2024-08-17T16:52:50.453 INFO:root:teuthology version: 1.2.0 2024-08-17T16:52:50.480 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-17T16:52:50.585 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-08-17_03:08:15-upgrade-main-distro-default-smithi/7858334 branch: main description: upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} email: ceph-qa@ceph.com first-half-sequence: - cephadm.shell: env: - sha1 mon.a: - ceph config set mgr mgr/cephadm/daemon_cache_timeout 60 - ceph config set global log_to_journald false --force - echo wait for mgr daemons to upgrade - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - echo wait for minority of mons to upgrade - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --limit 1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - sleep 60 - echo wait for majority of mons to upgrade - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --limit 1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - sleep 60 - echo wait for all mons to upgrade - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - sleep 60 - echo wait for half of osds to upgrade - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 4 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done first-half-tasks: - workunit: branch: reef clients: client.0: - rbd/test_librbd.sh env: RBD_FEATURES: '61' - print: '**** done rbd/test_librbd.sh 7-workload' first_in_suite: false job_id: '7858334' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'librbd C and C++ api tests ' - desc: 'run randomized correctness test for rados operations generate write load with rados bench ' - desc: 'run basic cls tests for rbd ' - desc: 'run basic import/export cli tests for rbd ' - desc: 'librbd C and C++ api tests ' - desc: 'randomized correctness test for rados operations on a replicated pool, using only reads, writes, and deletes ' - desc: 'randomized correctness test for rados operations on a replicated pool with snapshot operations ' - desc: 'run basic import/export cli tests for rbd ' name: teuthology-2024-08-17_03:08:15-upgrade-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true create_rbd_pool: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - mons down - mon down - MON_DOWN - out of quorum - PG_AVAILABILITY sha1: cd8bdbeb9fbdab5733aa3aaadc042c965473e453 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: cd8bdbeb9fbdab5733aa3aaadc042c965473e453 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main sha1: 91f88e4d132ac08dc53355fa3d9717a53236a3a9 owner: scheduled_teuthology@teuthology priority: 850 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b second-half-sequence: sequential: - cephadm.shell: env: - sha1 mon.a: - sleep 60 - echo wait for upgrade to complete - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - echo upgrade complete - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 second-half-tasks: - workunit: branch: reef clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' seed: 17 sha1: cd8bdbeb9fbdab5733aa3aaadc042c965473e453 sleep_before_teardown: 0 stress-tasks: - thrashosds: aggressive_pg_num_changes: false chance_force_recovery: 0 chance_pgnum_grow: 1 chance_pgpnum_fix: 1 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 chance_thrash_pg_upmap_items: 0 disable_objectstore_tool_tests: true timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - print: '**** done end radosbench.yaml' - workunit: branch: reef clients: client.0: - cls/test_cls_rbd.sh - print: '**** done cls/test_cls_rbd.sh 5-workload' - workunit: branch: reef clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' - workunit: branch: reef clients: client.0: - rbd/test_librbd.sh env: RBD_FEATURES: '61' - print: '**** done rbd/test_librbd.sh 7-workload' - full_sequential: - rados: clients: - client.0 objects: 500 op_weights: delete: 10 read: 45 write: 45 ops: 4000 write_append_excl: false - print: '**** done rados/readwrite 5-workload' - full_sequential: - rados: clients: - client.0 objects: 50 op_weights: delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 write_append_excl: false - print: '**** done rados/snaps-few-objects 5-workload' subset: 11/32 suite: upgrade suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_91f88e4d132ac08dc53355fa3d9717a53236a3a9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 91f88e4d132ac08dc53355fa3d9717a53236a3a9 targets: smithi136.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNywBWUEmqPINAQCxyBK25sFNFRHh0QdNHbsKrFm7Ynxt+AqbKtw0kwHZ7hMsSHN92pUmTgsA7u6PKFJns8l74c= smithi165.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOJqXeRDzgV4gLwS9vXiSF6o8lfcFUXnYB0+rHnuADbCiYB1Z9mz7N19h44GQ0Od+a/X3ncl8Nba4HIenUlnPKo= tasks: - pexec: all: - sudo dnf install runc nvmetcli nvme-cli -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: branch: reef exclude_packages: - ceph-volume - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:reef - cephadm.shell: mon.a: - ceph fs volume create foo - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph.healthy: null - print: '**** upgrading first half of cluster, with stress ****' - parallel: - first-half-tasks - first-half-sequence - print: '**** done upgrading first half of cluster ****' - ceph.healthy: null - print: '**** applying stress + thrashing to mixed-version cluster ****' - parallel: - stress-tasks - ceph.healthy: null - print: '**** finishing upgrade ****' - parallel: - second-half-tasks - second-half-sequence - ceph.healthy: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 67db276914d29505c9707d04f3a3b8fb9d8aee4b timestamp: 2024-08-17_03:08:15 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-08-17T16:52:50.586 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_91f88e4d132ac08dc53355fa3d9717a53236a3a9/qa; will attempt to use it 2024-08-17T16:52:50.587 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_91f88e4d132ac08dc53355fa3d9717a53236a3a9/qa/tasks 2024-08-17T16:52:50.587 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-08-17T16:52:50.588 INFO:teuthology.task.internal:Checking packages... 2024-08-17T16:52:50.612 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'cd8bdbeb9fbdab5733aa3aaadc042c965473e453' 2024-08-17T16:52:50.612 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-17T16:52:50.612 INFO:teuthology.packaging:ref: None 2024-08-17T16:52:50.612 INFO:teuthology.packaging:tag: None 2024-08-17T16:52:50.612 INFO:teuthology.packaging:branch: main 2024-08-17T16:52:50.612 INFO:teuthology.packaging:sha1: cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T16:52:50.613 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-08-17T16:52:50.844 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-5842.g4887e20f 2024-08-17T16:52:50.846 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-08-17T16:52:50.854 INFO:teuthology.task.internal:no buildpackages task found 2024-08-17T16:52:50.854 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-17T16:52:50.881 INFO:teuthology.task.internal:Saving configuration 2024-08-17T16:52:50.904 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-17T16:52:50.913 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-17T16:52:50.941 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-08-17_03:08:15-upgrade-main-distro-default-smithi/7858334', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-17 16:47:41.219058', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNywBWUEmqPINAQCxyBK25sFNFRHh0QdNHbsKrFm7Ynxt+AqbKtw0kwHZ7hMsSHN92pUmTgsA7u6PKFJns8l74c='} 2024-08-17T16:52:50.972 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi165.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-08-17_03:08:15-upgrade-main-distro-default-smithi/7858334', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-17 16:47:41.217423', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOJqXeRDzgV4gLwS9vXiSF6o8lfcFUXnYB0+rHnuADbCiYB1Z9mz7N19h44GQ0Od+a/X3ncl8Nba4HIenUlnPKo='} 2024-08-17T16:52:50.972 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-17T16:52:50.981 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-08-17T16:52:50.981 INFO:teuthology.task.internal:roles: ubuntu@smithi165.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-08-17T16:52:50.981 INFO:teuthology.run_tasks:Running task console_log... 2024-08-17T16:52:51.050 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f294fbf7ac0>, signals=[15]) 2024-08-17T16:52:51.051 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-17T16:52:51.056 INFO:teuthology.task.internal:Opening connections... 2024-08-17T16:52:51.056 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi136.front.sepia.ceph.com 2024-08-17T16:52:51.058 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-17T16:52:51.147 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi165.front.sepia.ceph.com 2024-08-17T16:52:51.148 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-17T16:52:51.229 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-17T16:52:51.235 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2024-08-17T16:52:51.253 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2024-08-17T16:52:51.253 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2024-08-17T16:52:51.310 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="CentOS Stream" 2024-08-17T16:52:51.311 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="9" 2024-08-17T16:52:51.311 INFO:teuthology.orchestra.run.smithi136.stdout:ID="centos" 2024-08-17T16:52:51.311 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="rhel fedora" 2024-08-17T16:52:51.311 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="9" 2024-08-17T16:52:51.311 INFO:teuthology.orchestra.run.smithi136.stdout:PLATFORM_ID="platform:el9" 2024-08-17T16:52:51.311 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-17T16:52:51.311 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31" 2024-08-17T16:52:51.311 INFO:teuthology.orchestra.run.smithi136.stdout:LOGO="fedora-logo-icon" 2024-08-17T16:52:51.311 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-17T16:52:51.311 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://centos.org/" 2024-08-17T16:52:51.311 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-08-17T16:52:51.311 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-17T16:52:51.312 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-17T16:52:51.312 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2024-08-17T16:52:51.343 DEBUG:teuthology.orchestra.run.smithi165:> uname -m 2024-08-17T16:52:51.361 INFO:teuthology.orchestra.run.smithi165.stdout:x86_64 2024-08-17T16:52:51.361 DEBUG:teuthology.orchestra.run.smithi165:> cat /etc/os-release 2024-08-17T16:52:51.418 INFO:teuthology.orchestra.run.smithi165.stdout:NAME="CentOS Stream" 2024-08-17T16:52:51.418 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION="9" 2024-08-17T16:52:51.418 INFO:teuthology.orchestra.run.smithi165.stdout:ID="centos" 2024-08-17T16:52:51.418 INFO:teuthology.orchestra.run.smithi165.stdout:ID_LIKE="rhel fedora" 2024-08-17T16:52:51.418 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_ID="9" 2024-08-17T16:52:51.418 INFO:teuthology.orchestra.run.smithi165.stdout:PLATFORM_ID="platform:el9" 2024-08-17T16:52:51.419 INFO:teuthology.orchestra.run.smithi165.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-17T16:52:51.419 INFO:teuthology.orchestra.run.smithi165.stdout:ANSI_COLOR="0;31" 2024-08-17T16:52:51.419 INFO:teuthology.orchestra.run.smithi165.stdout:LOGO="fedora-logo-icon" 2024-08-17T16:52:51.419 INFO:teuthology.orchestra.run.smithi165.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-17T16:52:51.419 INFO:teuthology.orchestra.run.smithi165.stdout:HOME_URL="https://centos.org/" 2024-08-17T16:52:51.419 INFO:teuthology.orchestra.run.smithi165.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-08-17T16:52:51.419 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-17T16:52:51.419 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-17T16:52:51.419 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server 2024-08-17T16:52:51.443 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-17T16:52:51.452 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-17T16:52:51.460 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-17T16:52:51.460 DEBUG:teuthology.orchestra.run.smithi136:> test '!' -e /home/ubuntu/cephtest 2024-08-17T16:52:51.463 DEBUG:teuthology.orchestra.run.smithi165:> test '!' -e /home/ubuntu/cephtest 2024-08-17T16:52:51.479 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-17T16:52:51.486 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-17T16:52:51.486 DEBUG:teuthology.orchestra.run.smithi136:> test -z $(ls -A /var/lib/ceph) 2024-08-17T16:52:51.519 DEBUG:teuthology.orchestra.run.smithi165:> test -z $(ls -A /var/lib/ceph) 2024-08-17T16:52:51.617 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-17T16:52:51.688 INFO:teuthology.run_tasks:Running task kernel... 2024-08-17T16:52:51.702 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-08-17T16:52:51.702 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-08-17T16:52:51.702 DEBUG:teuthology.orchestra.run.smithi136:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-17T16:52:51.703 DEBUG:teuthology.orchestra.run.smithi165:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-17T16:52:51.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T16:52:51.720 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-08-17T16:52:51.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T16:52:51.721 DEBUG:teuthology.orchestra.run.smithi165:> uname -r 2024-08-17T16:52:51.777 INFO:teuthology.orchestra.run.smithi136.stdout:5.14.0-437.el9.x86_64 2024-08-17T16:52:51.777 INFO:teuthology.task.kernel:Running kernel on smithi136: 5.14.0-437.el9.x86_64 2024-08-17T16:52:51.777 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum install -y kernel 2024-08-17T16:52:51.778 INFO:teuthology.orchestra.run.smithi165.stdout:5.14.0-437.el9.x86_64 2024-08-17T16:52:51.779 INFO:teuthology.task.kernel:Running kernel on smithi165: 5.14.0-437.el9.x86_64 2024-08-17T16:52:51.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum install -y kernel 2024-08-17T16:52:54.189 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.2 MB 00:00 2024-08-17T16:52:55.762 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - BaseOS 3.7 MB/s | 8.2 MB 00:02 2024-08-17T16:52:56.831 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - AppStream 13 MB/s | 20 MB 00:01 2024-08-17T16:52:58.879 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - AppStream 9.9 MB/s | 20 MB 00:02 2024-08-17T16:53:01.032 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.6 MB 00:00 2024-08-17T16:53:02.633 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - Extras packages 77 kB/s | 18 kB 00:00 2024-08-17T16:53:03.154 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.6 MB 00:00 2024-08-17T16:53:03.586 INFO:teuthology.orchestra.run.smithi165.stdout:Extra Packages for Enterprise Linux 27 MB/s | 22 MB 00:00 2024-08-17T16:53:04.896 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 18 kB 00:00 2024-08-17T16:53:06.076 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 21 MB/s | 22 MB 00:01 2024-08-17T16:53:08.983 INFO:teuthology.orchestra.run.smithi165.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-08-17T16:53:10.948 INFO:teuthology.orchestra.run.smithi165.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-17T16:53:10.948 INFO:teuthology.orchestra.run.smithi165.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-08-17T16:53:11.043 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T16:53:11.051 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T16:53:11.051 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-08-17T16:53:11.051 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T16:53:11.051 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-08-17T16:53:11.051 INFO:teuthology.orchestra.run.smithi165.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-17T16:53:11.051 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-08-17T16:53:11.051 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-08-17T16:53:11.051 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-08-17T16:53:11.051 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-08-17T16:53:11.052 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T16:53:11.052 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-08-17T16:53:11.052 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T16:53:11.052 INFO:teuthology.orchestra.run.smithi165.stdout:Install 4 Packages 2024-08-17T16:53:11.052 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T16:53:11.052 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 84 M 2024-08-17T16:53:11.052 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 126 M 2024-08-17T16:53:11.053 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-08-17T16:53:11.318 INFO:teuthology.orchestra.run.smithi165.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 6.6 MB/s | 1.5 MB 00:00 2024-08-17T16:53:11.439 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-08-17T16:53:11.585 INFO:teuthology.orchestra.run.smithi165.stdout:(2/4): kernel-core-5.14.0-480.el9.x86_64.rpm 34 MB/s | 17 MB 00:00 2024-08-17T16:53:11.977 INFO:teuthology.orchestra.run.smithi165.stdout:(3/4): kernel-modules-core-5.14.0-480.el9.x86_6 45 MB/s | 30 MB 00:00 2024-08-17T16:53:12.319 INFO:teuthology.orchestra.run.smithi165.stdout:(4/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 29 MB/s | 36 MB 00:01 2024-08-17T16:53:12.319 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:53:12.319 INFO:teuthology.orchestra.run.smithi165.stdout:Total 67 MB/s | 84 MB 00:01 2024-08-17T16:53:13.059 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-08-17T16:53:13.411 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-08-17T16:53:13.411 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-08-17T16:53:13.437 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-17T16:53:13.437 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-08-17T16:53:13.532 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T16:53:13.542 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T16:53:13.542 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-08-17T16:53:13.542 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T16:53:13.543 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-17T16:53:13.543 INFO:teuthology.orchestra.run.smithi136.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-17T16:53:13.543 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-08-17T16:53:13.543 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-08-17T16:53:13.543 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-08-17T16:53:13.543 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-08-17T16:53:13.543 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T16:53:13.543 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-17T16:53:13.543 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T16:53:13.543 INFO:teuthology.orchestra.run.smithi136.stdout:Install 4 Packages 2024-08-17T16:53:13.543 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T16:53:13.544 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 84 M 2024-08-17T16:53:13.544 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 126 M 2024-08-17T16:53:13.544 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-17T16:53:13.816 INFO:teuthology.orchestra.run.smithi136.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 6.5 MB/s | 1.5 MB 00:00 2024-08-17T16:53:14.091 INFO:teuthology.orchestra.run.smithi136.stdout:(2/4): kernel-core-5.14.0-480.el9.x86_64.rpm 34 MB/s | 17 MB 00:00 2024-08-17T16:53:14.407 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-08-17T16:53:14.408 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-08-17T16:53:14.474 INFO:teuthology.orchestra.run.smithi136.stdout:(3/4): kernel-modules-core-5.14.0-480.el9.x86_6 45 MB/s | 30 MB 00:00 2024-08-17T16:53:14.808 INFO:teuthology.orchestra.run.smithi136.stdout:(4/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 29 MB/s | 36 MB 00:01 2024-08-17T16:53:14.808 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:53:14.809 INFO:teuthology.orchestra.run.smithi136.stdout:Total 67 MB/s | 84 MB 00:01 2024-08-17T16:53:15.546 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-17T16:53:15.879 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-17T16:53:15.879 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-17T16:53:16.843 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-08-17T16:53:16.880 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-17T16:53:16.880 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-17T16:53:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-08-17T16:53:17.684 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-17T16:53:19.325 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-17T16:53:20.001 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-08-17T16:53:20.193 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-17T16:53:20.366 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-17T16:53:20.508 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-17T16:53:22.836 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-17T16:53:22.984 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-17T16:53:25.368 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-17T16:53:25.432 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-08-17T16:53:27.780 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-17T16:53:27.868 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-08-17T16:53:30.288 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-17T16:53:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-17T16:54:00.044 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-08-17T16:54:00.084 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-08-17T16:54:00.392 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-08-17T16:54:00.392 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-08-17T16:54:00.393 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-17T16:54:00.393 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-17T16:54:01.202 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-17T16:54:01.202 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T16:54:01.202 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-08-17T16:54:01.203 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-17T16:54:01.203 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-08-17T16:54:01.203 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-08-17T16:54:01.203 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-08-17T16:54:01.203 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T16:54:01.203 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T16:54:01.557 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q kernel | sort -rV | head -n 1 2024-08-17T16:54:01.679 INFO:teuthology.orchestra.run.smithi165.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-17T16:54:01.680 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-17T16:54:01.680 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-08-17T16:54:01.680 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-08-17T16:54:01.680 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-08-17T16:54:01.680 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-08-17T16:54:01.680 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi165.front.sepia.ceph.com, path=None, version=distro) 2024-08-17T16:54:01.680 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q kernel | sort -rV | head -n 1 2024-08-17T16:54:01.737 INFO:teuthology.orchestra.run.smithi165.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-17T16:54:01.737 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-17T16:54:01.737 DEBUG:teuthology.orchestra.run.smithi165:> sudo rpm -qi grub2-tools 2024-08-17T16:54:01.780 INFO:teuthology.orchestra.run.smithi165.stdout:Name : grub2-tools 2024-08-17T16:54:01.780 INFO:teuthology.orchestra.run.smithi165.stdout:Epoch : 1 2024-08-17T16:54:01.780 INFO:teuthology.orchestra.run.smithi165.stdout:Version : 2.06 2024-08-17T16:54:01.780 INFO:teuthology.orchestra.run.smithi165.stdout:Release : 68.el9 2024-08-17T16:54:01.780 INFO:teuthology.orchestra.run.smithi165.stdout:Architecture: x86_64 2024-08-17T16:54:01.781 INFO:teuthology.orchestra.run.smithi165.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-08-17T16:54:01.781 INFO:teuthology.orchestra.run.smithi165.stdout:Group : Unspecified 2024-08-17T16:54:01.781 INFO:teuthology.orchestra.run.smithi165.stdout:Size : 8264254 2024-08-17T16:54:01.781 INFO:teuthology.orchestra.run.smithi165.stdout:License : GPLv3+ 2024-08-17T16:54:01.781 INFO:teuthology.orchestra.run.smithi165.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-08-17T16:54:01.781 INFO:teuthology.orchestra.run.smithi165.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-08-17T16:54:01.781 INFO:teuthology.orchestra.run.smithi165.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-08-17T16:54:01.781 INFO:teuthology.orchestra.run.smithi165.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-08-17T16:54:01.781 INFO:teuthology.orchestra.run.smithi165.stdout:Packager : builder@centos.org 2024-08-17T16:54:01.781 INFO:teuthology.orchestra.run.smithi165.stdout:Vendor : CentOS 2024-08-17T16:54:01.781 INFO:teuthology.orchestra.run.smithi165.stdout:URL : http://www.gnu.org/software/grub/ 2024-08-17T16:54:01.781 INFO:teuthology.orchestra.run.smithi165.stdout:Summary : Support tools for GRUB. 2024-08-17T16:54:01.782 INFO:teuthology.orchestra.run.smithi165.stdout:Description : 2024-08-17T16:54:01.782 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T16:54:01.782 INFO:teuthology.orchestra.run.smithi165.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-08-17T16:54:01.782 INFO:teuthology.orchestra.run.smithi165.stdout:customizable bootloader with modular architecture. It supports a rich 2024-08-17T16:54:01.782 INFO:teuthology.orchestra.run.smithi165.stdout:variety of kernel formats, file systems, computer architectures and 2024-08-17T16:54:01.782 INFO:teuthology.orchestra.run.smithi165.stdout:hardware devices. 2024-08-17T16:54:01.782 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T16:54:01.782 INFO:teuthology.orchestra.run.smithi165.stdout:This subpackage provides tools for support of all platforms. 2024-08-17T16:54:01.783 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-08-17T16:54:01.783 INFO:teuthology.task.kernel:Updating grub on smithi165 to boot 5.14.0-480.el9.x86_64 2024-08-17T16:54:01.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-08-17T16:54:02.420 INFO:teuthology.orchestra.run.smithi165.stderr:Generating grub configuration file ... 2024-08-17T16:54:02.484 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-08-17T16:54:02.525 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-08-17T16:54:02.900 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-08-17T16:54:02.901 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-08-17T16:54:02.901 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-17T16:54:02.901 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-17T16:54:03.690 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-17T16:54:03.690 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T16:54:03.690 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-17T16:54:03.690 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-17T16:54:03.690 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-08-17T16:54:03.691 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-08-17T16:54:03.691 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-08-17T16:54:03.691 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T16:54:03.691 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T16:54:03.846 INFO:teuthology.orchestra.run.smithi165.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-08-17T16:54:03.874 INFO:teuthology.orchestra.run.smithi165.stderr:done 2024-08-17T16:54:03.876 DEBUG:teuthology.orchestra.run.smithi165:> mktemp 2024-08-17T16:54:03.891 INFO:teuthology.orchestra.run.smithi165.stdout:/tmp/tmp.C8hiBsC61Q 2024-08-17T16:54:03.892 DEBUG:teuthology.orchestra.run.smithi165:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.C8hiBsC61Q 2024-08-17T16:54:03.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0666 /tmp/tmp.C8hiBsC61Q 2024-08-17T16:54:04.098 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q kernel | sort -rV | head -n 1 2024-08-17T16:54:04.101 DEBUG:teuthology.orchestra.remote:smithi165:/tmp/tmp.C8hiBsC61Q is 6KB 2024-08-17T16:54:04.149 DEBUG:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.C8hiBsC61Q 2024-08-17T16:54:04.163 DEBUG:teuthology.orchestra.run.smithi165:> sudo /bin/ls /boot/loader/entries || true 2024-08-17T16:54:04.208 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-17T16:54:04.209 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-17T16:54:04.209 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-08-17T16:54:04.209 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-08-17T16:54:04.209 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-08-17T16:54:04.210 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-08-17T16:54:04.210 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi136.front.sepia.ceph.com, path=None, version=distro) 2024-08-17T16:54:04.210 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q kernel | sort -rV | head -n 1 2024-08-17T16:54:04.225 INFO:teuthology.orchestra.run.smithi165.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-08-17T16:54:04.226 INFO:teuthology.orchestra.run.smithi165.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-08-17T16:54:04.226 INFO:teuthology.orchestra.run.smithi165.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-08-17T16:54:04.226 INFO:teuthology.orchestra.run.smithi165.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-08-17T16:54:04.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-08-17T16:54:04.266 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-17T16:54:04.266 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-17T16:54:04.266 DEBUG:teuthology.orchestra.run.smithi136:> sudo rpm -qi grub2-tools 2024-08-17T16:54:04.296 INFO:teuthology.orchestra.run.smithi136.stdout:Name : grub2-tools 2024-08-17T16:54:04.296 INFO:teuthology.orchestra.run.smithi136.stdout:Epoch : 1 2024-08-17T16:54:04.296 INFO:teuthology.orchestra.run.smithi136.stdout:Version : 2.06 2024-08-17T16:54:04.297 INFO:teuthology.orchestra.run.smithi136.stdout:Release : 68.el9 2024-08-17T16:54:04.297 INFO:teuthology.orchestra.run.smithi136.stdout:Architecture: x86_64 2024-08-17T16:54:04.297 INFO:teuthology.orchestra.run.smithi136.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-08-17T16:54:04.297 INFO:teuthology.orchestra.run.smithi136.stdout:Group : Unspecified 2024-08-17T16:54:04.297 INFO:teuthology.orchestra.run.smithi136.stdout:Size : 8264254 2024-08-17T16:54:04.297 INFO:teuthology.orchestra.run.smithi136.stdout:License : GPLv3+ 2024-08-17T16:54:04.297 INFO:teuthology.orchestra.run.smithi136.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-08-17T16:54:04.297 INFO:teuthology.orchestra.run.smithi136.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-08-17T16:54:04.297 INFO:teuthology.orchestra.run.smithi136.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-08-17T16:54:04.297 INFO:teuthology.orchestra.run.smithi136.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-08-17T16:54:04.297 INFO:teuthology.orchestra.run.smithi136.stdout:Packager : builder@centos.org 2024-08-17T16:54:04.297 INFO:teuthology.orchestra.run.smithi136.stdout:Vendor : CentOS 2024-08-17T16:54:04.297 INFO:teuthology.orchestra.run.smithi136.stdout:URL : http://www.gnu.org/software/grub/ 2024-08-17T16:54:04.298 INFO:teuthology.orchestra.run.smithi136.stdout:Summary : Support tools for GRUB. 2024-08-17T16:54:04.298 INFO:teuthology.orchestra.run.smithi136.stdout:Description : 2024-08-17T16:54:04.298 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T16:54:04.298 INFO:teuthology.orchestra.run.smithi136.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-08-17T16:54:04.298 INFO:teuthology.orchestra.run.smithi136.stdout:customizable bootloader with modular architecture. It supports a rich 2024-08-17T16:54:04.298 INFO:teuthology.orchestra.run.smithi136.stdout:variety of kernel formats, file systems, computer architectures and 2024-08-17T16:54:04.298 INFO:teuthology.orchestra.run.smithi136.stdout:hardware devices. 2024-08-17T16:54:04.298 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T16:54:04.298 INFO:teuthology.orchestra.run.smithi136.stdout:This subpackage provides tools for support of all platforms. 2024-08-17T16:54:04.299 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-08-17T16:54:04.299 INFO:teuthology.task.kernel:Updating grub on smithi136 to boot 5.14.0-480.el9.x86_64 2024-08-17T16:54:04.299 DEBUG:teuthology.orchestra.run.smithi136:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-08-17T16:54:04.376 DEBUG:teuthology.orchestra.run.smithi165:> sudo shutdown -r now 2024-08-17T16:54:04.900 INFO:teuthology.orchestra.run.smithi136.stderr:Generating grub configuration file ... 2024-08-17T16:54:06.357 INFO:teuthology.orchestra.run.smithi136.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-08-17T16:54:06.397 INFO:teuthology.orchestra.run.smithi136.stderr:done 2024-08-17T16:54:06.399 DEBUG:teuthology.orchestra.run.smithi136:> mktemp 2024-08-17T16:54:06.420 INFO:teuthology.orchestra.run.smithi136.stdout:/tmp/tmp.ZLENzT8UeU 2024-08-17T16:54:06.420 DEBUG:teuthology.orchestra.run.smithi136:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ZLENzT8UeU 2024-08-17T16:54:06.487 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0666 /tmp/tmp.ZLENzT8UeU 2024-08-17T16:54:06.632 DEBUG:teuthology.orchestra.remote:smithi136:/tmp/tmp.ZLENzT8UeU is 6KB 2024-08-17T16:54:06.652 DEBUG:teuthology.orchestra.run.smithi136:> rm -fr /tmp/tmp.ZLENzT8UeU 2024-08-17T16:54:06.666 DEBUG:teuthology.orchestra.run.smithi136:> sudo /bin/ls /boot/loader/entries || true 2024-08-17T16:54:06.727 INFO:teuthology.orchestra.run.smithi136.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-08-17T16:54:06.727 INFO:teuthology.orchestra.run.smithi136.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-08-17T16:54:06.727 INFO:teuthology.orchestra.run.smithi136.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-08-17T16:54:06.727 INFO:teuthology.orchestra.run.smithi136.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-08-17T16:54:06.728 DEBUG:teuthology.orchestra.run.smithi136:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-08-17T16:54:06.882 DEBUG:teuthology.orchestra.run.smithi136:> sudo shutdown -r now 2024-08-17T16:54:34.406 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-08-17T16:54:34.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-08-17T16:54:34.408 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-17T16:54:36.886 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-08-17T16:54:36.886 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-08-17T16:54:36.887 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-17T16:55:09.027 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.165 2024-08-17T16:55:18.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-08-17T16:55:18.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-17T16:55:36.415 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.165 2024-08-17T16:55:36.898 DEBUG:teuthology.orchestra.remote:timed out 2024-08-17T16:55:45.907 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-08-17T16:55:45.909 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-17T16:55:48.418 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-08-17T16:55:48.419 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-17T16:55:49.312 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-08-17T16:55:49.774 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-08-17T16:55:49.774 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-480.el9.x86_64"... 2024-08-17T16:55:49.775 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-08-17T16:55:49.789 INFO:teuthology.orchestra.run.smithi136.stdout:5.14.0-480.el9.x86_64 2024-08-17T16:55:49.790 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-08-17T16:55:49.790 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-17T16:55:49.790 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-08-17T16:55:50.790 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-08-17T16:55:50.791 DEBUG:teuthology.orchestra.run.smithi136:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-17T16:55:50.871 INFO:teuthology.orchestra.run.smithi136.stdout:ttyS1 2024-08-17T16:55:50.918 DEBUG:teuthology.parallel:result is None 2024-08-17T16:55:50.943 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-08-17T16:55:51.412 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-08-17T16:55:51.413 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-480.el9.x86_64"... 2024-08-17T16:55:51.413 DEBUG:teuthology.orchestra.run.smithi165:> uname -r 2024-08-17T16:55:51.427 INFO:teuthology.orchestra.run.smithi165.stdout:5.14.0-480.el9.x86_64 2024-08-17T16:55:51.428 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-08-17T16:55:51.428 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-17T16:55:51.428 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-08-17T16:55:52.429 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-08-17T16:55:52.430 DEBUG:teuthology.orchestra.run.smithi165:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-17T16:55:52.530 INFO:teuthology.orchestra.run.smithi165.stdout:ttyS1 2024-08-17T16:55:52.559 DEBUG:teuthology.parallel:result is None 2024-08-17T16:55:52.560 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-17T16:55:52.569 INFO:teuthology.task.internal:Creating test directory... 2024-08-17T16:55:52.569 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-17T16:55:52.572 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-17T16:55:52.587 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-17T16:55:52.595 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-17T16:55:52.602 INFO:teuthology.task.internal:Creating archive directory... 2024-08-17T16:55:52.602 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-17T16:55:52.632 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-17T16:55:52.659 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-17T16:55:52.665 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-17T16:55:52.666 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-17T16:55:52.699 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-17T16:55:52.733 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-17T16:55:52.737 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-17T16:55:52.743 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-17T16:55:52.747 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-17T16:55:52.749 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-17T16:55:52.758 INFO:teuthology.task.internal:Configuring sudo... 2024-08-17T16:55:52.759 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-17T16:55:52.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-17T16:55:52.818 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-17T16:55:52.827 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-17T16:55:52.828 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-17T16:55:52.859 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-17T16:55:52.875 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-17T16:55:52.966 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-17T16:55:53.011 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-17T16:55:53.100 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-17T16:55:53.141 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-17T16:55:53.142 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-17T16:55:53.210 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-17T16:55:53.261 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-17T16:55:53.301 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-17T16:55:53.392 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-17T16:55:53.424 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-08-17T16:55:53.424 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-17T16:55:53.490 DEBUG:teuthology.orchestra.run.smithi136:> sudo service rsyslog restart 2024-08-17T16:55:53.492 DEBUG:teuthology.orchestra.run.smithi165:> sudo service rsyslog restart 2024-08-17T16:55:53.536 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-17T16:55:53.566 INFO:teuthology.orchestra.run.smithi165.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-17T16:55:54.021 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-17T16:55:54.029 INFO:teuthology.task.internal:Starting timer... 2024-08-17T16:55:54.030 INFO:teuthology.run_tasks:Running task pcp... 2024-08-17T16:55:54.075 INFO:teuthology.run_tasks:Running task selinux... 2024-08-17T16:55:54.084 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-08-17T16:55:54.084 DEBUG:teuthology.orchestra.run.smithi136:> sudo service auditd rotate 2024-08-17T16:55:54.267 INFO:teuthology.orchestra.run.smithi136.stdout:Rotating logs: 2024-08-17T16:55:54.269 DEBUG:teuthology.orchestra.run.smithi165:> sudo service auditd rotate 2024-08-17T16:55:54.346 INFO:teuthology.orchestra.run.smithi165.stdout:Rotating logs: 2024-08-17T16:55:54.348 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-17T16:55:54.348 DEBUG:teuthology.orchestra.run.smithi136:> /usr/sbin/getenforce 2024-08-17T16:55:54.375 INFO:teuthology.orchestra.run.smithi136.stdout:Permissive 2024-08-17T16:55:54.375 DEBUG:teuthology.orchestra.run.smithi165:> /usr/sbin/getenforce 2024-08-17T16:55:54.397 INFO:teuthology.orchestra.run.smithi165.stdout:Permissive 2024-08-17T16:55:54.398 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi136.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi165.front.sepia.ceph.com': 'permissive'} 2024-08-17T16:55:54.398 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-08-17T16:55:54.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T16:55:54.467 DEBUG:teuthology.orchestra.run.smithi165:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-08-17T16:55:54.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T16:55:54.493 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-17T16:55:54.493 DEBUG:teuthology.orchestra.run.smithi136:> sudo /usr/sbin/setenforce permissive 2024-08-17T16:55:54.792 DEBUG:teuthology.orchestra.run.smithi165:> sudo /usr/sbin/setenforce permissive 2024-08-17T16:55:54.817 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-17T16:55:54.826 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-08-17T16:55:54.829 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-08-17T16:55:54.979 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-17T16:55:54.990 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-17T16:55:54.991 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi136.front.sepia.ceph.com,smithi165.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-17T17:00:30.196 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi136.front.sepia.ceph.com'), Remote(name='ubuntu@smithi165.front.sepia.ceph.com')] 2024-08-17T17:00:30.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-08-17T17:00:30.199 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-17T17:00:30.282 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-08-17T17:00:30.361 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-08-17T17:00:30.361 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-08-17T17:00:30.361 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-17T17:00:30.482 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-08-17T17:00:30.514 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-08-17T17:00:30.514 INFO:teuthology.run_tasks:Running task clock... 2024-08-17T17:00:30.524 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-17T17:00:30.524 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-17T17:00:30.525 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-17T17:00:30.527 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-17T17:00:30.527 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-17T17:00:30.554 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-17T17:00:30.568 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-17T17:00:30.586 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-17T17:00:30.592 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: ntpd: command not found 2024-08-17T17:00:30.598 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-17T17:00:30.603 INFO:teuthology.orchestra.run.smithi136.stdout:506 Cannot talk to daemon 2024-08-17T17:00:30.617 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-17T17:00:30.622 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: ntpd: command not found 2024-08-17T17:00:30.631 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-17T17:00:30.633 INFO:teuthology.orchestra.run.smithi165.stdout:506 Cannot talk to daemon 2024-08-17T17:00:30.647 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-17T17:00:30.662 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-17T17:00:30.693 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-08-17T17:00:30.697 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-17T17:00:30.697 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-08-17T17:00:30.697 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-17T17:00:30.697 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-17T17:00:30.697 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-17T17:00:30.697 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-17T17:00:30.720 INFO:teuthology.orchestra.run.smithi165.stderr:bash: line 1: ntpq: command not found 2024-08-17T17:00:30.723 INFO:teuthology.orchestra.run.smithi165.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-17T17:00:30.723 INFO:teuthology.orchestra.run.smithi165.stdout:=============================================================================== 2024-08-17T17:00:30.723 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-17T17:00:30.723 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-17T17:00:30.723 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-17T17:00:30.723 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-17T17:00:30.724 INFO:teuthology.run_tasks:Running task pexec... 2024-08-17T17:00:30.734 INFO:teuthology.task.pexec:Executing custom commands... 2024-08-17T17:00:30.734 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi136.front.sepia.ceph.com 2024-08-17T17:00:30.734 DEBUG:teuthology.orchestra.run.smithi136:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-08-17T17:00:30.735 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi165.front.sepia.ceph.com 2024-08-17T17:00:30.735 DEBUG:teuthology.orchestra.run.smithi165:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-08-17T17:00:31.306 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:14 ago on Sat 17 Aug 2024 04:59:17 PM UTC. 2024-08-17T17:00:31.335 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:14 ago on Sat 17 Aug 2024 04:59:17 PM UTC. 2024-08-17T17:00:31.393 INFO:teuthology.orchestra.run.smithi136.stdout:Package nvme-cli-2.6-5.el9.x86_64 is already installed. 2024-08-17T17:00:31.421 INFO:teuthology.orchestra.run.smithi165.stdout:Package nvme-cli-2.6-5.el9.x86_64 is already installed. 2024-08-17T17:00:31.433 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:00:31.446 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:00:31.446 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-08-17T17:00:31.446 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:00:31.446 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-17T17:00:31.446 INFO:teuthology.orchestra.run.smithi136.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-08-17T17:00:31.446 INFO:teuthology.orchestra.run.smithi136.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-08-17T17:00:31.446 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrading: 2024-08-17T17:00:31.446 INFO:teuthology.orchestra.run.smithi136.stdout: libnvme x86_64 1.9-2.el9 baseos 106 k 2024-08-17T17:00:31.446 INFO:teuthology.orchestra.run.smithi136.stdout: nvme-cli x86_64 2.9.1-3.el9 baseos 903 k 2024-08-17T17:00:31.446 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-08-17T17:00:31.446 INFO:teuthology.orchestra.run.smithi136.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-08-17T17:00:31.446 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-08-17T17:00:31.446 INFO:teuthology.orchestra.run.smithi136.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-08-17T17:00:31.447 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:00:31.447 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-17T17:00:31.447 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:00:31.447 INFO:teuthology.orchestra.run.smithi136.stdout:Install 5 Packages 2024-08-17T17:00:31.447 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrade 2 Packages 2024-08-17T17:00:31.447 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:00:31.449 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 5.2 M 2024-08-17T17:00:31.449 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-17T17:00:31.462 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:00:31.477 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:00:31.478 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-08-17T17:00:31.478 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:00:31.478 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-08-17T17:00:31.478 INFO:teuthology.orchestra.run.smithi165.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-08-17T17:00:31.478 INFO:teuthology.orchestra.run.smithi165.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-08-17T17:00:31.478 INFO:teuthology.orchestra.run.smithi165.stdout:Upgrading: 2024-08-17T17:00:31.478 INFO:teuthology.orchestra.run.smithi165.stdout: libnvme x86_64 1.9-2.el9 baseos 106 k 2024-08-17T17:00:31.478 INFO:teuthology.orchestra.run.smithi165.stdout: nvme-cli x86_64 2.9.1-3.el9 baseos 903 k 2024-08-17T17:00:31.478 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-08-17T17:00:31.478 INFO:teuthology.orchestra.run.smithi165.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-08-17T17:00:31.478 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-08-17T17:00:31.478 INFO:teuthology.orchestra.run.smithi165.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-08-17T17:00:31.479 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:00:31.479 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-08-17T17:00:31.479 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:00:31.479 INFO:teuthology.orchestra.run.smithi165.stdout:Install 5 Packages 2024-08-17T17:00:31.479 INFO:teuthology.orchestra.run.smithi165.stdout:Upgrade 2 Packages 2024-08-17T17:00:31.479 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:00:31.488 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 5.2 M 2024-08-17T17:00:31.488 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-08-17T17:00:31.942 INFO:teuthology.orchestra.run.smithi136.stdout:(1/7): python3-kmod-0.9-32.el9.x86_64.rpm 499 kB/s | 84 kB 00:00 2024-08-17T17:00:31.967 INFO:teuthology.orchestra.run.smithi136.stdout:(2/7): nvmetcli-0.7-3.el9.noarch.rpm 229 kB/s | 44 kB 00:00 2024-08-17T17:00:31.993 INFO:teuthology.orchestra.run.smithi136.stdout:(3/7): python3-configshell-1.1.30-1.el9.noarch. 327 kB/s | 72 kB 00:00 2024-08-17T17:00:32.085 INFO:teuthology.orchestra.run.smithi136.stdout:(4/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 5.8 MB/s | 837 kB 00:00 2024-08-17T17:00:32.110 INFO:teuthology.orchestra.run.smithi136.stdout:(5/7): libnvme-1.9-2.el9.x86_64.rpm 906 kB/s | 106 kB 00:00 2024-08-17T17:00:32.161 INFO:teuthology.orchestra.run.smithi165.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 163 kB/s | 44 kB 00:00 2024-08-17T17:00:32.177 INFO:teuthology.orchestra.run.smithi136.stdout:(6/7): nvme-cli-2.9.1-3.el9.x86_64.rpm 9.6 MB/s | 903 kB 00:00 2024-08-17T17:00:32.194 INFO:teuthology.orchestra.run.smithi165.stdout:(2/7): python3-configshell-1.1.30-1.el9.noarch. 233 kB/s | 72 kB 00:00 2024-08-17T17:00:32.270 INFO:teuthology.orchestra.run.smithi165.stdout:(3/7): python3-kmod-0.9-32.el9.x86_64.rpm 220 kB/s | 84 kB 00:00 2024-08-17T17:00:32.570 INFO:teuthology.orchestra.run.smithi165.stdout:(4/7): libnvme-1.9-2.el9.x86_64.rpm 355 kB/s | 106 kB 00:00 2024-08-17T17:00:32.603 INFO:teuthology.orchestra.run.smithi165.stdout:(5/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 1.9 MB/s | 837 kB 00:00 2024-08-17T17:00:32.970 INFO:teuthology.orchestra.run.smithi165.stdout:(6/7): runc-1.1.13-4.el9.x86_64.rpm 4.1 MB/s | 3.2 MB 00:00 2024-08-17T17:00:33.012 INFO:teuthology.orchestra.run.smithi165.stdout:(7/7): nvme-cli-2.9.1-3.el9.x86_64.rpm 2.0 MB/s | 903 kB 00:00 2024-08-17T17:00:33.012 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:00:33.012 INFO:teuthology.orchestra.run.smithi165.stdout:Total 3.4 MB/s | 5.2 MB 00:01 2024-08-17T17:00:33.119 INFO:teuthology.orchestra.run.smithi136.stdout:(7/7): runc-1.1.13-4.el9.x86_64.rpm 2.7 MB/s | 3.2 MB 00:01 2024-08-17T17:00:33.119 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:00:33.119 INFO:teuthology.orchestra.run.smithi136.stdout:Total 3.1 MB/s | 5.2 MB 00:01 2024-08-17T17:00:33.122 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-08-17T17:00:33.139 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-08-17T17:00:33.140 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-08-17T17:00:33.231 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-17T17:00:33.248 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-17T17:00:33.248 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-17T17:00:33.367 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-08-17T17:00:33.368 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-08-17T17:00:33.470 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-17T17:00:33.471 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-17T17:00:33.775 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-08-17T17:00:33.863 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-17T17:00:33.987 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : libnvme-1.9-2.el9.x86_64 1/9 2024-08-17T17:00:34.050 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 2/9 2024-08-17T17:00:34.085 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 3/9 2024-08-17T17:00:34.089 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : libnvme-1.9-2.el9.x86_64 1/9 2024-08-17T17:00:34.117 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 4/9 2024-08-17T17:00:34.148 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 2/9 2024-08-17T17:00:34.151 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : nvmetcli-0.7-3.el9.noarch 5/9 2024-08-17T17:00:34.183 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 3/9 2024-08-17T17:00:34.216 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 4/9 2024-08-17T17:00:34.250 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : nvmetcli-0.7-3.el9.noarch 5/9 2024-08-17T17:00:34.433 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 5/9 2024-08-17T17:00:34.495 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : nvme-cli-2.9.1-3.el9.x86_64 6/9 2024-08-17T17:00:34.517 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 5/9 2024-08-17T17:00:34.578 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : nvme-cli-2.9.1-3.el9.x86_64 6/9 2024-08-17T17:00:34.880 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: nvme-cli-2.9.1-3.el9.x86_64 6/9 2024-08-17T17:00:34.916 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/9 2024-08-17T17:00:34.948 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : nvme-cli-2.6-5.el9.x86_64 8/9 2024-08-17T17:00:34.959 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: nvme-cli-2.9.1-3.el9.x86_64 6/9 2024-08-17T17:00:34.990 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/9 2024-08-17T17:00:35.010 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : libnvme-1.6-1.el9.x86_64 9/9 2024-08-17T17:00:35.023 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : nvme-cli-2.6-5.el9.x86_64 8/9 2024-08-17T17:00:35.083 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : libnvme-1.6-1.el9.x86_64 9/9 2024-08-17T17:00:35.979 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libnvme-1.6-1.el9.x86_64 9/9 2024-08-17T17:00:35.979 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 1/9 2024-08-17T17:00:35.979 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 2/9 2024-08-17T17:00:35.979 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 3/9 2024-08-17T17:00:35.979 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 4/9 2024-08-17T17:00:35.979 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 5/9 2024-08-17T17:00:35.979 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libnvme-1.9-2.el9.x86_64 6/9 2024-08-17T17:00:35.979 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libnvme-1.6-1.el9.x86_64 7/9 2024-08-17T17:00:35.979 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : nvme-cli-2.9.1-3.el9.x86_64 8/9 2024-08-17T17:00:36.002 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libnvme-1.6-1.el9.x86_64 9/9 2024-08-17T17:00:36.002 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 1/9 2024-08-17T17:00:36.002 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 2/9 2024-08-17T17:00:36.002 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 3/9 2024-08-17T17:00:36.002 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 4/9 2024-08-17T17:00:36.002 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 5/9 2024-08-17T17:00:36.002 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libnvme-1.9-2.el9.x86_64 6/9 2024-08-17T17:00:36.002 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libnvme-1.6-1.el9.x86_64 7/9 2024-08-17T17:00:36.002 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : nvme-cli-2.9.1-3.el9.x86_64 8/9 2024-08-17T17:00:36.328 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : nvme-cli-2.6-5.el9.x86_64 9/9 2024-08-17T17:00:36.329 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:00:36.329 INFO:teuthology.orchestra.run.smithi165.stdout:Upgraded: 2024-08-17T17:00:36.329 INFO:teuthology.orchestra.run.smithi165.stdout: libnvme-1.9-2.el9.x86_64 nvme-cli-2.9.1-3.el9.x86_64 2024-08-17T17:00:36.329 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-08-17T17:00:36.329 INFO:teuthology.orchestra.run.smithi165.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-08-17T17:00:36.329 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-08-17T17:00:36.329 INFO:teuthology.orchestra.run.smithi165.stdout: runc-4:1.1.13-4.el9.x86_64 2024-08-17T17:00:36.329 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:00:36.329 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:00:36.371 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : nvme-cli-2.6-5.el9.x86_64 9/9 2024-08-17T17:00:36.371 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:00:36.371 INFO:teuthology.orchestra.run.smithi136.stdout:Upgraded: 2024-08-17T17:00:36.371 INFO:teuthology.orchestra.run.smithi136.stdout: libnvme-1.9-2.el9.x86_64 nvme-cli-2.9.1-3.el9.x86_64 2024-08-17T17:00:36.371 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-17T17:00:36.371 INFO:teuthology.orchestra.run.smithi136.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-08-17T17:00:36.371 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-08-17T17:00:36.371 INFO:teuthology.orchestra.run.smithi136.stdout: runc-4:1.1.13-4.el9.x86_64 2024-08-17T17:00:36.372 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:00:36.372 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:00:36.712 DEBUG:teuthology.parallel:result is None 2024-08-17T17:00:36.770 DEBUG:teuthology.parallel:result is None 2024-08-17T17:00:36.770 INFO:teuthology.run_tasks:Running task install... 2024-08-17T17:00:36.779 DEBUG:teuthology.task.install:project ceph 2024-08-17T17:00:36.779 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'cd8bdbeb9fbdab5733aa3aaadc042c965473e453'}} 2024-08-17T17:00:36.780 DEBUG:teuthology.task.install:config {'branch': 'reef', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': 'cd8bdbeb9fbdab5733aa3aaadc042c965473e453'} 2024-08-17T17:00:36.780 INFO:teuthology.task.install:Using flavor: default 2024-08-17T17:00:36.785 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-08-17T17:00:36.785 INFO:teuthology.task.install:extra packages: [] 2024-08-17T17:00:36.785 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'cd8bdbeb9fbdab5733aa3aaadc042c965473e453', 'tag': None, 'wait_for_package': False} 2024-08-17T17:00:36.785 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-17T17:00:36.785 INFO:teuthology.packaging:ref: None 2024-08-17T17:00:36.785 INFO:teuthology.packaging:tag: None 2024-08-17T17:00:36.786 INFO:teuthology.packaging:branch: reef 2024-08-17T17:00:36.786 INFO:teuthology.packaging:sha1: cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:00:36.786 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-08-17T17:00:36.788 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'cd8bdbeb9fbdab5733aa3aaadc042c965473e453', 'tag': None, 'wait_for_package': False} 2024-08-17T17:00:36.788 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-17T17:00:36.788 INFO:teuthology.packaging:ref: None 2024-08-17T17:00:36.788 INFO:teuthology.packaging:tag: None 2024-08-17T17:00:36.788 INFO:teuthology.packaging:branch: reef 2024-08-17T17:00:36.788 INFO:teuthology.packaging:sha1: cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:00:36.789 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-08-17T17:00:37.029 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/reef/ced649f89259a8c5733cc18a50f42f20f41ba352/centos/9/flavors/default/ 2024-08-17T17:00:37.029 INFO:teuthology.task.install.rpm:Package version is 18.2.4-719.gced649f8 2024-08-17T17:00:37.064 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/reef/ced649f89259a8c5733cc18a50f42f20f41ba352/centos/9/flavors/default/ 2024-08-17T17:00:37.064 INFO:teuthology.task.install.rpm:Package version is 18.2.4-719.gced649f8 2024-08-17T17:00:37.161 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/reef/ced649f89259a8c5733cc18a50f42f20f41ba352/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/ced649f89259a8c5733cc18a50f42f20f41ba352/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/ced649f89259a8c5733cc18a50f42f20f41ba352/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-17T17:00:37.162 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-17T17:00:37.162 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-17T17:00:37.181 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/reef/ced649f89259a8c5733cc18a50f42f20f41ba352/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/ced649f89259a8c5733cc18a50f42f20f41ba352/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/ced649f89259a8c5733cc18a50f42f20f41ba352/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-17T17:00:37.182 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-08-17T17:00:37.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-17T17:00:37.188 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-17T17:00:37.188 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-17T17:00:37.188 INFO:teuthology.packaging:ref: None 2024-08-17T17:00:37.188 INFO:teuthology.packaging:tag: None 2024-08-17T17:00:37.188 INFO:teuthology.packaging:branch: reef 2024-08-17T17:00:37.188 INFO:teuthology.packaging:sha1: cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:00:37.188 DEBUG:teuthology.orchestra.run.smithi136:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-17T17:00:37.227 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-17T17:00:37.227 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-17T17:00:37.227 INFO:teuthology.packaging:ref: None 2024-08-17T17:00:37.227 INFO:teuthology.packaging:tag: None 2024-08-17T17:00:37.227 INFO:teuthology.packaging:branch: reef 2024-08-17T17:00:37.227 INFO:teuthology.packaging:sha1: cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:00:37.228 DEBUG:teuthology.orchestra.run.smithi165:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-17T17:00:37.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-17T17:00:37.293 DEBUG:teuthology.orchestra.run.smithi165:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-17T17:00:37.349 DEBUG:teuthology.orchestra.run.smithi136:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-17T17:00:37.368 DEBUG:teuthology.orchestra.run.smithi165:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-17T17:00:37.414 INFO:teuthology.orchestra.run.smithi136.stdout:check_obsoletes = 1 2024-08-17T17:00:37.415 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum clean all 2024-08-17T17:00:37.433 INFO:teuthology.orchestra.run.smithi165.stdout:check_obsoletes = 1 2024-08-17T17:00:37.434 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum clean all 2024-08-17T17:00:37.652 INFO:teuthology.orchestra.run.smithi136.stdout:45 files removed 2024-08-17T17:00:37.674 INFO:teuthology.orchestra.run.smithi165.stdout:45 files removed 2024-08-17T17:00:37.682 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-radosgw 2024-08-17T17:00:37.701 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-radosgw 2024-08-17T17:00:38.273 INFO:teuthology.orchestra.run.smithi136.stdout:ceph packages for x86_64 248 kB/s | 77 kB 00:00 2024-08-17T17:00:38.321 INFO:teuthology.orchestra.run.smithi165.stdout:ceph packages for x86_64 228 kB/s | 77 kB 00:00 2024-08-17T17:00:38.598 INFO:teuthology.orchestra.run.smithi136.stdout:ceph noarch packages 42 kB/s | 11 kB 00:00 2024-08-17T17:00:38.637 INFO:teuthology.orchestra.run.smithi165.stdout:ceph noarch packages 43 kB/s | 11 kB 00:00 2024-08-17T17:00:38.918 INFO:teuthology.orchestra.run.smithi136.stdout:ceph source packages 6.9 kB/s | 1.9 kB 00:00 2024-08-17T17:00:38.963 INFO:teuthology.orchestra.run.smithi165.stdout:ceph source packages 6.7 kB/s | 1.9 kB 00:00 2024-08-17T17:00:39.385 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.2 MB 00:00 2024-08-17T17:00:39.463 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.2 MB 00:00 2024-08-17T17:00:41.369 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - AppStream 23 MB/s | 20 MB 00:00 2024-08-17T17:00:42.831 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - AppStream 8.9 MB/s | 20 MB 00:02 2024-08-17T17:00:45.661 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.6 MB 00:00 2024-08-17T17:00:47.345 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 18 kB 00:00 2024-08-17T17:00:47.373 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - CRB 9.7 MB/s | 6.6 MB 00:00 2024-08-17T17:00:48.349 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 26 MB/s | 22 MB 00:00 2024-08-17T17:00:48.882 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 18 kB 00:00 2024-08-17T17:00:49.836 INFO:teuthology.orchestra.run.smithi165.stdout:Extra Packages for Enterprise Linux 27 MB/s | 22 MB 00:00 2024-08-17T17:00:53.771 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-08-17T17:00:55.300 INFO:teuthology.orchestra.run.smithi165.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-08-17T17:00:56.113 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:00:56.116 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:00:56.116 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-08-17T17:00:56.116 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:00:56.116 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-17T17:00:56.116 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw x86_64 2:18.2.4-719.gced649f8.el9 ceph 7.7 M 2024-08-17T17:00:56.116 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrading: 2024-08-17T17:00:56.116 INFO:teuthology.orchestra.run.smithi136.stdout: librados2 x86_64 2:18.2.4-719.gced649f8.el9 ceph 3.3 M 2024-08-17T17:00:56.116 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1 x86_64 2:18.2.4-719.gced649f8.el9 ceph 3.0 M 2024-08-17T17:00:56.116 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-08-17T17:00:56.116 INFO:teuthology.orchestra.run.smithi136.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-17T17:00:56.116 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base x86_64 2:18.2.4-719.gced649f8.el9 ceph 5.1 M 2024-08-17T17:00:56.117 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common x86_64 2:18.2.4-719.gced649f8.el9 ceph 18 M 2024-08-17T17:00:56.117 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux x86_64 2:18.2.4-719.gced649f8.el9 ceph 25 k 2024-08-17T17:00:56.117 INFO:teuthology.orchestra.run.smithi136.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-08-17T17:00:56.117 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-17T17:00:56.117 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-17T17:00:56.117 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-17T17:00:56.117 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2 x86_64 2:18.2.4-719.gced649f8.el9 ceph 712 k 2024-08-17T17:00:56.117 INFO:teuthology.orchestra.run.smithi136.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-17T17:00:56.117 INFO:teuthology.orchestra.run.smithi136.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-17T17:00:56.117 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-17T17:00:56.117 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1 x86_64 2:18.2.4-719.gced649f8.el9 ceph 477 k 2024-08-17T17:00:56.117 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-17T17:00:56.117 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2 x86_64 2:18.2.4-719.gced649f8.el9 ceph 4.5 M 2024-08-17T17:00:56.118 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-17T17:00:56.118 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-17T17:00:56.118 INFO:teuthology.orchestra.run.smithi136.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-17T17:00:56.118 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse x86_64 2:18.2.4-719.gced649f8.el9 ceph 45 k 2024-08-17T17:00:56.118 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common x86_64 2:18.2.4-719.gced649f8.el9 ceph 129 k 2024-08-17T17:00:56.118 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs x86_64 2:18.2.4-719.gced649f8.el9 ceph 188 k 2024-08-17T17:00:56.118 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-17T17:00:56.118 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados x86_64 2:18.2.4-719.gced649f8.el9 ceph 372 k 2024-08-17T17:00:56.118 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd x86_64 2:18.2.4-719.gced649f8.el9 ceph 347 k 2024-08-17T17:00:56.118 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw x86_64 2:18.2.4-719.gced649f8.el9 ceph 116 k 2024-08-17T17:00:56.119 INFO:teuthology.orchestra.run.smithi136.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-08-17T17:00:56.119 INFO:teuthology.orchestra.run.smithi136.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-17T17:00:56.119 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:00:56.119 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-17T17:00:56.119 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:00:56.119 INFO:teuthology.orchestra.run.smithi136.stdout:Install 28 Packages 2024-08-17T17:00:56.119 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrade 2 Packages 2024-08-17T17:00:56.119 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:00:56.120 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 53 M 2024-08-17T17:00:56.120 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-17T17:00:56.872 INFO:teuthology.orchestra.run.smithi136.stdout:(1/30): ceph-base-18.2.4-719.gced649f8.el9.x86_ 9.2 MB/s | 5.1 MB 00:00 2024-08-17T17:00:56.972 INFO:teuthology.orchestra.run.smithi136.stdout:(2/30): ceph-radosgw-18.2.4-719.gced649f8.el9.x 12 MB/s | 7.7 MB 00:00 2024-08-17T17:00:56.989 INFO:teuthology.orchestra.run.smithi136.stdout:(3/30): ceph-selinux-18.2.4-719.gced649f8.el9.x 215 kB/s | 25 kB 00:00 2024-08-17T17:00:57.039 INFO:teuthology.orchestra.run.smithi136.stdout:(4/30): libcephfs2-18.2.4-719.gced649f8.el9.x86 10 MB/s | 712 kB 00:00 2024-08-17T17:00:57.064 INFO:teuthology.orchestra.run.smithi136.stdout:(5/30): libradosstriper1-18.2.4-719.gced649f8.e 6.2 MB/s | 477 kB 00:00 2024-08-17T17:00:57.140 INFO:teuthology.orchestra.run.smithi136.stdout:(6/30): python3-ceph-argparse-18.2.4-719.gced64 600 kB/s | 45 kB 00:00 2024-08-17T17:00:57.256 INFO:teuthology.orchestra.run.smithi136.stdout:(7/30): librgw2-18.2.4-719.gced649f8.el9.x86_64 21 MB/s | 4.5 MB 00:00 2024-08-17T17:00:57.273 INFO:teuthology.orchestra.run.smithi136.stdout:(8/30): python3-ceph-common-18.2.4-719.gced649f 968 kB/s | 129 kB 00:00 2024-08-17T17:00:57.307 INFO:teuthology.orchestra.run.smithi136.stdout:(9/30): python3-cephfs-18.2.4-719.gced649f8.el9 3.6 MB/s | 188 kB 00:00 2024-08-17T17:00:57.332 INFO:teuthology.orchestra.run.smithi136.stdout:(10/30): python3-rados-18.2.4-719.gced649f8.el9 6.2 MB/s | 372 kB 00:00 2024-08-17T17:00:57.366 INFO:teuthology.orchestra.run.smithi136.stdout:(11/30): python3-rbd-18.2.4-719.gced649f8.el9.x 5.8 MB/s | 347 kB 00:00 2024-08-17T17:00:57.391 INFO:teuthology.orchestra.run.smithi136.stdout:(12/30): python3-rgw-18.2.4-719.gced649f8.el9.x 1.9 MB/s | 116 kB 00:00 2024-08-17T17:00:57.575 INFO:teuthology.orchestra.run.smithi136.stdout:(13/30): ceph-common-18.2.4-719.gced649f8.el9.x 15 MB/s | 18 MB 00:01 2024-08-17T17:00:57.633 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:00:57.636 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:00:57.636 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-08-17T17:00:57.636 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:00:57.636 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-08-17T17:00:57.636 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-radosgw x86_64 2:18.2.4-719.gced649f8.el9 ceph 7.7 M 2024-08-17T17:00:57.636 INFO:teuthology.orchestra.run.smithi165.stdout:Upgrading: 2024-08-17T17:00:57.636 INFO:teuthology.orchestra.run.smithi165.stdout: librados2 x86_64 2:18.2.4-719.gced649f8.el9 ceph 3.3 M 2024-08-17T17:00:57.636 INFO:teuthology.orchestra.run.smithi165.stdout: librbd1 x86_64 2:18.2.4-719.gced649f8.el9 ceph 3.0 M 2024-08-17T17:00:57.636 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-08-17T17:00:57.637 INFO:teuthology.orchestra.run.smithi165.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-17T17:00:57.637 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-base x86_64 2:18.2.4-719.gced649f8.el9 ceph 5.1 M 2024-08-17T17:00:57.637 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-common x86_64 2:18.2.4-719.gced649f8.el9 ceph 18 M 2024-08-17T17:00:57.637 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-selinux x86_64 2:18.2.4-719.gced649f8.el9 ceph 25 k 2024-08-17T17:00:57.637 INFO:teuthology.orchestra.run.smithi165.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-08-17T17:00:57.637 INFO:teuthology.orchestra.run.smithi165.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-17T17:00:57.637 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-17T17:00:57.637 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-17T17:00:57.637 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs2 x86_64 2:18.2.4-719.gced649f8.el9 ceph 712 k 2024-08-17T17:00:57.637 INFO:teuthology.orchestra.run.smithi165.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-17T17:00:57.637 INFO:teuthology.orchestra.run.smithi165.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-17T17:00:57.637 INFO:teuthology.orchestra.run.smithi165.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-17T17:00:57.637 INFO:teuthology.orchestra.run.smithi165.stdout: libradosstriper1 x86_64 2:18.2.4-719.gced649f8.el9 ceph 477 k 2024-08-17T17:00:57.637 INFO:teuthology.orchestra.run.smithi165.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-17T17:00:57.638 INFO:teuthology.orchestra.run.smithi165.stdout: librgw2 x86_64 2:18.2.4-719.gced649f8.el9 ceph 4.5 M 2024-08-17T17:00:57.638 INFO:teuthology.orchestra.run.smithi165.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-17T17:00:57.638 INFO:teuthology.orchestra.run.smithi165.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-17T17:00:57.638 INFO:teuthology.orchestra.run.smithi165.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-17T17:00:57.638 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-argparse x86_64 2:18.2.4-719.gced649f8.el9 ceph 45 k 2024-08-17T17:00:57.638 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-common x86_64 2:18.2.4-719.gced649f8.el9 ceph 129 k 2024-08-17T17:00:57.638 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cephfs x86_64 2:18.2.4-719.gced649f8.el9 ceph 188 k 2024-08-17T17:00:57.638 INFO:teuthology.orchestra.run.smithi165.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-17T17:00:57.638 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rados x86_64 2:18.2.4-719.gced649f8.el9 ceph 372 k 2024-08-17T17:00:57.638 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rbd x86_64 2:18.2.4-719.gced649f8.el9 ceph 347 k 2024-08-17T17:00:57.638 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rgw x86_64 2:18.2.4-719.gced649f8.el9 ceph 116 k 2024-08-17T17:00:57.638 INFO:teuthology.orchestra.run.smithi165.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-08-17T17:00:57.638 INFO:teuthology.orchestra.run.smithi165.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-17T17:00:57.639 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:00:57.639 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-08-17T17:00:57.639 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:00:57.639 INFO:teuthology.orchestra.run.smithi165.stdout:Install 28 Packages 2024-08-17T17:00:57.639 INFO:teuthology.orchestra.run.smithi165.stdout:Upgrade 2 Packages 2024-08-17T17:00:57.639 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:00:57.639 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 53 M 2024-08-17T17:00:57.639 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-08-17T17:00:58.175 INFO:teuthology.orchestra.run.smithi136.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 75 kB/s | 45 kB 00:00 2024-08-17T17:00:58.217 INFO:teuthology.orchestra.run.smithi136.stdout:(15/30): boost-program-options-1.75.0-8.el9.x86 126 kB/s | 107 kB 00:00 2024-08-17T17:00:58.242 INFO:teuthology.orchestra.run.smithi136.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 188 kB/s | 160 kB 00:00 2024-08-17T17:00:58.792 INFO:teuthology.orchestra.run.smithi165.stdout:(1/30): ceph-common-18.2.4-719.gced649f8.el9.x8 20 MB/s | 18 MB 00:00 2024-08-17T17:00:58.842 INFO:teuthology.orchestra.run.smithi136.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 70 kB/s | 42 kB 00:00 2024-08-17T17:00:58.893 INFO:teuthology.orchestra.run.smithi165.stdout:(2/30): ceph-radosgw-18.2.4-719.gced649f8.el9.x 7.6 MB/s | 7.7 MB 00:01 2024-08-17T17:00:58.909 INFO:teuthology.orchestra.run.smithi136.stdout:(18/30): fmt-8.1.1-5.el9.x86_64.rpm 1.6 MB/s | 111 kB 00:00 2024-08-17T17:00:58.918 INFO:teuthology.orchestra.run.smithi165.stdout:(3/30): ceph-selinux-18.2.4-719.gced649f8.el9.x 201 kB/s | 25 kB 00:00 2024-08-17T17:00:58.943 INFO:teuthology.orchestra.run.smithi136.stdout:(19/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 9.1 MB/s | 309 kB 00:00 2024-08-17T17:00:58.985 INFO:teuthology.orchestra.run.smithi165.stdout:(4/30): ceph-base-18.2.4-719.gced649f8.el9.x86_ 4.6 MB/s | 5.1 MB 00:01 2024-08-17T17:00:59.001 INFO:teuthology.orchestra.run.smithi136.stdout:(20/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 373 kB/s | 292 kB 00:00 2024-08-17T17:00:59.010 INFO:teuthology.orchestra.run.smithi165.stdout:(5/30): libcephfs2-18.2.4-719.gced649f8.el9.x86 5.9 MB/s | 712 kB 00:00 2024-08-17T17:00:59.044 INFO:teuthology.orchestra.run.smithi165.stdout:(6/30): libradosstriper1-18.2.4-719.gced649f8.e 3.7 MB/s | 477 kB 00:00 2024-08-17T17:00:59.068 INFO:teuthology.orchestra.run.smithi136.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 390 kB/s | 26 kB 00:00 2024-08-17T17:00:59.069 INFO:teuthology.orchestra.run.smithi165.stdout:(7/30): python3-ceph-argparse-18.2.4-719.gced64 771 kB/s | 45 kB 00:00 2024-08-17T17:00:59.094 INFO:teuthology.orchestra.run.smithi136.stdout:(22/30): librdkafka-1.6.1-102.el9.x86_64.rpm 722 kB/s | 662 kB 00:00 2024-08-17T17:00:59.094 INFO:teuthology.orchestra.run.smithi165.stdout:(8/30): python3-ceph-common-18.2.4-719.gced649f 2.5 MB/s | 129 kB 00:00 2024-08-17T17:00:59.110 INFO:teuthology.orchestra.run.smithi136.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-08-17T17:00:59.119 INFO:teuthology.orchestra.run.smithi165.stdout:(9/30): python3-cephfs-18.2.4-719.gced649f8.el9 3.6 MB/s | 188 kB 00:00 2024-08-17T17:00:59.145 INFO:teuthology.orchestra.run.smithi165.stdout:(10/30): python3-rados-18.2.4-719.gced649f8.el9 7.2 MB/s | 372 kB 00:00 2024-08-17T17:00:59.170 INFO:teuthology.orchestra.run.smithi165.stdout:(11/30): python3-rbd-18.2.4-719.gced649f8.el9.x 6.8 MB/s | 347 kB 00:00 2024-08-17T17:00:59.195 INFO:teuthology.orchestra.run.smithi165.stdout:(12/30): python3-rgw-18.2.4-719.gced649f8.el9.x 2.3 MB/s | 116 kB 00:00 2024-08-17T17:00:59.211 INFO:teuthology.orchestra.run.smithi136.stdout:(24/30): libarrow-9.0.0-11.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-08-17T17:00:59.244 INFO:teuthology.orchestra.run.smithi136.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 6.1 MB/s | 192 kB 00:00 2024-08-17T17:00:59.270 INFO:teuthology.orchestra.run.smithi136.stdout:(26/30): libunwind-1.6.2-1.el9.x86_64.rpm 383 kB/s | 67 kB 00:00 2024-08-17T17:00:59.270 INFO:teuthology.orchestra.run.smithi165.stdout:(13/30): librgw2-18.2.4-719.gced649f8.el9.x86_6 16 MB/s | 4.5 MB 00:00 2024-08-17T17:00:59.303 INFO:teuthology.orchestra.run.smithi136.stdout:(27/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.9 MB/s | 839 kB 00:00 2024-08-17T17:00:59.353 INFO:teuthology.orchestra.run.smithi136.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-08-17T17:00:59.570 INFO:teuthology.orchestra.run.smithi136.stdout:(29/30): librados2-18.2.4-719.gced649f8.el9.x86 11 MB/s | 3.3 MB 00:00 2024-08-17T17:00:59.620 INFO:teuthology.orchestra.run.smithi136.stdout:(30/30): librbd1-18.2.4-719.gced649f8.el9.x86_6 9.5 MB/s | 3.0 MB 00:00 2024-08-17T17:00:59.621 INFO:teuthology.orchestra.run.smithi165.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 237 kB/s | 107 kB 00:00 2024-08-17T17:00:59.626 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:00:59.626 INFO:teuthology.orchestra.run.smithi136.stdout:Total 15 MB/s | 53 MB 00:03 2024-08-17T17:00:59.646 INFO:teuthology.orchestra.run.smithi165.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 121 kB/s | 45 kB 00:00 2024-08-17T17:00:59.693 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-17T17:00:59.696 INFO:teuthology.orchestra.run.smithi165.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 320 kB/s | 160 kB 00:00 2024-08-17T17:00:59.727 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-17T17:00:59.727 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-17T17:00:59.797 INFO:teuthology.orchestra.run.smithi165.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 423 kB/s | 42 kB 00:00 2024-08-17T17:00:59.872 INFO:teuthology.orchestra.run.smithi165.stdout:(18/30): fmt-8.1.1-5.el9.x86_64.rpm 1.4 MB/s | 111 kB 00:00 2024-08-17T17:00:59.905 INFO:teuthology.orchestra.run.smithi165.stdout:(19/30): librdkafka-1.6.1-102.el9.x86_64.rpm 2.3 MB/s | 662 kB 00:00 2024-08-17T17:00:59.931 INFO:teuthology.orchestra.run.smithi165.stdout:(20/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.0 MB/s | 292 kB 00:00 2024-08-17T17:01:00.014 INFO:teuthology.orchestra.run.smithi165.stdout:(21/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.1 MB/s | 309 kB 00:00 2024-08-17T17:01:00.039 INFO:teuthology.orchestra.run.smithi165.stdout:(22/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 318 kB/s | 26 kB 00:00 2024-08-17T17:01:00.066 INFO:teuthology.orchestra.run.smithi165.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 974 kB/s | 49 kB 00:00 2024-08-17T17:01:00.120 INFO:teuthology.orchestra.run.smithi165.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.2 MB/s | 67 kB 00:00 2024-08-17T17:01:00.159 INFO:teuthology.orchestra.run.smithi165.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 4.8 MB/s | 192 kB 00:00 2024-08-17T17:01:00.163 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-17T17:01:00.163 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-17T17:01:00.194 INFO:teuthology.orchestra.run.smithi165.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.5 MB/s | 839 kB 00:00 2024-08-17T17:01:00.276 INFO:teuthology.orchestra.run.smithi165.stdout:(27/30): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-08-17T17:01:00.318 INFO:teuthology.orchestra.run.smithi165.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-08-17T17:01:00.468 INFO:teuthology.orchestra.run.smithi165.stdout:(29/30): librbd1-18.2.4-719.gced649f8.el9.x86_6 16 MB/s | 3.0 MB 00:00 2024-08-17T17:01:00.510 INFO:teuthology.orchestra.run.smithi165.stdout:(30/30): librados2-18.2.4-719.gced649f8.el9.x86 10 MB/s | 3.3 MB 00:00 2024-08-17T17:01:00.514 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:01:00.516 INFO:teuthology.orchestra.run.smithi165.stdout:Total 18 MB/s | 53 MB 00:02 2024-08-17T17:01:00.585 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-08-17T17:01:00.619 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-08-17T17:01:00.620 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-08-17T17:01:01.067 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-08-17T17:01:01.067 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-08-17T17:01:01.512 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-17T17:01:01.605 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-08-17T17:01:01.634 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-08-17T17:01:01.690 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-08-17T17:01:02.015 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-08-17T17:01:02.133 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librados2-2:18.2.4-719.gced649f8.el9.x86_64 5/32 2024-08-17T17:01:02.229 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:18.2.4-719.gced649f8.el9.x86_64 5/32 2024-08-17T17:01:02.317 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rados-2:18.2.4-719.gced649f8.el9.x86_64 6/32 2024-08-17T17:01:02.351 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs2-2:18.2.4-719.gced649f8.el9.x86_64 7/32 2024-08-17T17:01:02.414 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephfs2-2:18.2.4-719.gced649f8.el9.x86_64 7/32 2024-08-17T17:01:02.417 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-08-17T17:01:02.442 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-08-17T17:01:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-08-17T17:01:02.487 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-argparse-2:18.2.4-719.gced649f8.el9.x 9/32 2024-08-17T17:01:02.515 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-08-17T17:01:02.552 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cephfs-2:18.2.4-719.gced649f8.el9.x86_64 10/32 2024-08-17T17:01:02.569 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-08-17T17:01:02.578 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libradosstriper1-2:18.2.4-719.gced649f8.el9.x86_64 11/32 2024-08-17T17:01:02.635 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libradosstriper1-2:18.2.4-719.gced649f8.el9.x86_64 11/32 2024-08-17T17:01:02.751 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-08-17T17:01:02.785 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-08-17T17:01:02.835 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-08-17T17:01:02.875 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-08-17T17:01:02.902 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-08-17T17:01:02.938 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-08-17T17:01:02.970 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-08-17T17:01:02.974 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-08-17T17:01:03.015 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : librados2-2:18.2.4-719.gced649f8.el9.x86_64 5/32 2024-08-17T17:01:03.112 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librados2-2:18.2.4-719.gced649f8.el9.x86_64 5/32 2024-08-17T17:01:03.193 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rados-2:18.2.4-719.gced649f8.el9.x86_64 6/32 2024-08-17T17:01:03.226 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephfs2-2:18.2.4-719.gced649f8.el9.x86_64 7/32 2024-08-17T17:01:03.273 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libcephfs2-2:18.2.4-719.gced649f8.el9.x86_64 7/32 2024-08-17T17:01:03.275 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-08-17T17:01:03.309 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-08-17T17:01:03.325 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librbd1-2:18.2.4-719.gced649f8.el9.x86_64 20/32 2024-08-17T17:01:03.355 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ceph-argparse-2:18.2.4-719.gced649f8.el9.x 9/32 2024-08-17T17:01:03.419 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cephfs-2:18.2.4-719.gced649f8.el9.x86_64 10/32 2024-08-17T17:01:03.421 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:18.2.4-719.gced649f8.el9.x86_64 20/32 2024-08-17T17:01:03.445 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libradosstriper1-2:18.2.4-719.gced649f8.el9.x86_64 11/32 2024-08-17T17:01:03.447 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rbd-2:18.2.4-719.gced649f8.el9.x86_64 21/32 2024-08-17T17:01:03.492 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-08-17T17:01:03.495 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libradosstriper1-2:18.2.4-719.gced649f8.el9.x86_64 11/32 2024-08-17T17:01:03.602 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-08-17T17:01:03.636 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-08-17T17:01:03.682 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-08-17T17:01:03.717 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-08-17T17:01:03.744 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-08-17T17:01:03.780 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-08-17T17:01:03.813 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-08-17T17:01:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-08-17T17:01:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librgw2-2:18.2.4-719.gced649f8.el9.x86_64 24/32 2024-08-17T17:01:04.043 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librgw2-2:18.2.4-719.gced649f8.el9.x86_64 24/32 2024-08-17T17:01:04.112 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-08-17T17:01:04.119 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rgw-2:18.2.4-719.gced649f8.el9.x86_64 25/32 2024-08-17T17:01:04.147 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-common-2:18.2.4-719.gced649f8.el9.x86 26/32 2024-08-17T17:01:04.151 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : librbd1-2:18.2.4-719.gced649f8.el9.x86_64 20/32 2024-08-17T17:01:04.245 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librbd1-2:18.2.4-719.gced649f8.el9.x86_64 20/32 2024-08-17T17:01:04.273 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rbd-2:18.2.4-719.gced649f8.el9.x86_64 21/32 2024-08-17T17:01:04.309 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-08-17T17:01:04.713 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-08-17T17:01:04.754 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librgw2-2:18.2.4-719.gced649f8.el9.x86_64 24/32 2024-08-17T17:01:04.826 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librgw2-2:18.2.4-719.gced649f8.el9.x86_64 24/32 2024-08-17T17:01:04.900 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rgw-2:18.2.4-719.gced649f8.el9.x86_64 25/32 2024-08-17T17:01:04.931 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ceph-common-2:18.2.4-719.gced649f8.el9.x86 26/32 2024-08-17T17:01:06.208 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:18.2.4-719.gced649f8.el9.x86_64 27/32 2024-08-17T17:01:06.243 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-common-2:18.2.4-719.gced649f8.el9.x86_64 27/32 2024-08-17T17:01:06.778 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:18.2.4-719.gced649f8.el9.x86_64 27/32 2024-08-17T17:01:06.821 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-base-2:18.2.4-719.gced649f8.el9.x86_64 28/32 2024-08-17T17:01:06.916 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-base-2:18.2.4-719.gced649f8.el9.x86_64 28/32 2024-08-17T17:01:06.916 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-17T17:01:06.916 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-17T17:01:06.917 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:06.948 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-selinux-2:18.2.4-719.gced649f8.el9.x86_64 29/32 2024-08-17T17:01:07.009 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-common-2:18.2.4-719.gced649f8.el9.x86_64 27/32 2024-08-17T17:01:07.044 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-common-2:18.2.4-719.gced649f8.el9.x86_64 27/32 2024-08-17T17:01:07.573 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-common-2:18.2.4-719.gced649f8.el9.x86_64 27/32 2024-08-17T17:01:07.614 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-base-2:18.2.4-719.gced649f8.el9.x86_64 28/32 2024-08-17T17:01:07.712 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-base-2:18.2.4-719.gced649f8.el9.x86_64 28/32 2024-08-17T17:01:07.712 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-17T17:01:07.712 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-17T17:01:07.712 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:07.740 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-selinux-2:18.2.4-719.gced649f8.el9.x86_64 29/32 2024-08-17T17:01:16.773 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-selinux-2:18.2.4-719.gced649f8.el9.x86_64 29/32 2024-08-17T17:01:16.773 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /sys 2024-08-17T17:01:16.773 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /proc 2024-08-17T17:01:16.773 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /mnt 2024-08-17T17:01:16.773 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /var/tmp 2024-08-17T17:01:16.773 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /home 2024-08-17T17:01:16.774 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /root 2024-08-17T17:01:16.774 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /tmp 2024-08-17T17:01:16.774 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:17.371 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-selinux-2:18.2.4-719.gced649f8.el9.x86_64 29/32 2024-08-17T17:01:17.371 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /sys 2024-08-17T17:01:17.371 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /proc 2024-08-17T17:01:17.371 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /mnt 2024-08-17T17:01:17.371 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /var/tmp 2024-08-17T17:01:17.371 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /home 2024-08-17T17:01:17.371 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /root 2024-08-17T17:01:17.372 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /tmp 2024-08-17T17:01:17.372 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:17.555 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-radosgw-2:18.2.4-719.gced649f8.el9.x86_64 30/32 2024-08-17T17:01:17.581 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-719.gced649f8.el9.x86_64 30/32 2024-08-17T17:01:17.581 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T17:01:17.581 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-17T17:01:17.581 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-17T17:01:17.581 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-17T17:01:17.581 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:17.583 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-08-17T17:01:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-08-17T17:01:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-08-17T17:01:18.122 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-radosgw-2:18.2.4-719.gced649f8.el9.x86_64 30/32 2024-08-17T17:01:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-719.gced649f8.el9.x86_64 30/32 2024-08-17T17:01:18.148 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T17:01:18.148 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-17T17:01:18.148 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-17T17:01:18.149 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-17T17:01:18.149 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:18.150 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-08-17T17:01:18.242 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-08-17T17:01:18.242 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-08-17T17:01:18.730 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-08-17T17:01:18.731 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-base-2:18.2.4-719.gced649f8.el9.x86_64 1/32 2024-08-17T17:01:18.731 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-common-2:18.2.4-719.gced649f8.el9.x86_64 2/32 2024-08-17T17:01:18.731 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-radosgw-2:18.2.4-719.gced649f8.el9.x86_64 3/32 2024-08-17T17:01:18.731 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-selinux-2:18.2.4-719.gced649f8.el9.x86_64 4/32 2024-08-17T17:01:18.731 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs2-2:18.2.4-719.gced649f8.el9.x86_64 5/32 2024-08-17T17:01:18.731 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libradosstriper1-2:18.2.4-719.gced649f8.el9.x86_64 6/32 2024-08-17T17:01:18.731 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librgw2-2:18.2.4-719.gced649f8.el9.x86_64 7/32 2024-08-17T17:01:18.731 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-argparse-2:18.2.4-719.gced649f8.el9.x 8/32 2024-08-17T17:01:18.731 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-common-2:18.2.4-719.gced649f8.el9.x86 9/32 2024-08-17T17:01:18.731 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cephfs-2:18.2.4-719.gced649f8.el9.x86_64 10/32 2024-08-17T17:01:18.732 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rados-2:18.2.4-719.gced649f8.el9.x86_64 11/32 2024-08-17T17:01:18.732 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rbd-2:18.2.4-719.gced649f8.el9.x86_64 12/32 2024-08-17T17:01:18.732 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rgw-2:18.2.4-719.gced649f8.el9.x86_64 13/32 2024-08-17T17:01:18.732 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-08-17T17:01:18.732 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-08-17T17:01:18.732 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-08-17T17:01:18.732 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-08-17T17:01:18.732 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-08-17T17:01:18.732 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-08-17T17:01:18.732 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-08-17T17:01:18.732 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-08-17T17:01:18.732 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-08-17T17:01:18.732 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-08-17T17:01:18.733 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-08-17T17:01:18.733 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-08-17T17:01:18.733 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-08-17T17:01:18.733 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-08-17T17:01:18.733 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-08-17T17:01:18.733 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:18.2.4-719.gced649f8.el9.x86_64 29/32 2024-08-17T17:01:18.733 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-08-17T17:01:18.733 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:18.2.4-719.gced649f8.el9.x86_64 31/32 2024-08-17T17:01:19.309 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-08-17T17:01:19.309 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-base-2:18.2.4-719.gced649f8.el9.x86_64 1/32 2024-08-17T17:01:19.309 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-common-2:18.2.4-719.gced649f8.el9.x86_64 2/32 2024-08-17T17:01:19.309 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-radosgw-2:18.2.4-719.gced649f8.el9.x86_64 3/32 2024-08-17T17:01:19.309 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-selinux-2:18.2.4-719.gced649f8.el9.x86_64 4/32 2024-08-17T17:01:19.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephfs2-2:18.2.4-719.gced649f8.el9.x86_64 5/32 2024-08-17T17:01:19.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libradosstriper1-2:18.2.4-719.gced649f8.el9.x86_64 6/32 2024-08-17T17:01:19.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librgw2-2:18.2.4-719.gced649f8.el9.x86_64 7/32 2024-08-17T17:01:19.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ceph-argparse-2:18.2.4-719.gced649f8.el9.x 8/32 2024-08-17T17:01:19.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ceph-common-2:18.2.4-719.gced649f8.el9.x86 9/32 2024-08-17T17:01:19.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cephfs-2:18.2.4-719.gced649f8.el9.x86_64 10/32 2024-08-17T17:01:19.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rados-2:18.2.4-719.gced649f8.el9.x86_64 11/32 2024-08-17T17:01:19.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rbd-2:18.2.4-719.gced649f8.el9.x86_64 12/32 2024-08-17T17:01:19.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rgw-2:18.2.4-719.gced649f8.el9.x86_64 13/32 2024-08-17T17:01:19.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-08-17T17:01:19.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-08-17T17:01:19.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-08-17T17:01:19.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-08-17T17:01:19.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-08-17T17:01:19.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-08-17T17:01:19.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-08-17T17:01:19.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-08-17T17:01:19.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-08-17T17:01:19.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-08-17T17:01:19.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-08-17T17:01:19.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-08-17T17:01:19.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-08-17T17:01:19.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-08-17T17:01:19.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-08-17T17:01:19.311 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados2-2:18.2.4-719.gced649f8.el9.x86_64 29/32 2024-08-17T17:01:19.312 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-08-17T17:01:19.312 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librbd1-2:18.2.4-719.gced649f8.el9.x86_64 31/32 2024-08-17T17:01:19.703 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-08-17T17:01:19.704 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:19.704 INFO:teuthology.orchestra.run.smithi136.stdout:Upgraded: 2024-08-17T17:01:19.704 INFO:teuthology.orchestra.run.smithi136.stdout: librados2-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:19.704 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:19.704 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-17T17:01:19.704 INFO:teuthology.orchestra.run.smithi136.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-17T17:01:19.704 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:19.704 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:19.704 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:19.705 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:19.705 INFO:teuthology.orchestra.run.smithi136.stdout: fmt-8.1.1-5.el9.x86_64 2024-08-17T17:01:19.705 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-17T17:01:19.705 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-17T17:01:19.705 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-17T17:01:19.705 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:19.705 INFO:teuthology.orchestra.run.smithi136.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-17T17:01:19.705 INFO:teuthology.orchestra.run.smithi136.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-17T17:01:19.705 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-17T17:01:19.706 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:19.706 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-17T17:01:19.706 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:19.706 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-17T17:01:19.706 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-17T17:01:19.706 INFO:teuthology.orchestra.run.smithi136.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-17T17:01:19.706 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:19.707 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:19.707 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:19.707 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-17T17:01:19.707 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:19.707 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:19.707 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:19.707 INFO:teuthology.orchestra.run.smithi136.stdout: re2-1:20211101-3.el9.x86_64 2024-08-17T17:01:19.707 INFO:teuthology.orchestra.run.smithi136.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-17T17:01:19.708 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:19.708 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:01:20.083 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-test 2024-08-17T17:01:20.120 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-08-17T17:01:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:20.121 INFO:teuthology.orchestra.run.smithi165.stdout:Upgraded: 2024-08-17T17:01:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: librados2-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: librbd1-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:20.121 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-08-17T17:01:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-17T17:01:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-base-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-common-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-radosgw-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-selinux-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: fmt-8.1.1-5.el9.x86_64 2024-08-17T17:01:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-17T17:01:20.121 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-17T17:01:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-17T17:01:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs2-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-17T17:01:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-17T17:01:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-17T17:01:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: libradosstriper1-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-17T17:01:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: librgw2-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-17T17:01:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-17T17:01:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-17T17:01:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-argparse-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:20.122 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-common-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:20.123 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cephfs-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:20.123 INFO:teuthology.orchestra.run.smithi165.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-17T17:01:20.123 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rados-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:20.123 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rbd-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:20.123 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rgw-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:20.123 INFO:teuthology.orchestra.run.smithi165.stdout: re2-1:20211101-3.el9.x86_64 2024-08-17T17:01:20.123 INFO:teuthology.orchestra.run.smithi165.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-17T17:01:20.123 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:20.123 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:01:20.502 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-test 2024-08-17T17:01:20.634 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:27 ago on Sat 17 Aug 2024 05:00:53 PM UTC. 2024-08-17T17:01:20.759 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:01:20.760 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:01:20.760 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-08-17T17:01:20.760 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:01:20.760 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-17T17:01:20.760 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test x86_64 2:18.2.4-719.gced649f8.el9 ceph 40 M 2024-08-17T17:01:20.760 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-08-17T17:01:20.760 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite x86_64 2:18.2.4-719.gced649f8.el9 ceph 166 k 2024-08-17T17:01:20.761 INFO:teuthology.orchestra.run.smithi136.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-17T17:01:20.761 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-17T17:01:20.761 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:20.761 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-17T17:01:20.761 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:01:20.761 INFO:teuthology.orchestra.run.smithi136.stdout:Install 4 Packages 2024-08-17T17:01:20.761 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:20.762 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 40 M 2024-08-17T17:01:20.762 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 163 M 2024-08-17T17:01:20.762 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-17T17:01:21.054 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:26 ago on Sat 17 Aug 2024 05:00:55 PM UTC. 2024-08-17T17:01:21.179 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:01:21.180 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:01:21.180 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-08-17T17:01:21.180 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:01:21.180 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-08-17T17:01:21.180 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-test x86_64 2:18.2.4-719.gced649f8.el9 ceph 40 M 2024-08-17T17:01:21.180 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-08-17T17:01:21.180 INFO:teuthology.orchestra.run.smithi165.stdout: libcephsqlite x86_64 2:18.2.4-719.gced649f8.el9 ceph 166 k 2024-08-17T17:01:21.180 INFO:teuthology.orchestra.run.smithi165.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-17T17:01:21.180 INFO:teuthology.orchestra.run.smithi165.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-17T17:01:21.181 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:21.181 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-08-17T17:01:21.181 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:01:21.181 INFO:teuthology.orchestra.run.smithi165.stdout:Install 4 Packages 2024-08-17T17:01:21.181 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:21.181 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 40 M 2024-08-17T17:01:21.181 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 163 M 2024-08-17T17:01:21.182 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-08-17T17:01:21.286 INFO:teuthology.orchestra.run.smithi136.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.2 MB/s | 304 kB 00:00 2024-08-17T17:01:21.312 INFO:teuthology.orchestra.run.smithi136.stdout:(2/4): libcephsqlite-18.2.4-719.gced649f8.el9.x 590 kB/s | 166 kB 00:00 2024-08-17T17:01:21.437 INFO:teuthology.orchestra.run.smithi136.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 425 kB/s | 64 kB 00:00 2024-08-17T17:01:21.587 INFO:teuthology.orchestra.run.smithi165.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.1 MB/s | 304 kB 00:00 2024-08-17T17:01:21.645 INFO:teuthology.orchestra.run.smithi165.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-08-17T17:01:21.712 INFO:teuthology.orchestra.run.smithi165.stdout:(3/4): libcephsqlite-18.2.4-719.gced649f8.el9.x 419 kB/s | 166 kB 00:00 2024-08-17T17:01:22.395 INFO:teuthology.orchestra.run.smithi136.stdout:(4/4): ceph-test-18.2.4-719.gced649f8.el9.x86_6 29 MB/s | 40 MB 00:01 2024-08-17T17:01:22.398 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:01:22.398 INFO:teuthology.orchestra.run.smithi136.stdout:Total 25 MB/s | 40 MB 00:01 2024-08-17T17:01:22.425 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-17T17:01:22.445 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-17T17:01:22.445 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-17T17:01:22.765 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-17T17:01:22.765 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-17T17:01:23.237 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-17T17:01:23.314 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-17T17:01:23.361 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-17T17:01:23.389 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephsqlite-2:18.2.4-719.gced649f8.el9.x86_64 3/4 2024-08-17T17:01:23.737 INFO:teuthology.orchestra.run.smithi165.stdout:(4/4): ceph-test-18.2.4-719.gced649f8.el9.x86_6 16 MB/s | 40 MB 00:02 2024-08-17T17:01:23.739 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:01:23.740 INFO:teuthology.orchestra.run.smithi165.stdout:Total 16 MB/s | 40 MB 00:02 2024-08-17T17:01:23.766 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-08-17T17:01:23.786 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-08-17T17:01:23.786 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-08-17T17:01:24.103 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-08-17T17:01:24.104 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-08-17T17:01:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-08-17T17:01:24.646 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-17T17:01:24.686 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-17T17:01:24.714 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephsqlite-2:18.2.4-719.gced649f8.el9.x86_64 3/4 2024-08-17T17:01:27.093 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephsqlite-2:18.2.4-719.gced649f8.el9.x86_64 3/4 2024-08-17T17:01:27.169 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-test-2:18.2.4-719.gced649f8.el9.x86_64 4/4 2024-08-17T17:01:28.406 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libcephsqlite-2:18.2.4-719.gced649f8.el9.x86_64 3/4 2024-08-17T17:01:28.479 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-test-2:18.2.4-719.gced649f8.el9.x86_64 4/4 2024-08-17T17:01:29.232 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-test-2:18.2.4-719.gced649f8.el9.x86_64 4/4 2024-08-17T17:01:29.233 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-test-2:18.2.4-719.gced649f8.el9.x86_64 1/4 2024-08-17T17:01:29.233 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephsqlite-2:18.2.4-719.gced649f8.el9.x86_64 2/4 2024-08-17T17:01:29.233 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-17T17:01:29.515 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-17T17:01:29.515 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:29.515 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-17T17:01:29.515 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:29.516 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:29.516 INFO:teuthology.orchestra.run.smithi136.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-17T17:01:29.516 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-17T17:01:29.516 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:29.516 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:01:29.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph 2024-08-17T17:01:30.419 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:37 ago on Sat 17 Aug 2024 05:00:53 PM UTC. 2024-08-17T17:01:30.556 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:01:30.559 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:01:30.559 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-08-17T17:01:30.559 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:01:30.559 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-17T17:01:30.559 INFO:teuthology.orchestra.run.smithi136.stdout: ceph x86_64 2:18.2.4-719.gced649f8.el9 ceph 6.5 k 2024-08-17T17:01:30.559 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-08-17T17:01:30.560 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds x86_64 2:18.2.4-719.gced649f8.el9 ceph 2.1 M 2024-08-17T17:01:30.560 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr x86_64 2:18.2.4-719.gced649f8.el9 ceph 1.5 M 2024-08-17T17:01:30.560 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core noarch 2:18.2.4-719.gced649f8.el9 ceph-noarch 246 k 2024-08-17T17:01:30.560 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon x86_64 2:18.2.4-719.gced649f8.el9 ceph 4.7 M 2024-08-17T17:01:30.560 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd x86_64 2:18.2.4-719.gced649f8.el9 ceph 17 M 2024-08-17T17:01:30.560 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-17T17:01:30.560 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-17T17:01:30.560 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-17T17:01:30.560 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-17T17:01:30.560 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-17T17:01:30.560 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-17T17:01:30.560 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-17T17:01:30.560 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-08-17T17:01:30.560 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-17T17:01:30.561 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-17T17:01:30.561 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-17T17:01:30.561 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-17T17:01:30.561 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-17T17:01:30.561 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-17T17:01:30.561 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-17T17:01:30.561 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-17T17:01:30.561 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-17T17:01:30.561 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-17T17:01:30.561 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-17T17:01:30.561 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-17T17:01:30.561 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-17T17:01:30.561 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-17T17:01:30.561 INFO:teuthology.orchestra.run.smithi136.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-17T17:01:30.562 INFO:teuthology.orchestra.run.smithi136.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-17T17:01:30.562 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-17T17:01:30.562 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-17T17:01:30.562 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-17T17:01:30.562 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:30.562 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-17T17:01:30.562 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:01:30.562 INFO:teuthology.orchestra.run.smithi136.stdout:Install 33 Packages 2024-08-17T17:01:30.562 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:30.563 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 29 M 2024-08-17T17:01:30.563 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 104 M 2024-08-17T17:01:30.563 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-17T17:01:30.581 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-test-2:18.2.4-719.gced649f8.el9.x86_64 4/4 2024-08-17T17:01:30.581 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-test-2:18.2.4-719.gced649f8.el9.x86_64 1/4 2024-08-17T17:01:30.581 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephsqlite-2:18.2.4-719.gced649f8.el9.x86_64 2/4 2024-08-17T17:01:30.582 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-17T17:01:30.868 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-17T17:01:30.868 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:30.868 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-08-17T17:01:30.868 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-test-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:30.869 INFO:teuthology.orchestra.run.smithi165.stdout: libcephsqlite-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:30.869 INFO:teuthology.orchestra.run.smithi165.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-17T17:01:30.869 INFO:teuthology.orchestra.run.smithi165.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-17T17:01:30.869 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:30.869 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:01:31.079 INFO:teuthology.orchestra.run.smithi136.stdout:(1/33): ceph-18.2.4-719.gced649f8.el9.x86_64.rp 37 kB/s | 6.5 kB 00:00 2024-08-17T17:01:31.198 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph 2024-08-17T17:01:31.279 INFO:teuthology.orchestra.run.smithi136.stdout:(2/33): ceph-mgr-18.2.4-719.gced649f8.el9.x86_6 3.9 MB/s | 1.5 MB 00:00 2024-08-17T17:01:31.411 INFO:teuthology.orchestra.run.smithi136.stdout:(3/33): ceph-mds-18.2.4-719.gced649f8.el9.x86_6 4.9 MB/s | 2.1 MB 00:00 2024-08-17T17:01:31.421 INFO:teuthology.orchestra.run.smithi136.stdout:(4/33): ceph-mon-18.2.4-719.gced649f8.el9.x86_6 14 MB/s | 4.7 MB 00:00 2024-08-17T17:01:31.447 INFO:teuthology.orchestra.run.smithi136.stdout:(5/33): ceph-mgr-modules-core-18.2.4-719.gced64 2.1 MB/s | 246 kB 00:00 2024-08-17T17:01:31.597 INFO:teuthology.orchestra.run.smithi136.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-08-17T17:01:31.647 INFO:teuthology.orchestra.run.smithi136.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-08-17T17:01:31.753 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:36 ago on Sat 17 Aug 2024 05:00:55 PM UTC. 2024-08-17T17:01:31.831 INFO:teuthology.orchestra.run.smithi136.stdout:(8/33): ceph-osd-18.2.4-719.gced649f8.el9.x86_6 30 MB/s | 17 MB 00:00 2024-08-17T17:01:31.855 INFO:teuthology.orchestra.run.smithi136.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 648 kB/s | 135 kB 00:00 2024-08-17T17:01:31.885 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:01:31.887 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:01:31.887 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-08-17T17:01:31.887 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:01:31.887 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-08-17T17:01:31.887 INFO:teuthology.orchestra.run.smithi165.stdout: ceph x86_64 2:18.2.4-719.gced649f8.el9 ceph 6.5 k 2024-08-17T17:01:31.887 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-08-17T17:01:31.887 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mds x86_64 2:18.2.4-719.gced649f8.el9 ceph 2.1 M 2024-08-17T17:01:31.888 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr x86_64 2:18.2.4-719.gced649f8.el9 ceph 1.5 M 2024-08-17T17:01:31.888 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-modules-core noarch 2:18.2.4-719.gced649f8.el9 ceph-noarch 246 k 2024-08-17T17:01:31.888 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mon x86_64 2:18.2.4-719.gced649f8.el9 ceph 4.7 M 2024-08-17T17:01:31.888 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-osd x86_64 2:18.2.4-719.gced649f8.el9 ceph 17 M 2024-08-17T17:01:31.888 INFO:teuthology.orchestra.run.smithi165.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-17T17:01:31.888 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-17T17:01:31.888 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-17T17:01:31.888 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-17T17:01:31.888 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-17T17:01:31.888 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-17T17:01:31.888 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-17T17:01:31.888 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-08-17T17:01:31.888 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-17T17:01:31.888 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-17T17:01:31.889 INFO:teuthology.orchestra.run.smithi165.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-17T17:01:31.889 INFO:teuthology.orchestra.run.smithi165.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-17T17:01:31.889 INFO:teuthology.orchestra.run.smithi165.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-17T17:01:31.889 INFO:teuthology.orchestra.run.smithi165.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-17T17:01:31.889 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-17T17:01:31.889 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-17T17:01:31.889 INFO:teuthology.orchestra.run.smithi165.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-17T17:01:31.889 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-17T17:01:31.889 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-17T17:01:31.889 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-17T17:01:31.889 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-17T17:01:31.889 INFO:teuthology.orchestra.run.smithi165.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-17T17:01:31.889 INFO:teuthology.orchestra.run.smithi165.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-17T17:01:31.890 INFO:teuthology.orchestra.run.smithi165.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-17T17:01:31.890 INFO:teuthology.orchestra.run.smithi165.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-17T17:01:31.890 INFO:teuthology.orchestra.run.smithi165.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-17T17:01:31.890 INFO:teuthology.orchestra.run.smithi165.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-17T17:01:31.890 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:31.890 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-08-17T17:01:31.890 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:01:31.890 INFO:teuthology.orchestra.run.smithi165.stdout:Install 33 Packages 2024-08-17T17:01:31.890 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:31.891 INFO:teuthology.orchestra.run.smithi136.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 2.8 MB/s | 1.2 MB 00:00 2024-08-17T17:01:31.891 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 29 M 2024-08-17T17:01:31.891 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 104 M 2024-08-17T17:01:31.891 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-08-17T17:01:31.914 INFO:teuthology.orchestra.run.smithi136.stdout:(11/33): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-08-17T17:01:31.973 INFO:teuthology.orchestra.run.smithi136.stdout:(12/33): python3-requests-2.25.1-8.el9.noarch.r 878 kB/s | 125 kB 00:00 2024-08-17T17:01:32.065 INFO:teuthology.orchestra.run.smithi136.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 231 kB/s | 35 kB 00:00 2024-08-17T17:01:32.132 INFO:teuthology.orchestra.run.smithi136.stdout:(14/33): python3-mako-1.1.4-6.el9.noarch.rpm 711 kB/s | 172 kB 00:00 2024-08-17T17:01:32.149 INFO:teuthology.orchestra.run.smithi136.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 291 kB/s | 51 kB 00:00 2024-08-17T17:01:32.190 INFO:teuthology.orchestra.run.smithi136.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 332 kB/s | 42 kB 00:00 2024-08-17T17:01:32.232 INFO:teuthology.orchestra.run.smithi136.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-08-17T17:01:32.275 INFO:teuthology.orchestra.run.smithi136.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 746 kB/s | 11 kB 00:00 2024-08-17T17:01:32.300 INFO:teuthology.orchestra.run.smithi136.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-08-17T17:01:32.317 INFO:teuthology.orchestra.run.smithi136.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 446 kB/s | 18 kB 00:00 2024-08-17T17:01:32.334 INFO:teuthology.orchestra.run.smithi136.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 691 kB/s | 23 kB 00:00 2024-08-17T17:01:32.351 INFO:teuthology.orchestra.run.smithi136.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 577 kB/s | 19 kB 00:00 2024-08-17T17:01:32.367 INFO:teuthology.orchestra.run.smithi136.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 586 kB/s | 20 kB 00:00 2024-08-17T17:01:32.385 INFO:teuthology.orchestra.run.smithi136.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 1.3 MB/s | 46 kB 00:00 2024-08-17T17:01:32.410 INFO:teuthology.orchestra.run.smithi136.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-08-17T17:01:32.435 INFO:teuthology.orchestra.run.smithi136.stdout:(26/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.6 MB/s | 358 kB 00:00 2024-08-17T17:01:32.452 INFO:teuthology.orchestra.run.smithi136.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 392 kB/s | 16 kB 00:00 2024-08-17T17:01:32.463 INFO:teuthology.orchestra.run.smithi165.stdout:(1/33): ceph-18.2.4-719.gced649f8.el9.x86_64.rp 35 kB/s | 6.5 kB 00:00 2024-08-17T17:01:32.477 INFO:teuthology.orchestra.run.smithi136.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-08-17T17:01:32.503 INFO:teuthology.orchestra.run.smithi136.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-08-17T17:01:32.528 INFO:teuthology.orchestra.run.smithi136.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 473 kB/s | 36 kB 00:00 2024-08-17T17:01:32.553 INFO:teuthology.orchestra.run.smithi136.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 797 kB/s | 20 kB 00:00 2024-08-17T17:01:32.578 INFO:teuthology.orchestra.run.smithi136.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-08-17T17:01:32.603 INFO:teuthology.orchestra.run.smithi136.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-08-17T17:01:32.607 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:01:32.608 INFO:teuthology.orchestra.run.smithi136.stdout:Total 14 MB/s | 29 MB 00:02 2024-08-17T17:01:32.679 INFO:teuthology.orchestra.run.smithi165.stdout:(2/33): ceph-mds-18.2.4-719.gced649f8.el9.x86_6 5.2 MB/s | 2.1 MB 00:00 2024-08-17T17:01:32.721 INFO:teuthology.orchestra.run.smithi165.stdout:(3/33): ceph-mgr-18.2.4-719.gced649f8.el9.x86_6 3.3 MB/s | 1.5 MB 00:00 2024-08-17T17:01:32.721 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-17T17:01:32.759 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-17T17:01:32.759 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-17T17:01:32.821 INFO:teuthology.orchestra.run.smithi165.stdout:(4/33): ceph-mon-18.2.4-719.gced649f8.el9.x86_6 13 MB/s | 4.7 MB 00:00 2024-08-17T17:01:32.855 INFO:teuthology.orchestra.run.smithi165.stdout:(5/33): ceph-mgr-modules-core-18.2.4-719.gced64 1.8 MB/s | 246 kB 00:00 2024-08-17T17:01:33.006 INFO:teuthology.orchestra.run.smithi165.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-08-17T17:01:33.064 INFO:teuthology.orchestra.run.smithi165.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-08-17T17:01:33.123 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-17T17:01:33.123 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-17T17:01:33.240 INFO:teuthology.orchestra.run.smithi165.stdout:(8/33): ceph-osd-18.2.4-719.gced649f8.el9.x86_6 30 MB/s | 17 MB 00:00 2024-08-17T17:01:33.282 INFO:teuthology.orchestra.run.smithi165.stdout:(9/33): python3-cryptography-36.0.1-4.el9.x86_6 2.9 MB/s | 1.2 MB 00:00 2024-08-17T17:01:33.307 INFO:teuthology.orchestra.run.smithi165.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 558 kB/s | 135 kB 00:00 2024-08-17T17:01:33.332 INFO:teuthology.orchestra.run.smithi165.stdout:(11/33): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-08-17T17:01:33.399 INFO:teuthology.orchestra.run.smithi165.stdout:(12/33): python3-requests-2.25.1-8.el9.noarch.r 786 kB/s | 125 kB 00:00 2024-08-17T17:01:33.932 INFO:teuthology.orchestra.run.smithi165.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 58 kB/s | 35 kB 00:00 2024-08-17T17:01:34.008 INFO:teuthology.orchestra.run.smithi165.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 84 kB/s | 51 kB 00:00 2024-08-17T17:01:34.075 INFO:teuthology.orchestra.run.smithi165.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 657 kB/s | 43 kB 00:00 2024-08-17T17:01:34.100 INFO:teuthology.orchestra.run.smithi165.stdout:(16/33): python3-mako-1.1.4-6.el9.noarch.rpm 217 kB/s | 172 kB 00:00 2024-08-17T17:01:34.150 INFO:teuthology.orchestra.run.smithi165.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-08-17T17:01:34.184 INFO:teuthology.orchestra.run.smithi165.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 344 kB/s | 11 kB 00:00 2024-08-17T17:01:34.234 INFO:teuthology.orchestra.run.smithi165.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 667 kB/s | 18 kB 00:00 2024-08-17T17:01:34.259 INFO:teuthology.orchestra.run.smithi165.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 929 kB/s | 23 kB 00:00 2024-08-17T17:01:34.284 INFO:teuthology.orchestra.run.smithi165.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 778 kB/s | 19 kB 00:00 2024-08-17T17:01:34.310 INFO:teuthology.orchestra.run.smithi165.stdout:(22/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-08-17T17:01:34.335 INFO:teuthology.orchestra.run.smithi165.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 398 kB/s | 20 kB 00:00 2024-08-17T17:01:34.360 INFO:teuthology.orchestra.run.smithi165.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-08-17T17:01:34.385 INFO:teuthology.orchestra.run.smithi165.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-08-17T17:01:34.393 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-17T17:01:34.411 INFO:teuthology.orchestra.run.smithi165.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 656 kB/s | 16 kB 00:00 2024-08-17T17:01:34.436 INFO:teuthology.orchestra.run.smithi165.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-08-17T17:01:34.456 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-08-17T17:01:34.461 INFO:teuthology.orchestra.run.smithi165.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-08-17T17:01:34.486 INFO:teuthology.orchestra.run.smithi165.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 710 kB/s | 36 kB 00:00 2024-08-17T17:01:34.488 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-08-17T17:01:34.511 INFO:teuthology.orchestra.run.smithi165.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-08-17T17:01:34.521 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-08-17T17:01:34.537 INFO:teuthology.orchestra.run.smithi165.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 795 kB/s | 20 kB 00:00 2024-08-17T17:01:34.562 INFO:teuthology.orchestra.run.smithi165.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-08-17T17:01:34.587 INFO:teuthology.orchestra.run.smithi165.stdout:(33/33): python3-toml-0.10.2-6.el9.noarch.rpm 64 kB/s | 42 kB 00:00 2024-08-17T17:01:34.591 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:01:34.591 INFO:teuthology.orchestra.run.smithi165.stdout:Total 11 MB/s | 29 MB 00:02 2024-08-17T17:01:34.614 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-08-17T17:01:34.683 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-08-17T17:01:34.707 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-08-17T17:01:34.724 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-08-17T17:01:34.745 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-08-17T17:01:34.745 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-08-17T17:01:34.759 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-08-17T17:01:34.797 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-08-17T17:01:34.831 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-08-17T17:01:34.865 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-08-17T17:01:34.905 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-08-17T17:01:34.940 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-08-17T17:01:34.974 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-08-17T17:01:35.020 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-08-17T17:01:35.093 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-08-17T17:01:35.116 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-08-17T17:01:35.116 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-08-17T17:01:35.229 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-08-17T17:01:35.302 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-08-17T17:01:35.356 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-08-17T17:01:35.403 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-08-17T17:01:35.452 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-08-17T17:01:35.507 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-08-17T17:01:35.699 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-08-17T17:01:35.740 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-08-17T17:01:35.818 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-08-17T17:01:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-08-17T17:01:35.946 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-08-17T17:01:36.062 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-08-17T17:01:36.214 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-719.gced649f8.el9.n 28/33 2024-08-17T17:01:36.250 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-2:18.2.4-719.gced649f8.el9.x86_64 29/33 2024-08-17T17:01:36.275 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-2:18.2.4-719.gced649f8.el9.x86_64 29/33 2024-08-17T17:01:36.275 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T17:01:36.276 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-17T17:01:36.276 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-17T17:01:36.276 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-17T17:01:36.276 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:36.358 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-08-17T17:01:36.424 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-08-17T17:01:36.456 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-08-17T17:01:36.489 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-08-17T17:01:36.580 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-08-17T17:01:36.651 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-08-17T17:01:36.691 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-08-17T17:01:36.726 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-08-17T17:01:36.756 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-08-17T17:01:36.782 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-08-17T17:01:36.807 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-08-17T17:01:36.856 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-08-17T17:01:36.898 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-08-17T17:01:36.933 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-08-17T17:01:36.974 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-08-17T17:01:37.054 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-08-17T17:01:37.199 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-08-17T17:01:37.279 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-08-17T17:01:37.333 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-08-17T17:01:37.379 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-08-17T17:01:37.437 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-08-17T17:01:37.501 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-08-17T17:01:37.684 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-osd-2:18.2.4-719.gced649f8.el9.x86_64 30/33 2024-08-17T17:01:37.692 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-08-17T17:01:37.712 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-osd-2:18.2.4-719.gced649f8.el9.x86_64 30/33 2024-08-17T17:01:37.712 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T17:01:37.712 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-17T17:01:37.712 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-17T17:01:37.712 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-17T17:01:37.712 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:37.733 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-08-17T17:01:37.820 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-08-17T17:01:37.915 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-08-17T17:01:37.949 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-08-17T17:01:38.064 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-08-17T17:01:38.151 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mon-2:18.2.4-719.gced649f8.el9.x86_64 31/33 2024-08-17T17:01:38.175 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mon-2:18.2.4-719.gced649f8.el9.x86_64 31/33 2024-08-17T17:01:38.176 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T17:01:38.176 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-17T17:01:38.176 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-17T17:01:38.176 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-17T17:01:38.176 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:38.215 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-719.gced649f8.el9.n 28/33 2024-08-17T17:01:38.251 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-2:18.2.4-719.gced649f8.el9.x86_64 29/33 2024-08-17T17:01:38.278 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-2:18.2.4-719.gced649f8.el9.x86_64 29/33 2024-08-17T17:01:38.278 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T17:01:38.278 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-17T17:01:38.278 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-17T17:01:38.278 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-17T17:01:38.278 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:38.377 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mds-2:18.2.4-719.gced649f8.el9.x86_64 32/33 2024-08-17T17:01:38.401 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mds-2:18.2.4-719.gced649f8.el9.x86_64 32/33 2024-08-17T17:01:38.401 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T17:01:38.401 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-17T17:01:38.401 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-17T17:01:38.402 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-17T17:01:38.402 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:38.467 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-2:18.2.4-719.gced649f8.el9.x86_64 33/33 2024-08-17T17:01:39.676 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-osd-2:18.2.4-719.gced649f8.el9.x86_64 30/33 2024-08-17T17:01:39.704 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-osd-2:18.2.4-719.gced649f8.el9.x86_64 30/33 2024-08-17T17:01:39.704 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T17:01:39.704 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-17T17:01:39.704 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-17T17:01:39.705 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-17T17:01:39.705 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:40.096 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-2:18.2.4-719.gced649f8.el9.x86_64 33/33 2024-08-17T17:01:40.096 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-2:18.2.4-719.gced649f8.el9.x86_64 1/33 2024-08-17T17:01:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mds-2:18.2.4-719.gced649f8.el9.x86_64 2/33 2024-08-17T17:01:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-2:18.2.4-719.gced649f8.el9.x86_64 3/33 2024-08-17T17:01:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mon-2:18.2.4-719.gced649f8.el9.x86_64 4/33 2024-08-17T17:01:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-osd-2:18.2.4-719.gced649f8.el9.x86_64 5/33 2024-08-17T17:01:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-719.gced649f8.el9.n 6/33 2024-08-17T17:01:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-08-17T17:01:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-08-17T17:01:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-08-17T17:01:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-08-17T17:01:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-08-17T17:01:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-08-17T17:01:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-08-17T17:01:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-08-17T17:01:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-08-17T17:01:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-08-17T17:01:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-08-17T17:01:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-08-17T17:01:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-08-17T17:01:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-08-17T17:01:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-08-17T17:01:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-08-17T17:01:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-08-17T17:01:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-08-17T17:01:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-08-17T17:01:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-08-17T17:01:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-08-17T17:01:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-08-17T17:01:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-08-17T17:01:40.099 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-08-17T17:01:40.099 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-08-17T17:01:40.099 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-08-17T17:01:40.144 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mon-2:18.2.4-719.gced649f8.el9.x86_64 31/33 2024-08-17T17:01:40.168 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mon-2:18.2.4-719.gced649f8.el9.x86_64 31/33 2024-08-17T17:01:40.168 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T17:01:40.168 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-17T17:01:40.168 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-17T17:01:40.168 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-17T17:01:40.169 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:40.369 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mds-2:18.2.4-719.gced649f8.el9.x86_64 32/33 2024-08-17T17:01:40.393 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mds-2:18.2.4-719.gced649f8.el9.x86_64 32/33 2024-08-17T17:01:40.393 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T17:01:40.393 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-17T17:01:40.393 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-17T17:01:40.393 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-17T17:01:40.394 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:40.458 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-2:18.2.4-719.gced649f8.el9.x86_64 33/33 2024-08-17T17:01:41.077 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-08-17T17:01:41.078 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:41.078 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-17T17:01:41.078 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:41.078 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:41.078 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:41.078 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core-2:18.2.4-719.gced649f8.el9.noarch 2024-08-17T17:01:41.078 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:41.078 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:41.078 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-17T17:01:41.078 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-17T17:01:41.078 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-17T17:01:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-17T17:01:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-17T17:01:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-17T17:01:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-17T17:01:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-17T17:01:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-17T17:01:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-17T17:01:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-17T17:01:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-17T17:01:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-17T17:01:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-17T17:01:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-17T17:01:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ply-3.11-14.el9.noarch 2024-08-17T17:01:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-17T17:01:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-17T17:01:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-17T17:01:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-17T17:01:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-17T17:01:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-17T17:01:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-17T17:01:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-17T17:01:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-17T17:01:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-17T17:01:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-17T17:01:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:41.081 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:01:41.453 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-base 2024-08-17T17:01:42.018 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:49 ago on Sat 17 Aug 2024 05:00:53 PM UTC. 2024-08-17T17:01:42.103 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-2:18.2.4-719.gced649f8.el9.x86_64 33/33 2024-08-17T17:01:42.104 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-2:18.2.4-719.gced649f8.el9.x86_64 1/33 2024-08-17T17:01:42.104 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mds-2:18.2.4-719.gced649f8.el9.x86_64 2/33 2024-08-17T17:01:42.104 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-2:18.2.4-719.gced649f8.el9.x86_64 3/33 2024-08-17T17:01:42.104 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mon-2:18.2.4-719.gced649f8.el9.x86_64 4/33 2024-08-17T17:01:42.104 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-osd-2:18.2.4-719.gced649f8.el9.x86_64 5/33 2024-08-17T17:01:42.104 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-719.gced649f8.el9.n 6/33 2024-08-17T17:01:42.104 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-08-17T17:01:42.104 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-08-17T17:01:42.104 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-08-17T17:01:42.104 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-08-17T17:01:42.104 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-08-17T17:01:42.104 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-08-17T17:01:42.104 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-08-17T17:01:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-08-17T17:01:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-08-17T17:01:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-08-17T17:01:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-08-17T17:01:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-08-17T17:01:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-08-17T17:01:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-08-17T17:01:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-08-17T17:01:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-08-17T17:01:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-08-17T17:01:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-08-17T17:01:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-08-17T17:01:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-08-17T17:01:42.105 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-08-17T17:01:42.106 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-08-17T17:01:42.106 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-08-17T17:01:42.106 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-08-17T17:01:42.106 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-08-17T17:01:42.106 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-08-17T17:01:42.106 INFO:teuthology.orchestra.run.smithi136.stdout:Package ceph-base-2:18.2.4-719.gced649f8.el9.x86_64 is already installed. 2024-08-17T17:01:42.144 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:01:42.145 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-08-17T17:01:42.145 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:01:42.203 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install cephadm 2024-08-17T17:01:42.749 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:49 ago on Sat 17 Aug 2024 05:00:53 PM UTC. 2024-08-17T17:01:42.876 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:01:42.877 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:01:42.877 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-08-17T17:01:42.877 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:01:42.877 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-17T17:01:42.877 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm noarch 2:18.2.4-719.gced649f8.el9 ceph-noarch 224 k 2024-08-17T17:01:42.877 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:42.877 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-17T17:01:42.877 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:01:42.877 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-08-17T17:01:42.877 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:42.878 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 224 k 2024-08-17T17:01:42.878 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 216 k 2024-08-17T17:01:42.878 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-17T17:01:42.954 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-08-17T17:01:42.955 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:42.955 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-08-17T17:01:42.955 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:42.955 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mds-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:42.955 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:42.955 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-modules-core-2:18.2.4-719.gced649f8.el9.noarch 2024-08-17T17:01:42.955 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mon-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:42.955 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-osd-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:42.955 INFO:teuthology.orchestra.run.smithi165.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-17T17:01:42.956 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-17T17:01:42.956 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-17T17:01:42.956 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-17T17:01:42.956 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-17T17:01:42.956 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-17T17:01:42.956 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-17T17:01:42.956 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-17T17:01:42.956 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-17T17:01:42.956 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-17T17:01:42.956 INFO:teuthology.orchestra.run.smithi165.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-17T17:01:42.956 INFO:teuthology.orchestra.run.smithi165.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-17T17:01:42.957 INFO:teuthology.orchestra.run.smithi165.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-17T17:01:42.957 INFO:teuthology.orchestra.run.smithi165.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-17T17:01:42.957 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-17T17:01:42.957 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ply-3.11-14.el9.noarch 2024-08-17T17:01:42.957 INFO:teuthology.orchestra.run.smithi165.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-17T17:01:42.957 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-17T17:01:42.957 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-17T17:01:42.957 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-17T17:01:42.958 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-17T17:01:42.958 INFO:teuthology.orchestra.run.smithi165.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-17T17:01:42.958 INFO:teuthology.orchestra.run.smithi165.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-17T17:01:42.958 INFO:teuthology.orchestra.run.smithi165.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-17T17:01:42.958 INFO:teuthology.orchestra.run.smithi165.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-17T17:01:42.958 INFO:teuthology.orchestra.run.smithi165.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-17T17:01:42.958 INFO:teuthology.orchestra.run.smithi165.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-17T17:01:42.958 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:42.959 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:01:43.102 INFO:teuthology.orchestra.run.smithi136.stdout:cephadm-18.2.4-719.gced649f8.el9.noarch.rpm 1.0 MB/s | 224 kB 00:00 2024-08-17T17:01:43.102 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:01:43.103 INFO:teuthology.orchestra.run.smithi136.stdout:Total 995 kB/s | 224 kB 00:00 2024-08-17T17:01:43.103 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-17T17:01:43.108 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-17T17:01:43.108 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-17T17:01:43.110 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-17T17:01:43.111 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-17T17:01:43.179 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-17T17:01:43.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-base 2024-08-17T17:01:43.632 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephadm-2:18.2.4-719.gced649f8.el9.noarch 1/1 2024-08-17T17:01:43.714 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephadm-2:18.2.4-719.gced649f8.el9.noarch 1/1 2024-08-17T17:01:43.933 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:48 ago on Sat 17 Aug 2024 05:00:55 PM UTC. 2024-08-17T17:01:44.019 INFO:teuthology.orchestra.run.smithi165.stdout:Package ceph-base-2:18.2.4-719.gced649f8.el9.x86_64 is already installed. 2024-08-17T17:01:44.059 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephadm-2:18.2.4-719.gced649f8.el9.noarch 1/1 2024-08-17T17:01:44.060 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:01:44.061 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-08-17T17:01:44.061 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:01:44.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install cephadm 2024-08-17T17:01:44.317 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephadm-2:18.2.4-719.gced649f8.el9.noarch 1/1 2024-08-17T17:01:44.317 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:44.317 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-17T17:01:44.318 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm-2:18.2.4-719.gced649f8.el9.noarch 2024-08-17T17:01:44.318 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:44.318 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:01:44.514 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-immutable-object-cache 2024-08-17T17:01:44.685 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:49 ago on Sat 17 Aug 2024 05:00:55 PM UTC. 2024-08-17T17:01:44.811 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:01:44.812 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:01:44.812 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-08-17T17:01:44.812 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:01:44.812 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-08-17T17:01:44.812 INFO:teuthology.orchestra.run.smithi165.stdout: cephadm noarch 2:18.2.4-719.gced649f8.el9 ceph-noarch 224 k 2024-08-17T17:01:44.812 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:44.812 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-08-17T17:01:44.813 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:01:44.813 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-08-17T17:01:44.813 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:44.813 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 224 k 2024-08-17T17:01:44.813 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 216 k 2024-08-17T17:01:44.813 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-08-17T17:01:45.045 INFO:teuthology.orchestra.run.smithi165.stdout:cephadm-18.2.4-719.gced649f8.el9.noarch.rpm 970 kB/s | 224 kB 00:00 2024-08-17T17:01:45.045 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:01:45.046 INFO:teuthology.orchestra.run.smithi165.stdout:Total 963 kB/s | 224 kB 00:00 2024-08-17T17:01:45.046 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-08-17T17:01:45.051 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-08-17T17:01:45.051 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-08-17T17:01:45.054 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-08-17T17:01:45.054 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-08-17T17:01:45.072 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:52 ago on Sat 17 Aug 2024 05:00:53 PM UTC. 2024-08-17T17:01:45.124 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-08-17T17:01:45.201 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:01:45.202 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:01:45.202 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-08-17T17:01:45.202 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:01:45.202 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-17T17:01:45.202 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-719.gced649f8.el9 ceph 143 k 2024-08-17T17:01:45.202 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:45.202 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-17T17:01:45.202 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:01:45.202 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-08-17T17:01:45.203 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:45.203 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 143 k 2024-08-17T17:01:45.203 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 395 k 2024-08-17T17:01:45.203 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-17T17:01:45.409 INFO:teuthology.orchestra.run.smithi136.stdout:ceph-immutable-object-cache-18.2.4-719.gced649f 692 kB/s | 143 kB 00:00 2024-08-17T17:01:45.410 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:01:45.410 INFO:teuthology.orchestra.run.smithi136.stdout:Total 686 kB/s | 143 kB 00:00 2024-08-17T17:01:45.410 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-17T17:01:45.418 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-17T17:01:45.418 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-17T17:01:45.474 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-17T17:01:45.474 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-17T17:01:45.584 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: cephadm-2:18.2.4-719.gced649f8.el9.noarch 1/1 2024-08-17T17:01:45.617 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-17T17:01:45.673 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : cephadm-2:18.2.4-719.gced649f8.el9.noarch 1/1 2024-08-17T17:01:45.693 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-719.gced649f8.e 1/1 2024-08-17T17:01:45.718 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-719.gced649f8.e 1/1 2024-08-17T17:01:45.718 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T17:01:45.719 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-17T17:01:45.719 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:46.070 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: cephadm-2:18.2.4-719.gced649f8.el9.noarch 1/1 2024-08-17T17:01:46.345 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : cephadm-2:18.2.4-719.gced649f8.el9.noarch 1/1 2024-08-17T17:01:46.346 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:46.346 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-08-17T17:01:46.346 INFO:teuthology.orchestra.run.smithi165.stdout: cephadm-2:18.2.4-719.gced649f8.el9.noarch 2024-08-17T17:01:46.346 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:46.346 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:01:46.559 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-immutable-object-cache 2024-08-17T17:01:46.701 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-719.gced649f8.e 1/1 2024-08-17T17:01:46.701 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:46.701 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-17T17:01:46.701 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:46.701 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:46.701 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:01:47.006 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr 2024-08-17T17:01:47.130 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:52 ago on Sat 17 Aug 2024 05:00:55 PM UTC. 2024-08-17T17:01:47.260 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:01:47.261 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:01:47.261 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repo Size 2024-08-17T17:01:47.262 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:01:47.262 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-08-17T17:01:47.262 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-719.gced649f8.el9 ceph 143 k 2024-08-17T17:01:47.262 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:47.262 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-08-17T17:01:47.262 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:01:47.263 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-08-17T17:01:47.263 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:47.263 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 143 k 2024-08-17T17:01:47.263 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 395 k 2024-08-17T17:01:47.263 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-08-17T17:01:47.477 INFO:teuthology.orchestra.run.smithi165.stdout:ceph-immutable-object-cache-18.2.4-719.gced649f 662 kB/s | 143 kB 00:00 2024-08-17T17:01:47.478 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:01:47.478 INFO:teuthology.orchestra.run.smithi165.stdout:Total 656 kB/s | 143 kB 00:00 2024-08-17T17:01:47.478 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-08-17T17:01:47.486 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-08-17T17:01:47.486 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-08-17T17:01:47.543 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-08-17T17:01:47.543 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-08-17T17:01:47.574 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:54 ago on Sat 17 Aug 2024 05:00:53 PM UTC. 2024-08-17T17:01:47.661 INFO:teuthology.orchestra.run.smithi136.stdout:Package ceph-mgr-2:18.2.4-719.gced649f8.el9.x86_64 is already installed. 2024-08-17T17:01:47.687 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-08-17T17:01:47.703 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:01:47.704 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-08-17T17:01:47.704 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:01:47.761 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-dashboard 2024-08-17T17:01:47.786 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-719.gced649f8.e 1/1 2024-08-17T17:01:47.811 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-719.gced649f8.e 1/1 2024-08-17T17:01:47.811 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T17:01:47.811 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-17T17:01:47.812 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:48.307 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:55 ago on Sat 17 Aug 2024 05:00:53 PM UTC. 2024-08-17T17:01:48.433 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:01:48.434 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:01:48.434 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-08-17T17:01:48.434 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:01:48.434 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-17T17:01:48.435 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard noarch 2:18.2.4-719.gced649f8.el9 ceph-noarch 3.5 M 2024-08-17T17:01:48.435 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-08-17T17:01:48.435 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards noarch 2:18.2.4-719.gced649f8.el9 ceph-noarch 24 k 2024-08-17T17:01:48.435 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts noarch 2:18.2.4-719.gced649f8.el9 ceph-noarch 16 k 2024-08-17T17:01:48.435 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-17T17:01:48.435 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-17T17:01:48.435 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:48.435 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-17T17:01:48.435 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:01:48.435 INFO:teuthology.orchestra.run.smithi136.stdout:Install 5 Packages 2024-08-17T17:01:48.435 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:48.436 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 3.8 M 2024-08-17T17:01:48.436 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 87 M 2024-08-17T17:01:48.436 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-17T17:01:48.737 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-719.gced649f8.e 1/1 2024-08-17T17:01:48.737 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:48.737 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-08-17T17:01:48.737 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-immutable-object-cache-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:01:48.737 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:48.737 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:01:48.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr 2024-08-17T17:01:49.063 INFO:teuthology.orchestra.run.smithi136.stdout:(1/5): ceph-grafana-dashboards-18.2.4-719.gced6 121 kB/s | 24 kB 00:00 2024-08-17T17:01:49.089 INFO:teuthology.orchestra.run.smithi136.stdout:(2/5): ceph-prometheus-alerts-18.2.4-719.gced64 73 kB/s | 16 kB 00:00 2024-08-17T17:01:49.172 INFO:teuthology.orchestra.run.smithi136.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 283 kB/s | 31 kB 00:00 2024-08-17T17:01:49.206 INFO:teuthology.orchestra.run.smithi136.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-08-17T17:01:49.298 INFO:teuthology.orchestra.run.smithi136.stdout:(5/5): ceph-mgr-dashboard-18.2.4-719.gced649f8. 8.1 MB/s | 3.5 MB 00:00 2024-08-17T17:01:49.301 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:01:49.301 INFO:teuthology.orchestra.run.smithi136.stdout:Total 4.4 MB/s | 3.8 MB 00:00 2024-08-17T17:01:49.302 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-17T17:01:49.309 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-17T17:01:49.309 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-17T17:01:49.370 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-17T17:01:49.371 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-17T17:01:49.530 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:54 ago on Sat 17 Aug 2024 05:00:55 PM UTC. 2024-08-17T17:01:49.617 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-17T17:01:49.619 INFO:teuthology.orchestra.run.smithi165.stdout:Package ceph-mgr-2:18.2.4-719.gced649f8.el9.x86_64 is already installed. 2024-08-17T17:01:49.660 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:01:49.661 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-08-17T17:01:49.661 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:01:49.704 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-08-17T17:01:49.721 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr-dashboard 2024-08-17T17:01:49.736 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-08-17T17:01:49.765 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-719.gced649f8.el9.no 3/5 2024-08-17T17:01:50.266 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:55 ago on Sat 17 Aug 2024 05:00:55 PM UTC. 2024-08-17T17:01:50.394 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:01:50.395 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:01:50.395 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-08-17T17:01:50.395 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:01:50.395 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-08-17T17:01:50.395 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-dashboard noarch 2:18.2.4-719.gced649f8.el9 ceph-noarch 3.5 M 2024-08-17T17:01:50.395 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-08-17T17:01:50.395 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-grafana-dashboards noarch 2:18.2.4-719.gced649f8.el9 ceph-noarch 24 k 2024-08-17T17:01:50.395 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-prometheus-alerts noarch 2:18.2.4-719.gced649f8.el9 ceph-noarch 16 k 2024-08-17T17:01:50.396 INFO:teuthology.orchestra.run.smithi165.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-17T17:01:50.396 INFO:teuthology.orchestra.run.smithi165.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-17T17:01:50.396 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:50.396 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-08-17T17:01:50.396 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:01:50.396 INFO:teuthology.orchestra.run.smithi165.stdout:Install 5 Packages 2024-08-17T17:01:50.396 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:50.397 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 3.8 M 2024-08-17T17:01:50.397 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 87 M 2024-08-17T17:01:50.397 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-08-17T17:01:50.802 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-719.gced649f8.el9.n 4/5 2024-08-17T17:01:50.835 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-719.gced649f8.el9.noarch 5/5 2024-08-17T17:01:51.015 INFO:teuthology.orchestra.run.smithi165.stdout:(1/5): ceph-grafana-dashboards-18.2.4-719.gced6 130 kB/s | 24 kB 00:00 2024-08-17T17:01:51.032 INFO:teuthology.orchestra.run.smithi165.stdout:(2/5): ceph-prometheus-alerts-18.2.4-719.gced64 81 kB/s | 16 kB 00:00 2024-08-17T17:01:51.116 INFO:teuthology.orchestra.run.smithi165.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 307 kB/s | 31 kB 00:00 2024-08-17T17:01:51.141 INFO:teuthology.orchestra.run.smithi165.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2024-08-17T17:01:51.233 INFO:teuthology.orchestra.run.smithi165.stdout:(5/5): ceph-mgr-dashboard-18.2.4-719.gced649f8. 8.7 MB/s | 3.5 MB 00:00 2024-08-17T17:01:51.237 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:01:51.237 INFO:teuthology.orchestra.run.smithi165.stdout:Total 4.5 MB/s | 3.8 MB 00:00 2024-08-17T17:01:51.237 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-08-17T17:01:51.244 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-08-17T17:01:51.244 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-08-17T17:01:51.306 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-08-17T17:01:51.306 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-08-17T17:01:51.570 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-08-17T17:01:51.647 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-08-17T17:01:51.688 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-08-17T17:01:51.717 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-719.gced649f8.el9.no 3/5 2024-08-17T17:01:51.900 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-719.gced649f8.el9.noarch 5/5 2024-08-17T17:01:51.900 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-719.gced649f8.el9.n 1/5 2024-08-17T17:01:51.900 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-719.gced649f8.el9.noarch 2/5 2024-08-17T17:01:51.900 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-719.gced649f8.el9.no 3/5 2024-08-17T17:01:51.900 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-08-17T17:01:52.167 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-08-17T17:01:52.167 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:52.167 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-17T17:01:52.167 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards-2:18.2.4-719.gced649f8.el9.noarch 2024-08-17T17:01:52.167 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard-2:18.2.4-719.gced649f8.el9.noarch 2024-08-17T17:01:52.167 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts-2:18.2.4-719.gced649f8.el9.noarch 2024-08-17T17:01:52.167 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-17T17:01:52.167 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-17T17:01:52.167 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:52.167 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:01:52.381 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-17T17:01:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-719.gced649f8.el9.n 4/5 2024-08-17T17:01:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-719.gced649f8.el9.noarch 5/5 2024-08-17T17:01:52.922 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:59 ago on Sat 17 Aug 2024 05:00:53 PM UTC. 2024-08-17T17:01:53.050 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:01:53.051 INFO:teuthology.orchestra.run.smithi136.stdout:====================================================================================== 2024-08-17T17:01:53.051 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-08-17T17:01:53.051 INFO:teuthology.orchestra.run.smithi136.stdout:====================================================================================== 2024-08-17T17:01:53.052 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-17T17:01:53.052 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-719.gced649f8.el9 ceph-noarch 7.4 M 2024-08-17T17:01:53.052 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-08-17T17:01:53.052 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-17T17:01:53.052 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-17T17:01:53.052 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-17T17:01:53.052 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-17T17:01:53.052 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-17T17:01:53.052 INFO:teuthology.orchestra.run.smithi136.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-17T17:01:53.052 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-17T17:01:53.052 INFO:teuthology.orchestra.run.smithi136.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-08-17T17:01:53.052 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-17T17:01:53.052 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-17T17:01:53.053 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-17T17:01:53.053 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:53.053 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-17T17:01:53.053 INFO:teuthology.orchestra.run.smithi136.stdout:====================================================================================== 2024-08-17T17:01:53.053 INFO:teuthology.orchestra.run.smithi136.stdout:Install 12 Packages 2024-08-17T17:01:53.053 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:01:53.053 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 38 M 2024-08-17T17:01:53.053 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 209 M 2024-08-17T17:01:53.054 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-17T17:01:53.463 INFO:teuthology.orchestra.run.smithi136.stdout:(1/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.9 MB/s | 803 kB 00:00 2024-08-17T17:01:53.488 INFO:teuthology.orchestra.run.smithi136.stdout:(2/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-08-17T17:01:53.606 INFO:teuthology.orchestra.run.smithi136.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 230 kB/s | 33 kB 00:00 2024-08-17T17:01:53.656 INFO:teuthology.orchestra.run.smithi136.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-08-17T17:01:53.707 INFO:teuthology.orchestra.run.smithi136.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 703 kB/s | 35 kB 00:00 2024-08-17T17:01:53.807 INFO:teuthology.orchestra.run.smithi136.stdout:(6/12): ceph-mgr-diskprediction-local-18.2.4-71 15 MB/s | 7.4 MB 00:00 2024-08-17T17:01:53.831 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-719.gced649f8.el9.noarch 5/5 2024-08-17T17:01:53.832 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-719.gced649f8.el9.n 1/5 2024-08-17T17:01:53.832 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-719.gced649f8.el9.noarch 2/5 2024-08-17T17:01:53.832 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-719.gced649f8.el9.no 3/5 2024-08-17T17:01:53.832 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-08-17T17:01:53.974 INFO:teuthology.orchestra.run.smithi136.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.4 MB/s | 244 kB 00:00 2024-08-17T17:01:54.068 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-08-17T17:01:54.068 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:54.068 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-08-17T17:01:54.068 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-grafana-dashboards-2:18.2.4-719.gced649f8.el9.noarch 2024-08-17T17:01:54.068 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-dashboard-2:18.2.4-719.gced649f8.el9.noarch 2024-08-17T17:01:54.068 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-prometheus-alerts-2:18.2.4-719.gced649f8.el9.noarch 2024-08-17T17:01:54.068 INFO:teuthology.orchestra.run.smithi165.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-17T17:01:54.068 INFO:teuthology.orchestra.run.smithi165.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-17T17:01:54.068 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:54.069 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:01:54.174 INFO:teuthology.orchestra.run.smithi136.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.4 MB/s | 3.0 MB 00:00 2024-08-17T17:01:54.258 INFO:teuthology.orchestra.run.smithi136.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 250 kB 00:00 2024-08-17T17:01:54.301 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-17T17:01:54.650 INFO:teuthology.orchestra.run.smithi136.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 5.0 MB/s | 4.7 MB 00:00 2024-08-17T17:01:54.850 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:59 ago on Sat 17 Aug 2024 05:00:55 PM UTC. 2024-08-17T17:01:54.933 INFO:teuthology.orchestra.run.smithi136.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.3 MB/s | 5.1 MB 00:00 2024-08-17T17:01:54.979 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:01:54.980 INFO:teuthology.orchestra.run.smithi165.stdout:====================================================================================== 2024-08-17T17:01:54.980 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-08-17T17:01:54.980 INFO:teuthology.orchestra.run.smithi165.stdout:====================================================================================== 2024-08-17T17:01:54.980 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-08-17T17:01:54.980 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-719.gced649f8.el9 ceph-noarch 7.4 M 2024-08-17T17:01:54.980 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-08-17T17:01:54.980 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-17T17:01:54.980 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-17T17:01:54.980 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-17T17:01:54.980 INFO:teuthology.orchestra.run.smithi165.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-17T17:01:54.981 INFO:teuthology.orchestra.run.smithi165.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-17T17:01:54.981 INFO:teuthology.orchestra.run.smithi165.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-17T17:01:54.981 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-17T17:01:54.981 INFO:teuthology.orchestra.run.smithi165.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-08-17T17:01:54.981 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-17T17:01:54.981 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-17T17:01:54.981 INFO:teuthology.orchestra.run.smithi165.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-17T17:01:54.981 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:54.981 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-08-17T17:01:54.981 INFO:teuthology.orchestra.run.smithi165.stdout:====================================================================================== 2024-08-17T17:01:54.981 INFO:teuthology.orchestra.run.smithi165.stdout:Install 12 Packages 2024-08-17T17:01:54.981 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:01:54.982 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 38 M 2024-08-17T17:01:54.982 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 209 M 2024-08-17T17:01:54.982 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-08-17T17:01:55.374 INFO:teuthology.orchestra.run.smithi165.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 901 kB/s | 195 kB 00:00 2024-08-17T17:01:55.516 INFO:teuthology.orchestra.run.smithi165.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 803 kB 00:00 2024-08-17T17:01:55.616 INFO:teuthology.orchestra.run.smithi165.stdout:(3/12): ceph-mgr-diskprediction-local-18.2.4-71 16 MB/s | 7.4 MB 00:00 2024-08-17T17:01:56.008 INFO:teuthology.orchestra.run.smithi165.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 52 kB/s | 33 kB 00:00 2024-08-17T17:01:56.133 INFO:teuthology.orchestra.run.smithi165.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 37 kB/s | 18 kB 00:00 2024-08-17T17:01:56.391 INFO:teuthology.orchestra.run.smithi136.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 7.6 MB/s | 16 MB 00:02 2024-08-17T17:01:56.392 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:01:56.393 INFO:teuthology.orchestra.run.smithi136.stdout:Total 11 MB/s | 38 MB 00:03 2024-08-17T17:01:56.584 INFO:teuthology.orchestra.run.smithi165.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 61 kB/s | 35 kB 00:00 2024-08-17T17:01:56.651 INFO:teuthology.orchestra.run.smithi165.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.6 MB/s | 3.0 MB 00:01 2024-08-17T17:01:56.740 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-17T17:01:56.768 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-17T17:01:56.768 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-17T17:01:57.261 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-17T17:01:57.262 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-17T17:01:57.267 INFO:teuthology.orchestra.run.smithi165.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 4.2 MB/s | 4.7 MB 00:01 2024-08-17T17:01:57.368 INFO:teuthology.orchestra.run.smithi165.stdout:(9/12): python3-devel-3.9.18-3.el9.x86_64.rpm 311 kB/s | 244 kB 00:00 2024-08-17T17:01:57.810 INFO:teuthology.orchestra.run.smithi165.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.4 MB/s | 5.1 MB 00:01 2024-08-17T17:01:58.018 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-17T17:01:58.076 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-17T17:01:58.119 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-08-17T17:01:58.135 INFO:teuthology.orchestra.run.smithi165.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 289 kB/s | 250 kB 00:00 2024-08-17T17:01:58.231 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-08-17T17:01:58.282 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-08-17T17:01:58.459 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-08-17T17:01:58.490 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-08-17T17:01:58.565 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-17T17:01:58.877 INFO:teuthology.orchestra.run.smithi165.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-08-17T17:01:58.878 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:01:58.879 INFO:teuthology.orchestra.run.smithi165.stdout:Total 9.8 MB/s | 38 MB 00:03 2024-08-17T17:01:59.231 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-17T17:01:59.234 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-08-17T17:01:59.262 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-08-17T17:01:59.262 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-08-17T17:01:59.353 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-17T17:01:59.776 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-08-17T17:01:59.777 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-08-17T17:02:00.544 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-08-17T17:02:00.605 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-17T17:02:00.618 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-17T17:02:00.674 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-08-17T17:02:00.776 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-08-17T17:02:00.810 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-08-17T17:02:00.989 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-08-17T17:02:01.017 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-08-17T17:02:01.094 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-17T17:02:01.640 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-17T17:02:01.665 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-719.gced649 12/12 2024-08-17T17:02:01.777 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-17T17:02:01.898 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-17T17:02:03.152 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-17T17:02:04.187 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-17T17:02:04.226 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-719.gced649 12/12 2024-08-17T17:02:04.297 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-719.gced649 12/12 2024-08-17T17:02:04.297 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-719.gced649 1/12 2024-08-17T17:02:04.297 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-08-17T17:02:04.297 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-08-17T17:02:04.297 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-17T17:02:04.297 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-17T17:02:04.297 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-17T17:02:04.298 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-08-17T17:02:04.298 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-08-17T17:02:04.298 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-08-17T17:02:04.298 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-17T17:02:04.298 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-17T17:02:04.683 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-17T17:02:04.683 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:04.683 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-17T17:02:04.683 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local-2:18.2.4-719.gced649f8.el9.noarch 2024-08-17T17:02:04.683 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-17T17:02:04.683 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-17T17:02:04.683 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-17T17:02:04.683 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-17T17:02:04.683 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-17T17:02:04.683 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-17T17:02:04.683 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-17T17:02:04.684 INFO:teuthology.orchestra.run.smithi136.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-08-17T17:02:04.684 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-17T17:02:04.684 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-17T17:02:04.684 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-17T17:02:04.684 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:04.684 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:02:05.095 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-rook 2024-08-17T17:02:05.660 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:12 ago on Sat 17 Aug 2024 05:00:53 PM UTC. 2024-08-17T17:02:05.788 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:02:05.789 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:02:05.789 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-08-17T17:02:05.789 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:02:05.789 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-17T17:02:05.789 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook noarch 2:18.2.4-719.gced649f8.el9 ceph-noarch 49 k 2024-08-17T17:02:05.789 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-08-17T17:02:05.789 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-17T17:02:05.789 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-17T17:02:05.789 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth noarch 1:2.33.0-1.el9 epel 220 k 2024-08-17T17:02:05.790 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-17T17:02:05.790 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-17T17:02:05.790 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-17T17:02:05.790 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-17T17:02:05.790 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-17T17:02:05.790 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-17T17:02:05.790 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-17T17:02:05.790 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-17T17:02:05.790 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-17T17:02:05.790 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:05.790 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-17T17:02:05.790 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:02:05.791 INFO:teuthology.orchestra.run.smithi136.stdout:Install 13 Packages 2024-08-17T17:02:05.791 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:05.791 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 2.2 M 2024-08-17T17:02:05.791 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 26 M 2024-08-17T17:02:05.791 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-17T17:02:06.458 INFO:teuthology.orchestra.run.smithi136.stdout:(1/13): ceph-mgr-rook-18.2.4-719.gced649f8.el9. 277 kB/s | 49 kB 00:00 2024-08-17T17:02:06.542 INFO:teuthology.orchestra.run.smithi136.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 98 kB/s | 26 kB 00:00 2024-08-17T17:02:06.567 INFO:teuthology.orchestra.run.smithi136.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 67 kB/s | 19 kB 00:00 2024-08-17T17:02:06.816 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-719.gced649 12/12 2024-08-17T17:02:06.816 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-719.gced649 1/12 2024-08-17T17:02:06.816 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-08-17T17:02:06.816 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-08-17T17:02:06.816 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-17T17:02:06.816 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-17T17:02:06.816 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-17T17:02:06.817 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-08-17T17:02:06.817 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-08-17T17:02:06.817 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-08-17T17:02:06.817 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-17T17:02:06.817 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-17T17:02:06.951 INFO:teuthology.orchestra.run.smithi136.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 451 kB/s | 222 kB 00:00 2024-08-17T17:02:06.976 INFO:teuthology.orchestra.run.smithi136.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 366 kB/s | 159 kB 00:00 2024-08-17T17:02:07.043 INFO:teuthology.orchestra.run.smithi136.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 483 kB/s | 32 kB 00:00 2024-08-17T17:02:07.068 INFO:teuthology.orchestra.run.smithi136.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 568 kB/s | 14 kB 00:00 2024-08-17T17:02:07.094 INFO:teuthology.orchestra.run.smithi136.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 530 kB/s | 279 kB 00:00 2024-08-17T17:02:07.144 INFO:teuthology.orchestra.run.smithi136.stdout:(9/13): python3-google-auth-2.33.0-1.el9.noarch 2.9 MB/s | 220 kB 00:00 2024-08-17T17:02:07.178 INFO:teuthology.orchestra.run.smithi136.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.9 MB/s | 59 kB 00:00 2024-08-17T17:02:07.187 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-17T17:02:07.188 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:07.188 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-08-17T17:02:07.188 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-diskprediction-local-2:18.2.4-719.gced649f8.el9.noarch 2024-08-17T17:02:07.188 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-17T17:02:07.188 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-17T17:02:07.188 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-17T17:02:07.188 INFO:teuthology.orchestra.run.smithi165.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-17T17:02:07.188 INFO:teuthology.orchestra.run.smithi165.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-17T17:02:07.188 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-17T17:02:07.188 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-17T17:02:07.188 INFO:teuthology.orchestra.run.smithi165.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-08-17T17:02:07.188 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-17T17:02:07.188 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-17T17:02:07.189 INFO:teuthology.orchestra.run.smithi165.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-17T17:02:07.189 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:07.189 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:02:07.211 INFO:teuthology.orchestra.run.smithi136.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-08-17T17:02:07.294 INFO:teuthology.orchestra.run.smithi136.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 5.1 MB/s | 1.0 MB 00:00 2024-08-17T17:02:07.320 INFO:teuthology.orchestra.run.smithi136.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 145 kB/s | 54 kB 00:00 2024-08-17T17:02:07.322 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:02:07.322 INFO:teuthology.orchestra.run.smithi136.stdout:Total 1.4 MB/s | 2.2 MB 00:01 2024-08-17T17:02:07.384 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-17T17:02:07.398 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-17T17:02:07.398 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-17T17:02:07.485 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-17T17:02:07.486 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-17T17:02:07.623 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr-rook 2024-08-17T17:02:07.999 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-17T17:02:08.188 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:13 ago on Sat 17 Aug 2024 05:00:55 PM UTC. 2024-08-17T17:02:08.198 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-17T17:02:08.256 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-17T17:02:08.311 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-17T17:02:08.317 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:02:08.318 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:02:08.318 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-08-17T17:02:08.318 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:02:08.318 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-08-17T17:02:08.318 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-rook noarch 2:18.2.4-719.gced649f8.el9 ceph-noarch 49 k 2024-08-17T17:02:08.319 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-08-17T17:02:08.319 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-17T17:02:08.319 INFO:teuthology.orchestra.run.smithi165.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-17T17:02:08.319 INFO:teuthology.orchestra.run.smithi165.stdout: python3-google-auth noarch 1:2.33.0-1.el9 epel 220 k 2024-08-17T17:02:08.319 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-17T17:02:08.319 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-17T17:02:08.319 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-17T17:02:08.319 INFO:teuthology.orchestra.run.smithi165.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-17T17:02:08.319 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-17T17:02:08.319 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-17T17:02:08.319 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-17T17:02:08.319 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-17T17:02:08.319 INFO:teuthology.orchestra.run.smithi165.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-17T17:02:08.320 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:08.320 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-08-17T17:02:08.320 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:02:08.320 INFO:teuthology.orchestra.run.smithi165.stdout:Install 13 Packages 2024-08-17T17:02:08.320 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:08.320 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 2.2 M 2024-08-17T17:02:08.320 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 26 M 2024-08-17T17:02:08.321 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-08-17T17:02:08.348 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-17T17:02:08.386 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-17T17:02:08.489 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-17T17:02:08.591 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-google-auth-1:2.33.0-1.el9.noarch 7/13 2024-08-17T17:02:08.646 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-17T17:02:09.028 INFO:teuthology.orchestra.run.smithi165.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 137 kB/s | 19 kB 00:00 2024-08-17T17:02:09.054 INFO:teuthology.orchestra.run.smithi165.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 155 kB/s | 26 kB 00:00 2024-08-17T17:02:09.079 INFO:teuthology.orchestra.run.smithi165.stdout:(3/13): ceph-mgr-rook-18.2.4-719.gced649f8.el9. 259 kB/s | 49 kB 00:00 2024-08-17T17:02:09.188 INFO:teuthology.orchestra.run.smithi165.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-08-17T17:02:09.213 INFO:teuthology.orchestra.run.smithi165.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.2 MB/s | 222 kB 00:00 2024-08-17T17:02:09.255 INFO:teuthology.orchestra.run.smithi165.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 800 kB/s | 54 kB 00:00 2024-08-17T17:02:09.305 INFO:teuthology.orchestra.run.smithi165.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 279 kB 00:00 2024-08-17T17:02:09.322 INFO:teuthology.orchestra.run.smithi165.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 296 kB/s | 32 kB 00:00 2024-08-17T17:02:09.364 INFO:teuthology.orchestra.run.smithi165.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 132 kB/s | 14 kB 00:00 2024-08-17T17:02:09.370 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-17T17:02:09.389 INFO:teuthology.orchestra.run.smithi165.stdout:(10/13): python3-google-auth-2.33.0-1.el9.noarc 3.3 MB/s | 220 kB 00:00 2024-08-17T17:02:09.406 INFO:teuthology.orchestra.run.smithi165.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.5 MB/s | 59 kB 00:00 2024-08-17T17:02:09.411 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-17T17:02:09.423 INFO:teuthology.orchestra.run.smithi165.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-08-17T17:02:09.445 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-17T17:02:09.494 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-17T17:02:09.507 INFO:teuthology.orchestra.run.smithi165.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.6 MB/s | 1.0 MB 00:00 2024-08-17T17:02:09.509 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:02:09.509 INFO:teuthology.orchestra.run.smithi165.stdout:Total 1.9 MB/s | 2.2 MB 00:01 2024-08-17T17:02:09.524 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-rook-2:18.2.4-719.gced649f8.el9.noarch 13/13 2024-08-17T17:02:09.573 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-08-17T17:02:09.588 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-08-17T17:02:09.588 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-08-17T17:02:09.676 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-08-17T17:02:09.676 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-08-17T17:02:10.178 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-08-17T17:02:10.312 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-719.gced649f8.el9.noarch 13/13 2024-08-17T17:02:10.313 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-rook-2:18.2.4-719.gced649f8.el9.noarch 1/13 2024-08-17T17:02:10.313 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-17T17:02:10.313 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-17T17:02:10.313 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-17T17:02:10.313 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-17T17:02:10.313 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-17T17:02:10.313 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-17T17:02:10.313 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-17T17:02:10.313 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-17T17:02:10.313 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-google-auth-1:2.33.0-1.el9.noarch 10/13 2024-08-17T17:02:10.313 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-17T17:02:10.313 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-17T17:02:10.358 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-17T17:02:10.418 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-17T17:02:10.473 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-17T17:02:10.510 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-17T17:02:10.556 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-17T17:02:10.632 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-17T17:02:10.632 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:10.632 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-17T17:02:10.632 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook-2:18.2.4-719.gced649f8.el9.noarch 2024-08-17T17:02:10.633 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-17T17:02:10.633 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-17T17:02:10.633 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth-1:2.33.0-1.el9.noarch 2024-08-17T17:02:10.633 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-17T17:02:10.633 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-17T17:02:10.633 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-17T17:02:10.633 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-17T17:02:10.633 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-17T17:02:10.633 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-17T17:02:10.633 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-17T17:02:10.633 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-17T17:02:10.633 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-17T17:02:10.633 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:10.633 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:02:10.669 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-17T17:02:10.771 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-google-auth-1:2.33.0-1.el9.noarch 7/13 2024-08-17T17:02:10.834 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-17T17:02:10.901 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-cephadm 2024-08-17T17:02:11.455 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:18 ago on Sat 17 Aug 2024 05:00:53 PM UTC. 2024-08-17T17:02:11.574 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-17T17:02:11.582 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:02:11.582 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:02:11.582 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-08-17T17:02:11.583 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:02:11.583 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-17T17:02:11.583 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm noarch 2:18.2.4-719.gced649f8.el9 ceph-noarch 139 k 2024-08-17T17:02:11.583 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-08-17T17:02:11.583 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-17T17:02:11.583 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-17T17:02:11.583 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-17T17:02:11.584 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-17T17:02:11.584 INFO:teuthology.orchestra.run.smithi136.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-17T17:02:11.584 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:11.584 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-17T17:02:11.584 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:02:11.584 INFO:teuthology.orchestra.run.smithi136.stdout:Install 6 Packages 2024-08-17T17:02:11.584 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:11.585 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 7.0 M 2024-08-17T17:02:11.585 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 33 M 2024-08-17T17:02:11.585 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-17T17:02:11.623 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-17T17:02:11.657 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-17T17:02:11.706 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-17T17:02:11.736 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-rook-2:18.2.4-719.gced649f8.el9.noarch 13/13 2024-08-17T17:02:12.273 INFO:teuthology.orchestra.run.smithi136.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.8 MB/s | 248 kB 00:00 2024-08-17T17:02:12.356 INFO:teuthology.orchestra.run.smithi136.stdout:(2/6): ceph-mgr-cephadm-18.2.4-719.gced649f8.el 625 kB/s | 139 kB 00:00 2024-08-17T17:02:12.415 INFO:teuthology.orchestra.run.smithi136.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.8 MB/s | 548 kB 00:00 2024-08-17T17:02:12.448 INFO:teuthology.orchestra.run.smithi136.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 628 kB/s | 58 kB 00:00 2024-08-17T17:02:12.474 INFO:teuthology.orchestra.run.smithi136.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.3 MB/s | 74 kB 00:00 2024-08-17T17:02:12.529 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-719.gced649f8.el9.noarch 13/13 2024-08-17T17:02:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-rook-2:18.2.4-719.gced649f8.el9.noarch 1/13 2024-08-17T17:02:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-17T17:02:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-17T17:02:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-17T17:02:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-17T17:02:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-17T17:02:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-17T17:02:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-17T17:02:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-17T17:02:12.530 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-google-auth-1:2.33.0-1.el9.noarch 10/13 2024-08-17T17:02:12.531 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-17T17:02:12.531 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-17T17:02:12.841 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-17T17:02:12.842 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:12.842 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-08-17T17:02:12.842 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-rook-2:18.2.4-719.gced649f8.el9.noarch 2024-08-17T17:02:12.842 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-17T17:02:12.842 INFO:teuthology.orchestra.run.smithi165.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-17T17:02:12.842 INFO:teuthology.orchestra.run.smithi165.stdout: python3-google-auth-1:2.33.0-1.el9.noarch 2024-08-17T17:02:12.842 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-17T17:02:12.842 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-17T17:02:12.842 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-17T17:02:12.842 INFO:teuthology.orchestra.run.smithi165.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-17T17:02:12.842 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-17T17:02:12.843 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-17T17:02:12.843 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-17T17:02:12.843 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-17T17:02:12.843 INFO:teuthology.orchestra.run.smithi165.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-17T17:02:12.843 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:12.843 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:02:13.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr-cephadm 2024-08-17T17:02:13.641 INFO:teuthology.orchestra.run.smithi136.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.0 MB/s | 6.0 MB 00:01 2024-08-17T17:02:13.643 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:02:13.643 INFO:teuthology.orchestra.run.smithi136.stdout:Total 3.4 MB/s | 7.0 MB 00:02 2024-08-17T17:02:13.664 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:18 ago on Sat 17 Aug 2024 05:00:55 PM UTC. 2024-08-17T17:02:13.712 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-17T17:02:13.723 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-17T17:02:13.723 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-17T17:02:13.792 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:02:13.794 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:02:13.794 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-08-17T17:02:13.794 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:02:13.794 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-08-17T17:02:13.794 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-cephadm noarch 2:18.2.4-719.gced649f8.el9 ceph-noarch 139 k 2024-08-17T17:02:13.795 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-08-17T17:02:13.795 INFO:teuthology.orchestra.run.smithi165.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-17T17:02:13.795 INFO:teuthology.orchestra.run.smithi165.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-17T17:02:13.795 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-17T17:02:13.795 INFO:teuthology.orchestra.run.smithi165.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-17T17:02:13.795 INFO:teuthology.orchestra.run.smithi165.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-17T17:02:13.795 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:13.795 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-08-17T17:02:13.795 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:02:13.795 INFO:teuthology.orchestra.run.smithi165.stdout:Install 6 Packages 2024-08-17T17:02:13.795 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:13.795 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 7.0 M 2024-08-17T17:02:13.795 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 33 M 2024-08-17T17:02:13.795 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-08-17T17:02:13.819 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-17T17:02:13.820 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-17T17:02:14.119 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-17T17:02:14.297 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-17T17:02:14.351 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-17T17:02:14.426 INFO:teuthology.orchestra.run.smithi165.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-08-17T17:02:14.510 INFO:teuthology.orchestra.run.smithi165.stdout:(2/6): ceph-mgr-cephadm-18.2.4-719.gced649f8.el 622 kB/s | 139 kB 00:00 2024-08-17T17:02:14.560 INFO:teuthology.orchestra.run.smithi165.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.0 MB/s | 548 kB 00:00 2024-08-17T17:02:14.585 INFO:teuthology.orchestra.run.smithi165.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 3.2 MB/s | 74 kB 00:00 2024-08-17T17:02:14.602 INFO:teuthology.orchestra.run.smithi165.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 625 kB/s | 58 kB 00:00 2024-08-17T17:02:14.861 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-17T17:02:14.953 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-17T17:02:15.016 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-17T17:02:15.047 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-719.gced649f8.el9.noarch 6/6 2024-08-17T17:02:15.836 INFO:teuthology.orchestra.run.smithi165.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.9 MB/s | 6.0 MB 00:01 2024-08-17T17:02:15.838 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:02:15.838 INFO:teuthology.orchestra.run.smithi165.stdout:Total 3.4 MB/s | 7.0 MB 00:02 2024-08-17T17:02:15.859 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-719.gced649f8.el9.noarch 6/6 2024-08-17T17:02:15.859 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-719.gced649f8.el9.noarch 1/6 2024-08-17T17:02:15.859 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-17T17:02:15.859 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-17T17:02:15.859 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-17T17:02:15.859 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-17T17:02:15.909 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-08-17T17:02:15.920 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-08-17T17:02:15.920 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-08-17T17:02:16.018 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-08-17T17:02:16.020 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-08-17T17:02:16.130 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-17T17:02:16.130 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:16.130 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-17T17:02:16.130 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm-2:18.2.4-719.gced649f8.el9.noarch 2024-08-17T17:02:16.130 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-17T17:02:16.130 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-17T17:02:16.130 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-17T17:02:16.130 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-17T17:02:16.130 INFO:teuthology.orchestra.run.smithi136.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-17T17:02:16.130 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:16.130 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:02:16.331 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-08-17T17:02:16.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-fuse 2024-08-17T17:02:16.500 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-17T17:02:16.555 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-17T17:02:16.964 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:23 ago on Sat 17 Aug 2024 05:00:53 PM UTC. 2024-08-17T17:02:17.069 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-17T17:02:17.094 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:02:17.094 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:02:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-08-17T17:02:17.095 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:02:17.095 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-17T17:02:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse x86_64 2:18.2.4-719.gced649f8.el9 ceph 852 k 2024-08-17T17:02:17.095 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-08-17T17:02:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-17T17:02:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:17.095 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-17T17:02:17.095 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:02:17.095 INFO:teuthology.orchestra.run.smithi136.stdout:Install 2 Packages 2024-08-17T17:02:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:17.096 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 931 k 2024-08-17T17:02:17.096 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 2.7 M 2024-08-17T17:02:17.096 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-17T17:02:17.166 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-17T17:02:17.228 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-17T17:02:17.250 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-719.gced649f8.el9.noarch 6/6 2024-08-17T17:02:17.388 INFO:teuthology.orchestra.run.smithi136.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 475 kB/s | 80 kB 00:00 2024-08-17T17:02:17.487 INFO:teuthology.orchestra.run.smithi136.stdout:(2/2): ceph-fuse-18.2.4-719.gced649f8.el9.x86_6 3.1 MB/s | 852 kB 00:00 2024-08-17T17:02:17.487 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:02:17.487 INFO:teuthology.orchestra.run.smithi136.stdout:Total 2.3 MB/s | 931 kB 00:00 2024-08-17T17:02:17.499 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-17T17:02:17.520 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-17T17:02:17.520 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-17T17:02:17.599 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-17T17:02:17.600 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-17T17:02:17.785 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-17T17:02:17.910 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-17T17:02:17.977 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-fuse-2:18.2.4-719.gced649f8.el9.x86_64 2/2 2024-08-17T17:02:18.025 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-719.gced649f8.el9.noarch 6/6 2024-08-17T17:02:18.025 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-719.gced649f8.el9.noarch 1/6 2024-08-17T17:02:18.025 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-17T17:02:18.025 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-17T17:02:18.025 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-17T17:02:18.025 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-17T17:02:18.279 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-17T17:02:18.279 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:18.280 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-08-17T17:02:18.280 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-cephadm-2:18.2.4-719.gced649f8.el9.noarch 2024-08-17T17:02:18.280 INFO:teuthology.orchestra.run.smithi165.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-17T17:02:18.280 INFO:teuthology.orchestra.run.smithi165.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-17T17:02:18.280 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-17T17:02:18.280 INFO:teuthology.orchestra.run.smithi165.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-17T17:02:18.280 INFO:teuthology.orchestra.run.smithi165.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-17T17:02:18.280 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:18.281 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:02:18.534 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-fuse 2024-08-17T17:02:18.609 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-fuse-2:18.2.4-719.gced649f8.el9.x86_64 2/2 2024-08-17T17:02:18.610 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-fuse-2:18.2.4-719.gced649f8.el9.x86_64 1/2 2024-08-17T17:02:18.866 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-17T17:02:18.866 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:18.866 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-17T17:02:18.867 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse-2:18.2.4-719.gced649f8.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-17T17:02:18.867 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:18.867 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:02:19.088 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:24 ago on Sat 17 Aug 2024 05:00:55 PM UTC. 2024-08-17T17:02:19.135 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librados-devel 2024-08-17T17:02:19.216 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:02:19.216 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:02:19.216 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-08-17T17:02:19.216 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:02:19.216 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-08-17T17:02:19.216 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-fuse x86_64 2:18.2.4-719.gced649f8.el9 ceph 852 k 2024-08-17T17:02:19.217 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-08-17T17:02:19.217 INFO:teuthology.orchestra.run.smithi165.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-17T17:02:19.217 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:19.217 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-08-17T17:02:19.217 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:02:19.217 INFO:teuthology.orchestra.run.smithi165.stdout:Install 2 Packages 2024-08-17T17:02:19.217 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:19.217 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 931 k 2024-08-17T17:02:19.218 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 2.7 M 2024-08-17T17:02:19.218 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-08-17T17:02:19.456 INFO:teuthology.orchestra.run.smithi165.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 450 kB/s | 80 kB 00:00 2024-08-17T17:02:19.598 INFO:teuthology.orchestra.run.smithi165.stdout:(2/2): ceph-fuse-18.2.4-719.gced649f8.el9.x86_6 2.6 MB/s | 852 kB 00:00 2024-08-17T17:02:19.599 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:02:19.599 INFO:teuthology.orchestra.run.smithi165.stdout:Total 2.4 MB/s | 931 kB 00:00 2024-08-17T17:02:19.610 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-08-17T17:02:19.631 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-08-17T17:02:19.631 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-08-17T17:02:19.709 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:26 ago on Sat 17 Aug 2024 05:00:53 PM UTC. 2024-08-17T17:02:19.710 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-08-17T17:02:19.710 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-08-17T17:02:19.846 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:02:19.847 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:02:19.847 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-08-17T17:02:19.847 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:02:19.847 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-17T17:02:19.847 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel x86_64 2:18.2.4-719.gced649f8.el9 ceph 127 k 2024-08-17T17:02:19.847 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:19.847 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-17T17:02:19.847 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:02:19.847 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-08-17T17:02:19.848 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:19.848 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 127 k 2024-08-17T17:02:19.848 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 456 k 2024-08-17T17:02:19.848 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-17T17:02:19.892 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-08-17T17:02:20.038 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-17T17:02:20.072 INFO:teuthology.orchestra.run.smithi136.stdout:librados-devel-18.2.4-719.gced649f8.el9.x86_64. 565 kB/s | 127 kB 00:00 2024-08-17T17:02:20.072 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:02:20.073 INFO:teuthology.orchestra.run.smithi136.stdout:Total 561 kB/s | 127 kB 00:00 2024-08-17T17:02:20.073 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-17T17:02:20.079 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-17T17:02:20.079 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-17T17:02:20.105 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-fuse-2:18.2.4-719.gced649f8.el9.x86_64 2/2 2024-08-17T17:02:20.143 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-17T17:02:20.143 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-17T17:02:20.294 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-17T17:02:20.403 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librados-devel-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:20.760 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-fuse-2:18.2.4-719.gced649f8.el9.x86_64 2/2 2024-08-17T17:02:20.760 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-fuse-2:18.2.4-719.gced649f8.el9.x86_64 1/2 2024-08-17T17:02:20.846 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados-devel-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:21.024 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-17T17:02:21.024 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:21.024 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-08-17T17:02:21.024 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-fuse-2:18.2.4-719.gced649f8.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-17T17:02:21.024 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:21.024 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:02:21.092 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados-devel-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:21.092 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:21.092 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-17T17:02:21.092 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:02:21.092 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:21.092 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:02:21.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install librados-devel 2024-08-17T17:02:21.343 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install libcephfs2 2024-08-17T17:02:21.863 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:26 ago on Sat 17 Aug 2024 05:00:55 PM UTC. 2024-08-17T17:02:21.905 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:28 ago on Sat 17 Aug 2024 05:00:53 PM UTC. 2024-08-17T17:02:21.993 INFO:teuthology.orchestra.run.smithi136.stdout:Package libcephfs2-2:18.2.4-719.gced649f8.el9.x86_64 is already installed. 2024-08-17T17:02:21.994 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:02:21.995 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:02:21.995 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repo Size 2024-08-17T17:02:21.995 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:02:21.995 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-08-17T17:02:21.995 INFO:teuthology.orchestra.run.smithi165.stdout: librados-devel x86_64 2:18.2.4-719.gced649f8.el9 ceph 127 k 2024-08-17T17:02:21.995 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:21.995 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-08-17T17:02:21.995 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:02:21.995 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-08-17T17:02:21.996 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:21.996 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 127 k 2024-08-17T17:02:21.996 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 456 k 2024-08-17T17:02:21.996 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-08-17T17:02:22.037 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:02:22.038 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-08-17T17:02:22.038 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:02:22.101 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install libcephfs-devel 2024-08-17T17:02:22.226 INFO:teuthology.orchestra.run.smithi165.stdout:librados-devel-18.2.4-719.gced649f8.el9.x86_64. 553 kB/s | 127 kB 00:00 2024-08-17T17:02:22.226 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:02:22.227 INFO:teuthology.orchestra.run.smithi165.stdout:Total 548 kB/s | 127 kB 00:00 2024-08-17T17:02:22.227 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-08-17T17:02:22.232 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-08-17T17:02:22.232 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-08-17T17:02:22.297 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-08-17T17:02:22.297 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-08-17T17:02:22.448 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-08-17T17:02:22.573 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librados-devel-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:22.659 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:29 ago on Sat 17 Aug 2024 05:00:53 PM UTC. 2024-08-17T17:02:22.785 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:02:22.786 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:02:22.786 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-08-17T17:02:22.786 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:02:22.786 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-17T17:02:22.787 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel x86_64 2:18.2.4-719.gced649f8.el9 ceph 31 k 2024-08-17T17:02:22.787 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:22.787 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-17T17:02:22.787 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:02:22.787 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-08-17T17:02:22.787 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:22.787 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 31 k 2024-08-17T17:02:22.787 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 139 k 2024-08-17T17:02:22.787 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-17T17:02:22.955 INFO:teuthology.orchestra.run.smithi136.stdout:libcephfs-devel-18.2.4-719.gced649f8.el9.x86_64 185 kB/s | 31 kB 00:00 2024-08-17T17:02:22.956 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:02:22.956 INFO:teuthology.orchestra.run.smithi136.stdout:Total 184 kB/s | 31 kB 00:00 2024-08-17T17:02:22.956 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-17T17:02:22.959 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-17T17:02:22.959 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-17T17:02:22.975 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-17T17:02:22.975 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-17T17:02:22.979 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librados-devel-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:23.058 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-17T17:02:23.151 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs-devel-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:23.238 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados-devel-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:23.238 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:23.238 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-08-17T17:02:23.238 INFO:teuthology.orchestra.run.smithi165.stdout: librados-devel-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:02:23.238 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:23.239 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:02:23.505 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install libcephfs2 2024-08-17T17:02:23.518 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:23.745 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs-devel-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:23.745 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:23.745 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-17T17:02:23.745 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:02:23.745 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:23.745 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:02:23.919 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librados2 2024-08-17T17:02:24.060 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:29 ago on Sat 17 Aug 2024 05:00:55 PM UTC. 2024-08-17T17:02:24.146 INFO:teuthology.orchestra.run.smithi165.stdout:Package libcephfs2-2:18.2.4-719.gced649f8.el9.x86_64 is already installed. 2024-08-17T17:02:24.192 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:02:24.193 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-08-17T17:02:24.193 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:02:24.264 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install libcephfs-devel 2024-08-17T17:02:24.477 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:31 ago on Sat 17 Aug 2024 05:00:53 PM UTC. 2024-08-17T17:02:24.563 INFO:teuthology.orchestra.run.smithi136.stdout:Package librados2-2:18.2.4-719.gced649f8.el9.x86_64 is already installed. 2024-08-17T17:02:24.605 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:02:24.606 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-08-17T17:02:24.606 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:02:24.685 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librbd1 2024-08-17T17:02:24.818 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:29 ago on Sat 17 Aug 2024 05:00:55 PM UTC. 2024-08-17T17:02:24.949 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:02:24.950 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:02:24.950 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repo Size 2024-08-17T17:02:24.950 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:02:24.950 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-08-17T17:02:24.950 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs-devel x86_64 2:18.2.4-719.gced649f8.el9 ceph 31 k 2024-08-17T17:02:24.950 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:24.950 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-08-17T17:02:24.951 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:02:24.951 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-08-17T17:02:24.951 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:24.951 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 31 k 2024-08-17T17:02:24.951 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 139 k 2024-08-17T17:02:24.951 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-08-17T17:02:25.117 INFO:teuthology.orchestra.run.smithi165.stdout:libcephfs-devel-18.2.4-719.gced649f8.el9.x86_64 185 kB/s | 31 kB 00:00 2024-08-17T17:02:25.118 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:02:25.118 INFO:teuthology.orchestra.run.smithi165.stdout:Total 183 kB/s | 31 kB 00:00 2024-08-17T17:02:25.118 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-08-17T17:02:25.121 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-08-17T17:02:25.121 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-08-17T17:02:25.145 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-08-17T17:02:25.146 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-08-17T17:02:25.230 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-08-17T17:02:25.240 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:32 ago on Sat 17 Aug 2024 05:00:53 PM UTC. 2024-08-17T17:02:25.326 INFO:teuthology.orchestra.run.smithi136.stdout:Package librbd1-2:18.2.4-719.gced649f8.el9.x86_64 is already installed. 2024-08-17T17:02:25.331 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephfs-devel-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:25.369 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:02:25.370 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-08-17T17:02:25.370 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:02:25.432 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rados 2024-08-17T17:02:25.721 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:25.929 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephfs-devel-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:25.929 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:25.929 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-08-17T17:02:25.929 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs-devel-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:02:25.929 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:25.929 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:02:25.980 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:32 ago on Sat 17 Aug 2024 05:00:53 PM UTC. 2024-08-17T17:02:26.065 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-rados-2:18.2.4-719.gced649f8.el9.x86_64 is already installed. 2024-08-17T17:02:26.098 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install librados2 2024-08-17T17:02:26.109 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:02:26.111 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-08-17T17:02:26.111 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:02:26.177 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rgw 2024-08-17T17:02:26.655 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:31 ago on Sat 17 Aug 2024 05:00:55 PM UTC. 2024-08-17T17:02:26.721 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:33 ago on Sat 17 Aug 2024 05:00:53 PM UTC. 2024-08-17T17:02:26.741 INFO:teuthology.orchestra.run.smithi165.stdout:Package librados2-2:18.2.4-719.gced649f8.el9.x86_64 is already installed. 2024-08-17T17:02:26.783 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:02:26.784 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-08-17T17:02:26.784 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:02:26.808 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-rgw-2:18.2.4-719.gced649f8.el9.x86_64 is already installed. 2024-08-17T17:02:26.850 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install librbd1 2024-08-17T17:02:26.851 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:02:26.852 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-08-17T17:02:26.852 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:02:26.918 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-cephfs 2024-08-17T17:02:27.406 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:32 ago on Sat 17 Aug 2024 05:00:55 PM UTC. 2024-08-17T17:02:27.470 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:34 ago on Sat 17 Aug 2024 05:00:53 PM UTC. 2024-08-17T17:02:27.493 INFO:teuthology.orchestra.run.smithi165.stdout:Package librbd1-2:18.2.4-719.gced649f8.el9.x86_64 is already installed. 2024-08-17T17:02:27.535 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:02:27.536 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-08-17T17:02:27.536 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:02:27.554 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-cephfs-2:18.2.4-719.gced649f8.el9.x86_64 is already installed. 2024-08-17T17:02:27.597 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:02:27.598 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-08-17T17:02:27.598 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:02:27.605 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install python3-rados 2024-08-17T17:02:27.656 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rbd 2024-08-17T17:02:28.152 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:33 ago on Sat 17 Aug 2024 05:00:55 PM UTC. 2024-08-17T17:02:28.203 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:35 ago on Sat 17 Aug 2024 05:00:53 PM UTC. 2024-08-17T17:02:28.238 INFO:teuthology.orchestra.run.smithi165.stdout:Package python3-rados-2:18.2.4-719.gced649f8.el9.x86_64 is already installed. 2024-08-17T17:02:28.281 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:02:28.282 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-08-17T17:02:28.282 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:02:28.289 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-rbd-2:18.2.4-719.gced649f8.el9.x86_64 is already installed. 2024-08-17T17:02:28.331 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:02:28.332 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-08-17T17:02:28.333 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:02:28.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install python3-rgw 2024-08-17T17:02:28.406 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-fuse 2024-08-17T17:02:28.896 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:33 ago on Sat 17 Aug 2024 05:00:55 PM UTC. 2024-08-17T17:02:28.958 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:35 ago on Sat 17 Aug 2024 05:00:53 PM UTC. 2024-08-17T17:02:28.980 INFO:teuthology.orchestra.run.smithi165.stdout:Package python3-rgw-2:18.2.4-719.gced649f8.el9.x86_64 is already installed. 2024-08-17T17:02:29.024 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:02:29.025 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-08-17T17:02:29.025 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:02:29.086 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:02:29.086 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:02:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-08-17T17:02:29.087 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:02:29.087 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-17T17:02:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse x86_64 2:18.2.4-719.gced649f8.el9 ceph 87 k 2024-08-17T17:02:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:29.087 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-17T17:02:29.087 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:02:29.087 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-08-17T17:02:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:29.088 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 87 k 2024-08-17T17:02:29.088 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 227 k 2024-08-17T17:02:29.088 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-17T17:02:29.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install python3-cephfs 2024-08-17T17:02:29.308 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-fuse-18.2.4-719.gced649f8.el9.x86_64.rpm 394 kB/s | 87 kB 00:00 2024-08-17T17:02:29.309 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:02:29.309 INFO:teuthology.orchestra.run.smithi136.stdout:Total 391 kB/s | 87 kB 00:00 2024-08-17T17:02:29.309 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-17T17:02:29.316 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-17T17:02:29.316 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-17T17:02:29.378 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-17T17:02:29.378 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-17T17:02:29.519 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-17T17:02:29.621 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-fuse-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:29.640 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:34 ago on Sat 17 Aug 2024 05:00:55 PM UTC. 2024-08-17T17:02:29.727 INFO:teuthology.orchestra.run.smithi165.stdout:Package python3-cephfs-2:18.2.4-719.gced649f8.el9.x86_64 is already installed. 2024-08-17T17:02:29.769 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:02:29.770 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-08-17T17:02:29.770 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:02:29.832 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install python3-rbd 2024-08-17T17:02:29.991 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-fuse-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:30.257 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-fuse-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:30.257 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:30.257 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-17T17:02:30.257 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:02:30.257 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:30.258 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:02:30.385 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:35 ago on Sat 17 Aug 2024 05:00:55 PM UTC. 2024-08-17T17:02:30.472 INFO:teuthology.orchestra.run.smithi165.stdout:Package python3-rbd-2:18.2.4-719.gced649f8.el9.x86_64 is already installed. 2024-08-17T17:02:30.514 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:02:30.515 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-08-17T17:02:30.515 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:02:30.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-mirror 2024-08-17T17:02:30.582 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install rbd-fuse 2024-08-17T17:02:31.117 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:38 ago on Sat 17 Aug 2024 05:00:53 PM UTC. 2024-08-17T17:02:31.143 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:36 ago on Sat 17 Aug 2024 05:00:55 PM UTC. 2024-08-17T17:02:31.249 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:02:31.250 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:02:31.250 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-08-17T17:02:31.250 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:02:31.250 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-17T17:02:31.250 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror x86_64 2:18.2.4-719.gced649f8.el9 ceph 3.0 M 2024-08-17T17:02:31.250 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:31.250 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-17T17:02:31.250 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:02:31.251 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-08-17T17:02:31.251 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:31.251 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 3.0 M 2024-08-17T17:02:31.251 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 12 M 2024-08-17T17:02:31.251 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-17T17:02:31.272 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:02:31.272 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:02:31.273 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-08-17T17:02:31.273 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:02:31.273 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-08-17T17:02:31.273 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-fuse x86_64 2:18.2.4-719.gced649f8.el9 ceph 87 k 2024-08-17T17:02:31.273 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:31.273 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-08-17T17:02:31.273 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:02:31.273 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-08-17T17:02:31.273 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:31.274 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 87 k 2024-08-17T17:02:31.274 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 227 k 2024-08-17T17:02:31.274 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-08-17T17:02:31.487 INFO:teuthology.orchestra.run.smithi165.stdout:rbd-fuse-18.2.4-719.gced649f8.el9.x86_64.rpm 407 kB/s | 87 kB 00:00 2024-08-17T17:02:31.487 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:02:31.487 INFO:teuthology.orchestra.run.smithi165.stdout:Total 404 kB/s | 87 kB 00:00 2024-08-17T17:02:31.488 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-08-17T17:02:31.494 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-08-17T17:02:31.495 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-08-17T17:02:31.557 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-08-17T17:02:31.558 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-08-17T17:02:31.603 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-mirror-18.2.4-719.gced649f8.el9.x86_64.rpm 8.5 MB/s | 3.0 MB 00:00 2024-08-17T17:02:31.604 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:02:31.604 INFO:teuthology.orchestra.run.smithi136.stdout:Total 8.4 MB/s | 3.0 MB 00:00 2024-08-17T17:02:31.604 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-17T17:02:31.613 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-17T17:02:31.613 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-17T17:02:31.694 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-17T17:02:31.694 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-17T17:02:31.699 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-08-17T17:02:31.801 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-fuse-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:32.113 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-17T17:02:32.179 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-mirror-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:32.204 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-mirror-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:32.204 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T17:02:32.204 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-17T17:02:32.204 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-17T17:02:32.204 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-17T17:02:32.204 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:32.229 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: rbd-fuse-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:32.479 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-fuse-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:32.480 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:32.480 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-08-17T17:02:32.480 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-fuse-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:02:32.480 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:32.480 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:02:32.698 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install rbd-mirror 2024-08-17T17:02:33.262 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:38 ago on Sat 17 Aug 2024 05:00:55 PM UTC. 2024-08-17T17:02:33.294 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-mirror-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:33.294 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:33.294 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-17T17:02:33.294 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:02:33.294 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:33.294 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:02:33.391 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:02:33.392 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:02:33.392 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repo Size 2024-08-17T17:02:33.392 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:02:33.392 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-08-17T17:02:33.392 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-mirror x86_64 2:18.2.4-719.gced649f8.el9 ceph 3.0 M 2024-08-17T17:02:33.392 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:33.392 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-08-17T17:02:33.392 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:02:33.392 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-08-17T17:02:33.392 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:33.393 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 3.0 M 2024-08-17T17:02:33.393 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 12 M 2024-08-17T17:02:33.393 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-08-17T17:02:33.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-nbd 2024-08-17T17:02:33.741 INFO:teuthology.orchestra.run.smithi165.stdout:rbd-mirror-18.2.4-719.gced649f8.el9.x86_64.rpm 8.6 MB/s | 3.0 MB 00:00 2024-08-17T17:02:33.742 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:02:33.742 INFO:teuthology.orchestra.run.smithi165.stdout:Total 8.5 MB/s | 3.0 MB 00:00 2024-08-17T17:02:33.742 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-08-17T17:02:33.751 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-08-17T17:02:33.751 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-08-17T17:02:33.832 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-08-17T17:02:33.832 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-08-17T17:02:34.109 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:41 ago on Sat 17 Aug 2024 05:00:53 PM UTC. 2024-08-17T17:02:34.238 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-17T17:02:34.238 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:02:34.238 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-08-17T17:02:34.238 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:02:34.239 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-17T17:02:34.239 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd x86_64 2:18.2.4-719.gced649f8.el9 ceph 172 k 2024-08-17T17:02:34.239 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:34.239 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-17T17:02:34.239 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-17T17:02:34.239 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-08-17T17:02:34.239 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:34.239 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 172 k 2024-08-17T17:02:34.240 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 494 k 2024-08-17T17:02:34.240 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-17T17:02:34.250 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-08-17T17:02:34.309 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-mirror-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:34.334 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: rbd-mirror-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:34.334 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T17:02:34.334 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-17T17:02:34.334 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-17T17:02:34.334 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-17T17:02:34.334 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:34.444 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-nbd-18.2.4-719.gced649f8.el9.x86_64.rpm 845 kB/s | 172 kB 00:00 2024-08-17T17:02:34.444 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:02:34.445 INFO:teuthology.orchestra.run.smithi136.stdout:Total 838 kB/s | 172 kB 00:00 2024-08-17T17:02:34.445 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-17T17:02:34.453 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-17T17:02:34.453 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-17T17:02:34.515 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-17T17:02:34.516 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-17T17:02:34.670 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-17T17:02:34.765 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-nbd-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:35.155 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-nbd-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:35.399 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-nbd-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:35.399 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:35.400 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-17T17:02:35.400 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:02:35.400 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:02:35.400 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-17T17:02:35.453 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-mirror-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:35.453 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:35.453 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-08-17T17:02:35.453 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-mirror-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:02:35.453 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:35.453 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:02:35.681 DEBUG:teuthology.parallel:result is None 2024-08-17T17:02:35.708 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install rbd-nbd 2024-08-17T17:02:36.276 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:41 ago on Sat 17 Aug 2024 05:00:55 PM UTC. 2024-08-17T17:02:36.410 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-08-17T17:02:36.411 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:02:36.411 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-08-17T17:02:36.411 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:02:36.411 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-08-17T17:02:36.411 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-nbd x86_64 2:18.2.4-719.gced649f8.el9 ceph 172 k 2024-08-17T17:02:36.411 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:36.411 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-08-17T17:02:36.411 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-08-17T17:02:36.411 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-08-17T17:02:36.411 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:36.411 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 172 k 2024-08-17T17:02:36.411 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 494 k 2024-08-17T17:02:36.412 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-08-17T17:02:36.641 INFO:teuthology.orchestra.run.smithi165.stdout:rbd-nbd-18.2.4-719.gced649f8.el9.x86_64.rpm 750 kB/s | 172 kB 00:00 2024-08-17T17:02:36.642 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:02:36.642 INFO:teuthology.orchestra.run.smithi165.stdout:Total 745 kB/s | 172 kB 00:00 2024-08-17T17:02:36.642 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-08-17T17:02:36.650 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-08-17T17:02:36.650 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-08-17T17:02:36.719 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-08-17T17:02:36.720 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-08-17T17:02:36.881 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-08-17T17:02:36.992 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-nbd-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:37.473 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: rbd-nbd-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:37.719 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-nbd-2:18.2.4-719.gced649f8.el9.x86_64 1/1 2024-08-17T17:02:37.719 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:37.720 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-08-17T17:02:37.720 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-nbd-2:18.2.4-719.gced649f8.el9.x86_64 2024-08-17T17:02:37.720 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:02:37.720 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-08-17T17:02:37.988 DEBUG:teuthology.parallel:result is None 2024-08-17T17:02:37.989 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-17T17:02:37.989 INFO:teuthology.packaging:ref: None 2024-08-17T17:02:37.989 INFO:teuthology.packaging:tag: None 2024-08-17T17:02:37.989 INFO:teuthology.packaging:branch: reef 2024-08-17T17:02:37.989 INFO:teuthology.packaging:sha1: cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:02:37.990 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-08-17T17:02:38.331 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-17T17:02:38.357 INFO:teuthology.orchestra.run.smithi136.stdout:18.2.4-719.gced649f8.el9 2024-08-17T17:02:38.358 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-719.gced649f8.el9 2024-08-17T17:02:38.358 INFO:teuthology.task.install:The correct ceph version 18.2.4-719.gced649f8 is installed. 2024-08-17T17:02:38.360 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-17T17:02:38.360 INFO:teuthology.packaging:ref: None 2024-08-17T17:02:38.360 INFO:teuthology.packaging:tag: None 2024-08-17T17:02:38.360 INFO:teuthology.packaging:branch: reef 2024-08-17T17:02:38.360 INFO:teuthology.packaging:sha1: cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:02:38.360 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-08-17T17:02:38.633 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-17T17:02:38.658 INFO:teuthology.orchestra.run.smithi165.stdout:18.2.4-719.gced649f8.el9 2024-08-17T17:02:38.658 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-719.gced649f8.el9 2024-08-17T17:02:38.659 INFO:teuthology.task.install:The correct ceph version 18.2.4-719.gced649f8 is installed. 2024-08-17T17:02:38.661 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-17T17:02:38.661 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-17T17:02:38.661 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-17T17:02:38.691 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-08-17T17:02:38.692 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-17T17:02:38.730 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-17T17:02:38.730 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-17T17:02:38.730 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/daemon-helper 2024-08-17T17:02:38.761 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-17T17:02:38.828 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-08-17T17:02:38.828 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/daemon-helper 2024-08-17T17:02:38.855 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-17T17:02:38.921 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-17T17:02:38.922 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-17T17:02:38.922 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-17T17:02:38.949 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-17T17:02:39.015 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-08-17T17:02:39.016 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-17T17:02:39.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-17T17:02:39.107 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-17T17:02:39.108 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-17T17:02:39.108 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/stdin-killer 2024-08-17T17:02:39.133 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-17T17:02:39.199 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-08-17T17:02:39.200 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/stdin-killer 2024-08-17T17:02:39.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-17T17:02:39.291 INFO:teuthology.run_tasks:Running task cephadm... 2024-08-17T17:02:39.383 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}}, 'image': 'quay.ceph.io/ceph-ci/ceph:reef', 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'mons down', 'mon down', 'MON_DOWN', 'out of quorum', 'PG_AVAILABILITY'], 'sha1': 'cd8bdbeb9fbdab5733aa3aaadc042c965473e453'} 2024-08-17T17:02:39.383 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:reef 2024-08-17T17:02:39.384 INFO:tasks.cephadm:Cluster fsid is 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:02:39.384 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-08-17T17:02:39.384 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.136', 'mon.c': '[v2:172.21.15.136:3301,v1:172.21.15.136:6790]', 'mon.b': '172.21.15.165'} 2024-08-17T17:02:39.384 INFO:tasks.cephadm:First mon is mon.a on smithi136 2024-08-17T17:02:39.384 INFO:tasks.cephadm:First mgr is y 2024-08-17T17:02:39.385 INFO:tasks.cephadm:Normalizing hostnames... 2024-08-17T17:02:39.385 DEBUG:teuthology.orchestra.run.smithi136:> sudo hostname $(hostname -s) 2024-08-17T17:02:39.410 DEBUG:teuthology.orchestra.run.smithi165:> sudo hostname $(hostname -s) 2024-08-17T17:02:39.436 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-08-17T17:02:39.436 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:02:39.582 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'cd8bdbeb9fbdab5733aa3aaadc042c965473e453', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/55847/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-5836-gcd8bdbeb', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-5836.gcd8bdbeb'}, 'url': 'https://2.chacra.ceph.com/r/ceph/main/cd8bdbeb9fbdab5733aa3aaadc042c965473e453/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-08-15 21:47:48.216668', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'main', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/main/cd8bdbeb9fbdab5733aa3aaadc042c965473e453/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-08-17T17:02:39.818 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref reef, sha1 ced649f89259a8c5733cc18a50f42f20f41ba352 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-08-17T17:02:39.820 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/reef/ced649f89259a8c5733cc18a50f42f20f41ba352/centos/9/x86_64/flavors/default/cephadm 2024-08-17T17:02:39.820 INFO:tasks.cephadm:Downloading cephadm from url: https://3.chacra.ceph.com/binaries/ceph/reef/ced649f89259a8c5733cc18a50f42f20f41ba352/centos/9/x86_64/flavors/default/cephadm 2024-08-17T17:02:39.820 DEBUG:teuthology.orchestra.run.smithi136:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/reef/ced649f89259a8c5733cc18a50f42f20f41ba352/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-08-17T17:02:40.164 INFO:teuthology.orchestra.run.smithi136.stdout:-rw-r--r--. 1 ubuntu ubuntu 215594 Aug 17 17:02 /home/ubuntu/cephtest/cephadm 2024-08-17T17:02:40.164 DEBUG:teuthology.orchestra.run.smithi165:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/reef/ced649f89259a8c5733cc18a50f42f20f41ba352/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-08-17T17:02:40.550 INFO:teuthology.orchestra.run.smithi165.stdout:-rw-r--r--. 1 ubuntu ubuntu 215594 Aug 17 17:02 /home/ubuntu/cephtest/cephadm 2024-08-17T17:02:40.550 DEBUG:teuthology.orchestra.run.smithi136:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-08-17T17:02:40.567 DEBUG:teuthology.orchestra.run.smithi165:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-08-17T17:02:40.597 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:reef on all hosts... 2024-08-17T17:02:40.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-08-17T17:02:40.609 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-08-17T17:02:41.399 INFO:teuthology.orchestra.run.smithi136.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-08-17T17:02:41.640 INFO:teuthology.orchestra.run.smithi165.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-08-17T17:03:07.795 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T17:03:07.796 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_version": "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)", 2024-08-17T17:03:07.796 INFO:teuthology.orchestra.run.smithi165.stdout: "image_id": "c870753d67c2504e5ac044c83696772f8b7527a928cc764ed24eeea7fea57d6e", 2024-08-17T17:03:07.796 INFO:teuthology.orchestra.run.smithi165.stdout: "repo_digests": [ 2024-08-17T17:03:07.796 INFO:teuthology.orchestra.run.smithi165.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408" 2024-08-17T17:03:07.796 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:03:07.796 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T17:03:08.364 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:03:08.365 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_version": "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)", 2024-08-17T17:03:08.365 INFO:teuthology.orchestra.run.smithi136.stdout: "image_id": "c870753d67c2504e5ac044c83696772f8b7527a928cc764ed24eeea7fea57d6e", 2024-08-17T17:03:08.365 INFO:teuthology.orchestra.run.smithi136.stdout: "repo_digests": [ 2024-08-17T17:03:08.365 INFO:teuthology.orchestra.run.smithi136.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408" 2024-08-17T17:03:08.365 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:03:08.365 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:03:08.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /etc/ceph 2024-08-17T17:03:08.441 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /etc/ceph 2024-08-17T17:03:08.469 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 777 /etc/ceph 2024-08-17T17:03:08.493 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 777 /etc/ceph 2024-08-17T17:03:08.533 INFO:tasks.cephadm:Writing seed config... 2024-08-17T17:03:08.535 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-08-17T17:03:08.535 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-08-17T17:03:08.535 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-08-17T17:03:08.535 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-08-17T17:03:08.535 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-08-17T17:03:08.535 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-08-17T17:03:08.535 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-08-17T17:03:08.535 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-08-17T17:03:08.535 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-08-17T17:03:08.535 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-08-17T17:03:08.536 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-08-17T17:03:08.536 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-17T17:03:08.536 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-08-17T17:03:08.551 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 8270436c-5cba-11ef-bccf-c7b262605968 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-08-17T17:03:08.552 DEBUG:teuthology.orchestra.run.smithi136:mon.a> sudo journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@mon.a.service 2024-08-17T17:03:08.594 DEBUG:teuthology.orchestra.run.smithi136:mgr.y> sudo journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@mgr.y.service 2024-08-17T17:03:08.636 INFO:tasks.cephadm:Bootstrapping... 2024-08-17T17:03:08.637 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef -v bootstrap --fsid 8270436c-5cba-11ef-bccf-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.136 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-08-17T17:03:08.790 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:03:08.790 INFO:teuthology.orchestra.run.smithi136.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:reef', '-v', 'bootstrap', '--fsid', '8270436c-5cba-11ef-bccf-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.136', '--skip-admin-label'] 2024-08-17T17:03:08.815 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stdout 5.1.2 2024-08-17T17:03:08.815 INFO:teuthology.orchestra.run.smithi136.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-08-17T17:03:08.816 INFO:teuthology.orchestra.run.smithi136.stdout:Verifying podman|docker is present... 2024-08-17T17:03:08.838 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stdout 5.1.2 2024-08-17T17:03:08.839 INFO:teuthology.orchestra.run.smithi136.stdout:Verifying lvm2 is present... 2024-08-17T17:03:08.839 INFO:teuthology.orchestra.run.smithi136.stdout:Verifying time synchronization is in place... 2024-08-17T17:03:08.846 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-08-17T17:03:08.846 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-08-17T17:03:08.852 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-08-17T17:03:08.853 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout inactive 2024-08-17T17:03:08.859 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout enabled 2024-08-17T17:03:08.866 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout active 2024-08-17T17:03:08.866 INFO:teuthology.orchestra.run.smithi136.stdout:Unit chronyd.service is enabled and running 2024-08-17T17:03:08.866 INFO:teuthology.orchestra.run.smithi136.stdout:Repeating the final host check... 2024-08-17T17:03:08.889 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stdout 5.1.2 2024-08-17T17:03:08.889 INFO:teuthology.orchestra.run.smithi136.stdout:podman (/bin/podman) version 5.1.2 is present 2024-08-17T17:03:08.889 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl is present 2024-08-17T17:03:08.889 INFO:teuthology.orchestra.run.smithi136.stdout:lvcreate is present 2024-08-17T17:03:08.896 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-08-17T17:03:08.896 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-08-17T17:03:08.902 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-08-17T17:03:08.902 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout inactive 2024-08-17T17:03:08.909 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout enabled 2024-08-17T17:03:08.915 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout active 2024-08-17T17:03:08.915 INFO:teuthology.orchestra.run.smithi136.stdout:Unit chronyd.service is enabled and running 2024-08-17T17:03:08.915 INFO:teuthology.orchestra.run.smithi136.stdout:Host looks OK 2024-08-17T17:03:08.915 INFO:teuthology.orchestra.run.smithi136.stdout:Cluster fsid: 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:03:08.916 INFO:teuthology.orchestra.run.smithi136.stdout:Acquiring lock 139969808336640 on /run/cephadm/8270436c-5cba-11ef-bccf-c7b262605968.lock 2024-08-17T17:03:08.916 INFO:teuthology.orchestra.run.smithi136.stdout:Lock 139969808336640 acquired on /run/cephadm/8270436c-5cba-11ef-bccf-c7b262605968.lock 2024-08-17T17:03:08.916 INFO:teuthology.orchestra.run.smithi136.stdout:Verifying IP 172.21.15.136 port 3300 ... 2024-08-17T17:03:08.916 INFO:teuthology.orchestra.run.smithi136.stdout:Verifying IP 172.21.15.136 port 6789 ... 2024-08-17T17:03:08.916 INFO:teuthology.orchestra.run.smithi136.stdout:Base mon IP(s) is [172.21.15.136:3300, 172.21.15.136:6789], mon addrv is [v2:172.21.15.136:3300,v1:172.21.15.136:6789] 2024-08-17T17:03:08.919 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.136 metric 100 2024-08-17T17:03:08.919 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.136 metric 100 2024-08-17T17:03:08.922 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-08-17T17:03:08.922 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-08-17T17:03:08.922 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1796sec hoplimit 64 pref medium 2024-08-17T17:03:08.924 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-08-17T17:03:08.924 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-08-17T17:03:08.924 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-08-17T17:03:08.924 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-08-17T17:03:08.924 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:6e43/64 scope link 2024-08-17T17:03:08.925 INFO:teuthology.orchestra.run.smithi136.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-08-17T17:03:08.925 INFO:teuthology.orchestra.run.smithi136.stdout:Mon IP `172.21.15.136` is in CIDR network `172.21.0.0/20` 2024-08-17T17:03:08.925 INFO:teuthology.orchestra.run.smithi136.stdout:Mon IP `172.21.15.136` is in CIDR network `172.21.0.0/20` 2024-08-17T17:03:08.925 INFO:teuthology.orchestra.run.smithi136.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-08-17T17:03:08.926 INFO:teuthology.orchestra.run.smithi136.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-08-17T17:03:08.926 INFO:teuthology.orchestra.run.smithi136.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-08-17T17:03:09.418 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stdout c870753d67c2504e5ac044c83696772f8b7527a928cc764ed24eeea7fea57d6e 2024-08-17T17:03:09.418 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:reef... 2024-08-17T17:03:09.418 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stderr Getting image source signatures 2024-08-17T17:03:09.418 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-08-17T17:03:09.418 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stderr Copying blob sha256:090188efbbf262fdcde0da55307c932266150937ffe226cad16cfadb849e480e 2024-08-17T17:03:09.418 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stderr Copying config sha256:c870753d67c2504e5ac044c83696772f8b7527a928cc764ed24eeea7fea57d6e 2024-08-17T17:03:09.418 INFO:teuthology.orchestra.run.smithi136.stdout:/bin/podman: stderr Writing manifest to image destination 2024-08-17T17:03:10.372 INFO:teuthology.orchestra.run.smithi136.stdout:ceph: stdout ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable) 2024-08-17T17:03:10.372 INFO:teuthology.orchestra.run.smithi136.stdout:Ceph version: ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable) 2024-08-17T17:03:10.372 INFO:teuthology.orchestra.run.smithi136.stdout:Extracting ceph user uid/gid from container image... 2024-08-17T17:03:11.620 INFO:teuthology.orchestra.run.smithi136.stdout:stat: stdout 167 167 2024-08-17T17:03:11.620 INFO:teuthology.orchestra.run.smithi136.stdout:Creating initial keys... 2024-08-17T17:03:12.524 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph-authtool: stdout AQDP18Bm6kZQOBAAGe5gNGbLGWsa+nPwAsy4zg== 2024-08-17T17:03:13.413 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph-authtool: stdout AQDQ18BmpgeoNBAA4qnhA5K9mnu3C2UcuP8Yuw== 2024-08-17T17:03:14.393 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph-authtool: stdout AQDR18BmZGZ4MBAAEyS85B0ZNh9uzW/aOcqnbA== 2024-08-17T17:03:14.394 INFO:teuthology.orchestra.run.smithi136.stdout:Creating initial monmap... 2024-08-17T17:03:15.467 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-08-17T17:03:15.467 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-08-17T17:03:15.467 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:03:15.467 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-08-17T17:03:15.467 INFO:teuthology.orchestra.run.smithi136.stdout:monmaptool for a [v2:172.21.15.136:3300,v1:172.21.15.136:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-08-17T17:03:15.467 INFO:teuthology.orchestra.run.smithi136.stdout:setting min_mon_release = pacific 2024-08-17T17:03:15.467 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/monmaptool: set fsid to 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:03:15.467 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-08-17T17:03:15.467 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:03:15.467 INFO:teuthology.orchestra.run.smithi136.stdout:Creating mon... 2024-08-17T17:03:16.329 INFO:teuthology.orchestra.run.smithi136.stdout:create mon.a on 2024-08-17T17:03:16.539 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-08-17T17:03:16.680 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-08-17T17:03:16.848 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-8270436c-5cba-11ef-bccf-c7b262605968.target → /etc/systemd/system/ceph-8270436c-5cba-11ef-bccf-c7b262605968.target. 2024-08-17T17:03:16.848 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-8270436c-5cba-11ef-bccf-c7b262605968.target → /etc/systemd/system/ceph-8270436c-5cba-11ef-bccf-c7b262605968.target. 2024-08-17T17:03:17.062 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8270436c-5cba-11ef-bccf-c7b262605968@mon.a 2024-08-17T17:03:17.063 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8270436c-5cba-11ef-bccf-c7b262605968@mon.a.service: Unit ceph-8270436c-5cba-11ef-bccf-c7b262605968@mon.a.service not loaded. 2024-08-17T17:03:17.220 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8270436c-5cba-11ef-bccf-c7b262605968.target.wants/ceph-8270436c-5cba-11ef-bccf-c7b262605968@mon.a.service → /etc/systemd/system/ceph-8270436c-5cba-11ef-bccf-c7b262605968@.service. 2024-08-17T17:03:17.706 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:17 smithi136 ceph-mon[30564]: mon.a is new leader, mons a in quorum (ranks 0) 2024-08-17T17:03:17.719 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-17T17:03:17.719 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout disabled 2024-08-17T17:03:17.726 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-17T17:03:17.726 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout inactive 2024-08-17T17:03:17.726 INFO:teuthology.orchestra.run.smithi136.stdout:firewalld.service is not enabled 2024-08-17T17:03:17.726 INFO:teuthology.orchestra.run.smithi136.stdout:Not possible to enable service . firewalld.service is not available 2024-08-17T17:03:17.726 INFO:teuthology.orchestra.run.smithi136.stdout:Waiting for mon to start... 2024-08-17T17:03:17.726 INFO:teuthology.orchestra.run.smithi136.stdout:Waiting for mon... 2024-08-17T17:03:17.958 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:17 smithi136 bash[30526]: 34a34f0c51b8534e4df6e3019006cc13c00cb045abfd9a531f9888f1e5b02121 2024-08-17T17:03:17.958 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:17 smithi136 systemd[1]: Started Ceph mon.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:03:18.741 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout cluster: 2024-08-17T17:03:18.741 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout id: 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:03:18.741 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-08-17T17:03:18.741 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-08-17T17:03:18.742 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout services: 2024-08-17T17:03:18.742 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.538736s) 2024-08-17T17:03:18.742 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-08-17T17:03:18.742 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-08-17T17:03:18.742 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-08-17T17:03:18.742 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout data: 2024-08-17T17:03:18.742 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-08-17T17:03:18.742 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-08-17T17:03:18.742 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-08-17T17:03:18.743 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout pgs: 2024-08-17T17:03:18.743 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-08-17T17:03:18.743 INFO:teuthology.orchestra.run.smithi136.stdout:mon is available 2024-08-17T17:03:18.743 INFO:teuthology.orchestra.run.smithi136.stdout:Assimilating anything we can from ceph.conf... 2024-08-17T17:03:18.847 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:18 smithi136 ceph-mon[30564]: mon.a is new leader, mons a in quorum (ranks 0) 2024-08-17T17:03:18.847 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:18 smithi136 ceph-mon[30564]: monmap e1: 1 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-08-17T17:03:18.847 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:18 smithi136 ceph-mon[30564]: fsmap 2024-08-17T17:03:18.847 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:18 smithi136 ceph-mon[30564]: osdmap e1: 0 total, 0 up, 0 in 2024-08-17T17:03:18.847 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:18 smithi136 ceph-mon[30564]: mgrmap e1: no daemons active 2024-08-17T17:03:18.847 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:18 smithi136 ceph-mon[30564]: from='client.? 172.21.15.136:0/461382560' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-08-17T17:03:19.771 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-08-17T17:03:19.772 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout [global] 2024-08-17T17:03:19.772 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout fsid = 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:03:19.772 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.136:3300,v1:172.21.15.136:6789] 2024-08-17T17:03:19.772 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-08-17T17:03:19.772 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-08-17T17:03:19.772 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-08-17T17:03:19.772 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-08-17T17:03:19.772 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-08-17T17:03:19.772 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout [mgr] 2024-08-17T17:03:19.772 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-08-17T17:03:19.772 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-08-17T17:03:19.772 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout [osd] 2024-08-17T17:03:19.772 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-08-17T17:03:19.773 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-08-17T17:03:19.773 INFO:teuthology.orchestra.run.smithi136.stdout:Generating new minimal ceph.conf... 2024-08-17T17:03:19.885 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:19 smithi136 ceph-mon[30564]: from='client.? 172.21.15.136:0/4012314292' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-08-17T17:03:19.886 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:19 smithi136 ceph-mon[30564]: from='client.? 172.21.15.136:0/4012314292' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-08-17T17:03:20.761 INFO:teuthology.orchestra.run.smithi136.stdout:Restarting the monitor... 2024-08-17T17:03:20.782 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:20 smithi136 ceph-mon[30564]: from='client.? 172.21.15.136:0/3112869027' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:21.038 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:20 smithi136 systemd[1]: Stopping Ceph mon.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:03:21.419 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:21 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a[30541]: 2024-08-17T17:03:21.037+0000 7fba94fc5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:03:21.420 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:21 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a[30541]: 2024-08-17T17:03:21.037+0000 7fba94fc5640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-08-17T17:03:21.420 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:21 smithi136 podman[30879]: 2024-08-17 17:03:21.119080412 +0000 UTC m=+0.192828828 container died 34a34f0c51b8534e4df6e3019006cc13c00cb045abfd9a531f9888f1e5b02121 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a, org.label-schema.vendor=CentOS, RELEASE=reef-ced649f, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.build-date=20240731) 2024-08-17T17:03:21.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:21 smithi136 podman[30879]: 2024-08-17 17:03:21.526340449 +0000 UTC m=+0.600088868 container cleanup 34a34f0c51b8534e4df6e3019006cc13c00cb045abfd9a531f9888f1e5b02121 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=reef-ced649f, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-08-17T17:03:21.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:21 smithi136 bash[30879]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a 2024-08-17T17:03:22.053 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:21 smithi136 podman[30896]: 2024-08-17 17:03:21.718912937 +0000 UTC m=+0.591481539 container remove 34a34f0c51b8534e4df6e3019006cc13c00cb045abfd9a531f9888f1e5b02121 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , RELEASE=reef-ced649f, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-08-17T17:03:22.053 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:21 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@mon.a.service: Deactivated successfully. 2024-08-17T17:03:22.053 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:21 smithi136 systemd[1]: Stopped Ceph mon.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:03:22.053 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:21 smithi136 systemd[1]: Starting Ceph mon.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:03:22.332 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 podman[30995]: 2024-08-17 17:03:22.052906749 +0000 UTC m=+0.104030686 container create cc4e5e5ff75686490f40ecc7fb86f646c8220efc2d2968ba474c1a497629a5c2 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a, GIT_CLEAN=True, RELEASE=reef-ced649f, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-08-17T17:03:22.332 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 podman[30995]: 2024-08-17 17:03:21.965573007 +0000 UTC m=+0.016696953 image pull c870753d67c2504e5ac044c83696772f8b7527a928cc764ed24eeea7fea57d6e quay.ceph.io/ceph-ci/ceph:reef 2024-08-17T17:03:22.332 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 podman[30995]: 2024-08-17 17:03:22.238509319 +0000 UTC m=+0.289633259 container init cc4e5e5ff75686490f40ecc7fb86f646c8220efc2d2968ba474c1a497629a5c2 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-ced649f, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0) 2024-08-17T17:03:22.332 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 podman[30995]: 2024-08-17 17:03:22.243183666 +0000 UTC m=+0.294307603 container start cc4e5e5ff75686490f40ecc7fb86f646c8220efc2d2968ba474c1a497629a5c2 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=reef-ced649f, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-17T17:03:22.332 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: set uid:gid to 167:167 (ceph:ceph) 2024-08-17T17:03:22.332 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable), process ceph-mon, pid 7 2024-08-17T17:03:22.333 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: pidfile_write: ignore empty --pid-file 2024-08-17T17:03:22.333 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: load: jerasure load: lrc 2024-08-17T17:03:22.333 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: RocksDB version: 7.9.2 2024-08-17T17:03:22.333 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Git sha 0 2024-08-17T17:03:22.333 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Compile date 2024-08-17 15:23:20 2024-08-17T17:03:22.333 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: DB SUMMARY 2024-08-17T17:03:22.333 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: DB Session ID: MTGPD67ZE4RN1MRCHG4S 2024-08-17T17:03:22.333 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: CURRENT file: CURRENT 2024-08-17T17:03:22.333 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: IDENTITY file: IDENTITY 2024-08-17T17:03:22.333 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-08-17T17:03:22.333 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-08-17T17:03:22.333 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84572 ; 2024-08-17T17:03:22.333 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.error_if_exists: 0 2024-08-17T17:03:22.334 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.create_if_missing: 0 2024-08-17T17:03:22.334 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.paranoid_checks: 1 2024-08-17T17:03:22.334 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.flush_verify_memtable_count: 1 2024-08-17T17:03:22.338 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-08-17T17:03:22.338 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-08-17T17:03:22.338 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.env: 0x561c69cecbc0 2024-08-17T17:03:22.338 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.fs: PosixFileSystem 2024-08-17T17:03:22.338 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.info_log: 0x561c6ad62be0 2024-08-17T17:03:22.338 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_file_opening_threads: 16 2024-08-17T17:03:22.338 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.statistics: (nil) 2024-08-17T17:03:22.338 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.use_fsync: 0 2024-08-17T17:03:22.339 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_log_file_size: 0 2024-08-17T17:03:22.339 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-08-17T17:03:22.339 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.log_file_time_to_roll: 0 2024-08-17T17:03:22.339 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.keep_log_file_num: 1000 2024-08-17T17:03:22.339 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.recycle_log_file_num: 0 2024-08-17T17:03:22.339 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.allow_fallocate: 1 2024-08-17T17:03:22.339 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.allow_mmap_reads: 0 2024-08-17T17:03:22.339 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.allow_mmap_writes: 0 2024-08-17T17:03:22.339 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.use_direct_reads: 0 2024-08-17T17:03:22.339 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-08-17T17:03:22.339 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.create_missing_column_families: 0 2024-08-17T17:03:22.339 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.db_log_dir: 2024-08-17T17:03:22.339 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.wal_dir: 2024-08-17T17:03:22.339 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.table_cache_numshardbits: 6 2024-08-17T17:03:22.340 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.WAL_ttl_seconds: 0 2024-08-17T17:03:22.340 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.WAL_size_limit_MB: 0 2024-08-17T17:03:22.340 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-08-17T17:03:22.340 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-08-17T17:03:22.340 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.is_fd_close_on_exec: 1 2024-08-17T17:03:22.340 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.advise_random_on_open: 1 2024-08-17T17:03:22.340 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.db_write_buffer_size: 0 2024-08-17T17:03:22.340 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.write_buffer_manager: 0x561c6ad32140 2024-08-17T17:03:22.340 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-08-17T17:03:22.340 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-08-17T17:03:22.340 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.use_adaptive_mutex: 0 2024-08-17T17:03:22.340 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.rate_limiter: (nil) 2024-08-17T17:03:22.340 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-08-17T17:03:22.340 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.wal_recovery_mode: 2 2024-08-17T17:03:22.341 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.enable_thread_tracking: 0 2024-08-17T17:03:22.341 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.enable_pipelined_write: 0 2024-08-17T17:03:22.341 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.unordered_write: 0 2024-08-17T17:03:22.341 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-08-17T17:03:22.341 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-08-17T17:03:22.341 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-08-17T17:03:22.341 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-08-17T17:03:22.341 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.row_cache: None 2024-08-17T17:03:22.341 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.wal_filter: None 2024-08-17T17:03:22.341 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-08-17T17:03:22.341 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.allow_ingest_behind: 0 2024-08-17T17:03:22.341 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.two_write_queues: 0 2024-08-17T17:03:22.341 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.manual_wal_flush: 0 2024-08-17T17:03:22.342 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.wal_compression: 0 2024-08-17T17:03:22.342 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.atomic_flush: 0 2024-08-17T17:03:22.342 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-08-17T17:03:22.342 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.persist_stats_to_disk: 0 2024-08-17T17:03:22.342 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.write_dbid_to_manifest: 0 2024-08-17T17:03:22.342 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.log_readahead_size: 0 2024-08-17T17:03:22.342 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-08-17T17:03:22.342 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.best_efforts_recovery: 0 2024-08-17T17:03:22.342 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-08-17T17:03:22.342 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-08-17T17:03:22.342 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.allow_data_in_errors: 0 2024-08-17T17:03:22.342 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.db_host_id: __hostname__ 2024-08-17T17:03:22.342 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.enforce_single_del_contracts: true 2024-08-17T17:03:22.342 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_background_jobs: 2 2024-08-17T17:03:22.343 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_background_compactions: -1 2024-08-17T17:03:22.343 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_subcompactions: 1 2024-08-17T17:03:22.343 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-08-17T17:03:22.343 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-08-17T17:03:22.343 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.delayed_write_rate : 16777216 2024-08-17T17:03:22.343 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_total_wal_size: 0 2024-08-17T17:03:22.343 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-08-17T17:03:22.343 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.stats_dump_period_sec: 600 2024-08-17T17:03:22.343 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.stats_persist_period_sec: 600 2024-08-17T17:03:22.343 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-08-17T17:03:22.343 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_open_files: -1 2024-08-17T17:03:22.343 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.bytes_per_sync: 0 2024-08-17T17:03:22.343 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.wal_bytes_per_sync: 0 2024-08-17T17:03:22.344 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.strict_bytes_per_sync: 0 2024-08-17T17:03:22.344 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.compaction_readahead_size: 0 2024-08-17T17:03:22.344 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_background_flushes: -1 2024-08-17T17:03:22.344 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Compression algorithms supported: 2024-08-17T17:03:22.344 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: kZSTD supported: 0 2024-08-17T17:03:22.344 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: kXpressCompression supported: 0 2024-08-17T17:03:22.344 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: kBZip2Compression supported: 0 2024-08-17T17:03:22.344 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-08-17T17:03:22.344 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: kLZ4Compression supported: 1 2024-08-17T17:03:22.344 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: kZlibCompression supported: 1 2024-08-17T17:03:22.344 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: kLZ4HCCompression supported: 1 2024-08-17T17:03:22.344 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: kSnappyCompression supported: 1 2024-08-17T17:03:22.344 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Fast CRC32 supported: Supported on x86 2024-08-17T17:03:22.344 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: DMutex implementation: pthread_mutex_t 2024-08-17T17:03:22.345 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-08-17T17:03:22.345 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-08-17T17:03:22.345 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-08-17T17:03:22.345 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.merge_operator: 2024-08-17T17:03:22.345 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.compaction_filter: None 2024-08-17T17:03:22.345 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.compaction_filter_factory: None 2024-08-17T17:03:22.345 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.sst_partitioner_factory: None 2024-08-17T17:03:22.345 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.memtable_factory: SkipListFactory 2024-08-17T17:03:22.345 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.table_factory: BlockBasedTable 2024-08-17T17:03:22.345 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561c6ad62820) 2024-08-17T17:03:22.345 INFO:journalctl@ceph.mon.a.smithi136.stdout: cache_index_and_filter_blocks: 1 2024-08-17T17:03:22.345 INFO:journalctl@ceph.mon.a.smithi136.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-08-17T17:03:22.345 INFO:journalctl@ceph.mon.a.smithi136.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-08-17T17:03:22.345 INFO:journalctl@ceph.mon.a.smithi136.stdout: pin_top_level_index_and_filter: 1 2024-08-17T17:03:22.346 INFO:journalctl@ceph.mon.a.smithi136.stdout: index_type: 0 2024-08-17T17:03:22.346 INFO:journalctl@ceph.mon.a.smithi136.stdout: data_block_index_type: 0 2024-08-17T17:03:22.346 INFO:journalctl@ceph.mon.a.smithi136.stdout: index_shortening: 1 2024-08-17T17:03:22.346 INFO:journalctl@ceph.mon.a.smithi136.stdout: data_block_hash_table_util_ratio: 0.750000 2024-08-17T17:03:22.346 INFO:journalctl@ceph.mon.a.smithi136.stdout: checksum: 4 2024-08-17T17:03:22.346 INFO:journalctl@ceph.mon.a.smithi136.stdout: no_block_cache: 0 2024-08-17T17:03:22.346 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_cache: 0x561c6ad57350 2024-08-17T17:03:22.346 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_cache_name: BinnedLRUCache 2024-08-17T17:03:22.346 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_cache_options: 2024-08-17T17:03:22.346 INFO:journalctl@ceph.mon.a.smithi136.stdout: capacity : 536870912 2024-08-17T17:03:22.346 INFO:journalctl@ceph.mon.a.smithi136.stdout: num_shard_bits : 4 2024-08-17T17:03:22.346 INFO:journalctl@ceph.mon.a.smithi136.stdout: strict_capacity_limit : 0 2024-08-17T17:03:22.346 INFO:journalctl@ceph.mon.a.smithi136.stdout: high_pri_pool_ratio: 0.000 2024-08-17T17:03:22.346 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_cache_compressed: (nil) 2024-08-17T17:03:22.347 INFO:journalctl@ceph.mon.a.smithi136.stdout: persistent_cache: (nil) 2024-08-17T17:03:22.347 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_size: 4096 2024-08-17T17:03:22.347 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_size_deviation: 10 2024-08-17T17:03:22.347 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_restart_interval: 16 2024-08-17T17:03:22.347 INFO:journalctl@ceph.mon.a.smithi136.stdout: index_block_restart_interval: 1 2024-08-17T17:03:22.347 INFO:journalctl@ceph.mon.a.smithi136.stdout: metadata_block_size: 4096 2024-08-17T17:03:22.347 INFO:journalctl@ceph.mon.a.smithi136.stdout: partition_filters: 0 2024-08-17T17:03:22.347 INFO:journalctl@ceph.mon.a.smithi136.stdout: use_delta_encoding: 1 2024-08-17T17:03:22.347 INFO:journalctl@ceph.mon.a.smithi136.stdout: filter_policy: bloomfilter 2024-08-17T17:03:22.347 INFO:journalctl@ceph.mon.a.smithi136.stdout: whole_key_filtering: 1 2024-08-17T17:03:22.347 INFO:journalctl@ceph.mon.a.smithi136.stdout: verify_compression: 0 2024-08-17T17:03:22.347 INFO:journalctl@ceph.mon.a.smithi136.stdout: read_amp_bytes_per_bit: 0 2024-08-17T17:03:22.347 INFO:journalctl@ceph.mon.a.smithi136.stdout: format_version: 5 2024-08-17T17:03:22.347 INFO:journalctl@ceph.mon.a.smithi136.stdout: enable_index_compression: 1 2024-08-17T17:03:22.348 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_align: 0 2024-08-17T17:03:22.348 INFO:journalctl@ceph.mon.a.smithi136.stdout: max_auto_readahead_size: 262144 2024-08-17T17:03:22.348 INFO:journalctl@ceph.mon.a.smithi136.stdout: prepopulate_block_cache: 0 2024-08-17T17:03:22.348 INFO:journalctl@ceph.mon.a.smithi136.stdout: initial_auto_readahead_size: 8192 2024-08-17T17:03:22.348 INFO:journalctl@ceph.mon.a.smithi136.stdout: num_file_reads_for_auto_readahead: 2 2024-08-17T17:03:22.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.write_buffer_size: 33554432 2024-08-17T17:03:22.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_write_buffer_number: 2 2024-08-17T17:03:22.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.compression: NoCompression 2024-08-17T17:03:22.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.bottommost_compression: Disabled 2024-08-17T17:03:22.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.prefix_extractor: nullptr 2024-08-17T17:03:22.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-08-17T17:03:22.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.num_levels: 7 2024-08-17T17:03:22.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-08-17T17:03:22.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-08-17T17:03:22.349 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-08-17T17:03:22.349 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-08-17T17:03:22.349 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-08-17T17:03:22.349 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-08-17T17:03:22.349 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-08-17T17:03:22.349 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-08-17T17:03:22.349 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-08-17T17:03:22.349 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-08-17T17:03:22.349 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-08-17T17:03:22.349 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-08-17T17:03:22.349 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.compression_opts.window_bits: -14 2024-08-17T17:03:22.349 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.compression_opts.level: 32767 2024-08-17T17:03:22.349 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.compression_opts.strategy: 0 2024-08-17T17:03:22.349 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-08-17T17:03:22.350 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-08-17T17:03:22.350 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-08-17T17:03:22.350 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-08-17T17:03:22.350 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.compression_opts.enabled: false 2024-08-17T17:03:22.350 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-08-17T17:03:22.350 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-08-17T17:03:22.350 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-08-17T17:03:22.350 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-08-17T17:03:22.350 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.target_file_size_base: 67108864 2024-08-17T17:03:22.350 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.target_file_size_multiplier: 1 2024-08-17T17:03:22.350 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-08-17T17:03:22.350 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-08-17T17:03:22.350 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-08-17T17:03:22.351 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-08-17T17:03:22.351 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-08-17T17:03:22.351 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-08-17T17:03:22.351 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-08-17T17:03:22.351 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-08-17T17:03:22.351 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-08-17T17:03:22.351 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-08-17T17:03:22.351 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-08-17T17:03:22.351 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-08-17T17:03:22.351 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-08-17T17:03:22.351 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.arena_block_size: 1048576 2024-08-17T17:03:22.351 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-08-17T17:03:22.351 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-08-17T17:03:22.351 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.disable_auto_compactions: 0 2024-08-17T17:03:22.352 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-08-17T17:03:22.352 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-08-17T17:03:22.352 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-08-17T17:03:22.352 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-08-17T17:03:22.352 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-08-17T17:03:22.352 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-08-17T17:03:22.352 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-08-17T17:03:22.352 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-08-17T17:03:22.352 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-08-17T17:03:22.352 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-08-17T17:03:22.352 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.table_properties_collectors: 2024-08-17T17:03:22.352 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.inplace_update_support: 0 2024-08-17T17:03:22.352 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.inplace_update_num_locks: 10000 2024-08-17T17:03:22.352 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-08-17T17:03:22.353 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-08-17T17:03:22.353 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.memtable_huge_page_size: 0 2024-08-17T17:03:22.353 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.bloom_locality: 0 2024-08-17T17:03:22.353 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.max_successive_merges: 0 2024-08-17T17:03:22.353 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.optimize_filters_for_hits: 0 2024-08-17T17:03:22.353 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.paranoid_file_checks: 0 2024-08-17T17:03:22.353 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.force_consistency_checks: 1 2024-08-17T17:03:22.353 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.report_bg_io_stats: 0 2024-08-17T17:03:22.353 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.ttl: 2592000 2024-08-17T17:03:22.353 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.periodic_compaction_seconds: 0 2024-08-17T17:03:22.353 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-08-17T17:03:22.353 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-08-17T17:03:22.353 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.enable_blob_files: false 2024-08-17T17:03:22.354 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.min_blob_size: 0 2024-08-17T17:03:22.354 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.blob_file_size: 268435456 2024-08-17T17:03:22.354 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.blob_compression_type: NoCompression 2024-08-17T17:03:22.354 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.enable_blob_garbage_collection: false 2024-08-17T17:03:22.354 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-08-17T17:03:22.354 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-08-17T17:03:22.354 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-08-17T17:03:22.354 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.blob_file_starting_level: 0 2024-08-17T17:03:22.354 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-08-17T17:03:22.354 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-08-17T17:03:22.354 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-08-17T17:03:22.354 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 17e5137b-af42-4e49-a7e1-f98a5a53f78e 2024-08-17T17:03:22.354 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914202268913, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-08-17T17:03:22.354 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-08-17T17:03:22.355 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914202269807, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81096, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 256, "table_properties": {"data_size": 79272, "index_size": 234, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10267, "raw_average_key_size": 47, "raw_value_size": 73473, "raw_average_value_size": 341, "num_data_blocks": 10, "num_entries": 215, "num_filter_entries": 215, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1723914202, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "17e5137b-af42-4e49-a7e1-f98a5a53f78e", "db_session_id": "MTGPD67ZE4RN1MRCHG4S", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-08-17T17:03:22.355 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914202269900, "job": 1, "event": "recovery_finished"} 2024-08-17T17:03:22.355 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-08-17T17:03:22.355 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-08-17T17:03:22.355 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x561c6ad80e00 2024-08-17T17:03:22.355 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: DB pointer 0x561c6ae74000 2024-08-17T17:03:22.355 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-08-17T17:03:22.355 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-08-17T17:03:22.355 INFO:journalctl@ceph.mon.a.smithi136.stdout: ** DB Stats ** 2024-08-17T17:03:22.355 INFO:journalctl@ceph.mon.a.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-17T17:03:22.355 INFO:journalctl@ceph.mon.a.smithi136.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-08-17T17:03:22.355 INFO:journalctl@ceph.mon.a.smithi136.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-17T17:03:22.355 INFO:journalctl@ceph.mon.a.smithi136.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-17T17:03:22.356 INFO:journalctl@ceph.mon.a.smithi136.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-08-17T17:03:22.356 INFO:journalctl@ceph.mon.a.smithi136.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-17T17:03:22.356 INFO:journalctl@ceph.mon.a.smithi136.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-17T17:03:22.356 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-08-17T17:03:22.356 INFO:journalctl@ceph.mon.a.smithi136.stdout: ** Compaction Stats [default] ** 2024-08-17T17:03:22.356 INFO:journalctl@ceph.mon.a.smithi136.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-17T17:03:22.356 INFO:journalctl@ceph.mon.a.smithi136.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-08-17T17:03:22.356 INFO:journalctl@ceph.mon.a.smithi136.stdout: L0 2/0 81.03 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-08-17T17:03:22.356 INFO:journalctl@ceph.mon.a.smithi136.stdout: Sum 2/0 81.03 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-08-17T17:03:22.356 INFO:journalctl@ceph.mon.a.smithi136.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-08-17T17:03:22.356 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-08-17T17:03:22.356 INFO:journalctl@ceph.mon.a.smithi136.stdout: ** Compaction Stats [default] ** 2024-08-17T17:03:22.356 INFO:journalctl@ceph.mon.a.smithi136.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-17T17:03:22.356 INFO:journalctl@ceph.mon.a.smithi136.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-17T17:03:22.357 INFO:journalctl@ceph.mon.a.smithi136.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 138.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-08-17T17:03:22.357 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-08-17T17:03:22.357 INFO:journalctl@ceph.mon.a.smithi136.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-08-17T17:03:22.357 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-08-17T17:03:22.357 INFO:journalctl@ceph.mon.a.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-17T17:03:22.357 INFO:journalctl@ceph.mon.a.smithi136.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-08-17T17:03:22.357 INFO:journalctl@ceph.mon.a.smithi136.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-08-17T17:03:22.357 INFO:journalctl@ceph.mon.a.smithi136.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-08-17T17:03:22.357 INFO:journalctl@ceph.mon.a.smithi136.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-08-17T17:03:22.357 INFO:journalctl@ceph.mon.a.smithi136.stdout: AddFile(Keys): cumulative 0, interval 0 2024-08-17T17:03:22.357 INFO:journalctl@ceph.mon.a.smithi136.stdout: Cumulative compaction: 0.00 GB write, 25.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:03:22.357 INFO:journalctl@ceph.mon.a.smithi136.stdout: Interval compaction: 0.00 GB write, 25.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:03:22.357 INFO:journalctl@ceph.mon.a.smithi136.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-08-17T17:03:22.358 INFO:journalctl@ceph.mon.a.smithi136.stdout: Block cache BinnedLRUCache@0x561c6ad57350#7 capacity: 512.00 MB usage: 1.19 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-08-17T17:03:22.358 INFO:journalctl@ceph.mon.a.smithi136.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.42 KB,8.04663e-05%) Misc(1,0.00 KB,0%) 2024-08-17T17:03:22.358 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-08-17T17:03:22.358 INFO:journalctl@ceph.mon.a.smithi136.stdout: ** File Read Latency Histogram By Level [default] ** 2024-08-17T17:03:22.358 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: starting mon.a rank 0 at public addrs [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] at bind addrs [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:03:22.358 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: mon.a@-1(???) e1 preinit fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:03:22.358 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: mon.a@-1(???).mds e1 new map 2024-08-17T17:03:22.358 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: mon.a@-1(???).mds e1 print_map 2024-08-17T17:03:22.358 INFO:journalctl@ceph.mon.a.smithi136.stdout: e1 2024-08-17T17:03:22.358 INFO:journalctl@ceph.mon.a.smithi136.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-08-17T17:03:22.358 INFO:journalctl@ceph.mon.a.smithi136.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-08-17T17:03:22.358 INFO:journalctl@ceph.mon.a.smithi136.stdout: legacy client fscid: -1 2024-08-17T17:03:22.358 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-08-17T17:03:22.359 INFO:journalctl@ceph.mon.a.smithi136.stdout: No filesystems configured 2024-08-17T17:03:22.359 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-08-17T17:03:22.359 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-08-17T17:03:22.359 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-08-17T17:03:22.359 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-08-17T17:03:22.359 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-08-17T17:03:22.359 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: mon.a is new leader, mons a in quorum (ranks 0) 2024-08-17T17:03:22.359 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: monmap e1: 1 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-08-17T17:03:22.359 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: fsmap 2024-08-17T17:03:22.359 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: osdmap e1: 0 total, 0 up, 0 in 2024-08-17T17:03:22.359 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 ceph-mon[31034]: mgrmap e1: no daemons active 2024-08-17T17:03:22.360 INFO:teuthology.orchestra.run.smithi136.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-08-17T17:03:22.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 bash[30995]: cc4e5e5ff75686490f40ecc7fb86f646c8220efc2d2968ba474c1a497629a5c2 2024-08-17T17:03:22.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:22 smithi136 systemd[1]: Started Ceph mon.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:03:23.397 INFO:teuthology.orchestra.run.smithi136.stdout:Wrote config to /etc/ceph/ceph.conf 2024-08-17T17:03:23.419 INFO:teuthology.orchestra.run.smithi136.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-08-17T17:03:23.419 INFO:teuthology.orchestra.run.smithi136.stdout:Creating mgr... 2024-08-17T17:03:23.420 INFO:teuthology.orchestra.run.smithi136.stdout:Verifying port 0.0.0.0:9283 ... 2024-08-17T17:03:23.420 INFO:teuthology.orchestra.run.smithi136.stdout:Verifying port 0.0.0.0:8765 ... 2024-08-17T17:03:23.665 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8270436c-5cba-11ef-bccf-c7b262605968@mgr.y 2024-08-17T17:03:23.665 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8270436c-5cba-11ef-bccf-c7b262605968@mgr.y.service: Unit ceph-8270436c-5cba-11ef-bccf-c7b262605968@mgr.y.service not loaded. 2024-08-17T17:03:23.808 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8270436c-5cba-11ef-bccf-c7b262605968.target.wants/ceph-8270436c-5cba-11ef-bccf-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-8270436c-5cba-11ef-bccf-c7b262605968@.service. 2024-08-17T17:03:23.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:23 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/492536307' entity='client.admin' 2024-08-17T17:03:24.307 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:24 smithi136 podman[31269]: 2024-08-17 17:03:24.23643159 +0000 UTC m=+0.308704141 container start f74bb81f3a53c5eb2516d75214b8c56877f1a4e8ee014419a6da99081e0bfe1e (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=reef-ced649f, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-08-17T17:03:24.307 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:24 smithi136 bash[31269]: f74bb81f3a53c5eb2516d75214b8c56877f1a4e8ee014419a6da99081e0bfe1e 2024-08-17T17:03:24.320 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-17T17:03:24.320 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout disabled 2024-08-17T17:03:24.328 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-17T17:03:24.328 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout inactive 2024-08-17T17:03:24.328 INFO:teuthology.orchestra.run.smithi136.stdout:firewalld.service is not enabled 2024-08-17T17:03:24.328 INFO:teuthology.orchestra.run.smithi136.stdout:Not possible to enable service . firewalld.service is not available 2024-08-17T17:03:24.336 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-17T17:03:24.336 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout disabled 2024-08-17T17:03:24.344 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-17T17:03:24.344 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout inactive 2024-08-17T17:03:24.344 INFO:teuthology.orchestra.run.smithi136.stdout:firewalld.service is not enabled 2024-08-17T17:03:24.344 INFO:teuthology.orchestra.run.smithi136.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-08-17T17:03:24.344 INFO:teuthology.orchestra.run.smithi136.stdout:Waiting for mgr to start... 2024-08-17T17:03:24.344 INFO:teuthology.orchestra.run.smithi136.stdout:Waiting for mgr... 2024-08-17T17:03:24.559 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:24 smithi136 systemd[1]: Started Ceph mgr.y for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:03:24.559 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:24 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:24.360+0000 7f4624e00040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-17T17:03:24.559 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:24 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:24.508+0000 7f4624e00040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:03:24.824 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:24 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:24.579+0000 7f4624e00040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-17T17:03:25.121 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:24 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/1630454478' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-08-17T17:03:25.122 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:24 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:24.831+0000 7f4624e00040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-17T17:03:25.122 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:24 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:24.893+0000 7f4624e00040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-17T17:03:25.418 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-08-17T17:03:25.418 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout { 2024-08-17T17:03:25.418 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "fsid": "8270436c-5cba-11ef-bccf-c7b262605968", 2024-08-17T17:03:25.418 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "health": { 2024-08-17T17:03:25.418 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-08-17T17:03:25.419 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-08-17T17:03:25.419 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-08-17T17:03:25.419 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:03:25.419 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-08-17T17:03:25.419 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-08-17T17:03:25.419 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 0 2024-08-17T17:03:25.419 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-08-17T17:03:25.419 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-08-17T17:03:25.419 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "a" 2024-08-17T17:03:25.419 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-08-17T17:03:25.419 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-08-17T17:03:25.419 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "monmap": { 2024-08-17T17:03:25.419 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-17T17:03:25.420 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-08-17T17:03:25.420 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-08-17T17:03:25.420 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:03:25.420 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-08-17T17:03:25.420 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-17T17:03:25.420 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-08-17T17:03:25.420 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-08-17T17:03:25.420 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-08-17T17:03:25.423 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-08-17T17:03:25.423 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-08-17T17:03:25.424 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-08-17T17:03:25.424 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:03:25.424 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-08-17T17:03:25.424 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-08-17T17:03:25.424 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-08-17T17:03:25.424 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-08-17T17:03:25.424 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-08-17T17:03:25.424 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-08-17T17:03:25.424 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-08-17T17:03:25.424 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-08-17T17:03:25.424 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-08-17T17:03:25.424 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:03:25.424 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-08-17T17:03:25.425 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-17T17:03:25.425 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-08-17T17:03:25.425 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-08-17T17:03:25.425 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:03:25.425 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-08-17T17:03:25.425 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "available": false, 2024-08-17T17:03:25.425 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-08-17T17:03:25.426 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "modules": [ 2024-08-17T17:03:25.426 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "iostat", 2024-08-17T17:03:25.426 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "nfs", 2024-08-17T17:03:25.426 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "restful" 2024-08-17T17:03:25.426 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-08-17T17:03:25.426 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-17T17:03:25.426 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:03:25.426 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-08-17T17:03:25.426 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-17T17:03:25.426 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "modified": "2024-08-17T17:03:17.683210+0000", 2024-08-17T17:03:25.426 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-17T17:03:25.426 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:03:25.426 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-08-17T17:03:25.426 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout } 2024-08-17T17:03:25.427 INFO:teuthology.orchestra.run.smithi136.stdout:mgr not available, waiting (1/15)... 2024-08-17T17:03:26.194 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:25 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:25.820+0000 7f4624e00040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-17T17:03:26.194 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:25 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:25.932+0000 7f4624e00040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-17T17:03:26.194 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:25 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:25.988+0000 7f4624e00040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-17T17:03:26.195 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:26.193+0000 7f4624e00040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-17T17:03:26.446 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:26.253+0000 7f4624e00040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-17T17:03:26.446 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:26.318+0000 7f4624e00040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-17T17:03:26.870 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:26.460+0000 7f4624e00040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:03:27.176 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:26.869+0000 7f4624e00040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-17T17:03:27.176 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:27.175+0000 7f4624e00040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-17T17:03:27.446 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:27.240+0000 7f4624e00040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-17T17:03:27.446 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:27.384+0000 7f4624e00040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-17T17:03:27.874 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:27.701+0000 7f4624e00040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-17T17:03:27.875 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:27.762+0000 7f4624e00040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-17T17:03:28.170 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:27 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/2821030239' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-08-17T17:03:28.171 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:27.956+0000 7f4624e00040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-17T17:03:28.171 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:28 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:28.104+0000 7f4624e00040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-17T17:03:28.171 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:28 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:28.169+0000 7f4624e00040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-17T17:03:28.446 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:28 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:28.253+0000 7f4624e00040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-17T17:03:28.446 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:28 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:28.311+0000 7f4624e00040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-17T17:03:28.540 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-08-17T17:03:28.540 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout { 2024-08-17T17:03:28.541 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "fsid": "8270436c-5cba-11ef-bccf-c7b262605968", 2024-08-17T17:03:28.541 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "health": { 2024-08-17T17:03:28.541 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-08-17T17:03:28.541 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-08-17T17:03:28.541 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-08-17T17:03:28.541 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:03:28.541 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-08-17T17:03:28.541 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-08-17T17:03:28.541 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 0 2024-08-17T17:03:28.541 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-08-17T17:03:28.541 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-08-17T17:03:28.541 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "a" 2024-08-17T17:03:28.541 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-08-17T17:03:28.541 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-08-17T17:03:28.542 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "monmap": { 2024-08-17T17:03:28.542 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-17T17:03:28.542 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-08-17T17:03:28.542 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-08-17T17:03:28.542 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:03:28.542 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-08-17T17:03:28.542 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-17T17:03:28.542 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-08-17T17:03:28.542 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-08-17T17:03:28.542 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-08-17T17:03:28.542 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-08-17T17:03:28.546 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-08-17T17:03:28.546 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-08-17T17:03:28.546 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:03:28.546 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-08-17T17:03:28.546 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-08-17T17:03:28.546 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-08-17T17:03:28.546 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-08-17T17:03:28.546 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-08-17T17:03:28.546 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-08-17T17:03:28.547 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-08-17T17:03:28.547 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-08-17T17:03:28.547 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-08-17T17:03:28.547 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:03:28.547 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-08-17T17:03:28.547 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-17T17:03:28.547 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-08-17T17:03:28.547 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-08-17T17:03:28.547 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:03:28.547 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-08-17T17:03:28.547 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "available": false, 2024-08-17T17:03:28.547 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-08-17T17:03:28.547 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "modules": [ 2024-08-17T17:03:28.547 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "iostat", 2024-08-17T17:03:28.548 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "nfs", 2024-08-17T17:03:28.548 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "restful" 2024-08-17T17:03:28.548 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-08-17T17:03:28.548 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-17T17:03:28.548 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:03:28.548 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-08-17T17:03:28.548 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-17T17:03:28.548 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "modified": "2024-08-17T17:03:17.683210+0000", 2024-08-17T17:03:28.548 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-17T17:03:28.548 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:03:28.548 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-08-17T17:03:28.548 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout } 2024-08-17T17:03:28.548 INFO:teuthology.orchestra.run.smithi136.stdout:mgr not available, waiting (2/15)... 2024-08-17T17:03:29.196 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:28 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:28.837+0000 7f4624e00040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-17T17:03:29.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:28 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:28.992+0000 7f4624e00040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-17T17:03:29.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:29 smithi136 ceph-mon[31034]: Activating manager daemon y 2024-08-17T17:03:29.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:29 smithi136 ceph-mon[31034]: mgrmap e2: y(active, starting, since 0.00300266s) 2024-08-17T17:03:29.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:29 smithi136 ceph-mon[31034]: from='mgr.14100 172.21.15.136:0/1201362535' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-17T17:03:29.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:29 smithi136 ceph-mon[31034]: from='mgr.14100 172.21.15.136:0/1201362535' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-17T17:03:29.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:29 smithi136 ceph-mon[31034]: from='mgr.14100 172.21.15.136:0/1201362535' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-17T17:03:29.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:29 smithi136 ceph-mon[31034]: from='mgr.14100 172.21.15.136:0/1201362535' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:03:29.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:29 smithi136 ceph-mon[31034]: from='mgr.14100 172.21.15.136:0/1201362535' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-17T17:03:29.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:29 smithi136 ceph-mon[31034]: Manager daemon y is now available 2024-08-17T17:03:29.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:29 smithi136 ceph-mon[31034]: from='mgr.14100 172.21.15.136:0/1201362535' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:03:29.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:29 smithi136 ceph-mon[31034]: from='mgr.14100 172.21.15.136:0/1201362535' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:03:29.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:29 smithi136 ceph-mon[31034]: from='mgr.14100 172.21.15.136:0/1201362535' entity='mgr.y' 2024-08-17T17:03:29.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:29 smithi136 ceph-mon[31034]: from='mgr.14100 172.21.15.136:0/1201362535' entity='mgr.y' 2024-08-17T17:03:29.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:29 smithi136 ceph-mon[31034]: from='mgr.14100 172.21.15.136:0/1201362535' entity='mgr.y' 2024-08-17T17:03:31.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:30 smithi136 ceph-mon[31034]: mgrmap e3: y(active, since 1.00721s) 2024-08-17T17:03:31.754 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-08-17T17:03:31.755 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout { 2024-08-17T17:03:31.755 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "fsid": "8270436c-5cba-11ef-bccf-c7b262605968", 2024-08-17T17:03:31.755 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "health": { 2024-08-17T17:03:31.755 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-08-17T17:03:31.755 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-08-17T17:03:31.755 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-08-17T17:03:31.755 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:03:31.755 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-08-17T17:03:31.755 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-08-17T17:03:31.755 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 0 2024-08-17T17:03:31.755 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-08-17T17:03:31.755 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-08-17T17:03:31.755 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "a" 2024-08-17T17:03:31.756 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-08-17T17:03:31.756 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-08-17T17:03:31.756 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "monmap": { 2024-08-17T17:03:31.756 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-17T17:03:31.756 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-08-17T17:03:31.756 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-08-17T17:03:31.756 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:03:31.756 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-08-17T17:03:31.756 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-17T17:03:31.756 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-08-17T17:03:31.756 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-08-17T17:03:31.756 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-08-17T17:03:31.756 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-08-17T17:03:31.760 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-08-17T17:03:31.760 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-08-17T17:03:31.760 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:03:31.760 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-08-17T17:03:31.760 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-08-17T17:03:31.761 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-08-17T17:03:31.761 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-08-17T17:03:31.761 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-08-17T17:03:31.761 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-08-17T17:03:31.761 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-08-17T17:03:31.761 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-08-17T17:03:31.761 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-08-17T17:03:31.761 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:03:31.761 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-08-17T17:03:31.761 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-17T17:03:31.761 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-08-17T17:03:31.761 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-08-17T17:03:31.761 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:03:31.762 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-08-17T17:03:31.762 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "available": true, 2024-08-17T17:03:31.762 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-08-17T17:03:31.762 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "modules": [ 2024-08-17T17:03:31.762 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "iostat", 2024-08-17T17:03:31.762 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "nfs", 2024-08-17T17:03:31.762 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "restful" 2024-08-17T17:03:31.762 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ], 2024-08-17T17:03:31.762 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-17T17:03:31.762 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:03:31.762 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-08-17T17:03:31.762 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-17T17:03:31.762 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "modified": "2024-08-17T17:03:17.683210+0000", 2024-08-17T17:03:31.762 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-17T17:03:31.763 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:03:31.763 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-08-17T17:03:31.763 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout } 2024-08-17T17:03:31.763 INFO:teuthology.orchestra.run.smithi136.stdout:mgr is available 2024-08-17T17:03:32.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:32 smithi136 ceph-mon[31034]: mgrmap e4: y(active, since 2s) 2024-08-17T17:03:32.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:32 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/840498159' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-08-17T17:03:35.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:34 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/3694687669' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-08-17T17:03:40.839 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-08-17T17:03:40.839 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout [global] 2024-08-17T17:03:40.839 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout fsid = 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:03:40.839 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-08-17T17:03:40.839 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-08-17T17:03:40.839 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-08-17T17:03:40.839 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-08-17T17:03:40.839 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-08-17T17:03:40.839 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout [mgr] 2024-08-17T17:03:40.839 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-08-17T17:03:40.839 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 2024-08-17T17:03:40.839 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout [osd] 2024-08-17T17:03:40.839 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-08-17T17:03:40.840 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-08-17T17:03:40.840 INFO:teuthology.orchestra.run.smithi136.stdout:Enabling cephadm module... 2024-08-17T17:03:43.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:43 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/2023494193' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-08-17T17:03:43.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:43 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ignoring --setuser ceph since I am not root 2024-08-17T17:03:43.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:43 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ignoring --setgroup ceph since I am not root 2024-08-17T17:03:43.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:43 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:43.802+0000 7fe0dbf47040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-17T17:03:44.260 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:43 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:43.947+0000 7fe0dbf47040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:03:44.260 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:44 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:44.015+0000 7fe0dbf47040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-17T17:03:44.260 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:44 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:44.258+0000 7fe0dbf47040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-17T17:03:44.535 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:44 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:44.318+0000 7fe0dbf47040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-17T17:03:44.838 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:44 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/2023494193' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-08-17T17:03:44.838 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:44 smithi136 ceph-mon[31034]: mgrmap e5: y(active, since 14s) 2024-08-17T17:03:45.438 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:45.264+0000 7fe0dbf47040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-17T17:03:45.439 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:45.380+0000 7fe0dbf47040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-17T17:03:45.439 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:45.436+0000 7fe0dbf47040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-17T17:03:45.475 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout { 2024-08-17T17:03:45.475 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-08-17T17:03:45.475 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "available": true, 2024-08-17T17:03:45.475 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-08-17T17:03:45.475 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-08-17T17:03:45.475 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout } 2024-08-17T17:03:45.475 INFO:teuthology.orchestra.run.smithi136.stdout:Waiting for the mgr to restart... 2024-08-17T17:03:45.476 INFO:teuthology.orchestra.run.smithi136.stdout:Waiting for mgr epoch 5... 2024-08-17T17:03:45.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:45 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/1029466452' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-08-17T17:03:45.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:45.654+0000 7fe0dbf47040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-17T17:03:46.052 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:45.716+0000 7fe0dbf47040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-17T17:03:46.052 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:45.781+0000 7fe0dbf47040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-17T17:03:46.052 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:45.922+0000 7fe0dbf47040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:03:46.662 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:46 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:46.345+0000 7fe0dbf47040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-17T17:03:46.662 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:46 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:46.660+0000 7fe0dbf47040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-17T17:03:46.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:46 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:46.724+0000 7fe0dbf47040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-17T17:03:46.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:46 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:46.867+0000 7fe0dbf47040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-17T17:03:47.446 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:47.166+0000 7fe0dbf47040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-17T17:03:47.447 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:47.226+0000 7fe0dbf47040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-17T17:03:47.447 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:47.405+0000 7fe0dbf47040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-17T17:03:47.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:47.549+0000 7fe0dbf47040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-17T17:03:47.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:47.611+0000 7fe0dbf47040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-17T17:03:47.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:47.691+0000 7fe0dbf47040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-17T17:03:47.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:47.749+0000 7fe0dbf47040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-17T17:03:48.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:48 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:48.262+0000 7fe0dbf47040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-17T17:03:48.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:48 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:48.414+0000 7fe0dbf47040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-17T17:03:48.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:48 smithi136 ceph-mon[31034]: Active manager daemon y restarted 2024-08-17T17:03:48.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:48 smithi136 ceph-mon[31034]: Activating manager daemon y 2024-08-17T17:03:48.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:48 smithi136 ceph-mon[31034]: osdmap e2: 0 total, 0 up, 0 in 2024-08-17T17:03:48.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:48 smithi136 ceph-mon[31034]: mgrmap e6: y(active, starting, since 0.00296049s) 2024-08-17T17:03:48.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:48 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:03:48.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:48 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-17T17:03:48.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:48 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-17T17:03:48.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:48 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-17T17:03:48.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:48 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-17T17:03:48.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:48 smithi136 ceph-mon[31034]: Manager daemon y is now available 2024-08-17T17:03:48.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:48 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' 2024-08-17T17:03:48.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:48 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' 2024-08-17T17:03:48.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:48 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:03:48.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:48 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:03:48.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:48 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:03:48.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:48 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:03:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:49 smithi136 ceph-mon[31034]: Found migration_current of "None". Setting to last migration. 2024-08-17T17:03:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:49 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' 2024-08-17T17:03:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:49 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' 2024-08-17T17:03:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:49 smithi136 ceph-mon[31034]: mgrmap e7: y(active, since 1.00502s) 2024-08-17T17:03:49.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:49.611+0000 7fe043dd6640 -1 log_channel(cephadm) log [ERR] : [17/Aug/2024:17:03:49] ENGINE Error in HTTPServer.serve 2024-08-17T17:03:49.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: Traceback (most recent call last): 2024-08-17T17:03:49.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:03:49.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: self._connections.run(self.expiration_interval) 2024-08-17T17:03:49.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:03:49.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: self._run(expiration_interval) 2024-08-17T17:03:49.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:03:49.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:03:49.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:03:49.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:03:49.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:03:49.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: s = self.context.wrap_socket( 2024-08-17T17:03:49.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:03:49.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: return self.sslsocket_class._create( 2024-08-17T17:03:49.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:03:49.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: self.do_handshake() 2024-08-17T17:03:49.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:03:49.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: self._sslobj.do_handshake() 2024-08-17T17:03:49.699 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:03:49.699 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:03:49.956 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout { 2024-08-17T17:03:49.956 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-08-17T17:03:49.956 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "initialized": true 2024-08-17T17:03:49.956 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout } 2024-08-17T17:03:49.956 INFO:teuthology.orchestra.run.smithi136.stdout:mgr epoch 5 is available 2024-08-17T17:03:49.956 INFO:teuthology.orchestra.run.smithi136.stdout:Setting orchestrator backend to cephadm... 2024-08-17T17:03:50.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:50 smithi136 ceph-mon[31034]: [17/Aug/2024:17:03:49] ENGINE Bus STARTING 2024-08-17T17:03:50.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:50 smithi136 ceph-mon[31034]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-08-17T17:03:50.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:50 smithi136 ceph-mon[31034]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-08-17T17:03:50.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:50 smithi136 ceph-mon[31034]: [17/Aug/2024:17:03:49] ENGINE Serving on http://172.21.15.136:8765 2024-08-17T17:03:50.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:50 smithi136 ceph-mon[31034]: [17/Aug/2024:17:03:49] ENGINE Serving on https://172.21.15.136:7150 2024-08-17T17:03:50.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:50 smithi136 ceph-mon[31034]: [17/Aug/2024:17:03:49] ENGINE Bus STARTED 2024-08-17T17:03:50.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:50 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:03:50.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:50 smithi136 ceph-mon[31034]: [17/Aug/2024:17:03:49] ENGINE Error in HTTPServer.serve 2024-08-17T17:03:50.699 INFO:journalctl@ceph.mon.a.smithi136.stdout: Traceback (most recent call last): 2024-08-17T17:03:50.699 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:03:50.699 INFO:journalctl@ceph.mon.a.smithi136.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:03:50.699 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:03:50.699 INFO:journalctl@ceph.mon.a.smithi136.stdout: self._run(expiration_interval) 2024-08-17T17:03:50.699 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:03:50.700 INFO:journalctl@ceph.mon.a.smithi136.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:03:50.700 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:03:50.700 INFO:journalctl@ceph.mon.a.smithi136.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:03:50.700 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:03:50.700 INFO:journalctl@ceph.mon.a.smithi136.stdout: s = self.context.wrap_socket( 2024-08-17T17:03:50.700 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:03:50.700 INFO:journalctl@ceph.mon.a.smithi136.stdout: return self.sslsocket_class._create( 2024-08-17T17:03:50.701 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:03:50.701 INFO:journalctl@ceph.mon.a.smithi136.stdout: self.do_handshake() 2024-08-17T17:03:50.701 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:03:50.701 INFO:journalctl@ceph.mon.a.smithi136.stdout: self._sslobj.do_handshake() 2024-08-17T17:03:50.701 INFO:journalctl@ceph.mon.a.smithi136.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:03:51.640 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:51 smithi136 ceph-mon[31034]: mgrmap e8: y(active, since 2s) 2024-08-17T17:03:51.640 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:51 smithi136 ceph-mon[31034]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:03:51.640 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:51 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' 2024-08-17T17:03:51.640 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:51 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:03:52.218 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout value unchanged 2024-08-17T17:03:52.218 INFO:teuthology.orchestra.run.smithi136.stdout:Generating ssh key... 2024-08-17T17:03:52.881 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:52 smithi136 ceph-mon[31034]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:03:53.187 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: Generating public/private rsa key pair. 2024-08-17T17:03:53.187 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: Your identification has been saved in /tmp/tmpg93_60tb/key 2024-08-17T17:03:53.187 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: Your public key has been saved in /tmp/tmpg93_60tb/key.pub 2024-08-17T17:03:53.187 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: The key fingerprint is: 2024-08-17T17:03:53.187 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: SHA256:c6zmjYMBn5xcn8fuwPplRYn41NrVHa4tVHxGlFp/5Oo ceph-8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:03:53.187 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: The key's randomart image is: 2024-08-17T17:03:53.187 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: +---[RSA 3072]----+ 2024-08-17T17:03:53.187 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: | .=+| 2024-08-17T17:03:53.187 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: | . oo=B| 2024-08-17T17:03:53.187 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: | . o.*=*| 2024-08-17T17:03:53.187 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: | . .. o.=o.+| 2024-08-17T17:03:53.187 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: | = +S.oooooo.| 2024-08-17T17:03:53.188 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: | B =o o.o | 2024-08-17T17:03:53.188 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: | oo ooo. | 2024-08-17T17:03:53.188 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: | .o.+ +. E | 2024-08-17T17:03:53.188 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: | +oo.. | 2024-08-17T17:03:53.188 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:03:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: +----[SHA256]-----+ 2024-08-17T17:03:54.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:53 smithi136 ceph-mon[31034]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:03:54.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:53 smithi136 ceph-mon[31034]: Generating ssh key... 2024-08-17T17:03:54.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:53 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' 2024-08-17T17:03:54.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:53 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' 2024-08-17T17:03:54.537 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzoF5/ZXVutd9+by8TD8PUjNlDoGEh3vG8qpPoxjBeHnUw32eLt7DZ9XgTWm6zwCULiuxWMnQ3GaHzvn8rkRfnwPtKLg63Cm04wm25KwcKXSCUOixVFXySPMBSImEudSu0UwATAwN8Th4WrwskcesxRnoakGnyTWM+h62MMJez3vUzYGx63ALLYi1bXu5/GP/YeCJFrPaaWYIuZpEOdBipH6CECuMK0FSc97ZI0d71nJx8gNv5sRzDeeNnAUKdaEBMokg75eZuV8p89mKDXDrnMHeeeeU0IVrxrSfbx22XLAvRNaiPD8ZxpmzOM0SG+z2CeeO0vfEFDPkCg6+bvEFEIsRm5Oga1i+7SQJWXAzQHvdpU8UKz3C87RmhIPXrYWhNvtTY/T2pTSZLjyN1h/lJ9uaQxMwzXPn5Y/QzhhT/YJ5yHeogm4+T5UxWTtGUJT0yvTkwRe02WBvF3CaHUmJF0dmlQz9u+H8V60XydToE85HAz/p5054pUtxIwFUMujk= ceph-8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:03:54.537 INFO:teuthology.orchestra.run.smithi136.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-08-17T17:03:54.537 INFO:teuthology.orchestra.run.smithi136.stdout:Adding key to root@localhost authorized_keys... 2024-08-17T17:03:54.538 INFO:teuthology.orchestra.run.smithi136.stdout:Adding host smithi136... 2024-08-17T17:03:54.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:54 smithi136 ceph-mon[31034]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:03:56.138 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:55 smithi136 ceph-mon[31034]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi136", "addr": "172.21.15.136", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:03:57.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:56 smithi136 ceph-mon[31034]: Deploying cephadm binary to smithi136 2024-08-17T17:03:58.269 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout Added host 'smithi136' with addr '172.21.15.136' 2024-08-17T17:03:58.269 INFO:teuthology.orchestra.run.smithi136.stdout:Deploying unmanaged mon service... 2024-08-17T17:03:58.938 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:58 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' 2024-08-17T17:03:58.939 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:58 smithi136 ceph-mon[31034]: Added host smithi136 2024-08-17T17:03:58.939 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:03:58 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:03:59.778 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-08-17T17:03:59.779 INFO:teuthology.orchestra.run.smithi136.stdout:Deploying unmanaged mgr service... 2024-08-17T17:04:00.371 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:00 smithi136 ceph-mon[31034]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:04:00.372 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:00 smithi136 ceph-mon[31034]: Saving service mon spec with placement count:5 2024-08-17T17:04:00.372 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:00 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' 2024-08-17T17:04:01.345 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-08-17T17:04:01.670 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:01 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' 2024-08-17T17:04:01.670 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:01 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' 2024-08-17T17:04:01.670 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:01 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' 2024-08-17T17:04:02.616 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:02 smithi136 ceph-mon[31034]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:04:02.617 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:02 smithi136 ceph-mon[31034]: Saving service mgr spec with placement count:2 2024-08-17T17:04:02.617 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:02 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/3386975253' entity='client.admin' 2024-08-17T17:04:03.977 INFO:teuthology.orchestra.run.smithi136.stdout:Enabling the dashboard module... 2024-08-17T17:04:04.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:04 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/3258709299' entity='client.admin' 2024-08-17T17:04:04.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:04 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' 2024-08-17T17:04:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:05 smithi136 ceph-mon[31034]: from='mgr.14118 172.21.15.136:0/979465778' entity='mgr.y' 2024-08-17T17:04:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:05 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/3080075216' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-08-17T17:04:05.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:05 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ignoring --setuser ceph since I am not root 2024-08-17T17:04:05.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:05 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ignoring --setgroup ceph since I am not root 2024-08-17T17:04:05.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:05 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:05.843+0000 7f767a392040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-17T17:04:06.269 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:05 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:05.989+0000 7f767a392040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:04:06.270 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:06.055+0000 7f767a392040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-17T17:04:06.544 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:06.298+0000 7f767a392040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-17T17:04:06.545 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:06.358+0000 7f767a392040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-17T17:04:06.806 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:06 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/3080075216' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-08-17T17:04:06.806 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:06 smithi136 ceph-mon[31034]: mgrmap e9: y(active, since 17s) 2024-08-17T17:04:07.496 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:07.325+0000 7f767a392040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-17T17:04:07.497 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:07.438+0000 7f767a392040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-17T17:04:07.497 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:07.495+0000 7f767a392040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-17T17:04:07.769 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:07 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/3999647613' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-08-17T17:04:07.769 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:07.708+0000 7f767a392040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-17T17:04:07.769 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:07.767+0000 7f767a392040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-17T17:04:07.889 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout { 2024-08-17T17:04:07.890 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-08-17T17:04:07.890 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "available": true, 2024-08-17T17:04:07.890 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-08-17T17:04:07.890 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-08-17T17:04:07.890 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout } 2024-08-17T17:04:07.890 INFO:teuthology.orchestra.run.smithi136.stdout:Waiting for the mgr to restart... 2024-08-17T17:04:07.890 INFO:teuthology.orchestra.run.smithi136.stdout:Waiting for mgr epoch 9... 2024-08-17T17:04:08.101 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:07.834+0000 7f767a392040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-17T17:04:08.101 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:07.980+0000 7f767a392040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:04:08.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:08 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:08.407+0000 7f767a392040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-17T17:04:09.196 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:08 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:08.712+0000 7f767a392040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-17T17:04:09.196 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:08 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:08.776+0000 7f767a392040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-17T17:04:09.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:08 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:08.919+0000 7f767a392040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-17T17:04:09.602 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:09 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:09.217+0000 7f767a392040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-17T17:04:09.602 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:09 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:09.277+0000 7f767a392040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-17T17:04:09.602 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:09 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:09.456+0000 7f767a392040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-17T17:04:09.602 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:09 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:09.600+0000 7f767a392040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-17T17:04:09.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:09 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:09.663+0000 7f767a392040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-17T17:04:09.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:09 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:09.744+0000 7f767a392040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-17T17:04:09.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:09 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:09.801+0000 7f767a392040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-17T17:04:10.620 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:10 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:10.317+0000 7f767a392040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-17T17:04:10.621 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:10 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:10.469+0000 7f767a392040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-17T17:04:10.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:10 smithi136 ceph-mon[31034]: Active manager daemon y restarted 2024-08-17T17:04:10.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:10 smithi136 ceph-mon[31034]: Activating manager daemon y 2024-08-17T17:04:10.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:10 smithi136 ceph-mon[31034]: osdmap e3: 0 total, 0 up, 0 in 2024-08-17T17:04:10.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:10 smithi136 ceph-mon[31034]: mgrmap e10: y(active, starting, since 0.00326461s) 2024-08-17T17:04:10.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:10 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:04:10.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:10 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-17T17:04:10.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:10 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-17T17:04:10.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:10 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-17T17:04:10.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:10 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-17T17:04:10.622 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:10 smithi136 ceph-mon[31034]: Manager daemon y is now available 2024-08-17T17:04:10.622 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:10 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:10.622 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:10 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:04:10.622 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:10 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:04:12.141 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout { 2024-08-17T17:04:12.141 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-08-17T17:04:12.141 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout "initialized": true 2024-08-17T17:04:12.142 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout } 2024-08-17T17:04:12.142 INFO:teuthology.orchestra.run.smithi136.stdout:mgr epoch 9 is available 2024-08-17T17:04:12.142 INFO:teuthology.orchestra.run.smithi136.stdout:Generating a dashboard self-signed certificate... 2024-08-17T17:04:12.381 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:12 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:12.048+0000 7f75e09d0640 -1 log_channel(cephadm) log [ERR] : [17/Aug/2024:17:04:12] ENGINE Error in HTTPServer.serve 2024-08-17T17:04:12.382 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:12 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: Traceback (most recent call last): 2024-08-17T17:04:12.382 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:12 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:04:12.382 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:12 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: self._connections.run(self.expiration_interval) 2024-08-17T17:04:12.382 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:12 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:04:12.382 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:12 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: self._run(expiration_interval) 2024-08-17T17:04:12.382 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:12 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:04:12.382 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:12 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:04:12.382 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:12 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:04:12.382 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:12 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:04:12.382 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:12 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:04:12.382 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:12 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: s = self.context.wrap_socket( 2024-08-17T17:04:12.383 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:12 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:04:12.383 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:12 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: return self.sslsocket_class._create( 2024-08-17T17:04:12.383 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:12 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:04:12.383 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:12 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: self.do_handshake() 2024-08-17T17:04:12.383 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:12 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:04:12.383 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:12 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: self._sslobj.do_handshake() 2024-08-17T17:04:12.383 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:12 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:04:12.383 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:04:12 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:04:12.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:12 smithi136 ceph-mon[31034]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-08-17T17:04:12.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:12 smithi136 ceph-mon[31034]: mgrmap e11: y(active, since 1.00562s) 2024-08-17T17:04:12.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:12 smithi136 ceph-mon[31034]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-08-17T17:04:12.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:12 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:13.432 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-08-17T17:04:13.432 INFO:teuthology.orchestra.run.smithi136.stdout:Creating initial admin user... 2024-08-17T17:04:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:13 smithi136 ceph-mon[31034]: [17/Aug/2024:17:04:11] ENGINE Bus STARTING 2024-08-17T17:04:13.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:13 smithi136 ceph-mon[31034]: [17/Aug/2024:17:04:11] ENGINE Serving on http://172.21.15.136:8765 2024-08-17T17:04:13.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:13 smithi136 ceph-mon[31034]: [17/Aug/2024:17:04:12] ENGINE Serving on https://172.21.15.136:7150 2024-08-17T17:04:13.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:13 smithi136 ceph-mon[31034]: [17/Aug/2024:17:04:12] ENGINE Bus STARTED 2024-08-17T17:04:13.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:13 smithi136 ceph-mon[31034]: [17/Aug/2024:17:04:12] ENGINE Error in HTTPServer.serve 2024-08-17T17:04:13.947 INFO:journalctl@ceph.mon.a.smithi136.stdout: Traceback (most recent call last): 2024-08-17T17:04:13.947 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:04:13.947 INFO:journalctl@ceph.mon.a.smithi136.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:04:13.947 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:04:13.947 INFO:journalctl@ceph.mon.a.smithi136.stdout: self._run(expiration_interval) 2024-08-17T17:04:13.947 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:04:13.947 INFO:journalctl@ceph.mon.a.smithi136.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:04:13.948 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:04:13.948 INFO:journalctl@ceph.mon.a.smithi136.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:04:13.948 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:04:13.948 INFO:journalctl@ceph.mon.a.smithi136.stdout: s = self.context.wrap_socket( 2024-08-17T17:04:13.948 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:04:13.948 INFO:journalctl@ceph.mon.a.smithi136.stdout: return self.sslsocket_class._create( 2024-08-17T17:04:13.948 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:04:13.948 INFO:journalctl@ceph.mon.a.smithi136.stdout: self.do_handshake() 2024-08-17T17:04:13.948 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:04:13.948 INFO:journalctl@ceph.mon.a.smithi136.stdout: self._sslobj.do_handshake() 2024-08-17T17:04:13.948 INFO:journalctl@ceph.mon.a.smithi136.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:04:13.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:13 smithi136 ceph-mon[31034]: mgrmap e12: y(active, since 2s) 2024-08-17T17:04:13.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:13 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:13.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:13 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:13.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:13 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:14.789 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:14 smithi136 ceph-mon[31034]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:04:14.790 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:14 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:15.126 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$Y.RhFAFIv7.bJg7Y6e8zPOjqu45ul5MnCYvDup1CGg072kUHFaybS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1723914254, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-08-17T17:04:15.126 INFO:teuthology.orchestra.run.smithi136.stdout:Fetching dashboard port number... 2024-08-17T17:04:15.783 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:15 smithi136 ceph-mon[31034]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:04:16.740 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:16 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/1050084942' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-08-17T17:04:16.743 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stdout 8443 2024-08-17T17:04:16.753 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-17T17:04:16.753 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout disabled 2024-08-17T17:04:16.760 INFO:teuthology.orchestra.run.smithi136.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-17T17:04:16.760 INFO:teuthology.orchestra.run.smithi136.stdout:systemctl: stdout inactive 2024-08-17T17:04:16.760 INFO:teuthology.orchestra.run.smithi136.stdout:firewalld.service is not enabled 2024-08-17T17:04:16.760 INFO:teuthology.orchestra.run.smithi136.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-08-17T17:04:16.762 INFO:teuthology.orchestra.run.smithi136.stdout:Ceph Dashboard is now available at: 2024-08-17T17:04:16.762 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:04:16.762 INFO:teuthology.orchestra.run.smithi136.stdout: URL: https://smithi136.front.sepia.ceph.com:8443/ 2024-08-17T17:04:16.762 INFO:teuthology.orchestra.run.smithi136.stdout: User: admin 2024-08-17T17:04:16.763 INFO:teuthology.orchestra.run.smithi136.stdout: Password: jnofks6cfc 2024-08-17T17:04:16.763 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:04:16.763 INFO:teuthology.orchestra.run.smithi136.stdout:Saving cluster configuration to /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config directory 2024-08-17T17:04:16.763 INFO:teuthology.orchestra.run.smithi136.stdout:Enabling autotune for osd_memory_target 2024-08-17T17:04:17.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:17 smithi136 ceph-mon[31034]: mgrmap e13: y(active, since 6s) 2024-08-17T17:04:19.103 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-08-17T17:04:19.103 INFO:teuthology.orchestra.run.smithi136.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-08-17T17:04:19.103 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:04:19.103 INFO:teuthology.orchestra.run.smithi136.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-08-17T17:04:19.103 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:04:19.103 INFO:teuthology.orchestra.run.smithi136.stdout:Or, if you are only running a single cluster on this host: 2024-08-17T17:04:19.103 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:04:19.103 INFO:teuthology.orchestra.run.smithi136.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-08-17T17:04:19.103 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:04:19.103 INFO:teuthology.orchestra.run.smithi136.stdout:Please consider enabling telemetry to help improve Ceph: 2024-08-17T17:04:19.103 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:04:19.104 INFO:teuthology.orchestra.run.smithi136.stdout: ceph telemetry on 2024-08-17T17:04:19.104 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:04:19.104 INFO:teuthology.orchestra.run.smithi136.stdout:For more information see: 2024-08-17T17:04:19.104 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:04:19.104 INFO:teuthology.orchestra.run.smithi136.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-08-17T17:04:19.104 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:04:19.104 INFO:teuthology.orchestra.run.smithi136.stdout:Bootstrap complete. 2024-08-17T17:04:19.134 INFO:tasks.cephadm:Fetching config... 2024-08-17T17:04:19.135 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-17T17:04:19.135 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-08-17T17:04:19.151 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-08-17T17:04:19.151 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-17T17:04:19.151 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-08-17T17:04:19.209 INFO:tasks.cephadm:Fetching mon keyring... 2024-08-17T17:04:19.209 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-17T17:04:19.209 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/keyring of=/dev/stdout 2024-08-17T17:04:19.274 INFO:tasks.cephadm:Fetching pub ssh key... 2024-08-17T17:04:19.274 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-17T17:04:19.274 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-08-17T17:04:19.329 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-08-17T17:04:19.329 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzoF5/ZXVutd9+by8TD8PUjNlDoGEh3vG8qpPoxjBeHnUw32eLt7DZ9XgTWm6zwCULiuxWMnQ3GaHzvn8rkRfnwPtKLg63Cm04wm25KwcKXSCUOixVFXySPMBSImEudSu0UwATAwN8Th4WrwskcesxRnoakGnyTWM+h62MMJez3vUzYGx63ALLYi1bXu5/GP/YeCJFrPaaWYIuZpEOdBipH6CECuMK0FSc97ZI0d71nJx8gNv5sRzDeeNnAUKdaEBMokg75eZuV8p89mKDXDrnMHeeeeU0IVrxrSfbx22XLAvRNaiPD8ZxpmzOM0SG+z2CeeO0vfEFDPkCg6+bvEFEIsRm5Oga1i+7SQJWXAzQHvdpU8UKz3C87RmhIPXrYWhNvtTY/T2pTSZLjyN1h/lJ9uaQxMwzXPn5Y/QzhhT/YJ5yHeogm4+T5UxWTtGUJT0yvTkwRe02WBvF3CaHUmJF0dmlQz9u+H8V60XydToE85HAz/p5054pUtxIwFUMujk= ceph-8270436c-5cba-11ef-bccf-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-08-17T17:04:19.403 INFO:teuthology.orchestra.run.smithi136.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzoF5/ZXVutd9+by8TD8PUjNlDoGEh3vG8qpPoxjBeHnUw32eLt7DZ9XgTWm6zwCULiuxWMnQ3GaHzvn8rkRfnwPtKLg63Cm04wm25KwcKXSCUOixVFXySPMBSImEudSu0UwATAwN8Th4WrwskcesxRnoakGnyTWM+h62MMJez3vUzYGx63ALLYi1bXu5/GP/YeCJFrPaaWYIuZpEOdBipH6CECuMK0FSc97ZI0d71nJx8gNv5sRzDeeNnAUKdaEBMokg75eZuV8p89mKDXDrnMHeeeeU0IVrxrSfbx22XLAvRNaiPD8ZxpmzOM0SG+z2CeeO0vfEFDPkCg6+bvEFEIsRm5Oga1i+7SQJWXAzQHvdpU8UKz3C87RmhIPXrYWhNvtTY/T2pTSZLjyN1h/lJ9uaQxMwzXPn5Y/QzhhT/YJ5yHeogm4+T5UxWTtGUJT0yvTkwRe02WBvF3CaHUmJF0dmlQz9u+H8V60XydToE85HAz/p5054pUtxIwFUMujk= ceph-8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:04:19.423 DEBUG:teuthology.orchestra.run.smithi165:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzoF5/ZXVutd9+by8TD8PUjNlDoGEh3vG8qpPoxjBeHnUw32eLt7DZ9XgTWm6zwCULiuxWMnQ3GaHzvn8rkRfnwPtKLg63Cm04wm25KwcKXSCUOixVFXySPMBSImEudSu0UwATAwN8Th4WrwskcesxRnoakGnyTWM+h62MMJez3vUzYGx63ALLYi1bXu5/GP/YeCJFrPaaWYIuZpEOdBipH6CECuMK0FSc97ZI0d71nJx8gNv5sRzDeeNnAUKdaEBMokg75eZuV8p89mKDXDrnMHeeeeU0IVrxrSfbx22XLAvRNaiPD8ZxpmzOM0SG+z2CeeO0vfEFDPkCg6+bvEFEIsRm5Oga1i+7SQJWXAzQHvdpU8UKz3C87RmhIPXrYWhNvtTY/T2pTSZLjyN1h/lJ9uaQxMwzXPn5Y/QzhhT/YJ5yHeogm4+T5UxWTtGUJT0yvTkwRe02WBvF3CaHUmJF0dmlQz9u+H8V60XydToE85HAz/p5054pUtxIwFUMujk= ceph-8270436c-5cba-11ef-bccf-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-08-17T17:04:19.465 INFO:teuthology.orchestra.run.smithi165.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzoF5/ZXVutd9+by8TD8PUjNlDoGEh3vG8qpPoxjBeHnUw32eLt7DZ9XgTWm6zwCULiuxWMnQ3GaHzvn8rkRfnwPtKLg63Cm04wm25KwcKXSCUOixVFXySPMBSImEudSu0UwATAwN8Th4WrwskcesxRnoakGnyTWM+h62MMJez3vUzYGx63ALLYi1bXu5/GP/YeCJFrPaaWYIuZpEOdBipH6CECuMK0FSc97ZI0d71nJx8gNv5sRzDeeNnAUKdaEBMokg75eZuV8p89mKDXDrnMHeeeeU0IVrxrSfbx22XLAvRNaiPD8ZxpmzOM0SG+z2CeeO0vfEFDPkCg6+bvEFEIsRm5Oga1i+7SQJWXAzQHvdpU8UKz3C87RmhIPXrYWhNvtTY/T2pTSZLjyN1h/lJ9uaQxMwzXPn5Y/QzhhT/YJ5yHeogm4+T5UxWTtGUJT0yvTkwRe02WBvF3CaHUmJF0dmlQz9u+H8V60XydToE85HAz/p5054pUtxIwFUMujk= ceph-8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:04:19.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-08-17T17:04:19.678 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:04:19.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:19 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/2919176962' entity='client.admin' 2024-08-17T17:04:22.225 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-08-17T17:04:22.225 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-08-17T17:04:22.740 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:22 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/1262508384' entity='client.admin' 2024-08-17T17:04:22.854 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:04:25.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi136", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:25.989 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi165 2024-08-17T17:04:25.989 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-08-17T17:04:25.989 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/etc/ceph/ceph.conf 2024-08-17T17:04:26.007 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-08-17T17:04:26.008 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:04:26.066 INFO:tasks.cephadm:Adding host smithi165 to orchestrator... 2024-08-17T17:04:26.066 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph orch host add smithi165 2024-08-17T17:04:26.271 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:04:26.677 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:26 smithi136 ceph-mon[31034]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:04:26.677 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:26 smithi136 ceph-mon[31034]: Updating smithi136:/etc/ceph/ceph.conf 2024-08-17T17:04:26.677 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:26 smithi136 ceph-mon[31034]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:04:26.677 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:26 smithi136 ceph-mon[31034]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:04:26.677 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:26 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:26.677 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:26 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:26.677 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:26 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:27.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:27 smithi136 ceph-mon[31034]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:04:29.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:28 smithi136 ceph-mon[31034]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi165", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:04:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:29 smithi136 ceph-mon[31034]: Deploying cephadm binary to smithi165 2024-08-17T17:04:30.655 INFO:teuthology.orchestra.run.smithi136.stdout:Added host 'smithi165' with addr '172.21.15.165' 2024-08-17T17:04:31.191 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph orch host ls --format=json 2024-08-17T17:04:31.387 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:04:31.773 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:31 smithi136 ceph-mon[31034]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:31.773 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:31 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:31.773 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:31 smithi136 ceph-mon[31034]: Added host smithi165 2024-08-17T17:04:31.773 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:31 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:31.773 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:31 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:31.774 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:31 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:32.987 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:04:32.987 INFO:teuthology.orchestra.run.smithi136.stdout:[{"addr": "172.21.15.136", "hostname": "smithi136", "labels": [], "status": ""}, {"addr": "172.21.15.165", "hostname": "smithi165", "labels": [], "status": ""}] 2024-08-17T17:04:32.987 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:32 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:32.987 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:32 smithi136 ceph-mon[31034]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:33.537 INFO:tasks.cephadm:Setting crush tunables to default 2024-08-17T17:04:33.537 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd crush tunables default 2024-08-17T17:04:33.730 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:04:34.111 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:33 smithi136 ceph-mon[31034]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:04:35.103 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:34 smithi136 ceph-mon[31034]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:35.103 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:34 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:35.103 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:34 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:35.103 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:34 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:35.103 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:34 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:35.103 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:34 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:35.103 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:34 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:35.103 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:34 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:35.969 INFO:teuthology.orchestra.run.smithi136.stderr:adjusted tunables profile to default 2024-08-17T17:04:36.271 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:35 smithi136 ceph-mon[31034]: Updating smithi165:/etc/ceph/ceph.conf 2024-08-17T17:04:36.271 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:35 smithi136 ceph-mon[31034]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:04:36.271 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:35 smithi136 ceph-mon[31034]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:04:36.272 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:35 smithi136 ceph-mon[31034]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:04:36.272 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:35 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/3130059668' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-08-17T17:04:36.272 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:35 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:36.272 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:35 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:36.272 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:35 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:36.586 INFO:tasks.cephadm:Adding mon.a on smithi136 2024-08-17T17:04:36.586 INFO:tasks.cephadm:Adding mon.c on smithi136 2024-08-17T17:04:36.586 INFO:tasks.cephadm:Adding mon.b on smithi165 2024-08-17T17:04:36.587 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph orch apply mon '3;smithi136:172.21.15.136=a;smithi136:[v2:172.21.15.136:3301,v1:172.21.15.136:6790]=c;smithi165:172.21.15.165=b' 2024-08-17T17:04:36.791 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:04:37.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:36 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/3130059668' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-08-17T17:04:37.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:36 smithi136 ceph-mon[31034]: osdmap e4: 0 total, 0 up, 0 in 2024-08-17T17:04:37.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:36 smithi136 ceph-mon[31034]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:38.637 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled mon update... 2024-08-17T17:04:39.251 DEBUG:teuthology.orchestra.run.smithi136:mon.c> sudo journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@mon.c.service 2024-08-17T17:04:39.255 DEBUG:teuthology.orchestra.run.smithi165:mon.b> sudo journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@mon.b.service 2024-08-17T17:04:39.257 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-17T17:04:39.257 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph mon dump -f json 2024-08-17T17:04:39.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:39 smithi136 ceph-mon[31034]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:39.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:39 smithi136 ceph-mon[31034]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi136:172.21.15.136=a;smithi136:[v2:172.21.15.136:3301,v1:172.21.15.136:6790]=c;smithi165:172.21.15.165=b", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:04:39.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:39 smithi136 ceph-mon[31034]: Saving service mon spec with placement smithi136:172.21.15.136=a;smithi136:[v2:172.21.15.136:3301,v1:172.21.15.136:6790]=c;smithi165:172.21.15.165=b;count:3 2024-08-17T17:04:39.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:39 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:39.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:39 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:39.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:39 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:39.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:39 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:39.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:39 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:39.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:39 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:04:39.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:39 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:39.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:39 smithi136 ceph-mon[31034]: Deploying daemon mon.b on smithi165 2024-08-17T17:04:39.953 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:04:41.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:41 smithi136 ceph-mon[31034]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:43.135 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:04:43.135 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":1,"fsid":"8270436c-5cba-11ef-bccf-c7b262605968","modified":"2024-08-17T17:03:14.749093Z","created":"2024-08-17T17:03:14.749093Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-08-17T17:04:43.136 INFO:teuthology.orchestra.run.smithi165.stderr:dumped monmap epoch 1 2024-08-17T17:04:43.454 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:43 smithi165 systemd[1]: Starting Ceph mon.b for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:04:43.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:43 smithi136 ceph-mon[31034]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:43.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:43 smithi136 ceph-mon[31034]: from='client.? 172.21.15.165:0/1747041679' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-17T17:04:43.974 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:43 smithi165 podman[34830]: 2024-08-17 17:04:43.86299675 +0000 UTC m=+0.106895359 container create d587e16140e1c5633365c903f3f83b845de47b3221003b11b60d47804bd4a276 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=reef-ced649f, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240731) 2024-08-17T17:04:43.974 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:43 smithi165 podman[34830]: 2024-08-17 17:04:43.773468937 +0000 UTC m=+0.017367550 image pull c870753d67c2504e5ac044c83696772f8b7527a928cc764ed24eeea7fea57d6e quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408 2024-08-17T17:04:44.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 podman[34830]: 2024-08-17 17:04:44.048307501 +0000 UTC m=+0.292206128 container init d587e16140e1c5633365c903f3f83b845de47b3221003b11b60d47804bd4a276 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-b, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, RELEASE=reef-ced649f, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T17:04:44.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 podman[34830]: 2024-08-17 17:04:44.05289942 +0000 UTC m=+0.296798031 container start d587e16140e1c5633365c903f3f83b845de47b3221003b11b60d47804bd4a276 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=reef-ced649f, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-08-17T17:04:44.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: set uid:gid to 167:167 (ceph:ceph) 2024-08-17T17:04:44.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable), process ceph-mon, pid 7 2024-08-17T17:04:44.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: pidfile_write: ignore empty --pid-file 2024-08-17T17:04:44.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: load: jerasure load: lrc 2024-08-17T17:04:44.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: RocksDB version: 7.9.2 2024-08-17T17:04:44.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Git sha 0 2024-08-17T17:04:44.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Compile date 2024-08-17 15:23:20 2024-08-17T17:04:44.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: DB SUMMARY 2024-08-17T17:04:44.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: DB Session ID: PGJASQ3H277QAGCHEQ4H 2024-08-17T17:04:44.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: CURRENT file: CURRENT 2024-08-17T17:04:44.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: IDENTITY file: IDENTITY 2024-08-17T17:04:44.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-08-17T17:04:44.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-08-17T17:04:44.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-08-17T17:04:44.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.error_if_exists: 0 2024-08-17T17:04:44.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.create_if_missing: 0 2024-08-17T17:04:44.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.paranoid_checks: 1 2024-08-17T17:04:44.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.flush_verify_memtable_count: 1 2024-08-17T17:04:44.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-08-17T17:04:44.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-08-17T17:04:44.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.env: 0x559316441bc0 2024-08-17T17:04:44.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.fs: PosixFileSystem 2024-08-17T17:04:44.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.info_log: 0x559318aa2be0 2024-08-17T17:04:44.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_file_opening_threads: 16 2024-08-17T17:04:44.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.statistics: (nil) 2024-08-17T17:04:44.515 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.use_fsync: 0 2024-08-17T17:04:44.515 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_log_file_size: 0 2024-08-17T17:04:44.515 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-08-17T17:04:44.515 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.log_file_time_to_roll: 0 2024-08-17T17:04:44.515 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.keep_log_file_num: 1000 2024-08-17T17:04:44.515 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.recycle_log_file_num: 0 2024-08-17T17:04:44.515 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.allow_fallocate: 1 2024-08-17T17:04:44.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.allow_mmap_reads: 0 2024-08-17T17:04:44.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.allow_mmap_writes: 0 2024-08-17T17:04:44.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.use_direct_reads: 0 2024-08-17T17:04:44.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-08-17T17:04:44.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.create_missing_column_families: 0 2024-08-17T17:04:44.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.db_log_dir: 2024-08-17T17:04:44.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.wal_dir: 2024-08-17T17:04:44.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.table_cache_numshardbits: 6 2024-08-17T17:04:44.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.WAL_ttl_seconds: 0 2024-08-17T17:04:44.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.WAL_size_limit_MB: 0 2024-08-17T17:04:44.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-08-17T17:04:44.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-08-17T17:04:44.517 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.is_fd_close_on_exec: 1 2024-08-17T17:04:44.517 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.advise_random_on_open: 1 2024-08-17T17:04:44.517 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.db_write_buffer_size: 0 2024-08-17T17:04:44.517 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.write_buffer_manager: 0x559318a72140 2024-08-17T17:04:44.517 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-08-17T17:04:44.517 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-08-17T17:04:44.517 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.use_adaptive_mutex: 0 2024-08-17T17:04:44.517 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.rate_limiter: (nil) 2024-08-17T17:04:44.517 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-08-17T17:04:44.517 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.wal_recovery_mode: 2 2024-08-17T17:04:44.517 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.enable_thread_tracking: 0 2024-08-17T17:04:44.517 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.enable_pipelined_write: 0 2024-08-17T17:04:44.517 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.unordered_write: 0 2024-08-17T17:04:44.517 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-08-17T17:04:44.518 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-08-17T17:04:44.518 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-08-17T17:04:44.518 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-08-17T17:04:44.518 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.row_cache: None 2024-08-17T17:04:44.518 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.wal_filter: None 2024-08-17T17:04:44.518 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-08-17T17:04:44.518 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.allow_ingest_behind: 0 2024-08-17T17:04:44.518 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.two_write_queues: 0 2024-08-17T17:04:44.518 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.manual_wal_flush: 0 2024-08-17T17:04:44.518 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.wal_compression: 0 2024-08-17T17:04:44.518 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.atomic_flush: 0 2024-08-17T17:04:44.518 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-08-17T17:04:44.518 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.persist_stats_to_disk: 0 2024-08-17T17:04:44.518 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.write_dbid_to_manifest: 0 2024-08-17T17:04:44.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.log_readahead_size: 0 2024-08-17T17:04:44.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-08-17T17:04:44.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.best_efforts_recovery: 0 2024-08-17T17:04:44.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-08-17T17:04:44.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-08-17T17:04:44.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.allow_data_in_errors: 0 2024-08-17T17:04:44.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.db_host_id: __hostname__ 2024-08-17T17:04:44.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.enforce_single_del_contracts: true 2024-08-17T17:04:44.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_background_jobs: 2 2024-08-17T17:04:44.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_background_compactions: -1 2024-08-17T17:04:44.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_subcompactions: 1 2024-08-17T17:04:44.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-08-17T17:04:44.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-08-17T17:04:44.520 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.delayed_write_rate : 16777216 2024-08-17T17:04:44.520 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_total_wal_size: 0 2024-08-17T17:04:44.520 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-08-17T17:04:44.520 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.stats_dump_period_sec: 600 2024-08-17T17:04:44.520 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.stats_persist_period_sec: 600 2024-08-17T17:04:44.520 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-08-17T17:04:44.520 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_open_files: -1 2024-08-17T17:04:44.520 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.bytes_per_sync: 0 2024-08-17T17:04:44.520 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.wal_bytes_per_sync: 0 2024-08-17T17:04:44.520 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.strict_bytes_per_sync: 0 2024-08-17T17:04:44.520 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.compaction_readahead_size: 0 2024-08-17T17:04:44.520 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_background_flushes: -1 2024-08-17T17:04:44.520 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Compression algorithms supported: 2024-08-17T17:04:44.520 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: kZSTD supported: 0 2024-08-17T17:04:44.520 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: kXpressCompression supported: 0 2024-08-17T17:04:44.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: kBZip2Compression supported: 0 2024-08-17T17:04:44.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-08-17T17:04:44.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: kLZ4Compression supported: 1 2024-08-17T17:04:44.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: kZlibCompression supported: 1 2024-08-17T17:04:44.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: kLZ4HCCompression supported: 1 2024-08-17T17:04:44.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: kSnappyCompression supported: 1 2024-08-17T17:04:44.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Fast CRC32 supported: Supported on x86 2024-08-17T17:04:44.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: DMutex implementation: pthread_mutex_t 2024-08-17T17:04:44.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-08-17T17:04:44.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-08-17T17:04:44.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-08-17T17:04:44.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.merge_operator: 2024-08-17T17:04:44.522 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.compaction_filter: None 2024-08-17T17:04:44.522 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.compaction_filter_factory: None 2024-08-17T17:04:44.522 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.sst_partitioner_factory: None 2024-08-17T17:04:44.522 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.memtable_factory: SkipListFactory 2024-08-17T17:04:44.522 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.table_factory: BlockBasedTable 2024-08-17T17:04:44.522 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559318aa2820) 2024-08-17T17:04:44.522 INFO:journalctl@ceph.mon.b.smithi165.stdout: cache_index_and_filter_blocks: 1 2024-08-17T17:04:44.522 INFO:journalctl@ceph.mon.b.smithi165.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-08-17T17:04:44.522 INFO:journalctl@ceph.mon.b.smithi165.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-08-17T17:04:44.522 INFO:journalctl@ceph.mon.b.smithi165.stdout: pin_top_level_index_and_filter: 1 2024-08-17T17:04:44.522 INFO:journalctl@ceph.mon.b.smithi165.stdout: index_type: 0 2024-08-17T17:04:44.522 INFO:journalctl@ceph.mon.b.smithi165.stdout: data_block_index_type: 0 2024-08-17T17:04:44.522 INFO:journalctl@ceph.mon.b.smithi165.stdout: index_shortening: 1 2024-08-17T17:04:44.523 INFO:journalctl@ceph.mon.b.smithi165.stdout: data_block_hash_table_util_ratio: 0.750000 2024-08-17T17:04:44.523 INFO:journalctl@ceph.mon.b.smithi165.stdout: checksum: 4 2024-08-17T17:04:44.523 INFO:journalctl@ceph.mon.b.smithi165.stdout: no_block_cache: 0 2024-08-17T17:04:44.523 INFO:journalctl@ceph.mon.b.smithi165.stdout: block_cache: 0x559318a97350 2024-08-17T17:04:44.523 INFO:journalctl@ceph.mon.b.smithi165.stdout: block_cache_name: BinnedLRUCache 2024-08-17T17:04:44.523 INFO:journalctl@ceph.mon.b.smithi165.stdout: block_cache_options: 2024-08-17T17:04:44.523 INFO:journalctl@ceph.mon.b.smithi165.stdout: capacity : 536870912 2024-08-17T17:04:44.523 INFO:journalctl@ceph.mon.b.smithi165.stdout: num_shard_bits : 4 2024-08-17T17:04:44.523 INFO:journalctl@ceph.mon.b.smithi165.stdout: strict_capacity_limit : 0 2024-08-17T17:04:44.523 INFO:journalctl@ceph.mon.b.smithi165.stdout: high_pri_pool_ratio: 0.000 2024-08-17T17:04:44.523 INFO:journalctl@ceph.mon.b.smithi165.stdout: block_cache_compressed: (nil) 2024-08-17T17:04:44.523 INFO:journalctl@ceph.mon.b.smithi165.stdout: persistent_cache: (nil) 2024-08-17T17:04:44.523 INFO:journalctl@ceph.mon.b.smithi165.stdout: block_size: 4096 2024-08-17T17:04:44.524 INFO:journalctl@ceph.mon.b.smithi165.stdout: block_size_deviation: 10 2024-08-17T17:04:44.524 INFO:journalctl@ceph.mon.b.smithi165.stdout: block_restart_interval: 16 2024-08-17T17:04:44.524 INFO:journalctl@ceph.mon.b.smithi165.stdout: index_block_restart_interval: 1 2024-08-17T17:04:44.524 INFO:journalctl@ceph.mon.b.smithi165.stdout: metadata_block_size: 4096 2024-08-17T17:04:44.524 INFO:journalctl@ceph.mon.b.smithi165.stdout: partition_filters: 0 2024-08-17T17:04:44.524 INFO:journalctl@ceph.mon.b.smithi165.stdout: use_delta_encoding: 1 2024-08-17T17:04:44.524 INFO:journalctl@ceph.mon.b.smithi165.stdout: filter_policy: bloomfilter 2024-08-17T17:04:44.524 INFO:journalctl@ceph.mon.b.smithi165.stdout: whole_key_filtering: 1 2024-08-17T17:04:44.524 INFO:journalctl@ceph.mon.b.smithi165.stdout: verify_compression: 0 2024-08-17T17:04:44.524 INFO:journalctl@ceph.mon.b.smithi165.stdout: read_amp_bytes_per_bit: 0 2024-08-17T17:04:44.524 INFO:journalctl@ceph.mon.b.smithi165.stdout: format_version: 5 2024-08-17T17:04:44.524 INFO:journalctl@ceph.mon.b.smithi165.stdout: enable_index_compression: 1 2024-08-17T17:04:44.524 INFO:journalctl@ceph.mon.b.smithi165.stdout: block_align: 0 2024-08-17T17:04:44.525 INFO:journalctl@ceph.mon.b.smithi165.stdout: max_auto_readahead_size: 262144 2024-08-17T17:04:44.525 INFO:journalctl@ceph.mon.b.smithi165.stdout: prepopulate_block_cache: 0 2024-08-17T17:04:44.525 INFO:journalctl@ceph.mon.b.smithi165.stdout: initial_auto_readahead_size: 8192 2024-08-17T17:04:44.525 INFO:journalctl@ceph.mon.b.smithi165.stdout: num_file_reads_for_auto_readahead: 2 2024-08-17T17:04:44.525 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.write_buffer_size: 33554432 2024-08-17T17:04:44.525 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_write_buffer_number: 2 2024-08-17T17:04:44.525 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.compression: NoCompression 2024-08-17T17:04:44.525 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.bottommost_compression: Disabled 2024-08-17T17:04:44.525 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.prefix_extractor: nullptr 2024-08-17T17:04:44.525 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-08-17T17:04:44.525 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.num_levels: 7 2024-08-17T17:04:44.525 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-08-17T17:04:44.525 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-08-17T17:04:44.525 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-08-17T17:04:44.526 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-08-17T17:04:44.526 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-08-17T17:04:44.526 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-08-17T17:04:44.526 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-08-17T17:04:44.526 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-08-17T17:04:44.526 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-08-17T17:04:44.526 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-08-17T17:04:44.526 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-08-17T17:04:44.526 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-08-17T17:04:44.526 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.compression_opts.window_bits: -14 2024-08-17T17:04:44.526 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.compression_opts.level: 32767 2024-08-17T17:04:44.526 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.compression_opts.strategy: 0 2024-08-17T17:04:44.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-08-17T17:04:44.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-08-17T17:04:44.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-08-17T17:04:44.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-08-17T17:04:44.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.compression_opts.enabled: false 2024-08-17T17:04:44.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-08-17T17:04:44.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-08-17T17:04:44.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-08-17T17:04:44.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-08-17T17:04:44.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.target_file_size_base: 67108864 2024-08-17T17:04:44.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.target_file_size_multiplier: 1 2024-08-17T17:04:44.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-08-17T17:04:44.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-08-17T17:04:44.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-08-17T17:04:44.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-08-17T17:04:44.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-08-17T17:04:44.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-08-17T17:04:44.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-08-17T17:04:44.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-08-17T17:04:44.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-08-17T17:04:44.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-08-17T17:04:44.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-08-17T17:04:44.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-08-17T17:04:44.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-08-17T17:04:44.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.arena_block_size: 1048576 2024-08-17T17:04:44.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-08-17T17:04:44.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-08-17T17:04:44.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.disable_auto_compactions: 0 2024-08-17T17:04:44.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-08-17T17:04:44.529 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-08-17T17:04:44.529 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-08-17T17:04:44.529 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-08-17T17:04:44.529 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-08-17T17:04:44.529 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-08-17T17:04:44.529 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-08-17T17:04:44.529 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-08-17T17:04:44.529 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-08-17T17:04:44.529 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-08-17T17:04:44.529 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.table_properties_collectors: 2024-08-17T17:04:44.529 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.inplace_update_support: 0 2024-08-17T17:04:44.529 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.inplace_update_num_locks: 10000 2024-08-17T17:04:44.529 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-08-17T17:04:44.530 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-08-17T17:04:44.530 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.memtable_huge_page_size: 0 2024-08-17T17:04:44.530 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.bloom_locality: 0 2024-08-17T17:04:44.530 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.max_successive_merges: 0 2024-08-17T17:04:44.530 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.optimize_filters_for_hits: 0 2024-08-17T17:04:44.530 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.paranoid_file_checks: 0 2024-08-17T17:04:44.530 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.force_consistency_checks: 1 2024-08-17T17:04:44.530 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.report_bg_io_stats: 0 2024-08-17T17:04:44.530 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.ttl: 2592000 2024-08-17T17:04:44.530 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.periodic_compaction_seconds: 0 2024-08-17T17:04:44.530 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-08-17T17:04:44.530 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-08-17T17:04:44.531 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.enable_blob_files: false 2024-08-17T17:04:44.531 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.min_blob_size: 0 2024-08-17T17:04:44.531 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.blob_file_size: 268435456 2024-08-17T17:04:44.531 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.blob_compression_type: NoCompression 2024-08-17T17:04:44.531 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.enable_blob_garbage_collection: false 2024-08-17T17:04:44.531 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-08-17T17:04:44.531 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-08-17T17:04:44.531 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-08-17T17:04:44.531 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.blob_file_starting_level: 0 2024-08-17T17:04:44.531 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-08-17T17:04:44.531 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-08-17T17:04:44.531 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-08-17T17:04:44.531 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9c1e8a30-33ca-4b9e-a900-a08d3297168b 2024-08-17T17:04:44.532 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914284079913, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-08-17T17:04:44.532 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-08-17T17:04:44.532 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914284080355, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1723914284, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9c1e8a30-33ca-4b9e-a900-a08d3297168b", "db_session_id": "PGJASQ3H277QAGCHEQ4H", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-08-17T17:04:44.532 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914284080449, "job": 1, "event": "recovery_finished"} 2024-08-17T17:04:44.532 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-08-17T17:04:44.532 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-08-17T17:04:44.532 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559318ac0e00 2024-08-17T17:04:44.532 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: DB pointer 0x559318bb4000 2024-08-17T17:04:44.532 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-08-17T17:04:44.532 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-08-17T17:04:44.532 INFO:journalctl@ceph.mon.b.smithi165.stdout: ** DB Stats ** 2024-08-17T17:04:44.532 INFO:journalctl@ceph.mon.b.smithi165.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-17T17:04:44.532 INFO:journalctl@ceph.mon.b.smithi165.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-08-17T17:04:44.533 INFO:journalctl@ceph.mon.b.smithi165.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-17T17:04:44.533 INFO:journalctl@ceph.mon.b.smithi165.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-17T17:04:44.533 INFO:journalctl@ceph.mon.b.smithi165.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-08-17T17:04:44.533 INFO:journalctl@ceph.mon.b.smithi165.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-17T17:04:44.533 INFO:journalctl@ceph.mon.b.smithi165.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-17T17:04:44.533 INFO:journalctl@ceph.mon.b.smithi165.stdout: 2024-08-17T17:04:44.533 INFO:journalctl@ceph.mon.b.smithi165.stdout: ** Compaction Stats [default] ** 2024-08-17T17:04:44.533 INFO:journalctl@ceph.mon.b.smithi165.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-17T17:04:44.533 INFO:journalctl@ceph.mon.b.smithi165.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-08-17T17:04:44.533 INFO:journalctl@ceph.mon.b.smithi165.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-08-17T17:04:44.533 INFO:journalctl@ceph.mon.b.smithi165.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-08-17T17:04:44.534 INFO:journalctl@ceph.mon.b.smithi165.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-08-17T17:04:44.534 INFO:journalctl@ceph.mon.b.smithi165.stdout: 2024-08-17T17:04:44.534 INFO:journalctl@ceph.mon.b.smithi165.stdout: ** Compaction Stats [default] ** 2024-08-17T17:04:44.534 INFO:journalctl@ceph.mon.b.smithi165.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-17T17:04:44.534 INFO:journalctl@ceph.mon.b.smithi165.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-17T17:04:44.534 INFO:journalctl@ceph.mon.b.smithi165.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-08-17T17:04:44.534 INFO:journalctl@ceph.mon.b.smithi165.stdout: 2024-08-17T17:04:44.534 INFO:journalctl@ceph.mon.b.smithi165.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-08-17T17:04:44.534 INFO:journalctl@ceph.mon.b.smithi165.stdout: 2024-08-17T17:04:44.534 INFO:journalctl@ceph.mon.b.smithi165.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-17T17:04:44.534 INFO:journalctl@ceph.mon.b.smithi165.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-08-17T17:04:44.534 INFO:journalctl@ceph.mon.b.smithi165.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-08-17T17:04:44.534 INFO:journalctl@ceph.mon.b.smithi165.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-08-17T17:04:44.534 INFO:journalctl@ceph.mon.b.smithi165.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-08-17T17:04:44.534 INFO:journalctl@ceph.mon.b.smithi165.stdout: AddFile(Keys): cumulative 0, interval 0 2024-08-17T17:04:44.535 INFO:journalctl@ceph.mon.b.smithi165.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:04:44.535 INFO:journalctl@ceph.mon.b.smithi165.stdout: Interval compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:04:44.535 INFO:journalctl@ceph.mon.b.smithi165.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-08-17T17:04:44.535 INFO:journalctl@ceph.mon.b.smithi165.stdout: Block cache BinnedLRUCache@0x559318a97350#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-08-17T17:04:44.535 INFO:journalctl@ceph.mon.b.smithi165.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-08-17T17:04:44.535 INFO:journalctl@ceph.mon.b.smithi165.stdout: 2024-08-17T17:04:44.535 INFO:journalctl@ceph.mon.b.smithi165.stdout: ** File Read Latency Histogram By Level [default] ** 2024-08-17T17:04:44.535 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-08-17T17:04:44.535 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: using public_addr v2:172.21.15.165:0/0 -> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-08-17T17:04:44.535 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: starting mon.b rank -1 at public addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] at bind addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:04:44.535 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: mon.b@-1(???) e0 preinit fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:04:44.535 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: mon.b@-1(synchronizing).mds e1 new map 2024-08-17T17:04:44.535 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: mon.b@-1(synchronizing).mds e1 print_map 2024-08-17T17:04:44.535 INFO:journalctl@ceph.mon.b.smithi165.stdout: e1 2024-08-17T17:04:44.535 INFO:journalctl@ceph.mon.b.smithi165.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-08-17T17:04:44.536 INFO:journalctl@ceph.mon.b.smithi165.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-08-17T17:04:44.536 INFO:journalctl@ceph.mon.b.smithi165.stdout: legacy client fscid: -1 2024-08-17T17:04:44.536 INFO:journalctl@ceph.mon.b.smithi165.stdout: 2024-08-17T17:04:44.536 INFO:journalctl@ceph.mon.b.smithi165.stdout: No filesystems configured 2024-08-17T17:04:44.536 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-08-17T17:04:44.536 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-08-17T17:04:44.536 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-08-17T17:04:44.536 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-08-17T17:04:44.536 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-08-17T17:04:44.536 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-08-17T17:04:44.536 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-08-17T17:04:44.536 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-08-17T17:04:44.536 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-08-17T17:04:44.536 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-08-17T17:04:44.536 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: Updating smithi165:/etc/ceph/ceph.conf 2024-08-17T17:04:44.537 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:04:44.537 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:04:44.537 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:04:44.537 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/3130059668' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-08-17T17:04:44.537 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:44.537 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:44.537 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:44.537 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/3130059668' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-08-17T17:04:44.537 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: osdmap e4: 0 total, 0 up, 0 in 2024-08-17T17:04:44.537 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:44.537 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:44.538 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi136:172.21.15.136=a;smithi136:[v2:172.21.15.136:3301,v1:172.21.15.136:6790]=c;smithi165:172.21.15.165=b", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:04:44.538 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: Saving service mon spec with placement smithi136:172.21.15.136=a;smithi136:[v2:172.21.15.136:3301,v1:172.21.15.136:6790]=c;smithi165:172.21.15.165=b;count:3 2024-08-17T17:04:44.538 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:44.538 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:44.538 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:44.538 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:44.538 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:44.538 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:04:44.538 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:44.538 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: Deploying daemon mon.b on smithi165 2024-08-17T17:04:44.538 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:44.538 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:44.538 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: from='client.? 172.21.15.165:0/1747041679' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-17T17:04:44.538 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 ceph-mon[34868]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-08-17T17:04:44.538 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 bash[34830]: d587e16140e1c5633365c903f3f83b845de47b3221003b11b60d47804bd4a276 2024-08-17T17:04:44.539 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:44 smithi165 systemd[1]: Started Ceph mon.b for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:04:44.739 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-17T17:04:44.739 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph mon dump -f json 2024-08-17T17:04:44.937 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:04:49.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:49 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:04:49.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:49 smithi136 ceph-mon[31034]: mon.a calling monitor election 2024-08-17T17:04:49.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:49 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:49.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:49 smithi136 ceph-mon[31034]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:49.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:49 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:49.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:49 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:49.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:49 smithi136 ceph-mon[31034]: mon.b calling monitor election 2024-08-17T17:04:49.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:49 smithi136 ceph-mon[31034]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:49.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:49 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:49.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:49 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:49.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:49 smithi136 ceph-mon[31034]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:49.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:49 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:49.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:49 smithi136 ceph-mon[31034]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-08-17T17:04:49.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:49 smithi136 ceph-mon[31034]: monmap e2: 2 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],b=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-08-17T17:04:49.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:49 smithi136 ceph-mon[31034]: fsmap 2024-08-17T17:04:49.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:49 smithi136 ceph-mon[31034]: osdmap e4: 0 total, 0 up, 0 in 2024-08-17T17:04:49.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:49 smithi136 ceph-mon[31034]: mgrmap e13: y(active, since 38s) 2024-08-17T17:04:49.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:49 smithi136 ceph-mon[31034]: overall HEALTH_OK 2024-08-17T17:04:49.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:49 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:49.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:49 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:49.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:49 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:49.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:49 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:04:49.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:49 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:49.465 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:49 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:04:49.465 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:49 smithi165 ceph-mon[34868]: mon.a calling monitor election 2024-08-17T17:04:49.465 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:49 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:49.465 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:49 smithi165 ceph-mon[34868]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:49.465 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:49 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:49.465 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:49 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:49.465 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:49 smithi165 ceph-mon[34868]: mon.b calling monitor election 2024-08-17T17:04:49.465 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:49 smithi165 ceph-mon[34868]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:49.465 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:49 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:49.465 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:49 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:49.466 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:49 smithi165 ceph-mon[34868]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:49.466 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:49 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:49.466 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:49 smithi165 ceph-mon[34868]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-08-17T17:04:49.466 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:49 smithi165 ceph-mon[34868]: monmap e2: 2 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],b=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-08-17T17:04:49.466 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:49 smithi165 ceph-mon[34868]: fsmap 2024-08-17T17:04:49.466 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:49 smithi165 ceph-mon[34868]: osdmap e4: 0 total, 0 up, 0 in 2024-08-17T17:04:49.466 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:49 smithi165 ceph-mon[34868]: mgrmap e13: y(active, since 38s) 2024-08-17T17:04:49.466 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:49 smithi165 ceph-mon[34868]: overall HEALTH_OK 2024-08-17T17:04:49.466 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:49 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:49.466 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:49 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:49.466 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:49 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:49.466 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:49 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:04:49.466 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:49 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:49.655 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:04:49.655 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":2,"fsid":"8270436c-5cba-11ef-bccf-c7b262605968","modified":"2024-08-17T17:04:44.106833Z","created":"2024-08-17T17:03:14.749093Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-08-17T17:04:49.656 INFO:teuthology.orchestra.run.smithi165.stderr:dumped monmap epoch 2 2024-08-17T17:04:50.193 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:50 smithi165 ceph-mon[34868]: Deploying daemon mon.c on smithi136 2024-08-17T17:04:50.193 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:50 smithi165 ceph-mon[34868]: from='client.? 172.21.15.165:0/4018614874' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-17T17:04:50.193 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:50 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:50.394 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:50 smithi136 ceph-mon[31034]: Deploying daemon mon.c on smithi136 2024-08-17T17:04:50.394 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:50 smithi136 ceph-mon[31034]: from='client.? 172.21.15.165:0/4018614874' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-17T17:04:50.394 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:50 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:51.214 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-17T17:04:51.214 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph mon dump -f json 2024-08-17T17:04:51.400 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:51 smithi136 ceph-mon[31034]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:51.425 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:04:51.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:51 smithi165 ceph-mon[34868]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:51.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:51 smithi136 systemd[1]: Starting Ceph mon.c for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:04:52.381 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 podman[42542]: 2024-08-17 17:04:51.960017431 +0000 UTC m=+0.016967544 image pull c870753d67c2504e5ac044c83696772f8b7527a928cc764ed24eeea7fea57d6e quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408 2024-08-17T17:04:52.381 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 podman[42542]: 2024-08-17 17:04:52.061092203 +0000 UTC m=+0.118042351 container create b4e0f3c9e48509bb82a50145b049488b4893c7bfebd449c27534725ae8c67da5 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-c, ceph=True, org.label-schema.license=GPLv2, RELEASE=reef-ced649f, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:04:52.381 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 podman[42542]: 2024-08-17 17:04:52.380356956 +0000 UTC m=+0.437307070 container init b4e0f3c9e48509bb82a50145b049488b4893c7bfebd449c27534725ae8c67da5 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=reef-ced649f, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731) 2024-08-17T17:04:52.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 podman[42542]: 2024-08-17 17:04:52.385440631 +0000 UTC m=+0.442390734 container start b4e0f3c9e48509bb82a50145b049488b4893c7bfebd449c27534725ae8c67da5 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, RELEASE=reef-ced649f, ceph=True) 2024-08-17T17:04:52.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: set uid:gid to 167:167 (ceph:ceph) 2024-08-17T17:04:52.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable), process ceph-mon, pid 7 2024-08-17T17:04:52.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: pidfile_write: ignore empty --pid-file 2024-08-17T17:04:52.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: load: jerasure load: lrc 2024-08-17T17:04:52.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: RocksDB version: 7.9.2 2024-08-17T17:04:52.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Git sha 0 2024-08-17T17:04:52.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Compile date 2024-08-17 15:23:20 2024-08-17T17:04:52.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: DB SUMMARY 2024-08-17T17:04:52.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: DB Session ID: LBPINSKYXNMF0RC6WHMP 2024-08-17T17:04:52.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: CURRENT file: CURRENT 2024-08-17T17:04:52.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: IDENTITY file: IDENTITY 2024-08-17T17:04:52.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-08-17T17:04:52.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-08-17T17:04:52.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-08-17T17:04:52.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.error_if_exists: 0 2024-08-17T17:04:52.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.create_if_missing: 0 2024-08-17T17:04:52.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.paranoid_checks: 1 2024-08-17T17:04:52.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.flush_verify_memtable_count: 1 2024-08-17T17:04:52.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-08-17T17:04:52.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-08-17T17:04:52.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.env: 0x55586d1efbc0 2024-08-17T17:04:52.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.fs: PosixFileSystem 2024-08-17T17:04:52.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.info_log: 0x55586db7abe0 2024-08-17T17:04:52.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_file_opening_threads: 16 2024-08-17T17:04:52.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.statistics: (nil) 2024-08-17T17:04:52.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.use_fsync: 0 2024-08-17T17:04:52.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_log_file_size: 0 2024-08-17T17:04:52.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-08-17T17:04:52.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.log_file_time_to_roll: 0 2024-08-17T17:04:52.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.keep_log_file_num: 1000 2024-08-17T17:04:52.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.recycle_log_file_num: 0 2024-08-17T17:04:52.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.allow_fallocate: 1 2024-08-17T17:04:52.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.allow_mmap_reads: 0 2024-08-17T17:04:52.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.allow_mmap_writes: 0 2024-08-17T17:04:52.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.use_direct_reads: 0 2024-08-17T17:04:52.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-08-17T17:04:52.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.create_missing_column_families: 0 2024-08-17T17:04:52.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.db_log_dir: 2024-08-17T17:04:52.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.wal_dir: 2024-08-17T17:04:52.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.table_cache_numshardbits: 6 2024-08-17T17:04:52.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.WAL_ttl_seconds: 0 2024-08-17T17:04:52.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.WAL_size_limit_MB: 0 2024-08-17T17:04:52.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-08-17T17:04:52.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-08-17T17:04:52.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.is_fd_close_on_exec: 1 2024-08-17T17:04:52.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.advise_random_on_open: 1 2024-08-17T17:04:52.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.db_write_buffer_size: 0 2024-08-17T17:04:52.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.write_buffer_manager: 0x55586db4a140 2024-08-17T17:04:52.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-08-17T17:04:52.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-08-17T17:04:52.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.use_adaptive_mutex: 0 2024-08-17T17:04:52.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.rate_limiter: (nil) 2024-08-17T17:04:52.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-08-17T17:04:52.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.wal_recovery_mode: 2 2024-08-17T17:04:52.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.enable_thread_tracking: 0 2024-08-17T17:04:52.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.enable_pipelined_write: 0 2024-08-17T17:04:52.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.unordered_write: 0 2024-08-17T17:04:52.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-08-17T17:04:52.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-08-17T17:04:52.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-08-17T17:04:52.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-08-17T17:04:52.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.row_cache: None 2024-08-17T17:04:52.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.wal_filter: None 2024-08-17T17:04:52.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-08-17T17:04:52.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.allow_ingest_behind: 0 2024-08-17T17:04:52.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.two_write_queues: 0 2024-08-17T17:04:52.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.manual_wal_flush: 0 2024-08-17T17:04:52.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.wal_compression: 0 2024-08-17T17:04:52.704 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.atomic_flush: 0 2024-08-17T17:04:52.704 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-08-17T17:04:52.704 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.persist_stats_to_disk: 0 2024-08-17T17:04:52.704 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.write_dbid_to_manifest: 0 2024-08-17T17:04:52.704 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.log_readahead_size: 0 2024-08-17T17:04:52.704 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-08-17T17:04:52.704 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.best_efforts_recovery: 0 2024-08-17T17:04:52.704 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-08-17T17:04:52.704 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-08-17T17:04:52.704 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.allow_data_in_errors: 0 2024-08-17T17:04:52.704 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.db_host_id: __hostname__ 2024-08-17T17:04:52.704 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.enforce_single_del_contracts: true 2024-08-17T17:04:52.704 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_background_jobs: 2 2024-08-17T17:04:52.705 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_background_compactions: -1 2024-08-17T17:04:52.705 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_subcompactions: 1 2024-08-17T17:04:52.705 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-08-17T17:04:52.705 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-08-17T17:04:52.705 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.delayed_write_rate : 16777216 2024-08-17T17:04:52.705 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_total_wal_size: 0 2024-08-17T17:04:52.705 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-08-17T17:04:52.705 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.stats_dump_period_sec: 600 2024-08-17T17:04:52.705 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.stats_persist_period_sec: 600 2024-08-17T17:04:52.705 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-08-17T17:04:52.705 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_open_files: -1 2024-08-17T17:04:52.705 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.bytes_per_sync: 0 2024-08-17T17:04:52.705 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.wal_bytes_per_sync: 0 2024-08-17T17:04:52.706 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.strict_bytes_per_sync: 0 2024-08-17T17:04:52.706 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.compaction_readahead_size: 0 2024-08-17T17:04:52.706 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_background_flushes: -1 2024-08-17T17:04:52.706 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Compression algorithms supported: 2024-08-17T17:04:52.706 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: kZSTD supported: 0 2024-08-17T17:04:52.706 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: kXpressCompression supported: 0 2024-08-17T17:04:52.706 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: kBZip2Compression supported: 0 2024-08-17T17:04:52.706 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-08-17T17:04:52.706 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: kLZ4Compression supported: 1 2024-08-17T17:04:52.706 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: kZlibCompression supported: 1 2024-08-17T17:04:52.706 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: kLZ4HCCompression supported: 1 2024-08-17T17:04:52.706 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: kSnappyCompression supported: 1 2024-08-17T17:04:52.706 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Fast CRC32 supported: Supported on x86 2024-08-17T17:04:52.706 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: DMutex implementation: pthread_mutex_t 2024-08-17T17:04:52.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-08-17T17:04:52.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-08-17T17:04:52.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-08-17T17:04:52.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.merge_operator: 2024-08-17T17:04:52.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.compaction_filter: None 2024-08-17T17:04:52.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.compaction_filter_factory: None 2024-08-17T17:04:52.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.sst_partitioner_factory: None 2024-08-17T17:04:52.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.memtable_factory: SkipListFactory 2024-08-17T17:04:52.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.table_factory: BlockBasedTable 2024-08-17T17:04:52.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55586db7a820) 2024-08-17T17:04:52.707 INFO:journalctl@ceph.mon.c.smithi136.stdout: cache_index_and_filter_blocks: 1 2024-08-17T17:04:52.707 INFO:journalctl@ceph.mon.c.smithi136.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-08-17T17:04:52.707 INFO:journalctl@ceph.mon.c.smithi136.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-08-17T17:04:52.708 INFO:journalctl@ceph.mon.c.smithi136.stdout: pin_top_level_index_and_filter: 1 2024-08-17T17:04:52.708 INFO:journalctl@ceph.mon.c.smithi136.stdout: index_type: 0 2024-08-17T17:04:52.708 INFO:journalctl@ceph.mon.c.smithi136.stdout: data_block_index_type: 0 2024-08-17T17:04:52.708 INFO:journalctl@ceph.mon.c.smithi136.stdout: index_shortening: 1 2024-08-17T17:04:52.708 INFO:journalctl@ceph.mon.c.smithi136.stdout: data_block_hash_table_util_ratio: 0.750000 2024-08-17T17:04:52.708 INFO:journalctl@ceph.mon.c.smithi136.stdout: checksum: 4 2024-08-17T17:04:52.708 INFO:journalctl@ceph.mon.c.smithi136.stdout: no_block_cache: 0 2024-08-17T17:04:52.708 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_cache: 0x55586db6f350 2024-08-17T17:04:52.708 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_cache_name: BinnedLRUCache 2024-08-17T17:04:52.708 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_cache_options: 2024-08-17T17:04:52.708 INFO:journalctl@ceph.mon.c.smithi136.stdout: capacity : 536870912 2024-08-17T17:04:52.708 INFO:journalctl@ceph.mon.c.smithi136.stdout: num_shard_bits : 4 2024-08-17T17:04:52.708 INFO:journalctl@ceph.mon.c.smithi136.stdout: strict_capacity_limit : 0 2024-08-17T17:04:52.708 INFO:journalctl@ceph.mon.c.smithi136.stdout: high_pri_pool_ratio: 0.000 2024-08-17T17:04:52.709 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_cache_compressed: (nil) 2024-08-17T17:04:52.709 INFO:journalctl@ceph.mon.c.smithi136.stdout: persistent_cache: (nil) 2024-08-17T17:04:52.709 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_size: 4096 2024-08-17T17:04:52.709 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_size_deviation: 10 2024-08-17T17:04:52.709 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_restart_interval: 16 2024-08-17T17:04:52.709 INFO:journalctl@ceph.mon.c.smithi136.stdout: index_block_restart_interval: 1 2024-08-17T17:04:52.709 INFO:journalctl@ceph.mon.c.smithi136.stdout: metadata_block_size: 4096 2024-08-17T17:04:52.709 INFO:journalctl@ceph.mon.c.smithi136.stdout: partition_filters: 0 2024-08-17T17:04:52.709 INFO:journalctl@ceph.mon.c.smithi136.stdout: use_delta_encoding: 1 2024-08-17T17:04:52.709 INFO:journalctl@ceph.mon.c.smithi136.stdout: filter_policy: bloomfilter 2024-08-17T17:04:52.709 INFO:journalctl@ceph.mon.c.smithi136.stdout: whole_key_filtering: 1 2024-08-17T17:04:52.709 INFO:journalctl@ceph.mon.c.smithi136.stdout: verify_compression: 0 2024-08-17T17:04:52.709 INFO:journalctl@ceph.mon.c.smithi136.stdout: read_amp_bytes_per_bit: 0 2024-08-17T17:04:52.709 INFO:journalctl@ceph.mon.c.smithi136.stdout: format_version: 5 2024-08-17T17:04:52.710 INFO:journalctl@ceph.mon.c.smithi136.stdout: enable_index_compression: 1 2024-08-17T17:04:52.710 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_align: 0 2024-08-17T17:04:52.710 INFO:journalctl@ceph.mon.c.smithi136.stdout: max_auto_readahead_size: 262144 2024-08-17T17:04:52.710 INFO:journalctl@ceph.mon.c.smithi136.stdout: prepopulate_block_cache: 0 2024-08-17T17:04:52.710 INFO:journalctl@ceph.mon.c.smithi136.stdout: initial_auto_readahead_size: 8192 2024-08-17T17:04:52.710 INFO:journalctl@ceph.mon.c.smithi136.stdout: num_file_reads_for_auto_readahead: 2 2024-08-17T17:04:52.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.write_buffer_size: 33554432 2024-08-17T17:04:52.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_write_buffer_number: 2 2024-08-17T17:04:52.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.compression: NoCompression 2024-08-17T17:04:52.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.bottommost_compression: Disabled 2024-08-17T17:04:52.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.prefix_extractor: nullptr 2024-08-17T17:04:52.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-08-17T17:04:52.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.num_levels: 7 2024-08-17T17:04:52.711 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-08-17T17:04:52.711 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-08-17T17:04:52.711 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-08-17T17:04:52.711 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-08-17T17:04:52.711 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-08-17T17:04:52.711 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-08-17T17:04:52.711 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-08-17T17:04:52.711 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-08-17T17:04:52.711 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-08-17T17:04:52.711 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-08-17T17:04:52.711 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-08-17T17:04:52.711 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-08-17T17:04:52.711 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.compression_opts.window_bits: -14 2024-08-17T17:04:52.711 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.compression_opts.level: 32767 2024-08-17T17:04:52.712 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.compression_opts.strategy: 0 2024-08-17T17:04:52.712 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-08-17T17:04:52.712 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-08-17T17:04:52.712 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-08-17T17:04:52.712 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-08-17T17:04:52.712 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.compression_opts.enabled: false 2024-08-17T17:04:52.712 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-08-17T17:04:52.712 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-08-17T17:04:52.712 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-08-17T17:04:52.712 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-08-17T17:04:52.712 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.target_file_size_base: 67108864 2024-08-17T17:04:52.712 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.target_file_size_multiplier: 1 2024-08-17T17:04:52.712 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-08-17T17:04:52.712 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-08-17T17:04:52.713 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-08-17T17:04:52.713 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-08-17T17:04:52.713 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-08-17T17:04:52.713 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-08-17T17:04:52.713 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-08-17T17:04:52.713 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-08-17T17:04:52.713 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-08-17T17:04:52.713 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-08-17T17:04:52.713 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-08-17T17:04:52.713 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-08-17T17:04:52.713 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-08-17T17:04:52.713 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.arena_block_size: 1048576 2024-08-17T17:04:52.713 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-08-17T17:04:52.714 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-08-17T17:04:52.714 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.disable_auto_compactions: 0 2024-08-17T17:04:52.714 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-08-17T17:04:52.714 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-08-17T17:04:52.714 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-08-17T17:04:52.714 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-08-17T17:04:52.714 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-08-17T17:04:52.714 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-08-17T17:04:52.714 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-08-17T17:04:52.714 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-08-17T17:04:52.714 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-08-17T17:04:52.714 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-08-17T17:04:52.714 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.table_properties_collectors: 2024-08-17T17:04:52.714 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.inplace_update_support: 0 2024-08-17T17:04:52.715 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.inplace_update_num_locks: 10000 2024-08-17T17:04:52.715 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-08-17T17:04:52.715 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-08-17T17:04:52.715 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.memtable_huge_page_size: 0 2024-08-17T17:04:52.715 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.bloom_locality: 0 2024-08-17T17:04:52.715 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.max_successive_merges: 0 2024-08-17T17:04:52.715 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.optimize_filters_for_hits: 0 2024-08-17T17:04:52.715 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.paranoid_file_checks: 0 2024-08-17T17:04:52.715 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.force_consistency_checks: 1 2024-08-17T17:04:52.715 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.report_bg_io_stats: 0 2024-08-17T17:04:52.715 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.ttl: 2592000 2024-08-17T17:04:52.715 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.periodic_compaction_seconds: 0 2024-08-17T17:04:52.715 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-08-17T17:04:52.716 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-08-17T17:04:52.716 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.enable_blob_files: false 2024-08-17T17:04:52.716 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.min_blob_size: 0 2024-08-17T17:04:52.716 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.blob_file_size: 268435456 2024-08-17T17:04:52.716 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.blob_compression_type: NoCompression 2024-08-17T17:04:52.716 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.enable_blob_garbage_collection: false 2024-08-17T17:04:52.716 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-08-17T17:04:52.716 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-08-17T17:04:52.716 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-08-17T17:04:52.716 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.blob_file_starting_level: 0 2024-08-17T17:04:52.716 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-08-17T17:04:52.716 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-08-17T17:04:52.716 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-08-17T17:04:52.717 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a1d107e2-c9e3-41fd-8204-844cc2196212 2024-08-17T17:04:52.717 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914292412086, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-08-17T17:04:52.717 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-08-17T17:04:52.717 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914292412477, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1723914292, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a1d107e2-c9e3-41fd-8204-844cc2196212", "db_session_id": "LBPINSKYXNMF0RC6WHMP", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-08-17T17:04:52.717 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914292412572, "job": 1, "event": "recovery_finished"} 2024-08-17T17:04:52.717 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-08-17T17:04:52.717 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-08-17T17:04:52.717 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55586db98e00 2024-08-17T17:04:52.717 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: DB pointer 0x55586dc8c000 2024-08-17T17:04:52.717 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-08-17T17:04:52.717 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-08-17T17:04:52.717 INFO:journalctl@ceph.mon.c.smithi136.stdout: ** DB Stats ** 2024-08-17T17:04:52.718 INFO:journalctl@ceph.mon.c.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-17T17:04:52.718 INFO:journalctl@ceph.mon.c.smithi136.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-08-17T17:04:52.718 INFO:journalctl@ceph.mon.c.smithi136.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-17T17:04:52.718 INFO:journalctl@ceph.mon.c.smithi136.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-17T17:04:52.718 INFO:journalctl@ceph.mon.c.smithi136.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-08-17T17:04:52.718 INFO:journalctl@ceph.mon.c.smithi136.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-17T17:04:52.718 INFO:journalctl@ceph.mon.c.smithi136.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-17T17:04:52.718 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-08-17T17:04:52.718 INFO:journalctl@ceph.mon.c.smithi136.stdout: ** Compaction Stats [default] ** 2024-08-17T17:04:52.718 INFO:journalctl@ceph.mon.c.smithi136.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-17T17:04:52.718 INFO:journalctl@ceph.mon.c.smithi136.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-08-17T17:04:52.718 INFO:journalctl@ceph.mon.c.smithi136.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-08-17T17:04:52.718 INFO:journalctl@ceph.mon.c.smithi136.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-08-17T17:04:52.719 INFO:journalctl@ceph.mon.c.smithi136.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-08-17T17:04:52.719 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-08-17T17:04:52.719 INFO:journalctl@ceph.mon.c.smithi136.stdout: ** Compaction Stats [default] ** 2024-08-17T17:04:52.719 INFO:journalctl@ceph.mon.c.smithi136.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-17T17:04:52.719 INFO:journalctl@ceph.mon.c.smithi136.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-17T17:04:52.719 INFO:journalctl@ceph.mon.c.smithi136.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-08-17T17:04:52.719 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-08-17T17:04:52.719 INFO:journalctl@ceph.mon.c.smithi136.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-08-17T17:04:52.719 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-08-17T17:04:52.719 INFO:journalctl@ceph.mon.c.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-17T17:04:52.719 INFO:journalctl@ceph.mon.c.smithi136.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-08-17T17:04:52.719 INFO:journalctl@ceph.mon.c.smithi136.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-08-17T17:04:52.719 INFO:journalctl@ceph.mon.c.smithi136.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-08-17T17:04:52.719 INFO:journalctl@ceph.mon.c.smithi136.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-08-17T17:04:52.720 INFO:journalctl@ceph.mon.c.smithi136.stdout: AddFile(Keys): cumulative 0, interval 0 2024-08-17T17:04:52.720 INFO:journalctl@ceph.mon.c.smithi136.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:04:52.720 INFO:journalctl@ceph.mon.c.smithi136.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:04:52.720 INFO:journalctl@ceph.mon.c.smithi136.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-08-17T17:04:52.720 INFO:journalctl@ceph.mon.c.smithi136.stdout: Block cache BinnedLRUCache@0x55586db6f350#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-08-17T17:04:52.720 INFO:journalctl@ceph.mon.c.smithi136.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-08-17T17:04:52.720 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-08-17T17:04:52.720 INFO:journalctl@ceph.mon.c.smithi136.stdout: ** File Read Latency Histogram By Level [default] ** 2024-08-17T17:04:52.720 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-08-17T17:04:52.720 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: using public_addrv [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] 2024-08-17T17:04:52.720 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: starting mon.c rank -1 at public addrs [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] at bind addrs [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:04:52.720 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: mon.c@-1(???) e0 preinit fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:04:52.720 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: mon.c@-1(synchronizing).mds e1 new map 2024-08-17T17:04:52.721 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: mon.c@-1(synchronizing).mds e1 print_map 2024-08-17T17:04:52.721 INFO:journalctl@ceph.mon.c.smithi136.stdout: e1 2024-08-17T17:04:52.721 INFO:journalctl@ceph.mon.c.smithi136.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-08-17T17:04:52.721 INFO:journalctl@ceph.mon.c.smithi136.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-08-17T17:04:52.721 INFO:journalctl@ceph.mon.c.smithi136.stdout: legacy client fscid: -1 2024-08-17T17:04:52.721 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-08-17T17:04:52.721 INFO:journalctl@ceph.mon.c.smithi136.stdout: No filesystems configured 2024-08-17T17:04:52.721 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-08-17T17:04:52.721 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-08-17T17:04:52.721 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-08-17T17:04:52.721 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-08-17T17:04:52.721 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-08-17T17:04:52.721 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-08-17T17:04:52.721 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-08-17T17:04:52.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-08-17T17:04:52.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-08-17T17:04:52.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-08-17T17:04:52.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: Updating smithi165:/etc/ceph/ceph.conf 2024-08-17T17:04:52.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:04:52.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:04:52.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:04:52.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/3130059668' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-08-17T17:04:52.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:52.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:52.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:52.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/3130059668' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-08-17T17:04:52.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: osdmap e4: 0 total, 0 up, 0 in 2024-08-17T17:04:52.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:52.723 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:52.723 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi136:172.21.15.136=a;smithi136:[v2:172.21.15.136:3301,v1:172.21.15.136:6790]=c;smithi165:172.21.15.165=b", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:04:52.723 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: Saving service mon spec with placement smithi136:172.21.15.136=a;smithi136:[v2:172.21.15.136:3301,v1:172.21.15.136:6790]=c;smithi165:172.21.15.165=b;count:3 2024-08-17T17:04:52.723 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:52.723 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:52.723 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:52.723 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:52.723 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:52.723 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:04:52.723 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:52.723 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: Deploying daemon mon.b on smithi165 2024-08-17T17:04:52.723 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:52.723 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:52.724 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='client.? 172.21.15.165:0/1747041679' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-17T17:04:52.724 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:04:52.724 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: mon.a calling monitor election 2024-08-17T17:04:52.724 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:52.724 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:52.724 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:52.724 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:52.724 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: mon.b calling monitor election 2024-08-17T17:04:52.724 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:52.724 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:52.724 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:52.724 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:52.724 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:52.724 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-08-17T17:04:52.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: monmap e2: 2 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],b=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-08-17T17:04:52.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: fsmap 2024-08-17T17:04:52.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: osdmap e4: 0 total, 0 up, 0 in 2024-08-17T17:04:52.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: mgrmap e13: y(active, since 38s) 2024-08-17T17:04:52.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: overall HEALTH_OK 2024-08-17T17:04:52.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:52.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:52.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:52.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:04:52.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:52.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: Deploying daemon mon.c on smithi136 2024-08-17T17:04:52.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='client.? 172.21.15.165:0/4018614874' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-17T17:04:52.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:52.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:52.726 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 ceph-mon[42580]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-08-17T17:04:52.726 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 bash[42542]: b4e0f3c9e48509bb82a50145b049488b4893c7bfebd449c27534725ae8c67da5 2024-08-17T17:04:52.726 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:52 smithi136 systemd[1]: Started Ceph mon.c for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:04:57.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:04:57.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[31034]: mon.a calling monitor election 2024-08-17T17:04:57.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[31034]: mon.b calling monitor election 2024-08-17T17:04:57.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:57.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:57.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[31034]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:57.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:57.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:57.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[31034]: mon.c calling monitor election 2024-08-17T17:04:57.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[31034]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:57.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:57.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:57.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[31034]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:57.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:57.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[31034]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-08-17T17:04:57.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[31034]: monmap e3: 3 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],b=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0],c=[v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-08-17T17:04:57.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[31034]: fsmap 2024-08-17T17:04:57.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[31034]: osdmap e4: 0 total, 0 up, 0 in 2024-08-17T17:04:57.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[31034]: mgrmap e13: y(active, since 46s) 2024-08-17T17:04:57.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[31034]: overall HEALTH_OK 2024-08-17T17:04:57.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:57.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:57.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:57.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:57.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:04:57.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[42580]: mon.a calling monitor election 2024-08-17T17:04:57.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[42580]: mon.b calling monitor election 2024-08-17T17:04:57.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:57.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:57.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[42580]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:57.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:57.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:57.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[42580]: mon.c calling monitor election 2024-08-17T17:04:57.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[42580]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:57.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:57.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:57.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[42580]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:57.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:57.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[42580]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-08-17T17:04:57.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[42580]: monmap e3: 3 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],b=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0],c=[v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-08-17T17:04:57.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[42580]: fsmap 2024-08-17T17:04:57.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[42580]: osdmap e4: 0 total, 0 up, 0 in 2024-08-17T17:04:57.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[42580]: mgrmap e13: y(active, since 46s) 2024-08-17T17:04:57.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[42580]: overall HEALTH_OK 2024-08-17T17:04:57.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:57.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:57.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:57.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:57 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:57 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:04:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:57 smithi165 ceph-mon[34868]: mon.a calling monitor election 2024-08-17T17:04:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:57 smithi165 ceph-mon[34868]: mon.b calling monitor election 2024-08-17T17:04:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:57 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:58.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:57 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:58.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:57 smithi165 ceph-mon[34868]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:58.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:57 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:58.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:57 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:58.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:57 smithi165 ceph-mon[34868]: mon.c calling monitor election 2024-08-17T17:04:58.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:57 smithi165 ceph-mon[34868]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:58.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:57 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:58.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:57 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:58.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:57 smithi165 ceph-mon[34868]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:58.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:57 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:58.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:57 smithi165 ceph-mon[34868]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-08-17T17:04:58.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:57 smithi165 ceph-mon[34868]: monmap e3: 3 mons at {a=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0],b=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0],c=[v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-08-17T17:04:58.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:57 smithi165 ceph-mon[34868]: fsmap 2024-08-17T17:04:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:57 smithi165 ceph-mon[34868]: osdmap e4: 0 total, 0 up, 0 in 2024-08-17T17:04:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:57 smithi165 ceph-mon[34868]: mgrmap e13: y(active, since 46s) 2024-08-17T17:04:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:57 smithi165 ceph-mon[34868]: overall HEALTH_OK 2024-08-17T17:04:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:57 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:57 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:57 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:57 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:58.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:58 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:58.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:58 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:58.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:58 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:58.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:58 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:58.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:58 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:58.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:58 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:58.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:58 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:58.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:58 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:58 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:58 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:58 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:58 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.049 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:04:59.049 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":3,"fsid":"8270436c-5cba-11ef-bccf-c7b262605968","modified":"2024-08-17T17:04:52.440327Z","created":"2024-08-17T17:03:14.749093Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3301","nonce":0},{"type":"v1","addr":"172.21.15.136:6790","nonce":0}]},"addr":"172.21.15.136:6790/0","public_addr":"172.21.15.136:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-08-17T17:04:59.050 INFO:teuthology.orchestra.run.smithi165.stderr:dumped monmap epoch 3 2024-08-17T17:04:59.603 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-08-17T17:04:59.603 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph config generate-minimal-conf 2024-08-17T17:04:59.721 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[42580]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:59.721 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:59.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:59.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[42580]: Updating smithi136:/etc/ceph/ceph.conf 2024-08-17T17:04:59.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[42580]: Updating smithi165:/etc/ceph/ceph.conf 2024-08-17T17:04:59.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[42580]: from='client.? 172.21.15.165:0/3249281996' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-17T17:04:59.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.723 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.723 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.723 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.723 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.723 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:04:59.723 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-17T17:04:59.723 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:59.723 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[31034]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:04:59.723 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.723 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.723 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:59.723 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:59.724 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[31034]: Updating smithi136:/etc/ceph/ceph.conf 2024-08-17T17:04:59.724 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[31034]: Updating smithi165:/etc/ceph/ceph.conf 2024-08-17T17:04:59.724 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[31034]: from='client.? 172.21.15.165:0/3249281996' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-17T17:04:59.724 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.724 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.724 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.724 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.724 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.724 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.724 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.724 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.724 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:04:59.724 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:04:59.725 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-17T17:04:59.725 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:04:59 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:59 smithi165 ceph-mon[34868]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:59 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:59 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:59 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:59 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:00.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:59 smithi165 ceph-mon[34868]: Updating smithi136:/etc/ceph/ceph.conf 2024-08-17T17:05:00.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:59 smithi165 ceph-mon[34868]: Updating smithi165:/etc/ceph/ceph.conf 2024-08-17T17:05:00.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:59 smithi165 ceph-mon[34868]: from='client.? 172.21.15.165:0/3249281996' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-17T17:05:00.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:59 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:00.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:59 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:00.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:59 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:00.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:59 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:00.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:59 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:00.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:59 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:00.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:59 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:00.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:59 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:00.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:59 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:00.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:59 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:05:00.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:59 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-17T17:05:00.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:04:59 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:00.569 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:05:00.709 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:00 smithi136 ceph-mon[42580]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:05:00.709 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:00 smithi136 ceph-mon[42580]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:05:00.709 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:00 smithi136 ceph-mon[42580]: Reconfiguring mon.a (unknown last config time)... 2024-08-17T17:05:00.709 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:00 smithi136 ceph-mon[42580]: Reconfiguring daemon mon.a on smithi136 2024-08-17T17:05:00.709 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:00 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:00.709 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:00 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:00.709 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:00 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:05:00.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:00 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-17T17:05:00.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:00 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:00.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:00 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:00.710 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:00 smithi136 ceph-mon[31034]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:05:00.710 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:00 smithi136 ceph-mon[31034]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:05:00.710 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:00 smithi136 ceph-mon[31034]: Reconfiguring mon.a (unknown last config time)... 2024-08-17T17:05:00.711 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:00 smithi136 ceph-mon[31034]: Reconfiguring daemon mon.a on smithi136 2024-08-17T17:05:00.711 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:00 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:00.711 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:00 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:00.711 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:00 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:05:00.711 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:00 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-17T17:05:00.711 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:00 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:00.711 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:00 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:00 smithi165 ceph-mon[34868]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:05:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:00 smithi165 ceph-mon[34868]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:05:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:00 smithi165 ceph-mon[34868]: Reconfiguring mon.a (unknown last config time)... 2024-08-17T17:05:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:00 smithi165 ceph-mon[34868]: Reconfiguring daemon mon.a on smithi136 2024-08-17T17:05:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:00 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:00 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:00 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:05:01.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:00 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-17T17:05:01.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:00 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:01.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:00 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:01 smithi136 ceph-mon[31034]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:01 smithi136 ceph-mon[31034]: Reconfiguring mon.c (monmap changed)... 2024-08-17T17:05:01.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:01 smithi136 ceph-mon[31034]: Reconfiguring daemon mon.c on smithi136 2024-08-17T17:05:01.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:01 smithi136 ceph-mon[42580]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:01.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:01 smithi136 ceph-mon[42580]: Reconfiguring mon.c (monmap changed)... 2024-08-17T17:05:01.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:01 smithi136 ceph-mon[42580]: Reconfiguring daemon mon.c on smithi136 2024-08-17T17:05:02.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:01 smithi165 ceph-mon[34868]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:02.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:01 smithi165 ceph-mon[34868]: Reconfiguring mon.c (monmap changed)... 2024-08-17T17:05:02.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:01 smithi165 ceph-mon[34868]: Reconfiguring daemon mon.c on smithi136 2024-08-17T17:05:03.065 INFO:teuthology.orchestra.run.smithi136.stdout:# minimal ceph.conf for 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:05:03.065 INFO:teuthology.orchestra.run.smithi136.stdout:[global] 2024-08-17T17:05:03.065 INFO:teuthology.orchestra.run.smithi136.stdout: fsid = 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:05:03.065 INFO:teuthology.orchestra.run.smithi136.stdout: mon_host = [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] 2024-08-17T17:05:03.649 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-08-17T17:05:03.649 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-17T17:05:03.649 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.conf 2024-08-17T17:05:03.677 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-17T17:05:03.677 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:05:03.742 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-08-17T17:05:03.742 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.conf 2024-08-17T17:05:03.745 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:03 smithi165 ceph-mon[34868]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:03.745 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:03 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:03.745 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:03 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:03.746 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:03 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:05:03.746 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:03 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-17T17:05:03.746 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:03 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:03.746 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:03 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/34795021' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:03.770 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-08-17T17:05:03.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:05:03.836 INFO:tasks.cephadm:Adding mgr.y on smithi136 2024-08-17T17:05:03.836 INFO:tasks.cephadm:Adding mgr.x on smithi165 2024-08-17T17:05:03.836 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph orch apply mgr '2;smithi136=y;smithi165=x' 2024-08-17T17:05:03.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:03 smithi136 ceph-mon[31034]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:03.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:03 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:03.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:03 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:03.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:03 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:05:03.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:03 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-17T17:05:03.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:03 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:03.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:03 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/34795021' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:03 smithi136 ceph-mon[42580]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:03.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:03 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:03.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:03 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:03.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:03 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:05:03.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:03 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-17T17:05:03.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:03 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:03.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:03 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/34795021' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:04.094 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:05:04.844 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:04 smithi165 ceph-mon[34868]: Reconfiguring mon.b (monmap changed)... 2024-08-17T17:05:04.844 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:04 smithi165 ceph-mon[34868]: Reconfiguring daemon mon.b on smithi165 2024-08-17T17:05:04.844 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:04 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:04.844 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:04 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:04.844 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:04 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:04.844 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:04 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:04.844 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:04 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:04.844 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:04 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:04.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:04 smithi136 ceph-mon[31034]: Reconfiguring mon.b (monmap changed)... 2024-08-17T17:05:04.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:04 smithi136 ceph-mon[31034]: Reconfiguring daemon mon.b on smithi165 2024-08-17T17:05:04.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:04 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:04.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:04 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:04.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:04 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:04.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:04 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:04.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:04 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:04.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:04 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:04.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:04 smithi136 ceph-mon[42580]: Reconfiguring mon.b (monmap changed)... 2024-08-17T17:05:04.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:04 smithi136 ceph-mon[42580]: Reconfiguring daemon mon.b on smithi165 2024-08-17T17:05:04.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:04 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:04.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:04 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:04.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:04 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:04.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:04 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:04.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:04 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:04.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:04 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:05 smithi165 ceph-mon[34868]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:05.761 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled mgr update... 2024-08-17T17:05:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:05 smithi136 ceph-mon[31034]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:05.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:05 smithi136 ceph-mon[42580]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:06.349 DEBUG:teuthology.orchestra.run.smithi165:mgr.x> sudo journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@mgr.x.service 2024-08-17T17:05:06.351 DEBUG:tasks.cephadm:set 0 configs 2024-08-17T17:05:06.351 INFO:tasks.cephadm:Deploying OSDs... 2024-08-17T17:05:06.351 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-17T17:05:06.351 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2024-08-17T17:05:06.371 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-17T17:05:06.371 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_1 2024-08-17T17:05:06.429 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-17T17:05:06.429 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-17T17:05:06.429 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 978 Links: 1 2024-08-17T17:05:06.429 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-17T17:05:06.429 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-08-17T17:05:06.429 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-08-17 17:04:51.781386580 +0000 2024-08-17T17:05:06.429 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-08-17 17:00:34.986410507 +0000 2024-08-17T17:05:06.429 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-08-17 17:00:34.986410507 +0000 2024-08-17T17:05:06.430 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-08-17 17:00:34.986410507 +0000 2024-08-17T17:05:06.430 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-17T17:05:06.497 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-08-17T17:05:06.497 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-08-17T17:05:06.497 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000224059 s, 2.3 MB/s 2024-08-17T17:05:06.498 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-17T17:05:06.556 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_2 2024-08-17T17:05:06.613 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-17T17:05:06.613 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-17T17:05:06.613 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 969 Links: 1 2024-08-17T17:05:06.613 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-17T17:05:06.613 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-08-17T17:05:06.613 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-08-17 17:04:51.782386558 +0000 2024-08-17T17:05:06.614 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-08-17 17:00:34.986410507 +0000 2024-08-17T17:05:06.614 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-08-17 17:00:34.986410507 +0000 2024-08-17T17:05:06.614 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-08-17 17:00:34.986410507 +0000 2024-08-17T17:05:06.614 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-17T17:05:06.679 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-08-17T17:05:06.679 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-08-17T17:05:06.679 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000250283 s, 2.0 MB/s 2024-08-17T17:05:06.680 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-17T17:05:06.737 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_3 2024-08-17T17:05:06.794 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-17T17:05:06.794 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-17T17:05:06.794 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 983 Links: 1 2024-08-17T17:05:06.794 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-17T17:05:06.794 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-08-17T17:05:06.794 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-08-17 17:04:51.782386558 +0000 2024-08-17T17:05:06.795 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-08-17 17:00:34.987410483 +0000 2024-08-17T17:05:06.795 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-08-17 17:00:34.987410483 +0000 2024-08-17T17:05:06.795 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-08-17 17:00:34.987410483 +0000 2024-08-17T17:05:06.795 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-17T17:05:06.858 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-08-17T17:05:06.858 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-08-17T17:05:06.858 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000233815 s, 2.2 MB/s 2024-08-17T17:05:06.859 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-17T17:05:06.916 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_4 2024-08-17T17:05:06.974 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-17T17:05:06.974 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-17T17:05:06.974 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 977 Links: 1 2024-08-17T17:05:06.974 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-17T17:05:06.974 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-08-17T17:05:06.974 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-08-17 17:04:51.783386534 +0000 2024-08-17T17:05:06.974 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-08-17 17:00:34.986410507 +0000 2024-08-17T17:05:06.974 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-08-17 17:00:34.986410507 +0000 2024-08-17T17:05:06.974 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-08-17 17:00:34.986410507 +0000 2024-08-17T17:05:06.975 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-17T17:05:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:06 smithi165 ceph-mon[34868]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi136=y;smithi165=x", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:05:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:06 smithi165 ceph-mon[34868]: Saving service mgr spec with placement smithi136=y;smithi165=x;count:2 2024-08-17T17:05:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:06 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:06 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:06 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:06 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:06 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:06 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:05:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:06 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-17T17:05:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:06 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:05:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:06 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:07.034 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:06 smithi136 ceph-mon[42580]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi136=y;smithi165=x", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:05:07.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:06 smithi136 ceph-mon[42580]: Saving service mgr spec with placement smithi136=y;smithi165=x;count:2 2024-08-17T17:05:07.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:06 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:07.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:06 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:07.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:06 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:07.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:06 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:07.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:06 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:07.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:06 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:05:07.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:06 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-17T17:05:07.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:06 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:05:07.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:06 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:07.036 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:06 smithi136 ceph-mon[31034]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi136=y;smithi165=x", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:05:07.036 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:06 smithi136 ceph-mon[31034]: Saving service mgr spec with placement smithi136=y;smithi165=x;count:2 2024-08-17T17:05:07.036 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:06 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:07.036 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:06 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:07.036 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:06 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:07.036 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:06 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:07.036 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:06 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:07.036 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:06 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:05:07.036 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:06 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-17T17:05:07.036 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:06 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:05:07.036 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:06 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:07.037 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-08-17T17:05:07.037 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-08-17T17:05:07.037 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000260691 s, 2.0 MB/s 2024-08-17T17:05:07.038 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-17T17:05:07.095 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-08-17T17:05:07.095 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/scratch_devs of=/dev/stdout 2024-08-17T17:05:07.111 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-17T17:05:07.112 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_1 2024-08-17T17:05:07.167 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-17T17:05:07.167 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-17T17:05:07.167 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 983 Links: 1 2024-08-17T17:05:07.167 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-17T17:05:07.167 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-08-17T17:05:07.168 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-08-17 17:04:43.243754108 +0000 2024-08-17T17:05:07.168 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-08-17 17:00:34.904114405 +0000 2024-08-17T17:05:07.168 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-08-17 17:00:34.904114405 +0000 2024-08-17T17:05:07.168 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-08-17 17:00:34.904114405 +0000 2024-08-17T17:05:07.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-17T17:05:07.231 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-08-17T17:05:07.231 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-08-17T17:05:07.231 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000205667 s, 2.5 MB/s 2024-08-17T17:05:07.232 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-17T17:05:07.289 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_2 2024-08-17T17:05:07.344 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-17T17:05:07.344 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-17T17:05:07.344 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 977 Links: 1 2024-08-17T17:05:07.344 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-17T17:05:07.344 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-08-17T17:05:07.344 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-08-17 17:04:43.243754108 +0000 2024-08-17T17:05:07.344 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-08-17 17:00:34.904114405 +0000 2024-08-17T17:05:07.344 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-08-17 17:00:34.904114405 +0000 2024-08-17T17:05:07.345 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-08-17 17:00:34.904114405 +0000 2024-08-17T17:05:07.345 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-17T17:05:07.407 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-08-17T17:05:07.407 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-08-17T17:05:07.407 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000207248 s, 2.5 MB/s 2024-08-17T17:05:07.409 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-17T17:05:07.465 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_3 2024-08-17T17:05:07.520 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-17T17:05:07.521 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-17T17:05:07.521 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 967 Links: 1 2024-08-17T17:05:07.521 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-17T17:05:07.521 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-08-17T17:05:07.521 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-08-17 17:04:43.243754108 +0000 2024-08-17T17:05:07.521 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-08-17 17:00:34.902114431 +0000 2024-08-17T17:05:07.521 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-08-17 17:00:34.902114431 +0000 2024-08-17T17:05:07.521 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-08-17 17:00:34.902114431 +0000 2024-08-17T17:05:07.521 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-17T17:05:07.727 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-08-17T17:05:07.727 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-08-17T17:05:07.727 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000175047 s, 2.9 MB/s 2024-08-17T17:05:07.728 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-17T17:05:07.744 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_4 2024-08-17T17:05:07.804 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-17T17:05:07.805 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-17T17:05:07.805 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 970 Links: 1 2024-08-17T17:05:07.805 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-17T17:05:07.805 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-08-17T17:05:07.805 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-08-17 17:05:07.682426448 +0000 2024-08-17T17:05:07.805 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-08-17 17:00:34.903114418 +0000 2024-08-17T17:05:07.805 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-08-17 17:00:34.903114418 +0000 2024-08-17T17:05:07.805 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-08-17 17:00:34.903114418 +0000 2024-08-17T17:05:07.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-17T17:05:07.902 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-08-17T17:05:07.902 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-08-17T17:05:07.902 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000201103 s, 2.5 MB/s 2024-08-17T17:05:07.904 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-17T17:05:07.919 INFO:tasks.cephadm:Deploying osd.0 on smithi136 with /dev/vg_nvme/lv_4... 2024-08-17T17:05:07.919 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-08-17T17:05:08.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:07 smithi165 ceph-mon[34868]: Deploying daemon mgr.x on smithi165 2024-08-17T17:05:08.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:07 smithi165 ceph-mon[34868]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:08.012 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:07 smithi165 systemd[1]: Starting Ceph mgr.x for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:05:08.118 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:05:08.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:07 smithi136 ceph-mon[31034]: Deploying daemon mgr.x on smithi165 2024-08-17T17:05:08.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:07 smithi136 ceph-mon[31034]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:08.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:07 smithi136 ceph-mon[42580]: Deploying daemon mgr.x on smithi165 2024-08-17T17:05:08.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:07 smithi136 ceph-mon[42580]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:08.452 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:08 smithi165 podman[36944]: 2024-08-17 17:05:08.14083505 +0000 UTC m=+0.113912533 container create 5723e013d9096416f3d0e8fd2a8532584036e1588a99c227182261fed22745e5 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=reef-ced649f, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.build-date=20240731) 2024-08-17T17:05:08.452 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:08 smithi165 podman[36944]: 2024-08-17 17:05:08.043711622 +0000 UTC m=+0.016789081 image pull c870753d67c2504e5ac044c83696772f8b7527a928cc764ed24eeea7fea57d6e quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408 2024-08-17T17:05:08.452 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:08 smithi165 podman[36944]: 2024-08-17 17:05:08.343097078 +0000 UTC m=+0.316174535 container init 5723e013d9096416f3d0e8fd2a8532584036e1588a99c227182261fed22745e5 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x, ceph=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=reef-ced649f, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-08-17T17:05:08.452 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:08 smithi165 podman[36944]: 2024-08-17 17:05:08.348283028 +0000 UTC m=+0.321360467 container start 5723e013d9096416f3d0e8fd2a8532584036e1588a99c227182261fed22745e5 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=reef-ced649f) 2024-08-17T17:05:08.452 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:08 smithi165 ceph-mgr[36982]: -- 172.21.15.165:0/2842351016 <== mon.1 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5602e872ed00 con 0x5602e8703400 2024-08-17T17:05:08.452 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:08 smithi165 bash[36944]: 5723e013d9096416f3d0e8fd2a8532584036e1588a99c227182261fed22745e5 2024-08-17T17:05:08.761 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:08 smithi165 systemd[1]: Started Ceph mgr.x for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:05:08.761 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:08 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:08.468+0000 7fbcf5f80040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-17T17:05:08.761 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:08 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:08.626+0000 7fbcf5f80040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:05:08.761 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:08 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:08.696+0000 7fbcf5f80040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-17T17:05:09.261 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:08 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:08.966+0000 7fbcf5f80040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-17T17:05:09.261 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:09 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:09.025+0000 7fbcf5f80040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-17T17:05:09.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:09 smithi136 ceph-mon[42580]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:09.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:09 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:09.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:09 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:09 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:09 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:09 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:09 smithi136 ceph-mon[31034]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:09 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:09 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:09 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:09 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:09 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:09.986 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:09 smithi165 ceph-mon[34868]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:09.986 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:09 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:09.986 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:09 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:09.986 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:09 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:09.986 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:09 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:09.986 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:09 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:10.261 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:09 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:09.984+0000 7fbcf5f80040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-17T17:05:10.261 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:10 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:10.096+0000 7fbcf5f80040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-17T17:05:10.261 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:10 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:10.152+0000 7fbcf5f80040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-17T17:05:10.461 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:05:10.481 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi136:vg_nvme/lv_4 2024-08-17T17:05:10.626 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:10 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:10.358+0000 7fbcf5f80040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-17T17:05:10.626 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:10 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:10.417+0000 7fbcf5f80040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-17T17:05:10.627 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:10 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:10.482+0000 7fbcf5f80040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-17T17:05:10.627 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:10 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:10.625+0000 7fbcf5f80040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:05:10.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:10 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:10.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:10 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:10.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:10 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:10.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:10 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:10.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:10 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:10.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:10 smithi165 ceph-mon[34868]: Reconfiguring mgr.y (unknown last config time)... 2024-08-17T17:05:10.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:10 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:05:10.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:10 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:05:10.628 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:10 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:10.628 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:10 smithi165 ceph-mon[34868]: Reconfiguring daemon mgr.y on smithi136 2024-08-17T17:05:10.628 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:10 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:10.760 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:10 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:10.761 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:10 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:10.761 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:10 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:10.761 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:10 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:10.761 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:10 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:10.761 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:10 smithi136 ceph-mon[42580]: Reconfiguring mgr.y (unknown last config time)... 2024-08-17T17:05:10.761 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:10 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:05:10.761 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:10 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:05:10.761 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:10 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:10.761 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:10 smithi136 ceph-mon[42580]: Reconfiguring daemon mgr.y on smithi136 2024-08-17T17:05:10.761 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:10 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:10.761 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:10 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:10.761 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:10 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:10.762 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:10 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:10.762 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:10 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:10.762 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:10 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:10.762 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:10 smithi136 ceph-mon[31034]: Reconfiguring mgr.y (unknown last config time)... 2024-08-17T17:05:10.762 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:10 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:05:10.762 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:10 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:05:10.762 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:10 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:10.762 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:10 smithi136 ceph-mon[31034]: Reconfiguring daemon mgr.y on smithi136 2024-08-17T17:05:10.762 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:10 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:11.343 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:11.036+0000 7fbcf5f80040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-17T17:05:11.343 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:11.341+0000 7fbcf5f80040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-17T17:05:11.564 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:05:11.761 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:11.406+0000 7fbcf5f80040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-17T17:05:11.761 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:11.553+0000 7fbcf5f80040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-17T17:05:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:11 smithi165 ceph-mon[34868]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:11 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:11 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:11.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:11 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:11.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:11 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:11.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:11 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:11.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:11 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:11.836 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:11 smithi136 ceph-mon[42580]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:11.836 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:11 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:11.836 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:11 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:11.836 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:11 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:11.836 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:11 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:11.836 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:11 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:11.836 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:11 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:11.837 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:11 smithi136 ceph-mon[31034]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:11.837 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:11 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:11.837 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:11 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:11.837 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:11 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:11.837 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:11 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:11.837 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:11 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:11.837 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:11 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:12.246 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:11.859+0000 7fbcf5f80040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-17T17:05:12.246 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:11.919+0000 7fbcf5f80040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-17T17:05:12.246 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:12 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:12.099+0000 7fbcf5f80040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-17T17:05:12.246 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:12 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:12.244+0000 7fbcf5f80040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-17T17:05:12.511 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:12 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:12.306+0000 7fbcf5f80040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-17T17:05:12.511 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:12 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:12.388+0000 7fbcf5f80040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-17T17:05:12.511 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:12 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:12.446+0000 7fbcf5f80040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-17T17:05:13.261 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:12 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:12.965+0000 7fbcf5f80040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-17T17:05:13.261 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:05:13 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:05:13.120+0000 7fbcf5f80040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-17T17:05:13.937 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:13 smithi136 ceph-mon[42580]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:13.937 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:13 smithi136 ceph-mon[42580]: Standby manager daemon x started 2024-08-17T17:05:13.938 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:13 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.165:0/4132620753' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:05:13.938 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:13 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.165:0/4132620753' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:05:13.938 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:13 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.165:0/4132620753' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:05:13.938 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:13 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.165:0/4132620753' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:05:13.938 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:13 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:05:13.938 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:13 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:05:13.938 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:13 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:13.938 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:13 smithi136 ceph-mon[31034]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:13.938 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:13 smithi136 ceph-mon[31034]: Standby manager daemon x started 2024-08-17T17:05:13.938 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:13 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.165:0/4132620753' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:05:13.939 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:13 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.165:0/4132620753' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:05:13.939 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:13 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.165:0/4132620753' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:05:13.939 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:13 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.165:0/4132620753' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:05:13.939 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:13 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:05:13.939 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:13 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:05:13.939 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:13 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:13 smithi165 ceph-mon[34868]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:13 smithi165 ceph-mon[34868]: Standby manager daemon x started 2024-08-17T17:05:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:13 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.165:0/4132620753' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:05:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:13 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.165:0/4132620753' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:05:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:13 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.165:0/4132620753' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:05:14.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:13 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.165:0/4132620753' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:05:14.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:13 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:05:14.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:13 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:05:14.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:13 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:14.926 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:14 smithi136 ceph-mon[31034]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:05:14.926 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:14 smithi136 ceph-mon[31034]: mgrmap e14: y(active, since 63s), standbys: x 2024-08-17T17:05:14.926 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:14 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-17T17:05:14.926 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:14 smithi136 ceph-mon[42580]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:05:14.926 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:14 smithi136 ceph-mon[42580]: mgrmap e14: y(active, since 63s), standbys: x 2024-08-17T17:05:14.927 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:14 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-17T17:05:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:14 smithi165 ceph-mon[34868]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:05:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:14 smithi165 ceph-mon[34868]: mgrmap e14: y(active, since 63s), standbys: x 2024-08-17T17:05:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:14 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-17T17:05:15.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:15 smithi136 ceph-mon[31034]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:15.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:15 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/3401960716' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51eb458d-9f33-4b87-b7f5-8acb076c16ae"}]: dispatch 2024-08-17T17:05:15.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:15 smithi136 ceph-mon[31034]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51eb458d-9f33-4b87-b7f5-8acb076c16ae"}]: dispatch 2024-08-17T17:05:15.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:15 smithi136 ceph-mon[31034]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51eb458d-9f33-4b87-b7f5-8acb076c16ae"}]': finished 2024-08-17T17:05:15.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:15 smithi136 ceph-mon[31034]: osdmap e5: 1 total, 0 up, 1 in 2024-08-17T17:05:15.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:15 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:05:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:15 smithi136 ceph-mon[42580]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:15 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/3401960716' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51eb458d-9f33-4b87-b7f5-8acb076c16ae"}]: dispatch 2024-08-17T17:05:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:15 smithi136 ceph-mon[42580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51eb458d-9f33-4b87-b7f5-8acb076c16ae"}]: dispatch 2024-08-17T17:05:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:15 smithi136 ceph-mon[42580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51eb458d-9f33-4b87-b7f5-8acb076c16ae"}]': finished 2024-08-17T17:05:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:15 smithi136 ceph-mon[42580]: osdmap e5: 1 total, 0 up, 1 in 2024-08-17T17:05:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:15 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:05:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:15 smithi165 ceph-mon[34868]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:15 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/3401960716' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51eb458d-9f33-4b87-b7f5-8acb076c16ae"}]: dispatch 2024-08-17T17:05:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:15 smithi165 ceph-mon[34868]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51eb458d-9f33-4b87-b7f5-8acb076c16ae"}]: dispatch 2024-08-17T17:05:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:15 smithi165 ceph-mon[34868]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51eb458d-9f33-4b87-b7f5-8acb076c16ae"}]': finished 2024-08-17T17:05:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:15 smithi165 ceph-mon[34868]: osdmap e5: 1 total, 0 up, 1 in 2024-08-17T17:05:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:15 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:05:16.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:16 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/306220446' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:05:16.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:16 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/306220446' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:05:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:16 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/306220446' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:05:17.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:17 smithi136 ceph-mon[31034]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:17.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:17 smithi136 ceph-mon[42580]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:17 smithi165 ceph-mon[34868]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:19.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:19 smithi136 ceph-mon[31034]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:19.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:19 smithi136 ceph-mon[42580]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:19 smithi165 ceph-mon[34868]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:21.900 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:21 smithi136 ceph-mon[42580]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:21.900 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:21 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-08-17T17:05:21.900 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:21 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:21.901 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:21 smithi136 ceph-mon[31034]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:21.901 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:21 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-08-17T17:05:21.901 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:21 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:21 smithi165 ceph-mon[34868]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:21 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-08-17T17:05:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:21 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:22.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:22 smithi136 ceph-mon[31034]: Deploying daemon osd.0 on smithi136 2024-08-17T17:05:22.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:22 smithi136 ceph-mon[42580]: Deploying daemon osd.0 on smithi136 2024-08-17T17:05:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:22 smithi165 ceph-mon[34868]: Deploying daemon osd.0 on smithi136 2024-08-17T17:05:23.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:23 smithi136 ceph-mon[31034]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:23.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:23 smithi136 ceph-mon[42580]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:23 smithi165 ceph-mon[34868]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:25.860 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:25 smithi136 ceph-mon[31034]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:25.860 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:25 smithi136 ceph-mon[42580]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:25 smithi165 ceph-mon[34868]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:26.891 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:26 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:26.891 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:26 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:26.891 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:26 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:26.892 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:26 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:26.892 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:26 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:26.892 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:26 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:26 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:26 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:26 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:27.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:27 smithi136 ceph-mon[31034]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:27.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:27 smithi136 ceph-mon[42580]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:28.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:27 smithi165 ceph-mon[34868]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:29.773 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:29 smithi136 ceph-mon[42580]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:29.773 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:29 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:29.773 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:29 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:29.774 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:29 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:29.774 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:29 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:29.774 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:29 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:29.774 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:29 smithi136 ceph-mon[42580]: from='osd.0 [v2:172.21.15.136:6802/3864095089,v1:172.21.15.136:6803/3864095089]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-08-17T17:05:29.774 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:29 smithi136 ceph-mon[42580]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-08-17T17:05:29.774 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:29 smithi136 ceph-mon[31034]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:29.774 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:29 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:29.774 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:29 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:29.774 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:29 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:29.774 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:29 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:29.775 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:29 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:29.775 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:29 smithi136 ceph-mon[31034]: from='osd.0 [v2:172.21.15.136:6802/3864095089,v1:172.21.15.136:6803/3864095089]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-08-17T17:05:29.775 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:29 smithi136 ceph-mon[31034]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-08-17T17:05:29.807 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 0 on host 'smithi136' 2024-08-17T17:05:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:29 smithi165 ceph-mon[34868]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:29 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:29 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:30.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:29 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:30.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:29 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:30.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:29 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:30.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:29 smithi165 ceph-mon[34868]: from='osd.0 [v2:172.21.15.136:6802/3864095089,v1:172.21.15.136:6803/3864095089]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-08-17T17:05:30.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:29 smithi165 ceph-mon[34868]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-08-17T17:05:30.385 DEBUG:teuthology.orchestra.run.smithi136:osd.0> sudo journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.0.service 2024-08-17T17:05:30.388 INFO:tasks.cephadm:Deploying osd.1 on smithi136 with /dev/vg_nvme/lv_3... 2024-08-17T17:05:30.388 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-08-17T17:05:30.611 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:30 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:30.611 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:30 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:30.612 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:30 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:30.612 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:30 smithi136 ceph-mon[31034]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-17T17:05:30.612 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:30 smithi136 ceph-mon[31034]: osdmap e6: 1 total, 0 up, 1 in 2024-08-17T17:05:30.612 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:30 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:05:30.612 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:30 smithi136 ceph-mon[31034]: from='osd.0 [v2:172.21.15.136:6802/3864095089,v1:172.21.15.136:6803/3864095089]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-08-17T17:05:30.612 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:30 smithi136 ceph-mon[31034]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-08-17T17:05:30.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:30 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:30.889 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:30 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:30.889 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:30 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:30.889 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:30 smithi136 ceph-mon[42580]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-17T17:05:30.889 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:30 smithi136 ceph-mon[42580]: osdmap e6: 1 total, 0 up, 1 in 2024-08-17T17:05:30.889 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:30 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:05:30.889 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:30 smithi136 ceph-mon[42580]: from='osd.0 [v2:172.21.15.136:6802/3864095089,v1:172.21.15.136:6803/3864095089]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-08-17T17:05:30.889 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:30 smithi136 ceph-mon[42580]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-08-17T17:05:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:30 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:30 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:30 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:30 smithi165 ceph-mon[34868]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-17T17:05:31.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:30 smithi165 ceph-mon[34868]: osdmap e6: 1 total, 0 up, 1 in 2024-08-17T17:05:31.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:30 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:05:31.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:30 smithi165 ceph-mon[34868]: from='osd.0 [v2:172.21.15.136:6802/3864095089,v1:172.21.15.136:6803/3864095089]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-08-17T17:05:31.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:30 smithi165 ceph-mon[34868]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-08-17T17:05:31.078 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:05:31.744 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:31 smithi136 ceph-mon[31034]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:31.744 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:31 smithi136 ceph-mon[31034]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-08-17T17:05:31.744 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:31 smithi136 ceph-mon[31034]: osdmap e7: 1 total, 0 up, 1 in 2024-08-17T17:05:31.744 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:31 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:05:31.744 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:31 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:05:31.745 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:31 smithi136 ceph-mon[42580]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:31.745 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:31 smithi136 ceph-mon[42580]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-08-17T17:05:31.745 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:31 smithi136 ceph-mon[42580]: osdmap e7: 1 total, 0 up, 1 in 2024-08-17T17:05:31.745 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:31 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:05:31.745 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:31 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:05:31.745 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:05:31 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0[46890]: 2024-08-17T17:05:31.502+0000 7f1c7d2eb640 -1 osd.0 0 waiting for initial osdmap 2024-08-17T17:05:31.745 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:05:31 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0[46890]: 2024-08-17T17:05:31.504+0000 7f1c79114640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:05:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:31 smithi165 ceph-mon[34868]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:31 smithi165 ceph-mon[34868]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-08-17T17:05:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:31 smithi165 ceph-mon[34868]: osdmap e7: 1 total, 0 up, 1 in 2024-08-17T17:05:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:31 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:05:32.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:31 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:05:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:33 smithi165 ceph-mon[34868]: purged_snaps scrub starts 2024-08-17T17:05:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:33 smithi165 ceph-mon[34868]: purged_snaps scrub ok 2024-08-17T17:05:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:33 smithi165 ceph-mon[34868]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:33 smithi165 ceph-mon[34868]: osd.0 [v2:172.21.15.136:6802/3864095089,v1:172.21.15.136:6803/3864095089] boot 2024-08-17T17:05:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:33 smithi165 ceph-mon[34868]: osdmap e8: 1 total, 1 up, 1 in 2024-08-17T17:05:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:33 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:05:33.763 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:33 smithi136 ceph-mon[31034]: purged_snaps scrub starts 2024-08-17T17:05:33.763 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:33 smithi136 ceph-mon[31034]: purged_snaps scrub ok 2024-08-17T17:05:33.763 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:33 smithi136 ceph-mon[31034]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:33.763 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:33 smithi136 ceph-mon[31034]: osd.0 [v2:172.21.15.136:6802/3864095089,v1:172.21.15.136:6803/3864095089] boot 2024-08-17T17:05:33.764 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:33 smithi136 ceph-mon[31034]: osdmap e8: 1 total, 1 up, 1 in 2024-08-17T17:05:33.764 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:33 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:05:33.764 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:33 smithi136 ceph-mon[42580]: purged_snaps scrub starts 2024-08-17T17:05:33.764 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:33 smithi136 ceph-mon[42580]: purged_snaps scrub ok 2024-08-17T17:05:33.764 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:33 smithi136 ceph-mon[42580]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:05:33.764 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:33 smithi136 ceph-mon[42580]: osd.0 [v2:172.21.15.136:6802/3864095089,v1:172.21.15.136:6803/3864095089] boot 2024-08-17T17:05:33.764 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:33 smithi136 ceph-mon[42580]: osdmap e8: 1 total, 1 up, 1 in 2024-08-17T17:05:33.764 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:33 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:05:34.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:34 smithi136 ceph-mon[31034]: osdmap e9: 1 total, 1 up, 1 in 2024-08-17T17:05:34.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:34 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:34.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:34 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:34.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:34 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:34.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:34 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:34.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:34 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:34.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:34 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:34.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:34 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:34.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:34 smithi136 ceph-mon[42580]: osdmap e9: 1 total, 1 up, 1 in 2024-08-17T17:05:34.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:34 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:34.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:34 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:34.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:34 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:34.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:34 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:34.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:34 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:34.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:34 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:34.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:34 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:34.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:34 smithi165 ceph-mon[34868]: osdmap e9: 1 total, 1 up, 1 in 2024-08-17T17:05:34.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:34 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:34.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:34 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:34.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:34 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:34.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:34 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:34.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:34 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:34.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:34 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:34.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:34 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:35.365 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:05:35.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi136:vg_nvme/lv_3 2024-08-17T17:05:35.584 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:05:35.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:35 smithi136 ceph-mon[31034]: Detected new or changed devices on smithi136 2024-08-17T17:05:35.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:35 smithi136 ceph-mon[31034]: Adjusting osd_memory_target on smithi136 to 16064M 2024-08-17T17:05:35.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:35 smithi136 ceph-mon[31034]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:35 smithi136 ceph-mon[42580]: Detected new or changed devices on smithi136 2024-08-17T17:05:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:35 smithi136 ceph-mon[42580]: Adjusting osd_memory_target on smithi136 to 16064M 2024-08-17T17:05:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:35 smithi136 ceph-mon[42580]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:35 smithi165 ceph-mon[34868]: Detected new or changed devices on smithi136 2024-08-17T17:05:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:35 smithi165 ceph-mon[34868]: Adjusting osd_memory_target on smithi136 to 16064M 2024-08-17T17:05:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:35 smithi165 ceph-mon[34868]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:37.772 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:37 smithi136 ceph-mon[31034]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:37.772 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:37 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:05:37.772 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:37 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:05:37.773 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:37 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:37.773 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:37 smithi136 ceph-mon[42580]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:37.773 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:37 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:05:37.773 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:37 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:05:37.773 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:37 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:37 smithi165 ceph-mon[34868]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:37 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:05:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:37 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:05:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:37 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:38.601 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:38 smithi136 ceph-mon[31034]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:05:38.601 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:38 smithi136 ceph-mon[42580]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:05:39.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:38 smithi165 ceph-mon[34868]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:05:39.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:39 smithi136 ceph-mon[31034]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:39.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:39 smithi136 ceph-mon[42580]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:39 smithi165 ceph-mon[34868]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:40.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:40 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/2597132202' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa73fa16-8736-478c-b07a-e674c18698ef"}]: dispatch 2024-08-17T17:05:40.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:40 smithi136 ceph-mon[42580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa73fa16-8736-478c-b07a-e674c18698ef"}]: dispatch 2024-08-17T17:05:40.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:40 smithi136 ceph-mon[42580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa73fa16-8736-478c-b07a-e674c18698ef"}]': finished 2024-08-17T17:05:40.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:40 smithi136 ceph-mon[42580]: osdmap e10: 2 total, 1 up, 2 in 2024-08-17T17:05:40.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:40 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:05:40.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:40 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/858594140' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:05:40.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:40 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/2597132202' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa73fa16-8736-478c-b07a-e674c18698ef"}]: dispatch 2024-08-17T17:05:40.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:40 smithi136 ceph-mon[31034]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa73fa16-8736-478c-b07a-e674c18698ef"}]: dispatch 2024-08-17T17:05:40.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:40 smithi136 ceph-mon[31034]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa73fa16-8736-478c-b07a-e674c18698ef"}]': finished 2024-08-17T17:05:40.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:40 smithi136 ceph-mon[31034]: osdmap e10: 2 total, 1 up, 2 in 2024-08-17T17:05:40.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:40 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:05:40.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:40 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/858594140' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:05:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:40 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2597132202' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa73fa16-8736-478c-b07a-e674c18698ef"}]: dispatch 2024-08-17T17:05:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:40 smithi165 ceph-mon[34868]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa73fa16-8736-478c-b07a-e674c18698ef"}]: dispatch 2024-08-17T17:05:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:40 smithi165 ceph-mon[34868]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa73fa16-8736-478c-b07a-e674c18698ef"}]': finished 2024-08-17T17:05:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:40 smithi165 ceph-mon[34868]: osdmap e10: 2 total, 1 up, 2 in 2024-08-17T17:05:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:40 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:05:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:40 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/858594140' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:05:41.848 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:41 smithi136 ceph-mon[31034]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:41.849 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:41 smithi136 ceph-mon[42580]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:42.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:41 smithi165 ceph-mon[34868]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:43.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:43 smithi136 ceph-mon[31034]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:43.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:43 smithi136 ceph-mon[42580]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:43 smithi165 ceph-mon[34868]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:45 smithi136 ceph-mon[31034]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:45.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:45 smithi136 ceph-mon[42580]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:46.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:45 smithi165 ceph-mon[34868]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:46 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-08-17T17:05:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:46 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:46 smithi136 ceph-mon[42580]: Deploying daemon osd.1 on smithi136 2024-08-17T17:05:46.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:46 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-08-17T17:05:46.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:46 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:46.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:46 smithi136 ceph-mon[31034]: Deploying daemon osd.1 on smithi136 2024-08-17T17:05:47.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:46 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-08-17T17:05:47.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:46 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:47.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:46 smithi165 ceph-mon[34868]: Deploying daemon osd.1 on smithi136 2024-08-17T17:05:47.585 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:47 smithi136 ceph-mon[31034]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:47.900 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:47 smithi136 ceph-mon[42580]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:47 smithi165 ceph-mon[34868]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:49.882 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:49 smithi136 ceph-mon[31034]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:49.883 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:49 smithi136 ceph-mon[42580]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:49 smithi165 ceph-mon[34868]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:50.890 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:50 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:50.890 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:50 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:50.890 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:50 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:50.890 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:50 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:50.890 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:50 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:50.890 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:50 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:50 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:50 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:50 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:51.853 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:51 smithi136 ceph-mon[31034]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:51.854 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:51 smithi136 ceph-mon[42580]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:51 smithi165 ceph-mon[34868]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:53.088 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 1 on host 'smithi136' 2024-08-17T17:05:53.692 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:53 smithi136 ceph-mon[31034]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:53.692 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:53 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:53.692 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:53 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:53.692 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:53 smithi136 ceph-mon[31034]: from='osd.1 [v2:172.21.15.136:6810/3797355920,v1:172.21.15.136:6811/3797355920]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-08-17T17:05:53.692 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:53 smithi136 ceph-mon[42580]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:53.692 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:53 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:53.692 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:53 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:53.692 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:53 smithi136 ceph-mon[42580]: from='osd.1 [v2:172.21.15.136:6810/3797355920,v1:172.21.15.136:6811/3797355920]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-08-17T17:05:53.714 DEBUG:teuthology.orchestra.run.smithi136:osd.1> sudo journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.1.service 2024-08-17T17:05:53.716 INFO:tasks.cephadm:Deploying osd.2 on smithi136 with /dev/vg_nvme/lv_2... 2024-08-17T17:05:53.716 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-08-17T17:05:53.972 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:05:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:53 smithi165 ceph-mon[34868]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:53 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:53 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:53 smithi165 ceph-mon[34868]: from='osd.1 [v2:172.21.15.136:6810/3797355920,v1:172.21.15.136:6811/3797355920]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-08-17T17:05:55.099 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:54 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:55.099 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:54 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:55.099 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:54 smithi136 ceph-mon[31034]: from='osd.1 [v2:172.21.15.136:6810/3797355920,v1:172.21.15.136:6811/3797355920]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-17T17:05:55.099 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:54 smithi136 ceph-mon[31034]: osdmap e11: 2 total, 1 up, 2 in 2024-08-17T17:05:55.099 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:54 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:05:55.099 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:54 smithi136 ceph-mon[31034]: from='osd.1 [v2:172.21.15.136:6810/3797355920,v1:172.21.15.136:6811/3797355920]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-08-17T17:05:55.099 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:54 smithi136 ceph-mon[31034]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:55.099 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:54 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:55.099 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:54 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:55.099 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:54 smithi136 ceph-mon[42580]: from='osd.1 [v2:172.21.15.136:6810/3797355920,v1:172.21.15.136:6811/3797355920]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-17T17:05:55.100 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:54 smithi136 ceph-mon[42580]: osdmap e11: 2 total, 1 up, 2 in 2024-08-17T17:05:55.100 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:54 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:05:55.100 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:54 smithi136 ceph-mon[42580]: from='osd.1 [v2:172.21.15.136:6810/3797355920,v1:172.21.15.136:6811/3797355920]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-08-17T17:05:55.100 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:54 smithi136 ceph-mon[42580]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:54 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:54 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:05:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:54 smithi165 ceph-mon[34868]: from='osd.1 [v2:172.21.15.136:6810/3797355920,v1:172.21.15.136:6811/3797355920]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-17T17:05:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:54 smithi165 ceph-mon[34868]: osdmap e11: 2 total, 1 up, 2 in 2024-08-17T17:05:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:54 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:05:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:54 smithi165 ceph-mon[34868]: from='osd.1 [v2:172.21.15.136:6810/3797355920,v1:172.21.15.136:6811/3797355920]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-08-17T17:05:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:54 smithi165 ceph-mon[34868]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:05:55.446 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:05:55 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1[53185]: 2024-08-17T17:05:55.096+0000 7f44c48df640 -1 osd.1 0 waiting for initial osdmap 2024-08-17T17:05:55.446 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:05:55 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1[53185]: 2024-08-17T17:05:55.098+0000 7f44c0708640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:05:56.193 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:56 smithi136 ceph-mon[31034]: from='osd.1 [v2:172.21.15.136:6810/3797355920,v1:172.21.15.136:6811/3797355920]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-08-17T17:05:56.193 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:56 smithi136 ceph-mon[31034]: osdmap e12: 2 total, 1 up, 2 in 2024-08-17T17:05:56.193 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:56 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:05:56.193 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:56 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:05:56.193 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:56 smithi136 ceph-mon[42580]: from='osd.1 [v2:172.21.15.136:6810/3797355920,v1:172.21.15.136:6811/3797355920]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-08-17T17:05:56.194 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:56 smithi136 ceph-mon[42580]: osdmap e12: 2 total, 1 up, 2 in 2024-08-17T17:05:56.194 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:56 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:05:56.194 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:56 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:05:56.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:56 smithi165 ceph-mon[34868]: from='osd.1 [v2:172.21.15.136:6810/3797355920,v1:172.21.15.136:6811/3797355920]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-08-17T17:05:56.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:56 smithi165 ceph-mon[34868]: osdmap e12: 2 total, 1 up, 2 in 2024-08-17T17:05:56.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:56 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:05:56.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:56 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:05:57.226 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:05:57.243 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:57 smithi136 ceph-mon[31034]: purged_snaps scrub starts 2024-08-17T17:05:57.243 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:57 smithi136 ceph-mon[31034]: purged_snaps scrub ok 2024-08-17T17:05:57.244 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:57 smithi136 ceph-mon[31034]: osd.1 [v2:172.21.15.136:6810/3797355920,v1:172.21.15.136:6811/3797355920] boot 2024-08-17T17:05:57.244 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:57 smithi136 ceph-mon[31034]: osdmap e13: 2 total, 2 up, 2 in 2024-08-17T17:05:57.244 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:57 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:05:57.244 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:57 smithi136 ceph-mon[31034]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:05:57.244 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:57 smithi136 ceph-mon[42580]: purged_snaps scrub starts 2024-08-17T17:05:57.244 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:57 smithi136 ceph-mon[42580]: purged_snaps scrub ok 2024-08-17T17:05:57.244 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:57 smithi136 ceph-mon[42580]: osd.1 [v2:172.21.15.136:6810/3797355920,v1:172.21.15.136:6811/3797355920] boot 2024-08-17T17:05:57.244 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:57 smithi136 ceph-mon[42580]: osdmap e13: 2 total, 2 up, 2 in 2024-08-17T17:05:57.244 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:57 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:05:57.244 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:57 smithi136 ceph-mon[42580]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:05:57.245 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi136:vg_nvme/lv_2 2024-08-17T17:05:57.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:57 smithi165 ceph-mon[34868]: purged_snaps scrub starts 2024-08-17T17:05:57.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:57 smithi165 ceph-mon[34868]: purged_snaps scrub ok 2024-08-17T17:05:57.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:57 smithi165 ceph-mon[34868]: osd.1 [v2:172.21.15.136:6810/3797355920,v1:172.21.15.136:6811/3797355920] boot 2024-08-17T17:05:57.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:57 smithi165 ceph-mon[34868]: osdmap e13: 2 total, 2 up, 2 in 2024-08-17T17:05:57.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:57 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:05:57.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:57 smithi165 ceph-mon[34868]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:05:57.835 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:05:58.263 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:58 smithi136 ceph-mon[31034]: osdmap e14: 2 total, 2 up, 2 in 2024-08-17T17:05:58.264 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:58 smithi136 ceph-mon[42580]: osdmap e14: 2 total, 2 up, 2 in 2024-08-17T17:05:58.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:58 smithi165 ceph-mon[34868]: osdmap e14: 2 total, 2 up, 2 in 2024-08-17T17:05:59.127 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:05:59 smithi136 ceph-mon[31034]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:05:59.127 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:05:59 smithi136 ceph-mon[42580]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:05:59.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:05:59 smithi165 ceph-mon[34868]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:00.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[31034]: Detected new or changed devices on smithi136 2024-08-17T17:06:00.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:00.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:00.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:00.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:00.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[31034]: Adjusting osd_memory_target on smithi136 to 8032M 2024-08-17T17:06:00.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:00.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:00.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:06:00.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:00.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:06:00.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:00.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:06:00.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:00.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[42580]: Detected new or changed devices on smithi136 2024-08-17T17:06:00.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:00.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:00.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:00.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:00.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[42580]: Adjusting osd_memory_target on smithi136 to 8032M 2024-08-17T17:06:00.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:00.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:00.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:06:00.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:00.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:06:00.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:00.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:06:00.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:00 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:00 smithi165 ceph-mon[34868]: Detected new or changed devices on smithi136 2024-08-17T17:06:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:00 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:00 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:00 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:00 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:00 smithi165 ceph-mon[34868]: Adjusting osd_memory_target on smithi136 to 8032M 2024-08-17T17:06:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:00 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:00 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:00 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:06:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:00 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:00 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:06:00.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:00 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:00.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:00 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:06:00.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:00 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:01.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:01 smithi136 ceph-mon[42580]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:06:01.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:01 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:06:01.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:01 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:06:01.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:01 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:01.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:01 smithi136 ceph-mon[42580]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:01.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:01 smithi136 ceph-mon[31034]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:06:01.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:01 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:06:01.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:01 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:06:01.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:01 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:01.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:01 smithi136 ceph-mon[31034]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:01.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:01 smithi165 ceph-mon[34868]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:06:01.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:01 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:06:01.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:01 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:06:01.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:01 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:01.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:01 smithi165 ceph-mon[34868]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:02.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:02 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/3458403629' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d2612be-7068-48d9-bb0c-992ebccaa091"}]: dispatch 2024-08-17T17:06:02.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:02 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/3458403629' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d2612be-7068-48d9-bb0c-992ebccaa091"}]': finished 2024-08-17T17:06:02.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:02 smithi136 ceph-mon[42580]: osdmap e15: 3 total, 2 up, 3 in 2024-08-17T17:06:02.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:02 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:06:02.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:02 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/3458403629' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d2612be-7068-48d9-bb0c-992ebccaa091"}]: dispatch 2024-08-17T17:06:02.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:02 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/3458403629' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d2612be-7068-48d9-bb0c-992ebccaa091"}]': finished 2024-08-17T17:06:02.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:02 smithi136 ceph-mon[31034]: osdmap e15: 3 total, 2 up, 3 in 2024-08-17T17:06:02.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:02 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:06:03.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:02 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/3458403629' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d2612be-7068-48d9-bb0c-992ebccaa091"}]: dispatch 2024-08-17T17:06:03.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:02 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/3458403629' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d2612be-7068-48d9-bb0c-992ebccaa091"}]': finished 2024-08-17T17:06:03.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:02 smithi165 ceph-mon[34868]: osdmap e15: 3 total, 2 up, 3 in 2024-08-17T17:06:03.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:02 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:06:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:03 smithi136 ceph-mon[31034]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:03 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/1196339849' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:06:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:03 smithi136 ceph-mon[42580]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:03 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/1196339849' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:06:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:03 smithi165 ceph-mon[34868]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:03 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/1196339849' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:06:05.865 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:05 smithi136 ceph-mon[31034]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:05.865 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:05 smithi136 ceph-mon[42580]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:05 smithi165 ceph-mon[34868]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:07.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:07 smithi136 ceph-mon[31034]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:07.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:07 smithi136 ceph-mon[42580]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:08.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:07 smithi165 ceph-mon[34868]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:08.924 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:08 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-08-17T17:06:08.924 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:08 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:08.924 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:08 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-08-17T17:06:08.925 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:08 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:09.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:08 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-08-17T17:06:09.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:08 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:09.636 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:09 smithi136 ceph-mon[31034]: Deploying daemon osd.2 on smithi136 2024-08-17T17:06:09.637 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:09 smithi136 ceph-mon[31034]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:09.637 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:09 smithi136 ceph-mon[42580]: Deploying daemon osd.2 on smithi136 2024-08-17T17:06:09.637 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:09 smithi136 ceph-mon[42580]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:10.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:09 smithi165 ceph-mon[34868]: Deploying daemon osd.2 on smithi136 2024-08-17T17:06:10.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:09 smithi165 ceph-mon[34868]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:11.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:11 smithi136 ceph-mon[31034]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:11 smithi136 ceph-mon[42580]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:11 smithi165 ceph-mon[34868]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:13.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:13 smithi136 ceph-mon[31034]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:13.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:13 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:06:13.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:13 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:13.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:13 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:13.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:13 smithi136 ceph-mon[42580]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:13.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:13 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:06:13.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:13 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:13.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:13 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:13 smithi165 ceph-mon[34868]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:13 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:06:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:13 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:13 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:15.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:15 smithi136 ceph-mon[31034]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:15.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:15 smithi136 ceph-mon[42580]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:15 smithi165 ceph-mon[34868]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:16.858 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 2 on host 'smithi136' 2024-08-17T17:06:17.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:17 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:17.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:17 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:17.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:17 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:17.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:17 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:06:17.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:17 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:17.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:17 smithi136 ceph-mon[31034]: from='osd.2 [v2:172.21.15.136:6818/1674243363,v1:172.21.15.136:6819/1674243363]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-17T17:06:17.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:17 smithi136 ceph-mon[31034]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-17T17:06:17.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:17 smithi136 ceph-mon[31034]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:17.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:17 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:06:17.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:17 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:17.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:17 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:17.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:17 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:17.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:17 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:17.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:17 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:17.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:17 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:06:17.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:17 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:17.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:17 smithi136 ceph-mon[42580]: from='osd.2 [v2:172.21.15.136:6818/1674243363,v1:172.21.15.136:6819/1674243363]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-17T17:06:17.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:17 smithi136 ceph-mon[42580]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-17T17:06:17.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:17 smithi136 ceph-mon[42580]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:17.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:17 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:06:17.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:17 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:17.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:17 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:17.448 DEBUG:teuthology.orchestra.run.smithi136:osd.2> sudo journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.2.service 2024-08-17T17:06:17.450 INFO:tasks.cephadm:Deploying osd.3 on smithi136 with /dev/vg_nvme/lv_1... 2024-08-17T17:06:17.450 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-08-17T17:06:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:17 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:17 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:17 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:17 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:06:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:17 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:17 smithi165 ceph-mon[34868]: from='osd.2 [v2:172.21.15.136:6818/1674243363,v1:172.21.15.136:6819/1674243363]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-17T17:06:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:17 smithi165 ceph-mon[34868]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-17T17:06:17.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:17 smithi165 ceph-mon[34868]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:17.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:17 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:06:17.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:17 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:17.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:17 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:18.208 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:06:18.441 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:18 smithi136 ceph-mon[42580]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-17T17:06:18.441 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:18 smithi136 ceph-mon[42580]: osdmap e16: 3 total, 2 up, 3 in 2024-08-17T17:06:18.441 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:18 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:06:18.441 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:18 smithi136 ceph-mon[42580]: from='osd.2 [v2:172.21.15.136:6818/1674243363,v1:172.21.15.136:6819/1674243363]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-08-17T17:06:18.441 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:18 smithi136 ceph-mon[42580]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-08-17T17:06:18.441 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:18 smithi136 ceph-mon[31034]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-17T17:06:18.441 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:18 smithi136 ceph-mon[31034]: osdmap e16: 3 total, 2 up, 3 in 2024-08-17T17:06:18.442 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:18 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:06:18.442 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:18 smithi136 ceph-mon[31034]: from='osd.2 [v2:172.21.15.136:6818/1674243363,v1:172.21.15.136:6819/1674243363]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-08-17T17:06:18.442 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:18 smithi136 ceph-mon[31034]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-08-17T17:06:18.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:18 smithi165 ceph-mon[34868]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-17T17:06:18.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:18 smithi165 ceph-mon[34868]: osdmap e16: 3 total, 2 up, 3 in 2024-08-17T17:06:18.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:18 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:06:18.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:18 smithi165 ceph-mon[34868]: from='osd.2 [v2:172.21.15.136:6818/1674243363,v1:172.21.15.136:6819/1674243363]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-08-17T17:06:18.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:18 smithi165 ceph-mon[34868]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-08-17T17:06:18.697 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:06:18 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2[59404]: 2024-08-17T17:06:18.470+0000 7f5b3479d640 -1 osd.2 0 waiting for initial osdmap 2024-08-17T17:06:18.697 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:06:18 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2[59404]: 2024-08-17T17:06:18.473+0000 7f5b2bda7640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:06:19.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:19 smithi136 ceph-mon[31034]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-08-17T17:06:19.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:19 smithi136 ceph-mon[31034]: osdmap e17: 3 total, 2 up, 3 in 2024-08-17T17:06:19.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:19 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:06:19.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:19 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:06:19.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:19 smithi136 ceph-mon[31034]: from='osd.2 ' entity='osd.2' 2024-08-17T17:06:19.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:19 smithi136 ceph-mon[31034]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:19.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:19 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:06:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:19 smithi136 ceph-mon[42580]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-08-17T17:06:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:19 smithi136 ceph-mon[42580]: osdmap e17: 3 total, 2 up, 3 in 2024-08-17T17:06:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:19 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:06:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:19 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:06:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:19 smithi136 ceph-mon[42580]: from='osd.2 ' entity='osd.2' 2024-08-17T17:06:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:19 smithi136 ceph-mon[42580]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:19 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:06:19.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:19 smithi165 ceph-mon[34868]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-08-17T17:06:19.514 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:19 smithi165 ceph-mon[34868]: osdmap e17: 3 total, 2 up, 3 in 2024-08-17T17:06:19.514 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:19 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:06:19.514 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:19 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:06:19.514 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:19 smithi165 ceph-mon[34868]: from='osd.2 ' entity='osd.2' 2024-08-17T17:06:19.514 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:19 smithi165 ceph-mon[34868]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-17T17:06:19.514 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:19 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:06:20.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:20 smithi136 ceph-mon[31034]: purged_snaps scrub starts 2024-08-17T17:06:20.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:20 smithi136 ceph-mon[31034]: purged_snaps scrub ok 2024-08-17T17:06:20.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:20 smithi136 ceph-mon[31034]: osd.2 [v2:172.21.15.136:6818/1674243363,v1:172.21.15.136:6819/1674243363] boot 2024-08-17T17:06:20.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:20 smithi136 ceph-mon[31034]: osdmap e18: 3 total, 3 up, 3 in 2024-08-17T17:06:20.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:20 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:06:20.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:20 smithi136 ceph-mon[42580]: purged_snaps scrub starts 2024-08-17T17:06:20.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:20 smithi136 ceph-mon[42580]: purged_snaps scrub ok 2024-08-17T17:06:20.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:20 smithi136 ceph-mon[42580]: osd.2 [v2:172.21.15.136:6818/1674243363,v1:172.21.15.136:6819/1674243363] boot 2024-08-17T17:06:20.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:20 smithi136 ceph-mon[42580]: osdmap e18: 3 total, 3 up, 3 in 2024-08-17T17:06:20.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:20 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:06:20.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:20 smithi165 ceph-mon[34868]: purged_snaps scrub starts 2024-08-17T17:06:20.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:20 smithi165 ceph-mon[34868]: purged_snaps scrub ok 2024-08-17T17:06:20.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:20 smithi165 ceph-mon[34868]: osd.2 [v2:172.21.15.136:6818/1674243363,v1:172.21.15.136:6819/1674243363] boot 2024-08-17T17:06:20.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:20 smithi165 ceph-mon[34868]: osdmap e18: 3 total, 3 up, 3 in 2024-08-17T17:06:20.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:20 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:06:21.230 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:21 smithi136 ceph-mon[31034]: pgmap v73: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:21.230 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:21 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:06:21.231 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:21 smithi136 ceph-mon[42580]: pgmap v73: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:21.231 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:21 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:06:21.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:21 smithi165 ceph-mon[34868]: pgmap v73: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:21.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:21 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:06:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-08-17T17:06:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[42580]: osdmap e19: 3 total, 3 up, 3 in 2024-08-17T17:06:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:06:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[42580]: Detected new or changed devices on smithi136 2024-08-17T17:06:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[42580]: Adjusting osd_memory_target on smithi136 to 5354M 2024-08-17T17:06:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:22.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:22.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:06:22.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:22.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-08-17T17:06:22.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[31034]: osdmap e19: 3 total, 3 up, 3 in 2024-08-17T17:06:22.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:06:22.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[31034]: Detected new or changed devices on smithi136 2024-08-17T17:06:22.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:22.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:22.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:22.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:22.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:22.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[31034]: Adjusting osd_memory_target on smithi136 to 5354M 2024-08-17T17:06:22.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:22.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:22.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:06:22.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:22 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:22 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-08-17T17:06:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:22 smithi165 ceph-mon[34868]: osdmap e19: 3 total, 3 up, 3 in 2024-08-17T17:06:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:22 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:06:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:22 smithi165 ceph-mon[34868]: Detected new or changed devices on smithi136 2024-08-17T17:06:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:22 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:22 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:22 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:22 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:22.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:22 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:22.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:22 smithi165 ceph-mon[34868]: Adjusting osd_memory_target on smithi136 to 5354M 2024-08-17T17:06:22.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:22 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:22.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:22 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:22.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:22 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:06:22.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:22 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:22.592 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:06:22.612 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi136:vg_nvme/lv_1 2024-08-17T17:06:22.833 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:06:23.216 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:23 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:06:23.217 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:23 smithi136 ceph-mon[31034]: osdmap e20: 3 total, 3 up, 3 in 2024-08-17T17:06:23.217 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:23 smithi136 ceph-mon[31034]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:23.217 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:23 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:06:23.217 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:23 smithi136 ceph-mon[42580]: osdmap e20: 3 total, 3 up, 3 in 2024-08-17T17:06:23.217 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:23 smithi136 ceph-mon[42580]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:23.217 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:06:22 smithi136 sudo[63529]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-17T17:06:23.217 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:06:22 smithi136 sudo[63529]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-17T17:06:23.218 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:06:22 smithi136 sudo[63529]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-17T17:06:23.218 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:06:22 smithi136 sudo[63529]: pam_unix(sudo:session): session closed for user root 2024-08-17T17:06:23.501 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:06:23 smithi136 sudo[63560]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-17T17:06:23.501 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:06:23 smithi136 sudo[63560]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-17T17:06:23.501 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:06:23 smithi136 sudo[63560]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-17T17:06:23.501 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:06:23 smithi136 sudo[63560]: pam_unix(sudo:session): session closed for user root 2024-08-17T17:06:23.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:23 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:06:23.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:23 smithi165 ceph-mon[34868]: osdmap e20: 3 total, 3 up, 3 in 2024-08-17T17:06:23.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:23 smithi165 ceph-mon[34868]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:23.760 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:06:23 smithi136 sudo[63574]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-17T17:06:23.760 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:06:23 smithi136 sudo[63574]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-17T17:06:23.760 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:06:23 smithi136 sudo[63574]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-17T17:06:23.760 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:06:23 smithi136 sudo[63574]: pam_unix(sudo:session): session closed for user root 2024-08-17T17:06:24.019 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:06:23 smithi136 sudo[63583]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-17T17:06:24.020 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:06:23 smithi136 sudo[63583]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-17T17:06:24.020 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:06:23 smithi136 sudo[63583]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-17T17:06:24.020 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:06:23 smithi136 sudo[63583]: pam_unix(sudo:session): session closed for user root 2024-08-17T17:06:24.283 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:24 smithi136 ceph-mon[31034]: osdmap e21: 3 total, 3 up, 3 in 2024-08-17T17:06:24.284 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:24 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:24.284 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:24 smithi136 ceph-mon[42580]: osdmap e21: 3 total, 3 up, 3 in 2024-08-17T17:06:24.284 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:24 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:24.284 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:06:24 smithi136 sudo[63607]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-17T17:06:24.284 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:06:24 smithi136 sudo[63607]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-17T17:06:24.284 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:06:24 smithi136 sudo[63607]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-17T17:06:24.284 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:06:24 smithi136 sudo[63607]: pam_unix(sudo:session): session closed for user root 2024-08-17T17:06:24.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:24 smithi165 ceph-mon[34868]: osdmap e21: 3 total, 3 up, 3 in 2024-08-17T17:06:24.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:24 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:24.541 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:06:24 smithi136 sudo[63630]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-17T17:06:24.541 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:06:24 smithi136 sudo[63630]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-17T17:06:24.541 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:06:24 smithi136 sudo[63630]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-17T17:06:24.541 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:06:24 smithi136 sudo[63630]: pam_unix(sudo:session): session closed for user root 2024-08-17T17:06:24.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:24 smithi136 sudo[63659]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-17T17:06:24.798 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:24 smithi136 sudo[63659]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-17T17:06:24.798 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:24 smithi136 sudo[63659]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-17T17:06:24.798 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:24 smithi136 sudo[63659]: pam_unix(sudo:session): session closed for user root 2024-08-17T17:06:25.163 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:24 smithi136 sudo[63663]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-17T17:06:25.163 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:24 smithi136 sudo[63663]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-17T17:06:25.163 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:24 smithi136 sudo[63663]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-17T17:06:25.164 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:24 smithi136 sudo[63663]: pam_unix(sudo:session): session closed for user root 2024-08-17T17:06:25.294 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:25 smithi165 sudo[37634]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-17T17:06:25.294 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:25 smithi165 sudo[37634]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-17T17:06:25.294 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:25 smithi165 sudo[37634]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-17T17:06:25.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:25 smithi165 sudo[37634]: pam_unix(sudo:session): session closed for user root 2024-08-17T17:06:25.436 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[31034]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-17T17:06:25.436 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[31034]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:25.436 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[31034]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:06:25.436 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[31034]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-17T17:06:25.437 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:06:25.437 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:06:25.437 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:06:25.437 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:06:25.437 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:06:25.437 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:06:25.437 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:06:25.437 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:06:25.437 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:25.437 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[42580]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-17T17:06:25.437 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[42580]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:25.438 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[42580]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:06:25.438 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[42580]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-17T17:06:25.438 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:06:25.438 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:06:25.438 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:06:25.438 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:06:25.438 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:06:25.438 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:06:25.438 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:06:25.438 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:06:25.438 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:25 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:25.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:25 smithi136 sudo[63825]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-17T17:06:25.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:25 smithi136 sudo[63825]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-17T17:06:25.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:25 smithi136 sudo[63825]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-17T17:06:25.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:25 smithi136 sudo[63825]: pam_unix(sudo:session): session closed for user root 2024-08-17T17:06:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:25 smithi165 sudo[37637]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-17T17:06:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:25 smithi165 sudo[37637]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-17T17:06:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:25 smithi165 sudo[37637]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-17T17:06:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:25 smithi165 sudo[37637]: pam_unix(sudo:session): session closed for user root 2024-08-17T17:06:25.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:25 smithi165 ceph-mon[34868]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-17T17:06:25.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:25 smithi165 ceph-mon[34868]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:25.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:25 smithi165 ceph-mon[34868]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:06:25.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:25 smithi165 ceph-mon[34868]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-17T17:06:25.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:25 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:06:25.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:25 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:06:25.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:25 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:06:25.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:25 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:06:25.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:25 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:06:25.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:25 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:06:25.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:25 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:06:25.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:25 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:06:25.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:25 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:26.140 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:25 smithi136 sudo[63834]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-17T17:06:26.140 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:25 smithi136 sudo[63834]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-17T17:06:26.140 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:25 smithi136 sudo[63834]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-17T17:06:26.140 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:25 smithi136 sudo[63834]: pam_unix(sudo:session): session closed for user root 2024-08-17T17:06:26.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:26 smithi136 ceph-mon[31034]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-17T17:06:26.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:26 smithi136 ceph-mon[31034]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-17T17:06:26.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:26 smithi136 ceph-mon[31034]: mgrmap e15: y(active, since 2m), standbys: x 2024-08-17T17:06:26.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:26 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:26.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:26 smithi136 ceph-mon[31034]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-17T17:06:26.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:26 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:06:26.427 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:26 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:06:26.427 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:26 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:06:26.427 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:26 smithi136 ceph-mon[31034]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-17T17:06:26.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:26 smithi136 ceph-mon[42580]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-17T17:06:26.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:26 smithi136 ceph-mon[42580]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-17T17:06:26.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:26 smithi136 ceph-mon[42580]: mgrmap e15: y(active, since 2m), standbys: x 2024-08-17T17:06:26.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:26 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:26.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:26 smithi136 ceph-mon[42580]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-17T17:06:26.428 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:26 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:06:26.428 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:26 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:06:26.428 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:26 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:06:26.428 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:26 smithi136 ceph-mon[42580]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-17T17:06:26.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:26 smithi165 ceph-mon[34868]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-17T17:06:26.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:26 smithi165 ceph-mon[34868]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-17T17:06:26.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:26 smithi165 ceph-mon[34868]: mgrmap e15: y(active, since 2m), standbys: x 2024-08-17T17:06:26.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:26 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:26.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:26 smithi165 ceph-mon[34868]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-17T17:06:26.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:26 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:06:26.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:26 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:06:26.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:26 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:06:26.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:26 smithi165 ceph-mon[34868]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-17T17:06:27.651 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:27 smithi136 ceph-mon[31034]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:27.651 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:27 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/1744919438' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30b8a4f2-be6d-4d5f-8397-06c2e17cfc2b"}]: dispatch 2024-08-17T17:06:27.651 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:27 smithi136 ceph-mon[31034]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30b8a4f2-be6d-4d5f-8397-06c2e17cfc2b"}]: dispatch 2024-08-17T17:06:27.651 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:27 smithi136 ceph-mon[31034]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30b8a4f2-be6d-4d5f-8397-06c2e17cfc2b"}]': finished 2024-08-17T17:06:27.651 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:27 smithi136 ceph-mon[31034]: osdmap e22: 4 total, 3 up, 4 in 2024-08-17T17:06:27.652 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:27 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:06:27.652 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:27 smithi136 ceph-mon[42580]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:27.652 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:27 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/1744919438' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30b8a4f2-be6d-4d5f-8397-06c2e17cfc2b"}]: dispatch 2024-08-17T17:06:27.652 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:27 smithi136 ceph-mon[42580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30b8a4f2-be6d-4d5f-8397-06c2e17cfc2b"}]: dispatch 2024-08-17T17:06:27.652 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:27 smithi136 ceph-mon[42580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30b8a4f2-be6d-4d5f-8397-06c2e17cfc2b"}]': finished 2024-08-17T17:06:27.652 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:27 smithi136 ceph-mon[42580]: osdmap e22: 4 total, 3 up, 4 in 2024-08-17T17:06:27.652 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:27 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:06:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:27 smithi165 ceph-mon[34868]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:27 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/1744919438' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30b8a4f2-be6d-4d5f-8397-06c2e17cfc2b"}]: dispatch 2024-08-17T17:06:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:27 smithi165 ceph-mon[34868]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30b8a4f2-be6d-4d5f-8397-06c2e17cfc2b"}]: dispatch 2024-08-17T17:06:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:27 smithi165 ceph-mon[34868]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30b8a4f2-be6d-4d5f-8397-06c2e17cfc2b"}]': finished 2024-08-17T17:06:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:27 smithi165 ceph-mon[34868]: osdmap e22: 4 total, 3 up, 4 in 2024-08-17T17:06:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:27 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:06:28.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:28 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/770593253' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:06:28.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:28 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/770593253' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:06:28.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:28 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/770593253' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:06:29.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:29 smithi136 ceph-mon[31034]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:29 smithi136 ceph-mon[42580]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:29 smithi165 ceph-mon[34868]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:31.831 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:31 smithi136 ceph-mon[31034]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:31.832 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:31 smithi136 ceph-mon[42580]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:31 smithi165 ceph-mon[34868]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:33.675 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:33 smithi136 ceph-mon[31034]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:33.675 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:33 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-08-17T17:06:33.675 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:33 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:33.675 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:33 smithi136 ceph-mon[42580]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:33.676 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:33 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-08-17T17:06:33.676 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:33 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:33 smithi165 ceph-mon[34868]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:33 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-08-17T17:06:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:33 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:34.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:34 smithi136 ceph-mon[31034]: Deploying daemon osd.3 on smithi136 2024-08-17T17:06:34.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:34 smithi136 ceph-mon[42580]: Deploying daemon osd.3 on smithi136 2024-08-17T17:06:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:34 smithi165 ceph-mon[34868]: Deploying daemon osd.3 on smithi136 2024-08-17T17:06:35.770 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:35 smithi136 ceph-mon[31034]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:35.770 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:35 smithi136 ceph-mon[42580]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:35 smithi165 ceph-mon[34868]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:37 smithi136 ceph-mon[31034]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:37 smithi136 ceph-mon[42580]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:37 smithi165 ceph-mon[34868]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:38.902 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:38 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:06:38.902 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:38 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:38.902 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:38 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:38.902 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:38 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:06:38.902 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:38 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:38.902 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:38 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:39.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:38 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:06:39.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:38 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:39.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:38 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:39.775 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:39 smithi136 ceph-mon[31034]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:39.775 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:39 smithi136 ceph-mon[42580]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:39 smithi165 ceph-mon[34868]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:41.107 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 3 on host 'smithi136' 2024-08-17T17:06:41.663 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:41 smithi136 ceph-mon[31034]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:41.663 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:41 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:41.663 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:41 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:41.663 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:41 smithi136 ceph-mon[31034]: from='osd.3 [v2:172.21.15.136:6826/3486272897,v1:172.21.15.136:6827/3486272897]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-08-17T17:06:41.663 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:41 smithi136 ceph-mon[31034]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-08-17T17:06:41.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:41 smithi136 ceph-mon[42580]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:41.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:41 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:41.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:41 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:41.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:41 smithi136 ceph-mon[42580]: from='osd.3 [v2:172.21.15.136:6826/3486272897,v1:172.21.15.136:6827/3486272897]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-08-17T17:06:41.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:41 smithi136 ceph-mon[42580]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-08-17T17:06:41.683 DEBUG:teuthology.orchestra.run.smithi136:osd.3> sudo journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.3.service 2024-08-17T17:06:41.686 INFO:tasks.cephadm:Deploying osd.4 on smithi165 with /dev/vg_nvme/lv_4... 2024-08-17T17:06:41.686 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-08-17T17:06:41.896 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:06:42.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:41 smithi165 ceph-mon[34868]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:42.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:41 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:42.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:41 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:42.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:41 smithi165 ceph-mon[34868]: from='osd.3 [v2:172.21.15.136:6826/3486272897,v1:172.21.15.136:6827/3486272897]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-08-17T17:06:42.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:41 smithi165 ceph-mon[34868]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-08-17T17:06:43.019 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:42 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:43.019 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:42 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:43.019 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:42 smithi165 ceph-mon[34868]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-17T17:06:43.019 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:42 smithi165 ceph-mon[34868]: osdmap e23: 4 total, 3 up, 4 in 2024-08-17T17:06:43.019 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:42 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:06:43.019 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:42 smithi165 ceph-mon[34868]: from='osd.3 [v2:172.21.15.136:6826/3486272897,v1:172.21.15.136:6827/3486272897]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-08-17T17:06:43.019 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:42 smithi165 ceph-mon[34868]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-08-17T17:06:43.019 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:42 smithi165 ceph-mon[34868]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:43.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:42 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:43.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:42 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:43.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:42 smithi136 ceph-mon[42580]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-17T17:06:43.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:42 smithi136 ceph-mon[42580]: osdmap e23: 4 total, 3 up, 4 in 2024-08-17T17:06:43.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:42 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:06:43.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:42 smithi136 ceph-mon[42580]: from='osd.3 [v2:172.21.15.136:6826/3486272897,v1:172.21.15.136:6827/3486272897]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-08-17T17:06:43.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:42 smithi136 ceph-mon[42580]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-08-17T17:06:43.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:42 smithi136 ceph-mon[42580]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:43.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:42 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:43.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:42 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:43.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:42 smithi136 ceph-mon[31034]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-17T17:06:43.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:42 smithi136 ceph-mon[31034]: osdmap e23: 4 total, 3 up, 4 in 2024-08-17T17:06:43.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:42 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:06:43.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:42 smithi136 ceph-mon[31034]: from='osd.3 [v2:172.21.15.136:6826/3486272897,v1:172.21.15.136:6827/3486272897]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-08-17T17:06:43.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:42 smithi136 ceph-mon[31034]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-08-17T17:06:43.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:42 smithi136 ceph-mon[31034]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:43.595 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:06:43 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3[65713]: 2024-08-17T17:06:43.385+0000 7f8b6cdb6640 -1 osd.3 0 waiting for initial osdmap 2024-08-17T17:06:43.596 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:06:43 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3[65713]: 2024-08-17T17:06:43.389+0000 7f8b64bd7640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:06:44.067 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:06:44.086 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi165:vg_nvme/lv_4 2024-08-17T17:06:44.284 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:06:44.390 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:44 smithi165 ceph-mon[34868]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-08-17T17:06:44.390 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:44 smithi165 ceph-mon[34868]: osdmap e24: 4 total, 3 up, 4 in 2024-08-17T17:06:44.390 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:44 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:06:44.390 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:44 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:06:44.390 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:44 smithi165 ceph-mon[34868]: from='osd.3 ' entity='osd.3' 2024-08-17T17:06:44.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:44 smithi136 ceph-mon[31034]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-08-17T17:06:44.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:44 smithi136 ceph-mon[31034]: osdmap e24: 4 total, 3 up, 4 in 2024-08-17T17:06:44.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:44 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:06:44.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:44 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:06:44.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:44 smithi136 ceph-mon[31034]: from='osd.3 ' entity='osd.3' 2024-08-17T17:06:44.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:44 smithi136 ceph-mon[42580]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-08-17T17:06:44.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:44 smithi136 ceph-mon[42580]: osdmap e24: 4 total, 3 up, 4 in 2024-08-17T17:06:44.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:44 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:06:44.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:44 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:06:44.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:44 smithi136 ceph-mon[42580]: from='osd.3 ' entity='osd.3' 2024-08-17T17:06:45.403 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:45 smithi165 ceph-mon[34868]: purged_snaps scrub starts 2024-08-17T17:06:45.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:45 smithi165 ceph-mon[34868]: purged_snaps scrub ok 2024-08-17T17:06:45.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:45 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:06:45.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:45 smithi165 ceph-mon[34868]: osd.3 [v2:172.21.15.136:6826/3486272897,v1:172.21.15.136:6827/3486272897] boot 2024-08-17T17:06:45.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:45 smithi165 ceph-mon[34868]: osdmap e25: 4 total, 4 up, 4 in 2024-08-17T17:06:45.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:45 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:06:45.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:45 smithi165 ceph-mon[34868]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:45.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:45 smithi136 ceph-mon[31034]: purged_snaps scrub starts 2024-08-17T17:06:45.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:45 smithi136 ceph-mon[31034]: purged_snaps scrub ok 2024-08-17T17:06:45.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:45 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:06:45.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:45 smithi136 ceph-mon[31034]: osd.3 [v2:172.21.15.136:6826/3486272897,v1:172.21.15.136:6827/3486272897] boot 2024-08-17T17:06:45.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:45 smithi136 ceph-mon[31034]: osdmap e25: 4 total, 4 up, 4 in 2024-08-17T17:06:45.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:45 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:06:45.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:45 smithi136 ceph-mon[31034]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:45.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:45 smithi136 ceph-mon[42580]: purged_snaps scrub starts 2024-08-17T17:06:45.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:45 smithi136 ceph-mon[42580]: purged_snaps scrub ok 2024-08-17T17:06:45.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:45 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:06:45.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:45 smithi136 ceph-mon[42580]: osd.3 [v2:172.21.15.136:6826/3486272897,v1:172.21.15.136:6827/3486272897] boot 2024-08-17T17:06:45.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:45 smithi136 ceph-mon[42580]: osdmap e25: 4 total, 4 up, 4 in 2024-08-17T17:06:45.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:45 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:06:45.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:45 smithi136 ceph-mon[42580]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-17T17:06:46.403 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:46 smithi165 ceph-mon[34868]: Detected new or changed devices on smithi136 2024-08-17T17:06:46.403 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:46 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:46.403 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:46 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:46.403 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:46 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:46.403 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:46 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:46.403 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:46 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:46.403 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:46 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:46.403 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:46 smithi165 ceph-mon[34868]: Adjusting osd_memory_target on smithi136 to 4016M 2024-08-17T17:06:46.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:46 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:46.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:46 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:46.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:46 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:06:46.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:46 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:46.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:46 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:06:46.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:46 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:46.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:46 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:06:46.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:46 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:46.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:46 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:06:46.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:46 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:06:46.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:46 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:46.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[31034]: Detected new or changed devices on smithi136 2024-08-17T17:06:46.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:46.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:46.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:46.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:46.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:46.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:46.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[31034]: Adjusting osd_memory_target on smithi136 to 4016M 2024-08-17T17:06:46.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:46.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:46.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:06:46.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:46.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:06:46.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:46.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:06:46.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:46.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:06:46.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:06:46.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:46.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[42580]: Detected new or changed devices on smithi136 2024-08-17T17:06:46.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:46.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:46.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:46.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:46.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:46.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:46.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[42580]: Adjusting osd_memory_target on smithi136 to 4016M 2024-08-17T17:06:46.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:46.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:46.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:06:46.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:46.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:06:46.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:46.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:06:46.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:46.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:06:46.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:06:46.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:46 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:47.345 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:47 smithi165 ceph-mon[34868]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:06:47.345 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:47 smithi165 ceph-mon[34868]: osdmap e26: 4 total, 4 up, 4 in 2024-08-17T17:06:47.345 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:47 smithi165 ceph-mon[34868]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:06:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:47 smithi136 ceph-mon[31034]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:06:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:47 smithi136 ceph-mon[31034]: osdmap e26: 4 total, 4 up, 4 in 2024-08-17T17:06:47.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:47 smithi136 ceph-mon[31034]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:06:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:47 smithi136 ceph-mon[42580]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:06:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:47 smithi136 ceph-mon[42580]: osdmap e26: 4 total, 4 up, 4 in 2024-08-17T17:06:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:47 smithi136 ceph-mon[42580]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:06:48.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:48 smithi136 ceph-mon[31034]: from='client.? 172.21.15.165:0/1189053407' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae4074cf-0069-403f-acc7-74bedab266bd"}]: dispatch 2024-08-17T17:06:48.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:48 smithi136 ceph-mon[31034]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae4074cf-0069-403f-acc7-74bedab266bd"}]: dispatch 2024-08-17T17:06:48.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:48 smithi136 ceph-mon[31034]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae4074cf-0069-403f-acc7-74bedab266bd"}]': finished 2024-08-17T17:06:48.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:48 smithi136 ceph-mon[31034]: osdmap e27: 5 total, 4 up, 5 in 2024-08-17T17:06:48.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:48 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:06:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:48 smithi136 ceph-mon[42580]: from='client.? 172.21.15.165:0/1189053407' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae4074cf-0069-403f-acc7-74bedab266bd"}]: dispatch 2024-08-17T17:06:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:48 smithi136 ceph-mon[42580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae4074cf-0069-403f-acc7-74bedab266bd"}]: dispatch 2024-08-17T17:06:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:48 smithi136 ceph-mon[42580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae4074cf-0069-403f-acc7-74bedab266bd"}]': finished 2024-08-17T17:06:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:48 smithi136 ceph-mon[42580]: osdmap e27: 5 total, 4 up, 5 in 2024-08-17T17:06:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:48 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:06:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:48 smithi165 ceph-mon[34868]: from='client.? 172.21.15.165:0/1189053407' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae4074cf-0069-403f-acc7-74bedab266bd"}]: dispatch 2024-08-17T17:06:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:48 smithi165 ceph-mon[34868]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae4074cf-0069-403f-acc7-74bedab266bd"}]: dispatch 2024-08-17T17:06:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:48 smithi165 ceph-mon[34868]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae4074cf-0069-403f-acc7-74bedab266bd"}]': finished 2024-08-17T17:06:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:48 smithi165 ceph-mon[34868]: osdmap e27: 5 total, 4 up, 5 in 2024-08-17T17:06:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:48 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:06:49.541 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:49 smithi165 ceph-mon[34868]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:06:49.541 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:49 smithi165 ceph-mon[34868]: from='client.? 172.21.15.165:0/2751431665' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:06:49.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:49 smithi136 ceph-mon[31034]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:06:49.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:49 smithi136 ceph-mon[31034]: from='client.? 172.21.15.165:0/2751431665' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:06:49.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:49 smithi136 ceph-mon[42580]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:06:49.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:49 smithi136 ceph-mon[42580]: from='client.? 172.21.15.165:0/2751431665' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:06:51.660 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:51 smithi165 ceph-mon[34868]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:06:51.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:51 smithi136 ceph-mon[31034]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:06:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:51 smithi136 ceph-mon[42580]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:06:53.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:53 smithi165 ceph-mon[34868]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:06:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:53 smithi136 ceph-mon[31034]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:06:53.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:53 smithi136 ceph-mon[42580]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:06:54.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:54 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-08-17T17:06:54.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:54 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:54.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:54 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-08-17T17:06:54.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:54 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:54 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-08-17T17:06:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:54 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:55 smithi165 ceph-mon[34868]: Deploying daemon osd.4 on smithi165 2024-08-17T17:06:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:55 smithi165 ceph-mon[34868]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:06:55.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:55 smithi136 ceph-mon[31034]: Deploying daemon osd.4 on smithi165 2024-08-17T17:06:55.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:55 smithi136 ceph-mon[31034]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:06:55.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:55 smithi136 ceph-mon[42580]: Deploying daemon osd.4 on smithi165 2024-08-17T17:06:55.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:55 smithi136 ceph-mon[42580]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:06:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:57 smithi165 ceph-mon[34868]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:06:57.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:57 smithi136 ceph-mon[31034]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:06:57.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:57 smithi136 ceph-mon[42580]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:06:59.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:59 smithi165 ceph-mon[34868]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:06:59.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:59 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:06:59.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:59 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:59.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:06:59 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:59.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:59 smithi136 ceph-mon[42580]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:06:59.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:59 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:06:59.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:59 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:59.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:06:59 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:59.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:59 smithi136 ceph-mon[31034]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:06:59.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:59 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:06:59.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:59 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:06:59.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:06:59 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:01.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:01 smithi136 ceph-mon[31034]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:07:01.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:01 smithi136 ceph-mon[42580]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:07:01.999 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:01 smithi165 ceph-mon[34868]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:07:02.034 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 4 on host 'smithi165' 2024-08-17T17:07:02.593 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:02 smithi165 ceph-mon[34868]: from='osd.4 [v2:172.21.15.165:6800/842623090,v1:172.21.15.165:6801/842623090]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-08-17T17:07:02.593 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:02 smithi165 ceph-mon[34868]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-08-17T17:07:02.593 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:02 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:02.593 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:02 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:02.595 DEBUG:teuthology.orchestra.run.smithi165:osd.4> sudo journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.4.service 2024-08-17T17:07:02.597 INFO:tasks.cephadm:Deploying osd.5 on smithi165 with /dev/vg_nvme/lv_3... 2024-08-17T17:07:02.597 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-08-17T17:07:02.839 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:07:02.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:02 smithi136 ceph-mon[31034]: from='osd.4 [v2:172.21.15.165:6800/842623090,v1:172.21.15.165:6801/842623090]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-08-17T17:07:02.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:02 smithi136 ceph-mon[31034]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-08-17T17:07:02.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:02 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:02.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:02 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:02.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:02 smithi136 ceph-mon[42580]: from='osd.4 [v2:172.21.15.165:6800/842623090,v1:172.21.15.165:6801/842623090]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-08-17T17:07:02.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:02 smithi136 ceph-mon[42580]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-08-17T17:07:02.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:02 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:02.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:02 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:03.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:03 smithi136 ceph-mon[31034]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:07:03.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:03 smithi136 ceph-mon[31034]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-08-17T17:07:03.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:03 smithi136 ceph-mon[31034]: osdmap e28: 5 total, 4 up, 5 in 2024-08-17T17:07:03.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:03 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:07:03.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:03 smithi136 ceph-mon[31034]: from='osd.4 [v2:172.21.15.165:6800/842623090,v1:172.21.15.165:6801/842623090]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-08-17T17:07:03.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:03 smithi136 ceph-mon[31034]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-08-17T17:07:03.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:03 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:03.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:03 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:03.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:03 smithi136 ceph-mon[42580]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:07:03.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:03 smithi136 ceph-mon[42580]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-08-17T17:07:03.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:03 smithi136 ceph-mon[42580]: osdmap e28: 5 total, 4 up, 5 in 2024-08-17T17:07:03.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:03 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:07:03.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:03 smithi136 ceph-mon[42580]: from='osd.4 [v2:172.21.15.165:6800/842623090,v1:172.21.15.165:6801/842623090]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-08-17T17:07:03.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:03 smithi136 ceph-mon[42580]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-08-17T17:07:03.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:03 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:03.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:03 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:03.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:03 smithi165 ceph-mon[34868]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:07:03.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:03 smithi165 ceph-mon[34868]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-08-17T17:07:03.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:03 smithi165 ceph-mon[34868]: osdmap e28: 5 total, 4 up, 5 in 2024-08-17T17:07:03.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:03 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:07:03.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:03 smithi165 ceph-mon[34868]: from='osd.4 [v2:172.21.15.165:6800/842623090,v1:172.21.15.165:6801/842623090]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-08-17T17:07:03.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:03 smithi165 ceph-mon[34868]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-08-17T17:07:03.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:03 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:03.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:03 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:04.261 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 17:07:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4[40219]: 2024-08-17T17:07:04.052+0000 7ffbe5b01640 -1 osd.4 0 waiting for initial osdmap 2024-08-17T17:07:04.262 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 17:07:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4[40219]: 2024-08-17T17:07:04.057+0000 7ffbdd922640 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:07:04.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:04 smithi136 ceph-mon[31034]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-08-17T17:07:04.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:04 smithi136 ceph-mon[31034]: osdmap e29: 5 total, 4 up, 5 in 2024-08-17T17:07:04.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:04 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:07:04.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:04 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:07:04.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:04 smithi136 ceph-mon[31034]: from='osd.4 ' entity='osd.4' 2024-08-17T17:07:04.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:04 smithi136 ceph-mon[42580]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-08-17T17:07:04.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:04 smithi136 ceph-mon[42580]: osdmap e29: 5 total, 4 up, 5 in 2024-08-17T17:07:04.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:04 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:07:04.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:04 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:07:04.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:04 smithi136 ceph-mon[42580]: from='osd.4 ' entity='osd.4' 2024-08-17T17:07:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:04 smithi165 ceph-mon[34868]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-08-17T17:07:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:04 smithi165 ceph-mon[34868]: osdmap e29: 5 total, 4 up, 5 in 2024-08-17T17:07:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:04 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:07:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:04 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:07:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:04 smithi165 ceph-mon[34868]: from='osd.4 ' entity='osd.4' 2024-08-17T17:07:05.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:05 smithi165 ceph-mon[34868]: purged_snaps scrub starts 2024-08-17T17:07:05.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:05 smithi165 ceph-mon[34868]: purged_snaps scrub ok 2024-08-17T17:07:05.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:05 smithi165 ceph-mon[34868]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:07:05.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:05 smithi165 ceph-mon[34868]: osd.4 [v2:172.21.15.165:6800/842623090,v1:172.21.15.165:6801/842623090] boot 2024-08-17T17:07:05.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:05 smithi165 ceph-mon[34868]: osdmap e30: 5 total, 5 up, 5 in 2024-08-17T17:07:05.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:05 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:07:05.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:05 smithi165 ceph-mon[34868]: osdmap e31: 5 total, 5 up, 5 in 2024-08-17T17:07:05.774 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:07:05.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi165:vg_nvme/lv_3 2024-08-17T17:07:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:05 smithi136 ceph-mon[31034]: purged_snaps scrub starts 2024-08-17T17:07:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:05 smithi136 ceph-mon[31034]: purged_snaps scrub ok 2024-08-17T17:07:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:05 smithi136 ceph-mon[31034]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:07:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:05 smithi136 ceph-mon[31034]: osd.4 [v2:172.21.15.165:6800/842623090,v1:172.21.15.165:6801/842623090] boot 2024-08-17T17:07:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:05 smithi136 ceph-mon[31034]: osdmap e30: 5 total, 5 up, 5 in 2024-08-17T17:07:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:05 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:07:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:05 smithi136 ceph-mon[31034]: osdmap e31: 5 total, 5 up, 5 in 2024-08-17T17:07:05.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:05 smithi136 ceph-mon[42580]: purged_snaps scrub starts 2024-08-17T17:07:05.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:05 smithi136 ceph-mon[42580]: purged_snaps scrub ok 2024-08-17T17:07:05.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:05 smithi136 ceph-mon[42580]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-17T17:07:05.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:05 smithi136 ceph-mon[42580]: osd.4 [v2:172.21.15.165:6800/842623090,v1:172.21.15.165:6801/842623090] boot 2024-08-17T17:07:05.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:05 smithi136 ceph-mon[42580]: osdmap e30: 5 total, 5 up, 5 in 2024-08-17T17:07:05.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:05 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:07:05.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:05 smithi136 ceph-mon[42580]: osdmap e31: 5 total, 5 up, 5 in 2024-08-17T17:07:06.189 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:07:07.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:07 smithi165 ceph-mon[34868]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 535 MiB used, 446 GiB / 447 GiB avail 2024-08-17T17:07:07.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:07 smithi165 ceph-mon[34868]: osdmap e32: 5 total, 5 up, 5 in 2024-08-17T17:07:07.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:07 smithi165 ceph-mon[34868]: osdmap e33: 5 total, 5 up, 5 in 2024-08-17T17:07:07.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:07 smithi136 ceph-mon[31034]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 535 MiB used, 446 GiB / 447 GiB avail 2024-08-17T17:07:07.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:07 smithi136 ceph-mon[31034]: osdmap e32: 5 total, 5 up, 5 in 2024-08-17T17:07:07.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:07 smithi136 ceph-mon[31034]: osdmap e33: 5 total, 5 up, 5 in 2024-08-17T17:07:07.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:07 smithi136 ceph-mon[42580]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 535 MiB used, 446 GiB / 447 GiB avail 2024-08-17T17:07:07.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:07 smithi136 ceph-mon[42580]: osdmap e32: 5 total, 5 up, 5 in 2024-08-17T17:07:07.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:07 smithi136 ceph-mon[42580]: osdmap e33: 5 total, 5 up, 5 in 2024-08-17T17:07:09.565 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:09 smithi165 ceph-mon[34868]: osdmap e34: 5 total, 5 up, 5 in 2024-08-17T17:07:09.565 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:09 smithi165 ceph-mon[34868]: Detected new or changed devices on smithi165 2024-08-17T17:07:09.565 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:09 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:09.566 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:09 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:09.566 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:09 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:07:09.566 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:09 smithi165 ceph-mon[34868]: Adjusting osd_memory_target on smithi165 to 17088M 2024-08-17T17:07:09.566 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:09 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:09.566 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:09 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:09.566 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:09 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:09.566 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:09 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:09.566 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:09 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:09.566 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:09 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:09.566 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:09 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:09.566 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:09 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:09.566 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:09 smithi165 ceph-mon[34868]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 535 MiB used, 446 GiB / 447 GiB avail 2024-08-17T17:07:09.566 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:09 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:07:09.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:09 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:07:09.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:09 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[31034]: osdmap e34: 5 total, 5 up, 5 in 2024-08-17T17:07:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[31034]: Detected new or changed devices on smithi165 2024-08-17T17:07:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:07:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[31034]: Adjusting osd_memory_target on smithi165 to 17088M 2024-08-17T17:07:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:09.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:09.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:09.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:09.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:09.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:09.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[31034]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 535 MiB used, 446 GiB / 447 GiB avail 2024-08-17T17:07:09.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:07:09.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:07:09.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:09.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[42580]: osdmap e34: 5 total, 5 up, 5 in 2024-08-17T17:07:09.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[42580]: Detected new or changed devices on smithi165 2024-08-17T17:07:09.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:09.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:09.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:07:09.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[42580]: Adjusting osd_memory_target on smithi165 to 17088M 2024-08-17T17:07:09.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:09.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:09.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:09.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:09.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:09.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:09.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:09.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:09.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[42580]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 535 MiB used, 446 GiB / 447 GiB avail 2024-08-17T17:07:09.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:07:09.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:07:09.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:09 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:10.555 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:10 smithi165 ceph-mon[34868]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:07:10.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:10 smithi136 ceph-mon[31034]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:07:10.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:10 smithi136 ceph-mon[42580]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:07:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:11 smithi136 ceph-mon[31034]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-17T17:07:11.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:11 smithi136 ceph-mon[31034]: from='client.? 172.21.15.165:0/3122648597' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "586c23a7-4831-4b4d-9a06-c1bfcc1c5286"}]: dispatch 2024-08-17T17:07:11.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:11 smithi136 ceph-mon[31034]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "586c23a7-4831-4b4d-9a06-c1bfcc1c5286"}]: dispatch 2024-08-17T17:07:11.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:11 smithi136 ceph-mon[31034]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "586c23a7-4831-4b4d-9a06-c1bfcc1c5286"}]': finished 2024-08-17T17:07:11.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:11 smithi136 ceph-mon[31034]: osdmap e35: 6 total, 5 up, 6 in 2024-08-17T17:07:11.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:11 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:07:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:11 smithi136 ceph-mon[42580]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-17T17:07:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:11 smithi136 ceph-mon[42580]: from='client.? 172.21.15.165:0/3122648597' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "586c23a7-4831-4b4d-9a06-c1bfcc1c5286"}]: dispatch 2024-08-17T17:07:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:11 smithi136 ceph-mon[42580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "586c23a7-4831-4b4d-9a06-c1bfcc1c5286"}]: dispatch 2024-08-17T17:07:11.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:11 smithi136 ceph-mon[42580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "586c23a7-4831-4b4d-9a06-c1bfcc1c5286"}]': finished 2024-08-17T17:07:11.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:11 smithi136 ceph-mon[42580]: osdmap e35: 6 total, 5 up, 6 in 2024-08-17T17:07:11.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:11 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:07:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:11 smithi165 ceph-mon[34868]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-17T17:07:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:11 smithi165 ceph-mon[34868]: from='client.? 172.21.15.165:0/3122648597' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "586c23a7-4831-4b4d-9a06-c1bfcc1c5286"}]: dispatch 2024-08-17T17:07:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:11 smithi165 ceph-mon[34868]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "586c23a7-4831-4b4d-9a06-c1bfcc1c5286"}]: dispatch 2024-08-17T17:07:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:11 smithi165 ceph-mon[34868]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "586c23a7-4831-4b4d-9a06-c1bfcc1c5286"}]': finished 2024-08-17T17:07:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:11 smithi165 ceph-mon[34868]: osdmap e35: 6 total, 5 up, 6 in 2024-08-17T17:07:11.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:11 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:07:12.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:12 smithi136 ceph-mon[31034]: from='client.? 172.21.15.165:0/3540339718' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:07:12.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:12 smithi136 ceph-mon[42580]: from='client.? 172.21.15.165:0/3540339718' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:07:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:12 smithi165 ceph-mon[34868]: from='client.? 172.21.15.165:0/3540339718' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:07:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:13 smithi165 ceph-mon[34868]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-08-17T17:07:13.802 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:13 smithi136 ceph-mon[31034]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-08-17T17:07:13.802 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:13 smithi136 ceph-mon[42580]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-08-17T17:07:15.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:15 smithi136 ceph-mon[31034]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-08-17T17:07:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:15 smithi136 ceph-mon[42580]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-08-17T17:07:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:15 smithi165 ceph-mon[34868]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-08-17T17:07:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:17 smithi165 ceph-mon[34868]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-08-17T17:07:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:17 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-08-17T17:07:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:17 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:17.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:17 smithi136 ceph-mon[31034]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-08-17T17:07:17.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:17 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-08-17T17:07:17.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:17 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:17.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:17 smithi136 ceph-mon[42580]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-08-17T17:07:17.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:17 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-08-17T17:07:17.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:17 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:18.798 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:18 smithi165 ceph-mon[34868]: Deploying daemon osd.5 on smithi165 2024-08-17T17:07:18.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:18 smithi136 ceph-mon[31034]: Deploying daemon osd.5 on smithi165 2024-08-17T17:07:18.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:18 smithi136 ceph-mon[42580]: Deploying daemon osd.5 on smithi165 2024-08-17T17:07:19.644 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:19 smithi165 ceph-mon[34868]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-17T17:07:19.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:19 smithi136 ceph-mon[42580]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-17T17:07:19.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:19 smithi136 ceph-mon[31034]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-17T17:07:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:21 smithi165 ceph-mon[34868]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-17T17:07:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:21 smithi136 ceph-mon[31034]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-17T17:07:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:21 smithi136 ceph-mon[42580]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-17T17:07:22.831 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:22 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:22.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:22 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:22.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:22 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:22.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:22 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:22.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:22 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:22.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:22 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:22.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:22 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:22.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:22 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:22.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:22 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:23.677 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:23 smithi165 ceph-mon[34868]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-17T17:07:23.872 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:23 smithi136 ceph-mon[31034]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-17T17:07:23.873 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:23 smithi136 ceph-mon[42580]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-17T17:07:25.383 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 5 on host 'smithi165' 2024-08-17T17:07:25.737 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:25 smithi165 ceph-mon[34868]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-17T17:07:25.737 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:25 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:25.737 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:25 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:25.737 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:25 smithi165 ceph-mon[34868]: from='osd.5 [v2:172.21.15.165:6808/3602301990,v1:172.21.15.165:6809/3602301990]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-08-17T17:07:25.737 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:25 smithi165 ceph-mon[34868]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-08-17T17:07:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:25 smithi136 ceph-mon[31034]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-17T17:07:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:25 smithi136 ceph-mon[31034]: from='osd.5 [v2:172.21.15.165:6808/3602301990,v1:172.21.15.165:6809/3602301990]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-08-17T17:07:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:25 smithi136 ceph-mon[31034]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-08-17T17:07:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:25 smithi136 ceph-mon[42580]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-17T17:07:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:25 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:25.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:25 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:25.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:25 smithi136 ceph-mon[42580]: from='osd.5 [v2:172.21.15.165:6808/3602301990,v1:172.21.15.165:6809/3602301990]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-08-17T17:07:25.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:25 smithi136 ceph-mon[42580]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-08-17T17:07:26.024 DEBUG:teuthology.orchestra.run.smithi165:osd.5> sudo journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.5.service 2024-08-17T17:07:26.026 INFO:tasks.cephadm:Deploying osd.6 on smithi165 with /dev/vg_nvme/lv_2... 2024-08-17T17:07:26.026 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-08-17T17:07:26.831 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:07:26.924 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:26 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:26.925 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:26 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:26.925 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:26 smithi165 ceph-mon[34868]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-17T17:07:26.925 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:26 smithi165 ceph-mon[34868]: osdmap e36: 6 total, 5 up, 6 in 2024-08-17T17:07:26.925 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:26 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:07:26.925 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:26 smithi165 ceph-mon[34868]: from='osd.5 [v2:172.21.15.165:6808/3602301990,v1:172.21.15.165:6809/3602301990]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-08-17T17:07:26.925 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:26 smithi165 ceph-mon[34868]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-08-17T17:07:26.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:26 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:26.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:26 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:26.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:26 smithi136 ceph-mon[31034]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-17T17:07:26.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:26 smithi136 ceph-mon[31034]: osdmap e36: 6 total, 5 up, 6 in 2024-08-17T17:07:26.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:26 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:07:26.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:26 smithi136 ceph-mon[31034]: from='osd.5 [v2:172.21.15.165:6808/3602301990,v1:172.21.15.165:6809/3602301990]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-08-17T17:07:26.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:26 smithi136 ceph-mon[31034]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-08-17T17:07:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:26 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:26 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:26 smithi136 ceph-mon[42580]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-17T17:07:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:26 smithi136 ceph-mon[42580]: osdmap e36: 6 total, 5 up, 6 in 2024-08-17T17:07:26.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:26 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:07:26.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:26 smithi136 ceph-mon[42580]: from='osd.5 [v2:172.21.15.165:6808/3602301990,v1:172.21.15.165:6809/3602301990]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-08-17T17:07:26.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:26 smithi136 ceph-mon[42580]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-08-17T17:07:27.571 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 17:07:27 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T17:07:27.397+0000 7fedc7462640 -1 osd.5 0 waiting for initial osdmap 2024-08-17T17:07:27.571 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 17:07:27 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T17:07:27.401+0000 7fedc328b640 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:07:27.837 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:27 smithi165 ceph-mon[34868]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-17T17:07:27.837 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:27 smithi165 ceph-mon[34868]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-08-17T17:07:27.838 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:27 smithi165 ceph-mon[34868]: osdmap e37: 6 total, 5 up, 6 in 2024-08-17T17:07:27.838 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:27 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:07:27.838 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:27 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:07:27.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:27 smithi136 ceph-mon[31034]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-17T17:07:27.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:27 smithi136 ceph-mon[31034]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-08-17T17:07:27.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:27 smithi136 ceph-mon[31034]: osdmap e37: 6 total, 5 up, 6 in 2024-08-17T17:07:27.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:27 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:07:27.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:27 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:07:27.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:27 smithi136 ceph-mon[42580]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-17T17:07:27.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:27 smithi136 ceph-mon[42580]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-08-17T17:07:27.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:27 smithi136 ceph-mon[42580]: osdmap e37: 6 total, 5 up, 6 in 2024-08-17T17:07:27.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:27 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:07:27.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:27 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:07:29.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:29 smithi165 ceph-mon[34868]: purged_snaps scrub starts 2024-08-17T17:07:29.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:29 smithi165 ceph-mon[34868]: purged_snaps scrub ok 2024-08-17T17:07:29.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:29 smithi165 ceph-mon[34868]: osd.5 [v2:172.21.15.165:6808/3602301990,v1:172.21.15.165:6809/3602301990] boot 2024-08-17T17:07:29.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:29 smithi165 ceph-mon[34868]: osdmap e38: 6 total, 6 up, 6 in 2024-08-17T17:07:29.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:29 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:07:29.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:29 smithi165 ceph-mon[34868]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-17T17:07:29.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:29 smithi136 ceph-mon[42580]: purged_snaps scrub starts 2024-08-17T17:07:29.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:29 smithi136 ceph-mon[42580]: purged_snaps scrub ok 2024-08-17T17:07:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:29 smithi136 ceph-mon[42580]: osd.5 [v2:172.21.15.165:6808/3602301990,v1:172.21.15.165:6809/3602301990] boot 2024-08-17T17:07:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:29 smithi136 ceph-mon[42580]: osdmap e38: 6 total, 6 up, 6 in 2024-08-17T17:07:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:29 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:07:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:29 smithi136 ceph-mon[42580]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-17T17:07:29.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:29 smithi136 ceph-mon[31034]: purged_snaps scrub starts 2024-08-17T17:07:29.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:29 smithi136 ceph-mon[31034]: purged_snaps scrub ok 2024-08-17T17:07:29.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:29 smithi136 ceph-mon[31034]: osd.5 [v2:172.21.15.165:6808/3602301990,v1:172.21.15.165:6809/3602301990] boot 2024-08-17T17:07:29.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:29 smithi136 ceph-mon[31034]: osdmap e38: 6 total, 6 up, 6 in 2024-08-17T17:07:29.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:29 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:07:29.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:29 smithi136 ceph-mon[31034]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-17T17:07:30.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:30 smithi165 ceph-mon[34868]: osdmap e39: 6 total, 6 up, 6 in 2024-08-17T17:07:30.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:30 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:30.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:30 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:30.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:30 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:07:30.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:30 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:07:30.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:30 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:30.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:30 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:30.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:30 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:30.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:30 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:30.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:30 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:30.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:30 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:30.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:30 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:30.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:30 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:30.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[42580]: osdmap e39: 6 total, 6 up, 6 in 2024-08-17T17:07:30.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:30.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:30.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:07:30.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:07:30.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:30.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:30.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:30.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:30.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:30.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:30.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:30.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:30.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[31034]: osdmap e39: 6 total, 6 up, 6 in 2024-08-17T17:07:30.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:30.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:30.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:07:30.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:07:30.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:30.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:30.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:30.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:30.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:30.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:30.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:30.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:30 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:31.181 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:07:31.200 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi165:vg_nvme/lv_2 2024-08-17T17:07:31.401 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:07:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:31 smithi165 ceph-mon[34868]: Detected new or changed devices on smithi165 2024-08-17T17:07:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:31 smithi165 ceph-mon[34868]: Adjusting osd_memory_target on smithi165 to 8544M 2024-08-17T17:07:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:31 smithi165 ceph-mon[34868]: osdmap e40: 6 total, 6 up, 6 in 2024-08-17T17:07:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:31 smithi165 ceph-mon[34868]: pgmap v130: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-17T17:07:31.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:31 smithi136 ceph-mon[31034]: Detected new or changed devices on smithi165 2024-08-17T17:07:31.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:31 smithi136 ceph-mon[31034]: Adjusting osd_memory_target on smithi165 to 8544M 2024-08-17T17:07:31.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:31 smithi136 ceph-mon[31034]: osdmap e40: 6 total, 6 up, 6 in 2024-08-17T17:07:31.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:31 smithi136 ceph-mon[31034]: pgmap v130: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-17T17:07:31.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:31 smithi136 ceph-mon[42580]: Detected new or changed devices on smithi165 2024-08-17T17:07:31.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:31 smithi136 ceph-mon[42580]: Adjusting osd_memory_target on smithi165 to 8544M 2024-08-17T17:07:31.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:31 smithi136 ceph-mon[42580]: osdmap e40: 6 total, 6 up, 6 in 2024-08-17T17:07:31.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:31 smithi136 ceph-mon[42580]: pgmap v130: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-17T17:07:32.425 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:32 smithi165 ceph-mon[34868]: osdmap e41: 6 total, 6 up, 6 in 2024-08-17T17:07:32.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:32 smithi136 ceph-mon[42580]: osdmap e41: 6 total, 6 up, 6 in 2024-08-17T17:07:32.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:32 smithi136 ceph-mon[31034]: osdmap e41: 6 total, 6 up, 6 in 2024-08-17T17:07:33.688 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:33 smithi165 ceph-mon[34868]: osdmap e42: 6 total, 6 up, 6 in 2024-08-17T17:07:33.689 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:33 smithi165 ceph-mon[34868]: pgmap v133: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-17T17:07:33.689 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:33 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:07:33.689 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:33 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:07:33.689 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:33 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:33 smithi136 ceph-mon[31034]: osdmap e42: 6 total, 6 up, 6 in 2024-08-17T17:07:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:33 smithi136 ceph-mon[31034]: pgmap v133: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-17T17:07:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:33 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:07:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:33 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:07:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:33 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:33.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:33 smithi136 ceph-mon[42580]: osdmap e42: 6 total, 6 up, 6 in 2024-08-17T17:07:33.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:33 smithi136 ceph-mon[42580]: pgmap v133: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-17T17:07:33.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:33 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:07:33.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:33 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:07:33.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:33 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:34.620 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:34 smithi165 ceph-mon[34868]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:07:34.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:34 smithi136 ceph-mon[31034]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:07:34.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:34 smithi136 ceph-mon[42580]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:07:35.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:35 smithi136 ceph-mon[31034]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-08-17T17:07:35.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:35 smithi136 ceph-mon[31034]: from='client.? 172.21.15.165:0/1062308453' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e838c26b-5e4a-41c6-b35b-b438ba64b237"}]: dispatch 2024-08-17T17:07:35.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:35 smithi136 ceph-mon[31034]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e838c26b-5e4a-41c6-b35b-b438ba64b237"}]: dispatch 2024-08-17T17:07:35.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:35 smithi136 ceph-mon[31034]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e838c26b-5e4a-41c6-b35b-b438ba64b237"}]': finished 2024-08-17T17:07:35.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:35 smithi136 ceph-mon[31034]: osdmap e43: 7 total, 6 up, 7 in 2024-08-17T17:07:35.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:35 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:07:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:35 smithi136 ceph-mon[42580]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-08-17T17:07:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:35 smithi136 ceph-mon[42580]: from='client.? 172.21.15.165:0/1062308453' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e838c26b-5e4a-41c6-b35b-b438ba64b237"}]: dispatch 2024-08-17T17:07:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:35 smithi136 ceph-mon[42580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e838c26b-5e4a-41c6-b35b-b438ba64b237"}]: dispatch 2024-08-17T17:07:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:35 smithi136 ceph-mon[42580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e838c26b-5e4a-41c6-b35b-b438ba64b237"}]': finished 2024-08-17T17:07:35.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:35 smithi136 ceph-mon[42580]: osdmap e43: 7 total, 6 up, 7 in 2024-08-17T17:07:35.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:35 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:07:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:35 smithi165 ceph-mon[34868]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-08-17T17:07:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:35 smithi165 ceph-mon[34868]: from='client.? 172.21.15.165:0/1062308453' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e838c26b-5e4a-41c6-b35b-b438ba64b237"}]: dispatch 2024-08-17T17:07:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:35 smithi165 ceph-mon[34868]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e838c26b-5e4a-41c6-b35b-b438ba64b237"}]: dispatch 2024-08-17T17:07:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:35 smithi165 ceph-mon[34868]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e838c26b-5e4a-41c6-b35b-b438ba64b237"}]': finished 2024-08-17T17:07:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:35 smithi165 ceph-mon[34868]: osdmap e43: 7 total, 6 up, 7 in 2024-08-17T17:07:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:35 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:07:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:36 smithi136 ceph-mon[31034]: from='client.? 172.21.15.165:0/1325947886' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:07:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:36 smithi136 ceph-mon[42580]: from='client.? 172.21.15.165:0/1325947886' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:07:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:36 smithi165 ceph-mon[34868]: from='client.? 172.21.15.165:0/1325947886' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:07:37.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:37 smithi136 ceph-mon[42580]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-08-17T17:07:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:37 smithi136 ceph-mon[31034]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-08-17T17:07:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:37 smithi165 ceph-mon[34868]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-08-17T17:07:39.713 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:39 smithi165 ceph-mon[34868]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-08-17T17:07:39.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:39 smithi136 ceph-mon[31034]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-08-17T17:07:39.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:39 smithi136 ceph-mon[42580]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-08-17T17:07:41.872 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:41 smithi165 ceph-mon[34868]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-08-17T17:07:41.873 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:41 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-08-17T17:07:41.873 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:41 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:41.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:41 smithi136 ceph-mon[31034]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-08-17T17:07:41.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:41 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-08-17T17:07:41.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:41 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:41.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:41 smithi136 ceph-mon[42580]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-08-17T17:07:41.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:41 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-08-17T17:07:41.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:41 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:42.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:42 smithi165 ceph-mon[34868]: Deploying daemon osd.6 on smithi165 2024-08-17T17:07:42.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:42 smithi136 ceph-mon[31034]: Deploying daemon osd.6 on smithi165 2024-08-17T17:07:42.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:42 smithi136 ceph-mon[42580]: Deploying daemon osd.6 on smithi165 2024-08-17T17:07:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:43 smithi165 ceph-mon[34868]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-17T17:07:43.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:43 smithi136 ceph-mon[42580]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-17T17:07:43.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:43 smithi136 ceph-mon[31034]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-17T17:07:45.697 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:45 smithi165 ceph-mon[34868]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-17T17:07:45.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:45 smithi136 ceph-mon[42580]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-17T17:07:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:45 smithi136 ceph-mon[31034]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-17T17:07:46.836 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:46 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:46.836 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:46 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:46.836 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:46 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:46.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:46 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:46 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:46 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:46.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:46 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:46.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:46 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:46.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:46 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:47.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:47 smithi136 ceph-mon[42580]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-17T17:07:47.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:47 smithi136 ceph-mon[31034]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-17T17:07:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:47 smithi165 ceph-mon[34868]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-17T17:07:49.648 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 6 on host 'smithi165' 2024-08-17T17:07:49.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:49 smithi136 ceph-mon[42580]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-17T17:07:49.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:49 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:49.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:49 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:49.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:49 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:49.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:49 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:49.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:49 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:49.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:49 smithi136 ceph-mon[42580]: from='osd.6 [v2:172.21.15.165:6816/1795674907,v1:172.21.15.165:6817/1795674907]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-08-17T17:07:49.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:49 smithi136 ceph-mon[42580]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-08-17T17:07:49.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:49 smithi136 ceph-mon[31034]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-17T17:07:49.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:49 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:49.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:49 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:49.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:49 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:49.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:49 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:49.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:49 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:49.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:49 smithi136 ceph-mon[31034]: from='osd.6 [v2:172.21.15.165:6816/1795674907,v1:172.21.15.165:6817/1795674907]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-08-17T17:07:49.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:49 smithi136 ceph-mon[31034]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-08-17T17:07:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:49 smithi165 ceph-mon[34868]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-17T17:07:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:49 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:49 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:49 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:49 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:49 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:50.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:49 smithi165 ceph-mon[34868]: from='osd.6 [v2:172.21.15.165:6816/1795674907,v1:172.21.15.165:6817/1795674907]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-08-17T17:07:50.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:49 smithi165 ceph-mon[34868]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-08-17T17:07:50.313 DEBUG:teuthology.orchestra.run.smithi165:osd.6> sudo journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.6.service 2024-08-17T17:07:50.315 INFO:tasks.cephadm:Deploying osd.7 on smithi165 with /dev/vg_nvme/lv_1... 2024-08-17T17:07:50.315 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-08-17T17:07:50.805 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:50 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:50.805 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:50 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:50.805 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:50 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:50.806 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:50 smithi165 ceph-mon[34868]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-08-17T17:07:50.806 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:50 smithi165 ceph-mon[34868]: osdmap e44: 7 total, 6 up, 7 in 2024-08-17T17:07:50.806 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:50 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:07:50.806 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:50 smithi165 ceph-mon[34868]: from='osd.6 [v2:172.21.15.165:6816/1795674907,v1:172.21.15.165:6817/1795674907]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-08-17T17:07:50.806 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:50 smithi165 ceph-mon[34868]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-08-17T17:07:50.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:50 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:50.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:50 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:50 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:50 smithi136 ceph-mon[42580]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-08-17T17:07:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:50 smithi136 ceph-mon[42580]: osdmap e44: 7 total, 6 up, 7 in 2024-08-17T17:07:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:50 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:07:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:50 smithi136 ceph-mon[42580]: from='osd.6 [v2:172.21.15.165:6816/1795674907,v1:172.21.15.165:6817/1795674907]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-08-17T17:07:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:50 smithi136 ceph-mon[42580]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-08-17T17:07:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:50 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:50 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:50 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:50.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:50 smithi136 ceph-mon[31034]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-08-17T17:07:50.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:50 smithi136 ceph-mon[31034]: osdmap e44: 7 total, 6 up, 7 in 2024-08-17T17:07:50.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:50 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:07:50.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:50 smithi136 ceph-mon[31034]: from='osd.6 [v2:172.21.15.165:6816/1795674907,v1:172.21.15.165:6817/1795674907]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-08-17T17:07:50.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:50 smithi136 ceph-mon[31034]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-08-17T17:07:51.140 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:07:51.585 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 17:07:51 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[52331]: 2024-08-17T17:07:51.333+0000 7f914bcfa640 -1 osd.6 0 waiting for initial osdmap 2024-08-17T17:07:51.585 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 17:07:51 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[52331]: 2024-08-17T17:07:51.341+0000 7f9143b1b640 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:07:51.939 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:51 smithi165 ceph-mon[34868]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-17T17:07:51.939 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:51 smithi165 ceph-mon[34868]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-08-17T17:07:51.939 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:51 smithi165 ceph-mon[34868]: osdmap e45: 7 total, 6 up, 7 in 2024-08-17T17:07:51.939 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:51 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:07:51.939 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:51 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:07:51.939 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:51 smithi165 ceph-mon[34868]: from='osd.6 ' entity='osd.6' 2024-08-17T17:07:51.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:51 smithi136 ceph-mon[42580]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-17T17:07:51.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:51 smithi136 ceph-mon[42580]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-08-17T17:07:51.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:51 smithi136 ceph-mon[42580]: osdmap e45: 7 total, 6 up, 7 in 2024-08-17T17:07:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:51 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:07:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:51 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:07:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:51 smithi136 ceph-mon[42580]: from='osd.6 ' entity='osd.6' 2024-08-17T17:07:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:51 smithi136 ceph-mon[31034]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-17T17:07:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:51 smithi136 ceph-mon[31034]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-08-17T17:07:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:51 smithi136 ceph-mon[31034]: osdmap e45: 7 total, 6 up, 7 in 2024-08-17T17:07:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:51 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:07:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:51 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:07:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:51 smithi136 ceph-mon[31034]: from='osd.6 ' entity='osd.6' 2024-08-17T17:07:53.125 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:52 smithi165 ceph-mon[34868]: purged_snaps scrub starts 2024-08-17T17:07:53.126 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:52 smithi165 ceph-mon[34868]: purged_snaps scrub ok 2024-08-17T17:07:53.126 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:52 smithi165 ceph-mon[34868]: osd.6 [v2:172.21.15.165:6816/1795674907,v1:172.21.15.165:6817/1795674907] boot 2024-08-17T17:07:53.126 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:52 smithi165 ceph-mon[34868]: osdmap e46: 7 total, 7 up, 7 in 2024-08-17T17:07:53.126 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:52 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:07:53.126 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:52 smithi165 ceph-mon[34868]: osdmap e47: 7 total, 7 up, 7 in 2024-08-17T17:07:53.126 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:52 smithi165 ceph-mon[34868]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:07:53.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:52 smithi136 ceph-mon[42580]: purged_snaps scrub starts 2024-08-17T17:07:53.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:52 smithi136 ceph-mon[42580]: purged_snaps scrub ok 2024-08-17T17:07:53.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:52 smithi136 ceph-mon[42580]: osd.6 [v2:172.21.15.165:6816/1795674907,v1:172.21.15.165:6817/1795674907] boot 2024-08-17T17:07:53.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:52 smithi136 ceph-mon[42580]: osdmap e46: 7 total, 7 up, 7 in 2024-08-17T17:07:53.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:52 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:07:53.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:52 smithi136 ceph-mon[42580]: osdmap e47: 7 total, 7 up, 7 in 2024-08-17T17:07:53.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:52 smithi136 ceph-mon[42580]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:07:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:52 smithi136 ceph-mon[31034]: purged_snaps scrub starts 2024-08-17T17:07:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:52 smithi136 ceph-mon[31034]: purged_snaps scrub ok 2024-08-17T17:07:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:52 smithi136 ceph-mon[31034]: osd.6 [v2:172.21.15.165:6816/1795674907,v1:172.21.15.165:6817/1795674907] boot 2024-08-17T17:07:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:52 smithi136 ceph-mon[31034]: osdmap e46: 7 total, 7 up, 7 in 2024-08-17T17:07:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:52 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:07:53.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:52 smithi136 ceph-mon[31034]: osdmap e47: 7 total, 7 up, 7 in 2024-08-17T17:07:53.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:52 smithi136 ceph-mon[31034]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:07:54.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:54 smithi165 ceph-mon[34868]: osdmap e48: 7 total, 7 up, 7 in 2024-08-17T17:07:54.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:54 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:54.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:54 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:54.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:54 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:07:54.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:54 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:07:54.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:54 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:07:54.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:54 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:54.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:54 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:54.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:54 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:54.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:54 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:54 smithi136 ceph-mon[42580]: osdmap e48: 7 total, 7 up, 7 in 2024-08-17T17:07:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:54 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:54 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:54 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:07:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:54 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:07:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:54 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:07:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:54 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:54.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:54 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:54.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:54 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:54.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:54 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:54.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:54 smithi136 ceph-mon[31034]: osdmap e48: 7 total, 7 up, 7 in 2024-08-17T17:07:54.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:54 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:54.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:54 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:54.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:54 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:07:54.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:54 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:07:54.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:54 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:07:54.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:54 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:54.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:54 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:54.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:54 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:54.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:54 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:07:55.645 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:55 smithi165 ceph-mon[34868]: Detected new or changed devices on smithi165 2024-08-17T17:07:55.645 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:55 smithi165 ceph-mon[34868]: Adjusting osd_memory_target on smithi165 to 5696M 2024-08-17T17:07:55.645 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:55 smithi165 ceph-mon[34868]: osdmap e49: 7 total, 7 up, 7 in 2024-08-17T17:07:55.645 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:55 smithi165 ceph-mon[34868]: pgmap v151: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 159 KiB/s, 0 objects/s recovering 2024-08-17T17:07:55.647 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-08-17T17:07:55.666 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi165:vg_nvme/lv_1 2024-08-17T17:07:55.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:55 smithi136 ceph-mon[42580]: Detected new or changed devices on smithi165 2024-08-17T17:07:55.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:55 smithi136 ceph-mon[42580]: Adjusting osd_memory_target on smithi165 to 5696M 2024-08-17T17:07:55.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:55 smithi136 ceph-mon[42580]: osdmap e49: 7 total, 7 up, 7 in 2024-08-17T17:07:55.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:55 smithi136 ceph-mon[42580]: pgmap v151: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 159 KiB/s, 0 objects/s recovering 2024-08-17T17:07:55.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:55 smithi136 ceph-mon[31034]: Detected new or changed devices on smithi165 2024-08-17T17:07:55.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:55 smithi136 ceph-mon[31034]: Adjusting osd_memory_target on smithi165 to 5696M 2024-08-17T17:07:55.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:55 smithi136 ceph-mon[31034]: osdmap e49: 7 total, 7 up, 7 in 2024-08-17T17:07:55.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:55 smithi136 ceph-mon[31034]: pgmap v151: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 159 KiB/s, 0 objects/s recovering 2024-08-17T17:07:55.867 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:07:56.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:56 smithi165 ceph-mon[34868]: osdmap e50: 7 total, 7 up, 7 in 2024-08-17T17:07:56.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:56 smithi136 ceph-mon[31034]: osdmap e50: 7 total, 7 up, 7 in 2024-08-17T17:07:56.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:56 smithi136 ceph-mon[42580]: osdmap e50: 7 total, 7 up, 7 in 2024-08-17T17:07:57.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:57 smithi136 ceph-mon[31034]: pgmap v153: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-08-17T17:07:57.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:57 smithi136 ceph-mon[42580]: pgmap v153: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-08-17T17:07:57.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:57 smithi165 ceph-mon[34868]: pgmap v153: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-08-17T17:07:58.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:58 smithi136 ceph-mon[31034]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:07:58.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:58 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:07:58.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:58 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:07:58.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:58 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:58.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:58 smithi136 ceph-mon[42580]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:07:58.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:58 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:07:58.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:58 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:07:58.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:58 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:58.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:58 smithi165 ceph-mon[34868]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:07:58.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:58 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:07:58.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:58 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:07:58.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:58 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:59.405 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:07:59 smithi165 ceph-mon[34868]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-08-17T17:07:59.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:07:59 smithi136 ceph-mon[31034]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-08-17T17:07:59.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:07:59 smithi136 ceph-mon[42580]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-08-17T17:08:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:00 smithi165 ceph-mon[34868]: from='client.? 172.21.15.165:0/2623227574' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de2fab2e-177a-4eb0-9262-7bbef0d9837f"}]: dispatch 2024-08-17T17:08:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:00 smithi165 ceph-mon[34868]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de2fab2e-177a-4eb0-9262-7bbef0d9837f"}]: dispatch 2024-08-17T17:08:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:00 smithi165 ceph-mon[34868]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "de2fab2e-177a-4eb0-9262-7bbef0d9837f"}]': finished 2024-08-17T17:08:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:00 smithi165 ceph-mon[34868]: osdmap e51: 8 total, 7 up, 8 in 2024-08-17T17:08:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:00 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:08:00.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:00 smithi136 ceph-mon[31034]: from='client.? 172.21.15.165:0/2623227574' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de2fab2e-177a-4eb0-9262-7bbef0d9837f"}]: dispatch 2024-08-17T17:08:00.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:00 smithi136 ceph-mon[31034]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de2fab2e-177a-4eb0-9262-7bbef0d9837f"}]: dispatch 2024-08-17T17:08:00.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:00 smithi136 ceph-mon[31034]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "de2fab2e-177a-4eb0-9262-7bbef0d9837f"}]': finished 2024-08-17T17:08:00.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:00 smithi136 ceph-mon[31034]: osdmap e51: 8 total, 7 up, 8 in 2024-08-17T17:08:00.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:00 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:08:00.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:00 smithi136 ceph-mon[42580]: from='client.? 172.21.15.165:0/2623227574' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de2fab2e-177a-4eb0-9262-7bbef0d9837f"}]: dispatch 2024-08-17T17:08:00.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:00 smithi136 ceph-mon[42580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de2fab2e-177a-4eb0-9262-7bbef0d9837f"}]: dispatch 2024-08-17T17:08:00.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:00 smithi136 ceph-mon[42580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "de2fab2e-177a-4eb0-9262-7bbef0d9837f"}]': finished 2024-08-17T17:08:00.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:00 smithi136 ceph-mon[42580]: osdmap e51: 8 total, 7 up, 8 in 2024-08-17T17:08:00.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:00 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:08:01.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:01 smithi165 ceph-mon[34868]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:01.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:01 smithi165 ceph-mon[34868]: from='client.? 172.21.15.165:0/2363091632' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:08:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:01 smithi136 ceph-mon[31034]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:01 smithi136 ceph-mon[31034]: from='client.? 172.21.15.165:0/2363091632' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:08:01.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:01 smithi136 ceph-mon[42580]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:01.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:01 smithi136 ceph-mon[42580]: from='client.? 172.21.15.165:0/2363091632' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:08:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:03 smithi165 ceph-mon[34868]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:03 smithi136 ceph-mon[31034]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:03.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:03 smithi136 ceph-mon[42580]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:05.668 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:05 smithi165 ceph-mon[34868]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:05 smithi136 ceph-mon[31034]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:05.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:05 smithi136 ceph-mon[42580]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:06 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-08-17T17:08:06.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:06 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:08:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:06 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-08-17T17:08:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:06 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:08:06.948 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:06 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-08-17T17:08:06.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:06 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:08:07.667 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:07 smithi165 ceph-mon[34868]: Deploying daemon osd.7 on smithi165 2024-08-17T17:08:07.667 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:07 smithi165 ceph-mon[34868]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:07.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:07 smithi136 ceph-mon[31034]: Deploying daemon osd.7 on smithi165 2024-08-17T17:08:07.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:07 smithi136 ceph-mon[31034]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:07.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:07 smithi136 ceph-mon[42580]: Deploying daemon osd.7 on smithi165 2024-08-17T17:08:07.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:07 smithi136 ceph-mon[42580]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:09 smithi136 ceph-mon[31034]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:09.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:09 smithi136 ceph-mon[42580]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:09.974 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:09 smithi165 ceph-mon[34868]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:11.944 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:11 smithi165 ceph-mon[34868]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:11.944 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:11 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:08:11.944 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:11 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:11.944 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:11 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:11.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:11 smithi136 ceph-mon[31034]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:11.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:11 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:08:11.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:11 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:11.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:11 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:11.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:11 smithi136 ceph-mon[42580]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:11.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:11 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:08:11.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:11 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:11.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:11 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:13.878 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:13 smithi165 ceph-mon[34868]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:13 smithi136 ceph-mon[31034]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:13.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:13 smithi136 ceph-mon[42580]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:14.695 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 7 on host 'smithi165' 2024-08-17T17:08:15.026 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:14 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:15.027 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:14 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:15.027 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:14 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:08:15.027 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:14 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:08:15.027 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:14 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:15.027 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:14 smithi165 ceph-mon[34868]: from='osd.7 [v2:172.21.15.165:6824/4195921091,v1:172.21.15.165:6825/4195921091]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-08-17T17:08:15.027 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:14 smithi165 ceph-mon[34868]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-08-17T17:08:15.027 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:14 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:08:15.027 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:14 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:15.027 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:14 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:15.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:14 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:15.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:14 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:15.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:14 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:08:15.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:14 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:08:15.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:14 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:15.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:14 smithi136 ceph-mon[31034]: from='osd.7 [v2:172.21.15.165:6824/4195921091,v1:172.21.15.165:6825/4195921091]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-08-17T17:08:15.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:14 smithi136 ceph-mon[31034]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-08-17T17:08:15.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:14 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:08:15.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:14 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:15.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:14 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:15.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:14 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:15.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:14 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:15.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:14 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:08:15.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:14 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:08:15.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:14 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:15.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:14 smithi136 ceph-mon[42580]: from='osd.7 [v2:172.21.15.165:6824/4195921091,v1:172.21.15.165:6825/4195921091]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-08-17T17:08:15.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:14 smithi136 ceph-mon[42580]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-08-17T17:08:15.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:14 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:08:15.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:14 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:15.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:14 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:15.323 DEBUG:teuthology.orchestra.run.smithi165:osd.7> sudo journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.7.service 2024-08-17T17:08:15.326 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-08-17T17:08:15.326 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd stat -f json 2024-08-17T17:08:15.528 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:15.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:15 smithi136 ceph-mon[31034]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:15.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:15 smithi136 ceph-mon[31034]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-08-17T17:08:15.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:15 smithi136 ceph-mon[31034]: osdmap e52: 8 total, 7 up, 8 in 2024-08-17T17:08:15.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:15 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:08:15.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:15 smithi136 ceph-mon[31034]: from='osd.7 [v2:172.21.15.165:6824/4195921091,v1:172.21.15.165:6825/4195921091]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-08-17T17:08:15.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:15 smithi136 ceph-mon[31034]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-08-17T17:08:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:15 smithi136 ceph-mon[42580]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:15.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:15 smithi136 ceph-mon[42580]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-08-17T17:08:15.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:15 smithi136 ceph-mon[42580]: osdmap e52: 8 total, 7 up, 8 in 2024-08-17T17:08:15.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:15 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:08:15.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:15 smithi136 ceph-mon[42580]: from='osd.7 [v2:172.21.15.165:6824/4195921091,v1:172.21.15.165:6825/4195921091]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-08-17T17:08:15.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:15 smithi136 ceph-mon[42580]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-08-17T17:08:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:15 smithi165 ceph-mon[34868]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:15 smithi165 ceph-mon[34868]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-08-17T17:08:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:15 smithi165 ceph-mon[34868]: osdmap e52: 8 total, 7 up, 8 in 2024-08-17T17:08:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:15 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:08:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:15 smithi165 ceph-mon[34868]: from='osd.7 [v2:172.21.15.165:6824/4195921091,v1:172.21.15.165:6825/4195921091]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-08-17T17:08:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:15 smithi165 ceph-mon[34868]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-08-17T17:08:16.508 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 17:08:16 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T17:08:16.224+0000 7f639f5f6640 -1 osd.7 0 waiting for initial osdmap 2024-08-17T17:08:16.508 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 17:08:16 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T17:08:16.231+0000 7f639b41f640 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:08:17.391 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:08:17.392 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:17 smithi136 ceph-mon[31034]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-08-17T17:08:17.392 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:17 smithi136 ceph-mon[31034]: osdmap e53: 8 total, 7 up, 8 in 2024-08-17T17:08:17.392 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:17 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:08:17.392 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:17 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:08:17.392 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:17 smithi136 ceph-mon[31034]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:17.392 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:17 smithi136 ceph-mon[42580]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-08-17T17:08:17.392 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:17 smithi136 ceph-mon[42580]: osdmap e53: 8 total, 7 up, 8 in 2024-08-17T17:08:17.392 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:17 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:08:17.393 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:17 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:08:17.393 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:17 smithi136 ceph-mon[42580]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:17 smithi165 ceph-mon[34868]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-08-17T17:08:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:17 smithi165 ceph-mon[34868]: osdmap e53: 8 total, 7 up, 8 in 2024-08-17T17:08:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:17 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:08:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:17 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:08:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:17 smithi165 ceph-mon[34868]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-17T17:08:17.915 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1723914497,"num_in_osds":8,"osd_in_since":1723914479,"num_remapped_pgs":1} 2024-08-17T17:08:17.916 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-17T17:08:18.113 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:18.478 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[31034]: purged_snaps scrub starts 2024-08-17T17:08:18.478 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[31034]: purged_snaps scrub ok 2024-08-17T17:08:18.478 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[31034]: osd.7 [v2:172.21.15.165:6824/4195921091,v1:172.21.15.165:6825/4195921091] boot 2024-08-17T17:08:18.478 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[31034]: osdmap e54: 8 total, 8 up, 8 in 2024-08-17T17:08:18.478 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:08:18.479 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[31034]: osdmap e55: 8 total, 8 up, 8 in 2024-08-17T17:08:18.479 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/250912202' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-08-17T17:08:18.479 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:18.479 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:18.479 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:18.479 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:18.479 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:18.479 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:18.479 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:18.479 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:08:18.479 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:08:18.479 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:18.480 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[42580]: purged_snaps scrub starts 2024-08-17T17:08:18.480 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[42580]: purged_snaps scrub ok 2024-08-17T17:08:18.480 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[42580]: osd.7 [v2:172.21.15.165:6824/4195921091,v1:172.21.15.165:6825/4195921091] boot 2024-08-17T17:08:18.480 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[42580]: osdmap e54: 8 total, 8 up, 8 in 2024-08-17T17:08:18.480 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:08:18.480 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[42580]: osdmap e55: 8 total, 8 up, 8 in 2024-08-17T17:08:18.480 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/250912202' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-08-17T17:08:18.480 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:18.480 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:18.480 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:18.480 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:18.480 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:18.480 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:18.481 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:18.481 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:08:18.481 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:08:18.481 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:18 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:18.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:18 smithi165 ceph-mon[34868]: purged_snaps scrub starts 2024-08-17T17:08:18.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:18 smithi165 ceph-mon[34868]: purged_snaps scrub ok 2024-08-17T17:08:18.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:18 smithi165 ceph-mon[34868]: osd.7 [v2:172.21.15.165:6824/4195921091,v1:172.21.15.165:6825/4195921091] boot 2024-08-17T17:08:18.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:18 smithi165 ceph-mon[34868]: osdmap e54: 8 total, 8 up, 8 in 2024-08-17T17:08:18.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:18 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:08:18.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:18 smithi165 ceph-mon[34868]: osdmap e55: 8 total, 8 up, 8 in 2024-08-17T17:08:18.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:18 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/250912202' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-08-17T17:08:18.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:18 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:18.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:18 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:18.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:18 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:18.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:18 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:18.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:18 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:18.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:18 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:18.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:18 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:18.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:18 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:08:18.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:18 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:08:18.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:18 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:19.651 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:19 smithi136 ceph-mon[31034]: Detected new or changed devices on smithi165 2024-08-17T17:08:19.651 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:19 smithi136 ceph-mon[31034]: Adjusting osd_memory_target on smithi165 to 4272M 2024-08-17T17:08:19.651 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:19 smithi136 ceph-mon[31034]: osdmap e56: 8 total, 8 up, 8 in 2024-08-17T17:08:19.651 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:19 smithi136 ceph-mon[31034]: pgmap v170: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 175 KiB/s, 0 objects/s recovering 2024-08-17T17:08:19.651 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:19 smithi136 ceph-mon[42580]: Detected new or changed devices on smithi165 2024-08-17T17:08:19.651 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:19 smithi136 ceph-mon[42580]: Adjusting osd_memory_target on smithi165 to 4272M 2024-08-17T17:08:19.652 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:19 smithi136 ceph-mon[42580]: osdmap e56: 8 total, 8 up, 8 in 2024-08-17T17:08:19.652 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:19 smithi136 ceph-mon[42580]: pgmap v170: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 175 KiB/s, 0 objects/s recovering 2024-08-17T17:08:19.685 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:19 smithi165 ceph-mon[34868]: Detected new or changed devices on smithi165 2024-08-17T17:08:19.685 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:19 smithi165 ceph-mon[34868]: Adjusting osd_memory_target on smithi165 to 4272M 2024-08-17T17:08:19.685 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:19 smithi165 ceph-mon[34868]: osdmap e56: 8 total, 8 up, 8 in 2024-08-17T17:08:19.685 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:19 smithi165 ceph-mon[34868]: pgmap v170: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 175 KiB/s, 0 objects/s recovering 2024-08-17T17:08:20.014 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:08:20.014 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":57,"fsid":"8270436c-5cba-11ef-bccf-c7b262605968","created":"2024-08-17T17:03:17.682767+0000","modified":"2024-08-17T17:08:19.319552+0000","last_up_change":"2024-08-17T17:08:17.220684+0000","last_in_change":"2024-08-17T17:07:59.960068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-17T17:06:20.772596+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51eb458d-9f33-4b87-b7f5-8acb076c16ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3864095089},{"type":"v1","addr":"172.21.15.136:6803","nonce":3864095089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3864095089},{"type":"v1","addr":"172.21.15.136:6805","nonce":3864095089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":3864095089},{"type":"v1","addr":"172.21.15.136:6809","nonce":3864095089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3864095089},{"type":"v1","addr":"172.21.15.136:6807","nonce":3864095089}]},"public_addr":"172.21.15.136:6803/3864095089","cluster_addr":"172.21.15.136:6805/3864095089","heartbeat_back_addr":"172.21.15.136:6809/3864095089","heartbeat_front_addr":"172.21.15.136:6807/3864095089","state":["exists","up"]},{"osd":1,"uuid":"aa73fa16-8736-478c-b07a-e674c18698ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":3797355920},{"type":"v1","addr":"172.21.15.136:6811","nonce":3797355920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":3797355920},{"type":"v1","addr":"172.21.15.136:6813","nonce":3797355920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3797355920},{"type":"v1","addr":"172.21.15.136:6817","nonce":3797355920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":3797355920},{"type":"v1","addr":"172.21.15.136:6815","nonce":3797355920}]},"public_addr":"172.21.15.136:6811/3797355920","cluster_addr":"172.21.15.136:6813/3797355920","heartbeat_back_addr":"172.21.15.136:6817/3797355920","heartbeat_front_addr":"172.21.15.136:6815/3797355920","state":["exists","up"]},{"osd":2,"uuid":"4d2612be-7068-48d9-bb0c-992ebccaa091","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1674243363},{"type":"v1","addr":"172.21.15.136:6819","nonce":1674243363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1674243363},{"type":"v1","addr":"172.21.15.136:6821","nonce":1674243363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1674243363},{"type":"v1","addr":"172.21.15.136:6825","nonce":1674243363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1674243363},{"type":"v1","addr":"172.21.15.136:6823","nonce":1674243363}]},"public_addr":"172.21.15.136:6819/1674243363","cluster_addr":"172.21.15.136:6821/1674243363","heartbeat_back_addr":"172.21.15.136:6825/1674243363","heartbeat_front_addr":"172.21.15.136:6823/1674243363","state":["exists","up"]},{"osd":3,"uuid":"30b8a4f2-be6d-4d5f-8397-06c2e17cfc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3486272897},{"type":"v1","addr":"172.21.15.136:6827","nonce":3486272897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3486272897},{"type":"v1","addr":"172.21.15.136:6829","nonce":3486272897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3486272897},{"type":"v1","addr":"172.21.15.136:6833","nonce":3486272897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3486272897},{"type":"v1","addr":"172.21.15.136:6831","nonce":3486272897}]},"public_addr":"172.21.15.136:6827/3486272897","cluster_addr":"172.21.15.136:6829/3486272897","heartbeat_back_addr":"172.21.15.136:6833/3486272897","heartbeat_front_addr":"172.21.15.136:6831/3486272897","state":["exists","up"]},{"osd":4,"uuid":"ae4074cf-0069-403f-acc7-74bedab266bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6801","nonce":842623090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6803","nonce":842623090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6807","nonce":842623090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6805","nonce":842623090}]},"public_addr":"172.21.15.165:6801/842623090","cluster_addr":"172.21.15.165:6803/842623090","heartbeat_back_addr":"172.21.15.165:6807/842623090","heartbeat_front_addr":"172.21.15.165:6805/842623090","state":["exists","up"]},{"osd":5,"uuid":"586c23a7-4831-4b4d-9a06-c1bfcc1c5286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6809","nonce":3602301990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6811","nonce":3602301990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6815","nonce":3602301990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6813","nonce":3602301990}]},"public_addr":"172.21.15.165:6809/3602301990","cluster_addr":"172.21.15.165:6811/3602301990","heartbeat_back_addr":"172.21.15.165:6815/3602301990","heartbeat_front_addr":"172.21.15.165:6813/3602301990","state":["exists","up"]},{"osd":6,"uuid":"e838c26b-5e4a-41c6-b35b-b438ba64b237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6817","nonce":1795674907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6819","nonce":1795674907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6823","nonce":1795674907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6821","nonce":1795674907}]},"public_addr":"172.21.15.165:6817/1795674907","cluster_addr":"172.21.15.165:6819/1795674907","heartbeat_back_addr":"172.21.15.165:6823/1795674907","heartbeat_front_addr":"172.21.15.165:6821/1795674907","state":["exists","up"]},{"osd":7,"uuid":"de2fab2e-177a-4eb0-9262-7bbef0d9837f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6825","nonce":4195921091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6827","nonce":4195921091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6831","nonce":4195921091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6829","nonce":4195921091}]},"public_addr":"172.21.15.165:6825/4195921091","cluster_addr":"172.21.15.165:6827/4195921091","heartbeat_back_addr":"172.21.15.165:6831/4195921091","heartbeat_front_addr":"172.21.15.165:6829/4195921091","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:30.507144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:54.456854+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:06:17.338746+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:06:42.262947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:02.983751+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:26.594502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:50.363918+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:08:15.455798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.136:0/3933059683":"2024-08-18T17:04:10.472088+0000","172.21.15.136:6800/3058734149":"2024-08-18T17:04:10.472088+0000","172.21.15.136:6801/878395364":"2024-08-18T17:03:48.417471+0000","172.21.15.136:6801/3058734149":"2024-08-18T17:04:10.472088+0000","172.21.15.136:6800/878395364":"2024-08-18T17:03:48.417471+0000","172.21.15.136:0/2336022730":"2024-08-18T17:03:48.417471+0000","172.21.15.136:0/1414082347":"2024-08-18T17:04:10.472088+0000","172.21.15.136:0/3380731497":"2024-08-18T17:03:48.417471+0000","172.21.15.136:0/1577563918":"2024-08-18T17:04:10.472088+0000","172.21.15.136:0/1381030503":"2024-08-18T17:03:48.417471+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-17T17:08:20.593 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:20 smithi165 ceph-mon[34868]: osdmap e57: 8 total, 8 up, 8 in 2024-08-17T17:08:20.593 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:20 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/1878440007' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:08:20.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:20 smithi136 ceph-mon[31034]: osdmap e57: 8 total, 8 up, 8 in 2024-08-17T17:08:20.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:20 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/1878440007' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:08:20.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:20 smithi136 ceph-mon[42580]: osdmap e57: 8 total, 8 up, 8 in 2024-08-17T17:08:20.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:20 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/1878440007' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:08:20.728 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-17T17:06:20.772596+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-08-17T17:08:20.728 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-08-17T17:08:20.927 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:21 smithi136 ceph-mon[31034]: osdmap e58: 8 total, 8 up, 8 in 2024-08-17T17:08:21.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:21 smithi136 ceph-mon[31034]: pgmap v173: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 175 KiB/s, 0 objects/s recovering 2024-08-17T17:08:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:21 smithi136 ceph-mon[42580]: osdmap e58: 8 total, 8 up, 8 in 2024-08-17T17:08:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:21 smithi136 ceph-mon[42580]: pgmap v173: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 175 KiB/s, 0 objects/s recovering 2024-08-17T17:08:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:21 smithi165 ceph-mon[34868]: osdmap e58: 8 total, 8 up, 8 in 2024-08-17T17:08:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:21 smithi165 ceph-mon[34868]: pgmap v173: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 175 KiB/s, 0 objects/s recovering 2024-08-17T17:08:22.647 INFO:teuthology.orchestra.run.smithi136.stdout:pg_num: 1 2024-08-17T17:08:23.284 INFO:tasks.cephadm:Adding prometheus.a on smithi165 2024-08-17T17:08:23.285 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph orch apply prometheus '1;smithi165=a' 2024-08-17T17:08:23.502 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:08:23.917 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:23 smithi165 ceph-mon[34868]: pgmap v174: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-08-17T17:08:23.917 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:23 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/1964978519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-08-17T17:08:23.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:23 smithi136 ceph-mon[42580]: pgmap v174: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-08-17T17:08:23.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:23 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/1964978519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-08-17T17:08:23.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:23 smithi136 ceph-mon[31034]: pgmap v174: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-08-17T17:08:23.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:23 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/1964978519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-08-17T17:08:25.433 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled prometheus update... 2024-08-17T17:08:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:25 smithi165 ceph-mon[34868]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-08-17T17:08:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:25 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:25 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:08:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:25 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:08:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:25 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:08:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:25 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:25.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:25 smithi136 ceph-mon[31034]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-08-17T17:08:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:08:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:08:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:08:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:25 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:25 smithi136 ceph-mon[42580]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-08-17T17:08:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:25 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:25 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:08:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:25 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:08:25.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:25 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:08:25.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:25 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:26.064 DEBUG:teuthology.orchestra.run.smithi165:prometheus.a> sudo journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@prometheus.a.service 2024-08-17T17:08:26.067 INFO:tasks.cephadm:Adding node-exporter.a on smithi136 2024-08-17T17:08:26.067 INFO:tasks.cephadm:Adding node-exporter.b on smithi165 2024-08-17T17:08:26.067 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph orch apply node-exporter '2;smithi136=a;smithi165=b' 2024-08-17T17:08:26.316 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:08:26.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:26 smithi165 ceph-mon[34868]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:08:26.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:26 smithi165 ceph-mon[34868]: Saving service prometheus spec with placement smithi165=a;count:1 2024-08-17T17:08:26.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:26 smithi165 ceph-mon[34868]: Deploying daemon prometheus.a on smithi165 2024-08-17T17:08:26.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:26 smithi136 ceph-mon[42580]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:08:26.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:26 smithi136 ceph-mon[42580]: Saving service prometheus spec with placement smithi165=a;count:1 2024-08-17T17:08:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:26 smithi136 ceph-mon[42580]: Deploying daemon prometheus.a on smithi165 2024-08-17T17:08:26.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:26 smithi136 ceph-mon[31034]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:08:26.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:26 smithi136 ceph-mon[31034]: Saving service prometheus spec with placement smithi165=a;count:1 2024-08-17T17:08:26.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:26 smithi136 ceph-mon[31034]: Deploying daemon prometheus.a on smithi165 2024-08-17T17:08:27.882 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:27 smithi165 ceph-mon[34868]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:27.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:27 smithi136 ceph-mon[42580]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:27.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:27 smithi136 ceph-mon[31034]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:29.013 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled node-exporter update... 2024-08-17T17:08:29.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:29 smithi136 ceph-mon[42580]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:29.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:29 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:29.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:29 smithi136 ceph-mon[31034]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:29.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:29 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:29 smithi165 ceph-mon[34868]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:29 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:30.530 DEBUG:teuthology.orchestra.run.smithi136:node-exporter.a> sudo journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@node-exporter.a.service 2024-08-17T17:08:30.534 DEBUG:teuthology.orchestra.run.smithi165:node-exporter.b> sudo journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@node-exporter.b.service 2024-08-17T17:08:30.536 INFO:tasks.cephadm:Adding alertmanager.a on smithi136 2024-08-17T17:08:30.536 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph orch apply alertmanager '1;smithi136=a' 2024-08-17T17:08:30.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:30 smithi165 ceph-mon[34868]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi136=a;smithi165=b", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:08:30.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:30 smithi165 ceph-mon[34868]: Saving service node-exporter spec with placement smithi136=a;smithi165=b;count:2 2024-08-17T17:08:30.779 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:08:30.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:30 smithi136 ceph-mon[42580]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi136=a;smithi165=b", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:08:30.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:30 smithi136 ceph-mon[42580]: Saving service node-exporter spec with placement smithi136=a;smithi165=b;count:2 2024-08-17T17:08:30.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:30 smithi136 ceph-mon[31034]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi136=a;smithi165=b", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:08:30.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:30 smithi136 ceph-mon[31034]: Saving service node-exporter spec with placement smithi136=a;smithi165=b;count:2 2024-08-17T17:08:31.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:31 smithi165 ceph-mon[34868]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:31.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:31 smithi136 ceph-mon[42580]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:31.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:31 smithi136 ceph-mon[31034]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:33.016 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled alertmanager update... 2024-08-17T17:08:33.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:33 smithi136 ceph-mon[42580]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:33.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:33 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:33.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:33 smithi136 ceph-mon[31034]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:33.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:33 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:33 smithi165 ceph-mon[34868]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:33 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:34.436 DEBUG:teuthology.orchestra.run.smithi136:alertmanager.a> sudo journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@alertmanager.a.service 2024-08-17T17:08:34.439 INFO:tasks.cephadm:Adding grafana.a on smithi165 2024-08-17T17:08:34.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph orch apply grafana '1;smithi165=a' 2024-08-17T17:08:34.641 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:08:34.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:34 smithi165 ceph-mon[34868]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi136=a", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:08:34.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:34 smithi165 ceph-mon[34868]: Saving service alertmanager spec with placement smithi136=a;count:1 2024-08-17T17:08:34.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:34 smithi136 ceph-mon[31034]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi136=a", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:08:34.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:34 smithi136 ceph-mon[31034]: Saving service alertmanager spec with placement smithi136=a;count:1 2024-08-17T17:08:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:34 smithi136 ceph-mon[42580]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi136=a", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:08:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:34 smithi136 ceph-mon[42580]: Saving service alertmanager spec with placement smithi136=a;count:1 2024-08-17T17:08:35.916 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:35 smithi165 ceph-mon[34868]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:35.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:35 smithi136 ceph-mon[42580]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:35.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:35 smithi136 ceph-mon[31034]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:37.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:37 smithi136 ceph-mon[42580]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:37.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:37 smithi136 ceph-mon[31034]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:37 smithi165 ceph-mon[34868]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:38.416 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled grafana update... 2024-08-17T17:08:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:39 smithi136 ceph-mon[31034]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:08:39.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:39 smithi136 ceph-mon[31034]: Saving service grafana spec with placement smithi165=a;count:1 2024-08-17T17:08:39.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:39 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:39.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:39 smithi136 ceph-mon[31034]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:39 smithi136 ceph-mon[42580]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:08:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:39 smithi136 ceph-mon[42580]: Saving service grafana spec with placement smithi165=a;count:1 2024-08-17T17:08:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:39 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:39 smithi136 ceph-mon[42580]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:39.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:39 smithi165 ceph-mon[34868]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:08:39.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:39 smithi165 ceph-mon[34868]: Saving service grafana spec with placement smithi165=a;count:1 2024-08-17T17:08:39.751 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:39 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:39.751 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:39 smithi165 ceph-mon[34868]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:39.774 DEBUG:teuthology.orchestra.run.smithi165:grafana.a> sudo journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@grafana.a.service 2024-08-17T17:08:39.777 INFO:tasks.cephadm:Setting up client nodes... 2024-08-17T17:08:39.777 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-08-17T17:08:39.989 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:41.421 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 systemd[1]: Starting Ceph prometheus.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:08:41.673 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:41 smithi165 ceph-mon[34868]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:41.674 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 podman[63693]: 2024-08-17 17:08:41.323481195 +0000 UTC m=+0.018722678 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-08-17T17:08:41.674 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 podman[63693]: 2024-08-17 17:08:41.432944729 +0000 UTC m=+0.128186230 container create b55171200053202ccf2388527918b3031afc76a79e999f6e6b36759ec577ae94 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:08:41.674 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 podman[63693]: 2024-08-17 17:08:41.668771446 +0000 UTC m=+0.364012915 container init b55171200053202ccf2388527918b3031afc76a79e999f6e6b36759ec577ae94 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:08:41.674 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 podman[63693]: 2024-08-17 17:08:41.67350435 +0000 UTC m=+0.368745810 container start b55171200053202ccf2388527918b3031afc76a79e999f6e6b36759ec577ae94 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:08:41.925 INFO:teuthology.orchestra.run.smithi136.stdout:[client.0] 2024-08-17T17:08:41.926 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQAZ2cBmO+DJNhAAE4thsfI7CHnd/4F3Lf9VfQ== 2024-08-17T17:08:41.926 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:41 smithi136 ceph-mon[31034]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:41.926 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:41 smithi136 ceph-mon[42580]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:42.011 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:08:41.698Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-08-17T17:08:42.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:08:41.698Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-08-17T17:08:42.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:08:41.698Z caller=main.go:570 level=info host_details="(Linux 5.14.0-480.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024 x86_64 smithi165 (none))" 2024-08-17T17:08:42.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:08:41.698Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-08-17T17:08:42.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:08:41.698Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-08-17T17:08:42.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:08:41.699Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-08-17T17:08:42.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:08:41.700Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-08-17T17:08:42.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:08:41.701Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-08-17T17:08:42.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:08:41.701Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-08-17T17:08:42.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:08:41.703Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-08-17T17:08:42.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:08:41.703Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.77µs 2024-08-17T17:08:42.013 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:08:41.703Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-08-17T17:08:42.013 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:08:41.703Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-08-17T17:08:42.013 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:08:41.703Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=17.957µs wal_replay_duration=294.252µs wbl_replay_duration=120ns total_replay_duration=330.205µs 2024-08-17T17:08:42.013 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:08:41.704Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-08-17T17:08:42.013 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:08:41.704Z caller=main.go:1029 level=info msg="TSDB started" 2024-08-17T17:08:42.013 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:08:41.704Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-08-17T17:08:42.013 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:08:41.724Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.223673ms db_storage=1.112µs remote_storage=1.106µs web_handler=325ns query_engine=585ns scrape=452.004µs scrape_sd=94.154µs notify=821ns notify_sd=1.314µs rules=19.383012ms tracing=6.222µs 2024-08-17T17:08:42.013 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:08:41.724Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-08-17T17:08:42.013 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:08:41.724Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-08-17T17:08:42.013 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 bash[63693]: b55171200053202ccf2388527918b3031afc76a79e999f6e6b36759ec577ae94 2024-08-17T17:08:42.013 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:08:41 smithi165 systemd[1]: Started Ceph prometheus.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:08:42.524 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-17T17:08:42.524 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-08-17T17:08:42.524 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-08-17T17:08:42.563 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-08-17T17:08:42.769 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:08:43.158 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:42 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:43.158 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:42 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:43.158 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:42 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:43.158 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:42 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-08-17T17:08:43.158 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:42 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/72795677' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-17T17:08:43.159 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:42 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-17T17:08:43.159 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:42 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-17T17:08:43.159 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:42 smithi165 ceph-mon[34868]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:43.159 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:42 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: ignoring --setuser ceph since I am not root 2024-08-17T17:08:43.159 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:42 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: ignoring --setgroup ceph since I am not root 2024-08-17T17:08:43.160 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:42 smithi165 ceph-mgr[36982]: -- 172.21.15.165:0/742702960 <== mon.1 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ef296bad00 con 0x55ef2968f000 2024-08-17T17:08:43.160 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:43 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:43.004+0000 7ffb9858a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-17T17:08:43.160 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:43 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:43.155+0000 7ffb9858a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:08:43.168 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:42 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:43.168 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:42 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:43.168 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:42 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:43.168 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:42 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-08-17T17:08:43.169 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:42 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/72795677' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-17T17:08:43.169 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:42 smithi136 ceph-mon[31034]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-17T17:08:43.169 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:42 smithi136 ceph-mon[31034]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-17T17:08:43.169 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:42 smithi136 ceph-mon[31034]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:43.170 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:42 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ignoring --setuser ceph since I am not root 2024-08-17T17:08:43.170 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:42 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ignoring --setgroup ceph since I am not root 2024-08-17T17:08:43.170 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:43 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:43.017+0000 7fc2503be040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-17T17:08:43.170 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:43 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:43.166+0000 7fc2503be040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:08:43.170 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:42 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:43.171 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:42 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:43.171 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:42 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' 2024-08-17T17:08:43.171 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:42 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-08-17T17:08:43.171 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:42 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/72795677' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-17T17:08:43.171 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:42 smithi136 ceph-mon[42580]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-17T17:08:43.171 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:42 smithi136 ceph-mon[42580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-17T17:08:43.172 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:42 smithi136 ceph-mon[42580]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:43.446 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:43 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:43.235+0000 7fc2503be040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-17T17:08:43.471 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:43 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:43.223+0000 7ffb9858a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-17T17:08:43.755 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:43 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:43.482+0000 7fc2503be040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-17T17:08:43.755 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:43 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:43.543+0000 7fc2503be040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-17T17:08:43.760 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:43 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:43.470+0000 7ffb9858a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-17T17:08:43.761 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:43 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:43.530+0000 7ffb9858a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-17T17:08:44.195 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:43 smithi165 ceph-mon[34868]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-08-17T17:08:44.195 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:43 smithi165 ceph-mon[34868]: mgrmap e16: y(active, since 4m), standbys: x 2024-08-17T17:08:44.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:43 smithi136 ceph-mon[31034]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-08-17T17:08:44.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:43 smithi136 ceph-mon[31034]: mgrmap e16: y(active, since 4m), standbys: x 2024-08-17T17:08:44.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:43 smithi136 ceph-mon[42580]: from='mgr.14150 172.21.15.136:0/1088331175' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-08-17T17:08:44.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:43 smithi136 ceph-mon[42580]: mgrmap e16: y(active, since 4m), standbys: x 2024-08-17T17:08:44.759 INFO:teuthology.orchestra.run.smithi165.stdout:[client.1] 2024-08-17T17:08:44.759 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQAc2cBmZEb0LBAAmWIhPHky0r1CzEJYIlKqBg== 2024-08-17T17:08:44.759 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:44 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:44.470+0000 7ffb9858a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-17T17:08:44.760 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:44 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:44.590+0000 7ffb9858a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-17T17:08:44.760 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:44 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:44.647+0000 7ffb9858a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-17T17:08:44.845 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:44 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:44.465+0000 7fc2503be040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-17T17:08:44.845 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:44 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:44.579+0000 7fc2503be040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-17T17:08:44.845 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:44 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:44.634+0000 7fc2503be040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-17T17:08:44.845 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:44 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:44.843+0000 7fc2503be040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-17T17:08:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:44 smithi165 ceph-mon[34868]: from='client.? 172.21.15.165:0/4258022210' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-17T17:08:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:44 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-17T17:08:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:44 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-17T17:08:45.011 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:44 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:44.870+0000 7ffb9858a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-17T17:08:45.011 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:44 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:44.931+0000 7ffb9858a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-17T17:08:45.012 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:44 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:44.997+0000 7ffb9858a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-17T17:08:45.115 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:44 smithi136 ceph-mon[31034]: from='client.? 172.21.15.165:0/4258022210' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-17T17:08:45.115 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:44 smithi136 ceph-mon[31034]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-17T17:08:45.115 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:44 smithi136 ceph-mon[31034]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-17T17:08:45.115 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:44 smithi136 ceph-mon[42580]: from='client.? 172.21.15.165:0/4258022210' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-17T17:08:45.115 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:44 smithi136 ceph-mon[42580]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-17T17:08:45.115 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:44 smithi136 ceph-mon[42580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-17T17:08:45.116 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:44 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:44.905+0000 7fc2503be040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-17T17:08:45.116 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:44 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:44.971+0000 7fc2503be040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-17T17:08:45.116 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:45.113+0000 7fc2503be040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:08:45.411 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:45 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:45.141+0000 7ffb9858a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:08:45.433 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-08-17T17:08:45.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-08-17T17:08:45.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-08-17T17:08:45.471 INFO:tasks.cephadm:Creating RBD pool 2024-08-17T17:08:45.471 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool create rbd 8 2024-08-17T17:08:45.679 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:45.761 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:45 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:45.561+0000 7ffb9858a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-17T17:08:45.801 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:45.531+0000 7fc2503be040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-17T17:08:46.064 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:45.861+0000 7fc2503be040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-17T17:08:46.064 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:45.930+0000 7fc2503be040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-17T17:08:46.261 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:45 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:45.867+0000 7ffb9858a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-17T17:08:46.261 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:45 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:45.932+0000 7ffb9858a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-17T17:08:46.261 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:46.075+0000 7ffb9858a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-17T17:08:46.389 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:46 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:46.081+0000 7fc2503be040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-17T17:08:46.389 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:46 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:46.387+0000 7fc2503be040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-17T17:08:46.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:46 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:46.450+0000 7fc2503be040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-17T17:08:46.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:46 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:46.633+0000 7fc2503be040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-17T17:08:46.758 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:46.374+0000 7ffb9858a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-17T17:08:46.758 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:46.433+0000 7ffb9858a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-17T17:08:46.758 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:46.612+0000 7ffb9858a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-17T17:08:46.758 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:46.756+0000 7ffb9858a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-17T17:08:47.011 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:46.819+0000 7ffb9858a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-17T17:08:47.011 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:46.899+0000 7ffb9858a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-17T17:08:47.011 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:46.956+0000 7ffb9858a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-17T17:08:47.073 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:46 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:46.778+0000 7fc2503be040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-17T17:08:47.073 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:46 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:46.842+0000 7fc2503be040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-17T17:08:47.073 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:46 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:46.926+0000 7fc2503be040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-17T17:08:47.073 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:46 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:46.997+0000 7fc2503be040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-17T17:08:47.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:47.545+0000 7fc2503be040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-17T17:08:47.734 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:47.472+0000 7ffb9858a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-17T17:08:47.734 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:47.625+0000 7ffb9858a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-17T17:08:47.734 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: [17/Aug/2024:17:08:47] ENGINE Bus STARTING 2024-08-17T17:08:47.734 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: CherryPy Checker: 2024-08-17T17:08:47.734 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: The Application mounted at '' has an empty config. 2024-08-17T17:08:47.734 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: 2024-08-17T17:08:47.734 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: [17/Aug/2024:17:08:47] ENGINE Serving on http://:::9283 2024-08-17T17:08:47.734 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[36959]: [17/Aug/2024:17:08:47] ENGINE Bus STARTED 2024-08-17T17:08:47.851 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'rbd' created 2024-08-17T17:08:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/3183313594' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-08-17T17:08:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.165:0/1734158263' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:08:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.165:0/1734158263' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:08:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.165:0/1734158263' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:08:48.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.165:0/1734158263' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:08:48.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: Standby manager daemon x restarted 2024-08-17T17:08:48.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: Standby manager daemon x started 2024-08-17T17:08:48.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: Active manager daemon y restarted 2024-08-17T17:08:48.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: Activating manager daemon y 2024-08-17T17:08:48.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/3183313594' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-08-17T17:08:48.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: osdmap e59: 8 total, 8 up, 8 in 2024-08-17T17:08:48.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: mgrmap e17: y(active, starting, since 0.0098802s), standbys: x 2024-08-17T17:08:48.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:08:48.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:08:48.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:08:48.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-17T17:08:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-17T17:08:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:08:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:08:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:08:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:08:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:08:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:08:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:08:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:08:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-17T17:08:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-17T17:08:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-17T17:08:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: Manager daemon y is now available 2024-08-17T17:08:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:47 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:08:48.032 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/3183313594' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-08-17T17:08:48.032 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.165:0/1734158263' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:08:48.032 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.165:0/1734158263' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:08:48.032 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.165:0/1734158263' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:08:48.032 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.165:0/1734158263' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:08:48.032 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: Standby manager daemon x restarted 2024-08-17T17:08:48.033 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: Standby manager daemon x started 2024-08-17T17:08:48.033 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: Active manager daemon y restarted 2024-08-17T17:08:48.033 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: Activating manager daemon y 2024-08-17T17:08:48.033 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/3183313594' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-08-17T17:08:48.033 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: osdmap e59: 8 total, 8 up, 8 in 2024-08-17T17:08:48.033 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: mgrmap e17: y(active, starting, since 0.0098802s), standbys: x 2024-08-17T17:08:48.033 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:08:48.033 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:08:48.033 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:08:48.033 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-17T17:08:48.033 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-17T17:08:48.033 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:08:48.033 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:08:48.034 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:08:48.034 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:08:48.034 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:08:48.034 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:08:48.034 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:08:48.034 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:08:48.034 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-17T17:08:48.034 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-17T17:08:48.034 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-17T17:08:48.034 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: Manager daemon y is now available 2024-08-17T17:08:48.034 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:08:48.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/3183313594' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-08-17T17:08:48.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.165:0/1734158263' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:08:48.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.165:0/1734158263' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:08:48.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.165:0/1734158263' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:08:48.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.165:0/1734158263' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:08:48.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: Standby manager daemon x restarted 2024-08-17T17:08:48.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: Standby manager daemon x started 2024-08-17T17:08:48.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: Active manager daemon y restarted 2024-08-17T17:08:48.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: Activating manager daemon y 2024-08-17T17:08:48.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/3183313594' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-08-17T17:08:48.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: osdmap e59: 8 total, 8 up, 8 in 2024-08-17T17:08:48.036 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: mgrmap e17: y(active, starting, since 0.0098802s), standbys: x 2024-08-17T17:08:48.036 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:08:48.036 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:08:48.036 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:08:48.036 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-17T17:08:48.036 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-17T17:08:48.036 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:08:48.036 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:08:48.036 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:08:48.036 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:08:48.037 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:08:48.037 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:08:48.037 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:08:48.037 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:08:48.037 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-17T17:08:48.037 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-17T17:08:48.037 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-17T17:08:48.037 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: Manager daemon y is now available 2024-08-17T17:08:48.037 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:08:48.037 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:47.707+0000 7fc2503be040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-17T17:08:48.037 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:08:47] ENGINE Bus STARTING 2024-08-17T17:08:48.038 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: CherryPy Checker: 2024-08-17T17:08:48.038 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: The Application mounted at '' has an empty config. 2024-08-17T17:08:48.038 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:48.038 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:48 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:08:48] ENGINE Serving on http://:::9283 2024-08-17T17:08:48.038 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:48 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:08:48] ENGINE Bus STARTED 2024-08-17T17:08:48.511 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- rbd pool init rbd 2024-08-17T17:08:49.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:48 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:49.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:48 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:08:49.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:48 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:08:49.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:48 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:08:49.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:48 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:08:49.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:48 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:08:49.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:48 smithi136 ceph-mon[31034]: osdmap e60: 8 total, 8 up, 8 in 2024-08-17T17:08:49.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:48 smithi136 ceph-mon[31034]: mgrmap e18: y(active, since 1.02039s), standbys: x 2024-08-17T17:08:49.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:48 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:49.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:48 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:08:49.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:48 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:08:49.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:48 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:08:49.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:48 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:08:49.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:48 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:08:49.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:48 smithi136 ceph-mon[42580]: osdmap e60: 8 total, 8 up, 8 in 2024-08-17T17:08:49.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:48 smithi136 ceph-mon[42580]: mgrmap e18: y(active, since 1.02039s), standbys: x 2024-08-17T17:08:49.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:48 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:49.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:48 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:08:49.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:48 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:08:49.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:48 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:08:49.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:48 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:08:49.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:48 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:08:49.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:48 smithi165 ceph-mon[34868]: osdmap e60: 8 total, 8 up, 8 in 2024-08-17T17:08:49.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:48 smithi165 ceph-mon[34868]: mgrmap e18: y(active, since 1.02039s), standbys: x 2024-08-17T17:08:49.321 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:49.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:49.672+0000 7fc1a8ae1640 -1 log_channel(cephadm) log [ERR] : [17/Aug/2024:17:08:49] ENGINE Error in HTTPServer.serve 2024-08-17T17:08:49.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: Traceback (most recent call last): 2024-08-17T17:08:49.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:08:49.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: self._connections.run(self.expiration_interval) 2024-08-17T17:08:49.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:08:49.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: self._run(expiration_interval) 2024-08-17T17:08:49.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:08:49.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:08:49.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:08:49.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:08:49.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:08:49.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: s = self.context.wrap_socket( 2024-08-17T17:08:49.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:08:49.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: return self.sslsocket_class._create( 2024-08-17T17:08:49.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:08:49.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: self.do_handshake() 2024-08-17T17:08:49.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:08:49.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: self._sslobj.do_handshake() 2024-08-17T17:08:49.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:08:49.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:08:50.171 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-mon[31034]: pgmap v4: 9 pgs: 1 creating+activating, 8 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:50.171 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-mon[31034]: osdmap e61: 8 total, 8 up, 8 in 2024-08-17T17:08:50.172 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-mon[31034]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:08:50.172 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-mon[42580]: pgmap v4: 9 pgs: 1 creating+activating, 8 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:50.172 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-mon[42580]: osdmap e61: 8 total, 8 up, 8 in 2024-08-17T17:08:50.172 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:49 smithi136 ceph-mon[42580]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:08:50.206 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:49 smithi165 ceph-mon[34868]: pgmap v4: 9 pgs: 1 creating+activating, 8 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:50.206 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:49 smithi165 ceph-mon[34868]: osdmap e61: 8 total, 8 up, 8 in 2024-08-17T17:08:50.206 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:49 smithi165 ceph-mon[34868]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:08:51.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[31034]: [17/Aug/2024:17:08:49] ENGINE Bus STARTING 2024-08-17T17:08:51.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[31034]: [17/Aug/2024:17:08:49] ENGINE Serving on http://172.21.15.136:8765 2024-08-17T17:08:51.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[31034]: [17/Aug/2024:17:08:49] ENGINE Serving on https://172.21.15.136:7150 2024-08-17T17:08:51.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[31034]: [17/Aug/2024:17:08:49] ENGINE Bus STARTED 2024-08-17T17:08:51.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[31034]: [17/Aug/2024:17:08:49] ENGINE Error in HTTPServer.serve 2024-08-17T17:08:51.197 INFO:journalctl@ceph.mon.a.smithi136.stdout: Traceback (most recent call last): 2024-08-17T17:08:51.197 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:08:51.197 INFO:journalctl@ceph.mon.a.smithi136.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:08:51.197 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:08:51.197 INFO:journalctl@ceph.mon.a.smithi136.stdout: self._run(expiration_interval) 2024-08-17T17:08:51.197 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:08:51.198 INFO:journalctl@ceph.mon.a.smithi136.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:08:51.198 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:08:51.198 INFO:journalctl@ceph.mon.a.smithi136.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:08:51.198 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:08:51.198 INFO:journalctl@ceph.mon.a.smithi136.stdout: s = self.context.wrap_socket( 2024-08-17T17:08:51.198 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:08:51.198 INFO:journalctl@ceph.mon.a.smithi136.stdout: return self.sslsocket_class._create( 2024-08-17T17:08:51.198 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:08:51.198 INFO:journalctl@ceph.mon.a.smithi136.stdout: self.do_handshake() 2024-08-17T17:08:51.198 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:08:51.198 INFO:journalctl@ceph.mon.a.smithi136.stdout: self._sslobj.do_handshake() 2024-08-17T17:08:51.198 INFO:journalctl@ceph.mon.a.smithi136.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:08:51.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[31034]: pgmap v6: 9 pgs: 1 creating+activating, 8 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:51.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[31034]: mgrmap e19: y(active, since 2s), standbys: x 2024-08-17T17:08:51.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:51.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:51.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:51.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:51.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:51.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:51.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:51.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:51.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:51.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:51.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:51.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:51.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:51.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[42580]: [17/Aug/2024:17:08:49] ENGINE Bus STARTING 2024-08-17T17:08:51.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[42580]: [17/Aug/2024:17:08:49] ENGINE Serving on http://172.21.15.136:8765 2024-08-17T17:08:51.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[42580]: [17/Aug/2024:17:08:49] ENGINE Serving on https://172.21.15.136:7150 2024-08-17T17:08:51.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[42580]: [17/Aug/2024:17:08:49] ENGINE Bus STARTED 2024-08-17T17:08:51.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[42580]: [17/Aug/2024:17:08:49] ENGINE Error in HTTPServer.serve 2024-08-17T17:08:51.200 INFO:journalctl@ceph.mon.c.smithi136.stdout: Traceback (most recent call last): 2024-08-17T17:08:51.200 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:08:51.201 INFO:journalctl@ceph.mon.c.smithi136.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:08:51.201 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:08:51.201 INFO:journalctl@ceph.mon.c.smithi136.stdout: self._run(expiration_interval) 2024-08-17T17:08:51.201 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:08:51.201 INFO:journalctl@ceph.mon.c.smithi136.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:08:51.201 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:08:51.201 INFO:journalctl@ceph.mon.c.smithi136.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:08:51.201 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:08:51.201 INFO:journalctl@ceph.mon.c.smithi136.stdout: s = self.context.wrap_socket( 2024-08-17T17:08:51.201 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:08:51.201 INFO:journalctl@ceph.mon.c.smithi136.stdout: return self.sslsocket_class._create( 2024-08-17T17:08:51.202 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:08:51.202 INFO:journalctl@ceph.mon.c.smithi136.stdout: self.do_handshake() 2024-08-17T17:08:51.202 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:08:51.202 INFO:journalctl@ceph.mon.c.smithi136.stdout: self._sslobj.do_handshake() 2024-08-17T17:08:51.202 INFO:journalctl@ceph.mon.c.smithi136.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:08:51.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[42580]: pgmap v6: 9 pgs: 1 creating+activating, 8 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:51.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[42580]: mgrmap e19: y(active, since 2s), standbys: x 2024-08-17T17:08:51.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:51.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:51.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:51.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:51.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:51.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:51.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:51.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:51.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:51.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:51.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:51.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:51.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:50 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:51.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:50 smithi165 ceph-mon[34868]: [17/Aug/2024:17:08:49] ENGINE Bus STARTING 2024-08-17T17:08:51.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:50 smithi165 ceph-mon[34868]: [17/Aug/2024:17:08:49] ENGINE Serving on http://172.21.15.136:8765 2024-08-17T17:08:51.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:50 smithi165 ceph-mon[34868]: [17/Aug/2024:17:08:49] ENGINE Serving on https://172.21.15.136:7150 2024-08-17T17:08:51.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:50 smithi165 ceph-mon[34868]: [17/Aug/2024:17:08:49] ENGINE Bus STARTED 2024-08-17T17:08:51.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:50 smithi165 ceph-mon[34868]: [17/Aug/2024:17:08:49] ENGINE Error in HTTPServer.serve 2024-08-17T17:08:51.262 INFO:journalctl@ceph.mon.b.smithi165.stdout: Traceback (most recent call last): 2024-08-17T17:08:51.262 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:08:51.262 INFO:journalctl@ceph.mon.b.smithi165.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:08:51.262 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:08:51.262 INFO:journalctl@ceph.mon.b.smithi165.stdout: self._run(expiration_interval) 2024-08-17T17:08:51.262 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:08:51.262 INFO:journalctl@ceph.mon.b.smithi165.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:08:51.262 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:08:51.262 INFO:journalctl@ceph.mon.b.smithi165.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:08:51.262 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:08:51.263 INFO:journalctl@ceph.mon.b.smithi165.stdout: s = self.context.wrap_socket( 2024-08-17T17:08:51.263 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:08:51.263 INFO:journalctl@ceph.mon.b.smithi165.stdout: return self.sslsocket_class._create( 2024-08-17T17:08:51.263 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:08:51.263 INFO:journalctl@ceph.mon.b.smithi165.stdout: self.do_handshake() 2024-08-17T17:08:51.263 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:08:51.263 INFO:journalctl@ceph.mon.b.smithi165.stdout: self._sslobj.do_handshake() 2024-08-17T17:08:51.263 INFO:journalctl@ceph.mon.b.smithi165.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:08:51.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:50 smithi165 ceph-mon[34868]: pgmap v6: 9 pgs: 1 creating+activating, 8 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:51.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:50 smithi165 ceph-mon[34868]: mgrmap e19: y(active, since 2s), standbys: x 2024-08-17T17:08:51.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:50 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:51.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:50 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:51.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:50 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:51.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:50 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:51.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:50 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:51.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:50 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:51.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:50 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:51.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:50 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:51.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:50 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:51.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:50 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:51.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:50 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:51.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:50 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:08:51.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:50 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:52.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:51 smithi136 ceph-mon[31034]: Adjusting osd_memory_target on smithi165 to 4016M 2024-08-17T17:08:52.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:51 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/1844805592' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-08-17T17:08:52.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:51 smithi136 ceph-mon[31034]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-08-17T17:08:52.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:51 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:52.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:51 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:52.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:51 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:52.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:51 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:52.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:51 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:08:52.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:51 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:08:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:51 smithi136 ceph-mon[42580]: Adjusting osd_memory_target on smithi165 to 4016M 2024-08-17T17:08:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:51 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/1844805592' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-08-17T17:08:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:51 smithi136 ceph-mon[42580]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-08-17T17:08:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:51 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:51 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:52.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:51 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:52.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:51 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:52.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:51 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:08:52.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:51 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:08:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:51 smithi165 ceph-mon[34868]: Adjusting osd_memory_target on smithi165 to 4016M 2024-08-17T17:08:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:51 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/1844805592' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-08-17T17:08:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:51 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-08-17T17:08:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:51 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:51 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:51 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:51 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:51 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:08:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:51 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:08:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[31034]: Updating smithi136:/etc/ceph/ceph.conf 2024-08-17T17:08:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[31034]: Updating smithi165:/etc/ceph/ceph.conf 2024-08-17T17:08:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[31034]: pgmap v7: 9 pgs: 1 creating+activating, 8 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[31034]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-08-17T17:08:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[31034]: osdmap e62: 8 total, 8 up, 8 in 2024-08-17T17:08:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[31034]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:08:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[31034]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:08:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[31034]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:08:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[31034]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:08:53.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[31034]: mgrmap e20: y(active, since 4s), standbys: x 2024-08-17T17:08:53.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[31034]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:08:53.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[31034]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:08:53.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:53.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:53.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:53.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:53.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:53.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[31034]: Deploying daemon alertmanager.a on smithi136 2024-08-17T17:08:53.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[42580]: Updating smithi136:/etc/ceph/ceph.conf 2024-08-17T17:08:53.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[42580]: Updating smithi165:/etc/ceph/ceph.conf 2024-08-17T17:08:53.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[42580]: pgmap v7: 9 pgs: 1 creating+activating, 8 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:53.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[42580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-08-17T17:08:53.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[42580]: osdmap e62: 8 total, 8 up, 8 in 2024-08-17T17:08:53.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[42580]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:08:53.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[42580]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:08:53.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[42580]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:08:53.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[42580]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:08:53.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[42580]: mgrmap e20: y(active, since 4s), standbys: x 2024-08-17T17:08:53.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[42580]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:08:53.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[42580]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:08:53.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:53.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:53.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:53.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:53.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:53.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:52 smithi136 ceph-mon[42580]: Deploying daemon alertmanager.a on smithi136 2024-08-17T17:08:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:52 smithi165 ceph-mon[34868]: Updating smithi136:/etc/ceph/ceph.conf 2024-08-17T17:08:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:52 smithi165 ceph-mon[34868]: Updating smithi165:/etc/ceph/ceph.conf 2024-08-17T17:08:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:52 smithi165 ceph-mon[34868]: pgmap v7: 9 pgs: 1 creating+activating, 8 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:52 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-08-17T17:08:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:52 smithi165 ceph-mon[34868]: osdmap e62: 8 total, 8 up, 8 in 2024-08-17T17:08:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:52 smithi165 ceph-mon[34868]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:08:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:52 smithi165 ceph-mon[34868]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:08:53.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:52 smithi165 ceph-mon[34868]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:08:53.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:52 smithi165 ceph-mon[34868]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:08:53.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:52 smithi165 ceph-mon[34868]: mgrmap e20: y(active, since 4s), standbys: x 2024-08-17T17:08:53.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:52 smithi165 ceph-mon[34868]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:08:53.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:52 smithi165 ceph-mon[34868]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:08:53.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:52 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:53.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:52 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:53.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:52 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:53.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:52 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:53.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:52 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:08:53.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:52 smithi165 ceph-mon[34868]: Deploying daemon alertmanager.a on smithi136 2024-08-17T17:08:54.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:53 smithi136 ceph-mon[31034]: osdmap e63: 8 total, 8 up, 8 in 2024-08-17T17:08:54.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:53 smithi136 ceph-mon[31034]: pgmap v10: 9 pgs: 1 creating+activating, 8 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:54.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:53 smithi136 ceph-mon[42580]: osdmap e63: 8 total, 8 up, 8 in 2024-08-17T17:08:54.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:53 smithi136 ceph-mon[42580]: pgmap v10: 9 pgs: 1 creating+activating, 8 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:54.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:53 smithi165 ceph-mon[34868]: osdmap e63: 8 total, 8 up, 8 in 2024-08-17T17:08:54.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:53 smithi165 ceph-mon[34868]: pgmap v10: 9 pgs: 1 creating+activating, 8 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:08:55.073 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph config log 1 --format=json 2024-08-17T17:08:55.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:54 smithi136 ceph-mon[31034]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:08:55.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:54 smithi136 ceph-mon[31034]: Cluster is now healthy 2024-08-17T17:08:55.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:54 smithi136 ceph-mon[31034]: osdmap e64: 8 total, 8 up, 8 in 2024-08-17T17:08:55.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:54 smithi136 ceph-mon[42580]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:08:55.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:54 smithi136 ceph-mon[42580]: Cluster is now healthy 2024-08-17T17:08:55.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:54 smithi136 ceph-mon[42580]: osdmap e64: 8 total, 8 up, 8 in 2024-08-17T17:08:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:54 smithi165 ceph-mon[34868]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:08:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:54 smithi165 ceph-mon[34868]: Cluster is now healthy 2024-08-17T17:08:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:54 smithi165 ceph-mon[34868]: osdmap e64: 8 total, 8 up, 8 in 2024-08-17T17:08:55.282 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:56.165 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:55 smithi136 ceph-mon[31034]: pgmap v12: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 170 B/s wr, 19 op/s 2024-08-17T17:08:56.166 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:55 smithi136 ceph-mon[42580]: pgmap v12: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 170 B/s wr, 19 op/s 2024-08-17T17:08:56.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:55 smithi165 ceph-mon[34868]: pgmap v12: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 170 B/s wr, 19 op/s 2024-08-17T17:08:56.446 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:08:56 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:08:56] "GET /metrics HTTP/1.1" 200 25941 "" "Prometheus/2.43.0" 2024-08-17T17:08:58.922 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:08:59.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:58 smithi136 ceph-mon[31034]: pgmap v13: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 170 B/s wr, 19 op/s 2024-08-17T17:08:59.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:58 smithi136 ceph-mon[42580]: pgmap v13: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 170 B/s wr, 19 op/s 2024-08-17T17:08:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:58 smithi165 ceph-mon[34868]: pgmap v13: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 170 B/s wr, 19 op/s 2024-08-17T17:09:00.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:08:59 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/3296903764' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-08-17T17:09:00.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:08:59 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/3296903764' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-08-17T17:09:00.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:08:59 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/3296903764' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-08-17T17:09:01.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:00 smithi136 ceph-mon[31034]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 258 B/s wr, 14 op/s 2024-08-17T17:09:01.082 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:00 smithi136 ceph-mon[42580]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 258 B/s wr, 14 op/s 2024-08-17T17:09:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:00 smithi165 ceph-mon[34868]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 258 B/s wr, 14 op/s 2024-08-17T17:09:01.634 INFO:teuthology.orchestra.run.smithi136.stdout:[{"version":22,"timestamp":"2024-08-17T17:08:50.737496+0000","name":"","changes":[{"name":"osd/host:smithi165/osd_memory_target","previous_value":"4479562649","new_value":"4211127193"}]}] 2024-08-17T17:09:01.634 INFO:tasks.ceph_manager:config epoch is 22 2024-08-17T17:09:01.635 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-08-17T17:09:01.635 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-08-17T17:09:01.635 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph mgr dump --format=json 2024-08-17T17:09:02.165 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:01 smithi136 ceph-mon[31034]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 229 B/s wr, 13 op/s 2024-08-17T17:09:02.165 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:01 smithi136 ceph-mon[42580]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 229 B/s wr, 13 op/s 2024-08-17T17:09:02.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:01 smithi165 ceph-mon[34868]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 229 B/s wr, 13 op/s 2024-08-17T17:09:02.489 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:02.613 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:09:02 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:09:02] ENGINE Bus STOPPING 2024-08-17T17:09:02.920 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:09:02 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:09:02] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-08-17T17:09:02.920 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:09:02 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:09:02] ENGINE Bus STOPPED 2024-08-17T17:09:02.920 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:09:02 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:09:02] ENGINE Bus STARTING 2024-08-17T17:09:02.921 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:09:02 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:09:02] ENGINE Serving on http://:::9283 2024-08-17T17:09:02.921 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:09:02 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:09:02] ENGINE Bus STARTED 2024-08-17T17:09:03.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[42580]: osdmap e65: 8 total, 8 up, 8 in 2024-08-17T17:09:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[42580]: Regenerating cephadm self-signed grafana TLS certificates 2024-08-17T17:09:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-08-17T17:09:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[42580]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-08-17T17:09:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[42580]: Deploying daemon grafana.a on smithi165 2024-08-17T17:09:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:09:03.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[31034]: osdmap e65: 8 total, 8 up, 8 in 2024-08-17T17:09:03.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:03.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:03.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:03.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:03.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[31034]: Regenerating cephadm self-signed grafana TLS certificates 2024-08-17T17:09:03.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:03.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:03.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-08-17T17:09:03.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[31034]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-08-17T17:09:03.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:03.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[31034]: Deploying daemon grafana.a on smithi165 2024-08-17T17:09:03.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:03.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:03 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:09:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:03 smithi165 ceph-mon[34868]: osdmap e65: 8 total, 8 up, 8 in 2024-08-17T17:09:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:03 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:03 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:03 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:03 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:03 smithi165 ceph-mon[34868]: Regenerating cephadm self-signed grafana TLS certificates 2024-08-17T17:09:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:03 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:03.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:03 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:03.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:03 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-08-17T17:09:03.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:03 smithi165 ceph-mon[34868]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-08-17T17:09:03.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:03 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:03.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:03 smithi165 ceph-mon[34868]: Deploying daemon grafana.a on smithi165 2024-08-17T17:09:03.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:03 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:03.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:03 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:09:04.336 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:04 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[73386]: ts=2024-08-17T17:09:04.269Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000318712s 2024-08-17T17:09:04.613 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:04 smithi136 ceph-mon[31034]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 103 B/s wr, 0 op/s 2024-08-17T17:09:04.613 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:04 smithi136 ceph-mon[42580]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 103 B/s wr, 0 op/s 2024-08-17T17:09:04.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:04 smithi165 ceph-mon[34868]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 103 B/s wr, 0 op/s 2024-08-17T17:09:04.849 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:09:05.530 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":20,"flags":0,"active_gid":14442,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":803461847},{"type":"v1","addr":"172.21.15.136:6801","nonce":803461847}]},"active_addr":"172.21.15.136:6801/803461847","active_change":"2024-08-17T17:08:47.841296+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24346,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.136:8443/","prometheus":"http://172.21.15.136:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":59,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":2887741187}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":972617428}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":707907005}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":3166566423}]}]} 2024-08-17T17:09:05.537 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-08-17T17:09:05.537 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-08-17T17:09:05.537 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-17T17:09:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:05 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/2417846943' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-08-17T17:09:05.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:05 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/2417846943' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-08-17T17:09:05.738 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:05 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2417846943' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-08-17T17:09:06.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:06 smithi136 ceph-mon[31034]: pgmap v18: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 102 B/s wr, 0 op/s 2024-08-17T17:09:06.447 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:09:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:09:06] "GET /metrics HTTP/1.1" 200 28514 "" "Prometheus/2.43.0" 2024-08-17T17:09:06.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:06 smithi136 ceph-mon[42580]: pgmap v18: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 102 B/s wr, 0 op/s 2024-08-17T17:09:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:06 smithi165 ceph-mon[34868]: pgmap v18: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 102 B/s wr, 0 op/s 2024-08-17T17:09:07.673 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:09:07.674 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":65,"fsid":"8270436c-5cba-11ef-bccf-c7b262605968","created":"2024-08-17T17:03:17.682767+0000","modified":"2024-08-17T17:09:02.324301+0000","last_up_change":"2024-08-17T17:08:17.220684+0000","last_in_change":"2024-08-17T17:07:59.960068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-17T17:06:20.772596+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-17T17:08:47.840093+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51eb458d-9f33-4b87-b7f5-8acb076c16ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3864095089},{"type":"v1","addr":"172.21.15.136:6803","nonce":3864095089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3864095089},{"type":"v1","addr":"172.21.15.136:6805","nonce":3864095089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":3864095089},{"type":"v1","addr":"172.21.15.136:6809","nonce":3864095089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3864095089},{"type":"v1","addr":"172.21.15.136:6807","nonce":3864095089}]},"public_addr":"172.21.15.136:6803/3864095089","cluster_addr":"172.21.15.136:6805/3864095089","heartbeat_back_addr":"172.21.15.136:6809/3864095089","heartbeat_front_addr":"172.21.15.136:6807/3864095089","state":["exists","up"]},{"osd":1,"uuid":"aa73fa16-8736-478c-b07a-e674c18698ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":3797355920},{"type":"v1","addr":"172.21.15.136:6811","nonce":3797355920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":3797355920},{"type":"v1","addr":"172.21.15.136:6813","nonce":3797355920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3797355920},{"type":"v1","addr":"172.21.15.136:6817","nonce":3797355920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":3797355920},{"type":"v1","addr":"172.21.15.136:6815","nonce":3797355920}]},"public_addr":"172.21.15.136:6811/3797355920","cluster_addr":"172.21.15.136:6813/3797355920","heartbeat_back_addr":"172.21.15.136:6817/3797355920","heartbeat_front_addr":"172.21.15.136:6815/3797355920","state":["exists","up"]},{"osd":2,"uuid":"4d2612be-7068-48d9-bb0c-992ebccaa091","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1674243363},{"type":"v1","addr":"172.21.15.136:6819","nonce":1674243363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1674243363},{"type":"v1","addr":"172.21.15.136:6821","nonce":1674243363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1674243363},{"type":"v1","addr":"172.21.15.136:6825","nonce":1674243363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1674243363},{"type":"v1","addr":"172.21.15.136:6823","nonce":1674243363}]},"public_addr":"172.21.15.136:6819/1674243363","cluster_addr":"172.21.15.136:6821/1674243363","heartbeat_back_addr":"172.21.15.136:6825/1674243363","heartbeat_front_addr":"172.21.15.136:6823/1674243363","state":["exists","up"]},{"osd":3,"uuid":"30b8a4f2-be6d-4d5f-8397-06c2e17cfc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3486272897},{"type":"v1","addr":"172.21.15.136:6827","nonce":3486272897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3486272897},{"type":"v1","addr":"172.21.15.136:6829","nonce":3486272897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3486272897},{"type":"v1","addr":"172.21.15.136:6833","nonce":3486272897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3486272897},{"type":"v1","addr":"172.21.15.136:6831","nonce":3486272897}]},"public_addr":"172.21.15.136:6827/3486272897","cluster_addr":"172.21.15.136:6829/3486272897","heartbeat_back_addr":"172.21.15.136:6833/3486272897","heartbeat_front_addr":"172.21.15.136:6831/3486272897","state":["exists","up"]},{"osd":4,"uuid":"ae4074cf-0069-403f-acc7-74bedab266bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6801","nonce":842623090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6803","nonce":842623090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6807","nonce":842623090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6805","nonce":842623090}]},"public_addr":"172.21.15.165:6801/842623090","cluster_addr":"172.21.15.165:6803/842623090","heartbeat_back_addr":"172.21.15.165:6807/842623090","heartbeat_front_addr":"172.21.15.165:6805/842623090","state":["exists","up"]},{"osd":5,"uuid":"586c23a7-4831-4b4d-9a06-c1bfcc1c5286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6809","nonce":3602301990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6811","nonce":3602301990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6815","nonce":3602301990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6813","nonce":3602301990}]},"public_addr":"172.21.15.165:6809/3602301990","cluster_addr":"172.21.15.165:6811/3602301990","heartbeat_back_addr":"172.21.15.165:6815/3602301990","heartbeat_front_addr":"172.21.15.165:6813/3602301990","state":["exists","up"]},{"osd":6,"uuid":"e838c26b-5e4a-41c6-b35b-b438ba64b237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6817","nonce":1795674907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6819","nonce":1795674907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6823","nonce":1795674907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6821","nonce":1795674907}]},"public_addr":"172.21.15.165:6817/1795674907","cluster_addr":"172.21.15.165:6819/1795674907","heartbeat_back_addr":"172.21.15.165:6823/1795674907","heartbeat_front_addr":"172.21.15.165:6821/1795674907","state":["exists","up"]},{"osd":7,"uuid":"de2fab2e-177a-4eb0-9262-7bbef0d9837f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6825","nonce":4195921091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6827","nonce":4195921091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6831","nonce":4195921091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6829","nonce":4195921091}]},"public_addr":"172.21.15.165:6825/4195921091","cluster_addr":"172.21.15.165:6827/4195921091","heartbeat_back_addr":"172.21.15.165:6831/4195921091","heartbeat_front_addr":"172.21.15.165:6829/4195921091","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:30.507144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:54.456854+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:06:17.338746+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:06:42.262947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:02.983751+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:26.594502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:50.363918+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:08:15.455798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.136:0/1450531984":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/1905186682":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/2540800865":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/1577563918":"2024-08-18T17:04:10.472088+0000","172.21.15.136:0/1414082347":"2024-08-18T17:04:10.472088+0000","172.21.15.136:0/2336022730":"2024-08-18T17:03:48.417471+0000","172.21.15.136:6800/878395364":"2024-08-18T17:03:48.417471+0000","172.21.15.136:6801/3058734149":"2024-08-18T17:04:10.472088+0000","172.21.15.136:6801/3513268997":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/3380731497":"2024-08-18T17:03:48.417471+0000","172.21.15.136:6801/878395364":"2024-08-18T17:03:48.417471+0000","172.21.15.136:6800/3058734149":"2024-08-18T17:04:10.472088+0000","172.21.15.136:0/1381030503":"2024-08-18T17:03:48.417471+0000","172.21.15.136:6800/3513268997":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/3933059683":"2024-08-18T17:04:10.472088+0000","172.21.15.136:0/2931810343":"2024-08-18T17:08:47.841029+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-17T17:09:07.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:07 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/286394893' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:09:07.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:07 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/286394893' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:09:08.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:07 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/286394893' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:09:08.353 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-08-17T17:09:08.353 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-17T17:09:08.552 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:08.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:08 smithi136 ceph-mon[31034]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 102 B/s wr, 0 op/s 2024-08-17T17:09:08.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:08 smithi136 ceph-mon[42580]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 102 B/s wr, 0 op/s 2024-08-17T17:09:09.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:08 smithi165 ceph-mon[34868]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 102 B/s wr, 0 op/s 2024-08-17T17:09:10.417 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:09:10.417 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":65,"fsid":"8270436c-5cba-11ef-bccf-c7b262605968","created":"2024-08-17T17:03:17.682767+0000","modified":"2024-08-17T17:09:02.324301+0000","last_up_change":"2024-08-17T17:08:17.220684+0000","last_in_change":"2024-08-17T17:07:59.960068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-17T17:06:20.772596+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-17T17:08:47.840093+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51eb458d-9f33-4b87-b7f5-8acb076c16ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3864095089},{"type":"v1","addr":"172.21.15.136:6803","nonce":3864095089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3864095089},{"type":"v1","addr":"172.21.15.136:6805","nonce":3864095089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":3864095089},{"type":"v1","addr":"172.21.15.136:6809","nonce":3864095089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3864095089},{"type":"v1","addr":"172.21.15.136:6807","nonce":3864095089}]},"public_addr":"172.21.15.136:6803/3864095089","cluster_addr":"172.21.15.136:6805/3864095089","heartbeat_back_addr":"172.21.15.136:6809/3864095089","heartbeat_front_addr":"172.21.15.136:6807/3864095089","state":["exists","up"]},{"osd":1,"uuid":"aa73fa16-8736-478c-b07a-e674c18698ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":3797355920},{"type":"v1","addr":"172.21.15.136:6811","nonce":3797355920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":3797355920},{"type":"v1","addr":"172.21.15.136:6813","nonce":3797355920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3797355920},{"type":"v1","addr":"172.21.15.136:6817","nonce":3797355920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":3797355920},{"type":"v1","addr":"172.21.15.136:6815","nonce":3797355920}]},"public_addr":"172.21.15.136:6811/3797355920","cluster_addr":"172.21.15.136:6813/3797355920","heartbeat_back_addr":"172.21.15.136:6817/3797355920","heartbeat_front_addr":"172.21.15.136:6815/3797355920","state":["exists","up"]},{"osd":2,"uuid":"4d2612be-7068-48d9-bb0c-992ebccaa091","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1674243363},{"type":"v1","addr":"172.21.15.136:6819","nonce":1674243363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1674243363},{"type":"v1","addr":"172.21.15.136:6821","nonce":1674243363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1674243363},{"type":"v1","addr":"172.21.15.136:6825","nonce":1674243363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1674243363},{"type":"v1","addr":"172.21.15.136:6823","nonce":1674243363}]},"public_addr":"172.21.15.136:6819/1674243363","cluster_addr":"172.21.15.136:6821/1674243363","heartbeat_back_addr":"172.21.15.136:6825/1674243363","heartbeat_front_addr":"172.21.15.136:6823/1674243363","state":["exists","up"]},{"osd":3,"uuid":"30b8a4f2-be6d-4d5f-8397-06c2e17cfc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3486272897},{"type":"v1","addr":"172.21.15.136:6827","nonce":3486272897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3486272897},{"type":"v1","addr":"172.21.15.136:6829","nonce":3486272897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3486272897},{"type":"v1","addr":"172.21.15.136:6833","nonce":3486272897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3486272897},{"type":"v1","addr":"172.21.15.136:6831","nonce":3486272897}]},"public_addr":"172.21.15.136:6827/3486272897","cluster_addr":"172.21.15.136:6829/3486272897","heartbeat_back_addr":"172.21.15.136:6833/3486272897","heartbeat_front_addr":"172.21.15.136:6831/3486272897","state":["exists","up"]},{"osd":4,"uuid":"ae4074cf-0069-403f-acc7-74bedab266bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6801","nonce":842623090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6803","nonce":842623090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6807","nonce":842623090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6805","nonce":842623090}]},"public_addr":"172.21.15.165:6801/842623090","cluster_addr":"172.21.15.165:6803/842623090","heartbeat_back_addr":"172.21.15.165:6807/842623090","heartbeat_front_addr":"172.21.15.165:6805/842623090","state":["exists","up"]},{"osd":5,"uuid":"586c23a7-4831-4b4d-9a06-c1bfcc1c5286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6809","nonce":3602301990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6811","nonce":3602301990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6815","nonce":3602301990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6813","nonce":3602301990}]},"public_addr":"172.21.15.165:6809/3602301990","cluster_addr":"172.21.15.165:6811/3602301990","heartbeat_back_addr":"172.21.15.165:6815/3602301990","heartbeat_front_addr":"172.21.15.165:6813/3602301990","state":["exists","up"]},{"osd":6,"uuid":"e838c26b-5e4a-41c6-b35b-b438ba64b237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6817","nonce":1795674907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6819","nonce":1795674907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6823","nonce":1795674907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6821","nonce":1795674907}]},"public_addr":"172.21.15.165:6817/1795674907","cluster_addr":"172.21.15.165:6819/1795674907","heartbeat_back_addr":"172.21.15.165:6823/1795674907","heartbeat_front_addr":"172.21.15.165:6821/1795674907","state":["exists","up"]},{"osd":7,"uuid":"de2fab2e-177a-4eb0-9262-7bbef0d9837f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6825","nonce":4195921091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6827","nonce":4195921091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6831","nonce":4195921091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6829","nonce":4195921091}]},"public_addr":"172.21.15.165:6825/4195921091","cluster_addr":"172.21.15.165:6827/4195921091","heartbeat_back_addr":"172.21.15.165:6831/4195921091","heartbeat_front_addr":"172.21.15.165:6829/4195921091","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:30.507144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:54.456854+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:06:17.338746+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:06:42.262947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:02.983751+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:26.594502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:50.363918+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:08:15.455798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.136:0/1450531984":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/1905186682":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/2540800865":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/1577563918":"2024-08-18T17:04:10.472088+0000","172.21.15.136:0/1414082347":"2024-08-18T17:04:10.472088+0000","172.21.15.136:0/2336022730":"2024-08-18T17:03:48.417471+0000","172.21.15.136:6800/878395364":"2024-08-18T17:03:48.417471+0000","172.21.15.136:6801/3058734149":"2024-08-18T17:04:10.472088+0000","172.21.15.136:6801/3513268997":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/3380731497":"2024-08-18T17:03:48.417471+0000","172.21.15.136:6801/878395364":"2024-08-18T17:03:48.417471+0000","172.21.15.136:6800/3058734149":"2024-08-18T17:04:10.472088+0000","172.21.15.136:0/1381030503":"2024-08-18T17:03:48.417471+0000","172.21.15.136:6800/3513268997":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/3933059683":"2024-08-18T17:04:10.472088+0000","172.21.15.136:0/2931810343":"2024-08-18T17:08:47.841029+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-17T17:09:10.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:10 smithi136 ceph-mon[31034]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:10.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:10 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/2936898652' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:09:10.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:10 smithi136 ceph-mon[42580]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:10.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:10 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/2936898652' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:09:10.997 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-08-17T17:09:10.998 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-08-17T17:09:10.998 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-08-17T17:09:10.999 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-08-17T17:09:10.999 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-08-17T17:09:10.999 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-08-17T17:09:11.000 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-08-17T17:09:11.000 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-08-17T17:09:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:10 smithi165 ceph-mon[34868]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:10 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2936898652' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:09:11.407 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:11.407 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:11.410 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:11.412 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:11.412 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:11.416 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:11.420 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:11.440 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:12.446 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:12 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[73386]: ts=2024-08-17T17:09:12.270Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.00179794s 2024-08-17T17:09:13.131 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:12 smithi136 ceph-mon[31034]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:13.132 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:12 smithi136 ceph-mon[42580]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:13.209 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:12 smithi165 ceph-mon[34868]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:14.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:14 smithi136 ceph-mon[31034]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:14.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:14 smithi136 ceph-mon[42580]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:14 smithi165 ceph-mon[34868]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:16.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:16 smithi165 systemd[1]: Starting Ceph grafana.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:09:16.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:09:16 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:09:16] "GET /metrics HTTP/1.1" 200 28514 "" "Prometheus/2.43.0" 2024-08-17T17:09:16.875 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:16 smithi165 podman[65790]: 2024-08-17 17:09:16.502686507 +0000 UTC m=+0.016300336 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-08-17T17:09:16.876 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:16 smithi165 podman[65790]: 2024-08-17 17:09:16.605667918 +0000 UTC m=+0.119281747 container create be80a687c4027e4ca5ef4907b379acf45ad965f4750d3be4d8a82458d15667b7 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a, io.buildah.version=1.32.0, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, io.openshift.expose-services=, release=239.1651231664, maintainer=Paul Cuzner , description=Ceph Grafana Container, vcs-type=git, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, name=ubi8, vendor=Red Hat, Inc., architecture=x86_64, version=8.5) 2024-08-17T17:09:16.876 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:16 smithi165 podman[65790]: 2024-08-17 17:09:16.874959194 +0000 UTC m=+0.388573031 container init be80a687c4027e4ca5ef4907b379acf45ad965f4750d3be4d8a82458d15667b7 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, vcs-type=git, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, io.buildah.version=1.32.0, version=8.5) 2024-08-17T17:09:17.007 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:16 smithi136 ceph-mon[31034]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:17.007 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:16 smithi136 ceph-mon[42580]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:17.127 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:16 smithi165 ceph-mon[34868]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:17.127 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:16 smithi165 podman[65790]: 2024-08-17 17:09:16.879878692 +0000 UTC m=+0.393492520 container start be80a687c4027e4ca5ef4907b379acf45ad965f4750d3be4d8a82458d15667b7 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, io.buildah.version=1.32.0, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , description=Ceph Grafana Container, architecture=x86_64, build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, release=239.1651231664, vcs-type=git, com.redhat.component=ubi8-container) 2024-08-17T17:09:17.127 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:16 smithi165 bash[65790]: be80a687c4027e4ca5ef4907b379acf45ad965f4750d3be4d8a82458d15667b7 2024-08-17T17:09:17.127 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:16 smithi165 systemd[1]: Started Ceph grafana.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:09:17.127 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=settings t=2024-08-17T17:09:17.050914371Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-08-17T17:09:17.127 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=settings t=2024-08-17T17:09:17.051143472Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-08-17T17:09:17.127 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=settings t=2024-08-17T17:09:17.051179981Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-08-17T17:09:17.128 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=settings t=2024-08-17T17:09:17.051192943Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-08-17T17:09:17.128 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=settings t=2024-08-17T17:09:17.051204633Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-08-17T17:09:17.128 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=settings t=2024-08-17T17:09:17.051215981Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-08-17T17:09:17.128 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=settings t=2024-08-17T17:09:17.051226735Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-08-17T17:09:17.128 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=settings t=2024-08-17T17:09:17.051237816Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-08-17T17:09:17.128 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=settings t=2024-08-17T17:09:17.051248502Z level=info msg="Path Home" path=/usr/share/grafana 2024-08-17T17:09:17.128 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=settings t=2024-08-17T17:09:17.05125937Z level=info msg="Path Data" path=/var/lib/grafana 2024-08-17T17:09:17.128 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=settings t=2024-08-17T17:09:17.051268632Z level=info msg="Path Logs" path=/var/log/grafana 2024-08-17T17:09:17.129 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=settings t=2024-08-17T17:09:17.051279112Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-08-17T17:09:17.129 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=settings t=2024-08-17T17:09:17.05129024Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-08-17T17:09:17.129 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=settings t=2024-08-17T17:09:17.051301114Z level=info msg="App mode production" 2024-08-17T17:09:17.129 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=sqlstore t=2024-08-17T17:09:17.051371452Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-08-17T17:09:17.129 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=sqlstore t=2024-08-17T17:09:17.051397824Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-08-17T17:09:17.129 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.052533166Z level=info msg="Starting DB migrations" 2024-08-17T17:09:17.129 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.052633866Z level=info msg="Executing migration" id="create migration_log table" 2024-08-17T17:09:17.129 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.05316536Z level=info msg="Executing migration" id="create user table" 2024-08-17T17:09:17.129 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.053635353Z level=info msg="Executing migration" id="add unique index user.login" 2024-08-17T17:09:17.129 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.054412397Z level=info msg="Executing migration" id="add unique index user.email" 2024-08-17T17:09:17.129 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.055382877Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-08-17T17:09:17.129 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.055987721Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-08-17T17:09:17.129 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.056901654Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-08-17T17:09:17.130 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.059709962Z level=info msg="Executing migration" id="create user table v2" 2024-08-17T17:09:17.130 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.06232754Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-08-17T17:09:17.130 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.064984045Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-08-17T17:09:17.130 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.068048549Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-08-17T17:09:17.130 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.07058085Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-08-17T17:09:17.130 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.070973347Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-08-17T17:09:17.130 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.073896626Z level=info msg="Executing migration" id="Update user table charset" 2024-08-17T17:09:17.130 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.076561071Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-08-17T17:09:17.130 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.079337946Z level=info msg="Executing migration" id="Add missing user data" 2024-08-17T17:09:17.130 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.079857095Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-08-17T17:09:17.130 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.082704125Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-08-17T17:09:17.130 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.085396371Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-08-17T17:09:17.130 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.088254742Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-08-17T17:09:17.131 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.090888845Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-08-17T17:09:17.131 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.093823739Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-08-17T17:09:17.131 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.096714225Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-08-17T17:09:17.131 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.099423482Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-08-17T17:09:17.131 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.102184804Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-08-17T17:09:17.131 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.104880317Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-08-17T17:09:17.131 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.105198825Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-08-17T17:09:17.131 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.107899635Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-08-17T17:09:17.131 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.11080022Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-08-17T17:09:17.132 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.113603589Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-08-17T17:09:17.132 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.116260709Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-08-17T17:09:17.132 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.118943638Z level=info msg="Executing migration" id="create temp_user v2" 2024-08-17T17:09:17.132 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.121342236Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-08-17T17:09:17.132 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.124169816Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-08-17T17:09:17.132 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.126841153Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-08-17T17:09:17.380 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.129378733Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-08-17T17:09:17.380 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.132191995Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-08-17T17:09:17.380 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.135000379Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-08-17T17:09:17.380 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.137761952Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-08-17T17:09:17.381 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.140449935Z level=info msg="Executing migration" id="create star table" 2024-08-17T17:09:17.381 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.143214289Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-08-17T17:09:17.381 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.143645481Z level=info msg="Executing migration" id="create org table v1" 2024-08-17T17:09:17.381 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.144648906Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-08-17T17:09:17.381 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.145253788Z level=info msg="Executing migration" id="create org_user table v1" 2024-08-17T17:09:17.381 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.145619084Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-08-17T17:09:17.381 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.146437028Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-08-17T17:09:17.381 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.147231722Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-08-17T17:09:17.381 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.148192591Z level=info msg="Executing migration" id="Update org table charset" 2024-08-17T17:09:17.381 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.148565087Z level=info msg="Executing migration" id="Update org_user table charset" 2024-08-17T17:09:17.381 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.149402255Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-08-17T17:09:17.382 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.149734193Z level=info msg="Executing migration" id="create dashboard table" 2024-08-17T17:09:17.382 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.150195846Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-08-17T17:09:17.382 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.151164441Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-08-17T17:09:17.382 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.151658675Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-08-17T17:09:17.382 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.15252996Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-08-17T17:09:17.382 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.153342054Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-08-17T17:09:17.382 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.154153857Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-08-17T17:09:17.382 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.154995266Z level=info msg="Executing migration" id="create dashboard v2" 2024-08-17T17:09:17.382 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.155831547Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-08-17T17:09:17.383 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.156772272Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-08-17T17:09:17.383 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.157546669Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-08-17T17:09:17.383 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.157899146Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-08-17T17:09:17.383 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.158913543Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-08-17T17:09:17.383 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.159639327Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-08-17T17:09:17.383 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.160478292Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-08-17T17:09:17.383 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.161009496Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-08-17T17:09:17.383 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.161509849Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-08-17T17:09:17.383 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.162417578Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-08-17T17:09:17.383 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.163156505Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-08-17T17:09:17.383 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.163580175Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-08-17T17:09:17.383 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.16439735Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-08-17T17:09:17.384 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.164709834Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-08-17T17:09:17.384 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.164974714Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-08-17T17:09:17.384 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.165499499Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-08-17T17:09:17.384 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.16602198Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-08-17T17:09:17.384 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.166913419Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-08-17T17:09:17.384 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.167884248Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-08-17T17:09:17.384 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.168367428Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-08-17T17:09:17.384 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.169255254Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-08-17T17:09:17.384 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.170138171Z level=info msg="Executing migration" id="Update dashboard title length" 2024-08-17T17:09:17.384 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.173143183Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-08-17T17:09:17.384 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.173576997Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-08-17T17:09:17.384 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.176538673Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-08-17T17:09:17.384 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.179468493Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-08-17T17:09:17.385 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.182430468Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-08-17T17:09:17.385 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.182852803Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-08-17T17:09:17.385 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.185765092Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-08-17T17:09:17.385 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.188534745Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-08-17T17:09:17.385 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.191384556Z level=info msg="Executing migration" id="Add check_sum column" 2024-08-17T17:09:17.385 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.191979019Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-08-17T17:09:17.385 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.19491763Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-08-17T17:09:17.385 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.1977956Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-08-17T17:09:17.387 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.20048547Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-08-17T17:09:17.387 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.20093659Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-08-17T17:09:17.387 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.203902927Z level=info msg="Executing migration" id="create data_source table" 2024-08-17T17:09:17.387 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.206498489Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-08-17T17:09:17.387 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.206926143Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-08-17T17:09:17.387 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.209489748Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-08-17T17:09:17.387 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.212253461Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-08-17T17:09:17.388 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.214883678Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-08-17T17:09:17.388 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.21769708Z level=info msg="Executing migration" id="create data_source table v2" 2024-08-17T17:09:17.388 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.218200053Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-08-17T17:09:17.388 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.221124195Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-08-17T17:09:17.388 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.223889313Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-08-17T17:09:17.388 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.22658741Z level=info msg="Executing migration" id="Add column with_credentials" 2024-08-17T17:09:17.388 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.227168821Z level=info msg="Executing migration" id="Add secure json data column" 2024-08-17T17:09:17.388 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.230030067Z level=info msg="Executing migration" id="Update data_source table charset" 2024-08-17T17:09:17.388 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.232697031Z level=info msg="Executing migration" id="Update initial version to 1" 2024-08-17T17:09:17.388 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.235381678Z level=info msg="Executing migration" id="Add read_only data column" 2024-08-17T17:09:17.388 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.236032709Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-08-17T17:09:17.389 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.238695212Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-08-17T17:09:17.389 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.24146476Z level=info msg="Executing migration" id="Add uid column" 2024-08-17T17:09:17.389 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.244176672Z level=info msg="Executing migration" id="Update uid value" 2024-08-17T17:09:17.390 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.244598339Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-08-17T17:09:17.390 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.24722179Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-08-17T17:09:17.390 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.247726611Z level=info msg="Executing migration" id="create api_key table" 2024-08-17T17:09:17.390 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.250171821Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-08-17T17:09:17.390 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.250886687Z level=info msg="Executing migration" id="add index api_key.key" 2024-08-17T17:09:17.390 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.251330425Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-08-17T17:09:17.390 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.252315829Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-08-17T17:09:17.390 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.253053278Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-08-17T17:09:17.390 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.254021408Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-08-17T17:09:17.390 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.25493286Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-08-17T17:09:17.390 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.256179202Z level=info msg="Executing migration" id="create api_key table v2" 2024-08-17T17:09:17.390 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.256943811Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-08-17T17:09:17.391 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.25737724Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-08-17T17:09:17.391 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.257824159Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-08-17T17:09:17.391 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.258270679Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-08-17T17:09:17.391 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.25912357Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-08-17T17:09:17.391 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.260037289Z level=info msg="Executing migration" id="Update api_key table charset" 2024-08-17T17:09:17.391 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.260372011Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-08-17T17:09:17.391 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.260889576Z level=info msg="Executing migration" id="Add service account foreign key" 2024-08-17T17:09:17.391 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.26141939Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-08-17T17:09:17.391 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.261739847Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-08-17T17:09:17.391 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.26230481Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-08-17T17:09:17.391 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.262869878Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-08-17T17:09:17.391 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.263922595Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-08-17T17:09:17.391 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.26470714Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-08-17T17:09:17.392 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.265492204Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-08-17T17:09:17.392 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.265949869Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-08-17T17:09:17.392 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.266429974Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-08-17T17:09:17.392 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.267261629Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-08-17T17:09:17.392 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.268279039Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-08-17T17:09:17.392 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.269004076Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-08-17T17:09:17.392 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.269885672Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-08-17T17:09:17.392 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.270767295Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-08-17T17:09:17.392 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.271660952Z level=info msg="Executing migration" id="create quota table v1" 2024-08-17T17:09:17.392 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.272545878Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-08-17T17:09:17.392 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.273350147Z level=info msg="Executing migration" id="Update quota table charset" 2024-08-17T17:09:17.392 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.273685455Z level=info msg="Executing migration" id="create plugin_setting table" 2024-08-17T17:09:17.392 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.27409242Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-08-17T17:09:17.393 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.274550427Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-08-17T17:09:17.393 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.275121383Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-08-17T17:09:17.393 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.275428988Z level=info msg="Executing migration" id="create session table" 2024-08-17T17:09:17.393 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.27584983Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-08-17T17:09:17.393 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.276133218Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-08-17T17:09:17.393 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.276448695Z level=info msg="Executing migration" id="create playlist table v2" 2024-08-17T17:09:17.393 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.276833679Z level=info msg="Executing migration" id="create playlist item table v2" 2024-08-17T17:09:17.393 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.277257742Z level=info msg="Executing migration" id="Update playlist table charset" 2024-08-17T17:09:17.393 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.277545849Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-08-17T17:09:17.393 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.277813676Z level=info msg="Executing migration" id="drop preferences table v2" 2024-08-17T17:09:17.394 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.278091856Z level=info msg="Executing migration" id="drop preferences table v3" 2024-08-17T17:09:17.394 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.278403141Z level=info msg="Executing migration" id="create preferences table v3" 2024-08-17T17:09:17.394 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.278796164Z level=info msg="Executing migration" id="Update preferences table charset" 2024-08-17T17:09:17.394 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.279080556Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-08-17T17:09:17.394 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.279721543Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-08-17T17:09:17.394 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.280016838Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-08-17T17:09:17.394 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.280678946Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-08-17T17:09:17.394 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.281271756Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-08-17T17:09:17.394 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.281577306Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-08-17T17:09:17.394 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.282012307Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-08-17T17:09:17.394 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.282552425Z level=info msg="Executing migration" id="create alert table v1" 2024-08-17T17:09:17.394 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.283338031Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-08-17T17:09:17.394 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.284408437Z level=info msg="Executing migration" id="add index alert state" 2024-08-17T17:09:17.395 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.285033635Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-08-17T17:09:17.395 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.285491662Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-08-17T17:09:17.395 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.285879316Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-08-17T17:09:17.395 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.286309989Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-08-17T17:09:17.395 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.286727327Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-08-17T17:09:17.395 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.288125926Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-08-17T17:09:17.395 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.288522401Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-08-17T17:09:17.395 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.288967555Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-08-17T17:09:17.395 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.289304104Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-08-17T17:09:17.395 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.289669083Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-08-17T17:09:17.395 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.290034102Z level=info msg="Executing migration" id="Add column is_default" 2024-08-17T17:09:17.395 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.290713411Z level=info msg="Executing migration" id="Add column frequency" 2024-08-17T17:09:17.395 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.29134444Z level=info msg="Executing migration" id="Add column send_reminder" 2024-08-17T17:09:17.396 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.291993104Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-08-17T17:09:17.396 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.292654426Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-08-17T17:09:17.396 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.29309728Z level=info msg="Executing migration" id="Update alert table charset" 2024-08-17T17:09:17.396 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.293396482Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-08-17T17:09:17.396 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.293687149Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-08-17T17:09:17.396 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.294116448Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-08-17T17:09:17.396 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.294591522Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-08-17T17:09:17.396 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.294954574Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-08-17T17:09:17.396 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.295367272Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-08-17T17:09:17.396 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.295791674Z level=info msg="Executing migration" id="Add for to alert table" 2024-08-17T17:09:17.396 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.296453613Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-08-17T17:09:17.396 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.297095272Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-08-17T17:09:17.396 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.297428275Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-08-17T17:09:17.397 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.297885817Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-08-17T17:09:17.397 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.298897278Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-08-17T17:09:17.397 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.299741147Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-08-17T17:09:17.397 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.300538934Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-08-17T17:09:17.397 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.30099237Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-08-17T17:09:17.397 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.301456443Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-08-17T17:09:17.397 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.302294802Z level=info msg="Executing migration" id="create annotation table v5" 2024-08-17T17:09:17.397 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.303167073Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-08-17T17:09:17.397 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.304137227Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-08-17T17:09:17.397 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.304828671Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-08-17T17:09:17.397 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.305826448Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-08-17T17:09:17.397 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.306609015Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-08-17T17:09:17.397 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.307461868Z level=info msg="Executing migration" id="Update annotation table charset" 2024-08-17T17:09:17.397 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.308298436Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-08-17T17:09:17.398 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.30918795Z level=info msg="Executing migration" id="Drop category_id index" 2024-08-17T17:09:17.398 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.310145817Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-08-17T17:09:17.398 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.310988574Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-08-17T17:09:17.398 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.313731Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-08-17T17:09:17.398 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.316343637Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-08-17T17:09:17.398 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.319296122Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-08-17T17:09:17.398 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.321941629Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-08-17T17:09:17.398 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.324875553Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-08-17T17:09:17.398 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.327542454Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-08-17T17:09:17.398 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.330271898Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-08-17T17:09:17.398 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.33306225Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-08-17T17:09:17.398 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.333414461Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-08-17T17:09:17.398 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.336062805Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-08-17T17:09:17.399 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.336926754Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-08-17T17:09:17.399 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.338719306Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-08-17T17:09:17.399 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.341692499Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-08-17T17:09:17.399 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.342114729Z level=info msg="Executing migration" id="Add epoch_end column" 2024-08-17T17:09:17.399 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.344889242Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-08-17T17:09:17.399 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.347697274Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-08-17T17:09:17.399 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.350475417Z level=info msg="Executing migration" id="Move region to single row" 2024-08-17T17:09:17.399 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.3508787Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-08-17T17:09:17.399 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.353610453Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-08-17T17:09:17.399 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.356302786Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-08-17T17:09:17.399 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.358923312Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-08-17T17:09:17.399 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.359463818Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-08-17T17:09:17.399 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.362109555Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-08-17T17:09:17.399 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.364827966Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-08-17T17:09:17.400 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.367516155Z level=info msg="Executing migration" id="create test_data table" 2024-08-17T17:09:17.400 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.36794098Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-08-17T17:09:17.400 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.370649541Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-08-17T17:09:17.400 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.373288824Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-08-17T17:09:17.400 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.376139049Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-08-17T17:09:17.400 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.376463131Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-08-17T17:09:17.400 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.379181735Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-08-17T17:09:17.400 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.379531309Z level=info msg="Executing migration" id="create team table" 2024-08-17T17:09:17.400 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.381798726Z level=info msg="Executing migration" id="add index team.org_id" 2024-08-17T17:09:17.632 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.384563535Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-08-17T17:09:17.632 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.385022016Z level=info msg="Executing migration" id="create team member table" 2024-08-17T17:09:17.632 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.385852066Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-08-17T17:09:17.632 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.386633142Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-08-17T17:09:17.632 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.387449894Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-08-17T17:09:17.632 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.387905254Z level=info msg="Executing migration" id="Add column email to team table" 2024-08-17T17:09:17.632 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.388708004Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-08-17T17:09:17.632 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.389524165Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-08-17T17:09:17.633 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.390310727Z level=info msg="Executing migration" id="create dashboard acl table" 2024-08-17T17:09:17.633 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.390745326Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-08-17T17:09:17.633 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.391537511Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-08-17T17:09:17.633 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.39229034Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-08-17T17:09:17.633 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.393126575Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-08-17T17:09:17.633 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.393615137Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-08-17T17:09:17.633 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.394514115Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-08-17T17:09:17.633 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.395315284Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-08-17T17:09:17.634 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.396095167Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-08-17T17:09:17.634 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.396935496Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-08-17T17:09:17.634 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.397791851Z level=info msg="Executing migration" id="create tag table" 2024-08-17T17:09:17.635 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.398645671Z level=info msg="Executing migration" id="add index tag.key_value" 2024-08-17T17:09:17.635 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.399523539Z level=info msg="Executing migration" id="create login attempt table" 2024-08-17T17:09:17.635 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.400362041Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-08-17T17:09:17.635 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.401212839Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-08-17T17:09:17.635 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.40223661Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-08-17T17:09:17.635 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.404272734Z level=info msg="Executing migration" id="create login_attempt v2" 2024-08-17T17:09:17.635 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.405164722Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-08-17T17:09:17.636 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.406000883Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-08-17T17:09:17.636 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.406837658Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-08-17T17:09:17.636 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.407777475Z level=info msg="Executing migration" id="create user auth table" 2024-08-17T17:09:17.636 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.408546928Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-08-17T17:09:17.636 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.409553174Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-08-17T17:09:17.636 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.410279648Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-08-17T17:09:17.636 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.411123514Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-08-17T17:09:17.636 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.411912205Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-08-17T17:09:17.636 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.412764532Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-08-17T17:09:17.636 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.413664918Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-08-17T17:09:17.636 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.414541237Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-08-17T17:09:17.636 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.417289983Z level=info msg="Executing migration" id="create server_lock table" 2024-08-17T17:09:17.636 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.420042331Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-08-17T17:09:17.637 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.422825394Z level=info msg="Executing migration" id="create user auth token table" 2024-08-17T17:09:17.637 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.425828634Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-08-17T17:09:17.637 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.428434801Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-08-17T17:09:17.637 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.431236226Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-08-17T17:09:17.637 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.433945769Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-08-17T17:09:17.637 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.436843535Z level=info msg="Executing migration" id="create cache_data table" 2024-08-17T17:09:17.637 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.439567955Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-08-17T17:09:17.637 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.442318324Z level=info msg="Executing migration" id="create short_url table v1" 2024-08-17T17:09:17.637 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.444985188Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-08-17T17:09:17.637 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.447622261Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-08-17T17:09:17.637 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.450481292Z level=info msg="Executing migration" id="delete alert_definition table" 2024-08-17T17:09:17.637 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.453208247Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-08-17T17:09:17.638 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.455873392Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-08-17T17:09:17.638 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.458832704Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-08-17T17:09:17.638 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.461645726Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-08-17T17:09:17.638 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.464191336Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-08-17T17:09:17.638 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.466975294Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-08-17T17:09:17.638 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.469752123Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-08-17T17:09:17.638 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.472467563Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-08-17T17:09:17.638 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.475289678Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-08-17T17:09:17.638 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.477919854Z level=info msg="Executing migration" id="drop alert_definition table" 2024-08-17T17:09:17.638 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.4806404Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-08-17T17:09:17.638 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.483239439Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-08-17T17:09:17.638 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.486150401Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-08-17T17:09:17.639 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.488801185Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-08-17T17:09:17.639 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.491319538Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-08-17T17:09:17.639 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.494254795Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-08-17T17:09:17.639 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.496991615Z level=info msg="Executing migration" id="create alert_instance table" 2024-08-17T17:09:17.639 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.499538557Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-08-17T17:09:17.639 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.502359819Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-08-17T17:09:17.639 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.505109472Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-08-17T17:09:17.639 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.507722689Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-08-17T17:09:17.639 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.51064322Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-08-17T17:09:17.639 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.511336555Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-08-17T17:09:17.639 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.514627974Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-08-17T17:09:17.640 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.518016564Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-08-17T17:09:17.640 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.518967467Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-08-17T17:09:17.640 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.519625393Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-08-17T17:09:17.640 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.520551365Z level=info msg="Executing migration" id="create alert_rule table" 2024-08-17T17:09:17.640 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.52149234Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-08-17T17:09:17.640 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.52215438Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-08-17T17:09:17.640 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.523055666Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-08-17T17:09:17.640 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.523642054Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-08-17T17:09:17.640 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.523914537Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-08-17T17:09:17.640 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.524733733Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-08-17T17:09:17.640 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.525601277Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-08-17T17:09:17.640 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.526455806Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-08-17T17:09:17.640 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.526871485Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-08-17T17:09:17.641 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.527316853Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-08-17T17:09:17.641 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.528141147Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-08-17T17:09:17.641 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.528981949Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-08-17T17:09:17.641 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.529469219Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-08-17T17:09:17.641 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.530303389Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-08-17T17:09:17.641 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.531112136Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-08-17T17:09:17.641 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.531425099Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-08-17T17:09:17.641 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.531863542Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-08-17T17:09:17.641 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.534631954Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-08-17T17:09:17.642 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.535101562Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-08-17T17:09:17.642 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.535433975Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-08-17T17:09:17.642 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.53828347Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-08-17T17:09:17.642 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.540915775Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-08-17T17:09:17.642 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.543766506Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-08-17T17:09:17.642 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.546473092Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-08-17T17:09:17.642 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.549182716Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-08-17T17:09:17.642 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.551872807Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-08-17T17:09:17.642 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.554341836Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-08-17T17:09:17.642 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.55720045Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-08-17T17:09:17.642 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.56016475Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-08-17T17:09:17.643 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.56277966Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-08-17T17:09:17.643 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.565576504Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-08-17T17:09:17.643 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.568250296Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-08-17T17:09:17.643 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.571096405Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-08-17T17:09:17.643 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.573843612Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-08-17T17:09:17.643 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.576511033Z level=info msg="Executing migration" id="create provenance_type table" 2024-08-17T17:09:17.643 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.579281411Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-08-17T17:09:17.643 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.581865902Z level=info msg="Executing migration" id="create alert_image table" 2024-08-17T17:09:17.643 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.582854977Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-08-17T17:09:17.643 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.583578227Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-08-17T17:09:17.643 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.5845919Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-08-17T17:09:17.643 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.585307989Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-08-17T17:09:17.643 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.585744188Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-08-17T17:09:17.644 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.585816186Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-08-17T17:09:17.644 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.586652842Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-08-17T17:09:17.644 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.587233564Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-08-17T17:09:17.644 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.587657276Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-08-17T17:09:17.644 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.587769013Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-08-17T17:09:17.644 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.587873909Z level=info msg="alerts found to migrate" alerts=0 2024-08-17T17:09:17.644 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.58849869Z level=info msg="Executing migration" id="create library_element table v1" 2024-08-17T17:09:17.644 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.589466271Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-08-17T17:09:17.644 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.590172232Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-08-17T17:09:17.644 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.591198677Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-08-17T17:09:17.644 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.591889818Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-08-17T17:09:17.644 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.592836562Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-08-17T17:09:17.644 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.593491164Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-08-17T17:09:17.645 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.594314362Z level=info msg="Executing migration" id="create data_keys table" 2024-08-17T17:09:17.645 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.595197801Z level=info msg="Executing migration" id="create secrets table" 2024-08-17T17:09:17.645 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.596217586Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-08-17T17:09:17.645 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.600587103Z level=info msg="Executing migration" id="add name column into data_keys" 2024-08-17T17:09:17.645 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.601580747Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-08-17T17:09:17.645 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.602484887Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-08-17T17:09:17.645 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.606814242Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-08-17T17:09:17.645 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.61096967Z level=info msg="Executing migration" id="create kv_store table v1" 2024-08-17T17:09:17.645 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.611853088Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-08-17T17:09:17.645 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.612699685Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-08-17T17:09:17.645 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.613651802Z level=info msg="Executing migration" id="create permission table" 2024-08-17T17:09:17.645 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.614579691Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-08-17T17:09:17.645 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.615303639Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-08-17T17:09:17.645 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.616192376Z level=info msg="Executing migration" id="create role table" 2024-08-17T17:09:17.646 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.617159496Z level=info msg="Executing migration" id="add column display_name" 2024-08-17T17:09:17.646 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.618264773Z level=info msg="Executing migration" id="add column group_name" 2024-08-17T17:09:17.646 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.619178498Z level=info msg="Executing migration" id="add index role.org_id" 2024-08-17T17:09:17.646 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.620159467Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-08-17T17:09:17.646 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.621030033Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-08-17T17:09:17.646 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.621782585Z level=info msg="Executing migration" id="create team role table" 2024-08-17T17:09:17.646 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.62277024Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-08-17T17:09:17.646 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.625569378Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-08-17T17:09:17.646 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.628626267Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-08-17T17:09:17.646 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.631464741Z level=info msg="Executing migration" id="create user role table" 2024-08-17T17:09:17.646 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.634366713Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-08-17T17:09:17.876 INFO:teuthology.orchestra.run.smithi136.stdout:163208757271 2024-08-17T17:09:17.876 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-08-17T17:09:18.011 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.63729012Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-08-17T17:09:18.011 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.640158507Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-08-17T17:09:18.012 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.643021853Z level=info msg="Executing migration" id="create builtin role table" 2024-08-17T17:09:18.012 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.645840899Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-08-17T17:09:18.012 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.648695792Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-08-17T17:09:18.012 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.651536892Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-08-17T17:09:18.012 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.654340712Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-08-17T17:09:18.012 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.654868813Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-08-17T17:09:18.012 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.657831542Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-08-17T17:09:18.012 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.65833421Z level=info msg="Executing migration" id="add unique index role.uid" 2024-08-17T17:09:18.012 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.658791929Z level=info msg="Executing migration" id="create seed assignment table" 2024-08-17T17:09:18.012 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.659203707Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-08-17T17:09:18.012 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.659671983Z level=info msg="Executing migration" id="add column hidden to role table" 2024-08-17T17:09:18.012 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.660679025Z level=info msg="Executing migration" id="create query_history table v1" 2024-08-17T17:09:18.013 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.663453038Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-08-17T17:09:18.013 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.663954214Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-08-17T17:09:18.013 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.664252659Z level=info msg="Executing migration" id="teams permissions migration" 2024-08-17T17:09:18.013 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.664839783Z level=info msg="Executing migration" id="dashboard permissions" 2024-08-17T17:09:18.013 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.665332033Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-08-17T17:09:18.013 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.665727691Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-08-17T17:09:18.013 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.666045501Z level=info msg="Executing migration" id="alerting notification permissions" 2024-08-17T17:09:18.013 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.666446487Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-08-17T17:09:18.013 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.666866971Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-08-17T17:09:18.013 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.667342859Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-08-17T17:09:18.013 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.668335495Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-08-17T17:09:18.013 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.668826314Z level=info msg="Executing migration" id="create correlation table v1" 2024-08-17T17:09:18.013 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.66953515Z level=info msg="Executing migration" id="add index correlations.uid" 2024-08-17T17:09:18.013 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.670177798Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-08-17T17:09:18.014 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.671002507Z level=info msg="Executing migration" id="add correlation config column" 2024-08-17T17:09:18.014 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.672270697Z level=info msg="Executing migration" id="create entity_events table" 2024-08-17T17:09:18.014 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.672702495Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-08-17T17:09:18.014 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.675608578Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-08-17T17:09:18.014 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.675877401Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-08-17T17:09:18.014 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.676216368Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-08-17T17:09:18.014 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.676309345Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-08-17T17:09:18.014 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.676597663Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-08-17T17:09:18.014 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.679409583Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-08-17T17:09:18.015 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.679870314Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-08-17T17:09:18.015 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.682240132Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-08-17T17:09:18.015 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.685187381Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-08-17T17:09:18.015 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.687943408Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-08-17T17:09:18.015 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.690724327Z level=info msg="Executing migration" id="Drop public config table" 2024-08-17T17:09:18.015 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.693341876Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-08-17T17:09:18.015 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.696283751Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-08-17T17:09:18.015 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.698794211Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-08-17T17:09:18.015 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.701490643Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-08-17T17:09:18.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.70417236Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-08-17T17:09:18.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.707283827Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-08-17T17:09:18.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.710160583Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-08-17T17:09:18.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.712861553Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-08-17T17:09:18.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.715795101Z level=info msg="Executing migration" id="add share column" 2024-08-17T17:09:18.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.718456331Z level=info msg="Executing migration" id="create default alerting folders" 2024-08-17T17:09:18.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.721110526Z level=info msg="Executing migration" id="create file table" 2024-08-17T17:09:18.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.723906908Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-08-17T17:09:18.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.726480631Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-08-17T17:09:18.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.729156299Z level=info msg="Executing migration" id="create file_meta table" 2024-08-17T17:09:18.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.729598967Z level=info msg="Executing migration" id="file table idx: path key" 2024-08-17T17:09:18.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.732316106Z level=info msg="Executing migration" id="set path collation in file table" 2024-08-17T17:09:18.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.733224195Z level=info msg="Executing migration" id="managed permissions migration" 2024-08-17T17:09:18.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.734048683Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-08-17T17:09:18.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.734926012Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-08-17T17:09:18.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.735836186Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-08-17T17:09:18.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.737025823Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-08-17T17:09:18.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.737376937Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-08-17T17:09:18.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.738187458Z level=info msg="Executing migration" id="update group index for alert rules" 2024-08-17T17:09:18.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.738970067Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-08-17T17:09:18.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.739900844Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-08-17T17:09:18.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.740572434Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-08-17T17:09:18.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.741651002Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-08-17T17:09:18.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.742703709Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-08-17T17:09:18.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.743468566Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-08-17T17:09:18.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.753171648Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-08-17T17:09:18.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.753926266Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-08-17T17:09:18.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.754734817Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-08-17T17:09:18.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.758151263Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-08-17T17:09:18.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.758943457Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-08-17T17:09:18.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.759802687Z level=info msg="Executing migration" id="create folder table" 2024-08-17T17:09:18.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.760781602Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-08-17T17:09:18.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.761450686Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-08-17T17:09:18.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.762288378Z level=info msg="Executing migration" id="Update folder title length" 2024-08-17T17:09:18.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.763178041Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-08-17T17:09:18.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=migrator t=2024-08-17T17:09:17.764139236Z level=info msg="migrations completed" performed=485 skipped=0 duration=711.533864ms 2024-08-17T17:09:18.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=sqlstore t=2024-08-17T17:09:17.76458909Z level=info msg="Created default organization" 2024-08-17T17:09:18.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=plugin.loader t=2024-08-17T17:09:17.819810981Z level=info msg="Plugin registered" pluginID=input 2024-08-17T17:09:18.019 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=plugin.loader t=2024-08-17T17:09:17.847028148Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-08-17T17:09:18.019 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=plugin.loader t=2024-08-17T17:09:17.847046418Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-08-17T17:09:18.019 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=secrets t=2024-08-17T17:09:17.84708734Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-08-17T17:09:18.019 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=query_data t=2024-08-17T17:09:17.848855909Z level=info msg="Query Service initialization" 2024-08-17T17:09:18.019 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:17 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=live.push_http t=2024-08-17T17:09:17.851392043Z level=info msg="Live Push Gateway initialization" 2024-08-17T17:09:18.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:18 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:18.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:18 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:18.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:18 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:18.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:18 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:18.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:18 smithi136 ceph-mon[42580]: Deploying daemon node-exporter.a on smithi136 2024-08-17T17:09:18.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:18 smithi136 ceph-mon[42580]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:18.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:18 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:09:18.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:18 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:18.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:18 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:18.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:18 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:18.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:18 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:18.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:18 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:18.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:18 smithi136 ceph-mon[31034]: Deploying daemon node-exporter.a on smithi136 2024-08-17T17:09:18.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:18 smithi136 ceph-mon[31034]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:18.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:18 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:09:18.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:18 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:18.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:18 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:18.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:18 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:18.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:18 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:18.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:18 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:18.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:18 smithi165 ceph-mon[34868]: Deploying daemon node-exporter.a on smithi136 2024-08-17T17:09:18.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:18 smithi165 ceph-mon[34868]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:18.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:18 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:09:18.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:18 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:19.011 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:18 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=infra.usagestats.collector t=2024-08-17T17:09:18.585082639Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-08-17T17:09:19.011 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:18 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=provisioning.datasources t=2024-08-17T17:09:18.640403017Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-08-17T17:09:19.011 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:18 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=provisioning.datasources t=2024-08-17T17:09:18.672133689Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-08-17T17:09:19.012 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:18 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=provisioning.alerting t=2024-08-17T17:09:18.694060973Z level=info msg="starting to provision alerting" 2024-08-17T17:09:19.012 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:18 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=provisioning.alerting t=2024-08-17T17:09:18.69410782Z level=info msg="finished to provision alerting" 2024-08-17T17:09:19.012 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:18 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=grafanaStorageLogger t=2024-08-17T17:09:18.69423536Z level=info msg="storage starting" 2024-08-17T17:09:19.012 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:18 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=ngalert.state.manager t=2024-08-17T17:09:18.694372459Z level=info msg="Warming state cache for startup" 2024-08-17T17:09:19.012 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:18 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=http.server t=2024-08-17T17:09:18.695890966Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-08-17T17:09:19.012 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:18 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=ngalert.state.manager t=2024-08-17T17:09:18.704618571Z level=info msg="State cache has been initialized" states=0 duration=10.242727ms 2024-08-17T17:09:19.012 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:18 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=ngalert.multiorg.alertmanager t=2024-08-17T17:09:18.704732426Z level=info msg="starting MultiOrg Alertmanager" 2024-08-17T17:09:19.012 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:18 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=ticker t=2024-08-17T17:09:18.704744377Z level=info msg=starting first_tick=2024-08-17T17:09:20Z 2024-08-17T17:09:19.012 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:18 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=sqlstore.transactions t=2024-08-17T17:09:18.704742014Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-08-17T17:09:19.012 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:18 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=sqlstore.transactions t=2024-08-17T17:09:18.750246363Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-08-17T17:09:19.012 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:09:18 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[65805]: logger=sqlstore.transactions t=2024-08-17T17:09:18.760743361Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-08-17T17:09:20.429 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:20 smithi136 systemd[1]: Starting Ceph node-exporter.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:09:21.004 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:20 smithi136 ceph-mon[31034]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:21.004 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:20 smithi136 ceph-mon[42580]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:20 smithi165 ceph-mon[34868]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:22.281 INFO:teuthology.orchestra.run.smithi136.stdout:55834574891 2024-08-17T17:09:22.281 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-08-17T17:09:23.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:22 smithi136 ceph-mon[31034]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:23.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:22 smithi136 ceph-mon[42580]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:22 smithi165 ceph-mon[34868]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:23.387 INFO:teuthology.orchestra.run.smithi136.stdout:107374182434 2024-08-17T17:09:23.388 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-08-17T17:09:23.976 INFO:teuthology.orchestra.run.smithi136.stdout:231928233998 2024-08-17T17:09:23.977 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-08-17T17:09:25.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:24 smithi136 ceph-mon[31034]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:25.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:24 smithi136 ceph-mon[42580]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:25.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:24 smithi165 ceph-mon[34868]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:25.717 INFO:teuthology.orchestra.run.smithi136.stdout:34359738416 2024-08-17T17:09:25.718 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-08-17T17:09:26.225 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:09:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:09:26] "GET /metrics HTTP/1.1" 200 28518 "" "Prometheus/2.43.0" 2024-08-17T17:09:27.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:26 smithi136 ceph-mon[31034]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:27.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:26 smithi136 ceph-mon[42580]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:27.197 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:26 smithi136 bash[75951]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-08-17T17:09:27.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:26 smithi165 ceph-mon[34868]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:27.446 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:27 smithi136 bash[75951]: Getting image source signatures 2024-08-17T17:09:27.446 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:27 smithi136 bash[75951]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-08-17T17:09:27.447 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:27 smithi136 bash[75951]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-08-17T17:09:27.447 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:27 smithi136 bash[75951]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-08-17T17:09:27.798 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:27.798 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:27.881 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:28.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:27 smithi136 ceph-mon[31034]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:28.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:27 smithi136 ceph-mon[42580]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:28.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:27 smithi165 ceph-mon[34868]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:28.808 INFO:teuthology.orchestra.run.smithi136.stdout:77309411368 2024-08-17T17:09:28.809 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-08-17T17:09:29.696 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:29 smithi136 bash[75951]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-08-17T17:09:30.196 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:29 smithi136 bash[75951]: Writing manifest to image destination 2024-08-17T17:09:30.379 INFO:teuthology.orchestra.run.smithi136.stdout:128849018910 2024-08-17T17:09:30.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-08-17T17:09:30.467 INFO:teuthology.orchestra.run.smithi136.stdout:197568495637 2024-08-17T17:09:30.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-08-17T17:09:30.572 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 podman[75951]: 2024-08-17 17:09:30.423731989 +0000 UTC m=+4.094503890 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-08-17T17:09:30.572 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 podman[75951]: 2024-08-17 17:09:30.572014833 +0000 UTC m=+4.242786728 container create 6aef82a780f5bed31be412ba7b88ff8f2f8976f47f2f777be22a73b5cd6855fb (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-08-17T17:09:31.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-mon[31034]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:31.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-mon[42580]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:31.076 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 podman[75951]: 2024-08-17 17:09:30.883275369 +0000 UTC m=+4.554047254 container init 6aef82a780f5bed31be412ba7b88ff8f2f8976f47f2f777be22a73b5cd6855fb (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-08-17T17:09:31.076 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 podman[75951]: 2024-08-17 17:09:30.888527677 +0000 UTC m=+4.559299563 container start 6aef82a780f5bed31be412ba7b88ff8f2f8976f47f2f777be22a73b5cd6855fb (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-08-17T17:09:31.077 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.892Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-08-17T17:09:31.077 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.892Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-08-17T17:09:31.077 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.892Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-08-17T17:09:31.077 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.892Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-08-17T17:09:31.077 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-08-17T17:09:31.077 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-08-17T17:09:31.077 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-08-17T17:09:31.077 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=arp 2024-08-17T17:09:31.077 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=bcache 2024-08-17T17:09:31.077 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=bonding 2024-08-17T17:09:31.077 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=btrfs 2024-08-17T17:09:31.077 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=conntrack 2024-08-17T17:09:31.078 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=cpu 2024-08-17T17:09:31.078 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-08-17T17:09:31.078 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=diskstats 2024-08-17T17:09:31.078 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=dmi 2024-08-17T17:09:31.078 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=edac 2024-08-17T17:09:31.078 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=entropy 2024-08-17T17:09:31.078 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-08-17T17:09:31.078 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=filefd 2024-08-17T17:09:31.078 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=filesystem 2024-08-17T17:09:31.078 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=hwmon 2024-08-17T17:09:31.078 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=infiniband 2024-08-17T17:09:31.078 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=ipvs 2024-08-17T17:09:31.079 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=loadavg 2024-08-17T17:09:31.079 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=mdadm 2024-08-17T17:09:31.079 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=meminfo 2024-08-17T17:09:31.079 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=netclass 2024-08-17T17:09:31.079 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=netdev 2024-08-17T17:09:31.079 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=netstat 2024-08-17T17:09:31.079 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=nfs 2024-08-17T17:09:31.079 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=nfsd 2024-08-17T17:09:31.079 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=nvme 2024-08-17T17:09:31.079 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=os 2024-08-17T17:09:31.079 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-08-17T17:09:31.079 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=pressure 2024-08-17T17:09:31.079 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=rapl 2024-08-17T17:09:31.080 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=schedstat 2024-08-17T17:09:31.080 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=selinux 2024-08-17T17:09:31.080 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=sockstat 2024-08-17T17:09:31.080 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=softnet 2024-08-17T17:09:31.080 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=stat 2024-08-17T17:09:31.080 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=tapestats 2024-08-17T17:09:31.080 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=textfile 2024-08-17T17:09:31.080 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-08-17T17:09:31.081 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=time 2024-08-17T17:09:31.081 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-08-17T17:09:31.081 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=uname 2024-08-17T17:09:31.081 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=vmstat 2024-08-17T17:09:31.081 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=xfs 2024-08-17T17:09:31.081 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=node_exporter.go:117 level=info collector=zfs 2024-08-17T17:09:31.081 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-08-17T17:09:31.081 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[76386]: ts=2024-08-17T17:09:30.893Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-08-17T17:09:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:30 smithi165 ceph-mon[34868]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:31.696 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:31 smithi136 bash[75951]: 6aef82a780f5bed31be412ba7b88ff8f2f8976f47f2f777be22a73b5cd6855fb 2024-08-17T17:09:31.696 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:09:31 smithi136 systemd[1]: Started Ceph node-exporter.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:09:31.717 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:31.718 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:32.479 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:32 smithi165 bash[66081]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-08-17T17:09:32.740 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:32 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:32.740 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:32 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:32.740 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:32 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:32.740 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:32 smithi165 ceph-mon[34868]: Deploying daemon node-exporter.b on smithi165 2024-08-17T17:09:32.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:32 smithi165 ceph-mon[34868]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:32.775 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:32 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:32.775 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:32 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:32.775 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:32 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:32.775 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:32 smithi136 ceph-mon[31034]: Deploying daemon node-exporter.b on smithi165 2024-08-17T17:09:32.775 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:32 smithi136 ceph-mon[31034]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:32.776 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:32 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:32.776 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:32 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:32.776 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:32 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:32.776 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:32 smithi136 ceph-mon[42580]: Deploying daemon node-exporter.b on smithi165 2024-08-17T17:09:32.776 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:32 smithi136 ceph-mon[42580]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:33.011 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:32 smithi165 bash[66081]: Getting image source signatures 2024-08-17T17:09:33.011 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:32 smithi165 bash[66081]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-08-17T17:09:33.011 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:32 smithi165 bash[66081]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-08-17T17:09:33.011 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:32 smithi165 bash[66081]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-08-17T17:09:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:33 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:09:33.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:33 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:09:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:33 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:09:34.493 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 bash[66081]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-08-17T17:09:34.493 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 bash[66081]: Writing manifest to image destination 2024-08-17T17:09:34.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:34 smithi136 ceph-mon[31034]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:34.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:34 smithi136 ceph-mon[42580]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:34.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-mon[34868]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:34.761 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 podman[66081]: 2024-08-17 17:09:34.487700621 +0000 UTC m=+2.197595901 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-08-17T17:09:34.761 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 podman[66081]: 2024-08-17 17:09:34.594557209 +0000 UTC m=+2.304452487 container create 60b065dd0c2ad48a4f1a606481de4defb842a305c5a06f88623ea21a137f012e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-08-17T17:09:35.261 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 podman[66081]: 2024-08-17 17:09:34.86379052 +0000 UTC m=+2.573685800 container init 60b065dd0c2ad48a4f1a606481de4defb842a305c5a06f88623ea21a137f012e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-08-17T17:09:35.261 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 podman[66081]: 2024-08-17 17:09:34.868786747 +0000 UTC m=+2.578682019 container start 60b065dd0c2ad48a4f1a606481de4defb842a305c5a06f88623ea21a137f012e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-08-17T17:09:35.262 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.872Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-08-17T17:09:35.262 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.872Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-08-17T17:09:35.262 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.873Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-08-17T17:09:35.262 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.873Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-08-17T17:09:35.262 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.873Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-08-17T17:09:35.262 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.873Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-08-17T17:09:35.262 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.873Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-08-17T17:09:35.262 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.873Z caller=node_exporter.go:117 level=info collector=arp 2024-08-17T17:09:35.262 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.873Z caller=node_exporter.go:117 level=info collector=bcache 2024-08-17T17:09:35.262 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.873Z caller=node_exporter.go:117 level=info collector=bonding 2024-08-17T17:09:35.262 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.873Z caller=node_exporter.go:117 level=info collector=btrfs 2024-08-17T17:09:35.263 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.873Z caller=node_exporter.go:117 level=info collector=conntrack 2024-08-17T17:09:35.263 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.873Z caller=node_exporter.go:117 level=info collector=cpu 2024-08-17T17:09:35.263 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-08-17T17:09:35.263 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=diskstats 2024-08-17T17:09:35.263 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=dmi 2024-08-17T17:09:35.263 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=edac 2024-08-17T17:09:35.263 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=entropy 2024-08-17T17:09:35.264 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-08-17T17:09:35.264 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=filefd 2024-08-17T17:09:35.264 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=filesystem 2024-08-17T17:09:35.264 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=hwmon 2024-08-17T17:09:35.264 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=infiniband 2024-08-17T17:09:35.264 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=ipvs 2024-08-17T17:09:35.264 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=loadavg 2024-08-17T17:09:35.264 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=mdadm 2024-08-17T17:09:35.264 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=meminfo 2024-08-17T17:09:35.264 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=netclass 2024-08-17T17:09:35.264 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=netdev 2024-08-17T17:09:35.264 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=netstat 2024-08-17T17:09:35.264 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=nfs 2024-08-17T17:09:35.265 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=nfsd 2024-08-17T17:09:35.265 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=nvme 2024-08-17T17:09:35.265 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=os 2024-08-17T17:09:35.265 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-08-17T17:09:35.265 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=pressure 2024-08-17T17:09:35.265 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=rapl 2024-08-17T17:09:35.265 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=schedstat 2024-08-17T17:09:35.265 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=selinux 2024-08-17T17:09:35.265 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=sockstat 2024-08-17T17:09:35.265 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=softnet 2024-08-17T17:09:35.265 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=stat 2024-08-17T17:09:35.265 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=tapestats 2024-08-17T17:09:35.265 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=textfile 2024-08-17T17:09:35.266 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-08-17T17:09:35.266 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=time 2024-08-17T17:09:35.266 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-08-17T17:09:35.266 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=uname 2024-08-17T17:09:35.266 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=vmstat 2024-08-17T17:09:35.266 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=xfs 2024-08-17T17:09:35.266 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=node_exporter.go:117 level=info collector=zfs 2024-08-17T17:09:35.266 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-08-17T17:09:35.266 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[66132]: ts=2024-08-17T17:09:34.874Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-08-17T17:09:35.266 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 bash[66081]: 60b065dd0c2ad48a4f1a606481de4defb842a305c5a06f88623ea21a137f012e 2024-08-17T17:09:35.266 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:09:34 smithi165 systemd[1]: Started Ceph node-exporter.b for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:09:35.454 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:36.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:36 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:36.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:36 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:36.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:36 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:36.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:36 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:36.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:36 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:36.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:36 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:09:36.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:36 smithi136 ceph-mon[42580]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:36.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:36 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:36.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:36 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:36.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:36 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:36.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:36 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:36.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:36 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:36.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:36 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:09:36.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:36 smithi136 ceph-mon[31034]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:36.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:36 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:36.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:36 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:36.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:36 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:36.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:36 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:36.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:36 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:36.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:36 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:09:36.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:36 smithi165 ceph-mon[34868]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:36.569 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:09:36 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:09:36] "GET /metrics HTTP/1.1" 200 28515 "" "Prometheus/2.43.0" 2024-08-17T17:09:36.633 INFO:teuthology.orchestra.run.smithi136.stdout:231928234001 2024-08-17T17:09:37.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:37 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/3212416433' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-08-17T17:09:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:37 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/3391214412' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-08-17T17:09:37.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:37 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/3212416433' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-08-17T17:09:37.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:37 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/3391214412' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-08-17T17:09:37.339 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:37 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/3212416433' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-08-17T17:09:37.339 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:37 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/3391214412' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-08-17T17:09:37.424 INFO:teuthology.orchestra.run.smithi136.stdout:163208757275 2024-08-17T17:09:37.631 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:38.095 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757271 got 163208757275 for osd.5 2024-08-17T17:09:38.096 DEBUG:teuthology.parallel:result is None 2024-08-17T17:09:38.678 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233998 got 231928234001 for osd.7 2024-08-17T17:09:38.678 DEBUG:teuthology.parallel:result is None 2024-08-17T17:09:39.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:38 smithi136 ceph-mon[31034]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:39.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:38 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:39.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:38 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:39.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:38 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:39.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:38 smithi136 ceph-mon[42580]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:39.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:38 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:39.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:38 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:39.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:38 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:38 smithi165 ceph-mon[34868]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:38 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:38 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:38 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:39.578 INFO:teuthology.orchestra.run.smithi136.stdout:107374182437 2024-08-17T17:09:40.053 INFO:teuthology.orchestra.run.smithi136.stdout:34359738419 2024-08-17T17:09:40.053 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:39 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/2604577607' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-08-17T17:09:40.053 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:39 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/986757969' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-08-17T17:09:40.053 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:39 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/2604577607' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-08-17T17:09:40.054 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:39 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/986757969' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-08-17T17:09:40.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:39 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2604577607' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-08-17T17:09:40.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:39 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/986757969' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-08-17T17:09:40.867 INFO:teuthology.orchestra.run.smithi136.stdout:55834574895 2024-08-17T17:09:41.151 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:40 smithi136 ceph-mon[31034]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:41.151 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:40 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/1268899383' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-08-17T17:09:41.151 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:40 smithi136 ceph-mon[42580]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:41.151 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:40 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/1268899383' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-08-17T17:09:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:40 smithi165 ceph-mon[34868]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:40 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/1268899383' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-08-17T17:09:41.475 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182434 got 107374182437 for osd.3 2024-08-17T17:09:41.475 DEBUG:teuthology.parallel:result is None 2024-08-17T17:09:41.516 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:42.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:41 smithi136 ceph-mon[31034]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:42.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:41 smithi136 ceph-mon[42580]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:41 smithi165 ceph-mon[34868]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:42.764 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574891 got 55834574895 for osd.1 2024-08-17T17:09:42.764 DEBUG:teuthology.parallel:result is None 2024-08-17T17:09:44.175 INFO:teuthology.orchestra.run.smithi136.stdout:77309411371 2024-08-17T17:09:44.674 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738416 got 34359738419 for osd.0 2024-08-17T17:09:44.674 DEBUG:teuthology.parallel:result is None 2024-08-17T17:09:45.175 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:44 smithi136 ceph-mon[31034]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:45.175 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:44 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/3507451636' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-08-17T17:09:45.175 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:44 smithi136 ceph-mon[42580]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:45.175 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:44 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/3507451636' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-08-17T17:09:45.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:44 smithi165 ceph-mon[34868]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:45.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:44 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/3507451636' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-08-17T17:09:45.791 INFO:teuthology.orchestra.run.smithi136.stdout:128849018913 2024-08-17T17:09:46.034 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:45 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/1300766822' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-08-17T17:09:46.035 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:45 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/1300766822' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-08-17T17:09:46.057 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411368 got 77309411371 for osd.2 2024-08-17T17:09:46.057 DEBUG:teuthology.parallel:result is None 2024-08-17T17:09:46.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:45 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/1300766822' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-08-17T17:09:46.446 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:09:46 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:09:46] "GET /metrics HTTP/1.1" 200 28515 "" "Prometheus/2.43.0" 2024-08-17T17:09:47.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:46 smithi136 ceph-mon[31034]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:47.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:46 smithi136 ceph-mon[42580]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:46 smithi165 ceph-mon[34868]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:47.533 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018910 got 128849018913 for osd.4 2024-08-17T17:09:47.533 DEBUG:teuthology.parallel:result is None 2024-08-17T17:09:48.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:47 smithi136 ceph-mon[42580]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:48.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:47 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:09:48.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:47 smithi136 ceph-mon[31034]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:48.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:47 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:09:48.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:47 smithi165 ceph-mon[34868]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:48.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:47 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:09:48.274 INFO:teuthology.orchestra.run.smithi136.stdout:197568495641 2024-08-17T17:09:48.998 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495637 got 197568495641 for osd.6 2024-08-17T17:09:48.998 DEBUG:teuthology.parallel:result is None 2024-08-17T17:09:48.999 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-08-17T17:09:48.999 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:09:49.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:48 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/565165638' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-08-17T17:09:49.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:48 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/565165638' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-08-17T17:09:49.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:48 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/565165638' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-08-17T17:09:49.628 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:50.051 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:49 smithi136 ceph-mon[31034]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:50.052 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:49 smithi136 ceph-mon[42580]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:50.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:49 smithi165 ceph-mon[34868]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:52.178 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:09:52.179 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T17:09:52.695 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:52 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:52.695 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:52 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:52.695 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:52 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:09:52.695 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:52 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:09:52.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:52 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:52.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:52 smithi136 ceph-mon[31034]: Reconfiguring alertmanager.a (dependencies changed)... 2024-08-17T17:09:52.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:52 smithi136 ceph-mon[31034]: Reconfiguring daemon alertmanager.a on smithi136 2024-08-17T17:09:52.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:52 smithi136 ceph-mon[31034]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:52.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:52 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:52.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:52 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:52.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:52 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:09:52.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:52 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:09:52.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:52 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:52.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:52 smithi136 ceph-mon[42580]: Reconfiguring alertmanager.a (dependencies changed)... 2024-08-17T17:09:52.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:52 smithi136 ceph-mon[42580]: Reconfiguring daemon alertmanager.a on smithi136 2024-08-17T17:09:52.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:52 smithi136 ceph-mon[42580]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:52 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:52 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:52 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:09:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:52 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:09:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:52 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:09:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:52 smithi165 ceph-mon[34868]: Reconfiguring alertmanager.a (dependencies changed)... 2024-08-17T17:09:53.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:52 smithi165 ceph-mon[34868]: Reconfiguring daemon alertmanager.a on smithi136 2024-08-17T17:09:53.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:52 smithi165 ceph-mon[34868]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:53.201 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-08-17T17:09:51.869906+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":181,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131,"ondisk_log_size":131,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":223048,"kb_used_data":6888,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":749738168,"statfs":{"total":767960285184,"available":767731884032,"internally_reserved":0,"allocated":7053312,"data_stored":3988771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002219"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-08-17T17:09:02.570337+0000","last_change":"2024-08-17T17:08:53.945206+0000","last_active":"2024-08-17T17:09:02.570337+0000","last_peered":"2024-08-17T17:09:02.570337+0000","last_clean":"2024-08-17T17:09:02.570337+0000","last_became_active":"2024-08-17T17:08:48.853366+0000","last_became_peered":"2024-08-17T17:08:48.853366+0000","last_unstale":"2024-08-17T17:09:02.570337+0000","last_undegraded":"2024-08-17T17:09:02.570337+0000","last_fullsized":"2024-08-17T17:09:02.570337+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:34:32.938937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-08-17T17:09:02.326831+0000","last_change":"2024-08-17T17:08:53.939970+0000","last_active":"2024-08-17T17:09:02.326831+0000","last_peered":"2024-08-17T17:09:02.326831+0000","last_clean":"2024-08-17T17:09:02.326831+0000","last_became_active":"2024-08-17T17:08:48.852650+0000","last_became_peered":"2024-08-17T17:08:48.852650+0000","last_unstale":"2024-08-17T17:09:02.326831+0000","last_undegraded":"2024-08-17T17:09:02.326831+0000","last_fullsized":"2024-08-17T17:09:02.326831+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:40.181438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000256001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-08-17T17:09:02.626084+0000","last_change":"2024-08-17T17:08:54.068835+0000","last_active":"2024-08-17T17:09:02.626084+0000","last_peered":"2024-08-17T17:09:02.626084+0000","last_clean":"2024-08-17T17:09:02.626084+0000","last_became_active":"2024-08-17T17:08:48.853784+0000","last_became_peered":"2024-08-17T17:08:48.853784+0000","last_unstale":"2024-08-17T17:09:02.626084+0000","last_undegraded":"2024-08-17T17:09:02.626084+0000","last_fullsized":"2024-08-17T17:09:02.626084+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:43:23.734272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-08-17T17:09:02.326806+0000","last_change":"2024-08-17T17:08:53.939956+0000","last_active":"2024-08-17T17:09:02.326806+0000","last_peered":"2024-08-17T17:09:02.326806+0000","last_clean":"2024-08-17T17:09:02.326806+0000","last_became_active":"2024-08-17T17:08:48.851812+0000","last_became_peered":"2024-08-17T17:08:48.851812+0000","last_unstale":"2024-08-17T17:09:02.326806+0000","last_undegraded":"2024-08-17T17:09:02.326806+0000","last_fullsized":"2024-08-17T17:09:02.326806+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:25:20.326625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":22,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-08-17T17:09:02.836693+0000","last_change":"2024-08-17T17:08:53.944227+0000","last_active":"2024-08-17T17:09:02.836693+0000","last_peered":"2024-08-17T17:09:02.836693+0000","last_clean":"2024-08-17T17:09:02.836693+0000","last_became_active":"2024-08-17T17:08:48.854463+0000","last_became_peered":"2024-08-17T17:08:48.854463+0000","last_unstale":"2024-08-17T17:09:02.836693+0000","last_undegraded":"2024-08-17T17:09:02.836693+0000","last_fullsized":"2024-08-17T17:09:02.836693+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:06:41.281670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001023532,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-08-17T17:09:02.836401+0000","last_change":"2024-08-17T17:08:53.939867+0000","last_active":"2024-08-17T17:09:02.836401+0000","last_peered":"2024-08-17T17:09:02.836401+0000","last_clean":"2024-08-17T17:09:02.836401+0000","last_became_active":"2024-08-17T17:08:48.927330+0000","last_became_peered":"2024-08-17T17:08:48.927330+0000","last_unstale":"2024-08-17T17:09:02.836401+0000","last_undegraded":"2024-08-17T17:09:02.836401+0000","last_fullsized":"2024-08-17T17:09:02.836401+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:03:20.888081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-08-17T17:09:02.626116+0000","last_change":"2024-08-17T17:08:54.068863+0000","last_active":"2024-08-17T17:09:02.626116+0000","last_peered":"2024-08-17T17:09:02.626116+0000","last_clean":"2024-08-17T17:09:02.626116+0000","last_became_active":"2024-08-17T17:08:48.854469+0000","last_became_peered":"2024-08-17T17:08:48.854469+0000","last_unstale":"2024-08-17T17:09:02.626116+0000","last_undegraded":"2024-08-17T17:09:02.626116+0000","last_fullsized":"2024-08-17T17:09:02.626116+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:32:44.509162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":21,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-08-17T17:09:02.836876+0000","last_change":"2024-08-17T17:08:53.943617+0000","last_active":"2024-08-17T17:09:02.836876+0000","last_peered":"2024-08-17T17:09:02.836876+0000","last_clean":"2024-08-17T17:09:02.836876+0000","last_became_active":"2024-08-17T17:08:48.854004+0000","last_became_peered":"2024-08-17T17:08:48.854004+0000","last_unstale":"2024-08-17T17:09:02.836876+0000","last_undegraded":"2024-08-17T17:09:02.836876+0000","last_fullsized":"2024-08-17T17:09:02.836876+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:53:56.669365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041737399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"61'128","reported_seq":260,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-08-17T17:08:54.068545+0000","last_change":"2024-08-17T17:08:20.329502+0000","last_active":"2024-08-17T17:08:54.068545+0000","last_peered":"2024-08-17T17:08:54.068545+0000","last_clean":"2024-08-17T17:08:54.068545+0000","last_became_active":"2024-08-17T17:08:20.329063+0000","last_became_peered":"2024-08-17T17:08:20.329063+0000","last_unstale":"2024-08-17T17:08:54.068545+0000","last_undegraded":"2024-08-17T17:08:54.068545+0000","last_fullsized":"2024-08-17T17:08:54.068545+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T17:06:21.174471+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:52:20.712819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234004,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28240,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716912,"statfs":{"total":95995035648,"available":95966117888,"internally_reserved":0,"allocated":1249280,"data_stored":867435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:09:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.51700000000000002}]},{"osd":1,"last update":"Sat Aug 17 17:09:18 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.505},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.55200000000000005}]},{"osd":2,"last update":"Sat Aug 17 17:09:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.60499999999999998}]},{"osd":3,"last update":"Sat Aug 17 17:09:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.56399999999999995}]},{"osd":4,"last update":"Sat Aug 17 17:09:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.438},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.53900000000000003}]},{"osd":5,"last update":"Sat Aug 17 17:09:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.59599999999999997}]},{"osd":6,"last update":"Sat Aug 17 17:09:18 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.58499999999999996}]}]},{"osd":6,"up_from":46,"seq":197568495642,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28244,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716908,"statfs":{"total":95995035648,"available":95966113792,"internally_reserved":0,"allocated":1253376,"data_stored":867454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.56899999999999995}]},{"osd":1,"last update":"Sat Aug 17 17:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.60999999999999999}]},{"osd":2,"last update":"Sat Aug 17 17:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.42599999999999999}]},{"osd":3,"last update":"Sat Aug 17 17:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.69199999999999995}]},{"osd":4,"last update":"Sat Aug 17 17:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.59599999999999997}]},{"osd":5,"last update":"Sat Aug 17 17:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.55100000000000005}]},{"osd":7,"last update":"Sat Aug 17 17:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.63200000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757278,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":659456,"data_stored":277036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:09:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53200000000000003,"15min":0.52800000000000002},"min":{"1min":0.39500000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.52000000000000002,"15min":0.52200000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.60699999999999998,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.62}]},{"osd":1,"last update":"Sat Aug 17 17:09:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.63200000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.497,"15min":0.495},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.71799999999999997}]},{"osd":2,"last update":"Sat Aug 17 17:09:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55400000000000005,"15min":0.55300000000000005},"min":{"1min":0.45400000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53300000000000003,"15min":0.53500000000000003},"min":{"1min":0.38500000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.65700000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.68000000000000005}]},{"osd":3,"last update":"Sat Aug 17 17:09:31 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51400000000000001,"15min":0.51700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.63300000000000001,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55400000000000005,"15min":0.55200000000000005},"min":{"1min":0.437,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.73699999999999999}]},{"osd":4,"last update":"Sat Aug 17 17:09:31 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.29799999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.501,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.63},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.42799999999999999,"15min":0.435},"min":{"1min":0.28599999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.57499999999999996,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.50900000000000001}]},{"osd":6,"last update":"Sat Aug 17 17:08:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.60099999999999998}]},{"osd":7,"last update":"Sat Aug 17 17:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.53700000000000003}]}]},{"osd":4,"up_from":30,"seq":128849018915,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":659456,"data_stored":277348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:09:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.50900000000000001,"15min":0.50800000000000001},"min":{"1min":0.35799999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53700000000000003,"15min":0.53800000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.76300000000000001,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.52800000000000002}]},{"osd":1,"last update":"Sat Aug 17 17:09:07 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.502,"15min":0.502},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.496},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73699999999999999,"5min":0.751,"15min":0.751},"last":0.57899999999999996}]},{"osd":2,"last update":"Sat Aug 17 17:09:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.76600000000000001,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.504},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51900000000000002,"15min":0.52000000000000002},"min":{"1min":0.36299999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.66800000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.55600000000000005}]},{"osd":3,"last update":"Sat Aug 17 17:09:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54700000000000004,"15min":0.55000000000000004},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52700000000000002,"15min":0.52900000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.64400000000000002,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.51900000000000002}]},{"osd":5,"last update":"Sat Aug 17 17:09:34 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.374,"15min":0.36599999999999999},"min":{"1min":0.312,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.39800000000000002,"15min":0.38400000000000001},"min":{"1min":0.29899999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.47799999999999998}]},{"osd":6,"last update":"Sat Aug 17 17:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.61399999999999999}]},{"osd":7,"last update":"Sat Aug 17 17:09:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.443}]}]},{"osd":3,"up_from":25,"seq":107374182440,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27660,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717492,"statfs":{"total":95995035648,"available":95966711808,"internally_reserved":0,"allocated":655360,"data_stored":277067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.35799999999999998,"15min":0.34899999999999998},"min":{"1min":0.23300000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.39100000000000001,"15min":0.38800000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.317}]},{"osd":1,"last update":"Sat Aug 17 17:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40799999999999997,"15min":0.40600000000000003},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.39800000000000002,"15min":0.39300000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.25}]},{"osd":2,"last update":"Sat Aug 17 17:08:47 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42699999999999999,"15min":0.42299999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.41399999999999998,"15min":0.40999999999999998},"min":{"1min":0.28000000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.32900000000000001}]},{"osd":4,"last update":"Sat Aug 17 17:09:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54200000000000004,"15min":0.54300000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.71999999999999997}]},{"osd":5,"last update":"Sat Aug 17 17:09:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.63},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57199999999999995,"15min":0.57099999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.86899999999999999}]},{"osd":6,"last update":"Sat Aug 17 17:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.82099999999999995}]},{"osd":7,"last update":"Sat Aug 17 17:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.67200000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":659456,"data_stored":277348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35099999999999998,"15min":0.34999999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.35599999999999998,"15min":0.35699999999999998},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.503,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.31900000000000001}]},{"osd":1,"last update":"Sat Aug 17 17:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.36399999999999999,"15min":0.36199999999999999},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.52300000000000002,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38200000000000001,"15min":0.374},"min":{"1min":0.255,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.378}]},{"osd":3,"last update":"Sat Aug 17 17:09:46 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.41899999999999998,"15min":0.435},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.48399999999999999,"5min":0.753,"15min":0.753},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.375,"5min":0.41899999999999998,"15min":0.42699999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.61199999999999999,"5min":0.622,"15min":0.622},"last":0.34999999999999998}]},{"osd":4,"last update":"Sat Aug 17 17:09:07 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.55400000000000005,"15min":0.56399999999999995},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71299999999999997,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67900000000000005,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.56499999999999995}]},{"osd":5,"last update":"Sat Aug 17 17:09:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.54100000000000004,"15min":0.54300000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.67100000000000004,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.47599999999999998}]},{"osd":6,"last update":"Sat Aug 17 17:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.57199999999999995}]},{"osd":7,"last update":"Sat Aug 17 17:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.495}]}]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":663552,"data_stored":277367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:09:00 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.34000000000000002,"15min":0.32900000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.54900000000000004,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.315},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.35999999999999999,"15min":0.35299999999999998},"min":{"1min":0.23400000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.50600000000000001,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.26700000000000002}]},{"osd":2,"last update":"Sat Aug 17 17:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.39200000000000002,"15min":0.39800000000000002},"min":{"1min":0.23999999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.47899999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.39700000000000002,"15min":0.40400000000000003},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.56200000000000006,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.30099999999999999}]},{"osd":3,"last update":"Sat Aug 17 17:08:48 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.44600000000000001,"15min":0.45400000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.59099999999999997,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.42199999999999999,"15min":0.42599999999999999},"min":{"1min":0.25700000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.53400000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.377}]},{"osd":4,"last update":"Sat Aug 17 17:09:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.52500000000000002,"15min":0.51800000000000002},"min":{"1min":0.41899999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.432},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53600000000000003,"15min":0.53900000000000003},"min":{"1min":0.39900000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.53500000000000003}]},{"osd":5,"last update":"Sat Aug 17 17:09:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.58199999999999996,"15min":0.58799999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.56399999999999995,"15min":0.57099999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.505}]},{"osd":6,"last update":"Sat Aug 17 17:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.496}]},{"osd":7,"last update":"Sat Aug 17 17:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.48399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28244,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716908,"statfs":{"total":95995035648,"available":95966113792,"internally_reserved":0,"allocated":1253376,"data_stored":867716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 17 17:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.314,"15min":0.28699999999999998},"min":{"1min":0.189,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.313,"15min":0.28799999999999998},"min":{"1min":0.23899999999999999,"5min":0.19,"15min":0.19},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.28599999999999998}]},{"osd":2,"last update":"Sat Aug 17 17:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.35399999999999998,"15min":0.33500000000000002},"min":{"1min":0.246,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.371,"15min":0.35299999999999998},"min":{"1min":0.26200000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.22700000000000001}]},{"osd":3,"last update":"Sat Aug 17 17:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.42799999999999999,"15min":0.42199999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.253}]},{"osd":4,"last update":"Sat Aug 17 17:09:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.34999999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.55300000000000005,"15min":0.54700000000000004},"min":{"1min":0.46899999999999997,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.433}]},{"osd":5,"last update":"Sat Aug 17 17:09:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.41099999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71799999999999997,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.54200000000000004,"15min":0.53100000000000003},"min":{"1min":0.38,"5min":0.311,"15min":0.311},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.35199999999999998}]},{"osd":6,"last update":"Sat Aug 17 17:08:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.38100000000000001}]},{"osd":7,"last update":"Sat Aug 17 17:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.49299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:09:53.203 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:09:53.819 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:53 smithi136 ceph-mon[31034]: from='client.24521 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:09:53.819 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:53 smithi136 ceph-mon[42580]: from='client.24521 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:09:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:53 smithi165 ceph-mon[34868]: from='client.24521 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:09:54.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:54 smithi136 ceph-mon[31034]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:54.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:54 smithi136 ceph-mon[42580]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:54 smithi165 ceph-mon[34868]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:55.527 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:55.766 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:55 smithi136 systemd[1]: Stopping Ceph alertmanager.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:09:56.394 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:09:56 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:09:56] "GET /metrics HTTP/1.1" 200 28514 "" "Prometheus/2.43.0" 2024-08-17T17:09:56.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:56 smithi136 ceph-mon[31034]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:56.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:56 smithi136 ceph-mon[42580]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:56.947 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:56 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[73386]: ts=2024-08-17T17:09:56.849Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-08-17T17:09:57.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:56 smithi165 ceph-mon[34868]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:57.401 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:57 smithi136 podman[78694]: 2024-08-17 17:09:57.099069502 +0000 UTC m=+0.799585608 container died b95ab241773b9f4a45c290bcd46522dcd024cf7ba2046fc1720621ef49a222bc (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:09:57.696 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:57 smithi136 podman[78694]: 2024-08-17 17:09:57.500315483 +0000 UTC m=+1.200831586 container cleanup b95ab241773b9f4a45c290bcd46522dcd024cf7ba2046fc1720621ef49a222bc (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:09:57.696 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:57 smithi136 bash[78694]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a 2024-08-17T17:09:58.036 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:57 smithi136 podman[78731]: 2024-08-17 17:09:57.709882948 +0000 UTC m=+0.837335365 container remove b95ab241773b9f4a45c290bcd46522dcd024cf7ba2046fc1720621ef49a222bc (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:09:58.037 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:57 smithi136 podman[78731]: 2024-08-17 17:09:57.802372091 +0000 UTC m=+0.929824509 volume remove affc55ae8846a137225c6bdb7890cb665886b840a515b030ae409531d3218d82 2024-08-17T17:09:58.223 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:09:58.223 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T17:09:58.871 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-08-17T17:09:57.870808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":181,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131,"ondisk_log_size":131,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":223048,"kb_used_data":6888,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":749738168,"statfs":{"total":767960285184,"available":767731884032,"internally_reserved":0,"allocated":7053312,"data_stored":3988771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002057"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-08-17T17:09:02.570337+0000","last_change":"2024-08-17T17:08:53.945206+0000","last_active":"2024-08-17T17:09:02.570337+0000","last_peered":"2024-08-17T17:09:02.570337+0000","last_clean":"2024-08-17T17:09:02.570337+0000","last_became_active":"2024-08-17T17:08:48.853366+0000","last_became_peered":"2024-08-17T17:08:48.853366+0000","last_unstale":"2024-08-17T17:09:02.570337+0000","last_undegraded":"2024-08-17T17:09:02.570337+0000","last_fullsized":"2024-08-17T17:09:02.570337+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:34:32.938937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-08-17T17:09:02.326831+0000","last_change":"2024-08-17T17:08:53.939970+0000","last_active":"2024-08-17T17:09:02.326831+0000","last_peered":"2024-08-17T17:09:02.326831+0000","last_clean":"2024-08-17T17:09:02.326831+0000","last_became_active":"2024-08-17T17:08:48.852650+0000","last_became_peered":"2024-08-17T17:08:48.852650+0000","last_unstale":"2024-08-17T17:09:02.326831+0000","last_undegraded":"2024-08-17T17:09:02.326831+0000","last_fullsized":"2024-08-17T17:09:02.326831+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:40.181438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000256001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-08-17T17:09:02.626084+0000","last_change":"2024-08-17T17:08:54.068835+0000","last_active":"2024-08-17T17:09:02.626084+0000","last_peered":"2024-08-17T17:09:02.626084+0000","last_clean":"2024-08-17T17:09:02.626084+0000","last_became_active":"2024-08-17T17:08:48.853784+0000","last_became_peered":"2024-08-17T17:08:48.853784+0000","last_unstale":"2024-08-17T17:09:02.626084+0000","last_undegraded":"2024-08-17T17:09:02.626084+0000","last_fullsized":"2024-08-17T17:09:02.626084+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:43:23.734272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-08-17T17:09:02.326806+0000","last_change":"2024-08-17T17:08:53.939956+0000","last_active":"2024-08-17T17:09:02.326806+0000","last_peered":"2024-08-17T17:09:02.326806+0000","last_clean":"2024-08-17T17:09:02.326806+0000","last_became_active":"2024-08-17T17:08:48.851812+0000","last_became_peered":"2024-08-17T17:08:48.851812+0000","last_unstale":"2024-08-17T17:09:02.326806+0000","last_undegraded":"2024-08-17T17:09:02.326806+0000","last_fullsized":"2024-08-17T17:09:02.326806+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:25:20.326625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":22,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-08-17T17:09:02.836693+0000","last_change":"2024-08-17T17:08:53.944227+0000","last_active":"2024-08-17T17:09:02.836693+0000","last_peered":"2024-08-17T17:09:02.836693+0000","last_clean":"2024-08-17T17:09:02.836693+0000","last_became_active":"2024-08-17T17:08:48.854463+0000","last_became_peered":"2024-08-17T17:08:48.854463+0000","last_unstale":"2024-08-17T17:09:02.836693+0000","last_undegraded":"2024-08-17T17:09:02.836693+0000","last_fullsized":"2024-08-17T17:09:02.836693+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:06:41.281670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001023532,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-08-17T17:09:02.836401+0000","last_change":"2024-08-17T17:08:53.939867+0000","last_active":"2024-08-17T17:09:02.836401+0000","last_peered":"2024-08-17T17:09:02.836401+0000","last_clean":"2024-08-17T17:09:02.836401+0000","last_became_active":"2024-08-17T17:08:48.927330+0000","last_became_peered":"2024-08-17T17:08:48.927330+0000","last_unstale":"2024-08-17T17:09:02.836401+0000","last_undegraded":"2024-08-17T17:09:02.836401+0000","last_fullsized":"2024-08-17T17:09:02.836401+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:03:20.888081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-08-17T17:09:02.626116+0000","last_change":"2024-08-17T17:08:54.068863+0000","last_active":"2024-08-17T17:09:02.626116+0000","last_peered":"2024-08-17T17:09:02.626116+0000","last_clean":"2024-08-17T17:09:02.626116+0000","last_became_active":"2024-08-17T17:08:48.854469+0000","last_became_peered":"2024-08-17T17:08:48.854469+0000","last_unstale":"2024-08-17T17:09:02.626116+0000","last_undegraded":"2024-08-17T17:09:02.626116+0000","last_fullsized":"2024-08-17T17:09:02.626116+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:32:44.509162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":21,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-08-17T17:09:02.836876+0000","last_change":"2024-08-17T17:08:53.943617+0000","last_active":"2024-08-17T17:09:02.836876+0000","last_peered":"2024-08-17T17:09:02.836876+0000","last_clean":"2024-08-17T17:09:02.836876+0000","last_became_active":"2024-08-17T17:08:48.854004+0000","last_became_peered":"2024-08-17T17:08:48.854004+0000","last_unstale":"2024-08-17T17:09:02.836876+0000","last_undegraded":"2024-08-17T17:09:02.836876+0000","last_fullsized":"2024-08-17T17:09:02.836876+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:53:56.669365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041737399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"61'128","reported_seq":260,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-08-17T17:08:54.068545+0000","last_change":"2024-08-17T17:08:20.329502+0000","last_active":"2024-08-17T17:08:54.068545+0000","last_peered":"2024-08-17T17:08:54.068545+0000","last_clean":"2024-08-17T17:08:54.068545+0000","last_became_active":"2024-08-17T17:08:20.329063+0000","last_became_peered":"2024-08-17T17:08:20.329063+0000","last_unstale":"2024-08-17T17:08:54.068545+0000","last_undegraded":"2024-08-17T17:08:54.068545+0000","last_fullsized":"2024-08-17T17:08:54.068545+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T17:06:21.174471+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:52:20.712819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234005,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28240,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716912,"statfs":{"total":95995035648,"available":95966117888,"internally_reserved":0,"allocated":1249280,"data_stored":867435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:09:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.55300000000000005}]},{"osd":1,"last update":"Sat Aug 17 17:09:18 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.41399999999999998}]},{"osd":2,"last update":"Sat Aug 17 17:09:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.432},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.58699999999999997}]},{"osd":3,"last update":"Sat Aug 17 17:09:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.44600000000000001}]},{"osd":4,"last update":"Sat Aug 17 17:09:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.38}]},{"osd":5,"last update":"Sat Aug 17 17:09:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.52300000000000002}]},{"osd":6,"last update":"Sat Aug 17 17:09:18 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.505}]}]},{"osd":6,"up_from":46,"seq":197568495643,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28244,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716908,"statfs":{"total":95995035648,"available":95966113792,"internally_reserved":0,"allocated":1253376,"data_stored":867454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:09:53 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.52500000000000002,"15min":0.52800000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72199999999999998,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52800000000000002,"15min":0.53000000000000003},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.54500000000000004}]},{"osd":1,"last update":"Sat Aug 17 17:09:53 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.52800000000000002,"15min":0.53200000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68999999999999995,"5min":0.755,"15min":0.755},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.54500000000000004,"15min":0.54900000000000004},"min":{"1min":0.34200000000000003,"5min":0.317,"15min":0.317},"max":{"1min":0.70499999999999996,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.53800000000000003}]},{"osd":2,"last update":"Sat Aug 17 17:09:53 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.35799999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.66600000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54900000000000004,"15min":0.55300000000000005},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.52300000000000002}]},{"osd":3,"last update":"Sat Aug 17 17:09:53 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.53600000000000003,"15min":0.53900000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.52400000000000002,"15min":0.52300000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.51200000000000001}]},{"osd":4,"last update":"Sat Aug 17 17:09:53 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.40899999999999997,"15min":0.40200000000000002},"min":{"1min":0.31,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.40600000000000003,"15min":0.40300000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.38}]},{"osd":5,"last update":"Sat Aug 17 17:09:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.42399999999999999,"15min":0.41599999999999998},"min":{"1min":0.30399999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.42799999999999999,"15min":0.42199999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.48499999999999999}]},{"osd":7,"last update":"Sat Aug 17 17:09:17 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.63300000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757279,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":659456,"data_stored":277036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:09:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53200000000000003,"15min":0.52800000000000002},"min":{"1min":0.39500000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.52000000000000002,"15min":0.52200000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.60699999999999998,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.624}]},{"osd":1,"last update":"Sat Aug 17 17:09:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.63200000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.497,"15min":0.495},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.51700000000000002}]},{"osd":2,"last update":"Sat Aug 17 17:09:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55400000000000005,"15min":0.55300000000000005},"min":{"1min":0.45400000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.53300000000000003,"15min":0.53500000000000003},"min":{"1min":0.38500000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.65700000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.58199999999999996}]},{"osd":3,"last update":"Sat Aug 17 17:09:31 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51400000000000001,"15min":0.51700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.63300000000000001,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55400000000000005,"15min":0.55200000000000005},"min":{"1min":0.437,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.59499999999999997}]},{"osd":4,"last update":"Sat Aug 17 17:09:31 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.29799999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.501,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.42799999999999999,"15min":0.435},"min":{"1min":0.28599999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.57499999999999996,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.36099999999999999}]},{"osd":6,"last update":"Sat Aug 17 17:09:53 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.44400000000000001,"15min":0.44900000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.56599999999999995,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.49099999999999999,"15min":0.49299999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.63100000000000001,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.497}]},{"osd":7,"last update":"Sat Aug 17 17:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.48299999999999998}]}]},{"osd":4,"up_from":30,"seq":128849018916,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":659456,"data_stored":277348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:09:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.50900000000000001,"15min":0.50800000000000001},"min":{"1min":0.35799999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53700000000000003,"15min":0.53800000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.76300000000000001,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.58999999999999997}]},{"osd":1,"last update":"Sat Aug 17 17:09:07 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.502,"15min":0.502},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.622},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73699999999999999,"5min":0.751,"15min":0.751},"last":0.57499999999999996}]},{"osd":2,"last update":"Sat Aug 17 17:09:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.76600000000000001,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51900000000000002,"15min":0.52000000000000002},"min":{"1min":0.36299999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.66800000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.53700000000000003}]},{"osd":3,"last update":"Sat Aug 17 17:09:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54700000000000004,"15min":0.55000000000000004},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52700000000000002,"15min":0.52900000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.64400000000000002,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.51700000000000002}]},{"osd":5,"last update":"Sat Aug 17 17:09:34 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.374,"15min":0.36599999999999999},"min":{"1min":0.312,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.438},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.39800000000000002,"15min":0.38400000000000001},"min":{"1min":0.29899999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.68999999999999995}]},{"osd":6,"last update":"Sat Aug 17 17:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.67000000000000004}]},{"osd":7,"last update":"Sat Aug 17 17:09:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.56000000000000005}]}]},{"osd":3,"up_from":25,"seq":107374182441,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27660,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717492,"statfs":{"total":95995035648,"available":95966711808,"internally_reserved":0,"allocated":655360,"data_stored":277067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.35199999999999998,"15min":0.34699999999999998},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.52800000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.378,"15min":0.38400000000000001},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.442,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.25600000000000001}]},{"osd":1,"last update":"Sat Aug 17 17:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.39200000000000002,"15min":0.40000000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.47799999999999998,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.38400000000000001,"15min":0.38800000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.42799999999999999,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.24099999999999999}]},{"osd":2,"last update":"Sat Aug 17 17:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.41399999999999998,"15min":0.41899999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.52600000000000002,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.40999999999999998,"15min":0.40799999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.32600000000000001}]},{"osd":4,"last update":"Sat Aug 17 17:09:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54200000000000004,"15min":0.54300000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.44800000000000001}]},{"osd":5,"last update":"Sat Aug 17 17:09:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57199999999999995,"15min":0.57099999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.59499999999999997}]},{"osd":6,"last update":"Sat Aug 17 17:08:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.61299999999999999}]},{"osd":7,"last update":"Sat Aug 17 17:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.57899999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":659456,"data_stored":277348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35099999999999998,"15min":0.34999999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.254},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.35599999999999998,"15min":0.35699999999999998},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.503,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.24299999999999999}]},{"osd":1,"last update":"Sat Aug 17 17:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.36399999999999999,"15min":0.36199999999999999},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.52300000000000002,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38200000000000001,"15min":0.374},"min":{"1min":0.255,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.44700000000000001}]},{"osd":3,"last update":"Sat Aug 17 17:09:46 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.41899999999999998,"15min":0.435},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.48399999999999999,"5min":0.753,"15min":0.753},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.375,"5min":0.41899999999999998,"15min":0.42699999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.61199999999999999,"5min":0.622,"15min":0.622},"last":0.28499999999999998}]},{"osd":4,"last update":"Sat Aug 17 17:09:07 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.55400000000000005,"15min":0.56399999999999995},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71299999999999997,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67900000000000005,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.47299999999999998}]},{"osd":5,"last update":"Sat Aug 17 17:09:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.54100000000000004,"15min":0.54300000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.67100000000000004,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.48799999999999999}]},{"osd":6,"last update":"Sat Aug 17 17:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.501},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.42299999999999999}]},{"osd":7,"last update":"Sat Aug 17 17:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.434}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27668,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":663552,"data_stored":277367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:09:00 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.34000000000000002,"15min":0.32900000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.54900000000000004,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.35999999999999999,"15min":0.35299999999999998},"min":{"1min":0.23400000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.50600000000000001,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.35799999999999998}]},{"osd":2,"last update":"Sat Aug 17 17:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.39200000000000002,"15min":0.39800000000000002},"min":{"1min":0.23999999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.47899999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.39700000000000002,"15min":0.40400000000000003},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.56200000000000006,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.39300000000000002}]},{"osd":3,"last update":"Sat Aug 17 17:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.42899999999999999,"15min":0.44800000000000001},"min":{"1min":0.28399999999999997,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.54600000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.40999999999999998,"15min":0.42199999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.42599999999999999}]},{"osd":4,"last update":"Sat Aug 17 17:09:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.52500000000000002,"15min":0.51800000000000002},"min":{"1min":0.41899999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.53600000000000003,"15min":0.53900000000000003},"min":{"1min":0.39900000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.755}]},{"osd":5,"last update":"Sat Aug 17 17:09:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.58199999999999996,"15min":0.58799999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.56399999999999995,"15min":0.57099999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.68700000000000006}]},{"osd":6,"last update":"Sat Aug 17 17:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.745},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.66500000000000004}]},{"osd":7,"last update":"Sat Aug 17 17:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.78200000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28244,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716908,"statfs":{"total":95995035648,"available":95966113792,"internally_reserved":0,"allocated":1253376,"data_stored":867716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 17 17:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.314,"15min":0.28699999999999998},"min":{"1min":0.189,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.374,"5min":0.313,"15min":0.28799999999999998},"min":{"1min":0.23899999999999999,"5min":0.19,"15min":0.19},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.439}]},{"osd":2,"last update":"Sat Aug 17 17:09:23 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.35399999999999998,"15min":0.33500000000000002},"min":{"1min":0.246,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.371,"15min":0.35299999999999998},"min":{"1min":0.26200000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.495}]},{"osd":3,"last update":"Sat Aug 17 17:09:47 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.42799999999999999,"15min":0.42199999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.48499999999999999}]},{"osd":4,"last update":"Sat Aug 17 17:09:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.34999999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.55300000000000005,"15min":0.54700000000000004},"min":{"1min":0.46899999999999997,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.77600000000000002}]},{"osd":5,"last update":"Sat Aug 17 17:09:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.41099999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71799999999999997,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.54200000000000004,"15min":0.53100000000000003},"min":{"1min":0.38,"5min":0.311,"15min":0.311},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.84299999999999997}]},{"osd":6,"last update":"Sat Aug 17 17:09:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58599999999999997,"15min":0.58799999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.75900000000000001,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.62,"5min":0.57799999999999996,"15min":0.57099999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.851,"5min":1.851,"15min":1.851},"last":0.86599999999999999}]},{"osd":7,"last update":"Sat Aug 17 17:09:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.67300000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:09:58.873 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-08-17T17:09:58.873 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-08-17T17:09:58.873 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-08-17T17:09:58.873 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph health --format=json 2024-08-17T17:09:59.126 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:58 smithi136 ceph-mon[31034]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:59.126 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:58 smithi136 ceph-mon[42580]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:59.126 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:58 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-08-17T17:09:59.127 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:58 smithi136 systemd[1]: Stopped Ceph alertmanager.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:09:59.127 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:58 smithi136 systemd[1]: Starting Ceph alertmanager.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:09:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:58 smithi165 ceph-mon[34868]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:09:59.446 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:59 smithi136 podman[78926]: 2024-08-17 17:09:59.125554803 +0000 UTC m=+0.092710212 volume create 1ba1e01fddfd2399db117295d601f34cea22623bec0375b5c15526a93b3cb8ab 2024-08-17T17:09:59.446 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:59 smithi136 podman[78926]: 2024-08-17 17:09:59.051525197 +0000 UTC m=+0.018680581 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-08-17T17:09:59.446 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:59 smithi136 podman[78926]: 2024-08-17 17:09:59.209389909 +0000 UTC m=+0.176545316 container create 83b6bfc5306f207fbd8b48979a76ea59898ceca894fa5d772183019c8dcc78e1 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:09:59.620 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:59.745 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:59 smithi136 podman[78926]: 2024-08-17 17:09:59.48702313 +0000 UTC m=+0.454178503 container init 83b6bfc5306f207fbd8b48979a76ea59898ceca894fa5d772183019c8dcc78e1 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:09:59.745 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:59 smithi136 podman[78926]: 2024-08-17 17:09:59.492056663 +0000 UTC m=+0.459212027 container start 83b6bfc5306f207fbd8b48979a76ea59898ceca894fa5d772183019c8dcc78e1 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:09:59.745 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[78967]: ts=2024-08-17T17:09:59.504Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-08-17T17:09:59.745 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[78967]: ts=2024-08-17T17:09:59.504Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-08-17T17:09:59.745 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[78967]: ts=2024-08-17T17:09:59.505Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.136 port=9094 2024-08-17T17:09:59.746 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[78967]: ts=2024-08-17T17:09:59.506Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-08-17T17:09:59.746 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[78967]: ts=2024-08-17T17:09:59.528Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-08-17T17:09:59.746 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[78967]: ts=2024-08-17T17:09:59.529Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-08-17T17:09:59.746 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[78967]: ts=2024-08-17T17:09:59.531Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-08-17T17:09:59.746 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[78967]: ts=2024-08-17T17:09:59.531Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-08-17T17:09:59.746 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:59 smithi136 bash[78926]: 83b6bfc5306f207fbd8b48979a76ea59898ceca894fa5d772183019c8dcc78e1 2024-08-17T17:09:59.747 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:09:59 smithi136 systemd[1]: Started Ceph alertmanager.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:10:00.032 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:59 smithi136 ceph-mon[31034]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:10:00.032 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:59 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:00.033 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:09:59 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:00.033 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:59 smithi136 ceph-mon[42580]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:10:00.033 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:59 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:00.033 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:09:59 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:00.189 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:59 smithi165 ceph-mon[34868]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:10:00.189 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:59 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:00.189 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:09:59 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:01.177 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:00 smithi136 ceph-mon[31034]: Reconfiguring prometheus.a (dependencies changed)... 2024-08-17T17:10:01.177 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:00 smithi136 ceph-mon[31034]: Reconfiguring daemon prometheus.a on smithi165 2024-08-17T17:10:01.177 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:00 smithi136 ceph-mon[31034]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:01.177 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:00 smithi136 ceph-mon[31034]: overall HEALTH_OK 2024-08-17T17:10:01.177 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:00 smithi136 ceph-mon[42580]: Reconfiguring prometheus.a (dependencies changed)... 2024-08-17T17:10:01.178 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:00 smithi136 ceph-mon[42580]: Reconfiguring daemon prometheus.a on smithi165 2024-08-17T17:10:01.178 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:00 smithi136 ceph-mon[42580]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:01.178 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:00 smithi136 ceph-mon[42580]: overall HEALTH_OK 2024-08-17T17:10:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:00 smithi165 ceph-mon[34868]: Reconfiguring prometheus.a (dependencies changed)... 2024-08-17T17:10:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:00 smithi165 ceph-mon[34868]: Reconfiguring daemon prometheus.a on smithi165 2024-08-17T17:10:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:00 smithi165 ceph-mon[34868]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:00 smithi165 ceph-mon[34868]: overall HEALTH_OK 2024-08-17T17:10:01.568 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:10:01.568 INFO:teuthology.orchestra.run.smithi136.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-17T17:10:01.869 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:10:01 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[78967]: ts=2024-08-17T17:10:01.506Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000432212s 2024-08-17T17:10:02.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:01 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/572818917' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-08-17T17:10:02.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:01 smithi136 ceph-mon[31034]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:02.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:01 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/572818917' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-08-17T17:10:02.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:01 smithi136 ceph-mon[42580]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:02.219 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:01 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/572818917' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-08-17T17:10:02.219 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:01 smithi165 ceph-mon[34868]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:02.225 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-08-17T17:10:02.225 INFO:tasks.cephadm:Setup complete, yielding 2024-08-17T17:10:02.225 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-08-17T17:10:02.235 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi136.front.sepia.ceph.com 2024-08-17T17:10:02.236 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- bash -c 'ceph fs volume create foo' 2024-08-17T17:10:02.437 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:10:03.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:02 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:10:03.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:02 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:10:03.261 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:03 smithi165 systemd[1]: Stopping Ceph prometheus.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:10:03.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:02 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:10:03.561 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:03 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:10:03.282Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-08-17T17:10:03.561 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:03 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:10:03.282Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-08-17T17:10:03.562 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:03 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:10:03.282Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-08-17T17:10:03.562 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:03 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:10:03.282Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-08-17T17:10:03.562 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:03 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:10:03.282Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-08-17T17:10:03.562 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:03 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:10:03.282Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-08-17T17:10:03.562 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:03 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:10:03.282Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-08-17T17:10:03.562 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:03 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:10:03.282Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-08-17T17:10:03.562 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:03 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:10:03.282Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-08-17T17:10:03.562 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:03 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:10:03.283Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-08-17T17:10:03.562 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:03 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:10:03.283Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-08-17T17:10:03.562 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:03 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[63708]: ts=2024-08-17T17:10:03.283Z caller=main.go:1147 level=info msg="See you next time!" 2024-08-17T17:10:03.562 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:03 smithi165 podman[67022]: 2024-08-17 17:10:03.292770863 +0000 UTC m=+0.122544230 container died b55171200053202ccf2388527918b3031afc76a79e999f6e6b36759ec577ae94 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:10:03.903 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:03 smithi165 podman[67022]: 2024-08-17 17:10:03.718290004 +0000 UTC m=+0.548063358 container cleanup b55171200053202ccf2388527918b3031afc76a79e999f6e6b36759ec577ae94 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:10:03.903 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:03 smithi165 bash[67022]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a 2024-08-17T17:10:04.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:03 smithi136 ceph-mon[31034]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:04.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:03 smithi136 ceph-mon[42580]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:03 smithi165 ceph-mon[34868]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:04.261 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:03 smithi165 podman[67042]: 2024-08-17 17:10:03.903387563 +0000 UTC m=+0.613125808 container remove b55171200053202ccf2388527918b3031afc76a79e999f6e6b36759ec577ae94 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:10:04.261 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-08-17T17:10:04.261 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 systemd[1]: Stopped Ceph prometheus.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:10:04.261 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@prometheus.a.service: Consumed 1.171s CPU time. 2024-08-17T17:10:04.261 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 systemd[1]: Starting Ceph prometheus.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:10:04.632 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 podman[67136]: 2024-08-17 17:10:04.266921751 +0000 UTC m=+0.019097680 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-08-17T17:10:04.632 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 podman[67136]: 2024-08-17 17:10:04.387734284 +0000 UTC m=+0.139910248 container create 1e7f77a177081fc1ae830587072d9be4369fb97dafbe29fa4310210ef49f6530 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:10:04.940 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 podman[67136]: 2024-08-17 17:10:04.631943257 +0000 UTC m=+0.384119180 container init 1e7f77a177081fc1ae830587072d9be4369fb97dafbe29fa4310210ef49f6530 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:10:04.940 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 podman[67136]: 2024-08-17 17:10:04.637201908 +0000 UTC m=+0.389377832 container start 1e7f77a177081fc1ae830587072d9be4369fb97dafbe29fa4310210ef49f6530 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:10:04.940 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:10:04.660Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-08-17T17:10:04.940 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:10:04.660Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-08-17T17:10:04.940 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:10:04.660Z caller=main.go:570 level=info host_details="(Linux 5.14.0-480.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024 x86_64 smithi165 (none))" 2024-08-17T17:10:04.940 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:10:04.661Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-08-17T17:10:04.941 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:10:04.661Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-08-17T17:10:04.941 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:10:04.662Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-08-17T17:10:04.941 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:10:04.662Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-08-17T17:10:04.941 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:10:04.663Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-08-17T17:10:04.941 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:10:04.663Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-08-17T17:10:04.941 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:10:04.666Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-08-17T17:10:04.941 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:10:04.666Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.093µs 2024-08-17T17:10:04.941 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:10:04.666Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-08-17T17:10:04.941 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:10:04.668Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-08-17T17:10:04.941 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:10:04.668Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-08-17T17:10:04.941 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:10:04.668Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=20.086µs wal_replay_duration=1.706177ms wbl_replay_duration=100ns total_replay_duration=1.745118ms 2024-08-17T17:10:04.941 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:10:04.669Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-08-17T17:10:04.941 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:10:04.669Z caller=main.go:1029 level=info msg="TSDB started" 2024-08-17T17:10:04.942 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:10:04.669Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-08-17T17:10:04.942 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:10:04.689Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.14454ms db_storage=1.191µs remote_storage=992ns web_handler=346ns query_engine=538ns scrape=582.712µs scrape_sd=82.301µs notify=14.844µs notify_sd=36.236µs rules=19.10923ms tracing=6.324µs 2024-08-17T17:10:04.942 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:10:04.689Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-08-17T17:10:04.942 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:10:04.689Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-08-17T17:10:04.942 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 bash[67136]: 1e7f77a177081fc1ae830587072d9be4369fb97dafbe29fa4310210ef49f6530 2024-08-17T17:10:04.942 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:10:04 smithi165 systemd[1]: Started Ceph prometheus.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:10:05.196 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:10:04] ENGINE Bus STOPPING 2024-08-17T17:10:05.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:10:04] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-08-17T17:10:05.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:10:04] ENGINE Bus STOPPED 2024-08-17T17:10:05.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:10:04] ENGINE Bus STARTING 2024-08-17T17:10:05.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:10:04] ENGINE Serving on http://:::9283 2024-08-17T17:10:05.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:10:04] ENGINE Bus STARTED 2024-08-17T17:10:05.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:10:04] ENGINE Bus STOPPING 2024-08-17T17:10:05.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[31034]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:05.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-08-17T17:10:05.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-08-17T17:10:05.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:05.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:05.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-08-17T17:10:05.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[31034]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-08-17T17:10:05.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi136.front.sepia.ceph.com:9093"}]: dispatch 2024-08-17T17:10:05.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[31034]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi136.front.sepia.ceph.com:9093"}]: dispatch 2024-08-17T17:10:05.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:05.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:10:05.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[31034]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:10:05.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi165.front.sepia.ceph.com:9095"}]: dispatch 2024-08-17T17:10:05.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[31034]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi165.front.sepia.ceph.com:9095"}]: dispatch 2024-08-17T17:10:05.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:05.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-08-17T17:10:05.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[31034]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-08-17T17:10:05.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi165.front.sepia.ceph.com:3000"}]: dispatch 2024-08-17T17:10:05.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[31034]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi165.front.sepia.ceph.com:3000"}]: dispatch 2024-08-17T17:10:05.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:05.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:10:05.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[42580]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:05.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-08-17T17:10:05.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-08-17T17:10:05.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:05.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:05.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-08-17T17:10:05.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[42580]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-08-17T17:10:05.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi136.front.sepia.ceph.com:9093"}]: dispatch 2024-08-17T17:10:05.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[42580]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi136.front.sepia.ceph.com:9093"}]: dispatch 2024-08-17T17:10:05.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:05.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:10:05.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[42580]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:10:05.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi165.front.sepia.ceph.com:9095"}]: dispatch 2024-08-17T17:10:05.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[42580]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi165.front.sepia.ceph.com:9095"}]: dispatch 2024-08-17T17:10:05.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:05.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-08-17T17:10:05.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[42580]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-08-17T17:10:05.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi165.front.sepia.ceph.com:3000"}]: dispatch 2024-08-17T17:10:05.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[42580]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi165.front.sepia.ceph.com:3000"}]: dispatch 2024-08-17T17:10:05.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:05.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:04 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:10:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-mon[34868]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-08-17T17:10:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-08-17T17:10:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-08-17T17:10:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-mon[34868]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-08-17T17:10:05.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi136.front.sepia.ceph.com:9093"}]: dispatch 2024-08-17T17:10:05.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-mon[34868]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi136.front.sepia.ceph.com:9093"}]: dispatch 2024-08-17T17:10:05.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:05.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:10:05.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-mon[34868]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:10:05.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi165.front.sepia.ceph.com:9095"}]: dispatch 2024-08-17T17:10:05.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-mon[34868]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi165.front.sepia.ceph.com:9095"}]: dispatch 2024-08-17T17:10:05.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:05.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-08-17T17:10:05.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-mon[34868]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-08-17T17:10:05.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi165.front.sepia.ceph.com:3000"}]: dispatch 2024-08-17T17:10:05.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-mon[34868]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi165.front.sepia.ceph.com:3000"}]: dispatch 2024-08-17T17:10:05.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:05.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:04 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:10:05.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:10:05 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:10:05] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-08-17T17:10:05.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:10:05 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:10:05] ENGINE Bus STOPPED 2024-08-17T17:10:05.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:10:05 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:10:05] ENGINE Bus STARTING 2024-08-17T17:10:05.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:10:05 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:10:05] ENGINE Serving on http://:::9283 2024-08-17T17:10:05.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:10:05 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:10:05] ENGINE Bus STARTED 2024-08-17T17:10:05.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:10:05 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:10:05] ENGINE Bus STOPPING 2024-08-17T17:10:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:05 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-08-17T17:10:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:05 smithi165 ceph-mon[34868]: osdmap e66: 8 total, 8 up, 8 in 2024-08-17T17:10:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:05 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-08-17T17:10:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:05 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-08-17T17:10:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:05 smithi165 ceph-mon[34868]: pgmap v49: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:06.347 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:05 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-08-17T17:10:06.347 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:05 smithi136 ceph-mon[31034]: osdmap e66: 8 total, 8 up, 8 in 2024-08-17T17:10:06.347 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:05 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-08-17T17:10:06.347 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:05 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-08-17T17:10:06.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:05 smithi136 ceph-mon[31034]: pgmap v49: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:06.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:05 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a[31011]: 2024-08-17T17:10:05.962+0000 7f1924008640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-17T17:10:06.348 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:05 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-08-17T17:10:06.348 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:05 smithi136 ceph-mon[42580]: osdmap e66: 8 total, 8 up, 8 in 2024-08-17T17:10:06.348 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:05 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-08-17T17:10:06.348 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:05 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-08-17T17:10:06.348 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:05 smithi136 ceph-mon[42580]: pgmap v49: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:06.348 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:10:06] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-08-17T17:10:06.349 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:10:06] ENGINE Bus STOPPED 2024-08-17T17:10:06.349 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:10:06] ENGINE Bus STARTING 2024-08-17T17:10:06.349 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:10:06] ENGINE Serving on http://:::9283 2024-08-17T17:10:06.349 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:10:06] ENGINE Bus STARTED 2024-08-17T17:10:06.677 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-08-17T17:10:07.213 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-mon[31034]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:10:07.213 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-08-17T17:10:07.213 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-mon[31034]: osdmap e67: 8 total, 8 up, 8 in 2024-08-17T17:10:07.213 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-08-17T17:10:07.213 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-08-17T17:10:07.213 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-mon[31034]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-17T17:10:07.213 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-mon[31034]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-08-17T17:10:07.213 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-mon[31034]: osdmap e68: 8 total, 8 up, 8 in 2024-08-17T17:10:07.213 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-08-17T17:10:07.213 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-mon[31034]: fsmap foo:0 2024-08-17T17:10:07.214 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-mon[31034]: Saving service mds.foo spec with placement count:2 2024-08-17T17:10:07.214 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:07.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-mon[42580]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:10:07.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-08-17T17:10:07.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-mon[42580]: osdmap e67: 8 total, 8 up, 8 in 2024-08-17T17:10:07.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-08-17T17:10:07.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-08-17T17:10:07.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-mon[42580]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-17T17:10:07.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-mon[42580]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-08-17T17:10:07.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-mon[42580]: osdmap e68: 8 total, 8 up, 8 in 2024-08-17T17:10:07.215 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-08-17T17:10:07.215 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-mon[42580]: fsmap foo:0 2024-08-17T17:10:07.215 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-mon[42580]: Saving service mds.foo spec with placement count:2 2024-08-17T17:10:07.215 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:06 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:06 smithi165 ceph-mon[34868]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:10:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:06 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-08-17T17:10:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:06 smithi165 ceph-mon[34868]: osdmap e67: 8 total, 8 up, 8 in 2024-08-17T17:10:07.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:06 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-08-17T17:10:07.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:06 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-08-17T17:10:07.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:06 smithi165 ceph-mon[34868]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-17T17:10:07.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:06 smithi165 ceph-mon[34868]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-08-17T17:10:07.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:06 smithi165 ceph-mon[34868]: osdmap e68: 8 total, 8 up, 8 in 2024-08-17T17:10:07.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:06 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-08-17T17:10:07.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:06 smithi165 ceph-mon[34868]: fsmap foo:0 2024-08-17T17:10:07.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:06 smithi165 ceph-mon[34868]: Saving service mds.foo spec with placement count:2 2024-08-17T17:10:07.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:06 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:07.533 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:10:08.159 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:07 smithi165 ceph-mon[34868]: osdmap e69: 8 total, 8 up, 8 in 2024-08-17T17:10:08.159 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:07 smithi165 ceph-mon[34868]: pgmap v53: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:08.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:07 smithi136 ceph-mon[31034]: osdmap e69: 8 total, 8 up, 8 in 2024-08-17T17:10:08.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:07 smithi136 ceph-mon[31034]: pgmap v53: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:08.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:07 smithi136 ceph-mon[42580]: osdmap e69: 8 total, 8 up, 8 in 2024-08-17T17:10:08.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:07 smithi136 ceph-mon[42580]: pgmap v53: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:08 smithi165 ceph-mon[34868]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:10:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:08 smithi165 ceph-mon[34868]: osdmap e70: 8 total, 8 up, 8 in 2024-08-17T17:10:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:08 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:08 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:09.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:08 smithi136 ceph-mon[31034]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:10:09.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:08 smithi136 ceph-mon[31034]: osdmap e70: 8 total, 8 up, 8 in 2024-08-17T17:10:09.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:08 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:09.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:08 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:09.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:08 smithi136 ceph-mon[42580]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:10:09.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:08 smithi136 ceph-mon[42580]: osdmap e70: 8 total, 8 up, 8 in 2024-08-17T17:10:09.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:08 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:09.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:08 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:09.696 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:10:09 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[78967]: ts=2024-08-17T17:10:09.508Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.00220626s 2024-08-17T17:10:10.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:09 smithi165 ceph-mon[34868]: pgmap v55: 73 pgs: 73 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:10.347 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:09 smithi136 ceph-mon[42580]: pgmap v55: 73 pgs: 73 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:10.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:09 smithi136 ceph-mon[31034]: pgmap v55: 73 pgs: 73 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:10.727 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-08-17T17:10:11.294 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:11 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:11.294 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:11 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:11.294 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:11 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:10:11.294 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:11 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:10:11.294 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:11 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:11.295 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:11 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi136.ltsuqd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-17T17:10:11.295 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:11 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi136.ltsuqd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-17T17:10:11.295 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:11 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi136.ltsuqd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-17T17:10:11.295 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:11 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:10:11.295 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:11 smithi136 ceph-mon[31034]: Deploying daemon mds.foo.smithi136.ltsuqd on smithi136 2024-08-17T17:10:11.295 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:11 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:11.295 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:11 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:11.295 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:11 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:10:11.295 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:11 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:10:11.295 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:11 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:11.295 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:11 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi136.ltsuqd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-17T17:10:11.296 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:11 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi136.ltsuqd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-17T17:10:11.296 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:11 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi136.ltsuqd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-17T17:10:11.296 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:11 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:10:11.296 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:11 smithi136 ceph-mon[42580]: Deploying daemon mds.foo.smithi136.ltsuqd on smithi136 2024-08-17T17:10:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:11 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:11 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:11 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:10:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:11 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:10:11.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:11 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:11.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:11 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi136.ltsuqd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-17T17:10:11.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:11 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi136.ltsuqd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-17T17:10:11.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:11 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi136.ltsuqd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-17T17:10:11.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:11 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:10:11.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:11 smithi165 ceph-mon[34868]: Deploying daemon mds.foo.smithi136.ltsuqd on smithi136 2024-08-17T17:10:11.516 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:10:12.305 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:12 smithi136 ceph-mon[31034]: pgmap v56: 73 pgs: 73 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:12.305 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:12 smithi136 ceph-mon[42580]: pgmap v56: 73 pgs: 73 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:12.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:12 smithi165 ceph-mon[34868]: pgmap v56: 73 pgs: 73 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:14.924 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-08-17T17:10:14.938 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-08-17T17:10:14.938 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-08-17T17:10:14.938 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph mgr dump --format=json 2024-08-17T17:10:15.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:14 smithi136 ceph-mon[31034]: pgmap v57: 73 pgs: 73 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:15.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:14 smithi136 ceph-mon[42580]: pgmap v57: 73 pgs: 73 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:15.232 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:10:15.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:14 smithi165 ceph-mon[34868]: pgmap v57: 73 pgs: 73 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:16.402 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:10:16] "GET /metrics HTTP/1.1" 200 28512 "" "Prometheus/2.43.0" 2024-08-17T17:10:16.402 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:16.402 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:16.402 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:16.402 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi165.bmrbhh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-17T17:10:16.402 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi165.bmrbhh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-17T17:10:16.402 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi165.bmrbhh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-17T17:10:16.402 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:10:16.403 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[31034]: Deploying daemon mds.foo.smithi165.bmrbhh on smithi165 2024-08-17T17:10:16.403 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[31034]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:16.403 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[31034]: mds.? [v2:172.21.15.136:6834/764583248,v1:172.21.15.136:6835/764583248] up:boot 2024-08-17T17:10:16.403 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[31034]: daemon mds.foo.smithi136.ltsuqd assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-08-17T17:10:16.403 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[31034]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-08-17T17:10:16.403 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[31034]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-08-17T17:10:16.403 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[31034]: Cluster is now healthy 2024-08-17T17:10:16.403 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi136.ltsuqd"}]: dispatch 2024-08-17T17:10:16.403 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[31034]: fsmap foo:0 1 up:standby 2024-08-17T17:10:16.403 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[31034]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:creating} 2024-08-17T17:10:16.403 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[31034]: daemon mds.foo.smithi136.ltsuqd is now active in filesystem foo as rank 0 2024-08-17T17:10:16.403 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:16.404 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:16.404 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:16.404 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi165.bmrbhh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-17T17:10:16.404 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi165.bmrbhh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-17T17:10:16.404 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi165.bmrbhh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-17T17:10:16.404 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:10:16.404 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[42580]: Deploying daemon mds.foo.smithi165.bmrbhh on smithi165 2024-08-17T17:10:16.404 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[42580]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:16.404 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[42580]: mds.? [v2:172.21.15.136:6834/764583248,v1:172.21.15.136:6835/764583248] up:boot 2024-08-17T17:10:16.404 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[42580]: daemon mds.foo.smithi136.ltsuqd assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-08-17T17:10:16.404 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[42580]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-08-17T17:10:16.404 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[42580]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-08-17T17:10:16.404 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[42580]: Cluster is now healthy 2024-08-17T17:10:16.404 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi136.ltsuqd"}]: dispatch 2024-08-17T17:10:16.405 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[42580]: fsmap foo:0 1 up:standby 2024-08-17T17:10:16.405 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[42580]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:creating} 2024-08-17T17:10:16.405 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:16 smithi136 ceph-mon[42580]: daemon mds.foo.smithi136.ltsuqd is now active in filesystem foo as rank 0 2024-08-17T17:10:16.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:16 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:16.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:16 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:16.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:16 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:16.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:16 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi165.bmrbhh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-17T17:10:16.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:16 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi165.bmrbhh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-17T17:10:16.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:16 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi165.bmrbhh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-17T17:10:16.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:16 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:10:16.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:16 smithi165 ceph-mon[34868]: Deploying daemon mds.foo.smithi165.bmrbhh on smithi165 2024-08-17T17:10:16.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:16 smithi165 ceph-mon[34868]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:16.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:16 smithi165 ceph-mon[34868]: mds.? [v2:172.21.15.136:6834/764583248,v1:172.21.15.136:6835/764583248] up:boot 2024-08-17T17:10:16.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:16 smithi165 ceph-mon[34868]: daemon mds.foo.smithi136.ltsuqd assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-08-17T17:10:16.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:16 smithi165 ceph-mon[34868]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-08-17T17:10:16.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:16 smithi165 ceph-mon[34868]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-08-17T17:10:16.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:16 smithi165 ceph-mon[34868]: Cluster is now healthy 2024-08-17T17:10:16.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:16 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi136.ltsuqd"}]: dispatch 2024-08-17T17:10:16.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:16 smithi165 ceph-mon[34868]: fsmap foo:0 1 up:standby 2024-08-17T17:10:16.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:16 smithi165 ceph-mon[34868]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:creating} 2024-08-17T17:10:16.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:16 smithi165 ceph-mon[34868]: daemon mds.foo.smithi136.ltsuqd is now active in filesystem foo as rank 0 2024-08-17T17:10:17.109 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:10:17.759 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":20,"flags":0,"active_gid":14442,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":803461847},{"type":"v1","addr":"172.21.15.136:6801","nonce":803461847}]},"active_addr":"172.21.15.136:6801/803461847","active_change":"2024-08-17T17:08:47.841296+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24346,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.136:8443/","prometheus":"http://172.21.15.136:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":59,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":2887741187}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":972617428}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":707907005}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":3166566423}]}]} 2024-08-17T17:10:17.764 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-08-17T17:10:17.764 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-08-17T17:10:17.764 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-17T17:10:17.978 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:10:18.102 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:17 smithi136 ceph-mon[42580]: mds.? [v2:172.21.15.136:6834/764583248,v1:172.21.15.136:6835/764583248] up:active 2024-08-17T17:10:18.102 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:17 smithi136 ceph-mon[42580]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 2024-08-17T17:10:18.102 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:17 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/895952376' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-08-17T17:10:18.102 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:17 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:18.102 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:17 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:18.102 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:17 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:18.102 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:17 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:18.102 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:17 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:18.103 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:17 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:10:18.103 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:17 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:10:18.103 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:17 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:18.103 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:17 smithi136 ceph-mon[31034]: mds.? [v2:172.21.15.136:6834/764583248,v1:172.21.15.136:6835/764583248] up:active 2024-08-17T17:10:18.103 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:17 smithi136 ceph-mon[31034]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 2024-08-17T17:10:18.103 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:17 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/895952376' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-08-17T17:10:18.103 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:17 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:18.103 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:17 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:18.103 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:17 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:18.103 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:17 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:18.103 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:17 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:18.104 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:17 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:10:18.104 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:17 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:10:18.106 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:17 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:18.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:17 smithi165 ceph-mon[34868]: mds.? [v2:172.21.15.136:6834/764583248,v1:172.21.15.136:6835/764583248] up:active 2024-08-17T17:10:18.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:17 smithi165 ceph-mon[34868]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 2024-08-17T17:10:18.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:17 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/895952376' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-08-17T17:10:18.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:17 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:18.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:17 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:18.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:17 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:18.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:17 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:18.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:17 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:18.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:17 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:10:18.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:17 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:10:18.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:17 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:19.193 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:18 smithi136 ceph-mon[31034]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:19.193 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:18 smithi136 ceph-mon[31034]: mds.? [v2:172.21.15.165:6832/4276964188,v1:172.21.15.165:6833/4276964188] up:boot 2024-08-17T17:10:19.194 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:18 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi165.bmrbhh"}]: dispatch 2024-08-17T17:10:19.194 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:18 smithi136 ceph-mon[31034]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 1 up:standby 2024-08-17T17:10:19.194 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:18 smithi136 ceph-mon[31034]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 1 up:standby 2024-08-17T17:10:19.194 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:18 smithi136 ceph-mon[42580]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:19.194 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:18 smithi136 ceph-mon[42580]: mds.? [v2:172.21.15.165:6832/4276964188,v1:172.21.15.165:6833/4276964188] up:boot 2024-08-17T17:10:19.195 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:18 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi165.bmrbhh"}]: dispatch 2024-08-17T17:10:19.195 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:18 smithi136 ceph-mon[42580]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 1 up:standby 2024-08-17T17:10:19.195 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:18 smithi136 ceph-mon[42580]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 1 up:standby 2024-08-17T17:10:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:18 smithi165 ceph-mon[34868]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:18 smithi165 ceph-mon[34868]: mds.? [v2:172.21.15.165:6832/4276964188,v1:172.21.15.165:6833/4276964188] up:boot 2024-08-17T17:10:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:18 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi165.bmrbhh"}]: dispatch 2024-08-17T17:10:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:18 smithi165 ceph-mon[34868]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 1 up:standby 2024-08-17T17:10:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:18 smithi165 ceph-mon[34868]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 1 up:standby 2024-08-17T17:10:20.116 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:19 smithi136 ceph-mon[31034]: pgmap v60: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-08-17T17:10:20.116 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:19 smithi136 ceph-mon[42580]: pgmap v60: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-08-17T17:10:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:19 smithi165 ceph-mon[34868]: pgmap v60: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-08-17T17:10:20.371 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:10:20.371 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":70,"fsid":"8270436c-5cba-11ef-bccf-c7b262605968","created":"2024-08-17T17:03:17.682767+0000","modified":"2024-08-17T17:10:07.968862+0000","last_up_change":"2024-08-17T17:08:17.220684+0000","last_in_change":"2024-08-17T17:07:59.960068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-17T17:06:20.772596+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-17T17:08:47.840093+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-08-17T17:10:04.522913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-08-17T17:10:05.212219+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51eb458d-9f33-4b87-b7f5-8acb076c16ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3864095089},{"type":"v1","addr":"172.21.15.136:6803","nonce":3864095089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3864095089},{"type":"v1","addr":"172.21.15.136:6805","nonce":3864095089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":3864095089},{"type":"v1","addr":"172.21.15.136:6809","nonce":3864095089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3864095089},{"type":"v1","addr":"172.21.15.136:6807","nonce":3864095089}]},"public_addr":"172.21.15.136:6803/3864095089","cluster_addr":"172.21.15.136:6805/3864095089","heartbeat_back_addr":"172.21.15.136:6809/3864095089","heartbeat_front_addr":"172.21.15.136:6807/3864095089","state":["exists","up"]},{"osd":1,"uuid":"aa73fa16-8736-478c-b07a-e674c18698ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":3797355920},{"type":"v1","addr":"172.21.15.136:6811","nonce":3797355920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":3797355920},{"type":"v1","addr":"172.21.15.136:6813","nonce":3797355920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3797355920},{"type":"v1","addr":"172.21.15.136:6817","nonce":3797355920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":3797355920},{"type":"v1","addr":"172.21.15.136:6815","nonce":3797355920}]},"public_addr":"172.21.15.136:6811/3797355920","cluster_addr":"172.21.15.136:6813/3797355920","heartbeat_back_addr":"172.21.15.136:6817/3797355920","heartbeat_front_addr":"172.21.15.136:6815/3797355920","state":["exists","up"]},{"osd":2,"uuid":"4d2612be-7068-48d9-bb0c-992ebccaa091","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1674243363},{"type":"v1","addr":"172.21.15.136:6819","nonce":1674243363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1674243363},{"type":"v1","addr":"172.21.15.136:6821","nonce":1674243363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1674243363},{"type":"v1","addr":"172.21.15.136:6825","nonce":1674243363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1674243363},{"type":"v1","addr":"172.21.15.136:6823","nonce":1674243363}]},"public_addr":"172.21.15.136:6819/1674243363","cluster_addr":"172.21.15.136:6821/1674243363","heartbeat_back_addr":"172.21.15.136:6825/1674243363","heartbeat_front_addr":"172.21.15.136:6823/1674243363","state":["exists","up"]},{"osd":3,"uuid":"30b8a4f2-be6d-4d5f-8397-06c2e17cfc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3486272897},{"type":"v1","addr":"172.21.15.136:6827","nonce":3486272897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3486272897},{"type":"v1","addr":"172.21.15.136:6829","nonce":3486272897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3486272897},{"type":"v1","addr":"172.21.15.136:6833","nonce":3486272897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3486272897},{"type":"v1","addr":"172.21.15.136:6831","nonce":3486272897}]},"public_addr":"172.21.15.136:6827/3486272897","cluster_addr":"172.21.15.136:6829/3486272897","heartbeat_back_addr":"172.21.15.136:6833/3486272897","heartbeat_front_addr":"172.21.15.136:6831/3486272897","state":["exists","up"]},{"osd":4,"uuid":"ae4074cf-0069-403f-acc7-74bedab266bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6801","nonce":842623090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6803","nonce":842623090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6807","nonce":842623090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6805","nonce":842623090}]},"public_addr":"172.21.15.165:6801/842623090","cluster_addr":"172.21.15.165:6803/842623090","heartbeat_back_addr":"172.21.15.165:6807/842623090","heartbeat_front_addr":"172.21.15.165:6805/842623090","state":["exists","up"]},{"osd":5,"uuid":"586c23a7-4831-4b4d-9a06-c1bfcc1c5286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6809","nonce":3602301990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6811","nonce":3602301990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6815","nonce":3602301990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6813","nonce":3602301990}]},"public_addr":"172.21.15.165:6809/3602301990","cluster_addr":"172.21.15.165:6811/3602301990","heartbeat_back_addr":"172.21.15.165:6815/3602301990","heartbeat_front_addr":"172.21.15.165:6813/3602301990","state":["exists","up"]},{"osd":6,"uuid":"e838c26b-5e4a-41c6-b35b-b438ba64b237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6817","nonce":1795674907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6819","nonce":1795674907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6823","nonce":1795674907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6821","nonce":1795674907}]},"public_addr":"172.21.15.165:6817/1795674907","cluster_addr":"172.21.15.165:6819/1795674907","heartbeat_back_addr":"172.21.15.165:6823/1795674907","heartbeat_front_addr":"172.21.15.165:6821/1795674907","state":["exists","up"]},{"osd":7,"uuid":"de2fab2e-177a-4eb0-9262-7bbef0d9837f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6825","nonce":4195921091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6827","nonce":4195921091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6831","nonce":4195921091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6829","nonce":4195921091}]},"public_addr":"172.21.15.165:6825/4195921091","cluster_addr":"172.21.15.165:6827/4195921091","heartbeat_back_addr":"172.21.15.165:6831/4195921091","heartbeat_front_addr":"172.21.15.165:6829/4195921091","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:30.507144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:54.456854+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:06:17.338746+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:06:42.262947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:02.983751+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:26.594502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:50.363918+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:08:15.455798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.136:0/1450531984":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/1905186682":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/2540800865":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/1577563918":"2024-08-18T17:04:10.472088+0000","172.21.15.136:0/1414082347":"2024-08-18T17:04:10.472088+0000","172.21.15.136:0/2336022730":"2024-08-18T17:03:48.417471+0000","172.21.15.136:6800/878395364":"2024-08-18T17:03:48.417471+0000","172.21.15.136:6801/3058734149":"2024-08-18T17:04:10.472088+0000","172.21.15.136:6801/3513268997":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/3380731497":"2024-08-18T17:03:48.417471+0000","172.21.15.136:6801/878395364":"2024-08-18T17:03:48.417471+0000","172.21.15.136:6800/3058734149":"2024-08-18T17:04:10.472088+0000","172.21.15.136:0/1381030503":"2024-08-18T17:03:48.417471+0000","172.21.15.136:6800/3513268997":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/3933059683":"2024-08-18T17:04:10.472088+0000","172.21.15.136:0/2931810343":"2024-08-18T17:08:47.841029+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-17T17:10:21.084 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-08-17T17:10:21.084 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-17T17:10:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:20 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2715383335' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:10:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:20 smithi165 ceph-mon[34868]: mds.? [v2:172.21.15.136:6834/764583248,v1:172.21.15.136:6835/764583248] up:active 2024-08-17T17:10:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:20 smithi165 ceph-mon[34868]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 1 up:standby 2024-08-17T17:10:21.317 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:10:21.440 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:20 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/2715383335' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:10:21.441 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:20 smithi136 ceph-mon[31034]: mds.? [v2:172.21.15.136:6834/764583248,v1:172.21.15.136:6835/764583248] up:active 2024-08-17T17:10:21.441 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:20 smithi136 ceph-mon[31034]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 1 up:standby 2024-08-17T17:10:21.441 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:20 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/2715383335' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:10:21.441 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:20 smithi136 ceph-mon[42580]: mds.? [v2:172.21.15.136:6834/764583248,v1:172.21.15.136:6835/764583248] up:active 2024-08-17T17:10:21.441 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:20 smithi136 ceph-mon[42580]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 1 up:standby 2024-08-17T17:10:22.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:22 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:22.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:22 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:22.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:22 smithi136 ceph-mon[31034]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-08-17T17:10:22.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:22 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:22.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:22 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:22.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:22 smithi136 ceph-mon[42580]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-08-17T17:10:22.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:22 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:22.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:22 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:22.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:22 smithi165 ceph-mon[34868]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-08-17T17:10:23.629 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:23 smithi136 ceph-mon[31034]: mds.? [v2:172.21.15.165:6832/4276964188,v1:172.21.15.165:6833/4276964188] up:standby 2024-08-17T17:10:23.630 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:23 smithi136 ceph-mon[31034]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 1 up:standby 2024-08-17T17:10:23.630 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:23 smithi136 ceph-mon[42580]: mds.? [v2:172.21.15.165:6832/4276964188,v1:172.21.15.165:6833/4276964188] up:standby 2024-08-17T17:10:23.630 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:23 smithi136 ceph-mon[42580]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 1 up:standby 2024-08-17T17:10:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:23 smithi165 ceph-mon[34868]: mds.? [v2:172.21.15.165:6832/4276964188,v1:172.21.15.165:6833/4276964188] up:standby 2024-08-17T17:10:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:23 smithi165 ceph-mon[34868]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 1 up:standby 2024-08-17T17:10:23.898 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:10:23.898 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":70,"fsid":"8270436c-5cba-11ef-bccf-c7b262605968","created":"2024-08-17T17:03:17.682767+0000","modified":"2024-08-17T17:10:07.968862+0000","last_up_change":"2024-08-17T17:08:17.220684+0000","last_in_change":"2024-08-17T17:07:59.960068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-17T17:06:20.772596+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-17T17:08:47.840093+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-08-17T17:10:04.522913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-08-17T17:10:05.212219+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51eb458d-9f33-4b87-b7f5-8acb076c16ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3864095089},{"type":"v1","addr":"172.21.15.136:6803","nonce":3864095089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3864095089},{"type":"v1","addr":"172.21.15.136:6805","nonce":3864095089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":3864095089},{"type":"v1","addr":"172.21.15.136:6809","nonce":3864095089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3864095089},{"type":"v1","addr":"172.21.15.136:6807","nonce":3864095089}]},"public_addr":"172.21.15.136:6803/3864095089","cluster_addr":"172.21.15.136:6805/3864095089","heartbeat_back_addr":"172.21.15.136:6809/3864095089","heartbeat_front_addr":"172.21.15.136:6807/3864095089","state":["exists","up"]},{"osd":1,"uuid":"aa73fa16-8736-478c-b07a-e674c18698ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":3797355920},{"type":"v1","addr":"172.21.15.136:6811","nonce":3797355920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":3797355920},{"type":"v1","addr":"172.21.15.136:6813","nonce":3797355920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3797355920},{"type":"v1","addr":"172.21.15.136:6817","nonce":3797355920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":3797355920},{"type":"v1","addr":"172.21.15.136:6815","nonce":3797355920}]},"public_addr":"172.21.15.136:6811/3797355920","cluster_addr":"172.21.15.136:6813/3797355920","heartbeat_back_addr":"172.21.15.136:6817/3797355920","heartbeat_front_addr":"172.21.15.136:6815/3797355920","state":["exists","up"]},{"osd":2,"uuid":"4d2612be-7068-48d9-bb0c-992ebccaa091","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":1674243363},{"type":"v1","addr":"172.21.15.136:6819","nonce":1674243363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":1674243363},{"type":"v1","addr":"172.21.15.136:6821","nonce":1674243363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1674243363},{"type":"v1","addr":"172.21.15.136:6825","nonce":1674243363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":1674243363},{"type":"v1","addr":"172.21.15.136:6823","nonce":1674243363}]},"public_addr":"172.21.15.136:6819/1674243363","cluster_addr":"172.21.15.136:6821/1674243363","heartbeat_back_addr":"172.21.15.136:6825/1674243363","heartbeat_front_addr":"172.21.15.136:6823/1674243363","state":["exists","up"]},{"osd":3,"uuid":"30b8a4f2-be6d-4d5f-8397-06c2e17cfc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":3486272897},{"type":"v1","addr":"172.21.15.136:6827","nonce":3486272897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3486272897},{"type":"v1","addr":"172.21.15.136:6829","nonce":3486272897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3486272897},{"type":"v1","addr":"172.21.15.136:6833","nonce":3486272897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":3486272897},{"type":"v1","addr":"172.21.15.136:6831","nonce":3486272897}]},"public_addr":"172.21.15.136:6827/3486272897","cluster_addr":"172.21.15.136:6829/3486272897","heartbeat_back_addr":"172.21.15.136:6833/3486272897","heartbeat_front_addr":"172.21.15.136:6831/3486272897","state":["exists","up"]},{"osd":4,"uuid":"ae4074cf-0069-403f-acc7-74bedab266bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6801","nonce":842623090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6803","nonce":842623090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6807","nonce":842623090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6805","nonce":842623090}]},"public_addr":"172.21.15.165:6801/842623090","cluster_addr":"172.21.15.165:6803/842623090","heartbeat_back_addr":"172.21.15.165:6807/842623090","heartbeat_front_addr":"172.21.15.165:6805/842623090","state":["exists","up"]},{"osd":5,"uuid":"586c23a7-4831-4b4d-9a06-c1bfcc1c5286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6809","nonce":3602301990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6811","nonce":3602301990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6815","nonce":3602301990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6813","nonce":3602301990}]},"public_addr":"172.21.15.165:6809/3602301990","cluster_addr":"172.21.15.165:6811/3602301990","heartbeat_back_addr":"172.21.15.165:6815/3602301990","heartbeat_front_addr":"172.21.15.165:6813/3602301990","state":["exists","up"]},{"osd":6,"uuid":"e838c26b-5e4a-41c6-b35b-b438ba64b237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6817","nonce":1795674907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6819","nonce":1795674907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6823","nonce":1795674907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6821","nonce":1795674907}]},"public_addr":"172.21.15.165:6817/1795674907","cluster_addr":"172.21.15.165:6819/1795674907","heartbeat_back_addr":"172.21.15.165:6823/1795674907","heartbeat_front_addr":"172.21.15.165:6821/1795674907","state":["exists","up"]},{"osd":7,"uuid":"de2fab2e-177a-4eb0-9262-7bbef0d9837f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6825","nonce":4195921091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6827","nonce":4195921091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6831","nonce":4195921091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6829","nonce":4195921091}]},"public_addr":"172.21.15.165:6825/4195921091","cluster_addr":"172.21.15.165:6827/4195921091","heartbeat_back_addr":"172.21.15.165:6831/4195921091","heartbeat_front_addr":"172.21.15.165:6829/4195921091","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:30.507144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:54.456854+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:06:17.338746+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:06:42.262947+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:02.983751+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:26.594502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:50.363918+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:08:15.455798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.136:0/1450531984":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/1905186682":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/2540800865":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/1577563918":"2024-08-18T17:04:10.472088+0000","172.21.15.136:0/1414082347":"2024-08-18T17:04:10.472088+0000","172.21.15.136:0/2336022730":"2024-08-18T17:03:48.417471+0000","172.21.15.136:6800/878395364":"2024-08-18T17:03:48.417471+0000","172.21.15.136:6801/3058734149":"2024-08-18T17:04:10.472088+0000","172.21.15.136:6801/3513268997":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/3380731497":"2024-08-18T17:03:48.417471+0000","172.21.15.136:6801/878395364":"2024-08-18T17:03:48.417471+0000","172.21.15.136:6800/3058734149":"2024-08-18T17:04:10.472088+0000","172.21.15.136:0/1381030503":"2024-08-18T17:03:48.417471+0000","172.21.15.136:6800/3513268997":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/3933059683":"2024-08-18T17:04:10.472088+0000","172.21.15.136:0/2931810343":"2024-08-18T17:08:47.841029+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-17T17:10:24.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:24 smithi136 ceph-mon[31034]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-08-17T17:10:24.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:24 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/1153463586' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:10:24.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:24 smithi136 ceph-mon[42580]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-08-17T17:10:24.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:24 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/1153463586' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:10:24.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:24 smithi165 ceph-mon[34868]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-08-17T17:10:24.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:24 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/1153463586' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:10:24.767 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-08-17T17:10:24.767 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-08-17T17:10:24.768 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-08-17T17:10:24.768 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-08-17T17:10:24.768 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-08-17T17:10:24.769 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-08-17T17:10:24.769 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-08-17T17:10:24.770 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-08-17T17:10:25.390 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:10:25.391 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:10:25.394 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:10:25.395 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:10:25.396 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:10:25.403 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:10:25.458 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:10:26.447 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:10:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:10:26] "GET /metrics HTTP/1.1" 200 33422 "" "Prometheus/2.43.0" 2024-08-17T17:10:26.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:26 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:26.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:26 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:26.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:26 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:10:26.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:26 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:10:26.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:26 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:26.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:26 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:10:26.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:26 smithi136 ceph-mon[42580]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-08-17T17:10:26.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:26 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:26.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:26 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:26.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:26 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:10:26.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:26 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:10:26.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:26 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:26.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:26 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:10:26.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:26 smithi136 ceph-mon[31034]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-08-17T17:10:26.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:26 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:26.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:26 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:26.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:26 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:10:26.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:26 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:10:26.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:26 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:26.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:26 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:10:26.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:26 smithi165 ceph-mon[34868]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-08-17T17:10:29.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:28 smithi136 ceph-mon[31034]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-08-17T17:10:29.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:28 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:29.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:28 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:29.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:28 smithi136 ceph-mon[42580]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-08-17T17:10:29.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:28 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:29.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:28 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:29.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:28 smithi165 ceph-mon[34868]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-08-17T17:10:29.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:28 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:29.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:28 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:10:31.190 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:30 smithi136 ceph-mon[31034]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-08-17T17:10:31.191 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:30 smithi136 ceph-mon[42580]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-08-17T17:10:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:30 smithi165 ceph-mon[34868]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-08-17T17:10:32.190 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:31 smithi136 ceph-mon[31034]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-08-17T17:10:32.190 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:31 smithi136 ceph-mon[42580]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-08-17T17:10:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:31 smithi165 ceph-mon[34868]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-08-17T17:10:33.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:32 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:10:33.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:32 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:10:33.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:32 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:10:34.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:33 smithi136 ceph-mon[31034]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-08-17T17:10:34.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:33 smithi136 ceph-mon[42580]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-08-17T17:10:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:33 smithi165 ceph-mon[34868]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-08-17T17:10:35.687 INFO:teuthology.orchestra.run.smithi136.stdout:77309411383 2024-08-17T17:10:35.687 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-08-17T17:10:36.446 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:10:36 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:10:36] "GET /metrics HTTP/1.1" 200 33711 "" "Prometheus/2.43.0" 2024-08-17T17:10:36.625 INFO:teuthology.orchestra.run.smithi136.stdout:107374182449 2024-08-17T17:10:36.626 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-08-17T17:10:37.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:36 smithi136 ceph-mon[31034]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-08-17T17:10:37.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:36 smithi136 ceph-mon[42580]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-08-17T17:10:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:36 smithi165 ceph-mon[34868]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-08-17T17:10:37.861 INFO:teuthology.orchestra.run.smithi136.stdout:34359738432 2024-08-17T17:10:37.861 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-08-17T17:10:38.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:37 smithi136 ceph-mon[42580]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:38.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:37 smithi136 ceph-mon[31034]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:38.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:37 smithi165 ceph-mon[34868]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:38.378 INFO:teuthology.orchestra.run.smithi136.stdout:55834574907 2024-08-17T17:10:38.378 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-08-17T17:10:39.235 INFO:teuthology.orchestra.run.smithi136.stdout:128849018925 2024-08-17T17:10:39.236 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-08-17T17:10:39.598 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:10:40.111 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:10:40.112 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:10:40.216 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:10:41.034 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:40 smithi136 ceph-mon[31034]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:41.034 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:40 smithi136 ceph-mon[42580]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:41.056 INFO:teuthology.orchestra.run.smithi136.stdout:197568495653 2024-08-17T17:10:41.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-08-17T17:10:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:40 smithi165 ceph-mon[34868]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:42.185 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:41 smithi136 ceph-mon[31034]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:42.186 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:41 smithi136 ceph-mon[42580]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:41 smithi165 ceph-mon[34868]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:44.863 INFO:teuthology.orchestra.run.smithi136.stdout:231928234015 2024-08-17T17:10:44.864 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-08-17T17:10:45.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:44 smithi136 ceph-mon[42580]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:45.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:44 smithi136 ceph-mon[31034]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:45.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:44 smithi165 ceph-mon[34868]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:45.263 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:10:46.052 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:45 smithi136 ceph-mon[31034]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:46.053 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:45 smithi136 ceph-mon[42580]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:46.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:45 smithi165 ceph-mon[34868]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:46.402 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:10:46 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:10:46] "GET /metrics HTTP/1.1" 200 33711 "" "Prometheus/2.43.0" 2024-08-17T17:10:46.462 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:10:47.283 INFO:teuthology.orchestra.run.smithi136.stdout:77309411385 2024-08-17T17:10:47.362 INFO:teuthology.orchestra.run.smithi136.stdout:55834574909 2024-08-17T17:10:47.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:47 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/707202136' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-08-17T17:10:47.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:47 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/707202136' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-08-17T17:10:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:47 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/707202136' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-08-17T17:10:48.160 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574907 got 55834574909 for osd.1 2024-08-17T17:10:48.161 DEBUG:teuthology.parallel:result is None 2024-08-17T17:10:48.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:48 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/1808378314' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-08-17T17:10:48.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:48 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-08-17T17:10:48.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:48 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-08-17T17:10:48.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:48 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-08-17T17:10:48.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:48 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-08-17T17:10:48.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:48 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-08-17T17:10:48.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:48 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-08-17T17:10:48.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:48 smithi136 ceph-mon[31034]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:48.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:48 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:10:48.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:48 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/1808378314' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-08-17T17:10:48.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:48 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-08-17T17:10:48.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:48 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-08-17T17:10:48.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:48 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-08-17T17:10:48.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:48 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-08-17T17:10:48.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:48 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-08-17T17:10:48.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:48 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-08-17T17:10:48.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:48 smithi136 ceph-mon[42580]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:48.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:48 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:10:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:48 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/1808378314' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-08-17T17:10:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:48 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-08-17T17:10:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:48 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-08-17T17:10:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:48 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-08-17T17:10:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:48 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-08-17T17:10:48.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:48 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-08-17T17:10:48.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:48 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-08-17T17:10:48.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:48 smithi165 ceph-mon[34868]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:48.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:48 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:10:49.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:49 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-08-17T17:10:49.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:49 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-08-17T17:10:49.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:49 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-08-17T17:10:49.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:49 smithi136 ceph-mon[31034]: osdmap e71: 8 total, 8 up, 8 in 2024-08-17T17:10:49.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:49 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-08-17T17:10:49.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:49 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-08-17T17:10:49.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:49 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-08-17T17:10:49.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:49 smithi136 ceph-mon[42580]: osdmap e71: 8 total, 8 up, 8 in 2024-08-17T17:10:49.662 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:49 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-08-17T17:10:49.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:49 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-08-17T17:10:49.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:49 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-08-17T17:10:49.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:49 smithi165 ceph-mon[34868]: osdmap e71: 8 total, 8 up, 8 in 2024-08-17T17:10:49.966 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411383 got 77309411385 for osd.2 2024-08-17T17:10:49.967 DEBUG:teuthology.parallel:result is None 2024-08-17T17:10:50.524 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:50 smithi136 ceph-mon[31034]: osdmap e72: 8 total, 8 up, 8 in 2024-08-17T17:10:50.524 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:50 smithi136 ceph-mon[31034]: pgmap v77: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:50.525 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:50 smithi136 ceph-mon[42580]: osdmap e72: 8 total, 8 up, 8 in 2024-08-17T17:10:50.525 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:50 smithi136 ceph-mon[42580]: pgmap v77: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:50.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:50 smithi165 ceph-mon[34868]: osdmap e72: 8 total, 8 up, 8 in 2024-08-17T17:10:50.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:50 smithi165 ceph-mon[34868]: pgmap v77: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:51.376 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:10:51.377 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:10:51.552 INFO:teuthology.orchestra.run.smithi136.stdout:34359738435 2024-08-17T17:10:51.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:51 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/3167870426' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-08-17T17:10:51.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:51 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/3167870426' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-08-17T17:10:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:51 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/3167870426' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-08-17T17:10:52.568 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738432 got 34359738435 for osd.0 2024-08-17T17:10:52.568 DEBUG:teuthology.parallel:result is None 2024-08-17T17:10:52.888 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:52 smithi136 ceph-mon[31034]: pgmap v78: 73 pgs: 2 peering, 71 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:52.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:52 smithi136 ceph-mon[42580]: pgmap v78: 73 pgs: 2 peering, 71 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:52 smithi165 ceph-mon[34868]: pgmap v78: 73 pgs: 2 peering, 71 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:54.179 INFO:teuthology.orchestra.run.smithi136.stdout:163208757291 2024-08-17T17:10:54.180 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-08-17T17:10:55.163 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:54 smithi136 ceph-mon[31034]: pgmap v79: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:55.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:54 smithi136 ceph-mon[42580]: pgmap v79: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:54 smithi165 ceph-mon[34868]: pgmap v79: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:55.533 INFO:teuthology.orchestra.run.smithi136.stdout:128849018929 2024-08-17T17:10:56.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:55 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/2580812485' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-08-17T17:10:56.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:55 smithi136 ceph-mon[31034]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:56.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:55 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/2580812485' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-08-17T17:10:56.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:55 smithi136 ceph-mon[42580]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:56.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:55 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2580812485' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-08-17T17:10:56.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:55 smithi165 ceph-mon[34868]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:56.356 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018925 got 128849018929 for osd.4 2024-08-17T17:10:56.356 DEBUG:teuthology.parallel:result is None 2024-08-17T17:10:56.480 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:10:56 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:10:56] "GET /metrics HTTP/1.1" 200 33712 "" "Prometheus/2.43.0" 2024-08-17T17:10:56.958 INFO:teuthology.orchestra.run.smithi136.stdout:107374182454 2024-08-17T17:10:57.125 INFO:teuthology.orchestra.run.smithi136.stdout:231928234018 2024-08-17T17:10:57.125 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:57 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/2860341133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-08-17T17:10:57.126 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:57 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/2860341133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-08-17T17:10:57.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:57 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2860341133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-08-17T17:10:57.755 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234015 got 231928234018 for osd.7 2024-08-17T17:10:57.755 DEBUG:teuthology.parallel:result is None 2024-08-17T17:10:58.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:58 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/2736849879' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-08-17T17:10:58.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:58 smithi136 ceph-mon[31034]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:58.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:58 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/2736849879' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-08-17T17:10:58.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:58 smithi136 ceph-mon[42580]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:58.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:58 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2736849879' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-08-17T17:10:58.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:58 smithi165 ceph-mon[34868]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:10:59.582 INFO:teuthology.orchestra.run.smithi136.stdout:197568495656 2024-08-17T17:10:59.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:10:59 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/797178381' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-08-17T17:10:59.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:10:59 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/797178381' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-08-17T17:11:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:10:59 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/797178381' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-08-17T17:11:00.460 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182449 got 107374182454 for osd.3 2024-08-17T17:11:00.461 DEBUG:teuthology.parallel:result is None 2024-08-17T17:11:00.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:00 smithi136 ceph-mon[31034]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:00.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:00 smithi136 ceph-mon[42580]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:00.789 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495653 got 197568495656 for osd.6 2024-08-17T17:11:00.789 DEBUG:teuthology.parallel:result is None 2024-08-17T17:11:00.928 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:00 smithi165 ceph-mon[34868]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:01.208 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:11:02.438 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:02 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:11:02.438 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:02 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:11:02.439 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:02 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:11:02.439 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:02 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:11:02.439 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:02 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:11:02.439 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:02 smithi136 ceph-mon[31034]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:02.439 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:02 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:11:02.439 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:02 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:11:02.439 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:02 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:11:02.439 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:02 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:11:02.439 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:02 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:11:02.439 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:02 smithi136 ceph-mon[42580]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:02.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:02 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:11:02.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:02 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:11:02.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:02 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:11:02.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:02 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:11:02.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:02 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:11:02.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:02 smithi165 ceph-mon[34868]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:03.097 INFO:teuthology.orchestra.run.smithi136.stdout:163208757293 2024-08-17T17:11:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:03 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:03 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/4172969659' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-08-17T17:11:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:03 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:03 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/4172969659' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-08-17T17:11:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:03 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:03 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/4172969659' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-08-17T17:11:03.764 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757291 got 163208757293 for osd.5 2024-08-17T17:11:03.765 DEBUG:teuthology.parallel:result is None 2024-08-17T17:11:03.765 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-08-17T17:11:03.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:11:03.974 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:11:04.384 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:04 smithi136 ceph-mon[31034]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:04.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:04 smithi136 ceph-mon[42580]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:04.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:04 smithi165 ceph-mon[34868]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:05.799 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:11:05.800 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T17:11:06.161 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:05 smithi136 ceph-mon[31034]: from='client.24683 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:11:06.161 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:05 smithi136 ceph-mon[31034]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:06.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:05 smithi136 ceph-mon[42580]: from='client.24683 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:11:06.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:05 smithi136 ceph-mon[42580]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:05 smithi165 ceph-mon[34868]: from='client.24683 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:11:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:05 smithi165 ceph-mon[34868]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:06.446 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:11:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:11:06] "GET /metrics HTTP/1.1" 200 33719 "" "Prometheus/2.43.0" 2024-08-17T17:11:06.577 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-08-17T17:11:03.883775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592749,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":226,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":170,"ondisk_log_size":170,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":232036,"kb_used_data":7656,"kb_used_omap":12,"kb_used_meta":224243,"kb_avail":749729180,"statfs":{"total":767960285184,"available":767722680320,"internally_reserved":0,"allocated":7839744,"data_stored":4399945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":229625400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002552"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340955+0000","last_change":"2024-08-17T17:10:06.156584+0000","last_active":"2024-08-17T17:10:48.340955+0000","last_peered":"2024-08-17T17:10:48.340955+0000","last_clean":"2024-08-17T17:10:48.340955+0000","last_became_active":"2024-08-17T17:10:06.156252+0000","last_became_peered":"2024-08-17T17:10:06.156252+0000","last_unstale":"2024-08-17T17:10:48.340955+0000","last_undegraded":"2024-08-17T17:10:48.340955+0000","last_fullsized":"2024-08-17T17:10:48.340955+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:18:53.713960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.350239+0000","last_change":"2024-08-17T17:10:05.975682+0000","last_active":"2024-08-17T17:10:48.350239+0000","last_peered":"2024-08-17T17:10:48.350239+0000","last_clean":"2024-08-17T17:10:48.350239+0000","last_became_active":"2024-08-17T17:10:05.975420+0000","last_became_peered":"2024-08-17T17:10:05.975420+0000","last_unstale":"2024-08-17T17:10:48.350239+0000","last_undegraded":"2024-08-17T17:10:48.350239+0000","last_fullsized":"2024-08-17T17:10:48.350239+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:13:35.325266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340411+0000","last_change":"2024-08-17T17:10:07.397936+0000","last_active":"2024-08-17T17:10:48.340411+0000","last_peered":"2024-08-17T17:10:48.340411+0000","last_clean":"2024-08-17T17:10:48.340411+0000","last_became_active":"2024-08-17T17:10:07.397642+0000","last_became_peered":"2024-08-17T17:10:07.397642+0000","last_unstale":"2024-08-17T17:10:48.340411+0000","last_undegraded":"2024-08-17T17:10:48.340411+0000","last_fullsized":"2024-08-17T17:10:48.340411+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:27:27.497025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337171+0000","last_change":"2024-08-17T17:10:05.971463+0000","last_active":"2024-08-17T17:10:48.337171+0000","last_peered":"2024-08-17T17:10:48.337171+0000","last_clean":"2024-08-17T17:10:48.337171+0000","last_became_active":"2024-08-17T17:10:05.971112+0000","last_became_peered":"2024-08-17T17:10:05.971112+0000","last_unstale":"2024-08-17T17:10:48.337171+0000","last_undegraded":"2024-08-17T17:10:48.337171+0000","last_fullsized":"2024-08-17T17:10:48.337171+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:53:46.305291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337905+0000","last_change":"2024-08-17T17:10:06.157520+0000","last_active":"2024-08-17T17:10:48.337905+0000","last_peered":"2024-08-17T17:10:48.337905+0000","last_clean":"2024-08-17T17:10:48.337905+0000","last_became_active":"2024-08-17T17:10:06.157223+0000","last_became_peered":"2024-08-17T17:10:06.157223+0000","last_unstale":"2024-08-17T17:10:48.337905+0000","last_undegraded":"2024-08-17T17:10:48.337905+0000","last_fullsized":"2024-08-17T17:10:48.337905+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:37:58.194744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.350136+0000","last_change":"2024-08-17T17:10:05.975692+0000","last_active":"2024-08-17T17:10:48.350136+0000","last_peered":"2024-08-17T17:10:48.350136+0000","last_clean":"2024-08-17T17:10:48.350136+0000","last_became_active":"2024-08-17T17:10:05.975057+0000","last_became_peered":"2024-08-17T17:10:05.975057+0000","last_unstale":"2024-08-17T17:10:48.350136+0000","last_undegraded":"2024-08-17T17:10:48.350136+0000","last_fullsized":"2024-08-17T17:10:48.350136+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:44:53.106104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340968+0000","last_change":"2024-08-17T17:10:06.156570+0000","last_active":"2024-08-17T17:10:48.340968+0000","last_peered":"2024-08-17T17:10:48.340968+0000","last_clean":"2024-08-17T17:10:48.340968+0000","last_became_active":"2024-08-17T17:10:06.156267+0000","last_became_peered":"2024-08-17T17:10:06.156267+0000","last_unstale":"2024-08-17T17:10:48.340968+0000","last_undegraded":"2024-08-17T17:10:48.340968+0000","last_fullsized":"2024-08-17T17:10:48.340968+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:00:52.553271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":23,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.350096+0000","last_change":"2024-08-17T17:10:05.966582+0000","last_active":"2024-08-17T17:10:48.350096+0000","last_peered":"2024-08-17T17:10:48.350096+0000","last_clean":"2024-08-17T17:10:48.350096+0000","last_became_active":"2024-08-17T17:10:05.966324+0000","last_became_peered":"2024-08-17T17:10:05.966324+0000","last_unstale":"2024-08-17T17:10:48.350096+0000","last_undegraded":"2024-08-17T17:10:48.350096+0000","last_fullsized":"2024-08-17T17:10:48.350096+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:19:19.689962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.350525+0000","last_change":"2024-08-17T17:10:05.976856+0000","last_active":"2024-08-17T17:10:48.350525+0000","last_peered":"2024-08-17T17:10:48.350525+0000","last_clean":"2024-08-17T17:10:48.350525+0000","last_became_active":"2024-08-17T17:10:05.976575+0000","last_became_peered":"2024-08-17T17:10:05.976575+0000","last_unstale":"2024-08-17T17:10:48.350525+0000","last_undegraded":"2024-08-17T17:10:48.350525+0000","last_fullsized":"2024-08-17T17:10:48.350525+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:17:15.169128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340301+0000","last_change":"2024-08-17T17:10:06.994799+0000","last_active":"2024-08-17T17:10:48.340301+0000","last_peered":"2024-08-17T17:10:48.340301+0000","last_clean":"2024-08-17T17:10:48.340301+0000","last_became_active":"2024-08-17T17:10:06.994150+0000","last_became_peered":"2024-08-17T17:10:06.994150+0000","last_unstale":"2024-08-17T17:10:48.340301+0000","last_undegraded":"2024-08-17T17:10:48.340301+0000","last_fullsized":"2024-08-17T17:10:48.340301+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:50:47.293091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.350715+0000","last_change":"2024-08-17T17:10:05.974435+0000","last_active":"2024-08-17T17:10:48.350715+0000","last_peered":"2024-08-17T17:10:48.350715+0000","last_clean":"2024-08-17T17:10:48.350715+0000","last_became_active":"2024-08-17T17:10:05.974181+0000","last_became_peered":"2024-08-17T17:10:05.974181+0000","last_unstale":"2024-08-17T17:10:48.350715+0000","last_undegraded":"2024-08-17T17:10:48.350715+0000","last_fullsized":"2024-08-17T17:10:48.350715+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:52:43.173554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337279+0000","last_change":"2024-08-17T17:10:07.398836+0000","last_active":"2024-08-17T17:10:48.337279+0000","last_peered":"2024-08-17T17:10:48.337279+0000","last_clean":"2024-08-17T17:10:48.337279+0000","last_became_active":"2024-08-17T17:10:07.398548+0000","last_became_peered":"2024-08-17T17:10:07.398548+0000","last_unstale":"2024-08-17T17:10:48.337279+0000","last_undegraded":"2024-08-17T17:10:48.337279+0000","last_fullsized":"2024-08-17T17:10:48.337279+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:00:21.641230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.350775+0000","last_change":"2024-08-17T17:10:05.973491+0000","last_active":"2024-08-17T17:10:48.350775+0000","last_peered":"2024-08-17T17:10:48.350775+0000","last_clean":"2024-08-17T17:10:48.350775+0000","last_became_active":"2024-08-17T17:10:05.973228+0000","last_became_peered":"2024-08-17T17:10:05.973228+0000","last_unstale":"2024-08-17T17:10:48.350775+0000","last_undegraded":"2024-08-17T17:10:48.350775+0000","last_fullsized":"2024-08-17T17:10:48.350775+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:54.310135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.349990+0000","last_change":"2024-08-17T17:10:06.994297+0000","last_active":"2024-08-17T17:10:48.349990+0000","last_peered":"2024-08-17T17:10:48.349990+0000","last_clean":"2024-08-17T17:10:48.349990+0000","last_became_active":"2024-08-17T17:10:06.993979+0000","last_became_peered":"2024-08-17T17:10:06.993979+0000","last_unstale":"2024-08-17T17:10:48.349990+0000","last_undegraded":"2024-08-17T17:10:48.349990+0000","last_fullsized":"2024-08-17T17:10:48.349990+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:20:13.978272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340680+0000","last_change":"2024-08-17T17:10:05.960882+0000","last_active":"2024-08-17T17:10:48.340680+0000","last_peered":"2024-08-17T17:10:48.340680+0000","last_clean":"2024-08-17T17:10:48.340680+0000","last_became_active":"2024-08-17T17:10:05.960578+0000","last_became_peered":"2024-08-17T17:10:05.960578+0000","last_unstale":"2024-08-17T17:10:48.340680+0000","last_undegraded":"2024-08-17T17:10:48.340680+0000","last_fullsized":"2024-08-17T17:10:48.340680+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:06.791494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:10:49.342735+0000","last_change":"2024-08-17T17:10:49.342735+0000","last_active":"2024-08-17T17:10:49.342735+0000","last_peered":"2024-08-17T17:10:49.342735+0000","last_clean":"2024-08-17T17:10:49.342735+0000","last_became_active":"2024-08-17T17:10:49.342331+0000","last_became_peered":"2024-08-17T17:10:49.342331+0000","last_unstale":"2024-08-17T17:10:49.342735+0000","last_undegraded":"2024-08-17T17:10:49.342735+0000","last_fullsized":"2024-08-17T17:10:49.342735+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:56:02.469448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340461+0000","last_change":"2024-08-17T17:10:05.964640+0000","last_active":"2024-08-17T17:10:48.340461+0000","last_peered":"2024-08-17T17:10:48.340461+0000","last_clean":"2024-08-17T17:10:48.340461+0000","last_became_active":"2024-08-17T17:10:05.964085+0000","last_became_peered":"2024-08-17T17:10:05.964085+0000","last_unstale":"2024-08-17T17:10:48.340461+0000","last_undegraded":"2024-08-17T17:10:48.340461+0000","last_fullsized":"2024-08-17T17:10:48.340461+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:58:17.913779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337640+0000","last_change":"2024-08-17T17:10:07.397981+0000","last_active":"2024-08-17T17:10:48.337640+0000","last_peered":"2024-08-17T17:10:48.337640+0000","last_clean":"2024-08-17T17:10:48.337640+0000","last_became_active":"2024-08-17T17:10:07.397666+0000","last_became_peered":"2024-08-17T17:10:07.397666+0000","last_unstale":"2024-08-17T17:10:48.337640+0000","last_undegraded":"2024-08-17T17:10:48.337640+0000","last_fullsized":"2024-08-17T17:10:48.337640+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:42.103035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.350511+0000","last_change":"2024-08-17T17:10:05.966428+0000","last_active":"2024-08-17T17:10:48.350511+0000","last_peered":"2024-08-17T17:10:48.350511+0000","last_clean":"2024-08-17T17:10:48.350511+0000","last_became_active":"2024-08-17T17:10:05.966156+0000","last_became_peered":"2024-08-17T17:10:05.966156+0000","last_unstale":"2024-08-17T17:10:48.350511+0000","last_undegraded":"2024-08-17T17:10:48.350511+0000","last_fullsized":"2024-08-17T17:10:48.350511+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:32:45.657632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337977+0000","last_change":"2024-08-17T17:10:07.438535+0000","last_active":"2024-08-17T17:10:48.337977+0000","last_peered":"2024-08-17T17:10:48.337977+0000","last_clean":"2024-08-17T17:10:48.337977+0000","last_became_active":"2024-08-17T17:10:07.438245+0000","last_became_peered":"2024-08-17T17:10:07.438245+0000","last_unstale":"2024-08-17T17:10:48.337977+0000","last_undegraded":"2024-08-17T17:10:48.337977+0000","last_fullsized":"2024-08-17T17:10:48.337977+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:31:17.604018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337458+0000","last_change":"2024-08-17T17:10:05.973321+0000","last_active":"2024-08-17T17:10:48.337458+0000","last_peered":"2024-08-17T17:10:48.337458+0000","last_clean":"2024-08-17T17:10:48.337458+0000","last_became_active":"2024-08-17T17:10:05.973057+0000","last_became_peered":"2024-08-17T17:10:05.973057+0000","last_unstale":"2024-08-17T17:10:48.337458+0000","last_undegraded":"2024-08-17T17:10:48.337458+0000","last_fullsized":"2024-08-17T17:10:48.337458+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:29:50.938627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337938+0000","last_change":"2024-08-17T17:10:06.979766+0000","last_active":"2024-08-17T17:10:48.337938+0000","last_peered":"2024-08-17T17:10:48.337938+0000","last_clean":"2024-08-17T17:10:48.337938+0000","last_became_active":"2024-08-17T17:10:06.979504+0000","last_became_peered":"2024-08-17T17:10:06.979504+0000","last_unstale":"2024-08-17T17:10:48.337938+0000","last_undegraded":"2024-08-17T17:10:48.337938+0000","last_fullsized":"2024-08-17T17:10:48.337938+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:58:52.990126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340505+0000","last_change":"2024-08-17T17:10:05.969167+0000","last_active":"2024-08-17T17:10:48.340505+0000","last_peered":"2024-08-17T17:10:48.340505+0000","last_clean":"2024-08-17T17:10:48.340505+0000","last_became_active":"2024-08-17T17:10:05.968885+0000","last_became_peered":"2024-08-17T17:10:05.968885+0000","last_unstale":"2024-08-17T17:10:48.340505+0000","last_undegraded":"2024-08-17T17:10:48.340505+0000","last_fullsized":"2024-08-17T17:10:48.340505+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:56:34.762597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337299+0000","last_change":"2024-08-17T17:10:06.579964+0000","last_active":"2024-08-17T17:10:48.337299+0000","last_peered":"2024-08-17T17:10:48.337299+0000","last_clean":"2024-08-17T17:10:48.337299+0000","last_became_active":"2024-08-17T17:10:06.579633+0000","last_became_peered":"2024-08-17T17:10:06.579633+0000","last_unstale":"2024-08-17T17:10:48.337299+0000","last_undegraded":"2024-08-17T17:10:48.337299+0000","last_fullsized":"2024-08-17T17:10:48.337299+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:57:23.495634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"61'128","reported_seq":268,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.350258+0000","last_change":"2024-08-17T17:08:20.329502+0000","last_active":"2024-08-17T17:10:48.350258+0000","last_peered":"2024-08-17T17:10:48.350258+0000","last_clean":"2024-08-17T17:10:48.350258+0000","last_became_active":"2024-08-17T17:08:20.329063+0000","last_became_peered":"2024-08-17T17:08:20.329063+0000","last_unstale":"2024-08-17T17:10:48.350258+0000","last_undegraded":"2024-08-17T17:10:48.350258+0000","last_fullsized":"2024-08-17T17:10:48.350258+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T17:06:21.174471+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:52:20.712819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":29,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:10:49.341961+0000","last_change":"2024-08-17T17:08:53.943617+0000","last_active":"2024-08-17T17:10:49.341961+0000","last_peered":"2024-08-17T17:10:49.341961+0000","last_clean":"2024-08-17T17:10:49.341961+0000","last_became_active":"2024-08-17T17:08:48.854004+0000","last_became_peered":"2024-08-17T17:08:48.854004+0000","last_unstale":"2024-08-17T17:10:49.341961+0000","last_undegraded":"2024-08-17T17:10:49.341961+0000","last_fullsized":"2024-08-17T17:10:49.341961+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:53:56.669365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041737399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340294+0000","last_change":"2024-08-17T17:10:05.960101+0000","last_active":"2024-08-17T17:10:48.340294+0000","last_peered":"2024-08-17T17:10:48.340294+0000","last_clean":"2024-08-17T17:10:48.340294+0000","last_became_active":"2024-08-17T17:10:05.959775+0000","last_became_peered":"2024-08-17T17:10:05.959775+0000","last_unstale":"2024-08-17T17:10:48.340294+0000","last_undegraded":"2024-08-17T17:10:48.340294+0000","last_fullsized":"2024-08-17T17:10:48.340294+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:03:19.894065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.350586+0000","last_change":"2024-08-17T17:10:06.995023+0000","last_active":"2024-08-17T17:10:48.350586+0000","last_peered":"2024-08-17T17:10:48.350586+0000","last_clean":"2024-08-17T17:10:48.350586+0000","last_became_active":"2024-08-17T17:10:06.994757+0000","last_became_peered":"2024-08-17T17:10:06.994757+0000","last_unstale":"2024-08-17T17:10:48.350586+0000","last_undegraded":"2024-08-17T17:10:48.350586+0000","last_fullsized":"2024-08-17T17:10:48.350586+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:31:52.163518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":28,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.876366+0000","last_change":"2024-08-17T17:08:54.068863+0000","last_active":"2024-08-17T17:10:48.876366+0000","last_peered":"2024-08-17T17:10:48.876366+0000","last_clean":"2024-08-17T17:10:48.876366+0000","last_became_active":"2024-08-17T17:08:48.854469+0000","last_became_peered":"2024-08-17T17:08:48.854469+0000","last_unstale":"2024-08-17T17:10:48.876366+0000","last_undegraded":"2024-08-17T17:10:48.876366+0000","last_fullsized":"2024-08-17T17:10:48.876366+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:32:44.509162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337035+0000","last_change":"2024-08-17T17:10:05.968146+0000","last_active":"2024-08-17T17:10:48.337035+0000","last_peered":"2024-08-17T17:10:48.337035+0000","last_clean":"2024-08-17T17:10:48.337035+0000","last_became_active":"2024-08-17T17:10:05.967911+0000","last_became_peered":"2024-08-17T17:10:05.967911+0000","last_unstale":"2024-08-17T17:10:48.337035+0000","last_undegraded":"2024-08-17T17:10:48.337035+0000","last_fullsized":"2024-08-17T17:10:48.337035+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:26:04.782748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337847+0000","last_change":"2024-08-17T17:10:05.988330+0000","last_active":"2024-08-17T17:10:48.337847+0000","last_peered":"2024-08-17T17:10:48.337847+0000","last_clean":"2024-08-17T17:10:48.337847+0000","last_became_active":"2024-08-17T17:10:05.988015+0000","last_became_peered":"2024-08-17T17:10:05.988015+0000","last_unstale":"2024-08-17T17:10:48.337847+0000","last_undegraded":"2024-08-17T17:10:48.337847+0000","last_fullsized":"2024-08-17T17:10:48.337847+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:01:44.304295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":28,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:10:49.340343+0000","last_change":"2024-08-17T17:08:53.939867+0000","last_active":"2024-08-17T17:10:49.340343+0000","last_peered":"2024-08-17T17:10:49.340343+0000","last_clean":"2024-08-17T17:10:49.340343+0000","last_became_active":"2024-08-17T17:08:48.927330+0000","last_became_peered":"2024-08-17T17:08:48.927330+0000","last_unstale":"2024-08-17T17:10:49.340343+0000","last_undegraded":"2024-08-17T17:10:49.340343+0000","last_fullsized":"2024-08-17T17:10:49.340343+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:03:20.888081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.336756+0000","last_change":"2024-08-17T17:10:05.960071+0000","last_active":"2024-08-17T17:10:48.336756+0000","last_peered":"2024-08-17T17:10:48.336756+0000","last_clean":"2024-08-17T17:10:48.336756+0000","last_became_active":"2024-08-17T17:10:05.959830+0000","last_became_peered":"2024-08-17T17:10:05.959830+0000","last_unstale":"2024-08-17T17:10:48.336756+0000","last_undegraded":"2024-08-17T17:10:48.336756+0000","last_fullsized":"2024-08-17T17:10:48.336756+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:26:49.379433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337370+0000","last_change":"2024-08-17T17:10:05.984255+0000","last_active":"2024-08-17T17:10:48.337370+0000","last_peered":"2024-08-17T17:10:48.337370+0000","last_clean":"2024-08-17T17:10:48.337370+0000","last_became_active":"2024-08-17T17:10:05.983889+0000","last_became_peered":"2024-08-17T17:10:05.983889+0000","last_unstale":"2024-08-17T17:10:48.337370+0000","last_undegraded":"2024-08-17T17:10:48.337370+0000","last_fullsized":"2024-08-17T17:10:48.337370+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:50:29.637480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":30,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:10:49.341885+0000","last_change":"2024-08-17T17:08:53.944227+0000","last_active":"2024-08-17T17:10:49.341885+0000","last_peered":"2024-08-17T17:10:49.341885+0000","last_clean":"2024-08-17T17:10:49.341885+0000","last_became_active":"2024-08-17T17:08:48.854463+0000","last_became_peered":"2024-08-17T17:08:48.854463+0000","last_unstale":"2024-08-17T17:10:49.341885+0000","last_undegraded":"2024-08-17T17:10:49.341885+0000","last_fullsized":"2024-08-17T17:10:49.341885+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:06:41.281670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001023532,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.338060+0000","last_change":"2024-08-17T17:10:05.974383+0000","last_active":"2024-08-17T17:10:48.338060+0000","last_peered":"2024-08-17T17:10:48.338060+0000","last_clean":"2024-08-17T17:10:48.338060+0000","last_became_active":"2024-08-17T17:10:05.974059+0000","last_became_peered":"2024-08-17T17:10:05.974059+0000","last_unstale":"2024-08-17T17:10:48.338060+0000","last_undegraded":"2024-08-17T17:10:48.338060+0000","last_fullsized":"2024-08-17T17:10:48.338060+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:22:30.316285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:10:49.344184+0000","last_change":"2024-08-17T17:10:49.344184+0000","last_active":"2024-08-17T17:10:49.344184+0000","last_peered":"2024-08-17T17:10:49.344184+0000","last_clean":"2024-08-17T17:10:49.344184+0000","last_became_active":"2024-08-17T17:10:49.343838+0000","last_became_peered":"2024-08-17T17:10:49.343838+0000","last_unstale":"2024-08-17T17:10:49.344184+0000","last_undegraded":"2024-08-17T17:10:49.344184+0000","last_fullsized":"2024-08-17T17:10:49.344184+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:51:30.332750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340612+0000","last_change":"2024-08-17T17:10:05.960285+0000","last_active":"2024-08-17T17:10:48.340612+0000","last_peered":"2024-08-17T17:10:48.340612+0000","last_clean":"2024-08-17T17:10:48.340612+0000","last_became_active":"2024-08-17T17:10:05.959910+0000","last_became_peered":"2024-08-17T17:10:05.959910+0000","last_unstale":"2024-08-17T17:10:48.340612+0000","last_undegraded":"2024-08-17T17:10:48.340612+0000","last_fullsized":"2024-08-17T17:10:48.340612+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:15:55.327175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":28,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:10:49.342164+0000","last_change":"2024-08-17T17:08:53.939956+0000","last_active":"2024-08-17T17:10:49.342164+0000","last_peered":"2024-08-17T17:10:49.342164+0000","last_clean":"2024-08-17T17:10:49.342164+0000","last_became_active":"2024-08-17T17:08:48.851812+0000","last_became_peered":"2024-08-17T17:08:48.851812+0000","last_unstale":"2024-08-17T17:10:49.342164+0000","last_undegraded":"2024-08-17T17:10:49.342164+0000","last_fullsized":"2024-08-17T17:10:49.342164+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:25:20.326625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.336917+0000","last_change":"2024-08-17T17:10:06.579900+0000","last_active":"2024-08-17T17:10:48.336917+0000","last_peered":"2024-08-17T17:10:48.336917+0000","last_clean":"2024-08-17T17:10:48.336917+0000","last_became_active":"2024-08-17T17:10:06.579567+0000","last_became_peered":"2024-08-17T17:10:06.579567+0000","last_unstale":"2024-08-17T17:10:48.336917+0000","last_undegraded":"2024-08-17T17:10:48.336917+0000","last_fullsized":"2024-08-17T17:10:48.336917+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:29:50.641345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337807+0000","last_change":"2024-08-17T17:10:05.959878+0000","last_active":"2024-08-17T17:10:48.337807+0000","last_peered":"2024-08-17T17:10:48.337807+0000","last_clean":"2024-08-17T17:10:48.337807+0000","last_became_active":"2024-08-17T17:10:05.959595+0000","last_became_peered":"2024-08-17T17:10:05.959595+0000","last_unstale":"2024-08-17T17:10:48.337807+0000","last_undegraded":"2024-08-17T17:10:48.337807+0000","last_fullsized":"2024-08-17T17:10:48.337807+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:06:09.585988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":28,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:10:50.041355+0000","last_change":"2024-08-17T17:08:54.068835+0000","last_active":"2024-08-17T17:10:50.041355+0000","last_peered":"2024-08-17T17:10:50.041355+0000","last_clean":"2024-08-17T17:10:50.041355+0000","last_became_active":"2024-08-17T17:08:48.853784+0000","last_became_peered":"2024-08-17T17:08:48.853784+0000","last_unstale":"2024-08-17T17:10:50.041355+0000","last_undegraded":"2024-08-17T17:10:50.041355+0000","last_fullsized":"2024-08-17T17:10:50.041355+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:43:23.734272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337971+0000","last_change":"2024-08-17T17:10:06.156581+0000","last_active":"2024-08-17T17:10:48.337971+0000","last_peered":"2024-08-17T17:10:48.337971+0000","last_clean":"2024-08-17T17:10:48.337971+0000","last_became_active":"2024-08-17T17:10:06.156260+0000","last_became_peered":"2024-08-17T17:10:06.156260+0000","last_unstale":"2024-08-17T17:10:48.337971+0000","last_undegraded":"2024-08-17T17:10:48.337971+0000","last_fullsized":"2024-08-17T17:10:48.337971+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:32.420918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.341068+0000","last_change":"2024-08-17T17:10:05.968996+0000","last_active":"2024-08-17T17:10:48.341068+0000","last_peered":"2024-08-17T17:10:48.341068+0000","last_clean":"2024-08-17T17:10:48.341068+0000","last_became_active":"2024-08-17T17:10:05.968724+0000","last_became_peered":"2024-08-17T17:10:05.968724+0000","last_unstale":"2024-08-17T17:10:48.341068+0000","last_undegraded":"2024-08-17T17:10:48.341068+0000","last_fullsized":"2024-08-17T17:10:48.341068+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:50:37.286495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":28,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:10:49.341047+0000","last_change":"2024-08-17T17:08:53.939970+0000","last_active":"2024-08-17T17:10:49.341047+0000","last_peered":"2024-08-17T17:10:49.341047+0000","last_clean":"2024-08-17T17:10:49.341047+0000","last_became_active":"2024-08-17T17:08:48.852650+0000","last_became_peered":"2024-08-17T17:08:48.852650+0000","last_unstale":"2024-08-17T17:10:49.341047+0000","last_undegraded":"2024-08-17T17:10:49.341047+0000","last_fullsized":"2024-08-17T17:10:49.341047+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:40.181438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000256001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340249+0000","last_change":"2024-08-17T17:10:06.156505+0000","last_active":"2024-08-17T17:10:48.340249+0000","last_peered":"2024-08-17T17:10:48.340249+0000","last_clean":"2024-08-17T17:10:48.340249+0000","last_became_active":"2024-08-17T17:10:06.156197+0000","last_became_peered":"2024-08-17T17:10:06.156197+0000","last_unstale":"2024-08-17T17:10:48.340249+0000","last_undegraded":"2024-08-17T17:10:48.340249+0000","last_fullsized":"2024-08-17T17:10:48.340249+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:11:12.864042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337978+0000","last_change":"2024-08-17T17:10:05.968221+0000","last_active":"2024-08-17T17:10:48.337978+0000","last_peered":"2024-08-17T17:10:48.337978+0000","last_clean":"2024-08-17T17:10:48.337978+0000","last_became_active":"2024-08-17T17:10:05.967961+0000","last_became_peered":"2024-08-17T17:10:05.967961+0000","last_unstale":"2024-08-17T17:10:48.337978+0000","last_undegraded":"2024-08-17T17:10:48.337978+0000","last_fullsized":"2024-08-17T17:10:48.337978+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:24:58.119474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":28,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:10:50.041740+0000","last_change":"2024-08-17T17:08:53.945206+0000","last_active":"2024-08-17T17:10:50.041740+0000","last_peered":"2024-08-17T17:10:50.041740+0000","last_clean":"2024-08-17T17:10:50.041740+0000","last_became_active":"2024-08-17T17:08:48.853366+0000","last_became_peered":"2024-08-17T17:08:48.853366+0000","last_unstale":"2024-08-17T17:10:50.041740+0000","last_undegraded":"2024-08-17T17:10:50.041740+0000","last_fullsized":"2024-08-17T17:10:50.041740+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:34:32.938937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337410+0000","last_change":"2024-08-17T17:10:06.979492+0000","last_active":"2024-08-17T17:10:48.337410+0000","last_peered":"2024-08-17T17:10:48.337410+0000","last_clean":"2024-08-17T17:10:48.337410+0000","last_became_active":"2024-08-17T17:10:06.979158+0000","last_became_peered":"2024-08-17T17:10:06.979158+0000","last_unstale":"2024-08-17T17:10:48.337410+0000","last_undegraded":"2024-08-17T17:10:48.337410+0000","last_fullsized":"2024-08-17T17:10:48.337410+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:12:42.261365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337923+0000","last_change":"2024-08-17T17:10:06.979662+0000","last_active":"2024-08-17T17:10:48.337923+0000","last_peered":"2024-08-17T17:10:48.337923+0000","last_clean":"2024-08-17T17:10:48.337923+0000","last_became_active":"2024-08-17T17:10:06.979377+0000","last_became_peered":"2024-08-17T17:10:06.979377+0000","last_unstale":"2024-08-17T17:10:48.337923+0000","last_undegraded":"2024-08-17T17:10:48.337923+0000","last_fullsized":"2024-08-17T17:10:48.337923+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:29:06.898927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"70'6","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337732+0000","last_change":"2024-08-17T17:10:05.969131+0000","last_active":"2024-08-17T17:10:48.337732+0000","last_peered":"2024-08-17T17:10:48.337732+0000","last_clean":"2024-08-17T17:10:48.337732+0000","last_became_active":"2024-08-17T17:10:05.968856+0000","last_became_peered":"2024-08-17T17:10:05.968856+0000","last_unstale":"2024-08-17T17:10:48.337732+0000","last_undegraded":"2024-08-17T17:10:48.337732+0000","last_fullsized":"2024-08-17T17:10:48.337732+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:25:49.550065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.336840+0000","last_change":"2024-08-17T17:10:06.087073+0000","last_active":"2024-08-17T17:10:48.336840+0000","last_peered":"2024-08-17T17:10:48.336840+0000","last_clean":"2024-08-17T17:10:48.336840+0000","last_became_active":"2024-08-17T17:10:06.086742+0000","last_became_peered":"2024-08-17T17:10:06.086742+0000","last_unstale":"2024-08-17T17:10:48.336840+0000","last_undegraded":"2024-08-17T17:10:48.336840+0000","last_fullsized":"2024-08-17T17:10:48.336840+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:13:04.339226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":18,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.350729+0000","last_change":"2024-08-17T17:10:05.974976+0000","last_active":"2024-08-17T17:10:48.350729+0000","last_peered":"2024-08-17T17:10:48.350729+0000","last_clean":"2024-08-17T17:10:48.350729+0000","last_became_active":"2024-08-17T17:10:05.974726+0000","last_became_peered":"2024-08-17T17:10:05.974726+0000","last_unstale":"2024-08-17T17:10:48.350729+0000","last_undegraded":"2024-08-17T17:10:48.350729+0000","last_fullsized":"2024-08-17T17:10:48.350729+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:26:54.318334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340492+0000","last_change":"2024-08-17T17:10:05.966061+0000","last_active":"2024-08-17T17:10:48.340492+0000","last_peered":"2024-08-17T17:10:48.340492+0000","last_clean":"2024-08-17T17:10:48.340492+0000","last_became_active":"2024-08-17T17:10:05.965796+0000","last_became_peered":"2024-08-17T17:10:05.965796+0000","last_unstale":"2024-08-17T17:10:48.340492+0000","last_undegraded":"2024-08-17T17:10:48.340492+0000","last_fullsized":"2024-08-17T17:10:48.340492+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:57:12.028490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340716+0000","last_change":"2024-08-17T17:10:06.157618+0000","last_active":"2024-08-17T17:10:48.340716+0000","last_peered":"2024-08-17T17:10:48.340716+0000","last_clean":"2024-08-17T17:10:48.340716+0000","last_became_active":"2024-08-17T17:10:06.156724+0000","last_became_peered":"2024-08-17T17:10:06.156724+0000","last_unstale":"2024-08-17T17:10:48.340716+0000","last_undegraded":"2024-08-17T17:10:48.340716+0000","last_fullsized":"2024-08-17T17:10:48.340716+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:35:46.294434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337618+0000","last_change":"2024-08-17T17:10:05.965210+0000","last_active":"2024-08-17T17:10:48.337618+0000","last_peered":"2024-08-17T17:10:48.337618+0000","last_clean":"2024-08-17T17:10:48.337618+0000","last_became_active":"2024-08-17T17:10:05.964956+0000","last_became_peered":"2024-08-17T17:10:05.964956+0000","last_unstale":"2024-08-17T17:10:48.337618+0000","last_undegraded":"2024-08-17T17:10:48.337618+0000","last_fullsized":"2024-08-17T17:10:48.337618+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:42:36.519168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340160+0000","last_change":"2024-08-17T17:10:06.087172+0000","last_active":"2024-08-17T17:10:48.340160+0000","last_peered":"2024-08-17T17:10:48.340160+0000","last_clean":"2024-08-17T17:10:48.340160+0000","last_became_active":"2024-08-17T17:10:06.086840+0000","last_became_peered":"2024-08-17T17:10:06.086840+0000","last_unstale":"2024-08-17T17:10:48.340160+0000","last_undegraded":"2024-08-17T17:10:48.340160+0000","last_fullsized":"2024-08-17T17:10:48.340160+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:39:42.946969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.350321+0000","last_change":"2024-08-17T17:10:06.978485+0000","last_active":"2024-08-17T17:10:48.350321+0000","last_peered":"2024-08-17T17:10:48.350321+0000","last_clean":"2024-08-17T17:10:48.350321+0000","last_became_active":"2024-08-17T17:10:06.978145+0000","last_became_peered":"2024-08-17T17:10:06.978145+0000","last_unstale":"2024-08-17T17:10:48.350321+0000","last_undegraded":"2024-08-17T17:10:48.350321+0000","last_fullsized":"2024-08-17T17:10:48.350321+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:59.241653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340398+0000","last_change":"2024-08-17T17:10:05.962483+0000","last_active":"2024-08-17T17:10:48.340398+0000","last_peered":"2024-08-17T17:10:48.340398+0000","last_clean":"2024-08-17T17:10:48.340398+0000","last_became_active":"2024-08-17T17:10:05.962216+0000","last_became_peered":"2024-08-17T17:10:05.962216+0000","last_unstale":"2024-08-17T17:10:48.340398+0000","last_undegraded":"2024-08-17T17:10:48.340398+0000","last_fullsized":"2024-08-17T17:10:48.340398+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:31:59.156020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337786+0000","last_change":"2024-08-17T17:10:06.157231+0000","last_active":"2024-08-17T17:10:48.337786+0000","last_peered":"2024-08-17T17:10:48.337786+0000","last_clean":"2024-08-17T17:10:48.337786+0000","last_became_active":"2024-08-17T17:10:06.156943+0000","last_became_peered":"2024-08-17T17:10:06.156943+0000","last_unstale":"2024-08-17T17:10:48.337786+0000","last_undegraded":"2024-08-17T17:10:48.337786+0000","last_fullsized":"2024-08-17T17:10:48.337786+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:51:05.141785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.336808+0000","last_change":"2024-08-17T17:10:05.967087+0000","last_active":"2024-08-17T17:10:48.336808+0000","last_peered":"2024-08-17T17:10:48.336808+0000","last_clean":"2024-08-17T17:10:48.336808+0000","last_became_active":"2024-08-17T17:10:05.966793+0000","last_became_peered":"2024-08-17T17:10:05.966793+0000","last_unstale":"2024-08-17T17:10:48.336808+0000","last_undegraded":"2024-08-17T17:10:48.336808+0000","last_fullsized":"2024-08-17T17:10:48.336808+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:39.118660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:10:49.346039+0000","last_change":"2024-08-17T17:10:49.346039+0000","last_active":"2024-08-17T17:10:49.346039+0000","last_peered":"2024-08-17T17:10:49.346039+0000","last_clean":"2024-08-17T17:10:49.346039+0000","last_became_active":"2024-08-17T17:10:49.345706+0000","last_became_peered":"2024-08-17T17:10:49.345706+0000","last_unstale":"2024-08-17T17:10:49.346039+0000","last_undegraded":"2024-08-17T17:10:49.346039+0000","last_fullsized":"2024-08-17T17:10:49.346039+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:26:19.982527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337502+0000","last_change":"2024-08-17T17:10:05.964796+0000","last_active":"2024-08-17T17:10:48.337502+0000","last_peered":"2024-08-17T17:10:48.337502+0000","last_clean":"2024-08-17T17:10:48.337502+0000","last_became_active":"2024-08-17T17:10:05.964388+0000","last_became_peered":"2024-08-17T17:10:05.964388+0000","last_unstale":"2024-08-17T17:10:48.337502+0000","last_undegraded":"2024-08-17T17:10:48.337502+0000","last_fullsized":"2024-08-17T17:10:48.337502+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:20:06.478705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.339754+0000","last_change":"2024-08-17T17:10:05.984528+0000","last_active":"2024-08-17T17:10:48.339754+0000","last_peered":"2024-08-17T17:10:48.339754+0000","last_clean":"2024-08-17T17:10:48.339754+0000","last_became_active":"2024-08-17T17:10:05.984249+0000","last_became_peered":"2024-08-17T17:10:05.984249+0000","last_unstale":"2024-08-17T17:10:48.339754+0000","last_undegraded":"2024-08-17T17:10:48.339754+0000","last_fullsized":"2024-08-17T17:10:48.339754+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:52:21.938100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.338092+0000","last_change":"2024-08-17T17:10:05.971396+0000","last_active":"2024-08-17T17:10:48.338092+0000","last_peered":"2024-08-17T17:10:48.338092+0000","last_clean":"2024-08-17T17:10:48.338092+0000","last_became_active":"2024-08-17T17:10:05.971087+0000","last_became_peered":"2024-08-17T17:10:05.971087+0000","last_unstale":"2024-08-17T17:10:48.338092+0000","last_undegraded":"2024-08-17T17:10:48.338092+0000","last_fullsized":"2024-08-17T17:10:48.338092+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:32:36.084786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337823+0000","last_change":"2024-08-17T17:10:07.398001+0000","last_active":"2024-08-17T17:10:48.337823+0000","last_peered":"2024-08-17T17:10:48.337823+0000","last_clean":"2024-08-17T17:10:48.337823+0000","last_became_active":"2024-08-17T17:10:07.397733+0000","last_became_peered":"2024-08-17T17:10:07.397733+0000","last_unstale":"2024-08-17T17:10:48.337823+0000","last_undegraded":"2024-08-17T17:10:48.337823+0000","last_fullsized":"2024-08-17T17:10:48.337823+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:39:06.988794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340465+0000","last_change":"2024-08-17T17:10:05.964931+0000","last_active":"2024-08-17T17:10:48.340465+0000","last_peered":"2024-08-17T17:10:48.340465+0000","last_clean":"2024-08-17T17:10:48.340465+0000","last_became_active":"2024-08-17T17:10:05.964647+0000","last_became_peered":"2024-08-17T17:10:05.964647+0000","last_unstale":"2024-08-17T17:10:48.340465+0000","last_undegraded":"2024-08-17T17:10:48.340465+0000","last_fullsized":"2024-08-17T17:10:48.340465+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:53:14.162545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337416+0000","last_change":"2024-08-17T17:10:07.398175+0000","last_active":"2024-08-17T17:10:48.337416+0000","last_peered":"2024-08-17T17:10:48.337416+0000","last_clean":"2024-08-17T17:10:48.337416+0000","last_became_active":"2024-08-17T17:10:07.397596+0000","last_became_peered":"2024-08-17T17:10:07.397596+0000","last_unstale":"2024-08-17T17:10:48.337416+0000","last_undegraded":"2024-08-17T17:10:48.337416+0000","last_fullsized":"2024-08-17T17:10:48.337416+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:37.039485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340587+0000","last_change":"2024-08-17T17:10:05.965176+0000","last_active":"2024-08-17T17:10:48.340587+0000","last_peered":"2024-08-17T17:10:48.340587+0000","last_clean":"2024-08-17T17:10:48.340587+0000","last_became_active":"2024-08-17T17:10:05.964861+0000","last_became_peered":"2024-08-17T17:10:05.964861+0000","last_unstale":"2024-08-17T17:10:48.340587+0000","last_undegraded":"2024-08-17T17:10:48.340587+0000","last_fullsized":"2024-08-17T17:10:48.340587+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:59:04.277926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340612+0000","last_change":"2024-08-17T17:10:05.985507+0000","last_active":"2024-08-17T17:10:48.340612+0000","last_peered":"2024-08-17T17:10:48.340612+0000","last_clean":"2024-08-17T17:10:48.340612+0000","last_became_active":"2024-08-17T17:10:05.985191+0000","last_became_peered":"2024-08-17T17:10:05.985191+0000","last_unstale":"2024-08-17T17:10:48.340612+0000","last_undegraded":"2024-08-17T17:10:48.340612+0000","last_fullsized":"2024-08-17T17:10:48.340612+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:59.000088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340790+0000","last_change":"2024-08-17T17:10:05.959340+0000","last_active":"2024-08-17T17:10:48.340790+0000","last_peered":"2024-08-17T17:10:48.340790+0000","last_clean":"2024-08-17T17:10:48.340790+0000","last_became_active":"2024-08-17T17:10:05.959075+0000","last_became_peered":"2024-08-17T17:10:05.959075+0000","last_unstale":"2024-08-17T17:10:48.340790+0000","last_undegraded":"2024-08-17T17:10:48.340790+0000","last_fullsized":"2024-08-17T17:10:48.340790+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:20:46.147656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337278+0000","last_change":"2024-08-17T17:10:06.978790+0000","last_active":"2024-08-17T17:10:48.337278+0000","last_peered":"2024-08-17T17:10:48.337278+0000","last_clean":"2024-08-17T17:10:48.337278+0000","last_became_active":"2024-08-17T17:10:06.978497+0000","last_became_peered":"2024-08-17T17:10:06.978497+0000","last_unstale":"2024-08-17T17:10:48.337278+0000","last_undegraded":"2024-08-17T17:10:48.337278+0000","last_fullsized":"2024-08-17T17:10:48.337278+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:07:20.340330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":19,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337546+0000","last_change":"2024-08-17T17:10:05.968779+0000","last_active":"2024-08-17T17:10:48.337546+0000","last_peered":"2024-08-17T17:10:48.337546+0000","last_clean":"2024-08-17T17:10:48.337546+0000","last_became_active":"2024-08-17T17:10:05.968518+0000","last_became_peered":"2024-08-17T17:10:05.968518+0000","last_unstale":"2024-08-17T17:10:48.337546+0000","last_undegraded":"2024-08-17T17:10:48.337546+0000","last_fullsized":"2024-08-17T17:10:48.337546+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:52.221246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234020,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28356,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716796,"statfs":{"total":95995035648,"available":95965999104,"internally_reserved":0,"allocated":1363968,"data_stored":919726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:10:19 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.52400000000000002,"15min":0.52800000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67800000000000005,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.443},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53100000000000003,"15min":0.53300000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.66300000000000003,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.54200000000000004}]},{"osd":1,"last update":"Sat Aug 17 17:10:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.50700000000000001,"15min":0.505},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66800000000000004,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.55100000000000005,"15min":0.55300000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.68400000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.47399999999999998}]},{"osd":2,"last update":"Sat Aug 17 17:10:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52900000000000003,"15min":0.53200000000000003},"min":{"1min":0.371,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.65300000000000002,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.51800000000000002}]},{"osd":3,"last update":"Sat Aug 17 17:10:19 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52100000000000002,"15min":0.52300000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70599999999999996,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.49099999999999999}]},{"osd":4,"last update":"Sat Aug 17 17:10:19 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.35799999999999998}]},{"osd":5,"last update":"Sat Aug 17 17:10:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.48799999999999999,"15min":0.48399999999999999},"min":{"1min":0.33700000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.318},{"interface":"front","average":{"1min":0.495,"5min":0.52000000000000002,"15min":0.52400000000000002},"min":{"1min":0.34100000000000003,"5min":0.318,"15min":0.318},"max":{"1min":0.71899999999999997,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.40400000000000003}]},{"osd":6,"last update":"Sat Aug 17 17:10:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54400000000000004,"15min":0.54700000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.79100000000000004,"5min":1.024,"15min":1.024},"last":0.374},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.52700000000000002,"15min":0.52600000000000002},"min":{"1min":0.35599999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72099999999999997,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.39300000000000002}]}]},{"osd":6,"up_from":46,"seq":197568495658,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28344,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716808,"statfs":{"total":95995035648,"available":95966011392,"internally_reserved":0,"allocated":1355776,"data_stored":919599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:10:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52800000000000002,"15min":0.52900000000000003},"min":{"1min":0.372,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52800000000000002,"15min":0.53000000000000003},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.54900000000000004}]},{"osd":1,"last update":"Sat Aug 17 17:10:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.51900000000000002,"15min":0.52900000000000003},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.751,"5min":0.755,"15min":0.755},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54300000000000004,"15min":0.54900000000000004},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.72799999999999998}]},{"osd":2,"last update":"Sat Aug 17 17:10:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.76200000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54800000000000004,"15min":0.55300000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.747,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.57699999999999996}]},{"osd":3,"last update":"Sat Aug 17 17:10:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53900000000000003,"15min":0.54000000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.67200000000000004,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.65900000000000003}]},{"osd":4,"last update":"Sat Aug 17 17:10:54 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.41399999999999998,"15min":0.40400000000000003},"min":{"1min":0.29499999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.60399999999999998,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.497},{"interface":"front","average":{"1min":0.442,"5min":0.41399999999999998,"15min":0.40600000000000003},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.55400000000000005,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.443}]},{"osd":5,"last update":"Sat Aug 17 17:10:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.432,"15min":0.41899999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.65700000000000003,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.435,"15min":0.42499999999999999},"min":{"1min":0.314,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.61699999999999999,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.47999999999999998}]},{"osd":7,"last update":"Sat Aug 17 17:10:17 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33500000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.499,"15min":0.497},"min":{"1min":0.38800000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.52400000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757294,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27760,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":753664,"data_stored":328199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:10:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52900000000000003,"15min":0.52700000000000002},"min":{"1min":0.33900000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76800000000000002,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51800000000000002,"15min":0.52200000000000002},"min":{"1min":0.38200000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.63200000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.41799999999999998}]},{"osd":1,"last update":"Sat Aug 17 17:10:32 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.52700000000000002,"15min":0.53700000000000003},"min":{"1min":0.41499999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.495,"15min":0.49399999999999999},"min":{"1min":0.35599999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71799999999999997,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.32100000000000001}]},{"osd":2,"last update":"Sat Aug 17 17:10:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.70599999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.439},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52900000000000003,"15min":0.53300000000000003},"min":{"1min":0.38200000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.68000000000000005,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.49399999999999999}]},{"osd":3,"last update":"Sat Aug 17 17:10:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51400000000000001,"15min":0.51700000000000002},"min":{"1min":0.40100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72299999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55200000000000005,"15min":0.55100000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.73699999999999999,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.55800000000000005}]},{"osd":4,"last update":"Sat Aug 17 17:10:32 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40200000000000002,"15min":0.39600000000000002},"min":{"1min":0.312,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.432,"15min":0.436},"min":{"1min":0.28999999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.627,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.61499999999999999}]},{"osd":6,"last update":"Sat Aug 17 17:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.45000000000000001,"15min":0.45100000000000001},"min":{"1min":0.314,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.499,"15min":0.496},"min":{"1min":0.35299999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.71099999999999997,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.58899999999999997}]},{"osd":7,"last update":"Sat Aug 17 17:10:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.51200000000000001,"15min":0.50600000000000001},"min":{"1min":0.39300000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.66400000000000003,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.433,"15min":0.434},"min":{"1min":0.315,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.59899999999999998,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.51300000000000001}]}]},{"osd":4,"up_from":30,"seq":128849018931,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31872,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713280,"statfs":{"total":95995035648,"available":95962398720,"internally_reserved":0,"allocated":770048,"data_stored":329569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:10:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51200000000000001,"15min":0.50900000000000001},"min":{"1min":0.41599999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.71399999999999997,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.33700000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.77200000000000002,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.43099999999999999}]},{"osd":1,"last update":"Sat Aug 17 17:10:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.50600000000000001,"15min":0.503},"min":{"1min":0.28499999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68200000000000005,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.39800000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73899999999999999,"5min":0.751,"15min":0.751},"last":0.59999999999999998}]},{"osd":2,"last update":"Sat Aug 17 17:10:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54800000000000004,"15min":0.55200000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.78800000000000003,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.52400000000000002,"15min":0.52200000000000002},"min":{"1min":0.38300000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.505}]},{"osd":3,"last update":"Sat Aug 17 17:10:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.371,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53100000000000003,"15min":0.53000000000000003},"min":{"1min":0.35899999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.46100000000000002}]},{"osd":5,"last update":"Sat Aug 17 17:10:35 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.38700000000000001,"15min":0.371},"min":{"1min":0.249,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66300000000000003,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.41099999999999998,"15min":0.38900000000000001},"min":{"1min":0.28799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.68999999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.54300000000000004}]},{"osd":6,"last update":"Sat Aug 17 17:10:58 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.46500000000000002,"15min":0.443},"min":{"1min":0.32400000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.67600000000000005,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.46899999999999997,"15min":0.45100000000000001},"min":{"1min":0.309,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67700000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.44800000000000001}]},{"osd":7,"last update":"Sat Aug 17 17:10:24 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47199999999999998,"15min":0.46800000000000003},"min":{"1min":0.31900000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71299999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.46700000000000003,"15min":0.46300000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70199999999999996,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.442}]}]},{"osd":3,"up_from":25,"seq":107374182456,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27752,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717400,"statfs":{"total":95995035648,"available":95966617600,"internally_reserved":0,"allocated":745472,"data_stored":327668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:10:52 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.35599999999999998,"15min":0.34799999999999998},"min":{"1min":0.252,"5min":0.193,"15min":0.193},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.36499999999999999,"15min":0.379},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.497,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.317}]},{"osd":1,"last update":"Sat Aug 17 17:10:52 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.38300000000000001,"15min":0.39700000000000002},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.378,"15min":0.38600000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.33900000000000002}]},{"osd":2,"last update":"Sat Aug 17 17:10:52 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.40500000000000003,"15min":0.41599999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.40100000000000002,"15min":0.40500000000000003},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.91900000000000004,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.26700000000000002}]},{"osd":4,"last update":"Sat Aug 17 17:10:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54700000000000004,"15min":0.54500000000000004},"min":{"1min":0.42399999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.75600000000000001,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53800000000000003,"15min":0.54800000000000004},"min":{"1min":0.40400000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.42999999999999999}]},{"osd":5,"last update":"Sat Aug 17 17:10:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.41499999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57399999999999995,"15min":0.57199999999999995},"min":{"1min":0.44400000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.49099999999999999}]},{"osd":6,"last update":"Sat Aug 17 17:10:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.441,"5min":0.371,"15min":0.371},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61699999999999999,"15min":0.625},"min":{"1min":0.379,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.45800000000000002}]},{"osd":7,"last update":"Sat Aug 17 17:10:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.42199999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.437},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55000000000000004,"15min":0.54800000000000004},"min":{"1min":0.433,"5min":0.379,"15min":0.379},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.40999999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31856,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713296,"statfs":{"total":95995035648,"available":95962415104,"internally_reserved":0,"allocated":753664,"data_stored":328475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:10:24 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.34699999999999998,"15min":0.34899999999999998},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.35399999999999998,"15min":0.35599999999999998},"min":{"1min":0.24299999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.28199999999999997}]},{"osd":1,"last update":"Sat Aug 17 17:10:24 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.35799999999999998,"15min":0.35999999999999999},"min":{"1min":0.19,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.51400000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.38300000000000001,"15min":0.374},"min":{"1min":0.24099999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.61499999999999999,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.39600000000000002}]},{"osd":3,"last update":"Sat Aug 17 17:10:47 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40899999999999997,"15min":0.432},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.41399999999999998,"15min":0.42599999999999999},"min":{"1min":0.28499999999999998,"5min":0.214,"15min":0.214},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.32400000000000001}]},{"osd":4,"last update":"Sat Aug 17 17:10:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55100000000000005,"15min":0.56299999999999994},"min":{"1min":0.40799999999999997,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70999999999999996,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.42699999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.432}]},{"osd":5,"last update":"Sat Aug 17 17:10:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.52500000000000002},"min":{"1min":0.35499999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.73599999999999999,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.40100000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.47599999999999998}]},{"osd":6,"last update":"Sat Aug 17 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.39100000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.71999999999999997,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.41899999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.51000000000000001}]},{"osd":7,"last update":"Sat Aug 17 17:10:21 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52300000000000002,"15min":0.51800000000000002},"min":{"1min":0.41999999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69099999999999995,"5min":0.754,"15min":0.754},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.52600000000000002,"15min":0.52200000000000002},"min":{"1min":0.42999999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.54200000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27756,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717396,"statfs":{"total":95995035648,"available":95966613504,"internally_reserved":0,"allocated":749568,"data_stored":327878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34300000000000003,"15min":0.33000000000000002},"min":{"1min":0.23999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.51700000000000002,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.35399999999999998,"15min":0.35099999999999998},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.503,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.29499999999999998}]},{"osd":2,"last update":"Sat Aug 17 17:10:22 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.38600000000000001,"15min":0.39600000000000002},"min":{"1min":0.248,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.53100000000000003,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.38500000000000001,"15min":0.40000000000000002},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.52300000000000002,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.32500000000000001}]},{"osd":3,"last update":"Sat Aug 17 17:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.40999999999999998,"15min":0.442},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.48999999999999999,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.31},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.39600000000000002,"15min":0.41799999999999998},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.51800000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.33600000000000002}]},{"osd":4,"last update":"Sat Aug 17 17:10:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52500000000000002,"15min":0.51800000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.72799999999999998,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.624},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.59499999999999997}]},{"osd":5,"last update":"Sat Aug 17 17:10:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57799999999999996,"15min":0.58699999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.76900000000000002,"5min":0.88,"15min":0.88},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56699999999999995,"15min":0.57199999999999995},"min":{"1min":0.39500000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.83299999999999996,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.63900000000000001}]},{"osd":6,"last update":"Sat Aug 17 17:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.53700000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72499999999999998,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.35799999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70399999999999996,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.58399999999999996}]},{"osd":7,"last update":"Sat Aug 17 17:10:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.40400000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55800000000000005,"15min":0.55500000000000005},"min":{"1min":0.41499999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.86499999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.55700000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28340,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716812,"statfs":{"total":95995035648,"available":95966015488,"internally_reserved":0,"allocated":1347584,"data_stored":918831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 17 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.35599999999999998,"15min":0.30099999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.46100000000000002,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.35999999999999999,"15min":0.30399999999999999},"min":{"1min":0.20000000000000001,"5min":0.19,"15min":0.19},"max":{"1min":0.497,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":0.32400000000000001}]},{"osd":2,"last update":"Sat Aug 17 17:10:28 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.35299999999999998,"15min":0.33500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.379,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.22700000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.58199999999999996,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.36899999999999999}]},{"osd":3,"last update":"Sat Aug 17 17:10:48 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.42199999999999999,"15min":0.41999999999999998},"min":{"1min":0.28399999999999997,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.51800000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.39100000000000001,"15min":0.39700000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.54000000000000004,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.39700000000000002}]},{"osd":4,"last update":"Sat Aug 17 17:10:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.52800000000000002,"15min":0.51400000000000001},"min":{"1min":0.37,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81999999999999995,"5min":1.115,"15min":1.115},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.56200000000000006,"15min":0.55000000000000004},"min":{"1min":0.39600000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.55500000000000005}]},{"osd":5,"last update":"Sat Aug 17 17:10:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.54000000000000004,"15min":0.53900000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.70099999999999996,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54500000000000004,"15min":0.53200000000000003},"min":{"1min":0.35199999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.84299999999999997,"5min":1.821,"15min":1.821},"last":0.51900000000000002}]},{"osd":6,"last update":"Sat Aug 17 17:10:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57499999999999996,"15min":0.58399999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.71799999999999997,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.57199999999999995,"15min":0.56899999999999995},"min":{"1min":0.39800000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66900000000000004,"5min":1.851,"15min":1.851},"last":0.56499999999999995}]},{"osd":7,"last update":"Sat Aug 17 17:10:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79600000000000004,"5min":1.085,"15min":1.085},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59099999999999997,"15min":0.58999999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82999999999999996,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.53800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:11:06.582 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:11:06.786 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:11:08.255 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:07 smithi136 ceph-mon[31034]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:08.255 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:07 smithi136 ceph-mon[42580]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:07 smithi165 ceph-mon[34868]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:08.660 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:11:08.661 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T17:11:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:08 smithi165 ceph-mon[34868]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:11:09.384 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:08 smithi136 ceph-mon[31034]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:11:09.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:08 smithi136 ceph-mon[42580]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:11:09.406 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2024-08-17T17:11:07.884822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592749,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":226,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":170,"ondisk_log_size":170,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":232036,"kb_used_data":7656,"kb_used_omap":12,"kb_used_meta":224243,"kb_avail":749729180,"statfs":{"total":767960285184,"available":767722680320,"internally_reserved":0,"allocated":7839744,"data_stored":4399945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":229625400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002792"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340955+0000","last_change":"2024-08-17T17:10:06.156584+0000","last_active":"2024-08-17T17:10:48.340955+0000","last_peered":"2024-08-17T17:10:48.340955+0000","last_clean":"2024-08-17T17:10:48.340955+0000","last_became_active":"2024-08-17T17:10:06.156252+0000","last_became_peered":"2024-08-17T17:10:06.156252+0000","last_unstale":"2024-08-17T17:10:48.340955+0000","last_undegraded":"2024-08-17T17:10:48.340955+0000","last_fullsized":"2024-08-17T17:10:48.340955+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:18:53.713960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.350239+0000","last_change":"2024-08-17T17:10:05.975682+0000","last_active":"2024-08-17T17:10:48.350239+0000","last_peered":"2024-08-17T17:10:48.350239+0000","last_clean":"2024-08-17T17:10:48.350239+0000","last_became_active":"2024-08-17T17:10:05.975420+0000","last_became_peered":"2024-08-17T17:10:05.975420+0000","last_unstale":"2024-08-17T17:10:48.350239+0000","last_undegraded":"2024-08-17T17:10:48.350239+0000","last_fullsized":"2024-08-17T17:10:48.350239+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:13:35.325266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340411+0000","last_change":"2024-08-17T17:10:07.397936+0000","last_active":"2024-08-17T17:10:48.340411+0000","last_peered":"2024-08-17T17:10:48.340411+0000","last_clean":"2024-08-17T17:10:48.340411+0000","last_became_active":"2024-08-17T17:10:07.397642+0000","last_became_peered":"2024-08-17T17:10:07.397642+0000","last_unstale":"2024-08-17T17:10:48.340411+0000","last_undegraded":"2024-08-17T17:10:48.340411+0000","last_fullsized":"2024-08-17T17:10:48.340411+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:27:27.497025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337171+0000","last_change":"2024-08-17T17:10:05.971463+0000","last_active":"2024-08-17T17:10:48.337171+0000","last_peered":"2024-08-17T17:10:48.337171+0000","last_clean":"2024-08-17T17:10:48.337171+0000","last_became_active":"2024-08-17T17:10:05.971112+0000","last_became_peered":"2024-08-17T17:10:05.971112+0000","last_unstale":"2024-08-17T17:10:48.337171+0000","last_undegraded":"2024-08-17T17:10:48.337171+0000","last_fullsized":"2024-08-17T17:10:48.337171+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:53:46.305291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337905+0000","last_change":"2024-08-17T17:10:06.157520+0000","last_active":"2024-08-17T17:10:48.337905+0000","last_peered":"2024-08-17T17:10:48.337905+0000","last_clean":"2024-08-17T17:10:48.337905+0000","last_became_active":"2024-08-17T17:10:06.157223+0000","last_became_peered":"2024-08-17T17:10:06.157223+0000","last_unstale":"2024-08-17T17:10:48.337905+0000","last_undegraded":"2024-08-17T17:10:48.337905+0000","last_fullsized":"2024-08-17T17:10:48.337905+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:37:58.194744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.350136+0000","last_change":"2024-08-17T17:10:05.975692+0000","last_active":"2024-08-17T17:10:48.350136+0000","last_peered":"2024-08-17T17:10:48.350136+0000","last_clean":"2024-08-17T17:10:48.350136+0000","last_became_active":"2024-08-17T17:10:05.975057+0000","last_became_peered":"2024-08-17T17:10:05.975057+0000","last_unstale":"2024-08-17T17:10:48.350136+0000","last_undegraded":"2024-08-17T17:10:48.350136+0000","last_fullsized":"2024-08-17T17:10:48.350136+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:44:53.106104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340968+0000","last_change":"2024-08-17T17:10:06.156570+0000","last_active":"2024-08-17T17:10:48.340968+0000","last_peered":"2024-08-17T17:10:48.340968+0000","last_clean":"2024-08-17T17:10:48.340968+0000","last_became_active":"2024-08-17T17:10:06.156267+0000","last_became_peered":"2024-08-17T17:10:06.156267+0000","last_unstale":"2024-08-17T17:10:48.340968+0000","last_undegraded":"2024-08-17T17:10:48.340968+0000","last_fullsized":"2024-08-17T17:10:48.340968+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:00:52.553271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":23,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.350096+0000","last_change":"2024-08-17T17:10:05.966582+0000","last_active":"2024-08-17T17:10:48.350096+0000","last_peered":"2024-08-17T17:10:48.350096+0000","last_clean":"2024-08-17T17:10:48.350096+0000","last_became_active":"2024-08-17T17:10:05.966324+0000","last_became_peered":"2024-08-17T17:10:05.966324+0000","last_unstale":"2024-08-17T17:10:48.350096+0000","last_undegraded":"2024-08-17T17:10:48.350096+0000","last_fullsized":"2024-08-17T17:10:48.350096+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:19:19.689962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.350525+0000","last_change":"2024-08-17T17:10:05.976856+0000","last_active":"2024-08-17T17:10:48.350525+0000","last_peered":"2024-08-17T17:10:48.350525+0000","last_clean":"2024-08-17T17:10:48.350525+0000","last_became_active":"2024-08-17T17:10:05.976575+0000","last_became_peered":"2024-08-17T17:10:05.976575+0000","last_unstale":"2024-08-17T17:10:48.350525+0000","last_undegraded":"2024-08-17T17:10:48.350525+0000","last_fullsized":"2024-08-17T17:10:48.350525+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:17:15.169128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340301+0000","last_change":"2024-08-17T17:10:06.994799+0000","last_active":"2024-08-17T17:10:48.340301+0000","last_peered":"2024-08-17T17:10:48.340301+0000","last_clean":"2024-08-17T17:10:48.340301+0000","last_became_active":"2024-08-17T17:10:06.994150+0000","last_became_peered":"2024-08-17T17:10:06.994150+0000","last_unstale":"2024-08-17T17:10:48.340301+0000","last_undegraded":"2024-08-17T17:10:48.340301+0000","last_fullsized":"2024-08-17T17:10:48.340301+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:50:47.293091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.350715+0000","last_change":"2024-08-17T17:10:05.974435+0000","last_active":"2024-08-17T17:10:48.350715+0000","last_peered":"2024-08-17T17:10:48.350715+0000","last_clean":"2024-08-17T17:10:48.350715+0000","last_became_active":"2024-08-17T17:10:05.974181+0000","last_became_peered":"2024-08-17T17:10:05.974181+0000","last_unstale":"2024-08-17T17:10:48.350715+0000","last_undegraded":"2024-08-17T17:10:48.350715+0000","last_fullsized":"2024-08-17T17:10:48.350715+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:52:43.173554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337279+0000","last_change":"2024-08-17T17:10:07.398836+0000","last_active":"2024-08-17T17:10:48.337279+0000","last_peered":"2024-08-17T17:10:48.337279+0000","last_clean":"2024-08-17T17:10:48.337279+0000","last_became_active":"2024-08-17T17:10:07.398548+0000","last_became_peered":"2024-08-17T17:10:07.398548+0000","last_unstale":"2024-08-17T17:10:48.337279+0000","last_undegraded":"2024-08-17T17:10:48.337279+0000","last_fullsized":"2024-08-17T17:10:48.337279+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:00:21.641230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.350775+0000","last_change":"2024-08-17T17:10:05.973491+0000","last_active":"2024-08-17T17:10:48.350775+0000","last_peered":"2024-08-17T17:10:48.350775+0000","last_clean":"2024-08-17T17:10:48.350775+0000","last_became_active":"2024-08-17T17:10:05.973228+0000","last_became_peered":"2024-08-17T17:10:05.973228+0000","last_unstale":"2024-08-17T17:10:48.350775+0000","last_undegraded":"2024-08-17T17:10:48.350775+0000","last_fullsized":"2024-08-17T17:10:48.350775+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:54.310135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.349990+0000","last_change":"2024-08-17T17:10:06.994297+0000","last_active":"2024-08-17T17:10:48.349990+0000","last_peered":"2024-08-17T17:10:48.349990+0000","last_clean":"2024-08-17T17:10:48.349990+0000","last_became_active":"2024-08-17T17:10:06.993979+0000","last_became_peered":"2024-08-17T17:10:06.993979+0000","last_unstale":"2024-08-17T17:10:48.349990+0000","last_undegraded":"2024-08-17T17:10:48.349990+0000","last_fullsized":"2024-08-17T17:10:48.349990+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:20:13.978272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340680+0000","last_change":"2024-08-17T17:10:05.960882+0000","last_active":"2024-08-17T17:10:48.340680+0000","last_peered":"2024-08-17T17:10:48.340680+0000","last_clean":"2024-08-17T17:10:48.340680+0000","last_became_active":"2024-08-17T17:10:05.960578+0000","last_became_peered":"2024-08-17T17:10:05.960578+0000","last_unstale":"2024-08-17T17:10:48.340680+0000","last_undegraded":"2024-08-17T17:10:48.340680+0000","last_fullsized":"2024-08-17T17:10:48.340680+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:06.791494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:10:49.342735+0000","last_change":"2024-08-17T17:10:49.342735+0000","last_active":"2024-08-17T17:10:49.342735+0000","last_peered":"2024-08-17T17:10:49.342735+0000","last_clean":"2024-08-17T17:10:49.342735+0000","last_became_active":"2024-08-17T17:10:49.342331+0000","last_became_peered":"2024-08-17T17:10:49.342331+0000","last_unstale":"2024-08-17T17:10:49.342735+0000","last_undegraded":"2024-08-17T17:10:49.342735+0000","last_fullsized":"2024-08-17T17:10:49.342735+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:56:02.469448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340461+0000","last_change":"2024-08-17T17:10:05.964640+0000","last_active":"2024-08-17T17:10:48.340461+0000","last_peered":"2024-08-17T17:10:48.340461+0000","last_clean":"2024-08-17T17:10:48.340461+0000","last_became_active":"2024-08-17T17:10:05.964085+0000","last_became_peered":"2024-08-17T17:10:05.964085+0000","last_unstale":"2024-08-17T17:10:48.340461+0000","last_undegraded":"2024-08-17T17:10:48.340461+0000","last_fullsized":"2024-08-17T17:10:48.340461+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:58:17.913779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337640+0000","last_change":"2024-08-17T17:10:07.397981+0000","last_active":"2024-08-17T17:10:48.337640+0000","last_peered":"2024-08-17T17:10:48.337640+0000","last_clean":"2024-08-17T17:10:48.337640+0000","last_became_active":"2024-08-17T17:10:07.397666+0000","last_became_peered":"2024-08-17T17:10:07.397666+0000","last_unstale":"2024-08-17T17:10:48.337640+0000","last_undegraded":"2024-08-17T17:10:48.337640+0000","last_fullsized":"2024-08-17T17:10:48.337640+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:42.103035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.350511+0000","last_change":"2024-08-17T17:10:05.966428+0000","last_active":"2024-08-17T17:10:48.350511+0000","last_peered":"2024-08-17T17:10:48.350511+0000","last_clean":"2024-08-17T17:10:48.350511+0000","last_became_active":"2024-08-17T17:10:05.966156+0000","last_became_peered":"2024-08-17T17:10:05.966156+0000","last_unstale":"2024-08-17T17:10:48.350511+0000","last_undegraded":"2024-08-17T17:10:48.350511+0000","last_fullsized":"2024-08-17T17:10:48.350511+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:32:45.657632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337977+0000","last_change":"2024-08-17T17:10:07.438535+0000","last_active":"2024-08-17T17:10:48.337977+0000","last_peered":"2024-08-17T17:10:48.337977+0000","last_clean":"2024-08-17T17:10:48.337977+0000","last_became_active":"2024-08-17T17:10:07.438245+0000","last_became_peered":"2024-08-17T17:10:07.438245+0000","last_unstale":"2024-08-17T17:10:48.337977+0000","last_undegraded":"2024-08-17T17:10:48.337977+0000","last_fullsized":"2024-08-17T17:10:48.337977+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:31:17.604018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337458+0000","last_change":"2024-08-17T17:10:05.973321+0000","last_active":"2024-08-17T17:10:48.337458+0000","last_peered":"2024-08-17T17:10:48.337458+0000","last_clean":"2024-08-17T17:10:48.337458+0000","last_became_active":"2024-08-17T17:10:05.973057+0000","last_became_peered":"2024-08-17T17:10:05.973057+0000","last_unstale":"2024-08-17T17:10:48.337458+0000","last_undegraded":"2024-08-17T17:10:48.337458+0000","last_fullsized":"2024-08-17T17:10:48.337458+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:29:50.938627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337938+0000","last_change":"2024-08-17T17:10:06.979766+0000","last_active":"2024-08-17T17:10:48.337938+0000","last_peered":"2024-08-17T17:10:48.337938+0000","last_clean":"2024-08-17T17:10:48.337938+0000","last_became_active":"2024-08-17T17:10:06.979504+0000","last_became_peered":"2024-08-17T17:10:06.979504+0000","last_unstale":"2024-08-17T17:10:48.337938+0000","last_undegraded":"2024-08-17T17:10:48.337938+0000","last_fullsized":"2024-08-17T17:10:48.337938+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:58:52.990126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340505+0000","last_change":"2024-08-17T17:10:05.969167+0000","last_active":"2024-08-17T17:10:48.340505+0000","last_peered":"2024-08-17T17:10:48.340505+0000","last_clean":"2024-08-17T17:10:48.340505+0000","last_became_active":"2024-08-17T17:10:05.968885+0000","last_became_peered":"2024-08-17T17:10:05.968885+0000","last_unstale":"2024-08-17T17:10:48.340505+0000","last_undegraded":"2024-08-17T17:10:48.340505+0000","last_fullsized":"2024-08-17T17:10:48.340505+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:56:34.762597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337299+0000","last_change":"2024-08-17T17:10:06.579964+0000","last_active":"2024-08-17T17:10:48.337299+0000","last_peered":"2024-08-17T17:10:48.337299+0000","last_clean":"2024-08-17T17:10:48.337299+0000","last_became_active":"2024-08-17T17:10:06.579633+0000","last_became_peered":"2024-08-17T17:10:06.579633+0000","last_unstale":"2024-08-17T17:10:48.337299+0000","last_undegraded":"2024-08-17T17:10:48.337299+0000","last_fullsized":"2024-08-17T17:10:48.337299+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:57:23.495634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"61'128","reported_seq":268,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.350258+0000","last_change":"2024-08-17T17:08:20.329502+0000","last_active":"2024-08-17T17:10:48.350258+0000","last_peered":"2024-08-17T17:10:48.350258+0000","last_clean":"2024-08-17T17:10:48.350258+0000","last_became_active":"2024-08-17T17:08:20.329063+0000","last_became_peered":"2024-08-17T17:08:20.329063+0000","last_unstale":"2024-08-17T17:10:48.350258+0000","last_undegraded":"2024-08-17T17:10:48.350258+0000","last_fullsized":"2024-08-17T17:10:48.350258+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T17:06:21.174471+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:52:20.712819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":29,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:10:49.341961+0000","last_change":"2024-08-17T17:08:53.943617+0000","last_active":"2024-08-17T17:10:49.341961+0000","last_peered":"2024-08-17T17:10:49.341961+0000","last_clean":"2024-08-17T17:10:49.341961+0000","last_became_active":"2024-08-17T17:08:48.854004+0000","last_became_peered":"2024-08-17T17:08:48.854004+0000","last_unstale":"2024-08-17T17:10:49.341961+0000","last_undegraded":"2024-08-17T17:10:49.341961+0000","last_fullsized":"2024-08-17T17:10:49.341961+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:53:56.669365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041737399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340294+0000","last_change":"2024-08-17T17:10:05.960101+0000","last_active":"2024-08-17T17:10:48.340294+0000","last_peered":"2024-08-17T17:10:48.340294+0000","last_clean":"2024-08-17T17:10:48.340294+0000","last_became_active":"2024-08-17T17:10:05.959775+0000","last_became_peered":"2024-08-17T17:10:05.959775+0000","last_unstale":"2024-08-17T17:10:48.340294+0000","last_undegraded":"2024-08-17T17:10:48.340294+0000","last_fullsized":"2024-08-17T17:10:48.340294+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:03:19.894065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.350586+0000","last_change":"2024-08-17T17:10:06.995023+0000","last_active":"2024-08-17T17:10:48.350586+0000","last_peered":"2024-08-17T17:10:48.350586+0000","last_clean":"2024-08-17T17:10:48.350586+0000","last_became_active":"2024-08-17T17:10:06.994757+0000","last_became_peered":"2024-08-17T17:10:06.994757+0000","last_unstale":"2024-08-17T17:10:48.350586+0000","last_undegraded":"2024-08-17T17:10:48.350586+0000","last_fullsized":"2024-08-17T17:10:48.350586+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:31:52.163518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":28,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.876366+0000","last_change":"2024-08-17T17:08:54.068863+0000","last_active":"2024-08-17T17:10:48.876366+0000","last_peered":"2024-08-17T17:10:48.876366+0000","last_clean":"2024-08-17T17:10:48.876366+0000","last_became_active":"2024-08-17T17:08:48.854469+0000","last_became_peered":"2024-08-17T17:08:48.854469+0000","last_unstale":"2024-08-17T17:10:48.876366+0000","last_undegraded":"2024-08-17T17:10:48.876366+0000","last_fullsized":"2024-08-17T17:10:48.876366+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:32:44.509162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337035+0000","last_change":"2024-08-17T17:10:05.968146+0000","last_active":"2024-08-17T17:10:48.337035+0000","last_peered":"2024-08-17T17:10:48.337035+0000","last_clean":"2024-08-17T17:10:48.337035+0000","last_became_active":"2024-08-17T17:10:05.967911+0000","last_became_peered":"2024-08-17T17:10:05.967911+0000","last_unstale":"2024-08-17T17:10:48.337035+0000","last_undegraded":"2024-08-17T17:10:48.337035+0000","last_fullsized":"2024-08-17T17:10:48.337035+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:26:04.782748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337847+0000","last_change":"2024-08-17T17:10:05.988330+0000","last_active":"2024-08-17T17:10:48.337847+0000","last_peered":"2024-08-17T17:10:48.337847+0000","last_clean":"2024-08-17T17:10:48.337847+0000","last_became_active":"2024-08-17T17:10:05.988015+0000","last_became_peered":"2024-08-17T17:10:05.988015+0000","last_unstale":"2024-08-17T17:10:48.337847+0000","last_undegraded":"2024-08-17T17:10:48.337847+0000","last_fullsized":"2024-08-17T17:10:48.337847+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:01:44.304295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":28,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:10:49.340343+0000","last_change":"2024-08-17T17:08:53.939867+0000","last_active":"2024-08-17T17:10:49.340343+0000","last_peered":"2024-08-17T17:10:49.340343+0000","last_clean":"2024-08-17T17:10:49.340343+0000","last_became_active":"2024-08-17T17:08:48.927330+0000","last_became_peered":"2024-08-17T17:08:48.927330+0000","last_unstale":"2024-08-17T17:10:49.340343+0000","last_undegraded":"2024-08-17T17:10:49.340343+0000","last_fullsized":"2024-08-17T17:10:49.340343+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:03:20.888081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.336756+0000","last_change":"2024-08-17T17:10:05.960071+0000","last_active":"2024-08-17T17:10:48.336756+0000","last_peered":"2024-08-17T17:10:48.336756+0000","last_clean":"2024-08-17T17:10:48.336756+0000","last_became_active":"2024-08-17T17:10:05.959830+0000","last_became_peered":"2024-08-17T17:10:05.959830+0000","last_unstale":"2024-08-17T17:10:48.336756+0000","last_undegraded":"2024-08-17T17:10:48.336756+0000","last_fullsized":"2024-08-17T17:10:48.336756+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:26:49.379433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337370+0000","last_change":"2024-08-17T17:10:05.984255+0000","last_active":"2024-08-17T17:10:48.337370+0000","last_peered":"2024-08-17T17:10:48.337370+0000","last_clean":"2024-08-17T17:10:48.337370+0000","last_became_active":"2024-08-17T17:10:05.983889+0000","last_became_peered":"2024-08-17T17:10:05.983889+0000","last_unstale":"2024-08-17T17:10:48.337370+0000","last_undegraded":"2024-08-17T17:10:48.337370+0000","last_fullsized":"2024-08-17T17:10:48.337370+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:50:29.637480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":30,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:10:49.341885+0000","last_change":"2024-08-17T17:08:53.944227+0000","last_active":"2024-08-17T17:10:49.341885+0000","last_peered":"2024-08-17T17:10:49.341885+0000","last_clean":"2024-08-17T17:10:49.341885+0000","last_became_active":"2024-08-17T17:08:48.854463+0000","last_became_peered":"2024-08-17T17:08:48.854463+0000","last_unstale":"2024-08-17T17:10:49.341885+0000","last_undegraded":"2024-08-17T17:10:49.341885+0000","last_fullsized":"2024-08-17T17:10:49.341885+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:06:41.281670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001023532,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.338060+0000","last_change":"2024-08-17T17:10:05.974383+0000","last_active":"2024-08-17T17:10:48.338060+0000","last_peered":"2024-08-17T17:10:48.338060+0000","last_clean":"2024-08-17T17:10:48.338060+0000","last_became_active":"2024-08-17T17:10:05.974059+0000","last_became_peered":"2024-08-17T17:10:05.974059+0000","last_unstale":"2024-08-17T17:10:48.338060+0000","last_undegraded":"2024-08-17T17:10:48.338060+0000","last_fullsized":"2024-08-17T17:10:48.338060+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:22:30.316285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:10:49.344184+0000","last_change":"2024-08-17T17:10:49.344184+0000","last_active":"2024-08-17T17:10:49.344184+0000","last_peered":"2024-08-17T17:10:49.344184+0000","last_clean":"2024-08-17T17:10:49.344184+0000","last_became_active":"2024-08-17T17:10:49.343838+0000","last_became_peered":"2024-08-17T17:10:49.343838+0000","last_unstale":"2024-08-17T17:10:49.344184+0000","last_undegraded":"2024-08-17T17:10:49.344184+0000","last_fullsized":"2024-08-17T17:10:49.344184+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:51:30.332750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340612+0000","last_change":"2024-08-17T17:10:05.960285+0000","last_active":"2024-08-17T17:10:48.340612+0000","last_peered":"2024-08-17T17:10:48.340612+0000","last_clean":"2024-08-17T17:10:48.340612+0000","last_became_active":"2024-08-17T17:10:05.959910+0000","last_became_peered":"2024-08-17T17:10:05.959910+0000","last_unstale":"2024-08-17T17:10:48.340612+0000","last_undegraded":"2024-08-17T17:10:48.340612+0000","last_fullsized":"2024-08-17T17:10:48.340612+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:15:55.327175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":28,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:10:49.342164+0000","last_change":"2024-08-17T17:08:53.939956+0000","last_active":"2024-08-17T17:10:49.342164+0000","last_peered":"2024-08-17T17:10:49.342164+0000","last_clean":"2024-08-17T17:10:49.342164+0000","last_became_active":"2024-08-17T17:08:48.851812+0000","last_became_peered":"2024-08-17T17:08:48.851812+0000","last_unstale":"2024-08-17T17:10:49.342164+0000","last_undegraded":"2024-08-17T17:10:49.342164+0000","last_fullsized":"2024-08-17T17:10:49.342164+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:25:20.326625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.336917+0000","last_change":"2024-08-17T17:10:06.579900+0000","last_active":"2024-08-17T17:10:48.336917+0000","last_peered":"2024-08-17T17:10:48.336917+0000","last_clean":"2024-08-17T17:10:48.336917+0000","last_became_active":"2024-08-17T17:10:06.579567+0000","last_became_peered":"2024-08-17T17:10:06.579567+0000","last_unstale":"2024-08-17T17:10:48.336917+0000","last_undegraded":"2024-08-17T17:10:48.336917+0000","last_fullsized":"2024-08-17T17:10:48.336917+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:29:50.641345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337807+0000","last_change":"2024-08-17T17:10:05.959878+0000","last_active":"2024-08-17T17:10:48.337807+0000","last_peered":"2024-08-17T17:10:48.337807+0000","last_clean":"2024-08-17T17:10:48.337807+0000","last_became_active":"2024-08-17T17:10:05.959595+0000","last_became_peered":"2024-08-17T17:10:05.959595+0000","last_unstale":"2024-08-17T17:10:48.337807+0000","last_undegraded":"2024-08-17T17:10:48.337807+0000","last_fullsized":"2024-08-17T17:10:48.337807+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:06:09.585988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":28,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:10:50.041355+0000","last_change":"2024-08-17T17:08:54.068835+0000","last_active":"2024-08-17T17:10:50.041355+0000","last_peered":"2024-08-17T17:10:50.041355+0000","last_clean":"2024-08-17T17:10:50.041355+0000","last_became_active":"2024-08-17T17:08:48.853784+0000","last_became_peered":"2024-08-17T17:08:48.853784+0000","last_unstale":"2024-08-17T17:10:50.041355+0000","last_undegraded":"2024-08-17T17:10:50.041355+0000","last_fullsized":"2024-08-17T17:10:50.041355+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:43:23.734272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337971+0000","last_change":"2024-08-17T17:10:06.156581+0000","last_active":"2024-08-17T17:10:48.337971+0000","last_peered":"2024-08-17T17:10:48.337971+0000","last_clean":"2024-08-17T17:10:48.337971+0000","last_became_active":"2024-08-17T17:10:06.156260+0000","last_became_peered":"2024-08-17T17:10:06.156260+0000","last_unstale":"2024-08-17T17:10:48.337971+0000","last_undegraded":"2024-08-17T17:10:48.337971+0000","last_fullsized":"2024-08-17T17:10:48.337971+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:32.420918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.341068+0000","last_change":"2024-08-17T17:10:05.968996+0000","last_active":"2024-08-17T17:10:48.341068+0000","last_peered":"2024-08-17T17:10:48.341068+0000","last_clean":"2024-08-17T17:10:48.341068+0000","last_became_active":"2024-08-17T17:10:05.968724+0000","last_became_peered":"2024-08-17T17:10:05.968724+0000","last_unstale":"2024-08-17T17:10:48.341068+0000","last_undegraded":"2024-08-17T17:10:48.341068+0000","last_fullsized":"2024-08-17T17:10:48.341068+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:50:37.286495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":28,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:10:49.341047+0000","last_change":"2024-08-17T17:08:53.939970+0000","last_active":"2024-08-17T17:10:49.341047+0000","last_peered":"2024-08-17T17:10:49.341047+0000","last_clean":"2024-08-17T17:10:49.341047+0000","last_became_active":"2024-08-17T17:08:48.852650+0000","last_became_peered":"2024-08-17T17:08:48.852650+0000","last_unstale":"2024-08-17T17:10:49.341047+0000","last_undegraded":"2024-08-17T17:10:49.341047+0000","last_fullsized":"2024-08-17T17:10:49.341047+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:40.181438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000256001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340249+0000","last_change":"2024-08-17T17:10:06.156505+0000","last_active":"2024-08-17T17:10:48.340249+0000","last_peered":"2024-08-17T17:10:48.340249+0000","last_clean":"2024-08-17T17:10:48.340249+0000","last_became_active":"2024-08-17T17:10:06.156197+0000","last_became_peered":"2024-08-17T17:10:06.156197+0000","last_unstale":"2024-08-17T17:10:48.340249+0000","last_undegraded":"2024-08-17T17:10:48.340249+0000","last_fullsized":"2024-08-17T17:10:48.340249+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:11:12.864042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337978+0000","last_change":"2024-08-17T17:10:05.968221+0000","last_active":"2024-08-17T17:10:48.337978+0000","last_peered":"2024-08-17T17:10:48.337978+0000","last_clean":"2024-08-17T17:10:48.337978+0000","last_became_active":"2024-08-17T17:10:05.967961+0000","last_became_peered":"2024-08-17T17:10:05.967961+0000","last_unstale":"2024-08-17T17:10:48.337978+0000","last_undegraded":"2024-08-17T17:10:48.337978+0000","last_fullsized":"2024-08-17T17:10:48.337978+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:24:58.119474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":28,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:10:50.041740+0000","last_change":"2024-08-17T17:08:53.945206+0000","last_active":"2024-08-17T17:10:50.041740+0000","last_peered":"2024-08-17T17:10:50.041740+0000","last_clean":"2024-08-17T17:10:50.041740+0000","last_became_active":"2024-08-17T17:08:48.853366+0000","last_became_peered":"2024-08-17T17:08:48.853366+0000","last_unstale":"2024-08-17T17:10:50.041740+0000","last_undegraded":"2024-08-17T17:10:50.041740+0000","last_fullsized":"2024-08-17T17:10:50.041740+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:34:32.938937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337410+0000","last_change":"2024-08-17T17:10:06.979492+0000","last_active":"2024-08-17T17:10:48.337410+0000","last_peered":"2024-08-17T17:10:48.337410+0000","last_clean":"2024-08-17T17:10:48.337410+0000","last_became_active":"2024-08-17T17:10:06.979158+0000","last_became_peered":"2024-08-17T17:10:06.979158+0000","last_unstale":"2024-08-17T17:10:48.337410+0000","last_undegraded":"2024-08-17T17:10:48.337410+0000","last_fullsized":"2024-08-17T17:10:48.337410+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:12:42.261365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337923+0000","last_change":"2024-08-17T17:10:06.979662+0000","last_active":"2024-08-17T17:10:48.337923+0000","last_peered":"2024-08-17T17:10:48.337923+0000","last_clean":"2024-08-17T17:10:48.337923+0000","last_became_active":"2024-08-17T17:10:06.979377+0000","last_became_peered":"2024-08-17T17:10:06.979377+0000","last_unstale":"2024-08-17T17:10:48.337923+0000","last_undegraded":"2024-08-17T17:10:48.337923+0000","last_fullsized":"2024-08-17T17:10:48.337923+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:29:06.898927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"70'6","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337732+0000","last_change":"2024-08-17T17:10:05.969131+0000","last_active":"2024-08-17T17:10:48.337732+0000","last_peered":"2024-08-17T17:10:48.337732+0000","last_clean":"2024-08-17T17:10:48.337732+0000","last_became_active":"2024-08-17T17:10:05.968856+0000","last_became_peered":"2024-08-17T17:10:05.968856+0000","last_unstale":"2024-08-17T17:10:48.337732+0000","last_undegraded":"2024-08-17T17:10:48.337732+0000","last_fullsized":"2024-08-17T17:10:48.337732+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:25:49.550065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.336840+0000","last_change":"2024-08-17T17:10:06.087073+0000","last_active":"2024-08-17T17:10:48.336840+0000","last_peered":"2024-08-17T17:10:48.336840+0000","last_clean":"2024-08-17T17:10:48.336840+0000","last_became_active":"2024-08-17T17:10:06.086742+0000","last_became_peered":"2024-08-17T17:10:06.086742+0000","last_unstale":"2024-08-17T17:10:48.336840+0000","last_undegraded":"2024-08-17T17:10:48.336840+0000","last_fullsized":"2024-08-17T17:10:48.336840+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:13:04.339226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":18,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.350729+0000","last_change":"2024-08-17T17:10:05.974976+0000","last_active":"2024-08-17T17:10:48.350729+0000","last_peered":"2024-08-17T17:10:48.350729+0000","last_clean":"2024-08-17T17:10:48.350729+0000","last_became_active":"2024-08-17T17:10:05.974726+0000","last_became_peered":"2024-08-17T17:10:05.974726+0000","last_unstale":"2024-08-17T17:10:48.350729+0000","last_undegraded":"2024-08-17T17:10:48.350729+0000","last_fullsized":"2024-08-17T17:10:48.350729+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:26:54.318334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340492+0000","last_change":"2024-08-17T17:10:05.966061+0000","last_active":"2024-08-17T17:10:48.340492+0000","last_peered":"2024-08-17T17:10:48.340492+0000","last_clean":"2024-08-17T17:10:48.340492+0000","last_became_active":"2024-08-17T17:10:05.965796+0000","last_became_peered":"2024-08-17T17:10:05.965796+0000","last_unstale":"2024-08-17T17:10:48.340492+0000","last_undegraded":"2024-08-17T17:10:48.340492+0000","last_fullsized":"2024-08-17T17:10:48.340492+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:57:12.028490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340716+0000","last_change":"2024-08-17T17:10:06.157618+0000","last_active":"2024-08-17T17:10:48.340716+0000","last_peered":"2024-08-17T17:10:48.340716+0000","last_clean":"2024-08-17T17:10:48.340716+0000","last_became_active":"2024-08-17T17:10:06.156724+0000","last_became_peered":"2024-08-17T17:10:06.156724+0000","last_unstale":"2024-08-17T17:10:48.340716+0000","last_undegraded":"2024-08-17T17:10:48.340716+0000","last_fullsized":"2024-08-17T17:10:48.340716+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:35:46.294434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337618+0000","last_change":"2024-08-17T17:10:05.965210+0000","last_active":"2024-08-17T17:10:48.337618+0000","last_peered":"2024-08-17T17:10:48.337618+0000","last_clean":"2024-08-17T17:10:48.337618+0000","last_became_active":"2024-08-17T17:10:05.964956+0000","last_became_peered":"2024-08-17T17:10:05.964956+0000","last_unstale":"2024-08-17T17:10:48.337618+0000","last_undegraded":"2024-08-17T17:10:48.337618+0000","last_fullsized":"2024-08-17T17:10:48.337618+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:42:36.519168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340160+0000","last_change":"2024-08-17T17:10:06.087172+0000","last_active":"2024-08-17T17:10:48.340160+0000","last_peered":"2024-08-17T17:10:48.340160+0000","last_clean":"2024-08-17T17:10:48.340160+0000","last_became_active":"2024-08-17T17:10:06.086840+0000","last_became_peered":"2024-08-17T17:10:06.086840+0000","last_unstale":"2024-08-17T17:10:48.340160+0000","last_undegraded":"2024-08-17T17:10:48.340160+0000","last_fullsized":"2024-08-17T17:10:48.340160+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:39:42.946969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.350321+0000","last_change":"2024-08-17T17:10:06.978485+0000","last_active":"2024-08-17T17:10:48.350321+0000","last_peered":"2024-08-17T17:10:48.350321+0000","last_clean":"2024-08-17T17:10:48.350321+0000","last_became_active":"2024-08-17T17:10:06.978145+0000","last_became_peered":"2024-08-17T17:10:06.978145+0000","last_unstale":"2024-08-17T17:10:48.350321+0000","last_undegraded":"2024-08-17T17:10:48.350321+0000","last_fullsized":"2024-08-17T17:10:48.350321+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:59.241653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340398+0000","last_change":"2024-08-17T17:10:05.962483+0000","last_active":"2024-08-17T17:10:48.340398+0000","last_peered":"2024-08-17T17:10:48.340398+0000","last_clean":"2024-08-17T17:10:48.340398+0000","last_became_active":"2024-08-17T17:10:05.962216+0000","last_became_peered":"2024-08-17T17:10:05.962216+0000","last_unstale":"2024-08-17T17:10:48.340398+0000","last_undegraded":"2024-08-17T17:10:48.340398+0000","last_fullsized":"2024-08-17T17:10:48.340398+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:31:59.156020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337786+0000","last_change":"2024-08-17T17:10:06.157231+0000","last_active":"2024-08-17T17:10:48.337786+0000","last_peered":"2024-08-17T17:10:48.337786+0000","last_clean":"2024-08-17T17:10:48.337786+0000","last_became_active":"2024-08-17T17:10:06.156943+0000","last_became_peered":"2024-08-17T17:10:06.156943+0000","last_unstale":"2024-08-17T17:10:48.337786+0000","last_undegraded":"2024-08-17T17:10:48.337786+0000","last_fullsized":"2024-08-17T17:10:48.337786+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:51:05.141785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.336808+0000","last_change":"2024-08-17T17:10:05.967087+0000","last_active":"2024-08-17T17:10:48.336808+0000","last_peered":"2024-08-17T17:10:48.336808+0000","last_clean":"2024-08-17T17:10:48.336808+0000","last_became_active":"2024-08-17T17:10:05.966793+0000","last_became_peered":"2024-08-17T17:10:05.966793+0000","last_unstale":"2024-08-17T17:10:48.336808+0000","last_undegraded":"2024-08-17T17:10:48.336808+0000","last_fullsized":"2024-08-17T17:10:48.336808+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:39.118660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:10:49.346039+0000","last_change":"2024-08-17T17:10:49.346039+0000","last_active":"2024-08-17T17:10:49.346039+0000","last_peered":"2024-08-17T17:10:49.346039+0000","last_clean":"2024-08-17T17:10:49.346039+0000","last_became_active":"2024-08-17T17:10:49.345706+0000","last_became_peered":"2024-08-17T17:10:49.345706+0000","last_unstale":"2024-08-17T17:10:49.346039+0000","last_undegraded":"2024-08-17T17:10:49.346039+0000","last_fullsized":"2024-08-17T17:10:49.346039+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:26:19.982527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337502+0000","last_change":"2024-08-17T17:10:05.964796+0000","last_active":"2024-08-17T17:10:48.337502+0000","last_peered":"2024-08-17T17:10:48.337502+0000","last_clean":"2024-08-17T17:10:48.337502+0000","last_became_active":"2024-08-17T17:10:05.964388+0000","last_became_peered":"2024-08-17T17:10:05.964388+0000","last_unstale":"2024-08-17T17:10:48.337502+0000","last_undegraded":"2024-08-17T17:10:48.337502+0000","last_fullsized":"2024-08-17T17:10:48.337502+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:20:06.478705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.339754+0000","last_change":"2024-08-17T17:10:05.984528+0000","last_active":"2024-08-17T17:10:48.339754+0000","last_peered":"2024-08-17T17:10:48.339754+0000","last_clean":"2024-08-17T17:10:48.339754+0000","last_became_active":"2024-08-17T17:10:05.984249+0000","last_became_peered":"2024-08-17T17:10:05.984249+0000","last_unstale":"2024-08-17T17:10:48.339754+0000","last_undegraded":"2024-08-17T17:10:48.339754+0000","last_fullsized":"2024-08-17T17:10:48.339754+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:52:21.938100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.338092+0000","last_change":"2024-08-17T17:10:05.971396+0000","last_active":"2024-08-17T17:10:48.338092+0000","last_peered":"2024-08-17T17:10:48.338092+0000","last_clean":"2024-08-17T17:10:48.338092+0000","last_became_active":"2024-08-17T17:10:05.971087+0000","last_became_peered":"2024-08-17T17:10:05.971087+0000","last_unstale":"2024-08-17T17:10:48.338092+0000","last_undegraded":"2024-08-17T17:10:48.338092+0000","last_fullsized":"2024-08-17T17:10:48.338092+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:32:36.084786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337823+0000","last_change":"2024-08-17T17:10:07.398001+0000","last_active":"2024-08-17T17:10:48.337823+0000","last_peered":"2024-08-17T17:10:48.337823+0000","last_clean":"2024-08-17T17:10:48.337823+0000","last_became_active":"2024-08-17T17:10:07.397733+0000","last_became_peered":"2024-08-17T17:10:07.397733+0000","last_unstale":"2024-08-17T17:10:48.337823+0000","last_undegraded":"2024-08-17T17:10:48.337823+0000","last_fullsized":"2024-08-17T17:10:48.337823+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:39:06.988794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340465+0000","last_change":"2024-08-17T17:10:05.964931+0000","last_active":"2024-08-17T17:10:48.340465+0000","last_peered":"2024-08-17T17:10:48.340465+0000","last_clean":"2024-08-17T17:10:48.340465+0000","last_became_active":"2024-08-17T17:10:05.964647+0000","last_became_peered":"2024-08-17T17:10:05.964647+0000","last_unstale":"2024-08-17T17:10:48.340465+0000","last_undegraded":"2024-08-17T17:10:48.340465+0000","last_fullsized":"2024-08-17T17:10:48.340465+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:53:14.162545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337416+0000","last_change":"2024-08-17T17:10:07.398175+0000","last_active":"2024-08-17T17:10:48.337416+0000","last_peered":"2024-08-17T17:10:48.337416+0000","last_clean":"2024-08-17T17:10:48.337416+0000","last_became_active":"2024-08-17T17:10:07.397596+0000","last_became_peered":"2024-08-17T17:10:07.397596+0000","last_unstale":"2024-08-17T17:10:48.337416+0000","last_undegraded":"2024-08-17T17:10:48.337416+0000","last_fullsized":"2024-08-17T17:10:48.337416+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:37.039485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340587+0000","last_change":"2024-08-17T17:10:05.965176+0000","last_active":"2024-08-17T17:10:48.340587+0000","last_peered":"2024-08-17T17:10:48.340587+0000","last_clean":"2024-08-17T17:10:48.340587+0000","last_became_active":"2024-08-17T17:10:05.964861+0000","last_became_peered":"2024-08-17T17:10:05.964861+0000","last_unstale":"2024-08-17T17:10:48.340587+0000","last_undegraded":"2024-08-17T17:10:48.340587+0000","last_fullsized":"2024-08-17T17:10:48.340587+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:59:04.277926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340612+0000","last_change":"2024-08-17T17:10:05.985507+0000","last_active":"2024-08-17T17:10:48.340612+0000","last_peered":"2024-08-17T17:10:48.340612+0000","last_clean":"2024-08-17T17:10:48.340612+0000","last_became_active":"2024-08-17T17:10:05.985191+0000","last_became_peered":"2024-08-17T17:10:05.985191+0000","last_unstale":"2024-08-17T17:10:48.340612+0000","last_undegraded":"2024-08-17T17:10:48.340612+0000","last_fullsized":"2024-08-17T17:10:48.340612+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:59.000088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.340790+0000","last_change":"2024-08-17T17:10:05.959340+0000","last_active":"2024-08-17T17:10:48.340790+0000","last_peered":"2024-08-17T17:10:48.340790+0000","last_clean":"2024-08-17T17:10:48.340790+0000","last_became_active":"2024-08-17T17:10:05.959075+0000","last_became_peered":"2024-08-17T17:10:05.959075+0000","last_unstale":"2024-08-17T17:10:48.340790+0000","last_undegraded":"2024-08-17T17:10:48.340790+0000","last_fullsized":"2024-08-17T17:10:48.340790+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:20:46.147656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337278+0000","last_change":"2024-08-17T17:10:06.978790+0000","last_active":"2024-08-17T17:10:48.337278+0000","last_peered":"2024-08-17T17:10:48.337278+0000","last_clean":"2024-08-17T17:10:48.337278+0000","last_became_active":"2024-08-17T17:10:06.978497+0000","last_became_peered":"2024-08-17T17:10:06.978497+0000","last_unstale":"2024-08-17T17:10:48.337278+0000","last_undegraded":"2024-08-17T17:10:48.337278+0000","last_fullsized":"2024-08-17T17:10:48.337278+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:07:20.340330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":19,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-08-17T17:10:48.337546+0000","last_change":"2024-08-17T17:10:05.968779+0000","last_active":"2024-08-17T17:10:48.337546+0000","last_peered":"2024-08-17T17:10:48.337546+0000","last_clean":"2024-08-17T17:10:48.337546+0000","last_became_active":"2024-08-17T17:10:05.968518+0000","last_became_peered":"2024-08-17T17:10:05.968518+0000","last_unstale":"2024-08-17T17:10:48.337546+0000","last_undegraded":"2024-08-17T17:10:48.337546+0000","last_fullsized":"2024-08-17T17:10:48.337546+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:52.221246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234020,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28356,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716796,"statfs":{"total":95995035648,"available":95965999104,"internally_reserved":0,"allocated":1363968,"data_stored":919726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:10:19 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.52400000000000002,"15min":0.52800000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67800000000000005,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.443},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53100000000000003,"15min":0.53300000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.66300000000000003,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.54200000000000004}]},{"osd":1,"last update":"Sat Aug 17 17:10:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.50700000000000001,"15min":0.505},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66800000000000004,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.55100000000000005,"15min":0.55300000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.68400000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.47399999999999998}]},{"osd":2,"last update":"Sat Aug 17 17:10:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52900000000000003,"15min":0.53200000000000003},"min":{"1min":0.371,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.65300000000000002,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.51800000000000002}]},{"osd":3,"last update":"Sat Aug 17 17:10:19 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52100000000000002,"15min":0.52300000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70599999999999996,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.49099999999999999}]},{"osd":4,"last update":"Sat Aug 17 17:10:19 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.35799999999999998}]},{"osd":5,"last update":"Sat Aug 17 17:10:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.48799999999999999,"15min":0.48399999999999999},"min":{"1min":0.33700000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.318},{"interface":"front","average":{"1min":0.495,"5min":0.52000000000000002,"15min":0.52400000000000002},"min":{"1min":0.34100000000000003,"5min":0.318,"15min":0.318},"max":{"1min":0.71899999999999997,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.40400000000000003}]},{"osd":6,"last update":"Sat Aug 17 17:10:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54400000000000004,"15min":0.54700000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.79100000000000004,"5min":1.024,"15min":1.024},"last":0.374},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.52700000000000002,"15min":0.52600000000000002},"min":{"1min":0.35599999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72099999999999997,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.39300000000000002}]}]},{"osd":6,"up_from":46,"seq":197568495658,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28344,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716808,"statfs":{"total":95995035648,"available":95966011392,"internally_reserved":0,"allocated":1355776,"data_stored":919599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:10:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52800000000000002,"15min":0.52900000000000003},"min":{"1min":0.372,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52800000000000002,"15min":0.53000000000000003},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.54900000000000004}]},{"osd":1,"last update":"Sat Aug 17 17:10:54 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.51900000000000002,"15min":0.52900000000000003},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.751,"5min":0.755,"15min":0.755},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54300000000000004,"15min":0.54900000000000004},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.72799999999999998}]},{"osd":2,"last update":"Sat Aug 17 17:10:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.76200000000000001,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54800000000000004,"15min":0.55300000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.747,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.57699999999999996}]},{"osd":3,"last update":"Sat Aug 17 17:10:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.53900000000000003,"15min":0.54000000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.67200000000000004,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.65900000000000003}]},{"osd":4,"last update":"Sat Aug 17 17:10:54 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.41399999999999998,"15min":0.40400000000000003},"min":{"1min":0.29499999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.60399999999999998,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.497},{"interface":"front","average":{"1min":0.442,"5min":0.41399999999999998,"15min":0.40600000000000003},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.55400000000000005,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.443}]},{"osd":5,"last update":"Sat Aug 17 17:10:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.432,"15min":0.41899999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.65700000000000003,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.435,"15min":0.42499999999999999},"min":{"1min":0.314,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.61699999999999999,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.47999999999999998}]},{"osd":7,"last update":"Sat Aug 17 17:10:17 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33500000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.499,"15min":0.497},"min":{"1min":0.38800000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.52400000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757294,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27760,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":753664,"data_stored":328199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:10:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.52900000000000003,"15min":0.52700000000000002},"min":{"1min":0.33900000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76800000000000002,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51800000000000002,"15min":0.52200000000000002},"min":{"1min":0.38200000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.63200000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.41799999999999998}]},{"osd":1,"last update":"Sat Aug 17 17:10:32 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.52700000000000002,"15min":0.53700000000000003},"min":{"1min":0.41499999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.495,"15min":0.49399999999999999},"min":{"1min":0.35599999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71799999999999997,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.32100000000000001}]},{"osd":2,"last update":"Sat Aug 17 17:10:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.70599999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.439},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52900000000000003,"15min":0.53300000000000003},"min":{"1min":0.38200000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.68000000000000005,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.49399999999999999}]},{"osd":3,"last update":"Sat Aug 17 17:10:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51400000000000001,"15min":0.51700000000000002},"min":{"1min":0.40100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72299999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55200000000000005,"15min":0.55100000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.73699999999999999,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.55800000000000005}]},{"osd":4,"last update":"Sat Aug 17 17:10:32 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40200000000000002,"15min":0.39600000000000002},"min":{"1min":0.312,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.432,"15min":0.436},"min":{"1min":0.28999999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.627,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.61499999999999999}]},{"osd":6,"last update":"Sat Aug 17 17:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.45000000000000001,"15min":0.45100000000000001},"min":{"1min":0.314,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.499,"15min":0.496},"min":{"1min":0.35299999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.71099999999999997,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.58899999999999997}]},{"osd":7,"last update":"Sat Aug 17 17:10:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.51200000000000001,"15min":0.50600000000000001},"min":{"1min":0.39300000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.66400000000000003,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.433,"15min":0.434},"min":{"1min":0.315,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.59899999999999998,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.51300000000000001}]}]},{"osd":4,"up_from":30,"seq":128849018931,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31872,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713280,"statfs":{"total":95995035648,"available":95962398720,"internally_reserved":0,"allocated":770048,"data_stored":329569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:10:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51200000000000001,"15min":0.50900000000000001},"min":{"1min":0.41599999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.71399999999999997,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53600000000000003,"15min":0.53800000000000003},"min":{"1min":0.33700000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.77200000000000002,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.43099999999999999}]},{"osd":1,"last update":"Sat Aug 17 17:10:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.50600000000000001,"15min":0.503},"min":{"1min":0.28499999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68200000000000005,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.39800000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73899999999999999,"5min":0.751,"15min":0.751},"last":0.59999999999999998}]},{"osd":2,"last update":"Sat Aug 17 17:10:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54800000000000004,"15min":0.55200000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.78800000000000003,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.52400000000000002,"15min":0.52200000000000002},"min":{"1min":0.38300000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.505}]},{"osd":3,"last update":"Sat Aug 17 17:10:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.371,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53100000000000003,"15min":0.53000000000000003},"min":{"1min":0.35899999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.46100000000000002}]},{"osd":5,"last update":"Sat Aug 17 17:10:35 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.38700000000000001,"15min":0.371},"min":{"1min":0.249,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66300000000000003,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.41099999999999998,"15min":0.38900000000000001},"min":{"1min":0.28799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.68999999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.54300000000000004}]},{"osd":6,"last update":"Sat Aug 17 17:10:58 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.46500000000000002,"15min":0.443},"min":{"1min":0.32400000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.67600000000000005,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.46899999999999997,"15min":0.45100000000000001},"min":{"1min":0.309,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67700000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.44800000000000001}]},{"osd":7,"last update":"Sat Aug 17 17:10:24 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47199999999999998,"15min":0.46800000000000003},"min":{"1min":0.31900000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71299999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.46700000000000003,"15min":0.46300000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70199999999999996,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.442}]}]},{"osd":3,"up_from":25,"seq":107374182456,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27752,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717400,"statfs":{"total":95995035648,"available":95966617600,"internally_reserved":0,"allocated":745472,"data_stored":327668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:10:52 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.35599999999999998,"15min":0.34799999999999998},"min":{"1min":0.252,"5min":0.193,"15min":0.193},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.36499999999999999,"15min":0.379},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.497,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.317}]},{"osd":1,"last update":"Sat Aug 17 17:10:52 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.38300000000000001,"15min":0.39700000000000002},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.378,"15min":0.38600000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.33900000000000002}]},{"osd":2,"last update":"Sat Aug 17 17:10:52 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.40500000000000003,"15min":0.41599999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.40100000000000002,"15min":0.40500000000000003},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.91900000000000004,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.26700000000000002}]},{"osd":4,"last update":"Sat Aug 17 17:10:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54700000000000004,"15min":0.54500000000000004},"min":{"1min":0.42399999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.75600000000000001,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53800000000000003,"15min":0.54800000000000004},"min":{"1min":0.40400000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.42999999999999999}]},{"osd":5,"last update":"Sat Aug 17 17:10:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.41499999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57399999999999995,"15min":0.57199999999999995},"min":{"1min":0.44400000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.49099999999999999}]},{"osd":6,"last update":"Sat Aug 17 17:10:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.441,"5min":0.371,"15min":0.371},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61699999999999999,"15min":0.625},"min":{"1min":0.379,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.45800000000000002}]},{"osd":7,"last update":"Sat Aug 17 17:10:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.42199999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.437},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55000000000000004,"15min":0.54800000000000004},"min":{"1min":0.433,"5min":0.379,"15min":0.379},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.40999999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31856,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713296,"statfs":{"total":95995035648,"available":95962415104,"internally_reserved":0,"allocated":753664,"data_stored":328475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:10:24 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.34699999999999998,"15min":0.34899999999999998},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.35399999999999998,"15min":0.35599999999999998},"min":{"1min":0.24299999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.28199999999999997}]},{"osd":1,"last update":"Sat Aug 17 17:10:24 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.35799999999999998,"15min":0.35999999999999999},"min":{"1min":0.19,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.51400000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.38300000000000001,"15min":0.374},"min":{"1min":0.24099999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.61499999999999999,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.39600000000000002}]},{"osd":3,"last update":"Sat Aug 17 17:10:47 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40899999999999997,"15min":0.432},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.41399999999999998,"15min":0.42599999999999999},"min":{"1min":0.28499999999999998,"5min":0.214,"15min":0.214},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.32400000000000001}]},{"osd":4,"last update":"Sat Aug 17 17:10:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55100000000000005,"15min":0.56299999999999994},"min":{"1min":0.40799999999999997,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70999999999999996,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.42699999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.432}]},{"osd":5,"last update":"Sat Aug 17 17:10:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.52500000000000002},"min":{"1min":0.35499999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.73599999999999999,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.40100000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.47599999999999998}]},{"osd":6,"last update":"Sat Aug 17 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.39100000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.71999999999999997,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.41899999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.51000000000000001}]},{"osd":7,"last update":"Sat Aug 17 17:10:21 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52300000000000002,"15min":0.51800000000000002},"min":{"1min":0.41999999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69099999999999995,"5min":0.754,"15min":0.754},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.52600000000000002,"15min":0.52200000000000002},"min":{"1min":0.42999999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.54200000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27756,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717396,"statfs":{"total":95995035648,"available":95966613504,"internally_reserved":0,"allocated":749568,"data_stored":327878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34300000000000003,"15min":0.33000000000000002},"min":{"1min":0.23999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.51700000000000002,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.35399999999999998,"15min":0.35099999999999998},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.503,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.29499999999999998}]},{"osd":2,"last update":"Sat Aug 17 17:10:22 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.38600000000000001,"15min":0.39600000000000002},"min":{"1min":0.248,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.53100000000000003,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.38500000000000001,"15min":0.40000000000000002},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.52300000000000002,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.32500000000000001}]},{"osd":3,"last update":"Sat Aug 17 17:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.40999999999999998,"15min":0.442},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.48999999999999999,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.31},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.39600000000000002,"15min":0.41799999999999998},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.51800000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.33600000000000002}]},{"osd":4,"last update":"Sat Aug 17 17:10:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52500000000000002,"15min":0.51800000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.72799999999999998,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.624},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.59499999999999997}]},{"osd":5,"last update":"Sat Aug 17 17:10:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57799999999999996,"15min":0.58699999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.76900000000000002,"5min":0.88,"15min":0.88},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56699999999999995,"15min":0.57199999999999995},"min":{"1min":0.39500000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.83299999999999996,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.63900000000000001}]},{"osd":6,"last update":"Sat Aug 17 17:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.53700000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72499999999999998,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.35799999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70399999999999996,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.58399999999999996}]},{"osd":7,"last update":"Sat Aug 17 17:10:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.40400000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55800000000000005,"15min":0.55500000000000005},"min":{"1min":0.41499999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.86499999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.55700000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28340,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716812,"statfs":{"total":95995035648,"available":95966015488,"internally_reserved":0,"allocated":1347584,"data_stored":918831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 17 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.35599999999999998,"15min":0.30099999999999999},"min":{"1min":0.193,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.46100000000000002,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.35999999999999999,"15min":0.30399999999999999},"min":{"1min":0.20000000000000001,"5min":0.19,"15min":0.19},"max":{"1min":0.497,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":0.32400000000000001}]},{"osd":2,"last update":"Sat Aug 17 17:10:28 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.35299999999999998,"15min":0.33500000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.379,"5min":0.378,"15min":0.35599999999999998},"min":{"1min":0.22700000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.58199999999999996,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.36899999999999999}]},{"osd":3,"last update":"Sat Aug 17 17:10:48 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.42199999999999999,"15min":0.41999999999999998},"min":{"1min":0.28399999999999997,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.51800000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.39100000000000001,"15min":0.39700000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.54000000000000004,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.39700000000000002}]},{"osd":4,"last update":"Sat Aug 17 17:10:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.52800000000000002,"15min":0.51400000000000001},"min":{"1min":0.37,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81999999999999995,"5min":1.115,"15min":1.115},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.56200000000000006,"15min":0.55000000000000004},"min":{"1min":0.39600000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.55500000000000005}]},{"osd":5,"last update":"Sat Aug 17 17:10:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.54000000000000004,"15min":0.53900000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.70099999999999996,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54500000000000004,"15min":0.53200000000000003},"min":{"1min":0.35199999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.84299999999999997,"5min":1.821,"15min":1.821},"last":0.51900000000000002}]},{"osd":6,"last update":"Sat Aug 17 17:10:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57499999999999996,"15min":0.58399999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.71799999999999997,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.57199999999999995,"15min":0.56899999999999995},"min":{"1min":0.39800000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66900000000000004,"5min":1.851,"15min":1.851},"last":0.56499999999999995}]},{"osd":7,"last update":"Sat Aug 17 17:10:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79600000000000004,"5min":1.085,"15min":1.085},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59099999999999997,"15min":0.58999999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82999999999999996,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.53800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:11:09.410 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-08-17T17:11:09.410 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-08-17T17:11:09.410 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-08-17T17:11:09.410 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph health --format=json 2024-08-17T17:11:09.612 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:11:10.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:09 smithi165 ceph-mon[34868]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:10.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:09 smithi136 ceph-mon[31034]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:10.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:09 smithi136 ceph-mon[42580]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:11.505 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:11:11.505 INFO:teuthology.orchestra.run.smithi136.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-17T17:11:11.839 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:11 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/2803635023' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-08-17T17:11:11.839 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:11 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/2803635023' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-08-17T17:11:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:11 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2803635023' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-08-17T17:11:12.163 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-08-17T17:11:12.163 INFO:teuthology.run_tasks:Running task print... 2024-08-17T17:11:12.174 INFO:teuthology.task.print:**** upgrading first half of cluster, with stress **** 2024-08-17T17:11:12.174 INFO:teuthology.run_tasks:Running task parallel... 2024-08-17T17:11:12.186 INFO:teuthology.task.parallel:starting parallel... 2024-08-17T17:11:12.187 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-08-17T17:11:12.187 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-08-17T17:11:12.193 INFO:tasks.workunit:Pulling workunits from ref reef 2024-08-17T17:11:12.193 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-08-17T17:11:12.193 DEBUG:teuthology.orchestra.run.smithi136:> stat -- /home/ubuntu/cephtest/mnt.0 2024-08-17T17:11:12.194 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-08-17T17:11:12.194 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-08-17T17:11:12.194 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi136.front.sepia.ceph.com 2024-08-17T17:11:12.194 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'ceph config set mgr mgr/cephadm/daemon_cache_timeout 60' 2024-08-17T17:11:12.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:11:12.211 INFO:teuthology.orchestra.run.smithi136.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-17T17:11:12.211 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-08-17T17:11:12.267 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-08-17T17:11:12.267 DEBUG:teuthology.orchestra.run.smithi136:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-08-17T17:11:12.323 INFO:tasks.workunit:timeout=3h 2024-08-17T17:11:12.323 INFO:tasks.workunit:cleanup=True 2024-08-17T17:11:12.324 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch reef https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2024-08-17T17:11:12.380 INFO:tasks.workunit.client.0.smithi136.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-08-17T17:11:12.406 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:11:12.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:12 smithi136 ceph-mon[31034]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:12.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:12 smithi136 ceph-mon[42580]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:12 smithi165 ceph-mon[34868]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:14.035 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:13 smithi136 ceph-mon[31034]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:14.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:13 smithi136 ceph-mon[42580]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:13 smithi165 ceph-mon[34868]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:14.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:11:14 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:11:14] ENGINE Bus STOPPING 2024-08-17T17:11:15.031 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:11:14 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:11:14] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-08-17T17:11:15.031 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:11:14 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:11:14] ENGINE Bus STOPPED 2024-08-17T17:11:15.031 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:11:14 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:11:14] ENGINE Bus STARTING 2024-08-17T17:11:15.031 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:11:14 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:11:14] ENGINE Serving on http://:::9283 2024-08-17T17:11:15.031 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:11:14 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:11:14] ENGINE Bus STARTED 2024-08-17T17:11:15.053 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'ceph config set global log_to_journald false --force' 2024-08-17T17:11:15.302 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:11:15.594 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:15 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/953044411' entity='client.admin' 2024-08-17T17:11:15.594 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:15 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:11:15.594 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:15 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:11:15.594 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:15 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:11:15.594 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:15 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:11:15.594 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:15 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/953044411' entity='client.admin' 2024-08-17T17:11:15.595 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:15 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:11:15.595 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:15 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:11:15.595 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:15 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:11:15.595 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:15 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:11:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:15 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/953044411' entity='client.admin' 2024-08-17T17:11:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:15 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:11:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:15 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:11:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:15 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:11:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:15 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:11:16.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:16 smithi136 ceph-mon[31034]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:16.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:16 smithi136 ceph-mon[42580]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:16.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:11:16 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:11:16] "GET /metrics HTTP/1.1" 200 33719 "" "Prometheus/2.43.0" 2024-08-17T17:11:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:16 smithi165 ceph-mon[34868]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:17.862 DEBUG:teuthology.orchestra.run.smithi136:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-08-17T17:11:17.896 INFO:tasks.workunit.client.0.smithi136.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-08-17T17:11:17.898 INFO:tasks.workunit.client.0.smithi136.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-17T17:11:17.898 INFO:tasks.workunit.client.0.smithi136.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-08-17T17:11:18.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:17 smithi165 ceph-mon[34868]: pgmap v91: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:18.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:17 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:18.272 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:17 smithi136 ceph-mon[31034]: pgmap v91: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:18.272 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:17 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:18.272 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:17 smithi136 ceph-mon[42580]: pgmap v91: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:18.272 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:17 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:18.842 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'echo wait for mgr daemons to upgrade' 2024-08-17T17:11:19.090 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:11:19.714 INFO:tasks.workunit.client.0.smithi136.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-08-17T17:11:19.836 INFO:tasks.workunit.client.0.smithi136.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-08-17T17:11:19.880 INFO:tasks.workunit.client.0.smithi136.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-17T17:11:19.882 INFO:tasks.workunit.client.0.smithi136.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-17T17:11:19.882 INFO:tasks.workunit.client.0.smithi136.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-08-17T17:11:19.922 INFO:tasks.workunit.client.0.smithi136.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-17T17:11:19.926 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-17T17:11:19.926 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-08-17T17:11:19.942 INFO:tasks.workunit:Running workunits matching rbd/test_librbd.sh on client.0... 2024-08-17T17:11:19.943 INFO:tasks.workunit:Running workunit rbd/test_librbd.sh... 2024-08-17T17:11:19.944 DEBUG:teuthology.orchestra.run.smithi136:workunit test rbd/test_librbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh 2024-08-17T17:11:20.092 INFO:tasks.workunit.client.0.smithi136.stderr:warning: failed to enable lockdep 2024-08-17T17:11:20.093 INFO:tasks.workunit.client.0.smithi136.stdout:seed 87100 2024-08-17T17:11:20.093 INFO:tasks.workunit.client.0.smithi136.stdout:[==========] Running 490 tests from 24 test suites. 2024-08-17T17:11:20.093 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] Global test environment set-up. 2024-08-17T17:11:20.093 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 1 test from TestJournalStress 2024-08-17T17:11:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:19 smithi165 ceph-mon[34868]: pgmap v92: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:20.267 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:19 smithi136 ceph-mon[31034]: pgmap v92: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:20.267 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:19 smithi136 ceph-mon[42580]: pgmap v92: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:21.163 INFO:teuthology.orchestra.run.smithi136.stdout:wait for mgr daemons to upgrade 2024-08-17T17:11:21.918 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-08-17T17:11:22.005 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestJournalStress.DiscardWithPruneWriteOverlap 2024-08-17T17:11:22.139 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:11:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:22 smithi136 ceph-mon[31034]: osdmap e73: 8 total, 8 up, 8 in 2024-08-17T17:11:22.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:22 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/6110684' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:22.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:22 smithi136 ceph-mon[31034]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:22.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:22 smithi136 ceph-mon[31034]: pgmap v94: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:22 smithi136 ceph-mon[42580]: osdmap e73: 8 total, 8 up, 8 in 2024-08-17T17:11:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:22 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/6110684' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:22 smithi136 ceph-mon[42580]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:22 smithi136 ceph-mon[42580]: pgmap v94: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:22 smithi165 ceph-mon[34868]: osdmap e73: 8 total, 8 up, 8 in 2024-08-17T17:11:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:22 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/6110684' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:22 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:22 smithi165 ceph-mon[34868]: pgmap v94: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:23.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:23 smithi136 ceph-mon[31034]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:11:23.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:23 smithi136 ceph-mon[31034]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:11:23.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:23 smithi136 ceph-mon[31034]: osdmap e74: 8 total, 8 up, 8 in 2024-08-17T17:11:23.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:23 smithi136 ceph-mon[42580]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:11:23.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:23 smithi136 ceph-mon[42580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:11:23.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:23 smithi136 ceph-mon[42580]: osdmap e74: 8 total, 8 up, 8 in 2024-08-17T17:11:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:23 smithi165 ceph-mon[34868]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:11:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:23 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:11:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:23 smithi165 ceph-mon[34868]: osdmap e74: 8 total, 8 up, 8 in 2024-08-17T17:11:24.030 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/journal/test_Stress.cc:37: Skipped 2024-08-17T17:11:24.030 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:11:24.030 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalStress.DiscardWithPruneWriteOverlap (2024 ms) 2024-08-17T17:11:24.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:24 smithi136 ceph-mon[31034]: osdmap e75: 8 total, 8 up, 8 in 2024-08-17T17:11:24.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:24 smithi136 ceph-mon[31034]: pgmap v97: 105 pgs: 105 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-08-17T17:11:24.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:24 smithi136 ceph-mon[42580]: osdmap e75: 8 total, 8 up, 8 in 2024-08-17T17:11:24.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:24 smithi136 ceph-mon[42580]: pgmap v97: 105 pgs: 105 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-08-17T17:11:24.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:24 smithi165 ceph-mon[34868]: osdmap e75: 8 total, 8 up, 8 in 2024-08-17T17:11:24.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:24 smithi165 ceph-mon[34868]: pgmap v97: 105 pgs: 105 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-08-17T17:11:25.023 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 1 test from TestJournalStress (2024 ms total) 2024-08-17T17:11:25.023 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:11:25.023 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 16 tests from TestJournalReplay 2024-08-17T17:11:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:25 smithi136 ceph-mon[31034]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:11:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:25 smithi136 ceph-mon[31034]: Cluster is now healthy 2024-08-17T17:11:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:25 smithi136 ceph-mon[31034]: osdmap e76: 8 total, 8 up, 8 in 2024-08-17T17:11:25.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:25 smithi136 ceph-mon[31034]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:25.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:25 smithi136 ceph-mon[42580]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:11:25.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:25 smithi136 ceph-mon[42580]: Cluster is now healthy 2024-08-17T17:11:25.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:25 smithi136 ceph-mon[42580]: osdmap e76: 8 total, 8 up, 8 in 2024-08-17T17:11:25.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:25 smithi136 ceph-mon[42580]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:25 smithi165 ceph-mon[34868]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:11:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:25 smithi165 ceph-mon[34868]: Cluster is now healthy 2024-08-17T17:11:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:25 smithi165 ceph-mon[34868]: osdmap e76: 8 total, 8 up, 8 in 2024-08-17T17:11:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:25 smithi165 ceph-mon[34868]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:26.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:26 smithi136 ceph-mon[31034]: osdmap e77: 8 total, 8 up, 8 in 2024-08-17T17:11:26.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:26 smithi136 ceph-mon[31034]: pgmap v100: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:26.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:26 smithi136 ceph-mon[42580]: osdmap e77: 8 total, 8 up, 8 in 2024-08-17T17:11:26.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:26 smithi136 ceph-mon[42580]: pgmap v100: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:26.447 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:11:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:11:26] "GET /metrics HTTP/1.1" 200 33724 "" "Prometheus/2.43.0" 2024-08-17T17:11:26.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:26 smithi165 ceph-mon[34868]: osdmap e77: 8 total, 8 up, 8 in 2024-08-17T17:11:26.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:26 smithi165 ceph-mon[34868]: pgmap v100: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:27.037 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestJournalReplay.AioDiscardEvent 2024-08-17T17:11:27.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:27 smithi136 ceph-mon[31034]: osdmap e78: 8 total, 8 up, 8 in 2024-08-17T17:11:27.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:27 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/2508074309' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:27.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:27 smithi136 ceph-mon[42580]: osdmap e78: 8 total, 8 up, 8 in 2024-08-17T17:11:27.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:27 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/2508074309' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:27.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:27 smithi165 ceph-mon[34868]: osdmap e78: 8 total, 8 up, 8 in 2024-08-17T17:11:27.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:27 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2508074309' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:28.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:28 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/2508074309' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:11:28.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:28 smithi136 ceph-mon[31034]: osdmap e79: 8 total, 8 up, 8 in 2024-08-17T17:11:28.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:28 smithi136 ceph-mon[31034]: pgmap v103: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:28.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:28 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/2508074309' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:11:28.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:28 smithi136 ceph-mon[42580]: osdmap e79: 8 total, 8 up, 8 in 2024-08-17T17:11:28.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:28 smithi136 ceph-mon[42580]: pgmap v103: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:28.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:28 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2508074309' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:11:28.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:28 smithi165 ceph-mon[34868]: osdmap e79: 8 total, 8 up, 8 in 2024-08-17T17:11:28.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:28 smithi165 ceph-mon[34868]: pgmap v103: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:29.063 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/journal/test_Replay.cc:109: Skipped 2024-08-17T17:11:29.063 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:11:29.063 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.AioDiscardEvent (2025 ms) 2024-08-17T17:11:29.063 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestJournalReplay.AioWriteEvent 2024-08-17T17:11:29.072 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/journal/test_Replay.cc:200: Skipped 2024-08-17T17:11:29.072 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:11:29.072 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.AioWriteEvent (10 ms) 2024-08-17T17:11:29.072 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestJournalReplay.AioFlushEvent 2024-08-17T17:11:29.080 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/journal/test_Replay.cc:261: Skipped 2024-08-17T17:11:29.080 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:11:29.080 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.AioFlushEvent (8 ms) 2024-08-17T17:11:29.080 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestJournalReplay.SnapCreate 2024-08-17T17:11:29.087 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/journal/test_Replay.cc:307: Skipped 2024-08-17T17:11:29.088 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:11:29.088 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.SnapCreate (8 ms) 2024-08-17T17:11:29.088 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestJournalReplay.SnapProtect 2024-08-17T17:11:29.095 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/journal/test_Replay.cc:348: Skipped 2024-08-17T17:11:29.095 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:11:29.095 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.SnapProtect (7 ms) 2024-08-17T17:11:29.095 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestJournalReplay.SnapUnprotect 2024-08-17T17:11:29.103 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/journal/test_Replay.cc:394: Skipped 2024-08-17T17:11:29.103 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:11:29.103 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.SnapUnprotect (8 ms) 2024-08-17T17:11:29.103 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestJournalReplay.SnapRename 2024-08-17T17:11:29.110 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/journal/test_Replay.cc:450: Skipped 2024-08-17T17:11:29.110 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:11:29.110 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.SnapRename (7 ms) 2024-08-17T17:11:29.110 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestJournalReplay.SnapRollback 2024-08-17T17:11:29.118 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/journal/test_Replay.cc:500: Skipped 2024-08-17T17:11:29.118 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:11:29.118 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.SnapRollback (8 ms) 2024-08-17T17:11:29.118 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestJournalReplay.SnapRemove 2024-08-17T17:11:29.125 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/journal/test_Replay.cc:541: Skipped 2024-08-17T17:11:29.125 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:11:29.125 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.SnapRemove (7 ms) 2024-08-17T17:11:29.125 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestJournalReplay.Rename 2024-08-17T17:11:29.132 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/journal/test_Replay.cc:590: Skipped 2024-08-17T17:11:29.132 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:11:29.132 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.Rename (7 ms) 2024-08-17T17:11:29.132 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestJournalReplay.Resize 2024-08-17T17:11:29.139 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/journal/test_Replay.cc:624: Skipped 2024-08-17T17:11:29.139 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:11:29.139 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.Resize (7 ms) 2024-08-17T17:11:29.139 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestJournalReplay.Flatten 2024-08-17T17:11:29.146 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/journal/test_Replay.cc:657: Skipped 2024-08-17T17:11:29.146 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:11:29.146 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.Flatten (7 ms) 2024-08-17T17:11:29.146 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestJournalReplay.UpdateFeatures 2024-08-17T17:11:29.153 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/journal/test_Replay.cc:704: Skipped 2024-08-17T17:11:29.153 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:11:29.153 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.UpdateFeatures (7 ms) 2024-08-17T17:11:29.153 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestJournalReplay.MetadataSet 2024-08-17T17:11:29.160 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/journal/test_Replay.cc:741: Skipped 2024-08-17T17:11:29.160 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:11:29.160 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.MetadataSet (7 ms) 2024-08-17T17:11:29.160 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestJournalReplay.MetadataRemove 2024-08-17T17:11:29.167 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/journal/test_Replay.cc:781: Skipped 2024-08-17T17:11:29.167 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:11:29.167 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.MetadataRemove (7 ms) 2024-08-17T17:11:29.167 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestJournalReplay.ObjectPosition 2024-08-17T17:11:29.174 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/journal/test_Replay.cc:824: Skipped 2024-08-17T17:11:29.174 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:11:29.174 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.ObjectPosition (7 ms) 2024-08-17T17:11:29.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:29 smithi136 ceph-mon[31034]: osdmap e80: 8 total, 8 up, 8 in 2024-08-17T17:11:29.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:29 smithi136 ceph-mon[42580]: osdmap e80: 8 total, 8 up, 8 in 2024-08-17T17:11:29.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:29 smithi165 ceph-mon[34868]: osdmap e80: 8 total, 8 up, 8 in 2024-08-17T17:11:30.068 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 16 tests from TestJournalReplay (2137 ms total) 2024-08-17T17:11:30.068 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:11:30.068 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 4 tests from TestJournalEntries 2024-08-17T17:11:30.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:30 smithi136 ceph-mon[31034]: osdmap e81: 8 total, 8 up, 8 in 2024-08-17T17:11:30.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:30 smithi136 ceph-mon[31034]: pgmap v106: 105 pgs: 105 active+clean; 579 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:11:30.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:30 smithi136 ceph-mon[42580]: osdmap e81: 8 total, 8 up, 8 in 2024-08-17T17:11:30.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:30 smithi136 ceph-mon[42580]: pgmap v106: 105 pgs: 105 active+clean; 579 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:11:30.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:30 smithi165 ceph-mon[34868]: osdmap e81: 8 total, 8 up, 8 in 2024-08-17T17:11:30.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:30 smithi165 ceph-mon[34868]: pgmap v106: 105 pgs: 105 active+clean; 579 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:11:31.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:31 smithi136 ceph-mon[31034]: osdmap e82: 8 total, 8 up, 8 in 2024-08-17T17:11:31.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:31 smithi136 ceph-mon[42580]: osdmap e82: 8 total, 8 up, 8 in 2024-08-17T17:11:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:31 smithi165 ceph-mon[34868]: osdmap e82: 8 total, 8 up, 8 in 2024-08-17T17:11:32.072 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestJournalEntries.AioWrite 2024-08-17T17:11:32.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:32 smithi136 ceph-mon[31034]: osdmap e83: 8 total, 8 up, 8 in 2024-08-17T17:11:32.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:32 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/3745301588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:32.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:32 smithi136 ceph-mon[31034]: pgmap v109: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:32.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:32 smithi136 ceph-mon[42580]: osdmap e83: 8 total, 8 up, 8 in 2024-08-17T17:11:32.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:32 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/3745301588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:32.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:32 smithi136 ceph-mon[42580]: pgmap v109: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:32.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:32 smithi165 ceph-mon[34868]: osdmap e83: 8 total, 8 up, 8 in 2024-08-17T17:11:32.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:32 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/3745301588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:32.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:32 smithi165 ceph-mon[34868]: pgmap v109: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:33.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:33 smithi136 ceph-mon[42580]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:11:33.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:33 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/3745301588' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:11:33.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:33 smithi136 ceph-mon[42580]: osdmap e84: 8 total, 8 up, 8 in 2024-08-17T17:11:33.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:33 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:11:33.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:33 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:33.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:33 smithi136 ceph-mon[31034]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:11:33.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:33 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/3745301588' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:11:33.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:33 smithi136 ceph-mon[31034]: osdmap e84: 8 total, 8 up, 8 in 2024-08-17T17:11:33.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:33 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:11:33.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:33 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:33.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:33 smithi165 ceph-mon[34868]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:11:33.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:33 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/3745301588' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:11:33.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:33 smithi165 ceph-mon[34868]: osdmap e84: 8 total, 8 up, 8 in 2024-08-17T17:11:33.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:33 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:11:33.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:33 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:34.093 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/journal/test_Entries.cc:118: Skipped 2024-08-17T17:11:34.093 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:11:34.093 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalEntries.AioWrite (2021 ms) 2024-08-17T17:11:34.093 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestJournalEntries.AioDiscard 2024-08-17T17:11:34.102 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/journal/test_Entries.cc:163: Skipped 2024-08-17T17:11:34.102 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:11:34.102 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalEntries.AioDiscard (9 ms) 2024-08-17T17:11:34.102 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestJournalEntries.AioDiscardWithPrune 2024-08-17T17:11:34.109 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/journal/test_Entries.cc:200: Skipped 2024-08-17T17:11:34.109 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:11:34.110 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalEntries.AioDiscardWithPrune (7 ms) 2024-08-17T17:11:34.110 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestJournalEntries.AioFlush 2024-08-17T17:11:34.117 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/journal/test_Entries.cc:263: Skipped 2024-08-17T17:11:34.117 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:11:34.117 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalEntries.AioFlush (8 ms) 2024-08-17T17:11:34.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:34 smithi136 ceph-mon[31034]: osdmap e85: 8 total, 8 up, 8 in 2024-08-17T17:11:34.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:34 smithi136 ceph-mon[31034]: pgmap v112: 105 pgs: 105 active+clean; 579 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:34.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:34 smithi136 ceph-mon[42580]: osdmap e85: 8 total, 8 up, 8 in 2024-08-17T17:11:34.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:34 smithi136 ceph-mon[42580]: pgmap v112: 105 pgs: 105 active+clean; 579 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:34.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:34 smithi165 ceph-mon[34868]: osdmap e85: 8 total, 8 up, 8 in 2024-08-17T17:11:34.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:34 smithi165 ceph-mon[34868]: pgmap v112: 105 pgs: 105 active+clean; 579 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:35.088 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 4 tests from TestJournalEntries (2045 ms total) 2024-08-17T17:11:35.088 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:11:35.088 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 2 tests from TestTrash 2024-08-17T17:11:35.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:35 smithi136 ceph-mon[31034]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:11:35.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:35 smithi136 ceph-mon[31034]: Cluster is now healthy 2024-08-17T17:11:35.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:35 smithi136 ceph-mon[31034]: osdmap e86: 8 total, 8 up, 8 in 2024-08-17T17:11:35.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:35 smithi136 ceph-mon[42580]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:11:35.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:35 smithi136 ceph-mon[42580]: Cluster is now healthy 2024-08-17T17:11:35.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:35 smithi136 ceph-mon[42580]: osdmap e86: 8 total, 8 up, 8 in 2024-08-17T17:11:35.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:35 smithi165 ceph-mon[34868]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:11:35.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:35 smithi165 ceph-mon[34868]: Cluster is now healthy 2024-08-17T17:11:35.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:35 smithi165 ceph-mon[34868]: osdmap e86: 8 total, 8 up, 8 in 2024-08-17T17:11:36.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:36 smithi136 ceph-mon[31034]: osdmap e87: 8 total, 8 up, 8 in 2024-08-17T17:11:36.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:36 smithi136 ceph-mon[31034]: pgmap v115: 73 pgs: 73 active+clean; 579 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:36.447 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:11:36 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:11:36] "GET /metrics HTTP/1.1" 200 36099 "" "Prometheus/2.43.0" 2024-08-17T17:11:36.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:36 smithi136 ceph-mon[42580]: osdmap e87: 8 total, 8 up, 8 in 2024-08-17T17:11:36.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:36 smithi136 ceph-mon[42580]: pgmap v115: 73 pgs: 73 active+clean; 579 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:36.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:36 smithi165 ceph-mon[34868]: osdmap e87: 8 total, 8 up, 8 in 2024-08-17T17:11:36.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:36 smithi165 ceph-mon[34868]: pgmap v115: 73 pgs: 73 active+clean; 579 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:37.099 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestTrash.UserRemovingSource 2024-08-17T17:11:37.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:37 smithi136 ceph-mon[31034]: osdmap e88: 8 total, 8 up, 8 in 2024-08-17T17:11:37.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:37 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/747460666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:37 smithi136 ceph-mon[42580]: osdmap e88: 8 total, 8 up, 8 in 2024-08-17T17:11:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:37 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/747460666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:37 smithi165 ceph-mon[34868]: osdmap e88: 8 total, 8 up, 8 in 2024-08-17T17:11:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:37 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/747460666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:38.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:38 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/747460666' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:11:38.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:38 smithi136 ceph-mon[42580]: osdmap e89: 8 total, 8 up, 8 in 2024-08-17T17:11:38.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:38 smithi136 ceph-mon[42580]: pgmap v118: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:38.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:38 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/747460666' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:11:38.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:38 smithi136 ceph-mon[31034]: osdmap e89: 8 total, 8 up, 8 in 2024-08-17T17:11:38.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:38 smithi136 ceph-mon[31034]: pgmap v118: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:38.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:38 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/747460666' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:11:38.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:38 smithi165 ceph-mon[34868]: osdmap e89: 8 total, 8 up, 8 in 2024-08-17T17:11:38.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:38 smithi165 ceph-mon[34868]: pgmap v118: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:39.193 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestTrash.UserRemovingSource (2094 ms) 2024-08-17T17:11:39.194 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestTrash.RestoreMirroringSource 2024-08-17T17:11:39.234 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestTrash.RestoreMirroringSource (41 ms) 2024-08-17T17:11:39.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:39 smithi136 ceph-mon[31034]: osdmap e90: 8 total, 8 up, 8 in 2024-08-17T17:11:39.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:39 smithi136 ceph-mon[42580]: osdmap e90: 8 total, 8 up, 8 in 2024-08-17T17:11:39.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:39 smithi165 ceph-mon[34868]: osdmap e90: 8 total, 8 up, 8 in 2024-08-17T17:11:40.127 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 2 tests from TestTrash (2135 ms total) 2024-08-17T17:11:40.128 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:11:40.128 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 1 test from TestOperations 2024-08-17T17:11:40.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:40 smithi136 ceph-mon[31034]: osdmap e91: 8 total, 8 up, 8 in 2024-08-17T17:11:40.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:40 smithi136 ceph-mon[31034]: pgmap v121: 105 pgs: 105 active+clean; 579 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:11:40.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:40 smithi136 ceph-mon[42580]: osdmap e91: 8 total, 8 up, 8 in 2024-08-17T17:11:40.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:40 smithi136 ceph-mon[42580]: pgmap v121: 105 pgs: 105 active+clean; 579 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:11:40.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:40 smithi165 ceph-mon[34868]: osdmap e91: 8 total, 8 up, 8 in 2024-08-17T17:11:40.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:40 smithi165 ceph-mon[34868]: pgmap v121: 105 pgs: 105 active+clean; 579 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:11:41.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:41 smithi136 ceph-mon[31034]: osdmap e92: 8 total, 8 up, 8 in 2024-08-17T17:11:41.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:41 smithi136 ceph-mon[42580]: osdmap e92: 8 total, 8 up, 8 in 2024-08-17T17:11:41.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:41 smithi165 ceph-mon[34868]: osdmap e92: 8 total, 8 up, 8 in 2024-08-17T17:11:42.135 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestOperations.DisableJournalingCorrupt 2024-08-17T17:11:42.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:42 smithi136 ceph-mon[31034]: osdmap e93: 8 total, 8 up, 8 in 2024-08-17T17:11:42.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:42 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/1186375802' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:42.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:42 smithi136 ceph-mon[31034]: pgmap v124: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:42.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:42 smithi136 ceph-mon[42580]: osdmap e93: 8 total, 8 up, 8 in 2024-08-17T17:11:42.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:42 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/1186375802' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:42.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:42 smithi136 ceph-mon[42580]: pgmap v124: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:42.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:42 smithi165 ceph-mon[34868]: osdmap e93: 8 total, 8 up, 8 in 2024-08-17T17:11:42.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:42 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/1186375802' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:42.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:42 smithi165 ceph-mon[34868]: pgmap v124: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:43.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:43 smithi136 ceph-mon[31034]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:11:43.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:43 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/1186375802' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:11:43.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:43 smithi136 ceph-mon[31034]: osdmap e94: 8 total, 8 up, 8 in 2024-08-17T17:11:43.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:43 smithi136 ceph-mon[42580]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:11:43.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:43 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/1186375802' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:11:43.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:43 smithi136 ceph-mon[42580]: osdmap e94: 8 total, 8 up, 8 in 2024-08-17T17:11:43.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:43 smithi165 ceph-mon[34868]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:11:43.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:43 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/1186375802' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:11:43.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:43 smithi165 ceph-mon[34868]: osdmap e94: 8 total, 8 up, 8 in 2024-08-17T17:11:44.155 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_Operations.cc:18: Skipped 2024-08-17T17:11:44.155 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:11:44.155 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestOperations.DisableJournalingCorrupt (2020 ms) 2024-08-17T17:11:44.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:44 smithi136 ceph-mon[31034]: osdmap e95: 8 total, 8 up, 8 in 2024-08-17T17:11:44.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:44 smithi136 ceph-mon[31034]: pgmap v127: 105 pgs: 105 active+clean; 579 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:44.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:44 smithi136 ceph-mon[42580]: osdmap e95: 8 total, 8 up, 8 in 2024-08-17T17:11:44.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:44 smithi136 ceph-mon[42580]: pgmap v127: 105 pgs: 105 active+clean; 579 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:44.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:44 smithi165 ceph-mon[34868]: osdmap e95: 8 total, 8 up, 8 in 2024-08-17T17:11:44.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:44 smithi165 ceph-mon[34868]: pgmap v127: 105 pgs: 105 active+clean; 579 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:45.157 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 1 test from TestOperations (2020 ms total) 2024-08-17T17:11:45.158 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:11:45.158 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 4 tests from TestObjectMap 2024-08-17T17:11:45.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:45 smithi136 ceph-mon[31034]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:11:45.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:45 smithi136 ceph-mon[31034]: Cluster is now healthy 2024-08-17T17:11:45.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:45 smithi136 ceph-mon[31034]: osdmap e96: 8 total, 8 up, 8 in 2024-08-17T17:11:45.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:45 smithi136 ceph-mon[42580]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:11:45.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:45 smithi136 ceph-mon[42580]: Cluster is now healthy 2024-08-17T17:11:45.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:45 smithi136 ceph-mon[42580]: osdmap e96: 8 total, 8 up, 8 in 2024-08-17T17:11:45.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:45 smithi165 ceph-mon[34868]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:11:45.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:45 smithi165 ceph-mon[34868]: Cluster is now healthy 2024-08-17T17:11:45.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:45 smithi165 ceph-mon[34868]: osdmap e96: 8 total, 8 up, 8 in 2024-08-17T17:11:46.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:46 smithi136 ceph-mon[42580]: osdmap e97: 8 total, 8 up, 8 in 2024-08-17T17:11:46.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:46 smithi136 ceph-mon[42580]: pgmap v130: 73 pgs: 73 active+clean; 579 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:46.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:46 smithi136 ceph-mon[31034]: osdmap e97: 8 total, 8 up, 8 in 2024-08-17T17:11:46.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:46 smithi136 ceph-mon[31034]: pgmap v130: 73 pgs: 73 active+clean; 579 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:46.447 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:11:46 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:11:46] "GET /metrics HTTP/1.1" 200 36099 "" "Prometheus/2.43.0" 2024-08-17T17:11:46.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:46 smithi165 ceph-mon[34868]: osdmap e97: 8 total, 8 up, 8 in 2024-08-17T17:11:46.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:46 smithi165 ceph-mon[34868]: pgmap v130: 73 pgs: 73 active+clean; 579 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:47.166 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestObjectMap.RefreshInvalidatesWhenCorrupt 2024-08-17T17:11:47.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:47 smithi136 ceph-mon[42580]: osdmap e98: 8 total, 8 up, 8 in 2024-08-17T17:11:47.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:47 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/1863053238' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:47.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:47 smithi136 ceph-mon[42580]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:47.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:47 smithi136 ceph-mon[31034]: osdmap e98: 8 total, 8 up, 8 in 2024-08-17T17:11:47.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:47 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/1863053238' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:47.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:47 smithi136 ceph-mon[31034]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:47 smithi165 ceph-mon[34868]: osdmap e98: 8 total, 8 up, 8 in 2024-08-17T17:11:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:47 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/1863053238' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:47 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:48.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:48 smithi136 ceph-mon[31034]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:11:48.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:48 smithi136 ceph-mon[31034]: osdmap e99: 8 total, 8 up, 8 in 2024-08-17T17:11:48.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:48 smithi136 ceph-mon[31034]: pgmap v133: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:48.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:48 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:11:48.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:48 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:48.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:48 smithi136 ceph-mon[42580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:11:48.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:48 smithi136 ceph-mon[42580]: osdmap e99: 8 total, 8 up, 8 in 2024-08-17T17:11:48.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:48 smithi136 ceph-mon[42580]: pgmap v133: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:48.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:48 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:11:48.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:48 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:48.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:48 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:11:48.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:48 smithi165 ceph-mon[34868]: osdmap e99: 8 total, 8 up, 8 in 2024-08-17T17:11:48.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:48 smithi165 ceph-mon[34868]: pgmap v133: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:48.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:48 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:11:48.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:48 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:49.206 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestObjectMap.RefreshInvalidatesWhenCorrupt (2040 ms) 2024-08-17T17:11:49.206 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestObjectMap.RefreshInvalidatesWhenTooSmall 2024-08-17T17:11:49.233 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestObjectMap.RefreshInvalidatesWhenTooSmall (27 ms) 2024-08-17T17:11:49.233 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestObjectMap.InvalidateFlagOnDisk 2024-08-17T17:11:49.268 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestObjectMap.InvalidateFlagOnDisk (35 ms) 2024-08-17T17:11:49.268 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestObjectMap.AcquireLockInvalidatesWhenTooSmall 2024-08-17T17:11:49.294 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestObjectMap.AcquireLockInvalidatesWhenTooSmall (26 ms) 2024-08-17T17:11:49.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:49 smithi136 ceph-mon[31034]: osdmap e100: 8 total, 8 up, 8 in 2024-08-17T17:11:49.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:49 smithi136 ceph-mon[42580]: osdmap e100: 8 total, 8 up, 8 in 2024-08-17T17:11:49.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:49 smithi165 ceph-mon[34868]: osdmap e100: 8 total, 8 up, 8 in 2024-08-17T17:11:50.192 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 4 tests from TestObjectMap (2128 ms total) 2024-08-17T17:11:50.192 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:11:50.192 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 2 tests from TestMirroringWatcher 2024-08-17T17:11:50.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:50 smithi136 ceph-mon[31034]: osdmap e101: 8 total, 8 up, 8 in 2024-08-17T17:11:50.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:50 smithi136 ceph-mon[31034]: pgmap v136: 105 pgs: 105 active+clean; 579 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:11:50.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:50 smithi136 ceph-mon[42580]: osdmap e101: 8 total, 8 up, 8 in 2024-08-17T17:11:50.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:50 smithi136 ceph-mon[42580]: pgmap v136: 105 pgs: 105 active+clean; 579 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:11:50.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:50 smithi165 ceph-mon[34868]: osdmap e101: 8 total, 8 up, 8 in 2024-08-17T17:11:50.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:50 smithi165 ceph-mon[34868]: pgmap v136: 105 pgs: 105 active+clean; 579 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:11:51.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:51 smithi136 ceph-mon[31034]: osdmap e102: 8 total, 8 up, 8 in 2024-08-17T17:11:51.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:51 smithi136 ceph-mon[42580]: osdmap e102: 8 total, 8 up, 8 in 2024-08-17T17:11:51.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:51 smithi165 ceph-mon[34868]: osdmap e102: 8 total, 8 up, 8 in 2024-08-17T17:11:52.201 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroringWatcher.ModeUpdated 2024-08-17T17:11:52.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:52 smithi136 ceph-mon[31034]: osdmap e103: 8 total, 8 up, 8 in 2024-08-17T17:11:52.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:52 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/129213943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:52.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:52 smithi136 ceph-mon[31034]: pgmap v139: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:52.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:52 smithi136 ceph-mon[42580]: osdmap e103: 8 total, 8 up, 8 in 2024-08-17T17:11:52.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:52 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/129213943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:52.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:52 smithi136 ceph-mon[42580]: pgmap v139: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:52 smithi165 ceph-mon[34868]: osdmap e103: 8 total, 8 up, 8 in 2024-08-17T17:11:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:52 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/129213943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:52 smithi165 ceph-mon[34868]: pgmap v139: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:53.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:53 smithi136 ceph-mon[31034]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:11:53.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:53 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/129213943' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:11:53.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:53 smithi136 ceph-mon[31034]: osdmap e104: 8 total, 8 up, 8 in 2024-08-17T17:11:53.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:53 smithi136 ceph-mon[42580]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:11:53.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:53 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/129213943' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:11:53.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:53 smithi136 ceph-mon[42580]: osdmap e104: 8 total, 8 up, 8 in 2024-08-17T17:11:53.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:53 smithi165 ceph-mon[34868]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:11:53.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:53 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/129213943' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:11:53.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:53 smithi165 ceph-mon[34868]: osdmap e104: 8 total, 8 up, 8 in 2024-08-17T17:11:54.248 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroringWatcher.ModeUpdated (2047 ms) 2024-08-17T17:11:54.248 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroringWatcher.ImageStatusUpdated 2024-08-17T17:11:54.267 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroringWatcher.ImageStatusUpdated (20 ms) 2024-08-17T17:11:54.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:54 smithi165 ceph-mon[34868]: osdmap e105: 8 total, 8 up, 8 in 2024-08-17T17:11:54.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:54 smithi165 ceph-mon[34868]: pgmap v142: 105 pgs: 105 active+clean; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:54.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:54 smithi136 ceph-mon[31034]: osdmap e105: 8 total, 8 up, 8 in 2024-08-17T17:11:54.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:54 smithi136 ceph-mon[31034]: pgmap v142: 105 pgs: 105 active+clean; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:54 smithi136 ceph-mon[42580]: osdmap e105: 8 total, 8 up, 8 in 2024-08-17T17:11:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:54 smithi136 ceph-mon[42580]: pgmap v142: 105 pgs: 105 active+clean; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:55.230 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 2 tests from TestMirroringWatcher (2067 ms total) 2024-08-17T17:11:55.230 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:11:55.230 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 57 tests from TestMigration 2024-08-17T17:11:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:55 smithi165 ceph-mon[34868]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:11:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:55 smithi165 ceph-mon[34868]: Cluster is now healthy 2024-08-17T17:11:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:55 smithi165 ceph-mon[34868]: osdmap e106: 8 total, 8 up, 8 in 2024-08-17T17:11:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:55 smithi136 ceph-mon[31034]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:11:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:55 smithi136 ceph-mon[31034]: Cluster is now healthy 2024-08-17T17:11:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:55 smithi136 ceph-mon[31034]: osdmap e106: 8 total, 8 up, 8 in 2024-08-17T17:11:55.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:55 smithi136 ceph-mon[42580]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:11:55.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:55 smithi136 ceph-mon[42580]: Cluster is now healthy 2024-08-17T17:11:55.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:55 smithi136 ceph-mon[42580]: osdmap e106: 8 total, 8 up, 8 in 2024-08-17T17:11:56.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:56 smithi165 ceph-mon[34868]: osdmap e107: 8 total, 8 up, 8 in 2024-08-17T17:11:56.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:56 smithi165 ceph-mon[34868]: pgmap v145: 73 pgs: 73 active+clean; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:56.564 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:56 smithi136 ceph-mon[31034]: osdmap e107: 8 total, 8 up, 8 in 2024-08-17T17:11:56.564 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:56 smithi136 ceph-mon[31034]: pgmap v145: 73 pgs: 73 active+clean; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:56.565 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:11:56 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:11:56] "GET /metrics HTTP/1.1" 200 36156 "" "Prometheus/2.43.0" 2024-08-17T17:11:56.565 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:56 smithi136 ceph-mon[42580]: osdmap e107: 8 total, 8 up, 8 in 2024-08-17T17:11:56.565 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:56 smithi136 ceph-mon[42580]: pgmap v145: 73 pgs: 73 active+clean; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:57.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:57 smithi136 ceph-mon[42580]: osdmap e108: 8 total, 8 up, 8 in 2024-08-17T17:11:57.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:57 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/2773897634' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:57.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:57 smithi136 ceph-mon[31034]: osdmap e108: 8 total, 8 up, 8 in 2024-08-17T17:11:57.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:57 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/2773897634' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:57 smithi165 ceph-mon[34868]: osdmap e108: 8 total, 8 up, 8 in 2024-08-17T17:11:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:57 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2773897634' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:11:57.764 INFO:teuthology.orchestra.run.smithi136.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:11:58.237 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.Empty 2024-08-17T17:11:58.399 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-08-17T17:11:58.602 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:11:58.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:58 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/2773897634' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:11:58.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:58 smithi136 ceph-mon[31034]: osdmap e109: 8 total, 8 up, 8 in 2024-08-17T17:11:58.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:58 smithi136 ceph-mon[31034]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:11:58.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:58 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:11:58.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:58 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:11:58.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:58 smithi136 ceph-mon[31034]: pgmap v148: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:58.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:11:58 smithi136 ceph-mon[31034]: osdmap e110: 8 total, 8 up, 8 in 2024-08-17T17:11:58.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:58 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/2773897634' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:11:58.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:58 smithi136 ceph-mon[42580]: osdmap e109: 8 total, 8 up, 8 in 2024-08-17T17:11:58.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:58 smithi136 ceph-mon[42580]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:11:58.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:58 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:11:58.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:58 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:11:58.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:58 smithi136 ceph-mon[42580]: pgmap v148: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:58.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:11:58 smithi136 ceph-mon[42580]: osdmap e110: 8 total, 8 up, 8 in 2024-08-17T17:11:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:58 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2773897634' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:11:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:58 smithi165 ceph-mon[34868]: osdmap e109: 8 total, 8 up, 8 in 2024-08-17T17:11:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:58 smithi165 ceph-mon[34868]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:11:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:58 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:11:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:58 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:11:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:58 smithi165 ceph-mon[34868]: pgmap v148: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:11:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:11:58 smithi165 ceph-mon[34868]: osdmap e110: 8 total, 8 up, 8 in 2024-08-17T17:12:00.277 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:00.287 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:12:00.423 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:12:00.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:00 smithi136 ceph-mon[31034]: osdmap e111: 8 total, 8 up, 8 in 2024-08-17T17:12:00.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:00 smithi136 ceph-mon[31034]: pgmap v151: 137 pgs: 32 creating+peering, 105 active+clean; 579 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:12:00.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:00 smithi136 ceph-mon[42580]: osdmap e111: 8 total, 8 up, 8 in 2024-08-17T17:12:00.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:00 smithi136 ceph-mon[42580]: pgmap v151: 137 pgs: 32 creating+peering, 105 active+clean; 579 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:12:00.490 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:12:00.499 INFO:teuthology.orchestra.run.smithi136.stdout:true 2024-08-17T17:12:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:00 smithi165 ceph-mon[34868]: osdmap e111: 8 total, 8 up, 8 in 2024-08-17T17:12:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:00 smithi165 ceph-mon[34868]: pgmap v151: 137 pgs: 32 creating+peering, 105 active+clean; 579 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:12:00.574 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.Empty (2338 ms) 2024-08-17T17:12:00.574 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.OtherName 2024-08-17T17:12:00.604 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:00.615 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:12:00.737 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:12:00.796 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:12:00.877 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.OtherName (303 ms) 2024-08-17T17:12:00.878 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.OtherPool 2024-08-17T17:12:00.907 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:00.917 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:12:01.090 INFO:teuthology.orchestra.run.smithi136.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-17T17:12:01.090 INFO:teuthology.orchestra.run.smithi136.stdout:alertmanager.a smithi136 *:9093,9094 running (2m) 59s ago 2m 17.8M - 0.25.0 c8568f914cd2 83b6bfc5306f 2024-08-17T17:12:01.090 INFO:teuthology.orchestra.run.smithi136.stdout:grafana.a smithi165 *:3000 running (2m) 92s ago 2m 85.4M - 9.4.7 954c08fa6188 be80a687c402 2024-08-17T17:12:01.091 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi136.ltsuqd smithi136 running (107s) 59s ago 105s 15.7M - 18.2.4-719-gced649f8 c870753d67c2 faaa06f2950c 2024-08-17T17:12:01.091 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi165.bmrbhh smithi165 running (103s) 92s ago 103s 14.4M - 18.2.4-719-gced649f8 c870753d67c2 c74b00d929cf 2024-08-17T17:12:01.091 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.x smithi165 *:8443,8765 running (6m) 92s ago 6m 438M - 18.2.4-719-gced649f8 c870753d67c2 5723e013d909 2024-08-17T17:12:01.091 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.y smithi136 *:9283,8765 running (8m) 59s ago 8m 484M - 18.2.4-719-gced649f8 c870753d67c2 f74bb81f3a53 2024-08-17T17:12:01.091 INFO:teuthology.orchestra.run.smithi136.stdout:mon.a smithi136 running (8m) 59s ago 8m 44.9M 2048M 18.2.4-719-gced649f8 c870753d67c2 cc4e5e5ff756 2024-08-17T17:12:01.091 INFO:teuthology.orchestra.run.smithi136.stdout:mon.b smithi165 running (7m) 92s ago 7m 37.7M 2048M 18.2.4-719-gced649f8 c870753d67c2 d587e16140e1 2024-08-17T17:12:01.091 INFO:teuthology.orchestra.run.smithi136.stdout:mon.c smithi136 running (7m) 59s ago 7m 38.6M 2048M 18.2.4-719-gced649f8 c870753d67c2 b4e0f3c9e485 2024-08-17T17:12:01.091 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.a smithi136 *:9100 running (2m) 59s ago 2m 15.1M - 1.5.0 0da6a335fe13 6aef82a780f5 2024-08-17T17:12:01.091 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.b smithi165 *:9100 running (2m) 92s ago 2m 7898k - 1.5.0 0da6a335fe13 60b065dd0c2a 2024-08-17T17:12:01.091 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0 smithi136 running (6m) 59s ago 6m 44.3M 4016M 18.2.4-719-gced649f8 c870753d67c2 faab916734b0 2024-08-17T17:12:01.091 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1 smithi136 running (6m) 59s ago 6m 45.0M 4016M 18.2.4-719-gced649f8 c870753d67c2 861686b6ff82 2024-08-17T17:12:01.091 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2 smithi136 running (5m) 59s ago 5m 49.0M 4016M 18.2.4-719-gced649f8 c870753d67c2 640be4af4156 2024-08-17T17:12:01.091 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3 smithi136 running (5m) 59s ago 5m 47.3M 4016M 18.2.4-719-gced649f8 c870753d67c2 14b57a745d30 2024-08-17T17:12:01.092 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4 smithi165 running (5m) 92s ago 5m 49.6M 4016M 18.2.4-719-gced649f8 c870753d67c2 df366c73e1f2 2024-08-17T17:12:01.092 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5 smithi165 running (4m) 92s ago 4m 39.8M 4016M 18.2.4-719-gced649f8 c870753d67c2 1131665e203a 2024-08-17T17:12:01.092 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6 smithi165 running (4m) 92s ago 4m 49.2M 4016M 18.2.4-719-gced649f8 c870753d67c2 7bc5fd925293 2024-08-17T17:12:01.092 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7 smithi165 running (3m) 92s ago 3m 42.1M 4016M 18.2.4-719-gced649f8 c870753d67c2 18382e4ae3b8 2024-08-17T17:12:01.092 INFO:teuthology.orchestra.run.smithi136.stdout:prometheus.a smithi165 *:9095 running (116s) 92s ago 3m 33.5M - 2.43.0 a07b618ecd1d 1e7f77a17708 2024-08-17T17:12:01.430 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:12:01.430 INFO:teuthology.orchestra.run.smithi136.stdout: "mon": { 2024-08-17T17:12:01.430 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 3 2024-08-17T17:12:01.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:12:01.431 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr": { 2024-08-17T17:12:01.431 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 2 2024-08-17T17:12:01.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:12:01.431 INFO:teuthology.orchestra.run.smithi136.stdout: "osd": { 2024-08-17T17:12:01.431 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 8 2024-08-17T17:12:01.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:12:01.431 INFO:teuthology.orchestra.run.smithi136.stdout: "mds": { 2024-08-17T17:12:01.431 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 2 2024-08-17T17:12:01.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:12:01.431 INFO:teuthology.orchestra.run.smithi136.stdout: "overall": { 2024-08-17T17:12:01.432 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 15 2024-08-17T17:12:01.432 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:12:01.432 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:12:01.432 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:01 smithi136 ceph-mon[31034]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:12:01.432 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:01 smithi136 ceph-mon[31034]: osdmap e112: 8 total, 8 up, 8 in 2024-08-17T17:12:01.432 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:01 smithi136 ceph-mon[31034]: from='client.24821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:01.432 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:01 smithi136 ceph-mon[31034]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:01.432 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:01 smithi136 ceph-mon[42580]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:12:01.433 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:01 smithi136 ceph-mon[42580]: osdmap e112: 8 total, 8 up, 8 in 2024-08-17T17:12:01.433 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:01 smithi136 ceph-mon[42580]: from='client.24821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:01.433 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:01 smithi136 ceph-mon[42580]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:01.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:01 smithi165 ceph-mon[34868]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:12:01.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:01 smithi165 ceph-mon[34868]: osdmap e112: 8 total, 8 up, 8 in 2024-08-17T17:12:01.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:01 smithi165 ceph-mon[34868]: from='client.24821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:01.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:01 smithi165 ceph-mon[34868]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:01.732 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:12:01.733 INFO:teuthology.orchestra.run.smithi136.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453", 2024-08-17T17:12:01.733 INFO:teuthology.orchestra.run.smithi136.stdout: "in_progress": true, 2024-08-17T17:12:01.733 INFO:teuthology.orchestra.run.smithi136.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-08-17T17:12:01.733 INFO:teuthology.orchestra.run.smithi136.stdout: "services_complete": [], 2024-08-17T17:12:01.733 INFO:teuthology.orchestra.run.smithi136.stdout: "progress": "", 2024-08-17T17:12:01.733 INFO:teuthology.orchestra.run.smithi136.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 image", 2024-08-17T17:12:01.733 INFO:teuthology.orchestra.run.smithi136.stdout: "is_paused": false 2024-08-17T17:12:01.733 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:12:02.354 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:12:02.415 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:12:02.495 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.OtherPool (1618 ms) 2024-08-17T17:12:02.495 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.OtherNamespace 2024-08-17T17:12:02.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:02 smithi165 ceph-mon[34868]: from='client.24833 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:02.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:02 smithi165 ceph-mon[34868]: osdmap e113: 8 total, 8 up, 8 in 2024-08-17T17:12:02.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:02 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:02.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:02 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:02.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:02 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:12:02.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:02 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:12:02.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:02 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:02.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:02 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2702517746' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:12:02.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:02 smithi165 ceph-mon[34868]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:12:02.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:02 smithi165 ceph-mon[34868]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:02.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:02 smithi165 ceph-mon[34868]: pgmap v154: 137 pgs: 32 creating+peering, 105 active+clean; 579 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:12:02.524 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:02.530 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:02 smithi136 ceph-mon[31034]: from='client.24833 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:02.530 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:02 smithi136 ceph-mon[31034]: osdmap e113: 8 total, 8 up, 8 in 2024-08-17T17:12:02.530 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:02 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:02.530 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:02 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:02.530 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:02 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:12:02.530 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:02 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:12:02.530 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:02 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:02.530 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:02 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/2702517746' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:12:02.531 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:02 smithi136 ceph-mon[31034]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:12:02.531 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:02 smithi136 ceph-mon[31034]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:02.531 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:02 smithi136 ceph-mon[31034]: pgmap v154: 137 pgs: 32 creating+peering, 105 active+clean; 579 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:12:02.531 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:02 smithi136 ceph-mon[42580]: from='client.24833 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:02.531 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:02 smithi136 ceph-mon[42580]: osdmap e113: 8 total, 8 up, 8 in 2024-08-17T17:12:02.531 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:02 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:02.531 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:02 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:02.531 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:02 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:12:02.531 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:02 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:12:02.531 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:02 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:02.531 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:02 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/2702517746' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:12:02.532 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:02 smithi136 ceph-mon[42580]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:12:02.532 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:02 smithi136 ceph-mon[42580]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:02.532 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:02 smithi136 ceph-mon[42580]: pgmap v154: 137 pgs: 32 creating+peering, 105 active+clean; 579 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:12:02.538 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:12:02.669 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:12:02.732 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:12:02.811 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.OtherNamespace (316 ms) 2024-08-17T17:12:02.811 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.DataPool 2024-08-17T17:12:02.839 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:02.849 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:12:02.986 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:12:03.047 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:12:03.126 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.DataPool (315 ms) 2024-08-17T17:12:03.127 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.AbortAfterPrepare 2024-08-17T17:12:03.157 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:03.167 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:12:03.305 INFO:tasks.workunit.client.0.smithi136.stdout:migration_abort 2024-08-17T17:12:03.404 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.AbortAfterPrepare (278 ms) 2024-08-17T17:12:03.404 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.AbortAfterFailedPrepare 2024-08-17T17:12:03.436 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:03.446 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:12:03.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:03 smithi165 ceph-mon[34868]: osdmap e114: 8 total, 8 up, 8 in 2024-08-17T17:12:03.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:03 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:03.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:03 smithi165 ceph-mon[34868]: Upgrade: Target is version 19.0.0-5836-gcd8bdbeb (unknown) 2024-08-17T17:12:03.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:03 smithi165 ceph-mon[34868]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d'] 2024-08-17T17:12:03.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:03 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:12:03.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:03 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:03.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:03 smithi165 ceph-mon[34868]: Upgrade: Need to upgrade myself (mgr.y) 2024-08-17T17:12:03.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:03 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:03.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:03 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:12:03.549 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.AbortAfterFailedPrepare (145 ms) 2024-08-17T17:12:03.549 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.AbortAfterExecute 2024-08-17T17:12:03.577 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:03.586 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:12:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:03 smithi136 ceph-mon[31034]: osdmap e114: 8 total, 8 up, 8 in 2024-08-17T17:12:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:03 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:03 smithi136 ceph-mon[31034]: Upgrade: Target is version 19.0.0-5836-gcd8bdbeb (unknown) 2024-08-17T17:12:03.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:03 smithi136 ceph-mon[31034]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d'] 2024-08-17T17:12:03.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:03 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:12:03.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:03 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:03.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:03 smithi136 ceph-mon[31034]: Upgrade: Need to upgrade myself (mgr.y) 2024-08-17T17:12:03.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:03 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:03.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:03 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:12:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:03 smithi136 ceph-mon[42580]: osdmap e114: 8 total, 8 up, 8 in 2024-08-17T17:12:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:03 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:03 smithi136 ceph-mon[42580]: Upgrade: Target is version 19.0.0-5836-gcd8bdbeb (unknown) 2024-08-17T17:12:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:03 smithi136 ceph-mon[42580]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d'] 2024-08-17T17:12:03.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:03 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:12:03.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:03 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:03.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:03 smithi136 ceph-mon[42580]: Upgrade: Need to upgrade myself (mgr.y) 2024-08-17T17:12:03.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:03 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:03.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:03 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:12:03.716 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:12:03.781 INFO:tasks.workunit.client.0.smithi136.stdout:migration_abort 2024-08-17T17:12:03.871 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.AbortAfterExecute (322 ms) 2024-08-17T17:12:03.871 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.OtherPoolAbortAfterExecute 2024-08-17T17:12:03.900 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:03.910 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:12:04.039 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:12:04.101 INFO:tasks.workunit.client.0.smithi136.stdout:migration_abort 2024-08-17T17:12:04.197 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.OtherPoolAbortAfterExecute (326 ms) 2024-08-17T17:12:04.198 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.OtherNamespaceAbortAfterExecute 2024-08-17T17:12:04.228 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:04.241 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:12:04.366 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:12:04.426 INFO:tasks.workunit.client.0.smithi136.stdout:migration_abort 2024-08-17T17:12:04.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:04 smithi165 ceph-mon[34868]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d on smithi165 2024-08-17T17:12:04.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:04 smithi165 ceph-mon[34868]: pgmap v156: 137 pgs: 137 active+clean; 580 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1010 KiB/s rd, 56 KiB/s wr, 1.27k op/s 2024-08-17T17:12:04.530 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.OtherNamespaceAbortAfterExecute (333 ms) 2024-08-17T17:12:04.530 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.MirroringSamePool 2024-08-17T17:12:04.559 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:04.568 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_Migration.cc:635: Skipped 2024-08-17T17:12:04.568 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:12:04.574 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestMigration.MirroringSamePool (44 ms) 2024-08-17T17:12:04.574 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.MirroringAbort 2024-08-17T17:12:04.602 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:04.611 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_Migration.cc:653: Skipped 2024-08-17T17:12:04.611 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:12:04.617 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestMigration.MirroringAbort (43 ms) 2024-08-17T17:12:04.617 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.MirroringOtherPoolDisabled 2024-08-17T17:12:04.648 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:04.659 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_Migration.cc:676: Skipped 2024-08-17T17:12:04.659 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:12:04.665 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestMigration.MirroringOtherPoolDisabled (48 ms) 2024-08-17T17:12:04.665 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.MirroringOtherPoolEnabled 2024-08-17T17:12:04.693 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:04.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:04 smithi136 ceph-mon[31034]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d on smithi165 2024-08-17T17:12:04.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:04 smithi136 ceph-mon[31034]: pgmap v156: 137 pgs: 137 active+clean; 580 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1010 KiB/s rd, 56 KiB/s wr, 1.27k op/s 2024-08-17T17:12:04.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:04 smithi136 ceph-mon[42580]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d on smithi165 2024-08-17T17:12:04.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:04 smithi136 ceph-mon[42580]: pgmap v156: 137 pgs: 137 active+clean; 580 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1010 KiB/s rd, 56 KiB/s wr, 1.27k op/s 2024-08-17T17:12:04.706 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_Migration.cc:694: Skipped 2024-08-17T17:12:04.706 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:12:04.712 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestMigration.MirroringOtherPoolEnabled (47 ms) 2024-08-17T17:12:04.713 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.MirroringPool 2024-08-17T17:12:04.739 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:04.752 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_Migration.cc:714: Skipped 2024-08-17T17:12:04.752 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:12:04.758 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestMigration.MirroringPool (46 ms) 2024-08-17T17:12:04.758 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.Group 2024-08-17T17:12:04.787 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:04.806 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:12:04.943 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:12:05.008 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:12:05.096 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.Group (338 ms) 2024-08-17T17:12:05.096 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.GroupAbort 2024-08-17T17:12:05.124 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:05.142 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:12:05.285 INFO:tasks.workunit.client.0.smithi136.stdout:migration_abort 2024-08-17T17:12:05.395 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.GroupAbort (298 ms) 2024-08-17T17:12:05.395 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.NoSnaps 2024-08-17T17:12:05.422 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:05.432 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 0~8388609 2024-08-17T17:12:05.491 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 16777218~8388609 2024-08-17T17:12:05.543 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:12:05.720 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:12:05.828 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:12:05.946 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.NoSnaps (552 ms) 2024-08-17T17:12:05.946 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.NoSnapsOtherPool 2024-08-17T17:12:05.976 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:05.986 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 0~8388609 2024-08-17T17:12:06.042 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 16777218~8388609 2024-08-17T17:12:06.089 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 0~8388609 2024-08-17T17:12:06.095 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 16777218~8388609 2024-08-17T17:12:06.141 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:12:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:05 smithi165 ceph-mon[34868]: pgmap v157: 137 pgs: 137 active+clean; 580 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 784 KiB/s rd, 44 KiB/s wr, 989 op/s 2024-08-17T17:12:06.294 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:12:06.397 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:12:06.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:05 smithi136 ceph-mon[31034]: pgmap v157: 137 pgs: 137 active+clean; 580 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 784 KiB/s rd, 44 KiB/s wr, 989 op/s 2024-08-17T17:12:06.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:05 smithi136 ceph-mon[42580]: pgmap v157: 137 pgs: 137 active+clean; 580 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 784 KiB/s rd, 44 KiB/s wr, 989 op/s 2024-08-17T17:12:06.447 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:12:06] "GET /metrics HTTP/1.1" 200 38521 "" "Prometheus/2.43.0" 2024-08-17T17:12:06.505 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.NoSnapsOtherPool (559 ms) 2024-08-17T17:12:06.505 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.NoSnapsDataPool 2024-08-17T17:12:06.536 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:06.546 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 0~8388609 2024-08-17T17:12:06.613 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 16777218~8388609 2024-08-17T17:12:06.659 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:12:06.834 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:12:06.941 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:12:07.055 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.NoSnapsDataPool (550 ms) 2024-08-17T17:12:07.056 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.NoSnapsShrinkAfterPrepare 2024-08-17T17:12:07.086 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:07.095 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 0~8388609 2024-08-17T17:12:07.154 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 16777218~8388609 2024-08-17T17:12:07.205 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:12:07.379 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 41943040 2024-08-17T17:12:07.387 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:12:07.499 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:12:07.601 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.NoSnapsShrinkAfterPrepare (546 ms) 2024-08-17T17:12:07.601 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.NoSnapsShrinkToZeroBeforePrepare 2024-08-17T17:12:07.631 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:07.641 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 0~8388609 2024-08-17T17:12:07.686 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 16777218~8388609 2024-08-17T17:12:07.736 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 0 2024-08-17T17:12:07.750 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:12:07.846 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:12:07.909 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:12:07.952 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.NoSnapsShrinkToZeroBeforePrepare (351 ms) 2024-08-17T17:12:07.952 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.NoSnapsShrinkToZeroAfterPrepare 2024-08-17T17:12:07.980 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:07.990 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 0~8388609 2024-08-17T17:12:08.049 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 16777218~8388609 2024-08-17T17:12:08.106 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:12:08.293 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 0 2024-08-17T17:12:08.304 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:12:08.364 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:12:08.412 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.NoSnapsShrinkToZeroAfterPrepare (460 ms) 2024-08-17T17:12:08.412 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.NoSnapsExpandAfterPrepare 2024-08-17T17:12:08.439 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:08.449 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 0~8388609 2024-08-17T17:12:08.498 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 16777218~8388609 2024-08-17T17:12:08.546 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:12:08.723 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 167772160 2024-08-17T17:12:08.730 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:12:08.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:08 smithi165 ceph-mon[34868]: osdmap e115: 8 total, 8 up, 8 in 2024-08-17T17:12:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:08 smithi165 ceph-mon[34868]: pgmap v159: 137 pgs: 137 active+clean; 580 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 707 KiB/s rd, 40 KiB/s wr, 892 op/s 2024-08-17T17:12:08.832 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:12:08.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:08 smithi136 ceph-mon[31034]: osdmap e115: 8 total, 8 up, 8 in 2024-08-17T17:12:08.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:08 smithi136 ceph-mon[31034]: pgmap v159: 137 pgs: 137 active+clean; 580 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 707 KiB/s rd, 40 KiB/s wr, 892 op/s 2024-08-17T17:12:08.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:08 smithi136 ceph-mon[42580]: osdmap e115: 8 total, 8 up, 8 in 2024-08-17T17:12:08.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:08 smithi136 ceph-mon[42580]: pgmap v159: 137 pgs: 137 active+clean; 580 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 707 KiB/s rd, 40 KiB/s wr, 892 op/s 2024-08-17T17:12:08.960 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.NoSnapsExpandAfterPrepare (548 ms) 2024-08-17T17:12:08.960 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.NoSnapsSnapAfterPrepare 2024-08-17T17:12:08.986 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:08.996 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 0~8388609 2024-08-17T17:12:09.055 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 16777218~8388609 2024-08-17T17:12:09.109 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:12:09.286 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: after_prepare_snap 2024-08-17T17:12:10.509 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 41943040 2024-08-17T17:12:10.527 INFO:tasks.workunit.client.0.smithi136.stdout:write: * 0~1000 2024-08-17T17:12:10.532 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:12:10.649 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:12:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:10 smithi165 ceph-mon[34868]: osdmap e116: 8 total, 8 up, 8 in 2024-08-17T17:12:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:10 smithi165 ceph-mon[34868]: pgmap v161: 137 pgs: 137 active+clean; 161 MiB data, 736 MiB used, 714 GiB / 715 GiB avail; 55 MiB/s rd, 44 MiB/s wr, 2.28k op/s 2024-08-17T17:12:10.791 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.NoSnapsSnapAfterPrepare (1831 ms) 2024-08-17T17:12:10.791 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.Snaps 2024-08-17T17:12:10.820 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:10.829 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 0~8388609 2024-08-17T17:12:10.883 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:12:10.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:10 smithi136 ceph-mon[31034]: osdmap e116: 8 total, 8 up, 8 in 2024-08-17T17:12:10.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:10 smithi136 ceph-mon[31034]: pgmap v161: 137 pgs: 137 active+clean; 161 MiB data, 736 MiB used, 714 GiB / 715 GiB avail; 55 MiB/s rd, 44 MiB/s wr, 2.28k op/s 2024-08-17T17:12:10.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:10 smithi136 ceph-mon[42580]: osdmap e116: 8 total, 8 up, 8 in 2024-08-17T17:12:10.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:10 smithi136 ceph-mon[42580]: pgmap v161: 137 pgs: 137 active+clean; 161 MiB data, 736 MiB used, 714 GiB / 715 GiB avail; 55 MiB/s rd, 44 MiB/s wr, 2.28k op/s 2024-08-17T17:12:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:11 smithi165 ceph-mon[34868]: osdmap e117: 8 total, 8 up, 8 in 2024-08-17T17:12:11.848 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:11 smithi136 ceph-mon[31034]: osdmap e117: 8 total, 8 up, 8 in 2024-08-17T17:12:11.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:11 smithi136 ceph-mon[42580]: osdmap e117: 8 total, 8 up, 8 in 2024-08-17T17:12:12.518 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 8388609~8388609 2024-08-17T17:12:12.529 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 0~1000 2024-08-17T17:12:12.529 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 1010~1000 2024-08-17T17:12:12.529 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap2 2024-08-17T17:12:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:12 smithi165 ceph-mon[34868]: osdmap e118: 8 total, 8 up, 8 in 2024-08-17T17:12:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:12 smithi165 ceph-mon[34868]: pgmap v164: 137 pgs: 137 active+clean; 161 MiB data, 736 MiB used, 714 GiB / 715 GiB avail; 93 MiB/s rd, 76 MiB/s wr, 2.60k op/s 2024-08-17T17:12:12.861 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:12 smithi136 ceph-mon[31034]: osdmap e118: 8 total, 8 up, 8 in 2024-08-17T17:12:12.861 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:12 smithi136 ceph-mon[31034]: pgmap v164: 137 pgs: 137 active+clean; 161 MiB data, 736 MiB used, 714 GiB / 715 GiB avail; 93 MiB/s rd, 76 MiB/s wr, 2.60k op/s 2024-08-17T17:12:12.861 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:12 smithi136 ceph-mon[42580]: osdmap e118: 8 total, 8 up, 8 in 2024-08-17T17:12:12.861 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:12 smithi136 ceph-mon[42580]: pgmap v164: 137 pgs: 137 active+clean; 161 MiB data, 736 MiB used, 714 GiB / 715 GiB avail; 93 MiB/s rd, 76 MiB/s wr, 2.60k op/s 2024-08-17T17:12:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:13 smithi165 ceph-mon[34868]: osdmap e119: 8 total, 8 up, 8 in 2024-08-17T17:12:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:13 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:13 smithi136 ceph-mon[31034]: osdmap e119: 8 total, 8 up, 8 in 2024-08-17T17:12:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:13 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:13.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:13 smithi136 ceph-mon[42580]: osdmap e119: 8 total, 8 up, 8 in 2024-08-17T17:12:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:13 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:14.529 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 8388609~1000 2024-08-17T17:12:14.534 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 16777228~1000 2024-08-17T17:12:14.534 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 167772160 2024-08-17T17:12:14.542 INFO:tasks.workunit.client.0.smithi136.stdout:write: 2 83886079~8388609 2024-08-17T17:12:14.558 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap3 2024-08-17T17:12:14.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:14 smithi136 ceph-mon[31034]: osdmap e120: 8 total, 8 up, 8 in 2024-08-17T17:12:14.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:14 smithi136 ceph-mon[31034]: pgmap v167: 137 pgs: 137 active+clean; 225 MiB data, 945 MiB used, 714 GiB / 715 GiB avail; 35 MiB/s rd, 20 MiB/s wr, 456 op/s 2024-08-17T17:12:14.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:14 smithi136 ceph-mon[42580]: osdmap e120: 8 total, 8 up, 8 in 2024-08-17T17:12:14.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:14 smithi136 ceph-mon[42580]: pgmap v167: 137 pgs: 137 active+clean; 225 MiB data, 945 MiB used, 714 GiB / 715 GiB avail; 35 MiB/s rd, 20 MiB/s wr, 456 op/s 2024-08-17T17:12:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:14 smithi165 ceph-mon[34868]: osdmap e120: 8 total, 8 up, 8 in 2024-08-17T17:12:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:14 smithi165 ceph-mon[34868]: pgmap v167: 137 pgs: 137 active+clean; 225 MiB data, 945 MiB used, 714 GiB / 715 GiB avail; 35 MiB/s rd, 20 MiB/s wr, 456 op/s 2024-08-17T17:12:15.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:15 smithi136 ceph-mon[31034]: osdmap e121: 8 total, 8 up, 8 in 2024-08-17T17:12:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:15 smithi136 ceph-mon[42580]: osdmap e121: 8 total, 8 up, 8 in 2024-08-17T17:12:16.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:15 smithi165 ceph-mon[34868]: osdmap e121: 8 total, 8 up, 8 in 2024-08-17T17:12:16.525 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:16 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:12:16] "GET /metrics HTTP/1.1" 200 38521 "" "Prometheus/2.43.0" 2024-08-17T17:12:16.538 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:16.557 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 83886079~1 2024-08-17T17:12:16.565 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:12:16.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:16 smithi136 ceph-mon[31034]: osdmap e122: 8 total, 8 up, 8 in 2024-08-17T17:12:16.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:16 smithi136 ceph-mon[31034]: pgmap v170: 137 pgs: 137 active+clean; 225 MiB data, 945 MiB used, 714 GiB / 715 GiB avail; 35 MiB/s rd, 20 MiB/s wr, 456 op/s 2024-08-17T17:12:16.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:16 smithi136 ceph-mon[42580]: osdmap e122: 8 total, 8 up, 8 in 2024-08-17T17:12:16.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:16 smithi136 ceph-mon[42580]: pgmap v170: 137 pgs: 137 active+clean; 225 MiB data, 945 MiB used, 714 GiB / 715 GiB avail; 35 MiB/s rd, 20 MiB/s wr, 456 op/s 2024-08-17T17:12:17.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:16 smithi165 ceph-mon[34868]: osdmap e122: 8 total, 8 up, 8 in 2024-08-17T17:12:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:16 smithi165 ceph-mon[34868]: pgmap v170: 137 pgs: 137 active+clean; 225 MiB data, 945 MiB used, 714 GiB / 715 GiB avail; 35 MiB/s rd, 20 MiB/s wr, 456 op/s 2024-08-17T17:12:17.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:17 smithi136 ceph-mon[31034]: osdmap e123: 8 total, 8 up, 8 in 2024-08-17T17:12:17.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:17 smithi136 ceph-mon[42580]: osdmap e123: 8 total, 8 up, 8 in 2024-08-17T17:12:18.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:17 smithi165 ceph-mon[34868]: osdmap e123: 8 total, 8 up, 8 in 2024-08-17T17:12:18.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:18 smithi136 ceph-mon[31034]: osdmap e124: 8 total, 8 up, 8 in 2024-08-17T17:12:18.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:18 smithi136 ceph-mon[31034]: pgmap v173: 137 pgs: 137 active+clean; 225 MiB data, 945 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:12:18.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:18 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:12:18.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:18 smithi136 ceph-mon[42580]: osdmap e124: 8 total, 8 up, 8 in 2024-08-17T17:12:18.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:18 smithi136 ceph-mon[42580]: pgmap v173: 137 pgs: 137 active+clean; 225 MiB data, 945 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:12:18.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:18 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:12:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:18 smithi165 ceph-mon[34868]: osdmap e124: 8 total, 8 up, 8 in 2024-08-17T17:12:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:18 smithi165 ceph-mon[34868]: pgmap v173: 137 pgs: 137 active+clean; 225 MiB data, 945 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:12:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:18 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:12:19.910 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:12:19.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:19 smithi136 ceph-mon[31034]: osdmap e125: 8 total, 8 up, 8 in 2024-08-17T17:12:19.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:19 smithi136 ceph-mon[42580]: osdmap e125: 8 total, 8 up, 8 in 2024-08-17T17:12:20.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:19 smithi165 ceph-mon[34868]: osdmap e125: 8 total, 8 up, 8 in 2024-08-17T17:12:20.043 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:12:20.912 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:20 smithi165 ceph-mon[34868]: osdmap e126: 8 total, 8 up, 8 in 2024-08-17T17:12:20.912 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:20 smithi165 ceph-mon[34868]: pgmap v176: 137 pgs: 137 active+clean; 249 MiB data, 1002 MiB used, 714 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 169 op/s 2024-08-17T17:12:20.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:20 smithi136 ceph-mon[31034]: osdmap e126: 8 total, 8 up, 8 in 2024-08-17T17:12:20.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:20 smithi136 ceph-mon[31034]: pgmap v176: 137 pgs: 137 active+clean; 249 MiB data, 1002 MiB used, 714 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 169 op/s 2024-08-17T17:12:20.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:20 smithi136 ceph-mon[42580]: osdmap e126: 8 total, 8 up, 8 in 2024-08-17T17:12:20.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:20 smithi136 ceph-mon[42580]: pgmap v176: 137 pgs: 137 active+clean; 249 MiB data, 1002 MiB used, 714 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 169 op/s 2024-08-17T17:12:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:21 smithi136 ceph-mon[31034]: osdmap e127: 8 total, 8 up, 8 in 2024-08-17T17:12:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:21 smithi136 ceph-mon[42580]: osdmap e127: 8 total, 8 up, 8 in 2024-08-17T17:12:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:21 smithi165 ceph-mon[34868]: osdmap e127: 8 total, 8 up, 8 in 2024-08-17T17:12:22.826 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.Snaps (12035 ms) 2024-08-17T17:12:22.827 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.SnapsOtherPool 2024-08-17T17:12:22.855 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:22.864 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 0~8388609 2024-08-17T17:12:22.913 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:12:22.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:22 smithi136 ceph-mon[31034]: osdmap e128: 8 total, 8 up, 8 in 2024-08-17T17:12:22.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:22 smithi136 ceph-mon[31034]: pgmap v179: 137 pgs: 137 active+clean; 249 MiB data, 1002 MiB used, 714 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 169 op/s 2024-08-17T17:12:22.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:22 smithi136 ceph-mon[42580]: osdmap e128: 8 total, 8 up, 8 in 2024-08-17T17:12:22.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:22 smithi136 ceph-mon[42580]: pgmap v179: 137 pgs: 137 active+clean; 249 MiB data, 1002 MiB used, 714 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 169 op/s 2024-08-17T17:12:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:22 smithi165 ceph-mon[34868]: osdmap e128: 8 total, 8 up, 8 in 2024-08-17T17:12:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:22 smithi165 ceph-mon[34868]: pgmap v179: 137 pgs: 137 active+clean; 249 MiB data, 1002 MiB used, 714 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 169 op/s 2024-08-17T17:12:23.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:23 smithi136 ceph-mon[31034]: osdmap e129: 8 total, 8 up, 8 in 2024-08-17T17:12:23.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:23 smithi136 ceph-mon[42580]: osdmap e129: 8 total, 8 up, 8 in 2024-08-17T17:12:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:23 smithi165 ceph-mon[34868]: osdmap e129: 8 total, 8 up, 8 in 2024-08-17T17:12:24.586 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 8388609~8388609 2024-08-17T17:12:24.599 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 0~1000 2024-08-17T17:12:24.599 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 1010~1000 2024-08-17T17:12:24.599 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap2 2024-08-17T17:12:24.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:24 smithi136 ceph-mon[31034]: osdmap e130: 8 total, 8 up, 8 in 2024-08-17T17:12:24.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:24 smithi136 ceph-mon[31034]: pgmap v182: 137 pgs: 137 active+clean; 265 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 70 MiB/s rd, 10 MiB/s wr, 493 op/s 2024-08-17T17:12:24.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:24 smithi136 ceph-mon[42580]: osdmap e130: 8 total, 8 up, 8 in 2024-08-17T17:12:24.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:24 smithi136 ceph-mon[42580]: pgmap v182: 137 pgs: 137 active+clean; 265 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 70 MiB/s rd, 10 MiB/s wr, 493 op/s 2024-08-17T17:12:25.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:24 smithi165 ceph-mon[34868]: osdmap e130: 8 total, 8 up, 8 in 2024-08-17T17:12:25.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:24 smithi165 ceph-mon[34868]: pgmap v182: 137 pgs: 137 active+clean; 265 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 70 MiB/s rd, 10 MiB/s wr, 493 op/s 2024-08-17T17:12:25.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:25 smithi136 ceph-mon[31034]: osdmap e131: 8 total, 8 up, 8 in 2024-08-17T17:12:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:25 smithi136 ceph-mon[42580]: osdmap e131: 8 total, 8 up, 8 in 2024-08-17T17:12:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:25 smithi165 ceph-mon[34868]: osdmap e131: 8 total, 8 up, 8 in 2024-08-17T17:12:26.585 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:12:26] "GET /metrics HTTP/1.1" 200 38650 "" "Prometheus/2.43.0" 2024-08-17T17:12:26.598 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 8388609~1000 2024-08-17T17:12:26.604 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 16777228~1000 2024-08-17T17:12:26.604 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 167772160 2024-08-17T17:12:26.612 INFO:tasks.workunit.client.0.smithi136.stdout:write: 2 83886079~8388609 2024-08-17T17:12:26.628 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap3 2024-08-17T17:12:26.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:26 smithi136 ceph-mon[31034]: osdmap e132: 8 total, 8 up, 8 in 2024-08-17T17:12:26.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:26 smithi136 ceph-mon[31034]: pgmap v185: 137 pgs: 137 active+clean; 265 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 70 MiB/s rd, 10 MiB/s wr, 493 op/s 2024-08-17T17:12:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:26 smithi136 ceph-mon[42580]: osdmap e132: 8 total, 8 up, 8 in 2024-08-17T17:12:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:26 smithi136 ceph-mon[42580]: pgmap v185: 137 pgs: 137 active+clean; 265 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 70 MiB/s rd, 10 MiB/s wr, 493 op/s 2024-08-17T17:12:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:26 smithi165 ceph-mon[34868]: osdmap e132: 8 total, 8 up, 8 in 2024-08-17T17:12:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:26 smithi165 ceph-mon[34868]: pgmap v185: 137 pgs: 137 active+clean; 265 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 70 MiB/s rd, 10 MiB/s wr, 493 op/s 2024-08-17T17:12:27.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:27 smithi136 ceph-mon[31034]: osdmap e133: 8 total, 8 up, 8 in 2024-08-17T17:12:27.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:27 smithi136 ceph-mon[31034]: osdmap e134: 8 total, 8 up, 8 in 2024-08-17T17:12:27.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:27 smithi136 ceph-mon[42580]: osdmap e133: 8 total, 8 up, 8 in 2024-08-17T17:12:27.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:27 smithi136 ceph-mon[42580]: osdmap e134: 8 total, 8 up, 8 in 2024-08-17T17:12:28.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:27 smithi165 ceph-mon[34868]: osdmap e133: 8 total, 8 up, 8 in 2024-08-17T17:12:28.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:27 smithi165 ceph-mon[34868]: osdmap e134: 8 total, 8 up, 8 in 2024-08-17T17:12:28.496 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:28.520 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 83886079~1 2024-08-17T17:12:28.527 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 0~8388609 2024-08-17T17:12:28.536 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 16777218~8388609 2024-08-17T17:12:28.590 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:12:28.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:28 smithi136 ceph-mon[31034]: pgmap v188: 137 pgs: 137 active+clean; 265 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:12:28.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:28 smithi136 ceph-mon[31034]: osdmap e135: 8 total, 8 up, 8 in 2024-08-17T17:12:28.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:28 smithi136 ceph-mon[42580]: pgmap v188: 137 pgs: 137 active+clean; 265 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:12:28.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:28 smithi136 ceph-mon[42580]: osdmap e135: 8 total, 8 up, 8 in 2024-08-17T17:12:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:28 smithi165 ceph-mon[34868]: pgmap v188: 137 pgs: 137 active+clean; 265 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:12:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:28 smithi165 ceph-mon[34868]: osdmap e135: 8 total, 8 up, 8 in 2024-08-17T17:12:30.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:30 smithi136 ceph-mon[31034]: osdmap e136: 8 total, 8 up, 8 in 2024-08-17T17:12:30.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:30 smithi136 ceph-mon[31034]: pgmap v191: 137 pgs: 137 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 155 KiB/s rd, 16 MiB/s wr, 228 op/s 2024-08-17T17:12:30.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:30 smithi136 ceph-mon[42580]: osdmap e136: 8 total, 8 up, 8 in 2024-08-17T17:12:30.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:30 smithi136 ceph-mon[42580]: pgmap v191: 137 pgs: 137 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 155 KiB/s rd, 16 MiB/s wr, 228 op/s 2024-08-17T17:12:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:30 smithi165 ceph-mon[34868]: osdmap e136: 8 total, 8 up, 8 in 2024-08-17T17:12:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:30 smithi165 ceph-mon[34868]: pgmap v191: 137 pgs: 137 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 155 KiB/s rd, 16 MiB/s wr, 228 op/s 2024-08-17T17:12:31.895 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:31 smithi165 ceph-mon[34868]: osdmap e137: 8 total, 8 up, 8 in 2024-08-17T17:12:31.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:31 smithi136 ceph-mon[31034]: osdmap e137: 8 total, 8 up, 8 in 2024-08-17T17:12:31.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:31 smithi136 ceph-mon[42580]: osdmap e137: 8 total, 8 up, 8 in 2024-08-17T17:12:32.014 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:12:32.087 INFO:teuthology.orchestra.run.smithi136.stdout:true 2024-08-17T17:12:32.195 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:12:32.638 INFO:teuthology.orchestra.run.smithi136.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-17T17:12:32.639 INFO:teuthology.orchestra.run.smithi136.stdout:alertmanager.a smithi136 *:9093,9094 running (2m) 91s ago 3m 17.8M - 0.25.0 c8568f914cd2 83b6bfc5306f 2024-08-17T17:12:32.639 INFO:teuthology.orchestra.run.smithi136.stdout:grafana.a smithi165 *:3000 running (3m) 31s ago 3m 87.9M - 9.4.7 954c08fa6188 be80a687c402 2024-08-17T17:12:32.639 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi136.ltsuqd smithi136 running (2m) 91s ago 2m 15.7M - 18.2.4-719-gced649f8 c870753d67c2 faaa06f2950c 2024-08-17T17:12:32.639 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi165.bmrbhh smithi165 running (2m) 31s ago 2m 14.8M - 18.2.4-719-gced649f8 c870753d67c2 c74b00d929cf 2024-08-17T17:12:32.639 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.x smithi165 *:8443,8765 running (7m) 31s ago 7m 439M - 18.2.4-719-gced649f8 c870753d67c2 5723e013d909 2024-08-17T17:12:32.639 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.y smithi136 *:9283,8765 running (9m) 91s ago 9m 484M - 18.2.4-719-gced649f8 c870753d67c2 f74bb81f3a53 2024-08-17T17:12:32.639 INFO:teuthology.orchestra.run.smithi136.stdout:mon.a smithi136 running (9m) 91s ago 9m 44.9M 2048M 18.2.4-719-gced649f8 c870753d67c2 cc4e5e5ff756 2024-08-17T17:12:32.639 INFO:teuthology.orchestra.run.smithi136.stdout:mon.b smithi165 running (7m) 31s ago 7m 41.4M 2048M 18.2.4-719-gced649f8 c870753d67c2 d587e16140e1 2024-08-17T17:12:32.639 INFO:teuthology.orchestra.run.smithi136.stdout:mon.c smithi136 running (7m) 91s ago 7m 38.6M 2048M 18.2.4-719-gced649f8 c870753d67c2 b4e0f3c9e485 2024-08-17T17:12:32.640 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.a smithi136 *:9100 running (3m) 91s ago 3m 15.1M - 1.5.0 0da6a335fe13 6aef82a780f5 2024-08-17T17:12:32.640 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.b smithi165 *:9100 running (2m) 31s ago 2m 15.5M - 1.5.0 0da6a335fe13 60b065dd0c2a 2024-08-17T17:12:32.640 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0 smithi136 running (7m) 91s ago 7m 44.3M 4016M 18.2.4-719-gced649f8 c870753d67c2 faab916734b0 2024-08-17T17:12:32.640 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1 smithi136 running (6m) 91s ago 6m 45.0M 4016M 18.2.4-719-gced649f8 c870753d67c2 861686b6ff82 2024-08-17T17:12:32.640 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2 smithi136 running (6m) 91s ago 6m 49.0M 4016M 18.2.4-719-gced649f8 c870753d67c2 640be4af4156 2024-08-17T17:12:32.640 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3 smithi136 running (5m) 91s ago 5m 47.3M 4016M 18.2.4-719-gced649f8 c870753d67c2 14b57a745d30 2024-08-17T17:12:32.640 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4 smithi165 running (5m) 31s ago 5m 55.4M 4016M 18.2.4-719-gced649f8 c870753d67c2 df366c73e1f2 2024-08-17T17:12:32.640 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5 smithi165 running (5m) 31s ago 5m 47.0M 4016M 18.2.4-719-gced649f8 c870753d67c2 1131665e203a 2024-08-17T17:12:32.640 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6 smithi165 running (4m) 31s ago 4m 54.7M 4016M 18.2.4-719-gced649f8 c870753d67c2 7bc5fd925293 2024-08-17T17:12:32.640 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7 smithi165 running (4m) 31s ago 4m 48.6M 4016M 18.2.4-719-gced649f8 c870753d67c2 18382e4ae3b8 2024-08-17T17:12:32.640 INFO:teuthology.orchestra.run.smithi136.stdout:prometheus.a smithi165 *:9095 running (2m) 31s ago 3m 45.0M - 2.43.0 a07b618ecd1d 1e7f77a17708 2024-08-17T17:12:32.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:32 smithi165 ceph-mon[34868]: osdmap e138: 8 total, 8 up, 8 in 2024-08-17T17:12:32.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:32 smithi165 ceph-mon[34868]: pgmap v194: 137 pgs: 137 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 155 KiB/s rd, 16 MiB/s wr, 228 op/s 2024-08-17T17:12:32.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:32 smithi136 ceph-mon[31034]: osdmap e138: 8 total, 8 up, 8 in 2024-08-17T17:12:32.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:32 smithi136 ceph-mon[31034]: pgmap v194: 137 pgs: 137 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 155 KiB/s rd, 16 MiB/s wr, 228 op/s 2024-08-17T17:12:32.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:32 smithi136 ceph-mon[42580]: osdmap e138: 8 total, 8 up, 8 in 2024-08-17T17:12:32.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:32 smithi136 ceph-mon[42580]: pgmap v194: 137 pgs: 137 active+clean; 337 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 155 KiB/s rd, 16 MiB/s wr, 228 op/s 2024-08-17T17:12:32.960 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:12:32.960 INFO:teuthology.orchestra.run.smithi136.stdout: "mon": { 2024-08-17T17:12:32.960 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 3 2024-08-17T17:12:32.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:12:32.960 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr": { 2024-08-17T17:12:32.960 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 2 2024-08-17T17:12:32.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:12:32.961 INFO:teuthology.orchestra.run.smithi136.stdout: "osd": { 2024-08-17T17:12:32.961 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 8 2024-08-17T17:12:32.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:12:32.961 INFO:teuthology.orchestra.run.smithi136.stdout: "mds": { 2024-08-17T17:12:32.961 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 2 2024-08-17T17:12:32.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:12:32.961 INFO:teuthology.orchestra.run.smithi136.stdout: "overall": { 2024-08-17T17:12:32.961 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 15 2024-08-17T17:12:32.961 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:12:32.961 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:12:33.229 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:12:33.229 INFO:teuthology.orchestra.run.smithi136.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d", 2024-08-17T17:12:33.230 INFO:teuthology.orchestra.run.smithi136.stdout: "in_progress": true, 2024-08-17T17:12:33.230 INFO:teuthology.orchestra.run.smithi136.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-08-17T17:12:33.230 INFO:teuthology.orchestra.run.smithi136.stdout: "services_complete": [], 2024-08-17T17:12:33.230 INFO:teuthology.orchestra.run.smithi136.stdout: "progress": "0/2 daemons upgraded", 2024-08-17T17:12:33.230 INFO:teuthology.orchestra.run.smithi136.stdout: "message": "Currently upgrading mgr daemons", 2024-08-17T17:12:33.230 INFO:teuthology.orchestra.run.smithi136.stdout: "is_paused": false 2024-08-17T17:12:33.230 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:12:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:33 smithi165 ceph-mon[34868]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:33 smithi165 ceph-mon[34868]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:33 smithi165 ceph-mon[34868]: osdmap e139: 8 total, 8 up, 8 in 2024-08-17T17:12:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:33 smithi165 ceph-mon[34868]: from='client.24863 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:33.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:33 smithi165 ceph-mon[34868]: Upgrade: Updating mgr.x 2024-08-17T17:12:33.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:33 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:33.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:33 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:12:33.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:33 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:12:33.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:33 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:12:33.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:33 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:12:33.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:33 smithi165 ceph-mon[34868]: Deploying daemon mgr.x on smithi165 2024-08-17T17:12:33.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:33 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:12:33.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:33 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/55010899' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:12:33.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[31034]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:33.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[31034]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:33.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[31034]: osdmap e139: 8 total, 8 up, 8 in 2024-08-17T17:12:33.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[31034]: from='client.24863 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:33.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[31034]: Upgrade: Updating mgr.x 2024-08-17T17:12:33.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:33.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:12:33.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:12:33.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:12:33.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:12:33.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[31034]: Deploying daemon mgr.x on smithi165 2024-08-17T17:12:33.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:12:33.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/55010899' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:12:33.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[42580]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:33.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[42580]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:33.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[42580]: osdmap e139: 8 total, 8 up, 8 in 2024-08-17T17:12:33.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[42580]: from='client.24863 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:33.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[42580]: Upgrade: Updating mgr.x 2024-08-17T17:12:33.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:33.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:12:33.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:12:33.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:12:33.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:12:33.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[42580]: Deploying daemon mgr.x on smithi165 2024-08-17T17:12:33.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:12:33.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:33 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/55010899' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:12:34.761 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:34 smithi165 systemd[1]: Stopping Ceph mgr.x for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:12:34.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:34 smithi165 ceph-mon[34868]: from='client.24875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:34.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:34 smithi165 ceph-mon[34868]: osdmap e140: 8 total, 8 up, 8 in 2024-08-17T17:12:34.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:34 smithi165 ceph-mon[34868]: pgmap v197: 137 pgs: 137 active+clean; 373 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 46 MiB/s rd, 10 MiB/s wr, 380 op/s 2024-08-17T17:12:34.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:34 smithi136 ceph-mon[31034]: from='client.24875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:34.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:34 smithi136 ceph-mon[31034]: osdmap e140: 8 total, 8 up, 8 in 2024-08-17T17:12:34.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:34 smithi136 ceph-mon[31034]: pgmap v197: 137 pgs: 137 active+clean; 373 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 46 MiB/s rd, 10 MiB/s wr, 380 op/s 2024-08-17T17:12:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:34 smithi136 ceph-mon[42580]: from='client.24875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:34 smithi136 ceph-mon[42580]: osdmap e140: 8 total, 8 up, 8 in 2024-08-17T17:12:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:34 smithi136 ceph-mon[42580]: pgmap v197: 137 pgs: 137 active+clean; 373 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 46 MiB/s rd, 10 MiB/s wr, 380 op/s 2024-08-17T17:12:34.970 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.SnapsOtherPool (12144 ms) 2024-08-17T17:12:34.970 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.SnapsDataPool 2024-08-17T17:12:34.998 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:35.008 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 0~8388609 2024-08-17T17:12:35.066 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:12:35.097 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:34 smithi165 podman[70571]: 2024-08-17 17:12:34.803093412 +0000 UTC m=+0.159046063 container died 5723e013d9096416f3d0e8fd2a8532584036e1588a99c227182261fed22745e5 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=reef-ced649f, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-08-17T17:12:35.455 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:35 smithi165 podman[70571]: 2024-08-17 17:12:35.178232047 +0000 UTC m=+0.534184685 container cleanup 5723e013d9096416f3d0e8fd2a8532584036e1588a99c227182261fed22745e5 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-ced649f, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-08-17T17:12:35.455 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:35 smithi165 bash[70571]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x 2024-08-17T17:12:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:35 smithi165 ceph-mon[34868]: osdmap e141: 8 total, 8 up, 8 in 2024-08-17T17:12:35.761 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:35 smithi165 podman[70588]: 2024-08-17 17:12:35.45456369 +0000 UTC m=+0.655367196 container remove 5723e013d9096416f3d0e8fd2a8532584036e1588a99c227182261fed22745e5 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=reef-ced649f, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-08-17T17:12:35.761 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:35 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-08-17T17:12:35.761 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:35 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-08-17T17:12:35.761 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:35 smithi165 systemd[1]: Stopped Ceph mgr.x for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:12:35.761 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:35 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@mgr.x.service: Consumed 10.881s CPU time. 2024-08-17T17:12:35.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:35 smithi136 ceph-mon[31034]: osdmap e141: 8 total, 8 up, 8 in 2024-08-17T17:12:35.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:35 smithi136 ceph-mon[42580]: osdmap e141: 8 total, 8 up, 8 in 2024-08-17T17:12:36.134 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:35 smithi165 systemd[1]: Starting Ceph mgr.x for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:12:36.134 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:36 smithi165 podman[70725]: 2024-08-17 17:12:35.915740699 +0000 UTC m=+0.017449862 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:12:36.134 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:36 smithi165 podman[70725]: 2024-08-17 17:12:36.014144985 +0000 UTC m=+0.115854138 container create 7329884ca5de8a39a5c087b3fb36ad08ae81adb79f7b13b2ef70d795093e9059 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2) 2024-08-17T17:12:36.638 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:36 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:12:36] "GET /metrics HTTP/1.1" 200 38657 "" "Prometheus/2.43.0" 2024-08-17T17:12:36.638 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:36 smithi136 ceph-mon[31034]: osdmap e142: 8 total, 8 up, 8 in 2024-08-17T17:12:36.641 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:36 smithi165 podman[70725]: 2024-08-17 17:12:36.286605443 +0000 UTC m=+0.388314596 container init 7329884ca5de8a39a5c087b3fb36ad08ae81adb79f7b13b2ef70d795093e9059 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, GIT_BRANCH=HEAD) 2024-08-17T17:12:36.641 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:36 smithi165 podman[70725]: 2024-08-17 17:12:36.291827116 +0000 UTC m=+0.393536283 container start 7329884ca5de8a39a5c087b3fb36ad08ae81adb79f7b13b2ef70d795093e9059 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=main-cd8bdbe, ceph=True) 2024-08-17T17:12:36.641 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:36 smithi165 ceph-mgr[70763]: -- 172.21.15.165:0/3081497695 <== mon.1 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55889eaff0e0 con 0x55889eadd000 2024-08-17T17:12:36.641 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:36 smithi165 bash[70725]: 7329884ca5de8a39a5c087b3fb36ad08ae81adb79f7b13b2ef70d795093e9059 2024-08-17T17:12:36.641 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:36 smithi165 systemd[1]: Started Ceph mgr.x for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:12:36.653 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 8388609~8388609 2024-08-17T17:12:36.674 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 0~1000 2024-08-17T17:12:36.674 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 1010~1000 2024-08-17T17:12:36.674 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap2 2024-08-17T17:12:36.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:36 smithi136 ceph-mon[42580]: osdmap e142: 8 total, 8 up, 8 in 2024-08-17T17:12:36.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:36 smithi136 ceph-mon[42580]: pgmap v200: 137 pgs: 137 active+clean; 373 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 46 MiB/s rd, 10 MiB/s wr, 380 op/s 2024-08-17T17:12:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:36 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:36 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:36 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:12:36.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:36 smithi136 ceph-mon[31034]: pgmap v200: 137 pgs: 137 active+clean; 373 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 46 MiB/s rd, 10 MiB/s wr, 380 op/s 2024-08-17T17:12:36.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:36 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:36.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:36 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:36.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:36 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:12:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:36 smithi165 ceph-mon[34868]: osdmap e142: 8 total, 8 up, 8 in 2024-08-17T17:12:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:36 smithi165 ceph-mon[34868]: pgmap v200: 137 pgs: 137 active+clean; 373 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 46 MiB/s rd, 10 MiB/s wr, 380 op/s 2024-08-17T17:12:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:36 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:36 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:36 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:12:37.938 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:37 smithi165 ceph-mon[34868]: osdmap e143: 8 total, 8 up, 8 in 2024-08-17T17:12:37.938 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:37 smithi165 ceph-mon[34868]: osdmap e144: 8 total, 8 up, 8 in 2024-08-17T17:12:37.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:37 smithi136 ceph-mon[31034]: osdmap e143: 8 total, 8 up, 8 in 2024-08-17T17:12:37.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:37 smithi136 ceph-mon[31034]: osdmap e144: 8 total, 8 up, 8 in 2024-08-17T17:12:37.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:37 smithi136 ceph-mon[42580]: osdmap e143: 8 total, 8 up, 8 in 2024-08-17T17:12:37.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:37 smithi136 ceph-mon[42580]: osdmap e144: 8 total, 8 up, 8 in 2024-08-17T17:12:38.500 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 8388609~1000 2024-08-17T17:12:38.506 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 16777228~1000 2024-08-17T17:12:38.506 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 167772160 2024-08-17T17:12:38.515 INFO:tasks.workunit.client.0.smithi136.stdout:write: 2 83886079~8388609 2024-08-17T17:12:38.535 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap3 2024-08-17T17:12:38.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:38 smithi136 ceph-mon[31034]: pgmap v203: 137 pgs: 137 active+clean; 373 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:12:38.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:38 smithi136 ceph-mon[31034]: osdmap e145: 8 total, 8 up, 8 in 2024-08-17T17:12:38.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:38 smithi136 ceph-mon[42580]: pgmap v203: 137 pgs: 137 active+clean; 373 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:12:38.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:38 smithi136 ceph-mon[42580]: osdmap e145: 8 total, 8 up, 8 in 2024-08-17T17:12:39.004 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:38 smithi165 ceph-mon[34868]: pgmap v203: 137 pgs: 137 active+clean; 373 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:12:39.004 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:38 smithi165 ceph-mon[34868]: osdmap e145: 8 total, 8 up, 8 in 2024-08-17T17:12:40.508 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:40.529 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 83886079~1 2024-08-17T17:12:40.537 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:12:40.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:40 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:40.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:40 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:40.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:40 smithi136 ceph-mon[42580]: osdmap e146: 8 total, 8 up, 8 in 2024-08-17T17:12:40.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:40 smithi136 ceph-mon[42580]: pgmap v206: 137 pgs: 137 active+clean; 393 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 36 MiB/s rd, 12 MiB/s wr, 183 op/s 2024-08-17T17:12:40.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:40 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:40.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:40 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:40.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:40 smithi136 ceph-mon[31034]: osdmap e146: 8 total, 8 up, 8 in 2024-08-17T17:12:40.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:40 smithi136 ceph-mon[31034]: pgmap v206: 137 pgs: 137 active+clean; 393 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 36 MiB/s rd, 12 MiB/s wr, 183 op/s 2024-08-17T17:12:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:40 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:40 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:40 smithi165 ceph-mon[34868]: osdmap e146: 8 total, 8 up, 8 in 2024-08-17T17:12:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:40 smithi165 ceph-mon[34868]: pgmap v206: 137 pgs: 137 active+clean; 393 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 36 MiB/s rd, 12 MiB/s wr, 183 op/s 2024-08-17T17:12:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:41 smithi165 ceph-mon[34868]: osdmap e147: 8 total, 8 up, 8 in 2024-08-17T17:12:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:41 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:41 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:41.761 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: [17/Aug/2024:17:12:41] ENGINE Bus STARTING 2024-08-17T17:12:41.761 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: CherryPy Checker: 2024-08-17T17:12:41.762 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: The Application mounted at '' has an empty config. 2024-08-17T17:12:41.762 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: 2024-08-17T17:12:41.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:41 smithi136 ceph-mon[31034]: osdmap e147: 8 total, 8 up, 8 in 2024-08-17T17:12:41.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:41 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:41.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:41 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:41.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:41 smithi136 ceph-mon[42580]: osdmap e147: 8 total, 8 up, 8 in 2024-08-17T17:12:41.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:41 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:41.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:41 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:42.103 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: [17/Aug/2024:17:12:41] ENGINE Serving on http://:::9283 2024-08-17T17:12:42.103 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: [17/Aug/2024:17:12:41] ENGINE Bus STARTED 2024-08-17T17:12:42.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:42 smithi165 ceph-mon[34868]: osdmap e148: 8 total, 8 up, 8 in 2024-08-17T17:12:42.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:42 smithi165 ceph-mon[34868]: Standby manager daemon x restarted 2024-08-17T17:12:42.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:42 smithi165 ceph-mon[34868]: Standby manager daemon x started 2024-08-17T17:12:42.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:42 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:12:42.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:42 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:12:42.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:42 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:12:42.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:42 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:12:42.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:42 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:42.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:42 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:42.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:42 smithi165 ceph-mon[34868]: pgmap v209: 137 pgs: 137 active+clean; 393 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 36 MiB/s rd, 12 MiB/s wr, 183 op/s 2024-08-17T17:12:42.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:42 smithi165 ceph-mon[34868]: osdmap e149: 8 total, 8 up, 8 in 2024-08-17T17:12:42.862 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:42 smithi136 ceph-mon[42580]: osdmap e148: 8 total, 8 up, 8 in 2024-08-17T17:12:42.862 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:42 smithi136 ceph-mon[42580]: Standby manager daemon x restarted 2024-08-17T17:12:42.862 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:42 smithi136 ceph-mon[42580]: Standby manager daemon x started 2024-08-17T17:12:42.862 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:42 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:12:42.862 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:42 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:12:42.863 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:42 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:12:42.863 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:42 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:12:42.863 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:42 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:42.863 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:42 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:42.863 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:42 smithi136 ceph-mon[42580]: pgmap v209: 137 pgs: 137 active+clean; 393 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 36 MiB/s rd, 12 MiB/s wr, 183 op/s 2024-08-17T17:12:42.863 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:42 smithi136 ceph-mon[42580]: osdmap e149: 8 total, 8 up, 8 in 2024-08-17T17:12:42.863 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:42 smithi136 ceph-mon[31034]: osdmap e148: 8 total, 8 up, 8 in 2024-08-17T17:12:42.863 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:42 smithi136 ceph-mon[31034]: Standby manager daemon x restarted 2024-08-17T17:12:42.863 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:42 smithi136 ceph-mon[31034]: Standby manager daemon x started 2024-08-17T17:12:42.863 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:42 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:12:42.863 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:42 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:12:42.864 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:42 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:12:42.864 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:42 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:12:42.864 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:42 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:42.864 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:42 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:42.864 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:42 smithi136 ceph-mon[31034]: pgmap v209: 137 pgs: 137 active+clean; 393 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 36 MiB/s rd, 12 MiB/s wr, 183 op/s 2024-08-17T17:12:42.864 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:42 smithi136 ceph-mon[31034]: osdmap e149: 8 total, 8 up, 8 in 2024-08-17T17:12:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:43 smithi165 ceph-mon[34868]: mgrmap e21: y(active, since 3m), standbys: x 2024-08-17T17:12:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:43 smithi165 ceph-mon[34868]: osdmap e150: 8 total, 8 up, 8 in 2024-08-17T17:12:43.875 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:43 smithi136 ceph-mon[31034]: mgrmap e21: y(active, since 3m), standbys: x 2024-08-17T17:12:43.875 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:43 smithi136 ceph-mon[31034]: osdmap e150: 8 total, 8 up, 8 in 2024-08-17T17:12:43.875 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:43 smithi136 ceph-mon[42580]: mgrmap e21: y(active, since 3m), standbys: x 2024-08-17T17:12:43.875 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:43 smithi136 ceph-mon[42580]: osdmap e150: 8 total, 8 up, 8 in 2024-08-17T17:12:43.884 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:12:44.032 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:12:44.858 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:44 smithi165 ceph-mon[34868]: pgmap v212: 137 pgs: 137 active+clean; 393 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 32 MiB/s rd, 12 KiB/s wr, 288 op/s 2024-08-17T17:12:44.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:44 smithi136 ceph-mon[31034]: pgmap v212: 137 pgs: 137 active+clean; 393 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 32 MiB/s rd, 12 KiB/s wr, 288 op/s 2024-08-17T17:12:44.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:44 smithi136 ceph-mon[42580]: pgmap v212: 137 pgs: 137 active+clean; 393 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 32 MiB/s rd, 12 KiB/s wr, 288 op/s 2024-08-17T17:12:45.511 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: [17/Aug/2024:17:12:45] ENGINE Bus STOPPING 2024-08-17T17:12:45.620 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:45.327+0000 7fc24b35e640 -1 mgr handle_mgr_map I was active but no longer am 2024-08-17T17:12:45.620 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ignoring --setuser ceph since I am not root 2024-08-17T17:12:45.621 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ignoring --setgroup ceph since I am not root 2024-08-17T17:12:45.621 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:45.474+0000 7f944b291040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-17T17:12:45.621 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:45.619+0000 7f944b291040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:12:45.621 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: osdmap e151: 8 total, 8 up, 8 in 2024-08-17T17:12:45.621 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:45.621 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:45.621 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:12:45.622 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:12:45.622 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:45.622 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:12:45.622 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-08-17T17:12:45.622 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-08-17T17:12:45.622 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: Activating manager daemon x 2024-08-17T17:12:45.622 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: osdmap e152: 8 total, 8 up, 8 in 2024-08-17T17:12:45.622 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.14442 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-08-17T17:12:45.622 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: mgrmap e22: x(active, starting, since 0.00960304s) 2024-08-17T17:12:45.622 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:12:45.622 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:12:45.622 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:12:45.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi165.bmrbhh"}]: dispatch 2024-08-17T17:12:45.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi136.ltsuqd"}]: dispatch 2024-08-17T17:12:45.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-17T17:12:45.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:12:45.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:12:45.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:12:45.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:12:45.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:12:45.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:12:45.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:12:45.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:12:45.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-17T17:12:45.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-17T17:12:45.624 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-17T17:12:45.624 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: osdmap e151: 8 total, 8 up, 8 in 2024-08-17T17:12:45.624 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:45.624 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:45.624 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:12:45.624 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:12:45.624 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:45.624 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:12:45.624 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-08-17T17:12:45.624 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-08-17T17:12:45.624 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: Activating manager daemon x 2024-08-17T17:12:45.624 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: osdmap e152: 8 total, 8 up, 8 in 2024-08-17T17:12:45.625 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.14442 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-08-17T17:12:45.625 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: mgrmap e22: x(active, starting, since 0.00960304s) 2024-08-17T17:12:45.625 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:12:45.625 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:12:45.625 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:12:45.625 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi165.bmrbhh"}]: dispatch 2024-08-17T17:12:45.625 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi136.ltsuqd"}]: dispatch 2024-08-17T17:12:45.625 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-17T17:12:45.625 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:12:45.625 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:12:45.625 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:12:45.625 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:12:45.625 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:12:45.625 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:12:45.626 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:12:45.626 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:12:45.626 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-17T17:12:45.626 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-17T17:12:45.626 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-17T17:12:45.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: osdmap e151: 8 total, 8 up, 8 in 2024-08-17T17:12:45.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:45.815 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:45.815 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:12:45.815 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:12:45.815 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' 2024-08-17T17:12:45.815 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:12:45.815 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.14442 172.21.15.136:0/539403150' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-08-17T17:12:45.815 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-08-17T17:12:45.815 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: Activating manager daemon x 2024-08-17T17:12:45.815 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: osdmap e152: 8 total, 8 up, 8 in 2024-08-17T17:12:45.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.14442 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-08-17T17:12:45.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: mgrmap e22: x(active, starting, since 0.00960304s) 2024-08-17T17:12:45.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:12:45.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:12:45.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:12:45.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi165.bmrbhh"}]: dispatch 2024-08-17T17:12:45.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi136.ltsuqd"}]: dispatch 2024-08-17T17:12:45.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-17T17:12:45.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:12:45.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:12:45.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:12:45.817 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:12:45.817 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:12:45.817 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:12:45.817 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:12:45.817 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:12:45.817 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-17T17:12:45.817 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-17T17:12:45.817 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-17T17:12:45.925 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:45.686+0000 7f944b291040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-17T17:12:46.196 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:45.924+0000 7f944b291040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-17T17:12:46.196 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:45.986+0000 7f944b291040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-17T17:12:46.261 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: [17/Aug/2024:17:12:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-08-17T17:12:46.261 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:45 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: [17/Aug/2024:17:12:45] ENGINE Bus STOPPED 2024-08-17T17:12:46.625 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.SnapsDataPool (11655 ms) 2024-08-17T17:12:46.625 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.SnapsShrinkAfterPrepare 2024-08-17T17:12:46.656 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:46.668 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 0~8388609 2024-08-17T17:12:46.734 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:12:46.758 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: 2024-08-17T17:12:46.453+0000 7f471a7be640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:12:46.758 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: 2024-08-17T17:12:46.453+0000 7f471a7be640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:12:46.758 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: 2024-08-17T17:12:46.453+0000 7f471a7be640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:12:46.758 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: 2024-08-17T17:12:46.453+0000 7f471a7be640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:12:46.758 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: 2024-08-17T17:12:46.453+0000 7f471a7be640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:12:46.759 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: 2024-08-17T17:12:46.454+0000 7f471e7c6640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:12:46.759 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: 2024-08-17T17:12:46.454+0000 7f471e7c6640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:12:46.759 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: 2024-08-17T17:12:46.454+0000 7f471e7c6640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:12:46.759 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: 2024-08-17T17:12:46.454+0000 7f471e7c6640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:12:46.759 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: 2024-08-17T17:12:46.454+0000 7f471e7c6640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:12:46.759 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: [17/Aug/2024:17:12:46] ENGINE Bus STARTING 2024-08-17T17:12:46.759 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-mon[34868]: Manager daemon x is now available 2024-08-17T17:12:46.759 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-mon[34868]: osdmap e153: 8 total, 8 up, 8 in 2024-08-17T17:12:46.759 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:46.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:46.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:46.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:46.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:46.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:12:46.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:12:46.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:12:46.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:12:46.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:12:46.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-08-17T17:12:46.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-08-17T17:12:46.795 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[42580]: Manager daemon x is now available 2024-08-17T17:12:46.795 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[42580]: osdmap e153: 8 total, 8 up, 8 in 2024-08-17T17:12:46.795 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:46.795 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:46.796 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:46.796 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:46.796 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:46.796 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:12:46.796 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:12:46.796 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:12:46.796 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:12:46.796 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:12:46.796 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-08-17T17:12:46.796 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-08-17T17:12:46.796 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[31034]: Manager daemon x is now available 2024-08-17T17:12:46.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[31034]: osdmap e153: 8 total, 8 up, 8 in 2024-08-17T17:12:46.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:46.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:46.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:46.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:46.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:46.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:12:46.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:12:46.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:12:46.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:12:46.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:12:46.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-08-17T17:12:46.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:46 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-08-17T17:12:47.011 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: [17/Aug/2024:17:12:46] ENGINE Serving on http://:::9283 2024-08-17T17:12:47.011 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: [17/Aug/2024:17:12:46] ENGINE Bus STARTED 2024-08-17T17:12:47.134 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:47.017+0000 7f944b291040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-17T17:12:47.134 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:47.133+0000 7f944b291040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-17T17:12:47.403 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:47.189+0000 7f944b291040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-17T17:12:47.403 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:47.402+0000 7f944b291040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-17T17:12:47.678 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:47.463+0000 7f944b291040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-17T17:12:47.678 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:47.529+0000 7f944b291040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-17T17:12:47.678 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:47.677+0000 7f944b291040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:12:47.679 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:47 smithi136 ceph-mon[42580]: Checking for cert/key for grafana.a 2024-08-17T17:12:47.679 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:47 smithi136 ceph-mon[42580]: Migrating grafana.a cert to cert store 2024-08-17T17:12:47.679 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:47 smithi136 ceph-mon[42580]: Migrating grafana.a key to cert store 2024-08-17T17:12:47.679 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:47 smithi136 ceph-mon[42580]: mgrmap e23: x(active, since 1.21932s) 2024-08-17T17:12:47.679 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:47 smithi136 ceph-mon[42580]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:12:47.679 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:47 smithi136 ceph-mon[42580]: pgmap v4: 137 pgs: 137 active+clean; 393 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:12:47.679 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:47 smithi136 ceph-mon[42580]: Deploying cephadm binary to smithi165 2024-08-17T17:12:47.679 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:47 smithi136 ceph-mon[42580]: Deploying cephadm binary to smithi136 2024-08-17T17:12:47.679 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:47 smithi136 ceph-mon[31034]: Checking for cert/key for grafana.a 2024-08-17T17:12:47.680 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:47 smithi136 ceph-mon[31034]: Migrating grafana.a cert to cert store 2024-08-17T17:12:47.680 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:47 smithi136 ceph-mon[31034]: Migrating grafana.a key to cert store 2024-08-17T17:12:47.680 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:47 smithi136 ceph-mon[31034]: mgrmap e23: x(active, since 1.21932s) 2024-08-17T17:12:47.680 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:47 smithi136 ceph-mon[31034]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:12:47.680 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:47 smithi136 ceph-mon[31034]: pgmap v4: 137 pgs: 137 active+clean; 393 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:12:47.680 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:47 smithi136 ceph-mon[31034]: Deploying cephadm binary to smithi165 2024-08-17T17:12:47.680 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:47 smithi136 ceph-mon[31034]: Deploying cephadm binary to smithi136 2024-08-17T17:12:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-mon[34868]: Checking for cert/key for grafana.a 2024-08-17T17:12:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-mon[34868]: Migrating grafana.a cert to cert store 2024-08-17T17:12:48.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-mon[34868]: Migrating grafana.a key to cert store 2024-08-17T17:12:48.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-mon[34868]: mgrmap e23: x(active, since 1.21932s) 2024-08-17T17:12:48.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-mon[34868]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:12:48.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-mon[34868]: pgmap v4: 137 pgs: 137 active+clean; 393 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:12:48.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-mon[34868]: Deploying cephadm binary to smithi165 2024-08-17T17:12:48.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-mon[34868]: Deploying cephadm binary to smithi136 2024-08-17T17:12:48.012 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: 2024-08-17T17:12:47.757+0000 7f47014cf640 -1 log_channel(cephadm) log [ERR] : [17/Aug/2024:17:12:47] ENGINE Error in HTTPServer.serve 2024-08-17T17:12:48.012 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: Traceback (most recent call last): 2024-08-17T17:12:48.012 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:12:48.012 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: self._connections.run(self.expiration_interval) 2024-08-17T17:12:48.013 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:12:48.013 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: self._run(expiration_interval) 2024-08-17T17:12:48.013 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:12:48.013 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:12:48.013 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:12:48.013 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:12:48.013 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:12:48.013 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: s = self.context.wrap_socket( 2024-08-17T17:12:48.013 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:12:48.013 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: return self.sslsocket_class._create( 2024-08-17T17:12:48.013 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:12:48.013 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: self.do_handshake() 2024-08-17T17:12:48.013 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:12:48.014 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: self._sslobj.do_handshake() 2024-08-17T17:12:48.014 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:12:48.014 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:12:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: 2024-08-17T17:12:48.417 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:48 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:48.104+0000 7f944b291040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-17T17:12:48.417 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:48 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:48.416+0000 7f944b291040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-17T17:12:48.562 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 8388609~8388609 2024-08-17T17:12:48.573 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 0~1000 2024-08-17T17:12:48.573 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 1010~1000 2024-08-17T17:12:48.573 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap2 2024-08-17T17:12:48.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:48 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:48.484+0000 7f944b291040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-17T17:12:48.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:48 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:48.657+0000 7f944b291040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-17T17:12:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:48 smithi136 ceph-mon[42580]: pgmap v5: 137 pgs: 137 active+clean; 393 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:12:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:48 smithi136 ceph-mon[42580]: osdmap e154: 8 total, 8 up, 8 in 2024-08-17T17:12:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:48 smithi136 ceph-mon[42580]: [17/Aug/2024:17:12:47] ENGINE Bus STARTING 2024-08-17T17:12:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:48 smithi136 ceph-mon[42580]: [17/Aug/2024:17:12:47] ENGINE Serving on https://172.21.15.165:7150 2024-08-17T17:12:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:48 smithi136 ceph-mon[42580]: [17/Aug/2024:17:12:47] ENGINE Error in HTTPServer.serve 2024-08-17T17:12:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout: Traceback (most recent call last): 2024-08-17T17:12:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:12:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:12:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:12:48.698 INFO:journalctl@ceph.mon.c.smithi136.stdout: self._run(expiration_interval) 2024-08-17T17:12:48.698 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:12:48.698 INFO:journalctl@ceph.mon.c.smithi136.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:12:48.698 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:12:48.698 INFO:journalctl@ceph.mon.c.smithi136.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:12:48.698 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:12:48.698 INFO:journalctl@ceph.mon.c.smithi136.stdout: s = self.context.wrap_socket( 2024-08-17T17:12:48.698 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:12:48.698 INFO:journalctl@ceph.mon.c.smithi136.stdout: return self.sslsocket_class._create( 2024-08-17T17:12:48.698 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:12:48.698 INFO:journalctl@ceph.mon.c.smithi136.stdout: self.do_handshake() 2024-08-17T17:12:48.698 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:12:48.698 INFO:journalctl@ceph.mon.c.smithi136.stdout: self._sslobj.do_handshake() 2024-08-17T17:12:48.699 INFO:journalctl@ceph.mon.c.smithi136.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:12:48.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:48 smithi136 ceph-mon[42580]: [17/Aug/2024:17:12:47] ENGINE Serving on http://172.21.15.165:8765 2024-08-17T17:12:48.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:48 smithi136 ceph-mon[42580]: [17/Aug/2024:17:12:47] ENGINE Bus STARTED 2024-08-17T17:12:48.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:48 smithi136 ceph-mon[31034]: pgmap v5: 137 pgs: 137 active+clean; 393 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:12:48.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:48 smithi136 ceph-mon[31034]: osdmap e154: 8 total, 8 up, 8 in 2024-08-17T17:12:48.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:48 smithi136 ceph-mon[31034]: [17/Aug/2024:17:12:47] ENGINE Bus STARTING 2024-08-17T17:12:48.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:48 smithi136 ceph-mon[31034]: [17/Aug/2024:17:12:47] ENGINE Serving on https://172.21.15.165:7150 2024-08-17T17:12:48.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:48 smithi136 ceph-mon[31034]: [17/Aug/2024:17:12:47] ENGINE Error in HTTPServer.serve 2024-08-17T17:12:48.699 INFO:journalctl@ceph.mon.a.smithi136.stdout: Traceback (most recent call last): 2024-08-17T17:12:48.699 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:12:48.699 INFO:journalctl@ceph.mon.a.smithi136.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:12:48.700 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:12:48.700 INFO:journalctl@ceph.mon.a.smithi136.stdout: self._run(expiration_interval) 2024-08-17T17:12:48.700 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:12:48.700 INFO:journalctl@ceph.mon.a.smithi136.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:12:48.700 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:12:48.700 INFO:journalctl@ceph.mon.a.smithi136.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:12:48.700 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:12:48.700 INFO:journalctl@ceph.mon.a.smithi136.stdout: s = self.context.wrap_socket( 2024-08-17T17:12:48.700 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:12:48.700 INFO:journalctl@ceph.mon.a.smithi136.stdout: return self.sslsocket_class._create( 2024-08-17T17:12:48.700 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:12:48.700 INFO:journalctl@ceph.mon.a.smithi136.stdout: self.do_handshake() 2024-08-17T17:12:48.700 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:12:48.700 INFO:journalctl@ceph.mon.a.smithi136.stdout: self._sslobj.do_handshake() 2024-08-17T17:12:48.700 INFO:journalctl@ceph.mon.a.smithi136.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:12:48.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:48 smithi136 ceph-mon[31034]: [17/Aug/2024:17:12:47] ENGINE Serving on http://172.21.15.165:8765 2024-08-17T17:12:48.701 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:48 smithi136 ceph-mon[31034]: [17/Aug/2024:17:12:47] ENGINE Bus STARTED 2024-08-17T17:12:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:48 smithi165 ceph-mon[34868]: pgmap v5: 137 pgs: 137 active+clean; 393 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:12:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:48 smithi165 ceph-mon[34868]: osdmap e154: 8 total, 8 up, 8 in 2024-08-17T17:12:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:48 smithi165 ceph-mon[34868]: [17/Aug/2024:17:12:47] ENGINE Bus STARTING 2024-08-17T17:12:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:48 smithi165 ceph-mon[34868]: [17/Aug/2024:17:12:47] ENGINE Serving on https://172.21.15.165:7150 2024-08-17T17:12:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:48 smithi165 ceph-mon[34868]: [17/Aug/2024:17:12:47] ENGINE Error in HTTPServer.serve 2024-08-17T17:12:48.762 INFO:journalctl@ceph.mon.b.smithi165.stdout: Traceback (most recent call last): 2024-08-17T17:12:48.762 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:12:48.762 INFO:journalctl@ceph.mon.b.smithi165.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:12:48.762 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:12:48.763 INFO:journalctl@ceph.mon.b.smithi165.stdout: self._run(expiration_interval) 2024-08-17T17:12:48.763 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:12:48.763 INFO:journalctl@ceph.mon.b.smithi165.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:12:48.763 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:12:48.763 INFO:journalctl@ceph.mon.b.smithi165.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:12:48.763 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:12:48.763 INFO:journalctl@ceph.mon.b.smithi165.stdout: s = self.context.wrap_socket( 2024-08-17T17:12:48.763 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:12:48.763 INFO:journalctl@ceph.mon.b.smithi165.stdout: return self.sslsocket_class._create( 2024-08-17T17:12:48.763 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:12:48.763 INFO:journalctl@ceph.mon.b.smithi165.stdout: self.do_handshake() 2024-08-17T17:12:48.763 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:12:48.763 INFO:journalctl@ceph.mon.b.smithi165.stdout: self._sslobj.do_handshake() 2024-08-17T17:12:48.764 INFO:journalctl@ceph.mon.b.smithi165.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:12:48.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:48 smithi165 ceph-mon[34868]: [17/Aug/2024:17:12:47] ENGINE Serving on http://172.21.15.165:8765 2024-08-17T17:12:48.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:48 smithi165 ceph-mon[34868]: [17/Aug/2024:17:12:47] ENGINE Bus STARTED 2024-08-17T17:12:49.252 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:48 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:48.990+0000 7f944b291040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-17T17:12:49.252 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:49.052+0000 7f944b291040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-17T17:12:49.252 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:49.239+0000 7f944b291040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-17T17:12:49.503 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:49.395+0000 7f944b291040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-17T17:12:49.503 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:49.459+0000 7f944b291040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-17T17:12:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:49 smithi165 ceph-mon[34868]: osdmap e155: 8 total, 8 up, 8 in 2024-08-17T17:12:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:49 smithi165 ceph-mon[34868]: mgrmap e24: x(active, since 3s) 2024-08-17T17:12:49.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:49 smithi136 ceph-mon[31034]: osdmap e155: 8 total, 8 up, 8 in 2024-08-17T17:12:49.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:49 smithi136 ceph-mon[31034]: mgrmap e24: x(active, since 3s) 2024-08-17T17:12:49.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:49 smithi136 ceph-mon[42580]: osdmap e155: 8 total, 8 up, 8 in 2024-08-17T17:12:49.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:49 smithi136 ceph-mon[42580]: mgrmap e24: x(active, since 3s) 2024-08-17T17:12:49.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:49.544+0000 7f944b291040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-17T17:12:49.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:49.610+0000 7f944b291040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-17T17:12:50.446 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:50 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:50.179+0000 7f944b291040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-17T17:12:50.446 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:50 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:50.343+0000 7f944b291040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-17T17:12:50.447 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:50 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:12:50] ENGINE Bus STARTING 2024-08-17T17:12:50.447 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:50 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: CherryPy Checker: 2024-08-17T17:12:50.447 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:50 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: The Application mounted at '' has an empty config. 2024-08-17T17:12:50.447 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:50 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: 2024-08-17T17:12:50.580 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 8388609~1000 2024-08-17T17:12:50.585 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 16777228~1000 2024-08-17T17:12:50.585 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 167772160 2024-08-17T17:12:50.595 INFO:tasks.workunit.client.0.smithi136.stdout:write: 2 83886079~8388609 2024-08-17T17:12:50.608 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap3 2024-08-17T17:12:50.797 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:50 smithi165 ceph-mon[34868]: pgmap v8: 137 pgs: 137 active+clean; 393 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:12:50.797 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:50 smithi165 ceph-mon[34868]: osdmap e156: 8 total, 8 up, 8 in 2024-08-17T17:12:50.797 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:50 smithi165 ceph-mon[34868]: Standby manager daemon y started 2024-08-17T17:12:50.797 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:50 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.136:0/1110320205' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-08-17T17:12:50.797 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:50 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.136:0/1110320205' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:12:50.797 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:50 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.136:0/1110320205' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-08-17T17:12:50.797 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:50 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.136:0/1110320205' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:12:50.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:50 smithi136 ceph-mon[31034]: pgmap v8: 137 pgs: 137 active+clean; 393 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:12:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:50 smithi136 ceph-mon[31034]: osdmap e156: 8 total, 8 up, 8 in 2024-08-17T17:12:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:50 smithi136 ceph-mon[31034]: Standby manager daemon y started 2024-08-17T17:12:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:50 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.136:0/1110320205' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-08-17T17:12:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:50 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.136:0/1110320205' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:12:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:50 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.136:0/1110320205' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-08-17T17:12:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:50 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.136:0/1110320205' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:12:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:50 smithi136 ceph-mon[42580]: pgmap v8: 137 pgs: 137 active+clean; 393 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:12:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:50 smithi136 ceph-mon[42580]: osdmap e156: 8 total, 8 up, 8 in 2024-08-17T17:12:50.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:50 smithi136 ceph-mon[42580]: Standby manager daemon y started 2024-08-17T17:12:50.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:50 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.136:0/1110320205' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-08-17T17:12:50.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:50 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.136:0/1110320205' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:12:50.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:50 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.136:0/1110320205' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-08-17T17:12:50.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:50 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.136:0/1110320205' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:12:50.948 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:50 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:12:50] ENGINE Serving on http://:::9283 2024-08-17T17:12:50.948 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:50 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: [17/Aug/2024:17:12:50] ENGINE Bus STARTED 2024-08-17T17:12:51.827 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:51 smithi165 ceph-mon[34868]: osdmap e157: 8 total, 8 up, 8 in 2024-08-17T17:12:51.827 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:51 smithi165 ceph-mon[34868]: mgrmap e25: x(active, since 5s), standbys: y 2024-08-17T17:12:51.827 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:51 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-17T17:12:51.827 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:51 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:51.827 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:51 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:51.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:51 smithi136 ceph-mon[31034]: osdmap e157: 8 total, 8 up, 8 in 2024-08-17T17:12:51.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:51 smithi136 ceph-mon[31034]: mgrmap e25: x(active, since 5s), standbys: y 2024-08-17T17:12:51.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:51 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-17T17:12:51.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:51 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:51 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:51 smithi136 ceph-mon[42580]: osdmap e157: 8 total, 8 up, 8 in 2024-08-17T17:12:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:51 smithi136 ceph-mon[42580]: mgrmap e25: x(active, since 5s), standbys: y 2024-08-17T17:12:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:51 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-17T17:12:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:51 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:51 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:52.542 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:52.562 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 83886079~1 2024-08-17T17:12:52.569 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:12:52.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[31034]: pgmap v11: 137 pgs: 137 active+clean; 393 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:12:52.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[31034]: osdmap e158: 8 total, 8 up, 8 in 2024-08-17T17:12:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:12:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:12:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:12:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:12:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:12:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:12:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:12:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:12:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[31034]: Adjusting osd_memory_target on smithi136 to 2480M 2024-08-17T17:12:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[31034]: osdmap e159: 8 total, 8 up, 8 in 2024-08-17T17:12:52.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[42580]: pgmap v11: 137 pgs: 137 active+clean; 393 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:12:52.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[42580]: osdmap e158: 8 total, 8 up, 8 in 2024-08-17T17:12:52.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:52.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:52.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:12:52.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:12:52.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:12:52.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:12:52.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:12:52.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:12:52.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:12:52.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:12:52.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[42580]: Adjusting osd_memory_target on smithi136 to 2480M 2024-08-17T17:12:52.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:52.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:52 smithi136 ceph-mon[42580]: osdmap e159: 8 total, 8 up, 8 in 2024-08-17T17:12:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:52 smithi165 ceph-mon[34868]: pgmap v11: 137 pgs: 137 active+clean; 393 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:12:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:52 smithi165 ceph-mon[34868]: osdmap e158: 8 total, 8 up, 8 in 2024-08-17T17:12:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:52 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:52 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:52 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:12:53.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:52 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:12:53.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:52 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:12:53.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:52 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:12:53.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:52 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:12:53.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:52 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:12:53.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:52 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:12:53.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:52 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:12:53.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:52 smithi165 ceph-mon[34868]: Adjusting osd_memory_target on smithi136 to 2480M 2024-08-17T17:12:53.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:52 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:53.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:52 smithi165 ceph-mon[34868]: osdmap e159: 8 total, 8 up, 8 in 2024-08-17T17:12:54.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:54 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:54.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:54 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:54.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:54 smithi165 ceph-mon[34868]: osdmap e160: 8 total, 8 up, 8 in 2024-08-17T17:12:54.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:54 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:54.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:54 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:54.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:54 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:54.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:54 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:54.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:54 smithi136 ceph-mon[31034]: osdmap e160: 8 total, 8 up, 8 in 2024-08-17T17:12:54.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:54 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:54.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:54 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:54 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:54 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:54 smithi136 ceph-mon[42580]: osdmap e160: 8 total, 8 up, 8 in 2024-08-17T17:12:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:54 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:54 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:12:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:55 smithi165 ceph-mon[34868]: pgmap v14: 137 pgs: 137 active+clean; 449 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.1 MiB/s rd, 12 MiB/s wr, 169 op/s 2024-08-17T17:12:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:55 smithi165 ceph-mon[34868]: osdmap e161: 8 total, 8 up, 8 in 2024-08-17T17:12:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:55 smithi136 ceph-mon[31034]: pgmap v14: 137 pgs: 137 active+clean; 449 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.1 MiB/s rd, 12 MiB/s wr, 169 op/s 2024-08-17T17:12:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:55 smithi136 ceph-mon[31034]: osdmap e161: 8 total, 8 up, 8 in 2024-08-17T17:12:55.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:55 smithi136 ceph-mon[42580]: pgmap v14: 137 pgs: 137 active+clean; 449 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.1 MiB/s rd, 12 MiB/s wr, 169 op/s 2024-08-17T17:12:55.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:55 smithi136 ceph-mon[42580]: osdmap e161: 8 total, 8 up, 8 in 2024-08-17T17:12:56.018 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 41943040 2024-08-17T17:12:56.039 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:12:56.184 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:12:56.587 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:12:56 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:12:56] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:12:56.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:56 smithi136 ceph-mon[31034]: pgmap v17: 137 pgs: 137 active+clean; 449 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.1 MiB/s rd, 12 MiB/s wr, 169 op/s 2024-08-17T17:12:56.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:56 smithi136 ceph-mon[31034]: osdmap e162: 8 total, 8 up, 8 in 2024-08-17T17:12:56.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:56 smithi136 ceph-mon[42580]: pgmap v17: 137 pgs: 137 active+clean; 449 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.1 MiB/s rd, 12 MiB/s wr, 169 op/s 2024-08-17T17:12:56.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:56 smithi136 ceph-mon[42580]: osdmap e162: 8 total, 8 up, 8 in 2024-08-17T17:12:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:56 smithi165 ceph-mon[34868]: pgmap v17: 137 pgs: 137 active+clean; 449 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.1 MiB/s rd, 12 MiB/s wr, 169 op/s 2024-08-17T17:12:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:56 smithi165 ceph-mon[34868]: osdmap e162: 8 total, 8 up, 8 in 2024-08-17T17:12:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:57 smithi165 ceph-mon[34868]: osdmap e163: 8 total, 8 up, 8 in 2024-08-17T17:12:57.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:57 smithi136 ceph-mon[31034]: osdmap e163: 8 total, 8 up, 8 in 2024-08-17T17:12:57.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:57 smithi136 ceph-mon[42580]: osdmap e163: 8 total, 8 up, 8 in 2024-08-17T17:12:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:58 smithi165 ceph-mon[34868]: pgmap v20: 137 pgs: 137 active+clean; 477 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 511 op/s 2024-08-17T17:12:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:58 smithi165 ceph-mon[34868]: osdmap e164: 8 total, 8 up, 8 in 2024-08-17T17:12:58.927 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.SnapsShrinkAfterPrepare (12302 ms) 2024-08-17T17:12:58.927 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.SnapsShrinkToZeroBeforePrepare 2024-08-17T17:12:58.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:58 smithi136 ceph-mon[42580]: pgmap v20: 137 pgs: 137 active+clean; 477 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 511 op/s 2024-08-17T17:12:58.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:58 smithi136 ceph-mon[42580]: osdmap e164: 8 total, 8 up, 8 in 2024-08-17T17:12:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:58 smithi136 ceph-mon[31034]: pgmap v20: 137 pgs: 137 active+clean; 477 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 511 op/s 2024-08-17T17:12:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:58 smithi136 ceph-mon[31034]: osdmap e164: 8 total, 8 up, 8 in 2024-08-17T17:12:58.957 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:12:58.968 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 0~8388609 2024-08-17T17:12:59.028 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:12:59.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:12:59 smithi136 ceph-mon[31034]: osdmap e165: 8 total, 8 up, 8 in 2024-08-17T17:12:59.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:12:59 smithi136 ceph-mon[42580]: osdmap e165: 8 total, 8 up, 8 in 2024-08-17T17:13:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:12:59 smithi165 ceph-mon[34868]: osdmap e165: 8 total, 8 up, 8 in 2024-08-17T17:13:00.632 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 8388609~8388609 2024-08-17T17:13:00.645 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 0~1000 2024-08-17T17:13:00.645 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 1010~1000 2024-08-17T17:13:00.645 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap2 2024-08-17T17:13:00.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:00 smithi136 ceph-mon[31034]: pgmap v23: 137 pgs: 137 active+clean; 477 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 511 op/s 2024-08-17T17:13:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:00 smithi136 ceph-mon[31034]: osdmap e166: 8 total, 8 up, 8 in 2024-08-17T17:13:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:00 smithi136 ceph-mon[42580]: pgmap v23: 137 pgs: 137 active+clean; 477 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 511 op/s 2024-08-17T17:13:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:00 smithi136 ceph-mon[42580]: osdmap e166: 8 total, 8 up, 8 in 2024-08-17T17:13:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:00 smithi165 ceph-mon[34868]: pgmap v23: 137 pgs: 137 active+clean; 477 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 511 op/s 2024-08-17T17:13:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:00 smithi165 ceph-mon[34868]: osdmap e166: 8 total, 8 up, 8 in 2024-08-17T17:13:01.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:01 smithi136 ceph-mon[31034]: osdmap e167: 8 total, 8 up, 8 in 2024-08-17T17:13:01.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:01 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:13:01.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:01 smithi136 ceph-mon[42580]: osdmap e167: 8 total, 8 up, 8 in 2024-08-17T17:13:01.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:01 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:13:01.967 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:01 smithi165 ceph-mon[34868]: osdmap e167: 8 total, 8 up, 8 in 2024-08-17T17:13:01.967 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:01 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:13:02.640 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 8388609~1000 2024-08-17T17:13:02.646 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 16777228~1000 2024-08-17T17:13:02.646 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 167772160 2024-08-17T17:13:02.655 INFO:tasks.workunit.client.0.smithi136.stdout:write: 2 83886079~8388609 2024-08-17T17:13:02.674 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap3 2024-08-17T17:13:02.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:02 smithi136 ceph-mon[31034]: pgmap v26: 137 pgs: 137 active+clean; 477 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:13:02.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:02 smithi136 ceph-mon[31034]: osdmap e168: 8 total, 8 up, 8 in 2024-08-17T17:13:02.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:02 smithi136 ceph-mon[42580]: pgmap v26: 137 pgs: 137 active+clean; 477 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:13:02.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:02 smithi136 ceph-mon[42580]: osdmap e168: 8 total, 8 up, 8 in 2024-08-17T17:13:03.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:02 smithi165 ceph-mon[34868]: pgmap v26: 137 pgs: 137 active+clean; 477 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:13:03.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:02 smithi165 ceph-mon[34868]: osdmap e168: 8 total, 8 up, 8 in 2024-08-17T17:13:03.539 INFO:teuthology.orchestra.run.smithi136.stdout:true 2024-08-17T17:13:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:03 smithi136 ceph-mon[31034]: osdmap e169: 8 total, 8 up, 8 in 2024-08-17T17:13:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:03 smithi136 ceph-mon[42580]: osdmap e169: 8 total, 8 up, 8 in 2024-08-17T17:13:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:03 smithi165 ceph-mon[34868]: osdmap e169: 8 total, 8 up, 8 in 2024-08-17T17:13:04.093 INFO:teuthology.orchestra.run.smithi136.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-17T17:13:04.093 INFO:teuthology.orchestra.run.smithi136.stdout:alertmanager.a smithi136 *:9093,9094 running (3m) 12s ago 4m 27.8M - 0.25.0 c8568f914cd2 83b6bfc5306f 2024-08-17T17:13:04.093 INFO:teuthology.orchestra.run.smithi136.stdout:grafana.a smithi165 *:3000 running (3m) 10s ago 3m 90.0M - 9.4.7 954c08fa6188 be80a687c402 2024-08-17T17:13:04.093 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi136.ltsuqd smithi136 running (2m) 12s ago 2m 17.4M - 18.2.4-719-gced649f8 c870753d67c2 faaa06f2950c 2024-08-17T17:13:04.093 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi165.bmrbhh smithi165 running (2m) 10s ago 2m 15.4M - 18.2.4-719-gced649f8 c870753d67c2 c74b00d929cf 2024-08-17T17:13:04.093 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.x smithi165 *:8443,9283,8765 running (28s) 10s ago 7m 492M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 7329884ca5de 2024-08-17T17:13:04.094 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.y smithi136 *:9283,8765 running (9m) 12s ago 9m 312M - 18.2.4-719-gced649f8 c870753d67c2 f74bb81f3a53 2024-08-17T17:13:04.094 INFO:teuthology.orchestra.run.smithi136.stdout:mon.a smithi136 running (9m) 12s ago 9m 50.1M 2048M 18.2.4-719-gced649f8 c870753d67c2 cc4e5e5ff756 2024-08-17T17:13:04.094 INFO:teuthology.orchestra.run.smithi136.stdout:mon.b smithi165 running (8m) 10s ago 8m 45.0M 2048M 18.2.4-719-gced649f8 c870753d67c2 d587e16140e1 2024-08-17T17:13:04.094 INFO:teuthology.orchestra.run.smithi136.stdout:mon.c smithi136 running (8m) 12s ago 8m 44.4M 2048M 18.2.4-719-gced649f8 c870753d67c2 b4e0f3c9e485 2024-08-17T17:13:04.094 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.a smithi136 *:9100 running (3m) 12s ago 3m 15.8M - 1.5.0 0da6a335fe13 6aef82a780f5 2024-08-17T17:13:04.094 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.b smithi165 *:9100 running (3m) 10s ago 3m 15.7M - 1.5.0 0da6a335fe13 60b065dd0c2a 2024-08-17T17:13:04.094 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0 smithi136 running (7m) 12s ago 7m 118M 2480M 18.2.4-719-gced649f8 c870753d67c2 faab916734b0 2024-08-17T17:13:04.094 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1 smithi136 running (7m) 12s ago 7m 108M 2480M 18.2.4-719-gced649f8 c870753d67c2 861686b6ff82 2024-08-17T17:13:04.094 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2 smithi136 running (6m) 12s ago 6m 108M 2480M 18.2.4-719-gced649f8 c870753d67c2 640be4af4156 2024-08-17T17:13:04.094 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3 smithi136 running (6m) 12s ago 6m 87.0M 2480M 18.2.4-719-gced649f8 c870753d67c2 14b57a745d30 2024-08-17T17:13:04.094 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4 smithi165 running (6m) 10s ago 6m 91.4M 4016M 18.2.4-719-gced649f8 c870753d67c2 df366c73e1f2 2024-08-17T17:13:04.094 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5 smithi165 running (5m) 10s ago 5m 84.9M 4016M 18.2.4-719-gced649f8 c870753d67c2 1131665e203a 2024-08-17T17:13:04.094 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6 smithi165 running (5m) 10s ago 5m 119M 4016M 18.2.4-719-gced649f8 c870753d67c2 7bc5fd925293 2024-08-17T17:13:04.094 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7 smithi165 running (4m) 10s ago 4m 87.1M 4016M 18.2.4-719-gced649f8 c870753d67c2 18382e4ae3b8 2024-08-17T17:13:04.095 INFO:teuthology.orchestra.run.smithi136.stdout:prometheus.a smithi165 *:9095 running (2m) 10s ago 4m 52.5M - 2.43.0 a07b618ecd1d 1e7f77a17708 2024-08-17T17:13:04.423 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:13:04.423 INFO:teuthology.orchestra.run.smithi136.stdout: "mon": { 2024-08-17T17:13:04.423 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 3 2024-08-17T17:13:04.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:13:04.423 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr": { 2024-08-17T17:13:04.423 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 1, 2024-08-17T17:13:04.423 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 1 2024-08-17T17:13:04.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:13:04.423 INFO:teuthology.orchestra.run.smithi136.stdout: "osd": { 2024-08-17T17:13:04.423 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 8 2024-08-17T17:13:04.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:13:04.424 INFO:teuthology.orchestra.run.smithi136.stdout: "mds": { 2024-08-17T17:13:04.424 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 2 2024-08-17T17:13:04.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:13:04.424 INFO:teuthology.orchestra.run.smithi136.stdout: "overall": { 2024-08-17T17:13:04.424 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 14, 2024-08-17T17:13:04.424 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 1 2024-08-17T17:13:04.424 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:13:04.424 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:13:04.657 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:13:04.676 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 83886079~1 2024-08-17T17:13:04.684 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 0 2024-08-17T17:13:04.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:04 smithi136 ceph-mon[31034]: pgmap v29: 137 pgs: 137 active+clean; 481 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 32 MiB/s rd, 8.0 MiB/s wr, 155 op/s 2024-08-17T17:13:04.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:04 smithi136 ceph-mon[31034]: from='client.24917 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:04.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:04 smithi136 ceph-mon[31034]: osdmap e170: 8 total, 8 up, 8 in 2024-08-17T17:13:04.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:04 smithi136 ceph-mon[31034]: from='client.15015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:04.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:04 smithi136 ceph-mon[31034]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:04.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:04 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/496460733' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:04.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:04 smithi136 ceph-mon[42580]: pgmap v29: 137 pgs: 137 active+clean; 481 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 32 MiB/s rd, 8.0 MiB/s wr, 155 op/s 2024-08-17T17:13:04.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:04 smithi136 ceph-mon[42580]: from='client.24917 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:04.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:04 smithi136 ceph-mon[42580]: osdmap e170: 8 total, 8 up, 8 in 2024-08-17T17:13:04.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:04 smithi136 ceph-mon[42580]: from='client.15015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:04.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:04 smithi136 ceph-mon[42580]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:04.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:04 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/496460733' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:04.700 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:13:04.721 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:13:04.721 INFO:teuthology.orchestra.run.smithi136.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d", 2024-08-17T17:13:04.721 INFO:teuthology.orchestra.run.smithi136.stdout: "in_progress": true, 2024-08-17T17:13:04.721 INFO:teuthology.orchestra.run.smithi136.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-08-17T17:13:04.722 INFO:teuthology.orchestra.run.smithi136.stdout: "services_complete": [], 2024-08-17T17:13:04.722 INFO:teuthology.orchestra.run.smithi136.stdout: "progress": "1/2 daemons upgraded", 2024-08-17T17:13:04.722 INFO:teuthology.orchestra.run.smithi136.stdout: "message": "", 2024-08-17T17:13:04.722 INFO:teuthology.orchestra.run.smithi136.stdout: "is_paused": false 2024-08-17T17:13:04.722 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:13:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:04 smithi165 ceph-mon[34868]: pgmap v29: 137 pgs: 137 active+clean; 481 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 32 MiB/s rd, 8.0 MiB/s wr, 155 op/s 2024-08-17T17:13:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:04 smithi165 ceph-mon[34868]: from='client.24917 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:04 smithi165 ceph-mon[34868]: osdmap e170: 8 total, 8 up, 8 in 2024-08-17T17:13:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:04 smithi165 ceph-mon[34868]: from='client.15015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:04 smithi165 ceph-mon[34868]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:04 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/496460733' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:05.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:13:04.674Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.136:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.136:8765: connect: connection refused" 2024-08-17T17:13:05.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:13:04.674Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.136:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.136:8765: connect: connection refused" 2024-08-17T17:13:05.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:13:04.674Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.136:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.136:8765: connect: connection refused" 2024-08-17T17:13:05.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:13:04.675Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.136:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.136:8765: connect: connection refused" 2024-08-17T17:13:05.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:04 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:13:04.675Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.136:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.136:8765: connect: connection refused" 2024-08-17T17:13:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[31034]: osdmap e171: 8 total, 8 up, 8 in 2024-08-17T17:13:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[31034]: from='client.24941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:13:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:13:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:13:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:13:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:13:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:13:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:13:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:13:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[31034]: Adjusting osd_memory_target on smithi165 to 2480M 2024-08-17T17:13:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:05.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:13:05.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:13:05.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[31034]: Updating smithi136:/etc/ceph/ceph.conf 2024-08-17T17:13:05.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[31034]: Updating smithi165:/etc/ceph/ceph.conf 2024-08-17T17:13:05.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[42580]: osdmap e171: 8 total, 8 up, 8 in 2024-08-17T17:13:05.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[42580]: from='client.24941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:05.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:05.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:05.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:13:05.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:13:05.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:13:05.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:13:05.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:13:05.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:13:05.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:13:05.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:13:05.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[42580]: Adjusting osd_memory_target on smithi165 to 2480M 2024-08-17T17:13:05.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:05.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:13:05.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:13:05.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[42580]: Updating smithi136:/etc/ceph/ceph.conf 2024-08-17T17:13:05.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:05 smithi136 ceph-mon[42580]: Updating smithi165:/etc/ceph/ceph.conf 2024-08-17T17:13:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:05 smithi165 ceph-mon[34868]: osdmap e171: 8 total, 8 up, 8 in 2024-08-17T17:13:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:05 smithi165 ceph-mon[34868]: from='client.24941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:05 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:05 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:05 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:13:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:05 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:13:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:05 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:13:06.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:05 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:13:06.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:05 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:13:06.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:05 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:13:06.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:05 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:13:06.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:05 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:13:06.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:05 smithi165 ceph-mon[34868]: Adjusting osd_memory_target on smithi165 to 2480M 2024-08-17T17:13:06.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:05 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:06.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:05 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:13:06.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:05 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:13:06.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:05 smithi165 ceph-mon[34868]: Updating smithi136:/etc/ceph/ceph.conf 2024-08-17T17:13:06.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:05 smithi165 ceph-mon[34868]: Updating smithi165:/etc/ceph/ceph.conf 2024-08-17T17:13:06.651 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[31284]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:13:06] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:13:06.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[42580]: pgmap v32: 137 pgs: 137 active+clean; 481 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 32 MiB/s rd, 8.0 MiB/s wr, 155 op/s 2024-08-17T17:13:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[42580]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:13:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[42580]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:13:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[42580]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:13:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[42580]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:13:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[42580]: osdmap e172: 8 total, 8 up, 8 in 2024-08-17T17:13:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[42580]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:13:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[42580]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:13:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:06.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[42580]: Reconfiguring prometheus.a (dependencies changed)... 2024-08-17T17:13:06.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[31034]: pgmap v32: 137 pgs: 137 active+clean; 481 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 32 MiB/s rd, 8.0 MiB/s wr, 155 op/s 2024-08-17T17:13:06.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[31034]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:13:06.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[31034]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:13:06.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[31034]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:13:06.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[31034]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:13:06.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[31034]: osdmap e172: 8 total, 8 up, 8 in 2024-08-17T17:13:06.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[31034]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:13:06.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[31034]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:13:06.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:06.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:06.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:06.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:06.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:06.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:06 smithi136 ceph-mon[31034]: Reconfiguring prometheus.a (dependencies changed)... 2024-08-17T17:13:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:06 smithi165 ceph-mon[34868]: pgmap v32: 137 pgs: 137 active+clean; 481 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 32 MiB/s rd, 8.0 MiB/s wr, 155 op/s 2024-08-17T17:13:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:06 smithi165 ceph-mon[34868]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:13:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:06 smithi165 ceph-mon[34868]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:13:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:06 smithi165 ceph-mon[34868]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:13:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:06 smithi165 ceph-mon[34868]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:13:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:06 smithi165 ceph-mon[34868]: osdmap e172: 8 total, 8 up, 8 in 2024-08-17T17:13:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:06 smithi165 ceph-mon[34868]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:13:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:06 smithi165 ceph-mon[34868]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:13:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:06 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:06 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:06 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:06 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:06 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:06 smithi165 ceph-mon[34868]: Reconfiguring prometheus.a (dependencies changed)... 2024-08-17T17:13:07.819 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:13:07.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:07 smithi136 ceph-mon[31034]: Reconfiguring daemon prometheus.a on smithi165 2024-08-17T17:13:07.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:07 smithi136 ceph-mon[31034]: osdmap e173: 8 total, 8 up, 8 in 2024-08-17T17:13:07.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:07 smithi136 ceph-mon[31034]: osdmap e174: 8 total, 8 up, 8 in 2024-08-17T17:13:07.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:07 smithi136 ceph-mon[42580]: Reconfiguring daemon prometheus.a on smithi165 2024-08-17T17:13:07.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:07 smithi136 ceph-mon[42580]: osdmap e173: 8 total, 8 up, 8 in 2024-08-17T17:13:07.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:07 smithi136 ceph-mon[42580]: osdmap e174: 8 total, 8 up, 8 in 2024-08-17T17:13:07.956 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:13:08.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:07 smithi165 ceph-mon[34868]: Reconfiguring daemon prometheus.a on smithi165 2024-08-17T17:13:08.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:07 smithi165 ceph-mon[34868]: osdmap e173: 8 total, 8 up, 8 in 2024-08-17T17:13:08.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:07 smithi165 ceph-mon[34868]: osdmap e174: 8 total, 8 up, 8 in 2024-08-17T17:13:08.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:08 smithi136 ceph-mon[31034]: pgmap v35: 137 pgs: 137 active+clean; 505 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 127 KiB/s rd, 4.0 MiB/s wr, 185 op/s 2024-08-17T17:13:08.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:08 smithi136 ceph-mon[42580]: pgmap v35: 137 pgs: 137 active+clean; 505 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 127 KiB/s rd, 4.0 MiB/s wr, 185 op/s 2024-08-17T17:13:09.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:08 smithi165 ceph-mon[34868]: pgmap v35: 137 pgs: 137 active+clean; 505 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 127 KiB/s rd, 4.0 MiB/s wr, 185 op/s 2024-08-17T17:13:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:09 smithi136 ceph-mon[31034]: osdmap e175: 8 total, 8 up, 8 in 2024-08-17T17:13:09.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:09 smithi136 ceph-mon[42580]: osdmap e175: 8 total, 8 up, 8 in 2024-08-17T17:13:09.971 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:09 smithi165 ceph-mon[34868]: osdmap e175: 8 total, 8 up, 8 in 2024-08-17T17:13:09.971 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:09 smithi165 systemd[1]: Stopping Ceph prometheus.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:13:10.261 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:09 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:13:09.970Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-08-17T17:13:10.261 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:09 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:13:09.970Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-08-17T17:13:10.261 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:09 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:13:09.970Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-08-17T17:13:10.262 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:09 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:13:09.970Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-08-17T17:13:10.262 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:09 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:13:09.970Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-08-17T17:13:10.262 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:09 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:13:09.970Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-08-17T17:13:10.262 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:09 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:13:09.970Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-08-17T17:13:10.262 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:09 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:13:09.970Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-08-17T17:13:10.262 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:09 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:13:09.970Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-08-17T17:13:10.262 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:09 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:13:09.970Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-08-17T17:13:10.262 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:09 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:13:09.970Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-08-17T17:13:10.262 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:09 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[67150]: ts=2024-08-17T17:13:09.971Z caller=main.go:1147 level=info msg="See you next time!" 2024-08-17T17:13:10.262 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:09 smithi165 podman[79051]: 2024-08-17 17:13:09.980520796 +0000 UTC m=+0.133082729 container died 1e7f77a177081fc1ae830587072d9be4369fb97dafbe29fa4310210ef49f6530 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:13:10.677 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:10 smithi165 podman[79051]: 2024-08-17 17:13:10.451135352 +0000 UTC m=+0.603697272 container cleanup 1e7f77a177081fc1ae830587072d9be4369fb97dafbe29fa4310210ef49f6530 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:13:10.678 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:10 smithi165 bash[79051]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a 2024-08-17T17:13:10.928 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.SnapsShrinkToZeroBeforePrepare (12001 ms) 2024-08-17T17:13:10.929 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.SnapsShrinkToZeroAfterPrepare 2024-08-17T17:13:10.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:10 smithi136 ceph-mon[42580]: pgmap v38: 137 pgs: 137 active+clean; 505 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 127 KiB/s rd, 4.0 MiB/s wr, 185 op/s 2024-08-17T17:13:10.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:10 smithi136 ceph-mon[42580]: osdmap e176: 8 total, 8 up, 8 in 2024-08-17T17:13:10.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:10 smithi136 ceph-mon[31034]: pgmap v38: 137 pgs: 137 active+clean; 505 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 127 KiB/s rd, 4.0 MiB/s wr, 185 op/s 2024-08-17T17:13:10.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:10 smithi136 ceph-mon[31034]: osdmap e176: 8 total, 8 up, 8 in 2024-08-17T17:13:10.958 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:13:10.968 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 0~8388609 2024-08-17T17:13:11.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:10 smithi165 ceph-mon[34868]: pgmap v38: 137 pgs: 137 active+clean; 505 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 127 KiB/s rd, 4.0 MiB/s wr, 185 op/s 2024-08-17T17:13:11.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:10 smithi165 ceph-mon[34868]: osdmap e176: 8 total, 8 up, 8 in 2024-08-17T17:13:11.011 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:10 smithi165 podman[79069]: 2024-08-17 17:13:10.705309582 +0000 UTC m=+0.726271811 container remove 1e7f77a177081fc1ae830587072d9be4369fb97dafbe29fa4310210ef49f6530 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:13:11.011 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:10 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-08-17T17:13:11.011 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:10 smithi165 systemd[1]: Stopped Ceph prometheus.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:13:11.011 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:10 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@prometheus.a.service: Consumed 1.812s CPU time. 2024-08-17T17:13:11.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:10 smithi165 systemd[1]: Starting Ceph prometheus.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:13:11.028 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:13:11.384 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 podman[79166]: 2024-08-17 17:13:11.023010989 +0000 UTC m=+0.021770671 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-08-17T17:13:11.384 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 podman[79166]: 2024-08-17 17:13:11.136350297 +0000 UTC m=+0.135109979 container create 19e69ad66bef9ff92948aed7f30a06686a7e8946774e528d1e7635cd66025cf3 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:13:11.384 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 podman[79166]: 2024-08-17 17:13:11.354774806 +0000 UTC m=+0.353534503 container init 19e69ad66bef9ff92948aed7f30a06686a7e8946774e528d1e7635cd66025cf3 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:13:11.384 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 podman[79166]: 2024-08-17 17:13:11.359456177 +0000 UTC m=+0.358215859 container start 19e69ad66bef9ff92948aed7f30a06686a7e8946774e528d1e7635cd66025cf3 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:13:11.686 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-mon[34868]: osdmap e177: 8 total, 8 up, 8 in 2024-08-17T17:13:11.687 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:11.687 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:11.687 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:13:11.687 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:13:11.687 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:11.383Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-08-17T17:13:11.687 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:11.384Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-08-17T17:13:11.687 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:11.384Z caller=main.go:570 level=info host_details="(Linux 5.14.0-480.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024 x86_64 smithi165 (none))" 2024-08-17T17:13:11.688 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:11.384Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-08-17T17:13:11.688 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:11.384Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-08-17T17:13:11.688 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:11.385Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-08-17T17:13:11.688 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:11.385Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-08-17T17:13:11.688 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:11.386Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-08-17T17:13:11.688 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:11.386Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-08-17T17:13:11.688 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:11.387Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-08-17T17:13:11.688 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:11.387Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.694µs 2024-08-17T17:13:11.688 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:11.387Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-08-17T17:13:11.688 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:11.390Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-08-17T17:13:11.688 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:11.396Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-08-17T17:13:11.688 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:11.396Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-08-17T17:13:11.688 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:11.396Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.071µs wal_replay_duration=8.706476ms wbl_replay_duration=120ns total_replay_duration=8.742359ms 2024-08-17T17:13:11.689 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:11.397Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-08-17T17:13:11.689 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:11.397Z caller=main.go:1029 level=info msg="TSDB started" 2024-08-17T17:13:11.689 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:11.397Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-08-17T17:13:11.689 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:11.420Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=22.643834ms db_storage=791ns remote_storage=1.171µs web_handler=367ns query_engine=553ns scrape=578.285µs scrape_sd=118.349µs notify=16.922µs notify_sd=18.496µs rules=21.540649ms tracing=5.425µs 2024-08-17T17:13:11.689 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:11.420Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-08-17T17:13:11.689 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:11.420Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-08-17T17:13:11.689 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 bash[79166]: 19e69ad66bef9ff92948aed7f30a06686a7e8946774e528d1e7635cd66025cf3 2024-08-17T17:13:11.689 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:11 smithi165 systemd[1]: Started Ceph prometheus.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:13:11.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:11 smithi136 ceph-mon[42580]: osdmap e177: 8 total, 8 up, 8 in 2024-08-17T17:13:11.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:11 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:11.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:11 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:11.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:11 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:13:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:11 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:13:11.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:11 smithi136 ceph-mon[31034]: osdmap e177: 8 total, 8 up, 8 in 2024-08-17T17:13:11.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:11 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:11.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:11 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:11.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:11 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:13:11.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:11 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:13:12.701 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 8388609~8388609 2024-08-17T17:13:12.713 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 0~1000 2024-08-17T17:13:12.713 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 1010~1000 2024-08-17T17:13:12.713 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap2 2024-08-17T17:13:12.912 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:12 smithi136 ceph-mon[31034]: pgmap v41: 137 pgs: 137 active+clean; 505 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:13:12.912 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:12 smithi136 ceph-mon[31034]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:13:12.912 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:12 smithi136 ceph-mon[31034]: Upgrade: Need to upgrade myself (mgr.x) 2024-08-17T17:13:12.912 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:12 smithi136 ceph-mon[31034]: Upgrade: Need to upgrade myself (mgr.x) 2024-08-17T17:13:12.912 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:12 smithi136 ceph-mon[31034]: osdmap e178: 8 total, 8 up, 8 in 2024-08-17T17:13:12.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:12 smithi136 ceph-mon[42580]: pgmap v41: 137 pgs: 137 active+clean; 505 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:13:12.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:12 smithi136 ceph-mon[42580]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:13:12.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:12 smithi136 ceph-mon[42580]: Upgrade: Need to upgrade myself (mgr.x) 2024-08-17T17:13:12.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:12 smithi136 ceph-mon[42580]: Upgrade: Need to upgrade myself (mgr.x) 2024-08-17T17:13:12.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:12 smithi136 ceph-mon[42580]: osdmap e178: 8 total, 8 up, 8 in 2024-08-17T17:13:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:12 smithi165 ceph-mon[34868]: pgmap v41: 137 pgs: 137 active+clean; 505 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:13:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:12 smithi165 ceph-mon[34868]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:13:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:12 smithi165 ceph-mon[34868]: Upgrade: Need to upgrade myself (mgr.x) 2024-08-17T17:13:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:12 smithi165 ceph-mon[34868]: Upgrade: Need to upgrade myself (mgr.x) 2024-08-17T17:13:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:12 smithi165 ceph-mon[34868]: osdmap e178: 8 total, 8 up, 8 in 2024-08-17T17:13:13.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:13 smithi136 ceph-mon[42580]: osdmap e179: 8 total, 8 up, 8 in 2024-08-17T17:13:13.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:13 smithi136 ceph-mon[42580]: Upgrade: Updating mgr.y 2024-08-17T17:13:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:13 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:13 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:13:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:13 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:13:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:13 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:13:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:13 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:13:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:13 smithi136 ceph-mon[42580]: Deploying daemon mgr.y on smithi136 2024-08-17T17:13:13.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:13 smithi136 ceph-mon[31034]: osdmap e179: 8 total, 8 up, 8 in 2024-08-17T17:13:13.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:13 smithi136 ceph-mon[31034]: Upgrade: Updating mgr.y 2024-08-17T17:13:13.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:13 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:13.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:13 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:13:13.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:13 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:13:13.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:13 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:13:13.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:13 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:13:13.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:13 smithi136 ceph-mon[31034]: Deploying daemon mgr.y on smithi136 2024-08-17T17:13:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:13 smithi165 ceph-mon[34868]: osdmap e179: 8 total, 8 up, 8 in 2024-08-17T17:13:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:13 smithi165 ceph-mon[34868]: Upgrade: Updating mgr.y 2024-08-17T17:13:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:13 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:13 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:13:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:13 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:13:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:13 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:13:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:13 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:13:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:13 smithi165 ceph-mon[34868]: Deploying daemon mgr.y on smithi136 2024-08-17T17:13:14.715 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 8388609~1000 2024-08-17T17:13:14.720 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 16777228~1000 2024-08-17T17:13:14.720 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 167772160 2024-08-17T17:13:14.728 INFO:tasks.workunit.client.0.smithi136.stdout:write: 2 83886079~8388609 2024-08-17T17:13:14.751 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap3 2024-08-17T17:13:14.890 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:14 smithi136 systemd[1]: Stopping Ceph mgr.y for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:13:14.891 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:14 smithi136 ceph-mon[42580]: pgmap v44: 137 pgs: 137 active+clean; 521 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 54 MiB/s rd, 10 MiB/s wr, 481 op/s 2024-08-17T17:13:14.891 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:14 smithi136 ceph-mon[42580]: osdmap e180: 8 total, 8 up, 8 in 2024-08-17T17:13:14.891 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:14 smithi136 ceph-mon[31034]: pgmap v44: 137 pgs: 137 active+clean; 521 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 54 MiB/s rd, 10 MiB/s wr, 481 op/s 2024-08-17T17:13:14.891 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:14 smithi136 ceph-mon[31034]: osdmap e180: 8 total, 8 up, 8 in 2024-08-17T17:13:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:14 smithi165 ceph-mon[34868]: pgmap v44: 137 pgs: 137 active+clean; 521 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 54 MiB/s rd, 10 MiB/s wr, 481 op/s 2024-08-17T17:13:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:14 smithi165 ceph-mon[34868]: osdmap e180: 8 total, 8 up, 8 in 2024-08-17T17:13:15.167 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:14 smithi136 podman[91440]: 2024-08-17 17:13:14.89005266 +0000 UTC m=+0.168363860 container died f74bb81f3a53c5eb2516d75214b8c56877f1a4e8ee014419a6da99081e0bfe1e (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=reef-ced649f, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-08-17T17:13:15.446 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:15 smithi136 podman[91440]: 2024-08-17 17:13:15.337189458 +0000 UTC m=+0.615500647 container cleanup f74bb81f3a53c5eb2516d75214b8c56877f1a4e8ee014419a6da99081e0bfe1e (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, RELEASE=reef-ced649f, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True) 2024-08-17T17:13:15.446 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:15 smithi136 bash[91440]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y 2024-08-17T17:13:15.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:15 smithi136 ceph-mon[31034]: osdmap e181: 8 total, 8 up, 8 in 2024-08-17T17:13:15.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:15 smithi136 podman[91457]: 2024-08-17 17:13:15.609349871 +0000 UTC m=+0.718903765 container remove f74bb81f3a53c5eb2516d75214b8c56877f1a4e8ee014419a6da99081e0bfe1e (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=reef-ced649f, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-08-17T17:13:15.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:15 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-08-17T17:13:15.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:15 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-08-17T17:13:15.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:15 smithi136 systemd[1]: Stopped Ceph mgr.y for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:13:15.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:15 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@mgr.y.service: Consumed 41.942s CPU time. 2024-08-17T17:13:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:15 smithi136 ceph-mon[42580]: osdmap e181: 8 total, 8 up, 8 in 2024-08-17T17:13:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:15 smithi165 ceph-mon[34868]: osdmap e181: 8 total, 8 up, 8 in 2024-08-17T17:13:16.308 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:15 smithi136 systemd[1]: Starting Ceph mgr.y for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:13:16.308 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:16 smithi136 podman[91590]: 2024-08-17 17:13:16.09108733 +0000 UTC m=+0.017304640 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:13:16.308 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:16 smithi136 podman[91590]: 2024-08-17 17:13:16.194105346 +0000 UTC m=+0.120322691 container create 9acdc6775e486c1a13f35bf433e99d16609c8fdccecae3c56dff42fb2ec39d35 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_CLEAN=True, RELEASE=main-cd8bdbe, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD) 2024-08-17T17:13:16.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:16 smithi136 podman[91590]: 2024-08-17 17:13:16.42207639 +0000 UTC m=+0.348293701 container init 9acdc6775e486c1a13f35bf433e99d16609c8fdccecae3c56dff42fb2ec39d35 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=main-cd8bdbe, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-08-17T17:13:16.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:16 smithi136 podman[91590]: 2024-08-17 17:13:16.427272013 +0000 UTC m=+0.353489314 container start 9acdc6775e486c1a13f35bf433e99d16609c8fdccecae3c56dff42fb2ec39d35 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-08-17T17:13:16.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:16 smithi136 ceph-mgr[91628]: -- 172.21.15.136:0/3132257480 <== mon.2 v2:172.21.15.136:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5616db8550e0 con 0x5616db832800 2024-08-17T17:13:16.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:16 smithi136 bash[91590]: 9acdc6775e486c1a13f35bf433e99d16609c8fdccecae3c56dff42fb2ec39d35 2024-08-17T17:13:16.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:16 smithi136 systemd[1]: Started Ceph mgr.y for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:13:16.739 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:13:16.759 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 83886079~1 2024-08-17T17:13:16.767 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:13:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:16 smithi165 ceph-mon[34868]: pgmap v47: 137 pgs: 137 active+clean; 521 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 54 MiB/s rd, 10 MiB/s wr, 481 op/s 2024-08-17T17:13:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:16 smithi165 ceph-mon[34868]: osdmap e182: 8 total, 8 up, 8 in 2024-08-17T17:13:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:16 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:13:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:16 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:16 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:16 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:13:17.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:16 smithi136 ceph-mon[31034]: pgmap v47: 137 pgs: 137 active+clean; 521 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 54 MiB/s rd, 10 MiB/s wr, 481 op/s 2024-08-17T17:13:17.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:16 smithi136 ceph-mon[31034]: osdmap e182: 8 total, 8 up, 8 in 2024-08-17T17:13:17.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:16 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:13:17.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:16 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:17.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:16 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:17.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:16 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:13:17.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:16 smithi136 ceph-mon[42580]: pgmap v47: 137 pgs: 137 active+clean; 521 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 54 MiB/s rd, 10 MiB/s wr, 481 op/s 2024-08-17T17:13:17.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:16 smithi136 ceph-mon[42580]: osdmap e182: 8 total, 8 up, 8 in 2024-08-17T17:13:17.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:16 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:13:17.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:16 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:17.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:16 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:17.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:16 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:13:17.985 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:17 smithi165 ceph-mon[34868]: osdmap e183: 8 total, 8 up, 8 in 2024-08-17T17:13:17.985 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:17 smithi165 ceph-mon[34868]: osdmap e184: 8 total, 8 up, 8 in 2024-08-17T17:13:18.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:17 smithi136 ceph-mon[31034]: osdmap e183: 8 total, 8 up, 8 in 2024-08-17T17:13:18.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:17 smithi136 ceph-mon[31034]: osdmap e184: 8 total, 8 up, 8 in 2024-08-17T17:13:18.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:17 smithi136 ceph-mon[42580]: osdmap e183: 8 total, 8 up, 8 in 2024-08-17T17:13:18.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:17 smithi136 ceph-mon[42580]: osdmap e184: 8 total, 8 up, 8 in 2024-08-17T17:13:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:18 smithi165 ceph-mon[34868]: pgmap v50: 137 pgs: 137 active+clean; 561 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-08-17T17:13:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:18 smithi165 ceph-mon[34868]: osdmap e185: 8 total, 8 up, 8 in 2024-08-17T17:13:19.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:18 smithi136 ceph-mon[42580]: pgmap v50: 137 pgs: 137 active+clean; 561 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-08-17T17:13:19.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:18 smithi136 ceph-mon[42580]: osdmap e185: 8 total, 8 up, 8 in 2024-08-17T17:13:19.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:18 smithi136 ceph-mon[31034]: pgmap v50: 137 pgs: 137 active+clean; 561 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-08-17T17:13:19.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:18 smithi136 ceph-mon[31034]: osdmap e185: 8 total, 8 up, 8 in 2024-08-17T17:13:19.908 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 0 2024-08-17T17:13:19.982 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:13:20.111 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:13:20.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:20 smithi165 ceph-mon[34868]: pgmap v53: 137 pgs: 137 active+clean; 561 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-08-17T17:13:20.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:20 smithi165 ceph-mon[34868]: osdmap e186: 8 total, 8 up, 8 in 2024-08-17T17:13:20.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:20 smithi136 ceph-mon[31034]: pgmap v53: 137 pgs: 137 active+clean; 561 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-08-17T17:13:20.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:20 smithi136 ceph-mon[31034]: osdmap e186: 8 total, 8 up, 8 in 2024-08-17T17:13:20.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:20 smithi136 ceph-mon[42580]: pgmap v53: 137 pgs: 137 active+clean; 561 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-08-17T17:13:20.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:20 smithi136 ceph-mon[42580]: osdmap e186: 8 total, 8 up, 8 in 2024-08-17T17:13:21.824 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:21 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:21.825 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:21 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:21.825 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:21 smithi165 ceph-mon[34868]: osdmap e187: 8 total, 8 up, 8 in 2024-08-17T17:13:21.825 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:21 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:21.825 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:21 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:21.825 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:21 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:21.825 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:21 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:21.900 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:21 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:21.900 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:21 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:21.900 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:21 smithi136 ceph-mon[31034]: osdmap e187: 8 total, 8 up, 8 in 2024-08-17T17:13:21.900 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:21 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:21.900 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:21 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:21.900 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:21 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:21.901 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:21 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:21.901 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:21 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:21.901 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:21 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:21.901 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:21 smithi136 ceph-mon[42580]: osdmap e187: 8 total, 8 up, 8 in 2024-08-17T17:13:21.901 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:21 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:21.901 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:21 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:21.901 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:21 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:21.901 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:21 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:22.196 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:21 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:13:21] ENGINE Bus STARTING 2024-08-17T17:13:22.196 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:21 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: CherryPy Checker: 2024-08-17T17:13:22.196 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:21 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: The Application mounted at '' has an empty config. 2024-08-17T17:13:22.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:21 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:13:22.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:22 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:13:22] ENGINE Serving on http://:::9283 2024-08-17T17:13:22.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:22 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:13:22] ENGINE Bus STARTED 2024-08-17T17:13:22.261 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:21 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:13:21] "GET /metrics HTTP/1.1" 200 38733 "" "Prometheus/2.43.0" 2024-08-17T17:13:22.943 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:22 smithi136 ceph-mon[42580]: pgmap v56: 137 pgs: 137 active+clean; 561 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:22.943 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:22 smithi136 ceph-mon[42580]: osdmap e188: 8 total, 8 up, 8 in 2024-08-17T17:13:22.943 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:22 smithi136 ceph-mon[42580]: Standby manager daemon y restarted 2024-08-17T17:13:22.943 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:22 smithi136 ceph-mon[42580]: Standby manager daemon y started 2024-08-17T17:13:22.943 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:22 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-08-17T17:13:22.943 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:22 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:13:22.943 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:22 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-08-17T17:13:22.943 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:22 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:13:22.944 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:22 smithi136 ceph-mon[31034]: pgmap v56: 137 pgs: 137 active+clean; 561 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:22.944 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:22 smithi136 ceph-mon[31034]: osdmap e188: 8 total, 8 up, 8 in 2024-08-17T17:13:22.944 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:22 smithi136 ceph-mon[31034]: Standby manager daemon y restarted 2024-08-17T17:13:22.944 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:22 smithi136 ceph-mon[31034]: Standby manager daemon y started 2024-08-17T17:13:22.944 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:22 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-08-17T17:13:22.944 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:22 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:13:22.944 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:22 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-08-17T17:13:22.944 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:22 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:13:22.976 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.SnapsShrinkToZeroAfterPrepare (12048 ms) 2024-08-17T17:13:22.976 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.SnapsExpandAfterPrepare 2024-08-17T17:13:23.005 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:13:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:22 smithi165 ceph-mon[34868]: pgmap v56: 137 pgs: 137 active+clean; 561 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:22 smithi165 ceph-mon[34868]: osdmap e188: 8 total, 8 up, 8 in 2024-08-17T17:13:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:22 smithi165 ceph-mon[34868]: Standby manager daemon y restarted 2024-08-17T17:13:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:22 smithi165 ceph-mon[34868]: Standby manager daemon y started 2024-08-17T17:13:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:22 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-08-17T17:13:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:22 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:13:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:22 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-08-17T17:13:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:22 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:13:23.015 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 0~8388609 2024-08-17T17:13:23.073 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:13:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:23 smithi165 ceph-mon[34868]: mgrmap e26: x(active, since 37s), standbys: y 2024-08-17T17:13:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:23 smithi165 ceph-mon[34868]: osdmap e189: 8 total, 8 up, 8 in 2024-08-17T17:13:24.102 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:23 smithi136 ceph-mon[31034]: mgrmap e26: x(active, since 37s), standbys: y 2024-08-17T17:13:24.102 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:23 smithi136 ceph-mon[31034]: osdmap e189: 8 total, 8 up, 8 in 2024-08-17T17:13:24.102 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:23 smithi136 ceph-mon[42580]: mgrmap e26: x(active, since 37s), standbys: y 2024-08-17T17:13:24.102 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:23 smithi136 ceph-mon[42580]: osdmap e189: 8 total, 8 up, 8 in 2024-08-17T17:13:24.598 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 8388609~8388609 2024-08-17T17:13:24.620 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 0~1000 2024-08-17T17:13:24.621 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 1010~1000 2024-08-17T17:13:24.621 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap2 2024-08-17T17:13:24.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:13:24] ENGINE Bus STOPPING 2024-08-17T17:13:25.010 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: pgmap v59: 137 pgs: 137 active+clean; 581 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 42 MiB/s rd, 6.0 MiB/s wr, 542 op/s 2024-08-17T17:13:25.011 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: osdmap e190: 8 total, 8 up, 8 in 2024-08-17T17:13:25.011 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:25.011 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:25.011 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:13:25.011 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:13:25.011 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:25.011 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:13:25.011 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-08-17T17:13:25.011 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-08-17T17:13:25.011 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: Activating manager daemon y 2024-08-17T17:13:25.011 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: osdmap e191: 8 total, 8 up, 8 in 2024-08-17T17:13:25.011 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.24667 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-08-17T17:13:25.012 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: mgrmap e27: y(active, starting, since 0.00786419s) 2024-08-17T17:13:25.012 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:13:25.012 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:13:25.012 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:13:25.012 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi165.bmrbhh"}]: dispatch 2024-08-17T17:13:25.012 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi136.ltsuqd"}]: dispatch 2024-08-17T17:13:25.012 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-17T17:13:25.012 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:13:25.012 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:13:25.012 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:13:25.012 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:13:25.012 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:13:25.012 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:13:25.012 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:13:25.013 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:13:25.013 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-17T17:13:25.013 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-17T17:13:25.013 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-17T17:13:25.013 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: pgmap v59: 137 pgs: 137 active+clean; 581 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 42 MiB/s rd, 6.0 MiB/s wr, 542 op/s 2024-08-17T17:13:25.013 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: osdmap e190: 8 total, 8 up, 8 in 2024-08-17T17:13:25.013 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:25.013 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:25.013 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:13:25.013 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:13:25.013 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:25.014 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:13:25.014 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-08-17T17:13:25.014 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-08-17T17:13:25.014 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: Activating manager daemon y 2024-08-17T17:13:25.014 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: osdmap e191: 8 total, 8 up, 8 in 2024-08-17T17:13:25.014 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.24667 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-08-17T17:13:25.014 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: mgrmap e27: y(active, starting, since 0.00786419s) 2024-08-17T17:13:25.014 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:13:25.014 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:13:25.014 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:13:25.014 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi165.bmrbhh"}]: dispatch 2024-08-17T17:13:25.014 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi136.ltsuqd"}]: dispatch 2024-08-17T17:13:25.014 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-17T17:13:25.014 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:13:25.015 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:13:25.015 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:13:25.015 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:13:25.015 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:13:25.015 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:13:25.015 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:13:25.015 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:13:25.015 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-17T17:13:25.015 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-17T17:13:25.015 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:24 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-17T17:13:25.016 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: 2024-08-17T17:13:24.587+0000 7f47aa448640 -1 mgr handle_mgr_map I was active but no longer am 2024-08-17T17:13:25.016 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: ignoring --setuser ceph since I am not root 2024-08-17T17:13:25.016 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: ignoring --setgroup ceph since I am not root 2024-08-17T17:13:25.016 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mgr[70763]: -- 172.21.15.165:0/2806860285 <== mon.1 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f6f47150e0 con 0x55f6f46f2800 2024-08-17T17:13:25.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: pgmap v59: 137 pgs: 137 active+clean; 581 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 42 MiB/s rd, 6.0 MiB/s wr, 542 op/s 2024-08-17T17:13:25.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: osdmap e190: 8 total, 8 up, 8 in 2024-08-17T17:13:25.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:25.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:25.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:13:25.017 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:13:25.017 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' 2024-08-17T17:13:25.017 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:13:25.017 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.24667 172.21.15.165:0/825685939' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-08-17T17:13:25.017 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-08-17T17:13:25.017 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: Activating manager daemon y 2024-08-17T17:13:25.017 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: osdmap e191: 8 total, 8 up, 8 in 2024-08-17T17:13:25.017 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.24667 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-08-17T17:13:25.017 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: mgrmap e27: y(active, starting, since 0.00786419s) 2024-08-17T17:13:25.017 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:13:25.017 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:13:25.017 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:13:25.018 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi165.bmrbhh"}]: dispatch 2024-08-17T17:13:25.018 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi136.ltsuqd"}]: dispatch 2024-08-17T17:13:25.018 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-17T17:13:25.018 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:13:25.018 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:13:25.018 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:13:25.018 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:13:25.018 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:13:25.018 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:13:25.018 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:13:25.018 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:13:25.018 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-17T17:13:25.019 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-17T17:13:25.019 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:24 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-17T17:13:25.413 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:13:25] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-08-17T17:13:25.413 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:13:25] ENGINE Bus STOPPED 2024-08-17T17:13:25.413 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:13:25.411+0000 7f835fab8640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:13:25.413 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:13:25.411+0000 7f835fab8640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:13:25.413 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:13:25.411+0000 7f835fab8640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:13:25.413 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:13:25.411+0000 7f835fab8640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:13:25.413 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:13:25.411+0000 7f835fab8640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:13:25.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:13:25.413+0000 7f835cab2640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:13:25.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:13:25.413+0000 7f835cab2640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:13:25.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:13:25.413+0000 7f835cab2640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:13:25.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:13:25.413+0000 7f835cab2640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:13:25.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:13:25.413+0000 7f835cab2640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:13:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:25 smithi165 ceph-mon[34868]: Manager daemon y is now available 2024-08-17T17:13:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:25 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:13:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:25 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:13:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:25 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:13:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:25 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:13:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:25 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:13:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:25 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:13:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:25 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:13:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:25 smithi165 ceph-mon[34868]: osdmap e192: 8 total, 8 up, 8 in 2024-08-17T17:13:26.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-mon[42580]: Manager daemon y is now available 2024-08-17T17:13:26.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:13:26.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:13:26.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:13:26.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:13:26.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:13:26.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:13:26.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:13:26.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-mon[42580]: osdmap e192: 8 total, 8 up, 8 in 2024-08-17T17:13:26.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-mon[31034]: Manager daemon y is now available 2024-08-17T17:13:26.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:13:26.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:13:26.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:13:26.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:13:26.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:13:26.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:13:26.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:13:26.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-mon[31034]: osdmap e192: 8 total, 8 up, 8 in 2024-08-17T17:13:26.198 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:13:25] ENGINE Bus STARTING 2024-08-17T17:13:26.198 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:13:25] ENGINE Serving on http://:::9283 2024-08-17T17:13:26.198 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:25 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:13:25] ENGINE Bus STARTED 2024-08-17T17:13:26.600 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 8388609~1000 2024-08-17T17:13:26.605 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 16777228~1000 2024-08-17T17:13:26.605 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 167772160 2024-08-17T17:13:26.614 INFO:tasks.workunit.client.0.smithi136.stdout:write: 2 83886079~8388609 2024-08-17T17:13:26.626 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap3 2024-08-17T17:13:26.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:13:26.551+0000 7f8343002640 -1 log_channel(cephadm) log [ERR] : [17/Aug/2024:17:13:26] ENGINE Error in HTTPServer.serve 2024-08-17T17:13:26.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: Traceback (most recent call last): 2024-08-17T17:13:26.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:13:26.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: self._connections.run(self.expiration_interval) 2024-08-17T17:13:26.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:13:26.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: self._run(expiration_interval) 2024-08-17T17:13:26.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:13:26.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:13:26.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:13:26.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:13:26.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:13:26.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: s = self.context.wrap_socket( 2024-08-17T17:13:26.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:13:26.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: return self.sslsocket_class._create( 2024-08-17T17:13:26.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:13:26.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: self.do_handshake() 2024-08-17T17:13:26.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:13:26.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: self._sslobj.do_handshake() 2024-08-17T17:13:26.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:13:26.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:13:27.154 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:26 smithi165 ceph-mon[34868]: mgrmap e28: y(active, since 1.18495s) 2024-08-17T17:13:27.155 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:26 smithi165 ceph-mon[34868]: [17/Aug/2024:17:13:26] ENGINE Bus STARTING 2024-08-17T17:13:27.155 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:26 smithi165 ceph-mon[34868]: osdmap e193: 8 total, 8 up, 8 in 2024-08-17T17:13:27.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-mon[42580]: mgrmap e28: y(active, since 1.18495s) 2024-08-17T17:13:27.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-mon[42580]: [17/Aug/2024:17:13:26] ENGINE Bus STARTING 2024-08-17T17:13:27.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-mon[42580]: osdmap e193: 8 total, 8 up, 8 in 2024-08-17T17:13:27.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-mon[31034]: mgrmap e28: y(active, since 1.18495s) 2024-08-17T17:13:27.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-mon[31034]: [17/Aug/2024:17:13:26] ENGINE Bus STARTING 2024-08-17T17:13:27.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:26 smithi136 ceph-mon[31034]: osdmap e193: 8 total, 8 up, 8 in 2024-08-17T17:13:28.033 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:27 smithi165 ceph-mon[34868]: [17/Aug/2024:17:13:26] ENGINE Serving on https://172.21.15.136:7150 2024-08-17T17:13:28.033 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:27 smithi165 ceph-mon[34868]: [17/Aug/2024:17:13:26] ENGINE Error in HTTPServer.serve 2024-08-17T17:13:28.034 INFO:journalctl@ceph.mon.b.smithi165.stdout: Traceback (most recent call last): 2024-08-17T17:13:28.034 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:13:28.034 INFO:journalctl@ceph.mon.b.smithi165.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:13:28.034 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:13:28.034 INFO:journalctl@ceph.mon.b.smithi165.stdout: self._run(expiration_interval) 2024-08-17T17:13:28.034 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:13:28.034 INFO:journalctl@ceph.mon.b.smithi165.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:13:28.034 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:13:28.034 INFO:journalctl@ceph.mon.b.smithi165.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:13:28.034 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:13:28.034 INFO:journalctl@ceph.mon.b.smithi165.stdout: s = self.context.wrap_socket( 2024-08-17T17:13:28.034 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:13:28.035 INFO:journalctl@ceph.mon.b.smithi165.stdout: return self.sslsocket_class._create( 2024-08-17T17:13:28.035 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:13:28.035 INFO:journalctl@ceph.mon.b.smithi165.stdout: self.do_handshake() 2024-08-17T17:13:28.035 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:13:28.035 INFO:journalctl@ceph.mon.b.smithi165.stdout: self._sslobj.do_handshake() 2024-08-17T17:13:28.035 INFO:journalctl@ceph.mon.b.smithi165.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:13:28.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:27 smithi165 ceph-mon[34868]: pgmap v6: 137 pgs: 137 active+clean; 593 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:28.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:27 smithi165 ceph-mon[34868]: [17/Aug/2024:17:13:26] ENGINE Serving on http://172.21.15.136:8765 2024-08-17T17:13:28.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:27 smithi165 ceph-mon[34868]: [17/Aug/2024:17:13:26] ENGINE Bus STARTED 2024-08-17T17:13:28.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:27 smithi165 ceph-mon[34868]: mgrmap e29: y(active, since 2s) 2024-08-17T17:13:28.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:27 smithi165 ceph-mon[34868]: osdmap e194: 8 total, 8 up, 8 in 2024-08-17T17:13:28.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:27 smithi136 ceph-mon[31034]: [17/Aug/2024:17:13:26] ENGINE Serving on https://172.21.15.136:7150 2024-08-17T17:13:28.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:27 smithi136 ceph-mon[31034]: [17/Aug/2024:17:13:26] ENGINE Error in HTTPServer.serve 2024-08-17T17:13:28.190 INFO:journalctl@ceph.mon.a.smithi136.stdout: Traceback (most recent call last): 2024-08-17T17:13:28.190 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:13:28.190 INFO:journalctl@ceph.mon.a.smithi136.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:13:28.190 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:13:28.190 INFO:journalctl@ceph.mon.a.smithi136.stdout: self._run(expiration_interval) 2024-08-17T17:13:28.190 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:13:28.190 INFO:journalctl@ceph.mon.a.smithi136.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:13:28.190 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:13:28.190 INFO:journalctl@ceph.mon.a.smithi136.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:13:28.190 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:13:28.190 INFO:journalctl@ceph.mon.a.smithi136.stdout: s = self.context.wrap_socket( 2024-08-17T17:13:28.190 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:13:28.191 INFO:journalctl@ceph.mon.a.smithi136.stdout: return self.sslsocket_class._create( 2024-08-17T17:13:28.191 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:13:28.191 INFO:journalctl@ceph.mon.a.smithi136.stdout: self.do_handshake() 2024-08-17T17:13:28.191 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:13:28.191 INFO:journalctl@ceph.mon.a.smithi136.stdout: self._sslobj.do_handshake() 2024-08-17T17:13:28.191 INFO:journalctl@ceph.mon.a.smithi136.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:13:28.191 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:27 smithi136 ceph-mon[31034]: pgmap v6: 137 pgs: 137 active+clean; 593 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:28.191 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:27 smithi136 ceph-mon[31034]: [17/Aug/2024:17:13:26] ENGINE Serving on http://172.21.15.136:8765 2024-08-17T17:13:28.191 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:27 smithi136 ceph-mon[31034]: [17/Aug/2024:17:13:26] ENGINE Bus STARTED 2024-08-17T17:13:28.191 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:27 smithi136 ceph-mon[31034]: mgrmap e29: y(active, since 2s) 2024-08-17T17:13:28.191 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:27 smithi136 ceph-mon[31034]: osdmap e194: 8 total, 8 up, 8 in 2024-08-17T17:13:28.191 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:27 smithi136 ceph-mon[42580]: [17/Aug/2024:17:13:26] ENGINE Serving on https://172.21.15.136:7150 2024-08-17T17:13:28.192 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:27 smithi136 ceph-mon[42580]: [17/Aug/2024:17:13:26] ENGINE Error in HTTPServer.serve 2024-08-17T17:13:28.192 INFO:journalctl@ceph.mon.c.smithi136.stdout: Traceback (most recent call last): 2024-08-17T17:13:28.192 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:13:28.192 INFO:journalctl@ceph.mon.c.smithi136.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:13:28.192 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:13:28.192 INFO:journalctl@ceph.mon.c.smithi136.stdout: self._run(expiration_interval) 2024-08-17T17:13:28.192 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:13:28.192 INFO:journalctl@ceph.mon.c.smithi136.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:13:28.192 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:13:28.192 INFO:journalctl@ceph.mon.c.smithi136.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:13:28.192 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:13:28.192 INFO:journalctl@ceph.mon.c.smithi136.stdout: s = self.context.wrap_socket( 2024-08-17T17:13:28.192 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:13:28.193 INFO:journalctl@ceph.mon.c.smithi136.stdout: return self.sslsocket_class._create( 2024-08-17T17:13:28.193 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:13:28.193 INFO:journalctl@ceph.mon.c.smithi136.stdout: self.do_handshake() 2024-08-17T17:13:28.193 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:13:28.193 INFO:journalctl@ceph.mon.c.smithi136.stdout: self._sslobj.do_handshake() 2024-08-17T17:13:28.193 INFO:journalctl@ceph.mon.c.smithi136.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:13:28.193 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:27 smithi136 ceph-mon[42580]: pgmap v6: 137 pgs: 137 active+clean; 593 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:28.193 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:27 smithi136 ceph-mon[42580]: [17/Aug/2024:17:13:26] ENGINE Serving on http://172.21.15.136:8765 2024-08-17T17:13:28.193 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:27 smithi136 ceph-mon[42580]: [17/Aug/2024:17:13:26] ENGINE Bus STARTED 2024-08-17T17:13:28.193 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:27 smithi136 ceph-mon[42580]: mgrmap e29: y(active, since 2s) 2024-08-17T17:13:28.193 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:27 smithi136 ceph-mon[42580]: osdmap e194: 8 total, 8 up, 8 in 2024-08-17T17:13:28.514 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:13:28.533 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 83886079~1 2024-08-17T17:13:28.541 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:13:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:29 smithi165 ceph-mon[34868]: osdmap e195: 8 total, 8 up, 8 in 2024-08-17T17:13:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:29 smithi165 ceph-mon[34868]: pgmap v9: 137 pgs: 137 active+clean; 593 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:29 smithi165 ceph-mon[34868]: mgrmap e30: y(active, since 4s) 2024-08-17T17:13:29.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:29 smithi136 ceph-mon[42580]: osdmap e195: 8 total, 8 up, 8 in 2024-08-17T17:13:29.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:29 smithi136 ceph-mon[42580]: pgmap v9: 137 pgs: 137 active+clean; 593 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:29.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:29 smithi136 ceph-mon[42580]: mgrmap e30: y(active, since 4s) 2024-08-17T17:13:29.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:29 smithi136 ceph-mon[31034]: osdmap e195: 8 total, 8 up, 8 in 2024-08-17T17:13:29.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:29 smithi136 ceph-mon[31034]: pgmap v9: 137 pgs: 137 active+clean; 593 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:29.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:29 smithi136 ceph-mon[31034]: mgrmap e30: y(active, since 4s) 2024-08-17T17:13:30.261 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:29 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: [17/Aug/2024:17:13:29] ENGINE Bus STARTING 2024-08-17T17:13:30.261 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:29 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: CherryPy Checker: 2024-08-17T17:13:30.261 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:29 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: The Application mounted at '' has an empty config. 2024-08-17T17:13:30.261 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:29 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: 2024-08-17T17:13:30.261 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:30 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: [17/Aug/2024:17:13:30] ENGINE Serving on http://:::9283 2024-08-17T17:13:30.261 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:30 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: [17/Aug/2024:17:13:30] ENGINE Bus STARTED 2024-08-17T17:13:30.769 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:30 smithi136 ceph-mon[31034]: osdmap e196: 8 total, 8 up, 8 in 2024-08-17T17:13:30.770 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:30 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:30.770 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:30 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:30.770 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:30 smithi136 ceph-mon[31034]: Standby manager daemon x started 2024-08-17T17:13:30.770 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:30 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.165:0/3421457826' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:13:30.770 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:30 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.165:0/3421457826' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:13:30.770 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:30 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.165:0/3421457826' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:13:30.770 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:30 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.165:0/3421457826' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:13:30.770 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:30 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:30.770 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:30 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:30.770 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:30 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:30.770 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:30 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:30.771 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:30 smithi136 ceph-mon[42580]: osdmap e196: 8 total, 8 up, 8 in 2024-08-17T17:13:30.771 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:30 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:30.771 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:30 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:30.771 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:30 smithi136 ceph-mon[42580]: Standby manager daemon x started 2024-08-17T17:13:30.771 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:30 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.165:0/3421457826' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:13:30.771 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:30 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.165:0/3421457826' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:13:30.771 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:30 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.165:0/3421457826' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:13:30.771 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:30 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.165:0/3421457826' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:13:30.771 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:30 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:30.771 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:30 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:30.771 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:30 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:30.771 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:30 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:30 smithi165 ceph-mon[34868]: osdmap e196: 8 total, 8 up, 8 in 2024-08-17T17:13:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:30 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:30 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:30 smithi165 ceph-mon[34868]: Standby manager daemon x started 2024-08-17T17:13:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:30 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.165:0/3421457826' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:13:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:30 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.165:0/3421457826' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:13:31.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:30 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.165:0/3421457826' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:13:31.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:30 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.165:0/3421457826' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:13:31.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:30 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:31.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:30 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:31.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:30 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:31.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:30 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:31.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:31 smithi165 ceph-mon[34868]: osdmap e197: 8 total, 8 up, 8 in 2024-08-17T17:13:31.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:31 smithi165 ceph-mon[34868]: mgrmap e31: y(active, since 5s), standbys: x 2024-08-17T17:13:31.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:31 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-17T17:13:31.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:31 smithi165 ceph-mon[34868]: pgmap v12: 137 pgs: 137 active+clean; 593 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:31.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:31 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:31.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:31 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:31.859 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:31 smithi136 ceph-mon[31034]: osdmap e197: 8 total, 8 up, 8 in 2024-08-17T17:13:31.859 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:31 smithi136 ceph-mon[31034]: mgrmap e31: y(active, since 5s), standbys: x 2024-08-17T17:13:31.859 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:31 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-17T17:13:31.859 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:31 smithi136 ceph-mon[31034]: pgmap v12: 137 pgs: 137 active+clean; 593 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:31.859 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:31 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:31.859 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:31 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:31.859 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:31 smithi136 ceph-mon[42580]: osdmap e197: 8 total, 8 up, 8 in 2024-08-17T17:13:31.859 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:31 smithi136 ceph-mon[42580]: mgrmap e31: y(active, since 5s), standbys: x 2024-08-17T17:13:31.860 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:31 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-17T17:13:31.860 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:31 smithi136 ceph-mon[42580]: pgmap v12: 137 pgs: 137 active+clean; 593 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:31.860 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:31 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:31.860 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:31 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:31.934 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 167772160 2024-08-17T17:13:31.944 INFO:tasks.workunit.client.0.smithi136.stdout:write: * 83886080~1000 2024-08-17T17:13:31.979 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:13:32.112 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:13:32.261 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[70740]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:13:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:13:32.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:32 smithi136 ceph-mon[42580]: osdmap e198: 8 total, 8 up, 8 in 2024-08-17T17:13:32.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:32 smithi136 ceph-mon[31034]: osdmap e198: 8 total, 8 up, 8 in 2024-08-17T17:13:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:32 smithi165 ceph-mon[34868]: osdmap e198: 8 total, 8 up, 8 in 2024-08-17T17:13:33.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:33 smithi136 ceph-mon[31034]: osdmap e199: 8 total, 8 up, 8 in 2024-08-17T17:13:33.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:33 smithi136 ceph-mon[31034]: pgmap v15: 137 pgs: 137 active+clean; 617 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 163 KiB/s rd, 4.0 MiB/s wr, 187 op/s 2024-08-17T17:13:33.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:33 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:33.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:33 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:33.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:33 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:13:33.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:33 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:13:33.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:33 smithi136 ceph-mon[31034]: Updating smithi136:/etc/ceph/ceph.conf 2024-08-17T17:13:33.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:33 smithi136 ceph-mon[31034]: Updating smithi165:/etc/ceph/ceph.conf 2024-08-17T17:13:33.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:33 smithi136 ceph-mon[42580]: osdmap e199: 8 total, 8 up, 8 in 2024-08-17T17:13:33.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:33 smithi136 ceph-mon[42580]: pgmap v15: 137 pgs: 137 active+clean; 617 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 163 KiB/s rd, 4.0 MiB/s wr, 187 op/s 2024-08-17T17:13:33.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:33 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:33.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:33 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:33.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:33 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:13:33.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:33 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:13:33.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:33 smithi136 ceph-mon[42580]: Updating smithi136:/etc/ceph/ceph.conf 2024-08-17T17:13:33.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:33 smithi136 ceph-mon[42580]: Updating smithi165:/etc/ceph/ceph.conf 2024-08-17T17:13:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:33 smithi165 ceph-mon[34868]: osdmap e199: 8 total, 8 up, 8 in 2024-08-17T17:13:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:33 smithi165 ceph-mon[34868]: pgmap v15: 137 pgs: 137 active+clean; 617 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 163 KiB/s rd, 4.0 MiB/s wr, 187 op/s 2024-08-17T17:13:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:33 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:33 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:33 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:13:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:33 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:13:34.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:33 smithi165 ceph-mon[34868]: Updating smithi136:/etc/ceph/ceph.conf 2024-08-17T17:13:34.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:33 smithi165 ceph-mon[34868]: Updating smithi165:/etc/ceph/ceph.conf 2024-08-17T17:13:34.914 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.SnapsExpandAfterPrepare (11938 ms) 2024-08-17T17:13:34.914 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.SnapsExpandAfterPrepare2 2024-08-17T17:13:34.927 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:34 smithi165 ceph-mon[34868]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:13:34.927 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:34 smithi165 ceph-mon[34868]: osdmap e200: 8 total, 8 up, 8 in 2024-08-17T17:13:34.927 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:34 smithi165 ceph-mon[34868]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:13:34.927 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:34 smithi165 ceph-mon[34868]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:13:34.927 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:34 smithi165 ceph-mon[34868]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:13:34.927 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:34 smithi165 ceph-mon[34868]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:13:34.927 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:34 smithi165 ceph-mon[34868]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:13:34.927 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:34 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:34.927 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:34 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:34.927 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:34 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:34.928 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:34 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:34.928 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:34 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:34.928 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:34 smithi165 ceph-mon[34868]: Reconfiguring prometheus.a (dependencies changed)... 2024-08-17T17:13:34.943 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:13:34.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[42580]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:13:34.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[42580]: osdmap e200: 8 total, 8 up, 8 in 2024-08-17T17:13:34.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[42580]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:13:34.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[42580]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:13:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[42580]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:13:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[42580]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:13:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[42580]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:13:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[42580]: Reconfiguring prometheus.a (dependencies changed)... 2024-08-17T17:13:34.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[31034]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:13:34.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[31034]: osdmap e200: 8 total, 8 up, 8 in 2024-08-17T17:13:34.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[31034]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:13:34.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[31034]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:13:34.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[31034]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:13:34.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[31034]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:13:34.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[31034]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:13:34.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:34.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:34.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:34.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:34.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:34.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:34 smithi136 ceph-mon[31034]: Reconfiguring prometheus.a (dependencies changed)... 2024-08-17T17:13:34.953 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 41943040~10 2024-08-17T17:13:34.960 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:13:35.049 INFO:teuthology.orchestra.run.smithi136.stdout:true 2024-08-17T17:13:35.611 INFO:teuthology.orchestra.run.smithi136.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-17T17:13:35.611 INFO:teuthology.orchestra.run.smithi136.stdout:alertmanager.a smithi136 *:9093,9094 running (3m) 6s ago 4m 27.8M - 0.25.0 c8568f914cd2 83b6bfc5306f 2024-08-17T17:13:35.611 INFO:teuthology.orchestra.run.smithi136.stdout:grafana.a smithi165 *:3000 running (4m) 5s ago 4m 90.0M - 9.4.7 954c08fa6188 be80a687c402 2024-08-17T17:13:35.611 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi136.ltsuqd smithi136 running (3m) 6s ago 3m 18.2M - 18.2.4-719-gced649f8 c870753d67c2 faaa06f2950c 2024-08-17T17:13:35.611 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi165.bmrbhh smithi165 running (3m) 5s ago 3m 15.8M - 18.2.4-719-gced649f8 c870753d67c2 c74b00d929cf 2024-08-17T17:13:35.612 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.x smithi165 *:8443,9283,8765 running (59s) 5s ago 8m 143M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 7329884ca5de 2024-08-17T17:13:35.612 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.y smithi136 *:8443,9283,8765 running (19s) 6s ago 10m 489M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9acdc6775e48 2024-08-17T17:13:35.612 INFO:teuthology.orchestra.run.smithi136.stdout:mon.a smithi136 running (10m) 6s ago 10m 53.7M 2048M 18.2.4-719-gced649f8 c870753d67c2 cc4e5e5ff756 2024-08-17T17:13:35.612 INFO:teuthology.orchestra.run.smithi136.stdout:mon.b smithi165 running (8m) 5s ago 8m 47.1M 2048M 18.2.4-719-gced649f8 c870753d67c2 d587e16140e1 2024-08-17T17:13:35.612 INFO:teuthology.orchestra.run.smithi136.stdout:mon.c smithi136 running (8m) 6s ago 8m 45.8M 2048M 18.2.4-719-gced649f8 c870753d67c2 b4e0f3c9e485 2024-08-17T17:13:35.612 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.a smithi136 *:9100 running (4m) 6s ago 4m 15.9M - 1.5.0 0da6a335fe13 6aef82a780f5 2024-08-17T17:13:35.612 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.b smithi165 *:9100 running (4m) 5s ago 4m 15.7M - 1.5.0 0da6a335fe13 60b065dd0c2a 2024-08-17T17:13:35.612 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0 smithi136 running (8m) 6s ago 8m 115M 2480M 18.2.4-719-gced649f8 c870753d67c2 faab916734b0 2024-08-17T17:13:35.612 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1 smithi136 running (7m) 6s ago 7m 102M 2480M 18.2.4-719-gced649f8 c870753d67c2 861686b6ff82 2024-08-17T17:13:35.612 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2 smithi136 running (7m) 6s ago 7m 97.5M 2480M 18.2.4-719-gced649f8 c870753d67c2 640be4af4156 2024-08-17T17:13:35.612 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3 smithi136 running (6m) 6s ago 6m 87.9M 2480M 18.2.4-719-gced649f8 c870753d67c2 14b57a745d30 2024-08-17T17:13:35.612 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4 smithi165 running (6m) 5s ago 6m 107M 2480M 18.2.4-719-gced649f8 c870753d67c2 df366c73e1f2 2024-08-17T17:13:35.612 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5 smithi165 running (6m) 5s ago 6m 86.1M 2480M 18.2.4-719-gced649f8 c870753d67c2 1131665e203a 2024-08-17T17:13:35.612 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6 smithi165 running (5m) 5s ago 5m 124M 2480M 18.2.4-719-gced649f8 c870753d67c2 7bc5fd925293 2024-08-17T17:13:35.613 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7 smithi165 running (5m) 5s ago 5m 90.1M 2480M 18.2.4-719-gced649f8 c870753d67c2 18382e4ae3b8 2024-08-17T17:13:35.613 INFO:teuthology.orchestra.run.smithi136.stdout:prometheus.a smithi165 *:9095 running (24s) 5s ago 4m 36.8M - 2.43.0 a07b618ecd1d 19e69ad66bef 2024-08-17T17:13:35.943 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:13:35.944 INFO:teuthology.orchestra.run.smithi136.stdout: "mon": { 2024-08-17T17:13:35.944 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 3 2024-08-17T17:13:35.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:13:35.944 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr": { 2024-08-17T17:13:35.944 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 2 2024-08-17T17:13:35.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:13:35.944 INFO:teuthology.orchestra.run.smithi136.stdout: "osd": { 2024-08-17T17:13:35.944 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 8 2024-08-17T17:13:35.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:13:35.944 INFO:teuthology.orchestra.run.smithi136.stdout: "mds": { 2024-08-17T17:13:35.944 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 2 2024-08-17T17:13:35.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:13:35.945 INFO:teuthology.orchestra.run.smithi136.stdout: "overall": { 2024-08-17T17:13:35.945 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 13, 2024-08-17T17:13:35.945 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 2 2024-08-17T17:13:35.945 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:13:35.945 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:13:35.945 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:35 smithi136 ceph-mon[31034]: Reconfiguring daemon prometheus.a on smithi165 2024-08-17T17:13:35.945 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:35 smithi136 ceph-mon[31034]: osdmap e201: 8 total, 8 up, 8 in 2024-08-17T17:13:35.945 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:35 smithi136 ceph-mon[31034]: pgmap v18: 137 pgs: 137 active+clean; 617 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 163 KiB/s rd, 4.0 MiB/s wr, 187 op/s 2024-08-17T17:13:35.945 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:35 smithi136 ceph-mon[31034]: from='client.24989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:35.945 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:35 smithi136 ceph-mon[31034]: from='client.24995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:35.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:35 smithi136 ceph-mon[42580]: Reconfiguring daemon prometheus.a on smithi165 2024-08-17T17:13:35.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:35 smithi136 ceph-mon[42580]: osdmap e201: 8 total, 8 up, 8 in 2024-08-17T17:13:35.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:35 smithi136 ceph-mon[42580]: pgmap v18: 137 pgs: 137 active+clean; 617 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 163 KiB/s rd, 4.0 MiB/s wr, 187 op/s 2024-08-17T17:13:35.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:35 smithi136 ceph-mon[42580]: from='client.24989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:35.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:35 smithi136 ceph-mon[42580]: from='client.24995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:36.001 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:35 smithi165 ceph-mon[34868]: Reconfiguring daemon prometheus.a on smithi165 2024-08-17T17:13:36.001 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:35 smithi165 ceph-mon[34868]: osdmap e201: 8 total, 8 up, 8 in 2024-08-17T17:13:36.001 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:35 smithi165 ceph-mon[34868]: pgmap v18: 137 pgs: 137 active+clean; 617 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 163 KiB/s rd, 4.0 MiB/s wr, 187 op/s 2024-08-17T17:13:36.001 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:35 smithi165 ceph-mon[34868]: from='client.24989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:36.001 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:35 smithi165 ceph-mon[34868]: from='client.24995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:36.227 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:13:36.228 INFO:teuthology.orchestra.run.smithi136.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d", 2024-08-17T17:13:36.228 INFO:teuthology.orchestra.run.smithi136.stdout: "in_progress": true, 2024-08-17T17:13:36.228 INFO:teuthology.orchestra.run.smithi136.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-08-17T17:13:36.228 INFO:teuthology.orchestra.run.smithi136.stdout: "services_complete": [ 2024-08-17T17:13:36.228 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr" 2024-08-17T17:13:36.228 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-17T17:13:36.228 INFO:teuthology.orchestra.run.smithi136.stdout: "progress": "2/2 daemons upgraded", 2024-08-17T17:13:36.228 INFO:teuthology.orchestra.run.smithi136.stdout: "message": "", 2024-08-17T17:13:36.228 INFO:teuthology.orchestra.run.smithi136.stdout: "is_paused": false 2024-08-17T17:13:36.228 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:13:36.590 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 41943040 2024-08-17T17:13:36.608 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:13:36.720 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:36 smithi165 ceph-mon[34868]: osdmap e202: 8 total, 8 up, 8 in 2024-08-17T17:13:36.720 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:36 smithi165 ceph-mon[34868]: from='client.25001 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:36.720 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:36 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2449391947' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:36.720 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:36 smithi165 ceph-mon[34868]: from='client.25013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:36 smithi136 ceph-mon[31034]: osdmap e202: 8 total, 8 up, 8 in 2024-08-17T17:13:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:36 smithi136 ceph-mon[31034]: from='client.25001 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:36 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/2449391947' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:36.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:36 smithi136 ceph-mon[31034]: from='client.25013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:36 smithi136 ceph-mon[42580]: osdmap e202: 8 total, 8 up, 8 in 2024-08-17T17:13:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:36 smithi136 ceph-mon[42580]: from='client.25001 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:36 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/2449391947' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:36 smithi136 ceph-mon[42580]: from='client.25013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:37.740 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:13:37.750 INFO:tasks.workunit.client.0.smithi136.stdout:write: Y 41943040~5 2024-08-17T17:13:37.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:37 smithi165 ceph-mon[34868]: osdmap e203: 8 total, 8 up, 8 in 2024-08-17T17:13:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:37 smithi165 ceph-mon[34868]: pgmap v21: 137 pgs: 137 active+clean; 617 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 70 MiB/s rd, 6.0 MiB/s wr, 513 op/s 2024-08-17T17:13:37.841 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:13:37.915 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:13:37.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:37 smithi136 ceph-mon[42580]: osdmap e203: 8 total, 8 up, 8 in 2024-08-17T17:13:37.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:37 smithi136 ceph-mon[42580]: pgmap v21: 137 pgs: 137 active+clean; 617 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 70 MiB/s rd, 6.0 MiB/s wr, 513 op/s 2024-08-17T17:13:37.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:37 smithi136 ceph-mon[31034]: osdmap e203: 8 total, 8 up, 8 in 2024-08-17T17:13:37.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:37 smithi136 ceph-mon[31034]: pgmap v21: 137 pgs: 137 active+clean; 617 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 70 MiB/s rd, 6.0 MiB/s wr, 513 op/s 2024-08-17T17:13:38.261 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:38 smithi165 systemd[1]: Stopping Ceph prometheus.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:13:38.565 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:38 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:38.299Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-08-17T17:13:38.565 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:38 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:38.299Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-08-17T17:13:38.565 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:38 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:38.299Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-08-17T17:13:38.565 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:38 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:38.299Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-08-17T17:13:38.565 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:38 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:38.299Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-08-17T17:13:38.565 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:38 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:38.299Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-08-17T17:13:38.565 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:38 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:38.299Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-08-17T17:13:38.565 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:38 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:38.299Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-08-17T17:13:38.565 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:38 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:38.300Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-08-17T17:13:38.565 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:38 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:38.300Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-08-17T17:13:38.566 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:38 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:38.300Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-08-17T17:13:38.566 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:38 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[79181]: ts=2024-08-17T17:13:38.300Z caller=main.go:1147 level=info msg="See you next time!" 2024-08-17T17:13:38.566 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:38 smithi165 podman[81805]: 2024-08-17 17:13:38.309877953 +0000 UTC m=+0.109839523 container died 19e69ad66bef9ff92948aed7f30a06686a7e8946774e528d1e7635cd66025cf3 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:13:38.687 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.SnapsExpandAfterPrepare2 (3773 ms) 2024-08-17T17:13:38.687 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.SnapsSnapAfterPrepare 2024-08-17T17:13:38.718 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:13:38.728 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 0~8388609 2024-08-17T17:13:38.786 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:13:38.942 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:38 smithi165 ceph-mon[34868]: osdmap e204: 8 total, 8 up, 8 in 2024-08-17T17:13:38.942 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:38 smithi165 podman[81805]: 2024-08-17 17:13:38.663586384 +0000 UTC m=+0.463547956 container cleanup 19e69ad66bef9ff92948aed7f30a06686a7e8946774e528d1e7635cd66025cf3 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:13:38.942 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:38 smithi165 bash[81805]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a 2024-08-17T17:13:38.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:38 smithi136 ceph-mon[31034]: osdmap e204: 8 total, 8 up, 8 in 2024-08-17T17:13:38.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:38 smithi136 ceph-mon[42580]: osdmap e204: 8 total, 8 up, 8 in 2024-08-17T17:13:39.261 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:38 smithi165 podman[81824]: 2024-08-17 17:13:38.941886646 +0000 UTC m=+0.634053340 container remove 19e69ad66bef9ff92948aed7f30a06686a7e8946774e528d1e7635cd66025cf3 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:13:39.261 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-08-17T17:13:39.261 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 systemd[1]: Stopped Ceph prometheus.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:13:39.261 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@prometheus.a.service: Consumed 1.140s CPU time. 2024-08-17T17:13:39.261 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 systemd[1]: Starting Ceph prometheus.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:13:39.602 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 podman[81922]: 2024-08-17 17:13:39.221328033 +0000 UTC m=+0.017890454 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-08-17T17:13:39.603 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 podman[81922]: 2024-08-17 17:13:39.359510284 +0000 UTC m=+0.156072755 container create f3e579e6d2fb0001d3fa48ae0407be1fab5b829e265708cb98adde75e831c6cc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:13:39.862 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:39 smithi136 ceph-mon[31034]: pgmap v24: 137 pgs: 137 active+clean; 617 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 70 MiB/s rd, 6.0 MiB/s wr, 513 op/s 2024-08-17T17:13:39.862 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:39 smithi136 ceph-mon[31034]: osdmap e205: 8 total, 8 up, 8 in 2024-08-17T17:13:39.862 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:39 smithi136 ceph-mon[42580]: pgmap v24: 137 pgs: 137 active+clean; 617 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 70 MiB/s rd, 6.0 MiB/s wr, 513 op/s 2024-08-17T17:13:39.862 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:39 smithi136 ceph-mon[42580]: osdmap e205: 8 total, 8 up, 8 in 2024-08-17T17:13:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-mon[34868]: pgmap v24: 137 pgs: 137 active+clean; 617 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 70 MiB/s rd, 6.0 MiB/s wr, 513 op/s 2024-08-17T17:13:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-mon[34868]: osdmap e205: 8 total, 8 up, 8 in 2024-08-17T17:13:40.011 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 podman[81922]: 2024-08-17 17:13:39.627894692 +0000 UTC m=+0.424457105 container init f3e579e6d2fb0001d3fa48ae0407be1fab5b829e265708cb98adde75e831c6cc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:13:40.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 podman[81922]: 2024-08-17 17:13:39.632458998 +0000 UTC m=+0.429021402 container start f3e579e6d2fb0001d3fa48ae0407be1fab5b829e265708cb98adde75e831c6cc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:13:40.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:13:39.656Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-08-17T17:13:40.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:13:39.656Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-08-17T17:13:40.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:13:39.656Z caller=main.go:570 level=info host_details="(Linux 5.14.0-480.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024 x86_64 smithi165 (none))" 2024-08-17T17:13:40.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:13:39.656Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-08-17T17:13:40.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:13:39.656Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-08-17T17:13:40.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:13:39.657Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-08-17T17:13:40.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:13:39.658Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-08-17T17:13:40.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:13:39.659Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-08-17T17:13:40.012 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:13:39.659Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-08-17T17:13:40.013 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:13:39.662Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-08-17T17:13:40.013 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:13:39.662Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.789µs 2024-08-17T17:13:40.013 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:13:39.662Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-08-17T17:13:40.013 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:13:39.663Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-08-17T17:13:40.013 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:13:39.673Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-08-17T17:13:40.013 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:13:39.674Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-08-17T17:13:40.013 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:13:39.674Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-08-17T17:13:40.013 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:13:39.674Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=31.17µs wal_replay_duration=12.191819ms wbl_replay_duration=110ns total_replay_duration=12.241632ms 2024-08-17T17:13:40.013 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:13:39.675Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-08-17T17:13:40.013 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:13:39.675Z caller=main.go:1029 level=info msg="TSDB started" 2024-08-17T17:13:40.013 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:13:39.675Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-08-17T17:13:40.013 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:13:39.697Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.855259ms db_storage=1.21µs remote_storage=1.077µs web_handler=456ns query_engine=638ns scrape=411.343µs scrape_sd=95.487µs notify=9.665µs notify_sd=9.528µs rules=20.992504ms tracing=6.075µs 2024-08-17T17:13:40.014 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:13:39.697Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-08-17T17:13:40.014 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:13:39.697Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-08-17T17:13:40.014 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 bash[81922]: f3e579e6d2fb0001d3fa48ae0407be1fab5b829e265708cb98adde75e831c6cc 2024-08-17T17:13:40.014 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:13:39 smithi165 systemd[1]: Started Ceph prometheus.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:13:40.622 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 8388609~8388609 2024-08-17T17:13:40.644 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 0~1000 2024-08-17T17:13:40.644 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 1010~1000 2024-08-17T17:13:40.644 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap2 2024-08-17T17:13:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:40 smithi165 ceph-mon[34868]: osdmap e206: 8 total, 8 up, 8 in 2024-08-17T17:13:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:40 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:40 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:40 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:13:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:40 smithi165 ceph-mon[34868]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:13:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:40 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:13:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:40 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:13:40.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:40 smithi136 ceph-mon[31034]: osdmap e206: 8 total, 8 up, 8 in 2024-08-17T17:13:40.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:40 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:40.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:40 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:40.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:40 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:13:40.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:40 smithi136 ceph-mon[31034]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:13:40.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:40 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:13:40.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:40 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:13:40.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:40 smithi136 ceph-mon[42580]: osdmap e206: 8 total, 8 up, 8 in 2024-08-17T17:13:40.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:40 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:40.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:40 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:40.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:40 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:13:40.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:40 smithi136 ceph-mon[42580]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:13:40.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:40 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:13:40.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:40 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:13:41.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:41 smithi165 ceph-mon[34868]: pgmap v26: 137 pgs: 137 active+clean; 617 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:41.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:41 smithi165 ceph-mon[34868]: osdmap e207: 8 total, 8 up, 8 in 2024-08-17T17:13:41.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:41 smithi165 ceph-mon[34868]: Upgrade: Updating mgr.x 2024-08-17T17:13:41.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:41 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:13:41.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:41 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:13:41.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:41 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:13:41.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:41 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:13:41.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:41 smithi165 ceph-mon[34868]: Deploying daemon mgr.x on smithi165 2024-08-17T17:13:41.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:41 smithi136 ceph-mon[31034]: pgmap v26: 137 pgs: 137 active+clean; 617 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:41.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:41 smithi136 ceph-mon[31034]: osdmap e207: 8 total, 8 up, 8 in 2024-08-17T17:13:41.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:41 smithi136 ceph-mon[31034]: Upgrade: Updating mgr.x 2024-08-17T17:13:41.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:41 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:13:41.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:41 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:13:41.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:41 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:13:41.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:41 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:13:41.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:41 smithi136 ceph-mon[31034]: Deploying daemon mgr.x on smithi165 2024-08-17T17:13:41.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:41 smithi136 ceph-mon[42580]: pgmap v26: 137 pgs: 137 active+clean; 617 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:41.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:41 smithi136 ceph-mon[42580]: osdmap e207: 8 total, 8 up, 8 in 2024-08-17T17:13:41.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:41 smithi136 ceph-mon[42580]: Upgrade: Updating mgr.x 2024-08-17T17:13:41.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:41 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:13:41.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:41 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:13:41.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:41 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:13:41.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:41 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:13:41.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:41 smithi136 ceph-mon[42580]: Deploying daemon mgr.x on smithi165 2024-08-17T17:13:42.519 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 8388609~1000 2024-08-17T17:13:42.525 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 16777228~1000 2024-08-17T17:13:42.526 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 167772160 2024-08-17T17:13:42.534 INFO:tasks.workunit.client.0.smithi136.stdout:write: 2 83886079~8388609 2024-08-17T17:13:42.549 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap3 2024-08-17T17:13:42.819 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:42 smithi165 ceph-mon[34868]: osdmap e208: 8 total, 8 up, 8 in 2024-08-17T17:13:42.819 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:42 smithi165 ceph-mon[34868]: osdmap e209: 8 total, 8 up, 8 in 2024-08-17T17:13:42.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:42 smithi136 ceph-mon[31034]: osdmap e208: 8 total, 8 up, 8 in 2024-08-17T17:13:42.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:42 smithi136 ceph-mon[31034]: osdmap e209: 8 total, 8 up, 8 in 2024-08-17T17:13:42.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:42 smithi136 ceph-mon[42580]: osdmap e208: 8 total, 8 up, 8 in 2024-08-17T17:13:42.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:42 smithi136 ceph-mon[42580]: osdmap e209: 8 total, 8 up, 8 in 2024-08-17T17:13:43.396 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:43 smithi165 systemd[1]: Stopping Ceph mgr.x for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:13:43.696 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:43 smithi165 podman[82327]: 2024-08-17 17:13:43.395629173 +0000 UTC m=+0.142656501 container died 7329884ca5de8a39a5c087b3fb36ad08ae81adb79f7b13b2ef70d795093e9059 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe, maintainer=Guillaume Abrioux ) 2024-08-17T17:13:43.696 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:43 smithi165 ceph-mon[34868]: pgmap v30: 137 pgs: 137 active+clean; 649 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 325 KiB/s rd, 8.0 MiB/s wr, 426 op/s 2024-08-17T17:13:43.696 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:43 smithi165 ceph-mon[34868]: osdmap e210: 8 total, 8 up, 8 in 2024-08-17T17:13:43.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:43 smithi136 ceph-mon[31034]: pgmap v30: 137 pgs: 137 active+clean; 649 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 325 KiB/s rd, 8.0 MiB/s wr, 426 op/s 2024-08-17T17:13:43.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:43 smithi136 ceph-mon[31034]: osdmap e210: 8 total, 8 up, 8 in 2024-08-17T17:13:43.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:43 smithi136 ceph-mon[42580]: pgmap v30: 137 pgs: 137 active+clean; 649 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 325 KiB/s rd, 8.0 MiB/s wr, 426 op/s 2024-08-17T17:13:43.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:43 smithi136 ceph-mon[42580]: osdmap e210: 8 total, 8 up, 8 in 2024-08-17T17:13:44.011 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:43 smithi165 podman[82327]: 2024-08-17 17:13:43.861587841 +0000 UTC m=+0.608615161 container cleanup 7329884ca5de8a39a5c087b3fb36ad08ae81adb79f7b13b2ef70d795093e9059 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x, RELEASE=main-cd8bdbe, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-17T17:13:44.011 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:43 smithi165 bash[82327]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x 2024-08-17T17:13:44.485 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:44 smithi165 podman[82343]: 2024-08-17 17:13:44.137842116 +0000 UTC m=+0.740857600 container remove 7329884ca5de8a39a5c087b3fb36ad08ae81adb79f7b13b2ef70d795093e9059 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=main-cd8bdbe, org.label-schema.license=GPLv2) 2024-08-17T17:13:44.485 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:44 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-08-17T17:13:44.486 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:44 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-08-17T17:13:44.486 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:44 smithi165 systemd[1]: Stopped Ceph mgr.x for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:13:44.486 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:44 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@mgr.x.service: Consumed 17.628s CPU time. 2024-08-17T17:13:44.486 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:44 smithi165 systemd[1]: Starting Ceph mgr.x for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:13:44.522 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:13:44.543 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 83886079~1 2024-08-17T17:13:44.551 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:13:44.739 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:44 smithi165 podman[82480]: 2024-08-17 17:13:44.614656124 +0000 UTC m=+0.018396433 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:13:45.011 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:44 smithi165 podman[82480]: 2024-08-17 17:13:44.738743559 +0000 UTC m=+0.142483865 container create 837fd48ff7cb95d6e28e91b05d6b3c46c96f4b47af54d535abd4a912e08e261a (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-cd8bdbe, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T17:13:45.511 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:45 smithi165 podman[82480]: 2024-08-17 17:13:45.083280912 +0000 UTC m=+0.487021231 container init 837fd48ff7cb95d6e28e91b05d6b3c46c96f4b47af54d535abd4a912e08e261a (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=main-cd8bdbe, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-08-17T17:13:45.511 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:45 smithi165 podman[82480]: 2024-08-17 17:13:45.088579832 +0000 UTC m=+0.492320139 container start 837fd48ff7cb95d6e28e91b05d6b3c46c96f4b47af54d535abd4a912e08e261a (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe, io.buildah.version=1.36.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T17:13:45.511 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:45 smithi165 ceph-mgr[82518]: -- 172.21.15.165:0/3915993632 <== mon.1 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555702ef70e0 con 0x555702ed5000 2024-08-17T17:13:45.511 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:45 smithi165 bash[82480]: 837fd48ff7cb95d6e28e91b05d6b3c46c96f4b47af54d535abd4a912e08e261a 2024-08-17T17:13:45.511 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:45 smithi165 systemd[1]: Started Ceph mgr.x for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:13:45.936 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:45 smithi165 ceph-mon[34868]: osdmap e211: 8 total, 8 up, 8 in 2024-08-17T17:13:45.937 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:45 smithi165 ceph-mon[34868]: pgmap v33: 137 pgs: 137 active+clean; 649 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 326 KiB/s rd, 8.1 MiB/s wr, 428 op/s 2024-08-17T17:13:45.937 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:45 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:45.937 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:45 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:45.937 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:45 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:13:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:45 smithi136 ceph-mon[31034]: osdmap e211: 8 total, 8 up, 8 in 2024-08-17T17:13:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:45 smithi136 ceph-mon[31034]: pgmap v33: 137 pgs: 137 active+clean; 649 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 326 KiB/s rd, 8.1 MiB/s wr, 428 op/s 2024-08-17T17:13:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:45 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:45.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:45 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:45.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:45 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:13:45.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:45 smithi136 ceph-mon[42580]: osdmap e211: 8 total, 8 up, 8 in 2024-08-17T17:13:45.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:45 smithi136 ceph-mon[42580]: pgmap v33: 137 pgs: 137 active+clean; 649 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 326 KiB/s rd, 8.1 MiB/s wr, 428 op/s 2024-08-17T17:13:45.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:45 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:45.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:45 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:45.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:45 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:13:46.524 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:46 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:13:46] "GET /metrics HTTP/1.1" 200 38737 "" "Prometheus/2.43.0" 2024-08-17T17:13:46.841 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:46 smithi165 ceph-mon[34868]: osdmap e212: 8 total, 8 up, 8 in 2024-08-17T17:13:46.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:46 smithi136 ceph-mon[31034]: osdmap e212: 8 total, 8 up, 8 in 2024-08-17T17:13:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:46 smithi136 ceph-mon[42580]: osdmap e212: 8 total, 8 up, 8 in 2024-08-17T17:13:47.784 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:47 smithi165 ceph-mon[34868]: osdmap e213: 8 total, 8 up, 8 in 2024-08-17T17:13:47.784 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:47 smithi165 ceph-mon[34868]: pgmap v36: 137 pgs: 137 active+clean; 673 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-08-17T17:13:47.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:47 smithi136 ceph-mon[42580]: osdmap e213: 8 total, 8 up, 8 in 2024-08-17T17:13:47.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:47 smithi136 ceph-mon[42580]: pgmap v36: 137 pgs: 137 active+clean; 673 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-08-17T17:13:47.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:47 smithi136 ceph-mon[31034]: osdmap e213: 8 total, 8 up, 8 in 2024-08-17T17:13:47.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:47 smithi136 ceph-mon[31034]: pgmap v36: 137 pgs: 137 active+clean; 673 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-08-17T17:13:48.034 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: after_prepare_snap 2024-08-17T17:13:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:48 smithi165 ceph-mon[34868]: osdmap e214: 8 total, 8 up, 8 in 2024-08-17T17:13:48.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:48 smithi136 ceph-mon[31034]: osdmap e214: 8 total, 8 up, 8 in 2024-08-17T17:13:48.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:48 smithi136 ceph-mon[42580]: osdmap e214: 8 total, 8 up, 8 in 2024-08-17T17:13:49.553 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 41943040 2024-08-17T17:13:49.585 INFO:tasks.workunit.client.0.smithi136.stdout:write: * 0~1000 2024-08-17T17:13:49.591 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:13:49.727 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:13:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:49 smithi165 ceph-mon[34868]: osdmap e215: 8 total, 8 up, 8 in 2024-08-17T17:13:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:49 smithi165 ceph-mon[34868]: pgmap v39: 137 pgs: 137 active+clean; 673 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-08-17T17:13:49.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:49 smithi136 ceph-mon[31034]: osdmap e215: 8 total, 8 up, 8 in 2024-08-17T17:13:49.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:49 smithi136 ceph-mon[31034]: pgmap v39: 137 pgs: 137 active+clean; 673 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-08-17T17:13:49.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:49 smithi136 ceph-mon[42580]: osdmap e215: 8 total, 8 up, 8 in 2024-08-17T17:13:49.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:49 smithi136 ceph-mon[42580]: pgmap v39: 137 pgs: 137 active+clean; 673 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-08-17T17:13:50.815 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:50 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[82495]: [17/Aug/2024:17:13:50] ENGINE Bus STARTING 2024-08-17T17:13:50.815 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:50 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[82495]: CherryPy Checker: 2024-08-17T17:13:50.815 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:50 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[82495]: The Application mounted at '' has an empty config. 2024-08-17T17:13:50.815 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:50 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[82495]: 2024-08-17T17:13:50.816 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:50 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[82495]: [17/Aug/2024:17:13:50] ENGINE Serving on http://:::9283 2024-08-17T17:13:50.816 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:13:50 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[82495]: [17/Aug/2024:17:13:50] ENGINE Bus STARTED 2024-08-17T17:13:50.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:50 smithi165 ceph-mon[34868]: osdmap e216: 8 total, 8 up, 8 in 2024-08-17T17:13:50.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:50 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:50.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:50 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:50.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:50 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:50.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:50 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:50.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:50 smithi165 ceph-mon[34868]: Standby manager daemon x restarted 2024-08-17T17:13:50.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:50 smithi165 ceph-mon[34868]: Standby manager daemon x started 2024-08-17T17:13:50.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:50 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.165:0/2096407473' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:13:50.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:50 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.165:0/2096407473' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:13:50.817 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:50 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.165:0/2096407473' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:13:50.817 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:50 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.165:0/2096407473' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:13:50.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:50 smithi136 ceph-mon[31034]: osdmap e216: 8 total, 8 up, 8 in 2024-08-17T17:13:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:50 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:50 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:50 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:50 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:50 smithi136 ceph-mon[31034]: Standby manager daemon x restarted 2024-08-17T17:13:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:50 smithi136 ceph-mon[31034]: Standby manager daemon x started 2024-08-17T17:13:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:50 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.165:0/2096407473' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:13:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:50 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.165:0/2096407473' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:13:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:50 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.165:0/2096407473' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:13:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:50 smithi136 ceph-mon[31034]: from='mgr.? 172.21.15.165:0/2096407473' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:13:50.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:50 smithi136 ceph-mon[42580]: osdmap e216: 8 total, 8 up, 8 in 2024-08-17T17:13:50.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:50 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:50.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:50 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:50.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:50 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:50.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:50 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:50.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:50 smithi136 ceph-mon[42580]: Standby manager daemon x restarted 2024-08-17T17:13:50.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:50 smithi136 ceph-mon[42580]: Standby manager daemon x started 2024-08-17T17:13:50.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:50 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.165:0/2096407473' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:13:50.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:50 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.165:0/2096407473' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:13:50.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:50 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.165:0/2096407473' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:13:50.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:50 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.165:0/2096407473' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:13:51.888 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:51 smithi165 ceph-mon[34868]: pgmap v41: 137 pgs: 137 active+clean; 673 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:51.888 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:51 smithi165 ceph-mon[34868]: osdmap e217: 8 total, 8 up, 8 in 2024-08-17T17:13:51.888 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:51 smithi165 ceph-mon[34868]: mgrmap e32: y(active, since 26s), standbys: x 2024-08-17T17:13:52.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:51 smithi136 ceph-mon[31034]: pgmap v41: 137 pgs: 137 active+clean; 673 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:52.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:51 smithi136 ceph-mon[31034]: osdmap e217: 8 total, 8 up, 8 in 2024-08-17T17:13:52.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:51 smithi136 ceph-mon[31034]: mgrmap e32: y(active, since 26s), standbys: x 2024-08-17T17:13:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:51 smithi136 ceph-mon[42580]: pgmap v41: 137 pgs: 137 active+clean; 673 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:51 smithi136 ceph-mon[42580]: osdmap e217: 8 total, 8 up, 8 in 2024-08-17T17:13:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:51 smithi136 ceph-mon[42580]: mgrmap e32: y(active, since 26s), standbys: x 2024-08-17T17:13:52.820 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.SnapsSnapAfterPrepare (14133 ms) 2024-08-17T17:13:52.820 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.SnapsSnapExpandAfterPrepare 2024-08-17T17:13:52.850 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:13:52.864 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 0~8388609 2024-08-17T17:13:52.912 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:13:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:52 smithi165 ceph-mon[34868]: osdmap e218: 8 total, 8 up, 8 in 2024-08-17T17:13:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:52 smithi165 ceph-mon[34868]: osdmap e219: 8 total, 8 up, 8 in 2024-08-17T17:13:53.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:52 smithi136 ceph-mon[31034]: osdmap e218: 8 total, 8 up, 8 in 2024-08-17T17:13:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:52 smithi136 ceph-mon[31034]: osdmap e219: 8 total, 8 up, 8 in 2024-08-17T17:13:53.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:52 smithi136 ceph-mon[42580]: osdmap e218: 8 total, 8 up, 8 in 2024-08-17T17:13:53.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:52 smithi136 ceph-mon[42580]: osdmap e219: 8 total, 8 up, 8 in 2024-08-17T17:13:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:53 smithi165 ceph-mon[34868]: pgmap v45: 137 pgs: 137 active+clean; 693 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 50 MiB/s rd, 6.0 MiB/s wr, 541 op/s 2024-08-17T17:13:54.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:53 smithi136 ceph-mon[31034]: pgmap v45: 137 pgs: 137 active+clean; 693 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 50 MiB/s rd, 6.0 MiB/s wr, 541 op/s 2024-08-17T17:13:54.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:53 smithi136 ceph-mon[42580]: pgmap v45: 137 pgs: 137 active+clean; 693 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 50 MiB/s rd, 6.0 MiB/s wr, 541 op/s 2024-08-17T17:13:54.777 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 8388609~8388609 2024-08-17T17:13:54.800 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 0~1000 2024-08-17T17:13:54.800 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 1010~1000 2024-08-17T17:13:54.800 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap2 2024-08-17T17:13:54.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: osdmap e220: 8 total, 8 up, 8 in 2024-08-17T17:13:54.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:54.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:54.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:13:54.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:13:54.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:54.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:13:54.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:54.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: Upgrade: Setting container_image for all mgr 2024-08-17T17:13:54.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:54.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-08-17T17:13:54.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-08-17T17:13:54.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-08-17T17:13:54.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-08-17T17:13:54.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-08-17T17:13:54.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-08-17T17:13:54.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:54.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: Upgrade: Setting container_image for all crash 2024-08-17T17:13:54.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:54.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:54.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: Upgrade: Setting container_image for all rgw 2024-08-17T17:13:54.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:54.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:54.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: Upgrade: Setting container_image for all rbd-mirror 2024-08-17T17:13:54.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:54.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:54.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: Upgrade: Setting container_image for all cephfs-mirror 2024-08-17T17:13:54.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:54.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:54.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: Upgrade: Setting container_image for all ceph-exporter 2024-08-17T17:13:54.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:54.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:54.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: Upgrade: Setting container_image for all iscsi 2024-08-17T17:13:54.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:54.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:54.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: Upgrade: Setting container_image for all nfs 2024-08-17T17:13:54.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:54.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:54.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: Upgrade: Setting container_image for all nvmeof 2024-08-17T17:13:54.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:54.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: osdmap e220: 8 total, 8 up, 8 in 2024-08-17T17:13:54.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:54.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:54.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:13:54.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:13:54.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:54.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:13:54.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:54.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: Upgrade: Setting container_image for all mgr 2024-08-17T17:13:54.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:54.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-08-17T17:13:54.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-08-17T17:13:54.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-08-17T17:13:54.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-08-17T17:13:54.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-08-17T17:13:54.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-08-17T17:13:54.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:54.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: Upgrade: Setting container_image for all crash 2024-08-17T17:13:54.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:54.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:54.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: Upgrade: Setting container_image for all rgw 2024-08-17T17:13:54.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:54.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:54.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: Upgrade: Setting container_image for all rbd-mirror 2024-08-17T17:13:54.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:54.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:54.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: Upgrade: Setting container_image for all cephfs-mirror 2024-08-17T17:13:54.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:54.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:54.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: Upgrade: Setting container_image for all ceph-exporter 2024-08-17T17:13:54.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:54.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:54.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: Upgrade: Setting container_image for all iscsi 2024-08-17T17:13:54.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:54.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:54.953 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: Upgrade: Setting container_image for all nfs 2024-08-17T17:13:54.953 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:54.953 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:54.953 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: Upgrade: Setting container_image for all nvmeof 2024-08-17T17:13:54.953 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:54 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: osdmap e220: 8 total, 8 up, 8 in 2024-08-17T17:13:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:55.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:13:55.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:13:55.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:55.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:13:55.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:55.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: Upgrade: Setting container_image for all mgr 2024-08-17T17:13:55.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:55.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-08-17T17:13:55.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-08-17T17:13:55.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-08-17T17:13:55.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-08-17T17:13:55.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-08-17T17:13:55.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-08-17T17:13:55.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:55.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: Upgrade: Setting container_image for all crash 2024-08-17T17:13:55.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:55.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:55.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: Upgrade: Setting container_image for all rgw 2024-08-17T17:13:55.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:55.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:55.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: Upgrade: Setting container_image for all rbd-mirror 2024-08-17T17:13:55.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:55.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:55.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: Upgrade: Setting container_image for all cephfs-mirror 2024-08-17T17:13:55.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:55.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:55.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: Upgrade: Setting container_image for all ceph-exporter 2024-08-17T17:13:55.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:55.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:55.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: Upgrade: Setting container_image for all iscsi 2024-08-17T17:13:55.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:55.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:55.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: Upgrade: Setting container_image for all nfs 2024-08-17T17:13:55.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:55.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:13:55.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: Upgrade: Setting container_image for all nvmeof 2024-08-17T17:13:55.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:54 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:13:56.158 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:55 smithi136 ceph-mon[31034]: pgmap v47: 137 pgs: 137 active+clean; 693 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 50 MiB/s rd, 6.0 MiB/s wr, 541 op/s 2024-08-17T17:13:56.158 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:55 smithi136 ceph-mon[31034]: osdmap e221: 8 total, 8 up, 8 in 2024-08-17T17:13:56.158 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:55 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:13:56.158 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:55 smithi136 ceph-mon[31034]: Upgrade: Updating node-exporter.a (1/2) 2024-08-17T17:13:56.158 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:55 smithi136 ceph-mon[31034]: Deploying daemon node-exporter.a on smithi136 2024-08-17T17:13:56.159 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:55 smithi136 ceph-mon[42580]: pgmap v47: 137 pgs: 137 active+clean; 693 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 50 MiB/s rd, 6.0 MiB/s wr, 541 op/s 2024-08-17T17:13:56.159 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:55 smithi136 ceph-mon[42580]: osdmap e221: 8 total, 8 up, 8 in 2024-08-17T17:13:56.159 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:55 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:13:56.159 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:55 smithi136 ceph-mon[42580]: Upgrade: Updating node-exporter.a (1/2) 2024-08-17T17:13:56.159 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:55 smithi136 ceph-mon[42580]: Deploying daemon node-exporter.a on smithi136 2024-08-17T17:13:56.159 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:55 smithi136 systemd[1]: Stopping Ceph node-exporter.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:13:56.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:55 smithi165 ceph-mon[34868]: pgmap v47: 137 pgs: 137 active+clean; 693 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 50 MiB/s rd, 6.0 MiB/s wr, 541 op/s 2024-08-17T17:13:56.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:55 smithi165 ceph-mon[34868]: osdmap e221: 8 total, 8 up, 8 in 2024-08-17T17:13:56.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:55 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:13:56.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:55 smithi165 ceph-mon[34868]: Upgrade: Updating node-exporter.a (1/2) 2024-08-17T17:13:56.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:55 smithi165 ceph-mon[34868]: Deploying daemon node-exporter.a on smithi136 2024-08-17T17:13:56.446 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:56 smithi136 podman[100233]: 2024-08-17 17:13:56.157557545 +0000 UTC m=+0.099158124 container died 6aef82a780f5bed31be412ba7b88ff8f2f8976f47f2f777be22a73b5cd6855fb (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-08-17T17:13:56.447 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:13:56 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:13:56] "GET /metrics HTTP/1.1" 200 38726 "" "Prometheus/2.43.0" 2024-08-17T17:13:56.783 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:56 smithi136 podman[100233]: 2024-08-17 17:13:56.645111143 +0000 UTC m=+0.586711721 container cleanup 6aef82a780f5bed31be412ba7b88ff8f2f8976f47f2f777be22a73b5cd6855fb (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-08-17T17:13:56.784 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:56 smithi136 bash[100233]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a 2024-08-17T17:13:56.795 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 8388609~1000 2024-08-17T17:13:56.805 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 16777228~1000 2024-08-17T17:13:56.805 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 167772160 2024-08-17T17:13:56.812 INFO:tasks.workunit.client.0.smithi136.stdout:write: 2 83886079~8388609 2024-08-17T17:13:56.833 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap3 2024-08-17T17:13:57.051 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:56 smithi136 ceph-mon[31034]: osdmap e222: 8 total, 8 up, 8 in 2024-08-17T17:13:57.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:56 smithi136 ceph-mon[42580]: osdmap e222: 8 total, 8 up, 8 in 2024-08-17T17:13:57.051 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:56 smithi136 podman[100250]: 2024-08-17 17:13:56.897148268 +0000 UTC m=+0.739845307 container remove 6aef82a780f5bed31be412ba7b88ff8f2f8976f47f2f777be22a73b5cd6855fb (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-08-17T17:13:57.052 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:56 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-08-17T17:13:57.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:56 smithi165 ceph-mon[34868]: osdmap e222: 8 total, 8 up, 8 in 2024-08-17T17:13:57.447 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:57 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-08-17T17:13:57.447 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:57 smithi136 systemd[1]: Stopped Ceph node-exporter.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:13:57.447 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:57 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@node-exporter.a.service: Consumed 2.370s CPU time. 2024-08-17T17:13:57.447 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:57 smithi136 systemd[1]: Starting Ceph node-exporter.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:13:57.778 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:57 smithi136 bash[100387]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-08-17T17:13:58.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:57 smithi136 ceph-mon[31034]: pgmap v50: 137 pgs: 137 active+clean; 705 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 39 MiB/s rd, 7.8 MiB/s wr, 135 op/s 2024-08-17T17:13:58.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:57 smithi136 ceph-mon[31034]: osdmap e223: 8 total, 8 up, 8 in 2024-08-17T17:13:58.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:57 smithi136 ceph-mon[31034]: osdmap e224: 8 total, 8 up, 8 in 2024-08-17T17:13:58.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:57 smithi136 ceph-mon[42580]: pgmap v50: 137 pgs: 137 active+clean; 705 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 39 MiB/s rd, 7.8 MiB/s wr, 135 op/s 2024-08-17T17:13:58.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:57 smithi136 ceph-mon[42580]: osdmap e223: 8 total, 8 up, 8 in 2024-08-17T17:13:58.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:57 smithi136 ceph-mon[42580]: osdmap e224: 8 total, 8 up, 8 in 2024-08-17T17:13:58.197 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:57 smithi136 bash[100387]: Getting image source signatures 2024-08-17T17:13:58.197 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:57 smithi136 bash[100387]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-08-17T17:13:58.197 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:57 smithi136 bash[100387]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-08-17T17:13:58.197 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:57 smithi136 bash[100387]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-08-17T17:13:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:57 smithi165 ceph-mon[34868]: pgmap v50: 137 pgs: 137 active+clean; 705 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 39 MiB/s rd, 7.8 MiB/s wr, 135 op/s 2024-08-17T17:13:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:57 smithi165 ceph-mon[34868]: osdmap e223: 8 total, 8 up, 8 in 2024-08-17T17:13:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:57 smithi165 ceph-mon[34868]: osdmap e224: 8 total, 8 up, 8 in 2024-08-17T17:13:58.522 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:13:58.543 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 83886079~1 2024-08-17T17:13:58.552 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:13:59.446 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 bash[100387]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-08-17T17:13:59.447 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 bash[100387]: Writing manifest to image destination 2024-08-17T17:13:59.767 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-mon[42580]: osdmap e225: 8 total, 8 up, 8 in 2024-08-17T17:13:59.767 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-mon[42580]: pgmap v54: 137 pgs: 137 active+clean; 705 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 42 MiB/s rd, 8.4 MiB/s wr, 144 op/s 2024-08-17T17:13:59.767 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-mon[31034]: osdmap e225: 8 total, 8 up, 8 in 2024-08-17T17:13:59.767 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-mon[31034]: pgmap v54: 137 pgs: 137 active+clean; 705 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 42 MiB/s rd, 8.4 MiB/s wr, 144 op/s 2024-08-17T17:13:59.768 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 podman[100387]: 2024-08-17 17:13:59.537324363 +0000 UTC m=+2.171519797 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-08-17T17:13:59.768 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 podman[100387]: 2024-08-17 17:13:59.661177799 +0000 UTC m=+2.295373220 container create 1e4c644ae533e78f10c35439b2f88a95d12c25e345e9429afad64f85b253c5ea (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-08-17T17:14:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:59 smithi165 ceph-mon[34868]: osdmap e225: 8 total, 8 up, 8 in 2024-08-17T17:14:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:13:59 smithi165 ceph-mon[34868]: pgmap v54: 137 pgs: 137 active+clean; 705 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 42 MiB/s rd, 8.4 MiB/s wr, 144 op/s 2024-08-17T17:14:00.197 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 podman[100387]: 2024-08-17 17:13:59.880023155 +0000 UTC m=+2.514218575 container init 1e4c644ae533e78f10c35439b2f88a95d12c25e345e9429afad64f85b253c5ea (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-08-17T17:14:00.197 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 podman[100387]: 2024-08-17 17:13:59.884612921 +0000 UTC m=+2.518808342 container start 1e4c644ae533e78f10c35439b2f88a95d12c25e345e9429afad64f85b253c5ea (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-08-17T17:14:00.197 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.888Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-08-17T17:14:00.197 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.888Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-08-17T17:14:00.197 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.888Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-08-17T17:14:00.197 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.888Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-08-17T17:14:00.197 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.888Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-08-17T17:14:00.198 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.888Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-08-17T17:14:00.198 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-08-17T17:14:00.198 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=arp 2024-08-17T17:14:00.198 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=bcache 2024-08-17T17:14:00.198 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=bonding 2024-08-17T17:14:00.198 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=btrfs 2024-08-17T17:14:00.198 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=conntrack 2024-08-17T17:14:00.198 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=cpu 2024-08-17T17:14:00.198 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-08-17T17:14:00.199 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=diskstats 2024-08-17T17:14:00.199 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=dmi 2024-08-17T17:14:00.199 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=edac 2024-08-17T17:14:00.199 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=entropy 2024-08-17T17:14:00.199 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-08-17T17:14:00.199 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=filefd 2024-08-17T17:14:00.199 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=filesystem 2024-08-17T17:14:00.199 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=hwmon 2024-08-17T17:14:00.199 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=infiniband 2024-08-17T17:14:00.200 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=ipvs 2024-08-17T17:14:00.200 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=loadavg 2024-08-17T17:14:00.200 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=mdadm 2024-08-17T17:14:00.200 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=meminfo 2024-08-17T17:14:00.200 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=netclass 2024-08-17T17:14:00.200 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=netdev 2024-08-17T17:14:00.200 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=netstat 2024-08-17T17:14:00.200 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=nfs 2024-08-17T17:14:00.200 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=nfsd 2024-08-17T17:14:00.200 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=nvme 2024-08-17T17:14:00.200 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=os 2024-08-17T17:14:00.200 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-08-17T17:14:00.201 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=pressure 2024-08-17T17:14:00.201 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=rapl 2024-08-17T17:14:00.201 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=schedstat 2024-08-17T17:14:00.201 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=selinux 2024-08-17T17:14:00.201 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=sockstat 2024-08-17T17:14:00.201 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=softnet 2024-08-17T17:14:00.201 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=stat 2024-08-17T17:14:00.201 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=tapestats 2024-08-17T17:14:00.201 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=textfile 2024-08-17T17:14:00.201 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-08-17T17:14:00.201 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=time 2024-08-17T17:14:00.201 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-08-17T17:14:00.201 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=uname 2024-08-17T17:14:00.202 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=vmstat 2024-08-17T17:14:00.202 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=xfs 2024-08-17T17:14:00.202 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=node_exporter.go:117 level=info collector=zfs 2024-08-17T17:14:00.202 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-08-17T17:14:00.202 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a[100436]: ts=2024-08-17T17:13:59.889Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-08-17T17:14:00.202 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 bash[100387]: 1e4c644ae533e78f10c35439b2f88a95d12c25e345e9429afad64f85b253c5ea 2024-08-17T17:14:00.202 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 17:13:59 smithi136 systemd[1]: Started Ceph node-exporter.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:14:00.693 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:00 smithi165 ceph-mon[34868]: osdmap e226: 8 total, 8 up, 8 in 2024-08-17T17:14:00.693 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:00 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:00.693 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:00 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:00.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:00 smithi136 ceph-mon[31034]: osdmap e226: 8 total, 8 up, 8 in 2024-08-17T17:14:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:00 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:00 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:00 smithi136 ceph-mon[42580]: osdmap e226: 8 total, 8 up, 8 in 2024-08-17T17:14:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:00 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:00 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:01.935 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: after_prepare_snap 2024-08-17T17:14:01.939 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:01 smithi165 ceph-mon[34868]: osdmap e227: 8 total, 8 up, 8 in 2024-08-17T17:14:01.939 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:01 smithi165 ceph-mon[34868]: pgmap v57: 137 pgs: 137 active+clean; 705 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:01.939 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:01 smithi165 ceph-mon[34868]: Upgrade: Updating node-exporter.b (2/2) 2024-08-17T17:14:01.939 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:01 smithi165 ceph-mon[34868]: Deploying daemon node-exporter.b on smithi165 2024-08-17T17:14:01.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:01 smithi136 ceph-mon[31034]: osdmap e227: 8 total, 8 up, 8 in 2024-08-17T17:14:01.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:01 smithi136 ceph-mon[31034]: pgmap v57: 137 pgs: 137 active+clean; 705 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:01.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:01 smithi136 ceph-mon[31034]: Upgrade: Updating node-exporter.b (2/2) 2024-08-17T17:14:01.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:01 smithi136 ceph-mon[31034]: Deploying daemon node-exporter.b on smithi165 2024-08-17T17:14:01.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:01 smithi136 ceph-mon[42580]: osdmap e227: 8 total, 8 up, 8 in 2024-08-17T17:14:01.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:01 smithi136 ceph-mon[42580]: pgmap v57: 137 pgs: 137 active+clean; 705 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:01.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:01 smithi136 ceph-mon[42580]: Upgrade: Updating node-exporter.b (2/2) 2024-08-17T17:14:01.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:01 smithi136 ceph-mon[42580]: Deploying daemon node-exporter.b on smithi165 2024-08-17T17:14:02.222 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:01 smithi165 systemd[1]: Stopping Ceph node-exporter.b for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:14:02.494 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:02 smithi165 podman[86582]: 2024-08-17 17:14:02.221353463 +0000 UTC m=+0.132274268 container died 60b065dd0c2ad48a4f1a606481de4defb842a305c5a06f88623ea21a137f012e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-08-17T17:14:02.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:02 smithi165 ceph-mon[34868]: osdmap e228: 8 total, 8 up, 8 in 2024-08-17T17:14:02.761 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:02 smithi165 podman[86582]: 2024-08-17 17:14:02.575174857 +0000 UTC m=+0.486095667 container cleanup 60b065dd0c2ad48a4f1a606481de4defb842a305c5a06f88623ea21a137f012e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-08-17T17:14:02.761 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:02 smithi165 bash[86582]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b 2024-08-17T17:14:02.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:02 smithi136 ceph-mon[31034]: osdmap e228: 8 total, 8 up, 8 in 2024-08-17T17:14:02.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:02 smithi136 ceph-mon[42580]: osdmap e228: 8 total, 8 up, 8 in 2024-08-17T17:14:03.228 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:02 smithi165 podman[86600]: 2024-08-17 17:14:02.884485625 +0000 UTC m=+0.666969668 container remove 60b065dd0c2ad48a4f1a606481de4defb842a305c5a06f88623ea21a137f012e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-08-17T17:14:03.229 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:02 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-08-17T17:14:03.229 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:03 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-08-17T17:14:03.229 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:03 smithi165 systemd[1]: Stopped Ceph node-exporter.b for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:14:03.229 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:03 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@node-exporter.b.service: Consumed 2.322s CPU time. 2024-08-17T17:14:03.229 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:03 smithi165 systemd[1]: Starting Ceph node-exporter.b for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:14:03.511 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:03 smithi165 bash[86730]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-08-17T17:14:03.550 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 167772160 2024-08-17T17:14:03.562 INFO:tasks.workunit.client.0.smithi136.stdout:write: * 83886080~1000 2024-08-17T17:14:03.569 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:14:03.717 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:14:03.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:03 smithi165 ceph-mon[34868]: osdmap e229: 8 total, 8 up, 8 in 2024-08-17T17:14:03.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:03 smithi165 ceph-mon[34868]: pgmap v60: 137 pgs: 137 active+clean; 729 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 121 KiB/s rd, 4.0 MiB/s wr, 173 op/s 2024-08-17T17:14:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:03 smithi136 ceph-mon[31034]: osdmap e229: 8 total, 8 up, 8 in 2024-08-17T17:14:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:03 smithi136 ceph-mon[31034]: pgmap v60: 137 pgs: 137 active+clean; 729 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 121 KiB/s rd, 4.0 MiB/s wr, 173 op/s 2024-08-17T17:14:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:03 smithi136 ceph-mon[42580]: osdmap e229: 8 total, 8 up, 8 in 2024-08-17T17:14:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:03 smithi136 ceph-mon[42580]: pgmap v60: 137 pgs: 137 active+clean; 729 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 121 KiB/s rd, 4.0 MiB/s wr, 173 op/s 2024-08-17T17:14:04.261 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:03 smithi165 bash[86730]: Getting image source signatures 2024-08-17T17:14:04.261 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:03 smithi165 bash[86730]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-08-17T17:14:04.261 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:03 smithi165 bash[86730]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-08-17T17:14:04.261 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:03 smithi165 bash[86730]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-08-17T17:14:04.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:04 smithi136 ceph-mon[31034]: osdmap e230: 8 total, 8 up, 8 in 2024-08-17T17:14:04.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:04 smithi136 ceph-mon[42580]: osdmap e230: 8 total, 8 up, 8 in 2024-08-17T17:14:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:04 smithi165 ceph-mon[34868]: osdmap e230: 8 total, 8 up, 8 in 2024-08-17T17:14:05.511 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:05 smithi165 bash[86730]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-08-17T17:14:05.511 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:05 smithi165 bash[86730]: Writing manifest to image destination 2024-08-17T17:14:05.818 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:05 smithi165 ceph-mon[34868]: osdmap e231: 8 total, 8 up, 8 in 2024-08-17T17:14:05.818 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:05 smithi165 ceph-mon[34868]: pgmap v63: 137 pgs: 137 active+clean; 729 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 121 KiB/s rd, 4.0 MiB/s wr, 173 op/s 2024-08-17T17:14:05.818 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:05 smithi165 podman[86730]: 2024-08-17 17:14:05.69507419 +0000 UTC m=+2.368910076 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-08-17T17:14:05.819 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:05 smithi165 podman[86730]: 2024-08-17 17:14:05.818331321 +0000 UTC m=+2.492167205 container create 7af2d2d68b06803956a927505275969bec7debc0062a1c3981716600864cd82f (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-08-17T17:14:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:05 smithi136 ceph-mon[31034]: osdmap e231: 8 total, 8 up, 8 in 2024-08-17T17:14:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:05 smithi136 ceph-mon[31034]: pgmap v63: 137 pgs: 137 active+clean; 729 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 121 KiB/s rd, 4.0 MiB/s wr, 173 op/s 2024-08-17T17:14:05.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:05 smithi136 ceph-mon[42580]: osdmap e231: 8 total, 8 up, 8 in 2024-08-17T17:14:05.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:05 smithi136 ceph-mon[42580]: pgmap v63: 137 pgs: 137 active+clean; 729 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 121 KiB/s rd, 4.0 MiB/s wr, 173 op/s 2024-08-17T17:14:06.377 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 podman[86730]: 2024-08-17 17:14:06.070260469 +0000 UTC m=+2.744096365 container init 7af2d2d68b06803956a927505275969bec7debc0062a1c3981716600864cd82f (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-08-17T17:14:06.378 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 podman[86730]: 2024-08-17 17:14:06.075453276 +0000 UTC m=+2.749289162 container start 7af2d2d68b06803956a927505275969bec7debc0062a1c3981716600864cd82f (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-08-17T17:14:06.378 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.079Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-08-17T17:14:06.378 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.079Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-08-17T17:14:06.378 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.079Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-08-17T17:14:06.378 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.079Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-08-17T17:14:06.378 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-08-17T17:14:06.378 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-08-17T17:14:06.378 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-08-17T17:14:06.378 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=arp 2024-08-17T17:14:06.378 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=bcache 2024-08-17T17:14:06.378 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=bonding 2024-08-17T17:14:06.379 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=btrfs 2024-08-17T17:14:06.379 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=conntrack 2024-08-17T17:14:06.379 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=cpu 2024-08-17T17:14:06.379 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-08-17T17:14:06.379 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=diskstats 2024-08-17T17:14:06.379 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=dmi 2024-08-17T17:14:06.379 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=edac 2024-08-17T17:14:06.380 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=entropy 2024-08-17T17:14:06.380 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-08-17T17:14:06.380 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=filefd 2024-08-17T17:14:06.380 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=filesystem 2024-08-17T17:14:06.380 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=hwmon 2024-08-17T17:14:06.380 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=infiniband 2024-08-17T17:14:06.380 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=ipvs 2024-08-17T17:14:06.380 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=loadavg 2024-08-17T17:14:06.380 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=mdadm 2024-08-17T17:14:06.380 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=meminfo 2024-08-17T17:14:06.380 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=netclass 2024-08-17T17:14:06.380 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=netdev 2024-08-17T17:14:06.381 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=netstat 2024-08-17T17:14:06.381 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=nfs 2024-08-17T17:14:06.381 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=nfsd 2024-08-17T17:14:06.381 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=nvme 2024-08-17T17:14:06.381 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=os 2024-08-17T17:14:06.381 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-08-17T17:14:06.381 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=pressure 2024-08-17T17:14:06.381 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=rapl 2024-08-17T17:14:06.381 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=schedstat 2024-08-17T17:14:06.381 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=selinux 2024-08-17T17:14:06.381 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=sockstat 2024-08-17T17:14:06.381 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=softnet 2024-08-17T17:14:06.382 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=stat 2024-08-17T17:14:06.382 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=tapestats 2024-08-17T17:14:06.382 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=textfile 2024-08-17T17:14:06.382 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-08-17T17:14:06.382 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=time 2024-08-17T17:14:06.382 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-08-17T17:14:06.382 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=uname 2024-08-17T17:14:06.382 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=vmstat 2024-08-17T17:14:06.382 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=xfs 2024-08-17T17:14:06.382 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=node_exporter.go:117 level=info collector=zfs 2024-08-17T17:14:06.382 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-08-17T17:14:06.382 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b[86780]: ts=2024-08-17T17:14:06.080Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-08-17T17:14:06.382 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 bash[86730]: 7af2d2d68b06803956a927505275969bec7debc0062a1c3981716600864cd82f 2024-08-17T17:14:06.544 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:14:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:14:06] "GET /metrics HTTP/1.1" 200 38726 "" "Prometheus/2.43.0" 2024-08-17T17:14:06.545 INFO:teuthology.orchestra.run.smithi136.stdout:true 2024-08-17T17:14:06.761 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 17:14:06 smithi165 systemd[1]: Started Ceph node-exporter.b for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:14:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-mon[34868]: osdmap e232: 8 total, 8 up, 8 in 2024-08-17T17:14:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:06 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:14:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:06 smithi136 ceph-mon[31034]: osdmap e232: 8 total, 8 up, 8 in 2024-08-17T17:14:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:06 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:06.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:06 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:06.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:06 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:14:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:06 smithi136 ceph-mon[42580]: osdmap e232: 8 total, 8 up, 8 in 2024-08-17T17:14:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:06 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:06 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:06 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:14:06.997 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.SnapsSnapExpandAfterPrepare (14177 ms) 2024-08-17T17:14:06.997 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.Clone 2024-08-17T17:14:07.030 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:14:07.041 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 0~8388609 2024-08-17T17:14:07.112 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 16777218~8388609 2024-08-17T17:14:07.162 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap 2024-08-17T17:14:07.255 INFO:teuthology.orchestra.run.smithi136.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-17T17:14:07.256 INFO:teuthology.orchestra.run.smithi136.stdout:alertmanager.a smithi136 *:9093,9094 running (4m) 37s ago 5m 27.8M - 0.25.0 c8568f914cd2 83b6bfc5306f 2024-08-17T17:14:07.256 INFO:teuthology.orchestra.run.smithi136.stdout:grafana.a smithi165 *:3000 running (4m) 17s ago 4m 90.0M - 9.4.7 954c08fa6188 be80a687c402 2024-08-17T17:14:07.256 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi136.ltsuqd smithi136 running (3m) 37s ago 3m 18.2M - 18.2.4-719-gced649f8 c870753d67c2 faaa06f2950c 2024-08-17T17:14:07.256 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi165.bmrbhh smithi165 running (3m) 17s ago 3m 15.9M - 18.2.4-719-gced649f8 c870753d67c2 c74b00d929cf 2024-08-17T17:14:07.256 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.x smithi165 *:8443,9283,8765 running (22s) 17s ago 8m 55.9M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 837fd48ff7cb 2024-08-17T17:14:07.256 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.y smithi136 *:8443,9283,8765 running (51s) 37s ago 10m 489M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9acdc6775e48 2024-08-17T17:14:07.256 INFO:teuthology.orchestra.run.smithi136.stdout:mon.a smithi136 running (10m) 37s ago 10m 53.7M 2048M 18.2.4-719-gced649f8 c870753d67c2 cc4e5e5ff756 2024-08-17T17:14:07.256 INFO:teuthology.orchestra.run.smithi136.stdout:mon.b smithi165 running (9m) 17s ago 9m 47.3M 2048M 18.2.4-719-gced649f8 c870753d67c2 d587e16140e1 2024-08-17T17:14:07.256 INFO:teuthology.orchestra.run.smithi136.stdout:mon.c smithi136 running (9m) 37s ago 9m 45.8M 2048M 18.2.4-719-gced649f8 c870753d67c2 b4e0f3c9e485 2024-08-17T17:14:07.256 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.a smithi136 *:9100 starting - - - - 2024-08-17T17:14:07.256 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.b smithi165 *:9100 starting - - - - 2024-08-17T17:14:07.256 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0 smithi136 running (8m) 37s ago 8m 115M 2480M 18.2.4-719-gced649f8 c870753d67c2 faab916734b0 2024-08-17T17:14:07.256 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1 smithi136 running (8m) 37s ago 8m 102M 2480M 18.2.4-719-gced649f8 c870753d67c2 861686b6ff82 2024-08-17T17:14:07.257 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2 smithi136 running (7m) 37s ago 7m 97.5M 2480M 18.2.4-719-gced649f8 c870753d67c2 640be4af4156 2024-08-17T17:14:07.257 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3 smithi136 running (7m) 37s ago 7m 87.9M 2480M 18.2.4-719-gced649f8 c870753d67c2 14b57a745d30 2024-08-17T17:14:07.257 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4 smithi165 running (7m) 17s ago 7m 106M 2480M 18.2.4-719-gced649f8 c870753d67c2 df366c73e1f2 2024-08-17T17:14:07.257 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5 smithi165 running (6m) 17s ago 6m 88.9M 2480M 18.2.4-719-gced649f8 c870753d67c2 1131665e203a 2024-08-17T17:14:07.257 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6 smithi165 running (6m) 17s ago 6m 124M 2480M 18.2.4-719-gced649f8 c870753d67c2 7bc5fd925293 2024-08-17T17:14:07.257 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7 smithi165 running (5m) 17s ago 5m 92.3M 2480M 18.2.4-719-gced649f8 c870753d67c2 18382e4ae3b8 2024-08-17T17:14:07.257 INFO:teuthology.orchestra.run.smithi136.stdout:prometheus.a smithi165 *:9095 running (28s) 17s ago 5m 37.4M - 2.43.0 a07b618ecd1d f3e579e6d2fb 2024-08-17T17:14:07.597 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:14:07.597 INFO:teuthology.orchestra.run.smithi136.stdout: "mon": { 2024-08-17T17:14:07.597 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 3 2024-08-17T17:14:07.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:14:07.598 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr": { 2024-08-17T17:14:07.598 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 2 2024-08-17T17:14:07.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:14:07.598 INFO:teuthology.orchestra.run.smithi136.stdout: "osd": { 2024-08-17T17:14:07.598 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 8 2024-08-17T17:14:07.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:14:07.598 INFO:teuthology.orchestra.run.smithi136.stdout: "mds": { 2024-08-17T17:14:07.598 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 2 2024-08-17T17:14:07.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:14:07.598 INFO:teuthology.orchestra.run.smithi136.stdout: "overall": { 2024-08-17T17:14:07.598 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 13, 2024-08-17T17:14:07.598 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 2 2024-08-17T17:14:07.599 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:14:07.599 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:14:07.821 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:07 smithi136 ceph-mon[42580]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:07.821 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:07 smithi136 ceph-mon[42580]: osdmap e233: 8 total, 8 up, 8 in 2024-08-17T17:14:07.821 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:07 smithi136 ceph-mon[42580]: pgmap v66: 137 pgs: 137 active+clean; 749 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 501 op/s 2024-08-17T17:14:07.821 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:07 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:07.821 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:07 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:07.821 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:07 smithi136 ceph-mon[42580]: from='client.25031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:07.821 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:07 smithi136 ceph-mon[42580]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:07.822 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:07 smithi136 ceph-mon[31034]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:07.822 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:07 smithi136 ceph-mon[31034]: osdmap e233: 8 total, 8 up, 8 in 2024-08-17T17:14:07.822 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:07 smithi136 ceph-mon[31034]: pgmap v66: 137 pgs: 137 active+clean; 749 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 501 op/s 2024-08-17T17:14:07.822 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:07 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:07.822 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:07 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:07.822 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:07 smithi136 ceph-mon[31034]: from='client.25031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:07.822 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:07 smithi136 ceph-mon[31034]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:07.869 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:07 smithi165 ceph-mon[34868]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:07.869 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:07 smithi165 ceph-mon[34868]: osdmap e233: 8 total, 8 up, 8 in 2024-08-17T17:14:07.869 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:07 smithi165 ceph-mon[34868]: pgmap v66: 137 pgs: 137 active+clean; 749 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 501 op/s 2024-08-17T17:14:07.869 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:07 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:07.869 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:07 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:07.869 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:07 smithi165 ceph-mon[34868]: from='client.25031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:07.869 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:07 smithi165 ceph-mon[34868]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:07.892 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:14:07.892 INFO:teuthology.orchestra.run.smithi136.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d", 2024-08-17T17:14:07.892 INFO:teuthology.orchestra.run.smithi136.stdout: "in_progress": true, 2024-08-17T17:14:07.893 INFO:teuthology.orchestra.run.smithi136.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-08-17T17:14:07.893 INFO:teuthology.orchestra.run.smithi136.stdout: "services_complete": [ 2024-08-17T17:14:07.893 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr" 2024-08-17T17:14:07.893 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-17T17:14:07.893 INFO:teuthology.orchestra.run.smithi136.stdout: "progress": "2/2 daemons upgraded", 2024-08-17T17:14:07.893 INFO:teuthology.orchestra.run.smithi136.stdout: "message": "Currently upgrading node-exporter daemons", 2024-08-17T17:14:07.893 INFO:teuthology.orchestra.run.smithi136.stdout: "is_paused": false 2024-08-17T17:14:07.893 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:14:08.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:08 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/2036842661' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:08.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:08 smithi136 ceph-mon[31034]: osdmap e234: 8 total, 8 up, 8 in 2024-08-17T17:14:08.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:08 smithi136 ceph-mon[31034]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:08.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:08 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/2036842661' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:08.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:08 smithi136 ceph-mon[42580]: osdmap e234: 8 total, 8 up, 8 in 2024-08-17T17:14:08.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:08 smithi136 ceph-mon[42580]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:08.821 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap 2024-08-17T17:14:08.831 INFO:tasks.workunit.client.0.smithi136.stdout:clone image105 -> image108 2024-08-17T17:14:08.927 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 0~1000 2024-08-17T17:14:08.983 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:08 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2036842661' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:08.983 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:08 smithi165 ceph-mon[34868]: osdmap e234: 8 total, 8 up, 8 in 2024-08-17T17:14:08.983 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:08 smithi165 ceph-mon[34868]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:08.984 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 1010~1000 2024-08-17T17:14:08.987 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap 2024-08-17T17:14:10.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:09 smithi165 ceph-mon[34868]: pgmap v68: 137 pgs: 137 active+clean; 749 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 494 op/s 2024-08-17T17:14:10.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:09 smithi165 ceph-mon[34868]: osdmap e235: 8 total, 8 up, 8 in 2024-08-17T17:14:10.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:09 smithi136 ceph-mon[31034]: pgmap v68: 137 pgs: 137 active+clean; 749 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 494 op/s 2024-08-17T17:14:10.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:09 smithi136 ceph-mon[31034]: osdmap e235: 8 total, 8 up, 8 in 2024-08-17T17:14:10.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:09 smithi136 ceph-mon[42580]: pgmap v68: 137 pgs: 137 active+clean; 749 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 494 op/s 2024-08-17T17:14:10.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:09 smithi136 ceph-mon[42580]: osdmap e235: 8 total, 8 up, 8 in 2024-08-17T17:14:10.829 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap 2024-08-17T17:14:10.839 INFO:tasks.workunit.client.0.smithi136.stdout:clone image108 -> image110 2024-08-17T17:14:10.965 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 1000~1000 2024-08-17T17:14:11.022 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 2010~1000 2024-08-17T17:14:11.024 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:14:11.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:10 smithi136 ceph-mon[42580]: osdmap e236: 8 total, 8 up, 8 in 2024-08-17T17:14:11.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:10 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:14:11.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:10 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:11.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:10 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:11.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:10 smithi136 ceph-mon[31034]: osdmap e236: 8 total, 8 up, 8 in 2024-08-17T17:14:11.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:10 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:14:11.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:10 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:11.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:10 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:10 smithi165 ceph-mon[34868]: osdmap e236: 8 total, 8 up, 8 in 2024-08-17T17:14:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:10 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:14:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:10 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:10 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:11.340 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:14:11.540 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:14:11.749 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.Clone (4752 ms) 2024-08-17T17:14:11.749 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.CloneParent 2024-08-17T17:14:11.786 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:14:11.797 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap 2024-08-17T17:14:12.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:11 smithi136 ceph-mon[42580]: pgmap v71: 137 pgs: 137 active+clean; 749 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:12.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:11 smithi136 ceph-mon[42580]: osdmap e237: 8 total, 8 up, 8 in 2024-08-17T17:14:12.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:11 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:12.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:11 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:12.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:11 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:12.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:11 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:12.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:11 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:12.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:11 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:12.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:11 smithi136 ceph-mon[31034]: pgmap v71: 137 pgs: 137 active+clean; 749 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:12.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:11 smithi136 ceph-mon[31034]: osdmap e237: 8 total, 8 up, 8 in 2024-08-17T17:14:12.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:11 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:12.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:11 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:12.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:11 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:12.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:11 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:12.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:11 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:12.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:11 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:12.171 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:11 smithi165 ceph-mon[34868]: pgmap v71: 137 pgs: 137 active+clean; 749 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:12.171 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:11 smithi165 ceph-mon[34868]: osdmap e237: 8 total, 8 up, 8 in 2024-08-17T17:14:12.171 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:11 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:12.171 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:11 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:12.172 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:11 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:12.172 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:11 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:12.172 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:11 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:12.172 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:11 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:13.532 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap 2024-08-17T17:14:13.542 INFO:tasks.workunit.client.0.smithi136.stdout:clone image111 -> image114 2024-08-17T17:14:13.639 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:14:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:13 smithi165 ceph-mon[34868]: osdmap e238: 8 total, 8 up, 8 in 2024-08-17T17:14:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:13 smithi165 ceph-mon[34868]: pgmap v74: 137 pgs: 137 active+clean; 769 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 42 MiB/s rd, 10 MiB/s wr, 232 op/s 2024-08-17T17:14:13.792 INFO:tasks.workunit.client.0.smithi136.stdout:migration_abort 2024-08-17T17:14:13.929 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.CloneParent (2180 ms) 2024-08-17T17:14:13.929 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.CloneUpdateAfterPrepare 2024-08-17T17:14:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:13 smithi136 ceph-mon[31034]: osdmap e238: 8 total, 8 up, 8 in 2024-08-17T17:14:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:13 smithi136 ceph-mon[31034]: pgmap v74: 137 pgs: 137 active+clean; 769 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 42 MiB/s rd, 10 MiB/s wr, 232 op/s 2024-08-17T17:14:13.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:13 smithi136 ceph-mon[42580]: osdmap e238: 8 total, 8 up, 8 in 2024-08-17T17:14:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:13 smithi136 ceph-mon[42580]: pgmap v74: 137 pgs: 137 active+clean; 769 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 42 MiB/s rd, 10 MiB/s wr, 232 op/s 2024-08-17T17:14:13.960 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:14:13.972 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 0~10 2024-08-17T17:14:13.981 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap 2024-08-17T17:14:14.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:14 smithi136 ceph-mon[31034]: osdmap e239: 8 total, 8 up, 8 in 2024-08-17T17:14:14.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:14 smithi136 ceph-mon[42580]: osdmap e239: 8 total, 8 up, 8 in 2024-08-17T17:14:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:14 smithi165 ceph-mon[34868]: osdmap e239: 8 total, 8 up, 8 in 2024-08-17T17:14:15.537 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap 2024-08-17T17:14:15.549 INFO:tasks.workunit.client.0.smithi136.stdout:clone image115 -> image118 2024-08-17T17:14:15.642 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:14:15.683 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:15 smithi165 ceph-mon[34868]: osdmap e240: 8 total, 8 up, 8 in 2024-08-17T17:14:15.683 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:15 smithi165 ceph-mon[34868]: pgmap v77: 137 pgs: 137 active+clean; 769 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 42 MiB/s rd, 10 MiB/s wr, 232 op/s 2024-08-17T17:14:15.683 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:15 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:15.683 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:15 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:15.683 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:15 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:15.683 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:15 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:15.683 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:15 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:14:15.683 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:15 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:14:15.683 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:15 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:15.684 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:15 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:14:15.684 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:15 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.684 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:15 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.684 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:15 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.684 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:15 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.684 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:15 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.684 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:15 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.684 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:15 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.684 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:15 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.684 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:15 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.684 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:15 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.795 INFO:tasks.workunit.client.0.smithi136.stdout:write: Y 0~1 2024-08-17T17:14:15.819 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:14:15.897 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:14:15.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[42580]: osdmap e240: 8 total, 8 up, 8 in 2024-08-17T17:14:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[42580]: pgmap v77: 137 pgs: 137 active+clean; 769 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 42 MiB/s rd, 10 MiB/s wr, 232 op/s 2024-08-17T17:14:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:14:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:14:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:14:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[31034]: osdmap e240: 8 total, 8 up, 8 in 2024-08-17T17:14:15.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[31034]: pgmap v77: 137 pgs: 137 active+clean; 769 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 42 MiB/s rd, 10 MiB/s wr, 232 op/s 2024-08-17T17:14:15.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:15.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:15.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:15.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:15.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:14:15.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:14:15.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:15.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:14:15.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:15.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:15 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:16.000 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.CloneUpdateAfterPrepare (2071 ms) 2024-08-17T17:14:16.000 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.TriggerAssertSnapcSeq 2024-08-17T17:14:16.029 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:14:16.041 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 41943040~10 2024-08-17T17:14:16.050 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:14:16.526 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:14:16 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:14:16] "GET /metrics HTTP/1.1" 200 38732 "" "Prometheus/2.43.0" 2024-08-17T17:14:16.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:16 smithi165 ceph-mon[34868]: osdmap e241: 8 total, 8 up, 8 in 2024-08-17T17:14:16.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:16 smithi136 ceph-mon[31034]: osdmap e241: 8 total, 8 up, 8 in 2024-08-17T17:14:16.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:16 smithi136 ceph-mon[42580]: osdmap e241: 8 total, 8 up, 8 in 2024-08-17T17:14:17.523 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 41943041~10 2024-08-17T17:14:17.528 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:14:17.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:17 smithi136 ceph-mon[31034]: Upgrade: Updating prometheus.a 2024-08-17T17:14:17.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:17 smithi136 ceph-mon[31034]: osdmap e242: 8 total, 8 up, 8 in 2024-08-17T17:14:17.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:17 smithi136 ceph-mon[31034]: pgmap v80: 137 pgs: 137 active+clean; 777 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 3.0 MiB/s wr, 1.25k op/s 2024-08-17T17:14:17.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:17 smithi136 ceph-mon[31034]: Deploying daemon prometheus.a on smithi165 2024-08-17T17:14:17.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:17 smithi136 ceph-mon[31034]: osdmap e243: 8 total, 8 up, 8 in 2024-08-17T17:14:17.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:17 smithi136 ceph-mon[42580]: Upgrade: Updating prometheus.a 2024-08-17T17:14:17.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:17 smithi136 ceph-mon[42580]: osdmap e242: 8 total, 8 up, 8 in 2024-08-17T17:14:17.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:17 smithi136 ceph-mon[42580]: pgmap v80: 137 pgs: 137 active+clean; 777 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 3.0 MiB/s wr, 1.25k op/s 2024-08-17T17:14:17.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:17 smithi136 ceph-mon[42580]: Deploying daemon prometheus.a on smithi165 2024-08-17T17:14:17.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:17 smithi136 ceph-mon[42580]: osdmap e243: 8 total, 8 up, 8 in 2024-08-17T17:14:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:17 smithi165 ceph-mon[34868]: Upgrade: Updating prometheus.a 2024-08-17T17:14:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:17 smithi165 ceph-mon[34868]: osdmap e242: 8 total, 8 up, 8 in 2024-08-17T17:14:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:17 smithi165 ceph-mon[34868]: pgmap v80: 137 pgs: 137 active+clean; 777 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 3.0 MiB/s wr, 1.25k op/s 2024-08-17T17:14:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:17 smithi165 ceph-mon[34868]: Deploying daemon prometheus.a on smithi165 2024-08-17T17:14:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:17 smithi165 ceph-mon[34868]: osdmap e243: 8 total, 8 up, 8 in 2024-08-17T17:14:18.669 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 41943042~10 2024-08-17T17:14:18.684 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap2 2024-08-17T17:14:19.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:19 smithi136 ceph-mon[42580]: osdmap e244: 8 total, 8 up, 8 in 2024-08-17T17:14:19.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:19 smithi136 ceph-mon[42580]: pgmap v83: 137 pgs: 137 active+clean; 777 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 3.0 MiB/s wr, 1.25k op/s 2024-08-17T17:14:19.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:19 smithi136 ceph-mon[31034]: osdmap e244: 8 total, 8 up, 8 in 2024-08-17T17:14:19.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:19 smithi136 ceph-mon[31034]: pgmap v83: 137 pgs: 137 active+clean; 777 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 3.0 MiB/s wr, 1.25k op/s 2024-08-17T17:14:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:19 smithi165 ceph-mon[34868]: osdmap e244: 8 total, 8 up, 8 in 2024-08-17T17:14:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:19 smithi165 ceph-mon[34868]: pgmap v83: 137 pgs: 137 active+clean; 777 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 3.0 MiB/s wr, 1.25k op/s 2024-08-17T17:14:20.566 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 41943040 2024-08-17T17:14:20.594 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:14:20.651 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:14:20.912 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:20 smithi165 ceph-mon[34868]: osdmap e245: 8 total, 8 up, 8 in 2024-08-17T17:14:20.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:20 smithi136 ceph-mon[31034]: osdmap e245: 8 total, 8 up, 8 in 2024-08-17T17:14:20.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:20 smithi136 ceph-mon[42580]: osdmap e245: 8 total, 8 up, 8 in 2024-08-17T17:14:21.674 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.TriggerAssertSnapcSeq (5674 ms) 2024-08-17T17:14:21.674 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.SnapTrimBeforePrepare 2024-08-17T17:14:21.702 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:14:21.712 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 41943040~10 2024-08-17T17:14:21.722 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:14:21.931 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:21 smithi165 ceph-mon[34868]: osdmap e246: 8 total, 8 up, 8 in 2024-08-17T17:14:21.932 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:21 smithi165 ceph-mon[34868]: pgmap v86: 137 pgs: 137 active+clean; 777 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:21 smithi136 ceph-mon[31034]: osdmap e246: 8 total, 8 up, 8 in 2024-08-17T17:14:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:21 smithi136 ceph-mon[31034]: pgmap v86: 137 pgs: 137 active+clean; 777 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:21 smithi136 ceph-mon[42580]: osdmap e246: 8 total, 8 up, 8 in 2024-08-17T17:14:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:21 smithi136 ceph-mon[42580]: pgmap v86: 137 pgs: 137 active+clean; 777 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:22.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:22 smithi136 ceph-mon[31034]: osdmap e247: 8 total, 8 up, 8 in 2024-08-17T17:14:22.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:22 smithi136 ceph-mon[42580]: osdmap e247: 8 total, 8 up, 8 in 2024-08-17T17:14:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:22 smithi165 ceph-mon[34868]: osdmap e247: 8 total, 8 up, 8 in 2024-08-17T17:14:23.589 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 41943040 2024-08-17T17:14:23.606 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:14:23.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:23 smithi136 ceph-mon[42580]: osdmap e248: 8 total, 8 up, 8 in 2024-08-17T17:14:23.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:23 smithi136 ceph-mon[42580]: pgmap v89: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 473 op/s 2024-08-17T17:14:23.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:23 smithi136 ceph-mon[31034]: osdmap e248: 8 total, 8 up, 8 in 2024-08-17T17:14:23.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:23 smithi136 ceph-mon[31034]: pgmap v89: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 473 op/s 2024-08-17T17:14:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:23 smithi165 ceph-mon[34868]: osdmap e248: 8 total, 8 up, 8 in 2024-08-17T17:14:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:23 smithi165 ceph-mon[34868]: pgmap v89: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 473 op/s 2024-08-17T17:14:24.696 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:14:24.707 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap3 2024-08-17T17:14:24.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:24 smithi165 ceph-mon[34868]: osdmap e249: 8 total, 8 up, 8 in 2024-08-17T17:14:24.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:24 smithi136 ceph-mon[31034]: osdmap e249: 8 total, 8 up, 8 in 2024-08-17T17:14:24.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:24 smithi136 ceph-mon[42580]: osdmap e249: 8 total, 8 up, 8 in 2024-08-17T17:14:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:25 smithi165 ceph-mon[34868]: osdmap e250: 8 total, 8 up, 8 in 2024-08-17T17:14:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:25 smithi165 ceph-mon[34868]: pgmap v92: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 473 op/s 2024-08-17T17:14:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:25 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:14:25.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:25 smithi136 ceph-mon[31034]: osdmap e250: 8 total, 8 up, 8 in 2024-08-17T17:14:25.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:25 smithi136 ceph-mon[31034]: pgmap v92: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 473 op/s 2024-08-17T17:14:25.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:25 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:14:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:25 smithi136 ceph-mon[42580]: osdmap e250: 8 total, 8 up, 8 in 2024-08-17T17:14:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:25 smithi136 ceph-mon[42580]: pgmap v92: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 473 op/s 2024-08-17T17:14:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:25 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:14:26.597 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:14:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:14:26] "GET /metrics HTTP/1.1" 200 38733 "" "Prometheus/2.43.0" 2024-08-17T17:14:26.608 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 41943040~10 2024-08-17T17:14:26.614 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap4 2024-08-17T17:14:26.915 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:26 smithi165 ceph-mon[34868]: osdmap e251: 8 total, 8 up, 8 in 2024-08-17T17:14:26.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:26 smithi136 ceph-mon[31034]: osdmap e251: 8 total, 8 up, 8 in 2024-08-17T17:14:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:26 smithi136 ceph-mon[42580]: osdmap e251: 8 total, 8 up, 8 in 2024-08-17T17:14:27.927 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:27 smithi165 ceph-mon[34868]: osdmap e252: 8 total, 8 up, 8 in 2024-08-17T17:14:27.928 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:27 smithi165 ceph-mon[34868]: pgmap v95: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 162 KiB/s rd, 17 KiB/s wr, 221 op/s 2024-08-17T17:14:27.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:27 smithi136 ceph-mon[31034]: osdmap e252: 8 total, 8 up, 8 in 2024-08-17T17:14:27.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:27 smithi136 ceph-mon[31034]: pgmap v95: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 162 KiB/s rd, 17 KiB/s wr, 221 op/s 2024-08-17T17:14:27.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:27 smithi136 ceph-mon[42580]: osdmap e252: 8 total, 8 up, 8 in 2024-08-17T17:14:27.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:27 smithi136 ceph-mon[42580]: pgmap v95: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 162 KiB/s rd, 17 KiB/s wr, 221 op/s 2024-08-17T17:14:28.620 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 41943040 2024-08-17T17:14:28.641 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:14:28.738 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:14:28.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:28 smithi165 ceph-mon[34868]: osdmap e253: 8 total, 8 up, 8 in 2024-08-17T17:14:28.761 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:28 smithi165 systemd[1]: Stopping Ceph prometheus.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:14:28.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:28 smithi136 ceph-mon[31034]: osdmap e253: 8 total, 8 up, 8 in 2024-08-17T17:14:28.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:28 smithi136 ceph-mon[42580]: osdmap e253: 8 total, 8 up, 8 in 2024-08-17T17:14:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:29 smithi165 ceph-mon[34868]: pgmap v97: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 160 KiB/s rd, 17 KiB/s wr, 220 op/s 2024-08-17T17:14:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:29 smithi165 ceph-mon[34868]: osdmap e254: 8 total, 8 up, 8 in 2024-08-17T17:14:29.761 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:29 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:14:29.502Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-08-17T17:14:29.761 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:29 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:14:29.503Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-08-17T17:14:29.761 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:29 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:14:29.503Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-08-17T17:14:29.762 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:29 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:14:29.503Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-08-17T17:14:29.762 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:29 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:14:29.503Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-08-17T17:14:29.762 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:29 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:14:29.503Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-08-17T17:14:29.762 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:29 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:14:29.503Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-08-17T17:14:29.762 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:29 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:14:29.503Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-08-17T17:14:29.762 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:29 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:14:29.503Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-08-17T17:14:29.762 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:29 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:14:29.504Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-08-17T17:14:29.762 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:29 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:14:29.504Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-08-17T17:14:29.762 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:29 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[81936]: ts=2024-08-17T17:14:29.504Z caller=main.go:1147 level=info msg="See you next time!" 2024-08-17T17:14:29.762 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:29 smithi165 podman[91107]: 2024-08-17 17:14:29.513496978 +0000 UTC m=+0.973460883 container died f3e579e6d2fb0001d3fa48ae0407be1fab5b829e265708cb98adde75e831c6cc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:14:29.805 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.SnapTrimBeforePrepare (8131 ms) 2024-08-17T17:14:29.805 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.AbortInUseImage 2024-08-17T17:14:29.838 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:14:29.848 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:14:29.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:29 smithi136 ceph-mon[31034]: pgmap v97: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 160 KiB/s rd, 17 KiB/s wr, 220 op/s 2024-08-17T17:14:29.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:29 smithi136 ceph-mon[31034]: osdmap e254: 8 total, 8 up, 8 in 2024-08-17T17:14:29.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:29 smithi136 ceph-mon[42580]: pgmap v97: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 160 KiB/s rd, 17 KiB/s wr, 220 op/s 2024-08-17T17:14:29.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:29 smithi136 ceph-mon[42580]: osdmap e254: 8 total, 8 up, 8 in 2024-08-17T17:14:30.025 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.AbortInUseImage (220 ms) 2024-08-17T17:14:30.025 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.AbortWithoutSnapshots 2024-08-17T17:14:30.056 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:14:30.066 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 0~8388609 2024-08-17T17:14:30.122 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 16777218~8388609 2024-08-17T17:14:30.168 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:14:30.356 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 0~8388609 2024-08-17T17:14:30.398 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 16777218~8388609 2024-08-17T17:14:30.451 INFO:tasks.workunit.client.0.smithi136.stdout:migration_abort 2024-08-17T17:14:30.511 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:30 smithi165 podman[91107]: 2024-08-17 17:14:30.30043716 +0000 UTC m=+1.760401064 container cleanup f3e579e6d2fb0001d3fa48ae0407be1fab5b829e265708cb98adde75e831c6cc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:14:30.511 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:30 smithi165 bash[91107]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a 2024-08-17T17:14:30.623 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.AbortWithoutSnapshots (598 ms) 2024-08-17T17:14:30.623 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.AbortWithSnapshots 2024-08-17T17:14:30.654 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:14:30.665 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 0~8388609 2024-08-17T17:14:30.721 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:14:30.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:30 smithi136 ceph-mon[31034]: osdmap e255: 8 total, 8 up, 8 in 2024-08-17T17:14:30.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:30 smithi136 ceph-mon[42580]: osdmap e255: 8 total, 8 up, 8 in 2024-08-17T17:14:30.955 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:30 smithi165 podman[91126]: 2024-08-17 17:14:30.568947803 +0000 UTC m=+1.056178698 container remove f3e579e6d2fb0001d3fa48ae0407be1fab5b829e265708cb98adde75e831c6cc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:14:30.955 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:30 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-08-17T17:14:30.956 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:30 smithi165 systemd[1]: Stopped Ceph prometheus.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:14:30.956 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:30 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@prometheus.a.service: Consumed 1.257s CPU time. 2024-08-17T17:14:30.956 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:30 smithi165 ceph-mon[34868]: osdmap e255: 8 total, 8 up, 8 in 2024-08-17T17:14:31.261 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:30 smithi165 systemd[1]: Starting Ceph prometheus.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:14:31.261 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 podman[91258]: 2024-08-17 17:14:31.074250619 +0000 UTC m=+0.018056793 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-08-17T17:14:31.261 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 podman[91258]: 2024-08-17 17:14:31.17836253 +0000 UTC m=+0.122168697 container create ae8f8b69fe2ffa9eabdcdb85da64ea9055fd204f3d4a8afabb23368189d35700 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:14:31.680 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 podman[91258]: 2024-08-17 17:14:31.397297421 +0000 UTC m=+0.341103592 container init ae8f8b69fe2ffa9eabdcdb85da64ea9055fd204f3d4a8afabb23368189d35700 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:14:31.680 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 podman[91258]: 2024-08-17 17:14:31.401863501 +0000 UTC m=+0.345669669 container start ae8f8b69fe2ffa9eabdcdb85da64ea9055fd204f3d4a8afabb23368189d35700 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:14:31.680 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T17:14:31.421Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-08-17T17:14:31.680 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T17:14:31.421Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-08-17T17:14:31.680 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T17:14:31.421Z caller=main.go:623 level=info host_details="(Linux 5.14.0-480.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024 x86_64 smithi165 (none))" 2024-08-17T17:14:31.680 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T17:14:31.421Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-08-17T17:14:31.680 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T17:14:31.421Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-08-17T17:14:31.680 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T17:14:31.423Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-08-17T17:14:31.680 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T17:14:31.423Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-08-17T17:14:31.680 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T17:14:31.425Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-08-17T17:14:31.680 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T17:14:31.425Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-08-17T17:14:31.681 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T17:14:31.426Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-08-17T17:14:31.681 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T17:14:31.426Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.68µs 2024-08-17T17:14:31.681 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T17:14:31.426Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-08-17T17:14:31.681 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T17:14:31.427Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=4 2024-08-17T17:14:31.681 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T17:14:31.434Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=4 2024-08-17T17:14:31.681 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T17:14:31.435Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=4 2024-08-17T17:14:31.681 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T17:14:31.436Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=4 2024-08-17T17:14:31.681 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T17:14:31.436Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=4 2024-08-17T17:14:31.681 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T17:14:31.436Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=23.954µs wal_replay_duration=9.423336ms wbl_replay_duration=113ns total_replay_duration=9.483497ms 2024-08-17T17:14:31.681 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T17:14:31.438Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-08-17T17:14:31.682 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T17:14:31.438Z caller=main.go:1153 level=info msg="TSDB started" 2024-08-17T17:14:31.682 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T17:14:31.438Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-08-17T17:14:31.682 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T17:14:31.453Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.349344ms db_storage=1.5µs remote_storage=1.048µs web_handler=330ns query_engine=663ns scrape=429.235µs scrape_sd=89.844µs notify=13.204µs notify_sd=14.339µs rules=14.46469ms tracing=6.03µs 2024-08-17T17:14:31.682 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T17:14:31.453Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-08-17T17:14:31.682 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T17:14:31.453Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-08-17T17:14:31.682 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 bash[91258]: ae8f8b69fe2ffa9eabdcdb85da64ea9055fd204f3d4a8afabb23368189d35700 2024-08-17T17:14:31.682 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 17:14:31 smithi165 systemd[1]: Started Ceph prometheus.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:14:31.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:31 smithi136 ceph-mon[42580]: pgmap v100: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:31.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:31 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:31.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:31 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:31.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:31 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:14:31.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:31 smithi136 ceph-mon[31034]: pgmap v100: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:31.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:31 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:31.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:31 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:31.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:31 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:14:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-mon[34868]: pgmap v100: 137 pgs: 137 active+clean; 781 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:31 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:14:32.530 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 8388609~8388609 2024-08-17T17:14:32.557 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 0~1000 2024-08-17T17:14:32.557 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 1010~1000 2024-08-17T17:14:32.558 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap2 2024-08-17T17:14:32.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:32 smithi136 ceph-mon[42580]: osdmap e256: 8 total, 8 up, 8 in 2024-08-17T17:14:32.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:32 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:32.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:32 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:32.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:32 smithi136 ceph-mon[42580]: osdmap e257: 8 total, 8 up, 8 in 2024-08-17T17:14:32.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:32 smithi136 ceph-mon[31034]: osdmap e256: 8 total, 8 up, 8 in 2024-08-17T17:14:32.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:32 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:32.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:32 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:32.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:32 smithi136 ceph-mon[31034]: osdmap e257: 8 total, 8 up, 8 in 2024-08-17T17:14:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:32 smithi165 ceph-mon[34868]: osdmap e256: 8 total, 8 up, 8 in 2024-08-17T17:14:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:32 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:32 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:32 smithi165 ceph-mon[34868]: osdmap e257: 8 total, 8 up, 8 in 2024-08-17T17:14:33.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:33 smithi136 ceph-mon[31034]: pgmap v103: 137 pgs: 137 active+clean; 829 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 25 MiB/s wr, 639 op/s 2024-08-17T17:14:33.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:33 smithi136 ceph-mon[31034]: osdmap e258: 8 total, 8 up, 8 in 2024-08-17T17:14:33.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:33 smithi136 ceph-mon[42580]: pgmap v103: 137 pgs: 137 active+clean; 829 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 25 MiB/s wr, 639 op/s 2024-08-17T17:14:33.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:33 smithi136 ceph-mon[42580]: osdmap e258: 8 total, 8 up, 8 in 2024-08-17T17:14:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:33 smithi165 ceph-mon[34868]: pgmap v103: 137 pgs: 137 active+clean; 829 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 25 MiB/s wr, 639 op/s 2024-08-17T17:14:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:33 smithi165 ceph-mon[34868]: osdmap e258: 8 total, 8 up, 8 in 2024-08-17T17:14:34.535 INFO:tasks.workunit.client.0.smithi136.stdout:write: X 8388609~1000 2024-08-17T17:14:34.544 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 16777228~1000 2024-08-17T17:14:34.544 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 167772160 2024-08-17T17:14:34.552 INFO:tasks.workunit.client.0.smithi136.stdout:write: 2 83886079~8388609 2024-08-17T17:14:34.578 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap3 2024-08-17T17:14:35.825 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:35 smithi165 ceph-mon[34868]: osdmap e259: 8 total, 8 up, 8 in 2024-08-17T17:14:35.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:35 smithi165 ceph-mon[34868]: pgmap v106: 137 pgs: 137 active+clean; 829 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 25 MiB/s wr, 639 op/s 2024-08-17T17:14:35.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:35 smithi136 ceph-mon[42580]: osdmap e259: 8 total, 8 up, 8 in 2024-08-17T17:14:35.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:35 smithi136 ceph-mon[42580]: pgmap v106: 137 pgs: 137 active+clean; 829 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 25 MiB/s wr, 639 op/s 2024-08-17T17:14:35.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:35 smithi136 ceph-mon[31034]: osdmap e259: 8 total, 8 up, 8 in 2024-08-17T17:14:35.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:35 smithi136 ceph-mon[31034]: pgmap v106: 137 pgs: 137 active+clean; 829 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 25 MiB/s wr, 639 op/s 2024-08-17T17:14:36.549 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:14:36.571 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 83886079~1 2024-08-17T17:14:36.579 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:14:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:36 smithi136 ceph-mon[31034]: osdmap e260: 8 total, 8 up, 8 in 2024-08-17T17:14:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:36 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:36 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:36.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:36 smithi136 ceph-mon[42580]: osdmap e260: 8 total, 8 up, 8 in 2024-08-17T17:14:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:36 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:36 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:36 smithi165 ceph-mon[34868]: osdmap e260: 8 total, 8 up, 8 in 2024-08-17T17:14:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:36 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:36 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:37 smithi165 ceph-mon[34868]: osdmap e261: 8 total, 8 up, 8 in 2024-08-17T17:14:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:37 smithi165 ceph-mon[34868]: pgmap v109: 137 pgs: 137 active+clean; 869 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-08-17T17:14:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:37 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:37 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:37.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:37 smithi136 ceph-mon[31034]: osdmap e261: 8 total, 8 up, 8 in 2024-08-17T17:14:37.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:37 smithi136 ceph-mon[31034]: pgmap v109: 137 pgs: 137 active+clean; 869 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-08-17T17:14:37.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:37 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:37.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:37 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:37.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:37 smithi136 ceph-mon[42580]: osdmap e261: 8 total, 8 up, 8 in 2024-08-17T17:14:37.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:37 smithi136 ceph-mon[42580]: pgmap v109: 137 pgs: 137 active+clean; 869 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-08-17T17:14:37.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:37 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:37.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:37 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:37.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:14:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:14:37] "GET /metrics HTTP/1.1" 200 38733 "" "Prometheus/2.51.0" 2024-08-17T17:14:38.202 INFO:teuthology.orchestra.run.smithi136.stdout:true 2024-08-17T17:14:38.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:38 smithi165 ceph-mon[34868]: osdmap e262: 8 total, 8 up, 8 in 2024-08-17T17:14:38.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:38 smithi165 ceph-mon[34868]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:38.756 INFO:teuthology.orchestra.run.smithi136.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-17T17:14:38.757 INFO:teuthology.orchestra.run.smithi136.stdout:alertmanager.a smithi136 *:9093,9094 running (4m) 28s ago 5m 26.2M - 0.25.0 c8568f914cd2 83b6bfc5306f 2024-08-17T17:14:38.757 INFO:teuthology.orchestra.run.smithi136.stdout:grafana.a smithi165 *:3000 running (5m) 2s ago 5m 90.0M - 9.4.7 954c08fa6188 be80a687c402 2024-08-17T17:14:38.757 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi136.ltsuqd smithi136 running (4m) 28s ago 4m 18.9M - 18.2.4-719-gced649f8 c870753d67c2 faaa06f2950c 2024-08-17T17:14:38.757 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi165.bmrbhh smithi165 running (4m) 2s ago 4m 16.2M - 18.2.4-719-gced649f8 c870753d67c2 c74b00d929cf 2024-08-17T17:14:38.757 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.x smithi165 *:8443,9283,8765 running (54s) 2s ago 9m 451M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 837fd48ff7cb 2024-08-17T17:14:38.757 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.y smithi136 *:8443,9283,8765 running (82s) 28s ago 11m 511M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9acdc6775e48 2024-08-17T17:14:38.757 INFO:teuthology.orchestra.run.smithi136.stdout:mon.a smithi136 running (11m) 28s ago 11m 57.2M 2048M 18.2.4-719-gced649f8 c870753d67c2 cc4e5e5ff756 2024-08-17T17:14:38.757 INFO:teuthology.orchestra.run.smithi136.stdout:mon.b smithi165 running (9m) 2s ago 9m 52.2M 2048M 18.2.4-719-gced649f8 c870753d67c2 d587e16140e1 2024-08-17T17:14:38.757 INFO:teuthology.orchestra.run.smithi136.stdout:mon.c smithi136 running (9m) 28s ago 9m 48.9M 2048M 18.2.4-719-gced649f8 c870753d67c2 b4e0f3c9e485 2024-08-17T17:14:38.757 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.a smithi136 *:9100 running (39s) 28s ago 5m 3384k - 1.7.0 72c9c2088986 1e4c644ae533 2024-08-17T17:14:38.758 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.b smithi165 *:9100 running (33s) 2s ago 5m 9.88M - 1.7.0 72c9c2088986 7af2d2d68b06 2024-08-17T17:14:38.758 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0 smithi136 running (9m) 28s ago 9m 111M 2480M 18.2.4-719-gced649f8 c870753d67c2 faab916734b0 2024-08-17T17:14:38.758 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1 smithi136 running (8m) 28s ago 8m 122M 2480M 18.2.4-719-gced649f8 c870753d67c2 861686b6ff82 2024-08-17T17:14:38.758 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2 smithi136 running (8m) 28s ago 8m 117M 2480M 18.2.4-719-gced649f8 c870753d67c2 640be4af4156 2024-08-17T17:14:38.758 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3 smithi136 running (8m) 28s ago 8m 105M 2480M 18.2.4-719-gced649f8 c870753d67c2 14b57a745d30 2024-08-17T17:14:38.758 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4 smithi165 running (7m) 2s ago 7m 118M 2480M 18.2.4-719-gced649f8 c870753d67c2 df366c73e1f2 2024-08-17T17:14:38.758 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5 smithi165 running (7m) 2s ago 7m 89.7M 2480M 18.2.4-719-gced649f8 c870753d67c2 1131665e203a 2024-08-17T17:14:38.758 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6 smithi165 running (6m) 2s ago 6m 128M 2480M 18.2.4-719-gced649f8 c870753d67c2 7bc5fd925293 2024-08-17T17:14:38.758 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7 smithi165 running (6m) 2s ago 6m 105M 2480M 18.2.4-719-gced649f8 c870753d67c2 18382e4ae3b8 2024-08-17T17:14:38.758 INFO:teuthology.orchestra.run.smithi136.stdout:prometheus.a smithi165 *:9095 running (7s) 2s ago 5m 31.0M - 2.51.0 1d3b7f56885b ae8f8b69fe2f 2024-08-17T17:14:38.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:38 smithi136 ceph-mon[42580]: osdmap e262: 8 total, 8 up, 8 in 2024-08-17T17:14:38.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:38 smithi136 ceph-mon[42580]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:38.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:38 smithi136 ceph-mon[31034]: osdmap e262: 8 total, 8 up, 8 in 2024-08-17T17:14:38.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:38 smithi136 ceph-mon[31034]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:39.090 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:14:39.090 INFO:teuthology.orchestra.run.smithi136.stdout: "mon": { 2024-08-17T17:14:39.090 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 3 2024-08-17T17:14:39.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:14:39.091 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr": { 2024-08-17T17:14:39.091 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 2 2024-08-17T17:14:39.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:14:39.091 INFO:teuthology.orchestra.run.smithi136.stdout: "osd": { 2024-08-17T17:14:39.091 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 8 2024-08-17T17:14:39.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:14:39.091 INFO:teuthology.orchestra.run.smithi136.stdout: "mds": { 2024-08-17T17:14:39.091 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 2 2024-08-17T17:14:39.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:14:39.091 INFO:teuthology.orchestra.run.smithi136.stdout: "overall": { 2024-08-17T17:14:39.091 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 13, 2024-08-17T17:14:39.091 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 2 2024-08-17T17:14:39.091 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:14:39.091 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:14:39.383 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:14:39.383 INFO:teuthology.orchestra.run.smithi136.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d", 2024-08-17T17:14:39.384 INFO:teuthology.orchestra.run.smithi136.stdout: "in_progress": true, 2024-08-17T17:14:39.384 INFO:teuthology.orchestra.run.smithi136.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-08-17T17:14:39.384 INFO:teuthology.orchestra.run.smithi136.stdout: "services_complete": [ 2024-08-17T17:14:39.384 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr" 2024-08-17T17:14:39.384 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-17T17:14:39.384 INFO:teuthology.orchestra.run.smithi136.stdout: "progress": "2/2 daemons upgraded", 2024-08-17T17:14:39.384 INFO:teuthology.orchestra.run.smithi136.stdout: "message": "Currently upgrading prometheus daemons", 2024-08-17T17:14:39.384 INFO:teuthology.orchestra.run.smithi136.stdout: "is_paused": false 2024-08-17T17:14:39.384 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:14:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:39 smithi136 ceph-mon[31034]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:39 smithi136 ceph-mon[31034]: osdmap e263: 8 total, 8 up, 8 in 2024-08-17T17:14:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:39 smithi136 ceph-mon[31034]: pgmap v112: 137 pgs: 137 active+clean; 869 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-08-17T17:14:39.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:39 smithi136 ceph-mon[31034]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:39.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:39 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/2416457814' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:39.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:39 smithi136 ceph-mon[31034]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:39 smithi136 ceph-mon[42580]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:39 smithi136 ceph-mon[42580]: osdmap e263: 8 total, 8 up, 8 in 2024-08-17T17:14:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:39 smithi136 ceph-mon[42580]: pgmap v112: 137 pgs: 137 active+clean; 869 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-08-17T17:14:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:39 smithi136 ceph-mon[42580]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:39 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/2416457814' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:39 smithi136 ceph-mon[42580]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:39.963 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 0~8388609 2024-08-17T17:14:39.964 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:39 smithi165 ceph-mon[34868]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:39.964 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:39 smithi165 ceph-mon[34868]: osdmap e263: 8 total, 8 up, 8 in 2024-08-17T17:14:39.964 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:39 smithi165 ceph-mon[34868]: pgmap v112: 137 pgs: 137 active+clean; 869 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-08-17T17:14:39.964 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:39 smithi165 ceph-mon[34868]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:39.964 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:39 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2416457814' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:39.964 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:39 smithi165 ceph-mon[34868]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:40.056 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 16777218~8388609 2024-08-17T17:14:40.876 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[31034]: osdmap e264: 8 total, 8 up, 8 in 2024-08-17T17:14:40.876 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:40.876 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:40.876 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:14:40.876 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:14:40.876 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:40.876 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:14:40.877 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[31034]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:14:40.877 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:14:40.877 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:40.877 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:40.877 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:40.877 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:40.877 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:40.877 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:40.877 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:40.877 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:40.877 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:40.877 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:40.877 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:40.877 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:14:40.878 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[42580]: osdmap e264: 8 total, 8 up, 8 in 2024-08-17T17:14:40.878 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:40.878 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:40.878 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:14:40.878 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:14:40.878 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:40.878 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:14:40.878 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[42580]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:14:40.878 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:14:40.878 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:40.878 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:40.879 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:40.879 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:40.879 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:40.879 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:40.879 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:40.879 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:40.879 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:40.879 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:40.879 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:40.879 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:40 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:14:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:40 smithi165 ceph-mon[34868]: osdmap e264: 8 total, 8 up, 8 in 2024-08-17T17:14:41.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:40 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:41.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:40 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:41.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:40 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:14:41.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:40 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:14:41.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:40 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:41.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:40 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:14:41.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:40 smithi165 ceph-mon[34868]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:14:41.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:40 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:14:41.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:40 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:41.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:40 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:41.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:40 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:40 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:40 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:40 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:40 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:40 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:40 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:40 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:40 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:14:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:40 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:14:41.047 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 0~8388609 2024-08-17T17:14:41.062 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 16777218~8388609 2024-08-17T17:14:41.142 INFO:tasks.workunit.client.0.smithi136.stdout:migration_abort 2024-08-17T17:14:42.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:42 smithi136 ceph-mon[31034]: pgmap v114: 137 pgs: 137 active+clean; 869 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:42.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:42 smithi136 ceph-mon[31034]: osdmap e265: 8 total, 8 up, 8 in 2024-08-17T17:14:42.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:42 smithi136 ceph-mon[42580]: pgmap v114: 137 pgs: 137 active+clean; 869 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:42.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:42 smithi136 ceph-mon[42580]: osdmap e265: 8 total, 8 up, 8 in 2024-08-17T17:14:42.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:42 smithi165 ceph-mon[34868]: pgmap v114: 137 pgs: 137 active+clean; 869 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:42.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:42 smithi165 ceph-mon[34868]: osdmap e265: 8 total, 8 up, 8 in 2024-08-17T17:14:43.144 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:43 smithi136 ceph-mon[31034]: Upgrade: Updating alertmanager.a 2024-08-17T17:14:43.144 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:43 smithi136 ceph-mon[31034]: Deploying daemon alertmanager.a on smithi136 2024-08-17T17:14:43.144 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:43 smithi136 ceph-mon[31034]: osdmap e266: 8 total, 8 up, 8 in 2024-08-17T17:14:43.144 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:43 smithi136 ceph-mon[31034]: osdmap e267: 8 total, 8 up, 8 in 2024-08-17T17:14:43.144 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:43 smithi136 ceph-mon[42580]: Upgrade: Updating alertmanager.a 2024-08-17T17:14:43.144 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:43 smithi136 ceph-mon[42580]: Deploying daemon alertmanager.a on smithi136 2024-08-17T17:14:43.145 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:43 smithi136 ceph-mon[42580]: osdmap e266: 8 total, 8 up, 8 in 2024-08-17T17:14:43.145 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:43 smithi136 ceph-mon[42580]: osdmap e267: 8 total, 8 up, 8 in 2024-08-17T17:14:43.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:43 smithi165 ceph-mon[34868]: Upgrade: Updating alertmanager.a 2024-08-17T17:14:43.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:43 smithi165 ceph-mon[34868]: Deploying daemon alertmanager.a on smithi136 2024-08-17T17:14:43.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:43 smithi165 ceph-mon[34868]: osdmap e266: 8 total, 8 up, 8 in 2024-08-17T17:14:43.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:43 smithi165 ceph-mon[34868]: osdmap e267: 8 total, 8 up, 8 in 2024-08-17T17:14:44.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:44 smithi136 ceph-mon[31034]: pgmap v118: 137 pgs: 137 active+clean; 913 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 35 MiB/s rd, 11 MiB/s wr, 336 op/s 2024-08-17T17:14:44.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:44 smithi136 ceph-mon[31034]: osdmap e268: 8 total, 8 up, 8 in 2024-08-17T17:14:44.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:44 smithi136 ceph-mon[42580]: pgmap v118: 137 pgs: 137 active+clean; 913 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 35 MiB/s rd, 11 MiB/s wr, 336 op/s 2024-08-17T17:14:44.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:44 smithi136 ceph-mon[42580]: osdmap e268: 8 total, 8 up, 8 in 2024-08-17T17:14:44.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:44 smithi165 ceph-mon[34868]: pgmap v118: 137 pgs: 137 active+clean; 913 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 35 MiB/s rd, 11 MiB/s wr, 336 op/s 2024-08-17T17:14:44.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:44 smithi165 ceph-mon[34868]: osdmap e268: 8 total, 8 up, 8 in 2024-08-17T17:14:44.854 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.AbortWithSnapshots (14231 ms) 2024-08-17T17:14:44.855 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.CloneV1Parent 2024-08-17T17:14:44.884 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:14:44.894 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 0~10 2024-08-17T17:14:44.903 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:14:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:45 smithi136 ceph-mon[31034]: osdmap e269: 8 total, 8 up, 8 in 2024-08-17T17:14:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:45 smithi136 ceph-mon[31034]: pgmap v121: 137 pgs: 137 active+clean; 913 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:45.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:45 smithi136 ceph-mon[42580]: osdmap e269: 8 total, 8 up, 8 in 2024-08-17T17:14:45.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:45 smithi136 ceph-mon[42580]: pgmap v121: 137 pgs: 137 active+clean; 913 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:46.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:45 smithi165 ceph-mon[34868]: osdmap e269: 8 total, 8 up, 8 in 2024-08-17T17:14:46.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:45 smithi165 ceph-mon[34868]: pgmap v121: 137 pgs: 137 active+clean; 913 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:46.544 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap1 2024-08-17T17:14:46.604 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:14:46.856 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:46 smithi136 ceph-mon[31034]: osdmap e270: 8 total, 8 up, 8 in 2024-08-17T17:14:46.856 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:46 smithi136 ceph-mon[42580]: osdmap e270: 8 total, 8 up, 8 in 2024-08-17T17:14:47.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:46 smithi165 ceph-mon[34868]: osdmap e270: 8 total, 8 up, 8 in 2024-08-17T17:14:47.750 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:14:47.829 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:14:47.873 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:47 smithi136 ceph-mon[31034]: osdmap e271: 8 total, 8 up, 8 in 2024-08-17T17:14:47.873 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:47 smithi136 ceph-mon[31034]: pgmap v124: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 43 MiB/s rd, 12 MiB/s wr, 258 op/s 2024-08-17T17:14:47.873 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:47 smithi136 ceph-mon[42580]: osdmap e271: 8 total, 8 up, 8 in 2024-08-17T17:14:47.874 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:47 smithi136 ceph-mon[42580]: pgmap v124: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 43 MiB/s rd, 12 MiB/s wr, 258 op/s 2024-08-17T17:14:47.874 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:14:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:14:47] "GET /metrics HTTP/1.1" 200 38735 "" "Prometheus/2.51.0" 2024-08-17T17:14:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:47 smithi165 ceph-mon[34868]: osdmap e271: 8 total, 8 up, 8 in 2024-08-17T17:14:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:47 smithi165 ceph-mon[34868]: pgmap v124: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 43 MiB/s rd, 12 MiB/s wr, 258 op/s 2024-08-17T17:14:48.660 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.CloneV1Parent (3806 ms) 2024-08-17T17:14:48.660 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.CloneV2Parent 2024-08-17T17:14:48.695 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:14:48.708 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 0~10 2024-08-17T17:14:48.719 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:14:48.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:48 smithi136 ceph-mon[31034]: osdmap e272: 8 total, 8 up, 8 in 2024-08-17T17:14:48.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:48 smithi136 ceph-mon[42580]: osdmap e272: 8 total, 8 up, 8 in 2024-08-17T17:14:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:48 smithi165 ceph-mon[34868]: osdmap e272: 8 total, 8 up, 8 in 2024-08-17T17:14:49.561 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:49 smithi136 ceph-mon[31034]: osdmap e273: 8 total, 8 up, 8 in 2024-08-17T17:14:49.561 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:49 smithi136 ceph-mon[31034]: pgmap v127: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 43 MiB/s rd, 12 MiB/s wr, 258 op/s 2024-08-17T17:14:49.561 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:49 smithi136 ceph-mon[42580]: osdmap e273: 8 total, 8 up, 8 in 2024-08-17T17:14:49.562 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:49 smithi136 ceph-mon[42580]: pgmap v127: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 43 MiB/s rd, 12 MiB/s wr, 258 op/s 2024-08-17T17:14:49.890 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:49 smithi136 systemd[1]: Stopping Ceph alertmanager.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:14:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:49 smithi165 ceph-mon[34868]: osdmap e273: 8 total, 8 up, 8 in 2024-08-17T17:14:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:49 smithi165 ceph-mon[34868]: pgmap v127: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 43 MiB/s rd, 12 MiB/s wr, 258 op/s 2024-08-17T17:14:50.190 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:49 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[78967]: ts=2024-08-17T17:14:49.889Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-08-17T17:14:50.190 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:49 smithi136 podman[104572]: 2024-08-17 17:14:49.930325084 +0000 UTC m=+0.137753518 container died 83b6bfc5306f207fbd8b48979a76ea59898ceca894fa5d772183019c8dcc78e1 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:14:50.446 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:50 smithi136 podman[104572]: 2024-08-17 17:14:50.371009456 +0000 UTC m=+0.578437890 container cleanup 83b6bfc5306f207fbd8b48979a76ea59898ceca894fa5d772183019c8dcc78e1 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:14:50.446 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:50 smithi136 bash[104572]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a 2024-08-17T17:14:50.577 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap1 2024-08-17T17:14:50.637 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:14:50.911 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:50 smithi165 ceph-mon[34868]: osdmap e274: 8 total, 8 up, 8 in 2024-08-17T17:14:50.930 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:50 smithi136 ceph-mon[31034]: osdmap e274: 8 total, 8 up, 8 in 2024-08-17T17:14:50.930 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:50 smithi136 ceph-mon[42580]: osdmap e274: 8 total, 8 up, 8 in 2024-08-17T17:14:50.930 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:50 smithi136 podman[104588]: 2024-08-17 17:14:50.589142535 +0000 UTC m=+0.655603463 container remove 83b6bfc5306f207fbd8b48979a76ea59898ceca894fa5d772183019c8dcc78e1 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:14:50.931 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:50 smithi136 podman[104588]: 2024-08-17 17:14:50.665236422 +0000 UTC m=+0.731697353 volume remove 1ba1e01fddfd2399db117295d601f34cea22623bec0375b5c15526a93b3cb8ab 2024-08-17T17:14:51.196 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:50 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-08-17T17:14:51.197 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:50 smithi136 systemd[1]: Stopped Ceph alertmanager.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:14:51.197 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:50 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@alertmanager.a.service: Consumed 1.334s CPU time. 2024-08-17T17:14:51.197 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:51 smithi136 systemd[1]: Starting Ceph alertmanager.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:14:51.696 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:51 smithi136 podman[104724]: 2024-08-17 17:14:51.25891491 +0000 UTC m=+0.016963638 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-08-17T17:14:51.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:51 smithi136 ceph-mon[31034]: osdmap e275: 8 total, 8 up, 8 in 2024-08-17T17:14:51.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:51 smithi136 ceph-mon[31034]: pgmap v130: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-08-17T17:14:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:51 smithi136 ceph-mon[42580]: osdmap e275: 8 total, 8 up, 8 in 2024-08-17T17:14:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:51 smithi136 ceph-mon[42580]: pgmap v130: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-08-17T17:14:51.757 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:14:51.834 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:14:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:51 smithi165 ceph-mon[34868]: osdmap e275: 8 total, 8 up, 8 in 2024-08-17T17:14:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:51 smithi165 ceph-mon[34868]: pgmap v130: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-08-17T17:14:52.066 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:51 smithi136 podman[104724]: 2024-08-17 17:14:51.742351706 +0000 UTC m=+0.500400422 volume create cad8abfd9b1edfc26c36e185f8586f63b536d79159c1a5deb73503a3ab385a9c 2024-08-17T17:14:52.067 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:51 smithi136 podman[104724]: 2024-08-17 17:14:51.85128626 +0000 UTC m=+0.609334975 container create c3b9b5b223ab70bb8cc2b48b8b4f8ae2dbd700fb9c45af4178846d2c47fc9ec2 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:14:52.446 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:52 smithi136 podman[104724]: 2024-08-17 17:14:52.162458325 +0000 UTC m=+0.920507041 container init c3b9b5b223ab70bb8cc2b48b8b4f8ae2dbd700fb9c45af4178846d2c47fc9ec2 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:14:52.447 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:52 smithi136 podman[104724]: 2024-08-17 17:14:52.166916409 +0000 UTC m=+0.924965135 container start c3b9b5b223ab70bb8cc2b48b8b4f8ae2dbd700fb9c45af4178846d2c47fc9ec2 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:14:52.447 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[104739]: ts=2024-08-17T17:14:52.177Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-08-17T17:14:52.447 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[104739]: ts=2024-08-17T17:14:52.177Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-08-17T17:14:52.447 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[104739]: ts=2024-08-17T17:14:52.178Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.136 port=9094 2024-08-17T17:14:52.447 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[104739]: ts=2024-08-17T17:14:52.179Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-08-17T17:14:52.447 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[104739]: ts=2024-08-17T17:14:52.200Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-08-17T17:14:52.447 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[104739]: ts=2024-08-17T17:14:52.200Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-08-17T17:14:52.447 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[104739]: ts=2024-08-17T17:14:52.202Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-08-17T17:14:52.447 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[104739]: ts=2024-08-17T17:14:52.202Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-08-17T17:14:52.447 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:52 smithi136 bash[104724]: c3b9b5b223ab70bb8cc2b48b8b4f8ae2dbd700fb9c45af4178846d2c47fc9ec2 2024-08-17T17:14:52.447 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:52 smithi136 systemd[1]: Started Ceph alertmanager.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:14:52.637 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.CloneV2Parent (3977 ms) 2024-08-17T17:14:52.637 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.CloneV1ParentAbort 2024-08-17T17:14:52.672 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:14:52.683 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 0~10 2024-08-17T17:14:52.692 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:14:52.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:52 smithi136 ceph-mon[31034]: osdmap e276: 8 total, 8 up, 8 in 2024-08-17T17:14:52.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:52 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:52 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:52 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:14:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:52 smithi136 ceph-mon[31034]: osdmap e277: 8 total, 8 up, 8 in 2024-08-17T17:14:52.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:52 smithi136 ceph-mon[42580]: osdmap e276: 8 total, 8 up, 8 in 2024-08-17T17:14:52.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:52 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:52.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:52 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:52.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:52 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:14:52.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:52 smithi136 ceph-mon[42580]: osdmap e277: 8 total, 8 up, 8 in 2024-08-17T17:14:52.973 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:52 smithi165 ceph-mon[34868]: osdmap e276: 8 total, 8 up, 8 in 2024-08-17T17:14:52.974 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:52 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:52.974 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:52 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:52.974 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:52 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:14:52.974 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:52 smithi165 ceph-mon[34868]: osdmap e277: 8 total, 8 up, 8 in 2024-08-17T17:14:53.869 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:53 smithi136 ceph-mon[31034]: pgmap v133: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 421 KiB/s rd, 25 KiB/s wr, 533 op/s 2024-08-17T17:14:53.870 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:53 smithi136 ceph-mon[42580]: pgmap v133: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 421 KiB/s rd, 25 KiB/s wr, 533 op/s 2024-08-17T17:14:53.895 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:53 smithi165 ceph-mon[34868]: pgmap v133: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 421 KiB/s rd, 25 KiB/s wr, 533 op/s 2024-08-17T17:14:54.446 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:14:54 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[104739]: ts=2024-08-17T17:14:54.179Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000062089s 2024-08-17T17:14:54.599 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap1 2024-08-17T17:14:54.654 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:14:54.840 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:54 smithi165 ceph-mon[34868]: osdmap e278: 8 total, 8 up, 8 in 2024-08-17T17:14:54.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:54 smithi136 ceph-mon[31034]: osdmap e278: 8 total, 8 up, 8 in 2024-08-17T17:14:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:54 smithi136 ceph-mon[42580]: osdmap e278: 8 total, 8 up, 8 in 2024-08-17T17:14:55.735 INFO:tasks.workunit.client.0.smithi136.stdout:migration_abort 2024-08-17T17:14:55.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:55 smithi136 ceph-mon[42580]: osdmap e279: 8 total, 8 up, 8 in 2024-08-17T17:14:55.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:55 smithi136 ceph-mon[42580]: pgmap v136: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 421 KiB/s rd, 25 KiB/s wr, 533 op/s 2024-08-17T17:14:55.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:55 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:14:55.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:55 smithi136 ceph-mon[31034]: osdmap e279: 8 total, 8 up, 8 in 2024-08-17T17:14:55.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:55 smithi136 ceph-mon[31034]: pgmap v136: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 421 KiB/s rd, 25 KiB/s wr, 533 op/s 2024-08-17T17:14:55.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:55 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:14:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:55 smithi165 ceph-mon[34868]: osdmap e279: 8 total, 8 up, 8 in 2024-08-17T17:14:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:55 smithi165 ceph-mon[34868]: pgmap v136: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 421 KiB/s rd, 25 KiB/s wr, 533 op/s 2024-08-17T17:14:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:55 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:14:56.719 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.CloneV1ParentAbort (4082 ms) 2024-08-17T17:14:56.719 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.CloneV2ParentAbort 2024-08-17T17:14:56.751 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:14:56.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:56 smithi165 ceph-mon[34868]: osdmap e280: 8 total, 8 up, 8 in 2024-08-17T17:14:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:56 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:56 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:56 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:56 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:56.761 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 0~10 2024-08-17T17:14:56.770 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:14:56.901 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:56 smithi136 ceph-mon[31034]: osdmap e280: 8 total, 8 up, 8 in 2024-08-17T17:14:56.901 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:56 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:56.901 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:56 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:56.902 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:56 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:56.902 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:56 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:56.902 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:56 smithi136 ceph-mon[42580]: osdmap e280: 8 total, 8 up, 8 in 2024-08-17T17:14:56.902 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:56 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:56.902 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:56 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:56.902 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:56 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:56.902 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:56 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:57.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:14:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:14:57] "GET /metrics HTTP/1.1" 200 38727 "" "Prometheus/2.51.0" 2024-08-17T17:14:57.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:57 smithi136 ceph-mon[42580]: osdmap e281: 8 total, 8 up, 8 in 2024-08-17T17:14:57.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:57 smithi136 ceph-mon[42580]: pgmap v139: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 451 KiB/s rd, 25 KiB/s wr, 562 op/s 2024-08-17T17:14:57.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:57 smithi136 ceph-mon[42580]: mgrmap e33: y(active, since 92s), standbys: x 2024-08-17T17:14:57.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:57 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:57.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:57 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:57.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:57 smithi136 ceph-mon[31034]: osdmap e281: 8 total, 8 up, 8 in 2024-08-17T17:14:57.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:57 smithi136 ceph-mon[31034]: pgmap v139: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 451 KiB/s rd, 25 KiB/s wr, 562 op/s 2024-08-17T17:14:57.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:57 smithi136 ceph-mon[31034]: mgrmap e33: y(active, since 92s), standbys: x 2024-08-17T17:14:57.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:57 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:57.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:57 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:57 smithi165 ceph-mon[34868]: osdmap e281: 8 total, 8 up, 8 in 2024-08-17T17:14:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:57 smithi165 ceph-mon[34868]: pgmap v139: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 451 KiB/s rd, 25 KiB/s wr, 562 op/s 2024-08-17T17:14:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:57 smithi165 ceph-mon[34868]: mgrmap e33: y(active, since 92s), standbys: x 2024-08-17T17:14:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:57 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:57 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:14:58.619 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap1 2024-08-17T17:14:58.678 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:14:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:58 smithi136 ceph-mon[31034]: osdmap e282: 8 total, 8 up, 8 in 2024-08-17T17:14:58.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:58 smithi136 ceph-mon[42580]: osdmap e282: 8 total, 8 up, 8 in 2024-08-17T17:14:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:58 smithi165 ceph-mon[34868]: osdmap e282: 8 total, 8 up, 8 in 2024-08-17T17:14:59.765 INFO:tasks.workunit.client.0.smithi136.stdout:migration_abort 2024-08-17T17:14:59.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:59 smithi136 ceph-mon[31034]: osdmap e283: 8 total, 8 up, 8 in 2024-08-17T17:14:59.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:14:59 smithi136 ceph-mon[31034]: pgmap v142: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 451 KiB/s rd, 25 KiB/s wr, 562 op/s 2024-08-17T17:14:59.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:59 smithi136 ceph-mon[42580]: osdmap e283: 8 total, 8 up, 8 in 2024-08-17T17:14:59.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:14:59 smithi136 ceph-mon[42580]: pgmap v142: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 451 KiB/s rd, 25 KiB/s wr, 562 op/s 2024-08-17T17:15:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:59 smithi165 ceph-mon[34868]: osdmap e283: 8 total, 8 up, 8 in 2024-08-17T17:15:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:14:59 smithi165 ceph-mon[34868]: pgmap v142: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 451 KiB/s rd, 25 KiB/s wr, 562 op/s 2024-08-17T17:15:00.739 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.CloneV2ParentAbort (4020 ms) 2024-08-17T17:15:00.739 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.CloneV1ParentAbortFixIncompleteChildReattach 2024-08-17T17:15:00.771 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:15:00.780 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 0~10 2024-08-17T17:15:00.788 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:15:00.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[42580]: osdmap e284: 8 total, 8 up, 8 in 2024-08-17T17:15:00.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:15:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:15:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-08-17T17:15:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[42580]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-08-17T17:15:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:15:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:00.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:00.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:00.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:00.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:00.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:00.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:00.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[31034]: osdmap e284: 8 total, 8 up, 8 in 2024-08-17T17:15:00.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:00.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:00.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:15:00.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:15:00.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:00.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-08-17T17:15:00.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[31034]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-08-17T17:15:00.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:15:00.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:00.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:00.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:00.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:00.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:00.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:00.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:00.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:00.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:00.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:00.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:00.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:00 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:00 smithi165 ceph-mon[34868]: osdmap e284: 8 total, 8 up, 8 in 2024-08-17T17:15:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:00 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:00 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:01.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:00 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:15:01.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:00 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:15:01.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:00 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:01.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:00 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-08-17T17:15:01.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:00 smithi165 ceph-mon[34868]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-08-17T17:15:01.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:00 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:15:01.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:00 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:01.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:00 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:01.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:00 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:01.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:00 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:01.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:00 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:01.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:00 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:01.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:00 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:00 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:00 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:00 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:00 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:00 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:01 smithi165 ceph-mon[34868]: pgmap v144: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:01 smithi165 ceph-mon[34868]: osdmap e285: 8 total, 8 up, 8 in 2024-08-17T17:15:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:01 smithi165 ceph-mon[34868]: Upgrade: Updating grafana.a 2024-08-17T17:15:01.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:01 smithi136 ceph-mon[31034]: pgmap v144: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:01.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:01 smithi136 ceph-mon[31034]: osdmap e285: 8 total, 8 up, 8 in 2024-08-17T17:15:01.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:01 smithi136 ceph-mon[31034]: Upgrade: Updating grafana.a 2024-08-17T17:15:01.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:01 smithi136 ceph-mon[42580]: pgmap v144: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:01.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:01 smithi136 ceph-mon[42580]: osdmap e285: 8 total, 8 up, 8 in 2024-08-17T17:15:01.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:01 smithi136 ceph-mon[42580]: Upgrade: Updating grafana.a 2024-08-17T17:15:02.446 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 17:15:02 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[104739]: ts=2024-08-17T17:15:02.181Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001494913s 2024-08-17T17:15:02.531 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap1 2024-08-17T17:15:02.591 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:15:02.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:02 smithi136 ceph-mon[31034]: Deploying daemon grafana.a on smithi165 2024-08-17T17:15:02.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:02 smithi136 ceph-mon[31034]: osdmap e286: 8 total, 8 up, 8 in 2024-08-17T17:15:02.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:02 smithi136 ceph-mon[31034]: osdmap e287: 8 total, 8 up, 8 in 2024-08-17T17:15:02.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:02 smithi136 ceph-mon[42580]: Deploying daemon grafana.a on smithi165 2024-08-17T17:15:02.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:02 smithi136 ceph-mon[42580]: osdmap e286: 8 total, 8 up, 8 in 2024-08-17T17:15:02.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:02 smithi136 ceph-mon[42580]: osdmap e287: 8 total, 8 up, 8 in 2024-08-17T17:15:03.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:02 smithi165 ceph-mon[34868]: Deploying daemon grafana.a on smithi165 2024-08-17T17:15:03.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:02 smithi165 ceph-mon[34868]: osdmap e286: 8 total, 8 up, 8 in 2024-08-17T17:15:03.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:02 smithi165 ceph-mon[34868]: osdmap e287: 8 total, 8 up, 8 in 2024-08-17T17:15:03.810 INFO:tasks.workunit.client.0.smithi136.stdout:migration_abort 2024-08-17T17:15:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:03 smithi136 ceph-mon[31034]: pgmap v148: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 290 KiB/s rd, 27 KiB/s wr, 389 op/s 2024-08-17T17:15:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:03 smithi136 ceph-mon[42580]: pgmap v148: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 290 KiB/s rd, 27 KiB/s wr, 389 op/s 2024-08-17T17:15:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:03 smithi165 ceph-mon[34868]: pgmap v148: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 290 KiB/s rd, 27 KiB/s wr, 389 op/s 2024-08-17T17:15:04.769 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.CloneV1ParentAbortFixIncompleteChildReattach (4030 ms) 2024-08-17T17:15:04.770 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.CloneV1ParentAbortFixParentReattach 2024-08-17T17:15:04.800 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:15:04.810 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 0~10 2024-08-17T17:15:04.817 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:15:04.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:04 smithi136 ceph-mon[31034]: osdmap e288: 8 total, 8 up, 8 in 2024-08-17T17:15:04.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:04 smithi136 ceph-mon[42580]: osdmap e288: 8 total, 8 up, 8 in 2024-08-17T17:15:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:04 smithi165 ceph-mon[34868]: osdmap e288: 8 total, 8 up, 8 in 2024-08-17T17:15:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:05 smithi136 ceph-mon[31034]: pgmap v150: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 290 KiB/s rd, 27 KiB/s wr, 389 op/s 2024-08-17T17:15:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:05 smithi136 ceph-mon[31034]: osdmap e289: 8 total, 8 up, 8 in 2024-08-17T17:15:05.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:05 smithi136 ceph-mon[42580]: pgmap v150: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 290 KiB/s rd, 27 KiB/s wr, 389 op/s 2024-08-17T17:15:05.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:05 smithi136 ceph-mon[42580]: osdmap e289: 8 total, 8 up, 8 in 2024-08-17T17:15:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:05 smithi165 ceph-mon[34868]: pgmap v150: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 290 KiB/s rd, 27 KiB/s wr, 389 op/s 2024-08-17T17:15:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:05 smithi165 ceph-mon[34868]: osdmap e289: 8 total, 8 up, 8 in 2024-08-17T17:15:06.692 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap1 2024-08-17T17:15:06.752 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:15:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:06 smithi136 ceph-mon[31034]: osdmap e290: 8 total, 8 up, 8 in 2024-08-17T17:15:06.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:06 smithi136 ceph-mon[42580]: osdmap e290: 8 total, 8 up, 8 in 2024-08-17T17:15:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:06 smithi165 ceph-mon[34868]: osdmap e290: 8 total, 8 up, 8 in 2024-08-17T17:15:07.703 INFO:tasks.workunit.client.0.smithi136.stdout:migration_abort 2024-08-17T17:15:07.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:07 smithi136 ceph-mon[31034]: pgmap v153: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 282 KiB/s rd, 17 KiB/s wr, 359 op/s 2024-08-17T17:15:07.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:07 smithi136 ceph-mon[31034]: osdmap e291: 8 total, 8 up, 8 in 2024-08-17T17:15:07.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:07 smithi136 ceph-mon[31034]: osdmap e292: 8 total, 8 up, 8 in 2024-08-17T17:15:07.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:15:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:15:07] "GET /metrics HTTP/1.1" 200 38727 "" "Prometheus/2.51.0" 2024-08-17T17:15:07.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:07 smithi136 ceph-mon[42580]: pgmap v153: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 282 KiB/s rd, 17 KiB/s wr, 359 op/s 2024-08-17T17:15:07.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:07 smithi136 ceph-mon[42580]: osdmap e291: 8 total, 8 up, 8 in 2024-08-17T17:15:07.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:07 smithi136 ceph-mon[42580]: osdmap e292: 8 total, 8 up, 8 in 2024-08-17T17:15:08.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:07 smithi165 ceph-mon[34868]: pgmap v153: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 282 KiB/s rd, 17 KiB/s wr, 359 op/s 2024-08-17T17:15:08.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:07 smithi165 ceph-mon[34868]: osdmap e291: 8 total, 8 up, 8 in 2024-08-17T17:15:08.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:07 smithi165 ceph-mon[34868]: osdmap e292: 8 total, 8 up, 8 in 2024-08-17T17:15:08.779 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.CloneV1ParentAbortFixParentReattach (4010 ms) 2024-08-17T17:15:08.779 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.CloneV1ParentAbortRelinkNotNeeded 2024-08-17T17:15:08.809 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:15:08.819 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 0~10 2024-08-17T17:15:08.827 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:15:09.692 INFO:teuthology.orchestra.run.smithi136.stdout:true 2024-08-17T17:15:09.934 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:09 smithi165 ceph-mon[34868]: pgmap v156: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 288 KiB/s rd, 18 KiB/s wr, 367 op/s 2024-08-17T17:15:09.935 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:09 smithi165 ceph-mon[34868]: osdmap e293: 8 total, 8 up, 8 in 2024-08-17T17:15:09.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:09 smithi136 ceph-mon[42580]: pgmap v156: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 288 KiB/s rd, 18 KiB/s wr, 367 op/s 2024-08-17T17:15:09.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:09 smithi136 ceph-mon[42580]: osdmap e293: 8 total, 8 up, 8 in 2024-08-17T17:15:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:09 smithi136 ceph-mon[31034]: pgmap v156: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 288 KiB/s rd, 18 KiB/s wr, 367 op/s 2024-08-17T17:15:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:09 smithi136 ceph-mon[31034]: osdmap e293: 8 total, 8 up, 8 in 2024-08-17T17:15:10.245 INFO:teuthology.orchestra.run.smithi136.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-17T17:15:10.245 INFO:teuthology.orchestra.run.smithi136.stdout:alertmanager.a smithi136 *:9093,9094 running (18s) 14s ago 6m 17.5M - 0.27.0 11f11916f8cd c3b9b5b223ab 2024-08-17T17:15:10.246 INFO:teuthology.orchestra.run.smithi136.stdout:grafana.a smithi165 *:3000 running (5m) 13s ago 5m 90.6M - 9.4.7 954c08fa6188 be80a687c402 2024-08-17T17:15:10.246 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi136.ltsuqd smithi136 running (4m) 14s ago 4m 19.4M - 18.2.4-719-gced649f8 c870753d67c2 faaa06f2950c 2024-08-17T17:15:10.246 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi165.bmrbhh smithi165 running (4m) 13s ago 4m 16.3M - 18.2.4-719-gced649f8 c870753d67c2 c74b00d929cf 2024-08-17T17:15:10.246 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.x smithi165 *:8443,9283,8765 running (85s) 13s ago 10m 451M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 837fd48ff7cb 2024-08-17T17:15:10.246 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.y smithi136 *:8443,9283,8765 running (114s) 14s ago 11m 514M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9acdc6775e48 2024-08-17T17:15:10.246 INFO:teuthology.orchestra.run.smithi136.stdout:mon.a smithi136 running (11m) 14s ago 11m 59.8M 2048M 18.2.4-719-gced649f8 c870753d67c2 cc4e5e5ff756 2024-08-17T17:15:10.246 INFO:teuthology.orchestra.run.smithi136.stdout:mon.b smithi165 running (10m) 13s ago 10m 53.8M 2048M 18.2.4-719-gced649f8 c870753d67c2 d587e16140e1 2024-08-17T17:15:10.246 INFO:teuthology.orchestra.run.smithi136.stdout:mon.c smithi136 running (10m) 14s ago 10m 52.6M 2048M 18.2.4-719-gced649f8 c870753d67c2 b4e0f3c9e485 2024-08-17T17:15:10.246 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.a smithi136 *:9100 running (70s) 14s ago 5m 9625k - 1.7.0 72c9c2088986 1e4c644ae533 2024-08-17T17:15:10.246 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.b smithi165 *:9100 running (64s) 13s ago 5m 11.2M - 1.7.0 72c9c2088986 7af2d2d68b06 2024-08-17T17:15:10.246 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0 smithi136 running (9m) 14s ago 9m 127M 2480M 18.2.4-719-gced649f8 c870753d67c2 faab916734b0 2024-08-17T17:15:10.246 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1 smithi136 running (9m) 14s ago 9m 110M 2480M 18.2.4-719-gced649f8 c870753d67c2 861686b6ff82 2024-08-17T17:15:10.247 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2 smithi136 running (8m) 14s ago 8m 111M 2480M 18.2.4-719-gced649f8 c870753d67c2 640be4af4156 2024-08-17T17:15:10.247 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3 smithi136 running (8m) 14s ago 8m 105M 2480M 18.2.4-719-gced649f8 c870753d67c2 14b57a745d30 2024-08-17T17:15:10.247 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4 smithi165 running (8m) 13s ago 8m 126M 2480M 18.2.4-719-gced649f8 c870753d67c2 df366c73e1f2 2024-08-17T17:15:10.247 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5 smithi165 running (7m) 13s ago 7m 91.6M 2480M 18.2.4-719-gced649f8 c870753d67c2 1131665e203a 2024-08-17T17:15:10.247 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6 smithi165 running (7m) 13s ago 7m 131M 2480M 18.2.4-719-gced649f8 c870753d67c2 7bc5fd925293 2024-08-17T17:15:10.247 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7 smithi165 running (6m) 13s ago 6m 102M 2480M 18.2.4-719-gced649f8 c870753d67c2 18382e4ae3b8 2024-08-17T17:15:10.247 INFO:teuthology.orchestra.run.smithi136.stdout:prometheus.a smithi165 *:9095 running (39s) 13s ago 6m 35.4M - 2.51.0 1d3b7f56885b ae8f8b69fe2f 2024-08-17T17:15:10.577 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:15:10.577 INFO:teuthology.orchestra.run.smithi136.stdout: "mon": { 2024-08-17T17:15:10.578 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 3 2024-08-17T17:15:10.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:15:10.578 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr": { 2024-08-17T17:15:10.578 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 2 2024-08-17T17:15:10.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:15:10.578 INFO:teuthology.orchestra.run.smithi136.stdout: "osd": { 2024-08-17T17:15:10.578 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 8 2024-08-17T17:15:10.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:15:10.578 INFO:teuthology.orchestra.run.smithi136.stdout: "mds": { 2024-08-17T17:15:10.578 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 2 2024-08-17T17:15:10.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:15:10.578 INFO:teuthology.orchestra.run.smithi136.stdout: "overall": { 2024-08-17T17:15:10.578 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 13, 2024-08-17T17:15:10.579 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 2 2024-08-17T17:15:10.579 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:15:10.579 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:15:10.693 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap1 2024-08-17T17:15:10.749 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:15:10.884 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:15:10.884 INFO:teuthology.orchestra.run.smithi136.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d", 2024-08-17T17:15:10.884 INFO:teuthology.orchestra.run.smithi136.stdout: "in_progress": true, 2024-08-17T17:15:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-08-17T17:15:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: "services_complete": [ 2024-08-17T17:15:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr" 2024-08-17T17:15:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-17T17:15:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: "progress": "2/2 daemons upgraded", 2024-08-17T17:15:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: "message": "Currently upgrading grafana daemons", 2024-08-17T17:15:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: "is_paused": false 2024-08-17T17:15:10.885 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:15:10.885 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:10 smithi136 ceph-mon[31034]: osdmap e294: 8 total, 8 up, 8 in 2024-08-17T17:15:10.886 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:10 smithi136 ceph-mon[31034]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:10.886 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:10 smithi136 ceph-mon[31034]: from='client.25091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:10.886 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:10 smithi136 ceph-mon[31034]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:10.886 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:10 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:15:10.886 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:10 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/3683331446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:10.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:10 smithi136 ceph-mon[42580]: osdmap e294: 8 total, 8 up, 8 in 2024-08-17T17:15:10.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:10 smithi136 ceph-mon[42580]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:10.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:10 smithi136 ceph-mon[42580]: from='client.25091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:10.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:10 smithi136 ceph-mon[42580]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:10.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:10 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:15:10.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:10 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/3683331446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:10.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:10 smithi165 ceph-mon[34868]: osdmap e294: 8 total, 8 up, 8 in 2024-08-17T17:15:10.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:10 smithi165 ceph-mon[34868]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:10.955 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:10 smithi165 ceph-mon[34868]: from='client.25091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:10.955 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:10 smithi165 ceph-mon[34868]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:10.955 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:10 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:15:10.955 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:10 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/3683331446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:11.852 INFO:tasks.workunit.client.0.smithi136.stdout:migration_abort 2024-08-17T17:15:11.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:11 smithi136 ceph-mon[31034]: pgmap v159: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:11.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:11 smithi136 ceph-mon[31034]: osdmap e295: 8 total, 8 up, 8 in 2024-08-17T17:15:11.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:11 smithi136 ceph-mon[31034]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:11 smithi136 ceph-mon[42580]: pgmap v159: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:11 smithi136 ceph-mon[42580]: osdmap e295: 8 total, 8 up, 8 in 2024-08-17T17:15:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:11 smithi136 ceph-mon[42580]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:11 smithi165 ceph-mon[34868]: pgmap v159: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:11 smithi165 ceph-mon[34868]: osdmap e295: 8 total, 8 up, 8 in 2024-08-17T17:15:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:11 smithi165 ceph-mon[34868]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:12.806 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.CloneV1ParentAbortRelinkNotNeeded (4027 ms) 2024-08-17T17:15:12.806 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.CloneV2ParentAbortFixIncompleteChildReattach 2024-08-17T17:15:12.835 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:15:12.845 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 0~10 2024-08-17T17:15:12.852 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:15:12.870 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:12 smithi165 ceph-mon[34868]: osdmap e296: 8 total, 8 up, 8 in 2024-08-17T17:15:12.871 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:12 smithi165 systemd[1]: Stopping Ceph grafana.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:15:12.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:12 smithi136 ceph-mon[31034]: osdmap e296: 8 total, 8 up, 8 in 2024-08-17T17:15:12.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:12 smithi136 ceph-mon[42580]: osdmap e296: 8 total, 8 up, 8 in 2024-08-17T17:15:13.261 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:12 smithi165 podman[95869]: 2024-08-17 17:15:12.869809139 +0000 UTC m=+0.140910261 container died be80a687c4027e4ca5ef4907b379acf45ad965f4750d3be4d8a82458d15667b7 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a, com.redhat.component=ubi8-container, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, io.openshift.expose-services=, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, vcs-type=git, maintainer=Paul Cuzner , release=239.1651231664, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, name=ubi8, build-date=2022-04-29T11:48:11.055897, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-08-17T17:15:13.689 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:13 smithi165 podman[95869]: 2024-08-17 17:15:13.407031728 +0000 UTC m=+0.678132841 container cleanup be80a687c4027e4ca5ef4907b379acf45ad965f4750d3be4d8a82458d15667b7 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, version=8.5, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, description=Ceph Grafana Container, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, vendor=Red Hat, Inc., name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, architecture=x86_64, io.openshift.tags=base rhel8, maintainer=Paul Cuzner ) 2024-08-17T17:15:13.689 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:13 smithi165 bash[95869]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a 2024-08-17T17:15:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:13 smithi136 ceph-mon[31034]: pgmap v162: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 344 KiB/s rd, 20 KiB/s wr, 439 op/s 2024-08-17T17:15:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:13 smithi136 ceph-mon[31034]: osdmap e297: 8 total, 8 up, 8 in 2024-08-17T17:15:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:13 smithi136 ceph-mon[42580]: pgmap v162: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 344 KiB/s rd, 20 KiB/s wr, 439 op/s 2024-08-17T17:15:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:13 smithi136 ceph-mon[42580]: osdmap e297: 8 total, 8 up, 8 in 2024-08-17T17:15:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:13 smithi165 ceph-mon[34868]: pgmap v162: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 344 KiB/s rd, 20 KiB/s wr, 439 op/s 2024-08-17T17:15:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:13 smithi165 ceph-mon[34868]: osdmap e297: 8 total, 8 up, 8 in 2024-08-17T17:15:14.011 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:13 smithi165 podman[95889]: 2024-08-17 17:15:13.696017172 +0000 UTC m=+0.828140260 container remove be80a687c4027e4ca5ef4907b379acf45ad965f4750d3be4d8a82458d15667b7 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, io.buildah.version=1.32.0, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, distribution-scope=public, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, io.openshift.tags=base rhel8, version=8.5) 2024-08-17T17:15:14.011 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:13 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-08-17T17:15:14.011 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:13 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-08-17T17:15:14.011 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:13 smithi165 systemd[1]: Stopped Ceph grafana.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:15:14.012 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:13 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@grafana.a.service: Consumed 7.747s CPU time. 2024-08-17T17:15:14.402 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 systemd[1]: Starting Ceph grafana.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:15:14.403 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 podman[96023]: 2024-08-17 17:15:14.172552193 +0000 UTC m=+0.018173116 image pull c8b91775d855b99270fc5d22f3c6737e8cca01ef4c25c8b0362295e0746fa39b quay.io/ceph/grafana:10.4.0 2024-08-17T17:15:14.403 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 podman[96023]: 2024-08-17 17:15:14.293032935 +0000 UTC m=+0.138653885 container create 04db24fce5e9ecd97fe76d33e0bdff26ebbb2738fdce1ffc6d9eb4b3739292b0 (image=quay.io/ceph/grafana:10.4.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-08-17T17:15:14.720 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap1 2024-08-17T17:15:14.776 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:15:14.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-mon[34868]: osdmap e298: 8 total, 8 up, 8 in 2024-08-17T17:15:14.805 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 podman[96023]: 2024-08-17 17:15:14.553708383 +0000 UTC m=+0.399329302 container init 04db24fce5e9ecd97fe76d33e0bdff26ebbb2738fdce1ffc6d9eb4b3739292b0 (image=quay.io/ceph/grafana:10.4.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-08-17T17:15:14.805 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 podman[96023]: 2024-08-17 17:15:14.558687449 +0000 UTC m=+0.404308365 container start 04db24fce5e9ecd97fe76d33e0bdff26ebbb2738fdce1ffc6d9eb4b3739292b0 (image=quay.io/ceph/grafana:10.4.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-08-17T17:15:14.805 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 bash[96023]: 04db24fce5e9ecd97fe76d33e0bdff26ebbb2738fdce1ffc6d9eb4b3739292b0 2024-08-17T17:15:14.805 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 systemd[1]: Started Ceph grafana.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:15:14.805 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=settings t=2024-08-17T17:15:14.699206768Z level=info msg="Starting Grafana" version=10.4.0 commit=03f502a94d17f7dc4e6c34acdf8428aedd986e4c branch=HEAD compiled=2024-08-17T17:15:14Z 2024-08-17T17:15:14.805 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=settings t=2024-08-17T17:15:14.699427436Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-08-17T17:15:14.805 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=settings t=2024-08-17T17:15:14.699435298Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-08-17T17:15:14.805 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=settings t=2024-08-17T17:15:14.699438581Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-08-17T17:15:14.805 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=settings t=2024-08-17T17:15:14.699441397Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-08-17T17:15:14.805 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=settings t=2024-08-17T17:15:14.699443851Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-08-17T17:15:14.805 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=settings t=2024-08-17T17:15:14.699446521Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-08-17T17:15:14.805 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=settings t=2024-08-17T17:15:14.699449586Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-08-17T17:15:14.805 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=settings t=2024-08-17T17:15:14.699452336Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-08-17T17:15:14.806 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=settings t=2024-08-17T17:15:14.699454971Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-08-17T17:15:14.806 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=settings t=2024-08-17T17:15:14.699458085Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-08-17T17:15:14.806 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=settings t=2024-08-17T17:15:14.69946071Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-08-17T17:15:14.806 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=settings t=2024-08-17T17:15:14.699463734Z level=info msg=Target target=[all] 2024-08-17T17:15:14.806 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=settings t=2024-08-17T17:15:14.69946967Z level=info msg="Path Home" path=/usr/share/grafana 2024-08-17T17:15:14.807 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=settings t=2024-08-17T17:15:14.699474247Z level=info msg="Path Data" path=/var/lib/grafana 2024-08-17T17:15:14.807 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=settings t=2024-08-17T17:15:14.699476696Z level=info msg="Path Logs" path=/var/log/grafana 2024-08-17T17:15:14.807 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=settings t=2024-08-17T17:15:14.699478986Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-08-17T17:15:14.807 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=settings t=2024-08-17T17:15:14.699481408Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-08-17T17:15:14.807 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=settings t=2024-08-17T17:15:14.699483816Z level=info msg="App mode production" 2024-08-17T17:15:14.807 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=sqlstore t=2024-08-17T17:15:14.699762254Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-08-17T17:15:14.807 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=sqlstore t=2024-08-17T17:15:14.699775162Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-08-17T17:15:14.807 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.700252825Z level=info msg="Starting DB migrations" 2024-08-17T17:15:14.807 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.71370284Z level=info msg="Executing migration" id="Add uid column to user" 2024-08-17T17:15:14.807 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.720207094Z level=info msg="Migration successfully executed" id="Add uid column to user" duration=6.502241ms 2024-08-17T17:15:14.807 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.720661745Z level=info msg="Executing migration" id="Update uid column values for users" 2024-08-17T17:15:14.807 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.720812487Z level=info msg="Migration successfully executed" id="Update uid column values for users" duration=150.909µs 2024-08-17T17:15:14.808 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.721204478Z level=info msg="Executing migration" id="Add unique index user_uid" 2024-08-17T17:15:14.808 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.72197602Z level=info msg="Migration successfully executed" id="Add unique index user_uid" duration=771.315µs 2024-08-17T17:15:14.808 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.724144736Z level=info msg="Executing migration" id="Add playlist column created_at" 2024-08-17T17:15:14.808 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.729764021Z level=info msg="Migration successfully executed" id="Add playlist column created_at" duration=5.617823ms 2024-08-17T17:15:14.808 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.730224854Z level=info msg="Executing migration" id="Add playlist column updated_at" 2024-08-17T17:15:14.808 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.735958576Z level=info msg="Migration successfully executed" id="Add playlist column updated_at" duration=5.73333ms 2024-08-17T17:15:14.808 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.736500021Z level=info msg="Executing migration" id="Add column uid in team" 2024-08-17T17:15:14.808 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.742030826Z level=info msg="Migration successfully executed" id="Add column uid in team" duration=5.528508ms 2024-08-17T17:15:14.808 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.74246632Z level=info msg="Executing migration" id="Update uid column values in team" 2024-08-17T17:15:14.808 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.742597668Z level=info msg="Migration successfully executed" id="Update uid column values in team" duration=131.636µs 2024-08-17T17:15:14.808 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.743343626Z level=info msg="Executing migration" id="Add unique index team_org_id_uid" 2024-08-17T17:15:14.808 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.744095653Z level=info msg="Migration successfully executed" id="Add unique index team_org_id_uid" duration=751.533µs 2024-08-17T17:15:14.809 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.744517508Z level=info msg="Executing migration" id="add index user_auth_token.revoked_at" 2024-08-17T17:15:14.809 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.745208583Z level=info msg="Migration successfully executed" id="add index user_auth_token.revoked_at" duration=690.486µs 2024-08-17T17:15:14.809 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.745629888Z level=info msg="Executing migration" id="add result_fingerprint column to alert_instance" 2024-08-17T17:15:14.809 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.752131393Z level=info msg="Migration successfully executed" id="add result_fingerprint column to alert_instance" duration=6.498118ms 2024-08-17T17:15:14.809 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.752643556Z level=info msg="Executing migration" id="add last_applied column to alert_configuration_history" 2024-08-17T17:15:14.809 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.758111611Z level=info msg="Migration successfully executed" id="add last_applied column to alert_configuration_history" duration=5.464495ms 2024-08-17T17:15:14.809 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.758583249Z level=info msg="Executing migration" id="alter library_element model to mediumtext" 2024-08-17T17:15:14.809 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.75863109Z level=info msg="Migration successfully executed" id="alter library_element model to mediumtext" duration=47.701µs 2024-08-17T17:15:14.809 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.759460894Z level=info msg="Executing migration" id="permission kind migration" 2024-08-17T17:15:14.809 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.764882618Z level=info msg="Migration successfully executed" id="permission kind migration" duration=5.419485ms 2024-08-17T17:15:14.809 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.765325458Z level=info msg="Executing migration" id="permission attribute migration" 2024-08-17T17:15:14.809 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.770529457Z level=info msg="Migration successfully executed" id="permission attribute migration" duration=5.203182ms 2024-08-17T17:15:14.809 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.770960336Z level=info msg="Executing migration" id="permission identifier migration" 2024-08-17T17:15:14.810 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.776972283Z level=info msg="Migration successfully executed" id="permission identifier migration" duration=6.005847ms 2024-08-17T17:15:14.810 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.777468037Z level=info msg="Executing migration" id="add permission identifier index" 2024-08-17T17:15:14.810 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.778402184Z level=info msg="Migration successfully executed" id="add permission identifier index" duration=933.714µs 2024-08-17T17:15:14.810 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.778845688Z level=info msg="Executing migration" id="add permission action scope role_id index" 2024-08-17T17:15:14.810 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.779852908Z level=info msg="Migration successfully executed" id="add permission action scope role_id index" duration=1.00674ms 2024-08-17T17:15:14.810 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.780268881Z level=info msg="Executing migration" id="remove permission role_id action scope index" 2024-08-17T17:15:14.810 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.780977661Z level=info msg="Migration successfully executed" id="remove permission role_id action scope index" duration=708.847µs 2024-08-17T17:15:14.810 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.781362828Z level=info msg="Executing migration" id="rbac disabled migrator" 2024-08-17T17:15:14.811 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.781381722Z level=info msg="Migration successfully executed" id="rbac disabled migrator" duration=21.702µs 2024-08-17T17:15:14.811 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.782199615Z level=info msg="Executing migration" id="drop index IDX_correlation_uid - v1" 2024-08-17T17:15:14.811 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.783139754Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_uid - v1" duration=939.316µs 2024-08-17T17:15:14.811 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.784186917Z level=info msg="Executing migration" id="drop index IDX_correlation_source_uid - v1" 2024-08-17T17:15:14.811 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.785043543Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_source_uid - v1" duration=857.038µs 2024-08-17T17:15:14.811 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.7870171Z level=info msg="Executing migration" id="Rename table correlation to correlation_tmp_qwerty - v1" 2024-08-17T17:15:14.811 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.803692796Z level=info msg="Migration successfully executed" id="Rename table correlation to correlation_tmp_qwerty - v1" duration=16.671679ms 2024-08-17T17:15:14.811 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.804330307Z level=info msg="Executing migration" id="create correlation v2" 2024-08-17T17:15:14.811 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.805508396Z level=info msg="Migration successfully executed" id="create correlation v2" duration=1.177793ms 2024-08-17T17:15:14.811 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.807134263Z level=info msg="Executing migration" id="create index IDX_correlation_uid - v2" 2024-08-17T17:15:15.056 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.808178479Z level=info msg="Migration successfully executed" id="create index IDX_correlation_uid - v2" duration=1.043749ms 2024-08-17T17:15:15.056 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.809880288Z level=info msg="Executing migration" id="create index IDX_correlation_source_uid - v2" 2024-08-17T17:15:15.056 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.810752639Z level=info msg="Migration successfully executed" id="create index IDX_correlation_source_uid - v2" duration=871.517µs 2024-08-17T17:15:15.056 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.812746247Z level=info msg="Executing migration" id="create index IDX_correlation_org_id - v2" 2024-08-17T17:15:15.056 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.813861127Z level=info msg="Migration successfully executed" id="create index IDX_correlation_org_id - v2" duration=1.115254ms 2024-08-17T17:15:15.056 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.815503791Z level=info msg="Executing migration" id="copy correlation v1 to v2" 2024-08-17T17:15:15.056 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.815691162Z level=info msg="Migration successfully executed" id="copy correlation v1 to v2" duration=187.516µs 2024-08-17T17:15:15.056 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.818422396Z level=info msg="Executing migration" id="drop correlation_tmp_qwerty" 2024-08-17T17:15:15.056 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.819170558Z level=info msg="Migration successfully executed" id="drop correlation_tmp_qwerty" duration=756.006µs 2024-08-17T17:15:15.057 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.821191438Z level=info msg="Executing migration" id="add provisioning column" 2024-08-17T17:15:15.057 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.827250379Z level=info msg="Migration successfully executed" id="add provisioning column" duration=6.057205ms 2024-08-17T17:15:15.057 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.827790767Z level=info msg="Executing migration" id="backfill empty share column fields with default of public" 2024-08-17T17:15:15.057 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.82797544Z level=info msg="Migration successfully executed" id="backfill empty share column fields with default of public" duration=183.672µs 2024-08-17T17:15:15.057 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.830475592Z level=info msg="Executing migration" id="migrate contents column to mediumblob for MySQL" 2024-08-17T17:15:15.057 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.830522561Z level=info msg="Migration successfully executed" id="migrate contents column to mediumblob for MySQL" duration=46.873µs 2024-08-17T17:15:15.057 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.833267625Z level=info msg="Executing migration" id="add origin column to seed_assignment" 2024-08-17T17:15:15.057 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.839340904Z level=info msg="Migration successfully executed" id="add origin column to seed_assignment" duration=6.071387ms 2024-08-17T17:15:15.058 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.839759243Z level=info msg="Executing migration" id="add origin to plugin seed_assignment" 2024-08-17T17:15:15.058 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.839965022Z level=info msg="Migration successfully executed" id="add origin to plugin seed_assignment" duration=206.025µs 2024-08-17T17:15:15.059 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.840468162Z level=info msg="Executing migration" id="prevent seeding OnCall access" 2024-08-17T17:15:15.059 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.840581576Z level=info msg="Migration successfully executed" id="prevent seeding OnCall access" duration=113.637µs 2024-08-17T17:15:15.059 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.843228386Z level=info msg="Executing migration" id="managed folder permissions library panel actions migration" 2024-08-17T17:15:15.059 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.844175683Z level=info msg="Migration successfully executed" id="managed folder permissions library panel actions migration" duration=945.348µs 2024-08-17T17:15:15.059 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.845805673Z level=info msg="Executing migration" id="Remove unique index for folder.title and folder.parent_uid" 2024-08-17T17:15:15.059 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.846696232Z level=info msg="Migration successfully executed" id="Remove unique index for folder.title and folder.parent_uid" duration=889.935µs 2024-08-17T17:15:15.059 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.848497972Z level=info msg="Executing migration" id="Add unique index for title, parent_uid, and org_id" 2024-08-17T17:15:15.059 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.849303965Z level=info msg="Migration successfully executed" id="Add unique index for title, parent_uid, and org_id" duration=805.851µs 2024-08-17T17:15:15.059 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.851045834Z level=info msg="Executing migration" id="Sync dashboard and folder table" 2024-08-17T17:15:15.059 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.851462209Z level=info msg="Migration successfully executed" id="Sync dashboard and folder table" duration=416.487µs 2024-08-17T17:15:15.059 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.853719636Z level=info msg="Executing migration" id="Remove ghost folders from the folder table" 2024-08-17T17:15:15.059 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.853918628Z level=info msg="Migration successfully executed" id="Remove ghost folders from the folder table" duration=198.915µs 2024-08-17T17:15:15.060 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.856505324Z level=info msg="Executing migration" id="Remove unique index UQE_folder_uid_org_id" 2024-08-17T17:15:15.060 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.857217282Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_uid_org_id" duration=712.176µs 2024-08-17T17:15:15.060 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.859125381Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_uid" 2024-08-17T17:15:15.060 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.8598278Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_uid" duration=702.293µs 2024-08-17T17:15:15.060 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.861863651Z level=info msg="Executing migration" id="Remove unique index UQE_folder_title_parent_uid_org_id" 2024-08-17T17:15:15.060 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.862540786Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_title_parent_uid_org_id" duration=677.005µs 2024-08-17T17:15:15.060 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.864427994Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_parent_uid_title" 2024-08-17T17:15:15.060 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.865132691Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_parent_uid_title" duration=704.493µs 2024-08-17T17:15:15.060 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.867031061Z level=info msg="Executing migration" id="Remove index IDX_folder_parent_uid_org_id" 2024-08-17T17:15:15.061 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.867694762Z level=info msg="Migration successfully executed" id="Remove index IDX_folder_parent_uid_org_id" duration=663.633µs 2024-08-17T17:15:15.061 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.869647537Z level=info msg="Executing migration" id="create anon_device table" 2024-08-17T17:15:15.061 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.870227931Z level=info msg="Migration successfully executed" id="create anon_device table" duration=580.172µs 2024-08-17T17:15:15.061 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.872153533Z level=info msg="Executing migration" id="add unique index anon_device.device_id" 2024-08-17T17:15:15.061 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.872817735Z level=info msg="Migration successfully executed" id="add unique index anon_device.device_id" duration=663.976µs 2024-08-17T17:15:15.061 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.874801186Z level=info msg="Executing migration" id="add index anon_device.updated_at" 2024-08-17T17:15:15.061 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.875486802Z level=info msg="Migration successfully executed" id="add index anon_device.updated_at" duration=685.472µs 2024-08-17T17:15:15.061 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.877590727Z level=info msg="Executing migration" id="create signing_key table" 2024-08-17T17:15:15.061 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.878163061Z level=info msg="Migration successfully executed" id="create signing_key table" duration=572.142µs 2024-08-17T17:15:15.061 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.880270678Z level=info msg="Executing migration" id="add unique index signing_key.key_id" 2024-08-17T17:15:15.061 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.880944289Z level=info msg="Migration successfully executed" id="add unique index signing_key.key_id" duration=671.54µs 2024-08-17T17:15:15.061 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.882975145Z level=info msg="Executing migration" id="set legacy alert migration status in kvstore" 2024-08-17T17:15:15.061 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.88366741Z level=info msg="Migration successfully executed" id="set legacy alert migration status in kvstore" duration=692.26µs 2024-08-17T17:15:15.061 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.885688228Z level=info msg="Executing migration" id="migrate record of created folders during legacy migration to kvstore" 2024-08-17T17:15:15.062 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.886114387Z level=info msg="Migration successfully executed" id="migrate record of created folders during legacy migration to kvstore" duration=426.734µs 2024-08-17T17:15:15.062 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.888305099Z level=info msg="Executing migration" id="Add folder_uid for dashboard" 2024-08-17T17:15:15.062 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.893508236Z level=info msg="Migration successfully executed" id="Add folder_uid for dashboard" duration=5.202765ms 2024-08-17T17:15:15.062 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.893856054Z level=info msg="Executing migration" id="Populate dashboard folder_uid column" 2024-08-17T17:15:15.062 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.8955637Z level=info msg="Migration successfully executed" id="Populate dashboard folder_uid column" duration=1.710026ms 2024-08-17T17:15:15.062 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.89620531Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title" 2024-08-17T17:15:15.062 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.897092526Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title" duration=888.482µs 2024-08-17T17:15:15.062 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.897408653Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_id_title" 2024-08-17T17:15:15.062 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.898099034Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_id_title" duration=689.071µs 2024-08-17T17:15:15.062 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.898427168Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_uid_title" 2024-08-17T17:15:15.062 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.899098805Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_uid_title" duration=671.537µs 2024-08-17T17:15:15.062 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.89944227Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" 2024-08-17T17:15:15.063 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.900270613Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" duration=828.089µs 2024-08-17T17:15:15.063 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.900615212Z level=info msg="Executing migration" id="Restore index for dashboard_org_id_folder_id_title" 2024-08-17T17:15:15.063 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.901385343Z level=info msg="Migration successfully executed" id="Restore index for dashboard_org_id_folder_id_title" duration=769.929µs 2024-08-17T17:15:15.063 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.901732657Z level=info msg="Executing migration" id="create sso_setting table" 2024-08-17T17:15:15.063 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.902475687Z level=info msg="Migration successfully executed" id="create sso_setting table" duration=742.881µs 2024-08-17T17:15:15.063 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.902884821Z level=info msg="Executing migration" id="copy kvstore migration status to each org" 2024-08-17T17:15:15.063 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.903512998Z level=info msg="Migration successfully executed" id="copy kvstore migration status to each org" duration=628.352µs 2024-08-17T17:15:15.063 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.903830884Z level=info msg="Executing migration" id="add back entry for orgid=0 migrated status" 2024-08-17T17:15:15.063 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.904220039Z level=info msg="Migration successfully executed" id="add back entry for orgid=0 migrated status" duration=389.267µs 2024-08-17T17:15:15.063 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.904686533Z level=info msg="Executing migration" id="alter kv_store.value to longtext" 2024-08-17T17:15:15.063 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.904724724Z level=info msg="Migration successfully executed" id="alter kv_store.value to longtext" duration=38.522µs 2024-08-17T17:15:15.063 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.905559057Z level=info msg="Executing migration" id="add notification_settings column to alert_rule table" 2024-08-17T17:15:15.063 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.910907315Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule table" duration=5.344733ms 2024-08-17T17:15:15.063 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.911274182Z level=info msg="Executing migration" id="add notification_settings column to alert_rule_version table" 2024-08-17T17:15:15.064 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.91664357Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule_version table" duration=5.370846ms 2024-08-17T17:15:15.064 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.916994243Z level=info msg="Executing migration" id="removing scope from alert.instances:read action migration" 2024-08-17T17:15:15.064 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.917216152Z level=info msg="Migration successfully executed" id="removing scope from alert.instances:read action migration" duration=221.975µs 2024-08-17T17:15:15.064 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=migrator t=2024-08-17T17:15:14.917849546Z level=info msg="migrations completed" performed=65 skipped=482 duration=204.219733ms 2024-08-17T17:15:15.064 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=sqlstore t=2024-08-17T17:15:14.918519727Z level=info msg="Created default organization" 2024-08-17T17:15:15.064 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=secrets t=2024-08-17T17:15:14.920513089Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-08-17T17:15:15.064 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=plugin.store t=2024-08-17T17:15:14.937918447Z level=info msg="Loading plugins..." 2024-08-17T17:15:15.065 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=local.finder t=2024-08-17T17:15:14.982106374Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-08-17T17:15:15.065 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=plugin.store t=2024-08-17T17:15:14.982132811Z level=info msg="Plugins loaded" count=55 duration=44.215197ms 2024-08-17T17:15:15.065 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=query_data t=2024-08-17T17:15:14.984251993Z level=info msg="Query Service initialization" 2024-08-17T17:15:15.065 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=live.push_http t=2024-08-17T17:15:14.986843031Z level=info msg="Live Push Gateway initialization" 2024-08-17T17:15:15.065 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=ngalert.migration t=2024-08-17T17:15:14.988203313Z level=info msg=Starting 2024-08-17T17:15:15.065 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=ngalert.state.manager t=2024-08-17T17:15:14.994325878Z level=info msg="Running in alternative execution of Error/NoData mode" 2024-08-17T17:15:15.065 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=infra.usagestats.collector t=2024-08-17T17:15:14.995742237Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-08-17T17:15:15.066 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=provisioning.datasources t=2024-08-17T17:15:14.998482168Z level=info msg="deleted datasource based on configuration" name=Dashboard1 2024-08-17T17:15:15.066 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:14 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=provisioning.datasources t=2024-08-17T17:15:14.998809502Z level=info msg="inserting datasource from configuration" name=Dashboard1 uid=P43CA22E17D0F9596 2024-08-17T17:15:15.066 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=provisioning.alerting t=2024-08-17T17:15:15.037055949Z level=info msg="starting to provision alerting" 2024-08-17T17:15:15.066 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=provisioning.alerting t=2024-08-17T17:15:15.037079113Z level=info msg="finished to provision alerting" 2024-08-17T17:15:15.066 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=ngalert.state.manager t=2024-08-17T17:15:15.037195848Z level=info msg="Warming state cache for startup" 2024-08-17T17:15:15.066 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=grafanaStorageLogger t=2024-08-17T17:15:15.037225121Z level=info msg="Storage starting" 2024-08-17T17:15:15.066 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=ngalert.state.manager t=2024-08-17T17:15:15.037520093Z level=info msg="State cache has been initialized" states=0 duration=323.44µs 2024-08-17T17:15:15.066 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=ngalert.multiorg.alertmanager t=2024-08-17T17:15:15.037588424Z level=info msg="Starting MultiOrg Alertmanager" 2024-08-17T17:15:15.066 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=ngalert.scheduler t=2024-08-17T17:15:15.037623294Z level=info msg="Starting scheduler" tickInterval=10s maxAttempts=1 2024-08-17T17:15:15.066 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=ticker t=2024-08-17T17:15:15.03767551Z level=info msg=starting first_tick=2024-08-17T17:15:20Z 2024-08-17T17:15:15.066 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=http.server t=2024-08-17T17:15:15.039819174Z level=info msg="HTTP Server TLS settings" MinTLSVersion=TLS1.2 configuredciphers=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA 2024-08-17T17:15:15.066 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=http.server t=2024-08-17T17:15:15.040146086Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-08-17T17:15:15.067 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=provisioning.dashboard t=2024-08-17T17:15:15.055016135Z level=info msg="starting to provision dashboards" 2024-08-17T17:15:15.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:14 smithi136 ceph-mon[31034]: osdmap e298: 8 total, 8 up, 8 in 2024-08-17T17:15:15.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:14 smithi136 ceph-mon[42580]: osdmap e298: 8 total, 8 up, 8 in 2024-08-17T17:15:15.357 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=plugins.update.checker t=2024-08-17T17:15:15.121457862Z level=info msg="Update check succeeded" duration=84.19294ms 2024-08-17T17:15:15.357 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=grafana.update.checker t=2024-08-17T17:15:15.129996496Z level=info msg="Update check succeeded" duration=92.768573ms 2024-08-17T17:15:15.357 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=provisioning.dashboard t=2024-08-17T17:15:15.239282289Z level=info msg="finished to provision dashboards" 2024-08-17T17:15:15.357 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=grafana-apiserver t=2024-08-17T17:15:15.356910107Z level=info msg="Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager" 2024-08-17T17:15:15.674 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:15:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=grafana-apiserver t=2024-08-17T17:15:15.357444055Z level=info msg="Adding GroupVersion featuretoggle.grafana.app v0alpha1 to ResourceManager" 2024-08-17T17:15:15.893 INFO:tasks.workunit.client.0.smithi136.stdout:migration_abort 2024-08-17T17:15:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:15 smithi165 ceph-mon[34868]: pgmap v165: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 344 KiB/s rd, 20 KiB/s wr, 439 op/s 2024-08-17T17:15:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:15 smithi165 ceph-mon[34868]: osdmap e299: 8 total, 8 up, 8 in 2024-08-17T17:15:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:15 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:15 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:15 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:15:16.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:15 smithi136 ceph-mon[31034]: pgmap v165: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 344 KiB/s rd, 20 KiB/s wr, 439 op/s 2024-08-17T17:15:16.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:15 smithi136 ceph-mon[31034]: osdmap e299: 8 total, 8 up, 8 in 2024-08-17T17:15:16.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:15 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:16.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:15 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:16.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:15 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:15:16.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:15 smithi136 ceph-mon[42580]: pgmap v165: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 344 KiB/s rd, 20 KiB/s wr, 439 op/s 2024-08-17T17:15:16.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:15 smithi136 ceph-mon[42580]: osdmap e299: 8 total, 8 up, 8 in 2024-08-17T17:15:16.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:15 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:16.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:15 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:16.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:15 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:15:16.854 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.CloneV2ParentAbortFixIncompleteChildReattach (4048 ms) 2024-08-17T17:15:16.855 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.CloneV2ParentAbortFixParentReattach 2024-08-17T17:15:16.888 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:15:16.900 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 0~10 2024-08-17T17:15:16.908 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:15:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:16 smithi165 ceph-mon[34868]: osdmap e300: 8 total, 8 up, 8 in 2024-08-17T17:15:17.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:16 smithi136 ceph-mon[31034]: osdmap e300: 8 total, 8 up, 8 in 2024-08-17T17:15:17.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:16 smithi136 ceph-mon[42580]: osdmap e300: 8 total, 8 up, 8 in 2024-08-17T17:15:17.730 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:15:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:15:17] "GET /metrics HTTP/1.1" 200 38727 "" "Prometheus/2.51.0" 2024-08-17T17:15:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:17 smithi165 ceph-mon[34868]: pgmap v168: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 313 KiB/s rd, 23 KiB/s wr, 403 op/s 2024-08-17T17:15:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:17 smithi165 ceph-mon[34868]: osdmap e301: 8 total, 8 up, 8 in 2024-08-17T17:15:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:17 smithi165 ceph-mon[34868]: osdmap e302: 8 total, 8 up, 8 in 2024-08-17T17:15:18.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:17 smithi136 ceph-mon[42580]: pgmap v168: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 313 KiB/s rd, 23 KiB/s wr, 403 op/s 2024-08-17T17:15:18.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:17 smithi136 ceph-mon[42580]: osdmap e301: 8 total, 8 up, 8 in 2024-08-17T17:15:18.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:17 smithi136 ceph-mon[42580]: osdmap e302: 8 total, 8 up, 8 in 2024-08-17T17:15:18.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:17 smithi136 ceph-mon[31034]: pgmap v168: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 313 KiB/s rd, 23 KiB/s wr, 403 op/s 2024-08-17T17:15:18.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:17 smithi136 ceph-mon[31034]: osdmap e301: 8 total, 8 up, 8 in 2024-08-17T17:15:18.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:17 smithi136 ceph-mon[31034]: osdmap e302: 8 total, 8 up, 8 in 2024-08-17T17:15:18.536 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap1 2024-08-17T17:15:18.590 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:15:19.706 INFO:tasks.workunit.client.0.smithi136.stdout:migration_abort 2024-08-17T17:15:19.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:19 smithi136 ceph-mon[42580]: osdmap e303: 8 total, 8 up, 8 in 2024-08-17T17:15:19.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:19 smithi136 ceph-mon[42580]: pgmap v172: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 320 KiB/s rd, 24 KiB/s wr, 412 op/s 2024-08-17T17:15:19.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:19 smithi136 ceph-mon[31034]: osdmap e303: 8 total, 8 up, 8 in 2024-08-17T17:15:19.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:19 smithi136 ceph-mon[31034]: pgmap v172: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 320 KiB/s rd, 24 KiB/s wr, 412 op/s 2024-08-17T17:15:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:19 smithi165 ceph-mon[34868]: osdmap e303: 8 total, 8 up, 8 in 2024-08-17T17:15:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:19 smithi165 ceph-mon[34868]: pgmap v172: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 320 KiB/s rd, 24 KiB/s wr, 412 op/s 2024-08-17T17:15:20.671 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.CloneV2ParentAbortFixParentReattach (3817 ms) 2024-08-17T17:15:20.671 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.CloneV2ParentAbortRelinkNotNeeded 2024-08-17T17:15:20.702 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:15:20.714 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 0~10 2024-08-17T17:15:20.722 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:15:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:20 smithi165 ceph-mon[34868]: osdmap e304: 8 total, 8 up, 8 in 2024-08-17T17:15:20.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:20 smithi136 ceph-mon[31034]: osdmap e304: 8 total, 8 up, 8 in 2024-08-17T17:15:20.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:20 smithi136 ceph-mon[42580]: osdmap e304: 8 total, 8 up, 8 in 2024-08-17T17:15:21.895 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:21 smithi165 ceph-mon[34868]: osdmap e305: 8 total, 8 up, 8 in 2024-08-17T17:15:21.895 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:21 smithi165 ceph-mon[34868]: pgmap v175: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:21 smithi136 ceph-mon[31034]: osdmap e305: 8 total, 8 up, 8 in 2024-08-17T17:15:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:21 smithi136 ceph-mon[31034]: pgmap v175: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:21 smithi136 ceph-mon[42580]: osdmap e305: 8 total, 8 up, 8 in 2024-08-17T17:15:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:21 smithi136 ceph-mon[42580]: pgmap v175: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:22.564 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap1 2024-08-17T17:15:22.624 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:15:22.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:22 smithi136 ceph-mon[31034]: osdmap e306: 8 total, 8 up, 8 in 2024-08-17T17:15:22.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:22 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:22.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:22 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:22.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:22 smithi136 ceph-mon[42580]: osdmap e306: 8 total, 8 up, 8 in 2024-08-17T17:15:22.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:22 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:22.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:22 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:22 smithi165 ceph-mon[34868]: osdmap e306: 8 total, 8 up, 8 in 2024-08-17T17:15:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:22 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:22 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:23.719 INFO:tasks.workunit.client.0.smithi136.stdout:migration_abort 2024-08-17T17:15:23.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:23 smithi165 ceph-mon[34868]: osdmap e307: 8 total, 8 up, 8 in 2024-08-17T17:15:23.751 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:23 smithi165 ceph-mon[34868]: pgmap v178: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 365 KiB/s rd, 29 KiB/s wr, 477 op/s 2024-08-17T17:15:23.751 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:23 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:23.751 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:23 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:23.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:23 smithi136 ceph-mon[42580]: osdmap e307: 8 total, 8 up, 8 in 2024-08-17T17:15:23.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:23 smithi136 ceph-mon[42580]: pgmap v178: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 365 KiB/s rd, 29 KiB/s wr, 477 op/s 2024-08-17T17:15:23.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:23 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:23.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:23 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:23.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:23 smithi136 ceph-mon[31034]: osdmap e307: 8 total, 8 up, 8 in 2024-08-17T17:15:23.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:23 smithi136 ceph-mon[31034]: pgmap v178: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 365 KiB/s rd, 29 KiB/s wr, 477 op/s 2024-08-17T17:15:23.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:23 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:23.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:23 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:24.698 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.CloneV2ParentAbortRelinkNotNeeded (4027 ms) 2024-08-17T17:15:24.699 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.StressNoMigrate 2024-08-17T17:15:24.730 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:15:24.741 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 51155419~6603932 2024-08-17T17:15:24.791 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 15030660~3995958 2024-08-17T17:15:24.791 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 50473398~704226 2024-08-17T17:15:24.797 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 41006748~1265692 2024-08-17T17:15:24.797 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 51875869~861472 2024-08-17T17:15:24.804 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 75318472~6676323 2024-08-17T17:15:24.804 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 1602394~4875501 2024-08-17T17:15:24.834 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 31160781~2733925 2024-08-17T17:15:24.834 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap0 2024-08-17T17:15:24.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:24 smithi165 ceph-mon[34868]: osdmap e308: 8 total, 8 up, 8 in 2024-08-17T17:15:24.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:24 smithi136 ceph-mon[31034]: osdmap e308: 8 total, 8 up, 8 in 2024-08-17T17:15:24.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:24 smithi136 ceph-mon[42580]: osdmap e308: 8 total, 8 up, 8 in 2024-08-17T17:15:25.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:25 smithi136 ceph-mon[42580]: osdmap e309: 8 total, 8 up, 8 in 2024-08-17T17:15:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:25 smithi136 ceph-mon[42580]: pgmap v181: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 365 KiB/s rd, 29 KiB/s wr, 477 op/s 2024-08-17T17:15:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:25 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:15:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:25 smithi136 ceph-mon[31034]: osdmap e309: 8 total, 8 up, 8 in 2024-08-17T17:15:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:25 smithi136 ceph-mon[31034]: pgmap v181: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 365 KiB/s rd, 29 KiB/s wr, 477 op/s 2024-08-17T17:15:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:25 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:15:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:25 smithi165 ceph-mon[34868]: osdmap e309: 8 total, 8 up, 8 in 2024-08-17T17:15:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:25 smithi165 ceph-mon[34868]: pgmap v181: 137 pgs: 137 active+clean; 901 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 365 KiB/s rd, 29 KiB/s wr, 477 op/s 2024-08-17T17:15:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:25 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:15:26.594 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap0 2024-08-17T17:15:26.604 INFO:tasks.workunit.client.0.smithi136.stdout:clone image159 -> image162 2024-08-17T17:15:26.702 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 72391729~6326599 2024-08-17T17:15:26.746 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 78612219~4070762 2024-08-17T17:15:26.751 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 50182844~6483290 2024-08-17T17:15:26.797 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 48830019~382712 2024-08-17T17:15:26.799 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 33438469~121761 2024-08-17T17:15:26.806 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 3720340~4110766 2024-08-17T17:15:26.832 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 44269130~2337013 2024-08-17T17:15:26.851 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 23110187~4544111 2024-08-17T17:15:26.857 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:15:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: osdmap e310: 8 total, 8 up, 8 in 2024-08-17T17:15:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:15:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:15:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-08-17T17:15:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:15:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:26.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-08-17T17:15:26.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-08-17T17:15:26.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-08-17T17:15:26.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:26.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:26.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: osdmap e310: 8 total, 8 up, 8 in 2024-08-17T17:15:26.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:26.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:26.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:15:26.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:15:26.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:26.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-08-17T17:15:26.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:15:26.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:26.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:26.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-08-17T17:15:26.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-08-17T17:15:26.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-08-17T17:15:26.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:26.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:26.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-08-17T17:15:26.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-08-17T17:15:26.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-08-17T17:15:26.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-08-17T17:15:26.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-08-17T17:15:26.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-08-17T17:15:26.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-08-17T17:15:26.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-08-17T17:15:26.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-08-17T17:15:26.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-08-17T17:15:26.953 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-08-17T17:15:26.953 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-08-17T17:15:26.953 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-08-17T17:15:26.953 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-08-17T17:15:26.953 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-08-17T17:15:26.953 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-08-17T17:15:26.953 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-08-17T17:15:26.953 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-08-17T17:15:26.953 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-08-17T17:15:26.953 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:26 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-08-17T17:15:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: osdmap e310: 8 total, 8 up, 8 in 2024-08-17T17:15:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:27.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:15:27.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:15:27.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:27.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-08-17T17:15:27.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:15:27.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:27.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:27.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:27.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:27.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:27.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:27.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:27.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-08-17T17:15:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-08-17T17:15:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-08-17T17:15:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-08-17T17:15:27.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-08-17T17:15:27.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-08-17T17:15:27.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-08-17T17:15:27.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-08-17T17:15:27.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-08-17T17:15:27.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-08-17T17:15:27.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-08-17T17:15:27.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-08-17T17:15:27.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:26 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-08-17T17:15:27.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:15:27] "GET /metrics HTTP/1.1" 200 38730 "" "Prometheus/2.51.0" 2024-08-17T17:15:27.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-08-17T17:15:27.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: Upgrade: Finalizing container_image settings 2024-08-17T17:15:27.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-08-17T17:15:27.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: osdmap e311: 8 total, 8 up, 8 in 2024-08-17T17:15:27.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-08-17T17:15:27.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-08-17T17:15:27.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-08-17T17:15:27.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-08-17T17:15:27.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-08-17T17:15:27.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-08-17T17:15:27.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-08-17T17:15:27.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-08-17T17:15:27.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: pgmap v184: 137 pgs: 137 active+clean; 927 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 254 KiB/s rd, 6.2 MiB/s wr, 336 op/s 2024-08-17T17:15:27.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-08-17T17:15:27.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-08-17T17:15:27.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-08-17T17:15:27.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-08-17T17:15:27.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-08-17T17:15:27.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-08-17T17:15:27.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-08-17T17:15:27.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-08-17T17:15:27.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: Upgrade: Finalizing container_image settings 2024-08-17T17:15:27.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-08-17T17:15:27.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: osdmap e311: 8 total, 8 up, 8 in 2024-08-17T17:15:27.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-08-17T17:15:27.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-08-17T17:15:27.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-08-17T17:15:27.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-08-17T17:15:27.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-08-17T17:15:27.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-08-17T17:15:27.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-08-17T17:15:27.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-08-17T17:15:27.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: pgmap v184: 137 pgs: 137 active+clean; 927 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 254 KiB/s rd, 6.2 MiB/s wr, 336 op/s 2024-08-17T17:15:27.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-08-17T17:15:27.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-08-17T17:15:27.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-08-17T17:15:27.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-08-17T17:15:27.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-08-17T17:15:27.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-08-17T17:15:27.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-08-17T17:15:27.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.952 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.952 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.952 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.952 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.952 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.952 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.952 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: Upgrade: Complete! 2024-08-17T17:15:27.952 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-08-17T17:15:27.952 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-08-17T17:15:27.952 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-08-17T17:15:27.952 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:15:27.952 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:15:27.952 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:15:27.952 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:27.953 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:15:27.953 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:15:27.953 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:15:27.953 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:27.953 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[31034]: osdmap e312: 8 total, 8 up, 8 in 2024-08-17T17:15:27.953 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.953 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.953 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.953 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.953 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.953 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.953 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.954 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.954 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.954 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:27.954 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: Upgrade: Complete! 2024-08-17T17:15:27.954 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-08-17T17:15:27.954 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-08-17T17:15:27.954 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-08-17T17:15:27.954 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:15:27.954 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:15:27.954 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:15:27.954 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:27.954 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:15:27.954 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:15:27.954 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:15:27.955 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:27.955 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:27 smithi136 ceph-mon[42580]: osdmap e312: 8 total, 8 up, 8 in 2024-08-17T17:15:28.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-08-17T17:15:28.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: Upgrade: Finalizing container_image settings 2024-08-17T17:15:28.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-08-17T17:15:28.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: osdmap e311: 8 total, 8 up, 8 in 2024-08-17T17:15:28.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-08-17T17:15:28.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-08-17T17:15:28.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-08-17T17:15:28.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:28.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:28.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-08-17T17:15:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-08-17T17:15:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-08-17T17:15:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-08-17T17:15:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-08-17T17:15:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: pgmap v184: 137 pgs: 137 active+clean; 927 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 254 KiB/s rd, 6.2 MiB/s wr, 336 op/s 2024-08-17T17:15:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-08-17T17:15:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-08-17T17:15:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-08-17T17:15:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-08-17T17:15:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-08-17T17:15:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-08-17T17:15:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-08-17T17:15:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:28.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:28.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:28.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:28.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:28.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:28.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:28.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:28.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:28.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:28.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-08-17T17:15:28.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: Upgrade: Complete! 2024-08-17T17:15:28.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-08-17T17:15:28.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-08-17T17:15:28.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-08-17T17:15:28.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:15:28.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:15:28.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:15:28.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:28.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:15:28.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:15:28.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:15:28.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:28.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:27 smithi165 ceph-mon[34868]: osdmap e312: 8 total, 8 up, 8 in 2024-08-17T17:15:28.547 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap1 2024-08-17T17:15:28.560 INFO:tasks.workunit.client.0.smithi136.stdout:clone image162 -> image164 2024-08-17T17:15:28.686 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 107138017 2024-08-17T17:15:28.697 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 68773963~1845226 2024-08-17T17:15:28.717 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 57263577~7336193 2024-08-17T17:15:28.754 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 51461887~2599761 2024-08-17T17:15:28.803 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 48059083~8264949 2024-08-17T17:15:28.816 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 16621560~1502511 2024-08-17T17:15:28.830 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 49696904~5563006 2024-08-17T17:15:28.833 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 102835267~3809290 2024-08-17T17:15:28.855 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 65644293~2255191 2024-08-17T17:15:28.857 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap2 2024-08-17T17:15:29.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:29 smithi136 ceph-mon[31034]: osdmap e313: 8 total, 8 up, 8 in 2024-08-17T17:15:29.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:29 smithi136 ceph-mon[31034]: pgmap v187: 137 pgs: 137 active+clean; 927 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 254 KiB/s rd, 6.2 MiB/s wr, 336 op/s 2024-08-17T17:15:29.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:29 smithi136 ceph-mon[42580]: osdmap e313: 8 total, 8 up, 8 in 2024-08-17T17:15:29.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:29 smithi136 ceph-mon[42580]: pgmap v187: 137 pgs: 137 active+clean; 927 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 254 KiB/s rd, 6.2 MiB/s wr, 336 op/s 2024-08-17T17:15:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:29 smithi165 ceph-mon[34868]: osdmap e313: 8 total, 8 up, 8 in 2024-08-17T17:15:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:29 smithi165 ceph-mon[34868]: pgmap v187: 137 pgs: 137 active+clean; 927 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 254 KiB/s rd, 6.2 MiB/s wr, 336 op/s 2024-08-17T17:15:30.554 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap2 2024-08-17T17:15:30.566 INFO:tasks.workunit.client.0.smithi136.stdout:clone image164 -> image166 2024-08-17T17:15:30.719 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 117987247 2024-08-17T17:15:30.730 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 1134678~7438925 2024-08-17T17:15:30.790 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 66989537~7190787 2024-08-17T17:15:30.812 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 96335138~5948488 2024-08-17T17:15:30.861 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 42953562~5487173 2024-08-17T17:15:30.873 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 17923308~7472163 2024-08-17T17:15:30.922 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 97832964~7352329 2024-08-17T17:15:30.942 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 24614446~7911934 2024-08-17T17:15:30.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:30 smithi136 ceph-mon[31034]: osdmap e314: 8 total, 8 up, 8 in 2024-08-17T17:15:30.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:30 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:30.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:30 smithi136 ceph-mon[42580]: osdmap e314: 8 total, 8 up, 8 in 2024-08-17T17:15:30.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:30 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:30.988 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 6927502~4505861 2024-08-17T17:15:30.990 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap3 2024-08-17T17:15:31.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:30 smithi165 ceph-mon[34868]: osdmap e314: 8 total, 8 up, 8 in 2024-08-17T17:15:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:30 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:31.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:31 smithi136 ceph-mon[31034]: osdmap e315: 8 total, 8 up, 8 in 2024-08-17T17:15:31.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:31 smithi136 ceph-mon[31034]: pgmap v190: 137 pgs: 137 active+clean; 927 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-08-17T17:15:31.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:31 smithi136 ceph-mon[42580]: osdmap e315: 8 total, 8 up, 8 in 2024-08-17T17:15:31.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:31 smithi136 ceph-mon[42580]: pgmap v190: 137 pgs: 137 active+clean; 927 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-08-17T17:15:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:31 smithi165 ceph-mon[34868]: osdmap e315: 8 total, 8 up, 8 in 2024-08-17T17:15:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:31 smithi165 ceph-mon[34868]: pgmap v190: 137 pgs: 137 active+clean; 927 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-08-17T17:15:32.565 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap3 2024-08-17T17:15:32.577 INFO:tasks.workunit.client.0.smithi136.stdout:clone image166 -> image168 2024-08-17T17:15:32.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:32 smithi165 ceph-mon[34868]: osdmap e316: 8 total, 8 up, 8 in 2024-08-17T17:15:32.764 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 119870475 2024-08-17T17:15:32.925 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.StressNoMigrate (8227 ms) 2024-08-17T17:15:32.925 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.Stress 2024-08-17T17:15:32.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:32 smithi136 ceph-mon[31034]: osdmap e316: 8 total, 8 up, 8 in 2024-08-17T17:15:32.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:32 smithi136 ceph-mon[42580]: osdmap e316: 8 total, 8 up, 8 in 2024-08-17T17:15:32.955 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:15:32.965 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 63497214~1689077 2024-08-17T17:15:32.984 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 65972339~271805 2024-08-17T17:15:32.986 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 12469703~2961387 2024-08-17T17:15:33.013 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 38771010~1320739 2024-08-17T17:15:33.013 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 1872685~2616524 2024-08-17T17:15:33.036 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 51866314~7322965 2024-08-17T17:15:33.036 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 51494154~923003 2024-08-17T17:15:33.048 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 20861302~336845 2024-08-17T17:15:33.048 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap0 2024-08-17T17:15:33.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:33 smithi136 ceph-mon[31034]: osdmap e317: 8 total, 8 up, 8 in 2024-08-17T17:15:33.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:33 smithi136 ceph-mon[31034]: pgmap v193: 137 pgs: 137 active+clean; 1.0 GiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.1 MiB/s rd, 19 MiB/s wr, 615 op/s 2024-08-17T17:15:33.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:33 smithi136 ceph-mon[42580]: osdmap e317: 8 total, 8 up, 8 in 2024-08-17T17:15:33.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:33 smithi136 ceph-mon[42580]: pgmap v193: 137 pgs: 137 active+clean; 1.0 GiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.1 MiB/s rd, 19 MiB/s wr, 615 op/s 2024-08-17T17:15:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:33 smithi165 ceph-mon[34868]: osdmap e317: 8 total, 8 up, 8 in 2024-08-17T17:15:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:33 smithi165 ceph-mon[34868]: pgmap v193: 137 pgs: 137 active+clean; 1.0 GiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.1 MiB/s rd, 19 MiB/s wr, 615 op/s 2024-08-17T17:15:34.572 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap0 2024-08-17T17:15:34.586 INFO:tasks.workunit.client.0.smithi136.stdout:clone image169 -> image172 2024-08-17T17:15:34.690 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 118623164 2024-08-17T17:15:34.701 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 6692940~826378 2024-08-17T17:15:34.722 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 71739616~7249359 2024-08-17T17:15:34.728 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 102815417~6117868 2024-08-17T17:15:34.766 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 83975985~3065867 2024-08-17T17:15:34.766 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 109714736~1296851 2024-08-17T17:15:34.778 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 113113596~5509568 2024-08-17T17:15:34.780 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 67824130~2241865 2024-08-17T17:15:34.798 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 56387685~4357772 2024-08-17T17:15:34.804 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:15:34.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:34 smithi136 ceph-mon[31034]: osdmap e318: 8 total, 8 up, 8 in 2024-08-17T17:15:34.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:34 smithi136 ceph-mon[42580]: osdmap e318: 8 total, 8 up, 8 in 2024-08-17T17:15:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:34 smithi165 ceph-mon[34868]: osdmap e318: 8 total, 8 up, 8 in 2024-08-17T17:15:35.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:35 smithi136 ceph-mon[31034]: osdmap e319: 8 total, 8 up, 8 in 2024-08-17T17:15:35.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:35 smithi136 ceph-mon[31034]: pgmap v196: 137 pgs: 137 active+clean; 1.0 GiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.1 MiB/s rd, 19 MiB/s wr, 615 op/s 2024-08-17T17:15:35.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:35 smithi136 ceph-mon[42580]: osdmap e319: 8 total, 8 up, 8 in 2024-08-17T17:15:35.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:35 smithi136 ceph-mon[42580]: pgmap v196: 137 pgs: 137 active+clean; 1.0 GiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.1 MiB/s rd, 19 MiB/s wr, 615 op/s 2024-08-17T17:15:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:35 smithi165 ceph-mon[34868]: osdmap e319: 8 total, 8 up, 8 in 2024-08-17T17:15:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:35 smithi165 ceph-mon[34868]: pgmap v196: 137 pgs: 137 active+clean; 1.0 GiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.1 MiB/s rd, 19 MiB/s wr, 615 op/s 2024-08-17T17:15:36.587 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap1 2024-08-17T17:15:36.600 INFO:tasks.workunit.client.0.smithi136.stdout:clone image172 -> image174 2024-08-17T17:15:36.731 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 92445508 2024-08-17T17:15:36.747 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 27320148~1687275 2024-08-17T17:15:36.768 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 64655737~4298766 2024-08-17T17:15:36.799 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 20290785~4201450 2024-08-17T17:15:36.832 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 37238627~1784139 2024-08-17T17:15:36.839 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 48617513~917247 2024-08-17T17:15:36.851 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 66646463~6014332 2024-08-17T17:15:36.856 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 30544008~1332904 2024-08-17T17:15:36.870 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 73716536~960651 2024-08-17T17:15:36.873 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap2 2024-08-17T17:15:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:36 smithi136 ceph-mon[31034]: osdmap e320: 8 total, 8 up, 8 in 2024-08-17T17:15:36.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:36 smithi136 ceph-mon[42580]: osdmap e320: 8 total, 8 up, 8 in 2024-08-17T17:15:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:36 smithi165 ceph-mon[34868]: osdmap e320: 8 total, 8 up, 8 in 2024-08-17T17:15:37.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:15:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:15:37] "GET /metrics HTTP/1.1" 200 38730 "" "Prometheus/2.51.0" 2024-08-17T17:15:37.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:37 smithi136 ceph-mon[31034]: osdmap e321: 8 total, 8 up, 8 in 2024-08-17T17:15:37.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:37 smithi136 ceph-mon[31034]: pgmap v199: 137 pgs: 137 active+clean; 1.2 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 20 MiB/s rd, 23 MiB/s wr, 558 op/s 2024-08-17T17:15:37.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:37 smithi136 ceph-mon[31034]: osdmap e322: 8 total, 8 up, 8 in 2024-08-17T17:15:37.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:37 smithi136 ceph-mon[42580]: osdmap e321: 8 total, 8 up, 8 in 2024-08-17T17:15:37.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:37 smithi136 ceph-mon[42580]: pgmap v199: 137 pgs: 137 active+clean; 1.2 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 20 MiB/s rd, 23 MiB/s wr, 558 op/s 2024-08-17T17:15:37.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:37 smithi136 ceph-mon[42580]: osdmap e322: 8 total, 8 up, 8 in 2024-08-17T17:15:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:37 smithi165 ceph-mon[34868]: osdmap e321: 8 total, 8 up, 8 in 2024-08-17T17:15:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:37 smithi165 ceph-mon[34868]: pgmap v199: 137 pgs: 137 active+clean; 1.2 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 20 MiB/s rd, 23 MiB/s wr, 558 op/s 2024-08-17T17:15:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:37 smithi165 ceph-mon[34868]: osdmap e322: 8 total, 8 up, 8 in 2024-08-17T17:15:38.545 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 20240761~4188871 2024-08-17T17:15:38.557 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 51998548~641580 2024-08-17T17:15:38.557 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 53521814~3659266 2024-08-17T17:15:38.558 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 11212672~6983525 2024-08-17T17:15:38.559 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 89241826~696710 2024-08-17T17:15:38.559 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 6914918~2027882 2024-08-17T17:15:38.560 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 19296633~2126339 2024-08-17T17:15:38.604 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 47343225~3413754 2024-08-17T17:15:38.605 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap3 2024-08-17T17:15:39.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:39 smithi136 ceph-mon[31034]: osdmap e323: 8 total, 8 up, 8 in 2024-08-17T17:15:39.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:39 smithi136 ceph-mon[31034]: pgmap v202: 137 pgs: 137 active+clean; 1.2 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 20 MiB/s rd, 23 MiB/s wr, 558 op/s 2024-08-17T17:15:39.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:39 smithi136 ceph-mon[42580]: osdmap e323: 8 total, 8 up, 8 in 2024-08-17T17:15:39.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:39 smithi136 ceph-mon[42580]: pgmap v202: 137 pgs: 137 active+clean; 1.2 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 20 MiB/s rd, 23 MiB/s wr, 558 op/s 2024-08-17T17:15:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:39 smithi165 ceph-mon[34868]: osdmap e323: 8 total, 8 up, 8 in 2024-08-17T17:15:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:39 smithi165 ceph-mon[34868]: pgmap v202: 137 pgs: 137 active+clean; 1.2 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 20 MiB/s rd, 23 MiB/s wr, 558 op/s 2024-08-17T17:15:40.563 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap3 2024-08-17T17:15:40.573 INFO:tasks.workunit.client.0.smithi136.stdout:clone image174 -> image176 2024-08-17T17:15:40.739 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:15:40.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:40 smithi136 ceph-mon[31034]: osdmap e324: 8 total, 8 up, 8 in 2024-08-17T17:15:40.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:40 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:15:40.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:40 smithi136 ceph-mon[42580]: osdmap e324: 8 total, 8 up, 8 in 2024-08-17T17:15:40.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:40 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:15:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:40 smithi165 ceph-mon[34868]: osdmap e324: 8 total, 8 up, 8 in 2024-08-17T17:15:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:40 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:15:41.106 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:15:41.288 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:15:41.488 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.Stress (8563 ms) 2024-08-17T17:15:41.488 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.Stress2 2024-08-17T17:15:41.517 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:15:41.527 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 48063969~3258266 2024-08-17T17:15:41.553 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 29341337~3764891 2024-08-17T17:15:41.554 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 43333655~6627455 2024-08-17T17:15:41.588 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 33126183~6275413 2024-08-17T17:15:41.588 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 40790496~7964539 2024-08-17T17:15:41.625 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 37433547~694755 2024-08-17T17:15:41.627 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 4883626~6580792 2024-08-17T17:15:41.667 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 32388196~2028812 2024-08-17T17:15:41.668 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap0 2024-08-17T17:15:41.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:41 smithi136 ceph-mon[42580]: osdmap e325: 8 total, 8 up, 8 in 2024-08-17T17:15:41.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:41 smithi136 ceph-mon[42580]: pgmap v205: 137 pgs: 137 active+clean; 1.2 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-08-17T17:15:41.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:41 smithi136 ceph-mon[42580]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:41.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:41 smithi136 ceph-mon[31034]: osdmap e325: 8 total, 8 up, 8 in 2024-08-17T17:15:41.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:41 smithi136 ceph-mon[31034]: pgmap v205: 137 pgs: 137 active+clean; 1.2 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-08-17T17:15:41.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:41 smithi136 ceph-mon[31034]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:42.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:41 smithi165 ceph-mon[34868]: osdmap e325: 8 total, 8 up, 8 in 2024-08-17T17:15:42.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:41 smithi165 ceph-mon[34868]: pgmap v205: 137 pgs: 137 active+clean; 1.2 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-08-17T17:15:42.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:41 smithi165 ceph-mon[34868]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:42.210 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'echo wait for minority of mons to upgrade' 2024-08-17T17:15:42.414 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:15:43.578 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap0 2024-08-17T17:15:43.591 INFO:tasks.workunit.client.0.smithi136.stdout:clone image177 -> image180 2024-08-17T17:15:43.685 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 66940460~3561311 2024-08-17T17:15:43.724 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 37812453~2856655 2024-08-17T17:15:43.742 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 63604793~7920462 2024-08-17T17:15:43.783 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 68767057~5004991 2024-08-17T17:15:43.785 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 25195961~211086 2024-08-17T17:15:43.791 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 4547877~8083298 2024-08-17T17:15:43.805 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 70061820~7645341 2024-08-17T17:15:43.843 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 53353272~2280198 2024-08-17T17:15:43.843 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:43 smithi136 ceph-mon[31034]: osdmap e326: 8 total, 8 up, 8 in 2024-08-17T17:15:43.844 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:43 smithi136 ceph-mon[31034]: pgmap v207: 137 pgs: 137 active+clean; 1.3 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.6 MiB/s rd, 17 MiB/s wr, 549 op/s 2024-08-17T17:15:43.844 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:43 smithi136 ceph-mon[42580]: osdmap e326: 8 total, 8 up, 8 in 2024-08-17T17:15:43.844 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:43 smithi136 ceph-mon[42580]: pgmap v207: 137 pgs: 137 active+clean; 1.3 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.6 MiB/s rd, 17 MiB/s wr, 549 op/s 2024-08-17T17:15:43.857 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:15:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:43 smithi165 ceph-mon[34868]: osdmap e326: 8 total, 8 up, 8 in 2024-08-17T17:15:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:43 smithi165 ceph-mon[34868]: pgmap v207: 137 pgs: 137 active+clean; 1.3 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.6 MiB/s rd, 17 MiB/s wr, 549 op/s 2024-08-17T17:15:44.269 INFO:teuthology.orchestra.run.smithi136.stdout:wait for minority of mons to upgrade 2024-08-17T17:15:44.890 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:44 smithi136 ceph-mon[31034]: osdmap e327: 8 total, 8 up, 8 in 2024-08-17T17:15:44.890 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:44 smithi136 ceph-mon[42580]: osdmap e327: 8 total, 8 up, 8 in 2024-08-17T17:15:44.913 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --limit 1' 2024-08-17T17:15:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:44 smithi165 ceph-mon[34868]: osdmap e327: 8 total, 8 up, 8 in 2024-08-17T17:15:45.115 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:15:45.593 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap1 2024-08-17T17:15:45.607 INFO:tasks.workunit.client.0.smithi136.stdout:clone image180 -> image182 2024-08-17T17:15:45.729 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 130241359 2024-08-17T17:15:45.739 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 41162096~5015275 2024-08-17T17:15:45.796 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 126253189~3988170 2024-08-17T17:15:45.796 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 70563910~785551 2024-08-17T17:15:45.844 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 38787740~2054915 2024-08-17T17:15:45.846 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 104656605~1507490 2024-08-17T17:15:45.859 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 21793434~6905856 2024-08-17T17:15:45.867 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 14690190~4449991 2024-08-17T17:15:45.894 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 127073803~1663615 2024-08-17T17:15:45.894 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap2 2024-08-17T17:15:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:45 smithi136 ceph-mon[31034]: osdmap e328: 8 total, 8 up, 8 in 2024-08-17T17:15:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:45 smithi136 ceph-mon[31034]: pgmap v210: 137 pgs: 137 active+clean; 1.3 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.6 MiB/s rd, 17 MiB/s wr, 551 op/s 2024-08-17T17:15:45.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:45 smithi136 ceph-mon[42580]: osdmap e328: 8 total, 8 up, 8 in 2024-08-17T17:15:45.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:45 smithi136 ceph-mon[42580]: pgmap v210: 137 pgs: 137 active+clean; 1.3 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.6 MiB/s rd, 17 MiB/s wr, 551 op/s 2024-08-17T17:15:46.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:45 smithi165 ceph-mon[34868]: osdmap e328: 8 total, 8 up, 8 in 2024-08-17T17:15:46.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:45 smithi165 ceph-mon[34868]: pgmap v210: 137 pgs: 137 active+clean; 1.3 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.6 MiB/s rd, 17 MiB/s wr, 551 op/s 2024-08-17T17:15:46.671 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:46 smithi136 ceph-mon[31034]: osdmap e329: 8 total, 8 up, 8 in 2024-08-17T17:15:46.671 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:46 smithi136 ceph-mon[42580]: osdmap e329: 8 total, 8 up, 8 in 2024-08-17T17:15:47.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:46 smithi165 ceph-mon[34868]: osdmap e329: 8 total, 8 up, 8 in 2024-08-17T17:15:47.603 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap2 2024-08-17T17:15:47.612 INFO:tasks.workunit.client.0.smithi136.stdout:clone image182 -> image184 2024-08-17T17:15:47.755 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 109516075~5324396 2024-08-17T17:15:47.791 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 92781112~6759239 2024-08-17T17:15:47.797 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 81520609~7622191 2024-08-17T17:15:47.831 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 5291168~4611267 2024-08-17T17:15:47.839 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 48704191~5198706 2024-08-17T17:15:47.900 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 67106767~926401 2024-08-17T17:15:47.940 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 120766968~3656523 2024-08-17T17:15:47.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:15:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:15:47] "GET /metrics HTTP/1.1" 200 38744 "" "Prometheus/2.51.0" 2024-08-17T17:15:47.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:47 smithi136 ceph-mon[31034]: osdmap e330: 8 total, 8 up, 8 in 2024-08-17T17:15:47.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:47 smithi136 ceph-mon[31034]: pgmap v213: 137 pgs: 137 active+clean; 1.4 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 28 MiB/s rd, 24 MiB/s wr, 1.05k op/s 2024-08-17T17:15:47.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:47 smithi136 ceph-mon[31034]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453", "daemon_types": "mon", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:47.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:47 smithi136 ceph-mon[42580]: osdmap e330: 8 total, 8 up, 8 in 2024-08-17T17:15:47.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:47 smithi136 ceph-mon[42580]: pgmap v213: 137 pgs: 137 active+clean; 1.4 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 28 MiB/s rd, 24 MiB/s wr, 1.05k op/s 2024-08-17T17:15:47.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:47 smithi136 ceph-mon[42580]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453", "daemon_types": "mon", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:47.964 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 9005842~5808548 2024-08-17T17:15:47.988 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap3 2024-08-17T17:15:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:47 smithi165 ceph-mon[34868]: osdmap e330: 8 total, 8 up, 8 in 2024-08-17T17:15:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:47 smithi165 ceph-mon[34868]: pgmap v213: 137 pgs: 137 active+clean; 1.4 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 28 MiB/s rd, 24 MiB/s wr, 1.05k op/s 2024-08-17T17:15:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:47 smithi165 ceph-mon[34868]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453", "daemon_types": "mon", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:48.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:48 smithi136 ceph-mon[31034]: osdmap e331: 8 total, 8 up, 8 in 2024-08-17T17:15:48.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:48 smithi136 ceph-mon[42580]: osdmap e331: 8 total, 8 up, 8 in 2024-08-17T17:15:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:48 smithi165 ceph-mon[34868]: osdmap e331: 8 total, 8 up, 8 in 2024-08-17T17:15:49.173 INFO:teuthology.orchestra.run.smithi136.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:15:49.612 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap3 2024-08-17T17:15:49.621 INFO:tasks.workunit.client.0.smithi136.stdout:clone image184 -> image186 2024-08-17T17:15:49.794 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:15:49.836 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:49 smithi136 ceph-mon[42580]: osdmap e332: 8 total, 8 up, 8 in 2024-08-17T17:15:49.836 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:49 smithi136 ceph-mon[42580]: pgmap v216: 137 pgs: 137 active+clean; 1.4 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 28 MiB/s rd, 24 MiB/s wr, 1.05k op/s 2024-08-17T17:15:49.836 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:49 smithi136 ceph-mon[42580]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:15:49.836 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:49 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:49.836 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:49 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:15:49.836 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:49 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:15:49.837 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:49 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:15:49.837 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:49 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:49.837 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:49 smithi136 ceph-mon[42580]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:15:49.837 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:49 smithi136 ceph-mon[31034]: osdmap e332: 8 total, 8 up, 8 in 2024-08-17T17:15:49.837 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:49 smithi136 ceph-mon[31034]: pgmap v216: 137 pgs: 137 active+clean; 1.4 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 28 MiB/s rd, 24 MiB/s wr, 1.05k op/s 2024-08-17T17:15:49.837 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:49 smithi136 ceph-mon[31034]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:15:49.837 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:49 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:49.837 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:49 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:15:49.837 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:49 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:15:49.837 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:49 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:15:49.837 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:49 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:49.837 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:49 smithi136 ceph-mon[31034]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:15:49.861 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-08-17T17:15:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:49 smithi165 ceph-mon[34868]: osdmap e332: 8 total, 8 up, 8 in 2024-08-17T17:15:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:49 smithi165 ceph-mon[34868]: pgmap v216: 137 pgs: 137 active+clean; 1.4 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 28 MiB/s rd, 24 MiB/s wr, 1.05k op/s 2024-08-17T17:15:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:49 smithi165 ceph-mon[34868]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:15:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:49 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:49 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:15:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:49 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:15:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:49 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:15:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:49 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:50.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:49 smithi165 ceph-mon[34868]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:15:50.100 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:15:50.309 INFO:tasks.workunit.client.0.smithi136.stdout:write: a 46339034~7123371 2024-08-17T17:15:50.384 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 59887788~3874670 2024-08-17T17:15:50.421 INFO:tasks.workunit.client.0.smithi136.stdout:write: a 53648845~4359395 2024-08-17T17:15:50.450 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 84306858~7437821 2024-08-17T17:15:50.478 INFO:tasks.workunit.client.0.smithi136.stdout:write: a 11770196~740763 2024-08-17T17:15:50.498 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 45466150~740182 2024-08-17T17:15:50.529 INFO:tasks.workunit.client.0.smithi136.stdout:write: a 53781006~7260981 2024-08-17T17:15:50.558 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 84669583~6548962 2024-08-17T17:15:50.561 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: user0 2024-08-17T17:15:50.760 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:50 smithi136 ceph-mon[31034]: osdmap e333: 8 total, 8 up, 8 in 2024-08-17T17:15:50.761 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:50 smithi136 ceph-mon[42580]: osdmap e333: 8 total, 8 up, 8 in 2024-08-17T17:15:50.912 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:50 smithi165 ceph-mon[34868]: osdmap e333: 8 total, 8 up, 8 in 2024-08-17T17:15:51.627 INFO:tasks.workunit.client.0.smithi136.stdout:write: b 35948833~5782545 2024-08-17T17:15:51.645 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 98500293~7641771 2024-08-17T17:15:51.645 INFO:tasks.workunit.client.0.smithi136.stdout:write: b 104498260~5360219 2024-08-17T17:15:51.753 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 48172902~506878 2024-08-17T17:15:51.753 INFO:tasks.workunit.client.0.smithi136.stdout:write: b 42235876~2153969 2024-08-17T17:15:51.754 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 37197376~109303 2024-08-17T17:15:51.754 INFO:tasks.workunit.client.0.smithi136.stdout:write: b 113575957~7864354 2024-08-17T17:15:51.763 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 15311438~144818 2024-08-17T17:15:51.764 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: user1 2024-08-17T17:15:51.831 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:51 smithi136 ceph-mon[31034]: osdmap e334: 8 total, 8 up, 8 in 2024-08-17T17:15:51.831 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:51 smithi136 ceph-mon[31034]: pgmap v219: 137 pgs: 137 active+clean; 1.4 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-08-17T17:15:51.831 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:51 smithi136 ceph-mon[42580]: osdmap e334: 8 total, 8 up, 8 in 2024-08-17T17:15:51.831 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:51 smithi136 ceph-mon[42580]: pgmap v219: 137 pgs: 137 active+clean; 1.4 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-08-17T17:15:51.935 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:51 smithi165 ceph-mon[34868]: osdmap e334: 8 total, 8 up, 8 in 2024-08-17T17:15:51.936 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:51 smithi165 ceph-mon[34868]: pgmap v219: 137 pgs: 137 active+clean; 1.4 GiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-08-17T17:15:52.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:52 smithi136 ceph-mon[31034]: osdmap e335: 8 total, 8 up, 8 in 2024-08-17T17:15:52.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:52 smithi136 ceph-mon[31034]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:52.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:52 smithi136 ceph-mon[42580]: osdmap e335: 8 total, 8 up, 8 in 2024-08-17T17:15:52.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:52 smithi136 ceph-mon[42580]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:52 smithi165 ceph-mon[34868]: osdmap e335: 8 total, 8 up, 8 in 2024-08-17T17:15:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:52 smithi165 ceph-mon[34868]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:53.640 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 179652468 2024-08-17T17:15:53.654 INFO:tasks.workunit.client.0.smithi136.stdout:write: c 112952004~7305856 2024-08-17T17:15:53.666 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 119588954~7824892 2024-08-17T17:15:53.696 INFO:teuthology.orchestra.run.smithi136.stdout:true 2024-08-17T17:15:53.696 INFO:teuthology.orchestra.run.smithi136.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-17T17:15:53.696 INFO:teuthology.orchestra.run.smithi136.stdout:alertmanager.a smithi136 *:9093,9094 running (61s) 57s ago 6m 17.5M - 0.27.0 11f11916f8cd c3b9b5b223ab 2024-08-17T17:15:53.696 INFO:teuthology.orchestra.run.smithi136.stdout:grafana.a smithi165 *:3000 running (38s) 30s ago 6m 41.8M - c8b91775d855 04db24fce5e9 2024-08-17T17:15:53.696 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi136.ltsuqd smithi136 running (5m) 57s ago 5m 19.4M - 18.2.4-719-gced649f8 c870753d67c2 faaa06f2950c 2024-08-17T17:15:53.696 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi165.bmrbhh smithi165 running (5m) 30s ago 5m 16.4M - 18.2.4-719-gced649f8 c870753d67c2 c74b00d929cf 2024-08-17T17:15:53.696 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.x smithi165 *:8443,9283,8765 running (2m) 30s ago 10m 452M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 837fd48ff7cb 2024-08-17T17:15:53.697 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.y smithi136 *:8443,9283,8765 running (2m) 57s ago 12m 514M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9acdc6775e48 2024-08-17T17:15:53.697 INFO:teuthology.orchestra.run.smithi136.stdout:mon.a smithi136 running (12m) 57s ago 12m 59.8M 2048M 18.2.4-719-gced649f8 c870753d67c2 cc4e5e5ff756 2024-08-17T17:15:53.697 INFO:teuthology.orchestra.run.smithi136.stdout:mon.b smithi165 running (11m) 30s ago 11m 55.8M 2048M 18.2.4-719-gced649f8 c870753d67c2 d587e16140e1 2024-08-17T17:15:53.697 INFO:teuthology.orchestra.run.smithi136.stdout:mon.c smithi136 running (11m) 57s ago 11m 52.6M 2048M 18.2.4-719-gced649f8 c870753d67c2 b4e0f3c9e485 2024-08-17T17:15:53.697 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.a smithi136 *:9100 running (113s) 57s ago 6m 9625k - 1.7.0 72c9c2088986 1e4c644ae533 2024-08-17T17:15:53.697 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.b smithi165 *:9100 running (107s) 30s ago 6m 10.5M - 1.7.0 72c9c2088986 7af2d2d68b06 2024-08-17T17:15:53.697 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0 smithi136 running (10m) 57s ago 10m 127M 2480M 18.2.4-719-gced649f8 c870753d67c2 faab916734b0 2024-08-17T17:15:53.697 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1 smithi136 running (10m) 57s ago 10m 110M 2480M 18.2.4-719-gced649f8 c870753d67c2 861686b6ff82 2024-08-17T17:15:53.698 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2 smithi136 running (9m) 57s ago 9m 111M 2480M 18.2.4-719-gced649f8 c870753d67c2 640be4af4156 2024-08-17T17:15:53.698 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3 smithi136 running (9m) 57s ago 9m 105M 2480M 18.2.4-719-gced649f8 c870753d67c2 14b57a745d30 2024-08-17T17:15:53.698 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4 smithi165 running (8m) 30s ago 8m 126M 2480M 18.2.4-719-gced649f8 c870753d67c2 df366c73e1f2 2024-08-17T17:15:53.698 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5 smithi165 running (8m) 30s ago 8m 92.1M 2480M 18.2.4-719-gced649f8 c870753d67c2 1131665e203a 2024-08-17T17:15:53.698 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6 smithi165 running (8m) 30s ago 8m 133M 2480M 18.2.4-719-gced649f8 c870753d67c2 7bc5fd925293 2024-08-17T17:15:53.698 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7 smithi165 running (7m) 30s ago 7m 102M 2480M 18.2.4-719-gced649f8 c870753d67c2 18382e4ae3b8 2024-08-17T17:15:53.698 INFO:teuthology.orchestra.run.smithi136.stdout:prometheus.a smithi165 *:9095 running (81s) 30s ago 7m 37.1M - 2.51.0 1d3b7f56885b ae8f8b69fe2f 2024-08-17T17:15:53.698 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:15:53.698 INFO:teuthology.orchestra.run.smithi136.stdout: "mon": { 2024-08-17T17:15:53.698 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 3 2024-08-17T17:15:53.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:15:53.698 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr": { 2024-08-17T17:15:53.699 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 2 2024-08-17T17:15:53.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:15:53.699 INFO:teuthology.orchestra.run.smithi136.stdout: "osd": { 2024-08-17T17:15:53.699 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 8 2024-08-17T17:15:53.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:15:53.699 INFO:teuthology.orchestra.run.smithi136.stdout: "mds": { 2024-08-17T17:15:53.699 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 2 2024-08-17T17:15:53.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:15:53.699 INFO:teuthology.orchestra.run.smithi136.stdout: "overall": { 2024-08-17T17:15:53.699 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 13, 2024-08-17T17:15:53.699 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 2 2024-08-17T17:15:53.699 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:15:53.699 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:15:53.700 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:15:53.700 INFO:teuthology.orchestra.run.smithi136.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d", 2024-08-17T17:15:53.700 INFO:teuthology.orchestra.run.smithi136.stdout: "in_progress": true, 2024-08-17T17:15:53.700 INFO:teuthology.orchestra.run.smithi136.stdout: "which": "Upgrading daemons of type(s) mon. Upgrade limited to 1 daemons (1 remaining).", 2024-08-17T17:15:53.700 INFO:teuthology.orchestra.run.smithi136.stdout: "services_complete": [], 2024-08-17T17:15:53.700 INFO:teuthology.orchestra.run.smithi136.stdout: "progress": "0/3 daemons upgraded", 2024-08-17T17:15:53.700 INFO:teuthology.orchestra.run.smithi136.stdout: "message": "Currently upgrading mon daemons", 2024-08-17T17:15:53.700 INFO:teuthology.orchestra.run.smithi136.stdout: "is_paused": false 2024-08-17T17:15:53.700 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:15:53.717 INFO:tasks.workunit.client.0.smithi136.stdout:write: c 78457001~5322147 2024-08-17T17:15:53.719 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 50102273~78209 2024-08-17T17:15:53.719 INFO:tasks.workunit.client.0.smithi136.stdout:write: c 145257423~7410095 2024-08-17T17:15:53.729 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 121412327~2103963 2024-08-17T17:15:53.730 INFO:tasks.workunit.client.0.smithi136.stdout:write: c 72227597~784683 2024-08-17T17:15:53.732 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 18944194~4814612 2024-08-17T17:15:53.732 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: user2 2024-08-17T17:15:53.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:53 smithi136 ceph-mon[31034]: osdmap e336: 8 total, 8 up, 8 in 2024-08-17T17:15:53.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:53 smithi136 ceph-mon[31034]: pgmap v222: 137 pgs: 137 active+clean; 1.5 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 42 MiB/s rd, 44 MiB/s wr, 949 op/s 2024-08-17T17:15:53.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:53 smithi136 ceph-mon[31034]: from='client.15228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:53.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:53 smithi136 ceph-mon[31034]: from='client.15234 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:53.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:53 smithi136 ceph-mon[31034]: from='client.? 172.21.15.136:0/4088775889' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:53.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:53 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:53.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:53 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:15:53.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:53 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:53.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:53 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:53.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:53 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-08-17T17:15:53.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:53 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-08-17T17:15:53.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:53 smithi136 ceph-mon[42580]: osdmap e336: 8 total, 8 up, 8 in 2024-08-17T17:15:53.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:53 smithi136 ceph-mon[42580]: pgmap v222: 137 pgs: 137 active+clean; 1.5 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 42 MiB/s rd, 44 MiB/s wr, 949 op/s 2024-08-17T17:15:53.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:53 smithi136 ceph-mon[42580]: from='client.15228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:53.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:53 smithi136 ceph-mon[42580]: from='client.15234 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:53.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:53 smithi136 ceph-mon[42580]: from='client.? 172.21.15.136:0/4088775889' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:53.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:53 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:53.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:53 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:15:53.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:53 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:53.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:53 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:53.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:53 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-08-17T17:15:53.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:53 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-08-17T17:15:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:53 smithi165 ceph-mon[34868]: osdmap e336: 8 total, 8 up, 8 in 2024-08-17T17:15:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:53 smithi165 ceph-mon[34868]: pgmap v222: 137 pgs: 137 active+clean; 1.5 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 42 MiB/s rd, 44 MiB/s wr, 949 op/s 2024-08-17T17:15:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:53 smithi165 ceph-mon[34868]: from='client.15228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:53 smithi165 ceph-mon[34868]: from='client.15234 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:54.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:53 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/4088775889' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:54.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:53 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:54.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:53 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:15:54.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:53 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:15:54.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:53 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:54.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:53 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-08-17T17:15:54.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:53 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-08-17T17:15:54.903 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:54 smithi136 ceph-mon[31034]: Upgrade: Target is version 19.0.0-5836-gcd8bdbeb (squid) 2024-08-17T17:15:54.904 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:54 smithi136 ceph-mon[31034]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d'] 2024-08-17T17:15:54.904 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:54 smithi136 ceph-mon[31034]: Upgrade: Setting container_image for all mgr 2024-08-17T17:15:54.904 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:54 smithi136 ceph-mon[31034]: Upgrade: It appears safe to stop mon.a 2024-08-17T17:15:54.904 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:54 smithi136 ceph-mon[31034]: osdmap e337: 8 total, 8 up, 8 in 2024-08-17T17:15:54.904 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:54 smithi136 ceph-mon[31034]: from='client.25151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:54.904 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:54 smithi136 ceph-mon[42580]: Upgrade: Target is version 19.0.0-5836-gcd8bdbeb (squid) 2024-08-17T17:15:54.904 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:54 smithi136 ceph-mon[42580]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d'] 2024-08-17T17:15:54.904 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:54 smithi136 ceph-mon[42580]: Upgrade: Setting container_image for all mgr 2024-08-17T17:15:54.904 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:54 smithi136 ceph-mon[42580]: Upgrade: It appears safe to stop mon.a 2024-08-17T17:15:54.904 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:54 smithi136 ceph-mon[42580]: osdmap e337: 8 total, 8 up, 8 in 2024-08-17T17:15:54.905 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:54 smithi136 ceph-mon[42580]: from='client.25151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:54 smithi165 ceph-mon[34868]: Upgrade: Target is version 19.0.0-5836-gcd8bdbeb (squid) 2024-08-17T17:15:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:54 smithi165 ceph-mon[34868]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d'] 2024-08-17T17:15:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:54 smithi165 ceph-mon[34868]: Upgrade: Setting container_image for all mgr 2024-08-17T17:15:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:54 smithi165 ceph-mon[34868]: Upgrade: It appears safe to stop mon.a 2024-08-17T17:15:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:54 smithi165 ceph-mon[34868]: osdmap e337: 8 total, 8 up, 8 in 2024-08-17T17:15:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:54 smithi165 ceph-mon[34868]: from='client.25151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:55.652 INFO:tasks.workunit.client.0.smithi136.stdout:write: d 158642769~1786223 2024-08-17T17:15:55.660 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 150616262~7973864 2024-08-17T17:15:55.660 INFO:tasks.workunit.client.0.smithi136.stdout:write: d 169632987~689262 2024-08-17T17:15:55.661 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 156628601~4584889 2024-08-17T17:15:55.672 INFO:tasks.workunit.client.0.smithi136.stdout:write: d 149721645~5509558 2024-08-17T17:15:55.674 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 69305407~2244135 2024-08-17T17:15:55.677 INFO:tasks.workunit.client.0.smithi136.stdout:write: d 164143890~5089150 2024-08-17T17:15:55.680 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 164881102~7373268 2024-08-17T17:15:55.721 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: user3 2024-08-17T17:15:55.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:55 smithi136 ceph-mon[31034]: pgmap v224: 137 pgs: 137 active+clean; 1.5 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 42 MiB/s rd, 44 MiB/s wr, 945 op/s 2024-08-17T17:15:55.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:55 smithi136 ceph-mon[31034]: osdmap e338: 8 total, 8 up, 8 in 2024-08-17T17:15:55.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:55 smithi136 ceph-mon[31034]: Upgrade: Updating mon.a 2024-08-17T17:15:55.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:55 smithi136 ceph-mon[31034]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:55.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:55 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:15:55.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:55 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-17T17:15:55.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:55 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:15:55.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:55 smithi136 ceph-mon[31034]: Deploying daemon mon.a on smithi136 2024-08-17T17:15:55.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:55 smithi136 ceph-mon[31034]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:15:55.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:55 smithi136 ceph-mon[42580]: pgmap v224: 137 pgs: 137 active+clean; 1.5 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 42 MiB/s rd, 44 MiB/s wr, 945 op/s 2024-08-17T17:15:55.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:55 smithi136 ceph-mon[42580]: osdmap e338: 8 total, 8 up, 8 in 2024-08-17T17:15:55.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:55 smithi136 ceph-mon[42580]: Upgrade: Updating mon.a 2024-08-17T17:15:55.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:55 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:55.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:55 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:15:55.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:55 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-17T17:15:55.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:55 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:15:55.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:55 smithi136 ceph-mon[42580]: Deploying daemon mon.a on smithi136 2024-08-17T17:15:55.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:55 smithi136 ceph-mon[42580]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:15:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:55 smithi165 ceph-mon[34868]: pgmap v224: 137 pgs: 137 active+clean; 1.5 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 42 MiB/s rd, 44 MiB/s wr, 945 op/s 2024-08-17T17:15:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:55 smithi165 ceph-mon[34868]: osdmap e338: 8 total, 8 up, 8 in 2024-08-17T17:15:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:55 smithi165 ceph-mon[34868]: Upgrade: Updating mon.a 2024-08-17T17:15:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:55 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='mgr.y' 2024-08-17T17:15:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:55 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:15:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:55 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-17T17:15:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:55 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:15:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:55 smithi165 ceph-mon[34868]: Deploying daemon mon.a on smithi136 2024-08-17T17:15:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:55 smithi165 ceph-mon[34868]: from='mgr.15039 172.21.15.136:0/705148310' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:15:56.877 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:56 smithi136 systemd[1]: Stopping Ceph mon.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:15:56.877 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:56 smithi136 ceph-mon[31034]: osdmap e339: 8 total, 8 up, 8 in 2024-08-17T17:15:56.878 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:15:56 smithi136 ceph-mon[42580]: osdmap e339: 8 total, 8 up, 8 in 2024-08-17T17:15:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:15:56 smithi165 ceph-mon[34868]: osdmap e339: 8 total, 8 up, 8 in 2024-08-17T17:15:57.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:56 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a[31011]: 2024-08-17T17:15:56.874+0000 7f1929813640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:15:57.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:56 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a[31011]: 2024-08-17T17:15:56.874+0000 7f1929813640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-08-17T17:15:57.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:57 smithi136 podman[109673]: 2024-08-17 17:15:57.039780947 +0000 UTC m=+0.267429040 container died cc4e5e5ff75686490f40ecc7fb86f646c8220efc2d2968ba474c1a497629a5c2 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, RELEASE=reef-ced649f, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:15:57.676 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:15:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:15:57] "GET /metrics HTTP/1.1" 200 38740 "" "Prometheus/2.51.0" 2024-08-17T17:15:57.677 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:57 smithi136 podman[109673]: 2024-08-17 17:15:57.432821586 +0000 UTC m=+0.660469727 container cleanup cc4e5e5ff75686490f40ecc7fb86f646c8220efc2d2968ba474c1a497629a5c2 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=reef-ced649f, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-08-17T17:15:57.677 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:57 smithi136 bash[109673]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a 2024-08-17T17:15:57.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:57 smithi136 podman[109692]: 2024-08-17 17:15:57.675963119 +0000 UTC m=+0.628616146 container remove cc4e5e5ff75686490f40ecc7fb86f646c8220efc2d2968ba474c1a497629a5c2 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a, GIT_CLEAN=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, RELEASE=reef-ced649f) 2024-08-17T17:15:57.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:57 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@mon.a.service: Deactivated successfully. 2024-08-17T17:15:57.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:57 smithi136 systemd[1]: Stopped Ceph mon.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:15:57.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:57 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@mon.a.service: Consumed 16.840s CPU time. 2024-08-17T17:15:58.408 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 systemd[1]: Starting Ceph mon.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:15:58.408 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 podman[109823]: 2024-08-17 17:15:58.157538817 +0000 UTC m=+0.016642013 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:15:58.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 podman[109823]: 2024-08-17 17:15:58.407978877 +0000 UTC m=+0.267082102 container create 9cf82e613626d40f8c2d9c7046718ba80ebe1693cad532de3387d5e9a487fed5 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=main-cd8bdbe, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, ceph=True) 2024-08-17T17:15:59.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 podman[109823]: 2024-08-17 17:15:58.697060084 +0000 UTC m=+0.556163268 container init 9cf82e613626d40f8c2d9c7046718ba80ebe1693cad532de3387d5e9a487fed5 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-cd8bdbe) 2024-08-17T17:15:59.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 podman[109823]: 2024-08-17 17:15:58.70243531 +0000 UTC m=+0.561538496 container start 9cf82e613626d40f8c2d9c7046718ba80ebe1693cad532de3387d5e9a487fed5 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a, org.label-schema.vendor=CentOS, RELEASE=main-cd8bdbe, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-08-17T17:15:59.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: set uid:gid to 167:167 (ceph:ceph) 2024-08-17T17:15:59.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev), process ceph-mon, pid 7 2024-08-17T17:15:59.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: pidfile_write: ignore empty --pid-file 2024-08-17T17:15:59.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: load: jerasure load: lrc 2024-08-17T17:15:59.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: RocksDB version: 7.9.2 2024-08-17T17:15:59.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Git sha 0 2024-08-17T17:15:59.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Compile date 2024-08-15 20:43:31 2024-08-17T17:15:59.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: DB SUMMARY 2024-08-17T17:15:59.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: DB Session ID: BHIZ3YEZEWSW14OUPKDK 2024-08-17T17:15:59.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: CURRENT file: CURRENT 2024-08-17T17:15:59.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: IDENTITY file: IDENTITY 2024-08-17T17:15:59.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1520 Bytes 2024-08-17T17:15:59.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000032.sst 2024-08-17T17:15:59.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000030.log size: 6162040 ; 2024-08-17T17:15:59.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.error_if_exists: 0 2024-08-17T17:15:59.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.create_if_missing: 0 2024-08-17T17:15:59.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.paranoid_checks: 1 2024-08-17T17:15:59.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.flush_verify_memtable_count: 1 2024-08-17T17:15:59.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-08-17T17:15:59.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-08-17T17:15:59.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.env: 0x55a5f6353be0 2024-08-17T17:15:59.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.fs: PosixFileSystem 2024-08-17T17:15:59.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.info_log: 0x55a5f853f700 2024-08-17T17:15:59.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_file_opening_threads: 16 2024-08-17T17:15:59.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.statistics: (nil) 2024-08-17T17:15:59.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.use_fsync: 0 2024-08-17T17:15:59.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_log_file_size: 0 2024-08-17T17:15:59.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-08-17T17:15:59.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.log_file_time_to_roll: 0 2024-08-17T17:15:59.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.keep_log_file_num: 1000 2024-08-17T17:15:59.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.recycle_log_file_num: 0 2024-08-17T17:15:59.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.allow_fallocate: 1 2024-08-17T17:15:59.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.allow_mmap_reads: 0 2024-08-17T17:15:59.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.allow_mmap_writes: 0 2024-08-17T17:15:59.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.use_direct_reads: 0 2024-08-17T17:15:59.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-08-17T17:15:59.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.create_missing_column_families: 0 2024-08-17T17:15:59.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.db_log_dir: 2024-08-17T17:15:59.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.wal_dir: 2024-08-17T17:15:59.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.table_cache_numshardbits: 6 2024-08-17T17:15:59.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.WAL_ttl_seconds: 0 2024-08-17T17:15:59.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.WAL_size_limit_MB: 0 2024-08-17T17:15:59.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-08-17T17:15:59.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-08-17T17:15:59.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.is_fd_close_on_exec: 1 2024-08-17T17:15:59.202 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.advise_random_on_open: 1 2024-08-17T17:15:59.202 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.db_write_buffer_size: 0 2024-08-17T17:15:59.202 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.write_buffer_manager: 0x55a5f8542960 2024-08-17T17:15:59.202 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-08-17T17:15:59.202 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-08-17T17:15:59.202 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.use_adaptive_mutex: 0 2024-08-17T17:15:59.202 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.rate_limiter: (nil) 2024-08-17T17:15:59.202 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-08-17T17:15:59.202 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.wal_recovery_mode: 2 2024-08-17T17:15:59.202 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.enable_thread_tracking: 0 2024-08-17T17:15:59.202 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.enable_pipelined_write: 0 2024-08-17T17:15:59.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.unordered_write: 0 2024-08-17T17:15:59.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-08-17T17:15:59.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-08-17T17:15:59.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-08-17T17:15:59.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-08-17T17:15:59.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.row_cache: None 2024-08-17T17:15:59.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.wal_filter: None 2024-08-17T17:15:59.204 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-08-17T17:15:59.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.allow_ingest_behind: 0 2024-08-17T17:15:59.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.two_write_queues: 0 2024-08-17T17:15:59.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.manual_wal_flush: 0 2024-08-17T17:15:59.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.wal_compression: 0 2024-08-17T17:15:59.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.atomic_flush: 0 2024-08-17T17:15:59.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-08-17T17:15:59.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.persist_stats_to_disk: 0 2024-08-17T17:15:59.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.write_dbid_to_manifest: 0 2024-08-17T17:15:59.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.log_readahead_size: 0 2024-08-17T17:15:59.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-08-17T17:15:59.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.best_efforts_recovery: 0 2024-08-17T17:15:59.206 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-08-17T17:15:59.206 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-08-17T17:15:59.206 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.allow_data_in_errors: 0 2024-08-17T17:15:59.206 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.db_host_id: __hostname__ 2024-08-17T17:15:59.206 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.enforce_single_del_contracts: true 2024-08-17T17:15:59.206 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_background_jobs: 2 2024-08-17T17:15:59.206 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_background_compactions: -1 2024-08-17T17:15:59.206 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_subcompactions: 1 2024-08-17T17:15:59.206 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-08-17T17:15:59.206 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-08-17T17:15:59.207 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.delayed_write_rate : 16777216 2024-08-17T17:15:59.207 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_total_wal_size: 0 2024-08-17T17:15:59.207 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-08-17T17:15:59.207 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.stats_dump_period_sec: 600 2024-08-17T17:15:59.207 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.stats_persist_period_sec: 600 2024-08-17T17:15:59.207 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-08-17T17:15:59.207 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_open_files: -1 2024-08-17T17:15:59.207 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.bytes_per_sync: 0 2024-08-17T17:15:59.207 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.wal_bytes_per_sync: 0 2024-08-17T17:15:59.207 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.strict_bytes_per_sync: 0 2024-08-17T17:15:59.208 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.compaction_readahead_size: 0 2024-08-17T17:15:59.208 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_background_flushes: -1 2024-08-17T17:15:59.208 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Compression algorithms supported: 2024-08-17T17:15:59.208 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: kZSTD supported: 0 2024-08-17T17:15:59.208 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: kXpressCompression supported: 0 2024-08-17T17:15:59.208 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: kBZip2Compression supported: 0 2024-08-17T17:15:59.208 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-08-17T17:15:59.208 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: kLZ4Compression supported: 1 2024-08-17T17:15:59.208 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: kZlibCompression supported: 1 2024-08-17T17:15:59.208 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: kLZ4HCCompression supported: 1 2024-08-17T17:15:59.208 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: kSnappyCompression supported: 1 2024-08-17T17:15:59.209 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Fast CRC32 supported: Supported on x86 2024-08-17T17:15:59.209 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: DMutex implementation: pthread_mutex_t 2024-08-17T17:15:59.209 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-08-17T17:15:59.209 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-08-17T17:15:59.209 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-08-17T17:15:59.209 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.merge_operator: 2024-08-17T17:15:59.209 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.compaction_filter: None 2024-08-17T17:15:59.209 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.compaction_filter_factory: None 2024-08-17T17:15:59.209 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.sst_partitioner_factory: None 2024-08-17T17:15:59.209 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.memtable_factory: SkipListFactory 2024-08-17T17:15:59.209 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.table_factory: BlockBasedTable 2024-08-17T17:15:59.210 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a5f853f380) 2024-08-17T17:15:59.210 INFO:journalctl@ceph.mon.a.smithi136.stdout: cache_index_and_filter_blocks: 1 2024-08-17T17:15:59.210 INFO:journalctl@ceph.mon.a.smithi136.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-08-17T17:15:59.210 INFO:journalctl@ceph.mon.a.smithi136.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-08-17T17:15:59.210 INFO:journalctl@ceph.mon.a.smithi136.stdout: pin_top_level_index_and_filter: 1 2024-08-17T17:15:59.210 INFO:journalctl@ceph.mon.a.smithi136.stdout: index_type: 0 2024-08-17T17:15:59.210 INFO:journalctl@ceph.mon.a.smithi136.stdout: data_block_index_type: 0 2024-08-17T17:15:59.210 INFO:journalctl@ceph.mon.a.smithi136.stdout: index_shortening: 1 2024-08-17T17:15:59.210 INFO:journalctl@ceph.mon.a.smithi136.stdout: data_block_hash_table_util_ratio: 0.750000 2024-08-17T17:15:59.210 INFO:journalctl@ceph.mon.a.smithi136.stdout: checksum: 4 2024-08-17T17:15:59.210 INFO:journalctl@ceph.mon.a.smithi136.stdout: no_block_cache: 0 2024-08-17T17:15:59.211 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_cache: 0x55a5f8533610 2024-08-17T17:15:59.211 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_cache_name: BinnedLRUCache 2024-08-17T17:15:59.211 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_cache_options: 2024-08-17T17:15:59.211 INFO:journalctl@ceph.mon.a.smithi136.stdout: capacity : 536870912 2024-08-17T17:15:59.211 INFO:journalctl@ceph.mon.a.smithi136.stdout: num_shard_bits : 4 2024-08-17T17:15:59.211 INFO:journalctl@ceph.mon.a.smithi136.stdout: strict_capacity_limit : 0 2024-08-17T17:15:59.211 INFO:journalctl@ceph.mon.a.smithi136.stdout: high_pri_pool_ratio: 0.000 2024-08-17T17:15:59.211 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_cache_compressed: (nil) 2024-08-17T17:15:59.211 INFO:journalctl@ceph.mon.a.smithi136.stdout: persistent_cache: (nil) 2024-08-17T17:15:59.211 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_size: 4096 2024-08-17T17:15:59.211 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_size_deviation: 10 2024-08-17T17:15:59.211 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_restart_interval: 16 2024-08-17T17:15:59.212 INFO:journalctl@ceph.mon.a.smithi136.stdout: index_block_restart_interval: 1 2024-08-17T17:15:59.212 INFO:journalctl@ceph.mon.a.smithi136.stdout: metadata_block_size: 4096 2024-08-17T17:15:59.212 INFO:journalctl@ceph.mon.a.smithi136.stdout: partition_filters: 0 2024-08-17T17:15:59.212 INFO:journalctl@ceph.mon.a.smithi136.stdout: use_delta_encoding: 1 2024-08-17T17:15:59.212 INFO:journalctl@ceph.mon.a.smithi136.stdout: filter_policy: bloomfilter 2024-08-17T17:15:59.212 INFO:journalctl@ceph.mon.a.smithi136.stdout: whole_key_filtering: 1 2024-08-17T17:15:59.212 INFO:journalctl@ceph.mon.a.smithi136.stdout: verify_compression: 0 2024-08-17T17:15:59.212 INFO:journalctl@ceph.mon.a.smithi136.stdout: read_amp_bytes_per_bit: 0 2024-08-17T17:15:59.212 INFO:journalctl@ceph.mon.a.smithi136.stdout: format_version: 5 2024-08-17T17:15:59.212 INFO:journalctl@ceph.mon.a.smithi136.stdout: enable_index_compression: 1 2024-08-17T17:15:59.212 INFO:journalctl@ceph.mon.a.smithi136.stdout: block_align: 0 2024-08-17T17:15:59.212 INFO:journalctl@ceph.mon.a.smithi136.stdout: max_auto_readahead_size: 262144 2024-08-17T17:15:59.213 INFO:journalctl@ceph.mon.a.smithi136.stdout: prepopulate_block_cache: 0 2024-08-17T17:15:59.213 INFO:journalctl@ceph.mon.a.smithi136.stdout: initial_auto_readahead_size: 8192 2024-08-17T17:15:59.213 INFO:journalctl@ceph.mon.a.smithi136.stdout: num_file_reads_for_auto_readahead: 2 2024-08-17T17:15:59.213 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.write_buffer_size: 33554432 2024-08-17T17:15:59.213 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_write_buffer_number: 2 2024-08-17T17:15:59.213 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.compression: NoCompression 2024-08-17T17:15:59.213 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.bottommost_compression: Disabled 2024-08-17T17:15:59.213 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.prefix_extractor: nullptr 2024-08-17T17:15:59.213 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-08-17T17:15:59.213 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.num_levels: 7 2024-08-17T17:15:59.213 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-08-17T17:15:59.214 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-08-17T17:15:59.214 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-08-17T17:15:59.214 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-08-17T17:15:59.214 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-08-17T17:15:59.214 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-08-17T17:15:59.214 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-08-17T17:15:59.214 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-08-17T17:15:59.214 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-08-17T17:15:59.214 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-08-17T17:15:59.214 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-08-17T17:15:59.214 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-08-17T17:15:59.214 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.compression_opts.window_bits: -14 2024-08-17T17:15:59.215 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.compression_opts.level: 32767 2024-08-17T17:15:59.215 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.compression_opts.strategy: 0 2024-08-17T17:15:59.215 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-08-17T17:15:59.215 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-08-17T17:15:59.215 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-08-17T17:15:59.215 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-08-17T17:15:59.215 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.compression_opts.enabled: false 2024-08-17T17:15:59.215 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-08-17T17:15:59.215 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-08-17T17:15:59.215 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-08-17T17:15:59.215 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-08-17T17:15:59.216 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.target_file_size_base: 67108864 2024-08-17T17:15:59.216 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.target_file_size_multiplier: 1 2024-08-17T17:15:59.216 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-08-17T17:15:59.216 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-08-17T17:15:59.216 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-08-17T17:15:59.216 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-08-17T17:15:59.216 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-08-17T17:15:59.216 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-08-17T17:15:59.216 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-08-17T17:15:59.216 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-08-17T17:15:59.216 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-08-17T17:15:59.216 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-08-17T17:15:59.217 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-08-17T17:15:59.217 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-08-17T17:15:59.217 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-08-17T17:15:59.217 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.arena_block_size: 1048576 2024-08-17T17:15:59.217 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-08-17T17:15:59.217 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-08-17T17:15:59.217 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.disable_auto_compactions: 0 2024-08-17T17:15:59.217 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-08-17T17:15:59.217 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-08-17T17:15:59.217 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-08-17T17:15:59.217 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-08-17T17:15:59.218 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-08-17T17:15:59.218 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-08-17T17:15:59.218 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-08-17T17:15:59.218 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-08-17T17:15:59.218 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-08-17T17:15:59.218 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-08-17T17:15:59.218 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-08-17T17:15:59.218 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.inplace_update_support: 0 2024-08-17T17:15:59.218 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.inplace_update_num_locks: 10000 2024-08-17T17:15:59.218 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-08-17T17:15:59.218 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-08-17T17:15:59.218 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.memtable_huge_page_size: 0 2024-08-17T17:15:59.219 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.bloom_locality: 0 2024-08-17T17:15:59.219 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.max_successive_merges: 0 2024-08-17T17:15:59.219 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.optimize_filters_for_hits: 0 2024-08-17T17:15:59.219 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.paranoid_file_checks: 0 2024-08-17T17:15:59.219 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.force_consistency_checks: 1 2024-08-17T17:15:59.219 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.report_bg_io_stats: 0 2024-08-17T17:15:59.219 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.ttl: 2592000 2024-08-17T17:15:59.219 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.periodic_compaction_seconds: 0 2024-08-17T17:15:59.219 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-08-17T17:15:59.219 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-08-17T17:15:59.219 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.enable_blob_files: false 2024-08-17T17:15:59.219 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.min_blob_size: 0 2024-08-17T17:15:59.220 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.blob_file_size: 268435456 2024-08-17T17:15:59.220 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.blob_compression_type: NoCompression 2024-08-17T17:15:59.220 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.enable_blob_garbage_collection: false 2024-08-17T17:15:59.220 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-08-17T17:15:59.220 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-08-17T17:15:59.220 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-08-17T17:15:59.220 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.blob_file_starting_level: 0 2024-08-17T17:15:59.220 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-08-17T17:15:59.220 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-08-17T17:15:59.220 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 34, last_sequence is 12781, log_number is 30,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 30 2024-08-17T17:15:59.220 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 30 2024-08-17T17:15:59.221 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 17e5137b-af42-4e49-a7e1-f98a5a53f78e 2024-08-17T17:15:59.221 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914958730256, "job": 1, "event": "recovery_started", "wal_files": [30]} 2024-08-17T17:15:59.221 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #30 mode 2 2024-08-17T17:15:59.221 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914958747842, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 35, "file_size": 5744038, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12782, "largest_seqno": 16914, "table_properties": {"data_size": 5724174, "index_size": 14017, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 4805, "raw_key_size": 43544, "raw_average_key_size": 22, "raw_value_size": 5683063, "raw_average_value_size": 2986, "num_data_blocks": 606, "num_entries": 1903, "num_filter_entries": 1903, "num_deletions": 269, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1723914958, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "17e5137b-af42-4e49-a7e1-f98a5a53f78e", "db_session_id": "BHIZ3YEZEWSW14OUPKDK", "orig_file_number": 35, "seqno_to_time_mapping": "N/A"}} 2024-08-17T17:15:59.221 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914958747949, "job": 1, "event": "recovery_finished"} 2024-08-17T17:15:59.221 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: [db/version_set.cc:5047] Creating manifest 37 2024-08-17T17:15:59.221 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-08-17T17:15:59.221 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000030.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-08-17T17:15:59.221 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a5f8562e00 2024-08-17T17:15:59.221 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: DB pointer 0x55a5f8660000 2024-08-17T17:15:59.221 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-08-17T17:15:59.221 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-08-17T17:15:59.222 INFO:journalctl@ceph.mon.a.smithi136.stdout: ** DB Stats ** 2024-08-17T17:15:59.222 INFO:journalctl@ceph.mon.a.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-17T17:15:59.222 INFO:journalctl@ceph.mon.a.smithi136.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-08-17T17:15:59.222 INFO:journalctl@ceph.mon.a.smithi136.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-17T17:15:59.222 INFO:journalctl@ceph.mon.a.smithi136.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-17T17:15:59.222 INFO:journalctl@ceph.mon.a.smithi136.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-08-17T17:15:59.222 INFO:journalctl@ceph.mon.a.smithi136.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-17T17:15:59.222 INFO:journalctl@ceph.mon.a.smithi136.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-17T17:15:59.222 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-08-17T17:15:59.222 INFO:journalctl@ceph.mon.a.smithi136.stdout: ** Compaction Stats [default] ** 2024-08-17T17:15:59.222 INFO:journalctl@ceph.mon.a.smithi136.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-17T17:15:59.222 INFO:journalctl@ceph.mon.a.smithi136.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-08-17T17:15:59.223 INFO:journalctl@ceph.mon.a.smithi136.stdout: L0 1/0 5.48 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 698.0 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-08-17T17:15:59.223 INFO:journalctl@ceph.mon.a.smithi136.stdout: L6 1/0 8.76 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-08-17T17:15:59.223 INFO:journalctl@ceph.mon.a.smithi136.stdout: Sum 2/0 14.24 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 698.0 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-08-17T17:15:59.223 INFO:journalctl@ceph.mon.a.smithi136.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 698.0 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-08-17T17:15:59.223 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-08-17T17:15:59.223 INFO:journalctl@ceph.mon.a.smithi136.stdout: ** Compaction Stats [default] ** 2024-08-17T17:15:59.223 INFO:journalctl@ceph.mon.a.smithi136.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-17T17:15:59.223 INFO:journalctl@ceph.mon.a.smithi136.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-17T17:15:59.223 INFO:journalctl@ceph.mon.a.smithi136.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 698.0 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-08-17T17:15:59.223 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-08-17T17:15:59.223 INFO:journalctl@ceph.mon.a.smithi136.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-08-17T17:15:59.223 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-08-17T17:15:59.224 INFO:journalctl@ceph.mon.a.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-17T17:15:59.224 INFO:journalctl@ceph.mon.a.smithi136.stdout: Flush(GB): cumulative 0.005, interval 0.005 2024-08-17T17:15:59.224 INFO:journalctl@ceph.mon.a.smithi136.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-08-17T17:15:59.224 INFO:journalctl@ceph.mon.a.smithi136.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-08-17T17:15:59.224 INFO:journalctl@ceph.mon.a.smithi136.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-08-17T17:15:59.224 INFO:journalctl@ceph.mon.a.smithi136.stdout: AddFile(Keys): cumulative 0, interval 0 2024-08-17T17:15:59.224 INFO:journalctl@ceph.mon.a.smithi136.stdout: Cumulative compaction: 0.01 GB write, 253.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:15:59.224 INFO:journalctl@ceph.mon.a.smithi136.stdout: Interval compaction: 0.01 GB write, 253.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:15:59.224 INFO:journalctl@ceph.mon.a.smithi136.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-08-17T17:15:59.224 INFO:journalctl@ceph.mon.a.smithi136.stdout: Block cache BinnedLRUCache@0x55a5f8533610#7 capacity: 512.00 MB usage: 21.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-08-17T17:15:59.224 INFO:journalctl@ceph.mon.a.smithi136.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,5.03 KB,0.000959635%) IndexBlock(1,16.08 KB,0.00306666%) Misc(1,0.00 KB,0%) 2024-08-17T17:15:59.225 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-08-17T17:15:59.225 INFO:journalctl@ceph.mon.a.smithi136.stdout: ** File Read Latency Histogram By Level [default] ** 2024-08-17T17:15:59.225 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: starting mon.a rank 0 at public addrs [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] at bind addrs [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:15:59.225 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: mon.a@-1(???) e3 preinit fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:15:59.225 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: mon.a@-1(???).mds e9 new map 2024-08-17T17:15:59.225 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: mon.a@-1(???).mds e9 print_map 2024-08-17T17:15:59.225 INFO:journalctl@ceph.mon.a.smithi136.stdout: e9 2024-08-17T17:15:59.225 INFO:journalctl@ceph.mon.a.smithi136.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-08-17T17:15:59.225 INFO:journalctl@ceph.mon.a.smithi136.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-08-17T17:15:59.225 INFO:journalctl@ceph.mon.a.smithi136.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-08-17T17:15:59.225 INFO:journalctl@ceph.mon.a.smithi136.stdout: legacy client fscid: 1 2024-08-17T17:15:59.225 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-08-17T17:15:59.226 INFO:journalctl@ceph.mon.a.smithi136.stdout: Filesystem 'foo' (1) 2024-08-17T17:15:59.226 INFO:journalctl@ceph.mon.a.smithi136.stdout: fs_name foo 2024-08-17T17:15:59.226 INFO:journalctl@ceph.mon.a.smithi136.stdout: epoch 8 2024-08-17T17:15:59.226 INFO:journalctl@ceph.mon.a.smithi136.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-08-17T17:15:59.226 INFO:journalctl@ceph.mon.a.smithi136.stdout: created 2024-08-17T17:10:05.962795+0000 2024-08-17T17:15:59.226 INFO:journalctl@ceph.mon.a.smithi136.stdout: modified 2024-08-17T17:10:20.942940+0000 2024-08-17T17:15:59.226 INFO:journalctl@ceph.mon.a.smithi136.stdout: tableserver 0 2024-08-17T17:15:59.226 INFO:journalctl@ceph.mon.a.smithi136.stdout: root 0 2024-08-17T17:15:59.226 INFO:journalctl@ceph.mon.a.smithi136.stdout: session_timeout 60 2024-08-17T17:15:59.226 INFO:journalctl@ceph.mon.a.smithi136.stdout: session_autoclose 300 2024-08-17T17:15:59.226 INFO:journalctl@ceph.mon.a.smithi136.stdout: max_file_size 1099511627776 2024-08-17T17:15:59.227 INFO:journalctl@ceph.mon.a.smithi136.stdout: max_xattr_size 65536 2024-08-17T17:15:59.227 INFO:journalctl@ceph.mon.a.smithi136.stdout: required_client_features {} 2024-08-17T17:15:59.227 INFO:journalctl@ceph.mon.a.smithi136.stdout: last_failure 0 2024-08-17T17:15:59.227 INFO:journalctl@ceph.mon.a.smithi136.stdout: last_failure_osd_epoch 0 2024-08-17T17:15:59.227 INFO:journalctl@ceph.mon.a.smithi136.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-08-17T17:15:59.227 INFO:journalctl@ceph.mon.a.smithi136.stdout: max_mds 1 2024-08-17T17:15:59.227 INFO:journalctl@ceph.mon.a.smithi136.stdout: in 0 2024-08-17T17:15:59.227 INFO:journalctl@ceph.mon.a.smithi136.stdout: up {0=24557} 2024-08-17T17:15:59.227 INFO:journalctl@ceph.mon.a.smithi136.stdout: failed 2024-08-17T17:15:59.227 INFO:journalctl@ceph.mon.a.smithi136.stdout: damaged 2024-08-17T17:15:59.227 INFO:journalctl@ceph.mon.a.smithi136.stdout: stopped 2024-08-17T17:15:59.227 INFO:journalctl@ceph.mon.a.smithi136.stdout: data_pools [4] 2024-08-17T17:15:59.228 INFO:journalctl@ceph.mon.a.smithi136.stdout: metadata_pool 3 2024-08-17T17:15:59.228 INFO:journalctl@ceph.mon.a.smithi136.stdout: inline_data disabled 2024-08-17T17:15:59.228 INFO:journalctl@ceph.mon.a.smithi136.stdout: balancer 2024-08-17T17:15:59.228 INFO:journalctl@ceph.mon.a.smithi136.stdout: bal_rank_mask -1 2024-08-17T17:15:59.228 INFO:journalctl@ceph.mon.a.smithi136.stdout: standby_count_wanted 1 2024-08-17T17:15:59.228 INFO:journalctl@ceph.mon.a.smithi136.stdout: qdb_cluster leader: 0 members: 2024-08-17T17:15:59.228 INFO:journalctl@ceph.mon.a.smithi136.stdout: [mds.foo.smithi136.ltsuqd{0:24557} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.136:6834/764583248,v1:172.21.15.136:6835/764583248] compat {c=[1],r=[1],i=[7ff]}] 2024-08-17T17:15:59.228 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-08-17T17:15:59.228 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-08-17T17:15:59.228 INFO:journalctl@ceph.mon.a.smithi136.stdout: Standby daemons: 2024-08-17T17:15:59.228 INFO:journalctl@ceph.mon.a.smithi136.stdout: 2024-08-17T17:15:59.228 INFO:journalctl@ceph.mon.a.smithi136.stdout: [mds.foo.smithi165.bmrbhh{-1:24487} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.165:6832/4276964188,v1:172.21.15.165:6833/4276964188] compat {c=[1],r=[1],i=[7ff]}] 2024-08-17T17:15:59.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: mon.a@-1(???).osd e340 crush map has features 3314933000854323200, adjusting msgr requires 2024-08-17T17:15:59.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: mon.a@-1(???).osd e340 crush map has features 432629239337189376, adjusting msgr requires 2024-08-17T17:15:59.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: mon.a@-1(???).osd e340 crush map has features 432629239337189376, adjusting msgr requires 2024-08-17T17:15:59.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: mon.a@-1(???).osd e340 crush map has features 432629239337189376, adjusting msgr requires 2024-08-17T17:15:59.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 ceph-mon[109861]: mon.a@-1(???).paxosservice(auth 1..19) refresh upgraded, format 0 -> 3 2024-08-17T17:15:59.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 bash[109823]: 9cf82e613626d40f8c2d9c7046718ba80ebe1693cad532de3387d5e9a487fed5 2024-08-17T17:15:59.229 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:15:58 smithi136 systemd[1]: Started Ceph mon.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:16:00.253 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 183111317 2024-08-17T17:16:00.278 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 18311131~4577782 2024-08-17T17:16:00.282 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 19455576~2288891 2024-08-17T17:16:00.318 INFO:tasks.workunit.client.0.smithi136.stdout:write: 2 36622263~4577782 2024-08-17T17:16:00.319 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 37766708~2288891 2024-08-17T17:16:00.350 INFO:tasks.workunit.client.0.smithi136.stdout:write: 3 54933395~4577782 2024-08-17T17:16:00.351 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 56077840~2288891 2024-08-17T17:16:00.385 INFO:tasks.workunit.client.0.smithi136.stdout:write: 4 73244526~4577782 2024-08-17T17:16:00.386 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 74388971~2288891 2024-08-17T17:16:00.422 INFO:tasks.workunit.client.0.smithi136.stdout:write: 5 91555658~4577782 2024-08-17T17:16:00.424 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 92700103~2288891 2024-08-17T17:16:00.511 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[82495]: ignoring --setuser ceph since I am not root 2024-08-17T17:16:00.511 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[82495]: ignoring --setgroup ceph since I am not root 2024-08-17T17:16:00.511 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-mgr[82518]: -- 172.21.15.165:0/4017736379 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x555b7d9cf6c0 con 0x555b7d9a3000 2024-08-17T17:16:00.511 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-mgr[82518]: -- 172.21.15.165:0/4017736379 <== mon.2 v2:172.21.15.136:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555b7d9c50e0 con 0x555b7d9a3000 2024-08-17T17:16:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-mon[34868]: pgmap v227: 137 pgs: 137 active+clean; 1.8 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 18 MiB/s rd, 53 MiB/s wr, 413 op/s 2024-08-17T17:16:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-mon[34868]: pgmap v229: 137 pgs: 137 active+clean; 1.8 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 15 MiB/s rd, 42 MiB/s wr, 331 op/s 2024-08-17T17:16:00.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-mon[34868]: mon.a calling monitor election 2024-08-17T17:16:00.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-mon[34868]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-08-17T17:16:00.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-mon[34868]: monmap epoch 3 2024-08-17T17:16:00.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-mon[34868]: fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:16:00.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-mon[34868]: last_changed 2024-08-17T17:04:52.440327+0000 2024-08-17T17:16:00.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-mon[34868]: created 2024-08-17T17:03:14.749093+0000 2024-08-17T17:16:00.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-mon[34868]: min_mon_release 18 (reef) 2024-08-17T17:16:00.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-mon[34868]: election_strategy: 1 2024-08-17T17:16:00.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-mon[34868]: 0: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.a 2024-08-17T17:16:00.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-mon[34868]: 1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-08-17T17:16:00.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-mon[34868]: 2: [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] mon.c 2024-08-17T17:16:00.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-mon[34868]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 1 up:standby 2024-08-17T17:16:00.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-mon[34868]: osdmap e340: 8 total, 8 up, 8 in 2024-08-17T17:16:00.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-mon[34868]: mgrmap e33: y(active, since 2m), standbys: x 2024-08-17T17:16:00.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-mon[34868]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-08-17T17:16:00.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-mon[34868]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-08-17T17:16:00.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-mon[34868]: application not enabled on pool 'test-librbd-smithi136-87100-9' 2024-08-17T17:16:00.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-mon[34868]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-08-17T17:16:00.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-mon[34868]: from='mgr.15039 ' entity='' 2024-08-17T17:16:00.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:00 smithi165 ceph-mon[34868]: mgrmap e34: y(active, since 2m), standbys: x 2024-08-17T17:16:00.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ignoring --setuser ceph since I am not root 2024-08-17T17:16:00.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ignoring --setgroup ceph since I am not root 2024-08-17T17:16:00.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mgr[91628]: -- 172.21.15.136:0/1413760037 <== mon.0 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555ad339f0e0 con 0x555ad337c800 2024-08-17T17:16:00.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[109861]: pgmap v227: 137 pgs: 137 active+clean; 1.8 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 18 MiB/s rd, 53 MiB/s wr, 413 op/s 2024-08-17T17:16:00.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[109861]: pgmap v229: 137 pgs: 137 active+clean; 1.8 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 15 MiB/s rd, 42 MiB/s wr, 331 op/s 2024-08-17T17:16:00.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[109861]: mon.a calling monitor election 2024-08-17T17:16:00.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[109861]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-08-17T17:16:00.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[109861]: monmap epoch 3 2024-08-17T17:16:00.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[109861]: fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:16:00.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[109861]: last_changed 2024-08-17T17:04:52.440327+0000 2024-08-17T17:16:00.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[109861]: created 2024-08-17T17:03:14.749093+0000 2024-08-17T17:16:00.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[109861]: min_mon_release 18 (reef) 2024-08-17T17:16:00.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[109861]: election_strategy: 1 2024-08-17T17:16:00.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[109861]: 0: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.a 2024-08-17T17:16:00.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[109861]: 1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-08-17T17:16:00.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[109861]: 2: [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] mon.c 2024-08-17T17:16:00.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[109861]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 1 up:standby 2024-08-17T17:16:00.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[109861]: osdmap e340: 8 total, 8 up, 8 in 2024-08-17T17:16:00.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[109861]: mgrmap e33: y(active, since 2m), standbys: x 2024-08-17T17:16:00.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[109861]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-08-17T17:16:00.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[109861]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-08-17T17:16:00.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[109861]: application not enabled on pool 'test-librbd-smithi136-87100-9' 2024-08-17T17:16:00.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[109861]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-08-17T17:16:00.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[109861]: from='mgr.15039 ' entity='' 2024-08-17T17:16:00.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[109861]: mgrmap e34: y(active, since 2m), standbys: x 2024-08-17T17:16:00.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[42580]: pgmap v227: 137 pgs: 137 active+clean; 1.8 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 18 MiB/s rd, 53 MiB/s wr, 413 op/s 2024-08-17T17:16:00.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[42580]: pgmap v229: 137 pgs: 137 active+clean; 1.8 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 15 MiB/s rd, 42 MiB/s wr, 331 op/s 2024-08-17T17:16:00.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[42580]: mon.a calling monitor election 2024-08-17T17:16:00.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[42580]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-08-17T17:16:00.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[42580]: monmap epoch 3 2024-08-17T17:16:00.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[42580]: fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:16:00.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[42580]: last_changed 2024-08-17T17:04:52.440327+0000 2024-08-17T17:16:00.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[42580]: created 2024-08-17T17:03:14.749093+0000 2024-08-17T17:16:00.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[42580]: min_mon_release 18 (reef) 2024-08-17T17:16:00.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[42580]: election_strategy: 1 2024-08-17T17:16:00.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[42580]: 0: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.a 2024-08-17T17:16:00.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[42580]: 1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-08-17T17:16:00.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[42580]: 2: [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] mon.c 2024-08-17T17:16:00.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[42580]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 1 up:standby 2024-08-17T17:16:00.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[42580]: osdmap e340: 8 total, 8 up, 8 in 2024-08-17T17:16:00.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[42580]: mgrmap e33: y(active, since 2m), standbys: x 2024-08-17T17:16:00.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[42580]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-08-17T17:16:00.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[42580]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-08-17T17:16:00.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[42580]: application not enabled on pool 'test-librbd-smithi136-87100-9' 2024-08-17T17:16:00.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[42580]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-08-17T17:16:00.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[42580]: from='mgr.15039 ' entity='' 2024-08-17T17:16:00.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:00 smithi136 ceph-mon[42580]: mgrmap e34: y(active, since 2m), standbys: x 2024-08-17T17:16:01.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:01 smithi165 ceph-mon[34868]: osdmap e341: 8 total, 8 up, 8 in 2024-08-17T17:16:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:01 smithi136 ceph-mon[109861]: osdmap e341: 8 total, 8 up, 8 in 2024-08-17T17:16:01.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:01 smithi136 ceph-mon[42580]: osdmap e341: 8 total, 8 up, 8 in 2024-08-17T17:16:01.805 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:16:02.045 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:16:03.476 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.Stress2 (21988 ms) 2024-08-17T17:16:03.477 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMigration.StressLive 2024-08-17T17:16:03.509 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 83886080 2024-08-17T17:16:03.519 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 52548252~6604073 2024-08-17T17:16:03.562 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 27324898~6789503 2024-08-17T17:16:03.563 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 80041155~1238091 2024-08-17T17:16:03.575 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 22668407~4263038 2024-08-17T17:16:03.575 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 78557714~5328366 2024-08-17T17:16:03.607 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 64086469~2317195 2024-08-17T17:16:03.607 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 62823674~2839317 2024-08-17T17:16:03.630 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 61144829~3901506 2024-08-17T17:16:03.633 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap0 2024-08-17T17:16:04.704 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 21552501~69777 2024-08-17T17:16:04.711 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 30745159~6690086 2024-08-17T17:16:04.711 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 50877261~7674970 2024-08-17T17:16:04.716 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 4375513~5646746 2024-08-17T17:16:04.716 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 29797230~3176820 2024-08-17T17:16:04.717 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 30410135~5657439 2024-08-17T17:16:04.752 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 62593161~2954093 2024-08-17T17:16:04.755 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 76425165~7460915 2024-08-17T17:16:04.755 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap1 2024-08-17T17:16:04.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:04 smithi136 ceph-mon[109861]: osdmap e342: 8 total, 8 up, 8 in 2024-08-17T17:16:04.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:04 smithi136 ceph-mon[42580]: osdmap e342: 8 total, 8 up, 8 in 2024-08-17T17:16:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:04 smithi165 ceph-mon[34868]: osdmap e342: 8 total, 8 up, 8 in 2024-08-17T17:16:05.819 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap1 2024-08-17T17:16:05.830 INFO:tasks.workunit.client.0.smithi136.stdout:clone image187 -> image190 2024-08-17T17:16:05.916 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 156893615 2024-08-17T17:16:05.926 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 130193116~1906409 2024-08-17T17:16:05.946 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 56651339~2838430 2024-08-17T17:16:05.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:05 smithi136 ceph-mon[42580]: osdmap e343: 8 total, 8 up, 8 in 2024-08-17T17:16:05.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:05 smithi136 ceph-mon[42580]: Standby manager daemon x restarted 2024-08-17T17:16:05.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:05 smithi136 ceph-mon[42580]: Standby manager daemon x started 2024-08-17T17:16:05.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:05 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.165:0/175529447' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-08-17T17:16:05.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:05 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.165:0/175529447' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-08-17T17:16:05.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:05 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.165:0/175529447' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-08-17T17:16:05.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:05 smithi136 ceph-mon[42580]: from='mgr.? 172.21.15.165:0/175529447' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-08-17T17:16:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:05 smithi136 ceph-mon[109861]: osdmap e343: 8 total, 8 up, 8 in 2024-08-17T17:16:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:05 smithi136 ceph-mon[109861]: Standby manager daemon x restarted 2024-08-17T17:16:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:05 smithi136 ceph-mon[109861]: Standby manager daemon x started 2024-08-17T17:16:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:05 smithi136 ceph-mon[109861]: from='mgr.? 172.21.15.165:0/175529447' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-08-17T17:16:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:05 smithi136 ceph-mon[109861]: from='mgr.? 172.21.15.165:0/175529447' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-08-17T17:16:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:05 smithi136 ceph-mon[109861]: from='mgr.? 172.21.15.165:0/175529447' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-08-17T17:16:05.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:05 smithi136 ceph-mon[109861]: from='mgr.? 172.21.15.165:0/175529447' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-08-17T17:16:05.976 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 82516864~130177 2024-08-17T17:16:05.982 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 57055180~118057 2024-08-17T17:16:05.984 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 153176074~3717541 2024-08-17T17:16:06.005 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 30717742~4423991 2024-08-17T17:16:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:05 smithi165 ceph-mon[34868]: osdmap e343: 8 total, 8 up, 8 in 2024-08-17T17:16:06.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:05 smithi165 ceph-mon[34868]: Standby manager daemon x restarted 2024-08-17T17:16:06.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:05 smithi165 ceph-mon[34868]: Standby manager daemon x started 2024-08-17T17:16:06.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:05 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.165:0/175529447' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-08-17T17:16:06.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:05 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.165:0/175529447' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-08-17T17:16:06.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:05 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.165:0/175529447' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-08-17T17:16:06.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:05 smithi165 ceph-mon[34868]: from='mgr.? 172.21.15.165:0/175529447' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-08-17T17:16:06.012 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:16:05 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[82495]: [17/Aug/2024:17:16:05] ENGINE Bus STARTING 2024-08-17T17:16:06.012 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:16:05 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[82495]: CherryPy Checker: 2024-08-17T17:16:06.013 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:16:05 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[82495]: The Application mounted at '' has an empty config. 2024-08-17T17:16:06.013 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:16:05 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[82495]: 2024-08-17T17:16:06.013 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:16:05 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[82495]: [17/Aug/2024:17:16:05] ENGINE Serving on http://:::9283 2024-08-17T17:16:06.013 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 17:16:05 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x[82495]: [17/Aug/2024:17:16:05] ENGINE Bus STARTED 2024-08-17T17:16:06.018 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 30037569~2612408 2024-08-17T17:16:06.038 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 30601176~6609787 2024-08-17T17:16:06.041 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap2 2024-08-17T17:16:06.446 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:16:06.176+0000 7f0c93bb5640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:16:06.446 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:16:06.176+0000 7f0c93bb5640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:16:06.446 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:16:06.176+0000 7f0c93bb5640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:16:06.447 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:16:06.176+0000 7f0c93bb5640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:16:06.447 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:16:06.176+0000 7f0c93bb5640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:16:06.447 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:16:06.176+0000 7f0c90baf640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:16:06.447 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:16:06.176+0000 7f0c90baf640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:16:06.447 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:16:06.176+0000 7f0c90baf640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:16:06.447 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:16:06.176+0000 7f0c90baf640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:16:06.447 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:16:06.176+0000 7f0c90baf640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:16:06.961 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: mgrmap e35: y(active, since 2m), standbys: x 2024-08-17T17:16:06.961 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: osdmap e344: 8 total, 8 up, 8 in 2024-08-17T17:16:06.962 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: Active manager daemon y restarted 2024-08-17T17:16:06.962 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: Activating manager daemon y 2024-08-17T17:16:06.962 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: osdmap e345: 8 total, 8 up, 8 in 2024-08-17T17:16:06.962 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: mgrmap e36: y(active, starting, since 0.00910608s), standbys: x 2024-08-17T17:16:06.962 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-17T17:16:06.962 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-17T17:16:06.962 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-17T17:16:06.962 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi165.bmrbhh"} : dispatch 2024-08-17T17:16:06.962 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi136.ltsuqd"} : dispatch 2024-08-17T17:16:06.962 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-08-17T17:16:06.962 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-08-17T17:16:06.962 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-17T17:16:06.963 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-17T17:16:06.963 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-17T17:16:06.963 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-17T17:16:06.963 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-17T17:16:06.963 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-17T17:16:06.963 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-17T17:16:06.963 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-17T17:16:06.963 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-08-17T17:16:06.963 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-08-17T17:16:06.963 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-08-17T17:16:06.963 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: Manager daemon y is now available 2024-08-17T17:16:06.963 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:16:06.963 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:16:06.963 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-08-17T17:16:06.964 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-08-17T17:16:06.964 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-08-17T17:16:06.964 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: mgrmap e35: y(active, since 2m), standbys: x 2024-08-17T17:16:06.964 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: osdmap e344: 8 total, 8 up, 8 in 2024-08-17T17:16:06.964 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: Active manager daemon y restarted 2024-08-17T17:16:06.964 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: Activating manager daemon y 2024-08-17T17:16:06.964 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: osdmap e345: 8 total, 8 up, 8 in 2024-08-17T17:16:06.964 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: mgrmap e36: y(active, starting, since 0.00910608s), standbys: x 2024-08-17T17:16:06.965 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-17T17:16:06.965 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-17T17:16:06.965 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-17T17:16:06.965 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi165.bmrbhh"} : dispatch 2024-08-17T17:16:06.965 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi136.ltsuqd"} : dispatch 2024-08-17T17:16:06.965 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-08-17T17:16:06.965 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-08-17T17:16:06.965 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-17T17:16:06.965 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-17T17:16:06.965 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-17T17:16:06.965 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-17T17:16:06.965 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-17T17:16:06.965 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-17T17:16:06.965 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-17T17:16:06.966 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-17T17:16:06.966 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-08-17T17:16:06.966 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-08-17T17:16:06.966 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-08-17T17:16:06.966 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: Manager daemon y is now available 2024-08-17T17:16:06.966 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:16:06.966 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:16:06.966 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-08-17T17:16:06.966 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-08-17T17:16:06.966 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-08-17T17:16:06.966 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:16:06.815+0000 7f0cc2ea7640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-08-17T17:16:06.967 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:16:06] ENGINE Bus STARTING 2024-08-17T17:16:06.967 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: CherryPy Checker: 2024-08-17T17:16:06.967 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: The Application mounted at '' has an empty config. 2024-08-17T17:16:06.967 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:16:06.967 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:16:06] ENGINE Serving on http://:::9283 2024-08-17T17:16:06.967 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:16:06] ENGINE Bus STARTED 2024-08-17T17:16:06.967 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:16:06.959+0000 7f0c769c1640 -1 log_channel(cephadm) log [ERR] : [17/Aug/2024:17:16:06] ENGINE Error in HTTPServer.serve 2024-08-17T17:16:06.967 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: Traceback (most recent call last): 2024-08-17T17:16:06.967 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:16:06.967 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: self._connections.run(self.expiration_interval) 2024-08-17T17:16:06.967 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:16:06.967 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: self._run(expiration_interval) 2024-08-17T17:16:06.967 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:16:06.968 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:16:06.968 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:16:06.968 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:16:06.968 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:16:06.968 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: s = self.context.wrap_socket( 2024-08-17T17:16:06.968 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:16:06.968 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: return self.sslsocket_class._create( 2024-08-17T17:16:06.968 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:16:06.968 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: self.do_handshake() 2024-08-17T17:16:06.968 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:16:06.968 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: self._sslobj.do_handshake() 2024-08-17T17:16:06.968 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:16:06.968 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:16:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: mgrmap e35: y(active, since 2m), standbys: x 2024-08-17T17:16:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: osdmap e344: 8 total, 8 up, 8 in 2024-08-17T17:16:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: Active manager daemon y restarted 2024-08-17T17:16:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: Activating manager daemon y 2024-08-17T17:16:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: osdmap e345: 8 total, 8 up, 8 in 2024-08-17T17:16:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: mgrmap e36: y(active, starting, since 0.00910608s), standbys: x 2024-08-17T17:16:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-17T17:16:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-17T17:16:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-17T17:16:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi165.bmrbhh"} : dispatch 2024-08-17T17:16:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi136.ltsuqd"} : dispatch 2024-08-17T17:16:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-08-17T17:16:07.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-08-17T17:16:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-17T17:16:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-17T17:16:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-17T17:16:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-17T17:16:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-17T17:16:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-17T17:16:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-17T17:16:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-17T17:16:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-08-17T17:16:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-08-17T17:16:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-08-17T17:16:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: Manager daemon y is now available 2024-08-17T17:16:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:16:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:16:07.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-08-17T17:16:07.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-08-17T17:16:07.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-08-17T17:16:07.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:16:07] "GET /metrics HTTP/1.1" 200 35895 "" "Prometheus/2.51.0" 2024-08-17T17:16:07.846 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap2 2024-08-17T17:16:07.855 INFO:tasks.workunit.client.0.smithi136.stdout:clone image190 -> image192 2024-08-17T17:16:07.973 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 125113876 2024-08-17T17:16:07.990 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 25158025~4472544 2024-08-17T17:16:08.051 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 76717923~1692680 2024-08-17T17:16:08.057 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 107650121~7187924 2024-08-17T17:16:08.098 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 100159727~5614280 2024-08-17T17:16:08.104 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 31284452~6001761 2024-08-17T17:16:08.116 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:07 smithi136 ceph-mon[42580]: mgrmap e37: y(active, since 1.01323s), standbys: x 2024-08-17T17:16:08.117 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:07 smithi136 ceph-mon[42580]: osdmap e346: 8 total, 8 up, 8 in 2024-08-17T17:16:08.117 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:07 smithi136 ceph-mon[42580]: [17/Aug/2024:17:16:06] ENGINE Bus STARTING 2024-08-17T17:16:08.117 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:07 smithi136 ceph-mon[42580]: [17/Aug/2024:17:16:06] ENGINE Serving on https://172.21.15.136:7150 2024-08-17T17:16:08.117 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:07 smithi136 ceph-mon[42580]: [17/Aug/2024:17:16:06] ENGINE Error in HTTPServer.serve 2024-08-17T17:16:08.117 INFO:journalctl@ceph.mon.c.smithi136.stdout: Traceback (most recent call last): 2024-08-17T17:16:08.117 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:16:08.117 INFO:journalctl@ceph.mon.c.smithi136.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:16:08.117 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:16:08.117 INFO:journalctl@ceph.mon.c.smithi136.stdout: self._run(expiration_interval) 2024-08-17T17:16:08.117 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:16:08.117 INFO:journalctl@ceph.mon.c.smithi136.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:16:08.117 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:16:08.118 INFO:journalctl@ceph.mon.c.smithi136.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:16:08.118 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:16:08.118 INFO:journalctl@ceph.mon.c.smithi136.stdout: s = self.context.wrap_socket( 2024-08-17T17:16:08.118 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:16:08.118 INFO:journalctl@ceph.mon.c.smithi136.stdout: return self.sslsocket_class._create( 2024-08-17T17:16:08.118 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:16:08.118 INFO:journalctl@ceph.mon.c.smithi136.stdout: self.do_handshake() 2024-08-17T17:16:08.118 INFO:journalctl@ceph.mon.c.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:16:08.118 INFO:journalctl@ceph.mon.c.smithi136.stdout: self._sslobj.do_handshake() 2024-08-17T17:16:08.118 INFO:journalctl@ceph.mon.c.smithi136.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:16:08.118 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:07 smithi136 ceph-mon[42580]: [17/Aug/2024:17:16:07] ENGINE Serving on http://172.21.15.136:8765 2024-08-17T17:16:08.118 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:07 smithi136 ceph-mon[42580]: [17/Aug/2024:17:16:07] ENGINE Bus STARTED 2024-08-17T17:16:08.119 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:07 smithi136 ceph-mon[109861]: mgrmap e37: y(active, since 1.01323s), standbys: x 2024-08-17T17:16:08.119 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:07 smithi136 ceph-mon[109861]: osdmap e346: 8 total, 8 up, 8 in 2024-08-17T17:16:08.119 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:07 smithi136 ceph-mon[109861]: [17/Aug/2024:17:16:06] ENGINE Bus STARTING 2024-08-17T17:16:08.119 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:07 smithi136 ceph-mon[109861]: [17/Aug/2024:17:16:06] ENGINE Serving on https://172.21.15.136:7150 2024-08-17T17:16:08.119 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:07 smithi136 ceph-mon[109861]: [17/Aug/2024:17:16:06] ENGINE Error in HTTPServer.serve 2024-08-17T17:16:08.119 INFO:journalctl@ceph.mon.a.smithi136.stdout: Traceback (most recent call last): 2024-08-17T17:16:08.119 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:16:08.119 INFO:journalctl@ceph.mon.a.smithi136.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:16:08.119 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:16:08.119 INFO:journalctl@ceph.mon.a.smithi136.stdout: self._run(expiration_interval) 2024-08-17T17:16:08.119 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:16:08.119 INFO:journalctl@ceph.mon.a.smithi136.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:16:08.119 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:16:08.120 INFO:journalctl@ceph.mon.a.smithi136.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:16:08.120 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:16:08.120 INFO:journalctl@ceph.mon.a.smithi136.stdout: s = self.context.wrap_socket( 2024-08-17T17:16:08.120 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:16:08.120 INFO:journalctl@ceph.mon.a.smithi136.stdout: return self.sslsocket_class._create( 2024-08-17T17:16:08.120 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:16:08.120 INFO:journalctl@ceph.mon.a.smithi136.stdout: self.do_handshake() 2024-08-17T17:16:08.120 INFO:journalctl@ceph.mon.a.smithi136.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:16:08.120 INFO:journalctl@ceph.mon.a.smithi136.stdout: self._sslobj.do_handshake() 2024-08-17T17:16:08.120 INFO:journalctl@ceph.mon.a.smithi136.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:16:08.120 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:07 smithi136 ceph-mon[109861]: [17/Aug/2024:17:16:07] ENGINE Serving on http://172.21.15.136:8765 2024-08-17T17:16:08.120 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:07 smithi136 ceph-mon[109861]: [17/Aug/2024:17:16:07] ENGINE Bus STARTED 2024-08-17T17:16:08.140 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 26010993~6570630 2024-08-17T17:16:08.142 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 119949815~5164061 2024-08-17T17:16:08.170 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 83662226~6350255 2024-08-17T17:16:08.179 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: snap3 2024-08-17T17:16:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:07 smithi165 ceph-mon[34868]: mgrmap e37: y(active, since 1.01323s), standbys: x 2024-08-17T17:16:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:07 smithi165 ceph-mon[34868]: osdmap e346: 8 total, 8 up, 8 in 2024-08-17T17:16:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:07 smithi165 ceph-mon[34868]: [17/Aug/2024:17:16:06] ENGINE Bus STARTING 2024-08-17T17:16:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:07 smithi165 ceph-mon[34868]: [17/Aug/2024:17:16:06] ENGINE Serving on https://172.21.15.136:7150 2024-08-17T17:16:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:07 smithi165 ceph-mon[34868]: [17/Aug/2024:17:16:06] ENGINE Error in HTTPServer.serve 2024-08-17T17:16:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout: Traceback (most recent call last): 2024-08-17T17:16:08.262 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:16:08.262 INFO:journalctl@ceph.mon.b.smithi165.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:16:08.262 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:16:08.262 INFO:journalctl@ceph.mon.b.smithi165.stdout: self._run(expiration_interval) 2024-08-17T17:16:08.262 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:16:08.262 INFO:journalctl@ceph.mon.b.smithi165.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:16:08.262 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:16:08.262 INFO:journalctl@ceph.mon.b.smithi165.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:16:08.262 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:16:08.262 INFO:journalctl@ceph.mon.b.smithi165.stdout: s = self.context.wrap_socket( 2024-08-17T17:16:08.262 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:16:08.262 INFO:journalctl@ceph.mon.b.smithi165.stdout: return self.sslsocket_class._create( 2024-08-17T17:16:08.262 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:16:08.262 INFO:journalctl@ceph.mon.b.smithi165.stdout: self.do_handshake() 2024-08-17T17:16:08.263 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:16:08.263 INFO:journalctl@ceph.mon.b.smithi165.stdout: self._sslobj.do_handshake() 2024-08-17T17:16:08.263 INFO:journalctl@ceph.mon.b.smithi165.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:16:08.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:07 smithi165 ceph-mon[34868]: [17/Aug/2024:17:16:07] ENGINE Serving on http://172.21.15.136:8765 2024-08-17T17:16:08.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:07 smithi165 ceph-mon[34868]: [17/Aug/2024:17:16:07] ENGINE Bus STARTED 2024-08-17T17:16:09.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:08 smithi136 ceph-mon[109861]: pgmap v5: 137 pgs: 137 active+clean; 1.9 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 4.5 GiB/s rd, 1.8 GiB/s wr, 74.20k op/s; 1.7 MiB/s, 6 objects/s recovering 2024-08-17T17:16:09.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:08 smithi136 ceph-mon[109861]: mgrmap e38: y(active, since 2s), standbys: x 2024-08-17T17:16:09.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:08 smithi136 ceph-mon[109861]: osdmap e347: 8 total, 8 up, 8 in 2024-08-17T17:16:09.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:08 smithi136 ceph-mon[42580]: pgmap v5: 137 pgs: 137 active+clean; 1.9 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 4.5 GiB/s rd, 1.8 GiB/s wr, 74.20k op/s; 1.7 MiB/s, 6 objects/s recovering 2024-08-17T17:16:09.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:08 smithi136 ceph-mon[42580]: mgrmap e38: y(active, since 2s), standbys: x 2024-08-17T17:16:09.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:08 smithi136 ceph-mon[42580]: osdmap e347: 8 total, 8 up, 8 in 2024-08-17T17:16:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:08 smithi165 ceph-mon[34868]: pgmap v5: 137 pgs: 137 active+clean; 1.9 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 4.5 GiB/s rd, 1.8 GiB/s wr, 74.20k op/s; 1.7 MiB/s, 6 objects/s recovering 2024-08-17T17:16:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:08 smithi165 ceph-mon[34868]: mgrmap e38: y(active, since 2s), standbys: x 2024-08-17T17:16:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:08 smithi165 ceph-mon[34868]: osdmap e347: 8 total, 8 up, 8 in 2024-08-17T17:16:09.854 INFO:tasks.workunit.client.0.smithi136.stdout:snap_protect: snap3 2024-08-17T17:16:09.863 INFO:tasks.workunit.client.0.smithi136.stdout:clone image192 -> image194 2024-08-17T17:16:10.009 INFO:tasks.workunit.client.0.smithi136.stdout:migration_prepare 2024-08-17T17:16:10.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:09 smithi165 ceph-mon[34868]: osdmap e348: 8 total, 8 up, 8 in 2024-08-17T17:16:10.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:09 smithi136 ceph-mon[109861]: osdmap e348: 8 total, 8 up, 8 in 2024-08-17T17:16:10.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:09 smithi136 ceph-mon[42580]: osdmap e348: 8 total, 8 up, 8 in 2024-08-17T17:16:10.421 INFO:tasks.workunit.client.0.smithi136.stdout:migration_execute 2024-08-17T17:16:10.421 INFO:tasks.workunit.client.0.smithi136.stdout:write: a 64800984~2560859 2024-08-17T17:16:10.463 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 114349714~1495223 2024-08-17T17:16:10.479 INFO:tasks.workunit.client.0.smithi136.stdout:write: a 35878920~4782767 2024-08-17T17:16:10.527 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 38626944~7715898 2024-08-17T17:16:10.551 INFO:tasks.workunit.client.0.smithi136.stdout:write: a 66433609~8024712 2024-08-17T17:16:10.604 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 23763041~5690467 2024-08-17T17:16:10.645 INFO:tasks.workunit.client.0.smithi136.stdout:write: a 60494659~4580307 2024-08-17T17:16:10.686 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 47695103~346510 2024-08-17T17:16:10.687 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: user0 2024-08-17T17:16:11.098 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:10 smithi136 ceph-mon[109861]: pgmap v8: 137 pgs: 137 active+clean; 1.9 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.5 GiB/s rd, 603 MiB/s wr, 24.61k op/s; 578 KiB/s, 2 objects/s recovering 2024-08-17T17:16:11.098 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:10 smithi136 ceph-mon[109861]: osdmap e349: 8 total, 8 up, 8 in 2024-08-17T17:16:11.098 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:10 smithi136 ceph-mon[109861]: mgrmap e39: y(active, since 4s), standbys: x 2024-08-17T17:16:11.098 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:11.098 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:11.098 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:10 smithi136 ceph-mon[42580]: pgmap v8: 137 pgs: 137 active+clean; 1.9 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.5 GiB/s rd, 603 MiB/s wr, 24.61k op/s; 578 KiB/s, 2 objects/s recovering 2024-08-17T17:16:11.099 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:10 smithi136 ceph-mon[42580]: osdmap e349: 8 total, 8 up, 8 in 2024-08-17T17:16:11.099 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:10 smithi136 ceph-mon[42580]: mgrmap e39: y(active, since 4s), standbys: x 2024-08-17T17:16:11.099 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:10 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:11.099 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:10 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:10 smithi165 ceph-mon[34868]: pgmap v8: 137 pgs: 137 active+clean; 1.9 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.5 GiB/s rd, 603 MiB/s wr, 24.61k op/s; 578 KiB/s, 2 objects/s recovering 2024-08-17T17:16:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:10 smithi165 ceph-mon[34868]: osdmap e349: 8 total, 8 up, 8 in 2024-08-17T17:16:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:10 smithi165 ceph-mon[34868]: mgrmap e39: y(active, since 4s), standbys: x 2024-08-17T17:16:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:10 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:10 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:12.175 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:12.175 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:12.175 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:12.176 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:12.176 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:11 smithi136 ceph-mon[109861]: osdmap e350: 8 total, 8 up, 8 in 2024-08-17T17:16:12.176 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:12.176 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:12.176 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:11 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:12.176 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:11 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:12.176 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:11 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:12.176 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:11 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:12.176 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:11 smithi136 ceph-mon[42580]: osdmap e350: 8 total, 8 up, 8 in 2024-08-17T17:16:12.176 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:11 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:12.176 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:11 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:12.214 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:11 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:12.214 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:11 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:12.215 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:11 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:12.215 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:11 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:12.215 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:11 smithi165 ceph-mon[34868]: osdmap e350: 8 total, 8 up, 8 in 2024-08-17T17:16:12.215 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:11 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:12.215 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:11 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:12.365 INFO:tasks.workunit.client.0.smithi136.stdout:write: b 9683592~7534434 2024-08-17T17:16:12.390 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 81097849~741451 2024-08-17T17:16:12.391 INFO:tasks.workunit.client.0.smithi136.stdout:write: b 27928759~144305 2024-08-17T17:16:12.392 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 62570706~3588080 2024-08-17T17:16:12.392 INFO:tasks.workunit.client.0.smithi136.stdout:write: b 94003504~774556 2024-08-17T17:16:12.392 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 20709818~1617777 2024-08-17T17:16:12.393 INFO:tasks.workunit.client.0.smithi136.stdout:write: b 33144211~4096874 2024-08-17T17:16:12.396 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 114222798~5778218 2024-08-17T17:16:12.396 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: user1 2024-08-17T17:16:13.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:12 smithi136 ceph-mon[109861]: pgmap v11: 137 pgs: 137 active+clean; 1.9 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail 2024-08-17T17:16:13.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:12 smithi136 ceph-mon[109861]: osdmap e351: 8 total, 8 up, 8 in 2024-08-17T17:16:13.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:12 smithi136 ceph-mon[42580]: pgmap v11: 137 pgs: 137 active+clean; 1.9 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail 2024-08-17T17:16:13.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:12 smithi136 ceph-mon[42580]: osdmap e351: 8 total, 8 up, 8 in 2024-08-17T17:16:13.222 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:12 smithi165 ceph-mon[34868]: pgmap v11: 137 pgs: 137 active+clean; 1.9 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail 2024-08-17T17:16:13.222 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:12 smithi165 ceph-mon[34868]: osdmap e351: 8 total, 8 up, 8 in 2024-08-17T17:16:14.372 INFO:tasks.workunit.client.0.smithi136.stdout:write: c 89027354~2493100 2024-08-17T17:16:14.385 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 88761767~2305607 2024-08-17T17:16:14.394 INFO:tasks.workunit.client.0.smithi136.stdout:write: c 61377211~6443246 2024-08-17T17:16:14.400 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 48549121~950816 2024-08-17T17:16:14.400 INFO:tasks.workunit.client.0.smithi136.stdout:write: c 50335139~344608 2024-08-17T17:16:14.400 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 92965806~5519508 2024-08-17T17:16:14.400 INFO:tasks.workunit.client.0.smithi136.stdout:write: c 121142486~3415502 2024-08-17T17:16:14.404 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 87166078~8225552 2024-08-17T17:16:14.422 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: user2 2024-08-17T17:16:14.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:14 smithi136 ceph-mon[109861]: osdmap e352: 8 total, 8 up, 8 in 2024-08-17T17:16:14.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:14 smithi136 ceph-mon[109861]: pgmap v14: 137 pgs: 137 active+clean; 2.1 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 23 MiB/s rd, 34 MiB/s wr, 1.00k op/s 2024-08-17T17:16:14.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:14 smithi136 ceph-mon[42580]: osdmap e352: 8 total, 8 up, 8 in 2024-08-17T17:16:14.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:14 smithi136 ceph-mon[42580]: pgmap v14: 137 pgs: 137 active+clean; 2.1 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 23 MiB/s rd, 34 MiB/s wr, 1.00k op/s 2024-08-17T17:16:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:14 smithi165 ceph-mon[34868]: osdmap e352: 8 total, 8 up, 8 in 2024-08-17T17:16:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:14 smithi165 ceph-mon[34868]: pgmap v14: 137 pgs: 137 active+clean; 2.1 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 23 MiB/s rd, 34 MiB/s wr, 1.00k op/s 2024-08-17T17:16:15.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:15 smithi136 ceph-mon[109861]: osdmap e353: 8 total, 8 up, 8 in 2024-08-17T17:16:15.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:15 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:15.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:15 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:15.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:15 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:15.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:15 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:15.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:15 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:16:15.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:15 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:16:15.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:15 smithi136 ceph-mon[109861]: Updating smithi136:/etc/ceph/ceph.conf 2024-08-17T17:16:15.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:15 smithi136 ceph-mon[109861]: Updating smithi165:/etc/ceph/ceph.conf 2024-08-17T17:16:15.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:15 smithi136 ceph-mon[109861]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:16:15.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:15 smithi136 ceph-mon[109861]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:16:15.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:15 smithi136 ceph-mon[42580]: osdmap e353: 8 total, 8 up, 8 in 2024-08-17T17:16:15.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:15 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:15.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:15 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:15.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:15 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:15.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:15 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:15.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:15 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:16:15.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:15 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:16:15.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:15 smithi136 ceph-mon[42580]: Updating smithi136:/etc/ceph/ceph.conf 2024-08-17T17:16:15.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:15 smithi136 ceph-mon[42580]: Updating smithi165:/etc/ceph/ceph.conf 2024-08-17T17:16:15.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:15 smithi136 ceph-mon[42580]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:16:15.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:15 smithi136 ceph-mon[42580]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:16:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:15 smithi165 ceph-mon[34868]: osdmap e353: 8 total, 8 up, 8 in 2024-08-17T17:16:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:15 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:15 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:15 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:15 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:15 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:16:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:15 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:16:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:15 smithi165 ceph-mon[34868]: Updating smithi136:/etc/ceph/ceph.conf 2024-08-17T17:16:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:15 smithi165 ceph-mon[34868]: Updating smithi165:/etc/ceph/ceph.conf 2024-08-17T17:16:15.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:15 smithi165 ceph-mon[34868]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:16:15.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:15 smithi165 ceph-mon[34868]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:16:16.391 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 155001777 2024-08-17T17:16:16.404 INFO:tasks.workunit.client.0.smithi136.stdout:write: d 141854612~6090960 2024-08-17T17:16:16.425 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 111535212~442473 2024-08-17T17:16:16.425 INFO:tasks.workunit.client.0.smithi136.stdout:write: d 63127589~6161789 2024-08-17T17:16:16.435 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 123937652~2976260 2024-08-17T17:16:16.435 INFO:tasks.workunit.client.0.smithi136.stdout:write: d 141721106~4122432 2024-08-17T17:16:16.463 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 73554823~6768757 2024-08-17T17:16:16.463 INFO:tasks.workunit.client.0.smithi136.stdout:write: d 136590746~3262402 2024-08-17T17:16:16.464 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 87321966~1039284 2024-08-17T17:16:16.465 INFO:tasks.workunit.client.0.smithi136.stdout:snap_create: user3 2024-08-17T17:16:16.591 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[42580]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:16:16.591 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[42580]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:16:16.592 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[42580]: osdmap e354: 8 total, 8 up, 8 in 2024-08-17T17:16:16.592 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[42580]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:16:16.592 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[42580]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:16:16.592 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:16.592 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:16.592 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:16.592 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:16.592 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:16.592 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[42580]: pgmap v17: 137 pgs: 137 active+clean; 2.1 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 23 MiB/s rd, 34 MiB/s wr, 1.00k op/s 2024-08-17T17:16:16.592 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:16:16.592 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:16.592 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-08-17T17:16:16.593 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["c"]} : dispatch 2024-08-17T17:16:16.593 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[42580]: Upgrade: It appears safe to stop mon.c 2024-08-17T17:16:16.593 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[109861]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:16:16.593 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[109861]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:16:16.593 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[109861]: osdmap e354: 8 total, 8 up, 8 in 2024-08-17T17:16:16.593 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[109861]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:16:16.593 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[109861]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:16:16.593 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:16.593 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:16.593 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:16.593 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:16.594 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:16.594 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[109861]: pgmap v17: 137 pgs: 137 active+clean; 2.1 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 23 MiB/s rd, 34 MiB/s wr, 1.00k op/s 2024-08-17T17:16:16.594 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:16:16.594 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:16.594 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-08-17T17:16:16.594 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["c"]} : dispatch 2024-08-17T17:16:16.594 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:16 smithi136 ceph-mon[109861]: Upgrade: It appears safe to stop mon.c 2024-08-17T17:16:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:16 smithi165 ceph-mon[34868]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:16:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:16 smithi165 ceph-mon[34868]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:16:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:16 smithi165 ceph-mon[34868]: osdmap e354: 8 total, 8 up, 8 in 2024-08-17T17:16:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:16 smithi165 ceph-mon[34868]: Updating smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:16:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:16 smithi165 ceph-mon[34868]: Updating smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:16:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:16 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:16.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:16 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:16.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:16 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:16.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:16 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:16.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:16 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:16.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:16 smithi165 ceph-mon[34868]: pgmap v17: 137 pgs: 137 active+clean; 2.1 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 23 MiB/s rd, 34 MiB/s wr, 1.00k op/s 2024-08-17T17:16:16.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:16 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:16:16.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:16 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:16.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:16 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-08-17T17:16:16.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:16 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["c"]} : dispatch 2024-08-17T17:16:16.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:16 smithi165 ceph-mon[34868]: Upgrade: It appears safe to stop mon.c 2024-08-17T17:16:17.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:16:17] "GET /metrics HTTP/1.1" 200 35895 "" "Prometheus/2.51.0" 2024-08-17T17:16:17.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:17 smithi136 ceph-mon[109861]: osdmap e355: 8 total, 8 up, 8 in 2024-08-17T17:16:17.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:17 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:17.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:17 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:16:17.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:17 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:16:17.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:17 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:16:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:17 smithi136 ceph-mon[42580]: osdmap e355: 8 total, 8 up, 8 in 2024-08-17T17:16:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:17 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:17 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:16:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:17 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:16:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:17 smithi136 ceph-mon[42580]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:16:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:17 smithi165 ceph-mon[34868]: osdmap e355: 8 total, 8 up, 8 in 2024-08-17T17:16:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:17 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:17 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:16:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:17 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:16:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:17 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:16:18.408 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 238016419 2024-08-17T17:16:18.426 INFO:tasks.workunit.client.0.smithi136.stdout:write: 1 23801641~5950410 2024-08-17T17:16:18.428 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 25289243~2975205 2024-08-17T17:16:18.455 INFO:tasks.workunit.client.0.smithi136.stdout:write: 2 47603283~5950410 2024-08-17T17:16:18.457 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 49090885~2975205 2024-08-17T17:16:18.483 INFO:tasks.workunit.client.0.smithi136.stdout:write: 3 71404925~5950410 2024-08-17T17:16:18.485 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 72892527~2975205 2024-08-17T17:16:18.516 INFO:tasks.workunit.client.0.smithi136.stdout:write: 4 95206567~5950410 2024-08-17T17:16:18.518 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 96694169~2975205 2024-08-17T17:16:18.545 INFO:tasks.workunit.client.0.smithi136.stdout:write: 5 119008209~5950410 2024-08-17T17:16:18.546 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 120495811~2975205 2024-08-17T17:16:18.653 INFO:tasks.workunit.client.0.smithi136.stdout:migration_commit 2024-08-17T17:16:18.654 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:18 smithi136 ceph-mon[42580]: Upgrade: Updating mon.c 2024-08-17T17:16:18.654 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:18 smithi136 ceph-mon[42580]: Deploying daemon mon.c on smithi136 2024-08-17T17:16:18.654 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:18 smithi136 ceph-mon[42580]: osdmap e356: 8 total, 8 up, 8 in 2024-08-17T17:16:18.654 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:18 smithi136 ceph-mon[42580]: pgmap v20: 137 pgs: 137 active+clean; 2.2 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 6.4 MiB/s rd, 28 MiB/s wr, 312 op/s 2024-08-17T17:16:18.654 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:18 smithi136 ceph-mon[109861]: Upgrade: Updating mon.c 2024-08-17T17:16:18.654 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:18 smithi136 ceph-mon[109861]: Deploying daemon mon.c on smithi136 2024-08-17T17:16:18.655 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:18 smithi136 ceph-mon[109861]: osdmap e356: 8 total, 8 up, 8 in 2024-08-17T17:16:18.655 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:18 smithi136 ceph-mon[109861]: pgmap v20: 137 pgs: 137 active+clean; 2.2 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 6.4 MiB/s rd, 28 MiB/s wr, 312 op/s 2024-08-17T17:16:18.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:18 smithi165 ceph-mon[34868]: Upgrade: Updating mon.c 2024-08-17T17:16:18.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:18 smithi165 ceph-mon[34868]: Deploying daemon mon.c on smithi136 2024-08-17T17:16:18.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:18 smithi165 ceph-mon[34868]: osdmap e356: 8 total, 8 up, 8 in 2024-08-17T17:16:18.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:18 smithi165 ceph-mon[34868]: pgmap v20: 137 pgs: 137 active+clean; 2.2 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 6.4 MiB/s rd, 28 MiB/s wr, 312 op/s 2024-08-17T17:16:19.313 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:19 smithi136 systemd[1]: Stopping Ceph mon.c for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:16:19.677 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:19 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-c[42557]: 2024-08-17T17:16:19.310+0000 7f4e68c83640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:16:19.678 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:19 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-c[42557]: 2024-08-17T17:16:19.310+0000 7f4e68c83640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-08-17T17:16:19.678 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:19 smithi136 podman[114890]: 2024-08-17 17:16:19.381815149 +0000 UTC m=+0.196033128 container died b4e0f3c9e48509bb82a50145b049488b4893c7bfebd449c27534725ae8c67da5 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-c, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=reef-ced649f, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:16:19.738 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMigration.StressLive (16262 ms) 2024-08-17T17:16:19.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:19 smithi136 podman[114890]: 2024-08-17 17:16:19.826442379 +0000 UTC m=+0.640660400 container cleanup b4e0f3c9e48509bb82a50145b049488b4893c7bfebd449c27534725ae8c67da5 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=reef-ced649f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-08-17T17:16:19.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:19 smithi136 bash[114890]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-c 2024-08-17T17:16:20.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 podman[114906]: 2024-08-17 17:16:20.019939361 +0000 UTC m=+0.636124353 container remove b4e0f3c9e48509bb82a50145b049488b4893c7bfebd449c27534725ae8c67da5 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-c, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=reef-ced649f, ceph=True) 2024-08-17T17:16:20.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@mon.c.service: Deactivated successfully. 2024-08-17T17:16:20.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 systemd[1]: Stopped Ceph mon.c for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:16:20.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@mon.c.service: Consumed 9.274s CPU time. 2024-08-17T17:16:20.811 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 systemd[1]: Starting Ceph mon.c for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:16:20.811 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 podman[115040]: 2024-08-17 17:16:20.573209036 +0000 UTC m=+0.017084759 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:16:20.812 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 podman[115040]: 2024-08-17 17:16:20.687673797 +0000 UTC m=+0.131549550 container create 2c2ac98ad9d499e6f24973f19adb814ccd1e5a363b677247051e0e713aecae98 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-08-17T17:16:21.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 podman[115040]: 2024-08-17 17:16:20.932063206 +0000 UTC m=+0.375938916 container init 2c2ac98ad9d499e6f24973f19adb814ccd1e5a363b677247051e0e713aecae98 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-c, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=main-cd8bdbe, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731) 2024-08-17T17:16:21.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 podman[115040]: 2024-08-17 17:16:20.936570032 +0000 UTC m=+0.380445747 container start 2c2ac98ad9d499e6f24973f19adb814ccd1e5a363b677247051e0e713aecae98 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-c, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:16:21.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: set uid:gid to 167:167 (ceph:ceph) 2024-08-17T17:16:21.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev), process ceph-mon, pid 7 2024-08-17T17:16:21.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: pidfile_write: ignore empty --pid-file 2024-08-17T17:16:21.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: load: jerasure load: lrc 2024-08-17T17:16:21.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: RocksDB version: 7.9.2 2024-08-17T17:16:21.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Git sha 0 2024-08-17T17:16:21.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Compile date 2024-08-15 20:43:31 2024-08-17T17:16:21.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: DB SUMMARY 2024-08-17T17:16:21.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: DB Session ID: CRMND9KU1HM2HOB0GDKS 2024-08-17T17:16:21.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: CURRENT file: CURRENT 2024-08-17T17:16:21.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: IDENTITY file: IDENTITY 2024-08-17T17:16:21.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: MANIFEST file: MANIFEST-000010 size: 1413 Bytes 2024-08-17T17:16:21.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-08-17T17:16:21.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 12824320 ; 2024-08-17T17:16:21.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.error_if_exists: 0 2024-08-17T17:16:21.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.create_if_missing: 0 2024-08-17T17:16:21.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.paranoid_checks: 1 2024-08-17T17:16:21.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.flush_verify_memtable_count: 1 2024-08-17T17:16:21.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-08-17T17:16:21.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-08-17T17:16:21.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.env: 0x5567d4592be0 2024-08-17T17:16:21.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.fs: PosixFileSystem 2024-08-17T17:16:21.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.info_log: 0x5567d6829700 2024-08-17T17:16:21.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_file_opening_threads: 16 2024-08-17T17:16:21.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.statistics: (nil) 2024-08-17T17:16:21.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.use_fsync: 0 2024-08-17T17:16:21.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_log_file_size: 0 2024-08-17T17:16:21.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-08-17T17:16:21.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.log_file_time_to_roll: 0 2024-08-17T17:16:21.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.keep_log_file_num: 1000 2024-08-17T17:16:21.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.recycle_log_file_num: 0 2024-08-17T17:16:21.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.allow_fallocate: 1 2024-08-17T17:16:21.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.allow_mmap_reads: 0 2024-08-17T17:16:21.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.allow_mmap_writes: 0 2024-08-17T17:16:21.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.use_direct_reads: 0 2024-08-17T17:16:21.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-08-17T17:16:21.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.create_missing_column_families: 0 2024-08-17T17:16:21.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.db_log_dir: 2024-08-17T17:16:21.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.wal_dir: 2024-08-17T17:16:21.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.table_cache_numshardbits: 6 2024-08-17T17:16:21.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.WAL_ttl_seconds: 0 2024-08-17T17:16:21.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.WAL_size_limit_MB: 0 2024-08-17T17:16:21.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-08-17T17:16:21.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-08-17T17:16:21.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.is_fd_close_on_exec: 1 2024-08-17T17:16:21.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.advise_random_on_open: 1 2024-08-17T17:16:21.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.db_write_buffer_size: 0 2024-08-17T17:16:21.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.write_buffer_manager: 0x5567d682c960 2024-08-17T17:16:21.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-08-17T17:16:21.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-08-17T17:16:21.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.use_adaptive_mutex: 0 2024-08-17T17:16:21.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.rate_limiter: (nil) 2024-08-17T17:16:21.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-08-17T17:16:21.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.wal_recovery_mode: 2 2024-08-17T17:16:21.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.enable_thread_tracking: 0 2024-08-17T17:16:21.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.enable_pipelined_write: 0 2024-08-17T17:16:21.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.unordered_write: 0 2024-08-17T17:16:21.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-08-17T17:16:21.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-08-17T17:16:21.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-08-17T17:16:21.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-08-17T17:16:21.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.row_cache: None 2024-08-17T17:16:21.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.wal_filter: None 2024-08-17T17:16:21.204 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-08-17T17:16:21.204 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.allow_ingest_behind: 0 2024-08-17T17:16:21.204 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.two_write_queues: 0 2024-08-17T17:16:21.205 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.manual_wal_flush: 0 2024-08-17T17:16:21.205 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.wal_compression: 0 2024-08-17T17:16:21.205 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.atomic_flush: 0 2024-08-17T17:16:21.205 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-08-17T17:16:21.205 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.persist_stats_to_disk: 0 2024-08-17T17:16:21.205 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.write_dbid_to_manifest: 0 2024-08-17T17:16:21.205 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.log_readahead_size: 0 2024-08-17T17:16:21.205 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-08-17T17:16:21.205 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.best_efforts_recovery: 0 2024-08-17T17:16:21.205 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-08-17T17:16:21.205 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-08-17T17:16:21.205 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.allow_data_in_errors: 0 2024-08-17T17:16:21.205 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.db_host_id: __hostname__ 2024-08-17T17:16:21.206 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.enforce_single_del_contracts: true 2024-08-17T17:16:21.206 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_background_jobs: 2 2024-08-17T17:16:21.206 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_background_compactions: -1 2024-08-17T17:16:21.206 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_subcompactions: 1 2024-08-17T17:16:21.206 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-08-17T17:16:21.206 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-08-17T17:16:21.206 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.delayed_write_rate : 16777216 2024-08-17T17:16:21.206 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_total_wal_size: 0 2024-08-17T17:16:21.206 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-08-17T17:16:21.207 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.stats_dump_period_sec: 600 2024-08-17T17:16:21.207 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.stats_persist_period_sec: 600 2024-08-17T17:16:21.207 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-08-17T17:16:21.207 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_open_files: -1 2024-08-17T17:16:21.207 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.bytes_per_sync: 0 2024-08-17T17:16:21.207 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.wal_bytes_per_sync: 0 2024-08-17T17:16:21.207 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.strict_bytes_per_sync: 0 2024-08-17T17:16:21.207 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.compaction_readahead_size: 0 2024-08-17T17:16:21.207 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_background_flushes: -1 2024-08-17T17:16:21.207 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Compression algorithms supported: 2024-08-17T17:16:21.207 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: kZSTD supported: 0 2024-08-17T17:16:21.207 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: kXpressCompression supported: 0 2024-08-17T17:16:21.207 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: kBZip2Compression supported: 0 2024-08-17T17:16:21.208 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-08-17T17:16:21.208 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: kLZ4Compression supported: 1 2024-08-17T17:16:21.208 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: kZlibCompression supported: 1 2024-08-17T17:16:21.208 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: kLZ4HCCompression supported: 1 2024-08-17T17:16:21.208 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: kSnappyCompression supported: 1 2024-08-17T17:16:21.208 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Fast CRC32 supported: Supported on x86 2024-08-17T17:16:21.208 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: DMutex implementation: pthread_mutex_t 2024-08-17T17:16:21.208 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000010 2024-08-17T17:16:21.208 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-08-17T17:16:21.208 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-08-17T17:16:21.208 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.merge_operator: 2024-08-17T17:16:21.208 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.compaction_filter: None 2024-08-17T17:16:21.208 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.compaction_filter_factory: None 2024-08-17T17:16:21.208 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.sst_partitioner_factory: None 2024-08-17T17:16:21.209 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.memtable_factory: SkipListFactory 2024-08-17T17:16:21.209 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.table_factory: BlockBasedTable 2024-08-17T17:16:21.209 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5567d6829380) 2024-08-17T17:16:21.209 INFO:journalctl@ceph.mon.c.smithi136.stdout: cache_index_and_filter_blocks: 1 2024-08-17T17:16:21.209 INFO:journalctl@ceph.mon.c.smithi136.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-08-17T17:16:21.209 INFO:journalctl@ceph.mon.c.smithi136.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-08-17T17:16:21.209 INFO:journalctl@ceph.mon.c.smithi136.stdout: pin_top_level_index_and_filter: 1 2024-08-17T17:16:21.209 INFO:journalctl@ceph.mon.c.smithi136.stdout: index_type: 0 2024-08-17T17:16:21.209 INFO:journalctl@ceph.mon.c.smithi136.stdout: data_block_index_type: 0 2024-08-17T17:16:21.210 INFO:journalctl@ceph.mon.c.smithi136.stdout: index_shortening: 1 2024-08-17T17:16:21.210 INFO:journalctl@ceph.mon.c.smithi136.stdout: data_block_hash_table_util_ratio: 0.750000 2024-08-17T17:16:21.210 INFO:journalctl@ceph.mon.c.smithi136.stdout: checksum: 4 2024-08-17T17:16:21.210 INFO:journalctl@ceph.mon.c.smithi136.stdout: no_block_cache: 0 2024-08-17T17:16:21.210 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_cache: 0x5567d681d610 2024-08-17T17:16:21.210 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_cache_name: BinnedLRUCache 2024-08-17T17:16:21.210 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_cache_options: 2024-08-17T17:16:21.210 INFO:journalctl@ceph.mon.c.smithi136.stdout: capacity : 536870912 2024-08-17T17:16:21.210 INFO:journalctl@ceph.mon.c.smithi136.stdout: num_shard_bits : 4 2024-08-17T17:16:21.210 INFO:journalctl@ceph.mon.c.smithi136.stdout: strict_capacity_limit : 0 2024-08-17T17:16:21.210 INFO:journalctl@ceph.mon.c.smithi136.stdout: high_pri_pool_ratio: 0.000 2024-08-17T17:16:21.210 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_cache_compressed: (nil) 2024-08-17T17:16:21.210 INFO:journalctl@ceph.mon.c.smithi136.stdout: persistent_cache: (nil) 2024-08-17T17:16:21.211 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_size: 4096 2024-08-17T17:16:21.211 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_size_deviation: 10 2024-08-17T17:16:21.211 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_restart_interval: 16 2024-08-17T17:16:21.211 INFO:journalctl@ceph.mon.c.smithi136.stdout: index_block_restart_interval: 1 2024-08-17T17:16:21.211 INFO:journalctl@ceph.mon.c.smithi136.stdout: metadata_block_size: 4096 2024-08-17T17:16:21.211 INFO:journalctl@ceph.mon.c.smithi136.stdout: partition_filters: 0 2024-08-17T17:16:21.211 INFO:journalctl@ceph.mon.c.smithi136.stdout: use_delta_encoding: 1 2024-08-17T17:16:21.211 INFO:journalctl@ceph.mon.c.smithi136.stdout: filter_policy: bloomfilter 2024-08-17T17:16:21.211 INFO:journalctl@ceph.mon.c.smithi136.stdout: whole_key_filtering: 1 2024-08-17T17:16:21.211 INFO:journalctl@ceph.mon.c.smithi136.stdout: verify_compression: 0 2024-08-17T17:16:21.211 INFO:journalctl@ceph.mon.c.smithi136.stdout: read_amp_bytes_per_bit: 0 2024-08-17T17:16:21.211 INFO:journalctl@ceph.mon.c.smithi136.stdout: format_version: 5 2024-08-17T17:16:21.211 INFO:journalctl@ceph.mon.c.smithi136.stdout: enable_index_compression: 1 2024-08-17T17:16:21.212 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_align: 0 2024-08-17T17:16:21.212 INFO:journalctl@ceph.mon.c.smithi136.stdout: max_auto_readahead_size: 262144 2024-08-17T17:16:21.212 INFO:journalctl@ceph.mon.c.smithi136.stdout: prepopulate_block_cache: 0 2024-08-17T17:16:21.212 INFO:journalctl@ceph.mon.c.smithi136.stdout: initial_auto_readahead_size: 8192 2024-08-17T17:16:21.212 INFO:journalctl@ceph.mon.c.smithi136.stdout: num_file_reads_for_auto_readahead: 2 2024-08-17T17:16:21.212 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.write_buffer_size: 33554432 2024-08-17T17:16:21.212 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_write_buffer_number: 2 2024-08-17T17:16:21.212 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.compression: NoCompression 2024-08-17T17:16:21.212 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.bottommost_compression: Disabled 2024-08-17T17:16:21.212 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.prefix_extractor: nullptr 2024-08-17T17:16:21.212 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-08-17T17:16:21.212 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.num_levels: 7 2024-08-17T17:16:21.212 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-08-17T17:16:21.213 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-08-17T17:16:21.213 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-08-17T17:16:21.213 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-08-17T17:16:21.213 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-08-17T17:16:21.213 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-08-17T17:16:21.213 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-08-17T17:16:21.213 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-08-17T17:16:21.213 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-08-17T17:16:21.213 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-08-17T17:16:21.213 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-08-17T17:16:21.213 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-08-17T17:16:21.213 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.compression_opts.window_bits: -14 2024-08-17T17:16:21.213 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.compression_opts.level: 32767 2024-08-17T17:16:21.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.compression_opts.strategy: 0 2024-08-17T17:16:21.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-08-17T17:16:21.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-08-17T17:16:21.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-08-17T17:16:21.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-08-17T17:16:21.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.compression_opts.enabled: false 2024-08-17T17:16:21.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-08-17T17:16:21.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-08-17T17:16:21.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-08-17T17:16:21.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-08-17T17:16:21.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.target_file_size_base: 67108864 2024-08-17T17:16:21.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.target_file_size_multiplier: 1 2024-08-17T17:16:21.214 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-08-17T17:16:21.215 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-08-17T17:16:21.215 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-08-17T17:16:21.215 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-08-17T17:16:21.215 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-08-17T17:16:21.215 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-08-17T17:16:21.215 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-08-17T17:16:21.215 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-08-17T17:16:21.215 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-08-17T17:16:21.215 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-08-17T17:16:21.215 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-08-17T17:16:21.215 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-08-17T17:16:21.215 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-08-17T17:16:21.215 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.arena_block_size: 1048576 2024-08-17T17:16:21.216 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-08-17T17:16:21.216 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-08-17T17:16:21.216 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.disable_auto_compactions: 0 2024-08-17T17:16:21.216 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-08-17T17:16:21.216 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-08-17T17:16:21.216 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-08-17T17:16:21.216 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-08-17T17:16:21.216 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-08-17T17:16:21.216 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-08-17T17:16:21.216 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-08-17T17:16:21.216 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-08-17T17:16:21.216 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-08-17T17:16:21.216 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-08-17T17:16:21.216 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-08-17T17:16:21.217 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.inplace_update_support: 0 2024-08-17T17:16:21.217 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.inplace_update_num_locks: 10000 2024-08-17T17:16:21.217 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-08-17T17:16:21.217 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-08-17T17:16:21.217 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.memtable_huge_page_size: 0 2024-08-17T17:16:21.217 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.bloom_locality: 0 2024-08-17T17:16:21.217 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.max_successive_merges: 0 2024-08-17T17:16:21.217 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.optimize_filters_for_hits: 0 2024-08-17T17:16:21.217 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.paranoid_file_checks: 0 2024-08-17T17:16:21.217 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.force_consistency_checks: 1 2024-08-17T17:16:21.218 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.report_bg_io_stats: 0 2024-08-17T17:16:21.218 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.ttl: 2592000 2024-08-17T17:16:21.218 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.periodic_compaction_seconds: 0 2024-08-17T17:16:21.218 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-08-17T17:16:21.218 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-08-17T17:16:21.218 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.enable_blob_files: false 2024-08-17T17:16:21.218 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.min_blob_size: 0 2024-08-17T17:16:21.218 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.blob_file_size: 268435456 2024-08-17T17:16:21.218 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.blob_compression_type: NoCompression 2024-08-17T17:16:21.218 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.enable_blob_garbage_collection: false 2024-08-17T17:16:21.218 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-08-17T17:16:21.218 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-08-17T17:16:21.219 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-08-17T17:16:21.219 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.blob_file_starting_level: 0 2024-08-17T17:16:21.219 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-08-17T17:16:21.219 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-08-17T17:16:21.219 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 29, last_sequence is 13406, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 25 2024-08-17T17:16:21.219 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-08-17T17:16:21.219 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a1d107e2-c9e3-41fd-8204-844cc2196212 2024-08-17T17:16:21.219 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914980964685, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-08-17T17:16:21.219 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-08-17T17:16:21.219 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914980992326, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 8021743, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 13411, "largest_seqno": 18580, "table_properties": {"data_size": 7998821, "index_size": 16434, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 5445, "raw_key_size": 49821, "raw_average_key_size": 22, "raw_value_size": 7951590, "raw_average_value_size": 3654, "num_data_blocks": 706, "num_entries": 2176, "num_filter_entries": 2176, "num_deletions": 269, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1723914980, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a1d107e2-c9e3-41fd-8204-844cc2196212", "db_session_id": "CRMND9KU1HM2HOB0GDKS", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-08-17T17:16:21.220 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914980992491, "job": 1, "event": "recovery_finished"} 2024-08-17T17:16:21.220 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-08-17T17:16:21.220 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-08-17T17:16:21.220 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-08-17T17:16:21.220 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5567d684ce00 2024-08-17T17:16:21.220 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: DB pointer 0x5567d694a000 2024-08-17T17:16:21.220 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-08-17T17:16:21.220 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-08-17T17:16:21.220 INFO:journalctl@ceph.mon.c.smithi136.stdout: ** DB Stats ** 2024-08-17T17:16:21.220 INFO:journalctl@ceph.mon.c.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-17T17:16:21.220 INFO:journalctl@ceph.mon.c.smithi136.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-08-17T17:16:21.221 INFO:journalctl@ceph.mon.c.smithi136.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-17T17:16:21.221 INFO:journalctl@ceph.mon.c.smithi136.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-17T17:16:21.221 INFO:journalctl@ceph.mon.c.smithi136.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-08-17T17:16:21.221 INFO:journalctl@ceph.mon.c.smithi136.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-17T17:16:21.221 INFO:journalctl@ceph.mon.c.smithi136.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-17T17:16:21.221 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-08-17T17:16:21.221 INFO:journalctl@ceph.mon.c.smithi136.stdout: ** Compaction Stats [default] ** 2024-08-17T17:16:21.221 INFO:journalctl@ceph.mon.c.smithi136.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-17T17:16:21.221 INFO:journalctl@ceph.mon.c.smithi136.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-08-17T17:16:21.221 INFO:journalctl@ceph.mon.c.smithi136.stdout: L0 1/0 7.65 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 776.7 0.01 0.00 1 0.010 0 0 0.0 0.0 2024-08-17T17:16:21.221 INFO:journalctl@ceph.mon.c.smithi136.stdout: L6 1/0 8.76 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-08-17T17:16:21.221 INFO:journalctl@ceph.mon.c.smithi136.stdout: Sum 2/0 16.41 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 776.7 0.01 0.00 1 0.010 0 0 0.0 0.0 2024-08-17T17:16:21.222 INFO:journalctl@ceph.mon.c.smithi136.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 776.7 0.01 0.00 1 0.010 0 0 0.0 0.0 2024-08-17T17:16:21.222 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-08-17T17:16:21.222 INFO:journalctl@ceph.mon.c.smithi136.stdout: ** Compaction Stats [default] ** 2024-08-17T17:16:21.222 INFO:journalctl@ceph.mon.c.smithi136.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-17T17:16:21.222 INFO:journalctl@ceph.mon.c.smithi136.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-17T17:16:21.222 INFO:journalctl@ceph.mon.c.smithi136.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 776.7 0.01 0.00 1 0.010 0 0 0.0 0.0 2024-08-17T17:16:21.222 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-08-17T17:16:21.222 INFO:journalctl@ceph.mon.c.smithi136.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-08-17T17:16:21.222 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-08-17T17:16:21.222 INFO:journalctl@ceph.mon.c.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-17T17:16:21.222 INFO:journalctl@ceph.mon.c.smithi136.stdout: Flush(GB): cumulative 0.007, interval 0.007 2024-08-17T17:16:21.222 INFO:journalctl@ceph.mon.c.smithi136.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-08-17T17:16:21.223 INFO:journalctl@ceph.mon.c.smithi136.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-08-17T17:16:21.223 INFO:journalctl@ceph.mon.c.smithi136.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-08-17T17:16:21.223 INFO:journalctl@ceph.mon.c.smithi136.stdout: AddFile(Keys): cumulative 0, interval 0 2024-08-17T17:16:21.223 INFO:journalctl@ceph.mon.c.smithi136.stdout: Cumulative compaction: 0.01 GB write, 247.07 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:16:21.223 INFO:journalctl@ceph.mon.c.smithi136.stdout: Interval compaction: 0.01 GB write, 247.07 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:16:21.223 INFO:journalctl@ceph.mon.c.smithi136.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-08-17T17:16:21.223 INFO:journalctl@ceph.mon.c.smithi136.stdout: Block cache BinnedLRUCache@0x5567d681d610#7 capacity: 512.00 MB usage: 26.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-08-17T17:16:21.223 INFO:journalctl@ceph.mon.c.smithi136.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,6.03 KB,0.00115037%) IndexBlock(1,20.08 KB,0.0038296%) Misc(1,0.00 KB,0%) 2024-08-17T17:16:21.223 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-08-17T17:16:21.223 INFO:journalctl@ceph.mon.c.smithi136.stdout: ** File Read Latency Histogram By Level [default] ** 2024-08-17T17:16:21.223 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: starting mon.c rank 2 at public addrs [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] at bind addrs [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:16:21.224 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: mon.c@-1(???) e3 preinit fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:16:21.224 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: mon.c@-1(???).mds e9 new map 2024-08-17T17:16:21.224 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: mon.c@-1(???).mds e9 print_map 2024-08-17T17:16:21.224 INFO:journalctl@ceph.mon.c.smithi136.stdout: e9 2024-08-17T17:16:21.224 INFO:journalctl@ceph.mon.c.smithi136.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-08-17T17:16:21.224 INFO:journalctl@ceph.mon.c.smithi136.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-08-17T17:16:21.224 INFO:journalctl@ceph.mon.c.smithi136.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-08-17T17:16:21.224 INFO:journalctl@ceph.mon.c.smithi136.stdout: legacy client fscid: 1 2024-08-17T17:16:21.225 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-08-17T17:16:21.225 INFO:journalctl@ceph.mon.c.smithi136.stdout: Filesystem 'foo' (1) 2024-08-17T17:16:21.225 INFO:journalctl@ceph.mon.c.smithi136.stdout: fs_name foo 2024-08-17T17:16:21.225 INFO:journalctl@ceph.mon.c.smithi136.stdout: epoch 8 2024-08-17T17:16:21.225 INFO:journalctl@ceph.mon.c.smithi136.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-08-17T17:16:21.225 INFO:journalctl@ceph.mon.c.smithi136.stdout: created 2024-08-17T17:10:05.962795+0000 2024-08-17T17:16:21.225 INFO:journalctl@ceph.mon.c.smithi136.stdout: modified 2024-08-17T17:10:20.942940+0000 2024-08-17T17:16:21.225 INFO:journalctl@ceph.mon.c.smithi136.stdout: tableserver 0 2024-08-17T17:16:21.225 INFO:journalctl@ceph.mon.c.smithi136.stdout: root 0 2024-08-17T17:16:21.225 INFO:journalctl@ceph.mon.c.smithi136.stdout: session_timeout 60 2024-08-17T17:16:21.225 INFO:journalctl@ceph.mon.c.smithi136.stdout: session_autoclose 300 2024-08-17T17:16:21.225 INFO:journalctl@ceph.mon.c.smithi136.stdout: max_file_size 1099511627776 2024-08-17T17:16:21.226 INFO:journalctl@ceph.mon.c.smithi136.stdout: max_xattr_size 65536 2024-08-17T17:16:21.226 INFO:journalctl@ceph.mon.c.smithi136.stdout: required_client_features {} 2024-08-17T17:16:21.226 INFO:journalctl@ceph.mon.c.smithi136.stdout: last_failure 0 2024-08-17T17:16:21.226 INFO:journalctl@ceph.mon.c.smithi136.stdout: last_failure_osd_epoch 0 2024-08-17T17:16:21.226 INFO:journalctl@ceph.mon.c.smithi136.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-08-17T17:16:21.226 INFO:journalctl@ceph.mon.c.smithi136.stdout: max_mds 1 2024-08-17T17:16:21.226 INFO:journalctl@ceph.mon.c.smithi136.stdout: in 0 2024-08-17T17:16:21.226 INFO:journalctl@ceph.mon.c.smithi136.stdout: up {0=24557} 2024-08-17T17:16:21.227 INFO:journalctl@ceph.mon.c.smithi136.stdout: failed 2024-08-17T17:16:21.227 INFO:journalctl@ceph.mon.c.smithi136.stdout: damaged 2024-08-17T17:16:21.227 INFO:journalctl@ceph.mon.c.smithi136.stdout: stopped 2024-08-17T17:16:21.227 INFO:journalctl@ceph.mon.c.smithi136.stdout: data_pools [4] 2024-08-17T17:16:21.227 INFO:journalctl@ceph.mon.c.smithi136.stdout: metadata_pool 3 2024-08-17T17:16:21.227 INFO:journalctl@ceph.mon.c.smithi136.stdout: inline_data disabled 2024-08-17T17:16:21.227 INFO:journalctl@ceph.mon.c.smithi136.stdout: balancer 2024-08-17T17:16:21.227 INFO:journalctl@ceph.mon.c.smithi136.stdout: bal_rank_mask -1 2024-08-17T17:16:21.227 INFO:journalctl@ceph.mon.c.smithi136.stdout: standby_count_wanted 1 2024-08-17T17:16:21.227 INFO:journalctl@ceph.mon.c.smithi136.stdout: qdb_cluster leader: 0 members: 2024-08-17T17:16:21.227 INFO:journalctl@ceph.mon.c.smithi136.stdout: [mds.foo.smithi136.ltsuqd{0:24557} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.136:6834/764583248,v1:172.21.15.136:6835/764583248] compat {c=[1],r=[1],i=[7ff]}] 2024-08-17T17:16:21.228 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-08-17T17:16:21.228 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-08-17T17:16:21.228 INFO:journalctl@ceph.mon.c.smithi136.stdout: Standby daemons: 2024-08-17T17:16:21.228 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-08-17T17:16:21.228 INFO:journalctl@ceph.mon.c.smithi136.stdout: [mds.foo.smithi165.bmrbhh{-1:24487} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.165:6832/4276964188,v1:172.21.15.165:6833/4276964188] compat {c=[1],r=[1],i=[7ff]}] 2024-08-17T17:16:21.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: mon.c@-1(???).osd e357 crush map has features 3314933000854323200, adjusting msgr requires 2024-08-17T17:16:21.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: mon.c@-1(???).osd e357 crush map has features 432629239337189376, adjusting msgr requires 2024-08-17T17:16:21.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: mon.c@-1(???).osd e357 crush map has features 432629239337189376, adjusting msgr requires 2024-08-17T17:16:21.228 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: mon.c@-1(???).osd e357 crush map has features 432629239337189376, adjusting msgr requires 2024-08-17T17:16:21.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:20 smithi136 ceph-mon[115078]: mon.c@-1(???).paxosservice(auth 1..20) refresh upgraded, format 0 -> 3 2024-08-17T17:16:21.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:21 smithi136 bash[115040]: 2c2ac98ad9d499e6f24973f19adb814ccd1e5a363b677247051e0e713aecae98 2024-08-17T17:16:21.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:21 smithi136 systemd[1]: Started Ceph mon.c for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:16:26.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:26 smithi165 ceph-mon[34868]: osdmap e357: 8 total, 8 up, 8 in 2024-08-17T17:16:26.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:26 smithi136 ceph-mon[109861]: osdmap e357: 8 total, 8 up, 8 in 2024-08-17T17:16:27.024 INFO:teuthology.orchestra.run.smithi136.stdout:true 2024-08-17T17:16:27.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:27 smithi165 ceph-mon[34868]: mon.c calling monitor election 2024-08-17T17:16:27.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:27 smithi165 ceph-mon[34868]: mon.b calling monitor election 2024-08-17T17:16:27.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:27 smithi165 ceph-mon[34868]: mon.a calling monitor election 2024-08-17T17:16:27.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:27 smithi165 ceph-mon[34868]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-08-17T17:16:27.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:27 smithi165 ceph-mon[34868]: monmap epoch 3 2024-08-17T17:16:27.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:27 smithi165 ceph-mon[34868]: fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:16:27.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:27 smithi165 ceph-mon[34868]: last_changed 2024-08-17T17:04:52.440327+0000 2024-08-17T17:16:27.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:27 smithi165 ceph-mon[34868]: created 2024-08-17T17:03:14.749093+0000 2024-08-17T17:16:27.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:27 smithi165 ceph-mon[34868]: min_mon_release 18 (reef) 2024-08-17T17:16:27.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:27 smithi165 ceph-mon[34868]: election_strategy: 1 2024-08-17T17:16:27.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:27 smithi165 ceph-mon[34868]: 0: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.a 2024-08-17T17:16:27.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:27 smithi165 ceph-mon[34868]: 1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-08-17T17:16:27.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:27 smithi165 ceph-mon[34868]: 2: [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] mon.c 2024-08-17T17:16:27.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:27 smithi165 ceph-mon[34868]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 1 up:standby 2024-08-17T17:16:27.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:27 smithi165 ceph-mon[34868]: osdmap e357: 8 total, 8 up, 8 in 2024-08-17T17:16:27.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:27 smithi165 ceph-mon[34868]: mgrmap e39: y(active, since 21s), standbys: x 2024-08-17T17:16:27.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:27 smithi165 ceph-mon[34868]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-08-17T17:16:27.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:27 smithi165 ceph-mon[34868]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-08-17T17:16:27.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:27 smithi165 ceph-mon[34868]: application not enabled on pool 'test-librbd-smithi136-87100-9' 2024-08-17T17:16:27.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:27 smithi165 ceph-mon[34868]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-08-17T17:16:27.619 INFO:teuthology.orchestra.run.smithi136.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-17T17:16:27.619 INFO:teuthology.orchestra.run.smithi136.stdout:alertmanager.a smithi136 *:9093,9094 running (96s) 17s ago 7m 18.8M - 0.27.0 11f11916f8cd c3b9b5b223ab 2024-08-17T17:16:27.619 INFO:teuthology.orchestra.run.smithi136.stdout:grafana.a smithi165 *:3000 running (73s) 16s ago 7m 72.4M - c8b91775d855 04db24fce5e9 2024-08-17T17:16:27.619 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi136.ltsuqd smithi136 running (6m) 17s ago 6m 20.4M - 18.2.4-719-gced649f8 c870753d67c2 faaa06f2950c 2024-08-17T17:16:27.619 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi165.bmrbhh smithi165 running (6m) 16s ago 6m 16.7M - 18.2.4-719-gced649f8 c870753d67c2 c74b00d929cf 2024-08-17T17:16:27.619 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.x smithi165 *:8443,9283,8765 running (2m) 16s ago 11m 452M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 837fd48ff7cb 2024-08-17T17:16:27.619 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.y smithi136 *:8443,9283,8765 running (3m) 17s ago 13m 488M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9acdc6775e48 2024-08-17T17:16:27.619 INFO:teuthology.orchestra.run.smithi136.stdout:mon.a smithi136 running (29s) 17s ago 13m 33.7M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9cf82e613626 2024-08-17T17:16:27.619 INFO:teuthology.orchestra.run.smithi136.stdout:mon.b smithi165 running (11m) 16s ago 11m 64.1M 2048M 18.2.4-719-gced649f8 c870753d67c2 d587e16140e1 2024-08-17T17:16:27.619 INFO:teuthology.orchestra.run.smithi136.stdout:mon.c smithi136 starting - - - 2048M 2024-08-17T17:16:27.620 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.a smithi136 *:9100 running (2m) 17s ago 6m 10.7M - 1.7.0 72c9c2088986 1e4c644ae533 2024-08-17T17:16:27.620 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.b smithi165 *:9100 running (2m) 16s ago 6m 10.9M - 1.7.0 72c9c2088986 7af2d2d68b06 2024-08-17T17:16:27.620 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0 smithi136 running (11m) 17s ago 11m 169M 2480M 18.2.4-719-gced649f8 c870753d67c2 faab916734b0 2024-08-17T17:16:27.620 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1 smithi136 running (10m) 17s ago 10m 124M 2480M 18.2.4-719-gced649f8 c870753d67c2 861686b6ff82 2024-08-17T17:16:27.620 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2 smithi136 running (10m) 17s ago 10m 130M 2480M 18.2.4-719-gced649f8 c870753d67c2 640be4af4156 2024-08-17T17:16:27.620 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3 smithi136 running (9m) 17s ago 9m 122M 2480M 18.2.4-719-gced649f8 c870753d67c2 14b57a745d30 2024-08-17T17:16:27.620 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4 smithi165 running (9m) 16s ago 9m 142M 2480M 18.2.4-719-gced649f8 c870753d67c2 df366c73e1f2 2024-08-17T17:16:27.620 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5 smithi165 running (9m) 16s ago 9m 110M 2480M 18.2.4-719-gced649f8 c870753d67c2 1131665e203a 2024-08-17T17:16:27.620 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6 smithi165 running (8m) 16s ago 8m 153M 2480M 18.2.4-719-gced649f8 c870753d67c2 7bc5fd925293 2024-08-17T17:16:27.620 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7 smithi165 running (8m) 16s ago 8m 116M 2480M 18.2.4-719-gced649f8 c870753d67c2 18382e4ae3b8 2024-08-17T17:16:27.620 INFO:teuthology.orchestra.run.smithi136.stdout:prometheus.a smithi165 *:9095 running (116s) 16s ago 7m 41.4M - 2.51.0 1d3b7f56885b ae8f8b69fe2f 2024-08-17T17:16:27.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[109861]: mon.c calling monitor election 2024-08-17T17:16:27.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[109861]: mon.b calling monitor election 2024-08-17T17:16:27.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[109861]: mon.a calling monitor election 2024-08-17T17:16:27.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[109861]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-08-17T17:16:27.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[109861]: monmap epoch 3 2024-08-17T17:16:27.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[109861]: fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:16:27.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[109861]: last_changed 2024-08-17T17:04:52.440327+0000 2024-08-17T17:16:27.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[109861]: created 2024-08-17T17:03:14.749093+0000 2024-08-17T17:16:27.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[109861]: min_mon_release 18 (reef) 2024-08-17T17:16:27.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[109861]: election_strategy: 1 2024-08-17T17:16:27.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[109861]: 0: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.a 2024-08-17T17:16:27.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[109861]: 1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-08-17T17:16:27.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[109861]: 2: [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] mon.c 2024-08-17T17:16:27.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[109861]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 1 up:standby 2024-08-17T17:16:27.622 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[109861]: osdmap e357: 8 total, 8 up, 8 in 2024-08-17T17:16:27.622 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[109861]: mgrmap e39: y(active, since 21s), standbys: x 2024-08-17T17:16:27.622 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[109861]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-08-17T17:16:27.622 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[109861]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-08-17T17:16:27.622 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[109861]: application not enabled on pool 'test-librbd-smithi136-87100-9' 2024-08-17T17:16:27.622 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[109861]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-08-17T17:16:27.622 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[115078]: osdmap e357: 8 total, 8 up, 8 in 2024-08-17T17:16:27.622 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[115078]: mon.c calling monitor election 2024-08-17T17:16:27.622 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[115078]: mon.b calling monitor election 2024-08-17T17:16:27.622 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[115078]: mon.a calling monitor election 2024-08-17T17:16:27.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[115078]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-08-17T17:16:27.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[115078]: monmap epoch 3 2024-08-17T17:16:27.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[115078]: fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:16:27.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[115078]: last_changed 2024-08-17T17:04:52.440327+0000 2024-08-17T17:16:27.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[115078]: created 2024-08-17T17:03:14.749093+0000 2024-08-17T17:16:27.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[115078]: min_mon_release 18 (reef) 2024-08-17T17:16:27.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[115078]: election_strategy: 1 2024-08-17T17:16:27.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[115078]: 0: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.a 2024-08-17T17:16:27.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[115078]: 1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-08-17T17:16:27.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[115078]: 2: [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] mon.c 2024-08-17T17:16:27.624 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[115078]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 1 up:standby 2024-08-17T17:16:27.624 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[115078]: osdmap e357: 8 total, 8 up, 8 in 2024-08-17T17:16:27.624 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[115078]: mgrmap e39: y(active, since 21s), standbys: x 2024-08-17T17:16:27.624 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[115078]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-08-17T17:16:27.624 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[115078]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-08-17T17:16:27.624 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[115078]: application not enabled on pool 'test-librbd-smithi136-87100-9' 2024-08-17T17:16:27.624 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-mon[115078]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-08-17T17:16:27.624 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:16:27] "GET /metrics HTTP/1.1" 200 38672 "" "Prometheus/2.51.0" 2024-08-17T17:16:27.964 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:16:27.964 INFO:teuthology.orchestra.run.smithi136.stdout: "mon": { 2024-08-17T17:16:27.964 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 1, 2024-08-17T17:16:27.964 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 2 2024-08-17T17:16:27.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:16:27.964 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr": { 2024-08-17T17:16:27.964 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 2 2024-08-17T17:16:27.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:16:27.964 INFO:teuthology.orchestra.run.smithi136.stdout: "osd": { 2024-08-17T17:16:27.965 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 8 2024-08-17T17:16:27.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:16:27.965 INFO:teuthology.orchestra.run.smithi136.stdout: "mds": { 2024-08-17T17:16:27.965 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 2 2024-08-17T17:16:27.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:16:27.965 INFO:teuthology.orchestra.run.smithi136.stdout: "overall": { 2024-08-17T17:16:27.965 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 11, 2024-08-17T17:16:27.965 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 4 2024-08-17T17:16:27.965 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:16:27.965 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:16:28.257 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 57 tests from TestMigration (261502 ms total) 2024-08-17T17:16:28.257 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:16:28.258 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 6 tests from TestGroup 2024-08-17T17:16:28.273 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:16:28.273 INFO:teuthology.orchestra.run.smithi136.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d", 2024-08-17T17:16:28.273 INFO:teuthology.orchestra.run.smithi136.stdout: "in_progress": true, 2024-08-17T17:16:28.273 INFO:teuthology.orchestra.run.smithi136.stdout: "which": "Upgrading daemons of type(s) mon. Upgrade limited to 1 daemons (0 remaining).", 2024-08-17T17:16:28.274 INFO:teuthology.orchestra.run.smithi136.stdout: "services_complete": [], 2024-08-17T17:16:28.274 INFO:teuthology.orchestra.run.smithi136.stdout: "progress": "1/3 daemons upgraded", 2024-08-17T17:16:28.274 INFO:teuthology.orchestra.run.smithi136.stdout: "message": "Currently upgrading mon daemons", 2024-08-17T17:16:28.274 INFO:teuthology.orchestra.run.smithi136.stdout: "is_paused": false 2024-08-17T17:16:28.274 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:16:28.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:28 smithi165 ceph-mon[34868]: from='client.24958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:28.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:28 smithi165 ceph-mon[34868]: osdmap e358: 8 total, 8 up, 8 in 2024-08-17T17:16:28.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:28 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/229450986' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:28.542 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:28 smithi136 ceph-mon[109861]: from='client.24958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:28.543 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:28 smithi136 ceph-mon[109861]: osdmap e358: 8 total, 8 up, 8 in 2024-08-17T17:16:28.543 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:28 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/229450986' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:28.543 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:28 smithi136 ceph-mon[115078]: from='client.24958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:28.543 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:28 smithi136 ceph-mon[115078]: osdmap e358: 8 total, 8 up, 8 in 2024-08-17T17:16:28.543 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:28 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/229450986' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:29.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:29 smithi165 ceph-mon[34868]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:29.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:29 smithi165 ceph-mon[34868]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:29.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:29 smithi165 ceph-mon[34868]: pgmap v27: 105 pgs: 105 active+clean; 2.2 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-08-17T17:16:29.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:29 smithi165 ceph-mon[34868]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:16:29.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:29 smithi165 ceph-mon[34868]: Cluster is now healthy 2024-08-17T17:16:29.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:29 smithi165 ceph-mon[34868]: osdmap e359: 8 total, 8 up, 8 in 2024-08-17T17:16:29.539 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:29 smithi136 ceph-mon[109861]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:29.539 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:29 smithi136 ceph-mon[109861]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:29.539 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:29 smithi136 ceph-mon[109861]: pgmap v27: 105 pgs: 105 active+clean; 2.2 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-08-17T17:16:29.540 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:29 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:16:29.540 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:29 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T17:16:29.540 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:29 smithi136 ceph-mon[109861]: osdmap e359: 8 total, 8 up, 8 in 2024-08-17T17:16:29.540 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:29 smithi136 ceph-mon[115078]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:29.540 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:29 smithi136 ceph-mon[115078]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:29.540 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:29 smithi136 ceph-mon[115078]: pgmap v27: 105 pgs: 105 active+clean; 2.2 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-08-17T17:16:29.540 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:29 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:16:29.540 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:29 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T17:16:29.540 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:29 smithi136 ceph-mon[115078]: osdmap e359: 8 total, 8 up, 8 in 2024-08-17T17:16:30.265 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestGroup.group_create 2024-08-17T17:16:30.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:30 smithi165 ceph-mon[34868]: from='client.34170 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:30.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:30 smithi165 ceph-mon[34868]: osdmap e360: 8 total, 8 up, 8 in 2024-08-17T17:16:30.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:30 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/4256127349' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:16:30.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:30 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:30.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:30 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:30.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:30 smithi136 ceph-mon[109861]: from='client.34170 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:30.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:30 smithi136 ceph-mon[109861]: osdmap e360: 8 total, 8 up, 8 in 2024-08-17T17:16:30.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:30 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4256127349' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:16:30.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:30.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:30.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:30 smithi136 ceph-mon[115078]: from='client.34170 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:30.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:30 smithi136 ceph-mon[115078]: osdmap e360: 8 total, 8 up, 8 in 2024-08-17T17:16:30.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:30 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4256127349' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:16:30.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:30.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:31.144 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:31 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:16:31.004+0000 7f0cc2ea7640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-08-17T17:16:31.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:31 smithi136 ceph-mon[109861]: pgmap v30: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-08-17T17:16:31.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:31 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:16:31.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:31 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4256127349' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:16:31.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:31 smithi136 ceph-mon[109861]: osdmap e361: 8 total, 8 up, 8 in 2024-08-17T17:16:31.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:31 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:31.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:31 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:31.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:31 smithi136 ceph-mon[115078]: pgmap v30: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-08-17T17:16:31.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:31 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:16:31.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:31 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4256127349' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:16:31.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:31 smithi136 ceph-mon[115078]: osdmap e361: 8 total, 8 up, 8 in 2024-08-17T17:16:31.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:31 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:31.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:31 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:31 smithi165 ceph-mon[34868]: pgmap v30: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-08-17T17:16:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:31 smithi165 ceph-mon[34868]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:16:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:31 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/4256127349' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:16:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:31 smithi165 ceph-mon[34868]: osdmap e361: 8 total, 8 up, 8 in 2024-08-17T17:16:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:31 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:31 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:32.307 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestGroup.group_create (2041 ms) 2024-08-17T17:16:32.307 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestGroup.group_createPP 2024-08-17T17:16:32.323 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestGroup.group_createPP (17 ms) 2024-08-17T17:16:32.323 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestGroup.add_image 2024-08-17T17:16:32.359 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestGroup.add_image (36 ms) 2024-08-17T17:16:32.359 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestGroup.add_imagePP 2024-08-17T17:16:32.396 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestGroup.add_imagePP (37 ms) 2024-08-17T17:16:32.396 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestGroup.add_snapshot 2024-08-17T17:16:32.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:32 smithi136 ceph-mon[109861]: osdmap e362: 8 total, 8 up, 8 in 2024-08-17T17:16:32.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:32 smithi136 ceph-mon[109861]: pgmap v33: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-08-17T17:16:32.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:32 smithi136 ceph-mon[115078]: osdmap e362: 8 total, 8 up, 8 in 2024-08-17T17:16:32.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:32 smithi136 ceph-mon[115078]: pgmap v33: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-08-17T17:16:32.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:32 smithi165 ceph-mon[34868]: osdmap e362: 8 total, 8 up, 8 in 2024-08-17T17:16:32.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:32 smithi165 ceph-mon[34868]: pgmap v33: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-08-17T17:16:33.591 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:33 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:16:33.591 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:33 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T17:16:33.591 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:33 smithi136 ceph-mon[109861]: osdmap e363: 8 total, 8 up, 8 in 2024-08-17T17:16:33.591 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:33 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:16:33.591 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:33 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T17:16:33.591 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:33 smithi136 ceph-mon[115078]: osdmap e363: 8 total, 8 up, 8 in 2024-08-17T17:16:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:33 smithi165 ceph-mon[34868]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:16:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:33 smithi165 ceph-mon[34868]: Cluster is now healthy 2024-08-17T17:16:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:33 smithi165 ceph-mon[34868]: osdmap e363: 8 total, 8 up, 8 in 2024-08-17T17:16:34.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: osdmap e364: 8 total, 8 up, 8 in 2024-08-17T17:16:34.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:34.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:34.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:16:34.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:16:34.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:34.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:16:34.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:34.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:34.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all nvmeof 2024-08-17T17:16:34.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:34.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:34.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all node-exporter 2024-08-17T17:16:34.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:34.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-08-17T17:16:34.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-08-17T17:16:34.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-08-17T17:16:34.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-08-17T17:16:34.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:34.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all prometheus 2024-08-17T17:16:34.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-08-17T17:16:34.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-08-17T17:16:34.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:34.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all alertmanager 2024-08-17T17:16:34.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-08-17T17:16:34.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-08-17T17:16:34.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:34.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all grafana 2024-08-17T17:16:34.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-08-17T17:16:34.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: osdmap e364: 8 total, 8 up, 8 in 2024-08-17T17:16:34.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:34.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:34.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:16:34.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:16:34.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:34.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:16:34.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:34.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:34.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all nvmeof 2024-08-17T17:16:34.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:34.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:34.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all node-exporter 2024-08-17T17:16:34.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:34.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-08-17T17:16:34.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-08-17T17:16:34.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-08-17T17:16:34.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-08-17T17:16:34.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:34.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all prometheus 2024-08-17T17:16:34.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-08-17T17:16:34.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-08-17T17:16:34.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:34.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all alertmanager 2024-08-17T17:16:34.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-08-17T17:16:34.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-08-17T17:16:34.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:34.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all grafana 2024-08-17T17:16:34.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-08-17T17:16:34.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-08-17T17:16:34.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:34.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all loki 2024-08-17T17:16:34.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-08-17T17:16:34.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-08-17T17:16:34.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:34.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all promtail 2024-08-17T17:16:34.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-08-17T17:16:34.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-08-17T17:16:34.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: Upgrade: Finalizing container_image settings 2024-08-17T17:16:34.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-08-17T17:16:34.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-08-17T17:16:34.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:16:34.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-08-17T17:16:34.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-08-17T17:16:34.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-08-17T17:16:34.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-08-17T17:16:34.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-08-17T17:16:34.703 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-08-17T17:16:34.704 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:16:34.704 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-08-17T17:16:34.704 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-08-17T17:16:34.704 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-08-17T17:16:34.704 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-08-17T17:16:34.704 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:34.704 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all loki 2024-08-17T17:16:34.705 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-08-17T17:16:34.705 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-08-17T17:16:34.705 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:34.705 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all promtail 2024-08-17T17:16:34.705 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-08-17T17:16:34.705 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-08-17T17:16:34.705 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: Upgrade: Finalizing container_image settings 2024-08-17T17:16:34.705 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-08-17T17:16:34.705 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-08-17T17:16:34.705 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:16:34.705 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-08-17T17:16:34.705 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-08-17T17:16:34.705 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-08-17T17:16:34.705 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-08-17T17:16:34.706 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-08-17T17:16:34.706 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-08-17T17:16:34.706 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:16:34.706 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-08-17T17:16:34.706 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-08-17T17:16:34.706 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-08-17T17:16:34.706 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-08-17T17:16:34.706 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-08-17T17:16:34.706 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:16:34.706 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:16:34.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:16:34.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:16:34.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:16:34.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:16:34.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: Upgrade: Complete! 2024-08-17T17:16:34.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-08-17T17:16:34.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-08-17T17:16:34.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:16:34.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:16:34.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:16:34.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:34.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: pgmap v36: 105 pgs: 105 active+clean; 582 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:16:34.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:16:34.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:16:34.708 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:16:34.708 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:34.708 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-08-17T17:16:34.708 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-08-17T17:16:34.708 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:16:34.708 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:16:34.708 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:16:34.708 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:16:34.708 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:16:34.708 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:16:34.708 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: Upgrade: Complete! 2024-08-17T17:16:34.708 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-08-17T17:16:34.709 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-08-17T17:16:34.709 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:16:34.709 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:16:34.709 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:16:34.709 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:34.709 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: pgmap v36: 105 pgs: 105 active+clean; 582 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:16:34.709 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:16:34.709 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:16:34.709 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:16:34.709 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:34.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: osdmap e364: 8 total, 8 up, 8 in 2024-08-17T17:16:34.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:34.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:34.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:16:34.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:16:34.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:34.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:16:34.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:34.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:34.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: Upgrade: Setting container_image for all nvmeof 2024-08-17T17:16:34.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:34.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:34.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: Upgrade: Setting container_image for all node-exporter 2024-08-17T17:16:34.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:34.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-08-17T17:16:34.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-08-17T17:16:34.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-08-17T17:16:34.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-08-17T17:16:34.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:34.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: Upgrade: Setting container_image for all prometheus 2024-08-17T17:16:34.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-08-17T17:16:34.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-08-17T17:16:34.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:34.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: Upgrade: Setting container_image for all alertmanager 2024-08-17T17:16:34.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-08-17T17:16:34.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-08-17T17:16:34.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:34.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: Upgrade: Setting container_image for all grafana 2024-08-17T17:16:34.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-08-17T17:16:34.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-08-17T17:16:34.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:34.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: Upgrade: Setting container_image for all loki 2024-08-17T17:16:34.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-08-17T17:16:34.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-08-17T17:16:34.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:34.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: Upgrade: Setting container_image for all promtail 2024-08-17T17:16:34.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-08-17T17:16:34.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-08-17T17:16:34.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: Upgrade: Finalizing container_image settings 2024-08-17T17:16:34.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-08-17T17:16:34.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-08-17T17:16:34.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:16:34.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-08-17T17:16:34.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-08-17T17:16:34.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-08-17T17:16:34.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-08-17T17:16:34.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-08-17T17:16:34.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-08-17T17:16:34.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:16:34.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-08-17T17:16:34.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-08-17T17:16:34.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-08-17T17:16:34.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-08-17T17:16:34.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-08-17T17:16:34.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:16:34.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:16:34.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:16:34.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:16:34.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:16:34.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:16:34.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: Upgrade: Complete! 2024-08-17T17:16:34.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-08-17T17:16:34.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-08-17T17:16:34.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:16:34.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:16:34.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:16:34.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:34.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: pgmap v36: 105 pgs: 105 active+clean; 582 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:16:34.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:16:34.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:16:34.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:16:34.768 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:35.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:35 smithi136 ceph-mon[109861]: osdmap e365: 8 total, 8 up, 8 in 2024-08-17T17:16:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:35 smithi136 ceph-mon[115078]: osdmap e365: 8 total, 8 up, 8 in 2024-08-17T17:16:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:35 smithi165 ceph-mon[34868]: osdmap e365: 8 total, 8 up, 8 in 2024-08-17T17:16:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:36 smithi136 ceph-mon[109861]: osdmap e366: 8 total, 8 up, 8 in 2024-08-17T17:16:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:36 smithi136 ceph-mon[109861]: pgmap v39: 105 pgs: 105 active+clean; 582 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:16:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:16:36.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:36 smithi136 ceph-mon[115078]: osdmap e366: 8 total, 8 up, 8 in 2024-08-17T17:16:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:36 smithi136 ceph-mon[115078]: pgmap v39: 105 pgs: 105 active+clean; 582 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:16:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:16:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:36 smithi165 ceph-mon[34868]: osdmap e366: 8 total, 8 up, 8 in 2024-08-17T17:16:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:36 smithi165 ceph-mon[34868]: pgmap v39: 105 pgs: 105 active+clean; 582 KiB data, 374 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:16:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:36 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:36 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:16:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:36 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:37 smithi165 ceph-mon[34868]: osdmap e367: 8 total, 8 up, 8 in 2024-08-17T17:16:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:37 smithi136 ceph-mon[109861]: osdmap e367: 8 total, 8 up, 8 in 2024-08-17T17:16:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:37 smithi136 ceph-mon[115078]: osdmap e367: 8 total, 8 up, 8 in 2024-08-17T17:16:37.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:16:37] "GET /metrics HTTP/1.1" 200 36257 "" "Prometheus/2.51.0" 2024-08-17T17:16:38.344 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestGroup.add_snapshot (5947 ms) 2024-08-17T17:16:38.344 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestGroup.add_snapshotPP 2024-08-17T17:16:38.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:38 smithi136 ceph-mon[109861]: osdmap e368: 8 total, 8 up, 8 in 2024-08-17T17:16:38.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:38 smithi136 ceph-mon[109861]: pgmap v42: 105 pgs: 105 active+clean; 582 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 164 KiB/s rd, 18 KiB/s wr, 221 op/s 2024-08-17T17:16:38.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:38 smithi136 ceph-mon[115078]: osdmap e368: 8 total, 8 up, 8 in 2024-08-17T17:16:38.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:38 smithi136 ceph-mon[115078]: pgmap v42: 105 pgs: 105 active+clean; 582 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 164 KiB/s rd, 18 KiB/s wr, 221 op/s 2024-08-17T17:16:38.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:38 smithi165 ceph-mon[34868]: osdmap e368: 8 total, 8 up, 8 in 2024-08-17T17:16:38.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:38 smithi165 ceph-mon[34868]: pgmap v42: 105 pgs: 105 active+clean; 582 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 164 KiB/s rd, 18 KiB/s wr, 221 op/s 2024-08-17T17:16:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:39 smithi136 ceph-mon[109861]: osdmap e369: 8 total, 8 up, 8 in 2024-08-17T17:16:39.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:39 smithi136 ceph-mon[109861]: osdmap e370: 8 total, 8 up, 8 in 2024-08-17T17:16:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:39 smithi136 ceph-mon[115078]: osdmap e369: 8 total, 8 up, 8 in 2024-08-17T17:16:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:39 smithi136 ceph-mon[115078]: osdmap e370: 8 total, 8 up, 8 in 2024-08-17T17:16:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:39 smithi165 ceph-mon[34868]: osdmap e369: 8 total, 8 up, 8 in 2024-08-17T17:16:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:39 smithi165 ceph-mon[34868]: osdmap e370: 8 total, 8 up, 8 in 2024-08-17T17:16:41.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:40 smithi136 ceph-mon[109861]: osdmap e371: 8 total, 8 up, 8 in 2024-08-17T17:16:41.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:40 smithi136 ceph-mon[109861]: pgmap v46: 105 pgs: 105 active+clean; 582 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 187 KiB/s rd, 21 KiB/s wr, 252 op/s 2024-08-17T17:16:41.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:40 smithi136 ceph-mon[115078]: osdmap e371: 8 total, 8 up, 8 in 2024-08-17T17:16:41.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:40 smithi136 ceph-mon[115078]: pgmap v46: 105 pgs: 105 active+clean; 582 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 187 KiB/s rd, 21 KiB/s wr, 252 op/s 2024-08-17T17:16:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:40 smithi165 ceph-mon[34868]: osdmap e371: 8 total, 8 up, 8 in 2024-08-17T17:16:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:40 smithi165 ceph-mon[34868]: pgmap v46: 105 pgs: 105 active+clean; 582 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 187 KiB/s rd, 21 KiB/s wr, 252 op/s 2024-08-17T17:16:42.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:41 smithi136 ceph-mon[109861]: osdmap e372: 8 total, 8 up, 8 in 2024-08-17T17:16:42.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:41 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:42.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:41 smithi136 ceph-mon[115078]: osdmap e372: 8 total, 8 up, 8 in 2024-08-17T17:16:42.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:41 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:41 smithi165 ceph-mon[34868]: osdmap e372: 8 total, 8 up, 8 in 2024-08-17T17:16:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:41 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:16:43.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:42 smithi136 ceph-mon[109861]: osdmap e373: 8 total, 8 up, 8 in 2024-08-17T17:16:43.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:42 smithi136 ceph-mon[109861]: pgmap v49: 105 pgs: 105 active+clean; 582 KiB data, 375 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:16:43.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:42 smithi136 ceph-mon[115078]: osdmap e373: 8 total, 8 up, 8 in 2024-08-17T17:16:43.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:42 smithi136 ceph-mon[115078]: pgmap v49: 105 pgs: 105 active+clean; 582 KiB data, 375 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:16:43.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:42 smithi165 ceph-mon[34868]: osdmap e373: 8 total, 8 up, 8 in 2024-08-17T17:16:43.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:42 smithi165 ceph-mon[34868]: pgmap v49: 105 pgs: 105 active+clean; 582 KiB data, 375 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:16:43.796 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestGroup.add_snapshotPP (5453 ms) 2024-08-17T17:16:44.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:43 smithi136 ceph-mon[109861]: osdmap e374: 8 total, 8 up, 8 in 2024-08-17T17:16:44.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:43 smithi136 ceph-mon[109861]: osdmap e375: 8 total, 8 up, 8 in 2024-08-17T17:16:44.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:43 smithi136 ceph-mon[115078]: osdmap e374: 8 total, 8 up, 8 in 2024-08-17T17:16:44.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:43 smithi136 ceph-mon[115078]: osdmap e375: 8 total, 8 up, 8 in 2024-08-17T17:16:44.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:43 smithi165 ceph-mon[34868]: osdmap e374: 8 total, 8 up, 8 in 2024-08-17T17:16:44.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:43 smithi165 ceph-mon[34868]: osdmap e375: 8 total, 8 up, 8 in 2024-08-17T17:16:44.808 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 6 tests from TestGroup (13531 ms total) 2024-08-17T17:16:44.808 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:16:44.808 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 31 tests from TestDeepCopy 2024-08-17T17:16:45.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:44 smithi136 ceph-mon[109861]: pgmap v52: 105 pgs: 105 active+clean; 583 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 12 KiB/s wr, 161 op/s 2024-08-17T17:16:45.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:44 smithi136 ceph-mon[115078]: pgmap v52: 105 pgs: 105 active+clean; 583 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 12 KiB/s wr, 161 op/s 2024-08-17T17:16:45.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:44 smithi165 ceph-mon[34868]: pgmap v52: 105 pgs: 105 active+clean; 583 KiB data, 375 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 12 KiB/s wr, 161 op/s 2024-08-17T17:16:46.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:45 smithi136 ceph-mon[109861]: osdmap e376: 8 total, 8 up, 8 in 2024-08-17T17:16:46.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:45 smithi136 ceph-mon[115078]: osdmap e376: 8 total, 8 up, 8 in 2024-08-17T17:16:46.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:45 smithi165 ceph-mon[34868]: osdmap e376: 8 total, 8 up, 8 in 2024-08-17T17:16:46.821 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.Empty 2024-08-17T17:16:47.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:46 smithi136 ceph-mon[109861]: osdmap e377: 8 total, 8 up, 8 in 2024-08-17T17:16:47.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:46 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1119805645' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:16:47.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:46 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:16:47.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:46 smithi136 ceph-mon[109861]: pgmap v55: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 375 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:16:47.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:46 smithi136 ceph-mon[115078]: osdmap e377: 8 total, 8 up, 8 in 2024-08-17T17:16:47.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:46 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1119805645' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:16:47.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:46 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:16:47.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:46 smithi136 ceph-mon[115078]: pgmap v55: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 375 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:16:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:46 smithi165 ceph-mon[34868]: osdmap e377: 8 total, 8 up, 8 in 2024-08-17T17:16:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:46 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/1119805645' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:16:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:46 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:16:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:46 smithi165 ceph-mon[34868]: pgmap v55: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 375 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:16:47.831 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:16:47] "GET /metrics HTTP/1.1" 200 36257 "" "Prometheus/2.51.0" 2024-08-17T17:16:48.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:47 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:16:48.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:47 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:16:48.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:47 smithi136 ceph-mon[109861]: osdmap e378: 8 total, 8 up, 8 in 2024-08-17T17:16:48.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:47 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:16:48.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:47 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:16:48.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:47 smithi136 ceph-mon[115078]: osdmap e378: 8 total, 8 up, 8 in 2024-08-17T17:16:48.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:47 smithi165 ceph-mon[34868]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:16:48.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:47 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:16:48.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:47 smithi165 ceph-mon[34868]: osdmap e378: 8 total, 8 up, 8 in 2024-08-17T17:16:48.959 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestDeepCopy.Empty (2139 ms) 2024-08-17T17:16:48.960 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.NoSnaps 2024-08-17T17:16:49.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:48 smithi136 ceph-mon[109861]: pgmap v57: 105 pgs: 12 creating+peering, 20 creating+activating, 73 active+clean; 582 KiB data, 380 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:16:49.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:48 smithi136 ceph-mon[109861]: osdmap e379: 8 total, 8 up, 8 in 2024-08-17T17:16:49.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:48 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:16:49.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:48 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T17:16:49.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:48 smithi136 ceph-mon[115078]: pgmap v57: 105 pgs: 12 creating+peering, 20 creating+activating, 73 active+clean; 582 KiB data, 380 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:16:49.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:48 smithi136 ceph-mon[115078]: osdmap e379: 8 total, 8 up, 8 in 2024-08-17T17:16:49.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:48 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:16:49.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:48 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T17:16:49.197 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestDeepCopy.NoSnaps (239 ms) 2024-08-17T17:16:49.198 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.Snaps 2024-08-17T17:16:49.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:48 smithi165 ceph-mon[34868]: pgmap v57: 105 pgs: 12 creating+peering, 20 creating+activating, 73 active+clean; 582 KiB data, 380 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:16:49.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:48 smithi165 ceph-mon[34868]: osdmap e379: 8 total, 8 up, 8 in 2024-08-17T17:16:49.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:48 smithi165 ceph-mon[34868]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:16:49.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:48 smithi165 ceph-mon[34868]: Cluster is now healthy 2024-08-17T17:16:50.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:49 smithi136 ceph-mon[109861]: osdmap e380: 8 total, 8 up, 8 in 2024-08-17T17:16:50.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:49 smithi136 ceph-mon[115078]: osdmap e380: 8 total, 8 up, 8 in 2024-08-17T17:16:50.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:49 smithi165 ceph-mon[34868]: osdmap e380: 8 total, 8 up, 8 in 2024-08-17T17:16:51.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:50 smithi136 ceph-mon[109861]: pgmap v60: 105 pgs: 12 creating+peering, 20 creating+activating, 73 active+clean; 582 KiB data, 380 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:16:51.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:50 smithi136 ceph-mon[109861]: osdmap e381: 8 total, 8 up, 8 in 2024-08-17T17:16:51.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:50 smithi136 ceph-mon[115078]: pgmap v60: 105 pgs: 12 creating+peering, 20 creating+activating, 73 active+clean; 582 KiB data, 380 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:16:51.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:50 smithi136 ceph-mon[115078]: osdmap e381: 8 total, 8 up, 8 in 2024-08-17T17:16:51.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:50 smithi165 ceph-mon[34868]: pgmap v60: 105 pgs: 12 creating+peering, 20 creating+activating, 73 active+clean; 582 KiB data, 380 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:16:51.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:50 smithi165 ceph-mon[34868]: osdmap e381: 8 total, 8 up, 8 in 2024-08-17T17:16:52.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:51 smithi136 ceph-mon[109861]: osdmap e382: 8 total, 8 up, 8 in 2024-08-17T17:16:52.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:16:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:51 smithi136 ceph-mon[115078]: osdmap e382: 8 total, 8 up, 8 in 2024-08-17T17:16:52.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:16:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:51 smithi165 ceph-mon[34868]: osdmap e382: 8 total, 8 up, 8 in 2024-08-17T17:16:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:51 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:16:53.143 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestDeepCopy.Snaps (3945 ms) 2024-08-17T17:16:53.144 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.SnapDiscard 2024-08-17T17:16:53.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:52 smithi136 ceph-mon[115078]: pgmap v63: 105 pgs: 5 creating+peering, 20 creating+activating, 80 active+clean; 13 MiB data, 416 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 62 op/s 2024-08-17T17:16:53.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:52 smithi136 ceph-mon[115078]: osdmap e383: 8 total, 8 up, 8 in 2024-08-17T17:16:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:52 smithi136 ceph-mon[109861]: pgmap v63: 105 pgs: 5 creating+peering, 20 creating+activating, 80 active+clean; 13 MiB data, 416 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 62 op/s 2024-08-17T17:16:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:52 smithi136 ceph-mon[109861]: osdmap e383: 8 total, 8 up, 8 in 2024-08-17T17:16:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:52 smithi165 ceph-mon[34868]: pgmap v63: 105 pgs: 5 creating+peering, 20 creating+activating, 80 active+clean; 13 MiB data, 416 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.0 MiB/s wr, 62 op/s 2024-08-17T17:16:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:52 smithi165 ceph-mon[34868]: osdmap e383: 8 total, 8 up, 8 in 2024-08-17T17:16:54.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:53 smithi136 ceph-mon[109861]: osdmap e384: 8 total, 8 up, 8 in 2024-08-17T17:16:54.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:53 smithi136 ceph-mon[115078]: osdmap e384: 8 total, 8 up, 8 in 2024-08-17T17:16:54.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:53 smithi165 ceph-mon[34868]: osdmap e384: 8 total, 8 up, 8 in 2024-08-17T17:16:54.995 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestDeepCopy.SnapDiscard (1851 ms) 2024-08-17T17:16:54.995 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.CloneDiscard 2024-08-17T17:16:55.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:54 smithi136 ceph-mon[109861]: pgmap v66: 105 pgs: 105 active+clean; 49 MiB data, 542 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 12 MiB/s wr, 174 op/s 2024-08-17T17:16:55.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:54 smithi136 ceph-mon[109861]: osdmap e385: 8 total, 8 up, 8 in 2024-08-17T17:16:55.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:54 smithi136 ceph-mon[115078]: pgmap v66: 105 pgs: 105 active+clean; 49 MiB data, 542 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 12 MiB/s wr, 174 op/s 2024-08-17T17:16:55.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:54 smithi136 ceph-mon[115078]: osdmap e385: 8 total, 8 up, 8 in 2024-08-17T17:16:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:54 smithi165 ceph-mon[34868]: pgmap v66: 105 pgs: 105 active+clean; 49 MiB data, 542 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 12 MiB/s wr, 174 op/s 2024-08-17T17:16:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:54 smithi165 ceph-mon[34868]: osdmap e385: 8 total, 8 up, 8 in 2024-08-17T17:16:56.088 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestDeepCopy.CloneDiscard (1094 ms) 2024-08-17T17:16:56.088 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.CloneShrink 2024-08-17T17:16:56.162 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:55 smithi165 ceph-mon[34868]: osdmap e386: 8 total, 8 up, 8 in 2024-08-17T17:16:56.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:55 smithi136 ceph-mon[109861]: osdmap e386: 8 total, 8 up, 8 in 2024-08-17T17:16:56.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:55 smithi136 ceph-mon[115078]: osdmap e386: 8 total, 8 up, 8 in 2024-08-17T17:16:57.050 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestDeepCopy.CloneShrink (962 ms) 2024-08-17T17:16:57.050 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.CloneExpand 2024-08-17T17:16:57.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:56 smithi136 ceph-mon[109861]: pgmap v69: 105 pgs: 105 active+clean; 49 MiB data, 542 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 9.0 MiB/s wr, 111 op/s 2024-08-17T17:16:57.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:56 smithi136 ceph-mon[109861]: osdmap e387: 8 total, 8 up, 8 in 2024-08-17T17:16:57.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:56 smithi136 ceph-mon[115078]: pgmap v69: 105 pgs: 105 active+clean; 49 MiB data, 542 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 9.0 MiB/s wr, 111 op/s 2024-08-17T17:16:57.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:56 smithi136 ceph-mon[115078]: osdmap e387: 8 total, 8 up, 8 in 2024-08-17T17:16:57.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:56 smithi165 ceph-mon[34868]: pgmap v69: 105 pgs: 105 active+clean; 49 MiB data, 542 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 9.0 MiB/s wr, 111 op/s 2024-08-17T17:16:57.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:56 smithi165 ceph-mon[34868]: osdmap e387: 8 total, 8 up, 8 in 2024-08-17T17:16:57.904 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:16:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:16:57] "GET /metrics HTTP/1.1" 200 36229 "" "Prometheus/2.51.0" 2024-08-17T17:16:58.121 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestDeepCopy.CloneExpand (1071 ms) 2024-08-17T17:16:58.121 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.CloneHideParent 2024-08-17T17:16:58.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:57 smithi136 ceph-mon[109861]: osdmap e388: 8 total, 8 up, 8 in 2024-08-17T17:16:58.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:57 smithi136 ceph-mon[115078]: osdmap e388: 8 total, 8 up, 8 in 2024-08-17T17:16:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:57 smithi165 ceph-mon[34868]: osdmap e388: 8 total, 8 up, 8 in 2024-08-17T17:16:59.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:58 smithi136 ceph-mon[109861]: pgmap v72: 105 pgs: 105 active+clean; 65 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 24 MiB/s rd, 4.0 MiB/s wr, 289 op/s 2024-08-17T17:16:59.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:58 smithi136 ceph-mon[109861]: osdmap e389: 8 total, 8 up, 8 in 2024-08-17T17:16:59.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:58 smithi136 ceph-mon[109861]: osdmap e390: 8 total, 8 up, 8 in 2024-08-17T17:16:59.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:58 smithi136 ceph-mon[115078]: pgmap v72: 105 pgs: 105 active+clean; 65 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 24 MiB/s rd, 4.0 MiB/s wr, 289 op/s 2024-08-17T17:16:59.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:58 smithi136 ceph-mon[115078]: osdmap e389: 8 total, 8 up, 8 in 2024-08-17T17:16:59.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:58 smithi136 ceph-mon[115078]: osdmap e390: 8 total, 8 up, 8 in 2024-08-17T17:16:59.224 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'sleep 60' 2024-08-17T17:16:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:58 smithi165 ceph-mon[34868]: pgmap v72: 105 pgs: 105 active+clean; 65 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 24 MiB/s rd, 4.0 MiB/s wr, 289 op/s 2024-08-17T17:16:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:58 smithi165 ceph-mon[34868]: osdmap e389: 8 total, 8 up, 8 in 2024-08-17T17:16:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:58 smithi165 ceph-mon[34868]: osdmap e390: 8 total, 8 up, 8 in 2024-08-17T17:16:59.428 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:17:00.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:59 smithi136 ceph-mon[109861]: from='client.44114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:00.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:16:59 smithi136 ceph-mon[109861]: osdmap e391: 8 total, 8 up, 8 in 2024-08-17T17:17:00.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:59 smithi136 ceph-mon[115078]: from='client.44114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:00.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:16:59 smithi136 ceph-mon[115078]: osdmap e391: 8 total, 8 up, 8 in 2024-08-17T17:17:00.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:59 smithi165 ceph-mon[34868]: from='client.44114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:00.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:16:59 smithi165 ceph-mon[34868]: osdmap e391: 8 total, 8 up, 8 in 2024-08-17T17:17:01.015 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:00 smithi136 ceph-mon[109861]: pgmap v76: 105 pgs: 105 active+clean; 65 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 25 MiB/s rd, 4.1 MiB/s wr, 294 op/s 2024-08-17T17:17:01.015 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:00 smithi136 ceph-mon[109861]: osdmap e392: 8 total, 8 up, 8 in 2024-08-17T17:17:01.015 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:00 smithi136 ceph-mon[115078]: pgmap v76: 105 pgs: 105 active+clean; 65 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 25 MiB/s rd, 4.1 MiB/s wr, 294 op/s 2024-08-17T17:17:01.015 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:00 smithi136 ceph-mon[115078]: osdmap e392: 8 total, 8 up, 8 in 2024-08-17T17:17:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:00 smithi165 ceph-mon[34868]: pgmap v76: 105 pgs: 105 active+clean; 65 MiB data, 599 MiB used, 715 GiB / 715 GiB avail; 25 MiB/s rd, 4.1 MiB/s wr, 294 op/s 2024-08-17T17:17:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:00 smithi165 ceph-mon[34868]: osdmap e392: 8 total, 8 up, 8 in 2024-08-17T17:17:03.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:02 smithi136 ceph-mon[109861]: osdmap e393: 8 total, 8 up, 8 in 2024-08-17T17:17:03.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:02 smithi136 ceph-mon[109861]: pgmap v79: 105 pgs: 105 active+clean; 65 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 4.3 KiB/s wr, 59 op/s 2024-08-17T17:17:03.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:02 smithi136 ceph-mon[115078]: osdmap e393: 8 total, 8 up, 8 in 2024-08-17T17:17:03.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:02 smithi136 ceph-mon[115078]: pgmap v79: 105 pgs: 105 active+clean; 65 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 4.3 KiB/s wr, 59 op/s 2024-08-17T17:17:03.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:02 smithi165 ceph-mon[34868]: osdmap e393: 8 total, 8 up, 8 in 2024-08-17T17:17:03.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:02 smithi165 ceph-mon[34868]: pgmap v79: 105 pgs: 105 active+clean; 65 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 4.3 KiB/s wr, 59 op/s 2024-08-17T17:17:04.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:03 smithi136 ceph-mon[109861]: osdmap e394: 8 total, 8 up, 8 in 2024-08-17T17:17:04.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:03 smithi136 ceph-mon[115078]: osdmap e394: 8 total, 8 up, 8 in 2024-08-17T17:17:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:03 smithi165 ceph-mon[34868]: osdmap e394: 8 total, 8 up, 8 in 2024-08-17T17:17:04.929 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestDeepCopy.CloneHideParent (6808 ms) 2024-08-17T17:17:04.929 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.Clone 2024-08-17T17:17:05.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:04 smithi136 ceph-mon[109861]: osdmap e395: 8 total, 8 up, 8 in 2024-08-17T17:17:05.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:04 smithi136 ceph-mon[109861]: pgmap v82: 105 pgs: 105 active+clean; 65 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 326 KiB/s rd, 26 KiB/s wr, 445 op/s 2024-08-17T17:17:05.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:04 smithi136 ceph-mon[115078]: osdmap e395: 8 total, 8 up, 8 in 2024-08-17T17:17:05.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:04 smithi136 ceph-mon[115078]: pgmap v82: 105 pgs: 105 active+clean; 65 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 326 KiB/s rd, 26 KiB/s wr, 445 op/s 2024-08-17T17:17:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:04 smithi165 ceph-mon[34868]: osdmap e395: 8 total, 8 up, 8 in 2024-08-17T17:17:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:04 smithi165 ceph-mon[34868]: pgmap v82: 105 pgs: 105 active+clean; 65 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 326 KiB/s rd, 26 KiB/s wr, 445 op/s 2024-08-17T17:17:06.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:05 smithi136 ceph-mon[109861]: osdmap e396: 8 total, 8 up, 8 in 2024-08-17T17:17:06.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:05 smithi136 ceph-mon[115078]: osdmap e396: 8 total, 8 up, 8 in 2024-08-17T17:17:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:05 smithi165 ceph-mon[34868]: osdmap e396: 8 total, 8 up, 8 in 2024-08-17T17:17:07.147 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestDeepCopy.Clone (2218 ms) 2024-08-17T17:17:07.147 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.CloneFlatten 2024-08-17T17:17:07.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:06 smithi136 ceph-mon[109861]: osdmap e397: 8 total, 8 up, 8 in 2024-08-17T17:17:07.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:06 smithi136 ceph-mon[109861]: pgmap v85: 105 pgs: 105 active+clean; 65 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 286 KiB/s rd, 22 KiB/s wr, 386 op/s 2024-08-17T17:17:07.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:17:07.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:06 smithi136 ceph-mon[115078]: osdmap e397: 8 total, 8 up, 8 in 2024-08-17T17:17:07.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:06 smithi136 ceph-mon[115078]: pgmap v85: 105 pgs: 105 active+clean; 65 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 286 KiB/s rd, 22 KiB/s wr, 386 op/s 2024-08-17T17:17:07.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:17:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:06 smithi165 ceph-mon[34868]: osdmap e397: 8 total, 8 up, 8 in 2024-08-17T17:17:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:06 smithi165 ceph-mon[34868]: pgmap v85: 105 pgs: 105 active+clean; 65 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 286 KiB/s rd, 22 KiB/s wr, 386 op/s 2024-08-17T17:17:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:17:07.820 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:17:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:17:07] "GET /metrics HTTP/1.1" 200 36291 "" "Prometheus/2.51.0" 2024-08-17T17:17:08.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:07 smithi136 ceph-mon[109861]: osdmap e398: 8 total, 8 up, 8 in 2024-08-17T17:17:08.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:07 smithi136 ceph-mon[115078]: osdmap e398: 8 total, 8 up, 8 in 2024-08-17T17:17:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:07 smithi165 ceph-mon[34868]: osdmap e398: 8 total, 8 up, 8 in 2024-08-17T17:17:09.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:08 smithi136 ceph-mon[109861]: osdmap e399: 8 total, 8 up, 8 in 2024-08-17T17:17:09.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:08 smithi136 ceph-mon[109861]: pgmap v88: 105 pgs: 105 active+clean; 85 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 MiB/s wr, 174 op/s 2024-08-17T17:17:09.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:08 smithi136 ceph-mon[115078]: osdmap e399: 8 total, 8 up, 8 in 2024-08-17T17:17:09.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:08 smithi136 ceph-mon[115078]: pgmap v88: 105 pgs: 105 active+clean; 85 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 MiB/s wr, 174 op/s 2024-08-17T17:17:09.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:08 smithi165 ceph-mon[34868]: osdmap e399: 8 total, 8 up, 8 in 2024-08-17T17:17:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:08 smithi165 ceph-mon[34868]: pgmap v88: 105 pgs: 105 active+clean; 85 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 MiB/s wr, 174 op/s 2024-08-17T17:17:09.386 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestDeepCopy.CloneFlatten (2239 ms) 2024-08-17T17:17:09.386 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.Stress 2024-08-17T17:17:09.409 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 12033065~7509720 2024-08-17T17:17:09.433 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 33020152~2839027 2024-08-17T17:17:09.433 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 58690183~786973 2024-08-17T17:17:09.437 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 54534123~156838 2024-08-17T17:17:09.438 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 33548214~616606 2024-08-17T17:17:09.442 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 35559167~1760522 2024-08-17T17:17:09.443 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 33017306~5236783 2024-08-17T17:17:09.459 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 47622297~7391968 2024-08-17T17:17:09.460 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap0 2024-08-17T17:17:09.851 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 97979503 2024-08-17T17:17:09.857 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 70243530~3797616 2024-08-17T17:17:09.861 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 36028288~1438639 2024-08-17T17:17:09.861 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 91879427~4535838 2024-08-17T17:17:09.861 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 74190058~1043130 2024-08-17T17:17:09.862 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 8283245~2667398 2024-08-17T17:17:09.862 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 70879519~8102729 2024-08-17T17:17:09.879 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 27512715~5732251 2024-08-17T17:17:09.879 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 18954165~4600348 2024-08-17T17:17:09.905 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap1 2024-08-17T17:17:10.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:09 smithi136 ceph-mon[109861]: osdmap e400: 8 total, 8 up, 8 in 2024-08-17T17:17:10.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:09 smithi136 ceph-mon[115078]: osdmap e400: 8 total, 8 up, 8 in 2024-08-17T17:17:10.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:09 smithi165 ceph-mon[34868]: osdmap e400: 8 total, 8 up, 8 in 2024-08-17T17:17:10.862 INFO:tasks.workunit.client.0.smithi136.stdout:clone image240 -> image241 2024-08-17T17:17:10.912 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 171103595 2024-08-17T17:17:10.919 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 10364095~8348027 2024-08-17T17:17:10.945 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 32062979~5348591 2024-08-17T17:17:10.966 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 163911978~7191617 2024-08-17T17:17:10.990 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 96984009~1182815 2024-08-17T17:17:10.994 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 32347209~7876023 2024-08-17T17:17:11.011 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 30478095~2971518 2024-08-17T17:17:11.013 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 104853370~5801788 2024-08-17T17:17:11.029 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 70931462~6630869 2024-08-17T17:17:11.035 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap2 2024-08-17T17:17:11.097 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:10 smithi136 ceph-mon[109861]: pgmap v90: 105 pgs: 105 active+clean; 85 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 MiB/s wr, 173 op/s 2024-08-17T17:17:11.098 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:10 smithi136 ceph-mon[109861]: osdmap e401: 8 total, 8 up, 8 in 2024-08-17T17:17:11.098 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:10 smithi136 ceph-mon[115078]: pgmap v90: 105 pgs: 105 active+clean; 85 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 MiB/s wr, 173 op/s 2024-08-17T17:17:11.098 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:10 smithi136 ceph-mon[115078]: osdmap e401: 8 total, 8 up, 8 in 2024-08-17T17:17:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:10 smithi165 ceph-mon[34868]: pgmap v90: 105 pgs: 105 active+clean; 85 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.0 MiB/s wr, 173 op/s 2024-08-17T17:17:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:10 smithi165 ceph-mon[34868]: osdmap e401: 8 total, 8 up, 8 in 2024-08-17T17:17:11.261 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:17:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=infra.usagestats t=2024-08-17T17:17:11.043950035Z level=info msg="Usage stats are ready to report" 2024-08-17T17:17:11.860 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 206072279 2024-08-17T17:17:11.867 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 25498065~2294916 2024-08-17T17:17:11.870 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 98261269~2848177 2024-08-17T17:17:11.870 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 19509191~4299093 2024-08-17T17:17:11.871 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 106254439~5367616 2024-08-17T17:17:11.872 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 92296481~5389791 2024-08-17T17:17:11.872 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 143859562~5081109 2024-08-17T17:17:11.872 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 56188574~1083978 2024-08-17T17:17:11.872 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 60421409~7704583 2024-08-17T17:17:11.923 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap3 2024-08-17T17:17:12.108 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:11 smithi136 ceph-mon[109861]: osdmap e402: 8 total, 8 up, 8 in 2024-08-17T17:17:12.108 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:11 smithi136 ceph-mon[115078]: osdmap e402: 8 total, 8 up, 8 in 2024-08-17T17:17:12.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:11 smithi165 ceph-mon[34868]: osdmap e402: 8 total, 8 up, 8 in 2024-08-17T17:17:12.874 INFO:tasks.workunit.client.0.smithi136.stdout:clone image241 -> image242 2024-08-17T17:17:12.931 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 270216658 2024-08-17T17:17:13.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:12 smithi136 ceph-mon[109861]: pgmap v93: 105 pgs: 105 active+clean; 231 MiB data, 940 MiB used, 714 GiB / 715 GiB avail; 32 MiB/s rd, 33 MiB/s wr, 284 op/s 2024-08-17T17:17:13.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:12 smithi136 ceph-mon[109861]: osdmap e403: 8 total, 8 up, 8 in 2024-08-17T17:17:13.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:12 smithi136 ceph-mon[115078]: pgmap v93: 105 pgs: 105 active+clean; 231 MiB data, 940 MiB used, 714 GiB / 715 GiB avail; 32 MiB/s rd, 33 MiB/s wr, 284 op/s 2024-08-17T17:17:13.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:12 smithi136 ceph-mon[115078]: osdmap e403: 8 total, 8 up, 8 in 2024-08-17T17:17:13.235 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestDeepCopy.Stress (3849 ms) 2024-08-17T17:17:13.235 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.NoSnaps_LargerDstObjSize 2024-08-17T17:17:13.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:12 smithi165 ceph-mon[34868]: pgmap v93: 105 pgs: 105 active+clean; 231 MiB data, 940 MiB used, 714 GiB / 715 GiB avail; 32 MiB/s rd, 33 MiB/s wr, 284 op/s 2024-08-17T17:17:13.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:12 smithi165 ceph-mon[34868]: osdmap e403: 8 total, 8 up, 8 in 2024-08-17T17:17:13.489 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestDeepCopy.NoSnaps_LargerDstObjSize (255 ms) 2024-08-17T17:17:13.490 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.Snaps_LargerDstObjSize 2024-08-17T17:17:14.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:13 smithi136 ceph-mon[109861]: osdmap e404: 8 total, 8 up, 8 in 2024-08-17T17:17:14.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:13 smithi136 ceph-mon[115078]: osdmap e404: 8 total, 8 up, 8 in 2024-08-17T17:17:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:13 smithi165 ceph-mon[34868]: osdmap e404: 8 total, 8 up, 8 in 2024-08-17T17:17:15.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:14 smithi136 ceph-mon[115078]: pgmap v96: 105 pgs: 105 active+clean; 300 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 49 MiB/s rd, 47 MiB/s wr, 615 op/s 2024-08-17T17:17:15.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:14 smithi136 ceph-mon[115078]: osdmap e405: 8 total, 8 up, 8 in 2024-08-17T17:17:15.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:14 smithi136 ceph-mon[109861]: pgmap v96: 105 pgs: 105 active+clean; 300 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 49 MiB/s rd, 47 MiB/s wr, 615 op/s 2024-08-17T17:17:15.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:14 smithi136 ceph-mon[109861]: osdmap e405: 8 total, 8 up, 8 in 2024-08-17T17:17:15.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:14 smithi165 ceph-mon[34868]: pgmap v96: 105 pgs: 105 active+clean; 300 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 49 MiB/s rd, 47 MiB/s wr, 615 op/s 2024-08-17T17:17:15.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:14 smithi165 ceph-mon[34868]: osdmap e405: 8 total, 8 up, 8 in 2024-08-17T17:17:16.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:15 smithi136 ceph-mon[115078]: osdmap e406: 8 total, 8 up, 8 in 2024-08-17T17:17:16.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:15 smithi136 ceph-mon[109861]: osdmap e406: 8 total, 8 up, 8 in 2024-08-17T17:17:16.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:15 smithi165 ceph-mon[34868]: osdmap e406: 8 total, 8 up, 8 in 2024-08-17T17:17:17.156 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestDeepCopy.Snaps_LargerDstObjSize (3666 ms) 2024-08-17T17:17:17.156 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.Clone_LargerDstObjSize 2024-08-17T17:17:17.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:16 smithi136 ceph-mon[109861]: pgmap v99: 105 pgs: 105 active+clean; 300 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 14 MiB/s wr, 330 op/s 2024-08-17T17:17:17.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:16 smithi136 ceph-mon[109861]: osdmap e407: 8 total, 8 up, 8 in 2024-08-17T17:17:17.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:16 smithi136 ceph-mon[115078]: pgmap v99: 105 pgs: 105 active+clean; 300 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 14 MiB/s wr, 330 op/s 2024-08-17T17:17:17.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:16 smithi136 ceph-mon[115078]: osdmap e407: 8 total, 8 up, 8 in 2024-08-17T17:17:17.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:16 smithi165 ceph-mon[34868]: pgmap v99: 105 pgs: 105 active+clean; 300 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 14 MiB/s wr, 330 op/s 2024-08-17T17:17:17.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:16 smithi165 ceph-mon[34868]: osdmap e407: 8 total, 8 up, 8 in 2024-08-17T17:17:17.879 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:17:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:17:17] "GET /metrics HTTP/1.1" 200 36291 "" "Prometheus/2.51.0" 2024-08-17T17:17:18.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:17 smithi136 ceph-mon[109861]: osdmap e408: 8 total, 8 up, 8 in 2024-08-17T17:17:18.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:17 smithi136 ceph-mon[115078]: osdmap e408: 8 total, 8 up, 8 in 2024-08-17T17:17:18.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:17 smithi165 ceph-mon[34868]: osdmap e408: 8 total, 8 up, 8 in 2024-08-17T17:17:19.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:18 smithi136 ceph-mon[109861]: pgmap v102: 105 pgs: 105 active+clean; 377 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 35 MiB/s rd, 17 MiB/s wr, 406 op/s 2024-08-17T17:17:19.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:18 smithi136 ceph-mon[109861]: osdmap e409: 8 total, 8 up, 8 in 2024-08-17T17:17:19.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:18 smithi136 ceph-mon[115078]: pgmap v102: 105 pgs: 105 active+clean; 377 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 35 MiB/s rd, 17 MiB/s wr, 406 op/s 2024-08-17T17:17:19.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:18 smithi136 ceph-mon[115078]: osdmap e409: 8 total, 8 up, 8 in 2024-08-17T17:17:19.240 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestDeepCopy.Clone_LargerDstObjSize (2084 ms) 2024-08-17T17:17:19.241 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.CloneFlatten_LargerDstObjSize 2024-08-17T17:17:19.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:18 smithi165 ceph-mon[34868]: pgmap v102: 105 pgs: 105 active+clean; 377 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 35 MiB/s rd, 17 MiB/s wr, 406 op/s 2024-08-17T17:17:19.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:18 smithi165 ceph-mon[34868]: osdmap e409: 8 total, 8 up, 8 in 2024-08-17T17:17:20.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:19 smithi136 ceph-mon[109861]: osdmap e410: 8 total, 8 up, 8 in 2024-08-17T17:17:20.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:19 smithi136 ceph-mon[115078]: osdmap e410: 8 total, 8 up, 8 in 2024-08-17T17:17:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:19 smithi165 ceph-mon[34868]: osdmap e410: 8 total, 8 up, 8 in 2024-08-17T17:17:21.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:20 smithi136 ceph-mon[115078]: pgmap v105: 105 pgs: 105 active+clean; 377 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 35 MiB/s rd, 17 MiB/s wr, 406 op/s 2024-08-17T17:17:21.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:20 smithi136 ceph-mon[115078]: osdmap e411: 8 total, 8 up, 8 in 2024-08-17T17:17:21.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:20 smithi136 ceph-mon[109861]: pgmap v105: 105 pgs: 105 active+clean; 377 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 35 MiB/s rd, 17 MiB/s wr, 406 op/s 2024-08-17T17:17:21.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:20 smithi136 ceph-mon[109861]: osdmap e411: 8 total, 8 up, 8 in 2024-08-17T17:17:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:20 smithi165 ceph-mon[34868]: pgmap v105: 105 pgs: 105 active+clean; 377 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 35 MiB/s rd, 17 MiB/s wr, 406 op/s 2024-08-17T17:17:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:20 smithi165 ceph-mon[34868]: osdmap e411: 8 total, 8 up, 8 in 2024-08-17T17:17:21.410 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestDeepCopy.CloneFlatten_LargerDstObjSize (2170 ms) 2024-08-17T17:17:21.410 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.Stress_LargerDstObjSize 2024-08-17T17:17:21.430 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 56524035~1460278 2024-08-17T17:17:21.441 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 2198877~3404086 2024-08-17T17:17:21.442 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 15375210~4434649 2024-08-17T17:17:21.453 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 12693632~7473313 2024-08-17T17:17:21.455 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 9768230~3223640 2024-08-17T17:17:21.468 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 56403465~6842797 2024-08-17T17:17:21.469 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 19530498~3388659 2024-08-17T17:17:21.477 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 33922099~6263637 2024-08-17T17:17:21.478 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap0 2024-08-17T17:17:21.921 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 50288265~1492227 2024-08-17T17:17:21.925 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 36130637~5620812 2024-08-17T17:17:21.925 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 30102611~1835999 2024-08-17T17:17:21.925 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 69627255~5380125 2024-08-17T17:17:21.926 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 42921291~6449528 2024-08-17T17:17:21.926 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 66169941~1007151 2024-08-17T17:17:21.929 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 69576294~6969157 2024-08-17T17:17:21.929 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 10373243~5253465 2024-08-17T17:17:21.957 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap1 2024-08-17T17:17:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:21 smithi136 ceph-mon[109861]: osdmap e412: 8 total, 8 up, 8 in 2024-08-17T17:17:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:17:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:21 smithi136 ceph-mon[115078]: osdmap e412: 8 total, 8 up, 8 in 2024-08-17T17:17:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:17:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:21 smithi165 ceph-mon[34868]: osdmap e412: 8 total, 8 up, 8 in 2024-08-17T17:17:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:21 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:17:22.932 INFO:tasks.workunit.client.0.smithi136.stdout:clone image261 -> image262 2024-08-17T17:17:22.982 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 20528228~582629 2024-08-17T17:17:22.996 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 45749309~5604530 2024-08-17T17:17:23.016 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 20835935~5130634 2024-08-17T17:17:23.030 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 8519294~1489478 2024-08-17T17:17:23.037 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 24530331~6812001 2024-08-17T17:17:23.063 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 10076117~498353 2024-08-17T17:17:23.064 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 27156404~2629327 2024-08-17T17:17:23.074 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 82241954~1644126 2024-08-17T17:17:23.077 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap2 2024-08-17T17:17:23.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:22 smithi136 ceph-mon[109861]: pgmap v108: 105 pgs: 105 active+clean; 523 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 57 MiB/s rd, 37 MiB/s wr, 502 op/s 2024-08-17T17:17:23.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:22 smithi136 ceph-mon[109861]: osdmap e413: 8 total, 8 up, 8 in 2024-08-17T17:17:23.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:22 smithi136 ceph-mon[115078]: pgmap v108: 105 pgs: 105 active+clean; 523 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 57 MiB/s rd, 37 MiB/s wr, 502 op/s 2024-08-17T17:17:23.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:22 smithi136 ceph-mon[115078]: osdmap e413: 8 total, 8 up, 8 in 2024-08-17T17:17:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:22 smithi165 ceph-mon[34868]: pgmap v108: 105 pgs: 105 active+clean; 523 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 57 MiB/s rd, 37 MiB/s wr, 502 op/s 2024-08-17T17:17:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:22 smithi165 ceph-mon[34868]: osdmap e413: 8 total, 8 up, 8 in 2024-08-17T17:17:23.934 INFO:tasks.workunit.client.0.smithi136.stdout:clone image262 -> image263 2024-08-17T17:17:23.994 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 51889951~4798318 2024-08-17T17:17:24.016 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 36413671~5998517 2024-08-17T17:17:24.032 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 40560100~2536554 2024-08-17T17:17:24.038 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 9505712~1920316 2024-08-17T17:17:24.043 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 40728213~1084490 2024-08-17T17:17:24.048 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 80784341~3101739 2024-08-17T17:17:24.051 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 79181841~84679 2024-08-17T17:17:24.067 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 39243641~1113202 2024-08-17T17:17:24.068 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap3 2024-08-17T17:17:24.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:23 smithi136 ceph-mon[109861]: osdmap e414: 8 total, 8 up, 8 in 2024-08-17T17:17:24.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:23 smithi136 ceph-mon[115078]: osdmap e414: 8 total, 8 up, 8 in 2024-08-17T17:17:24.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:23 smithi165 ceph-mon[34868]: osdmap e414: 8 total, 8 up, 8 in 2024-08-17T17:17:24.952 INFO:tasks.workunit.client.0.smithi136.stdout:clone image263 -> image264 2024-08-17T17:17:25.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:24 smithi136 ceph-mon[109861]: pgmap v111: 105 pgs: 105 active+clean; 557 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 72 MiB/s rd, 45 MiB/s wr, 568 op/s 2024-08-17T17:17:25.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:24 smithi136 ceph-mon[109861]: osdmap e415: 8 total, 8 up, 8 in 2024-08-17T17:17:25.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:24 smithi136 ceph-mon[115078]: pgmap v111: 105 pgs: 105 active+clean; 557 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 72 MiB/s rd, 45 MiB/s wr, 568 op/s 2024-08-17T17:17:25.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:24 smithi136 ceph-mon[115078]: osdmap e415: 8 total, 8 up, 8 in 2024-08-17T17:17:25.254 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestDeepCopy.Stress_LargerDstObjSize (3844 ms) 2024-08-17T17:17:25.254 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.NoSnaps_SmallerDstObjSize 2024-08-17T17:17:25.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:24 smithi165 ceph-mon[34868]: pgmap v111: 105 pgs: 105 active+clean; 557 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 72 MiB/s rd, 45 MiB/s wr, 568 op/s 2024-08-17T17:17:25.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:24 smithi165 ceph-mon[34868]: osdmap e415: 8 total, 8 up, 8 in 2024-08-17T17:17:25.481 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestDeepCopy.NoSnaps_SmallerDstObjSize (227 ms) 2024-08-17T17:17:25.481 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.Snaps_SmallerDstObjSize 2024-08-17T17:17:26.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:25 smithi136 ceph-mon[109861]: osdmap e416: 8 total, 8 up, 8 in 2024-08-17T17:17:26.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:25 smithi136 ceph-mon[115078]: osdmap e416: 8 total, 8 up, 8 in 2024-08-17T17:17:26.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:25 smithi165 ceph-mon[34868]: osdmap e416: 8 total, 8 up, 8 in 2024-08-17T17:17:27.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:26 smithi136 ceph-mon[109861]: pgmap v114: 105 pgs: 105 active+clean; 557 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 8.4 MiB/s wr, 66 op/s 2024-08-17T17:17:27.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:26 smithi136 ceph-mon[109861]: osdmap e417: 8 total, 8 up, 8 in 2024-08-17T17:17:27.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:26 smithi136 ceph-mon[115078]: pgmap v114: 105 pgs: 105 active+clean; 557 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 8.4 MiB/s wr, 66 op/s 2024-08-17T17:17:27.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:26 smithi136 ceph-mon[115078]: osdmap e417: 8 total, 8 up, 8 in 2024-08-17T17:17:27.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:26 smithi165 ceph-mon[34868]: pgmap v114: 105 pgs: 105 active+clean; 557 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 8.4 MiB/s wr, 66 op/s 2024-08-17T17:17:27.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:26 smithi165 ceph-mon[34868]: osdmap e417: 8 total, 8 up, 8 in 2024-08-17T17:17:27.944 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:17:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:17:27] "GET /metrics HTTP/1.1" 200 36300 "" "Prometheus/2.51.0" 2024-08-17T17:17:28.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:27 smithi136 ceph-mon[109861]: osdmap e418: 8 total, 8 up, 8 in 2024-08-17T17:17:28.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:27 smithi136 ceph-mon[115078]: osdmap e418: 8 total, 8 up, 8 in 2024-08-17T17:17:28.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:27 smithi165 ceph-mon[34868]: osdmap e418: 8 total, 8 up, 8 in 2024-08-17T17:17:29.208 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestDeepCopy.Snaps_SmallerDstObjSize (3727 ms) 2024-08-17T17:17:29.208 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.Clone_SmallerDstObjSize 2024-08-17T17:17:29.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:28 smithi165 ceph-mon[34868]: pgmap v117: 105 pgs: 105 active+clean; 671 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 32 MiB/s rd, 25 MiB/s wr, 588 op/s 2024-08-17T17:17:29.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:28 smithi165 ceph-mon[34868]: osdmap e419: 8 total, 8 up, 8 in 2024-08-17T17:17:29.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:28 smithi136 ceph-mon[109861]: pgmap v117: 105 pgs: 105 active+clean; 671 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 32 MiB/s rd, 25 MiB/s wr, 588 op/s 2024-08-17T17:17:29.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:28 smithi136 ceph-mon[109861]: osdmap e419: 8 total, 8 up, 8 in 2024-08-17T17:17:29.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:28 smithi136 ceph-mon[115078]: pgmap v117: 105 pgs: 105 active+clean; 671 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 32 MiB/s rd, 25 MiB/s wr, 588 op/s 2024-08-17T17:17:29.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:28 smithi136 ceph-mon[115078]: osdmap e419: 8 total, 8 up, 8 in 2024-08-17T17:17:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:29 smithi165 ceph-mon[34868]: osdmap e420: 8 total, 8 up, 8 in 2024-08-17T17:17:30.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:29 smithi136 ceph-mon[109861]: osdmap e420: 8 total, 8 up, 8 in 2024-08-17T17:17:30.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:29 smithi136 ceph-mon[115078]: osdmap e420: 8 total, 8 up, 8 in 2024-08-17T17:17:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:30 smithi165 ceph-mon[34868]: pgmap v120: 105 pgs: 105 active+clean; 671 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 32 MiB/s rd, 25 MiB/s wr, 588 op/s 2024-08-17T17:17:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:30 smithi165 ceph-mon[34868]: osdmap e421: 8 total, 8 up, 8 in 2024-08-17T17:17:31.275 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestDeepCopy.Clone_SmallerDstObjSize (2067 ms) 2024-08-17T17:17:31.275 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.CloneFlatten_SmallerDstObjSize 2024-08-17T17:17:31.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:30 smithi136 ceph-mon[109861]: pgmap v120: 105 pgs: 105 active+clean; 671 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 32 MiB/s rd, 25 MiB/s wr, 588 op/s 2024-08-17T17:17:31.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:30 smithi136 ceph-mon[109861]: osdmap e421: 8 total, 8 up, 8 in 2024-08-17T17:17:31.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:30 smithi136 ceph-mon[115078]: pgmap v120: 105 pgs: 105 active+clean; 671 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 32 MiB/s rd, 25 MiB/s wr, 588 op/s 2024-08-17T17:17:31.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:30 smithi136 ceph-mon[115078]: osdmap e421: 8 total, 8 up, 8 in 2024-08-17T17:17:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:31 smithi165 ceph-mon[34868]: osdmap e422: 8 total, 8 up, 8 in 2024-08-17T17:17:32.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:31 smithi136 ceph-mon[109861]: osdmap e422: 8 total, 8 up, 8 in 2024-08-17T17:17:32.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:31 smithi136 ceph-mon[115078]: osdmap e422: 8 total, 8 up, 8 in 2024-08-17T17:17:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:32 smithi165 ceph-mon[34868]: pgmap v123: 105 pgs: 105 active+clean; 717 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 31 MiB/s rd, 12 MiB/s wr, 266 op/s 2024-08-17T17:17:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:32 smithi165 ceph-mon[34868]: osdmap e423: 8 total, 8 up, 8 in 2024-08-17T17:17:33.394 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestDeepCopy.CloneFlatten_SmallerDstObjSize (2119 ms) 2024-08-17T17:17:33.394 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.Stress_SmallerDstObjSize 2024-08-17T17:17:33.419 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 67064880~8171322 2024-08-17T17:17:33.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:32 smithi136 ceph-mon[109861]: pgmap v123: 105 pgs: 105 active+clean; 717 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 31 MiB/s rd, 12 MiB/s wr, 266 op/s 2024-08-17T17:17:33.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:32 smithi136 ceph-mon[109861]: osdmap e423: 8 total, 8 up, 8 in 2024-08-17T17:17:33.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:32 smithi136 ceph-mon[115078]: pgmap v123: 105 pgs: 105 active+clean; 717 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 31 MiB/s rd, 12 MiB/s wr, 266 op/s 2024-08-17T17:17:33.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:32 smithi136 ceph-mon[115078]: osdmap e423: 8 total, 8 up, 8 in 2024-08-17T17:17:33.449 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 43413263~6872876 2024-08-17T17:17:33.450 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 59195104~2585344 2024-08-17T17:17:33.457 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 27612699~3114350 2024-08-17T17:17:33.458 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 56926124~5041634 2024-08-17T17:17:33.471 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 73708801~7245194 2024-08-17T17:17:33.479 INFO:tasks.workunit.client.0.smithi136.stdout:write: A 43658865~4204385 2024-08-17T17:17:33.489 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 36375877~747884 2024-08-17T17:17:33.489 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap0 2024-08-17T17:17:33.998 INFO:tasks.workunit.client.0.smithi136.stdout:clone image283 -> image284 2024-08-17T17:17:34.052 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 52752963~3905936 2024-08-17T17:17:34.079 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 9656733~4966727 2024-08-17T17:17:34.079 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 42205528~2505994 2024-08-17T17:17:34.094 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 33361028~2381327 2024-08-17T17:17:34.099 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 25402380~7753127 2024-08-17T17:17:34.119 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 25121852~1165916 2024-08-17T17:17:34.122 INFO:tasks.workunit.client.0.smithi136.stdout:write: B 42891670~6611078 2024-08-17T17:17:34.137 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 13483954~8300652 2024-08-17T17:17:34.140 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap1 2024-08-17T17:17:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:33 smithi165 ceph-mon[34868]: osdmap e424: 8 total, 8 up, 8 in 2024-08-17T17:17:34.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:33 smithi136 ceph-mon[109861]: osdmap e424: 8 total, 8 up, 8 in 2024-08-17T17:17:34.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:33 smithi136 ceph-mon[115078]: osdmap e424: 8 total, 8 up, 8 in 2024-08-17T17:17:35.001 INFO:tasks.workunit.client.0.smithi136.stdout:clone image284 -> image285 2024-08-17T17:17:35.063 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 55864852~5570195 2024-08-17T17:17:35.098 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 44166261~4944264 2024-08-17T17:17:35.110 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 20578066~3739072 2024-08-17T17:17:35.126 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 58275105~434551 2024-08-17T17:17:35.128 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 17959651~5516675 2024-08-17T17:17:35.138 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 76261680~2376294 2024-08-17T17:17:35.141 INFO:tasks.workunit.client.0.smithi136.stdout:write: C 23059446~3047423 2024-08-17T17:17:35.161 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 24791366~3185523 2024-08-17T17:17:35.163 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap2 2024-08-17T17:17:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:34 smithi165 ceph-mon[34868]: pgmap v126: 105 pgs: 105 active+clean; 739 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 38 MiB/s rd, 17 MiB/s wr, 419 op/s 2024-08-17T17:17:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:34 smithi165 ceph-mon[34868]: osdmap e425: 8 total, 8 up, 8 in 2024-08-17T17:17:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:34 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:17:35.442 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:34 smithi136 ceph-mon[109861]: pgmap v126: 105 pgs: 105 active+clean; 739 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 38 MiB/s rd, 17 MiB/s wr, 419 op/s 2024-08-17T17:17:35.443 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:34 smithi136 ceph-mon[109861]: osdmap e425: 8 total, 8 up, 8 in 2024-08-17T17:17:35.443 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:17:35.443 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:34 smithi136 ceph-mon[115078]: pgmap v126: 105 pgs: 105 active+clean; 739 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 38 MiB/s rd, 17 MiB/s wr, 419 op/s 2024-08-17T17:17:35.443 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:34 smithi136 ceph-mon[115078]: osdmap e425: 8 total, 8 up, 8 in 2024-08-17T17:17:35.443 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:17:36.003 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 60263331~8276639 2024-08-17T17:17:36.005 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 16029766~5498114 2024-08-17T17:17:36.006 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 65384286~1779631 2024-08-17T17:17:36.044 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 50333750~7597186 2024-08-17T17:17:36.046 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 76273419~7612661 2024-08-17T17:17:36.046 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 10031515~5097448 2024-08-17T17:17:36.047 INFO:tasks.workunit.client.0.smithi136.stdout:write: D 2279035~1618011 2024-08-17T17:17:36.048 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 31728099~7176111 2024-08-17T17:17:36.094 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap3 2024-08-17T17:17:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:35 smithi165 ceph-mon[34868]: osdmap e426: 8 total, 8 up, 8 in 2024-08-17T17:17:36.329 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:35 smithi136 ceph-mon[115078]: osdmap e426: 8 total, 8 up, 8 in 2024-08-17T17:17:36.329 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:35 smithi136 ceph-mon[109861]: osdmap e426: 8 total, 8 up, 8 in 2024-08-17T17:17:37.011 INFO:tasks.workunit.client.0.smithi136.stdout:clone image285 -> image286 2024-08-17T17:17:37.083 INFO:tasks.workunit.client.0.smithi136.stdout:resize: 136192245 2024-08-17T17:17:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:36 smithi165 ceph-mon[34868]: pgmap v129: 105 pgs: 105 active+clean; 739 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 5.5 MiB/s wr, 153 op/s 2024-08-17T17:17:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:36 smithi165 ceph-mon[34868]: osdmap e427: 8 total, 8 up, 8 in 2024-08-17T17:17:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:36 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:17:37.310 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:36 smithi136 ceph-mon[115078]: pgmap v129: 105 pgs: 105 active+clean; 739 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 5.5 MiB/s wr, 153 op/s 2024-08-17T17:17:37.310 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:36 smithi136 ceph-mon[115078]: osdmap e427: 8 total, 8 up, 8 in 2024-08-17T17:17:37.310 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:17:37.310 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:36 smithi136 ceph-mon[109861]: pgmap v129: 105 pgs: 105 active+clean; 739 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 5.5 MiB/s wr, 153 op/s 2024-08-17T17:17:37.310 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:36 smithi136 ceph-mon[109861]: osdmap e427: 8 total, 8 up, 8 in 2024-08-17T17:17:37.310 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:17:37.331 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestDeepCopy.Stress_SmallerDstObjSize (3938 ms) 2024-08-17T17:17:37.332 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.NoSnaps_StrippingLargerDstObjSize 2024-08-17T17:17:37.353 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_DeepCopy.cc:621: Skipped 2024-08-17T17:17:37.353 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:17:37.423 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestDeepCopy.NoSnaps_StrippingLargerDstObjSize (91 ms) 2024-08-17T17:17:37.423 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.Snaps_StrippingLargerDstObjSize 2024-08-17T17:17:37.445 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_DeepCopy.cc:635: Skipped 2024-08-17T17:17:37.445 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:17:37.519 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestDeepCopy.Snaps_StrippingLargerDstObjSize (96 ms) 2024-08-17T17:17:37.519 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.Clone_StrippingLargerDstObjSize 2024-08-17T17:17:37.541 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_DeepCopy.cc:649: Skipped 2024-08-17T17:17:37.542 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:17:37.621 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestDeepCopy.Clone_StrippingLargerDstObjSize (102 ms) 2024-08-17T17:17:37.621 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.CloneFlatten_StrippingLargerDstObjSize 2024-08-17T17:17:37.646 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_DeepCopy.cc:663: Skipped 2024-08-17T17:17:37.646 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:17:37.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:17:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:17:37] "GET /metrics HTTP/1.1" 200 36301 "" "Prometheus/2.51.0" 2024-08-17T17:17:37.721 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestDeepCopy.CloneFlatten_StrippingLargerDstObjSize (100 ms) 2024-08-17T17:17:37.721 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.Stress_StrippingLargerDstObjSize 2024-08-17T17:17:37.744 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_DeepCopy.cc:679: Skipped 2024-08-17T17:17:37.744 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:17:37.817 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestDeepCopy.Stress_StrippingLargerDstObjSize (96 ms) 2024-08-17T17:17:37.817 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.NoSnaps_StrippingSmallerDstObjSize 2024-08-17T17:17:37.839 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_DeepCopy.cc:693: Skipped 2024-08-17T17:17:37.840 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:17:37.914 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestDeepCopy.NoSnaps_StrippingSmallerDstObjSize (97 ms) 2024-08-17T17:17:37.914 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.Snaps_StrippingSmallerDstObjSize 2024-08-17T17:17:37.938 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_DeepCopy.cc:707: Skipped 2024-08-17T17:17:37.938 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:17:38.013 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestDeepCopy.Snaps_StrippingSmallerDstObjSize (99 ms) 2024-08-17T17:17:38.013 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.Clone_StrippingSmallerDstObjSize 2024-08-17T17:17:38.037 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_DeepCopy.cc:721: Skipped 2024-08-17T17:17:38.037 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:17:38.114 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestDeepCopy.Clone_StrippingSmallerDstObjSize (101 ms) 2024-08-17T17:17:38.114 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.CloneFlatten_StrippingSmallerDstObjSize 2024-08-17T17:17:38.134 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_DeepCopy.cc:735: Skipped 2024-08-17T17:17:38.134 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:17:38.206 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestDeepCopy.CloneFlatten_StrippingSmallerDstObjSize (92 ms) 2024-08-17T17:17:38.206 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestDeepCopy.Stress_StrippingSmallerDstObjSize 2024-08-17T17:17:38.211 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:37 smithi165 ceph-mon[34868]: osdmap e428: 8 total, 8 up, 8 in 2024-08-17T17:17:38.211 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:37 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:17:38.211 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:37 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:17:38.228 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_DeepCopy.cc:751: Skipped 2024-08-17T17:17:38.228 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:17:38.298 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestDeepCopy.Stress_StrippingSmallerDstObjSize (92 ms) 2024-08-17T17:17:38.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:37 smithi136 ceph-mon[109861]: osdmap e428: 8 total, 8 up, 8 in 2024-08-17T17:17:38.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:17:38.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:17:38.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:37 smithi136 ceph-mon[115078]: osdmap e428: 8 total, 8 up, 8 in 2024-08-17T17:17:38.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:17:38.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:17:39.012 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 31 tests from TestDeepCopy (51478 ms total) 2024-08-17T17:17:39.012 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:17:39.013 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 51 tests from TestMirroring 2024-08-17T17:17:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:38 smithi165 ceph-mon[34868]: pgmap v132: 105 pgs: 105 active+clean; 952 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 40 MiB/s rd, 48 MiB/s wr, 663 op/s 2024-08-17T17:17:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:38 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:17:39.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:38 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:17:39.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:38 smithi165 ceph-mon[34868]: mgrmap e40: y(active, since 92s), standbys: x 2024-08-17T17:17:39.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:38 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:17:39.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:38 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:17:39.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:38 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:17:39.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:38 smithi136 ceph-mon[109861]: pgmap v132: 105 pgs: 105 active+clean; 952 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 40 MiB/s rd, 48 MiB/s wr, 663 op/s 2024-08-17T17:17:39.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:17:39.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:17:39.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:38 smithi136 ceph-mon[109861]: mgrmap e40: y(active, since 92s), standbys: x 2024-08-17T17:17:39.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:17:39.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:17:39.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:17:39.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:38 smithi136 ceph-mon[115078]: pgmap v132: 105 pgs: 105 active+clean; 952 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 40 MiB/s rd, 48 MiB/s wr, 663 op/s 2024-08-17T17:17:39.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:17:39.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:17:39.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:38 smithi136 ceph-mon[115078]: mgrmap e40: y(active, since 92s), standbys: x 2024-08-17T17:17:39.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:17:39.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:17:39.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:17:40.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:40 smithi165 ceph-mon[34868]: osdmap e429: 8 total, 8 up, 8 in 2024-08-17T17:17:40.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:40 smithi136 ceph-mon[109861]: osdmap e429: 8 total, 8 up, 8 in 2024-08-17T17:17:40.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:40 smithi136 ceph-mon[115078]: osdmap e429: 8 total, 8 up, 8 in 2024-08-17T17:17:41.016 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeImage 2024-08-17T17:17:41.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:41 smithi136 ceph-mon[109861]: pgmap v134: 73 pgs: 73 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-08-17T17:17:41.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:41 smithi136 ceph-mon[109861]: osdmap e430: 8 total, 8 up, 8 in 2024-08-17T17:17:41.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:41 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-12","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:17:41.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:41 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-12","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:17:41.349 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:41 smithi136 ceph-mon[115078]: pgmap v134: 73 pgs: 73 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-08-17T17:17:41.349 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:41 smithi136 ceph-mon[115078]: osdmap e430: 8 total, 8 up, 8 in 2024-08-17T17:17:41.349 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:41 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-12","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:17:41.349 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:41 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-12","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:17:41.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:41 smithi165 ceph-mon[34868]: pgmap v134: 73 pgs: 73 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-08-17T17:17:41.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:41 smithi165 ceph-mon[34868]: osdmap e430: 8 total, 8 up, 8 in 2024-08-17T17:17:41.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:41 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-12","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:17:41.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:41 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-12","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:17:42.365 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:42 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:17:42.365 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:42 smithi136 ceph-mon[109861]: osdmap e431: 8 total, 8 up, 8 in 2024-08-17T17:17:42.365 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:42 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:17:42.365 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:42 smithi136 ceph-mon[115078]: osdmap e431: 8 total, 8 up, 8 in 2024-08-17T17:17:42.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:42 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:17:42.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:42 smithi165 ceph-mon[34868]: osdmap e431: 8 total, 8 up, 8 in 2024-08-17T17:17:43.137 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeImage (2121 ms) 2024-08-17T17:17:43.138 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModePool 2024-08-17T17:17:43.238 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModePool (101 ms) 2024-08-17T17:17:43.238 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeDisabled 2024-08-17T17:17:43.312 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeDisabled (74 ms) 2024-08-17T17:17:43.312 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.DisableImageMirror_In_MirrorModeImage 2024-08-17T17:17:43.402 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.DisableImageMirror_In_MirrorModeImage (90 ms) 2024-08-17T17:17:43.402 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.DisableImageMirror_In_MirrorModeImage_NoObjectMap 2024-08-17T17:17:43.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:43 smithi136 ceph-mon[109861]: pgmap v137: 105 pgs: 14 unknown, 91 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:43.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:43 smithi136 ceph-mon[109861]: osdmap e432: 8 total, 8 up, 8 in 2024-08-17T17:17:43.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:43 smithi136 ceph-mon[115078]: pgmap v137: 105 pgs: 14 unknown, 91 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:43.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:43 smithi136 ceph-mon[115078]: osdmap e432: 8 total, 8 up, 8 in 2024-08-17T17:17:43.489 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.DisableImageMirror_In_MirrorModeImage_NoObjectMap (87 ms) 2024-08-17T17:17:43.489 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.DisableImageMirror_In_MirrorModePool 2024-08-17T17:17:43.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:43 smithi165 ceph-mon[34868]: pgmap v137: 105 pgs: 14 unknown, 91 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:43.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:43 smithi165 ceph-mon[34868]: osdmap e432: 8 total, 8 up, 8 in 2024-08-17T17:17:43.576 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.DisableImageMirror_In_MirrorModePool (87 ms) 2024-08-17T17:17:43.576 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.DisableImageMirror_In_MirrorModeDisabled 2024-08-17T17:17:43.679 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.DisableImageMirror_In_MirrorModeDisabled (103 ms) 2024-08-17T17:17:43.679 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.DisableImageMirrorWithPeer 2024-08-17T17:17:43.679 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_mirroring.cc:482: Skipped 2024-08-17T17:17:43.679 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:17:43.679 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestMirroring.DisableImageMirrorWithPeer (0 ms) 2024-08-17T17:17:43.679 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.DisableJournalingWithPeer 2024-08-17T17:17:43.679 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_mirroring.cc:520: Skipped 2024-08-17T17:17:43.679 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:17:43.679 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestMirroring.DisableJournalingWithPeer (0 ms) 2024-08-17T17:17:43.680 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeDisabled_WithoutJournaling 2024-08-17T17:17:43.748 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeDisabled_WithoutJournaling (70 ms) 2024-08-17T17:17:43.749 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModePool_WithoutJournaling 2024-08-17T17:17:43.820 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModePool_WithoutJournaling (71 ms) 2024-08-17T17:17:43.820 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeImage_WithoutJournaling 2024-08-17T17:17:44.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:44 smithi136 ceph-mon[109861]: osdmap e433: 8 total, 8 up, 8 in 2024-08-17T17:17:44.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:44 smithi136 ceph-mon[115078]: osdmap e433: 8 total, 8 up, 8 in 2024-08-17T17:17:44.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:44 smithi165 ceph-mon[34868]: osdmap e433: 8 total, 8 up, 8 in 2024-08-17T17:17:45.093 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeImage_WithoutJournaling (1273 ms) 2024-08-17T17:17:45.093 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeImage_WithoutExclusiveLock 2024-08-17T17:17:45.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:45 smithi136 ceph-mon[115078]: pgmap v140: 105 pgs: 105 active+clean; 582 KiB data, 442 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:17:45.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:45 smithi136 ceph-mon[115078]: osdmap e434: 8 total, 8 up, 8 in 2024-08-17T17:17:45.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:45 smithi136 ceph-mon[109861]: pgmap v140: 105 pgs: 105 active+clean; 582 KiB data, 442 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:17:45.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:45 smithi136 ceph-mon[109861]: osdmap e434: 8 total, 8 up, 8 in 2024-08-17T17:17:45.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:45 smithi165 ceph-mon[34868]: pgmap v140: 105 pgs: 105 active+clean; 582 KiB data, 442 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:17:45.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:45 smithi165 ceph-mon[34868]: osdmap e434: 8 total, 8 up, 8 in 2024-08-17T17:17:46.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:46 smithi136 ceph-mon[109861]: osdmap e435: 8 total, 8 up, 8 in 2024-08-17T17:17:46.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:46 smithi136 ceph-mon[115078]: osdmap e435: 8 total, 8 up, 8 in 2024-08-17T17:17:46.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:46 smithi165 ceph-mon[34868]: osdmap e435: 8 total, 8 up, 8 in 2024-08-17T17:17:47.107 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeImage_WithoutExclusiveLock (2014 ms) 2024-08-17T17:17:47.107 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModeDisabled 2024-08-17T17:17:47.190 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModeDisabled (83 ms) 2024-08-17T17:17:47.190 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModeImage 2024-08-17T17:17:47.273 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModeImage (83 ms) 2024-08-17T17:17:47.274 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModePool 2024-08-17T17:17:47.362 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModePool (89 ms) 2024-08-17T17:17:47.362 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModePool_WithoutJournaling 2024-08-17T17:17:47.437 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModePool_WithoutJournaling (75 ms) 2024-08-17T17:17:47.437 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModeImage_WithoutJournaling 2024-08-17T17:17:47.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:47 smithi136 ceph-mon[109861]: pgmap v143: 105 pgs: 105 active+clean; 582 KiB data, 442 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:17:47.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:47 smithi136 ceph-mon[109861]: osdmap e436: 8 total, 8 up, 8 in 2024-08-17T17:17:47.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:47 smithi136 ceph-mon[115078]: pgmap v143: 105 pgs: 105 active+clean; 582 KiB data, 442 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:17:47.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:47 smithi136 ceph-mon[115078]: osdmap e436: 8 total, 8 up, 8 in 2024-08-17T17:17:47.509 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModeImage_WithoutJournaling (72 ms) 2024-08-17T17:17:47.509 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.EnableJournaling_In_MirrorModeDisabled 2024-08-17T17:17:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:47 smithi165 ceph-mon[34868]: pgmap v143: 105 pgs: 105 active+clean; 582 KiB data, 442 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:17:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:47 smithi165 ceph-mon[34868]: osdmap e436: 8 total, 8 up, 8 in 2024-08-17T17:17:47.595 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.EnableJournaling_In_MirrorModeDisabled (86 ms) 2024-08-17T17:17:47.595 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.EnableJournaling_In_MirrorModeImage 2024-08-17T17:17:47.684 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.EnableJournaling_In_MirrorModeImage (89 ms) 2024-08-17T17:17:47.685 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.EnableJournaling_In_MirrorModeImage_SnapshotMirroringEnabled 2024-08-17T17:17:47.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:17:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:17:47] "GET /metrics HTTP/1.1" 200 36301 "" "Prometheus/2.51.0" 2024-08-17T17:17:48.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:48 smithi136 ceph-mon[109861]: osdmap e437: 8 total, 8 up, 8 in 2024-08-17T17:17:48.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:48 smithi136 ceph-mon[115078]: osdmap e437: 8 total, 8 up, 8 in 2024-08-17T17:17:48.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:48 smithi165 ceph-mon[34868]: osdmap e437: 8 total, 8 up, 8 in 2024-08-17T17:17:48.835 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.EnableJournaling_In_MirrorModeImage_SnapshotMirroringEnabled (1151 ms) 2024-08-17T17:17:48.835 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.EnableJournaling_In_MirrorModePool 2024-08-17T17:17:48.934 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.EnableJournaling_In_MirrorModePool (99 ms) 2024-08-17T17:17:48.934 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.DisableJournaling_In_MirrorModePool 2024-08-17T17:17:49.028 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.DisableJournaling_In_MirrorModePool (93 ms) 2024-08-17T17:17:49.028 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.DisableJournaling_In_MirrorModeImage 2024-08-17T17:17:49.121 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.DisableJournaling_In_MirrorModeImage (94 ms) 2024-08-17T17:17:49.121 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.MirrorModeSet_DisabledMode_To_PoolMode 2024-08-17T17:17:49.266 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.MirrorModeSet_DisabledMode_To_PoolMode (146 ms) 2024-08-17T17:17:49.267 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.MirrorModeSet_PoolMode_To_DisabledMode 2024-08-17T17:17:49.410 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.MirrorModeSet_PoolMode_To_DisabledMode (143 ms) 2024-08-17T17:17:49.410 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.MirrorModeSet_DisabledMode_To_ImageMode 2024-08-17T17:17:49.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:49 smithi136 ceph-mon[109861]: pgmap v146: 105 pgs: 105 active+clean; 582 KiB data, 432 MiB used, 715 GiB / 715 GiB avail; 433 KiB/s rd, 58 KiB/s wr, 624 op/s 2024-08-17T17:17:49.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:49 smithi136 ceph-mon[109861]: osdmap e438: 8 total, 8 up, 8 in 2024-08-17T17:17:49.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:49 smithi136 ceph-mon[109861]: osdmap e439: 8 total, 8 up, 8 in 2024-08-17T17:17:49.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:49 smithi136 ceph-mon[115078]: pgmap v146: 105 pgs: 105 active+clean; 582 KiB data, 432 MiB used, 715 GiB / 715 GiB avail; 433 KiB/s rd, 58 KiB/s wr, 624 op/s 2024-08-17T17:17:49.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:49 smithi136 ceph-mon[115078]: osdmap e438: 8 total, 8 up, 8 in 2024-08-17T17:17:49.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:49 smithi136 ceph-mon[115078]: osdmap e439: 8 total, 8 up, 8 in 2024-08-17T17:17:49.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:49 smithi165 ceph-mon[34868]: pgmap v146: 105 pgs: 105 active+clean; 582 KiB data, 432 MiB used, 715 GiB / 715 GiB avail; 433 KiB/s rd, 58 KiB/s wr, 624 op/s 2024-08-17T17:17:49.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:49 smithi165 ceph-mon[34868]: osdmap e438: 8 total, 8 up, 8 in 2024-08-17T17:17:49.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:49 smithi165 ceph-mon[34868]: osdmap e439: 8 total, 8 up, 8 in 2024-08-17T17:17:49.535 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.MirrorModeSet_DisabledMode_To_ImageMode (125 ms) 2024-08-17T17:17:49.535 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.MirrorModeSet_PoolMode_To_ImageMode 2024-08-17T17:17:49.673 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.MirrorModeSet_PoolMode_To_ImageMode (138 ms) 2024-08-17T17:17:49.673 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.MirrorModeSet_ImageMode_To_PoolMode 2024-08-17T17:17:49.817 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.MirrorModeSet_ImageMode_To_PoolMode (144 ms) 2024-08-17T17:17:49.817 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.MirrorModeSet_ImageMode_To_DisabledMode 2024-08-17T17:17:49.977 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.MirrorModeSet_ImageMode_To_DisabledMode (160 ms) 2024-08-17T17:17:49.977 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.RemoveImage_With_MirrorImageEnabled 2024-08-17T17:17:50.057 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.RemoveImage_With_MirrorImageEnabled (80 ms) 2024-08-17T17:17:50.057 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.RemoveImage_With_MirrorImageDisabled 2024-08-17T17:17:50.128 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.RemoveImage_With_MirrorImageDisabled (71 ms) 2024-08-17T17:17:50.128 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.RemoveImage_With_ImageWithoutJournal 2024-08-17T17:17:50.195 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.RemoveImage_With_ImageWithoutJournal (67 ms) 2024-08-17T17:17:50.195 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.RemoveImage_With_MirrorImageDemoted 2024-08-17T17:17:50.309 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.RemoveImage_With_MirrorImageDemoted (114 ms) 2024-08-17T17:17:50.309 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.TrashMoveRestore_PoolMode 2024-08-17T17:17:50.417 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.TrashMoveRestore_PoolMode (108 ms) 2024-08-17T17:17:50.417 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.TrashMoveRestore_ImageMode_MirroringDisabled 2024-08-17T17:17:50.518 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.TrashMoveRestore_ImageMode_MirroringDisabled (101 ms) 2024-08-17T17:17:50.518 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.TrashMoveRestore_ImageMode_MirroringEnabled 2024-08-17T17:17:50.625 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.TrashMoveRestore_ImageMode_MirroringEnabled (107 ms) 2024-08-17T17:17:50.625 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.MirrorStatusList 2024-08-17T17:17:50.694 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.MirrorStatusList (69 ms) 2024-08-17T17:17:50.694 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.RemoveBootstrapped 2024-08-17T17:17:50.780 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.RemoveBootstrapped (86 ms) 2024-08-17T17:17:50.780 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.AioPromoteDemote 2024-08-17T17:17:51.116 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.AioPromoteDemote (336 ms) 2024-08-17T17:17:51.116 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.AioGetInfo 2024-08-17T17:17:51.442 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.AioGetInfo (326 ms) 2024-08-17T17:17:51.443 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.AioGetStatus 2024-08-17T17:17:51.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:51 smithi136 ceph-mon[109861]: pgmap v149: 105 pgs: 105 active+clean; 582 KiB data, 432 MiB used, 715 GiB / 715 GiB avail; 433 KiB/s rd, 58 KiB/s wr, 624 op/s 2024-08-17T17:17:51.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:51 smithi136 ceph-mon[115078]: pgmap v149: 105 pgs: 105 active+clean; 582 KiB data, 432 MiB used, 715 GiB / 715 GiB avail; 433 KiB/s rd, 58 KiB/s wr, 624 op/s 2024-08-17T17:17:51.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:51 smithi165 ceph-mon[34868]: pgmap v149: 105 pgs: 105 active+clean; 582 KiB data, 432 MiB used, 715 GiB / 715 GiB avail; 433 KiB/s rd, 58 KiB/s wr, 624 op/s 2024-08-17T17:17:51.648 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.AioGetStatus (206 ms) 2024-08-17T17:17:51.649 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.SiteName 2024-08-17T17:17:51.658 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.SiteName (10 ms) 2024-08-17T17:17:51.658 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.Bootstrap 2024-08-17T17:17:52.422 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.Bootstrap (765 ms) 2024-08-17T17:17:52.423 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.PeerDirection 2024-08-17T17:17:52.430 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.PeerDirection (8 ms) 2024-08-17T17:17:52.431 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.Snapshot 2024-08-17T17:17:52.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:17:52.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:52 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' 2024-08-17T17:17:52.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:52 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-08-17T17:17:52.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:52 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/site_name"} : dispatch 2024-08-17T17:17:52.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:52 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/site_name"} : dispatch 2024-08-17T17:17:52.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:52 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]': finished 2024-08-17T17:17:52.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:52 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-08-17T17:17:52.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:17:52.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:52 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' 2024-08-17T17:17:52.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:52 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-08-17T17:17:52.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:52 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/site_name"} : dispatch 2024-08-17T17:17:52.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:52 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/site_name"} : dispatch 2024-08-17T17:17:52.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:52 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]': finished 2024-08-17T17:17:52.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:52 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-08-17T17:17:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:52 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:17:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:52 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' 2024-08-17T17:17:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:52 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-08-17T17:17:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:52 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/site_name"} : dispatch 2024-08-17T17:17:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:52 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/site_name"} : dispatch 2024-08-17T17:17:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:52 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]': finished 2024-08-17T17:17:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:52 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-08-17T17:17:53.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:53 smithi136 ceph-mon[109861]: pgmap v150: 105 pgs: 105 active+clean; 583 KiB data, 453 MiB used, 715 GiB / 715 GiB avail; 1.5 MiB/s rd, 238 KiB/s wr, 2.12k op/s 2024-08-17T17:17:53.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:53 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer_client_id"} : dispatch 2024-08-17T17:17:53.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:53 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' 2024-08-17T17:17:53.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:53 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"} : dispatch 2024-08-17T17:17:53.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:53 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"} : dispatch 2024-08-17T17:17:53.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:53 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]': finished 2024-08-17T17:17:53.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:53 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-08-17T17:17:53.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:53 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/85b54dcb-5fc0-4464-90b0-9a85034788da"} : dispatch 2024-08-17T17:17:53.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:53 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/85b54dcb-5fc0-4464-90b0-9a85034788da"} : dispatch 2024-08-17T17:17:53.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:53 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/85b54dcb-5fc0-4464-90b0-9a85034788da"}]': finished 2024-08-17T17:17:53.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:53 smithi136 ceph-mon[115078]: pgmap v150: 105 pgs: 105 active+clean; 583 KiB data, 453 MiB used, 715 GiB / 715 GiB avail; 1.5 MiB/s rd, 238 KiB/s wr, 2.12k op/s 2024-08-17T17:17:53.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:53 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer_client_id"} : dispatch 2024-08-17T17:17:53.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:53 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' 2024-08-17T17:17:53.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:53 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"} : dispatch 2024-08-17T17:17:53.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:53 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"} : dispatch 2024-08-17T17:17:53.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:53 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]': finished 2024-08-17T17:17:53.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:53 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-08-17T17:17:53.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:53 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/85b54dcb-5fc0-4464-90b0-9a85034788da"} : dispatch 2024-08-17T17:17:53.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:53 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/85b54dcb-5fc0-4464-90b0-9a85034788da"} : dispatch 2024-08-17T17:17:53.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:53 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/85b54dcb-5fc0-4464-90b0-9a85034788da"}]': finished 2024-08-17T17:17:53.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:53 smithi165 ceph-mon[34868]: pgmap v150: 105 pgs: 105 active+clean; 583 KiB data, 453 MiB used, 715 GiB / 715 GiB avail; 1.5 MiB/s rd, 238 KiB/s wr, 2.12k op/s 2024-08-17T17:17:53.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:53 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer_client_id"} : dispatch 2024-08-17T17:17:53.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:53 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' 2024-08-17T17:17:53.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:53 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"} : dispatch 2024-08-17T17:17:53.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:53 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' cmd={ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"} : dispatch 2024-08-17T17:17:53.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:53 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' cmd='[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]': finished 2024-08-17T17:17:53.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:53 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-08-17T17:17:53.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:53 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/85b54dcb-5fc0-4464-90b0-9a85034788da"} : dispatch 2024-08-17T17:17:53.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:53 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/85b54dcb-5fc0-4464-90b0-9a85034788da"} : dispatch 2024-08-17T17:17:53.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:53 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/85b54dcb-5fc0-4464-90b0-9a85034788da"}]': finished 2024-08-17T17:17:54.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:54 smithi136 ceph-mon[109861]: osdmap e440: 8 total, 8 up, 8 in 2024-08-17T17:17:54.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:54 smithi136 ceph-mon[109861]: pgmap v152: 105 pgs: 105 active+clean; 583 KiB data, 457 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 216 KiB/s wr, 1.94k op/s 2024-08-17T17:17:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:54 smithi136 ceph-mon[115078]: osdmap e440: 8 total, 8 up, 8 in 2024-08-17T17:17:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:54 smithi136 ceph-mon[115078]: pgmap v152: 105 pgs: 105 active+clean; 583 KiB data, 457 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 216 KiB/s wr, 1.94k op/s 2024-08-17T17:17:54.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:54 smithi165 ceph-mon[34868]: osdmap e440: 8 total, 8 up, 8 in 2024-08-17T17:17:54.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:54 smithi165 ceph-mon[34868]: pgmap v152: 105 pgs: 105 active+clean; 583 KiB data, 457 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 216 KiB/s wr, 1.94k op/s 2024-08-17T17:17:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:55 smithi136 ceph-mon[109861]: osdmap e441: 8 total, 8 up, 8 in 2024-08-17T17:17:55.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:55 smithi136 ceph-mon[115078]: osdmap e441: 8 total, 8 up, 8 in 2024-08-17T17:17:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:55 smithi165 ceph-mon[34868]: osdmap e441: 8 total, 8 up, 8 in 2024-08-17T17:17:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:56 smithi165 ceph-mon[34868]: osdmap e442: 8 total, 8 up, 8 in 2024-08-17T17:17:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:56 smithi165 ceph-mon[34868]: pgmap v155: 105 pgs: 105 active+clean; 583 KiB data, 457 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 216 KiB/s wr, 1.94k op/s 2024-08-17T17:17:56.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:56 smithi136 ceph-mon[109861]: osdmap e442: 8 total, 8 up, 8 in 2024-08-17T17:17:56.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:56 smithi136 ceph-mon[109861]: pgmap v155: 105 pgs: 105 active+clean; 583 KiB data, 457 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 216 KiB/s wr, 1.94k op/s 2024-08-17T17:17:56.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:56 smithi136 ceph-mon[115078]: osdmap e442: 8 total, 8 up, 8 in 2024-08-17T17:17:56.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:56 smithi136 ceph-mon[115078]: pgmap v155: 105 pgs: 105 active+clean; 583 KiB data, 457 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 216 KiB/s wr, 1.94k op/s 2024-08-17T17:17:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:57 smithi165 ceph-mon[34868]: osdmap e443: 8 total, 8 up, 8 in 2024-08-17T17:17:57.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:57 smithi136 ceph-mon[109861]: osdmap e443: 8 total, 8 up, 8 in 2024-08-17T17:17:57.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:57 smithi136 ceph-mon[115078]: osdmap e443: 8 total, 8 up, 8 in 2024-08-17T17:17:57.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:17:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:17:57] "GET /metrics HTTP/1.1" 200 36264 "" "Prometheus/2.51.0" 2024-08-17T17:17:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:58 smithi165 ceph-mon[34868]: osdmap e444: 8 total, 8 up, 8 in 2024-08-17T17:17:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:58 smithi165 ceph-mon[34868]: pgmap v158: 105 pgs: 105 active+clean; 583 KiB data, 463 MiB used, 715 GiB / 715 GiB avail; 333 KiB/s rd, 29 KiB/s wr, 452 op/s 2024-08-17T17:17:58.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:58 smithi136 ceph-mon[109861]: osdmap e444: 8 total, 8 up, 8 in 2024-08-17T17:17:58.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:58 smithi136 ceph-mon[109861]: pgmap v158: 105 pgs: 105 active+clean; 583 KiB data, 463 MiB used, 715 GiB / 715 GiB avail; 333 KiB/s rd, 29 KiB/s wr, 452 op/s 2024-08-17T17:17:58.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:58 smithi136 ceph-mon[115078]: osdmap e444: 8 total, 8 up, 8 in 2024-08-17T17:17:58.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:58 smithi136 ceph-mon[115078]: pgmap v158: 105 pgs: 105 active+clean; 583 KiB data, 463 MiB used, 715 GiB / 715 GiB avail; 333 KiB/s rd, 29 KiB/s wr, 452 op/s 2024-08-17T17:17:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:59 smithi165 ceph-mon[34868]: osdmap e445: 8 total, 8 up, 8 in 2024-08-17T17:17:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:17:59 smithi165 ceph-mon[34868]: osdmap e446: 8 total, 8 up, 8 in 2024-08-17T17:17:59.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:59 smithi136 ceph-mon[109861]: osdmap e445: 8 total, 8 up, 8 in 2024-08-17T17:17:59.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:17:59 smithi136 ceph-mon[109861]: osdmap e446: 8 total, 8 up, 8 in 2024-08-17T17:17:59.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:59 smithi136 ceph-mon[115078]: osdmap e445: 8 total, 8 up, 8 in 2024-08-17T17:17:59.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:17:59 smithi136 ceph-mon[115078]: osdmap e446: 8 total, 8 up, 8 in 2024-08-17T17:18:01.131 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:00 smithi136 ceph-mon[109861]: osdmap e447: 8 total, 8 up, 8 in 2024-08-17T17:18:01.131 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:00 smithi136 ceph-mon[109861]: pgmap v162: 105 pgs: 105 active+clean; 583 KiB data, 463 MiB used, 715 GiB / 715 GiB avail; 394 KiB/s rd, 34 KiB/s wr, 534 op/s 2024-08-17T17:18:01.131 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:00 smithi136 ceph-mon[115078]: osdmap e447: 8 total, 8 up, 8 in 2024-08-17T17:18:01.131 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:00 smithi136 ceph-mon[115078]: pgmap v162: 105 pgs: 105 active+clean; 583 KiB data, 463 MiB used, 715 GiB / 715 GiB avail; 394 KiB/s rd, 34 KiB/s wr, 534 op/s 2024-08-17T17:18:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:00 smithi165 ceph-mon[34868]: osdmap e447: 8 total, 8 up, 8 in 2024-08-17T17:18:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:00 smithi165 ceph-mon[34868]: pgmap v162: 105 pgs: 105 active+clean; 583 KiB data, 463 MiB used, 715 GiB / 715 GiB avail; 394 KiB/s rd, 34 KiB/s wr, 534 op/s 2024-08-17T17:18:01.801 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'echo wait for majority of mons to upgrade' 2024-08-17T17:18:02.014 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:18:02.136 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:01 smithi136 ceph-mon[109861]: osdmap e448: 8 total, 8 up, 8 in 2024-08-17T17:18:02.137 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:01 smithi136 ceph-mon[115078]: osdmap e448: 8 total, 8 up, 8 in 2024-08-17T17:18:02.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:01 smithi165 ceph-mon[34868]: osdmap e448: 8 total, 8 up, 8 in 2024-08-17T17:18:03.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:02 smithi136 ceph-mon[109861]: osdmap e449: 8 total, 8 up, 8 in 2024-08-17T17:18:03.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:02 smithi136 ceph-mon[109861]: pgmap v165: 105 pgs: 105 active+clean; 583 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 9.5 KiB/s wr, 131 op/s 2024-08-17T17:18:03.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:02 smithi136 ceph-mon[115078]: osdmap e449: 8 total, 8 up, 8 in 2024-08-17T17:18:03.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:02 smithi136 ceph-mon[115078]: pgmap v165: 105 pgs: 105 active+clean; 583 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 9.5 KiB/s wr, 131 op/s 2024-08-17T17:18:03.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:02 smithi165 ceph-mon[34868]: osdmap e449: 8 total, 8 up, 8 in 2024-08-17T17:18:03.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:02 smithi165 ceph-mon[34868]: pgmap v165: 105 pgs: 105 active+clean; 583 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 9.5 KiB/s wr, 131 op/s 2024-08-17T17:18:03.802 INFO:teuthology.orchestra.run.smithi136.stdout:wait for majority of mons to upgrade 2024-08-17T17:18:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:03 smithi136 ceph-mon[109861]: osdmap e450: 8 total, 8 up, 8 in 2024-08-17T17:18:03.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:03 smithi136 ceph-mon[109861]: osdmap e451: 8 total, 8 up, 8 in 2024-08-17T17:18:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:03 smithi136 ceph-mon[115078]: osdmap e450: 8 total, 8 up, 8 in 2024-08-17T17:18:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:03 smithi136 ceph-mon[115078]: osdmap e451: 8 total, 8 up, 8 in 2024-08-17T17:18:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:03 smithi165 ceph-mon[34868]: osdmap e450: 8 total, 8 up, 8 in 2024-08-17T17:18:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:03 smithi165 ceph-mon[34868]: osdmap e451: 8 total, 8 up, 8 in 2024-08-17T17:18:04.495 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --limit 1' 2024-08-17T17:18:04.699 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:18:05.102 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:04 smithi136 ceph-mon[109861]: pgmap v168: 105 pgs: 105 active+clean; 583 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 8.0 KiB/s wr, 110 op/s 2024-08-17T17:18:05.102 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:04 smithi136 ceph-mon[115078]: pgmap v168: 105 pgs: 105 active+clean; 583 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 8.0 KiB/s wr, 110 op/s 2024-08-17T17:18:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:04 smithi165 ceph-mon[34868]: pgmap v168: 105 pgs: 105 active+clean; 583 KiB data, 448 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 8.0 KiB/s wr, 110 op/s 2024-08-17T17:18:06.134 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:05 smithi136 ceph-mon[109861]: osdmap e452: 8 total, 8 up, 8 in 2024-08-17T17:18:06.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:05 smithi136 ceph-mon[115078]: osdmap e452: 8 total, 8 up, 8 in 2024-08-17T17:18:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:05 smithi165 ceph-mon[34868]: osdmap e452: 8 total, 8 up, 8 in 2024-08-17T17:18:07.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:06 smithi136 ceph-mon[109861]: osdmap e453: 8 total, 8 up, 8 in 2024-08-17T17:18:07.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:06 smithi136 ceph-mon[109861]: pgmap v171: 105 pgs: 105 active+clean; 583 KiB data, 448 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:18:07.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:18:07.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:06 smithi136 ceph-mon[115078]: osdmap e453: 8 total, 8 up, 8 in 2024-08-17T17:18:07.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:06 smithi136 ceph-mon[115078]: pgmap v171: 105 pgs: 105 active+clean; 583 KiB data, 448 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:18:07.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:18:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:06 smithi165 ceph-mon[34868]: osdmap e453: 8 total, 8 up, 8 in 2024-08-17T17:18:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:06 smithi165 ceph-mon[34868]: pgmap v171: 105 pgs: 105 active+clean; 583 KiB data, 448 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:18:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:06 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:18:07.631 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:18:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:18:07] "GET /metrics HTTP/1.1" 200 36271 "" "Prometheus/2.51.0" 2024-08-17T17:18:07.916 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:07 smithi136 ceph-mon[109861]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453", "daemon_types": "mon", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:07.916 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:07 smithi136 ceph-mon[109861]: osdmap e454: 8 total, 8 up, 8 in 2024-08-17T17:18:07.917 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:07 smithi136 ceph-mon[115078]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453", "daemon_types": "mon", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:07.917 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:07 smithi136 ceph-mon[115078]: osdmap e454: 8 total, 8 up, 8 in 2024-08-17T17:18:07.917 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.Snapshot (15486 ms) 2024-08-17T17:18:07.917 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.SnapshotRemoveOnDisable 2024-08-17T17:18:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:07 smithi165 ceph-mon[34868]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453", "daemon_types": "mon", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:07 smithi165 ceph-mon[34868]: osdmap e454: 8 total, 8 up, 8 in 2024-08-17T17:18:08.647 INFO:teuthology.orchestra.run.smithi136.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:18:08.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:08 smithi136 ceph-mon[115078]: osdmap e455: 8 total, 8 up, 8 in 2024-08-17T17:18:08.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:08 smithi136 ceph-mon[115078]: pgmap v174: 105 pgs: 105 active+clean; 583 KiB data, 436 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 4.0 KiB/s wr, 63 op/s 2024-08-17T17:18:08.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:08 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/bf15be50-475a-4079-a4b0-2b1e95ecb306"} : dispatch 2024-08-17T17:18:08.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:08 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/bf15be50-475a-4079-a4b0-2b1e95ecb306"} : dispatch 2024-08-17T17:18:08.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:08 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/bf15be50-475a-4079-a4b0-2b1e95ecb306"}]': finished 2024-08-17T17:18:08.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:08.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:18:08.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:08.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:18:08.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:08.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:08 smithi136 ceph-mon[109861]: osdmap e455: 8 total, 8 up, 8 in 2024-08-17T17:18:08.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:08 smithi136 ceph-mon[109861]: pgmap v174: 105 pgs: 105 active+clean; 583 KiB data, 436 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 4.0 KiB/s wr, 63 op/s 2024-08-17T17:18:08.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:08 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/bf15be50-475a-4079-a4b0-2b1e95ecb306"} : dispatch 2024-08-17T17:18:08.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:08 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/bf15be50-475a-4079-a4b0-2b1e95ecb306"} : dispatch 2024-08-17T17:18:08.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:08 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/bf15be50-475a-4079-a4b0-2b1e95ecb306"}]': finished 2024-08-17T17:18:08.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:08.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:18:08.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:08.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:18:08.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:08 smithi165 ceph-mon[34868]: osdmap e455: 8 total, 8 up, 8 in 2024-08-17T17:18:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:08 smithi165 ceph-mon[34868]: pgmap v174: 105 pgs: 105 active+clean; 583 KiB data, 436 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 4.0 KiB/s wr, 63 op/s 2024-08-17T17:18:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:08 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/bf15be50-475a-4079-a4b0-2b1e95ecb306"} : dispatch 2024-08-17T17:18:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:08 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/bf15be50-475a-4079-a4b0-2b1e95ecb306"} : dispatch 2024-08-17T17:18:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:08 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/bf15be50-475a-4079-a4b0-2b1e95ecb306"}]': finished 2024-08-17T17:18:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:08 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:08 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:18:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:08 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:09.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:08 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:18:09.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:08 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:09.263 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-08-17T17:18:09.482 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:18:10.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:09 smithi136 ceph-mon[115078]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:18:10.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:09 smithi136 ceph-mon[115078]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:18:10.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:09 smithi136 ceph-mon[115078]: osdmap e456: 8 total, 8 up, 8 in 2024-08-17T17:18:10.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:09 smithi136 ceph-mon[109861]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:18:10.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:09 smithi136 ceph-mon[109861]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:18:10.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:09 smithi136 ceph-mon[109861]: osdmap e456: 8 total, 8 up, 8 in 2024-08-17T17:18:10.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:09 smithi165 ceph-mon[34868]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:18:10.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:09 smithi165 ceph-mon[34868]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:18:10.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:09 smithi165 ceph-mon[34868]: osdmap e456: 8 total, 8 up, 8 in 2024-08-17T17:18:11.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:10 smithi136 ceph-mon[115078]: pgmap v176: 105 pgs: 105 active+clean; 583 KiB data, 436 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 4.0 KiB/s wr, 63 op/s 2024-08-17T17:18:11.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:10 smithi136 ceph-mon[115078]: osdmap e457: 8 total, 8 up, 8 in 2024-08-17T17:18:11.049 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:10 smithi136 ceph-mon[109861]: pgmap v176: 105 pgs: 105 active+clean; 583 KiB data, 436 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 4.0 KiB/s wr, 63 op/s 2024-08-17T17:18:11.049 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:10 smithi136 ceph-mon[109861]: osdmap e457: 8 total, 8 up, 8 in 2024-08-17T17:18:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:10 smithi165 ceph-mon[34868]: pgmap v176: 105 pgs: 105 active+clean; 583 KiB data, 436 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 4.0 KiB/s wr, 63 op/s 2024-08-17T17:18:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:10 smithi165 ceph-mon[34868]: osdmap e457: 8 total, 8 up, 8 in 2024-08-17T17:18:11.673 INFO:teuthology.orchestra.run.smithi136.stdout:true 2024-08-17T17:18:12.021 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.SnapshotRemoveOnDisable (4104 ms) 2024-08-17T17:18:12.021 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.SnapshotUnlinkPeer 2024-08-17T17:18:12.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:11 smithi136 ceph-mon[109861]: osdmap e458: 8 total, 8 up, 8 in 2024-08-17T17:18:12.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:11 smithi136 ceph-mon[115078]: osdmap e458: 8 total, 8 up, 8 in 2024-08-17T17:18:12.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:11 smithi165 ceph-mon[34868]: osdmap e458: 8 total, 8 up, 8 in 2024-08-17T17:18:12.272 INFO:teuthology.orchestra.run.smithi136.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-17T17:18:12.272 INFO:teuthology.orchestra.run.smithi136.stdout:alertmanager.a smithi136 *:9093,9094 running (3m) 34s ago 9m 20.9M - 0.27.0 11f11916f8cd c3b9b5b223ab 2024-08-17T17:18:12.272 INFO:teuthology.orchestra.run.smithi136.stdout:grafana.a smithi165 *:3000 running (2m) 33s ago 8m 76.9M - c8b91775d855 04db24fce5e9 2024-08-17T17:18:12.272 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi136.ltsuqd smithi136 running (7m) 34s ago 7m 21.4M - 18.2.4-719-gced649f8 c870753d67c2 faaa06f2950c 2024-08-17T17:18:12.272 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi165.bmrbhh smithi165 running (7m) 33s ago 7m 17.1M - 18.2.4-719-gced649f8 c870753d67c2 c74b00d929cf 2024-08-17T17:18:12.272 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.x smithi165 *:8443,9283,8765 running (4m) 33s ago 13m 455M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 837fd48ff7cb 2024-08-17T17:18:12.272 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.y smithi136 *:8443,9283,8765 running (4m) 34s ago 14m 510M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9acdc6775e48 2024-08-17T17:18:12.272 INFO:teuthology.orchestra.run.smithi136.stdout:mon.a smithi136 running (2m) 34s ago 14m 47.3M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9cf82e613626 2024-08-17T17:18:12.272 INFO:teuthology.orchestra.run.smithi136.stdout:mon.b smithi165 running (13m) 33s ago 13m 59.8M 2048M 18.2.4-719-gced649f8 c870753d67c2 d587e16140e1 2024-08-17T17:18:12.272 INFO:teuthology.orchestra.run.smithi136.stdout:mon.c smithi136 running (111s) 34s ago 13m 41.8M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 2c2ac98ad9d4 2024-08-17T17:18:12.273 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.a smithi136 *:9100 running (4m) 34s ago 8m 10.6M - 1.7.0 72c9c2088986 1e4c644ae533 2024-08-17T17:18:12.273 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.b smithi165 *:9100 running (4m) 33s ago 8m 10.2M - 1.7.0 72c9c2088986 7af2d2d68b06 2024-08-17T17:18:12.273 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0 smithi136 running (12m) 34s ago 12m 110M 2480M 18.2.4-719-gced649f8 c870753d67c2 faab916734b0 2024-08-17T17:18:12.273 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1 smithi136 running (12m) 34s ago 12m 145M 2480M 18.2.4-719-gced649f8 c870753d67c2 861686b6ff82 2024-08-17T17:18:12.273 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2 smithi136 running (11m) 34s ago 11m 185M 2480M 18.2.4-719-gced649f8 c870753d67c2 640be4af4156 2024-08-17T17:18:12.273 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3 smithi136 running (11m) 34s ago 11m 139M 2480M 18.2.4-719-gced649f8 c870753d67c2 14b57a745d30 2024-08-17T17:18:12.273 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4 smithi165 running (11m) 33s ago 11m 149M 2480M 18.2.4-719-gced649f8 c870753d67c2 df366c73e1f2 2024-08-17T17:18:12.273 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5 smithi165 running (10m) 33s ago 10m 107M 2480M 18.2.4-719-gced649f8 c870753d67c2 1131665e203a 2024-08-17T17:18:12.273 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6 smithi165 running (10m) 33s ago 10m 165M 2480M 18.2.4-719-gced649f8 c870753d67c2 7bc5fd925293 2024-08-17T17:18:12.273 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7 smithi165 running (10m) 33s ago 10m 128M 2480M 18.2.4-719-gced649f8 c870753d67c2 18382e4ae3b8 2024-08-17T17:18:12.273 INFO:teuthology.orchestra.run.smithi136.stdout:prometheus.a smithi165 *:9095 running (3m) 33s ago 9m 47.8M - 2.51.0 1d3b7f56885b ae8f8b69fe2f 2024-08-17T17:18:12.638 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:18:12.638 INFO:teuthology.orchestra.run.smithi136.stdout: "mon": { 2024-08-17T17:18:12.638 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 1, 2024-08-17T17:18:12.638 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 2 2024-08-17T17:18:12.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:18:12.638 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr": { 2024-08-17T17:18:12.638 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 2 2024-08-17T17:18:12.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:18:12.638 INFO:teuthology.orchestra.run.smithi136.stdout: "osd": { 2024-08-17T17:18:12.638 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 8 2024-08-17T17:18:12.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:18:12.639 INFO:teuthology.orchestra.run.smithi136.stdout: "mds": { 2024-08-17T17:18:12.639 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 2 2024-08-17T17:18:12.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:18:12.639 INFO:teuthology.orchestra.run.smithi136.stdout: "overall": { 2024-08-17T17:18:12.639 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 11, 2024-08-17T17:18:12.639 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 4 2024-08-17T17:18:12.639 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:18:12.639 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:18:12.931 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:18:12.931 INFO:teuthology.orchestra.run.smithi136.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d", 2024-08-17T17:18:12.931 INFO:teuthology.orchestra.run.smithi136.stdout: "in_progress": true, 2024-08-17T17:18:12.931 INFO:teuthology.orchestra.run.smithi136.stdout: "which": "Upgrading daemons of type(s) mon. Upgrade limited to 1 daemons (1 remaining).", 2024-08-17T17:18:12.931 INFO:teuthology.orchestra.run.smithi136.stdout: "services_complete": [], 2024-08-17T17:18:12.931 INFO:teuthology.orchestra.run.smithi136.stdout: "progress": "2/3 daemons upgraded", 2024-08-17T17:18:12.931 INFO:teuthology.orchestra.run.smithi136.stdout: "message": "Currently upgrading mon daemons", 2024-08-17T17:18:12.931 INFO:teuthology.orchestra.run.smithi136.stdout: "is_paused": false 2024-08-17T17:18:12.931 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:18:13.096 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:12 smithi165 ceph-mon[34868]: from='client.44144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:13.096 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:12 smithi165 ceph-mon[34868]: pgmap v179: 105 pgs: 105 active+clean; 583 KiB data, 437 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 9.5 KiB/s wr, 110 op/s 2024-08-17T17:18:13.097 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:12 smithi165 ceph-mon[34868]: osdmap e459: 8 total, 8 up, 8 in 2024-08-17T17:18:13.097 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:12 smithi165 ceph-mon[34868]: from='client.34236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:13.097 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:12 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/86320b6c-3f31-42d6-89eb-4672ddb1415c"} : dispatch 2024-08-17T17:18:13.097 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:12 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/86320b6c-3f31-42d6-89eb-4672ddb1415c"} : dispatch 2024-08-17T17:18:13.097 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:12 smithi165 ceph-mon[34868]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/86320b6c-3f31-42d6-89eb-4672ddb1415c"}]': finished 2024-08-17T17:18:13.097 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:12 smithi165 ceph-mon[34868]: from='client.34242 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:13.097 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:12 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:13.097 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:12 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:18:13.097 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:12 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:13.097 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:12 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:13.097 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:12 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-08-17T17:18:13.097 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:12 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-08-17T17:18:13.097 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:12 smithi165 ceph-mon[34868]: from='client.? 172.21.15.136:0/2796296601' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:13.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[109861]: from='client.44144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:13.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[109861]: pgmap v179: 105 pgs: 105 active+clean; 583 KiB data, 437 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 9.5 KiB/s wr, 110 op/s 2024-08-17T17:18:13.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[109861]: osdmap e459: 8 total, 8 up, 8 in 2024-08-17T17:18:13.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[109861]: from='client.34236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:13.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/86320b6c-3f31-42d6-89eb-4672ddb1415c"} : dispatch 2024-08-17T17:18:13.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/86320b6c-3f31-42d6-89eb-4672ddb1415c"} : dispatch 2024-08-17T17:18:13.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/86320b6c-3f31-42d6-89eb-4672ddb1415c"}]': finished 2024-08-17T17:18:13.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[109861]: from='client.34242 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:13.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:13.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:18:13.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:13.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:13.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-08-17T17:18:13.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-08-17T17:18:13.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2796296601' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:13.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[115078]: from='client.44144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:13.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[115078]: pgmap v179: 105 pgs: 105 active+clean; 583 KiB data, 437 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 9.5 KiB/s wr, 110 op/s 2024-08-17T17:18:13.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[115078]: osdmap e459: 8 total, 8 up, 8 in 2024-08-17T17:18:13.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[115078]: from='client.34236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:13.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/86320b6c-3f31-42d6-89eb-4672ddb1415c"} : dispatch 2024-08-17T17:18:13.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/86320b6c-3f31-42d6-89eb-4672ddb1415c"} : dispatch 2024-08-17T17:18:13.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/86320b6c-3f31-42d6-89eb-4672ddb1415c"}]': finished 2024-08-17T17:18:13.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[115078]: from='client.34242 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:13.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:13.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:18:13.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:13.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:13.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-08-17T17:18:13.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-08-17T17:18:13.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:12 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2796296601' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:14 smithi165 ceph-mon[34868]: Upgrade: Target is version 19.0.0-5836-gcd8bdbeb (squid) 2024-08-17T17:18:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:14 smithi165 ceph-mon[34868]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d'] 2024-08-17T17:18:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:14 smithi165 ceph-mon[34868]: Upgrade: Setting container_image for all mgr 2024-08-17T17:18:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:14 smithi165 ceph-mon[34868]: Upgrade: It appears safe to stop mon.b 2024-08-17T17:18:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:14 smithi165 ceph-mon[34868]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:14 smithi165 ceph-mon[34868]: osdmap e460: 8 total, 8 up, 8 in 2024-08-17T17:18:14.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:14 smithi165 ceph-mon[34868]: osdmap e461: 8 total, 8 up, 8 in 2024-08-17T17:18:14.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:14 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:14.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:14 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:18:14.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:14 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:18:14.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:14 smithi165 ceph-mon[34868]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:14.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:14 smithi136 ceph-mon[109861]: Upgrade: Target is version 19.0.0-5836-gcd8bdbeb (squid) 2024-08-17T17:18:14.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:14 smithi136 ceph-mon[109861]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d'] 2024-08-17T17:18:14.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:14 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all mgr 2024-08-17T17:18:14.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:14 smithi136 ceph-mon[109861]: Upgrade: It appears safe to stop mon.b 2024-08-17T17:18:14.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:14 smithi136 ceph-mon[109861]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:14.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:14 smithi136 ceph-mon[109861]: osdmap e460: 8 total, 8 up, 8 in 2024-08-17T17:18:14.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:14 smithi136 ceph-mon[109861]: osdmap e461: 8 total, 8 up, 8 in 2024-08-17T17:18:14.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:14 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:14.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:14 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:18:14.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:14 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:18:14.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:14 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:14.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:14 smithi136 ceph-mon[115078]: Upgrade: Target is version 19.0.0-5836-gcd8bdbeb (squid) 2024-08-17T17:18:14.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:14 smithi136 ceph-mon[115078]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d'] 2024-08-17T17:18:14.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:14 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all mgr 2024-08-17T17:18:14.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:14 smithi136 ceph-mon[115078]: Upgrade: It appears safe to stop mon.b 2024-08-17T17:18:14.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:14 smithi136 ceph-mon[115078]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:14.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:14 smithi136 ceph-mon[115078]: osdmap e460: 8 total, 8 up, 8 in 2024-08-17T17:18:14.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:14 smithi136 ceph-mon[115078]: osdmap e461: 8 total, 8 up, 8 in 2024-08-17T17:18:14.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:14 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:14.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:14 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:18:14.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:14 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:18:14.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:14 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:15.432 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:15 smithi165 ceph-mon[34868]: pgmap v183: 105 pgs: 105 active+clean; 583 KiB data, 437 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 9.7 KiB/s wr, 113 op/s 2024-08-17T17:18:15.432 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:15 smithi165 ceph-mon[34868]: Upgrade: Updating mon.b 2024-08-17T17:18:15.432 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:15 smithi165 ceph-mon[34868]: Deploying daemon mon.b on smithi165 2024-08-17T17:18:15.432 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:15 smithi165 ceph-mon[34868]: osdmap e462: 8 total, 8 up, 8 in 2024-08-17T17:18:15.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:15 smithi136 ceph-mon[109861]: pgmap v183: 105 pgs: 105 active+clean; 583 KiB data, 437 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 9.7 KiB/s wr, 113 op/s 2024-08-17T17:18:15.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:15 smithi136 ceph-mon[109861]: Upgrade: Updating mon.b 2024-08-17T17:18:15.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:15 smithi136 ceph-mon[109861]: Deploying daemon mon.b on smithi165 2024-08-17T17:18:15.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:15 smithi136 ceph-mon[109861]: osdmap e462: 8 total, 8 up, 8 in 2024-08-17T17:18:15.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:15 smithi136 ceph-mon[115078]: pgmap v183: 105 pgs: 105 active+clean; 583 KiB data, 437 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 9.7 KiB/s wr, 113 op/s 2024-08-17T17:18:15.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:15 smithi136 ceph-mon[115078]: Upgrade: Updating mon.b 2024-08-17T17:18:15.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:15 smithi136 ceph-mon[115078]: Deploying daemon mon.b on smithi165 2024-08-17T17:18:15.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:15 smithi136 ceph-mon[115078]: osdmap e462: 8 total, 8 up, 8 in 2024-08-17T17:18:15.972 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:15 smithi165 systemd[1]: Stopping Ceph mon.b for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:18:16.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-b[34845]: 2024-08-17T17:18:15.969+0000 7f24d3574640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:18:16.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-b[34845]: 2024-08-17T17:18:15.969+0000 7f24d3574640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-08-17T17:18:16.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:16 smithi165 podman[105328]: 2024-08-17 17:18:16.021791026 +0000 UTC m=+0.155126843 container died d587e16140e1c5633365c903f3f83b845de47b3221003b11b60d47804bd4a276 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=reef-ced649f, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux ) 2024-08-17T17:18:16.628 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:16 smithi165 podman[105328]: 2024-08-17 17:18:16.385260176 +0000 UTC m=+0.518595986 container cleanup d587e16140e1c5633365c903f3f83b845de47b3221003b11b60d47804bd4a276 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=reef-ced649f, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=) 2024-08-17T17:18:16.628 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:16 smithi165 bash[105328]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-b 2024-08-17T17:18:16.972 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:16 smithi165 podman[105346]: 2024-08-17 17:18:16.628183173 +0000 UTC m=+0.601190848 container remove d587e16140e1c5633365c903f3f83b845de47b3221003b11b60d47804bd4a276 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-b, RELEASE=reef-ced649f, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-08-17T17:18:16.972 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:16 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@mon.b.service: Deactivated successfully. 2024-08-17T17:18:16.972 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:16 smithi165 systemd[1]: Stopped Ceph mon.b for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:18:16.973 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:16 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@mon.b.service: Consumed 11.576s CPU time. 2024-08-17T17:18:17.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:16 smithi165 systemd[1]: Starting Ceph mon.b for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:18:17.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 podman[105479]: 2024-08-17 17:18:17.179166298 +0000 UTC m=+0.106727866 container create 1fd357308b8d9c5756303eb35cb2497df3ef60b5fe965f83d94c0fbfeec85b53 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-cd8bdbe, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T17:18:17.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 podman[105479]: 2024-08-17 17:18:17.090037847 +0000 UTC m=+0.017599432 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:18:17.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 podman[105479]: 2024-08-17 17:18:17.389675943 +0000 UTC m=+0.317237520 container init 1fd357308b8d9c5756303eb35cb2497df3ef60b5fe965f83d94c0fbfeec85b53 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=main-cd8bdbe, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T17:18:17.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 podman[105479]: 2024-08-17 17:18:17.394628302 +0000 UTC m=+0.322189866 container start 1fd357308b8d9c5756303eb35cb2497df3ef60b5fe965f83d94c0fbfeec85b53 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-b, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-cd8bdbe, GIT_CLEAN=True) 2024-08-17T17:18:17.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: set uid:gid to 167:167 (ceph:ceph) 2024-08-17T17:18:17.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev), process ceph-mon, pid 7 2024-08-17T17:18:17.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: pidfile_write: ignore empty --pid-file 2024-08-17T17:18:17.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: load: jerasure load: lrc 2024-08-17T17:18:17.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: RocksDB version: 7.9.2 2024-08-17T17:18:17.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Git sha 0 2024-08-17T17:18:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Compile date 2024-08-15 20:43:31 2024-08-17T17:18:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: DB SUMMARY 2024-08-17T17:18:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: DB Session ID: OREJKVJ5PIU9JRQDPMCZ 2024-08-17T17:18:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: CURRENT file: CURRENT 2024-08-17T17:18:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: IDENTITY file: IDENTITY 2024-08-17T17:18:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: MANIFEST file: MANIFEST-000010 size: 1656 Bytes 2024-08-17T17:18:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000030.sst 2024-08-17T17:18:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000028.log size: 4408254 ; 2024-08-17T17:18:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.error_if_exists: 0 2024-08-17T17:18:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.create_if_missing: 0 2024-08-17T17:18:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.paranoid_checks: 1 2024-08-17T17:18:17.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.flush_verify_memtable_count: 1 2024-08-17T17:18:17.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-08-17T17:18:17.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-08-17T17:18:17.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.env: 0x55c6aae1fbe0 2024-08-17T17:18:17.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.fs: PosixFileSystem 2024-08-17T17:18:17.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.info_log: 0x55c6abbf3700 2024-08-17T17:18:17.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_file_opening_threads: 16 2024-08-17T17:18:17.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.statistics: (nil) 2024-08-17T17:18:17.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.use_fsync: 0 2024-08-17T17:18:17.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_log_file_size: 0 2024-08-17T17:18:17.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-08-17T17:18:17.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.log_file_time_to_roll: 0 2024-08-17T17:18:17.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.keep_log_file_num: 1000 2024-08-17T17:18:17.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.recycle_log_file_num: 0 2024-08-17T17:18:17.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.allow_fallocate: 1 2024-08-17T17:18:17.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.allow_mmap_reads: 0 2024-08-17T17:18:17.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.allow_mmap_writes: 0 2024-08-17T17:18:17.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.use_direct_reads: 0 2024-08-17T17:18:17.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-08-17T17:18:17.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.create_missing_column_families: 0 2024-08-17T17:18:17.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.db_log_dir: 2024-08-17T17:18:17.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.wal_dir: 2024-08-17T17:18:17.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.table_cache_numshardbits: 6 2024-08-17T17:18:17.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.WAL_ttl_seconds: 0 2024-08-17T17:18:17.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.WAL_size_limit_MB: 0 2024-08-17T17:18:17.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-08-17T17:18:17.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-08-17T17:18:17.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.is_fd_close_on_exec: 1 2024-08-17T17:18:17.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.advise_random_on_open: 1 2024-08-17T17:18:17.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.db_write_buffer_size: 0 2024-08-17T17:18:17.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.write_buffer_manager: 0x55c6abbf6960 2024-08-17T17:18:17.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-08-17T17:18:17.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-08-17T17:18:17.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.use_adaptive_mutex: 0 2024-08-17T17:18:17.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.rate_limiter: (nil) 2024-08-17T17:18:17.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-08-17T17:18:17.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.wal_recovery_mode: 2 2024-08-17T17:18:17.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.enable_thread_tracking: 0 2024-08-17T17:18:17.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.enable_pipelined_write: 0 2024-08-17T17:18:17.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.unordered_write: 0 2024-08-17T17:18:17.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-08-17T17:18:17.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-08-17T17:18:17.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-08-17T17:18:17.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-08-17T17:18:17.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.row_cache: None 2024-08-17T17:18:17.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.wal_filter: None 2024-08-17T17:18:17.768 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-08-17T17:18:17.769 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.allow_ingest_behind: 0 2024-08-17T17:18:17.769 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.two_write_queues: 0 2024-08-17T17:18:17.769 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.manual_wal_flush: 0 2024-08-17T17:18:17.769 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.wal_compression: 0 2024-08-17T17:18:17.769 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.atomic_flush: 0 2024-08-17T17:18:17.769 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-08-17T17:18:17.769 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.persist_stats_to_disk: 0 2024-08-17T17:18:17.769 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.write_dbid_to_manifest: 0 2024-08-17T17:18:17.769 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.log_readahead_size: 0 2024-08-17T17:18:17.769 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-08-17T17:18:17.769 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.best_efforts_recovery: 0 2024-08-17T17:18:17.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-08-17T17:18:17.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-08-17T17:18:17.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.allow_data_in_errors: 0 2024-08-17T17:18:17.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.db_host_id: __hostname__ 2024-08-17T17:18:17.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.enforce_single_del_contracts: true 2024-08-17T17:18:17.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_background_jobs: 2 2024-08-17T17:18:17.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_background_compactions: -1 2024-08-17T17:18:17.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_subcompactions: 1 2024-08-17T17:18:17.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-08-17T17:18:17.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-08-17T17:18:17.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.delayed_write_rate : 16777216 2024-08-17T17:18:17.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_total_wal_size: 0 2024-08-17T17:18:17.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-08-17T17:18:17.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.stats_dump_period_sec: 600 2024-08-17T17:18:17.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.stats_persist_period_sec: 600 2024-08-17T17:18:17.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-08-17T17:18:17.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_open_files: -1 2024-08-17T17:18:17.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.bytes_per_sync: 0 2024-08-17T17:18:17.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.wal_bytes_per_sync: 0 2024-08-17T17:18:17.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.strict_bytes_per_sync: 0 2024-08-17T17:18:17.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.compaction_readahead_size: 0 2024-08-17T17:18:17.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_background_flushes: -1 2024-08-17T17:18:17.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Compression algorithms supported: 2024-08-17T17:18:17.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: kZSTD supported: 0 2024-08-17T17:18:17.772 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: kXpressCompression supported: 0 2024-08-17T17:18:17.772 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: kBZip2Compression supported: 0 2024-08-17T17:18:17.772 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-08-17T17:18:17.772 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: kLZ4Compression supported: 1 2024-08-17T17:18:17.772 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: kZlibCompression supported: 1 2024-08-17T17:18:17.773 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: kLZ4HCCompression supported: 1 2024-08-17T17:18:17.773 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: kSnappyCompression supported: 1 2024-08-17T17:18:17.773 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Fast CRC32 supported: Supported on x86 2024-08-17T17:18:17.773 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: DMutex implementation: pthread_mutex_t 2024-08-17T17:18:17.773 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000010 2024-08-17T17:18:17.774 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-08-17T17:18:17.774 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-08-17T17:18:17.774 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.merge_operator: 2024-08-17T17:18:17.774 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.compaction_filter: None 2024-08-17T17:18:17.774 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.compaction_filter_factory: None 2024-08-17T17:18:17.774 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.sst_partitioner_factory: None 2024-08-17T17:18:17.774 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.memtable_factory: SkipListFactory 2024-08-17T17:18:17.774 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.table_factory: BlockBasedTable 2024-08-17T17:18:17.775 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c6abbf3380) 2024-08-17T17:18:17.775 INFO:journalctl@ceph.mon.b.smithi165.stdout: cache_index_and_filter_blocks: 1 2024-08-17T17:18:17.775 INFO:journalctl@ceph.mon.b.smithi165.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-08-17T17:18:17.775 INFO:journalctl@ceph.mon.b.smithi165.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-08-17T17:18:17.775 INFO:journalctl@ceph.mon.b.smithi165.stdout: pin_top_level_index_and_filter: 1 2024-08-17T17:18:17.775 INFO:journalctl@ceph.mon.b.smithi165.stdout: index_type: 0 2024-08-17T17:18:17.775 INFO:journalctl@ceph.mon.b.smithi165.stdout: data_block_index_type: 0 2024-08-17T17:18:17.775 INFO:journalctl@ceph.mon.b.smithi165.stdout: index_shortening: 1 2024-08-17T17:18:17.775 INFO:journalctl@ceph.mon.b.smithi165.stdout: data_block_hash_table_util_ratio: 0.750000 2024-08-17T17:18:17.775 INFO:journalctl@ceph.mon.b.smithi165.stdout: checksum: 4 2024-08-17T17:18:17.775 INFO:journalctl@ceph.mon.b.smithi165.stdout: no_block_cache: 0 2024-08-17T17:18:17.775 INFO:journalctl@ceph.mon.b.smithi165.stdout: block_cache: 0x55c6abbe7610 2024-08-17T17:18:17.775 INFO:journalctl@ceph.mon.b.smithi165.stdout: block_cache_name: BinnedLRUCache 2024-08-17T17:18:17.776 INFO:journalctl@ceph.mon.b.smithi165.stdout: block_cache_options: 2024-08-17T17:18:17.776 INFO:journalctl@ceph.mon.b.smithi165.stdout: capacity : 536870912 2024-08-17T17:18:17.776 INFO:journalctl@ceph.mon.b.smithi165.stdout: num_shard_bits : 4 2024-08-17T17:18:17.776 INFO:journalctl@ceph.mon.b.smithi165.stdout: strict_capacity_limit : 0 2024-08-17T17:18:17.776 INFO:journalctl@ceph.mon.b.smithi165.stdout: high_pri_pool_ratio: 0.000 2024-08-17T17:18:17.776 INFO:journalctl@ceph.mon.b.smithi165.stdout: block_cache_compressed: (nil) 2024-08-17T17:18:17.776 INFO:journalctl@ceph.mon.b.smithi165.stdout: persistent_cache: (nil) 2024-08-17T17:18:17.776 INFO:journalctl@ceph.mon.b.smithi165.stdout: block_size: 4096 2024-08-17T17:18:17.776 INFO:journalctl@ceph.mon.b.smithi165.stdout: block_size_deviation: 10 2024-08-17T17:18:17.776 INFO:journalctl@ceph.mon.b.smithi165.stdout: block_restart_interval: 16 2024-08-17T17:18:17.776 INFO:journalctl@ceph.mon.b.smithi165.stdout: index_block_restart_interval: 1 2024-08-17T17:18:17.776 INFO:journalctl@ceph.mon.b.smithi165.stdout: metadata_block_size: 4096 2024-08-17T17:18:17.776 INFO:journalctl@ceph.mon.b.smithi165.stdout: partition_filters: 0 2024-08-17T17:18:17.776 INFO:journalctl@ceph.mon.b.smithi165.stdout: use_delta_encoding: 1 2024-08-17T17:18:17.777 INFO:journalctl@ceph.mon.b.smithi165.stdout: filter_policy: bloomfilter 2024-08-17T17:18:17.777 INFO:journalctl@ceph.mon.b.smithi165.stdout: whole_key_filtering: 1 2024-08-17T17:18:17.777 INFO:journalctl@ceph.mon.b.smithi165.stdout: verify_compression: 0 2024-08-17T17:18:17.777 INFO:journalctl@ceph.mon.b.smithi165.stdout: read_amp_bytes_per_bit: 0 2024-08-17T17:18:17.777 INFO:journalctl@ceph.mon.b.smithi165.stdout: format_version: 5 2024-08-17T17:18:17.777 INFO:journalctl@ceph.mon.b.smithi165.stdout: enable_index_compression: 1 2024-08-17T17:18:17.777 INFO:journalctl@ceph.mon.b.smithi165.stdout: block_align: 0 2024-08-17T17:18:17.777 INFO:journalctl@ceph.mon.b.smithi165.stdout: max_auto_readahead_size: 262144 2024-08-17T17:18:17.777 INFO:journalctl@ceph.mon.b.smithi165.stdout: prepopulate_block_cache: 0 2024-08-17T17:18:17.777 INFO:journalctl@ceph.mon.b.smithi165.stdout: initial_auto_readahead_size: 8192 2024-08-17T17:18:17.777 INFO:journalctl@ceph.mon.b.smithi165.stdout: num_file_reads_for_auto_readahead: 2 2024-08-17T17:18:17.777 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.write_buffer_size: 33554432 2024-08-17T17:18:17.777 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_write_buffer_number: 2 2024-08-17T17:18:17.777 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.compression: NoCompression 2024-08-17T17:18:17.778 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.bottommost_compression: Disabled 2024-08-17T17:18:17.778 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.prefix_extractor: nullptr 2024-08-17T17:18:17.778 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-08-17T17:18:17.778 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.num_levels: 7 2024-08-17T17:18:17.778 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-08-17T17:18:17.778 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-08-17T17:18:17.778 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-08-17T17:18:17.778 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-08-17T17:18:17.778 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-08-17T17:18:17.778 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-08-17T17:18:17.778 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-08-17T17:18:17.778 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-08-17T17:18:17.778 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-08-17T17:18:17.779 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-08-17T17:18:17.779 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-08-17T17:18:17.779 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-08-17T17:18:17.779 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.compression_opts.window_bits: -14 2024-08-17T17:18:17.779 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.compression_opts.level: 32767 2024-08-17T17:18:17.779 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.compression_opts.strategy: 0 2024-08-17T17:18:17.779 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-08-17T17:18:17.779 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-08-17T17:18:17.779 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-08-17T17:18:17.779 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-08-17T17:18:17.779 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.compression_opts.enabled: false 2024-08-17T17:18:17.779 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-08-17T17:18:17.780 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-08-17T17:18:17.780 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-08-17T17:18:17.780 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-08-17T17:18:17.780 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.target_file_size_base: 67108864 2024-08-17T17:18:17.780 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.target_file_size_multiplier: 1 2024-08-17T17:18:17.780 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-08-17T17:18:17.780 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-08-17T17:18:17.780 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-08-17T17:18:17.780 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-08-17T17:18:17.780 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-08-17T17:18:17.780 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-08-17T17:18:17.780 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-08-17T17:18:17.780 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-08-17T17:18:17.781 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-08-17T17:18:17.781 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-08-17T17:18:17.781 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-08-17T17:18:17.781 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-08-17T17:18:17.781 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-08-17T17:18:17.781 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.arena_block_size: 1048576 2024-08-17T17:18:17.781 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-08-17T17:18:17.781 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-08-17T17:18:17.781 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.disable_auto_compactions: 0 2024-08-17T17:18:17.781 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-08-17T17:18:17.781 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-08-17T17:18:17.781 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-08-17T17:18:17.781 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-08-17T17:18:17.782 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-08-17T17:18:17.782 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-08-17T17:18:17.782 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-08-17T17:18:17.782 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-08-17T17:18:17.782 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-08-17T17:18:17.782 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-08-17T17:18:17.782 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-08-17T17:18:17.782 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.inplace_update_support: 0 2024-08-17T17:18:17.782 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.inplace_update_num_locks: 10000 2024-08-17T17:18:17.782 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-08-17T17:18:17.782 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-08-17T17:18:17.782 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.memtable_huge_page_size: 0 2024-08-17T17:18:17.782 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.bloom_locality: 0 2024-08-17T17:18:17.783 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.max_successive_merges: 0 2024-08-17T17:18:17.783 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.optimize_filters_for_hits: 0 2024-08-17T17:18:17.783 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.paranoid_file_checks: 0 2024-08-17T17:18:17.783 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.force_consistency_checks: 1 2024-08-17T17:18:17.783 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.report_bg_io_stats: 0 2024-08-17T17:18:17.783 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.ttl: 2592000 2024-08-17T17:18:17.783 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.periodic_compaction_seconds: 0 2024-08-17T17:18:17.783 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-08-17T17:18:17.783 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-08-17T17:18:17.783 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.enable_blob_files: false 2024-08-17T17:18:17.783 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.min_blob_size: 0 2024-08-17T17:18:17.783 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.blob_file_size: 268435456 2024-08-17T17:18:17.783 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.blob_compression_type: NoCompression 2024-08-17T17:18:17.784 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.enable_blob_garbage_collection: false 2024-08-17T17:18:17.784 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-08-17T17:18:17.784 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-08-17T17:18:17.784 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-08-17T17:18:17.784 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.blob_file_starting_level: 0 2024-08-17T17:18:17.784 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-08-17T17:18:17.784 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-08-17T17:18:17.784 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 32, last_sequence is 19562, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 28 2024-08-17T17:18:17.784 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-08-17T17:18:17.784 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9c1e8a30-33ca-4b9e-a900-a08d3297168b 2024-08-17T17:18:17.784 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723915097421746, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-08-17T17:18:17.784 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-08-17T17:18:17.784 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723915097432520, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 2873412, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 19567, "largest_seqno": 21820, "table_properties": {"data_size": 2862459, "index_size": 7795, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2117, "raw_key_size": 18521, "raw_average_key_size": 22, "raw_value_size": 2842705, "raw_average_value_size": 3445, "num_data_blocks": 338, "num_entries": 825, "num_filter_entries": 825, "num_deletions": 8, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1723915097, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9c1e8a30-33ca-4b9e-a900-a08d3297168b", "db_session_id": "OREJKVJ5PIU9JRQDPMCZ", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-08-17T17:18:17.784 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723915097432657, "job": 1, "event": "recovery_finished"} 2024-08-17T17:18:17.785 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-08-17T17:18:17.785 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-08-17T17:18:17.785 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-08-17T17:18:17.785 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c6abc16e00 2024-08-17T17:18:17.785 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: DB pointer 0x55c6abd14000 2024-08-17T17:18:17.785 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-08-17T17:18:17.785 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-08-17T17:18:17.785 INFO:journalctl@ceph.mon.b.smithi165.stdout: ** DB Stats ** 2024-08-17T17:18:17.785 INFO:journalctl@ceph.mon.b.smithi165.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-17T17:18:17.785 INFO:journalctl@ceph.mon.b.smithi165.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-08-17T17:18:17.785 INFO:journalctl@ceph.mon.b.smithi165.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-17T17:18:17.785 INFO:journalctl@ceph.mon.b.smithi165.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-17T17:18:17.785 INFO:journalctl@ceph.mon.b.smithi165.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-08-17T17:18:17.786 INFO:journalctl@ceph.mon.b.smithi165.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-17T17:18:17.786 INFO:journalctl@ceph.mon.b.smithi165.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-17T17:18:17.786 INFO:journalctl@ceph.mon.b.smithi165.stdout: 2024-08-17T17:18:17.786 INFO:journalctl@ceph.mon.b.smithi165.stdout: ** Compaction Stats [default] ** 2024-08-17T17:18:17.786 INFO:journalctl@ceph.mon.b.smithi165.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-17T17:18:17.786 INFO:journalctl@ceph.mon.b.smithi165.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-08-17T17:18:17.786 INFO:journalctl@ceph.mon.b.smithi165.stdout: L0 1/0 2.74 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 626.2 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-08-17T17:18:17.786 INFO:journalctl@ceph.mon.b.smithi165.stdout: L6 1/0 12.78 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-08-17T17:18:17.786 INFO:journalctl@ceph.mon.b.smithi165.stdout: Sum 2/0 15.52 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 626.2 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-08-17T17:18:17.786 INFO:journalctl@ceph.mon.b.smithi165.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 626.2 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-08-17T17:18:17.786 INFO:journalctl@ceph.mon.b.smithi165.stdout: 2024-08-17T17:18:17.786 INFO:journalctl@ceph.mon.b.smithi165.stdout: ** Compaction Stats [default] ** 2024-08-17T17:18:17.786 INFO:journalctl@ceph.mon.b.smithi165.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-17T17:18:17.787 INFO:journalctl@ceph.mon.b.smithi165.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-17T17:18:17.787 INFO:journalctl@ceph.mon.b.smithi165.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 626.2 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-08-17T17:18:17.787 INFO:journalctl@ceph.mon.b.smithi165.stdout: 2024-08-17T17:18:17.787 INFO:journalctl@ceph.mon.b.smithi165.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-08-17T17:18:17.787 INFO:journalctl@ceph.mon.b.smithi165.stdout: 2024-08-17T17:18:17.787 INFO:journalctl@ceph.mon.b.smithi165.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-17T17:18:17.787 INFO:journalctl@ceph.mon.b.smithi165.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-08-17T17:18:17.787 INFO:journalctl@ceph.mon.b.smithi165.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-08-17T17:18:17.788 INFO:journalctl@ceph.mon.b.smithi165.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-08-17T17:18:17.788 INFO:journalctl@ceph.mon.b.smithi165.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-08-17T17:18:17.788 INFO:journalctl@ceph.mon.b.smithi165.stdout: AddFile(Keys): cumulative 0, interval 0 2024-08-17T17:18:17.788 INFO:journalctl@ceph.mon.b.smithi165.stdout: Cumulative compaction: 0.00 GB write, 183.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:18:17.788 INFO:journalctl@ceph.mon.b.smithi165.stdout: Interval compaction: 0.00 GB write, 183.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:18:17.788 INFO:journalctl@ceph.mon.b.smithi165.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-08-17T17:18:17.788 INFO:journalctl@ceph.mon.b.smithi165.stdout: Block cache BinnedLRUCache@0x55c6abbe7610#7 capacity: 512.00 MB usage: 10.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-08-17T17:18:17.788 INFO:journalctl@ceph.mon.b.smithi165.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.28 KB,0.000435114%) IndexBlock(1,8.08 KB,0.00154078%) Misc(1,0.00 KB,0%) 2024-08-17T17:18:17.788 INFO:journalctl@ceph.mon.b.smithi165.stdout: 2024-08-17T17:18:17.788 INFO:journalctl@ceph.mon.b.smithi165.stdout: ** File Read Latency Histogram By Level [default] ** 2024-08-17T17:18:17.788 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: starting mon.b rank 1 at public addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] at bind addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:18:17.788 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: mon.b@-1(???) e3 preinit fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:18:17.789 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: mon.b@-1(???).mds e9 new map 2024-08-17T17:18:17.789 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: mon.b@-1(???).mds e9 print_map 2024-08-17T17:18:17.789 INFO:journalctl@ceph.mon.b.smithi165.stdout: e9 2024-08-17T17:18:17.789 INFO:journalctl@ceph.mon.b.smithi165.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-08-17T17:18:17.789 INFO:journalctl@ceph.mon.b.smithi165.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-08-17T17:18:17.789 INFO:journalctl@ceph.mon.b.smithi165.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-08-17T17:18:17.789 INFO:journalctl@ceph.mon.b.smithi165.stdout: legacy client fscid: 1 2024-08-17T17:18:17.789 INFO:journalctl@ceph.mon.b.smithi165.stdout: 2024-08-17T17:18:17.789 INFO:journalctl@ceph.mon.b.smithi165.stdout: Filesystem 'foo' (1) 2024-08-17T17:18:17.789 INFO:journalctl@ceph.mon.b.smithi165.stdout: fs_name foo 2024-08-17T17:18:17.789 INFO:journalctl@ceph.mon.b.smithi165.stdout: epoch 8 2024-08-17T17:18:17.790 INFO:journalctl@ceph.mon.b.smithi165.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-08-17T17:18:17.790 INFO:journalctl@ceph.mon.b.smithi165.stdout: created 2024-08-17T17:10:05.962795+0000 2024-08-17T17:18:17.790 INFO:journalctl@ceph.mon.b.smithi165.stdout: modified 2024-08-17T17:10:20.942940+0000 2024-08-17T17:18:17.790 INFO:journalctl@ceph.mon.b.smithi165.stdout: tableserver 0 2024-08-17T17:18:17.790 INFO:journalctl@ceph.mon.b.smithi165.stdout: root 0 2024-08-17T17:18:17.790 INFO:journalctl@ceph.mon.b.smithi165.stdout: session_timeout 60 2024-08-17T17:18:17.790 INFO:journalctl@ceph.mon.b.smithi165.stdout: session_autoclose 300 2024-08-17T17:18:17.790 INFO:journalctl@ceph.mon.b.smithi165.stdout: max_file_size 1099511627776 2024-08-17T17:18:17.790 INFO:journalctl@ceph.mon.b.smithi165.stdout: max_xattr_size 65536 2024-08-17T17:18:17.790 INFO:journalctl@ceph.mon.b.smithi165.stdout: required_client_features {} 2024-08-17T17:18:17.790 INFO:journalctl@ceph.mon.b.smithi165.stdout: last_failure 0 2024-08-17T17:18:17.790 INFO:journalctl@ceph.mon.b.smithi165.stdout: last_failure_osd_epoch 0 2024-08-17T17:18:17.790 INFO:journalctl@ceph.mon.b.smithi165.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-08-17T17:18:17.791 INFO:journalctl@ceph.mon.b.smithi165.stdout: max_mds 1 2024-08-17T17:18:17.791 INFO:journalctl@ceph.mon.b.smithi165.stdout: in 0 2024-08-17T17:18:17.791 INFO:journalctl@ceph.mon.b.smithi165.stdout: up {0=24557} 2024-08-17T17:18:17.791 INFO:journalctl@ceph.mon.b.smithi165.stdout: failed 2024-08-17T17:18:17.791 INFO:journalctl@ceph.mon.b.smithi165.stdout: damaged 2024-08-17T17:18:17.791 INFO:journalctl@ceph.mon.b.smithi165.stdout: stopped 2024-08-17T17:18:17.791 INFO:journalctl@ceph.mon.b.smithi165.stdout: data_pools [4] 2024-08-17T17:18:17.791 INFO:journalctl@ceph.mon.b.smithi165.stdout: metadata_pool 3 2024-08-17T17:18:17.791 INFO:journalctl@ceph.mon.b.smithi165.stdout: inline_data disabled 2024-08-17T17:18:17.791 INFO:journalctl@ceph.mon.b.smithi165.stdout: balancer 2024-08-17T17:18:17.791 INFO:journalctl@ceph.mon.b.smithi165.stdout: bal_rank_mask -1 2024-08-17T17:18:17.791 INFO:journalctl@ceph.mon.b.smithi165.stdout: standby_count_wanted 1 2024-08-17T17:18:17.791 INFO:journalctl@ceph.mon.b.smithi165.stdout: qdb_cluster leader: 0 members: 2024-08-17T17:18:17.791 INFO:journalctl@ceph.mon.b.smithi165.stdout: [mds.foo.smithi136.ltsuqd{0:24557} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.136:6834/764583248,v1:172.21.15.136:6835/764583248] compat {c=[1],r=[1],i=[7ff]}] 2024-08-17T17:18:17.792 INFO:journalctl@ceph.mon.b.smithi165.stdout: 2024-08-17T17:18:17.792 INFO:journalctl@ceph.mon.b.smithi165.stdout: 2024-08-17T17:18:17.792 INFO:journalctl@ceph.mon.b.smithi165.stdout: Standby daemons: 2024-08-17T17:18:17.792 INFO:journalctl@ceph.mon.b.smithi165.stdout: 2024-08-17T17:18:17.792 INFO:journalctl@ceph.mon.b.smithi165.stdout: [mds.foo.smithi165.bmrbhh{-1:24487} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.165:6832/4276964188,v1:172.21.15.165:6833/4276964188] compat {c=[1],r=[1],i=[7ff]}] 2024-08-17T17:18:17.792 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: mon.b@-1(???).osd e463 crush map has features 3314933000854323200, adjusting msgr requires 2024-08-17T17:18:17.792 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: mon.b@-1(???).osd e463 crush map has features 432629239337189376, adjusting msgr requires 2024-08-17T17:18:17.792 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: mon.b@-1(???).osd e463 crush map has features 432629239337189376, adjusting msgr requires 2024-08-17T17:18:17.792 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: mon.b@-1(???).osd e463 crush map has features 432629239337189376, adjusting msgr requires 2024-08-17T17:18:17.792 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 ceph-mon[105517]: mon.b@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-08-17T17:18:17.792 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 bash[105479]: 1fd357308b8d9c5756303eb35cb2497df3ef60b5fe965f83d94c0fbfeec85b53 2024-08-17T17:18:17.792 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:17 smithi165 systemd[1]: Started Ceph mon.b for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:18:17.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:18:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:18:17] "GET /metrics HTTP/1.1" 200 36271 "" "Prometheus/2.51.0" 2024-08-17T17:18:22.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:18:22.444+0000 7f0cc2ea7640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-08-17T17:18:22.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: mgrmap e40: y(active, since 2m), standbys: x 2024-08-17T17:18:22.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: overall HEALTH_OK 2024-08-17T17:18:22.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: mon.a calling monitor election 2024-08-17T17:18:22.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: mon.b calling monitor election 2024-08-17T17:18:22.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-17T17:18:22.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-17T17:18:22.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-17T17:18:22.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: pgmap v187: 105 pgs: 105 active+clean; 583 KiB data, 442 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 9.5 KiB/s wr, 131 op/s 2024-08-17T17:18:22.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: pgmap v188: 105 pgs: 105 active+clean; 583 KiB data, 442 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 7.6 KiB/s wr, 104 op/s 2024-08-17T17:18:22.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:18:22.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: pgmap v189: 105 pgs: 105 active+clean; 583 KiB data, 442 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 5.7 KiB/s wr, 79 op/s 2024-08-17T17:18:22.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-08-17T17:18:22.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: monmap epoch 4 2024-08-17T17:18:22.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:18:22.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: last_changed 2024-08-17T17:18:17.598790+0000 2024-08-17T17:18:22.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: created 2024-08-17T17:03:14.749093+0000 2024-08-17T17:18:22.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: min_mon_release 19 (squid) 2024-08-17T17:18:22.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: election_strategy: 1 2024-08-17T17:18:22.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: 0: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.a 2024-08-17T17:18:22.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: 1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-08-17T17:18:22.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: 2: [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] mon.c 2024-08-17T17:18:22.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 1 up:standby 2024-08-17T17:18:22.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: osdmap e463: 8 total, 8 up, 8 in 2024-08-17T17:18:22.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: mgrmap e40: y(active, since 2m), standbys: x 2024-08-17T17:18:22.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-08-17T17:18:22.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-08-17T17:18:22.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-08-17T17:18:22.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: mon.c (rank 2) addr [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] is down (out of quorum) 2024-08-17T17:18:22.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:22.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:22.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:22.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:18:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: mgrmap e40: y(active, since 2m), standbys: x 2024-08-17T17:18:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: overall HEALTH_OK 2024-08-17T17:18:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: mon.a calling monitor election 2024-08-17T17:18:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: mon.b calling monitor election 2024-08-17T17:18:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-17T17:18:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-17T17:18:23.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-17T17:18:23.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: pgmap v187: 105 pgs: 105 active+clean; 583 KiB data, 442 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 9.5 KiB/s wr, 131 op/s 2024-08-17T17:18:23.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: pgmap v188: 105 pgs: 105 active+clean; 583 KiB data, 442 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 7.6 KiB/s wr, 104 op/s 2024-08-17T17:18:23.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:18:23.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: pgmap v189: 105 pgs: 105 active+clean; 583 KiB data, 442 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 5.7 KiB/s wr, 79 op/s 2024-08-17T17:18:23.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-08-17T17:18:23.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: monmap epoch 4 2024-08-17T17:18:23.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:18:23.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: last_changed 2024-08-17T17:18:17.598790+0000 2024-08-17T17:18:23.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: created 2024-08-17T17:03:14.749093+0000 2024-08-17T17:18:23.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: min_mon_release 19 (squid) 2024-08-17T17:18:23.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: election_strategy: 1 2024-08-17T17:18:23.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: 0: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.a 2024-08-17T17:18:23.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: 1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-08-17T17:18:23.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: 2: [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] mon.c 2024-08-17T17:18:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 1 up:standby 2024-08-17T17:18:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: osdmap e463: 8 total, 8 up, 8 in 2024-08-17T17:18:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: mgrmap e40: y(active, since 2m), standbys: x 2024-08-17T17:18:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-08-17T17:18:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-08-17T17:18:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-08-17T17:18:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: mon.c (rank 2) addr [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] is down (out of quorum) 2024-08-17T17:18:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:18:23.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[109861]: mon.c calling monitor election 2024-08-17T17:18:23.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[109861]: mon.b calling monitor election 2024-08-17T17:18:23.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[109861]: mon.a calling monitor election 2024-08-17T17:18:23.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[109861]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-08-17T17:18:23.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[109861]: monmap epoch 4 2024-08-17T17:18:23.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[109861]: fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:18:23.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[109861]: last_changed 2024-08-17T17:18:17.598790+0000 2024-08-17T17:18:23.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[109861]: created 2024-08-17T17:03:14.749093+0000 2024-08-17T17:18:23.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[109861]: min_mon_release 19 (squid) 2024-08-17T17:18:23.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[109861]: election_strategy: 1 2024-08-17T17:18:23.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[109861]: 0: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.a 2024-08-17T17:18:23.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[109861]: 1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-08-17T17:18:23.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[109861]: 2: [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] mon.c 2024-08-17T17:18:23.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[109861]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 1 up:standby 2024-08-17T17:18:23.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[109861]: osdmap e463: 8 total, 8 up, 8 in 2024-08-17T17:18:23.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[109861]: mgrmap e40: y(active, since 2m), standbys: x 2024-08-17T17:18:23.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[109861]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-08-17T17:18:23.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T17:18:23.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[109861]: overall HEALTH_OK 2024-08-17T17:18:23.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: mgrmap e40: y(active, since 2m), standbys: x 2024-08-17T17:18:23.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: overall HEALTH_OK 2024-08-17T17:18:23.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: mon.a calling monitor election 2024-08-17T17:18:23.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: mon.b calling monitor election 2024-08-17T17:18:23.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-17T17:18:23.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-17T17:18:23.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-17T17:18:23.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: pgmap v187: 105 pgs: 105 active+clean; 583 KiB data, 442 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 9.5 KiB/s wr, 131 op/s 2024-08-17T17:18:23.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: pgmap v188: 105 pgs: 105 active+clean; 583 KiB data, 442 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 7.6 KiB/s wr, 104 op/s 2024-08-17T17:18:23.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:18:23.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: pgmap v189: 105 pgs: 105 active+clean; 583 KiB data, 442 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 5.7 KiB/s wr, 79 op/s 2024-08-17T17:18:23.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-08-17T17:18:23.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: monmap epoch 4 2024-08-17T17:18:23.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:18:23.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: last_changed 2024-08-17T17:18:17.598790+0000 2024-08-17T17:18:23.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: created 2024-08-17T17:03:14.749093+0000 2024-08-17T17:18:23.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: min_mon_release 19 (squid) 2024-08-17T17:18:23.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: election_strategy: 1 2024-08-17T17:18:23.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: 0: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.a 2024-08-17T17:18:23.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: 1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-08-17T17:18:23.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: 2: [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] mon.c 2024-08-17T17:18:23.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 1 up:standby 2024-08-17T17:18:23.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: osdmap e463: 8 total, 8 up, 8 in 2024-08-17T17:18:23.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: mgrmap e40: y(active, since 2m), standbys: x 2024-08-17T17:18:23.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-08-17T17:18:23.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-08-17T17:18:23.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-08-17T17:18:23.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: mon.c (rank 2) addr [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] is down (out of quorum) 2024-08-17T17:18:23.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:23.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:23.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:23.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:18:23.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: mon.c calling monitor election 2024-08-17T17:18:23.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: mon.b calling monitor election 2024-08-17T17:18:23.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: mon.a calling monitor election 2024-08-17T17:18:23.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-08-17T17:18:23.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: monmap epoch 4 2024-08-17T17:18:23.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:18:23.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: last_changed 2024-08-17T17:18:17.598790+0000 2024-08-17T17:18:23.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: created 2024-08-17T17:03:14.749093+0000 2024-08-17T17:18:23.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: min_mon_release 19 (squid) 2024-08-17T17:18:23.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: election_strategy: 1 2024-08-17T17:18:23.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: 0: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.a 2024-08-17T17:18:23.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: 1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-08-17T17:18:23.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: 2: [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] mon.c 2024-08-17T17:18:23.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 1 up:standby 2024-08-17T17:18:23.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: osdmap e463: 8 total, 8 up, 8 in 2024-08-17T17:18:23.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: mgrmap e40: y(active, since 2m), standbys: x 2024-08-17T17:18:23.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-08-17T17:18:23.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T17:18:23.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:23 smithi136 ceph-mon[115078]: overall HEALTH_OK 2024-08-17T17:18:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:23 smithi165 ceph-mon[105517]: mon.c calling monitor election 2024-08-17T17:18:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:23 smithi165 ceph-mon[105517]: mon.b calling monitor election 2024-08-17T17:18:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:23 smithi165 ceph-mon[105517]: mon.a calling monitor election 2024-08-17T17:18:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:23 smithi165 ceph-mon[105517]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-08-17T17:18:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:23 smithi165 ceph-mon[105517]: monmap epoch 4 2024-08-17T17:18:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:23 smithi165 ceph-mon[105517]: fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:18:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:23 smithi165 ceph-mon[105517]: last_changed 2024-08-17T17:18:17.598790+0000 2024-08-17T17:18:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:23 smithi165 ceph-mon[105517]: created 2024-08-17T17:03:14.749093+0000 2024-08-17T17:18:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:23 smithi165 ceph-mon[105517]: min_mon_release 19 (squid) 2024-08-17T17:18:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:23 smithi165 ceph-mon[105517]: election_strategy: 1 2024-08-17T17:18:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:23 smithi165 ceph-mon[105517]: 0: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.a 2024-08-17T17:18:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:23 smithi165 ceph-mon[105517]: 1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-08-17T17:18:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:23 smithi165 ceph-mon[105517]: 2: [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] mon.c 2024-08-17T17:18:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:23 smithi165 ceph-mon[105517]: fsmap foo:1 {0=foo.smithi136.ltsuqd=up:active} 1 up:standby 2024-08-17T17:18:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:23 smithi165 ceph-mon[105517]: osdmap e463: 8 total, 8 up, 8 in 2024-08-17T17:18:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:23 smithi165 ceph-mon[105517]: mgrmap e40: y(active, since 2m), standbys: x 2024-08-17T17:18:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:23 smithi165 ceph-mon[105517]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-08-17T17:18:24.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:23 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T17:18:24.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:23 smithi165 ceph-mon[105517]: overall HEALTH_OK 2024-08-17T17:18:24.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:24 smithi136 ceph-mon[109861]: osdmap e464: 8 total, 8 up, 8 in 2024-08-17T17:18:24.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:24 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/e020b3d5-c0cc-4ee1-a36c-5268ca5da971"} : dispatch 2024-08-17T17:18:24.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:24 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/e020b3d5-c0cc-4ee1-a36c-5268ca5da971"} : dispatch 2024-08-17T17:18:24.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:24 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/e020b3d5-c0cc-4ee1-a36c-5268ca5da971"}]': finished 2024-08-17T17:18:24.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:24 smithi136 ceph-mon[109861]: osdmap e465: 8 total, 8 up, 8 in 2024-08-17T17:18:24.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:24 smithi136 ceph-mon[109861]: pgmap v192: 105 pgs: 105 active+clean; 583 KiB data, 442 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 5.7 KiB/s wr, 79 op/s 2024-08-17T17:18:24.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:24 smithi136 ceph-mon[115078]: osdmap e464: 8 total, 8 up, 8 in 2024-08-17T17:18:24.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:24 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/e020b3d5-c0cc-4ee1-a36c-5268ca5da971"} : dispatch 2024-08-17T17:18:24.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:24 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/e020b3d5-c0cc-4ee1-a36c-5268ca5da971"} : dispatch 2024-08-17T17:18:24.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:24 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/e020b3d5-c0cc-4ee1-a36c-5268ca5da971"}]': finished 2024-08-17T17:18:24.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:24 smithi136 ceph-mon[115078]: osdmap e465: 8 total, 8 up, 8 in 2024-08-17T17:18:24.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:24 smithi136 ceph-mon[115078]: pgmap v192: 105 pgs: 105 active+clean; 583 KiB data, 442 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 5.7 KiB/s wr, 79 op/s 2024-08-17T17:18:24.948 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:24 smithi165 ceph-mon[105517]: osdmap e464: 8 total, 8 up, 8 in 2024-08-17T17:18:24.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:24 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/e020b3d5-c0cc-4ee1-a36c-5268ca5da971"} : dispatch 2024-08-17T17:18:24.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:24 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/e020b3d5-c0cc-4ee1-a36c-5268ca5da971"} : dispatch 2024-08-17T17:18:24.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:24 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/e020b3d5-c0cc-4ee1-a36c-5268ca5da971"}]': finished 2024-08-17T17:18:24.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:24 smithi165 ceph-mon[105517]: osdmap e465: 8 total, 8 up, 8 in 2024-08-17T17:18:24.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:24 smithi165 ceph-mon[105517]: pgmap v192: 105 pgs: 105 active+clean; 583 KiB data, 442 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 5.7 KiB/s wr, 79 op/s 2024-08-17T17:18:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:25 smithi165 ceph-mon[105517]: osdmap e466: 8 total, 8 up, 8 in 2024-08-17T17:18:26.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:25 smithi136 ceph-mon[109861]: osdmap e466: 8 total, 8 up, 8 in 2024-08-17T17:18:26.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:25 smithi136 ceph-mon[115078]: osdmap e466: 8 total, 8 up, 8 in 2024-08-17T17:18:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:26 smithi165 ceph-mon[105517]: osdmap e467: 8 total, 8 up, 8 in 2024-08-17T17:18:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:26 smithi165 ceph-mon[105517]: pgmap v195: 105 pgs: 105 active+clean; 583 KiB data, 442 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:18:27.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:26 smithi136 ceph-mon[109861]: osdmap e467: 8 total, 8 up, 8 in 2024-08-17T17:18:27.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:26 smithi136 ceph-mon[109861]: pgmap v195: 105 pgs: 105 active+clean; 583 KiB data, 442 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:18:27.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:26 smithi136 ceph-mon[115078]: osdmap e467: 8 total, 8 up, 8 in 2024-08-17T17:18:27.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:26 smithi136 ceph-mon[115078]: pgmap v195: 105 pgs: 105 active+clean; 583 KiB data, 442 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:18:27.808 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:18:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:18:27] "GET /metrics HTTP/1.1" 200 36284 "" "Prometheus/2.51.0" 2024-08-17T17:18:27.880 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.SnapshotUnlinkPeer (15859 ms) 2024-08-17T17:18:27.880 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.SnapshotImageState 2024-08-17T17:18:28.132 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:27 smithi165 ceph-mon[105517]: osdmap e468: 8 total, 8 up, 8 in 2024-08-17T17:18:28.132 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:27 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:28.132 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:27 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:28.133 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:27 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:28.133 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:27 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:28.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:27 smithi136 ceph-mon[109861]: osdmap e468: 8 total, 8 up, 8 in 2024-08-17T17:18:28.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:27 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:28.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:27 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:28.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:27 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:28.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:27 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:28.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:27 smithi136 ceph-mon[115078]: osdmap e468: 8 total, 8 up, 8 in 2024-08-17T17:18:28.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:27 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:28.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:27 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:28.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:27 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:28.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:27 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:29.073 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:28 smithi165 ceph-mon[105517]: osdmap e469: 8 total, 8 up, 8 in 2024-08-17T17:18:29.073 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:28 smithi165 ceph-mon[105517]: pgmap v198: 105 pgs: 105 active+clean; 583 KiB data, 442 MiB used, 715 GiB / 715 GiB avail; 161 KiB/s rd, 9.0 KiB/s wr, 215 op/s 2024-08-17T17:18:29.074 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:28 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/a2b962b0-75dd-4f4b-a771-4c06a6a041cd"} : dispatch 2024-08-17T17:18:29.074 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:28 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/a2b962b0-75dd-4f4b-a771-4c06a6a041cd"} : dispatch 2024-08-17T17:18:29.074 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:28 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/a2b962b0-75dd-4f4b-a771-4c06a6a041cd"}]': finished 2024-08-17T17:18:29.074 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:28 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/35532f13-fa39-4d38-9c58-987d5de95819"} : dispatch 2024-08-17T17:18:29.074 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:28 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/35532f13-fa39-4d38-9c58-987d5de95819"} : dispatch 2024-08-17T17:18:29.074 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:28 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/35532f13-fa39-4d38-9c58-987d5de95819"}]': finished 2024-08-17T17:18:29.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:28 smithi136 ceph-mon[109861]: osdmap e469: 8 total, 8 up, 8 in 2024-08-17T17:18:29.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:28 smithi136 ceph-mon[109861]: pgmap v198: 105 pgs: 105 active+clean; 583 KiB data, 442 MiB used, 715 GiB / 715 GiB avail; 161 KiB/s rd, 9.0 KiB/s wr, 215 op/s 2024-08-17T17:18:29.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:28 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/a2b962b0-75dd-4f4b-a771-4c06a6a041cd"} : dispatch 2024-08-17T17:18:29.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:28 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/a2b962b0-75dd-4f4b-a771-4c06a6a041cd"} : dispatch 2024-08-17T17:18:29.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:28 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/a2b962b0-75dd-4f4b-a771-4c06a6a041cd"}]': finished 2024-08-17T17:18:29.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:28 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/35532f13-fa39-4d38-9c58-987d5de95819"} : dispatch 2024-08-17T17:18:29.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:28 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/35532f13-fa39-4d38-9c58-987d5de95819"} : dispatch 2024-08-17T17:18:29.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:28 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/35532f13-fa39-4d38-9c58-987d5de95819"}]': finished 2024-08-17T17:18:29.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:28 smithi136 ceph-mon[115078]: osdmap e469: 8 total, 8 up, 8 in 2024-08-17T17:18:29.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:28 smithi136 ceph-mon[115078]: pgmap v198: 105 pgs: 105 active+clean; 583 KiB data, 442 MiB used, 715 GiB / 715 GiB avail; 161 KiB/s rd, 9.0 KiB/s wr, 215 op/s 2024-08-17T17:18:29.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:28 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/a2b962b0-75dd-4f4b-a771-4c06a6a041cd"} : dispatch 2024-08-17T17:18:29.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:28 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/a2b962b0-75dd-4f4b-a771-4c06a6a041cd"} : dispatch 2024-08-17T17:18:29.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:28 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/a2b962b0-75dd-4f4b-a771-4c06a6a041cd"}]': finished 2024-08-17T17:18:29.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:28 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/35532f13-fa39-4d38-9c58-987d5de95819"} : dispatch 2024-08-17T17:18:29.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:28 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/35532f13-fa39-4d38-9c58-987d5de95819"} : dispatch 2024-08-17T17:18:29.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:28 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/35532f13-fa39-4d38-9c58-987d5de95819"}]': finished 2024-08-17T17:18:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:29 smithi136 ceph-mon[109861]: osdmap e470: 8 total, 8 up, 8 in 2024-08-17T17:18:30.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:29 smithi136 ceph-mon[115078]: osdmap e470: 8 total, 8 up, 8 in 2024-08-17T17:18:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:29 smithi165 ceph-mon[105517]: osdmap e470: 8 total, 8 up, 8 in 2024-08-17T17:18:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:30 smithi165 ceph-mon[105517]: pgmap v200: 105 pgs: 105 active+clean; 583 KiB data, 442 MiB used, 715 GiB / 715 GiB avail; 159 KiB/s rd, 8.9 KiB/s wr, 212 op/s 2024-08-17T17:18:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:30 smithi165 ceph-mon[105517]: osdmap e471: 8 total, 8 up, 8 in 2024-08-17T17:18:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:30 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:30 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:30 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:30 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:18:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:30 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:30 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:18:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:30 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:18:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:30 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:31.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:30 smithi136 ceph-mon[109861]: pgmap v200: 105 pgs: 105 active+clean; 583 KiB data, 442 MiB used, 715 GiB / 715 GiB avail; 159 KiB/s rd, 8.9 KiB/s wr, 212 op/s 2024-08-17T17:18:31.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:30 smithi136 ceph-mon[109861]: osdmap e471: 8 total, 8 up, 8 in 2024-08-17T17:18:31.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:31.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:31.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:31.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:18:31.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:31.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:18:31.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:18:31.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:31.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:30 smithi136 ceph-mon[115078]: pgmap v200: 105 pgs: 105 active+clean; 583 KiB data, 442 MiB used, 715 GiB / 715 GiB avail; 159 KiB/s rd, 8.9 KiB/s wr, 212 op/s 2024-08-17T17:18:31.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:30 smithi136 ceph-mon[115078]: osdmap e471: 8 total, 8 up, 8 in 2024-08-17T17:18:31.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:31.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:31.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:31.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:18:31.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:31.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:18:31.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:18:31.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:31.944 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.SnapshotImageState (4064 ms) 2024-08-17T17:18:31.944 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.SnapshotPromoteDemote 2024-08-17T17:18:32.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:31 smithi136 ceph-mon[109861]: Reconfiguring mon.a (monmap changed)... 2024-08-17T17:18:32.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:31 smithi136 ceph-mon[109861]: Reconfiguring daemon mon.a on smithi136 2024-08-17T17:18:32.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:31 smithi136 ceph-mon[109861]: osdmap e472: 8 total, 8 up, 8 in 2024-08-17T17:18:32.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:31 smithi136 ceph-mon[115078]: Reconfiguring mon.a (monmap changed)... 2024-08-17T17:18:32.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:31 smithi136 ceph-mon[115078]: Reconfiguring daemon mon.a on smithi136 2024-08-17T17:18:32.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:31 smithi136 ceph-mon[115078]: osdmap e472: 8 total, 8 up, 8 in 2024-08-17T17:18:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:31 smithi165 ceph-mon[105517]: Reconfiguring mon.a (monmap changed)... 2024-08-17T17:18:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:31 smithi165 ceph-mon[105517]: Reconfiguring daemon mon.a on smithi136 2024-08-17T17:18:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:31 smithi165 ceph-mon[105517]: osdmap e472: 8 total, 8 up, 8 in 2024-08-17T17:18:32.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:32 smithi136 ceph-mon[109861]: pgmap v203: 105 pgs: 105 active+clean; 583 KiB data, 443 MiB used, 715 GiB / 715 GiB avail; 150 KiB/s rd, 23 KiB/s wr, 216 op/s 2024-08-17T17:18:32.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:32 smithi136 ceph-mon[109861]: osdmap e473: 8 total, 8 up, 8 in 2024-08-17T17:18:32.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:32.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:32.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-17T17:18:32.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-08-17T17:18:32.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:32.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:32 smithi136 ceph-mon[115078]: pgmap v203: 105 pgs: 105 active+clean; 583 KiB data, 443 MiB used, 715 GiB / 715 GiB avail; 150 KiB/s rd, 23 KiB/s wr, 216 op/s 2024-08-17T17:18:32.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:32 smithi136 ceph-mon[115078]: osdmap e473: 8 total, 8 up, 8 in 2024-08-17T17:18:32.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:32.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:32.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-17T17:18:32.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-08-17T17:18:32.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:32 smithi165 ceph-mon[105517]: pgmap v203: 105 pgs: 105 active+clean; 583 KiB data, 443 MiB used, 715 GiB / 715 GiB avail; 150 KiB/s rd, 23 KiB/s wr, 216 op/s 2024-08-17T17:18:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:32 smithi165 ceph-mon[105517]: osdmap e473: 8 total, 8 up, 8 in 2024-08-17T17:18:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-17T17:18:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-08-17T17:18:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:34.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:33 smithi136 ceph-mon[109861]: Reconfiguring mgr.y (monmap changed)... 2024-08-17T17:18:34.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:33 smithi136 ceph-mon[109861]: Reconfiguring daemon mgr.y on smithi136 2024-08-17T17:18:34.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:33 smithi136 ceph-mon[109861]: osdmap e474: 8 total, 8 up, 8 in 2024-08-17T17:18:34.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:33 smithi136 ceph-mon[109861]: osdmap e475: 8 total, 8 up, 8 in 2024-08-17T17:18:34.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:33 smithi136 ceph-mon[115078]: Reconfiguring mgr.y (monmap changed)... 2024-08-17T17:18:34.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:33 smithi136 ceph-mon[115078]: Reconfiguring daemon mgr.y on smithi136 2024-08-17T17:18:34.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:33 smithi136 ceph-mon[115078]: osdmap e474: 8 total, 8 up, 8 in 2024-08-17T17:18:34.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:33 smithi136 ceph-mon[115078]: osdmap e475: 8 total, 8 up, 8 in 2024-08-17T17:18:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:33 smithi165 ceph-mon[105517]: Reconfiguring mgr.y (monmap changed)... 2024-08-17T17:18:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:33 smithi165 ceph-mon[105517]: Reconfiguring daemon mgr.y on smithi136 2024-08-17T17:18:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:33 smithi165 ceph-mon[105517]: osdmap e474: 8 total, 8 up, 8 in 2024-08-17T17:18:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:33 smithi165 ceph-mon[105517]: osdmap e475: 8 total, 8 up, 8 in 2024-08-17T17:18:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:34 smithi165 ceph-mon[105517]: pgmap v207: 105 pgs: 105 active+clean; 583 KiB data, 443 MiB used, 715 GiB / 715 GiB avail; 153 KiB/s rd, 24 KiB/s wr, 221 op/s 2024-08-17T17:18:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:34 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:34 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:34 smithi165 ceph-mon[105517]: Reconfiguring mon.c (monmap changed)... 2024-08-17T17:18:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:34 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:18:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:34 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:18:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:34 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:34 smithi165 ceph-mon[105517]: Reconfiguring daemon mon.c on smithi136 2024-08-17T17:18:35.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:34 smithi165 ceph-mon[105517]: osdmap e476: 8 total, 8 up, 8 in 2024-08-17T17:18:35.370 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:34 smithi136 ceph-mon[109861]: pgmap v207: 105 pgs: 105 active+clean; 583 KiB data, 443 MiB used, 715 GiB / 715 GiB avail; 153 KiB/s rd, 24 KiB/s wr, 221 op/s 2024-08-17T17:18:35.370 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:35.370 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:35.371 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:34 smithi136 ceph-mon[109861]: Reconfiguring mon.c (monmap changed)... 2024-08-17T17:18:35.371 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:18:35.371 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:18:35.371 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:35.371 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:34 smithi136 ceph-mon[109861]: Reconfiguring daemon mon.c on smithi136 2024-08-17T17:18:35.371 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:34 smithi136 ceph-mon[109861]: osdmap e476: 8 total, 8 up, 8 in 2024-08-17T17:18:35.371 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:34 smithi136 ceph-mon[115078]: pgmap v207: 105 pgs: 105 active+clean; 583 KiB data, 443 MiB used, 715 GiB / 715 GiB avail; 153 KiB/s rd, 24 KiB/s wr, 221 op/s 2024-08-17T17:18:35.371 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:35.371 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:35.371 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:34 smithi136 ceph-mon[115078]: Reconfiguring mon.c (monmap changed)... 2024-08-17T17:18:35.371 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:18:35.371 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:18:35.372 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:35.372 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:34 smithi136 ceph-mon[115078]: Reconfiguring daemon mon.c on smithi136 2024-08-17T17:18:35.372 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:34 smithi136 ceph-mon[115078]: osdmap e476: 8 total, 8 up, 8 in 2024-08-17T17:18:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:36.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:36 smithi136 ceph-mon[109861]: Reconfiguring osd.0 (monmap changed)... 2024-08-17T17:18:36.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-08-17T17:18:36.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:36.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:36 smithi136 ceph-mon[109861]: Reconfiguring daemon osd.0 on smithi136 2024-08-17T17:18:36.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:36 smithi136 ceph-mon[109861]: osdmap e477: 8 total, 8 up, 8 in 2024-08-17T17:18:36.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:36 smithi136 ceph-mon[109861]: pgmap v210: 105 pgs: 105 active+clean; 583 KiB data, 443 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:18:36.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:18:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:36 smithi136 ceph-mon[115078]: Reconfiguring osd.0 (monmap changed)... 2024-08-17T17:18:36.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-08-17T17:18:36.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:36.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:36 smithi136 ceph-mon[115078]: Reconfiguring daemon osd.0 on smithi136 2024-08-17T17:18:36.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:36 smithi136 ceph-mon[115078]: osdmap e477: 8 total, 8 up, 8 in 2024-08-17T17:18:36.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:36 smithi136 ceph-mon[115078]: pgmap v210: 105 pgs: 105 active+clean; 583 KiB data, 443 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:18:36.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:18:36.736 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:36.736 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:36.736 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:36 smithi165 ceph-mon[105517]: Reconfiguring osd.0 (monmap changed)... 2024-08-17T17:18:36.736 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-08-17T17:18:36.736 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:36.736 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:36 smithi165 ceph-mon[105517]: Reconfiguring daemon osd.0 on smithi136 2024-08-17T17:18:36.736 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:36 smithi165 ceph-mon[105517]: osdmap e477: 8 total, 8 up, 8 in 2024-08-17T17:18:36.736 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:36 smithi165 ceph-mon[105517]: pgmap v210: 105 pgs: 105 active+clean; 583 KiB data, 443 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:18:36.736 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:18:37.539 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:18:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:18:37] "GET /metrics HTTP/1.1" 200 36283 "" "Prometheus/2.51.0" 2024-08-17T17:18:38.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:37 smithi136 ceph-mon[109861]: osdmap e478: 8 total, 8 up, 8 in 2024-08-17T17:18:38.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:38.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:38.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:37 smithi136 ceph-mon[109861]: Reconfiguring osd.1 (monmap changed)... 2024-08-17T17:18:38.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-08-17T17:18:38.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:38.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:37 smithi136 ceph-mon[109861]: Reconfiguring daemon osd.1 on smithi136 2024-08-17T17:18:38.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:37 smithi136 ceph-mon[115078]: osdmap e478: 8 total, 8 up, 8 in 2024-08-17T17:18:38.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:38.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:38.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:37 smithi136 ceph-mon[115078]: Reconfiguring osd.1 (monmap changed)... 2024-08-17T17:18:38.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-08-17T17:18:38.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:38.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:37 smithi136 ceph-mon[115078]: Reconfiguring daemon osd.1 on smithi136 2024-08-17T17:18:38.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:37 smithi165 ceph-mon[105517]: osdmap e478: 8 total, 8 up, 8 in 2024-08-17T17:18:38.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:38.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:38.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:37 smithi165 ceph-mon[105517]: Reconfiguring osd.1 (monmap changed)... 2024-08-17T17:18:38.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-08-17T17:18:38.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:38.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:37 smithi165 ceph-mon[105517]: Reconfiguring daemon osd.1 on smithi136 2024-08-17T17:18:39.067 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:38 smithi136 ceph-mon[109861]: osdmap e479: 8 total, 8 up, 8 in 2024-08-17T17:18:39.067 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:38 smithi136 ceph-mon[109861]: pgmap v213: 105 pgs: 105 active+clean; 583 KiB data, 451 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s rd, 10 KiB/s wr, 171 op/s 2024-08-17T17:18:39.067 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:39.067 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:39.067 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-08-17T17:18:39.067 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:39.068 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:38 smithi136 ceph-mon[115078]: osdmap e479: 8 total, 8 up, 8 in 2024-08-17T17:18:39.068 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:38 smithi136 ceph-mon[115078]: pgmap v213: 105 pgs: 105 active+clean; 583 KiB data, 451 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s rd, 10 KiB/s wr, 171 op/s 2024-08-17T17:18:39.068 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:39.068 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:39.068 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-08-17T17:18:39.068 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:38 smithi165 ceph-mon[105517]: osdmap e479: 8 total, 8 up, 8 in 2024-08-17T17:18:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:38 smithi165 ceph-mon[105517]: pgmap v213: 105 pgs: 105 active+clean; 583 KiB data, 451 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s rd, 10 KiB/s wr, 171 op/s 2024-08-17T17:18:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:38 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:38 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:38 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-08-17T17:18:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:38 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:39.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:39 smithi136 ceph-mon[109861]: Reconfiguring osd.2 (monmap changed)... 2024-08-17T17:18:39.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:39 smithi136 ceph-mon[109861]: Reconfiguring daemon osd.2 on smithi136 2024-08-17T17:18:39.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:39 smithi136 ceph-mon[109861]: osdmap e480: 8 total, 8 up, 8 in 2024-08-17T17:18:39.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:39 smithi136 ceph-mon[115078]: Reconfiguring osd.2 (monmap changed)... 2024-08-17T17:18:39.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:39 smithi136 ceph-mon[115078]: Reconfiguring daemon osd.2 on smithi136 2024-08-17T17:18:39.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:39 smithi136 ceph-mon[115078]: osdmap e480: 8 total, 8 up, 8 in 2024-08-17T17:18:40.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:39 smithi165 ceph-mon[105517]: Reconfiguring osd.2 (monmap changed)... 2024-08-17T17:18:40.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:39 smithi165 ceph-mon[105517]: Reconfiguring daemon osd.2 on smithi136 2024-08-17T17:18:40.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:39 smithi165 ceph-mon[105517]: osdmap e480: 8 total, 8 up, 8 in 2024-08-17T17:18:41.167 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:40 smithi136 ceph-mon[109861]: osdmap e481: 8 total, 8 up, 8 in 2024-08-17T17:18:41.168 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:40 smithi136 ceph-mon[109861]: pgmap v216: 105 pgs: 105 active+clean; 583 KiB data, 451 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s rd, 10 KiB/s wr, 171 op/s 2024-08-17T17:18:41.168 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:40 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:41.168 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:40 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:41.168 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:40 smithi136 ceph-mon[109861]: Reconfiguring osd.3 (monmap changed)... 2024-08-17T17:18:41.168 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:40 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-08-17T17:18:41.168 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:40 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:41.168 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:40 smithi136 ceph-mon[109861]: Reconfiguring daemon osd.3 on smithi136 2024-08-17T17:18:41.168 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:40 smithi136 ceph-mon[115078]: osdmap e481: 8 total, 8 up, 8 in 2024-08-17T17:18:41.169 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:40 smithi136 ceph-mon[115078]: pgmap v216: 105 pgs: 105 active+clean; 583 KiB data, 451 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s rd, 10 KiB/s wr, 171 op/s 2024-08-17T17:18:41.169 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:40 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:41.169 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:40 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:41.169 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:40 smithi136 ceph-mon[115078]: Reconfiguring osd.3 (monmap changed)... 2024-08-17T17:18:41.169 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:40 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-08-17T17:18:41.169 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:40 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:41.169 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:40 smithi136 ceph-mon[115078]: Reconfiguring daemon osd.3 on smithi136 2024-08-17T17:18:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:40 smithi165 ceph-mon[105517]: osdmap e481: 8 total, 8 up, 8 in 2024-08-17T17:18:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:40 smithi165 ceph-mon[105517]: pgmap v216: 105 pgs: 105 active+clean; 583 KiB data, 451 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s rd, 10 KiB/s wr, 171 op/s 2024-08-17T17:18:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:40 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:40 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:40 smithi165 ceph-mon[105517]: Reconfiguring osd.3 (monmap changed)... 2024-08-17T17:18:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:40 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-08-17T17:18:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:40 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:40 smithi165 ceph-mon[105517]: Reconfiguring daemon osd.3 on smithi136 2024-08-17T17:18:41.890 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.SnapshotPromoteDemote (9946 ms) 2024-08-17T17:18:41.890 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestMirroring.AioSnapshotCreate 2024-08-17T17:18:42.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:41 smithi136 ceph-mon[109861]: osdmap e482: 8 total, 8 up, 8 in 2024-08-17T17:18:42.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:41 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:42.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:41 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:42.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:41 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi136.ltsuqd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-17T17:18:42.081 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:41 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:42.082 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:41 smithi136 ceph-mon[115078]: osdmap e482: 8 total, 8 up, 8 in 2024-08-17T17:18:42.082 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:41 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:42.082 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:41 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:42.082 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:41 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi136.ltsuqd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-17T17:18:42.082 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:41 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:41 smithi165 ceph-mon[105517]: osdmap e482: 8 total, 8 up, 8 in 2024-08-17T17:18:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:41 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:41 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:41 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi136.ltsuqd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-17T17:18:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:41 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:42.993 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:42 smithi136 ceph-mon[109861]: Reconfiguring mds.foo.smithi136.ltsuqd (monmap changed)... 2024-08-17T17:18:42.993 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:42 smithi136 ceph-mon[109861]: Reconfiguring daemon mds.foo.smithi136.ltsuqd on smithi136 2024-08-17T17:18:42.993 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:42 smithi136 ceph-mon[109861]: osdmap e483: 8 total, 8 up, 8 in 2024-08-17T17:18:42.994 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:42 smithi136 ceph-mon[109861]: pgmap v219: 105 pgs: 105 active+clean; 583 KiB data, 456 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 5.2 KiB/s wr, 47 op/s 2024-08-17T17:18:42.994 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:42 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/ca39296d-625e-4844-8023-8f3efc5b45a0"} : dispatch 2024-08-17T17:18:42.994 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:42 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/ca39296d-625e-4844-8023-8f3efc5b45a0"} : dispatch 2024-08-17T17:18:42.994 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:42 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/ca39296d-625e-4844-8023-8f3efc5b45a0"}]': finished 2024-08-17T17:18:42.994 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:42 smithi136 ceph-mon[115078]: Reconfiguring mds.foo.smithi136.ltsuqd (monmap changed)... 2024-08-17T17:18:42.994 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:42 smithi136 ceph-mon[115078]: Reconfiguring daemon mds.foo.smithi136.ltsuqd on smithi136 2024-08-17T17:18:42.994 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:42 smithi136 ceph-mon[115078]: osdmap e483: 8 total, 8 up, 8 in 2024-08-17T17:18:42.994 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:42 smithi136 ceph-mon[115078]: pgmap v219: 105 pgs: 105 active+clean; 583 KiB data, 456 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 5.2 KiB/s wr, 47 op/s 2024-08-17T17:18:42.994 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:42 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/ca39296d-625e-4844-8023-8f3efc5b45a0"} : dispatch 2024-08-17T17:18:42.994 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:42 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/ca39296d-625e-4844-8023-8f3efc5b45a0"} : dispatch 2024-08-17T17:18:42.994 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:42 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/ca39296d-625e-4844-8023-8f3efc5b45a0"}]': finished 2024-08-17T17:18:43.247 INFO:teuthology.orchestra.run.smithi136.stdout:true 2024-08-17T17:18:43.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:42 smithi165 ceph-mon[105517]: Reconfiguring mds.foo.smithi136.ltsuqd (monmap changed)... 2024-08-17T17:18:43.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:42 smithi165 ceph-mon[105517]: Reconfiguring daemon mds.foo.smithi136.ltsuqd on smithi136 2024-08-17T17:18:43.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:42 smithi165 ceph-mon[105517]: osdmap e483: 8 total, 8 up, 8 in 2024-08-17T17:18:43.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:42 smithi165 ceph-mon[105517]: pgmap v219: 105 pgs: 105 active+clean; 583 KiB data, 456 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 5.2 KiB/s wr, 47 op/s 2024-08-17T17:18:43.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:42 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/ca39296d-625e-4844-8023-8f3efc5b45a0"} : dispatch 2024-08-17T17:18:43.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:42 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/ca39296d-625e-4844-8023-8f3efc5b45a0"} : dispatch 2024-08-17T17:18:43.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:42 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/ca39296d-625e-4844-8023-8f3efc5b45a0"}]': finished 2024-08-17T17:18:43.807 INFO:teuthology.orchestra.run.smithi136.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-17T17:18:43.807 INFO:teuthology.orchestra.run.smithi136.stdout:alertmanager.a smithi136 *:9093,9094 running (3m) 66s ago 9m 20.9M - 0.27.0 11f11916f8cd c3b9b5b223ab 2024-08-17T17:18:43.807 INFO:teuthology.orchestra.run.smithi136.stdout:grafana.a smithi165 *:3000 running (3m) 16s ago 9m 75.9M - c8b91775d855 04db24fce5e9 2024-08-17T17:18:43.807 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi136.ltsuqd smithi136 running (8m) 66s ago 8m 21.4M - 18.2.4-719-gced649f8 c870753d67c2 faaa06f2950c 2024-08-17T17:18:43.807 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi165.bmrbhh smithi165 running (8m) 16s ago 8m 17.5M - 18.2.4-719-gced649f8 c870753d67c2 c74b00d929cf 2024-08-17T17:18:43.807 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.x smithi165 *:8443,9283,8765 running (4m) 16s ago 13m 456M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 837fd48ff7cb 2024-08-17T17:18:43.807 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.y smithi136 *:8443,9283,8765 running (5m) 66s ago 15m 510M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9acdc6775e48 2024-08-17T17:18:43.807 INFO:teuthology.orchestra.run.smithi136.stdout:mon.a smithi136 running (2m) 66s ago 15m 47.3M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9cf82e613626 2024-08-17T17:18:43.807 INFO:teuthology.orchestra.run.smithi136.stdout:mon.b smithi165 running (26s) 16s ago 13m 24.2M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 1fd357308b8d 2024-08-17T17:18:43.807 INFO:teuthology.orchestra.run.smithi136.stdout:mon.c smithi136 running (2m) 66s ago 13m 41.8M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 2c2ac98ad9d4 2024-08-17T17:18:43.807 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.a smithi136 *:9100 running (4m) 66s ago 9m 10.6M - 1.7.0 72c9c2088986 1e4c644ae533 2024-08-17T17:18:43.808 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.b smithi165 *:9100 running (4m) 16s ago 9m 10.5M - 1.7.0 72c9c2088986 7af2d2d68b06 2024-08-17T17:18:43.808 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0 smithi136 running (13m) 66s ago 13m 110M 2480M 18.2.4-719-gced649f8 c870753d67c2 faab916734b0 2024-08-17T17:18:43.808 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1 smithi136 running (12m) 66s ago 12m 145M 2480M 18.2.4-719-gced649f8 c870753d67c2 861686b6ff82 2024-08-17T17:18:43.808 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2 smithi136 running (12m) 66s ago 12m 185M 2480M 18.2.4-719-gced649f8 c870753d67c2 640be4af4156 2024-08-17T17:18:43.808 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3 smithi136 running (12m) 66s ago 12m 139M 2480M 18.2.4-719-gced649f8 c870753d67c2 14b57a745d30 2024-08-17T17:18:43.808 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4 smithi165 running (11m) 16s ago 11m 124M 2480M 18.2.4-719-gced649f8 c870753d67c2 df366c73e1f2 2024-08-17T17:18:43.808 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5 smithi165 running (11m) 16s ago 11m 99.9M 2480M 18.2.4-719-gced649f8 c870753d67c2 1131665e203a 2024-08-17T17:18:43.808 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6 smithi165 running (10m) 16s ago 10m 135M 2480M 18.2.4-719-gced649f8 c870753d67c2 7bc5fd925293 2024-08-17T17:18:43.808 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7 smithi165 running (10m) 16s ago 10m 109M 2480M 18.2.4-719-gced649f8 c870753d67c2 18382e4ae3b8 2024-08-17T17:18:43.808 INFO:teuthology.orchestra.run.smithi136.stdout:prometheus.a smithi165 *:9095 running (4m) 16s ago 10m 47.9M - 2.51.0 1d3b7f56885b ae8f8b69fe2f 2024-08-17T17:18:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:43 smithi165 ceph-mon[105517]: osdmap e484: 8 total, 8 up, 8 in 2024-08-17T17:18:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:43 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:43 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:43 smithi165 ceph-mon[105517]: Reconfiguring mon.b (monmap changed)... 2024-08-17T17:18:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:43 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:18:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:43 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:18:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:43 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:43 smithi165 ceph-mon[105517]: Reconfiguring daemon mon.b on smithi165 2024-08-17T17:18:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:43 smithi165 ceph-mon[105517]: from='client.44174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:44.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:43 smithi165 ceph-mon[105517]: osdmap e485: 8 total, 8 up, 8 in 2024-08-17T17:18:44.143 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:18:44.143 INFO:teuthology.orchestra.run.smithi136.stdout: "mon": { 2024-08-17T17:18:44.143 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 3 2024-08-17T17:18:44.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:18:44.144 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr": { 2024-08-17T17:18:44.144 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 2 2024-08-17T17:18:44.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:18:44.144 INFO:teuthology.orchestra.run.smithi136.stdout: "osd": { 2024-08-17T17:18:44.144 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 8 2024-08-17T17:18:44.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:18:44.144 INFO:teuthology.orchestra.run.smithi136.stdout: "mds": { 2024-08-17T17:18:44.145 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 2 2024-08-17T17:18:44.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:18:44.145 INFO:teuthology.orchestra.run.smithi136.stdout: "overall": { 2024-08-17T17:18:44.145 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 10, 2024-08-17T17:18:44.145 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 5 2024-08-17T17:18:44.145 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:18:44.145 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:18:44.146 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:43 smithi136 ceph-mon[109861]: osdmap e484: 8 total, 8 up, 8 in 2024-08-17T17:18:44.146 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:43 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:44.146 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:43 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:44.146 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:43 smithi136 ceph-mon[109861]: Reconfiguring mon.b (monmap changed)... 2024-08-17T17:18:44.146 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:43 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:18:44.146 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:43 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:18:44.146 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:43 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:44.147 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:43 smithi136 ceph-mon[109861]: Reconfiguring daemon mon.b on smithi165 2024-08-17T17:18:44.147 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:43 smithi136 ceph-mon[109861]: from='client.44174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:44.147 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:43 smithi136 ceph-mon[109861]: osdmap e485: 8 total, 8 up, 8 in 2024-08-17T17:18:44.147 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:43 smithi136 ceph-mon[115078]: osdmap e484: 8 total, 8 up, 8 in 2024-08-17T17:18:44.147 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:43 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:44.147 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:43 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:44.148 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:43 smithi136 ceph-mon[115078]: Reconfiguring mon.b (monmap changed)... 2024-08-17T17:18:44.148 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:43 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:18:44.148 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:43 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:18:44.148 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:43 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:44.148 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:43 smithi136 ceph-mon[115078]: Reconfiguring daemon mon.b on smithi165 2024-08-17T17:18:44.148 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:43 smithi136 ceph-mon[115078]: from='client.44174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:44.148 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:43 smithi136 ceph-mon[115078]: osdmap e485: 8 total, 8 up, 8 in 2024-08-17T17:18:44.427 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:18:44.427 INFO:teuthology.orchestra.run.smithi136.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d", 2024-08-17T17:18:44.427 INFO:teuthology.orchestra.run.smithi136.stdout: "in_progress": true, 2024-08-17T17:18:44.427 INFO:teuthology.orchestra.run.smithi136.stdout: "which": "Upgrading daemons of type(s) mon. Upgrade limited to 1 daemons (0 remaining).", 2024-08-17T17:18:44.427 INFO:teuthology.orchestra.run.smithi136.stdout: "services_complete": [ 2024-08-17T17:18:44.427 INFO:teuthology.orchestra.run.smithi136.stdout: "mon" 2024-08-17T17:18:44.427 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-17T17:18:44.427 INFO:teuthology.orchestra.run.smithi136.stdout: "progress": "3/3 daemons upgraded", 2024-08-17T17:18:44.427 INFO:teuthology.orchestra.run.smithi136.stdout: "message": "Currently upgrading mon daemons", 2024-08-17T17:18:44.427 INFO:teuthology.orchestra.run.smithi136.stdout: "is_paused": false 2024-08-17T17:18:44.428 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:18:45.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:44 smithi136 ceph-mon[109861]: from='client.44180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:45.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:44 smithi136 ceph-mon[109861]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:45.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:44 smithi136 ceph-mon[109861]: pgmap v222: 105 pgs: 105 active+clean; 583 KiB data, 456 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 5.2 KiB/s wr, 47 op/s 2024-08-17T17:18:45.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:44 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1477269964' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:45.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:44 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:45.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:44 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:45.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:44 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-17T17:18:45.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:44 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-08-17T17:18:45.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:44 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:45.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:44 smithi136 ceph-mon[109861]: osdmap e486: 8 total, 8 up, 8 in 2024-08-17T17:18:45.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:44 smithi136 ceph-mon[115078]: from='client.44180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:45.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:44 smithi136 ceph-mon[115078]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:45.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:44 smithi136 ceph-mon[115078]: pgmap v222: 105 pgs: 105 active+clean; 583 KiB data, 456 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 5.2 KiB/s wr, 47 op/s 2024-08-17T17:18:45.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:44 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1477269964' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:45.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:44 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:45.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:44 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:45.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:44 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-17T17:18:45.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:44 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-08-17T17:18:45.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:44 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:45.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:44 smithi136 ceph-mon[115078]: osdmap e486: 8 total, 8 up, 8 in 2024-08-17T17:18:45.222 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:44 smithi165 ceph-mon[105517]: from='client.44180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:45.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:44 smithi165 ceph-mon[105517]: from='client.34272 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:45.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:44 smithi165 ceph-mon[105517]: pgmap v222: 105 pgs: 105 active+clean; 583 KiB data, 456 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 5.2 KiB/s wr, 47 op/s 2024-08-17T17:18:45.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:44 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1477269964' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:45.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:44 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:45.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:44 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:45.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:44 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-17T17:18:45.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:44 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-08-17T17:18:45.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:44 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:45.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:44 smithi165 ceph-mon[105517]: osdmap e486: 8 total, 8 up, 8 in 2024-08-17T17:18:45.976 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:45 smithi165 ceph-mon[105517]: from='client.44198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:45.977 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:45 smithi165 ceph-mon[105517]: Reconfiguring mgr.x (monmap changed)... 2024-08-17T17:18:45.977 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:45 smithi165 ceph-mon[105517]: Reconfiguring daemon mgr.x on smithi165 2024-08-17T17:18:45.977 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:45 smithi165 ceph-mon[105517]: osdmap e487: 8 total, 8 up, 8 in 2024-08-17T17:18:46.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:45 smithi136 ceph-mon[109861]: from='client.44198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:46.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:45 smithi136 ceph-mon[109861]: Reconfiguring mgr.x (monmap changed)... 2024-08-17T17:18:46.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:45 smithi136 ceph-mon[109861]: Reconfiguring daemon mgr.x on smithi165 2024-08-17T17:18:46.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:45 smithi136 ceph-mon[109861]: osdmap e487: 8 total, 8 up, 8 in 2024-08-17T17:18:46.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:45 smithi136 ceph-mon[115078]: from='client.44198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:46.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:45 smithi136 ceph-mon[115078]: Reconfiguring mgr.x (monmap changed)... 2024-08-17T17:18:46.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:45 smithi136 ceph-mon[115078]: Reconfiguring daemon mgr.x on smithi165 2024-08-17T17:18:46.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:45 smithi136 ceph-mon[115078]: osdmap e487: 8 total, 8 up, 8 in 2024-08-17T17:18:47.324 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:47 smithi165 ceph-mon[105517]: pgmap v225: 105 pgs: 105 active+clean; 583 KiB data, 456 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:18:47.324 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:47 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:47.324 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:47 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:47.324 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:47 smithi165 ceph-mon[105517]: Reconfiguring osd.4 (monmap changed)... 2024-08-17T17:18:47.324 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:47 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-08-17T17:18:47.324 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:47 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:47.325 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:47 smithi165 ceph-mon[105517]: Reconfiguring daemon osd.4 on smithi165 2024-08-17T17:18:47.325 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:47 smithi165 ceph-mon[105517]: osdmap e488: 8 total, 8 up, 8 in 2024-08-17T17:18:47.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:47 smithi136 ceph-mon[109861]: pgmap v225: 105 pgs: 105 active+clean; 583 KiB data, 456 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:18:47.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:47 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:47.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:47 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:47.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:47 smithi136 ceph-mon[109861]: Reconfiguring osd.4 (monmap changed)... 2024-08-17T17:18:47.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:47 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-08-17T17:18:47.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:47 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:47.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:47 smithi136 ceph-mon[109861]: Reconfiguring daemon osd.4 on smithi165 2024-08-17T17:18:47.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:47 smithi136 ceph-mon[109861]: osdmap e488: 8 total, 8 up, 8 in 2024-08-17T17:18:47.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:47 smithi136 ceph-mon[115078]: pgmap v225: 105 pgs: 105 active+clean; 583 KiB data, 456 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:18:47.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:47 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:47.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:47 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:47.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:47 smithi136 ceph-mon[115078]: Reconfiguring osd.4 (monmap changed)... 2024-08-17T17:18:47.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:47 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-08-17T17:18:47.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:47 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:47.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:47 smithi136 ceph-mon[115078]: Reconfiguring daemon osd.4 on smithi165 2024-08-17T17:18:47.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:47 smithi136 ceph-mon[115078]: osdmap e488: 8 total, 8 up, 8 in 2024-08-17T17:18:47.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:18:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:18:47] "GET /metrics HTTP/1.1" 200 36283 "" "Prometheus/2.51.0" 2024-08-17T17:18:48.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:48 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:48.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:48 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:48.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:48 smithi136 ceph-mon[109861]: Reconfiguring osd.5 (monmap changed)... 2024-08-17T17:18:48.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:48 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-08-17T17:18:48.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:48 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:48.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:48 smithi136 ceph-mon[109861]: Reconfiguring daemon osd.5 on smithi165 2024-08-17T17:18:48.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:48 smithi136 ceph-mon[109861]: osdmap e489: 8 total, 8 up, 8 in 2024-08-17T17:18:48.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:48 smithi136 ceph-mon[109861]: pgmap v228: 105 pgs: 105 active+clean; 583 KiB data, 457 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 16 KiB/s wr, 147 op/s 2024-08-17T17:18:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:48 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:48 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:48 smithi136 ceph-mon[115078]: Reconfiguring osd.5 (monmap changed)... 2024-08-17T17:18:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:48 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-08-17T17:18:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:48 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:48 smithi136 ceph-mon[115078]: Reconfiguring daemon osd.5 on smithi165 2024-08-17T17:18:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:48 smithi136 ceph-mon[115078]: osdmap e489: 8 total, 8 up, 8 in 2024-08-17T17:18:48.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:48 smithi136 ceph-mon[115078]: pgmap v228: 105 pgs: 105 active+clean; 583 KiB data, 457 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 16 KiB/s wr, 147 op/s 2024-08-17T17:18:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:48 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:48 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:48 smithi165 ceph-mon[105517]: Reconfiguring osd.5 (monmap changed)... 2024-08-17T17:18:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:48 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-08-17T17:18:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:48 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:48 smithi165 ceph-mon[105517]: Reconfiguring daemon osd.5 on smithi165 2024-08-17T17:18:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:48 smithi165 ceph-mon[105517]: osdmap e489: 8 total, 8 up, 8 in 2024-08-17T17:18:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:48 smithi165 ceph-mon[105517]: pgmap v228: 105 pgs: 105 active+clean; 583 KiB data, 457 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 16 KiB/s wr, 147 op/s 2024-08-17T17:18:50.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:49 smithi136 ceph-mon[109861]: osdmap e490: 8 total, 8 up, 8 in 2024-08-17T17:18:50.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:49 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:50.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:49 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:50.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:49 smithi136 ceph-mon[109861]: Reconfiguring osd.6 (monmap changed)... 2024-08-17T17:18:50.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:49 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-08-17T17:18:50.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:49 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:50.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:49 smithi136 ceph-mon[109861]: Reconfiguring daemon osd.6 on smithi165 2024-08-17T17:18:50.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:49 smithi136 ceph-mon[115078]: osdmap e490: 8 total, 8 up, 8 in 2024-08-17T17:18:50.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:49 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:50.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:49 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:50.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:49 smithi136 ceph-mon[115078]: Reconfiguring osd.6 (monmap changed)... 2024-08-17T17:18:50.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:49 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-08-17T17:18:50.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:49 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:50.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:49 smithi136 ceph-mon[115078]: Reconfiguring daemon osd.6 on smithi165 2024-08-17T17:18:50.212 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:49 smithi165 ceph-mon[105517]: osdmap e490: 8 total, 8 up, 8 in 2024-08-17T17:18:50.212 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:49 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:50.212 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:49 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:50.212 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:49 smithi165 ceph-mon[105517]: Reconfiguring osd.6 (monmap changed)... 2024-08-17T17:18:50.212 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:49 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-08-17T17:18:50.212 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:49 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:50.212 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:49 smithi165 ceph-mon[105517]: Reconfiguring daemon osd.6 on smithi165 2024-08-17T17:18:50.965 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:50 smithi165 ceph-mon[105517]: osdmap e491: 8 total, 8 up, 8 in 2024-08-17T17:18:50.965 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:50 smithi165 ceph-mon[105517]: pgmap v231: 105 pgs: 105 active+clean; 583 KiB data, 457 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 16 KiB/s wr, 147 op/s 2024-08-17T17:18:50.966 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:50 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:50.966 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:50 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:50.966 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:50 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-08-17T17:18:50.966 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:50 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:51.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:50 smithi136 ceph-mon[109861]: osdmap e491: 8 total, 8 up, 8 in 2024-08-17T17:18:51.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:50 smithi136 ceph-mon[109861]: pgmap v231: 105 pgs: 105 active+clean; 583 KiB data, 457 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 16 KiB/s wr, 147 op/s 2024-08-17T17:18:51.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:50 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:51.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:50 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:51.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:50 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-08-17T17:18:51.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:50 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:51.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:50 smithi136 ceph-mon[115078]: osdmap e491: 8 total, 8 up, 8 in 2024-08-17T17:18:51.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:50 smithi136 ceph-mon[115078]: pgmap v231: 105 pgs: 105 active+clean; 583 KiB data, 457 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 16 KiB/s wr, 147 op/s 2024-08-17T17:18:51.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:50 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:51.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:50 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:51.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:50 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-08-17T17:18:51.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:50 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:51 smithi165 ceph-mon[105517]: Reconfiguring osd.7 (monmap changed)... 2024-08-17T17:18:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:51 smithi165 ceph-mon[105517]: Reconfiguring daemon osd.7 on smithi165 2024-08-17T17:18:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:51 smithi165 ceph-mon[105517]: osdmap e492: 8 total, 8 up, 8 in 2024-08-17T17:18:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:18:52.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:51 smithi136 ceph-mon[109861]: Reconfiguring osd.7 (monmap changed)... 2024-08-17T17:18:52.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:51 smithi136 ceph-mon[109861]: Reconfiguring daemon osd.7 on smithi165 2024-08-17T17:18:52.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:51 smithi136 ceph-mon[109861]: osdmap e492: 8 total, 8 up, 8 in 2024-08-17T17:18:52.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:18:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:51 smithi136 ceph-mon[115078]: Reconfiguring osd.7 (monmap changed)... 2024-08-17T17:18:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:51 smithi136 ceph-mon[115078]: Reconfiguring daemon osd.7 on smithi165 2024-08-17T17:18:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:51 smithi136 ceph-mon[115078]: osdmap e492: 8 total, 8 up, 8 in 2024-08-17T17:18:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:18:53.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:52 smithi136 ceph-mon[109861]: osdmap e493: 8 total, 8 up, 8 in 2024-08-17T17:18:53.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:52 smithi136 ceph-mon[109861]: pgmap v234: 105 pgs: 105 active+clean; 584 KiB data, 458 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 11 KiB/s wr, 103 op/s 2024-08-17T17:18:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:52 smithi136 ceph-mon[109861]: Reconfiguring mds.foo.smithi165.bmrbhh (monmap changed)... 2024-08-17T17:18:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi165.bmrbhh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-17T17:18:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:52 smithi136 ceph-mon[109861]: Reconfiguring daemon mds.foo.smithi165.bmrbhh on smithi165 2024-08-17T17:18:53.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:52 smithi136 ceph-mon[115078]: osdmap e493: 8 total, 8 up, 8 in 2024-08-17T17:18:53.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:52 smithi136 ceph-mon[115078]: pgmap v234: 105 pgs: 105 active+clean; 584 KiB data, 458 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 11 KiB/s wr, 103 op/s 2024-08-17T17:18:53.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:53.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:53.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:52 smithi136 ceph-mon[115078]: Reconfiguring mds.foo.smithi165.bmrbhh (monmap changed)... 2024-08-17T17:18:53.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi165.bmrbhh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-17T17:18:53.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:53.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:52 smithi136 ceph-mon[115078]: Reconfiguring daemon mds.foo.smithi165.bmrbhh on smithi165 2024-08-17T17:18:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:52 smithi165 ceph-mon[105517]: osdmap e493: 8 total, 8 up, 8 in 2024-08-17T17:18:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:52 smithi165 ceph-mon[105517]: pgmap v234: 105 pgs: 105 active+clean; 584 KiB data, 458 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 11 KiB/s wr, 103 op/s 2024-08-17T17:18:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:52 smithi165 ceph-mon[105517]: Reconfiguring mds.foo.smithi165.bmrbhh (monmap changed)... 2024-08-17T17:18:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi165.bmrbhh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-17T17:18:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:52 smithi165 ceph-mon[105517]: Reconfiguring daemon mds.foo.smithi165.bmrbhh on smithi165 2024-08-17T17:18:54.112 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: osdmap e494: 8 total, 8 up, 8 in 2024-08-17T17:18:54.112 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:54.113 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:54.113 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:18:54.113 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:54.113 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:54.113 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:54.113 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:54.113 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:54.113 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-08-17T17:18:54.113 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-08-17T17:18:54.113 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:54.113 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-08-17T17:18:54.113 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:54.114 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-08-17T17:18:54.114 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:54.114 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-08-17T17:18:54.114 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:54.114 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-08-17T17:18:54.114 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:54.114 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-08-17T17:18:54.114 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-08-17T17:18:54.114 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-08-17T17:18:54.114 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:18:54.114 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-08-17T17:18:54.114 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-08-17T17:18:54.115 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: osdmap e494: 8 total, 8 up, 8 in 2024-08-17T17:18:54.115 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:54.115 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:54.115 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:18:54.115 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:54.115 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:54.115 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:54.115 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:54.115 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:54.115 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-08-17T17:18:54.116 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-08-17T17:18:54.116 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:54.116 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-08-17T17:18:54.116 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:54.116 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-08-17T17:18:54.116 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:54.116 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-08-17T17:18:54.116 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:54.116 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-08-17T17:18:54.116 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:54.116 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-08-17T17:18:54.116 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-08-17T17:18:54.116 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-08-17T17:18:54.116 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:18:54.117 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-08-17T17:18:54.117 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-08-17T17:18:54.117 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-08-17T17:18:54.117 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-08-17T17:18:54.117 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-08-17T17:18:54.117 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-08-17T17:18:54.117 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:18:54.117 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-08-17T17:18:54.117 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-08-17T17:18:54.118 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-08-17T17:18:54.118 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-08-17T17:18:54.118 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-08-17T17:18:54.118 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:18:54.118 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:18:54.118 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:18:54.118 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:18:54.118 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:18:54.118 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:18:54.118 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-08-17T17:18:54.118 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-08-17T17:18:54.118 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:18:54.118 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[115078]: osdmap e495: 8 total, 8 up, 8 in 2024-08-17T17:18:54.119 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-08-17T17:18:54.119 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-08-17T17:18:54.119 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-08-17T17:18:54.119 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-08-17T17:18:54.119 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:18:54.119 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-08-17T17:18:54.119 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-08-17T17:18:54.119 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-08-17T17:18:54.119 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-08-17T17:18:54.119 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-08-17T17:18:54.119 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:18:54.119 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:18:54.120 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:18:54.120 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:18:54.120 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:18:54.120 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:18:54.120 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-08-17T17:18:54.120 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-08-17T17:18:54.120 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:18:54.120 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:53 smithi136 ceph-mon[109861]: osdmap e495: 8 total, 8 up, 8 in 2024-08-17T17:18:54.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: osdmap e494: 8 total, 8 up, 8 in 2024-08-17T17:18:54.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:54.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:54.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:18:54.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:54.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:54.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:54.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:54.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:54.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-08-17T17:18:54.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-08-17T17:18:54.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:54.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-08-17T17:18:54.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:54.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-08-17T17:18:54.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:54.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-08-17T17:18:54.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:54.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-08-17T17:18:54.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:18:54.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-08-17T17:18:54.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-08-17T17:18:54.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-08-17T17:18:54.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:18:54.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-08-17T17:18:54.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-08-17T17:18:54.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-08-17T17:18:54.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-08-17T17:18:54.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-08-17T17:18:54.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-08-17T17:18:54.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:18:54.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-08-17T17:18:54.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-08-17T17:18:54.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-08-17T17:18:54.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-08-17T17:18:54.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-08-17T17:18:54.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:18:54.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:18:54.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:18:54.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:18:54.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:18:54.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:18:54.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-08-17T17:18:54.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-08-17T17:18:54.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:18:54.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:53 smithi165 ceph-mon[105517]: osdmap e495: 8 total, 8 up, 8 in 2024-08-17T17:18:55.126 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:54 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all nvmeof 2024-08-17T17:18:55.126 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:54 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all node-exporter 2024-08-17T17:18:55.126 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:54 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all prometheus 2024-08-17T17:18:55.126 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:54 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all alertmanager 2024-08-17T17:18:55.127 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:54 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all grafana 2024-08-17T17:18:55.127 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:54 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all loki 2024-08-17T17:18:55.127 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:54 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all promtail 2024-08-17T17:18:55.127 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:54 smithi136 ceph-mon[115078]: Upgrade: Finalizing container_image settings 2024-08-17T17:18:55.127 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:54 smithi136 ceph-mon[115078]: Upgrade: Complete! 2024-08-17T17:18:55.127 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:54 smithi136 ceph-mon[115078]: pgmap v237: 105 pgs: 105 active+clean; 584 KiB data, 458 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 11 KiB/s wr, 103 op/s 2024-08-17T17:18:55.127 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:54 smithi136 ceph-mon[115078]: osdmap e496: 8 total, 8 up, 8 in 2024-08-17T17:18:55.127 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:54 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all nvmeof 2024-08-17T17:18:55.127 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:54 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all node-exporter 2024-08-17T17:18:55.127 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:54 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all prometheus 2024-08-17T17:18:55.127 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:54 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all alertmanager 2024-08-17T17:18:55.127 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:54 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all grafana 2024-08-17T17:18:55.128 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:54 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all loki 2024-08-17T17:18:55.128 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:54 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all promtail 2024-08-17T17:18:55.128 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:54 smithi136 ceph-mon[109861]: Upgrade: Finalizing container_image settings 2024-08-17T17:18:55.128 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:54 smithi136 ceph-mon[109861]: Upgrade: Complete! 2024-08-17T17:18:55.128 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:54 smithi136 ceph-mon[109861]: pgmap v237: 105 pgs: 105 active+clean; 584 KiB data, 458 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 11 KiB/s wr, 103 op/s 2024-08-17T17:18:55.128 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:54 smithi136 ceph-mon[109861]: osdmap e496: 8 total, 8 up, 8 in 2024-08-17T17:18:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:54 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all nvmeof 2024-08-17T17:18:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:54 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all node-exporter 2024-08-17T17:18:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:54 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all prometheus 2024-08-17T17:18:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:54 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all alertmanager 2024-08-17T17:18:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:54 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all grafana 2024-08-17T17:18:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:54 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all loki 2024-08-17T17:18:55.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:54 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all promtail 2024-08-17T17:18:55.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:54 smithi165 ceph-mon[105517]: Upgrade: Finalizing container_image settings 2024-08-17T17:18:55.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:54 smithi165 ceph-mon[105517]: Upgrade: Complete! 2024-08-17T17:18:55.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:54 smithi165 ceph-mon[105517]: pgmap v237: 105 pgs: 105 active+clean; 584 KiB data, 458 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 11 KiB/s wr, 103 op/s 2024-08-17T17:18:55.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:54 smithi165 ceph-mon[105517]: osdmap e496: 8 total, 8 up, 8 in 2024-08-17T17:18:57.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:56 smithi136 ceph-mon[109861]: pgmap v239: 105 pgs: 105 active+clean; 584 KiB data, 458 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 4.7 KiB/s wr, 62 op/s 2024-08-17T17:18:57.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:56 smithi136 ceph-mon[109861]: osdmap e497: 8 total, 8 up, 8 in 2024-08-17T17:18:57.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:56 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:57.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:56 smithi136 ceph-mon[115078]: pgmap v239: 105 pgs: 105 active+clean; 584 KiB data, 458 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 4.7 KiB/s wr, 62 op/s 2024-08-17T17:18:57.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:56 smithi136 ceph-mon[115078]: osdmap e497: 8 total, 8 up, 8 in 2024-08-17T17:18:57.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:56 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:57.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:56 smithi165 ceph-mon[105517]: pgmap v239: 105 pgs: 105 active+clean; 584 KiB data, 458 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 4.7 KiB/s wr, 62 op/s 2024-08-17T17:18:57.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:56 smithi165 ceph-mon[105517]: osdmap e497: 8 total, 8 up, 8 in 2024-08-17T17:18:57.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:56 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:57.870 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:18:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:18:57] "GET /metrics HTTP/1.1" 200 36285 "" "Prometheus/2.51.0" 2024-08-17T17:18:58.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:57 smithi136 ceph-mon[109861]: osdmap e498: 8 total, 8 up, 8 in 2024-08-17T17:18:58.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:57 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:58.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:57 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:58.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:57 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:58.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:57 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:18:58.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:57 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:58.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:57 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:18:58.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:57 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:58.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:57 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:18:58.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:57 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:58.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:57 smithi136 ceph-mon[115078]: osdmap e498: 8 total, 8 up, 8 in 2024-08-17T17:18:58.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:57 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:58.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:57 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:58.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:57 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:58.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:57 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:18:58.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:57 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:58.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:57 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:18:58.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:57 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:58.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:57 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:18:58.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:57 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:57 smithi165 ceph-mon[105517]: osdmap e498: 8 total, 8 up, 8 in 2024-08-17T17:18:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:57 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:57 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:57 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:57 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:18:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:57 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:57 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:18:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:57 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:18:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:57 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:18:58.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:57 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:18:59.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:58 smithi136 ceph-mon[109861]: pgmap v242: 105 pgs: 105 active+clean; 584 KiB data, 462 MiB used, 715 GiB / 715 GiB avail; 171 KiB/s rd, 20 KiB/s wr, 231 op/s 2024-08-17T17:18:59.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:58 smithi136 ceph-mon[109861]: osdmap e499: 8 total, 8 up, 8 in 2024-08-17T17:18:59.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:18:58 smithi136 ceph-mon[109861]: osdmap e500: 8 total, 8 up, 8 in 2024-08-17T17:18:59.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:58 smithi136 ceph-mon[115078]: pgmap v242: 105 pgs: 105 active+clean; 584 KiB data, 462 MiB used, 715 GiB / 715 GiB avail; 171 KiB/s rd, 20 KiB/s wr, 231 op/s 2024-08-17T17:18:59.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:58 smithi136 ceph-mon[115078]: osdmap e499: 8 total, 8 up, 8 in 2024-08-17T17:18:59.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:18:58 smithi136 ceph-mon[115078]: osdmap e500: 8 total, 8 up, 8 in 2024-08-17T17:18:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:58 smithi165 ceph-mon[105517]: pgmap v242: 105 pgs: 105 active+clean; 584 KiB data, 462 MiB used, 715 GiB / 715 GiB avail; 171 KiB/s rd, 20 KiB/s wr, 231 op/s 2024-08-17T17:18:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:58 smithi165 ceph-mon[105517]: osdmap e499: 8 total, 8 up, 8 in 2024-08-17T17:18:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:18:58 smithi165 ceph-mon[105517]: osdmap e500: 8 total, 8 up, 8 in 2024-08-17T17:19:01.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:00 smithi136 ceph-mon[109861]: pgmap v245: 105 pgs: 105 active+clean; 584 KiB data, 462 MiB used, 715 GiB / 715 GiB avail; 173 KiB/s rd, 20 KiB/s wr, 234 op/s 2024-08-17T17:19:01.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:00 smithi136 ceph-mon[109861]: osdmap e501: 8 total, 8 up, 8 in 2024-08-17T17:19:01.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:00 smithi136 ceph-mon[115078]: pgmap v245: 105 pgs: 105 active+clean; 584 KiB data, 462 MiB used, 715 GiB / 715 GiB avail; 173 KiB/s rd, 20 KiB/s wr, 234 op/s 2024-08-17T17:19:01.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:00 smithi136 ceph-mon[115078]: osdmap e501: 8 total, 8 up, 8 in 2024-08-17T17:19:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:00 smithi165 ceph-mon[105517]: pgmap v245: 105 pgs: 105 active+clean; 584 KiB data, 462 MiB used, 715 GiB / 715 GiB avail; 173 KiB/s rd, 20 KiB/s wr, 234 op/s 2024-08-17T17:19:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:00 smithi165 ceph-mon[105517]: osdmap e501: 8 total, 8 up, 8 in 2024-08-17T17:19:02.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:01 smithi136 ceph-mon[109861]: osdmap e502: 8 total, 8 up, 8 in 2024-08-17T17:19:02.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:01 smithi136 ceph-mon[115078]: osdmap e502: 8 total, 8 up, 8 in 2024-08-17T17:19:02.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:01 smithi165 ceph-mon[105517]: osdmap e502: 8 total, 8 up, 8 in 2024-08-17T17:19:03.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:02 smithi136 ceph-mon[109861]: pgmap v248: 105 pgs: 105 active+clean; 584 KiB data, 463 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 8.7 KiB/s wr, 160 op/s 2024-08-17T17:19:03.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:02 smithi136 ceph-mon[109861]: osdmap e503: 8 total, 8 up, 8 in 2024-08-17T17:19:03.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:02 smithi136 ceph-mon[115078]: pgmap v248: 105 pgs: 105 active+clean; 584 KiB data, 463 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 8.7 KiB/s wr, 160 op/s 2024-08-17T17:19:03.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:02 smithi136 ceph-mon[115078]: osdmap e503: 8 total, 8 up, 8 in 2024-08-17T17:19:03.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:02 smithi165 ceph-mon[105517]: pgmap v248: 105 pgs: 105 active+clean; 584 KiB data, 463 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 8.7 KiB/s wr, 160 op/s 2024-08-17T17:19:03.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:02 smithi165 ceph-mon[105517]: osdmap e503: 8 total, 8 up, 8 in 2024-08-17T17:19:04.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:03 smithi136 ceph-mon[109861]: osdmap e504: 8 total, 8 up, 8 in 2024-08-17T17:19:04.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:03 smithi136 ceph-mon[109861]: osdmap e505: 8 total, 8 up, 8 in 2024-08-17T17:19:04.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:03 smithi136 ceph-mon[115078]: osdmap e504: 8 total, 8 up, 8 in 2024-08-17T17:19:04.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:03 smithi136 ceph-mon[115078]: osdmap e505: 8 total, 8 up, 8 in 2024-08-17T17:19:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:03 smithi165 ceph-mon[105517]: osdmap e504: 8 total, 8 up, 8 in 2024-08-17T17:19:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:03 smithi165 ceph-mon[105517]: osdmap e505: 8 total, 8 up, 8 in 2024-08-17T17:19:05.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:04 smithi136 ceph-mon[115078]: pgmap v252: 105 pgs: 105 active+clean; 584 KiB data, 463 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 8.8 KiB/s wr, 162 op/s 2024-08-17T17:19:05.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:04 smithi136 ceph-mon[115078]: osdmap e506: 8 total, 8 up, 8 in 2024-08-17T17:19:05.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:04 smithi136 ceph-mon[109861]: pgmap v252: 105 pgs: 105 active+clean; 584 KiB data, 463 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 8.8 KiB/s wr, 162 op/s 2024-08-17T17:19:05.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:04 smithi136 ceph-mon[109861]: osdmap e506: 8 total, 8 up, 8 in 2024-08-17T17:19:05.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:04 smithi165 ceph-mon[105517]: pgmap v252: 105 pgs: 105 active+clean; 584 KiB data, 463 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 8.8 KiB/s wr, 162 op/s 2024-08-17T17:19:05.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:04 smithi165 ceph-mon[105517]: osdmap e506: 8 total, 8 up, 8 in 2024-08-17T17:19:07.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:06 smithi136 ceph-mon[109861]: osdmap e507: 8 total, 8 up, 8 in 2024-08-17T17:19:07.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:06 smithi136 ceph-mon[109861]: pgmap v255: 105 pgs: 105 active+clean; 584 KiB data, 463 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:19:07.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:19:07.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:06 smithi136 ceph-mon[115078]: osdmap e507: 8 total, 8 up, 8 in 2024-08-17T17:19:07.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:06 smithi136 ceph-mon[115078]: pgmap v255: 105 pgs: 105 active+clean; 584 KiB data, 463 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:19:07.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:19:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:06 smithi165 ceph-mon[105517]: osdmap e507: 8 total, 8 up, 8 in 2024-08-17T17:19:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:06 smithi165 ceph-mon[105517]: pgmap v255: 105 pgs: 105 active+clean; 584 KiB data, 463 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:19:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:19:07.818 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:19:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:19:07] "GET /metrics HTTP/1.1" 200 36284 "" "Prometheus/2.51.0" 2024-08-17T17:19:08.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:07 smithi136 ceph-mon[109861]: osdmap e508: 8 total, 8 up, 8 in 2024-08-17T17:19:08.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:07 smithi136 ceph-mon[115078]: osdmap e508: 8 total, 8 up, 8 in 2024-08-17T17:19:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:07 smithi165 ceph-mon[105517]: osdmap e508: 8 total, 8 up, 8 in 2024-08-17T17:19:09.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:08 smithi136 ceph-mon[115078]: osdmap e509: 8 total, 8 up, 8 in 2024-08-17T17:19:09.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:08 smithi136 ceph-mon[115078]: pgmap v258: 105 pgs: 1 active+clean+snaptrim_wait, 104 active+clean; 583 KiB data, 463 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 7.7 KiB/s wr, 120 op/s 2024-08-17T17:19:09.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:08 smithi136 ceph-mon[109861]: osdmap e509: 8 total, 8 up, 8 in 2024-08-17T17:19:09.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:08 smithi136 ceph-mon[109861]: pgmap v258: 105 pgs: 1 active+clean+snaptrim_wait, 104 active+clean; 583 KiB data, 463 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 7.7 KiB/s wr, 120 op/s 2024-08-17T17:19:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:08 smithi165 ceph-mon[105517]: osdmap e509: 8 total, 8 up, 8 in 2024-08-17T17:19:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:08 smithi165 ceph-mon[105517]: pgmap v258: 105 pgs: 1 active+clean+snaptrim_wait, 104 active+clean; 583 KiB data, 463 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 7.7 KiB/s wr, 120 op/s 2024-08-17T17:19:10.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:09 smithi136 ceph-mon[109861]: osdmap e510: 8 total, 8 up, 8 in 2024-08-17T17:19:10.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:09 smithi136 ceph-mon[115078]: osdmap e510: 8 total, 8 up, 8 in 2024-08-17T17:19:10.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:09 smithi165 ceph-mon[105517]: osdmap e510: 8 total, 8 up, 8 in 2024-08-17T17:19:11.098 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:10 smithi136 ceph-mon[109861]: osdmap e511: 8 total, 8 up, 8 in 2024-08-17T17:19:11.098 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:10 smithi136 ceph-mon[109861]: pgmap v261: 105 pgs: 1 active+clean+snaptrim_wait, 104 active+clean; 583 KiB data, 463 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 7.7 KiB/s wr, 120 op/s 2024-08-17T17:19:11.098 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:10 smithi136 ceph-mon[115078]: osdmap e511: 8 total, 8 up, 8 in 2024-08-17T17:19:11.098 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:10 smithi136 ceph-mon[115078]: pgmap v261: 105 pgs: 1 active+clean+snaptrim_wait, 104 active+clean; 583 KiB data, 463 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 7.7 KiB/s wr, 120 op/s 2024-08-17T17:19:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:10 smithi165 ceph-mon[105517]: osdmap e511: 8 total, 8 up, 8 in 2024-08-17T17:19:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:10 smithi165 ceph-mon[105517]: pgmap v261: 105 pgs: 1 active+clean+snaptrim_wait, 104 active+clean; 583 KiB data, 463 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 7.7 KiB/s wr, 120 op/s 2024-08-17T17:19:12.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:11 smithi136 ceph-mon[109861]: osdmap e512: 8 total, 8 up, 8 in 2024-08-17T17:19:12.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:11 smithi136 ceph-mon[115078]: osdmap e512: 8 total, 8 up, 8 in 2024-08-17T17:19:12.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:11 smithi165 ceph-mon[105517]: osdmap e512: 8 total, 8 up, 8 in 2024-08-17T17:19:12.919 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestMirroring.AioSnapshotCreate (31029 ms) 2024-08-17T17:19:13.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:12 smithi136 ceph-mon[109861]: pgmap v263: 105 pgs: 105 active+clean; 583 KiB data, 468 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 8.4 KiB/s wr, 155 op/s 2024-08-17T17:19:13.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:12 smithi136 ceph-mon[109861]: osdmap e513: 8 total, 8 up, 8 in 2024-08-17T17:19:13.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:12 smithi136 ceph-mon[115078]: pgmap v263: 105 pgs: 105 active+clean; 583 KiB data, 468 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 8.4 KiB/s wr, 155 op/s 2024-08-17T17:19:13.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:12 smithi136 ceph-mon[115078]: osdmap e513: 8 total, 8 up, 8 in 2024-08-17T17:19:13.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:12 smithi165 ceph-mon[105517]: pgmap v263: 105 pgs: 105 active+clean; 583 KiB data, 468 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 8.4 KiB/s wr, 155 op/s 2024-08-17T17:19:13.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:12 smithi165 ceph-mon[105517]: osdmap e513: 8 total, 8 up, 8 in 2024-08-17T17:19:13.808 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 51 tests from TestMirroring (91904 ms total) 2024-08-17T17:19:13.808 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:19:13.808 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 34 tests from TestInternal 2024-08-17T17:19:14.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:13 smithi136 ceph-mon[109861]: osdmap e514: 8 total, 8 up, 8 in 2024-08-17T17:19:14.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:13 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/1f3e9c76-0eb8-4c9d-8f23-e2ad738b500f"} : dispatch 2024-08-17T17:19:14.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:13 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/1f3e9c76-0eb8-4c9d-8f23-e2ad738b500f"} : dispatch 2024-08-17T17:19:14.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:13 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/1f3e9c76-0eb8-4c9d-8f23-e2ad738b500f"}]': finished 2024-08-17T17:19:14.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:13 smithi136 ceph-mon[109861]: osdmap e515: 8 total, 8 up, 8 in 2024-08-17T17:19:14.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:13 smithi136 ceph-mon[115078]: osdmap e514: 8 total, 8 up, 8 in 2024-08-17T17:19:14.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:13 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/1f3e9c76-0eb8-4c9d-8f23-e2ad738b500f"} : dispatch 2024-08-17T17:19:14.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:13 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/1f3e9c76-0eb8-4c9d-8f23-e2ad738b500f"} : dispatch 2024-08-17T17:19:14.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:13 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/1f3e9c76-0eb8-4c9d-8f23-e2ad738b500f"}]': finished 2024-08-17T17:19:14.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:13 smithi136 ceph-mon[115078]: osdmap e515: 8 total, 8 up, 8 in 2024-08-17T17:19:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:13 smithi165 ceph-mon[105517]: osdmap e514: 8 total, 8 up, 8 in 2024-08-17T17:19:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:13 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2738657198' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/1f3e9c76-0eb8-4c9d-8f23-e2ad738b500f"} : dispatch 2024-08-17T17:19:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:13 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/1f3e9c76-0eb8-4c9d-8f23-e2ad738b500f"} : dispatch 2024-08-17T17:19:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:13 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/1f3e9c76-0eb8-4c9d-8f23-e2ad738b500f"}]': finished 2024-08-17T17:19:14.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:13 smithi165 ceph-mon[105517]: osdmap e515: 8 total, 8 up, 8 in 2024-08-17T17:19:15.046 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:14 smithi136 ceph-mon[109861]: pgmap v267: 73 pgs: 73 active+clean; 582 KiB data, 468 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:19:15.046 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:14 smithi136 ceph-mon[109861]: osdmap e516: 8 total, 8 up, 8 in 2024-08-17T17:19:15.046 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:14 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3699063369' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:19:15.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:14 smithi136 ceph-mon[115078]: pgmap v267: 73 pgs: 73 active+clean; 582 KiB data, 468 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:19:15.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:14 smithi136 ceph-mon[115078]: osdmap e516: 8 total, 8 up, 8 in 2024-08-17T17:19:15.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:14 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3699063369' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:19:15.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:14 smithi165 ceph-mon[105517]: pgmap v267: 73 pgs: 73 active+clean; 582 KiB data, 468 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:19:15.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:14 smithi165 ceph-mon[105517]: osdmap e516: 8 total, 8 up, 8 in 2024-08-17T17:19:15.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:14 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3699063369' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:19:15.411 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'sleep 60' 2024-08-17T17:19:15.616 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:19:15.810 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.OpenByID 2024-08-17T17:19:16.042 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:15 smithi136 ceph-mon[109861]: from='client.44216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:19:16.042 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:15 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3699063369' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:19:16.042 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:15 smithi136 ceph-mon[109861]: osdmap e517: 8 total, 8 up, 8 in 2024-08-17T17:19:16.043 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:15 smithi136 ceph-mon[115078]: from='client.44216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:19:16.043 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:15 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3699063369' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:19:16.043 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:15 smithi136 ceph-mon[115078]: osdmap e517: 8 total, 8 up, 8 in 2024-08-17T17:19:16.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:15 smithi165 ceph-mon[105517]: from='client.44216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:19:16.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:15 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3699063369' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:19:16.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:15 smithi165 ceph-mon[105517]: osdmap e517: 8 total, 8 up, 8 in 2024-08-17T17:19:17.150 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:16 smithi136 ceph-mon[109861]: pgmap v270: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 468 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:19:17.150 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:16 smithi136 ceph-mon[109861]: osdmap e518: 8 total, 8 up, 8 in 2024-08-17T17:19:17.150 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:16 smithi136 ceph-mon[115078]: pgmap v270: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 468 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:19:17.150 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:16 smithi136 ceph-mon[115078]: osdmap e518: 8 total, 8 up, 8 in 2024-08-17T17:19:17.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:16 smithi165 ceph-mon[105517]: pgmap v270: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 468 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:19:17.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:16 smithi165 ceph-mon[105517]: osdmap e518: 8 total, 8 up, 8 in 2024-08-17T17:19:17.855 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.OpenByID (2045 ms) 2024-08-17T17:19:17.855 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.OpenSnapDNE 2024-08-17T17:19:17.878 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.OpenSnapDNE (23 ms) 2024-08-17T17:19:17.878 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.IsExclusiveLockOwner 2024-08-17T17:19:17.899 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.IsExclusiveLockOwner (21 ms) 2024-08-17T17:19:17.899 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.ResizeLocksImage 2024-08-17T17:19:17.924 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.ResizeLocksImage (25 ms) 2024-08-17T17:19:17.924 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.ResizeFailsToLockImage 2024-08-17T17:19:17.941 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.ResizeFailsToLockImage (17 ms) 2024-08-17T17:19:17.941 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.SnapCreateLocksImage 2024-08-17T17:19:17.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:19:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:19:17] "GET /metrics HTTP/1.1" 200 36284 "" "Prometheus/2.51.0" 2024-08-17T17:19:19.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:18 smithi136 ceph-mon[109861]: osdmap e519: 8 total, 8 up, 8 in 2024-08-17T17:19:19.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:18 smithi136 ceph-mon[109861]: pgmap v273: 105 pgs: 105 active+clean; 582 KiB data, 468 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:19:19.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:18 smithi136 ceph-mon[115078]: osdmap e519: 8 total, 8 up, 8 in 2024-08-17T17:19:19.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:18 smithi136 ceph-mon[115078]: pgmap v273: 105 pgs: 105 active+clean; 582 KiB data, 468 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:19:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:18 smithi165 ceph-mon[105517]: osdmap e519: 8 total, 8 up, 8 in 2024-08-17T17:19:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:18 smithi165 ceph-mon[105517]: pgmap v273: 105 pgs: 105 active+clean; 582 KiB data, 468 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:19:19.837 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.SnapCreateLocksImage (1896 ms) 2024-08-17T17:19:19.838 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.SnapCreateFailsToLockImage 2024-08-17T17:19:19.862 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.SnapCreateFailsToLockImage (25 ms) 2024-08-17T17:19:19.862 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.SnapRollbackLocksImage 2024-08-17T17:19:20.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:19 smithi136 ceph-mon[109861]: osdmap e520: 8 total, 8 up, 8 in 2024-08-17T17:19:20.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:19 smithi136 ceph-mon[115078]: osdmap e520: 8 total, 8 up, 8 in 2024-08-17T17:19:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:19 smithi165 ceph-mon[105517]: osdmap e520: 8 total, 8 up, 8 in 2024-08-17T17:19:21.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:20 smithi136 ceph-mon[109861]: osdmap e521: 8 total, 8 up, 8 in 2024-08-17T17:19:21.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:20 smithi136 ceph-mon[109861]: pgmap v276: 105 pgs: 105 active+clean; 582 KiB data, 468 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:19:21.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:20 smithi136 ceph-mon[115078]: osdmap e521: 8 total, 8 up, 8 in 2024-08-17T17:19:21.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:20 smithi136 ceph-mon[115078]: pgmap v276: 105 pgs: 105 active+clean; 582 KiB data, 468 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:19:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:20 smithi165 ceph-mon[105517]: osdmap e521: 8 total, 8 up, 8 in 2024-08-17T17:19:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:20 smithi165 ceph-mon[105517]: pgmap v276: 105 pgs: 105 active+clean; 582 KiB data, 468 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:19:21.858 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.SnapRollbackLocksImage (1995 ms) 2024-08-17T17:19:21.858 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.SnapRollbackFailsToLockImage 2024-08-17T17:19:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:21 smithi136 ceph-mon[109861]: osdmap e522: 8 total, 8 up, 8 in 2024-08-17T17:19:22.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:19:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:21 smithi136 ceph-mon[115078]: osdmap e522: 8 total, 8 up, 8 in 2024-08-17T17:19:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:19:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:21 smithi165 ceph-mon[105517]: osdmap e522: 8 total, 8 up, 8 in 2024-08-17T17:19:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:19:23.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:22 smithi136 ceph-mon[109861]: osdmap e523: 8 total, 8 up, 8 in 2024-08-17T17:19:23.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:22 smithi136 ceph-mon[109861]: pgmap v279: 105 pgs: 105 active+clean; 582 KiB data, 473 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 12 KiB/s wr, 157 op/s 2024-08-17T17:19:23.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:22 smithi136 ceph-mon[115078]: osdmap e523: 8 total, 8 up, 8 in 2024-08-17T17:19:23.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:22 smithi136 ceph-mon[115078]: pgmap v279: 105 pgs: 105 active+clean; 582 KiB data, 473 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 12 KiB/s wr, 157 op/s 2024-08-17T17:19:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:22 smithi165 ceph-mon[105517]: osdmap e523: 8 total, 8 up, 8 in 2024-08-17T17:19:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:22 smithi165 ceph-mon[105517]: pgmap v279: 105 pgs: 105 active+clean; 582 KiB data, 473 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 12 KiB/s wr, 157 op/s 2024-08-17T17:19:23.825 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.SnapRollbackFailsToLockImage (1968 ms) 2024-08-17T17:19:23.825 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.SnapSetReleasesLock 2024-08-17T17:19:24.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:23 smithi136 ceph-mon[109861]: osdmap e524: 8 total, 8 up, 8 in 2024-08-17T17:19:24.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:23 smithi136 ceph-mon[109861]: osdmap e525: 8 total, 8 up, 8 in 2024-08-17T17:19:24.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:23 smithi136 ceph-mon[115078]: osdmap e524: 8 total, 8 up, 8 in 2024-08-17T17:19:24.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:23 smithi136 ceph-mon[115078]: osdmap e525: 8 total, 8 up, 8 in 2024-08-17T17:19:24.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:23 smithi165 ceph-mon[105517]: osdmap e524: 8 total, 8 up, 8 in 2024-08-17T17:19:24.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:23 smithi165 ceph-mon[105517]: osdmap e525: 8 total, 8 up, 8 in 2024-08-17T17:19:25.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:24 smithi136 ceph-mon[109861]: pgmap v282: 105 pgs: 105 active+clean; 582 KiB data, 473 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 12 KiB/s wr, 157 op/s 2024-08-17T17:19:25.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:24 smithi136 ceph-mon[109861]: osdmap e526: 8 total, 8 up, 8 in 2024-08-17T17:19:25.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:24 smithi136 ceph-mon[115078]: pgmap v282: 105 pgs: 105 active+clean; 582 KiB data, 473 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 12 KiB/s wr, 157 op/s 2024-08-17T17:19:25.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:24 smithi136 ceph-mon[115078]: osdmap e526: 8 total, 8 up, 8 in 2024-08-17T17:19:25.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:24 smithi165 ceph-mon[105517]: pgmap v282: 105 pgs: 105 active+clean; 582 KiB data, 473 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 12 KiB/s wr, 157 op/s 2024-08-17T17:19:25.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:24 smithi165 ceph-mon[105517]: osdmap e526: 8 total, 8 up, 8 in 2024-08-17T17:19:25.877 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.SnapSetReleasesLock (2052 ms) 2024-08-17T17:19:25.877 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.FlattenLocksImage 2024-08-17T17:19:27.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:26 smithi136 ceph-mon[109861]: pgmap v284: 105 pgs: 105 active+clean; 582 KiB data, 473 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 12 KiB/s wr, 157 op/s 2024-08-17T17:19:27.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:26 smithi136 ceph-mon[109861]: osdmap e527: 8 total, 8 up, 8 in 2024-08-17T17:19:27.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:26 smithi136 ceph-mon[115078]: pgmap v284: 105 pgs: 105 active+clean; 582 KiB data, 473 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 12 KiB/s wr, 157 op/s 2024-08-17T17:19:27.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:26 smithi136 ceph-mon[115078]: osdmap e527: 8 total, 8 up, 8 in 2024-08-17T17:19:27.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:26 smithi165 ceph-mon[105517]: pgmap v284: 105 pgs: 105 active+clean; 582 KiB data, 473 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 12 KiB/s wr, 157 op/s 2024-08-17T17:19:27.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:26 smithi165 ceph-mon[105517]: osdmap e527: 8 total, 8 up, 8 in 2024-08-17T17:19:27.871 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:19:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:19:27] "GET /metrics HTTP/1.1" 200 36260 "" "Prometheus/2.51.0" 2024-08-17T17:19:27.897 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.FlattenLocksImage (2020 ms) 2024-08-17T17:19:27.897 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.FlattenFailsToLockImage 2024-08-17T17:19:28.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:27 smithi136 ceph-mon[109861]: osdmap e528: 8 total, 8 up, 8 in 2024-08-17T17:19:28.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:27 smithi136 ceph-mon[115078]: osdmap e528: 8 total, 8 up, 8 in 2024-08-17T17:19:28.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:27 smithi165 ceph-mon[105517]: osdmap e528: 8 total, 8 up, 8 in 2024-08-17T17:19:29.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:28 smithi136 ceph-mon[109861]: pgmap v287: 105 pgs: 105 active+clean; 583 KiB data, 457 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 6.7 KiB/s wr, 98 op/s 2024-08-17T17:19:29.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:28 smithi136 ceph-mon[109861]: osdmap e529: 8 total, 8 up, 8 in 2024-08-17T17:19:29.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:28 smithi136 ceph-mon[109861]: osdmap e530: 8 total, 8 up, 8 in 2024-08-17T17:19:29.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:28 smithi136 ceph-mon[115078]: pgmap v287: 105 pgs: 105 active+clean; 583 KiB data, 457 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 6.7 KiB/s wr, 98 op/s 2024-08-17T17:19:29.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:28 smithi136 ceph-mon[115078]: osdmap e529: 8 total, 8 up, 8 in 2024-08-17T17:19:29.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:28 smithi136 ceph-mon[115078]: osdmap e530: 8 total, 8 up, 8 in 2024-08-17T17:19:29.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:28 smithi165 ceph-mon[105517]: pgmap v287: 105 pgs: 105 active+clean; 583 KiB data, 457 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 6.7 KiB/s wr, 98 op/s 2024-08-17T17:19:29.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:28 smithi165 ceph-mon[105517]: osdmap e529: 8 total, 8 up, 8 in 2024-08-17T17:19:29.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:28 smithi165 ceph-mon[105517]: osdmap e530: 8 total, 8 up, 8 in 2024-08-17T17:19:29.893 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.FlattenFailsToLockImage (1996 ms) 2024-08-17T17:19:29.894 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.WriteFailsToLockImageBlocklisted 2024-08-17T17:19:30.891 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.WriteFailsToLockImageBlocklisted (998 ms) 2024-08-17T17:19:30.892 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.WriteFailsToLockImageBlocklistedWatch 2024-08-17T17:19:31.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:30 smithi136 ceph-mon[109861]: pgmap v290: 105 pgs: 105 active+clean; 583 KiB data, 457 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 6.7 KiB/s wr, 99 op/s 2024-08-17T17:19:31.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:30 smithi136 ceph-mon[109861]: osdmap e531: 8 total, 8 up, 8 in 2024-08-17T17:19:31.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:30 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1504550954' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.136:0/1504550954"} : dispatch 2024-08-17T17:19:31.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:30 smithi136 ceph-mon[115078]: pgmap v290: 105 pgs: 105 active+clean; 583 KiB data, 457 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 6.7 KiB/s wr, 99 op/s 2024-08-17T17:19:31.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:30 smithi136 ceph-mon[115078]: osdmap e531: 8 total, 8 up, 8 in 2024-08-17T17:19:31.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:30 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1504550954' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.136:0/1504550954"} : dispatch 2024-08-17T17:19:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:30 smithi165 ceph-mon[105517]: pgmap v290: 105 pgs: 105 active+clean; 583 KiB data, 457 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 6.7 KiB/s wr, 99 op/s 2024-08-17T17:19:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:30 smithi165 ceph-mon[105517]: osdmap e531: 8 total, 8 up, 8 in 2024-08-17T17:19:31.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:30 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1504550954' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.136:0/1504550954"} : dispatch 2024-08-17T17:19:32.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:31 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1504550954' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.136:0/1504550954"}]': finished 2024-08-17T17:19:32.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:31 smithi136 ceph-mon[109861]: osdmap e532: 8 total, 8 up, 8 in 2024-08-17T17:19:32.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:31 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2190679281' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.136:0/2190679281"} : dispatch 2024-08-17T17:19:32.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:31 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.136:0/2190679281"} : dispatch 2024-08-17T17:19:32.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:31 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1504550954' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.136:0/1504550954"}]': finished 2024-08-17T17:19:32.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:31 smithi136 ceph-mon[115078]: osdmap e532: 8 total, 8 up, 8 in 2024-08-17T17:19:32.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:31 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2190679281' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.136:0/2190679281"} : dispatch 2024-08-17T17:19:32.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:31 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.136:0/2190679281"} : dispatch 2024-08-17T17:19:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:31 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1504550954' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.136:0/1504550954"}]': finished 2024-08-17T17:19:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:31 smithi165 ceph-mon[105517]: osdmap e532: 8 total, 8 up, 8 in 2024-08-17T17:19:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:31 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2190679281' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.136:0/2190679281"} : dispatch 2024-08-17T17:19:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:31 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.136:0/2190679281"} : dispatch 2024-08-17T17:19:32.896 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.WriteFailsToLockImageBlocklistedWatch (2005 ms) 2024-08-17T17:19:32.896 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.AioWriteRequestsLock 2024-08-17T17:19:32.930 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.AioWriteRequestsLock (34 ms) 2024-08-17T17:19:32.931 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.AioDiscardRequestsLock 2024-08-17T17:19:32.957 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.AioDiscardRequestsLock (27 ms) 2024-08-17T17:19:32.957 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.CancelAsyncResize 2024-08-17T17:19:33.002 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.CancelAsyncResize (45 ms) 2024-08-17T17:19:33.002 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.MultipleResize 2024-08-17T17:19:33.057 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.MultipleResize (55 ms) 2024-08-17T17:19:33.057 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.Metadata 2024-08-17T17:19:33.103 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.Metadata (46 ms) 2024-08-17T17:19:33.103 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.MetadataConfApply 2024-08-17T17:19:33.131 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.MetadataConfApply (28 ms) 2024-08-17T17:19:33.131 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.SnapshotCopyup 2024-08-17T17:19:33.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:32 smithi136 ceph-mon[109861]: pgmap v293: 105 pgs: 105 active+clean; 583 KiB data, 462 MiB used, 715 GiB / 715 GiB avail; 184 KiB/s rd, 12 KiB/s wr, 247 op/s 2024-08-17T17:19:33.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:32 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.136:0/2190679281"}]': finished 2024-08-17T17:19:33.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:32 smithi136 ceph-mon[109861]: osdmap e533: 8 total, 8 up, 8 in 2024-08-17T17:19:33.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:32 smithi136 ceph-mon[115078]: pgmap v293: 105 pgs: 105 active+clean; 583 KiB data, 462 MiB used, 715 GiB / 715 GiB avail; 184 KiB/s rd, 12 KiB/s wr, 247 op/s 2024-08-17T17:19:33.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:32 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.136:0/2190679281"}]': finished 2024-08-17T17:19:33.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:32 smithi136 ceph-mon[115078]: osdmap e533: 8 total, 8 up, 8 in 2024-08-17T17:19:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:32 smithi165 ceph-mon[105517]: pgmap v293: 105 pgs: 105 active+clean; 583 KiB data, 462 MiB used, 715 GiB / 715 GiB avail; 184 KiB/s rd, 12 KiB/s wr, 247 op/s 2024-08-17T17:19:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:32 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.136:0/2190679281"}]': finished 2024-08-17T17:19:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:32 smithi165 ceph-mon[105517]: osdmap e533: 8 total, 8 up, 8 in 2024-08-17T17:19:35.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:34 smithi136 ceph-mon[109861]: osdmap e534: 8 total, 8 up, 8 in 2024-08-17T17:19:35.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:34 smithi136 ceph-mon[109861]: pgmap v296: 105 pgs: 105 active+clean; 583 KiB data, 462 MiB used, 715 GiB / 715 GiB avail; 184 KiB/s rd, 12 KiB/s wr, 247 op/s 2024-08-17T17:19:35.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:34 smithi136 ceph-mon[115078]: osdmap e534: 8 total, 8 up, 8 in 2024-08-17T17:19:35.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:34 smithi136 ceph-mon[115078]: pgmap v296: 105 pgs: 105 active+clean; 583 KiB data, 462 MiB used, 715 GiB / 715 GiB avail; 184 KiB/s rd, 12 KiB/s wr, 247 op/s 2024-08-17T17:19:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:34 smithi165 ceph-mon[105517]: osdmap e534: 8 total, 8 up, 8 in 2024-08-17T17:19:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:34 smithi165 ceph-mon[105517]: pgmap v296: 105 pgs: 105 active+clean; 583 KiB data, 462 MiB used, 715 GiB / 715 GiB avail; 184 KiB/s rd, 12 KiB/s wr, 247 op/s 2024-08-17T17:19:35.892 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.SnapshotCopyup (2761 ms) 2024-08-17T17:19:35.892 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.SnapshotCopyupZeros 2024-08-17T17:19:36.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:35 smithi136 ceph-mon[109861]: osdmap e535: 8 total, 8 up, 8 in 2024-08-17T17:19:36.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:35 smithi136 ceph-mon[115078]: osdmap e535: 8 total, 8 up, 8 in 2024-08-17T17:19:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:35 smithi165 ceph-mon[105517]: osdmap e535: 8 total, 8 up, 8 in 2024-08-17T17:19:37.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:36 smithi136 ceph-mon[109861]: osdmap e536: 8 total, 8 up, 8 in 2024-08-17T17:19:37.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:36 smithi136 ceph-mon[109861]: pgmap v299: 105 pgs: 105 active+clean; 583 KiB data, 462 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:19:37.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:19:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:36 smithi136 ceph-mon[115078]: osdmap e536: 8 total, 8 up, 8 in 2024-08-17T17:19:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:36 smithi136 ceph-mon[115078]: pgmap v299: 105 pgs: 105 active+clean; 583 KiB data, 462 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:19:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:19:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:36 smithi165 ceph-mon[105517]: osdmap e536: 8 total, 8 up, 8 in 2024-08-17T17:19:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:36 smithi165 ceph-mon[105517]: pgmap v299: 105 pgs: 105 active+clean; 583 KiB data, 462 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:19:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:19:37.824 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:19:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:19:37] "GET /metrics HTTP/1.1" 200 36279 "" "Prometheus/2.51.0" 2024-08-17T17:19:37.863 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.SnapshotCopyupZeros (1971 ms) 2024-08-17T17:19:37.864 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.SnapshotCopyupZerosMigration 2024-08-17T17:19:38.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:37 smithi136 ceph-mon[109861]: osdmap e537: 8 total, 8 up, 8 in 2024-08-17T17:19:38.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:37 smithi136 ceph-mon[115078]: osdmap e537: 8 total, 8 up, 8 in 2024-08-17T17:19:38.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:37 smithi165 ceph-mon[105517]: osdmap e537: 8 total, 8 up, 8 in 2024-08-17T17:19:38.868 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.SnapshotCopyupZerosMigration (1005 ms) 2024-08-17T17:19:38.869 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.ResizeCopyup 2024-08-17T17:19:39.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:38 smithi136 ceph-mon[115078]: osdmap e538: 8 total, 8 up, 8 in 2024-08-17T17:19:39.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:38 smithi136 ceph-mon[115078]: pgmap v302: 105 pgs: 105 active+clean; 586 KiB data, 475 MiB used, 715 GiB / 715 GiB avail; 255 KiB/s rd, 30 KiB/s wr, 359 op/s 2024-08-17T17:19:39.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:38 smithi136 ceph-mon[109861]: osdmap e538: 8 total, 8 up, 8 in 2024-08-17T17:19:39.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:38 smithi136 ceph-mon[109861]: pgmap v302: 105 pgs: 105 active+clean; 586 KiB data, 475 MiB used, 715 GiB / 715 GiB avail; 255 KiB/s rd, 30 KiB/s wr, 359 op/s 2024-08-17T17:19:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:38 smithi165 ceph-mon[105517]: osdmap e538: 8 total, 8 up, 8 in 2024-08-17T17:19:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:38 smithi165 ceph-mon[105517]: pgmap v302: 105 pgs: 105 active+clean; 586 KiB data, 475 MiB used, 715 GiB / 715 GiB avail; 255 KiB/s rd, 30 KiB/s wr, 359 op/s 2024-08-17T17:19:40.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:39 smithi136 ceph-mon[109861]: osdmap e539: 8 total, 8 up, 8 in 2024-08-17T17:19:40.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:39 smithi136 ceph-mon[115078]: osdmap e539: 8 total, 8 up, 8 in 2024-08-17T17:19:40.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:39 smithi165 ceph-mon[105517]: osdmap e539: 8 total, 8 up, 8 in 2024-08-17T17:19:40.906 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.ResizeCopyup (2037 ms) 2024-08-17T17:19:40.906 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.DiscardCopyup 2024-08-17T17:19:40.915 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_internal.cc:1056: Skipped 2024-08-17T17:19:40.915 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:19:40.915 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestInternal.DiscardCopyup (9 ms) 2024-08-17T17:19:40.915 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.ImageOptions 2024-08-17T17:19:40.922 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.ImageOptions (8 ms) 2024-08-17T17:19:40.922 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.WriteFullCopyup 2024-08-17T17:19:41.118 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:40 smithi136 ceph-mon[109861]: osdmap e540: 8 total, 8 up, 8 in 2024-08-17T17:19:41.118 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:40 smithi136 ceph-mon[109861]: pgmap v305: 105 pgs: 105 active+clean; 586 KiB data, 475 MiB used, 715 GiB / 715 GiB avail; 255 KiB/s rd, 30 KiB/s wr, 359 op/s 2024-08-17T17:19:41.118 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:40 smithi136 ceph-mon[115078]: osdmap e540: 8 total, 8 up, 8 in 2024-08-17T17:19:41.118 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:40 smithi136 ceph-mon[115078]: pgmap v305: 105 pgs: 105 active+clean; 586 KiB data, 475 MiB used, 715 GiB / 715 GiB avail; 255 KiB/s rd, 30 KiB/s wr, 359 op/s 2024-08-17T17:19:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:40 smithi165 ceph-mon[105517]: osdmap e540: 8 total, 8 up, 8 in 2024-08-17T17:19:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:40 smithi165 ceph-mon[105517]: pgmap v305: 105 pgs: 105 active+clean; 586 KiB data, 475 MiB used, 715 GiB / 715 GiB avail; 255 KiB/s rd, 30 KiB/s wr, 359 op/s 2024-08-17T17:19:42.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:41 smithi136 ceph-mon[109861]: osdmap e541: 8 total, 8 up, 8 in 2024-08-17T17:19:42.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:41 smithi136 ceph-mon[115078]: osdmap e541: 8 total, 8 up, 8 in 2024-08-17T17:19:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:41 smithi165 ceph-mon[105517]: osdmap e541: 8 total, 8 up, 8 in 2024-08-17T17:19:43.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:42 smithi136 ceph-mon[109861]: pgmap v308: 105 pgs: 105 active+clean; 4.6 MiB data, 488 MiB used, 715 GiB / 715 GiB avail; 228 KiB/s rd, 1.0 MiB/s wr, 316 op/s 2024-08-17T17:19:43.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:42 smithi136 ceph-mon[109861]: osdmap e542: 8 total, 8 up, 8 in 2024-08-17T17:19:43.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:42 smithi136 ceph-mon[115078]: pgmap v308: 105 pgs: 105 active+clean; 4.6 MiB data, 488 MiB used, 715 GiB / 715 GiB avail; 228 KiB/s rd, 1.0 MiB/s wr, 316 op/s 2024-08-17T17:19:43.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:42 smithi136 ceph-mon[115078]: osdmap e542: 8 total, 8 up, 8 in 2024-08-17T17:19:43.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:42 smithi165 ceph-mon[105517]: pgmap v308: 105 pgs: 105 active+clean; 4.6 MiB data, 488 MiB used, 715 GiB / 715 GiB avail; 228 KiB/s rd, 1.0 MiB/s wr, 316 op/s 2024-08-17T17:19:43.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:42 smithi165 ceph-mon[105517]: osdmap e542: 8 total, 8 up, 8 in 2024-08-17T17:19:44.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:43 smithi136 ceph-mon[109861]: osdmap e543: 8 total, 8 up, 8 in 2024-08-17T17:19:44.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:43 smithi136 ceph-mon[115078]: osdmap e543: 8 total, 8 up, 8 in 2024-08-17T17:19:44.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:43 smithi165 ceph-mon[105517]: osdmap e543: 8 total, 8 up, 8 in 2024-08-17T17:19:44.895 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.WriteFullCopyup (3972 ms) 2024-08-17T17:19:44.895 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.DiffIterateCloneOverwrite 2024-08-17T17:19:45.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:44 smithi136 ceph-mon[109861]: pgmap v310: 105 pgs: 105 active+clean; 4.6 MiB data, 488 MiB used, 715 GiB / 715 GiB avail; 228 KiB/s rd, 1.0 MiB/s wr, 315 op/s 2024-08-17T17:19:45.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:44 smithi136 ceph-mon[109861]: osdmap e544: 8 total, 8 up, 8 in 2024-08-17T17:19:45.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:44 smithi136 ceph-mon[115078]: pgmap v310: 105 pgs: 105 active+clean; 4.6 MiB data, 488 MiB used, 715 GiB / 715 GiB avail; 228 KiB/s rd, 1.0 MiB/s wr, 315 op/s 2024-08-17T17:19:45.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:44 smithi136 ceph-mon[115078]: osdmap e544: 8 total, 8 up, 8 in 2024-08-17T17:19:45.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:44 smithi165 ceph-mon[105517]: pgmap v310: 105 pgs: 105 active+clean; 4.6 MiB data, 488 MiB used, 715 GiB / 715 GiB avail; 228 KiB/s rd, 1.0 MiB/s wr, 315 op/s 2024-08-17T17:19:45.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:44 smithi165 ceph-mon[105517]: osdmap e544: 8 total, 8 up, 8 in 2024-08-17T17:19:46.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:45 smithi136 ceph-mon[109861]: osdmap e545: 8 total, 8 up, 8 in 2024-08-17T17:19:46.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:45 smithi136 ceph-mon[115078]: osdmap e545: 8 total, 8 up, 8 in 2024-08-17T17:19:46.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:45 smithi165 ceph-mon[105517]: osdmap e545: 8 total, 8 up, 8 in 2024-08-17T17:19:46.918 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.DiffIterateCloneOverwrite (2023 ms) 2024-08-17T17:19:46.918 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.TestCoR 2024-08-17T17:19:46.929 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_internal.cc:1313: Skipped 2024-08-17T17:19:46.929 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to disabled copy-on-read 2024-08-17T17:19:46.929 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestInternal.TestCoR (11 ms) 2024-08-17T17:19:46.929 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.FlattenNoEmptyObjects 2024-08-17T17:19:47.132 INFO:tasks.workunit.client.0.smithi136.stdout:made parent image "image397": 4194304K (1024 * 4K) 2024-08-17T17:19:47.132 INFO:tasks.workunit.client.0.smithi136.stdout:generated random write map: 2024-08-17T17:19:47.132 INFO:tasks.workunit.client.0.smithi136.stdout: [8 , 34217 ] 2024-08-17T17:19:47.132 INFO:tasks.workunit.client.0.smithi136.stdout: [32 , 133747 ] 2024-08-17T17:19:47.133 INFO:tasks.workunit.client.0.smithi136.stdout: [42 , 175310 ] 2024-08-17T17:19:47.133 INFO:tasks.workunit.client.0.smithi136.stdout: [51 , 210303 ] 2024-08-17T17:19:47.133 INFO:tasks.workunit.client.0.smithi136.stdout: [60 , 247276 ] 2024-08-17T17:19:47.133 INFO:tasks.workunit.client.0.smithi136.stdout: [61 , 252392 ] 2024-08-17T17:19:47.133 INFO:tasks.workunit.client.0.smithi136.stdout: [74 , 305499 ] 2024-08-17T17:19:47.133 INFO:tasks.workunit.client.0.smithi136.stdout: [80 , 328280 ] 2024-08-17T17:19:47.133 INFO:tasks.workunit.client.0.smithi136.stdout: [84 , 344156 ] 2024-08-17T17:19:47.133 INFO:tasks.workunit.client.0.smithi136.stdout: [85 , 349472 ] 2024-08-17T17:19:47.133 INFO:tasks.workunit.client.0.smithi136.stdout: [91 , 374044 ] 2024-08-17T17:19:47.133 INFO:tasks.workunit.client.0.smithi136.stdout: [100 , 410067 ] 2024-08-17T17:19:47.133 INFO:tasks.workunit.client.0.smithi136.stdout: [101 , 414440 ] 2024-08-17T17:19:47.133 INFO:tasks.workunit.client.0.smithi136.stdout: [133 , 545208 ] 2024-08-17T17:19:47.134 INFO:tasks.workunit.client.0.smithi136.stdout: [134 , 550551 ] 2024-08-17T17:19:47.134 INFO:tasks.workunit.client.0.smithi136.stdout: [151 , 621447 ] 2024-08-17T17:19:47.134 INFO:tasks.workunit.client.0.smithi136.stdout: [160 , 657219 ] 2024-08-17T17:19:47.134 INFO:tasks.workunit.client.0.smithi136.stdout: [176 , 721434 ] 2024-08-17T17:19:47.134 INFO:tasks.workunit.client.0.smithi136.stdout: [177 , 727476 ] 2024-08-17T17:19:47.134 INFO:tasks.workunit.client.0.smithi136.stdout: [182 , 745700 ] 2024-08-17T17:19:47.134 INFO:tasks.workunit.client.0.smithi136.stdout: [190 , 778615 ] 2024-08-17T17:19:47.134 INFO:tasks.workunit.client.0.smithi136.stdout: [219 , 898454 ] 2024-08-17T17:19:47.134 INFO:tasks.workunit.client.0.smithi136.stdout: [234 , 959355 ] 2024-08-17T17:19:47.134 INFO:tasks.workunit.client.0.smithi136.stdout: [251 , 1030130 ] 2024-08-17T17:19:47.134 INFO:tasks.workunit.client.0.smithi136.stdout: [257 , 1056079 ] 2024-08-17T17:19:47.134 INFO:tasks.workunit.client.0.smithi136.stdout: [258 , 1057614 ] 2024-08-17T17:19:47.134 INFO:tasks.workunit.client.0.smithi136.stdout: [293 , 1200521 ] 2024-08-17T17:19:47.135 INFO:tasks.workunit.client.0.smithi136.stdout: [305 , 1251489 ] 2024-08-17T17:19:47.135 INFO:tasks.workunit.client.0.smithi136.stdout: [314 , 1286808 ] 2024-08-17T17:19:47.135 INFO:tasks.workunit.client.0.smithi136.stdout: [316 , 1296663 ] 2024-08-17T17:19:47.135 INFO:tasks.workunit.client.0.smithi136.stdout: [334 , 1369691 ] 2024-08-17T17:19:47.135 INFO:tasks.workunit.client.0.smithi136.stdout: [336 , 1378691 ] 2024-08-17T17:19:47.135 INFO:tasks.workunit.client.0.smithi136.stdout: [347 , 1421841 ] 2024-08-17T17:19:47.135 INFO:tasks.workunit.client.0.smithi136.stdout: [361 , 1482039 ] 2024-08-17T17:19:47.135 INFO:tasks.workunit.client.0.smithi136.stdout: [370 , 1518406 ] 2024-08-17T17:19:47.135 INFO:tasks.workunit.client.0.smithi136.stdout: [374 , 1531990 ] 2024-08-17T17:19:47.135 INFO:tasks.workunit.client.0.smithi136.stdout: [385 , 1580000 ] 2024-08-17T17:19:47.135 INFO:tasks.workunit.client.0.smithi136.stdout: [391 , 1604759 ] 2024-08-17T17:19:47.135 INFO:tasks.workunit.client.0.smithi136.stdout: [392 , 1608776 ] 2024-08-17T17:19:47.135 INFO:tasks.workunit.client.0.smithi136.stdout: [408 , 1671638 ] 2024-08-17T17:19:47.136 INFO:tasks.workunit.client.0.smithi136.stdout: [409 , 1678609 ] 2024-08-17T17:19:47.136 INFO:tasks.workunit.client.0.smithi136.stdout: [413 , 1694837 ] 2024-08-17T17:19:47.136 INFO:tasks.workunit.client.0.smithi136.stdout: [419 , 1717010 ] 2024-08-17T17:19:47.136 INFO:tasks.workunit.client.0.smithi136.stdout: [436 , 1787127 ] 2024-08-17T17:19:47.136 INFO:tasks.workunit.client.0.smithi136.stdout: [438 , 1795857 ] 2024-08-17T17:19:47.136 INFO:tasks.workunit.client.0.smithi136.stdout: [441 , 1808106 ] 2024-08-17T17:19:47.136 INFO:tasks.workunit.client.0.smithi136.stdout: [443 , 1814683 ] 2024-08-17T17:19:47.136 INFO:tasks.workunit.client.0.smithi136.stdout: [449 , 1839676 ] 2024-08-17T17:19:47.136 INFO:tasks.workunit.client.0.smithi136.stdout: [450 , 1845938 ] 2024-08-17T17:19:47.136 INFO:tasks.workunit.client.0.smithi136.stdout: [451 , 1850187 ] 2024-08-17T17:19:47.136 INFO:tasks.workunit.client.0.smithi136.stdout: [462 , 1894596 ] 2024-08-17T17:19:47.136 INFO:tasks.workunit.client.0.smithi136.stdout: [463 , 1896721 ] 2024-08-17T17:19:47.137 INFO:tasks.workunit.client.0.smithi136.stdout: [481 , 1971968 ] 2024-08-17T17:19:47.137 INFO:tasks.workunit.client.0.smithi136.stdout: [483 , 1980654 ] 2024-08-17T17:19:47.137 INFO:tasks.workunit.client.0.smithi136.stdout: [496 , 2032559 ] 2024-08-17T17:19:47.137 INFO:tasks.workunit.client.0.smithi136.stdout: [497 , 2038665 ] 2024-08-17T17:19:47.137 INFO:tasks.workunit.client.0.smithi136.stdout: [506 , 2072972 ] 2024-08-17T17:19:47.137 INFO:tasks.workunit.client.0.smithi136.stdout: [511 , 2094380 ] 2024-08-17T17:19:47.137 INFO:tasks.workunit.client.0.smithi136.stdout: [512 , 2097252 ] 2024-08-17T17:19:47.137 INFO:tasks.workunit.client.0.smithi136.stdout: [550 , 2256373 ] 2024-08-17T17:19:47.137 INFO:tasks.workunit.client.0.smithi136.stdout: [575 , 2356078 ] 2024-08-17T17:19:47.137 INFO:tasks.workunit.client.0.smithi136.stdout: [580 , 2378750 ] 2024-08-17T17:19:47.137 INFO:tasks.workunit.client.0.smithi136.stdout: [595 , 2437808 ] 2024-08-17T17:19:47.137 INFO:tasks.workunit.client.0.smithi136.stdout: [599 , 2453895 ] 2024-08-17T17:19:47.137 INFO:tasks.workunit.client.0.smithi136.stdout: [601 , 2463257 ] 2024-08-17T17:19:47.138 INFO:tasks.workunit.client.0.smithi136.stdout: [621 , 2544507 ] 2024-08-17T17:19:47.138 INFO:tasks.workunit.client.0.smithi136.stdout: [637 , 2610894 ] 2024-08-17T17:19:47.138 INFO:tasks.workunit.client.0.smithi136.stdout: [639 , 2620166 ] 2024-08-17T17:19:47.138 INFO:tasks.workunit.client.0.smithi136.stdout: [642 , 2632886 ] 2024-08-17T17:19:47.138 INFO:tasks.workunit.client.0.smithi136.stdout: [643 , 2636416 ] 2024-08-17T17:19:47.138 INFO:tasks.workunit.client.0.smithi136.stdout: [646 , 2647987 ] 2024-08-17T17:19:47.138 INFO:tasks.workunit.client.0.smithi136.stdout: [654 , 2681197 ] 2024-08-17T17:19:47.138 INFO:tasks.workunit.client.0.smithi136.stdout: [655 , 2683149 ] 2024-08-17T17:19:47.138 INFO:tasks.workunit.client.0.smithi136.stdout: [715 , 2930054 ] 2024-08-17T17:19:47.138 INFO:tasks.workunit.client.0.smithi136.stdout: [718 , 2940942 ] 2024-08-17T17:19:47.138 INFO:tasks.workunit.client.0.smithi136.stdout: [723 , 2964361 ] 2024-08-17T17:19:47.138 INFO:tasks.workunit.client.0.smithi136.stdout: [725 , 2970322 ] 2024-08-17T17:19:47.138 INFO:tasks.workunit.client.0.smithi136.stdout: [758 , 3106423 ] 2024-08-17T17:19:47.138 INFO:tasks.workunit.client.0.smithi136.stdout: [763 , 3127987 ] 2024-08-17T17:19:47.139 INFO:tasks.workunit.client.0.smithi136.stdout: [802 , 3285458 ] 2024-08-17T17:19:47.139 INFO:tasks.workunit.client.0.smithi136.stdout: [816 , 3344814 ] 2024-08-17T17:19:47.139 INFO:tasks.workunit.client.0.smithi136.stdout: [817 , 3348620 ] 2024-08-17T17:19:47.139 INFO:tasks.workunit.client.0.smithi136.stdout: [823 , 3374412 ] 2024-08-17T17:19:47.139 INFO:tasks.workunit.client.0.smithi136.stdout: [860 , 3525339 ] 2024-08-17T17:19:47.139 INFO:tasks.workunit.client.0.smithi136.stdout: [874 , 3581373 ] 2024-08-17T17:19:47.139 INFO:tasks.workunit.client.0.smithi136.stdout: [881 , 3610472 ] 2024-08-17T17:19:47.139 INFO:tasks.workunit.client.0.smithi136.stdout: [897 , 3674220 ] 2024-08-17T17:19:47.139 INFO:tasks.workunit.client.0.smithi136.stdout: [900 , 3689532 ] 2024-08-17T17:19:47.139 INFO:tasks.workunit.client.0.smithi136.stdout: [905 , 3707576 ] 2024-08-17T17:19:47.139 INFO:tasks.workunit.client.0.smithi136.stdout: [917 , 3757170 ] 2024-08-17T17:19:47.139 INFO:tasks.workunit.client.0.smithi136.stdout: [937 , 3839042 ] 2024-08-17T17:19:47.139 INFO:tasks.workunit.client.0.smithi136.stdout: [985 , 4037912 ] 2024-08-17T17:19:47.140 INFO:tasks.workunit.client.0.smithi136.stdout: [987 , 4046311 ] 2024-08-17T17:19:47.140 INFO:tasks.workunit.client.0.smithi136.stdout: [998 , 4090810 ] 2024-08-17T17:19:47.140 INFO:tasks.workunit.client.0.smithi136.stdout: [1004 , 4112400 ] 2024-08-17T17:19:47.140 INFO:tasks.workunit.client.0.smithi136.stdout: [1016 , 4163970 ] 2024-08-17T17:19:47.140 INFO:tasks.workunit.client.0.smithi136.stdout: [1020 , 4179343 ] 2024-08-17T17:19:47.140 INFO:tasks.workunit.client.0.smithi136.stdout: [1022 , 4186174 ] 2024-08-17T17:19:47.140 INFO:tasks.workunit.client.0.smithi136.stdout:write data based on random map 2024-08-17T17:19:47.140 INFO:tasks.workunit.client.0.smithi136.stdout: write object-8 2024-08-17T17:19:47.140 INFO:tasks.workunit.client.0.smithi136.stdout: write object-32 2024-08-17T17:19:47.140 INFO:tasks.workunit.client.0.smithi136.stdout: write object-42 2024-08-17T17:19:47.140 INFO:tasks.workunit.client.0.smithi136.stdout: write object-51 2024-08-17T17:19:47.140 INFO:tasks.workunit.client.0.smithi136.stdout: write object-60 2024-08-17T17:19:47.140 INFO:tasks.workunit.client.0.smithi136.stdout: write object-61 2024-08-17T17:19:47.140 INFO:tasks.workunit.client.0.smithi136.stdout: write object-74 2024-08-17T17:19:47.141 INFO:tasks.workunit.client.0.smithi136.stdout: write object-80 2024-08-17T17:19:47.141 INFO:tasks.workunit.client.0.smithi136.stdout: write object-84 2024-08-17T17:19:47.141 INFO:tasks.workunit.client.0.smithi136.stdout: write object-85 2024-08-17T17:19:47.141 INFO:tasks.workunit.client.0.smithi136.stdout: write object-91 2024-08-17T17:19:47.141 INFO:tasks.workunit.client.0.smithi136.stdout: write object-100 2024-08-17T17:19:47.141 INFO:tasks.workunit.client.0.smithi136.stdout: write object-101 2024-08-17T17:19:47.141 INFO:tasks.workunit.client.0.smithi136.stdout: write object-133 2024-08-17T17:19:47.141 INFO:tasks.workunit.client.0.smithi136.stdout: write object-134 2024-08-17T17:19:47.141 INFO:tasks.workunit.client.0.smithi136.stdout: write object-151 2024-08-17T17:19:47.141 INFO:tasks.workunit.client.0.smithi136.stdout: write object-160 2024-08-17T17:19:47.141 INFO:tasks.workunit.client.0.smithi136.stdout: write object-176 2024-08-17T17:19:47.141 INFO:tasks.workunit.client.0.smithi136.stdout: write object-177 2024-08-17T17:19:47.141 INFO:tasks.workunit.client.0.smithi136.stdout: write object-182 2024-08-17T17:19:47.141 INFO:tasks.workunit.client.0.smithi136.stdout: write object-190 2024-08-17T17:19:47.142 INFO:tasks.workunit.client.0.smithi136.stdout: write object-219 2024-08-17T17:19:47.142 INFO:tasks.workunit.client.0.smithi136.stdout: write object-234 2024-08-17T17:19:47.142 INFO:tasks.workunit.client.0.smithi136.stdout: write object-251 2024-08-17T17:19:47.142 INFO:tasks.workunit.client.0.smithi136.stdout: write object-257 2024-08-17T17:19:47.142 INFO:tasks.workunit.client.0.smithi136.stdout: write object-258 2024-08-17T17:19:47.142 INFO:tasks.workunit.client.0.smithi136.stdout: write object-293 2024-08-17T17:19:47.142 INFO:tasks.workunit.client.0.smithi136.stdout: write object-305 2024-08-17T17:19:47.142 INFO:tasks.workunit.client.0.smithi136.stdout: write object-314 2024-08-17T17:19:47.142 INFO:tasks.workunit.client.0.smithi136.stdout: write object-316 2024-08-17T17:19:47.142 INFO:tasks.workunit.client.0.smithi136.stdout: write object-334 2024-08-17T17:19:47.142 INFO:tasks.workunit.client.0.smithi136.stdout: write object-336 2024-08-17T17:19:47.142 INFO:tasks.workunit.client.0.smithi136.stdout: write object-347 2024-08-17T17:19:47.142 INFO:tasks.workunit.client.0.smithi136.stdout: write object-361 2024-08-17T17:19:47.142 INFO:tasks.workunit.client.0.smithi136.stdout: write object-370 2024-08-17T17:19:47.143 INFO:tasks.workunit.client.0.smithi136.stdout: write object-374 2024-08-17T17:19:47.143 INFO:tasks.workunit.client.0.smithi136.stdout: write object-385 2024-08-17T17:19:47.143 INFO:tasks.workunit.client.0.smithi136.stdout: write object-391 2024-08-17T17:19:47.143 INFO:tasks.workunit.client.0.smithi136.stdout: write object-392 2024-08-17T17:19:47.143 INFO:tasks.workunit.client.0.smithi136.stdout: write object-408 2024-08-17T17:19:47.143 INFO:tasks.workunit.client.0.smithi136.stdout: write object-409 2024-08-17T17:19:47.143 INFO:tasks.workunit.client.0.smithi136.stdout: write object-413 2024-08-17T17:19:47.143 INFO:tasks.workunit.client.0.smithi136.stdout: write object-419 2024-08-17T17:19:47.143 INFO:tasks.workunit.client.0.smithi136.stdout: write object-436 2024-08-17T17:19:47.143 INFO:tasks.workunit.client.0.smithi136.stdout: write object-438 2024-08-17T17:19:47.143 INFO:tasks.workunit.client.0.smithi136.stdout: write object-441 2024-08-17T17:19:47.143 INFO:tasks.workunit.client.0.smithi136.stdout: write object-443 2024-08-17T17:19:47.143 INFO:tasks.workunit.client.0.smithi136.stdout: write object-449 2024-08-17T17:19:47.143 INFO:tasks.workunit.client.0.smithi136.stdout: write object-450 2024-08-17T17:19:47.144 INFO:tasks.workunit.client.0.smithi136.stdout: write object-451 2024-08-17T17:19:47.144 INFO:tasks.workunit.client.0.smithi136.stdout: write object-462 2024-08-17T17:19:47.144 INFO:tasks.workunit.client.0.smithi136.stdout: write object-463 2024-08-17T17:19:47.144 INFO:tasks.workunit.client.0.smithi136.stdout: write object-481 2024-08-17T17:19:47.144 INFO:tasks.workunit.client.0.smithi136.stdout: write object-483 2024-08-17T17:19:47.144 INFO:tasks.workunit.client.0.smithi136.stdout: write object-496 2024-08-17T17:19:47.144 INFO:tasks.workunit.client.0.smithi136.stdout: write object-497 2024-08-17T17:19:47.144 INFO:tasks.workunit.client.0.smithi136.stdout: write object-506 2024-08-17T17:19:47.144 INFO:tasks.workunit.client.0.smithi136.stdout: write object-511 2024-08-17T17:19:47.144 INFO:tasks.workunit.client.0.smithi136.stdout: write object-512 2024-08-17T17:19:47.144 INFO:tasks.workunit.client.0.smithi136.stdout: write object-550 2024-08-17T17:19:47.144 INFO:tasks.workunit.client.0.smithi136.stdout: write object-575 2024-08-17T17:19:47.144 INFO:tasks.workunit.client.0.smithi136.stdout: write object-580 2024-08-17T17:19:47.144 INFO:tasks.workunit.client.0.smithi136.stdout: write object-595 2024-08-17T17:19:47.144 INFO:tasks.workunit.client.0.smithi136.stdout: write object-599 2024-08-17T17:19:47.144 INFO:tasks.workunit.client.0.smithi136.stdout: write object-601 2024-08-17T17:19:47.145 INFO:tasks.workunit.client.0.smithi136.stdout: write object-621 2024-08-17T17:19:47.145 INFO:tasks.workunit.client.0.smithi136.stdout: write object-637 2024-08-17T17:19:47.145 INFO:tasks.workunit.client.0.smithi136.stdout: write object-639 2024-08-17T17:19:47.145 INFO:tasks.workunit.client.0.smithi136.stdout: write object-642 2024-08-17T17:19:47.145 INFO:tasks.workunit.client.0.smithi136.stdout: write object-643 2024-08-17T17:19:47.145 INFO:tasks.workunit.client.0.smithi136.stdout: write object-646 2024-08-17T17:19:47.145 INFO:tasks.workunit.client.0.smithi136.stdout: write object-654 2024-08-17T17:19:47.145 INFO:tasks.workunit.client.0.smithi136.stdout: write object-655 2024-08-17T17:19:47.145 INFO:tasks.workunit.client.0.smithi136.stdout: write object-715 2024-08-17T17:19:47.145 INFO:tasks.workunit.client.0.smithi136.stdout: write object-718 2024-08-17T17:19:47.145 INFO:tasks.workunit.client.0.smithi136.stdout: write object-723 2024-08-17T17:19:47.145 INFO:tasks.workunit.client.0.smithi136.stdout: write object-725 2024-08-17T17:19:47.145 INFO:tasks.workunit.client.0.smithi136.stdout: write object-758 2024-08-17T17:19:47.145 INFO:tasks.workunit.client.0.smithi136.stdout: write object-763 2024-08-17T17:19:47.146 INFO:tasks.workunit.client.0.smithi136.stdout: write object-802 2024-08-17T17:19:47.146 INFO:tasks.workunit.client.0.smithi136.stdout: write object-816 2024-08-17T17:19:47.146 INFO:tasks.workunit.client.0.smithi136.stdout: write object-817 2024-08-17T17:19:47.146 INFO:tasks.workunit.client.0.smithi136.stdout: write object-823 2024-08-17T17:19:47.146 INFO:tasks.workunit.client.0.smithi136.stdout: write object-860 2024-08-17T17:19:47.146 INFO:tasks.workunit.client.0.smithi136.stdout: write object-874 2024-08-17T17:19:47.146 INFO:tasks.workunit.client.0.smithi136.stdout: write object-881 2024-08-17T17:19:47.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:46 smithi136 ceph-mon[109861]: pgmap v313: 105 pgs: 105 active+clean; 4.6 MiB data, 488 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:19:47.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:46 smithi136 ceph-mon[109861]: osdmap e546: 8 total, 8 up, 8 in 2024-08-17T17:19:47.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:46 smithi136 ceph-mon[115078]: pgmap v313: 105 pgs: 105 active+clean; 4.6 MiB data, 488 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:19:47.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:46 smithi136 ceph-mon[115078]: osdmap e546: 8 total, 8 up, 8 in 2024-08-17T17:19:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:46 smithi165 ceph-mon[105517]: pgmap v313: 105 pgs: 105 active+clean; 4.6 MiB data, 488 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:19:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:46 smithi165 ceph-mon[105517]: osdmap e546: 8 total, 8 up, 8 in 2024-08-17T17:19:47.897 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:19:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:19:47] "GET /metrics HTTP/1.1" 200 36279 "" "Prometheus/2.51.0" 2024-08-17T17:19:48.076 INFO:tasks.workunit.client.0.smithi136.stdout: write object-897 2024-08-17T17:19:48.076 INFO:tasks.workunit.client.0.smithi136.stdout: write object-900 2024-08-17T17:19:48.076 INFO:tasks.workunit.client.0.smithi136.stdout: write object-905 2024-08-17T17:19:48.076 INFO:tasks.workunit.client.0.smithi136.stdout: write object-917 2024-08-17T17:19:48.076 INFO:tasks.workunit.client.0.smithi136.stdout: write object-937 2024-08-17T17:19:48.076 INFO:tasks.workunit.client.0.smithi136.stdout: write object-985 2024-08-17T17:19:48.076 INFO:tasks.workunit.client.0.smithi136.stdout: write object-987 2024-08-17T17:19:48.076 INFO:tasks.workunit.client.0.smithi136.stdout: write object-998 2024-08-17T17:19:48.076 INFO:tasks.workunit.client.0.smithi136.stdout: write object-1004 2024-08-17T17:19:48.076 INFO:tasks.workunit.client.0.smithi136.stdout: write object-1016 2024-08-17T17:19:48.076 INFO:tasks.workunit.client.0.smithi136.stdout: write object-1020 2024-08-17T17:19:48.077 INFO:tasks.workunit.client.0.smithi136.stdout: write object-1022 2024-08-17T17:19:48.077 INFO:tasks.workunit.client.0.smithi136.stdout:verify written data by reading 2024-08-17T17:19:48.077 INFO:tasks.workunit.client.0.smithi136.stdout: read object-8 2024-08-17T17:19:48.077 INFO:tasks.workunit.client.0.smithi136.stdout:made snapshot "image397@parent_snap" and protect it 2024-08-17T17:19:48.077 INFO:tasks.workunit.client.0.smithi136.stdout:made and opened clone "image398" 2024-08-17T17:19:48.077 INFO:tasks.workunit.client.0.smithi136.stdout:flattening clone: "image398" 2024-08-17T17:19:48.077 INFO:tasks.workunit.client.0.smithi136.stdout:check whether child image has the same set of objects as parent 2024-08-17T17:19:48.077 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.FlattenNoEmptyObjects (1147 ms) 2024-08-17T17:19:48.077 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.PoolMetadataConfApply 2024-08-17T17:19:48.147 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.PoolMetadataConfApply (71 ms) 2024-08-17T17:19:48.147 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.Sparsify 2024-08-17T17:19:48.169 INFO:tasks.workunit.client.0.smithi136.stdout:sparsify_supported=1 2024-08-17T17:19:48.169 INFO:tasks.workunit.client.0.smithi136.stdout:sparse_read_supported=1 2024-08-17T17:19:48.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:47 smithi136 ceph-mon[109861]: osdmap e547: 8 total, 8 up, 8 in 2024-08-17T17:19:48.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:47 smithi136 ceph-mon[115078]: osdmap e547: 8 total, 8 up, 8 in 2024-08-17T17:19:48.196 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:19:48 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:19:48] ENGINE Bus STOPPING 2024-08-17T17:19:48.197 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.Sparsify (50 ms) 2024-08-17T17:19:48.197 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.SparsifyClone 2024-08-17T17:19:48.213 INFO:tasks.workunit.client.0.smithi136.stdout:sparsify_supported=1 2024-08-17T17:19:48.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:47 smithi165 ceph-mon[105517]: osdmap e547: 8 total, 8 up, 8 in 2024-08-17T17:19:48.897 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:19:48 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:19:48] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-08-17T17:19:48.897 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:19:48 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:19:48] ENGINE Bus STOPPED 2024-08-17T17:19:48.897 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:19:48 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:19:48] ENGINE Bus STARTING 2024-08-17T17:19:48.897 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:19:48 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:19:48] ENGINE Serving on http://:::9283 2024-08-17T17:19:48.897 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:19:48 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:19:48] ENGINE Bus STARTED 2024-08-17T17:19:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:48 smithi165 ceph-mon[105517]: pgmap v316: 105 pgs: 105 active+clean; 4.6 MiB data, 488 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.0 MiB/s wr, 219 op/s 2024-08-17T17:19:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:48 smithi165 ceph-mon[105517]: osdmap e548: 8 total, 8 up, 8 in 2024-08-17T17:19:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:48 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3699063369' entity='client.admin' 2024-08-17T17:19:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:48 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:19:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:48 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-08-17T17:19:49.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:48 smithi165 ceph-mon[105517]: osdmap e549: 8 total, 8 up, 8 in 2024-08-17T17:19:49.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:48 smithi136 ceph-mon[109861]: pgmap v316: 105 pgs: 105 active+clean; 4.6 MiB data, 488 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.0 MiB/s wr, 219 op/s 2024-08-17T17:19:49.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:48 smithi136 ceph-mon[109861]: osdmap e548: 8 total, 8 up, 8 in 2024-08-17T17:19:49.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:48 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3699063369' entity='client.admin' 2024-08-17T17:19:49.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:48 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:19:49.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:48 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-08-17T17:19:49.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:48 smithi136 ceph-mon[109861]: osdmap e549: 8 total, 8 up, 8 in 2024-08-17T17:19:49.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:48 smithi136 ceph-mon[115078]: pgmap v316: 105 pgs: 105 active+clean; 4.6 MiB data, 488 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.0 MiB/s wr, 219 op/s 2024-08-17T17:19:49.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:48 smithi136 ceph-mon[115078]: osdmap e548: 8 total, 8 up, 8 in 2024-08-17T17:19:49.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:48 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3699063369' entity='client.admin' 2024-08-17T17:19:49.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:48 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:19:49.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:48 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-08-17T17:19:49.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:48 smithi136 ceph-mon[115078]: osdmap e549: 8 total, 8 up, 8 in 2024-08-17T17:19:49.926 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.SparsifyClone (1729 ms) 2024-08-17T17:19:49.926 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestInternal.MissingDataPool 2024-08-17T17:19:50.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:49 smithi136 ceph-mon[109861]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:19:50.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:49 smithi136 ceph-mon[115078]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:19:50.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:49 smithi165 ceph-mon[105517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:19:51.018 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestInternal.MissingDataPool (1092 ms) 2024-08-17T17:19:51.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:50 smithi136 ceph-mon[115078]: pgmap v319: 105 pgs: 105 active+clean; 4.6 MiB data, 488 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.0 MiB/s wr, 219 op/s 2024-08-17T17:19:51.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:50 smithi136 ceph-mon[115078]: osdmap e550: 8 total, 8 up, 8 in 2024-08-17T17:19:51.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:50 smithi136 ceph-mon[109861]: pgmap v319: 105 pgs: 105 active+clean; 4.6 MiB data, 488 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.0 MiB/s wr, 219 op/s 2024-08-17T17:19:51.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:50 smithi136 ceph-mon[109861]: osdmap e550: 8 total, 8 up, 8 in 2024-08-17T17:19:51.230 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:50 smithi165 ceph-mon[105517]: pgmap v319: 105 pgs: 105 active+clean; 4.6 MiB data, 488 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.0 MiB/s wr, 219 op/s 2024-08-17T17:19:51.230 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:50 smithi165 ceph-mon[105517]: osdmap e550: 8 total, 8 up, 8 in 2024-08-17T17:19:51.923 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 34 tests from TestInternal (35208 ms total) 2024-08-17T17:19:51.923 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:19:51.923 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 17 tests from TestImageWatcher 2024-08-17T17:19:52.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:51 smithi136 ceph-mon[115078]: osdmap e551: 8 total, 8 up, 8 in 2024-08-17T17:19:52.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:19:52.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:51 smithi136 ceph-mon[109861]: osdmap e551: 8 total, 8 up, 8 in 2024-08-17T17:19:52.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:19:52.197 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:51 smithi165 ceph-mon[105517]: osdmap e551: 8 total, 8 up, 8 in 2024-08-17T17:19:52.197 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:19:53.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:52 smithi136 ceph-mon[109861]: pgmap v322: 105 pgs: 105 active+clean; 5.0 MiB data, 500 MiB used, 715 GiB / 715 GiB avail; 609 KiB/s rd, 196 KiB/s wr, 961 op/s 2024-08-17T17:19:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:52 smithi136 ceph-mon[109861]: osdmap e552: 8 total, 8 up, 8 in 2024-08-17T17:19:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:19:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:19:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:19:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:19:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:19:53.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:52 smithi136 ceph-mon[115078]: pgmap v322: 105 pgs: 105 active+clean; 5.0 MiB data, 500 MiB used, 715 GiB / 715 GiB avail; 609 KiB/s rd, 196 KiB/s wr, 961 op/s 2024-08-17T17:19:53.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:52 smithi136 ceph-mon[115078]: osdmap e552: 8 total, 8 up, 8 in 2024-08-17T17:19:53.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:19:53.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:19:53.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:19:53.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:19:53.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:19:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:52 smithi165 ceph-mon[105517]: pgmap v322: 105 pgs: 105 active+clean; 5.0 MiB data, 500 MiB used, 715 GiB / 715 GiB avail; 609 KiB/s rd, 196 KiB/s wr, 961 op/s 2024-08-17T17:19:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:52 smithi165 ceph-mon[105517]: osdmap e552: 8 total, 8 up, 8 in 2024-08-17T17:19:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:19:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:19:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:19:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:19:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:19:53.935 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestImageWatcher.NotifyHeaderUpdate 2024-08-17T17:19:54.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:53 smithi136 ceph-mon[109861]: osdmap e553: 8 total, 8 up, 8 in 2024-08-17T17:19:54.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:53 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/915986418' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:19:54.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:53 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:19:54.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:53 smithi136 ceph-mon[115078]: osdmap e553: 8 total, 8 up, 8 in 2024-08-17T17:19:54.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:53 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/915986418' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:19:54.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:53 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:19:54.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:53 smithi165 ceph-mon[105517]: osdmap e553: 8 total, 8 up, 8 in 2024-08-17T17:19:54.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:53 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/915986418' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:19:54.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:53 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:19:55.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:54 smithi136 ceph-mon[109861]: pgmap v325: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 500 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:19:55.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:54 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:19:55.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:54 smithi136 ceph-mon[109861]: osdmap e554: 8 total, 8 up, 8 in 2024-08-17T17:19:55.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:54 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:19:55.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:54 smithi136 ceph-mon[115078]: pgmap v325: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 500 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:19:55.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:54 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:19:55.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:54 smithi136 ceph-mon[115078]: osdmap e554: 8 total, 8 up, 8 in 2024-08-17T17:19:55.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:54 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:19:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:54 smithi165 ceph-mon[105517]: pgmap v325: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 500 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:19:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:54 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:19:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:54 smithi165 ceph-mon[105517]: osdmap e554: 8 total, 8 up, 8 in 2024-08-17T17:19:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:54 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:19:55.975 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestImageWatcher.NotifyHeaderUpdate (2040 ms) 2024-08-17T17:19:55.976 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestImageWatcher.NotifyFlatten 2024-08-17T17:19:56.000 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestImageWatcher.NotifyFlatten (25 ms) 2024-08-17T17:19:56.000 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestImageWatcher.NotifyResize 2024-08-17T17:19:56.022 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestImageWatcher.NotifyResize (22 ms) 2024-08-17T17:19:56.022 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestImageWatcher.NotifyRebuildObjectMap 2024-08-17T17:19:56.044 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestImageWatcher.NotifyRebuildObjectMap (22 ms) 2024-08-17T17:19:56.044 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestImageWatcher.NotifyUpdateFeatures 2024-08-17T17:19:56.065 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestImageWatcher.NotifyUpdateFeatures (22 ms) 2024-08-17T17:19:56.066 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestImageWatcher.NotifySnapCreate 2024-08-17T17:19:56.086 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestImageWatcher.NotifySnapCreate (20 ms) 2024-08-17T17:19:56.086 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestImageWatcher.NotifySnapCreateError 2024-08-17T17:19:56.108 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestImageWatcher.NotifySnapCreateError (22 ms) 2024-08-17T17:19:56.108 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestImageWatcher.NotifySnapRename 2024-08-17T17:19:56.127 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestImageWatcher.NotifySnapRename (19 ms) 2024-08-17T17:19:56.127 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestImageWatcher.NotifySnapRenameError 2024-08-17T17:19:56.146 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestImageWatcher.NotifySnapRenameError (19 ms) 2024-08-17T17:19:56.147 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestImageWatcher.NotifySnapRemove 2024-08-17T17:19:56.166 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestImageWatcher.NotifySnapRemove (19 ms) 2024-08-17T17:19:56.166 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestImageWatcher.NotifySnapProtect 2024-08-17T17:19:56.185 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestImageWatcher.NotifySnapProtect (19 ms) 2024-08-17T17:19:56.185 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestImageWatcher.NotifySnapUnprotect 2024-08-17T17:19:56.204 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestImageWatcher.NotifySnapUnprotect (19 ms) 2024-08-17T17:19:56.204 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestImageWatcher.NotifyRename 2024-08-17T17:19:56.226 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestImageWatcher.NotifyRename (19 ms) 2024-08-17T17:19:56.226 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestImageWatcher.NotifyAsyncTimedOut 2024-08-17T17:19:56.241 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestImageWatcher.NotifyAsyncTimedOut (18 ms) 2024-08-17T17:19:56.241 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestImageWatcher.NotifyAsyncError 2024-08-17T17:19:56.259 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestImageWatcher.NotifyAsyncError (19 ms) 2024-08-17T17:19:56.260 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestImageWatcher.NotifyAsyncCompleteError 2024-08-17T17:19:56.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:55 smithi165 ceph-mon[105517]: osdmap e555: 8 total, 8 up, 8 in 2024-08-17T17:19:56.279 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestImageWatcher.NotifyAsyncCompleteError (19 ms) 2024-08-17T17:19:56.279 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestImageWatcher.NotifyAsyncRequestTimedOut 2024-08-17T17:19:56.297 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestImageWatcher.NotifyAsyncRequestTimedOut (18 ms) 2024-08-17T17:19:56.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:55 smithi136 ceph-mon[109861]: osdmap e555: 8 total, 8 up, 8 in 2024-08-17T17:19:56.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:55 smithi136 ceph-mon[115078]: osdmap e555: 8 total, 8 up, 8 in 2024-08-17T17:19:56.955 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 17 tests from TestImageWatcher (2362 ms total) 2024-08-17T17:19:56.955 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:19:56.955 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 170 tests from TestLibRBD 2024-08-17T17:19:56.984 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.CreateAndStat 2024-08-17T17:19:57.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:56 smithi165 ceph-mon[105517]: pgmap v328: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 500 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:19:57.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:56 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:19:57.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:56 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T17:19:57.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:56 smithi165 ceph-mon[105517]: osdmap e556: 8 total, 8 up, 8 in 2024-08-17T17:19:57.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:56 smithi136 ceph-mon[109861]: pgmap v328: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 500 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:19:57.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:56 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:19:57.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:56 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T17:19:57.452 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:56 smithi136 ceph-mon[109861]: osdmap e556: 8 total, 8 up, 8 in 2024-08-17T17:19:57.452 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:56 smithi136 ceph-mon[115078]: pgmap v328: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 500 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:19:57.452 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:56 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:19:57.452 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:56 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T17:19:57.453 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:56 smithi136 ceph-mon[115078]: osdmap e556: 8 total, 8 up, 8 in 2024-08-17T17:19:57.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:19:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:19:57] "GET /metrics HTTP/1.1" 200 36292 "" "Prometheus/2.51.0" 2024-08-17T17:19:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:57 smithi165 ceph-mon[105517]: osdmap e557: 8 total, 8 up, 8 in 2024-08-17T17:19:58.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:57 smithi136 ceph-mon[109861]: osdmap e557: 8 total, 8 up, 8 in 2024-08-17T17:19:58.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:57 smithi136 ceph-mon[115078]: osdmap e557: 8 total, 8 up, 8 in 2024-08-17T17:19:58.981 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:19:59.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:58 smithi165 ceph-mon[105517]: pgmap v331: 73 pgs: 73 active+clean; 582 KiB data, 469 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:19:59.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:58 smithi165 ceph-mon[105517]: osdmap e558: 8 total, 8 up, 8 in 2024-08-17T17:19:59.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:58 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/774601202' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:19:59.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:58 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:19:59.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:58 smithi136 ceph-mon[109861]: pgmap v331: 73 pgs: 73 active+clean; 582 KiB data, 469 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:19:59.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:58 smithi136 ceph-mon[109861]: osdmap e558: 8 total, 8 up, 8 in 2024-08-17T17:19:59.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:58 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/774601202' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:19:59.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:58 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:19:59.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:58 smithi136 ceph-mon[115078]: pgmap v331: 73 pgs: 73 active+clean; 582 KiB data, 469 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:19:59.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:58 smithi136 ceph-mon[115078]: osdmap e558: 8 total, 8 up, 8 in 2024-08-17T17:19:59.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:58 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/774601202' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:19:59.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:58 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:00.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:59 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:00.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:19:59 smithi165 ceph-mon[105517]: osdmap e559: 8 total, 8 up, 8 in 2024-08-17T17:20:00.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:59 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:00.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:19:59 smithi136 ceph-mon[109861]: osdmap e559: 8 total, 8 up, 8 in 2024-08-17T17:20:00.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:59 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:00.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:19:59 smithi136 ceph-mon[115078]: osdmap e559: 8 total, 8 up, 8 in 2024-08-17T17:20:01.007 INFO:tasks.workunit.client.0.smithi136.stdout:image has size 2097152 and order 22 2024-08-17T17:20:01.007 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.CreateAndStat (4023 ms) 2024-08-17T17:20:01.007 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.CreateWithSameDataPool 2024-08-17T17:20:01.007 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:01.027 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.CreateWithSameDataPool (20 ms) 2024-08-17T17:20:01.027 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.CreateAndStatPP 2024-08-17T17:20:01.028 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:01.049 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.CreateAndStatPP (22 ms) 2024-08-17T17:20:01.049 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.GetId 2024-08-17T17:20:01.049 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:01.074 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.GetId (25 ms) 2024-08-17T17:20:01.074 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.GetIdPP 2024-08-17T17:20:01.074 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:01.097 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.GetIdPP (23 ms) 2024-08-17T17:20:01.097 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefix 2024-08-17T17:20:01.097 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:01.115 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.GetBlockNamePrefix (18 ms) 2024-08-17T17:20:01.115 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefixPP 2024-08-17T17:20:01.115 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:01.130 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.GetBlockNamePrefixPP (15 ms) 2024-08-17T17:20:01.130 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestGetCreateTimestamp 2024-08-17T17:20:01.130 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:01.146 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestGetCreateTimestamp (16 ms) 2024-08-17T17:20:01.146 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.GetCreateTimestampPP 2024-08-17T17:20:01.146 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:01.160 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.GetCreateTimestampPP (14 ms) 2024-08-17T17:20:01.160 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.OpenAio 2024-08-17T17:20:01.160 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:01.175 INFO:tasks.workunit.client.0.smithi136.stdout:image has size 2097152 and order 22 2024-08-17T17:20:01.175 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.OpenAio (16 ms) 2024-08-17T17:20:01.175 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.OpenAioFail 2024-08-17T17:20:01.178 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.OpenAioFail (2 ms) 2024-08-17T17:20:01.178 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.OpenAioPP 2024-08-17T17:20:01.178 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:01.199 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.OpenAioPP (22 ms) 2024-08-17T17:20:01.200 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.OpenAioFailPP 2024-08-17T17:20:01.202 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.OpenAioFailPP (2 ms) 2024-08-17T17:20:01.202 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.ResizeAndStat 2024-08-17T17:20:01.203 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:01.233 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.ResizeAndStat (31 ms) 2024-08-17T17:20:01.233 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.ResizeAndStatPP 2024-08-17T17:20:01.233 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:01.259 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.ResizeAndStatPP (26 ms) 2024-08-17T17:20:01.259 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResize 2024-08-17T17:20:01.259 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:00 smithi165 ceph-mon[105517]: pgmap v334: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 469 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:20:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:00 smithi165 ceph-mon[105517]: osdmap e560: 8 total, 8 up, 8 in 2024-08-17T17:20:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:00 smithi165 ceph-mon[105517]: overall HEALTH_OK 2024-08-17T17:20:01.291 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.UpdateWatchAndResize (32 ms) 2024-08-17T17:20:01.291 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResizePP 2024-08-17T17:20:01.291 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:01.323 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.UpdateWatchAndResizePP (32 ms) 2024-08-17T17:20:01.323 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestCreateLsDelete 2024-08-17T17:20:01.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:00 smithi136 ceph-mon[109861]: pgmap v334: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 469 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:20:01.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:00 smithi136 ceph-mon[109861]: osdmap e560: 8 total, 8 up, 8 in 2024-08-17T17:20:01.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:00 smithi136 ceph-mon[109861]: overall HEALTH_OK 2024-08-17T17:20:01.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:00 smithi136 ceph-mon[115078]: pgmap v334: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 469 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:20:01.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:00 smithi136 ceph-mon[115078]: osdmap e560: 8 total, 8 up, 8 in 2024-08-17T17:20:01.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:00 smithi136 ceph-mon[115078]: overall HEALTH_OK 2024-08-17T17:20:02.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:01 smithi165 ceph-mon[105517]: osdmap e561: 8 total, 8 up, 8 in 2024-08-17T17:20:02.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:01 smithi136 ceph-mon[109861]: osdmap e561: 8 total, 8 up, 8 in 2024-08-17T17:20:02.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:01 smithi136 ceph-mon[115078]: osdmap e561: 8 total, 8 up, 8 in 2024-08-17T17:20:03.006 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:03.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:02 smithi165 ceph-mon[105517]: pgmap v337: 105 pgs: 105 active+clean; 583 KiB data, 470 MiB used, 715 GiB / 715 GiB avail; 181 KiB/s rd, 20 KiB/s wr, 250 op/s 2024-08-17T17:20:03.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:02 smithi165 ceph-mon[105517]: osdmap e562: 8 total, 8 up, 8 in 2024-08-17T17:20:03.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:02 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3446551553' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-16","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:03.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:02 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-16","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:03.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:02 smithi136 ceph-mon[109861]: pgmap v337: 105 pgs: 105 active+clean; 583 KiB data, 470 MiB used, 715 GiB / 715 GiB avail; 181 KiB/s rd, 20 KiB/s wr, 250 op/s 2024-08-17T17:20:03.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:02 smithi136 ceph-mon[109861]: osdmap e562: 8 total, 8 up, 8 in 2024-08-17T17:20:03.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:02 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3446551553' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-16","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:03.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:02 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-16","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:03.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:02 smithi136 ceph-mon[115078]: pgmap v337: 105 pgs: 105 active+clean; 583 KiB data, 470 MiB used, 715 GiB / 715 GiB avail; 181 KiB/s rd, 20 KiB/s wr, 250 op/s 2024-08-17T17:20:03.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:02 smithi136 ceph-mon[115078]: osdmap e562: 8 total, 8 up, 8 in 2024-08-17T17:20:03.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:02 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3446551553' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-16","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:03.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:02 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-16","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:03 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:03 smithi165 ceph-mon[105517]: osdmap e563: 8 total, 8 up, 8 in 2024-08-17T17:20:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:03 smithi165 ceph-mon[105517]: osdmap e564: 8 total, 8 up, 8 in 2024-08-17T17:20:04.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:03 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:04.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:03 smithi136 ceph-mon[109861]: osdmap e563: 8 total, 8 up, 8 in 2024-08-17T17:20:04.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:03 smithi136 ceph-mon[109861]: osdmap e564: 8 total, 8 up, 8 in 2024-08-17T17:20:04.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:03 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:04.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:03 smithi136 ceph-mon[115078]: osdmap e563: 8 total, 8 up, 8 in 2024-08-17T17:20:04.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:03 smithi136 ceph-mon[115078]: osdmap e564: 8 total, 8 up, 8 in 2024-08-17T17:20:04.848 INFO:tasks.workunit.client.0.smithi136.stdout:image: image18 2024-08-17T17:20:04.848 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image18 2024-08-17T17:20:04.848 INFO:tasks.workunit.client.0.smithi136.stdout:found image18 2024-08-17T17:20:04.848 INFO:tasks.workunit.client.0.smithi136.stdout:erased image18 2024-08-17T17:20:04.848 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:04.923 INFO:tasks.workunit.client.0.smithi136.stdout:image: image18 2024-08-17T17:20:04.923 INFO:tasks.workunit.client.0.smithi136.stdout:image: image19 2024-08-17T17:20:04.923 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image18 2024-08-17T17:20:04.923 INFO:tasks.workunit.client.0.smithi136.stdout:found image18 2024-08-17T17:20:04.923 INFO:tasks.workunit.client.0.smithi136.stdout:erased image18 2024-08-17T17:20:04.923 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image19 2024-08-17T17:20:04.923 INFO:tasks.workunit.client.0.smithi136.stdout:found image19 2024-08-17T17:20:04.923 INFO:tasks.workunit.client.0.smithi136.stdout:erased image19 2024-08-17T17:20:04.923 INFO:tasks.workunit.client.0.smithi136.stdout:image: image19 2024-08-17T17:20:04.923 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image19 2024-08-17T17:20:04.923 INFO:tasks.workunit.client.0.smithi136.stdout:found image19 2024-08-17T17:20:04.924 INFO:tasks.workunit.client.0.smithi136.stdout:erased image19 2024-08-17T17:20:04.924 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestCreateLsDelete (3600 ms) 2024-08-17T17:20:04.924 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestCreateLsDeletePP 2024-08-17T17:20:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:04 smithi165 ceph-mon[105517]: pgmap v341: 137 pgs: 32 unknown, 105 active+clean; 583 KiB data, 470 MiB used, 715 GiB / 715 GiB avail; 187 KiB/s rd, 21 KiB/s wr, 258 op/s 2024-08-17T17:20:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:04 smithi165 ceph-mon[105517]: osdmap e565: 8 total, 8 up, 8 in 2024-08-17T17:20:05.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:04 smithi136 ceph-mon[109861]: pgmap v341: 137 pgs: 32 unknown, 105 active+clean; 583 KiB data, 470 MiB used, 715 GiB / 715 GiB avail; 187 KiB/s rd, 21 KiB/s wr, 258 op/s 2024-08-17T17:20:05.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:04 smithi136 ceph-mon[109861]: osdmap e565: 8 total, 8 up, 8 in 2024-08-17T17:20:05.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:05 smithi136 ceph-mon[115078]: pgmap v341: 137 pgs: 32 unknown, 105 active+clean; 583 KiB data, 470 MiB used, 715 GiB / 715 GiB avail; 187 KiB/s rd, 21 KiB/s wr, 258 op/s 2024-08-17T17:20:05.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:05 smithi136 ceph-mon[115078]: osdmap e565: 8 total, 8 up, 8 in 2024-08-17T17:20:06.843 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:07.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:06 smithi136 ceph-mon[109861]: osdmap e566: 8 total, 8 up, 8 in 2024-08-17T17:20:07.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:06 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/349644327' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:07.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:06 smithi136 ceph-mon[109861]: pgmap v344: 169 pgs: 64 unknown, 105 active+clean; 583 KiB data, 470 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:20:07.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:20:07.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:06 smithi136 ceph-mon[115078]: osdmap e566: 8 total, 8 up, 8 in 2024-08-17T17:20:07.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:06 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/349644327' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:07.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:06 smithi136 ceph-mon[115078]: pgmap v344: 169 pgs: 64 unknown, 105 active+clean; 583 KiB data, 470 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:20:07.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:20:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:06 smithi165 ceph-mon[105517]: osdmap e566: 8 total, 8 up, 8 in 2024-08-17T17:20:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:06 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/349644327' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:06 smithi165 ceph-mon[105517]: pgmap v344: 169 pgs: 64 unknown, 105 active+clean; 583 KiB data, 470 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:20:07.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:20:07.832 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:20:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:20:07] "GET /metrics HTTP/1.1" 200 41001 "" "Prometheus/2.51.0" 2024-08-17T17:20:08.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:07 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:20:08.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:07 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/349644327' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:08.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:07 smithi136 ceph-mon[115078]: osdmap e567: 8 total, 8 up, 8 in 2024-08-17T17:20:08.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:07 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:20:08.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:07 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/349644327' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:08.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:07 smithi136 ceph-mon[109861]: osdmap e567: 8 total, 8 up, 8 in 2024-08-17T17:20:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:07 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:20:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:07 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/349644327' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:07 smithi165 ceph-mon[105517]: osdmap e567: 8 total, 8 up, 8 in 2024-08-17T17:20:08.863 INFO:tasks.workunit.client.0.smithi136.stdout:num images is: 1 2024-08-17T17:20:08.864 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 1 2024-08-17T17:20:08.864 INFO:tasks.workunit.client.0.smithi136.stdout:image: image20 2024-08-17T17:20:08.864 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image20 2024-08-17T17:20:08.864 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:08.876 INFO:tasks.workunit.client.0.smithi136.stdout:num images is: 2 2024-08-17T17:20:08.877 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 2 2024-08-17T17:20:08.877 INFO:tasks.workunit.client.0.smithi136.stdout:image: image20 2024-08-17T17:20:08.877 INFO:tasks.workunit.client.0.smithi136.stdout:image: image21 2024-08-17T17:20:08.877 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image20 2024-08-17T17:20:08.877 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image21 2024-08-17T17:20:08.929 INFO:tasks.workunit.client.0.smithi136.stdout:num images is: 1 2024-08-17T17:20:08.930 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 1 2024-08-17T17:20:08.930 INFO:tasks.workunit.client.0.smithi136.stdout:image: image21 2024-08-17T17:20:08.930 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image21 2024-08-17T17:20:08.930 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestCreateLsDeletePP (4007 ms) 2024-08-17T17:20:08.930 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestCopy 2024-08-17T17:20:09.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:08 smithi136 ceph-mon[115078]: osdmap e568: 8 total, 8 up, 8 in 2024-08-17T17:20:09.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:08 smithi136 ceph-mon[115078]: pgmap v347: 169 pgs: 19 creating+activating, 9 creating+peering, 141 active+clean; 583 KiB data, 470 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.2 KiB/s wr, 41 op/s 2024-08-17T17:20:09.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:08 smithi136 ceph-mon[109861]: osdmap e568: 8 total, 8 up, 8 in 2024-08-17T17:20:09.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:08 smithi136 ceph-mon[109861]: pgmap v347: 169 pgs: 19 creating+activating, 9 creating+peering, 141 active+clean; 583 KiB data, 470 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.2 KiB/s wr, 41 op/s 2024-08-17T17:20:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:08 smithi165 ceph-mon[105517]: osdmap e568: 8 total, 8 up, 8 in 2024-08-17T17:20:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:08 smithi165 ceph-mon[105517]: pgmap v347: 169 pgs: 19 creating+activating, 9 creating+peering, 141 active+clean; 583 KiB data, 470 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.2 KiB/s wr, 41 op/s 2024-08-17T17:20:10.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:09 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:20:10.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:09 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T17:20:10.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:09 smithi136 ceph-mon[115078]: osdmap e569: 8 total, 8 up, 8 in 2024-08-17T17:20:10.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:09 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:20:10.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:09 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T17:20:10.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:09 smithi136 ceph-mon[109861]: osdmap e569: 8 total, 8 up, 8 in 2024-08-17T17:20:10.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:09 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:20:10.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:09 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T17:20:10.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:09 smithi165 ceph-mon[105517]: osdmap e569: 8 total, 8 up, 8 in 2024-08-17T17:20:10.865 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:11.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:10 smithi136 ceph-mon[115078]: osdmap e570: 8 total, 8 up, 8 in 2024-08-17T17:20:11.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:10 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/804456427' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:11.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:10 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:11.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:10 smithi136 ceph-mon[115078]: pgmap v350: 201 pgs: 32 unknown, 19 creating+activating, 9 creating+peering, 141 active+clean; 583 KiB data, 470 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.2 KiB/s wr, 41 op/s 2024-08-17T17:20:11.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:10 smithi136 ceph-mon[109861]: osdmap e570: 8 total, 8 up, 8 in 2024-08-17T17:20:11.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:10 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/804456427' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:11.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:10 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:11.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:10 smithi136 ceph-mon[109861]: pgmap v350: 201 pgs: 32 unknown, 19 creating+activating, 9 creating+peering, 141 active+clean; 583 KiB data, 470 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.2 KiB/s wr, 41 op/s 2024-08-17T17:20:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:10 smithi165 ceph-mon[105517]: osdmap e570: 8 total, 8 up, 8 in 2024-08-17T17:20:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:10 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/804456427' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:10 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:10 smithi165 ceph-mon[105517]: pgmap v350: 201 pgs: 32 unknown, 19 creating+activating, 9 creating+peering, 141 active+clean; 583 KiB data, 470 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.2 KiB/s wr, 41 op/s 2024-08-17T17:20:12.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:11 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:20:12.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:11 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:12.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:11 smithi136 ceph-mon[115078]: osdmap e571: 8 total, 8 up, 8 in 2024-08-17T17:20:12.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:11 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:20:12.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:11 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:12.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:11 smithi136 ceph-mon[109861]: osdmap e571: 8 total, 8 up, 8 in 2024-08-17T17:20:12.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:11 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:20:12.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:11 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:12.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:11 smithi165 ceph-mon[105517]: osdmap e571: 8 total, 8 up, 8 in 2024-08-17T17:20:13.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:12 smithi136 ceph-mon[109861]: pgmap v353: 201 pgs: 201 active+clean; 583 KiB data, 471 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.2 KiB/s wr, 40 op/s 2024-08-17T17:20:13.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:12 smithi136 ceph-mon[109861]: osdmap e572: 8 total, 8 up, 8 in 2024-08-17T17:20:13.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:12 smithi136 ceph-mon[115078]: pgmap v353: 201 pgs: 201 active+clean; 583 KiB data, 471 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.2 KiB/s wr, 40 op/s 2024-08-17T17:20:13.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:12 smithi136 ceph-mon[115078]: osdmap e572: 8 total, 8 up, 8 in 2024-08-17T17:20:13.202 INFO:tasks.workunit.client.0.smithi136.stdout:image: image22 2024-08-17T17:20:13.202 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image22 2024-08-17T17:20:13.202 INFO:tasks.workunit.client.0.smithi136.stdout:found image22 2024-08-17T17:20:13.202 INFO:tasks.workunit.client.0.smithi136.stdout:erased image22 2024-08-17T17:20:13.202 INFO:tasks.workunit.client.0.smithi136.stdout:image: image22 2024-08-17T17:20:13.202 INFO:tasks.workunit.client.0.smithi136.stdout:image: image23 2024-08-17T17:20:13.202 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image22 2024-08-17T17:20:13.202 INFO:tasks.workunit.client.0.smithi136.stdout:found image22 2024-08-17T17:20:13.203 INFO:tasks.workunit.client.0.smithi136.stdout:erased image22 2024-08-17T17:20:13.203 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image23 2024-08-17T17:20:13.203 INFO:tasks.workunit.client.0.smithi136.stdout:found image23 2024-08-17T17:20:13.203 INFO:tasks.workunit.client.0.smithi136.stdout:erased image23 2024-08-17T17:20:13.203 INFO:tasks.workunit.client.0.smithi136.stdout:100.00% done 2024-08-17T17:20:13.203 INFO:tasks.workunit.client.0.smithi136.stdout:image: image22 2024-08-17T17:20:13.203 INFO:tasks.workunit.client.0.smithi136.stdout:image: image23 2024-08-17T17:20:13.203 INFO:tasks.workunit.client.0.smithi136.stdout:image: image24 2024-08-17T17:20:13.203 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image22 2024-08-17T17:20:13.203 INFO:tasks.workunit.client.0.smithi136.stdout:found image22 2024-08-17T17:20:13.203 INFO:tasks.workunit.client.0.smithi136.stdout:erased image22 2024-08-17T17:20:13.203 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image23 2024-08-17T17:20:13.204 INFO:tasks.workunit.client.0.smithi136.stdout:found image23 2024-08-17T17:20:13.204 INFO:tasks.workunit.client.0.smithi136.stdout:erased image23 2024-08-17T17:20:13.204 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image24 2024-08-17T17:20:13.204 INFO:tasks.workunit.client.0.smithi136.stdout:found image24 2024-08-17T17:20:13.204 INFO:tasks.workunit.client.0.smithi136.stdout:erased image24 2024-08-17T17:20:13.204 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestCopy (4272 ms) 2024-08-17T17:20:13.204 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestCopyPP 2024-08-17T17:20:13.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:12 smithi165 ceph-mon[105517]: pgmap v353: 201 pgs: 201 active+clean; 583 KiB data, 471 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.2 KiB/s wr, 40 op/s 2024-08-17T17:20:13.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:12 smithi165 ceph-mon[105517]: osdmap e572: 8 total, 8 up, 8 in 2024-08-17T17:20:14.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:13 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:20:14.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:13 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T17:20:14.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:13 smithi136 ceph-mon[109861]: osdmap e573: 8 total, 8 up, 8 in 2024-08-17T17:20:14.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:13 smithi136 ceph-mon[109861]: osdmap e574: 8 total, 8 up, 8 in 2024-08-17T17:20:14.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:13 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/277495998' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:14.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:13 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:14.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:13 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:20:14.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:13 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T17:20:14.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:13 smithi136 ceph-mon[115078]: osdmap e573: 8 total, 8 up, 8 in 2024-08-17T17:20:14.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:13 smithi136 ceph-mon[115078]: osdmap e574: 8 total, 8 up, 8 in 2024-08-17T17:20:14.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:13 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/277495998' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:14.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:13 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:13 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:20:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:13 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T17:20:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:13 smithi165 ceph-mon[105517]: osdmap e573: 8 total, 8 up, 8 in 2024-08-17T17:20:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:13 smithi165 ceph-mon[105517]: osdmap e574: 8 total, 8 up, 8 in 2024-08-17T17:20:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:13 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/277495998' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:14.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:13 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:14.832 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:15.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:14 smithi136 ceph-mon[109861]: pgmap v356: 233 pgs: 32 unknown, 201 active+clean; 583 KiB data, 471 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.2 KiB/s wr, 40 op/s 2024-08-17T17:20:15.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:14 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:20:15.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:14 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:15.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:14 smithi136 ceph-mon[109861]: osdmap e575: 8 total, 8 up, 8 in 2024-08-17T17:20:15.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:14 smithi136 ceph-mon[115078]: pgmap v356: 233 pgs: 32 unknown, 201 active+clean; 583 KiB data, 471 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.2 KiB/s wr, 40 op/s 2024-08-17T17:20:15.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:14 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:20:15.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:14 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:15.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:14 smithi136 ceph-mon[115078]: osdmap e575: 8 total, 8 up, 8 in 2024-08-17T17:20:15.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:14 smithi165 ceph-mon[105517]: pgmap v356: 233 pgs: 32 unknown, 201 active+clean; 583 KiB data, 471 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 3.2 KiB/s wr, 40 op/s 2024-08-17T17:20:15.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:14 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:20:15.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:14 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:15.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:14 smithi165 ceph-mon[105517]: osdmap e575: 8 total, 8 up, 8 in 2024-08-17T17:20:17.066 INFO:tasks.workunit.client.0.smithi136.stdout:num images is: 1 2024-08-17T17:20:17.067 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 1 2024-08-17T17:20:17.067 INFO:tasks.workunit.client.0.smithi136.stdout:image: image25 2024-08-17T17:20:17.067 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image25 2024-08-17T17:20:17.084 INFO:tasks.workunit.client.0.smithi136.stdout:num images is: 2 2024-08-17T17:20:17.084 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 2 2024-08-17T17:20:17.084 INFO:tasks.workunit.client.0.smithi136.stdout:image: image25 2024-08-17T17:20:17.084 INFO:tasks.workunit.client.0.smithi136.stdout:image: image26 2024-08-17T17:20:17.084 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image25 2024-08-17T17:20:17.084 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image26 2024-08-17T17:20:17.123 INFO:tasks.workunit.client.0.smithi136.stdout:100.00% done 2024-08-17T17:20:17.123 INFO:tasks.workunit.client.0.smithi136.stdout:num images is: 3 2024-08-17T17:20:17.123 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 3 2024-08-17T17:20:17.123 INFO:tasks.workunit.client.0.smithi136.stdout:image: image25 2024-08-17T17:20:17.123 INFO:tasks.workunit.client.0.smithi136.stdout:image: image26 2024-08-17T17:20:17.123 INFO:tasks.workunit.client.0.smithi136.stdout:image: image27 2024-08-17T17:20:17.124 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image25 2024-08-17T17:20:17.124 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image26 2024-08-17T17:20:17.124 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image27 2024-08-17T17:20:17.149 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestCopyPP (3948 ms) 2024-08-17T17:20:17.149 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestDeepCopy 2024-08-17T17:20:17.158 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:16 smithi136 ceph-mon[109861]: osdmap e576: 8 total, 8 up, 8 in 2024-08-17T17:20:17.158 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:16 smithi136 ceph-mon[109861]: pgmap v359: 233 pgs: 32 unknown, 201 active+clean; 583 KiB data, 471 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:20:17.158 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:16 smithi136 ceph-mon[115078]: osdmap e576: 8 total, 8 up, 8 in 2024-08-17T17:20:17.158 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:16 smithi136 ceph-mon[115078]: pgmap v359: 233 pgs: 32 unknown, 201 active+clean; 583 KiB data, 471 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:20:17.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:16 smithi165 ceph-mon[105517]: osdmap e576: 8 total, 8 up, 8 in 2024-08-17T17:20:17.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:16 smithi165 ceph-mon[105517]: pgmap v359: 233 pgs: 32 unknown, 201 active+clean; 583 KiB data, 471 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:20:17.831 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:20:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:20:17] "GET /metrics HTTP/1.1" 200 41001 "" "Prometheus/2.51.0" 2024-08-17T17:20:17.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'echo wait for all mons to upgrade' 2024-08-17T17:20:18.088 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:20:18.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:17 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:20:18.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:17 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T17:20:18.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:17 smithi136 ceph-mon[109861]: osdmap e577: 8 total, 8 up, 8 in 2024-08-17T17:20:18.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:17 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:20:18.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:17 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T17:20:18.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:17 smithi136 ceph-mon[115078]: osdmap e577: 8 total, 8 up, 8 in 2024-08-17T17:20:18.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:17 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:20:18.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:17 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T17:20:18.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:17 smithi165 ceph-mon[105517]: osdmap e577: 8 total, 8 up, 8 in 2024-08-17T17:20:18.861 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:19.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:18 smithi136 ceph-mon[109861]: osdmap e578: 8 total, 8 up, 8 in 2024-08-17T17:20:19.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:18 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2768364734' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:19.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:18 smithi136 ceph-mon[109861]: pgmap v362: 265 pgs: 32 unknown, 233 active+clean; 583 KiB data, 484 MiB used, 715 GiB / 715 GiB avail; 342 KiB/s rd, 23 KiB/s wr, 401 op/s 2024-08-17T17:20:19.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:18 smithi136 ceph-mon[115078]: osdmap e578: 8 total, 8 up, 8 in 2024-08-17T17:20:19.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:18 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2768364734' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:19.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:18 smithi136 ceph-mon[115078]: pgmap v362: 265 pgs: 32 unknown, 233 active+clean; 583 KiB data, 484 MiB used, 715 GiB / 715 GiB avail; 342 KiB/s rd, 23 KiB/s wr, 401 op/s 2024-08-17T17:20:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:18 smithi165 ceph-mon[105517]: osdmap e578: 8 total, 8 up, 8 in 2024-08-17T17:20:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:18 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2768364734' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:18 smithi165 ceph-mon[105517]: pgmap v362: 265 pgs: 32 unknown, 233 active+clean; 583 KiB data, 484 MiB used, 715 GiB / 715 GiB avail; 342 KiB/s rd, 23 KiB/s wr, 401 op/s 2024-08-17T17:20:19.801 INFO:teuthology.orchestra.run.smithi136.stdout:wait for all mons to upgrade 2024-08-17T17:20:20.163 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:19 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:20:20.163 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:19 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2768364734' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:20.163 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:19 smithi136 ceph-mon[109861]: osdmap e579: 8 total, 8 up, 8 in 2024-08-17T17:20:20.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:19 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:20:20.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:19 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2768364734' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:20.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:19 smithi136 ceph-mon[115078]: osdmap e579: 8 total, 8 up, 8 in 2024-08-17T17:20:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:19 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:20:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:19 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2768364734' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:19 smithi165 ceph-mon[105517]: osdmap e579: 8 total, 8 up, 8 in 2024-08-17T17:20:20.435 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon' 2024-08-17T17:20:20.635 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:20:21.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:20 smithi136 ceph-mon[115078]: osdmap e580: 8 total, 8 up, 8 in 2024-08-17T17:20:21.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:20 smithi136 ceph-mon[115078]: pgmap v365: 265 pgs: 32 unknown, 233 active+clean; 583 KiB data, 484 MiB used, 715 GiB / 715 GiB avail; 342 KiB/s rd, 23 KiB/s wr, 401 op/s 2024-08-17T17:20:21.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:20 smithi136 ceph-mon[109861]: osdmap e580: 8 total, 8 up, 8 in 2024-08-17T17:20:21.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:20 smithi136 ceph-mon[109861]: pgmap v365: 265 pgs: 32 unknown, 233 active+clean; 583 KiB data, 484 MiB used, 715 GiB / 715 GiB avail; 342 KiB/s rd, 23 KiB/s wr, 401 op/s 2024-08-17T17:20:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:20 smithi165 ceph-mon[105517]: osdmap e580: 8 total, 8 up, 8 in 2024-08-17T17:20:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:20 smithi165 ceph-mon[105517]: pgmap v365: 265 pgs: 32 unknown, 233 active+clean; 583 KiB data, 484 MiB used, 715 GiB / 715 GiB avail; 342 KiB/s rd, 23 KiB/s wr, 401 op/s 2024-08-17T17:20:22.010 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:21 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:20:22.010 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:21 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T17:20:22.010 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:21 smithi136 ceph-mon[109861]: osdmap e581: 8 total, 8 up, 8 in 2024-08-17T17:20:22.010 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:20:22.011 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:21 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:20:22.011 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:21 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T17:20:22.011 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:21 smithi136 ceph-mon[115078]: osdmap e581: 8 total, 8 up, 8 in 2024-08-17T17:20:22.011 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:20:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:21 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:20:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:21 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T17:20:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:21 smithi165 ceph-mon[105517]: osdmap e581: 8 total, 8 up, 8 in 2024-08-17T17:20:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:20:23.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:22 smithi136 ceph-mon[109861]: pgmap v367: 265 pgs: 265 active+clean; 583 KiB data, 492 MiB used, 715 GiB / 715 GiB avail; 684 KiB/s rd, 42 KiB/s wr, 799 op/s 2024-08-17T17:20:23.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:22 smithi136 ceph-mon[109861]: osdmap e582: 8 total, 8 up, 8 in 2024-08-17T17:20:23.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:22 smithi136 ceph-mon[115078]: pgmap v367: 265 pgs: 265 active+clean; 583 KiB data, 492 MiB used, 715 GiB / 715 GiB avail; 684 KiB/s rd, 42 KiB/s wr, 799 op/s 2024-08-17T17:20:23.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:22 smithi136 ceph-mon[115078]: osdmap e582: 8 total, 8 up, 8 in 2024-08-17T17:20:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:22 smithi165 ceph-mon[105517]: pgmap v367: 265 pgs: 265 active+clean; 583 KiB data, 492 MiB used, 715 GiB / 715 GiB avail; 684 KiB/s rd, 42 KiB/s wr, 799 op/s 2024-08-17T17:20:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:22 smithi165 ceph-mon[105517]: osdmap e582: 8 total, 8 up, 8 in 2024-08-17T17:20:24.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:23 smithi136 ceph-mon[109861]: from='client.44294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453", "daemon_types": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:20:24.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:23 smithi136 ceph-mon[109861]: osdmap e583: 8 total, 8 up, 8 in 2024-08-17T17:20:24.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:23 smithi136 ceph-mon[109861]: osdmap e584: 8 total, 8 up, 8 in 2024-08-17T17:20:24.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:23 smithi136 ceph-mon[115078]: from='client.44294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453", "daemon_types": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:20:24.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:23 smithi136 ceph-mon[115078]: osdmap e583: 8 total, 8 up, 8 in 2024-08-17T17:20:24.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:23 smithi136 ceph-mon[115078]: osdmap e584: 8 total, 8 up, 8 in 2024-08-17T17:20:24.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:23 smithi165 ceph-mon[105517]: from='client.44294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453", "daemon_types": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:20:24.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:23 smithi165 ceph-mon[105517]: osdmap e583: 8 total, 8 up, 8 in 2024-08-17T17:20:24.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:23 smithi165 ceph-mon[105517]: osdmap e584: 8 total, 8 up, 8 in 2024-08-17T17:20:24.293 INFO:teuthology.orchestra.run.smithi136.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:20:24.925 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:24 smithi136 ceph-mon[109861]: pgmap v371: 265 pgs: 265 active+clean; 583 KiB data, 492 MiB used, 715 GiB / 715 GiB avail; 688 KiB/s rd, 42 KiB/s wr, 804 op/s 2024-08-17T17:20:24.925 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:24 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:24.925 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:24 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:20:24.925 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:24 smithi136 ceph-mon[115078]: pgmap v371: 265 pgs: 265 active+clean; 583 KiB data, 492 MiB used, 715 GiB / 715 GiB avail; 688 KiB/s rd, 42 KiB/s wr, 804 op/s 2024-08-17T17:20:24.925 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:24 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:24.925 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:24 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:20:24.947 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-08-17T17:20:24.952 INFO:tasks.workunit.client.0.smithi136.stdout:image: image28 2024-08-17T17:20:24.952 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image28 2024-08-17T17:20:24.952 INFO:tasks.workunit.client.0.smithi136.stdout:found image28 2024-08-17T17:20:24.952 INFO:tasks.workunit.client.0.smithi136.stdout:erased image28 2024-08-17T17:20:24.952 INFO:tasks.workunit.client.0.smithi136.stdout:image: image28 2024-08-17T17:20:24.952 INFO:tasks.workunit.client.0.smithi136.stdout:image: image29 2024-08-17T17:20:24.952 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image28 2024-08-17T17:20:24.952 INFO:tasks.workunit.client.0.smithi136.stdout:found image28 2024-08-17T17:20:24.952 INFO:tasks.workunit.client.0.smithi136.stdout:erased image28 2024-08-17T17:20:24.953 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image29 2024-08-17T17:20:24.953 INFO:tasks.workunit.client.0.smithi136.stdout:found image29 2024-08-17T17:20:24.953 INFO:tasks.workunit.client.0.smithi136.stdout:erased image29 2024-08-17T17:20:24.953 INFO:tasks.workunit.client.0.smithi136.stdout:100.00% done 2024-08-17T17:20:24.953 INFO:tasks.workunit.client.0.smithi136.stdout:image: image28 2024-08-17T17:20:24.953 INFO:tasks.workunit.client.0.smithi136.stdout:image: image29 2024-08-17T17:20:24.953 INFO:tasks.workunit.client.0.smithi136.stdout:image: image30 2024-08-17T17:20:24.953 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image28 2024-08-17T17:20:24.953 INFO:tasks.workunit.client.0.smithi136.stdout:found image28 2024-08-17T17:20:24.953 INFO:tasks.workunit.client.0.smithi136.stdout:erased image28 2024-08-17T17:20:24.953 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image29 2024-08-17T17:20:24.953 INFO:tasks.workunit.client.0.smithi136.stdout:found image29 2024-08-17T17:20:24.953 INFO:tasks.workunit.client.0.smithi136.stdout:erased image29 2024-08-17T17:20:24.953 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image30 2024-08-17T17:20:24.953 INFO:tasks.workunit.client.0.smithi136.stdout:found image30 2024-08-17T17:20:24.954 INFO:tasks.workunit.client.0.smithi136.stdout:erased image30 2024-08-17T17:20:24.954 INFO:tasks.workunit.client.0.smithi136.stdout:image: image28 2024-08-17T17:20:24.954 INFO:tasks.workunit.client.0.smithi136.stdout:image: image29 2024-08-17T17:20:24.954 INFO:tasks.workunit.client.0.smithi136.stdout:image: image30 2024-08-17T17:20:24.954 INFO:tasks.workunit.client.0.smithi136.stdout:image: image31 2024-08-17T17:20:24.954 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image28 2024-08-17T17:20:24.954 INFO:tasks.workunit.client.0.smithi136.stdout:found image28 2024-08-17T17:20:24.954 INFO:tasks.workunit.client.0.smithi136.stdout:erased image28 2024-08-17T17:20:24.954 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image29 2024-08-17T17:20:24.954 INFO:tasks.workunit.client.0.smithi136.stdout:found image29 2024-08-17T17:20:24.954 INFO:tasks.workunit.client.0.smithi136.stdout:erased image29 2024-08-17T17:20:24.954 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image30 2024-08-17T17:20:24.954 INFO:tasks.workunit.client.0.smithi136.stdout:found image30 2024-08-17T17:20:24.955 INFO:tasks.workunit.client.0.smithi136.stdout:erased image30 2024-08-17T17:20:24.955 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image31 2024-08-17T17:20:24.955 INFO:tasks.workunit.client.0.smithi136.stdout:found image31 2024-08-17T17:20:24.955 INFO:tasks.workunit.client.0.smithi136.stdout:erased image31 2024-08-17T17:20:24.955 INFO:tasks.workunit.client.0.smithi136.stdout:image: image28 2024-08-17T17:20:24.955 INFO:tasks.workunit.client.0.smithi136.stdout:image: image29 2024-08-17T17:20:24.955 INFO:tasks.workunit.client.0.smithi136.stdout:image: image30 2024-08-17T17:20:24.955 INFO:tasks.workunit.client.0.smithi136.stdout:image: image31 2024-08-17T17:20:24.955 INFO:tasks.workunit.client.0.smithi136.stdout:image: image32 2024-08-17T17:20:24.955 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image28 2024-08-17T17:20:24.955 INFO:tasks.workunit.client.0.smithi136.stdout:found image28 2024-08-17T17:20:24.955 INFO:tasks.workunit.client.0.smithi136.stdout:erased image28 2024-08-17T17:20:24.955 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image29 2024-08-17T17:20:24.955 INFO:tasks.workunit.client.0.smithi136.stdout:found image29 2024-08-17T17:20:24.955 INFO:tasks.workunit.client.0.smithi136.stdout:erased image29 2024-08-17T17:20:24.956 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image30 2024-08-17T17:20:24.956 INFO:tasks.workunit.client.0.smithi136.stdout:found image30 2024-08-17T17:20:24.956 INFO:tasks.workunit.client.0.smithi136.stdout:erased image30 2024-08-17T17:20:24.956 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image31 2024-08-17T17:20:24.956 INFO:tasks.workunit.client.0.smithi136.stdout:found image31 2024-08-17T17:20:24.956 INFO:tasks.workunit.client.0.smithi136.stdout:erased image31 2024-08-17T17:20:24.956 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image32 2024-08-17T17:20:24.956 INFO:tasks.workunit.client.0.smithi136.stdout:found image32 2024-08-17T17:20:24.956 INFO:tasks.workunit.client.0.smithi136.stdout:erased image32 2024-08-17T17:20:24.956 INFO:tasks.workunit.client.0.smithi136.stdout:100.00% done 2024-08-17T17:20:24.956 INFO:tasks.workunit.client.0.smithi136.stdout:image: image28 2024-08-17T17:20:24.956 INFO:tasks.workunit.client.0.smithi136.stdout:image: image29 2024-08-17T17:20:24.956 INFO:tasks.workunit.client.0.smithi136.stdout:image: image30 2024-08-17T17:20:24.956 INFO:tasks.workunit.client.0.smithi136.stdout:image: image31 2024-08-17T17:20:24.957 INFO:tasks.workunit.client.0.smithi136.stdout:image: image32 2024-08-17T17:20:24.957 INFO:tasks.workunit.client.0.smithi136.stdout:image: image33 2024-08-17T17:20:24.957 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image28 2024-08-17T17:20:24.957 INFO:tasks.workunit.client.0.smithi136.stdout:found image28 2024-08-17T17:20:24.957 INFO:tasks.workunit.client.0.smithi136.stdout:erased image28 2024-08-17T17:20:24.957 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image29 2024-08-17T17:20:24.957 INFO:tasks.workunit.client.0.smithi136.stdout:found image29 2024-08-17T17:20:24.957 INFO:tasks.workunit.client.0.smithi136.stdout:erased image29 2024-08-17T17:20:24.957 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image30 2024-08-17T17:20:24.957 INFO:tasks.workunit.client.0.smithi136.stdout:found image30 2024-08-17T17:20:24.957 INFO:tasks.workunit.client.0.smithi136.stdout:erased image30 2024-08-17T17:20:24.957 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image31 2024-08-17T17:20:24.957 INFO:tasks.workunit.client.0.smithi136.stdout:found image31 2024-08-17T17:20:24.957 INFO:tasks.workunit.client.0.smithi136.stdout:erased image31 2024-08-17T17:20:24.958 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image32 2024-08-17T17:20:24.958 INFO:tasks.workunit.client.0.smithi136.stdout:found image32 2024-08-17T17:20:24.958 INFO:tasks.workunit.client.0.smithi136.stdout:erased image32 2024-08-17T17:20:24.958 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image33 2024-08-17T17:20:24.958 INFO:tasks.workunit.client.0.smithi136.stdout:found image33 2024-08-17T17:20:24.958 INFO:tasks.workunit.client.0.smithi136.stdout:erased image33 2024-08-17T17:20:24.958 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestDeepCopy (7802 ms) 2024-08-17T17:20:24.958 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestDeepCopyPP 2024-08-17T17:20:25.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:24 smithi165 ceph-mon[105517]: pgmap v371: 265 pgs: 265 active+clean; 583 KiB data, 492 MiB used, 715 GiB / 715 GiB avail; 688 KiB/s rd, 42 KiB/s wr, 804 op/s 2024-08-17T17:20:25.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:24 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:25.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:24 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:20:25.751 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:20:26.141 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:25 smithi136 ceph-mon[109861]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:20:26.141 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:25 smithi136 ceph-mon[109861]: osdmap e585: 8 total, 8 up, 8 in 2024-08-17T17:20:26.141 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:25 smithi136 ceph-mon[115078]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:20:26.141 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:25 smithi136 ceph-mon[115078]: osdmap e585: 8 total, 8 up, 8 in 2024-08-17T17:20:26.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:25 smithi165 ceph-mon[105517]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:20:26.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:25 smithi165 ceph-mon[105517]: osdmap e585: 8 total, 8 up, 8 in 2024-08-17T17:20:26.917 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:27.090 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:26 smithi136 ceph-mon[109861]: pgmap v373: 265 pgs: 265 active+clean; 583 KiB data, 492 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:20:27.090 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:26 smithi136 ceph-mon[109861]: osdmap e586: 8 total, 8 up, 8 in 2024-08-17T17:20:27.090 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:26 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4003654246' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:27.090 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:26 smithi136 ceph-mon[115078]: pgmap v373: 265 pgs: 265 active+clean; 583 KiB data, 492 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:20:27.090 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:26 smithi136 ceph-mon[115078]: osdmap e586: 8 total, 8 up, 8 in 2024-08-17T17:20:27.090 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:26 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4003654246' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:27.162 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:26 smithi165 ceph-mon[105517]: pgmap v373: 265 pgs: 265 active+clean; 583 KiB data, 492 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:20:27.162 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:26 smithi165 ceph-mon[105517]: osdmap e586: 8 total, 8 up, 8 in 2024-08-17T17:20:27.162 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:26 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4003654246' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:27.505 INFO:teuthology.orchestra.run.smithi136.stdout:true 2024-08-17T17:20:27.828 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:20:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:20:27] "GET /metrics HTTP/1.1" 200 48245 "" "Prometheus/2.51.0" 2024-08-17T17:20:28.102 INFO:teuthology.orchestra.run.smithi136.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-17T17:20:28.102 INFO:teuthology.orchestra.run.smithi136.stdout:alertmanager.a smithi136 *:9093,9094 running (5m) 91s ago 11m 19.4M - 0.27.0 11f11916f8cd c3b9b5b223ab 2024-08-17T17:20:28.103 INFO:teuthology.orchestra.run.smithi136.stdout:grafana.a smithi165 *:3000 running (5m) 35s ago 11m 77.5M - c8b91775d855 04db24fce5e9 2024-08-17T17:20:28.103 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi136.ltsuqd smithi136 running (10m) 91s ago 10m 22.2M - 18.2.4-719-gced649f8 c870753d67c2 faaa06f2950c 2024-08-17T17:20:28.103 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi165.bmrbhh smithi165 running (10m) 35s ago 10m 17.8M - 18.2.4-719-gced649f8 c870753d67c2 c74b00d929cf 2024-08-17T17:20:28.103 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.x smithi165 *:8443,9283,8765 running (6m) 35s ago 15m 456M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 837fd48ff7cb 2024-08-17T17:20:28.103 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.y smithi136 *:8443,9283,8765 running (7m) 91s ago 17m 513M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9acdc6775e48 2024-08-17T17:20:28.103 INFO:teuthology.orchestra.run.smithi136.stdout:mon.a smithi136 running (4m) 91s ago 17m 54.9M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9cf82e613626 2024-08-17T17:20:28.103 INFO:teuthology.orchestra.run.smithi136.stdout:mon.b smithi165 running (2m) 35s ago 15m 41.3M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 1fd357308b8d 2024-08-17T17:20:28.103 INFO:teuthology.orchestra.run.smithi136.stdout:mon.c smithi136 running (4m) 91s ago 15m 49.2M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 2c2ac98ad9d4 2024-08-17T17:20:28.103 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.a smithi136 *:9100 running (6m) 91s ago 10m 11.1M - 1.7.0 72c9c2088986 1e4c644ae533 2024-08-17T17:20:28.103 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.b smithi165 *:9100 running (6m) 35s ago 10m 11.1M - 1.7.0 72c9c2088986 7af2d2d68b06 2024-08-17T17:20:28.103 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0 smithi136 running (15m) 91s ago 15m 115M 2480M 18.2.4-719-gced649f8 c870753d67c2 faab916734b0 2024-08-17T17:20:28.103 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1 smithi136 running (14m) 91s ago 14m 122M 2480M 18.2.4-719-gced649f8 c870753d67c2 861686b6ff82 2024-08-17T17:20:28.104 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2 smithi136 running (14m) 91s ago 14m 137M 2480M 18.2.4-719-gced649f8 c870753d67c2 640be4af4156 2024-08-17T17:20:28.104 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3 smithi136 running (13m) 91s ago 13m 125M 2480M 18.2.4-719-gced649f8 c870753d67c2 14b57a745d30 2024-08-17T17:20:28.104 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4 smithi165 running (13m) 35s ago 13m 127M 2480M 18.2.4-719-gced649f8 c870753d67c2 df366c73e1f2 2024-08-17T17:20:28.104 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5 smithi165 running (13m) 35s ago 13m 103M 2480M 18.2.4-719-gced649f8 c870753d67c2 1131665e203a 2024-08-17T17:20:28.104 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6 smithi165 running (12m) 35s ago 12m 137M 2480M 18.2.4-719-gced649f8 c870753d67c2 7bc5fd925293 2024-08-17T17:20:28.104 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7 smithi165 running (12m) 35s ago 12m 118M 2480M 18.2.4-719-gced649f8 c870753d67c2 18382e4ae3b8 2024-08-17T17:20:28.104 INFO:teuthology.orchestra.run.smithi136.stdout:prometheus.a smithi165 *:9095 running (5m) 35s ago 11m 48.0M - 2.51.0 1d3b7f56885b ae8f8b69fe2f 2024-08-17T17:20:28.104 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:27 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4003654246' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:28.104 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:27 smithi136 ceph-mon[109861]: osdmap e587: 8 total, 8 up, 8 in 2024-08-17T17:20:28.105 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:27 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4003654246' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:28.105 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:27 smithi136 ceph-mon[115078]: osdmap e587: 8 total, 8 up, 8 in 2024-08-17T17:20:28.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:27 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4003654246' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:28.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:27 smithi165 ceph-mon[105517]: osdmap e587: 8 total, 8 up, 8 in 2024-08-17T17:20:28.470 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:20:28.470 INFO:teuthology.orchestra.run.smithi136.stdout: "mon": { 2024-08-17T17:20:28.470 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 3 2024-08-17T17:20:28.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:20:28.470 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr": { 2024-08-17T17:20:28.470 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 2 2024-08-17T17:20:28.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:20:28.470 INFO:teuthology.orchestra.run.smithi136.stdout: "osd": { 2024-08-17T17:20:28.470 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 8 2024-08-17T17:20:28.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:20:28.470 INFO:teuthology.orchestra.run.smithi136.stdout: "mds": { 2024-08-17T17:20:28.470 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 2 2024-08-17T17:20:28.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:20:28.471 INFO:teuthology.orchestra.run.smithi136.stdout: "overall": { 2024-08-17T17:20:28.471 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 10, 2024-08-17T17:20:28.471 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 5 2024-08-17T17:20:28.471 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:20:28.471 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:20:28.761 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:20:28.761 INFO:teuthology.orchestra.run.smithi136.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453", 2024-08-17T17:20:28.761 INFO:teuthology.orchestra.run.smithi136.stdout: "in_progress": true, 2024-08-17T17:20:28.761 INFO:teuthology.orchestra.run.smithi136.stdout: "which": "Upgrading daemons of type(s) mon", 2024-08-17T17:20:28.762 INFO:teuthology.orchestra.run.smithi136.stdout: "services_complete": [], 2024-08-17T17:20:28.762 INFO:teuthology.orchestra.run.smithi136.stdout: "progress": "", 2024-08-17T17:20:28.762 INFO:teuthology.orchestra.run.smithi136.stdout: "message": "Currently upgrading mon daemons", 2024-08-17T17:20:28.762 INFO:teuthology.orchestra.run.smithi136.stdout: "is_paused": false 2024-08-17T17:20:28.762 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:20:29.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:28 smithi136 ceph-mon[109861]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:20:29.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:28 smithi136 ceph-mon[109861]: from='client.44312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:20:29.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:28 smithi136 ceph-mon[109861]: pgmap v376: 297 pgs: 32 creating+peering, 265 active+clean; 584 KiB data, 493 MiB used, 715 GiB / 715 GiB avail; 198 KiB/s rd, 10 KiB/s wr, 239 op/s 2024-08-17T17:20:29.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:28 smithi136 ceph-mon[109861]: osdmap e588: 8 total, 8 up, 8 in 2024-08-17T17:20:29.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:28 smithi136 ceph-mon[109861]: from='client.44318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:20:29.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:28 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3280547300' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:29.082 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:28 smithi136 ceph-mon[115078]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:20:29.082 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:28 smithi136 ceph-mon[115078]: from='client.44312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:20:29.082 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:28 smithi136 ceph-mon[115078]: pgmap v376: 297 pgs: 32 creating+peering, 265 active+clean; 584 KiB data, 493 MiB used, 715 GiB / 715 GiB avail; 198 KiB/s rd, 10 KiB/s wr, 239 op/s 2024-08-17T17:20:29.083 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:28 smithi136 ceph-mon[115078]: osdmap e588: 8 total, 8 up, 8 in 2024-08-17T17:20:29.083 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:28 smithi136 ceph-mon[115078]: from='client.44318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:20:29.083 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:28 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3280547300' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:29.207 INFO:tasks.workunit.client.0.smithi136.stdout:num images is: 1 2024-08-17T17:20:29.208 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 1 2024-08-17T17:20:29.208 INFO:tasks.workunit.client.0.smithi136.stdout:image: image34 2024-08-17T17:20:29.208 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image34 2024-08-17T17:20:29.230 INFO:tasks.workunit.client.0.smithi136.stdout:num images is: 2 2024-08-17T17:20:29.230 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 2 2024-08-17T17:20:29.230 INFO:tasks.workunit.client.0.smithi136.stdout:image: image34 2024-08-17T17:20:29.230 INFO:tasks.workunit.client.0.smithi136.stdout:image: image35 2024-08-17T17:20:29.230 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image34 2024-08-17T17:20:29.230 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image35 2024-08-17T17:20:29.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:28 smithi165 ceph-mon[105517]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:20:29.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:28 smithi165 ceph-mon[105517]: from='client.44312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:20:29.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:28 smithi165 ceph-mon[105517]: pgmap v376: 297 pgs: 32 creating+peering, 265 active+clean; 584 KiB data, 493 MiB used, 715 GiB / 715 GiB avail; 198 KiB/s rd, 10 KiB/s wr, 239 op/s 2024-08-17T17:20:29.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:28 smithi165 ceph-mon[105517]: osdmap e588: 8 total, 8 up, 8 in 2024-08-17T17:20:29.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:28 smithi165 ceph-mon[105517]: from='client.44318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:20:29.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:28 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3280547300' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:29.280 INFO:tasks.workunit.client.0.smithi136.stdout:100.00% done 2024-08-17T17:20:29.280 INFO:tasks.workunit.client.0.smithi136.stdout:num images is: 3 2024-08-17T17:20:29.280 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 3 2024-08-17T17:20:29.280 INFO:tasks.workunit.client.0.smithi136.stdout:image: image34 2024-08-17T17:20:29.280 INFO:tasks.workunit.client.0.smithi136.stdout:image: image35 2024-08-17T17:20:29.280 INFO:tasks.workunit.client.0.smithi136.stdout:image: image36 2024-08-17T17:20:29.280 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image34 2024-08-17T17:20:29.280 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image35 2024-08-17T17:20:29.280 INFO:tasks.workunit.client.0.smithi136.stdout:expected = image36 2024-08-17T17:20:29.318 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestDeepCopyPP (4366 ms) 2024-08-17T17:20:29.318 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnap 2024-08-17T17:20:29.318 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:29 smithi136 ceph-mon[109861]: from='client.44330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:20:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:29 smithi136 ceph-mon[109861]: osdmap e589: 8 total, 8 up, 8 in 2024-08-17T17:20:30.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:29 smithi136 ceph-mon[115078]: from='client.44330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:20:30.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:29 smithi136 ceph-mon[115078]: osdmap e589: 8 total, 8 up, 8 in 2024-08-17T17:20:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:29 smithi165 ceph-mon[105517]: from='client.44330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:20:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:29 smithi165 ceph-mon[105517]: osdmap e589: 8 total, 8 up, 8 in 2024-08-17T17:20:31.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:30 smithi136 ceph-mon[109861]: pgmap v379: 297 pgs: 32 creating+peering, 265 active+clean; 584 KiB data, 493 MiB used, 715 GiB / 715 GiB avail; 200 KiB/s rd, 10 KiB/s wr, 241 op/s 2024-08-17T17:20:31.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:30 smithi136 ceph-mon[109861]: osdmap e590: 8 total, 8 up, 8 in 2024-08-17T17:20:31.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:31.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:31.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:20:31.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:20:31.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:31.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:30 smithi136 ceph-mon[115078]: pgmap v379: 297 pgs: 32 creating+peering, 265 active+clean; 584 KiB data, 493 MiB used, 715 GiB / 715 GiB avail; 200 KiB/s rd, 10 KiB/s wr, 241 op/s 2024-08-17T17:20:31.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:30 smithi136 ceph-mon[115078]: osdmap e590: 8 total, 8 up, 8 in 2024-08-17T17:20:31.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:31.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:31.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:20:31.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:20:31.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:30 smithi165 ceph-mon[105517]: pgmap v379: 297 pgs: 32 creating+peering, 265 active+clean; 584 KiB data, 493 MiB used, 715 GiB / 715 GiB avail; 200 KiB/s rd, 10 KiB/s wr, 241 op/s 2024-08-17T17:20:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:30 smithi165 ceph-mon[105517]: osdmap e590: 8 total, 8 up, 8 in 2024-08-17T17:20:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:30 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:30 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:30 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:20:31.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:30 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:20:31.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:30 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:32.004 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:31 smithi136 ceph-mon[109861]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:20:32.004 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:31 smithi136 ceph-mon[109861]: osdmap e591: 8 total, 8 up, 8 in 2024-08-17T17:20:32.004 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:31 smithi136 ceph-mon[115078]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:20:32.004 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:31 smithi136 ceph-mon[115078]: osdmap e591: 8 total, 8 up, 8 in 2024-08-17T17:20:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:31 smithi165 ceph-mon[105517]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:20:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:31 smithi165 ceph-mon[105517]: osdmap e591: 8 total, 8 up, 8 in 2024-08-17T17:20:32.963 INFO:tasks.workunit.client.0.smithi136.stdout:num snaps is: 1 2024-08-17T17:20:32.963 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 1 2024-08-17T17:20:32.963 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap1 2024-08-17T17:20:32.963 INFO:tasks.workunit.client.0.smithi136.stdout:found snap1 with size 2097152 2024-08-17T17:20:32.963 INFO:tasks.workunit.client.0.smithi136.stdout:num snaps is: 2 2024-08-17T17:20:32.963 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 2 2024-08-17T17:20:32.963 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap1 2024-08-17T17:20:32.963 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap2 2024-08-17T17:20:32.963 INFO:tasks.workunit.client.0.smithi136.stdout:found snap1 with size 2097152 2024-08-17T17:20:32.963 INFO:tasks.workunit.client.0.smithi136.stdout:found snap2 with size 4194304 2024-08-17T17:20:32.963 INFO:tasks.workunit.client.0.smithi136.stdout:num snaps is: 1 2024-08-17T17:20:32.964 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 1 2024-08-17T17:20:32.964 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap2 2024-08-17T17:20:32.964 INFO:tasks.workunit.client.0.smithi136.stdout:found snap2 with size 4194304 2024-08-17T17:20:32.964 INFO:tasks.workunit.client.0.smithi136.stdout:num snaps is: 0 2024-08-17T17:20:32.964 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 0 2024-08-17T17:20:32.964 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnap (3645 ms) 2024-08-17T17:20:32.964 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestGetSnapShotTimeStamp 2024-08-17T17:20:32.964 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:33.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: pgmap v382: 297 pgs: 297 active+clean; 584 KiB data, 498 MiB used, 715 GiB / 715 GiB avail; 374 KiB/s rd, 26 KiB/s wr, 448 op/s 2024-08-17T17:20:33.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: osdmap e592: 8 total, 8 up, 8 in 2024-08-17T17:20:33.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:20:33.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-08-17T17:20:33.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-08-17T17:20:33.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: pgmap v382: 297 pgs: 297 active+clean; 584 KiB data, 498 MiB used, 715 GiB / 715 GiB avail; 374 KiB/s rd, 26 KiB/s wr, 448 op/s 2024-08-17T17:20:33.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: osdmap e592: 8 total, 8 up, 8 in 2024-08-17T17:20:33.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:20:33.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.202 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.202 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.202 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-08-17T17:20:33.202 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-08-17T17:20:33.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:20:33.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-08-17T17:20:33.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-08-17T17:20:33.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-08-17T17:20:33.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-08-17T17:20:33.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-08-17T17:20:33.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-08-17T17:20:33.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-08-17T17:20:33.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-08-17T17:20:33.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-08-17T17:20:33.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:20:33.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-08-17T17:20:33.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-08-17T17:20:33.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-08-17T17:20:33.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-08-17T17:20:33.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-08-17T17:20:33.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-08-17T17:20:33.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-08-17T17:20:33.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-08-17T17:20:33.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:20:33.204 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:20:33.204 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:20:33.204 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-08-17T17:20:33.204 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-08-17T17:20:33.204 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-08-17T17:20:33.204 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-08-17T17:20:33.204 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-08-17T17:20:33.204 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-08-17T17:20:33.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-08-17T17:20:33.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-08-17T17:20:33.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-08-17T17:20:33.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:20:33.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-08-17T17:20:33.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-08-17T17:20:33.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-08-17T17:20:33.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-08-17T17:20:33.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-08-17T17:20:33.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-08-17T17:20:33.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-08-17T17:20:33.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-08-17T17:20:33.205 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:20:33.206 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:20:33.206 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:20:33.206 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:20:33.206 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:20:33.206 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:20:33.206 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-08-17T17:20:33.206 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-08-17T17:20:33.206 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:20:33.207 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:20:33.207 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:20:33.207 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.207 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:20:33.207 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:20:33.207 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:20:33.207 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:20:33.207 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-08-17T17:20:33.207 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-08-17T17:20:33.207 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:20:33.207 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:20:33.208 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:20:33.208 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: pgmap v382: 297 pgs: 297 active+clean; 584 KiB data, 498 MiB used, 715 GiB / 715 GiB avail; 374 KiB/s rd, 26 KiB/s wr, 448 op/s 2024-08-17T17:20:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: osdmap e592: 8 total, 8 up, 8 in 2024-08-17T17:20:33.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:20:33.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:33.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:20:33.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-08-17T17:20:33.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-08-17T17:20:33.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:20:33.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-08-17T17:20:33.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-08-17T17:20:33.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-08-17T17:20:33.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-08-17T17:20:33.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-08-17T17:20:33.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-08-17T17:20:33.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-08-17T17:20:33.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-08-17T17:20:33.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-08-17T17:20:33.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:20:33.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-08-17T17:20:33.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-08-17T17:20:33.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-08-17T17:20:33.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-08-17T17:20:33.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-08-17T17:20:33.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-08-17T17:20:33.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-08-17T17:20:33.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-08-17T17:20:33.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:20:33.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:20:33.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:20:33.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:20:33.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:20:33.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:20:33.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-08-17T17:20:33.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-08-17T17:20:33.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:20:33.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:20:33.267 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:20:33.267 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:33 smithi165 ceph-mon[105517]: Upgrade: Target is version 19.0.0-5836-gcd8bdbeb (squid) 2024-08-17T17:20:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:33 smithi165 ceph-mon[105517]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d'] 2024-08-17T17:20:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:33 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all mgr 2024-08-17T17:20:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:33 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all mon 2024-08-17T17:20:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:33 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all crash 2024-08-17T17:20:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:33 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all rgw 2024-08-17T17:20:34.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:33 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all rbd-mirror 2024-08-17T17:20:34.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:33 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all cephfs-mirror 2024-08-17T17:20:34.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:33 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all ceph-exporter 2024-08-17T17:20:34.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:33 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all iscsi 2024-08-17T17:20:34.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:33 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all nfs 2024-08-17T17:20:34.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:33 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all nvmeof 2024-08-17T17:20:34.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:33 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all node-exporter 2024-08-17T17:20:34.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:33 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all prometheus 2024-08-17T17:20:34.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:33 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all alertmanager 2024-08-17T17:20:34.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:33 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all grafana 2024-08-17T17:20:34.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:33 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all loki 2024-08-17T17:20:34.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:33 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all promtail 2024-08-17T17:20:34.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:33 smithi165 ceph-mon[105517]: Upgrade: Finalizing container_image settings 2024-08-17T17:20:34.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:33 smithi165 ceph-mon[105517]: Upgrade: Complete! 2024-08-17T17:20:34.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:33 smithi165 ceph-mon[105517]: osdmap e593: 8 total, 8 up, 8 in 2024-08-17T17:20:34.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:33 smithi165 ceph-mon[105517]: osdmap e594: 8 total, 8 up, 8 in 2024-08-17T17:20:34.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[109861]: Upgrade: Target is version 19.0.0-5836-gcd8bdbeb (squid) 2024-08-17T17:20:34.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[109861]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d'] 2024-08-17T17:20:34.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all mgr 2024-08-17T17:20:34.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all mon 2024-08-17T17:20:34.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all crash 2024-08-17T17:20:34.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all rgw 2024-08-17T17:20:34.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all rbd-mirror 2024-08-17T17:20:34.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all cephfs-mirror 2024-08-17T17:20:34.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all ceph-exporter 2024-08-17T17:20:34.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all iscsi 2024-08-17T17:20:34.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all nfs 2024-08-17T17:20:34.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all nvmeof 2024-08-17T17:20:34.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all node-exporter 2024-08-17T17:20:34.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all prometheus 2024-08-17T17:20:34.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all alertmanager 2024-08-17T17:20:34.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all grafana 2024-08-17T17:20:34.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all loki 2024-08-17T17:20:34.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all promtail 2024-08-17T17:20:34.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[109861]: Upgrade: Finalizing container_image settings 2024-08-17T17:20:34.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[109861]: Upgrade: Complete! 2024-08-17T17:20:34.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[109861]: osdmap e593: 8 total, 8 up, 8 in 2024-08-17T17:20:34.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[109861]: osdmap e594: 8 total, 8 up, 8 in 2024-08-17T17:20:34.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[115078]: Upgrade: Target is version 19.0.0-5836-gcd8bdbeb (squid) 2024-08-17T17:20:34.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[115078]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d'] 2024-08-17T17:20:34.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all mgr 2024-08-17T17:20:34.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all mon 2024-08-17T17:20:34.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all crash 2024-08-17T17:20:34.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all rgw 2024-08-17T17:20:34.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all rbd-mirror 2024-08-17T17:20:34.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all cephfs-mirror 2024-08-17T17:20:34.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all ceph-exporter 2024-08-17T17:20:34.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all iscsi 2024-08-17T17:20:34.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all nfs 2024-08-17T17:20:34.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all nvmeof 2024-08-17T17:20:34.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all node-exporter 2024-08-17T17:20:34.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all prometheus 2024-08-17T17:20:34.450 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all alertmanager 2024-08-17T17:20:34.450 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all grafana 2024-08-17T17:20:34.450 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all loki 2024-08-17T17:20:34.450 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all promtail 2024-08-17T17:20:34.450 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[115078]: Upgrade: Finalizing container_image settings 2024-08-17T17:20:34.450 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[115078]: Upgrade: Complete! 2024-08-17T17:20:34.450 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[115078]: osdmap e593: 8 total, 8 up, 8 in 2024-08-17T17:20:34.450 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:33 smithi136 ceph-mon[115078]: osdmap e594: 8 total, 8 up, 8 in 2024-08-17T17:20:34.859 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestGetSnapShotTimeStamp (1896 ms) 2024-08-17T17:20:34.859 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnapPP 2024-08-17T17:20:34.859 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:34 smithi165 ceph-mon[105517]: pgmap v386: 297 pgs: 297 active+clean; 584 KiB data, 498 MiB used, 715 GiB / 715 GiB avail; 380 KiB/s rd, 26 KiB/s wr, 455 op/s 2024-08-17T17:20:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:34 smithi165 ceph-mon[105517]: osdmap e595: 8 total, 8 up, 8 in 2024-08-17T17:20:35.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:34 smithi136 ceph-mon[109861]: pgmap v386: 297 pgs: 297 active+clean; 584 KiB data, 498 MiB used, 715 GiB / 715 GiB avail; 380 KiB/s rd, 26 KiB/s wr, 455 op/s 2024-08-17T17:20:35.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:34 smithi136 ceph-mon[109861]: osdmap e595: 8 total, 8 up, 8 in 2024-08-17T17:20:35.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:34 smithi136 ceph-mon[115078]: pgmap v386: 297 pgs: 297 active+clean; 584 KiB data, 498 MiB used, 715 GiB / 715 GiB avail; 380 KiB/s rd, 26 KiB/s wr, 455 op/s 2024-08-17T17:20:35.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:34 smithi136 ceph-mon[115078]: osdmap e595: 8 total, 8 up, 8 in 2024-08-17T17:20:35.851 INFO:tasks.workunit.client.0.smithi136.stdout:num snaps is: 1 2024-08-17T17:20:35.851 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 1 2024-08-17T17:20:35.851 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap1 2024-08-17T17:20:35.851 INFO:tasks.workunit.client.0.smithi136.stdout:found snap1 with size 2097152 2024-08-17T17:20:36.863 INFO:tasks.workunit.client.0.smithi136.stdout:num snaps is: 2 2024-08-17T17:20:36.863 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 2 2024-08-17T17:20:36.863 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap1 2024-08-17T17:20:36.863 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap2 2024-08-17T17:20:36.863 INFO:tasks.workunit.client.0.smithi136.stdout:found snap1 with size 2097152 2024-08-17T17:20:36.863 INFO:tasks.workunit.client.0.smithi136.stdout:found snap2 with size 4194304 2024-08-17T17:20:37.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:36 smithi136 ceph-mon[109861]: osdmap e596: 8 total, 8 up, 8 in 2024-08-17T17:20:37.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:36 smithi136 ceph-mon[109861]: pgmap v389: 297 pgs: 297 active+clean; 584 KiB data, 498 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:20:37.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:20:37.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:36 smithi136 ceph-mon[115078]: osdmap e596: 8 total, 8 up, 8 in 2024-08-17T17:20:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:36 smithi136 ceph-mon[115078]: pgmap v389: 297 pgs: 297 active+clean; 584 KiB data, 498 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:20:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:20:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:36 smithi165 ceph-mon[105517]: osdmap e596: 8 total, 8 up, 8 in 2024-08-17T17:20:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:36 smithi165 ceph-mon[105517]: pgmap v389: 297 pgs: 297 active+clean; 584 KiB data, 498 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:20:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:20:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:37.839 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:20:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:20:37] "GET /metrics HTTP/1.1" 200 50711 "" "Prometheus/2.51.0" 2024-08-17T17:20:37.863 INFO:tasks.workunit.client.0.smithi136.stdout:num snaps is: 1 2024-08-17T17:20:37.863 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 1 2024-08-17T17:20:37.863 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap2 2024-08-17T17:20:37.863 INFO:tasks.workunit.client.0.smithi136.stdout:found snap2 with size 4194304 2024-08-17T17:20:38.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:37 smithi136 ceph-mon[109861]: osdmap e597: 8 total, 8 up, 8 in 2024-08-17T17:20:38.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:37 smithi136 ceph-mon[115078]: osdmap e597: 8 total, 8 up, 8 in 2024-08-17T17:20:38.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:37 smithi165 ceph-mon[105517]: osdmap e597: 8 total, 8 up, 8 in 2024-08-17T17:20:38.870 INFO:tasks.workunit.client.0.smithi136.stdout:num snaps is: 0 2024-08-17T17:20:38.870 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 0 2024-08-17T17:20:38.881 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnapPP (4022 ms) 2024-08-17T17:20:38.881 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestGetNameIdSnapPP 2024-08-17T17:20:38.881 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:39.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:38 smithi136 ceph-mon[109861]: osdmap e598: 8 total, 8 up, 8 in 2024-08-17T17:20:39.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:38 smithi136 ceph-mon[109861]: pgmap v392: 297 pgs: 297 active+clean; 584 KiB data, 483 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 6.5 KiB/s wr, 81 op/s 2024-08-17T17:20:39.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:38 smithi136 ceph-mon[115078]: osdmap e598: 8 total, 8 up, 8 in 2024-08-17T17:20:39.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:38 smithi136 ceph-mon[115078]: pgmap v392: 297 pgs: 297 active+clean; 584 KiB data, 483 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 6.5 KiB/s wr, 81 op/s 2024-08-17T17:20:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:38 smithi165 ceph-mon[105517]: osdmap e598: 8 total, 8 up, 8 in 2024-08-17T17:20:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:38 smithi165 ceph-mon[105517]: pgmap v392: 297 pgs: 297 active+clean; 584 KiB data, 483 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 6.5 KiB/s wr, 81 op/s 2024-08-17T17:20:40.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:39 smithi136 ceph-mon[109861]: osdmap e599: 8 total, 8 up, 8 in 2024-08-17T17:20:40.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:39 smithi136 ceph-mon[115078]: osdmap e599: 8 total, 8 up, 8 in 2024-08-17T17:20:40.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:39 smithi165 ceph-mon[105517]: osdmap e599: 8 total, 8 up, 8 in 2024-08-17T17:20:41.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:40 smithi136 ceph-mon[109861]: osdmap e600: 8 total, 8 up, 8 in 2024-08-17T17:20:41.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:40 smithi136 ceph-mon[109861]: pgmap v395: 297 pgs: 297 active+clean; 584 KiB data, 483 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 6.5 KiB/s wr, 81 op/s 2024-08-17T17:20:41.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:40 smithi136 ceph-mon[115078]: osdmap e600: 8 total, 8 up, 8 in 2024-08-17T17:20:41.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:40 smithi136 ceph-mon[115078]: pgmap v395: 297 pgs: 297 active+clean; 584 KiB data, 483 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 6.5 KiB/s wr, 81 op/s 2024-08-17T17:20:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:40 smithi165 ceph-mon[105517]: osdmap e600: 8 total, 8 up, 8 in 2024-08-17T17:20:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:40 smithi165 ceph-mon[105517]: pgmap v395: 297 pgs: 297 active+clean; 584 KiB data, 483 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 6.5 KiB/s wr, 81 op/s 2024-08-17T17:20:42.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:41 smithi136 ceph-mon[109861]: osdmap e601: 8 total, 8 up, 8 in 2024-08-17T17:20:42.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:41 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:42.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:41 smithi136 ceph-mon[115078]: osdmap e601: 8 total, 8 up, 8 in 2024-08-17T17:20:42.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:41 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:41 smithi165 ceph-mon[105517]: osdmap e601: 8 total, 8 up, 8 in 2024-08-17T17:20:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:41 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:43.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:42 smithi136 ceph-mon[109861]: pgmap v397: 297 pgs: 297 active+clean; 584 KiB data, 483 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 4.7 KiB/s wr, 62 op/s 2024-08-17T17:20:43.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:42 smithi136 ceph-mon[109861]: osdmap e602: 8 total, 8 up, 8 in 2024-08-17T17:20:43.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:42 smithi136 ceph-mon[115078]: pgmap v397: 297 pgs: 297 active+clean; 584 KiB data, 483 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 4.7 KiB/s wr, 62 op/s 2024-08-17T17:20:43.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:42 smithi136 ceph-mon[115078]: osdmap e602: 8 total, 8 up, 8 in 2024-08-17T17:20:43.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:42 smithi165 ceph-mon[105517]: pgmap v397: 297 pgs: 297 active+clean; 584 KiB data, 483 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 4.7 KiB/s wr, 62 op/s 2024-08-17T17:20:43.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:42 smithi165 ceph-mon[105517]: osdmap e602: 8 total, 8 up, 8 in 2024-08-17T17:20:44.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:43 smithi136 ceph-mon[109861]: osdmap e603: 8 total, 8 up, 8 in 2024-08-17T17:20:44.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:43 smithi136 ceph-mon[109861]: osdmap e604: 8 total, 8 up, 8 in 2024-08-17T17:20:44.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:43 smithi136 ceph-mon[115078]: osdmap e603: 8 total, 8 up, 8 in 2024-08-17T17:20:44.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:43 smithi136 ceph-mon[115078]: osdmap e604: 8 total, 8 up, 8 in 2024-08-17T17:20:44.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:43 smithi165 ceph-mon[105517]: osdmap e603: 8 total, 8 up, 8 in 2024-08-17T17:20:44.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:43 smithi165 ceph-mon[105517]: osdmap e604: 8 total, 8 up, 8 in 2024-08-17T17:20:44.844 INFO:tasks.workunit.client.0.smithi136.stdout:num snaps is: 0 2024-08-17T17:20:44.844 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 0 2024-08-17T17:20:44.848 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestGetNameIdSnapPP (5967 ms) 2024-08-17T17:20:44.848 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestCreateLsRenameSnapPP 2024-08-17T17:20:44.848 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:45.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:44 smithi136 ceph-mon[109861]: pgmap v401: 297 pgs: 297 active+clean; 584 KiB data, 483 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 4.7 KiB/s wr, 62 op/s 2024-08-17T17:20:45.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:44 smithi136 ceph-mon[109861]: osdmap e605: 8 total, 8 up, 8 in 2024-08-17T17:20:45.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:44 smithi136 ceph-mon[115078]: pgmap v401: 297 pgs: 297 active+clean; 584 KiB data, 483 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 4.7 KiB/s wr, 62 op/s 2024-08-17T17:20:45.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:44 smithi136 ceph-mon[115078]: osdmap e605: 8 total, 8 up, 8 in 2024-08-17T17:20:45.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:44 smithi165 ceph-mon[105517]: pgmap v401: 297 pgs: 297 active+clean; 584 KiB data, 483 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 4.7 KiB/s wr, 62 op/s 2024-08-17T17:20:45.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:44 smithi165 ceph-mon[105517]: osdmap e605: 8 total, 8 up, 8 in 2024-08-17T17:20:45.852 INFO:tasks.workunit.client.0.smithi136.stdout:num snaps is: 1 2024-08-17T17:20:45.852 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 1 2024-08-17T17:20:45.853 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap1 2024-08-17T17:20:45.853 INFO:tasks.workunit.client.0.smithi136.stdout:found snap1 with size 2097152 2024-08-17T17:20:46.859 INFO:tasks.workunit.client.0.smithi136.stdout:num snaps is: 2 2024-08-17T17:20:46.859 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 2 2024-08-17T17:20:46.859 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap1 2024-08-17T17:20:46.860 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap2 2024-08-17T17:20:46.860 INFO:tasks.workunit.client.0.smithi136.stdout:found snap1 with size 2097152 2024-08-17T17:20:46.860 INFO:tasks.workunit.client.0.smithi136.stdout:found snap2 with size 4194304 2024-08-17T17:20:46.864 INFO:tasks.workunit.client.0.smithi136.stdout:num snaps is: 2 2024-08-17T17:20:46.865 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 2 2024-08-17T17:20:46.865 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap1-rename 2024-08-17T17:20:46.865 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap2 2024-08-17T17:20:46.865 INFO:tasks.workunit.client.0.smithi136.stdout:found snap1-rename with size 2097152 2024-08-17T17:20:46.865 INFO:tasks.workunit.client.0.smithi136.stdout:found snap2 with size 4194304 2024-08-17T17:20:47.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:46 smithi136 ceph-mon[109861]: osdmap e606: 8 total, 8 up, 8 in 2024-08-17T17:20:47.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:46 smithi136 ceph-mon[109861]: pgmap v404: 297 pgs: 297 active+clean; 584 KiB data, 483 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:20:47.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:46 smithi136 ceph-mon[115078]: osdmap e606: 8 total, 8 up, 8 in 2024-08-17T17:20:47.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:46 smithi136 ceph-mon[115078]: pgmap v404: 297 pgs: 297 active+clean; 584 KiB data, 483 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:20:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:46 smithi165 ceph-mon[105517]: osdmap e606: 8 total, 8 up, 8 in 2024-08-17T17:20:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:46 smithi165 ceph-mon[105517]: pgmap v404: 297 pgs: 297 active+clean; 584 KiB data, 483 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:20:47.841 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:20:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:20:47] "GET /metrics HTTP/1.1" 200 50711 "" "Prometheus/2.51.0" 2024-08-17T17:20:47.865 INFO:tasks.workunit.client.0.smithi136.stdout:num snaps is: 1 2024-08-17T17:20:47.865 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 1 2024-08-17T17:20:47.866 INFO:tasks.workunit.client.0.smithi136.stdout:snap: snap2-rename 2024-08-17T17:20:47.866 INFO:tasks.workunit.client.0.smithi136.stdout:found snap2-rename with size 4194304 2024-08-17T17:20:48.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:47 smithi136 ceph-mon[109861]: osdmap e607: 8 total, 8 up, 8 in 2024-08-17T17:20:48.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:47 smithi136 ceph-mon[115078]: osdmap e607: 8 total, 8 up, 8 in 2024-08-17T17:20:48.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:47 smithi165 ceph-mon[105517]: osdmap e607: 8 total, 8 up, 8 in 2024-08-17T17:20:48.866 INFO:tasks.workunit.client.0.smithi136.stdout:num snaps is: 0 2024-08-17T17:20:48.866 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 0 2024-08-17T17:20:48.876 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestCreateLsRenameSnapPP (4027 ms) 2024-08-17T17:20:48.876 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.ConcurrentCreatesUnvalidatedPool 2024-08-17T17:20:49.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:48 smithi136 ceph-mon[109861]: osdmap e608: 8 total, 8 up, 8 in 2024-08-17T17:20:49.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:48 smithi136 ceph-mon[109861]: pgmap v407: 297 pgs: 297 active+clean; 584 KiB data, 488 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 6.5 KiB/s wr, 77 op/s 2024-08-17T17:20:49.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:48 smithi136 ceph-mon[115078]: osdmap e608: 8 total, 8 up, 8 in 2024-08-17T17:20:49.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:48 smithi136 ceph-mon[115078]: pgmap v407: 297 pgs: 297 active+clean; 584 KiB data, 488 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 6.5 KiB/s wr, 77 op/s 2024-08-17T17:20:49.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:48 smithi165 ceph-mon[105517]: osdmap e608: 8 total, 8 up, 8 in 2024-08-17T17:20:49.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:48 smithi165 ceph-mon[105517]: pgmap v407: 297 pgs: 297 active+clean; 584 KiB data, 488 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 6.5 KiB/s wr, 77 op/s 2024-08-17T17:20:50.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:49 smithi136 ceph-mon[115078]: osdmap e609: 8 total, 8 up, 8 in 2024-08-17T17:20:50.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:49 smithi136 ceph-mon[109861]: osdmap e609: 8 total, 8 up, 8 in 2024-08-17T17:20:50.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:49 smithi165 ceph-mon[105517]: osdmap e609: 8 total, 8 up, 8 in 2024-08-17T17:20:50.879 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:50.880 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:50.880 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:50.880 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:51.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:50 smithi136 ceph-mon[109861]: osdmap e610: 8 total, 8 up, 8 in 2024-08-17T17:20:51.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:50 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1159248386' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-22","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:51.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:50 smithi136 ceph-mon[109861]: pgmap v410: 329 pgs: 32 unknown, 297 active+clean; 584 KiB data, 488 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 6.5 KiB/s wr, 77 op/s 2024-08-17T17:20:51.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:50 smithi136 ceph-mon[115078]: osdmap e610: 8 total, 8 up, 8 in 2024-08-17T17:20:51.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:50 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1159248386' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-22","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:51.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:50 smithi136 ceph-mon[115078]: pgmap v410: 329 pgs: 32 unknown, 297 active+clean; 584 KiB data, 488 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 6.5 KiB/s wr, 77 op/s 2024-08-17T17:20:51.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:50 smithi165 ceph-mon[105517]: osdmap e610: 8 total, 8 up, 8 in 2024-08-17T17:20:51.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:50 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1159248386' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-22","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:51.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:50 smithi165 ceph-mon[105517]: pgmap v410: 329 pgs: 32 unknown, 297 active+clean; 584 KiB data, 488 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 6.5 KiB/s wr, 77 op/s 2024-08-17T17:20:52.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:51 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:20:52.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:51 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1159248386' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:52.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:51 smithi136 ceph-mon[115078]: osdmap e611: 8 total, 8 up, 8 in 2024-08-17T17:20:52.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:20:52.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:51 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:20:52.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:51 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1159248386' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:52.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:51 smithi136 ceph-mon[109861]: osdmap e611: 8 total, 8 up, 8 in 2024-08-17T17:20:52.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:52.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:20:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:51 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:20:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:51 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1159248386' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:51 smithi165 ceph-mon[105517]: osdmap e611: 8 total, 8 up, 8 in 2024-08-17T17:20:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:20:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:20:53.130 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.ConcurrentCreatesUnvalidatedPool (4255 ms) 2024-08-17T17:20:53.131 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.RemoveFullTry 2024-08-17T17:20:53.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:52 smithi136 ceph-mon[109861]: pgmap v412: 329 pgs: 329 active+clean; 584 KiB data, 489 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 2.7 KiB/s wr, 45 op/s 2024-08-17T17:20:53.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:52 smithi136 ceph-mon[109861]: osdmap e612: 8 total, 8 up, 8 in 2024-08-17T17:20:53.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:52 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:20:53.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:52 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T17:20:53.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:52 smithi136 ceph-mon[115078]: pgmap v412: 329 pgs: 329 active+clean; 584 KiB data, 489 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 2.7 KiB/s wr, 45 op/s 2024-08-17T17:20:53.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:52 smithi136 ceph-mon[115078]: osdmap e612: 8 total, 8 up, 8 in 2024-08-17T17:20:53.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:52 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:20:53.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:52 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T17:20:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:52 smithi165 ceph-mon[105517]: pgmap v412: 329 pgs: 329 active+clean; 584 KiB data, 489 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 2.7 KiB/s wr, 45 op/s 2024-08-17T17:20:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:52 smithi165 ceph-mon[105517]: osdmap e612: 8 total, 8 up, 8 in 2024-08-17T17:20:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:52 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:20:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:52 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T17:20:54.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:53 smithi136 ceph-mon[115078]: osdmap e613: 8 total, 8 up, 8 in 2024-08-17T17:20:54.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:53 smithi136 ceph-mon[115078]: osdmap e614: 8 total, 8 up, 8 in 2024-08-17T17:20:54.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:53 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1188199602' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:54.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:53 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:54.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:53 smithi136 ceph-mon[109861]: osdmap e613: 8 total, 8 up, 8 in 2024-08-17T17:20:54.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:53 smithi136 ceph-mon[109861]: osdmap e614: 8 total, 8 up, 8 in 2024-08-17T17:20:54.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:53 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1188199602' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:54.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:53 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:54.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:53 smithi165 ceph-mon[105517]: osdmap e613: 8 total, 8 up, 8 in 2024-08-17T17:20:54.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:53 smithi165 ceph-mon[105517]: osdmap e614: 8 total, 8 up, 8 in 2024-08-17T17:20:54.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:53 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1188199602' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:54.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:53 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:55.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:54 smithi136 ceph-mon[115078]: pgmap v416: 361 pgs: 32 unknown, 329 active+clean; 584 KiB data, 489 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 2.7 KiB/s wr, 45 op/s 2024-08-17T17:20:55.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:54 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:20:55.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:54 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:55.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:54 smithi136 ceph-mon[115078]: osdmap e615: 8 total, 8 up, 8 in 2024-08-17T17:20:55.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:54 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-23","app": "rbd","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:55.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:54 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-23","app": "rbd","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:55.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:54 smithi136 ceph-mon[109861]: pgmap v416: 361 pgs: 32 unknown, 329 active+clean; 584 KiB data, 489 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 2.7 KiB/s wr, 45 op/s 2024-08-17T17:20:55.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:54 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:20:55.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:54 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:55.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:54 smithi136 ceph-mon[109861]: osdmap e615: 8 total, 8 up, 8 in 2024-08-17T17:20:55.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:54 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-23","app": "rbd","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:55.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:54 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-23","app": "rbd","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:54 smithi165 ceph-mon[105517]: pgmap v416: 361 pgs: 32 unknown, 329 active+clean; 584 KiB data, 489 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 2.7 KiB/s wr, 45 op/s 2024-08-17T17:20:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:54 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:20:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:54 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:54 smithi165 ceph-mon[105517]: osdmap e615: 8 total, 8 up, 8 in 2024-08-17T17:20:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:54 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-23","app": "rbd","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:54 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-23","app": "rbd","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:20:57.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:56 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-23","app": "rbd","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:57.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:56 smithi136 ceph-mon[115078]: osdmap e616: 8 total, 8 up, 8 in 2024-08-17T17:20:57.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:56 smithi136 ceph-mon[115078]: pgmap v419: 361 pgs: 32 unknown, 329 active+clean; 584 KiB data, 489 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:20:57.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:56 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-23","app": "rbd","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:57.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:56 smithi136 ceph-mon[109861]: osdmap e616: 8 total, 8 up, 8 in 2024-08-17T17:20:57.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:56 smithi136 ceph-mon[109861]: pgmap v419: 361 pgs: 32 unknown, 329 active+clean; 584 KiB data, 489 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:20:57.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:56 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-23","app": "rbd","yes_i_really_mean_it": true}]': finished 2024-08-17T17:20:57.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:56 smithi165 ceph-mon[105517]: osdmap e616: 8 total, 8 up, 8 in 2024-08-17T17:20:57.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:56 smithi165 ceph-mon[105517]: pgmap v419: 361 pgs: 32 unknown, 329 active+clean; 584 KiB data, 489 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:20:57.848 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:20:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:20:57] "GET /metrics HTTP/1.1" 200 53073 "" "Prometheus/2.51.0" 2024-08-17T17:20:57.865 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:20:58.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:57 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:20:58.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:57 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T17:20:58.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:57 smithi136 ceph-mon[109861]: osdmap e617: 8 total, 8 up, 8 in 2024-08-17T17:20:58.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:57 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:20:58.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:57 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T17:20:58.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:57 smithi136 ceph-mon[115078]: osdmap e617: 8 total, 8 up, 8 in 2024-08-17T17:20:58.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:20:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:20:57] ENGINE Bus STOPPING 2024-08-17T17:20:58.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:20:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:20:57] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-08-17T17:20:58.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:20:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:20:57] ENGINE Bus STOPPED 2024-08-17T17:20:58.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:20:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:20:57] ENGINE Bus STARTING 2024-08-17T17:20:58.198 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:20:58 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:20:58] ENGINE Serving on http://:::9283 2024-08-17T17:20:58.198 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:20:58 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:20:58] ENGINE Bus STARTED 2024-08-17T17:20:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:57 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:20:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:57 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T17:20:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:57 smithi165 ceph-mon[105517]: osdmap e617: 8 total, 8 up, 8 in 2024-08-17T17:20:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:58 smithi165 ceph-mon[105517]: osdmap e618: 8 total, 8 up, 8 in 2024-08-17T17:20:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:58 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' 2024-08-17T17:20:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:58 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:20:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:58 smithi165 ceph-mon[105517]: pgmap v422: 361 pgs: 361 active+clean; 584 KiB data, 477 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 8.5 KiB/s wr, 144 op/s 2024-08-17T17:20:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:58 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-23", "field": "max_bytes", "val": "10485760"} : dispatch 2024-08-17T17:20:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:58 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-23", "field": "max_bytes", "val": "10485760"} : dispatch 2024-08-17T17:20:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:58 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-08-17T17:20:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:58 smithi165 ceph-mon[105517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:20:59.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:58 smithi136 ceph-mon[115078]: osdmap e618: 8 total, 8 up, 8 in 2024-08-17T17:20:59.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:58 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' 2024-08-17T17:20:59.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:58 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:20:59.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:58 smithi136 ceph-mon[115078]: pgmap v422: 361 pgs: 361 active+clean; 584 KiB data, 477 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 8.5 KiB/s wr, 144 op/s 2024-08-17T17:20:59.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:58 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-23", "field": "max_bytes", "val": "10485760"} : dispatch 2024-08-17T17:20:59.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:58 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-23", "field": "max_bytes", "val": "10485760"} : dispatch 2024-08-17T17:20:59.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:58 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-08-17T17:20:59.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:58 smithi136 ceph-mon[115078]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:20:59.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:58 smithi136 ceph-mon[109861]: osdmap e618: 8 total, 8 up, 8 in 2024-08-17T17:20:59.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:58 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' 2024-08-17T17:20:59.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:58 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:20:59.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:58 smithi136 ceph-mon[109861]: pgmap v422: 361 pgs: 361 active+clean; 584 KiB data, 477 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 8.5 KiB/s wr, 144 op/s 2024-08-17T17:20:59.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:58 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-23", "field": "max_bytes", "val": "10485760"} : dispatch 2024-08-17T17:20:59.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:58 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-23", "field": "max_bytes", "val": "10485760"} : dispatch 2024-08-17T17:20:59.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:58 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-08-17T17:20:59.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:58 smithi136 ceph-mon[109861]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:20:59.767 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'sleep 60' 2024-08-17T17:21:00.015 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:21:00.137 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:59 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-23", "field": "max_bytes", "val": "10485760"}]': finished 2024-08-17T17:21:00.138 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:59 smithi136 ceph-mon[109861]: osdmap e619: 8 total, 8 up, 8 in 2024-08-17T17:21:00.138 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:20:59 smithi136 ceph-mon[109861]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:21:00.138 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:59 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-23", "field": "max_bytes", "val": "10485760"}]': finished 2024-08-17T17:21:00.138 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:59 smithi136 ceph-mon[115078]: osdmap e619: 8 total, 8 up, 8 in 2024-08-17T17:21:00.138 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:20:59 smithi136 ceph-mon[115078]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:21:00.194 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:59 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-23", "field": "max_bytes", "val": "10485760"}]': finished 2024-08-17T17:21:00.194 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:59 smithi165 ceph-mon[105517]: osdmap e619: 8 total, 8 up, 8 in 2024-08-17T17:21:00.194 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:20:59 smithi165 ceph-mon[105517]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:21:01.144 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:00 smithi136 ceph-mon[109861]: pgmap v424: 361 pgs: 361 active+clean; 584 KiB data, 477 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 8.4 KiB/s wr, 142 op/s 2024-08-17T17:21:01.144 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:00 smithi136 ceph-mon[115078]: pgmap v424: 361 pgs: 361 active+clean; 584 KiB data, 477 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 8.4 KiB/s wr, 142 op/s 2024-08-17T17:21:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:00 smithi165 ceph-mon[105517]: pgmap v424: 361 pgs: 361 active+clean; 584 KiB data, 477 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 8.4 KiB/s wr, 142 op/s 2024-08-17T17:21:03.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:03 smithi136 ceph-mon[115078]: pgmap v425: 361 pgs: 361 active+clean; 3.6 MiB data, 487 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 519 KiB/s wr, 106 op/s 2024-08-17T17:21:03.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:03.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:03.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:21:03.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:21:03.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:03.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:03 smithi136 ceph-mon[109861]: pgmap v425: 361 pgs: 361 active+clean; 3.6 MiB data, 487 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 519 KiB/s wr, 106 op/s 2024-08-17T17:21:03.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:03.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:03.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:21:03.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:21:03.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:03.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:03 smithi165 ceph-mon[105517]: pgmap v425: 361 pgs: 361 active+clean; 3.6 MiB data, 487 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 519 KiB/s wr, 106 op/s 2024-08-17T17:21:03.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:03.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:03.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:21:03.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:21:03.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:05.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:04 smithi136 ceph-mon[109861]: osdmap e620: 8 total, 8 up, 8 in 2024-08-17T17:21:05.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:04 smithi136 ceph-mon[109861]: pgmap v427: 361 pgs: 361 active+clean; 3.6 MiB data, 487 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 510 KiB/s wr, 9 op/s 2024-08-17T17:21:05.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:04 smithi136 ceph-mon[115078]: osdmap e620: 8 total, 8 up, 8 in 2024-08-17T17:21:05.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:04 smithi136 ceph-mon[115078]: pgmap v427: 361 pgs: 361 active+clean; 3.6 MiB data, 487 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 510 KiB/s wr, 9 op/s 2024-08-17T17:21:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:04 smithi165 ceph-mon[105517]: osdmap e620: 8 total, 8 up, 8 in 2024-08-17T17:21:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:04 smithi165 ceph-mon[105517]: pgmap v427: 361 pgs: 361 active+clean; 3.6 MiB data, 487 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 510 KiB/s wr, 9 op/s 2024-08-17T17:21:06.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.0", "id": [7, 5]} : dispatch 2024-08-17T17:21:06.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.c", "id": [7, 4]} : dispatch 2024-08-17T17:21:06.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.d", "id": [7, 0]} : dispatch 2024-08-17T17:21:06.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1c", "id": [7, 5]} : dispatch 2024-08-17T17:21:06.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.16", "id": [7, 0]} : dispatch 2024-08-17T17:21:06.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1a", "id": [7, 2]} : dispatch 2024-08-17T17:21:06.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1c", "id": [7, 0]} : dispatch 2024-08-17T17:21:06.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.d", "id": [0, 2]} : dispatch 2024-08-17T17:21:06.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.11", "id": [0, 2]} : dispatch 2024-08-17T17:21:06.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.0", "id": [7, 5]} : dispatch 2024-08-17T17:21:06.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.c", "id": [7, 4]} : dispatch 2024-08-17T17:21:06.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.d", "id": [7, 0]} : dispatch 2024-08-17T17:21:06.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1c", "id": [7, 5]} : dispatch 2024-08-17T17:21:06.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.16", "id": [7, 0]} : dispatch 2024-08-17T17:21:06.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1a", "id": [7, 2]} : dispatch 2024-08-17T17:21:06.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1c", "id": [7, 0]} : dispatch 2024-08-17T17:21:06.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.d", "id": [0, 2]} : dispatch 2024-08-17T17:21:06.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.11", "id": [0, 2]} : dispatch 2024-08-17T17:21:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.0", "id": [7, 5]} : dispatch 2024-08-17T17:21:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.c", "id": [7, 4]} : dispatch 2024-08-17T17:21:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.d", "id": [7, 0]} : dispatch 2024-08-17T17:21:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1c", "id": [7, 5]} : dispatch 2024-08-17T17:21:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.16", "id": [7, 0]} : dispatch 2024-08-17T17:21:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1a", "id": [7, 2]} : dispatch 2024-08-17T17:21:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1c", "id": [7, 0]} : dispatch 2024-08-17T17:21:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.d", "id": [0, 2]} : dispatch 2024-08-17T17:21:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.11", "id": [0, 2]} : dispatch 2024-08-17T17:21:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:06 smithi165 ceph-mon[105517]: pgmap v428: 361 pgs: 361 active+clean; 3.6 MiB data, 487 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 385 KiB/s wr, 7 op/s 2024-08-17T17:21:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.0", "id": [7, 5]}]': finished 2024-08-17T17:21:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.c", "id": [7, 4]}]': finished 2024-08-17T17:21:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.d", "id": [7, 0]}]': finished 2024-08-17T17:21:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1c", "id": [7, 5]}]': finished 2024-08-17T17:21:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.16", "id": [7, 0]}]': finished 2024-08-17T17:21:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1a", "id": [7, 2]}]': finished 2024-08-17T17:21:07.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1c", "id": [7, 0]}]': finished 2024-08-17T17:21:07.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.d", "id": [0, 2]}]': finished 2024-08-17T17:21:07.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.11", "id": [0, 2]}]': finished 2024-08-17T17:21:07.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:06 smithi165 ceph-mon[105517]: osdmap e621: 8 total, 8 up, 8 in 2024-08-17T17:21:07.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:07.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:21:07.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:06 smithi165 ceph-mon[105517]: osdmap e622: 8 total, 8 up, 8 in 2024-08-17T17:21:07.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[109861]: pgmap v428: 361 pgs: 361 active+clean; 3.6 MiB data, 487 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 385 KiB/s wr, 7 op/s 2024-08-17T17:21:07.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.0", "id": [7, 5]}]': finished 2024-08-17T17:21:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.c", "id": [7, 4]}]': finished 2024-08-17T17:21:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.d", "id": [7, 0]}]': finished 2024-08-17T17:21:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1c", "id": [7, 5]}]': finished 2024-08-17T17:21:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.16", "id": [7, 0]}]': finished 2024-08-17T17:21:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1a", "id": [7, 2]}]': finished 2024-08-17T17:21:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1c", "id": [7, 0]}]': finished 2024-08-17T17:21:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.d", "id": [0, 2]}]': finished 2024-08-17T17:21:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.11", "id": [0, 2]}]': finished 2024-08-17T17:21:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[109861]: osdmap e621: 8 total, 8 up, 8 in 2024-08-17T17:21:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:21:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[109861]: osdmap e622: 8 total, 8 up, 8 in 2024-08-17T17:21:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[115078]: pgmap v428: 361 pgs: 361 active+clean; 3.6 MiB data, 487 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 385 KiB/s wr, 7 op/s 2024-08-17T17:21:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.0", "id": [7, 5]}]': finished 2024-08-17T17:21:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.c", "id": [7, 4]}]': finished 2024-08-17T17:21:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.d", "id": [7, 0]}]': finished 2024-08-17T17:21:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1c", "id": [7, 5]}]': finished 2024-08-17T17:21:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.16", "id": [7, 0]}]': finished 2024-08-17T17:21:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1a", "id": [7, 2]}]': finished 2024-08-17T17:21:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1c", "id": [7, 0]}]': finished 2024-08-17T17:21:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.d", "id": [0, 2]}]': finished 2024-08-17T17:21:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.11", "id": [0, 2]}]': finished 2024-08-17T17:21:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[115078]: osdmap e621: 8 total, 8 up, 8 in 2024-08-17T17:21:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:21:07.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:06 smithi136 ceph-mon[115078]: osdmap e622: 8 total, 8 up, 8 in 2024-08-17T17:21:07.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:21:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:21:07] "GET /metrics HTTP/1.1" 200 55532 "" "Prometheus/2.51.0" 2024-08-17T17:21:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:08 smithi165 ceph-mon[105517]: pgmap v431: 361 pgs: 9 peering, 352 active+clean; 8.6 MiB data, 519 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 KiB/s wr, 2 op/s 2024-08-17T17:21:09.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:08 smithi136 ceph-mon[109861]: pgmap v431: 361 pgs: 9 peering, 352 active+clean; 8.6 MiB data, 519 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 KiB/s wr, 2 op/s 2024-08-17T17:21:09.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:08 smithi136 ceph-mon[115078]: pgmap v431: 361 pgs: 9 peering, 352 active+clean; 8.6 MiB data, 519 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 KiB/s wr, 2 op/s 2024-08-17T17:21:11.249 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:10 smithi136 ceph-mon[109861]: pgmap v432: 361 pgs: 9 peering, 352 active+clean; 8.6 MiB data, 519 MiB used, 715 GiB / 715 GiB avail; 845 B/s rd, 846 KiB/s wr, 2 op/s 2024-08-17T17:21:11.249 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:10 smithi136 ceph-mon[115078]: pgmap v432: 361 pgs: 9 peering, 352 active+clean; 8.6 MiB data, 519 MiB used, 715 GiB / 715 GiB avail; 845 B/s rd, 846 KiB/s wr, 2 op/s 2024-08-17T17:21:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:10 smithi165 ceph-mon[105517]: pgmap v432: 361 pgs: 9 peering, 352 active+clean; 8.6 MiB data, 519 MiB used, 715 GiB / 715 GiB avail; 845 B/s rd, 846 KiB/s wr, 2 op/s 2024-08-17T17:21:13.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:12 smithi165 ceph-mon[105517]: pgmap v433: 361 pgs: 361 active+clean; 14 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.3 MiB/s wr, 4 op/s; 8 B/s, 10 keys/s, 0 objects/s recovering 2024-08-17T17:21:13.441 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:12 smithi136 ceph-mon[109861]: pgmap v433: 361 pgs: 361 active+clean; 14 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.3 MiB/s wr, 4 op/s; 8 B/s, 10 keys/s, 0 objects/s recovering 2024-08-17T17:21:13.442 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:12 smithi136 ceph-mon[115078]: pgmap v433: 361 pgs: 361 active+clean; 14 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.3 MiB/s wr, 4 op/s; 8 B/s, 10 keys/s, 0 objects/s recovering 2024-08-17T17:21:14.058 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.RemoveFullTry (20928 ms) 2024-08-17T17:21:14.058 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.RemoveFullTryDataPool 2024-08-17T17:21:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:13 smithi165 ceph-mon[105517]: pool 'test-librbd-smithi136-87100-23' is full (reached quota's max_bytes: 10 MiB) 2024-08-17T17:21:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:13 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-08-17T17:21:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:13 smithi165 ceph-mon[105517]: osdmap e623: 8 total, 8 up, 8 in 2024-08-17T17:21:14.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:13 smithi136 ceph-mon[109861]: pool 'test-librbd-smithi136-87100-23' is full (reached quota's max_bytes: 10 MiB) 2024-08-17T17:21:14.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:13 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-08-17T17:21:14.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:13 smithi136 ceph-mon[109861]: osdmap e623: 8 total, 8 up, 8 in 2024-08-17T17:21:14.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:13 smithi136 ceph-mon[115078]: pool 'test-librbd-smithi136-87100-23' is full (reached quota's max_bytes: 10 MiB) 2024-08-17T17:21:14.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:13 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-08-17T17:21:14.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:13 smithi136 ceph-mon[115078]: osdmap e623: 8 total, 8 up, 8 in 2024-08-17T17:21:15.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:14 smithi165 ceph-mon[105517]: pgmap v435: 361 pgs: 361 active+clean; 14 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.3 MiB/s wr, 4 op/s; 8 B/s, 10 keys/s, 0 objects/s recovering 2024-08-17T17:21:15.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:14 smithi136 ceph-mon[109861]: pgmap v435: 361 pgs: 361 active+clean; 14 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.3 MiB/s wr, 4 op/s; 8 B/s, 10 keys/s, 0 objects/s recovering 2024-08-17T17:21:15.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:14 smithi136 ceph-mon[115078]: pgmap v435: 361 pgs: 361 active+clean; 14 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.3 MiB/s wr, 4 op/s; 8 B/s, 10 keys/s, 0 objects/s recovering 2024-08-17T17:21:16.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:15 smithi165 ceph-mon[105517]: osdmap e624: 8 total, 8 up, 8 in 2024-08-17T17:21:16.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:15 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4260290133' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:21:16.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:15 smithi136 ceph-mon[109861]: osdmap e624: 8 total, 8 up, 8 in 2024-08-17T17:21:16.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:15 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4260290133' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:21:16.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:15 smithi136 ceph-mon[115078]: osdmap e624: 8 total, 8 up, 8 in 2024-08-17T17:21:16.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:15 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4260290133' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:21:17.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:16 smithi165 ceph-mon[105517]: pgmap v437: 393 pgs: 32 unknown, 361 active+clean; 14 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 640 KiB/s wr, 2 op/s; 8 B/s, 10 keys/s, 0 objects/s recovering 2024-08-17T17:21:17.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:16 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:21:17.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:16 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4260290133' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:21:17.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:16 smithi165 ceph-mon[105517]: osdmap e625: 8 total, 8 up, 8 in 2024-08-17T17:21:17.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:16 smithi136 ceph-mon[109861]: pgmap v437: 393 pgs: 32 unknown, 361 active+clean; 14 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 640 KiB/s wr, 2 op/s; 8 B/s, 10 keys/s, 0 objects/s recovering 2024-08-17T17:21:17.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:16 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:21:17.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:16 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4260290133' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:21:17.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:16 smithi136 ceph-mon[109861]: osdmap e625: 8 total, 8 up, 8 in 2024-08-17T17:21:17.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:16 smithi136 ceph-mon[115078]: pgmap v437: 393 pgs: 32 unknown, 361 active+clean; 14 MiB data, 532 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 640 KiB/s wr, 2 op/s; 8 B/s, 10 keys/s, 0 objects/s recovering 2024-08-17T17:21:17.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:16 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:21:17.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:16 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4260290133' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:21:17.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:16 smithi136 ceph-mon[115078]: osdmap e625: 8 total, 8 up, 8 in 2024-08-17T17:21:17.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:21:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:21:17] "GET /metrics HTTP/1.1" 200 55532 "" "Prometheus/2.51.0" 2024-08-17T17:21:18.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:18 smithi165 ceph-mon[105517]: osdmap e626: 8 total, 8 up, 8 in 2024-08-17T17:21:18.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:18 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2773686289' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:21:18.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:18 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:21:18.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:17 smithi136 ceph-mon[109861]: osdmap e626: 8 total, 8 up, 8 in 2024-08-17T17:21:18.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:17 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2773686289' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:21:18.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:17 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:21:18.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:17 smithi136 ceph-mon[115078]: osdmap e626: 8 total, 8 up, 8 in 2024-08-17T17:21:18.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:17 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2773686289' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:21:18.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:17 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:21:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:18 smithi165 ceph-mon[105517]: pgmap v440: 425 pgs: 32 unknown, 393 active+clean; 584 KiB data, 504 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 507 KiB/s wr, 39 op/s 2024-08-17T17:21:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:18 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:21:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:18 smithi165 ceph-mon[105517]: osdmap e627: 8 total, 8 up, 8 in 2024-08-17T17:21:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:18 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-24","app": "rbd","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:21:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:18 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-24","app": "rbd","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:21:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:18 smithi165 ceph-mon[105517]: pool 'test-librbd-smithi136-87100-23' no longer out of quota; removing NO_QUOTA flag 2024-08-17T17:21:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:18 smithi165 ceph-mon[105517]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-08-17T17:21:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:18 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-24","app": "rbd","yes_i_really_mean_it": true}]': finished 2024-08-17T17:21:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:18 smithi165 ceph-mon[105517]: osdmap e628: 8 total, 8 up, 8 in 2024-08-17T17:21:19.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:18 smithi136 ceph-mon[109861]: pgmap v440: 425 pgs: 32 unknown, 393 active+clean; 584 KiB data, 504 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 507 KiB/s wr, 39 op/s 2024-08-17T17:21:19.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:18 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:21:19.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:18 smithi136 ceph-mon[109861]: osdmap e627: 8 total, 8 up, 8 in 2024-08-17T17:21:19.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:18 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-24","app": "rbd","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:21:19.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:18 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-24","app": "rbd","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:21:19.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:18 smithi136 ceph-mon[109861]: pool 'test-librbd-smithi136-87100-23' no longer out of quota; removing NO_QUOTA flag 2024-08-17T17:21:19.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:18 smithi136 ceph-mon[109861]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-08-17T17:21:19.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:18 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-24","app": "rbd","yes_i_really_mean_it": true}]': finished 2024-08-17T17:21:19.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:18 smithi136 ceph-mon[109861]: osdmap e628: 8 total, 8 up, 8 in 2024-08-17T17:21:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:18 smithi136 ceph-mon[115078]: pgmap v440: 425 pgs: 32 unknown, 393 active+clean; 584 KiB data, 504 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 507 KiB/s wr, 39 op/s 2024-08-17T17:21:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:18 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:21:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:18 smithi136 ceph-mon[115078]: osdmap e627: 8 total, 8 up, 8 in 2024-08-17T17:21:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:18 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-24","app": "rbd","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:21:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:18 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-24","app": "rbd","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:21:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:18 smithi136 ceph-mon[115078]: pool 'test-librbd-smithi136-87100-23' no longer out of quota; removing NO_QUOTA flag 2024-08-17T17:21:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:18 smithi136 ceph-mon[115078]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-08-17T17:21:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:18 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-24","app": "rbd","yes_i_really_mean_it": true}]': finished 2024-08-17T17:21:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:18 smithi136 ceph-mon[115078]: osdmap e628: 8 total, 8 up, 8 in 2024-08-17T17:21:20.870 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:21:21.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:20 smithi136 ceph-mon[109861]: osdmap e629: 8 total, 8 up, 8 in 2024-08-17T17:21:21.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:20 smithi136 ceph-mon[109861]: pgmap v444: 425 pgs: 32 unknown, 393 active+clean; 584 KiB data, 504 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 525 KiB/s wr, 40 op/s 2024-08-17T17:21:21.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:20 smithi136 ceph-mon[115078]: osdmap e629: 8 total, 8 up, 8 in 2024-08-17T17:21:21.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:20 smithi136 ceph-mon[115078]: pgmap v444: 425 pgs: 32 unknown, 393 active+clean; 584 KiB data, 504 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 525 KiB/s wr, 40 op/s 2024-08-17T17:21:21.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:21:20 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:21:20] ENGINE Bus STOPPING 2024-08-17T17:21:21.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:21:20 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:21:20] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-08-17T17:21:21.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:21:20 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:21:20] ENGINE Bus STOPPED 2024-08-17T17:21:21.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:21:20 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:21:20] ENGINE Bus STARTING 2024-08-17T17:21:21.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:21:21 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:21:21] ENGINE Serving on http://:::9283 2024-08-17T17:21:21.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:21:21 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:21:21] ENGINE Bus STARTED 2024-08-17T17:21:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:20 smithi165 ceph-mon[105517]: osdmap e629: 8 total, 8 up, 8 in 2024-08-17T17:21:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:20 smithi165 ceph-mon[105517]: pgmap v444: 425 pgs: 32 unknown, 393 active+clean; 584 KiB data, 504 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 525 KiB/s wr, 40 op/s 2024-08-17T17:21:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:21 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:21:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:21 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T17:21:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:21 smithi136 ceph-mon[109861]: osdmap e630: 8 total, 8 up, 8 in 2024-08-17T17:21:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:21 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' 2024-08-17T17:21:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:21:22.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:21:22.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:21:22.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:22.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-08-17T17:21:22.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:21 smithi136 ceph-mon[109861]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:21:22.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:21:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:21 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:21:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:21 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T17:21:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:21 smithi136 ceph-mon[115078]: osdmap e630: 8 total, 8 up, 8 in 2024-08-17T17:21:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:21 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' 2024-08-17T17:21:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:21:22.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:21:22.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:21:22.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:22.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-08-17T17:21:22.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:21 smithi136 ceph-mon[115078]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:21:22.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:21:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:21 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:21:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:21 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T17:21:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:21 smithi165 ceph-mon[105517]: osdmap e630: 8 total, 8 up, 8 in 2024-08-17T17:21:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:21 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' 2024-08-17T17:21:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:21:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:21:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:21:22.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:22.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-08-17T17:21:22.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:21 smithi165 ceph-mon[105517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:21:22.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:21:23.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:22 smithi136 ceph-mon[109861]: osdmap e631: 8 total, 8 up, 8 in 2024-08-17T17:21:23.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:22 smithi136 ceph-mon[109861]: pgmap v447: 425 pgs: 425 active+clean; 584 KiB data, 501 MiB used, 715 GiB / 715 GiB avail; 262 B/s rd, 786 B/s wr, 1 op/s 2024-08-17T17:21:23.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:22 smithi136 ceph-mon[115078]: osdmap e631: 8 total, 8 up, 8 in 2024-08-17T17:21:23.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:22 smithi136 ceph-mon[115078]: pgmap v447: 425 pgs: 425 active+clean; 584 KiB data, 501 MiB used, 715 GiB / 715 GiB avail; 262 B/s rd, 786 B/s wr, 1 op/s 2024-08-17T17:21:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:22 smithi165 ceph-mon[105517]: osdmap e631: 8 total, 8 up, 8 in 2024-08-17T17:21:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:22 smithi165 ceph-mon[105517]: pgmap v447: 425 pgs: 425 active+clean; 584 KiB data, 501 MiB used, 715 GiB / 715 GiB avail; 262 B/s rd, 786 B/s wr, 1 op/s 2024-08-17T17:21:24.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:23 smithi136 ceph-mon[115078]: osdmap e632: 8 total, 8 up, 8 in 2024-08-17T17:21:24.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:23 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-25", "field": "max_bytes", "val": "10485760"} : dispatch 2024-08-17T17:21:24.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:23 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-25", "field": "max_bytes", "val": "10485760"} : dispatch 2024-08-17T17:21:24.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:23 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-25", "field": "max_bytes", "val": "10485760"}]': finished 2024-08-17T17:21:24.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:23 smithi136 ceph-mon[115078]: osdmap e633: 8 total, 8 up, 8 in 2024-08-17T17:21:24.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:23 smithi136 ceph-mon[109861]: osdmap e632: 8 total, 8 up, 8 in 2024-08-17T17:21:24.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:23 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-25", "field": "max_bytes", "val": "10485760"} : dispatch 2024-08-17T17:21:24.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:23 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-25", "field": "max_bytes", "val": "10485760"} : dispatch 2024-08-17T17:21:24.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:23 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-25", "field": "max_bytes", "val": "10485760"}]': finished 2024-08-17T17:21:24.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:23 smithi136 ceph-mon[109861]: osdmap e633: 8 total, 8 up, 8 in 2024-08-17T17:21:24.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:23 smithi165 ceph-mon[105517]: osdmap e632: 8 total, 8 up, 8 in 2024-08-17T17:21:24.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:23 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-25", "field": "max_bytes", "val": "10485760"} : dispatch 2024-08-17T17:21:24.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:23 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-25", "field": "max_bytes", "val": "10485760"} : dispatch 2024-08-17T17:21:24.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:23 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-25", "field": "max_bytes", "val": "10485760"}]': finished 2024-08-17T17:21:24.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:23 smithi165 ceph-mon[105517]: osdmap e633: 8 total, 8 up, 8 in 2024-08-17T17:21:25.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:24 smithi136 ceph-mon[109861]: pgmap v450: 425 pgs: 425 active+clean; 584 KiB data, 501 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 767 B/s wr, 0 op/s 2024-08-17T17:21:25.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:24 smithi136 ceph-mon[115078]: pgmap v450: 425 pgs: 425 active+clean; 584 KiB data, 501 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 767 B/s wr, 0 op/s 2024-08-17T17:21:25.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:24 smithi165 ceph-mon[105517]: pgmap v450: 425 pgs: 425 active+clean; 584 KiB data, 501 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 767 B/s wr, 0 op/s 2024-08-17T17:21:27.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:26 smithi136 ceph-mon[109861]: pgmap v451: 425 pgs: 425 active+clean; 584 KiB data, 501 MiB used, 715 GiB / 715 GiB avail; 203 B/s rd, 609 B/s wr, 0 op/s 2024-08-17T17:21:27.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:26 smithi136 ceph-mon[115078]: pgmap v451: 425 pgs: 425 active+clean; 584 KiB data, 501 MiB used, 715 GiB / 715 GiB avail; 203 B/s rd, 609 B/s wr, 0 op/s 2024-08-17T17:21:27.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:26 smithi165 ceph-mon[105517]: pgmap v451: 425 pgs: 425 active+clean; 584 KiB data, 501 MiB used, 715 GiB / 715 GiB avail; 203 B/s rd, 609 B/s wr, 0 op/s 2024-08-17T17:21:27.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:21:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:21:27] "GET /metrics HTTP/1.1" 200 60255 "" "Prometheus/2.51.0" 2024-08-17T17:21:29.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:28 smithi136 ceph-mon[109861]: pgmap v452: 425 pgs: 425 active+clean; 3.6 MiB data, 510 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 512 KiB/s wr, 9 op/s 2024-08-17T17:21:29.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:28 smithi136 ceph-mon[109861]: osdmap e634: 8 total, 8 up, 8 in 2024-08-17T17:21:29.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:28 smithi136 ceph-mon[115078]: pgmap v452: 425 pgs: 425 active+clean; 3.6 MiB data, 510 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 512 KiB/s wr, 9 op/s 2024-08-17T17:21:29.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:28 smithi136 ceph-mon[115078]: osdmap e634: 8 total, 8 up, 8 in 2024-08-17T17:21:29.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:28 smithi165 ceph-mon[105517]: pgmap v452: 425 pgs: 425 active+clean; 3.6 MiB data, 510 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 512 KiB/s wr, 9 op/s 2024-08-17T17:21:29.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:28 smithi165 ceph-mon[105517]: osdmap e634: 8 total, 8 up, 8 in 2024-08-17T17:21:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:30 smithi165 ceph-mon[105517]: pgmap v454: 425 pgs: 425 active+clean; 3.6 MiB data, 510 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 439 KiB/s wr, 8 op/s 2024-08-17T17:21:31.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:30 smithi136 ceph-mon[109861]: pgmap v454: 425 pgs: 425 active+clean; 3.6 MiB data, 510 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 439 KiB/s wr, 8 op/s 2024-08-17T17:21:31.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:30 smithi136 ceph-mon[115078]: pgmap v454: 425 pgs: 425 active+clean; 3.6 MiB data, 510 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 439 KiB/s wr, 8 op/s 2024-08-17T17:21:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:32 smithi165 ceph-mon[105517]: pgmap v455: 425 pgs: 425 active+clean; 8.6 MiB data, 525 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 1018 KiB/s wr, 8 op/s 2024-08-17T17:21:33.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:32 smithi136 ceph-mon[109861]: pgmap v455: 425 pgs: 425 active+clean; 8.6 MiB data, 525 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 1018 KiB/s wr, 8 op/s 2024-08-17T17:21:33.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:32 smithi136 ceph-mon[115078]: pgmap v455: 425 pgs: 425 active+clean; 8.6 MiB data, 525 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 1018 KiB/s wr, 8 op/s 2024-08-17T17:21:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:34 smithi165 ceph-mon[105517]: pgmap v456: 425 pgs: 425 active+clean; 8.6 MiB data, 525 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 820 KiB/s wr, 7 op/s 2024-08-17T17:21:35.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:34 smithi136 ceph-mon[109861]: pgmap v456: 425 pgs: 425 active+clean; 8.6 MiB data, 525 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 820 KiB/s wr, 7 op/s 2024-08-17T17:21:35.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:34 smithi136 ceph-mon[115078]: pgmap v456: 425 pgs: 425 active+clean; 8.6 MiB data, 525 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 820 KiB/s wr, 7 op/s 2024-08-17T17:21:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:36 smithi165 ceph-mon[105517]: pgmap v457: 425 pgs: 425 active+clean; 8.6 MiB data, 525 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 820 KiB/s wr, 7 op/s 2024-08-17T17:21:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:21:37.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:36 smithi136 ceph-mon[109861]: pgmap v457: 425 pgs: 425 active+clean; 8.6 MiB data, 525 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 820 KiB/s wr, 7 op/s 2024-08-17T17:21:37.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:21:37.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:36 smithi136 ceph-mon[115078]: pgmap v457: 425 pgs: 425 active+clean; 8.6 MiB data, 525 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 820 KiB/s wr, 7 op/s 2024-08-17T17:21:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:21:37.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:21:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:21:37] "GET /metrics HTTP/1.1" 200 60339 "" "Prometheus/2.51.0" 2024-08-17T17:21:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:38 smithi165 ceph-mon[105517]: pgmap v458: 425 pgs: 425 active+clean; 14 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 MiB/s wr, 3 op/s 2024-08-17T17:21:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:38 smithi165 ceph-mon[105517]: pool 'test-librbd-smithi136-87100-25' is full (reached quota's max_bytes: 10 MiB) 2024-08-17T17:21:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:38 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-08-17T17:21:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:38 smithi165 ceph-mon[105517]: osdmap e635: 8 total, 8 up, 8 in 2024-08-17T17:21:39.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:38 smithi136 ceph-mon[109861]: pgmap v458: 425 pgs: 425 active+clean; 14 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 MiB/s wr, 3 op/s 2024-08-17T17:21:39.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:38 smithi136 ceph-mon[109861]: pool 'test-librbd-smithi136-87100-25' is full (reached quota's max_bytes: 10 MiB) 2024-08-17T17:21:39.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:38 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-08-17T17:21:39.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:38 smithi136 ceph-mon[109861]: osdmap e635: 8 total, 8 up, 8 in 2024-08-17T17:21:39.450 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:38 smithi136 ceph-mon[115078]: pgmap v458: 425 pgs: 425 active+clean; 14 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 MiB/s wr, 3 op/s 2024-08-17T17:21:39.450 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:38 smithi136 ceph-mon[115078]: pool 'test-librbd-smithi136-87100-25' is full (reached quota's max_bytes: 10 MiB) 2024-08-17T17:21:39.450 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:38 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-08-17T17:21:39.450 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:38 smithi136 ceph-mon[115078]: osdmap e635: 8 total, 8 up, 8 in 2024-08-17T17:21:39.976 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:21:40.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:39 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-24", "field": "max_objects", "val": "6"} : dispatch 2024-08-17T17:21:40.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:39 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-24", "field": "max_objects", "val": "6"} : dispatch 2024-08-17T17:21:40.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:39 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-24", "field": "max_objects", "val": "6"} : dispatch 2024-08-17T17:21:40.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:39 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-24", "field": "max_objects", "val": "6"} : dispatch 2024-08-17T17:21:40.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:39 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-24", "field": "max_objects", "val": "6"} : dispatch 2024-08-17T17:21:40.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:39 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-24", "field": "max_objects", "val": "6"} : dispatch 2024-08-17T17:21:40.990 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:21:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:40 smithi165 ceph-mon[105517]: pgmap v460: 425 pgs: 425 active+clean; 14 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 MiB/s wr, 3 op/s 2024-08-17T17:21:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:40 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-24", "field": "max_objects", "val": "6"}]': finished 2024-08-17T17:21:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:40 smithi165 ceph-mon[105517]: osdmap e636: 8 total, 8 up, 8 in 2024-08-17T17:21:41.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:40 smithi136 ceph-mon[109861]: pgmap v460: 425 pgs: 425 active+clean; 14 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 MiB/s wr, 3 op/s 2024-08-17T17:21:41.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:40 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-24", "field": "max_objects", "val": "6"}]': finished 2024-08-17T17:21:41.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:40 smithi136 ceph-mon[109861]: osdmap e636: 8 total, 8 up, 8 in 2024-08-17T17:21:41.348 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:40 smithi136 ceph-mon[115078]: pgmap v460: 425 pgs: 425 active+clean; 14 MiB data, 540 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 MiB/s wr, 3 op/s 2024-08-17T17:21:41.348 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:40 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-24", "field": "max_objects", "val": "6"}]': finished 2024-08-17T17:21:41.348 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:40 smithi136 ceph-mon[115078]: osdmap e636: 8 total, 8 up, 8 in 2024-08-17T17:21:42.001 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:21:43.011 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:21:43.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:42 smithi165 ceph-mon[105517]: pgmap v462: 425 pgs: 425 active+clean; 16 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 897 KiB/s wr, 6 op/s 2024-08-17T17:21:43.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:42 smithi136 ceph-mon[109861]: pgmap v462: 425 pgs: 425 active+clean; 16 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 897 KiB/s wr, 6 op/s 2024-08-17T17:21:43.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:42 smithi136 ceph-mon[115078]: pgmap v462: 425 pgs: 425 active+clean; 16 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 897 KiB/s wr, 6 op/s 2024-08-17T17:21:44.023 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:21:44.102 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.RemoveFullTryDataPool (30044 ms) 2024-08-17T17:21:44.102 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.RemoveFullTryNamespace 2024-08-17T17:21:44.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:43 smithi165 ceph-mon[105517]: pool 'test-librbd-smithi136-87100-24' is full (reached quota's max_objects: 6) 2024-08-17T17:21:44.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:43 smithi165 ceph-mon[105517]: Health check update: 2 pool(s) full (POOL_FULL) 2024-08-17T17:21:44.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:43 smithi165 ceph-mon[105517]: osdmap e637: 8 total, 8 up, 8 in 2024-08-17T17:21:44.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:43 smithi136 ceph-mon[109861]: pool 'test-librbd-smithi136-87100-24' is full (reached quota's max_objects: 6) 2024-08-17T17:21:44.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:43 smithi136 ceph-mon[109861]: Health check update: 2 pool(s) full (POOL_FULL) 2024-08-17T17:21:44.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:43 smithi136 ceph-mon[109861]: osdmap e637: 8 total, 8 up, 8 in 2024-08-17T17:21:44.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:43 smithi136 ceph-mon[115078]: pool 'test-librbd-smithi136-87100-24' is full (reached quota's max_objects: 6) 2024-08-17T17:21:44.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:43 smithi136 ceph-mon[115078]: Health check update: 2 pool(s) full (POOL_FULL) 2024-08-17T17:21:44.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:43 smithi136 ceph-mon[115078]: osdmap e637: 8 total, 8 up, 8 in 2024-08-17T17:21:45.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:44 smithi165 ceph-mon[105517]: pgmap v464: 425 pgs: 425 active+clean; 16 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 342 KiB/s wr, 4 op/s 2024-08-17T17:21:45.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:44 smithi136 ceph-mon[109861]: pgmap v464: 425 pgs: 425 active+clean; 16 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 342 KiB/s wr, 4 op/s 2024-08-17T17:21:45.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:44 smithi136 ceph-mon[115078]: pgmap v464: 425 pgs: 425 active+clean; 16 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 342 KiB/s wr, 4 op/s 2024-08-17T17:21:46.027 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:21:46.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:45 smithi165 ceph-mon[105517]: osdmap e638: 8 total, 8 up, 8 in 2024-08-17T17:21:46.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:45 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1469807951' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-26","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:21:46.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:45 smithi136 ceph-mon[115078]: osdmap e638: 8 total, 8 up, 8 in 2024-08-17T17:21:46.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:45 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1469807951' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-26","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:21:46.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:45 smithi136 ceph-mon[109861]: osdmap e638: 8 total, 8 up, 8 in 2024-08-17T17:21:46.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:45 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1469807951' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-26","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:21:46.447 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:21:46 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:21:46] ENGINE Bus STOPPING 2024-08-17T17:21:46.799 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:21:46 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:21:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-08-17T17:21:46.799 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:21:46 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:21:46] ENGINE Bus STOPPED 2024-08-17T17:21:46.799 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:21:46 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:21:46] ENGINE Bus STARTING 2024-08-17T17:21:46.799 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:21:46 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:21:46] ENGINE Serving on http://:::9283 2024-08-17T17:21:46.799 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:21:46 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:21:46] ENGINE Bus STARTED 2024-08-17T17:21:47.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:46 smithi136 ceph-mon[109861]: pgmap v466: 457 pgs: 32 unknown, 425 active+clean; 16 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 343 KiB/s wr, 4 op/s 2024-08-17T17:21:47.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:46 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:21:47.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:46 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1469807951' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:21:47.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:46 smithi136 ceph-mon[109861]: osdmap e639: 8 total, 8 up, 8 in 2024-08-17T17:21:47.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:46 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' 2024-08-17T17:21:47.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:46 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:21:47.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:46 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-08-17T17:21:47.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:46 smithi136 ceph-mon[115078]: pgmap v466: 457 pgs: 32 unknown, 425 active+clean; 16 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 343 KiB/s wr, 4 op/s 2024-08-17T17:21:47.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:46 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:21:47.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:46 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1469807951' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:21:47.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:46 smithi136 ceph-mon[115078]: osdmap e639: 8 total, 8 up, 8 in 2024-08-17T17:21:47.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:46 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' 2024-08-17T17:21:47.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:46 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:21:47.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:46 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-08-17T17:21:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:46 smithi165 ceph-mon[105517]: pgmap v466: 457 pgs: 32 unknown, 425 active+clean; 16 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 343 KiB/s wr, 4 op/s 2024-08-17T17:21:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:46 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:21:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:46 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1469807951' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:21:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:46 smithi165 ceph-mon[105517]: osdmap e639: 8 total, 8 up, 8 in 2024-08-17T17:21:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:46 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' 2024-08-17T17:21:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:46 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:21:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:46 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-08-17T17:21:47.825 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:21:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:21:47] "GET /metrics HTTP/1.1" 200 60339 "" "Prometheus/2.51.0" 2024-08-17T17:21:48.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:48 smithi136 ceph-mon[109861]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:21:48.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:48 smithi136 ceph-mon[109861]: osdmap e640: 8 total, 8 up, 8 in 2024-08-17T17:21:48.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:48 smithi136 ceph-mon[115078]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:21:48.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:48 smithi136 ceph-mon[115078]: osdmap e640: 8 total, 8 up, 8 in 2024-08-17T17:21:48.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:47 smithi165 ceph-mon[105517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:21:48.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:47 smithi165 ceph-mon[105517]: osdmap e640: 8 total, 8 up, 8 in 2024-08-17T17:21:49.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:48 smithi165 ceph-mon[105517]: pgmap v469: 457 pgs: 457 active+clean; 584 KiB data, 520 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 4.7 KiB/s wr, 47 op/s 2024-08-17T17:21:49.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:48 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:21:49.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:48 smithi165 ceph-mon[105517]: osdmap e641: 8 total, 8 up, 8 in 2024-08-17T17:21:49.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:48 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-26", "field": "max_bytes", "val": "10485760"} : dispatch 2024-08-17T17:21:49.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:48 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-26", "field": "max_bytes", "val": "10485760"} : dispatch 2024-08-17T17:21:49.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:48 smithi165 ceph-mon[105517]: pool 'test-librbd-smithi136-87100-25' no longer out of quota; removing NO_QUOTA flag 2024-08-17T17:21:49.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:48 smithi165 ceph-mon[105517]: Health check update: 1 pool(s) full (POOL_FULL) 2024-08-17T17:21:49.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:49 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-26", "field": "max_bytes", "val": "10485760"}]': finished 2024-08-17T17:21:49.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:49 smithi165 ceph-mon[105517]: osdmap e642: 8 total, 8 up, 8 in 2024-08-17T17:21:49.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:48 smithi136 ceph-mon[109861]: pgmap v469: 457 pgs: 457 active+clean; 584 KiB data, 520 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 4.7 KiB/s wr, 47 op/s 2024-08-17T17:21:49.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:48 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:21:49.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:48 smithi136 ceph-mon[109861]: osdmap e641: 8 total, 8 up, 8 in 2024-08-17T17:21:49.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:48 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-26", "field": "max_bytes", "val": "10485760"} : dispatch 2024-08-17T17:21:49.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:48 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-26", "field": "max_bytes", "val": "10485760"} : dispatch 2024-08-17T17:21:49.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:48 smithi136 ceph-mon[109861]: pool 'test-librbd-smithi136-87100-25' no longer out of quota; removing NO_QUOTA flag 2024-08-17T17:21:49.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:48 smithi136 ceph-mon[109861]: Health check update: 1 pool(s) full (POOL_FULL) 2024-08-17T17:21:49.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:48 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-26", "field": "max_bytes", "val": "10485760"}]': finished 2024-08-17T17:21:49.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:48 smithi136 ceph-mon[109861]: osdmap e642: 8 total, 8 up, 8 in 2024-08-17T17:21:49.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:48 smithi136 ceph-mon[115078]: pgmap v469: 457 pgs: 457 active+clean; 584 KiB data, 520 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 4.7 KiB/s wr, 47 op/s 2024-08-17T17:21:49.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:48 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:21:49.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:48 smithi136 ceph-mon[115078]: osdmap e641: 8 total, 8 up, 8 in 2024-08-17T17:21:49.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:48 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-26", "field": "max_bytes", "val": "10485760"} : dispatch 2024-08-17T17:21:49.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:48 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-26", "field": "max_bytes", "val": "10485760"} : dispatch 2024-08-17T17:21:49.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:48 smithi136 ceph-mon[115078]: pool 'test-librbd-smithi136-87100-25' no longer out of quota; removing NO_QUOTA flag 2024-08-17T17:21:49.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:48 smithi136 ceph-mon[115078]: Health check update: 1 pool(s) full (POOL_FULL) 2024-08-17T17:21:49.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:48 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-26", "field": "max_bytes", "val": "10485760"}]': finished 2024-08-17T17:21:49.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:48 smithi136 ceph-mon[115078]: osdmap e642: 8 total, 8 up, 8 in 2024-08-17T17:21:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:50 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:50 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:50 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:21:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:50 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:21:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:50 smithi136 ceph-mon[109861]: pgmap v472: 457 pgs: 457 active+clean; 584 KiB data, 520 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 4.7 KiB/s wr, 48 op/s 2024-08-17T17:21:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:50 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:50.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:50 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:50.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:50 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:50.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:50 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:21:50.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:50 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:21:50.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:50 smithi136 ceph-mon[115078]: pgmap v472: 457 pgs: 457 active+clean; 584 KiB data, 520 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 4.7 KiB/s wr, 48 op/s 2024-08-17T17:21:50.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:50 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:50 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:50 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:50 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:21:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:50 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:21:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:50 smithi165 ceph-mon[105517]: pgmap v472: 457 pgs: 457 active+clean; 584 KiB data, 520 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 4.7 KiB/s wr, 48 op/s 2024-08-17T17:21:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:50 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:52.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:52.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:21:52.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:52.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:21:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:21:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:21:53.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:53 smithi136 ceph-mon[109861]: pgmap v473: 457 pgs: 457 active+clean; 3.6 MiB data, 529 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 524 KiB/s wr, 42 op/s 2024-08-17T17:21:53.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:53 smithi136 ceph-mon[115078]: pgmap v473: 457 pgs: 457 active+clean; 3.6 MiB data, 529 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 524 KiB/s wr, 42 op/s 2024-08-17T17:21:53.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:53 smithi165 ceph-mon[105517]: pgmap v473: 457 pgs: 457 active+clean; 3.6 MiB data, 529 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 524 KiB/s wr, 42 op/s 2024-08-17T17:21:55.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:54 smithi136 ceph-mon[109861]: osdmap e643: 8 total, 8 up, 8 in 2024-08-17T17:21:55.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:54 smithi136 ceph-mon[109861]: pgmap v475: 457 pgs: 457 active+clean; 3.6 MiB data, 529 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 513 KiB/s wr, 9 op/s 2024-08-17T17:21:55.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:54 smithi136 ceph-mon[115078]: osdmap e643: 8 total, 8 up, 8 in 2024-08-17T17:21:55.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:54 smithi136 ceph-mon[115078]: pgmap v475: 457 pgs: 457 active+clean; 3.6 MiB data, 529 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 513 KiB/s wr, 9 op/s 2024-08-17T17:21:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:54 smithi165 ceph-mon[105517]: osdmap e643: 8 total, 8 up, 8 in 2024-08-17T17:21:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:54 smithi165 ceph-mon[105517]: pgmap v475: 457 pgs: 457 active+clean; 3.6 MiB data, 529 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 513 KiB/s wr, 9 op/s 2024-08-17T17:21:57.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:56 smithi165 ceph-mon[105517]: pgmap v476: 457 pgs: 457 active+clean; 3.6 MiB data, 529 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 390 KiB/s wr, 7 op/s 2024-08-17T17:21:57.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:56 smithi136 ceph-mon[109861]: pgmap v476: 457 pgs: 457 active+clean; 3.6 MiB data, 529 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 390 KiB/s wr, 7 op/s 2024-08-17T17:21:57.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:56 smithi136 ceph-mon[115078]: pgmap v476: 457 pgs: 457 active+clean; 3.6 MiB data, 529 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 390 KiB/s wr, 7 op/s 2024-08-17T17:21:57.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:21:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:21:57] "GET /metrics HTTP/1.1" 200 62779 "" "Prometheus/2.51.0" 2024-08-17T17:21:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:21:58 smithi165 ceph-mon[105517]: pgmap v477: 457 pgs: 457 active+clean; 8.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 905 KiB/s wr, 7 op/s 2024-08-17T17:21:59.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:21:58 smithi136 ceph-mon[109861]: pgmap v477: 457 pgs: 457 active+clean; 8.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 905 KiB/s wr, 7 op/s 2024-08-17T17:21:59.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:21:58 smithi136 ceph-mon[115078]: pgmap v477: 457 pgs: 457 active+clean; 8.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 905 KiB/s wr, 7 op/s 2024-08-17T17:22:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:00 smithi165 ceph-mon[105517]: pgmap v478: 457 pgs: 457 active+clean; 8.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 820 KiB/s wr, 7 op/s 2024-08-17T17:22:01.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:00 smithi136 ceph-mon[109861]: pgmap v478: 457 pgs: 457 active+clean; 8.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 820 KiB/s wr, 7 op/s 2024-08-17T17:22:01.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:00 smithi136 ceph-mon[115078]: pgmap v478: 457 pgs: 457 active+clean; 8.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 820 KiB/s wr, 7 op/s 2024-08-17T17:22:02.226 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'echo wait for half of osds to upgrade' 2024-08-17T17:22:02.432 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:22:03.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:02 smithi136 ceph-mon[109861]: pgmap v479: 457 pgs: 457 active+clean; 14 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 MiB/s wr, 3 op/s 2024-08-17T17:22:03.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:02 smithi136 ceph-mon[115078]: pgmap v479: 457 pgs: 457 active+clean; 14 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 MiB/s wr, 3 op/s 2024-08-17T17:22:03.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:02 smithi165 ceph-mon[105517]: pgmap v479: 457 pgs: 457 active+clean; 14 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 MiB/s wr, 3 op/s 2024-08-17T17:22:04.025 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.RemoveFullTryNamespace (19923 ms) 2024-08-17T17:22:04.025 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.RemoveFullTryNamespaceDataPool 2024-08-17T17:22:04.078 INFO:teuthology.orchestra.run.smithi136.stdout:wait for half of osds to upgrade 2024-08-17T17:22:04.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:03 smithi136 ceph-mon[115078]: pool 'test-librbd-smithi136-87100-26' is full (reached quota's max_bytes: 10 MiB) 2024-08-17T17:22:04.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:03 smithi136 ceph-mon[115078]: Health check update: 2 pool(s) full (POOL_FULL) 2024-08-17T17:22:04.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:03 smithi136 ceph-mon[115078]: osdmap e644: 8 total, 8 up, 8 in 2024-08-17T17:22:04.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:03 smithi136 ceph-mon[109861]: pool 'test-librbd-smithi136-87100-26' is full (reached quota's max_bytes: 10 MiB) 2024-08-17T17:22:04.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:03 smithi136 ceph-mon[109861]: Health check update: 2 pool(s) full (POOL_FULL) 2024-08-17T17:22:04.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:03 smithi136 ceph-mon[109861]: osdmap e644: 8 total, 8 up, 8 in 2024-08-17T17:22:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:03 smithi165 ceph-mon[105517]: pool 'test-librbd-smithi136-87100-26' is full (reached quota's max_bytes: 10 MiB) 2024-08-17T17:22:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:03 smithi165 ceph-mon[105517]: Health check update: 2 pool(s) full (POOL_FULL) 2024-08-17T17:22:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:03 smithi165 ceph-mon[105517]: osdmap e644: 8 total, 8 up, 8 in 2024-08-17T17:22:04.603 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 4' 2024-08-17T17:22:04.804 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:22:05.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:04 smithi136 ceph-mon[115078]: pgmap v481: 457 pgs: 457 active+clean; 14 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 MiB/s wr, 3 op/s 2024-08-17T17:22:05.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:04 smithi136 ceph-mon[109861]: pgmap v481: 457 pgs: 457 active+clean; 14 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 MiB/s wr, 3 op/s 2024-08-17T17:22:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:04 smithi165 ceph-mon[105517]: pgmap v481: 457 pgs: 457 active+clean; 14 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 MiB/s wr, 3 op/s 2024-08-17T17:22:06.171 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:05 smithi136 ceph-mon[109861]: osdmap e645: 8 total, 8 up, 8 in 2024-08-17T17:22:06.171 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:05 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3077839533' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:22:06.171 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:05 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:22:06.171 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:05 smithi136 ceph-mon[115078]: osdmap e645: 8 total, 8 up, 8 in 2024-08-17T17:22:06.171 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:05 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3077839533' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:22:06.171 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:05 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:22:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:05 smithi165 ceph-mon[105517]: osdmap e645: 8 total, 8 up, 8 in 2024-08-17T17:22:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:05 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3077839533' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:22:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:05 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:22:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:07 smithi165 ceph-mon[105517]: pgmap v483: 489 pgs: 32 unknown, 457 active+clean; 14 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 640 KiB/s wr, 2 op/s 2024-08-17T17:22:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:07 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:22:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:07 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:22:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:07 smithi165 ceph-mon[105517]: osdmap e646: 8 total, 8 up, 8 in 2024-08-17T17:22:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:22:07.321 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:07 smithi136 ceph-mon[115078]: pgmap v483: 489 pgs: 32 unknown, 457 active+clean; 14 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 640 KiB/s wr, 2 op/s 2024-08-17T17:22:07.321 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:07 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:22:07.321 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:07 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:22:07.321 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:07 smithi136 ceph-mon[115078]: osdmap e646: 8 total, 8 up, 8 in 2024-08-17T17:22:07.321 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:07.321 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:22:07.321 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:07 smithi136 ceph-mon[109861]: pgmap v483: 489 pgs: 32 unknown, 457 active+clean; 14 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 640 KiB/s wr, 2 op/s 2024-08-17T17:22:07.322 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:07 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:22:07.322 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:07 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:22:07.322 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:07 smithi136 ceph-mon[109861]: osdmap e646: 8 total, 8 up, 8 in 2024-08-17T17:22:07.322 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:07.322 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:22:07.615 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:22:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:22:07] "GET /metrics HTTP/1.1" 200 65195 "" "Prometheus/2.51.0" 2024-08-17T17:22:08.035 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:22:08.401 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:08 smithi136 ceph-mon[109861]: from='client.44384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453", "daemon_types": "osd", "limit": 4, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:08.402 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:08 smithi136 ceph-mon[109861]: osdmap e647: 8 total, 8 up, 8 in 2024-08-17T17:22:08.402 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:08 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3712314138' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-28","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:22:08.402 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:08 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3712314138' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:22:08.402 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:08 smithi136 ceph-mon[109861]: osdmap e648: 8 total, 8 up, 8 in 2024-08-17T17:22:08.402 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:22:08 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:22:08] ENGINE Bus STOPPING 2024-08-17T17:22:08.403 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:08 smithi136 ceph-mon[115078]: from='client.44384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453", "daemon_types": "osd", "limit": 4, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:08.403 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:08 smithi136 ceph-mon[115078]: osdmap e647: 8 total, 8 up, 8 in 2024-08-17T17:22:08.403 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:08 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3712314138' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-28","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:22:08.403 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:08 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3712314138' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:22:08.403 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:08 smithi136 ceph-mon[115078]: osdmap e648: 8 total, 8 up, 8 in 2024-08-17T17:22:08.434 INFO:teuthology.orchestra.run.smithi136.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:22:08.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:08 smithi165 ceph-mon[105517]: from='client.44384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453", "daemon_types": "osd", "limit": 4, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:08.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:08 smithi165 ceph-mon[105517]: osdmap e647: 8 total, 8 up, 8 in 2024-08-17T17:22:08.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:08 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3712314138' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-28","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:22:08.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:08 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3712314138' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:22:08.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:08 smithi165 ceph-mon[105517]: osdmap e648: 8 total, 8 up, 8 in 2024-08-17T17:22:08.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:22:08 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:22:08] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-08-17T17:22:08.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:22:08 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:22:08] ENGINE Bus STOPPED 2024-08-17T17:22:08.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:22:08 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:22:08] ENGINE Bus STARTING 2024-08-17T17:22:08.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:22:08 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:22:08] ENGINE Serving on http://:::9283 2024-08-17T17:22:08.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:22:08 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:22:08] ENGINE Bus STARTED 2024-08-17T17:22:09.032 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-08-17T17:22:09.234 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:22:09.356 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:09 smithi136 ceph-mon[109861]: pgmap v486: 521 pgs: 32 unknown, 489 active+clean; 584 KiB data, 509 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 507 KiB/s wr, 39 op/s 2024-08-17T17:22:09.356 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:09 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' 2024-08-17T17:22:09.356 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:09 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:22:09.356 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:09 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:09.356 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:09 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-08-17T17:22:09.356 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:09 smithi136 ceph-mon[109861]: pool 'test-librbd-smithi136-87100-26' no longer out of quota; removing NO_QUOTA flag 2024-08-17T17:22:09.356 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:09 smithi136 ceph-mon[109861]: Health check update: 1 pool(s) full (POOL_FULL) 2024-08-17T17:22:09.357 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:09 smithi136 ceph-mon[109861]: osdmap e649: 8 total, 8 up, 8 in 2024-08-17T17:22:09.357 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:09 smithi136 ceph-mon[115078]: pgmap v486: 521 pgs: 32 unknown, 489 active+clean; 584 KiB data, 509 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 507 KiB/s wr, 39 op/s 2024-08-17T17:22:09.357 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:09 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' 2024-08-17T17:22:09.357 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:09 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:22:09.357 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:09 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:09.357 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:09 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-08-17T17:22:09.357 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:09 smithi136 ceph-mon[115078]: pool 'test-librbd-smithi136-87100-26' no longer out of quota; removing NO_QUOTA flag 2024-08-17T17:22:09.357 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:09 smithi136 ceph-mon[115078]: Health check update: 1 pool(s) full (POOL_FULL) 2024-08-17T17:22:09.357 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:09 smithi136 ceph-mon[115078]: osdmap e649: 8 total, 8 up, 8 in 2024-08-17T17:22:09.385 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:09 smithi165 ceph-mon[105517]: pgmap v486: 521 pgs: 32 unknown, 489 active+clean; 584 KiB data, 509 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 507 KiB/s wr, 39 op/s 2024-08-17T17:22:09.385 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:09 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' 2024-08-17T17:22:09.385 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:09 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:22:09.385 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:09 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:09.385 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:09 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-08-17T17:22:09.385 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:09 smithi165 ceph-mon[105517]: pool 'test-librbd-smithi136-87100-26' no longer out of quota; removing NO_QUOTA flag 2024-08-17T17:22:09.385 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:09 smithi165 ceph-mon[105517]: Health check update: 1 pool(s) full (POOL_FULL) 2024-08-17T17:22:09.385 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:09 smithi165 ceph-mon[105517]: osdmap e649: 8 total, 8 up, 8 in 2024-08-17T17:22:10.289 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:10 smithi165 ceph-mon[105517]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:22:10.289 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:10 smithi165 ceph-mon[105517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:22:10.289 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:10 smithi165 ceph-mon[105517]: osdmap e650: 8 total, 8 up, 8 in 2024-08-17T17:22:10.289 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:10 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-28", "field": "max_bytes", "val": "10485760"} : dispatch 2024-08-17T17:22:10.289 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:10 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-28", "field": "max_bytes", "val": "10485760"} : dispatch 2024-08-17T17:22:10.312 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:10 smithi136 ceph-mon[109861]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:22:10.312 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:10 smithi136 ceph-mon[109861]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:22:10.312 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:10 smithi136 ceph-mon[109861]: osdmap e650: 8 total, 8 up, 8 in 2024-08-17T17:22:10.313 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:10 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-28", "field": "max_bytes", "val": "10485760"} : dispatch 2024-08-17T17:22:10.313 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:10 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-28", "field": "max_bytes", "val": "10485760"} : dispatch 2024-08-17T17:22:10.313 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:10 smithi136 ceph-mon[115078]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:22:10.313 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:10 smithi136 ceph-mon[115078]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:22:10.313 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:10 smithi136 ceph-mon[115078]: osdmap e650: 8 total, 8 up, 8 in 2024-08-17T17:22:10.313 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:10 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-28", "field": "max_bytes", "val": "10485760"} : dispatch 2024-08-17T17:22:10.313 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:10 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-28", "field": "max_bytes", "val": "10485760"} : dispatch 2024-08-17T17:22:11.042 INFO:teuthology.orchestra.run.smithi136.stdout:true 2024-08-17T17:22:11.310 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:11 smithi165 ceph-mon[105517]: pgmap v490: 521 pgs: 32 unknown, 489 active+clean; 584 KiB data, 509 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 524 KiB/s wr, 40 op/s 2024-08-17T17:22:11.310 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:11 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:22:11.310 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:11 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-28", "field": "max_bytes", "val": "10485760"}]': finished 2024-08-17T17:22:11.310 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:11 smithi165 ceph-mon[105517]: osdmap e651: 8 total, 8 up, 8 in 2024-08-17T17:22:11.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:11 smithi136 ceph-mon[115078]: pgmap v490: 521 pgs: 32 unknown, 489 active+clean; 584 KiB data, 509 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 524 KiB/s wr, 40 op/s 2024-08-17T17:22:11.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:11 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:22:11.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:11 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-28", "field": "max_bytes", "val": "10485760"}]': finished 2024-08-17T17:22:11.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:11 smithi136 ceph-mon[115078]: osdmap e651: 8 total, 8 up, 8 in 2024-08-17T17:22:11.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:11 smithi136 ceph-mon[109861]: pgmap v490: 521 pgs: 32 unknown, 489 active+clean; 584 KiB data, 509 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 524 KiB/s wr, 40 op/s 2024-08-17T17:22:11.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:11 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:22:11.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:11 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-28", "field": "max_bytes", "val": "10485760"}]': finished 2024-08-17T17:22:11.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:11 smithi136 ceph-mon[109861]: osdmap e651: 8 total, 8 up, 8 in 2024-08-17T17:22:11.594 INFO:teuthology.orchestra.run.smithi136.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-17T17:22:11.594 INFO:teuthology.orchestra.run.smithi136.stdout:alertmanager.a smithi136 *:9093,9094 running (7m) 22s ago 13m 19.4M - 0.27.0 11f11916f8cd c3b9b5b223ab 2024-08-17T17:22:11.594 INFO:teuthology.orchestra.run.smithi136.stdout:grafana.a smithi165 *:3000 running (6m) 69s ago 12m 77.9M - c8b91775d855 04db24fce5e9 2024-08-17T17:22:11.595 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi136.ltsuqd smithi136 running (11m) 22s ago 11m 23.9M - 18.2.4-719-gced649f8 c870753d67c2 faaa06f2950c 2024-08-17T17:22:11.595 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi165.bmrbhh smithi165 running (11m) 69s ago 11m 18.1M - 18.2.4-719-gced649f8 c870753d67c2 c74b00d929cf 2024-08-17T17:22:11.595 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.x smithi165 *:8443,9283,8765 running (8m) 69s ago 17m 457M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 837fd48ff7cb 2024-08-17T17:22:11.595 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.y smithi136 *:8443,9283,8765 running (8m) 22s ago 18m 527M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9acdc6775e48 2024-08-17T17:22:11.595 INFO:teuthology.orchestra.run.smithi136.stdout:mon.a smithi136 running (6m) 22s ago 18m 70.3M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9cf82e613626 2024-08-17T17:22:11.595 INFO:teuthology.orchestra.run.smithi136.stdout:mon.b smithi165 running (3m) 69s ago 17m 48.6M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 1fd357308b8d 2024-08-17T17:22:11.595 INFO:teuthology.orchestra.run.smithi136.stdout:mon.c smithi136 running (5m) 22s ago 17m 62.5M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 2c2ac98ad9d4 2024-08-17T17:22:11.595 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.a smithi136 *:9100 running (8m) 22s ago 12m 10.3M - 1.7.0 72c9c2088986 1e4c644ae533 2024-08-17T17:22:11.595 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.b smithi165 *:9100 running (8m) 69s ago 12m 10.9M - 1.7.0 72c9c2088986 7af2d2d68b06 2024-08-17T17:22:11.595 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0 smithi136 running (16m) 22s ago 16m 126M 2480M 18.2.4-719-gced649f8 c870753d67c2 faab916734b0 2024-08-17T17:22:11.595 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1 smithi136 running (16m) 22s ago 16m 131M 2480M 18.2.4-719-gced649f8 c870753d67c2 861686b6ff82 2024-08-17T17:22:11.595 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2 smithi136 running (15m) 22s ago 15m 140M 2480M 18.2.4-719-gced649f8 c870753d67c2 640be4af4156 2024-08-17T17:22:11.595 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3 smithi136 running (15m) 22s ago 15m 134M 2480M 18.2.4-719-gced649f8 c870753d67c2 14b57a745d30 2024-08-17T17:22:11.595 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4 smithi165 running (15m) 69s ago 15m 132M 2480M 18.2.4-719-gced649f8 c870753d67c2 df366c73e1f2 2024-08-17T17:22:11.596 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5 smithi165 running (14m) 69s ago 14m 108M 2480M 18.2.4-719-gced649f8 c870753d67c2 1131665e203a 2024-08-17T17:22:11.596 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6 smithi165 running (14m) 69s ago 14m 142M 2480M 18.2.4-719-gced649f8 c870753d67c2 7bc5fd925293 2024-08-17T17:22:11.596 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7 smithi165 running (14m) 69s ago 14m 121M 2480M 18.2.4-719-gced649f8 c870753d67c2 18382e4ae3b8 2024-08-17T17:22:11.596 INFO:teuthology.orchestra.run.smithi136.stdout:prometheus.a smithi165 *:9095 running (7m) 69s ago 13m 48.0M - 2.51.0 1d3b7f56885b ae8f8b69fe2f 2024-08-17T17:22:11.930 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:22:11.931 INFO:teuthology.orchestra.run.smithi136.stdout: "mon": { 2024-08-17T17:22:11.931 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 3 2024-08-17T17:22:11.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:22:11.931 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr": { 2024-08-17T17:22:11.931 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 2 2024-08-17T17:22:11.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:22:11.931 INFO:teuthology.orchestra.run.smithi136.stdout: "osd": { 2024-08-17T17:22:11.931 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 8 2024-08-17T17:22:11.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:22:11.931 INFO:teuthology.orchestra.run.smithi136.stdout: "mds": { 2024-08-17T17:22:11.931 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 2 2024-08-17T17:22:11.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:22:11.931 INFO:teuthology.orchestra.run.smithi136.stdout: "overall": { 2024-08-17T17:22:11.931 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 10, 2024-08-17T17:22:11.932 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 5 2024-08-17T17:22:11.932 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:22:11.932 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:22:12.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:12 smithi136 ceph-mon[109861]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:12.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:12 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3091351686' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:22:12.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:12 smithi136 ceph-mon[115078]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:12.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:12 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3091351686' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:22:12.217 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:22:12.217 INFO:teuthology.orchestra.run.smithi136.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453", 2024-08-17T17:22:12.218 INFO:teuthology.orchestra.run.smithi136.stdout: "in_progress": true, 2024-08-17T17:22:12.218 INFO:teuthology.orchestra.run.smithi136.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 4 daemons (4 remaining).", 2024-08-17T17:22:12.218 INFO:teuthology.orchestra.run.smithi136.stdout: "services_complete": [], 2024-08-17T17:22:12.218 INFO:teuthology.orchestra.run.smithi136.stdout: "progress": "", 2024-08-17T17:22:12.218 INFO:teuthology.orchestra.run.smithi136.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 image", 2024-08-17T17:22:12.218 INFO:teuthology.orchestra.run.smithi136.stdout: "is_paused": false 2024-08-17T17:22:12.218 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:22:12.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:12 smithi165 ceph-mon[105517]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:12.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:12 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3091351686' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:22:13.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:13 smithi136 ceph-mon[109861]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:13.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:13 smithi136 ceph-mon[109861]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:13.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:13 smithi136 ceph-mon[109861]: pgmap v492: 521 pgs: 521 active+clean; 1.6 MiB data, 513 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 259 KiB/s wr, 15 op/s 2024-08-17T17:22:13.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:13 smithi136 ceph-mon[109861]: from='client.44414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:13.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:13 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:13.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:13 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:13.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:13 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:22:13.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:13 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:22:13.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:13 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:13.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:13 smithi136 ceph-mon[115078]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:13.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:13 smithi136 ceph-mon[115078]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:13.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:13 smithi136 ceph-mon[115078]: pgmap v492: 521 pgs: 521 active+clean; 1.6 MiB data, 513 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 259 KiB/s wr, 15 op/s 2024-08-17T17:22:13.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:13 smithi136 ceph-mon[115078]: from='client.44414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:13.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:13 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:13.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:13 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:13.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:13 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:22:13.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:13 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:22:13.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:13 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:13 smithi165 ceph-mon[105517]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:13 smithi165 ceph-mon[105517]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:13 smithi165 ceph-mon[105517]: pgmap v492: 521 pgs: 521 active+clean; 1.6 MiB data, 513 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 259 KiB/s wr, 15 op/s 2024-08-17T17:22:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:13 smithi165 ceph-mon[105517]: from='client.44414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:13 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:13 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:13 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:22:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:13 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:22:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:13 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:14.101 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:14 smithi136 ceph-mon[109861]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:22:14.101 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:14 smithi136 ceph-mon[109861]: osdmap e652: 8 total, 8 up, 8 in 2024-08-17T17:22:14.102 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:14 smithi136 ceph-mon[115078]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:22:14.102 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:14 smithi136 ceph-mon[115078]: osdmap e652: 8 total, 8 up, 8 in 2024-08-17T17:22:14.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:14 smithi165 ceph-mon[105517]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:22:14.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:14 smithi165 ceph-mon[105517]: osdmap e652: 8 total, 8 up, 8 in 2024-08-17T17:22:15.289 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:15 smithi136 ceph-mon[109861]: pgmap v494: 521 pgs: 521 active+clean; 1.6 MiB data, 513 MiB used, 715 GiB / 715 GiB avail; 6.9 KiB/s rd, 204 KiB/s wr, 12 op/s 2024-08-17T17:22:15.289 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:15 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:15.289 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:15 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:22:15.289 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:15 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:22:15.289 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:15 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:15.289 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:15 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:22:15.289 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:15 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:15.289 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:15 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:22:15.289 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:15 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:15.290 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:15 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-08-17T17:22:15.290 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:15 smithi136 ceph-mon[115078]: pgmap v494: 521 pgs: 521 active+clean; 1.6 MiB data, 513 MiB used, 715 GiB / 715 GiB avail; 6.9 KiB/s rd, 204 KiB/s wr, 12 op/s 2024-08-17T17:22:15.290 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:15 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:15.290 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:15 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:22:15.290 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:15 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:22:15.290 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:15 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:15.290 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:15 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:22:15.290 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:15 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:15.290 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:15 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:22:15.290 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:15 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:15.290 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:15 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-08-17T17:22:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:15 smithi165 ceph-mon[105517]: pgmap v494: 521 pgs: 521 active+clean; 1.6 MiB data, 513 MiB used, 715 GiB / 715 GiB avail; 6.9 KiB/s rd, 204 KiB/s wr, 12 op/s 2024-08-17T17:22:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:15 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:15 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:22:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:15 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:22:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:15 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:15 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:22:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:15 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:15 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:22:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:15 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:15.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:15 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-08-17T17:22:16.091 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:16 smithi136 ceph-mon[115078]: Upgrade: Target is version 19.0.0-5836-gcd8bdbeb (squid) 2024-08-17T17:22:16.091 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:16 smithi136 ceph-mon[115078]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d'] 2024-08-17T17:22:16.091 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:16 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all mgr 2024-08-17T17:22:16.091 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:16 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all mon 2024-08-17T17:22:16.091 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:16 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all crash 2024-08-17T17:22:16.091 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:16 smithi136 ceph-mon[115078]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-08-17T17:22:16.091 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:16 smithi136 ceph-mon[115078]: Upgrade: osd.0 is safe to restart 2024-08-17T17:22:16.092 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:16 smithi136 ceph-mon[109861]: Upgrade: Target is version 19.0.0-5836-gcd8bdbeb (squid) 2024-08-17T17:22:16.092 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:16 smithi136 ceph-mon[109861]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d'] 2024-08-17T17:22:16.092 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:16 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all mgr 2024-08-17T17:22:16.092 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:16 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all mon 2024-08-17T17:22:16.092 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:16 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all crash 2024-08-17T17:22:16.092 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:16 smithi136 ceph-mon[109861]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-08-17T17:22:16.092 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:16 smithi136 ceph-mon[109861]: Upgrade: osd.0 is safe to restart 2024-08-17T17:22:16.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:16 smithi165 ceph-mon[105517]: Upgrade: Target is version 19.0.0-5836-gcd8bdbeb (squid) 2024-08-17T17:22:16.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:16 smithi165 ceph-mon[105517]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d'] 2024-08-17T17:22:16.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:16 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all mgr 2024-08-17T17:22:16.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:16 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all mon 2024-08-17T17:22:16.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:16 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all crash 2024-08-17T17:22:16.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:16 smithi165 ceph-mon[105517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-08-17T17:22:16.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:16 smithi165 ceph-mon[105517]: Upgrade: osd.0 is safe to restart 2024-08-17T17:22:17.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:17 smithi136 ceph-mon[109861]: pgmap v495: 521 pgs: 521 active+clean; 1.6 MiB data, 513 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 171 KiB/s wr, 10 op/s 2024-08-17T17:22:17.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:17 smithi136 ceph-mon[109861]: Upgrade: Updating osd.0 2024-08-17T17:22:17.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:17 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:17.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:17 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-08-17T17:22:17.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:17 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:22:17.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:17 smithi136 ceph-mon[109861]: Deploying daemon osd.0 on smithi136 2024-08-17T17:22:17.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:17 smithi136 ceph-mon[115078]: pgmap v495: 521 pgs: 521 active+clean; 1.6 MiB data, 513 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 171 KiB/s wr, 10 op/s 2024-08-17T17:22:17.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:17 smithi136 ceph-mon[115078]: Upgrade: Updating osd.0 2024-08-17T17:22:17.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:17 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:17.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:17 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-08-17T17:22:17.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:17 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:22:17.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:17 smithi136 ceph-mon[115078]: Deploying daemon osd.0 on smithi136 2024-08-17T17:22:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:17 smithi165 ceph-mon[105517]: pgmap v495: 521 pgs: 521 active+clean; 1.6 MiB data, 513 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 171 KiB/s wr, 10 op/s 2024-08-17T17:22:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:17 smithi165 ceph-mon[105517]: Upgrade: Updating osd.0 2024-08-17T17:22:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:17 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:17 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-08-17T17:22:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:17 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:22:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:17 smithi165 ceph-mon[105517]: Deploying daemon osd.0 on smithi136 2024-08-17T17:22:17.791 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:22:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:22:17] "GET /metrics HTTP/1.1" 200 65195 "" "Prometheus/2.51.0" 2024-08-17T17:22:19.416 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:19 smithi136 ceph-mon[109861]: pgmap v496: 521 pgs: 521 active+clean; 6.6 MiB data, 528 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 769 KiB/s wr, 9 op/s 2024-08-17T17:22:19.416 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:19 smithi136 ceph-mon[115078]: pgmap v496: 521 pgs: 521 active+clean; 6.6 MiB data, 528 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 769 KiB/s wr, 9 op/s 2024-08-17T17:22:19.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:19 smithi165 ceph-mon[105517]: pgmap v496: 521 pgs: 521 active+clean; 6.6 MiB data, 528 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 769 KiB/s wr, 9 op/s 2024-08-17T17:22:19.696 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:19 smithi136 systemd[1]: Stopping Ceph osd.0 for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:22:19.696 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:19 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0[46890]: 2024-08-17T17:22:19.644+0000 7f1c7e303640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:22:19.696 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:19 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0[46890]: 2024-08-17T17:22:19.644+0000 7f1c7e303640 -1 osd.0 652 *** Got signal Terminated *** 2024-08-17T17:22:19.697 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:19 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0[46890]: 2024-08-17T17:22:19.644+0000 7f1c7e303640 -1 osd.0 652 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T17:22:20.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:20 smithi136 ceph-mon[109861]: osd.0 marked itself down and dead 2024-08-17T17:22:20.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:20 smithi136 ceph-mon[115078]: osd.0 marked itself down and dead 2024-08-17T17:22:20.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:20 smithi165 ceph-mon[105517]: osd.0 marked itself down and dead 2024-08-17T17:22:20.968 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:20 smithi136 podman[128200]: 2024-08-17 17:22:20.702541002 +0000 UTC m=+1.141556404 container died faab916734b02380693f6854c039f79fd50a23913711f871a137a511bdbb97cb (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=reef-ced649f, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T17:22:21.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:21 smithi136 ceph-mon[109861]: pgmap v497: 521 pgs: 521 active+clean; 6.6 MiB data, 528 MiB used, 715 GiB / 715 GiB avail; 4.4 KiB/s rd, 680 KiB/s wr, 8 op/s 2024-08-17T17:22:21.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:21 smithi136 ceph-mon[109861]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:22:21.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:21 smithi136 ceph-mon[109861]: osdmap e653: 8 total, 7 up, 8 in 2024-08-17T17:22:21.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:21 smithi136 ceph-mon[109861]: osd.0 marked itself dead as of e653 2024-08-17T17:22:21.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:21 smithi136 ceph-mon[115078]: pgmap v497: 521 pgs: 521 active+clean; 6.6 MiB data, 528 MiB used, 715 GiB / 715 GiB avail; 4.4 KiB/s rd, 680 KiB/s wr, 8 op/s 2024-08-17T17:22:21.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:21 smithi136 ceph-mon[115078]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:22:21.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:21 smithi136 ceph-mon[115078]: osdmap e653: 8 total, 7 up, 8 in 2024-08-17T17:22:21.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:21 smithi136 ceph-mon[115078]: osd.0 marked itself dead as of e653 2024-08-17T17:22:21.447 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:20 smithi136 podman[128200]: 2024-08-17 17:22:20.967790952 +0000 UTC m=+1.406806352 container cleanup faab916734b02380693f6854c039f79fd50a23913711f871a137a511bdbb97cb (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=reef-ced649f) 2024-08-17T17:22:21.447 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:20 smithi136 bash[128200]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0 2024-08-17T17:22:21.448 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:21 smithi136 podman[128221]: 2024-08-17 17:22:21.135847539 +0000 UTC m=+0.436159618 container remove faab916734b02380693f6854c039f79fd50a23913711f871a137a511bdbb97cb (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=reef-ced649f, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-17T17:22:21.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:21 smithi165 ceph-mon[105517]: pgmap v497: 521 pgs: 521 active+clean; 6.6 MiB data, 528 MiB used, 715 GiB / 715 GiB avail; 4.4 KiB/s rd, 680 KiB/s wr, 8 op/s 2024-08-17T17:22:21.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:21 smithi165 ceph-mon[105517]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:22:21.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:21 smithi165 ceph-mon[105517]: osdmap e653: 8 total, 7 up, 8 in 2024-08-17T17:22:21.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:21 smithi165 ceph-mon[105517]: osd.0 marked itself dead as of e653 2024-08-17T17:22:21.906 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:21 smithi136 podman[128315]: 2024-08-17 17:22:21.578660477 +0000 UTC m=+0.102766468 container create d23222f1906e27c057949e9dee2c793933f9fdd82b193e4c466350c626fbd36f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=main-cd8bdbe) 2024-08-17T17:22:21.906 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:21 smithi136 podman[128315]: 2024-08-17 17:22:21.493041093 +0000 UTC m=+0.017147058 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:22:21.906 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:21 smithi136 podman[128315]: 2024-08-17 17:22:21.806087924 +0000 UTC m=+0.330193872 container init d23222f1906e27c057949e9dee2c793933f9fdd82b193e4c466350c626fbd36f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-deactivate, RELEASE=main-cd8bdbe, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-08-17T17:22:21.907 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:21 smithi136 podman[128315]: 2024-08-17 17:22:21.810441811 +0000 UTC m=+0.334547766 container start d23222f1906e27c057949e9dee2c793933f9fdd82b193e4c466350c626fbd36f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=main-cd8bdbe, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-08-17T17:22:21.907 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:21 smithi136 podman[128315]: 2024-08-17 17:22:21.906047595 +0000 UTC m=+0.430153553 container attach d23222f1906e27c057949e9dee2c793933f9fdd82b193e4c466350c626fbd36f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-08-17T17:22:22.196 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:22 smithi136 podman[128315]: 2024-08-17 17:22:22.003957514 +0000 UTC m=+0.528063474 container died d23222f1906e27c057949e9dee2c793933f9fdd82b193e4c466350c626fbd36f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe) 2024-08-17T17:22:22.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:22 smithi136 ceph-mon[109861]: osdmap e654: 8 total, 7 up, 8 in 2024-08-17T17:22:22.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:22.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:22:22.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:22 smithi136 ceph-mon[115078]: osdmap e654: 8 total, 7 up, 8 in 2024-08-17T17:22:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:22:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:22 smithi165 ceph-mon[105517]: osdmap e654: 8 total, 7 up, 8 in 2024-08-17T17:22:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:22:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:23.137 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:22 smithi136 podman[128359]: 2024-08-17 17:22:22.760299131 +0000 UTC m=+0.750359363 container remove d23222f1906e27c057949e9dee2c793933f9fdd82b193e4c466350c626fbd36f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, maintainer=Guillaume Abrioux ) 2024-08-17T17:22:23.137 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:22 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.0.service: Deactivated successfully. 2024-08-17T17:22:23.137 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:22 smithi136 systemd[1]: Stopped Ceph osd.0 for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:22:23.137 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:22 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.0.service: Consumed 31.005s CPU time. 2024-08-17T17:22:23.137 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:23 smithi136 systemd[1]: Starting Ceph osd.0 for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:22:23.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:23 smithi136 ceph-mon[109861]: pgmap v500: 521 pgs: 134 active+undersized, 49 active+undersized+degraded, 338 active+clean; 12 MiB data, 551 MiB used, 715 GiB / 715 GiB avail; 1.3 MiB/s wr, 1 op/s; 76/591 objects degraded (12.860%) 2024-08-17T17:22:23.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:23 smithi136 ceph-mon[109861]: Health check failed: Degraded data redundancy: 76/591 objects degraded (12.860%), 49 pgs degraded (PG_DEGRADED) 2024-08-17T17:22:23.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:23 smithi136 ceph-mon[115078]: pgmap v500: 521 pgs: 134 active+undersized, 49 active+undersized+degraded, 338 active+clean; 12 MiB data, 551 MiB used, 715 GiB / 715 GiB avail; 1.3 MiB/s wr, 1 op/s; 76/591 objects degraded (12.860%) 2024-08-17T17:22:23.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:23 smithi136 ceph-mon[115078]: Health check failed: Degraded data redundancy: 76/591 objects degraded (12.860%), 49 pgs degraded (PG_DEGRADED) 2024-08-17T17:22:23.447 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:23 smithi136 podman[128462]: 2024-08-17 17:22:23.123255727 +0000 UTC m=+0.017199866 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:22:23.447 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:23 smithi136 podman[128462]: 2024-08-17 17:22:23.228058927 +0000 UTC m=+0.122003108 container create a830fa2ea0ac6e285d129597a3ac4218c640d997b9a5fe05024ca69c7286a9cc (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=main-cd8bdbe, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True) 2024-08-17T17:22:23.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:23 smithi165 ceph-mon[105517]: pgmap v500: 521 pgs: 134 active+undersized, 49 active+undersized+degraded, 338 active+clean; 12 MiB data, 551 MiB used, 715 GiB / 715 GiB avail; 1.3 MiB/s wr, 1 op/s; 76/591 objects degraded (12.860%) 2024-08-17T17:22:23.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:23 smithi165 ceph-mon[105517]: Health check failed: Degraded data redundancy: 76/591 objects degraded (12.860%), 49 pgs degraded (PG_DEGRADED) 2024-08-17T17:22:23.947 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:23 smithi136 podman[128462]: 2024-08-17 17:22:23.580572833 +0000 UTC m=+0.474516985 container init a830fa2ea0ac6e285d129597a3ac4218c640d997b9a5fe05024ca69c7286a9cc (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-activate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, org.label-schema.license=GPLv2) 2024-08-17T17:22:23.947 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:23 smithi136 podman[128462]: 2024-08-17 17:22:23.58531699 +0000 UTC m=+0.479261138 container start a830fa2ea0ac6e285d129597a3ac4218c640d997b9a5fe05024ca69c7286a9cc (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-activate, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=main-cd8bdbe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-08-17T17:22:23.947 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:23 smithi136 podman[128462]: 2024-08-17 17:22:23.663673696 +0000 UTC m=+0.557617854 container attach a830fa2ea0ac6e285d129597a3ac4218c640d997b9a5fe05024ca69c7286a9cc (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-activate, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=main-cd8bdbe, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-08-17T17:22:23.947 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:23 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-activate[128477]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:22:23.947 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:23 smithi136 bash[128462]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:22:23.947 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:23 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-activate[128477]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:22:23.947 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:23 smithi136 bash[128462]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:22:23.947 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:23 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-activate[128477]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-17T17:22:23.947 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:23 smithi136 bash[128462]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-17T17:22:23.948 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:23 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-activate[128477]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:22:23.948 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:23 smithi136 bash[128462]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:22:23.948 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:23 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-activate[128477]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:22:23.948 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:23 smithi136 bash[128462]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:22:23.948 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:23 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-activate[128477]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-08-17T17:22:23.948 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:23 smithi136 bash[128462]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-08-17T17:22:23.948 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:23 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-activate[128477]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2024-08-17T17:22:23.948 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:23 smithi136 bash[128462]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2024-08-17T17:22:24.442 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:24 smithi136 ceph-mon[109861]: pool 'test-librbd-smithi136-87100-28' is full (reached quota's max_bytes: 10 MiB) 2024-08-17T17:22:24.442 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:24 smithi136 ceph-mon[109861]: Health check update: 2 pool(s) full (POOL_FULL) 2024-08-17T17:22:24.443 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:24 smithi136 ceph-mon[109861]: osdmap e655: 8 total, 7 up, 8 in 2024-08-17T17:22:24.443 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:24 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-27", "field": "max_objects", "val": "7"} : dispatch 2024-08-17T17:22:24.443 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:24 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-27", "field": "max_objects", "val": "7"} : dispatch 2024-08-17T17:22:24.443 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:24 smithi136 ceph-mon[115078]: pool 'test-librbd-smithi136-87100-28' is full (reached quota's max_bytes: 10 MiB) 2024-08-17T17:22:24.443 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:24 smithi136 ceph-mon[115078]: Health check update: 2 pool(s) full (POOL_FULL) 2024-08-17T17:22:24.443 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:24 smithi136 ceph-mon[115078]: osdmap e655: 8 total, 7 up, 8 in 2024-08-17T17:22:24.443 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:24 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-27", "field": "max_objects", "val": "7"} : dispatch 2024-08-17T17:22:24.443 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:24 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-27", "field": "max_objects", "val": "7"} : dispatch 2024-08-17T17:22:24.444 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:24 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-activate[128477]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2024-08-17T17:22:24.444 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:24 smithi136 bash[128462]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2024-08-17T17:22:24.444 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:24 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-activate[128477]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2024-08-17T17:22:24.444 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:24 smithi136 bash[128462]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2024-08-17T17:22:24.444 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:24 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-activate[128477]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-08-17T17:22:24.444 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:24 smithi136 bash[128462]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-08-17T17:22:24.444 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:24 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-activate[128477]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-08-17T17:22:24.444 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:24 smithi136 bash[128462]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-08-17T17:22:24.444 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:24 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-activate[128477]: --> ceph-volume lvm activate successful for osd ID: 0 2024-08-17T17:22:24.444 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:24 smithi136 bash[128462]: --> ceph-volume lvm activate successful for osd ID: 0 2024-08-17T17:22:24.444 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:24 smithi136 podman[128462]: 2024-08-17 17:22:24.136160211 +0000 UTC m=+1.030104352 container died a830fa2ea0ac6e285d129597a3ac4218c640d997b9a5fe05024ca69c7286a9cc (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731) 2024-08-17T17:22:24.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:24 smithi165 ceph-mon[105517]: pool 'test-librbd-smithi136-87100-28' is full (reached quota's max_bytes: 10 MiB) 2024-08-17T17:22:24.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:24 smithi165 ceph-mon[105517]: Health check update: 2 pool(s) full (POOL_FULL) 2024-08-17T17:22:24.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:24 smithi165 ceph-mon[105517]: osdmap e655: 8 total, 7 up, 8 in 2024-08-17T17:22:24.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:24 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-27", "field": "max_objects", "val": "7"} : dispatch 2024-08-17T17:22:24.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:24 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-27", "field": "max_objects", "val": "7"} : dispatch 2024-08-17T17:22:24.868 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:22:25.068 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:24 smithi136 podman[128604]: 2024-08-17 17:22:24.759408993 +0000 UTC m=+0.613640487 container remove a830fa2ea0ac6e285d129597a3ac4218c640d997b9a5fe05024ca69c7286a9cc (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0) 2024-08-17T17:22:25.068 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:24 smithi136 podman[128667]: 2024-08-17 17:22:24.952103678 +0000 UTC m=+0.095549430 container create 71b7314ae4a5b549076d7ef75cfd4ef44101068cbfeb22f4feed628422ec7557 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:22:25.068 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:24 smithi136 podman[128667]: 2024-08-17 17:22:24.884614208 +0000 UTC m=+0.028059918 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:22:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:25 smithi136 ceph-mon[109861]: pgmap v502: 521 pgs: 134 active+undersized, 49 active+undersized+degraded, 338 active+clean; 12 MiB data, 551 MiB used, 715 GiB / 715 GiB avail; 853 KiB/s wr, 1 op/s; 76/591 objects degraded (12.860%) 2024-08-17T17:22:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:25 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-27", "field": "max_objects", "val": "7"}]': finished 2024-08-17T17:22:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:25 smithi136 ceph-mon[109861]: osdmap e656: 8 total, 7 up, 8 in 2024-08-17T17:22:25.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:25 smithi136 ceph-mon[115078]: pgmap v502: 521 pgs: 134 active+undersized, 49 active+undersized+degraded, 338 active+clean; 12 MiB data, 551 MiB used, 715 GiB / 715 GiB avail; 853 KiB/s wr, 1 op/s; 76/591 objects degraded (12.860%) 2024-08-17T17:22:25.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:25 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-27", "field": "max_objects", "val": "7"}]': finished 2024-08-17T17:22:25.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:25 smithi136 ceph-mon[115078]: osdmap e656: 8 total, 7 up, 8 in 2024-08-17T17:22:25.447 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:25 smithi136 podman[128667]: 2024-08-17 17:22:25.154578524 +0000 UTC m=+0.298024242 container init 71b7314ae4a5b549076d7ef75cfd4ef44101068cbfeb22f4feed628422ec7557 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0, ceph=True, GIT_CLEAN=True, RELEASE=main-cd8bdbe, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:22:25.447 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:25 smithi136 podman[128667]: 2024-08-17 17:22:25.159454583 +0000 UTC m=+0.302900294 container start 71b7314ae4a5b549076d7ef75cfd4ef44101068cbfeb22f4feed628422ec7557 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=main-cd8bdbe, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-08-17T17:22:25.447 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:25 smithi136 ceph-osd[128705]: -- 172.21.15.136:0/3747165924 <== mon.0 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56413aad3a40 con 0x56413aab1c00 2024-08-17T17:22:25.447 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:25 smithi136 bash[128667]: 71b7314ae4a5b549076d7ef75cfd4ef44101068cbfeb22f4feed628422ec7557 2024-08-17T17:22:25.447 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:25 smithi136 systemd[1]: Started Ceph osd.0 for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:22:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:25 smithi165 ceph-mon[105517]: pgmap v502: 521 pgs: 134 active+undersized, 49 active+undersized+degraded, 338 active+clean; 12 MiB data, 551 MiB used, 715 GiB / 715 GiB avail; 853 KiB/s wr, 1 op/s; 76/591 objects degraded (12.860%) 2024-08-17T17:22:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:25 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi136-87100-27", "field": "max_objects", "val": "7"}]': finished 2024-08-17T17:22:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:25 smithi165 ceph-mon[105517]: osdmap e656: 8 total, 7 up, 8 in 2024-08-17T17:22:25.882 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:22:26.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:26 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:26.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:26 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:26.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:26 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:26.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:26 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:22:26.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:26 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:26.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:26 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:26.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:26 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:26.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:26 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:22:26.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:26 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:26.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:26 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:26.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:26 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:26.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:26 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:22:26.892 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:22:27.196 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0[128682]: 2024-08-17T17:22:27.054+0000 7f1cf2301640 -1 Falling back to public interface 2024-08-17T17:22:27.631 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:27 smithi136 ceph-mon[109861]: pgmap v504: 521 pgs: 134 active+undersized, 49 active+undersized+degraded, 338 active+clean; 12 MiB data, 551 MiB used, 715 GiB / 715 GiB avail; 885 KiB/s wr, 1 op/s; 76/591 objects degraded (12.860%) 2024-08-17T17:22:27.631 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:27 smithi136 ceph-mon[115078]: pgmap v504: 521 pgs: 134 active+undersized, 49 active+undersized+degraded, 338 active+clean; 12 MiB data, 551 MiB used, 715 GiB / 715 GiB avail; 885 KiB/s wr, 1 op/s; 76/591 objects degraded (12.860%) 2024-08-17T17:22:27.632 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:22:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:22:27] "GET /metrics HTTP/1.1" 200 67689 "" "Prometheus/2.51.0" 2024-08-17T17:22:27.662 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:27 smithi165 ceph-mon[105517]: pgmap v504: 521 pgs: 134 active+undersized, 49 active+undersized+degraded, 338 active+clean; 12 MiB data, 551 MiB used, 715 GiB / 715 GiB avail; 885 KiB/s wr, 1 op/s; 76/591 objects degraded (12.860%) 2024-08-17T17:22:27.901 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:22:28.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:28 smithi136 ceph-mon[109861]: pgmap v505: 521 pgs: 134 active+undersized, 49 active+undersized+degraded, 338 active+clean; 14 MiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.0 MiB/s wr, 3 op/s; 77/612 objects degraded (12.582%) 2024-08-17T17:22:28.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:28 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 77/612 objects degraded (12.582%), 49 pgs degraded (PG_DEGRADED) 2024-08-17T17:22:28.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:28 smithi136 ceph-mon[115078]: pgmap v505: 521 pgs: 134 active+undersized, 49 active+undersized+degraded, 338 active+clean; 14 MiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.0 MiB/s wr, 3 op/s; 77/612 objects degraded (12.582%) 2024-08-17T17:22:28.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:28 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 77/612 objects degraded (12.582%), 49 pgs degraded (PG_DEGRADED) 2024-08-17T17:22:28.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:28 smithi165 ceph-mon[105517]: pgmap v505: 521 pgs: 134 active+undersized, 49 active+undersized+degraded, 338 active+clean; 14 MiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.0 MiB/s wr, 3 op/s; 77/612 objects degraded (12.582%) 2024-08-17T17:22:28.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:28 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 77/612 objects degraded (12.582%), 49 pgs degraded (PG_DEGRADED) 2024-08-17T17:22:28.910 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:22:28.974 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.RemoveFullTryNamespaceDataPool (24949 ms) 2024-08-17T17:22:28.974 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestIO 2024-08-17T17:22:28.974 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:22:30.115 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:29 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:30.115 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:29 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:30.115 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:29 smithi136 ceph-mon[109861]: pool 'test-librbd-smithi136-87100-27' is full (reached quota's max_objects: 7) 2024-08-17T17:22:30.115 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:29 smithi136 ceph-mon[109861]: Health check update: 3 pool(s) full (POOL_FULL) 2024-08-17T17:22:30.115 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:29 smithi136 ceph-mon[109861]: osdmap e657: 8 total, 7 up, 8 in 2024-08-17T17:22:30.115 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:29 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:30.116 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:29 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:30.116 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:29 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:30.116 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:29 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:30.116 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:29 smithi136 ceph-mon[115078]: pool 'test-librbd-smithi136-87100-27' is full (reached quota's max_objects: 7) 2024-08-17T17:22:30.116 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:29 smithi136 ceph-mon[115078]: Health check update: 3 pool(s) full (POOL_FULL) 2024-08-17T17:22:30.116 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:29 smithi136 ceph-mon[115078]: osdmap e657: 8 total, 7 up, 8 in 2024-08-17T17:22:30.116 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:29 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:30.116 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:29 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:29 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:29 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:29 smithi165 ceph-mon[105517]: pool 'test-librbd-smithi136-87100-27' is full (reached quota's max_objects: 7) 2024-08-17T17:22:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:29 smithi165 ceph-mon[105517]: Health check update: 3 pool(s) full (POOL_FULL) 2024-08-17T17:22:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:29 smithi165 ceph-mon[105517]: osdmap e657: 8 total, 7 up, 8 in 2024-08-17T17:22:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:29 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:29 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:30.560 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:22:30.560 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:22:30.560 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:22:30.560 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:22:30.561 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:22:30.561 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:30.561 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:22:30.561 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:30.561 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:30.561 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:22:30.562 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:30.562 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:22:30.562 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:30.562 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:30.562 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:22:30.563 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:30.563 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:22:30.563 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:30.563 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:30.563 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:22:30.563 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:30.563 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:22:30.563 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:30.563 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:30.563 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:22:30.563 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:30.564 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:22:30.564 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:30.564 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:30.564 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:22:30.564 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:22:30.564 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:22:30.564 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:22:30.565 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:22:30.565 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:22:30.565 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:22:30.565 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:22:30.565 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:22:30.565 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:22:30.565 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:22:30.566 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:30.566 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:22:30.566 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:30.566 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:30.566 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:22:30.566 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:30.566 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:22:30.567 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:30.567 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:30.567 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:22:30.567 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:30.567 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:22:30.567 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:30.567 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:30.567 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:22:30.568 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:30.568 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:22:30.568 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:30.568 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:30.568 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:22:30.568 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:30.569 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:22:30.569 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:30.569 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:30.569 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:22:30.569 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:30.569 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:30.569 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:30.569 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:30.569 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:30.569 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:30.570 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:22:30.570 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:22:30.570 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:22:30.570 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:30.570 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:22:30.570 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:22:30.570 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:22:30.570 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:30.570 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:22:30.570 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:22:30.570 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:22:30.570 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:30.571 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:22:30.571 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:22:30.571 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:22:30.571 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:30.571 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:22:30.571 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:22:30.571 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:22:30.571 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 512~512 = 512 2024-08-17T17:22:30.571 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:30.571 INFO:tasks.workunit.client.0.smithi136.stdout:aio discard: 1536~512 = 0 2024-08-17T17:22:30.571 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:30.571 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:30.571 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:30.571 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:30.571 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:30.572 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 0 2024-08-17T17:22:30.572 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.572 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.572 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 0 2024-08-17T17:22:30.572 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.572 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.572 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 16384 2024-08-17T17:22:30.572 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.572 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.572 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 16384 2024-08-17T17:22:30.572 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.572 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.572 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:30.572 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:30.573 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 49152 2024-08-17T17:22:30.573 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.573 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.573 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 49152 2024-08-17T17:22:30.573 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.573 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.573 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 65536 2024-08-17T17:22:30.573 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.573 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.573 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 65536 2024-08-17T17:22:30.573 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.573 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.573 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:30.574 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:30.574 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 98304 2024-08-17T17:22:30.574 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.574 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.574 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 98304 2024-08-17T17:22:30.575 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.575 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.575 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 114688 2024-08-17T17:22:30.575 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.575 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.575 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 114688 2024-08-17T17:22:30.575 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.575 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.575 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:30.575 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:30.575 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 147456 2024-08-17T17:22:30.575 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.576 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.576 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 147456 2024-08-17T17:22:30.576 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.576 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.576 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 163840 2024-08-17T17:22:30.576 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.576 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.576 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 163840 2024-08-17T17:22:30.576 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.576 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.576 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:30.576 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:30.576 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 196608 2024-08-17T17:22:30.576 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.576 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.577 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 196608 2024-08-17T17:22:30.577 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.577 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.577 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 212992 2024-08-17T17:22:30.577 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.577 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.577 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 212992 2024-08-17T17:22:30.577 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.577 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.577 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:30.577 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:30.577 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:30.577 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:30.577 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:30.578 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:30.578 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:30.578 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.578 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.578 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:30.578 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:30.578 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:30.578 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:30.578 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:30.578 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.578 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.578 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:30.578 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:30.579 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:30.579 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:30.579 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:30.579 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.579 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.579 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:30.579 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:30.579 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:30.579 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:30.579 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:30.579 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.579 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.579 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:30.579 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:30.579 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:30.579 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:30.580 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:30.580 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:30.580 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:30.580 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:30.580 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:30.580 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:30.580 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:30.580 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.580 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.580 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:30.580 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:30.580 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:30.580 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:30.580 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:30.581 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:30.581 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:30.581 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:30.696 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0[128682]: 2024-08-17T17:22:30.491+0000 7f1cf2301640 -1 osd.0 653 log_to_monitors true 2024-08-17T17:22:31.021 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 17:22:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0[128682]: 2024-08-17T17:22:30.859+0000 7f1cea0ab640 -1 osd.0 653 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:22:31.021 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:30 smithi136 ceph-mon[109861]: pgmap v507: 521 pgs: 134 active+undersized, 49 active+undersized+degraded, 338 active+clean; 14 MiB data, 561 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 340 KiB/s wr, 6 op/s; 77/612 objects degraded (12.582%) 2024-08-17T17:22:31.021 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:30 smithi136 ceph-mon[109861]: from='osd.0 [v2:172.21.15.136:6802/1285058127,v1:172.21.15.136:6803/1285058127]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-08-17T17:22:31.021 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:30 smithi136 ceph-mon[109861]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-08-17T17:22:31.021 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:30 smithi136 ceph-mon[115078]: pgmap v507: 521 pgs: 134 active+undersized, 49 active+undersized+degraded, 338 active+clean; 14 MiB data, 561 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 340 KiB/s wr, 6 op/s; 77/612 objects degraded (12.582%) 2024-08-17T17:22:31.022 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:30 smithi136 ceph-mon[115078]: from='osd.0 [v2:172.21.15.136:6802/1285058127,v1:172.21.15.136:6803/1285058127]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-08-17T17:22:31.022 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:30 smithi136 ceph-mon[115078]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-08-17T17:22:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:30 smithi165 ceph-mon[105517]: pgmap v507: 521 pgs: 134 active+undersized, 49 active+undersized+degraded, 338 active+clean; 14 MiB data, 561 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 340 KiB/s wr, 6 op/s; 77/612 objects degraded (12.582%) 2024-08-17T17:22:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:30 smithi165 ceph-mon[105517]: from='osd.0 [v2:172.21.15.136:6802/1285058127,v1:172.21.15.136:6803/1285058127]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-08-17T17:22:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:30 smithi165 ceph-mon[105517]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-08-17T17:22:32.160 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:31 smithi136 ceph-mon[109861]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-17T17:22:32.160 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:31 smithi136 ceph-mon[109861]: osdmap e658: 8 total, 7 up, 8 in 2024-08-17T17:22:32.160 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:31 smithi136 ceph-mon[109861]: from='osd.0 [v2:172.21.15.136:6802/1285058127,v1:172.21.15.136:6803/1285058127]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-17T17:22:32.160 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:31 smithi136 ceph-mon[109861]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-17T17:22:32.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:31 smithi136 ceph-mon[115078]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-17T17:22:32.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:31 smithi136 ceph-mon[115078]: osdmap e658: 8 total, 7 up, 8 in 2024-08-17T17:22:32.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:31 smithi136 ceph-mon[115078]: from='osd.0 [v2:172.21.15.136:6802/1285058127,v1:172.21.15.136:6803/1285058127]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-17T17:22:32.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:31 smithi136 ceph-mon[115078]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-17T17:22:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:31 smithi165 ceph-mon[105517]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-17T17:22:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:31 smithi165 ceph-mon[105517]: osdmap e658: 8 total, 7 up, 8 in 2024-08-17T17:22:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:31 smithi165 ceph-mon[105517]: from='osd.0 [v2:172.21.15.136:6802/1285058127,v1:172.21.15.136:6803/1285058127]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-17T17:22:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:31 smithi165 ceph-mon[105517]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-17T17:22:32.990 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:32.990 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:32.990 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:32.990 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:32.990 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:32.990 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:32.990 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:32.991 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:32.991 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:32.991 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:32.991 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:32.991 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:32.991 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:32.991 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:32.991 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:32.991 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:32.991 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:32.991 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:32.991 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:32.991 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:32.991 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:32.991 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:32.991 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:32.992 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:32.992 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:32.992 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:32.992 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:32.992 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:32.992 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:32.992 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:32.992 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:32.992 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:32.992 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:32.992 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:32.992 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:32.992 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:32.992 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:32.992 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:32.993 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:32.993 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:32.993 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:32.993 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:32.993 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:32.993 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:32.993 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:32.993 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:32.993 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:32.993 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:32.993 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:32.993 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:32.993 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:32.993 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:32.993 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:32.993 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:32.994 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:32.994 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:32.994 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:32.994 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:32.994 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:32.994 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:32.994 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:32.994 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:32.994 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:32.994 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:32.994 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:32.994 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:32.994 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:32.995 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:32.995 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:32.995 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:32.995 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:32.995 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:32.995 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:32.995 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:32.995 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:32.995 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:32.995 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:32.995 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:32.995 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:32.995 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:32.995 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:32.995 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:32.996 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:32.996 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:32.996 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:32.996 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:32.996 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:32.996 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:32.996 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:32.996 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:32.996 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:32.996 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:32.996 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:32.996 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:32.996 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:32.997 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:32.997 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:32.997 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:32.997 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:32.997 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:32.997 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:32.997 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:32.997 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:32.997 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:32.997 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:32.997 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:32.997 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:32.997 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:32.997 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:32.998 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:32.998 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:32.998 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:32.998 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:32.998 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:32.998 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:32.998 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:32.998 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:32.998 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:32.998 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:32.998 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:32.998 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:32.998 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:22:32.999 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:22:32.999 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:22:32.999 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:22:32.999 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestIO (4016 ms) 2024-08-17T17:22:32.999 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS1 2024-08-17T17:22:32.999 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:22:33.196 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:22:32 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:22:32.941+0000 7f0cc2ea7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (47 PGs are or would become offline) 2024-08-17T17:22:33.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:32 smithi136 ceph-mon[109861]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:22:33.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:32 smithi136 ceph-mon[109861]: osd.0 [v2:172.21.15.136:6802/1285058127,v1:172.21.15.136:6803/1285058127] boot 2024-08-17T17:22:33.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:32 smithi136 ceph-mon[109861]: osdmap e659: 8 total, 8 up, 8 in 2024-08-17T17:22:33.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-17T17:22:33.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:32 smithi136 ceph-mon[109861]: pgmap v510: 521 pgs: 121 peering, 45 active+undersized, 17 active+undersized+degraded, 338 active+clean; 585 KiB data, 491 MiB used, 715 GiB / 715 GiB avail; 726 KiB/s rd, 696 KiB/s wr, 1.38k op/s; 23/618 objects degraded (3.722%) 2024-08-17T17:22:33.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:33.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:33.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:22:33.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:22:33.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:33.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:32 smithi136 ceph-mon[109861]: osdmap e660: 8 total, 8 up, 8 in 2024-08-17T17:22:33.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:32 smithi136 ceph-mon[115078]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:22:33.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:32 smithi136 ceph-mon[115078]: osd.0 [v2:172.21.15.136:6802/1285058127,v1:172.21.15.136:6803/1285058127] boot 2024-08-17T17:22:33.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:32 smithi136 ceph-mon[115078]: osdmap e659: 8 total, 8 up, 8 in 2024-08-17T17:22:33.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-17T17:22:33.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:32 smithi136 ceph-mon[115078]: pgmap v510: 521 pgs: 121 peering, 45 active+undersized, 17 active+undersized+degraded, 338 active+clean; 585 KiB data, 491 MiB used, 715 GiB / 715 GiB avail; 726 KiB/s rd, 696 KiB/s wr, 1.38k op/s; 23/618 objects degraded (3.722%) 2024-08-17T17:22:33.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:33.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:33.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:22:33.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:22:33.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:33.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:32 smithi136 ceph-mon[115078]: osdmap e660: 8 total, 8 up, 8 in 2024-08-17T17:22:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:32 smithi165 ceph-mon[105517]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:22:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:32 smithi165 ceph-mon[105517]: osd.0 [v2:172.21.15.136:6802/1285058127,v1:172.21.15.136:6803/1285058127] boot 2024-08-17T17:22:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:32 smithi165 ceph-mon[105517]: osdmap e659: 8 total, 8 up, 8 in 2024-08-17T17:22:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-17T17:22:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:32 smithi165 ceph-mon[105517]: pgmap v510: 521 pgs: 121 peering, 45 active+undersized, 17 active+undersized+degraded, 338 active+clean; 585 KiB data, 491 MiB used, 715 GiB / 715 GiB avail; 726 KiB/s rd, 696 KiB/s wr, 1.38k op/s; 23/618 objects degraded (3.722%) 2024-08-17T17:22:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:22:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:22:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:33.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:32 smithi165 ceph-mon[105517]: osdmap e660: 8 total, 8 up, 8 in 2024-08-17T17:22:34.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:33 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:22:34.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:33 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:22:34.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:33 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:22:34.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:33 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:22:34.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:33 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-08-17T17:22:34.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:33 smithi136 ceph-mon[115078]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-08-17T17:22:34.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:33 smithi136 ceph-mon[115078]: Upgrade: unsafe to stop osd(s) at this time (47 PGs are or would become offline) 2024-08-17T17:22:34.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:33 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:22:34.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:33 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:22:34.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:33 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:22:34.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:33 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:22:34.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:33 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-08-17T17:22:34.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:33 smithi136 ceph-mon[109861]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-08-17T17:22:34.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:33 smithi136 ceph-mon[109861]: Upgrade: unsafe to stop osd(s) at this time (47 PGs are or would become offline) 2024-08-17T17:22:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:33 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:22:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:33 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:22:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:33 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:22:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:33 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:22:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:33 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-08-17T17:22:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:33 smithi165 ceph-mon[105517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-08-17T17:22:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:33 smithi165 ceph-mon[105517]: Upgrade: unsafe to stop osd(s) at this time (47 PGs are or would become offline) 2024-08-17T17:22:35.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:34 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 23/618 objects degraded (3.722%), 17 pgs degraded (PG_DEGRADED) 2024-08-17T17:22:35.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:34 smithi136 ceph-mon[109861]: pool 'test-librbd-smithi136-87100-28' no longer out of quota; removing NO_QUOTA flag 2024-08-17T17:22:35.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:34 smithi136 ceph-mon[109861]: Health check update: 2 pool(s) full (POOL_FULL) 2024-08-17T17:22:35.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:34 smithi136 ceph-mon[109861]: osdmap e661: 8 total, 8 up, 8 in 2024-08-17T17:22:35.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:34 smithi136 ceph-mon[109861]: pgmap v513: 521 pgs: 121 peering, 45 active+undersized, 17 active+undersized+degraded, 338 active+clean; 585 KiB data, 491 MiB used, 715 GiB / 715 GiB avail; 1.1 MiB/s rd, 530 KiB/s wr, 2.05k op/s; 23/618 objects degraded (3.722%) 2024-08-17T17:22:35.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:34 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 23/618 objects degraded (3.722%), 17 pgs degraded (PG_DEGRADED) 2024-08-17T17:22:35.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:34 smithi136 ceph-mon[115078]: pool 'test-librbd-smithi136-87100-28' no longer out of quota; removing NO_QUOTA flag 2024-08-17T17:22:35.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:34 smithi136 ceph-mon[115078]: Health check update: 2 pool(s) full (POOL_FULL) 2024-08-17T17:22:35.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:34 smithi136 ceph-mon[115078]: osdmap e661: 8 total, 8 up, 8 in 2024-08-17T17:22:35.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:34 smithi136 ceph-mon[115078]: pgmap v513: 521 pgs: 121 peering, 45 active+undersized, 17 active+undersized+degraded, 338 active+clean; 585 KiB data, 491 MiB used, 715 GiB / 715 GiB avail; 1.1 MiB/s rd, 530 KiB/s wr, 2.05k op/s; 23/618 objects degraded (3.722%) 2024-08-17T17:22:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:34 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 23/618 objects degraded (3.722%), 17 pgs degraded (PG_DEGRADED) 2024-08-17T17:22:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:34 smithi165 ceph-mon[105517]: pool 'test-librbd-smithi136-87100-28' no longer out of quota; removing NO_QUOTA flag 2024-08-17T17:22:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:34 smithi165 ceph-mon[105517]: Health check update: 2 pool(s) full (POOL_FULL) 2024-08-17T17:22:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:34 smithi165 ceph-mon[105517]: osdmap e661: 8 total, 8 up, 8 in 2024-08-17T17:22:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:34 smithi165 ceph-mon[105517]: pgmap v513: 521 pgs: 121 peering, 45 active+undersized, 17 active+undersized+degraded, 338 active+clean; 585 KiB data, 491 MiB used, 715 GiB / 715 GiB avail; 1.1 MiB/s rd, 530 KiB/s wr, 2.05k op/s; 23/618 objects degraded (3.722%) 2024-08-17T17:22:36.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:36 smithi136 ceph-mon[109861]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/618 objects degraded (3.722%), 17 pgs degraded) 2024-08-17T17:22:36.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:36 smithi136 ceph-mon[115078]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/618 objects degraded (3.722%), 17 pgs degraded) 2024-08-17T17:22:36.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:36 smithi165 ceph-mon[105517]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/618 objects degraded (3.722%), 17 pgs degraded) 2024-08-17T17:22:37.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:37 smithi136 ceph-mon[109861]: pgmap v514: 521 pgs: 121 peering, 400 active+clean; 585 KiB data, 491 MiB used, 715 GiB / 715 GiB avail; 849 KiB/s rd, 414 KiB/s wr, 1.61k op/s; 7 B/s, 0 objects/s recovering 2024-08-17T17:22:37.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:37.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:22:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:37 smithi136 ceph-mon[115078]: pgmap v514: 521 pgs: 121 peering, 400 active+clean; 585 KiB data, 491 MiB used, 715 GiB / 715 GiB avail; 849 KiB/s rd, 414 KiB/s wr, 1.61k op/s; 7 B/s, 0 objects/s recovering 2024-08-17T17:22:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:22:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:37 smithi165 ceph-mon[105517]: pgmap v514: 521 pgs: 121 peering, 400 active+clean; 585 KiB data, 491 MiB used, 715 GiB / 715 GiB avail; 849 KiB/s rd, 414 KiB/s wr, 1.61k op/s; 7 B/s, 0 objects/s recovering 2024-08-17T17:22:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:22:37.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:22:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:22:37] "GET /metrics HTTP/1.1" 200 67697 "" "Prometheus/2.51.0" 2024-08-17T17:22:38.232 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:22:38.232 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:22:38.232 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:22:38.232 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:22:38.232 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:22:38.232 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:38.232 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:22:38.232 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:38.232 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:38.232 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:22:38.232 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:38.233 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:22:38.233 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:38.233 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:38.233 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:22:38.233 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:38.233 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:22:38.233 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:38.233 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:38.233 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:22:38.233 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:38.233 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:22:38.233 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:38.233 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:38.233 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:22:38.234 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:38.234 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:22:38.234 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:38.234 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:38.234 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:22:38.234 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:22:38.234 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:22:38.234 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:22:38.234 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:22:38.234 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:22:38.234 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:22:38.234 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:22:38.234 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:22:38.235 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:22:38.235 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:22:38.235 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:38.235 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:22:38.235 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:38.235 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:38.235 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:22:38.235 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:38.235 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:22:38.235 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:38.235 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:38.235 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:22:38.235 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:38.235 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:22:38.235 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:38.236 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:38.236 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:22:38.236 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:38.236 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:22:38.236 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:38.236 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:38.236 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:22:38.236 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:38.236 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:22:38.236 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:38.236 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:38.236 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:22:38.236 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:38.236 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:38.237 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:38.237 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:38.237 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:38.237 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:38.237 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:22:38.237 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:22:38.237 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:22:38.237 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:38.237 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:22:38.237 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:22:38.237 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:22:38.237 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:38.237 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:22:38.237 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:22:38.238 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:22:38.238 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:38.238 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:22:38.238 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:22:38.238 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:22:38.238 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:38.238 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:22:38.238 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:22:38.238 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:22:38.238 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 512~512 = 512 2024-08-17T17:22:38.238 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:38.238 INFO:tasks.workunit.client.0.smithi136.stdout:aio discard: 1536~512 = 0 2024-08-17T17:22:38.238 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:38.238 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:38.238 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:38.239 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:38.239 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:38.239 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 0 2024-08-17T17:22:38.239 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.239 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.239 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 0 2024-08-17T17:22:38.239 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.239 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.239 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 16384 2024-08-17T17:22:38.239 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.239 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.239 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 16384 2024-08-17T17:22:38.239 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.239 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.240 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:38.240 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:38.240 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 49152 2024-08-17T17:22:38.240 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.240 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.240 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 49152 2024-08-17T17:22:38.240 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.240 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.240 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 65536 2024-08-17T17:22:38.240 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.240 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.240 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 65536 2024-08-17T17:22:38.240 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.241 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.241 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:38.241 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:38.241 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 98304 2024-08-17T17:22:38.241 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.241 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.241 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 98304 2024-08-17T17:22:38.241 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.241 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.241 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 114688 2024-08-17T17:22:38.241 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.241 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.241 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 114688 2024-08-17T17:22:38.241 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.241 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.242 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:38.242 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:38.242 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 147456 2024-08-17T17:22:38.242 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.242 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.242 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 147456 2024-08-17T17:22:38.242 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.242 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.242 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 163840 2024-08-17T17:22:38.242 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.242 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.242 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 163840 2024-08-17T17:22:38.242 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.242 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.243 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:38.243 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:38.243 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 196608 2024-08-17T17:22:38.243 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.243 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.243 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 196608 2024-08-17T17:22:38.243 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.243 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.243 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 212992 2024-08-17T17:22:38.243 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.243 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.243 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 212992 2024-08-17T17:22:38.243 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.243 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.243 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:38.243 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:38.244 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:38.244 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:38.244 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:38.244 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:38.244 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:38.244 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.244 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.244 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:38.244 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:38.244 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:38.244 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:38.244 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:38.244 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.244 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.245 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:38.245 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:38.245 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:38.245 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:38.245 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:38.245 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.245 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.245 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:38.245 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:38.245 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:38.245 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:38.245 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:38.245 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.245 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.245 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:38.245 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:38.246 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:38.246 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:38.246 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:38.246 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:38.246 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:38.246 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:38.246 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:38.246 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:38.246 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:38.246 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.246 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.247 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:38.247 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:38.247 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:38.247 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:38.247 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:38.247 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:38.248 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:38.248 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:39 smithi165 ceph-mon[105517]: pgmap v515: 521 pgs: 521 active+clean; 6.7 MiB data, 499 MiB used, 715 GiB / 715 GiB avail; 903 KiB/s rd, 1.1 MiB/s wr, 1.59k op/s; 90 B/s, 2 keys/s, 0 objects/s recovering 2024-08-17T17:22:39.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:39 smithi136 ceph-mon[109861]: pgmap v515: 521 pgs: 521 active+clean; 6.7 MiB data, 499 MiB used, 715 GiB / 715 GiB avail; 903 KiB/s rd, 1.1 MiB/s wr, 1.59k op/s; 90 B/s, 2 keys/s, 0 objects/s recovering 2024-08-17T17:22:39.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:39 smithi136 ceph-mon[115078]: pgmap v515: 521 pgs: 521 active+clean; 6.7 MiB data, 499 MiB used, 715 GiB / 715 GiB avail; 903 KiB/s rd, 1.1 MiB/s wr, 1.59k op/s; 90 B/s, 2 keys/s, 0 objects/s recovering 2024-08-17T17:22:41.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:41 smithi136 ceph-mon[109861]: pgmap v516: 521 pgs: 521 active+clean; 6.7 MiB data, 499 MiB used, 715 GiB / 715 GiB avail; 148 KiB/s rd, 586 KiB/s wr, 191 op/s; 68 B/s, 2 keys/s, 0 objects/s recovering 2024-08-17T17:22:41.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:41 smithi136 ceph-mon[115078]: pgmap v516: 521 pgs: 521 active+clean; 6.7 MiB data, 499 MiB used, 715 GiB / 715 GiB avail; 148 KiB/s rd, 586 KiB/s wr, 191 op/s; 68 B/s, 2 keys/s, 0 objects/s recovering 2024-08-17T17:22:41.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:41 smithi165 ceph-mon[105517]: pgmap v516: 521 pgs: 521 active+clean; 6.7 MiB data, 499 MiB used, 715 GiB / 715 GiB avail; 148 KiB/s rd, 586 KiB/s wr, 191 op/s; 68 B/s, 2 keys/s, 0 objects/s recovering 2024-08-17T17:22:42.533 INFO:teuthology.orchestra.run.smithi136.stdout:true 2024-08-17T17:22:43.109 INFO:teuthology.orchestra.run.smithi136.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-17T17:22:43.110 INFO:teuthology.orchestra.run.smithi136.stdout:alertmanager.a smithi136 *:9093,9094 running (7m) 14s ago 13m 19.0M - 0.27.0 11f11916f8cd c3b9b5b223ab 2024-08-17T17:22:43.110 INFO:teuthology.orchestra.run.smithi136.stdout:grafana.a smithi165 *:3000 running (7m) 30s ago 13m 77.9M - c8b91775d855 04db24fce5e9 2024-08-17T17:22:43.110 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi136.ltsuqd smithi136 running (12m) 14s ago 12m 24.2M - 18.2.4-719-gced649f8 c870753d67c2 faaa06f2950c 2024-08-17T17:22:43.110 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi165.bmrbhh smithi165 running (12m) 30s ago 12m 18.4M - 18.2.4-719-gced649f8 c870753d67c2 c74b00d929cf 2024-08-17T17:22:43.110 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.x smithi165 *:8443,9283,8765 running (8m) 30s ago 17m 458M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 837fd48ff7cb 2024-08-17T17:22:43.110 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.y smithi136 *:8443,9283,8765 running (9m) 14s ago 19m 532M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9acdc6775e48 2024-08-17T17:22:43.110 INFO:teuthology.orchestra.run.smithi136.stdout:mon.a smithi136 running (6m) 14s ago 19m 70.9M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9cf82e613626 2024-08-17T17:22:43.110 INFO:teuthology.orchestra.run.smithi136.stdout:mon.b smithi165 running (4m) 30s ago 17m 50.0M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 1fd357308b8d 2024-08-17T17:22:43.110 INFO:teuthology.orchestra.run.smithi136.stdout:mon.c smithi136 running (6m) 14s ago 17m 62.6M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 2c2ac98ad9d4 2024-08-17T17:22:43.110 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.a smithi136 *:9100 running (8m) 14s ago 13m 10.7M - 1.7.0 72c9c2088986 1e4c644ae533 2024-08-17T17:22:43.110 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.b smithi165 *:9100 running (8m) 30s ago 13m 11.0M - 1.7.0 72c9c2088986 7af2d2d68b06 2024-08-17T17:22:43.111 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0 smithi136 running (18s) 14s ago 17m 11.4M 2480M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 71b7314ae4a5 2024-08-17T17:22:43.111 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1 smithi136 running (16m) 14s ago 16m 128M 2480M 18.2.4-719-gced649f8 c870753d67c2 861686b6ff82 2024-08-17T17:22:43.111 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2 smithi136 running (16m) 14s ago 16m 140M 2480M 18.2.4-719-gced649f8 c870753d67c2 640be4af4156 2024-08-17T17:22:43.111 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3 smithi136 running (16m) 14s ago 16m 133M 2480M 18.2.4-719-gced649f8 c870753d67c2 14b57a745d30 2024-08-17T17:22:43.111 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4 smithi165 running (15m) 30s ago 15m 132M 2480M 18.2.4-719-gced649f8 c870753d67c2 df366c73e1f2 2024-08-17T17:22:43.111 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5 smithi165 running (15m) 30s ago 15m 112M 2480M 18.2.4-719-gced649f8 c870753d67c2 1131665e203a 2024-08-17T17:22:43.111 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6 smithi165 running (14m) 30s ago 14m 142M 2480M 18.2.4-719-gced649f8 c870753d67c2 7bc5fd925293 2024-08-17T17:22:43.111 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7 smithi165 running (14m) 30s ago 14m 122M 2480M 18.2.4-719-gced649f8 c870753d67c2 18382e4ae3b8 2024-08-17T17:22:43.111 INFO:teuthology.orchestra.run.smithi136.stdout:prometheus.a smithi165 *:9095 running (8m) 30s ago 14m 48.0M - 2.51.0 1d3b7f56885b ae8f8b69fe2f 2024-08-17T17:22:43.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:43 smithi136 ceph-mon[109861]: pgmap v517: 521 pgs: 521 active+clean; 11 MiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1.8 MiB/s rd, 911 KiB/s wr, 974 op/s; 60 B/s, 1 keys/s, 0 objects/s recovering 2024-08-17T17:22:43.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:43 smithi136 ceph-mon[115078]: pgmap v517: 521 pgs: 521 active+clean; 11 MiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1.8 MiB/s rd, 911 KiB/s wr, 974 op/s; 60 B/s, 1 keys/s, 0 objects/s recovering 2024-08-17T17:22:43.456 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:22:43.456 INFO:teuthology.orchestra.run.smithi136.stdout: "mon": { 2024-08-17T17:22:43.456 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 3 2024-08-17T17:22:43.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:22:43.456 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr": { 2024-08-17T17:22:43.456 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 2 2024-08-17T17:22:43.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:22:43.457 INFO:teuthology.orchestra.run.smithi136.stdout: "osd": { 2024-08-17T17:22:43.457 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 7, 2024-08-17T17:22:43.457 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 1 2024-08-17T17:22:43.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:22:43.457 INFO:teuthology.orchestra.run.smithi136.stdout: "mds": { 2024-08-17T17:22:43.457 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 2 2024-08-17T17:22:43.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:22:43.457 INFO:teuthology.orchestra.run.smithi136.stdout: "overall": { 2024-08-17T17:22:43.457 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 9, 2024-08-17T17:22:43.457 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 6 2024-08-17T17:22:43.457 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:22:43.457 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:22:43.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:43 smithi165 ceph-mon[105517]: pgmap v517: 521 pgs: 521 active+clean; 11 MiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1.8 MiB/s rd, 911 KiB/s wr, 974 op/s; 60 B/s, 1 keys/s, 0 objects/s recovering 2024-08-17T17:22:43.742 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:22:43.743 INFO:teuthology.orchestra.run.smithi136.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d", 2024-08-17T17:22:43.743 INFO:teuthology.orchestra.run.smithi136.stdout: "in_progress": true, 2024-08-17T17:22:43.743 INFO:teuthology.orchestra.run.smithi136.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 4 daemons (3 remaining).", 2024-08-17T17:22:43.743 INFO:teuthology.orchestra.run.smithi136.stdout: "services_complete": [], 2024-08-17T17:22:43.743 INFO:teuthology.orchestra.run.smithi136.stdout: "progress": "1/8 daemons upgraded", 2024-08-17T17:22:43.743 INFO:teuthology.orchestra.run.smithi136.stdout: "message": "Currently upgrading osd daemons", 2024-08-17T17:22:43.743 INFO:teuthology.orchestra.run.smithi136.stdout: "is_paused": false 2024-08-17T17:22:43.743 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:22:43.977 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:43.977 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:43.977 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:43.977 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:43.977 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:43.977 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:43.977 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:43.977 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:43.977 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:43.977 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:43.978 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:43.978 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:43.978 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:43.978 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:43.978 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:43.978 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:43.978 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:43.978 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:43.978 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:43.978 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:43.978 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:43.978 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:43.978 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:43.979 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:43.979 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:43.979 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:43.979 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:43.979 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:43.979 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:43.979 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:43.979 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:43.979 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:43.979 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:43.979 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:43.979 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:43.979 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:43.979 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:43.979 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:43.980 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:43.980 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:43.980 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:43.980 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:43.980 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:43.980 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:43.980 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:43.980 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:43.980 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:43.980 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:43.980 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:43.980 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:43.980 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:43.980 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:43.980 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:43.981 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:43.981 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:43.981 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:43.981 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:43.981 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:43.981 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:43.981 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:43.981 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:43.981 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:43.981 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:43.981 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:43.981 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:43.981 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:43.981 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:43.981 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:43.982 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:43.982 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:43.982 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:43.982 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:43.982 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:43.982 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:43.982 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:43.982 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:43.982 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:43.982 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:43.983 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:43.983 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:43.983 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:43.983 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:43.983 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:43.983 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:43.983 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:43.983 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:43.983 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:43.983 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:43.983 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:43.983 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:43.983 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:43.983 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:43.984 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:43.984 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:43.984 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:43.984 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:43.984 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:43.984 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:43.984 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:43.984 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:43.984 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:43.984 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:43.984 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:43.984 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:43.984 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:43.984 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:43.985 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:43.985 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:43.985 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:43.985 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:43.985 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:43.985 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:43.985 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:43.985 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:43.985 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:43.985 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:43.985 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:43.985 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:43.985 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:43.985 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:43.986 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:43.986 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:22:43.986 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:22:43.986 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:22:43.986 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:22:43.986 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 4 2024-08-17T17:22:43.986 INFO:tasks.workunit.client.0.smithi136.stdout:read: 4 2024-08-17T17:22:43.986 INFO:tasks.workunit.client.0.smithi136.stdout:read: 4 2024-08-17T17:22:43.986 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestEncryptionLUKS1 (10987 ms) 2024-08-17T17:22:43.986 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS2 2024-08-17T17:22:43.986 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:22:44.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:44 smithi136 ceph-mon[109861]: from='client.34512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:44.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:44 smithi136 ceph-mon[109861]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:44.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:44 smithi136 ceph-mon[109861]: from='client.34524 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:44.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:44 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3889648006' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:22:44.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:44 smithi136 ceph-mon[115078]: from='client.34512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:44.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:44 smithi136 ceph-mon[115078]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:44.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:44 smithi136 ceph-mon[115078]: from='client.34524 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:44.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:44 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3889648006' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:22:44.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:44 smithi165 ceph-mon[105517]: from='client.34512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:44.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:44 smithi165 ceph-mon[105517]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:44.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:44 smithi165 ceph-mon[105517]: from='client.34524 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:44.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:44 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3889648006' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:22:45.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:45 smithi136 ceph-mon[109861]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:45.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:45 smithi136 ceph-mon[109861]: pgmap v518: 521 pgs: 521 active+clean; 11 MiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1.7 MiB/s rd, 825 KiB/s wr, 882 op/s; 54 B/s, 1 keys/s, 0 objects/s recovering 2024-08-17T17:22:45.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:45 smithi136 ceph-mon[115078]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:45.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:45 smithi136 ceph-mon[115078]: pgmap v518: 521 pgs: 521 active+clean; 11 MiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1.7 MiB/s rd, 825 KiB/s wr, 882 op/s; 54 B/s, 1 keys/s, 0 objects/s recovering 2024-08-17T17:22:45.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:45 smithi165 ceph-mon[105517]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:22:45.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:45 smithi165 ceph-mon[105517]: pgmap v518: 521 pgs: 521 active+clean; 11 MiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1.7 MiB/s rd, 825 KiB/s wr, 882 op/s; 54 B/s, 1 keys/s, 0 objects/s recovering 2024-08-17T17:22:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:47 smithi165 ceph-mon[105517]: pgmap v519: 521 pgs: 521 active+clean; 11 MiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 692 KiB/s wr, 740 op/s; 45 B/s, 1 keys/s, 0 objects/s recovering 2024-08-17T17:22:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:47 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:47 smithi136 ceph-mon[109861]: pgmap v519: 521 pgs: 521 active+clean; 11 MiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 692 KiB/s wr, 740 op/s; 45 B/s, 1 keys/s, 0 objects/s recovering 2024-08-17T17:22:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:47 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:47.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:22:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:22:47] "GET /metrics HTTP/1.1" 200 67697 "" "Prometheus/2.51.0" 2024-08-17T17:22:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:47 smithi136 ceph-mon[115078]: pgmap v519: 521 pgs: 521 active+clean; 11 MiB data, 511 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 692 KiB/s wr, 740 op/s; 45 B/s, 1 keys/s, 0 objects/s recovering 2024-08-17T17:22:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:47 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:48.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:48 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-08-17T17:22:48.590 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:48 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-08-17T17:22:48.590 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:48 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-08-17T17:22:49.502 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:49 smithi136 ceph-mon[109861]: pgmap v520: 521 pgs: 521 active+clean; 11 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 693 KiB/s wr, 742 op/s; 42 B/s, 1 keys/s, 0 objects/s recovering 2024-08-17T17:22:49.502 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:49 smithi136 ceph-mon[109861]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-08-17T17:22:49.502 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:49 smithi136 ceph-mon[109861]: Upgrade: osd.1 is safe to restart 2024-08-17T17:22:49.502 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:49 smithi136 ceph-mon[115078]: pgmap v520: 521 pgs: 521 active+clean; 11 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 693 KiB/s wr, 742 op/s; 42 B/s, 1 keys/s, 0 objects/s recovering 2024-08-17T17:22:49.503 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:49 smithi136 ceph-mon[115078]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-08-17T17:22:49.503 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:49 smithi136 ceph-mon[115078]: Upgrade: osd.1 is safe to restart 2024-08-17T17:22:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:49 smithi165 ceph-mon[105517]: pgmap v520: 521 pgs: 521 active+clean; 11 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 693 KiB/s wr, 742 op/s; 42 B/s, 1 keys/s, 0 objects/s recovering 2024-08-17T17:22:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:49 smithi165 ceph-mon[105517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-08-17T17:22:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:49 smithi165 ceph-mon[105517]: Upgrade: osd.1 is safe to restart 2024-08-17T17:22:50.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:50 smithi136 ceph-mon[109861]: Upgrade: Updating osd.1 2024-08-17T17:22:50.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:50 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:50.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:50 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-08-17T17:22:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:50 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:22:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:50 smithi136 ceph-mon[109861]: Deploying daemon osd.1 on smithi136 2024-08-17T17:22:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:50 smithi136 ceph-mon[109861]: pgmap v521: 521 pgs: 521 active+clean; 11 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 1.3 MiB/s rd, 302 KiB/s wr, 615 op/s 2024-08-17T17:22:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:50 smithi136 ceph-mon[115078]: Upgrade: Updating osd.1 2024-08-17T17:22:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:50 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:50 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-08-17T17:22:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:50 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:22:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:50 smithi136 ceph-mon[115078]: Deploying daemon osd.1 on smithi136 2024-08-17T17:22:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:50 smithi136 ceph-mon[115078]: pgmap v521: 521 pgs: 521 active+clean; 11 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 1.3 MiB/s rd, 302 KiB/s wr, 615 op/s 2024-08-17T17:22:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:50 smithi165 ceph-mon[105517]: Upgrade: Updating osd.1 2024-08-17T17:22:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:50 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:22:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:50 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-08-17T17:22:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:50 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:22:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:50 smithi165 ceph-mon[105517]: Deploying daemon osd.1 on smithi136 2024-08-17T17:22:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:50 smithi165 ceph-mon[105517]: pgmap v521: 521 pgs: 521 active+clean; 11 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 1.3 MiB/s rd, 302 KiB/s wr, 615 op/s 2024-08-17T17:22:51.533 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:22:51.533 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:22:51.533 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:22:51.533 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:22:51.534 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:22:51.534 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:51.534 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:22:51.534 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:51.534 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:51.534 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:22:51.534 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:51.534 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:22:51.534 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:51.535 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:51.535 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:22:51.535 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:51.535 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:22:51.535 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:51.535 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:51.535 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:22:51.535 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:51.535 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:22:51.535 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:51.535 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:51.536 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:22:51.536 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:51.536 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:22:51.536 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:51.536 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:51.536 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:22:51.536 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:22:51.536 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:22:51.536 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:22:51.536 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:22:51.536 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:22:51.536 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:22:51.537 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:22:51.537 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:22:51.537 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:22:51.537 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:22:51.537 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:51.537 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:22:51.537 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:51.537 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:51.537 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:22:51.537 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:51.537 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:22:51.537 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:51.538 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:51.538 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:22:51.538 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:51.538 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:22:51.538 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:51.538 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:51.538 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:22:51.538 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:51.538 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:22:51.539 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:51.539 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:51.539 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:22:51.539 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:51.539 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:22:51.539 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:51.539 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:51.539 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:22:51.539 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:51.539 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:51.539 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:51.539 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:51.540 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:51.540 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:51.540 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:22:51.540 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:22:51.540 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:22:51.540 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:51.540 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:22:51.540 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:22:51.540 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:22:51.540 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:51.541 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:22:51.541 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:22:51.541 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:22:51.541 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:51.541 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:22:51.541 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:22:51.541 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:22:51.541 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:51.542 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:22:51.542 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:22:51.542 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:22:51.542 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 512~512 = 512 2024-08-17T17:22:51.542 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:51.542 INFO:tasks.workunit.client.0.smithi136.stdout:aio discard: 1536~512 = 0 2024-08-17T17:22:51.542 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:51.542 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:51.542 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:51.542 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:51.543 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:22:51.543 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 0 2024-08-17T17:22:51.543 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.543 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.543 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 0 2024-08-17T17:22:51.543 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.543 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.543 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 16384 2024-08-17T17:22:51.543 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.543 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.543 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 16384 2024-08-17T17:22:51.544 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.544 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.544 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:51.544 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:51.544 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 49152 2024-08-17T17:22:51.544 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.544 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.544 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 49152 2024-08-17T17:22:51.544 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.544 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.544 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 65536 2024-08-17T17:22:51.545 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.545 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.545 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 65536 2024-08-17T17:22:51.545 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.545 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.545 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:51.545 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:51.545 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 98304 2024-08-17T17:22:51.545 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.545 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.545 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 98304 2024-08-17T17:22:51.545 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.546 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.546 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 114688 2024-08-17T17:22:51.546 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.546 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.546 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 114688 2024-08-17T17:22:51.546 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.546 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.546 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:51.546 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:51.546 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 147456 2024-08-17T17:22:51.546 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.547 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.547 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 147456 2024-08-17T17:22:51.547 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.547 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.547 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 163840 2024-08-17T17:22:51.547 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.547 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.547 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 163840 2024-08-17T17:22:51.547 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.547 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.547 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:51.548 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:51.548 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 196608 2024-08-17T17:22:51.548 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.548 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.548 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 196608 2024-08-17T17:22:51.548 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.548 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.548 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 212992 2024-08-17T17:22:51.548 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.548 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.548 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 212992 2024-08-17T17:22:51.548 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.549 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.549 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:51.549 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:51.549 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:51.549 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:51.549 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:51.549 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:51.549 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:51.549 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.549 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.549 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:51.549 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:51.550 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:51.550 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:51.550 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:51.550 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.550 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.550 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:51.550 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:51.550 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:51.550 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:51.551 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:51.551 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.551 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.551 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:51.551 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:51.551 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:51.551 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:51.552 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:51.552 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.552 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.552 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:51.552 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:51.552 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:51.552 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:51.552 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:51.552 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:51.552 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:51.552 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:51.552 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:51.552 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:51.553 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:51.553 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.553 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.553 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:51.553 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:51.553 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:51.553 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:51.553 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:51.553 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:51.553 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:51.553 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:51.637 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:22:51.637 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:22:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:22:52.562 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:52 smithi136 ceph-mon[109861]: pgmap v522: 521 pgs: 521 active+clean; 27 MiB data, 556 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.9 MiB/s wr, 1.07k op/s 2024-08-17T17:22:52.562 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:52 smithi136 ceph-mon[115078]: pgmap v522: 521 pgs: 521 active+clean; 27 MiB data, 556 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.9 MiB/s wr, 1.07k op/s 2024-08-17T17:22:52.936 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:52 smithi136 systemd[1]: Stopping Ceph osd.1 for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:22:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:52 smithi165 ceph-mon[105517]: pgmap v522: 521 pgs: 521 active+clean; 27 MiB data, 556 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.9 MiB/s wr, 1.07k op/s 2024-08-17T17:22:53.196 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1[53185]: 2024-08-17T17:22:52.934+0000 7f44c58f7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:22:53.196 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1[53185]: 2024-08-17T17:22:52.934+0000 7f44c58f7640 -1 osd.1 661 *** Got signal Terminated *** 2024-08-17T17:22:53.196 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1[53185]: 2024-08-17T17:22:52.934+0000 7f44c58f7640 -1 osd.1 661 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T17:22:53.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:53 smithi136 ceph-mon[109861]: osd.1 marked itself down and dead 2024-08-17T17:22:53.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:53 smithi136 ceph-mon[115078]: osd.1 marked itself down and dead 2024-08-17T17:22:54.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:53 smithi165 ceph-mon[105517]: osd.1 marked itself down and dead 2024-08-17T17:22:54.447 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:54 smithi136 podman[133843]: 2024-08-17 17:22:54.184845333 +0000 UTC m=+1.345744426 container died 861686b6ff828e90d0c74511d0015ffaa89e02663940c2aedacc21157b996fe7 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=reef-ced649f, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True) 2024-08-17T17:22:54.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:54 smithi136 ceph-mon[109861]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:22:54.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:54 smithi136 ceph-mon[109861]: osdmap e662: 8 total, 7 up, 8 in 2024-08-17T17:22:54.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:54 smithi136 ceph-mon[109861]: osd.1 marked itself dead as of e662 2024-08-17T17:22:54.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:54 smithi136 ceph-mon[109861]: pgmap v524: 521 pgs: 75 stale+active+clean, 446 active+clean; 27 MiB data, 556 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 546 op/s 2024-08-17T17:22:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:54 smithi136 ceph-mon[115078]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:22:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:54 smithi136 ceph-mon[115078]: osdmap e662: 8 total, 7 up, 8 in 2024-08-17T17:22:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:54 smithi136 ceph-mon[115078]: osd.1 marked itself dead as of e662 2024-08-17T17:22:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:54 smithi136 ceph-mon[115078]: pgmap v524: 521 pgs: 75 stale+active+clean, 446 active+clean; 27 MiB data, 556 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 546 op/s 2024-08-17T17:22:54.947 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:54 smithi136 podman[133843]: 2024-08-17 17:22:54.54040308 +0000 UTC m=+1.701302179 container cleanup 861686b6ff828e90d0c74511d0015ffaa89e02663940c2aedacc21157b996fe7 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, RELEASE=reef-ced649f, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:22:54.948 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:54 smithi136 bash[133843]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1 2024-08-17T17:22:54.948 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:54 smithi136 podman[133933]: 2024-08-17 17:22:54.767066392 +0000 UTC m=+0.578387666 container remove 861686b6ff828e90d0c74511d0015ffaa89e02663940c2aedacc21157b996fe7 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=reef-ced649f, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0) 2024-08-17T17:22:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:54 smithi165 ceph-mon[105517]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:22:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:54 smithi165 ceph-mon[105517]: osdmap e662: 8 total, 7 up, 8 in 2024-08-17T17:22:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:54 smithi165 ceph-mon[105517]: osd.1 marked itself dead as of e662 2024-08-17T17:22:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:54 smithi165 ceph-mon[105517]: pgmap v524: 521 pgs: 75 stale+active+clean, 446 active+clean; 27 MiB data, 556 MiB used, 715 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 546 op/s 2024-08-17T17:22:55.606 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:55 smithi136 ceph-mon[109861]: osdmap e663: 8 total, 7 up, 8 in 2024-08-17T17:22:55.606 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:55 smithi136 podman[134092]: 2024-08-17 17:22:55.242585764 +0000 UTC m=+0.024506424 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:22:55.607 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:55 smithi136 podman[134092]: 2024-08-17 17:22:55.351161903 +0000 UTC m=+0.133082563 container create 26805eb4441d63db86639bb71cfff1cd54e1c2af509c87d93b5fdb6d573c85d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:22:55.607 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:55 smithi136 podman[134092]: 2024-08-17 17:22:55.604892935 +0000 UTC m=+0.386813609 container init 26805eb4441d63db86639bb71cfff1cd54e1c2af509c87d93b5fdb6d573c85d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:22:55.607 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:55 smithi136 ceph-mon[115078]: osdmap e663: 8 total, 7 up, 8 in 2024-08-17T17:22:55.902 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:55 smithi136 podman[134092]: 2024-08-17 17:22:55.612723815 +0000 UTC m=+0.394644486 container start 26805eb4441d63db86639bb71cfff1cd54e1c2af509c87d93b5fdb6d573c85d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-08-17T17:22:55.902 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:55 smithi136 podman[134092]: 2024-08-17 17:22:55.704216992 +0000 UTC m=+0.486137669 container attach 26805eb4441d63db86639bb71cfff1cd54e1c2af509c87d93b5fdb6d573c85d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-08-17T17:22:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:55 smithi165 ceph-mon[105517]: osdmap e663: 8 total, 7 up, 8 in 2024-08-17T17:22:56.196 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:55 smithi136 podman[134092]: 2024-08-17 17:22:55.902180943 +0000 UTC m=+0.684101614 container died 26805eb4441d63db86639bb71cfff1cd54e1c2af509c87d93b5fdb6d573c85d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-08-17T17:22:56.839 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:56 smithi136 ceph-mon[109861]: pgmap v526: 521 pgs: 11 active+undersized, 61 stale+active+clean, 11 active+undersized+degraded, 438 active+clean; 27 MiB data, 556 MiB used, 715 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.3 MiB/s wr, 659 op/s; 19/663 objects degraded (2.866%) 2024-08-17T17:22:56.839 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:56 smithi136 podman[134171]: 2024-08-17 17:22:56.532953158 +0000 UTC m=+0.623275263 container remove 26805eb4441d63db86639bb71cfff1cd54e1c2af509c87d93b5fdb6d573c85d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-cd8bdbe, GIT_BRANCH=HEAD) 2024-08-17T17:22:56.840 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:56 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.1.service: Deactivated successfully. 2024-08-17T17:22:56.840 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:56 smithi136 systemd[1]: Stopped Ceph osd.1 for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:22:56.840 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:56 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.1.service: Consumed 31.736s CPU time. 2024-08-17T17:22:56.840 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:56 smithi136 ceph-mon[115078]: pgmap v526: 521 pgs: 11 active+undersized, 61 stale+active+clean, 11 active+undersized+degraded, 438 active+clean; 27 MiB data, 556 MiB used, 715 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.3 MiB/s wr, 659 op/s; 19/663 objects degraded (2.866%) 2024-08-17T17:22:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:56 smithi165 ceph-mon[105517]: pgmap v526: 521 pgs: 11 active+undersized, 61 stale+active+clean, 11 active+undersized+degraded, 438 active+clean; 27 MiB data, 556 MiB used, 715 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.3 MiB/s wr, 659 op/s; 19/663 objects degraded (2.866%) 2024-08-17T17:22:57.122 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:56 smithi136 systemd[1]: Starting Ceph osd.1 for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:22:57.395 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 podman[134348]: 2024-08-17 17:22:57.024051533 +0000 UTC m=+0.023594044 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:22:57.395 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 podman[134348]: 2024-08-17 17:22:57.150662775 +0000 UTC m=+0.150205266 container create 102b31d832f43fc0e8676042b35efb9763e4ff7e016315a852dff52836a00cc8 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-cd8bdbe, io.buildah.version=1.36.0) 2024-08-17T17:22:57.472 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:57.472 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:57.472 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:57.472 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:57.472 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:57.472 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:57.473 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:57.473 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:57.473 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:57.473 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:57.473 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:57.473 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:57.473 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:57.473 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:57.473 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:57.473 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:57.473 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:57.473 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:57.473 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:57.473 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:57.473 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:57.474 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:57.474 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:57.474 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:57.474 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:57.474 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:57.474 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:57.474 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:57.474 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:57.474 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:57.474 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:57.474 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:57.474 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:57.474 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:57.475 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:57.475 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:57.475 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:57.475 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:57.475 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:57.475 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:57.475 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:57.475 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:57.475 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:57.475 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:57.475 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:57.475 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:57.475 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:57.475 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:57.476 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:57.476 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:57.476 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:57.476 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:57.476 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:57.476 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:57.476 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:57.476 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:57.476 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:57.476 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:57.476 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:57.476 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:57.476 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:57.476 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:57.477 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:57.477 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:57.477 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:57.477 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:57.477 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:57.477 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:57.477 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:57.477 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:57.477 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:57.478 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:57.478 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:57.478 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:57.478 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:57.478 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:57.478 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:57.478 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:57.478 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:57.478 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:57.478 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:57.478 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:57.478 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:57.478 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:57.479 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:57.479 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:57.479 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:57.479 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:57.479 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:57.479 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:57.479 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:57.479 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:57.479 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:57.479 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:57.479 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:57.479 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:57.479 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:57.479 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:57.480 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:57.480 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:57.480 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:57.480 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:57.480 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:57.480 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:57.480 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:57.480 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:57.480 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:57.480 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:57.480 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:57.480 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:57.480 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:22:57.480 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:22:57.481 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:22:57.481 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:22:57.481 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:22:57.481 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:57.481 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:57.481 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:57.481 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:22:57.481 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:22:57.481 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:22:57.481 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:22:57.481 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:22:57.481 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:22:57.481 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:22:57.481 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 4 2024-08-17T17:22:57.481 INFO:tasks.workunit.client.0.smithi136.stdout:read: 4 2024-08-17T17:22:57.482 INFO:tasks.workunit.client.0.smithi136.stdout:read: 4 2024-08-17T17:22:57.482 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestEncryptionLUKS2 (13495 ms) 2024-08-17T17:22:57.482 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestCloneEncryption 2024-08-17T17:22:57.482 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:22:57.648 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:57 smithi136 ceph-mon[109861]: Health check failed: Degraded data redundancy: 19/663 objects degraded (2.866%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T17:22:57.648 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:57 smithi136 ceph-mon[115078]: Health check failed: Degraded data redundancy: 19/663 objects degraded (2.866%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T17:22:57.649 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:22:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:22:57] "GET /metrics HTTP/1.1" 200 67703 "" "Prometheus/2.51.0" 2024-08-17T17:22:57.649 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 podman[134348]: 2024-08-17 17:22:57.394746186 +0000 UTC m=+0.394288688 container init 102b31d832f43fc0e8676042b35efb9763e4ff7e016315a852dff52836a00cc8 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-activate, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, RELEASE=main-cd8bdbe, GIT_BRANCH=HEAD) 2024-08-17T17:22:57.649 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 podman[134348]: 2024-08-17 17:22:57.399683155 +0000 UTC m=+0.399225652 container start 102b31d832f43fc0e8676042b35efb9763e4ff7e016315a852dff52836a00cc8 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-activate, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-cd8bdbe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-08-17T17:22:57.649 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 podman[134348]: 2024-08-17 17:22:57.485990705 +0000 UTC m=+0.485533202 container attach 102b31d832f43fc0e8676042b35efb9763e4ff7e016315a852dff52836a00cc8 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-activate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=main-cd8bdbe) 2024-08-17T17:22:57.649 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-activate[134386]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:22:57.649 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 bash[134348]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:22:57.649 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-activate[134386]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:22:57.649 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 bash[134348]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:22:57.649 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-activate[134386]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-17T17:22:57.650 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 bash[134348]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-17T17:22:57.650 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-activate[134386]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:22:57.650 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 bash[134348]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:22:57.650 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-activate[134386]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:22:57.650 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 bash[134348]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:22:57.650 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-activate[134386]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-17T17:22:57.650 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 bash[134348]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-17T17:22:57.912 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:57 smithi165 ceph-mon[105517]: Health check failed: Degraded data redundancy: 19/663 objects degraded (2.866%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T17:22:57.933 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-activate[134386]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2024-08-17T17:22:57.933 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 bash[134348]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2024-08-17T17:22:58.196 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-activate[134386]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2024-08-17T17:22:58.197 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 bash[134348]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2024-08-17T17:22:58.197 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-activate[134386]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2024-08-17T17:22:58.197 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 bash[134348]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2024-08-17T17:22:58.197 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-activate[134386]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-08-17T17:22:58.197 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 bash[134348]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-08-17T17:22:58.197 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-activate[134386]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-17T17:22:58.197 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 bash[134348]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-17T17:22:58.197 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-activate[134386]: --> ceph-volume lvm activate successful for osd ID: 1 2024-08-17T17:22:58.197 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 bash[134348]: --> ceph-volume lvm activate successful for osd ID: 1 2024-08-17T17:22:58.197 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:57 smithi136 podman[134348]: 2024-08-17 17:22:57.961901758 +0000 UTC m=+0.961444251 container died 102b31d832f43fc0e8676042b35efb9763e4ff7e016315a852dff52836a00cc8 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-activate, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-08-17T17:22:58.880 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:58 smithi136 ceph-mon[109861]: osdmap e664: 8 total, 7 up, 8 in 2024-08-17T17:22:58.881 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:22:58 smithi136 ceph-mon[109861]: pgmap v528: 521 pgs: 136 active+undersized, 52 active+undersized+degraded, 333 active+clean; 31 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 3.6 MiB/s rd, 261 KiB/s wr, 509 op/s; 83/666 objects degraded (12.462%) 2024-08-17T17:22:58.881 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:58 smithi136 ceph-mon[115078]: osdmap e664: 8 total, 7 up, 8 in 2024-08-17T17:22:58.881 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:22:58 smithi136 ceph-mon[115078]: pgmap v528: 521 pgs: 136 active+undersized, 52 active+undersized+degraded, 333 active+clean; 31 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 3.6 MiB/s rd, 261 KiB/s wr, 509 op/s; 83/666 objects degraded (12.462%) 2024-08-17T17:22:58.881 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:58 smithi136 podman[134527]: 2024-08-17 17:22:58.548228622 +0000 UTC m=+0.580661588 container remove 102b31d832f43fc0e8676042b35efb9763e4ff7e016315a852dff52836a00cc8 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-activate, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-cd8bdbe, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T17:22:58.881 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:58 smithi136 podman[134590]: 2024-08-17 17:22:58.666466972 +0000 UTC m=+0.018220975 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:22:58.881 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:58 smithi136 podman[134590]: 2024-08-17 17:22:58.774195392 +0000 UTC m=+0.125949391 container create a0bc94c66763d209c210caf3f644d339d3c688d725a46bb6e40552dd49c09380 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1, org.label-schema.license=GPLv2, RELEASE=main-cd8bdbe, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True) 2024-08-17T17:22:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:58 smithi165 ceph-mon[105517]: osdmap e664: 8 total, 7 up, 8 in 2024-08-17T17:22:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:22:58 smithi165 ceph-mon[105517]: pgmap v528: 521 pgs: 136 active+undersized, 52 active+undersized+degraded, 333 active+clean; 31 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 3.6 MiB/s rd, 261 KiB/s wr, 509 op/s; 83/666 objects degraded (12.462%) 2024-08-17T17:22:59.446 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:59 smithi136 podman[134590]: 2024-08-17 17:22:59.002008846 +0000 UTC m=+0.353762843 container init a0bc94c66763d209c210caf3f644d339d3c688d725a46bb6e40552dd49c09380 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-08-17T17:22:59.446 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:59 smithi136 podman[134590]: 2024-08-17 17:22:59.007012977 +0000 UTC m=+0.358766972 container start a0bc94c66763d209c210caf3f644d339d3c688d725a46bb6e40552dd49c09380 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-cd8bdbe) 2024-08-17T17:22:59.447 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:59 smithi136 ceph-osd[134629]: -- 172.21.15.136:0/1649291841 <== mon.0 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5561ab857a40 con 0x5561ab9fa400 2024-08-17T17:22:59.447 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:59 smithi136 bash[134590]: a0bc94c66763d209c210caf3f644d339d3c688d725a46bb6e40552dd49c09380 2024-08-17T17:22:59.447 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:22:59 smithi136 systemd[1]: Started Ceph osd.1 for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:23:00.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:00 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:00.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:00 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:00.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:00 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:00.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:00 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:23:00.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:00 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:00.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:00 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:00.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:00 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:00.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:00 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:23:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:00 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:00 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:00 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:00 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:23:01.170 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:23:00 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1[134606]: 2024-08-17T17:23:00.886+0000 7f3d06bb0640 -1 Falling back to public interface 2024-08-17T17:23:01.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:01 smithi136 ceph-mon[109861]: pgmap v529: 521 pgs: 136 active+undersized, 52 active+undersized+degraded, 333 active+clean; 31 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 3.4 MiB/s rd, 246 KiB/s wr, 479 op/s; 83/666 objects degraded (12.462%) 2024-08-17T17:23:01.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:01 smithi136 ceph-mon[115078]: pgmap v529: 521 pgs: 136 active+undersized, 52 active+undersized+degraded, 333 active+clean; 31 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 3.4 MiB/s rd, 246 KiB/s wr, 479 op/s; 83/666 objects degraded (12.462%) 2024-08-17T17:23:01.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:01 smithi165 ceph-mon[105517]: pgmap v529: 521 pgs: 136 active+undersized, 52 active+undersized+degraded, 333 active+clean; 31 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 3.4 MiB/s rd, 246 KiB/s wr, 479 op/s; 83/666 objects degraded (12.462%) 2024-08-17T17:23:02.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:02 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 86/696 objects degraded (12.356%), 52 pgs degraded (PG_DEGRADED) 2024-08-17T17:23:02.657 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:02 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 86/696 objects degraded (12.356%), 52 pgs degraded (PG_DEGRADED) 2024-08-17T17:23:02.657 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:02 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 86/696 objects degraded (12.356%), 52 pgs degraded (PG_DEGRADED) 2024-08-17T17:23:03.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:03 smithi136 ceph-mon[109861]: pgmap v530: 521 pgs: 136 active+undersized, 52 active+undersized+degraded, 333 active+clean; 35 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 2.8 MiB/s rd, 710 KiB/s wr, 418 op/s; 86/696 objects degraded (12.356%) 2024-08-17T17:23:03.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:03 smithi136 ceph-mon[115078]: pgmap v530: 521 pgs: 136 active+undersized, 52 active+undersized+degraded, 333 active+clean; 35 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 2.8 MiB/s rd, 710 KiB/s wr, 418 op/s; 86/696 objects degraded (12.356%) 2024-08-17T17:23:03.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:03 smithi165 ceph-mon[105517]: pgmap v530: 521 pgs: 136 active+undersized, 52 active+undersized+degraded, 333 active+clean; 35 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 2.8 MiB/s rd, 710 KiB/s wr, 418 op/s; 86/696 objects degraded (12.356%) 2024-08-17T17:23:04.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:04 smithi165 ceph-mon[105517]: osdmap e665: 8 total, 7 up, 8 in 2024-08-17T17:23:04.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:04 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:04.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:04 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:04.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:04 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:04.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:04 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:04.553 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:04 smithi136 ceph-mon[109861]: osdmap e665: 8 total, 7 up, 8 in 2024-08-17T17:23:04.553 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:04 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:04.553 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:04 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:04.554 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:04 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:04.554 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:04 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:04.554 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:04 smithi136 ceph-mon[115078]: osdmap e665: 8 total, 7 up, 8 in 2024-08-17T17:23:04.554 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:04 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:04.554 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:04 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:04.554 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:04 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:04.554 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:04 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:04.823 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:23:04 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1[134606]: 2024-08-17T17:23:04.611+0000 7f3d06bb0640 -1 osd.1 662 log_to_monitors true 2024-08-17T17:23:05.692 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:05 smithi136 ceph-mon[109861]: pgmap v532: 521 pgs: 136 active+undersized, 52 active+undersized+degraded, 333 active+clean; 35 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 2.8 MiB/s rd, 710 KiB/s wr, 437 op/s; 86/696 objects degraded (12.356%) 2024-08-17T17:23:05.692 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:05 smithi136 ceph-mon[109861]: from='osd.1 [v2:172.21.15.136:6810/717195127,v1:172.21.15.136:6811/717195127]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-17T17:23:05.693 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:05 smithi136 ceph-mon[115078]: pgmap v532: 521 pgs: 136 active+undersized, 52 active+undersized+degraded, 333 active+clean; 35 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 2.8 MiB/s rd, 710 KiB/s wr, 437 op/s; 86/696 objects degraded (12.356%) 2024-08-17T17:23:05.693 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:05 smithi136 ceph-mon[115078]: from='osd.1 [v2:172.21.15.136:6810/717195127,v1:172.21.15.136:6811/717195127]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-17T17:23:05.693 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 17:23:05 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1[134606]: 2024-08-17T17:23:05.370+0000 7f3cfe95a640 -1 osd.1 662 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:23:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:05 smithi165 ceph-mon[105517]: pgmap v532: 521 pgs: 136 active+undersized, 52 active+undersized+degraded, 333 active+clean; 35 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 2.8 MiB/s rd, 710 KiB/s wr, 437 op/s; 86/696 objects degraded (12.356%) 2024-08-17T17:23:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:05 smithi165 ceph-mon[105517]: from='osd.1 [v2:172.21.15.136:6810/717195127,v1:172.21.15.136:6811/717195127]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-17T17:23:06.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:06 smithi136 ceph-mon[109861]: from='osd.1 [v2:172.21.15.136:6810/717195127,v1:172.21.15.136:6811/717195127]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-17T17:23:06.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:06 smithi136 ceph-mon[109861]: osdmap e666: 8 total, 7 up, 8 in 2024-08-17T17:23:06.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:06 smithi136 ceph-mon[109861]: from='osd.1 [v2:172.21.15.136:6810/717195127,v1:172.21.15.136:6811/717195127]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-17T17:23:06.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:06 smithi136 ceph-mon[109861]: pgmap v534: 521 pgs: 136 active+undersized, 52 active+undersized+degraded, 333 active+clean; 35 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 514 KiB/s wr, 36 op/s; 86/696 objects degraded (12.356%) 2024-08-17T17:23:06.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:06.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:23:06.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:06 smithi136 ceph-mon[115078]: from='osd.1 [v2:172.21.15.136:6810/717195127,v1:172.21.15.136:6811/717195127]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-17T17:23:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:06 smithi136 ceph-mon[115078]: osdmap e666: 8 total, 7 up, 8 in 2024-08-17T17:23:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:06 smithi136 ceph-mon[115078]: from='osd.1 [v2:172.21.15.136:6810/717195127,v1:172.21.15.136:6811/717195127]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-17T17:23:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:06 smithi136 ceph-mon[115078]: pgmap v534: 521 pgs: 136 active+undersized, 52 active+undersized+degraded, 333 active+clean; 35 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 514 KiB/s wr, 36 op/s; 86/696 objects degraded (12.356%) 2024-08-17T17:23:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:06.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:23:06.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:06 smithi165 ceph-mon[105517]: from='osd.1 [v2:172.21.15.136:6810/717195127,v1:172.21.15.136:6811/717195127]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-17T17:23:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:06 smithi165 ceph-mon[105517]: osdmap e666: 8 total, 7 up, 8 in 2024-08-17T17:23:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:06 smithi165 ceph-mon[105517]: from='osd.1 [v2:172.21.15.136:6810/717195127,v1:172.21.15.136:6811/717195127]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-17T17:23:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:06 smithi165 ceph-mon[105517]: pgmap v534: 521 pgs: 136 active+undersized, 52 active+undersized+degraded, 333 active+clean; 35 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 514 KiB/s wr, 36 op/s; 86/696 objects degraded (12.356%) 2024-08-17T17:23:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:23:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:07 smithi136 ceph-mon[109861]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:23:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:07 smithi136 ceph-mon[109861]: osd.1 [v2:172.21.15.136:6810/717195127,v1:172.21.15.136:6811/717195127] boot 2024-08-17T17:23:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:07 smithi136 ceph-mon[109861]: osdmap e667: 8 total, 8 up, 8 in 2024-08-17T17:23:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-17T17:23:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:07 smithi136 ceph-mon[115078]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:23:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:07 smithi136 ceph-mon[115078]: osd.1 [v2:172.21.15.136:6810/717195127,v1:172.21.15.136:6811/717195127] boot 2024-08-17T17:23:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:07 smithi136 ceph-mon[115078]: osdmap e667: 8 total, 8 up, 8 in 2024-08-17T17:23:07.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-17T17:23:07.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:07 smithi165 ceph-mon[105517]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:23:07.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:07 smithi165 ceph-mon[105517]: osd.1 [v2:172.21.15.136:6810/717195127,v1:172.21.15.136:6811/717195127] boot 2024-08-17T17:23:07.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:07 smithi165 ceph-mon[105517]: osdmap e667: 8 total, 8 up, 8 in 2024-08-17T17:23:07.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-17T17:23:07.934 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:23:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:23:07] "GET /metrics HTTP/1.1" 200 67645 "" "Prometheus/2.51.0" 2024-08-17T17:23:08.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[109861]: osdmap e668: 8 total, 8 up, 8 in 2024-08-17T17:23:08.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[109861]: pgmap v537: 521 pgs: 105 peering, 53 active+undersized, 30 active+undersized+degraded, 333 active+clean; 35 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 2.8 KiB/s wr, 61 op/s; 49/708 objects degraded (6.921%) 2024-08-17T17:23:08.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:08.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:08.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:23:08.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:23:08.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:08.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:23:08.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:23:08.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:23:08.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:23:08.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-08-17T17:23:08.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[109861]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-08-17T17:23:08.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[109861]: Upgrade: unsafe to stop osd(s) at this time (64 PGs are or would become offline) 2024-08-17T17:23:08.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[115078]: osdmap e668: 8 total, 8 up, 8 in 2024-08-17T17:23:08.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[115078]: pgmap v537: 521 pgs: 105 peering, 53 active+undersized, 30 active+undersized+degraded, 333 active+clean; 35 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 2.8 KiB/s wr, 61 op/s; 49/708 objects degraded (6.921%) 2024-08-17T17:23:08.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:08.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:08.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:23:08.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:23:08.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:08.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:23:08.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:23:08.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:23:08.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:23:08.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-08-17T17:23:08.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[115078]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-08-17T17:23:08.702 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-mon[115078]: Upgrade: unsafe to stop osd(s) at this time (64 PGs are or would become offline) 2024-08-17T17:23:08.703 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:23:08 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:23:08.197+0000 7f0cc2ea7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (64 PGs are or would become offline) 2024-08-17T17:23:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:08 smithi165 ceph-mon[105517]: osdmap e668: 8 total, 8 up, 8 in 2024-08-17T17:23:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:08 smithi165 ceph-mon[105517]: pgmap v537: 521 pgs: 105 peering, 53 active+undersized, 30 active+undersized+degraded, 333 active+clean; 35 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 2.8 KiB/s wr, 61 op/s; 49/708 objects degraded (6.921%) 2024-08-17T17:23:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:23:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:23:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:23:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:23:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:23:08.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:23:08.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-08-17T17:23:08.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:08 smithi165 ceph-mon[105517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-08-17T17:23:08.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:08 smithi165 ceph-mon[105517]: Upgrade: unsafe to stop osd(s) at this time (64 PGs are or would become offline) 2024-08-17T17:23:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:09 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 49/708 objects degraded (6.921%), 30 pgs degraded (PG_DEGRADED) 2024-08-17T17:23:09.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:09 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 49/708 objects degraded (6.921%), 30 pgs degraded (PG_DEGRADED) 2024-08-17T17:23:09.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:09 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 49/708 objects degraded (6.921%), 30 pgs degraded (PG_DEGRADED) 2024-08-17T17:23:10.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:10 smithi136 ceph-mon[109861]: pgmap v538: 521 pgs: 105 peering, 53 active+undersized, 30 active+undersized+degraded, 333 active+clean; 35 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 2.2 KiB/s wr, 48 op/s; 49/708 objects degraded (6.921%) 2024-08-17T17:23:10.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:10 smithi136 ceph-mon[115078]: pgmap v538: 521 pgs: 105 peering, 53 active+undersized, 30 active+undersized+degraded, 333 active+clean; 35 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 2.2 KiB/s wr, 48 op/s; 49/708 objects degraded (6.921%) 2024-08-17T17:23:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:10 smithi165 ceph-mon[105517]: pgmap v538: 521 pgs: 105 peering, 53 active+undersized, 30 active+undersized+degraded, 333 active+clean; 35 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 2.2 KiB/s wr, 48 op/s; 49/708 objects degraded (6.921%) 2024-08-17T17:23:12.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:12 smithi136 ceph-mon[109861]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 49/708 objects degraded (6.921%), 30 pgs degraded) 2024-08-17T17:23:12.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:12 smithi136 ceph-mon[115078]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 49/708 objects degraded (6.921%), 30 pgs degraded) 2024-08-17T17:23:12.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:12 smithi165 ceph-mon[105517]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 49/708 objects degraded (6.921%), 30 pgs degraded) 2024-08-17T17:23:13.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:13 smithi136 ceph-mon[109861]: pgmap v539: 521 pgs: 521 active+clean; 51 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s rd, 2.4 MiB/s wr, 50 op/s; 617 KiB/s, 0 objects/s recovering 2024-08-17T17:23:13.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:13 smithi136 ceph-mon[115078]: pgmap v539: 521 pgs: 521 active+clean; 51 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s rd, 2.4 MiB/s wr, 50 op/s; 617 KiB/s, 0 objects/s recovering 2024-08-17T17:23:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:13 smithi165 ceph-mon[105517]: pgmap v539: 521 pgs: 521 active+clean; 51 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s rd, 2.4 MiB/s wr, 50 op/s; 617 KiB/s, 0 objects/s recovering 2024-08-17T17:23:14.056 INFO:teuthology.orchestra.run.smithi136.stdout:true 2024-08-17T17:23:14.634 INFO:teuthology.orchestra.run.smithi136.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-17T17:23:14.634 INFO:teuthology.orchestra.run.smithi136.stdout:alertmanager.a smithi136 *:9093,9094 running (8m) 11s ago 14m 19.0M - 0.27.0 11f11916f8cd c3b9b5b223ab 2024-08-17T17:23:14.635 INFO:teuthology.orchestra.run.smithi136.stdout:grafana.a smithi165 *:3000 running (8m) 62s ago 13m 77.9M - c8b91775d855 04db24fce5e9 2024-08-17T17:23:14.635 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi136.ltsuqd smithi136 running (13m) 11s ago 12m 24.4M - 18.2.4-719-gced649f8 c870753d67c2 faaa06f2950c 2024-08-17T17:23:14.635 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi165.bmrbhh smithi165 running (12m) 62s ago 12m 18.4M - 18.2.4-719-gced649f8 c870753d67c2 c74b00d929cf 2024-08-17T17:23:14.635 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.x smithi165 *:8443,9283,8765 running (9m) 62s ago 18m 458M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 837fd48ff7cb 2024-08-17T17:23:14.635 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.y smithi136 *:8443,9283,8765 running (9m) 11s ago 19m 533M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9acdc6775e48 2024-08-17T17:23:14.635 INFO:teuthology.orchestra.run.smithi136.stdout:mon.a smithi136 running (7m) 11s ago 19m 73.5M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9cf82e613626 2024-08-17T17:23:14.635 INFO:teuthology.orchestra.run.smithi136.stdout:mon.b smithi165 running (4m) 62s ago 18m 50.0M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 1fd357308b8d 2024-08-17T17:23:14.635 INFO:teuthology.orchestra.run.smithi136.stdout:mon.c smithi136 running (6m) 11s ago 18m 65.1M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 2c2ac98ad9d4 2024-08-17T17:23:14.635 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.a smithi136 *:9100 running (9m) 11s ago 13m 11.0M - 1.7.0 72c9c2088986 1e4c644ae533 2024-08-17T17:23:14.635 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.b smithi165 *:9100 running (9m) 62s ago 13m 11.0M - 1.7.0 72c9c2088986 7af2d2d68b06 2024-08-17T17:23:14.635 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0 smithi136 running (49s) 11s ago 17m 55.9M 2480M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 71b7314ae4a5 2024-08-17T17:23:14.635 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1 smithi136 running (15s) 11s ago 17m 11.9M 2480M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb a0bc94c66763 2024-08-17T17:23:14.636 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2 smithi136 running (17m) 11s ago 17m 142M 2480M 18.2.4-719-gced649f8 c870753d67c2 640be4af4156 2024-08-17T17:23:14.636 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3 smithi136 running (16m) 11s ago 16m 135M 2480M 18.2.4-719-gced649f8 c870753d67c2 14b57a745d30 2024-08-17T17:23:14.636 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4 smithi165 running (16m) 62s ago 16m 132M 2480M 18.2.4-719-gced649f8 c870753d67c2 df366c73e1f2 2024-08-17T17:23:14.636 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5 smithi165 running (15m) 62s ago 15m 112M 2480M 18.2.4-719-gced649f8 c870753d67c2 1131665e203a 2024-08-17T17:23:14.636 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6 smithi165 running (15m) 62s ago 15m 142M 2480M 18.2.4-719-gced649f8 c870753d67c2 7bc5fd925293 2024-08-17T17:23:14.636 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7 smithi165 running (15m) 62s ago 15m 122M 2480M 18.2.4-719-gced649f8 c870753d67c2 18382e4ae3b8 2024-08-17T17:23:14.636 INFO:teuthology.orchestra.run.smithi136.stdout:prometheus.a smithi165 *:9095 running (8m) 62s ago 14m 48.0M - 2.51.0 1d3b7f56885b ae8f8b69fe2f 2024-08-17T17:23:14.982 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:23:14.982 INFO:teuthology.orchestra.run.smithi136.stdout: "mon": { 2024-08-17T17:23:14.982 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 3 2024-08-17T17:23:14.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:23:14.982 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr": { 2024-08-17T17:23:14.982 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 2 2024-08-17T17:23:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:23:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: "osd": { 2024-08-17T17:23:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 6, 2024-08-17T17:23:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 2 2024-08-17T17:23:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:23:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: "mds": { 2024-08-17T17:23:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 2 2024-08-17T17:23:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:23:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: "overall": { 2024-08-17T17:23:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 8, 2024-08-17T17:23:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 7 2024-08-17T17:23:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:23:14.983 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:23:15.334 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:23:15.334 INFO:teuthology.orchestra.run.smithi136.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d", 2024-08-17T17:23:15.334 INFO:teuthology.orchestra.run.smithi136.stdout: "in_progress": true, 2024-08-17T17:23:15.334 INFO:teuthology.orchestra.run.smithi136.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 4 daemons (2 remaining).", 2024-08-17T17:23:15.334 INFO:teuthology.orchestra.run.smithi136.stdout: "services_complete": [], 2024-08-17T17:23:15.334 INFO:teuthology.orchestra.run.smithi136.stdout: "progress": "2/8 daemons upgraded", 2024-08-17T17:23:15.335 INFO:teuthology.orchestra.run.smithi136.stdout: "message": "Currently upgrading osd daemons", 2024-08-17T17:23:15.335 INFO:teuthology.orchestra.run.smithi136.stdout: "is_paused": false 2024-08-17T17:23:15.335 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:23:15.335 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:15 smithi136 ceph-mon[109861]: pgmap v540: 521 pgs: 521 active+clean; 51 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 2.0 MiB/s wr, 42 op/s; 512 KiB/s, 0 objects/s recovering 2024-08-17T17:23:15.335 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:15 smithi136 ceph-mon[109861]: from='client.44459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:23:15.335 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:15 smithi136 ceph-mon[109861]: osdmap e669: 8 total, 8 up, 8 in 2024-08-17T17:23:15.335 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:15 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1798022229' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:23:15.335 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:15 smithi136 ceph-mon[115078]: pgmap v540: 521 pgs: 521 active+clean; 51 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 2.0 MiB/s wr, 42 op/s; 512 KiB/s, 0 objects/s recovering 2024-08-17T17:23:15.336 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:15 smithi136 ceph-mon[115078]: from='client.44459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:23:15.336 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:15 smithi136 ceph-mon[115078]: osdmap e669: 8 total, 8 up, 8 in 2024-08-17T17:23:15.336 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:15 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1798022229' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:23:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:15 smithi165 ceph-mon[105517]: pgmap v540: 521 pgs: 521 active+clean; 51 MiB data, 606 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 2.0 MiB/s wr, 42 op/s; 512 KiB/s, 0 objects/s recovering 2024-08-17T17:23:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:15 smithi165 ceph-mon[105517]: from='client.44459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:23:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:15 smithi165 ceph-mon[105517]: osdmap e669: 8 total, 8 up, 8 in 2024-08-17T17:23:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:15 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1798022229' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:23:16.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:16 smithi136 ceph-mon[109861]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:23:16.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:16 smithi136 ceph-mon[109861]: from='client.44471 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:23:16.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:16 smithi136 ceph-mon[115078]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:23:16.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:16 smithi136 ceph-mon[115078]: from='client.44471 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:23:16.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:16 smithi165 ceph-mon[105517]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:23:16.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:16 smithi165 ceph-mon[105517]: from='client.44471 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:23:17.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:17 smithi136 ceph-mon[115078]: from='client.54319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:23:17.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:17 smithi136 ceph-mon[115078]: pgmap v542: 521 pgs: 521 active+clean; 51 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 1.9 MiB/s wr, 11 op/s; 475 KiB/s, 0 objects/s recovering 2024-08-17T17:23:17.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:17 smithi136 ceph-mon[109861]: from='client.54319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:23:17.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:17 smithi136 ceph-mon[109861]: pgmap v542: 521 pgs: 521 active+clean; 51 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 1.9 MiB/s wr, 11 op/s; 475 KiB/s, 0 objects/s recovering 2024-08-17T17:23:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:17 smithi165 ceph-mon[105517]: from='client.54319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:23:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:17 smithi165 ceph-mon[105517]: pgmap v542: 521 pgs: 521 active+clean; 51 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 1.9 MiB/s wr, 11 op/s; 475 KiB/s, 0 objects/s recovering 2024-08-17T17:23:17.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:23:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:23:17] "GET /metrics HTTP/1.1" 200 67645 "" "Prometheus/2.51.0" 2024-08-17T17:23:19.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:19 smithi136 ceph-mon[109861]: pgmap v543: 521 pgs: 521 active+clean; 55 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 2.0 MiB/s wr, 40 op/s; 409 KiB/s, 0 objects/s recovering 2024-08-17T17:23:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:19 smithi136 ceph-mon[115078]: pgmap v543: 521 pgs: 521 active+clean; 55 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 2.0 MiB/s wr, 40 op/s; 409 KiB/s, 0 objects/s recovering 2024-08-17T17:23:19.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:19 smithi165 ceph-mon[105517]: pgmap v543: 521 pgs: 521 active+clean; 55 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 2.0 MiB/s wr, 40 op/s; 409 KiB/s, 0 objects/s recovering 2024-08-17T17:23:21.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:21 smithi136 ceph-mon[109861]: pgmap v544: 521 pgs: 521 active+clean; 55 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 2.0 MiB/s wr, 40 op/s; 409 KiB/s, 0 objects/s recovering 2024-08-17T17:23:21.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:21 smithi136 ceph-mon[115078]: pgmap v544: 521 pgs: 521 active+clean; 55 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 2.0 MiB/s wr, 40 op/s; 409 KiB/s, 0 objects/s recovering 2024-08-17T17:23:21.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:21 smithi165 ceph-mon[105517]: pgmap v544: 521 pgs: 521 active+clean; 55 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 2.0 MiB/s wr, 40 op/s; 409 KiB/s, 0 objects/s recovering 2024-08-17T17:23:22.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:22.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:23:22.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:23:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:23:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:23.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:23 smithi136 ceph-mon[109861]: pgmap v545: 521 pgs: 521 active+clean; 67 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 1.6 MiB/s wr, 40 op/s 2024-08-17T17:23:23.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:23 smithi136 ceph-mon[115078]: pgmap v545: 521 pgs: 521 active+clean; 67 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 1.6 MiB/s wr, 40 op/s 2024-08-17T17:23:23.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:23 smithi165 ceph-mon[105517]: pgmap v545: 521 pgs: 521 active+clean; 67 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 1.6 MiB/s wr, 40 op/s 2024-08-17T17:23:24.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:24 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-08-17T17:23:24.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:24 smithi136 ceph-mon[109861]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-08-17T17:23:24.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:24 smithi136 ceph-mon[109861]: Upgrade: osd.2 is safe to restart 2024-08-17T17:23:24.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:24 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-08-17T17:23:24.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:24 smithi136 ceph-mon[115078]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-08-17T17:23:24.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:24 smithi136 ceph-mon[115078]: Upgrade: osd.2 is safe to restart 2024-08-17T17:23:24.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:24 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-08-17T17:23:24.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:24 smithi165 ceph-mon[105517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-08-17T17:23:24.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:24 smithi165 ceph-mon[105517]: Upgrade: osd.2 is safe to restart 2024-08-17T17:23:25.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:25 smithi136 ceph-mon[109861]: pgmap v546: 521 pgs: 521 active+clean; 67 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 1.6 MiB/s wr, 40 op/s 2024-08-17T17:23:25.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:25 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:25.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:25 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-08-17T17:23:25.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:25 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:23:25.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:25 smithi136 ceph-mon[115078]: pgmap v546: 521 pgs: 521 active+clean; 67 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 1.6 MiB/s wr, 40 op/s 2024-08-17T17:23:25.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:25 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:25.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:25 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-08-17T17:23:25.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:25 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:23:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:25 smithi165 ceph-mon[105517]: pgmap v546: 521 pgs: 521 active+clean; 67 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 1.6 MiB/s wr, 40 op/s 2024-08-17T17:23:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:25 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:25 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-08-17T17:23:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:25 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:23:26.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:26 smithi136 ceph-mon[109861]: Upgrade: Updating osd.2 2024-08-17T17:23:26.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:26 smithi136 ceph-mon[109861]: Deploying daemon osd.2 on smithi136 2024-08-17T17:23:26.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:26 smithi136 ceph-mon[115078]: Upgrade: Updating osd.2 2024-08-17T17:23:26.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:26 smithi136 ceph-mon[115078]: Deploying daemon osd.2 on smithi136 2024-08-17T17:23:26.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:26 smithi165 ceph-mon[105517]: Upgrade: Updating osd.2 2024-08-17T17:23:26.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:26 smithi165 ceph-mon[105517]: Deploying daemon osd.2 on smithi136 2024-08-17T17:23:27.254 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:27 smithi136 ceph-mon[109861]: pgmap v547: 521 pgs: 521 active+clean; 71 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 395 KiB/s rd, 1.7 MiB/s wr, 34 op/s 2024-08-17T17:23:27.255 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:27 smithi136 ceph-mon[115078]: pgmap v547: 521 pgs: 521 active+clean; 71 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 395 KiB/s rd, 1.7 MiB/s wr, 34 op/s 2024-08-17T17:23:27.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:27 smithi165 ceph-mon[105517]: pgmap v547: 521 pgs: 521 active+clean; 71 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 395 KiB/s rd, 1.7 MiB/s wr, 34 op/s 2024-08-17T17:23:27.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:23:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:23:27] "GET /metrics HTTP/1.1" 200 67785 "" "Prometheus/2.51.0" 2024-08-17T17:23:28.432 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:28 smithi136 systemd[1]: Stopping Ceph osd.2 for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:23:29.171 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:28 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2[59404]: 2024-08-17T17:23:28.752+0000 7f5b34f9e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:23:29.172 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:28 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2[59404]: 2024-08-17T17:23:28.752+0000 7f5b34f9e640 -1 osd.2 669 *** Got signal Terminated *** 2024-08-17T17:23:29.172 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:28 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2[59404]: 2024-08-17T17:23:28.752+0000 7f5b34f9e640 -1 osd.2 669 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T17:23:29.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:29 smithi136 ceph-mon[109861]: pgmap v548: 521 pgs: 521 active+clean; 71 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 29 op/s 2024-08-17T17:23:29.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:29 smithi136 ceph-mon[109861]: osd.2 marked itself down and dead 2024-08-17T17:23:29.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:29 smithi136 ceph-mon[115078]: pgmap v548: 521 pgs: 521 active+clean; 71 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 29 op/s 2024-08-17T17:23:29.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:29 smithi136 ceph-mon[115078]: osd.2 marked itself down and dead 2024-08-17T17:23:29.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:29 smithi165 ceph-mon[105517]: pgmap v548: 521 pgs: 521 active+clean; 71 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 29 op/s 2024-08-17T17:23:29.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:29 smithi165 ceph-mon[105517]: osd.2 marked itself down and dead 2024-08-17T17:23:30.067 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:29 smithi136 podman[141666]: 2024-08-17 17:23:29.780297672 +0000 UTC m=+1.156783908 container died 640be4af415619c7e508a17afc8c7adf365d5118089da6f13de92cb31a2813f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2, RELEASE=reef-ced649f, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731) 2024-08-17T17:23:30.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:30 smithi136 ceph-mon[109861]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:23:30.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:30 smithi136 ceph-mon[109861]: osdmap e670: 8 total, 7 up, 8 in 2024-08-17T17:23:30.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:30 smithi136 ceph-mon[115078]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:23:30.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:30 smithi136 ceph-mon[115078]: osdmap e670: 8 total, 7 up, 8 in 2024-08-17T17:23:30.447 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:30 smithi136 podman[141666]: 2024-08-17 17:23:30.162422763 +0000 UTC m=+1.538908998 container cleanup 640be4af415619c7e508a17afc8c7adf365d5118089da6f13de92cb31a2813f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=reef-ced649f, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-17T17:23:30.447 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:30 smithi136 bash[141666]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2 2024-08-17T17:23:30.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:30 smithi165 ceph-mon[105517]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:23:30.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:30 smithi165 ceph-mon[105517]: osdmap e670: 8 total, 7 up, 8 in 2024-08-17T17:23:30.924 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:30 smithi136 podman[141770]: 2024-08-17 17:23:30.447476597 +0000 UTC m=+0.666272522 container remove 640be4af415619c7e508a17afc8c7adf365d5118089da6f13de92cb31a2813f2 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=reef-ced649f, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-08-17T17:23:31.196 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:30 smithi136 podman[141903]: 2024-08-17 17:23:30.826244801 +0000 UTC m=+0.017064818 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:23:31.196 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:30 smithi136 podman[141903]: 2024-08-17 17:23:30.93183066 +0000 UTC m=+0.122650654 container create 9216f5fcff4e05507af60ba1f8a78c3e938da0a1b9b50b4e06278d0889e15072 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, ceph=True, RELEASE=main-cd8bdbe, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-08-17T17:23:31.196 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:31 smithi136 podman[141903]: 2024-08-17 17:23:31.151556621 +0000 UTC m=+0.342376634 container init 9216f5fcff4e05507af60ba1f8a78c3e938da0a1b9b50b4e06278d0889e15072 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, RELEASE=main-cd8bdbe) 2024-08-17T17:23:31.197 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:31 smithi136 podman[141903]: 2024-08-17 17:23:31.156503313 +0000 UTC m=+0.347323303 container start 9216f5fcff4e05507af60ba1f8a78c3e938da0a1b9b50b4e06278d0889e15072 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T17:23:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:31 smithi165 ceph-mon[105517]: pgmap v550: 521 pgs: 53 stale+active+clean, 468 active+clean; 71 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2024-08-17T17:23:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:31 smithi165 ceph-mon[105517]: osdmap e671: 8 total, 7 up, 8 in 2024-08-17T17:23:31.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:31 smithi136 ceph-mon[109861]: pgmap v550: 521 pgs: 53 stale+active+clean, 468 active+clean; 71 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2024-08-17T17:23:31.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:31 smithi136 ceph-mon[109861]: osdmap e671: 8 total, 7 up, 8 in 2024-08-17T17:23:31.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:31 smithi136 ceph-mon[115078]: pgmap v550: 521 pgs: 53 stale+active+clean, 468 active+clean; 71 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 4 op/s 2024-08-17T17:23:31.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:31 smithi136 ceph-mon[115078]: osdmap e671: 8 total, 7 up, 8 in 2024-08-17T17:23:31.697 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:31 smithi136 podman[141903]: 2024-08-17 17:23:31.243217553 +0000 UTC m=+0.434037542 container attach 9216f5fcff4e05507af60ba1f8a78c3e938da0a1b9b50b4e06278d0889e15072 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=main-cd8bdbe, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-08-17T17:23:31.697 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:31 smithi136 podman[141903]: 2024-08-17 17:23:31.344041796 +0000 UTC m=+0.534861791 container died 9216f5fcff4e05507af60ba1f8a78c3e938da0a1b9b50b4e06278d0889e15072 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, ceph=True, RELEASE=main-cd8bdbe, CEPH_POINT_RELEASE=) 2024-08-17T17:23:32.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:32 smithi136 ceph-mon[109861]: Health check failed: Degraded data redundancy: 118/750 objects degraded (15.733%), 59 pgs degraded (PG_DEGRADED) 2024-08-17T17:23:32.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:32 smithi136 ceph-mon[115078]: Health check failed: Degraded data redundancy: 118/750 objects degraded (15.733%), 59 pgs degraded (PG_DEGRADED) 2024-08-17T17:23:32.447 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:32 smithi136 podman[141945]: 2024-08-17 17:23:32.130868318 +0000 UTC m=+0.769409368 container remove 9216f5fcff4e05507af60ba1f8a78c3e938da0a1b9b50b4e06278d0889e15072 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-cd8bdbe, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-08-17T17:23:32.447 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:32 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.2.service: Deactivated successfully. 2024-08-17T17:23:32.447 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:32 smithi136 systemd[1]: Stopped Ceph osd.2 for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:23:32.447 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:32 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.2.service: Consumed 31.462s CPU time. 2024-08-17T17:23:32.447 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:32 smithi136 systemd[1]: Starting Ceph osd.2 for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:23:32.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:32 smithi165 ceph-mon[105517]: Health check failed: Degraded data redundancy: 118/750 objects degraded (15.733%), 59 pgs degraded (PG_DEGRADED) 2024-08-17T17:23:32.902 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:32 smithi136 podman[142049]: 2024-08-17 17:23:32.475752711 +0000 UTC m=+0.018385088 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:23:32.902 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:32 smithi136 podman[142049]: 2024-08-17 17:23:32.599144509 +0000 UTC m=+0.141776876 container create 2afbef18fd4429dcfd95b5ab53ca13b7cf2e4508f0c9264a0a1cc6fb22b90bfb (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-cd8bdbe, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:23:32.902 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:32 smithi136 podman[142049]: 2024-08-17 17:23:32.810379159 +0000 UTC m=+0.353011530 container init 2afbef18fd4429dcfd95b5ab53ca13b7cf2e4508f0c9264a0a1cc6fb22b90bfb (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0) 2024-08-17T17:23:32.902 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:32 smithi136 podman[142049]: 2024-08-17 17:23:32.815223679 +0000 UTC m=+0.357856035 container start 2afbef18fd4429dcfd95b5ab53ca13b7cf2e4508f0c9264a0a1cc6fb22b90bfb (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2) 2024-08-17T17:23:32.902 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:32 smithi136 podman[142049]: 2024-08-17 17:23:32.901882928 +0000 UTC m=+0.444515288 container attach 2afbef18fd4429dcfd95b5ab53ca13b7cf2e4508f0c9264a0a1cc6fb22b90bfb (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=main-cd8bdbe, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-08-17T17:23:33.196 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:32 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate[142064]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:23:33.197 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:32 smithi136 bash[142049]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:23:33.197 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:32 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate[142064]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:23:33.197 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:32 smithi136 bash[142049]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:23:33.197 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:32 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate[142064]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-17T17:23:33.197 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:32 smithi136 bash[142049]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-17T17:23:33.197 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:32 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate[142064]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:23:33.197 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:32 smithi136 bash[142049]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:23:33.197 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:32 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate[142064]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:23:33.197 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:32 smithi136 bash[142049]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:23:33.197 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:33 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate[142064]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-17T17:23:33.197 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:33 smithi136 bash[142049]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-17T17:23:33.197 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:33 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate[142064]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2024-08-17T17:23:33.198 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:33 smithi136 bash[142049]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2024-08-17T17:23:33.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:33 smithi165 ceph-mon[105517]: pgmap v552: 521 pgs: 142 active+undersized, 59 active+undersized+degraded, 320 active+clean; 71 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 512 KiB/s wr, 0 op/s; 118/750 objects degraded (15.733%) 2024-08-17T17:23:33.674 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:33 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate[142064]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2024-08-17T17:23:33.675 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:33 smithi136 bash[142049]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2024-08-17T17:23:33.675 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:33 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate[142064]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2024-08-17T17:23:33.675 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:33 smithi136 bash[142049]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2024-08-17T17:23:33.675 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:33 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate[142064]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-08-17T17:23:33.675 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:33 smithi136 bash[142049]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-08-17T17:23:33.675 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:33 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate[142064]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-17T17:23:33.675 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:33 smithi136 bash[142049]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-17T17:23:33.675 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:33 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate[142064]: --> ceph-volume lvm activate successful for osd ID: 2 2024-08-17T17:23:33.675 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:33 smithi136 bash[142049]: --> ceph-volume lvm activate successful for osd ID: 2 2024-08-17T17:23:33.675 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:33 smithi136 podman[142049]: 2024-08-17 17:23:33.37105985 +0000 UTC m=+0.913692221 container died 2afbef18fd4429dcfd95b5ab53ca13b7cf2e4508f0c9264a0a1cc6fb22b90bfb (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-08-17T17:23:33.676 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:33 smithi136 ceph-mon[109861]: pgmap v552: 521 pgs: 142 active+undersized, 59 active+undersized+degraded, 320 active+clean; 71 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 512 KiB/s wr, 0 op/s; 118/750 objects degraded (15.733%) 2024-08-17T17:23:33.676 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:33 smithi136 ceph-mon[115078]: pgmap v552: 521 pgs: 142 active+undersized, 59 active+undersized+degraded, 320 active+clean; 71 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 512 KiB/s wr, 0 op/s; 118/750 objects degraded (15.733%) 2024-08-17T17:23:34.338 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:34 smithi136 ceph-mon[109861]: osdmap e672: 8 total, 7 up, 8 in 2024-08-17T17:23:34.339 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:34 smithi136 podman[142209]: 2024-08-17 17:23:34.0636222 +0000 UTC m=+0.679747522 container remove 2afbef18fd4429dcfd95b5ab53ca13b7cf2e4508f0c9264a0a1cc6fb22b90bfb (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe) 2024-08-17T17:23:34.339 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:34 smithi136 ceph-mon[115078]: osdmap e672: 8 total, 7 up, 8 in 2024-08-17T17:23:34.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:34 smithi165 ceph-mon[105517]: osdmap e672: 8 total, 7 up, 8 in 2024-08-17T17:23:34.697 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:34 smithi136 podman[142331]: 2024-08-17 17:23:34.242886835 +0000 UTC m=+0.029667622 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:23:34.697 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:34 smithi136 podman[142331]: 2024-08-17 17:23:34.366111513 +0000 UTC m=+0.152892276 container create 933eab0a702a51e3fa99de6e373a49e883425ed23ba22ee45d4aace6e1a3eda9 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=main-cd8bdbe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, ceph=True, org.label-schema.schema-version=1.0) 2024-08-17T17:23:35.197 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:34 smithi136 podman[142331]: 2024-08-17 17:23:34.728188031 +0000 UTC m=+0.514968803 container init 933eab0a702a51e3fa99de6e373a49e883425ed23ba22ee45d4aace6e1a3eda9 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2, RELEASE=main-cd8bdbe, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2) 2024-08-17T17:23:35.197 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:34 smithi136 podman[142331]: 2024-08-17 17:23:34.73801739 +0000 UTC m=+0.524798154 container start 933eab0a702a51e3fa99de6e373a49e883425ed23ba22ee45d4aace6e1a3eda9 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2, org.label-schema.vendor=CentOS, RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-08-17T17:23:35.197 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:34 smithi136 ceph-osd[142410]: -- 172.21.15.136:0/4012342005 <== mon.0 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55721b0dfa40 con 0x55721beac000 2024-08-17T17:23:35.197 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:34 smithi136 bash[142331]: 933eab0a702a51e3fa99de6e373a49e883425ed23ba22ee45d4aace6e1a3eda9 2024-08-17T17:23:35.197 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:34 smithi136 systemd[1]: Started Ceph osd.2 for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:23:35.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:35 smithi165 ceph-mon[105517]: pgmap v554: 521 pgs: 142 active+undersized, 59 active+undersized+degraded, 320 active+clean; 71 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 118/750 objects degraded (15.733%) 2024-08-17T17:23:35.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:35 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:35.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:35 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:35.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:35 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:35.626 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:35 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:23:35.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:35 smithi136 ceph-mon[109861]: pgmap v554: 521 pgs: 142 active+undersized, 59 active+undersized+degraded, 320 active+clean; 71 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 118/750 objects degraded (15.733%) 2024-08-17T17:23:35.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:35 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:35.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:35 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:35.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:35 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:35.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:35 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:23:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:35 smithi136 ceph-mon[115078]: pgmap v554: 521 pgs: 142 active+undersized, 59 active+undersized+degraded, 320 active+clean; 71 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 118/750 objects degraded (15.733%) 2024-08-17T17:23:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:35 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:35 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:35 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:35 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:23:36.696 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:36 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2[142367]: 2024-08-17T17:23:36.393+0000 7f202b11b640 -1 Falling back to public interface 2024-08-17T17:23:37.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:37 smithi136 ceph-mon[109861]: pgmap v555: 521 pgs: 142 active+undersized, 59 active+undersized+degraded, 320 active+clean; 71 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 1.0 KiB/s wr, 20 op/s; 120/759 objects degraded (15.810%) 2024-08-17T17:23:37.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:37.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:23:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:37 smithi136 ceph-mon[115078]: pgmap v555: 521 pgs: 142 active+undersized, 59 active+undersized+degraded, 320 active+clean; 71 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 1.0 KiB/s wr, 20 op/s; 120/759 objects degraded (15.810%) 2024-08-17T17:23:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:23:37.503 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:37 smithi165 ceph-mon[105517]: pgmap v555: 521 pgs: 142 active+undersized, 59 active+undersized+degraded, 320 active+clean; 71 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 1.0 KiB/s wr, 20 op/s; 120/759 objects degraded (15.810%) 2024-08-17T17:23:37.503 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:37.503 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:23:37.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:23:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:23:37] "GET /metrics HTTP/1.1" 200 67794 "" "Prometheus/2.51.0" 2024-08-17T17:23:38.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:38 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 121/762 objects degraded (15.879%), 59 pgs degraded (PG_DEGRADED) 2024-08-17T17:23:38.543 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:38 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 121/762 objects degraded (15.879%), 59 pgs degraded (PG_DEGRADED) 2024-08-17T17:23:38.543 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:38 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 121/762 objects degraded (15.879%), 59 pgs degraded (PG_DEGRADED) 2024-08-17T17:23:39.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:39 smithi165 ceph-mon[105517]: pgmap v556: 521 pgs: 142 active+undersized, 59 active+undersized+degraded, 320 active+clean; 71 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 1.4 KiB/s wr, 46 op/s; 121/762 objects degraded (15.879%) 2024-08-17T17:23:39.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:39 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:39.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:39 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:39.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:39 smithi136 ceph-mon[115078]: pgmap v556: 521 pgs: 142 active+undersized, 59 active+undersized+degraded, 320 active+clean; 71 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 1.4 KiB/s wr, 46 op/s; 121/762 objects degraded (15.879%) 2024-08-17T17:23:39.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:39 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:39.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:39 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:39.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:39 smithi136 ceph-mon[109861]: pgmap v556: 521 pgs: 142 active+undersized, 59 active+undersized+degraded, 320 active+clean; 71 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 1.4 KiB/s wr, 46 op/s; 121/762 objects degraded (15.879%) 2024-08-17T17:23:39.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:39 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:39.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:39 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:40.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:40 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:40.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:40 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:40.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:40 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:40.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:40 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:40.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:40 smithi136 ceph-mon[109861]: from='osd.2 [v2:172.21.15.136:6818/762368443,v1:172.21.15.136:6819/762368443]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-17T17:23:40.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:40 smithi136 ceph-mon[109861]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-17T17:23:40.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:40 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:40.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:40 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:40.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:40 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:40.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:40 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:40.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:40 smithi136 ceph-mon[115078]: from='osd.2 [v2:172.21.15.136:6818/762368443,v1:172.21.15.136:6819/762368443]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-17T17:23:40.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:40 smithi136 ceph-mon[115078]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-17T17:23:40.448 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:40 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2[142367]: 2024-08-17T17:23:40.180+0000 7f202b11b640 -1 osd.2 670 log_to_monitors true 2024-08-17T17:23:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:40 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:40 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:40 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:40 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:40 smithi165 ceph-mon[105517]: from='osd.2 [v2:172.21.15.136:6818/762368443,v1:172.21.15.136:6819/762368443]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-17T17:23:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:40 smithi165 ceph-mon[105517]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-17T17:23:41.196 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 17:23:40 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2[142367]: 2024-08-17T17:23:40.888+0000 7f201eebd640 -1 osd.2 670 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:23:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:41 smithi136 ceph-mon[109861]: pgmap v557: 521 pgs: 142 active+undersized, 59 active+undersized+degraded, 320 active+clean; 71 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 1.1 KiB/s wr, 38 op/s; 121/762 objects degraded (15.879%) 2024-08-17T17:23:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:41 smithi136 ceph-mon[109861]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-17T17:23:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:41 smithi136 ceph-mon[109861]: osdmap e673: 8 total, 7 up, 8 in 2024-08-17T17:23:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:41 smithi136 ceph-mon[109861]: from='osd.2 [v2:172.21.15.136:6818/762368443,v1:172.21.15.136:6819/762368443]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-17T17:23:41.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:41 smithi136 ceph-mon[109861]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-17T17:23:41.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:41 smithi136 ceph-mon[115078]: pgmap v557: 521 pgs: 142 active+undersized, 59 active+undersized+degraded, 320 active+clean; 71 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 1.1 KiB/s wr, 38 op/s; 121/762 objects degraded (15.879%) 2024-08-17T17:23:41.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:41 smithi136 ceph-mon[115078]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-17T17:23:41.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:41 smithi136 ceph-mon[115078]: osdmap e673: 8 total, 7 up, 8 in 2024-08-17T17:23:41.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:41 smithi136 ceph-mon[115078]: from='osd.2 [v2:172.21.15.136:6818/762368443,v1:172.21.15.136:6819/762368443]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-17T17:23:41.576 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:41 smithi136 ceph-mon[115078]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-17T17:23:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:41 smithi165 ceph-mon[105517]: pgmap v557: 521 pgs: 142 active+undersized, 59 active+undersized+degraded, 320 active+clean; 71 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 1.1 KiB/s wr, 38 op/s; 121/762 objects degraded (15.879%) 2024-08-17T17:23:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:41 smithi165 ceph-mon[105517]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-17T17:23:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:41 smithi165 ceph-mon[105517]: osdmap e673: 8 total, 7 up, 8 in 2024-08-17T17:23:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:41 smithi165 ceph-mon[105517]: from='osd.2 [v2:172.21.15.136:6818/762368443,v1:172.21.15.136:6819/762368443]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-17T17:23:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:41 smithi165 ceph-mon[105517]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-17T17:23:42.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:42 smithi136 ceph-mon[109861]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:23:42.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:42 smithi136 ceph-mon[109861]: osd.2 [v2:172.21.15.136:6818/762368443,v1:172.21.15.136:6819/762368443] boot 2024-08-17T17:23:42.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:42 smithi136 ceph-mon[109861]: osdmap e674: 8 total, 8 up, 8 in 2024-08-17T17:23:42.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:42 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-17T17:23:42.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:42 smithi136 ceph-mon[115078]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:23:42.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:42 smithi136 ceph-mon[115078]: osd.2 [v2:172.21.15.136:6818/762368443,v1:172.21.15.136:6819/762368443] boot 2024-08-17T17:23:42.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:42 smithi136 ceph-mon[115078]: osdmap e674: 8 total, 8 up, 8 in 2024-08-17T17:23:42.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:42 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-17T17:23:42.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:42 smithi165 ceph-mon[105517]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:23:42.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:42 smithi165 ceph-mon[105517]: osd.2 [v2:172.21.15.136:6818/762368443,v1:172.21.15.136:6819/762368443] boot 2024-08-17T17:23:42.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:42 smithi165 ceph-mon[105517]: osdmap e674: 8 total, 8 up, 8 in 2024-08-17T17:23:42.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:42 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-17T17:23:43.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:43 smithi136 ceph-mon[109861]: pgmap v560: 521 pgs: 100 peering, 73 active+undersized, 28 active+undersized+degraded, 320 active+clean; 199 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 187 KiB/s rd, 16 MiB/s wr, 65 op/s; 57/858 objects degraded (6.643%) 2024-08-17T17:23:43.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:43 smithi136 ceph-mon[109861]: osdmap e675: 8 total, 8 up, 8 in 2024-08-17T17:23:43.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:43 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:43.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:43 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:43.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:43 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:23:43.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:43 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:23:43.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:43 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:43.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:43 smithi136 ceph-mon[115078]: pgmap v560: 521 pgs: 100 peering, 73 active+undersized, 28 active+undersized+degraded, 320 active+clean; 199 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 187 KiB/s rd, 16 MiB/s wr, 65 op/s; 57/858 objects degraded (6.643%) 2024-08-17T17:23:43.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:43 smithi136 ceph-mon[115078]: osdmap e675: 8 total, 8 up, 8 in 2024-08-17T17:23:43.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:43 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:43.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:43 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:43.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:43 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:23:43.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:43 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:23:43.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:43 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:43.448 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:23:43 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T17:23:43.295+0000 7f0cc2ea7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (65 PGs are or would become offline) 2024-08-17T17:23:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:43 smithi165 ceph-mon[105517]: pgmap v560: 521 pgs: 100 peering, 73 active+undersized, 28 active+undersized+degraded, 320 active+clean; 199 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 187 KiB/s rd, 16 MiB/s wr, 65 op/s; 57/858 objects degraded (6.643%) 2024-08-17T17:23:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:43 smithi165 ceph-mon[105517]: osdmap e675: 8 total, 8 up, 8 in 2024-08-17T17:23:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:43 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:43 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:43 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:23:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:43 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:23:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:43 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:44.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:44 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:23:44.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:44 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:23:44.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:44 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:23:44.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:44 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:23:44.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:44 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-08-17T17:23:44.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:44 smithi136 ceph-mon[109861]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-08-17T17:23:44.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:44 smithi136 ceph-mon[109861]: Upgrade: unsafe to stop osd(s) at this time (65 PGs are or would become offline) 2024-08-17T17:23:44.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:44 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 57/858 objects degraded (6.643%), 28 pgs degraded (PG_DEGRADED) 2024-08-17T17:23:44.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:44 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:23:44.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:44 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:23:44.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:44 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:23:44.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:44 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:23:44.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:44 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-08-17T17:23:44.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:44 smithi136 ceph-mon[115078]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-08-17T17:23:44.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:44 smithi136 ceph-mon[115078]: Upgrade: unsafe to stop osd(s) at this time (65 PGs are or would become offline) 2024-08-17T17:23:44.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:44 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 57/858 objects degraded (6.643%), 28 pgs degraded (PG_DEGRADED) 2024-08-17T17:23:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:44 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:23:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:44 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:23:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:44 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:23:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:44 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:23:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:44 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-08-17T17:23:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:44 smithi165 ceph-mon[105517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-08-17T17:23:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:44 smithi165 ceph-mon[105517]: Upgrade: unsafe to stop osd(s) at this time (65 PGs are or would become offline) 2024-08-17T17:23:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:44 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 57/858 objects degraded (6.643%), 28 pgs degraded (PG_DEGRADED) 2024-08-17T17:23:45.652 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:45 smithi136 ceph-mon[109861]: pgmap v562: 521 pgs: 100 peering, 73 active+undersized, 28 active+undersized+degraded, 320 active+clean; 199 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 21 MiB/s wr, 16 op/s; 57/858 objects degraded (6.643%) 2024-08-17T17:23:45.652 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:45 smithi136 ceph-mon[115078]: pgmap v562: 521 pgs: 100 peering, 73 active+undersized, 28 active+undersized+degraded, 320 active+clean; 199 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 21 MiB/s wr, 16 op/s; 57/858 objects degraded (6.643%) 2024-08-17T17:23:45.653 INFO:teuthology.orchestra.run.smithi136.stdout:true 2024-08-17T17:23:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:45 smithi165 ceph-mon[105517]: pgmap v562: 521 pgs: 100 peering, 73 active+undersized, 28 active+undersized+degraded, 320 active+clean; 199 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 21 MiB/s wr, 16 op/s; 57/858 objects degraded (6.643%) 2024-08-17T17:23:46.207 INFO:teuthology.orchestra.run.smithi136.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-17T17:23:46.207 INFO:teuthology.orchestra.run.smithi136.stdout:alertmanager.a smithi136 *:9093,9094 running (8m) 7s ago 14m 19.2M - 0.27.0 11f11916f8cd c3b9b5b223ab 2024-08-17T17:23:46.208 INFO:teuthology.orchestra.run.smithi136.stdout:grafana.a smithi165 *:3000 running (8m) 6s ago 14m 77.6M - c8b91775d855 04db24fce5e9 2024-08-17T17:23:46.208 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi136.ltsuqd smithi136 running (13m) 7s ago 13m 24.7M - 18.2.4-719-gced649f8 c870753d67c2 faaa06f2950c 2024-08-17T17:23:46.208 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi165.bmrbhh smithi165 running (13m) 6s ago 13m 18.8M - 18.2.4-719-gced649f8 c870753d67c2 c74b00d929cf 2024-08-17T17:23:46.208 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.x smithi165 *:8443,9283,8765 running (10m) 6s ago 18m 458M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 837fd48ff7cb 2024-08-17T17:23:46.208 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.y smithi136 *:8443,9283,8765 running (10m) 7s ago 20m 535M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9acdc6775e48 2024-08-17T17:23:46.208 INFO:teuthology.orchestra.run.smithi136.stdout:mon.a smithi136 running (7m) 7s ago 20m 76.3M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9cf82e613626 2024-08-17T17:23:46.208 INFO:teuthology.orchestra.run.smithi136.stdout:mon.b smithi165 running (5m) 6s ago 19m 55.6M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 1fd357308b8d 2024-08-17T17:23:46.208 INFO:teuthology.orchestra.run.smithi136.stdout:mon.c smithi136 running (7m) 7s ago 18m 67.5M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 2c2ac98ad9d4 2024-08-17T17:23:46.208 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.a smithi136 *:9100 running (9m) 7s ago 14m 11.2M - 1.7.0 72c9c2088986 1e4c644ae533 2024-08-17T17:23:46.208 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.b smithi165 *:9100 running (9m) 6s ago 14m 11.6M - 1.7.0 72c9c2088986 7af2d2d68b06 2024-08-17T17:23:46.208 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0 smithi136 running (81s) 7s ago 18m 59.9M 2480M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 71b7314ae4a5 2024-08-17T17:23:46.208 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1 smithi136 running (47s) 7s ago 17m 64.4M 2480M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb a0bc94c66763 2024-08-17T17:23:46.208 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2 smithi136 running (11s) 7s ago 17m 11.4M 2480M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 933eab0a702a 2024-08-17T17:23:46.208 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3 smithi136 running (17m) 7s ago 17m 141M 2480M 18.2.4-719-gced649f8 c870753d67c2 14b57a745d30 2024-08-17T17:23:46.209 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4 smithi165 running (16m) 6s ago 16m 134M 2480M 18.2.4-719-gced649f8 c870753d67c2 df366c73e1f2 2024-08-17T17:23:46.209 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5 smithi165 running (16m) 6s ago 16m 115M 2480M 18.2.4-719-gced649f8 c870753d67c2 1131665e203a 2024-08-17T17:23:46.209 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6 smithi165 running (16m) 6s ago 15m 149M 2480M 18.2.4-719-gced649f8 c870753d67c2 7bc5fd925293 2024-08-17T17:23:46.209 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7 smithi165 running (15m) 6s ago 15m 127M 2480M 18.2.4-719-gced649f8 c870753d67c2 18382e4ae3b8 2024-08-17T17:23:46.209 INFO:teuthology.orchestra.run.smithi136.stdout:prometheus.a smithi165 *:9095 running (9m) 6s ago 15m 48.0M - 2.51.0 1d3b7f56885b ae8f8b69fe2f 2024-08-17T17:23:46.576 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:23:46.576 INFO:teuthology.orchestra.run.smithi136.stdout: "mon": { 2024-08-17T17:23:46.576 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 3 2024-08-17T17:23:46.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:23:46.576 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr": { 2024-08-17T17:23:46.576 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 2 2024-08-17T17:23:46.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:23:46.576 INFO:teuthology.orchestra.run.smithi136.stdout: "osd": { 2024-08-17T17:23:46.576 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 5, 2024-08-17T17:23:46.577 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 3 2024-08-17T17:23:46.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:23:46.577 INFO:teuthology.orchestra.run.smithi136.stdout: "mds": { 2024-08-17T17:23:46.577 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 2 2024-08-17T17:23:46.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:23:46.577 INFO:teuthology.orchestra.run.smithi136.stdout: "overall": { 2024-08-17T17:23:46.577 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 7, 2024-08-17T17:23:46.577 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 8 2024-08-17T17:23:46.577 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:23:46.577 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:23:46.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:46 smithi136 ceph-mon[109861]: from='client.44495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:23:46.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:46 smithi136 ceph-mon[109861]: from='client.44501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:23:46.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:46 smithi136 ceph-mon[109861]: pgmap v563: 521 pgs: 100 peering, 421 active+clean; 199 MiB data, 945 MiB used, 714 GiB / 715 GiB avail; 227 KiB/s rd, 33 MiB/s wr, 152 op/s; 3.3 MiB/s, 0 keys/s, 1 objects/s recovering 2024-08-17T17:23:46.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:46 smithi136 ceph-mon[109861]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:23:46.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:46 smithi136 ceph-mon[109861]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/858 objects degraded (6.643%), 28 pgs degraded) 2024-08-17T17:23:46.578 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:46 smithi136 ceph-mon[109861]: osdmap e676: 8 total, 8 up, 8 in 2024-08-17T17:23:46.578 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:46 smithi136 ceph-mon[115078]: from='client.44495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:23:46.578 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:46 smithi136 ceph-mon[115078]: from='client.44501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:23:46.578 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:46 smithi136 ceph-mon[115078]: pgmap v563: 521 pgs: 100 peering, 421 active+clean; 199 MiB data, 945 MiB used, 714 GiB / 715 GiB avail; 227 KiB/s rd, 33 MiB/s wr, 152 op/s; 3.3 MiB/s, 0 keys/s, 1 objects/s recovering 2024-08-17T17:23:46.579 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:46 smithi136 ceph-mon[115078]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:23:46.579 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:46 smithi136 ceph-mon[115078]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/858 objects degraded (6.643%), 28 pgs degraded) 2024-08-17T17:23:46.579 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:46 smithi136 ceph-mon[115078]: osdmap e676: 8 total, 8 up, 8 in 2024-08-17T17:23:46.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:46 smithi165 ceph-mon[105517]: from='client.44495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:23:46.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:46 smithi165 ceph-mon[105517]: from='client.44501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:23:46.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:46 smithi165 ceph-mon[105517]: pgmap v563: 521 pgs: 100 peering, 421 active+clean; 199 MiB data, 945 MiB used, 714 GiB / 715 GiB avail; 227 KiB/s rd, 33 MiB/s wr, 152 op/s; 3.3 MiB/s, 0 keys/s, 1 objects/s recovering 2024-08-17T17:23:46.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:46 smithi165 ceph-mon[105517]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:23:46.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:46 smithi165 ceph-mon[105517]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/858 objects degraded (6.643%), 28 pgs degraded) 2024-08-17T17:23:46.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:46 smithi165 ceph-mon[105517]: osdmap e676: 8 total, 8 up, 8 in 2024-08-17T17:23:46.859 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:23:46.859 INFO:teuthology.orchestra.run.smithi136.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d", 2024-08-17T17:23:46.859 INFO:teuthology.orchestra.run.smithi136.stdout: "in_progress": true, 2024-08-17T17:23:46.859 INFO:teuthology.orchestra.run.smithi136.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 4 daemons (1 remaining).", 2024-08-17T17:23:46.859 INFO:teuthology.orchestra.run.smithi136.stdout: "services_complete": [], 2024-08-17T17:23:46.859 INFO:teuthology.orchestra.run.smithi136.stdout: "progress": "3/8 daemons upgraded", 2024-08-17T17:23:46.859 INFO:teuthology.orchestra.run.smithi136.stdout: "message": "Currently upgrading osd daemons", 2024-08-17T17:23:46.859 INFO:teuthology.orchestra.run.smithi136.stdout: "is_paused": false 2024-08-17T17:23:46.860 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:23:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:47 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3883248098' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:23:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:47 smithi136 ceph-mon[109861]: from='client.34608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:23:47.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:23:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:23:47] "GET /metrics HTTP/1.1" 200 67794 "" "Prometheus/2.51.0" 2024-08-17T17:23:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:47 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3883248098' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:23:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:47 smithi136 ceph-mon[115078]: from='client.34608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:23:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:47 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3883248098' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:23:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:47 smithi165 ceph-mon[105517]: from='client.34608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:23:48.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:48 smithi136 ceph-mon[115078]: osdmap e677: 8 total, 8 up, 8 in 2024-08-17T17:23:48.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:48 smithi136 ceph-mon[115078]: pgmap v566: 521 pgs: 521 active+clean; 59 MiB data, 881 MiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 21 MiB/s wr, 176 op/s; 15 MiB/s, 0 keys/s, 4 objects/s recovering 2024-08-17T17:23:48.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:48 smithi136 ceph-mon[109861]: osdmap e677: 8 total, 8 up, 8 in 2024-08-17T17:23:48.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:48 smithi136 ceph-mon[109861]: pgmap v566: 521 pgs: 521 active+clean; 59 MiB data, 881 MiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 21 MiB/s wr, 176 op/s; 15 MiB/s, 0 keys/s, 4 objects/s recovering 2024-08-17T17:23:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:48 smithi165 ceph-mon[105517]: osdmap e677: 8 total, 8 up, 8 in 2024-08-17T17:23:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:48 smithi165 ceph-mon[105517]: pgmap v566: 521 pgs: 521 active+clean; 59 MiB data, 881 MiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 21 MiB/s wr, 176 op/s; 15 MiB/s, 0 keys/s, 4 objects/s recovering 2024-08-17T17:23:49.448 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 4 2024-08-17T17:23:49.448 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 4 2024-08-17T17:23:49.448 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 4 2024-08-17T17:23:49.448 INFO:tasks.workunit.client.0.smithi136.stdout:read: 4 2024-08-17T17:23:49.448 INFO:tasks.workunit.client.0.smithi136.stdout:read: 4 2024-08-17T17:23:49.448 INFO:tasks.workunit.client.0.smithi136.stdout:read: 4 2024-08-17T17:23:49.448 INFO:tasks.workunit.client.0.smithi136.stdout:read: 4 2024-08-17T17:23:49.448 INFO:tasks.workunit.client.0.smithi136.stdout:read: 4 2024-08-17T17:23:49.448 INFO:tasks.workunit.client.0.smithi136.stdout:read: 4 2024-08-17T17:23:49.448 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestCloneEncryption (51976 ms) 2024-08-17T17:23:49.449 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.LUKS1UnderLUKS2WithoutResize 2024-08-17T17:23:49.449 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:23:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:49 smithi136 ceph-mon[109861]: osdmap e678: 8 total, 8 up, 8 in 2024-08-17T17:23:49.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:49 smithi136 ceph-mon[115078]: osdmap e678: 8 total, 8 up, 8 in 2024-08-17T17:23:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:49 smithi165 ceph-mon[105517]: osdmap e678: 8 total, 8 up, 8 in 2024-08-17T17:23:50.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:50 smithi136 ceph-mon[109861]: osdmap e679: 8 total, 8 up, 8 in 2024-08-17T17:23:50.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:50 smithi136 ceph-mon[109861]: pgmap v569: 521 pgs: 521 active+clean; 59 MiB data, 881 MiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 14 MiB/s wr, 60 op/s; 17 MiB/s, 4 objects/s recovering 2024-08-17T17:23:50.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:50 smithi136 ceph-mon[115078]: osdmap e679: 8 total, 8 up, 8 in 2024-08-17T17:23:50.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:50 smithi136 ceph-mon[115078]: pgmap v569: 521 pgs: 521 active+clean; 59 MiB data, 881 MiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 14 MiB/s wr, 60 op/s; 17 MiB/s, 4 objects/s recovering 2024-08-17T17:23:50.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:50 smithi165 ceph-mon[105517]: osdmap e679: 8 total, 8 up, 8 in 2024-08-17T17:23:50.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:50 smithi165 ceph-mon[105517]: pgmap v569: 521 pgs: 521 active+clean; 59 MiB data, 881 MiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 14 MiB/s wr, 60 op/s; 17 MiB/s, 4 objects/s recovering 2024-08-17T17:23:52.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:52.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:23:52.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:52.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:23:52.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:23:53.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:53 smithi136 ceph-mon[109861]: pgmap v570: 521 pgs: 521 active+clean; 31 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 1.5 MiB/s rd, 9.9 MiB/s wr, 179 op/s; 12 MiB/s, 3 objects/s recovering 2024-08-17T17:23:53.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:53 smithi136 ceph-mon[115078]: pgmap v570: 521 pgs: 521 active+clean; 31 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 1.5 MiB/s rd, 9.9 MiB/s wr, 179 op/s; 12 MiB/s, 3 objects/s recovering 2024-08-17T17:23:53.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:53 smithi165 ceph-mon[105517]: pgmap v570: 521 pgs: 521 active+clean; 31 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 1.5 MiB/s rd, 9.9 MiB/s wr, 179 op/s; 12 MiB/s, 3 objects/s recovering 2024-08-17T17:23:54.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:54 smithi136 ceph-mon[109861]: osdmap e680: 8 total, 8 up, 8 in 2024-08-17T17:23:54.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:54 smithi136 ceph-mon[109861]: osdmap e681: 8 total, 8 up, 8 in 2024-08-17T17:23:54.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:54 smithi136 ceph-mon[115078]: osdmap e680: 8 total, 8 up, 8 in 2024-08-17T17:23:54.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:54 smithi136 ceph-mon[115078]: osdmap e681: 8 total, 8 up, 8 in 2024-08-17T17:23:54.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:54 smithi165 ceph-mon[105517]: osdmap e680: 8 total, 8 up, 8 in 2024-08-17T17:23:54.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:54 smithi165 ceph-mon[105517]: osdmap e681: 8 total, 8 up, 8 in 2024-08-17T17:23:55.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:55 smithi136 ceph-mon[109861]: pgmap v573: 521 pgs: 521 active+clean; 31 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 7.5 KiB/s wr, 137 op/s 2024-08-17T17:23:55.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:55 smithi136 ceph-mon[115078]: pgmap v573: 521 pgs: 521 active+clean; 31 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 7.5 KiB/s wr, 137 op/s 2024-08-17T17:23:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:55 smithi165 ceph-mon[105517]: pgmap v573: 521 pgs: 521 active+clean; 31 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 7.5 KiB/s wr, 137 op/s 2024-08-17T17:23:57.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:57 smithi136 ceph-mon[109861]: pgmap v574: 521 pgs: 521 active+clean; 39 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 1.2 MiB/s wr, 130 op/s 2024-08-17T17:23:57.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:57 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:57.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:57 smithi136 ceph-mon[115078]: pgmap v574: 521 pgs: 521 active+clean; 39 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 1.2 MiB/s wr, 130 op/s 2024-08-17T17:23:57.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:57 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:57.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:57 smithi165 ceph-mon[105517]: pgmap v574: 521 pgs: 521 active+clean; 39 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 1.2 MiB/s wr, 130 op/s 2024-08-17T17:23:57.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:57 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:23:57.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:23:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:23:57] "GET /metrics HTTP/1.1" 200 67792 "" "Prometheus/2.51.0" 2024-08-17T17:23:59.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:59 smithi136 ceph-mon[109861]: pgmap v575: 521 pgs: 521 active+clean; 60 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 3.6 MiB/s wr, 136 op/s 2024-08-17T17:23:59.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:59 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-08-17T17:23:59.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:59 smithi136 ceph-mon[109861]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-08-17T17:23:59.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:23:59 smithi136 ceph-mon[109861]: Upgrade: osd.3 is safe to restart 2024-08-17T17:23:59.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:59 smithi136 ceph-mon[115078]: pgmap v575: 521 pgs: 521 active+clean; 60 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 3.6 MiB/s wr, 136 op/s 2024-08-17T17:23:59.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:59 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-08-17T17:23:59.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:59 smithi136 ceph-mon[115078]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-08-17T17:23:59.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:23:59 smithi136 ceph-mon[115078]: Upgrade: osd.3 is safe to restart 2024-08-17T17:23:59.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:59 smithi165 ceph-mon[105517]: pgmap v575: 521 pgs: 521 active+clean; 60 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 3.6 MiB/s wr, 136 op/s 2024-08-17T17:23:59.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:59 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-08-17T17:23:59.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:59 smithi165 ceph-mon[105517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-08-17T17:23:59.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:23:59 smithi165 ceph-mon[105517]: Upgrade: osd.3 is safe to restart 2024-08-17T17:24:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:00 smithi165 ceph-mon[105517]: pgmap v576: 521 pgs: 521 active+clean; 60 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 3.6 MiB/s wr, 40 op/s 2024-08-17T17:24:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:00 smithi165 ceph-mon[105517]: Upgrade: Updating osd.3 2024-08-17T17:24:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:00 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:00 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-08-17T17:24:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:00 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:24:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:00 smithi165 ceph-mon[105517]: Deploying daemon osd.3 on smithi136 2024-08-17T17:24:01.411 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:00 smithi136 ceph-mon[109861]: pgmap v576: 521 pgs: 521 active+clean; 60 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 3.6 MiB/s wr, 40 op/s 2024-08-17T17:24:01.412 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:00 smithi136 ceph-mon[109861]: Upgrade: Updating osd.3 2024-08-17T17:24:01.412 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:00 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:01.412 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:00 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-08-17T17:24:01.412 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:00 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:24:01.412 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:00 smithi136 ceph-mon[109861]: Deploying daemon osd.3 on smithi136 2024-08-17T17:24:01.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:00 smithi136 ceph-mon[115078]: pgmap v576: 521 pgs: 521 active+clean; 60 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 3.6 MiB/s wr, 40 op/s 2024-08-17T17:24:01.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:00 smithi136 ceph-mon[115078]: Upgrade: Updating osd.3 2024-08-17T17:24:01.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:00 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:01.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:00 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-08-17T17:24:01.412 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:00 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:24:01.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:00 smithi136 ceph-mon[115078]: Deploying daemon osd.3 on smithi136 2024-08-17T17:24:02.898 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.LUKS1UnderLUKS2WithoutResize (13449 ms) 2024-08-17T17:24:02.898 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.LUKS2UnderLUKS1WithoutResize 2024-08-17T17:24:02.898 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:24:03.194 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:02 smithi136 ceph-mon[109861]: pgmap v577: 521 pgs: 521 active+clean; 76 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 5.1 MiB/s wr, 41 op/s 2024-08-17T17:24:03.194 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:02 smithi136 ceph-mon[115078]: pgmap v577: 521 pgs: 521 active+clean; 76 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 5.1 MiB/s wr, 41 op/s 2024-08-17T17:24:03.194 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:02 smithi136 systemd[1]: Stopping Ceph osd.3 for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:24:03.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:02 smithi165 ceph-mon[105517]: pgmap v577: 521 pgs: 521 active+clean; 76 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 5.1 MiB/s wr, 41 op/s 2024-08-17T17:24:03.446 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:03 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3[65713]: 2024-08-17T17:24:03.192+0000 7f8b6ddce640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:24:03.446 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:03 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3[65713]: 2024-08-17T17:24:03.192+0000 7f8b6ddce640 -1 osd.3 681 *** Got signal Terminated *** 2024-08-17T17:24:03.446 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:03 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3[65713]: 2024-08-17T17:24:03.192+0000 7f8b6ddce640 -1 osd.3 681 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T17:24:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:04 smithi165 ceph-mon[105517]: osd.3 marked itself down and dead 2024-08-17T17:24:04.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:04 smithi136 ceph-mon[109861]: osd.3 marked itself down and dead 2024-08-17T17:24:04.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:04 smithi136 ceph-mon[115078]: osd.3 marked itself down and dead 2024-08-17T17:24:04.893 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:04 smithi136 podman[147797]: 2024-08-17 17:24:04.602547312 +0000 UTC m=+1.534712623 container died 14b57a745d3052ab2d98769b9a0918121d712d5d7154797345e6d93a0af16842 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=reef-ced649f, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.36.0) 2024-08-17T17:24:04.893 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:04 smithi136 podman[147797]: 2024-08-17 17:24:04.890881924 +0000 UTC m=+1.823047243 container cleanup 14b57a745d3052ab2d98769b9a0918121d712d5d7154797345e6d93a0af16842 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=reef-ced649f, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-08-17T17:24:04.893 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:04 smithi136 bash[147797]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3 2024-08-17T17:24:05.177 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:05 smithi136 ceph-mon[109861]: pgmap v578: 521 pgs: 521 active+clean; 76 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 4.5 MiB/s wr, 36 op/s 2024-08-17T17:24:05.177 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:05 smithi136 ceph-mon[109861]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:24:05.177 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:05 smithi136 ceph-mon[109861]: osdmap e682: 8 total, 7 up, 8 in 2024-08-17T17:24:05.177 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:05 smithi136 ceph-mon[109861]: osd.3 marked itself dead as of e682 2024-08-17T17:24:05.178 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:05 smithi136 ceph-mon[115078]: pgmap v578: 521 pgs: 521 active+clean; 76 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 4.5 MiB/s wr, 36 op/s 2024-08-17T17:24:05.178 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:05 smithi136 ceph-mon[115078]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:24:05.178 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:05 smithi136 ceph-mon[115078]: osdmap e682: 8 total, 7 up, 8 in 2024-08-17T17:24:05.178 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:05 smithi136 ceph-mon[115078]: osd.3 marked itself dead as of e682 2024-08-17T17:24:05.178 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:05 smithi136 podman[148074]: 2024-08-17 17:24:05.175887461 +0000 UTC m=+0.566517405 container remove 14b57a745d3052ab2d98769b9a0918121d712d5d7154797345e6d93a0af16842 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=reef-ced649f, ceph=True, org.label-schema.build-date=20240731) 2024-08-17T17:24:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:05 smithi165 ceph-mon[105517]: pgmap v578: 521 pgs: 521 active+clean; 76 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 4.5 MiB/s wr, 36 op/s 2024-08-17T17:24:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:05 smithi165 ceph-mon[105517]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:24:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:05 smithi165 ceph-mon[105517]: osdmap e682: 8 total, 7 up, 8 in 2024-08-17T17:24:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:05 smithi165 ceph-mon[105517]: osd.3 marked itself dead as of e682 2024-08-17T17:24:05.898 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:05 smithi136 podman[148223]: 2024-08-17 17:24:05.530508825 +0000 UTC m=+0.022976528 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:24:05.898 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:05 smithi136 podman[148223]: 2024-08-17 17:24:05.635361531 +0000 UTC m=+0.127829218 container create 0b11b5f303e533e6583f802982a5896467696ccce579059bfc82d17aebd4c341 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-08-17T17:24:05.898 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:05 smithi136 podman[148223]: 2024-08-17 17:24:05.897344376 +0000 UTC m=+0.389812073 container init 0b11b5f303e533e6583f802982a5896467696ccce579059bfc82d17aebd4c341 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0) 2024-08-17T17:24:06.181 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:06 smithi136 ceph-mon[109861]: osdmap e683: 8 total, 7 up, 8 in 2024-08-17T17:24:06.182 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:06 smithi136 ceph-mon[115078]: osdmap e683: 8 total, 7 up, 8 in 2024-08-17T17:24:06.182 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:05 smithi136 podman[148223]: 2024-08-17 17:24:05.906783102 +0000 UTC m=+0.399250798 container start 0b11b5f303e533e6583f802982a5896467696ccce579059bfc82d17aebd4c341 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:24:06.182 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:05 smithi136 podman[148223]: 2024-08-17 17:24:05.979888899 +0000 UTC m=+0.472356578 container attach 0b11b5f303e533e6583f802982a5896467696ccce579059bfc82d17aebd4c341 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-08-17T17:24:06.446 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:06 smithi136 podman[148223]: 2024-08-17 17:24:06.183242461 +0000 UTC m=+0.675710155 container died 0b11b5f303e533e6583f802982a5896467696ccce579059bfc82d17aebd4c341 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-08-17T17:24:06.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:06 smithi165 ceph-mon[105517]: osdmap e683: 8 total, 7 up, 8 in 2024-08-17T17:24:07.027 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:06 smithi136 podman[148317]: 2024-08-17 17:24:06.750822217 +0000 UTC m=+0.556405401 container remove 0b11b5f303e533e6583f802982a5896467696ccce579059bfc82d17aebd4c341 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=main-cd8bdbe, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux ) 2024-08-17T17:24:07.027 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:06 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.3.service: Deactivated successfully. 2024-08-17T17:24:07.027 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:06 smithi136 systemd[1]: Stopped Ceph osd.3 for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:24:07.027 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:06 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.3.service: Consumed 32.482s CPU time. 2024-08-17T17:24:07.342 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:07 smithi136 ceph-mon[109861]: pgmap v581: 521 pgs: 66 active+undersized, 47 stale+active+clean, 28 active+undersized+degraded, 380 active+clean; 76 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s wr, 1 op/s; 53/732 objects degraded (7.240%) 2024-08-17T17:24:07.342 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:07 smithi136 ceph-mon[109861]: Health check failed: Degraded data redundancy: 53/732 objects degraded (7.240%), 28 pgs degraded (PG_DEGRADED) 2024-08-17T17:24:07.342 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:07.342 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:24:07.343 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:07 smithi136 ceph-mon[115078]: pgmap v581: 521 pgs: 66 active+undersized, 47 stale+active+clean, 28 active+undersized+degraded, 380 active+clean; 76 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s wr, 1 op/s; 53/732 objects degraded (7.240%) 2024-08-17T17:24:07.343 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:07 smithi136 ceph-mon[115078]: Health check failed: Degraded data redundancy: 53/732 objects degraded (7.240%), 28 pgs degraded (PG_DEGRADED) 2024-08-17T17:24:07.343 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:07.343 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:24:07.343 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:07 smithi136 systemd[1]: Starting Ceph osd.3 for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:24:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:07 smithi165 ceph-mon[105517]: pgmap v581: 521 pgs: 66 active+undersized, 47 stale+active+clean, 28 active+undersized+degraded, 380 active+clean; 76 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s wr, 1 op/s; 53/732 objects degraded (7.240%) 2024-08-17T17:24:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:07 smithi165 ceph-mon[105517]: Health check failed: Degraded data redundancy: 53/732 objects degraded (7.240%), 28 pgs degraded (PG_DEGRADED) 2024-08-17T17:24:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:24:07.623 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:07 smithi136 podman[148482]: 2024-08-17 17:24:07.245891821 +0000 UTC m=+0.029136642 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:24:07.623 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:07 smithi136 podman[148482]: 2024-08-17 17:24:07.387308746 +0000 UTC m=+0.170553571 container create 93bb393c11361c3a025b4d8eca6d8ec9910e9834ad1e2427b4459eee5509ee08 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-activate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, RELEASE=main-cd8bdbe, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-08-17T17:24:07.623 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:07 smithi136 podman[148482]: 2024-08-17 17:24:07.622747134 +0000 UTC m=+0.405991974 container init 93bb393c11361c3a025b4d8eca6d8ec9910e9834ad1e2427b4459eee5509ee08 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-activate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-08-17T17:24:07.624 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:24:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:24:07] "GET /metrics HTTP/1.1" 200 67792 "" "Prometheus/2.51.0" 2024-08-17T17:24:07.947 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:07 smithi136 podman[148482]: 2024-08-17 17:24:07.632453716 +0000 UTC m=+0.415698542 container start 93bb393c11361c3a025b4d8eca6d8ec9910e9834ad1e2427b4459eee5509ee08 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-activate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=main-cd8bdbe, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.36.0) 2024-08-17T17:24:07.947 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:07 smithi136 podman[148482]: 2024-08-17 17:24:07.713924517 +0000 UTC m=+0.497169343 container attach 93bb393c11361c3a025b4d8eca6d8ec9910e9834ad1e2427b4459eee5509ee08 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-activate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-08-17T17:24:07.947 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-activate[148515]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:24:07.947 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:07 smithi136 bash[148482]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:24:07.947 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-activate[148515]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:24:07.947 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:07 smithi136 bash[148482]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:24:07.947 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-activate[148515]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-17T17:24:07.948 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:07 smithi136 bash[148482]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-17T17:24:07.948 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-activate[148515]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:24:07.948 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:07 smithi136 bash[148482]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:24:07.948 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-activate[148515]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:24:07.948 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:07 smithi136 bash[148482]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:24:08.252 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-activate[148515]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-08-17T17:24:08.252 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:07 smithi136 bash[148482]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-08-17T17:24:08.252 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-activate[148515]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2024-08-17T17:24:08.252 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:07 smithi136 bash[148482]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2024-08-17T17:24:08.252 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:08 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-activate[148515]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3/block 2024-08-17T17:24:08.252 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:08 smithi136 bash[148482]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3/block 2024-08-17T17:24:08.588 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:08 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-activate[148515]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2024-08-17T17:24:08.588 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:08 smithi136 bash[148482]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2024-08-17T17:24:08.588 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:08 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-activate[148515]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-08-17T17:24:08.588 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:08 smithi136 bash[148482]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-08-17T17:24:08.588 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:08 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-activate[148515]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-08-17T17:24:08.588 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:08 smithi136 bash[148482]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-08-17T17:24:08.588 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:08 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-activate[148515]: --> ceph-volume lvm activate successful for osd ID: 3 2024-08-17T17:24:08.588 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:08 smithi136 bash[148482]: --> ceph-volume lvm activate successful for osd ID: 3 2024-08-17T17:24:08.588 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:08 smithi136 podman[148482]: 2024-08-17 17:24:08.323702727 +0000 UTC m=+1.106947551 container died 93bb393c11361c3a025b4d8eca6d8ec9910e9834ad1e2427b4459eee5509ee08 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-activate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-cd8bdbe, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0) 2024-08-17T17:24:09.178 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:09 smithi136 ceph-mon[109861]: pgmap v582: 521 pgs: 159 active+undersized, 61 active+undersized+degraded, 301 active+clean; 76 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s wr, 2 op/s; 106/738 objects degraded (14.363%) 2024-08-17T17:24:09.178 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:09 smithi136 ceph-mon[115078]: pgmap v582: 521 pgs: 159 active+undersized, 61 active+undersized+degraded, 301 active+clean; 76 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s wr, 2 op/s; 106/738 objects degraded (14.363%) 2024-08-17T17:24:09.178 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:08 smithi136 podman[148720]: 2024-08-17 17:24:08.926665505 +0000 UTC m=+0.594380795 container remove 93bb393c11361c3a025b4d8eca6d8ec9910e9834ad1e2427b4459eee5509ee08 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, RELEASE=main-cd8bdbe, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-08-17T17:24:09.178 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:09 smithi136 podman[148802]: 2024-08-17 17:24:09.050926055 +0000 UTC m=+0.028424310 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:24:09.446 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:09 smithi136 podman[148802]: 2024-08-17 17:24:09.17781371 +0000 UTC m=+0.155311951 container create 64f31d4557dbb44e30de087730f06fde260c03935a1d253e7d84e2357094233f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-cd8bdbe, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-08-17T17:24:09.447 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:09 smithi136 podman[148802]: 2024-08-17 17:24:09.380396515 +0000 UTC m=+0.357894756 container init 64f31d4557dbb44e30de087730f06fde260c03935a1d253e7d84e2357094233f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-cd8bdbe, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:24:09.447 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:09 smithi136 podman[148802]: 2024-08-17 17:24:09.385567538 +0000 UTC m=+0.363065796 container start 64f31d4557dbb44e30de087730f06fde260c03935a1d253e7d84e2357094233f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.license=GPLv2, RELEASE=main-cd8bdbe, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2024-08-17T17:24:09.447 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:09 smithi136 ceph-osd[148839]: -- 172.21.15.136:0/1401404135 <== mon.0 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f736759a40 con 0x55f736737c00 2024-08-17T17:24:09.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:09 smithi165 ceph-mon[105517]: pgmap v582: 521 pgs: 159 active+undersized, 61 active+undersized+degraded, 301 active+clean; 76 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 2.0 MiB/s wr, 2 op/s; 106/738 objects degraded (14.363%) 2024-08-17T17:24:09.946 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:09 smithi136 bash[148802]: 64f31d4557dbb44e30de087730f06fde260c03935a1d253e7d84e2357094233f 2024-08-17T17:24:09.946 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:09 smithi136 systemd[1]: Started Ceph osd.3 for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:24:10.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:10 smithi136 ceph-mon[109861]: osdmap e684: 8 total, 7 up, 8 in 2024-08-17T17:24:10.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:10.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:10.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:10.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:24:10.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:10.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:10.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:10 smithi136 ceph-mon[115078]: osdmap e684: 8 total, 7 up, 8 in 2024-08-17T17:24:10.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:10.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:10.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:10.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:24:10.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:10.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:10.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:10 smithi165 ceph-mon[105517]: osdmap e684: 8 total, 7 up, 8 in 2024-08-17T17:24:10.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:10.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:10.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:10.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:24:10.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:10.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:11.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:11 smithi136 ceph-mon[109861]: pgmap v584: 521 pgs: 159 active+undersized, 61 active+undersized+degraded, 301 active+clean; 76 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 682 B/s wr, 0 op/s; 106/738 objects degraded (14.363%) 2024-08-17T17:24:11.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:11 smithi136 ceph-mon[115078]: pgmap v584: 521 pgs: 159 active+undersized, 61 active+undersized+degraded, 301 active+clean; 76 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 682 B/s wr, 0 op/s; 106/738 objects degraded (14.363%) 2024-08-17T17:24:11.447 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:11 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3[148816]: 2024-08-17T17:24:11.262+0000 7f453d9d5640 -1 Falling back to public interface 2024-08-17T17:24:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:11 smithi165 ceph-mon[105517]: pgmap v584: 521 pgs: 159 active+undersized, 61 active+undersized+degraded, 301 active+clean; 76 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 682 B/s wr, 0 op/s; 106/738 objects degraded (14.363%) 2024-08-17T17:24:12.378 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:12 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 111/783 objects degraded (14.176%), 61 pgs degraded (PG_DEGRADED) 2024-08-17T17:24:12.379 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:12 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 111/783 objects degraded (14.176%), 61 pgs degraded (PG_DEGRADED) 2024-08-17T17:24:12.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:12 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 111/783 objects degraded (14.176%), 61 pgs degraded (PG_DEGRADED) 2024-08-17T17:24:13.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:13 smithi136 ceph-mon[109861]: pgmap v585: 521 pgs: 159 active+undersized, 61 active+undersized+degraded, 301 active+clean; 117 MiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.1 MiB/s wr, 31 op/s; 111/783 objects degraded (14.176%) 2024-08-17T17:24:13.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:13 smithi136 ceph-mon[115078]: pgmap v585: 521 pgs: 159 active+undersized, 61 active+undersized+degraded, 301 active+clean; 117 MiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.1 MiB/s wr, 31 op/s; 111/783 objects degraded (14.176%) 2024-08-17T17:24:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:13 smithi165 ceph-mon[105517]: pgmap v585: 521 pgs: 159 active+undersized, 61 active+undersized+degraded, 301 active+clean; 117 MiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.1 MiB/s wr, 31 op/s; 111/783 objects degraded (14.176%) 2024-08-17T17:24:14.615 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:14 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:14.615 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:14 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:14.615 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:14 smithi136 ceph-mon[109861]: pgmap v586: 521 pgs: 159 active+undersized, 61 active+undersized+degraded, 301 active+clean; 117 MiB data, 776 MiB used, 714 GiB / 715 GiB avail; 961 KiB/s rd, 4.6 MiB/s wr, 27 op/s; 111/783 objects degraded (14.176%) 2024-08-17T17:24:14.615 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:14 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:14.615 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:14 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:14.615 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:14 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:14.615 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:14 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:14.616 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:14 smithi136 ceph-mon[115078]: pgmap v586: 521 pgs: 159 active+undersized, 61 active+undersized+degraded, 301 active+clean; 117 MiB data, 776 MiB used, 714 GiB / 715 GiB avail; 961 KiB/s rd, 4.6 MiB/s wr, 27 op/s; 111/783 objects degraded (14.176%) 2024-08-17T17:24:14.616 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:14 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:14.616 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:14 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:14 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:14 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:14 smithi165 ceph-mon[105517]: pgmap v586: 521 pgs: 159 active+undersized, 61 active+undersized+degraded, 301 active+clean; 117 MiB data, 776 MiB used, 714 GiB / 715 GiB avail; 961 KiB/s rd, 4.6 MiB/s wr, 27 op/s; 111/783 objects degraded (14.176%) 2024-08-17T17:24:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:14 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:14.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:14 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:15.367 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:15 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3[148816]: 2024-08-17T17:24:15.094+0000 7f453d9d5640 -1 osd.3 682 log_to_monitors true 2024-08-17T17:24:15.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:15 smithi136 ceph-mon[109861]: from='osd.3 [v2:172.21.15.136:6826/946860336,v1:172.21.15.136:6827/946860336]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-08-17T17:24:15.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:15 smithi136 ceph-mon[115078]: from='osd.3 [v2:172.21.15.136:6826/946860336,v1:172.21.15.136:6827/946860336]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-08-17T17:24:15.697 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 17:24:15 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3[148816]: 2024-08-17T17:24:15.481+0000 7f453577f640 -1 osd.3 682 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:24:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:15 smithi165 ceph-mon[105517]: from='osd.3 [v2:172.21.15.136:6826/946860336,v1:172.21.15.136:6827/946860336]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-08-17T17:24:16.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:16 smithi136 ceph-mon[109861]: from='osd.3 [v2:172.21.15.136:6826/946860336,v1:172.21.15.136:6827/946860336]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-17T17:24:16.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:16 smithi136 ceph-mon[109861]: osdmap e685: 8 total, 7 up, 8 in 2024-08-17T17:24:16.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:16 smithi136 ceph-mon[109861]: from='osd.3 [v2:172.21.15.136:6826/946860336,v1:172.21.15.136:6827/946860336]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-17T17:24:16.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:16 smithi136 ceph-mon[109861]: pgmap v588: 521 pgs: 159 active+undersized, 61 active+undersized+degraded, 301 active+clean; 117 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 67 KiB/s rd, 5.1 MiB/s wr, 45 op/s; 111/783 objects degraded (14.176%) 2024-08-17T17:24:16.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:16 smithi136 ceph-mon[115078]: from='osd.3 [v2:172.21.15.136:6826/946860336,v1:172.21.15.136:6827/946860336]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-17T17:24:16.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:16 smithi136 ceph-mon[115078]: osdmap e685: 8 total, 7 up, 8 in 2024-08-17T17:24:16.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:16 smithi136 ceph-mon[115078]: from='osd.3 [v2:172.21.15.136:6826/946860336,v1:172.21.15.136:6827/946860336]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-17T17:24:16.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:16 smithi136 ceph-mon[115078]: pgmap v588: 521 pgs: 159 active+undersized, 61 active+undersized+degraded, 301 active+clean; 117 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 67 KiB/s rd, 5.1 MiB/s wr, 45 op/s; 111/783 objects degraded (14.176%) 2024-08-17T17:24:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:16 smithi165 ceph-mon[105517]: from='osd.3 [v2:172.21.15.136:6826/946860336,v1:172.21.15.136:6827/946860336]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-17T17:24:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:16 smithi165 ceph-mon[105517]: osdmap e685: 8 total, 7 up, 8 in 2024-08-17T17:24:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:16 smithi165 ceph-mon[105517]: from='osd.3 [v2:172.21.15.136:6826/946860336,v1:172.21.15.136:6827/946860336]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-17T17:24:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:16 smithi165 ceph-mon[105517]: pgmap v588: 521 pgs: 159 active+undersized, 61 active+undersized+degraded, 301 active+clean; 117 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 67 KiB/s rd, 5.1 MiB/s wr, 45 op/s; 111/783 objects degraded (14.176%) 2024-08-17T17:24:17.179 INFO:teuthology.orchestra.run.smithi136.stdout:true 2024-08-17T17:24:17.524 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.LUKS2UnderLUKS1WithoutResize (14627 ms) 2024-08-17T17:24:17.524 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.EncryptionFormatNoData 2024-08-17T17:24:17.524 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:24:17.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:24:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:24:17] "GET /metrics HTTP/1.1" 200 67792 "" "Prometheus/2.51.0" 2024-08-17T17:24:17.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:17 smithi136 ceph-mon[109861]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:24:17.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:17 smithi136 ceph-mon[109861]: osd.3 [v2:172.21.15.136:6826/946860336,v1:172.21.15.136:6827/946860336] boot 2024-08-17T17:24:17.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:17 smithi136 ceph-mon[109861]: osdmap e686: 8 total, 8 up, 8 in 2024-08-17T17:24:17.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:17 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-17T17:24:17.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:17 smithi136 ceph-mon[109861]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:24:17.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:17 smithi136 ceph-mon[109861]: osdmap e687: 8 total, 8 up, 8 in 2024-08-17T17:24:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:17 smithi136 ceph-mon[115078]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:24:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:17 smithi136 ceph-mon[115078]: osd.3 [v2:172.21.15.136:6826/946860336,v1:172.21.15.136:6827/946860336] boot 2024-08-17T17:24:17.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:17 smithi136 ceph-mon[115078]: osdmap e686: 8 total, 8 up, 8 in 2024-08-17T17:24:17.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:17 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-17T17:24:17.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:17 smithi136 ceph-mon[115078]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:24:17.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:17 smithi136 ceph-mon[115078]: osdmap e687: 8 total, 8 up, 8 in 2024-08-17T17:24:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:17 smithi165 ceph-mon[105517]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:24:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:17 smithi165 ceph-mon[105517]: osd.3 [v2:172.21.15.136:6826/946860336,v1:172.21.15.136:6827/946860336] boot 2024-08-17T17:24:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:17 smithi165 ceph-mon[105517]: osdmap e686: 8 total, 8 up, 8 in 2024-08-17T17:24:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:17 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-17T17:24:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:17 smithi165 ceph-mon[105517]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:24:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:17 smithi165 ceph-mon[105517]: osdmap e687: 8 total, 8 up, 8 in 2024-08-17T17:24:17.821 INFO:teuthology.orchestra.run.smithi136.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-17T17:24:17.821 INFO:teuthology.orchestra.run.smithi136.stdout:alertmanager.a smithi136 *:9093,9094 running (9m) 4s ago 15m 19.4M - 0.27.0 11f11916f8cd c3b9b5b223ab 2024-08-17T17:24:17.821 INFO:teuthology.orchestra.run.smithi136.stdout:grafana.a smithi165 *:3000 running (9m) 38s ago 15m 77.6M - c8b91775d855 04db24fce5e9 2024-08-17T17:24:17.821 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi136.ltsuqd smithi136 running (14m) 4s ago 14m 24.9M - 18.2.4-719-gced649f8 c870753d67c2 faaa06f2950c 2024-08-17T17:24:17.821 INFO:teuthology.orchestra.run.smithi136.stdout:mds.foo.smithi165.bmrbhh smithi165 running (14m) 38s ago 14m 18.8M - 18.2.4-719-gced649f8 c870753d67c2 c74b00d929cf 2024-08-17T17:24:17.821 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.x smithi165 *:8443,9283,8765 running (10m) 38s ago 19m 458M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 837fd48ff7cb 2024-08-17T17:24:17.822 INFO:teuthology.orchestra.run.smithi136.stdout:mgr.y smithi136 *:8443,9283,8765 running (11m) 4s ago 20m 537M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9acdc6775e48 2024-08-17T17:24:17.822 INFO:teuthology.orchestra.run.smithi136.stdout:mon.a smithi136 running (8m) 4s ago 21m 78.9M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9cf82e613626 2024-08-17T17:24:17.822 INFO:teuthology.orchestra.run.smithi136.stdout:mon.b smithi165 running (6m) 38s ago 19m 55.6M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 1fd357308b8d 2024-08-17T17:24:17.822 INFO:teuthology.orchestra.run.smithi136.stdout:mon.c smithi136 running (7m) 4s ago 19m 66.0M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 2c2ac98ad9d4 2024-08-17T17:24:17.822 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.a smithi136 *:9100 running (10m) 4s ago 14m 11.2M - 1.7.0 72c9c2088986 1e4c644ae533 2024-08-17T17:24:17.822 INFO:teuthology.orchestra.run.smithi136.stdout:node-exporter.b smithi165 *:9100 running (10m) 38s ago 14m 11.6M - 1.7.0 72c9c2088986 7af2d2d68b06 2024-08-17T17:24:17.822 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0 smithi136 running (112s) 4s ago 18m 64.7M 2480M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 71b7314ae4a5 2024-08-17T17:24:17.822 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1 smithi136 running (79s) 4s ago 18m 66.8M 2480M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb a0bc94c66763 2024-08-17T17:24:17.822 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2 smithi136 running (43s) 4s ago 18m 63.0M 2480M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 933eab0a702a 2024-08-17T17:24:17.822 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3 smithi136 running (8s) 4s ago 17m 12.2M 2480M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 64f31d4557db 2024-08-17T17:24:17.822 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4 smithi165 running (17m) 38s ago 17m 134M 2480M 18.2.4-719-gced649f8 c870753d67c2 df366c73e1f2 2024-08-17T17:24:17.822 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5 smithi165 running (16m) 38s ago 16m 115M 2480M 18.2.4-719-gced649f8 c870753d67c2 1131665e203a 2024-08-17T17:24:17.823 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6 smithi165 running (16m) 38s ago 16m 149M 2480M 18.2.4-719-gced649f8 c870753d67c2 7bc5fd925293 2024-08-17T17:24:17.823 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7 smithi165 running (16m) 38s ago 16m 127M 2480M 18.2.4-719-gced649f8 c870753d67c2 18382e4ae3b8 2024-08-17T17:24:17.823 INFO:teuthology.orchestra.run.smithi136.stdout:prometheus.a smithi165 *:9095 running (9m) 38s ago 15m 48.0M - 2.51.0 1d3b7f56885b ae8f8b69fe2f 2024-08-17T17:24:18.191 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:24:18.191 INFO:teuthology.orchestra.run.smithi136.stdout: "mon": { 2024-08-17T17:24:18.191 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 3 2024-08-17T17:24:18.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:24:18.191 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr": { 2024-08-17T17:24:18.191 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 2 2024-08-17T17:24:18.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:24:18.191 INFO:teuthology.orchestra.run.smithi136.stdout: "osd": { 2024-08-17T17:24:18.191 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 4, 2024-08-17T17:24:18.191 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 4 2024-08-17T17:24:18.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:24:18.192 INFO:teuthology.orchestra.run.smithi136.stdout: "mds": { 2024-08-17T17:24:18.192 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 2 2024-08-17T17:24:18.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:24:18.192 INFO:teuthology.orchestra.run.smithi136.stdout: "overall": { 2024-08-17T17:24:18.192 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 18.2.4-719-gced649f8 (ced649f89259a8c5733cc18a50f42f20f41ba352) reef (stable)": 6, 2024-08-17T17:24:18.192 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 9 2024-08-17T17:24:18.193 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:24:18.193 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:24:18.517 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:24:18.517 INFO:teuthology.orchestra.run.smithi136.stdout: "target_image": null, 2024-08-17T17:24:18.517 INFO:teuthology.orchestra.run.smithi136.stdout: "in_progress": false, 2024-08-17T17:24:18.517 INFO:teuthology.orchestra.run.smithi136.stdout: "which": "", 2024-08-17T17:24:18.517 INFO:teuthology.orchestra.run.smithi136.stdout: "services_complete": [], 2024-08-17T17:24:18.517 INFO:teuthology.orchestra.run.smithi136.stdout: "progress": null, 2024-08-17T17:24:18.517 INFO:teuthology.orchestra.run.smithi136.stdout: "message": "", 2024-08-17T17:24:18.518 INFO:teuthology.orchestra.run.smithi136.stdout: "is_paused": false 2024-08-17T17:24:18.518 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:24:19.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:24:19.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='client.44543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:24:19.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:19.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: pgmap v591: 521 pgs: 79 peering, 103 active+undersized, 38 active+undersized+degraded, 301 active+clean; 121 MiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 682 KiB/s wr, 3 op/s; 70/786 objects degraded (8.906%) 2024-08-17T17:24:19.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:19.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:24:19.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:24:19.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:19.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:24:19.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: Upgrade: Setting container_image for all nvmeof 2024-08-17T17:24:19.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:19.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: Upgrade: Finalizing container_image settings 2024-08-17T17:24:19.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-08-17T17:24:19.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-08-17T17:24:19.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:24:19.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-08-17T17:24:19.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-08-17T17:24:19.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-08-17T17:24:19.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-08-17T17:24:19.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-08-17T17:24:19.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-08-17T17:24:19.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-08-17T17:24:19.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:24:19.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-08-17T17:24:19.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-08-17T17:24:19.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-08-17T17:24:19.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-08-17T17:24:19.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-08-17T17:24:19.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:24:19.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:24:19.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:24:19.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:24:19.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:24:19.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:24:19.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: Upgrade: Complete! 2024-08-17T17:24:19.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-08-17T17:24:19.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-08-17T17:24:19.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:24:19.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:24:19.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:24:19.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:19.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3384138282' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:24:19.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:24:19.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:24:19.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:19.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:24:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='client.44543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:24:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: pgmap v591: 521 pgs: 79 peering, 103 active+undersized, 38 active+undersized+degraded, 301 active+clean; 121 MiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 682 KiB/s wr, 3 op/s; 70/786 objects degraded (8.906%) 2024-08-17T17:24:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:24:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:24:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:24:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: Upgrade: Setting container_image for all nvmeof 2024-08-17T17:24:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: Upgrade: Finalizing container_image settings 2024-08-17T17:24:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-08-17T17:24:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-08-17T17:24:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:24:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-08-17T17:24:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-08-17T17:24:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-08-17T17:24:19.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:24:19.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='client.44543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:24:19.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:19.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: pgmap v591: 521 pgs: 79 peering, 103 active+undersized, 38 active+undersized+degraded, 301 active+clean; 121 MiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 682 KiB/s wr, 3 op/s; 70/786 objects degraded (8.906%) 2024-08-17T17:24:19.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:19.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:24:19.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:24:19.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:19.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:24:19.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: Upgrade: Setting container_image for all nvmeof 2024-08-17T17:24:19.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:19.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: Upgrade: Finalizing container_image settings 2024-08-17T17:24:19.450 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-08-17T17:24:19.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-08-17T17:24:19.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:24:19.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-08-17T17:24:19.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-08-17T17:24:19.451 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-08-17T17:24:19.451 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-08-17T17:24:19.451 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-08-17T17:24:19.452 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-08-17T17:24:19.452 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-08-17T17:24:19.452 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:24:19.452 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-08-17T17:24:19.452 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-08-17T17:24:19.452 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-08-17T17:24:19.452 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-08-17T17:24:19.452 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-08-17T17:24:19.452 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:24:19.452 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:24:19.452 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:24:19.452 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:24:19.452 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:24:19.453 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:24:19.453 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: Upgrade: Complete! 2024-08-17T17:24:19.453 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-08-17T17:24:19.453 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-08-17T17:24:19.453 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:24:19.453 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:24:19.453 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:24:19.453 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:19.453 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3384138282' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.453 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:24:19.453 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:24:19.453 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:24:19.453 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:19.454 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-08-17T17:24:19.454 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-08-17T17:24:19.454 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-08-17T17:24:19.454 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-08-17T17:24:19.454 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:24:19.454 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-08-17T17:24:19.454 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-08-17T17:24:19.454 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-08-17T17:24:19.454 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-08-17T17:24:19.455 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-08-17T17:24:19.455 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:24:19.455 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:24:19.455 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:24:19.455 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:24:19.455 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:24:19.455 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-08-17T17:24:19.455 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: Upgrade: Complete! 2024-08-17T17:24:19.455 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-08-17T17:24:19.455 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-08-17T17:24:19.455 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:24:19.455 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:24:19.455 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:24:19.456 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:19.456 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3384138282' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:24:19.456 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:24:19.456 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:24:19.456 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:24:19.456 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:19 smithi165 ceph-mon[105517]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:24:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:19 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 70/786 objects degraded (8.906%), 38 pgs degraded (PG_DEGRADED) 2024-08-17T17:24:20.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:19 smithi136 ceph-mon[109861]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:24:20.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:19 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 70/786 objects degraded (8.906%), 38 pgs degraded (PG_DEGRADED) 2024-08-17T17:24:20.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:19 smithi136 ceph-mon[115078]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:24:20.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:19 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 70/786 objects degraded (8.906%), 38 pgs degraded (PG_DEGRADED) 2024-08-17T17:24:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:20 smithi165 ceph-mon[105517]: pgmap v592: 521 pgs: 79 peering, 103 active+undersized, 38 active+undersized+degraded, 301 active+clean; 121 MiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 683 KiB/s wr, 3 op/s; 70/786 objects degraded (8.906%) 2024-08-17T17:24:21.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:20 smithi136 ceph-mon[109861]: pgmap v592: 521 pgs: 79 peering, 103 active+undersized, 38 active+undersized+degraded, 301 active+clean; 121 MiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 683 KiB/s wr, 3 op/s; 70/786 objects degraded (8.906%) 2024-08-17T17:24:21.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:20 smithi136 ceph-mon[115078]: pgmap v592: 521 pgs: 79 peering, 103 active+undersized, 38 active+undersized+degraded, 301 active+clean; 121 MiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 683 KiB/s wr, 3 op/s; 70/786 objects degraded (8.906%) 2024-08-17T17:24:21.714 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.EncryptionFormatNoData (4190 ms) 2024-08-17T17:24:21.714 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.EncryptionLoadBadSize 2024-08-17T17:24:21.714 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:24:22.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:22.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:24:22.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:24:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:24:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:23.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:23 smithi136 ceph-mon[109861]: pgmap v593: 521 pgs: 521 active+clean; 121 MiB data, 771 MiB used, 714 GiB / 715 GiB avail; 3.9 MiB/s rd, 621 KiB/s wr, 20 op/s; 3.0 MiB/s, 9 keys/s, 1 objects/s recovering 2024-08-17T17:24:23.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:23 smithi136 ceph-mon[109861]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 70/786 objects degraded (8.906%), 38 pgs degraded) 2024-08-17T17:24:23.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:23 smithi136 ceph-mon[115078]: pgmap v593: 521 pgs: 521 active+clean; 121 MiB data, 771 MiB used, 714 GiB / 715 GiB avail; 3.9 MiB/s rd, 621 KiB/s wr, 20 op/s; 3.0 MiB/s, 9 keys/s, 1 objects/s recovering 2024-08-17T17:24:23.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:23 smithi136 ceph-mon[115078]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 70/786 objects degraded (8.906%), 38 pgs degraded) 2024-08-17T17:24:23.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:23 smithi165 ceph-mon[105517]: pgmap v593: 521 pgs: 521 active+clean; 121 MiB data, 771 MiB used, 714 GiB / 715 GiB avail; 3.9 MiB/s rd, 621 KiB/s wr, 20 op/s; 3.0 MiB/s, 9 keys/s, 1 objects/s recovering 2024-08-17T17:24:23.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:23 smithi165 ceph-mon[105517]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 70/786 objects degraded (8.906%), 38 pgs degraded) 2024-08-17T17:24:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:25 smithi136 ceph-mon[109861]: pgmap v594: 521 pgs: 521 active+clean; 121 MiB data, 771 MiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 513 KiB/s wr, 14 op/s; 2.5 MiB/s, 8 keys/s, 1 objects/s recovering 2024-08-17T17:24:25.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:25 smithi136 ceph-mon[115078]: pgmap v594: 521 pgs: 521 active+clean; 121 MiB data, 771 MiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 513 KiB/s wr, 14 op/s; 2.5 MiB/s, 8 keys/s, 1 objects/s recovering 2024-08-17T17:24:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:25 smithi165 ceph-mon[105517]: pgmap v594: 521 pgs: 521 active+clean; 121 MiB data, 771 MiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 513 KiB/s wr, 14 op/s; 2.5 MiB/s, 8 keys/s, 1 objects/s recovering 2024-08-17T17:24:27.203 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.EncryptionLoadBadSize (5488 ms) 2024-08-17T17:24:27.203 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.EncryptionLoadBadStripePattern 2024-08-17T17:24:27.203 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_librbd.cc:2905: Skipped 2024-08-17T17:24:27.203 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:24:27.203 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadBadStripePattern (0 ms) 2024-08-17T17:24:27.203 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.EncryptionLoadFormatMismatch 2024-08-17T17:24:27.203 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:24:27.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:27 smithi136 ceph-mon[109861]: pgmap v595: 521 pgs: 521 active+clean; 125 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 854 KiB/s wr, 20 op/s; 2.1 MiB/s, 6 keys/s, 1 objects/s recovering 2024-08-17T17:24:27.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:27 smithi136 ceph-mon[115078]: pgmap v595: 521 pgs: 521 active+clean; 125 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 854 KiB/s wr, 20 op/s; 2.1 MiB/s, 6 keys/s, 1 objects/s recovering 2024-08-17T17:24:27.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:27 smithi165 ceph-mon[105517]: pgmap v595: 521 pgs: 521 active+clean; 125 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 854 KiB/s wr, 20 op/s; 2.1 MiB/s, 6 keys/s, 1 objects/s recovering 2024-08-17T17:24:27.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:24:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:24:27] "GET /metrics HTTP/1.1" 200 67789 "" "Prometheus/2.51.0" 2024-08-17T17:24:29.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:29 smithi136 ceph-mon[109861]: pgmap v596: 521 pgs: 521 active+clean; 129 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 773 KiB/s wr, 19 op/s; 1.9 MiB/s, 6 keys/s, 0 objects/s recovering 2024-08-17T17:24:29.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:29 smithi136 ceph-mon[115078]: pgmap v596: 521 pgs: 521 active+clean; 129 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 773 KiB/s wr, 19 op/s; 1.9 MiB/s, 6 keys/s, 0 objects/s recovering 2024-08-17T17:24:29.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:29 smithi165 ceph-mon[105517]: pgmap v596: 521 pgs: 521 active+clean; 129 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 773 KiB/s wr, 19 op/s; 1.9 MiB/s, 6 keys/s, 0 objects/s recovering 2024-08-17T17:24:31.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:31 smithi136 ceph-mon[109861]: pgmap v597: 521 pgs: 521 active+clean; 129 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 684 KiB/s wr, 17 op/s; 1.7 MiB/s, 5 keys/s, 0 objects/s recovering 2024-08-17T17:24:31.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:31 smithi136 ceph-mon[115078]: pgmap v597: 521 pgs: 521 active+clean; 129 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 684 KiB/s wr, 17 op/s; 1.7 MiB/s, 5 keys/s, 0 objects/s recovering 2024-08-17T17:24:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:31 smithi165 ceph-mon[105517]: pgmap v597: 521 pgs: 521 active+clean; 129 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 684 KiB/s wr, 17 op/s; 1.7 MiB/s, 5 keys/s, 0 objects/s recovering 2024-08-17T17:24:32.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:32 smithi136 ceph-mon[109861]: osdmap e688: 8 total, 8 up, 8 in 2024-08-17T17:24:32.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:32.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:32 smithi136 ceph-mon[115078]: osdmap e688: 8 total, 8 up, 8 in 2024-08-17T17:24:32.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:32.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:32 smithi165 ceph-mon[105517]: osdmap e688: 8 total, 8 up, 8 in 2024-08-17T17:24:32.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:33.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:33 smithi136 ceph-mon[115078]: pgmap v599: 521 pgs: 521 active+clean; 133 MiB data, 803 MiB used, 714 GiB / 715 GiB avail; 85 KiB/s rd, 1.2 MiB/s wr, 38 op/s 2024-08-17T17:24:33.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:33 smithi136 ceph-mon[115078]: osdmap e689: 8 total, 8 up, 8 in 2024-08-17T17:24:33.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:33 smithi136 ceph-mon[109861]: pgmap v599: 521 pgs: 521 active+clean; 133 MiB data, 803 MiB used, 714 GiB / 715 GiB avail; 85 KiB/s rd, 1.2 MiB/s wr, 38 op/s 2024-08-17T17:24:33.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:33 smithi136 ceph-mon[109861]: osdmap e689: 8 total, 8 up, 8 in 2024-08-17T17:24:33.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:33 smithi165 ceph-mon[105517]: pgmap v599: 521 pgs: 521 active+clean; 133 MiB data, 803 MiB used, 714 GiB / 715 GiB avail; 85 KiB/s rd, 1.2 MiB/s wr, 38 op/s 2024-08-17T17:24:33.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:33 smithi165 ceph-mon[105517]: osdmap e689: 8 total, 8 up, 8 in 2024-08-17T17:24:35.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:35 smithi136 ceph-mon[109861]: pgmap v601: 521 pgs: 521 active+clean; 133 MiB data, 803 MiB used, 714 GiB / 715 GiB avail; 100 KiB/s rd, 1.0 MiB/s wr, 38 op/s 2024-08-17T17:24:35.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:35 smithi136 ceph-mon[115078]: pgmap v601: 521 pgs: 521 active+clean; 133 MiB data, 803 MiB used, 714 GiB / 715 GiB avail; 100 KiB/s rd, 1.0 MiB/s wr, 38 op/s 2024-08-17T17:24:35.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:35 smithi165 ceph-mon[105517]: pgmap v601: 521 pgs: 521 active+clean; 133 MiB data, 803 MiB used, 714 GiB / 715 GiB avail; 100 KiB/s rd, 1.0 MiB/s wr, 38 op/s 2024-08-17T17:24:37.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:37 smithi136 ceph-mon[109861]: pgmap v602: 521 pgs: 521 active+clean; 133 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 84 KiB/s rd, 515 KiB/s wr, 63 op/s 2024-08-17T17:24:37.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:37.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:24:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:37 smithi136 ceph-mon[115078]: pgmap v602: 521 pgs: 521 active+clean; 133 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 84 KiB/s rd, 515 KiB/s wr, 63 op/s 2024-08-17T17:24:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:24:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:37 smithi165 ceph-mon[105517]: pgmap v602: 521 pgs: 521 active+clean; 133 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 84 KiB/s rd, 515 KiB/s wr, 63 op/s 2024-08-17T17:24:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:24:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:24:37.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:24:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:24:37] "GET /metrics HTTP/1.1" 200 67787 "" "Prometheus/2.51.0" 2024-08-17T17:24:39.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:39 smithi136 ceph-mon[109861]: pgmap v603: 521 pgs: 521 active+clean; 133 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 98 KiB/s rd, 515 KiB/s wr, 80 op/s 2024-08-17T17:24:39.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:39 smithi136 ceph-mon[115078]: pgmap v603: 521 pgs: 521 active+clean; 133 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 98 KiB/s rd, 515 KiB/s wr, 80 op/s 2024-08-17T17:24:39.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:39 smithi165 ceph-mon[105517]: pgmap v603: 521 pgs: 521 active+clean; 133 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 98 KiB/s rd, 515 KiB/s wr, 80 op/s 2024-08-17T17:24:41.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:41 smithi136 ceph-mon[109861]: pgmap v604: 521 pgs: 521 active+clean; 133 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 83 KiB/s rd, 467 KiB/s wr, 66 op/s 2024-08-17T17:24:41.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:41 smithi136 ceph-mon[115078]: pgmap v604: 521 pgs: 521 active+clean; 133 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 83 KiB/s rd, 467 KiB/s wr, 66 op/s 2024-08-17T17:24:41.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:41 smithi165 ceph-mon[105517]: pgmap v604: 521 pgs: 521 active+clean; 133 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 83 KiB/s rd, 467 KiB/s wr, 66 op/s 2024-08-17T17:24:43.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:43 smithi136 ceph-mon[109861]: pgmap v605: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.6 MiB/s wr, 52 op/s 2024-08-17T17:24:43.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:43 smithi136 ceph-mon[115078]: pgmap v605: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.6 MiB/s wr, 52 op/s 2024-08-17T17:24:43.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:43 smithi165 ceph-mon[105517]: pgmap v605: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 2.5 MiB/s rd, 1.6 MiB/s wr, 52 op/s 2024-08-17T17:24:45.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:45 smithi136 ceph-mon[109861]: pgmap v606: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 44 op/s 2024-08-17T17:24:45.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:45 smithi136 ceph-mon[115078]: pgmap v606: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 44 op/s 2024-08-17T17:24:45.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:45 smithi165 ceph-mon[105517]: pgmap v606: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 44 op/s 2024-08-17T17:24:47.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:47 smithi136 ceph-mon[109861]: pgmap v607: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.3 MiB/s wr, 43 op/s 2024-08-17T17:24:47.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:47 smithi136 ceph-mon[115078]: pgmap v607: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.3 MiB/s wr, 43 op/s 2024-08-17T17:24:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:47 smithi165 ceph-mon[105517]: pgmap v607: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.3 MiB/s wr, 43 op/s 2024-08-17T17:24:47.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:24:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:24:47] "GET /metrics HTTP/1.1" 200 67787 "" "Prometheus/2.51.0" 2024-08-17T17:24:49.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:49 smithi136 ceph-mon[109861]: pgmap v608: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 1.3 MiB/s wr, 27 op/s 2024-08-17T17:24:49.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:49 smithi136 ceph-mon[115078]: pgmap v608: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 1.3 MiB/s wr, 27 op/s 2024-08-17T17:24:49.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:49 smithi165 ceph-mon[105517]: pgmap v608: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 1.3 MiB/s wr, 27 op/s 2024-08-17T17:24:49.574 DEBUG:teuthology.parallel:result is None 2024-08-17T17:24:50.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:50 smithi136 ceph-mon[109861]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:24:50.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:50 smithi136 ceph-mon[115078]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:24:50.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:50 smithi165 ceph-mon[105517]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:24:51.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:51 smithi136 ceph-mon[109861]: pgmap v609: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2024-08-17T17:24:51.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:24:51.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:51 smithi136 ceph-mon[115078]: pgmap v609: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2024-08-17T17:24:51.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:24:51.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:51 smithi165 ceph-mon[105517]: pgmap v609: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 5.5 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2024-08-17T17:24:51.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:24:53.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:53 smithi136 ceph-mon[109861]: pgmap v610: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 6.5 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-08-17T17:24:53.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:53 smithi136 ceph-mon[115078]: pgmap v610: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 6.5 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-08-17T17:24:53.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:53 smithi165 ceph-mon[105517]: pgmap v610: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 6.5 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2024-08-17T17:24:55.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:55 smithi136 ceph-mon[109861]: pgmap v611: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 2 op/s 2024-08-17T17:24:55.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:55 smithi136 ceph-mon[115078]: pgmap v611: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 2 op/s 2024-08-17T17:24:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:55 smithi165 ceph-mon[105517]: pgmap v611: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 2 op/s 2024-08-17T17:24:57.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:57 smithi136 ceph-mon[109861]: pgmap v612: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 2 op/s 2024-08-17T17:24:57.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:57 smithi136 ceph-mon[115078]: pgmap v612: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 2 op/s 2024-08-17T17:24:57.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:57 smithi165 ceph-mon[105517]: pgmap v612: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 2 op/s 2024-08-17T17:24:57.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:24:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:24:57] "GET /metrics HTTP/1.1" 200 67775 "" "Prometheus/2.51.0" 2024-08-17T17:24:59.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:24:59 smithi165 ceph-mon[105517]: pgmap v613: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 4.8 MiB/s rd, 2 op/s 2024-08-17T17:24:59.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:24:59 smithi136 ceph-mon[109861]: pgmap v613: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 4.8 MiB/s rd, 2 op/s 2024-08-17T17:24:59.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:24:59 smithi136 ceph-mon[115078]: pgmap v613: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 4.8 MiB/s rd, 2 op/s 2024-08-17T17:25:01.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:01 smithi165 ceph-mon[105517]: pgmap v614: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 1 op/s 2024-08-17T17:25:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:01 smithi136 ceph-mon[109861]: pgmap v614: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 1 op/s 2024-08-17T17:25:01.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:01 smithi136 ceph-mon[115078]: pgmap v614: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 1 op/s 2024-08-17T17:25:03.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:03 smithi165 ceph-mon[105517]: pgmap v615: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1 op/s 2024-08-17T17:25:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:03 smithi136 ceph-mon[109861]: pgmap v615: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1 op/s 2024-08-17T17:25:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:03 smithi136 ceph-mon[115078]: pgmap v615: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1 op/s 2024-08-17T17:25:05.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:05 smithi165 ceph-mon[105517]: pgmap v616: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 0 op/s 2024-08-17T17:25:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:05 smithi136 ceph-mon[109861]: pgmap v616: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 0 op/s 2024-08-17T17:25:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:05 smithi136 ceph-mon[115078]: pgmap v616: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 0 op/s 2024-08-17T17:25:06.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:25:06.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.4", "id": [6, 4]} : dispatch 2024-08-17T17:25:06.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.5", "id": [6, 5]} : dispatch 2024-08-17T17:25:06.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "29.1a", "id": [6, 0]} : dispatch 2024-08-17T17:25:06.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:25:06.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.4", "id": [6, 4]} : dispatch 2024-08-17T17:25:06.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.5", "id": [6, 5]} : dispatch 2024-08-17T17:25:06.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "29.1a", "id": [6, 0]} : dispatch 2024-08-17T17:25:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:25:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.4", "id": [6, 4]} : dispatch 2024-08-17T17:25:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.5", "id": [6, 5]} : dispatch 2024-08-17T17:25:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "29.1a", "id": [6, 0]} : dispatch 2024-08-17T17:25:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:07 smithi165 ceph-mon[105517]: pgmap v617: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 0 op/s 2024-08-17T17:25:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.4", "id": [6, 4]}]': finished 2024-08-17T17:25:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.5", "id": [6, 5]}]': finished 2024-08-17T17:25:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "29.1a", "id": [6, 0]}]': finished 2024-08-17T17:25:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:07 smithi165 ceph-mon[105517]: osdmap e690: 8 total, 8 up, 8 in 2024-08-17T17:25:07.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:07 smithi136 ceph-mon[109861]: pgmap v617: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 0 op/s 2024-08-17T17:25:07.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.4", "id": [6, 4]}]': finished 2024-08-17T17:25:07.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.5", "id": [6, 5]}]': finished 2024-08-17T17:25:07.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "29.1a", "id": [6, 0]}]': finished 2024-08-17T17:25:07.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:07 smithi136 ceph-mon[109861]: osdmap e690: 8 total, 8 up, 8 in 2024-08-17T17:25:07.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:25:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:25:07] "GET /metrics HTTP/1.1" 200 67778 "" "Prometheus/2.51.0" 2024-08-17T17:25:07.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:07 smithi136 ceph-mon[115078]: pgmap v617: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 0 op/s 2024-08-17T17:25:07.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.4", "id": [6, 4]}]': finished 2024-08-17T17:25:07.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.5", "id": [6, 5]}]': finished 2024-08-17T17:25:07.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "29.1a", "id": [6, 0]}]': finished 2024-08-17T17:25:07.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:07 smithi136 ceph-mon[115078]: osdmap e690: 8 total, 8 up, 8 in 2024-08-17T17:25:08.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:08 smithi165 ceph-mon[105517]: osdmap e691: 8 total, 8 up, 8 in 2024-08-17T17:25:08.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:08 smithi136 ceph-mon[109861]: osdmap e691: 8 total, 8 up, 8 in 2024-08-17T17:25:08.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:08 smithi136 ceph-mon[115078]: osdmap e691: 8 total, 8 up, 8 in 2024-08-17T17:25:09.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:09 smithi165 ceph-mon[105517]: pgmap v620: 521 pgs: 3 peering, 518 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1 op/s 2024-08-17T17:25:09.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:09 smithi165 ceph-mon[105517]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-08-17T17:25:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:09 smithi136 ceph-mon[109861]: pgmap v620: 521 pgs: 3 peering, 518 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1 op/s 2024-08-17T17:25:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:09 smithi136 ceph-mon[109861]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-08-17T17:25:09.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:09 smithi136 ceph-mon[115078]: pgmap v620: 521 pgs: 3 peering, 518 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1 op/s 2024-08-17T17:25:09.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:09 smithi136 ceph-mon[115078]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-08-17T17:25:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:11 smithi165 ceph-mon[105517]: pgmap v621: 521 pgs: 3 peering, 518 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 0 op/s 2024-08-17T17:25:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:11 smithi136 ceph-mon[109861]: pgmap v621: 521 pgs: 3 peering, 518 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 0 op/s 2024-08-17T17:25:11.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:11 smithi136 ceph-mon[115078]: pgmap v621: 521 pgs: 3 peering, 518 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 0 op/s 2024-08-17T17:25:12.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:12 smithi136 ceph-mon[109861]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-08-17T17:25:12.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:12 smithi136 ceph-mon[115078]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-08-17T17:25:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:12 smithi165 ceph-mon[105517]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-08-17T17:25:13.597 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:13 smithi136 ceph-mon[109861]: pgmap v622: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1 op/s; 4 B/s, 0 objects/s recovering 2024-08-17T17:25:13.598 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:13 smithi136 ceph-mon[115078]: pgmap v622: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1 op/s; 4 B/s, 0 objects/s recovering 2024-08-17T17:25:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:13 smithi165 ceph-mon[105517]: pgmap v622: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1 op/s; 4 B/s, 0 objects/s recovering 2024-08-17T17:25:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:15 smithi165 ceph-mon[105517]: pgmap v623: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1 op/s; 4 B/s, 0 objects/s recovering 2024-08-17T17:25:15.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:25:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=sqlstore.transactions t=2024-08-17T17:25:15.048823687Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-08-17T17:25:15.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:25:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=cleanup t=2024-08-17T17:25:15.049722214Z level=info msg="Completed cleanup jobs" duration=12.398657ms 2024-08-17T17:25:15.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:25:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=sqlstore.transactions t=2024-08-17T17:25:15.060212302Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-08-17T17:25:15.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:25:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=plugins.update.checker t=2024-08-17T17:25:15.19075803Z level=info msg="Update check succeeded" duration=69.142605ms 2024-08-17T17:25:15.512 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:25:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=grafana.update.checker t=2024-08-17T17:25:15.201458818Z level=info msg="Update check succeeded" duration=70.351298ms 2024-08-17T17:25:15.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:15 smithi136 ceph-mon[109861]: pgmap v623: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1 op/s; 4 B/s, 0 objects/s recovering 2024-08-17T17:25:15.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:15 smithi136 ceph-mon[115078]: pgmap v623: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 1 op/s; 4 B/s, 0 objects/s recovering 2024-08-17T17:25:17.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:17 smithi136 ceph-mon[109861]: pgmap v624: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 1 op/s; 3 B/s, 0 objects/s recovering 2024-08-17T17:25:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:17 smithi136 ceph-mon[115078]: pgmap v624: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 1 op/s; 3 B/s, 0 objects/s recovering 2024-08-17T17:25:17.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:25:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:25:17] "GET /metrics HTTP/1.1" 200 67778 "" "Prometheus/2.51.0" 2024-08-17T17:25:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:17 smithi165 ceph-mon[105517]: pgmap v624: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 1 op/s; 3 B/s, 0 objects/s recovering 2024-08-17T17:25:19.579 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:19 smithi165 ceph-mon[105517]: pgmap v625: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 892 KiB/s rd, 0 op/s; 3 B/s, 0 objects/s recovering 2024-08-17T17:25:19.579 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:19 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:25:19.579 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:19 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:25:19.579 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:19 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:25:19.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:19 smithi136 ceph-mon[109861]: pgmap v625: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 892 KiB/s rd, 0 op/s; 3 B/s, 0 objects/s recovering 2024-08-17T17:25:19.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:19 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:25:19.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:19 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:25:19.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:19 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:25:19.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:19 smithi136 ceph-mon[115078]: pgmap v625: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 892 KiB/s rd, 0 op/s; 3 B/s, 0 objects/s recovering 2024-08-17T17:25:19.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:19 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:25:19.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:19 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:25:19.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:19 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:25:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:21 smithi136 ceph-mon[109861]: pgmap v626: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 802 KiB/s rd, 0 op/s; 2 B/s, 0 objects/s recovering 2024-08-17T17:25:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:25:21.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:21 smithi136 ceph-mon[115078]: pgmap v626: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 802 KiB/s rd, 0 op/s; 2 B/s, 0 objects/s recovering 2024-08-17T17:25:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:25:21.756 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:21 smithi165 ceph-mon[105517]: pgmap v626: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 802 KiB/s rd, 0 op/s; 2 B/s, 0 objects/s recovering 2024-08-17T17:25:21.756 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:25:23.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:23 smithi136 ceph-mon[109861]: pgmap v627: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 881 KiB/s rd, 8 op/s; 2 B/s, 0 objects/s recovering 2024-08-17T17:25:23.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:23 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:25:23.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:23 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:25:23.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:23 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:25:23.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:23 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:25:23.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:23 smithi136 ceph-mon[115078]: pgmap v627: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 881 KiB/s rd, 8 op/s; 2 B/s, 0 objects/s recovering 2024-08-17T17:25:23.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:23 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:25:23.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:23 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:25:23.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:23 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:25:23.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:23 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:25:23.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:23 smithi165 ceph-mon[105517]: pgmap v627: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 881 KiB/s rd, 8 op/s; 2 B/s, 0 objects/s recovering 2024-08-17T17:25:23.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:23 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:25:23.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:23 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:25:23.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:23 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:25:23.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:23 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:25:24.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:24 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:25:24.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:24 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:25:24.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:24 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:25:24.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:24 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:25:24.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:24 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:25:24.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:24 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:25:24.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:24 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:25:24.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:24 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:25:24.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:24 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:25:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:25 smithi136 ceph-mon[109861]: pgmap v628: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 127 KiB/s rd, 8 op/s 2024-08-17T17:25:25.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:25 smithi136 ceph-mon[115078]: pgmap v628: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 127 KiB/s rd, 8 op/s 2024-08-17T17:25:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:25 smithi165 ceph-mon[105517]: pgmap v628: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 127 KiB/s rd, 8 op/s 2024-08-17T17:25:27.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:27 smithi136 ceph-mon[109861]: pgmap v629: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 129 KiB/s rd, 11 op/s 2024-08-17T17:25:27.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:27 smithi136 ceph-mon[115078]: pgmap v629: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 129 KiB/s rd, 11 op/s 2024-08-17T17:25:27.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:27 smithi165 ceph-mon[105517]: pgmap v629: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 129 KiB/s rd, 11 op/s 2024-08-17T17:25:27.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:25:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:25:27] "GET /metrics HTTP/1.1" 200 67785 "" "Prometheus/2.51.0" 2024-08-17T17:25:29.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:29 smithi136 ceph-mon[109861]: pgmap v630: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 182 KiB/s rd, 16 op/s 2024-08-17T17:25:29.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:29 smithi136 ceph-mon[115078]: pgmap v630: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 182 KiB/s rd, 16 op/s 2024-08-17T17:25:29.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:29 smithi165 ceph-mon[105517]: pgmap v630: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 182 KiB/s rd, 16 op/s 2024-08-17T17:25:31.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:31 smithi136 ceph-mon[109861]: pgmap v631: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-08-17T17:25:31.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:31 smithi136 ceph-mon[115078]: pgmap v631: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-08-17T17:25:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:31 smithi165 ceph-mon[105517]: pgmap v631: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-08-17T17:25:33.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:33 smithi136 ceph-mon[109861]: pgmap v632: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 212 KiB/s rd, 85 B/s wr, 24 op/s 2024-08-17T17:25:33.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:33 smithi136 ceph-mon[115078]: pgmap v632: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 212 KiB/s rd, 85 B/s wr, 24 op/s 2024-08-17T17:25:33.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:33 smithi165 ceph-mon[105517]: pgmap v632: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 212 KiB/s rd, 85 B/s wr, 24 op/s 2024-08-17T17:25:35.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:35 smithi165 ceph-mon[105517]: pgmap v633: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 85 B/s wr, 16 op/s 2024-08-17T17:25:35.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:35 smithi136 ceph-mon[109861]: pgmap v633: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 85 B/s wr, 16 op/s 2024-08-17T17:25:35.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:35 smithi136 ceph-mon[115078]: pgmap v633: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 85 B/s wr, 16 op/s 2024-08-17T17:25:36.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:25:36.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:25:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:25:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:37 smithi165 ceph-mon[105517]: pgmap v634: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 85 B/s wr, 16 op/s 2024-08-17T17:25:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:37 smithi136 ceph-mon[109861]: pgmap v634: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 85 B/s wr, 16 op/s 2024-08-17T17:25:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:37 smithi136 ceph-mon[115078]: pgmap v634: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 85 B/s wr, 16 op/s 2024-08-17T17:25:37.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:25:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:25:37] "GET /metrics HTTP/1.1" 200 67781 "" "Prometheus/2.51.0" 2024-08-17T17:25:39.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:39 smithi165 ceph-mon[105517]: pgmap v635: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 183 KiB/s rd, 170 B/s wr, 18 op/s 2024-08-17T17:25:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:39 smithi136 ceph-mon[109861]: pgmap v635: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 183 KiB/s rd, 170 B/s wr, 18 op/s 2024-08-17T17:25:39.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:39 smithi136 ceph-mon[115078]: pgmap v635: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 183 KiB/s rd, 170 B/s wr, 18 op/s 2024-08-17T17:25:41.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:41 smithi165 ceph-mon[105517]: pgmap v636: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 131 KiB/s rd, 170 B/s wr, 13 op/s 2024-08-17T17:25:41.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:41 smithi136 ceph-mon[109861]: pgmap v636: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 131 KiB/s rd, 170 B/s wr, 13 op/s 2024-08-17T17:25:41.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:41 smithi136 ceph-mon[115078]: pgmap v636: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 131 KiB/s rd, 170 B/s wr, 13 op/s 2024-08-17T17:25:43.498 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:43 smithi136 ceph-mon[109861]: pgmap v637: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 184 KiB/s rd, 255 B/s wr, 19 op/s 2024-08-17T17:25:43.498 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:43 smithi136 ceph-mon[115078]: pgmap v637: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 184 KiB/s rd, 255 B/s wr, 19 op/s 2024-08-17T17:25:43.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:43 smithi165 ceph-mon[105517]: pgmap v637: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 184 KiB/s rd, 255 B/s wr, 19 op/s 2024-08-17T17:25:45.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:45 smithi165 ceph-mon[105517]: pgmap v638: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 105 KiB/s rd, 170 B/s wr, 10 op/s 2024-08-17T17:25:45.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:45 smithi136 ceph-mon[109861]: pgmap v638: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 105 KiB/s rd, 170 B/s wr, 10 op/s 2024-08-17T17:25:45.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:45 smithi136 ceph-mon[115078]: pgmap v638: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 105 KiB/s rd, 170 B/s wr, 10 op/s 2024-08-17T17:25:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:47 smithi165 ceph-mon[105517]: pgmap v639: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 107 KiB/s rd, 170 B/s wr, 14 op/s 2024-08-17T17:25:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:47 smithi136 ceph-mon[109861]: pgmap v639: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 107 KiB/s rd, 170 B/s wr, 14 op/s 2024-08-17T17:25:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:47 smithi136 ceph-mon[115078]: pgmap v639: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 107 KiB/s rd, 170 B/s wr, 14 op/s 2024-08-17T17:25:47.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:25:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:25:47] "GET /metrics HTTP/1.1" 200 67781 "" "Prometheus/2.51.0" 2024-08-17T17:25:49.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:49 smithi165 ceph-mon[105517]: pgmap v640: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 182 KiB/s rd, 170 B/s wr, 16 op/s 2024-08-17T17:25:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:49 smithi136 ceph-mon[109861]: pgmap v640: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 182 KiB/s rd, 170 B/s wr, 16 op/s 2024-08-17T17:25:49.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:49 smithi136 ceph-mon[115078]: pgmap v640: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 182 KiB/s rd, 170 B/s wr, 16 op/s 2024-08-17T17:25:51.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:51 smithi165 ceph-mon[105517]: pgmap v641: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 129 KiB/s rd, 85 B/s wr, 11 op/s 2024-08-17T17:25:51.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:25:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:51 smithi136 ceph-mon[109861]: pgmap v641: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 129 KiB/s rd, 85 B/s wr, 11 op/s 2024-08-17T17:25:51.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:25:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:51 smithi136 ceph-mon[115078]: pgmap v641: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 129 KiB/s rd, 85 B/s wr, 11 op/s 2024-08-17T17:25:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:25:53.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:53 smithi165 ceph-mon[105517]: pgmap v642: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 184 KiB/s rd, 85 B/s wr, 19 op/s 2024-08-17T17:25:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:53 smithi136 ceph-mon[109861]: pgmap v642: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 184 KiB/s rd, 85 B/s wr, 19 op/s 2024-08-17T17:25:53.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:53 smithi136 ceph-mon[115078]: pgmap v642: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 184 KiB/s rd, 85 B/s wr, 19 op/s 2024-08-17T17:25:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:55 smithi165 ceph-mon[105517]: pgmap v643: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 132 KiB/s rd, 14 op/s 2024-08-17T17:25:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:55 smithi136 ceph-mon[109861]: pgmap v643: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 132 KiB/s rd, 14 op/s 2024-08-17T17:25:55.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:55 smithi136 ceph-mon[115078]: pgmap v643: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 132 KiB/s rd, 14 op/s 2024-08-17T17:25:57.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:57 smithi165 ceph-mon[105517]: pgmap v644: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 17 op/s 2024-08-17T17:25:57.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:57 smithi136 ceph-mon[109861]: pgmap v644: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 17 op/s 2024-08-17T17:25:57.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:57 smithi136 ceph-mon[115078]: pgmap v644: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 17 op/s 2024-08-17T17:25:57.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:25:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:25:57] "GET /metrics HTTP/1.1" 200 67784 "" "Prometheus/2.51.0" 2024-08-17T17:25:59.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:25:59 smithi165 ceph-mon[105517]: pgmap v645: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 208 KiB/s rd, 19 op/s 2024-08-17T17:25:59.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:25:59 smithi136 ceph-mon[109861]: pgmap v645: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 208 KiB/s rd, 19 op/s 2024-08-17T17:25:59.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:25:59 smithi136 ceph-mon[115078]: pgmap v645: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 208 KiB/s rd, 19 op/s 2024-08-17T17:26:01.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:01 smithi165 ceph-mon[105517]: pgmap v646: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-08-17T17:26:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:01 smithi136 ceph-mon[109861]: pgmap v646: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-08-17T17:26:01.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:01 smithi136 ceph-mon[115078]: pgmap v646: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-08-17T17:26:03.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:03 smithi165 ceph-mon[105517]: pgmap v647: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 188 KiB/s rd, 24 op/s 2024-08-17T17:26:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:03 smithi136 ceph-mon[109861]: pgmap v647: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 188 KiB/s rd, 24 op/s 2024-08-17T17:26:03.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:03 smithi136 ceph-mon[115078]: pgmap v647: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 188 KiB/s rd, 24 op/s 2024-08-17T17:26:05.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:05 smithi165 ceph-mon[105517]: pgmap v648: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-08-17T17:26:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:05 smithi136 ceph-mon[109861]: pgmap v648: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-08-17T17:26:05.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:05 smithi136 ceph-mon[115078]: pgmap v648: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-08-17T17:26:06.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:26:06.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:26:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:26:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:07 smithi165 ceph-mon[105517]: pgmap v649: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-08-17T17:26:07.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:07 smithi136 ceph-mon[109861]: pgmap v649: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-08-17T17:26:07.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:07 smithi136 ceph-mon[115078]: pgmap v649: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-08-17T17:26:07.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:26:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:26:07] "GET /metrics HTTP/1.1" 200 67779 "" "Prometheus/2.51.0" 2024-08-17T17:26:09.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:09 smithi165 ceph-mon[105517]: pgmap v650: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 207 KiB/s rd, 18 op/s 2024-08-17T17:26:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:09 smithi136 ceph-mon[109861]: pgmap v650: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 207 KiB/s rd, 18 op/s 2024-08-17T17:26:09.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:09 smithi136 ceph-mon[115078]: pgmap v650: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 207 KiB/s rd, 18 op/s 2024-08-17T17:26:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:11 smithi136 ceph-mon[109861]: pgmap v651: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 131 KiB/s rd, 13 op/s 2024-08-17T17:26:11.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:11 smithi136 ceph-mon[115078]: pgmap v651: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 131 KiB/s rd, 13 op/s 2024-08-17T17:26:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:11 smithi165 ceph-mon[105517]: pgmap v651: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 131 KiB/s rd, 13 op/s 2024-08-17T17:26:13.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:13 smithi136 ceph-mon[115078]: pgmap v652: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 182 KiB/s rd, 16 op/s 2024-08-17T17:26:13.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:13 smithi136 ceph-mon[109861]: pgmap v652: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 182 KiB/s rd, 16 op/s 2024-08-17T17:26:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:13 smithi165 ceph-mon[105517]: pgmap v652: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 182 KiB/s rd, 16 op/s 2024-08-17T17:26:15.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:15 smithi136 ceph-mon[109861]: pgmap v653: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 127 KiB/s rd, 8 op/s 2024-08-17T17:26:15.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:15 smithi136 ceph-mon[115078]: pgmap v653: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 127 KiB/s rd, 8 op/s 2024-08-17T17:26:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:15 smithi165 ceph-mon[105517]: pgmap v653: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 127 KiB/s rd, 8 op/s 2024-08-17T17:26:17.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:17 smithi136 ceph-mon[109861]: pgmap v654: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 129 KiB/s rd, 11 op/s 2024-08-17T17:26:17.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:26:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:26:17] "GET /metrics HTTP/1.1" 200 67779 "" "Prometheus/2.51.0" 2024-08-17T17:26:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:17 smithi136 ceph-mon[115078]: pgmap v654: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 129 KiB/s rd, 11 op/s 2024-08-17T17:26:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:17 smithi165 ceph-mon[105517]: pgmap v654: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 129 KiB/s rd, 11 op/s 2024-08-17T17:26:19.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:19 smithi136 ceph-mon[109861]: pgmap v655: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 206 KiB/s rd, 16 op/s 2024-08-17T17:26:19.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:19 smithi136 ceph-mon[115078]: pgmap v655: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 206 KiB/s rd, 16 op/s 2024-08-17T17:26:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:19 smithi165 ceph-mon[105517]: pgmap v655: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 206 KiB/s rd, 16 op/s 2024-08-17T17:26:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:21 smithi136 ceph-mon[109861]: pgmap v656: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 129 KiB/s rd, 11 op/s 2024-08-17T17:26:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:26:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:21 smithi136 ceph-mon[115078]: pgmap v656: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 129 KiB/s rd, 11 op/s 2024-08-17T17:26:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:26:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:21 smithi165 ceph-mon[105517]: pgmap v656: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 129 KiB/s rd, 11 op/s 2024-08-17T17:26:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:26:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:23 smithi136 ceph-mon[109861]: pgmap v657: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 184 KiB/s rd, 19 op/s 2024-08-17T17:26:23.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:23 smithi136 ceph-mon[115078]: pgmap v657: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 184 KiB/s rd, 19 op/s 2024-08-17T17:26:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:23 smithi165 ceph-mon[105517]: pgmap v657: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 184 KiB/s rd, 19 op/s 2024-08-17T17:26:24.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:24 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:26:24.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:24 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:26:24.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:24 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:26:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:25 smithi136 ceph-mon[109861]: pgmap v658: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-08-17T17:26:25.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:25 smithi136 ceph-mon[115078]: pgmap v658: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-08-17T17:26:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:25 smithi165 ceph-mon[105517]: pgmap v658: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-08-17T17:26:27.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:27 smithi136 ceph-mon[109861]: pgmap v659: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 136 KiB/s rd, 19 op/s 2024-08-17T17:26:27.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:27 smithi136 ceph-mon[115078]: pgmap v659: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 136 KiB/s rd, 19 op/s 2024-08-17T17:26:27.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:26:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:26:27] "GET /metrics HTTP/1.1" 200 67778 "" "Prometheus/2.51.0" 2024-08-17T17:26:27.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:27 smithi165 ceph-mon[105517]: pgmap v659: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 136 KiB/s rd, 19 op/s 2024-08-17T17:26:28.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:28 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:26:28.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:28 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:26:28.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:28 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:26:28.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:28 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:26:28.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:28 smithi165 ceph-mon[105517]: pgmap v660: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 210 KiB/s rd, 21 op/s 2024-08-17T17:26:28.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:28 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:26:28.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:28 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:26:28.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:28 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:26:28.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:28 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:26:28.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:28 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:26:28.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:28 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:26:28.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:28 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:26:28.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:28 smithi136 ceph-mon[115078]: pgmap v660: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 210 KiB/s rd, 21 op/s 2024-08-17T17:26:28.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:28 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:26:28.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:28 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:26:28.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:28 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:26:28.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:28 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:26:28.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:28 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:26:28.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:28 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:26:28.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:28 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:26:28.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:28 smithi136 ceph-mon[109861]: pgmap v660: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 210 KiB/s rd, 21 op/s 2024-08-17T17:26:28.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:28 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:26:28.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:28 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:26:28.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:28 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:26:31.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:31 smithi136 ceph-mon[109861]: pgmap v661: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-08-17T17:26:31.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:31 smithi136 ceph-mon[115078]: pgmap v661: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-08-17T17:26:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:31 smithi165 ceph-mon[105517]: pgmap v661: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-08-17T17:26:33.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:33 smithi136 ceph-mon[109861]: pgmap v662: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 186 KiB/s rd, 21 op/s 2024-08-17T17:26:33.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:33 smithi136 ceph-mon[115078]: pgmap v662: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 186 KiB/s rd, 21 op/s 2024-08-17T17:26:33.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:33 smithi165 ceph-mon[105517]: pgmap v662: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 186 KiB/s rd, 21 op/s 2024-08-17T17:26:35.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:35 smithi136 ceph-mon[109861]: pgmap v663: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 131 KiB/s rd, 13 op/s 2024-08-17T17:26:35.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:35 smithi136 ceph-mon[115078]: pgmap v663: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 131 KiB/s rd, 13 op/s 2024-08-17T17:26:35.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:35 smithi165 ceph-mon[105517]: pgmap v663: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 131 KiB/s rd, 13 op/s 2024-08-17T17:26:37.244 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.EncryptionLoadFormatMismatch (130041 ms) 2024-08-17T17:26:37.244 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.EncryptedResize 2024-08-17T17:26:37.244 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:26:37.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:37 smithi136 ceph-mon[109861]: pgmap v664: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-08-17T17:26:37.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:26:37.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:37 smithi136 ceph-mon[115078]: pgmap v664: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-08-17T17:26:37.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:26:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:37 smithi165 ceph-mon[105517]: pgmap v664: 521 pgs: 521 active+clean; 149 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 16 op/s 2024-08-17T17:26:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:26:37.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:26:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:26:37] "GET /metrics HTTP/1.1" 200 67780 "" "Prometheus/2.51.0" 2024-08-17T17:26:39.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:39 smithi136 ceph-mon[115078]: pgmap v665: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 179 KiB/s rd, 85 B/s wr, 13 op/s 2024-08-17T17:26:39.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:39 smithi136 ceph-mon[109861]: pgmap v665: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 179 KiB/s rd, 85 B/s wr, 13 op/s 2024-08-17T17:26:39.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:39 smithi165 ceph-mon[105517]: pgmap v665: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 179 KiB/s rd, 85 B/s wr, 13 op/s 2024-08-17T17:26:41.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:41 smithi136 ceph-mon[109861]: pgmap v666: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 103 KiB/s rd, 85 B/s wr, 8 op/s 2024-08-17T17:26:41.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:41 smithi136 ceph-mon[115078]: pgmap v666: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 103 KiB/s rd, 85 B/s wr, 8 op/s 2024-08-17T17:26:41.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:41 smithi165 ceph-mon[105517]: pgmap v666: 521 pgs: 521 active+clean; 149 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 103 KiB/s rd, 85 B/s wr, 8 op/s 2024-08-17T17:26:43.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:43 smithi136 ceph-mon[109861]: pgmap v667: 521 pgs: 521 active+clean; 149 MiB data, 839 MiB used, 714 GiB / 715 GiB avail; 105 KiB/s rd, 426 B/s wr, 11 op/s 2024-08-17T17:26:43.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:43 smithi136 ceph-mon[115078]: pgmap v667: 521 pgs: 521 active+clean; 149 MiB data, 839 MiB used, 714 GiB / 715 GiB avail; 105 KiB/s rd, 426 B/s wr, 11 op/s 2024-08-17T17:26:43.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:43 smithi165 ceph-mon[105517]: pgmap v667: 521 pgs: 521 active+clean; 149 MiB data, 839 MiB used, 714 GiB / 715 GiB avail; 105 KiB/s rd, 426 B/s wr, 11 op/s 2024-08-17T17:26:45.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:45 smithi136 ceph-mon[109861]: pgmap v668: 521 pgs: 521 active+clean; 149 MiB data, 839 MiB used, 714 GiB / 715 GiB avail; 53 KiB/s rd, 426 B/s wr, 6 op/s 2024-08-17T17:26:45.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:45 smithi136 ceph-mon[115078]: pgmap v668: 521 pgs: 521 active+clean; 149 MiB data, 839 MiB used, 714 GiB / 715 GiB avail; 53 KiB/s rd, 426 B/s wr, 6 op/s 2024-08-17T17:26:45.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:45 smithi165 ceph-mon[105517]: pgmap v668: 521 pgs: 521 active+clean; 149 MiB data, 839 MiB used, 714 GiB / 715 GiB avail; 53 KiB/s rd, 426 B/s wr, 6 op/s 2024-08-17T17:26:47.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:47 smithi136 ceph-mon[109861]: pgmap v669: 521 pgs: 521 active+clean; 157 MiB data, 874 MiB used, 714 GiB / 715 GiB avail; 56 KiB/s rd, 598 KiB/s wr, 12 op/s 2024-08-17T17:26:47.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:47 smithi136 ceph-mon[115078]: pgmap v669: 521 pgs: 521 active+clean; 157 MiB data, 874 MiB used, 714 GiB / 715 GiB avail; 56 KiB/s rd, 598 KiB/s wr, 12 op/s 2024-08-17T17:26:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:47 smithi165 ceph-mon[105517]: pgmap v669: 521 pgs: 521 active+clean; 157 MiB data, 874 MiB used, 714 GiB / 715 GiB avail; 56 KiB/s rd, 598 KiB/s wr, 12 op/s 2024-08-17T17:26:47.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:26:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:26:47] "GET /metrics HTTP/1.1" 200 67780 "" "Prometheus/2.51.0" 2024-08-17T17:26:48.862 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.EncryptedResize (11619 ms) 2024-08-17T17:26:48.863 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.EncryptedFlattenSmallData 2024-08-17T17:26:48.863 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:26:49.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:49 smithi136 ceph-mon[109861]: pgmap v670: 521 pgs: 521 active+clean; 169 MiB data, 896 MiB used, 714 GiB / 715 GiB avail; 136 KiB/s rd, 1.6 MiB/s wr, 22 op/s 2024-08-17T17:26:49.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:49 smithi136 ceph-mon[115078]: pgmap v670: 521 pgs: 521 active+clean; 169 MiB data, 896 MiB used, 714 GiB / 715 GiB avail; 136 KiB/s rd, 1.6 MiB/s wr, 22 op/s 2024-08-17T17:26:49.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:49 smithi165 ceph-mon[105517]: pgmap v670: 521 pgs: 521 active+clean; 169 MiB data, 896 MiB used, 714 GiB / 715 GiB avail; 136 KiB/s rd, 1.6 MiB/s wr, 22 op/s 2024-08-17T17:26:51.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:51 smithi136 ceph-mon[109861]: pgmap v671: 521 pgs: 521 active+clean; 169 MiB data, 896 MiB used, 714 GiB / 715 GiB avail; 88 KiB/s rd, 1.6 MiB/s wr, 21 op/s 2024-08-17T17:26:51.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:51 smithi136 ceph-mon[115078]: pgmap v671: 521 pgs: 521 active+clean; 169 MiB data, 896 MiB used, 714 GiB / 715 GiB avail; 88 KiB/s rd, 1.6 MiB/s wr, 21 op/s 2024-08-17T17:26:51.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:51 smithi165 ceph-mon[105517]: pgmap v671: 521 pgs: 521 active+clean; 169 MiB data, 896 MiB used, 714 GiB / 715 GiB avail; 88 KiB/s rd, 1.6 MiB/s wr, 21 op/s 2024-08-17T17:26:52.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:26:52.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:26:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:26:53.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:53 smithi136 ceph-mon[109861]: pgmap v672: 521 pgs: 521 active+clean; 165 MiB data, 895 MiB used, 714 GiB / 715 GiB avail; 95 KiB/s rd, 1.6 MiB/s wr, 32 op/s 2024-08-17T17:26:53.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:53 smithi136 ceph-mon[115078]: pgmap v672: 521 pgs: 521 active+clean; 165 MiB data, 895 MiB used, 714 GiB / 715 GiB avail; 95 KiB/s rd, 1.6 MiB/s wr, 32 op/s 2024-08-17T17:26:53.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:53 smithi165 ceph-mon[105517]: pgmap v672: 521 pgs: 521 active+clean; 165 MiB data, 895 MiB used, 714 GiB / 715 GiB avail; 95 KiB/s rd, 1.6 MiB/s wr, 32 op/s 2024-08-17T17:26:55.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:55 smithi136 ceph-mon[109861]: pgmap v673: 521 pgs: 521 active+clean; 165 MiB data, 895 MiB used, 714 GiB / 715 GiB avail; 93 KiB/s rd, 1.6 MiB/s wr, 29 op/s 2024-08-17T17:26:55.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:55 smithi136 ceph-mon[115078]: pgmap v673: 521 pgs: 521 active+clean; 165 MiB data, 895 MiB used, 714 GiB / 715 GiB avail; 93 KiB/s rd, 1.6 MiB/s wr, 29 op/s 2024-08-17T17:26:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:55 smithi165 ceph-mon[105517]: pgmap v673: 521 pgs: 521 active+clean; 165 MiB data, 895 MiB used, 714 GiB / 715 GiB avail; 93 KiB/s rd, 1.6 MiB/s wr, 29 op/s 2024-08-17T17:26:56.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:56 smithi136 ceph-mon[109861]: osdmap e692: 8 total, 8 up, 8 in 2024-08-17T17:26:56.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:56 smithi136 ceph-mon[115078]: osdmap e692: 8 total, 8 up, 8 in 2024-08-17T17:26:56.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:56 smithi165 ceph-mon[105517]: osdmap e692: 8 total, 8 up, 8 in 2024-08-17T17:26:57.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:57 smithi136 ceph-mon[109861]: pgmap v675: 521 pgs: 521 active+clean; 177 MiB data, 927 MiB used, 714 GiB / 715 GiB avail; 138 KiB/s rd, 2.4 MiB/s wr, 30 op/s 2024-08-17T17:26:57.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:57 smithi136 ceph-mon[115078]: pgmap v675: 521 pgs: 521 active+clean; 177 MiB data, 927 MiB used, 714 GiB / 715 GiB avail; 138 KiB/s rd, 2.4 MiB/s wr, 30 op/s 2024-08-17T17:26:57.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:57 smithi165 ceph-mon[105517]: pgmap v675: 521 pgs: 521 active+clean; 177 MiB data, 927 MiB used, 714 GiB / 715 GiB avail; 138 KiB/s rd, 2.4 MiB/s wr, 30 op/s 2024-08-17T17:26:57.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:26:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:26:57] "GET /metrics HTTP/1.1" 200 67777 "" "Prometheus/2.51.0" 2024-08-17T17:26:59.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:26:59 smithi136 ceph-mon[109861]: pgmap v676: 521 pgs: 521 active+clean; 181 MiB data, 940 MiB used, 714 GiB / 715 GiB avail; 60 KiB/s rd, 1.6 MiB/s wr, 42 op/s 2024-08-17T17:26:59.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:26:59 smithi136 ceph-mon[115078]: pgmap v676: 521 pgs: 521 active+clean; 181 MiB data, 940 MiB used, 714 GiB / 715 GiB avail; 60 KiB/s rd, 1.6 MiB/s wr, 42 op/s 2024-08-17T17:26:59.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:26:59 smithi165 ceph-mon[105517]: pgmap v676: 521 pgs: 521 active+clean; 181 MiB data, 940 MiB used, 714 GiB / 715 GiB avail; 60 KiB/s rd, 1.6 MiB/s wr, 42 op/s 2024-08-17T17:27:01.045 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.EncryptedFlattenSmallData (12183 ms) 2024-08-17T17:27:01.045 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestIOWithIOHint 2024-08-17T17:27:01.045 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:01.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:01 smithi136 ceph-mon[109861]: pgmap v677: 521 pgs: 521 active+clean; 181 MiB data, 940 MiB used, 714 GiB / 715 GiB avail; 60 KiB/s rd, 1.6 MiB/s wr, 42 op/s 2024-08-17T17:27:01.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:01 smithi136 ceph-mon[115078]: pgmap v677: 521 pgs: 521 active+clean; 181 MiB data, 940 MiB used, 714 GiB / 715 GiB avail; 60 KiB/s rd, 1.6 MiB/s wr, 42 op/s 2024-08-17T17:27:01.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:01 smithi165 ceph-mon[105517]: pgmap v677: 521 pgs: 521 active+clean; 181 MiB data, 940 MiB used, 714 GiB / 715 GiB avail; 60 KiB/s rd, 1.6 MiB/s wr, 42 op/s 2024-08-17T17:27:02.849 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:02.849 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:02.849 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:02.849 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:02.849 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:02.849 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:02.849 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:02.849 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:02.849 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:02.850 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:02.850 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:02.850 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:02.850 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:02.850 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:02.850 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:02.850 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:02.850 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:02.850 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:02.850 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:02.850 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:02.850 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:02.851 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:02.851 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:02.851 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:02.851 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:02.851 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:02.851 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:02.851 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:02.851 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:02.851 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:02.851 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:27:02.851 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:27:02.851 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:27:02.852 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:27:02.852 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:27:02.852 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:27:02.852 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:27:02.852 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:27:02.852 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:27:02.852 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:27:02.852 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:02.852 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:27:02.852 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:02.852 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:02.852 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:27:02.852 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:02.853 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:27:02.853 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:02.853 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:02.853 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:27:02.853 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:02.853 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:27:02.853 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:02.853 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:02.853 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:27:02.853 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:02.853 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:27:02.853 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:02.854 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:02.854 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:27:02.854 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:02.854 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:27:02.854 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:02.854 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:02.854 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:27:02.854 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:02.854 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:02.854 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:02.854 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:02.854 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:02.854 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:02.855 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:02.855 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:02.855 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:02.855 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:02.855 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:02.855 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:02.855 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:02.855 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:02.855 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:02.855 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:02.855 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:02.855 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:02.855 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:02.856 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:02.856 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:02.856 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:02.856 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:02.856 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:02.856 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:02.856 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 512~512 = 512 2024-08-17T17:27:02.856 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:02.856 INFO:tasks.workunit.client.0.smithi136.stdout:aio discard: 1536~512 = 0 2024-08-17T17:27:02.856 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:02.856 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:02.856 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:02.856 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:02.857 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:02.857 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 0 2024-08-17T17:27:02.857 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.857 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.857 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 0 2024-08-17T17:27:02.857 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.857 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.857 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 16384 2024-08-17T17:27:02.857 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.857 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.857 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 16384 2024-08-17T17:27:02.857 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.858 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.858 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:02.858 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:02.858 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 49152 2024-08-17T17:27:02.858 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.858 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.858 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 49152 2024-08-17T17:27:02.858 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.858 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.858 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 65536 2024-08-17T17:27:02.858 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.858 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.858 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 65536 2024-08-17T17:27:02.859 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.859 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.859 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:02.859 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:02.859 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 98304 2024-08-17T17:27:02.859 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.859 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.859 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 98304 2024-08-17T17:27:02.859 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.859 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.859 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 114688 2024-08-17T17:27:02.859 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.859 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.860 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 114688 2024-08-17T17:27:02.860 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.860 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.860 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:02.860 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:02.860 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 147456 2024-08-17T17:27:02.860 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.860 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.860 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 147456 2024-08-17T17:27:02.860 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.860 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.860 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 163840 2024-08-17T17:27:02.860 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.860 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.861 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 163840 2024-08-17T17:27:02.861 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.861 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.861 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:02.861 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:02.861 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 196608 2024-08-17T17:27:02.861 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.861 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.861 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 196608 2024-08-17T17:27:02.861 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.861 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.861 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 212992 2024-08-17T17:27:02.861 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.862 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.862 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 212992 2024-08-17T17:27:02.862 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.862 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.862 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:02.862 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:02.862 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:02.862 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:02.862 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:02.862 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:02.862 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:02.862 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.862 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.863 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:02.863 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:02.863 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:02.863 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:02.863 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:02.863 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.863 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.863 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:02.863 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:02.863 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:02.863 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:02.863 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:02.863 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.863 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.864 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:02.864 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:02.864 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:02.864 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:02.864 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:02.864 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.864 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.864 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:02.864 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:02.864 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:02.864 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:02.864 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:02.864 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:02.865 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:02.865 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:02.865 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:02.865 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:02.865 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:02.865 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.865 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.865 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:02.865 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:02.865 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:02.865 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:02.865 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:02.865 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:02.865 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:02.865 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:03.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:03 smithi136 ceph-mon[109861]: pgmap v678: 521 pgs: 521 active+clean; 197 MiB data, 988 MiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 261 op/s 2024-08-17T17:27:03.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:03 smithi136 ceph-mon[115078]: pgmap v678: 521 pgs: 521 active+clean; 197 MiB data, 988 MiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 261 op/s 2024-08-17T17:27:03.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:03 smithi165 ceph-mon[105517]: pgmap v678: 521 pgs: 521 active+clean; 197 MiB data, 988 MiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 261 op/s 2024-08-17T17:27:04.404 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:04.404 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:04.404 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:04.404 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:04.404 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:04.404 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:04.404 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:04.405 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:04.405 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:04.405 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:04.405 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:04.405 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:04.405 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:04.405 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:04.405 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:04.405 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:04.405 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:04.405 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:04.405 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:04.405 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:04.405 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:04.406 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:04.406 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:04.406 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:04.406 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:04.406 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:04.406 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:04.406 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:04.406 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:04.406 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:04.406 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:04.406 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:04.406 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:04.406 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:04.407 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:04.407 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:04.407 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:04.407 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:04.407 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:04.407 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:04.407 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:04.407 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:04.407 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:04.407 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:04.407 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:04.407 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:04.407 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:04.407 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:04.408 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:04.408 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:04.408 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:04.408 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:04.408 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:04.408 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:04.408 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:04.408 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:04.408 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:04.408 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:04.408 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:04.408 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:04.408 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:04.408 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:04.409 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:04.409 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:04.409 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:04.409 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:04.409 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:04.409 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:04.409 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:04.409 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:04.409 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:04.409 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:04.409 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:04.409 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:04.409 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:04.410 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:04.410 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:04.410 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:04.410 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:04.410 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:04.410 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:04.410 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:04.410 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:04.410 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:04.410 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:04.410 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:04.411 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:04.411 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:04.411 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:04.411 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:04.411 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:04.411 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:04.411 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:04.411 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:04.411 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:04.411 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:04.411 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:04.411 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:04.412 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:04.412 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:04.412 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:04.412 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:04.412 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:04.412 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:04.412 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:04.412 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:04.412 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:04.412 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:04.413 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:04.413 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:04.413 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:04.413 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:04.413 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:04.413 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:04.413 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:04.413 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:04.413 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:04.413 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:04.413 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:04.414 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:04.414 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:04.414 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:04.414 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:04.414 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:04.414 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestIOWithIOHint (3359 ms) 2024-08-17T17:27:04.414 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestDataPoolIO 2024-08-17T17:27:05.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:05 smithi136 ceph-mon[109861]: pgmap v679: 521 pgs: 521 active+clean; 197 MiB data, 988 MiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 262 op/s 2024-08-17T17:27:05.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:05 smithi136 ceph-mon[115078]: pgmap v679: 521 pgs: 521 active+clean; 197 MiB data, 988 MiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 262 op/s 2024-08-17T17:27:05.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:05 smithi165 ceph-mon[105517]: pgmap v679: 521 pgs: 521 active+clean; 197 MiB data, 988 MiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 262 op/s 2024-08-17T17:27:06.220 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:06.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:06 smithi165 ceph-mon[105517]: osdmap e693: 8 total, 8 up, 8 in 2024-08-17T17:27:06.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:06 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2226203479' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-29","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:06.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:27:06.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:06 smithi136 ceph-mon[109861]: osdmap e693: 8 total, 8 up, 8 in 2024-08-17T17:27:06.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:06 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2226203479' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-29","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:06.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:27:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:06 smithi136 ceph-mon[115078]: osdmap e693: 8 total, 8 up, 8 in 2024-08-17T17:27:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:06 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2226203479' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-29","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:27:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:07 smithi165 ceph-mon[105517]: pgmap v681: 553 pgs: 22 creating+peering, 10 unknown, 521 active+clean; 197 MiB data, 990 MiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 263 op/s 2024-08-17T17:27:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:07 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:27:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:07 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2226203479' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:27:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:07 smithi165 ceph-mon[105517]: osdmap e694: 8 total, 8 up, 8 in 2024-08-17T17:27:07.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:27:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:27:07] "GET /metrics HTTP/1.1" 200 70151 "" "Prometheus/2.51.0" 2024-08-17T17:27:07.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:07 smithi136 ceph-mon[109861]: pgmap v681: 553 pgs: 22 creating+peering, 10 unknown, 521 active+clean; 197 MiB data, 990 MiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 263 op/s 2024-08-17T17:27:07.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:07 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:27:07.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:07 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2226203479' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:27:07.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:07 smithi136 ceph-mon[109861]: osdmap e694: 8 total, 8 up, 8 in 2024-08-17T17:27:07.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:07 smithi136 ceph-mon[115078]: pgmap v681: 553 pgs: 22 creating+peering, 10 unknown, 521 active+clean; 197 MiB data, 990 MiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 263 op/s 2024-08-17T17:27:07.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:07 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:27:07.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:07 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2226203479' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:27:07.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:07 smithi136 ceph-mon[115078]: osdmap e694: 8 total, 8 up, 8 in 2024-08-17T17:27:08.293 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:08.294 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:08.294 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:08.294 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:08.294 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:08.294 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:08.294 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:08.294 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:08.294 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:08.294 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:08.294 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:08.295 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:08.295 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:08.295 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:08.295 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:08.295 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:08.295 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:08.295 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:08.295 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:08.295 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:08.295 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:08.295 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:08.295 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:08.295 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:08.295 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:08.296 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:08.296 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:08.296 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:08.296 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:08.296 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:08.296 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:08.296 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:08.296 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:08.296 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:08.296 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:08.296 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:08.297 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:08.297 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:08.297 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:08.297 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:08.297 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:08.297 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:08.297 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:08.297 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:08.297 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:08.297 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:08.297 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:08.297 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:08.297 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:08.298 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:08.298 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:08.298 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:08.298 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:08.298 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:08.298 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:08.298 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 512~512 = 512 2024-08-17T17:27:08.298 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:08.298 INFO:tasks.workunit.client.0.smithi136.stdout:aio discard: 1536~512 = 0 2024-08-17T17:27:08.298 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:08.298 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:08.298 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:08.298 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:08.299 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:08.299 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:08.299 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:08.299 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestDataPoolIO (3889 ms) 2024-08-17T17:27:08.299 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatch 2024-08-17T17:27:08.299 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:08.325 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatch (31 ms) 2024-08-17T17:27:08.325 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatch 2024-08-17T17:27:08.325 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:08.356 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatch (32 ms) 2024-08-17T17:27:08.357 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccess 2024-08-17T17:27:08.357 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:08.382 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccess (25 ms) 2024-08-17T17:27:08.382 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccess 2024-08-17T17:27:08.382 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:08.408 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccess (27 ms) 2024-08-17T17:27:08.408 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitUnaligned 2024-08-17T17:27:08.408 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:08.473 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitUnaligned (65 ms) 2024-08-17T17:27:08.473 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnaligned 2024-08-17T17:27:08.474 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:08.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:08 smithi165 ceph-mon[105517]: osdmap e695: 8 total, 8 up, 8 in 2024-08-17T17:27:08.539 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnaligned (66 ms) 2024-08-17T17:27:08.539 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteTooLarge 2024-08-17T17:27:08.539 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:08.608 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestCompareAndWriteTooLarge (68 ms) 2024-08-17T17:27:08.608 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteTooLarge 2024-08-17T17:27:08.608 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:08.663 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteTooLarge (55 ms) 2024-08-17T17:27:08.663 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitSuccess 2024-08-17T17:27:08.663 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:08.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:08 smithi136 ceph-mon[109861]: osdmap e695: 8 total, 8 up, 8 in 2024-08-17T17:27:08.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:08 smithi136 ceph-mon[115078]: osdmap e695: 8 total, 8 up, 8 in 2024-08-17T17:27:08.900 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitSuccess (237 ms) 2024-08-17T17:27:08.900 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccess 2024-08-17T17:27:08.900 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:09.164 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccess (264 ms) 2024-08-17T17:27:09.165 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVIovecLenDiffers 2024-08-17T17:27:09.165 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:09.190 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVIovecLenDiffers (26 ms) 2024-08-17T17:27:09.190 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVMismatch 2024-08-17T17:27:09.190 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:09.215 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVMismatch (25 ms) 2024-08-17T17:27:09.215 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVSuccess 2024-08-17T17:27:09.215 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:09.242 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVSuccess (27 ms) 2024-08-17T17:27:09.243 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestScatterGatherIO 2024-08-17T17:27:09.243 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:09.265 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestScatterGatherIO (23 ms) 2024-08-17T17:27:09.265 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestEmptyDiscard 2024-08-17T17:27:09.265 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:09.289 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:09.289 INFO:tasks.workunit.client.0.smithi136.stdout:aio discard: 0~1048576 = 0 2024-08-17T17:27:09.289 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:09.290 INFO:tasks.workunit.client.0.smithi136.stdout:aio discard: 0~4194304 = 0 2024-08-17T17:27:09.290 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:09.290 INFO:tasks.workunit.client.0.smithi136.stdout:aio discard: 3145728~1048576 = 0 2024-08-17T17:27:09.290 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestEmptyDiscard (24 ms) 2024-08-17T17:27:09.290 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestFUA 2024-08-17T17:27:09.290 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:09.395 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:09.395 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:09.395 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:09.395 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:09.395 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:09.395 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:09.395 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:09.395 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:09.395 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:09.395 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:09.395 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.396 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:09.396 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:09.396 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:09.396 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:09.396 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.396 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:09.396 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:09.396 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:09.396 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:09.396 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.396 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:09.396 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:09.396 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:09.396 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:09.397 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.397 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:09.397 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:09.397 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:09.397 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:09.397 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.397 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:09.397 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:09.397 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:09.397 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:09.397 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.398 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:09.398 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:09.398 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:09.398 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.398 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:09.398 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:09.398 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:09.398 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.398 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:09.398 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:09.398 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:09.398 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.398 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:09.399 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:09.399 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:09.399 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.399 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:09.399 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:09.399 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:09.399 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestFUA (106 ms) 2024-08-17T17:27:09.399 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestIOPP 2024-08-17T17:27:09.399 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:09.413 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:09.413 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:09.413 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:09.413 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:09.413 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:09.413 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.413 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:09.413 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:09.413 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:09.414 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:09.414 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.414 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:09.414 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:09.414 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:09.414 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:09.414 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.414 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:09.414 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:09.414 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:09.414 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:09.414 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.414 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:09.414 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:09.415 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:09.415 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:09.415 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.415 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:09.415 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:09.429 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:09.429 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:09.429 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:27:09.429 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:27:09.429 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:27:09.429 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:27:09.429 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:27:09.429 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:27:09.429 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:27:09.429 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:27:09.429 INFO:tasks.workunit.client.0.smithi136.stdout:start compare and write 2024-08-17T17:27:09.429 INFO:tasks.workunit.client.0.smithi136.stdout:compare and wrote: 512 2024-08-17T17:27:09.430 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.430 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:27:09.430 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:09.430 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:09.430 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:27:09.430 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.430 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:27:09.430 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:09.430 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:09.431 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:27:09.431 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.431 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:27:09.431 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:09.431 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:09.431 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:27:09.431 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.431 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:27:09.431 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:09.432 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:09.432 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:27:09.432 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.432 INFO:tasks.workunit.client.0.smithi136.stdout:started aio compare and write 2024-08-17T17:27:09.432 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:09.433 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:09.433 INFO:tasks.workunit.client.0.smithi136.stdout:finished aio compare and write 2024-08-17T17:27:09.434 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:09.434 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:09.434 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:09.434 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:09.434 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:09.434 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.434 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:09.434 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:09.434 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:09.434 INFO:tasks.workunit.client.0.smithi136.stdout:finished read 2024-08-17T17:27:09.434 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.435 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:09.435 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:09.435 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:09.435 INFO:tasks.workunit.client.0.smithi136.stdout:finished read 2024-08-17T17:27:09.435 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.436 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:09.436 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:09.436 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:09.436 INFO:tasks.workunit.client.0.smithi136.stdout:finished read 2024-08-17T17:27:09.436 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.436 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:09.436 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:09.436 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:09.436 INFO:tasks.workunit.client.0.smithi136.stdout:finished read 2024-08-17T17:27:09.436 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:09.436 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:09.436 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:09.436 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:09.436 INFO:tasks.workunit.client.0.smithi136.stdout:finished read 2024-08-17T17:27:09.437 INFO:tasks.workunit.client.0.smithi136.stdout:discard: 512~512 2024-08-17T17:27:09.437 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:09.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:09 smithi165 ceph-mon[105517]: pgmap v684: 553 pgs: 22 creating+peering, 531 active+clean; 199 MiB data, 989 MiB used, 714 GiB / 715 GiB avail; 503 KiB/s rd, 241 KiB/s wr, 1.04k op/s 2024-08-17T17:27:09.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:09 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:27:09.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:09 smithi165 ceph-mon[105517]: osdmap e696: 8 total, 8 up, 8 in 2024-08-17T17:27:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:09 smithi136 ceph-mon[109861]: pgmap v684: 553 pgs: 22 creating+peering, 531 active+clean; 199 MiB data, 989 MiB used, 714 GiB / 715 GiB avail; 503 KiB/s rd, 241 KiB/s wr, 1.04k op/s 2024-08-17T17:27:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:09 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:27:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:09 smithi136 ceph-mon[109861]: osdmap e696: 8 total, 8 up, 8 in 2024-08-17T17:27:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:09 smithi136 ceph-mon[115078]: pgmap v684: 553 pgs: 22 creating+peering, 531 active+clean; 199 MiB data, 989 MiB used, 714 GiB / 715 GiB avail; 503 KiB/s rd, 241 KiB/s wr, 1.04k op/s 2024-08-17T17:27:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:09 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:27:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:09 smithi136 ceph-mon[115078]: osdmap e696: 8 total, 8 up, 8 in 2024-08-17T17:27:10.326 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:10.326 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:10.326 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:10.326 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:10.327 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:10.327 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 0 2024-08-17T17:27:10.327 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.327 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.327 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 0 2024-08-17T17:27:10.327 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.327 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.327 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 16384 2024-08-17T17:27:10.327 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.327 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.327 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 16384 2024-08-17T17:27:10.327 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.327 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.328 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:10.328 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:10.328 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 49152 2024-08-17T17:27:10.328 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.328 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.328 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 49152 2024-08-17T17:27:10.328 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.328 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.328 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 65536 2024-08-17T17:27:10.328 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.328 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.328 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 65536 2024-08-17T17:27:10.328 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.328 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.329 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:10.329 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:10.329 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 98304 2024-08-17T17:27:10.329 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.329 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.329 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 98304 2024-08-17T17:27:10.329 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.329 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.329 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 114688 2024-08-17T17:27:10.329 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.329 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.329 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 114688 2024-08-17T17:27:10.329 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.330 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.330 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:10.330 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:10.330 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 147456 2024-08-17T17:27:10.330 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.330 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.330 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 147456 2024-08-17T17:27:10.330 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.330 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.330 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 163840 2024-08-17T17:27:10.330 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.330 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.330 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 163840 2024-08-17T17:27:10.330 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.331 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.331 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:10.331 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:10.331 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 196608 2024-08-17T17:27:10.331 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.331 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.331 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 196608 2024-08-17T17:27:10.331 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.331 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.331 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 212992 2024-08-17T17:27:10.331 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.331 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.331 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 212992 2024-08-17T17:27:10.331 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.332 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.332 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:10.332 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:10.332 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.332 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.332 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:10.332 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:10.332 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:10.332 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.332 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.332 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.332 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.332 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:10.332 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:10.333 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:10.333 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.333 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.333 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.333 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.333 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:10.333 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:10.333 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:10.333 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.334 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.334 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.334 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.334 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:10.340 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:10.341 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:10.341 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.341 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.341 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.341 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.341 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:10.341 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.341 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.341 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:10.341 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.341 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.341 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:10.361 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:10.361 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:10.361 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.361 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.362 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.362 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.362 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:10.380 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:10.380 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:10.380 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.381 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.381 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.381 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.381 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:10.411 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:10.411 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:10.411 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.411 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.411 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.411 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.411 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:10.439 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:10.440 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:10.440 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.440 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.440 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.440 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.440 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:10.440 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.440 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.440 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:10.440 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.440 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.440 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:10.478 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:10.479 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:10.479 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.479 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.479 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.479 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.479 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:10.515 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:10.515 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:10.515 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.515 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.515 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.515 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.515 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:10.559 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:10.559 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:10.559 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.559 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.559 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.559 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.559 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:10.600 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:10.601 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:10.601 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.601 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.601 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.601 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.601 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:10.601 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.601 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.601 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:10.601 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.601 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.601 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:10.679 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:10.679 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:10.679 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.679 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.679 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.679 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.679 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:10.733 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:10.734 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:10.734 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.734 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.734 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.734 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.734 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:10.796 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:10.796 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:10.796 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.796 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.796 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.796 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.796 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:10.857 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:10.858 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:10.858 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.858 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.858 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.858 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.858 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:10.859 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.859 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.859 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:10.859 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.859 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.859 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:10.978 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:10.978 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:10.979 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:10.979 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:10.979 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:10.979 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:10.979 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:11.048 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:11.048 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:11.048 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:11.048 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:11.049 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:11.049 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:11.049 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:11.131 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:11.131 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:11.131 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:11.131 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:11.132 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:11.132 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:11.132 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:11.221 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:11.221 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:11.221 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:11.221 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:11.221 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:11.221 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:11.221 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:11.222 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:11.222 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:11.222 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:11.222 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestIOPP (1826 ms) 2024-08-17T17:27:11.222 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteCompareTooSmallPP 2024-08-17T17:27:11.222 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:11.246 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestCompareAndWriteCompareTooSmallPP (25 ms) 2024-08-17T17:27:11.247 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteCompareTooSmallPP 2024-08-17T17:27:11.247 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:11.269 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteCompareTooSmallPP (23 ms) 2024-08-17T17:27:11.269 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteWriteTooSmallPP 2024-08-17T17:27:11.269 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:11.294 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestCompareAndWriteWriteTooSmallPP (25 ms) 2024-08-17T17:27:11.294 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteWriteTooSmallPP 2024-08-17T17:27:11.294 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:11.317 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteWriteTooSmallPP (23 ms) 2024-08-17T17:27:11.317 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatchPP 2024-08-17T17:27:11.317 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:11.345 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatchPP (28 ms) 2024-08-17T17:27:11.345 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatchPP 2024-08-17T17:27:11.345 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:11.368 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:11.372 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatchPP (27 ms) 2024-08-17T17:27:11.372 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatchBufferlistGreaterLenPP 2024-08-17T17:27:11.372 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:11.400 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatchBufferlistGreaterLenPP (28 ms) 2024-08-17T17:27:11.400 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatchBufferlistGreaterLenPP 2024-08-17T17:27:11.400 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:11.420 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:11.424 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatchBufferlistGreaterLenPP (24 ms) 2024-08-17T17:27:11.424 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccessPP 2024-08-17T17:27:11.424 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:11.447 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccessPP (23 ms) 2024-08-17T17:27:11.447 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccessPP 2024-08-17T17:27:11.447 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:11.471 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:11.474 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccessPP (27 ms) 2024-08-17T17:27:11.474 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccessBufferlistGreaterLenPP 2024-08-17T17:27:11.474 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:11.504 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccessBufferlistGreaterLenPP (30 ms) 2024-08-17T17:27:11.504 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccessBufferlistGreaterLenPP 2024-08-17T17:27:11.504 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:11.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:11 smithi165 ceph-mon[105517]: pgmap v686: 553 pgs: 22 creating+peering, 531 active+clean; 199 MiB data, 989 MiB used, 714 GiB / 715 GiB avail; 616 KiB/s rd, 295 KiB/s wr, 1.28k op/s 2024-08-17T17:27:11.528 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:11.532 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccessBufferlistGreaterLenPP (28 ms) 2024-08-17T17:27:11.532 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitUnalignedPP 2024-08-17T17:27:11.533 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:11.610 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitUnalignedPP (78 ms) 2024-08-17T17:27:11.610 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnalignedPP 2024-08-17T17:27:11.611 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:11.664 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:11.686 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnalignedPP (76 ms) 2024-08-17T17:27:11.686 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteTooLargePP 2024-08-17T17:27:11.687 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:11 smithi136 ceph-mon[109861]: pgmap v686: 553 pgs: 22 creating+peering, 531 active+clean; 199 MiB data, 989 MiB used, 714 GiB / 715 GiB avail; 616 KiB/s rd, 295 KiB/s wr, 1.28k op/s 2024-08-17T17:27:11.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:11 smithi136 ceph-mon[115078]: pgmap v686: 553 pgs: 22 creating+peering, 531 active+clean; 199 MiB data, 989 MiB used, 714 GiB / 715 GiB avail; 616 KiB/s rd, 295 KiB/s wr, 1.28k op/s 2024-08-17T17:27:11.763 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestCompareAndWriteTooLargePP (77 ms) 2024-08-17T17:27:11.764 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteTooLargePP 2024-08-17T17:27:11.764 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:11.822 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:11.842 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteTooLargePP (79 ms) 2024-08-17T17:27:11.842 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitSuccessPP 2024-08-17T17:27:11.843 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:12.106 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitSuccessPP (263 ms) 2024-08-17T17:27:12.106 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccessPP 2024-08-17T17:27:12.106 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:12.327 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:12.346 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccessPP (241 ms) 2024-08-17T17:27:12.347 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestIOPPWithIOHint 2024-08-17T17:27:12.347 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:12.376 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:12.376 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:12.376 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:12.377 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:12.377 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:27:12.377 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:12.377 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:12.377 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:12.377 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:12.377 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:12.377 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:12.378 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:12.378 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:12.378 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:12.378 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:12.378 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:12.378 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:12.378 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:12.379 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:12.379 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:12.379 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:12.379 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:12.379 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:12.383 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:12.383 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:12.383 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:12.383 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:27:12.383 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:12.384 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:12.384 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:27:12.384 INFO:tasks.workunit.client.0.smithi136.stdout:read: 512 2024-08-17T17:27:12.384 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:12.384 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:12.384 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:12.385 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:12.385 INFO:tasks.workunit.client.0.smithi136.stdout:finished read 2024-08-17T17:27:12.385 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:12.385 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:12.385 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:12.385 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:12.386 INFO:tasks.workunit.client.0.smithi136.stdout:finished read 2024-08-17T17:27:12.386 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:12.386 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:12.386 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:12.386 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:12.386 INFO:tasks.workunit.client.0.smithi136.stdout:finished read 2024-08-17T17:27:12.386 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:12.386 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:12.386 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:12.386 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:12.386 INFO:tasks.workunit.client.0.smithi136.stdout:finished read 2024-08-17T17:27:12.386 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:12.386 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:27:12.387 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:27:13.406 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:27:13.406 INFO:tasks.workunit.client.0.smithi136.stdout:finished read 2024-08-17T17:27:13.406 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 0 2024-08-17T17:27:13.406 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.406 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.406 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 0 2024-08-17T17:27:13.406 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.406 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.406 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 16384 2024-08-17T17:27:13.406 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.406 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.407 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 16384 2024-08-17T17:27:13.407 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.407 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.407 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:13.407 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:13.407 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 49152 2024-08-17T17:27:13.407 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.407 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.407 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 49152 2024-08-17T17:27:13.407 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.407 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.407 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 65536 2024-08-17T17:27:13.407 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.408 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.408 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 65536 2024-08-17T17:27:13.408 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.408 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.408 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:13.408 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:13.408 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 98304 2024-08-17T17:27:13.408 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.408 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.408 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 98304 2024-08-17T17:27:13.408 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.408 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.408 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 114688 2024-08-17T17:27:13.409 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.409 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.409 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 114688 2024-08-17T17:27:13.409 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.409 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.409 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:13.409 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:13.409 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 147456 2024-08-17T17:27:13.409 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.409 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.409 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 147456 2024-08-17T17:27:13.409 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.409 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.409 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 163840 2024-08-17T17:27:13.410 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.410 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.410 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 163840 2024-08-17T17:27:13.410 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.410 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.410 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:13.410 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:13.410 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 196608 2024-08-17T17:27:13.410 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.410 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.410 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 196608 2024-08-17T17:27:13.410 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.410 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.410 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 212992 2024-08-17T17:27:13.411 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.411 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.411 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 212992 2024-08-17T17:27:13.411 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.411 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.411 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:13.411 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:13.411 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:13.411 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:13.411 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:13.411 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:13.411 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:13.411 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.412 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.412 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:13.412 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:13.412 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:13.412 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:13.412 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:13.412 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.412 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.412 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:13.412 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:13.413 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:13.415 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:13.415 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:13.415 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.415 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.416 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:13.416 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:13.416 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:13.424 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:13.424 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:13.424 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.424 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.424 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:13.424 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:13.425 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:13.425 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:13.425 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:13.425 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:13.425 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:13.425 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:13.425 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:13.450 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:13.450 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:13.450 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.450 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.450 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:13.450 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:13.450 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:13.478 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:13.478 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:13.478 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.479 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.479 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:13.479 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:13.479 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:13.498 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:13 smithi136 ceph-mon[109861]: pgmap v687: 553 pgs: 553 active+clean; 273 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 14 MiB/s wr, 2.21k op/s 2024-08-17T17:27:13.499 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:13 smithi136 ceph-mon[115078]: pgmap v687: 553 pgs: 553 active+clean; 273 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 14 MiB/s wr, 2.21k op/s 2024-08-17T17:27:13.507 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:13.507 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:13.507 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.508 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.508 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:13.508 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:13.508 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:13.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:13 smithi165 ceph-mon[105517]: pgmap v687: 553 pgs: 553 active+clean; 273 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 14 MiB/s wr, 2.21k op/s 2024-08-17T17:27:13.534 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:13.534 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:13.534 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.534 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.534 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:13.534 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:13.534 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:13.534 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:13.534 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:13.534 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:13.535 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:13.535 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:13.535 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:13.576 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:13.577 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:13.577 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.577 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.577 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:13.577 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:13.577 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:13.627 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:13.627 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:13.627 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.627 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.628 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:13.628 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:13.628 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:13.687 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:13.687 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:13.687 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.687 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.687 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:13.688 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:13.688 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:13.741 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:13.741 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:13.741 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.741 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.741 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:13.741 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:13.741 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:13.742 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:13.742 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:13.742 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:13.742 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:13.742 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:13.742 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:13.810 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:13.810 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:13.810 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.810 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.810 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:13.810 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:13.810 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:13.873 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:13.873 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:13.873 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.873 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.873 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:13.873 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:13.874 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:13.952 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:13.952 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:13.952 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:13.952 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:13.952 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:13.952 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:13.952 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:14.015 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:14.015 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:14.015 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:14.015 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:14.015 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:14.015 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:14.015 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:14.015 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:14.015 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:14.015 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:14.015 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:14.016 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:14.016 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:14.099 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:14.099 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:14.099 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:14.099 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:14.099 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:14.099 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:14.099 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:14.175 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:14.175 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:14.175 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:14.175 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:14.175 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:14.175 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:14.175 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:14.266 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:14.266 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:14.266 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:14.266 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:14.266 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:14.266 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:14.266 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:27:14.362 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:27:14.362 INFO:tasks.workunit.client.0.smithi136.stdout:finished writesame 2024-08-17T17:27:14.362 INFO:tasks.workunit.client.0.smithi136.stdout:to verify the data 2024-08-17T17:27:14.362 INFO:tasks.workunit.client.0.smithi136.stdout:verified 2024-08-17T17:27:14.362 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:14.362 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:14.362 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:14.362 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:27:14.362 INFO:tasks.workunit.client.0.smithi136.stdout:started writesame 2024-08-17T17:27:14.362 INFO:tasks.workunit.client.0.smithi136.stdout:expected fail, finished writesame 2024-08-17T17:27:14.362 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestIOPPWithIOHint (2015 ms) 2024-08-17T17:27:14.362 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestIOToSnapshot 2024-08-17T17:27:14.363 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:15.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:14 smithi136 ceph-mon[109861]: osdmap e697: 8 total, 8 up, 8 in 2024-08-17T17:27:15.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:14 smithi136 ceph-mon[109861]: pgmap v689: 553 pgs: 553 active+clean; 273 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 12 MiB/s wr, 1.03k op/s 2024-08-17T17:27:15.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:14 smithi136 ceph-mon[115078]: osdmap e697: 8 total, 8 up, 8 in 2024-08-17T17:27:15.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:14 smithi136 ceph-mon[115078]: pgmap v689: 553 pgs: 553 active+clean; 273 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 12 MiB/s wr, 1.03k op/s 2024-08-17T17:27:15.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:14 smithi165 ceph-mon[105517]: osdmap e697: 8 total, 8 up, 8 in 2024-08-17T17:27:15.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:14 smithi165 ceph-mon[105517]: pgmap v689: 553 pgs: 553 active+clean; 273 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 12 MiB/s wr, 1.03k op/s 2024-08-17T17:27:15.957 INFO:tasks.workunit.client.0.smithi136.stdout:num snaps is: 0 2024-08-17T17:27:15.958 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 0 2024-08-17T17:27:15.958 INFO:tasks.workunit.client.0.smithi136.stdout:num snaps is: 1 2024-08-17T17:27:15.958 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 1 2024-08-17T17:27:15.958 INFO:tasks.workunit.client.0.smithi136.stdout:snap: orig 2024-08-17T17:27:15.958 INFO:tasks.workunit.client.0.smithi136.stdout:found orig with size 2097152 2024-08-17T17:27:15.958 INFO:tasks.workunit.client.0.smithi136.stdout:read: 80 2024-08-17T17:27:15.958 INFO:tasks.workunit.client.0.smithi136.stdout:write test data! 2024-08-17T17:27:15.958 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 80 2024-08-17T17:27:15.958 INFO:tasks.workunit.client.0.smithi136.stdout:num snaps is: 2 2024-08-17T17:27:15.958 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 2 2024-08-17T17:27:15.958 INFO:tasks.workunit.client.0.smithi136.stdout:snap: orig 2024-08-17T17:27:15.959 INFO:tasks.workunit.client.0.smithi136.stdout:snap: written 2024-08-17T17:27:15.959 INFO:tasks.workunit.client.0.smithi136.stdout:found orig with size 2097152 2024-08-17T17:27:15.959 INFO:tasks.workunit.client.0.smithi136.stdout:found written with size 2097152 2024-08-17T17:27:15.959 INFO:tasks.workunit.client.0.smithi136.stdout:read: 80 2024-08-17T17:27:15.959 INFO:tasks.workunit.client.0.smithi136.stdout:read: 80 2024-08-17T17:27:15.959 INFO:tasks.workunit.client.0.smithi136.stdout:read: 80 2024-08-17T17:27:15.959 INFO:tasks.workunit.client.0.smithi136.stdout:write to snapshot returned -30 2024-08-17T17:27:15.959 INFO:tasks.workunit.client.0.smithi136.stdout:Read-only file system 2024-08-17T17:27:15.977 INFO:tasks.workunit.client.0.smithi136.stdout:read: 80 2024-08-17T17:27:15.977 INFO:tasks.workunit.client.0.smithi136.stdout:read: 80 2024-08-17T17:27:15.977 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 80 2024-08-17T17:27:15.977 INFO:tasks.workunit.client.0.smithi136.stdout:opening testimg@orig 2024-08-17T17:27:15.977 INFO:tasks.workunit.client.0.smithi136.stdout:read: 80 2024-08-17T17:27:15.978 INFO:tasks.workunit.client.0.smithi136.stdout:write to snapshot returned -30 2024-08-17T17:27:15.978 INFO:tasks.workunit.client.0.smithi136.stdout:Read-only file system 2024-08-17T17:27:16.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:15 smithi136 ceph-mon[109861]: osdmap e698: 8 total, 8 up, 8 in 2024-08-17T17:27:16.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:15 smithi136 ceph-mon[115078]: osdmap e698: 8 total, 8 up, 8 in 2024-08-17T17:27:16.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:15 smithi165 ceph-mon[105517]: osdmap e698: 8 total, 8 up, 8 in 2024-08-17T17:27:17.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:16 smithi136 ceph-mon[109861]: osdmap e699: 8 total, 8 up, 8 in 2024-08-17T17:27:17.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:16 smithi136 ceph-mon[109861]: pgmap v692: 553 pgs: 553 active+clean; 293 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 18 MiB/s wr, 1.84k op/s 2024-08-17T17:27:17.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:16 smithi136 ceph-mon[115078]: osdmap e699: 8 total, 8 up, 8 in 2024-08-17T17:27:17.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:16 smithi136 ceph-mon[115078]: pgmap v692: 553 pgs: 553 active+clean; 293 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 18 MiB/s wr, 1.84k op/s 2024-08-17T17:27:17.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:16 smithi165 ceph-mon[105517]: osdmap e699: 8 total, 8 up, 8 in 2024-08-17T17:27:17.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:16 smithi165 ceph-mon[105517]: pgmap v692: 553 pgs: 553 active+clean; 293 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 18 MiB/s wr, 1.84k op/s 2024-08-17T17:27:17.938 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:27:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:27:17] "GET /metrics HTTP/1.1" 200 70151 "" "Prometheus/2.51.0" 2024-08-17T17:27:17.969 INFO:tasks.workunit.client.0.smithi136.stdout:num snaps is: 2 2024-08-17T17:27:17.970 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 2 2024-08-17T17:27:17.970 INFO:tasks.workunit.client.0.smithi136.stdout:snap: orig 2024-08-17T17:27:17.970 INFO:tasks.workunit.client.0.smithi136.stdout:snap: written 2024-08-17T17:27:17.970 INFO:tasks.workunit.client.0.smithi136.stdout:found orig with size 2097152 2024-08-17T17:27:17.970 INFO:tasks.workunit.client.0.smithi136.stdout:found written with size 2097152 2024-08-17T17:27:17.970 INFO:tasks.workunit.client.0.smithi136.stdout:num snaps is: 1 2024-08-17T17:27:17.970 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 1 2024-08-17T17:27:17.970 INFO:tasks.workunit.client.0.smithi136.stdout:snap: orig 2024-08-17T17:27:17.970 INFO:tasks.workunit.client.0.smithi136.stdout:found orig with size 2097152 2024-08-17T17:27:17.971 INFO:tasks.workunit.client.0.smithi136.stdout:num snaps is: 0 2024-08-17T17:27:17.971 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 0 2024-08-17T17:27:17.971 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestIOToSnapshot (3608 ms) 2024-08-17T17:27:17.971 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestSnapshotDeletedIo 2024-08-17T17:27:17.971 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:18.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:17 smithi136 ceph-mon[109861]: osdmap e700: 8 total, 8 up, 8 in 2024-08-17T17:27:18.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:17 smithi136 ceph-mon[115078]: osdmap e700: 8 total, 8 up, 8 in 2024-08-17T17:27:18.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:17 smithi165 ceph-mon[105517]: osdmap e700: 8 total, 8 up, 8 in 2024-08-17T17:27:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:18 smithi165 ceph-mon[105517]: osdmap e701: 8 total, 8 up, 8 in 2024-08-17T17:27:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:18 smithi165 ceph-mon[105517]: pgmap v695: 553 pgs: 553 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.5 MiB/s rd, 8.7 MiB/s wr, 3.10k op/s 2024-08-17T17:27:19.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:18 smithi136 ceph-mon[109861]: osdmap e701: 8 total, 8 up, 8 in 2024-08-17T17:27:19.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:18 smithi136 ceph-mon[109861]: pgmap v695: 553 pgs: 553 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.5 MiB/s rd, 8.7 MiB/s wr, 3.10k op/s 2024-08-17T17:27:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:18 smithi136 ceph-mon[115078]: osdmap e701: 8 total, 8 up, 8 in 2024-08-17T17:27:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:18 smithi136 ceph-mon[115078]: pgmap v695: 553 pgs: 553 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.5 MiB/s rd, 8.7 MiB/s wr, 3.10k op/s 2024-08-17T17:27:19.977 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestSnapshotDeletedIo (2008 ms) 2024-08-17T17:27:19.977 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestClone 2024-08-17T17:27:19.977 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:19 smithi165 ceph-mon[105517]: osdmap e702: 8 total, 8 up, 8 in 2024-08-17T17:27:20.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:19 smithi136 ceph-mon[109861]: osdmap e702: 8 total, 8 up, 8 in 2024-08-17T17:27:20.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:19 smithi136 ceph-mon[115078]: osdmap e702: 8 total, 8 up, 8 in 2024-08-17T17:27:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:20 smithi165 ceph-mon[105517]: osdmap e703: 8 total, 8 up, 8 in 2024-08-17T17:27:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:20 smithi165 ceph-mon[105517]: pgmap v698: 553 pgs: 553 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 2.11k op/s 2024-08-17T17:27:21.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:20 smithi136 ceph-mon[109861]: osdmap e703: 8 total, 8 up, 8 in 2024-08-17T17:27:21.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:20 smithi136 ceph-mon[109861]: pgmap v698: 553 pgs: 553 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 2.11k op/s 2024-08-17T17:27:21.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:20 smithi136 ceph-mon[115078]: osdmap e703: 8 total, 8 up, 8 in 2024-08-17T17:27:21.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:20 smithi136 ceph-mon[115078]: pgmap v698: 553 pgs: 553 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 2.11k op/s 2024-08-17T17:27:21.989 INFO:tasks.workunit.client.0.smithi136.stdout:made parent image "parent" 2024-08-17T17:27:21.990 INFO:tasks.workunit.client.0.smithi136.stdout:parent has no parent info 2024-08-17T17:27:21.990 INFO:tasks.workunit.client.0.smithi136.stdout:made snapshot "parent@parent_snap" 2024-08-17T17:27:21.990 INFO:tasks.workunit.client.0.smithi136.stdout:can't unprotect an unprotected snap 2024-08-17T17:27:21.990 INFO:tasks.workunit.client.0.smithi136.stdout:can't protect a protected snap 2024-08-17T17:27:21.990 INFO:tasks.workunit.client.0.smithi136.stdout:made and opened clone "child" 2024-08-17T17:27:21.990 INFO:tasks.workunit.client.0.smithi136.stdout:read: 8 2024-08-17T17:27:21.990 INFO:tasks.workunit.client.0.smithi136.stdout:read: 8 2024-08-17T17:27:21.990 INFO:tasks.workunit.client.0.smithi136.stdout:read: 8 2024-08-17T17:27:21.990 INFO:tasks.workunit.client.0.smithi136.stdout:sizes and overlaps are good between parent and child 2024-08-17T17:27:21.990 INFO:tasks.workunit.client.0.smithi136.stdout:child image successfully cloned all image-meta pairs 2024-08-17T17:27:21.991 INFO:tasks.workunit.client.0.smithi136.stdout:sized down clone, changed overlap 2024-08-17T17:27:21.991 INFO:tasks.workunit.client.0.smithi136.stdout:parent info: size 4194304 obj_size 4194304 parent_pool 18446744073709551615 2024-08-17T17:27:21.991 INFO:tasks.workunit.client.0.smithi136.stdout:sized up clone, changed size but not overlap or parent's size 2024-08-17T17:27:21.991 INFO:tasks.workunit.client.0.smithi136.stdout:can't remove parent while child still exists 2024-08-17T17:27:21.991 INFO:tasks.workunit.client.0.smithi136.stdout:can't remove parent while still protected 2024-08-17T17:27:21.991 INFO:tasks.workunit.client.0.smithi136.stdout:removed parent snap after unprotecting 2024-08-17T17:27:21.991 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestClone (2012 ms) 2024-08-17T17:27:21.991 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestClone2 2024-08-17T17:27:21.991 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:21 smithi165 ceph-mon[105517]: osdmap e704: 8 total, 8 up, 8 in 2024-08-17T17:27:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:27:22.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:21 smithi136 ceph-mon[109861]: osdmap e704: 8 total, 8 up, 8 in 2024-08-17T17:27:22.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:27:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:21 smithi136 ceph-mon[115078]: osdmap e704: 8 total, 8 up, 8 in 2024-08-17T17:27:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:27:23.097 INFO:tasks.workunit.client.0.smithi136.stdout:made parent image "parent" 2024-08-17T17:27:23.097 INFO:tasks.workunit.client.0.smithi136.stdout:parent has no parent info 2024-08-17T17:27:23.098 INFO:tasks.workunit.client.0.smithi136.stdout:made snapshot "parent@parent_snap" 2024-08-17T17:27:23.098 INFO:tasks.workunit.client.0.smithi136.stdout:made and opened clone "child" 2024-08-17T17:27:23.098 INFO:tasks.workunit.client.0.smithi136.stdout:child image successfully cloned all image-meta pairs 2024-08-17T17:27:23.098 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestClone2 (1108 ms) 2024-08-17T17:27:23.098 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.ListChildren 2024-08-17T17:27:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:22 smithi165 ceph-mon[105517]: osdmap e705: 8 total, 8 up, 8 in 2024-08-17T17:27:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:22 smithi165 ceph-mon[105517]: pgmap v701: 553 pgs: 553 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 403 KiB/s rd, 30 KiB/s wr, 493 op/s 2024-08-17T17:27:23.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:22 smithi136 ceph-mon[109861]: osdmap e705: 8 total, 8 up, 8 in 2024-08-17T17:27:23.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:22 smithi136 ceph-mon[109861]: pgmap v701: 553 pgs: 553 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 403 KiB/s rd, 30 KiB/s wr, 493 op/s 2024-08-17T17:27:23.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:22 smithi136 ceph-mon[115078]: osdmap e705: 8 total, 8 up, 8 in 2024-08-17T17:27:23.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:22 smithi136 ceph-mon[115078]: pgmap v701: 553 pgs: 553 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 403 KiB/s rd, 30 KiB/s wr, 493 op/s 2024-08-17T17:27:24.242 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:23 smithi136 ceph-mon[109861]: osdmap e706: 8 total, 8 up, 8 in 2024-08-17T17:27:24.242 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:23 smithi136 ceph-mon[109861]: osdmap e707: 8 total, 8 up, 8 in 2024-08-17T17:27:24.242 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:23 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/371540674' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-30","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:24.243 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:23 smithi136 ceph-mon[115078]: osdmap e706: 8 total, 8 up, 8 in 2024-08-17T17:27:24.243 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:23 smithi136 ceph-mon[115078]: osdmap e707: 8 total, 8 up, 8 in 2024-08-17T17:27:24.243 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:23 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/371540674' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-30","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:24.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:23 smithi165 ceph-mon[105517]: osdmap e706: 8 total, 8 up, 8 in 2024-08-17T17:27:24.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:23 smithi165 ceph-mon[105517]: osdmap e707: 8 total, 8 up, 8 in 2024-08-17T17:27:24.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:23 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/371540674' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-30","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:25.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:24 smithi136 ceph-mon[115078]: pgmap v704: 585 pgs: 32 unknown, 553 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 403 KiB/s rd, 30 KiB/s wr, 493 op/s 2024-08-17T17:27:25.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:24 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/371540674' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:27:25.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:24 smithi136 ceph-mon[115078]: osdmap e708: 8 total, 8 up, 8 in 2024-08-17T17:27:25.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:24 smithi136 ceph-mon[109861]: pgmap v704: 585 pgs: 32 unknown, 553 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 403 KiB/s rd, 30 KiB/s wr, 493 op/s 2024-08-17T17:27:25.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:24 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/371540674' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:27:25.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:24 smithi136 ceph-mon[109861]: osdmap e708: 8 total, 8 up, 8 in 2024-08-17T17:27:25.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:24 smithi165 ceph-mon[105517]: pgmap v704: 585 pgs: 32 unknown, 553 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 403 KiB/s rd, 30 KiB/s wr, 493 op/s 2024-08-17T17:27:25.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:24 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/371540674' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:27:25.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:24 smithi165 ceph-mon[105517]: osdmap e708: 8 total, 8 up, 8 in 2024-08-17T17:27:26.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:25 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:27:26.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:25 smithi165 ceph-mon[105517]: osdmap e709: 8 total, 8 up, 8 in 2024-08-17T17:27:26.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:25 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/758223136' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-31","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:26.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:25 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:27:26.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:25 smithi136 ceph-mon[109861]: osdmap e709: 8 total, 8 up, 8 in 2024-08-17T17:27:26.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:25 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/758223136' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-31","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:26.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:25 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:27:26.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:25 smithi136 ceph-mon[115078]: osdmap e709: 8 total, 8 up, 8 in 2024-08-17T17:27:26.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:25 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/758223136' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-31","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:26.941 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:27.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:26 smithi165 ceph-mon[105517]: pgmap v707: 617 pgs: 46 unknown, 571 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 396 KiB/s rd, 26 KiB/s wr, 472 op/s 2024-08-17T17:27:27.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:26 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/758223136' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:27:27.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:26 smithi165 ceph-mon[105517]: osdmap e710: 8 total, 8 up, 8 in 2024-08-17T17:27:27.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:26 smithi136 ceph-mon[115078]: pgmap v707: 617 pgs: 46 unknown, 571 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 396 KiB/s rd, 26 KiB/s wr, 472 op/s 2024-08-17T17:27:27.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:26 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/758223136' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:27:27.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:26 smithi136 ceph-mon[115078]: osdmap e710: 8 total, 8 up, 8 in 2024-08-17T17:27:27.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:26 smithi136 ceph-mon[109861]: pgmap v707: 617 pgs: 46 unknown, 571 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 396 KiB/s rd, 26 KiB/s wr, 472 op/s 2024-08-17T17:27:27.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:26 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/758223136' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:27:27.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:26 smithi136 ceph-mon[109861]: osdmap e710: 8 total, 8 up, 8 in 2024-08-17T17:27:27.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:27:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:27:27] "GET /metrics HTTP/1.1" 200 70191 "" "Prometheus/2.51.0" 2024-08-17T17:27:29.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:28 smithi136 ceph-mon[115078]: osdmap e711: 8 total, 8 up, 8 in 2024-08-17T17:27:29.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:28 smithi136 ceph-mon[115078]: pgmap v710: 617 pgs: 12 unknown, 605 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 434 KiB/s rd, 31 KiB/s wr, 520 op/s 2024-08-17T17:27:29.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:28 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:27:29.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:28 smithi136 ceph-mon[109861]: osdmap e711: 8 total, 8 up, 8 in 2024-08-17T17:27:29.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:28 smithi136 ceph-mon[109861]: pgmap v710: 617 pgs: 12 unknown, 605 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 434 KiB/s rd, 31 KiB/s wr, 520 op/s 2024-08-17T17:27:29.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:28 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:27:29.208 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:28 smithi165 ceph-mon[105517]: osdmap e711: 8 total, 8 up, 8 in 2024-08-17T17:27:29.208 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:28 smithi165 ceph-mon[105517]: pgmap v710: 617 pgs: 12 unknown, 605 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 434 KiB/s rd, 31 KiB/s wr, 520 op/s 2024-08-17T17:27:29.208 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:28 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:27:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:29 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:27:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:29 smithi165 ceph-mon[105517]: osdmap e712: 8 total, 8 up, 8 in 2024-08-17T17:27:30.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:29 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:27:30.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:29 smithi136 ceph-mon[115078]: osdmap e712: 8 total, 8 up, 8 in 2024-08-17T17:27:30.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:29 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:27:30.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:29 smithi136 ceph-mon[109861]: osdmap e712: 8 total, 8 up, 8 in 2024-08-17T17:27:31.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:30 smithi136 ceph-mon[109861]: osdmap e713: 8 total, 8 up, 8 in 2024-08-17T17:27:31.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:30 smithi136 ceph-mon[109861]: pgmap v713: 617 pgs: 12 unknown, 605 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 38 KiB/s rd, 4.5 KiB/s wr, 47 op/s 2024-08-17T17:27:31.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:30 smithi136 ceph-mon[115078]: osdmap e713: 8 total, 8 up, 8 in 2024-08-17T17:27:31.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:30 smithi136 ceph-mon[115078]: pgmap v713: 617 pgs: 12 unknown, 605 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 38 KiB/s rd, 4.5 KiB/s wr, 47 op/s 2024-08-17T17:27:31.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:30 smithi165 ceph-mon[105517]: osdmap e713: 8 total, 8 up, 8 in 2024-08-17T17:27:31.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:30 smithi165 ceph-mon[105517]: pgmap v713: 617 pgs: 12 unknown, 605 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 38 KiB/s rd, 4.5 KiB/s wr, 47 op/s 2024-08-17T17:27:32.225 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:31 smithi165 ceph-mon[105517]: osdmap e714: 8 total, 8 up, 8 in 2024-08-17T17:27:32.225 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:31 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:27:32.225 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:31 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:27:32.401 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.402 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-31/image125 2024-08-17T17:27:32.402 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-31/image125 2024-08-17T17:27:32.402 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image125 2024-08-17T17:27:32.402 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.402 INFO:tasks.workunit.client.0.smithi136.stdout:num children is: 1 2024-08-17T17:27:32.402 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 1 2024-08-17T17:27:32.402 INFO:tasks.workunit.client.0.smithi136.stdout:child: image125 2024-08-17T17:27:32.402 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image125/acd61d51d844 2024-08-17T17:27:32.402 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.402 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.402 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-31/image125 2024-08-17T17:27:32.403 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:32.403 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-31/image125 2024-08-17T17:27:32.403 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image125 2024-08-17T17:27:32.403 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.403 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.403 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:32.403 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:32.403 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:32.403 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.403 INFO:tasks.workunit.client.0.smithi136.stdout:num children is: 2 2024-08-17T17:27:32.403 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 2 2024-08-17T17:27:32.404 INFO:tasks.workunit.client.0.smithi136.stdout:child: image126 2024-08-17T17:27:32.404 INFO:tasks.workunit.client.0.smithi136.stdout:child: image125 2024-08-17T17:27:32.404 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image125/acd61d51d844 2024-08-17T17:27:32.404 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.404 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-30/image126/acd6a8fba43d 2024-08-17T17:27:32.404 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.404 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.404 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-31/image125 2024-08-17T17:27:32.404 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:32.404 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-31/image125 2024-08-17T17:27:32.404 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image125 2024-08-17T17:27:32.404 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.404 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.405 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:32.405 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:32.405 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:32.405 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.405 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.405 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-31/image127 2024-08-17T17:27:32.405 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:32.405 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-31/image125 2024-08-17T17:27:32.405 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-31/image127 2024-08-17T17:27:32.405 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image127 2024-08-17T17:27:32.405 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.405 INFO:tasks.workunit.client.0.smithi136.stdout:num children is: 3 2024-08-17T17:27:32.405 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 3 2024-08-17T17:27:32.406 INFO:tasks.workunit.client.0.smithi136.stdout:child: image126 2024-08-17T17:27:32.406 INFO:tasks.workunit.client.0.smithi136.stdout:child: image125 2024-08-17T17:27:32.406 INFO:tasks.workunit.client.0.smithi136.stdout:child: image127 2024-08-17T17:27:32.406 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image125/acd61d51d844 2024-08-17T17:27:32.406 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.406 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-30/image126/acd6a8fba43d 2024-08-17T17:27:32.406 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.406 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image127/acd67ff0f3c2 2024-08-17T17:27:32.406 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.406 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.406 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-31/image125 2024-08-17T17:27:32.407 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:32.407 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-31/image125 2024-08-17T17:27:32.407 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image125 2024-08-17T17:27:32.407 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.407 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.407 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:32.407 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:32.407 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:32.407 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.407 INFO:tasks.workunit.client.0.smithi136.stdout:num children is: 3 2024-08-17T17:27:32.408 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 3 2024-08-17T17:27:32.408 INFO:tasks.workunit.client.0.smithi136.stdout:child: image126 2024-08-17T17:27:32.408 INFO:tasks.workunit.client.0.smithi136.stdout:child: image125 2024-08-17T17:27:32.408 INFO:tasks.workunit.client.0.smithi136.stdout:child: image127 2024-08-17T17:27:32.408 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image125/acd61d51d844 2024-08-17T17:27:32.408 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.408 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-30/image126/acd6a8fba43d 2024-08-17T17:27:32.408 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.408 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image127/acd67ff0f3c2 2024-08-17T17:27:32.408 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.408 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.408 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-31/image125 2024-08-17T17:27:32.409 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:32.409 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-31/image125 2024-08-17T17:27:32.409 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image125 2024-08-17T17:27:32.409 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.409 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.409 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:32.409 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:32.409 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:32.409 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.409 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.409 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-31/image128 2024-08-17T17:27:32.409 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:32.409 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-31/image125 2024-08-17T17:27:32.410 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-31/image128 2024-08-17T17:27:32.410 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image128 2024-08-17T17:27:32.410 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.410 INFO:tasks.workunit.client.0.smithi136.stdout:num children is: 4 2024-08-17T17:27:32.410 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 4 2024-08-17T17:27:32.410 INFO:tasks.workunit.client.0.smithi136.stdout:child: image126 2024-08-17T17:27:32.410 INFO:tasks.workunit.client.0.smithi136.stdout:child: image125 2024-08-17T17:27:32.410 INFO:tasks.workunit.client.0.smithi136.stdout:child: image127 2024-08-17T17:27:32.410 INFO:tasks.workunit.client.0.smithi136.stdout:child: image128 2024-08-17T17:27:32.410 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image125/acd61d51d844 2024-08-17T17:27:32.410 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.410 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-30/image126/acd6a8fba43d 2024-08-17T17:27:32.410 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.411 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image127/acd67ff0f3c2 2024-08-17T17:27:32.411 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.411 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image128/acd6a18f5be 2024-08-17T17:27:32.411 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.411 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.411 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-31/image125 2024-08-17T17:27:32.411 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:32.411 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-31/image125 2024-08-17T17:27:32.411 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image125 2024-08-17T17:27:32.411 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.411 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.411 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:32.411 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:32.411 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:32.412 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.412 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.412 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-31/image127 2024-08-17T17:27:32.412 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:32.412 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-31/image125 2024-08-17T17:27:32.412 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-31/image127 2024-08-17T17:27:32.412 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image127 2024-08-17T17:27:32.412 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.412 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:32.412 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-31/image128 2024-08-17T17:27:32.412 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:32.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:31 smithi136 ceph-mon[115078]: osdmap e714: 8 total, 8 up, 8 in 2024-08-17T17:27:32.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:31 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:27:32.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:31 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:27:32.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:31 smithi136 ceph-mon[109861]: osdmap e714: 8 total, 8 up, 8 in 2024-08-17T17:27:32.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:31 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:27:32.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:31 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:27:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:32 smithi165 ceph-mon[105517]: osdmap e715: 8 total, 8 up, 8 in 2024-08-17T17:27:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:32 smithi165 ceph-mon[105517]: pgmap v716: 617 pgs: 617 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 2.5 KiB/s wr, 42 op/s 2024-08-17T17:27:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:27:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:27:33.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:32 smithi136 ceph-mon[109861]: osdmap e715: 8 total, 8 up, 8 in 2024-08-17T17:27:33.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:32 smithi136 ceph-mon[109861]: pgmap v716: 617 pgs: 617 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 2.5 KiB/s wr, 42 op/s 2024-08-17T17:27:33.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:27:33.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:27:33.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:32 smithi136 ceph-mon[115078]: osdmap e715: 8 total, 8 up, 8 in 2024-08-17T17:27:33.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:32 smithi136 ceph-mon[115078]: pgmap v716: 617 pgs: 617 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 2.5 KiB/s wr, 42 op/s 2024-08-17T17:27:33.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:27:33.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:27:33.722 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-31/image125 2024-08-17T17:27:33.722 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-31/image127 2024-08-17T17:27:33.722 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-31/image128 2024-08-17T17:27:33.722 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image128 2024-08-17T17:27:33.722 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:33.722 INFO:tasks.workunit.client.0.smithi136.stdout:num children is: 4 2024-08-17T17:27:33.723 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 4 2024-08-17T17:27:33.723 INFO:tasks.workunit.client.0.smithi136.stdout:child: image126 2024-08-17T17:27:33.723 INFO:tasks.workunit.client.0.smithi136.stdout:child: image125 2024-08-17T17:27:33.723 INFO:tasks.workunit.client.0.smithi136.stdout:child: image127 2024-08-17T17:27:33.723 INFO:tasks.workunit.client.0.smithi136.stdout:child: image128 2024-08-17T17:27:33.723 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image125/acd61d51d844 2024-08-17T17:27:33.723 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:33.723 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-30/image126/acd6a8fba43d 2024-08-17T17:27:33.723 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:33.723 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image127/acd67ff0f3c2 2024-08-17T17:27:33.723 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:33.723 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image128/acd6a18f5be 2024-08-17T17:27:33.723 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:33.724 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:33.724 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:33.724 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:33.724 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:33.724 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:33.724 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:33.724 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-31/image127 2024-08-17T17:27:33.724 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:33.724 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-31/image127 2024-08-17T17:27:33.724 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image127 2024-08-17T17:27:33.724 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:33.724 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:33.724 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-31/image128 2024-08-17T17:27:33.725 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:33.725 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-31/image127 2024-08-17T17:27:33.725 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-31/image128 2024-08-17T17:27:33.725 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image128 2024-08-17T17:27:33.725 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:33.725 INFO:tasks.workunit.client.0.smithi136.stdout:num children is: 3 2024-08-17T17:27:33.725 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 3 2024-08-17T17:27:33.725 INFO:tasks.workunit.client.0.smithi136.stdout:child: image126 2024-08-17T17:27:33.725 INFO:tasks.workunit.client.0.smithi136.stdout:child: image127 2024-08-17T17:27:33.725 INFO:tasks.workunit.client.0.smithi136.stdout:child: image128 2024-08-17T17:27:33.725 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-30/image126/acd6a8fba43d 2024-08-17T17:27:33.725 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:33.725 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image127/acd67ff0f3c2 2024-08-17T17:27:33.726 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:33.726 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image128/acd6a18f5be 2024-08-17T17:27:33.726 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:33.726 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:33.726 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:33.726 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:33.726 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:33.726 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:33.726 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:33.726 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-31/image128 2024-08-17T17:27:33.726 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:33.726 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-31/image128 2024-08-17T17:27:33.726 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image128 2024-08-17T17:27:33.727 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:33.727 INFO:tasks.workunit.client.0.smithi136.stdout:num children is: 2 2024-08-17T17:27:33.727 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 2 2024-08-17T17:27:33.727 INFO:tasks.workunit.client.0.smithi136.stdout:child: image126 2024-08-17T17:27:33.727 INFO:tasks.workunit.client.0.smithi136.stdout:child: image128 2024-08-17T17:27:33.727 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-30/image126/acd6a8fba43d 2024-08-17T17:27:33.727 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:33.727 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-31/image128/acd6a18f5be 2024-08-17T17:27:33.727 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:33.727 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:33.727 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:33.727 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:33.727 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-30/image126 2024-08-17T17:27:33.727 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:33.728 INFO:tasks.workunit.client.0.smithi136.stdout:num children is: 1 2024-08-17T17:27:33.728 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 1 2024-08-17T17:27:33.728 INFO:tasks.workunit.client.0.smithi136.stdout:child: image126 2024-08-17T17:27:33.728 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-30/image126/acd6a8fba43d 2024-08-17T17:27:33.728 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:33.728 INFO:tasks.workunit.client.0.smithi136.stdout:num children is: 0 2024-08-17T17:27:33.728 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 0 2024-08-17T17:27:33.728 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.ListChildren (10622 ms) 2024-08-17T17:27:33.728 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.ListChildrenTiered 2024-08-17T17:27:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:33 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:27:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:33 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:27:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:33 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:27:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:33 smithi165 ceph-mon[105517]: osdmap e716: 8 total, 8 up, 8 in 2024-08-17T17:27:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:33 smithi165 ceph-mon[105517]: osdmap e717: 8 total, 8 up, 8 in 2024-08-17T17:27:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:33 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/531025704' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-32","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:33 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-32","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:34.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:33 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:27:34.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:33 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:27:34.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:33 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:27:34.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:33 smithi136 ceph-mon[109861]: osdmap e716: 8 total, 8 up, 8 in 2024-08-17T17:27:34.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:33 smithi136 ceph-mon[109861]: osdmap e717: 8 total, 8 up, 8 in 2024-08-17T17:27:34.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:33 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/531025704' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-32","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:34.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:33 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-32","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:34.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:33 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:27:34.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:33 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:27:34.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:33 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:27:34.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:33 smithi136 ceph-mon[115078]: osdmap e716: 8 total, 8 up, 8 in 2024-08-17T17:27:34.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:33 smithi136 ceph-mon[115078]: osdmap e717: 8 total, 8 up, 8 in 2024-08-17T17:27:34.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:33 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/531025704' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-32","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:34.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:33 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-32","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:34 smithi165 ceph-mon[105517]: pgmap v719: 649 pgs: 32 unknown, 617 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 2.5 KiB/s wr, 42 op/s 2024-08-17T17:27:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:34 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:27:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:34 smithi165 ceph-mon[105517]: osdmap e718: 8 total, 8 up, 8 in 2024-08-17T17:27:35.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:34 smithi136 ceph-mon[109861]: pgmap v719: 649 pgs: 32 unknown, 617 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 2.5 KiB/s wr, 42 op/s 2024-08-17T17:27:35.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:34 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:27:35.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:34 smithi136 ceph-mon[109861]: osdmap e718: 8 total, 8 up, 8 in 2024-08-17T17:27:35.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:34 smithi136 ceph-mon[115078]: pgmap v719: 649 pgs: 32 unknown, 617 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 2.5 KiB/s wr, 42 op/s 2024-08-17T17:27:35.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:34 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:27:35.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:34 smithi136 ceph-mon[115078]: osdmap e718: 8 total, 8 up, 8 in 2024-08-17T17:27:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:35 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:27:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:35 smithi165 ceph-mon[105517]: osdmap e719: 8 total, 8 up, 8 in 2024-08-17T17:27:36.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:35 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3828434117' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:36.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:35 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:36.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:35 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:27:36.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:35 smithi136 ceph-mon[109861]: osdmap e719: 8 total, 8 up, 8 in 2024-08-17T17:27:36.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:35 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3828434117' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:36.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:35 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:36.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:35 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:27:36.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:35 smithi136 ceph-mon[115078]: osdmap e719: 8 total, 8 up, 8 in 2024-08-17T17:27:36.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:35 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3828434117' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:36.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:35 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:37.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:37 smithi136 ceph-mon[109861]: pgmap v722: 681 pgs: 44 unknown, 637 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 335 KiB/s rd, 5.7 KiB/s wr, 406 op/s 2024-08-17T17:27:37.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:27:37.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:27:37.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:37 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:27:37.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:37 smithi136 ceph-mon[109861]: osdmap e720: 8 total, 8 up, 8 in 2024-08-17T17:27:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:37 smithi136 ceph-mon[115078]: pgmap v722: 681 pgs: 44 unknown, 637 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 335 KiB/s rd, 5.7 KiB/s wr, 406 op/s 2024-08-17T17:27:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:27:37.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:27:37.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:37 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:27:37.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:37 smithi136 ceph-mon[115078]: osdmap e720: 8 total, 8 up, 8 in 2024-08-17T17:27:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:37 smithi165 ceph-mon[105517]: pgmap v722: 681 pgs: 44 unknown, 637 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 335 KiB/s rd, 5.7 KiB/s wr, 406 op/s 2024-08-17T17:27:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:27:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:27:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:37 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:27:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:37 smithi165 ceph-mon[105517]: osdmap e720: 8 total, 8 up, 8 in 2024-08-17T17:27:37.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:27:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:27:37] "GET /metrics HTTP/1.1" 200 79717 "" "Prometheus/2.51.0" 2024-08-17T17:27:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:38 smithi165 ceph-mon[105517]: osdmap e721: 8 total, 8 up, 8 in 2024-08-17T17:27:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:38 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3984153247' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-34","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:38 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-34","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:38 smithi165 ceph-mon[105517]: pgmap v725: 713 pgs: 3 creating+activating, 49 unknown, 661 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 564 KiB/s rd, 14 KiB/s wr, 687 op/s 2024-08-17T17:27:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:38 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:27:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:38 smithi165 ceph-mon[105517]: osdmap e722: 8 total, 8 up, 8 in 2024-08-17T17:27:39.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:38 smithi136 ceph-mon[109861]: osdmap e721: 8 total, 8 up, 8 in 2024-08-17T17:27:39.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:38 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3984153247' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-34","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:39.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:38 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-34","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:39.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:38 smithi136 ceph-mon[109861]: pgmap v725: 713 pgs: 3 creating+activating, 49 unknown, 661 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 564 KiB/s rd, 14 KiB/s wr, 687 op/s 2024-08-17T17:27:39.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:38 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:27:39.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:38 smithi136 ceph-mon[109861]: osdmap e722: 8 total, 8 up, 8 in 2024-08-17T17:27:39.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:38 smithi136 ceph-mon[115078]: osdmap e721: 8 total, 8 up, 8 in 2024-08-17T17:27:39.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:38 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3984153247' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-34","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:39.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:38 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-34","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:27:39.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:38 smithi136 ceph-mon[115078]: pgmap v725: 713 pgs: 3 creating+activating, 49 unknown, 661 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 564 KiB/s rd, 14 KiB/s wr, 687 op/s 2024-08-17T17:27:39.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:38 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:27:39.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:38 smithi136 ceph-mon[115078]: osdmap e722: 8 total, 8 up, 8 in 2024-08-17T17:27:40.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:39 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-librbd-smithi136-87100-32", "tierpool":"test-librbd-smithi136-87100-34", "force_nonempty":""} : dispatch 2024-08-17T17:27:40.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:39 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-librbd-smithi136-87100-32", "tierpool":"test-librbd-smithi136-87100-34", "force_nonempty":""} : dispatch 2024-08-17T17:27:40.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:39 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-librbd-smithi136-87100-32", "tierpool":"test-librbd-smithi136-87100-34", "force_nonempty":""}]': finished 2024-08-17T17:27:40.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:39 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-librbd-smithi136-87100-32", "tierpool":"test-librbd-smithi136-87100-34", "force_nonempty":""} : dispatch 2024-08-17T17:27:40.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:39 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-librbd-smithi136-87100-32", "tierpool":"test-librbd-smithi136-87100-34", "force_nonempty":""} : dispatch 2024-08-17T17:27:40.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:39 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-librbd-smithi136-87100-32", "tierpool":"test-librbd-smithi136-87100-34", "force_nonempty":""}]': finished 2024-08-17T17:27:40.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:39 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-librbd-smithi136-87100-32", "tierpool":"test-librbd-smithi136-87100-34", "force_nonempty":""} : dispatch 2024-08-17T17:27:40.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:39 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-librbd-smithi136-87100-32", "tierpool":"test-librbd-smithi136-87100-34", "force_nonempty":""} : dispatch 2024-08-17T17:27:40.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:39 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-librbd-smithi136-87100-32", "tierpool":"test-librbd-smithi136-87100-34", "force_nonempty":""}]': finished 2024-08-17T17:27:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:40 smithi165 ceph-mon[105517]: osdmap e723: 8 total, 8 up, 8 in 2024-08-17T17:27:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:40 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-librbd-smithi136-87100-34", "mode":"writeback"} : dispatch 2024-08-17T17:27:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:40 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-librbd-smithi136-87100-34", "mode":"writeback"} : dispatch 2024-08-17T17:27:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:40 smithi165 ceph-mon[105517]: pgmap v728: 713 pgs: 3 creating+activating, 49 unknown, 661 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 230 KiB/s rd, 8.0 KiB/s wr, 280 op/s 2024-08-17T17:27:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:40 smithi165 ceph-mon[105517]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-08-17T17:27:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:40 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-librbd-smithi136-87100-34", "mode":"writeback"}]': finished 2024-08-17T17:27:41.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:40 smithi165 ceph-mon[105517]: osdmap e724: 8 total, 8 up, 8 in 2024-08-17T17:27:41.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:40 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-librbd-smithi136-87100-32", "overlaypool":"test-librbd-smithi136-87100-34"} : dispatch 2024-08-17T17:27:41.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:40 smithi136 ceph-mon[109861]: osdmap e723: 8 total, 8 up, 8 in 2024-08-17T17:27:41.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:40 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-librbd-smithi136-87100-34", "mode":"writeback"} : dispatch 2024-08-17T17:27:41.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:40 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-librbd-smithi136-87100-34", "mode":"writeback"} : dispatch 2024-08-17T17:27:41.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:40 smithi136 ceph-mon[109861]: pgmap v728: 713 pgs: 3 creating+activating, 49 unknown, 661 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 230 KiB/s rd, 8.0 KiB/s wr, 280 op/s 2024-08-17T17:27:41.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:40 smithi136 ceph-mon[109861]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-08-17T17:27:41.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:40 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-librbd-smithi136-87100-34", "mode":"writeback"}]': finished 2024-08-17T17:27:41.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:40 smithi136 ceph-mon[109861]: osdmap e724: 8 total, 8 up, 8 in 2024-08-17T17:27:41.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:40 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-librbd-smithi136-87100-32", "overlaypool":"test-librbd-smithi136-87100-34"} : dispatch 2024-08-17T17:27:41.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:40 smithi136 ceph-mon[115078]: osdmap e723: 8 total, 8 up, 8 in 2024-08-17T17:27:41.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:40 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-librbd-smithi136-87100-34", "mode":"writeback"} : dispatch 2024-08-17T17:27:41.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:40 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-librbd-smithi136-87100-34", "mode":"writeback"} : dispatch 2024-08-17T17:27:41.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:40 smithi136 ceph-mon[115078]: pgmap v728: 713 pgs: 3 creating+activating, 49 unknown, 661 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 230 KiB/s rd, 8.0 KiB/s wr, 280 op/s 2024-08-17T17:27:41.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:40 smithi136 ceph-mon[115078]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-08-17T17:27:41.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:40 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-librbd-smithi136-87100-34", "mode":"writeback"}]': finished 2024-08-17T17:27:41.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:40 smithi136 ceph-mon[115078]: osdmap e724: 8 total, 8 up, 8 in 2024-08-17T17:27:41.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:40 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-librbd-smithi136-87100-32", "overlaypool":"test-librbd-smithi136-87100-34"} : dispatch 2024-08-17T17:27:41.962 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:41 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-librbd-smithi136-87100-32", "overlaypool":"test-librbd-smithi136-87100-34"} : dispatch 2024-08-17T17:27:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:41 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:27:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:41 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-librbd-smithi136-87100-32", "overlaypool":"test-librbd-smithi136-87100-34"}]': finished 2024-08-17T17:27:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:41 smithi165 ceph-mon[105517]: osdmap e725: 8 total, 8 up, 8 in 2024-08-17T17:27:42.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:41 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-librbd-smithi136-87100-32", "overlaypool":"test-librbd-smithi136-87100-34"} : dispatch 2024-08-17T17:27:42.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:41 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:27:42.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:41 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-librbd-smithi136-87100-32", "overlaypool":"test-librbd-smithi136-87100-34"}]': finished 2024-08-17T17:27:42.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:41 smithi136 ceph-mon[109861]: osdmap e725: 8 total, 8 up, 8 in 2024-08-17T17:27:42.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:41 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-librbd-smithi136-87100-32", "overlaypool":"test-librbd-smithi136-87100-34"} : dispatch 2024-08-17T17:27:42.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:41 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:27:42.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:41 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-librbd-smithi136-87100-32", "overlaypool":"test-librbd-smithi136-87100-34"}]': finished 2024-08-17T17:27:42.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:41 smithi136 ceph-mon[115078]: osdmap e725: 8 total, 8 up, 8 in 2024-08-17T17:27:43.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:42 smithi165 ceph-mon[105517]: pgmap v731: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:27:43.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:42 smithi136 ceph-mon[109861]: pgmap v731: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:27:43.465 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:42 smithi136 ceph-mon[115078]: pgmap v731: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:27:44.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:43 smithi165 ceph-mon[105517]: osdmap e726: 8 total, 8 up, 8 in 2024-08-17T17:27:44.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:43 smithi165 ceph-mon[105517]: osdmap e727: 8 total, 8 up, 8 in 2024-08-17T17:27:44.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:43 smithi136 ceph-mon[109861]: osdmap e726: 8 total, 8 up, 8 in 2024-08-17T17:27:44.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:43 smithi136 ceph-mon[109861]: osdmap e727: 8 total, 8 up, 8 in 2024-08-17T17:27:44.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:43 smithi136 ceph-mon[115078]: osdmap e726: 8 total, 8 up, 8 in 2024-08-17T17:27:44.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:43 smithi136 ceph-mon[115078]: osdmap e727: 8 total, 8 up, 8 in 2024-08-17T17:27:45.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:44 smithi165 ceph-mon[105517]: pgmap v734: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:27:45.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:44 smithi165 ceph-mon[105517]: osdmap e728: 8 total, 8 up, 8 in 2024-08-17T17:27:45.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:44 smithi136 ceph-mon[109861]: pgmap v734: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:27:45.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:44 smithi136 ceph-mon[109861]: osdmap e728: 8 total, 8 up, 8 in 2024-08-17T17:27:45.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:44 smithi136 ceph-mon[115078]: pgmap v734: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:27:45.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:44 smithi136 ceph-mon[115078]: osdmap e728: 8 total, 8 up, 8 in 2024-08-17T17:27:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:46 smithi165 ceph-mon[105517]: osdmap e729: 8 total, 8 up, 8 in 2024-08-17T17:27:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:46 smithi165 ceph-mon[105517]: pgmap v737: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-08-17T17:27:47.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:46 smithi136 ceph-mon[109861]: osdmap e729: 8 total, 8 up, 8 in 2024-08-17T17:27:47.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:46 smithi136 ceph-mon[109861]: pgmap v737: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-08-17T17:27:47.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:46 smithi136 ceph-mon[115078]: osdmap e729: 8 total, 8 up, 8 in 2024-08-17T17:27:47.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:46 smithi136 ceph-mon[115078]: pgmap v737: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-08-17T17:27:47.509 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.509 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-33/image130 2024-08-17T17:27:47.509 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-33/image130 2024-08-17T17:27:47.509 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image130 2024-08-17T17:27:47.509 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.509 INFO:tasks.workunit.client.0.smithi136.stdout:num children is: 1 2024-08-17T17:27:47.509 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 1 2024-08-17T17:27:47.509 INFO:tasks.workunit.client.0.smithi136.stdout:child: image130 2024-08-17T17:27:47.510 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image130/acd6d63091c2 2024-08-17T17:27:47.510 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.510 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.510 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-33/image130 2024-08-17T17:27:47.510 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:47.510 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-33/image130 2024-08-17T17:27:47.510 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image130 2024-08-17T17:27:47.510 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.510 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.510 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:47.510 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:47.510 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:47.510 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.511 INFO:tasks.workunit.client.0.smithi136.stdout:num children is: 2 2024-08-17T17:27:47.511 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 2 2024-08-17T17:27:47.511 INFO:tasks.workunit.client.0.smithi136.stdout:child: image131 2024-08-17T17:27:47.511 INFO:tasks.workunit.client.0.smithi136.stdout:child: image130 2024-08-17T17:27:47.511 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image130/acd6d63091c2 2024-08-17T17:27:47.511 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.511 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-32/image131/acd63f227902 2024-08-17T17:27:47.511 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.511 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.511 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-33/image130 2024-08-17T17:27:47.511 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:47.511 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-33/image130 2024-08-17T17:27:47.511 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image130 2024-08-17T17:27:47.512 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.512 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.512 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:47.512 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:47.512 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:47.512 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.512 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.512 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-33/image132 2024-08-17T17:27:47.512 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:47.512 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-33/image130 2024-08-17T17:27:47.512 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-33/image132 2024-08-17T17:27:47.512 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image132 2024-08-17T17:27:47.512 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.512 INFO:tasks.workunit.client.0.smithi136.stdout:num children is: 3 2024-08-17T17:27:47.513 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 3 2024-08-17T17:27:47.513 INFO:tasks.workunit.client.0.smithi136.stdout:child: image131 2024-08-17T17:27:47.513 INFO:tasks.workunit.client.0.smithi136.stdout:child: image130 2024-08-17T17:27:47.513 INFO:tasks.workunit.client.0.smithi136.stdout:child: image132 2024-08-17T17:27:47.513 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image130/acd6d63091c2 2024-08-17T17:27:47.513 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.513 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-32/image131/acd63f227902 2024-08-17T17:27:47.513 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.513 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image132/acd6fdb05ac0 2024-08-17T17:27:47.513 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.513 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.513 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-33/image130 2024-08-17T17:27:47.513 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:47.513 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-33/image130 2024-08-17T17:27:47.514 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image130 2024-08-17T17:27:47.514 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.514 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.514 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:47.514 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:47.514 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:47.514 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.514 INFO:tasks.workunit.client.0.smithi136.stdout:num children is: 3 2024-08-17T17:27:47.514 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 3 2024-08-17T17:27:47.514 INFO:tasks.workunit.client.0.smithi136.stdout:child: image131 2024-08-17T17:27:47.514 INFO:tasks.workunit.client.0.smithi136.stdout:child: image130 2024-08-17T17:27:47.514 INFO:tasks.workunit.client.0.smithi136.stdout:child: image132 2024-08-17T17:27:47.514 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image130/acd6d63091c2 2024-08-17T17:27:47.514 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.515 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-32/image131/acd63f227902 2024-08-17T17:27:47.515 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.515 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image132/acd6fdb05ac0 2024-08-17T17:27:47.515 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.515 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.515 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-33/image130 2024-08-17T17:27:47.515 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:47.515 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-33/image130 2024-08-17T17:27:47.515 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image130 2024-08-17T17:27:47.515 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.515 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.515 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:47.515 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:47.515 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:47.516 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.516 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.516 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-33/image133 2024-08-17T17:27:47.516 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:47.516 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-33/image130 2024-08-17T17:27:47.516 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-33/image133 2024-08-17T17:27:47.516 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image133 2024-08-17T17:27:47.516 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.516 INFO:tasks.workunit.client.0.smithi136.stdout:num children is: 4 2024-08-17T17:27:47.516 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 4 2024-08-17T17:27:47.516 INFO:tasks.workunit.client.0.smithi136.stdout:child: image131 2024-08-17T17:27:47.516 INFO:tasks.workunit.client.0.smithi136.stdout:child: image130 2024-08-17T17:27:47.516 INFO:tasks.workunit.client.0.smithi136.stdout:child: image132 2024-08-17T17:27:47.516 INFO:tasks.workunit.client.0.smithi136.stdout:child: image133 2024-08-17T17:27:47.516 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image130/acd6d63091c2 2024-08-17T17:27:47.517 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.517 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-32/image131/acd63f227902 2024-08-17T17:27:47.517 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.517 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image132/acd6fdb05ac0 2024-08-17T17:27:47.517 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.517 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image133/acd6f903b99f 2024-08-17T17:27:47.517 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.517 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.517 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-33/image130 2024-08-17T17:27:47.517 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:47.517 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-33/image130 2024-08-17T17:27:47.517 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image130 2024-08-17T17:27:47.518 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.518 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.518 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:47.518 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:47.518 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:47.518 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.518 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.518 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-33/image132 2024-08-17T17:27:47.518 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:47.518 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-33/image130 2024-08-17T17:27:47.519 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-33/image132 2024-08-17T17:27:47.519 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image132 2024-08-17T17:27:47.519 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.519 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:47.519 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-33/image133 2024-08-17T17:27:47.519 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:47.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:27:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:27:47] "GET /metrics HTTP/1.1" 200 79717 "" "Prometheus/2.51.0" 2024-08-17T17:27:48.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:48 smithi165 ceph-mon[105517]: osdmap e730: 8 total, 8 up, 8 in 2024-08-17T17:27:48.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:48 smithi136 ceph-mon[109861]: osdmap e730: 8 total, 8 up, 8 in 2024-08-17T17:27:48.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:48 smithi136 ceph-mon[115078]: osdmap e730: 8 total, 8 up, 8 in 2024-08-17T17:27:49.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:49 smithi136 ceph-mon[109861]: osdmap e731: 8 total, 8 up, 8 in 2024-08-17T17:27:49.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:49 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-librbd-smithi136-87100-32"} : dispatch 2024-08-17T17:27:49.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:49 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-librbd-smithi136-87100-32"} : dispatch 2024-08-17T17:27:49.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:49 smithi136 ceph-mon[109861]: pgmap v740: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 2.5 KiB/s wr, 42 op/s 2024-08-17T17:27:49.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:49 smithi136 ceph-mon[115078]: osdmap e731: 8 total, 8 up, 8 in 2024-08-17T17:27:49.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:49 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-librbd-smithi136-87100-32"} : dispatch 2024-08-17T17:27:49.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:49 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-librbd-smithi136-87100-32"} : dispatch 2024-08-17T17:27:49.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:49 smithi136 ceph-mon[115078]: pgmap v740: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 2.5 KiB/s wr, 42 op/s 2024-08-17T17:27:49.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:49 smithi165 ceph-mon[105517]: osdmap e731: 8 total, 8 up, 8 in 2024-08-17T17:27:49.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:49 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-librbd-smithi136-87100-32"} : dispatch 2024-08-17T17:27:49.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:49 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-librbd-smithi136-87100-32"} : dispatch 2024-08-17T17:27:49.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:49 smithi165 ceph-mon[105517]: pgmap v740: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 2.5 KiB/s wr, 42 op/s 2024-08-17T17:27:50.020 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-33/image130 2024-08-17T17:27:50.020 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-33/image132 2024-08-17T17:27:50.020 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-33/image133 2024-08-17T17:27:50.020 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image133 2024-08-17T17:27:50.020 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:50.020 INFO:tasks.workunit.client.0.smithi136.stdout:num children is: 4 2024-08-17T17:27:50.020 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 4 2024-08-17T17:27:50.020 INFO:tasks.workunit.client.0.smithi136.stdout:child: image131 2024-08-17T17:27:50.021 INFO:tasks.workunit.client.0.smithi136.stdout:child: image130 2024-08-17T17:27:50.021 INFO:tasks.workunit.client.0.smithi136.stdout:child: image132 2024-08-17T17:27:50.021 INFO:tasks.workunit.client.0.smithi136.stdout:child: image133 2024-08-17T17:27:50.021 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image130/acd6d63091c2 2024-08-17T17:27:50.021 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:50.021 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-32/image131/acd63f227902 2024-08-17T17:27:50.021 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:50.021 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image132/acd6fdb05ac0 2024-08-17T17:27:50.021 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:50.021 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image133/acd6f903b99f 2024-08-17T17:27:50.021 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:50.021 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:50.021 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:50.021 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:50.022 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:50.022 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:50.022 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:50.022 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-33/image132 2024-08-17T17:27:50.022 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:50.022 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-33/image132 2024-08-17T17:27:50.022 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image132 2024-08-17T17:27:50.022 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:50.022 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:50.022 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-33/image133 2024-08-17T17:27:50.022 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:50.022 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-33/image132 2024-08-17T17:27:50.023 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-33/image133 2024-08-17T17:27:50.023 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image133 2024-08-17T17:27:50.023 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:50.023 INFO:tasks.workunit.client.0.smithi136.stdout:num children is: 3 2024-08-17T17:27:50.023 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 3 2024-08-17T17:27:50.023 INFO:tasks.workunit.client.0.smithi136.stdout:child: image131 2024-08-17T17:27:50.023 INFO:tasks.workunit.client.0.smithi136.stdout:child: image132 2024-08-17T17:27:50.023 INFO:tasks.workunit.client.0.smithi136.stdout:child: image133 2024-08-17T17:27:50.023 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-32/image131/acd63f227902 2024-08-17T17:27:50.023 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:50.023 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image132/acd6fdb05ac0 2024-08-17T17:27:50.023 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:50.023 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image133/acd6f903b99f 2024-08-17T17:27:50.023 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:50.024 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:50.024 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:50.024 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:50.024 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:50.024 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:50.024 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:50.024 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-33/image133 2024-08-17T17:27:50.024 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:50.024 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-33/image133 2024-08-17T17:27:50.024 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image133 2024-08-17T17:27:50.024 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:50.024 INFO:tasks.workunit.client.0.smithi136.stdout:num children is: 2 2024-08-17T17:27:50.024 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 2 2024-08-17T17:27:50.024 INFO:tasks.workunit.client.0.smithi136.stdout:child: image131 2024-08-17T17:27:50.025 INFO:tasks.workunit.client.0.smithi136.stdout:child: image133 2024-08-17T17:27:50.025 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-32/image131/acd63f227902 2024-08-17T17:27:50.025 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:50.025 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-33/image133/acd6f903b99f 2024-08-17T17:27:50.025 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:50.025 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:50.025 INFO:tasks.workunit.client.0.smithi136.stdout:trying to find test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:50.025 INFO:tasks.workunit.client.0.smithi136.stdout:checking test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:50.025 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-32/image131 2024-08-17T17:27:50.025 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:50.025 INFO:tasks.workunit.client.0.smithi136.stdout:num children is: 1 2024-08-17T17:27:50.025 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 1 2024-08-17T17:27:50.025 INFO:tasks.workunit.client.0.smithi136.stdout:child: image131 2024-08-17T17:27:50.025 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-32/image131/acd63f227902 2024-08-17T17:27:50.025 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:27:50.026 INFO:tasks.workunit.client.0.smithi136.stdout:num children is: 0 2024-08-17T17:27:50.026 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 0 2024-08-17T17:27:50.026 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.ListChildrenTiered (16300 ms) 2024-08-17T17:27:50.026 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.LockingPP 2024-08-17T17:27:50.026 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:50.079 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.LockingPP (60 ms) 2024-08-17T17:27:50.079 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.FlushAio 2024-08-17T17:27:50.079 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:50.316 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.FlushAio (237 ms) 2024-08-17T17:27:50.316 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.FlushAioPP 2024-08-17T17:27:50.316 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:50.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:50 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-librbd-smithi136-87100-32"}]': finished 2024-08-17T17:27:50.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:50 smithi136 ceph-mon[115078]: osdmap e732: 8 total, 8 up, 8 in 2024-08-17T17:27:50.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:50 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-librbd-smithi136-87100-32", "tierpool":"test-librbd-smithi136-87100-34"} : dispatch 2024-08-17T17:27:50.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:50 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-librbd-smithi136-87100-32", "tierpool":"test-librbd-smithi136-87100-34"} : dispatch 2024-08-17T17:27:50.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:50 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-librbd-smithi136-87100-32"}]': finished 2024-08-17T17:27:50.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:50 smithi136 ceph-mon[109861]: osdmap e732: 8 total, 8 up, 8 in 2024-08-17T17:27:50.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:50 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-librbd-smithi136-87100-32", "tierpool":"test-librbd-smithi136-87100-34"} : dispatch 2024-08-17T17:27:50.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:50 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-librbd-smithi136-87100-32", "tierpool":"test-librbd-smithi136-87100-34"} : dispatch 2024-08-17T17:27:50.502 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.FlushAioPP (186 ms) 2024-08-17T17:27:50.502 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.ZeroLengthWrite 2024-08-17T17:27:50.502 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:50.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:50 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-librbd-smithi136-87100-32"}]': finished 2024-08-17T17:27:50.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:50 smithi165 ceph-mon[105517]: osdmap e732: 8 total, 8 up, 8 in 2024-08-17T17:27:50.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:50 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-librbd-smithi136-87100-32", "tierpool":"test-librbd-smithi136-87100-34"} : dispatch 2024-08-17T17:27:50.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:50 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-librbd-smithi136-87100-32", "tierpool":"test-librbd-smithi136-87100-34"} : dispatch 2024-08-17T17:27:50.523 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.ZeroLengthWrite (21 ms) 2024-08-17T17:27:50.523 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.ZeroLengthDiscard 2024-08-17T17:27:50.523 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:50.548 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.ZeroLengthDiscard (25 ms) 2024-08-17T17:27:50.549 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.ZeroLengthRead 2024-08-17T17:27:50.549 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:50.566 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.ZeroLengthRead (18 ms) 2024-08-17T17:27:50.567 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.LargeCacheRead 2024-08-17T17:27:50.567 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:50.601 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.LargeCacheRead (35 ms) 2024-08-17T17:27:50.601 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestPendingAio 2024-08-17T17:27:50.601 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:50.826 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestPendingAio (225 ms) 2024-08-17T17:27:50.827 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.CompareAndWriteCopyup 2024-08-17T17:27:50.827 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:51.124 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:51.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:51 smithi136 ceph-mon[109861]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-08-17T17:27:51.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:51 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-librbd-smithi136-87100-32", "tierpool":"test-librbd-smithi136-87100-34"}]': finished 2024-08-17T17:27:51.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:51 smithi136 ceph-mon[109861]: osdmap e733: 8 total, 8 up, 8 in 2024-08-17T17:27:51.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:51 smithi136 ceph-mon[109861]: pgmap v743: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 2.2 KiB/s wr, 39 op/s 2024-08-17T17:27:51.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:51 smithi136 ceph-mon[115078]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-08-17T17:27:51.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:51 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-librbd-smithi136-87100-32", "tierpool":"test-librbd-smithi136-87100-34"}]': finished 2024-08-17T17:27:51.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:51 smithi136 ceph-mon[115078]: osdmap e733: 8 total, 8 up, 8 in 2024-08-17T17:27:51.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:51 smithi136 ceph-mon[115078]: pgmap v743: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 2.2 KiB/s wr, 39 op/s 2024-08-17T17:27:51.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:51 smithi165 ceph-mon[105517]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-08-17T17:27:51.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:51 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-librbd-smithi136-87100-32", "tierpool":"test-librbd-smithi136-87100-34"}]': finished 2024-08-17T17:27:51.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:51 smithi165 ceph-mon[105517]: osdmap e733: 8 total, 8 up, 8 in 2024-08-17T17:27:51.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:51 smithi165 ceph-mon[105517]: pgmap v743: 713 pgs: 713 active+clean; 297 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 2.2 KiB/s wr, 39 op/s 2024-08-17T17:27:52.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:52 smithi136 ceph-mon[109861]: osdmap e734: 8 total, 8 up, 8 in 2024-08-17T17:27:52.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:27:52.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:27:52.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:52 smithi136 ceph-mon[115078]: osdmap e734: 8 total, 8 up, 8 in 2024-08-17T17:27:52.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:27:52.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:27:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:52 smithi165 ceph-mon[105517]: osdmap e734: 8 total, 8 up, 8 in 2024-08-17T17:27:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:27:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:27:53.104 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.CompareAndWriteCopyup (2278 ms) 2024-08-17T17:27:53.105 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.CompareAndWriteCopyupMismatch 2024-08-17T17:27:53.105 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:53.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:53 smithi136 ceph-mon[109861]: osdmap e735: 8 total, 8 up, 8 in 2024-08-17T17:27:53.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:53 smithi136 ceph-mon[109861]: pgmap v746: 713 pgs: 713 active+clean; 301 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s rd, 669 KiB/s wr, 1.04k op/s 2024-08-17T17:27:53.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:53 smithi136 ceph-mon[115078]: osdmap e735: 8 total, 8 up, 8 in 2024-08-17T17:27:53.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:53 smithi136 ceph-mon[115078]: pgmap v746: 713 pgs: 713 active+clean; 301 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s rd, 669 KiB/s wr, 1.04k op/s 2024-08-17T17:27:53.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:53 smithi165 ceph-mon[105517]: osdmap e735: 8 total, 8 up, 8 in 2024-08-17T17:27:53.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:53 smithi165 ceph-mon[105517]: pgmap v746: 713 pgs: 713 active+clean; 301 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s rd, 669 KiB/s wr, 1.04k op/s 2024-08-17T17:27:54.043 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:54.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:54 smithi136 ceph-mon[109861]: osdmap e736: 8 total, 8 up, 8 in 2024-08-17T17:27:54.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:54 smithi136 ceph-mon[109861]: osdmap e737: 8 total, 8 up, 8 in 2024-08-17T17:27:54.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:54 smithi136 ceph-mon[115078]: osdmap e736: 8 total, 8 up, 8 in 2024-08-17T17:27:54.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:54 smithi136 ceph-mon[115078]: osdmap e737: 8 total, 8 up, 8 in 2024-08-17T17:27:54.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:54 smithi165 ceph-mon[105517]: osdmap e736: 8 total, 8 up, 8 in 2024-08-17T17:27:54.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:54 smithi165 ceph-mon[105517]: osdmap e737: 8 total, 8 up, 8 in 2024-08-17T17:27:55.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:55 smithi136 ceph-mon[109861]: pgmap v749: 713 pgs: 713 active+clean; 301 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s rd, 669 KiB/s wr, 1.04k op/s 2024-08-17T17:27:55.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:55 smithi136 ceph-mon[115078]: pgmap v749: 713 pgs: 713 active+clean; 301 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s rd, 669 KiB/s wr, 1.04k op/s 2024-08-17T17:27:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:55 smithi165 ceph-mon[105517]: pgmap v749: 713 pgs: 713 active+clean; 301 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 MiB/s rd, 669 KiB/s wr, 1.04k op/s 2024-08-17T17:27:56.115 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.CompareAndWriteCopyupMismatch (3011 ms) 2024-08-17T17:27:56.115 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.Flatten 2024-08-17T17:27:56.116 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:56.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:56 smithi136 ceph-mon[109861]: osdmap e738: 8 total, 8 up, 8 in 2024-08-17T17:27:56.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:56 smithi136 ceph-mon[115078]: osdmap e738: 8 total, 8 up, 8 in 2024-08-17T17:27:56.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:56 smithi165 ceph-mon[105517]: osdmap e738: 8 total, 8 up, 8 in 2024-08-17T17:27:57.170 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.Flatten (1055 ms) 2024-08-17T17:27:57.171 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.Sparsify 2024-08-17T17:27:57.171 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:57.211 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 32768 2024-08-17T17:27:57.211 INFO:tasks.workunit.client.0.smithi136.stdout:read: 32768 2024-08-17T17:27:57.211 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.Sparsify (41 ms) 2024-08-17T17:27:57.211 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.SparsifyPP 2024-08-17T17:27:57.211 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:57.241 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.SparsifyPP (30 ms) 2024-08-17T17:27:57.241 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.SnapshotLimit 2024-08-17T17:27:57.241 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:27:57.412 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:57 smithi165 ceph-mon[105517]: osdmap e739: 8 total, 8 up, 8 in 2024-08-17T17:27:57.412 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:57 smithi165 ceph-mon[105517]: pgmap v752: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 143 KiB/s rd, 77 KiB/s wr, 238 op/s 2024-08-17T17:27:57.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:57 smithi136 ceph-mon[109861]: osdmap e739: 8 total, 8 up, 8 in 2024-08-17T17:27:57.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:57 smithi136 ceph-mon[109861]: pgmap v752: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 143 KiB/s rd, 77 KiB/s wr, 238 op/s 2024-08-17T17:27:57.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:57 smithi136 ceph-mon[115078]: osdmap e739: 8 total, 8 up, 8 in 2024-08-17T17:27:57.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:57 smithi136 ceph-mon[115078]: pgmap v752: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 143 KiB/s rd, 77 KiB/s wr, 238 op/s 2024-08-17T17:27:57.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:27:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:27:57] "GET /metrics HTTP/1.1" 200 82164 "" "Prometheus/2.51.0" 2024-08-17T17:27:58.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:58 smithi136 ceph-mon[109861]: osdmap e740: 8 total, 8 up, 8 in 2024-08-17T17:27:58.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:58 smithi136 ceph-mon[115078]: osdmap e740: 8 total, 8 up, 8 in 2024-08-17T17:27:58.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:58 smithi165 ceph-mon[105517]: osdmap e740: 8 total, 8 up, 8 in 2024-08-17T17:27:59.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:59 smithi136 ceph-mon[109861]: osdmap e741: 8 total, 8 up, 8 in 2024-08-17T17:27:59.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:27:59 smithi136 ceph-mon[109861]: pgmap v755: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 277 KiB/s rd, 90 KiB/s wr, 418 op/s 2024-08-17T17:27:59.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:59 smithi136 ceph-mon[115078]: osdmap e741: 8 total, 8 up, 8 in 2024-08-17T17:27:59.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:27:59 smithi136 ceph-mon[115078]: pgmap v755: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 277 KiB/s rd, 90 KiB/s wr, 418 op/s 2024-08-17T17:27:59.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:59 smithi165 ceph-mon[105517]: osdmap e741: 8 total, 8 up, 8 in 2024-08-17T17:27:59.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:27:59 smithi165 ceph-mon[105517]: pgmap v755: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 277 KiB/s rd, 90 KiB/s wr, 418 op/s 2024-08-17T17:28:00.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:00 smithi136 ceph-mon[109861]: osdmap e742: 8 total, 8 up, 8 in 2024-08-17T17:28:00.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:00 smithi136 ceph-mon[115078]: osdmap e742: 8 total, 8 up, 8 in 2024-08-17T17:28:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:00 smithi165 ceph-mon[105517]: osdmap e742: 8 total, 8 up, 8 in 2024-08-17T17:28:01.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:01 smithi136 ceph-mon[109861]: osdmap e743: 8 total, 8 up, 8 in 2024-08-17T17:28:01.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:01 smithi136 ceph-mon[109861]: pgmap v758: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 13 KiB/s wr, 180 op/s 2024-08-17T17:28:01.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:01 smithi136 ceph-mon[115078]: osdmap e743: 8 total, 8 up, 8 in 2024-08-17T17:28:01.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:01 smithi136 ceph-mon[115078]: pgmap v758: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 13 KiB/s wr, 180 op/s 2024-08-17T17:28:01.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:01 smithi165 ceph-mon[105517]: osdmap e743: 8 total, 8 up, 8 in 2024-08-17T17:28:01.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:01 smithi165 ceph-mon[105517]: pgmap v758: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 134 KiB/s rd, 13 KiB/s wr, 180 op/s 2024-08-17T17:28:02.153 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.SnapshotLimit (4911 ms) 2024-08-17T17:28:02.153 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.SnapshotLimitPP 2024-08-17T17:28:02.153 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:02.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:02 smithi136 ceph-mon[109861]: osdmap e744: 8 total, 8 up, 8 in 2024-08-17T17:28:02.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:02 smithi136 ceph-mon[115078]: osdmap e744: 8 total, 8 up, 8 in 2024-08-17T17:28:02.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:02 smithi165 ceph-mon[105517]: osdmap e744: 8 total, 8 up, 8 in 2024-08-17T17:28:03.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:03 smithi136 ceph-mon[109861]: pgmap v760: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 142 KiB/s rd, 31 KiB/s wr, 171 op/s 2024-08-17T17:28:03.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:03 smithi136 ceph-mon[109861]: osdmap e745: 8 total, 8 up, 8 in 2024-08-17T17:28:03.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:03 smithi136 ceph-mon[115078]: pgmap v760: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 142 KiB/s rd, 31 KiB/s wr, 171 op/s 2024-08-17T17:28:03.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:03 smithi136 ceph-mon[115078]: osdmap e745: 8 total, 8 up, 8 in 2024-08-17T17:28:03.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:03 smithi165 ceph-mon[105517]: pgmap v760: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 142 KiB/s rd, 31 KiB/s wr, 171 op/s 2024-08-17T17:28:03.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:03 smithi165 ceph-mon[105517]: osdmap e745: 8 total, 8 up, 8 in 2024-08-17T17:28:04.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:04 smithi136 ceph-mon[115078]: osdmap e746: 8 total, 8 up, 8 in 2024-08-17T17:28:04.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:04 smithi136 ceph-mon[109861]: osdmap e746: 8 total, 8 up, 8 in 2024-08-17T17:28:04.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:04 smithi165 ceph-mon[105517]: osdmap e746: 8 total, 8 up, 8 in 2024-08-17T17:28:05.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:05 smithi136 ceph-mon[115078]: pgmap v763: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 143 KiB/s rd, 31 KiB/s wr, 171 op/s 2024-08-17T17:28:05.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:05 smithi136 ceph-mon[115078]: osdmap e747: 8 total, 8 up, 8 in 2024-08-17T17:28:05.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:05 smithi136 ceph-mon[109861]: pgmap v763: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 143 KiB/s rd, 31 KiB/s wr, 171 op/s 2024-08-17T17:28:05.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:05 smithi136 ceph-mon[109861]: osdmap e747: 8 total, 8 up, 8 in 2024-08-17T17:28:05.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:05 smithi165 ceph-mon[105517]: pgmap v763: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 143 KiB/s rd, 31 KiB/s wr, 171 op/s 2024-08-17T17:28:05.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:05 smithi165 ceph-mon[105517]: osdmap e747: 8 total, 8 up, 8 in 2024-08-17T17:28:06.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:06 smithi136 ceph-mon[115078]: osdmap e748: 8 total, 8 up, 8 in 2024-08-17T17:28:06.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:28:06.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:06 smithi136 ceph-mon[109861]: osdmap e748: 8 total, 8 up, 8 in 2024-08-17T17:28:06.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:28:06.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:06 smithi165 ceph-mon[105517]: osdmap e748: 8 total, 8 up, 8 in 2024-08-17T17:28:06.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:28:07.240 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.SnapshotLimitPP (5088 ms) 2024-08-17T17:28:07.240 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.RebuildObjectMapViaLockOwner 2024-08-17T17:28:07.241 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:07.291 INFO:tasks.workunit.client.0.smithi136.stdout:0.00% done 2024-08-17T17:28:07.292 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.RebuildObjectMapViaLockOwner (51 ms) 2024-08-17T17:28:07.292 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.RenameViaLockOwner 2024-08-17T17:28:07.292 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:07.357 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.RenameViaLockOwner (66 ms) 2024-08-17T17:28:07.358 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.SnapCreateViaLockOwner 2024-08-17T17:28:07.358 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:07.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-mon[109861]: pgmap v766: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 1.7 KiB/s wr, 28 op/s 2024-08-17T17:28:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-mon[109861]: osdmap e749: 8 total, 8 up, 8 in 2024-08-17T17:28:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.c", "id": [2, 4]} : dispatch 2024-08-17T17:28:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.14", "id": [2, 5]} : dispatch 2024-08-17T17:28:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.3", "id": [0, 3]} : dispatch 2024-08-17T17:28:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.4", "id": [0, 3]} : dispatch 2024-08-17T17:28:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.1", "id": [0, 1]} : dispatch 2024-08-17T17:28:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.4", "id": [7, 4]} : dispatch 2024-08-17T17:28:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.c", "id": [0, 1]} : dispatch 2024-08-17T17:28:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.f", "id": [0, 4]} : dispatch 2024-08-17T17:28:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.12", "id": [7, 3]} : dispatch 2024-08-17T17:28:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.15", "id": [0, 3]} : dispatch 2024-08-17T17:28:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-mon[115078]: pgmap v766: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 1.7 KiB/s wr, 28 op/s 2024-08-17T17:28:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-mon[115078]: osdmap e749: 8 total, 8 up, 8 in 2024-08-17T17:28:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.c", "id": [2, 4]} : dispatch 2024-08-17T17:28:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.14", "id": [2, 5]} : dispatch 2024-08-17T17:28:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.3", "id": [0, 3]} : dispatch 2024-08-17T17:28:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.4", "id": [0, 3]} : dispatch 2024-08-17T17:28:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.1", "id": [0, 1]} : dispatch 2024-08-17T17:28:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.4", "id": [7, 4]} : dispatch 2024-08-17T17:28:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.c", "id": [0, 1]} : dispatch 2024-08-17T17:28:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.f", "id": [0, 4]} : dispatch 2024-08-17T17:28:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.12", "id": [7, 3]} : dispatch 2024-08-17T17:28:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.15", "id": [0, 3]} : dispatch 2024-08-17T17:28:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:07 smithi165 ceph-mon[105517]: pgmap v766: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 1.7 KiB/s wr, 28 op/s 2024-08-17T17:28:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:07 smithi165 ceph-mon[105517]: osdmap e749: 8 total, 8 up, 8 in 2024-08-17T17:28:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.c", "id": [2, 4]} : dispatch 2024-08-17T17:28:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.14", "id": [2, 5]} : dispatch 2024-08-17T17:28:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.3", "id": [0, 3]} : dispatch 2024-08-17T17:28:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.4", "id": [0, 3]} : dispatch 2024-08-17T17:28:07.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.1", "id": [0, 1]} : dispatch 2024-08-17T17:28:07.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.4", "id": [7, 4]} : dispatch 2024-08-17T17:28:07.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.c", "id": [0, 1]} : dispatch 2024-08-17T17:28:07.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.f", "id": [0, 4]} : dispatch 2024-08-17T17:28:07.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.12", "id": [7, 3]} : dispatch 2024-08-17T17:28:07.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.15", "id": [0, 3]} : dispatch 2024-08-17T17:28:07.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:28:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:28:07] "GET /metrics HTTP/1.1" 200 82154 "" "Prometheus/2.51.0" 2024-08-17T17:28:08.257 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.SnapCreateViaLockOwner (900 ms) 2024-08-17T17:28:08.257 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.SnapRemoveViaLockOwner 2024-08-17T17:28:08.258 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:08.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.c", "id": [2, 4]}]': finished 2024-08-17T17:28:08.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.14", "id": [2, 5]}]': finished 2024-08-17T17:28:08.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.3", "id": [0, 3]}]': finished 2024-08-17T17:28:08.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.4", "id": [0, 3]}]': finished 2024-08-17T17:28:08.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.1", "id": [0, 1]}]': finished 2024-08-17T17:28:08.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.4", "id": [7, 4]}]': finished 2024-08-17T17:28:08.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.c", "id": [0, 1]}]': finished 2024-08-17T17:28:08.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.f", "id": [0, 4]}]': finished 2024-08-17T17:28:08.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.12", "id": [7, 3]}]': finished 2024-08-17T17:28:08.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.15", "id": [0, 3]}]': finished 2024-08-17T17:28:08.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:08 smithi165 ceph-mon[105517]: osdmap e750: 8 total, 8 up, 8 in 2024-08-17T17:28:08.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.c", "id": [2, 4]}]': finished 2024-08-17T17:28:08.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.14", "id": [2, 5]}]': finished 2024-08-17T17:28:08.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.3", "id": [0, 3]}]': finished 2024-08-17T17:28:08.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.4", "id": [0, 3]}]': finished 2024-08-17T17:28:08.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.1", "id": [0, 1]}]': finished 2024-08-17T17:28:08.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.4", "id": [7, 4]}]': finished 2024-08-17T17:28:08.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.c", "id": [0, 1]}]': finished 2024-08-17T17:28:08.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.f", "id": [0, 4]}]': finished 2024-08-17T17:28:08.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.12", "id": [7, 3]}]': finished 2024-08-17T17:28:08.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.15", "id": [0, 3]}]': finished 2024-08-17T17:28:08.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:08 smithi136 ceph-mon[109861]: osdmap e750: 8 total, 8 up, 8 in 2024-08-17T17:28:08.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.c", "id": [2, 4]}]': finished 2024-08-17T17:28:08.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.14", "id": [2, 5]}]': finished 2024-08-17T17:28:08.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.3", "id": [0, 3]}]': finished 2024-08-17T17:28:08.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.4", "id": [0, 3]}]': finished 2024-08-17T17:28:08.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.1", "id": [0, 1]}]': finished 2024-08-17T17:28:08.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.4", "id": [7, 4]}]': finished 2024-08-17T17:28:08.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.c", "id": [0, 1]}]': finished 2024-08-17T17:28:08.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.f", "id": [0, 4]}]': finished 2024-08-17T17:28:08.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.12", "id": [7, 3]}]': finished 2024-08-17T17:28:08.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.15", "id": [0, 3]}]': finished 2024-08-17T17:28:08.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:08 smithi136 ceph-mon[115078]: osdmap e750: 8 total, 8 up, 8 in 2024-08-17T17:28:09.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:09 smithi165 ceph-mon[105517]: pgmap v769: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 3.5 KiB/s wr, 37 op/s 2024-08-17T17:28:09.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:09 smithi165 ceph-mon[105517]: osdmap e751: 8 total, 8 up, 8 in 2024-08-17T17:28:09.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:09 smithi165 ceph-mon[105517]: osdmap e752: 8 total, 8 up, 8 in 2024-08-17T17:28:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:09 smithi136 ceph-mon[109861]: pgmap v769: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 3.5 KiB/s wr, 37 op/s 2024-08-17T17:28:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:09 smithi136 ceph-mon[109861]: osdmap e751: 8 total, 8 up, 8 in 2024-08-17T17:28:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:09 smithi136 ceph-mon[109861]: osdmap e752: 8 total, 8 up, 8 in 2024-08-17T17:28:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:09 smithi136 ceph-mon[115078]: pgmap v769: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 3.5 KiB/s wr, 37 op/s 2024-08-17T17:28:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:09 smithi136 ceph-mon[115078]: osdmap e751: 8 total, 8 up, 8 in 2024-08-17T17:28:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:09 smithi136 ceph-mon[115078]: osdmap e752: 8 total, 8 up, 8 in 2024-08-17T17:28:09.971 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.SnapRemoveViaLockOwner (1714 ms) 2024-08-17T17:28:09.971 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.UpdateFeaturesViaLockOwner 2024-08-17T17:28:09.971 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:10.020 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.UpdateFeaturesViaLockOwner (49 ms) 2024-08-17T17:28:10.021 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.EnableJournalingViaLockOwner 2024-08-17T17:28:10.021 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_librbd.cc:9736: Skipped 2024-08-17T17:28:10.021 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:28:10.021 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestLibRBD.EnableJournalingViaLockOwner (0 ms) 2024-08-17T17:28:10.021 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.SnapRemove2 2024-08-17T17:28:10.021 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:11.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:10 smithi136 ceph-mon[109861]: osdmap e753: 8 total, 8 up, 8 in 2024-08-17T17:28:11.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:10 smithi136 ceph-mon[109861]: pgmap v773: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-08-17T17:28:11.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:10 smithi136 ceph-mon[115078]: osdmap e753: 8 total, 8 up, 8 in 2024-08-17T17:28:11.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:10 smithi136 ceph-mon[115078]: pgmap v773: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-08-17T17:28:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:10 smithi165 ceph-mon[105517]: osdmap e753: 8 total, 8 up, 8 in 2024-08-17T17:28:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:10 smithi165 ceph-mon[105517]: pgmap v773: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-08-17T17:28:11.975 INFO:tasks.workunit.client.0.smithi136.stdout:100.00% done 2024-08-17T17:28:11.975 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.SnapRemove2 (1955 ms) 2024-08-17T17:28:11.975 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.SnapRenameViaLockOwner 2024-08-17T17:28:11.976 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:12.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:11 smithi165 ceph-mon[105517]: osdmap e754: 8 total, 8 up, 8 in 2024-08-17T17:28:12.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:11 smithi136 ceph-mon[109861]: osdmap e754: 8 total, 8 up, 8 in 2024-08-17T17:28:12.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:11 smithi136 ceph-mon[115078]: osdmap e754: 8 total, 8 up, 8 in 2024-08-17T17:28:13.007 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.SnapRenameViaLockOwner (1031 ms) 2024-08-17T17:28:13.007 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.SnapProtectViaLockOwner 2024-08-17T17:28:13.007 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:13.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:12 smithi165 ceph-mon[105517]: osdmap e755: 8 total, 8 up, 8 in 2024-08-17T17:28:13.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:12 smithi165 ceph-mon[105517]: pgmap v776: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 180 KiB/s rd, 16 KiB/s wr, 249 op/s; 0 B/s, 0 objects/s recovering 2024-08-17T17:28:13.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:12 smithi136 ceph-mon[109861]: osdmap e755: 8 total, 8 up, 8 in 2024-08-17T17:28:13.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:12 smithi136 ceph-mon[109861]: pgmap v776: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 180 KiB/s rd, 16 KiB/s wr, 249 op/s; 0 B/s, 0 objects/s recovering 2024-08-17T17:28:13.348 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:12 smithi136 ceph-mon[115078]: osdmap e755: 8 total, 8 up, 8 in 2024-08-17T17:28:13.348 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:12 smithi136 ceph-mon[115078]: pgmap v776: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 180 KiB/s rd, 16 KiB/s wr, 249 op/s; 0 B/s, 0 objects/s recovering 2024-08-17T17:28:13.984 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.SnapProtectViaLockOwner (977 ms) 2024-08-17T17:28:13.984 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.SnapUnprotectViaLockOwner 2024-08-17T17:28:13.984 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:13 smithi165 ceph-mon[105517]: osdmap e756: 8 total, 8 up, 8 in 2024-08-17T17:28:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:13 smithi165 ceph-mon[105517]: osdmap e757: 8 total, 8 up, 8 in 2024-08-17T17:28:14.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:13 smithi136 ceph-mon[109861]: osdmap e756: 8 total, 8 up, 8 in 2024-08-17T17:28:14.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:13 smithi136 ceph-mon[109861]: osdmap e757: 8 total, 8 up, 8 in 2024-08-17T17:28:14.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:13 smithi136 ceph-mon[115078]: osdmap e756: 8 total, 8 up, 8 in 2024-08-17T17:28:14.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:13 smithi136 ceph-mon[115078]: osdmap e757: 8 total, 8 up, 8 in 2024-08-17T17:28:15.021 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.SnapUnprotectViaLockOwner (1038 ms) 2024-08-17T17:28:15.022 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.FlattenViaLockOwner 2024-08-17T17:28:15.022 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:15.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:14 smithi165 ceph-mon[105517]: pgmap v779: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 176 KiB/s rd, 16 KiB/s wr, 244 op/s; 0 B/s, 0 objects/s recovering 2024-08-17T17:28:15.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:14 smithi136 ceph-mon[109861]: pgmap v779: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 176 KiB/s rd, 16 KiB/s wr, 244 op/s; 0 B/s, 0 objects/s recovering 2024-08-17T17:28:15.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:14 smithi136 ceph-mon[115078]: pgmap v779: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 176 KiB/s rd, 16 KiB/s wr, 244 op/s; 0 B/s, 0 objects/s recovering 2024-08-17T17:28:16.082 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.FlattenViaLockOwner (1061 ms) 2024-08-17T17:28:16.082 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.ResizeViaLockOwner 2024-08-17T17:28:16.082 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:16.119 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.ResizeViaLockOwner (37 ms) 2024-08-17T17:28:16.120 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.SparsifyViaLockOwner 2024-08-17T17:28:16.120 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:16.158 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.SparsifyViaLockOwner (39 ms) 2024-08-17T17:28:16.158 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.ObjectMapConsistentSnap 2024-08-17T17:28:16.158 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:16.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:15 smithi165 ceph-mon[105517]: osdmap e758: 8 total, 8 up, 8 in 2024-08-17T17:28:16.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:15 smithi136 ceph-mon[109861]: osdmap e758: 8 total, 8 up, 8 in 2024-08-17T17:28:16.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:15 smithi136 ceph-mon[115078]: osdmap e758: 8 total, 8 up, 8 in 2024-08-17T17:28:17.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:16 smithi165 ceph-mon[105517]: osdmap e759: 8 total, 8 up, 8 in 2024-08-17T17:28:17.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:16 smithi165 ceph-mon[105517]: pgmap v782: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 57 KiB/s rd, 4.5 KiB/s wr, 78 op/s 2024-08-17T17:28:17.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:16 smithi136 ceph-mon[109861]: osdmap e759: 8 total, 8 up, 8 in 2024-08-17T17:28:17.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:16 smithi136 ceph-mon[109861]: pgmap v782: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 57 KiB/s rd, 4.5 KiB/s wr, 78 op/s 2024-08-17T17:28:17.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:16 smithi136 ceph-mon[115078]: osdmap e759: 8 total, 8 up, 8 in 2024-08-17T17:28:17.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:16 smithi136 ceph-mon[115078]: pgmap v782: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 57 KiB/s rd, 4.5 KiB/s wr, 78 op/s 2024-08-17T17:28:17.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:28:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:28:17] "GET /metrics HTTP/1.1" 200 82154 "" "Prometheus/2.51.0" 2024-08-17T17:28:18.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:17 smithi165 ceph-mon[105517]: osdmap e760: 8 total, 8 up, 8 in 2024-08-17T17:28:18.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:17 smithi136 ceph-mon[109861]: osdmap e760: 8 total, 8 up, 8 in 2024-08-17T17:28:18.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:17 smithi136 ceph-mon[115078]: osdmap e760: 8 total, 8 up, 8 in 2024-08-17T17:28:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:19 smithi165 ceph-mon[105517]: osdmap e761: 8 total, 8 up, 8 in 2024-08-17T17:28:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:19 smithi165 ceph-mon[105517]: pgmap v785: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 223 KiB/s rd, 13 KiB/s wr, 290 op/s 2024-08-17T17:28:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:19 smithi165 ceph-mon[105517]: osdmap e762: 8 total, 8 up, 8 in 2024-08-17T17:28:19.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:19 smithi136 ceph-mon[109861]: osdmap e761: 8 total, 8 up, 8 in 2024-08-17T17:28:19.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:19 smithi136 ceph-mon[109861]: pgmap v785: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 223 KiB/s rd, 13 KiB/s wr, 290 op/s 2024-08-17T17:28:19.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:19 smithi136 ceph-mon[109861]: osdmap e762: 8 total, 8 up, 8 in 2024-08-17T17:28:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:19 smithi136 ceph-mon[115078]: osdmap e761: 8 total, 8 up, 8 in 2024-08-17T17:28:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:19 smithi136 ceph-mon[115078]: pgmap v785: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 223 KiB/s rd, 13 KiB/s wr, 290 op/s 2024-08-17T17:28:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:19 smithi136 ceph-mon[115078]: osdmap e762: 8 total, 8 up, 8 in 2024-08-17T17:28:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:20 smithi165 ceph-mon[105517]: osdmap e763: 8 total, 8 up, 8 in 2024-08-17T17:28:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:20 smithi165 ceph-mon[105517]: pgmap v788: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 166 KiB/s rd, 8.5 KiB/s wr, 211 op/s 2024-08-17T17:28:21.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:20 smithi136 ceph-mon[109861]: osdmap e763: 8 total, 8 up, 8 in 2024-08-17T17:28:21.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:20 smithi136 ceph-mon[109861]: pgmap v788: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 166 KiB/s rd, 8.5 KiB/s wr, 211 op/s 2024-08-17T17:28:21.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:20 smithi136 ceph-mon[115078]: osdmap e763: 8 total, 8 up, 8 in 2024-08-17T17:28:21.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:20 smithi136 ceph-mon[115078]: pgmap v788: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 166 KiB/s rd, 8.5 KiB/s wr, 211 op/s 2024-08-17T17:28:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:21 smithi165 ceph-mon[105517]: osdmap e764: 8 total, 8 up, 8 in 2024-08-17T17:28:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:28:22.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:21 smithi136 ceph-mon[115078]: osdmap e764: 8 total, 8 up, 8 in 2024-08-17T17:28:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:28:22.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:21 smithi136 ceph-mon[109861]: osdmap e764: 8 total, 8 up, 8 in 2024-08-17T17:28:22.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:28:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:22 smithi165 ceph-mon[105517]: osdmap e765: 8 total, 8 up, 8 in 2024-08-17T17:28:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:22 smithi165 ceph-mon[105517]: pgmap v791: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 75 KiB/s rd, 6.2 KiB/s wr, 105 op/s 2024-08-17T17:28:23.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:22 smithi136 ceph-mon[115078]: osdmap e765: 8 total, 8 up, 8 in 2024-08-17T17:28:23.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:22 smithi136 ceph-mon[115078]: pgmap v791: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 75 KiB/s rd, 6.2 KiB/s wr, 105 op/s 2024-08-17T17:28:23.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:22 smithi136 ceph-mon[109861]: osdmap e765: 8 total, 8 up, 8 in 2024-08-17T17:28:23.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:22 smithi136 ceph-mon[109861]: pgmap v791: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 75 KiB/s rd, 6.2 KiB/s wr, 105 op/s 2024-08-17T17:28:24.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:23 smithi165 ceph-mon[105517]: osdmap e766: 8 total, 8 up, 8 in 2024-08-17T17:28:24.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:23 smithi136 ceph-mon[115078]: osdmap e766: 8 total, 8 up, 8 in 2024-08-17T17:28:24.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:23 smithi136 ceph-mon[109861]: osdmap e766: 8 total, 8 up, 8 in 2024-08-17T17:28:25.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:24 smithi165 ceph-mon[105517]: osdmap e767: 8 total, 8 up, 8 in 2024-08-17T17:28:25.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:24 smithi165 ceph-mon[105517]: pgmap v794: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 75 KiB/s rd, 6.2 KiB/s wr, 105 op/s 2024-08-17T17:28:25.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:24 smithi136 ceph-mon[115078]: osdmap e767: 8 total, 8 up, 8 in 2024-08-17T17:28:25.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:24 smithi136 ceph-mon[115078]: pgmap v794: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 75 KiB/s rd, 6.2 KiB/s wr, 105 op/s 2024-08-17T17:28:25.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:24 smithi136 ceph-mon[109861]: osdmap e767: 8 total, 8 up, 8 in 2024-08-17T17:28:25.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:24 smithi136 ceph-mon[109861]: pgmap v794: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 75 KiB/s rd, 6.2 KiB/s wr, 105 op/s 2024-08-17T17:28:26.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:25 smithi165 ceph-mon[105517]: osdmap e768: 8 total, 8 up, 8 in 2024-08-17T17:28:26.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:25 smithi136 ceph-mon[109861]: osdmap e768: 8 total, 8 up, 8 in 2024-08-17T17:28:26.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:25 smithi136 ceph-mon[115078]: osdmap e768: 8 total, 8 up, 8 in 2024-08-17T17:28:26.587 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.ObjectMapConsistentSnap (10429 ms) 2024-08-17T17:28:26.587 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.Metadata 2024-08-17T17:28:26.587 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:27.104 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.Metadata (517 ms) 2024-08-17T17:28:27.104 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.MetadataPP 2024-08-17T17:28:27.104 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:27.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:27 smithi165 ceph-mon[105517]: osdmap e769: 8 total, 8 up, 8 in 2024-08-17T17:28:27.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:27 smithi165 ceph-mon[105517]: pgmap v797: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 74 KiB/s rd, 3.5 KiB/s wr, 98 op/s 2024-08-17T17:28:27.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:27 smithi136 ceph-mon[115078]: osdmap e769: 8 total, 8 up, 8 in 2024-08-17T17:28:27.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:27 smithi136 ceph-mon[115078]: pgmap v797: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 74 KiB/s rd, 3.5 KiB/s wr, 98 op/s 2024-08-17T17:28:27.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:27 smithi136 ceph-mon[109861]: osdmap e769: 8 total, 8 up, 8 in 2024-08-17T17:28:27.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:27 smithi136 ceph-mon[109861]: pgmap v797: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 74 KiB/s rd, 3.5 KiB/s wr, 98 op/s 2024-08-17T17:28:27.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:28:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:28:27] "GET /metrics HTTP/1.1" 200 82171 "" "Prometheus/2.51.0" 2024-08-17T17:28:28.114 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.MetadataPP (1010 ms) 2024-08-17T17:28:28.115 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.UpdateFeatures 2024-08-17T17:28:28.115 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:28.209 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.UpdateFeatures (94 ms) 2024-08-17T17:28:28.209 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.FeaturesBitmaskString 2024-08-17T17:28:28.209 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.FeaturesBitmaskString (0 ms) 2024-08-17T17:28:28.209 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.RebuildObjectMap 2024-08-17T17:28:28.209 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:28.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:28 smithi165 ceph-mon[105517]: osdmap e770: 8 total, 8 up, 8 in 2024-08-17T17:28:28.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:28 smithi136 ceph-mon[109861]: osdmap e770: 8 total, 8 up, 8 in 2024-08-17T17:28:28.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:28 smithi136 ceph-mon[115078]: osdmap e770: 8 total, 8 up, 8 in 2024-08-17T17:28:29.089 INFO:tasks.workunit.client.0.smithi136.stdout:0.00% done 2024-08-17T17:28:29.089 INFO:tasks.workunit.client.0.smithi136.stdout:25.00% done 2024-08-17T17:28:29.089 INFO:tasks.workunit.client.0.smithi136.stdout:50.00% done 2024-08-17T17:28:29.089 INFO:tasks.workunit.client.0.smithi136.stdout:75.00% done 2024-08-17T17:28:29.089 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.RebuildObjectMap (880 ms) 2024-08-17T17:28:29.089 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.RebuildNewObjectMap 2024-08-17T17:28:29.123 INFO:tasks.workunit.client.0.smithi136.stdout:0.00% done 2024-08-17T17:28:29.123 INFO:tasks.workunit.client.0.smithi136.stdout:25.00% done 2024-08-17T17:28:29.123 INFO:tasks.workunit.client.0.smithi136.stdout:50.00% done 2024-08-17T17:28:29.123 INFO:tasks.workunit.client.0.smithi136.stdout:75.00% done 2024-08-17T17:28:29.123 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.RebuildNewObjectMap (35 ms) 2024-08-17T17:28:29.123 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.CheckObjectMap 2024-08-17T17:28:29.123 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:29.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:29 smithi136 ceph-mon[109861]: osdmap e771: 8 total, 8 up, 8 in 2024-08-17T17:28:29.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:29 smithi136 ceph-mon[109861]: pgmap v800: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 106 KiB/s rd, 38 KiB/s wr, 173 op/s 2024-08-17T17:28:29.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:29 smithi136 ceph-mon[115078]: osdmap e771: 8 total, 8 up, 8 in 2024-08-17T17:28:29.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:29 smithi136 ceph-mon[115078]: pgmap v800: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 106 KiB/s rd, 38 KiB/s wr, 173 op/s 2024-08-17T17:28:29.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:29 smithi165 ceph-mon[105517]: osdmap e771: 8 total, 8 up, 8 in 2024-08-17T17:28:29.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:29 smithi165 ceph-mon[105517]: pgmap v800: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 106 KiB/s rd, 38 KiB/s wr, 173 op/s 2024-08-17T17:28:30.099 INFO:tasks.workunit.client.0.smithi136.stdout:0.00% done 2024-08-17T17:28:30.099 INFO:tasks.workunit.client.0.smithi136.stdout:25.00% done 2024-08-17T17:28:30.100 INFO:tasks.workunit.client.0.smithi136.stdout:50.00% done 2024-08-17T17:28:30.100 INFO:tasks.workunit.client.0.smithi136.stdout:75.00% done 2024-08-17T17:28:30.100 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.CheckObjectMap (976 ms) 2024-08-17T17:28:30.100 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.BlockingAIO 2024-08-17T17:28:30.100 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:30.126 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.BlockingAIO (28 ms) 2024-08-17T17:28:30.127 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.ExclusiveLockTransition 2024-08-17T17:28:30.127 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:30.255 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.ExclusiveLockTransition (127 ms) 2024-08-17T17:28:30.255 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.ExclusiveLockReadTransition 2024-08-17T17:28:30.255 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_librbd.cc:10793: Skipped 2024-08-17T17:28:30.255 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:28:30.255 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockReadTransition (0 ms) 2024-08-17T17:28:30.255 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.CacheMayCopyOnWrite 2024-08-17T17:28:30.255 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:30.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:30 smithi136 ceph-mon[109861]: osdmap e772: 8 total, 8 up, 8 in 2024-08-17T17:28:30.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:30 smithi136 ceph-mon[115078]: osdmap e772: 8 total, 8 up, 8 in 2024-08-17T17:28:30.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:30 smithi165 ceph-mon[105517]: osdmap e772: 8 total, 8 up, 8 in 2024-08-17T17:28:31.107 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.CacheMayCopyOnWrite (853 ms) 2024-08-17T17:28:31.107 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.FlushEmptyOpsOnExternalSnapshot 2024-08-17T17:28:31.107 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:31.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:31 smithi136 ceph-mon[109861]: osdmap e773: 8 total, 8 up, 8 in 2024-08-17T17:28:31.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:31 smithi136 ceph-mon[109861]: pgmap v803: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 32 KiB/s rd, 34 KiB/s wr, 74 op/s 2024-08-17T17:28:31.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:31 smithi136 ceph-mon[115078]: osdmap e773: 8 total, 8 up, 8 in 2024-08-17T17:28:31.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:31 smithi136 ceph-mon[115078]: pgmap v803: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 32 KiB/s rd, 34 KiB/s wr, 74 op/s 2024-08-17T17:28:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:31 smithi165 ceph-mon[105517]: osdmap e773: 8 total, 8 up, 8 in 2024-08-17T17:28:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:31 smithi165 ceph-mon[105517]: pgmap v803: 713 pgs: 713 active+clean; 305 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 32 KiB/s rd, 34 KiB/s wr, 74 op/s 2024-08-17T17:28:32.073 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.FlushEmptyOpsOnExternalSnapshot (966 ms) 2024-08-17T17:28:32.073 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestImageOptions 2024-08-17T17:28:32.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:32 smithi136 ceph-mon[109861]: osdmap e774: 8 total, 8 up, 8 in 2024-08-17T17:28:32.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:32 smithi136 ceph-mon[115078]: osdmap e774: 8 total, 8 up, 8 in 2024-08-17T17:28:32.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:32 smithi165 ceph-mon[105517]: osdmap e774: 8 total, 8 up, 8 in 2024-08-17T17:28:33.173 INFO:tasks.workunit.client.0.smithi136.stdout:0.00% done 2024-08-17T17:28:33.174 INFO:tasks.workunit.client.0.smithi136.stdout:100.00% done 2024-08-17T17:28:33.174 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestImageOptions (1100 ms) 2024-08-17T17:28:33.174 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestImageOptionsPP 2024-08-17T17:28:33.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:33 smithi136 ceph-mon[109861]: osdmap e775: 8 total, 8 up, 8 in 2024-08-17T17:28:33.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:33 smithi136 ceph-mon[109861]: pgmap v806: 713 pgs: 713 active+clean; 305 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 888 KiB/s rd, 292 KiB/s wr, 1.28k op/s 2024-08-17T17:28:33.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:33 smithi136 ceph-mon[115078]: osdmap e775: 8 total, 8 up, 8 in 2024-08-17T17:28:33.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:33 smithi136 ceph-mon[115078]: pgmap v806: 713 pgs: 713 active+clean; 305 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 888 KiB/s rd, 292 KiB/s wr, 1.28k op/s 2024-08-17T17:28:33.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:33 smithi165 ceph-mon[105517]: osdmap e775: 8 total, 8 up, 8 in 2024-08-17T17:28:33.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:33 smithi165 ceph-mon[105517]: pgmap v806: 713 pgs: 713 active+clean; 305 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 888 KiB/s rd, 292 KiB/s wr, 1.28k op/s 2024-08-17T17:28:34.156 INFO:tasks.workunit.client.0.smithi136.stdout:0.00% done 2024-08-17T17:28:34.156 INFO:tasks.workunit.client.0.smithi136.stdout:100.00% done 2024-08-17T17:28:34.156 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestImageOptionsPP (982 ms) 2024-08-17T17:28:34.156 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.EventSocketPipe 2024-08-17T17:28:34.156 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.EventSocketPipe (0 ms) 2024-08-17T17:28:34.156 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.EventSocketEventfd 2024-08-17T17:28:34.156 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.EventSocketEventfd (0 ms) 2024-08-17T17:28:34.156 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.ImagePollIO 2024-08-17T17:28:34.157 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:34.198 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:28:34.199 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:28:34.199 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:28:34.199 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:28:34.199 INFO:tasks.workunit.client.0.smithi136.stdout:wrote: 512 2024-08-17T17:28:34.199 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:28:34.199 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:28:34.200 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:28:34.200 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:28:34.200 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:28:34.200 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:28:34.200 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:28:34.200 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:28:34.201 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:28:34.201 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:28:34.201 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:28:34.201 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:28:34.201 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:28:34.201 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:28:34.202 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:28:34.202 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:28:34.202 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:28:34.202 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:28:34.202 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:28:34.203 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:28:34.203 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:28:34.203 INFO:tasks.workunit.client.0.smithi136.stdout:started write 2024-08-17T17:28:34.203 INFO:tasks.workunit.client.0.smithi136.stdout:write completion cb called! 2024-08-17T17:28:34.203 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 0 2024-08-17T17:28:34.203 INFO:tasks.workunit.client.0.smithi136.stdout:finished write 2024-08-17T17:28:34.204 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:28:34.204 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:28:34.204 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:28:34.204 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:28:34.204 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:28:34.205 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:28:34.205 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:28:34.205 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:28:34.205 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:28:34.206 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:28:34.206 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:28:34.206 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:28:34.206 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:28:34.206 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:28:34.206 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:28:34.207 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:28:34.207 INFO:tasks.workunit.client.0.smithi136.stdout:created completion 2024-08-17T17:28:34.207 INFO:tasks.workunit.client.0.smithi136.stdout:started read 2024-08-17T17:28:34.207 INFO:tasks.workunit.client.0.smithi136.stdout:read completion cb called! 2024-08-17T17:28:34.207 INFO:tasks.workunit.client.0.smithi136.stdout:return value is: 512 2024-08-17T17:28:34.207 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.ImagePollIO (43 ms) 2024-08-17T17:28:34.208 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.Mirror 2024-08-17T17:28:34.208 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:34.210 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:34.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:34 smithi136 ceph-mon[109861]: osdmap e776: 8 total, 8 up, 8 in 2024-08-17T17:28:34.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:28:34.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:34 smithi136 ceph-mon[115078]: osdmap e776: 8 total, 8 up, 8 in 2024-08-17T17:28:34.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:28:34.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:34 smithi165 ceph-mon[105517]: osdmap e776: 8 total, 8 up, 8 in 2024-08-17T17:28:34.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:34 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:28:35.213 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.Mirror (1015 ms) 2024-08-17T17:28:35.213 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.MirrorPeerAttributes 2024-08-17T17:28:35.230 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.MirrorPeerAttributes (17 ms) 2024-08-17T17:28:35.231 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.CreateWithMirrorEnabled 2024-08-17T17:28:35.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:35 smithi136 ceph-mon[109861]: osdmap e777: 8 total, 8 up, 8 in 2024-08-17T17:28:35.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:35 smithi136 ceph-mon[109861]: pgmap v809: 713 pgs: 713 active+clean; 305 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 888 KiB/s rd, 292 KiB/s wr, 1.28k op/s 2024-08-17T17:28:35.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:35 smithi136 ceph-mon[115078]: osdmap e777: 8 total, 8 up, 8 in 2024-08-17T17:28:35.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:35 smithi136 ceph-mon[115078]: pgmap v809: 713 pgs: 713 active+clean; 305 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 888 KiB/s rd, 292 KiB/s wr, 1.28k op/s 2024-08-17T17:28:35.465 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:35 smithi165 ceph-mon[105517]: osdmap e777: 8 total, 8 up, 8 in 2024-08-17T17:28:35.465 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:35 smithi165 ceph-mon[105517]: pgmap v809: 713 pgs: 713 active+clean; 305 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 888 KiB/s rd, 292 KiB/s wr, 1.28k op/s 2024-08-17T17:28:36.345 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[115078]: osdmap e778: 8 total, 8 up, 8 in 2024-08-17T17:28:36.346 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2823250899' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/uuid4"} : dispatch 2024-08-17T17:28:36.346 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/uuid4"} : dispatch 2024-08-17T17:28:36.346 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/uuid4"}]': finished 2024-08-17T17:28:36.346 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2823250899' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/88c2cb73-8715-4632-9cc3-8459a9e4303e"} : dispatch 2024-08-17T17:28:36.346 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/88c2cb73-8715-4632-9cc3-8459a9e4303e"} : dispatch 2024-08-17T17:28:36.346 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/88c2cb73-8715-4632-9cc3-8459a9e4303e"}]': finished 2024-08-17T17:28:36.346 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2823250899' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/bcf1c8d8-2ca0-420b-839c-8d39a5974452"} : dispatch 2024-08-17T17:28:36.346 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/bcf1c8d8-2ca0-420b-839c-8d39a5974452"} : dispatch 2024-08-17T17:28:36.347 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/bcf1c8d8-2ca0-420b-839c-8d39a5974452"}]': finished 2024-08-17T17:28:36.347 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2823250899' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/0f762cd2-43a2-4759-a1d8-3f6f8ec6efb3"} : dispatch 2024-08-17T17:28:36.347 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/0f762cd2-43a2-4759-a1d8-3f6f8ec6efb3"} : dispatch 2024-08-17T17:28:36.347 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/0f762cd2-43a2-4759-a1d8-3f6f8ec6efb3"}]': finished 2024-08-17T17:28:36.347 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2823250899' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer/19/c5726797-d525-4899-a0bf-352f97ef868e"} : dispatch 2024-08-17T17:28:36.347 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' 2024-08-17T17:28:36.347 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2823250899' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer/19/c5726797-d525-4899-a0bf-352f97ef868e"} : dispatch 2024-08-17T17:28:36.347 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2823250899' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/c5726797-d525-4899-a0bf-352f97ef868e"} : dispatch 2024-08-17T17:28:36.347 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/c5726797-d525-4899-a0bf-352f97ef868e"} : dispatch 2024-08-17T17:28:36.348 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/c5726797-d525-4899-a0bf-352f97ef868e"}]': finished 2024-08-17T17:28:36.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[109861]: osdmap e778: 8 total, 8 up, 8 in 2024-08-17T17:28:36.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2823250899' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/uuid4"} : dispatch 2024-08-17T17:28:36.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/uuid4"} : dispatch 2024-08-17T17:28:36.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/uuid4"}]': finished 2024-08-17T17:28:36.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2823250899' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/88c2cb73-8715-4632-9cc3-8459a9e4303e"} : dispatch 2024-08-17T17:28:36.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/88c2cb73-8715-4632-9cc3-8459a9e4303e"} : dispatch 2024-08-17T17:28:36.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/88c2cb73-8715-4632-9cc3-8459a9e4303e"}]': finished 2024-08-17T17:28:36.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2823250899' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/bcf1c8d8-2ca0-420b-839c-8d39a5974452"} : dispatch 2024-08-17T17:28:36.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/bcf1c8d8-2ca0-420b-839c-8d39a5974452"} : dispatch 2024-08-17T17:28:36.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/bcf1c8d8-2ca0-420b-839c-8d39a5974452"}]': finished 2024-08-17T17:28:36.349 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2823250899' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/0f762cd2-43a2-4759-a1d8-3f6f8ec6efb3"} : dispatch 2024-08-17T17:28:36.349 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/0f762cd2-43a2-4759-a1d8-3f6f8ec6efb3"} : dispatch 2024-08-17T17:28:36.349 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/0f762cd2-43a2-4759-a1d8-3f6f8ec6efb3"}]': finished 2024-08-17T17:28:36.349 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2823250899' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer/19/c5726797-d525-4899-a0bf-352f97ef868e"} : dispatch 2024-08-17T17:28:36.349 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' 2024-08-17T17:28:36.349 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2823250899' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer/19/c5726797-d525-4899-a0bf-352f97ef868e"} : dispatch 2024-08-17T17:28:36.349 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2823250899' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/c5726797-d525-4899-a0bf-352f97ef868e"} : dispatch 2024-08-17T17:28:36.349 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/c5726797-d525-4899-a0bf-352f97ef868e"} : dispatch 2024-08-17T17:28:36.349 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:36 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/c5726797-d525-4899-a0bf-352f97ef868e"}]': finished 2024-08-17T17:28:36.374 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:36 smithi165 ceph-mon[105517]: osdmap e778: 8 total, 8 up, 8 in 2024-08-17T17:28:36.374 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:36 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2823250899' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/uuid4"} : dispatch 2024-08-17T17:28:36.374 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:36 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/uuid4"} : dispatch 2024-08-17T17:28:36.374 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:36 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/uuid4"}]': finished 2024-08-17T17:28:36.374 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:36 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2823250899' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/88c2cb73-8715-4632-9cc3-8459a9e4303e"} : dispatch 2024-08-17T17:28:36.374 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:36 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/88c2cb73-8715-4632-9cc3-8459a9e4303e"} : dispatch 2024-08-17T17:28:36.374 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:36 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/88c2cb73-8715-4632-9cc3-8459a9e4303e"}]': finished 2024-08-17T17:28:36.374 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:36 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2823250899' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/bcf1c8d8-2ca0-420b-839c-8d39a5974452"} : dispatch 2024-08-17T17:28:36.374 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:36 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/bcf1c8d8-2ca0-420b-839c-8d39a5974452"} : dispatch 2024-08-17T17:28:36.375 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:36 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/bcf1c8d8-2ca0-420b-839c-8d39a5974452"}]': finished 2024-08-17T17:28:36.375 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:36 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2823250899' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/0f762cd2-43a2-4759-a1d8-3f6f8ec6efb3"} : dispatch 2024-08-17T17:28:36.375 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:36 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/0f762cd2-43a2-4759-a1d8-3f6f8ec6efb3"} : dispatch 2024-08-17T17:28:36.375 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:36 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/0f762cd2-43a2-4759-a1d8-3f6f8ec6efb3"}]': finished 2024-08-17T17:28:36.375 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:36 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2823250899' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer/19/c5726797-d525-4899-a0bf-352f97ef868e"} : dispatch 2024-08-17T17:28:36.375 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:36 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' 2024-08-17T17:28:36.375 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:36 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2823250899' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer/19/c5726797-d525-4899-a0bf-352f97ef868e"} : dispatch 2024-08-17T17:28:36.375 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:36 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2823250899' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/c5726797-d525-4899-a0bf-352f97ef868e"} : dispatch 2024-08-17T17:28:36.375 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:36 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/c5726797-d525-4899-a0bf-352f97ef868e"} : dispatch 2024-08-17T17:28:36.375 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:36 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/c5726797-d525-4899-a0bf-352f97ef868e"}]': finished 2024-08-17T17:28:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:37 smithi165 ceph-mon[105517]: pgmap v811: 713 pgs: 713 active+clean; 309 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 80 KiB/s rd, 1008 KiB/s wr, 110 op/s 2024-08-17T17:28:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:28:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:37 smithi165 ceph-mon[105517]: osdmap e779: 8 total, 8 up, 8 in 2024-08-17T17:28:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:28:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:28:37.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:37 smithi136 ceph-mon[109861]: pgmap v811: 713 pgs: 713 active+clean; 309 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 80 KiB/s rd, 1008 KiB/s wr, 110 op/s 2024-08-17T17:28:37.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:28:37.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:37 smithi136 ceph-mon[109861]: osdmap e779: 8 total, 8 up, 8 in 2024-08-17T17:28:37.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:28:37.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:28:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:37 smithi136 ceph-mon[115078]: pgmap v811: 713 pgs: 713 active+clean; 309 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 80 KiB/s rd, 1008 KiB/s wr, 110 op/s 2024-08-17T17:28:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:28:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:37 smithi136 ceph-mon[115078]: osdmap e779: 8 total, 8 up, 8 in 2024-08-17T17:28:37.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:28:37.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:28:37.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:28:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:28:37] "GET /metrics HTTP/1.1" 200 82168 "" "Prometheus/2.51.0" 2024-08-17T17:28:38.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:38 smithi165 ceph-mon[105517]: osdmap e780: 8 total, 8 up, 8 in 2024-08-17T17:28:38.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:38 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:28:38.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:38 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:28:38.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:38 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:28:38.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:38 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:28:38.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:38 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:28:38.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:38 smithi136 ceph-mon[109861]: osdmap e780: 8 total, 8 up, 8 in 2024-08-17T17:28:38.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:28:38.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:28:38.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:28:38.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:28:38.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:28:38.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:38 smithi136 ceph-mon[115078]: osdmap e780: 8 total, 8 up, 8 in 2024-08-17T17:28:38.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:28:38.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:28:38.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:28:38.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:28:38.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:28:39.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:39 smithi165 ceph-mon[105517]: pgmap v814: 713 pgs: 713 active+clean; 313 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 255 KiB/s rd, 2.0 MiB/s wr, 348 op/s 2024-08-17T17:28:39.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:39 smithi165 ceph-mon[105517]: osdmap e781: 8 total, 8 up, 8 in 2024-08-17T17:28:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:39 smithi136 ceph-mon[109861]: pgmap v814: 713 pgs: 713 active+clean; 313 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 255 KiB/s rd, 2.0 MiB/s wr, 348 op/s 2024-08-17T17:28:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:39 smithi136 ceph-mon[109861]: osdmap e781: 8 total, 8 up, 8 in 2024-08-17T17:28:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:39 smithi136 ceph-mon[115078]: pgmap v814: 713 pgs: 713 active+clean; 313 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 255 KiB/s rd, 2.0 MiB/s wr, 348 op/s 2024-08-17T17:28:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:39 smithi136 ceph-mon[115078]: osdmap e781: 8 total, 8 up, 8 in 2024-08-17T17:28:40.297 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.CreateWithMirrorEnabled (5067 ms) 2024-08-17T17:28:40.297 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot 2024-08-17T17:28:40.297 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:40.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:40 smithi165 ceph-mon[105517]: osdmap e782: 8 total, 8 up, 8 in 2024-08-17T17:28:40.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:40 smithi136 ceph-mon[109861]: osdmap e782: 8 total, 8 up, 8 in 2024-08-17T17:28:40.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:40 smithi136 ceph-mon[115078]: osdmap e782: 8 total, 8 up, 8 in 2024-08-17T17:28:41.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:41 smithi165 ceph-mon[105517]: pgmap v817: 713 pgs: 713 active+clean; 313 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 178 KiB/s rd, 1.0 MiB/s wr, 242 op/s 2024-08-17T17:28:41.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:41 smithi165 ceph-mon[105517]: osdmap e783: 8 total, 8 up, 8 in 2024-08-17T17:28:41.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:41 smithi136 ceph-mon[109861]: pgmap v817: 713 pgs: 713 active+clean; 313 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 178 KiB/s rd, 1.0 MiB/s wr, 242 op/s 2024-08-17T17:28:41.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:41 smithi136 ceph-mon[109861]: osdmap e783: 8 total, 8 up, 8 in 2024-08-17T17:28:41.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:41 smithi136 ceph-mon[115078]: pgmap v817: 713 pgs: 713 active+clean; 313 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 178 KiB/s rd, 1.0 MiB/s wr, 242 op/s 2024-08-17T17:28:41.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:41 smithi136 ceph-mon[115078]: osdmap e783: 8 total, 8 up, 8 in 2024-08-17T17:28:42.296 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot (1999 ms) 2024-08-17T17:28:42.296 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.ExclusiveLock 2024-08-17T17:28:42.297 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:42.347 INFO:tasks.workunit.client.0.smithi136.stdout:1: acquiring exclusive lock 2024-08-17T17:28:42.347 INFO:tasks.workunit.client.0.smithi136.stdout:2: acquiring exclusive lock 2024-08-17T17:28:42.350 INFO:tasks.workunit.client.0.smithi136.stdout:2: exclusive lock acquired 2024-08-17T17:28:42.393 INFO:tasks.workunit.client.0.smithi136.stdout:2: releasing exclusive lock 2024-08-17T17:28:42.396 INFO:tasks.workunit.client.0.smithi136.stdout:2: exclusive lock released 2024-08-17T17:28:42.396 INFO:tasks.workunit.client.0.smithi136.stdout:2: acquiring exclusive lock 2024-08-17T17:28:42.399 INFO:tasks.workunit.client.0.smithi136.stdout:1: exclusive lock acquired 2024-08-17T17:28:42.404 INFO:tasks.workunit.client.0.smithi136.stdout:1: releasing exclusive lock 2024-08-17T17:28:42.407 INFO:tasks.workunit.client.0.smithi136.stdout:1: exclusive lock released 2024-08-17T17:28:42.407 INFO:tasks.workunit.client.0.smithi136.stdout:1: acquiring exclusive lock 2024-08-17T17:28:42.410 INFO:tasks.workunit.client.0.smithi136.stdout:2: exclusive lock acquired 2024-08-17T17:28:42.441 INFO:tasks.workunit.client.0.smithi136.stdout:2: releasing exclusive lock 2024-08-17T17:28:42.443 INFO:tasks.workunit.client.0.smithi136.stdout:2: exclusive lock released 2024-08-17T17:28:42.444 INFO:tasks.workunit.client.0.smithi136.stdout:2: acquiring exclusive lock 2024-08-17T17:28:42.447 INFO:tasks.workunit.client.0.smithi136.stdout:1: exclusive lock acquired 2024-08-17T17:28:42.471 INFO:tasks.workunit.client.0.smithi136.stdout:1: releasing exclusive lock 2024-08-17T17:28:42.474 INFO:tasks.workunit.client.0.smithi136.stdout:1: exclusive lock released 2024-08-17T17:28:42.474 INFO:tasks.workunit.client.0.smithi136.stdout:1: acquiring exclusive lock 2024-08-17T17:28:42.477 INFO:tasks.workunit.client.0.smithi136.stdout:2: exclusive lock acquired 2024-08-17T17:28:42.501 INFO:tasks.workunit.client.0.smithi136.stdout:2: releasing exclusive lock 2024-08-17T17:28:42.504 INFO:tasks.workunit.client.0.smithi136.stdout:2: exclusive lock released 2024-08-17T17:28:42.504 INFO:tasks.workunit.client.0.smithi136.stdout:2: acquiring exclusive lock 2024-08-17T17:28:42.507 INFO:tasks.workunit.client.0.smithi136.stdout:1: exclusive lock acquired 2024-08-17T17:28:42.513 INFO:tasks.workunit.client.0.smithi136.stdout:1: releasing exclusive lock 2024-08-17T17:28:42.516 INFO:tasks.workunit.client.0.smithi136.stdout:1: exclusive lock released 2024-08-17T17:28:42.516 INFO:tasks.workunit.client.0.smithi136.stdout:1: acquiring exclusive lock 2024-08-17T17:28:42.519 INFO:tasks.workunit.client.0.smithi136.stdout:1: exclusive lock acquired 2024-08-17T17:28:42.568 INFO:tasks.workunit.client.0.smithi136.stdout:1: releasing exclusive lock 2024-08-17T17:28:42.570 INFO:tasks.workunit.client.0.smithi136.stdout:1: exclusive lock released 2024-08-17T17:28:42.570 INFO:tasks.workunit.client.0.smithi136.stdout:1: acquiring exclusive lock 2024-08-17T17:28:42.574 INFO:tasks.workunit.client.0.smithi136.stdout:1: exclusive lock acquired 2024-08-17T17:28:42.584 INFO:tasks.workunit.client.0.smithi136.stdout:1: releasing exclusive lock 2024-08-17T17:28:42.587 INFO:tasks.workunit.client.0.smithi136.stdout:1: exclusive lock released 2024-08-17T17:28:42.589 INFO:tasks.workunit.client.0.smithi136.stdout:2: exclusive lock acquired 2024-08-17T17:28:42.620 INFO:tasks.workunit.client.0.smithi136.stdout:2: releasing exclusive lock 2024-08-17T17:28:42.623 INFO:tasks.workunit.client.0.smithi136.stdout:2: exclusive lock released 2024-08-17T17:28:42.623 INFO:tasks.workunit.client.0.smithi136.stdout:2: acquiring exclusive lock 2024-08-17T17:28:42.626 INFO:tasks.workunit.client.0.smithi136.stdout:2: exclusive lock acquired 2024-08-17T17:28:42.673 INFO:tasks.workunit.client.0.smithi136.stdout:2: releasing exclusive lock 2024-08-17T17:28:42.676 INFO:tasks.workunit.client.0.smithi136.stdout:2: exclusive lock released 2024-08-17T17:28:42.688 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.ExclusiveLock (392 ms) 2024-08-17T17:28:42.689 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.BreakLock 2024-08-17T17:28:42.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:42 smithi136 ceph-mon[109861]: osdmap e784: 8 total, 8 up, 8 in 2024-08-17T17:28:42.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:42 smithi136 ceph-mon[115078]: osdmap e784: 8 total, 8 up, 8 in 2024-08-17T17:28:42.704 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:42.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:42 smithi165 ceph-mon[105517]: osdmap e784: 8 total, 8 up, 8 in 2024-08-17T17:28:43.302 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.BreakLock (614 ms) 2024-08-17T17:28:43.302 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.DiscardAfterWrite 2024-08-17T17:28:43.302 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:43.329 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.DiscardAfterWrite (24 ms) 2024-08-17T17:28:43.330 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.DefaultFeatures 2024-08-17T17:28:43.330 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.DefaultFeatures (0 ms) 2024-08-17T17:28:43.330 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndPurge 2024-08-17T17:28:43.330 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:43.391 INFO:tasks.workunit.client.0.smithi136.stdout:0.00% done 2024-08-17T17:28:43.391 INFO:tasks.workunit.client.0.smithi136.stdout:1.56% done 2024-08-17T17:28:43.391 INFO:tasks.workunit.client.0.smithi136.stdout:3.12% done 2024-08-17T17:28:43.391 INFO:tasks.workunit.client.0.smithi136.stdout:4.69% done 2024-08-17T17:28:43.391 INFO:tasks.workunit.client.0.smithi136.stdout:6.25% done 2024-08-17T17:28:43.391 INFO:tasks.workunit.client.0.smithi136.stdout:7.81% done 2024-08-17T17:28:43.391 INFO:tasks.workunit.client.0.smithi136.stdout:9.38% done 2024-08-17T17:28:43.391 INFO:tasks.workunit.client.0.smithi136.stdout:10.94% done 2024-08-17T17:28:43.391 INFO:tasks.workunit.client.0.smithi136.stdout:12.50% done 2024-08-17T17:28:43.391 INFO:tasks.workunit.client.0.smithi136.stdout:14.06% done 2024-08-17T17:28:43.392 INFO:tasks.workunit.client.0.smithi136.stdout:15.62% done 2024-08-17T17:28:43.392 INFO:tasks.workunit.client.0.smithi136.stdout:17.19% done 2024-08-17T17:28:43.392 INFO:tasks.workunit.client.0.smithi136.stdout:18.75% done 2024-08-17T17:28:43.392 INFO:tasks.workunit.client.0.smithi136.stdout:20.31% done 2024-08-17T17:28:43.392 INFO:tasks.workunit.client.0.smithi136.stdout:21.88% done 2024-08-17T17:28:43.392 INFO:tasks.workunit.client.0.smithi136.stdout:23.44% done 2024-08-17T17:28:43.392 INFO:tasks.workunit.client.0.smithi136.stdout:25.00% done 2024-08-17T17:28:43.392 INFO:tasks.workunit.client.0.smithi136.stdout:26.56% done 2024-08-17T17:28:43.392 INFO:tasks.workunit.client.0.smithi136.stdout:28.12% done 2024-08-17T17:28:43.392 INFO:tasks.workunit.client.0.smithi136.stdout:29.69% done 2024-08-17T17:28:43.392 INFO:tasks.workunit.client.0.smithi136.stdout:31.25% done 2024-08-17T17:28:43.392 INFO:tasks.workunit.client.0.smithi136.stdout:32.81% done 2024-08-17T17:28:43.392 INFO:tasks.workunit.client.0.smithi136.stdout:34.38% done 2024-08-17T17:28:43.392 INFO:tasks.workunit.client.0.smithi136.stdout:35.94% done 2024-08-17T17:28:43.393 INFO:tasks.workunit.client.0.smithi136.stdout:37.50% done 2024-08-17T17:28:43.393 INFO:tasks.workunit.client.0.smithi136.stdout:39.06% done 2024-08-17T17:28:43.393 INFO:tasks.workunit.client.0.smithi136.stdout:40.62% done 2024-08-17T17:28:43.393 INFO:tasks.workunit.client.0.smithi136.stdout:42.19% done 2024-08-17T17:28:43.393 INFO:tasks.workunit.client.0.smithi136.stdout:43.75% done 2024-08-17T17:28:43.393 INFO:tasks.workunit.client.0.smithi136.stdout:45.31% done 2024-08-17T17:28:43.393 INFO:tasks.workunit.client.0.smithi136.stdout:46.88% done 2024-08-17T17:28:43.393 INFO:tasks.workunit.client.0.smithi136.stdout:48.44% done 2024-08-17T17:28:43.393 INFO:tasks.workunit.client.0.smithi136.stdout:50.00% done 2024-08-17T17:28:43.393 INFO:tasks.workunit.client.0.smithi136.stdout:51.56% done 2024-08-17T17:28:43.393 INFO:tasks.workunit.client.0.smithi136.stdout:53.12% done 2024-08-17T17:28:43.393 INFO:tasks.workunit.client.0.smithi136.stdout:54.69% done 2024-08-17T17:28:43.393 INFO:tasks.workunit.client.0.smithi136.stdout:56.25% done 2024-08-17T17:28:43.393 INFO:tasks.workunit.client.0.smithi136.stdout:57.81% done 2024-08-17T17:28:43.393 INFO:tasks.workunit.client.0.smithi136.stdout:59.38% done 2024-08-17T17:28:43.394 INFO:tasks.workunit.client.0.smithi136.stdout:60.94% done 2024-08-17T17:28:43.394 INFO:tasks.workunit.client.0.smithi136.stdout:62.50% done 2024-08-17T17:28:43.394 INFO:tasks.workunit.client.0.smithi136.stdout:64.06% done 2024-08-17T17:28:43.394 INFO:tasks.workunit.client.0.smithi136.stdout:65.62% done 2024-08-17T17:28:43.394 INFO:tasks.workunit.client.0.smithi136.stdout:67.19% done 2024-08-17T17:28:43.394 INFO:tasks.workunit.client.0.smithi136.stdout:68.75% done 2024-08-17T17:28:43.394 INFO:tasks.workunit.client.0.smithi136.stdout:70.31% done 2024-08-17T17:28:43.394 INFO:tasks.workunit.client.0.smithi136.stdout:71.88% done 2024-08-17T17:28:43.394 INFO:tasks.workunit.client.0.smithi136.stdout:73.44% done 2024-08-17T17:28:43.394 INFO:tasks.workunit.client.0.smithi136.stdout:75.00% done 2024-08-17T17:28:43.394 INFO:tasks.workunit.client.0.smithi136.stdout:76.56% done 2024-08-17T17:28:43.394 INFO:tasks.workunit.client.0.smithi136.stdout:78.12% done 2024-08-17T17:28:43.394 INFO:tasks.workunit.client.0.smithi136.stdout:79.69% done 2024-08-17T17:28:43.395 INFO:tasks.workunit.client.0.smithi136.stdout:81.25% done 2024-08-17T17:28:43.395 INFO:tasks.workunit.client.0.smithi136.stdout:82.81% done 2024-08-17T17:28:43.395 INFO:tasks.workunit.client.0.smithi136.stdout:84.38% done 2024-08-17T17:28:43.395 INFO:tasks.workunit.client.0.smithi136.stdout:85.94% done 2024-08-17T17:28:43.395 INFO:tasks.workunit.client.0.smithi136.stdout:87.50% done 2024-08-17T17:28:43.395 INFO:tasks.workunit.client.0.smithi136.stdout:89.06% done 2024-08-17T17:28:43.395 INFO:tasks.workunit.client.0.smithi136.stdout:90.62% done 2024-08-17T17:28:43.395 INFO:tasks.workunit.client.0.smithi136.stdout:92.19% done 2024-08-17T17:28:43.395 INFO:tasks.workunit.client.0.smithi136.stdout:93.75% done 2024-08-17T17:28:43.395 INFO:tasks.workunit.client.0.smithi136.stdout:95.31% done 2024-08-17T17:28:43.395 INFO:tasks.workunit.client.0.smithi136.stdout:96.88% done 2024-08-17T17:28:43.395 INFO:tasks.workunit.client.0.smithi136.stdout:98.44% done 2024-08-17T17:28:43.395 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestTrashMoveAndPurge (64 ms) 2024-08-17T17:28:43.395 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay 2024-08-17T17:28:43.396 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:43.445 INFO:tasks.workunit.client.0.smithi136.stdout:0.00% done 2024-08-17T17:28:43.445 INFO:tasks.workunit.client.0.smithi136.stdout:1.56% done 2024-08-17T17:28:43.445 INFO:tasks.workunit.client.0.smithi136.stdout:3.12% done 2024-08-17T17:28:43.445 INFO:tasks.workunit.client.0.smithi136.stdout:4.69% done 2024-08-17T17:28:43.445 INFO:tasks.workunit.client.0.smithi136.stdout:6.25% done 2024-08-17T17:28:43.445 INFO:tasks.workunit.client.0.smithi136.stdout:7.81% done 2024-08-17T17:28:43.445 INFO:tasks.workunit.client.0.smithi136.stdout:9.38% done 2024-08-17T17:28:43.445 INFO:tasks.workunit.client.0.smithi136.stdout:10.94% done 2024-08-17T17:28:43.445 INFO:tasks.workunit.client.0.smithi136.stdout:12.50% done 2024-08-17T17:28:43.445 INFO:tasks.workunit.client.0.smithi136.stdout:14.06% done 2024-08-17T17:28:43.445 INFO:tasks.workunit.client.0.smithi136.stdout:15.62% done 2024-08-17T17:28:43.445 INFO:tasks.workunit.client.0.smithi136.stdout:17.19% done 2024-08-17T17:28:43.446 INFO:tasks.workunit.client.0.smithi136.stdout:18.75% done 2024-08-17T17:28:43.446 INFO:tasks.workunit.client.0.smithi136.stdout:20.31% done 2024-08-17T17:28:43.446 INFO:tasks.workunit.client.0.smithi136.stdout:21.88% done 2024-08-17T17:28:43.446 INFO:tasks.workunit.client.0.smithi136.stdout:23.44% done 2024-08-17T17:28:43.446 INFO:tasks.workunit.client.0.smithi136.stdout:25.00% done 2024-08-17T17:28:43.446 INFO:tasks.workunit.client.0.smithi136.stdout:26.56% done 2024-08-17T17:28:43.446 INFO:tasks.workunit.client.0.smithi136.stdout:28.12% done 2024-08-17T17:28:43.446 INFO:tasks.workunit.client.0.smithi136.stdout:29.69% done 2024-08-17T17:28:43.446 INFO:tasks.workunit.client.0.smithi136.stdout:31.25% done 2024-08-17T17:28:43.446 INFO:tasks.workunit.client.0.smithi136.stdout:32.81% done 2024-08-17T17:28:43.446 INFO:tasks.workunit.client.0.smithi136.stdout:34.38% done 2024-08-17T17:28:43.446 INFO:tasks.workunit.client.0.smithi136.stdout:35.94% done 2024-08-17T17:28:43.446 INFO:tasks.workunit.client.0.smithi136.stdout:37.50% done 2024-08-17T17:28:43.447 INFO:tasks.workunit.client.0.smithi136.stdout:39.06% done 2024-08-17T17:28:43.447 INFO:tasks.workunit.client.0.smithi136.stdout:40.62% done 2024-08-17T17:28:43.447 INFO:tasks.workunit.client.0.smithi136.stdout:42.19% done 2024-08-17T17:28:43.447 INFO:tasks.workunit.client.0.smithi136.stdout:43.75% done 2024-08-17T17:28:43.447 INFO:tasks.workunit.client.0.smithi136.stdout:45.31% done 2024-08-17T17:28:43.447 INFO:tasks.workunit.client.0.smithi136.stdout:46.88% done 2024-08-17T17:28:43.447 INFO:tasks.workunit.client.0.smithi136.stdout:48.44% done 2024-08-17T17:28:43.447 INFO:tasks.workunit.client.0.smithi136.stdout:50.00% done 2024-08-17T17:28:43.447 INFO:tasks.workunit.client.0.smithi136.stdout:51.56% done 2024-08-17T17:28:43.447 INFO:tasks.workunit.client.0.smithi136.stdout:53.12% done 2024-08-17T17:28:43.447 INFO:tasks.workunit.client.0.smithi136.stdout:54.69% done 2024-08-17T17:28:43.447 INFO:tasks.workunit.client.0.smithi136.stdout:56.25% done 2024-08-17T17:28:43.447 INFO:tasks.workunit.client.0.smithi136.stdout:57.81% done 2024-08-17T17:28:43.447 INFO:tasks.workunit.client.0.smithi136.stdout:59.38% done 2024-08-17T17:28:43.448 INFO:tasks.workunit.client.0.smithi136.stdout:60.94% done 2024-08-17T17:28:43.448 INFO:tasks.workunit.client.0.smithi136.stdout:62.50% done 2024-08-17T17:28:43.448 INFO:tasks.workunit.client.0.smithi136.stdout:64.06% done 2024-08-17T17:28:43.448 INFO:tasks.workunit.client.0.smithi136.stdout:65.62% done 2024-08-17T17:28:43.448 INFO:tasks.workunit.client.0.smithi136.stdout:67.19% done 2024-08-17T17:28:43.448 INFO:tasks.workunit.client.0.smithi136.stdout:68.75% done 2024-08-17T17:28:43.448 INFO:tasks.workunit.client.0.smithi136.stdout:70.31% done 2024-08-17T17:28:43.448 INFO:tasks.workunit.client.0.smithi136.stdout:71.88% done 2024-08-17T17:28:43.448 INFO:tasks.workunit.client.0.smithi136.stdout:73.44% done 2024-08-17T17:28:43.448 INFO:tasks.workunit.client.0.smithi136.stdout:75.00% done 2024-08-17T17:28:43.448 INFO:tasks.workunit.client.0.smithi136.stdout:76.56% done 2024-08-17T17:28:43.448 INFO:tasks.workunit.client.0.smithi136.stdout:78.12% done 2024-08-17T17:28:43.449 INFO:tasks.workunit.client.0.smithi136.stdout:79.69% done 2024-08-17T17:28:43.449 INFO:tasks.workunit.client.0.smithi136.stdout:81.25% done 2024-08-17T17:28:43.449 INFO:tasks.workunit.client.0.smithi136.stdout:82.81% done 2024-08-17T17:28:43.449 INFO:tasks.workunit.client.0.smithi136.stdout:84.38% done 2024-08-17T17:28:43.449 INFO:tasks.workunit.client.0.smithi136.stdout:85.94% done 2024-08-17T17:28:43.449 INFO:tasks.workunit.client.0.smithi136.stdout:87.50% done 2024-08-17T17:28:43.449 INFO:tasks.workunit.client.0.smithi136.stdout:89.06% done 2024-08-17T17:28:43.449 INFO:tasks.workunit.client.0.smithi136.stdout:90.62% done 2024-08-17T17:28:43.449 INFO:tasks.workunit.client.0.smithi136.stdout:92.19% done 2024-08-17T17:28:43.449 INFO:tasks.workunit.client.0.smithi136.stdout:93.75% done 2024-08-17T17:28:43.449 INFO:tasks.workunit.client.0.smithi136.stdout:95.31% done 2024-08-17T17:28:43.449 INFO:tasks.workunit.client.0.smithi136.stdout:96.88% done 2024-08-17T17:28:43.449 INFO:tasks.workunit.client.0.smithi136.stdout:98.44% done 2024-08-17T17:28:43.449 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay (54 ms) 2024-08-17T17:28:43.450 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestTrashPurge 2024-08-17T17:28:43.450 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:43.452 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:43.565 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestTrashPurge (121 ms) 2024-08-17T17:28:43.565 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndRestore 2024-08-17T17:28:43.565 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:43.597 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:43 smithi136 ceph-mon[109861]: pgmap v820: 713 pgs: 713 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 213 KiB/s rd, 138 KiB/s wr, 196 op/s 2024-08-17T17:28:43.597 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:43 smithi136 ceph-mon[109861]: osdmap e785: 8 total, 8 up, 8 in 2024-08-17T17:28:43.597 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:43 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.136:0/3515054633"} : dispatch 2024-08-17T17:28:43.597 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:43 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.136:0/3515054633"} : dispatch 2024-08-17T17:28:43.597 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:43 smithi136 ceph-mon[115078]: pgmap v820: 713 pgs: 713 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 213 KiB/s rd, 138 KiB/s wr, 196 op/s 2024-08-17T17:28:43.597 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:43 smithi136 ceph-mon[115078]: osdmap e785: 8 total, 8 up, 8 in 2024-08-17T17:28:43.598 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:43 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.136:0/3515054633"} : dispatch 2024-08-17T17:28:43.598 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:43 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.136:0/3515054633"} : dispatch 2024-08-17T17:28:43.607 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestTrashMoveAndRestore (42 ms) 2024-08-17T17:28:43.607 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestListWatchers 2024-08-17T17:28:43.607 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:43.638 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestListWatchers (31 ms) 2024-08-17T17:28:43.638 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestSetSnapById 2024-08-17T17:28:43.638 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:43 smithi165 ceph-mon[105517]: pgmap v820: 713 pgs: 713 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 213 KiB/s rd, 138 KiB/s wr, 196 op/s 2024-08-17T17:28:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:43 smithi165 ceph-mon[105517]: osdmap e785: 8 total, 8 up, 8 in 2024-08-17T17:28:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:43 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2657741561' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.136:0/3515054633"} : dispatch 2024-08-17T17:28:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:43 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.136:0/3515054633"} : dispatch 2024-08-17T17:28:43.975 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestSetSnapById (336 ms) 2024-08-17T17:28:43.975 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.Namespaces 2024-08-17T17:28:43.995 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.Namespaces (21 ms) 2024-08-17T17:28:43.995 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.NamespacesPP 2024-08-17T17:28:44.025 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:44.030 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:44.105 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.NamespacesPP (109 ms) 2024-08-17T17:28:44.105 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.Migration 2024-08-17T17:28:44.105 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:44.105 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:44.482 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.Migration (377 ms) 2024-08-17T17:28:44.483 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.MigrationPP 2024-08-17T17:28:44.483 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:44.483 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:44.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:44 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.136:0/3515054633"}]': finished 2024-08-17T17:28:44.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:44 smithi136 ceph-mon[115078]: osdmap e786: 8 total, 8 up, 8 in 2024-08-17T17:28:44.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:44 smithi136 ceph-mon[115078]: osdmap e787: 8 total, 8 up, 8 in 2024-08-17T17:28:44.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:44 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.136:0/3515054633"}]': finished 2024-08-17T17:28:44.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:44 smithi136 ceph-mon[109861]: osdmap e786: 8 total, 8 up, 8 in 2024-08-17T17:28:44.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:44 smithi136 ceph-mon[109861]: osdmap e787: 8 total, 8 up, 8 in 2024-08-17T17:28:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:44 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.136:0/3515054633"}]': finished 2024-08-17T17:28:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:44 smithi165 ceph-mon[105517]: osdmap e786: 8 total, 8 up, 8 in 2024-08-17T17:28:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:44 smithi165 ceph-mon[105517]: osdmap e787: 8 total, 8 up, 8 in 2024-08-17T17:28:44.866 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.MigrationPP (384 ms) 2024-08-17T17:28:44.866 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestGetAccessTimestamp 2024-08-17T17:28:44.866 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:44.882 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestGetAccessTimestamp (16 ms) 2024-08-17T17:28:44.883 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.TestGetModifyTimestamp 2024-08-17T17:28:44.883 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:44.898 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.TestGetModifyTimestamp (16 ms) 2024-08-17T17:28:44.898 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.ZeroOverlapFlatten 2024-08-17T17:28:44.898 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:45.365 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.ZeroOverlapFlatten (467 ms) 2024-08-17T17:28:45.366 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.PoolMetadata 2024-08-17T17:28:45.392 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.PoolMetadata (27 ms) 2024-08-17T17:28:45.393 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.PoolMetadataPP 2024-08-17T17:28:45.411 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.PoolMetadataPP (19 ms) 2024-08-17T17:28:45.411 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.Config 2024-08-17T17:28:45.417 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:45.453 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.Config (42 ms) 2024-08-17T17:28:45.453 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.ConfigPP 2024-08-17T17:28:45.457 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:45.491 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.ConfigPP (38 ms) 2024-08-17T17:28:45.491 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.PoolStatsPP 2024-08-17T17:28:45.559 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:45 smithi136 ceph-mon[109861]: pgmap v824: 713 pgs: 713 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 220 KiB/s rd, 143 KiB/s wr, 203 op/s 2024-08-17T17:28:45.559 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:45 smithi136 ceph-mon[115078]: pgmap v824: 713 pgs: 713 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 220 KiB/s rd, 143 KiB/s wr, 203 op/s 2024-08-17T17:28:45.560 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:28:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:28:45] ENGINE Bus STOPPING 2024-08-17T17:28:45.560 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:28:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:28:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-08-17T17:28:45.560 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:28:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:28:45] ENGINE Bus STOPPED 2024-08-17T17:28:45.560 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:28:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:28:45] ENGINE Bus STARTING 2024-08-17T17:28:45.560 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:28:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:28:45] ENGINE Serving on http://:::9283 2024-08-17T17:28:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:45 smithi165 ceph-mon[105517]: pgmap v824: 713 pgs: 713 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 220 KiB/s rd, 143 KiB/s wr, 203 op/s 2024-08-17T17:28:45.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:28:45 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:17:28:45] ENGINE Bus STARTED 2024-08-17T17:28:46.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:46 smithi136 ceph-mon[109861]: osdmap e788: 8 total, 8 up, 8 in 2024-08-17T17:28:46.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:46 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' 2024-08-17T17:28:46.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:46 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:28:46.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:46 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:28:46.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:46 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:28:46.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:46 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:28:46.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:46 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-08-17T17:28:46.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:46 smithi136 ceph-mon[115078]: osdmap e788: 8 total, 8 up, 8 in 2024-08-17T17:28:46.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:46 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' 2024-08-17T17:28:46.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:46 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:28:46.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:46 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:28:46.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:46 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:28:46.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:46 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:28:46.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:46 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-08-17T17:28:46.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:46 smithi165 ceph-mon[105517]: osdmap e788: 8 total, 8 up, 8 in 2024-08-17T17:28:46.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:46 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' 2024-08-17T17:28:46.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:46 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:28:46.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:46 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:28:46.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:46 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:28:46.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:46 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:28:46.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:46 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-08-17T17:28:47.414 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:47.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:28:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:28:47] "GET /metrics HTTP/1.1" 200 82168 "" "Prometheus/2.51.0" 2024-08-17T17:28:47.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:47 smithi136 ceph-mon[109861]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:28:47.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:47 smithi136 ceph-mon[109861]: pgmap v826: 713 pgs: 713 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 393 KiB/s rd, 23 KiB/s wr, 542 op/s 2024-08-17T17:28:47.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:47 smithi136 ceph-mon[109861]: osdmap e789: 8 total, 8 up, 8 in 2024-08-17T17:28:47.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:47 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1467991218' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:28:47.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:47 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:28:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:47 smithi136 ceph-mon[115078]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:28:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:47 smithi136 ceph-mon[115078]: pgmap v826: 713 pgs: 713 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 393 KiB/s rd, 23 KiB/s wr, 542 op/s 2024-08-17T17:28:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:47 smithi136 ceph-mon[115078]: osdmap e789: 8 total, 8 up, 8 in 2024-08-17T17:28:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:47 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1467991218' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:28:47.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:47 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:28:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:47 smithi165 ceph-mon[105517]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:28:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:47 smithi165 ceph-mon[105517]: pgmap v826: 713 pgs: 713 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 393 KiB/s rd, 23 KiB/s wr, 542 op/s 2024-08-17T17:28:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:47 smithi165 ceph-mon[105517]: osdmap e789: 8 total, 8 up, 8 in 2024-08-17T17:28:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:47 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1467991218' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:28:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:47 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:28:48.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:48 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:28:48.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:48 smithi136 ceph-mon[109861]: osdmap e790: 8 total, 8 up, 8 in 2024-08-17T17:28:48.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:48 smithi136 ceph-mon[109861]: pgmap v829: 745 pgs: 14 creating+peering, 18 unknown, 713 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 877 KiB/s rd, 57 KiB/s wr, 1.18k op/s 2024-08-17T17:28:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:48 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:28:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:48 smithi136 ceph-mon[115078]: osdmap e790: 8 total, 8 up, 8 in 2024-08-17T17:28:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:48 smithi136 ceph-mon[115078]: pgmap v829: 745 pgs: 14 creating+peering, 18 unknown, 713 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 877 KiB/s rd, 57 KiB/s wr, 1.18k op/s 2024-08-17T17:28:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:48 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:28:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:48 smithi165 ceph-mon[105517]: osdmap e790: 8 total, 8 up, 8 in 2024-08-17T17:28:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:48 smithi165 ceph-mon[105517]: pgmap v829: 745 pgs: 14 creating+peering, 18 unknown, 713 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 877 KiB/s rd, 57 KiB/s wr, 1.18k op/s 2024-08-17T17:28:49.441 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:49.450 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:49.461 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:49 smithi136 ceph-mon[109861]: osdmap e791: 8 total, 8 up, 8 in 2024-08-17T17:28:49.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:49 smithi136 ceph-mon[115078]: osdmap e791: 8 total, 8 up, 8 in 2024-08-17T17:28:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:49 smithi165 ceph-mon[105517]: osdmap e791: 8 total, 8 up, 8 in 2024-08-17T17:28:50.490 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.PoolStatsPP (4999 ms) 2024-08-17T17:28:50.490 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.ImageSpec 2024-08-17T17:28:50.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:50 smithi136 ceph-mon[109861]: osdmap e792: 8 total, 8 up, 8 in 2024-08-17T17:28:50.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:50 smithi136 ceph-mon[109861]: pgmap v832: 745 pgs: 14 creating+peering, 18 unknown, 713 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 527 KiB/s rd, 37 KiB/s wr, 696 op/s 2024-08-17T17:28:50.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:50 smithi136 ceph-mon[115078]: osdmap e792: 8 total, 8 up, 8 in 2024-08-17T17:28:50.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:50 smithi136 ceph-mon[115078]: pgmap v832: 745 pgs: 14 creating+peering, 18 unknown, 713 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 527 KiB/s rd, 37 KiB/s wr, 696 op/s 2024-08-17T17:28:50.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:50 smithi165 ceph-mon[105517]: osdmap e792: 8 total, 8 up, 8 in 2024-08-17T17:28:50.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:50 smithi165 ceph-mon[105517]: pgmap v832: 745 pgs: 14 creating+peering, 18 unknown, 713 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 527 KiB/s rd, 37 KiB/s wr, 696 op/s 2024-08-17T17:28:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:51 smithi136 ceph-mon[109861]: osdmap e793: 8 total, 8 up, 8 in 2024-08-17T17:28:51.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:28:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:51 smithi136 ceph-mon[115078]: osdmap e793: 8 total, 8 up, 8 in 2024-08-17T17:28:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:28:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:51 smithi165 ceph-mon[105517]: osdmap e793: 8 total, 8 up, 8 in 2024-08-17T17:28:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:28:52.456 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:52.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:52 smithi136 ceph-mon[109861]: osdmap e794: 8 total, 8 up, 8 in 2024-08-17T17:28:52.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:52 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1149475780' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-36","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:28:52.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:52 smithi136 ceph-mon[109861]: pgmap v835: 777 pgs: 15 creating+peering, 17 unknown, 745 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 42 KiB/s rd, 6.2 KiB/s wr, 58 op/s 2024-08-17T17:28:52.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:52 smithi136 ceph-mon[115078]: osdmap e794: 8 total, 8 up, 8 in 2024-08-17T17:28:52.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:52 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1149475780' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-36","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:28:52.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:52 smithi136 ceph-mon[115078]: pgmap v835: 777 pgs: 15 creating+peering, 17 unknown, 745 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 42 KiB/s rd, 6.2 KiB/s wr, 58 op/s 2024-08-17T17:28:52.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:52 smithi165 ceph-mon[105517]: osdmap e794: 8 total, 8 up, 8 in 2024-08-17T17:28:52.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:52 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1149475780' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-36","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:28:52.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:52 smithi165 ceph-mon[105517]: pgmap v835: 777 pgs: 15 creating+peering, 17 unknown, 745 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 42 KiB/s rd, 6.2 KiB/s wr, 58 op/s 2024-08-17T17:28:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:53 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:28:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:53 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1149475780' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:28:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:53 smithi136 ceph-mon[109861]: osdmap e795: 8 total, 8 up, 8 in 2024-08-17T17:28:53.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:53 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:28:53.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:53 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1149475780' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:28:53.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:53 smithi136 ceph-mon[115078]: osdmap e795: 8 total, 8 up, 8 in 2024-08-17T17:28:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:53 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:28:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:53 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1149475780' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:28:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:53 smithi165 ceph-mon[105517]: osdmap e795: 8 total, 8 up, 8 in 2024-08-17T17:28:54.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:54 smithi136 ceph-mon[109861]: osdmap e796: 8 total, 8 up, 8 in 2024-08-17T17:28:54.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:54 smithi136 ceph-mon[109861]: osdmap e797: 8 total, 8 up, 8 in 2024-08-17T17:28:54.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:54 smithi136 ceph-mon[109861]: pgmap v839: 777 pgs: 15 creating+peering, 17 unknown, 745 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 45 KiB/s rd, 6.7 KiB/s wr, 62 op/s 2024-08-17T17:28:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:54 smithi136 ceph-mon[115078]: osdmap e796: 8 total, 8 up, 8 in 2024-08-17T17:28:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:54 smithi136 ceph-mon[115078]: osdmap e797: 8 total, 8 up, 8 in 2024-08-17T17:28:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:54 smithi136 ceph-mon[115078]: pgmap v839: 777 pgs: 15 creating+peering, 17 unknown, 745 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 45 KiB/s rd, 6.7 KiB/s wr, 62 op/s 2024-08-17T17:28:54.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:54 smithi165 ceph-mon[105517]: osdmap e796: 8 total, 8 up, 8 in 2024-08-17T17:28:54.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:54 smithi165 ceph-mon[105517]: osdmap e797: 8 total, 8 up, 8 in 2024-08-17T17:28:54.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:54 smithi165 ceph-mon[105517]: pgmap v839: 777 pgs: 15 creating+peering, 17 unknown, 745 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 45 KiB/s rd, 6.7 KiB/s wr, 62 op/s 2024-08-17T17:28:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:55 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:28:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:55 smithi136 ceph-mon[109861]: osdmap e798: 8 total, 8 up, 8 in 2024-08-17T17:28:55.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:55 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:28:55.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:55 smithi136 ceph-mon[115078]: osdmap e798: 8 total, 8 up, 8 in 2024-08-17T17:28:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:55 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:28:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:55 smithi165 ceph-mon[105517]: osdmap e798: 8 total, 8 up, 8 in 2024-08-17T17:28:56.100 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.ImageSpec (5610 ms) 2024-08-17T17:28:56.100 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.SnapRemoveWithChildMissing 2024-08-17T17:28:57.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:56 smithi165 ceph-mon[105517]: osdmap e799: 8 total, 8 up, 8 in 2024-08-17T17:28:57.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:56 smithi165 ceph-mon[105517]: pgmap v842: 777 pgs: 15 creating+peering, 762 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 40 KiB/s rd, 2.4 KiB/s wr, 51 op/s 2024-08-17T17:28:57.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:56 smithi136 ceph-mon[115078]: osdmap e799: 8 total, 8 up, 8 in 2024-08-17T17:28:57.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:56 smithi136 ceph-mon[115078]: pgmap v842: 777 pgs: 15 creating+peering, 762 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 40 KiB/s rd, 2.4 KiB/s wr, 51 op/s 2024-08-17T17:28:57.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:56 smithi136 ceph-mon[109861]: osdmap e799: 8 total, 8 up, 8 in 2024-08-17T17:28:57.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:56 smithi136 ceph-mon[109861]: pgmap v842: 777 pgs: 15 creating+peering, 762 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 40 KiB/s rd, 2.4 KiB/s wr, 51 op/s 2024-08-17T17:28:57.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:28:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:28:57] "GET /metrics HTTP/1.1" 200 84533 "" "Prometheus/2.51.0" 2024-08-17T17:28:57.976 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:28:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:57 smithi165 ceph-mon[105517]: osdmap e800: 8 total, 8 up, 8 in 2024-08-17T17:28:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:57 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2865831406' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:28:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:57 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:28:58.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:57 smithi136 ceph-mon[109861]: osdmap e800: 8 total, 8 up, 8 in 2024-08-17T17:28:58.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:57 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2865831406' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:28:58.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:57 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:28:58.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:57 smithi136 ceph-mon[115078]: osdmap e800: 8 total, 8 up, 8 in 2024-08-17T17:28:58.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:57 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2865831406' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:28:58.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:57 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:28:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:58 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:28:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:58 smithi165 ceph-mon[105517]: osdmap e801: 8 total, 8 up, 8 in 2024-08-17T17:28:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:58 smithi165 ceph-mon[105517]: pgmap v845: 809 pgs: 19 creating+peering, 13 unknown, 777 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 106 KiB/s rd, 5.7 KiB/s wr, 137 op/s 2024-08-17T17:28:59.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:58 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:28:59.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:58 smithi136 ceph-mon[109861]: osdmap e801: 8 total, 8 up, 8 in 2024-08-17T17:28:59.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:58 smithi136 ceph-mon[109861]: pgmap v845: 809 pgs: 19 creating+peering, 13 unknown, 777 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 106 KiB/s rd, 5.7 KiB/s wr, 137 op/s 2024-08-17T17:28:59.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:58 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:28:59.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:58 smithi136 ceph-mon[115078]: osdmap e801: 8 total, 8 up, 8 in 2024-08-17T17:28:59.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:58 smithi136 ceph-mon[115078]: pgmap v845: 809 pgs: 19 creating+peering, 13 unknown, 777 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 106 KiB/s rd, 5.7 KiB/s wr, 137 op/s 2024-08-17T17:29:00.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:28:59 smithi165 ceph-mon[105517]: osdmap e802: 8 total, 8 up, 8 in 2024-08-17T17:29:00.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:28:59 smithi136 ceph-mon[109861]: osdmap e802: 8 total, 8 up, 8 in 2024-08-17T17:29:00.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:28:59 smithi136 ceph-mon[115078]: osdmap e802: 8 total, 8 up, 8 in 2024-08-17T17:29:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:00 smithi165 ceph-mon[105517]: osdmap e803: 8 total, 8 up, 8 in 2024-08-17T17:29:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:00 smithi165 ceph-mon[105517]: pgmap v848: 809 pgs: 19 creating+peering, 13 unknown, 777 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 69 KiB/s rd, 3.5 KiB/s wr, 89 op/s 2024-08-17T17:29:01.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:00 smithi136 ceph-mon[109861]: osdmap e803: 8 total, 8 up, 8 in 2024-08-17T17:29:01.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:00 smithi136 ceph-mon[109861]: pgmap v848: 809 pgs: 19 creating+peering, 13 unknown, 777 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 69 KiB/s rd, 3.5 KiB/s wr, 89 op/s 2024-08-17T17:29:01.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:00 smithi136 ceph-mon[115078]: osdmap e803: 8 total, 8 up, 8 in 2024-08-17T17:29:01.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:00 smithi136 ceph-mon[115078]: pgmap v848: 809 pgs: 19 creating+peering, 13 unknown, 777 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 69 KiB/s rd, 3.5 KiB/s wr, 89 op/s 2024-08-17T17:29:02.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:02 smithi165 ceph-mon[105517]: osdmap e804: 8 total, 8 up, 8 in 2024-08-17T17:29:02.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:02 smithi136 ceph-mon[109861]: osdmap e804: 8 total, 8 up, 8 in 2024-08-17T17:29:02.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:02 smithi136 ceph-mon[115078]: osdmap e804: 8 total, 8 up, 8 in 2024-08-17T17:29:03.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:03 smithi165 ceph-mon[105517]: osdmap e805: 8 total, 8 up, 8 in 2024-08-17T17:29:03.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:03 smithi165 ceph-mon[105517]: pgmap v851: 809 pgs: 809 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 48 KiB/s rd, 3.0 KiB/s wr, 60 op/s 2024-08-17T17:29:03.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:03 smithi136 ceph-mon[109861]: osdmap e805: 8 total, 8 up, 8 in 2024-08-17T17:29:03.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:03 smithi136 ceph-mon[109861]: pgmap v851: 809 pgs: 809 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 48 KiB/s rd, 3.0 KiB/s wr, 60 op/s 2024-08-17T17:29:03.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:03 smithi136 ceph-mon[115078]: osdmap e805: 8 total, 8 up, 8 in 2024-08-17T17:29:03.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:03 smithi136 ceph-mon[115078]: pgmap v851: 809 pgs: 809 active+clean; 314 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 48 KiB/s rd, 3.0 KiB/s wr, 60 op/s 2024-08-17T17:29:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:04 smithi165 ceph-mon[105517]: osdmap e806: 8 total, 8 up, 8 in 2024-08-17T17:29:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:04 smithi165 ceph-mon[105517]: osdmap e807: 8 total, 8 up, 8 in 2024-08-17T17:29:04.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:04 smithi136 ceph-mon[109861]: osdmap e806: 8 total, 8 up, 8 in 2024-08-17T17:29:04.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:04 smithi136 ceph-mon[109861]: osdmap e807: 8 total, 8 up, 8 in 2024-08-17T17:29:04.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:04 smithi136 ceph-mon[115078]: osdmap e806: 8 total, 8 up, 8 in 2024-08-17T17:29:04.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:04 smithi136 ceph-mon[115078]: osdmap e807: 8 total, 8 up, 8 in 2024-08-17T17:29:05.048 INFO:tasks.workunit.client.0.smithi136.stdout:num children is: 3 2024-08-17T17:29:05.048 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 3 2024-08-17T17:29:05.048 INFO:tasks.workunit.client.0.smithi136.stdout:child: image230 2024-08-17T17:29:05.048 INFO:tasks.workunit.client.0.smithi136.stdout:child: image232 2024-08-17T17:29:05.048 INFO:tasks.workunit.client.0.smithi136.stdout:child: image231 2024-08-17T17:29:05.048 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-15/image230/acd6bb4e712b 2024-08-17T17:29:05.048 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:29:05.048 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-37/image231/acd63dbf1fa9 2024-08-17T17:29:05.048 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:29:05.048 INFO:tasks.workunit.client.0.smithi136.stdout:found child test-librbd-smithi136-87100-15/image232/acd686951e1 2024-08-17T17:29:05.048 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:29:05.048 INFO:tasks.workunit.client.0.smithi136.stdout:num children is: 0 2024-08-17T17:29:05.048 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 0 2024-08-17T17:29:05.049 INFO:tasks.workunit.client.0.smithi136.stdout:num snaps is: 0 2024-08-17T17:29:05.049 INFO:tasks.workunit.client.0.smithi136.stdout:expected: 0 2024-08-17T17:29:05.049 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.SnapRemoveWithChildMissing (8947 ms) 2024-08-17T17:29:05.049 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.QuiesceWatch 2024-08-17T17:29:05.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:05 smithi136 ceph-mon[109861]: pgmap v854: 777 pgs: 777 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:29:05.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:05 smithi136 ceph-mon[115078]: pgmap v854: 777 pgs: 777 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:29:05.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:05 smithi165 ceph-mon[105517]: pgmap v854: 777 pgs: 777 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:29:06.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:06 smithi136 ceph-mon[109861]: osdmap e808: 8 total, 8 up, 8 in 2024-08-17T17:29:06.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:06 smithi136 ceph-mon[115078]: osdmap e808: 8 total, 8 up, 8 in 2024-08-17T17:29:06.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:06 smithi165 ceph-mon[105517]: osdmap e808: 8 total, 8 up, 8 in 2024-08-17T17:29:07.045 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:29:07.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:07 smithi136 ceph-mon[109861]: osdmap e809: 8 total, 8 up, 8 in 2024-08-17T17:29:07.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:07 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3583714242' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-38","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:29:07.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:07 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-38","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:29:07.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:07 smithi136 ceph-mon[109861]: pgmap v857: 809 pgs: 32 unknown, 777 active+clean; 2.6 MiB data, 940 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:29:07.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:29:07.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:07 smithi136 ceph-mon[115078]: osdmap e809: 8 total, 8 up, 8 in 2024-08-17T17:29:07.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:07 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3583714242' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-38","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:29:07.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:07 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-38","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:29:07.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:07 smithi136 ceph-mon[115078]: pgmap v857: 809 pgs: 32 unknown, 777 active+clean; 2.6 MiB data, 940 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:29:07.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:29:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:07 smithi165 ceph-mon[105517]: osdmap e809: 8 total, 8 up, 8 in 2024-08-17T17:29:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:07 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3583714242' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-38","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:29:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:07 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-38","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:29:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:07 smithi165 ceph-mon[105517]: pgmap v857: 809 pgs: 32 unknown, 777 active+clean; 2.6 MiB data, 940 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:29:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:29:07.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:29:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:29:07] "GET /metrics HTTP/1.1" 200 89309 "" "Prometheus/2.51.0" 2024-08-17T17:29:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:08 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:29:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:08 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:29:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:08 smithi165 ceph-mon[105517]: osdmap e810: 8 total, 8 up, 8 in 2024-08-17T17:29:08.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:08 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:29:08.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:08 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:29:08.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:08 smithi136 ceph-mon[109861]: osdmap e810: 8 total, 8 up, 8 in 2024-08-17T17:29:08.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:08 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:29:08.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:08 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:29:08.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:08 smithi136 ceph-mon[115078]: osdmap e810: 8 total, 8 up, 8 in 2024-08-17T17:29:09.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:09 smithi136 ceph-mon[109861]: osdmap e811: 8 total, 8 up, 8 in 2024-08-17T17:29:09.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:09 smithi136 ceph-mon[109861]: pgmap v860: 809 pgs: 19 creating+peering, 13 unknown, 777 active+clean; 2.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 131 KiB/s rd, 8.0 KiB/s wr, 168 op/s 2024-08-17T17:29:09.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:09 smithi136 ceph-mon[115078]: osdmap e811: 8 total, 8 up, 8 in 2024-08-17T17:29:09.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:09 smithi136 ceph-mon[115078]: pgmap v860: 809 pgs: 19 creating+peering, 13 unknown, 777 active+clean; 2.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 131 KiB/s rd, 8.0 KiB/s wr, 168 op/s 2024-08-17T17:29:09.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:09 smithi165 ceph-mon[105517]: osdmap e811: 8 total, 8 up, 8 in 2024-08-17T17:29:09.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:09 smithi165 ceph-mon[105517]: pgmap v860: 809 pgs: 19 creating+peering, 13 unknown, 777 active+clean; 2.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 131 KiB/s rd, 8.0 KiB/s wr, 168 op/s 2024-08-17T17:29:10.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:10 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:29:10.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:10 smithi136 ceph-mon[109861]: osdmap e812: 8 total, 8 up, 8 in 2024-08-17T17:29:10.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:10 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:29:10.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:10 smithi136 ceph-mon[115078]: osdmap e812: 8 total, 8 up, 8 in 2024-08-17T17:29:10.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:10 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:29:10.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:10 smithi165 ceph-mon[105517]: osdmap e812: 8 total, 8 up, 8 in 2024-08-17T17:29:11.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:11 smithi136 ceph-mon[109861]: osdmap e813: 8 total, 8 up, 8 in 2024-08-17T17:29:11.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:11 smithi136 ceph-mon[109861]: pgmap v863: 809 pgs: 19 creating+peering, 13 unknown, 777 active+clean; 2.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 126 KiB/s rd, 7.0 KiB/s wr, 160 op/s 2024-08-17T17:29:11.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:11 smithi136 ceph-mon[115078]: osdmap e813: 8 total, 8 up, 8 in 2024-08-17T17:29:11.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:11 smithi136 ceph-mon[115078]: pgmap v863: 809 pgs: 19 creating+peering, 13 unknown, 777 active+clean; 2.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 126 KiB/s rd, 7.0 KiB/s wr, 160 op/s 2024-08-17T17:29:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:11 smithi165 ceph-mon[105517]: osdmap e813: 8 total, 8 up, 8 in 2024-08-17T17:29:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:11 smithi165 ceph-mon[105517]: pgmap v863: 809 pgs: 19 creating+peering, 13 unknown, 777 active+clean; 2.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 126 KiB/s rd, 7.0 KiB/s wr, 160 op/s 2024-08-17T17:29:12.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:12 smithi136 ceph-mon[109861]: osdmap e814: 8 total, 8 up, 8 in 2024-08-17T17:29:12.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:12 smithi136 ceph-mon[115078]: osdmap e814: 8 total, 8 up, 8 in 2024-08-17T17:29:12.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:12 smithi165 ceph-mon[105517]: osdmap e814: 8 total, 8 up, 8 in 2024-08-17T17:29:13.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:13 smithi136 ceph-mon[109861]: osdmap e815: 8 total, 8 up, 8 in 2024-08-17T17:29:13.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:13 smithi136 ceph-mon[109861]: pgmap v866: 809 pgs: 809 active+clean; 2.6 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 2.7 KiB/s wr, 44 op/s 2024-08-17T17:29:13.348 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:13 smithi136 ceph-mon[115078]: osdmap e815: 8 total, 8 up, 8 in 2024-08-17T17:29:13.348 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:13 smithi136 ceph-mon[115078]: pgmap v866: 809 pgs: 809 active+clean; 2.6 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 2.7 KiB/s wr, 44 op/s 2024-08-17T17:29:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:13 smithi165 ceph-mon[105517]: osdmap e815: 8 total, 8 up, 8 in 2024-08-17T17:29:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:13 smithi165 ceph-mon[105517]: pgmap v866: 809 pgs: 809 active+clean; 2.6 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 2.7 KiB/s wr, 44 op/s 2024-08-17T17:29:14.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:14 smithi136 ceph-mon[109861]: osdmap e816: 8 total, 8 up, 8 in 2024-08-17T17:29:14.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:14 smithi136 ceph-mon[109861]: osdmap e817: 8 total, 8 up, 8 in 2024-08-17T17:29:14.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:14 smithi136 ceph-mon[115078]: osdmap e816: 8 total, 8 up, 8 in 2024-08-17T17:29:14.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:14 smithi136 ceph-mon[115078]: osdmap e817: 8 total, 8 up, 8 in 2024-08-17T17:29:14.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:14 smithi165 ceph-mon[105517]: osdmap e816: 8 total, 8 up, 8 in 2024-08-17T17:29:14.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:14 smithi165 ceph-mon[105517]: osdmap e817: 8 total, 8 up, 8 in 2024-08-17T17:29:15.043 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.QuiesceWatch (9996 ms) 2024-08-17T17:29:15.044 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.QuiesceWatchPP 2024-08-17T17:29:15.044 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:29:15.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:15 smithi136 ceph-mon[109861]: pgmap v869: 809 pgs: 809 active+clean; 2.6 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 2.7 KiB/s wr, 44 op/s 2024-08-17T17:29:15.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:15 smithi136 ceph-mon[109861]: osdmap e818: 8 total, 8 up, 8 in 2024-08-17T17:29:15.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:15 smithi136 ceph-mon[115078]: pgmap v869: 809 pgs: 809 active+clean; 2.6 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 2.7 KiB/s wr, 44 op/s 2024-08-17T17:29:15.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:15 smithi136 ceph-mon[115078]: osdmap e818: 8 total, 8 up, 8 in 2024-08-17T17:29:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:15 smithi165 ceph-mon[105517]: pgmap v869: 809 pgs: 809 active+clean; 2.6 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 2.7 KiB/s wr, 44 op/s 2024-08-17T17:29:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:15 smithi165 ceph-mon[105517]: osdmap e818: 8 total, 8 up, 8 in 2024-08-17T17:29:17.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:16 smithi165 ceph-mon[105517]: osdmap e819: 8 total, 8 up, 8 in 2024-08-17T17:29:17.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:16 smithi165 ceph-mon[105517]: pgmap v872: 809 pgs: 809 active+clean; 2.6 MiB data, 553 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 1.2 KiB/s wr, 23 op/s 2024-08-17T17:29:17.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:16 smithi136 ceph-mon[109861]: osdmap e819: 8 total, 8 up, 8 in 2024-08-17T17:29:17.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:16 smithi136 ceph-mon[109861]: pgmap v872: 809 pgs: 809 active+clean; 2.6 MiB data, 553 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 1.2 KiB/s wr, 23 op/s 2024-08-17T17:29:17.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:16 smithi136 ceph-mon[115078]: osdmap e819: 8 total, 8 up, 8 in 2024-08-17T17:29:17.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:16 smithi136 ceph-mon[115078]: pgmap v872: 809 pgs: 809 active+clean; 2.6 MiB data, 553 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 1.2 KiB/s wr, 23 op/s 2024-08-17T17:29:17.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:29:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:29:17] "GET /metrics HTTP/1.1" 200 89309 "" "Prometheus/2.51.0" 2024-08-17T17:29:18.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:17 smithi165 ceph-mon[105517]: osdmap e820: 8 total, 8 up, 8 in 2024-08-17T17:29:18.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:17 smithi136 ceph-mon[109861]: osdmap e820: 8 total, 8 up, 8 in 2024-08-17T17:29:18.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:17 smithi136 ceph-mon[115078]: osdmap e820: 8 total, 8 up, 8 in 2024-08-17T17:29:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:18 smithi165 ceph-mon[105517]: osdmap e821: 8 total, 8 up, 8 in 2024-08-17T17:29:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:18 smithi165 ceph-mon[105517]: pgmap v875: 809 pgs: 809 active+clean; 2.6 MiB data, 553 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 5.7 KiB/s wr, 97 op/s 2024-08-17T17:29:19.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:18 smithi136 ceph-mon[109861]: osdmap e821: 8 total, 8 up, 8 in 2024-08-17T17:29:19.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:18 smithi136 ceph-mon[109861]: pgmap v875: 809 pgs: 809 active+clean; 2.6 MiB data, 553 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 5.7 KiB/s wr, 97 op/s 2024-08-17T17:29:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:18 smithi136 ceph-mon[115078]: osdmap e821: 8 total, 8 up, 8 in 2024-08-17T17:29:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:18 smithi136 ceph-mon[115078]: pgmap v875: 809 pgs: 809 active+clean; 2.6 MiB data, 553 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 5.7 KiB/s wr, 97 op/s 2024-08-17T17:29:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:19 smithi165 ceph-mon[105517]: osdmap e822: 8 total, 8 up, 8 in 2024-08-17T17:29:20.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:19 smithi136 ceph-mon[109861]: osdmap e822: 8 total, 8 up, 8 in 2024-08-17T17:29:20.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:19 smithi136 ceph-mon[115078]: osdmap e822: 8 total, 8 up, 8 in 2024-08-17T17:29:21.081 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.QuiesceWatchPP (6038 ms) 2024-08-17T17:29:21.081 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.QuiesceWatchError 2024-08-17T17:29:21.081 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:29:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:20 smithi165 ceph-mon[105517]: osdmap e823: 8 total, 8 up, 8 in 2024-08-17T17:29:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:20 smithi165 ceph-mon[105517]: pgmap v878: 809 pgs: 809 active+clean; 2.6 MiB data, 553 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 4.5 KiB/s wr, 73 op/s 2024-08-17T17:29:21.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:20 smithi136 ceph-mon[109861]: osdmap e823: 8 total, 8 up, 8 in 2024-08-17T17:29:21.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:20 smithi136 ceph-mon[109861]: pgmap v878: 809 pgs: 809 active+clean; 2.6 MiB data, 553 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 4.5 KiB/s wr, 73 op/s 2024-08-17T17:29:21.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:20 smithi136 ceph-mon[115078]: osdmap e823: 8 total, 8 up, 8 in 2024-08-17T17:29:21.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:20 smithi136 ceph-mon[115078]: pgmap v878: 809 pgs: 809 active+clean; 2.6 MiB data, 553 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 4.5 KiB/s wr, 73 op/s 2024-08-17T17:29:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:22 smithi165 ceph-mon[105517]: osdmap e824: 8 total, 8 up, 8 in 2024-08-17T17:29:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:29:22.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:22 smithi136 ceph-mon[109861]: osdmap e824: 8 total, 8 up, 8 in 2024-08-17T17:29:22.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:29:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:22 smithi136 ceph-mon[115078]: osdmap e824: 8 total, 8 up, 8 in 2024-08-17T17:29:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:29:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:23 smithi165 ceph-mon[105517]: osdmap e825: 8 total, 8 up, 8 in 2024-08-17T17:29:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:23 smithi165 ceph-mon[105517]: pgmap v881: 809 pgs: 809 active+clean; 2.6 MiB data, 542 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 6.5 KiB/s wr, 89 op/s 2024-08-17T17:29:23.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:23 smithi136 ceph-mon[109861]: osdmap e825: 8 total, 8 up, 8 in 2024-08-17T17:29:23.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:23 smithi136 ceph-mon[109861]: pgmap v881: 809 pgs: 809 active+clean; 2.6 MiB data, 542 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 6.5 KiB/s wr, 89 op/s 2024-08-17T17:29:23.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:23 smithi136 ceph-mon[115078]: osdmap e825: 8 total, 8 up, 8 in 2024-08-17T17:29:23.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:23 smithi136 ceph-mon[115078]: pgmap v881: 809 pgs: 809 active+clean; 2.6 MiB data, 542 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 6.5 KiB/s wr, 89 op/s 2024-08-17T17:29:24.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:24 smithi165 ceph-mon[105517]: osdmap e826: 8 total, 8 up, 8 in 2024-08-17T17:29:24.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:24 smithi165 ceph-mon[105517]: osdmap e827: 8 total, 8 up, 8 in 2024-08-17T17:29:24.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:24 smithi136 ceph-mon[109861]: osdmap e826: 8 total, 8 up, 8 in 2024-08-17T17:29:24.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:24 smithi136 ceph-mon[109861]: osdmap e827: 8 total, 8 up, 8 in 2024-08-17T17:29:24.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:24 smithi136 ceph-mon[115078]: osdmap e826: 8 total, 8 up, 8 in 2024-08-17T17:29:24.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:24 smithi136 ceph-mon[115078]: osdmap e827: 8 total, 8 up, 8 in 2024-08-17T17:29:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:25 smithi136 ceph-mon[109861]: pgmap v884: 809 pgs: 809 active+clean; 2.6 MiB data, 542 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 6.5 KiB/s wr, 89 op/s 2024-08-17T17:29:25.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:25 smithi136 ceph-mon[109861]: osdmap e828: 8 total, 8 up, 8 in 2024-08-17T17:29:25.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:25 smithi136 ceph-mon[115078]: pgmap v884: 809 pgs: 809 active+clean; 2.6 MiB data, 542 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 6.5 KiB/s wr, 89 op/s 2024-08-17T17:29:25.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:25 smithi136 ceph-mon[115078]: osdmap e828: 8 total, 8 up, 8 in 2024-08-17T17:29:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:25 smithi165 ceph-mon[105517]: pgmap v884: 809 pgs: 809 active+clean; 2.6 MiB data, 542 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 6.5 KiB/s wr, 89 op/s 2024-08-17T17:29:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:25 smithi165 ceph-mon[105517]: osdmap e828: 8 total, 8 up, 8 in 2024-08-17T17:29:27.053 INFO:tasks.workunit.client.0.smithi136.stdout:10.00% done 2024-08-17T17:29:27.054 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.QuiesceWatchError (5972 ms) 2024-08-17T17:29:27.054 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.QuiesceWatchTimeout 2024-08-17T17:29:27.054 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:29:27.067 INFO:tasks.workunit.client.0.smithi136.stderr:test quiesce is not long enough to time out 2024-08-17T17:29:27.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:26 smithi165 ceph-mon[105517]: osdmap e829: 8 total, 8 up, 8 in 2024-08-17T17:29:27.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:26 smithi165 ceph-mon[105517]: pgmap v887: 809 pgs: 809 active+clean; 2.6 MiB data, 542 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 767 B/s wr, 32 op/s 2024-08-17T17:29:27.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:26 smithi136 ceph-mon[109861]: osdmap e829: 8 total, 8 up, 8 in 2024-08-17T17:29:27.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:26 smithi136 ceph-mon[109861]: pgmap v887: 809 pgs: 809 active+clean; 2.6 MiB data, 542 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 767 B/s wr, 32 op/s 2024-08-17T17:29:27.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:26 smithi136 ceph-mon[115078]: osdmap e829: 8 total, 8 up, 8 in 2024-08-17T17:29:27.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:26 smithi136 ceph-mon[115078]: pgmap v887: 809 pgs: 809 active+clean; 2.6 MiB data, 542 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 767 B/s wr, 32 op/s 2024-08-17T17:29:27.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:29:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:29:27] "GET /metrics HTTP/1.1" 200 89355 "" "Prometheus/2.51.0" 2024-08-17T17:29:28.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:27 smithi165 ceph-mon[105517]: osdmap e830: 8 total, 8 up, 8 in 2024-08-17T17:29:28.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:27 smithi136 ceph-mon[109861]: osdmap e830: 8 total, 8 up, 8 in 2024-08-17T17:29:28.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:27 smithi136 ceph-mon[115078]: osdmap e830: 8 total, 8 up, 8 in 2024-08-17T17:29:29.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:28 smithi165 ceph-mon[105517]: pgmap v889: 809 pgs: 809 active+clean; 2.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 5.5 KiB/s wr, 95 op/s 2024-08-17T17:29:29.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:28 smithi165 ceph-mon[105517]: osdmap e831: 8 total, 8 up, 8 in 2024-08-17T17:29:29.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:28 smithi136 ceph-mon[109861]: pgmap v889: 809 pgs: 809 active+clean; 2.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 5.5 KiB/s wr, 95 op/s 2024-08-17T17:29:29.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:28 smithi136 ceph-mon[109861]: osdmap e831: 8 total, 8 up, 8 in 2024-08-17T17:29:29.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:28 smithi136 ceph-mon[115078]: pgmap v889: 809 pgs: 809 active+clean; 2.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 5.5 KiB/s wr, 95 op/s 2024-08-17T17:29:29.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:28 smithi136 ceph-mon[115078]: osdmap e831: 8 total, 8 up, 8 in 2024-08-17T17:29:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:31 smithi165 ceph-mon[105517]: pgmap v891: 809 pgs: 809 active+clean; 2.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 4.4 KiB/s wr, 77 op/s 2024-08-17T17:29:31.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:31 smithi136 ceph-mon[109861]: pgmap v891: 809 pgs: 809 active+clean; 2.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 4.4 KiB/s wr, 77 op/s 2024-08-17T17:29:31.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:31 smithi136 ceph-mon[115078]: pgmap v891: 809 pgs: 809 active+clean; 2.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 4.4 KiB/s wr, 77 op/s 2024-08-17T17:29:33.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:33 smithi165 ceph-mon[105517]: pgmap v892: 809 pgs: 809 active+clean; 2.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 4.4 KiB/s wr, 73 op/s 2024-08-17T17:29:33.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:33 smithi136 ceph-mon[109861]: pgmap v892: 809 pgs: 809 active+clean; 2.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 4.4 KiB/s wr, 73 op/s 2024-08-17T17:29:33.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:33 smithi136 ceph-mon[115078]: pgmap v892: 809 pgs: 809 active+clean; 2.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 4.4 KiB/s wr, 73 op/s 2024-08-17T17:29:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:34 smithi165 ceph-mon[105517]: osdmap e832: 8 total, 8 up, 8 in 2024-08-17T17:29:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:34 smithi165 ceph-mon[105517]: pgmap v894: 809 pgs: 809 active+clean; 2.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.8 KiB/s wr, 63 op/s 2024-08-17T17:29:35.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:34 smithi136 ceph-mon[115078]: osdmap e832: 8 total, 8 up, 8 in 2024-08-17T17:29:35.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:34 smithi136 ceph-mon[115078]: pgmap v894: 809 pgs: 809 active+clean; 2.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.8 KiB/s wr, 63 op/s 2024-08-17T17:29:35.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:34 smithi136 ceph-mon[109861]: osdmap e832: 8 total, 8 up, 8 in 2024-08-17T17:29:35.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:34 smithi136 ceph-mon[109861]: pgmap v894: 809 pgs: 809 active+clean; 2.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.8 KiB/s wr, 63 op/s 2024-08-17T17:29:35.994 INFO:tasks.workunit.client.0.smithi136.stderr:test quiesce is timed out 2024-08-17T17:29:36.002 INFO:tasks.workunit.client.0.smithi136.stderr:waiting for timed out ... 0 2024-08-17T17:29:36.998 INFO:tasks.workunit.client.0.smithi136.stderr:waiting for timed out ... 1 2024-08-17T17:29:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:36 smithi165 ceph-mon[105517]: osdmap e833: 8 total, 8 up, 8 in 2024-08-17T17:29:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:29:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:36 smithi165 ceph-mon[105517]: pgmap v896: 809 pgs: 809 active+clean; 2.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 994 B/s wr, 25 op/s 2024-08-17T17:29:37.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:36 smithi136 ceph-mon[109861]: osdmap e833: 8 total, 8 up, 8 in 2024-08-17T17:29:37.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:29:37.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:36 smithi136 ceph-mon[109861]: pgmap v896: 809 pgs: 809 active+clean; 2.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 994 B/s wr, 25 op/s 2024-08-17T17:29:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:36 smithi136 ceph-mon[115078]: osdmap e833: 8 total, 8 up, 8 in 2024-08-17T17:29:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:29:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:36 smithi136 ceph-mon[115078]: pgmap v896: 809 pgs: 809 active+clean; 2.6 MiB data, 543 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 994 B/s wr, 25 op/s 2024-08-17T17:29:37.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:29:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:29:37] "GET /metrics HTTP/1.1" 200 89333 "" "Prometheus/2.51.0" 2024-08-17T17:29:37.998 INFO:tasks.workunit.client.0.smithi136.stderr:waiting for timed out ... 2 2024-08-17T17:29:38.998 INFO:tasks.workunit.client.0.smithi136.stderr:waiting for timed out ... 3 2024-08-17T17:29:39.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:39 smithi165 ceph-mon[105517]: pgmap v897: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 1.2 KiB/s wr, 27 op/s 2024-08-17T17:29:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:39 smithi136 ceph-mon[109861]: pgmap v897: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 1.2 KiB/s wr, 27 op/s 2024-08-17T17:29:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:39 smithi136 ceph-mon[115078]: pgmap v897: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 1.2 KiB/s wr, 27 op/s 2024-08-17T17:29:39.998 INFO:tasks.workunit.client.0.smithi136.stderr:waiting for timed out ... 4 2024-08-17T17:29:40.998 INFO:tasks.workunit.client.0.smithi136.stderr:waiting for timed out ... 5 2024-08-17T17:29:41.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:41 smithi165 ceph-mon[105517]: pgmap v898: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 383 B/s wr, 4 op/s 2024-08-17T17:29:41.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:41 smithi136 ceph-mon[109861]: pgmap v898: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 383 B/s wr, 4 op/s 2024-08-17T17:29:41.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:41 smithi136 ceph-mon[115078]: pgmap v898: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 383 B/s wr, 4 op/s 2024-08-17T17:29:41.998 INFO:tasks.workunit.client.0.smithi136.stderr:waiting for timed out ... 6 2024-08-17T17:29:42.998 INFO:tasks.workunit.client.0.smithi136.stderr:waiting for timed out ... 7 2024-08-17T17:29:43.499 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:43 smithi136 ceph-mon[109861]: pgmap v899: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 374 B/s wr, 4 op/s 2024-08-17T17:29:43.499 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:43 smithi136 ceph-mon[115078]: pgmap v899: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 374 B/s wr, 4 op/s 2024-08-17T17:29:43.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:43 smithi165 ceph-mon[105517]: pgmap v899: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 374 B/s wr, 4 op/s 2024-08-17T17:29:43.998 INFO:tasks.workunit.client.0.smithi136.stderr:waiting for timed out ... 8 2024-08-17T17:29:44.999 INFO:tasks.workunit.client.0.smithi136.stderr:waiting for timed out ... 9 2024-08-17T17:29:45.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:45 smithi165 ceph-mon[105517]: pgmap v900: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 306 B/s wr, 3 op/s 2024-08-17T17:29:45.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:45 smithi136 ceph-mon[109861]: pgmap v900: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 306 B/s wr, 3 op/s 2024-08-17T17:29:45.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:45 smithi136 ceph-mon[115078]: pgmap v900: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 306 B/s wr, 3 op/s 2024-08-17T17:29:45.999 INFO:tasks.workunit.client.0.smithi136.stderr:waiting for timed out ... 10 2024-08-17T17:29:46.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:46 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:29:46.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:46 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:29:46.489 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:46 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:29:47.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:47 smithi136 ceph-mon[109861]: pgmap v901: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 301 B/s wr, 3 op/s 2024-08-17T17:29:47.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:47 smithi136 ceph-mon[115078]: pgmap v901: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 301 B/s wr, 3 op/s 2024-08-17T17:29:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:47 smithi165 ceph-mon[105517]: pgmap v901: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 301 B/s wr, 3 op/s 2024-08-17T17:29:47.875 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:29:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:29:47] "GET /metrics HTTP/1.1" 200 89333 "" "Prometheus/2.51.0" 2024-08-17T17:29:49.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:49 smithi165 ceph-mon[105517]: pgmap v902: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 255 B/s wr, 3 op/s 2024-08-17T17:29:49.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:49 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:29:49.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:49 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:29:49.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:49 smithi136 ceph-mon[109861]: pgmap v902: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 255 B/s wr, 3 op/s 2024-08-17T17:29:49.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:49 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:29:49.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:49 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:29:49.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:49 smithi136 ceph-mon[115078]: pgmap v902: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 255 B/s wr, 3 op/s 2024-08-17T17:29:49.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:49 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:29:49.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:49 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:29:51.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:50 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:29:51.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:50 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:29:51.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:50 smithi136 ceph-mon[109861]: pgmap v903: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-08-17T17:29:51.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:50 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:29:51.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:50 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:29:51.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:50 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:29:51.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:50 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:29:51.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:50 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:29:51.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:50 smithi136 ceph-mon[115078]: pgmap v903: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-08-17T17:29:51.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:50 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:29:51.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:50 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:29:51.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:50 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:29:51.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:50 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:29:51.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:50 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:29:51.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:50 smithi165 ceph-mon[105517]: pgmap v903: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-08-17T17:29:51.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:50 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:29:51.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:50 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:29:51.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:50 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:29:52.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:29:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:29:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:29:53.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:52 smithi136 ceph-mon[109861]: pgmap v904: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-08-17T17:29:53.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:52 smithi136 ceph-mon[115078]: pgmap v904: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-08-17T17:29:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:52 smithi165 ceph-mon[105517]: pgmap v904: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-08-17T17:29:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:55 smithi165 ceph-mon[105517]: pgmap v905: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:29:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:55 smithi136 ceph-mon[109861]: pgmap v905: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:29:55.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:55 smithi136 ceph-mon[115078]: pgmap v905: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:29:56.999 INFO:tasks.workunit.client.0.smithi136.stderr:test retry succeeds 2024-08-17T17:29:57.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:57 smithi165 ceph-mon[105517]: pgmap v906: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:29:57.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:57 smithi136 ceph-mon[109861]: pgmap v906: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:29:57.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:57 smithi136 ceph-mon[115078]: pgmap v906: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:29:57.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:29:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:29:57] "GET /metrics HTTP/1.1" 200 89359 "" "Prometheus/2.51.0" 2024-08-17T17:29:58.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:58 smithi165 ceph-mon[105517]: osdmap e834: 8 total, 8 up, 8 in 2024-08-17T17:29:58.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:58 smithi136 ceph-mon[109861]: osdmap e834: 8 total, 8 up, 8 in 2024-08-17T17:29:58.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:58 smithi136 ceph-mon[115078]: osdmap e834: 8 total, 8 up, 8 in 2024-08-17T17:29:59.336 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.QuiesceWatchTimeout (32283 ms) 2024-08-17T17:29:59.336 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.WriteZeroes 2024-08-17T17:29:59.336 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:29:59.397 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.WriteZeroes (61 ms) 2024-08-17T17:29:59.397 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.WriteZeroesThickProvision 2024-08-17T17:29:59.397 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:29:59.465 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.WriteZeroesThickProvision (68 ms) 2024-08-17T17:29:59.466 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestLibRBD.ConcurentOperations 2024-08-17T17:29:59.466 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:29:59.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:59 smithi136 ceph-mon[109861]: pgmap v908: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:29:59.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:29:59 smithi136 ceph-mon[109861]: osdmap e835: 8 total, 8 up, 8 in 2024-08-17T17:29:59.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:59 smithi136 ceph-mon[115078]: pgmap v908: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:29:59.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:29:59 smithi136 ceph-mon[115078]: osdmap e835: 8 total, 8 up, 8 in 2024-08-17T17:29:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:59 smithi165 ceph-mon[105517]: pgmap v908: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:29:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:29:59 smithi165 ceph-mon[105517]: osdmap e835: 8 total, 8 up, 8 in 2024-08-17T17:30:00.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:00 smithi136 ceph-mon[109861]: osdmap e836: 8 total, 8 up, 8 in 2024-08-17T17:30:00.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:00 smithi136 ceph-mon[109861]: Health detail: HEALTH_WARN 2 pool(s) full 2024-08-17T17:30:00.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:00 smithi136 ceph-mon[109861]: [WRN] POOL_FULL: 2 pool(s) full 2024-08-17T17:30:00.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:00 smithi136 ceph-mon[109861]: pool 'test-librbd-smithi136-87100-24' is full (running out of quota) 2024-08-17T17:30:00.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:00 smithi136 ceph-mon[109861]: pool 'test-librbd-smithi136-87100-27' is full (running out of quota) 2024-08-17T17:30:00.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:00 smithi136 ceph-mon[115078]: osdmap e836: 8 total, 8 up, 8 in 2024-08-17T17:30:00.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:00 smithi136 ceph-mon[115078]: Health detail: HEALTH_WARN 2 pool(s) full 2024-08-17T17:30:00.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:00 smithi136 ceph-mon[115078]: [WRN] POOL_FULL: 2 pool(s) full 2024-08-17T17:30:00.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:00 smithi136 ceph-mon[115078]: pool 'test-librbd-smithi136-87100-24' is full (running out of quota) 2024-08-17T17:30:00.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:00 smithi136 ceph-mon[115078]: pool 'test-librbd-smithi136-87100-27' is full (running out of quota) 2024-08-17T17:30:00.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:00 smithi165 ceph-mon[105517]: osdmap e836: 8 total, 8 up, 8 in 2024-08-17T17:30:00.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:00 smithi165 ceph-mon[105517]: Health detail: HEALTH_WARN 2 pool(s) full 2024-08-17T17:30:00.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:00 smithi165 ceph-mon[105517]: [WRN] POOL_FULL: 2 pool(s) full 2024-08-17T17:30:00.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:00 smithi165 ceph-mon[105517]: pool 'test-librbd-smithi136-87100-24' is full (running out of quota) 2024-08-17T17:30:00.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:00 smithi165 ceph-mon[105517]: pool 'test-librbd-smithi136-87100-27' is full (running out of quota) 2024-08-17T17:30:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:01 smithi136 ceph-mon[109861]: pgmap v911: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:01 smithi136 ceph-mon[109861]: osdmap e837: 8 total, 8 up, 8 in 2024-08-17T17:30:01.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:01 smithi136 ceph-mon[115078]: pgmap v911: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:01.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:01 smithi136 ceph-mon[115078]: osdmap e837: 8 total, 8 up, 8 in 2024-08-17T17:30:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:01 smithi165 ceph-mon[105517]: pgmap v911: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:01 smithi165 ceph-mon[105517]: osdmap e837: 8 total, 8 up, 8 in 2024-08-17T17:30:02.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:02 smithi136 ceph-mon[109861]: osdmap e838: 8 total, 8 up, 8 in 2024-08-17T17:30:02.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:02 smithi136 ceph-mon[115078]: osdmap e838: 8 total, 8 up, 8 in 2024-08-17T17:30:02.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:02 smithi165 ceph-mon[105517]: osdmap e838: 8 total, 8 up, 8 in 2024-08-17T17:30:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:03 smithi136 ceph-mon[109861]: pgmap v914: 809 pgs: 809 active+clean; 2.6 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s rd, 10 KiB/s wr, 180 op/s 2024-08-17T17:30:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:03 smithi136 ceph-mon[109861]: osdmap e839: 8 total, 8 up, 8 in 2024-08-17T17:30:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:03 smithi136 ceph-mon[115078]: pgmap v914: 809 pgs: 809 active+clean; 2.6 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s rd, 10 KiB/s wr, 180 op/s 2024-08-17T17:30:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:03 smithi136 ceph-mon[115078]: osdmap e839: 8 total, 8 up, 8 in 2024-08-17T17:30:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:03 smithi165 ceph-mon[105517]: pgmap v914: 809 pgs: 809 active+clean; 2.6 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s rd, 10 KiB/s wr, 180 op/s 2024-08-17T17:30:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:03 smithi165 ceph-mon[105517]: osdmap e839: 8 total, 8 up, 8 in 2024-08-17T17:30:04.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:04 smithi136 ceph-mon[109861]: osdmap e840: 8 total, 8 up, 8 in 2024-08-17T17:30:04.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:04 smithi136 ceph-mon[109861]: osdmap e841: 8 total, 8 up, 8 in 2024-08-17T17:30:04.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:04 smithi136 ceph-mon[115078]: osdmap e840: 8 total, 8 up, 8 in 2024-08-17T17:30:04.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:04 smithi136 ceph-mon[115078]: osdmap e841: 8 total, 8 up, 8 in 2024-08-17T17:30:04.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:04 smithi165 ceph-mon[105517]: osdmap e840: 8 total, 8 up, 8 in 2024-08-17T17:30:04.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:04 smithi165 ceph-mon[105517]: osdmap e841: 8 total, 8 up, 8 in 2024-08-17T17:30:05.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:05 smithi136 ceph-mon[109861]: pgmap v918: 809 pgs: 809 active+clean; 2.6 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 131 KiB/s rd, 10 KiB/s wr, 184 op/s 2024-08-17T17:30:05.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:05 smithi136 ceph-mon[109861]: osdmap e842: 8 total, 8 up, 8 in 2024-08-17T17:30:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:05 smithi136 ceph-mon[115078]: pgmap v918: 809 pgs: 809 active+clean; 2.6 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 131 KiB/s rd, 10 KiB/s wr, 184 op/s 2024-08-17T17:30:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:05 smithi136 ceph-mon[115078]: osdmap e842: 8 total, 8 up, 8 in 2024-08-17T17:30:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:05 smithi165 ceph-mon[105517]: pgmap v918: 809 pgs: 809 active+clean; 2.6 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 131 KiB/s rd, 10 KiB/s wr, 184 op/s 2024-08-17T17:30:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:05 smithi165 ceph-mon[105517]: osdmap e842: 8 total, 8 up, 8 in 2024-08-17T17:30:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:06 smithi165 ceph-mon[105517]: osdmap e843: 8 total, 8 up, 8 in 2024-08-17T17:30:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:30:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:06 smithi165 ceph-mon[105517]: pgmap v921: 809 pgs: 809 active+clean; 2.6 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 3.6 KiB/s wr, 16 op/s 2024-08-17T17:30:07.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:06 smithi136 ceph-mon[109861]: osdmap e843: 8 total, 8 up, 8 in 2024-08-17T17:30:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:30:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:06 smithi136 ceph-mon[109861]: pgmap v921: 809 pgs: 809 active+clean; 2.6 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 3.6 KiB/s wr, 16 op/s 2024-08-17T17:30:07.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:06 smithi136 ceph-mon[115078]: osdmap e843: 8 total, 8 up, 8 in 2024-08-17T17:30:07.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:30:07.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:06 smithi136 ceph-mon[115078]: pgmap v921: 809 pgs: 809 active+clean; 2.6 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 3.6 KiB/s wr, 16 op/s 2024-08-17T17:30:07.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:30:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:30:07] "GET /metrics HTTP/1.1" 200 89361 "" "Prometheus/2.51.0" 2024-08-17T17:30:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:07 smithi165 ceph-mon[105517]: osdmap e844: 8 total, 8 up, 8 in 2024-08-17T17:30:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.15", "id": [3, 4]} : dispatch 2024-08-17T17:30:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "35.17", "id": [4, 2]} : dispatch 2024-08-17T17:30:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "36.b", "id": [6, 3]} : dispatch 2024-08-17T17:30:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]} : dispatch 2024-08-17T17:30:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.16", "id": [6, 1]} : dispatch 2024-08-17T17:30:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1d", "id": [3, 7]} : dispatch 2024-08-17T17:30:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1e", "id": [3, 2]} : dispatch 2024-08-17T17:30:08.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1f", "id": [5, 1]} : dispatch 2024-08-17T17:30:08.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:07 smithi136 ceph-mon[115078]: osdmap e844: 8 total, 8 up, 8 in 2024-08-17T17:30:08.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.15", "id": [3, 4]} : dispatch 2024-08-17T17:30:08.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "35.17", "id": [4, 2]} : dispatch 2024-08-17T17:30:08.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "36.b", "id": [6, 3]} : dispatch 2024-08-17T17:30:08.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]} : dispatch 2024-08-17T17:30:08.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.16", "id": [6, 1]} : dispatch 2024-08-17T17:30:08.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1d", "id": [3, 7]} : dispatch 2024-08-17T17:30:08.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1e", "id": [3, 2]} : dispatch 2024-08-17T17:30:08.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1f", "id": [5, 1]} : dispatch 2024-08-17T17:30:08.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:07 smithi136 ceph-mon[109861]: osdmap e844: 8 total, 8 up, 8 in 2024-08-17T17:30:08.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.15", "id": [3, 4]} : dispatch 2024-08-17T17:30:08.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "35.17", "id": [4, 2]} : dispatch 2024-08-17T17:30:08.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "36.b", "id": [6, 3]} : dispatch 2024-08-17T17:30:08.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]} : dispatch 2024-08-17T17:30:08.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.16", "id": [6, 1]} : dispatch 2024-08-17T17:30:08.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1d", "id": [3, 7]} : dispatch 2024-08-17T17:30:08.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1e", "id": [3, 2]} : dispatch 2024-08-17T17:30:08.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1f", "id": [5, 1]} : dispatch 2024-08-17T17:30:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.15", "id": [3, 4]}]': finished 2024-08-17T17:30:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "35.17", "id": [4, 2]}]': finished 2024-08-17T17:30:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "36.b", "id": [6, 3]}]': finished 2024-08-17T17:30:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]}]': finished 2024-08-17T17:30:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.16", "id": [6, 1]}]': finished 2024-08-17T17:30:09.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1d", "id": [3, 7]}]': finished 2024-08-17T17:30:09.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1e", "id": [3, 2]}]': finished 2024-08-17T17:30:09.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1f", "id": [5, 1]}]': finished 2024-08-17T17:30:09.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:08 smithi165 ceph-mon[105517]: osdmap e845: 8 total, 8 up, 8 in 2024-08-17T17:30:09.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:08 smithi165 ceph-mon[105517]: pgmap v924: 809 pgs: 809 active+clean; 2.6 MiB data, 550 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 4.2 KiB/s wr, 16 op/s 2024-08-17T17:30:09.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.15", "id": [3, 4]}]': finished 2024-08-17T17:30:09.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "35.17", "id": [4, 2]}]': finished 2024-08-17T17:30:09.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "36.b", "id": [6, 3]}]': finished 2024-08-17T17:30:09.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]}]': finished 2024-08-17T17:30:09.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.16", "id": [6, 1]}]': finished 2024-08-17T17:30:09.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1d", "id": [3, 7]}]': finished 2024-08-17T17:30:09.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1e", "id": [3, 2]}]': finished 2024-08-17T17:30:09.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1f", "id": [5, 1]}]': finished 2024-08-17T17:30:09.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:08 smithi136 ceph-mon[109861]: osdmap e845: 8 total, 8 up, 8 in 2024-08-17T17:30:09.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:08 smithi136 ceph-mon[109861]: pgmap v924: 809 pgs: 809 active+clean; 2.6 MiB data, 550 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 4.2 KiB/s wr, 16 op/s 2024-08-17T17:30:09.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.15", "id": [3, 4]}]': finished 2024-08-17T17:30:09.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "35.17", "id": [4, 2]}]': finished 2024-08-17T17:30:09.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "36.b", "id": [6, 3]}]': finished 2024-08-17T17:30:09.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]}]': finished 2024-08-17T17:30:09.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.16", "id": [6, 1]}]': finished 2024-08-17T17:30:09.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1d", "id": [3, 7]}]': finished 2024-08-17T17:30:09.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1e", "id": [3, 2]}]': finished 2024-08-17T17:30:09.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1f", "id": [5, 1]}]': finished 2024-08-17T17:30:09.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:08 smithi136 ceph-mon[115078]: osdmap e845: 8 total, 8 up, 8 in 2024-08-17T17:30:09.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:08 smithi136 ceph-mon[115078]: pgmap v924: 809 pgs: 809 active+clean; 2.6 MiB data, 550 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 4.2 KiB/s wr, 16 op/s 2024-08-17T17:30:10.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:09 smithi165 ceph-mon[105517]: osdmap e846: 8 total, 8 up, 8 in 2024-08-17T17:30:10.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:09 smithi136 ceph-mon[109861]: osdmap e846: 8 total, 8 up, 8 in 2024-08-17T17:30:10.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:09 smithi136 ceph-mon[115078]: osdmap e846: 8 total, 8 up, 8 in 2024-08-17T17:30:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:11 smithi165 ceph-mon[105517]: osdmap e847: 8 total, 8 up, 8 in 2024-08-17T17:30:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:11 smithi165 ceph-mon[105517]: pgmap v927: 809 pgs: 809 active+clean; 2.6 MiB data, 550 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-08-17T17:30:11.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:11 smithi136 ceph-mon[109861]: osdmap e847: 8 total, 8 up, 8 in 2024-08-17T17:30:11.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:11 smithi136 ceph-mon[109861]: pgmap v927: 809 pgs: 809 active+clean; 2.6 MiB data, 550 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-08-17T17:30:11.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:11 smithi136 ceph-mon[115078]: osdmap e847: 8 total, 8 up, 8 in 2024-08-17T17:30:11.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:11 smithi136 ceph-mon[115078]: pgmap v927: 809 pgs: 809 active+clean; 2.6 MiB data, 550 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-08-17T17:30:12.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:12 smithi136 ceph-mon[109861]: osdmap e848: 8 total, 8 up, 8 in 2024-08-17T17:30:12.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:12 smithi136 ceph-mon[115078]: osdmap e848: 8 total, 8 up, 8 in 2024-08-17T17:30:12.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:12 smithi165 ceph-mon[105517]: osdmap e848: 8 total, 8 up, 8 in 2024-08-17T17:30:13.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:13 smithi136 ceph-mon[109861]: osdmap e849: 8 total, 8 up, 8 in 2024-08-17T17:30:13.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:13 smithi136 ceph-mon[109861]: pgmap v930: 809 pgs: 1 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 807 active+clean; 2.6 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 178 KiB/s rd, 4.2 KiB/s wr, 242 op/s; 0 B/s, 2 keys/s, 0 objects/s recovering 2024-08-17T17:30:13.348 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:13 smithi136 ceph-mon[115078]: osdmap e849: 8 total, 8 up, 8 in 2024-08-17T17:30:13.348 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:13 smithi136 ceph-mon[115078]: pgmap v930: 809 pgs: 1 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 807 active+clean; 2.6 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 178 KiB/s rd, 4.2 KiB/s wr, 242 op/s; 0 B/s, 2 keys/s, 0 objects/s recovering 2024-08-17T17:30:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:13 smithi165 ceph-mon[105517]: osdmap e849: 8 total, 8 up, 8 in 2024-08-17T17:30:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:13 smithi165 ceph-mon[105517]: pgmap v930: 809 pgs: 1 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 807 active+clean; 2.6 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 178 KiB/s rd, 4.2 KiB/s wr, 242 op/s; 0 B/s, 2 keys/s, 0 objects/s recovering 2024-08-17T17:30:14.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:14 smithi136 ceph-mon[115078]: osdmap e850: 8 total, 8 up, 8 in 2024-08-17T17:30:14.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:14 smithi136 ceph-mon[115078]: osdmap e851: 8 total, 8 up, 8 in 2024-08-17T17:30:14.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:14 smithi136 ceph-mon[109861]: osdmap e850: 8 total, 8 up, 8 in 2024-08-17T17:30:14.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:14 smithi136 ceph-mon[109861]: osdmap e851: 8 total, 8 up, 8 in 2024-08-17T17:30:14.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:14 smithi165 ceph-mon[105517]: osdmap e850: 8 total, 8 up, 8 in 2024-08-17T17:30:14.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:14 smithi165 ceph-mon[105517]: osdmap e851: 8 total, 8 up, 8 in 2024-08-17T17:30:15.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:15 smithi165 ceph-mon[105517]: pgmap v933: 809 pgs: 1 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 807 active+clean; 2.6 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 178 KiB/s rd, 4.2 KiB/s wr, 242 op/s; 0 B/s, 2 keys/s, 0 objects/s recovering 2024-08-17T17:30:15.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:15 smithi165 ceph-mon[105517]: osdmap e852: 8 total, 8 up, 8 in 2024-08-17T17:30:15.261 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:30:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=sqlstore.transactions t=2024-08-17T17:30:15.00235901Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-08-17T17:30:15.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:15 smithi136 ceph-mon[109861]: pgmap v933: 809 pgs: 1 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 807 active+clean; 2.6 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 178 KiB/s rd, 4.2 KiB/s wr, 242 op/s; 0 B/s, 2 keys/s, 0 objects/s recovering 2024-08-17T17:30:15.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:15 smithi136 ceph-mon[109861]: osdmap e852: 8 total, 8 up, 8 in 2024-08-17T17:30:15.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:15 smithi136 ceph-mon[115078]: pgmap v933: 809 pgs: 1 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 807 active+clean; 2.6 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 178 KiB/s rd, 4.2 KiB/s wr, 242 op/s; 0 B/s, 2 keys/s, 0 objects/s recovering 2024-08-17T17:30:15.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:15 smithi136 ceph-mon[115078]: osdmap e852: 8 total, 8 up, 8 in 2024-08-17T17:30:17.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:16 smithi165 ceph-mon[105517]: osdmap e853: 8 total, 8 up, 8 in 2024-08-17T17:30:17.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:16 smithi165 ceph-mon[105517]: pgmap v936: 809 pgs: 1 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 807 active+clean; 2.6 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 5.0 KiB/s wr, 21 op/s 2024-08-17T17:30:17.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:16 smithi136 ceph-mon[109861]: osdmap e853: 8 total, 8 up, 8 in 2024-08-17T17:30:17.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:16 smithi136 ceph-mon[109861]: pgmap v936: 809 pgs: 1 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 807 active+clean; 2.6 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 5.0 KiB/s wr, 21 op/s 2024-08-17T17:30:17.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:16 smithi136 ceph-mon[115078]: osdmap e853: 8 total, 8 up, 8 in 2024-08-17T17:30:17.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:16 smithi136 ceph-mon[115078]: pgmap v936: 809 pgs: 1 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 807 active+clean; 2.6 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 5.0 KiB/s wr, 21 op/s 2024-08-17T17:30:17.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:30:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:30:17] "GET /metrics HTTP/1.1" 200 89361 "" "Prometheus/2.51.0" 2024-08-17T17:30:18.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:17 smithi165 ceph-mon[105517]: osdmap e854: 8 total, 8 up, 8 in 2024-08-17T17:30:18.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:17 smithi136 ceph-mon[115078]: osdmap e854: 8 total, 8 up, 8 in 2024-08-17T17:30:18.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:17 smithi136 ceph-mon[109861]: osdmap e854: 8 total, 8 up, 8 in 2024-08-17T17:30:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:19 smithi165 ceph-mon[105517]: osdmap e855: 8 total, 8 up, 8 in 2024-08-17T17:30:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:19 smithi165 ceph-mon[105517]: pgmap v939: 809 pgs: 809 active+clean; 2.6 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 5.0 KiB/s wr, 26 op/s 2024-08-17T17:30:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:19 smithi165 ceph-mon[105517]: osdmap e856: 8 total, 8 up, 8 in 2024-08-17T17:30:19.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:19 smithi136 ceph-mon[109861]: osdmap e855: 8 total, 8 up, 8 in 2024-08-17T17:30:19.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:19 smithi136 ceph-mon[109861]: pgmap v939: 809 pgs: 809 active+clean; 2.6 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 5.0 KiB/s wr, 26 op/s 2024-08-17T17:30:19.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:19 smithi136 ceph-mon[109861]: osdmap e856: 8 total, 8 up, 8 in 2024-08-17T17:30:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:19 smithi136 ceph-mon[115078]: osdmap e855: 8 total, 8 up, 8 in 2024-08-17T17:30:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:19 smithi136 ceph-mon[115078]: pgmap v939: 809 pgs: 809 active+clean; 2.6 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 5.0 KiB/s wr, 26 op/s 2024-08-17T17:30:19.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:19 smithi136 ceph-mon[115078]: osdmap e856: 8 total, 8 up, 8 in 2024-08-17T17:30:21.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:21 smithi136 ceph-mon[109861]: osdmap e857: 8 total, 8 up, 8 in 2024-08-17T17:30:21.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:21 smithi136 ceph-mon[109861]: pgmap v942: 809 pgs: 809 active+clean; 2.6 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 5 op/s 2024-08-17T17:30:21.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:21 smithi136 ceph-mon[115078]: osdmap e857: 8 total, 8 up, 8 in 2024-08-17T17:30:21.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:21 smithi136 ceph-mon[115078]: pgmap v942: 809 pgs: 809 active+clean; 2.6 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 5 op/s 2024-08-17T17:30:21.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:21 smithi165 ceph-mon[105517]: osdmap e857: 8 total, 8 up, 8 in 2024-08-17T17:30:21.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:21 smithi165 ceph-mon[105517]: pgmap v942: 809 pgs: 809 active+clean; 2.6 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 5 op/s 2024-08-17T17:30:22.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:22 smithi136 ceph-mon[115078]: osdmap e858: 8 total, 8 up, 8 in 2024-08-17T17:30:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:30:22.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:22 smithi136 ceph-mon[109861]: osdmap e858: 8 total, 8 up, 8 in 2024-08-17T17:30:22.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:30:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:22 smithi165 ceph-mon[105517]: osdmap e858: 8 total, 8 up, 8 in 2024-08-17T17:30:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:30:23.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:23 smithi136 ceph-mon[109861]: osdmap e859: 8 total, 8 up, 8 in 2024-08-17T17:30:23.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:23 smithi136 ceph-mon[109861]: pgmap v945: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 4.0 KiB/s wr, 43 op/s 2024-08-17T17:30:23.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:23 smithi136 ceph-mon[115078]: osdmap e859: 8 total, 8 up, 8 in 2024-08-17T17:30:23.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:23 smithi136 ceph-mon[115078]: pgmap v945: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 4.0 KiB/s wr, 43 op/s 2024-08-17T17:30:23.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:23 smithi165 ceph-mon[105517]: osdmap e859: 8 total, 8 up, 8 in 2024-08-17T17:30:23.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:23 smithi165 ceph-mon[105517]: pgmap v945: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 4.0 KiB/s wr, 43 op/s 2024-08-17T17:30:24.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:24 smithi136 ceph-mon[115078]: osdmap e860: 8 total, 8 up, 8 in 2024-08-17T17:30:24.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:24 smithi136 ceph-mon[115078]: osdmap e861: 8 total, 8 up, 8 in 2024-08-17T17:30:24.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:24 smithi136 ceph-mon[109861]: osdmap e860: 8 total, 8 up, 8 in 2024-08-17T17:30:24.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:24 smithi136 ceph-mon[109861]: osdmap e861: 8 total, 8 up, 8 in 2024-08-17T17:30:24.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:24 smithi165 ceph-mon[105517]: osdmap e860: 8 total, 8 up, 8 in 2024-08-17T17:30:24.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:24 smithi165 ceph-mon[105517]: osdmap e861: 8 total, 8 up, 8 in 2024-08-17T17:30:25.058 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestLibRBD.ConcurentOperations (25593 ms) 2024-08-17T17:30:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:25 smithi136 ceph-mon[109861]: pgmap v948: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 4.0 KiB/s wr, 43 op/s 2024-08-17T17:30:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:25 smithi136 ceph-mon[109861]: osdmap e862: 8 total, 8 up, 8 in 2024-08-17T17:30:25.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:25 smithi136 ceph-mon[115078]: pgmap v948: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 4.0 KiB/s wr, 43 op/s 2024-08-17T17:30:25.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:25 smithi136 ceph-mon[115078]: osdmap e862: 8 total, 8 up, 8 in 2024-08-17T17:30:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:25 smithi165 ceph-mon[105517]: pgmap v948: 809 pgs: 809 active+clean; 2.6 MiB data, 544 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 4.0 KiB/s wr, 43 op/s 2024-08-17T17:30:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:25 smithi165 ceph-mon[105517]: osdmap e862: 8 total, 8 up, 8 in 2024-08-17T17:30:27.313 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:27 smithi165 ceph-mon[105517]: osdmap e863: 8 total, 8 up, 8 in 2024-08-17T17:30:27.313 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:27 smithi165 ceph-mon[105517]: pgmap v951: 777 pgs: 777 active+clean; 2.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 1.2 KiB/s wr, 70 op/s 2024-08-17T17:30:27.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:27 smithi136 ceph-mon[109861]: osdmap e863: 8 total, 8 up, 8 in 2024-08-17T17:30:27.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:27 smithi136 ceph-mon[109861]: pgmap v951: 777 pgs: 777 active+clean; 2.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 1.2 KiB/s wr, 70 op/s 2024-08-17T17:30:27.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:27 smithi136 ceph-mon[115078]: osdmap e863: 8 total, 8 up, 8 in 2024-08-17T17:30:27.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:27 smithi136 ceph-mon[115078]: pgmap v951: 777 pgs: 777 active+clean; 2.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 1.2 KiB/s wr, 70 op/s 2024-08-17T17:30:27.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:30:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:30:27] "GET /metrics HTTP/1.1" 200 89363 "" "Prometheus/2.51.0" 2024-08-17T17:30:28.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:28 smithi136 ceph-mon[109861]: osdmap e864: 8 total, 8 up, 8 in 2024-08-17T17:30:28.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:28 smithi136 ceph-mon[115078]: osdmap e864: 8 total, 8 up, 8 in 2024-08-17T17:30:28.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:28 smithi165 ceph-mon[105517]: osdmap e864: 8 total, 8 up, 8 in 2024-08-17T17:30:29.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:29 smithi136 ceph-mon[109861]: osdmap e865: 8 total, 8 up, 8 in 2024-08-17T17:30:29.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:29 smithi136 ceph-mon[109861]: pgmap v954: 713 pgs: 713 active+clean; 2.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:29.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:29 smithi136 ceph-mon[115078]: osdmap e865: 8 total, 8 up, 8 in 2024-08-17T17:30:29.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:29 smithi136 ceph-mon[115078]: pgmap v954: 713 pgs: 713 active+clean; 2.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:29.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:29 smithi165 ceph-mon[105517]: osdmap e865: 8 total, 8 up, 8 in 2024-08-17T17:30:29.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:29 smithi165 ceph-mon[105517]: pgmap v954: 713 pgs: 713 active+clean; 2.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:30.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:30 smithi136 ceph-mon[109861]: osdmap e866: 8 total, 8 up, 8 in 2024-08-17T17:30:30.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:30 smithi136 ceph-mon[115078]: osdmap e866: 8 total, 8 up, 8 in 2024-08-17T17:30:30.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:30 smithi165 ceph-mon[105517]: osdmap e866: 8 total, 8 up, 8 in 2024-08-17T17:30:31.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:31 smithi136 ceph-mon[109861]: osdmap e867: 8 total, 8 up, 8 in 2024-08-17T17:30:31.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:31 smithi136 ceph-mon[109861]: pgmap v957: 649 pgs: 649 active+clean; 2.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:31.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:31 smithi136 ceph-mon[109861]: osdmap e868: 8 total, 8 up, 8 in 2024-08-17T17:30:31.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:31 smithi136 ceph-mon[115078]: osdmap e867: 8 total, 8 up, 8 in 2024-08-17T17:30:31.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:31 smithi136 ceph-mon[115078]: pgmap v957: 649 pgs: 649 active+clean; 2.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:31.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:31 smithi136 ceph-mon[115078]: osdmap e868: 8 total, 8 up, 8 in 2024-08-17T17:30:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:31 smithi165 ceph-mon[105517]: osdmap e867: 8 total, 8 up, 8 in 2024-08-17T17:30:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:31 smithi165 ceph-mon[105517]: pgmap v957: 649 pgs: 649 active+clean; 2.6 MiB data, 537 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:31 smithi165 ceph-mon[105517]: osdmap e868: 8 total, 8 up, 8 in 2024-08-17T17:30:33.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:33 smithi136 ceph-mon[109861]: osdmap e869: 8 total, 8 up, 8 in 2024-08-17T17:30:33.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:33 smithi136 ceph-mon[109861]: pgmap v960: 585 pgs: 585 active+clean; 2.6 MiB data, 538 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:33.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:33 smithi136 ceph-mon[115078]: osdmap e869: 8 total, 8 up, 8 in 2024-08-17T17:30:33.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:33 smithi136 ceph-mon[115078]: pgmap v960: 585 pgs: 585 active+clean; 2.6 MiB data, 538 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:33.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:33 smithi165 ceph-mon[105517]: osdmap e869: 8 total, 8 up, 8 in 2024-08-17T17:30:33.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:33 smithi165 ceph-mon[105517]: pgmap v960: 585 pgs: 585 active+clean; 2.6 MiB data, 538 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:34.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:34 smithi136 ceph-mon[109861]: osdmap e870: 8 total, 8 up, 8 in 2024-08-17T17:30:34.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:34 smithi136 ceph-mon[109861]: Health check update: 1 pool(s) full (POOL_FULL) 2024-08-17T17:30:34.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:34 smithi136 ceph-mon[109861]: osdmap e871: 8 total, 8 up, 8 in 2024-08-17T17:30:34.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:34 smithi136 ceph-mon[115078]: osdmap e870: 8 total, 8 up, 8 in 2024-08-17T17:30:34.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:34 smithi136 ceph-mon[115078]: Health check update: 1 pool(s) full (POOL_FULL) 2024-08-17T17:30:34.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:34 smithi136 ceph-mon[115078]: osdmap e871: 8 total, 8 up, 8 in 2024-08-17T17:30:34.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:34 smithi165 ceph-mon[105517]: osdmap e870: 8 total, 8 up, 8 in 2024-08-17T17:30:34.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:34 smithi165 ceph-mon[105517]: Health check update: 1 pool(s) full (POOL_FULL) 2024-08-17T17:30:34.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:34 smithi165 ceph-mon[105517]: osdmap e871: 8 total, 8 up, 8 in 2024-08-17T17:30:35.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:35 smithi136 ceph-mon[109861]: pgmap v963: 521 pgs: 521 active+clean; 2.6 MiB data, 538 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:35.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:35 smithi136 ceph-mon[109861]: osdmap e872: 8 total, 8 up, 8 in 2024-08-17T17:30:35.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:35 smithi136 ceph-mon[115078]: pgmap v963: 521 pgs: 521 active+clean; 2.6 MiB data, 538 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:35.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:35 smithi136 ceph-mon[115078]: osdmap e872: 8 total, 8 up, 8 in 2024-08-17T17:30:35.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:35 smithi165 ceph-mon[105517]: pgmap v963: 521 pgs: 521 active+clean; 2.6 MiB data, 538 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:35.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:35 smithi165 ceph-mon[105517]: osdmap e872: 8 total, 8 up, 8 in 2024-08-17T17:30:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:36 smithi165 ceph-mon[105517]: osdmap e873: 8 total, 8 up, 8 in 2024-08-17T17:30:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:30:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:36 smithi165 ceph-mon[105517]: pgmap v966: 457 pgs: 457 active+clean; 2.6 MiB data, 538 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:37.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:36 smithi136 ceph-mon[109861]: osdmap e873: 8 total, 8 up, 8 in 2024-08-17T17:30:37.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:30:37.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:36 smithi136 ceph-mon[109861]: pgmap v966: 457 pgs: 457 active+clean; 2.6 MiB data, 538 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:36 smithi136 ceph-mon[115078]: osdmap e873: 8 total, 8 up, 8 in 2024-08-17T17:30:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:30:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:36 smithi136 ceph-mon[115078]: pgmap v966: 457 pgs: 457 active+clean; 2.6 MiB data, 538 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:37.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:30:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:30:37] "GET /metrics HTTP/1.1" 200 62864 "" "Prometheus/2.51.0" 2024-08-17T17:30:38.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:37 smithi165 ceph-mon[105517]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-08-17T17:30:38.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:37 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T17:30:38.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:37 smithi165 ceph-mon[105517]: osdmap e874: 8 total, 8 up, 8 in 2024-08-17T17:30:38.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:37 smithi136 ceph-mon[109861]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-08-17T17:30:38.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:37 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T17:30:38.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:37 smithi136 ceph-mon[109861]: osdmap e874: 8 total, 8 up, 8 in 2024-08-17T17:30:38.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:37 smithi136 ceph-mon[115078]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-08-17T17:30:38.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:37 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T17:30:38.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:37 smithi136 ceph-mon[115078]: osdmap e874: 8 total, 8 up, 8 in 2024-08-17T17:30:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:38 smithi165 ceph-mon[105517]: osdmap e875: 8 total, 8 up, 8 in 2024-08-17T17:30:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:38 smithi165 ceph-mon[105517]: pgmap v969: 393 pgs: 393 active+clean; 2.6 MiB data, 560 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:39.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:38 smithi136 ceph-mon[115078]: osdmap e875: 8 total, 8 up, 8 in 2024-08-17T17:30:39.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:38 smithi136 ceph-mon[115078]: pgmap v969: 393 pgs: 393 active+clean; 2.6 MiB data, 560 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:39.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:38 smithi136 ceph-mon[109861]: osdmap e875: 8 total, 8 up, 8 in 2024-08-17T17:30:39.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:38 smithi136 ceph-mon[109861]: pgmap v969: 393 pgs: 393 active+clean; 2.6 MiB data, 560 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:40.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:40 smithi165 ceph-mon[105517]: osdmap e876: 8 total, 8 up, 8 in 2024-08-17T17:30:40.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:40 smithi136 ceph-mon[109861]: osdmap e876: 8 total, 8 up, 8 in 2024-08-17T17:30:40.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:40 smithi136 ceph-mon[115078]: osdmap e876: 8 total, 8 up, 8 in 2024-08-17T17:30:41.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:41 smithi136 ceph-mon[109861]: osdmap e877: 8 total, 8 up, 8 in 2024-08-17T17:30:41.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:41 smithi136 ceph-mon[109861]: pgmap v972: 329 pgs: 329 active+clean; 587 KiB data, 560 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:41.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:41 smithi136 ceph-mon[115078]: osdmap e877: 8 total, 8 up, 8 in 2024-08-17T17:30:41.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:41 smithi136 ceph-mon[115078]: pgmap v972: 329 pgs: 329 active+clean; 587 KiB data, 560 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:41.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:41 smithi165 ceph-mon[105517]: osdmap e877: 8 total, 8 up, 8 in 2024-08-17T17:30:41.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:41 smithi165 ceph-mon[105517]: pgmap v972: 329 pgs: 329 active+clean; 587 KiB data, 560 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:42.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:42 smithi136 ceph-mon[109861]: osdmap e878: 8 total, 8 up, 8 in 2024-08-17T17:30:42.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:42 smithi136 ceph-mon[115078]: osdmap e878: 8 total, 8 up, 8 in 2024-08-17T17:30:42.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:42 smithi165 ceph-mon[105517]: osdmap e878: 8 total, 8 up, 8 in 2024-08-17T17:30:43.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:43 smithi136 ceph-mon[109861]: osdmap e879: 8 total, 8 up, 8 in 2024-08-17T17:30:43.348 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:43 smithi136 ceph-mon[109861]: pgmap v975: 265 pgs: 265 active+clean; 587 KiB data, 561 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:43.348 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:43 smithi136 ceph-mon[115078]: osdmap e879: 8 total, 8 up, 8 in 2024-08-17T17:30:43.348 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:43 smithi136 ceph-mon[115078]: pgmap v975: 265 pgs: 265 active+clean; 587 KiB data, 561 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:43.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:43 smithi165 ceph-mon[105517]: osdmap e879: 8 total, 8 up, 8 in 2024-08-17T17:30:43.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:43 smithi165 ceph-mon[105517]: pgmap v975: 265 pgs: 265 active+clean; 587 KiB data, 561 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:44.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:44 smithi136 ceph-mon[109861]: osdmap e880: 8 total, 8 up, 8 in 2024-08-17T17:30:44.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:44 smithi136 ceph-mon[115078]: osdmap e880: 8 total, 8 up, 8 in 2024-08-17T17:30:44.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:44 smithi165 ceph-mon[105517]: osdmap e880: 8 total, 8 up, 8 in 2024-08-17T17:30:45.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:45 smithi136 ceph-mon[109861]: osdmap e881: 8 total, 8 up, 8 in 2024-08-17T17:30:45.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:45 smithi136 ceph-mon[109861]: pgmap v978: 201 pgs: 201 active+clean; 587 KiB data, 561 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:45.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:45 smithi136 ceph-mon[115078]: osdmap e881: 8 total, 8 up, 8 in 2024-08-17T17:30:45.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:45 smithi136 ceph-mon[115078]: pgmap v978: 201 pgs: 201 active+clean; 587 KiB data, 561 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:45.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:45 smithi165 ceph-mon[105517]: osdmap e881: 8 total, 8 up, 8 in 2024-08-17T17:30:45.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:45 smithi165 ceph-mon[105517]: pgmap v978: 201 pgs: 201 active+clean; 587 KiB data, 561 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:46.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:46 smithi136 ceph-mon[109861]: osdmap e882: 8 total, 8 up, 8 in 2024-08-17T17:30:46.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:46 smithi136 ceph-mon[115078]: osdmap e882: 8 total, 8 up, 8 in 2024-08-17T17:30:46.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:46 smithi165 ceph-mon[105517]: osdmap e882: 8 total, 8 up, 8 in 2024-08-17T17:30:47.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:47 smithi136 ceph-mon[115078]: osdmap e883: 8 total, 8 up, 8 in 2024-08-17T17:30:47.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:47 smithi136 ceph-mon[115078]: pgmap v981: 137 pgs: 137 active+clean; 586 KiB data, 565 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:47.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:47 smithi136 ceph-mon[109861]: osdmap e883: 8 total, 8 up, 8 in 2024-08-17T17:30:47.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:47 smithi136 ceph-mon[109861]: pgmap v981: 137 pgs: 137 active+clean; 586 KiB data, 565 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:47 smithi165 ceph-mon[105517]: osdmap e883: 8 total, 8 up, 8 in 2024-08-17T17:30:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:47 smithi165 ceph-mon[105517]: pgmap v981: 137 pgs: 137 active+clean; 586 KiB data, 565 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:47.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:30:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:30:47] "GET /metrics HTTP/1.1" 200 62864 "" "Prometheus/2.51.0" 2024-08-17T17:30:48.072 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 170 tests from TestLibRBD (628074 ms total) 2024-08-17T17:30:48.072 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:30:48.072 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 7 tests from EncryptedFlattenTest/0, where TypeParam = PlaintextUnderLUKS1 2024-08-17T17:30:48.103 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/0.Simple 2024-08-17T17:30:48.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:48 smithi136 ceph-mon[115078]: osdmap e884: 8 total, 8 up, 8 in 2024-08-17T17:30:48.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:48 smithi136 ceph-mon[109861]: osdmap e884: 8 total, 8 up, 8 in 2024-08-17T17:30:48.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:48 smithi165 ceph-mon[105517]: osdmap e884: 8 total, 8 up, 8 in 2024-08-17T17:30:49.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:49 smithi136 ceph-mon[109861]: osdmap e885: 8 total, 8 up, 8 in 2024-08-17T17:30:49.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:49 smithi136 ceph-mon[109861]: pgmap v984: 73 pgs: 73 active+clean; 582 KiB data, 565 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:49.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:49 smithi136 ceph-mon[115078]: osdmap e885: 8 total, 8 up, 8 in 2024-08-17T17:30:49.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:49 smithi136 ceph-mon[115078]: pgmap v984: 73 pgs: 73 active+clean; 582 KiB data, 565 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:49.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:49 smithi165 ceph-mon[105517]: osdmap e885: 8 total, 8 up, 8 in 2024-08-17T17:30:49.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:49 smithi165 ceph-mon[105517]: pgmap v984: 73 pgs: 73 active+clean; 582 KiB data, 565 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:50.100 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:30:50.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:50 smithi136 ceph-mon[109861]: osdmap e886: 8 total, 8 up, 8 in 2024-08-17T17:30:50.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:50 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3173501995' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:30:50.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:50 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:30:50.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:50 smithi136 ceph-mon[115078]: osdmap e886: 8 total, 8 up, 8 in 2024-08-17T17:30:50.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:50 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3173501995' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:30:50.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:50 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:30:50.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:50 smithi165 ceph-mon[105517]: osdmap e886: 8 total, 8 up, 8 in 2024-08-17T17:30:50.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:50 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3173501995' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:30:50.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:50 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:30:51.189 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:51 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:30:51.189 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:51 smithi136 ceph-mon[115078]: osdmap e887: 8 total, 8 up, 8 in 2024-08-17T17:30:51.189 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:51 smithi136 ceph-mon[115078]: pgmap v987: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 565 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:51.189 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:30:51.189 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:51 smithi136 ceph-mon[115078]: osdmap e888: 8 total, 8 up, 8 in 2024-08-17T17:30:51.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:51 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:30:51.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:51 smithi136 ceph-mon[109861]: osdmap e887: 8 total, 8 up, 8 in 2024-08-17T17:30:51.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:51 smithi136 ceph-mon[109861]: pgmap v987: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 565 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:51.189 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:30:51.190 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:51 smithi136 ceph-mon[109861]: osdmap e888: 8 total, 8 up, 8 in 2024-08-17T17:30:51.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:51 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:30:51.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:51 smithi165 ceph-mon[105517]: osdmap e887: 8 total, 8 up, 8 in 2024-08-17T17:30:51.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:51 smithi165 ceph-mon[105517]: pgmap v987: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 565 MiB used, 715 GiB / 715 GiB avail 2024-08-17T17:30:51.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:30:51.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:51 smithi165 ceph-mon[105517]: osdmap e888: 8 total, 8 up, 8 in 2024-08-17T17:30:52.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:30:52.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:30:52.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:52 smithi136 ceph-mon[115078]: osdmap e889: 8 total, 8 up, 8 in 2024-08-17T17:30:52.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:30:52.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:30:52.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:52 smithi136 ceph-mon[109861]: osdmap e889: 8 total, 8 up, 8 in 2024-08-17T17:30:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:30:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:30:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:52 smithi165 ceph-mon[105517]: osdmap e889: 8 total, 8 up, 8 in 2024-08-17T17:30:53.442 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:53 smithi136 ceph-mon[109861]: pgmap v990: 105 pgs: 105 active+clean; 582 KiB data, 565 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:30:53.443 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:53 smithi136 ceph-mon[115078]: pgmap v990: 105 pgs: 105 active+clean; 582 KiB data, 565 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:30:53.453 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:53 smithi165 ceph-mon[105517]: pgmap v990: 105 pgs: 105 active+clean; 582 KiB data, 565 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:30:54.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:54 smithi136 ceph-mon[109861]: osdmap e890: 8 total, 8 up, 8 in 2024-08-17T17:30:54.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:54 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:30:54.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:54 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:30:54.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:54 smithi136 ceph-mon[115078]: osdmap e890: 8 total, 8 up, 8 in 2024-08-17T17:30:54.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:54 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:30:54.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:54 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:30:54.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:54 smithi165 ceph-mon[105517]: osdmap e890: 8 total, 8 up, 8 in 2024-08-17T17:30:54.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:54 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:30:54.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:54 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:30:55.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:55 smithi136 ceph-mon[109861]: pgmap v992: 105 pgs: 105 active+clean; 582 KiB data, 565 MiB used, 715 GiB / 715 GiB avail; 247 B/s wr, 0 op/s 2024-08-17T17:30:55.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:55 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:30:55.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:55 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:30:55.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:55 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:30:55.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:55 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:30:55.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:55 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:30:55.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:55 smithi136 ceph-mon[115078]: pgmap v992: 105 pgs: 105 active+clean; 582 KiB data, 565 MiB used, 715 GiB / 715 GiB avail; 247 B/s wr, 0 op/s 2024-08-17T17:30:55.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:55 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:30:55.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:55 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:30:55.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:55 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:30:55.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:55 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:30:55.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:55 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:30:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:55 smithi165 ceph-mon[105517]: pgmap v992: 105 pgs: 105 active+clean; 582 KiB data, 565 MiB used, 715 GiB / 715 GiB avail; 247 B/s wr, 0 op/s 2024-08-17T17:30:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:55 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:30:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:55 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:30:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:55 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:30:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:55 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:30:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:55 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:30:57.580 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:57 smithi165 ceph-mon[105517]: pgmap v993: 105 pgs: 105 active+clean; 9.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 1.5 MiB/s wr, 42 op/s 2024-08-17T17:30:57.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:57 smithi136 ceph-mon[109861]: pgmap v993: 105 pgs: 105 active+clean; 9.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 1.5 MiB/s wr, 42 op/s 2024-08-17T17:30:57.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:57 smithi136 ceph-mon[115078]: pgmap v993: 105 pgs: 105 active+clean; 9.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 1.5 MiB/s wr, 42 op/s 2024-08-17T17:30:57.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:30:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:30:57] "GET /metrics HTTP/1.1" 200 36414 "" "Prometheus/2.51.0" 2024-08-17T17:30:59.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:59 smithi136 ceph-mon[115078]: pgmap v994: 105 pgs: 105 active+clean; 26 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 3.5 MiB/s wr, 55 op/s 2024-08-17T17:30:59.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:30:59 smithi136 ceph-mon[115078]: osdmap e891: 8 total, 8 up, 8 in 2024-08-17T17:30:59.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:59 smithi136 ceph-mon[109861]: pgmap v994: 105 pgs: 105 active+clean; 26 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 3.5 MiB/s wr, 55 op/s 2024-08-17T17:30:59.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:30:59 smithi136 ceph-mon[109861]: osdmap e891: 8 total, 8 up, 8 in 2024-08-17T17:30:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:59 smithi165 ceph-mon[105517]: pgmap v994: 105 pgs: 105 active+clean; 26 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 3.5 MiB/s wr, 55 op/s 2024-08-17T17:30:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:30:59 smithi165 ceph-mon[105517]: osdmap e891: 8 total, 8 up, 8 in 2024-08-17T17:31:00.961 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/0.Simple (12858 ms) 2024-08-17T17:31:00.961 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/0.Grow 2024-08-17T17:31:00.961 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:31:01.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:01 smithi136 ceph-mon[115078]: pgmap v996: 105 pgs: 105 active+clean; 26 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 3.1 MiB/s wr, 48 op/s 2024-08-17T17:31:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:01 smithi136 ceph-mon[109861]: pgmap v996: 105 pgs: 105 active+clean; 26 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 3.1 MiB/s wr, 48 op/s 2024-08-17T17:31:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:01 smithi165 ceph-mon[105517]: pgmap v996: 105 pgs: 105 active+clean; 26 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 3.1 MiB/s wr, 48 op/s 2024-08-17T17:31:02.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:02 smithi136 ceph-mon[109861]: osdmap e892: 8 total, 8 up, 8 in 2024-08-17T17:31:02.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:02 smithi136 ceph-mon[115078]: osdmap e892: 8 total, 8 up, 8 in 2024-08-17T17:31:02.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:02 smithi165 ceph-mon[105517]: osdmap e892: 8 total, 8 up, 8 in 2024-08-17T17:31:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:03 smithi136 ceph-mon[109861]: pgmap v998: 105 pgs: 105 active+clean; 68 MiB data, 767 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.9 MiB/s wr, 122 op/s 2024-08-17T17:31:03.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:03 smithi136 ceph-mon[115078]: pgmap v998: 105 pgs: 105 active+clean; 68 MiB data, 767 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.9 MiB/s wr, 122 op/s 2024-08-17T17:31:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:03 smithi165 ceph-mon[105517]: pgmap v998: 105 pgs: 105 active+clean; 68 MiB data, 767 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.9 MiB/s wr, 122 op/s 2024-08-17T17:31:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:05 smithi136 ceph-mon[109861]: pgmap v999: 105 pgs: 105 active+clean; 68 MiB data, 767 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.8 MiB/s wr, 90 op/s 2024-08-17T17:31:05.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:05 smithi136 ceph-mon[115078]: pgmap v999: 105 pgs: 105 active+clean; 68 MiB data, 767 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.8 MiB/s wr, 90 op/s 2024-08-17T17:31:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:05 smithi165 ceph-mon[105517]: pgmap v999: 105 pgs: 105 active+clean; 68 MiB data, 767 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.8 MiB/s wr, 90 op/s 2024-08-17T17:31:06.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:31:06.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:31:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:31:07.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:07 smithi136 ceph-mon[109861]: pgmap v1000: 105 pgs: 105 active+clean; 80 MiB data, 803 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 7.3 MiB/s wr, 76 op/s 2024-08-17T17:31:07.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:07 smithi136 ceph-mon[115078]: pgmap v1000: 105 pgs: 105 active+clean; 80 MiB data, 803 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 7.3 MiB/s wr, 76 op/s 2024-08-17T17:31:07.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:31:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:31:07] "GET /metrics HTTP/1.1" 200 36475 "" "Prometheus/2.51.0" 2024-08-17T17:31:07.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:07 smithi165 ceph-mon[105517]: pgmap v1000: 105 pgs: 105 active+clean; 80 MiB data, 803 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 7.3 MiB/s wr, 76 op/s 2024-08-17T17:31:09.143 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/0.Grow (8182 ms) 2024-08-17T17:31:09.143 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/0.Shrink 2024-08-17T17:31:09.143 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:31:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:09 smithi136 ceph-mon[109861]: pgmap v1001: 105 pgs: 105 active+clean; 101 MiB data, 806 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 9.0 MiB/s wr, 94 op/s 2024-08-17T17:31:09.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:09 smithi136 ceph-mon[115078]: pgmap v1001: 105 pgs: 105 active+clean; 101 MiB data, 806 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 9.0 MiB/s wr, 94 op/s 2024-08-17T17:31:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:09 smithi165 ceph-mon[105517]: pgmap v1001: 105 pgs: 105 active+clean; 101 MiB data, 806 MiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 9.0 MiB/s wr, 94 op/s 2024-08-17T17:31:10.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:10 smithi136 ceph-mon[109861]: osdmap e893: 8 total, 8 up, 8 in 2024-08-17T17:31:10.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:10 smithi136 ceph-mon[115078]: osdmap e893: 8 total, 8 up, 8 in 2024-08-17T17:31:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:10 smithi165 ceph-mon[105517]: osdmap e893: 8 total, 8 up, 8 in 2024-08-17T17:31:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:11 smithi136 ceph-mon[109861]: pgmap v1003: 105 pgs: 105 active+clean; 101 MiB data, 806 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 6.9 MiB/s wr, 81 op/s 2024-08-17T17:31:11.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:11 smithi136 ceph-mon[115078]: pgmap v1003: 105 pgs: 105 active+clean; 101 MiB data, 806 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 6.9 MiB/s wr, 81 op/s 2024-08-17T17:31:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:11 smithi165 ceph-mon[105517]: pgmap v1003: 105 pgs: 105 active+clean; 101 MiB data, 806 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 6.9 MiB/s wr, 81 op/s 2024-08-17T17:31:13.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:13 smithi136 ceph-mon[109861]: pgmap v1004: 105 pgs: 105 active+clean; 134 MiB data, 960 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 67 op/s 2024-08-17T17:31:13.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:13 smithi136 ceph-mon[115078]: pgmap v1004: 105 pgs: 105 active+clean; 134 MiB data, 960 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 67 op/s 2024-08-17T17:31:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:13 smithi165 ceph-mon[105517]: pgmap v1004: 105 pgs: 105 active+clean; 134 MiB data, 960 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 67 op/s 2024-08-17T17:31:15.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:15 smithi136 ceph-mon[109861]: pgmap v1005: 105 pgs: 105 active+clean; 134 MiB data, 960 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 67 op/s 2024-08-17T17:31:15.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:15 smithi136 ceph-mon[115078]: pgmap v1005: 105 pgs: 105 active+clean; 134 MiB data, 960 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 67 op/s 2024-08-17T17:31:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:15 smithi165 ceph-mon[105517]: pgmap v1005: 105 pgs: 105 active+clean; 134 MiB data, 960 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 67 op/s 2024-08-17T17:31:17.159 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/0.Shrink (8015 ms) 2024-08-17T17:31:17.159 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/0.ShrinkToOne 2024-08-17T17:31:17.159 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:31:17.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:17 smithi136 ceph-mon[109861]: pgmap v1006: 105 pgs: 105 active+clean; 158 MiB data, 995 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 8.2 MiB/s wr, 77 op/s 2024-08-17T17:31:17.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:31:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:31:17] "GET /metrics HTTP/1.1" 200 36475 "" "Prometheus/2.51.0" 2024-08-17T17:31:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:17 smithi136 ceph-mon[115078]: pgmap v1006: 105 pgs: 105 active+clean; 158 MiB data, 995 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 8.2 MiB/s wr, 77 op/s 2024-08-17T17:31:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:17 smithi165 ceph-mon[105517]: pgmap v1006: 105 pgs: 105 active+clean; 158 MiB data, 995 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 8.2 MiB/s wr, 77 op/s 2024-08-17T17:31:18.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:18 smithi136 ceph-mon[109861]: osdmap e894: 8 total, 8 up, 8 in 2024-08-17T17:31:18.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:18 smithi136 ceph-mon[115078]: osdmap e894: 8 total, 8 up, 8 in 2024-08-17T17:31:18.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:18 smithi165 ceph-mon[105517]: osdmap e894: 8 total, 8 up, 8 in 2024-08-17T17:31:19.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:19 smithi136 ceph-mon[109861]: pgmap v1008: 105 pgs: 105 active+clean; 163 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 7.4 MiB/s wr, 75 op/s 2024-08-17T17:31:19.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:19 smithi136 ceph-mon[115078]: pgmap v1008: 105 pgs: 105 active+clean; 163 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 7.4 MiB/s wr, 75 op/s 2024-08-17T17:31:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:19 smithi165 ceph-mon[105517]: pgmap v1008: 105 pgs: 105 active+clean; 163 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 7.4 MiB/s wr, 75 op/s 2024-08-17T17:31:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:21 smithi136 ceph-mon[109861]: pgmap v1009: 105 pgs: 105 active+clean; 163 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 6.6 MiB/s wr, 67 op/s 2024-08-17T17:31:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:31:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:21 smithi136 ceph-mon[115078]: pgmap v1009: 105 pgs: 105 active+clean; 163 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 6.6 MiB/s wr, 67 op/s 2024-08-17T17:31:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:31:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:21 smithi165 ceph-mon[105517]: pgmap v1009: 105 pgs: 105 active+clean; 163 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 6.6 MiB/s wr, 67 op/s 2024-08-17T17:31:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:31:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:23 smithi136 ceph-mon[109861]: pgmap v1010: 105 pgs: 105 active+clean; 192 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.2 MiB/s wr, 74 op/s 2024-08-17T17:31:23.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:23 smithi136 ceph-mon[115078]: pgmap v1010: 105 pgs: 105 active+clean; 192 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.2 MiB/s wr, 74 op/s 2024-08-17T17:31:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:23 smithi165 ceph-mon[105517]: pgmap v1010: 105 pgs: 105 active+clean; 192 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.2 MiB/s wr, 74 op/s 2024-08-17T17:31:25.051 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/0.ShrinkToOne (7893 ms) 2024-08-17T17:31:25.051 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/0.ShrinkToOneAfterSnapshot 2024-08-17T17:31:25.051 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:31:25.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:24 smithi136 ceph-mon[109861]: pgmap v1011: 105 pgs: 105 active+clean; 192 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.2 MiB/s wr, 74 op/s 2024-08-17T17:31:25.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:24 smithi136 ceph-mon[115078]: pgmap v1011: 105 pgs: 105 active+clean; 192 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.2 MiB/s wr, 74 op/s 2024-08-17T17:31:25.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:24 smithi165 ceph-mon[105517]: pgmap v1011: 105 pgs: 105 active+clean; 192 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 6.2 MiB/s wr, 74 op/s 2024-08-17T17:31:27.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:27 smithi165 ceph-mon[105517]: osdmap e895: 8 total, 8 up, 8 in 2024-08-17T17:31:27.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:27 smithi165 ceph-mon[105517]: pgmap v1013: 105 pgs: 105 active+clean; 201 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 4.7 MiB/s wr, 69 op/s 2024-08-17T17:31:27.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:27 smithi136 ceph-mon[109861]: osdmap e895: 8 total, 8 up, 8 in 2024-08-17T17:31:27.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:27 smithi136 ceph-mon[109861]: pgmap v1013: 105 pgs: 105 active+clean; 201 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 4.7 MiB/s wr, 69 op/s 2024-08-17T17:31:27.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:27 smithi136 ceph-mon[115078]: osdmap e895: 8 total, 8 up, 8 in 2024-08-17T17:31:27.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:27 smithi136 ceph-mon[115078]: pgmap v1013: 105 pgs: 105 active+clean; 201 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 4.7 MiB/s wr, 69 op/s 2024-08-17T17:31:27.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:31:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:31:27] "GET /metrics HTTP/1.1" 200 36480 "" "Prometheus/2.51.0" 2024-08-17T17:31:29.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:29 smithi136 ceph-mon[109861]: pgmap v1014: 105 pgs: 105 active+clean; 221 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 6.6 MiB/s wr, 86 op/s 2024-08-17T17:31:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:29 smithi136 ceph-mon[115078]: pgmap v1014: 105 pgs: 105 active+clean; 221 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 6.6 MiB/s wr, 86 op/s 2024-08-17T17:31:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:29 smithi165 ceph-mon[105517]: pgmap v1014: 105 pgs: 105 active+clean; 221 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 6.6 MiB/s wr, 86 op/s 2024-08-17T17:31:31.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:31 smithi136 ceph-mon[109861]: pgmap v1015: 105 pgs: 105 active+clean; 221 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 6.6 MiB/s wr, 86 op/s 2024-08-17T17:31:31.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:31 smithi136 ceph-mon[115078]: pgmap v1015: 105 pgs: 105 active+clean; 221 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 6.6 MiB/s wr, 86 op/s 2024-08-17T17:31:31.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:31 smithi165 ceph-mon[105517]: pgmap v1015: 105 pgs: 105 active+clean; 221 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 6.6 MiB/s wr, 86 op/s 2024-08-17T17:31:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:33 smithi136 ceph-mon[109861]: pgmap v1016: 105 pgs: 105 active+clean; 225 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 975 KiB/s rd, 4.1 MiB/s wr, 57 op/s 2024-08-17T17:31:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:33 smithi136 ceph-mon[109861]: osdmap e896: 8 total, 8 up, 8 in 2024-08-17T17:31:33.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:33 smithi136 ceph-mon[115078]: pgmap v1016: 105 pgs: 105 active+clean; 225 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 975 KiB/s rd, 4.1 MiB/s wr, 57 op/s 2024-08-17T17:31:33.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:33 smithi136 ceph-mon[115078]: osdmap e896: 8 total, 8 up, 8 in 2024-08-17T17:31:33.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:33 smithi165 ceph-mon[105517]: pgmap v1016: 105 pgs: 105 active+clean; 225 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 975 KiB/s rd, 4.1 MiB/s wr, 57 op/s 2024-08-17T17:31:33.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:33 smithi165 ceph-mon[105517]: osdmap e896: 8 total, 8 up, 8 in 2024-08-17T17:31:34.578 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/0.ShrinkToOneAfterSnapshot (9526 ms) 2024-08-17T17:31:34.578 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/0.MinOverlap 2024-08-17T17:31:34.578 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:31:35.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:35 smithi136 ceph-mon[109861]: pgmap v1018: 105 pgs: 105 active+clean; 225 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 641 KiB/s rd, 3.4 MiB/s wr, 49 op/s 2024-08-17T17:31:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:35 smithi136 ceph-mon[115078]: pgmap v1018: 105 pgs: 105 active+clean; 225 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 641 KiB/s rd, 3.4 MiB/s wr, 49 op/s 2024-08-17T17:31:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:35 smithi165 ceph-mon[105517]: pgmap v1018: 105 pgs: 105 active+clean; 225 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 641 KiB/s rd, 3.4 MiB/s wr, 49 op/s 2024-08-17T17:31:36.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:36 smithi136 ceph-mon[115078]: osdmap e897: 8 total, 8 up, 8 in 2024-08-17T17:31:36.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:31:36.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:36 smithi136 ceph-mon[109861]: osdmap e897: 8 total, 8 up, 8 in 2024-08-17T17:31:36.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:31:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:36 smithi165 ceph-mon[105517]: osdmap e897: 8 total, 8 up, 8 in 2024-08-17T17:31:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:31:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:37 smithi136 ceph-mon[109861]: pgmap v1020: 105 pgs: 105 active+clean; 265 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.5 MiB/s wr, 58 op/s 2024-08-17T17:31:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:37 smithi136 ceph-mon[115078]: pgmap v1020: 105 pgs: 105 active+clean; 265 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.5 MiB/s wr, 58 op/s 2024-08-17T17:31:37.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:31:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:31:37] "GET /metrics HTTP/1.1" 200 36482 "" "Prometheus/2.51.0" 2024-08-17T17:31:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:37 smithi165 ceph-mon[105517]: pgmap v1020: 105 pgs: 105 active+clean; 265 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.5 MiB/s wr, 58 op/s 2024-08-17T17:31:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:39 smithi136 ceph-mon[109861]: pgmap v1021: 105 pgs: 105 active+clean; 307 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 11 MiB/s wr, 104 op/s 2024-08-17T17:31:39.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:39 smithi136 ceph-mon[115078]: pgmap v1021: 105 pgs: 105 active+clean; 307 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 11 MiB/s wr, 104 op/s 2024-08-17T17:31:39.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:39 smithi165 ceph-mon[105517]: pgmap v1021: 105 pgs: 105 active+clean; 307 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 11 MiB/s wr, 104 op/s 2024-08-17T17:31:41.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:41 smithi136 ceph-mon[109861]: pgmap v1022: 105 pgs: 105 active+clean; 307 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 10 MiB/s wr, 83 op/s 2024-08-17T17:31:41.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:41 smithi136 ceph-mon[115078]: pgmap v1022: 105 pgs: 105 active+clean; 307 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 10 MiB/s wr, 83 op/s 2024-08-17T17:31:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:41 smithi165 ceph-mon[105517]: pgmap v1022: 105 pgs: 105 active+clean; 307 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 10 MiB/s wr, 83 op/s 2024-08-17T17:31:42.922 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/0.MinOverlap (8344 ms) 2024-08-17T17:31:42.922 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/0.ZeroOverlap 2024-08-17T17:31:42.922 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:31:43.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:43 smithi136 ceph-mon[109861]: pgmap v1023: 105 pgs: 105 active+clean; 311 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 9.1 MiB/s wr, 79 op/s 2024-08-17T17:31:43.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:43 smithi136 ceph-mon[115078]: pgmap v1023: 105 pgs: 105 active+clean; 311 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 9.1 MiB/s wr, 79 op/s 2024-08-17T17:31:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:43 smithi165 ceph-mon[105517]: pgmap v1023: 105 pgs: 105 active+clean; 311 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 9.1 MiB/s wr, 79 op/s 2024-08-17T17:31:44.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:44 smithi136 ceph-mon[109861]: osdmap e898: 8 total, 8 up, 8 in 2024-08-17T17:31:44.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:44 smithi136 ceph-mon[115078]: osdmap e898: 8 total, 8 up, 8 in 2024-08-17T17:31:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:44 smithi165 ceph-mon[105517]: osdmap e898: 8 total, 8 up, 8 in 2024-08-17T17:31:45.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:45 smithi136 ceph-mon[109861]: pgmap v1025: 105 pgs: 105 active+clean; 311 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 10 MiB/s wr, 88 op/s 2024-08-17T17:31:45.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:45 smithi136 ceph-mon[115078]: pgmap v1025: 105 pgs: 105 active+clean; 311 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 10 MiB/s wr, 88 op/s 2024-08-17T17:31:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:45 smithi165 ceph-mon[105517]: pgmap v1025: 105 pgs: 105 active+clean; 311 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 10 MiB/s wr, 88 op/s 2024-08-17T17:31:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:47 smithi136 ceph-mon[109861]: pgmap v1026: 105 pgs: 105 active+clean; 327 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.1 MiB/s rd, 7.0 MiB/s wr, 74 op/s 2024-08-17T17:31:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:47 smithi136 ceph-mon[115078]: pgmap v1026: 105 pgs: 105 active+clean; 327 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.1 MiB/s rd, 7.0 MiB/s wr, 74 op/s 2024-08-17T17:31:47.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:31:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:31:47] "GET /metrics HTTP/1.1" 200 36482 "" "Prometheus/2.51.0" 2024-08-17T17:31:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:47 smithi165 ceph-mon[105517]: pgmap v1026: 105 pgs: 105 active+clean; 327 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.1 MiB/s rd, 7.0 MiB/s wr, 74 op/s 2024-08-17T17:31:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:49 smithi136 ceph-mon[109861]: pgmap v1027: 105 pgs: 105 active+clean; 340 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 62 op/s 2024-08-17T17:31:49.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:49 smithi136 ceph-mon[115078]: pgmap v1027: 105 pgs: 105 active+clean; 340 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 62 op/s 2024-08-17T17:31:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:49 smithi165 ceph-mon[105517]: pgmap v1027: 105 pgs: 105 active+clean; 340 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 62 op/s 2024-08-17T17:31:50.999 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/0.ZeroOverlap (8077 ms) 2024-08-17T17:31:51.335 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 7 tests from EncryptedFlattenTest/0 (62896 ms total) 2024-08-17T17:31:51.335 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:31:51.335 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 7 tests from EncryptedFlattenTest/1, where TypeParam = PlaintextUnderLUKS2 2024-08-17T17:31:51.362 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/1.Simple 2024-08-17T17:31:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:51 smithi136 ceph-mon[109861]: pgmap v1028: 105 pgs: 105 active+clean; 340 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 62 op/s 2024-08-17T17:31:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:31:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:51 smithi136 ceph-mon[115078]: pgmap v1028: 105 pgs: 105 active+clean; 340 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 62 op/s 2024-08-17T17:31:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:31:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:51 smithi165 ceph-mon[105517]: pgmap v1028: 105 pgs: 105 active+clean; 340 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 62 op/s 2024-08-17T17:31:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:31:52.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:52 smithi136 ceph-mon[109861]: osdmap e899: 8 total, 8 up, 8 in 2024-08-17T17:31:52.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:52 smithi136 ceph-mon[115078]: osdmap e899: 8 total, 8 up, 8 in 2024-08-17T17:31:52.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:52 smithi165 ceph-mon[105517]: osdmap e899: 8 total, 8 up, 8 in 2024-08-17T17:31:53.359 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:31:53.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:53 smithi136 ceph-mon[115078]: pgmap v1030: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:31:53.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:53 smithi136 ceph-mon[115078]: osdmap e900: 8 total, 8 up, 8 in 2024-08-17T17:31:53.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:53 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/889165381' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-40","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:31:53.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:53 smithi136 ceph-mon[109861]: pgmap v1030: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:31:53.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:53 smithi136 ceph-mon[109861]: osdmap e900: 8 total, 8 up, 8 in 2024-08-17T17:31:53.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:53 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/889165381' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-40","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:31:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:53 smithi165 ceph-mon[105517]: pgmap v1030: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:31:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:53 smithi165 ceph-mon[105517]: osdmap e900: 8 total, 8 up, 8 in 2024-08-17T17:31:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:53 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/889165381' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-40","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:31:54.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:54 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/889165381' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:31:54.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:54 smithi136 ceph-mon[109861]: osdmap e901: 8 total, 8 up, 8 in 2024-08-17T17:31:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:54 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/889165381' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:31:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:54 smithi136 ceph-mon[115078]: osdmap e901: 8 total, 8 up, 8 in 2024-08-17T17:31:54.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:54 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/889165381' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:31:54.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:54 smithi165 ceph-mon[105517]: osdmap e901: 8 total, 8 up, 8 in 2024-08-17T17:31:55.639 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:55 smithi136 ceph-mon[109861]: pgmap v1033: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:31:55.640 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:55 smithi136 ceph-mon[109861]: osdmap e902: 8 total, 8 up, 8 in 2024-08-17T17:31:55.640 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:55 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:31:55.640 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:55 smithi136 ceph-mon[115078]: pgmap v1033: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:31:55.640 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:55 smithi136 ceph-mon[115078]: osdmap e902: 8 total, 8 up, 8 in 2024-08-17T17:31:55.640 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:55 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:31:55.643 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:55 smithi165 ceph-mon[105517]: pgmap v1033: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:31:55.643 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:55 smithi165 ceph-mon[105517]: osdmap e902: 8 total, 8 up, 8 in 2024-08-17T17:31:55.643 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:55 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:31:56.687 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:56 smithi165 ceph-mon[105517]: osdmap e903: 8 total, 8 up, 8 in 2024-08-17T17:31:56.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:56 smithi136 ceph-mon[109861]: osdmap e903: 8 total, 8 up, 8 in 2024-08-17T17:31:56.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:56 smithi136 ceph-mon[115078]: osdmap e903: 8 total, 8 up, 8 in 2024-08-17T17:31:57.588 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:57 smithi165 ceph-mon[105517]: pgmap v1036: 105 pgs: 16 unknown, 89 active+clean; 13 MiB data, 975 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 3.0 MiB/s wr, 1 op/s 2024-08-17T17:31:57.588 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:57 smithi165 ceph-mon[105517]: osdmap e904: 8 total, 8 up, 8 in 2024-08-17T17:31:57.686 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:31:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:31:57] "GET /metrics HTTP/1.1" 200 36484 "" "Prometheus/2.51.0" 2024-08-17T17:31:57.686 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:57 smithi136 ceph-mon[109861]: pgmap v1036: 105 pgs: 16 unknown, 89 active+clean; 13 MiB data, 975 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 3.0 MiB/s wr, 1 op/s 2024-08-17T17:31:57.686 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:57 smithi136 ceph-mon[109861]: osdmap e904: 8 total, 8 up, 8 in 2024-08-17T17:31:57.687 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:57 smithi136 ceph-mon[115078]: pgmap v1036: 105 pgs: 16 unknown, 89 active+clean; 13 MiB data, 975 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 3.0 MiB/s wr, 1 op/s 2024-08-17T17:31:57.687 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:57 smithi136 ceph-mon[115078]: osdmap e904: 8 total, 8 up, 8 in 2024-08-17T17:31:59.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:59 smithi136 ceph-mon[109861]: pgmap v1038: 105 pgs: 105 active+clean; 26 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 5.1 MiB/s wr, 71 op/s 2024-08-17T17:31:59.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:59 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:31:59.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:59 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:31:59.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:31:59 smithi136 ceph-mon[109861]: osdmap e905: 8 total, 8 up, 8 in 2024-08-17T17:31:59.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:59 smithi136 ceph-mon[115078]: pgmap v1038: 105 pgs: 105 active+clean; 26 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 5.1 MiB/s wr, 71 op/s 2024-08-17T17:31:59.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:59 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:31:59.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:59 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:31:59.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:31:59 smithi136 ceph-mon[115078]: osdmap e905: 8 total, 8 up, 8 in 2024-08-17T17:31:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:59 smithi165 ceph-mon[105517]: pgmap v1038: 105 pgs: 105 active+clean; 26 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 5.1 MiB/s wr, 71 op/s 2024-08-17T17:31:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:59 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:31:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:59 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:31:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:31:59 smithi165 ceph-mon[105517]: osdmap e905: 8 total, 8 up, 8 in 2024-08-17T17:32:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:00 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:32:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:00 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:32:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:00 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:32:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:00 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:32:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:00 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:32:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:00 smithi136 ceph-mon[109861]: pgmap v1040: 105 pgs: 105 active+clean; 26 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 4.3 MiB/s wr, 59 op/s 2024-08-17T17:32:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:00 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:32:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:00 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:32:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:00 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:32:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:00 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:32:00.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:00 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:32:00.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:00 smithi136 ceph-mon[115078]: pgmap v1040: 105 pgs: 105 active+clean; 26 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 4.3 MiB/s wr, 59 op/s 2024-08-17T17:32:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:00 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:32:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:00 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:32:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:00 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:32:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:00 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:32:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:00 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:32:01.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:00 smithi165 ceph-mon[105517]: pgmap v1040: 105 pgs: 105 active+clean; 26 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 4.3 MiB/s wr, 59 op/s 2024-08-17T17:32:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:03 smithi136 ceph-mon[109861]: pgmap v1041: 105 pgs: 105 active+clean; 26 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 3.6 MiB/s wr, 54 op/s 2024-08-17T17:32:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:03 smithi136 ceph-mon[115078]: pgmap v1041: 105 pgs: 105 active+clean; 26 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 3.6 MiB/s wr, 54 op/s 2024-08-17T17:32:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:03 smithi165 ceph-mon[105517]: pgmap v1041: 105 pgs: 105 active+clean; 26 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 3.6 MiB/s wr, 54 op/s 2024-08-17T17:32:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:05 smithi136 ceph-mon[109861]: pgmap v1042: 105 pgs: 105 active+clean; 26 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 1.6 MiB/s wr, 45 op/s 2024-08-17T17:32:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:05 smithi136 ceph-mon[115078]: pgmap v1042: 105 pgs: 105 active+clean; 26 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 1.6 MiB/s wr, 45 op/s 2024-08-17T17:32:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:05 smithi165 ceph-mon[105517]: pgmap v1042: 105 pgs: 105 active+clean; 26 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 1.6 MiB/s wr, 45 op/s 2024-08-17T17:32:06.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:32:06.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:32:06.746 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/1.Simple (15384 ms) 2024-08-17T17:32:06.746 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/1.Grow 2024-08-17T17:32:06.746 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:32:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:32:07.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:07 smithi136 ceph-mon[109861]: pgmap v1043: 105 pgs: 105 active+clean; 46 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 4.5 MiB/s rd, 4.2 MiB/s wr, 50 op/s 2024-08-17T17:32:07.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:07 smithi136 ceph-mon[115078]: pgmap v1043: 105 pgs: 105 active+clean; 46 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 4.5 MiB/s rd, 4.2 MiB/s wr, 50 op/s 2024-08-17T17:32:07.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:32:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:32:07] "GET /metrics HTTP/1.1" 200 36468 "" "Prometheus/2.51.0" 2024-08-17T17:32:07.759 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:07 smithi165 ceph-mon[105517]: pgmap v1043: 105 pgs: 105 active+clean; 46 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 4.5 MiB/s rd, 4.2 MiB/s wr, 50 op/s 2024-08-17T17:32:08.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:08 smithi136 ceph-mon[109861]: osdmap e906: 8 total, 8 up, 8 in 2024-08-17T17:32:08.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:08 smithi136 ceph-mon[115078]: osdmap e906: 8 total, 8 up, 8 in 2024-08-17T17:32:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:08 smithi165 ceph-mon[105517]: osdmap e906: 8 total, 8 up, 8 in 2024-08-17T17:32:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:09 smithi136 ceph-mon[109861]: pgmap v1045: 105 pgs: 105 active+clean; 87 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 8.3 MiB/s wr, 42 op/s 2024-08-17T17:32:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:09 smithi136 ceph-mon[115078]: pgmap v1045: 105 pgs: 105 active+clean; 87 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 8.3 MiB/s wr, 42 op/s 2024-08-17T17:32:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:09 smithi165 ceph-mon[105517]: pgmap v1045: 105 pgs: 105 active+clean; 87 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 8.3 MiB/s wr, 42 op/s 2024-08-17T17:32:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:11 smithi136 ceph-mon[109861]: pgmap v1046: 105 pgs: 105 active+clean; 87 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 7.7 MiB/s wr, 38 op/s 2024-08-17T17:32:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:11 smithi136 ceph-mon[115078]: pgmap v1046: 105 pgs: 105 active+clean; 87 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 7.7 MiB/s wr, 38 op/s 2024-08-17T17:32:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:11 smithi165 ceph-mon[105517]: pgmap v1046: 105 pgs: 105 active+clean; 87 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 7.7 MiB/s wr, 38 op/s 2024-08-17T17:32:13.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:13 smithi136 ceph-mon[109861]: pgmap v1047: 105 pgs: 105 active+clean; 92 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 66 op/s 2024-08-17T17:32:13.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:13 smithi136 ceph-mon[115078]: pgmap v1047: 105 pgs: 105 active+clean; 92 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 66 op/s 2024-08-17T17:32:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:13 smithi165 ceph-mon[105517]: pgmap v1047: 105 pgs: 105 active+clean; 92 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 66 op/s 2024-08-17T17:32:15.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:15 smithi136 ceph-mon[109861]: pgmap v1048: 105 pgs: 105 active+clean; 92 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 66 op/s 2024-08-17T17:32:15.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:15 smithi136 ceph-mon[115078]: pgmap v1048: 105 pgs: 105 active+clean; 92 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 66 op/s 2024-08-17T17:32:15.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:15 smithi165 ceph-mon[105517]: pgmap v1048: 105 pgs: 105 active+clean; 92 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 66 op/s 2024-08-17T17:32:17.684 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/1.Grow (10938 ms) 2024-08-17T17:32:17.685 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/1.Shrink 2024-08-17T17:32:17.685 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:32:17.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:17 smithi136 ceph-mon[109861]: pgmap v1049: 105 pgs: 105 active+clean; 108 MiB data, 846 MiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 7.8 MiB/s wr, 67 op/s 2024-08-17T17:32:17.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:17 smithi136 ceph-mon[115078]: pgmap v1049: 105 pgs: 105 active+clean; 108 MiB data, 846 MiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 7.8 MiB/s wr, 67 op/s 2024-08-17T17:32:17.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:32:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:32:17] "GET /metrics HTTP/1.1" 200 36468 "" "Prometheus/2.51.0" 2024-08-17T17:32:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:17 smithi165 ceph-mon[105517]: pgmap v1049: 105 pgs: 105 active+clean; 108 MiB data, 846 MiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 7.8 MiB/s wr, 67 op/s 2024-08-17T17:32:19.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:19 smithi136 ceph-mon[109861]: pgmap v1050: 105 pgs: 105 active+clean; 121 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 8.0 MiB/s rd, 4.2 MiB/s wr, 54 op/s 2024-08-17T17:32:19.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:19 smithi136 ceph-mon[109861]: osdmap e907: 8 total, 8 up, 8 in 2024-08-17T17:32:19.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:19 smithi136 ceph-mon[115078]: pgmap v1050: 105 pgs: 105 active+clean; 121 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 8.0 MiB/s rd, 4.2 MiB/s wr, 54 op/s 2024-08-17T17:32:19.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:19 smithi136 ceph-mon[115078]: osdmap e907: 8 total, 8 up, 8 in 2024-08-17T17:32:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:19 smithi165 ceph-mon[105517]: pgmap v1050: 105 pgs: 105 active+clean; 121 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 8.0 MiB/s rd, 4.2 MiB/s wr, 54 op/s 2024-08-17T17:32:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:19 smithi165 ceph-mon[105517]: osdmap e907: 8 total, 8 up, 8 in 2024-08-17T17:32:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:21 smithi136 ceph-mon[109861]: pgmap v1052: 105 pgs: 105 active+clean; 121 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 4.6 MiB/s wr, 59 op/s 2024-08-17T17:32:21.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:32:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:21 smithi136 ceph-mon[115078]: pgmap v1052: 105 pgs: 105 active+clean; 121 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 4.6 MiB/s wr, 59 op/s 2024-08-17T17:32:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:32:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:21 smithi165 ceph-mon[105517]: pgmap v1052: 105 pgs: 105 active+clean; 121 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 4.6 MiB/s wr, 59 op/s 2024-08-17T17:32:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:32:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:23 smithi136 ceph-mon[109861]: pgmap v1053: 105 pgs: 105 active+clean; 158 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 68 op/s 2024-08-17T17:32:23.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:23 smithi136 ceph-mon[115078]: pgmap v1053: 105 pgs: 105 active+clean; 158 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 68 op/s 2024-08-17T17:32:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:23 smithi165 ceph-mon[105517]: pgmap v1053: 105 pgs: 105 active+clean; 158 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 68 op/s 2024-08-17T17:32:25.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:25 smithi136 ceph-mon[109861]: pgmap v1054: 105 pgs: 105 active+clean; 158 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 68 op/s 2024-08-17T17:32:25.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:25 smithi136 ceph-mon[115078]: pgmap v1054: 105 pgs: 105 active+clean; 158 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 68 op/s 2024-08-17T17:32:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:25 smithi165 ceph-mon[105517]: pgmap v1054: 105 pgs: 105 active+clean; 158 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 68 op/s 2024-08-17T17:32:27.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:27 smithi136 ceph-mon[115078]: pgmap v1055: 105 pgs: 105 active+clean; 174 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.4 MiB/s wr, 59 op/s 2024-08-17T17:32:27.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:27 smithi136 ceph-mon[109861]: pgmap v1055: 105 pgs: 105 active+clean; 174 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.4 MiB/s wr, 59 op/s 2024-08-17T17:32:27.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:32:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:32:27] "GET /metrics HTTP/1.1" 200 36483 "" "Prometheus/2.51.0" 2024-08-17T17:32:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:27 smithi165 ceph-mon[105517]: pgmap v1055: 105 pgs: 105 active+clean; 174 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.4 MiB/s wr, 59 op/s 2024-08-17T17:32:27.811 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/1.Shrink (10127 ms) 2024-08-17T17:32:27.811 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/1.ShrinkToOne 2024-08-17T17:32:27.811 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:32:29.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:29 smithi136 ceph-mon[115078]: pgmap v1056: 105 pgs: 105 active+clean; 191 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 8.6 MiB/s wr, 69 op/s 2024-08-17T17:32:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:29 smithi136 ceph-mon[115078]: osdmap e908: 8 total, 8 up, 8 in 2024-08-17T17:32:29.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:29 smithi136 ceph-mon[109861]: pgmap v1056: 105 pgs: 105 active+clean; 191 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 8.6 MiB/s wr, 69 op/s 2024-08-17T17:32:29.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:29 smithi136 ceph-mon[109861]: osdmap e908: 8 total, 8 up, 8 in 2024-08-17T17:32:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:29 smithi165 ceph-mon[105517]: pgmap v1056: 105 pgs: 105 active+clean; 191 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 8.6 MiB/s wr, 69 op/s 2024-08-17T17:32:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:29 smithi165 ceph-mon[105517]: osdmap e908: 8 total, 8 up, 8 in 2024-08-17T17:32:31.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:31 smithi136 ceph-mon[109861]: pgmap v1058: 105 pgs: 105 active+clean; 191 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 8.6 MiB/s wr, 69 op/s 2024-08-17T17:32:31.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:31 smithi136 ceph-mon[115078]: pgmap v1058: 105 pgs: 105 active+clean; 191 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 8.6 MiB/s wr, 69 op/s 2024-08-17T17:32:31.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:31 smithi165 ceph-mon[105517]: pgmap v1058: 105 pgs: 105 active+clean; 191 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 8.6 MiB/s wr, 69 op/s 2024-08-17T17:32:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:33 smithi136 ceph-mon[109861]: pgmap v1059: 105 pgs: 105 active+clean; 224 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 69 op/s 2024-08-17T17:32:33.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:33 smithi136 ceph-mon[115078]: pgmap v1059: 105 pgs: 105 active+clean; 224 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 69 op/s 2024-08-17T17:32:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:33 smithi165 ceph-mon[105517]: pgmap v1059: 105 pgs: 105 active+clean; 224 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 69 op/s 2024-08-17T17:32:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:35 smithi136 ceph-mon[115078]: pgmap v1060: 105 pgs: 105 active+clean; 224 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 69 op/s 2024-08-17T17:32:35.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:35 smithi136 ceph-mon[109861]: pgmap v1060: 105 pgs: 105 active+clean; 224 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 69 op/s 2024-08-17T17:32:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:35 smithi165 ceph-mon[105517]: pgmap v1060: 105 pgs: 105 active+clean; 224 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 69 op/s 2024-08-17T17:32:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:32:36.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:32:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:32:37.649 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/1.ShrinkToOne (9838 ms) 2024-08-17T17:32:37.650 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/1.ShrinkToOneAfterSnapshot 2024-08-17T17:32:37.650 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:32:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:37 smithi136 ceph-mon[109861]: pgmap v1061: 105 pgs: 105 active+clean; 236 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 8.6 MiB/s wr, 68 op/s 2024-08-17T17:32:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:37 smithi136 ceph-mon[115078]: pgmap v1061: 105 pgs: 105 active+clean; 236 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 8.6 MiB/s wr, 68 op/s 2024-08-17T17:32:37.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:32:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:32:37] "GET /metrics HTTP/1.1" 200 36485 "" "Prometheus/2.51.0" 2024-08-17T17:32:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:37 smithi165 ceph-mon[105517]: pgmap v1061: 105 pgs: 105 active+clean; 236 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 8.6 MiB/s wr, 68 op/s 2024-08-17T17:32:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:39 smithi136 ceph-mon[109861]: pgmap v1062: 105 pgs: 105 active+clean; 244 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 7.3 MiB/s wr, 61 op/s 2024-08-17T17:32:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:39 smithi136 ceph-mon[109861]: osdmap e909: 8 total, 8 up, 8 in 2024-08-17T17:32:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:39 smithi136 ceph-mon[115078]: pgmap v1062: 105 pgs: 105 active+clean; 244 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 7.3 MiB/s wr, 61 op/s 2024-08-17T17:32:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:39 smithi136 ceph-mon[115078]: osdmap e909: 8 total, 8 up, 8 in 2024-08-17T17:32:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:39 smithi165 ceph-mon[105517]: pgmap v1062: 105 pgs: 105 active+clean; 244 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 7.3 MiB/s wr, 61 op/s 2024-08-17T17:32:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:39 smithi165 ceph-mon[105517]: osdmap e909: 8 total, 8 up, 8 in 2024-08-17T17:32:41.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:41 smithi136 ceph-mon[109861]: pgmap v1064: 105 pgs: 105 active+clean; 244 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 7.3 MiB/s wr, 61 op/s 2024-08-17T17:32:41.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:41 smithi136 ceph-mon[115078]: pgmap v1064: 105 pgs: 105 active+clean; 244 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 7.3 MiB/s wr, 61 op/s 2024-08-17T17:32:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:41 smithi165 ceph-mon[105517]: pgmap v1064: 105 pgs: 105 active+clean; 244 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 7.3 MiB/s wr, 61 op/s 2024-08-17T17:32:43.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:43 smithi136 ceph-mon[109861]: pgmap v1065: 105 pgs: 105 active+clean; 269 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 60 op/s 2024-08-17T17:32:43.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:43 smithi136 ceph-mon[115078]: pgmap v1065: 105 pgs: 105 active+clean; 269 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 60 op/s 2024-08-17T17:32:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:43 smithi165 ceph-mon[105517]: pgmap v1065: 105 pgs: 105 active+clean; 269 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 60 op/s 2024-08-17T17:32:45.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:45 smithi136 ceph-mon[109861]: pgmap v1066: 105 pgs: 105 active+clean; 269 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 60 op/s 2024-08-17T17:32:45.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:45 smithi136 ceph-mon[115078]: pgmap v1066: 105 pgs: 105 active+clean; 269 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 60 op/s 2024-08-17T17:32:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:45 smithi165 ceph-mon[105517]: pgmap v1066: 105 pgs: 105 active+clean; 269 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 60 op/s 2024-08-17T17:32:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:47 smithi136 ceph-mon[109861]: pgmap v1067: 105 pgs: 105 active+clean; 277 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 5.3 MiB/s wr, 56 op/s 2024-08-17T17:32:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:47 smithi136 ceph-mon[109861]: osdmap e910: 8 total, 8 up, 8 in 2024-08-17T17:32:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:47 smithi136 ceph-mon[115078]: pgmap v1067: 105 pgs: 105 active+clean; 277 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 5.3 MiB/s wr, 56 op/s 2024-08-17T17:32:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:47 smithi136 ceph-mon[115078]: osdmap e910: 8 total, 8 up, 8 in 2024-08-17T17:32:47.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:32:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:32:47] "GET /metrics HTTP/1.1" 200 36485 "" "Prometheus/2.51.0" 2024-08-17T17:32:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:47 smithi165 ceph-mon[105517]: pgmap v1067: 105 pgs: 105 active+clean; 277 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 5.3 MiB/s wr, 56 op/s 2024-08-17T17:32:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:47 smithi165 ceph-mon[105517]: osdmap e910: 8 total, 8 up, 8 in 2024-08-17T17:32:48.520 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/1.ShrinkToOneAfterSnapshot (10870 ms) 2024-08-17T17:32:48.520 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/1.MinOverlap 2024-08-17T17:32:48.520 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:32:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:48 smithi165 ceph-mon[105517]: pgmap v1069: 105 pgs: 105 active+clean; 333 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.3 MiB/s rd, 9.4 MiB/s wr, 75 op/s 2024-08-17T17:32:48.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:48 smithi136 ceph-mon[109861]: pgmap v1069: 105 pgs: 105 active+clean; 333 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.3 MiB/s rd, 9.4 MiB/s wr, 75 op/s 2024-08-17T17:32:48.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:48 smithi136 ceph-mon[115078]: pgmap v1069: 105 pgs: 105 active+clean; 333 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.3 MiB/s rd, 9.4 MiB/s wr, 75 op/s 2024-08-17T17:32:50.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:50 smithi165 ceph-mon[105517]: osdmap e911: 8 total, 8 up, 8 in 2024-08-17T17:32:50.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:50 smithi165 ceph-mon[105517]: pgmap v1071: 105 pgs: 105 active+clean; 333 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.3 MiB/s rd, 8.0 MiB/s wr, 46 op/s 2024-08-17T17:32:50.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:50 smithi136 ceph-mon[109861]: osdmap e911: 8 total, 8 up, 8 in 2024-08-17T17:32:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:50 smithi136 ceph-mon[109861]: pgmap v1071: 105 pgs: 105 active+clean; 333 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.3 MiB/s rd, 8.0 MiB/s wr, 46 op/s 2024-08-17T17:32:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:50 smithi136 ceph-mon[115078]: osdmap e911: 8 total, 8 up, 8 in 2024-08-17T17:32:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:50 smithi136 ceph-mon[115078]: pgmap v1071: 105 pgs: 105 active+clean; 333 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.3 MiB/s rd, 8.0 MiB/s wr, 46 op/s 2024-08-17T17:32:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:32:51.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:32:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:32:52.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:52 smithi165 ceph-mon[105517]: pgmap v1072: 105 pgs: 105 active+clean; 379 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 14 MiB/s wr, 102 op/s 2024-08-17T17:32:52.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:52 smithi136 ceph-mon[109861]: pgmap v1072: 105 pgs: 105 active+clean; 379 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 14 MiB/s wr, 102 op/s 2024-08-17T17:32:52.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:52 smithi136 ceph-mon[115078]: pgmap v1072: 105 pgs: 105 active+clean; 379 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 14 MiB/s wr, 102 op/s 2024-08-17T17:32:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:55 smithi136 ceph-mon[109861]: pgmap v1073: 105 pgs: 105 active+clean; 379 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 13 MiB/s wr, 101 op/s 2024-08-17T17:32:55.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:55 smithi136 ceph-mon[115078]: pgmap v1073: 105 pgs: 105 active+clean; 379 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 13 MiB/s wr, 101 op/s 2024-08-17T17:32:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:55 smithi165 ceph-mon[105517]: pgmap v1073: 105 pgs: 105 active+clean; 379 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.3 MiB/s rd, 13 MiB/s wr, 101 op/s 2024-08-17T17:32:57.563 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:57 smithi165 ceph-mon[105517]: pgmap v1074: 105 pgs: 105 active+clean; 383 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 4.3 MiB/s rd, 11 MiB/s wr, 86 op/s 2024-08-17T17:32:57.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:57 smithi136 ceph-mon[109861]: pgmap v1074: 105 pgs: 105 active+clean; 383 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 4.3 MiB/s rd, 11 MiB/s wr, 86 op/s 2024-08-17T17:32:57.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:57 smithi136 ceph-mon[115078]: pgmap v1074: 105 pgs: 105 active+clean; 383 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 4.3 MiB/s rd, 11 MiB/s wr, 86 op/s 2024-08-17T17:32:57.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:32:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:32:57] "GET /metrics HTTP/1.1" 200 36489 "" "Prometheus/2.51.0" 2024-08-17T17:32:58.893 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/1.MinOverlap (10374 ms) 2024-08-17T17:32:58.894 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/1.ZeroOverlap 2024-08-17T17:32:58.894 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:32:59.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:32:59 smithi136 ceph-mon[109861]: pgmap v1075: 105 pgs: 105 active+clean; 395 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 7.4 MiB/s wr, 68 op/s 2024-08-17T17:32:59.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:32:59 smithi136 ceph-mon[115078]: pgmap v1075: 105 pgs: 105 active+clean; 395 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 7.4 MiB/s wr, 68 op/s 2024-08-17T17:32:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:32:59 smithi165 ceph-mon[105517]: pgmap v1075: 105 pgs: 105 active+clean; 395 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 7.4 MiB/s wr, 68 op/s 2024-08-17T17:33:00.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:00 smithi136 ceph-mon[109861]: osdmap e912: 8 total, 8 up, 8 in 2024-08-17T17:33:00.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:00 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:33:00.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:00 smithi136 ceph-mon[115078]: osdmap e912: 8 total, 8 up, 8 in 2024-08-17T17:33:00.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:00 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:33:00.708 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:00 smithi165 ceph-mon[105517]: osdmap e912: 8 total, 8 up, 8 in 2024-08-17T17:33:00.708 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:00 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:33:01.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:01 smithi165 ceph-mon[105517]: pgmap v1077: 105 pgs: 105 active+clean; 395 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 7.4 MiB/s wr, 68 op/s 2024-08-17T17:33:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:01 smithi136 ceph-mon[109861]: pgmap v1077: 105 pgs: 105 active+clean; 395 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 7.4 MiB/s wr, 68 op/s 2024-08-17T17:33:01.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:01 smithi136 ceph-mon[115078]: pgmap v1077: 105 pgs: 105 active+clean; 395 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 7.4 MiB/s wr, 68 op/s 2024-08-17T17:33:03.657 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:03 smithi136 ceph-mon[109861]: pgmap v1078: 105 pgs: 105 active+clean; 424 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 6.5 MiB/s wr, 62 op/s 2024-08-17T17:33:03.657 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:03 smithi136 ceph-mon[115078]: pgmap v1078: 105 pgs: 105 active+clean; 424 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 6.5 MiB/s wr, 62 op/s 2024-08-17T17:33:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:03 smithi165 ceph-mon[105517]: pgmap v1078: 105 pgs: 105 active+clean; 424 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 6.5 MiB/s wr, 62 op/s 2024-08-17T17:33:05.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:33:05.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:33:05.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:33:05.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:05 smithi136 ceph-mon[109861]: pgmap v1079: 105 pgs: 105 active+clean; 424 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 6.5 MiB/s wr, 62 op/s 2024-08-17T17:33:05.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:33:05.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:33:05.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:33:05.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:33:05.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:33:05.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:33:05.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:33:05.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:05 smithi136 ceph-mon[115078]: pgmap v1079: 105 pgs: 105 active+clean; 424 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 6.5 MiB/s wr, 62 op/s 2024-08-17T17:33:05.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:33:05.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:33:05.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:33:05.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:33:05.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:33:05.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:33:05.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:33:05.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:05 smithi165 ceph-mon[105517]: pgmap v1079: 105 pgs: 105 active+clean; 424 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 6.5 MiB/s wr, 62 op/s 2024-08-17T17:33:05.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:33:05.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:33:05.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:33:05.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:33:06.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:33:06.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:33:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:33:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:07 smithi165 ceph-mon[105517]: pgmap v1080: 105 pgs: 105 active+clean; 424 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 6.1 MiB/s wr, 58 op/s 2024-08-17T17:33:07.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:07 smithi136 ceph-mon[109861]: pgmap v1080: 105 pgs: 105 active+clean; 424 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 6.1 MiB/s wr, 58 op/s 2024-08-17T17:33:07.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:07 smithi136 ceph-mon[115078]: pgmap v1080: 105 pgs: 105 active+clean; 424 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 6.1 MiB/s wr, 58 op/s 2024-08-17T17:33:07.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:33:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:33:07] "GET /metrics HTTP/1.1" 200 36485 "" "Prometheus/2.51.0" 2024-08-17T17:33:09.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:09 smithi165 ceph-mon[105517]: pgmap v1081: 105 pgs: 105 active+clean; 428 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 38 op/s 2024-08-17T17:33:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:09 smithi136 ceph-mon[109861]: pgmap v1081: 105 pgs: 105 active+clean; 428 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 38 op/s 2024-08-17T17:33:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:09 smithi136 ceph-mon[115078]: pgmap v1081: 105 pgs: 105 active+clean; 428 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 38 op/s 2024-08-17T17:33:09.917 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/1.ZeroOverlap (11024 ms) 2024-08-17T17:33:10.269 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 7 tests from EncryptedFlattenTest/1 (78555 ms total) 2024-08-17T17:33:10.269 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:33:10.269 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 7 tests from EncryptedFlattenTest/2, where TypeParam = UnformattedLUKS1 2024-08-17T17:33:10.298 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/2.Simple 2024-08-17T17:33:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:11 smithi136 ceph-mon[109861]: pgmap v1082: 105 pgs: 105 active+clean; 428 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 35 op/s 2024-08-17T17:33:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:11 smithi136 ceph-mon[109861]: osdmap e913: 8 total, 8 up, 8 in 2024-08-17T17:33:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:11 smithi136 ceph-mon[115078]: pgmap v1082: 105 pgs: 105 active+clean; 428 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 35 op/s 2024-08-17T17:33:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:11 smithi136 ceph-mon[115078]: osdmap e913: 8 total, 8 up, 8 in 2024-08-17T17:33:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:11 smithi165 ceph-mon[105517]: pgmap v1082: 105 pgs: 105 active+clean; 428 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 35 op/s 2024-08-17T17:33:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:11 smithi165 ceph-mon[105517]: osdmap e913: 8 total, 8 up, 8 in 2024-08-17T17:33:12.288 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:33:12.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:12 smithi136 ceph-mon[109861]: osdmap e914: 8 total, 8 up, 8 in 2024-08-17T17:33:12.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:12 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3778822077' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-41","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:33:12.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:12 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-41","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:33:12.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:12 smithi136 ceph-mon[115078]: osdmap e914: 8 total, 8 up, 8 in 2024-08-17T17:33:12.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:12 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3778822077' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-41","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:33:12.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:12 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-41","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:33:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:12 smithi165 ceph-mon[105517]: osdmap e914: 8 total, 8 up, 8 in 2024-08-17T17:33:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:12 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3778822077' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-41","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:33:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:12 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-41","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:33:13.598 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:13 smithi136 ceph-mon[109861]: pgmap v1085: 105 pgs: 14 creating+peering, 18 unknown, 73 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:33:13.598 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:13 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:33:13.598 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:13 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:33:13.598 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:13 smithi136 ceph-mon[109861]: osdmap e915: 8 total, 8 up, 8 in 2024-08-17T17:33:13.598 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:13 smithi136 ceph-mon[115078]: pgmap v1085: 105 pgs: 14 creating+peering, 18 unknown, 73 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:33:13.599 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:13 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:33:13.599 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:13 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:33:13.599 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:13 smithi136 ceph-mon[115078]: osdmap e915: 8 total, 8 up, 8 in 2024-08-17T17:33:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:13 smithi165 ceph-mon[105517]: pgmap v1085: 105 pgs: 14 creating+peering, 18 unknown, 73 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:33:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:13 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:33:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:13 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:33:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:13 smithi165 ceph-mon[105517]: osdmap e915: 8 total, 8 up, 8 in 2024-08-17T17:33:14.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:14 smithi136 ceph-mon[109861]: osdmap e916: 8 total, 8 up, 8 in 2024-08-17T17:33:14.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:14 smithi136 ceph-mon[115078]: osdmap e916: 8 total, 8 up, 8 in 2024-08-17T17:33:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:14 smithi165 ceph-mon[105517]: osdmap e916: 8 total, 8 up, 8 in 2024-08-17T17:33:15.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:15 smithi136 ceph-mon[109861]: pgmap v1088: 105 pgs: 14 creating+peering, 18 unknown, 73 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:33:15.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:15 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:33:15.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:15 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T17:33:15.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:15 smithi136 ceph-mon[109861]: osdmap e917: 8 total, 8 up, 8 in 2024-08-17T17:33:15.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:15 smithi136 ceph-mon[115078]: pgmap v1088: 105 pgs: 14 creating+peering, 18 unknown, 73 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:33:15.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:15 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:33:15.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:15 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T17:33:15.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:15 smithi136 ceph-mon[115078]: osdmap e917: 8 total, 8 up, 8 in 2024-08-17T17:33:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:15 smithi165 ceph-mon[105517]: pgmap v1088: 105 pgs: 14 creating+peering, 18 unknown, 73 active+clean; 582 KiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:33:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:15 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:33:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:15 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T17:33:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:15 smithi165 ceph-mon[105517]: osdmap e917: 8 total, 8 up, 8 in 2024-08-17T17:33:17.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:17 smithi136 ceph-mon[109861]: pgmap v1090: 105 pgs: 14 creating+peering, 91 active+clean; 582 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 616 B/s rd, 616 B/s wr, 1 op/s 2024-08-17T17:33:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:17 smithi136 ceph-mon[115078]: pgmap v1090: 105 pgs: 14 creating+peering, 91 active+clean; 582 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 616 B/s rd, 616 B/s wr, 1 op/s 2024-08-17T17:33:17.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:33:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:33:17] "GET /metrics HTTP/1.1" 200 36485 "" "Prometheus/2.51.0" 2024-08-17T17:33:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:17 smithi165 ceph-mon[105517]: pgmap v1090: 105 pgs: 14 creating+peering, 91 active+clean; 582 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 616 B/s rd, 616 B/s wr, 1 op/s 2024-08-17T17:33:19.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:19 smithi136 ceph-mon[109861]: pgmap v1091: 105 pgs: 105 active+clean; 582 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-08-17T17:33:19.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:19 smithi136 ceph-mon[109861]: osdmap e918: 8 total, 8 up, 8 in 2024-08-17T17:33:19.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:19 smithi136 ceph-mon[109861]: osdmap e919: 8 total, 8 up, 8 in 2024-08-17T17:33:19.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:19 smithi136 ceph-mon[115078]: pgmap v1091: 105 pgs: 105 active+clean; 582 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-08-17T17:33:19.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:19 smithi136 ceph-mon[115078]: osdmap e918: 8 total, 8 up, 8 in 2024-08-17T17:33:19.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:19 smithi136 ceph-mon[115078]: osdmap e919: 8 total, 8 up, 8 in 2024-08-17T17:33:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:19 smithi165 ceph-mon[105517]: pgmap v1091: 105 pgs: 105 active+clean; 582 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-08-17T17:33:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:19 smithi165 ceph-mon[105517]: osdmap e918: 8 total, 8 up, 8 in 2024-08-17T17:33:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:19 smithi165 ceph-mon[105517]: osdmap e919: 8 total, 8 up, 8 in 2024-08-17T17:33:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:21 smithi136 ceph-mon[109861]: pgmap v1094: 105 pgs: 105 active+clean; 582 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-08-17T17:33:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:33:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:21 smithi136 ceph-mon[115078]: pgmap v1094: 105 pgs: 105 active+clean; 582 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-08-17T17:33:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:33:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:21 smithi165 ceph-mon[105517]: pgmap v1094: 105 pgs: 105 active+clean; 582 KiB data, 572 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-08-17T17:33:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:33:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:23 smithi136 ceph-mon[109861]: pgmap v1095: 105 pgs: 105 active+clean; 30 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 3.6 MiB/s wr, 56 op/s 2024-08-17T17:33:23.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:23 smithi136 ceph-mon[115078]: pgmap v1095: 105 pgs: 105 active+clean; 30 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 3.6 MiB/s wr, 56 op/s 2024-08-17T17:33:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:23 smithi165 ceph-mon[105517]: pgmap v1095: 105 pgs: 105 active+clean; 30 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 3.6 MiB/s wr, 56 op/s 2024-08-17T17:33:24.799 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/2.Simple (14501 ms) 2024-08-17T17:33:24.800 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/2.Grow 2024-08-17T17:33:24.800 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:33:25.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:25 smithi136 ceph-mon[109861]: pgmap v1096: 105 pgs: 105 active+clean; 30 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 3.6 MiB/s wr, 55 op/s 2024-08-17T17:33:25.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:25 smithi136 ceph-mon[115078]: pgmap v1096: 105 pgs: 105 active+clean; 30 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 3.6 MiB/s wr, 55 op/s 2024-08-17T17:33:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:25 smithi165 ceph-mon[105517]: pgmap v1096: 105 pgs: 105 active+clean; 30 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 3.6 MiB/s wr, 55 op/s 2024-08-17T17:33:27.563 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:27 smithi165 ceph-mon[105517]: pgmap v1097: 105 pgs: 105 active+clean; 42 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 2.7 MiB/s rd, 5.1 MiB/s wr, 73 op/s 2024-08-17T17:33:27.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:27 smithi136 ceph-mon[109861]: pgmap v1097: 105 pgs: 105 active+clean; 42 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 2.7 MiB/s rd, 5.1 MiB/s wr, 73 op/s 2024-08-17T17:33:27.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:27 smithi136 ceph-mon[115078]: pgmap v1097: 105 pgs: 105 active+clean; 42 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 2.7 MiB/s rd, 5.1 MiB/s wr, 73 op/s 2024-08-17T17:33:27.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:33:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:33:27] "GET /metrics HTTP/1.1" 200 36450 "" "Prometheus/2.51.0" 2024-08-17T17:33:29.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:29 smithi136 ceph-mon[109861]: pgmap v1098: 105 pgs: 105 active+clean; 59 MiB data, 747 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 63 op/s 2024-08-17T17:33:29.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:29 smithi136 ceph-mon[109861]: osdmap e920: 8 total, 8 up, 8 in 2024-08-17T17:33:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:29 smithi136 ceph-mon[115078]: pgmap v1098: 105 pgs: 105 active+clean; 59 MiB data, 747 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 63 op/s 2024-08-17T17:33:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:29 smithi136 ceph-mon[115078]: osdmap e920: 8 total, 8 up, 8 in 2024-08-17T17:33:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:29 smithi165 ceph-mon[105517]: pgmap v1098: 105 pgs: 105 active+clean; 59 MiB data, 747 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 63 op/s 2024-08-17T17:33:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:29 smithi165 ceph-mon[105517]: osdmap e920: 8 total, 8 up, 8 in 2024-08-17T17:33:31.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:31 smithi136 ceph-mon[109861]: pgmap v1100: 105 pgs: 105 active+clean; 59 MiB data, 747 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 62 op/s 2024-08-17T17:33:31.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:31 smithi136 ceph-mon[115078]: pgmap v1100: 105 pgs: 105 active+clean; 59 MiB data, 747 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 62 op/s 2024-08-17T17:33:31.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:31 smithi165 ceph-mon[105517]: pgmap v1100: 105 pgs: 105 active+clean; 59 MiB data, 747 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 62 op/s 2024-08-17T17:33:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:33 smithi136 ceph-mon[109861]: pgmap v1101: 105 pgs: 105 active+clean; 88 MiB data, 834 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 62 op/s 2024-08-17T17:33:33.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:33 smithi136 ceph-mon[115078]: pgmap v1101: 105 pgs: 105 active+clean; 88 MiB data, 834 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 62 op/s 2024-08-17T17:33:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:33 smithi165 ceph-mon[105517]: pgmap v1101: 105 pgs: 105 active+clean; 88 MiB data, 834 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 62 op/s 2024-08-17T17:33:34.767 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/2.Grow (9968 ms) 2024-08-17T17:33:34.767 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/2.Shrink 2024-08-17T17:33:34.767 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:33:35.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:35 smithi136 ceph-mon[109861]: pgmap v1102: 105 pgs: 105 active+clean; 88 MiB data, 834 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 62 op/s 2024-08-17T17:33:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:35 smithi136 ceph-mon[115078]: pgmap v1102: 105 pgs: 105 active+clean; 88 MiB data, 834 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 62 op/s 2024-08-17T17:33:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:35 smithi165 ceph-mon[105517]: pgmap v1102: 105 pgs: 105 active+clean; 88 MiB data, 834 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 62 op/s 2024-08-17T17:33:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:33:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:33:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:33:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:37 smithi136 ceph-mon[109861]: pgmap v1103: 105 pgs: 105 active+clean; 100 MiB data, 891 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 5.8 MiB/s wr, 63 op/s 2024-08-17T17:33:37.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:37 smithi136 ceph-mon[115078]: pgmap v1103: 105 pgs: 105 active+clean; 100 MiB data, 891 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 5.8 MiB/s wr, 63 op/s 2024-08-17T17:33:37.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:33:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:33:37] "GET /metrics HTTP/1.1" 200 36472 "" "Prometheus/2.51.0" 2024-08-17T17:33:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:37 smithi165 ceph-mon[105517]: pgmap v1103: 105 pgs: 105 active+clean; 100 MiB data, 891 MiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 5.8 MiB/s wr, 63 op/s 2024-08-17T17:33:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:39 smithi136 ceph-mon[109861]: pgmap v1104: 105 pgs: 105 active+clean; 117 MiB data, 921 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-08-17T17:33:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:39 smithi136 ceph-mon[109861]: osdmap e921: 8 total, 8 up, 8 in 2024-08-17T17:33:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:39 smithi136 ceph-mon[115078]: pgmap v1104: 105 pgs: 105 active+clean; 117 MiB data, 921 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-08-17T17:33:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:39 smithi136 ceph-mon[115078]: osdmap e921: 8 total, 8 up, 8 in 2024-08-17T17:33:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:39 smithi165 ceph-mon[105517]: pgmap v1104: 105 pgs: 105 active+clean; 117 MiB data, 921 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-08-17T17:33:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:39 smithi165 ceph-mon[105517]: osdmap e921: 8 total, 8 up, 8 in 2024-08-17T17:33:41.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:41 smithi136 ceph-mon[109861]: pgmap v1106: 105 pgs: 105 active+clean; 117 MiB data, 921 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-08-17T17:33:41.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:41 smithi136 ceph-mon[115078]: pgmap v1106: 105 pgs: 105 active+clean; 117 MiB data, 921 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-08-17T17:33:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:41 smithi165 ceph-mon[105517]: pgmap v1106: 105 pgs: 105 active+clean; 117 MiB data, 921 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-08-17T17:33:43.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:43 smithi136 ceph-mon[109861]: pgmap v1107: 105 pgs: 105 active+clean; 146 MiB data, 1008 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-08-17T17:33:43.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:43 smithi136 ceph-mon[115078]: pgmap v1107: 105 pgs: 105 active+clean; 146 MiB data, 1008 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-08-17T17:33:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:43 smithi165 ceph-mon[105517]: pgmap v1107: 105 pgs: 105 active+clean; 146 MiB data, 1008 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-08-17T17:33:44.832 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/2.Shrink (10065 ms) 2024-08-17T17:33:44.832 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/2.ShrinkToOne 2024-08-17T17:33:44.832 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:33:45.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:45 smithi136 ceph-mon[109861]: pgmap v1108: 105 pgs: 105 active+clean; 146 MiB data, 1008 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-08-17T17:33:45.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:45 smithi136 ceph-mon[115078]: pgmap v1108: 105 pgs: 105 active+clean; 146 MiB data, 1008 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-08-17T17:33:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:45 smithi165 ceph-mon[105517]: pgmap v1108: 105 pgs: 105 active+clean; 146 MiB data, 1008 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-08-17T17:33:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:47 smithi136 ceph-mon[109861]: pgmap v1109: 105 pgs: 105 active+clean; 154 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 8.1 MiB/s rd, 5.4 MiB/s wr, 59 op/s 2024-08-17T17:33:47.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:47 smithi136 ceph-mon[115078]: pgmap v1109: 105 pgs: 105 active+clean; 154 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 8.1 MiB/s rd, 5.4 MiB/s wr, 59 op/s 2024-08-17T17:33:47.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:33:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:33:47] "GET /metrics HTTP/1.1" 200 36472 "" "Prometheus/2.51.0" 2024-08-17T17:33:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:47 smithi165 ceph-mon[105517]: pgmap v1109: 105 pgs: 105 active+clean; 154 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 8.1 MiB/s rd, 5.4 MiB/s wr, 59 op/s 2024-08-17T17:33:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:49 smithi136 ceph-mon[109861]: pgmap v1110: 105 pgs: 105 active+clean; 175 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-08-17T17:33:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:49 smithi136 ceph-mon[109861]: osdmap e922: 8 total, 8 up, 8 in 2024-08-17T17:33:49.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:49 smithi136 ceph-mon[115078]: pgmap v1110: 105 pgs: 105 active+clean; 175 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-08-17T17:33:49.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:49 smithi136 ceph-mon[115078]: osdmap e922: 8 total, 8 up, 8 in 2024-08-17T17:33:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:49 smithi165 ceph-mon[105517]: pgmap v1110: 105 pgs: 105 active+clean; 175 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-08-17T17:33:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:49 smithi165 ceph-mon[105517]: osdmap e922: 8 total, 8 up, 8 in 2024-08-17T17:33:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:51 smithi136 ceph-mon[109861]: pgmap v1112: 105 pgs: 105 active+clean; 175 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-08-17T17:33:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:33:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:51 smithi136 ceph-mon[115078]: pgmap v1112: 105 pgs: 105 active+clean; 175 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-08-17T17:33:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:33:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:51 smithi165 ceph-mon[105517]: pgmap v1112: 105 pgs: 105 active+clean; 175 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-08-17T17:33:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:33:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:53 smithi136 ceph-mon[109861]: pgmap v1113: 105 pgs: 105 active+clean; 204 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-08-17T17:33:53.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:53 smithi136 ceph-mon[115078]: pgmap v1113: 105 pgs: 105 active+clean; 204 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-08-17T17:33:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:53 smithi165 ceph-mon[105517]: pgmap v1113: 105 pgs: 105 active+clean; 204 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-08-17T17:33:54.733 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/2.ShrinkToOne (9901 ms) 2024-08-17T17:33:54.733 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/2.ShrinkToOneAfterSnapshot 2024-08-17T17:33:54.734 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:33:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:55 smithi136 ceph-mon[109861]: pgmap v1114: 105 pgs: 105 active+clean; 204 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-08-17T17:33:55.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:55 smithi136 ceph-mon[115078]: pgmap v1114: 105 pgs: 105 active+clean; 204 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-08-17T17:33:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:55 smithi165 ceph-mon[105517]: pgmap v1114: 105 pgs: 105 active+clean; 204 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.8 MiB/s wr, 65 op/s 2024-08-17T17:33:57.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:57 smithi136 ceph-mon[109861]: pgmap v1115: 105 pgs: 105 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.4 MiB/s rd, 5.4 MiB/s wr, 59 op/s 2024-08-17T17:33:57.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:57 smithi136 ceph-mon[115078]: pgmap v1115: 105 pgs: 105 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.4 MiB/s rd, 5.4 MiB/s wr, 59 op/s 2024-08-17T17:33:57.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:33:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:33:57] "GET /metrics HTTP/1.1" 200 36483 "" "Prometheus/2.51.0" 2024-08-17T17:33:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:57 smithi165 ceph-mon[105517]: pgmap v1115: 105 pgs: 105 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.4 MiB/s rd, 5.4 MiB/s wr, 59 op/s 2024-08-17T17:33:59.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:59 smithi136 ceph-mon[109861]: pgmap v1116: 105 pgs: 105 active+clean; 212 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 946 KiB/s rd, 4.1 MiB/s wr, 56 op/s 2024-08-17T17:33:59.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:33:59 smithi136 ceph-mon[109861]: osdmap e923: 8 total, 8 up, 8 in 2024-08-17T17:33:59.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:59 smithi136 ceph-mon[115078]: pgmap v1116: 105 pgs: 105 active+clean; 212 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 946 KiB/s rd, 4.1 MiB/s wr, 56 op/s 2024-08-17T17:33:59.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:33:59 smithi136 ceph-mon[115078]: osdmap e923: 8 total, 8 up, 8 in 2024-08-17T17:33:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:59 smithi165 ceph-mon[105517]: pgmap v1116: 105 pgs: 105 active+clean; 212 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 946 KiB/s rd, 4.1 MiB/s wr, 56 op/s 2024-08-17T17:33:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:33:59 smithi165 ceph-mon[105517]: osdmap e923: 8 total, 8 up, 8 in 2024-08-17T17:34:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:01 smithi136 ceph-mon[109861]: pgmap v1118: 105 pgs: 105 active+clean; 212 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 946 KiB/s rd, 4.1 MiB/s wr, 56 op/s 2024-08-17T17:34:01.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:01 smithi136 ceph-mon[115078]: pgmap v1118: 105 pgs: 105 active+clean; 212 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 946 KiB/s rd, 4.1 MiB/s wr, 56 op/s 2024-08-17T17:34:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:01 smithi165 ceph-mon[105517]: pgmap v1118: 105 pgs: 105 active+clean; 212 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 946 KiB/s rd, 4.1 MiB/s wr, 56 op/s 2024-08-17T17:34:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:03 smithi136 ceph-mon[109861]: pgmap v1119: 105 pgs: 105 active+clean; 241 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 946 KiB/s rd, 4.1 MiB/s wr, 56 op/s 2024-08-17T17:34:03.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:03 smithi136 ceph-mon[115078]: pgmap v1119: 105 pgs: 105 active+clean; 241 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 946 KiB/s rd, 4.1 MiB/s wr, 56 op/s 2024-08-17T17:34:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:03 smithi165 ceph-mon[105517]: pgmap v1119: 105 pgs: 105 active+clean; 241 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 946 KiB/s rd, 4.1 MiB/s wr, 56 op/s 2024-08-17T17:34:04.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:04 smithi136 ceph-mon[109861]: osdmap e924: 8 total, 8 up, 8 in 2024-08-17T17:34:04.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:04 smithi136 ceph-mon[115078]: osdmap e924: 8 total, 8 up, 8 in 2024-08-17T17:34:04.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:04 smithi165 ceph-mon[105517]: osdmap e924: 8 total, 8 up, 8 in 2024-08-17T17:34:05.687 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:05 smithi136 ceph-mon[109861]: pgmap v1121: 105 pgs: 105 active+clean; 241 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 117 KiB/s rd, 4.6 MiB/s wr, 60 op/s 2024-08-17T17:34:05.687 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:34:05.687 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:05 smithi136 ceph-mon[115078]: pgmap v1121: 105 pgs: 105 active+clean; 241 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 117 KiB/s rd, 4.6 MiB/s wr, 60 op/s 2024-08-17T17:34:05.688 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:34:05.730 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:05 smithi165 ceph-mon[105517]: pgmap v1121: 105 pgs: 105 active+clean; 241 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 117 KiB/s rd, 4.6 MiB/s wr, 60 op/s 2024-08-17T17:34:05.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:34:05.757 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/2.ShrinkToOneAfterSnapshot (11024 ms) 2024-08-17T17:34:05.758 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/2.MinOverlap 2024-08-17T17:34:05.758 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:34:06.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:34:06.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:34:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:34:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:07 smithi165 ceph-mon[105517]: pgmap v1122: 105 pgs: 105 active+clean; 270 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 7.3 MiB/s wr, 78 op/s 2024-08-17T17:34:07.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:07 smithi136 ceph-mon[109861]: pgmap v1122: 105 pgs: 105 active+clean; 270 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 7.3 MiB/s wr, 78 op/s 2024-08-17T17:34:07.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:07 smithi136 ceph-mon[115078]: pgmap v1122: 105 pgs: 105 active+clean; 270 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 7.3 MiB/s wr, 78 op/s 2024-08-17T17:34:07.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:34:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:34:07] "GET /metrics HTTP/1.1" 200 36486 "" "Prometheus/2.51.0" 2024-08-17T17:34:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:09 smithi136 ceph-mon[109861]: pgmap v1123: 105 pgs: 105 active+clean; 302 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 9.1 MiB/s wr, 75 op/s 2024-08-17T17:34:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:09 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:34:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:09 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:34:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:09 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:34:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:09 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:34:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:09 smithi136 ceph-mon[115078]: pgmap v1123: 105 pgs: 105 active+clean; 302 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 9.1 MiB/s wr, 75 op/s 2024-08-17T17:34:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:09 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:34:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:09 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:34:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:09 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:34:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:09 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:34:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:09 smithi165 ceph-mon[105517]: pgmap v1123: 105 pgs: 105 active+clean; 302 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 9.1 MiB/s wr, 75 op/s 2024-08-17T17:34:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:09 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:34:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:09 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:34:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:09 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:34:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:09 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:34:10.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:34:10.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:34:10.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:34:10.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:10 smithi136 ceph-mon[109861]: osdmap e925: 8 total, 8 up, 8 in 2024-08-17T17:34:10.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:34:10.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:34:10.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:34:10.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:10 smithi136 ceph-mon[115078]: osdmap e925: 8 total, 8 up, 8 in 2024-08-17T17:34:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:34:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:34:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:34:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:10 smithi165 ceph-mon[105517]: osdmap e925: 8 total, 8 up, 8 in 2024-08-17T17:34:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:11 smithi136 ceph-mon[109861]: pgmap v1124: 105 pgs: 105 active+clean; 302 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 9.0 MiB/s wr, 74 op/s 2024-08-17T17:34:11.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:11 smithi136 ceph-mon[115078]: pgmap v1124: 105 pgs: 105 active+clean; 302 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 9.0 MiB/s wr, 74 op/s 2024-08-17T17:34:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:11 smithi165 ceph-mon[105517]: pgmap v1124: 105 pgs: 105 active+clean; 302 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 9.0 MiB/s wr, 74 op/s 2024-08-17T17:34:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:12 smithi165 ceph-mon[105517]: pgmap v1126: 105 pgs: 105 active+clean; 331 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 10 MiB/s wr, 84 op/s 2024-08-17T17:34:12.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:12 smithi136 ceph-mon[109861]: pgmap v1126: 105 pgs: 105 active+clean; 331 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 10 MiB/s wr, 84 op/s 2024-08-17T17:34:12.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:12 smithi136 ceph-mon[115078]: pgmap v1126: 105 pgs: 105 active+clean; 331 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.8 MiB/s rd, 10 MiB/s wr, 84 op/s 2024-08-17T17:34:15.649 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:15 smithi136 ceph-mon[109861]: pgmap v1127: 105 pgs: 105 active+clean; 331 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 9.0 MiB/s wr, 75 op/s 2024-08-17T17:34:15.649 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:15 smithi136 ceph-mon[115078]: pgmap v1127: 105 pgs: 105 active+clean; 331 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 9.0 MiB/s wr, 75 op/s 2024-08-17T17:34:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:15 smithi165 ceph-mon[105517]: pgmap v1127: 105 pgs: 105 active+clean; 331 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 9.0 MiB/s wr, 75 op/s 2024-08-17T17:34:16.412 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/2.MinOverlap (10655 ms) 2024-08-17T17:34:16.413 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/2.ZeroOverlap 2024-08-17T17:34:16.413 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:34:17.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:17 smithi136 ceph-mon[109861]: pgmap v1128: 105 pgs: 105 active+clean; 339 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 7.3 MiB/s wr, 66 op/s 2024-08-17T17:34:17.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:17 smithi136 ceph-mon[115078]: pgmap v1128: 105 pgs: 105 active+clean; 339 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 7.3 MiB/s wr, 66 op/s 2024-08-17T17:34:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:17 smithi165 ceph-mon[105517]: pgmap v1128: 105 pgs: 105 active+clean; 339 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 7.3 MiB/s wr, 66 op/s 2024-08-17T17:34:17.871 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:34:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:34:17] "GET /metrics HTTP/1.1" 200 36486 "" "Prometheus/2.51.0" 2024-08-17T17:34:19.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:19 smithi136 ceph-mon[109861]: pgmap v1129: 105 pgs: 105 active+clean; 339 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 53 op/s 2024-08-17T17:34:19.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:19 smithi136 ceph-mon[115078]: pgmap v1129: 105 pgs: 105 active+clean; 339 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 53 op/s 2024-08-17T17:34:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:19 smithi165 ceph-mon[105517]: pgmap v1129: 105 pgs: 105 active+clean; 339 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 53 op/s 2024-08-17T17:34:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:21 smithi136 ceph-mon[109861]: pgmap v1130: 105 pgs: 105 active+clean; 339 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 53 op/s 2024-08-17T17:34:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:21 smithi136 ceph-mon[109861]: osdmap e926: 8 total, 8 up, 8 in 2024-08-17T17:34:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:34:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:21 smithi136 ceph-mon[115078]: pgmap v1130: 105 pgs: 105 active+clean; 339 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 53 op/s 2024-08-17T17:34:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:21 smithi136 ceph-mon[115078]: osdmap e926: 8 total, 8 up, 8 in 2024-08-17T17:34:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:34:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:21 smithi165 ceph-mon[105517]: pgmap v1130: 105 pgs: 105 active+clean; 339 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 53 op/s 2024-08-17T17:34:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:21 smithi165 ceph-mon[105517]: osdmap e926: 8 total, 8 up, 8 in 2024-08-17T17:34:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:34:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:23 smithi136 ceph-mon[109861]: pgmap v1132: 105 pgs: 105 active+clean; 368 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 58 op/s 2024-08-17T17:34:23.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:23 smithi136 ceph-mon[115078]: pgmap v1132: 105 pgs: 105 active+clean; 368 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 58 op/s 2024-08-17T17:34:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:23 smithi165 ceph-mon[105517]: pgmap v1132: 105 pgs: 105 active+clean; 368 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 58 op/s 2024-08-17T17:34:25.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:25 smithi136 ceph-mon[109861]: pgmap v1133: 105 pgs: 105 active+clean; 368 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 58 op/s 2024-08-17T17:34:25.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:25 smithi136 ceph-mon[115078]: pgmap v1133: 105 pgs: 105 active+clean; 368 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 58 op/s 2024-08-17T17:34:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:25 smithi165 ceph-mon[105517]: pgmap v1133: 105 pgs: 105 active+clean; 368 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 58 op/s 2024-08-17T17:34:26.439 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/2.ZeroOverlap (10027 ms) 2024-08-17T17:34:27.363 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 7 tests from EncryptedFlattenTest/2 (76141 ms total) 2024-08-17T17:34:27.363 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:34:27.363 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 7 tests from EncryptedFlattenTest/3, where TypeParam = LUKS1UnderLUKS1 2024-08-17T17:34:27.391 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/3.Simple 2024-08-17T17:34:27.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:27 smithi136 ceph-mon[109861]: pgmap v1134: 105 pgs: 105 active+clean; 368 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 472 KiB/s rd, 2.9 MiB/s wr, 45 op/s 2024-08-17T17:34:27.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:27 smithi136 ceph-mon[115078]: pgmap v1134: 105 pgs: 105 active+clean; 368 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 472 KiB/s rd, 2.9 MiB/s wr, 45 op/s 2024-08-17T17:34:27.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:34:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:34:27] "GET /metrics HTTP/1.1" 200 36478 "" "Prometheus/2.51.0" 2024-08-17T17:34:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:27 smithi165 ceph-mon[105517]: pgmap v1134: 105 pgs: 105 active+clean; 368 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 472 KiB/s rd, 2.9 MiB/s wr, 45 op/s 2024-08-17T17:34:28.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:28 smithi136 ceph-mon[109861]: osdmap e927: 8 total, 8 up, 8 in 2024-08-17T17:34:28.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:28 smithi136 ceph-mon[115078]: osdmap e927: 8 total, 8 up, 8 in 2024-08-17T17:34:28.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:28 smithi165 ceph-mon[105517]: osdmap e927: 8 total, 8 up, 8 in 2024-08-17T17:34:29.382 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:34:29.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:29 smithi136 ceph-mon[109861]: pgmap v1136: 73 pgs: 73 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:34:29.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:29 smithi136 ceph-mon[109861]: osdmap e928: 8 total, 8 up, 8 in 2024-08-17T17:34:29.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:29 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3458930408' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-42","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:34:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:29 smithi136 ceph-mon[115078]: pgmap v1136: 73 pgs: 73 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:34:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:29 smithi136 ceph-mon[115078]: osdmap e928: 8 total, 8 up, 8 in 2024-08-17T17:34:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:29 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3458930408' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-42","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:34:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:29 smithi165 ceph-mon[105517]: pgmap v1136: 73 pgs: 73 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:34:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:29 smithi165 ceph-mon[105517]: osdmap e928: 8 total, 8 up, 8 in 2024-08-17T17:34:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:29 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3458930408' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-42","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:34:30.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:30 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3458930408' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:34:30.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:30 smithi136 ceph-mon[109861]: osdmap e929: 8 total, 8 up, 8 in 2024-08-17T17:34:30.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:30 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3458930408' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:34:30.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:30 smithi136 ceph-mon[115078]: osdmap e929: 8 total, 8 up, 8 in 2024-08-17T17:34:30.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:30 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3458930408' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:34:30.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:30 smithi165 ceph-mon[105517]: osdmap e929: 8 total, 8 up, 8 in 2024-08-17T17:34:31.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:31 smithi136 ceph-mon[109861]: pgmap v1139: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:34:31.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:31 smithi136 ceph-mon[109861]: osdmap e930: 8 total, 8 up, 8 in 2024-08-17T17:34:31.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:31 smithi136 ceph-mon[115078]: pgmap v1139: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:34:31.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:31 smithi136 ceph-mon[115078]: osdmap e930: 8 total, 8 up, 8 in 2024-08-17T17:34:31.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:31 smithi165 ceph-mon[105517]: pgmap v1139: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:34:31.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:31 smithi165 ceph-mon[105517]: osdmap e930: 8 total, 8 up, 8 in 2024-08-17T17:34:32.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:32 smithi136 ceph-mon[109861]: osdmap e931: 8 total, 8 up, 8 in 2024-08-17T17:34:32.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:32 smithi136 ceph-mon[115078]: osdmap e931: 8 total, 8 up, 8 in 2024-08-17T17:34:32.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:32 smithi165 ceph-mon[105517]: osdmap e931: 8 total, 8 up, 8 in 2024-08-17T17:34:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:33 smithi136 ceph-mon[109861]: pgmap v1142: 105 pgs: 105 active+clean; 582 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-08-17T17:34:33.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:33 smithi136 ceph-mon[115078]: pgmap v1142: 105 pgs: 105 active+clean; 582 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-08-17T17:34:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:33 smithi165 ceph-mon[105517]: pgmap v1142: 105 pgs: 105 active+clean; 582 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-08-17T17:34:35.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:35 smithi136 ceph-mon[109861]: pgmap v1143: 105 pgs: 105 active+clean; 582 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 4.4 KiB/s rd, 865 B/s wr, 6 op/s 2024-08-17T17:34:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:35 smithi136 ceph-mon[115078]: pgmap v1143: 105 pgs: 105 active+clean; 582 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 4.4 KiB/s rd, 865 B/s wr, 6 op/s 2024-08-17T17:34:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:35 smithi165 ceph-mon[105517]: pgmap v1143: 105 pgs: 105 active+clean; 582 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 4.4 KiB/s rd, 865 B/s wr, 6 op/s 2024-08-17T17:34:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:36 smithi136 ceph-mon[109861]: osdmap e932: 8 total, 8 up, 8 in 2024-08-17T17:34:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:34:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:36 smithi136 ceph-mon[115078]: osdmap e932: 8 total, 8 up, 8 in 2024-08-17T17:34:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:34:36.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:36 smithi165 ceph-mon[105517]: osdmap e932: 8 total, 8 up, 8 in 2024-08-17T17:34:36.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:34:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:37 smithi136 ceph-mon[109861]: pgmap v1145: 105 pgs: 105 active+clean; 26 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 4.2 MiB/s wr, 24 op/s 2024-08-17T17:34:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:37 smithi136 ceph-mon[115078]: pgmap v1145: 105 pgs: 105 active+clean; 26 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 4.2 MiB/s wr, 24 op/s 2024-08-17T17:34:37.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:34:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:34:37] "GET /metrics HTTP/1.1" 200 36446 "" "Prometheus/2.51.0" 2024-08-17T17:34:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:37 smithi165 ceph-mon[105517]: pgmap v1145: 105 pgs: 105 active+clean; 26 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 4.2 MiB/s wr, 24 op/s 2024-08-17T17:34:38.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:38 smithi136 ceph-mon[109861]: pgmap v1146: 105 pgs: 105 active+clean; 30 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 3.7 MiB/s wr, 48 op/s 2024-08-17T17:34:38.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:38 smithi136 ceph-mon[115078]: pgmap v1146: 105 pgs: 105 active+clean; 30 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 3.7 MiB/s wr, 48 op/s 2024-08-17T17:34:38.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:38 smithi165 ceph-mon[105517]: pgmap v1146: 105 pgs: 105 active+clean; 30 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 3.7 MiB/s wr, 48 op/s 2024-08-17T17:34:40.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:40 smithi136 ceph-mon[109861]: osdmap e933: 8 total, 8 up, 8 in 2024-08-17T17:34:40.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:40 smithi136 ceph-mon[115078]: osdmap e933: 8 total, 8 up, 8 in 2024-08-17T17:34:40.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:40 smithi165 ceph-mon[105517]: osdmap e933: 8 total, 8 up, 8 in 2024-08-17T17:34:41.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:41 smithi136 ceph-mon[109861]: pgmap v1148: 105 pgs: 105 active+clean; 30 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 3.6 MiB/s wr, 42 op/s 2024-08-17T17:34:41.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:41 smithi136 ceph-mon[115078]: pgmap v1148: 105 pgs: 105 active+clean; 30 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 3.6 MiB/s wr, 42 op/s 2024-08-17T17:34:41.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:41 smithi165 ceph-mon[105517]: pgmap v1148: 105 pgs: 105 active+clean; 30 MiB data, 660 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 3.6 MiB/s wr, 42 op/s 2024-08-17T17:34:43.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:43 smithi136 ceph-mon[109861]: pgmap v1149: 105 pgs: 105 active+clean; 34 MiB data, 681 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 4.1 MiB/s wr, 53 op/s 2024-08-17T17:34:43.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:43 smithi136 ceph-mon[115078]: pgmap v1149: 105 pgs: 105 active+clean; 34 MiB data, 681 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 4.1 MiB/s wr, 53 op/s 2024-08-17T17:34:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:43 smithi165 ceph-mon[105517]: pgmap v1149: 105 pgs: 105 active+clean; 34 MiB data, 681 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 4.1 MiB/s wr, 53 op/s 2024-08-17T17:34:45.258 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/3.Simple (17866 ms) 2024-08-17T17:34:45.258 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/3.Grow 2024-08-17T17:34:45.258 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:34:45.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:45 smithi136 ceph-mon[109861]: pgmap v1150: 105 pgs: 105 active+clean; 34 MiB data, 681 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 3.7 MiB/s wr, 48 op/s 2024-08-17T17:34:45.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:45 smithi136 ceph-mon[115078]: pgmap v1150: 105 pgs: 105 active+clean; 34 MiB data, 681 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 3.7 MiB/s wr, 48 op/s 2024-08-17T17:34:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:45 smithi165 ceph-mon[105517]: pgmap v1150: 105 pgs: 105 active+clean; 34 MiB data, 681 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 3.7 MiB/s wr, 48 op/s 2024-08-17T17:34:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:47 smithi136 ceph-mon[109861]: pgmap v1151: 105 pgs: 105 active+clean; 43 MiB data, 719 MiB used, 715 GiB / 715 GiB avail; 6.1 MiB/s rd, 1.7 MiB/s wr, 39 op/s 2024-08-17T17:34:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:47 smithi136 ceph-mon[115078]: pgmap v1151: 105 pgs: 105 active+clean; 43 MiB data, 719 MiB used, 715 GiB / 715 GiB avail; 6.1 MiB/s rd, 1.7 MiB/s wr, 39 op/s 2024-08-17T17:34:47.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:34:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:34:47] "GET /metrics HTTP/1.1" 200 36446 "" "Prometheus/2.51.0" 2024-08-17T17:34:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:47 smithi165 ceph-mon[105517]: pgmap v1151: 105 pgs: 105 active+clean; 43 MiB data, 719 MiB used, 715 GiB / 715 GiB avail; 6.1 MiB/s rd, 1.7 MiB/s wr, 39 op/s 2024-08-17T17:34:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:49 smithi136 ceph-mon[109861]: pgmap v1152: 105 pgs: 105 active+clean; 59 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 3.3 MiB/s wr, 31 op/s 2024-08-17T17:34:49.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:49 smithi136 ceph-mon[115078]: pgmap v1152: 105 pgs: 105 active+clean; 59 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 3.3 MiB/s wr, 31 op/s 2024-08-17T17:34:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:49 smithi165 ceph-mon[105517]: pgmap v1152: 105 pgs: 105 active+clean; 59 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 3.3 MiB/s wr, 31 op/s 2024-08-17T17:34:50.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:50 smithi136 ceph-mon[109861]: osdmap e934: 8 total, 8 up, 8 in 2024-08-17T17:34:50.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:50 smithi136 ceph-mon[115078]: osdmap e934: 8 total, 8 up, 8 in 2024-08-17T17:34:50.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:50 smithi165 ceph-mon[105517]: osdmap e934: 8 total, 8 up, 8 in 2024-08-17T17:34:51.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:51 smithi136 ceph-mon[115078]: pgmap v1154: 105 pgs: 105 active+clean; 59 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 3.3 MiB/s wr, 31 op/s 2024-08-17T17:34:51.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:34:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:51 smithi136 ceph-mon[109861]: pgmap v1154: 105 pgs: 105 active+clean; 59 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 3.3 MiB/s wr, 31 op/s 2024-08-17T17:34:51.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:34:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:51 smithi165 ceph-mon[105517]: pgmap v1154: 105 pgs: 105 active+clean; 59 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 3.3 MiB/s wr, 31 op/s 2024-08-17T17:34:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:34:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:53 smithi136 ceph-mon[109861]: pgmap v1155: 105 pgs: 105 active+clean; 88 MiB data, 844 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 55 op/s 2024-08-17T17:34:53.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:53 smithi136 ceph-mon[115078]: pgmap v1155: 105 pgs: 105 active+clean; 88 MiB data, 844 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 55 op/s 2024-08-17T17:34:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:53 smithi165 ceph-mon[105517]: pgmap v1155: 105 pgs: 105 active+clean; 88 MiB data, 844 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 55 op/s 2024-08-17T17:34:55.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:55 smithi136 ceph-mon[115078]: pgmap v1156: 105 pgs: 105 active+clean; 88 MiB data, 844 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 55 op/s 2024-08-17T17:34:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:55 smithi136 ceph-mon[109861]: pgmap v1156: 105 pgs: 105 active+clean; 88 MiB data, 844 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 55 op/s 2024-08-17T17:34:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:55 smithi165 ceph-mon[105517]: pgmap v1156: 105 pgs: 105 active+clean; 88 MiB data, 844 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 55 op/s 2024-08-17T17:34:57.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:57 smithi136 ceph-mon[109861]: pgmap v1157: 105 pgs: 105 active+clean; 92 MiB data, 848 MiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 5.3 MiB/s wr, 53 op/s 2024-08-17T17:34:57.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:57 smithi136 ceph-mon[115078]: pgmap v1157: 105 pgs: 105 active+clean; 92 MiB data, 848 MiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 5.3 MiB/s wr, 53 op/s 2024-08-17T17:34:57.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:34:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:34:57] "GET /metrics HTTP/1.1" 200 36474 "" "Prometheus/2.51.0" 2024-08-17T17:34:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:57 smithi165 ceph-mon[105517]: pgmap v1157: 105 pgs: 105 active+clean; 92 MiB data, 848 MiB used, 714 GiB / 715 GiB avail; 4.5 MiB/s rd, 5.3 MiB/s wr, 53 op/s 2024-08-17T17:34:59.201 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/3.Grow (13944 ms) 2024-08-17T17:34:59.201 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/3.Shrink 2024-08-17T17:34:59.201 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:34:59.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:34:59 smithi136 ceph-mon[109861]: pgmap v1158: 105 pgs: 105 active+clean; 96 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 44 op/s 2024-08-17T17:34:59.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:34:59 smithi136 ceph-mon[115078]: pgmap v1158: 105 pgs: 105 active+clean; 96 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 44 op/s 2024-08-17T17:34:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:34:59 smithi165 ceph-mon[105517]: pgmap v1158: 105 pgs: 105 active+clean; 96 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 44 op/s 2024-08-17T17:35:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:01 smithi136 ceph-mon[109861]: pgmap v1159: 105 pgs: 105 active+clean; 96 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 40 op/s 2024-08-17T17:35:01.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:01 smithi136 ceph-mon[115078]: pgmap v1159: 105 pgs: 105 active+clean; 96 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 40 op/s 2024-08-17T17:35:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:01 smithi165 ceph-mon[105517]: pgmap v1159: 105 pgs: 105 active+clean; 96 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 40 op/s 2024-08-17T17:35:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:03 smithi136 ceph-mon[109861]: pgmap v1160: 105 pgs: 105 active+clean; 117 MiB data, 931 MiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 5.2 MiB/s wr, 55 op/s 2024-08-17T17:35:03.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:03 smithi136 ceph-mon[115078]: pgmap v1160: 105 pgs: 105 active+clean; 117 MiB data, 931 MiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 5.2 MiB/s wr, 55 op/s 2024-08-17T17:35:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:03 smithi165 ceph-mon[105517]: pgmap v1160: 105 pgs: 105 active+clean; 117 MiB data, 931 MiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 5.2 MiB/s wr, 55 op/s 2024-08-17T17:35:04.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:04 smithi136 ceph-mon[115078]: osdmap e935: 8 total, 8 up, 8 in 2024-08-17T17:35:04.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:04 smithi136 ceph-mon[109861]: osdmap e935: 8 total, 8 up, 8 in 2024-08-17T17:35:04.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:04 smithi165 ceph-mon[105517]: osdmap e935: 8 total, 8 up, 8 in 2024-08-17T17:35:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:05 smithi136 ceph-mon[109861]: pgmap v1162: 105 pgs: 105 active+clean; 117 MiB data, 931 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.3 MiB/s wr, 33 op/s 2024-08-17T17:35:05.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:05 smithi136 ceph-mon[115078]: pgmap v1162: 105 pgs: 105 active+clean; 117 MiB data, 931 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.3 MiB/s wr, 33 op/s 2024-08-17T17:35:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:05 smithi165 ceph-mon[105517]: pgmap v1162: 105 pgs: 105 active+clean; 117 MiB data, 931 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.3 MiB/s wr, 33 op/s 2024-08-17T17:35:06.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:35:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:35:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:35:07.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:07 smithi136 ceph-mon[109861]: pgmap v1163: 105 pgs: 105 active+clean; 133 MiB data, 984 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 4.5 MiB/s wr, 36 op/s 2024-08-17T17:35:07.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:07 smithi136 ceph-mon[115078]: pgmap v1163: 105 pgs: 105 active+clean; 133 MiB data, 984 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 4.5 MiB/s wr, 36 op/s 2024-08-17T17:35:07.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:35:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:35:07] "GET /metrics HTTP/1.1" 200 36475 "" "Prometheus/2.51.0" 2024-08-17T17:35:07.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:07 smithi165 ceph-mon[105517]: pgmap v1163: 105 pgs: 105 active+clean; 133 MiB data, 984 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 4.5 MiB/s wr, 36 op/s 2024-08-17T17:35:08.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.2", "id": [0, 4]} : dispatch 2024-08-17T17:35:08.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.f", "id": [0, 4]} : dispatch 2024-08-17T17:35:08.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.2", "id": [0, 4]} : dispatch 2024-08-17T17:35:08.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.f", "id": [0, 4]} : dispatch 2024-08-17T17:35:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.2", "id": [0, 4]} : dispatch 2024-08-17T17:35:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.f", "id": [0, 4]} : dispatch 2024-08-17T17:35:09.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:09 smithi136 ceph-mon[115078]: pgmap v1164: 105 pgs: 105 active+clean; 146 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 5.4 MiB/s wr, 57 op/s 2024-08-17T17:35:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:09 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.2", "id": [0, 4]}]': finished 2024-08-17T17:35:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:09 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.f", "id": [0, 4]}]': finished 2024-08-17T17:35:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:09 smithi136 ceph-mon[115078]: osdmap e936: 8 total, 8 up, 8 in 2024-08-17T17:35:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:09 smithi136 ceph-mon[109861]: pgmap v1164: 105 pgs: 105 active+clean; 146 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 5.4 MiB/s wr, 57 op/s 2024-08-17T17:35:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:09 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.2", "id": [0, 4]}]': finished 2024-08-17T17:35:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:09 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.f", "id": [0, 4]}]': finished 2024-08-17T17:35:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:09 smithi136 ceph-mon[109861]: osdmap e936: 8 total, 8 up, 8 in 2024-08-17T17:35:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:09 smithi165 ceph-mon[105517]: pgmap v1164: 105 pgs: 105 active+clean; 146 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 7.1 MiB/s rd, 5.4 MiB/s wr, 57 op/s 2024-08-17T17:35:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:09 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.2", "id": [0, 4]}]': finished 2024-08-17T17:35:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:09 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.f", "id": [0, 4]}]': finished 2024-08-17T17:35:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:09 smithi165 ceph-mon[105517]: osdmap e936: 8 total, 8 up, 8 in 2024-08-17T17:35:10.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:10 smithi136 ceph-mon[109861]: osdmap e937: 8 total, 8 up, 8 in 2024-08-17T17:35:10.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:35:10.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:35:10.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:35:10.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:10 smithi136 ceph-mon[115078]: osdmap e937: 8 total, 8 up, 8 in 2024-08-17T17:35:10.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:35:10.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:35:10.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:35:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:10 smithi165 ceph-mon[105517]: osdmap e937: 8 total, 8 up, 8 in 2024-08-17T17:35:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:35:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:35:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:35:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:11 smithi165 ceph-mon[105517]: pgmap v1167: 105 pgs: 105 active+clean; 146 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 40 KiB/s rd, 4.2 MiB/s wr, 52 op/s 2024-08-17T17:35:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:11 smithi136 ceph-mon[109861]: pgmap v1167: 105 pgs: 105 active+clean; 146 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 40 KiB/s rd, 4.2 MiB/s wr, 52 op/s 2024-08-17T17:35:11.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:11 smithi136 ceph-mon[115078]: pgmap v1167: 105 pgs: 105 active+clean; 146 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 40 KiB/s rd, 4.2 MiB/s wr, 52 op/s 2024-08-17T17:35:13.418 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/3.Shrink (14217 ms) 2024-08-17T17:35:13.418 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/3.ShrinkToOne 2024-08-17T17:35:13.418 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:35:13.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:13 smithi136 ceph-mon[109861]: pgmap v1168: 105 pgs: 105 active+clean; 158 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 5.1 MiB/s wr, 53 op/s; 512 KiB/s, 0 keys/s, 0 objects/s recovering 2024-08-17T17:35:13.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:13 smithi136 ceph-mon[115078]: pgmap v1168: 105 pgs: 105 active+clean; 158 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 5.1 MiB/s wr, 53 op/s; 512 KiB/s, 0 keys/s, 0 objects/s recovering 2024-08-17T17:35:13.718 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:13 smithi165 ceph-mon[105517]: pgmap v1168: 105 pgs: 105 active+clean; 158 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 5.1 MiB/s wr, 53 op/s; 512 KiB/s, 0 keys/s, 0 objects/s recovering 2024-08-17T17:35:14.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:14 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:35:14.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:14 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:35:14.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:14 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:35:14.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:14 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:35:14.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:14 smithi136 ceph-mon[109861]: pgmap v1169: 105 pgs: 105 active+clean; 158 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 3.1 MiB/s wr, 43 op/s; 512 KiB/s, 0 keys/s, 0 objects/s recovering 2024-08-17T17:35:14.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:14 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:35:14.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:14 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:35:14.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:14 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:35:14.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:14 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:35:14.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:14 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:35:14.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:14 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:35:14.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:14 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:35:14.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:14 smithi136 ceph-mon[115078]: pgmap v1169: 105 pgs: 105 active+clean; 158 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 3.1 MiB/s wr, 43 op/s; 512 KiB/s, 0 keys/s, 0 objects/s recovering 2024-08-17T17:35:14.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:14 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:35:14.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:14 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:35:14.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:14 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:35:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:14 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:35:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:14 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:35:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:14 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:35:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:14 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:35:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:14 smithi165 ceph-mon[105517]: pgmap v1169: 105 pgs: 105 active+clean; 158 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 3.1 MiB/s wr, 43 op/s; 512 KiB/s, 0 keys/s, 0 objects/s recovering 2024-08-17T17:35:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:14 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:35:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:14 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:35:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:14 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:35:15.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:35:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=cleanup t=2024-08-17T17:35:15.059458723Z level=info msg="Completed cleanup jobs" duration=21.307833ms 2024-08-17T17:35:15.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:35:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=grafana.update.checker t=2024-08-17T17:35:15.199179901Z level=info msg="Update check succeeded" duration=68.674766ms 2024-08-17T17:35:15.512 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:35:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=plugins.update.checker t=2024-08-17T17:35:15.200668789Z level=info msg="Update check succeeded" duration=78.4702ms 2024-08-17T17:35:17.650 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:17 smithi136 ceph-mon[109861]: pgmap v1170: 105 pgs: 105 active+clean; 175 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 36 op/s; 512 KiB/s, 0 keys/s, 0 objects/s recovering 2024-08-17T17:35:17.650 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:17 smithi136 ceph-mon[115078]: pgmap v1170: 105 pgs: 105 active+clean; 175 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 36 op/s; 512 KiB/s, 0 keys/s, 0 objects/s recovering 2024-08-17T17:35:17.650 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:35:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:35:17] "GET /metrics HTTP/1.1" 200 36475 "" "Prometheus/2.51.0" 2024-08-17T17:35:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:17 smithi165 ceph-mon[105517]: pgmap v1170: 105 pgs: 105 active+clean; 175 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 36 op/s; 512 KiB/s, 0 keys/s, 0 objects/s recovering 2024-08-17T17:35:18.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:18 smithi136 ceph-mon[109861]: osdmap e938: 8 total, 8 up, 8 in 2024-08-17T17:35:18.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:18 smithi136 ceph-mon[115078]: osdmap e938: 8 total, 8 up, 8 in 2024-08-17T17:35:18.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:18 smithi165 ceph-mon[105517]: osdmap e938: 8 total, 8 up, 8 in 2024-08-17T17:35:19.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:19 smithi136 ceph-mon[109861]: pgmap v1172: 105 pgs: 105 active+clean; 184 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.7 MiB/s wr, 34 op/s; 461 KiB/s, 0 keys/s, 0 objects/s recovering 2024-08-17T17:35:19.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:19 smithi136 ceph-mon[115078]: pgmap v1172: 105 pgs: 105 active+clean; 184 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.7 MiB/s wr, 34 op/s; 461 KiB/s, 0 keys/s, 0 objects/s recovering 2024-08-17T17:35:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:19 smithi165 ceph-mon[105517]: pgmap v1172: 105 pgs: 105 active+clean; 184 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.7 MiB/s wr, 34 op/s; 461 KiB/s, 0 keys/s, 0 objects/s recovering 2024-08-17T17:35:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:21 smithi136 ceph-mon[109861]: pgmap v1173: 105 pgs: 105 active+clean; 184 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 4.2 MiB/s wr, 30 op/s; 410 KiB/s, 0 keys/s, 0 objects/s recovering 2024-08-17T17:35:21.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:35:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:21 smithi136 ceph-mon[115078]: pgmap v1173: 105 pgs: 105 active+clean; 184 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 4.2 MiB/s wr, 30 op/s; 410 KiB/s, 0 keys/s, 0 objects/s recovering 2024-08-17T17:35:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:35:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:21 smithi165 ceph-mon[105517]: pgmap v1173: 105 pgs: 105 active+clean; 184 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 4.2 MiB/s wr, 30 op/s; 410 KiB/s, 0 keys/s, 0 objects/s recovering 2024-08-17T17:35:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:35:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:23 smithi136 ceph-mon[109861]: pgmap v1174: 105 pgs: 105 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.2 MiB/s rd, 5.4 MiB/s wr, 57 op/s 2024-08-17T17:35:23.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:23 smithi136 ceph-mon[115078]: pgmap v1174: 105 pgs: 105 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.2 MiB/s rd, 5.4 MiB/s wr, 57 op/s 2024-08-17T17:35:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:23 smithi165 ceph-mon[105517]: pgmap v1174: 105 pgs: 105 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.2 MiB/s rd, 5.4 MiB/s wr, 57 op/s 2024-08-17T17:35:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:25 smithi136 ceph-mon[109861]: pgmap v1175: 105 pgs: 105 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.2 MiB/s rd, 5.4 MiB/s wr, 57 op/s 2024-08-17T17:35:25.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:25 smithi136 ceph-mon[115078]: pgmap v1175: 105 pgs: 105 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.2 MiB/s rd, 5.4 MiB/s wr, 57 op/s 2024-08-17T17:35:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:25 smithi165 ceph-mon[105517]: pgmap v1175: 105 pgs: 105 active+clean; 208 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.2 MiB/s rd, 5.4 MiB/s wr, 57 op/s 2024-08-17T17:35:26.987 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/3.ShrinkToOne (13569 ms) 2024-08-17T17:35:26.988 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/3.ShrinkToOneAfterSnapshot 2024-08-17T17:35:26.988 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:35:27.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:27 smithi136 ceph-mon[109861]: pgmap v1176: 105 pgs: 105 active+clean; 212 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.1 MiB/s wr, 52 op/s 2024-08-17T17:35:27.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:35:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:35:27] "GET /metrics HTTP/1.1" 200 36495 "" "Prometheus/2.51.0" 2024-08-17T17:35:27.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:27 smithi136 ceph-mon[115078]: pgmap v1176: 105 pgs: 105 active+clean; 212 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.1 MiB/s wr, 52 op/s 2024-08-17T17:35:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:27 smithi165 ceph-mon[105517]: pgmap v1176: 105 pgs: 105 active+clean; 212 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.1 MiB/s wr, 52 op/s 2024-08-17T17:35:29.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:29 smithi136 ceph-mon[109861]: pgmap v1177: 105 pgs: 105 active+clean; 212 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 863 KiB/s rd, 3.3 MiB/s wr, 47 op/s 2024-08-17T17:35:29.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:29 smithi136 ceph-mon[115078]: pgmap v1177: 105 pgs: 105 active+clean; 212 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 863 KiB/s rd, 3.3 MiB/s wr, 47 op/s 2024-08-17T17:35:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:29 smithi165 ceph-mon[105517]: pgmap v1177: 105 pgs: 105 active+clean; 212 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 863 KiB/s rd, 3.3 MiB/s wr, 47 op/s 2024-08-17T17:35:31.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:31 smithi136 ceph-mon[115078]: pgmap v1178: 105 pgs: 105 active+clean; 212 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 787 KiB/s rd, 3.0 MiB/s wr, 43 op/s 2024-08-17T17:35:31.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:31 smithi136 ceph-mon[109861]: pgmap v1178: 105 pgs: 105 active+clean; 212 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 787 KiB/s rd, 3.0 MiB/s wr, 43 op/s 2024-08-17T17:35:31.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:31 smithi165 ceph-mon[105517]: pgmap v1178: 105 pgs: 105 active+clean; 212 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 787 KiB/s rd, 3.0 MiB/s wr, 43 op/s 2024-08-17T17:35:32.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:32 smithi136 ceph-mon[115078]: osdmap e939: 8 total, 8 up, 8 in 2024-08-17T17:35:32.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:32 smithi136 ceph-mon[109861]: osdmap e939: 8 total, 8 up, 8 in 2024-08-17T17:35:32.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:32 smithi165 ceph-mon[105517]: osdmap e939: 8 total, 8 up, 8 in 2024-08-17T17:35:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:33 smithi136 ceph-mon[109861]: pgmap v1180: 105 pgs: 105 active+clean; 241 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 900 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-08-17T17:35:33.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:33 smithi136 ceph-mon[115078]: pgmap v1180: 105 pgs: 105 active+clean; 241 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 900 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-08-17T17:35:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:33 smithi165 ceph-mon[105517]: pgmap v1180: 105 pgs: 105 active+clean; 241 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 900 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-08-17T17:35:35.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:35 smithi136 ceph-mon[109861]: pgmap v1181: 105 pgs: 105 active+clean; 241 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 900 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-08-17T17:35:35.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:35 smithi136 ceph-mon[115078]: pgmap v1181: 105 pgs: 105 active+clean; 241 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 900 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-08-17T17:35:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:35 smithi165 ceph-mon[105517]: pgmap v1181: 105 pgs: 105 active+clean; 241 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 900 KiB/s rd, 4.1 MiB/s wr, 35 op/s 2024-08-17T17:35:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:35:36.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:35:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:35:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:37 smithi136 ceph-mon[109861]: pgmap v1182: 105 pgs: 105 active+clean; 245 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 500 KiB/s rd, 3.7 MiB/s wr, 45 op/s 2024-08-17T17:35:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:37 smithi136 ceph-mon[115078]: pgmap v1182: 105 pgs: 105 active+clean; 245 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 500 KiB/s rd, 3.7 MiB/s wr, 45 op/s 2024-08-17T17:35:37.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:35:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:35:37] "GET /metrics HTTP/1.1" 200 36482 "" "Prometheus/2.51.0" 2024-08-17T17:35:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:37 smithi165 ceph-mon[105517]: pgmap v1182: 105 pgs: 105 active+clean; 245 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 500 KiB/s rd, 3.7 MiB/s wr, 45 op/s 2024-08-17T17:35:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:39 smithi136 ceph-mon[109861]: pgmap v1183: 105 pgs: 105 active+clean; 245 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 62 KiB/s rd, 3.3 MiB/s wr, 46 op/s 2024-08-17T17:35:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:39 smithi136 ceph-mon[115078]: pgmap v1183: 105 pgs: 105 active+clean; 245 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 62 KiB/s rd, 3.3 MiB/s wr, 46 op/s 2024-08-17T17:35:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:39 smithi165 ceph-mon[105517]: pgmap v1183: 105 pgs: 105 active+clean; 245 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 62 KiB/s rd, 3.3 MiB/s wr, 46 op/s 2024-08-17T17:35:40.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:40 smithi136 ceph-mon[109861]: osdmap e940: 8 total, 8 up, 8 in 2024-08-17T17:35:40.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:40 smithi136 ceph-mon[115078]: osdmap e940: 8 total, 8 up, 8 in 2024-08-17T17:35:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:40 smithi165 ceph-mon[105517]: osdmap e940: 8 total, 8 up, 8 in 2024-08-17T17:35:41.677 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/3.ShrinkToOneAfterSnapshot (14690 ms) 2024-08-17T17:35:41.677 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/3.MinOverlap 2024-08-17T17:35:41.677 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:35:41.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:41 smithi136 ceph-mon[109861]: pgmap v1185: 105 pgs: 105 active+clean; 245 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 60 KiB/s rd, 460 KiB/s wr, 36 op/s 2024-08-17T17:35:41.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:41 smithi136 ceph-mon[115078]: pgmap v1185: 105 pgs: 105 active+clean; 245 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 60 KiB/s rd, 460 KiB/s wr, 36 op/s 2024-08-17T17:35:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:41 smithi165 ceph-mon[105517]: pgmap v1185: 105 pgs: 105 active+clean; 245 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 60 KiB/s rd, 460 KiB/s wr, 36 op/s 2024-08-17T17:35:43.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:43 smithi136 ceph-mon[109861]: pgmap v1186: 105 pgs: 105 active+clean; 302 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 6.1 MiB/s wr, 60 op/s 2024-08-17T17:35:43.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:43 smithi136 ceph-mon[115078]: pgmap v1186: 105 pgs: 105 active+clean; 302 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 6.1 MiB/s wr, 60 op/s 2024-08-17T17:35:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:43 smithi165 ceph-mon[105517]: pgmap v1186: 105 pgs: 105 active+clean; 302 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 6.1 MiB/s wr, 60 op/s 2024-08-17T17:35:45.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:45 smithi136 ceph-mon[109861]: pgmap v1187: 105 pgs: 105 active+clean; 302 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 6.1 MiB/s wr, 60 op/s 2024-08-17T17:35:45.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:45 smithi136 ceph-mon[115078]: pgmap v1187: 105 pgs: 105 active+clean; 302 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 6.1 MiB/s wr, 60 op/s 2024-08-17T17:35:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:45 smithi165 ceph-mon[105517]: pgmap v1187: 105 pgs: 105 active+clean; 302 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 6.1 MiB/s wr, 60 op/s 2024-08-17T17:35:46.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:46 smithi136 ceph-mon[109861]: osdmap e941: 8 total, 8 up, 8 in 2024-08-17T17:35:46.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:46 smithi136 ceph-mon[115078]: osdmap e941: 8 total, 8 up, 8 in 2024-08-17T17:35:46.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:46 smithi165 ceph-mon[105517]: osdmap e941: 8 total, 8 up, 8 in 2024-08-17T17:35:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:47 smithi136 ceph-mon[109861]: pgmap v1189: 105 pgs: 105 active+clean; 310 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 8.1 MiB/s wr, 76 op/s 2024-08-17T17:35:47.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:35:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:35:47] "GET /metrics HTTP/1.1" 200 36482 "" "Prometheus/2.51.0" 2024-08-17T17:35:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:47 smithi136 ceph-mon[115078]: pgmap v1189: 105 pgs: 105 active+clean; 310 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 8.1 MiB/s wr, 76 op/s 2024-08-17T17:35:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:47 smithi165 ceph-mon[105517]: pgmap v1189: 105 pgs: 105 active+clean; 310 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.7 MiB/s rd, 8.1 MiB/s wr, 76 op/s 2024-08-17T17:35:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:49 smithi136 ceph-mon[109861]: pgmap v1190: 105 pgs: 105 active+clean; 331 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 9.7 MiB/s wr, 76 op/s 2024-08-17T17:35:49.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:49 smithi136 ceph-mon[115078]: pgmap v1190: 105 pgs: 105 active+clean; 331 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 9.7 MiB/s wr, 76 op/s 2024-08-17T17:35:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:49 smithi165 ceph-mon[105517]: pgmap v1190: 105 pgs: 105 active+clean; 331 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 9.7 MiB/s wr, 76 op/s 2024-08-17T17:35:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:51 smithi136 ceph-mon[109861]: pgmap v1191: 105 pgs: 105 active+clean; 331 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 8.6 MiB/s wr, 68 op/s 2024-08-17T17:35:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:35:51.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:51 smithi136 ceph-mon[115078]: pgmap v1191: 105 pgs: 105 active+clean; 331 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 8.6 MiB/s wr, 68 op/s 2024-08-17T17:35:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:35:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:51 smithi165 ceph-mon[105517]: pgmap v1191: 105 pgs: 105 active+clean; 331 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 8.6 MiB/s wr, 68 op/s 2024-08-17T17:35:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:35:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:53 smithi136 ceph-mon[109861]: pgmap v1192: 105 pgs: 105 active+clean; 335 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 62 KiB/s rd, 3.3 MiB/s wr, 45 op/s 2024-08-17T17:35:53.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:53 smithi136 ceph-mon[115078]: pgmap v1192: 105 pgs: 105 active+clean; 335 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 62 KiB/s rd, 3.3 MiB/s wr, 45 op/s 2024-08-17T17:35:53.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:53 smithi165 ceph-mon[105517]: pgmap v1192: 105 pgs: 105 active+clean; 335 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 62 KiB/s rd, 3.3 MiB/s wr, 45 op/s 2024-08-17T17:35:55.193 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/3.MinOverlap (13516 ms) 2024-08-17T17:35:55.193 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/3.ZeroOverlap 2024-08-17T17:35:55.193 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:35:55.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:55 smithi136 ceph-mon[109861]: pgmap v1193: 105 pgs: 105 active+clean; 335 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 62 KiB/s rd, 3.3 MiB/s wr, 45 op/s 2024-08-17T17:35:55.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:55 smithi136 ceph-mon[115078]: pgmap v1193: 105 pgs: 105 active+clean; 335 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 62 KiB/s rd, 3.3 MiB/s wr, 45 op/s 2024-08-17T17:35:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:55 smithi165 ceph-mon[105517]: pgmap v1193: 105 pgs: 105 active+clean; 335 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 62 KiB/s rd, 3.3 MiB/s wr, 45 op/s 2024-08-17T17:35:57.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:57 smithi136 ceph-mon[109861]: pgmap v1194: 105 pgs: 105 active+clean; 339 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 47 op/s 2024-08-17T17:35:57.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:57 smithi136 ceph-mon[115078]: pgmap v1194: 105 pgs: 105 active+clean; 339 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 47 op/s 2024-08-17T17:35:57.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:35:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:35:57] "GET /metrics HTTP/1.1" 200 36484 "" "Prometheus/2.51.0" 2024-08-17T17:35:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:57 smithi165 ceph-mon[105517]: pgmap v1194: 105 pgs: 105 active+clean; 339 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s rd, 4.1 MiB/s wr, 47 op/s 2024-08-17T17:35:59.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:35:59 smithi136 ceph-mon[109861]: pgmap v1195: 105 pgs: 105 active+clean; 339 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 28 op/s 2024-08-17T17:35:59.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:35:59 smithi136 ceph-mon[115078]: pgmap v1195: 105 pgs: 105 active+clean; 339 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 28 op/s 2024-08-17T17:35:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:35:59 smithi165 ceph-mon[105517]: pgmap v1195: 105 pgs: 105 active+clean; 339 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 28 op/s 2024-08-17T17:36:00.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:00 smithi136 ceph-mon[109861]: osdmap e942: 8 total, 8 up, 8 in 2024-08-17T17:36:00.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:00 smithi136 ceph-mon[115078]: osdmap e942: 8 total, 8 up, 8 in 2024-08-17T17:36:00.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:00 smithi165 ceph-mon[105517]: osdmap e942: 8 total, 8 up, 8 in 2024-08-17T17:36:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:01 smithi136 ceph-mon[109861]: pgmap v1197: 105 pgs: 105 active+clean; 339 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 27 op/s 2024-08-17T17:36:01.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:01 smithi136 ceph-mon[115078]: pgmap v1197: 105 pgs: 105 active+clean; 339 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 27 op/s 2024-08-17T17:36:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:01 smithi165 ceph-mon[105517]: pgmap v1197: 105 pgs: 105 active+clean; 339 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 27 op/s 2024-08-17T17:36:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:03 smithi136 ceph-mon[109861]: pgmap v1198: 105 pgs: 105 active+clean; 368 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 51 op/s 2024-08-17T17:36:03.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:03 smithi136 ceph-mon[115078]: pgmap v1198: 105 pgs: 105 active+clean; 368 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 51 op/s 2024-08-17T17:36:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:03 smithi165 ceph-mon[105517]: pgmap v1198: 105 pgs: 105 active+clean; 368 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 51 op/s 2024-08-17T17:36:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:05 smithi136 ceph-mon[109861]: pgmap v1199: 105 pgs: 105 active+clean; 368 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 51 op/s 2024-08-17T17:36:05.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:05 smithi136 ceph-mon[115078]: pgmap v1199: 105 pgs: 105 active+clean; 368 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 51 op/s 2024-08-17T17:36:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:05 smithi165 ceph-mon[105517]: pgmap v1199: 105 pgs: 105 active+clean; 368 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.1 MiB/s wr, 51 op/s 2024-08-17T17:36:06.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:36:06.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:36:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:36:07.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:07 smithi136 ceph-mon[109861]: pgmap v1200: 105 pgs: 105 active+clean; 372 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 92 KiB/s rd, 3.3 MiB/s wr, 46 op/s 2024-08-17T17:36:07.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:07 smithi136 ceph-mon[115078]: pgmap v1200: 105 pgs: 105 active+clean; 372 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 92 KiB/s rd, 3.3 MiB/s wr, 46 op/s 2024-08-17T17:36:07.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:36:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:36:07] "GET /metrics HTTP/1.1" 200 36491 "" "Prometheus/2.51.0" 2024-08-17T17:36:07.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:07 smithi165 ceph-mon[105517]: pgmap v1200: 105 pgs: 105 active+clean; 372 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 92 KiB/s rd, 3.3 MiB/s wr, 46 op/s 2024-08-17T17:36:09.088 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/3.ZeroOverlap (13895 ms) 2024-08-17T17:36:09.384 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 7 tests from EncryptedFlattenTest/3 (101697 ms total) 2024-08-17T17:36:09.384 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:36:09.384 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 7 tests from EncryptedFlattenTest/4, where TypeParam = LUKS1UnderLUKS2 2024-08-17T17:36:09.412 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/4.Simple 2024-08-17T17:36:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:09 smithi136 ceph-mon[109861]: pgmap v1201: 105 pgs: 105 active+clean; 372 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 96 KiB/s rd, 3.3 MiB/s wr, 51 op/s 2024-08-17T17:36:09.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:09 smithi136 ceph-mon[115078]: pgmap v1201: 105 pgs: 105 active+clean; 372 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 96 KiB/s rd, 3.3 MiB/s wr, 51 op/s 2024-08-17T17:36:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:09 smithi165 ceph-mon[105517]: pgmap v1201: 105 pgs: 105 active+clean; 372 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 96 KiB/s rd, 3.3 MiB/s wr, 51 op/s 2024-08-17T17:36:10.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:10 smithi136 ceph-mon[109861]: osdmap e943: 8 total, 8 up, 8 in 2024-08-17T17:36:10.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:10 smithi136 ceph-mon[115078]: osdmap e943: 8 total, 8 up, 8 in 2024-08-17T17:36:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:10 smithi165 ceph-mon[105517]: osdmap e943: 8 total, 8 up, 8 in 2024-08-17T17:36:11.398 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:36:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:11 smithi136 ceph-mon[109861]: pgmap v1203: 73 pgs: 73 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:36:11.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:11 smithi136 ceph-mon[109861]: osdmap e944: 8 total, 8 up, 8 in 2024-08-17T17:36:11.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:11 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3057617461' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-43","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:36:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:11 smithi136 ceph-mon[115078]: pgmap v1203: 73 pgs: 73 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:36:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:11 smithi136 ceph-mon[115078]: osdmap e944: 8 total, 8 up, 8 in 2024-08-17T17:36:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:11 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3057617461' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-43","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:36:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:11 smithi165 ceph-mon[105517]: pgmap v1203: 73 pgs: 73 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:36:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:11 smithi165 ceph-mon[105517]: osdmap e944: 8 total, 8 up, 8 in 2024-08-17T17:36:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:11 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3057617461' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-43","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:36:12.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:12 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3057617461' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:36:12.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:12 smithi136 ceph-mon[109861]: osdmap e945: 8 total, 8 up, 8 in 2024-08-17T17:36:12.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:12 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3057617461' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:36:12.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:12 smithi136 ceph-mon[115078]: osdmap e945: 8 total, 8 up, 8 in 2024-08-17T17:36:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:12 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3057617461' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:36:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:12 smithi165 ceph-mon[105517]: osdmap e945: 8 total, 8 up, 8 in 2024-08-17T17:36:13.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:13 smithi136 ceph-mon[115078]: pgmap v1206: 105 pgs: 15 creating+peering, 90 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:36:13.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:13 smithi136 ceph-mon[115078]: osdmap e946: 8 total, 8 up, 8 in 2024-08-17T17:36:13.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:13 smithi136 ceph-mon[109861]: pgmap v1206: 105 pgs: 15 creating+peering, 90 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:36:13.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:13 smithi136 ceph-mon[109861]: osdmap e946: 8 total, 8 up, 8 in 2024-08-17T17:36:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:13 smithi165 ceph-mon[105517]: pgmap v1206: 105 pgs: 15 creating+peering, 90 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:36:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:13 smithi165 ceph-mon[105517]: osdmap e946: 8 total, 8 up, 8 in 2024-08-17T17:36:14.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:14 smithi136 ceph-mon[115078]: osdmap e947: 8 total, 8 up, 8 in 2024-08-17T17:36:14.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:14 smithi136 ceph-mon[109861]: osdmap e947: 8 total, 8 up, 8 in 2024-08-17T17:36:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:14 smithi165 ceph-mon[105517]: osdmap e947: 8 total, 8 up, 8 in 2024-08-17T17:36:15.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:15 smithi136 ceph-mon[109861]: pgmap v1209: 105 pgs: 15 creating+peering, 90 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:36:15.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:15 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:36:15.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:15 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:36:15.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:15 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:36:15.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:15 smithi136 ceph-mon[115078]: pgmap v1209: 105 pgs: 15 creating+peering, 90 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:36:15.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:15 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:36:15.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:15 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:36:15.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:15 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:36:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:15 smithi165 ceph-mon[105517]: pgmap v1209: 105 pgs: 15 creating+peering, 90 active+clean; 582 KiB data, 746 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:36:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:15 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:36:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:15 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:36:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:15 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:36:16.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:16 smithi136 ceph-mon[109861]: pgmap v1210: 105 pgs: 105 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 520 B/s rd, 347 B/s wr, 1 op/s 2024-08-17T17:36:16.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:16 smithi136 ceph-mon[115078]: pgmap v1210: 105 pgs: 105 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 520 B/s rd, 347 B/s wr, 1 op/s 2024-08-17T17:36:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:16 smithi165 ceph-mon[105517]: pgmap v1210: 105 pgs: 105 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 520 B/s rd, 347 B/s wr, 1 op/s 2024-08-17T17:36:17.928 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:36:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:36:17] "GET /metrics HTTP/1.1" 200 36491 "" "Prometheus/2.51.0" 2024-08-17T17:36:18.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:18 smithi136 ceph-mon[109861]: osdmap e948: 8 total, 8 up, 8 in 2024-08-17T17:36:18.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:18 smithi136 ceph-mon[109861]: pgmap v1212: 105 pgs: 105 active+clean; 21 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 3.3 MiB/s wr, 26 op/s 2024-08-17T17:36:18.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:36:18.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:36:18.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:18 smithi136 ceph-mon[115078]: osdmap e948: 8 total, 8 up, 8 in 2024-08-17T17:36:18.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:18 smithi136 ceph-mon[115078]: pgmap v1212: 105 pgs: 105 active+clean; 21 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 3.3 MiB/s wr, 26 op/s 2024-08-17T17:36:18.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:36:18.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:36:18.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:18 smithi165 ceph-mon[105517]: osdmap e948: 8 total, 8 up, 8 in 2024-08-17T17:36:18.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:18 smithi165 ceph-mon[105517]: pgmap v1212: 105 pgs: 105 active+clean; 21 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 3.3 MiB/s wr, 26 op/s 2024-08-17T17:36:18.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:36:18.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:36:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:20 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:36:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:20 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:36:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:20 smithi165 ceph-mon[105517]: osdmap e949: 8 total, 8 up, 8 in 2024-08-17T17:36:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:20 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:36:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:20 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:36:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:20 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:36:20.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:20 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:36:20.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:20 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:36:20.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:20 smithi136 ceph-mon[109861]: osdmap e949: 8 total, 8 up, 8 in 2024-08-17T17:36:20.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:20 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:36:20.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:20 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:36:20.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:20 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:36:20.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:20 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:36:20.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:20 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:36:20.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:20 smithi136 ceph-mon[115078]: osdmap e949: 8 total, 8 up, 8 in 2024-08-17T17:36:20.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:20 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:36:20.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:20 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:36:20.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:20 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:36:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:21 smithi165 ceph-mon[105517]: pgmap v1214: 105 pgs: 105 active+clean; 21 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 2.9 MiB/s wr, 23 op/s 2024-08-17T17:36:21.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:21 smithi136 ceph-mon[109861]: pgmap v1214: 105 pgs: 105 active+clean; 21 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 2.9 MiB/s wr, 23 op/s 2024-08-17T17:36:21.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:21 smithi136 ceph-mon[115078]: pgmap v1214: 105 pgs: 105 active+clean; 21 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 2.9 MiB/s wr, 23 op/s 2024-08-17T17:36:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:36:22.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:36:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:36:23.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:23 smithi136 ceph-mon[109861]: pgmap v1215: 105 pgs: 105 active+clean; 30 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 3.6 MiB/s wr, 50 op/s 2024-08-17T17:36:23.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:23 smithi136 ceph-mon[115078]: pgmap v1215: 105 pgs: 105 active+clean; 30 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 3.6 MiB/s wr, 50 op/s 2024-08-17T17:36:23.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:23 smithi165 ceph-mon[105517]: pgmap v1215: 105 pgs: 105 active+clean; 30 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 3.6 MiB/s wr, 50 op/s 2024-08-17T17:36:25.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:25 smithi136 ceph-mon[109861]: pgmap v1216: 105 pgs: 105 active+clean; 30 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 3.6 MiB/s wr, 49 op/s 2024-08-17T17:36:25.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:25 smithi136 ceph-mon[115078]: pgmap v1216: 105 pgs: 105 active+clean; 30 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 3.6 MiB/s wr, 49 op/s 2024-08-17T17:36:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:25 smithi165 ceph-mon[105517]: pgmap v1216: 105 pgs: 105 active+clean; 30 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 3.6 MiB/s wr, 49 op/s 2024-08-17T17:36:27.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:27 smithi136 ceph-mon[109861]: pgmap v1217: 105 pgs: 105 active+clean; 38 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1.9 MiB/s wr, 31 op/s 2024-08-17T17:36:27.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:27 smithi136 ceph-mon[115078]: pgmap v1217: 105 pgs: 105 active+clean; 38 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1.9 MiB/s wr, 31 op/s 2024-08-17T17:36:27.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:36:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:36:27] "GET /metrics HTTP/1.1" 200 36467 "" "Prometheus/2.51.0" 2024-08-17T17:36:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:27 smithi165 ceph-mon[105517]: pgmap v1217: 105 pgs: 105 active+clean; 38 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 1.9 MiB/s wr, 31 op/s 2024-08-17T17:36:29.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:29 smithi136 ceph-mon[109861]: pgmap v1218: 105 pgs: 105 active+clean; 46 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 2.5 MiB/s wr, 35 op/s 2024-08-17T17:36:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:29 smithi136 ceph-mon[115078]: pgmap v1218: 105 pgs: 105 active+clean; 46 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 2.5 MiB/s wr, 35 op/s 2024-08-17T17:36:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:29 smithi165 ceph-mon[105517]: pgmap v1218: 105 pgs: 105 active+clean; 46 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 2.5 MiB/s wr, 35 op/s 2024-08-17T17:36:30.772 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/4.Simple (21360 ms) 2024-08-17T17:36:30.772 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/4.Grow 2024-08-17T17:36:30.772 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:36:31.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:31 smithi136 ceph-mon[109861]: pgmap v1219: 105 pgs: 105 active+clean; 46 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 2.2 MiB/s wr, 31 op/s 2024-08-17T17:36:31.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:31 smithi136 ceph-mon[115078]: pgmap v1219: 105 pgs: 105 active+clean; 46 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 2.2 MiB/s wr, 31 op/s 2024-08-17T17:36:31.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:31 smithi165 ceph-mon[105517]: pgmap v1219: 105 pgs: 105 active+clean; 46 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 2.2 MiB/s wr, 31 op/s 2024-08-17T17:36:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:33 smithi136 ceph-mon[109861]: pgmap v1220: 105 pgs: 105 active+clean; 71 MiB data, 816 MiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 5.5 MiB/s wr, 47 op/s 2024-08-17T17:36:33.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:33 smithi136 ceph-mon[115078]: pgmap v1220: 105 pgs: 105 active+clean; 71 MiB data, 816 MiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 5.5 MiB/s wr, 47 op/s 2024-08-17T17:36:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:33 smithi165 ceph-mon[105517]: pgmap v1220: 105 pgs: 105 active+clean; 71 MiB data, 816 MiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 5.5 MiB/s wr, 47 op/s 2024-08-17T17:36:35.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:35 smithi136 ceph-mon[109861]: pgmap v1221: 105 pgs: 105 active+clean; 71 MiB data, 816 MiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 4.8 MiB/s wr, 27 op/s 2024-08-17T17:36:35.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:35 smithi136 ceph-mon[109861]: osdmap e950: 8 total, 8 up, 8 in 2024-08-17T17:36:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:35 smithi136 ceph-mon[115078]: pgmap v1221: 105 pgs: 105 active+clean; 71 MiB data, 816 MiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 4.8 MiB/s wr, 27 op/s 2024-08-17T17:36:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:35 smithi136 ceph-mon[115078]: osdmap e950: 8 total, 8 up, 8 in 2024-08-17T17:36:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:35 smithi165 ceph-mon[105517]: pgmap v1221: 105 pgs: 105 active+clean; 71 MiB data, 816 MiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 4.8 MiB/s wr, 27 op/s 2024-08-17T17:36:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:35 smithi165 ceph-mon[105517]: osdmap e950: 8 total, 8 up, 8 in 2024-08-17T17:36:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:36:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:36:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:36:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:37 smithi136 ceph-mon[109861]: pgmap v1223: 105 pgs: 105 active+clean; 80 MiB data, 850 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 5.8 MiB/s wr, 45 op/s 2024-08-17T17:36:37.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:36:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:36:37] "GET /metrics HTTP/1.1" 200 36473 "" "Prometheus/2.51.0" 2024-08-17T17:36:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:37 smithi136 ceph-mon[115078]: pgmap v1223: 105 pgs: 105 active+clean; 80 MiB data, 850 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 5.8 MiB/s wr, 45 op/s 2024-08-17T17:36:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:37 smithi165 ceph-mon[105517]: pgmap v1223: 105 pgs: 105 active+clean; 80 MiB data, 850 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 5.8 MiB/s wr, 45 op/s 2024-08-17T17:36:39.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:39 smithi136 ceph-mon[109861]: pgmap v1224: 105 pgs: 105 active+clean; 100 MiB data, 903 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 58 op/s 2024-08-17T17:36:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:39 smithi136 ceph-mon[115078]: pgmap v1224: 105 pgs: 105 active+clean; 100 MiB data, 903 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 58 op/s 2024-08-17T17:36:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:39 smithi165 ceph-mon[105517]: pgmap v1224: 105 pgs: 105 active+clean; 100 MiB data, 903 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 58 op/s 2024-08-17T17:36:41.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:41 smithi136 ceph-mon[115078]: pgmap v1225: 105 pgs: 105 active+clean; 100 MiB data, 903 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 58 op/s 2024-08-17T17:36:41.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:41 smithi136 ceph-mon[109861]: pgmap v1225: 105 pgs: 105 active+clean; 100 MiB data, 903 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 58 op/s 2024-08-17T17:36:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:41 smithi165 ceph-mon[105517]: pgmap v1225: 105 pgs: 105 active+clean; 100 MiB data, 903 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 58 op/s 2024-08-17T17:36:43.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:43 smithi136 ceph-mon[109861]: pgmap v1226: 105 pgs: 105 active+clean; 116 MiB data, 939 MiB used, 714 GiB / 715 GiB avail; 911 KiB/s rd, 4.5 MiB/s wr, 48 op/s 2024-08-17T17:36:43.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:43 smithi136 ceph-mon[115078]: pgmap v1226: 105 pgs: 105 active+clean; 116 MiB data, 939 MiB used, 714 GiB / 715 GiB avail; 911 KiB/s rd, 4.5 MiB/s wr, 48 op/s 2024-08-17T17:36:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:43 smithi165 ceph-mon[105517]: pgmap v1226: 105 pgs: 105 active+clean; 116 MiB data, 939 MiB used, 714 GiB / 715 GiB avail; 911 KiB/s rd, 4.5 MiB/s wr, 48 op/s 2024-08-17T17:36:45.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:45 smithi136 ceph-mon[109861]: pgmap v1227: 105 pgs: 105 active+clean; 116 MiB data, 939 MiB used, 714 GiB / 715 GiB avail; 911 KiB/s rd, 4.5 MiB/s wr, 48 op/s 2024-08-17T17:36:45.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:45 smithi136 ceph-mon[115078]: pgmap v1227: 105 pgs: 105 active+clean; 116 MiB data, 939 MiB used, 714 GiB / 715 GiB avail; 911 KiB/s rd, 4.5 MiB/s wr, 48 op/s 2024-08-17T17:36:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:45 smithi165 ceph-mon[105517]: pgmap v1227: 105 pgs: 105 active+clean; 116 MiB data, 939 MiB used, 714 GiB / 715 GiB avail; 911 KiB/s rd, 4.5 MiB/s wr, 48 op/s 2024-08-17T17:36:45.811 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/4.Grow (15039 ms) 2024-08-17T17:36:45.811 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/4.Shrink 2024-08-17T17:36:45.812 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:36:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:47 smithi136 ceph-mon[109861]: pgmap v1228: 105 pgs: 105 active+clean; 129 MiB data, 996 MiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 5.5 MiB/s wr, 51 op/s 2024-08-17T17:36:47.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:36:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:36:47] "GET /metrics HTTP/1.1" 200 36473 "" "Prometheus/2.51.0" 2024-08-17T17:36:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:47 smithi136 ceph-mon[115078]: pgmap v1228: 105 pgs: 105 active+clean; 129 MiB data, 996 MiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 5.5 MiB/s wr, 51 op/s 2024-08-17T17:36:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:47 smithi165 ceph-mon[105517]: pgmap v1228: 105 pgs: 105 active+clean; 129 MiB data, 996 MiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 5.5 MiB/s wr, 51 op/s 2024-08-17T17:36:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:49 smithi136 ceph-mon[109861]: pgmap v1229: 105 pgs: 105 active+clean; 141 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 9.3 MiB/s rd, 6.4 MiB/s wr, 47 op/s 2024-08-17T17:36:49.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:49 smithi136 ceph-mon[115078]: pgmap v1229: 105 pgs: 105 active+clean; 141 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 9.3 MiB/s rd, 6.4 MiB/s wr, 47 op/s 2024-08-17T17:36:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:49 smithi165 ceph-mon[105517]: pgmap v1229: 105 pgs: 105 active+clean; 141 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 9.3 MiB/s rd, 6.4 MiB/s wr, 47 op/s 2024-08-17T17:36:50.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:50 smithi136 ceph-mon[109861]: osdmap e951: 8 total, 8 up, 8 in 2024-08-17T17:36:50.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:50 smithi136 ceph-mon[115078]: osdmap e951: 8 total, 8 up, 8 in 2024-08-17T17:36:50.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:50 smithi165 ceph-mon[105517]: osdmap e951: 8 total, 8 up, 8 in 2024-08-17T17:36:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:51 smithi136 ceph-mon[109861]: pgmap v1231: 105 pgs: 105 active+clean; 141 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.7 MiB/s wr, 36 op/s 2024-08-17T17:36:51.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:36:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:51 smithi136 ceph-mon[115078]: pgmap v1231: 105 pgs: 105 active+clean; 141 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.7 MiB/s wr, 36 op/s 2024-08-17T17:36:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:36:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:51 smithi165 ceph-mon[105517]: pgmap v1231: 105 pgs: 105 active+clean; 141 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 5.7 MiB/s wr, 36 op/s 2024-08-17T17:36:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:36:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:53 smithi136 ceph-mon[109861]: pgmap v1232: 105 pgs: 105 active+clean; 170 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 60 op/s 2024-08-17T17:36:53.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:53 smithi136 ceph-mon[115078]: pgmap v1232: 105 pgs: 105 active+clean; 170 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 60 op/s 2024-08-17T17:36:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:53 smithi165 ceph-mon[105517]: pgmap v1232: 105 pgs: 105 active+clean; 170 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 60 op/s 2024-08-17T17:36:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:55 smithi136 ceph-mon[109861]: pgmap v1233: 105 pgs: 105 active+clean; 170 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 60 op/s 2024-08-17T17:36:55.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:55 smithi136 ceph-mon[115078]: pgmap v1233: 105 pgs: 105 active+clean; 170 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 60 op/s 2024-08-17T17:36:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:55 smithi165 ceph-mon[105517]: pgmap v1233: 105 pgs: 105 active+clean; 170 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 60 op/s 2024-08-17T17:36:57.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:57 smithi136 ceph-mon[109861]: pgmap v1234: 105 pgs: 105 active+clean; 178 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 5.7 MiB/s wr, 55 op/s 2024-08-17T17:36:57.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:57 smithi136 ceph-mon[115078]: pgmap v1234: 105 pgs: 105 active+clean; 178 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 5.7 MiB/s wr, 55 op/s 2024-08-17T17:36:57.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:36:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:36:57] "GET /metrics HTTP/1.1" 200 36483 "" "Prometheus/2.51.0" 2024-08-17T17:36:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:57 smithi165 ceph-mon[105517]: pgmap v1234: 105 pgs: 105 active+clean; 178 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.0 MiB/s rd, 5.7 MiB/s wr, 55 op/s 2024-08-17T17:36:59.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:36:59 smithi136 ceph-mon[109861]: pgmap v1235: 105 pgs: 105 active+clean; 186 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 575 KiB/s rd, 4.5 MiB/s wr, 46 op/s 2024-08-17T17:36:59.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:36:59 smithi136 ceph-mon[115078]: pgmap v1235: 105 pgs: 105 active+clean; 186 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 575 KiB/s rd, 4.5 MiB/s wr, 46 op/s 2024-08-17T17:36:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:36:59 smithi165 ceph-mon[105517]: pgmap v1235: 105 pgs: 105 active+clean; 186 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 575 KiB/s rd, 4.5 MiB/s wr, 46 op/s 2024-08-17T17:37:00.919 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/4.Shrink (15108 ms) 2024-08-17T17:37:00.920 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/4.ShrinkToOne 2024-08-17T17:37:00.920 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:37:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:01 smithi136 ceph-mon[109861]: pgmap v1236: 105 pgs: 105 active+clean; 186 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 527 KiB/s rd, 4.1 MiB/s wr, 43 op/s 2024-08-17T17:37:01.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:01 smithi136 ceph-mon[115078]: pgmap v1236: 105 pgs: 105 active+clean; 186 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 527 KiB/s rd, 4.1 MiB/s wr, 43 op/s 2024-08-17T17:37:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:01 smithi165 ceph-mon[105517]: pgmap v1236: 105 pgs: 105 active+clean; 186 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 527 KiB/s rd, 4.1 MiB/s wr, 43 op/s 2024-08-17T17:37:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:03 smithi136 ceph-mon[109861]: pgmap v1237: 105 pgs: 105 active+clean; 211 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.5 MiB/s rd, 7.2 MiB/s wr, 59 op/s 2024-08-17T17:37:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:03 smithi136 ceph-mon[115078]: pgmap v1237: 105 pgs: 105 active+clean; 211 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.5 MiB/s rd, 7.2 MiB/s wr, 59 op/s 2024-08-17T17:37:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:03 smithi165 ceph-mon[105517]: pgmap v1237: 105 pgs: 105 active+clean; 211 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.5 MiB/s rd, 7.2 MiB/s wr, 59 op/s 2024-08-17T17:37:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:05 smithi136 ceph-mon[109861]: pgmap v1238: 105 pgs: 105 active+clean; 211 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 4.8 MiB/s wr, 30 op/s 2024-08-17T17:37:05.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:05 smithi136 ceph-mon[109861]: osdmap e952: 8 total, 8 up, 8 in 2024-08-17T17:37:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:05 smithi136 ceph-mon[115078]: pgmap v1238: 105 pgs: 105 active+clean; 211 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 4.8 MiB/s wr, 30 op/s 2024-08-17T17:37:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:05 smithi136 ceph-mon[115078]: osdmap e952: 8 total, 8 up, 8 in 2024-08-17T17:37:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:05 smithi165 ceph-mon[105517]: pgmap v1238: 105 pgs: 105 active+clean; 211 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 4.8 MiB/s wr, 30 op/s 2024-08-17T17:37:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:05 smithi165 ceph-mon[105517]: osdmap e952: 8 total, 8 up, 8 in 2024-08-17T17:37:06.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:37:06.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:06 smithi136 ceph-mon[109861]: pgmap v1240: 105 pgs: 105 active+clean; 223 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.1 MiB/s wr, 31 op/s 2024-08-17T17:37:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:37:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:06 smithi136 ceph-mon[115078]: pgmap v1240: 105 pgs: 105 active+clean; 223 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.1 MiB/s wr, 31 op/s 2024-08-17T17:37:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:37:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:06 smithi165 ceph-mon[105517]: pgmap v1240: 105 pgs: 105 active+clean; 223 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.1 MiB/s wr, 31 op/s 2024-08-17T17:37:07.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:37:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:37:07] "GET /metrics HTTP/1.1" 200 36481 "" "Prometheus/2.51.0" 2024-08-17T17:37:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:09 smithi136 ceph-mon[109861]: pgmap v1241: 105 pgs: 105 active+clean; 240 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 60 op/s 2024-08-17T17:37:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:09 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "47.9", "id": [1, 2]} : dispatch 2024-08-17T17:37:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:09 smithi136 ceph-mon[115078]: pgmap v1241: 105 pgs: 105 active+clean; 240 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 60 op/s 2024-08-17T17:37:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:09 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "47.9", "id": [1, 2]} : dispatch 2024-08-17T17:37:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:09 smithi165 ceph-mon[105517]: pgmap v1241: 105 pgs: 105 active+clean; 240 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 7.0 MiB/s wr, 60 op/s 2024-08-17T17:37:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:09 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "47.9", "id": [1, 2]} : dispatch 2024-08-17T17:37:10.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "47.9", "id": [1, 2]}]': finished 2024-08-17T17:37:10.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:10 smithi136 ceph-mon[109861]: osdmap e953: 8 total, 8 up, 8 in 2024-08-17T17:37:10.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "47.9", "id": [1, 2]}]': finished 2024-08-17T17:37:10.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:10 smithi136 ceph-mon[115078]: osdmap e953: 8 total, 8 up, 8 in 2024-08-17T17:37:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "47.9", "id": [1, 2]}]': finished 2024-08-17T17:37:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:10 smithi165 ceph-mon[105517]: osdmap e953: 8 total, 8 up, 8 in 2024-08-17T17:37:11.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:11 smithi136 ceph-mon[115078]: pgmap v1243: 105 pgs: 105 active+clean; 240 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.6 MiB/s wr, 45 op/s 2024-08-17T17:37:11.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:11 smithi136 ceph-mon[115078]: osdmap e954: 8 total, 8 up, 8 in 2024-08-17T17:37:11.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:11 smithi136 ceph-mon[109861]: pgmap v1243: 105 pgs: 105 active+clean; 240 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.6 MiB/s wr, 45 op/s 2024-08-17T17:37:11.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:11 smithi136 ceph-mon[109861]: osdmap e954: 8 total, 8 up, 8 in 2024-08-17T17:37:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:11 smithi165 ceph-mon[105517]: pgmap v1243: 105 pgs: 105 active+clean; 240 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.0 MiB/s rd, 3.6 MiB/s wr, 45 op/s 2024-08-17T17:37:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:11 smithi165 ceph-mon[105517]: osdmap e954: 8 total, 8 up, 8 in 2024-08-17T17:37:13.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:13 smithi136 ceph-mon[109861]: pgmap v1245: 105 pgs: 105 active+clean; 256 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.7 MiB/s wr, 60 op/s; 1.5 MiB/s, 0 objects/s recovering 2024-08-17T17:37:13.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:13 smithi136 ceph-mon[115078]: pgmap v1245: 105 pgs: 105 active+clean; 256 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.7 MiB/s wr, 60 op/s; 1.5 MiB/s, 0 objects/s recovering 2024-08-17T17:37:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:13 smithi165 ceph-mon[105517]: pgmap v1245: 105 pgs: 105 active+clean; 256 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 5.7 MiB/s wr, 60 op/s; 1.5 MiB/s, 0 objects/s recovering 2024-08-17T17:37:15.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:15 smithi136 ceph-mon[109861]: pgmap v1246: 105 pgs: 105 active+clean; 256 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 113 KiB/s rd, 4.1 MiB/s wr, 55 op/s; 1.5 MiB/s, 0 objects/s recovering 2024-08-17T17:37:15.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:15 smithi136 ceph-mon[115078]: pgmap v1246: 105 pgs: 105 active+clean; 256 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 113 KiB/s rd, 4.1 MiB/s wr, 55 op/s; 1.5 MiB/s, 0 objects/s recovering 2024-08-17T17:37:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:15 smithi165 ceph-mon[105517]: pgmap v1246: 105 pgs: 105 active+clean; 256 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 113 KiB/s rd, 4.1 MiB/s wr, 55 op/s; 1.5 MiB/s, 0 objects/s recovering 2024-08-17T17:37:15.808 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/4.ShrinkToOne (14889 ms) 2024-08-17T17:37:15.808 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/4.ShrinkToOneAfterSnapshot 2024-08-17T17:37:15.808 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:37:17.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:17 smithi136 ceph-mon[109861]: pgmap v1247: 105 pgs: 105 active+clean; 260 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 16 op/s; 1.5 MiB/s, 0 objects/s recovering 2024-08-17T17:37:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:17 smithi136 ceph-mon[115078]: pgmap v1247: 105 pgs: 105 active+clean; 260 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 16 op/s; 1.5 MiB/s, 0 objects/s recovering 2024-08-17T17:37:17.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:37:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:37:17] "GET /metrics HTTP/1.1" 200 36481 "" "Prometheus/2.51.0" 2024-08-17T17:37:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:17 smithi165 ceph-mon[105517]: pgmap v1247: 105 pgs: 105 active+clean; 260 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 16 op/s; 1.5 MiB/s, 0 objects/s recovering 2024-08-17T17:37:19.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:19 smithi136 ceph-mon[109861]: pgmap v1248: 105 pgs: 105 active+clean; 260 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 4.5 MiB/s wr, 30 op/s; 1.3 MiB/s, 0 objects/s recovering 2024-08-17T17:37:19.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:19 smithi136 ceph-mon[115078]: pgmap v1248: 105 pgs: 105 active+clean; 260 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 4.5 MiB/s wr, 30 op/s; 1.3 MiB/s, 0 objects/s recovering 2024-08-17T17:37:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:19 smithi165 ceph-mon[105517]: pgmap v1248: 105 pgs: 105 active+clean; 260 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 4.5 MiB/s wr, 30 op/s; 1.3 MiB/s, 0 objects/s recovering 2024-08-17T17:37:20.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:20 smithi136 ceph-mon[109861]: osdmap e955: 8 total, 8 up, 8 in 2024-08-17T17:37:20.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:20 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:37:20.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:20 smithi136 ceph-mon[115078]: osdmap e955: 8 total, 8 up, 8 in 2024-08-17T17:37:20.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:20 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:37:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:20 smithi165 ceph-mon[105517]: osdmap e955: 8 total, 8 up, 8 in 2024-08-17T17:37:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:20 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:37:21.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:21 smithi165 ceph-mon[105517]: pgmap v1250: 105 pgs: 105 active+clean; 260 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.0 MiB/s wr, 27 op/s; 1.2 MiB/s, 0 objects/s recovering 2024-08-17T17:37:21.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:37:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:21 smithi136 ceph-mon[109861]: pgmap v1250: 105 pgs: 105 active+clean; 260 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.0 MiB/s wr, 27 op/s; 1.2 MiB/s, 0 objects/s recovering 2024-08-17T17:37:21.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:37:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:21 smithi136 ceph-mon[115078]: pgmap v1250: 105 pgs: 105 active+clean; 260 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.0 MiB/s wr, 27 op/s; 1.2 MiB/s, 0 objects/s recovering 2024-08-17T17:37:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:37:23.571 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:23 smithi165 ceph-mon[105517]: pgmap v1251: 105 pgs: 105 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 51 op/s 2024-08-17T17:37:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:23 smithi136 ceph-mon[109861]: pgmap v1251: 105 pgs: 105 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 51 op/s 2024-08-17T17:37:23.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:23 smithi136 ceph-mon[115078]: pgmap v1251: 105 pgs: 105 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 51 op/s 2024-08-17T17:37:24.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:24 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:37:24.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:24 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:37:24.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:24 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:37:24.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:24 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:37:24.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:24 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:37:24.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:24 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:37:24.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:24 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:37:24.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:24 smithi165 ceph-mon[105517]: pgmap v1252: 105 pgs: 105 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 51 op/s 2024-08-17T17:37:24.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:24 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:37:24.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:24 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:37:24.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:24 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:37:24.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:24 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:37:24.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:24 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:37:24.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:24 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:37:24.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:24 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:37:24.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:24 smithi136 ceph-mon[109861]: pgmap v1252: 105 pgs: 105 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 51 op/s 2024-08-17T17:37:24.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:24 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:37:24.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:24 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:37:24.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:24 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:37:24.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:24 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:37:24.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:24 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:37:24.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:24 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:37:24.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:24 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:37:24.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:24 smithi136 ceph-mon[115078]: pgmap v1252: 105 pgs: 105 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 51 op/s 2024-08-17T17:37:27.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:27 smithi136 ceph-mon[109861]: pgmap v1253: 105 pgs: 105 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 446 KiB/s rd, 3.3 MiB/s wr, 49 op/s 2024-08-17T17:37:27.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:27 smithi136 ceph-mon[115078]: pgmap v1253: 105 pgs: 105 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 446 KiB/s rd, 3.3 MiB/s wr, 49 op/s 2024-08-17T17:37:27.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:37:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:37:27] "GET /metrics HTTP/1.1" 200 36495 "" "Prometheus/2.51.0" 2024-08-17T17:37:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:27 smithi165 ceph-mon[105517]: pgmap v1253: 105 pgs: 105 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 446 KiB/s rd, 3.3 MiB/s wr, 49 op/s 2024-08-17T17:37:29.667 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:29 smithi165 ceph-mon[105517]: pgmap v1254: 105 pgs: 105 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 2.9 MiB/s wr, 36 op/s 2024-08-17T17:37:29.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:29 smithi136 ceph-mon[109861]: pgmap v1254: 105 pgs: 105 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 2.9 MiB/s wr, 36 op/s 2024-08-17T17:37:29.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:29 smithi136 ceph-mon[115078]: pgmap v1254: 105 pgs: 105 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 2.9 MiB/s wr, 36 op/s 2024-08-17T17:37:31.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:31 smithi136 ceph-mon[109861]: pgmap v1255: 105 pgs: 105 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.7 MiB/s wr, 33 op/s 2024-08-17T17:37:31.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:31 smithi136 ceph-mon[115078]: pgmap v1255: 105 pgs: 105 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.7 MiB/s wr, 33 op/s 2024-08-17T17:37:31.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:31 smithi165 ceph-mon[105517]: pgmap v1255: 105 pgs: 105 active+clean; 289 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.7 MiB/s wr, 33 op/s 2024-08-17T17:37:32.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:32 smithi136 ceph-mon[109861]: osdmap e956: 8 total, 8 up, 8 in 2024-08-17T17:37:32.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:32 smithi136 ceph-mon[115078]: osdmap e956: 8 total, 8 up, 8 in 2024-08-17T17:37:32.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:32 smithi165 ceph-mon[105517]: osdmap e956: 8 total, 8 up, 8 in 2024-08-17T17:37:33.530 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/4.ShrinkToOneAfterSnapshot (17722 ms) 2024-08-17T17:37:33.531 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/4.MinOverlap 2024-08-17T17:37:33.531 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:37:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:33 smithi136 ceph-mon[109861]: pgmap v1257: 105 pgs: 105 active+clean; 354 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 6.5 MiB/s wr, 42 op/s 2024-08-17T17:37:33.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:33 smithi136 ceph-mon[115078]: pgmap v1257: 105 pgs: 105 active+clean; 354 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 6.5 MiB/s wr, 42 op/s 2024-08-17T17:37:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:33 smithi165 ceph-mon[105517]: pgmap v1257: 105 pgs: 105 active+clean; 354 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 6.5 MiB/s wr, 42 op/s 2024-08-17T17:37:35.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:35 smithi136 ceph-mon[115078]: pgmap v1258: 105 pgs: 105 active+clean; 354 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 6.5 MiB/s wr, 42 op/s 2024-08-17T17:37:35.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:35 smithi136 ceph-mon[109861]: pgmap v1258: 105 pgs: 105 active+clean; 354 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 6.5 MiB/s wr, 42 op/s 2024-08-17T17:37:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:35 smithi165 ceph-mon[105517]: pgmap v1258: 105 pgs: 105 active+clean; 354 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 6.5 MiB/s wr, 42 op/s 2024-08-17T17:37:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:37:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:37:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:37:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:37 smithi136 ceph-mon[109861]: pgmap v1259: 105 pgs: 105 active+clean; 374 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 8.5 MiB/s wr, 49 op/s 2024-08-17T17:37:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:37 smithi136 ceph-mon[115078]: pgmap v1259: 105 pgs: 105 active+clean; 374 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 8.5 MiB/s wr, 49 op/s 2024-08-17T17:37:37.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:37:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:37:37] "GET /metrics HTTP/1.1" 200 36488 "" "Prometheus/2.51.0" 2024-08-17T17:37:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:37 smithi165 ceph-mon[105517]: pgmap v1259: 105 pgs: 105 active+clean; 374 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 8.5 MiB/s wr, 49 op/s 2024-08-17T17:37:38.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:38 smithi136 ceph-mon[109861]: osdmap e957: 8 total, 8 up, 8 in 2024-08-17T17:37:38.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:38 smithi136 ceph-mon[115078]: osdmap e957: 8 total, 8 up, 8 in 2024-08-17T17:37:38.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:38 smithi165 ceph-mon[105517]: osdmap e957: 8 total, 8 up, 8 in 2024-08-17T17:37:39.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:39 smithi165 ceph-mon[105517]: pgmap v1261: 105 pgs: 105 active+clean; 387 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 5.3 MiB/s rd, 12 MiB/s wr, 62 op/s 2024-08-17T17:37:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:39 smithi136 ceph-mon[109861]: pgmap v1261: 105 pgs: 105 active+clean; 387 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 5.3 MiB/s rd, 12 MiB/s wr, 62 op/s 2024-08-17T17:37:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:39 smithi136 ceph-mon[115078]: pgmap v1261: 105 pgs: 105 active+clean; 387 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 5.3 MiB/s rd, 12 MiB/s wr, 62 op/s 2024-08-17T17:37:41.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:41 smithi136 ceph-mon[109861]: pgmap v1262: 105 pgs: 105 active+clean; 387 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 9.6 MiB/s wr, 55 op/s 2024-08-17T17:37:41.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:41 smithi136 ceph-mon[115078]: pgmap v1262: 105 pgs: 105 active+clean; 387 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 9.6 MiB/s wr, 55 op/s 2024-08-17T17:37:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:41 smithi165 ceph-mon[105517]: pgmap v1262: 105 pgs: 105 active+clean; 387 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 9.6 MiB/s wr, 55 op/s 2024-08-17T17:37:43.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:43 smithi136 ceph-mon[109861]: pgmap v1263: 105 pgs: 105 active+clean; 403 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 4.9 MiB/s wr, 42 op/s 2024-08-17T17:37:43.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:43 smithi136 ceph-mon[115078]: pgmap v1263: 105 pgs: 105 active+clean; 403 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 4.9 MiB/s wr, 42 op/s 2024-08-17T17:37:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:43 smithi165 ceph-mon[105517]: pgmap v1263: 105 pgs: 105 active+clean; 403 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 4.9 MiB/s wr, 42 op/s 2024-08-17T17:37:45.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:45 smithi136 ceph-mon[109861]: pgmap v1264: 105 pgs: 105 active+clean; 403 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 4.9 MiB/s wr, 42 op/s 2024-08-17T17:37:45.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:45 smithi136 ceph-mon[115078]: pgmap v1264: 105 pgs: 105 active+clean; 403 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 4.9 MiB/s wr, 42 op/s 2024-08-17T17:37:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:45 smithi165 ceph-mon[105517]: pgmap v1264: 105 pgs: 105 active+clean; 403 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 4.9 MiB/s wr, 42 op/s 2024-08-17T17:37:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:47 smithi136 ceph-mon[109861]: pgmap v1265: 105 pgs: 105 active+clean; 411 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 90 KiB/s rd, 3.7 MiB/s wr, 42 op/s 2024-08-17T17:37:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:47 smithi136 ceph-mon[115078]: pgmap v1265: 105 pgs: 105 active+clean; 411 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 90 KiB/s rd, 3.7 MiB/s wr, 42 op/s 2024-08-17T17:37:47.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:37:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:37:47] "GET /metrics HTTP/1.1" 200 36488 "" "Prometheus/2.51.0" 2024-08-17T17:37:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:47 smithi165 ceph-mon[105517]: pgmap v1265: 105 pgs: 105 active+clean; 411 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 90 KiB/s rd, 3.7 MiB/s wr, 42 op/s 2024-08-17T17:37:48.823 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/4.MinOverlap (15293 ms) 2024-08-17T17:37:48.824 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/4.ZeroOverlap 2024-08-17T17:37:48.824 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:37:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:49 smithi136 ceph-mon[109861]: pgmap v1266: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 45 op/s 2024-08-17T17:37:49.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:49 smithi136 ceph-mon[115078]: pgmap v1266: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 45 op/s 2024-08-17T17:37:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:49 smithi165 ceph-mon[105517]: pgmap v1266: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 45 op/s 2024-08-17T17:37:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:51 smithi136 ceph-mon[109861]: pgmap v1267: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 41 op/s 2024-08-17T17:37:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:37:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:51 smithi136 ceph-mon[115078]: pgmap v1267: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 41 op/s 2024-08-17T17:37:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:37:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:51 smithi165 ceph-mon[105517]: pgmap v1267: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 41 op/s 2024-08-17T17:37:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:37:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:53 smithi136 ceph-mon[109861]: pgmap v1268: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 4.7 MiB/s wr, 54 op/s 2024-08-17T17:37:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:53 smithi136 ceph-mon[109861]: osdmap e958: 8 total, 8 up, 8 in 2024-08-17T17:37:53.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:53 smithi136 ceph-mon[115078]: pgmap v1268: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 4.7 MiB/s wr, 54 op/s 2024-08-17T17:37:53.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:53 smithi136 ceph-mon[115078]: osdmap e958: 8 total, 8 up, 8 in 2024-08-17T17:37:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:53 smithi165 ceph-mon[105517]: pgmap v1268: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 4.7 MiB/s wr, 54 op/s 2024-08-17T17:37:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:53 smithi165 ceph-mon[105517]: osdmap e958: 8 total, 8 up, 8 in 2024-08-17T17:37:54.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:54 smithi165 ceph-mon[105517]: pgmap v1270: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 4.0 MiB/s wr, 30 op/s 2024-08-17T17:37:54.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:54 smithi136 ceph-mon[109861]: pgmap v1270: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 4.0 MiB/s wr, 30 op/s 2024-08-17T17:37:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:54 smithi136 ceph-mon[115078]: pgmap v1270: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 4.0 MiB/s wr, 30 op/s 2024-08-17T17:37:57.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:57 smithi136 ceph-mon[109861]: pgmap v1271: 105 pgs: 105 active+clean; 431 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 4.0 MiB/s wr, 43 op/s 2024-08-17T17:37:57.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:57 smithi136 ceph-mon[115078]: pgmap v1271: 105 pgs: 105 active+clean; 431 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 4.0 MiB/s wr, 43 op/s 2024-08-17T17:37:57.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:37:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:37:57] "GET /metrics HTTP/1.1" 200 36486 "" "Prometheus/2.51.0" 2024-08-17T17:37:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:57 smithi165 ceph-mon[105517]: pgmap v1271: 105 pgs: 105 active+clean; 431 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 4.0 MiB/s wr, 43 op/s 2024-08-17T17:37:59.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:37:59 smithi136 ceph-mon[109861]: pgmap v1272: 105 pgs: 105 active+clean; 452 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 49 op/s 2024-08-17T17:37:59.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:37:59 smithi136 ceph-mon[115078]: pgmap v1272: 105 pgs: 105 active+clean; 452 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 49 op/s 2024-08-17T17:37:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:37:59 smithi165 ceph-mon[105517]: pgmap v1272: 105 pgs: 105 active+clean; 452 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 49 op/s 2024-08-17T17:38:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:01 smithi136 ceph-mon[109861]: pgmap v1273: 105 pgs: 105 active+clean; 452 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 49 op/s 2024-08-17T17:38:01.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:01 smithi136 ceph-mon[115078]: pgmap v1273: 105 pgs: 105 active+clean; 452 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 49 op/s 2024-08-17T17:38:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:01 smithi165 ceph-mon[105517]: pgmap v1273: 105 pgs: 105 active+clean; 452 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 49 op/s 2024-08-17T17:38:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:03 smithi136 ceph-mon[109861]: pgmap v1274: 105 pgs: 105 active+clean; 468 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 919 KiB/s rd, 5.3 MiB/s wr, 57 op/s 2024-08-17T17:38:03.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:03 smithi136 ceph-mon[115078]: pgmap v1274: 105 pgs: 105 active+clean; 468 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 919 KiB/s rd, 5.3 MiB/s wr, 57 op/s 2024-08-17T17:38:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:03 smithi165 ceph-mon[105517]: pgmap v1274: 105 pgs: 105 active+clean; 468 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 919 KiB/s rd, 5.3 MiB/s wr, 57 op/s 2024-08-17T17:38:03.782 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/4.ZeroOverlap (14959 ms) 2024-08-17T17:38:04.392 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 7 tests from EncryptedFlattenTest/4 (114370 ms total) 2024-08-17T17:38:04.393 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:38:04.393 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 7 tests from EncryptedFlattenTest/5, where TypeParam = UnformattedLUKS2 2024-08-17T17:38:04.422 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/5.Simple 2024-08-17T17:38:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:05 smithi136 ceph-mon[109861]: pgmap v1275: 105 pgs: 105 active+clean; 468 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 773 KiB/s rd, 4.5 MiB/s wr, 48 op/s 2024-08-17T17:38:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:05 smithi136 ceph-mon[109861]: osdmap e959: 8 total, 8 up, 8 in 2024-08-17T17:38:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:05 smithi136 ceph-mon[115078]: pgmap v1275: 105 pgs: 105 active+clean; 468 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 773 KiB/s rd, 4.5 MiB/s wr, 48 op/s 2024-08-17T17:38:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:05 smithi136 ceph-mon[115078]: osdmap e959: 8 total, 8 up, 8 in 2024-08-17T17:38:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:05 smithi165 ceph-mon[105517]: pgmap v1275: 105 pgs: 105 active+clean; 468 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 773 KiB/s rd, 4.5 MiB/s wr, 48 op/s 2024-08-17T17:38:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:05 smithi165 ceph-mon[105517]: osdmap e959: 8 total, 8 up, 8 in 2024-08-17T17:38:06.399 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:38:06.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:06 smithi136 ceph-mon[109861]: osdmap e960: 8 total, 8 up, 8 in 2024-08-17T17:38:06.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:06 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1067907889' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-44","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:38:06.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:06 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-44","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:38:06.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:38:06.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:06 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:38:06.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:06 smithi136 ceph-mon[109861]: osdmap e961: 8 total, 8 up, 8 in 2024-08-17T17:38:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:06 smithi136 ceph-mon[115078]: osdmap e960: 8 total, 8 up, 8 in 2024-08-17T17:38:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:06 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1067907889' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-44","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:38:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:06 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-44","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:38:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:38:06.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:06 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:38:06.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:06 smithi136 ceph-mon[115078]: osdmap e961: 8 total, 8 up, 8 in 2024-08-17T17:38:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:06 smithi165 ceph-mon[105517]: osdmap e960: 8 total, 8 up, 8 in 2024-08-17T17:38:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:06 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1067907889' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-44","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:38:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:06 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-44","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:38:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:38:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:06 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:38:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:06 smithi165 ceph-mon[105517]: osdmap e961: 8 total, 8 up, 8 in 2024-08-17T17:38:07.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:38:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:38:07] "GET /metrics HTTP/1.1" 200 36416 "" "Prometheus/2.51.0" 2024-08-17T17:38:07.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:07 smithi136 ceph-mon[109861]: pgmap v1278: 105 pgs: 16 creating+peering, 16 unknown, 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:38:07.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:07 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:38:07.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:07 smithi136 ceph-mon[109861]: osdmap e962: 8 total, 8 up, 8 in 2024-08-17T17:38:07.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:07 smithi136 ceph-mon[115078]: pgmap v1278: 105 pgs: 16 creating+peering, 16 unknown, 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:38:07.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:07 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:38:07.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:07 smithi136 ceph-mon[115078]: osdmap e962: 8 total, 8 up, 8 in 2024-08-17T17:38:07.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:07 smithi165 ceph-mon[105517]: pgmap v1278: 105 pgs: 16 creating+peering, 16 unknown, 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:38:07.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:07 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:38:07.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:07 smithi165 ceph-mon[105517]: osdmap e962: 8 total, 8 up, 8 in 2024-08-17T17:38:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:09 smithi136 ceph-mon[109861]: pgmap v1281: 105 pgs: 16 creating+peering, 89 active+clean; 582 KiB data, 918 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:38:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:09 smithi136 ceph-mon[109861]: osdmap e963: 8 total, 8 up, 8 in 2024-08-17T17:38:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:09 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:38:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:09 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T17:38:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:09 smithi136 ceph-mon[115078]: pgmap v1281: 105 pgs: 16 creating+peering, 89 active+clean; 582 KiB data, 918 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:38:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:09 smithi136 ceph-mon[115078]: osdmap e963: 8 total, 8 up, 8 in 2024-08-17T17:38:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:09 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:38:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:09 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T17:38:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:09 smithi165 ceph-mon[105517]: pgmap v1281: 105 pgs: 16 creating+peering, 89 active+clean; 582 KiB data, 918 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:38:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:09 smithi165 ceph-mon[105517]: osdmap e963: 8 total, 8 up, 8 in 2024-08-17T17:38:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:09 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:38:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:09 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T17:38:11.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:11 smithi136 ceph-mon[109861]: pgmap v1283: 105 pgs: 16 creating+peering, 89 active+clean; 582 KiB data, 918 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:38:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:11 smithi136 ceph-mon[115078]: pgmap v1283: 105 pgs: 16 creating+peering, 89 active+clean; 582 KiB data, 918 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:38:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:11 smithi165 ceph-mon[105517]: pgmap v1283: 105 pgs: 16 creating+peering, 89 active+clean; 582 KiB data, 918 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:38:13.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:13 smithi136 ceph-mon[109861]: pgmap v1284: 105 pgs: 105 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-08-17T17:38:13.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:13 smithi136 ceph-mon[115078]: pgmap v1284: 105 pgs: 105 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-08-17T17:38:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:13 smithi165 ceph-mon[105517]: pgmap v1284: 105 pgs: 105 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-08-17T17:38:15.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:15 smithi136 ceph-mon[109861]: osdmap e964: 8 total, 8 up, 8 in 2024-08-17T17:38:15.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:15 smithi136 ceph-mon[109861]: pgmap v1286: 105 pgs: 105 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-08-17T17:38:15.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:15 smithi136 ceph-mon[115078]: osdmap e964: 8 total, 8 up, 8 in 2024-08-17T17:38:15.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:15 smithi136 ceph-mon[115078]: pgmap v1286: 105 pgs: 105 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-08-17T17:38:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:15 smithi165 ceph-mon[105517]: osdmap e964: 8 total, 8 up, 8 in 2024-08-17T17:38:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:15 smithi165 ceph-mon[105517]: pgmap v1286: 105 pgs: 105 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-08-17T17:38:16.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:16 smithi136 ceph-mon[109861]: osdmap e965: 8 total, 8 up, 8 in 2024-08-17T17:38:16.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:16 smithi136 ceph-mon[115078]: osdmap e965: 8 total, 8 up, 8 in 2024-08-17T17:38:16.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:16 smithi165 ceph-mon[105517]: osdmap e965: 8 total, 8 up, 8 in 2024-08-17T17:38:17.347 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:17 smithi136 ceph-mon[109861]: pgmap v1288: 105 pgs: 105 active+clean; 22 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 2.6 MiB/s wr, 8 op/s 2024-08-17T17:38:17.348 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:17 smithi136 ceph-mon[115078]: pgmap v1288: 105 pgs: 105 active+clean; 22 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 2.6 MiB/s wr, 8 op/s 2024-08-17T17:38:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:17 smithi165 ceph-mon[105517]: pgmap v1288: 105 pgs: 105 active+clean; 22 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 2.6 MiB/s wr, 8 op/s 2024-08-17T17:38:17.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:38:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:38:17] "GET /metrics HTTP/1.1" 200 36416 "" "Prometheus/2.51.0" 2024-08-17T17:38:19.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:19 smithi136 ceph-mon[109861]: pgmap v1289: 105 pgs: 105 active+clean; 42 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 79 KiB/s rd, 5.1 MiB/s wr, 59 op/s 2024-08-17T17:38:19.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:19 smithi136 ceph-mon[115078]: pgmap v1289: 105 pgs: 105 active+clean; 42 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 79 KiB/s rd, 5.1 MiB/s wr, 59 op/s 2024-08-17T17:38:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:19 smithi165 ceph-mon[105517]: pgmap v1289: 105 pgs: 105 active+clean; 42 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 79 KiB/s rd, 5.1 MiB/s wr, 59 op/s 2024-08-17T17:38:21.030 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/5.Simple (16608 ms) 2024-08-17T17:38:21.030 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/5.Grow 2024-08-17T17:38:21.030 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:38:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:21 smithi136 ceph-mon[109861]: pgmap v1290: 105 pgs: 105 active+clean; 42 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 75 KiB/s rd, 5.1 MiB/s wr, 52 op/s 2024-08-17T17:38:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:38:21.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:21 smithi136 ceph-mon[115078]: pgmap v1290: 105 pgs: 105 active+clean; 42 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 75 KiB/s rd, 5.1 MiB/s wr, 52 op/s 2024-08-17T17:38:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:38:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:21 smithi165 ceph-mon[105517]: pgmap v1290: 105 pgs: 105 active+clean; 42 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 75 KiB/s rd, 5.1 MiB/s wr, 52 op/s 2024-08-17T17:38:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:38:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:23 smithi136 ceph-mon[109861]: pgmap v1291: 105 pgs: 105 active+clean; 83 MiB data, 905 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 9.9 MiB/s wr, 76 op/s 2024-08-17T17:38:23.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:23 smithi136 ceph-mon[115078]: pgmap v1291: 105 pgs: 105 active+clean; 83 MiB data, 905 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 9.9 MiB/s wr, 76 op/s 2024-08-17T17:38:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:23 smithi165 ceph-mon[105517]: pgmap v1291: 105 pgs: 105 active+clean; 83 MiB data, 905 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 9.9 MiB/s wr, 76 op/s 2024-08-17T17:38:24.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:24 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:38:24.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:24 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:38:24.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:24 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:38:25.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:25 smithi136 ceph-mon[115078]: pgmap v1292: 105 pgs: 105 active+clean; 83 MiB data, 905 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 8.2 MiB/s wr, 63 op/s 2024-08-17T17:38:25.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:25 smithi136 ceph-mon[109861]: pgmap v1292: 105 pgs: 105 active+clean; 83 MiB data, 905 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 8.2 MiB/s wr, 63 op/s 2024-08-17T17:38:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:25 smithi165 ceph-mon[105517]: pgmap v1292: 105 pgs: 105 active+clean; 83 MiB data, 905 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 8.2 MiB/s wr, 63 op/s 2024-08-17T17:38:27.628 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:27 smithi165 ceph-mon[105517]: pgmap v1293: 105 pgs: 105 active+clean; 83 MiB data, 905 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 7.3 MiB/s wr, 60 op/s 2024-08-17T17:38:27.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:27 smithi136 ceph-mon[109861]: pgmap v1293: 105 pgs: 105 active+clean; 83 MiB data, 905 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 7.3 MiB/s wr, 60 op/s 2024-08-17T17:38:27.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:27 smithi136 ceph-mon[115078]: pgmap v1293: 105 pgs: 105 active+clean; 83 MiB data, 905 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 7.3 MiB/s wr, 60 op/s 2024-08-17T17:38:27.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:38:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:38:27] "GET /metrics HTTP/1.1" 200 36470 "" "Prometheus/2.51.0" 2024-08-17T17:38:28.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:28 smithi136 ceph-mon[109861]: osdmap e966: 8 total, 8 up, 8 in 2024-08-17T17:38:28.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:28 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:38:28.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:28 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:38:28.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:28 smithi136 ceph-mon[115078]: osdmap e966: 8 total, 8 up, 8 in 2024-08-17T17:38:28.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:28 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:38:28.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:28 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:38:28.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:28 smithi165 ceph-mon[105517]: osdmap e966: 8 total, 8 up, 8 in 2024-08-17T17:38:28.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:28 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:38:28.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:28 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:38:29.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:29 smithi136 ceph-mon[109861]: pgmap v1295: 105 pgs: 105 active+clean; 111 MiB data, 914 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.9 MiB/s wr, 27 op/s 2024-08-17T17:38:29.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:29 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:38:29.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:29 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:38:29.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:29 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:38:29.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:29 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:38:29.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:29 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:38:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:29 smithi136 ceph-mon[115078]: pgmap v1295: 105 pgs: 105 active+clean; 111 MiB data, 914 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.9 MiB/s wr, 27 op/s 2024-08-17T17:38:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:29 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:38:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:29 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:38:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:29 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:38:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:29 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:38:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:29 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:38:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:29 smithi165 ceph-mon[105517]: pgmap v1295: 105 pgs: 105 active+clean; 111 MiB data, 914 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.9 MiB/s wr, 27 op/s 2024-08-17T17:38:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:29 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:38:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:29 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:38:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:29 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:38:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:29 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:38:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:29 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:38:31.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:31 smithi136 ceph-mon[109861]: pgmap v1296: 105 pgs: 105 active+clean; 111 MiB data, 914 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.9 MiB/s wr, 27 op/s 2024-08-17T17:38:31.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:31 smithi136 ceph-mon[115078]: pgmap v1296: 105 pgs: 105 active+clean; 111 MiB data, 914 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.9 MiB/s wr, 27 op/s 2024-08-17T17:38:31.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:31 smithi165 ceph-mon[105517]: pgmap v1296: 105 pgs: 105 active+clean; 111 MiB data, 914 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.9 MiB/s wr, 27 op/s 2024-08-17T17:38:33.651 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/5.Grow (12622 ms) 2024-08-17T17:38:33.652 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/5.Shrink 2024-08-17T17:38:33.652 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:38:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:33 smithi136 ceph-mon[109861]: pgmap v1297: 105 pgs: 105 active+clean; 153 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.5 MiB/s rd, 7.0 MiB/s wr, 58 op/s 2024-08-17T17:38:33.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:33 smithi136 ceph-mon[115078]: pgmap v1297: 105 pgs: 105 active+clean; 153 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.5 MiB/s rd, 7.0 MiB/s wr, 58 op/s 2024-08-17T17:38:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:33 smithi165 ceph-mon[105517]: pgmap v1297: 105 pgs: 105 active+clean; 153 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.5 MiB/s rd, 7.0 MiB/s wr, 58 op/s 2024-08-17T17:38:35.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:35 smithi136 ceph-mon[109861]: pgmap v1298: 105 pgs: 105 active+clean; 153 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.5 MiB/s rd, 7.0 MiB/s wr, 58 op/s 2024-08-17T17:38:35.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:35 smithi136 ceph-mon[115078]: pgmap v1298: 105 pgs: 105 active+clean; 153 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.5 MiB/s rd, 7.0 MiB/s wr, 58 op/s 2024-08-17T17:38:35.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:35 smithi165 ceph-mon[105517]: pgmap v1298: 105 pgs: 105 active+clean; 153 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.5 MiB/s rd, 7.0 MiB/s wr, 58 op/s 2024-08-17T17:38:36.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:38:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:38:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:38:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:37 smithi165 ceph-mon[105517]: pgmap v1299: 105 pgs: 105 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 8.2 MiB/s wr, 65 op/s 2024-08-17T17:38:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:37 smithi136 ceph-mon[109861]: pgmap v1299: 105 pgs: 105 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 8.2 MiB/s wr, 65 op/s 2024-08-17T17:38:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:37 smithi136 ceph-mon[115078]: pgmap v1299: 105 pgs: 105 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 8.2 MiB/s wr, 65 op/s 2024-08-17T17:38:37.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:38:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:38:37] "GET /metrics HTTP/1.1" 200 36477 "" "Prometheus/2.51.0" 2024-08-17T17:38:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:39 smithi136 ceph-mon[109861]: pgmap v1300: 105 pgs: 105 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 4.9 MiB/s wr, 64 op/s 2024-08-17T17:38:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:39 smithi136 ceph-mon[115078]: pgmap v1300: 105 pgs: 105 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 4.9 MiB/s wr, 64 op/s 2024-08-17T17:38:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:39 smithi165 ceph-mon[105517]: pgmap v1300: 105 pgs: 105 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 4.9 MiB/s wr, 64 op/s 2024-08-17T17:38:40.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:40 smithi136 ceph-mon[109861]: osdmap e967: 8 total, 8 up, 8 in 2024-08-17T17:38:40.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:40 smithi136 ceph-mon[115078]: osdmap e967: 8 total, 8 up, 8 in 2024-08-17T17:38:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:40 smithi165 ceph-mon[105517]: osdmap e967: 8 total, 8 up, 8 in 2024-08-17T17:38:41.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:41 smithi136 ceph-mon[109861]: pgmap v1302: 105 pgs: 105 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 5.4 MiB/s wr, 69 op/s 2024-08-17T17:38:41.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:41 smithi136 ceph-mon[115078]: pgmap v1302: 105 pgs: 105 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 5.4 MiB/s wr, 69 op/s 2024-08-17T17:38:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:41 smithi165 ceph-mon[105517]: pgmap v1302: 105 pgs: 105 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 5.4 MiB/s wr, 69 op/s 2024-08-17T17:38:43.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:43 smithi136 ceph-mon[109861]: pgmap v1303: 105 pgs: 105 active+clean; 206 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 5.3 MiB/s wr, 59 op/s 2024-08-17T17:38:43.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:43 smithi136 ceph-mon[115078]: pgmap v1303: 105 pgs: 105 active+clean; 206 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 5.3 MiB/s wr, 59 op/s 2024-08-17T17:38:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:43 smithi165 ceph-mon[105517]: pgmap v1303: 105 pgs: 105 active+clean; 206 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 5.3 MiB/s wr, 59 op/s 2024-08-17T17:38:45.392 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/5.Shrink (11741 ms) 2024-08-17T17:38:45.393 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/5.ShrinkToOne 2024-08-17T17:38:45.393 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:38:45.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:45 smithi136 ceph-mon[109861]: pgmap v1304: 105 pgs: 105 active+clean; 206 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 5.3 MiB/s wr, 59 op/s 2024-08-17T17:38:45.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:45 smithi136 ceph-mon[115078]: pgmap v1304: 105 pgs: 105 active+clean; 206 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 5.3 MiB/s wr, 59 op/s 2024-08-17T17:38:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:45 smithi165 ceph-mon[105517]: pgmap v1304: 105 pgs: 105 active+clean; 206 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.6 MiB/s rd, 5.3 MiB/s wr, 59 op/s 2024-08-17T17:38:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:47 smithi136 ceph-mon[109861]: pgmap v1305: 105 pgs: 105 active+clean; 231 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 6.6 MiB/s wr, 52 op/s 2024-08-17T17:38:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:47 smithi136 ceph-mon[115078]: pgmap v1305: 105 pgs: 105 active+clean; 231 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 6.6 MiB/s wr, 52 op/s 2024-08-17T17:38:47.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:38:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:38:47] "GET /metrics HTTP/1.1" 200 36477 "" "Prometheus/2.51.0" 2024-08-17T17:38:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:47 smithi165 ceph-mon[105517]: pgmap v1305: 105 pgs: 105 active+clean; 231 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 6.6 MiB/s wr, 52 op/s 2024-08-17T17:38:49.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:49 smithi136 ceph-mon[109861]: pgmap v1306: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 71 op/s 2024-08-17T17:38:49.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:49 smithi136 ceph-mon[115078]: pgmap v1306: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 71 op/s 2024-08-17T17:38:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:49 smithi165 ceph-mon[105517]: pgmap v1306: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 8.2 MiB/s wr, 71 op/s 2024-08-17T17:38:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:51 smithi136 ceph-mon[109861]: pgmap v1307: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 7.5 MiB/s wr, 65 op/s 2024-08-17T17:38:51.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:38:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:51 smithi136 ceph-mon[115078]: pgmap v1307: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 7.5 MiB/s wr, 65 op/s 2024-08-17T17:38:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:38:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:51 smithi165 ceph-mon[105517]: pgmap v1307: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 7.5 MiB/s wr, 65 op/s 2024-08-17T17:38:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:38:52.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:52 smithi136 ceph-mon[109861]: osdmap e968: 8 total, 8 up, 8 in 2024-08-17T17:38:52.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:52 smithi136 ceph-mon[115078]: osdmap e968: 8 total, 8 up, 8 in 2024-08-17T17:38:52.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:52 smithi165 ceph-mon[105517]: osdmap e968: 8 total, 8 up, 8 in 2024-08-17T17:38:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:53 smithi136 ceph-mon[109861]: pgmap v1309: 105 pgs: 105 active+clean; 268 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.2 MiB/s wr, 60 op/s 2024-08-17T17:38:53.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:53 smithi136 ceph-mon[115078]: pgmap v1309: 105 pgs: 105 active+clean; 268 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.2 MiB/s wr, 60 op/s 2024-08-17T17:38:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:53 smithi165 ceph-mon[105517]: pgmap v1309: 105 pgs: 105 active+clean; 268 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.2 MiB/s wr, 60 op/s 2024-08-17T17:38:54.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:54 smithi165 ceph-mon[105517]: pgmap v1310: 105 pgs: 105 active+clean; 268 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.2 MiB/s wr, 60 op/s 2024-08-17T17:38:54.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:54 smithi136 ceph-mon[109861]: pgmap v1310: 105 pgs: 105 active+clean; 268 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.2 MiB/s wr, 60 op/s 2024-08-17T17:38:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:54 smithi136 ceph-mon[115078]: pgmap v1310: 105 pgs: 105 active+clean; 268 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 6.2 MiB/s wr, 60 op/s 2024-08-17T17:38:57.306 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/5.ShrinkToOne (11914 ms) 2024-08-17T17:38:57.306 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/5.ShrinkToOneAfterSnapshot 2024-08-17T17:38:57.307 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:38:57.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:57 smithi136 ceph-mon[109861]: pgmap v1311: 105 pgs: 105 active+clean; 304 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 7.3 MiB/s wr, 74 op/s 2024-08-17T17:38:57.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:57 smithi136 ceph-mon[115078]: pgmap v1311: 105 pgs: 105 active+clean; 304 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 7.3 MiB/s wr, 74 op/s 2024-08-17T17:38:57.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:38:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:38:57] "GET /metrics HTTP/1.1" 200 36481 "" "Prometheus/2.51.0" 2024-08-17T17:38:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:57 smithi165 ceph-mon[105517]: pgmap v1311: 105 pgs: 105 active+clean; 304 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 7.3 MiB/s wr, 74 op/s 2024-08-17T17:38:59.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:38:59 smithi136 ceph-mon[109861]: pgmap v1312: 105 pgs: 105 active+clean; 308 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 56 op/s 2024-08-17T17:38:59.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:38:59 smithi136 ceph-mon[115078]: pgmap v1312: 105 pgs: 105 active+clean; 308 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 56 op/s 2024-08-17T17:38:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:38:59 smithi165 ceph-mon[105517]: pgmap v1312: 105 pgs: 105 active+clean; 308 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 56 op/s 2024-08-17T17:39:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:01 smithi136 ceph-mon[109861]: pgmap v1313: 105 pgs: 105 active+clean; 308 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 56 op/s 2024-08-17T17:39:01.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:01 smithi136 ceph-mon[115078]: pgmap v1313: 105 pgs: 105 active+clean; 308 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 56 op/s 2024-08-17T17:39:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:01 smithi165 ceph-mon[105517]: pgmap v1313: 105 pgs: 105 active+clean; 308 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 56 op/s 2024-08-17T17:39:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:03 smithi136 ceph-mon[109861]: pgmap v1314: 105 pgs: 105 active+clean; 308 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s rd, 6.0 MiB/s wr, 56 op/s 2024-08-17T17:39:03.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:03 smithi136 ceph-mon[115078]: pgmap v1314: 105 pgs: 105 active+clean; 308 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s rd, 6.0 MiB/s wr, 56 op/s 2024-08-17T17:39:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:03 smithi165 ceph-mon[105517]: pgmap v1314: 105 pgs: 105 active+clean; 308 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.9 MiB/s rd, 6.0 MiB/s wr, 56 op/s 2024-08-17T17:39:04.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:04 smithi136 ceph-mon[109861]: osdmap e969: 8 total, 8 up, 8 in 2024-08-17T17:39:04.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:04 smithi136 ceph-mon[115078]: osdmap e969: 8 total, 8 up, 8 in 2024-08-17T17:39:04.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:04 smithi165 ceph-mon[105517]: osdmap e969: 8 total, 8 up, 8 in 2024-08-17T17:39:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:05 smithi136 ceph-mon[109861]: pgmap v1316: 105 pgs: 105 active+clean; 308 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.4 MiB/s wr, 30 op/s 2024-08-17T17:39:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:05 smithi136 ceph-mon[115078]: pgmap v1316: 105 pgs: 105 active+clean; 308 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.4 MiB/s wr, 30 op/s 2024-08-17T17:39:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:05 smithi165 ceph-mon[105517]: pgmap v1316: 105 pgs: 105 active+clean; 308 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 4.4 MiB/s wr, 30 op/s 2024-08-17T17:39:06.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:39:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:39:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:39:07.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:07 smithi136 ceph-mon[109861]: pgmap v1317: 105 pgs: 105 active+clean; 321 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 50 op/s 2024-08-17T17:39:07.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:07 smithi136 ceph-mon[115078]: pgmap v1317: 105 pgs: 105 active+clean; 321 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 50 op/s 2024-08-17T17:39:07.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:39:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:39:07] "GET /metrics HTTP/1.1" 200 36478 "" "Prometheus/2.51.0" 2024-08-17T17:39:07.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:07 smithi165 ceph-mon[105517]: pgmap v1317: 105 pgs: 105 active+clean; 321 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 50 op/s 2024-08-17T17:39:08.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:08 smithi136 ceph-mon[109861]: osdmap e970: 8 total, 8 up, 8 in 2024-08-17T17:39:08.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:08 smithi136 ceph-mon[115078]: osdmap e970: 8 total, 8 up, 8 in 2024-08-17T17:39:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:08 smithi165 ceph-mon[105517]: osdmap e970: 8 total, 8 up, 8 in 2024-08-17T17:39:09.634 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/5.ShrinkToOneAfterSnapshot (12328 ms) 2024-08-17T17:39:09.634 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/5.MinOverlap 2024-08-17T17:39:09.634 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:39:09.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:09 smithi136 ceph-mon[115078]: pgmap v1319: 105 pgs: 105 active+clean; 349 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 115 KiB/s rd, 5.1 MiB/s wr, 59 op/s 2024-08-17T17:39:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:09 smithi136 ceph-mon[109861]: pgmap v1319: 105 pgs: 105 active+clean; 349 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 115 KiB/s rd, 5.1 MiB/s wr, 59 op/s 2024-08-17T17:39:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:09 smithi165 ceph-mon[105517]: pgmap v1319: 105 pgs: 105 active+clean; 349 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 115 KiB/s rd, 5.1 MiB/s wr, 59 op/s 2024-08-17T17:39:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:11 smithi136 ceph-mon[109861]: pgmap v1320: 105 pgs: 105 active+clean; 349 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 111 KiB/s rd, 5.1 MiB/s wr, 52 op/s 2024-08-17T17:39:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:11 smithi136 ceph-mon[115078]: pgmap v1320: 105 pgs: 105 active+clean; 349 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 111 KiB/s rd, 5.1 MiB/s wr, 52 op/s 2024-08-17T17:39:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:11 smithi165 ceph-mon[105517]: pgmap v1320: 105 pgs: 105 active+clean; 349 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 111 KiB/s rd, 5.1 MiB/s wr, 52 op/s 2024-08-17T17:39:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:13 smithi165 ceph-mon[105517]: pgmap v1321: 105 pgs: 105 active+clean; 434 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.5 MiB/s rd, 14 MiB/s wr, 94 op/s 2024-08-17T17:39:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:13 smithi136 ceph-mon[109861]: pgmap v1321: 105 pgs: 105 active+clean; 434 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.5 MiB/s rd, 14 MiB/s wr, 94 op/s 2024-08-17T17:39:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:13 smithi136 ceph-mon[115078]: pgmap v1321: 105 pgs: 105 active+clean; 434 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.5 MiB/s rd, 14 MiB/s wr, 94 op/s 2024-08-17T17:39:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:14 smithi165 ceph-mon[105517]: pgmap v1322: 105 pgs: 105 active+clean; 434 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 13 MiB/s wr, 84 op/s 2024-08-17T17:39:14.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:14 smithi136 ceph-mon[109861]: pgmap v1322: 105 pgs: 105 active+clean; 434 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 13 MiB/s wr, 84 op/s 2024-08-17T17:39:14.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:14 smithi136 ceph-mon[115078]: pgmap v1322: 105 pgs: 105 active+clean; 434 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 13 MiB/s wr, 84 op/s 2024-08-17T17:39:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:16 smithi165 ceph-mon[105517]: osdmap e971: 8 total, 8 up, 8 in 2024-08-17T17:39:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:16 smithi165 ceph-mon[105517]: pgmap v1324: 105 pgs: 105 active+clean; 463 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.5 MiB/s rd, 13 MiB/s wr, 58 op/s 2024-08-17T17:39:16.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:16 smithi136 ceph-mon[109861]: osdmap e971: 8 total, 8 up, 8 in 2024-08-17T17:39:16.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:16 smithi136 ceph-mon[109861]: pgmap v1324: 105 pgs: 105 active+clean; 463 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.5 MiB/s rd, 13 MiB/s wr, 58 op/s 2024-08-17T17:39:16.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:16 smithi136 ceph-mon[115078]: osdmap e971: 8 total, 8 up, 8 in 2024-08-17T17:39:16.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:16 smithi136 ceph-mon[115078]: pgmap v1324: 105 pgs: 105 active+clean; 463 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.5 MiB/s rd, 13 MiB/s wr, 58 op/s 2024-08-17T17:39:17.848 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:39:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:39:17] "GET /metrics HTTP/1.1" 200 36478 "" "Prometheus/2.51.0" 2024-08-17T17:39:19.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:19 smithi136 ceph-mon[109861]: pgmap v1325: 105 pgs: 105 active+clean; 475 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 13 MiB/s wr, 75 op/s 2024-08-17T17:39:19.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:19 smithi136 ceph-mon[115078]: pgmap v1325: 105 pgs: 105 active+clean; 475 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 13 MiB/s wr, 75 op/s 2024-08-17T17:39:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:19 smithi165 ceph-mon[105517]: pgmap v1325: 105 pgs: 105 active+clean; 475 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 13 MiB/s wr, 75 op/s 2024-08-17T17:39:21.413 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/5.MinOverlap (11779 ms) 2024-08-17T17:39:21.413 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/5.ZeroOverlap 2024-08-17T17:39:21.414 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:39:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:21 smithi136 ceph-mon[109861]: pgmap v1326: 105 pgs: 105 active+clean; 475 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 13 MiB/s wr, 75 op/s 2024-08-17T17:39:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:39:21.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:21 smithi136 ceph-mon[115078]: pgmap v1326: 105 pgs: 105 active+clean; 475 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 13 MiB/s wr, 75 op/s 2024-08-17T17:39:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:39:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:21 smithi165 ceph-mon[105517]: pgmap v1326: 105 pgs: 105 active+clean; 475 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.8 MiB/s rd, 13 MiB/s wr, 75 op/s 2024-08-17T17:39:21.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:39:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:23 smithi136 ceph-mon[109861]: pgmap v1327: 105 pgs: 105 active+clean; 495 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 6.5 MiB/s wr, 61 op/s 2024-08-17T17:39:23.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:23 smithi136 ceph-mon[115078]: pgmap v1327: 105 pgs: 105 active+clean; 495 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 6.5 MiB/s wr, 61 op/s 2024-08-17T17:39:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:23 smithi165 ceph-mon[105517]: pgmap v1327: 105 pgs: 105 active+clean; 495 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 6.5 MiB/s wr, 61 op/s 2024-08-17T17:39:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:25 smithi136 ceph-mon[109861]: pgmap v1328: 105 pgs: 105 active+clean; 495 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 6.5 MiB/s wr, 61 op/s 2024-08-17T17:39:25.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:25 smithi136 ceph-mon[115078]: pgmap v1328: 105 pgs: 105 active+clean; 495 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 6.5 MiB/s wr, 61 op/s 2024-08-17T17:39:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:25 smithi165 ceph-mon[105517]: pgmap v1328: 105 pgs: 105 active+clean; 495 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 6.5 MiB/s wr, 61 op/s 2024-08-17T17:39:27.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:27 smithi136 ceph-mon[109861]: pgmap v1329: 105 pgs: 105 active+clean; 495 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 6.0 MiB/s wr, 59 op/s 2024-08-17T17:39:27.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:27 smithi136 ceph-mon[115078]: pgmap v1329: 105 pgs: 105 active+clean; 495 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 6.0 MiB/s wr, 59 op/s 2024-08-17T17:39:27.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:39:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:39:27] "GET /metrics HTTP/1.1" 200 36482 "" "Prometheus/2.51.0" 2024-08-17T17:39:27.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:27 smithi165 ceph-mon[105517]: pgmap v1329: 105 pgs: 105 active+clean; 495 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 6.0 MiB/s wr, 59 op/s 2024-08-17T17:39:28.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:28 smithi136 ceph-mon[109861]: osdmap e972: 8 total, 8 up, 8 in 2024-08-17T17:39:28.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:28 smithi136 ceph-mon[115078]: osdmap e972: 8 total, 8 up, 8 in 2024-08-17T17:39:28.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:28 smithi165 ceph-mon[105517]: osdmap e972: 8 total, 8 up, 8 in 2024-08-17T17:39:29.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:29 smithi136 ceph-mon[109861]: pgmap v1331: 105 pgs: 105 active+clean; 507 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 39 op/s 2024-08-17T17:39:29.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:29 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:39:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:29 smithi136 ceph-mon[115078]: pgmap v1331: 105 pgs: 105 active+clean; 507 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 39 op/s 2024-08-17T17:39:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:29 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:39:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:29 smithi165 ceph-mon[105517]: pgmap v1331: 105 pgs: 105 active+clean; 507 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 39 op/s 2024-08-17T17:39:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:29 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:39:31.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:31 smithi136 ceph-mon[109861]: pgmap v1332: 105 pgs: 105 active+clean; 507 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 39 op/s 2024-08-17T17:39:31.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:31 smithi136 ceph-mon[115078]: pgmap v1332: 105 pgs: 105 active+clean; 507 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 39 op/s 2024-08-17T17:39:31.745 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:31 smithi165 ceph-mon[105517]: pgmap v1332: 105 pgs: 105 active+clean; 507 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 39 op/s 2024-08-17T17:39:33.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:33 smithi165 ceph-mon[105517]: pgmap v1333: 105 pgs: 105 active+clean; 544 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.9 MiB/s wr, 51 op/s 2024-08-17T17:39:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:33 smithi136 ceph-mon[109861]: pgmap v1333: 105 pgs: 105 active+clean; 544 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.9 MiB/s wr, 51 op/s 2024-08-17T17:39:33.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:33 smithi136 ceph-mon[115078]: pgmap v1333: 105 pgs: 105 active+clean; 544 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.9 MiB/s wr, 51 op/s 2024-08-17T17:39:33.708 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/5.ZeroOverlap (12295 ms) 2024-08-17T17:39:34.714 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 7 tests from EncryptedFlattenTest/5 (89287 ms total) 2024-08-17T17:39:34.715 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:39:34.715 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 7 tests from EncryptedFlattenTest/6, where TypeParam = LUKS2UnderLUKS2 2024-08-17T17:39:34.744 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/6.Simple 2024-08-17T17:39:34.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:34 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:39:34.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:34 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:39:34.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:34 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:39:34.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:34 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:39:34.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:34 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:39:34.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:34 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:39:34.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:34 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:39:34.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:34 smithi165 ceph-mon[105517]: pgmap v1334: 105 pgs: 105 active+clean; 544 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.9 MiB/s wr, 51 op/s 2024-08-17T17:39:34.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:39:34.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:39:34.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:39:34.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:39:34.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:39:34.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:39:34.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:39:34.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:34 smithi136 ceph-mon[109861]: pgmap v1334: 105 pgs: 105 active+clean; 544 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.9 MiB/s wr, 51 op/s 2024-08-17T17:39:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:39:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:39:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:39:34.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:39:34.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:39:34.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:39:34.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:39:34.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:34 smithi136 ceph-mon[115078]: pgmap v1334: 105 pgs: 105 active+clean; 544 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.9 MiB/s wr, 51 op/s 2024-08-17T17:39:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:35 smithi165 ceph-mon[105517]: osdmap e973: 8 total, 8 up, 8 in 2024-08-17T17:39:36.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:35 smithi136 ceph-mon[109861]: osdmap e973: 8 total, 8 up, 8 in 2024-08-17T17:39:36.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:35 smithi136 ceph-mon[115078]: osdmap e973: 8 total, 8 up, 8 in 2024-08-17T17:39:36.733 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:39:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:36 smithi165 ceph-mon[105517]: osdmap e974: 8 total, 8 up, 8 in 2024-08-17T17:39:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:36 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1654591035' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-45","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:39:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:39:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:36 smithi165 ceph-mon[105517]: pgmap v1337: 105 pgs: 6 creating+peering, 26 unknown, 73 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:39:37.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:36 smithi136 ceph-mon[109861]: osdmap e974: 8 total, 8 up, 8 in 2024-08-17T17:39:37.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:36 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1654591035' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-45","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:39:37.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:39:37.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:36 smithi136 ceph-mon[109861]: pgmap v1337: 105 pgs: 6 creating+peering, 26 unknown, 73 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:39:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:36 smithi136 ceph-mon[115078]: osdmap e974: 8 total, 8 up, 8 in 2024-08-17T17:39:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:36 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1654591035' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-45","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:39:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:39:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:36 smithi136 ceph-mon[115078]: pgmap v1337: 105 pgs: 6 creating+peering, 26 unknown, 73 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:39:37.723 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:39:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:39:37] "GET /metrics HTTP/1.1" 200 36416 "" "Prometheus/2.51.0" 2024-08-17T17:39:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:37 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:39:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:37 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1654591035' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:39:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:37 smithi165 ceph-mon[105517]: osdmap e975: 8 total, 8 up, 8 in 2024-08-17T17:39:38.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:37 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:39:38.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:37 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1654591035' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:39:38.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:37 smithi136 ceph-mon[109861]: osdmap e975: 8 total, 8 up, 8 in 2024-08-17T17:39:38.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:37 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:39:38.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:37 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1654591035' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:39:38.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:37 smithi136 ceph-mon[115078]: osdmap e975: 8 total, 8 up, 8 in 2024-08-17T17:39:39.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:38 smithi165 ceph-mon[105517]: osdmap e976: 8 total, 8 up, 8 in 2024-08-17T17:39:39.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:38 smithi165 ceph-mon[105517]: pgmap v1340: 105 pgs: 6 creating+peering, 8 unknown, 91 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:39:39.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:38 smithi136 ceph-mon[109861]: osdmap e976: 8 total, 8 up, 8 in 2024-08-17T17:39:39.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:38 smithi136 ceph-mon[109861]: pgmap v1340: 105 pgs: 6 creating+peering, 8 unknown, 91 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:39:39.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:38 smithi136 ceph-mon[115078]: osdmap e976: 8 total, 8 up, 8 in 2024-08-17T17:39:39.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:38 smithi136 ceph-mon[115078]: pgmap v1340: 105 pgs: 6 creating+peering, 8 unknown, 91 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:39:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:39 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:39:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:39 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T17:39:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:39 smithi165 ceph-mon[105517]: osdmap e977: 8 total, 8 up, 8 in 2024-08-17T17:39:40.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:39 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:39:40.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:39 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T17:39:40.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:39 smithi136 ceph-mon[109861]: osdmap e977: 8 total, 8 up, 8 in 2024-08-17T17:39:40.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:39 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:39:40.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:39 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T17:39:40.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:39 smithi136 ceph-mon[115078]: osdmap e977: 8 total, 8 up, 8 in 2024-08-17T17:39:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:40 smithi165 ceph-mon[105517]: pgmap v1342: 105 pgs: 6 creating+peering, 8 unknown, 91 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:39:41.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:40 smithi136 ceph-mon[109861]: pgmap v1342: 105 pgs: 6 creating+peering, 8 unknown, 91 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:39:41.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:40 smithi136 ceph-mon[115078]: pgmap v1342: 105 pgs: 6 creating+peering, 8 unknown, 91 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:39:43.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:43 smithi136 ceph-mon[109861]: pgmap v1343: 105 pgs: 105 active+clean; 582 KiB data, 665 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-08-17T17:39:43.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:43 smithi136 ceph-mon[115078]: pgmap v1343: 105 pgs: 105 active+clean; 582 KiB data, 665 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-08-17T17:39:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:43 smithi165 ceph-mon[105517]: pgmap v1343: 105 pgs: 105 active+clean; 582 KiB data, 665 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-08-17T17:39:45.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:45 smithi136 ceph-mon[109861]: osdmap e978: 8 total, 8 up, 8 in 2024-08-17T17:39:45.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:45 smithi136 ceph-mon[109861]: pgmap v1345: 105 pgs: 105 active+clean; 582 KiB data, 665 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-08-17T17:39:45.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:45 smithi136 ceph-mon[115078]: osdmap e978: 8 total, 8 up, 8 in 2024-08-17T17:39:45.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:45 smithi136 ceph-mon[115078]: pgmap v1345: 105 pgs: 105 active+clean; 582 KiB data, 665 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-08-17T17:39:45.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:45 smithi165 ceph-mon[105517]: osdmap e978: 8 total, 8 up, 8 in 2024-08-17T17:39:45.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:45 smithi165 ceph-mon[105517]: pgmap v1345: 105 pgs: 105 active+clean; 582 KiB data, 665 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-08-17T17:39:46.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:46 smithi136 ceph-mon[109861]: osdmap e979: 8 total, 8 up, 8 in 2024-08-17T17:39:46.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:46 smithi136 ceph-mon[115078]: osdmap e979: 8 total, 8 up, 8 in 2024-08-17T17:39:46.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:46 smithi165 ceph-mon[105517]: osdmap e979: 8 total, 8 up, 8 in 2024-08-17T17:39:47.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:47 smithi136 ceph-mon[109861]: pgmap v1347: 105 pgs: 105 active+clean; 9.6 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 1.2 MiB/s wr, 49 op/s 2024-08-17T17:39:47.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:47 smithi136 ceph-mon[115078]: pgmap v1347: 105 pgs: 105 active+clean; 9.6 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 1.2 MiB/s wr, 49 op/s 2024-08-17T17:39:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:47 smithi165 ceph-mon[105517]: pgmap v1347: 105 pgs: 105 active+clean; 9.6 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 1.2 MiB/s wr, 49 op/s 2024-08-17T17:39:47.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:39:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:39:47] "GET /metrics HTTP/1.1" 200 36416 "" "Prometheus/2.51.0" 2024-08-17T17:39:49.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:49 smithi136 ceph-mon[109861]: pgmap v1348: 105 pgs: 105 active+clean; 42 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 36 KiB/s rd, 5.1 MiB/s wr, 50 op/s 2024-08-17T17:39:49.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:49 smithi136 ceph-mon[115078]: pgmap v1348: 105 pgs: 105 active+clean; 42 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 36 KiB/s rd, 5.1 MiB/s wr, 50 op/s 2024-08-17T17:39:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:49 smithi165 ceph-mon[105517]: pgmap v1348: 105 pgs: 105 active+clean; 42 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 36 KiB/s rd, 5.1 MiB/s wr, 50 op/s 2024-08-17T17:39:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:51 smithi136 ceph-mon[109861]: pgmap v1349: 105 pgs: 105 active+clean; 42 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 32 KiB/s rd, 5.1 MiB/s wr, 44 op/s 2024-08-17T17:39:51.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:39:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:51 smithi136 ceph-mon[115078]: pgmap v1349: 105 pgs: 105 active+clean; 42 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 32 KiB/s rd, 5.1 MiB/s wr, 44 op/s 2024-08-17T17:39:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:39:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:51 smithi165 ceph-mon[105517]: pgmap v1349: 105 pgs: 105 active+clean; 42 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 32 KiB/s rd, 5.1 MiB/s wr, 44 op/s 2024-08-17T17:39:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:39:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:53 smithi136 ceph-mon[109861]: pgmap v1350: 105 pgs: 105 active+clean; 58 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 66 KiB/s rd, 6.9 MiB/s wr, 43 op/s 2024-08-17T17:39:53.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:53 smithi136 ceph-mon[115078]: pgmap v1350: 105 pgs: 105 active+clean; 58 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 66 KiB/s rd, 6.9 MiB/s wr, 43 op/s 2024-08-17T17:39:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:53 smithi165 ceph-mon[105517]: pgmap v1350: 105 pgs: 105 active+clean; 58 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 66 KiB/s rd, 6.9 MiB/s wr, 43 op/s 2024-08-17T17:39:55.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:55 smithi136 ceph-mon[109861]: pgmap v1351: 105 pgs: 105 active+clean; 58 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 5.7 MiB/s wr, 36 op/s 2024-08-17T17:39:55.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:55 smithi136 ceph-mon[115078]: pgmap v1351: 105 pgs: 105 active+clean; 58 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 5.7 MiB/s wr, 36 op/s 2024-08-17T17:39:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:55 smithi165 ceph-mon[105517]: pgmap v1351: 105 pgs: 105 active+clean; 58 MiB data, 820 MiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 5.7 MiB/s wr, 36 op/s 2024-08-17T17:39:56.734 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/6.Simple (21989 ms) 2024-08-17T17:39:56.734 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/6.Grow 2024-08-17T17:39:56.734 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:39:57.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:57 smithi136 ceph-mon[109861]: pgmap v1352: 105 pgs: 105 active+clean; 62 MiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.4 MiB/s wr, 52 op/s 2024-08-17T17:39:57.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:57 smithi136 ceph-mon[115078]: pgmap v1352: 105 pgs: 105 active+clean; 62 MiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.4 MiB/s wr, 52 op/s 2024-08-17T17:39:57.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:39:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:39:57] "GET /metrics HTTP/1.1" 200 36467 "" "Prometheus/2.51.0" 2024-08-17T17:39:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:57 smithi165 ceph-mon[105517]: pgmap v1352: 105 pgs: 105 active+clean; 62 MiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.4 MiB/s wr, 52 op/s 2024-08-17T17:39:59.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:39:59 smithi136 ceph-mon[109861]: pgmap v1353: 105 pgs: 105 active+clean; 83 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 7.4 MiB/s wr, 30 op/s 2024-08-17T17:39:59.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:39:59 smithi136 ceph-mon[115078]: pgmap v1353: 105 pgs: 105 active+clean; 83 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 7.4 MiB/s wr, 30 op/s 2024-08-17T17:39:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:39:59 smithi165 ceph-mon[105517]: pgmap v1353: 105 pgs: 105 active+clean; 83 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 7.4 MiB/s wr, 30 op/s 2024-08-17T17:40:00.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:00 smithi136 ceph-mon[109861]: overall HEALTH_OK 2024-08-17T17:40:00.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:00 smithi136 ceph-mon[115078]: overall HEALTH_OK 2024-08-17T17:40:00.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:00 smithi165 ceph-mon[105517]: overall HEALTH_OK 2024-08-17T17:40:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:01 smithi136 ceph-mon[109861]: pgmap v1354: 105 pgs: 105 active+clean; 83 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 4.8 MiB/s wr, 28 op/s 2024-08-17T17:40:01.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:01 smithi136 ceph-mon[115078]: pgmap v1354: 105 pgs: 105 active+clean; 83 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 4.8 MiB/s wr, 28 op/s 2024-08-17T17:40:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:01 smithi165 ceph-mon[105517]: pgmap v1354: 105 pgs: 105 active+clean; 83 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 4.8 MiB/s wr, 28 op/s 2024-08-17T17:40:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:03 smithi136 ceph-mon[109861]: pgmap v1355: 105 pgs: 105 active+clean; 83 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 32 op/s 2024-08-17T17:40:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:03 smithi136 ceph-mon[109861]: osdmap e980: 8 total, 8 up, 8 in 2024-08-17T17:40:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:03 smithi136 ceph-mon[115078]: pgmap v1355: 105 pgs: 105 active+clean; 83 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 32 op/s 2024-08-17T17:40:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:03 smithi136 ceph-mon[115078]: osdmap e980: 8 total, 8 up, 8 in 2024-08-17T17:40:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:03 smithi165 ceph-mon[105517]: pgmap v1355: 105 pgs: 105 active+clean; 83 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 32 op/s 2024-08-17T17:40:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:03 smithi165 ceph-mon[105517]: osdmap e980: 8 total, 8 up, 8 in 2024-08-17T17:40:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:05 smithi136 ceph-mon[109861]: pgmap v1357: 105 pgs: 105 active+clean; 83 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 38 op/s 2024-08-17T17:40:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:05 smithi136 ceph-mon[115078]: pgmap v1357: 105 pgs: 105 active+clean; 83 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 38 op/s 2024-08-17T17:40:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:05 smithi165 ceph-mon[105517]: pgmap v1357: 105 pgs: 105 active+clean; 83 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 38 op/s 2024-08-17T17:40:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:40:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:40:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:40:07.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:07 smithi165 ceph-mon[105517]: pgmap v1358: 105 pgs: 105 active+clean; 95 MiB data, 968 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 4.9 MiB/s wr, 23 op/s 2024-08-17T17:40:07.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:07 smithi136 ceph-mon[109861]: pgmap v1358: 105 pgs: 105 active+clean; 95 MiB data, 968 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 4.9 MiB/s wr, 23 op/s 2024-08-17T17:40:07.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:07 smithi136 ceph-mon[115078]: pgmap v1358: 105 pgs: 105 active+clean; 95 MiB data, 968 MiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 4.9 MiB/s wr, 23 op/s 2024-08-17T17:40:07.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:40:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:40:07] "GET /metrics HTTP/1.1" 200 36470 "" "Prometheus/2.51.0" 2024-08-17T17:40:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:09 smithi165 ceph-mon[105517]: pgmap v1359: 105 pgs: 105 active+clean; 124 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 436 KiB/s rd, 4.1 MiB/s wr, 36 op/s 2024-08-17T17:40:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:09 smithi136 ceph-mon[109861]: pgmap v1359: 105 pgs: 105 active+clean; 124 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 436 KiB/s rd, 4.1 MiB/s wr, 36 op/s 2024-08-17T17:40:09.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:09 smithi136 ceph-mon[115078]: pgmap v1359: 105 pgs: 105 active+clean; 124 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 436 KiB/s rd, 4.1 MiB/s wr, 36 op/s 2024-08-17T17:40:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:10 smithi165 ceph-mon[105517]: pgmap v1360: 105 pgs: 105 active+clean; 124 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 436 KiB/s rd, 4.1 MiB/s wr, 36 op/s 2024-08-17T17:40:10.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:10 smithi136 ceph-mon[109861]: pgmap v1360: 105 pgs: 105 active+clean; 124 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 436 KiB/s rd, 4.1 MiB/s wr, 36 op/s 2024-08-17T17:40:10.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:10 smithi136 ceph-mon[115078]: pgmap v1360: 105 pgs: 105 active+clean; 124 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 436 KiB/s rd, 4.1 MiB/s wr, 36 op/s 2024-08-17T17:40:13.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:13 smithi136 ceph-mon[109861]: pgmap v1361: 105 pgs: 105 active+clean; 140 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 6.5 MiB/s wr, 43 op/s 2024-08-17T17:40:13.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:13 smithi136 ceph-mon[115078]: pgmap v1361: 105 pgs: 105 active+clean; 140 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 6.5 MiB/s wr, 43 op/s 2024-08-17T17:40:13.714 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/6.Grow (16981 ms) 2024-08-17T17:40:13.714 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/6.Shrink 2024-08-17T17:40:13.714 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:40:13.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:13 smithi165 ceph-mon[105517]: pgmap v1361: 105 pgs: 105 active+clean; 140 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.5 MiB/s rd, 6.5 MiB/s wr, 43 op/s 2024-08-17T17:40:15.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:15 smithi136 ceph-mon[109861]: pgmap v1362: 105 pgs: 105 active+clean; 140 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 5.5 MiB/s wr, 36 op/s 2024-08-17T17:40:15.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:15 smithi136 ceph-mon[115078]: pgmap v1362: 105 pgs: 105 active+clean; 140 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 5.5 MiB/s wr, 36 op/s 2024-08-17T17:40:15.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:15 smithi165 ceph-mon[105517]: pgmap v1362: 105 pgs: 105 active+clean; 140 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.9 MiB/s rd, 5.5 MiB/s wr, 36 op/s 2024-08-17T17:40:17.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:17 smithi136 ceph-mon[109861]: pgmap v1363: 105 pgs: 105 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 8.2 MiB/s wr, 52 op/s 2024-08-17T17:40:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:17 smithi136 ceph-mon[115078]: pgmap v1363: 105 pgs: 105 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 8.2 MiB/s wr, 52 op/s 2024-08-17T17:40:17.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:40:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:40:17] "GET /metrics HTTP/1.1" 200 36470 "" "Prometheus/2.51.0" 2024-08-17T17:40:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:17 smithi165 ceph-mon[105517]: pgmap v1363: 105 pgs: 105 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 8.2 MiB/s wr, 52 op/s 2024-08-17T17:40:19.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:19 smithi136 ceph-mon[109861]: pgmap v1364: 105 pgs: 105 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 7.2 MiB/s wr, 49 op/s 2024-08-17T17:40:19.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:19 smithi136 ceph-mon[115078]: pgmap v1364: 105 pgs: 105 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 7.2 MiB/s wr, 49 op/s 2024-08-17T17:40:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:19 smithi165 ceph-mon[105517]: pgmap v1364: 105 pgs: 105 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 7.2 MiB/s wr, 49 op/s 2024-08-17T17:40:20.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:20 smithi136 ceph-mon[109861]: osdmap e981: 8 total, 8 up, 8 in 2024-08-17T17:40:20.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:20 smithi136 ceph-mon[115078]: osdmap e981: 8 total, 8 up, 8 in 2024-08-17T17:40:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:20 smithi165 ceph-mon[105517]: osdmap e981: 8 total, 8 up, 8 in 2024-08-17T17:40:21.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:21 smithi136 ceph-mon[109861]: pgmap v1366: 105 pgs: 105 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 5.7 MiB/s wr, 36 op/s 2024-08-17T17:40:21.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:40:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:21 smithi136 ceph-mon[115078]: pgmap v1366: 105 pgs: 105 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 5.7 MiB/s wr, 36 op/s 2024-08-17T17:40:21.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:40:21.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:21 smithi165 ceph-mon[105517]: pgmap v1366: 105 pgs: 105 active+clean; 165 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 5.7 MiB/s wr, 36 op/s 2024-08-17T17:40:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:40:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:23 smithi165 ceph-mon[105517]: pgmap v1367: 105 pgs: 105 active+clean; 206 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.3 MiB/s rd, 7.4 MiB/s wr, 59 op/s 2024-08-17T17:40:23.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:23 smithi136 ceph-mon[109861]: pgmap v1367: 105 pgs: 105 active+clean; 206 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.3 MiB/s rd, 7.4 MiB/s wr, 59 op/s 2024-08-17T17:40:23.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:23 smithi136 ceph-mon[115078]: pgmap v1367: 105 pgs: 105 active+clean; 206 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.3 MiB/s rd, 7.4 MiB/s wr, 59 op/s 2024-08-17T17:40:25.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:25 smithi165 ceph-mon[105517]: pgmap v1368: 105 pgs: 105 active+clean; 206 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.3 MiB/s rd, 7.4 MiB/s wr, 59 op/s 2024-08-17T17:40:25.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:25 smithi136 ceph-mon[109861]: pgmap v1368: 105 pgs: 105 active+clean; 206 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.3 MiB/s rd, 7.4 MiB/s wr, 59 op/s 2024-08-17T17:40:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:25 smithi136 ceph-mon[115078]: pgmap v1368: 105 pgs: 105 active+clean; 206 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.3 MiB/s rd, 7.4 MiB/s wr, 59 op/s 2024-08-17T17:40:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:27 smithi165 ceph-mon[105517]: pgmap v1369: 105 pgs: 105 active+clean; 210 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 4.5 MiB/s wr, 43 op/s 2024-08-17T17:40:27.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:27 smithi136 ceph-mon[109861]: pgmap v1369: 105 pgs: 105 active+clean; 210 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 4.5 MiB/s wr, 43 op/s 2024-08-17T17:40:27.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:27 smithi136 ceph-mon[115078]: pgmap v1369: 105 pgs: 105 active+clean; 210 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 4.5 MiB/s wr, 43 op/s 2024-08-17T17:40:27.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:40:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:40:27] "GET /metrics HTTP/1.1" 200 36481 "" "Prometheus/2.51.0" 2024-08-17T17:40:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:29 smithi165 ceph-mon[105517]: pgmap v1370: 105 pgs: 105 active+clean; 222 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 62 KiB/s rd, 5.7 MiB/s wr, 46 op/s 2024-08-17T17:40:29.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:29 smithi136 ceph-mon[109861]: pgmap v1370: 105 pgs: 105 active+clean; 222 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 62 KiB/s rd, 5.7 MiB/s wr, 46 op/s 2024-08-17T17:40:29.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:29 smithi136 ceph-mon[115078]: pgmap v1370: 105 pgs: 105 active+clean; 222 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 62 KiB/s rd, 5.7 MiB/s wr, 46 op/s 2024-08-17T17:40:30.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:30 smithi165 ceph-mon[105517]: pgmap v1371: 105 pgs: 105 active+clean; 222 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 57 KiB/s rd, 5.2 MiB/s wr, 42 op/s 2024-08-17T17:40:30.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:30 smithi136 ceph-mon[109861]: pgmap v1371: 105 pgs: 105 active+clean; 222 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 57 KiB/s rd, 5.2 MiB/s wr, 42 op/s 2024-08-17T17:40:30.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:30 smithi136 ceph-mon[115078]: pgmap v1371: 105 pgs: 105 active+clean; 222 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 57 KiB/s rd, 5.2 MiB/s wr, 42 op/s 2024-08-17T17:40:31.546 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/6.Shrink (17832 ms) 2024-08-17T17:40:31.546 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/6.ShrinkToOne 2024-08-17T17:40:31.546 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:40:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:33 smithi136 ceph-mon[109861]: pgmap v1372: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 8.2 MiB/s wr, 57 op/s 2024-08-17T17:40:33.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:33 smithi136 ceph-mon[115078]: pgmap v1372: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 8.2 MiB/s wr, 57 op/s 2024-08-17T17:40:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:33 smithi165 ceph-mon[105517]: pgmap v1372: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 8.2 MiB/s wr, 57 op/s 2024-08-17T17:40:34.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:40:34.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:40:34.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:34 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:40:35.442 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:35 smithi136 ceph-mon[115078]: pgmap v1373: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 4.8 MiB/s wr, 28 op/s 2024-08-17T17:40:35.442 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:35 smithi136 ceph-mon[109861]: pgmap v1373: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 4.8 MiB/s wr, 28 op/s 2024-08-17T17:40:35.692 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:35 smithi165 ceph-mon[105517]: pgmap v1373: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.4 MiB/s rd, 4.8 MiB/s wr, 28 op/s 2024-08-17T17:40:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:40:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:40:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:40:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:37 smithi165 ceph-mon[105517]: pgmap v1374: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 31 op/s 2024-08-17T17:40:37.726 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:37 smithi136 ceph-mon[109861]: pgmap v1374: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 31 op/s 2024-08-17T17:40:37.727 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:37 smithi136 ceph-mon[115078]: pgmap v1374: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 4.8 MiB/s wr, 31 op/s 2024-08-17T17:40:37.727 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:40:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:40:37] "GET /metrics HTTP/1.1" 200 36485 "" "Prometheus/2.51.0" 2024-08-17T17:40:38.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:38 smithi165 ceph-mon[105517]: osdmap e982: 8 total, 8 up, 8 in 2024-08-17T17:40:38.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:38 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:40:38.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:38 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:40:38.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:38 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:40:38.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:38 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:40:38.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:38 smithi136 ceph-mon[109861]: osdmap e982: 8 total, 8 up, 8 in 2024-08-17T17:40:38.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:40:38.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:40:38.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:40:38.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:40:38.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:38 smithi136 ceph-mon[115078]: osdmap e982: 8 total, 8 up, 8 in 2024-08-17T17:40:38.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:40:38.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:40:38.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:40:38.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:40:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:39 smithi165 ceph-mon[105517]: pgmap v1376: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 26 op/s 2024-08-17T17:40:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:39 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:40:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:39 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:40:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:39 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:40:39.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:39 smithi136 ceph-mon[109861]: pgmap v1376: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 26 op/s 2024-08-17T17:40:39.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:39 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:40:39.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:39 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:40:39.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:39 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:40:39.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:39 smithi136 ceph-mon[115078]: pgmap v1376: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 26 op/s 2024-08-17T17:40:39.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:39 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:40:39.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:39 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:40:39.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:39 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:40:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:41 smithi165 ceph-mon[105517]: pgmap v1377: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 26 op/s 2024-08-17T17:40:41.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:41 smithi136 ceph-mon[109861]: pgmap v1377: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 26 op/s 2024-08-17T17:40:41.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:41 smithi136 ceph-mon[115078]: pgmap v1377: 105 pgs: 105 active+clean; 247 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 26 op/s 2024-08-17T17:40:42.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:42 smithi136 ceph-mon[109861]: pgmap v1378: 105 pgs: 105 active+clean; 288 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 438 KiB/s rd, 4.1 MiB/s wr, 38 op/s 2024-08-17T17:40:42.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:42 smithi136 ceph-mon[115078]: pgmap v1378: 105 pgs: 105 active+clean; 288 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 438 KiB/s rd, 4.1 MiB/s wr, 38 op/s 2024-08-17T17:40:43.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:42 smithi165 ceph-mon[105517]: pgmap v1378: 105 pgs: 105 active+clean; 288 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 438 KiB/s rd, 4.1 MiB/s wr, 38 op/s 2024-08-17T17:40:45.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:45 smithi136 ceph-mon[109861]: pgmap v1379: 105 pgs: 105 active+clean; 288 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 438 KiB/s rd, 4.1 MiB/s wr, 38 op/s 2024-08-17T17:40:45.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:45 smithi136 ceph-mon[115078]: pgmap v1379: 105 pgs: 105 active+clean; 288 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 438 KiB/s rd, 4.1 MiB/s wr, 38 op/s 2024-08-17T17:40:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:45 smithi165 ceph-mon[105517]: pgmap v1379: 105 pgs: 105 active+clean; 288 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 438 KiB/s rd, 4.1 MiB/s wr, 38 op/s 2024-08-17T17:40:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:47 smithi136 ceph-mon[109861]: pgmap v1380: 105 pgs: 105 active+clean; 296 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 4.9 MiB/s wr, 45 op/s 2024-08-17T17:40:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:47 smithi136 ceph-mon[115078]: pgmap v1380: 105 pgs: 105 active+clean; 296 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 4.9 MiB/s wr, 45 op/s 2024-08-17T17:40:47.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:40:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:40:47] "GET /metrics HTTP/1.1" 200 36485 "" "Prometheus/2.51.0" 2024-08-17T17:40:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:47 smithi165 ceph-mon[105517]: pgmap v1380: 105 pgs: 105 active+clean; 296 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 4.9 MiB/s wr, 45 op/s 2024-08-17T17:40:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:49 smithi136 ceph-mon[109861]: pgmap v1381: 105 pgs: 105 active+clean; 304 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 83 KiB/s rd, 5.2 MiB/s wr, 42 op/s 2024-08-17T17:40:49.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:49 smithi136 ceph-mon[115078]: pgmap v1381: 105 pgs: 105 active+clean; 304 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 83 KiB/s rd, 5.2 MiB/s wr, 42 op/s 2024-08-17T17:40:49.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:49 smithi165 ceph-mon[105517]: pgmap v1381: 105 pgs: 105 active+clean; 304 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 83 KiB/s rd, 5.2 MiB/s wr, 42 op/s 2024-08-17T17:40:49.877 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/6.ShrinkToOne (18331 ms) 2024-08-17T17:40:49.877 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/6.ShrinkToOneAfterSnapshot 2024-08-17T17:40:49.877 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:40:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:51 smithi136 ceph-mon[109861]: pgmap v1382: 105 pgs: 105 active+clean; 304 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 76 KiB/s rd, 4.8 MiB/s wr, 39 op/s 2024-08-17T17:40:51.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:40:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:51 smithi136 ceph-mon[115078]: pgmap v1382: 105 pgs: 105 active+clean; 304 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 76 KiB/s rd, 4.8 MiB/s wr, 39 op/s 2024-08-17T17:40:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:40:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:51 smithi165 ceph-mon[105517]: pgmap v1382: 105 pgs: 105 active+clean; 304 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 76 KiB/s rd, 4.8 MiB/s wr, 39 op/s 2024-08-17T17:40:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:40:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:53 smithi136 ceph-mon[109861]: pgmap v1383: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 6.8 MiB/s wr, 53 op/s 2024-08-17T17:40:53.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:53 smithi136 ceph-mon[115078]: pgmap v1383: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 6.8 MiB/s wr, 53 op/s 2024-08-17T17:40:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:53 smithi165 ceph-mon[105517]: pgmap v1383: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 6.8 MiB/s wr, 53 op/s 2024-08-17T17:40:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:55 smithi165 ceph-mon[105517]: pgmap v1384: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 23 op/s 2024-08-17T17:40:55.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:55 smithi136 ceph-mon[109861]: pgmap v1384: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 23 op/s 2024-08-17T17:40:55.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:55 smithi136 ceph-mon[115078]: pgmap v1384: 105 pgs: 105 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.8 MiB/s rd, 3.3 MiB/s wr, 23 op/s 2024-08-17T17:40:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:56 smithi165 ceph-mon[105517]: osdmap e983: 8 total, 8 up, 8 in 2024-08-17T17:40:56.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:56 smithi136 ceph-mon[115078]: osdmap e983: 8 total, 8 up, 8 in 2024-08-17T17:40:56.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:56 smithi136 ceph-mon[109861]: osdmap e983: 8 total, 8 up, 8 in 2024-08-17T17:40:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:57 smithi165 ceph-mon[105517]: pgmap v1386: 105 pgs: 105 active+clean; 329 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 42 op/s 2024-08-17T17:40:57.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:57 smithi136 ceph-mon[109861]: pgmap v1386: 105 pgs: 105 active+clean; 329 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 42 op/s 2024-08-17T17:40:57.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:57 smithi136 ceph-mon[115078]: pgmap v1386: 105 pgs: 105 active+clean; 329 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 5.3 MiB/s wr, 42 op/s 2024-08-17T17:40:57.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:40:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:40:57] "GET /metrics HTTP/1.1" 200 36496 "" "Prometheus/2.51.0" 2024-08-17T17:40:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:40:59 smithi165 ceph-mon[105517]: pgmap v1387: 105 pgs: 105 active+clean; 349 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 52 op/s 2024-08-17T17:40:59.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:40:59 smithi136 ceph-mon[109861]: pgmap v1387: 105 pgs: 105 active+clean; 349 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 52 op/s 2024-08-17T17:40:59.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:40:59 smithi136 ceph-mon[115078]: pgmap v1387: 105 pgs: 105 active+clean; 349 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 52 op/s 2024-08-17T17:41:00.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:00 smithi165 ceph-mon[105517]: pgmap v1388: 105 pgs: 105 active+clean; 349 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 52 op/s 2024-08-17T17:41:00.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:00 smithi136 ceph-mon[109861]: pgmap v1388: 105 pgs: 105 active+clean; 349 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 52 op/s 2024-08-17T17:41:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:00 smithi136 ceph-mon[115078]: pgmap v1388: 105 pgs: 105 active+clean; 349 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.1 MiB/s rd, 6.5 MiB/s wr, 52 op/s 2024-08-17T17:41:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:03 smithi136 ceph-mon[109861]: pgmap v1389: 105 pgs: 105 active+clean; 365 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 5.7 MiB/s wr, 36 op/s 2024-08-17T17:41:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:03 smithi136 ceph-mon[115078]: pgmap v1389: 105 pgs: 105 active+clean; 365 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 5.7 MiB/s wr, 36 op/s 2024-08-17T17:41:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:03 smithi165 ceph-mon[105517]: pgmap v1389: 105 pgs: 105 active+clean; 365 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 5.7 MiB/s wr, 36 op/s 2024-08-17T17:41:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:05 smithi136 ceph-mon[109861]: pgmap v1390: 105 pgs: 105 active+clean; 365 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 5.7 MiB/s wr, 36 op/s 2024-08-17T17:41:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:05 smithi136 ceph-mon[115078]: pgmap v1390: 105 pgs: 105 active+clean; 365 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 5.7 MiB/s wr, 36 op/s 2024-08-17T17:41:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:05 smithi165 ceph-mon[105517]: pgmap v1390: 105 pgs: 105 active+clean; 365 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 5.7 MiB/s wr, 36 op/s 2024-08-17T17:41:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:06 smithi165 ceph-mon[105517]: osdmap e984: 8 total, 8 up, 8 in 2024-08-17T17:41:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:41:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:06 smithi136 ceph-mon[109861]: osdmap e984: 8 total, 8 up, 8 in 2024-08-17T17:41:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:41:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:06 smithi136 ceph-mon[115078]: osdmap e984: 8 total, 8 up, 8 in 2024-08-17T17:41:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:41:07.618 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/6.ShrinkToOneAfterSnapshot (17740 ms) 2024-08-17T17:41:07.618 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/6.MinOverlap 2024-08-17T17:41:07.618 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:41:07.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:07 smithi165 ceph-mon[105517]: pgmap v1392: 105 pgs: 105 active+clean; 386 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 972 KiB/s rd, 5.7 MiB/s wr, 33 op/s 2024-08-17T17:41:07.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:07 smithi136 ceph-mon[109861]: pgmap v1392: 105 pgs: 105 active+clean; 386 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 972 KiB/s rd, 5.7 MiB/s wr, 33 op/s 2024-08-17T17:41:07.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:07 smithi136 ceph-mon[115078]: pgmap v1392: 105 pgs: 105 active+clean; 386 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 972 KiB/s rd, 5.7 MiB/s wr, 33 op/s 2024-08-17T17:41:07.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:41:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:41:07] "GET /metrics HTTP/1.1" 200 36498 "" "Prometheus/2.51.0" 2024-08-17T17:41:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:09 smithi165 ceph-mon[105517]: pgmap v1393: 105 pgs: 105 active+clean; 426 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 7.7 MiB/s wr, 31 op/s 2024-08-17T17:41:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:09 smithi136 ceph-mon[109861]: pgmap v1393: 105 pgs: 105 active+clean; 426 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 7.7 MiB/s wr, 31 op/s 2024-08-17T17:41:09.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:09 smithi136 ceph-mon[115078]: pgmap v1393: 105 pgs: 105 active+clean; 426 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 7.7 MiB/s wr, 31 op/s 2024-08-17T17:41:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:11 smithi165 ceph-mon[105517]: pgmap v1394: 105 pgs: 105 active+clean; 426 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 7.7 MiB/s wr, 31 op/s 2024-08-17T17:41:11.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:11 smithi136 ceph-mon[109861]: pgmap v1394: 105 pgs: 105 active+clean; 426 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 7.7 MiB/s wr, 31 op/s 2024-08-17T17:41:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:11 smithi136 ceph-mon[115078]: pgmap v1394: 105 pgs: 105 active+clean; 426 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 7.7 MiB/s wr, 31 op/s 2024-08-17T17:41:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:13 smithi165 ceph-mon[105517]: pgmap v1395: 105 pgs: 105 active+clean; 434 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 6.9 MiB/s wr, 48 op/s 2024-08-17T17:41:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:13 smithi136 ceph-mon[109861]: pgmap v1395: 105 pgs: 105 active+clean; 434 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 6.9 MiB/s wr, 48 op/s 2024-08-17T17:41:13.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:13 smithi136 ceph-mon[115078]: pgmap v1395: 105 pgs: 105 active+clean; 434 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 6.9 MiB/s wr, 48 op/s 2024-08-17T17:41:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:14 smithi165 ceph-mon[105517]: osdmap e985: 8 total, 8 up, 8 in 2024-08-17T17:41:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:14 smithi165 ceph-mon[105517]: pgmap v1397: 105 pgs: 105 active+clean; 434 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 7.7 MiB/s wr, 54 op/s 2024-08-17T17:41:14.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:14 smithi136 ceph-mon[109861]: osdmap e985: 8 total, 8 up, 8 in 2024-08-17T17:41:14.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:14 smithi136 ceph-mon[109861]: pgmap v1397: 105 pgs: 105 active+clean; 434 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 7.7 MiB/s wr, 54 op/s 2024-08-17T17:41:14.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:14 smithi136 ceph-mon[115078]: osdmap e985: 8 total, 8 up, 8 in 2024-08-17T17:41:14.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:14 smithi136 ceph-mon[115078]: pgmap v1397: 105 pgs: 105 active+clean; 434 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 7.7 MiB/s wr, 54 op/s 2024-08-17T17:41:17.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:17 smithi136 ceph-mon[109861]: pgmap v1398: 105 pgs: 105 active+clean; 442 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 5.6 MiB/s wr, 44 op/s 2024-08-17T17:41:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:17 smithi136 ceph-mon[115078]: pgmap v1398: 105 pgs: 105 active+clean; 442 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 5.6 MiB/s wr, 44 op/s 2024-08-17T17:41:17.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:41:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:41:17] "GET /metrics HTTP/1.1" 200 36498 "" "Prometheus/2.51.0" 2024-08-17T17:41:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:17 smithi165 ceph-mon[105517]: pgmap v1398: 105 pgs: 105 active+clean; 442 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 5.6 MiB/s wr, 44 op/s 2024-08-17T17:41:19.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:19 smithi136 ceph-mon[109861]: pgmap v1399: 105 pgs: 105 active+clean; 475 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 4.9 MiB/s wr, 53 op/s 2024-08-17T17:41:19.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:19 smithi136 ceph-mon[115078]: pgmap v1399: 105 pgs: 105 active+clean; 475 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 4.9 MiB/s wr, 53 op/s 2024-08-17T17:41:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:19 smithi165 ceph-mon[105517]: pgmap v1399: 105 pgs: 105 active+clean; 475 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 4.9 MiB/s wr, 53 op/s 2024-08-17T17:41:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:21 smithi165 ceph-mon[105517]: pgmap v1400: 105 pgs: 105 active+clean; 475 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 4.9 MiB/s wr, 53 op/s 2024-08-17T17:41:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:41:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:21 smithi136 ceph-mon[109861]: pgmap v1400: 105 pgs: 105 active+clean; 475 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 4.9 MiB/s wr, 53 op/s 2024-08-17T17:41:21.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:41:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:21 smithi136 ceph-mon[115078]: pgmap v1400: 105 pgs: 105 active+clean; 475 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 4.9 MiB/s wr, 53 op/s 2024-08-17T17:41:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:41:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:23 smithi165 ceph-mon[105517]: pgmap v1401: 105 pgs: 105 active+clean; 491 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 91 KiB/s rd, 5.7 MiB/s wr, 46 op/s 2024-08-17T17:41:23.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:23 smithi136 ceph-mon[109861]: pgmap v1401: 105 pgs: 105 active+clean; 491 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 91 KiB/s rd, 5.7 MiB/s wr, 46 op/s 2024-08-17T17:41:23.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:23 smithi136 ceph-mon[115078]: pgmap v1401: 105 pgs: 105 active+clean; 491 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 91 KiB/s rd, 5.7 MiB/s wr, 46 op/s 2024-08-17T17:41:24.729 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/6.MinOverlap (17112 ms) 2024-08-17T17:41:24.729 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/6.ZeroOverlap 2024-08-17T17:41:24.729 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:41:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:25 smithi165 ceph-mon[105517]: pgmap v1402: 105 pgs: 105 active+clean; 491 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 84 KiB/s rd, 5.2 MiB/s wr, 42 op/s 2024-08-17T17:41:25.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:25 smithi136 ceph-mon[109861]: pgmap v1402: 105 pgs: 105 active+clean; 491 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 84 KiB/s rd, 5.2 MiB/s wr, 42 op/s 2024-08-17T17:41:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:25 smithi136 ceph-mon[115078]: pgmap v1402: 105 pgs: 105 active+clean; 491 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 84 KiB/s rd, 5.2 MiB/s wr, 42 op/s 2024-08-17T17:41:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:27 smithi165 ceph-mon[105517]: pgmap v1403: 105 pgs: 105 active+clean; 491 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 76 KiB/s rd, 4.8 MiB/s wr, 40 op/s 2024-08-17T17:41:27.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:27 smithi136 ceph-mon[109861]: pgmap v1403: 105 pgs: 105 active+clean; 491 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 76 KiB/s rd, 4.8 MiB/s wr, 40 op/s 2024-08-17T17:41:27.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:27 smithi136 ceph-mon[115078]: pgmap v1403: 105 pgs: 105 active+clean; 491 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 76 KiB/s rd, 4.8 MiB/s wr, 40 op/s 2024-08-17T17:41:27.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:41:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:41:27] "GET /metrics HTTP/1.1" 200 36493 "" "Prometheus/2.51.0" 2024-08-17T17:41:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:29 smithi165 ceph-mon[105517]: pgmap v1404: 105 pgs: 105 active+clean; 495 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 6.1 MiB/s wr, 40 op/s 2024-08-17T17:41:29.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:29 smithi136 ceph-mon[109861]: pgmap v1404: 105 pgs: 105 active+clean; 495 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 6.1 MiB/s wr, 40 op/s 2024-08-17T17:41:29.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:29 smithi136 ceph-mon[115078]: pgmap v1404: 105 pgs: 105 active+clean; 495 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 6.1 MiB/s wr, 40 op/s 2024-08-17T17:41:31.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:31 smithi165 ceph-mon[105517]: pgmap v1405: 105 pgs: 105 active+clean; 495 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 25 op/s 2024-08-17T17:41:31.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:31 smithi165 ceph-mon[105517]: osdmap e986: 8 total, 8 up, 8 in 2024-08-17T17:41:31.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:31 smithi136 ceph-mon[109861]: pgmap v1405: 105 pgs: 105 active+clean; 495 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 25 op/s 2024-08-17T17:41:31.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:31 smithi136 ceph-mon[109861]: osdmap e986: 8 total, 8 up, 8 in 2024-08-17T17:41:31.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:31 smithi136 ceph-mon[115078]: pgmap v1405: 105 pgs: 105 active+clean; 495 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.8 MiB/s rd, 3.3 MiB/s wr, 25 op/s 2024-08-17T17:41:31.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:31 smithi136 ceph-mon[115078]: osdmap e986: 8 total, 8 up, 8 in 2024-08-17T17:41:32.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:32 smithi165 ceph-mon[105517]: pgmap v1407: 105 pgs: 105 active+clean; 536 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 6.5 MiB/s wr, 54 op/s 2024-08-17T17:41:32.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:32 smithi136 ceph-mon[109861]: pgmap v1407: 105 pgs: 105 active+clean; 536 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 6.5 MiB/s wr, 54 op/s 2024-08-17T17:41:32.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:32 smithi136 ceph-mon[115078]: pgmap v1407: 105 pgs: 105 active+clean; 536 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 6.5 MiB/s wr, 54 op/s 2024-08-17T17:41:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:35 smithi165 ceph-mon[105517]: pgmap v1408: 105 pgs: 105 active+clean; 536 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 6.5 MiB/s wr, 54 op/s 2024-08-17T17:41:35.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:35 smithi136 ceph-mon[109861]: pgmap v1408: 105 pgs: 105 active+clean; 536 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 6.5 MiB/s wr, 54 op/s 2024-08-17T17:41:35.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:35 smithi136 ceph-mon[115078]: pgmap v1408: 105 pgs: 105 active+clean; 536 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 6.5 MiB/s wr, 54 op/s 2024-08-17T17:41:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:41:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:41:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:41:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:37 smithi165 ceph-mon[105517]: pgmap v1409: 105 pgs: 105 active+clean; 540 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 6.9 MiB/s wr, 56 op/s 2024-08-17T17:41:37.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:37 smithi136 ceph-mon[109861]: pgmap v1409: 105 pgs: 105 active+clean; 540 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 6.9 MiB/s wr, 56 op/s 2024-08-17T17:41:37.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:37 smithi136 ceph-mon[115078]: pgmap v1409: 105 pgs: 105 active+clean; 540 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 6.9 MiB/s wr, 56 op/s 2024-08-17T17:41:37.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:41:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:41:37] "GET /metrics HTTP/1.1" 200 36497 "" "Prometheus/2.51.0" 2024-08-17T17:41:39.757 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:39 smithi136 ceph-mon[109861]: pgmap v1410: 105 pgs: 105 active+clean; 548 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 31 KiB/s rd, 5.3 MiB/s wr, 43 op/s 2024-08-17T17:41:39.758 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:39 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:41:39.758 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:39 smithi136 ceph-mon[115078]: pgmap v1410: 105 pgs: 105 active+clean; 548 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 31 KiB/s rd, 5.3 MiB/s wr, 43 op/s 2024-08-17T17:41:39.758 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:39 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:41:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:39 smithi165 ceph-mon[105517]: pgmap v1410: 105 pgs: 105 active+clean; 548 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 31 KiB/s rd, 5.3 MiB/s wr, 43 op/s 2024-08-17T17:41:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:39 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:41:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:41 smithi165 ceph-mon[105517]: pgmap v1411: 105 pgs: 105 active+clean; 548 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 31 KiB/s rd, 5.3 MiB/s wr, 43 op/s 2024-08-17T17:41:41.857 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:41 smithi136 ceph-mon[109861]: pgmap v1411: 105 pgs: 105 active+clean; 548 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 31 KiB/s rd, 5.3 MiB/s wr, 43 op/s 2024-08-17T17:41:41.857 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:41 smithi136 ceph-mon[115078]: pgmap v1411: 105 pgs: 105 active+clean; 548 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 31 KiB/s rd, 5.3 MiB/s wr, 43 op/s 2024-08-17T17:41:42.038 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/6.ZeroOverlap (17309 ms) 2024-08-17T17:41:42.474 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 7 tests from EncryptedFlattenTest/6 (127294 ms total) 2024-08-17T17:41:42.475 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:41:42.475 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 7 tests from EncryptedFlattenTest/7, where TypeParam = LUKS2UnderLUKS1 2024-08-17T17:41:42.502 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/7.Simple 2024-08-17T17:41:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:43 smithi165 ceph-mon[105517]: pgmap v1412: 105 pgs: 105 active+clean; 552 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 MiB/s rd, 6.1 MiB/s wr, 49 op/s 2024-08-17T17:41:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:43 smithi165 ceph-mon[105517]: osdmap e987: 8 total, 8 up, 8 in 2024-08-17T17:41:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:43 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:41:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:43 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:41:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:43 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:41:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:43 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:41:43.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:43 smithi136 ceph-mon[109861]: pgmap v1412: 105 pgs: 105 active+clean; 552 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 MiB/s rd, 6.1 MiB/s wr, 49 op/s 2024-08-17T17:41:43.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:43 smithi136 ceph-mon[109861]: osdmap e987: 8 total, 8 up, 8 in 2024-08-17T17:41:43.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:43 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:41:43.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:43 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:41:43.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:43 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:41:43.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:43 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:41:43.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:43 smithi136 ceph-mon[115078]: pgmap v1412: 105 pgs: 105 active+clean; 552 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 MiB/s rd, 6.1 MiB/s wr, 49 op/s 2024-08-17T17:41:43.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:43 smithi136 ceph-mon[115078]: osdmap e987: 8 total, 8 up, 8 in 2024-08-17T17:41:43.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:43 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:41:43.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:43 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:41:43.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:43 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:41:43.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:43 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:41:44.494 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:41:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:44 smithi165 ceph-mon[105517]: osdmap e988: 8 total, 8 up, 8 in 2024-08-17T17:41:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:44 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1625699392' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-46","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:41:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:44 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-46","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:41:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:44 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:41:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:44 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:41:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:44 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:41:44.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:44 smithi165 ceph-mon[105517]: pgmap v1415: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:41:44.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:44 smithi136 ceph-mon[109861]: osdmap e988: 8 total, 8 up, 8 in 2024-08-17T17:41:44.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:44 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1625699392' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-46","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:41:44.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:44 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-46","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:41:44.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:44 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:41:44.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:44 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:41:44.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:44 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:41:44.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:44 smithi136 ceph-mon[109861]: pgmap v1415: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:41:44.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:44 smithi136 ceph-mon[115078]: osdmap e988: 8 total, 8 up, 8 in 2024-08-17T17:41:44.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:44 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1625699392' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-46","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:41:44.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:44 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-46","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:41:44.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:44 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:41:44.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:44 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:41:44.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:44 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:41:44.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:44 smithi136 ceph-mon[115078]: pgmap v1415: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:41:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:45 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-46","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:41:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:45 smithi165 ceph-mon[105517]: osdmap e989: 8 total, 8 up, 8 in 2024-08-17T17:41:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:45 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:41:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:45 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-46","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:41:45.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:45 smithi136 ceph-mon[109861]: osdmap e989: 8 total, 8 up, 8 in 2024-08-17T17:41:45.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:45 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:41:45.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:45 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-46","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:41:45.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:45 smithi136 ceph-mon[115078]: osdmap e989: 8 total, 8 up, 8 in 2024-08-17T17:41:45.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:45 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:41:46.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:46 smithi165 ceph-mon[105517]: osdmap e990: 8 total, 8 up, 8 in 2024-08-17T17:41:46.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:46 smithi165 ceph-mon[105517]: pgmap v1418: 105 pgs: 21 unknown, 84 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:41:46.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:46 smithi136 ceph-mon[115078]: osdmap e990: 8 total, 8 up, 8 in 2024-08-17T17:41:46.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:46 smithi136 ceph-mon[115078]: pgmap v1418: 105 pgs: 21 unknown, 84 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:41:46.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:46 smithi136 ceph-mon[109861]: osdmap e990: 8 total, 8 up, 8 in 2024-08-17T17:41:46.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:46 smithi136 ceph-mon[109861]: pgmap v1418: 105 pgs: 21 unknown, 84 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:41:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:47 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:41:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:47 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T17:41:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:47 smithi165 ceph-mon[105517]: osdmap e991: 8 total, 8 up, 8 in 2024-08-17T17:41:47.847 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:47 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:41:47.847 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:47 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T17:41:47.847 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:47 smithi136 ceph-mon[109861]: osdmap e991: 8 total, 8 up, 8 in 2024-08-17T17:41:47.848 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:41:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:41:47] "GET /metrics HTTP/1.1" 200 36497 "" "Prometheus/2.51.0" 2024-08-17T17:41:47.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:47 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:41:47.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:47 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T17:41:47.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:47 smithi136 ceph-mon[115078]: osdmap e991: 8 total, 8 up, 8 in 2024-08-17T17:41:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:48 smithi165 ceph-mon[105517]: pgmap v1420: 105 pgs: 105 active+clean; 582 KiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-08-17T17:41:48.867 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:48 smithi136 ceph-mon[115078]: pgmap v1420: 105 pgs: 105 active+clean; 582 KiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-08-17T17:41:48.867 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:48 smithi136 ceph-mon[109861]: pgmap v1420: 105 pgs: 105 active+clean; 582 KiB data, 696 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-08-17T17:41:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:51 smithi165 ceph-mon[105517]: pgmap v1421: 105 pgs: 105 active+clean; 582 KiB data, 696 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 3 op/s 2024-08-17T17:41:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:41:51.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:51 smithi136 ceph-mon[109861]: pgmap v1421: 105 pgs: 105 active+clean; 582 KiB data, 696 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 3 op/s 2024-08-17T17:41:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:41:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:51 smithi136 ceph-mon[115078]: pgmap v1421: 105 pgs: 105 active+clean; 582 KiB data, 696 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 3 op/s 2024-08-17T17:41:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:41:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:53 smithi165 ceph-mon[105517]: pgmap v1422: 105 pgs: 105 active+clean; 582 KiB data, 696 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.0 KiB/s wr, 6 op/s 2024-08-17T17:41:53.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:53 smithi136 ceph-mon[109861]: pgmap v1422: 105 pgs: 105 active+clean; 582 KiB data, 696 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.0 KiB/s wr, 6 op/s 2024-08-17T17:41:53.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:53 smithi136 ceph-mon[115078]: pgmap v1422: 105 pgs: 105 active+clean; 582 KiB data, 696 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.0 KiB/s wr, 6 op/s 2024-08-17T17:41:54.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:54 smithi165 ceph-mon[105517]: osdmap e992: 8 total, 8 up, 8 in 2024-08-17T17:41:54.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:54 smithi165 ceph-mon[105517]: osdmap e993: 8 total, 8 up, 8 in 2024-08-17T17:41:54.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:54 smithi136 ceph-mon[109861]: osdmap e992: 8 total, 8 up, 8 in 2024-08-17T17:41:54.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:54 smithi136 ceph-mon[109861]: osdmap e993: 8 total, 8 up, 8 in 2024-08-17T17:41:54.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:54 smithi136 ceph-mon[115078]: osdmap e992: 8 total, 8 up, 8 in 2024-08-17T17:41:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:54 smithi136 ceph-mon[115078]: osdmap e993: 8 total, 8 up, 8 in 2024-08-17T17:41:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:55 smithi165 ceph-mon[105517]: pgmap v1425: 105 pgs: 105 active+clean; 582 KiB data, 696 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.0 KiB/s wr, 6 op/s 2024-08-17T17:41:55.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:55 smithi136 ceph-mon[109861]: pgmap v1425: 105 pgs: 105 active+clean; 582 KiB data, 696 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.0 KiB/s wr, 6 op/s 2024-08-17T17:41:55.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:55 smithi136 ceph-mon[115078]: pgmap v1425: 105 pgs: 105 active+clean; 582 KiB data, 696 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.0 KiB/s wr, 6 op/s 2024-08-17T17:41:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:57 smithi165 ceph-mon[105517]: pgmap v1426: 105 pgs: 105 active+clean; 17 MiB data, 742 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.0 MiB/s wr, 33 op/s 2024-08-17T17:41:57.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:57 smithi136 ceph-mon[109861]: pgmap v1426: 105 pgs: 105 active+clean; 17 MiB data, 742 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.0 MiB/s wr, 33 op/s 2024-08-17T17:41:57.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:57 smithi136 ceph-mon[115078]: pgmap v1426: 105 pgs: 105 active+clean; 17 MiB data, 742 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.0 MiB/s wr, 33 op/s 2024-08-17T17:41:57.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:41:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:41:57] "GET /metrics HTTP/1.1" 200 36456 "" "Prometheus/2.51.0" 2024-08-17T17:41:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:41:58 smithi165 ceph-mon[105517]: pgmap v1427: 105 pgs: 105 active+clean; 46 MiB data, 824 MiB used, 714 GiB / 715 GiB avail; 72 KiB/s rd, 5.6 MiB/s wr, 50 op/s 2024-08-17T17:41:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:41:58 smithi136 ceph-mon[109861]: pgmap v1427: 105 pgs: 105 active+clean; 46 MiB data, 824 MiB used, 714 GiB / 715 GiB avail; 72 KiB/s rd, 5.6 MiB/s wr, 50 op/s 2024-08-17T17:41:58.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:41:58 smithi136 ceph-mon[115078]: pgmap v1427: 105 pgs: 105 active+clean; 46 MiB data, 824 MiB used, 714 GiB / 715 GiB avail; 72 KiB/s rd, 5.6 MiB/s wr, 50 op/s 2024-08-17T17:42:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:01 smithi165 ceph-mon[105517]: pgmap v1428: 105 pgs: 105 active+clean; 46 MiB data, 824 MiB used, 714 GiB / 715 GiB avail; 69 KiB/s rd, 5.6 MiB/s wr, 45 op/s 2024-08-17T17:42:01.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:01 smithi136 ceph-mon[109861]: pgmap v1428: 105 pgs: 105 active+clean; 46 MiB data, 824 MiB used, 714 GiB / 715 GiB avail; 69 KiB/s rd, 5.6 MiB/s wr, 45 op/s 2024-08-17T17:42:01.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:01 smithi136 ceph-mon[115078]: pgmap v1428: 105 pgs: 105 active+clean; 46 MiB data, 824 MiB used, 714 GiB / 715 GiB avail; 69 KiB/s rd, 5.6 MiB/s wr, 45 op/s 2024-08-17T17:42:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:03 smithi165 ceph-mon[105517]: pgmap v1429: 105 pgs: 105 active+clean; 46 MiB data, 840 MiB used, 714 GiB / 715 GiB avail; 245 KiB/s rd, 5.0 MiB/s wr, 48 op/s 2024-08-17T17:42:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:03 smithi136 ceph-mon[109861]: pgmap v1429: 105 pgs: 105 active+clean; 46 MiB data, 840 MiB used, 714 GiB / 715 GiB avail; 245 KiB/s rd, 5.0 MiB/s wr, 48 op/s 2024-08-17T17:42:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:03 smithi136 ceph-mon[115078]: pgmap v1429: 105 pgs: 105 active+clean; 46 MiB data, 840 MiB used, 714 GiB / 715 GiB avail; 245 KiB/s rd, 5.0 MiB/s wr, 48 op/s 2024-08-17T17:42:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:05 smithi165 ceph-mon[105517]: pgmap v1430: 105 pgs: 105 active+clean; 46 MiB data, 840 MiB used, 714 GiB / 715 GiB avail; 213 KiB/s rd, 4.4 MiB/s wr, 41 op/s 2024-08-17T17:42:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:05 smithi136 ceph-mon[109861]: pgmap v1430: 105 pgs: 105 active+clean; 46 MiB data, 840 MiB used, 714 GiB / 715 GiB avail; 213 KiB/s rd, 4.4 MiB/s wr, 41 op/s 2024-08-17T17:42:05.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:05 smithi136 ceph-mon[115078]: pgmap v1430: 105 pgs: 105 active+clean; 46 MiB data, 840 MiB used, 714 GiB / 715 GiB avail; 213 KiB/s rd, 4.4 MiB/s wr, 41 op/s 2024-08-17T17:42:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:42:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:42:06.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:42:07.196 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/7.Simple (24694 ms) 2024-08-17T17:42:07.196 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/7.Grow 2024-08-17T17:42:07.196 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:42:07.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:07 smithi165 ceph-mon[105517]: pgmap v1431: 105 pgs: 105 active+clean; 54 MiB data, 864 MiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 44 op/s 2024-08-17T17:42:07.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:07 smithi136 ceph-mon[109861]: pgmap v1431: 105 pgs: 105 active+clean; 54 MiB data, 864 MiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 44 op/s 2024-08-17T17:42:07.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:07 smithi136 ceph-mon[115078]: pgmap v1431: 105 pgs: 105 active+clean; 54 MiB data, 864 MiB used, 714 GiB / 715 GiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 44 op/s 2024-08-17T17:42:07.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:42:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:42:07] "GET /metrics HTTP/1.1" 200 36478 "" "Prometheus/2.51.0" 2024-08-17T17:42:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:09 smithi165 ceph-mon[105517]: pgmap v1432: 105 pgs: 105 active+clean; 74 MiB data, 884 MiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 5.1 MiB/s wr, 32 op/s 2024-08-17T17:42:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:09 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "50.15", "id": [6, 1]} : dispatch 2024-08-17T17:42:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:09 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "50.19", "id": [5, 3]} : dispatch 2024-08-17T17:42:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:09 smithi136 ceph-mon[109861]: pgmap v1432: 105 pgs: 105 active+clean; 74 MiB data, 884 MiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 5.1 MiB/s wr, 32 op/s 2024-08-17T17:42:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:09 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "50.15", "id": [6, 1]} : dispatch 2024-08-17T17:42:09.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:09 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "50.19", "id": [5, 3]} : dispatch 2024-08-17T17:42:09.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:09 smithi136 ceph-mon[115078]: pgmap v1432: 105 pgs: 105 active+clean; 74 MiB data, 884 MiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 5.1 MiB/s wr, 32 op/s 2024-08-17T17:42:09.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:09 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "50.15", "id": [6, 1]} : dispatch 2024-08-17T17:42:09.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:09 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "50.19", "id": [5, 3]} : dispatch 2024-08-17T17:42:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "50.15", "id": [6, 1]}]': finished 2024-08-17T17:42:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "50.19", "id": [5, 3]}]': finished 2024-08-17T17:42:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:10 smithi165 ceph-mon[105517]: osdmap e994: 8 total, 8 up, 8 in 2024-08-17T17:42:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:10 smithi165 ceph-mon[105517]: pgmap v1434: 105 pgs: 105 active+clean; 74 MiB data, 884 MiB used, 714 GiB / 715 GiB avail; 8.0 MiB/s rd, 3.2 MiB/s wr, 24 op/s 2024-08-17T17:42:10.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "50.15", "id": [6, 1]}]': finished 2024-08-17T17:42:10.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "50.19", "id": [5, 3]}]': finished 2024-08-17T17:42:10.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:10 smithi136 ceph-mon[109861]: osdmap e994: 8 total, 8 up, 8 in 2024-08-17T17:42:10.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:10 smithi136 ceph-mon[109861]: pgmap v1434: 105 pgs: 105 active+clean; 74 MiB data, 884 MiB used, 714 GiB / 715 GiB avail; 8.0 MiB/s rd, 3.2 MiB/s wr, 24 op/s 2024-08-17T17:42:10.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "50.15", "id": [6, 1]}]': finished 2024-08-17T17:42:10.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "50.19", "id": [5, 3]}]': finished 2024-08-17T17:42:10.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:10 smithi136 ceph-mon[115078]: osdmap e994: 8 total, 8 up, 8 in 2024-08-17T17:42:10.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:10 smithi136 ceph-mon[115078]: pgmap v1434: 105 pgs: 105 active+clean; 74 MiB data, 884 MiB used, 714 GiB / 715 GiB avail; 8.0 MiB/s rd, 3.2 MiB/s wr, 24 op/s 2024-08-17T17:42:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:11 smithi165 ceph-mon[105517]: osdmap e995: 8 total, 8 up, 8 in 2024-08-17T17:42:11.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:11 smithi136 ceph-mon[109861]: osdmap e995: 8 total, 8 up, 8 in 2024-08-17T17:42:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:11 smithi136 ceph-mon[115078]: osdmap e995: 8 total, 8 up, 8 in 2024-08-17T17:42:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:12 smithi165 ceph-mon[105517]: pgmap v1436: 105 pgs: 105 active+clean; 74 MiB data, 888 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 4.0 MiB/s wr, 30 op/s; 1.5 MiB/s, 0 objects/s recovering 2024-08-17T17:42:12.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:12 smithi136 ceph-mon[109861]: pgmap v1436: 105 pgs: 105 active+clean; 74 MiB data, 888 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 4.0 MiB/s wr, 30 op/s; 1.5 MiB/s, 0 objects/s recovering 2024-08-17T17:42:12.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:12 smithi136 ceph-mon[115078]: pgmap v1436: 105 pgs: 105 active+clean; 74 MiB data, 888 MiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 4.0 MiB/s wr, 30 op/s; 1.5 MiB/s, 0 objects/s recovering 2024-08-17T17:42:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:14 smithi165 ceph-mon[105517]: osdmap e996: 8 total, 8 up, 8 in 2024-08-17T17:42:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:14 smithi165 ceph-mon[105517]: pgmap v1438: 105 pgs: 105 active+clean; 74 MiB data, 888 MiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 1023 B/s wr, 9 op/s; 2.0 MiB/s, 0 objects/s recovering 2024-08-17T17:42:14.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:14 smithi136 ceph-mon[115078]: osdmap e996: 8 total, 8 up, 8 in 2024-08-17T17:42:14.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:14 smithi136 ceph-mon[115078]: pgmap v1438: 105 pgs: 105 active+clean; 74 MiB data, 888 MiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 1023 B/s wr, 9 op/s; 2.0 MiB/s, 0 objects/s recovering 2024-08-17T17:42:14.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:14 smithi136 ceph-mon[109861]: osdmap e996: 8 total, 8 up, 8 in 2024-08-17T17:42:14.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:14 smithi136 ceph-mon[109861]: pgmap v1438: 105 pgs: 105 active+clean; 74 MiB data, 888 MiB used, 714 GiB / 715 GiB avail; 4.2 MiB/s rd, 1023 B/s wr, 9 op/s; 2.0 MiB/s, 0 objects/s recovering 2024-08-17T17:42:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:17 smithi165 ceph-mon[105517]: pgmap v1439: 105 pgs: 105 active+clean; 90 MiB data, 968 MiB used, 714 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.3 MiB/s wr, 19 op/s; 1.7 MiB/s, 0 objects/s recovering 2024-08-17T17:42:17.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:17 smithi136 ceph-mon[109861]: pgmap v1439: 105 pgs: 105 active+clean; 90 MiB data, 968 MiB used, 714 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.3 MiB/s wr, 19 op/s; 1.7 MiB/s, 0 objects/s recovering 2024-08-17T17:42:17.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:17 smithi136 ceph-mon[115078]: pgmap v1439: 105 pgs: 105 active+clean; 90 MiB data, 968 MiB used, 714 GiB / 715 GiB avail; 3.6 MiB/s rd, 2.3 MiB/s wr, 19 op/s; 1.7 MiB/s, 0 objects/s recovering 2024-08-17T17:42:17.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:42:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:42:17] "GET /metrics HTTP/1.1" 200 36478 "" "Prometheus/2.51.0" 2024-08-17T17:42:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:19 smithi165 ceph-mon[105517]: pgmap v1440: 105 pgs: 105 active+clean; 119 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 5.6 MiB/s wr, 51 op/s; 1.5 MiB/s, 0 objects/s recovering 2024-08-17T17:42:19.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:19 smithi136 ceph-mon[109861]: pgmap v1440: 105 pgs: 105 active+clean; 119 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 5.6 MiB/s wr, 51 op/s; 1.5 MiB/s, 0 objects/s recovering 2024-08-17T17:42:19.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:19 smithi136 ceph-mon[115078]: pgmap v1440: 105 pgs: 105 active+clean; 119 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 MiB/s rd, 5.6 MiB/s wr, 51 op/s; 1.5 MiB/s, 0 objects/s recovering 2024-08-17T17:42:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:21 smithi165 ceph-mon[105517]: pgmap v1441: 105 pgs: 105 active+clean; 119 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 4.5 MiB/s wr, 41 op/s; 1.2 MiB/s, 0 objects/s recovering 2024-08-17T17:42:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:42:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:21 smithi136 ceph-mon[109861]: pgmap v1441: 105 pgs: 105 active+clean; 119 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 4.5 MiB/s wr, 41 op/s; 1.2 MiB/s, 0 objects/s recovering 2024-08-17T17:42:21.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:42:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:21 smithi136 ceph-mon[115078]: pgmap v1441: 105 pgs: 105 active+clean; 119 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.6 MiB/s rd, 4.5 MiB/s wr, 41 op/s; 1.2 MiB/s, 0 objects/s recovering 2024-08-17T17:42:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:42:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:23 smithi165 ceph-mon[105517]: pgmap v1442: 105 pgs: 105 active+clean; 123 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 221 KiB/s rd, 4.9 MiB/s wr, 44 op/s 2024-08-17T17:42:23.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:23 smithi136 ceph-mon[109861]: pgmap v1442: 105 pgs: 105 active+clean; 123 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 221 KiB/s rd, 4.9 MiB/s wr, 44 op/s 2024-08-17T17:42:23.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:23 smithi136 ceph-mon[115078]: pgmap v1442: 105 pgs: 105 active+clean; 123 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 221 KiB/s rd, 4.9 MiB/s wr, 44 op/s 2024-08-17T17:42:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:25 smithi165 ceph-mon[105517]: pgmap v1443: 105 pgs: 105 active+clean; 123 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 202 KiB/s rd, 4.5 MiB/s wr, 41 op/s 2024-08-17T17:42:25.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:25 smithi136 ceph-mon[109861]: pgmap v1443: 105 pgs: 105 active+clean; 123 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 202 KiB/s rd, 4.5 MiB/s wr, 41 op/s 2024-08-17T17:42:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:25 smithi136 ceph-mon[115078]: pgmap v1443: 105 pgs: 105 active+clean; 123 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 202 KiB/s rd, 4.5 MiB/s wr, 41 op/s 2024-08-17T17:42:27.050 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/7.Grow (19854 ms) 2024-08-17T17:42:27.050 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/7.Shrink 2024-08-17T17:42:27.050 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:42:27.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:27 smithi136 ceph-mon[109861]: pgmap v1444: 105 pgs: 105 active+clean; 139 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 5.4 MiB/s wr, 50 op/s 2024-08-17T17:42:27.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:27 smithi136 ceph-mon[115078]: pgmap v1444: 105 pgs: 105 active+clean; 139 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 5.4 MiB/s wr, 50 op/s 2024-08-17T17:42:27.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:42:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:42:27] "GET /metrics HTTP/1.1" 200 36501 "" "Prometheus/2.51.0" 2024-08-17T17:42:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:27 smithi165 ceph-mon[105517]: pgmap v1444: 105 pgs: 105 active+clean; 139 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 5.4 MiB/s wr, 50 op/s 2024-08-17T17:42:28.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:28 smithi136 ceph-mon[109861]: pgmap v1445: 105 pgs: 105 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.5 MiB/s rd, 5.1 MiB/s wr, 51 op/s 2024-08-17T17:42:28.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:28 smithi136 ceph-mon[115078]: pgmap v1445: 105 pgs: 105 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.5 MiB/s rd, 5.1 MiB/s wr, 51 op/s 2024-08-17T17:42:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:28 smithi165 ceph-mon[105517]: pgmap v1445: 105 pgs: 105 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.5 MiB/s rd, 5.1 MiB/s wr, 51 op/s 2024-08-17T17:42:31.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:31 smithi165 ceph-mon[105517]: pgmap v1446: 105 pgs: 105 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.4 MiB/s rd, 2.7 MiB/s wr, 28 op/s 2024-08-17T17:42:31.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:31 smithi136 ceph-mon[109861]: pgmap v1446: 105 pgs: 105 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.4 MiB/s rd, 2.7 MiB/s wr, 28 op/s 2024-08-17T17:42:31.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:31 smithi136 ceph-mon[115078]: pgmap v1446: 105 pgs: 105 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.4 MiB/s rd, 2.7 MiB/s wr, 28 op/s 2024-08-17T17:42:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:33 smithi165 ceph-mon[105517]: pgmap v1447: 105 pgs: 105 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 2.7 MiB/s wr, 28 op/s 2024-08-17T17:42:33.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:33 smithi136 ceph-mon[109861]: pgmap v1447: 105 pgs: 105 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 2.7 MiB/s wr, 28 op/s 2024-08-17T17:42:33.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:33 smithi136 ceph-mon[115078]: pgmap v1447: 105 pgs: 105 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 2.7 MiB/s wr, 28 op/s 2024-08-17T17:42:34.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:34 smithi165 ceph-mon[105517]: osdmap e997: 8 total, 8 up, 8 in 2024-08-17T17:42:34.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:34 smithi136 ceph-mon[109861]: osdmap e997: 8 total, 8 up, 8 in 2024-08-17T17:42:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:34 smithi136 ceph-mon[115078]: osdmap e997: 8 total, 8 up, 8 in 2024-08-17T17:42:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:35 smithi165 ceph-mon[105517]: pgmap v1449: 105 pgs: 105 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.8 MiB/s wr, 25 op/s 2024-08-17T17:42:35.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:35 smithi136 ceph-mon[109861]: pgmap v1449: 105 pgs: 105 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.8 MiB/s wr, 25 op/s 2024-08-17T17:42:35.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:35 smithi136 ceph-mon[115078]: pgmap v1449: 105 pgs: 105 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.8 MiB/s wr, 25 op/s 2024-08-17T17:42:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:42:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:36 smithi165 ceph-mon[105517]: pgmap v1450: 105 pgs: 105 active+clean; 148 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 1.3 MiB/s wr, 9 op/s 2024-08-17T17:42:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:42:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:36 smithi136 ceph-mon[109861]: pgmap v1450: 105 pgs: 105 active+clean; 148 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 1.3 MiB/s wr, 9 op/s 2024-08-17T17:42:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:42:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:36 smithi136 ceph-mon[115078]: pgmap v1450: 105 pgs: 105 active+clean; 148 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.9 MiB/s rd, 1.3 MiB/s wr, 9 op/s 2024-08-17T17:42:37.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:42:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:42:37] "GET /metrics HTTP/1.1" 200 36486 "" "Prometheus/2.51.0" 2024-08-17T17:42:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:39 smithi165 ceph-mon[105517]: pgmap v1451: 105 pgs: 105 active+clean; 188 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 37 op/s 2024-08-17T17:42:39.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:39 smithi136 ceph-mon[109861]: pgmap v1451: 105 pgs: 105 active+clean; 188 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 37 op/s 2024-08-17T17:42:39.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:39 smithi136 ceph-mon[115078]: pgmap v1451: 105 pgs: 105 active+clean; 188 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 37 op/s 2024-08-17T17:42:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:41 smithi165 ceph-mon[105517]: pgmap v1452: 105 pgs: 105 active+clean; 188 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 37 op/s 2024-08-17T17:42:41.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:41 smithi136 ceph-mon[109861]: pgmap v1452: 105 pgs: 105 active+clean; 188 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 37 op/s 2024-08-17T17:42:41.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:41 smithi136 ceph-mon[115078]: pgmap v1452: 105 pgs: 105 active+clean; 188 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 4.1 MiB/s wr, 37 op/s 2024-08-17T17:42:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:43 smithi165 ceph-mon[105517]: pgmap v1453: 105 pgs: 105 active+clean; 196 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 119 KiB/s rd, 4.9 MiB/s wr, 46 op/s 2024-08-17T17:42:43.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:43 smithi136 ceph-mon[109861]: pgmap v1453: 105 pgs: 105 active+clean; 196 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 119 KiB/s rd, 4.9 MiB/s wr, 46 op/s 2024-08-17T17:42:43.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:43 smithi136 ceph-mon[115078]: pgmap v1453: 105 pgs: 105 active+clean; 196 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 119 KiB/s rd, 4.9 MiB/s wr, 46 op/s 2024-08-17T17:42:44.572 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:44 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:42:44.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:44 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:42:44.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:44 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:42:45.480 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:45 smithi165 ceph-mon[105517]: pgmap v1454: 105 pgs: 105 active+clean; 196 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 109 KiB/s rd, 4.5 MiB/s wr, 42 op/s 2024-08-17T17:42:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:45 smithi136 ceph-mon[109861]: pgmap v1454: 105 pgs: 105 active+clean; 196 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 109 KiB/s rd, 4.5 MiB/s wr, 42 op/s 2024-08-17T17:42:45.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:45 smithi136 ceph-mon[115078]: pgmap v1454: 105 pgs: 105 active+clean; 196 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 109 KiB/s rd, 4.5 MiB/s wr, 42 op/s 2024-08-17T17:42:46.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:46 smithi165 ceph-mon[105517]: pgmap v1455: 105 pgs: 105 active+clean; 208 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 5.4 MiB/s wr, 39 op/s 2024-08-17T17:42:46.850 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:46 smithi136 ceph-mon[109861]: pgmap v1455: 105 pgs: 105 active+clean; 208 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 5.4 MiB/s wr, 39 op/s 2024-08-17T17:42:46.851 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:46 smithi136 ceph-mon[115078]: pgmap v1455: 105 pgs: 105 active+clean; 208 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 5.4 MiB/s wr, 39 op/s 2024-08-17T17:42:47.295 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/7.Shrink (20245 ms) 2024-08-17T17:42:47.295 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/7.ShrinkToOne 2024-08-17T17:42:47.295 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:42:47.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:42:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:42:47] "GET /metrics HTTP/1.1" 200 36486 "" "Prometheus/2.51.0" 2024-08-17T17:42:48.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:48 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:42:48.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:48 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:42:48.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:48 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:42:48.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:48 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:42:48.551 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:48 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:42:48.551 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:48 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:42:48.551 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:48 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:42:48.551 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:48 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:42:48.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:48 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:42:48.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:48 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:42:48.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:48 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:42:48.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:48 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:42:49.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:49 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:42:49.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:49 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:42:49.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:49 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:42:49.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:49 smithi165 ceph-mon[105517]: pgmap v1456: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 6.3 MiB/s wr, 53 op/s 2024-08-17T17:42:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:49 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:42:49.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:49 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:42:49.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:49 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:42:49.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:49 smithi136 ceph-mon[109861]: pgmap v1456: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 6.3 MiB/s wr, 53 op/s 2024-08-17T17:42:49.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:49 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:42:49.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:49 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:42:49.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:49 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:42:49.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:49 smithi136 ceph-mon[115078]: pgmap v1456: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 6.3 MiB/s wr, 53 op/s 2024-08-17T17:42:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:51 smithi165 ceph-mon[105517]: pgmap v1457: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 3.0 MiB/s wr, 22 op/s 2024-08-17T17:42:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:42:51.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:51 smithi136 ceph-mon[109861]: pgmap v1457: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 3.0 MiB/s wr, 22 op/s 2024-08-17T17:42:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:42:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:51 smithi136 ceph-mon[115078]: pgmap v1457: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 3.0 MiB/s wr, 22 op/s 2024-08-17T17:42:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:42:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:53 smithi165 ceph-mon[105517]: pgmap v1458: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 3.0 MiB/s wr, 27 op/s 2024-08-17T17:42:53.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:53 smithi136 ceph-mon[109861]: pgmap v1458: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 3.0 MiB/s wr, 27 op/s 2024-08-17T17:42:53.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:53 smithi136 ceph-mon[115078]: pgmap v1458: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.8 MiB/s rd, 3.0 MiB/s wr, 27 op/s 2024-08-17T17:42:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:55 smithi165 ceph-mon[105517]: pgmap v1459: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 2.3 MiB/s wr, 20 op/s 2024-08-17T17:42:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:55 smithi165 ceph-mon[105517]: osdmap e998: 8 total, 8 up, 8 in 2024-08-17T17:42:55.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:55 smithi136 ceph-mon[109861]: pgmap v1459: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 2.3 MiB/s wr, 20 op/s 2024-08-17T17:42:55.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:55 smithi136 ceph-mon[109861]: osdmap e998: 8 total, 8 up, 8 in 2024-08-17T17:42:55.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:55 smithi136 ceph-mon[115078]: pgmap v1459: 105 pgs: 105 active+clean; 220 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 2.3 MiB/s wr, 20 op/s 2024-08-17T17:42:55.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:55 smithi136 ceph-mon[115078]: osdmap e998: 8 total, 8 up, 8 in 2024-08-17T17:42:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:56 smithi165 ceph-mon[105517]: pgmap v1461: 105 pgs: 105 active+clean; 241 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 3.3 MiB/s wr, 39 op/s 2024-08-17T17:42:56.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:56 smithi136 ceph-mon[109861]: pgmap v1461: 105 pgs: 105 active+clean; 241 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 3.3 MiB/s wr, 39 op/s 2024-08-17T17:42:56.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:56 smithi136 ceph-mon[115078]: pgmap v1461: 105 pgs: 105 active+clean; 241 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 3.3 MiB/s wr, 39 op/s 2024-08-17T17:42:57.923 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:42:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:42:57] "GET /metrics HTTP/1.1" 200 36494 "" "Prometheus/2.51.0" 2024-08-17T17:42:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:42:59 smithi165 ceph-mon[105517]: pgmap v1462: 105 pgs: 105 active+clean; 261 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.5 MiB/s rd, 4.1 MiB/s wr, 41 op/s 2024-08-17T17:42:59.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:42:59 smithi136 ceph-mon[109861]: pgmap v1462: 105 pgs: 105 active+clean; 261 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.5 MiB/s rd, 4.1 MiB/s wr, 41 op/s 2024-08-17T17:42:59.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:42:59 smithi136 ceph-mon[115078]: pgmap v1462: 105 pgs: 105 active+clean; 261 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.5 MiB/s rd, 4.1 MiB/s wr, 41 op/s 2024-08-17T17:43:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:01 smithi165 ceph-mon[105517]: pgmap v1463: 105 pgs: 105 active+clean; 261 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.5 MiB/s rd, 4.1 MiB/s wr, 41 op/s 2024-08-17T17:43:01.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:01 smithi136 ceph-mon[109861]: pgmap v1463: 105 pgs: 105 active+clean; 261 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.5 MiB/s rd, 4.1 MiB/s wr, 41 op/s 2024-08-17T17:43:01.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:01 smithi136 ceph-mon[115078]: pgmap v1463: 105 pgs: 105 active+clean; 261 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.5 MiB/s rd, 4.1 MiB/s wr, 41 op/s 2024-08-17T17:43:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:03 smithi165 ceph-mon[105517]: pgmap v1464: 105 pgs: 105 active+clean; 265 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 117 KiB/s rd, 4.5 MiB/s wr, 42 op/s 2024-08-17T17:43:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:03 smithi136 ceph-mon[109861]: pgmap v1464: 105 pgs: 105 active+clean; 265 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 117 KiB/s rd, 4.5 MiB/s wr, 42 op/s 2024-08-17T17:43:03.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:03 smithi136 ceph-mon[115078]: pgmap v1464: 105 pgs: 105 active+clean; 265 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 117 KiB/s rd, 4.5 MiB/s wr, 42 op/s 2024-08-17T17:43:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:05 smithi165 ceph-mon[105517]: pgmap v1465: 105 pgs: 105 active+clean; 265 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 117 KiB/s rd, 4.5 MiB/s wr, 42 op/s 2024-08-17T17:43:05.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:05 smithi136 ceph-mon[115078]: pgmap v1465: 105 pgs: 105 active+clean; 265 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 117 KiB/s rd, 4.5 MiB/s wr, 42 op/s 2024-08-17T17:43:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:05 smithi136 ceph-mon[109861]: pgmap v1465: 105 pgs: 105 active+clean; 265 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 117 KiB/s rd, 4.5 MiB/s wr, 42 op/s 2024-08-17T17:43:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:43:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:43:06.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:43:07.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:07 smithi165 ceph-mon[105517]: pgmap v1466: 105 pgs: 105 active+clean; 269 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 534 KiB/s rd, 4.1 MiB/s wr, 44 op/s 2024-08-17T17:43:07.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:07 smithi136 ceph-mon[109861]: pgmap v1466: 105 pgs: 105 active+clean; 269 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 534 KiB/s rd, 4.1 MiB/s wr, 44 op/s 2024-08-17T17:43:07.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:07 smithi136 ceph-mon[115078]: pgmap v1466: 105 pgs: 105 active+clean; 269 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 534 KiB/s rd, 4.1 MiB/s wr, 44 op/s 2024-08-17T17:43:07.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:43:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:43:07] "GET /metrics HTTP/1.1" 200 36493 "" "Prometheus/2.51.0" 2024-08-17T17:43:08.065 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/7.ShrinkToOne (20768 ms) 2024-08-17T17:43:08.065 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/7.ShrinkToOneAfterSnapshot 2024-08-17T17:43:08.065 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:43:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:08 smithi165 ceph-mon[105517]: pgmap v1467: 105 pgs: 105 active+clean; 273 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 913 KiB/s rd, 3.3 MiB/s wr, 35 op/s 2024-08-17T17:43:08.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:08 smithi136 ceph-mon[109861]: pgmap v1467: 105 pgs: 105 active+clean; 273 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 913 KiB/s rd, 3.3 MiB/s wr, 35 op/s 2024-08-17T17:43:08.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:08 smithi136 ceph-mon[115078]: pgmap v1467: 105 pgs: 105 active+clean; 273 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 913 KiB/s rd, 3.3 MiB/s wr, 35 op/s 2024-08-17T17:43:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:11 smithi165 ceph-mon[105517]: pgmap v1468: 105 pgs: 105 active+clean; 273 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 901 KiB/s rd, 1.7 MiB/s wr, 19 op/s 2024-08-17T17:43:11.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:11 smithi136 ceph-mon[109861]: pgmap v1468: 105 pgs: 105 active+clean; 273 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 901 KiB/s rd, 1.7 MiB/s wr, 19 op/s 2024-08-17T17:43:11.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:11 smithi136 ceph-mon[115078]: pgmap v1468: 105 pgs: 105 active+clean; 273 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 901 KiB/s rd, 1.7 MiB/s wr, 19 op/s 2024-08-17T17:43:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:13 smithi165 ceph-mon[105517]: pgmap v1469: 105 pgs: 105 active+clean; 269 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 904 KiB/s rd, 1.7 MiB/s wr, 24 op/s 2024-08-17T17:43:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:13 smithi136 ceph-mon[109861]: pgmap v1469: 105 pgs: 105 active+clean; 269 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 904 KiB/s rd, 1.7 MiB/s wr, 24 op/s 2024-08-17T17:43:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:13 smithi136 ceph-mon[115078]: pgmap v1469: 105 pgs: 105 active+clean; 269 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 904 KiB/s rd, 1.7 MiB/s wr, 24 op/s 2024-08-17T17:43:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:14 smithi165 ceph-mon[105517]: osdmap e999: 8 total, 8 up, 8 in 2024-08-17T17:43:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:14 smithi165 ceph-mon[105517]: pgmap v1471: 105 pgs: 105 active+clean; 269 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 994 KiB/s rd, 1.6 MiB/s wr, 21 op/s 2024-08-17T17:43:14.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:14 smithi136 ceph-mon[109861]: osdmap e999: 8 total, 8 up, 8 in 2024-08-17T17:43:14.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:14 smithi136 ceph-mon[109861]: pgmap v1471: 105 pgs: 105 active+clean; 269 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 994 KiB/s rd, 1.6 MiB/s wr, 21 op/s 2024-08-17T17:43:14.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:14 smithi136 ceph-mon[115078]: osdmap e999: 8 total, 8 up, 8 in 2024-08-17T17:43:14.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:14 smithi136 ceph-mon[115078]: pgmap v1471: 105 pgs: 105 active+clean; 269 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 994 KiB/s rd, 1.6 MiB/s wr, 21 op/s 2024-08-17T17:43:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:17 smithi165 ceph-mon[105517]: pgmap v1472: 105 pgs: 105 active+clean; 286 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 480 KiB/s rd, 2.9 MiB/s wr, 22 op/s 2024-08-17T17:43:17.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:17 smithi136 ceph-mon[109861]: pgmap v1472: 105 pgs: 105 active+clean; 286 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 480 KiB/s rd, 2.9 MiB/s wr, 22 op/s 2024-08-17T17:43:17.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:17 smithi136 ceph-mon[115078]: pgmap v1472: 105 pgs: 105 active+clean; 286 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 480 KiB/s rd, 2.9 MiB/s wr, 22 op/s 2024-08-17T17:43:17.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:43:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:43:17] "GET /metrics HTTP/1.1" 200 36493 "" "Prometheus/2.51.0" 2024-08-17T17:43:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:19 smithi165 ceph-mon[105517]: pgmap v1473: 105 pgs: 105 active+clean; 310 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 4.1 MiB/s wr, 42 op/s 2024-08-17T17:43:19.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:19 smithi136 ceph-mon[109861]: pgmap v1473: 105 pgs: 105 active+clean; 310 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 4.1 MiB/s wr, 42 op/s 2024-08-17T17:43:19.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:19 smithi136 ceph-mon[115078]: pgmap v1473: 105 pgs: 105 active+clean; 310 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 4.1 MiB/s wr, 42 op/s 2024-08-17T17:43:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:21 smithi165 ceph-mon[105517]: pgmap v1474: 105 pgs: 105 active+clean; 310 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 4.1 MiB/s wr, 42 op/s 2024-08-17T17:43:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:43:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:21 smithi136 ceph-mon[109861]: pgmap v1474: 105 pgs: 105 active+clean; 310 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 4.1 MiB/s wr, 42 op/s 2024-08-17T17:43:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:43:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:21 smithi136 ceph-mon[115078]: pgmap v1474: 105 pgs: 105 active+clean; 310 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 4.1 MiB/s wr, 42 op/s 2024-08-17T17:43:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:43:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:23 smithi165 ceph-mon[105517]: pgmap v1475: 105 pgs: 105 active+clean; 318 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 193 KiB/s rd, 4.9 MiB/s wr, 45 op/s 2024-08-17T17:43:23.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:23 smithi136 ceph-mon[109861]: pgmap v1475: 105 pgs: 105 active+clean; 318 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 193 KiB/s rd, 4.9 MiB/s wr, 45 op/s 2024-08-17T17:43:23.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:23 smithi136 ceph-mon[115078]: pgmap v1475: 105 pgs: 105 active+clean; 318 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 193 KiB/s rd, 4.9 MiB/s wr, 45 op/s 2024-08-17T17:43:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:25 smithi136 ceph-mon[109861]: pgmap v1476: 105 pgs: 105 active+clean; 318 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 176 KiB/s rd, 4.5 MiB/s wr, 41 op/s 2024-08-17T17:43:25.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:25 smithi136 ceph-mon[115078]: pgmap v1476: 105 pgs: 105 active+clean; 318 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 176 KiB/s rd, 4.5 MiB/s wr, 41 op/s 2024-08-17T17:43:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:25 smithi165 ceph-mon[105517]: pgmap v1476: 105 pgs: 105 active+clean; 318 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 176 KiB/s rd, 4.5 MiB/s wr, 41 op/s 2024-08-17T17:43:26.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:26 smithi136 ceph-mon[109861]: osdmap e1000: 8 total, 8 up, 8 in 2024-08-17T17:43:26.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:26 smithi136 ceph-mon[115078]: osdmap e1000: 8 total, 8 up, 8 in 2024-08-17T17:43:26.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:26 smithi165 ceph-mon[105517]: osdmap e1000: 8 total, 8 up, 8 in 2024-08-17T17:43:27.408 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/7.ShrinkToOneAfterSnapshot (19345 ms) 2024-08-17T17:43:27.409 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/7.MinOverlap 2024-08-17T17:43:27.409 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:43:27.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:27 smithi136 ceph-mon[109861]: pgmap v1478: 105 pgs: 105 active+clean; 342 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 5.6 MiB/s wr, 36 op/s 2024-08-17T17:43:27.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:27 smithi136 ceph-mon[115078]: pgmap v1478: 105 pgs: 105 active+clean; 342 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 5.6 MiB/s wr, 36 op/s 2024-08-17T17:43:27.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:27 smithi165 ceph-mon[105517]: pgmap v1478: 105 pgs: 105 active+clean; 342 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.4 MiB/s rd, 5.6 MiB/s wr, 36 op/s 2024-08-17T17:43:27.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:43:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:43:27] "GET /metrics HTTP/1.1" 200 36494 "" "Prometheus/2.51.0" 2024-08-17T17:43:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:29 smithi165 ceph-mon[105517]: pgmap v1479: 105 pgs: 105 active+clean; 374 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 6.4 MiB/s wr, 39 op/s 2024-08-17T17:43:29.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:29 smithi136 ceph-mon[109861]: pgmap v1479: 105 pgs: 105 active+clean; 374 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 6.4 MiB/s wr, 39 op/s 2024-08-17T17:43:29.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:29 smithi136 ceph-mon[115078]: pgmap v1479: 105 pgs: 105 active+clean; 374 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 6.4 MiB/s wr, 39 op/s 2024-08-17T17:43:31.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:31 smithi165 ceph-mon[105517]: pgmap v1480: 105 pgs: 105 active+clean; 374 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 6.4 MiB/s wr, 39 op/s 2024-08-17T17:43:31.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:31 smithi136 ceph-mon[109861]: pgmap v1480: 105 pgs: 105 active+clean; 374 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 6.4 MiB/s wr, 39 op/s 2024-08-17T17:43:31.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:31 smithi136 ceph-mon[115078]: pgmap v1480: 105 pgs: 105 active+clean; 374 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 6.4 MiB/s wr, 39 op/s 2024-08-17T17:43:32.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:32 smithi136 ceph-mon[109861]: pgmap v1481: 105 pgs: 105 active+clean; 374 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 5.6 MiB/s wr, 35 op/s 2024-08-17T17:43:32.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:32 smithi136 ceph-mon[115078]: pgmap v1481: 105 pgs: 105 active+clean; 374 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 5.6 MiB/s wr, 35 op/s 2024-08-17T17:43:33.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:32 smithi165 ceph-mon[105517]: pgmap v1481: 105 pgs: 105 active+clean; 374 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 2.9 MiB/s rd, 5.6 MiB/s wr, 35 op/s 2024-08-17T17:43:34.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:34 smithi136 ceph-mon[109861]: osdmap e1001: 8 total, 8 up, 8 in 2024-08-17T17:43:34.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:34 smithi136 ceph-mon[109861]: pgmap v1483: 105 pgs: 105 active+clean; 374 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 6.0 MiB/s wr, 38 op/s 2024-08-17T17:43:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:34 smithi136 ceph-mon[115078]: osdmap e1001: 8 total, 8 up, 8 in 2024-08-17T17:43:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:34 smithi136 ceph-mon[115078]: pgmap v1483: 105 pgs: 105 active+clean; 374 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 6.0 MiB/s wr, 38 op/s 2024-08-17T17:43:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:34 smithi165 ceph-mon[105517]: osdmap e1001: 8 total, 8 up, 8 in 2024-08-17T17:43:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:34 smithi165 ceph-mon[105517]: pgmap v1483: 105 pgs: 105 active+clean; 374 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 6.0 MiB/s wr, 38 op/s 2024-08-17T17:43:36.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:43:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:43:36.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:43:37.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:37 smithi165 ceph-mon[105517]: pgmap v1484: 105 pgs: 105 active+clean; 390 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 60 op/s 2024-08-17T17:43:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:37 smithi136 ceph-mon[109861]: pgmap v1484: 105 pgs: 105 active+clean; 390 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 60 op/s 2024-08-17T17:43:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:37 smithi136 ceph-mon[115078]: pgmap v1484: 105 pgs: 105 active+clean; 390 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 60 op/s 2024-08-17T17:43:37.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:43:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:43:37] "GET /metrics HTTP/1.1" 200 36492 "" "Prometheus/2.51.0" 2024-08-17T17:43:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:39 smithi165 ceph-mon[105517]: pgmap v1485: 105 pgs: 105 active+clean; 419 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s rd, 4.5 MiB/s wr, 42 op/s 2024-08-17T17:43:39.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:39 smithi136 ceph-mon[109861]: pgmap v1485: 105 pgs: 105 active+clean; 419 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s rd, 4.5 MiB/s wr, 42 op/s 2024-08-17T17:43:39.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:39 smithi136 ceph-mon[115078]: pgmap v1485: 105 pgs: 105 active+clean; 419 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s rd, 4.5 MiB/s wr, 42 op/s 2024-08-17T17:43:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:41 smithi165 ceph-mon[105517]: pgmap v1486: 105 pgs: 105 active+clean; 419 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s rd, 4.5 MiB/s wr, 42 op/s 2024-08-17T17:43:41.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:41 smithi136 ceph-mon[109861]: pgmap v1486: 105 pgs: 105 active+clean; 419 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s rd, 4.5 MiB/s wr, 42 op/s 2024-08-17T17:43:41.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:41 smithi136 ceph-mon[115078]: pgmap v1486: 105 pgs: 105 active+clean; 419 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s rd, 4.5 MiB/s wr, 42 op/s 2024-08-17T17:43:42.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:42 smithi136 ceph-mon[109861]: pgmap v1487: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 222 KiB/s rd, 4.9 MiB/s wr, 46 op/s 2024-08-17T17:43:42.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:42 smithi136 ceph-mon[115078]: pgmap v1487: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 222 KiB/s rd, 4.9 MiB/s wr, 46 op/s 2024-08-17T17:43:43.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:42 smithi165 ceph-mon[105517]: pgmap v1487: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 222 KiB/s rd, 4.9 MiB/s wr, 46 op/s 2024-08-17T17:43:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:45 smithi165 ceph-mon[105517]: pgmap v1488: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 204 KiB/s rd, 4.5 MiB/s wr, 42 op/s 2024-08-17T17:43:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:45 smithi136 ceph-mon[109861]: pgmap v1488: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 204 KiB/s rd, 4.5 MiB/s wr, 42 op/s 2024-08-17T17:43:45.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:45 smithi136 ceph-mon[115078]: pgmap v1488: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 204 KiB/s rd, 4.5 MiB/s wr, 42 op/s 2024-08-17T17:43:47.158 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/7.MinOverlap (19749 ms) 2024-08-17T17:43:47.158 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] EncryptedFlattenTest/7.ZeroOverlap 2024-08-17T17:43:47.158 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:43:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:47 smithi136 ceph-mon[109861]: pgmap v1489: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 185 KiB/s rd, 4.1 MiB/s wr, 38 op/s 2024-08-17T17:43:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:47 smithi136 ceph-mon[115078]: pgmap v1489: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 185 KiB/s rd, 4.1 MiB/s wr, 38 op/s 2024-08-17T17:43:47.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:43:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:43:47] "GET /metrics HTTP/1.1" 200 36492 "" "Prometheus/2.51.0" 2024-08-17T17:43:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:47 smithi165 ceph-mon[105517]: pgmap v1489: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 185 KiB/s rd, 4.1 MiB/s wr, 38 op/s 2024-08-17T17:43:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:48 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:43:48.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:48 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:43:48.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:48 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:43:49.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:49 smithi136 ceph-mon[115078]: pgmap v1490: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 27 op/s 2024-08-17T17:43:49.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:49 smithi136 ceph-mon[109861]: pgmap v1490: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 27 op/s 2024-08-17T17:43:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:49 smithi165 ceph-mon[105517]: pgmap v1490: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 27 op/s 2024-08-17T17:43:50.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:50 smithi165 ceph-mon[105517]: pgmap v1491: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 19 op/s 2024-08-17T17:43:50.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:50 smithi136 ceph-mon[109861]: pgmap v1491: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 19 op/s 2024-08-17T17:43:50.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:50 smithi136 ceph-mon[115078]: pgmap v1491: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 19 op/s 2024-08-17T17:43:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:43:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:43:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:43:52.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:52 smithi165 ceph-mon[105517]: pgmap v1492: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 24 op/s 2024-08-17T17:43:52.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:52 smithi136 ceph-mon[109861]: pgmap v1492: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 24 op/s 2024-08-17T17:43:52.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:52 smithi136 ceph-mon[115078]: pgmap v1492: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 24 op/s 2024-08-17T17:43:53.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:43:53.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:43:53.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:43:53.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:43:53.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:43:53.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:43:53.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:43:53.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:43:53.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:43:53.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:43:53.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:43:53.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:43:53.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:43:53.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:43:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:43:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:43:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:43:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:43:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:43:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:43:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:43:55.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:54 smithi136 ceph-mon[109861]: osdmap e1002: 8 total, 8 up, 8 in 2024-08-17T17:43:55.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:54 smithi136 ceph-mon[109861]: pgmap v1494: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.2 MiB/s wr, 20 op/s 2024-08-17T17:43:55.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:54 smithi136 ceph-mon[115078]: osdmap e1002: 8 total, 8 up, 8 in 2024-08-17T17:43:55.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:54 smithi136 ceph-mon[115078]: pgmap v1494: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.2 MiB/s wr, 20 op/s 2024-08-17T17:43:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:54 smithi165 ceph-mon[105517]: osdmap e1002: 8 total, 8 up, 8 in 2024-08-17T17:43:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:54 smithi165 ceph-mon[105517]: pgmap v1494: 105 pgs: 105 active+clean; 423 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.2 MiB/s wr, 20 op/s 2024-08-17T17:43:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:57 smithi165 ceph-mon[105517]: pgmap v1495: 105 pgs: 105 active+clean; 439 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 37 op/s 2024-08-17T17:43:57.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:57 smithi136 ceph-mon[109861]: pgmap v1495: 105 pgs: 105 active+clean; 439 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 37 op/s 2024-08-17T17:43:57.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:57 smithi136 ceph-mon[115078]: pgmap v1495: 105 pgs: 105 active+clean; 439 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 37 op/s 2024-08-17T17:43:57.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:43:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:43:57] "GET /metrics HTTP/1.1" 200 36491 "" "Prometheus/2.51.0" 2024-08-17T17:43:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:43:58 smithi136 ceph-mon[109861]: pgmap v1496: 105 pgs: 105 active+clean; 464 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 438 KiB/s rd, 4.1 MiB/s wr, 40 op/s 2024-08-17T17:43:58.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:43:58 smithi136 ceph-mon[115078]: pgmap v1496: 105 pgs: 105 active+clean; 464 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 438 KiB/s rd, 4.1 MiB/s wr, 40 op/s 2024-08-17T17:43:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:43:58 smithi165 ceph-mon[105517]: pgmap v1496: 105 pgs: 105 active+clean; 464 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 438 KiB/s rd, 4.1 MiB/s wr, 40 op/s 2024-08-17T17:44:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:01 smithi165 ceph-mon[105517]: pgmap v1497: 105 pgs: 105 active+clean; 464 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 438 KiB/s rd, 4.1 MiB/s wr, 40 op/s 2024-08-17T17:44:01.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:01 smithi136 ceph-mon[109861]: pgmap v1497: 105 pgs: 105 active+clean; 464 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 438 KiB/s rd, 4.1 MiB/s wr, 40 op/s 2024-08-17T17:44:01.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:01 smithi136 ceph-mon[115078]: pgmap v1497: 105 pgs: 105 active+clean; 464 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 438 KiB/s rd, 4.1 MiB/s wr, 40 op/s 2024-08-17T17:44:02.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:02 smithi165 ceph-mon[105517]: pgmap v1498: 105 pgs: 105 active+clean; 472 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 218 KiB/s rd, 4.9 MiB/s wr, 41 op/s 2024-08-17T17:44:02.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:02 smithi136 ceph-mon[109861]: pgmap v1498: 105 pgs: 105 active+clean; 472 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 218 KiB/s rd, 4.9 MiB/s wr, 41 op/s 2024-08-17T17:44:02.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:02 smithi136 ceph-mon[115078]: pgmap v1498: 105 pgs: 105 active+clean; 472 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 218 KiB/s rd, 4.9 MiB/s wr, 41 op/s 2024-08-17T17:44:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:05 smithi165 ceph-mon[105517]: pgmap v1499: 105 pgs: 105 active+clean; 472 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 205 KiB/s rd, 4.6 MiB/s wr, 38 op/s 2024-08-17T17:44:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:05 smithi136 ceph-mon[109861]: pgmap v1499: 105 pgs: 105 active+clean; 472 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 205 KiB/s rd, 4.6 MiB/s wr, 38 op/s 2024-08-17T17:44:05.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:05 smithi136 ceph-mon[115078]: pgmap v1499: 105 pgs: 105 active+clean; 472 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 205 KiB/s rd, 4.6 MiB/s wr, 38 op/s 2024-08-17T17:44:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:44:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:06 smithi165 ceph-mon[105517]: pgmap v1500: 105 pgs: 105 active+clean; 472 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 186 KiB/s rd, 4.1 MiB/s wr, 41 op/s 2024-08-17T17:44:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:44:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:06 smithi136 ceph-mon[109861]: pgmap v1500: 105 pgs: 105 active+clean; 472 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 186 KiB/s rd, 4.1 MiB/s wr, 41 op/s 2024-08-17T17:44:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:44:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:06 smithi136 ceph-mon[115078]: pgmap v1500: 105 pgs: 105 active+clean; 472 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 186 KiB/s rd, 4.1 MiB/s wr, 41 op/s 2024-08-17T17:44:07.306 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] EncryptedFlattenTest/7.ZeroOverlap (20148 ms) 2024-08-17T17:44:07.506 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 7 tests from EncryptedFlattenTest/7 (144804 ms total) 2024-08-17T17:44:07.506 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:44:07.506 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 19 tests from DiffIterateTest/0, where TypeParam = DiffIterateParams 2024-08-17T17:44:07.535 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/0.DiffIterate 2024-08-17T17:44:07.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:44:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:44:07] "GET /metrics HTTP/1.1" 200 36493 "" "Prometheus/2.51.0" 2024-08-17T17:44:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:08 smithi165 ceph-mon[105517]: osdmap e1003: 8 total, 8 up, 8 in 2024-08-17T17:44:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:08 smithi165 ceph-mon[105517]: pgmap v1502: 73 pgs: 73 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:44:08.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:08 smithi136 ceph-mon[109861]: osdmap e1003: 8 total, 8 up, 8 in 2024-08-17T17:44:08.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:08 smithi136 ceph-mon[109861]: pgmap v1502: 73 pgs: 73 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:44:08.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:08 smithi136 ceph-mon[115078]: osdmap e1003: 8 total, 8 up, 8 in 2024-08-17T17:44:08.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:08 smithi136 ceph-mon[115078]: pgmap v1502: 73 pgs: 73 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:44:09.521 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:44:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:09 smithi136 ceph-mon[109861]: osdmap e1004: 8 total, 8 up, 8 in 2024-08-17T17:44:09.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:09 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2274018038' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-47","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:44:09.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:09 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-47","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:44:09.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:09 smithi136 ceph-mon[115078]: osdmap e1004: 8 total, 8 up, 8 in 2024-08-17T17:44:09.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:09 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2274018038' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-47","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:44:09.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:09 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-47","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:44:10.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:09 smithi165 ceph-mon[105517]: osdmap e1004: 8 total, 8 up, 8 in 2024-08-17T17:44:10.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:09 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2274018038' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-47","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:44:10.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:09 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-47","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:44:10.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:10 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:44:10.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:10 smithi136 ceph-mon[109861]: osdmap e1005: 8 total, 8 up, 8 in 2024-08-17T17:44:10.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:10 smithi136 ceph-mon[109861]: pgmap v1505: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:44:10.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:10 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:44:10.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:10 smithi136 ceph-mon[115078]: osdmap e1005: 8 total, 8 up, 8 in 2024-08-17T17:44:10.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:10 smithi136 ceph-mon[115078]: pgmap v1505: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:44:11.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:10 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:44:11.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:10 smithi165 ceph-mon[105517]: osdmap e1005: 8 total, 8 up, 8 in 2024-08-17T17:44:11.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:10 smithi165 ceph-mon[105517]: pgmap v1505: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:44:11.555 INFO:tasks.workunit.client.0.smithi136.stdout: wrote [4386700~22628,4623995~8131,5329861~47128,6160260~79524,8257169~43858,8517087~92612,9922872~18196,12993918~64348,15094126~35651,19403011~922] 2024-08-17T17:44:11.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:11 smithi136 ceph-mon[109861]: osdmap e1006: 8 total, 8 up, 8 in 2024-08-17T17:44:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:11 smithi136 ceph-mon[115078]: osdmap e1006: 8 total, 8 up, 8 in 2024-08-17T17:44:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:11 smithi165 ceph-mon[105517]: osdmap e1006: 8 total, 8 up, 8 in 2024-08-17T17:44:12.552 INFO:tasks.workunit.client.0.smithi136.stdout: wrote [1061602~40377,3914769~64458,4076986~44562,5097325~22747,9739736~101890,9903937~76064,14950362~83473,16547548~68760,20268544~66212,20535714~63475] 2024-08-17T17:44:12.560 INFO:tasks.workunit.client.0.smithi136.stdout: diff was [0~4121548,5097325~22747,9739736~101890,9903937~76064,14950362~83473,15129777~1486531,19403933~1195256] 2024-08-17T17:44:12.563 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/0.DiffIterate (5028 ms) 2024-08-17T17:44:12.563 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/0.DiffIterateDeterministic 2024-08-17T17:44:12.564 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:44:12.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:12 smithi136 ceph-mon[109861]: osdmap e1007: 8 total, 8 up, 8 in 2024-08-17T17:44:12.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:12 smithi136 ceph-mon[109861]: pgmap v1508: 105 pgs: 105 active+clean; 4.5 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 50 KiB/s wr, 2 op/s 2024-08-17T17:44:12.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:12 smithi136 ceph-mon[115078]: osdmap e1007: 8 total, 8 up, 8 in 2024-08-17T17:44:12.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:12 smithi136 ceph-mon[115078]: pgmap v1508: 105 pgs: 105 active+clean; 4.5 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 50 KiB/s wr, 2 op/s 2024-08-17T17:44:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:12 smithi165 ceph-mon[105517]: osdmap e1007: 8 total, 8 up, 8 in 2024-08-17T17:44:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:12 smithi165 ceph-mon[105517]: pgmap v1508: 105 pgs: 105 active+clean; 4.5 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 50 KiB/s wr, 2 op/s 2024-08-17T17:44:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:13 smithi165 ceph-mon[105517]: osdmap e1008: 8 total, 8 up, 8 in 2024-08-17T17:44:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:13 smithi136 ceph-mon[109861]: osdmap e1008: 8 total, 8 up, 8 in 2024-08-17T17:44:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:13 smithi136 ceph-mon[115078]: osdmap e1008: 8 total, 8 up, 8 in 2024-08-17T17:44:14.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:14 smithi136 ceph-mon[109861]: osdmap e1009: 8 total, 8 up, 8 in 2024-08-17T17:44:14.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:14 smithi136 ceph-mon[109861]: pgmap v1511: 105 pgs: 105 active+clean; 4.5 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 50 KiB/s wr, 2 op/s 2024-08-17T17:44:14.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:14 smithi136 ceph-mon[115078]: osdmap e1009: 8 total, 8 up, 8 in 2024-08-17T17:44:14.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:14 smithi136 ceph-mon[115078]: pgmap v1511: 105 pgs: 105 active+clean; 4.5 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 50 KiB/s wr, 2 op/s 2024-08-17T17:44:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:14 smithi165 ceph-mon[105517]: osdmap e1009: 8 total, 8 up, 8 in 2024-08-17T17:44:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:14 smithi165 ceph-mon[105517]: pgmap v1511: 105 pgs: 105 active+clean; 4.5 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 50 KiB/s wr, 2 op/s 2024-08-17T17:44:15.587 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:44:15.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:15 smithi136 ceph-mon[109861]: osdmap e1010: 8 total, 8 up, 8 in 2024-08-17T17:44:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:15 smithi136 ceph-mon[115078]: osdmap e1010: 8 total, 8 up, 8 in 2024-08-17T17:44:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:15 smithi165 ceph-mon[105517]: osdmap e1010: 8 total, 8 up, 8 in 2024-08-17T17:44:16.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:16 smithi136 ceph-mon[109861]: osdmap e1011: 8 total, 8 up, 8 in 2024-08-17T17:44:16.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:16 smithi136 ceph-mon[109861]: pgmap v1514: 105 pgs: 105 active+clean; 18 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 209 KiB/s wr, 49 op/s 2024-08-17T17:44:16.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:16 smithi136 ceph-mon[115078]: osdmap e1011: 8 total, 8 up, 8 in 2024-08-17T17:44:16.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:16 smithi136 ceph-mon[115078]: pgmap v1514: 105 pgs: 105 active+clean; 18 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 209 KiB/s wr, 49 op/s 2024-08-17T17:44:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:16 smithi165 ceph-mon[105517]: osdmap e1011: 8 total, 8 up, 8 in 2024-08-17T17:44:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:16 smithi165 ceph-mon[105517]: pgmap v1514: 105 pgs: 105 active+clean; 18 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 209 KiB/s wr, 49 op/s 2024-08-17T17:44:17.848 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:17 smithi136 ceph-mon[109861]: osdmap e1012: 8 total, 8 up, 8 in 2024-08-17T17:44:17.848 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:44:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:44:17] "GET /metrics HTTP/1.1" 200 36493 "" "Prometheus/2.51.0" 2024-08-17T17:44:17.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:17 smithi136 ceph-mon[115078]: osdmap e1012: 8 total, 8 up, 8 in 2024-08-17T17:44:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:17 smithi165 ceph-mon[105517]: osdmap e1012: 8 total, 8 up, 8 in 2024-08-17T17:44:18.603 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:44:18.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:18 smithi136 ceph-mon[109861]: osdmap e1013: 8 total, 8 up, 8 in 2024-08-17T17:44:18.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:18 smithi136 ceph-mon[109861]: pgmap v1517: 105 pgs: 105 active+clean; 18 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 221 KiB/s wr, 124 op/s 2024-08-17T17:44:18.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:18 smithi136 ceph-mon[115078]: osdmap e1013: 8 total, 8 up, 8 in 2024-08-17T17:44:18.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:18 smithi136 ceph-mon[115078]: pgmap v1517: 105 pgs: 105 active+clean; 18 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 221 KiB/s wr, 124 op/s 2024-08-17T17:44:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:18 smithi165 ceph-mon[105517]: osdmap e1013: 8 total, 8 up, 8 in 2024-08-17T17:44:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:18 smithi165 ceph-mon[105517]: pgmap v1517: 105 pgs: 105 active+clean; 18 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 221 KiB/s wr, 124 op/s 2024-08-17T17:44:19.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:19 smithi136 ceph-mon[115078]: osdmap e1014: 8 total, 8 up, 8 in 2024-08-17T17:44:19.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:19 smithi136 ceph-mon[115078]: osdmap e1015: 8 total, 8 up, 8 in 2024-08-17T17:44:19.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:19 smithi136 ceph-mon[109861]: osdmap e1014: 8 total, 8 up, 8 in 2024-08-17T17:44:19.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:19 smithi136 ceph-mon[109861]: osdmap e1015: 8 total, 8 up, 8 in 2024-08-17T17:44:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:19 smithi165 ceph-mon[105517]: osdmap e1014: 8 total, 8 up, 8 in 2024-08-17T17:44:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:19 smithi165 ceph-mon[105517]: osdmap e1015: 8 total, 8 up, 8 in 2024-08-17T17:44:21.173 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:44:21.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:21 smithi136 ceph-mon[115078]: osdmap e1016: 8 total, 8 up, 8 in 2024-08-17T17:44:21.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:21 smithi136 ceph-mon[115078]: pgmap v1521: 105 pgs: 105 active+clean; 18 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 13 KiB/s wr, 77 op/s 2024-08-17T17:44:21.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:21 smithi136 ceph-mon[109861]: osdmap e1016: 8 total, 8 up, 8 in 2024-08-17T17:44:21.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:21 smithi136 ceph-mon[109861]: pgmap v1521: 105 pgs: 105 active+clean; 18 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 13 KiB/s wr, 77 op/s 2024-08-17T17:44:21.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:21 smithi165 ceph-mon[105517]: osdmap e1016: 8 total, 8 up, 8 in 2024-08-17T17:44:21.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:21 smithi165 ceph-mon[105517]: pgmap v1521: 105 pgs: 105 active+clean; 18 MiB data, 666 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 13 KiB/s wr, 77 op/s 2024-08-17T17:44:22.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:22 smithi136 ceph-mon[109861]: osdmap e1017: 8 total, 8 up, 8 in 2024-08-17T17:44:22.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:44:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:22 smithi136 ceph-mon[115078]: osdmap e1017: 8 total, 8 up, 8 in 2024-08-17T17:44:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:44:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:22 smithi165 ceph-mon[105517]: osdmap e1017: 8 total, 8 up, 8 in 2024-08-17T17:44:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:44:23.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:23 smithi136 ceph-mon[109861]: osdmap e1018: 8 total, 8 up, 8 in 2024-08-17T17:44:23.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:23 smithi136 ceph-mon[109861]: pgmap v1524: 105 pgs: 105 active+clean; 22 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 8.0 KiB/s wr, 80 op/s 2024-08-17T17:44:23.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:23 smithi136 ceph-mon[115078]: osdmap e1018: 8 total, 8 up, 8 in 2024-08-17T17:44:23.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:23 smithi136 ceph-mon[115078]: pgmap v1524: 105 pgs: 105 active+clean; 22 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 8.0 KiB/s wr, 80 op/s 2024-08-17T17:44:23.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:23 smithi165 ceph-mon[105517]: osdmap e1018: 8 total, 8 up, 8 in 2024-08-17T17:44:23.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:23 smithi165 ceph-mon[105517]: pgmap v1524: 105 pgs: 105 active+clean; 22 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 8.0 KiB/s wr, 80 op/s 2024-08-17T17:44:24.186 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:44:24.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:24 smithi136 ceph-mon[109861]: osdmap e1019: 8 total, 8 up, 8 in 2024-08-17T17:44:24.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:24 smithi136 ceph-mon[115078]: osdmap e1019: 8 total, 8 up, 8 in 2024-08-17T17:44:24.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:24 smithi165 ceph-mon[105517]: osdmap e1019: 8 total, 8 up, 8 in 2024-08-17T17:44:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:25 smithi136 ceph-mon[109861]: osdmap e1020: 8 total, 8 up, 8 in 2024-08-17T17:44:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:25 smithi136 ceph-mon[109861]: pgmap v1527: 105 pgs: 105 active+clean; 22 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 7.7 KiB/s wr, 77 op/s 2024-08-17T17:44:25.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:25 smithi136 ceph-mon[115078]: osdmap e1020: 8 total, 8 up, 8 in 2024-08-17T17:44:25.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:25 smithi136 ceph-mon[115078]: pgmap v1527: 105 pgs: 105 active+clean; 22 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 7.7 KiB/s wr, 77 op/s 2024-08-17T17:44:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:25 smithi165 ceph-mon[105517]: osdmap e1020: 8 total, 8 up, 8 in 2024-08-17T17:44:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:25 smithi165 ceph-mon[105517]: pgmap v1527: 105 pgs: 105 active+clean; 22 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 7.7 KiB/s wr, 77 op/s 2024-08-17T17:44:26.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:26 smithi136 ceph-mon[109861]: osdmap e1021: 8 total, 8 up, 8 in 2024-08-17T17:44:26.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:26 smithi136 ceph-mon[115078]: osdmap e1021: 8 total, 8 up, 8 in 2024-08-17T17:44:26.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:26 smithi165 ceph-mon[105517]: osdmap e1021: 8 total, 8 up, 8 in 2024-08-17T17:44:27.214 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/0.DiffIterateDeterministic (14650 ms) 2024-08-17T17:44:27.214 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/0.DiffIterateDeterministicPP 2024-08-17T17:44:27.214 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:44:27.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:27 smithi136 ceph-mon[109861]: osdmap e1022: 8 total, 8 up, 8 in 2024-08-17T17:44:27.449 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:27 smithi136 ceph-mon[109861]: pgmap v1530: 105 pgs: 105 active+clean; 24 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 4.0 KiB/s wr, 65 op/s 2024-08-17T17:44:27.450 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:27 smithi136 ceph-mon[115078]: osdmap e1022: 8 total, 8 up, 8 in 2024-08-17T17:44:27.450 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:27 smithi136 ceph-mon[115078]: pgmap v1530: 105 pgs: 105 active+clean; 24 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 4.0 KiB/s wr, 65 op/s 2024-08-17T17:44:27.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:27 smithi165 ceph-mon[105517]: osdmap e1022: 8 total, 8 up, 8 in 2024-08-17T17:44:27.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:27 smithi165 ceph-mon[105517]: pgmap v1530: 105 pgs: 105 active+clean; 24 MiB data, 671 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 4.0 KiB/s wr, 65 op/s 2024-08-17T17:44:27.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:44:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:44:27] "GET /metrics HTTP/1.1" 200 36477 "" "Prometheus/2.51.0" 2024-08-17T17:44:28.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:28 smithi136 ceph-mon[109861]: osdmap e1023: 8 total, 8 up, 8 in 2024-08-17T17:44:28.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:28 smithi136 ceph-mon[115078]: osdmap e1023: 8 total, 8 up, 8 in 2024-08-17T17:44:28.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:28 smithi165 ceph-mon[105517]: osdmap e1023: 8 total, 8 up, 8 in 2024-08-17T17:44:29.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:29 smithi136 ceph-mon[109861]: osdmap e1024: 8 total, 8 up, 8 in 2024-08-17T17:44:29.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:29 smithi136 ceph-mon[109861]: pgmap v1533: 105 pgs: 105 active+clean; 28 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 9.5 KiB/s wr, 109 op/s 2024-08-17T17:44:29.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:29 smithi136 ceph-mon[115078]: osdmap e1024: 8 total, 8 up, 8 in 2024-08-17T17:44:29.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:29 smithi136 ceph-mon[115078]: pgmap v1533: 105 pgs: 105 active+clean; 28 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 9.5 KiB/s wr, 109 op/s 2024-08-17T17:44:29.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:29 smithi165 ceph-mon[105517]: osdmap e1024: 8 total, 8 up, 8 in 2024-08-17T17:44:29.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:29 smithi165 ceph-mon[105517]: pgmap v1533: 105 pgs: 105 active+clean; 28 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 9.5 KiB/s wr, 109 op/s 2024-08-17T17:44:30.228 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:44:30.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:30 smithi136 ceph-mon[109861]: osdmap e1025: 8 total, 8 up, 8 in 2024-08-17T17:44:30.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:30 smithi136 ceph-mon[115078]: osdmap e1025: 8 total, 8 up, 8 in 2024-08-17T17:44:30.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:30 smithi165 ceph-mon[105517]: osdmap e1025: 8 total, 8 up, 8 in 2024-08-17T17:44:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:31 smithi165 ceph-mon[105517]: osdmap e1026: 8 total, 8 up, 8 in 2024-08-17T17:44:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:31 smithi165 ceph-mon[105517]: pgmap v1536: 105 pgs: 105 active+clean; 28 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 5.5 KiB/s wr, 43 op/s 2024-08-17T17:44:31.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:31 smithi136 ceph-mon[115078]: osdmap e1026: 8 total, 8 up, 8 in 2024-08-17T17:44:31.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:31 smithi136 ceph-mon[115078]: pgmap v1536: 105 pgs: 105 active+clean; 28 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 5.5 KiB/s wr, 43 op/s 2024-08-17T17:44:31.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:31 smithi136 ceph-mon[109861]: osdmap e1026: 8 total, 8 up, 8 in 2024-08-17T17:44:31.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:31 smithi136 ceph-mon[109861]: pgmap v1536: 105 pgs: 105 active+clean; 28 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 5.5 KiB/s wr, 43 op/s 2024-08-17T17:44:32.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:32 smithi165 ceph-mon[105517]: osdmap e1027: 8 total, 8 up, 8 in 2024-08-17T17:44:32.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:32 smithi136 ceph-mon[109861]: osdmap e1027: 8 total, 8 up, 8 in 2024-08-17T17:44:32.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:32 smithi136 ceph-mon[115078]: osdmap e1027: 8 total, 8 up, 8 in 2024-08-17T17:44:33.254 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:44:33.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:33 smithi165 ceph-mon[105517]: osdmap e1028: 8 total, 8 up, 8 in 2024-08-17T17:44:33.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:33 smithi165 ceph-mon[105517]: pgmap v1539: 105 pgs: 105 active+clean; 32 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 8.0 KiB/s wr, 91 op/s 2024-08-17T17:44:33.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:33 smithi136 ceph-mon[115078]: osdmap e1028: 8 total, 8 up, 8 in 2024-08-17T17:44:33.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:33 smithi136 ceph-mon[115078]: pgmap v1539: 105 pgs: 105 active+clean; 32 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 8.0 KiB/s wr, 91 op/s 2024-08-17T17:44:33.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:33 smithi136 ceph-mon[109861]: osdmap e1028: 8 total, 8 up, 8 in 2024-08-17T17:44:33.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:33 smithi136 ceph-mon[109861]: pgmap v1539: 105 pgs: 105 active+clean; 32 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 8.0 KiB/s wr, 91 op/s 2024-08-17T17:44:34.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:34 smithi165 ceph-mon[105517]: osdmap e1029: 8 total, 8 up, 8 in 2024-08-17T17:44:34.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:34 smithi136 ceph-mon[109861]: osdmap e1029: 8 total, 8 up, 8 in 2024-08-17T17:44:34.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:34 smithi136 ceph-mon[115078]: osdmap e1029: 8 total, 8 up, 8 in 2024-08-17T17:44:35.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:35 smithi165 ceph-mon[105517]: osdmap e1030: 8 total, 8 up, 8 in 2024-08-17T17:44:35.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:35 smithi165 ceph-mon[105517]: pgmap v1542: 105 pgs: 105 active+clean; 32 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 8.0 KiB/s wr, 91 op/s 2024-08-17T17:44:35.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:35 smithi136 ceph-mon[109861]: osdmap e1030: 8 total, 8 up, 8 in 2024-08-17T17:44:35.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:35 smithi136 ceph-mon[109861]: pgmap v1542: 105 pgs: 105 active+clean; 32 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 8.0 KiB/s wr, 91 op/s 2024-08-17T17:44:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:35 smithi136 ceph-mon[115078]: osdmap e1030: 8 total, 8 up, 8 in 2024-08-17T17:44:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:35 smithi136 ceph-mon[115078]: pgmap v1542: 105 pgs: 105 active+clean; 32 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 8.0 KiB/s wr, 91 op/s 2024-08-17T17:44:36.277 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:44:36.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:36 smithi165 ceph-mon[105517]: osdmap e1031: 8 total, 8 up, 8 in 2024-08-17T17:44:36.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:44:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:36 smithi136 ceph-mon[109861]: osdmap e1031: 8 total, 8 up, 8 in 2024-08-17T17:44:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:44:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:36 smithi136 ceph-mon[115078]: osdmap e1031: 8 total, 8 up, 8 in 2024-08-17T17:44:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:44:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:37 smithi165 ceph-mon[105517]: osdmap e1032: 8 total, 8 up, 8 in 2024-08-17T17:44:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:37 smithi165 ceph-mon[105517]: pgmap v1545: 105 pgs: 105 active+clean; 35 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 5.7 KiB/s wr, 53 op/s 2024-08-17T17:44:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:37 smithi136 ceph-mon[109861]: osdmap e1032: 8 total, 8 up, 8 in 2024-08-17T17:44:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:37 smithi136 ceph-mon[109861]: pgmap v1545: 105 pgs: 105 active+clean; 35 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 5.7 KiB/s wr, 53 op/s 2024-08-17T17:44:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:37 smithi136 ceph-mon[115078]: osdmap e1032: 8 total, 8 up, 8 in 2024-08-17T17:44:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:37 smithi136 ceph-mon[115078]: pgmap v1545: 105 pgs: 105 active+clean; 35 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 5.7 KiB/s wr, 53 op/s 2024-08-17T17:44:37.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:44:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:44:37] "GET /metrics HTTP/1.1" 200 36478 "" "Prometheus/2.51.0" 2024-08-17T17:44:38.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:38 smithi165 ceph-mon[105517]: osdmap e1033: 8 total, 8 up, 8 in 2024-08-17T17:44:38.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:38 smithi136 ceph-mon[109861]: osdmap e1033: 8 total, 8 up, 8 in 2024-08-17T17:44:38.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:38 smithi136 ceph-mon[115078]: osdmap e1033: 8 total, 8 up, 8 in 2024-08-17T17:44:39.300 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:44:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:39 smithi136 ceph-mon[109861]: osdmap e1034: 8 total, 8 up, 8 in 2024-08-17T17:44:39.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:39 smithi136 ceph-mon[109861]: pgmap v1548: 105 pgs: 105 active+clean; 44 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 9.5 KiB/s wr, 109 op/s 2024-08-17T17:44:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:39 smithi136 ceph-mon[115078]: osdmap e1034: 8 total, 8 up, 8 in 2024-08-17T17:44:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:39 smithi136 ceph-mon[115078]: pgmap v1548: 105 pgs: 105 active+clean; 44 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 9.5 KiB/s wr, 109 op/s 2024-08-17T17:44:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:39 smithi165 ceph-mon[105517]: osdmap e1034: 8 total, 8 up, 8 in 2024-08-17T17:44:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:39 smithi165 ceph-mon[105517]: pgmap v1548: 105 pgs: 105 active+clean; 44 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 9.5 KiB/s wr, 109 op/s 2024-08-17T17:44:40.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:40 smithi136 ceph-mon[109861]: osdmap e1035: 8 total, 8 up, 8 in 2024-08-17T17:44:40.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:40 smithi136 ceph-mon[115078]: osdmap e1035: 8 total, 8 up, 8 in 2024-08-17T17:44:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:40 smithi165 ceph-mon[105517]: osdmap e1035: 8 total, 8 up, 8 in 2024-08-17T17:44:41.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:41 smithi136 ceph-mon[115078]: osdmap e1036: 8 total, 8 up, 8 in 2024-08-17T17:44:41.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:41 smithi136 ceph-mon[115078]: pgmap v1551: 105 pgs: 105 active+clean; 44 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 3.7 KiB/s wr, 56 op/s 2024-08-17T17:44:41.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:41 smithi136 ceph-mon[109861]: osdmap e1036: 8 total, 8 up, 8 in 2024-08-17T17:44:41.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:41 smithi136 ceph-mon[109861]: pgmap v1551: 105 pgs: 105 active+clean; 44 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 3.7 KiB/s wr, 56 op/s 2024-08-17T17:44:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:41 smithi165 ceph-mon[105517]: osdmap e1036: 8 total, 8 up, 8 in 2024-08-17T17:44:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:41 smithi165 ceph-mon[105517]: pgmap v1551: 105 pgs: 105 active+clean; 44 MiB data, 673 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 3.7 KiB/s wr, 56 op/s 2024-08-17T17:44:42.315 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/0.DiffIterateDeterministicPP (15102 ms) 2024-08-17T17:44:42.315 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/0.DiffIterateDeterministicLUKS1 2024-08-17T17:44:42.315 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:44:42.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:42 smithi136 ceph-mon[109861]: osdmap e1037: 8 total, 8 up, 8 in 2024-08-17T17:44:42.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:42 smithi136 ceph-mon[115078]: osdmap e1037: 8 total, 8 up, 8 in 2024-08-17T17:44:42.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:42 smithi165 ceph-mon[105517]: osdmap e1037: 8 total, 8 up, 8 in 2024-08-17T17:44:43.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:43 smithi136 ceph-mon[109861]: osdmap e1038: 8 total, 8 up, 8 in 2024-08-17T17:44:43.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:43 smithi136 ceph-mon[109861]: pgmap v1554: 105 pgs: 105 active+clean; 58 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 9.0 KiB/s wr, 89 op/s 2024-08-17T17:44:43.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:43 smithi136 ceph-mon[115078]: osdmap e1038: 8 total, 8 up, 8 in 2024-08-17T17:44:43.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:43 smithi136 ceph-mon[115078]: pgmap v1554: 105 pgs: 105 active+clean; 58 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 9.0 KiB/s wr, 89 op/s 2024-08-17T17:44:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:43 smithi165 ceph-mon[105517]: osdmap e1038: 8 total, 8 up, 8 in 2024-08-17T17:44:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:43 smithi165 ceph-mon[105517]: pgmap v1554: 105 pgs: 105 active+clean; 58 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 9.0 KiB/s wr, 89 op/s 2024-08-17T17:44:44.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:44 smithi136 ceph-mon[109861]: pgmap v1555: 105 pgs: 105 active+clean; 58 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 7.0 KiB/s wr, 69 op/s 2024-08-17T17:44:44.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:44 smithi136 ceph-mon[115078]: pgmap v1555: 105 pgs: 105 active+clean; 58 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 7.0 KiB/s wr, 69 op/s 2024-08-17T17:44:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:44 smithi165 ceph-mon[105517]: pgmap v1555: 105 pgs: 105 active+clean; 58 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 7.0 KiB/s wr, 69 op/s 2024-08-17T17:44:47.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:46 smithi136 ceph-mon[109861]: osdmap e1039: 8 total, 8 up, 8 in 2024-08-17T17:44:47.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:46 smithi136 ceph-mon[109861]: pgmap v1557: 105 pgs: 105 active+clean; 62 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 689 KiB/s wr, 62 op/s 2024-08-17T17:44:47.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:46 smithi136 ceph-mon[115078]: osdmap e1039: 8 total, 8 up, 8 in 2024-08-17T17:44:47.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:46 smithi136 ceph-mon[115078]: pgmap v1557: 105 pgs: 105 active+clean; 62 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 689 KiB/s wr, 62 op/s 2024-08-17T17:44:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:46 smithi165 ceph-mon[105517]: osdmap e1039: 8 total, 8 up, 8 in 2024-08-17T17:44:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:46 smithi165 ceph-mon[105517]: pgmap v1557: 105 pgs: 105 active+clean; 62 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 689 KiB/s wr, 62 op/s 2024-08-17T17:44:47.795 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:44:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:44:47] "GET /metrics HTTP/1.1" 200 36478 "" "Prometheus/2.51.0" 2024-08-17T17:44:47.820 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:44:48.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:47 smithi136 ceph-mon[109861]: osdmap e1040: 8 total, 8 up, 8 in 2024-08-17T17:44:48.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:47 smithi136 ceph-mon[115078]: osdmap e1040: 8 total, 8 up, 8 in 2024-08-17T17:44:48.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:47 smithi165 ceph-mon[105517]: osdmap e1040: 8 total, 8 up, 8 in 2024-08-17T17:44:49.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:48 smithi136 ceph-mon[109861]: osdmap e1041: 8 total, 8 up, 8 in 2024-08-17T17:44:49.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:48 smithi136 ceph-mon[109861]: pgmap v1560: 105 pgs: 105 active+clean; 62 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-08-17T17:44:49.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:48 smithi136 ceph-mon[115078]: osdmap e1041: 8 total, 8 up, 8 in 2024-08-17T17:44:49.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:48 smithi136 ceph-mon[115078]: pgmap v1560: 105 pgs: 105 active+clean; 62 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-08-17T17:44:49.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:48 smithi165 ceph-mon[105517]: osdmap e1041: 8 total, 8 up, 8 in 2024-08-17T17:44:49.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:48 smithi165 ceph-mon[105517]: pgmap v1560: 105 pgs: 105 active+clean; 62 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-08-17T17:44:50.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:50 smithi136 ceph-mon[109861]: pgmap v1561: 105 pgs: 105 active+clean; 62 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-08-17T17:44:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:50 smithi136 ceph-mon[115078]: pgmap v1561: 105 pgs: 105 active+clean; 62 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-08-17T17:44:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:50 smithi165 ceph-mon[105517]: pgmap v1561: 105 pgs: 105 active+clean; 62 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-08-17T17:44:51.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:44:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:44:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:44:52.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:52 smithi136 ceph-mon[109861]: osdmap e1042: 8 total, 8 up, 8 in 2024-08-17T17:44:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:52 smithi136 ceph-mon[109861]: pgmap v1563: 105 pgs: 105 active+clean; 62 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 6.2 KiB/s wr, 62 op/s 2024-08-17T17:44:52.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:52 smithi136 ceph-mon[115078]: osdmap e1042: 8 total, 8 up, 8 in 2024-08-17T17:44:52.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:52 smithi136 ceph-mon[115078]: pgmap v1563: 105 pgs: 105 active+clean; 62 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 6.2 KiB/s wr, 62 op/s 2024-08-17T17:44:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:52 smithi165 ceph-mon[105517]: osdmap e1042: 8 total, 8 up, 8 in 2024-08-17T17:44:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:52 smithi165 ceph-mon[105517]: pgmap v1563: 105 pgs: 105 active+clean; 62 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 6.2 KiB/s wr, 62 op/s 2024-08-17T17:44:53.590 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:44:53.930 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:53 smithi136 ceph-mon[109861]: osdmap e1043: 8 total, 8 up, 8 in 2024-08-17T17:44:53.930 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:44:53.931 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:53 smithi136 ceph-mon[115078]: osdmap e1043: 8 total, 8 up, 8 in 2024-08-17T17:44:53.931 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:44:53.987 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:53 smithi165 ceph-mon[105517]: osdmap e1043: 8 total, 8 up, 8 in 2024-08-17T17:44:53.987 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:44:54.905 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:54 smithi165 ceph-mon[105517]: osdmap e1044: 8 total, 8 up, 8 in 2024-08-17T17:44:54.906 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:54 smithi165 ceph-mon[105517]: pgmap v1566: 105 pgs: 105 active+clean; 62 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 2.3 KiB/s wr, 24 op/s 2024-08-17T17:44:54.925 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:54 smithi136 ceph-mon[109861]: osdmap e1044: 8 total, 8 up, 8 in 2024-08-17T17:44:54.925 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:54 smithi136 ceph-mon[109861]: pgmap v1566: 105 pgs: 105 active+clean; 62 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 2.3 KiB/s wr, 24 op/s 2024-08-17T17:44:54.925 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:54 smithi136 ceph-mon[115078]: osdmap e1044: 8 total, 8 up, 8 in 2024-08-17T17:44:54.925 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:54 smithi136 ceph-mon[115078]: pgmap v1566: 105 pgs: 105 active+clean; 62 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 2.3 KiB/s wr, 24 op/s 2024-08-17T17:44:56.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:56 smithi165 ceph-mon[105517]: pgmap v1567: 105 pgs: 105 active+clean; 67 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 686 KiB/s wr, 48 op/s 2024-08-17T17:44:56.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:56 smithi136 ceph-mon[109861]: pgmap v1567: 105 pgs: 105 active+clean; 67 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 686 KiB/s wr, 48 op/s 2024-08-17T17:44:56.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:56 smithi136 ceph-mon[115078]: pgmap v1567: 105 pgs: 105 active+clean; 67 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 686 KiB/s wr, 48 op/s 2024-08-17T17:44:57.893 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:44:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:44:57] "GET /metrics HTTP/1.1" 200 36480 "" "Prometheus/2.51.0" 2024-08-17T17:44:58.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:57 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:44:58.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:57 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:44:58.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:57 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:44:58.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:57 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:44:58.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:57 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:44:58.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:57 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:44:58.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:57 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:44:58.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:57 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:44:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:57 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:44:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:57 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:44:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:57 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:44:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:57 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:44:59.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:58 smithi136 ceph-mon[109861]: osdmap e1045: 8 total, 8 up, 8 in 2024-08-17T17:44:59.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:58 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:44:59.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:58 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:44:59.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:58 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:44:59.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:58 smithi136 ceph-mon[109861]: pgmap v1569: 105 pgs: 105 active+clean; 68 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 686 KiB/s wr, 36 op/s 2024-08-17T17:44:59.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:58 smithi136 ceph-mon[115078]: osdmap e1045: 8 total, 8 up, 8 in 2024-08-17T17:44:59.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:58 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:44:59.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:58 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:44:59.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:58 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:44:59.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:58 smithi136 ceph-mon[115078]: pgmap v1569: 105 pgs: 105 active+clean; 68 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 686 KiB/s wr, 36 op/s 2024-08-17T17:44:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:58 smithi165 ceph-mon[105517]: osdmap e1045: 8 total, 8 up, 8 in 2024-08-17T17:44:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:58 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:44:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:58 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:44:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:58 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:44:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:58 smithi165 ceph-mon[105517]: pgmap v1569: 105 pgs: 105 active+clean; 68 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 686 KiB/s wr, 36 op/s 2024-08-17T17:44:59.949 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:45:00.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:44:59 smithi136 ceph-mon[109861]: osdmap e1046: 8 total, 8 up, 8 in 2024-08-17T17:45:00.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:44:59 smithi136 ceph-mon[115078]: osdmap e1046: 8 total, 8 up, 8 in 2024-08-17T17:45:00.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:44:59 smithi165 ceph-mon[105517]: osdmap e1046: 8 total, 8 up, 8 in 2024-08-17T17:45:01.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:00 smithi136 ceph-mon[115078]: osdmap e1047: 8 total, 8 up, 8 in 2024-08-17T17:45:01.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:00 smithi136 ceph-mon[115078]: pgmap v1572: 105 pgs: 105 active+clean; 68 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 686 KiB/s wr, 36 op/s 2024-08-17T17:45:01.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:00 smithi136 ceph-mon[109861]: osdmap e1047: 8 total, 8 up, 8 in 2024-08-17T17:45:01.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:00 smithi136 ceph-mon[109861]: pgmap v1572: 105 pgs: 105 active+clean; 68 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 686 KiB/s wr, 36 op/s 2024-08-17T17:45:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:00 smithi165 ceph-mon[105517]: osdmap e1047: 8 total, 8 up, 8 in 2024-08-17T17:45:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:00 smithi165 ceph-mon[105517]: pgmap v1572: 105 pgs: 105 active+clean; 68 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 686 KiB/s wr, 36 op/s 2024-08-17T17:45:02.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:02 smithi165 ceph-mon[105517]: pgmap v1573: 105 pgs: 105 active+clean; 74 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 688 KiB/s wr, 52 op/s 2024-08-17T17:45:02.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:02 smithi136 ceph-mon[115078]: pgmap v1573: 105 pgs: 105 active+clean; 74 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 688 KiB/s wr, 52 op/s 2024-08-17T17:45:02.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:02 smithi136 ceph-mon[109861]: pgmap v1573: 105 pgs: 105 active+clean; 74 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 688 KiB/s wr, 52 op/s 2024-08-17T17:45:04.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:04 smithi136 ceph-mon[109861]: osdmap e1048: 8 total, 8 up, 8 in 2024-08-17T17:45:04.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:04 smithi136 ceph-mon[109861]: pgmap v1575: 105 pgs: 105 active+clean; 74 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 39 op/s 2024-08-17T17:45:04.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:04 smithi136 ceph-mon[115078]: osdmap e1048: 8 total, 8 up, 8 in 2024-08-17T17:45:04.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:04 smithi136 ceph-mon[115078]: pgmap v1575: 105 pgs: 105 active+clean; 74 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 39 op/s 2024-08-17T17:45:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:04 smithi165 ceph-mon[105517]: osdmap e1048: 8 total, 8 up, 8 in 2024-08-17T17:45:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:04 smithi165 ceph-mon[105517]: pgmap v1575: 105 pgs: 105 active+clean; 74 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 39 op/s 2024-08-17T17:45:05.590 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:45:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:05 smithi136 ceph-mon[109861]: osdmap e1049: 8 total, 8 up, 8 in 2024-08-17T17:45:05.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:05 smithi136 ceph-mon[115078]: osdmap e1049: 8 total, 8 up, 8 in 2024-08-17T17:45:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:05 smithi165 ceph-mon[105517]: osdmap e1049: 8 total, 8 up, 8 in 2024-08-17T17:45:06.950 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:06 smithi136 ceph-mon[109861]: osdmap e1050: 8 total, 8 up, 8 in 2024-08-17T17:45:06.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:45:06.951 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:06 smithi136 ceph-mon[109861]: pgmap v1578: 105 pgs: 105 active+clean; 76 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 689 KiB/s wr, 52 op/s 2024-08-17T17:45:06.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:06 smithi136 ceph-mon[115078]: osdmap e1050: 8 total, 8 up, 8 in 2024-08-17T17:45:06.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:45:06.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:06 smithi136 ceph-mon[115078]: pgmap v1578: 105 pgs: 105 active+clean; 76 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 689 KiB/s wr, 52 op/s 2024-08-17T17:45:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:06 smithi165 ceph-mon[105517]: osdmap e1050: 8 total, 8 up, 8 in 2024-08-17T17:45:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:45:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:06 smithi165 ceph-mon[105517]: pgmap v1578: 105 pgs: 105 active+clean; 76 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 689 KiB/s wr, 52 op/s 2024-08-17T17:45:07.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:45:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:45:07] "GET /metrics HTTP/1.1" 200 36482 "" "Prometheus/2.51.0" 2024-08-17T17:45:08.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:08 smithi136 ceph-mon[109861]: pgmap v1579: 105 pgs: 105 active+clean; 80 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 686 KiB/s wr, 35 op/s 2024-08-17T17:45:08.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:08 smithi136 ceph-mon[115078]: pgmap v1579: 105 pgs: 105 active+clean; 80 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 686 KiB/s wr, 35 op/s 2024-08-17T17:45:09.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:08 smithi165 ceph-mon[105517]: pgmap v1579: 105 pgs: 105 active+clean; 80 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 686 KiB/s wr, 35 op/s 2024-08-17T17:45:10.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:10 smithi136 ceph-mon[109861]: osdmap e1051: 8 total, 8 up, 8 in 2024-08-17T17:45:10.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:10 smithi136 ceph-mon[109861]: pgmap v1581: 105 pgs: 105 active+clean; 80 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 686 KiB/s wr, 35 op/s 2024-08-17T17:45:10.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:10 smithi136 ceph-mon[115078]: osdmap e1051: 8 total, 8 up, 8 in 2024-08-17T17:45:10.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:10 smithi136 ceph-mon[115078]: pgmap v1581: 105 pgs: 105 active+clean; 80 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 686 KiB/s wr, 35 op/s 2024-08-17T17:45:11.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:10 smithi165 ceph-mon[105517]: osdmap e1051: 8 total, 8 up, 8 in 2024-08-17T17:45:11.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:10 smithi165 ceph-mon[105517]: pgmap v1581: 105 pgs: 105 active+clean; 80 MiB data, 724 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 686 KiB/s wr, 35 op/s 2024-08-17T17:45:11.592 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/0.DiffIterateDeterministicLUKS1 (29277 ms) 2024-08-17T17:45:11.592 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/0.DiffIterateDeterministicLUKS1PP 2024-08-17T17:45:11.592 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:45:11.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:11 smithi136 ceph-mon[109861]: osdmap e1052: 8 total, 8 up, 8 in 2024-08-17T17:45:11.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:11 smithi136 ceph-mon[115078]: osdmap e1052: 8 total, 8 up, 8 in 2024-08-17T17:45:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:11 smithi165 ceph-mon[105517]: osdmap e1052: 8 total, 8 up, 8 in 2024-08-17T17:45:12.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:12 smithi165 ceph-mon[105517]: osdmap e1053: 8 total, 8 up, 8 in 2024-08-17T17:45:12.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:12 smithi165 ceph-mon[105517]: pgmap v1584: 105 pgs: 105 active+clean; 92 MiB data, 736 MiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 1.3 MiB/s wr, 52 op/s 2024-08-17T17:45:12.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:12 smithi136 ceph-mon[115078]: osdmap e1053: 8 total, 8 up, 8 in 2024-08-17T17:45:12.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:12 smithi136 ceph-mon[115078]: pgmap v1584: 105 pgs: 105 active+clean; 92 MiB data, 736 MiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 1.3 MiB/s wr, 52 op/s 2024-08-17T17:45:12.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:12 smithi136 ceph-mon[109861]: osdmap e1053: 8 total, 8 up, 8 in 2024-08-17T17:45:12.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:12 smithi136 ceph-mon[109861]: pgmap v1584: 105 pgs: 105 active+clean; 92 MiB data, 736 MiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 1.3 MiB/s wr, 52 op/s 2024-08-17T17:45:14.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:14 smithi136 ceph-mon[109861]: pgmap v1585: 105 pgs: 105 active+clean; 92 MiB data, 736 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 686 KiB/s wr, 30 op/s 2024-08-17T17:45:14.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:14 smithi136 ceph-mon[115078]: pgmap v1585: 105 pgs: 105 active+clean; 92 MiB data, 736 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 686 KiB/s wr, 30 op/s 2024-08-17T17:45:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:14 smithi165 ceph-mon[105517]: pgmap v1585: 105 pgs: 105 active+clean; 92 MiB data, 736 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 686 KiB/s wr, 30 op/s 2024-08-17T17:45:15.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:45:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=cleanup t=2024-08-17T17:45:15.059570031Z level=info msg="Completed cleanup jobs" duration=21.479581ms 2024-08-17T17:45:15.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:45:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=plugins.update.checker t=2024-08-17T17:45:15.18282765Z level=info msg="Update check succeeded" duration=60.378644ms 2024-08-17T17:45:15.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:45:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=grafana.update.checker t=2024-08-17T17:45:15.200288116Z level=info msg="Update check succeeded" duration=69.437177ms 2024-08-17T17:45:16.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:16 smithi136 ceph-mon[109861]: osdmap e1054: 8 total, 8 up, 8 in 2024-08-17T17:45:16.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:16 smithi136 ceph-mon[109861]: pgmap v1587: 105 pgs: 105 active+clean; 96 MiB data, 745 MiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 1.3 MiB/s wr, 45 op/s 2024-08-17T17:45:16.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:16 smithi136 ceph-mon[115078]: osdmap e1054: 8 total, 8 up, 8 in 2024-08-17T17:45:16.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:16 smithi136 ceph-mon[115078]: pgmap v1587: 105 pgs: 105 active+clean; 96 MiB data, 745 MiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 1.3 MiB/s wr, 45 op/s 2024-08-17T17:45:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:16 smithi165 ceph-mon[105517]: osdmap e1054: 8 total, 8 up, 8 in 2024-08-17T17:45:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:16 smithi165 ceph-mon[105517]: pgmap v1587: 105 pgs: 105 active+clean; 96 MiB data, 745 MiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 1.3 MiB/s wr, 45 op/s 2024-08-17T17:45:17.595 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:45:17.847 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:17 smithi136 ceph-mon[109861]: osdmap e1055: 8 total, 8 up, 8 in 2024-08-17T17:45:17.847 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:17 smithi136 ceph-mon[115078]: osdmap e1055: 8 total, 8 up, 8 in 2024-08-17T17:45:17.847 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:45:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:45:17] "GET /metrics HTTP/1.1" 200 36482 "" "Prometheus/2.51.0" 2024-08-17T17:45:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:17 smithi165 ceph-mon[105517]: osdmap e1055: 8 total, 8 up, 8 in 2024-08-17T17:45:18.862 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:18 smithi136 ceph-mon[109861]: osdmap e1056: 8 total, 8 up, 8 in 2024-08-17T17:45:18.862 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:18 smithi136 ceph-mon[109861]: pgmap v1590: 105 pgs: 105 active+clean; 96 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 686 KiB/s wr, 34 op/s 2024-08-17T17:45:18.862 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:18 smithi136 ceph-mon[115078]: osdmap e1056: 8 total, 8 up, 8 in 2024-08-17T17:45:18.863 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:18 smithi136 ceph-mon[115078]: pgmap v1590: 105 pgs: 105 active+clean; 96 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 686 KiB/s wr, 34 op/s 2024-08-17T17:45:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:18 smithi165 ceph-mon[105517]: osdmap e1056: 8 total, 8 up, 8 in 2024-08-17T17:45:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:18 smithi165 ceph-mon[105517]: pgmap v1590: 105 pgs: 105 active+clean; 96 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 686 KiB/s wr, 34 op/s 2024-08-17T17:45:20.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:20 smithi136 ceph-mon[109861]: pgmap v1591: 105 pgs: 105 active+clean; 96 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 686 KiB/s wr, 34 op/s 2024-08-17T17:45:20.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:20 smithi136 ceph-mon[115078]: pgmap v1591: 105 pgs: 105 active+clean; 96 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 686 KiB/s wr, 34 op/s 2024-08-17T17:45:21.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:20 smithi165 ceph-mon[105517]: pgmap v1591: 105 pgs: 105 active+clean; 96 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 686 KiB/s wr, 34 op/s 2024-08-17T17:45:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:45:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:45:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:45:22.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:22 smithi136 ceph-mon[109861]: osdmap e1057: 8 total, 8 up, 8 in 2024-08-17T17:45:22.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:22 smithi136 ceph-mon[109861]: pgmap v1593: 105 pgs: 105 active+clean; 103 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 688 KiB/s wr, 50 op/s 2024-08-17T17:45:22.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:22 smithi136 ceph-mon[115078]: osdmap e1057: 8 total, 8 up, 8 in 2024-08-17T17:45:22.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:22 smithi136 ceph-mon[115078]: pgmap v1593: 105 pgs: 105 active+clean; 103 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 688 KiB/s wr, 50 op/s 2024-08-17T17:45:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:22 smithi165 ceph-mon[105517]: osdmap e1057: 8 total, 8 up, 8 in 2024-08-17T17:45:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:22 smithi165 ceph-mon[105517]: pgmap v1593: 105 pgs: 105 active+clean; 103 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 688 KiB/s wr, 50 op/s 2024-08-17T17:45:23.594 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:45:23.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:23 smithi136 ceph-mon[109861]: osdmap e1058: 8 total, 8 up, 8 in 2024-08-17T17:45:23.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:23 smithi136 ceph-mon[115078]: osdmap e1058: 8 total, 8 up, 8 in 2024-08-17T17:45:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:23 smithi165 ceph-mon[105517]: osdmap e1058: 8 total, 8 up, 8 in 2024-08-17T17:45:24.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:24 smithi136 ceph-mon[109861]: osdmap e1059: 8 total, 8 up, 8 in 2024-08-17T17:45:24.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:24 smithi136 ceph-mon[109861]: pgmap v1596: 105 pgs: 105 active+clean; 103 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 685 KiB/s wr, 30 op/s 2024-08-17T17:45:24.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:24 smithi136 ceph-mon[115078]: osdmap e1059: 8 total, 8 up, 8 in 2024-08-17T17:45:24.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:24 smithi136 ceph-mon[115078]: pgmap v1596: 105 pgs: 105 active+clean; 103 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 685 KiB/s wr, 30 op/s 2024-08-17T17:45:25.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:24 smithi165 ceph-mon[105517]: osdmap e1059: 8 total, 8 up, 8 in 2024-08-17T17:45:25.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:24 smithi165 ceph-mon[105517]: pgmap v1596: 105 pgs: 105 active+clean; 103 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 685 KiB/s wr, 30 op/s 2024-08-17T17:45:26.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:26 smithi136 ceph-mon[109861]: pgmap v1597: 105 pgs: 105 active+clean; 103 MiB data, 761 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 686 KiB/s wr, 34 op/s 2024-08-17T17:45:26.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:26 smithi136 ceph-mon[115078]: pgmap v1597: 105 pgs: 105 active+clean; 103 MiB data, 761 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 686 KiB/s wr, 34 op/s 2024-08-17T17:45:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:26 smithi165 ceph-mon[105517]: pgmap v1597: 105 pgs: 105 active+clean; 103 MiB data, 761 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 686 KiB/s wr, 34 op/s 2024-08-17T17:45:27.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:45:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:45:27] "GET /metrics HTTP/1.1" 200 36479 "" "Prometheus/2.51.0" 2024-08-17T17:45:28.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:28 smithi136 ceph-mon[109861]: osdmap e1060: 8 total, 8 up, 8 in 2024-08-17T17:45:28.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:28 smithi136 ceph-mon[109861]: pgmap v1599: 105 pgs: 105 active+clean; 106 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 2.7 KiB/s wr, 30 op/s 2024-08-17T17:45:28.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:28 smithi136 ceph-mon[115078]: osdmap e1060: 8 total, 8 up, 8 in 2024-08-17T17:45:28.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:28 smithi136 ceph-mon[115078]: pgmap v1599: 105 pgs: 105 active+clean; 106 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 2.7 KiB/s wr, 30 op/s 2024-08-17T17:45:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:28 smithi165 ceph-mon[105517]: osdmap e1060: 8 total, 8 up, 8 in 2024-08-17T17:45:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:28 smithi165 ceph-mon[105517]: pgmap v1599: 105 pgs: 105 active+clean; 106 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 2.7 KiB/s wr, 30 op/s 2024-08-17T17:45:29.593 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:45:29.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:29 smithi136 ceph-mon[109861]: osdmap e1061: 8 total, 8 up, 8 in 2024-08-17T17:45:29.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:29 smithi136 ceph-mon[115078]: osdmap e1061: 8 total, 8 up, 8 in 2024-08-17T17:45:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:29 smithi165 ceph-mon[105517]: osdmap e1061: 8 total, 8 up, 8 in 2024-08-17T17:45:30.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:30 smithi136 ceph-mon[109861]: osdmap e1062: 8 total, 8 up, 8 in 2024-08-17T17:45:30.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:30 smithi136 ceph-mon[109861]: pgmap v1602: 105 pgs: 105 active+clean; 106 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 2.7 KiB/s wr, 30 op/s 2024-08-17T17:45:30.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:30 smithi136 ceph-mon[115078]: osdmap e1062: 8 total, 8 up, 8 in 2024-08-17T17:45:30.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:30 smithi136 ceph-mon[115078]: pgmap v1602: 105 pgs: 105 active+clean; 106 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 2.7 KiB/s wr, 30 op/s 2024-08-17T17:45:31.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:30 smithi165 ceph-mon[105517]: osdmap e1062: 8 total, 8 up, 8 in 2024-08-17T17:45:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:30 smithi165 ceph-mon[105517]: pgmap v1602: 105 pgs: 105 active+clean; 106 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 2.7 KiB/s wr, 30 op/s 2024-08-17T17:45:32.813 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:32 smithi165 ceph-mon[105517]: pgmap v1603: 105 pgs: 105 active+clean; 116 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 42 KiB/s rd, 688 KiB/s wr, 66 op/s 2024-08-17T17:45:32.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:32 smithi136 ceph-mon[109861]: pgmap v1603: 105 pgs: 105 active+clean; 116 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 42 KiB/s rd, 688 KiB/s wr, 66 op/s 2024-08-17T17:45:32.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:32 smithi136 ceph-mon[115078]: pgmap v1603: 105 pgs: 105 active+clean; 116 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 42 KiB/s rd, 688 KiB/s wr, 66 op/s 2024-08-17T17:45:34.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:34 smithi136 ceph-mon[109861]: osdmap e1063: 8 total, 8 up, 8 in 2024-08-17T17:45:34.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:34 smithi136 ceph-mon[109861]: pgmap v1605: 105 pgs: 105 active+clean; 116 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 686 KiB/s wr, 39 op/s 2024-08-17T17:45:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:34 smithi136 ceph-mon[115078]: osdmap e1063: 8 total, 8 up, 8 in 2024-08-17T17:45:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:34 smithi136 ceph-mon[115078]: pgmap v1605: 105 pgs: 105 active+clean; 116 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 686 KiB/s wr, 39 op/s 2024-08-17T17:45:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:34 smithi165 ceph-mon[105517]: osdmap e1063: 8 total, 8 up, 8 in 2024-08-17T17:45:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:34 smithi165 ceph-mon[105517]: pgmap v1605: 105 pgs: 105 active+clean; 116 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 686 KiB/s wr, 39 op/s 2024-08-17T17:45:35.593 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:45:35.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:35 smithi136 ceph-mon[115078]: osdmap e1064: 8 total, 8 up, 8 in 2024-08-17T17:45:35.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:35 smithi136 ceph-mon[109861]: osdmap e1064: 8 total, 8 up, 8 in 2024-08-17T17:45:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:35 smithi165 ceph-mon[105517]: osdmap e1064: 8 total, 8 up, 8 in 2024-08-17T17:45:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:36 smithi136 ceph-mon[109861]: osdmap e1065: 8 total, 8 up, 8 in 2024-08-17T17:45:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:45:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:36 smithi136 ceph-mon[109861]: pgmap v1608: 105 pgs: 105 active+clean; 126 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 1.3 MiB/s wr, 53 op/s 2024-08-17T17:45:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:36 smithi136 ceph-mon[115078]: osdmap e1065: 8 total, 8 up, 8 in 2024-08-17T17:45:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:45:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:36 smithi136 ceph-mon[115078]: pgmap v1608: 105 pgs: 105 active+clean; 126 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 1.3 MiB/s wr, 53 op/s 2024-08-17T17:45:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:36 smithi165 ceph-mon[105517]: osdmap e1065: 8 total, 8 up, 8 in 2024-08-17T17:45:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:45:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:36 smithi165 ceph-mon[105517]: pgmap v1608: 105 pgs: 105 active+clean; 126 MiB data, 762 MiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 1.3 MiB/s wr, 53 op/s 2024-08-17T17:45:37.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:45:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:45:37] "GET /metrics HTTP/1.1" 200 36482 "" "Prometheus/2.51.0" 2024-08-17T17:45:38.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:38 smithi136 ceph-mon[109861]: pgmap v1609: 105 pgs: 105 active+clean; 126 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 686 KiB/s wr, 35 op/s 2024-08-17T17:45:38.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:38 smithi136 ceph-mon[115078]: pgmap v1609: 105 pgs: 105 active+clean; 126 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 686 KiB/s wr, 35 op/s 2024-08-17T17:45:39.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:38 smithi165 ceph-mon[105517]: pgmap v1609: 105 pgs: 105 active+clean; 126 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 686 KiB/s wr, 35 op/s 2024-08-17T17:45:40.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:40 smithi136 ceph-mon[115078]: osdmap e1066: 8 total, 8 up, 8 in 2024-08-17T17:45:40.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:40 smithi136 ceph-mon[115078]: pgmap v1611: 105 pgs: 105 active+clean; 126 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 686 KiB/s wr, 35 op/s 2024-08-17T17:45:40.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:40 smithi136 ceph-mon[109861]: osdmap e1066: 8 total, 8 up, 8 in 2024-08-17T17:45:40.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:40 smithi136 ceph-mon[109861]: pgmap v1611: 105 pgs: 105 active+clean; 126 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 686 KiB/s wr, 35 op/s 2024-08-17T17:45:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:40 smithi165 ceph-mon[105517]: osdmap e1066: 8 total, 8 up, 8 in 2024-08-17T17:45:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:40 smithi165 ceph-mon[105517]: pgmap v1611: 105 pgs: 105 active+clean; 126 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 686 KiB/s wr, 35 op/s 2024-08-17T17:45:41.594 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/0.DiffIterateDeterministicLUKS1PP (30001 ms) 2024-08-17T17:45:41.594 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/0.DiffIterateDeterministicLUKS2 2024-08-17T17:45:41.594 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:45:41.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:41 smithi136 ceph-mon[109861]: osdmap e1067: 8 total, 8 up, 8 in 2024-08-17T17:45:41.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:41 smithi136 ceph-mon[115078]: osdmap e1067: 8 total, 8 up, 8 in 2024-08-17T17:45:42.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:41 smithi165 ceph-mon[105517]: osdmap e1067: 8 total, 8 up, 8 in 2024-08-17T17:45:42.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:42 smithi136 ceph-mon[109861]: osdmap e1068: 8 total, 8 up, 8 in 2024-08-17T17:45:42.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:42 smithi136 ceph-mon[109861]: pgmap v1614: 105 pgs: 105 active+clean; 138 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 688 KiB/s wr, 50 op/s 2024-08-17T17:45:42.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:42 smithi136 ceph-mon[115078]: osdmap e1068: 8 total, 8 up, 8 in 2024-08-17T17:45:42.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:42 smithi136 ceph-mon[115078]: pgmap v1614: 105 pgs: 105 active+clean; 138 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 688 KiB/s wr, 50 op/s 2024-08-17T17:45:43.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:42 smithi165 ceph-mon[105517]: osdmap e1068: 8 total, 8 up, 8 in 2024-08-17T17:45:43.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:42 smithi165 ceph-mon[105517]: pgmap v1614: 105 pgs: 105 active+clean; 138 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 688 KiB/s wr, 50 op/s 2024-08-17T17:45:44.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:44 smithi136 ceph-mon[109861]: pgmap v1615: 105 pgs: 105 active+clean; 138 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 686 KiB/s wr, 28 op/s 2024-08-17T17:45:44.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:44 smithi136 ceph-mon[115078]: pgmap v1615: 105 pgs: 105 active+clean; 138 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 686 KiB/s wr, 28 op/s 2024-08-17T17:45:45.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:44 smithi165 ceph-mon[105517]: pgmap v1615: 105 pgs: 105 active+clean; 138 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 686 KiB/s wr, 28 op/s 2024-08-17T17:45:46.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:46 smithi136 ceph-mon[109861]: pgmap v1616: 105 pgs: 105 active+clean; 138 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 597 KiB/s wr, 38 op/s 2024-08-17T17:45:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:46 smithi136 ceph-mon[115078]: pgmap v1616: 105 pgs: 105 active+clean; 138 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 597 KiB/s wr, 38 op/s 2024-08-17T17:45:47.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:46 smithi165 ceph-mon[105517]: pgmap v1616: 105 pgs: 105 active+clean; 138 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 597 KiB/s wr, 38 op/s 2024-08-17T17:45:47.848 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:45:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:45:47] "GET /metrics HTTP/1.1" 200 36482 "" "Prometheus/2.51.0" 2024-08-17T17:45:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:48 smithi165 ceph-mon[105517]: osdmap e1069: 8 total, 8 up, 8 in 2024-08-17T17:45:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:48 smithi165 ceph-mon[105517]: pgmap v1618: 105 pgs: 105 active+clean; 138 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 522 KiB/s wr, 33 op/s 2024-08-17T17:45:49.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:48 smithi136 ceph-mon[115078]: osdmap e1069: 8 total, 8 up, 8 in 2024-08-17T17:45:49.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:48 smithi136 ceph-mon[115078]: pgmap v1618: 105 pgs: 105 active+clean; 138 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 522 KiB/s wr, 33 op/s 2024-08-17T17:45:49.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:48 smithi136 ceph-mon[109861]: osdmap e1069: 8 total, 8 up, 8 in 2024-08-17T17:45:49.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:48 smithi136 ceph-mon[109861]: pgmap v1618: 105 pgs: 105 active+clean; 138 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 522 KiB/s wr, 33 op/s 2024-08-17T17:45:49.765 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:45:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:49 smithi165 ceph-mon[105517]: osdmap e1070: 8 total, 8 up, 8 in 2024-08-17T17:45:50.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:49 smithi136 ceph-mon[115078]: osdmap e1070: 8 total, 8 up, 8 in 2024-08-17T17:45:50.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:49 smithi136 ceph-mon[109861]: osdmap e1070: 8 total, 8 up, 8 in 2024-08-17T17:45:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:50 smithi165 ceph-mon[105517]: osdmap e1071: 8 total, 8 up, 8 in 2024-08-17T17:45:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:50 smithi165 ceph-mon[105517]: pgmap v1621: 105 pgs: 105 active+clean; 138 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 511 B/s wr, 15 op/s 2024-08-17T17:45:51.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:50 smithi136 ceph-mon[109861]: osdmap e1071: 8 total, 8 up, 8 in 2024-08-17T17:45:51.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:50 smithi136 ceph-mon[109861]: pgmap v1621: 105 pgs: 105 active+clean; 138 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 511 B/s wr, 15 op/s 2024-08-17T17:45:51.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:50 smithi136 ceph-mon[115078]: osdmap e1071: 8 total, 8 up, 8 in 2024-08-17T17:45:51.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:50 smithi136 ceph-mon[115078]: pgmap v1621: 105 pgs: 105 active+clean; 138 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 511 B/s wr, 15 op/s 2024-08-17T17:45:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:45:52.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:45:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:45:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:52 smithi165 ceph-mon[105517]: pgmap v1622: 105 pgs: 105 active+clean; 154 MiB data, 831 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 MiB/s wr, 44 op/s 2024-08-17T17:45:53.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:52 smithi136 ceph-mon[109861]: pgmap v1622: 105 pgs: 105 active+clean; 154 MiB data, 831 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 MiB/s wr, 44 op/s 2024-08-17T17:45:53.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:52 smithi136 ceph-mon[115078]: pgmap v1622: 105 pgs: 105 active+clean; 154 MiB data, 831 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 MiB/s wr, 44 op/s 2024-08-17T17:45:54.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:54 smithi136 ceph-mon[109861]: pgmap v1623: 105 pgs: 105 active+clean; 154 MiB data, 831 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.4 MiB/s wr, 39 op/s 2024-08-17T17:45:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:54 smithi136 ceph-mon[115078]: pgmap v1623: 105 pgs: 105 active+clean; 154 MiB data, 831 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.4 MiB/s wr, 39 op/s 2024-08-17T17:45:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:54 smithi165 ceph-mon[105517]: pgmap v1623: 105 pgs: 105 active+clean; 154 MiB data, 831 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.4 MiB/s wr, 39 op/s 2024-08-17T17:45:57.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:57 smithi136 ceph-mon[115078]: osdmap e1072: 8 total, 8 up, 8 in 2024-08-17T17:45:57.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:57 smithi136 ceph-mon[115078]: pgmap v1625: 105 pgs: 105 active+clean; 154 MiB data, 831 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.1 MiB/s wr, 34 op/s 2024-08-17T17:45:57.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:57 smithi136 ceph-mon[109861]: osdmap e1072: 8 total, 8 up, 8 in 2024-08-17T17:45:57.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:57 smithi136 ceph-mon[109861]: pgmap v1625: 105 pgs: 105 active+clean; 154 MiB data, 831 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.1 MiB/s wr, 34 op/s 2024-08-17T17:45:57.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:57 smithi165 ceph-mon[105517]: osdmap e1072: 8 total, 8 up, 8 in 2024-08-17T17:45:57.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:57 smithi165 ceph-mon[105517]: pgmap v1625: 105 pgs: 105 active+clean; 154 MiB data, 831 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.1 MiB/s wr, 34 op/s 2024-08-17T17:45:57.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:45:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:45:57] "GET /metrics HTTP/1.1" 200 36479 "" "Prometheus/2.51.0" 2024-08-17T17:45:58.179 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:45:58.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:58 smithi136 ceph-mon[109861]: osdmap e1073: 8 total, 8 up, 8 in 2024-08-17T17:45:58.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:58 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:45:58.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:58 smithi136 ceph-mon[115078]: osdmap e1073: 8 total, 8 up, 8 in 2024-08-17T17:45:58.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:58 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:45:58.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:58 smithi165 ceph-mon[105517]: osdmap e1073: 8 total, 8 up, 8 in 2024-08-17T17:45:58.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:58 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:45:59.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:59 smithi136 ceph-mon[109861]: osdmap e1074: 8 total, 8 up, 8 in 2024-08-17T17:45:59.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:59 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:45:59.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:59 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:45:59.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:45:59 smithi136 ceph-mon[109861]: pgmap v1628: 105 pgs: 105 active+clean; 163 MiB data, 851 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.3 MiB/s wr, 8 op/s 2024-08-17T17:45:59.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:59 smithi136 ceph-mon[115078]: osdmap e1074: 8 total, 8 up, 8 in 2024-08-17T17:45:59.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:59 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:45:59.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:59 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:45:59.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:45:59 smithi136 ceph-mon[115078]: pgmap v1628: 105 pgs: 105 active+clean; 163 MiB data, 851 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.3 MiB/s wr, 8 op/s 2024-08-17T17:45:59.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:59 smithi165 ceph-mon[105517]: osdmap e1074: 8 total, 8 up, 8 in 2024-08-17T17:45:59.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:59 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:45:59.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:59 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:45:59.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:45:59 smithi165 ceph-mon[105517]: pgmap v1628: 105 pgs: 105 active+clean; 163 MiB data, 851 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.3 MiB/s wr, 8 op/s 2024-08-17T17:46:00.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:00 smithi136 ceph-mon[109861]: pgmap v1629: 105 pgs: 105 active+clean; 163 MiB data, 851 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.3 MiB/s wr, 8 op/s 2024-08-17T17:46:00.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:00 smithi136 ceph-mon[115078]: pgmap v1629: 105 pgs: 105 active+clean; 163 MiB data, 851 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.3 MiB/s wr, 8 op/s 2024-08-17T17:46:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:00 smithi165 ceph-mon[105517]: pgmap v1629: 105 pgs: 105 active+clean; 163 MiB data, 851 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.3 MiB/s wr, 8 op/s 2024-08-17T17:46:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:03 smithi165 ceph-mon[105517]: pgmap v1630: 105 pgs: 105 active+clean; 172 MiB data, 897 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.5 MiB/s wr, 42 op/s 2024-08-17T17:46:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:46:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:46:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:46:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:46:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:46:03.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:46:03.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:46:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:03 smithi136 ceph-mon[115078]: pgmap v1630: 105 pgs: 105 active+clean; 172 MiB data, 897 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.5 MiB/s wr, 42 op/s 2024-08-17T17:46:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:46:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:46:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:46:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:46:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:46:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:46:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:46:03.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:03 smithi136 ceph-mon[109861]: pgmap v1630: 105 pgs: 105 active+clean; 172 MiB data, 897 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.5 MiB/s wr, 42 op/s 2024-08-17T17:46:03.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:46:03.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:46:03.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:46:03.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:46:03.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:46:03.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:46:03.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:46:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:05 smithi165 ceph-mon[105517]: pgmap v1631: 105 pgs: 105 active+clean; 172 MiB data, 897 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.0 MiB/s wr, 33 op/s 2024-08-17T17:46:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:05 smithi136 ceph-mon[109861]: pgmap v1631: 105 pgs: 105 active+clean; 172 MiB data, 897 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.0 MiB/s wr, 33 op/s 2024-08-17T17:46:05.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:05 smithi136 ceph-mon[115078]: pgmap v1631: 105 pgs: 105 active+clean; 172 MiB data, 897 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.0 MiB/s wr, 33 op/s 2024-08-17T17:46:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:46:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:46:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:46:07.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:07 smithi165 ceph-mon[105517]: pgmap v1632: 105 pgs: 105 active+clean; 184 MiB data, 913 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 2.6 MiB/s wr, 28 op/s 2024-08-17T17:46:07.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:07 smithi165 ceph-mon[105517]: osdmap e1075: 8 total, 8 up, 8 in 2024-08-17T17:46:07.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:07 smithi136 ceph-mon[109861]: pgmap v1632: 105 pgs: 105 active+clean; 184 MiB data, 913 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 2.6 MiB/s wr, 28 op/s 2024-08-17T17:46:07.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:07 smithi136 ceph-mon[109861]: osdmap e1075: 8 total, 8 up, 8 in 2024-08-17T17:46:07.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:46:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:46:07] "GET /metrics HTTP/1.1" 200 36486 "" "Prometheus/2.51.0" 2024-08-17T17:46:07.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:07 smithi136 ceph-mon[115078]: pgmap v1632: 105 pgs: 105 active+clean; 184 MiB data, 913 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 2.6 MiB/s wr, 28 op/s 2024-08-17T17:46:07.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:07 smithi136 ceph-mon[115078]: osdmap e1075: 8 total, 8 up, 8 in 2024-08-17T17:46:08.528 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:46:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:08 smithi165 ceph-mon[105517]: osdmap e1076: 8 total, 8 up, 8 in 2024-08-17T17:46:08.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:08 smithi136 ceph-mon[109861]: osdmap e1076: 8 total, 8 up, 8 in 2024-08-17T17:46:08.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:08 smithi136 ceph-mon[115078]: osdmap e1076: 8 total, 8 up, 8 in 2024-08-17T17:46:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:09 smithi165 ceph-mon[105517]: pgmap v1635: 105 pgs: 105 active+clean; 185 MiB data, 933 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.5 MiB/s wr, 37 op/s 2024-08-17T17:46:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:09 smithi165 ceph-mon[105517]: osdmap e1077: 8 total, 8 up, 8 in 2024-08-17T17:46:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:09 smithi136 ceph-mon[109861]: pgmap v1635: 105 pgs: 105 active+clean; 185 MiB data, 933 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.5 MiB/s wr, 37 op/s 2024-08-17T17:46:09.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:09 smithi136 ceph-mon[109861]: osdmap e1077: 8 total, 8 up, 8 in 2024-08-17T17:46:09.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:09 smithi136 ceph-mon[115078]: pgmap v1635: 105 pgs: 105 active+clean; 185 MiB data, 933 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.5 MiB/s wr, 37 op/s 2024-08-17T17:46:09.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:09 smithi136 ceph-mon[115078]: osdmap e1077: 8 total, 8 up, 8 in 2024-08-17T17:46:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:11 smithi165 ceph-mon[105517]: pgmap v1637: 105 pgs: 105 active+clean; 185 MiB data, 933 MiB used, 714 GiB / 715 GiB avail; 9.0 KiB/s rd, 2.0 MiB/s wr, 14 op/s 2024-08-17T17:46:11.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:11 smithi136 ceph-mon[109861]: pgmap v1637: 105 pgs: 105 active+clean; 185 MiB data, 933 MiB used, 714 GiB / 715 GiB avail; 9.0 KiB/s rd, 2.0 MiB/s wr, 14 op/s 2024-08-17T17:46:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:11 smithi136 ceph-mon[115078]: pgmap v1637: 105 pgs: 105 active+clean; 185 MiB data, 933 MiB used, 714 GiB / 715 GiB avail; 9.0 KiB/s rd, 2.0 MiB/s wr, 14 op/s 2024-08-17T17:46:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:13 smithi165 ceph-mon[105517]: pgmap v1638: 105 pgs: 105 active+clean; 190 MiB data, 946 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 690 KiB/s wr, 44 op/s 2024-08-17T17:46:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:13 smithi136 ceph-mon[109861]: pgmap v1638: 105 pgs: 105 active+clean; 190 MiB data, 946 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 690 KiB/s wr, 44 op/s 2024-08-17T17:46:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:13 smithi136 ceph-mon[115078]: pgmap v1638: 105 pgs: 105 active+clean; 190 MiB data, 946 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 690 KiB/s wr, 44 op/s 2024-08-17T17:46:14.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:14 smithi136 ceph-mon[109861]: pgmap v1639: 105 pgs: 105 active+clean; 190 MiB data, 946 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 519 KiB/s wr, 33 op/s 2024-08-17T17:46:14.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:14 smithi136 ceph-mon[115078]: pgmap v1639: 105 pgs: 105 active+clean; 190 MiB data, 946 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 519 KiB/s wr, 33 op/s 2024-08-17T17:46:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:14 smithi165 ceph-mon[105517]: pgmap v1639: 105 pgs: 105 active+clean; 190 MiB data, 946 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 519 KiB/s wr, 33 op/s 2024-08-17T17:46:15.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:15 smithi136 ceph-mon[109861]: osdmap e1078: 8 total, 8 up, 8 in 2024-08-17T17:46:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:15 smithi136 ceph-mon[115078]: osdmap e1078: 8 total, 8 up, 8 in 2024-08-17T17:46:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:15 smithi165 ceph-mon[105517]: osdmap e1078: 8 total, 8 up, 8 in 2024-08-17T17:46:16.556 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:46:16.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:16 smithi136 ceph-mon[109861]: osdmap e1079: 8 total, 8 up, 8 in 2024-08-17T17:46:16.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:16 smithi136 ceph-mon[109861]: pgmap v1642: 105 pgs: 105 active+clean; 200 MiB data, 970 MiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 1.5 MiB/s wr, 31 op/s 2024-08-17T17:46:16.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:16 smithi136 ceph-mon[115078]: osdmap e1079: 8 total, 8 up, 8 in 2024-08-17T17:46:16.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:16 smithi136 ceph-mon[115078]: pgmap v1642: 105 pgs: 105 active+clean; 200 MiB data, 970 MiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 1.5 MiB/s wr, 31 op/s 2024-08-17T17:46:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:16 smithi165 ceph-mon[105517]: osdmap e1079: 8 total, 8 up, 8 in 2024-08-17T17:46:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:16 smithi165 ceph-mon[105517]: pgmap v1642: 105 pgs: 105 active+clean; 200 MiB data, 970 MiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 1.5 MiB/s wr, 31 op/s 2024-08-17T17:46:17.847 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:17 smithi136 ceph-mon[109861]: osdmap e1080: 8 total, 8 up, 8 in 2024-08-17T17:46:17.847 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:17 smithi136 ceph-mon[115078]: osdmap e1080: 8 total, 8 up, 8 in 2024-08-17T17:46:17.848 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:46:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:46:17] "GET /metrics HTTP/1.1" 200 36486 "" "Prometheus/2.51.0" 2024-08-17T17:46:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:17 smithi165 ceph-mon[105517]: osdmap e1080: 8 total, 8 up, 8 in 2024-08-17T17:46:18.871 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:18 smithi136 ceph-mon[109861]: pgmap v1644: 105 pgs: 105 active+clean; 208 MiB data, 994 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.7 MiB/s wr, 37 op/s 2024-08-17T17:46:18.872 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:18 smithi136 ceph-mon[115078]: pgmap v1644: 105 pgs: 105 active+clean; 208 MiB data, 994 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.7 MiB/s wr, 37 op/s 2024-08-17T17:46:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:18 smithi165 ceph-mon[105517]: pgmap v1644: 105 pgs: 105 active+clean; 208 MiB data, 994 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.7 MiB/s wr, 37 op/s 2024-08-17T17:46:20.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:20 smithi136 ceph-mon[109861]: pgmap v1645: 105 pgs: 105 active+clean; 208 MiB data, 994 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.7 MiB/s wr, 37 op/s 2024-08-17T17:46:20.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:20 smithi136 ceph-mon[115078]: pgmap v1645: 105 pgs: 105 active+clean; 208 MiB data, 994 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.7 MiB/s wr, 37 op/s 2024-08-17T17:46:21.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:20 smithi165 ceph-mon[105517]: pgmap v1645: 105 pgs: 105 active+clean; 208 MiB data, 994 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.7 MiB/s wr, 37 op/s 2024-08-17T17:46:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:46:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:46:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:46:22.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:22 smithi136 ceph-mon[109861]: pgmap v1646: 105 pgs: 105 active+clean; 208 MiB data, 994 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.0 MiB/s wr, 33 op/s 2024-08-17T17:46:22.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:22 smithi136 ceph-mon[115078]: pgmap v1646: 105 pgs: 105 active+clean; 208 MiB data, 994 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.0 MiB/s wr, 33 op/s 2024-08-17T17:46:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:22 smithi165 ceph-mon[105517]: pgmap v1646: 105 pgs: 105 active+clean; 208 MiB data, 994 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.0 MiB/s wr, 33 op/s 2024-08-17T17:46:23.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:23 smithi136 ceph-mon[109861]: osdmap e1081: 8 total, 8 up, 8 in 2024-08-17T17:46:23.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:23 smithi136 ceph-mon[115078]: osdmap e1081: 8 total, 8 up, 8 in 2024-08-17T17:46:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:23 smithi165 ceph-mon[105517]: osdmap e1081: 8 total, 8 up, 8 in 2024-08-17T17:46:24.610 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/0.DiffIterateDeterministicLUKS2 (43017 ms) 2024-08-17T17:46:24.653 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/0.DiffIterateDeterministicLUKS2PP 2024-08-17T17:46:24.653 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:46:24.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:24 smithi136 ceph-mon[109861]: osdmap e1082: 8 total, 8 up, 8 in 2024-08-17T17:46:24.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:24 smithi136 ceph-mon[109861]: pgmap v1649: 105 pgs: 105 active+clean; 208 MiB data, 994 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 1.0 MiB/s wr, 24 op/s 2024-08-17T17:46:24.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:24 smithi136 ceph-mon[115078]: osdmap e1082: 8 total, 8 up, 8 in 2024-08-17T17:46:24.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:24 smithi136 ceph-mon[115078]: pgmap v1649: 105 pgs: 105 active+clean; 208 MiB data, 994 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 1.0 MiB/s wr, 24 op/s 2024-08-17T17:46:25.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:24 smithi165 ceph-mon[105517]: osdmap e1082: 8 total, 8 up, 8 in 2024-08-17T17:46:25.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:24 smithi165 ceph-mon[105517]: pgmap v1649: 105 pgs: 105 active+clean; 208 MiB data, 994 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 1.0 MiB/s wr, 24 op/s 2024-08-17T17:46:25.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:25 smithi136 ceph-mon[109861]: osdmap e1083: 8 total, 8 up, 8 in 2024-08-17T17:46:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:25 smithi136 ceph-mon[115078]: osdmap e1083: 8 total, 8 up, 8 in 2024-08-17T17:46:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:25 smithi165 ceph-mon[105517]: osdmap e1083: 8 total, 8 up, 8 in 2024-08-17T17:46:26.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:26 smithi136 ceph-mon[109861]: pgmap v1651: 105 pgs: 105 active+clean; 224 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.3 MiB/s wr, 14 op/s 2024-08-17T17:46:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:26 smithi136 ceph-mon[115078]: pgmap v1651: 105 pgs: 105 active+clean; 224 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.3 MiB/s wr, 14 op/s 2024-08-17T17:46:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:26 smithi165 ceph-mon[105517]: pgmap v1651: 105 pgs: 105 active+clean; 224 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.3 MiB/s wr, 14 op/s 2024-08-17T17:46:27.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:46:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:46:27] "GET /metrics HTTP/1.1" 200 36489 "" "Prometheus/2.51.0" 2024-08-17T17:46:28.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:28 smithi136 ceph-mon[109861]: pgmap v1652: 105 pgs: 105 active+clean; 232 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 MiB/s wr, 44 op/s 2024-08-17T17:46:28.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:28 smithi136 ceph-mon[115078]: pgmap v1652: 105 pgs: 105 active+clean; 232 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 MiB/s wr, 44 op/s 2024-08-17T17:46:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:28 smithi165 ceph-mon[105517]: pgmap v1652: 105 pgs: 105 active+clean; 232 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 MiB/s wr, 44 op/s 2024-08-17T17:46:31.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:31 smithi136 ceph-mon[109861]: osdmap e1084: 8 total, 8 up, 8 in 2024-08-17T17:46:31.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:31 smithi136 ceph-mon[109861]: pgmap v1654: 105 pgs: 105 active+clean; 232 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.3 MiB/s wr, 38 op/s 2024-08-17T17:46:31.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:31 smithi136 ceph-mon[115078]: osdmap e1084: 8 total, 8 up, 8 in 2024-08-17T17:46:31.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:31 smithi136 ceph-mon[115078]: pgmap v1654: 105 pgs: 105 active+clean; 232 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.3 MiB/s wr, 38 op/s 2024-08-17T17:46:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:31 smithi165 ceph-mon[105517]: osdmap e1084: 8 total, 8 up, 8 in 2024-08-17T17:46:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:31 smithi165 ceph-mon[105517]: pgmap v1654: 105 pgs: 105 active+clean; 232 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.3 MiB/s wr, 38 op/s 2024-08-17T17:46:32.063 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:46:32.412 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:32 smithi165 ceph-mon[105517]: osdmap e1085: 8 total, 8 up, 8 in 2024-08-17T17:46:32.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:32 smithi136 ceph-mon[109861]: osdmap e1085: 8 total, 8 up, 8 in 2024-08-17T17:46:32.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:32 smithi136 ceph-mon[115078]: osdmap e1085: 8 total, 8 up, 8 in 2024-08-17T17:46:33.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:33 smithi136 ceph-mon[109861]: osdmap e1086: 8 total, 8 up, 8 in 2024-08-17T17:46:33.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:33 smithi136 ceph-mon[109861]: pgmap v1657: 105 pgs: 105 active+clean; 248 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 42 KiB/s rd, 4.0 MiB/s wr, 61 op/s 2024-08-17T17:46:33.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:33 smithi136 ceph-mon[115078]: osdmap e1086: 8 total, 8 up, 8 in 2024-08-17T17:46:33.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:33 smithi136 ceph-mon[115078]: pgmap v1657: 105 pgs: 105 active+clean; 248 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 42 KiB/s rd, 4.0 MiB/s wr, 61 op/s 2024-08-17T17:46:33.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:33 smithi165 ceph-mon[105517]: osdmap e1086: 8 total, 8 up, 8 in 2024-08-17T17:46:33.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:33 smithi165 ceph-mon[105517]: pgmap v1657: 105 pgs: 105 active+clean; 248 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 42 KiB/s rd, 4.0 MiB/s wr, 61 op/s 2024-08-17T17:46:34.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:34 smithi136 ceph-mon[115078]: pgmap v1658: 105 pgs: 105 active+clean; 248 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 24 op/s 2024-08-17T17:46:34.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:34 smithi136 ceph-mon[109861]: pgmap v1658: 105 pgs: 105 active+clean; 248 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 24 op/s 2024-08-17T17:46:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:34 smithi165 ceph-mon[105517]: pgmap v1658: 105 pgs: 105 active+clean; 248 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 24 op/s 2024-08-17T17:46:36.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:46:36.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:46:36.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:46:37.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:37 smithi165 ceph-mon[105517]: pgmap v1659: 105 pgs: 105 active+clean; 248 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.5 MiB/s wr, 27 op/s 2024-08-17T17:46:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:37 smithi136 ceph-mon[109861]: pgmap v1659: 105 pgs: 105 active+clean; 248 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.5 MiB/s wr, 27 op/s 2024-08-17T17:46:37.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:46:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:46:37] "GET /metrics HTTP/1.1" 200 36489 "" "Prometheus/2.51.0" 2024-08-17T17:46:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:37 smithi136 ceph-mon[115078]: pgmap v1659: 105 pgs: 105 active+clean; 248 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.5 MiB/s wr, 27 op/s 2024-08-17T17:46:39.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:39 smithi165 ceph-mon[105517]: osdmap e1087: 8 total, 8 up, 8 in 2024-08-17T17:46:39.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:39 smithi165 ceph-mon[105517]: pgmap v1661: 105 pgs: 105 active+clean; 248 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 1.6 MiB/s wr, 35 op/s 2024-08-17T17:46:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:39 smithi136 ceph-mon[109861]: osdmap e1087: 8 total, 8 up, 8 in 2024-08-17T17:46:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:39 smithi136 ceph-mon[109861]: pgmap v1661: 105 pgs: 105 active+clean; 248 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 1.6 MiB/s wr, 35 op/s 2024-08-17T17:46:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:39 smithi136 ceph-mon[115078]: osdmap e1087: 8 total, 8 up, 8 in 2024-08-17T17:46:39.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:39 smithi136 ceph-mon[115078]: pgmap v1661: 105 pgs: 105 active+clean; 248 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 1.6 MiB/s wr, 35 op/s 2024-08-17T17:46:40.294 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:46:40.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:40 smithi136 ceph-mon[109861]: osdmap e1088: 8 total, 8 up, 8 in 2024-08-17T17:46:40.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:40 smithi136 ceph-mon[115078]: osdmap e1088: 8 total, 8 up, 8 in 2024-08-17T17:46:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:40 smithi165 ceph-mon[105517]: osdmap e1088: 8 total, 8 up, 8 in 2024-08-17T17:46:41.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:41 smithi136 ceph-mon[109861]: osdmap e1089: 8 total, 8 up, 8 in 2024-08-17T17:46:41.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:41 smithi136 ceph-mon[109861]: pgmap v1664: 105 pgs: 105 active+clean; 248 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 2.0 KiB/s wr, 20 op/s 2024-08-17T17:46:41.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:41 smithi136 ceph-mon[115078]: osdmap e1089: 8 total, 8 up, 8 in 2024-08-17T17:46:41.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:41 smithi136 ceph-mon[115078]: pgmap v1664: 105 pgs: 105 active+clean; 248 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 2.0 KiB/s wr, 20 op/s 2024-08-17T17:46:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:41 smithi165 ceph-mon[105517]: osdmap e1089: 8 total, 8 up, 8 in 2024-08-17T17:46:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:41 smithi165 ceph-mon[105517]: pgmap v1664: 105 pgs: 105 active+clean; 248 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 2.0 KiB/s wr, 20 op/s 2024-08-17T17:46:42.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:42 smithi136 ceph-mon[109861]: pgmap v1665: 105 pgs: 105 active+clean; 270 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 38 KiB/s rd, 2.7 MiB/s wr, 60 op/s 2024-08-17T17:46:42.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:42 smithi136 ceph-mon[115078]: pgmap v1665: 105 pgs: 105 active+clean; 270 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 38 KiB/s rd, 2.7 MiB/s wr, 60 op/s 2024-08-17T17:46:43.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:42 smithi165 ceph-mon[105517]: pgmap v1665: 105 pgs: 105 active+clean; 270 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 38 KiB/s rd, 2.7 MiB/s wr, 60 op/s 2024-08-17T17:46:44.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:44 smithi136 ceph-mon[109861]: pgmap v1666: 105 pgs: 105 active+clean; 270 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 2.6 MiB/s wr, 43 op/s 2024-08-17T17:46:44.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:44 smithi136 ceph-mon[115078]: pgmap v1666: 105 pgs: 105 active+clean; 270 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 2.6 MiB/s wr, 43 op/s 2024-08-17T17:46:45.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:44 smithi165 ceph-mon[105517]: pgmap v1666: 105 pgs: 105 active+clean; 270 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 2.6 MiB/s wr, 43 op/s 2024-08-17T17:46:46.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:46 smithi136 ceph-mon[109861]: osdmap e1090: 8 total, 8 up, 8 in 2024-08-17T17:46:46.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:46 smithi136 ceph-mon[109861]: pgmap v1668: 105 pgs: 105 active+clean; 281 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 3.3 MiB/s wr, 42 op/s 2024-08-17T17:46:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:46 smithi136 ceph-mon[115078]: osdmap e1090: 8 total, 8 up, 8 in 2024-08-17T17:46:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:46 smithi136 ceph-mon[115078]: pgmap v1668: 105 pgs: 105 active+clean; 281 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 3.3 MiB/s wr, 42 op/s 2024-08-17T17:46:47.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:46 smithi165 ceph-mon[105517]: osdmap e1090: 8 total, 8 up, 8 in 2024-08-17T17:46:47.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:46 smithi165 ceph-mon[105517]: pgmap v1668: 105 pgs: 105 active+clean; 281 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 3.3 MiB/s wr, 42 op/s 2024-08-17T17:46:47.707 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:46:47.847 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:47 smithi136 ceph-mon[109861]: osdmap e1091: 8 total, 8 up, 8 in 2024-08-17T17:46:47.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:47 smithi136 ceph-mon[115078]: osdmap e1091: 8 total, 8 up, 8 in 2024-08-17T17:46:47.848 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:46:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:46:47] "GET /metrics HTTP/1.1" 200 36489 "" "Prometheus/2.51.0" 2024-08-17T17:46:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:47 smithi165 ceph-mon[105517]: osdmap e1091: 8 total, 8 up, 8 in 2024-08-17T17:46:48.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:48 smithi136 ceph-mon[109861]: osdmap e1092: 8 total, 8 up, 8 in 2024-08-17T17:46:48.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:48 smithi136 ceph-mon[109861]: pgmap v1671: 105 pgs: 105 active+clean; 288 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 2.0 MiB/s wr, 17 op/s 2024-08-17T17:46:48.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:48 smithi136 ceph-mon[115078]: osdmap e1092: 8 total, 8 up, 8 in 2024-08-17T17:46:48.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:48 smithi136 ceph-mon[115078]: pgmap v1671: 105 pgs: 105 active+clean; 288 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 2.0 MiB/s wr, 17 op/s 2024-08-17T17:46:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:48 smithi165 ceph-mon[105517]: osdmap e1092: 8 total, 8 up, 8 in 2024-08-17T17:46:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:48 smithi165 ceph-mon[105517]: pgmap v1671: 105 pgs: 105 active+clean; 288 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 2.0 MiB/s wr, 17 op/s 2024-08-17T17:46:50.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:50 smithi136 ceph-mon[109861]: pgmap v1672: 105 pgs: 105 active+clean; 288 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 2.0 MiB/s wr, 17 op/s 2024-08-17T17:46:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:50 smithi136 ceph-mon[115078]: pgmap v1672: 105 pgs: 105 active+clean; 288 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 2.0 MiB/s wr, 17 op/s 2024-08-17T17:46:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:50 smithi165 ceph-mon[105517]: pgmap v1672: 105 pgs: 105 active+clean; 288 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 2.0 MiB/s wr, 17 op/s 2024-08-17T17:46:51.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:46:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:46:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:46:52.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:52 smithi136 ceph-mon[109861]: pgmap v1673: 105 pgs: 105 active+clean; 292 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 1.8 MiB/s wr, 39 op/s 2024-08-17T17:46:52.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:52 smithi136 ceph-mon[115078]: pgmap v1673: 105 pgs: 105 active+clean; 292 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 1.8 MiB/s wr, 39 op/s 2024-08-17T17:46:53.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:52 smithi165 ceph-mon[105517]: pgmap v1673: 105 pgs: 105 active+clean; 292 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 1.8 MiB/s wr, 39 op/s 2024-08-17T17:46:54.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:54 smithi136 ceph-mon[109861]: osdmap e1093: 8 total, 8 up, 8 in 2024-08-17T17:46:54.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:54 smithi136 ceph-mon[109861]: pgmap v1675: 105 pgs: 105 active+clean; 292 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 1.0 MiB/s wr, 29 op/s 2024-08-17T17:46:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:54 smithi136 ceph-mon[115078]: osdmap e1093: 8 total, 8 up, 8 in 2024-08-17T17:46:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:54 smithi136 ceph-mon[115078]: pgmap v1675: 105 pgs: 105 active+clean; 292 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 1.0 MiB/s wr, 29 op/s 2024-08-17T17:46:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:54 smithi165 ceph-mon[105517]: osdmap e1093: 8 total, 8 up, 8 in 2024-08-17T17:46:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:54 smithi165 ceph-mon[105517]: pgmap v1675: 105 pgs: 105 active+clean; 292 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 1.0 MiB/s wr, 29 op/s 2024-08-17T17:46:55.616 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:46:55.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:55 smithi136 ceph-mon[109861]: osdmap e1094: 8 total, 8 up, 8 in 2024-08-17T17:46:55.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:55 smithi136 ceph-mon[115078]: osdmap e1094: 8 total, 8 up, 8 in 2024-08-17T17:46:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:55 smithi165 ceph-mon[105517]: osdmap e1094: 8 total, 8 up, 8 in 2024-08-17T17:46:56.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:56 smithi136 ceph-mon[109861]: osdmap e1095: 8 total, 8 up, 8 in 2024-08-17T17:46:56.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:56 smithi136 ceph-mon[109861]: pgmap v1678: 105 pgs: 105 active+clean; 311 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 44 KiB/s rd, 3.3 MiB/s wr, 67 op/s 2024-08-17T17:46:56.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:56 smithi136 ceph-mon[115078]: osdmap e1095: 8 total, 8 up, 8 in 2024-08-17T17:46:56.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:56 smithi136 ceph-mon[115078]: pgmap v1678: 105 pgs: 105 active+clean; 311 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 44 KiB/s rd, 3.3 MiB/s wr, 67 op/s 2024-08-17T17:46:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:56 smithi165 ceph-mon[105517]: osdmap e1095: 8 total, 8 up, 8 in 2024-08-17T17:46:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:56 smithi165 ceph-mon[105517]: pgmap v1678: 105 pgs: 105 active+clean; 311 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 44 KiB/s rd, 3.3 MiB/s wr, 67 op/s 2024-08-17T17:46:57.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:46:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:46:57] "GET /metrics HTTP/1.1" 200 36486 "" "Prometheus/2.51.0" 2024-08-17T17:46:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:46:58 smithi136 ceph-mon[109861]: pgmap v1679: 105 pgs: 105 active+clean; 314 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 MiB/s wr, 44 op/s 2024-08-17T17:46:58.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:46:58 smithi136 ceph-mon[115078]: pgmap v1679: 105 pgs: 105 active+clean; 314 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 MiB/s wr, 44 op/s 2024-08-17T17:46:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:46:58 smithi165 ceph-mon[105517]: pgmap v1679: 105 pgs: 105 active+clean; 314 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 MiB/s wr, 44 op/s 2024-08-17T17:47:00.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:00 smithi136 ceph-mon[109861]: pgmap v1680: 105 pgs: 105 active+clean; 314 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.3 MiB/s wr, 38 op/s 2024-08-17T17:47:00.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:00 smithi136 ceph-mon[115078]: pgmap v1680: 105 pgs: 105 active+clean; 314 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.3 MiB/s wr, 38 op/s 2024-08-17T17:47:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:00 smithi165 ceph-mon[105517]: pgmap v1680: 105 pgs: 105 active+clean; 314 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.3 MiB/s wr, 38 op/s 2024-08-17T17:47:02.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:02 smithi136 ceph-mon[109861]: osdmap e1096: 8 total, 8 up, 8 in 2024-08-17T17:47:02.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:02 smithi136 ceph-mon[109861]: pgmap v1682: 105 pgs: 105 active+clean; 334 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 4.1 MiB/s wr, 39 op/s 2024-08-17T17:47:02.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:02 smithi136 ceph-mon[115078]: osdmap e1096: 8 total, 8 up, 8 in 2024-08-17T17:47:02.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:02 smithi136 ceph-mon[115078]: pgmap v1682: 105 pgs: 105 active+clean; 334 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 4.1 MiB/s wr, 39 op/s 2024-08-17T17:47:03.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:02 smithi165 ceph-mon[105517]: osdmap e1096: 8 total, 8 up, 8 in 2024-08-17T17:47:03.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:02 smithi165 ceph-mon[105517]: pgmap v1682: 105 pgs: 105 active+clean; 334 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 4.1 MiB/s wr, 39 op/s 2024-08-17T17:47:03.602 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/0.DiffIterateDeterministicLUKS2PP (38992 ms) 2024-08-17T17:47:03.602 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/0.DiffIterateDiscard 2024-08-17T17:47:03.602 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:47:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:03 smithi136 ceph-mon[109861]: osdmap e1097: 8 total, 8 up, 8 in 2024-08-17T17:47:03.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:47:03.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:47:03.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:47:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:03 smithi136 ceph-mon[115078]: osdmap e1097: 8 total, 8 up, 8 in 2024-08-17T17:47:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:47:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:47:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:47:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:03 smithi165 ceph-mon[105517]: osdmap e1097: 8 total, 8 up, 8 in 2024-08-17T17:47:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:47:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:47:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:47:04.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:04 smithi165 ceph-mon[105517]: osdmap e1098: 8 total, 8 up, 8 in 2024-08-17T17:47:04.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:04 smithi165 ceph-mon[105517]: pgmap v1685: 105 pgs: 105 active+clean; 334 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.7 MiB/s wr, 7 op/s 2024-08-17T17:47:04.933 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:04 smithi136 ceph-mon[109861]: osdmap e1098: 8 total, 8 up, 8 in 2024-08-17T17:47:04.933 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:04 smithi136 ceph-mon[109861]: pgmap v1685: 105 pgs: 105 active+clean; 334 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.7 MiB/s wr, 7 op/s 2024-08-17T17:47:04.934 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:04 smithi136 ceph-mon[115078]: osdmap e1098: 8 total, 8 up, 8 in 2024-08-17T17:47:04.934 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:04 smithi136 ceph-mon[115078]: pgmap v1685: 105 pgs: 105 active+clean; 334 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.7 MiB/s wr, 7 op/s 2024-08-17T17:47:05.883 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:05 smithi165 ceph-mon[105517]: osdmap e1099: 8 total, 8 up, 8 in 2024-08-17T17:47:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:05 smithi136 ceph-mon[109861]: osdmap e1099: 8 total, 8 up, 8 in 2024-08-17T17:47:05.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:05 smithi136 ceph-mon[115078]: osdmap e1099: 8 total, 8 up, 8 in 2024-08-17T17:47:06.624 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/0.DiffIterateDiscard (3022 ms) 2024-08-17T17:47:06.625 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/0.DiffIterateTruncate 2024-08-17T17:47:06.625 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:47:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:06 smithi136 ceph-mon[109861]: osdmap e1100: 8 total, 8 up, 8 in 2024-08-17T17:47:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:47:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:06 smithi136 ceph-mon[109861]: pgmap v1688: 105 pgs: 105 active+clean; 334 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 42 KiB/s rd, 5.0 KiB/s wr, 56 op/s 2024-08-17T17:47:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:06 smithi136 ceph-mon[115078]: osdmap e1100: 8 total, 8 up, 8 in 2024-08-17T17:47:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:47:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:06 smithi136 ceph-mon[115078]: pgmap v1688: 105 pgs: 105 active+clean; 334 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 42 KiB/s rd, 5.0 KiB/s wr, 56 op/s 2024-08-17T17:47:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:06 smithi165 ceph-mon[105517]: osdmap e1100: 8 total, 8 up, 8 in 2024-08-17T17:47:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:47:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:06 smithi165 ceph-mon[105517]: pgmap v1688: 105 pgs: 105 active+clean; 334 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 42 KiB/s rd, 5.0 KiB/s wr, 56 op/s 2024-08-17T17:47:07.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:07 smithi136 ceph-mon[115078]: osdmap e1101: 8 total, 8 up, 8 in 2024-08-17T17:47:07.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:47:07.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:47:07.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:07 smithi136 ceph-mon[109861]: osdmap e1101: 8 total, 8 up, 8 in 2024-08-17T17:47:07.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:47:07.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:47:07.948 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:47:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:47:07] "GET /metrics HTTP/1.1" 200 36490 "" "Prometheus/2.51.0" 2024-08-17T17:47:08.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:07 smithi165 ceph-mon[105517]: osdmap e1101: 8 total, 8 up, 8 in 2024-08-17T17:47:08.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:47:08.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:47:08.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:08 smithi136 ceph-mon[115078]: osdmap e1102: 8 total, 8 up, 8 in 2024-08-17T17:47:08.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:47:08.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:47:08.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:47:08.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:47:08.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:47:08.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:08 smithi136 ceph-mon[115078]: pgmap v1691: 105 pgs: 105 active+clean; 338 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 64 KiB/s rd, 13 KiB/s wr, 104 op/s 2024-08-17T17:47:08.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:08 smithi136 ceph-mon[109861]: osdmap e1102: 8 total, 8 up, 8 in 2024-08-17T17:47:08.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:47:08.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:47:08.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:47:08.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:47:08.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:47:08.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:08 smithi136 ceph-mon[109861]: pgmap v1691: 105 pgs: 105 active+clean; 338 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 64 KiB/s rd, 13 KiB/s wr, 104 op/s 2024-08-17T17:47:09.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:08 smithi165 ceph-mon[105517]: osdmap e1102: 8 total, 8 up, 8 in 2024-08-17T17:47:09.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:47:09.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:47:09.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:47:09.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:47:09.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:47:09.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:08 smithi165 ceph-mon[105517]: pgmap v1691: 105 pgs: 105 active+clean; 338 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 64 KiB/s rd, 13 KiB/s wr, 104 op/s 2024-08-17T17:47:09.654 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/0.DiffIterateTruncate (3028 ms) 2024-08-17T17:47:09.654 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/0.DiffIterateWriteAndTruncate 2024-08-17T17:47:09.654 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:47:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:09 smithi136 ceph-mon[109861]: osdmap e1103: 8 total, 8 up, 8 in 2024-08-17T17:47:09.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:09 smithi136 ceph-mon[115078]: osdmap e1103: 8 total, 8 up, 8 in 2024-08-17T17:47:10.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:09 smithi165 ceph-mon[105517]: osdmap e1103: 8 total, 8 up, 8 in 2024-08-17T17:47:10.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:10 smithi136 ceph-mon[115078]: osdmap e1104: 8 total, 8 up, 8 in 2024-08-17T17:47:10.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:10 smithi136 ceph-mon[115078]: pgmap v1694: 105 pgs: 105 active+clean; 338 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 8.5 KiB/s wr, 47 op/s 2024-08-17T17:47:10.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:10 smithi136 ceph-mon[109861]: osdmap e1104: 8 total, 8 up, 8 in 2024-08-17T17:47:10.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:10 smithi136 ceph-mon[109861]: pgmap v1694: 105 pgs: 105 active+clean; 338 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 8.5 KiB/s wr, 47 op/s 2024-08-17T17:47:11.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:10 smithi165 ceph-mon[105517]: osdmap e1104: 8 total, 8 up, 8 in 2024-08-17T17:47:11.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:10 smithi165 ceph-mon[105517]: pgmap v1694: 105 pgs: 105 active+clean; 338 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 8.5 KiB/s wr, 47 op/s 2024-08-17T17:47:11.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:47:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=infra.usagestats t=2024-08-17T17:47:11.052404512Z level=info msg="Usage stats are ready to report" 2024-08-17T17:47:11.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:11 smithi136 ceph-mon[109861]: osdmap e1105: 8 total, 8 up, 8 in 2024-08-17T17:47:11.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:11 smithi136 ceph-mon[115078]: osdmap e1105: 8 total, 8 up, 8 in 2024-08-17T17:47:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:11 smithi165 ceph-mon[105517]: osdmap e1105: 8 total, 8 up, 8 in 2024-08-17T17:47:12.682 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/0.DiffIterateWriteAndTruncate (3029 ms) 2024-08-17T17:47:12.682 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/0.DiffIterateStress 2024-08-17T17:47:12.682 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:47:12.754 INFO:tasks.workunit.client.0.smithi136.stdout: i=0 exists [58317638~3227581,83023353~3639184,122432837~3020879,204458483~1439192,238250114~5030627,272197461~7881822,326072099~3531549,395510660~8058270,407715299~6516896] wrote [58317638~3227581,83023353~3639184,122432837~3020879,204458483~1439192,238250114~5030627,272197461~7881822,326072099~3531549,395510660~8058270,407715299~6516896] 2024-08-17T17:47:12.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:12 smithi136 ceph-mon[109861]: osdmap e1106: 8 total, 8 up, 8 in 2024-08-17T17:47:12.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:12 smithi136 ceph-mon[109861]: pgmap v1697: 105 pgs: 105 active+clean; 339 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 60 KiB/s rd, 393 KiB/s wr, 94 op/s 2024-08-17T17:47:12.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:12 smithi136 ceph-mon[115078]: osdmap e1106: 8 total, 8 up, 8 in 2024-08-17T17:47:12.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:12 smithi136 ceph-mon[115078]: pgmap v1697: 105 pgs: 105 active+clean; 339 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 60 KiB/s rd, 393 KiB/s wr, 94 op/s 2024-08-17T17:47:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:12 smithi165 ceph-mon[105517]: osdmap e1106: 8 total, 8 up, 8 in 2024-08-17T17:47:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:12 smithi165 ceph-mon[105517]: pgmap v1697: 105 pgs: 105 active+clean; 339 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 60 KiB/s rd, 393 KiB/s wr, 94 op/s 2024-08-17T17:47:13.700 INFO:tasks.workunit.client.0.smithi136.stdout: i=1 exists [11579176~9299441,58317638~3227581,66519528~3755847,83023353~3639184,99070072~2132985,122432837~3020879,128259422~7709409,181050411~8001537,204458483~1439192,214151721~1221802,238250114~5030627,244079590~6931492,272197461~7881822,326072099~3531549,338264527~4074143,395510660~9255171,407715299~6516896] wrote [11579176~9299441,66519528~3755847,99070072~2132985,128259422~7709409,181050411~8001537,214151721~1221802,244079590~6931492,338264527~4074143,399426072~5339759] 2024-08-17T17:47:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:13 smithi136 ceph-mon[109861]: osdmap e1107: 8 total, 8 up, 8 in 2024-08-17T17:47:13.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:13 smithi136 ceph-mon[115078]: osdmap e1107: 8 total, 8 up, 8 in 2024-08-17T17:47:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:13 smithi165 ceph-mon[105517]: osdmap e1107: 8 total, 8 up, 8 in 2024-08-17T17:47:14.721 INFO:tasks.workunit.client.0.smithi136.stdout: i=2 exists [8385954~756889,11579176~9299441,56172858~5372361,66519528~6828280,83023353~3639184,87976177~4444660,99070072~2132985,122432837~3020879,128259422~7709409,146521237~911867,178804667~10247281,204458483~1439192,214151721~1221802,234720472~2583266,238250114~5030627,244079590~6931492,272197461~7881822,300553585~6793976,326072099~3531549,338264527~4074143,395510660~9255171,406190492~620407,407715299~6516896] wrote [8385954~756889,56172858~4472149,67669518~5678290,87976177~4444660,146521237~911867,178804667~5897510,234720472~2583266,274099473~5145862,300553585~6793976,406190492~620407] 2024-08-17T17:47:14.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:14 smithi136 ceph-mon[115078]: osdmap e1108: 8 total, 8 up, 8 in 2024-08-17T17:47:14.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:14 smithi136 ceph-mon[115078]: pgmap v1700: 105 pgs: 105 active+clean; 339 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 60 KiB/s rd, 393 KiB/s wr, 94 op/s 2024-08-17T17:47:14.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:14 smithi136 ceph-mon[109861]: osdmap e1108: 8 total, 8 up, 8 in 2024-08-17T17:47:14.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:14 smithi136 ceph-mon[109861]: pgmap v1700: 105 pgs: 105 active+clean; 339 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 60 KiB/s rd, 393 KiB/s wr, 94 op/s 2024-08-17T17:47:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:14 smithi165 ceph-mon[105517]: osdmap e1108: 8 total, 8 up, 8 in 2024-08-17T17:47:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:14 smithi165 ceph-mon[105517]: pgmap v1700: 105 pgs: 105 active+clean; 339 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 60 KiB/s rd, 393 KiB/s wr, 94 op/s 2024-08-17T17:47:15.712 INFO:tasks.workunit.client.0.smithi136.stdout: i=3 exists [8385954~756889,11579176~9299441,56172858~5372361,66519528~6828280,83023353~3639184,87976177~4444660,99070072~2132985,122432837~3020879,128259422~12458763,141162200~123733,146521237~911867,170236347~879778,178804667~10247281,204458483~1439192,214151721~1221802,234720472~2583266,238250114~5030627,244079590~6931492,266927138~2271880,272197461~7881822,300553585~6793976,316072980~3026910,326072099~3531549,338264527~4074143,370259166~7251446,392747039~1807197,395510660~9255171,406190492~620407,407715299~6516896] wrote [134789056~5929129,141162200~123733,170236347~879778,179506671~1438996,266927138~2271880,316072980~3026910,370259166~7251446,392747039~1807197,398184564~2539553] 2024-08-17T17:47:15.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:15 smithi136 ceph-mon[109861]: osdmap e1109: 8 total, 8 up, 8 in 2024-08-17T17:47:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:15 smithi136 ceph-mon[115078]: osdmap e1109: 8 total, 8 up, 8 in 2024-08-17T17:47:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:15 smithi165 ceph-mon[105517]: osdmap e1109: 8 total, 8 up, 8 in 2024-08-17T17:47:16.757 INFO:tasks.workunit.client.0.smithi136.stdout: i=4 exists [8385954~756889,11579176~9299441,27213408~7936521,52524078~2996302,56172858~5372361,66519528~6828280,83023353~3639184,87976177~4444660,99070072~2132985,122432837~3020879,128259422~12458763,141162200~123733,146521237~911867,170236347~879778,178804667~10247281,204458483~1439192,214151721~1221802,216026965~5893923,234720472~2583266,238250114~5030627,244079590~6931492,266927138~2271880,272197461~7881822,300553585~12233905,316072980~3026910,326072099~3531549,338264527~4074143,370259166~7251446,385127346~19638485,405783969~8448226] wrote [27213408~7936521,52524078~2996302,132770613~4547368,184623919~975441,216026965~5893923,305250582~7536908,385127346~8726798,393974446~4117399,405783969~3077668] 2024-08-17T17:47:16.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:16 smithi136 ceph-mon[109861]: osdmap e1110: 8 total, 8 up, 8 in 2024-08-17T17:47:16.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:16 smithi136 ceph-mon[109861]: pgmap v1703: 105 pgs: 105 active+clean; 421 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 143 KiB/s rd, 12 MiB/s wr, 230 op/s 2024-08-17T17:47:16.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:16 smithi136 ceph-mon[115078]: osdmap e1110: 8 total, 8 up, 8 in 2024-08-17T17:47:16.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:16 smithi136 ceph-mon[115078]: pgmap v1703: 105 pgs: 105 active+clean; 421 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 143 KiB/s rd, 12 MiB/s wr, 230 op/s 2024-08-17T17:47:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:16 smithi165 ceph-mon[105517]: osdmap e1110: 8 total, 8 up, 8 in 2024-08-17T17:47:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:16 smithi165 ceph-mon[105517]: pgmap v1703: 105 pgs: 105 active+clean; 421 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 143 KiB/s rd, 12 MiB/s wr, 230 op/s 2024-08-17T17:47:17.751 INFO:tasks.workunit.client.0.smithi136.stdout: i=5 exists [7294659~3604783,11579176~9299441,27213408~7936521,46387139~9133241,56172858~5372361,66519528~6828280,83023353~3639184,87976177~4444660,99070072~2132985,112459935~7349748,122432837~3020879,128259422~12458763,141162200~123733,146521237~911867,170236347~879778,178804667~10247281,204458483~1439192,214151721~1221802,216026965~5893923,234720472~2583266,238250114~5030627,244079590~6931492,266927138~13152145,300553585~12233905,316072980~3026910,326072099~3531549,332951667~1703830,338264527~4074143,344366004~4958418,366512993~16138442,385127346~19638485,405783969~8448226] wrote [7294659~3604783,14418625~2503052,46387139~6646744,88210204~2654548,112459935~7349748,267519243~7295128,332951667~1703830,344366004~4958418,366512993~8145527,376903755~5747680] 2024-08-17T17:47:17.848 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:17 smithi136 ceph-mon[109861]: osdmap e1111: 8 total, 8 up, 8 in 2024-08-17T17:47:17.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:17 smithi136 ceph-mon[115078]: osdmap e1111: 8 total, 8 up, 8 in 2024-08-17T17:47:17.848 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:47:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:47:17] "GET /metrics HTTP/1.1" 200 36490 "" "Prometheus/2.51.0" 2024-08-17T17:47:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:17 smithi165 ceph-mon[105517]: osdmap e1111: 8 total, 8 up, 8 in 2024-08-17T17:47:18.738 INFO:tasks.workunit.client.0.smithi136.stdout: i=6 exists [7294659~3604783,11579176~9299441,22298619~3398016,27213408~7936521,46387139~9133241,56172858~5372361,66519528~6828280,83023353~3639184,87976177~4444660,99070072~2132985,112459935~7349748,122432837~3020879,128259422~12458763,141162200~123733,143631694~2819484,146521237~911867,170236347~879778,177148887~11903061,203371987~4800717,210384164~3396269,214151721~1221802,216026965~5893923,229813410~1621406,234720472~2583266,238250114~5030627,244079590~6931492,265447020~14632263,300553585~12233905,316072980~3026910,326072099~3531549,332951667~1703830,338264527~4074143,344366004~4958418,366512993~16138442,385127346~19638485,405783969~8448226] wrote [18397559~2377817,22298619~3398016,71871766~729179,143631694~2819484,177148887~3686146,203371987~4800717,210384164~3396269,229813410~1621406,265447020~5750407,375036141~1226037] 2024-08-17T17:47:18.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:18 smithi136 ceph-mon[115078]: osdmap e1112: 8 total, 8 up, 8 in 2024-08-17T17:47:18.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:18 smithi136 ceph-mon[115078]: pgmap v1706: 105 pgs: 105 active+clean; 576 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 146 KiB/s rd, 43 MiB/s wr, 250 op/s 2024-08-17T17:47:18.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:18 smithi136 ceph-mon[109861]: osdmap e1112: 8 total, 8 up, 8 in 2024-08-17T17:47:18.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:18 smithi136 ceph-mon[109861]: pgmap v1706: 105 pgs: 105 active+clean; 576 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 146 KiB/s rd, 43 MiB/s wr, 250 op/s 2024-08-17T17:47:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:18 smithi165 ceph-mon[105517]: osdmap e1112: 8 total, 8 up, 8 in 2024-08-17T17:47:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:18 smithi165 ceph-mon[105517]: pgmap v1706: 105 pgs: 105 active+clean; 576 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 146 KiB/s rd, 43 MiB/s wr, 250 op/s 2024-08-17T17:47:19.736 INFO:tasks.workunit.client.0.smithi136.stdout: i=7 exists [7294659~3604783,11579176~9299441,22298619~3398016,27213408~7936521,46387139~9133241,56172858~5372361,66519528~7765941,83023353~3639184,87976177~4444660,98323116~4579656,112459935~7349748,122432837~3020879,128259422~12458763,141162200~123733,143631694~2819484,146521237~911867,170236347~879778,173661827~3057406,177148887~11903061,203371987~5918596,210384164~3396269,214151721~1221802,216026965~5893923,229813410~1621406,231872994~29434,234720472~2583266,238250114~5030627,244079590~6931492,259164751~280375,265447020~14632263,295831353~16956137,316072980~4616109,326072099~3531549,332951667~1703830,338264527~4074143,344366004~4958418,366512993~16138442,385127346~19638485,405783969~8448226] wrote [72693380~1592089,98323116~4579656,130078852~4344656,173661827~3057406,205047266~4243317,231872994~29434,259164751~280375,295831353~7859301,317833139~2855950,401465291~1068983] 2024-08-17T17:47:19.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:19 smithi136 ceph-mon[109861]: osdmap e1113: 8 total, 8 up, 8 in 2024-08-17T17:47:19.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:19 smithi136 ceph-mon[115078]: osdmap e1113: 8 total, 8 up, 8 in 2024-08-17T17:47:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:19 smithi165 ceph-mon[105517]: osdmap e1113: 8 total, 8 up, 8 in 2024-08-17T17:47:20.776 INFO:tasks.workunit.client.0.smithi136.stdout: i=8 exists [7294659~3604783,11579176~9299441,22298619~3398016,27213408~7936521,46126522~9393858,56172858~5372361,66519528~7765941,83023353~3639184,87976177~4444660,96231003~7638279,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~911867,151863990~3083981,170236347~879778,173661827~3057406,177148887~11903061,203371987~5918596,210384164~3396269,214151721~1221802,216026965~5893923,229813410~2722880,234720472~2583266,238250114~5030627,244079590~6931492,259164751~280375,265447020~14632263,295831353~16956137,316072980~4616109,326072099~3531549,332951667~1703830,338264527~4074143,344366004~4958418,366512993~16138442,385127346~19638485,405783969~8448226] wrote [46126522~3904741,96231003~7638279,121612481~8169769,151863990~3083981,182474064~5726314,231388101~1148189,316577159~3150955,391986145~4334286] 2024-08-17T17:47:21.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:20 smithi165 ceph-mon[105517]: osdmap e1114: 8 total, 8 up, 8 in 2024-08-17T17:47:21.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:20 smithi165 ceph-mon[105517]: pgmap v1709: 105 pgs: 105 active+clean; 576 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 31 MiB/s wr, 20 op/s 2024-08-17T17:47:21.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:20 smithi136 ceph-mon[109861]: osdmap e1114: 8 total, 8 up, 8 in 2024-08-17T17:47:21.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:20 smithi136 ceph-mon[109861]: pgmap v1709: 105 pgs: 105 active+clean; 576 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 31 MiB/s wr, 20 op/s 2024-08-17T17:47:21.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:20 smithi136 ceph-mon[115078]: osdmap e1114: 8 total, 8 up, 8 in 2024-08-17T17:47:21.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:20 smithi136 ceph-mon[115078]: pgmap v1709: 105 pgs: 105 active+clean; 576 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 31 MiB/s wr, 20 op/s 2024-08-17T17:47:21.776 INFO:tasks.workunit.client.0.smithi136.stdout: i=9 exists [7294659~3604783,11579176~9894787,22298619~3398016,27213408~7936521,46126522~9393858,56172858~5372361,66519528~7765941,83023353~3639184,87976177~4444660,96231003~7638279,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~911867,151863990~3892627,170236347~879778,173661827~3057406,177148887~11903061,199192089~1285316,203371987~5918596,210384164~3396269,214151721~7769167,229813410~2722880,234720472~8560269,244079590~6931492,259164751~280375,265447020~14632263,295831353~16956137,316072980~4616109,326072099~3531549,332951667~1703830,338264527~11059895,366512993~16138442,385127346~19638485,404966874~9265321] wrote [14137310~7336653,46931142~7840484,153465260~2291357,199192089~1285316,214907306~4822959,235471051~6158445,272649616~7204210,342231213~2195203,404966874~7561094] 2024-08-17T17:47:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:21 smithi165 ceph-mon[105517]: osdmap e1115: 8 total, 8 up, 8 in 2024-08-17T17:47:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:47:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:21 smithi136 ceph-mon[109861]: osdmap e1115: 8 total, 8 up, 8 in 2024-08-17T17:47:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:47:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:21 smithi136 ceph-mon[115078]: osdmap e1115: 8 total, 8 up, 8 in 2024-08-17T17:47:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:47:22.766 INFO:tasks.workunit.client.0.smithi136.stdout: i=10 exists [7294659~3604783,11579176~9894787,22298619~3398016,27213408~7936521,46126522~9393858,56172858~5372361,66519528~7765941,83023353~3639184,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~911867,151863990~7332122,170236347~879778,173661827~3057406,177148887~11903061,199192089~1285316,200751176~8539407,210384164~3396269,214151721~7769167,229813410~2722880,234720472~8560269,244079590~6931492,257231876~3214859,265447020~14632263,284082763~2045905,290745721~4774032,295831353~16956137,316072980~4616109,326072099~3531549,332951667~1703830,338264527~11059895,366512993~16138442,385127346~19638485,404966874~9265321] wrote [104856556~4244827,135726836~4282324,153474666~5721446,187306936~599389,200751176~7563019,257231876~3214859,284082763~2045905,290745721~4774032,299299990~6835465] 2024-08-17T17:47:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:22 smithi165 ceph-mon[105517]: osdmap e1116: 8 total, 8 up, 8 in 2024-08-17T17:47:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:22 smithi165 ceph-mon[105517]: pgmap v1712: 105 pgs: 105 active+clean; 791 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 171 KiB/s rd, 46 MiB/s wr, 281 op/s 2024-08-17T17:47:23.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:22 smithi136 ceph-mon[109861]: osdmap e1116: 8 total, 8 up, 8 in 2024-08-17T17:47:23.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:22 smithi136 ceph-mon[109861]: pgmap v1712: 105 pgs: 105 active+clean; 791 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 171 KiB/s rd, 46 MiB/s wr, 281 op/s 2024-08-17T17:47:23.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:22 smithi136 ceph-mon[115078]: osdmap e1116: 8 total, 8 up, 8 in 2024-08-17T17:47:23.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:22 smithi136 ceph-mon[115078]: pgmap v1712: 105 pgs: 105 active+clean; 791 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 171 KiB/s rd, 46 MiB/s wr, 281 op/s 2024-08-17T17:47:23.785 INFO:tasks.workunit.client.0.smithi136.stdout: i=11 exists [7294659~3604783,11579176~9894787,22298619~3398016,27213408~7936521,43810632~11709748,56172858~6654831,64609489~9675980,76085135~2537370,83023353~3639184,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~911867,151863990~7332122,170236347~879778,173661827~3057406,177148887~11903061,199192089~1285316,200751176~8539407,210384164~3396269,214151721~11450493,229813410~2722880,234720472~8560269,244079590~6931492,257231876~3214859,265447020~14632263,284082763~2045905,290745721~4774032,295831353~16956137,316072980~4616109,326072099~3531549,332951667~1703830,338264527~11059895,366512993~38252838,404966874~9265321] wrote [43810632~6570505,56731919~6095770,64609489~4058806,76085135~2537370,98515291~4618384,126663946~3417085,217564579~8037635,345979605~2021847,378717919~7112881,389343167~1935343] 2024-08-17T17:47:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:23 smithi165 ceph-mon[105517]: osdmap e1117: 8 total, 8 up, 8 in 2024-08-17T17:47:24.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:23 smithi136 ceph-mon[109861]: osdmap e1117: 8 total, 8 up, 8 in 2024-08-17T17:47:24.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:23 smithi136 ceph-mon[115078]: osdmap e1117: 8 total, 8 up, 8 in 2024-08-17T17:47:24.808 INFO:tasks.workunit.client.0.smithi136.stdout: i=12 exists [3528496~7370946,11579176~9894787,22298619~3398016,27213408~7936521,43810632~11709748,56172858~6654831,64609489~14013016,83023353~3639184,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~12674875,170236347~879778,173661827~3057406,177148887~16010685,199192089~1285316,200751176~8539407,210384164~3396269,214151721~11450493,229813410~2722880,234720472~8560269,244079590~6931492,257231876~3214859,265447020~20681648,290745721~4774032,295831353~16956137,316072980~4616109,326072099~8583398,338264527~11059895,357458885~7280173,366512993~38252838,404966874~9265321] wrote [3528496~4240299,72723682~4360656,146925241~7131805,156994941~1404463,186686136~6473436,279830462~4729102,327775763~6410563,357458885~7280173] 2024-08-17T17:47:25.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:24 smithi165 ceph-mon[105517]: osdmap e1118: 8 total, 8 up, 8 in 2024-08-17T17:47:25.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:24 smithi165 ceph-mon[105517]: pgmap v1715: 105 pgs: 105 active+clean; 791 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 171 KiB/s rd, 46 MiB/s wr, 281 op/s 2024-08-17T17:47:25.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:24 smithi136 ceph-mon[109861]: osdmap e1118: 8 total, 8 up, 8 in 2024-08-17T17:47:25.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:24 smithi136 ceph-mon[109861]: pgmap v1715: 105 pgs: 105 active+clean; 791 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 171 KiB/s rd, 46 MiB/s wr, 281 op/s 2024-08-17T17:47:25.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:24 smithi136 ceph-mon[115078]: osdmap e1118: 8 total, 8 up, 8 in 2024-08-17T17:47:25.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:24 smithi136 ceph-mon[115078]: pgmap v1715: 105 pgs: 105 active+clean; 791 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 171 KiB/s rd, 46 MiB/s wr, 281 op/s 2024-08-17T17:47:25.817 INFO:tasks.workunit.client.0.smithi136.stdout: i=13 exists [3528496~7370946,11579176~9894787,22298619~3398016,27213408~7936521,43111238~12409142,56172858~6654831,64609489~14013016,83023353~3639184,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~12674875,170236347~879778,173661827~3057406,177148887~16010685,199192089~1285316,200751176~8539407,210384164~3396269,214151721~11450493,229463800~13816941,244079590~6931492,257231876~3214859,265447020~21856450,290745721~4774032,295831353~16956137,316072980~4616109,326072099~8583398,338264527~11059895,357458885~7280173,366512993~38252838,404966874~9265321] wrote [43111238~709358,97918767~799692,201183648~1421815,229463800~6605226,285018335~2285135,301620409~5170183,372530391~10621123] 2024-08-17T17:47:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:25 smithi165 ceph-mon[105517]: osdmap e1119: 8 total, 8 up, 8 in 2024-08-17T17:47:26.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:25 smithi136 ceph-mon[109861]: osdmap e1119: 8 total, 8 up, 8 in 2024-08-17T17:47:26.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:25 smithi136 ceph-mon[115078]: osdmap e1119: 8 total, 8 up, 8 in 2024-08-17T17:47:26.805 INFO:tasks.workunit.client.0.smithi136.stdout: i=14 exists [1598369~392670,3528496~7370946,11579176~9894787,22298619~3398016,27213408~7936521,43111238~12409142,56172858~22449647,83023353~3639184,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~12674875,170236347~879778,173661827~3057406,177148887~16010685,199192089~1285316,200751176~8539407,210384164~3396269,214151721~11450493,229463800~13816941,244079590~9571059,257231876~3214859,265447020~21856450,290745721~4774032,295831353~16956137,316072980~4616109,326072099~8583398,338264527~11059895,357458885~7280173,366512993~38252838,404966874~9265321] wrote [1598369~392670,60835320~6300977,68158340~5780466,151816603~6128341,216394763~3014593,249341666~4308983,295434938~62703,339992078~3460192,397711277~3779285] 2024-08-17T17:47:27.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:26 smithi165 ceph-mon[105517]: osdmap e1120: 8 total, 8 up, 8 in 2024-08-17T17:47:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:26 smithi165 ceph-mon[105517]: pgmap v1718: 105 pgs: 105 active+clean; 890 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 207 KiB/s rd, 25 MiB/s wr, 319 op/s 2024-08-17T17:47:27.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:26 smithi136 ceph-mon[109861]: osdmap e1120: 8 total, 8 up, 8 in 2024-08-17T17:47:27.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:26 smithi136 ceph-mon[109861]: pgmap v1718: 105 pgs: 105 active+clean; 890 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 207 KiB/s rd, 25 MiB/s wr, 319 op/s 2024-08-17T17:47:27.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:26 smithi136 ceph-mon[115078]: osdmap e1120: 8 total, 8 up, 8 in 2024-08-17T17:47:27.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:26 smithi136 ceph-mon[115078]: pgmap v1718: 105 pgs: 105 active+clean; 890 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 207 KiB/s rd, 25 MiB/s wr, 319 op/s 2024-08-17T17:47:27.741 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:47:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:47:27] "GET /metrics HTTP/1.1" 200 36489 "" "Prometheus/2.51.0" 2024-08-17T17:47:27.813 INFO:tasks.workunit.client.0.smithi136.stdout: i=15 exists [1598369~392670,3528496~7370946,11579176~9894787,22298619~3398016,27213408~11717112,43111238~35511267,83023353~3639184,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~12674875,170236347~879778,173661827~3057406,177148887~16010685,199192089~1285316,200751176~8539407,210384164~3396269,214151721~11450493,229463800~13816941,244079590~16367145,265447020~21856450,290745721~4774032,295831353~16956137,316072980~4616109,326072099~8583398,338264527~11059895,357458885~7280173,366512993~38252838,404966874~9265321] wrote [18178412~2116490,31126039~7804481,54625176~7223321,65607886~7657595,252993055~6837523,270348712~4014591,385974724~5541505,395832430~1285281] 2024-08-17T17:47:28.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:27 smithi165 ceph-mon[105517]: osdmap e1121: 8 total, 8 up, 8 in 2024-08-17T17:47:28.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:27 smithi136 ceph-mon[109861]: osdmap e1121: 8 total, 8 up, 8 in 2024-08-17T17:47:28.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:27 smithi136 ceph-mon[115078]: osdmap e1121: 8 total, 8 up, 8 in 2024-08-17T17:47:28.870 INFO:tasks.workunit.client.0.smithi136.stdout: i=16 exists [1598369~392670,3528496~17945467,22298619~3398016,27213408~11717112,43111238~35511267,83023353~9397484,96231003~7638279,104378282~4723101,112459935~7349748,121612481~37583631,170236347~879778,173661827~3057406,177148887~16010685,199192089~10098494,210384164~3396269,214151721~11450493,229463800~13816941,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,326072099~8583398,338264527~11059895,357458885~7280173,364955231~39810600,404966874~9265321] wrote [4299867~7607183,13709905~5440474,83664186~7312323,104378282~1274498,137168385~12023792,199871847~7240303,263839725~5665871,364955231~4891605] 2024-08-17T17:47:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:28 smithi165 ceph-mon[105517]: osdmap e1122: 8 total, 8 up, 8 in 2024-08-17T17:47:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:28 smithi165 ceph-mon[105517]: pgmap v1721: 105 pgs: 105 active+clean; 990 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 207 KiB/s rd, 51 MiB/s wr, 333 op/s 2024-08-17T17:47:29.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:28 smithi136 ceph-mon[115078]: osdmap e1122: 8 total, 8 up, 8 in 2024-08-17T17:47:29.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:28 smithi136 ceph-mon[115078]: pgmap v1721: 105 pgs: 105 active+clean; 990 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 207 KiB/s rd, 51 MiB/s wr, 333 op/s 2024-08-17T17:47:29.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:28 smithi136 ceph-mon[109861]: osdmap e1122: 8 total, 8 up, 8 in 2024-08-17T17:47:29.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:28 smithi136 ceph-mon[109861]: pgmap v1721: 105 pgs: 105 active+clean; 990 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 207 KiB/s rd, 51 MiB/s wr, 333 op/s 2024-08-17T17:47:29.825 INFO:tasks.workunit.client.0.smithi136.stdout: i=17 exists [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37003803,83023353~9397484,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,173661827~3057406,177148887~16010685,199192089~10098494,210384164~3396269,214151721~14137263,229463800~13816941,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,326072099~8583398,338264527~11897077,353675391~14545,357458885~7280173,364955231~39810600,404966874~9265321] wrote [18316322~3379950,60476408~2367307,74032760~6082281,100092746~4080651,168323939~2295785,179399560~3642611,223587612~4701372,345408086~4753518,353675391~14545] 2024-08-17T17:47:29.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:29 smithi136 ceph-mon[109861]: osdmap e1123: 8 total, 8 up, 8 in 2024-08-17T17:47:29.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:29 smithi136 ceph-mon[115078]: osdmap e1123: 8 total, 8 up, 8 in 2024-08-17T17:47:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:29 smithi165 ceph-mon[105517]: osdmap e1123: 8 total, 8 up, 8 in 2024-08-17T17:47:30.843 INFO:tasks.workunit.client.0.smithi136.stdout: i=18 exists [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,83023353~9397484,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,173661827~3057406,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229463800~13816941,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,326072099~8583398,338264527~11897077,353675391~14545,357458885~7280173,364955231~39810600,404966874~9265321] wrote [15854147~4971299,69672745~5644370,80009864~128371,88081530~2792272,138916670~5337091,191922365~3183201,202869045~5347991,216722147~6387656,280132049~79251,339854108~7964355] 2024-08-17T17:47:31.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:30 smithi136 ceph-mon[109861]: osdmap e1124: 8 total, 8 up, 8 in 2024-08-17T17:47:31.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:30 smithi136 ceph-mon[109861]: pgmap v1724: 105 pgs: 105 active+clean; 990 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 27 MiB/s wr, 13 op/s 2024-08-17T17:47:31.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:30 smithi136 ceph-mon[115078]: osdmap e1124: 8 total, 8 up, 8 in 2024-08-17T17:47:31.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:30 smithi136 ceph-mon[115078]: pgmap v1724: 105 pgs: 105 active+clean; 990 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 27 MiB/s wr, 13 op/s 2024-08-17T17:47:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:30 smithi165 ceph-mon[105517]: osdmap e1124: 8 total, 8 up, 8 in 2024-08-17T17:47:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:30 smithi165 ceph-mon[105517]: pgmap v1724: 105 pgs: 105 active+clean; 990 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 27 MiB/s wr, 13 op/s 2024-08-17T17:47:31.850 INFO:tasks.workunit.client.0.smithi136.stdout: i=19 exists [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83023353~9397484,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~13945402,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,326072099~8583398,338264527~11897077,353675391~14545,357458885~47306946,404966874~9265321] wrote [30130057~7268094,81614709~784962,140106859~4041063,171913277~2498215,182522330~5465509,229335339~7013526,281196283~5188907,299679355~367126,359888874~121026,360457183~4536253] 2024-08-17T17:47:32.162 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:31 smithi165 ceph-mon[105517]: osdmap e1125: 8 total, 8 up, 8 in 2024-08-17T17:47:32.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:31 smithi136 ceph-mon[109861]: osdmap e1125: 8 total, 8 up, 8 in 2024-08-17T17:47:32.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:31 smithi136 ceph-mon[115078]: osdmap e1125: 8 total, 8 up, 8 in 2024-08-17T17:47:32.808 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 1 diff [11579176~9299441,66519528~3755847,99070072~2132985,128259422~7709409,181050411~8001537,214151721~1221802,244079590~6931492,338264527~4074143,399426072~5339759] 2024-08-17T17:47:32.808 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [11579176~9299441,66519528~3755847,99070072~2132985,128259422~7709409,181050411~8001537,214151721~1221802,244079590~6931492,338264527~4074143,399426072~5339759] 2024-08-17T17:47:32.826 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [8388608~12490009,62914560~7360815,96468992~4734065,125829120~10139711,180355072~8696876,213909504~1464019,243280741~7730341,335544320~6794350,399426072~5339759] 2024-08-17T17:47:32.826 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 2 diff [8385954~756889,11579176~9299441,56172858~4472149,66519528~6828280,87976177~4444660,99070072~2132985,128259422~7709409,146521237~911867,178804667~10247281,214151721~1221802,234720472~2583266,244079590~6931492,274099473~5145862,300553585~6793976,338264527~4074143,399426072~5339759,406190492~620407] 2024-08-17T17:47:32.826 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [8385954~756889,11579176~9299441,56172858~4472149,66519528~6828280,87976177~4444660,99070072~2132985,128259422~7709409,146521237~911867,178804667~10247281,214151721~1221802,234720472~2583266,244079590~6931492,274099473~5145862,300553585~6793976,338264527~4074143,399426072~5339759,406190492~620407] 2024-08-17T17:47:32.832 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~16684313,56172858~4472149,62914560~10433248,86662537~5758300,96468992~4734065,125829120~10139711,142606336~4826768,176160768~12891180,213909504~1464019,230686720~6617018,243280741~7730341,274099473~5145862,297795584~9551977,335544320~6794350,399426072~7384827] 2024-08-17T17:47:32.832 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 3 diff [8385954~756889,11579176~9299441,56172858~4472149,66519528~6828280,87976177~4444660,99070072~2132985,128259422~12458763,141162200~123733,146521237~911867,170236347~879778,178804667~10247281,214151721~1221802,234720472~2583266,244079590~6931492,266927138~2271880,274099473~5145862,300553585~6793976,316072980~3026910,338264527~4074143,370259166~7251446,392747039~1807197,398184564~6581267,406190492~620407] 2024-08-17T17:47:32.832 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [8385954~756889,11579176~9299441,56172858~4472149,66519528~6828280,87976177~4444660,99070072~2132985,128259422~12458763,141162200~123733,146521237~911867,170236347~879778,178804667~10247281,214151721~1221802,234720472~2583266,244079590~6931492,266927138~2271880,274099473~5145862,300553585~6793976,316072980~3026910,338264527~4074143,370259166~7251446,392747039~1807197,398184564~6581267,406190492~620407] 2024-08-17T17:47:32.839 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~16684313,56172858~4472149,62914560~10433248,86662537~5758300,96468992~4734065,125829120~15456813,142606336~4826768,167772160~3343965,176160768~12891180,213909504~1464019,230686720~6617018,243280741~7730341,264241152~4957866,274099473~5145862,297795584~9551977,314572800~4527090,335544320~6794350,369098752~8411860,390070272~4483964,398184564~8626335] 2024-08-17T17:47:32.839 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 4 diff [8385954~756889,11579176~9299441,27213408~7936521,52524078~2996302,56172858~4472149,66519528~6828280,87976177~4444660,99070072~2132985,128259422~12458763,141162200~123733,146521237~911867,170236347~879778,178804667~10247281,214151721~1221802,216026965~5893923,234720472~2583266,244079590~6931492,266927138~2271880,274099473~5145862,300553585~12233905,316072980~3026910,338264527~4074143,370259166~7251446,385127346~12964499,398184564~6581267,405783969~3077668] 2024-08-17T17:47:32.839 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [8385954~756889,11579176~9299441,27213408~7936521,52524078~2996302,56172858~4472149,66519528~6828280,87976177~4444660,99070072~2132985,128259422~12458763,141162200~123733,146521237~911867,170236347~879778,178804667~10247281,214151721~1221802,216026965~5893923,234720472~2583266,244079590~6931492,266927138~2271880,274099473~5145862,300553585~12233905,316072980~3026910,338264527~4074143,370259166~7251446,385127346~12964499,398184564~6581267,405783969~3077668] 2024-08-17T17:47:32.845 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~16684313,25165824~9984105,50331648~5188732,56172858~4472149,62914560~10433248,86662537~5758300,96468992~4734065,125829120~15456813,142606336~4826768,167772160~3343965,176160768~12891180,213909504~8011384,230686720~6617018,243280741~7730341,264241152~4957866,274099473~5145862,297795584~14991906,314572800~4527090,335544320~6794350,369098752~8411860,381681664~16410181,398184564~10677073] 2024-08-17T17:47:32.845 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 5 diff [7294659~3604783,11579176~9299441,27213408~7936521,46387139~9133241,56172858~4472149,66519528~6828280,87976177~4444660,99070072~2132985,112459935~7349748,128259422~12458763,141162200~123733,146521237~911867,170236347~879778,178804667~10247281,214151721~1221802,216026965~5893923,234720472~2583266,244079590~6931492,266927138~12318197,300553585~12233905,316072980~3026910,332951667~1703830,338264527~4074143,344366004~4958418,366512993~16138442,385127346~12964499,398184564~6581267,405783969~3077668] 2024-08-17T17:47:32.845 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,11579176~9299441,27213408~7936521,46387139~9133241,56172858~4472149,66519528~6828280,87976177~4444660,99070072~2132985,112459935~7349748,128259422~12458763,141162200~123733,146521237~911867,170236347~879778,178804667~10247281,214151721~1221802,216026965~5893923,234720472~2583266,244079590~6931492,266927138~12318197,300553585~12233905,316072980~3026910,332951667~1703830,338264527~4074143,344366004~4958418,366512993~16138442,385127346~12964499,398184564~6581267,405783969~3077668] 2024-08-17T17:47:32.851 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~16684313,25165824~9984105,46137344~9383036,56172858~4472149,62914560~10433248,86662537~5758300,96468992~4734065,109051904~10757779,125829120~15456813,142606336~4826768,167772160~3343965,176160768~12891180,213909504~8011384,230686720~6617018,243280741~7730341,264241152~15004183,297795584~14991906,314572800~4527090,331350016~3305481,335544320~6794350,343932928~5391494,364904448~33187397,398184564~10677073] 2024-08-17T17:47:32.851 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 6 diff [7294659~3604783,11579176~9299441,22298619~3398016,27213408~7936521,46387139~9133241,56172858~4472149,66519528~6828280,87976177~4444660,99070072~2132985,112459935~7349748,128259422~12458763,141162200~123733,143631694~2819484,146521237~911867,170236347~879778,177148887~11903061,203371987~4800717,210384164~3396269,214151721~1221802,216026965~5893923,229813410~1621406,234720472~2583266,244079590~6931492,265447020~13798315,300553585~12233905,316072980~3026910,332951667~1703830,338264527~4074143,344366004~4958418,366512993~16138442,385127346~12964499,398184564~6581267,405783969~3077668] 2024-08-17T17:47:32.851 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,11579176~9299441,22298619~3398016,27213408~7936521,46387139~9133241,56172858~4472149,66519528~6828280,87976177~4444660,99070072~2132985,112459935~7349748,128259422~12458763,141162200~123733,143631694~2819484,146521237~911867,170236347~879778,177148887~11903061,203371987~4800717,210384164~3396269,214151721~1221802,216026965~5893923,229813410~1621406,234720472~2583266,244079590~6931492,265447020~13798315,300553585~12233905,316072980~3026910,332951667~1703830,338264527~4074143,344366004~4958418,366512993~16138442,385127346~12964499,398184564~6581267,405783969~3077668] 2024-08-17T17:47:32.857 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~16684313,20971520~14178409,46137344~9383036,56172858~4472149,62914560~10433248,86662537~5758300,96468992~4734065,109051904~10757779,125829120~15456813,142606336~4826768,167772160~3343965,176160768~12891180,203371987~4800717,209715200~4065233,213909504~8011384,226492416~10811322,243280741~7730341,264241152~15004183,297795584~14991906,314572800~4527090,331350016~3305481,335544320~6794350,343932928~5391494,364904448~33187397,398184564~10677073] 2024-08-17T17:47:32.858 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 7 diff [7294659~3604783,11579176~9299441,22298619~3398016,27213408~7936521,46387139~9133241,56172858~4472149,66519528~7765941,87976177~4444660,98323116~4579656,112459935~7349748,128259422~12458763,141162200~123733,143631694~2819484,146521237~911867,170236347~879778,173661827~3057406,177148887~11903061,203371987~5918596,210384164~3396269,214151721~1221802,216026965~5893923,229813410~1621406,231872994~29434,234720472~2583266,244079590~6931492,259164751~280375,265447020~13798315,295831353~16956137,316072980~4616109,332951667~1703830,338264527~4074143,344366004~4958418,366512993~16138442,385127346~12964499,398184564~6581267,405783969~3077668] 2024-08-17T17:47:32.858 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,11579176~9299441,22298619~3398016,27213408~7936521,46387139~9133241,56172858~4472149,66519528~7765941,87976177~4444660,98323116~4579656,112459935~7349748,128259422~12458763,141162200~123733,143631694~2819484,146521237~911867,170236347~879778,173661827~3057406,177148887~11903061,203371987~5918596,210384164~3396269,214151721~1221802,216026965~5893923,229813410~1621406,231872994~29434,234720472~2583266,244079590~6931492,259164751~280375,265447020~13798315,295831353~16956137,316072980~4616109,332951667~1703830,338264527~4074143,344366004~4958418,366512993~16138442,385127346~12964499,398184564~6581267,405783969~3077668] 2024-08-17T17:47:32.863 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~16684313,20971520~14178409,46137344~9383036,56172858~4472149,62914560~11370909,86662537~5758300,96468992~6433780,109051904~10757779,125829120~15456813,142606336~4826768,167772160~3343965,171966464~17085484,203371987~5918596,209715200~4065233,213909504~8011384,226492416~10811322,243280741~7730341,255852544~3592582,264241152~15004183,293601280~19186210,314572800~6116289,331350016~3305481,335544320~6794350,343932928~5391494,364904448~33187397,398184564~10677073] 2024-08-17T17:47:32.864 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 8 diff [7294659~3604783,11579176~9299441,22298619~3398016,27213408~7936521,46126522~9393858,56172858~4472149,66519528~7765941,87976177~4444660,96231003~7638279,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~911867,151863990~3083981,170236347~879778,173661827~3057406,177148887~11903061,203371987~5918596,210384164~3396269,214151721~1221802,216026965~5893923,229813410~2722880,234720472~2583266,244079590~6931492,259164751~280375,265447020~13798315,295831353~16956137,316072980~4616109,332951667~1703830,338264527~4074143,344366004~4958418,366512993~16138442,385127346~12964499,398184564~6581267,405783969~3077668] 2024-08-17T17:47:32.864 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,11579176~9299441,22298619~3398016,27213408~7936521,46126522~9393858,56172858~4472149,66519528~7765941,87976177~4444660,96231003~7638279,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~911867,151863990~3083981,170236347~879778,173661827~3057406,177148887~11903061,203371987~5918596,210384164~3396269,214151721~1221802,216026965~5893923,229813410~2722880,234720472~2583266,244079590~6931492,259164751~280375,265447020~13798315,295831353~16956137,316072980~4616109,332951667~1703830,338264527~4074143,344366004~4958418,366512993~16138442,385127346~12964499,398184564~6581267,405783969~3077668] 2024-08-17T17:47:32.870 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~16684313,20971520~14178409,41943040~13577340,56172858~4472149,62914560~11370909,86662537~17206745,109051904~32234029,142606336~4826768,150994944~3953027,167772160~3343965,171966464~17085484,203371987~5918596,209715200~4065233,213909504~8011384,226492416~10811322,243280741~7730341,255852544~3592582,264241152~15004183,293601280~19186210,314572800~6116289,331350016~3305481,335544320~6794350,343932928~5391494,364904448~33187397,398184564~10677073] 2024-08-17T17:47:32.870 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 9 diff [7294659~3604783,11579176~9894787,22298619~3398016,27213408~7936521,46126522~9393858,56172858~4472149,66519528~7765941,87976177~4444660,96231003~7638279,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~911867,151863990~3892627,170236347~879778,173661827~3057406,177148887~11903061,199192089~1285316,203371987~5918596,210384164~3396269,214151721~7769167,229813410~2722880,234720472~6909024,244079590~6931492,259164751~280375,265447020~14406806,295831353~16956137,316072980~4616109,332951667~1703830,338264527~11059895,366512993~16138442,385127346~12964499,398184564~6581267,404966874~7561094] 2024-08-17T17:47:32.870 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,11579176~9894787,22298619~3398016,27213408~7936521,46126522~9393858,56172858~4472149,66519528~7765941,87976177~4444660,96231003~7638279,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~911867,151863990~3892627,170236347~879778,173661827~3057406,177148887~11903061,199192089~1285316,203371987~5918596,210384164~3396269,214151721~7769167,229813410~2722880,234720472~6909024,244079590~6931492,259164751~280375,265447020~14406806,295831353~16956137,316072980~4616109,332951667~1703830,338264527~11059895,366512993~16138442,385127346~12964499,398184564~6581267,404966874~7561094] 2024-08-17T17:47:32.876 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~30955625,41943040~13577340,56172858~4472149,62914560~11370909,86662537~17206745,109051904~32234029,142606336~4826768,150994944~4761673,167772160~3343965,171966464~17085484,197132288~3345117,203371987~5918596,209715200~4065233,213909504~8011384,226492416~15137080,243280741~7730341,255852544~3592582,264241152~15612674,293601280~19186210,314572800~6116289,331350016~3305481,335544320~13780102,364904448~33187397,398184564~14343404] 2024-08-17T17:47:32.876 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 10 diff [7294659~3604783,11579176~9894787,22298619~3398016,27213408~7936521,46126522~9393858,56172858~4472149,66519528~7765941,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~911867,151863990~7332122,170236347~879778,173661827~3057406,177148887~11903061,199192089~1285316,200751176~8539407,210384164~3396269,214151721~7769167,229813410~2722880,234720472~6909024,244079590~6931492,257231876~3214859,265447020~14406806,284082763~2045905,290745721~4774032,295831353~16956137,316072980~4616109,332951667~1703830,338264527~11059895,366512993~16138442,385127346~12964499,398184564~6581267,404966874~7561094] 2024-08-17T17:47:32.876 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,11579176~9894787,22298619~3398016,27213408~7936521,46126522~9393858,56172858~4472149,66519528~7765941,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~911867,151863990~7332122,170236347~879778,173661827~3057406,177148887~11903061,199192089~1285316,200751176~8539407,210384164~3396269,214151721~7769167,229813410~2722880,234720472~6909024,244079590~6931492,257231876~3214859,265447020~14406806,284082763~2045905,290745721~4774032,295831353~16956137,316072980~4616109,332951667~1703830,338264527~11059895,366512993~16138442,385127346~12964499,398184564~6581267,404966874~7561094] 2024-08-17T17:47:32.882 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~30955625,41943040~13577340,56172858~4472149,62914560~11370909,86662537~54623396,142606336~4826768,150994944~8201168,167772160~3343965,171966464~17085484,197132288~12158295,209715200~4065233,213909504~8011384,226492416~15137080,243280741~7730341,255852544~4594191,264241152~15612674,281018368~5110300,289406976~23380514,314572800~6116289,331350016~3305481,335544320~13780102,364904448~33187397,398184564~14343404] 2024-08-17T17:47:32.882 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 11 diff [7294659~3604783,11579176~9894787,22298619~3398016,27213408~7936521,43810632~11709748,56172858~6654831,64609489~9675980,76085135~2537370,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~911867,151863990~7332122,170236347~879778,173661827~3057406,177148887~11903061,199192089~1285316,200751176~8539407,210384164~3396269,214151721~11450493,229813410~2722880,234720472~6909024,244079590~6931492,257231876~3214859,265447020~14406806,284082763~2045905,290745721~4774032,295831353~16956137,316072980~4616109,332951667~1703830,338264527~11059895,366512993~31578852,398184564~6581267,404966874~7561094] 2024-08-17T17:47:32.882 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,11579176~9894787,22298619~3398016,27213408~7936521,43810632~11709748,56172858~6654831,64609489~9675980,76085135~2537370,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~911867,151863990~7332122,170236347~879778,173661827~3057406,177148887~11903061,199192089~1285316,200751176~8539407,210384164~3396269,214151721~11450493,229813410~2722880,234720472~6909024,244079590~6931492,257231876~3214859,265447020~14406806,284082763~2045905,290745721~4774032,295831353~16956137,316072980~4616109,332951667~1703830,338264527~11059895,366512993~31578852,398184564~6581267,404966874~7561094] 2024-08-17T17:47:32.888 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~30955625,41943040~13577340,56172858~6654831,62914560~11370909,75497472~3125033,86662537~54623396,142606336~4826768,150994944~8201168,167772160~3343965,171966464~17085484,197132288~12158295,209715200~4065233,213909504~11692710,226492416~15137080,243280741~7730341,255852544~4594191,264241152~15612674,281018368~5110300,289406976~23380514,314572800~6116289,331350016~3305481,335544320~13780102,364904448~33187397,398184564~14343404] 2024-08-17T17:47:32.888 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 12 diff [3528496~7370946,11579176~9894787,22298619~3398016,27213408~7936521,43810632~11709748,56172858~6654831,64609489~14013016,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~12674875,170236347~879778,173661827~3057406,177148887~16010685,199192089~1285316,200751176~8539407,210384164~3396269,214151721~11450493,229813410~2722880,234720472~6909024,244079590~6931492,257231876~3214859,265447020~20681648,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11059895,357458885~7280173,366512993~31578852,398184564~6581267,404966874~7561094] 2024-08-17T17:47:32.888 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [3528496~7370946,11579176~9894787,22298619~3398016,27213408~7936521,43810632~11709748,56172858~6654831,64609489~14013016,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~12674875,170236347~879778,173661827~3057406,177148887~16010685,199192089~1285316,200751176~8539407,210384164~3396269,214151721~11450493,229813410~2722880,234720472~6909024,244079590~6931492,257231876~3214859,265447020~20681648,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11059895,357458885~7280173,366512993~31578852,398184564~6581267,404966874~7561094] 2024-08-17T17:47:32.894 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~35149929,41943040~13577340,56172858~6654831,62914560~15707945,86662537~54623396,142606336~16589776,167772160~3343965,171966464~21193108,197132288~12158295,209715200~4065233,213909504~11692710,226492416~15137080,243280741~7730341,255852544~4594191,264241152~21887516,289406976~23380514,314572800~6116289,327775763~6879734,335544320~13780102,356515840~8223218,364904448~33187397,398184564~14343404] 2024-08-17T17:47:32.894 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 13 diff [3528496~7370946,11579176~9894787,22298619~3398016,27213408~7936521,43111238~12409142,56172858~6654831,64609489~14013016,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~12674875,170236347~879778,173661827~3057406,177148887~16010685,199192089~1285316,200751176~8539407,210384164~3396269,214151721~11450493,229463800~12165696,244079590~6931492,257231876~3214859,265447020~21856450,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11059895,357458885~7280173,366512993~31578852,398184564~6581267,404966874~7561094] 2024-08-17T17:47:32.895 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [3528496~7370946,11579176~9894787,22298619~3398016,27213408~7936521,43111238~12409142,56172858~6654831,64609489~14013016,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~12674875,170236347~879778,173661827~3057406,177148887~16010685,199192089~1285316,200751176~8539407,210384164~3396269,214151721~11450493,229463800~12165696,244079590~6931492,257231876~3214859,265447020~21856450,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11059895,357458885~7280173,366512993~31578852,398184564~6581267,404966874~7561094] 2024-08-17T17:47:32.900 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~35149929,41943040~13577340,56172858~6654831,62914560~15707945,86662537~54623396,142606336~16589776,167772160~3343965,171966464~21193108,197132288~12158295,209715200~4065233,213909504~11692710,226492416~15137080,243280741~7730341,255852544~4594191,264241152~23062318,289406976~23380514,314572800~6116289,327775763~6879734,335544320~13780102,356515840~8223218,364904448~33187397,398184564~14343404] 2024-08-17T17:47:32.900 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 14 diff [1598369~392670,3528496~7370946,11579176~9894787,22298619~3398016,27213408~7936521,43111238~12409142,56172858~22449647,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~12674875,170236347~879778,173661827~3057406,177148887~16010685,199192089~1285316,200751176~8539407,210384164~3396269,214151721~11450493,229463800~12165696,244079590~9571059,257231876~3214859,265447020~21856450,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11059895,357458885~7280173,366512993~38252838,404966874~7561094] 2024-08-17T17:47:32.900 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~7370946,11579176~9894787,22298619~3398016,27213408~7936521,43111238~12409142,56172858~22449647,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~12674875,170236347~879778,173661827~3057406,177148887~16010685,199192089~1285316,200751176~8539407,210384164~3396269,214151721~11450493,229463800~12165696,244079590~9571059,257231876~3214859,265447020~21856450,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11059895,357458885~7280173,366512993~38252838,404966874~7561094] 2024-08-17T17:47:32.906 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~35149929,41943040~13577340,56172858~22449647,86662537~54623396,142606336~16589776,167772160~3343965,171966464~21193108,197132288~12158295,209715200~4065233,213909504~11692710,226492416~15137080,243280741~10369908,255852544~4594191,264241152~23062318,289406976~23380514,314572800~6116289,327775763~6879734,335544320~13780102,356515840~8223218,364904448~47623520] 2024-08-17T17:47:32.906 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 15 diff [1598369~392670,3528496~7370946,11579176~9894787,22298619~3398016,27213408~11717112,43111238~35511267,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~12674875,170236347~879778,173661827~3057406,177148887~16010685,199192089~1285316,200751176~8539407,210384164~3396269,214151721~11450493,229463800~12165696,244079590~16367145,265447020~21856450,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11059895,357458885~7280173,366512993~38252838,404966874~7561094] 2024-08-17T17:47:32.907 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~7370946,11579176~9894787,22298619~3398016,27213408~11717112,43111238~35511267,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~12674875,170236347~879778,173661827~3057406,177148887~16010685,199192089~1285316,200751176~8539407,210384164~3396269,214151721~11450493,229463800~12165696,244079590~16367145,265447020~21856450,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11059895,357458885~7280173,366512993~38252838,404966874~7561094] 2024-08-17T17:47:32.912 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~38930520,41943040~36679465,86662537~54623396,142606336~16589776,167772160~3343965,171966464~21193108,197132288~12158295,209715200~4065233,213909504~11692710,226492416~15137080,243280741~17165994,264241152~23062318,289406976~23380514,314572800~6116289,327775763~6879734,335544320~13780102,356515840~8223218,364904448~47623520] 2024-08-17T17:47:32.912 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 16 diff [1598369~392670,3528496~17945467,22298619~3398016,27213408~11717112,43111238~35511267,83664186~8756651,96231003~7638279,104378282~4723101,112459935~7349748,121612481~37583631,170236347~879778,173661827~3057406,177148887~16010685,199192089~10098494,210384164~3396269,214151721~11450493,229463800~12165696,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11059895,357458885~7280173,364955231~39810600,404966874~7561094] 2024-08-17T17:47:32.912 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~17945467,22298619~3398016,27213408~11717112,43111238~35511267,83664186~8756651,96231003~7638279,104378282~4723101,112459935~7349748,121612481~37583631,170236347~879778,173661827~3057406,177148887~16010685,199192089~10098494,210384164~3396269,214151721~11450493,229463800~12165696,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11059895,357458885~7280173,364955231~39810600,404966874~7561094] 2024-08-17T17:47:32.918 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~38930520,41943040~36679465,83664186~75531926,167772160~3343965,171966464~21193108,197132288~12158295,209715200~4065233,213909504~11692710,226492416~15137080,243280741~44022729,289406976~23380514,314572800~6116289,327775763~6879734,335544320~13780102,356515840~8223218,364904448~47623520] 2024-08-17T17:47:32.918 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 17 diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37003803,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,173661827~3057406,177148887~16010685,199192089~10098494,210384164~3396269,214151721~14137263,229463800~12165696,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~7280173,364955231~39810600,404966874~7561094] 2024-08-17T17:47:32.918 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37003803,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,173661827~3057406,177148887~16010685,199192089~10098494,210384164~3396269,214151721~14137263,229463800~12165696,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~7280173,364955231~39810600,404966874~7561094] 2024-08-17T17:47:32.924 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~38930520,41943040~38172001,83664186~75531926,167772160~3343965,171966464~21193108,197132288~12158295,209715200~4065233,213909504~27719992,243280741~44022729,289406976~23380514,314572800~6116289,327775763~6879734,335544320~14617284,352321536~1368400,356515840~8223218,364904448~47623520] 2024-08-17T17:47:32.924 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 18 diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,173661827~3057406,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229463800~12165696,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~7280173,364955231~39810600,404966874~7561094] 2024-08-17T17:47:32.924 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,173661827~3057406,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229463800~12165696,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~7280173,364955231~39810600,404966874~7561094] 2024-08-17T17:47:32.930 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~38930520,41943040~38195195,83664186~75531926,167772160~3343965,171966464~23139102,197132288~12158295,209715200~4065233,213909504~27719992,243280741~44022729,289406976~23380514,314572800~6116289,327775763~6879734,335544320~14617284,352321536~1368400,356515840~8223218,364904448~47623520] 2024-08-17T17:47:32.930 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 19 diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:32.930 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:32.936 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~38930520,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,213909504~27719992,243280741~44022729,289406976~23380514,314572800~6116289,327775763~6879734,335544320~14617284,352321536~1368400,356515840~56012128] 2024-08-17T17:47:32.936 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 2 diff [8385954~756889,56172858~4472149,67669518~5678290,87976177~4444660,146521237~911867,178804667~5897510,234720472~2583266,274099473~5145862,300553585~6793976,406190492~620407] 2024-08-17T17:47:32.937 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [8385954~756889,56172858~4472149,67669518~5678290,87976177~4444660,146521237~911867,178804667~5897510,234720472~2583266,274099473~5145862,300553585~6793976,406190492~620407] 2024-08-17T17:47:32.942 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~4948539,56172858~4472149,67669518~5678290,86662537~5758300,142606336~4826768,176160768~8541409,230686720~6617018,274099473~5145862,297795584~9551977,404765831~2045068] 2024-08-17T17:47:32.942 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 3 diff [8385954~756889,56172858~4472149,67669518~5678290,87976177~4444660,134789056~5929129,141162200~123733,146521237~911867,170236347~879778,178804667~5897510,234720472~2583266,266927138~2271880,274099473~5145862,300553585~6793976,316072980~3026910,370259166~7251446,392747039~1807197,398184564~2539553,406190492~620407] 2024-08-17T17:47:32.942 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [8385954~756889,56172858~4472149,67669518~5678290,87976177~4444660,134789056~5929129,141162200~123733,146521237~911867,170236347~879778,178804667~5897510,234720472~2583266,266927138~2271880,274099473~5145862,300553585~6793976,316072980~3026910,370259166~7251446,392747039~1807197,398184564~2539553,406190492~620407] 2024-08-17T17:47:32.948 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~4948539,56172858~4472149,67669518~5678290,86662537~5758300,134789056~6496877,142606336~4826768,167772160~3343965,176160768~8541409,230686720~6617018,264241152~4957866,274099473~5145862,297795584~9551977,314572800~4527090,369098752~8411860,390070272~4483964,398184564~2539553,404765831~2045068] 2024-08-17T17:47:32.948 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 4 diff [8385954~756889,27213408~7936521,52524078~2996302,56172858~4472149,67669518~5678290,87976177~4444660,132770613~7947572,141162200~123733,146521237~911867,170236347~879778,178804667~6794693,216026965~5893923,234720472~2583266,266927138~2271880,274099473~5145862,300553585~12233905,316072980~3026910,370259166~7251446,385127346~12964499,398184564~2539553,405783969~3077668] 2024-08-17T17:47:32.948 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [8385954~756889,27213408~7936521,52524078~2996302,56172858~4472149,67669518~5678290,87976177~4444660,132770613~7947572,141162200~123733,146521237~911867,170236347~879778,178804667~6794693,216026965~5893923,234720472~2583266,266927138~2271880,274099473~5145862,300553585~12233905,316072980~3026910,370259166~7251446,385127346~12964499,398184564~2539553,405783969~3077668] 2024-08-17T17:47:32.954 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~4948539,25165824~9984105,50331648~5188732,56172858~4472149,67669518~5678290,86662537~5758300,132770613~8515320,142606336~4826768,167772160~3343965,176160768~9438592,215373523~6547365,230686720~6617018,264241152~4957866,274099473~5145862,297795584~14991906,314572800~4527090,369098752~8411860,381681664~16410181,398184564~2539553,404765831~4095806] 2024-08-17T17:47:32.954 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 5 diff [7294659~3604783,14418625~2503052,27213408~7936521,46387139~9133241,56172858~4472149,67669518~5678290,87976177~4444660,112459935~7349748,132770613~7947572,141162200~123733,146521237~911867,170236347~879778,178804667~6794693,216026965~5893923,234720472~2583266,266927138~12318197,300553585~12233905,316072980~3026910,332951667~1703830,344366004~4958418,366512993~16138442,385127346~12964499,398184564~2539553,405783969~3077668] 2024-08-17T17:47:32.954 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14418625~2503052,27213408~7936521,46387139~9133241,56172858~4472149,67669518~5678290,87976177~4444660,112459935~7349748,132770613~7947572,141162200~123733,146521237~911867,170236347~879778,178804667~6794693,216026965~5893923,234720472~2583266,266927138~12318197,300553585~12233905,316072980~3026910,332951667~1703830,344366004~4958418,366512993~16138442,385127346~12964499,398184564~2539553,405783969~3077668] 2024-08-17T17:47:32.960 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~6705138,14418625~2503052,25165824~9984105,46137344~9383036,56172858~4472149,67669518~5678290,86662537~5758300,109051904~10757779,132770613~8515320,142606336~4826768,167772160~3343965,176160768~9438592,215373523~6547365,230686720~6617018,264241152~15004183,297795584~14991906,314572800~4527090,331350016~3305481,343932928~5391494,364904448~33187397,398184564~2539553,404765831~4095806] 2024-08-17T17:47:32.960 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 6 diff [7294659~3604783,14418625~2503052,18397559~2377817,22298619~3398016,27213408~7936521,46387139~9133241,56172858~4472149,67669518~5678290,87976177~4444660,112459935~7349748,132770613~7947572,141162200~123733,143631694~2819484,146521237~911867,170236347~879778,177148887~8450473,203371987~4800717,210384164~3396269,216026965~5893923,229813410~1621406,234720472~2583266,265447020~13798315,300553585~12233905,316072980~3026910,332951667~1703830,344366004~4958418,366512993~16138442,385127346~12964499,398184564~2539553,405783969~3077668] 2024-08-17T17:47:32.960 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14418625~2503052,18397559~2377817,22298619~3398016,27213408~7936521,46387139~9133241,56172858~4472149,67669518~5678290,87976177~4444660,112459935~7349748,132770613~7947572,141162200~123733,143631694~2819484,146521237~911867,170236347~879778,177148887~8450473,203371987~4800717,210384164~3396269,216026965~5893923,229813410~1621406,234720472~2583266,265447020~13798315,300553585~12233905,316072980~3026910,332951667~1703830,344366004~4958418,366512993~16138442,385127346~12964499,398184564~2539553,405783969~3077668] 2024-08-17T17:47:32.965 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~6705138,14418625~2503052,18397559~2377817,20971520~14178409,46137344~9383036,56172858~4472149,67669518~5678290,86662537~5758300,109051904~10757779,132770613~8515320,142606336~4826768,167772160~3343965,176160768~9438592,203371987~4800717,209715200~4065233,215373523~6547365,226492416~10811322,264241152~15004183,297795584~14991906,314572800~4527090,331350016~3305481,343932928~5391494,364904448~33187397,398184564~2539553,404765831~4095806] 2024-08-17T17:47:32.966 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 7 diff [7294659~3604783,14418625~2503052,18397559~2377817,22298619~3398016,27213408~7936521,46387139~9133241,56172858~4472149,67669518~6615951,87976177~4444660,98323116~4579656,112459935~7349748,130078852~10639333,141162200~123733,143631694~2819484,146521237~911867,170236347~879778,173661827~3057406,177148887~8450473,203371987~5918596,210384164~3396269,216026965~5893923,229813410~1621406,231872994~29434,234720472~2583266,259164751~280375,265447020~13798315,295831353~16956137,316072980~4616109,332951667~1703830,344366004~4958418,366512993~16138442,385127346~12964499,398184564~2539553,401465291~1068983,405783969~3077668] 2024-08-17T17:47:32.966 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14418625~2503052,18397559~2377817,22298619~3398016,27213408~7936521,46387139~9133241,56172858~4472149,67669518~6615951,87976177~4444660,98323116~4579656,112459935~7349748,130078852~10639333,141162200~123733,143631694~2819484,146521237~911867,170236347~879778,173661827~3057406,177148887~8450473,203371987~5918596,210384164~3396269,216026965~5893923,229813410~1621406,231872994~29434,234720472~2583266,259164751~280375,265447020~13798315,295831353~16956137,316072980~4616109,332951667~1703830,344366004~4958418,366512993~16138442,385127346~12964499,398184564~2539553,401465291~1068983,405783969~3077668] 2024-08-17T17:47:32.971 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~6705138,14418625~2503052,18397559~2377817,20971520~14178409,46137344~9383036,56172858~4472149,67669518~6615951,86662537~5758300,98323116~4579656,109051904~10757779,130078852~11207081,142606336~4826768,167772160~3343965,171966464~13632896,203371987~5918596,209715200~4065233,215373523~6547365,226492416~10811322,255852544~3592582,264241152~15004183,293601280~19186210,314572800~6116289,331350016~3305481,343932928~5391494,364904448~33187397,398184564~2539553,401465291~1068983,404765831~4095806] 2024-08-17T17:47:32.971 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 8 diff [7294659~3604783,14418625~2503052,18397559~2377817,22298619~3398016,27213408~7936521,46126522~9393858,56172858~4472149,67669518~6615951,87976177~4444660,96231003~7638279,112459935~7349748,121612481~8169769,130078852~10639333,141162200~123733,143631694~2819484,146521237~911867,151863990~3083981,170236347~879778,173661827~3057406,177148887~11051491,203371987~5918596,210384164~3396269,216026965~5893923,229813410~2722880,234720472~2583266,259164751~280375,265447020~13798315,295831353~16956137,316072980~4616109,332951667~1703830,344366004~4958418,366512993~16138442,385127346~12964499,398184564~2539553,401465291~1068983,405783969~3077668] 2024-08-17T17:47:32.971 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14418625~2503052,18397559~2377817,22298619~3398016,27213408~7936521,46126522~9393858,56172858~4472149,67669518~6615951,87976177~4444660,96231003~7638279,112459935~7349748,121612481~8169769,130078852~10639333,141162200~123733,143631694~2819484,146521237~911867,151863990~3083981,170236347~879778,173661827~3057406,177148887~11051491,203371987~5918596,210384164~3396269,216026965~5893923,229813410~2722880,234720472~2583266,259164751~280375,265447020~13798315,295831353~16956137,316072980~4616109,332951667~1703830,344366004~4958418,366512993~16138442,385127346~12964499,398184564~2539553,401465291~1068983,405783969~3077668] 2024-08-17T17:47:32.977 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~6705138,14418625~2503052,18397559~2377817,20971520~14178409,41943040~13577340,56172858~4472149,67669518~6615951,86662537~17206745,109051904~20730346,130078852~11207081,142606336~4826768,150994944~3953027,167772160~3343965,171966464~16233914,203371987~5918596,209715200~4065233,215373523~6547365,226492416~10811322,255852544~3592582,264241152~15004183,293601280~19186210,314572800~6116289,331350016~3305481,343932928~5391494,364904448~33187397,398184564~2539553,401465291~1068983,404765831~4095806] 2024-08-17T17:47:32.977 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 9 diff [7294659~3604783,14137310~7336653,22298619~3398016,27213408~7936521,46126522~9393858,56172858~4472149,67669518~6615951,87976177~4444660,96231003~7638279,112459935~7349748,121612481~8169769,130078852~10639333,141162200~123733,143631694~2819484,146521237~911867,151863990~3892627,170236347~879778,173661827~3057406,177148887~11051491,199192089~1285316,203371987~5918596,210384164~3396269,214907306~7013582,229813410~2722880,234720472~6909024,259164751~280375,265447020~14406806,295831353~16956137,316072980~4616109,332951667~1703830,342231213~7093209,366512993~16138442,385127346~12964499,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:32.978 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14137310~7336653,22298619~3398016,27213408~7936521,46126522~9393858,56172858~4472149,67669518~6615951,87976177~4444660,96231003~7638279,112459935~7349748,121612481~8169769,130078852~10639333,141162200~123733,143631694~2819484,146521237~911867,151863990~3892627,170236347~879778,173661827~3057406,177148887~11051491,199192089~1285316,203371987~5918596,210384164~3396269,214907306~7013582,229813410~2722880,234720472~6909024,259164751~280375,265447020~14406806,295831353~16956137,316072980~4616109,332951667~1703830,342231213~7093209,366512993~16138442,385127346~12964499,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:32.983 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~6705138,14137310~21012619,41943040~13577340,56172858~4472149,67669518~6615951,86662537~17206745,109051904~20730346,130078852~11207081,142606336~4826768,150994944~4761673,167772160~3343965,171966464~16233914,197132288~3345117,203371987~5918596,209715200~4065233,214907306~7013582,226492416~15137080,255852544~3592582,264241152~15612674,293601280~19186210,314572800~6116289,331350016~3305481,342231213~7093209,364904448~33187397,398184564~2539553,401465291~1068983,404765831~7762137] 2024-08-17T17:47:32.983 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 10 diff [7294659~3604783,14137310~7336653,22298619~3398016,27213408~7936521,46126522~9393858,56172858~4472149,67669518~6615951,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~8169769,130078852~10639333,141162200~123733,143631694~2819484,146521237~911867,151863990~7332122,170236347~879778,173661827~3057406,177148887~11051491,199192089~1285316,200751176~8539407,210384164~3396269,214907306~7013582,229813410~2722880,234720472~6909024,257231876~3214859,265447020~14406806,284082763~2045905,290745721~4774032,295831353~16956137,316072980~4616109,332951667~1703830,342231213~7093209,366512993~16138442,385127346~12964499,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:32.983 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14137310~7336653,22298619~3398016,27213408~7936521,46126522~9393858,56172858~4472149,67669518~6615951,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~8169769,130078852~10639333,141162200~123733,143631694~2819484,146521237~911867,151863990~7332122,170236347~879778,173661827~3057406,177148887~11051491,199192089~1285316,200751176~8539407,210384164~3396269,214907306~7013582,229813410~2722880,234720472~6909024,257231876~3214859,265447020~14406806,284082763~2045905,290745721~4774032,295831353~16956137,316072980~4616109,332951667~1703830,342231213~7093209,366512993~16138442,385127346~12964499,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:32.988 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~6705138,14137310~21012619,41943040~13577340,56172858~4472149,67669518~6615951,86662537~43119713,130078852~11207081,142606336~4826768,150994944~8201168,167772160~3343965,171966464~16233914,197132288~12158295,209715200~4065233,214907306~7013582,226492416~15137080,255852544~4594191,264241152~15612674,281018368~5110300,289406976~23380514,314572800~6116289,331350016~3305481,342231213~7093209,364904448~33187397,398184564~2539553,401465291~1068983,404765831~7762137] 2024-08-17T17:47:32.988 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 11 diff [7294659~3604783,14137310~7336653,22298619~3398016,27213408~7936521,43810632~11709748,56172858~6654831,64609489~9675980,76085135~2537370,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~911867,151863990~7332122,170236347~879778,173661827~3057406,177148887~11051491,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229813410~2722880,234720472~6909024,257231876~3214859,265447020~14406806,284082763~2045905,290745721~4774032,295831353~16956137,316072980~4616109,332951667~1703830,342231213~7093209,366512993~31578852,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:32.989 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14137310~7336653,22298619~3398016,27213408~7936521,43810632~11709748,56172858~6654831,64609489~9675980,76085135~2537370,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~911867,151863990~7332122,170236347~879778,173661827~3057406,177148887~11051491,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229813410~2722880,234720472~6909024,257231876~3214859,265447020~14406806,284082763~2045905,290745721~4774032,295831353~16956137,316072980~4616109,332951667~1703830,342231213~7093209,366512993~31578852,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:32.994 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~6705138,14137310~21012619,41943040~13577340,56172858~6654831,64609489~9675980,75497472~3125033,86662537~54623396,142606336~4826768,150994944~8201168,167772160~3343965,171966464~16233914,197132288~12158295,209715200~4065233,214907306~10694908,226492416~15137080,255852544~4594191,264241152~15612674,281018368~5110300,289406976~23380514,314572800~6116289,331350016~3305481,342231213~7093209,364904448~33187397,398184564~2539553,401465291~1068983,404765831~7762137] 2024-08-17T17:47:32.994 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 12 diff [3528496~7370946,14137310~7336653,22298619~3398016,27213408~7936521,43810632~11709748,56172858~6654831,64609489~14013016,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~12674875,170236347~879778,173661827~3057406,177148887~16010685,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229813410~2722880,234720472~6909024,257231876~3214859,265447020~20681648,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,342231213~7093209,357458885~7280173,366512993~31578852,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:32.995 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [3528496~7370946,14137310~7336653,22298619~3398016,27213408~7936521,43810632~11709748,56172858~6654831,64609489~14013016,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~12674875,170236347~879778,173661827~3057406,177148887~16010685,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229813410~2722880,234720472~6909024,257231876~3214859,265447020~20681648,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,342231213~7093209,357458885~7280173,366512993~31578852,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.000 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~10899442,14137310~21012619,41943040~13577340,56172858~6654831,64609489~14013016,86662537~54623396,142606336~16589776,167772160~3343965,171966464~21193108,197132288~12158295,209715200~4065233,214907306~10694908,226492416~15137080,255852544~4594191,264241152~21887516,289406976~23380514,314572800~6116289,327775763~6879734,342231213~7093209,356515840~8223218,364904448~33187397,398184564~2539553,401465291~1068983,404765831~7762137] 2024-08-17T17:47:33.000 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 13 diff [3528496~7370946,14137310~7336653,22298619~3398016,27213408~7936521,43111238~12409142,56172858~6654831,64609489~14013016,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~12674875,170236347~879778,173661827~3057406,177148887~16010685,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,257231876~3214859,265447020~21856450,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,342231213~7093209,357458885~7280173,366512993~31578852,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.000 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [3528496~7370946,14137310~7336653,22298619~3398016,27213408~7936521,43111238~12409142,56172858~6654831,64609489~14013016,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~12674875,170236347~879778,173661827~3057406,177148887~16010685,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,257231876~3214859,265447020~21856450,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,342231213~7093209,357458885~7280173,366512993~31578852,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.006 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~10899442,14137310~21012619,41943040~13577340,56172858~6654831,64609489~14013016,86662537~54623396,142606336~16589776,167772160~3343965,171966464~21193108,197132288~12158295,209715200~4065233,214907306~10694908,226492416~15137080,255852544~4594191,264241152~23062318,289406976~23380514,314572800~6116289,327775763~6879734,342231213~7093209,356515840~8223218,364904448~33187397,398184564~2539553,401465291~1068983,404765831~7762137] 2024-08-17T17:47:33.006 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 14 diff [1598369~392670,3528496~7370946,14137310~7336653,22298619~3398016,27213408~7936521,43111238~12409142,56172858~22449647,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~12674875,170236347~879778,173661827~3057406,177148887~16010685,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,249341666~4308983,257231876~3214859,265447020~21856450,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339992078~9332344,357458885~7280173,366512993~36021281,404966874~7561094] 2024-08-17T17:47:33.007 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~7370946,14137310~7336653,22298619~3398016,27213408~7936521,43111238~12409142,56172858~22449647,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~12674875,170236347~879778,173661827~3057406,177148887~16010685,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,249341666~4308983,257231876~3214859,265447020~21856450,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339992078~9332344,357458885~7280173,366512993~36021281,404966874~7561094] 2024-08-17T17:47:33.012 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~10899442,14137310~21012619,41943040~13577340,56172858~22449647,86662537~54623396,142606336~16589776,167772160~3343965,171966464~21193108,197132288~12158295,209715200~4065233,214907306~10694908,226492416~15137080,249341666~4308983,255852544~4594191,264241152~23062318,289406976~23380514,314572800~6116289,327775763~6879734,339992078~9332344,356515840~8223218,364904448~37629826,404765831~7762137] 2024-08-17T17:47:33.012 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 15 diff [1598369~392670,3528496~7370946,14137310~7336653,22298619~3398016,27213408~11717112,43111238~35511267,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~12674875,170236347~879778,173661827~3057406,177148887~16010685,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,249341666~11105069,265447020~21856450,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339992078~9332344,357458885~7280173,366512993~36021281,404966874~7561094] 2024-08-17T17:47:33.012 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~7370946,14137310~7336653,22298619~3398016,27213408~11717112,43111238~35511267,87976177~4444660,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146521237~12674875,170236347~879778,173661827~3057406,177148887~16010685,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,249341666~11105069,265447020~21856450,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339992078~9332344,357458885~7280173,366512993~36021281,404966874~7561094] 2024-08-17T17:47:33.017 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~10899442,14137310~24793210,41943040~36679465,86662537~54623396,142606336~16589776,167772160~3343965,171966464~21193108,197132288~12158295,209715200~4065233,214907306~10694908,226492416~15137080,249341666~11105069,264241152~23062318,289406976~23380514,314572800~6116289,327775763~6879734,339992078~9332344,356515840~8223218,364904448~37629826,404765831~7762137] 2024-08-17T17:47:33.017 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 16 diff [1598369~392670,3528496~8378554,13709905~7764058,22298619~3398016,27213408~11717112,43111238~35511267,83664186~8756651,96231003~7638279,104378282~4723101,112459935~7349748,121612481~37583631,170236347~879778,173661827~3057406,177148887~16010685,199192089~10098494,210384164~3396269,214907306~10694908,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339992078~9332344,357458885~7280173,364955231~37579043,404966874~7561094] 2024-08-17T17:47:33.017 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7764058,22298619~3398016,27213408~11717112,43111238~35511267,83664186~8756651,96231003~7638279,104378282~4723101,112459935~7349748,121612481~37583631,170236347~879778,173661827~3057406,177148887~16010685,199192089~10098494,210384164~3396269,214907306~10694908,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339992078~9332344,357458885~7280173,364955231~37579043,404966874~7561094] 2024-08-17T17:47:33.023 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~36679465,83664186~75531926,167772160~3343965,171966464~21193108,197132288~12158295,209715200~4065233,214907306~10694908,226492416~15137080,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339992078~9332344,356515840~8223218,364904448~37629826,404765831~7762137] 2024-08-17T17:47:33.024 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 17 diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37003803,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,173661827~3057406,177148887~16010685,199192089~10098494,210384164~3396269,214907306~13381678,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339992078~10169526,353675391~14545,357458885~7280173,364955231~37579043,404966874~7561094] 2024-08-17T17:47:33.024 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37003803,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,173661827~3057406,177148887~16010685,199192089~10098494,210384164~3396269,214907306~13381678,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339992078~10169526,353675391~14545,357458885~7280173,364955231~37579043,404966874~7561094] 2024-08-17T17:47:33.029 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38172001,83664186~75531926,167772160~3343965,171966464~21193108,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339992078~10169526,352321536~1368400,356515840~8223218,364904448~37629826,404765831~7762137] 2024-08-17T17:47:33.029 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 18 diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,173661827~3057406,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~7280173,364955231~37579043,404966874~7561094] 2024-08-17T17:47:33.029 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,173661827~3057406,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~7280173,364955231~37579043,404966874~7561094] 2024-08-17T17:47:33.035 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,83664186~75531926,167772160~3343965,171966464~23139102,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~8223218,364904448~37629826,404765831~7762137] 2024-08-17T17:47:33.035 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 19 diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:33.035 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:33.041 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404765831~7762137] 2024-08-17T17:47:33.041 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 3 diff [134789056~5929129,141162200~123733,170236347~879778,179506671~1438996,266927138~2271880,316072980~3026910,370259166~7251446,392747039~1807197,398184564~2539553] 2024-08-17T17:47:33.041 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [134789056~5929129,141162200~123733,170236347~879778,179506671~1438996,266927138~2271880,316072980~3026910,370259166~7251446,392747039~1807197,398184564~2539553] 2024-08-17T17:47:33.047 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [134789056~6496877,167772160~3343965,179506671~1438996,264241152~4957866,314572800~4527090,369098752~8411860,390070272~4483964,398184564~2539553] 2024-08-17T17:47:33.047 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 4 diff [27213408~7936521,52524078~2996302,132770613~7947572,141162200~123733,170236347~879778,179506671~1438996,184623919~975441,216026965~5893923,266927138~2271880,305250582~7536908,316072980~3026910,370259166~7251446,385127346~12964499,398184564~2539553,405783969~3077668] 2024-08-17T17:47:33.047 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [27213408~7936521,52524078~2996302,132770613~7947572,141162200~123733,170236347~879778,179506671~1438996,184623919~975441,216026965~5893923,266927138~2271880,305250582~7536908,316072980~3026910,370259166~7251446,385127346~12964499,398184564~2539553,405783969~3077668] 2024-08-17T17:47:33.052 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [25165824~9984105,50331648~5188732,132770613~8515320,167772160~3343965,179506671~1438996,184623919~975441,215373523~6547365,264241152~4957866,305250582~7536908,314572800~4527090,369098752~8411860,381681664~16410181,398184564~2539553,405783969~3077668] 2024-08-17T17:47:33.052 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 5 diff [7294659~3604783,14418625~2503052,27213408~7936521,46387139~9133241,88210204~2654548,112459935~7349748,132770613~7947572,141162200~123733,170236347~879778,179506671~1438996,184623919~975441,216026965~5893923,266927138~7887233,305250582~7536908,316072980~3026910,332951667~1703830,344366004~4958418,366512993~16138442,385127346~12964499,398184564~2539553,405783969~3077668] 2024-08-17T17:47:33.052 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14418625~2503052,27213408~7936521,46387139~9133241,88210204~2654548,112459935~7349748,132770613~7947572,141162200~123733,170236347~879778,179506671~1438996,184623919~975441,216026965~5893923,266927138~7887233,305250582~7536908,316072980~3026910,332951667~1703830,344366004~4958418,366512993~16138442,385127346~12964499,398184564~2539553,405783969~3077668] 2024-08-17T17:47:33.058 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [7294659~3604783,14418625~2503052,25165824~9984105,46137344~9383036,88210204~2654548,109051904~10757779,132770613~8515320,167772160~3343965,179506671~1438996,184623919~975441,215373523~6547365,264241152~10573219,305250582~7536908,314572800~4527090,331350016~3305481,343932928~5391494,364904448~33187397,398184564~2539553,405783969~3077668] 2024-08-17T17:47:33.058 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 6 diff [7294659~3604783,14418625~2503052,18397559~2377817,22298619~3398016,27213408~7936521,46387139~9133241,71871766~729179,88210204~2654548,112459935~7349748,132770613~7947572,141162200~123733,143631694~2819484,170236347~879778,177148887~3796780,184623919~975441,203371987~4800717,210384164~3396269,216026965~5893923,229813410~1621406,265447020~9367351,305250582~7536908,316072980~3026910,332951667~1703830,344366004~4958418,366512993~16138442,385127346~12964499,398184564~2539553,405783969~3077668] 2024-08-17T17:47:33.058 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14418625~2503052,18397559~2377817,22298619~3398016,27213408~7936521,46387139~9133241,71871766~729179,88210204~2654548,112459935~7349748,132770613~7947572,141162200~123733,143631694~2819484,170236347~879778,177148887~3796780,184623919~975441,203371987~4800717,210384164~3396269,216026965~5893923,229813410~1621406,265447020~9367351,305250582~7536908,316072980~3026910,332951667~1703830,344366004~4958418,366512993~16138442,385127346~12964499,398184564~2539553,405783969~3077668] 2024-08-17T17:47:33.063 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [7294659~3604783,14418625~2503052,18397559~2377817,20971520~14178409,46137344~9383036,71871766~729179,88210204~2654548,109051904~10757779,132770613~8515320,143631694~2819484,167772160~3343965,177148887~3796780,184623919~975441,203371987~4800717,209715200~4065233,215373523~6547365,226492416~4942400,264241152~10573219,305250582~7536908,314572800~4527090,331350016~3305481,343932928~5391494,364904448~33187397,398184564~2539553,405783969~3077668] 2024-08-17T17:47:33.064 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 7 diff [7294659~3604783,14418625~2503052,18397559~2377817,22298619~3398016,27213408~7936521,46387139~9133241,71871766~729179,72693380~1592089,88210204~2654548,98323116~4579656,112459935~7349748,130078852~10639333,141162200~123733,143631694~2819484,170236347~879778,173661827~3057406,177148887~3796780,184623919~975441,203371987~5918596,210384164~3396269,216026965~5893923,229813410~1621406,231872994~29434,259164751~280375,265447020~9367351,295831353~7859301,305250582~7536908,316072980~4616109,332951667~1703830,344366004~4958418,366512993~16138442,385127346~12964499,398184564~2539553,401465291~1068983,405783969~3077668] 2024-08-17T17:47:33.064 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14418625~2503052,18397559~2377817,22298619~3398016,27213408~7936521,46387139~9133241,71871766~729179,72693380~1592089,88210204~2654548,98323116~4579656,112459935~7349748,130078852~10639333,141162200~123733,143631694~2819484,170236347~879778,173661827~3057406,177148887~3796780,184623919~975441,203371987~5918596,210384164~3396269,216026965~5893923,229813410~1621406,231872994~29434,259164751~280375,265447020~9367351,295831353~7859301,305250582~7536908,316072980~4616109,332951667~1703830,344366004~4958418,366512993~16138442,385127346~12964499,398184564~2539553,401465291~1068983,405783969~3077668] 2024-08-17T17:47:33.069 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [7294659~3604783,14418625~2503052,18397559~2377817,20971520~14178409,46137344~9383036,71871766~729179,72693380~1592089,88210204~2654548,98323116~4579656,109051904~10757779,130078852~11207081,143631694~2819484,167772160~3343965,171966464~4752769,177148887~3796780,184623919~975441,203371987~5918596,209715200~4065233,215373523~6547365,226492416~4942400,231872994~29434,255852544~3592582,264241152~10573219,293601280~10089374,305250582~7536908,314572800~6116289,331350016~3305481,343932928~5391494,364904448~33187397,398184564~2539553,401465291~1068983,405783969~3077668] 2024-08-17T17:47:33.069 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 8 diff [7294659~3604783,14418625~2503052,18397559~2377817,22298619~3398016,27213408~7936521,46126522~9393858,71871766~729179,72693380~1592089,88210204~2654548,96231003~7638279,112459935~7349748,121612481~8169769,130078852~10639333,141162200~123733,143631694~2819484,151863990~3083981,170236347~879778,173661827~3057406,177148887~3796780,182474064~5726314,203371987~5918596,210384164~3396269,216026965~5893923,229813410~2722880,259164751~280375,265447020~9367351,295831353~7859301,305250582~7536908,316072980~4616109,332951667~1703830,344366004~4958418,366512993~16138442,385127346~12964499,398184564~2539553,401465291~1068983,405783969~3077668] 2024-08-17T17:47:33.069 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14418625~2503052,18397559~2377817,22298619~3398016,27213408~7936521,46126522~9393858,71871766~729179,72693380~1592089,88210204~2654548,96231003~7638279,112459935~7349748,121612481~8169769,130078852~10639333,141162200~123733,143631694~2819484,151863990~3083981,170236347~879778,173661827~3057406,177148887~3796780,182474064~5726314,203371987~5918596,210384164~3396269,216026965~5893923,229813410~2722880,259164751~280375,265447020~9367351,295831353~7859301,305250582~7536908,316072980~4616109,332951667~1703830,344366004~4958418,366512993~16138442,385127346~12964499,398184564~2539553,401465291~1068983,405783969~3077668] 2024-08-17T17:47:33.075 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [7294659~3604783,14418625~2503052,18397559~2377817,20971520~14178409,41943040~13577340,71871766~729179,72693380~1592089,88210204~2654548,92420837~11448445,109051904~20730346,130078852~11207081,143631694~2819484,150994944~3953027,167772160~3343965,171966464~4752769,177148887~3796780,182474064~5726314,203371987~5918596,209715200~4065233,215373523~6547365,226492416~6043874,255852544~3592582,264241152~10573219,293601280~10089374,305250582~7536908,314572800~6116289,331350016~3305481,343932928~5391494,364904448~33187397,398184564~2539553,401465291~1068983,405783969~3077668] 2024-08-17T17:47:33.075 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 9 diff [7294659~3604783,14137310~7336653,22298619~3398016,27213408~7936521,46126522~9393858,71871766~729179,72693380~1592089,88210204~2654548,96231003~7638279,112459935~7349748,121612481~8169769,130078852~10639333,141162200~123733,143631694~2819484,151863990~3892627,170236347~879778,173661827~3057406,177148887~3796780,182474064~5726314,199192089~1285316,203371987~5918596,210384164~3396269,214907306~7013582,229813410~2722880,235471051~6158445,259164751~280375,265447020~14406806,295831353~7859301,305250582~7536908,316072980~4616109,332951667~1703830,342231213~7093209,366512993~16138442,385127346~12964499,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.075 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14137310~7336653,22298619~3398016,27213408~7936521,46126522~9393858,71871766~729179,72693380~1592089,88210204~2654548,96231003~7638279,112459935~7349748,121612481~8169769,130078852~10639333,141162200~123733,143631694~2819484,151863990~3892627,170236347~879778,173661827~3057406,177148887~3796780,182474064~5726314,199192089~1285316,203371987~5918596,210384164~3396269,214907306~7013582,229813410~2722880,235471051~6158445,259164751~280375,265447020~14406806,295831353~7859301,305250582~7536908,316072980~4616109,332951667~1703830,342231213~7093209,366512993~16138442,385127346~12964499,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.080 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [7294659~3604783,14137310~21012619,41943040~13577340,71871766~729179,72693380~1592089,88210204~2654548,92420837~11448445,109051904~20730346,130078852~11207081,143631694~2819484,150994944~4761673,167772160~3343965,171966464~4752769,177148887~3796780,182474064~5726314,197132288~3345117,203371987~5918596,209715200~4065233,214907306~7013582,226492416~6043874,235471051~6158445,255852544~3592582,264241152~15612674,293601280~10089374,305250582~7536908,314572800~6116289,331350016~3305481,342231213~7093209,364904448~33187397,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.080 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 10 diff [7294659~3604783,14137310~7336653,22298619~3398016,27213408~7936521,46126522~9393858,71871766~729179,72693380~1592089,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~8169769,130078852~10639333,141162200~123733,143631694~2819484,151863990~7332122,170236347~879778,173661827~3057406,177148887~3796780,182474064~5726314,199192089~1285316,200751176~8539407,210384164~3396269,214907306~7013582,229813410~2722880,235471051~6158445,257231876~3214859,265447020~14406806,284082763~2045905,290745721~4774032,295831353~16956137,316072980~4616109,332951667~1703830,342231213~7093209,366512993~16138442,385127346~12964499,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.081 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14137310~7336653,22298619~3398016,27213408~7936521,46126522~9393858,71871766~729179,72693380~1592089,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~8169769,130078852~10639333,141162200~123733,143631694~2819484,151863990~7332122,170236347~879778,173661827~3057406,177148887~3796780,182474064~5726314,199192089~1285316,200751176~8539407,210384164~3396269,214907306~7013582,229813410~2722880,235471051~6158445,257231876~3214859,265447020~14406806,284082763~2045905,290745721~4774032,295831353~16956137,316072980~4616109,332951667~1703830,342231213~7093209,366512993~16138442,385127346~12964499,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.086 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [7294659~3604783,14137310~21012619,41943040~13577340,71871766~729179,72693380~1592089,88210204~2654548,92420837~37361413,130078852~11207081,143631694~2819484,150994944~8201168,167772160~3343965,171966464~4752769,177148887~3796780,182474064~5726314,197132288~12158295,209715200~4065233,214907306~7013582,226492416~6043874,235471051~6158445,255852544~4594191,264241152~15612674,281018368~5110300,289406976~23380514,314572800~6116289,331350016~3305481,342231213~7093209,364904448~33187397,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.086 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 11 diff [7294659~3604783,14137310~7336653,22298619~3398016,27213408~7936521,43810632~11709748,56731919~6095770,64609489~4058806,71871766~729179,72693380~1592089,76085135~2537370,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,151863990~7332122,170236347~879778,173661827~3057406,177148887~3796780,182474064~5726314,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229813410~2722880,235471051~6158445,257231876~3214859,265447020~14406806,284082763~2045905,290745721~4774032,295831353~16956137,316072980~4616109,332951667~1703830,342231213~7093209,366512993~31578852,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.087 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14137310~7336653,22298619~3398016,27213408~7936521,43810632~11709748,56731919~6095770,64609489~4058806,71871766~729179,72693380~1592089,76085135~2537370,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,151863990~7332122,170236347~879778,173661827~3057406,177148887~3796780,182474064~5726314,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229813410~2722880,235471051~6158445,257231876~3214859,265447020~14406806,284082763~2045905,290745721~4774032,295831353~16956137,316072980~4616109,332951667~1703830,342231213~7093209,366512993~31578852,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.092 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [7294659~3604783,14137310~21012619,41943040~13577340,56731919~6095770,64609489~4058806,71871766~729179,72693380~1592089,75497472~3125033,88210204~2654548,92420837~48865096,143631694~2819484,150994944~8201168,167772160~3343965,171966464~4752769,177148887~3796780,182474064~5726314,197132288~12158295,209715200~4065233,214907306~10694908,226492416~6043874,235471051~6158445,255852544~4594191,264241152~15612674,281018368~5110300,289406976~23380514,314572800~6116289,331350016~3305481,342231213~7093209,364904448~33187397,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.092 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 12 diff [3528496~7370946,14137310~7336653,22298619~3398016,27213408~7936521,43810632~11709748,56731919~6095770,64609489~4058806,71871766~729179,72693380~5929125,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146925241~12270871,170236347~879778,173661827~3057406,177148887~3796780,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229813410~2722880,235471051~6158445,257231876~3214859,265447020~20681648,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,342231213~7093209,357458885~7280173,366512993~31578852,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.093 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [3528496~7370946,14137310~7336653,22298619~3398016,27213408~7936521,43810632~11709748,56731919~6095770,64609489~4058806,71871766~729179,72693380~5929125,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146925241~12270871,170236347~879778,173661827~3057406,177148887~3796780,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229813410~2722880,235471051~6158445,257231876~3214859,265447020~20681648,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,342231213~7093209,357458885~7280173,366512993~31578852,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.098 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~10899442,14137310~21012619,41943040~13577340,56731919~6095770,64609489~4058806,71871766~729179,72693380~5929125,88210204~2654548,92420837~48865096,143631694~2819484,146925241~12270871,167772160~3343965,171966464~4752769,177148887~3796780,182474064~10685508,197132288~12158295,209715200~4065233,214907306~10694908,226492416~6043874,235471051~6158445,255852544~4594191,264241152~21887516,289406976~23380514,314572800~6116289,327775763~6879734,342231213~7093209,356515840~8223218,364904448~33187397,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.098 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 13 diff [3528496~7370946,14137310~7336653,22298619~3398016,27213408~7936521,43111238~12409142,56731919~6095770,64609489~4058806,71871766~729179,72693380~5929125,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146925241~12270871,170236347~879778,173661827~3057406,177148887~3796780,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,257231876~3214859,265447020~21856450,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,342231213~7093209,357458885~7280173,366512993~31578852,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.098 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [3528496~7370946,14137310~7336653,22298619~3398016,27213408~7936521,43111238~12409142,56731919~6095770,64609489~4058806,71871766~729179,72693380~5929125,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146925241~12270871,170236347~879778,173661827~3057406,177148887~3796780,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,257231876~3214859,265447020~21856450,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,342231213~7093209,357458885~7280173,366512993~31578852,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.104 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~10899442,14137310~21012619,41943040~13577340,56731919~6095770,64609489~4058806,71871766~729179,72693380~5929125,88210204~2654548,92420837~48865096,143631694~2819484,146925241~12270871,167772160~3343965,171966464~4752769,177148887~3796780,182474064~10685508,197132288~12158295,209715200~4065233,214907306~10694908,226492416~15137080,255852544~4594191,264241152~23062318,289406976~23380514,314572800~6116289,327775763~6879734,342231213~7093209,356515840~8223218,364904448~33187397,398184564~2539553,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.104 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 14 diff [1598369~392670,3528496~7370946,14137310~7336653,22298619~3398016,27213408~7936521,43111238~12409142,56731919~21890586,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146925241~12270871,170236347~879778,173661827~3057406,177148887~3796780,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,249341666~4308983,257231876~3214859,265447020~21856450,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339992078~9332344,357458885~7280173,366512993~36021281,404966874~7561094] 2024-08-17T17:47:33.105 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~7370946,14137310~7336653,22298619~3398016,27213408~7936521,43111238~12409142,56731919~21890586,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146925241~12270871,170236347~879778,173661827~3057406,177148887~3796780,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,249341666~4308983,257231876~3214859,265447020~21856450,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339992078~9332344,357458885~7280173,366512993~36021281,404966874~7561094] 2024-08-17T17:47:33.110 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~10899442,14137310~21012619,41943040~13577340,56731919~21890586,88210204~2654548,92420837~48865096,143631694~2819484,146925241~12270871,167772160~3343965,171966464~4752769,177148887~3796780,182474064~10685508,197132288~12158295,209715200~4065233,214907306~10694908,226492416~15137080,249341666~4308983,255852544~4594191,264241152~23062318,289406976~23380514,314572800~6116289,327775763~6879734,339992078~9332344,356515840~8223218,364904448~37629826,404966874~7561094] 2024-08-17T17:47:33.110 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 15 diff [1598369~392670,3528496~7370946,14137310~7336653,22298619~3398016,27213408~11717112,43111238~35511267,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146925241~12270871,170236347~879778,173661827~3057406,177148887~3796780,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,249341666~11105069,265447020~21856450,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339992078~9332344,357458885~7280173,366512993~36021281,404966874~7561094] 2024-08-17T17:47:33.110 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~7370946,14137310~7336653,22298619~3398016,27213408~11717112,43111238~35511267,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~19105704,141162200~123733,143631694~2819484,146925241~12270871,170236347~879778,173661827~3057406,177148887~3796780,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,249341666~11105069,265447020~21856450,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339992078~9332344,357458885~7280173,366512993~36021281,404966874~7561094] 2024-08-17T17:47:33.116 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~10899442,14137310~24793210,41943040~36679465,88210204~2654548,92420837~48865096,143631694~2819484,146925241~12270871,167772160~3343965,171966464~4752769,177148887~3796780,182474064~10685508,197132288~12158295,209715200~4065233,214907306~10694908,226492416~15137080,249341666~11105069,264241152~23062318,289406976~23380514,314572800~6116289,327775763~6879734,339992078~9332344,356515840~8223218,364904448~37629826,404966874~7561094] 2024-08-17T17:47:33.116 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 16 diff [1598369~392670,3528496~8378554,13709905~7764058,22298619~3398016,27213408~11717112,43111238~35511267,83664186~7312323,96231003~7638279,104378282~4723101,112459935~7349748,121612481~37583631,170236347~879778,173661827~3057406,177148887~3796780,182474064~10685508,199192089~10098494,210384164~3396269,214907306~10694908,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339992078~9332344,357458885~7280173,364955231~37579043,404966874~7561094] 2024-08-17T17:47:33.116 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7764058,22298619~3398016,27213408~11717112,43111238~35511267,83664186~7312323,96231003~7638279,104378282~4723101,112459935~7349748,121612481~37583631,170236347~879778,173661827~3057406,177148887~3796780,182474064~10685508,199192089~10098494,210384164~3396269,214907306~10694908,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339992078~9332344,357458885~7280173,364955231~37579043,404966874~7561094] 2024-08-17T17:47:33.122 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~36679465,83664186~7312323,92420837~66775275,167772160~3343965,171966464~4752769,177148887~3796780,182474064~10685508,197132288~12158295,209715200~4065233,214907306~10694908,226492416~15137080,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339992078~9332344,356515840~8223218,364904448~37629826,404966874~7561094] 2024-08-17T17:47:33.122 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 17 diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37003803,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,173661827~3057406,177148887~16010685,199192089~10098494,210384164~3396269,214907306~13381678,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339992078~10169526,353675391~14545,357458885~7280173,364955231~37579043,404966874~7561094] 2024-08-17T17:47:33.122 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37003803,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,173661827~3057406,177148887~16010685,199192089~10098494,210384164~3396269,214907306~13381678,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339992078~10169526,353675391~14545,357458885~7280173,364955231~37579043,404966874~7561094] 2024-08-17T17:47:33.127 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38172001,83664186~7312323,92420837~66775275,167772160~3343965,171966464~4752769,177148887~16010685,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339992078~10169526,352321536~1368400,356515840~8223218,364904448~37629826,404966874~7561094] 2024-08-17T17:47:33.127 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 18 diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,173661827~3057406,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~7280173,364955231~37579043,404966874~7561094] 2024-08-17T17:47:33.128 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,173661827~3057406,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~7280173,364955231~37579043,404966874~7561094] 2024-08-17T17:47:33.133 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,83664186~7312323,92420837~66775275,167772160~3343965,171966464~4752769,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~8223218,364904448~37629826,404966874~7561094] 2024-08-17T17:47:33.134 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 19 diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:33.134 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:33.139 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,167772160~8947073,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:33.139 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 4 diff [27213408~7936521,52524078~2996302,132770613~4547368,184623919~975441,216026965~5893923,305250582~7536908,385127346~8726798,393974446~4117399,405783969~3077668] 2024-08-17T17:47:33.139 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [27213408~7936521,52524078~2996302,132770613~4547368,184623919~975441,216026965~5893923,305250582~7536908,385127346~8726798,393974446~4117399,405783969~3077668] 2024-08-17T17:47:33.144 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [25165824~9984105,50331648~5188732,132770613~4547368,184623919~975441,215373523~6547365,305250582~7536908,381681664~12172480,393974446~4117399,405783969~3077668] 2024-08-17T17:47:33.145 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 5 diff [7294659~3604783,14418625~2503052,27213408~7936521,46387139~9133241,88210204~2654548,112459935~7349748,132770613~4547368,184623919~975441,216026965~5893923,267519243~7295128,305250582~7536908,332951667~1703830,344366004~4958418,366512993~8145527,376903755~5747680,385127346~8726798,393974446~4117399,405783969~3077668] 2024-08-17T17:47:33.145 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14418625~2503052,27213408~7936521,46387139~9133241,88210204~2654548,112459935~7349748,132770613~4547368,184623919~975441,216026965~5893923,267519243~7295128,305250582~7536908,332951667~1703830,344366004~4958418,366512993~8145527,376903755~5747680,385127346~8726798,393974446~4117399,405783969~3077668] 2024-08-17T17:47:33.150 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [7294659~3604783,14418625~2503052,25165824~9984105,46137344~9383036,88210204~2654548,109051904~10757779,132770613~4547368,184623919~975441,215373523~6547365,267519243~7295128,305250582~7536908,331350016~3305481,343932928~5391494,364904448~9754072,376903755~16950389,393974446~4117399,405783969~3077668] 2024-08-17T17:47:33.150 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 6 diff [7294659~3604783,14418625~2503052,18397559~2377817,22298619~3398016,27213408~7936521,46387139~9133241,71871766~729179,88210204~2654548,112459935~7349748,132770613~4547368,143631694~2819484,177148887~3686146,184623919~975441,203371987~4800717,210384164~3396269,216026965~5893923,229813410~1621406,265447020~9367351,305250582~7536908,332951667~1703830,344366004~4958418,366512993~8145527,375036141~1226037,376903755~5747680,385127346~8726798,393974446~4117399,405783969~3077668] 2024-08-17T17:47:33.150 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14418625~2503052,18397559~2377817,22298619~3398016,27213408~7936521,46387139~9133241,71871766~729179,88210204~2654548,112459935~7349748,132770613~4547368,143631694~2819484,177148887~3686146,184623919~975441,203371987~4800717,210384164~3396269,216026965~5893923,229813410~1621406,265447020~9367351,305250582~7536908,332951667~1703830,344366004~4958418,366512993~8145527,375036141~1226037,376903755~5747680,385127346~8726798,393974446~4117399,405783969~3077668] 2024-08-17T17:47:33.156 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [7294659~3604783,14418625~2503052,18397559~2377817,20971520~14178409,46137344~9383036,71871766~729179,88210204~2654548,109051904~10757779,132770613~4547368,143631694~2819484,177148887~3686146,184623919~975441,203371987~4800717,209715200~4065233,215373523~6547365,226492416~4942400,265447020~9367351,305250582~7536908,331350016~3305481,343932928~5391494,364904448~9754072,375036141~1226037,376903755~16950389,393974446~4117399,405783969~3077668] 2024-08-17T17:47:33.156 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 7 diff [7294659~3604783,14418625~2503052,18397559~2377817,22298619~3398016,27213408~7936521,46387139~9133241,71871766~729179,72693380~1592089,88210204~2654548,98323116~4579656,112459935~7349748,130078852~7239129,143631694~2819484,173661827~3057406,177148887~3686146,184623919~975441,203371987~5918596,210384164~3396269,216026965~5893923,229813410~1621406,231872994~29434,259164751~280375,265447020~9367351,295831353~7859301,305250582~7536908,317833139~2855950,332951667~1703830,344366004~4958418,366512993~8145527,375036141~1226037,376903755~5747680,385127346~8726798,393974446~4117399,401465291~1068983,405783969~3077668] 2024-08-17T17:47:33.156 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14418625~2503052,18397559~2377817,22298619~3398016,27213408~7936521,46387139~9133241,71871766~729179,72693380~1592089,88210204~2654548,98323116~4579656,112459935~7349748,130078852~7239129,143631694~2819484,173661827~3057406,177148887~3686146,184623919~975441,203371987~5918596,210384164~3396269,216026965~5893923,229813410~1621406,231872994~29434,259164751~280375,265447020~9367351,295831353~7859301,305250582~7536908,317833139~2855950,332951667~1703830,344366004~4958418,366512993~8145527,375036141~1226037,376903755~5747680,385127346~8726798,393974446~4117399,401465291~1068983,405783969~3077668] 2024-08-17T17:47:33.161 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [7294659~3604783,14418625~2503052,18397559~2377817,20971520~14178409,46137344~9383036,71871766~729179,72693380~1592089,88210204~2654548,98323116~4579656,109051904~10757779,130078852~7239129,143631694~2819484,171966464~4752769,177148887~3686146,184623919~975441,203371987~5918596,209715200~4065233,215373523~6547365,226492416~4942400,231872994~29434,255852544~3592582,265447020~9367351,293601280~10089374,305250582~7536908,317833139~2855950,331350016~3305481,343932928~5391494,364904448~9754072,375036141~1226037,376903755~16950389,393974446~4117399,401465291~1068983,405783969~3077668] 2024-08-17T17:47:33.161 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 8 diff [7294659~3604783,14418625~2503052,18397559~2377817,22298619~3398016,27213408~7936521,46126522~9393858,71871766~729179,72693380~1592089,88210204~2654548,96231003~7638279,112459935~7349748,121612481~8169769,130078852~7239129,143631694~2819484,151863990~3083981,173661827~3057406,177148887~3686146,182474064~5726314,203371987~5918596,210384164~3396269,216026965~5893923,229813410~2722880,259164751~280375,265447020~9367351,295831353~7859301,305250582~7536908,316577159~4111930,332951667~1703830,344366004~4958418,366512993~8145527,375036141~1226037,376903755~5747680,385127346~12964499,401465291~1068983,405783969~3077668] 2024-08-17T17:47:33.161 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14418625~2503052,18397559~2377817,22298619~3398016,27213408~7936521,46126522~9393858,71871766~729179,72693380~1592089,88210204~2654548,96231003~7638279,112459935~7349748,121612481~8169769,130078852~7239129,143631694~2819484,151863990~3083981,173661827~3057406,177148887~3686146,182474064~5726314,203371987~5918596,210384164~3396269,216026965~5893923,229813410~2722880,259164751~280375,265447020~9367351,295831353~7859301,305250582~7536908,316577159~4111930,332951667~1703830,344366004~4958418,366512993~8145527,375036141~1226037,376903755~5747680,385127346~12964499,401465291~1068983,405783969~3077668] 2024-08-17T17:47:33.167 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [7294659~3604783,14418625~2503052,18397559~2377817,20971520~14178409,41943040~13577340,71871766~729179,72693380~1592089,88210204~2654548,92420837~11448445,109051904~20730346,130078852~7239129,143631694~2819484,150994944~3953027,171966464~4752769,177148887~3686146,182474064~5726314,203371987~5918596,209715200~4065233,215373523~6547365,226492416~6043874,255852544~3592582,265447020~9367351,293601280~10089374,305250582~7536908,316577159~4111930,331350016~3305481,343932928~5391494,364904448~9754072,375036141~1226037,376903755~21188090,401465291~1068983,405783969~3077668] 2024-08-17T17:47:33.168 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 9 diff [7294659~3604783,14137310~7336653,22298619~3398016,27213408~7936521,46126522~9393858,71871766~729179,72693380~1592089,88210204~2654548,96231003~7638279,112459935~7349748,121612481~8169769,130078852~7239129,143631694~2819484,151863990~3892627,173661827~3057406,177148887~3686146,182474064~5726314,199192089~1285316,203371987~5918596,210384164~3396269,214907306~7013582,229813410~2722880,235471051~6158445,259164751~280375,265447020~14406806,295831353~7859301,305250582~7536908,316577159~4111930,332951667~1703830,342231213~7093209,366512993~8145527,375036141~1226037,376903755~5747680,385127346~12964499,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.168 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14137310~7336653,22298619~3398016,27213408~7936521,46126522~9393858,71871766~729179,72693380~1592089,88210204~2654548,96231003~7638279,112459935~7349748,121612481~8169769,130078852~7239129,143631694~2819484,151863990~3892627,173661827~3057406,177148887~3686146,182474064~5726314,199192089~1285316,203371987~5918596,210384164~3396269,214907306~7013582,229813410~2722880,235471051~6158445,259164751~280375,265447020~14406806,295831353~7859301,305250582~7536908,316577159~4111930,332951667~1703830,342231213~7093209,366512993~8145527,375036141~1226037,376903755~5747680,385127346~12964499,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.173 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [7294659~3604783,14137310~21012619,41943040~13577340,71871766~729179,72693380~1592089,88210204~2654548,92420837~11448445,109051904~20730346,130078852~7239129,143631694~2819484,150994944~4761673,171966464~4752769,177148887~3686146,182474064~5726314,197132288~3345117,203371987~5918596,209715200~4065233,214907306~7013582,226492416~6043874,235471051~6158445,255852544~3592582,265447020~14406806,293601280~10089374,305250582~7536908,316577159~4111930,331350016~3305481,342231213~7093209,364904448~9754072,375036141~1226037,376903755~21188090,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.173 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 10 diff [7294659~3604783,14137310~7336653,22298619~3398016,27213408~7936521,46126522~9393858,71871766~729179,72693380~1592089,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~8169769,130078852~9930308,143631694~2819484,151863990~7332122,173661827~3057406,177148887~3686146,182474064~5726314,199192089~1285316,200751176~8539407,210384164~3396269,214907306~7013582,229813410~2722880,235471051~6158445,257231876~3214859,265447020~14406806,284082763~2045905,290745721~4774032,295831353~16956137,316577159~4111930,332951667~1703830,342231213~7093209,366512993~8145527,375036141~1226037,376903755~5747680,385127346~12964499,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.173 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14137310~7336653,22298619~3398016,27213408~7936521,46126522~9393858,71871766~729179,72693380~1592089,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~8169769,130078852~9930308,143631694~2819484,151863990~7332122,173661827~3057406,177148887~3686146,182474064~5726314,199192089~1285316,200751176~8539407,210384164~3396269,214907306~7013582,229813410~2722880,235471051~6158445,257231876~3214859,265447020~14406806,284082763~2045905,290745721~4774032,295831353~16956137,316577159~4111930,332951667~1703830,342231213~7093209,366512993~8145527,375036141~1226037,376903755~5747680,385127346~12964499,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.179 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [7294659~3604783,14137310~21012619,41943040~13577340,71871766~729179,72693380~1592089,88210204~2654548,92420837~37361413,130078852~9930308,143631694~2819484,150994944~8201168,171966464~4752769,177148887~3686146,182474064~5726314,197132288~12158295,209715200~4065233,214907306~7013582,226492416~6043874,235471051~6158445,255852544~4594191,265447020~14406806,281018368~5110300,289406976~23380514,316577159~4111930,331350016~3305481,342231213~7093209,364904448~9754072,375036141~1226037,376903755~21188090,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.179 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 11 diff [7294659~3604783,14137310~7336653,22298619~3398016,27213408~7936521,43810632~11709748,56731919~6095770,64609489~4058806,71871766~729179,72693380~1592089,76085135~2537370,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~18396679,143631694~2819484,151863990~7332122,173661827~3057406,177148887~3686146,182474064~5726314,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229813410~2722880,235471051~6158445,257231876~3214859,265447020~14406806,284082763~2045905,290745721~4774032,295831353~16956137,316577159~4111930,332951667~1703830,342231213~7093209,366512993~8145527,375036141~1226037,376903755~21188090,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.179 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14137310~7336653,22298619~3398016,27213408~7936521,43810632~11709748,56731919~6095770,64609489~4058806,71871766~729179,72693380~1592089,76085135~2537370,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~18396679,143631694~2819484,151863990~7332122,173661827~3057406,177148887~3686146,182474064~5726314,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229813410~2722880,235471051~6158445,257231876~3214859,265447020~14406806,284082763~2045905,290745721~4774032,295831353~16956137,316577159~4111930,332951667~1703830,342231213~7093209,366512993~8145527,375036141~1226037,376903755~21188090,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.185 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [7294659~3604783,14137310~21012619,41943040~13577340,56731919~6095770,64609489~4058806,71871766~729179,72693380~1592089,75497472~3125033,88210204~2654548,92420837~47588323,143631694~2819484,150994944~8201168,171966464~4752769,177148887~3686146,182474064~5726314,197132288~12158295,209715200~4065233,214907306~10694908,226492416~6043874,235471051~6158445,255852544~4594191,265447020~14406806,281018368~5110300,289406976~23380514,316577159~4111930,331350016~3305481,342231213~7093209,364904448~9754072,375036141~1226037,376903755~21188090,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.185 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 12 diff [3528496~7370946,14137310~7336653,22298619~3398016,27213408~7936521,43810632~11709748,56731919~6095770,64609489~4058806,71871766~729179,72693380~5929125,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~18396679,143631694~2819484,146925241~12270871,173661827~3057406,177148887~3686146,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229813410~2722880,235471051~6158445,257231876~3214859,265447020~20681648,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,342231213~7093209,357458885~7280173,366512993~8145527,375036141~1226037,376903755~21188090,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.185 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [3528496~7370946,14137310~7336653,22298619~3398016,27213408~7936521,43810632~11709748,56731919~6095770,64609489~4058806,71871766~729179,72693380~5929125,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~18396679,143631694~2819484,146925241~12270871,173661827~3057406,177148887~3686146,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229813410~2722880,235471051~6158445,257231876~3214859,265447020~20681648,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,342231213~7093209,357458885~7280173,366512993~8145527,375036141~1226037,376903755~21188090,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.190 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~10899442,14137310~21012619,41943040~13577340,56731919~6095770,64609489~4058806,71871766~729179,72693380~5929125,88210204~2654548,92420837~47588323,143631694~2819484,146925241~12270871,171966464~4752769,177148887~3686146,182474064~10685508,197132288~12158295,209715200~4065233,214907306~10694908,226492416~6043874,235471051~6158445,255852544~4594191,265447020~20681648,289406976~23380514,316577159~4111930,327775763~6879734,342231213~7093209,356515840~8223218,364904448~9754072,375036141~1226037,376903755~21188090,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.191 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 13 diff [3528496~7370946,14137310~7336653,22298619~3398016,27213408~7936521,43111238~12409142,56731919~6095770,64609489~4058806,71871766~729179,72693380~5929125,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~18396679,143631694~2819484,146925241~12270871,173661827~3057406,177148887~3686146,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,257231876~3214859,265447020~21856450,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,342231213~7093209,357458885~7280173,366512993~31578852,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.191 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [3528496~7370946,14137310~7336653,22298619~3398016,27213408~7936521,43111238~12409142,56731919~6095770,64609489~4058806,71871766~729179,72693380~5929125,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~18396679,143631694~2819484,146925241~12270871,173661827~3057406,177148887~3686146,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,257231876~3214859,265447020~21856450,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,342231213~7093209,357458885~7280173,366512993~31578852,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:32 smithi136 ceph-mon[115078]: osdmap e1126: 8 total, 8 up, 8 in 2024-08-17T17:47:33.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:32 smithi136 ceph-mon[115078]: pgmap v1727: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 234 KiB/s rd, 46 MiB/s wr, 364 op/s 2024-08-17T17:47:33.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:32 smithi136 ceph-mon[109861]: osdmap e1126: 8 total, 8 up, 8 in 2024-08-17T17:47:33.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:32 smithi136 ceph-mon[109861]: pgmap v1727: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 234 KiB/s rd, 46 MiB/s wr, 364 op/s 2024-08-17T17:47:33.197 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~10899442,14137310~21012619,41943040~13577340,56731919~6095770,64609489~4058806,71871766~729179,72693380~5929125,88210204~2654548,92420837~47588323,143631694~2819484,146925241~12270871,171966464~4752769,177148887~3686146,182474064~10685508,197132288~12158295,209715200~4065233,214907306~10694908,226492416~15137080,255852544~4594191,265447020~21856450,289406976~23380514,316577159~4111930,327775763~6879734,342231213~7093209,356515840~8223218,364904448~33187397,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.197 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 14 diff [1598369~392670,3528496~7370946,14137310~7336653,22298619~3398016,27213408~7936521,43111238~12409142,56731919~21890586,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~18396679,143631694~2819484,146925241~12270871,173661827~3057406,177148887~3686146,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,249341666~4308983,257231876~3214859,265447020~21856450,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339992078~9332344,357458885~7280173,366512993~36021281,404966874~7561094] 2024-08-17T17:47:33.197 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~7370946,14137310~7336653,22298619~3398016,27213408~7936521,43111238~12409142,56731919~21890586,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~18396679,143631694~2819484,146925241~12270871,173661827~3057406,177148887~3686146,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,249341666~4308983,257231876~3214859,265447020~21856450,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339992078~9332344,357458885~7280173,366512993~36021281,404966874~7561094] 2024-08-17T17:47:33.202 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~10899442,14137310~21012619,41943040~13577340,56731919~21890586,88210204~2654548,92420837~47588323,143631694~2819484,146925241~12270871,171966464~4752769,177148887~3686146,182474064~10685508,197132288~12158295,209715200~4065233,214907306~10694908,226492416~15137080,249341666~4308983,255852544~4594191,265447020~21856450,289406976~23380514,316577159~4111930,327775763~6879734,339992078~9332344,356515840~8223218,364904448~37629826,404966874~7561094] 2024-08-17T17:47:33.202 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 15 diff [1598369~392670,3528496~7370946,14137310~7336653,22298619~3398016,27213408~11717112,43111238~35511267,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~18396679,143631694~2819484,146925241~12270871,173661827~3057406,177148887~3686146,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,249341666~11105069,265447020~21856450,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339992078~9332344,357458885~7280173,366512993~36021281,404966874~7561094] 2024-08-17T17:47:33.202 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~7370946,14137310~7336653,22298619~3398016,27213408~11717112,43111238~35511267,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~18396679,143631694~2819484,146925241~12270871,173661827~3057406,177148887~3686146,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,249341666~11105069,265447020~21856450,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339992078~9332344,357458885~7280173,366512993~36021281,404966874~7561094] 2024-08-17T17:47:33.208 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~10899442,14137310~24793210,41943040~36679465,88210204~2654548,92420837~47588323,143631694~2819484,146925241~12270871,171966464~4752769,177148887~3686146,182474064~10685508,197132288~12158295,209715200~4065233,214907306~10694908,226492416~15137080,249341666~11105069,265447020~21856450,289406976~23380514,316577159~4111930,327775763~6879734,339992078~9332344,356515840~8223218,364904448~37629826,404966874~7561094] 2024-08-17T17:47:33.208 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 16 diff [1598369~392670,3528496~8378554,13709905~7764058,22298619~3398016,27213408~11717112,43111238~35511267,83664186~7312323,96231003~7638279,104378282~4723101,112459935~7349748,121612481~37583631,173661827~3057406,177148887~3686146,182474064~10685508,199192089~10098494,210384164~3396269,214907306~10694908,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339992078~9332344,357458885~7280173,364955231~37579043,404966874~7561094] 2024-08-17T17:47:33.208 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7764058,22298619~3398016,27213408~11717112,43111238~35511267,83664186~7312323,96231003~7638279,104378282~4723101,112459935~7349748,121612481~37583631,173661827~3057406,177148887~3686146,182474064~10685508,199192089~10098494,210384164~3396269,214907306~10694908,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339992078~9332344,357458885~7280173,364955231~37579043,404966874~7561094] 2024-08-17T17:47:33.213 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~36679465,83664186~7312323,92420837~66775275,171966464~4752769,177148887~3686146,182474064~10685508,197132288~12158295,209715200~4065233,214907306~10694908,226492416~15137080,249341666~37961804,289406976~23380514,316577159~4111930,327775763~6879734,339992078~9332344,356515840~8223218,364904448~37629826,404966874~7561094] 2024-08-17T17:47:33.213 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 17 diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37003803,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,173661827~3057406,177148887~16010685,199192089~10098494,210384164~3396269,214907306~13381678,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339992078~10169526,353675391~14545,357458885~7280173,364955231~37579043,404966874~7561094] 2024-08-17T17:47:33.213 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37003803,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,173661827~3057406,177148887~16010685,199192089~10098494,210384164~3396269,214907306~13381678,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339992078~10169526,353675391~14545,357458885~7280173,364955231~37579043,404966874~7561094] 2024-08-17T17:47:33.219 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38172001,83664186~7312323,92420837~66775275,168323939~2295785,171966464~4752769,177148887~16010685,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,316577159~4111930,327775763~6879734,339992078~10169526,352321536~1368400,356515840~8223218,364904448~37629826,404966874~7561094] 2024-08-17T17:47:33.219 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 18 diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,173661827~3057406,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~7280173,364955231~37579043,404966874~7561094] 2024-08-17T17:47:33.219 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,173661827~3057406,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~7280173,364955231~37579043,404966874~7561094] 2024-08-17T17:47:33.225 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,83664186~7312323,92420837~66775275,168323939~2295785,171966464~4752769,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~8223218,364904448~37629826,404966874~7561094] 2024-08-17T17:47:33.225 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 19 diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:33.225 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:33.230 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:33.231 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 5 diff [7294659~3604783,14418625~2503052,46387139~6646744,88210204~2654548,112459935~7349748,267519243~7295128,332951667~1703830,344366004~4958418,366512993~8145527,376903755~5747680] 2024-08-17T17:47:33.231 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14418625~2503052,46387139~6646744,88210204~2654548,112459935~7349748,267519243~7295128,332951667~1703830,344366004~4958418,366512993~8145527,376903755~5747680] 2024-08-17T17:47:33.236 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [7294659~3604783,14418625~2503052,46137344~6896539,88210204~2654548,109051904~10757779,267519243~7295128,331350016~3305481,343932928~5391494,364904448~9754072,376903755~5747680] 2024-08-17T17:47:33.236 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 6 diff [7294659~3604783,14418625~2503052,18397559~2377817,22298619~3398016,46387139~6646744,71871766~729179,88210204~2654548,112459935~7349748,143631694~2819484,177148887~3686146,203371987~4800717,210384164~3396269,229813410~1621406,265447020~9367351,332951667~1703830,344366004~4958418,366512993~8145527,375036141~1226037,376903755~5747680] 2024-08-17T17:47:33.236 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14418625~2503052,18397559~2377817,22298619~3398016,46387139~6646744,71871766~729179,88210204~2654548,112459935~7349748,143631694~2819484,177148887~3686146,203371987~4800717,210384164~3396269,229813410~1621406,265447020~9367351,332951667~1703830,344366004~4958418,366512993~8145527,375036141~1226037,376903755~5747680] 2024-08-17T17:47:33.242 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [7294659~3604783,14418625~2503052,18397559~2377817,20971520~4725115,46137344~6896539,71871766~729179,88210204~2654548,109051904~10757779,143631694~2819484,177148887~3686146,203371987~4800717,209715200~4065233,226492416~4942400,265447020~9367351,331350016~3305481,343932928~5391494,364904448~9754072,375036141~1226037,376903755~5747680] 2024-08-17T17:47:33.242 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 7 diff [7294659~3604783,14418625~2503052,18397559~2377817,22298619~3398016,46387139~6646744,71871766~729179,72693380~1592089,88210204~2654548,98323116~4579656,112459935~7349748,130078852~4344656,143631694~2819484,173661827~3057406,177148887~3686146,203371987~5918596,210384164~3396269,229813410~1621406,231872994~29434,259164751~280375,265447020~9367351,295831353~7859301,317833139~2855950,332951667~1703830,344366004~4958418,366512993~8145527,375036141~1226037,376903755~5747680,401465291~1068983] 2024-08-17T17:47:33.242 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14418625~2503052,18397559~2377817,22298619~3398016,46387139~6646744,71871766~729179,72693380~1592089,88210204~2654548,98323116~4579656,112459935~7349748,130078852~4344656,143631694~2819484,173661827~3057406,177148887~3686146,203371987~5918596,210384164~3396269,229813410~1621406,231872994~29434,259164751~280375,265447020~9367351,295831353~7859301,317833139~2855950,332951667~1703830,344366004~4958418,366512993~8145527,375036141~1226037,376903755~5747680,401465291~1068983] 2024-08-17T17:47:33.252 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [7294659~3604783,14418625~2503052,18397559~2377817,20971520~4725115,46137344~6896539,71871766~729179,72693380~1592089,88210204~2654548,98323116~4579656,109051904~10757779,130078852~4344656,143631694~2819484,171966464~4752769,177148887~3686146,203371987~5918596,209715200~4065233,226492416~4942400,231872994~29434,255852544~3592582,265447020~9367351,293601280~10089374,317833139~2855950,331350016~3305481,343932928~5391494,364904448~9754072,375036141~1226037,376903755~5747680,401465291~1068983] 2024-08-17T17:47:33.252 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 8 diff [7294659~3604783,14418625~2503052,18397559~2377817,22298619~3398016,46126522~6907361,71871766~729179,72693380~1592089,88210204~2654548,96231003~7638279,112459935~7349748,121612481~8169769,130078852~4344656,143631694~2819484,151863990~3083981,173661827~3057406,177148887~3686146,182474064~5726314,203371987~5918596,210384164~3396269,229813410~2722880,259164751~280375,265447020~9367351,295831353~7859301,316577159~4111930,332951667~1703830,344366004~4958418,366512993~8145527,375036141~1226037,376903755~5747680,391986145~4334286,401465291~1068983] 2024-08-17T17:47:33.252 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14418625~2503052,18397559~2377817,22298619~3398016,46126522~6907361,71871766~729179,72693380~1592089,88210204~2654548,96231003~7638279,112459935~7349748,121612481~8169769,130078852~4344656,143631694~2819484,151863990~3083981,173661827~3057406,177148887~3686146,182474064~5726314,203371987~5918596,210384164~3396269,229813410~2722880,259164751~280375,265447020~9367351,295831353~7859301,316577159~4111930,332951667~1703830,344366004~4958418,366512993~8145527,375036141~1226037,376903755~5747680,391986145~4334286,401465291~1068983] 2024-08-17T17:47:33.258 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [7294659~3604783,14418625~2503052,18397559~2377817,20971520~4725115,41943040~11090843,71871766~729179,72693380~1592089,88210204~2654548,92420837~11448445,109051904~20730346,130078852~4344656,143631694~2819484,150994944~3953027,171966464~4752769,177148887~3686146,182474064~5726314,203371987~5918596,209715200~4065233,226492416~6043874,255852544~3592582,265447020~9367351,293601280~10089374,316577159~4111930,331350016~3305481,343932928~5391494,364904448~9754072,375036141~1226037,376903755~5747680,391986145~4334286,401465291~1068983] 2024-08-17T17:47:33.259 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 9 diff [7294659~3604783,14137310~7336653,22298619~3398016,46126522~8645104,71871766~729179,72693380~1592089,88210204~2654548,96231003~7638279,112459935~7349748,121612481~8169769,130078852~4344656,143631694~2819484,151863990~3892627,173661827~3057406,177148887~3686146,182474064~5726314,199192089~1285316,203371987~5918596,210384164~3396269,214907306~4822959,229813410~2722880,235471051~6158445,259164751~280375,265447020~14406806,295831353~7859301,316577159~4111930,332951667~1703830,342231213~7093209,366512993~8145527,375036141~1226037,376903755~5747680,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.259 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14137310~7336653,22298619~3398016,46126522~8645104,71871766~729179,72693380~1592089,88210204~2654548,96231003~7638279,112459935~7349748,121612481~8169769,130078852~4344656,143631694~2819484,151863990~3892627,173661827~3057406,177148887~3686146,182474064~5726314,199192089~1285316,203371987~5918596,210384164~3396269,214907306~4822959,229813410~2722880,235471051~6158445,259164751~280375,265447020~14406806,295831353~7859301,316577159~4111930,332951667~1703830,342231213~7093209,366512993~8145527,375036141~1226037,376903755~5747680,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:32 smithi165 ceph-mon[105517]: osdmap e1126: 8 total, 8 up, 8 in 2024-08-17T17:47:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:32 smithi165 ceph-mon[105517]: pgmap v1727: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 234 KiB/s rd, 46 MiB/s wr, 364 op/s 2024-08-17T17:47:33.264 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [7294659~3604783,14137310~11559325,41943040~12828586,71871766~729179,72693380~1592089,88210204~2654548,92420837~11448445,109051904~20730346,130078852~4344656,143631694~2819484,150994944~4761673,171966464~4752769,177148887~3686146,182474064~5726314,197132288~3345117,203371987~5918596,209715200~4065233,214907306~4822959,226492416~6043874,235471051~6158445,255852544~3592582,265447020~14406806,293601280~10089374,316577159~4111930,331350016~3305481,342231213~7093209,364904448~9754072,375036141~1226037,376903755~5747680,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.264 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 10 diff [7294659~3604783,14137310~7336653,22298619~3398016,46126522~8645104,71871766~729179,72693380~1592089,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~8169769,130078852~4344656,135726836~4282324,143631694~2819484,151863990~7332122,173661827~3057406,177148887~3686146,182474064~5726314,199192089~1285316,200751176~8539407,210384164~3396269,214907306~4822959,229813410~2722880,235471051~6158445,257231876~3214859,265447020~14406806,284082763~2045905,290745721~4774032,295831353~10304102,316577159~4111930,332951667~1703830,342231213~7093209,366512993~8145527,375036141~1226037,376903755~5747680,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.265 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14137310~7336653,22298619~3398016,46126522~8645104,71871766~729179,72693380~1592089,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~8169769,130078852~4344656,135726836~4282324,143631694~2819484,151863990~7332122,173661827~3057406,177148887~3686146,182474064~5726314,199192089~1285316,200751176~8539407,210384164~3396269,214907306~4822959,229813410~2722880,235471051~6158445,257231876~3214859,265447020~14406806,284082763~2045905,290745721~4774032,295831353~10304102,316577159~4111930,332951667~1703830,342231213~7093209,366512993~8145527,375036141~1226037,376903755~5747680,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.270 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [7294659~3604783,14137310~11559325,41943040~12828586,71871766~729179,72693380~1592089,88210204~2654548,92420837~37361413,130078852~4344656,135726836~4282324,143631694~2819484,150994944~8201168,171966464~4752769,177148887~3686146,182474064~5726314,197132288~12158295,209715200~4065233,214907306~4822959,226492416~6043874,235471051~6158445,255852544~4594191,265447020~14406806,281018368~5110300,289406976~16728479,316577159~4111930,331350016~3305481,342231213~7093209,364904448~9754072,375036141~1226037,376903755~5747680,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.271 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 11 diff [7294659~3604783,14137310~7336653,22298619~3398016,43810632~10960994,56731919~6095770,64609489~4058806,71871766~729179,72693380~1592089,76085135~2537370,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~12811027,135726836~4282324,143631694~2819484,151863990~7332122,173661827~3057406,177148887~3686146,182474064~5726314,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229813410~2722880,235471051~6158445,257231876~3214859,265447020~14406806,284082763~2045905,290745721~4774032,295831353~10304102,316577159~4111930,332951667~1703830,342231213~7093209,366512993~8145527,375036141~1226037,376903755~8927045,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.271 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [7294659~3604783,14137310~7336653,22298619~3398016,43810632~10960994,56731919~6095770,64609489~4058806,71871766~729179,72693380~1592089,76085135~2537370,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~12811027,135726836~4282324,143631694~2819484,151863990~7332122,173661827~3057406,177148887~3686146,182474064~5726314,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229813410~2722880,235471051~6158445,257231876~3214859,265447020~14406806,284082763~2045905,290745721~4774032,295831353~10304102,316577159~4111930,332951667~1703830,342231213~7093209,366512993~8145527,375036141~1226037,376903755~8927045,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.277 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [7294659~3604783,14137310~11559325,41943040~12828586,56731919~6095770,64609489~4058806,71871766~729179,72693380~1592089,75497472~3125033,88210204~2654548,92420837~42002671,135726836~4282324,143631694~2819484,150994944~8201168,171966464~4752769,177148887~3686146,182474064~5726314,197132288~12158295,209715200~4065233,214907306~10694908,226492416~6043874,235471051~6158445,255852544~4594191,265447020~14406806,281018368~5110300,289406976~16728479,316577159~4111930,331350016~3305481,342231213~7093209,364904448~9754072,375036141~1226037,376903755~8927045,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.277 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 12 diff [3528496~7370946,14137310~7336653,22298619~3398016,43810632~10960994,56731919~6095770,64609489~4058806,71871766~729179,72693380~5929125,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~12811027,135726836~4282324,143631694~2819484,146925241~12270871,173661827~3057406,177148887~3686146,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229813410~2722880,235471051~6158445,257231876~3214859,265447020~20681648,290745721~4774032,295831353~10304102,316577159~4111930,327775763~6879734,342231213~7093209,357458885~7280173,366512993~8145527,375036141~1226037,376903755~8927045,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.277 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [3528496~7370946,14137310~7336653,22298619~3398016,43810632~10960994,56731919~6095770,64609489~4058806,71871766~729179,72693380~5929125,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~12811027,135726836~4282324,143631694~2819484,146925241~12270871,173661827~3057406,177148887~3686146,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229813410~2722880,235471051~6158445,257231876~3214859,265447020~20681648,290745721~4774032,295831353~10304102,316577159~4111930,327775763~6879734,342231213~7093209,357458885~7280173,366512993~8145527,375036141~1226037,376903755~8927045,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.282 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~10899442,14137310~11559325,41943040~12828586,56731919~6095770,64609489~4058806,71871766~729179,72693380~5929125,88210204~2654548,92420837~42002671,135726836~4282324,143631694~2819484,146925241~12270871,171966464~4752769,177148887~3686146,182474064~10685508,197132288~12158295,209715200~4065233,214907306~10694908,226492416~6043874,235471051~6158445,255852544~4594191,265447020~20681648,289406976~16728479,316577159~4111930,327775763~6879734,342231213~7093209,356515840~8223218,364904448~9754072,375036141~1226037,376903755~8927045,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.282 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 13 diff [3528496~7370946,14137310~7336653,22298619~3398016,43111238~11660388,56731919~6095770,64609489~4058806,71871766~729179,72693380~5929125,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~12811027,135726836~4282324,143631694~2819484,146925241~12270871,173661827~3057406,177148887~3686146,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,257231876~3214859,265447020~21856450,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,342231213~7093209,357458885~7280173,366512993~19317807,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.282 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [3528496~7370946,14137310~7336653,22298619~3398016,43111238~11660388,56731919~6095770,64609489~4058806,71871766~729179,72693380~5929125,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~12811027,135726836~4282324,143631694~2819484,146925241~12270871,173661827~3057406,177148887~3686146,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,257231876~3214859,265447020~21856450,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,342231213~7093209,357458885~7280173,366512993~19317807,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.288 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~10899442,14137310~11559325,41943040~12828586,56731919~6095770,64609489~4058806,71871766~729179,72693380~5929125,88210204~2654548,92420837~42002671,135726836~4282324,143631694~2819484,146925241~12270871,171966464~4752769,177148887~3686146,182474064~10685508,197132288~12158295,209715200~4065233,214907306~10694908,226492416~15137080,255852544~4594191,265447020~21856450,289406976~17383616,316577159~4111930,327775763~6879734,342231213~7093209,356515840~8223218,364904448~20926352,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.288 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 14 diff [1598369~392670,3528496~7370946,14137310~7336653,22298619~3398016,43111238~11660388,56731919~21890586,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~12811027,135726836~4282324,143631694~2819484,146925241~12270871,173661827~3057406,177148887~3686146,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,249341666~4308983,257231876~3214859,265447020~21856450,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339992078~9332344,357458885~7280173,366512993~19317807,389343167~1935343,391986145~4334286,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.288 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~7370946,14137310~7336653,22298619~3398016,43111238~11660388,56731919~21890586,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~12811027,135726836~4282324,143631694~2819484,146925241~12270871,173661827~3057406,177148887~3686146,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,249341666~4308983,257231876~3214859,265447020~21856450,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339992078~9332344,357458885~7280173,366512993~19317807,389343167~1935343,391986145~4334286,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.294 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~10899442,14137310~11559325,41943040~12828586,56731919~21890586,88210204~2654548,92420837~42002671,135726836~4282324,143631694~2819484,146925241~12270871,171966464~4752769,177148887~3686146,182474064~10685508,197132288~12158295,209715200~4065233,214907306~10694908,226492416~15137080,249341666~4308983,255852544~4594191,265447020~21856450,289406976~17383616,316577159~4111930,327775763~6879734,339992078~9332344,356515840~8223218,364904448~20926352,389343167~1935343,391986145~4334286,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.294 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 15 diff [1598369~392670,3528496~7370946,14137310~7336653,22298619~3398016,31126039~7804481,43111238~35511267,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~12811027,135726836~4282324,143631694~2819484,146925241~12270871,173661827~3057406,177148887~3686146,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,249341666~11105069,265447020~21856450,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339992078~9332344,357458885~7280173,366512993~19317807,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.294 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~7370946,14137310~7336653,22298619~3398016,31126039~7804481,43111238~35511267,88210204~2654548,96231003~7638279,104856556~4244827,112459935~7349748,121612481~12811027,135726836~4282324,143631694~2819484,146925241~12270871,173661827~3057406,177148887~3686146,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,249341666~11105069,265447020~21856450,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339992078~9332344,357458885~7280173,366512993~19317807,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.299 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~10899442,14137310~11559325,31126039~7804481,41943040~36679465,88210204~2654548,92420837~42002671,135726836~4282324,143631694~2819484,146925241~12270871,171966464~4752769,177148887~3686146,182474064~10685508,197132288~12158295,209715200~4065233,214907306~10694908,226492416~15137080,249341666~11105069,265447020~21856450,289406976~17383616,316577159~4111930,327775763~6879734,339992078~9332344,356515840~8223218,364904448~20926352,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.300 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 16 diff [1598369~392670,3528496~8378554,13709905~7764058,22298619~3398016,31126039~7804481,43111238~35511267,83664186~7312323,96231003~7638279,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,173661827~3057406,177148887~3686146,182474064~10685508,199192089~10098494,210384164~3396269,214907306~10694908,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339992078~9332344,357458885~7280173,364955231~20875569,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.300 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7764058,22298619~3398016,31126039~7804481,43111238~35511267,83664186~7312323,96231003~7638279,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,173661827~3057406,177148887~3686146,182474064~10685508,199192089~10098494,210384164~3396269,214907306~10694908,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339992078~9332344,357458885~7280173,364955231~20875569,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.306 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,31126039~7804481,41943040~36679465,83664186~7312323,92420837~42002671,135726836~23469276,171966464~4752769,177148887~3686146,182474064~10685508,197132288~12158295,209715200~4065233,214907306~10694908,226492416~15137080,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6879734,339992078~9332344,356515840~8223218,364904448~20926352,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.306 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 17 diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,31126039~7804481,43111238~37003803,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,173661827~3057406,177148887~16010685,199192089~10098494,210384164~3396269,214907306~13381678,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339992078~10169526,353675391~14545,357458885~7280173,364955231~20875569,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.306 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,31126039~7804481,43111238~37003803,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,173661827~3057406,177148887~16010685,199192089~10098494,210384164~3396269,214907306~13381678,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339992078~10169526,353675391~14545,357458885~7280173,364955231~20875569,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.311 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,31126039~7804481,41943040~38172001,83664186~7312323,92420837~42002671,135726836~23469276,168323939~2295785,171966464~4752769,177148887~16010685,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6879734,339992078~10169526,352321536~1368400,356515840~8223218,364904448~20926352,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.311 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 18 diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,31126039~7804481,43111238~37026997,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,173661827~3057406,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~7280173,364955231~20875569,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.311 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,31126039~7804481,43111238~37026997,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,173661827~3057406,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~7280173,364955231~20875569,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.317 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,31126039~7804481,41943040~38195195,83664186~7312323,92420837~42002671,135726836~23469276,168323939~2295785,171966464~4752769,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~8223218,364904448~20926352,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.317 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 19 diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.317 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.323 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~42002671,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~29314960,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.323 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 6 diff [18397559~2377817,22298619~3398016,71871766~729179,143631694~2819484,177148887~3686146,203371987~4800717,210384164~3396269,229813410~1621406,265447020~5750407,375036141~1226037] 2024-08-17T17:47:33.323 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [18397559~2377817,22298619~3398016,71871766~729179,143631694~2819484,177148887~3686146,203371987~4800717,210384164~3396269,229813410~1621406,265447020~5750407,375036141~1226037] 2024-08-17T17:47:33.328 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [18397559~2377817,20971520~4725115,71871766~729179,143631694~2819484,177148887~3686146,203371987~4800717,209715200~4065233,226492416~4942400,265447020~5750407,375036141~1226037] 2024-08-17T17:47:33.328 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 7 diff [18397559~2377817,22298619~3398016,71871766~729179,72693380~1592089,98323116~4579656,130078852~4344656,143631694~2819484,173661827~3057406,177148887~3686146,203371987~5918596,210384164~3396269,229813410~1621406,231872994~29434,259164751~280375,265447020~5750407,295831353~7859301,317833139~2855950,375036141~1226037,401465291~1068983] 2024-08-17T17:47:33.328 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [18397559~2377817,22298619~3398016,71871766~729179,72693380~1592089,98323116~4579656,130078852~4344656,143631694~2819484,173661827~3057406,177148887~3686146,203371987~5918596,210384164~3396269,229813410~1621406,231872994~29434,259164751~280375,265447020~5750407,295831353~7859301,317833139~2855950,375036141~1226037,401465291~1068983] 2024-08-17T17:47:33.334 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [18397559~2377817,20971520~4725115,71871766~729179,72693380~1592089,98323116~4579656,130078852~4344656,143631694~2819484,171966464~4752769,177148887~3686146,203371987~5918596,209715200~4065233,226492416~4942400,231872994~29434,255852544~3592582,265447020~5750407,293601280~10089374,317833139~2855950,375036141~1226037,401465291~1068983] 2024-08-17T17:47:33.334 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 8 diff [18397559~2377817,22298619~3398016,46126522~3904741,71871766~729179,72693380~1592089,96231003~7638279,121612481~8169769,130078852~4344656,143631694~2819484,151863990~3083981,173661827~3057406,177148887~3686146,182474064~5726314,203371987~5918596,210384164~3396269,229813410~2722880,259164751~280375,265447020~5750407,295831353~7859301,316577159~4111930,375036141~1226037,391986145~4334286,401465291~1068983] 2024-08-17T17:47:33.334 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [18397559~2377817,22298619~3398016,46126522~3904741,71871766~729179,72693380~1592089,96231003~7638279,121612481~8169769,130078852~4344656,143631694~2819484,151863990~3083981,173661827~3057406,177148887~3686146,182474064~5726314,203371987~5918596,210384164~3396269,229813410~2722880,259164751~280375,265447020~5750407,295831353~7859301,316577159~4111930,375036141~1226037,391986145~4334286,401465291~1068983] 2024-08-17T17:47:33.339 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [18397559~2377817,20971520~4725115,41943040~8088223,71871766~729179,72693380~1592089,92420837~11448445,119809683~9972567,130078852~4344656,143631694~2819484,150994944~3953027,171966464~4752769,177148887~3686146,182474064~5726314,203371987~5918596,209715200~4065233,226492416~6043874,255852544~3592582,265447020~5750407,293601280~10089374,316577159~4111930,375036141~1226037,391986145~4334286,401465291~1068983] 2024-08-17T17:47:33.339 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 9 diff [14137310~7336653,22298619~3398016,46126522~8645104,71871766~729179,72693380~1592089,96231003~7638279,121612481~8169769,130078852~4344656,143631694~2819484,151863990~3892627,173661827~3057406,177148887~3686146,182474064~5726314,199192089~1285316,203371987~5918596,210384164~3396269,214907306~4822959,229813410~2722880,235471051~6158445,259164751~280375,265447020~5750407,272649616~7204210,295831353~7859301,316577159~4111930,342231213~2195203,375036141~1226037,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.340 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [14137310~7336653,22298619~3398016,46126522~8645104,71871766~729179,72693380~1592089,96231003~7638279,121612481~8169769,130078852~4344656,143631694~2819484,151863990~3892627,173661827~3057406,177148887~3686146,182474064~5726314,199192089~1285316,203371987~5918596,210384164~3396269,214907306~4822959,229813410~2722880,235471051~6158445,259164751~280375,265447020~5750407,272649616~7204210,295831353~7859301,316577159~4111930,342231213~2195203,375036141~1226037,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.345 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [14137310~11559325,41943040~12828586,71871766~729179,72693380~1592089,92420837~11448445,119809683~9972567,130078852~4344656,143631694~2819484,150994944~4761673,171966464~4752769,177148887~3686146,182474064~5726314,197132288~3345117,203371987~5918596,209715200~4065233,214907306~4822959,226492416~6043874,235471051~6158445,255852544~3592582,265447020~5750407,272649616~7204210,293601280~10089374,316577159~4111930,342231213~2195203,375036141~1226037,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.345 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 10 diff [14137310~7336653,22298619~3398016,46126522~8645104,71871766~729179,72693380~1592089,96231003~7638279,104856556~4244827,121612481~8169769,130078852~4344656,135726836~4282324,143631694~2819484,151863990~7332122,173661827~3057406,177148887~3686146,182474064~5726314,199192089~1285316,200751176~8539407,210384164~3396269,214907306~4822959,229813410~2722880,235471051~6158445,257231876~3214859,265447020~5750407,272649616~7204210,284082763~2045905,290745721~4774032,295831353~10304102,316577159~4111930,342231213~2195203,375036141~1226037,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.345 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [14137310~7336653,22298619~3398016,46126522~8645104,71871766~729179,72693380~1592089,96231003~7638279,104856556~4244827,121612481~8169769,130078852~4344656,135726836~4282324,143631694~2819484,151863990~7332122,173661827~3057406,177148887~3686146,182474064~5726314,199192089~1285316,200751176~8539407,210384164~3396269,214907306~4822959,229813410~2722880,235471051~6158445,257231876~3214859,265447020~5750407,272649616~7204210,284082763~2045905,290745721~4774032,295831353~10304102,316577159~4111930,342231213~2195203,375036141~1226037,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.351 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [14137310~11559325,41943040~12828586,71871766~729179,72693380~1592089,92420837~16680546,119809683~9972567,130078852~4344656,135726836~4282324,143631694~2819484,150994944~8201168,171966464~4752769,177148887~3686146,182474064~5726314,197132288~12158295,209715200~4065233,214907306~4822959,226492416~6043874,235471051~6158445,255852544~4594191,265447020~5750407,272649616~7204210,281018368~5110300,289406976~16728479,316577159~4111930,342231213~2195203,375036141~1226037,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.351 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 11 diff [14137310~7336653,22298619~3398016,43810632~10960994,56731919~6095770,64609489~4058806,71871766~729179,72693380~1592089,76085135~2537370,96231003~7638279,104856556~4244827,121612481~12811027,135726836~4282324,143631694~2819484,151863990~7332122,173661827~3057406,177148887~3686146,182474064~5726314,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229813410~2722880,235471051~6158445,257231876~3214859,265447020~5750407,272649616~7204210,284082763~2045905,290745721~4774032,295831353~10304102,316577159~4111930,342231213~2195203,345979605~2021847,375036141~1226037,378717919~7112881,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.351 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [14137310~7336653,22298619~3398016,43810632~10960994,56731919~6095770,64609489~4058806,71871766~729179,72693380~1592089,76085135~2537370,96231003~7638279,104856556~4244827,121612481~12811027,135726836~4282324,143631694~2819484,151863990~7332122,173661827~3057406,177148887~3686146,182474064~5726314,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229813410~2722880,235471051~6158445,257231876~3214859,265447020~5750407,272649616~7204210,284082763~2045905,290745721~4774032,295831353~10304102,316577159~4111930,342231213~2195203,345979605~2021847,375036141~1226037,378717919~7112881,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.357 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [14137310~11559325,41943040~12828586,56731919~6095770,64609489~4058806,71871766~729179,72693380~1592089,75497472~3125033,92420837~16680546,119809683~14613825,135726836~4282324,143631694~2819484,150994944~8201168,171966464~4752769,177148887~3686146,182474064~5726314,197132288~12158295,209715200~4065233,214907306~10694908,226492416~6043874,235471051~6158445,255852544~4594191,265447020~5750407,272649616~7204210,281018368~5110300,289406976~16728479,316577159~4111930,342231213~2195203,345979605~2021847,375036141~1226037,378717919~7112881,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.357 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 12 diff [3528496~4240299,14137310~7336653,22298619~3398016,43810632~10960994,56731919~6095770,64609489~4058806,71871766~729179,72693380~5929125,96231003~7638279,104856556~4244827,121612481~12811027,135726836~4282324,143631694~2819484,146925241~12270871,173661827~3057406,177148887~3686146,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229813410~2722880,235471051~6158445,257231876~3214859,265447020~5750407,272649616~13479052,290745721~4774032,295831353~10304102,316577159~4111930,327775763~6410563,342231213~2195203,345979605~2021847,357458885~7280173,375036141~1226037,378717919~7112881,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.357 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [3528496~4240299,14137310~7336653,22298619~3398016,43810632~10960994,56731919~6095770,64609489~4058806,71871766~729179,72693380~5929125,96231003~7638279,104856556~4244827,121612481~12811027,135726836~4282324,143631694~2819484,146925241~12270871,173661827~3057406,177148887~3686146,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229813410~2722880,235471051~6158445,257231876~3214859,265447020~5750407,272649616~13479052,290745721~4774032,295831353~10304102,316577159~4111930,327775763~6410563,342231213~2195203,345979605~2021847,357458885~7280173,375036141~1226037,378717919~7112881,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.363 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,14137310~11559325,41943040~12828586,56731919~6095770,64609489~4058806,71871766~729179,72693380~5929125,92420837~16680546,119809683~14613825,135726836~4282324,143631694~2819484,146925241~12270871,171966464~4752769,177148887~3686146,182474064~10685508,197132288~12158295,209715200~4065233,214907306~10694908,226492416~6043874,235471051~6158445,255852544~4594191,265447020~5750407,272649616~13479052,289406976~16728479,316577159~4111930,327775763~6410563,342231213~2195203,345979605~2021847,356515840~8223218,375036141~1226037,378717919~7112881,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.363 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 13 diff [3528496~4240299,14137310~7336653,22298619~3398016,43111238~11660388,56731919~6095770,64609489~4058806,71871766~729179,72693380~5929125,96231003~7638279,104856556~4244827,121612481~12811027,135726836~4282324,143631694~2819484,146925241~12270871,173661827~3057406,177148887~3686146,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,257231876~3214859,265447020~5750407,272649616~14653854,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,342231213~2195203,345979605~2021847,357458885~7280173,372530391~13300409,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.363 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [3528496~4240299,14137310~7336653,22298619~3398016,43111238~11660388,56731919~6095770,64609489~4058806,71871766~729179,72693380~5929125,96231003~7638279,104856556~4244827,121612481~12811027,135726836~4282324,143631694~2819484,146925241~12270871,173661827~3057406,177148887~3686146,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,257231876~3214859,265447020~5750407,272649616~14653854,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,342231213~2195203,345979605~2021847,357458885~7280173,372530391~13300409,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.369 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,14137310~11559325,41943040~12828586,56731919~6095770,64609489~4058806,71871766~729179,72693380~5929125,92420837~16680546,119809683~14613825,135726836~4282324,143631694~2819484,146925241~12270871,171966464~4752769,177148887~3686146,182474064~10685508,197132288~12158295,209715200~4065233,214907306~10694908,226492416~15137080,255852544~4594191,265447020~5750407,272649616~14653854,289406976~17383616,316577159~4111930,327775763~6410563,342231213~2195203,345979605~2021847,356515840~8223218,372530391~13300409,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.369 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 14 diff [1598369~392670,3528496~4240299,14137310~7336653,22298619~3398016,43111238~11660388,56731919~21890586,96231003~7638279,104856556~4244827,121612481~12811027,135726836~4282324,143631694~2819484,146925241~12270871,173661827~3057406,177148887~3686146,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,249341666~4308983,257231876~3214859,265447020~5750407,272649616~14653854,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339992078~4434338,345979605~2021847,357458885~7280173,372530391~13300409,389343167~1935343,391986145~4334286,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.369 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~4240299,14137310~7336653,22298619~3398016,43111238~11660388,56731919~21890586,96231003~7638279,104856556~4244827,121612481~12811027,135726836~4282324,143631694~2819484,146925241~12270871,173661827~3057406,177148887~3686146,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,249341666~4308983,257231876~3214859,265447020~5750407,272649616~14653854,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339992078~4434338,345979605~2021847,357458885~7280173,372530391~13300409,389343167~1935343,391986145~4334286,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.375 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,14137310~11559325,41943040~12828586,56731919~21890586,92420837~16680546,119809683~14613825,135726836~4282324,143631694~2819484,146925241~12270871,171966464~4752769,177148887~3686146,182474064~10685508,197132288~12158295,209715200~4065233,214907306~10694908,226492416~15137080,249341666~4308983,255852544~4594191,265447020~5750407,272649616~14653854,289406976~17383616,316577159~4111930,327775763~6410563,339992078~4434338,345979605~2021847,356515840~8223218,372530391~13300409,389343167~1935343,391986145~4334286,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.375 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 15 diff [1598369~392670,3528496~4240299,14137310~7336653,22298619~3398016,31126039~7804481,43111238~35511267,96231003~7638279,104856556~4244827,121612481~12811027,135726836~4282324,143631694~2819484,146925241~12270871,173661827~3057406,177148887~3686146,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,249341666~11105069,265447020~21856450,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339992078~4434338,345979605~2021847,357458885~7280173,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.377 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~4240299,14137310~7336653,22298619~3398016,31126039~7804481,43111238~35511267,96231003~7638279,104856556~4244827,121612481~12811027,135726836~4282324,143631694~2819484,146925241~12270871,173661827~3057406,177148887~3686146,182474064~10685508,199192089~1285316,200751176~8539407,210384164~3396269,214907306~10694908,229463800~12165696,249341666~11105069,265447020~21856450,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339992078~4434338,345979605~2021847,357458885~7280173,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.381 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,14137310~11559325,31126039~7804481,41943040~36679465,92420837~16680546,119809683~14613825,135726836~4282324,143631694~2819484,146925241~12270871,171966464~4752769,177148887~3686146,182474064~10685508,197132288~12158295,209715200~4065233,214907306~10694908,226492416~15137080,249341666~11105069,265447020~21856450,289406976~17383616,316577159~4111930,327775763~6410563,339992078~4434338,345979605~2021847,356515840~8223218,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.381 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 16 diff [1598369~392670,3528496~8378554,13709905~7764058,22298619~3398016,31126039~7804481,43111238~35511267,83664186~7312323,96231003~7638279,104378282~4723101,121612481~12811027,135726836~23469276,173661827~3057406,177148887~3686146,182474064~10685508,199192089~10098494,210384164~3396269,214907306~10694908,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339992078~4434338,345979605~2021847,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.381 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7764058,22298619~3398016,31126039~7804481,43111238~35511267,83664186~7312323,96231003~7638279,104378282~4723101,121612481~12811027,135726836~23469276,173661827~3057406,177148887~3686146,182474064~10685508,199192089~10098494,210384164~3396269,214907306~10694908,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339992078~4434338,345979605~2021847,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.386 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,31126039~7804481,41943040~36679465,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,171966464~4752769,177148887~3686146,182474064~10685508,197132288~12158295,209715200~4065233,214907306~10694908,226492416~15137080,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6410563,339992078~4434338,345979605~2021847,356515840~8223218,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.387 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 17 diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,31126039~7804481,43111238~37003803,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,173661827~3057406,177148887~16010685,199192089~10098494,210384164~3396269,214907306~13381678,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339992078~4434338,345408086~4753518,353675391~14545,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.387 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,31126039~7804481,43111238~37003803,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,173661827~3057406,177148887~16010685,199192089~10098494,210384164~3396269,214907306~13381678,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339992078~4434338,345408086~4753518,353675391~14545,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.392 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,31126039~7804481,41943040~38172001,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171966464~4752769,177148887~16010685,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6410563,339992078~4434338,345408086~4753518,352321536~1368400,356515840~8223218,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.392 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 18 diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,31126039~7804481,43111238~37026997,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,173661827~3057406,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.393 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,31126039~7804481,43111238~37026997,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,173661827~3057406,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229463800~12165696,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.398 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,31126039~7804481,41943040~38195195,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171966464~4752769,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~8223218,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.398 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 19 diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.398 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.404 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.404 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 7 diff [72693380~1592089,98323116~4579656,130078852~4344656,173661827~3057406,205047266~4243317,231872994~29434,259164751~280375,295831353~7859301,317833139~2855950,401465291~1068983] 2024-08-17T17:47:33.404 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [72693380~1592089,98323116~4579656,130078852~4344656,173661827~3057406,205047266~4243317,231872994~29434,259164751~280375,295831353~7859301,317833139~2855950,401465291~1068983] 2024-08-17T17:47:33.410 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [72693380~1592089,98323116~4579656,130078852~4344656,171966464~4752769,205047266~4243317,231872994~29434,255852544~3592582,293601280~10089374,317833139~2855950,401465291~1068983] 2024-08-17T17:47:33.410 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 8 diff [46126522~3904741,72693380~1592089,96231003~7638279,121612481~8169769,130078852~4344656,151863990~3083981,173661827~3057406,182474064~5726314,205047266~4243317,231388101~1148189,259164751~280375,295831353~7859301,316577159~4111930,391986145~4334286,401465291~1068983] 2024-08-17T17:47:33.410 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [46126522~3904741,72693380~1592089,96231003~7638279,121612481~8169769,130078852~4344656,151863990~3083981,173661827~3057406,182474064~5726314,205047266~4243317,231388101~1148189,259164751~280375,295831353~7859301,316577159~4111930,391986145~4334286,401465291~1068983] 2024-08-17T17:47:33.415 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [41943040~8088223,72693380~1592089,92420837~11448445,119809683~9972567,130078852~4344656,150994944~3953027,171966464~4752769,182474064~5726314,205047266~4243317,231388101~1148189,255852544~3592582,293601280~10089374,316577159~4111930,391986145~4334286,401465291~1068983] 2024-08-17T17:47:33.415 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 9 diff [14137310~7336653,46126522~8645104,72693380~1592089,96231003~7638279,121612481~8169769,130078852~4344656,151863990~3892627,173661827~3057406,182474064~5726314,199192089~1285316,205047266~4243317,214907306~4822959,231388101~1148189,235471051~6158445,259164751~280375,272649616~7204210,295831353~7859301,316577159~4111930,342231213~2195203,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.415 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [14137310~7336653,46126522~8645104,72693380~1592089,96231003~7638279,121612481~8169769,130078852~4344656,151863990~3892627,173661827~3057406,182474064~5726314,199192089~1285316,205047266~4243317,214907306~4822959,231388101~1148189,235471051~6158445,259164751~280375,272649616~7204210,295831353~7859301,316577159~4111930,342231213~2195203,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.421 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [14137310~7336653,41943040~12828586,72693380~1592089,92420837~11448445,119809683~9972567,130078852~4344656,150994944~4761673,171966464~4752769,182474064~5726314,197132288~3345117,205047266~4243317,214907306~4822959,231388101~1148189,235471051~6158445,255852544~3592582,272649616~7204210,293601280~10089374,316577159~4111930,342231213~2195203,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.421 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 10 diff [14137310~7336653,46126522~8645104,72693380~1592089,96231003~7638279,104856556~4244827,121612481~8169769,130078852~4344656,135726836~4282324,151863990~7332122,173661827~3057406,182474064~5726314,199192089~1285316,200751176~8539407,214907306~4822959,231388101~1148189,235471051~6158445,257231876~3214859,272649616~7204210,284082763~2045905,290745721~4774032,295831353~10304102,316577159~4111930,342231213~2195203,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.421 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [14137310~7336653,46126522~8645104,72693380~1592089,96231003~7638279,104856556~4244827,121612481~8169769,130078852~4344656,135726836~4282324,151863990~7332122,173661827~3057406,182474064~5726314,199192089~1285316,200751176~8539407,214907306~4822959,231388101~1148189,235471051~6158445,257231876~3214859,272649616~7204210,284082763~2045905,290745721~4774032,295831353~10304102,316577159~4111930,342231213~2195203,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.427 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [14137310~7336653,41943040~12828586,72693380~1592089,92420837~16680546,119809683~9972567,130078852~4344656,135726836~4282324,150994944~8201168,171966464~4752769,182474064~5726314,197132288~12158295,214907306~4822959,231388101~1148189,235471051~6158445,255852544~4594191,272649616~7204210,281018368~5110300,289406976~16728479,316577159~4111930,342231213~2195203,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.427 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 11 diff [14137310~7336653,43810632~10960994,56731919~6095770,64609489~4058806,72693380~1592089,76085135~2537370,96231003~7638279,104856556~4244827,121612481~12811027,135726836~4282324,151863990~7332122,173661827~3057406,182474064~5726314,199192089~1285316,200751176~8539407,214907306~10694908,231388101~1148189,235471051~6158445,257231876~3214859,272649616~7204210,284082763~2045905,290745721~4774032,295831353~10304102,316577159~4111930,342231213~2195203,345979605~2021847,378717919~7112881,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.427 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [14137310~7336653,43810632~10960994,56731919~6095770,64609489~4058806,72693380~1592089,76085135~2537370,96231003~7638279,104856556~4244827,121612481~12811027,135726836~4282324,151863990~7332122,173661827~3057406,182474064~5726314,199192089~1285316,200751176~8539407,214907306~10694908,231388101~1148189,235471051~6158445,257231876~3214859,272649616~7204210,284082763~2045905,290745721~4774032,295831353~10304102,316577159~4111930,342231213~2195203,345979605~2021847,378717919~7112881,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.432 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [14137310~7336653,41943040~12828586,56731919~6095770,64609489~4058806,72693380~1592089,75497472~3125033,92420837~16680546,119809683~14613825,135726836~4282324,150994944~8201168,171966464~4752769,182474064~5726314,197132288~12158295,214907306~10694908,231388101~1148189,235471051~6158445,255852544~4594191,272649616~7204210,281018368~5110300,289406976~16728479,316577159~4111930,342231213~2195203,345979605~2021847,378717919~7112881,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.432 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 12 diff [3528496~4240299,14137310~7336653,43810632~10960994,56731919~6095770,64609489~4058806,72693380~5929125,96231003~7638279,104856556~4244827,121612481~12811027,135726836~4282324,146925241~12270871,173661827~3057406,182474064~10685508,199192089~1285316,200751176~8539407,214907306~10694908,231388101~1148189,235471051~6158445,257231876~3214859,272649616~13479052,290745721~4774032,295831353~10304102,316577159~4111930,327775763~6410563,342231213~2195203,345979605~2021847,357458885~7280173,378717919~7112881,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.432 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [3528496~4240299,14137310~7336653,43810632~10960994,56731919~6095770,64609489~4058806,72693380~5929125,96231003~7638279,104856556~4244827,121612481~12811027,135726836~4282324,146925241~12270871,173661827~3057406,182474064~10685508,199192089~1285316,200751176~8539407,214907306~10694908,231388101~1148189,235471051~6158445,257231876~3214859,272649616~13479052,290745721~4774032,295831353~10304102,316577159~4111930,327775763~6410563,342231213~2195203,345979605~2021847,357458885~7280173,378717919~7112881,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.438 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,14137310~7336653,41943040~12828586,56731919~6095770,64609489~4058806,72693380~5929125,92420837~16680546,119809683~14613825,135726836~4282324,146925241~12270871,171966464~4752769,182474064~10685508,197132288~12158295,214907306~10694908,231388101~1148189,235471051~6158445,255852544~4594191,272649616~13479052,289406976~16728479,316577159~4111930,327775763~6410563,342231213~2195203,345979605~2021847,356515840~8223218,378717919~7112881,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.438 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 13 diff [3528496~4240299,14137310~7336653,43111238~11660388,56731919~6095770,64609489~4058806,72693380~5929125,96231003~7638279,104856556~4244827,121612481~12811027,135726836~4282324,146925241~12270871,173661827~3057406,182474064~10685508,199192089~1285316,200751176~8539407,214907306~10694908,229463800~12165696,257231876~3214859,272649616~14653854,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,342231213~2195203,345979605~2021847,357458885~7280173,372530391~13300409,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.438 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [3528496~4240299,14137310~7336653,43111238~11660388,56731919~6095770,64609489~4058806,72693380~5929125,96231003~7638279,104856556~4244827,121612481~12811027,135726836~4282324,146925241~12270871,173661827~3057406,182474064~10685508,199192089~1285316,200751176~8539407,214907306~10694908,229463800~12165696,257231876~3214859,272649616~14653854,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,342231213~2195203,345979605~2021847,357458885~7280173,372530391~13300409,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.444 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,14137310~7336653,41943040~12828586,56731919~6095770,64609489~4058806,72693380~5929125,92420837~16680546,119809683~14613825,135726836~4282324,146925241~12270871,171966464~4752769,182474064~10685508,197132288~12158295,214907306~10694908,229463800~12165696,255852544~4594191,272649616~14653854,289406976~17383616,316577159~4111930,327775763~6410563,342231213~2195203,345979605~2021847,356515840~8223218,372530391~13300409,389343167~1935343,391986145~4334286,401465291~1068983,404966874~7561094] 2024-08-17T17:47:33.444 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 14 diff [1598369~392670,3528496~4240299,14137310~7336653,43111238~11660388,56731919~21890586,96231003~7638279,104856556~4244827,121612481~12811027,135726836~4282324,146925241~12270871,173661827~3057406,182474064~10685508,199192089~1285316,200751176~8539407,214907306~10694908,229463800~12165696,249341666~4308983,257231876~3214859,272649616~14653854,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339992078~4434338,345979605~2021847,357458885~7280173,372530391~13300409,389343167~1935343,391986145~4334286,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.444 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~4240299,14137310~7336653,43111238~11660388,56731919~21890586,96231003~7638279,104856556~4244827,121612481~12811027,135726836~4282324,146925241~12270871,173661827~3057406,182474064~10685508,199192089~1285316,200751176~8539407,214907306~10694908,229463800~12165696,249341666~4308983,257231876~3214859,272649616~14653854,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339992078~4434338,345979605~2021847,357458885~7280173,372530391~13300409,389343167~1935343,391986145~4334286,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.449 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,14137310~7336653,41943040~12828586,56731919~21890586,92420837~16680546,119809683~14613825,135726836~4282324,146925241~12270871,171966464~4752769,182474064~10685508,197132288~12158295,214907306~10694908,229463800~12165696,249341666~4308983,255852544~4594191,272649616~14653854,289406976~17383616,316577159~4111930,327775763~6410563,339992078~4434338,345979605~2021847,356515840~8223218,372530391~13300409,389343167~1935343,391986145~4334286,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.450 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 15 diff [1598369~392670,3528496~4240299,14137310~7336653,31126039~7804481,43111238~35511267,96231003~7638279,104856556~4244827,121612481~12811027,135726836~4282324,146925241~12270871,173661827~3057406,182474064~10685508,199192089~1285316,200751176~8539407,214907306~10694908,229463800~12165696,249341666~11105069,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339992078~4434338,345979605~2021847,357458885~7280173,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.450 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~4240299,14137310~7336653,31126039~7804481,43111238~35511267,96231003~7638279,104856556~4244827,121612481~12811027,135726836~4282324,146925241~12270871,173661827~3057406,182474064~10685508,199192089~1285316,200751176~8539407,214907306~10694908,229463800~12165696,249341666~11105069,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339992078~4434338,345979605~2021847,357458885~7280173,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.455 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,14137310~7336653,31126039~7804481,41943040~36679465,92420837~16680546,119809683~14613825,135726836~4282324,146925241~12270871,171966464~4752769,182474064~10685508,197132288~12158295,214907306~10694908,229463800~12165696,249341666~11105069,270348712~16954758,289406976~17383616,316577159~4111930,327775763~6410563,339992078~4434338,345979605~2021847,356515840~8223218,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.455 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 16 diff [1598369~392670,3528496~8378554,13709905~7764058,31126039~7804481,43111238~35511267,83664186~7312323,96231003~7638279,104378282~4723101,121612481~12811027,135726836~23469276,173661827~3057406,182474064~10685508,199192089~10098494,214907306~10694908,229463800~12165696,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339992078~4434338,345979605~2021847,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.455 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7764058,31126039~7804481,43111238~35511267,83664186~7312323,96231003~7638279,104378282~4723101,121612481~12811027,135726836~23469276,173661827~3057406,182474064~10685508,199192089~10098494,214907306~10694908,229463800~12165696,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339992078~4434338,345979605~2021847,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.460 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7764058,31126039~7804481,41943040~36679465,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,171966464~4752769,182474064~10685508,197132288~12158295,214907306~10694908,229463800~12165696,249341666~20163930,270348712~16954758,289406976~17383616,316577159~4111930,327775763~6410563,339992078~4434338,345979605~2021847,356515840~8223218,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.461 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 17 diff [1598369~392670,3528496~8378554,13709905~7986367,31126039~7804481,43111238~37003803,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,173661827~3057406,179399560~13760012,199192089~10098494,214907306~13381678,229463800~12165696,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339992078~4434338,345408086~4753518,353675391~14545,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.461 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,31126039~7804481,43111238~37003803,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,173661827~3057406,179399560~13760012,199192089~10098494,214907306~13381678,229463800~12165696,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339992078~4434338,345408086~4753518,353675391~14545,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.467 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,31126039~7804481,41943040~38172001,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171966464~4752769,179399560~13760012,197132288~12158295,214907306~13381678,229463800~12165696,249341666~20163930,270348712~16954758,289406976~17383616,316577159~4111930,327775763~6410563,339992078~4434338,345408086~4753518,352321536~1368400,356515840~8223218,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.467 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 18 diff [1598369~392670,3528496~8378554,13709905~7986367,31126039~7804481,43111238~37026997,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,173661827~3057406,179399560~15706006,199192089~10098494,214907306~13381678,229463800~12165696,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.467 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,31126039~7804481,43111238~37026997,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,173661827~3057406,179399560~15706006,199192089~10098494,214907306~13381678,229463800~12165696,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.473 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,31126039~7804481,41943040~38195195,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171966464~4752769,179399560~15706006,197132288~12158295,214907306~13381678,229463800~12165696,249341666~20163930,270348712~16954758,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~8223218,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.473 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 19 diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.473 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.478 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,179399560~15706006,197132288~12158295,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:33.479 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to 8 diff [46126522~3904741,96231003~7638279,121612481~8169769,151863990~3083981,182474064~5726314,231388101~1148189,316577159~3150955,391986145~4334286] 2024-08-17T17:47:33.479 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [46126522~3904741,96231003~7638279,121612481~8169769,151863990~3083981,182474064~5726314,231388101~1148189,316577159~3150955,391986145~4334286] 2024-08-17T17:47:33.484 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [41943040~8088223,92420837~11448445,119809683~9972567,150994944~3953027,182474064~5726314,231388101~1148189,316577159~3150955,391986145~4334286] 2024-08-17T17:47:33.484 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to 9 diff [14137310~7336653,46126522~8645104,96231003~7638279,121612481~8169769,151863990~3892627,182474064~5726314,199192089~1285316,214907306~4822959,231388101~1148189,235471051~6158445,272649616~7204210,316577159~3150955,342231213~2195203,391986145~4334286,404966874~7561094] 2024-08-17T17:47:33.484 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [14137310~7336653,46126522~8645104,96231003~7638279,121612481~8169769,151863990~3892627,182474064~5726314,199192089~1285316,214907306~4822959,231388101~1148189,235471051~6158445,272649616~7204210,316577159~3150955,342231213~2195203,391986145~4334286,404966874~7561094] 2024-08-17T17:47:33.490 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [14137310~7336653,41943040~12828586,92420837~11448445,119809683~9972567,150994944~4761673,182474064~5726314,197132288~3345117,214907306~4822959,231388101~1148189,235471051~6158445,272649616~7204210,316577159~3150955,342231213~2195203,391986145~4334286,404966874~7561094] 2024-08-17T17:47:33.490 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to 10 diff [14137310~7336653,46126522~8645104,96231003~7638279,104856556~4244827,121612481~8169769,135726836~4282324,151863990~7332122,182474064~5726314,199192089~1285316,200751176~7563019,214907306~4822959,231388101~1148189,235471051~6158445,257231876~3214859,272649616~7204210,284082763~2045905,290745721~4774032,299299990~6835465,316577159~3150955,342231213~2195203,391986145~4334286,404966874~7561094] 2024-08-17T17:47:33.490 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [14137310~7336653,46126522~8645104,96231003~7638279,104856556~4244827,121612481~8169769,135726836~4282324,151863990~7332122,182474064~5726314,199192089~1285316,200751176~7563019,214907306~4822959,231388101~1148189,235471051~6158445,257231876~3214859,272649616~7204210,284082763~2045905,290745721~4774032,299299990~6835465,316577159~3150955,342231213~2195203,391986145~4334286,404966874~7561094] 2024-08-17T17:47:33.496 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [14137310~7336653,41943040~12828586,92420837~16680546,119809683~9972567,135726836~4282324,150994944~8201168,182474064~5726314,197132288~11181907,214907306~4822959,231388101~1148189,235471051~6158445,257231876~3214859,272649616~7204210,281018368~5110300,289406976~6112777,299299990~6835465,316577159~3150955,342231213~2195203,391986145~4334286,404966874~7561094] 2024-08-17T17:47:33.496 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to 11 diff [14137310~7336653,43810632~10960994,56731919~6095770,64609489~4058806,76085135~2537370,96231003~7638279,104856556~4244827,121612481~8468550,135726836~4282324,151863990~7332122,182474064~5726314,199192089~1285316,200751176~7563019,214907306~10694908,231388101~1148189,235471051~6158445,257231876~3214859,272649616~7204210,284082763~2045905,290745721~4774032,299299990~6835465,316577159~3150955,342231213~2195203,345979605~2021847,378717919~7112881,389343167~1935343,391986145~4334286,404966874~7561094] 2024-08-17T17:47:33.496 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [14137310~7336653,43810632~10960994,56731919~6095770,64609489~4058806,76085135~2537370,96231003~7638279,104856556~4244827,121612481~8468550,135726836~4282324,151863990~7332122,182474064~5726314,199192089~1285316,200751176~7563019,214907306~10694908,231388101~1148189,235471051~6158445,257231876~3214859,272649616~7204210,284082763~2045905,290745721~4774032,299299990~6835465,316577159~3150955,342231213~2195203,345979605~2021847,378717919~7112881,389343167~1935343,391986145~4334286,404966874~7561094] 2024-08-17T17:47:33.501 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [14137310~7336653,41943040~12828586,56731919~6095770,64609489~4058806,75497472~3125033,92420837~16680546,119809683~10271348,135726836~4282324,150994944~8201168,182474064~5726314,197132288~11181907,214907306~10694908,231388101~1148189,235471051~6158445,257231876~3214859,272649616~7204210,281018368~5110300,289406976~6112777,299299990~6835465,316577159~3150955,342231213~2195203,345979605~2021847,378717919~7112881,389343167~1935343,391986145~4334286,404966874~7561094] 2024-08-17T17:47:33.501 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to 12 diff [3528496~4240299,14137310~7336653,43810632~10960994,56731919~6095770,64609489~4058806,72723682~5898823,96231003~7638279,104856556~4244827,121612481~8468550,135726836~4282324,146925241~12270871,182474064~10685508,199192089~1285316,200751176~7563019,214907306~10694908,231388101~1148189,235471051~6158445,257231876~3214859,272649616~13479052,290745721~4774032,299299990~6835465,316577159~3150955,327775763~6410563,342231213~2195203,345979605~2021847,357458885~7280173,378717919~7112881,389343167~1935343,391986145~4334286,404966874~7561094] 2024-08-17T17:47:33.501 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [3528496~4240299,14137310~7336653,43810632~10960994,56731919~6095770,64609489~4058806,72723682~5898823,96231003~7638279,104856556~4244827,121612481~8468550,135726836~4282324,146925241~12270871,182474064~10685508,199192089~1285316,200751176~7563019,214907306~10694908,231388101~1148189,235471051~6158445,257231876~3214859,272649616~13479052,290745721~4774032,299299990~6835465,316577159~3150955,327775763~6410563,342231213~2195203,345979605~2021847,357458885~7280173,378717919~7112881,389343167~1935343,391986145~4334286,404966874~7561094] 2024-08-17T17:47:33.507 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,14137310~7336653,41943040~12828586,56731919~6095770,64609489~4058806,72723682~5898823,92420837~16680546,119809683~10271348,135726836~4282324,146925241~12270871,182474064~10685508,197132288~11181907,214907306~10694908,231388101~1148189,235471051~6158445,257231876~3214859,272649616~13479052,289406976~6112777,299299990~6835465,316577159~3150955,327775763~6410563,342231213~2195203,345979605~2021847,356515840~8223218,378717919~7112881,389343167~1935343,391986145~4334286,404966874~7561094] 2024-08-17T17:47:33.507 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to 13 diff [3528496~4240299,14137310~7336653,43111238~11660388,56731919~6095770,64609489~4058806,72723682~5898823,96231003~7638279,104856556~4244827,121612481~8468550,135726836~4282324,146925241~12270871,182474064~10685508,199192089~1285316,200751176~7563019,214907306~10694908,229463800~12165696,257231876~3214859,272649616~14653854,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,342231213~2195203,345979605~2021847,357458885~7280173,372530391~13300409,389343167~1935343,391986145~4334286,404966874~7561094] 2024-08-17T17:47:33.507 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [3528496~4240299,14137310~7336653,43111238~11660388,56731919~6095770,64609489~4058806,72723682~5898823,96231003~7638279,104856556~4244827,121612481~8468550,135726836~4282324,146925241~12270871,182474064~10685508,199192089~1285316,200751176~7563019,214907306~10694908,229463800~12165696,257231876~3214859,272649616~14653854,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,342231213~2195203,345979605~2021847,357458885~7280173,372530391~13300409,389343167~1935343,391986145~4334286,404966874~7561094] 2024-08-17T17:47:33.513 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,14137310~7336653,41943040~12828586,56731919~6095770,64609489~4058806,72723682~5898823,92420837~16680546,119809683~10271348,135726836~4282324,146925241~12270871,182474064~10685508,197132288~11181907,214907306~10694908,229463800~12165696,257231876~3214859,272649616~14653854,289406976~6112777,299299990~7490602,316577159~3150955,327775763~6410563,342231213~2195203,345979605~2021847,356515840~8223218,372530391~13300409,389343167~1935343,391986145~4334286,404966874~7561094] 2024-08-17T17:47:33.513 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to 14 diff [1598369~392670,3528496~4240299,14137310~7336653,43111238~11660388,56731919~21890586,96231003~7638279,104856556~4244827,121612481~8468550,135726836~4282324,146925241~12270871,182474064~10685508,199192089~1285316,200751176~7563019,214907306~10694908,229463800~12165696,249341666~4308983,257231876~3214859,272649616~14653854,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339992078~4434338,345979605~2021847,357458885~7280173,372530391~13300409,389343167~1935343,391986145~4334286,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.513 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~4240299,14137310~7336653,43111238~11660388,56731919~21890586,96231003~7638279,104856556~4244827,121612481~8468550,135726836~4282324,146925241~12270871,182474064~10685508,199192089~1285316,200751176~7563019,214907306~10694908,229463800~12165696,249341666~4308983,257231876~3214859,272649616~14653854,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339992078~4434338,345979605~2021847,357458885~7280173,372530391~13300409,389343167~1935343,391986145~4334286,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.518 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,14137310~7336653,41943040~12828586,56731919~21890586,92420837~16680546,119809683~10271348,135726836~4282324,146925241~12270871,182474064~10685508,197132288~11181907,214907306~10694908,229463800~12165696,249341666~4308983,257231876~3214859,272649616~14653854,289406976~6112777,299299990~7490602,316577159~3150955,327775763~6410563,339992078~4434338,345979605~2021847,356515840~8223218,372530391~13300409,389343167~1935343,391986145~4334286,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.518 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to 15 diff [1598369~392670,3528496~4240299,14137310~7336653,31126039~7804481,43111238~35511267,96231003~7638279,104856556~4244827,121612481~8468550,135726836~4282324,146925241~12270871,182474064~10685508,199192089~1285316,200751176~7563019,214907306~10694908,229463800~12165696,249341666~11105069,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339992078~4434338,345979605~2021847,357458885~7280173,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.519 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~4240299,14137310~7336653,31126039~7804481,43111238~35511267,96231003~7638279,104856556~4244827,121612481~8468550,135726836~4282324,146925241~12270871,182474064~10685508,199192089~1285316,200751176~7563019,214907306~10694908,229463800~12165696,249341666~11105069,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339992078~4434338,345979605~2021847,357458885~7280173,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.524 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,14137310~7336653,31126039~7804481,41943040~36679465,92420837~16680546,119809683~10271348,135726836~4282324,146925241~12270871,182474064~10685508,197132288~11181907,214907306~10694908,229463800~12165696,249341666~11105069,270348712~16954758,289406976~6112777,299299990~7490602,316577159~3150955,327775763~6410563,339992078~4434338,345979605~2021847,356515840~8223218,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.524 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to 16 diff [1598369~392670,3528496~8378554,13709905~7764058,31126039~7804481,43111238~35511267,83664186~7312323,96231003~7638279,104378282~4723101,121612481~8468550,135726836~23469276,182474064~10685508,199192089~9122106,214907306~10694908,229463800~12165696,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339992078~4434338,345979605~2021847,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.524 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7764058,31126039~7804481,43111238~35511267,83664186~7312323,96231003~7638279,104378282~4723101,121612481~8468550,135726836~23469276,182474064~10685508,199192089~9122106,214907306~10694908,229463800~12165696,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339992078~4434338,345979605~2021847,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.530 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7764058,31126039~7804481,41943040~36679465,83664186~7312323,92420837~16680546,119809683~10271348,135726836~23469276,182474064~10685508,197132288~11181907,214907306~10694908,229463800~12165696,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,316577159~3150955,327775763~6410563,339992078~4434338,345979605~2021847,356515840~8223218,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.530 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to 17 diff [1598369~392670,3528496~8378554,13709905~7986367,31126039~7804481,43111238~37003803,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,179399560~13760012,199192089~9122106,214907306~13381678,229463800~12165696,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339992078~4434338,345408086~4753518,353675391~14545,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.530 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,31126039~7804481,43111238~37003803,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,179399560~13760012,199192089~9122106,214907306~13381678,229463800~12165696,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339992078~4434338,345408086~4753518,353675391~14545,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.536 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,31126039~7804481,41943040~38172001,83664186~7312323,92420837~16680546,119809683~10271348,135726836~23469276,168323939~2295785,179399560~13760012,197132288~11181907,214907306~13381678,229463800~12165696,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,316577159~3150955,327775763~6410563,339992078~4434338,345408086~4753518,352321536~1368400,356515840~8223218,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.536 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to 18 diff [1598369~392670,3528496~8378554,13709905~7986367,31126039~7804481,43111238~37026997,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,179399560~15706006,199192089~9122106,214907306~13381678,229463800~12165696,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.536 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,31126039~7804481,43111238~37026997,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,179399560~15706006,199192089~9122106,214907306~13381678,229463800~12165696,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.542 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,31126039~7804481,41943040~38195195,83664186~7312323,92420837~16680546,119809683~10271348,135726836~23469276,168323939~2295785,179399560~15706006,197132288~11181907,214907306~13381678,229463800~12165696,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,352321536~1368400,356515840~8223218,364955231~4891605,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.542 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to 19 diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.542 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.548 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~10271348,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,197132288~11181907,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.548 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to 9 diff [14137310~7336653,46931142~7840484,153465260~2291357,199192089~1285316,214907306~4822959,235471051~6158445,272649616~7204210,342231213~2195203,404966874~7561094] 2024-08-17T17:47:33.548 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [14137310~7336653,46931142~7840484,153465260~2291357,199192089~1285316,214907306~4822959,235471051~6158445,272649616~7204210,342231213~2195203,404966874~7561094] 2024-08-17T17:47:33.553 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [14137310~7336653,46931142~7840484,153465260~2291357,197132288~3345117,214907306~4822959,235471051~6158445,272649616~7204210,342231213~2195203,404966874~7561094] 2024-08-17T17:47:33.553 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to 10 diff [14137310~7336653,46931142~7840484,104856556~4244827,135726836~4282324,153465260~5730852,187306936~599389,199192089~1285316,200751176~7563019,214907306~4822959,235471051~6158445,257231876~3214859,272649616~7204210,284082763~2045905,290745721~4774032,299299990~6835465,342231213~2195203,404966874~7561094] 2024-08-17T17:47:33.553 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [14137310~7336653,46931142~7840484,104856556~4244827,135726836~4282324,153465260~5730852,187306936~599389,199192089~1285316,200751176~7563019,214907306~4822959,235471051~6158445,257231876~3214859,272649616~7204210,284082763~2045905,290745721~4774032,299299990~6835465,342231213~2195203,404966874~7561094] 2024-08-17T17:47:33.559 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [14137310~7336653,46931142~7840484,103869282~5232101,135726836~4282324,153465260~5730852,187306936~599389,197132288~11181907,214907306~4822959,235471051~6158445,257231876~3214859,272649616~7204210,281018368~5110300,289406976~6112777,299299990~6835465,342231213~2195203,404966874~7561094] 2024-08-17T17:47:33.559 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to 11 diff [14137310~7336653,43810632~10960994,56731919~6095770,64609489~4058806,76085135~2537370,98515291~4618384,104856556~4244827,126663946~3417085,135726836~4282324,153465260~5730852,187306936~599389,199192089~1285316,200751176~7563019,214907306~10694908,235471051~6158445,257231876~3214859,272649616~7204210,284082763~2045905,290745721~4774032,299299990~6835465,342231213~2195203,345979605~2021847,378717919~7112881,389343167~1935343,404966874~7561094] 2024-08-17T17:47:33.559 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [14137310~7336653,43810632~10960994,56731919~6095770,64609489~4058806,76085135~2537370,98515291~4618384,104856556~4244827,126663946~3417085,135726836~4282324,153465260~5730852,187306936~599389,199192089~1285316,200751176~7563019,214907306~10694908,235471051~6158445,257231876~3214859,272649616~7204210,284082763~2045905,290745721~4774032,299299990~6835465,342231213~2195203,345979605~2021847,378717919~7112881,389343167~1935343,404966874~7561094] 2024-08-17T17:47:33.564 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [14137310~7336653,43810632~10960994,56731919~6095770,64609489~4058806,75497472~3125033,98515291~4618384,103869282~5232101,126663946~3417085,135726836~4282324,153465260~5730852,187306936~599389,197132288~11181907,214907306~10694908,235471051~6158445,257231876~3214859,272649616~7204210,281018368~5110300,289406976~6112777,299299990~6835465,342231213~2195203,345979605~2021847,378717919~7112881,389343167~1935343,404966874~7561094] 2024-08-17T17:47:33.565 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to 12 diff [3528496~4240299,14137310~7336653,43810632~10960994,56731919~6095770,64609489~4058806,72723682~5898823,98515291~4618384,104856556~4244827,126663946~3417085,135726836~4282324,146925241~12270871,186686136~6473436,199192089~1285316,200751176~7563019,214907306~10694908,235471051~6158445,257231876~3214859,272649616~13479052,290745721~4774032,299299990~6835465,327775763~6410563,342231213~2195203,345979605~2021847,357458885~7280173,378717919~7112881,389343167~1935343,404966874~7561094] 2024-08-17T17:47:33.565 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [3528496~4240299,14137310~7336653,43810632~10960994,56731919~6095770,64609489~4058806,72723682~5898823,98515291~4618384,104856556~4244827,126663946~3417085,135726836~4282324,146925241~12270871,186686136~6473436,199192089~1285316,200751176~7563019,214907306~10694908,235471051~6158445,257231876~3214859,272649616~13479052,290745721~4774032,299299990~6835465,327775763~6410563,342231213~2195203,345979605~2021847,357458885~7280173,378717919~7112881,389343167~1935343,404966874~7561094] 2024-08-17T17:47:33.570 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,14137310~7336653,43810632~10960994,56731919~6095770,64609489~4058806,72723682~5898823,98515291~4618384,103869282~5232101,126663946~3417085,135726836~4282324,146925241~12270871,186686136~6473436,197132288~11181907,214907306~10694908,235471051~6158445,257231876~3214859,272649616~13479052,289406976~6112777,299299990~6835465,327775763~6410563,342231213~2195203,345979605~2021847,356515840~8223218,378717919~7112881,389343167~1935343,404966874~7561094] 2024-08-17T17:47:33.570 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to 13 diff [3528496~4240299,14137310~7336653,43111238~11660388,56731919~6095770,64609489~4058806,72723682~5898823,97918767~5214908,104856556~4244827,126663946~3417085,135726836~4282324,146925241~12270871,186686136~6473436,199192089~1285316,200751176~7563019,214907306~10694908,229463800~12165696,257231876~3214859,272649616~14653854,290745721~4774032,299299990~7490602,327775763~6410563,342231213~2195203,345979605~2021847,357458885~7280173,372530391~13300409,389343167~1935343,404966874~7561094] 2024-08-17T17:47:33.570 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [3528496~4240299,14137310~7336653,43111238~11660388,56731919~6095770,64609489~4058806,72723682~5898823,97918767~5214908,104856556~4244827,126663946~3417085,135726836~4282324,146925241~12270871,186686136~6473436,199192089~1285316,200751176~7563019,214907306~10694908,229463800~12165696,257231876~3214859,272649616~14653854,290745721~4774032,299299990~7490602,327775763~6410563,342231213~2195203,345979605~2021847,357458885~7280173,372530391~13300409,389343167~1935343,404966874~7561094] 2024-08-17T17:47:33.575 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,14137310~7336653,43111238~11660388,56731919~6095770,64609489~4058806,72723682~5898823,97918767~5214908,103869282~5232101,126663946~3417085,135726836~4282324,146925241~12270871,186686136~6473436,197132288~11181907,214907306~10694908,229463800~12165696,257231876~3214859,272649616~14653854,289406976~6112777,299299990~7490602,327775763~6410563,342231213~2195203,345979605~2021847,356515840~8223218,372530391~13300409,389343167~1935343,404966874~7561094] 2024-08-17T17:47:33.576 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to 14 diff [1598369~392670,3528496~4240299,14137310~7336653,43111238~11660388,56731919~21890586,97918767~5214908,104856556~4244827,126663946~3417085,135726836~4282324,146925241~12270871,186686136~6473436,199192089~1285316,200751176~7563019,214907306~10694908,229463800~12165696,249341666~4308983,257231876~3214859,272649616~14653854,290745721~4774032,299299990~7490602,327775763~6410563,339992078~4434338,345979605~2021847,357458885~7280173,372530391~13300409,389343167~1935343,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.576 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~4240299,14137310~7336653,43111238~11660388,56731919~21890586,97918767~5214908,104856556~4244827,126663946~3417085,135726836~4282324,146925241~12270871,186686136~6473436,199192089~1285316,200751176~7563019,214907306~10694908,229463800~12165696,249341666~4308983,257231876~3214859,272649616~14653854,290745721~4774032,299299990~7490602,327775763~6410563,339992078~4434338,345979605~2021847,357458885~7280173,372530391~13300409,389343167~1935343,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.581 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,14137310~7336653,43111238~11660388,56731919~21890586,97918767~5214908,103869282~5232101,126663946~3417085,135726836~4282324,146925241~12270871,186686136~6473436,197132288~11181907,214907306~10694908,229463800~12165696,249341666~4308983,257231876~3214859,272649616~14653854,289406976~6112777,299299990~7490602,327775763~6410563,339992078~4434338,345979605~2021847,356515840~8223218,372530391~13300409,389343167~1935343,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.581 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to 15 diff [1598369~392670,3528496~4240299,14137310~7336653,31126039~7804481,43111238~35511267,97918767~5214908,104856556~4244827,126663946~3417085,135726836~4282324,146925241~12270871,186686136~6473436,199192089~1285316,200751176~7563019,214907306~10694908,229463800~12165696,249341666~11105069,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339992078~4434338,345979605~2021847,357458885~7280173,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.581 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~4240299,14137310~7336653,31126039~7804481,43111238~35511267,97918767~5214908,104856556~4244827,126663946~3417085,135726836~4282324,146925241~12270871,186686136~6473436,199192089~1285316,200751176~7563019,214907306~10694908,229463800~12165696,249341666~11105069,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339992078~4434338,345979605~2021847,357458885~7280173,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.586 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,14137310~7336653,31126039~7804481,43111238~35511267,97918767~5214908,103869282~5232101,126663946~3417085,135726836~4282324,146925241~12270871,186686136~6473436,197132288~11181907,214907306~10694908,229463800~12165696,249341666~11105069,270348712~16954758,289406976~6112777,299299990~7490602,327775763~6410563,339992078~4434338,345979605~2021847,356515840~8223218,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.586 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to 16 diff [1598369~392670,3528496~8378554,13709905~7764058,31126039~7804481,43111238~35511267,83664186~7312323,97918767~5214908,104378282~4723101,126663946~3417085,135726836~23469276,186686136~6473436,199192089~9122106,214907306~10694908,229463800~12165696,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339992078~4434338,345979605~2021847,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.587 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7764058,31126039~7804481,43111238~35511267,83664186~7312323,97918767~5214908,104378282~4723101,126663946~3417085,135726836~23469276,186686136~6473436,199192089~9122106,214907306~10694908,229463800~12165696,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339992078~4434338,345979605~2021847,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.592 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7764058,31126039~7804481,43111238~35511267,83664186~7312323,97918767~5214908,103869282~5232101,126663946~3417085,135726836~23469276,186686136~6473436,197132288~11181907,214907306~10694908,229463800~12165696,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,327775763~6410563,339992078~4434338,345979605~2021847,356515840~8223218,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.592 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to 17 diff [1598369~392670,3528496~8378554,13709905~7986367,31126039~7804481,43111238~37003803,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,179399560~3642611,186686136~6473436,199192089~9122106,214907306~13381678,229463800~12165696,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339992078~4434338,345408086~4753518,353675391~14545,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.592 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,31126039~7804481,43111238~37003803,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,179399560~3642611,186686136~6473436,199192089~9122106,214907306~13381678,229463800~12165696,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339992078~4434338,345408086~4753518,353675391~14545,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.597 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,31126039~7804481,43111238~37003803,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,179399560~3642611,186686136~6473436,197132288~11181907,214907306~13381678,229463800~12165696,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,327775763~6410563,339992078~4434338,345408086~4753518,352321536~1368400,356515840~8223218,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.597 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to 18 diff [1598369~392670,3528496~8378554,13709905~7986367,31126039~7804481,43111238~37026997,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,179399560~3642611,186686136~8419430,199192089~9122106,214907306~13381678,229463800~12165696,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.597 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,31126039~7804481,43111238~37026997,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,179399560~3642611,186686136~8419430,199192089~9122106,214907306~13381678,229463800~12165696,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.602 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,31126039~7804481,43111238~37026997,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,179399560~3642611,186686136~8419430,197132288~11181907,214907306~13381678,229463800~12165696,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,327775763~6410563,339854108~10307496,352321536~1368400,356515840~8223218,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.603 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to 19 diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.603 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.608 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,197132288~11181907,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:33.608 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to 10 diff [104856556~4244827,135726836~4282324,153474666~5721446,187306936~599389,200751176~7563019,257231876~3214859,284082763~2045905,290745721~4774032,299299990~6835465] 2024-08-17T17:47:33.608 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [104856556~4244827,135726836~4282324,153474666~5721446,187306936~599389,200751176~7563019,257231876~3214859,284082763~2045905,290745721~4774032,299299990~6835465] 2024-08-17T17:47:33.614 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [103869282~5232101,135726836~4282324,153474666~5721446,187306936~599389,200477405~7836790,257231876~3214859,281018368~5110300,289406976~6112777,299299990~6835465] 2024-08-17T17:47:33.614 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to 11 diff [43810632~6570505,56731919~6095770,64609489~4058806,76085135~2537370,98515291~4618384,104856556~4244827,126663946~3417085,135726836~4282324,153474666~5721446,187306936~599389,200751176~7563019,217564579~8037635,257231876~3214859,284082763~2045905,290745721~4774032,299299990~6835465,345979605~2021847,378717919~7112881,389343167~1935343] 2024-08-17T17:47:33.614 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [43810632~6570505,56731919~6095770,64609489~4058806,76085135~2537370,98515291~4618384,104856556~4244827,126663946~3417085,135726836~4282324,153474666~5721446,187306936~599389,200751176~7563019,217564579~8037635,257231876~3214859,284082763~2045905,290745721~4774032,299299990~6835465,345979605~2021847,378717919~7112881,389343167~1935343] 2024-08-17T17:47:33.619 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [43810632~6570505,56731919~6095770,64609489~4058806,75497472~3125033,98515291~4618384,103869282~5232101,126663946~3417085,135726836~4282324,153474666~5721446,187306936~599389,200477405~7836790,217564579~8037635,257231876~3214859,281018368~5110300,289406976~6112777,299299990~6835465,345979605~2021847,378717919~7112881,389343167~1935343] 2024-08-17T17:47:33.619 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to 12 diff [3528496~4240299,43810632~6570505,56731919~6095770,64609489~4058806,72723682~5898823,98515291~4618384,104856556~4244827,126663946~3417085,135726836~4282324,146925241~12270871,186686136~6473436,200751176~7563019,217564579~8037635,257231876~3214859,279830462~6298206,290745721~4774032,299299990~6835465,327775763~6410563,345979605~2021847,357458885~7280173,378717919~7112881,389343167~1935343] 2024-08-17T17:47:33.620 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [3528496~4240299,43810632~6570505,56731919~6095770,64609489~4058806,72723682~5898823,98515291~4618384,104856556~4244827,126663946~3417085,135726836~4282324,146925241~12270871,186686136~6473436,200751176~7563019,217564579~8037635,257231876~3214859,279830462~6298206,290745721~4774032,299299990~6835465,327775763~6410563,345979605~2021847,357458885~7280173,378717919~7112881,389343167~1935343] 2024-08-17T17:47:33.625 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,43810632~6570505,56731919~6095770,64609489~4058806,72723682~5898823,98515291~4618384,103869282~5232101,126663946~3417085,135726836~4282324,146925241~12270871,186686136~6473436,200477405~7836790,217564579~8037635,257231876~3214859,279830462~6298206,289406976~6112777,299299990~6835465,327775763~6410563,345979605~2021847,356515840~8223218,378717919~7112881,389343167~1935343] 2024-08-17T17:47:33.625 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to 13 diff [3528496~4240299,43111238~7269899,56731919~6095770,64609489~4058806,72723682~5898823,97918767~5214908,104856556~4244827,126663946~3417085,135726836~4282324,146925241~12270871,186686136~6473436,200751176~7563019,217564579~8037635,229463800~6605226,257231876~3214859,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,345979605~2021847,357458885~7280173,372530391~13300409,389343167~1935343] 2024-08-17T17:47:33.625 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [3528496~4240299,43111238~7269899,56731919~6095770,64609489~4058806,72723682~5898823,97918767~5214908,104856556~4244827,126663946~3417085,135726836~4282324,146925241~12270871,186686136~6473436,200751176~7563019,217564579~8037635,229463800~6605226,257231876~3214859,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,345979605~2021847,357458885~7280173,372530391~13300409,389343167~1935343] 2024-08-17T17:47:33.631 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,43111238~7269899,56731919~6095770,64609489~4058806,72723682~5898823,97918767~5214908,103869282~5232101,126663946~3417085,135726836~4282324,146925241~12270871,186686136~6473436,200477405~7836790,217564579~8037635,229463800~6605226,257231876~3214859,279830462~7473008,289406976~6112777,299299990~7490602,327775763~6410563,345979605~2021847,356515840~8223218,372530391~13300409,389343167~1935343] 2024-08-17T17:47:33.631 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to 14 diff [1598369~392670,3528496~4240299,43111238~7269899,56731919~21890586,97918767~5214908,104856556~4244827,126663946~3417085,135726836~4282324,146925241~12270871,186686136~6473436,200751176~7563019,216394763~9207451,229463800~6605226,249341666~4308983,257231876~3214859,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339992078~3460192,345979605~2021847,357458885~7280173,372530391~13300409,389343167~1935343,397711277~3779285] 2024-08-17T17:47:33.631 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~4240299,43111238~7269899,56731919~21890586,97918767~5214908,104856556~4244827,126663946~3417085,135726836~4282324,146925241~12270871,186686136~6473436,200751176~7563019,216394763~9207451,229463800~6605226,249341666~4308983,257231876~3214859,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339992078~3460192,345979605~2021847,357458885~7280173,372530391~13300409,389343167~1935343,397711277~3779285] 2024-08-17T17:47:33.636 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,43111238~7269899,56731919~21890586,97918767~5214908,103869282~5232101,126663946~3417085,135726836~4282324,146925241~12270871,186686136~6473436,200477405~7836790,216394763~9207451,229463800~6605226,249341666~4308983,257231876~3214859,279830462~7473008,289406976~6112777,299299990~7490602,327775763~6410563,339992078~3460192,345979605~2021847,356515840~8223218,372530391~13300409,389343167~1935343,397711277~3779285] 2024-08-17T17:47:33.636 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to 15 diff [1598369~392670,3528496~4240299,18178412~2116490,31126039~7804481,43111238~7269899,54625176~23997329,97918767~5214908,104856556~4244827,126663946~3417085,135726836~4282324,146925241~12270871,186686136~6473436,200751176~7563019,216394763~9207451,229463800~6605226,249341666~11105069,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339992078~3460192,345979605~2021847,357458885~7280173,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.636 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~4240299,18178412~2116490,31126039~7804481,43111238~7269899,54625176~23997329,97918767~5214908,104856556~4244827,126663946~3417085,135726836~4282324,146925241~12270871,186686136~6473436,200751176~7563019,216394763~9207451,229463800~6605226,249341666~11105069,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339992078~3460192,345979605~2021847,357458885~7280173,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.642 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,18178412~2116490,31126039~7804481,43111238~7269899,54625176~23997329,97918767~5214908,103869282~5232101,126663946~3417085,135726836~4282324,146925241~12270871,186686136~6473436,200477405~7836790,216394763~9207451,229463800~6605226,249341666~11105069,270348712~4014591,279830462~7473008,289406976~6112777,299299990~7490602,327775763~6410563,339992078~3460192,345979605~2021847,356515840~8223218,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.642 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to 16 diff [1598369~392670,3528496~8378554,13709905~6584997,31126039~7804481,43111238~7269899,54625176~23997329,83664186~7312323,97918767~5214908,104378282~4723101,126663946~3417085,135726836~23469276,186686136~6473436,199871847~8442348,216394763~9207451,229463800~6605226,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339992078~3460192,345979605~2021847,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.642 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~6584997,31126039~7804481,43111238~7269899,54625176~23997329,83664186~7312323,97918767~5214908,104378282~4723101,126663946~3417085,135726836~23469276,186686136~6473436,199871847~8442348,216394763~9207451,229463800~6605226,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339992078~3460192,345979605~2021847,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.647 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~6584997,31126039~7804481,43111238~7269899,54625176~23997329,83664186~7312323,97918767~5214908,103869282~5232101,126663946~3417085,135726836~23469276,186686136~6473436,199871847~8442348,216394763~9207451,229463800~6605226,249341666~20163930,270348712~4014591,279830462~7473008,289406976~6112777,299299990~7490602,327775763~6410563,339992078~3460192,345979605~2021847,356515840~8223218,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.647 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to 17 diff [1598369~392670,3528496~8378554,13709905~7986367,31126039~7804481,43111238~7269899,54625176~25489865,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,179399560~3642611,186686136~6473436,199871847~8442348,216394763~11894221,229463800~6605226,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339992078~3460192,345408086~4753518,353675391~14545,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.647 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,31126039~7804481,43111238~7269899,54625176~25489865,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,179399560~3642611,186686136~6473436,199871847~8442348,216394763~11894221,229463800~6605226,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339992078~3460192,345408086~4753518,353675391~14545,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.653 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,31126039~7804481,43111238~7269899,54625176~25489865,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,179399560~3642611,186686136~6473436,199871847~8442348,216394763~11894221,229463800~6605226,249341666~20163930,270348712~4014591,279830462~7473008,289406976~6112777,299299990~7490602,327775763~6410563,339992078~3460192,345408086~4753518,352321536~1368400,356515840~8223218,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.653 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to 18 diff [1598369~392670,3528496~8378554,13709905~7986367,31126039~7804481,43111238~7269899,54625176~25513059,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,179399560~3642611,186686136~8419430,199871847~8442348,216394763~11894221,229463800~6605226,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.653 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,31126039~7804481,43111238~7269899,54625176~25513059,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,179399560~3642611,186686136~8419430,199871847~8442348,216394763~11894221,229463800~6605226,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.659 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,31126039~7804481,43111238~7269899,54625176~25513059,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,179399560~3642611,186686136~8419430,199871847~8442348,216394763~11894221,229463800~6605226,249341666~20163930,270348712~4014591,279830462~7473008,289406976~6112777,299299990~7490602,327775763~6410563,339854108~10307496,352321536~1368400,356515840~8223218,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.659 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to 19 diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.659 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.664 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~20163930,270348712~4014591,279830462~7473008,289406976~6112777,299299990~7490602,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.664 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to 11 diff [43810632~6570505,56731919~6095770,64609489~4058806,76085135~2537370,98515291~4618384,126663946~3417085,217564579~8037635,345979605~2021847,378717919~7112881,389343167~1935343] 2024-08-17T17:47:33.664 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [43810632~6570505,56731919~6095770,64609489~4058806,76085135~2537370,98515291~4618384,126663946~3417085,217564579~8037635,345979605~2021847,378717919~7112881,389343167~1935343] 2024-08-17T17:47:33.670 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [43810632~6570505,56731919~6095770,64609489~4058806,75497472~3125033,98515291~4618384,126663946~3417085,217564579~8037635,345979605~2021847,378717919~7112881,389343167~1935343] 2024-08-17T17:47:33.670 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to 12 diff [3528496~4240299,43810632~6570505,56731919~6095770,64609489~4058806,72723682~5898823,98515291~4618384,126663946~3417085,146925241~7131805,156994941~1404463,186686136~6473436,217564579~8037635,279830462~4729102,327775763~6410563,345979605~2021847,357458885~7280173,378717919~7112881,389343167~1935343] 2024-08-17T17:47:33.670 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [3528496~4240299,43810632~6570505,56731919~6095770,64609489~4058806,72723682~5898823,98515291~4618384,126663946~3417085,146925241~7131805,156994941~1404463,186686136~6473436,217564579~8037635,279830462~4729102,327775763~6410563,345979605~2021847,357458885~7280173,378717919~7112881,389343167~1935343] 2024-08-17T17:47:33.675 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,43810632~6570505,56731919~6095770,64609489~4058806,72723682~5898823,98515291~4618384,126663946~3417085,146925241~7131805,156994941~1404463,186686136~6473436,217564579~8037635,279830462~4729102,327775763~6410563,345979605~2021847,356515840~8223218,378717919~7112881,389343167~1935343] 2024-08-17T17:47:33.676 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to 13 diff [3528496~4240299,43111238~7269899,56731919~6095770,64609489~4058806,72723682~5898823,97918767~5214908,126663946~3417085,146925241~7131805,156994941~1404463,186686136~6473436,201183648~1421815,217564579~8037635,229463800~6605226,279830462~4729102,285018335~2285135,301620409~5170183,327775763~6410563,345979605~2021847,357458885~7280173,372530391~13300409,389343167~1935343] 2024-08-17T17:47:33.676 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [3528496~4240299,43111238~7269899,56731919~6095770,64609489~4058806,72723682~5898823,97918767~5214908,126663946~3417085,146925241~7131805,156994941~1404463,186686136~6473436,201183648~1421815,217564579~8037635,229463800~6605226,279830462~4729102,285018335~2285135,301620409~5170183,327775763~6410563,345979605~2021847,357458885~7280173,372530391~13300409,389343167~1935343] 2024-08-17T17:47:33.681 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,43111238~7269899,56731919~6095770,64609489~4058806,72723682~5898823,97918767~5214908,126663946~3417085,146925241~7131805,156994941~1404463,186686136~6473436,201183648~1421815,217564579~8037635,229463800~6605226,279830462~4729102,285018335~2285135,301620409~5170183,327775763~6410563,345979605~2021847,356515840~8223218,372530391~13300409,389343167~1935343] 2024-08-17T17:47:33.681 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to 14 diff [1598369~392670,3528496~4240299,43111238~7269899,56731919~21890586,97918767~5214908,126663946~3417085,146925241~11474163,186686136~6473436,201183648~1421815,216394763~9207451,229463800~6605226,249341666~4308983,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339992078~3460192,345979605~2021847,357458885~7280173,372530391~13300409,389343167~1935343,397711277~3779285] 2024-08-17T17:47:33.681 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~4240299,43111238~7269899,56731919~21890586,97918767~5214908,126663946~3417085,146925241~11474163,186686136~6473436,201183648~1421815,216394763~9207451,229463800~6605226,249341666~4308983,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339992078~3460192,345979605~2021847,357458885~7280173,372530391~13300409,389343167~1935343,397711277~3779285] 2024-08-17T17:47:33.686 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,43111238~7269899,56731919~21890586,97918767~5214908,126663946~3417085,146925241~11474163,186686136~6473436,201183648~1421815,216394763~9207451,229463800~6605226,249341666~4308983,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339992078~3460192,345979605~2021847,356515840~8223218,372530391~13300409,389343167~1935343,397711277~3779285] 2024-08-17T17:47:33.686 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to 15 diff [1598369~392670,3528496~4240299,18178412~2116490,31126039~7804481,43111238~7269899,54625176~23997329,97918767~5214908,126663946~3417085,146925241~11474163,186686136~6473436,201183648~1421815,216394763~9207451,229463800~6605226,249341666~10488912,270348712~4014591,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339992078~3460192,345979605~2021847,357458885~7280173,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.687 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~4240299,18178412~2116490,31126039~7804481,43111238~7269899,54625176~23997329,97918767~5214908,126663946~3417085,146925241~11474163,186686136~6473436,201183648~1421815,216394763~9207451,229463800~6605226,249341666~10488912,270348712~4014591,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339992078~3460192,345979605~2021847,357458885~7280173,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.692 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,18178412~2116490,31126039~7804481,43111238~7269899,54625176~23997329,97918767~5214908,126663946~3417085,146925241~11474163,186686136~6473436,201183648~1421815,216394763~9207451,229463800~6605226,249341666~10488912,270348712~4014591,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339992078~3460192,345979605~2021847,356515840~8223218,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.692 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to 16 diff [1598369~392670,3528496~8378554,13709905~6584997,31126039~7804481,43111238~7269899,54625176~23997329,83664186~7312323,97918767~5214908,104378282~1274498,126663946~3417085,137168385~21231019,186686136~6473436,199871847~7240303,216394763~9207451,229463800~6605226,249341666~10488912,263839725~5665871,270348712~4014591,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339992078~3460192,345979605~2021847,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.692 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~6584997,31126039~7804481,43111238~7269899,54625176~23997329,83664186~7312323,97918767~5214908,104378282~1274498,126663946~3417085,137168385~21231019,186686136~6473436,199871847~7240303,216394763~9207451,229463800~6605226,249341666~10488912,263839725~5665871,270348712~4014591,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339992078~3460192,345979605~2021847,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.697 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~6584997,31126039~7804481,43111238~7269899,54625176~23997329,83664186~7312323,97918767~5214908,104378282~1274498,126663946~3417085,137168385~21231019,186686136~6473436,199871847~7240303,216394763~9207451,229463800~6605226,249341666~10488912,260446735~9058861,270348712~4014591,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339992078~3460192,345979605~2021847,356515840~8223218,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.698 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to 17 diff [1598369~392670,3528496~8378554,13709905~7986367,31126039~7804481,43111238~7269899,54625176~25489865,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,179399560~3642611,186686136~6473436,199871847~7240303,216394763~11894221,229463800~6605226,249341666~10488912,263839725~5665871,270348712~4014591,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339992078~3460192,345408086~4753518,353675391~14545,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.698 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,31126039~7804481,43111238~7269899,54625176~25489865,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,179399560~3642611,186686136~6473436,199871847~7240303,216394763~11894221,229463800~6605226,249341666~10488912,263839725~5665871,270348712~4014591,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339992078~3460192,345408086~4753518,353675391~14545,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.703 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,31126039~7804481,43111238~7269899,54625176~25489865,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,179399560~3642611,186686136~6473436,199871847~7240303,216394763~11894221,229463800~6605226,249341666~10488912,260446735~9058861,270348712~4014591,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339992078~3460192,345408086~4753518,352321536~1368400,356515840~8223218,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.703 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to 18 diff [1598369~392670,3528496~8378554,13709905~7986367,31126039~7804481,43111238~7269899,54625176~25513059,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,179399560~3642611,186686136~8419430,199871847~8345189,216394763~11894221,229463800~6605226,249341666~10488912,263839725~5665871,270348712~4014591,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.703 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,31126039~7804481,43111238~7269899,54625176~25513059,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,179399560~3642611,186686136~8419430,199871847~8345189,216394763~11894221,229463800~6605226,249341666~10488912,263839725~5665871,270348712~4014591,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~7280173,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.709 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,31126039~7804481,43111238~7269899,54625176~25513059,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,179399560~3642611,186686136~8419430,199871847~8345189,216394763~11894221,229463800~6605226,249341666~10488912,260446735~9058861,270348712~4014591,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339854108~10307496,352321536~1368400,356515840~8223218,364955231~4891605,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.709 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to 19 diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.709 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.715 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.715 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to 12 diff [3528496~4240299,72723682~4360656,146925241~7131805,156994941~1404463,186686136~6473436,279830462~4729102,327775763~6410563,357458885~7280173] 2024-08-17T17:47:33.715 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [3528496~4240299,72723682~4360656,146925241~7131805,156994941~1404463,186686136~6473436,279830462~4729102,327775763~6410563,357458885~7280173] 2024-08-17T17:47:33.720 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,72723682~4360656,146925241~7131805,156994941~1404463,186686136~6473436,279830462~4729102,327775763~6410563,356515840~8223218] 2024-08-17T17:47:33.720 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to 13 diff [3528496~4240299,43111238~709358,72723682~4360656,97918767~799692,146925241~7131805,156994941~1404463,186686136~6473436,201183648~1421815,229463800~6605226,279830462~4729102,285018335~2285135,301620409~5170183,327775763~6410563,357458885~7280173,372530391~10621123] 2024-08-17T17:47:33.720 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [3528496~4240299,43111238~709358,72723682~4360656,97918767~799692,146925241~7131805,156994941~1404463,186686136~6473436,201183648~1421815,229463800~6605226,279830462~4729102,285018335~2285135,301620409~5170183,327775763~6410563,357458885~7280173,372530391~10621123] 2024-08-17T17:47:33.726 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,43111238~709358,72723682~4360656,97918767~799692,146925241~7131805,156994941~1404463,186686136~6473436,201183648~1421815,229463800~6605226,279830462~4729102,285018335~2285135,301620409~5170183,327775763~6410563,356515840~8223218,372530391~10621123] 2024-08-17T17:47:33.726 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to 14 diff [1598369~392670,3528496~4240299,43111238~709358,60835320~6300977,68158340~8925998,97918767~799692,146925241~11474163,186686136~6473436,201183648~1421815,216394763~3014593,229463800~6605226,249341666~4308983,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339992078~3460192,357458885~7280173,372530391~10621123,397711277~3779285] 2024-08-17T17:47:33.726 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~4240299,43111238~709358,60835320~6300977,68158340~8925998,97918767~799692,146925241~11474163,186686136~6473436,201183648~1421815,216394763~3014593,229463800~6605226,249341666~4308983,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339992078~3460192,357458885~7280173,372530391~10621123,397711277~3779285] 2024-08-17T17:47:33.732 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,43111238~709358,60835320~6300977,68158340~8925998,97918767~799692,146925241~11474163,186686136~6473436,201183648~1421815,216394763~3014593,229463800~6605226,249341666~4308983,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339992078~3460192,356515840~8223218,372530391~10621123,397711277~3779285] 2024-08-17T17:47:33.732 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to 15 diff [1598369~392670,3528496~4240299,18178412~2116490,31126039~7804481,43111238~709358,54625176~22459162,97918767~799692,146925241~11474163,186686136~6473436,201183648~1421815,216394763~3014593,229463800~6605226,249341666~10488912,270348712~4014591,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339992078~3460192,357458885~7280173,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.732 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~4240299,18178412~2116490,31126039~7804481,43111238~709358,54625176~22459162,97918767~799692,146925241~11474163,186686136~6473436,201183648~1421815,216394763~3014593,229463800~6605226,249341666~10488912,270348712~4014591,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339992078~3460192,357458885~7280173,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.737 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~7768795,18178412~2116490,31126039~7804481,43111238~709358,54625176~22459162,97918767~799692,146925241~11474163,186686136~6473436,201183648~1421815,216394763~3014593,229463800~6605226,249341666~10488912,270348712~4014591,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339992078~3460192,356515840~8223218,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.737 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to 16 diff [1598369~392670,3528496~8378554,13709905~6584997,31126039~7804481,43111238~709358,54625176~22459162,83664186~7312323,97918767~799692,104378282~1274498,137168385~21231019,186686136~6473436,199871847~7240303,216394763~3014593,229463800~6605226,249341666~10488912,263839725~5665871,270348712~4014591,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339992078~3460192,357458885~7280173,364955231~4891605,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.737 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~6584997,31126039~7804481,43111238~709358,54625176~22459162,83664186~7312323,97918767~799692,104378282~1274498,137168385~21231019,186686136~6473436,199871847~7240303,216394763~3014593,229463800~6605226,249341666~10488912,263839725~5665871,270348712~4014591,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339992078~3460192,357458885~7280173,364955231~4891605,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.743 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~6584997,31126039~7804481,43111238~709358,54625176~22459162,83664186~7312323,97918767~799692,104378282~1274498,137168385~21231019,186686136~6473436,199871847~7240303,216394763~3014593,229463800~6605226,249341666~10488912,260446735~9058861,270348712~4014591,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339992078~3460192,356515840~8223218,364955231~4891605,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.743 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to 17 diff [1598369~392670,3528496~8378554,13709905~7986367,31126039~7804481,43111238~709358,54625176~25489865,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,179399560~3642611,186686136~6473436,199871847~7240303,216394763~3014593,223587612~4701372,229463800~6605226,249341666~10488912,263839725~5665871,270348712~4014591,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339992078~3460192,345408086~4753518,353675391~14545,357458885~7280173,364955231~4891605,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.743 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,31126039~7804481,43111238~709358,54625176~25489865,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,179399560~3642611,186686136~6473436,199871847~7240303,216394763~3014593,223587612~4701372,229463800~6605226,249341666~10488912,263839725~5665871,270348712~4014591,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339992078~3460192,345408086~4753518,353675391~14545,357458885~7280173,364955231~4891605,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.748 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,31126039~7804481,43111238~709358,54625176~25489865,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,179399560~3642611,186686136~6473436,199871847~7240303,216394763~3014593,223587612~4701372,229463800~6605226,249341666~10488912,260446735~9058861,270348712~4014591,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339992078~3460192,345408086~4753518,352321536~1368400,356515840~8223218,364955231~4891605,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.749 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to 18 diff [1598369~392670,3528496~8378554,13709905~7986367,31126039~7804481,43111238~709358,54625176~25513059,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,179399560~3642611,186686136~8419430,199871847~8345189,216394763~6715040,223587612~4701372,229463800~6605226,249341666~10488912,263839725~5665871,270348712~4014591,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~7280173,364955231~4891605,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.749 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,31126039~7804481,43111238~709358,54625176~25513059,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,179399560~3642611,186686136~8419430,199871847~8345189,216394763~6715040,223587612~4701372,229463800~6605226,249341666~10488912,263839725~5665871,270348712~4014591,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~7280173,364955231~4891605,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.754 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,31126039~7804481,43111238~709358,54625176~25513059,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,179399560~3642611,186686136~8419430,199871847~8345189,216394763~6715040,223587612~4701372,229463800~6605226,249341666~10488912,260446735~9058861,270348712~4014591,279830462~4729102,285018335~2285135,295434938~62703,301620409~5170183,327775763~6410563,339854108~10307496,352321536~1368400,356515840~8223218,364955231~4891605,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.755 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to 19 diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.755 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.761 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.761 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to 13 diff [43111238~709358,97918767~799692,201183648~1421815,229463800~6605226,285018335~2285135,301620409~5170183,372530391~10621123] 2024-08-17T17:47:33.761 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [43111238~709358,97918767~799692,201183648~1421815,229463800~6605226,285018335~2285135,301620409~5170183,372530391~10621123] 2024-08-17T17:47:33.766 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [43111238~709358,97918767~799692,201183648~1421815,229463800~6605226,285018335~2285135,301620409~5170183,372530391~10621123] 2024-08-17T17:47:33.766 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to 14 diff [1598369~392670,43111238~709358,60835320~6300977,68158340~5780466,97918767~799692,151816603~6128341,201183648~1421815,216394763~3014593,229463800~6605226,249341666~4308983,285018335~2285135,295434938~62703,301620409~5170183,339992078~3460192,372530391~10621123,397711277~3779285] 2024-08-17T17:47:33.766 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,43111238~709358,60835320~6300977,68158340~5780466,97918767~799692,151816603~6128341,201183648~1421815,216394763~3014593,229463800~6605226,249341666~4308983,285018335~2285135,295434938~62703,301620409~5170183,339992078~3460192,372530391~10621123,397711277~3779285] 2024-08-17T17:47:33.772 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [1598369~392670,43111238~709358,60835320~6300977,68158340~5780466,97918767~799692,151816603~6128341,201183648~1421815,216394763~3014593,229463800~6605226,249341666~4308983,285018335~2285135,295434938~62703,301620409~5170183,339992078~3460192,372530391~10621123,397711277~3779285] 2024-08-17T17:47:33.772 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to 15 diff [1598369~392670,18178412~2116490,31126039~7804481,43111238~709358,54625176~19313630,97918767~799692,151816603~6128341,201183648~1421815,216394763~3014593,229463800~6605226,249341666~10488912,270348712~4014591,285018335~2285135,295434938~62703,301620409~5170183,339992078~3460192,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.772 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,18178412~2116490,31126039~7804481,43111238~709358,54625176~19313630,97918767~799692,151816603~6128341,201183648~1421815,216394763~3014593,229463800~6605226,249341666~10488912,270348712~4014591,285018335~2285135,295434938~62703,301620409~5170183,339992078~3460192,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.778 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [1598369~392670,18178412~2116490,31126039~7804481,43111238~709358,54625176~19313630,97918767~799692,151816603~6128341,201183648~1421815,216394763~3014593,229463800~6605226,249341666~10488912,270348712~4014591,285018335~2285135,295434938~62703,301620409~5170183,339992078~3460192,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.778 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to 16 diff [1598369~392670,4299867~7607183,13709905~6584997,31126039~7804481,43111238~709358,54625176~19313630,83664186~7312323,97918767~799692,104378282~1274498,137168385~12023792,151816603~6128341,199871847~7240303,216394763~3014593,229463800~6605226,249341666~10488912,263839725~5665871,270348712~4014591,285018335~2285135,295434938~62703,301620409~5170183,339992078~3460192,364955231~4891605,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.778 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,4299867~7607183,13709905~6584997,31126039~7804481,43111238~709358,54625176~19313630,83664186~7312323,97918767~799692,104378282~1274498,137168385~12023792,151816603~6128341,199871847~7240303,216394763~3014593,229463800~6605226,249341666~10488912,263839725~5665871,270348712~4014591,285018335~2285135,295434938~62703,301620409~5170183,339992078~3460192,364955231~4891605,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.783 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [1598369~392670,4299867~7607183,13709905~6584997,31126039~7804481,43111238~709358,54625176~19313630,83664186~7312323,97918767~799692,104378282~1274498,137168385~12023792,151816603~6128341,199871847~7240303,216394763~3014593,229463800~6605226,249341666~10488912,260446735~9058861,270348712~4014591,285018335~2285135,295434938~62703,301620409~5170183,339992078~3460192,364955231~4891605,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.783 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to 17 diff [1598369~392670,4299867~7607183,13709905~7986367,31126039~7804481,43111238~709358,54625176~19313630,74032760~6082281,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,179399560~3642611,199871847~7240303,216394763~3014593,223587612~4701372,229463800~6605226,249341666~10488912,263839725~5665871,270348712~4014591,285018335~2285135,295434938~62703,301620409~5170183,339992078~3460192,345408086~4753518,353675391~14545,364955231~4891605,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.783 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,4299867~7607183,13709905~7986367,31126039~7804481,43111238~709358,54625176~19313630,74032760~6082281,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,179399560~3642611,199871847~7240303,216394763~3014593,223587612~4701372,229463800~6605226,249341666~10488912,263839725~5665871,270348712~4014591,285018335~2285135,295434938~62703,301620409~5170183,339992078~3460192,345408086~4753518,353675391~14545,364955231~4891605,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.789 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [1598369~392670,4299867~7607183,13709905~7986367,31126039~7804481,43111238~709358,54625176~19313630,74032760~6082281,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,179399560~3642611,199871847~7240303,216394763~3014593,223587612~4701372,229463800~6605226,249341666~10488912,260446735~9058861,270348712~4014591,285018335~2285135,295434938~62703,301620409~5170183,339992078~3460192,345408086~4753518,352321536~1368400,364955231~4891605,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.789 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to 18 diff [1598369~392670,4299867~7607183,13709905~7986367,31126039~7804481,43111238~709358,54625176~25513059,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,179399560~3642611,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229463800~6605226,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,285018335~2285135,295434938~62703,301620409~5170183,339854108~10307496,353675391~14545,364955231~4891605,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.789 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,4299867~7607183,13709905~7986367,31126039~7804481,43111238~709358,54625176~25513059,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,179399560~3642611,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229463800~6605226,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,285018335~2285135,295434938~62703,301620409~5170183,339854108~10307496,353675391~14545,364955231~4891605,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.794 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [1598369~392670,4299867~7607183,13709905~7986367,31126039~7804481,43111238~709358,54625176~25513059,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,179399560~3642611,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229463800~6605226,249341666~10488912,260446735~9058861,270348712~4014591,280132049~79251,285018335~2285135,295434938~62703,301620409~5170183,339854108~10307496,352321536~1368400,364955231~4891605,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.794 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to 19 diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~6107187,295434938~62703,299679355~367126,301620409~5170183,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.795 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~6107187,295434938~62703,299679355~367126,301620409~5170183,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.801 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,280132049~79251,281196283~6107187,295434938~62703,299679355~367126,301620409~5170183,339854108~10307496,352321536~1368400,359888874~121026,360457183~9389653,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.801 INFO:tasks.workunit.client.0.smithi136.stdout:from 13 to 14 diff [1598369~392670,60835320~6300977,68158340~5780466,151816603~6128341,216394763~3014593,249341666~4308983,295434938~62703,339992078~3460192,397711277~3779285] 2024-08-17T17:47:33.801 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,60835320~6300977,68158340~5780466,151816603~6128341,216394763~3014593,249341666~4308983,295434938~62703,339992078~3460192,397711277~3779285] 2024-08-17T17:47:33.806 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [1598369~392670,60835320~6300977,68158340~5780466,151816603~6128341,216394763~3014593,249341666~4308983,295434938~62703,339992078~3460192,397711277~3779285] 2024-08-17T17:47:33.807 INFO:tasks.workunit.client.0.smithi136.stdout:from 13 to 15 diff [1598369~392670,18178412~2116490,31126039~7804481,54625176~19313630,151816603~6128341,216394763~3014593,249341666~10488912,270348712~4014591,295434938~62703,339992078~3460192,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.807 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,18178412~2116490,31126039~7804481,54625176~19313630,151816603~6128341,216394763~3014593,249341666~10488912,270348712~4014591,295434938~62703,339992078~3460192,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.812 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [1598369~392670,18178412~2116490,31126039~7804481,54625176~19313630,151816603~6128341,216394763~3014593,249341666~10488912,270348712~4014591,295434938~62703,339992078~3460192,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.812 INFO:tasks.workunit.client.0.smithi136.stdout:from 13 to 16 diff [1598369~392670,4299867~7607183,13709905~6584997,31126039~7804481,54625176~19313630,83664186~7312323,104378282~1274498,137168385~12023792,151816603~6128341,199871847~7240303,216394763~3014593,249341666~10488912,263839725~5665871,270348712~4014591,295434938~62703,339992078~3460192,364955231~4891605,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.812 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,4299867~7607183,13709905~6584997,31126039~7804481,54625176~19313630,83664186~7312323,104378282~1274498,137168385~12023792,151816603~6128341,199871847~7240303,216394763~3014593,249341666~10488912,263839725~5665871,270348712~4014591,295434938~62703,339992078~3460192,364955231~4891605,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.817 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [1598369~392670,4299867~7607183,13709905~6584997,31126039~7804481,54625176~19313630,83664186~7312323,104378282~1274498,137168385~12023792,151816603~6128341,199871847~7240303,216394763~3014593,249341666~10488912,260446735~9058861,270348712~4014591,295434938~62703,339992078~3460192,364955231~4891605,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.817 INFO:tasks.workunit.client.0.smithi136.stdout:from 13 to 17 diff [1598369~392670,4299867~7607183,13709905~7986367,31126039~7804481,54625176~19313630,74032760~6082281,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,179399560~3642611,199871847~7240303,216394763~3014593,223587612~4701372,249341666~10488912,263839725~5665871,270348712~4014591,295434938~62703,339992078~3460192,345408086~4753518,353675391~14545,364955231~4891605,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.818 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,4299867~7607183,13709905~7986367,31126039~7804481,54625176~19313630,74032760~6082281,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,179399560~3642611,199871847~7240303,216394763~3014593,223587612~4701372,249341666~10488912,263839725~5665871,270348712~4014591,295434938~62703,339992078~3460192,345408086~4753518,353675391~14545,364955231~4891605,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.823 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [1598369~392670,4299867~7607183,13709905~7986367,31126039~7804481,54625176~19313630,74032760~6082281,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,179399560~3642611,199871847~7240303,216394763~3014593,223587612~4701372,249341666~10488912,260446735~9058861,270348712~4014591,295434938~62703,339992078~3460192,345408086~4753518,352321536~1368400,364955231~4891605,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.823 INFO:tasks.workunit.client.0.smithi136.stdout:from 13 to 18 diff [1598369~392670,4299867~7607183,13709905~7986367,31126039~7804481,54625176~25513059,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,179399560~3642611,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,295434938~62703,339854108~10307496,353675391~14545,364955231~4891605,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.823 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,4299867~7607183,13709905~7986367,31126039~7804481,54625176~25513059,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,179399560~3642611,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,295434938~62703,339854108~10307496,353675391~14545,364955231~4891605,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.828 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [1598369~392670,4299867~7607183,13709905~7986367,31126039~7804481,54625176~25513059,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,179399560~3642611,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,249341666~10488912,260446735~9058861,270348712~4014591,280132049~79251,295434938~62703,339854108~10307496,352321536~1368400,364955231~4891605,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.829 INFO:tasks.workunit.client.0.smithi136.stdout:from 13 to 19 diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,54625176~25513059,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~5188907,295434938~62703,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.829 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,54625176~25513059,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~5188907,295434938~62703,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.834 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,54625176~25513059,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,280132049~79251,281196283~5188907,295434938~62703,299679355~367126,339854108~10307496,352321536~1368400,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:33.834 INFO:tasks.workunit.client.0.smithi136.stdout:from 14 to 15 diff [18178412~2116490,31126039~7804481,54625176~7223321,65607886~7657595,252993055~6837523,270348712~4014591,385974724~5541505,395832430~1285281] 2024-08-17T17:47:33.834 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [18178412~2116490,31126039~7804481,54625176~7223321,65607886~7657595,252993055~6837523,270348712~4014591,385974724~5541505,395832430~1285281] 2024-08-17T17:47:33.840 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [18178412~2116490,31126039~7804481,54625176~7223321,65607886~7657595,252993055~6837523,270348712~4014591,385974724~5541505,395832430~1285281] 2024-08-17T17:47:33.840 INFO:tasks.workunit.client.0.smithi136.stdout:from 14 to 16 diff [4299867~7607183,13709905~6584997,31126039~7804481,54625176~7223321,65607886~7657595,83664186~7312323,104378282~1274498,137168385~12023792,199871847~7240303,252993055~6837523,263839725~5665871,270348712~4014591,364955231~4891605,385974724~5541505,395832430~1285281] 2024-08-17T17:47:33.840 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4299867~7607183,13709905~6584997,31126039~7804481,54625176~7223321,65607886~7657595,83664186~7312323,104378282~1274498,137168385~12023792,199871847~7240303,252993055~6837523,263839725~5665871,270348712~4014591,364955231~4891605,385974724~5541505,395832430~1285281] 2024-08-17T17:47:33.845 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4299867~7607183,13709905~6584997,31126039~7804481,54625176~7223321,65607886~7657595,83664186~7312323,104378282~1274498,137168385~12023792,199871847~7240303,252993055~6837523,260446735~9058861,270348712~4014591,364955231~4891605,385974724~5541505,395832430~1285281] 2024-08-17T17:47:33.846 INFO:tasks.workunit.client.0.smithi136.stdout:from 14 to 17 diff [4299867~7607183,13709905~7986367,31126039~7804481,54625176~8218539,65607886~7657595,74032760~6082281,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,179399560~3642611,199871847~7240303,223587612~4701372,252993055~6837523,263839725~5665871,270348712~4014591,345408086~4753518,353675391~14545,364955231~4891605,385974724~5541505,395832430~1285281] 2024-08-17T17:47:33.846 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4299867~7607183,13709905~7986367,31126039~7804481,54625176~8218539,65607886~7657595,74032760~6082281,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,179399560~3642611,199871847~7240303,223587612~4701372,252993055~6837523,263839725~5665871,270348712~4014591,345408086~4753518,353675391~14545,364955231~4891605,385974724~5541505,395832430~1285281] 2024-08-17T17:47:33.851 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4299867~7607183,13709905~7986367,31126039~7804481,54625176~8218539,65607886~7657595,74032760~6082281,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,179399560~3642611,199871847~7240303,223587612~4701372,252993055~6837523,260446735~9058861,270348712~4014591,345408086~4753518,352321536~1368400,364955231~4891605,385974724~5541505,395832430~1285281] 2024-08-17T17:47:33.851 INFO:tasks.workunit.client.0.smithi136.stdout:from 14 to 18 diff [4299867~7607183,13709905~7986367,31126039~7804481,54625176~8218539,65607886~14530349,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,179399560~3642611,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,252993055~6837523,263839725~5665871,270348712~4014591,280132049~79251,339854108~10307496,353675391~14545,364955231~4891605,385974724~5541505,395832430~1285281] 2024-08-17T17:47:33.851 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4299867~7607183,13709905~7986367,31126039~7804481,54625176~8218539,65607886~14530349,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,179399560~3642611,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,252993055~6837523,263839725~5665871,270348712~4014591,280132049~79251,339854108~10307496,353675391~14545,364955231~4891605,385974724~5541505,395832430~1285281] 2024-08-17T17:47:33.857 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4299867~7607183,13709905~7986367,31126039~7804481,54625176~8218539,65607886~14530349,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,179399560~3642611,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,252993055~6837523,260446735~9058861,270348712~4014591,280132049~79251,339854108~10307496,352321536~1368400,364955231~4891605,385974724~5541505,395832430~1285281] 2024-08-17T17:47:33.857 INFO:tasks.workunit.client.0.smithi136.stdout:from 14 to 19 diff [4299867~7607183,13709905~7986367,30130057~8800463,54625176~8218539,65607886~14530349,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,252993055~6837523,263839725~5665871,270348712~4014591,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281] 2024-08-17T17:47:33.857 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4299867~7607183,13709905~7986367,30130057~8800463,54625176~8218539,65607886~14530349,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,252993055~6837523,263839725~5665871,270348712~4014591,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281] 2024-08-17T17:47:33.862 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4299867~7607183,13709905~7986367,30130057~8800463,54625176~8218539,65607886~14530349,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,252993055~6837523,260446735~9058861,270348712~4014591,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,352321536~1368400,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281] 2024-08-17T17:47:33.862 INFO:tasks.workunit.client.0.smithi136.stdout:from 15 to 16 diff [4299867~7607183,13709905~5440474,83664186~7312323,104378282~1274498,137168385~12023792,199871847~7240303,263839725~5665871,364955231~4891605] 2024-08-17T17:47:33.863 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4299867~7607183,13709905~5440474,83664186~7312323,104378282~1274498,137168385~12023792,199871847~7240303,263839725~5665871,364955231~4891605] 2024-08-17T17:47:33.868 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4299867~7607183,13709905~5440474,83664186~7312323,104378282~1274498,137168385~12023792,199871847~7240303,260446735~9058861,364955231~4891605] 2024-08-17T17:47:33.868 INFO:tasks.workunit.client.0.smithi136.stdout:from 15 to 17 diff [4299867~7607183,13709905~7986367,60476408~2367307,74032760~6082281,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,179399560~3642611,199871847~7240303,223587612~4701372,263839725~5665871,345408086~4753518,353675391~14545,364955231~4891605] 2024-08-17T17:47:33.868 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4299867~7607183,13709905~7986367,60476408~2367307,74032760~6082281,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,179399560~3642611,199871847~7240303,223587612~4701372,263839725~5665871,345408086~4753518,353675391~14545,364955231~4891605] 2024-08-17T17:47:33.874 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4299867~7607183,13709905~7986367,60476408~2367307,74032760~6082281,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,179399560~3642611,199871847~7240303,223587612~4701372,260446735~9058861,345408086~4753518,352321536~1368400,364955231~4891605] 2024-08-17T17:47:33.874 INFO:tasks.workunit.client.0.smithi136.stdout:from 15 to 18 diff [4299867~7607183,13709905~7986367,60476408~2367307,69672745~10465490,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,179399560~3642611,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,263839725~5665871,280132049~79251,339854108~10307496,353675391~14545,364955231~4891605] 2024-08-17T17:47:33.874 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4299867~7607183,13709905~7986367,60476408~2367307,69672745~10465490,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,179399560~3642611,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,263839725~5665871,280132049~79251,339854108~10307496,353675391~14545,364955231~4891605] 2024-08-17T17:47:33.879 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4299867~7607183,13709905~7986367,60476408~2367307,69672745~10465490,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,179399560~3642611,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,260446735~9058861,280132049~79251,339854108~10307496,352321536~1368400,364955231~4891605] 2024-08-17T17:47:33.879 INFO:tasks.workunit.client.0.smithi136.stdout:from 15 to 19 diff [4299867~7607183,13709905~7986367,30130057~7268094,60476408~2367307,69672745~10465490,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,263839725~5665871,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653] 2024-08-17T17:47:33.879 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4299867~7607183,13709905~7986367,30130057~7268094,60476408~2367307,69672745~10465490,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,263839725~5665871,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653] 2024-08-17T17:47:33.885 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4299867~7607183,13709905~7986367,30130057~7268094,60476408~2367307,69672745~10465490,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,260446735~9058861,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,352321536~1368400,359888874~121026,360457183~9389653] 2024-08-17T17:47:33.885 INFO:tasks.workunit.client.0.smithi136.stdout:from 16 to 17 diff [18316322~3379950,60476408~2367307,74032760~6082281,100092746~4080651,168323939~2295785,179399560~3642611,223587612~4701372,345408086~4753518,353675391~14545] 2024-08-17T17:47:33.885 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [18316322~3379950,60476408~2367307,74032760~6082281,100092746~4080651,168323939~2295785,179399560~3642611,223587612~4701372,345408086~4753518,353675391~14545] 2024-08-17T17:47:33.891 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [18316322~3379950,60476408~2367307,74032760~6082281,100092746~4080651,168323939~2295785,179399560~3642611,223587612~4701372,345408086~4753518,352321536~1368400] 2024-08-17T17:47:33.891 INFO:tasks.workunit.client.0.smithi136.stdout:from 16 to 18 diff [15854147~5842125,60476408~2367307,69672745~10465490,88081530~2792272,100092746~4080651,138916670~5337091,168323939~2295785,179399560~3642611,191922365~3183201,202869045~5347991,216722147~6387656,223587612~4701372,280132049~79251,339854108~10307496,353675391~14545] 2024-08-17T17:47:33.891 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [15854147~5842125,60476408~2367307,69672745~10465490,88081530~2792272,100092746~4080651,138916670~5337091,168323939~2295785,179399560~3642611,191922365~3183201,202869045~5347991,216722147~6387656,223587612~4701372,280132049~79251,339854108~10307496,353675391~14545] 2024-08-17T17:47:33.896 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [15854147~5842125,60476408~2367307,69672745~10465490,88081530~2792272,100092746~4080651,138916670~5337091,168323939~2295785,179399560~3642611,191922365~3183201,202869045~5347991,216722147~6387656,223587612~4701372,280132049~79251,339854108~10307496,352321536~1368400] 2024-08-17T17:47:33.896 INFO:tasks.workunit.client.0.smithi136.stdout:from 16 to 19 diff [15854147~5842125,30130057~7268094,60476408~2367307,69672745~10465490,81614709~784962,88081530~2792272,100092746~4080651,138916670~5337091,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,202869045~5347991,216722147~6387656,223587612~4701372,229335339~7013526,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~4536253] 2024-08-17T17:47:33.896 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [15854147~5842125,30130057~7268094,60476408~2367307,69672745~10465490,81614709~784962,88081530~2792272,100092746~4080651,138916670~5337091,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,202869045~5347991,216722147~6387656,223587612~4701372,229335339~7013526,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~4536253] 2024-08-17T17:47:33.901 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [15854147~5842125,30130057~7268094,60476408~2367307,69672745~10465490,81614709~784962,88081530~2792272,100092746~4080651,138916670~5337091,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,202869045~5347991,216722147~6387656,223587612~4701372,229335339~7013526,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,352321536~1368400,359888874~121026,360457183~4536253] 2024-08-17T17:47:33.901 INFO:tasks.workunit.client.0.smithi136.stdout:from 17 to 18 diff [15854147~4971299,69672745~5644370,80009864~128371,88081530~2792272,138916670~5337091,191922365~3183201,202869045~5347991,216722147~6387656,280132049~79251,339854108~7964355] 2024-08-17T17:47:33.901 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [15854147~4971299,69672745~5644370,80009864~128371,88081530~2792272,138916670~5337091,191922365~3183201,202869045~5347991,216722147~6387656,280132049~79251,339854108~7964355] 2024-08-17T17:47:33.907 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [15854147~4971299,69672745~5644370,80009864~128371,88081530~2792272,138916670~5337091,191922365~3183201,202869045~5347991,216722147~6387656,280132049~79251,339854108~7964355] 2024-08-17T17:47:33.907 INFO:tasks.workunit.client.0.smithi136.stdout:from 17 to 19 diff [15854147~4971299,30130057~7268094,69672745~5644370,80009864~128371,81614709~784962,88081530~2792272,138916670~5337091,171913277~2498215,182522330~5465509,191922365~3183201,202869045~5347991,216722147~6387656,229335339~7013526,280132049~79251,281196283~5188907,299679355~367126,339854108~7964355,359888874~121026,360457183~4536253] 2024-08-17T17:47:33.907 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [15854147~4971299,30130057~7268094,69672745~5644370,80009864~128371,81614709~784962,88081530~2792272,138916670~5337091,171913277~2498215,182522330~5465509,191922365~3183201,202869045~5347991,216722147~6387656,229335339~7013526,280132049~79251,281196283~5188907,299679355~367126,339854108~7964355,359888874~121026,360457183~4536253] 2024-08-17T17:47:33.912 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [15854147~4971299,30130057~7268094,69672745~5644370,80009864~128371,81614709~784962,88081530~2792272,138916670~5337091,171116125~3295367,182522330~5465509,191922365~3183201,202869045~5347991,216722147~6387656,229335339~7013526,280132049~79251,281196283~5188907,299679355~367126,339854108~7964355,359888874~121026,360457183~4536253] 2024-08-17T17:47:33.912 INFO:tasks.workunit.client.0.smithi136.stdout:from 18 to 19 diff [30130057~7268094,81614709~784962,140106859~4041063,171913277~2498215,182522330~5465509,229335339~7013526,281196283~5188907,299679355~367126,359888874~121026,360457183~4536253] 2024-08-17T17:47:33.912 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [30130057~7268094,81614709~784962,140106859~4041063,171913277~2498215,182522330~5465509,229335339~7013526,281196283~5188907,299679355~367126,359888874~121026,360457183~4536253] 2024-08-17T17:47:33.917 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [30130057~7268094,81614709~784962,140106859~4041063,171116125~3295367,182522330~5465509,229335339~7013526,281196283~5188907,299679355~367126,359888874~121026,360457183~4536253] 2024-08-17T17:47:34.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:33 smithi136 ceph-mon[109861]: osdmap e1127: 8 total, 8 up, 8 in 2024-08-17T17:47:34.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:33 smithi136 ceph-mon[115078]: osdmap e1127: 8 total, 8 up, 8 in 2024-08-17T17:47:34.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:33 smithi165 ceph-mon[105517]: osdmap e1127: 8 total, 8 up, 8 in 2024-08-17T17:47:34.816 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:34.816 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:34.834 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~38930520,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,213909504~27719992,243280741~44022729,289406976~23380514,314572800~6116289,327775763~6879734,335544320~14617284,352321536~1368400,356515840~56012128] 2024-08-17T17:47:34.834 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:34.835 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:34.840 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404765831~7762137] 2024-08-17T17:47:34.841 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:34.841 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:34.846 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,167772160~8947073,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:34.846 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:34.847 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:34.852 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:34.852 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:34.852 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:34.858 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~42002671,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~29314960,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:34.858 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:34.859 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:34.864 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:34.865 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:34.865 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:34.870 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,179399560~15706006,197132288~12158295,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:34.870 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:34.870 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:34.875 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~10271348,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,197132288~11181907,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:34.875 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:34.876 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:34.881 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,197132288~11181907,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:34.881 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:34.881 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:34.887 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~20163930,270348712~4014591,279830462~7473008,289406976~6112777,299299990~7490602,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:34.887 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:34.887 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:34.892 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:34.892 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:34.892 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:34.898 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:34.898 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to HEAD diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~6107187,295434938~62703,299679355~367126,301620409~5170183,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:34.898 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~6107187,295434938~62703,299679355~367126,301620409~5170183,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:34.903 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,280132049~79251,281196283~6107187,295434938~62703,299679355~367126,301620409~5170183,339854108~10307496,352321536~1368400,359888874~121026,360457183~9389653,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:34.903 INFO:tasks.workunit.client.0.smithi136.stdout:from 13 to HEAD diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,54625176~25513059,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~5188907,295434938~62703,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:34.903 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,54625176~25513059,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~5188907,295434938~62703,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:34.908 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,54625176~25513059,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,280132049~79251,281196283~5188907,295434938~62703,299679355~367126,339854108~10307496,352321536~1368400,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:34.909 INFO:tasks.workunit.client.0.smithi136.stdout:from 14 to HEAD diff [4299867~7607183,13709905~7986367,30130057~8800463,54625176~8218539,65607886~14530349,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,252993055~6837523,263839725~5665871,270348712~4014591,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281] 2024-08-17T17:47:34.909 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4299867~7607183,13709905~7986367,30130057~8800463,54625176~8218539,65607886~14530349,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,252993055~6837523,263839725~5665871,270348712~4014591,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281] 2024-08-17T17:47:34.914 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4299867~7607183,13709905~7986367,30130057~8800463,54625176~8218539,65607886~14530349,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,252993055~6837523,260446735~9058861,270348712~4014591,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,352321536~1368400,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281] 2024-08-17T17:47:34.914 INFO:tasks.workunit.client.0.smithi136.stdout:from 15 to HEAD diff [4299867~7607183,13709905~7986367,30130057~7268094,60476408~2367307,69672745~10465490,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,263839725~5665871,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653] 2024-08-17T17:47:34.914 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4299867~7607183,13709905~7986367,30130057~7268094,60476408~2367307,69672745~10465490,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,263839725~5665871,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653] 2024-08-17T17:47:34.919 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4299867~7607183,13709905~7986367,30130057~7268094,60476408~2367307,69672745~10465490,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,260446735~9058861,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,352321536~1368400,359888874~121026,360457183~9389653] 2024-08-17T17:47:34.919 INFO:tasks.workunit.client.0.smithi136.stdout:from 16 to HEAD diff [15854147~5842125,30130057~7268094,60476408~2367307,69672745~10465490,81614709~784962,88081530~2792272,100092746~4080651,138916670~5337091,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,202869045~5347991,216722147~6387656,223587612~4701372,229335339~7013526,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~4536253] 2024-08-17T17:47:34.919 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [15854147~5842125,30130057~7268094,60476408~2367307,69672745~10465490,81614709~784962,88081530~2792272,100092746~4080651,138916670~5337091,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,202869045~5347991,216722147~6387656,223587612~4701372,229335339~7013526,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~4536253] 2024-08-17T17:47:34.924 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [15854147~5842125,30130057~7268094,60476408~2367307,69672745~10465490,81614709~784962,88081530~2792272,100092746~4080651,138916670~5337091,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,202869045~5347991,216722147~6387656,223587612~4701372,229335339~7013526,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,352321536~1368400,359888874~121026,360457183~4536253] 2024-08-17T17:47:34.925 INFO:tasks.workunit.client.0.smithi136.stdout:from 17 to HEAD diff [15854147~4971299,30130057~7268094,69672745~5644370,80009864~128371,81614709~784962,88081530~2792272,138916670~5337091,171913277~2498215,182522330~5465509,191922365~3183201,202869045~5347991,216722147~6387656,229335339~7013526,280132049~79251,281196283~5188907,299679355~367126,339854108~7964355,359888874~121026,360457183~4536253] 2024-08-17T17:47:34.925 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [15854147~4971299,30130057~7268094,69672745~5644370,80009864~128371,81614709~784962,88081530~2792272,138916670~5337091,171913277~2498215,182522330~5465509,191922365~3183201,202869045~5347991,216722147~6387656,229335339~7013526,280132049~79251,281196283~5188907,299679355~367126,339854108~7964355,359888874~121026,360457183~4536253] 2024-08-17T17:47:34.930 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [15854147~4971299,30130057~7268094,69672745~5644370,80009864~128371,81614709~784962,88081530~2792272,138916670~5337091,171116125~3295367,182522330~5465509,191922365~3183201,202869045~5347991,216722147~6387656,229335339~7013526,280132049~79251,281196283~5188907,299679355~367126,339854108~7964355,359888874~121026,360457183~4536253] 2024-08-17T17:47:35.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:34 smithi136 ceph-mon[109861]: pgmap v1729: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 198 KiB/s rd, 39 MiB/s wr, 308 op/s 2024-08-17T17:47:35.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:34 smithi136 ceph-mon[115078]: pgmap v1729: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 198 KiB/s rd, 39 MiB/s wr, 308 op/s 2024-08-17T17:47:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:34 smithi165 ceph-mon[105517]: pgmap v1729: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 198 KiB/s rd, 39 MiB/s wr, 308 op/s 2024-08-17T17:47:35.824 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:35.824 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:35.836 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~38930520,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,213909504~27719992,243280741~44022729,289406976~23380514,314572800~6116289,327775763~6879734,335544320~14617284,352321536~1368400,356515840~56012128] 2024-08-17T17:47:35.836 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:35.836 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:35.842 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404765831~7762137] 2024-08-17T17:47:35.842 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:35.842 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:35.848 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,167772160~8947073,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:35.848 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:35.848 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:35.854 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:35.854 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:35.854 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:35.860 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~42002671,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~29314960,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:35.860 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:35.860 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:35.866 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:35.866 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:35.866 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:35.871 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,179399560~15706006,197132288~12158295,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:35.872 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:35.872 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:35.877 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~10271348,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,197132288~11181907,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:35.877 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:35.877 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:35.883 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,197132288~11181907,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:35.883 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:35.883 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:35.889 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~20163930,270348712~4014591,279830462~7473008,289406976~6112777,299299990~7490602,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:35.889 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:35.889 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:35.894 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:35.894 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:35.894 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:35.900 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:35.900 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to HEAD diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~6107187,295434938~62703,299679355~367126,301620409~5170183,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:35.900 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~6107187,295434938~62703,299679355~367126,301620409~5170183,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:35.905 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,280132049~79251,281196283~6107187,295434938~62703,299679355~367126,301620409~5170183,339854108~10307496,352321536~1368400,359888874~121026,360457183~9389653,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:35.905 INFO:tasks.workunit.client.0.smithi136.stdout:from 13 to HEAD diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,54625176~25513059,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~5188907,295434938~62703,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:35.906 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,54625176~25513059,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~5188907,295434938~62703,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:35.911 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,54625176~25513059,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,280132049~79251,281196283~5188907,295434938~62703,299679355~367126,339854108~10307496,352321536~1368400,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:35.911 INFO:tasks.workunit.client.0.smithi136.stdout:from 14 to HEAD diff [4299867~7607183,13709905~7986367,30130057~8800463,54625176~8218539,65607886~14530349,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,252993055~6837523,263839725~5665871,270348712~4014591,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281] 2024-08-17T17:47:35.911 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4299867~7607183,13709905~7986367,30130057~8800463,54625176~8218539,65607886~14530349,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,252993055~6837523,263839725~5665871,270348712~4014591,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281] 2024-08-17T17:47:35.916 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4299867~7607183,13709905~7986367,30130057~8800463,54625176~8218539,65607886~14530349,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,252993055~6837523,260446735~9058861,270348712~4014591,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,352321536~1368400,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281] 2024-08-17T17:47:35.917 INFO:tasks.workunit.client.0.smithi136.stdout:from 15 to HEAD diff [4299867~7607183,13709905~7986367,30130057~7268094,60476408~2367307,69672745~10465490,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,263839725~5665871,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653] 2024-08-17T17:47:35.917 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4299867~7607183,13709905~7986367,30130057~7268094,60476408~2367307,69672745~10465490,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,263839725~5665871,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653] 2024-08-17T17:47:35.922 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4299867~7607183,13709905~7986367,30130057~7268094,60476408~2367307,69672745~10465490,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,260446735~9058861,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,352321536~1368400,359888874~121026,360457183~9389653] 2024-08-17T17:47:35.922 INFO:tasks.workunit.client.0.smithi136.stdout:from 16 to HEAD diff [15854147~5842125,30130057~7268094,60476408~2367307,69672745~10465490,81614709~784962,88081530~2792272,100092746~4080651,138916670~5337091,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,202869045~5347991,216722147~6387656,223587612~4701372,229335339~7013526,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~4536253] 2024-08-17T17:47:35.922 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [15854147~5842125,30130057~7268094,60476408~2367307,69672745~10465490,81614709~784962,88081530~2792272,100092746~4080651,138916670~5337091,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,202869045~5347991,216722147~6387656,223587612~4701372,229335339~7013526,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~4536253] 2024-08-17T17:47:35.927 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [15854147~5842125,30130057~7268094,60476408~2367307,69672745~10465490,81614709~784962,88081530~2792272,100092746~4080651,138916670~5337091,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,202869045~5347991,216722147~6387656,223587612~4701372,229335339~7013526,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,352321536~1368400,359888874~121026,360457183~4536253] 2024-08-17T17:47:36.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:35 smithi136 ceph-mon[109861]: osdmap e1128: 8 total, 8 up, 8 in 2024-08-17T17:47:36.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:35 smithi136 ceph-mon[115078]: osdmap e1128: 8 total, 8 up, 8 in 2024-08-17T17:47:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:35 smithi165 ceph-mon[105517]: osdmap e1128: 8 total, 8 up, 8 in 2024-08-17T17:47:36.827 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:36.827 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:36.845 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~38930520,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,213909504~27719992,243280741~44022729,289406976~23380514,314572800~6116289,327775763~6879734,335544320~14617284,352321536~1368400,356515840~56012128] 2024-08-17T17:47:36.845 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:36.845 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:36.851 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404765831~7762137] 2024-08-17T17:47:36.851 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:36.851 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:36.857 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,167772160~8947073,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:36.857 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:36.857 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:36.862 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:36.862 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:36.863 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:36.868 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~42002671,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~29314960,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:36.869 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:36.869 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:36.874 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:36.875 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:36.875 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:36.880 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,179399560~15706006,197132288~12158295,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:36.880 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:36.880 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:36.886 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~10271348,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,197132288~11181907,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:36.886 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:36.886 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:36.892 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,197132288~11181907,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:36.892 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:36.892 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:36.897 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~20163930,270348712~4014591,279830462~7473008,289406976~6112777,299299990~7490602,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:36.897 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:36.897 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:36.903 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:36.903 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:36.903 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:36.908 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:36.908 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to HEAD diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~6107187,295434938~62703,299679355~367126,301620409~5170183,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:36.909 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~6107187,295434938~62703,299679355~367126,301620409~5170183,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:36.914 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,280132049~79251,281196283~6107187,295434938~62703,299679355~367126,301620409~5170183,339854108~10307496,352321536~1368400,359888874~121026,360457183~9389653,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:36.914 INFO:tasks.workunit.client.0.smithi136.stdout:from 13 to HEAD diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,54625176~25513059,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~5188907,295434938~62703,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:36.914 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,54625176~25513059,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~5188907,295434938~62703,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:36.919 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,54625176~25513059,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,280132049~79251,281196283~5188907,295434938~62703,299679355~367126,339854108~10307496,352321536~1368400,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:36.919 INFO:tasks.workunit.client.0.smithi136.stdout:from 14 to HEAD diff [4299867~7607183,13709905~7986367,30130057~8800463,54625176~8218539,65607886~14530349,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,252993055~6837523,263839725~5665871,270348712~4014591,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281] 2024-08-17T17:47:36.919 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4299867~7607183,13709905~7986367,30130057~8800463,54625176~8218539,65607886~14530349,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,252993055~6837523,263839725~5665871,270348712~4014591,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281] 2024-08-17T17:47:36.925 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4299867~7607183,13709905~7986367,30130057~8800463,54625176~8218539,65607886~14530349,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,252993055~6837523,260446735~9058861,270348712~4014591,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,352321536~1368400,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281] 2024-08-17T17:47:36.925 INFO:tasks.workunit.client.0.smithi136.stdout:from 15 to HEAD diff [4299867~7607183,13709905~7986367,30130057~7268094,60476408~2367307,69672745~10465490,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,263839725~5665871,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653] 2024-08-17T17:47:36.925 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4299867~7607183,13709905~7986367,30130057~7268094,60476408~2367307,69672745~10465490,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,263839725~5665871,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653] 2024-08-17T17:47:36.930 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4299867~7607183,13709905~7986367,30130057~7268094,60476408~2367307,69672745~10465490,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,260446735~9058861,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,352321536~1368400,359888874~121026,360457183~9389653] 2024-08-17T17:47:37.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:36 smithi136 ceph-mon[109861]: osdmap e1129: 8 total, 8 up, 8 in 2024-08-17T17:47:37.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:47:37.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:36 smithi136 ceph-mon[109861]: pgmap v1732: 105 pgs: 105 active+clean; 1.2 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 147 KiB/s rd, 24 MiB/s wr, 2.37k op/s 2024-08-17T17:47:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:36 smithi136 ceph-mon[115078]: osdmap e1129: 8 total, 8 up, 8 in 2024-08-17T17:47:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:47:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:36 smithi136 ceph-mon[115078]: pgmap v1732: 105 pgs: 105 active+clean; 1.2 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 147 KiB/s rd, 24 MiB/s wr, 2.37k op/s 2024-08-17T17:47:37.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:36 smithi165 ceph-mon[105517]: osdmap e1129: 8 total, 8 up, 8 in 2024-08-17T17:47:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:47:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:36 smithi165 ceph-mon[105517]: pgmap v1732: 105 pgs: 105 active+clean; 1.2 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 147 KiB/s rd, 24 MiB/s wr, 2.37k op/s 2024-08-17T17:47:37.830 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:47:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:47:37] "GET /metrics HTTP/1.1" 200 36492 "" "Prometheus/2.51.0" 2024-08-17T17:47:37.837 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:37.838 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:37.856 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~38930520,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,213909504~27719992,243280741~44022729,289406976~23380514,314572800~6116289,327775763~6879734,335544320~14617284,352321536~1368400,356515840~56012128] 2024-08-17T17:47:37.856 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:37.856 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:37.862 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404765831~7762137] 2024-08-17T17:47:37.862 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:37.862 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:37.868 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,167772160~8947073,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:37.868 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:37.868 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:37.874 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:37.874 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:37.874 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:37.880 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~42002671,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~29314960,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:37.880 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:37.880 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:37.885 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:37.885 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:37.886 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:37.891 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,179399560~15706006,197132288~12158295,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:37.891 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:37.891 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:37.897 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~10271348,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,197132288~11181907,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:37.897 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:37.897 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:37.902 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,197132288~11181907,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:37.902 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:37.902 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:37.908 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~20163930,270348712~4014591,279830462~7473008,289406976~6112777,299299990~7490602,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:37.908 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:37.908 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:37.913 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:37.913 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:37.913 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:37.919 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:37.919 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to HEAD diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~6107187,295434938~62703,299679355~367126,301620409~5170183,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:37.919 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~6107187,295434938~62703,299679355~367126,301620409~5170183,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:37.924 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,280132049~79251,281196283~6107187,295434938~62703,299679355~367126,301620409~5170183,339854108~10307496,352321536~1368400,359888874~121026,360457183~9389653,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:37.924 INFO:tasks.workunit.client.0.smithi136.stdout:from 13 to HEAD diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,54625176~25513059,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~5188907,295434938~62703,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:37.924 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,54625176~25513059,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~5188907,295434938~62703,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:37.929 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,54625176~25513059,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,280132049~79251,281196283~5188907,295434938~62703,299679355~367126,339854108~10307496,352321536~1368400,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:37.930 INFO:tasks.workunit.client.0.smithi136.stdout:from 14 to HEAD diff [4299867~7607183,13709905~7986367,30130057~8800463,54625176~8218539,65607886~14530349,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,252993055~6837523,263839725~5665871,270348712~4014591,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281] 2024-08-17T17:47:37.930 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4299867~7607183,13709905~7986367,30130057~8800463,54625176~8218539,65607886~14530349,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,252993055~6837523,263839725~5665871,270348712~4014591,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281] 2024-08-17T17:47:37.935 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4299867~7607183,13709905~7986367,30130057~8800463,54625176~8218539,65607886~14530349,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,199871847~8345189,216722147~6387656,223587612~4701372,229335339~7013526,252993055~6837523,260446735~9058861,270348712~4014591,280132049~79251,281196283~5188907,299679355~367126,339854108~10307496,352321536~1368400,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281] 2024-08-17T17:47:38.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:37 smithi136 ceph-mon[109861]: osdmap e1130: 8 total, 8 up, 8 in 2024-08-17T17:47:38.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:37 smithi136 ceph-mon[115078]: osdmap e1130: 8 total, 8 up, 8 in 2024-08-17T17:47:38.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:37 smithi165 ceph-mon[105517]: osdmap e1130: 8 total, 8 up, 8 in 2024-08-17T17:47:38.856 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:38.857 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:38.872 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~38930520,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,213909504~27719992,243280741~44022729,289406976~23380514,314572800~6116289,327775763~6879734,335544320~14617284,352321536~1368400,356515840~56012128] 2024-08-17T17:47:38.872 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:38.873 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:38.879 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404765831~7762137] 2024-08-17T17:47:38.879 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:38.879 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:38.885 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,167772160~8947073,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:38.885 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:38.885 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:38.891 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:38.891 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:38.891 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:38.897 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~42002671,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~29314960,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:38.897 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:38.897 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:38.902 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:38.902 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:38.902 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:38.908 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,179399560~15706006,197132288~12158295,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:38.908 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:38.908 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:38.913 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~10271348,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,197132288~11181907,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:38.914 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:38.914 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:38.919 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,197132288~11181907,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:38.919 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:38.920 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:38.925 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~20163930,270348712~4014591,279830462~7473008,289406976~6112777,299299990~7490602,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:38.925 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:38.925 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:38.930 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:38.931 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:38.931 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:38.936 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:38.936 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to HEAD diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~6107187,295434938~62703,299679355~367126,301620409~5170183,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:38.936 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~6107187,295434938~62703,299679355~367126,301620409~5170183,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:38.941 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,280132049~79251,281196283~6107187,295434938~62703,299679355~367126,301620409~5170183,339854108~10307496,352321536~1368400,359888874~121026,360457183~9389653,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:38.941 INFO:tasks.workunit.client.0.smithi136.stdout:from 13 to HEAD diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,54625176~25513059,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~5188907,295434938~62703,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:38.941 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,54625176~25513059,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~5188907,295434938~62703,299679355~367126,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:38.946 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,54625176~25513059,81614709~784962,83664186~7312323,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,280132049~79251,281196283~5188907,295434938~62703,299679355~367126,339854108~10307496,352321536~1368400,359888874~121026,360457183~9389653,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:39.186 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:39.187 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:39.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:38 smithi136 ceph-mon[109861]: osdmap e1131: 8 total, 8 up, 8 in 2024-08-17T17:47:39.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:38 smithi136 ceph-mon[109861]: pgmap v1735: 105 pgs: 4 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 96 active+clean; 1.2 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 195 KiB/s rd, 9.9 MiB/s wr, 5.63k op/s 2024-08-17T17:47:39.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:38 smithi136 ceph-mon[115078]: osdmap e1131: 8 total, 8 up, 8 in 2024-08-17T17:47:39.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:38 smithi136 ceph-mon[115078]: pgmap v1735: 105 pgs: 4 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 96 active+clean; 1.2 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 195 KiB/s rd, 9.9 MiB/s wr, 5.63k op/s 2024-08-17T17:47:39.205 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~38930520,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,213909504~27719992,243280741~44022729,289406976~23380514,314572800~6116289,327775763~6879734,335544320~14617284,352321536~1368400,356515840~56012128] 2024-08-17T17:47:39.205 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:39.205 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:39.211 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404765831~7762137] 2024-08-17T17:47:39.211 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:39.211 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:39.216 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,167772160~8947073,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:39.216 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:39.216 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:39.221 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:39.221 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:39.221 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:39.226 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~42002671,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~29314960,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:39.227 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:39.227 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:39.232 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:39.232 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:39.232 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:39.237 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,179399560~15706006,197132288~12158295,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:39.237 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:39.237 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:39.242 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~10271348,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,197132288~11181907,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:39.242 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:39.243 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:39.248 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,197132288~11181907,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:39.249 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:39.249 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:39.254 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~20163930,270348712~4014591,279830462~7473008,289406976~6112777,299299990~7490602,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:39.254 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:39.254 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:39.259 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:39.259 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:39.259 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:39.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:38 smithi165 ceph-mon[105517]: osdmap e1131: 8 total, 8 up, 8 in 2024-08-17T17:47:39.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:38 smithi165 ceph-mon[105517]: pgmap v1735: 105 pgs: 4 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 96 active+clean; 1.2 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 195 KiB/s rd, 9.9 MiB/s wr, 5.63k op/s 2024-08-17T17:47:39.264 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:39.265 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to HEAD diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~6107187,295434938~62703,299679355~367126,301620409~5170183,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:39.265 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171913277~2498215,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,280132049~79251,281196283~6107187,295434938~62703,299679355~367126,301620409~5170183,339854108~10307496,353675391~14545,359888874~121026,360457183~9389653,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:39.269 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [1598369~392670,4299867~7607183,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~12023792,151816603~6128341,168323939~2295785,171116125~3295367,179399560~8588279,191922365~3183201,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,280132049~79251,281196283~6107187,295434938~62703,299679355~367126,301620409~5170183,339854108~10307496,352321536~1368400,359888874~121026,360457183~9389653,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:40.193 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:40.193 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:40.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:39 smithi136 ceph-mon[109861]: osdmap e1132: 8 total, 8 up, 8 in 2024-08-17T17:47:40.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:39 smithi136 ceph-mon[109861]: osdmap e1133: 8 total, 8 up, 8 in 2024-08-17T17:47:40.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:39 smithi136 ceph-mon[115078]: osdmap e1132: 8 total, 8 up, 8 in 2024-08-17T17:47:40.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:39 smithi136 ceph-mon[115078]: osdmap e1133: 8 total, 8 up, 8 in 2024-08-17T17:47:40.210 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~38930520,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,213909504~27719992,243280741~44022729,289406976~23380514,314572800~6116289,327775763~6879734,335544320~14617284,352321536~1368400,356515840~56012128] 2024-08-17T17:47:40.211 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:40.211 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:40.216 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404765831~7762137] 2024-08-17T17:47:40.216 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:40.216 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:40.222 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,167772160~8947073,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:40.222 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:40.222 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:40.227 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:40.227 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:40.228 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:40.233 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~42002671,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~29314960,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:40.233 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:40.233 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:40.239 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:40.239 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:40.239 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:40.244 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,179399560~15706006,197132288~12158295,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:40.244 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:40.244 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:40.249 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~10271348,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,197132288~11181907,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:40.250 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:40.250 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:40.255 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,197132288~11181907,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:40.255 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:40.255 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:40.260 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~20163930,270348712~4014591,279830462~7473008,289406976~6112777,299299990~7490602,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:40.260 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:40.260 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:40.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:39 smithi165 ceph-mon[105517]: osdmap e1132: 8 total, 8 up, 8 in 2024-08-17T17:47:40.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:39 smithi165 ceph-mon[105517]: osdmap e1133: 8 total, 8 up, 8 in 2024-08-17T17:47:40.266 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:40.266 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:40.266 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:40.271 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~709358,54625176~25513059,81614709~784962,83664186~7312323,97918767~799692,100092746~4080651,104378282~1274498,137168385~21231019,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8345189,216394763~6715040,223587612~4701372,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~10621123,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:41.202 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:41.202 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:41.220 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~38930520,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,213909504~27719992,243280741~44022729,289406976~23380514,314572800~6116289,327775763~6879734,335544320~14617284,352321536~1368400,356515840~56012128] 2024-08-17T17:47:41.220 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:41.221 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:41.226 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404765831~7762137] 2024-08-17T17:47:41.226 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:41.226 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:41.232 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,167772160~8947073,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:41.232 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:41.232 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:41.238 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:41.238 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:41.238 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:41.243 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~42002671,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~29314960,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:41.244 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:41.244 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:41.249 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:41.249 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:41.249 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:41.255 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,179399560~15706006,197132288~12158295,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:41.255 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:41.255 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:41.260 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~10271348,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,197132288~11181907,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:41.260 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:41.260 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:41.266 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,197132288~11181907,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:41.266 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:41.266 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:41.271 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~20163930,270348712~4014591,279830462~7473008,289406976~6112777,299299990~7490602,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:41.271 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:41.271 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,263839725~5665871,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:41.277 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~1274498,126663946~3417085,137168385~21231019,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8345189,216394763~11894221,229335339~7013526,249341666~10488912,260446735~9058861,270348712~4014591,279830462~7473008,295434938~62703,299679355~367126,301620409~5170183,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:41.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:41 smithi136 ceph-mon[109861]: osdmap e1134: 8 total, 8 up, 8 in 2024-08-17T17:47:41.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:41 smithi136 ceph-mon[109861]: pgmap v1739: 105 pgs: 4 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 96 active+clean; 1.2 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 2.6 MiB/s wr, 3.11k op/s 2024-08-17T17:47:41.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:41 smithi136 ceph-mon[115078]: osdmap e1134: 8 total, 8 up, 8 in 2024-08-17T17:47:41.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:41 smithi136 ceph-mon[115078]: pgmap v1739: 105 pgs: 4 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 96 active+clean; 1.2 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 2.6 MiB/s wr, 3.11k op/s 2024-08-17T17:47:41.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:41 smithi165 ceph-mon[105517]: osdmap e1134: 8 total, 8 up, 8 in 2024-08-17T17:47:41.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:41 smithi165 ceph-mon[105517]: pgmap v1739: 105 pgs: 4 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 96 active+clean; 1.2 GiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 2.6 MiB/s wr, 3.11k op/s 2024-08-17T17:47:42.205 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:42.205 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:42.226 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~38930520,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,213909504~27719992,243280741~44022729,289406976~23380514,314572800~6116289,327775763~6879734,335544320~14617284,352321536~1368400,356515840~56012128] 2024-08-17T17:47:42.226 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:42.226 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:42.231 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404765831~7762137] 2024-08-17T17:47:42.231 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:42.231 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:42.237 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,167772160~8947073,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:42.237 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:42.237 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:42.243 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:42.243 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:42.243 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:42.249 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~42002671,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~29314960,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:42.249 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:42.250 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:42.258 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:42.258 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:42.258 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:42.263 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,179399560~15706006,197132288~12158295,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:42.263 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:42.263 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:42.269 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~10271348,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,197132288~11181907,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:42.269 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:42.269 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:42.275 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,197132288~11181907,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:42.275 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:42.275 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~11105069,263839725~5665871,270348712~4014591,279830462~7473008,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:42.280 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~7269899,54625176~25513059,81614709~784962,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,199871847~8442348,216394763~11894221,229335339~7013526,249341666~20163930,270348712~4014591,279830462~7473008,289406976~6112777,299299990~7490602,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285] 2024-08-17T17:47:42.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:42 smithi136 ceph-mon[109861]: osdmap e1135: 8 total, 8 up, 8 in 2024-08-17T17:47:42.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:42 smithi136 ceph-mon[115078]: osdmap e1135: 8 total, 8 up, 8 in 2024-08-17T17:47:42.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:42 smithi165 ceph-mon[105517]: osdmap e1135: 8 total, 8 up, 8 in 2024-08-17T17:47:43.213 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:43.213 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:43.233 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~38930520,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,213909504~27719992,243280741~44022729,289406976~23380514,314572800~6116289,327775763~6879734,335544320~14617284,352321536~1368400,356515840~56012128] 2024-08-17T17:47:43.233 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:43.233 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:43.239 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404765831~7762137] 2024-08-17T17:47:43.239 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:43.239 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:43.245 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,167772160~8947073,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:43.245 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:43.245 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:43.250 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:43.250 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:43.250 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:43.256 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~42002671,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~29314960,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:43.256 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:43.256 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:43.262 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:43.262 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:43.262 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:43.268 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,179399560~15706006,197132288~12158295,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:43.268 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:43.268 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:43.274 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~10271348,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,197132288~11181907,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:43.274 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:43.274 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~6254630,104378282~4723101,126663946~3417085,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:43.279 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,97918767~11182616,126663946~3417085,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,197132288~11181907,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,395832430~1285281,397711277~3779285,404966874~7561094] 2024-08-17T17:47:43.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:43 smithi165 ceph-mon[105517]: osdmap e1136: 8 total, 8 up, 8 in 2024-08-17T17:47:43.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:43 smithi165 ceph-mon[105517]: pgmap v1742: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 174 KiB/s rd, 7.9 KiB/s wr, 2.39k op/s 2024-08-17T17:47:43.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:43 smithi136 ceph-mon[109861]: osdmap e1136: 8 total, 8 up, 8 in 2024-08-17T17:47:43.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:43 smithi136 ceph-mon[109861]: pgmap v1742: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 174 KiB/s rd, 7.9 KiB/s wr, 2.39k op/s 2024-08-17T17:47:43.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:43 smithi136 ceph-mon[115078]: osdmap e1136: 8 total, 8 up, 8 in 2024-08-17T17:47:43.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:43 smithi136 ceph-mon[115078]: pgmap v1742: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 174 KiB/s rd, 7.9 KiB/s wr, 2.39k op/s 2024-08-17T17:47:44.186 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:44.186 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:44.217 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~38930520,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,213909504~27719992,243280741~44022729,289406976~23380514,314572800~6116289,327775763~6879734,335544320~14617284,352321536~1368400,356515840~56012128] 2024-08-17T17:47:44.218 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:44.218 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:44.223 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404765831~7762137] 2024-08-17T17:47:44.223 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:44.223 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:44.229 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,167772160~8947073,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:44.229 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:44.229 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:44.234 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:44.235 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:44.235 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:44.240 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~42002671,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~29314960,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:44.240 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:44.240 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:44.245 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:44.246 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:44.246 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:44.251 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,179399560~15706006,197132288~12158295,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:44.251 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:44.251 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~8468550,135726836~23469276,168323939~2295785,171913277~2498215,179399560~15706006,199192089~9122106,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:44.256 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~10271348,135726836~23469276,168323939~2295785,171116125~3295367,179399560~15706006,197132288~11181907,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~6112777,299299990~7490602,316577159~3150955,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~3779285,404966874~7561094] 2024-08-17T17:47:44.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:44 smithi165 ceph-mon[105517]: osdmap e1137: 8 total, 8 up, 8 in 2024-08-17T17:47:44.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:44 smithi165 ceph-mon[105517]: osdmap e1138: 8 total, 8 up, 8 in 2024-08-17T17:47:44.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:44 smithi136 ceph-mon[109861]: osdmap e1137: 8 total, 8 up, 8 in 2024-08-17T17:47:44.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:44 smithi136 ceph-mon[109861]: osdmap e1138: 8 total, 8 up, 8 in 2024-08-17T17:47:44.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:44 smithi136 ceph-mon[115078]: osdmap e1137: 8 total, 8 up, 8 in 2024-08-17T17:47:44.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:44 smithi136 ceph-mon[115078]: osdmap e1138: 8 total, 8 up, 8 in 2024-08-17T17:47:45.220 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:45.220 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:45.240 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~38930520,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,213909504~27719992,243280741~44022729,289406976~23380514,314572800~6116289,327775763~6879734,335544320~14617284,352321536~1368400,356515840~56012128] 2024-08-17T17:47:45.240 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:45.240 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:45.246 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404765831~7762137] 2024-08-17T17:47:45.246 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:45.246 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:45.251 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,167772160~8947073,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:45.252 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:45.252 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:45.257 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:45.257 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:45.257 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:45.263 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~42002671,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~29314960,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:45.263 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:45.263 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:45.269 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:45.269 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:45.269 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,179399560~15706006,199192089~10098494,214907306~13381678,229335339~12294157,249341666~11105069,263839725~5665871,270348712~16954758,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:45.274 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~7986367,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,179399560~15706006,197132288~12158295,214907306~13381678,229335339~12294157,249341666~20163930,270348712~16954758,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:45.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:45 smithi165 ceph-mon[105517]: pgmap v1745: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 159 KiB/s rd, 7.2 KiB/s wr, 2.18k op/s 2024-08-17T17:47:45.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:45 smithi136 ceph-mon[109861]: pgmap v1745: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 159 KiB/s rd, 7.2 KiB/s wr, 2.18k op/s 2024-08-17T17:47:45.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:45 smithi136 ceph-mon[115078]: pgmap v1745: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 159 KiB/s rd, 7.2 KiB/s wr, 2.18k op/s 2024-08-17T17:47:46.225 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:46.225 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:46.248 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~38930520,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,213909504~27719992,243280741~44022729,289406976~23380514,314572800~6116289,327775763~6879734,335544320~14617284,352321536~1368400,356515840~56012128] 2024-08-17T17:47:46.248 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:46.248 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:46.254 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404765831~7762137] 2024-08-17T17:47:46.254 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:46.254 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:46.259 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,167772160~8947073,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:46.260 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:46.260 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:46.265 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:46.265 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:46.265 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:46.271 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~42002671,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~29314960,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:46.271 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:46.271 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6410563,339854108~10307496,353675391~14545,357458885~12387951,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:46.276 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~16680546,119809683~14613825,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6410563,339854108~10307496,352321536~1368400,356515840~13330996,372530391~13300409,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:46.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:46 smithi165 ceph-mon[105517]: osdmap e1139: 8 total, 8 up, 8 in 2024-08-17T17:47:46.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:46 smithi136 ceph-mon[109861]: osdmap e1139: 8 total, 8 up, 8 in 2024-08-17T17:47:46.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:46 smithi136 ceph-mon[115078]: osdmap e1139: 8 total, 8 up, 8 in 2024-08-17T17:47:47.241 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:47.241 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:47.256 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~38930520,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,213909504~27719992,243280741~44022729,289406976~23380514,314572800~6116289,327775763~6879734,335544320~14617284,352321536~1368400,356515840~56012128] 2024-08-17T17:47:47.256 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:47.256 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:47.262 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404765831~7762137] 2024-08-17T17:47:47.262 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:47.262 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:47.268 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,167772160~8947073,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:47.268 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:47.268 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:47.273 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:47.273 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:47.273 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,30130057~8800463,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~12811027,135726836~23469276,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~10959239,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~28371915,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:47.279 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~11986730,30130057~8800463,41943040~38195195,81614709~784962,83664186~7312323,92420837~42002671,135726836~23469276,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~17383616,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~29314960,385974724~5541505,391986145~5131566,397711277~4822997,404966874~7561094] 2024-08-17T17:47:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:47 smithi165 ceph-mon[105517]: osdmap e1140: 8 total, 8 up, 8 in 2024-08-17T17:47:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:47 smithi165 ceph-mon[105517]: pgmap v1748: 105 pgs: 105 active+clean; 1.0 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 97 KiB/s rd, 6.0 KiB/s wr, 663 op/s 2024-08-17T17:47:47.598 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:47 smithi136 ceph-mon[109861]: osdmap e1140: 8 total, 8 up, 8 in 2024-08-17T17:47:47.598 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:47 smithi136 ceph-mon[109861]: pgmap v1748: 105 pgs: 105 active+clean; 1.0 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 97 KiB/s rd, 6.0 KiB/s wr, 663 op/s 2024-08-17T17:47:47.598 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:47 smithi136 ceph-mon[115078]: osdmap e1140: 8 total, 8 up, 8 in 2024-08-17T17:47:47.598 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:47 smithi136 ceph-mon[115078]: pgmap v1748: 105 pgs: 105 active+clean; 1.0 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 97 KiB/s rd, 6.0 KiB/s wr, 663 op/s 2024-08-17T17:47:47.599 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:47:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:47:47] "GET /metrics HTTP/1.1" 200 36492 "" "Prometheus/2.51.0" 2024-08-17T17:47:48.232 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:48.232 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:48.258 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~38930520,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,213909504~27719992,243280741~44022729,289406976~23380514,314572800~6116289,327775763~6879734,335544320~14617284,352321536~1368400,356515840~56012128] 2024-08-17T17:47:48.258 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:48.258 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:48.263 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404765831~7762137] 2024-08-17T17:47:48.264 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:48.264 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:48.269 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,167772160~8947073,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:48.269 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:48.269 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2295785,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316577159~4111930,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:48.274 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,168323939~2295785,171116125~5603108,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,316577159~4111930,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:48.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:48 smithi165 ceph-mon[105517]: osdmap e1141: 8 total, 8 up, 8 in 2024-08-17T17:47:48.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:48 smithi136 ceph-mon[109861]: osdmap e1141: 8 total, 8 up, 8 in 2024-08-17T17:47:48.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:48 smithi136 ceph-mon[115078]: osdmap e1141: 8 total, 8 up, 8 in 2024-08-17T17:47:49.190 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:49.190 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:49.211 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~38930520,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,213909504~27719992,243280741~44022729,289406976~23380514,314572800~6116289,327775763~6879734,335544320~14617284,352321536~1368400,356515840~56012128] 2024-08-17T17:47:49.211 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:49.211 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:49.216 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404765831~7762137] 2024-08-17T17:47:49.216 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:49.216 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~7312323,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:49.221 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~7312323,92420837~66775275,167772160~8947073,177148887~17956679,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404966874~7561094] 2024-08-17T17:47:49.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:49 smithi165 ceph-mon[105517]: osdmap e1142: 8 total, 8 up, 8 in 2024-08-17T17:47:49.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:49 smithi165 ceph-mon[105517]: pgmap v1751: 105 pgs: 105 active+clean; 1007 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 97 KiB/s rd, 6.0 KiB/s wr, 1.16k op/s 2024-08-17T17:47:49.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:49 smithi165 ceph-mon[105517]: osdmap e1143: 8 total, 8 up, 8 in 2024-08-17T17:47:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:49 smithi136 ceph-mon[109861]: osdmap e1142: 8 total, 8 up, 8 in 2024-08-17T17:47:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:49 smithi136 ceph-mon[109861]: pgmap v1751: 105 pgs: 105 active+clean; 1007 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 97 KiB/s rd, 6.0 KiB/s wr, 1.16k op/s 2024-08-17T17:47:49.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:49 smithi136 ceph-mon[109861]: osdmap e1143: 8 total, 8 up, 8 in 2024-08-17T17:47:49.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:49 smithi136 ceph-mon[115078]: osdmap e1142: 8 total, 8 up, 8 in 2024-08-17T17:47:49.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:49 smithi136 ceph-mon[115078]: pgmap v1751: 105 pgs: 105 active+clean; 1007 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 97 KiB/s rd, 6.0 KiB/s wr, 1.16k op/s 2024-08-17T17:47:49.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:49 smithi136 ceph-mon[115078]: osdmap e1143: 8 total, 8 up, 8 in 2024-08-17T17:47:50.238 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:50.238 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:50.261 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~38930520,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,213909504~27719992,243280741~44022729,289406976~23380514,314572800~6116289,327775763~6879734,335544320~14617284,352321536~1368400,356515840~56012128] 2024-08-17T17:47:50.261 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:50.262 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~8378554,13709905~7986367,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214907306~13381678,229335339~12294157,249341666~11105069,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,339854108~10307496,353675391~14545,357458885~45075389,404966874~7561094] 2024-08-17T17:47:50.267 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~11907050,13709905~25220615,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,214907306~26722190,249341666~37961804,289406976~23380514,314572800~6116289,327775763~6879734,339854108~10307496,352321536~1368400,356515840~46018434,404765831~7762137] 2024-08-17T17:47:51.249 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:51.249 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [1598369~392670,3528496~18167776,22298619~3398016,27213408~11717112,43111238~37026997,81614709~784962,83664186~8756651,96231003~7942394,104378282~4723101,112459935~7349748,121612481~37583631,168323939~2792186,171913277~4805956,177148887~17956679,199192089~10098494,210384164~3396269,214151721~14137263,229335339~12294157,244079590~16367145,263839725~23463745,290745721~4774032,295831353~16956137,316072980~4616109,327775763~6879734,338264527~11897077,353675391~14545,357458885~47306946,404966874~7561094] 2024-08-17T17:47:51.269 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~38930520,41943040~38195195,81614709~784962,83664186~75531926,167772160~27333406,197132288~12158295,209715200~4065233,213909504~27719992,243280741~44022729,289406976~23380514,314572800~6116289,327775763~6879734,335544320~14617284,352321536~1368400,356515840~56012128] 2024-08-17T17:47:51.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:51 smithi165 ceph-mon[105517]: osdmap e1144: 8 total, 8 up, 8 in 2024-08-17T17:47:51.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:51 smithi165 ceph-mon[105517]: pgmap v1754: 105 pgs: 105 active+clean; 1007 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 501 op/s 2024-08-17T17:47:51.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:47:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:51 smithi136 ceph-mon[109861]: osdmap e1144: 8 total, 8 up, 8 in 2024-08-17T17:47:51.719 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:51 smithi136 ceph-mon[109861]: pgmap v1754: 105 pgs: 105 active+clean; 1007 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 501 op/s 2024-08-17T17:47:51.719 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:47:51.720 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:51 smithi136 ceph-mon[115078]: osdmap e1144: 8 total, 8 up, 8 in 2024-08-17T17:47:51.720 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:51 smithi136 ceph-mon[115078]: pgmap v1754: 105 pgs: 105 active+clean; 1007 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 501 op/s 2024-08-17T17:47:51.720 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:47:52.271 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/0.DiffIterateStress (39589 ms) 2024-08-17T17:47:52.272 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/0.DiffIterateParent 2024-08-17T17:47:52.272 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:47:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:52 smithi165 ceph-mon[105517]: osdmap e1145: 8 total, 8 up, 8 in 2024-08-17T17:47:52.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:52 smithi136 ceph-mon[115078]: osdmap e1145: 8 total, 8 up, 8 in 2024-08-17T17:47:52.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:52 smithi136 ceph-mon[109861]: osdmap e1145: 8 total, 8 up, 8 in 2024-08-17T17:47:53.365 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/0.DiffIterateParent (1094 ms) 2024-08-17T17:47:53.365 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/0.DiffIterateIgnoreParent 2024-08-17T17:47:53.366 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:47:53.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:53 smithi165 ceph-mon[105517]: osdmap e1146: 8 total, 8 up, 8 in 2024-08-17T17:47:53.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:53 smithi165 ceph-mon[105517]: pgmap v1757: 105 pgs: 105 active+clean; 829 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 65 KiB/s rd, 5.7 KiB/s wr, 514 op/s 2024-08-17T17:47:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:53 smithi136 ceph-mon[109861]: osdmap e1146: 8 total, 8 up, 8 in 2024-08-17T17:47:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:53 smithi136 ceph-mon[109861]: pgmap v1757: 105 pgs: 105 active+clean; 829 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 65 KiB/s rd, 5.7 KiB/s wr, 514 op/s 2024-08-17T17:47:53.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:53 smithi136 ceph-mon[115078]: osdmap e1146: 8 total, 8 up, 8 in 2024-08-17T17:47:53.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:53 smithi136 ceph-mon[115078]: pgmap v1757: 105 pgs: 105 active+clean; 829 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 65 KiB/s rd, 5.7 KiB/s wr, 514 op/s 2024-08-17T17:47:54.300 INFO:tasks.workunit.client.0.smithi136.stdout: wrote [1334730~71218,7569329~36383,14885838~76837,15106441~81908,15406091~42878,15894005~88964,16071438~4449,17071043~57180,18452650~32209,18790485~13683] to clone 2024-08-17T17:47:54.300 INFO:tasks.workunit.client.0.smithi136.stdout: diff was [0~1405948,4194304~3411408,12582912~3492975,16777216~2026952] 2024-08-17T17:47:54.303 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/0.DiffIterateIgnoreParent (939 ms) 2024-08-17T17:47:54.304 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/0.DiffIterateCallbackError 2024-08-17T17:47:54.304 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:47:54.322 INFO:tasks.workunit.client.0.smithi136.stdout: wrote [1462781~75008,3012824~49915,3262123~38296,6347278~43466,9718143~73453,10578948~17246,11173620~3609,12513410~31732,14001902~4648,18063956~102386] 2024-08-17T17:47:54.332 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/0.DiffIterateCallbackError (28 ms) 2024-08-17T17:47:54.332 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/0.DiffIterateParentDiscard 2024-08-17T17:47:54.332 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:47:54.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:54 smithi165 ceph-mon[105517]: osdmap e1147: 8 total, 8 up, 8 in 2024-08-17T17:47:54.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:54 smithi165 ceph-mon[105517]: osdmap e1148: 8 total, 8 up, 8 in 2024-08-17T17:47:54.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:54 smithi136 ceph-mon[109861]: osdmap e1147: 8 total, 8 up, 8 in 2024-08-17T17:47:54.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:54 smithi136 ceph-mon[109861]: osdmap e1148: 8 total, 8 up, 8 in 2024-08-17T17:47:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:54 smithi136 ceph-mon[115078]: osdmap e1147: 8 total, 8 up, 8 in 2024-08-17T17:47:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:54 smithi136 ceph-mon[115078]: osdmap e1148: 8 total, 8 up, 8 in 2024-08-17T17:47:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:55 smithi136 ceph-mon[109861]: pgmap v1760: 105 pgs: 105 active+clean; 829 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 65 KiB/s rd, 5.7 KiB/s wr, 514 op/s 2024-08-17T17:47:55.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:55 smithi136 ceph-mon[115078]: pgmap v1760: 105 pgs: 105 active+clean; 829 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 65 KiB/s rd, 5.7 KiB/s wr, 514 op/s 2024-08-17T17:47:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:55 smithi165 ceph-mon[105517]: pgmap v1760: 105 pgs: 105 active+clean; 829 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 65 KiB/s rd, 5.7 KiB/s wr, 514 op/s 2024-08-17T17:47:56.369 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/0.DiffIterateParentDiscard (2037 ms) 2024-08-17T17:47:56.369 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/0.DiffIterateUnalignedSmall 2024-08-17T17:47:56.369 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:47:56.416 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/0.DiffIterateUnalignedSmall (47 ms) 2024-08-17T17:47:56.416 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/0.DiffIterateUnaligned 2024-08-17T17:47:56.416 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:47:56.491 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/0.DiffIterateUnaligned (75 ms) 2024-08-17T17:47:56.491 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/0.DiffIterateTryAcquireLock 2024-08-17T17:47:56.491 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:47:56.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:56 smithi136 ceph-mon[115078]: osdmap e1149: 8 total, 8 up, 8 in 2024-08-17T17:47:56.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:56 smithi136 ceph-mon[109861]: osdmap e1149: 8 total, 8 up, 8 in 2024-08-17T17:47:56.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:56 smithi165 ceph-mon[105517]: osdmap e1149: 8 total, 8 up, 8 in 2024-08-17T17:47:57.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:57 smithi136 ceph-mon[109861]: osdmap e1150: 8 total, 8 up, 8 in 2024-08-17T17:47:57.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:57 smithi136 ceph-mon[109861]: pgmap v1763: 105 pgs: 105 active+clean; 818 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 127 op/s 2024-08-17T17:47:57.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:47:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:47:57] "GET /metrics HTTP/1.1" 200 36494 "" "Prometheus/2.51.0" 2024-08-17T17:47:57.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:57 smithi136 ceph-mon[115078]: osdmap e1150: 8 total, 8 up, 8 in 2024-08-17T17:47:57.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:57 smithi136 ceph-mon[115078]: pgmap v1763: 105 pgs: 105 active+clean; 818 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 127 op/s 2024-08-17T17:47:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:57 smithi165 ceph-mon[105517]: osdmap e1150: 8 total, 8 up, 8 in 2024-08-17T17:47:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:57 smithi165 ceph-mon[105517]: pgmap v1763: 105 pgs: 105 active+clean; 818 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 127 op/s 2024-08-17T17:47:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:47:58 smithi136 ceph-mon[109861]: pgmap v1764: 105 pgs: 105 active+clean; 854 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.0 MiB/s rd, 9.8 MiB/s wr, 302 op/s 2024-08-17T17:47:58.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:47:58 smithi136 ceph-mon[115078]: pgmap v1764: 105 pgs: 105 active+clean; 854 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.0 MiB/s rd, 9.8 MiB/s wr, 302 op/s 2024-08-17T17:47:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:47:58 smithi165 ceph-mon[105517]: pgmap v1764: 105 pgs: 105 active+clean; 854 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.0 MiB/s rd, 9.8 MiB/s wr, 302 op/s 2024-08-17T17:48:00.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:00 smithi136 ceph-mon[109861]: osdmap e1151: 8 total, 8 up, 8 in 2024-08-17T17:48:00.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:00 smithi136 ceph-mon[115078]: osdmap e1151: 8 total, 8 up, 8 in 2024-08-17T17:48:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:00 smithi165 ceph-mon[105517]: osdmap e1151: 8 total, 8 up, 8 in 2024-08-17T17:48:01.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:01 smithi136 ceph-mon[109861]: pgmap v1766: 105 pgs: 105 active+clean; 854 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 8.6 MiB/s wr, 264 op/s 2024-08-17T17:48:01.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:01 smithi136 ceph-mon[115078]: pgmap v1766: 105 pgs: 105 active+clean; 854 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 8.6 MiB/s wr, 264 op/s 2024-08-17T17:48:01.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:01 smithi165 ceph-mon[105517]: pgmap v1766: 105 pgs: 105 active+clean; 854 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.5 MiB/s rd, 8.6 MiB/s wr, 264 op/s 2024-08-17T17:48:01.527 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/0.DiffIterateTryAcquireLock (5036 ms) 2024-08-17T17:48:01.528 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/0.DiffIterateStriping 2024-08-17T17:48:01.528 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_librbd.cc:8958: Skipped 2024-08-17T17:48:01.528 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:48:01.528 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] DiffIterateTest/0.DiffIterateStriping (0 ms) 2024-08-17T17:48:02.197 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 19 tests from DiffIterateTest/0 (233992 ms total) 2024-08-17T17:48:02.197 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:48:02.197 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 19 tests from DiffIterateTest/1, where TypeParam = DiffIterateParams 2024-08-17T17:48:02.229 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/1.DiffIterate 2024-08-17T17:48:03.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:03 smithi136 ceph-mon[109861]: osdmap e1152: 8 total, 8 up, 8 in 2024-08-17T17:48:03.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:03 smithi136 ceph-mon[109861]: pgmap v1768: 73 pgs: 73 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:48:03.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:03 smithi136 ceph-mon[115078]: osdmap e1152: 8 total, 8 up, 8 in 2024-08-17T17:48:03.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:03 smithi136 ceph-mon[115078]: pgmap v1768: 73 pgs: 73 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:48:03.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:03 smithi165 ceph-mon[105517]: osdmap e1152: 8 total, 8 up, 8 in 2024-08-17T17:48:03.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:03 smithi165 ceph-mon[105517]: pgmap v1768: 73 pgs: 73 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:48:04.217 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:48:04.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:04 smithi136 ceph-mon[109861]: osdmap e1153: 8 total, 8 up, 8 in 2024-08-17T17:48:04.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:04 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2324058082' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-48","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:48:04.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:04 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-48","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:48:04.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:04 smithi136 ceph-mon[115078]: osdmap e1153: 8 total, 8 up, 8 in 2024-08-17T17:48:04.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:04 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2324058082' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-48","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:48:04.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:04 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-48","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:48:04.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:04 smithi165 ceph-mon[105517]: osdmap e1153: 8 total, 8 up, 8 in 2024-08-17T17:48:04.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:04 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2324058082' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-48","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:48:04.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:04 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-48","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:48:05.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:05 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-48","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:48:05.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:05 smithi165 ceph-mon[105517]: osdmap e1154: 8 total, 8 up, 8 in 2024-08-17T17:48:05.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:05 smithi165 ceph-mon[105517]: pgmap v1771: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:48:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:05 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-48","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:48:05.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:05 smithi136 ceph-mon[109861]: osdmap e1154: 8 total, 8 up, 8 in 2024-08-17T17:48:05.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:05 smithi136 ceph-mon[109861]: pgmap v1771: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:48:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:05 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi136-87100-48","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:48:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:05 smithi136 ceph-mon[115078]: osdmap e1154: 8 total, 8 up, 8 in 2024-08-17T17:48:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:05 smithi136 ceph-mon[115078]: pgmap v1771: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:48:06.254 INFO:tasks.workunit.client.0.smithi136.stdout: wrote [604208~77441,6278229~17491,10547692~17053,12018281~51966,13150062~85408,13800997~87446,14100346~40843,17288109~97461,19560648~97657] 2024-08-17T17:48:06.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:06 smithi165 ceph-mon[105517]: osdmap e1155: 8 total, 8 up, 8 in 2024-08-17T17:48:06.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:48:06.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:06 smithi136 ceph-mon[109861]: osdmap e1155: 8 total, 8 up, 8 in 2024-08-17T17:48:06.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:48:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:06 smithi136 ceph-mon[115078]: osdmap e1155: 8 total, 8 up, 8 in 2024-08-17T17:48:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:48:07.242 INFO:tasks.workunit.client.0.smithi136.stdout: wrote [0~20971520] 2024-08-17T17:48:07.251 INFO:tasks.workunit.client.0.smithi136.stdout: diff was [0~20971520] 2024-08-17T17:48:07.255 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/1.DiffIterate (5026 ms) 2024-08-17T17:48:07.255 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/1.DiffIterateDeterministic 2024-08-17T17:48:07.256 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:48:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:07 smithi165 ceph-mon[105517]: osdmap e1156: 8 total, 8 up, 8 in 2024-08-17T17:48:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:07 smithi165 ceph-mon[105517]: pgmap v1774: 105 pgs: 16 unknown, 89 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:48:07.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:07 smithi136 ceph-mon[109861]: osdmap e1156: 8 total, 8 up, 8 in 2024-08-17T17:48:07.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:07 smithi136 ceph-mon[109861]: pgmap v1774: 105 pgs: 16 unknown, 89 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:48:07.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:07 smithi136 ceph-mon[115078]: osdmap e1156: 8 total, 8 up, 8 in 2024-08-17T17:48:07.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:07 smithi136 ceph-mon[115078]: pgmap v1774: 105 pgs: 16 unknown, 89 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:48:07.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:48:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:48:07] "GET /metrics HTTP/1.1" 200 36420 "" "Prometheus/2.51.0" 2024-08-17T17:48:08.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:08 smithi165 ceph-mon[105517]: osdmap e1157: 8 total, 8 up, 8 in 2024-08-17T17:48:08.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:48:08.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:08 smithi136 ceph-mon[109861]: osdmap e1157: 8 total, 8 up, 8 in 2024-08-17T17:48:08.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:48:08.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:08 smithi136 ceph-mon[115078]: osdmap e1157: 8 total, 8 up, 8 in 2024-08-17T17:48:08.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:48:09.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:09 smithi136 ceph-mon[115078]: osdmap e1158: 8 total, 8 up, 8 in 2024-08-17T17:48:09.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:09 smithi136 ceph-mon[115078]: pgmap v1777: 105 pgs: 105 active+clean; 6.8 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 77 KiB/s wr, 21 op/s 2024-08-17T17:48:09.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:09 smithi136 ceph-mon[109861]: osdmap e1158: 8 total, 8 up, 8 in 2024-08-17T17:48:09.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:09 smithi136 ceph-mon[109861]: pgmap v1777: 105 pgs: 105 active+clean; 6.8 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 77 KiB/s wr, 21 op/s 2024-08-17T17:48:09.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:09 smithi165 ceph-mon[105517]: osdmap e1158: 8 total, 8 up, 8 in 2024-08-17T17:48:09.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:09 smithi165 ceph-mon[105517]: pgmap v1777: 105 pgs: 105 active+clean; 6.8 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 77 KiB/s wr, 21 op/s 2024-08-17T17:48:10.262 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:48:10.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:10 smithi136 ceph-mon[109861]: osdmap e1159: 8 total, 8 up, 8 in 2024-08-17T17:48:10.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:10 smithi136 ceph-mon[115078]: osdmap e1159: 8 total, 8 up, 8 in 2024-08-17T17:48:10.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:10 smithi165 ceph-mon[105517]: osdmap e1159: 8 total, 8 up, 8 in 2024-08-17T17:48:11.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:11 smithi165 ceph-mon[105517]: osdmap e1160: 8 total, 8 up, 8 in 2024-08-17T17:48:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:11 smithi165 ceph-mon[105517]: pgmap v1780: 105 pgs: 105 active+clean; 6.8 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 77 KiB/s wr, 21 op/s 2024-08-17T17:48:11.586 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:11 smithi136 ceph-mon[109861]: osdmap e1160: 8 total, 8 up, 8 in 2024-08-17T17:48:11.586 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:11 smithi136 ceph-mon[109861]: pgmap v1780: 105 pgs: 105 active+clean; 6.8 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 77 KiB/s wr, 21 op/s 2024-08-17T17:48:11.586 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:11 smithi136 ceph-mon[115078]: osdmap e1160: 8 total, 8 up, 8 in 2024-08-17T17:48:11.587 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:11 smithi136 ceph-mon[115078]: pgmap v1780: 105 pgs: 105 active+clean; 6.8 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 77 KiB/s wr, 21 op/s 2024-08-17T17:48:12.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:12 smithi165 ceph-mon[105517]: osdmap e1161: 8 total, 8 up, 8 in 2024-08-17T17:48:12.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:12 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:48:12.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:12 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:48:12.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:12 smithi136 ceph-mon[115078]: osdmap e1161: 8 total, 8 up, 8 in 2024-08-17T17:48:12.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:48:12.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:48:12.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:12 smithi136 ceph-mon[109861]: osdmap e1161: 8 total, 8 up, 8 in 2024-08-17T17:48:12.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:48:12.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:48:13.284 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:48:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:13 smithi165 ceph-mon[105517]: osdmap e1162: 8 total, 8 up, 8 in 2024-08-17T17:48:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:13 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:48:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:13 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:48:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:13 smithi165 ceph-mon[105517]: pgmap v1783: 105 pgs: 105 active+clean; 17 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 229 KiB/s wr, 101 op/s 2024-08-17T17:48:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:13 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:48:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:13 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:48:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:13 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:48:13.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:13 smithi136 ceph-mon[109861]: osdmap e1162: 8 total, 8 up, 8 in 2024-08-17T17:48:13.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:13 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:48:13.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:13 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:48:13.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:13 smithi136 ceph-mon[109861]: pgmap v1783: 105 pgs: 105 active+clean; 17 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 229 KiB/s wr, 101 op/s 2024-08-17T17:48:13.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:13 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:48:13.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:13 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:48:13.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:13 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:48:13.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:13 smithi136 ceph-mon[115078]: osdmap e1162: 8 total, 8 up, 8 in 2024-08-17T17:48:13.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:13 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:48:13.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:13 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:48:13.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:13 smithi136 ceph-mon[115078]: pgmap v1783: 105 pgs: 105 active+clean; 17 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 229 KiB/s wr, 101 op/s 2024-08-17T17:48:13.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:13 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:48:13.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:13 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:48:13.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:13 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:48:14.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:14 smithi165 ceph-mon[105517]: osdmap e1163: 8 total, 8 up, 8 in 2024-08-17T17:48:14.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:14 smithi165 ceph-mon[105517]: osdmap e1164: 8 total, 8 up, 8 in 2024-08-17T17:48:14.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:14 smithi136 ceph-mon[109861]: osdmap e1163: 8 total, 8 up, 8 in 2024-08-17T17:48:14.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:14 smithi136 ceph-mon[109861]: osdmap e1164: 8 total, 8 up, 8 in 2024-08-17T17:48:14.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:14 smithi136 ceph-mon[115078]: osdmap e1163: 8 total, 8 up, 8 in 2024-08-17T17:48:14.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:14 smithi136 ceph-mon[115078]: osdmap e1164: 8 total, 8 up, 8 in 2024-08-17T17:48:15.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:15 smithi136 ceph-mon[109861]: pgmap v1786: 105 pgs: 105 active+clean; 17 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 229 KiB/s wr, 101 op/s 2024-08-17T17:48:15.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:15 smithi136 ceph-mon[109861]: osdmap e1165: 8 total, 8 up, 8 in 2024-08-17T17:48:15.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:15 smithi136 ceph-mon[115078]: pgmap v1786: 105 pgs: 105 active+clean; 17 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 229 KiB/s wr, 101 op/s 2024-08-17T17:48:15.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:15 smithi136 ceph-mon[115078]: osdmap e1165: 8 total, 8 up, 8 in 2024-08-17T17:48:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:15 smithi165 ceph-mon[105517]: pgmap v1786: 105 pgs: 105 active+clean; 17 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 229 KiB/s wr, 101 op/s 2024-08-17T17:48:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:15 smithi165 ceph-mon[105517]: osdmap e1165: 8 total, 8 up, 8 in 2024-08-17T17:48:16.223 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:48:17.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:17 smithi136 ceph-mon[109861]: osdmap e1166: 8 total, 8 up, 8 in 2024-08-17T17:48:17.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:17 smithi136 ceph-mon[109861]: pgmap v1789: 105 pgs: 105 active+clean; 19 MiB data, 687 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 5.7 KiB/s wr, 68 op/s 2024-08-17T17:48:17.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:17 smithi136 ceph-mon[115078]: osdmap e1166: 8 total, 8 up, 8 in 2024-08-17T17:48:17.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:17 smithi136 ceph-mon[115078]: pgmap v1789: 105 pgs: 105 active+clean; 19 MiB data, 687 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 5.7 KiB/s wr, 68 op/s 2024-08-17T17:48:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:17 smithi165 ceph-mon[105517]: osdmap e1166: 8 total, 8 up, 8 in 2024-08-17T17:48:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:17 smithi165 ceph-mon[105517]: pgmap v1789: 105 pgs: 105 active+clean; 19 MiB data, 687 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 5.7 KiB/s wr, 68 op/s 2024-08-17T17:48:17.720 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:48:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:48:17] "GET /metrics HTTP/1.1" 200 36420 "" "Prometheus/2.51.0" 2024-08-17T17:48:18.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:18 smithi165 ceph-mon[105517]: osdmap e1167: 8 total, 8 up, 8 in 2024-08-17T17:48:18.645 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:18 smithi136 ceph-mon[109861]: osdmap e1167: 8 total, 8 up, 8 in 2024-08-17T17:48:18.646 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:18 smithi136 ceph-mon[115078]: osdmap e1167: 8 total, 8 up, 8 in 2024-08-17T17:48:19.255 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:48:19.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:19 smithi165 ceph-mon[105517]: osdmap e1168: 8 total, 8 up, 8 in 2024-08-17T17:48:19.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:19 smithi165 ceph-mon[105517]: pgmap v1792: 105 pgs: 105 active+clean; 20 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 8.5 KiB/s wr, 94 op/s 2024-08-17T17:48:19.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:19 smithi136 ceph-mon[115078]: osdmap e1168: 8 total, 8 up, 8 in 2024-08-17T17:48:19.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:19 smithi136 ceph-mon[115078]: pgmap v1792: 105 pgs: 105 active+clean; 20 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 8.5 KiB/s wr, 94 op/s 2024-08-17T17:48:19.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:19 smithi136 ceph-mon[109861]: osdmap e1168: 8 total, 8 up, 8 in 2024-08-17T17:48:19.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:19 smithi136 ceph-mon[109861]: pgmap v1792: 105 pgs: 105 active+clean; 20 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 8.5 KiB/s wr, 94 op/s 2024-08-17T17:48:20.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:20 smithi165 ceph-mon[105517]: osdmap e1169: 8 total, 8 up, 8 in 2024-08-17T17:48:20.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:20 smithi136 ceph-mon[109861]: osdmap e1169: 8 total, 8 up, 8 in 2024-08-17T17:48:20.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:20 smithi136 ceph-mon[115078]: osdmap e1169: 8 total, 8 up, 8 in 2024-08-17T17:48:21.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:21 smithi165 ceph-mon[105517]: osdmap e1170: 8 total, 8 up, 8 in 2024-08-17T17:48:21.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:21 smithi165 ceph-mon[105517]: pgmap v1795: 105 pgs: 105 active+clean; 20 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 2.7 KiB/s wr, 25 op/s 2024-08-17T17:48:21.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:48:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:21 smithi136 ceph-mon[109861]: osdmap e1170: 8 total, 8 up, 8 in 2024-08-17T17:48:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:21 smithi136 ceph-mon[109861]: pgmap v1795: 105 pgs: 105 active+clean; 20 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 2.7 KiB/s wr, 25 op/s 2024-08-17T17:48:21.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:48:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:21 smithi136 ceph-mon[115078]: osdmap e1170: 8 total, 8 up, 8 in 2024-08-17T17:48:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:21 smithi136 ceph-mon[115078]: pgmap v1795: 105 pgs: 105 active+clean; 20 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 2.7 KiB/s wr, 25 op/s 2024-08-17T17:48:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:48:22.283 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/1.DiffIterateDeterministic (15028 ms) 2024-08-17T17:48:22.283 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/1.DiffIterateDeterministicPP 2024-08-17T17:48:22.283 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:48:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:22 smithi165 ceph-mon[105517]: osdmap e1171: 8 total, 8 up, 8 in 2024-08-17T17:48:22.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:22 smithi136 ceph-mon[109861]: osdmap e1171: 8 total, 8 up, 8 in 2024-08-17T17:48:22.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:22 smithi136 ceph-mon[115078]: osdmap e1171: 8 total, 8 up, 8 in 2024-08-17T17:48:23.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:23 smithi165 ceph-mon[105517]: osdmap e1172: 8 total, 8 up, 8 in 2024-08-17T17:48:23.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:23 smithi165 ceph-mon[105517]: pgmap v1798: 105 pgs: 105 active+clean; 30 MiB data, 692 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 9.2 KiB/s wr, 98 op/s 2024-08-17T17:48:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:23 smithi136 ceph-mon[109861]: osdmap e1172: 8 total, 8 up, 8 in 2024-08-17T17:48:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:23 smithi136 ceph-mon[109861]: pgmap v1798: 105 pgs: 105 active+clean; 30 MiB data, 692 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 9.2 KiB/s wr, 98 op/s 2024-08-17T17:48:23.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:23 smithi136 ceph-mon[115078]: osdmap e1172: 8 total, 8 up, 8 in 2024-08-17T17:48:23.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:23 smithi136 ceph-mon[115078]: pgmap v1798: 105 pgs: 105 active+clean; 30 MiB data, 692 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 9.2 KiB/s wr, 98 op/s 2024-08-17T17:48:24.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:24 smithi165 ceph-mon[105517]: osdmap e1173: 8 total, 8 up, 8 in 2024-08-17T17:48:24.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:24 smithi136 ceph-mon[109861]: osdmap e1173: 8 total, 8 up, 8 in 2024-08-17T17:48:24.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:24 smithi136 ceph-mon[115078]: osdmap e1173: 8 total, 8 up, 8 in 2024-08-17T17:48:25.308 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:48:25.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:25 smithi136 ceph-mon[109861]: osdmap e1174: 8 total, 8 up, 8 in 2024-08-17T17:48:25.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:25 smithi136 ceph-mon[109861]: pgmap v1801: 105 pgs: 105 active+clean; 30 MiB data, 692 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 9.2 KiB/s wr, 98 op/s 2024-08-17T17:48:25.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:25 smithi136 ceph-mon[115078]: osdmap e1174: 8 total, 8 up, 8 in 2024-08-17T17:48:25.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:25 smithi136 ceph-mon[115078]: pgmap v1801: 105 pgs: 105 active+clean; 30 MiB data, 692 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 9.2 KiB/s wr, 98 op/s 2024-08-17T17:48:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:25 smithi165 ceph-mon[105517]: osdmap e1174: 8 total, 8 up, 8 in 2024-08-17T17:48:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:25 smithi165 ceph-mon[105517]: pgmap v1801: 105 pgs: 105 active+clean; 30 MiB data, 692 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 9.2 KiB/s wr, 98 op/s 2024-08-17T17:48:26.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:26 smithi136 ceph-mon[109861]: osdmap e1175: 8 total, 8 up, 8 in 2024-08-17T17:48:26.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:26 smithi136 ceph-mon[115078]: osdmap e1175: 8 total, 8 up, 8 in 2024-08-17T17:48:26.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:26 smithi165 ceph-mon[105517]: osdmap e1175: 8 total, 8 up, 8 in 2024-08-17T17:48:27.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:27 smithi136 ceph-mon[115078]: osdmap e1176: 8 total, 8 up, 8 in 2024-08-17T17:48:27.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:27 smithi136 ceph-mon[115078]: pgmap v1804: 105 pgs: 105 active+clean; 30 MiB data, 692 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 3.7 KiB/s wr, 48 op/s 2024-08-17T17:48:27.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:27 smithi136 ceph-mon[109861]: osdmap e1176: 8 total, 8 up, 8 in 2024-08-17T17:48:27.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:27 smithi136 ceph-mon[109861]: pgmap v1804: 105 pgs: 105 active+clean; 30 MiB data, 692 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 3.7 KiB/s wr, 48 op/s 2024-08-17T17:48:27.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:48:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:48:27] "GET /metrics HTTP/1.1" 200 36471 "" "Prometheus/2.51.0" 2024-08-17T17:48:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:27 smithi165 ceph-mon[105517]: osdmap e1176: 8 total, 8 up, 8 in 2024-08-17T17:48:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:27 smithi165 ceph-mon[105517]: pgmap v1804: 105 pgs: 105 active+clean; 30 MiB data, 692 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 3.7 KiB/s wr, 48 op/s 2024-08-17T17:48:28.339 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:48:28.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:28 smithi136 ceph-mon[109861]: osdmap e1177: 8 total, 8 up, 8 in 2024-08-17T17:48:28.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:28 smithi136 ceph-mon[115078]: osdmap e1177: 8 total, 8 up, 8 in 2024-08-17T17:48:28.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:28 smithi165 ceph-mon[105517]: osdmap e1177: 8 total, 8 up, 8 in 2024-08-17T17:48:29.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:29 smithi136 ceph-mon[109861]: osdmap e1178: 8 total, 8 up, 8 in 2024-08-17T17:48:29.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:29 smithi136 ceph-mon[109861]: pgmap v1807: 105 pgs: 105 active+clean; 30 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 8.5 KiB/s wr, 108 op/s 2024-08-17T17:48:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:29 smithi136 ceph-mon[115078]: osdmap e1178: 8 total, 8 up, 8 in 2024-08-17T17:48:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:29 smithi136 ceph-mon[115078]: pgmap v1807: 105 pgs: 105 active+clean; 30 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 8.5 KiB/s wr, 108 op/s 2024-08-17T17:48:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:29 smithi165 ceph-mon[105517]: osdmap e1178: 8 total, 8 up, 8 in 2024-08-17T17:48:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:29 smithi165 ceph-mon[105517]: pgmap v1807: 105 pgs: 105 active+clean; 30 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 8.5 KiB/s wr, 108 op/s 2024-08-17T17:48:30.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:30 smithi136 ceph-mon[109861]: osdmap e1179: 8 total, 8 up, 8 in 2024-08-17T17:48:30.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:30 smithi136 ceph-mon[115078]: osdmap e1179: 8 total, 8 up, 8 in 2024-08-17T17:48:30.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:30 smithi165 ceph-mon[105517]: osdmap e1179: 8 total, 8 up, 8 in 2024-08-17T17:48:31.360 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:48:31.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:31 smithi136 ceph-mon[115078]: osdmap e1180: 8 total, 8 up, 8 in 2024-08-17T17:48:31.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:31 smithi136 ceph-mon[115078]: pgmap v1810: 105 pgs: 105 active+clean; 30 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 4.7 KiB/s wr, 59 op/s 2024-08-17T17:48:31.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:31 smithi136 ceph-mon[109861]: osdmap e1180: 8 total, 8 up, 8 in 2024-08-17T17:48:31.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:31 smithi136 ceph-mon[109861]: pgmap v1810: 105 pgs: 105 active+clean; 30 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 4.7 KiB/s wr, 59 op/s 2024-08-17T17:48:31.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:31 smithi165 ceph-mon[105517]: osdmap e1180: 8 total, 8 up, 8 in 2024-08-17T17:48:31.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:31 smithi165 ceph-mon[105517]: pgmap v1810: 105 pgs: 105 active+clean; 30 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 4.7 KiB/s wr, 59 op/s 2024-08-17T17:48:32.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:32 smithi136 ceph-mon[109861]: osdmap e1181: 8 total, 8 up, 8 in 2024-08-17T17:48:32.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:32 smithi136 ceph-mon[115078]: osdmap e1181: 8 total, 8 up, 8 in 2024-08-17T17:48:32.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:32 smithi165 ceph-mon[105517]: osdmap e1181: 8 total, 8 up, 8 in 2024-08-17T17:48:33.662 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:33 smithi165 ceph-mon[105517]: osdmap e1182: 8 total, 8 up, 8 in 2024-08-17T17:48:33.662 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:33 smithi165 ceph-mon[105517]: pgmap v1813: 105 pgs: 105 active+clean; 42 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 8.2 KiB/s wr, 93 op/s 2024-08-17T17:48:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:33 smithi136 ceph-mon[109861]: osdmap e1182: 8 total, 8 up, 8 in 2024-08-17T17:48:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:33 smithi136 ceph-mon[109861]: pgmap v1813: 105 pgs: 105 active+clean; 42 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 8.2 KiB/s wr, 93 op/s 2024-08-17T17:48:33.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:33 smithi136 ceph-mon[115078]: osdmap e1182: 8 total, 8 up, 8 in 2024-08-17T17:48:33.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:33 smithi136 ceph-mon[115078]: pgmap v1813: 105 pgs: 105 active+clean; 42 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 8.2 KiB/s wr, 93 op/s 2024-08-17T17:48:34.372 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:48:34.674 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:34 smithi165 ceph-mon[105517]: osdmap e1183: 8 total, 8 up, 8 in 2024-08-17T17:48:34.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:34 smithi136 ceph-mon[109861]: osdmap e1183: 8 total, 8 up, 8 in 2024-08-17T17:48:34.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:34 smithi136 ceph-mon[115078]: osdmap e1183: 8 total, 8 up, 8 in 2024-08-17T17:48:35.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:35 smithi136 ceph-mon[109861]: osdmap e1184: 8 total, 8 up, 8 in 2024-08-17T17:48:35.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:35 smithi136 ceph-mon[109861]: pgmap v1816: 105 pgs: 105 active+clean; 42 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 8.2 KiB/s wr, 93 op/s 2024-08-17T17:48:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:35 smithi136 ceph-mon[115078]: osdmap e1184: 8 total, 8 up, 8 in 2024-08-17T17:48:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:35 smithi136 ceph-mon[115078]: pgmap v1816: 105 pgs: 105 active+clean; 42 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 8.2 KiB/s wr, 93 op/s 2024-08-17T17:48:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:35 smithi165 ceph-mon[105517]: osdmap e1184: 8 total, 8 up, 8 in 2024-08-17T17:48:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:35 smithi165 ceph-mon[105517]: pgmap v1816: 105 pgs: 105 active+clean; 42 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 8.2 KiB/s wr, 93 op/s 2024-08-17T17:48:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:36 smithi136 ceph-mon[109861]: osdmap e1185: 8 total, 8 up, 8 in 2024-08-17T17:48:36.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:48:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:36 smithi136 ceph-mon[115078]: osdmap e1185: 8 total, 8 up, 8 in 2024-08-17T17:48:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:48:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:36 smithi165 ceph-mon[105517]: osdmap e1185: 8 total, 8 up, 8 in 2024-08-17T17:48:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:48:37.393 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/1.DiffIterateDeterministicPP (15110 ms) 2024-08-17T17:48:37.393 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/1.DiffIterateDeterministicLUKS1 2024-08-17T17:48:37.394 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:48:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:37 smithi136 ceph-mon[109861]: osdmap e1186: 8 total, 8 up, 8 in 2024-08-17T17:48:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:37 smithi136 ceph-mon[109861]: pgmap v1819: 105 pgs: 105 active+clean; 45 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 2.7 KiB/s wr, 33 op/s 2024-08-17T17:48:37.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:37 smithi136 ceph-mon[115078]: osdmap e1186: 8 total, 8 up, 8 in 2024-08-17T17:48:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:37 smithi136 ceph-mon[115078]: pgmap v1819: 105 pgs: 105 active+clean; 45 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 2.7 KiB/s wr, 33 op/s 2024-08-17T17:48:37.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:48:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:48:37] "GET /metrics HTTP/1.1" 200 36474 "" "Prometheus/2.51.0" 2024-08-17T17:48:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:37 smithi165 ceph-mon[105517]: osdmap e1186: 8 total, 8 up, 8 in 2024-08-17T17:48:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:37 smithi165 ceph-mon[105517]: pgmap v1819: 105 pgs: 105 active+clean; 45 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 2.7 KiB/s wr, 33 op/s 2024-08-17T17:48:38.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:38 smithi136 ceph-mon[109861]: osdmap e1187: 8 total, 8 up, 8 in 2024-08-17T17:48:38.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:38 smithi136 ceph-mon[115078]: osdmap e1187: 8 total, 8 up, 8 in 2024-08-17T17:48:38.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:38 smithi165 ceph-mon[105517]: osdmap e1187: 8 total, 8 up, 8 in 2024-08-17T17:48:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:39 smithi136 ceph-mon[109861]: pgmap v1821: 105 pgs: 105 active+clean; 56 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 9.4 KiB/s wr, 104 op/s 2024-08-17T17:48:39.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:39 smithi136 ceph-mon[115078]: pgmap v1821: 105 pgs: 105 active+clean; 56 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 9.4 KiB/s wr, 104 op/s 2024-08-17T17:48:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:39 smithi165 ceph-mon[105517]: pgmap v1821: 105 pgs: 105 active+clean; 56 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 9.4 KiB/s wr, 104 op/s 2024-08-17T17:48:40.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:40 smithi136 ceph-mon[109861]: pgmap v1822: 105 pgs: 105 active+clean; 56 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 6.5 KiB/s wr, 72 op/s 2024-08-17T17:48:40.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:40 smithi136 ceph-mon[115078]: pgmap v1822: 105 pgs: 105 active+clean; 56 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 6.5 KiB/s wr, 72 op/s 2024-08-17T17:48:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:40 smithi165 ceph-mon[105517]: pgmap v1822: 105 pgs: 105 active+clean; 56 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 6.5 KiB/s wr, 72 op/s 2024-08-17T17:48:42.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:42 smithi136 ceph-mon[109861]: osdmap e1188: 8 total, 8 up, 8 in 2024-08-17T17:48:42.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:42 smithi136 ceph-mon[109861]: pgmap v1824: 105 pgs: 105 active+clean; 60 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 673 KiB/s wr, 80 op/s 2024-08-17T17:48:42.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:42 smithi136 ceph-mon[115078]: osdmap e1188: 8 total, 8 up, 8 in 2024-08-17T17:48:42.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:42 smithi136 ceph-mon[115078]: pgmap v1824: 105 pgs: 105 active+clean; 60 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 673 KiB/s wr, 80 op/s 2024-08-17T17:48:43.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:42 smithi165 ceph-mon[105517]: osdmap e1188: 8 total, 8 up, 8 in 2024-08-17T17:48:43.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:42 smithi165 ceph-mon[105517]: pgmap v1824: 105 pgs: 105 active+clean; 60 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 673 KiB/s wr, 80 op/s 2024-08-17T17:48:43.623 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:48:43.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:43 smithi136 ceph-mon[109861]: osdmap e1189: 8 total, 8 up, 8 in 2024-08-17T17:48:43.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:43 smithi136 ceph-mon[115078]: osdmap e1189: 8 total, 8 up, 8 in 2024-08-17T17:48:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:43 smithi165 ceph-mon[105517]: osdmap e1189: 8 total, 8 up, 8 in 2024-08-17T17:48:44.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:44 smithi136 ceph-mon[109861]: osdmap e1190: 8 total, 8 up, 8 in 2024-08-17T17:48:44.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:44 smithi136 ceph-mon[109861]: pgmap v1827: 105 pgs: 105 active+clean; 60 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 685 KiB/s wr, 32 op/s 2024-08-17T17:48:44.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:44 smithi136 ceph-mon[115078]: osdmap e1190: 8 total, 8 up, 8 in 2024-08-17T17:48:44.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:44 smithi136 ceph-mon[115078]: pgmap v1827: 105 pgs: 105 active+clean; 60 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 685 KiB/s wr, 32 op/s 2024-08-17T17:48:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:44 smithi165 ceph-mon[105517]: osdmap e1190: 8 total, 8 up, 8 in 2024-08-17T17:48:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:44 smithi165 ceph-mon[105517]: pgmap v1827: 105 pgs: 105 active+clean; 60 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 685 KiB/s wr, 32 op/s 2024-08-17T17:48:46.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:46 smithi136 ceph-mon[109861]: pgmap v1828: 105 pgs: 105 active+clean; 60 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 44 op/s 2024-08-17T17:48:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:46 smithi136 ceph-mon[115078]: pgmap v1828: 105 pgs: 105 active+clean; 60 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 44 op/s 2024-08-17T17:48:47.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:46 smithi165 ceph-mon[105517]: pgmap v1828: 105 pgs: 105 active+clean; 60 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 44 op/s 2024-08-17T17:48:47.847 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:48:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:48:47] "GET /metrics HTTP/1.1" 200 36474 "" "Prometheus/2.51.0" 2024-08-17T17:48:48.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:48 smithi136 ceph-mon[109861]: osdmap e1191: 8 total, 8 up, 8 in 2024-08-17T17:48:48.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:48 smithi136 ceph-mon[109861]: pgmap v1830: 105 pgs: 105 active+clean; 60 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 2.7 KiB/s wr, 31 op/s 2024-08-17T17:48:48.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:48 smithi136 ceph-mon[115078]: osdmap e1191: 8 total, 8 up, 8 in 2024-08-17T17:48:48.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:48 smithi136 ceph-mon[115078]: pgmap v1830: 105 pgs: 105 active+clean; 60 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 2.7 KiB/s wr, 31 op/s 2024-08-17T17:48:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:48 smithi165 ceph-mon[105517]: osdmap e1191: 8 total, 8 up, 8 in 2024-08-17T17:48:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:48 smithi165 ceph-mon[105517]: pgmap v1830: 105 pgs: 105 active+clean; 60 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 2.7 KiB/s wr, 31 op/s 2024-08-17T17:48:49.620 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:48:49.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:49 smithi136 ceph-mon[109861]: osdmap e1192: 8 total, 8 up, 8 in 2024-08-17T17:48:49.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:49 smithi136 ceph-mon[115078]: osdmap e1192: 8 total, 8 up, 8 in 2024-08-17T17:48:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:49 smithi165 ceph-mon[105517]: osdmap e1192: 8 total, 8 up, 8 in 2024-08-17T17:48:50.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:50 smithi136 ceph-mon[109861]: osdmap e1193: 8 total, 8 up, 8 in 2024-08-17T17:48:50.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:50 smithi136 ceph-mon[109861]: pgmap v1833: 105 pgs: 105 active+clean; 60 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 2.7 KiB/s wr, 31 op/s 2024-08-17T17:48:50.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:50 smithi136 ceph-mon[115078]: osdmap e1193: 8 total, 8 up, 8 in 2024-08-17T17:48:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:50 smithi136 ceph-mon[115078]: pgmap v1833: 105 pgs: 105 active+clean; 60 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 2.7 KiB/s wr, 31 op/s 2024-08-17T17:48:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:50 smithi165 ceph-mon[105517]: osdmap e1193: 8 total, 8 up, 8 in 2024-08-17T17:48:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:50 smithi165 ceph-mon[105517]: pgmap v1833: 105 pgs: 105 active+clean; 60 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 2.7 KiB/s wr, 31 op/s 2024-08-17T17:48:51.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:48:51.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:48:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:48:52.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:52 smithi136 ceph-mon[109861]: pgmap v1834: 105 pgs: 105 active+clean; 66 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 688 KiB/s wr, 61 op/s 2024-08-17T17:48:52.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:52 smithi136 ceph-mon[115078]: pgmap v1834: 105 pgs: 105 active+clean; 66 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 688 KiB/s wr, 61 op/s 2024-08-17T17:48:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:52 smithi165 ceph-mon[105517]: pgmap v1834: 105 pgs: 105 active+clean; 66 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 688 KiB/s wr, 61 op/s 2024-08-17T17:48:54.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:54 smithi136 ceph-mon[109861]: osdmap e1194: 8 total, 8 up, 8 in 2024-08-17T17:48:54.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:54 smithi136 ceph-mon[109861]: pgmap v1836: 105 pgs: 105 active+clean; 66 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-08-17T17:48:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:54 smithi136 ceph-mon[115078]: osdmap e1194: 8 total, 8 up, 8 in 2024-08-17T17:48:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:54 smithi136 ceph-mon[115078]: pgmap v1836: 105 pgs: 105 active+clean; 66 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-08-17T17:48:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:54 smithi165 ceph-mon[105517]: osdmap e1194: 8 total, 8 up, 8 in 2024-08-17T17:48:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:54 smithi165 ceph-mon[105517]: pgmap v1836: 105 pgs: 105 active+clean; 66 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-08-17T17:48:55.649 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:48:55.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:55 smithi136 ceph-mon[109861]: osdmap e1195: 8 total, 8 up, 8 in 2024-08-17T17:48:55.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:55 smithi136 ceph-mon[115078]: osdmap e1195: 8 total, 8 up, 8 in 2024-08-17T17:48:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:55 smithi165 ceph-mon[105517]: osdmap e1195: 8 total, 8 up, 8 in 2024-08-17T17:48:56.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:56 smithi136 ceph-mon[109861]: osdmap e1196: 8 total, 8 up, 8 in 2024-08-17T17:48:56.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:56 smithi136 ceph-mon[109861]: pgmap v1839: 105 pgs: 105 active+clean; 70 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 1.3 MiB/s wr, 72 op/s 2024-08-17T17:48:56.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:56 smithi136 ceph-mon[115078]: osdmap e1196: 8 total, 8 up, 8 in 2024-08-17T17:48:56.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:56 smithi136 ceph-mon[115078]: pgmap v1839: 105 pgs: 105 active+clean; 70 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 1.3 MiB/s wr, 72 op/s 2024-08-17T17:48:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:56 smithi165 ceph-mon[105517]: osdmap e1196: 8 total, 8 up, 8 in 2024-08-17T17:48:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:56 smithi165 ceph-mon[105517]: pgmap v1839: 105 pgs: 105 active+clean; 70 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 1.3 MiB/s wr, 72 op/s 2024-08-17T17:48:57.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:48:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:48:57] "GET /metrics HTTP/1.1" 200 36479 "" "Prometheus/2.51.0" 2024-08-17T17:48:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:48:58 smithi136 ceph-mon[109861]: pgmap v1840: 105 pgs: 105 active+clean; 72 MiB data, 736 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 686 KiB/s wr, 38 op/s 2024-08-17T17:48:58.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:48:58 smithi136 ceph-mon[115078]: pgmap v1840: 105 pgs: 105 active+clean; 72 MiB data, 736 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 686 KiB/s wr, 38 op/s 2024-08-17T17:48:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:48:58 smithi165 ceph-mon[105517]: pgmap v1840: 105 pgs: 105 active+clean; 72 MiB data, 736 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 686 KiB/s wr, 38 op/s 2024-08-17T17:49:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:00 smithi136 ceph-mon[109861]: osdmap e1197: 8 total, 8 up, 8 in 2024-08-17T17:49:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:00 smithi136 ceph-mon[109861]: pgmap v1842: 105 pgs: 105 active+clean; 72 MiB data, 736 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 686 KiB/s wr, 38 op/s 2024-08-17T17:49:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:00 smithi136 ceph-mon[115078]: osdmap e1197: 8 total, 8 up, 8 in 2024-08-17T17:49:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:00 smithi136 ceph-mon[115078]: pgmap v1842: 105 pgs: 105 active+clean; 72 MiB data, 736 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 686 KiB/s wr, 38 op/s 2024-08-17T17:49:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:00 smithi165 ceph-mon[105517]: osdmap e1197: 8 total, 8 up, 8 in 2024-08-17T17:49:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:00 smithi165 ceph-mon[105517]: pgmap v1842: 105 pgs: 105 active+clean; 72 MiB data, 736 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 686 KiB/s wr, 38 op/s 2024-08-17T17:49:01.623 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:49:01.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:01 smithi136 ceph-mon[109861]: osdmap e1198: 8 total, 8 up, 8 in 2024-08-17T17:49:01.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:01 smithi136 ceph-mon[115078]: osdmap e1198: 8 total, 8 up, 8 in 2024-08-17T17:49:02.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:01 smithi165 ceph-mon[105517]: osdmap e1198: 8 total, 8 up, 8 in 2024-08-17T17:49:02.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:02 smithi136 ceph-mon[109861]: osdmap e1199: 8 total, 8 up, 8 in 2024-08-17T17:49:02.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:02 smithi136 ceph-mon[109861]: pgmap v1845: 105 pgs: 105 active+clean; 78 MiB data, 748 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 42 op/s 2024-08-17T17:49:02.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:02 smithi136 ceph-mon[115078]: osdmap e1199: 8 total, 8 up, 8 in 2024-08-17T17:49:02.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:02 smithi136 ceph-mon[115078]: pgmap v1845: 105 pgs: 105 active+clean; 78 MiB data, 748 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 42 op/s 2024-08-17T17:49:03.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:02 smithi165 ceph-mon[105517]: osdmap e1199: 8 total, 8 up, 8 in 2024-08-17T17:49:03.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:02 smithi165 ceph-mon[105517]: pgmap v1845: 105 pgs: 105 active+clean; 78 MiB data, 748 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 42 op/s 2024-08-17T17:49:04.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:04 smithi136 ceph-mon[109861]: pgmap v1846: 105 pgs: 105 active+clean; 78 MiB data, 748 MiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 686 KiB/s wr, 34 op/s 2024-08-17T17:49:04.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:04 smithi136 ceph-mon[115078]: pgmap v1846: 105 pgs: 105 active+clean; 78 MiB data, 748 MiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 686 KiB/s wr, 34 op/s 2024-08-17T17:49:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:04 smithi165 ceph-mon[105517]: pgmap v1846: 105 pgs: 105 active+clean; 78 MiB data, 748 MiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 686 KiB/s wr, 34 op/s 2024-08-17T17:49:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:06 smithi136 ceph-mon[109861]: osdmap e1200: 8 total, 8 up, 8 in 2024-08-17T17:49:06.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:49:06.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:06 smithi136 ceph-mon[109861]: pgmap v1848: 105 pgs: 105 active+clean; 82 MiB data, 752 MiB used, 714 GiB / 715 GiB avail; 35 KiB/s rd, 1.3 MiB/s wr, 53 op/s 2024-08-17T17:49:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:06 smithi136 ceph-mon[115078]: osdmap e1200: 8 total, 8 up, 8 in 2024-08-17T17:49:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:49:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:06 smithi136 ceph-mon[115078]: pgmap v1848: 105 pgs: 105 active+clean; 82 MiB data, 752 MiB used, 714 GiB / 715 GiB avail; 35 KiB/s rd, 1.3 MiB/s wr, 53 op/s 2024-08-17T17:49:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:06 smithi165 ceph-mon[105517]: osdmap e1200: 8 total, 8 up, 8 in 2024-08-17T17:49:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:49:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:06 smithi165 ceph-mon[105517]: pgmap v1848: 105 pgs: 105 active+clean; 82 MiB data, 752 MiB used, 714 GiB / 715 GiB avail; 35 KiB/s rd, 1.3 MiB/s wr, 53 op/s 2024-08-17T17:49:07.621 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/1.DiffIterateDeterministicLUKS1 (30228 ms) 2024-08-17T17:49:07.621 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/1.DiffIterateDeterministicLUKS1PP 2024-08-17T17:49:07.621 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:49:07.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:07 smithi136 ceph-mon[109861]: osdmap e1201: 8 total, 8 up, 8 in 2024-08-17T17:49:07.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:07 smithi136 ceph-mon[115078]: osdmap e1201: 8 total, 8 up, 8 in 2024-08-17T17:49:07.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:49:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:49:07] "GET /metrics HTTP/1.1" 200 36480 "" "Prometheus/2.51.0" 2024-08-17T17:49:08.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:07 smithi165 ceph-mon[105517]: osdmap e1201: 8 total, 8 up, 8 in 2024-08-17T17:49:08.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:08 smithi136 ceph-mon[109861]: osdmap e1202: 8 total, 8 up, 8 in 2024-08-17T17:49:08.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:08 smithi136 ceph-mon[109861]: pgmap v1851: 105 pgs: 105 active+clean; 86 MiB data, 760 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 684 KiB/s wr, 19 op/s 2024-08-17T17:49:08.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:08 smithi136 ceph-mon[115078]: osdmap e1202: 8 total, 8 up, 8 in 2024-08-17T17:49:08.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:08 smithi136 ceph-mon[115078]: pgmap v1851: 105 pgs: 105 active+clean; 86 MiB data, 760 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 684 KiB/s wr, 19 op/s 2024-08-17T17:49:09.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:08 smithi165 ceph-mon[105517]: osdmap e1202: 8 total, 8 up, 8 in 2024-08-17T17:49:09.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:08 smithi165 ceph-mon[105517]: pgmap v1851: 105 pgs: 105 active+clean; 86 MiB data, 760 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 684 KiB/s wr, 19 op/s 2024-08-17T17:49:10.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:10 smithi136 ceph-mon[109861]: pgmap v1852: 105 pgs: 105 active+clean; 86 MiB data, 760 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 684 KiB/s wr, 19 op/s 2024-08-17T17:49:10.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:10 smithi136 ceph-mon[115078]: pgmap v1852: 105 pgs: 105 active+clean; 86 MiB data, 760 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 684 KiB/s wr, 19 op/s 2024-08-17T17:49:11.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:10 smithi165 ceph-mon[105517]: pgmap v1852: 105 pgs: 105 active+clean; 86 MiB data, 760 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 684 KiB/s wr, 19 op/s 2024-08-17T17:49:12.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:12 smithi136 ceph-mon[115078]: osdmap e1203: 8 total, 8 up, 8 in 2024-08-17T17:49:12.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:12 smithi136 ceph-mon[115078]: pgmap v1854: 105 pgs: 105 active+clean; 90 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 3.2 KiB/s wr, 32 op/s 2024-08-17T17:49:12.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:12 smithi136 ceph-mon[109861]: osdmap e1203: 8 total, 8 up, 8 in 2024-08-17T17:49:12.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:12 smithi136 ceph-mon[109861]: pgmap v1854: 105 pgs: 105 active+clean; 90 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 3.2 KiB/s wr, 32 op/s 2024-08-17T17:49:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:12 smithi165 ceph-mon[105517]: osdmap e1203: 8 total, 8 up, 8 in 2024-08-17T17:49:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:12 smithi165 ceph-mon[105517]: pgmap v1854: 105 pgs: 105 active+clean; 90 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 3.2 KiB/s wr, 32 op/s 2024-08-17T17:49:13.621 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:49:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:13 smithi165 ceph-mon[105517]: osdmap e1204: 8 total, 8 up, 8 in 2024-08-17T17:49:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:13 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:49:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:13 smithi136 ceph-mon[109861]: osdmap e1204: 8 total, 8 up, 8 in 2024-08-17T17:49:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:13 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:49:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:13 smithi136 ceph-mon[115078]: osdmap e1204: 8 total, 8 up, 8 in 2024-08-17T17:49:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:13 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:49:14.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:14 smithi136 ceph-mon[115078]: osdmap e1205: 8 total, 8 up, 8 in 2024-08-17T17:49:14.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:14 smithi136 ceph-mon[115078]: pgmap v1857: 105 pgs: 105 active+clean; 90 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 3.0 KiB/s wr, 32 op/s 2024-08-17T17:49:14.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:14 smithi136 ceph-mon[109861]: osdmap e1205: 8 total, 8 up, 8 in 2024-08-17T17:49:14.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:14 smithi136 ceph-mon[109861]: pgmap v1857: 105 pgs: 105 active+clean; 90 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 3.0 KiB/s wr, 32 op/s 2024-08-17T17:49:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:14 smithi165 ceph-mon[105517]: osdmap e1205: 8 total, 8 up, 8 in 2024-08-17T17:49:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:14 smithi165 ceph-mon[105517]: pgmap v1857: 105 pgs: 105 active+clean; 90 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 3.0 KiB/s wr, 32 op/s 2024-08-17T17:49:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:17 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:17 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:17 smithi165 ceph-mon[105517]: pgmap v1858: 105 pgs: 105 active+clean; 94 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 49 KiB/s rd, 689 KiB/s wr, 71 op/s 2024-08-17T17:49:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:17 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:17 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:17.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:17 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:17.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:17 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:17.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:17 smithi136 ceph-mon[109861]: pgmap v1858: 105 pgs: 105 active+clean; 94 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 49 KiB/s rd, 689 KiB/s wr, 71 op/s 2024-08-17T17:49:17.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:17 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:17.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:17 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:17.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:49:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:49:17] "GET /metrics HTTP/1.1" 200 36480 "" "Prometheus/2.51.0" 2024-08-17T17:49:17.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:17 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:17.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:17 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:17.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:17 smithi136 ceph-mon[115078]: pgmap v1858: 105 pgs: 105 active+clean; 94 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 49 KiB/s rd, 689 KiB/s wr, 71 op/s 2024-08-17T17:49:17.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:17 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:17.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:17 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:18.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:18.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:18.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:18 smithi136 ceph-mon[109861]: osdmap e1206: 8 total, 8 up, 8 in 2024-08-17T17:49:18.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:18 smithi136 ceph-mon[109861]: pgmap v1860: 105 pgs: 105 active+clean; 94 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 686 KiB/s wr, 41 op/s 2024-08-17T17:49:18.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:18.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:18.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:18 smithi136 ceph-mon[115078]: osdmap e1206: 8 total, 8 up, 8 in 2024-08-17T17:49:18.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:18 smithi136 ceph-mon[115078]: pgmap v1860: 105 pgs: 105 active+clean; 94 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 686 KiB/s wr, 41 op/s 2024-08-17T17:49:18.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:18.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:18.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:18 smithi165 ceph-mon[105517]: osdmap e1206: 8 total, 8 up, 8 in 2024-08-17T17:49:18.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:18 smithi165 ceph-mon[105517]: pgmap v1860: 105 pgs: 105 active+clean; 94 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 686 KiB/s wr, 41 op/s 2024-08-17T17:49:20.122 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:49:20.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:20 smithi165 ceph-mon[105517]: osdmap e1207: 8 total, 8 up, 8 in 2024-08-17T17:49:20.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:20 smithi136 ceph-mon[109861]: osdmap e1207: 8 total, 8 up, 8 in 2024-08-17T17:49:20.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:20 smithi136 ceph-mon[115078]: osdmap e1207: 8 total, 8 up, 8 in 2024-08-17T17:49:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:21 smithi165 ceph-mon[105517]: osdmap e1208: 8 total, 8 up, 8 in 2024-08-17T17:49:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:21 smithi165 ceph-mon[105517]: pgmap v1863: 105 pgs: 105 active+clean; 94 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 686 KiB/s wr, 41 op/s 2024-08-17T17:49:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:49:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:49:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:21.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:21 smithi136 ceph-mon[115078]: osdmap e1208: 8 total, 8 up, 8 in 2024-08-17T17:49:21.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:21 smithi136 ceph-mon[115078]: pgmap v1863: 105 pgs: 105 active+clean; 94 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 686 KiB/s wr, 41 op/s 2024-08-17T17:49:21.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:21.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:21.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:49:21.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:49:21.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:21.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:21 smithi136 ceph-mon[109861]: osdmap e1208: 8 total, 8 up, 8 in 2024-08-17T17:49:21.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:21 smithi136 ceph-mon[109861]: pgmap v1863: 105 pgs: 105 active+clean; 94 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 686 KiB/s wr, 41 op/s 2024-08-17T17:49:21.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:21.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:21.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:49:21.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:49:21.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:49:22.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:49:22.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:49:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:49:23.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:23 smithi136 ceph-mon[109861]: pgmap v1864: 105 pgs: 105 active+clean; 104 MiB data, 786 MiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 687 KiB/s wr, 44 op/s 2024-08-17T17:49:23.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:23 smithi136 ceph-mon[115078]: pgmap v1864: 105 pgs: 105 active+clean; 104 MiB data, 786 MiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 687 KiB/s wr, 44 op/s 2024-08-17T17:49:23.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:23 smithi165 ceph-mon[105517]: pgmap v1864: 105 pgs: 105 active+clean; 104 MiB data, 786 MiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 687 KiB/s wr, 44 op/s 2024-08-17T17:49:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:25 smithi136 ceph-mon[109861]: osdmap e1209: 8 total, 8 up, 8 in 2024-08-17T17:49:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:25 smithi136 ceph-mon[109861]: pgmap v1866: 105 pgs: 105 active+clean; 104 MiB data, 786 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-08-17T17:49:25.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:25 smithi136 ceph-mon[115078]: osdmap e1209: 8 total, 8 up, 8 in 2024-08-17T17:49:25.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:25 smithi136 ceph-mon[115078]: pgmap v1866: 105 pgs: 105 active+clean; 104 MiB data, 786 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-08-17T17:49:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:25 smithi165 ceph-mon[105517]: osdmap e1209: 8 total, 8 up, 8 in 2024-08-17T17:49:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:25 smithi165 ceph-mon[105517]: pgmap v1866: 105 pgs: 105 active+clean; 104 MiB data, 786 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-08-17T17:49:26.152 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:49:26.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:26 smithi136 ceph-mon[109861]: osdmap e1210: 8 total, 8 up, 8 in 2024-08-17T17:49:26.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:26 smithi136 ceph-mon[115078]: osdmap e1210: 8 total, 8 up, 8 in 2024-08-17T17:49:26.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:26 smithi165 ceph-mon[105517]: osdmap e1210: 8 total, 8 up, 8 in 2024-08-17T17:49:27.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:27 smithi136 ceph-mon[109861]: osdmap e1211: 8 total, 8 up, 8 in 2024-08-17T17:49:27.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:27 smithi136 ceph-mon[109861]: pgmap v1869: 105 pgs: 105 active+clean; 114 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 1.3 MiB/s wr, 54 op/s 2024-08-17T17:49:27.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:27 smithi136 ceph-mon[115078]: osdmap e1211: 8 total, 8 up, 8 in 2024-08-17T17:49:27.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:27 smithi136 ceph-mon[115078]: pgmap v1869: 105 pgs: 105 active+clean; 114 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 1.3 MiB/s wr, 54 op/s 2024-08-17T17:49:27.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:27 smithi165 ceph-mon[105517]: osdmap e1211: 8 total, 8 up, 8 in 2024-08-17T17:49:27.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:27 smithi165 ceph-mon[105517]: pgmap v1869: 105 pgs: 105 active+clean; 114 MiB data, 790 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 1.3 MiB/s wr, 54 op/s 2024-08-17T17:49:27.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:49:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:49:27] "GET /metrics HTTP/1.1" 200 36481 "" "Prometheus/2.51.0" 2024-08-17T17:49:28.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:28 smithi136 ceph-mon[109861]: pgmap v1870: 105 pgs: 105 active+clean; 114 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 686 KiB/s wr, 30 op/s 2024-08-17T17:49:28.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:28 smithi136 ceph-mon[115078]: pgmap v1870: 105 pgs: 105 active+clean; 114 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 686 KiB/s wr, 30 op/s 2024-08-17T17:49:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:28 smithi165 ceph-mon[105517]: pgmap v1870: 105 pgs: 105 active+clean; 114 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 686 KiB/s wr, 30 op/s 2024-08-17T17:49:30.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:30 smithi136 ceph-mon[109861]: osdmap e1212: 8 total, 8 up, 8 in 2024-08-17T17:49:30.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:30 smithi136 ceph-mon[109861]: pgmap v1872: 105 pgs: 105 active+clean; 114 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 686 KiB/s wr, 30 op/s 2024-08-17T17:49:30.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:30 smithi136 ceph-mon[115078]: osdmap e1212: 8 total, 8 up, 8 in 2024-08-17T17:49:30.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:30 smithi136 ceph-mon[115078]: pgmap v1872: 105 pgs: 105 active+clean; 114 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 686 KiB/s wr, 30 op/s 2024-08-17T17:49:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:30 smithi165 ceph-mon[105517]: osdmap e1212: 8 total, 8 up, 8 in 2024-08-17T17:49:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:30 smithi165 ceph-mon[105517]: pgmap v1872: 105 pgs: 105 active+clean; 114 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 686 KiB/s wr, 30 op/s 2024-08-17T17:49:31.633 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:49:31.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:31 smithi136 ceph-mon[109861]: osdmap e1213: 8 total, 8 up, 8 in 2024-08-17T17:49:31.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:31 smithi136 ceph-mon[115078]: osdmap e1213: 8 total, 8 up, 8 in 2024-08-17T17:49:32.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:31 smithi165 ceph-mon[105517]: osdmap e1213: 8 total, 8 up, 8 in 2024-08-17T17:49:32.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:32 smithi136 ceph-mon[109861]: osdmap e1214: 8 total, 8 up, 8 in 2024-08-17T17:49:32.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:32 smithi136 ceph-mon[109861]: pgmap v1875: 105 pgs: 105 active+clean; 124 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 44 KiB/s rd, 689 KiB/s wr, 66 op/s 2024-08-17T17:49:32.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:32 smithi136 ceph-mon[115078]: osdmap e1214: 8 total, 8 up, 8 in 2024-08-17T17:49:32.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:32 smithi136 ceph-mon[115078]: pgmap v1875: 105 pgs: 105 active+clean; 124 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 44 KiB/s rd, 689 KiB/s wr, 66 op/s 2024-08-17T17:49:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:32 smithi165 ceph-mon[105517]: osdmap e1214: 8 total, 8 up, 8 in 2024-08-17T17:49:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:32 smithi165 ceph-mon[105517]: pgmap v1875: 105 pgs: 105 active+clean; 124 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 44 KiB/s rd, 689 KiB/s wr, 66 op/s 2024-08-17T17:49:34.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:34 smithi136 ceph-mon[109861]: pgmap v1876: 105 pgs: 105 active+clean; 124 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 687 KiB/s wr, 48 op/s 2024-08-17T17:49:34.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:34 smithi136 ceph-mon[115078]: pgmap v1876: 105 pgs: 105 active+clean; 124 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 687 KiB/s wr, 48 op/s 2024-08-17T17:49:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:34 smithi165 ceph-mon[105517]: pgmap v1876: 105 pgs: 105 active+clean; 124 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 687 KiB/s wr, 48 op/s 2024-08-17T17:49:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:36 smithi136 ceph-mon[109861]: osdmap e1215: 8 total, 8 up, 8 in 2024-08-17T17:49:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:49:36.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:36 smithi136 ceph-mon[109861]: pgmap v1878: 105 pgs: 105 active+clean; 128 MiB data, 819 MiB used, 714 GiB / 715 GiB avail; 38 KiB/s rd, 1.3 MiB/s wr, 57 op/s 2024-08-17T17:49:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:36 smithi136 ceph-mon[115078]: osdmap e1215: 8 total, 8 up, 8 in 2024-08-17T17:49:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:49:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:36 smithi136 ceph-mon[115078]: pgmap v1878: 105 pgs: 105 active+clean; 128 MiB data, 819 MiB used, 714 GiB / 715 GiB avail; 38 KiB/s rd, 1.3 MiB/s wr, 57 op/s 2024-08-17T17:49:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:36 smithi165 ceph-mon[105517]: osdmap e1215: 8 total, 8 up, 8 in 2024-08-17T17:49:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:49:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:36 smithi165 ceph-mon[105517]: pgmap v1878: 105 pgs: 105 active+clean; 128 MiB data, 819 MiB used, 714 GiB / 715 GiB avail; 38 KiB/s rd, 1.3 MiB/s wr, 57 op/s 2024-08-17T17:49:37.631 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/1.DiffIterateDeterministicLUKS1PP (30010 ms) 2024-08-17T17:49:37.631 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/1.DiffIterateDeterministicLUKS2 2024-08-17T17:49:37.632 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:49:37.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:37 smithi136 ceph-mon[109861]: osdmap e1216: 8 total, 8 up, 8 in 2024-08-17T17:49:37.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:37 smithi136 ceph-mon[115078]: osdmap e1216: 8 total, 8 up, 8 in 2024-08-17T17:49:37.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:49:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:49:37] "GET /metrics HTTP/1.1" 200 36484 "" "Prometheus/2.51.0" 2024-08-17T17:49:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:37 smithi165 ceph-mon[105517]: osdmap e1216: 8 total, 8 up, 8 in 2024-08-17T17:49:38.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:38 smithi136 ceph-mon[115078]: osdmap e1217: 8 total, 8 up, 8 in 2024-08-17T17:49:38.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:38 smithi136 ceph-mon[115078]: pgmap v1881: 105 pgs: 105 active+clean; 136 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 685 KiB/s wr, 21 op/s 2024-08-17T17:49:38.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:38 smithi136 ceph-mon[109861]: osdmap e1217: 8 total, 8 up, 8 in 2024-08-17T17:49:38.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:38 smithi136 ceph-mon[109861]: pgmap v1881: 105 pgs: 105 active+clean; 136 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 685 KiB/s wr, 21 op/s 2024-08-17T17:49:39.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:38 smithi165 ceph-mon[105517]: osdmap e1217: 8 total, 8 up, 8 in 2024-08-17T17:49:39.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:38 smithi165 ceph-mon[105517]: pgmap v1881: 105 pgs: 105 active+clean; 136 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 685 KiB/s wr, 21 op/s 2024-08-17T17:49:40.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:40 smithi136 ceph-mon[109861]: pgmap v1882: 105 pgs: 105 active+clean; 136 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 685 KiB/s wr, 21 op/s 2024-08-17T17:49:40.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:40 smithi136 ceph-mon[115078]: pgmap v1882: 105 pgs: 105 active+clean; 136 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 685 KiB/s wr, 21 op/s 2024-08-17T17:49:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:40 smithi165 ceph-mon[105517]: pgmap v1882: 105 pgs: 105 active+clean; 136 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 685 KiB/s wr, 21 op/s 2024-08-17T17:49:42.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:42 smithi136 ceph-mon[109861]: pgmap v1883: 105 pgs: 105 active+clean; 136 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 595 KiB/s wr, 40 op/s 2024-08-17T17:49:42.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:42 smithi136 ceph-mon[115078]: pgmap v1883: 105 pgs: 105 active+clean; 136 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 595 KiB/s wr, 40 op/s 2024-08-17T17:49:43.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:42 smithi165 ceph-mon[105517]: pgmap v1883: 105 pgs: 105 active+clean; 136 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 595 KiB/s wr, 40 op/s 2024-08-17T17:49:44.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:44 smithi136 ceph-mon[109861]: osdmap e1218: 8 total, 8 up, 8 in 2024-08-17T17:49:44.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:44 smithi136 ceph-mon[109861]: pgmap v1885: 105 pgs: 105 active+clean; 136 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 2.3 KiB/s wr, 28 op/s 2024-08-17T17:49:44.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:44 smithi136 ceph-mon[115078]: osdmap e1218: 8 total, 8 up, 8 in 2024-08-17T17:49:44.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:44 smithi136 ceph-mon[115078]: pgmap v1885: 105 pgs: 105 active+clean; 136 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 2.3 KiB/s wr, 28 op/s 2024-08-17T17:49:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:44 smithi165 ceph-mon[105517]: osdmap e1218: 8 total, 8 up, 8 in 2024-08-17T17:49:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:44 smithi165 ceph-mon[105517]: pgmap v1885: 105 pgs: 105 active+clean; 136 MiB data, 823 MiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 2.3 KiB/s wr, 28 op/s 2024-08-17T17:49:45.633 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:49:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:45 smithi136 ceph-mon[109861]: osdmap e1219: 8 total, 8 up, 8 in 2024-08-17T17:49:45.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:45 smithi136 ceph-mon[115078]: osdmap e1219: 8 total, 8 up, 8 in 2024-08-17T17:49:46.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:45 smithi165 ceph-mon[105517]: osdmap e1219: 8 total, 8 up, 8 in 2024-08-17T17:49:46.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:46 smithi136 ceph-mon[115078]: osdmap e1220: 8 total, 8 up, 8 in 2024-08-17T17:49:46.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:46 smithi136 ceph-mon[115078]: pgmap v1888: 105 pgs: 105 active+clean; 148 MiB data, 843 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.0 MiB/s wr, 30 op/s 2024-08-17T17:49:46.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:46 smithi136 ceph-mon[109861]: osdmap e1220: 8 total, 8 up, 8 in 2024-08-17T17:49:46.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:46 smithi136 ceph-mon[109861]: pgmap v1888: 105 pgs: 105 active+clean; 148 MiB data, 843 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.0 MiB/s wr, 30 op/s 2024-08-17T17:49:47.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:46 smithi165 ceph-mon[105517]: osdmap e1220: 8 total, 8 up, 8 in 2024-08-17T17:49:47.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:46 smithi165 ceph-mon[105517]: pgmap v1888: 105 pgs: 105 active+clean; 148 MiB data, 843 MiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.0 MiB/s wr, 30 op/s 2024-08-17T17:49:47.847 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:49:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:49:47] "GET /metrics HTTP/1.1" 200 36484 "" "Prometheus/2.51.0" 2024-08-17T17:49:48.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:48 smithi136 ceph-mon[109861]: pgmap v1889: 105 pgs: 105 active+clean; 152 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 MiB/s wr, 42 op/s 2024-08-17T17:49:48.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:48 smithi136 ceph-mon[115078]: pgmap v1889: 105 pgs: 105 active+clean; 152 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 MiB/s wr, 42 op/s 2024-08-17T17:49:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:48 smithi165 ceph-mon[105517]: pgmap v1889: 105 pgs: 105 active+clean; 152 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 MiB/s wr, 42 op/s 2024-08-17T17:49:50.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:50 smithi136 ceph-mon[109861]: pgmap v1890: 105 pgs: 105 active+clean; 152 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.3 MiB/s wr, 36 op/s 2024-08-17T17:49:50.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:50 smithi136 ceph-mon[115078]: pgmap v1890: 105 pgs: 105 active+clean; 152 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.3 MiB/s wr, 36 op/s 2024-08-17T17:49:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:50 smithi165 ceph-mon[105517]: pgmap v1890: 105 pgs: 105 active+clean; 152 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.3 MiB/s wr, 36 op/s 2024-08-17T17:49:51.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:49:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:49:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:49:52.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:52 smithi136 ceph-mon[109861]: osdmap e1221: 8 total, 8 up, 8 in 2024-08-17T17:49:52.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:52 smithi136 ceph-mon[109861]: pgmap v1892: 105 pgs: 105 active+clean; 157 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 2.5 MiB/s wr, 41 op/s 2024-08-17T17:49:52.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:52 smithi136 ceph-mon[115078]: osdmap e1221: 8 total, 8 up, 8 in 2024-08-17T17:49:52.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:52 smithi136 ceph-mon[115078]: pgmap v1892: 105 pgs: 105 active+clean; 157 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 2.5 MiB/s wr, 41 op/s 2024-08-17T17:49:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:52 smithi165 ceph-mon[105517]: osdmap e1221: 8 total, 8 up, 8 in 2024-08-17T17:49:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:52 smithi165 ceph-mon[105517]: pgmap v1892: 105 pgs: 105 active+clean; 157 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 2.5 MiB/s wr, 41 op/s 2024-08-17T17:49:53.635 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:49:53.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:53 smithi136 ceph-mon[109861]: osdmap e1222: 8 total, 8 up, 8 in 2024-08-17T17:49:53.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:53 smithi136 ceph-mon[115078]: osdmap e1222: 8 total, 8 up, 8 in 2024-08-17T17:49:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:53 smithi165 ceph-mon[105517]: osdmap e1222: 8 total, 8 up, 8 in 2024-08-17T17:49:54.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:54 smithi136 ceph-mon[109861]: osdmap e1223: 8 total, 8 up, 8 in 2024-08-17T17:49:54.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:54 smithi136 ceph-mon[109861]: pgmap v1895: 105 pgs: 105 active+clean; 157 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 6.8 KiB/s rd, 685 KiB/s wr, 12 op/s 2024-08-17T17:49:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:54 smithi136 ceph-mon[115078]: osdmap e1223: 8 total, 8 up, 8 in 2024-08-17T17:49:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:54 smithi136 ceph-mon[115078]: pgmap v1895: 105 pgs: 105 active+clean; 157 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 6.8 KiB/s rd, 685 KiB/s wr, 12 op/s 2024-08-17T17:49:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:54 smithi165 ceph-mon[105517]: osdmap e1223: 8 total, 8 up, 8 in 2024-08-17T17:49:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:54 smithi165 ceph-mon[105517]: pgmap v1895: 105 pgs: 105 active+clean; 157 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 6.8 KiB/s rd, 685 KiB/s wr, 12 op/s 2024-08-17T17:49:56.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:56 smithi136 ceph-mon[109861]: pgmap v1896: 105 pgs: 105 active+clean; 170 MiB data, 904 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.7 MiB/s wr, 36 op/s 2024-08-17T17:49:56.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:56 smithi136 ceph-mon[115078]: pgmap v1896: 105 pgs: 105 active+clean; 170 MiB data, 904 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.7 MiB/s wr, 36 op/s 2024-08-17T17:49:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:56 smithi165 ceph-mon[105517]: pgmap v1896: 105 pgs: 105 active+clean; 170 MiB data, 904 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.7 MiB/s wr, 36 op/s 2024-08-17T17:49:57.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:49:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:49:57] "GET /metrics HTTP/1.1" 200 36486 "" "Prometheus/2.51.0" 2024-08-17T17:49:58.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:49:58 smithi136 ceph-mon[109861]: pgmap v1897: 105 pgs: 105 active+clean; 170 MiB data, 904 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 1.7 MiB/s wr, 39 op/s 2024-08-17T17:49:58.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:49:58 smithi136 ceph-mon[115078]: pgmap v1897: 105 pgs: 105 active+clean; 170 MiB data, 904 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 1.7 MiB/s wr, 39 op/s 2024-08-17T17:49:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:49:58 smithi165 ceph-mon[105517]: pgmap v1897: 105 pgs: 105 active+clean; 170 MiB data, 904 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 1.7 MiB/s wr, 39 op/s 2024-08-17T17:50:00.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:00 smithi136 ceph-mon[109861]: osdmap e1224: 8 total, 8 up, 8 in 2024-08-17T17:50:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:00 smithi136 ceph-mon[109861]: overall HEALTH_OK 2024-08-17T17:50:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:00 smithi136 ceph-mon[109861]: pgmap v1899: 105 pgs: 105 active+clean; 170 MiB data, 904 MiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 1.5 MiB/s wr, 28 op/s 2024-08-17T17:50:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:00 smithi136 ceph-mon[115078]: osdmap e1224: 8 total, 8 up, 8 in 2024-08-17T17:50:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:00 smithi136 ceph-mon[115078]: overall HEALTH_OK 2024-08-17T17:50:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:00 smithi136 ceph-mon[115078]: pgmap v1899: 105 pgs: 105 active+clean; 170 MiB data, 904 MiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 1.5 MiB/s wr, 28 op/s 2024-08-17T17:50:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:00 smithi165 ceph-mon[105517]: osdmap e1224: 8 total, 8 up, 8 in 2024-08-17T17:50:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:00 smithi165 ceph-mon[105517]: overall HEALTH_OK 2024-08-17T17:50:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:00 smithi165 ceph-mon[105517]: pgmap v1899: 105 pgs: 105 active+clean; 170 MiB data, 904 MiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 1.5 MiB/s wr, 28 op/s 2024-08-17T17:50:01.630 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:50:01.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:01 smithi136 ceph-mon[109861]: osdmap e1225: 8 total, 8 up, 8 in 2024-08-17T17:50:01.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:01 smithi136 ceph-mon[115078]: osdmap e1225: 8 total, 8 up, 8 in 2024-08-17T17:50:02.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:01 smithi165 ceph-mon[105517]: osdmap e1225: 8 total, 8 up, 8 in 2024-08-17T17:50:02.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:02 smithi136 ceph-mon[109861]: osdmap e1226: 8 total, 8 up, 8 in 2024-08-17T17:50:02.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:02 smithi136 ceph-mon[109861]: pgmap v1902: 105 pgs: 105 active+clean; 188 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 2.7 MiB/s wr, 50 op/s 2024-08-17T17:50:02.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:02 smithi136 ceph-mon[115078]: osdmap e1226: 8 total, 8 up, 8 in 2024-08-17T17:50:02.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:02 smithi136 ceph-mon[115078]: pgmap v1902: 105 pgs: 105 active+clean; 188 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 2.7 MiB/s wr, 50 op/s 2024-08-17T17:50:03.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:02 smithi165 ceph-mon[105517]: osdmap e1226: 8 total, 8 up, 8 in 2024-08-17T17:50:03.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:02 smithi165 ceph-mon[105517]: pgmap v1902: 105 pgs: 105 active+clean; 188 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 2.7 MiB/s wr, 50 op/s 2024-08-17T17:50:04.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:04 smithi136 ceph-mon[109861]: pgmap v1903: 105 pgs: 105 active+clean; 188 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.7 MiB/s wr, 37 op/s 2024-08-17T17:50:04.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:04 smithi136 ceph-mon[115078]: pgmap v1903: 105 pgs: 105 active+clean; 188 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.7 MiB/s wr, 37 op/s 2024-08-17T17:50:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:04 smithi165 ceph-mon[105517]: pgmap v1903: 105 pgs: 105 active+clean; 188 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.7 MiB/s wr, 37 op/s 2024-08-17T17:50:06.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:50:06.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:50:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:50:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:07 smithi165 ceph-mon[105517]: pgmap v1904: 105 pgs: 105 active+clean; 188 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.3 MiB/s wr, 40 op/s 2024-08-17T17:50:07.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:07 smithi136 ceph-mon[109861]: pgmap v1904: 105 pgs: 105 active+clean; 188 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.3 MiB/s wr, 40 op/s 2024-08-17T17:50:07.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:07 smithi136 ceph-mon[115078]: pgmap v1904: 105 pgs: 105 active+clean; 188 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.3 MiB/s wr, 40 op/s 2024-08-17T17:50:07.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:50:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:50:07] "GET /metrics HTTP/1.1" 200 36489 "" "Prometheus/2.51.0" 2024-08-17T17:50:08.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:08 smithi165 ceph-mon[105517]: osdmap e1227: 8 total, 8 up, 8 in 2024-08-17T17:50:08.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:08 smithi136 ceph-mon[109861]: osdmap e1227: 8 total, 8 up, 8 in 2024-08-17T17:50:08.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:08 smithi136 ceph-mon[115078]: osdmap e1227: 8 total, 8 up, 8 in 2024-08-17T17:50:09.281 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:50:09.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:09 smithi165 ceph-mon[105517]: osdmap e1228: 8 total, 8 up, 8 in 2024-08-17T17:50:09.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:09 smithi165 ceph-mon[105517]: pgmap v1907: 105 pgs: 105 active+clean; 188 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 597 KiB/s wr, 39 op/s 2024-08-17T17:50:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:09 smithi136 ceph-mon[109861]: osdmap e1228: 8 total, 8 up, 8 in 2024-08-17T17:50:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:09 smithi136 ceph-mon[109861]: pgmap v1907: 105 pgs: 105 active+clean; 188 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 597 KiB/s wr, 39 op/s 2024-08-17T17:50:09.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:09 smithi136 ceph-mon[115078]: osdmap e1228: 8 total, 8 up, 8 in 2024-08-17T17:50:09.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:09 smithi136 ceph-mon[115078]: pgmap v1907: 105 pgs: 105 active+clean; 188 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 597 KiB/s wr, 39 op/s 2024-08-17T17:50:10.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:10 smithi165 ceph-mon[105517]: osdmap e1229: 8 total, 8 up, 8 in 2024-08-17T17:50:10.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:10 smithi136 ceph-mon[109861]: osdmap e1229: 8 total, 8 up, 8 in 2024-08-17T17:50:10.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:10 smithi136 ceph-mon[115078]: osdmap e1229: 8 total, 8 up, 8 in 2024-08-17T17:50:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:11 smithi136 ceph-mon[109861]: pgmap v1909: 105 pgs: 105 active+clean; 188 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 5.8 KiB/s rd, 853 B/s wr, 8 op/s 2024-08-17T17:50:11.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:11 smithi136 ceph-mon[115078]: pgmap v1909: 105 pgs: 105 active+clean; 188 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 5.8 KiB/s rd, 853 B/s wr, 8 op/s 2024-08-17T17:50:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:11 smithi165 ceph-mon[105517]: pgmap v1909: 105 pgs: 105 active+clean; 188 MiB data, 956 MiB used, 714 GiB / 715 GiB avail; 5.8 KiB/s rd, 853 B/s wr, 8 op/s 2024-08-17T17:50:12.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:12 smithi136 ceph-mon[109861]: pgmap v1910: 105 pgs: 105 active+clean; 206 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 2.7 MiB/s wr, 46 op/s 2024-08-17T17:50:12.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:12 smithi136 ceph-mon[115078]: pgmap v1910: 105 pgs: 105 active+clean; 206 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 2.7 MiB/s wr, 46 op/s 2024-08-17T17:50:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:12 smithi165 ceph-mon[105517]: pgmap v1910: 105 pgs: 105 active+clean; 206 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 2.7 MiB/s wr, 46 op/s 2024-08-17T17:50:14.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:14 smithi136 ceph-mon[109861]: pgmap v1911: 105 pgs: 105 active+clean; 206 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.2 MiB/s wr, 38 op/s 2024-08-17T17:50:14.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:14 smithi136 ceph-mon[115078]: pgmap v1911: 105 pgs: 105 active+clean; 206 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.2 MiB/s wr, 38 op/s 2024-08-17T17:50:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:14 smithi165 ceph-mon[105517]: pgmap v1911: 105 pgs: 105 active+clean; 206 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.2 MiB/s wr, 38 op/s 2024-08-17T17:50:16.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:16 smithi136 ceph-mon[109861]: osdmap e1230: 8 total, 8 up, 8 in 2024-08-17T17:50:16.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:16 smithi136 ceph-mon[109861]: pgmap v1913: 105 pgs: 105 active+clean; 214 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 3.0 MiB/s wr, 41 op/s 2024-08-17T17:50:16.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:16 smithi136 ceph-mon[115078]: osdmap e1230: 8 total, 8 up, 8 in 2024-08-17T17:50:16.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:16 smithi136 ceph-mon[115078]: pgmap v1913: 105 pgs: 105 active+clean; 214 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 3.0 MiB/s wr, 41 op/s 2024-08-17T17:50:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:16 smithi165 ceph-mon[105517]: osdmap e1230: 8 total, 8 up, 8 in 2024-08-17T17:50:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:16 smithi165 ceph-mon[105517]: pgmap v1913: 105 pgs: 105 active+clean; 214 MiB data, 1005 MiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 3.0 MiB/s wr, 41 op/s 2024-08-17T17:50:17.638 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/1.DiffIterateDeterministicLUKS2 (40007 ms) 2024-08-17T17:50:17.638 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/1.DiffIterateDeterministicLUKS2PP 2024-08-17T17:50:17.638 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:50:17.847 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:17 smithi136 ceph-mon[109861]: osdmap e1231: 8 total, 8 up, 8 in 2024-08-17T17:50:17.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:17 smithi136 ceph-mon[115078]: osdmap e1231: 8 total, 8 up, 8 in 2024-08-17T17:50:17.848 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:50:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:50:17] "GET /metrics HTTP/1.1" 200 36489 "" "Prometheus/2.51.0" 2024-08-17T17:50:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:17 smithi165 ceph-mon[105517]: osdmap e1231: 8 total, 8 up, 8 in 2024-08-17T17:50:18.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:18 smithi136 ceph-mon[109861]: osdmap e1232: 8 total, 8 up, 8 in 2024-08-17T17:50:18.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:18 smithi136 ceph-mon[109861]: pgmap v1916: 105 pgs: 105 active+clean; 230 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-08-17T17:50:18.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:18 smithi136 ceph-mon[115078]: osdmap e1232: 8 total, 8 up, 8 in 2024-08-17T17:50:18.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:18 smithi136 ceph-mon[115078]: pgmap v1916: 105 pgs: 105 active+clean; 230 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-08-17T17:50:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:18 smithi165 ceph-mon[105517]: osdmap e1232: 8 total, 8 up, 8 in 2024-08-17T17:50:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:18 smithi165 ceph-mon[105517]: pgmap v1916: 105 pgs: 105 active+clean; 230 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-08-17T17:50:20.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:20 smithi136 ceph-mon[109861]: pgmap v1917: 105 pgs: 105 active+clean; 230 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-08-17T17:50:20.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:20 smithi136 ceph-mon[115078]: pgmap v1917: 105 pgs: 105 active+clean; 230 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-08-17T17:50:21.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:20 smithi165 ceph-mon[105517]: pgmap v1917: 105 pgs: 105 active+clean; 230 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-08-17T17:50:21.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:50:21.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:50:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:50:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:50:21.985 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:50:21.985 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:50:22.871 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:22 smithi165 ceph-mon[105517]: pgmap v1918: 105 pgs: 105 active+clean; 230 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.3 MiB/s wr, 40 op/s 2024-08-17T17:50:22.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:22 smithi136 ceph-mon[109861]: pgmap v1918: 105 pgs: 105 active+clean; 230 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.3 MiB/s wr, 40 op/s 2024-08-17T17:50:22.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:22 smithi136 ceph-mon[115078]: pgmap v1918: 105 pgs: 105 active+clean; 230 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.3 MiB/s wr, 40 op/s 2024-08-17T17:50:24.865 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:24 smithi136 ceph-mon[109861]: osdmap e1233: 8 total, 8 up, 8 in 2024-08-17T17:50:24.866 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:24 smithi136 ceph-mon[109861]: pgmap v1920: 105 pgs: 105 active+clean; 230 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 1.0 MiB/s wr, 28 op/s 2024-08-17T17:50:24.866 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:24 smithi136 ceph-mon[115078]: osdmap e1233: 8 total, 8 up, 8 in 2024-08-17T17:50:24.866 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:24 smithi136 ceph-mon[115078]: pgmap v1920: 105 pgs: 105 active+clean; 230 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 1.0 MiB/s wr, 28 op/s 2024-08-17T17:50:24.959 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:24 smithi165 ceph-mon[105517]: osdmap e1233: 8 total, 8 up, 8 in 2024-08-17T17:50:24.959 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:24 smithi165 ceph-mon[105517]: pgmap v1920: 105 pgs: 105 active+clean; 230 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 1.0 MiB/s wr, 28 op/s 2024-08-17T17:50:25.644 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:50:25.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:25 smithi136 ceph-mon[109861]: osdmap e1234: 8 total, 8 up, 8 in 2024-08-17T17:50:25.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:25 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:50:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:25 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:50:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:25 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:50:25.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:25 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:50:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:25 smithi136 ceph-mon[115078]: osdmap e1234: 8 total, 8 up, 8 in 2024-08-17T17:50:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:25 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:50:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:25 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:50:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:25 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:50:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:25 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:50:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:25 smithi165 ceph-mon[105517]: osdmap e1234: 8 total, 8 up, 8 in 2024-08-17T17:50:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:25 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:50:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:25 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:50:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:25 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:50:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:25 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:50:26.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:26 smithi136 ceph-mon[109861]: osdmap e1235: 8 total, 8 up, 8 in 2024-08-17T17:50:26.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:26 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:50:26.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:26 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:50:26.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:26 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:50:26.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:26 smithi136 ceph-mon[109861]: pgmap v1923: 105 pgs: 105 active+clean; 238 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 1.3 MiB/s wr, 51 op/s 2024-08-17T17:50:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:26 smithi136 ceph-mon[115078]: osdmap e1235: 8 total, 8 up, 8 in 2024-08-17T17:50:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:26 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:50:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:26 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:50:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:26 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:50:26.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:26 smithi136 ceph-mon[115078]: pgmap v1923: 105 pgs: 105 active+clean; 238 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 1.3 MiB/s wr, 51 op/s 2024-08-17T17:50:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:26 smithi165 ceph-mon[105517]: osdmap e1235: 8 total, 8 up, 8 in 2024-08-17T17:50:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:26 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:50:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:26 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:50:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:26 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:50:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:26 smithi165 ceph-mon[105517]: pgmap v1923: 105 pgs: 105 active+clean; 238 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 1.3 MiB/s wr, 51 op/s 2024-08-17T17:50:27.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:50:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:50:27] "GET /metrics HTTP/1.1" 200 36491 "" "Prometheus/2.51.0" 2024-08-17T17:50:28.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:28 smithi136 ceph-mon[109861]: pgmap v1924: 105 pgs: 105 active+clean; 246 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 2.7 MiB/s wr, 44 op/s 2024-08-17T17:50:28.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:28 smithi136 ceph-mon[115078]: pgmap v1924: 105 pgs: 105 active+clean; 246 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 2.7 MiB/s wr, 44 op/s 2024-08-17T17:50:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:28 smithi165 ceph-mon[105517]: pgmap v1924: 105 pgs: 105 active+clean; 246 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 2.7 MiB/s wr, 44 op/s 2024-08-17T17:50:30.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:30 smithi136 ceph-mon[109861]: pgmap v1925: 105 pgs: 105 active+clean; 246 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.3 MiB/s wr, 38 op/s 2024-08-17T17:50:30.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:30 smithi136 ceph-mon[115078]: pgmap v1925: 105 pgs: 105 active+clean; 246 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.3 MiB/s wr, 38 op/s 2024-08-17T17:50:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:30 smithi165 ceph-mon[105517]: pgmap v1925: 105 pgs: 105 active+clean; 246 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.3 MiB/s wr, 38 op/s 2024-08-17T17:50:32.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:32 smithi136 ceph-mon[109861]: osdmap e1236: 8 total, 8 up, 8 in 2024-08-17T17:50:32.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:32 smithi136 ceph-mon[109861]: pgmap v1927: 105 pgs: 105 active+clean; 261 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 3.5 MiB/s wr, 42 op/s 2024-08-17T17:50:32.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:32 smithi136 ceph-mon[115078]: osdmap e1236: 8 total, 8 up, 8 in 2024-08-17T17:50:32.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:32 smithi136 ceph-mon[115078]: pgmap v1927: 105 pgs: 105 active+clean; 261 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 3.5 MiB/s wr, 42 op/s 2024-08-17T17:50:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:32 smithi165 ceph-mon[105517]: osdmap e1236: 8 total, 8 up, 8 in 2024-08-17T17:50:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:32 smithi165 ceph-mon[105517]: pgmap v1927: 105 pgs: 105 active+clean; 261 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 3.5 MiB/s wr, 42 op/s 2024-08-17T17:50:33.638 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:50:33.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:33 smithi136 ceph-mon[109861]: osdmap e1237: 8 total, 8 up, 8 in 2024-08-17T17:50:33.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:33 smithi136 ceph-mon[115078]: osdmap e1237: 8 total, 8 up, 8 in 2024-08-17T17:50:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:33 smithi165 ceph-mon[105517]: osdmap e1237: 8 total, 8 up, 8 in 2024-08-17T17:50:34.912 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:34 smithi165 ceph-mon[105517]: osdmap e1238: 8 total, 8 up, 8 in 2024-08-17T17:50:34.912 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:34 smithi165 ceph-mon[105517]: pgmap v1930: 105 pgs: 105 active+clean; 261 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.3 KiB/s rd, 2.0 MiB/s wr, 10 op/s 2024-08-17T17:50:34.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:34 smithi136 ceph-mon[109861]: osdmap e1238: 8 total, 8 up, 8 in 2024-08-17T17:50:34.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:34 smithi136 ceph-mon[109861]: pgmap v1930: 105 pgs: 105 active+clean; 261 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.3 KiB/s rd, 2.0 MiB/s wr, 10 op/s 2024-08-17T17:50:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:34 smithi136 ceph-mon[115078]: osdmap e1238: 8 total, 8 up, 8 in 2024-08-17T17:50:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:34 smithi136 ceph-mon[115078]: pgmap v1930: 105 pgs: 105 active+clean; 261 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.3 KiB/s rd, 2.0 MiB/s wr, 10 op/s 2024-08-17T17:50:36.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:50:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:50:36.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:50:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:37 smithi165 ceph-mon[105517]: pgmap v1931: 105 pgs: 105 active+clean; 268 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 MiB/s wr, 43 op/s 2024-08-17T17:50:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:37 smithi136 ceph-mon[109861]: pgmap v1931: 105 pgs: 105 active+clean; 268 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 MiB/s wr, 43 op/s 2024-08-17T17:50:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:37 smithi136 ceph-mon[115078]: pgmap v1931: 105 pgs: 105 active+clean; 268 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 MiB/s wr, 43 op/s 2024-08-17T17:50:37.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:50:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:50:37] "GET /metrics HTTP/1.1" 200 36490 "" "Prometheus/2.51.0" 2024-08-17T17:50:38.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:38 smithi136 ceph-mon[109861]: pgmap v1932: 105 pgs: 105 active+clean; 268 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.3 MiB/s wr, 39 op/s 2024-08-17T17:50:38.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:38 smithi136 ceph-mon[115078]: pgmap v1932: 105 pgs: 105 active+clean; 268 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.3 MiB/s wr, 39 op/s 2024-08-17T17:50:39.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:38 smithi165 ceph-mon[105517]: pgmap v1932: 105 pgs: 105 active+clean; 268 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.3 MiB/s wr, 39 op/s 2024-08-17T17:50:40.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:40 smithi136 ceph-mon[109861]: osdmap e1239: 8 total, 8 up, 8 in 2024-08-17T17:50:40.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:40 smithi136 ceph-mon[109861]: pgmap v1934: 105 pgs: 105 active+clean; 268 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 519 KiB/s wr, 27 op/s 2024-08-17T17:50:40.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:40 smithi136 ceph-mon[115078]: osdmap e1239: 8 total, 8 up, 8 in 2024-08-17T17:50:40.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:40 smithi136 ceph-mon[115078]: pgmap v1934: 105 pgs: 105 active+clean; 268 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 519 KiB/s wr, 27 op/s 2024-08-17T17:50:41.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:40 smithi165 ceph-mon[105517]: osdmap e1239: 8 total, 8 up, 8 in 2024-08-17T17:50:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:40 smithi165 ceph-mon[105517]: pgmap v1934: 105 pgs: 105 active+clean; 268 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 519 KiB/s wr, 27 op/s 2024-08-17T17:50:41.646 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:50:41.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:41 smithi136 ceph-mon[115078]: osdmap e1240: 8 total, 8 up, 8 in 2024-08-17T17:50:41.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:41 smithi136 ceph-mon[109861]: osdmap e1240: 8 total, 8 up, 8 in 2024-08-17T17:50:42.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:41 smithi165 ceph-mon[105517]: osdmap e1240: 8 total, 8 up, 8 in 2024-08-17T17:50:42.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:42 smithi136 ceph-mon[109861]: osdmap e1241: 8 total, 8 up, 8 in 2024-08-17T17:50:42.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:42 smithi136 ceph-mon[109861]: pgmap v1937: 105 pgs: 105 active+clean; 290 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.7 MiB/s wr, 37 op/s 2024-08-17T17:50:42.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:42 smithi136 ceph-mon[115078]: osdmap e1241: 8 total, 8 up, 8 in 2024-08-17T17:50:42.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:42 smithi136 ceph-mon[115078]: pgmap v1937: 105 pgs: 105 active+clean; 290 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.7 MiB/s wr, 37 op/s 2024-08-17T17:50:43.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:42 smithi165 ceph-mon[105517]: osdmap e1241: 8 total, 8 up, 8 in 2024-08-17T17:50:43.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:42 smithi165 ceph-mon[105517]: pgmap v1937: 105 pgs: 105 active+clean; 290 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.7 MiB/s wr, 37 op/s 2024-08-17T17:50:44.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:44 smithi136 ceph-mon[109861]: pgmap v1938: 105 pgs: 105 active+clean; 290 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.7 MiB/s wr, 32 op/s 2024-08-17T17:50:44.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:44 smithi136 ceph-mon[115078]: pgmap v1938: 105 pgs: 105 active+clean; 290 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.7 MiB/s wr, 32 op/s 2024-08-17T17:50:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:44 smithi165 ceph-mon[105517]: pgmap v1938: 105 pgs: 105 active+clean; 290 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 2.7 MiB/s wr, 32 op/s 2024-08-17T17:50:46.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:46 smithi136 ceph-mon[109861]: pgmap v1939: 105 pgs: 105 active+clean; 290 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.3 MiB/s wr, 40 op/s 2024-08-17T17:50:46.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:46 smithi136 ceph-mon[115078]: pgmap v1939: 105 pgs: 105 active+clean; 290 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.3 MiB/s wr, 40 op/s 2024-08-17T17:50:47.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:46 smithi165 ceph-mon[105517]: pgmap v1939: 105 pgs: 105 active+clean; 290 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.3 MiB/s wr, 40 op/s 2024-08-17T17:50:47.848 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:50:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:50:47] "GET /metrics HTTP/1.1" 200 36490 "" "Prometheus/2.51.0" 2024-08-17T17:50:48.868 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:48 smithi136 ceph-mon[109861]: osdmap e1242: 8 total, 8 up, 8 in 2024-08-17T17:50:48.868 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:48 smithi136 ceph-mon[109861]: pgmap v1941: 105 pgs: 105 active+clean; 290 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 2.0 MiB/s wr, 35 op/s 2024-08-17T17:50:48.868 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:48 smithi136 ceph-mon[115078]: osdmap e1242: 8 total, 8 up, 8 in 2024-08-17T17:50:48.868 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:48 smithi136 ceph-mon[115078]: pgmap v1941: 105 pgs: 105 active+clean; 290 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 2.0 MiB/s wr, 35 op/s 2024-08-17T17:50:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:48 smithi165 ceph-mon[105517]: osdmap e1242: 8 total, 8 up, 8 in 2024-08-17T17:50:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:48 smithi165 ceph-mon[105517]: pgmap v1941: 105 pgs: 105 active+clean; 290 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 2.0 MiB/s wr, 35 op/s 2024-08-17T17:50:49.639 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:50:49.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:49 smithi136 ceph-mon[109861]: osdmap e1243: 8 total, 8 up, 8 in 2024-08-17T17:50:49.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:49 smithi136 ceph-mon[115078]: osdmap e1243: 8 total, 8 up, 8 in 2024-08-17T17:50:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:49 smithi165 ceph-mon[105517]: osdmap e1243: 8 total, 8 up, 8 in 2024-08-17T17:50:50.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:50 smithi136 ceph-mon[109861]: osdmap e1244: 8 total, 8 up, 8 in 2024-08-17T17:50:50.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:50 smithi136 ceph-mon[109861]: pgmap v1944: 105 pgs: 105 active+clean; 290 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.8 KiB/s rd, 1023 B/s wr, 13 op/s 2024-08-17T17:50:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:50 smithi136 ceph-mon[115078]: osdmap e1244: 8 total, 8 up, 8 in 2024-08-17T17:50:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:50 smithi136 ceph-mon[115078]: pgmap v1944: 105 pgs: 105 active+clean; 290 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.8 KiB/s rd, 1023 B/s wr, 13 op/s 2024-08-17T17:50:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:50 smithi165 ceph-mon[105517]: osdmap e1244: 8 total, 8 up, 8 in 2024-08-17T17:50:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:50 smithi165 ceph-mon[105517]: pgmap v1944: 105 pgs: 105 active+clean; 290 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.8 KiB/s rd, 1023 B/s wr, 13 op/s 2024-08-17T17:50:51.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:50:51.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:50:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:50:52.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:52 smithi136 ceph-mon[109861]: pgmap v1945: 105 pgs: 105 active+clean; 312 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 2.7 MiB/s wr, 46 op/s 2024-08-17T17:50:52.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:52 smithi136 ceph-mon[115078]: pgmap v1945: 105 pgs: 105 active+clean; 312 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 2.7 MiB/s wr, 46 op/s 2024-08-17T17:50:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:52 smithi165 ceph-mon[105517]: pgmap v1945: 105 pgs: 105 active+clean; 312 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 2.7 MiB/s wr, 46 op/s 2024-08-17T17:50:54.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:54 smithi136 ceph-mon[115078]: pgmap v1946: 105 pgs: 105 active+clean; 312 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.3 MiB/s wr, 40 op/s 2024-08-17T17:50:54.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:54 smithi136 ceph-mon[109861]: pgmap v1946: 105 pgs: 105 active+clean; 312 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.3 MiB/s wr, 40 op/s 2024-08-17T17:50:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:54 smithi165 ceph-mon[105517]: pgmap v1946: 105 pgs: 105 active+clean; 312 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 2.3 MiB/s wr, 40 op/s 2024-08-17T17:50:56.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:56 smithi136 ceph-mon[109861]: osdmap e1245: 8 total, 8 up, 8 in 2024-08-17T17:50:56.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:56 smithi136 ceph-mon[109861]: pgmap v1948: 105 pgs: 105 active+clean; 324 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 3.5 MiB/s wr, 41 op/s 2024-08-17T17:50:56.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:56 smithi136 ceph-mon[115078]: osdmap e1245: 8 total, 8 up, 8 in 2024-08-17T17:50:56.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:56 smithi136 ceph-mon[115078]: pgmap v1948: 105 pgs: 105 active+clean; 324 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 3.5 MiB/s wr, 41 op/s 2024-08-17T17:50:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:56 smithi165 ceph-mon[105517]: osdmap e1245: 8 total, 8 up, 8 in 2024-08-17T17:50:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:56 smithi165 ceph-mon[105517]: pgmap v1948: 105 pgs: 105 active+clean; 324 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 3.5 MiB/s wr, 41 op/s 2024-08-17T17:50:57.648 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/1.DiffIterateDeterministicLUKS2PP (40010 ms) 2024-08-17T17:50:57.648 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/1.DiffIterateDiscard 2024-08-17T17:50:57.649 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:50:57.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:57 smithi136 ceph-mon[109861]: osdmap e1246: 8 total, 8 up, 8 in 2024-08-17T17:50:57.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:57 smithi136 ceph-mon[115078]: osdmap e1246: 8 total, 8 up, 8 in 2024-08-17T17:50:57.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:50:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:50:57] "GET /metrics HTTP/1.1" 200 36490 "" "Prometheus/2.51.0" 2024-08-17T17:50:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:57 smithi165 ceph-mon[105517]: osdmap e1246: 8 total, 8 up, 8 in 2024-08-17T17:50:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:58 smithi136 ceph-mon[109861]: osdmap e1247: 8 total, 8 up, 8 in 2024-08-17T17:50:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:58 smithi136 ceph-mon[109861]: pgmap v1951: 105 pgs: 105 active+clean; 332 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-08-17T17:50:58.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:58 smithi136 ceph-mon[115078]: osdmap e1247: 8 total, 8 up, 8 in 2024-08-17T17:50:58.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:58 smithi136 ceph-mon[115078]: pgmap v1951: 105 pgs: 105 active+clean; 332 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-08-17T17:50:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:58 smithi165 ceph-mon[105517]: osdmap e1247: 8 total, 8 up, 8 in 2024-08-17T17:50:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:58 smithi165 ceph-mon[105517]: pgmap v1951: 105 pgs: 105 active+clean; 332 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 2.7 MiB/s wr, 21 op/s 2024-08-17T17:50:59.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:50:59 smithi136 ceph-mon[109861]: osdmap e1248: 8 total, 8 up, 8 in 2024-08-17T17:50:59.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:50:59 smithi136 ceph-mon[115078]: osdmap e1248: 8 total, 8 up, 8 in 2024-08-17T17:51:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:50:59 smithi165 ceph-mon[105517]: osdmap e1248: 8 total, 8 up, 8 in 2024-08-17T17:51:00.673 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/1.DiffIterateDiscard (3025 ms) 2024-08-17T17:51:00.673 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/1.DiffIterateTruncate 2024-08-17T17:51:00.673 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:51:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:00 smithi136 ceph-mon[109861]: osdmap e1249: 8 total, 8 up, 8 in 2024-08-17T17:51:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:00 smithi136 ceph-mon[109861]: pgmap v1954: 105 pgs: 105 active+clean; 332 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 1.0 MiB/s wr, 18 op/s 2024-08-17T17:51:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:00 smithi136 ceph-mon[115078]: osdmap e1249: 8 total, 8 up, 8 in 2024-08-17T17:51:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:00 smithi136 ceph-mon[115078]: pgmap v1954: 105 pgs: 105 active+clean; 332 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 1.0 MiB/s wr, 18 op/s 2024-08-17T17:51:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:00 smithi165 ceph-mon[105517]: osdmap e1249: 8 total, 8 up, 8 in 2024-08-17T17:51:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:00 smithi165 ceph-mon[105517]: pgmap v1954: 105 pgs: 105 active+clean; 332 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 1.0 MiB/s wr, 18 op/s 2024-08-17T17:51:01.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:01 smithi136 ceph-mon[109861]: osdmap e1250: 8 total, 8 up, 8 in 2024-08-17T17:51:01.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:01 smithi136 ceph-mon[115078]: osdmap e1250: 8 total, 8 up, 8 in 2024-08-17T17:51:02.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:01 smithi165 ceph-mon[105517]: osdmap e1250: 8 total, 8 up, 8 in 2024-08-17T17:51:02.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:02 smithi136 ceph-mon[109861]: osdmap e1251: 8 total, 8 up, 8 in 2024-08-17T17:51:02.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:02 smithi136 ceph-mon[109861]: pgmap v1957: 105 pgs: 105 active+clean; 336 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 76 KiB/s rd, 141 KiB/s wr, 116 op/s 2024-08-17T17:51:02.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:02 smithi136 ceph-mon[115078]: osdmap e1251: 8 total, 8 up, 8 in 2024-08-17T17:51:02.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:02 smithi136 ceph-mon[115078]: pgmap v1957: 105 pgs: 105 active+clean; 336 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 76 KiB/s rd, 141 KiB/s wr, 116 op/s 2024-08-17T17:51:03.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:02 smithi165 ceph-mon[105517]: osdmap e1251: 8 total, 8 up, 8 in 2024-08-17T17:51:03.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:02 smithi165 ceph-mon[105517]: pgmap v1957: 105 pgs: 105 active+clean; 336 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 76 KiB/s rd, 141 KiB/s wr, 116 op/s 2024-08-17T17:51:03.711 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/1.DiffIterateTruncate (3038 ms) 2024-08-17T17:51:03.711 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/1.DiffIterateWriteAndTruncate 2024-08-17T17:51:03.711 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:51:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:03 smithi136 ceph-mon[109861]: osdmap e1252: 8 total, 8 up, 8 in 2024-08-17T17:51:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:03 smithi136 ceph-mon[115078]: osdmap e1252: 8 total, 8 up, 8 in 2024-08-17T17:51:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:03 smithi165 ceph-mon[105517]: osdmap e1252: 8 total, 8 up, 8 in 2024-08-17T17:51:04.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:04 smithi136 ceph-mon[109861]: osdmap e1253: 8 total, 8 up, 8 in 2024-08-17T17:51:04.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:04 smithi136 ceph-mon[109861]: pgmap v1960: 105 pgs: 105 active+clean; 336 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 76 KiB/s rd, 141 KiB/s wr, 116 op/s 2024-08-17T17:51:04.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:04 smithi136 ceph-mon[115078]: osdmap e1253: 8 total, 8 up, 8 in 2024-08-17T17:51:04.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:04 smithi136 ceph-mon[115078]: pgmap v1960: 105 pgs: 105 active+clean; 336 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 76 KiB/s rd, 141 KiB/s wr, 116 op/s 2024-08-17T17:51:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:04 smithi165 ceph-mon[105517]: osdmap e1253: 8 total, 8 up, 8 in 2024-08-17T17:51:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:04 smithi165 ceph-mon[105517]: pgmap v1960: 105 pgs: 105 active+clean; 336 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 76 KiB/s rd, 141 KiB/s wr, 116 op/s 2024-08-17T17:51:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:05 smithi136 ceph-mon[109861]: osdmap e1254: 8 total, 8 up, 8 in 2024-08-17T17:51:05.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:05 smithi136 ceph-mon[115078]: osdmap e1254: 8 total, 8 up, 8 in 2024-08-17T17:51:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:05 smithi165 ceph-mon[105517]: osdmap e1254: 8 total, 8 up, 8 in 2024-08-17T17:51:06.735 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/1.DiffIterateWriteAndTruncate (3024 ms) 2024-08-17T17:51:06.735 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/1.DiffIterateStress 2024-08-17T17:51:06.735 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:51:06.813 INFO:tasks.workunit.client.0.smithi136.stdout: i=0 exists [48462114~4677674,71211351~6146357,237346036~2799056,309475977~5513858,318350437~3654919,354835730~3643829,366249771~7694512,385823160~7611208,404586323~155760] wrote [48462114~4677674,71211351~6146357,237346036~2799056,309475977~5513858,318350437~3654919,354835730~3643829,366249771~7694512,385823160~7611208,404586323~155760] 2024-08-17T17:51:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:06 smithi136 ceph-mon[109861]: osdmap e1255: 8 total, 8 up, 8 in 2024-08-17T17:51:06.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:51:06.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:06 smithi136 ceph-mon[109861]: pgmap v1963: 105 pgs: 105 active+clean; 337 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 57 KiB/s rd, 134 KiB/s wr, 82 op/s 2024-08-17T17:51:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:06 smithi136 ceph-mon[115078]: osdmap e1255: 8 total, 8 up, 8 in 2024-08-17T17:51:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:51:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:06 smithi136 ceph-mon[115078]: pgmap v1963: 105 pgs: 105 active+clean; 337 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 57 KiB/s rd, 134 KiB/s wr, 82 op/s 2024-08-17T17:51:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:06 smithi165 ceph-mon[105517]: osdmap e1255: 8 total, 8 up, 8 in 2024-08-17T17:51:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:51:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:06 smithi165 ceph-mon[105517]: pgmap v1963: 105 pgs: 105 active+clean; 337 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 57 KiB/s rd, 134 KiB/s wr, 82 op/s 2024-08-17T17:51:07.786 INFO:tasks.workunit.client.0.smithi136.stdout: i=1 exists [48462114~4677674,71211351~6146357,90018778~4508801,122766672~6849844,166155864~6586300,183675439~2552659,237346036~2799056,262157706~7676757,309475977~5513858,318350437~10297345,354835730~3643829,366249771~7694512,385823160~7611208,397957808~4649991,404504589~5397524] wrote [90018778~4508801,122766672~6849844,166155864~6586300,183675439~2552659,262157706~7676757,319982018~8665764,397957808~4649991,404504589~5397524] 2024-08-17T17:51:07.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:07 smithi136 ceph-mon[115078]: osdmap e1256: 8 total, 8 up, 8 in 2024-08-17T17:51:07.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:07 smithi136 ceph-mon[109861]: osdmap e1256: 8 total, 8 up, 8 in 2024-08-17T17:51:07.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:51:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:51:07] "GET /metrics HTTP/1.1" 200 36491 "" "Prometheus/2.51.0" 2024-08-17T17:51:08.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:07 smithi165 ceph-mon[105517]: osdmap e1256: 8 total, 8 up, 8 in 2024-08-17T17:51:08.744 INFO:tasks.workunit.client.0.smithi136.stdout: i=2 exists [16432083~6114399,48462114~4677674,71211351~6146357,85454305~267064,88964979~5562600,122766672~6849844,149680190~5248650,166155864~6586300,183675439~2552659,206888403~3117896,237346036~6110719,262157706~7676757,288112741~3360412,309475977~5513858,318350437~10297345,354835730~3643829,363287350~1402505,366249771~7694512,385823160~7611208,397957808~4649991,404504589~5397524] wrote [16432083~6114399,85454305~267064,88964979~2787846,149680190~5248650,206888403~3117896,240108067~3348688,288112741~3360412,357876380~31152,363287350~1402505] 2024-08-17T17:51:09.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:08 smithi165 ceph-mon[105517]: osdmap e1257: 8 total, 8 up, 8 in 2024-08-17T17:51:09.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:08 smithi165 ceph-mon[105517]: pgmap v1966: 105 pgs: 105 active+clean; 359 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 94 KiB/s rd, 3.8 MiB/s wr, 149 op/s 2024-08-17T17:51:09.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:08 smithi136 ceph-mon[109861]: osdmap e1257: 8 total, 8 up, 8 in 2024-08-17T17:51:09.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:08 smithi136 ceph-mon[109861]: pgmap v1966: 105 pgs: 105 active+clean; 359 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 94 KiB/s rd, 3.8 MiB/s wr, 149 op/s 2024-08-17T17:51:09.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:08 smithi136 ceph-mon[115078]: osdmap e1257: 8 total, 8 up, 8 in 2024-08-17T17:51:09.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:08 smithi136 ceph-mon[115078]: pgmap v1966: 105 pgs: 105 active+clean; 359 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 94 KiB/s rd, 3.8 MiB/s wr, 149 op/s 2024-08-17T17:51:09.787 INFO:tasks.workunit.client.0.smithi136.stdout: i=3 exists [16432083~6114399,46654331~6485457,68343469~9505840,85454305~267064,88964979~5562600,122766672~6849844,147143781~7785059,166155864~6586300,183675439~2552659,204503998~6105578,226465344~4844181,237346036~6110719,262157706~7676757,288112741~3360412,292687332~3771753,309475977~7595439,318350437~10297345,354835730~3643829,363287350~1402505,366249771~7694512,373958893~1270515,385823160~7611208,397957808~4649991,404504589~5397524] wrote [46654331~6114476,68343469~9505840,125516267~3656656,147143781~6312764,204503998~6105578,226465344~4844181,292687332~3771753,310648761~6422655,373958893~1270515] 2024-08-17T17:51:10.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:09 smithi165 ceph-mon[105517]: osdmap e1258: 8 total, 8 up, 8 in 2024-08-17T17:51:10.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:09 smithi136 ceph-mon[115078]: osdmap e1258: 8 total, 8 up, 8 in 2024-08-17T17:51:10.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:09 smithi136 ceph-mon[109861]: osdmap e1258: 8 total, 8 up, 8 in 2024-08-17T17:51:10.783 INFO:tasks.workunit.client.0.smithi136.stdout: i=4 exists [16432083~6114399,31362958~4825337,46654331~6485457,60776595~50057,68343469~9505840,85454305~267064,88964979~5562600,112520365~4973588,122766672~6849844,147143781~10112691,166155864~6586300,183675439~2552659,194362671~6689846,204503998~6105578,221317665~9991860,237346036~6110719,262157706~7676757,288112741~3360412,292687332~3771753,309475977~7595439,318350437~10297345,349623108~8856451,363287350~1402505,366249771~7694512,373958893~1270515,385823160~7611208,397957808~4649991,404504589~6066063] wrote [31362958~4825337,60776595~50057,112520365~4973588,152629666~4626806,194362671~6689846,221317665~5973462,349623108~6453513,369086838~3223052,409439115~1131537] 2024-08-17T17:51:11.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:10 smithi165 ceph-mon[105517]: osdmap e1259: 8 total, 8 up, 8 in 2024-08-17T17:51:11.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:10 smithi165 ceph-mon[105517]: pgmap v1969: 105 pgs: 105 active+clean; 359 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 36 KiB/s rd, 3.7 MiB/s wr, 67 op/s 2024-08-17T17:51:11.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:10 smithi136 ceph-mon[109861]: osdmap e1259: 8 total, 8 up, 8 in 2024-08-17T17:51:11.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:10 smithi136 ceph-mon[109861]: pgmap v1969: 105 pgs: 105 active+clean; 359 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 36 KiB/s rd, 3.7 MiB/s wr, 67 op/s 2024-08-17T17:51:11.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:10 smithi136 ceph-mon[115078]: osdmap e1259: 8 total, 8 up, 8 in 2024-08-17T17:51:11.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:10 smithi136 ceph-mon[115078]: pgmap v1969: 105 pgs: 105 active+clean; 359 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 36 KiB/s rd, 3.7 MiB/s wr, 67 op/s 2024-08-17T17:51:11.822 INFO:tasks.workunit.client.0.smithi136.stdout: i=5 exists [13034637~9511845,31362958~4825337,46654331~6485457,56171032~6428386,68343469~9505840,85454305~267064,87292046~10922823,112520365~4973588,122766672~6849844,147143781~10112691,166155864~6586300,183675439~2552659,194362671~6689846,204503998~6105578,221317665~9991860,231497307~3206498,237346036~6110719,262157706~7676757,288112741~3360412,292687332~3771753,309475977~8820185,318350437~10297345,349623108~8927097,363287350~1402505,366249771~7694512,373958893~1270515,385823160~7611208,397957808~4649991,404504589~6066063] wrote [13034637~4514256,34096319~470983,56171032~6428386,87292046~10922823,183982354~331135,231497307~3206498,311275751~7020411,354352721~4197484] 2024-08-17T17:51:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:11 smithi165 ceph-mon[105517]: osdmap e1260: 8 total, 8 up, 8 in 2024-08-17T17:51:12.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:11 smithi136 ceph-mon[109861]: osdmap e1260: 8 total, 8 up, 8 in 2024-08-17T17:51:12.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:11 smithi136 ceph-mon[115078]: osdmap e1260: 8 total, 8 up, 8 in 2024-08-17T17:51:12.828 INFO:tasks.workunit.client.0.smithi136.stdout: i=6 exists [8078668~14467814,31362958~4825337,46654331~6485457,56171032~6428386,68343469~9505840,85454305~267064,87292046~10922823,112520365~4973588,119489825~10126691,147143781~10112691,166155864~6586300,183675439~2552659,194362671~6689846,204503998~6105578,221317665~14094546,237346036~6110719,262157706~7676757,278298118~7969082,288112741~3360412,292687332~3771753,309475977~8820185,318350437~10297345,336430313~970901,349623108~8927097,363287350~1402505,366249771~8979637,376897304~3880693,385823160~7611208,397957808~4649991,403613365~6957287] wrote [8078668~5761910,94876840~1467218,119489825~7233800,229924838~5487373,278298118~7969082,336430313~970901,369190139~4944283,376897304~3880693,403613365~1214947] 2024-08-17T17:51:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:12 smithi165 ceph-mon[105517]: osdmap e1261: 8 total, 8 up, 8 in 2024-08-17T17:51:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:12 smithi165 ceph-mon[105517]: pgmap v1972: 105 pgs: 105 active+clean; 599 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 160 KiB/s rd, 47 MiB/s wr, 273 op/s 2024-08-17T17:51:13.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:12 smithi136 ceph-mon[109861]: osdmap e1261: 8 total, 8 up, 8 in 2024-08-17T17:51:13.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:12 smithi136 ceph-mon[109861]: pgmap v1972: 105 pgs: 105 active+clean; 599 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 160 KiB/s rd, 47 MiB/s wr, 273 op/s 2024-08-17T17:51:13.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:12 smithi136 ceph-mon[115078]: osdmap e1261: 8 total, 8 up, 8 in 2024-08-17T17:51:13.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:12 smithi136 ceph-mon[115078]: pgmap v1972: 105 pgs: 105 active+clean; 599 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 160 KiB/s rd, 47 MiB/s wr, 273 op/s 2024-08-17T17:51:13.805 INFO:tasks.workunit.client.0.smithi136.stdout: i=7 exists [8078668~14467814,31362958~4825337,42050326~11089462,56171032~6428386,68343469~9505840,85454305~267064,87292046~10922823,99139028~3763239,112520365~4973588,119489825~11554946,147143781~10112691,166155864~6586300,183675439~2552659,194362671~6689846,204503998~6105578,221317665~14094546,237346036~6110719,262157706~7676757,278298118~7969082,288112741~3360412,292687332~3771753,300894819~6578324,309475977~8820185,318350437~10297345,336430313~970901,349623108~8927097,363287350~1402505,366249771~8979637,375793367~5596838,385823160~7611208,397957808~4649991,403613365~6957287] wrote [14477675~3171457,35718840~262860,42050326~5363462,99139028~3763239,125668744~5376027,169038305~132958,300894819~6578324,311185945~2997493,375793367~5596838] 2024-08-17T17:51:14.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:13 smithi165 ceph-mon[105517]: osdmap e1262: 8 total, 8 up, 8 in 2024-08-17T17:51:14.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:13 smithi136 ceph-mon[109861]: osdmap e1262: 8 total, 8 up, 8 in 2024-08-17T17:51:14.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:13 smithi136 ceph-mon[115078]: osdmap e1262: 8 total, 8 up, 8 in 2024-08-17T17:51:14.820 INFO:tasks.workunit.client.0.smithi136.stdout: i=8 exists [8078668~14467814,31362958~4825337,38431595~706215,42050326~11089462,55699791~6899627,68343469~9505840,85454305~267064,87292046~10922823,99139028~3763239,112520365~4973588,119489825~11554946,147143781~10112691,158729525~5114585,166155864~6586300,182798492~7553585,194362671~6689846,204503998~6105578,217915750~3376341,221317665~14094546,237346036~6110719,262157706~7676757,278298118~7969082,288112741~3360412,292687332~5472244,300894819~6578324,309475977~8820185,318350437~12380046,336430313~970901,349623108~8927097,363287350~1402505,366249771~8979637,375793367~5596838,385823160~7611208,397957808~4649991,403613365~6957287] wrote [9608637~5079587,38431595~706215,55699791~3439789,92046344~5921121,158729525~5114585,182798492~7553585,217915750~3376341,295785121~2374455,311507565~1418406,328457062~2273421] 2024-08-17T17:51:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:14 smithi165 ceph-mon[105517]: osdmap e1263: 8 total, 8 up, 8 in 2024-08-17T17:51:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:14 smithi165 ceph-mon[105517]: pgmap v1975: 105 pgs: 105 active+clean; 599 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 160 KiB/s rd, 47 MiB/s wr, 273 op/s 2024-08-17T17:51:15.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:14 smithi136 ceph-mon[109861]: osdmap e1263: 8 total, 8 up, 8 in 2024-08-17T17:51:15.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:14 smithi136 ceph-mon[109861]: pgmap v1975: 105 pgs: 105 active+clean; 599 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 160 KiB/s rd, 47 MiB/s wr, 273 op/s 2024-08-17T17:51:15.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:14 smithi136 ceph-mon[115078]: osdmap e1263: 8 total, 8 up, 8 in 2024-08-17T17:51:15.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:14 smithi136 ceph-mon[115078]: pgmap v1975: 105 pgs: 105 active+clean; 599 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 160 KiB/s rd, 47 MiB/s wr, 273 op/s 2024-08-17T17:51:15.822 INFO:tasks.workunit.client.0.smithi136.stdout: i=9 exists [3325953~19220529,27560069~8628226,38431595~706215,42050326~11089462,55699791~6899627,68343469~9505840,83027470~2693899,87292046~10922823,99139028~3763239,112520365~4973588,119489825~11554946,147143781~10112691,158729525~5114585,166155864~6586300,182798492~7553585,194362671~6689846,202522110~8087466,217915750~3376341,221317665~14094546,237346036~6110719,262157706~7676757,278298118~7969082,288112741~3360412,292687332~5472244,300894819~6578324,309475977~8820185,318350437~12380046,336430313~970901,349181703~9368502,363287350~1402505,366249771~8979637,375793367~5596838,385823160~7611208,397957808~4649991,403613365~6957287] wrote [3325953~7207897,27560069~6237608,83027470~2627747,183875784~4569634,202522110~2534113,208650808~1061319,349181703~5313058,368306469~4802134,405003824~824880] 2024-08-17T17:51:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:15 smithi165 ceph-mon[105517]: osdmap e1264: 8 total, 8 up, 8 in 2024-08-17T17:51:16.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:15 smithi136 ceph-mon[109861]: osdmap e1264: 8 total, 8 up, 8 in 2024-08-17T17:51:16.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:15 smithi136 ceph-mon[115078]: osdmap e1264: 8 total, 8 up, 8 in 2024-08-17T17:51:16.880 INFO:tasks.workunit.client.0.smithi136.stdout: i=10 exists [1031351~22775917,27560069~8628226,38431595~706215,42050326~11089462,55699791~10584242,68343469~9505840,83027470~2693899,87292046~10922823,99139028~3763239,112520365~4973588,119489825~11554946,147143781~10112691,158729525~5114585,166155864~6586300,182798492~7553585,194362671~6689846,202522110~8087466,217915750~3376341,221317665~14094546,237346036~6110719,262157706~7676757,278298118~7969082,288112741~3360412,292687332~14785811,309475977~8820185,318350437~12380046,332069850~6161067,349181703~9368502,363287350~1402505,366249771~8979637,375793367~6495094,384220755~9213613,397957808~4649991,403613365~6957287] wrote [1031351~8021239,10469887~13337381,58387363~7896670,296271052~5142949,323853686~868366,332069850~6161067,350753359~3180236,375798482~6489979,384220755~7012355] 2024-08-17T17:51:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:16 smithi165 ceph-mon[105517]: osdmap e1265: 8 total, 8 up, 8 in 2024-08-17T17:51:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:16 smithi165 ceph-mon[105517]: pgmap v1978: 105 pgs: 105 active+clean; 723 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 64 KiB/s rd, 27 MiB/s wr, 98 op/s 2024-08-17T17:51:17.097 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:16 smithi136 ceph-mon[109861]: osdmap e1265: 8 total, 8 up, 8 in 2024-08-17T17:51:17.098 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:16 smithi136 ceph-mon[109861]: pgmap v1978: 105 pgs: 105 active+clean; 723 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 64 KiB/s rd, 27 MiB/s wr, 98 op/s 2024-08-17T17:51:17.098 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:16 smithi136 ceph-mon[115078]: osdmap e1265: 8 total, 8 up, 8 in 2024-08-17T17:51:17.098 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:16 smithi136 ceph-mon[115078]: pgmap v1978: 105 pgs: 105 active+clean; 723 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 64 KiB/s rd, 27 MiB/s wr, 98 op/s 2024-08-17T17:51:17.767 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:51:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:51:17] "GET /metrics HTTP/1.1" 200 36491 "" "Prometheus/2.51.0" 2024-08-17T17:51:17.838 INFO:tasks.workunit.client.0.smithi136.stdout: i=11 exists [1031351~35156944,38431595~706215,42050326~11089462,55699791~22149518,83027470~2693899,87292046~10922823,99139028~3763239,112520365~6325671,119489825~11554946,147143781~10112691,158729525~5114585,166155864~6586300,180401384~9950693,194362671~6689846,202522110~8087466,217915750~3376341,221317665~14094546,237346036~6110719,249474220~5612504,262157706~7676757,275134981~11132219,288112741~3360412,292687332~25608830,318350437~12380046,332069850~6161067,349181703~9368502,363287350~1402505,365929346~9300062,375793367~6495094,384220755~9213613,397957808~4649991,403613365~8351552] wrote [22588262~5276547,64276422~4485538,112715633~6130403,180401384~4581201,231668476~1933064,249474220~5612504,275134981~4050308,306334790~5933513,365929346~488668,405394681~6570236] 2024-08-17T17:51:18.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:17 smithi136 ceph-mon[109861]: osdmap e1266: 8 total, 8 up, 8 in 2024-08-17T17:51:18.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:17 smithi136 ceph-mon[115078]: osdmap e1266: 8 total, 8 up, 8 in 2024-08-17T17:51:18.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:17 smithi165 ceph-mon[105517]: osdmap e1266: 8 total, 8 up, 8 in 2024-08-17T17:51:18.840 INFO:tasks.workunit.client.0.smithi136.stdout: i=12 exists [1031351~35156944,38431595~706215,39402060~13737728,55699791~22149518,83027470~2693899,87292046~10922823,99139028~3763239,112520365~6325671,119060682~11984089,147143781~10112691,158729525~14012639,180401384~10618906,194362671~6689846,202522110~8087466,217915750~3376341,221317665~14094546,237346036~6110719,249474220~5612504,262157706~7676757,275134981~11132219,288112741~3360412,292655901~25640261,318350437~12380046,332069850~6161067,349181703~9368502,363287350~1402505,365929346~9300062,375793367~6495094,384220755~9213613,397957808~4649991,403613365~8351552] wrote [24253462~6626407,39402060~3366271,73042165~1410471,119060682~2918709,125361285~2768521,162390622~5677118,183218788~7801502,196446664~4038153,237936817~1972601,292655901~2372939] 2024-08-17T17:51:19.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:18 smithi136 ceph-mon[115078]: osdmap e1267: 8 total, 8 up, 8 in 2024-08-17T17:51:19.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:18 smithi136 ceph-mon[115078]: pgmap v1981: 105 pgs: 105 active+clean; 810 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 181 KiB/s rd, 45 MiB/s wr, 297 op/s 2024-08-17T17:51:19.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:18 smithi136 ceph-mon[109861]: osdmap e1267: 8 total, 8 up, 8 in 2024-08-17T17:51:19.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:18 smithi136 ceph-mon[109861]: pgmap v1981: 105 pgs: 105 active+clean; 810 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 181 KiB/s rd, 45 MiB/s wr, 297 op/s 2024-08-17T17:51:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:18 smithi165 ceph-mon[105517]: osdmap e1267: 8 total, 8 up, 8 in 2024-08-17T17:51:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:18 smithi165 ceph-mon[105517]: pgmap v1981: 105 pgs: 105 active+clean; 810 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 181 KiB/s rd, 45 MiB/s wr, 297 op/s 2024-08-17T17:51:19.857 INFO:tasks.workunit.client.0.smithi136.stdout: i=13 exists [1031351~35156944,38431595~706215,39402060~14079959,53748579~26882433,81472010~16742859,99139028~3763239,112520365~6325671,119060682~11984089,147143781~10112691,158729525~14012639,174690623~1524415,180401384~10618906,194362671~6689846,202522110~8087466,213928485~7363606,221317665~14094546,237346036~6110719,249474220~5612504,262157706~7676757,275134981~11132219,288112741~3360412,292655901~25640261,318350437~12380046,332069850~6161067,349181703~9368502,363287350~1402505,365929346~9300062,375793367~6495094,384220755~9213613,397957808~4649991,403613365~8351552] wrote [25123218~2074858,33994698~1144405,47581238~5900781,53748579~7640037,73302258~7328754,81472010~6013022,166197396~6112810,174690623~1524415,213928485~4221488,336227776~243612] 2024-08-17T17:51:20.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:19 smithi136 ceph-mon[109861]: osdmap e1268: 8 total, 8 up, 8 in 2024-08-17T17:51:20.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:19 smithi136 ceph-mon[115078]: osdmap e1268: 8 total, 8 up, 8 in 2024-08-17T17:51:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:19 smithi165 ceph-mon[105517]: osdmap e1268: 8 total, 8 up, 8 in 2024-08-17T17:51:20.868 INFO:tasks.workunit.client.0.smithi136.stdout: i=14 exists [1031351~35156944,38431595~706215,39402060~14079959,53748579~26882433,81472010~16742859,99139028~3763239,112520365~6325671,119060682~11984089,147143781~10112691,158729525~14012639,174690623~1524415,180401384~10618906,194362671~6689846,202522110~8087466,212201669~1540869,213928485~7363606,221317665~14094546,237346036~6110719,244304172~189167,249474220~5612504,256504781~13329682,275134981~16338172,292655901~38074582,332069850~6161067,349181703~9368502,363287350~1402505,365929346~9300062,375793367~6495094,384220755~9213613,397957808~4649991,403613365~10491442] wrote [92611126~2785338,159118599~1278459,212201669~1540869,244304172~189167,256504781~7671633,283757367~4390085,318060498~3573009,406154270~382826,408487524~5617283] 2024-08-17T17:51:21.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:20 smithi136 ceph-mon[115078]: osdmap e1269: 8 total, 8 up, 8 in 2024-08-17T17:51:21.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:20 smithi136 ceph-mon[115078]: pgmap v1984: 105 pgs: 105 active+clean; 810 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 117 KiB/s rd, 18 MiB/s wr, 199 op/s 2024-08-17T17:51:21.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:20 smithi136 ceph-mon[109861]: osdmap e1269: 8 total, 8 up, 8 in 2024-08-17T17:51:21.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:20 smithi136 ceph-mon[109861]: pgmap v1984: 105 pgs: 105 active+clean; 810 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 117 KiB/s rd, 18 MiB/s wr, 199 op/s 2024-08-17T17:51:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:20 smithi165 ceph-mon[105517]: osdmap e1269: 8 total, 8 up, 8 in 2024-08-17T17:51:21.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:20 smithi165 ceph-mon[105517]: pgmap v1984: 105 pgs: 105 active+clean; 810 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 117 KiB/s rd, 18 MiB/s wr, 199 op/s 2024-08-17T17:51:21.855 INFO:tasks.workunit.client.0.smithi136.stdout: i=15 exists [1031351~35156944,38431595~706215,39402060~14079959,53748579~26882433,81472010~16742859,99139028~3763239,112520365~6325671,119060682~11984089,132150039~6699759,147143781~10112691,158128092~14614072,174690623~1524415,180401384~10618906,194362671~6689846,202522110~8087466,212201669~1540869,213928485~7363606,221317665~14094546,237346036~6110719,244304172~189167,249474220~20360243,275134981~16338172,292655901~38074582,332069850~7735404,349181703~9368502,363287350~1402505,365534802~9694606,375793367~6495094,384220755~9213613,397957808~4649991,403613365~10491442] wrote [44934226~1796540,123727152~3929100,132150039~6699759,149748826~2794684,158128092~4113802,254887809~4912077,260505077~4024439,302811047~4771049,333975259~5829995,365534802~1319261] 2024-08-17T17:51:22.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:21 smithi136 ceph-mon[115078]: osdmap e1270: 8 total, 8 up, 8 in 2024-08-17T17:51:22.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:51:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:21 smithi136 ceph-mon[109861]: osdmap e1270: 8 total, 8 up, 8 in 2024-08-17T17:51:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:51:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:21 smithi165 ceph-mon[105517]: osdmap e1270: 8 total, 8 up, 8 in 2024-08-17T17:51:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:51:22.835 INFO:tasks.workunit.client.0.smithi136.stdout: i=16 exists [1031351~35156944,38431595~706215,39402060~14079959,53748579~26882433,81472010~16742859,99139028~3763239,109916240~2037465,112520365~6325671,119060682~11984089,132150039~6699759,147143781~10112691,158128092~14614072,174690623~1524415,180401384~10618906,194362671~6689846,202522110~8087466,212201669~1540869,213928485~7363606,221317665~14094546,237346036~6110719,244304172~189167,249474220~20360243,273807794~175066,274843948~16629205,292655901~47149353,349181703~9368502,363287350~1402505,365534802~9694606,375793367~6495094,384220755~9213613,397957808~4649991,403613365~10491442] wrote [22862910~3139692,30297612~601675,67373046~1071267,75333452~4254817,109916240~2037465,134312963~1611810,230104368~5033067,273807794~175066,274843948~1232140,329083757~4406345] 2024-08-17T17:51:23.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:22 smithi136 ceph-mon[115078]: osdmap e1271: 8 total, 8 up, 8 in 2024-08-17T17:51:23.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:22 smithi136 ceph-mon[115078]: pgmap v1987: 105 pgs: 105 active+clean; 1.0 GiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 220 KiB/s rd, 53 MiB/s wr, 352 op/s 2024-08-17T17:51:23.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:22 smithi136 ceph-mon[109861]: osdmap e1271: 8 total, 8 up, 8 in 2024-08-17T17:51:23.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:22 smithi136 ceph-mon[109861]: pgmap v1987: 105 pgs: 105 active+clean; 1.0 GiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 220 KiB/s rd, 53 MiB/s wr, 352 op/s 2024-08-17T17:51:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:22 smithi165 ceph-mon[105517]: osdmap e1271: 8 total, 8 up, 8 in 2024-08-17T17:51:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:22 smithi165 ceph-mon[105517]: pgmap v1987: 105 pgs: 105 active+clean; 1.0 GiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 220 KiB/s rd, 53 MiB/s wr, 352 op/s 2024-08-17T17:51:23.897 INFO:tasks.workunit.client.0.smithi136.stdout: i=17 exists [1031351~35156944,38431595~706215,39402060~14079959,53748579~26882433,81472010~16742859,99139028~3763239,109916240~2037465,112520365~6325671,119060682~11984089,132150039~6699759,147143781~10112691,158128092~14614072,174690623~1524415,180401384~10618906,194362671~6689846,202522110~8087466,212201669~1540869,213928485~7363606,221317665~14094546,237346036~6110719,244304172~189167,249474220~20379811,273807794~175066,274843948~16629205,292655901~47149353,349181703~9368502,363287350~1402505,365534802~9694606,375793367~6495094,384220755~9213613,397957808~4649991,403613365~10491442] wrote [15710350~7014579,63698027~6129846,90853287~7175082,116605234~661702,148977276~2195903,216461935~2719895,255667311~6917167,262802574~7051457,318779617~5303334] 2024-08-17T17:51:24.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:23 smithi136 ceph-mon[109861]: osdmap e1272: 8 total, 8 up, 8 in 2024-08-17T17:51:24.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:23 smithi136 ceph-mon[115078]: osdmap e1272: 8 total, 8 up, 8 in 2024-08-17T17:51:24.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:23 smithi165 ceph-mon[105517]: osdmap e1272: 8 total, 8 up, 8 in 2024-08-17T17:51:24.888 INFO:tasks.workunit.client.0.smithi136.stdout: i=18 exists [1031351~35156944,38431595~706215,39402060~14079959,53748579~26882433,81472010~16742859,99139028~3763239,103726598~5627320,109916240~2037465,112520365~6325671,119060682~11984089,132150039~6699759,147143781~10112691,158128092~14614072,174690623~1524415,180401384~10618906,194362671~6689846,202522110~8087466,212201669~1540869,213928485~7363606,221317665~14094546,237346036~6110719,244304172~189167,249474220~20379811,273807794~175066,274843948~16629205,292655901~49960397,349181703~9669183,363287350~1402505,365534802~9694606,375793367~6495094,384220755~9213613,397957808~16146999] wrote [3796867~2278938,81534112~3175391,103726598~5627320,161865435~7112767,205911180~346270,330077135~4075706,337050256~5566042,351941882~6909004,400603502~7334768] 2024-08-17T17:51:25.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:24 smithi136 ceph-mon[109861]: osdmap e1273: 8 total, 8 up, 8 in 2024-08-17T17:51:25.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:24 smithi136 ceph-mon[109861]: pgmap v1990: 105 pgs: 105 active+clean; 1.0 GiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 220 KiB/s rd, 53 MiB/s wr, 352 op/s 2024-08-17T17:51:25.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:24 smithi136 ceph-mon[115078]: osdmap e1273: 8 total, 8 up, 8 in 2024-08-17T17:51:25.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:24 smithi136 ceph-mon[115078]: pgmap v1990: 105 pgs: 105 active+clean; 1.0 GiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 220 KiB/s rd, 53 MiB/s wr, 352 op/s 2024-08-17T17:51:25.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:24 smithi165 ceph-mon[105517]: osdmap e1273: 8 total, 8 up, 8 in 2024-08-17T17:51:25.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:24 smithi165 ceph-mon[105517]: pgmap v1990: 105 pgs: 105 active+clean; 1.0 GiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 220 KiB/s rd, 53 MiB/s wr, 352 op/s 2024-08-17T17:51:25.904 INFO:tasks.workunit.client.0.smithi136.stdout: i=19 exists [1031351~35156944,38431595~706215,39402060~14079959,53748579~26882433,81472010~16742859,99139028~3763239,103726598~5627320,109916240~2037465,112520365~6325671,119060682~11984089,132150039~6699759,147143781~10112691,158128092~14614072,174690623~1524415,180401384~10618906,193566721~7485796,202522110~8087466,212201669~1540869,213928485~7363606,221317665~14094546,237346036~6110719,244304172~189167,249474220~20379811,273807794~175066,274843948~16629205,292655901~49960397,349181703~9669183,363287350~1402505,365010930~10218478,375793367~6495094,384220755~9213613,395203737~23492676] wrote [193566721~3577973,223379120~8666932,259629347~3266163,280148875~3906392,365010930~3297429,395203737~7828211,408539727~1275621,411115772~7580641] 2024-08-17T17:51:26.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:25 smithi136 ceph-mon[109861]: osdmap e1274: 8 total, 8 up, 8 in 2024-08-17T17:51:26.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:25 smithi136 ceph-mon[115078]: osdmap e1274: 8 total, 8 up, 8 in 2024-08-17T17:51:26.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:25 smithi165 ceph-mon[105517]: osdmap e1274: 8 total, 8 up, 8 in 2024-08-17T17:51:26.847 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 1 diff [88080384~8388608,121634816~8388608,163577856~8388608,180355072~4194304,260046848~8388608,318767104~12582912,394264576~16777216] 2024-08-17T17:51:26.847 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [88080384~8388608,121634816~8388608,163577856~8388608,180355072~4194304,260046848~8388608,318767104~12582912,394264576~16777216] 2024-08-17T17:51:26.856 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [88080384~8388608,121634816~8388608,163577856~12582912,180355072~8388608,260046848~12582912,318767104~12582912,394264576~16777216] 2024-08-17T17:51:26.856 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 2 diff [12582912~8388608,83886080~12582912,121634816~8388608,146800640~8388608,163577856~8388608,180355072~4194304,205520896~4194304,239075328~4194304,260046848~8388608,285212672~4194304,318767104~12582912,356515840~8388608,394264576~16777216] 2024-08-17T17:51:26.857 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [12582912~8388608,83886080~12582912,121634816~8388608,146800640~8388608,163577856~8388608,180355072~4194304,205520896~4194304,239075328~4194304,260046848~8388608,285212672~4194304,318767104~12582912,356515840~8388608,394264576~16777216] 2024-08-17T17:51:26.858 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [12582912~12582912,83886080~12582912,121634816~8388608,146800640~8388608,163577856~12582912,180355072~8388608,205520896~8388608,239075328~8388608,260046848~12582912,285212672~8388608,318767104~12582912,356515840~8388608,394264576~16777216] 2024-08-17T17:51:26.858 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 3 diff [12582912~8388608,46137344~8388608,67108864~12582912,83886080~12582912,121634816~8388608,146800640~8388608,163577856~8388608,180355072~4194304,201326592~8388608,222298112~8388608,239075328~4194304,260046848~8388608,285212672~8388608,310378496~20971520,356515840~8388608,373293056~4194304,394264576~16777216] 2024-08-17T17:51:26.858 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [12582912~8388608,46137344~8388608,67108864~12582912,83886080~12582912,121634816~8388608,146800640~8388608,163577856~8388608,180355072~4194304,201326592~8388608,222298112~8388608,239075328~4194304,260046848~8388608,285212672~8388608,310378496~20971520,356515840~8388608,373293056~4194304,394264576~16777216] 2024-08-17T17:51:26.860 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [12582912~12582912,46137344~8388608,67108864~12582912,83886080~12582912,121634816~8388608,146800640~8388608,163577856~12582912,180355072~8388608,201326592~12582912,222298112~12582912,239075328~8388608,260046848~12582912,285212672~12582912,310378496~20971520,356515840~8388608,373293056~4194304,394264576~16777216] 2024-08-17T17:51:26.860 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 4 diff [12582912~8388608,29360128~8388608,46137344~8388608,58720256~4194304,67108864~12582912,83886080~12582912,109051904~8388608,121634816~8388608,146800640~12582912,163577856~8388608,180355072~4194304,192937984~16777216,218103808~12582912,239075328~4194304,260046848~8388608,285212672~8388608,310378496~20971520,348127232~20971520,373293056~4194304,394264576~16777216] 2024-08-17T17:51:26.860 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [12582912~8388608,29360128~8388608,46137344~8388608,58720256~4194304,67108864~12582912,83886080~12582912,109051904~8388608,121634816~8388608,146800640~12582912,163577856~8388608,180355072~4194304,192937984~16777216,218103808~12582912,239075328~4194304,260046848~8388608,285212672~8388608,310378496~20971520,348127232~20971520,373293056~4194304,394264576~16777216] 2024-08-17T17:51:26.862 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [12582912~12582912,29360128~8388608,46137344~8388608,58720256~4194304,67108864~12582912,83886080~12582912,109051904~20971520,146800640~12582912,163577856~12582912,180355072~8388608,192937984~20971520,218103808~16777216,239075328~8388608,260046848~12582912,285212672~12582912,310378496~20971520,348127232~29360128,394264576~16777216] 2024-08-17T17:51:26.863 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 5 diff [12582912~12582912,29360128~8388608,46137344~16777216,67108864~12582912,83886080~12582912,109051904~8388608,121634816~8388608,146800640~12582912,163577856~8388608,180355072~4194304,192937984~16777216,218103808~16777216,239075328~4194304,260046848~8388608,285212672~8388608,310378496~20971520,348127232~20971520,373293056~4194304,394264576~16777216] 2024-08-17T17:51:26.863 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [12582912~12582912,29360128~8388608,46137344~16777216,67108864~12582912,83886080~12582912,109051904~8388608,121634816~8388608,146800640~12582912,163577856~8388608,180355072~4194304,192937984~16777216,218103808~16777216,239075328~4194304,260046848~8388608,285212672~8388608,310378496~20971520,348127232~20971520,373293056~4194304,394264576~16777216] 2024-08-17T17:51:26.866 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [12582912~12582912,29360128~8388608,46137344~16777216,67108864~12582912,83886080~16777216,109051904~20971520,146800640~12582912,163577856~12582912,180355072~8388608,192937984~20971520,218103808~16777216,239075328~8388608,260046848~12582912,285212672~12582912,310378496~20971520,348127232~29360128,394264576~16777216] 2024-08-17T17:51:26.866 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 6 diff [4194304~16777216,29360128~8388608,46137344~16777216,67108864~12582912,83886080~12582912,109051904~20971520,146800640~12582912,163577856~8388608,180355072~4194304,192937984~16777216,218103808~16777216,239075328~4194304,260046848~8388608,276824064~16777216,310378496~20971520,335544320~4194304,348127232~29360128,394264576~16777216] 2024-08-17T17:51:26.866 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4194304~16777216,29360128~8388608,46137344~16777216,67108864~12582912,83886080~12582912,109051904~20971520,146800640~12582912,163577856~8388608,180355072~4194304,192937984~16777216,218103808~16777216,239075328~4194304,260046848~8388608,276824064~16777216,310378496~20971520,335544320~4194304,348127232~29360128,394264576~16777216] 2024-08-17T17:51:26.868 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~20971520,29360128~8388608,46137344~16777216,67108864~12582912,83886080~16777216,109051904~20971520,146800640~12582912,163577856~12582912,180355072~8388608,192937984~20971520,218103808~29360128,260046848~12582912,276824064~20971520,310378496~20971520,335544320~4194304,348127232~33554432,394264576~16777216] 2024-08-17T17:51:26.868 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 7 diff [4194304~16777216,29360128~8388608,41943040~20971520,67108864~12582912,83886080~16777216,109051904~20971520,146800640~12582912,163577856~8388608,180355072~4194304,192937984~16777216,218103808~16777216,239075328~4194304,260046848~8388608,276824064~16777216,297795584~8388608,310378496~20971520,335544320~4194304,348127232~33554432,394264576~16777216] 2024-08-17T17:51:26.868 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4194304~16777216,29360128~8388608,41943040~20971520,67108864~12582912,83886080~16777216,109051904~20971520,146800640~12582912,163577856~8388608,180355072~4194304,192937984~16777216,218103808~16777216,239075328~4194304,260046848~8388608,276824064~16777216,297795584~8388608,310378496~20971520,335544320~4194304,348127232~33554432,394264576~16777216] 2024-08-17T17:51:26.871 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~20971520,29360128~8388608,41943040~20971520,67108864~12582912,83886080~20971520,109051904~25165824,146800640~12582912,163577856~12582912,180355072~8388608,192937984~20971520,218103808~29360128,260046848~12582912,276824064~54525952,335544320~4194304,348127232~33554432,394264576~16777216] 2024-08-17T17:51:26.871 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 8 diff [4194304~16777216,29360128~33554432,67108864~12582912,83886080~16777216,109051904~20971520,146800640~25165824,180355072~8388608,192937984~16777216,213909504~20971520,239075328~4194304,260046848~8388608,276824064~29360128,310378496~20971520,335544320~4194304,348127232~33554432,394264576~16777216] 2024-08-17T17:51:26.871 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4194304~16777216,29360128~33554432,67108864~12582912,83886080~16777216,109051904~20971520,146800640~25165824,180355072~8388608,192937984~16777216,213909504~20971520,239075328~4194304,260046848~8388608,276824064~29360128,310378496~20971520,335544320~4194304,348127232~33554432,394264576~16777216] 2024-08-17T17:51:26.875 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~20971520,29360128~33554432,67108864~12582912,83886080~20971520,109051904~25165824,146800640~29360128,180355072~67108864,260046848~12582912,276824064~54525952,335544320~4194304,348127232~33554432,394264576~16777216] 2024-08-17T17:51:26.875 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 9 diff [0~20971520,25165824~37748736,67108864~33554432,109051904~20971520,146800640~25165824,180355072~8388608,192937984~16777216,213909504~20971520,239075328~4194304,260046848~8388608,276824064~29360128,310378496~20971520,335544320~4194304,348127232~33554432,394264576~16777216] 2024-08-17T17:51:26.876 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~20971520,25165824~37748736,67108864~33554432,109051904~20971520,146800640~25165824,180355072~8388608,192937984~16777216,213909504~20971520,239075328~4194304,260046848~8388608,276824064~29360128,310378496~20971520,335544320~4194304,348127232~33554432,394264576~16777216] 2024-08-17T17:51:26.880 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~62914560,67108864~37748736,109051904~25165824,146800640~29360128,180355072~67108864,260046848~12582912,276824064~54525952,335544320~4194304,348127232~33554432,394264576~16777216] 2024-08-17T17:51:26.880 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 10 diff [0~100663296,109051904~20971520,146800640~25165824,180355072~8388608,192937984~16777216,213909504~20971520,239075328~4194304,260046848~8388608,276824064~29360128,310378496~29360128,348127232~41943040,394264576~16777216] 2024-08-17T17:51:26.880 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~100663296,109051904~20971520,146800640~25165824,180355072~8388608,192937984~16777216,213909504~20971520,239075328~4194304,260046848~8388608,276824064~29360128,310378496~29360128,348127232~41943040,394264576~16777216] 2024-08-17T17:51:26.887 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~25165824,146800640~29360128,180355072~67108864,260046848~12582912,276824064~62914560,348127232~62914560] 2024-08-17T17:51:26.887 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 11 diff [0~100663296,109051904~20971520,146800640~25165824,180355072~29360128,213909504~20971520,239075328~4194304,247463936~8388608,260046848~8388608,272629760~67108864,348127232~41943040,394264576~16777216] 2024-08-17T17:51:26.887 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~100663296,109051904~20971520,146800640~25165824,180355072~29360128,213909504~20971520,239075328~4194304,247463936~8388608,260046848~8388608,272629760~67108864,348127232~41943040,394264576~16777216] 2024-08-17T17:51:26.892 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~25165824,146800640~29360128,180355072~75497472,260046848~79691776,348127232~67108864] 2024-08-17T17:51:26.892 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 12 diff [0~100663296,109051904~20971520,146800640~25165824,180355072~29360128,213909504~29360128,247463936~8388608,260046848~8388608,272629760~67108864,348127232~41943040,394264576~16777216] 2024-08-17T17:51:26.892 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~100663296,109051904~20971520,146800640~25165824,180355072~29360128,213909504~29360128,247463936~8388608,260046848~8388608,272629760~67108864,348127232~41943040,394264576~16777216] 2024-08-17T17:51:26.897 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~25165824,146800640~29360128,180355072~75497472,260046848~79691776,348127232~67108864] 2024-08-17T17:51:26.897 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 13 diff [0~100663296,109051904~20971520,146800640~29360128,180355072~29360128,213909504~29360128,247463936~8388608,260046848~8388608,272629760~67108864,348127232~41943040,394264576~16777216] 2024-08-17T17:51:26.897 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~100663296,109051904~20971520,146800640~29360128,180355072~29360128,213909504~29360128,247463936~8388608,260046848~8388608,272629760~67108864,348127232~41943040,394264576~16777216] 2024-08-17T17:51:26.904 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~25165824,146800640~109051904,260046848~79691776,348127232~67108864] 2024-08-17T17:51:26.904 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 14 diff [0~100663296,109051904~20971520,146800640~29360128,180355072~159383552,348127232~41943040,394264576~20971520] 2024-08-17T17:51:26.904 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~100663296,109051904~20971520,146800640~29360128,180355072~159383552,348127232~41943040,394264576~20971520] 2024-08-17T17:51:26.910 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~25165824,146800640~192937984,348127232~67108864] 2024-08-17T17:51:26.910 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 15 diff [0~100663296,109051904~29360128,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~20971520] 2024-08-17T17:51:26.910 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~100663296,109051904~29360128,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~20971520] 2024-08-17T17:51:26.916 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~33554432,146800640~197132288,348127232~67108864] 2024-08-17T17:51:26.916 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 16 diff [0~100663296,109051904~29360128,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~20971520] 2024-08-17T17:51:26.916 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~100663296,109051904~29360128,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~20971520] 2024-08-17T17:51:26.922 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~33554432,146800640~197132288,348127232~67108864] 2024-08-17T17:51:26.922 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 17 diff [0~100663296,109051904~29360128,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~20971520] 2024-08-17T17:51:26.922 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~100663296,109051904~29360128,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~20971520] 2024-08-17T17:51:26.929 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~33554432,146800640~197132288,348127232~67108864] 2024-08-17T17:51:26.929 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 18 diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~16777216] 2024-08-17T17:51:26.929 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~16777216] 2024-08-17T17:51:26.936 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~67108864] 2024-08-17T17:51:26.936 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to 19 diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:26.936 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:26.944 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:26.944 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 2 diff [12582912~8388608,83886080~8388608,146800640~8388608,205520896~4194304,239075328~4194304,285212672~4194304,356515840~8388608] 2024-08-17T17:51:26.944 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [12582912~8388608,83886080~8388608,146800640~8388608,205520896~4194304,239075328~4194304,285212672~4194304,356515840~8388608] 2024-08-17T17:51:26.945 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [12582912~12582912,83886080~8388608,146800640~8388608,205520896~8388608,239075328~8388608,285212672~8388608,356515840~8388608] 2024-08-17T17:51:26.945 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 3 diff [12582912~8388608,46137344~8388608,67108864~12582912,83886080~8388608,121634816~4194304,146800640~8388608,201326592~8388608,222298112~8388608,239075328~4194304,285212672~8388608,310378496~8388608,356515840~8388608,373293056~4194304] 2024-08-17T17:51:26.945 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [12582912~8388608,46137344~8388608,67108864~12582912,83886080~8388608,121634816~4194304,146800640~8388608,201326592~8388608,222298112~8388608,239075328~4194304,285212672~8388608,310378496~8388608,356515840~8388608,373293056~4194304] 2024-08-17T17:51:26.947 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [12582912~12582912,46137344~8388608,67108864~12582912,83886080~8388608,121634816~8388608,146800640~8388608,201326592~12582912,222298112~12582912,239075328~8388608,285212672~12582912,310378496~8388608,356515840~8388608,373293056~4194304] 2024-08-17T17:51:26.947 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 4 diff [12582912~8388608,29360128~8388608,46137344~8388608,58720256~4194304,67108864~12582912,83886080~8388608,109051904~8388608,121634816~4194304,146800640~12582912,192937984~16777216,218103808~12582912,239075328~4194304,285212672~8388608,310378496~8388608,348127232~20971520,373293056~4194304,406847488~4194304] 2024-08-17T17:51:26.947 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [12582912~8388608,29360128~8388608,46137344~8388608,58720256~4194304,67108864~12582912,83886080~8388608,109051904~8388608,121634816~4194304,146800640~12582912,192937984~16777216,218103808~12582912,239075328~4194304,285212672~8388608,310378496~8388608,348127232~20971520,373293056~4194304,406847488~4194304] 2024-08-17T17:51:26.949 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [12582912~12582912,29360128~8388608,46137344~8388608,58720256~4194304,67108864~12582912,83886080~8388608,109051904~20971520,146800640~12582912,192937984~20971520,218103808~16777216,239075328~8388608,285212672~12582912,310378496~8388608,348127232~29360128,406847488~4194304] 2024-08-17T17:51:26.949 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 5 diff [12582912~12582912,29360128~8388608,46137344~16777216,67108864~12582912,83886080~12582912,109051904~8388608,121634816~4194304,146800640~12582912,180355072~4194304,192937984~16777216,218103808~16777216,239075328~4194304,285212672~8388608,310378496~8388608,348127232~20971520,373293056~4194304,406847488~4194304] 2024-08-17T17:51:26.949 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [12582912~12582912,29360128~8388608,46137344~16777216,67108864~12582912,83886080~12582912,109051904~8388608,121634816~4194304,146800640~12582912,180355072~4194304,192937984~16777216,218103808~16777216,239075328~4194304,285212672~8388608,310378496~8388608,348127232~20971520,373293056~4194304,406847488~4194304] 2024-08-17T17:51:26.951 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [12582912~12582912,29360128~8388608,46137344~16777216,67108864~12582912,83886080~16777216,109051904~20971520,146800640~12582912,180355072~4194304,192937984~20971520,218103808~16777216,239075328~8388608,285212672~12582912,310378496~8388608,348127232~29360128,406847488~4194304] 2024-08-17T17:51:26.951 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 6 diff [4194304~16777216,29360128~8388608,46137344~16777216,67108864~12582912,83886080~12582912,109051904~20971520,146800640~12582912,180355072~4194304,192937984~16777216,218103808~16777216,239075328~4194304,276824064~16777216,310378496~8388608,335544320~4194304,348127232~29360128,402653184~8388608] 2024-08-17T17:51:26.951 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4194304~16777216,29360128~8388608,46137344~16777216,67108864~12582912,83886080~12582912,109051904~20971520,146800640~12582912,180355072~4194304,192937984~16777216,218103808~16777216,239075328~4194304,276824064~16777216,310378496~8388608,335544320~4194304,348127232~29360128,402653184~8388608] 2024-08-17T17:51:26.953 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~20971520,29360128~8388608,46137344~16777216,67108864~12582912,83886080~16777216,109051904~20971520,146800640~12582912,180355072~4194304,192937984~20971520,218103808~29360128,276824064~20971520,310378496~8388608,335544320~4194304,348127232~33554432,402653184~8388608] 2024-08-17T17:51:26.953 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 7 diff [4194304~16777216,29360128~8388608,41943040~20971520,67108864~12582912,83886080~16777216,109051904~20971520,146800640~12582912,167772160~4194304,180355072~4194304,192937984~16777216,218103808~16777216,239075328~4194304,276824064~16777216,297795584~8388608,310378496~8388608,335544320~4194304,348127232~33554432,402653184~8388608] 2024-08-17T17:51:26.954 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4194304~16777216,29360128~8388608,41943040~20971520,67108864~12582912,83886080~16777216,109051904~20971520,146800640~12582912,167772160~4194304,180355072~4194304,192937984~16777216,218103808~16777216,239075328~4194304,276824064~16777216,297795584~8388608,310378496~8388608,335544320~4194304,348127232~33554432,402653184~8388608] 2024-08-17T17:51:26.956 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~20971520,29360128~8388608,41943040~20971520,67108864~12582912,83886080~20971520,109051904~25165824,146800640~12582912,167772160~4194304,180355072~4194304,192937984~20971520,218103808~29360128,276824064~41943040,335544320~4194304,348127232~33554432,402653184~8388608] 2024-08-17T17:51:26.957 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 8 diff [4194304~16777216,29360128~33554432,67108864~12582912,83886080~16777216,109051904~20971520,146800640~16777216,167772160~4194304,180355072~8388608,192937984~16777216,213909504~20971520,239075328~4194304,276824064~29360128,310378496~8388608,327155712~4194304,335544320~4194304,348127232~33554432,402653184~8388608] 2024-08-17T17:51:26.957 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4194304~16777216,29360128~33554432,67108864~12582912,83886080~16777216,109051904~20971520,146800640~16777216,167772160~4194304,180355072~8388608,192937984~16777216,213909504~20971520,239075328~4194304,276824064~29360128,310378496~8388608,327155712~4194304,335544320~4194304,348127232~33554432,402653184~8388608] 2024-08-17T17:51:26.960 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~20971520,29360128~33554432,67108864~12582912,83886080~20971520,109051904~25165824,146800640~25165824,180355072~67108864,276824064~41943040,327155712~4194304,335544320~4194304,348127232~33554432,402653184~8388608] 2024-08-17T17:51:26.960 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 9 diff [0~20971520,25165824~37748736,67108864~33554432,109051904~20971520,146800640~16777216,167772160~4194304,180355072~8388608,192937984~16777216,213909504~20971520,239075328~4194304,276824064~29360128,310378496~8388608,327155712~4194304,335544320~4194304,348127232~33554432,402653184~8388608] 2024-08-17T17:51:26.960 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~20971520,25165824~37748736,67108864~33554432,109051904~20971520,146800640~16777216,167772160~4194304,180355072~8388608,192937984~16777216,213909504~20971520,239075328~4194304,276824064~29360128,310378496~8388608,327155712~4194304,335544320~4194304,348127232~33554432,402653184~8388608] 2024-08-17T17:51:26.963 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~62914560,67108864~37748736,109051904~25165824,146800640~25165824,180355072~67108864,276824064~41943040,327155712~4194304,335544320~4194304,348127232~33554432,402653184~8388608] 2024-08-17T17:51:26.963 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 10 diff [0~100663296,109051904~20971520,146800640~16777216,167772160~4194304,180355072~8388608,192937984~16777216,213909504~20971520,239075328~4194304,276824064~29360128,310378496~8388608,322961408~16777216,348127232~41943040,402653184~8388608] 2024-08-17T17:51:26.963 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~100663296,109051904~20971520,146800640~16777216,167772160~4194304,180355072~8388608,192937984~16777216,213909504~20971520,239075328~4194304,276824064~29360128,310378496~8388608,322961408~16777216,348127232~41943040,402653184~8388608] 2024-08-17T17:51:26.967 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~25165824,146800640~25165824,180355072~67108864,276824064~41943040,322961408~16777216,348127232~46137344,402653184~8388608] 2024-08-17T17:51:26.967 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 11 diff [0~100663296,109051904~20971520,146800640~16777216,167772160~4194304,180355072~29360128,213909504~20971520,239075328~4194304,247463936~8388608,272629760~46137344,322961408~16777216,348127232~41943040,402653184~8388608] 2024-08-17T17:51:26.967 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~100663296,109051904~20971520,146800640~16777216,167772160~4194304,180355072~29360128,213909504~20971520,239075328~4194304,247463936~8388608,272629760~46137344,322961408~16777216,348127232~41943040,402653184~8388608] 2024-08-17T17:51:26.971 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~25165824,146800640~25165824,180355072~75497472,272629760~46137344,322961408~16777216,348127232~46137344,402653184~12582912] 2024-08-17T17:51:26.971 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 12 diff [0~100663296,109051904~20971520,146800640~25165824,180355072~29360128,213909504~29360128,247463936~8388608,272629760~46137344,322961408~16777216,348127232~41943040,402653184~8388608] 2024-08-17T17:51:26.971 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~100663296,109051904~20971520,146800640~25165824,180355072~29360128,213909504~29360128,247463936~8388608,272629760~46137344,322961408~16777216,348127232~41943040,402653184~8388608] 2024-08-17T17:51:26.975 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~25165824,146800640~25165824,180355072~75497472,272629760~46137344,322961408~16777216,348127232~46137344,402653184~12582912] 2024-08-17T17:51:26.976 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 13 diff [0~100663296,109051904~20971520,146800640~29360128,180355072~29360128,213909504~29360128,247463936~8388608,272629760~46137344,322961408~16777216,348127232~41943040,402653184~8388608] 2024-08-17T17:51:26.976 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~100663296,109051904~20971520,146800640~29360128,180355072~29360128,213909504~29360128,247463936~8388608,272629760~46137344,322961408~16777216,348127232~41943040,402653184~8388608] 2024-08-17T17:51:26.980 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~25165824,146800640~109051904,272629760~46137344,322961408~16777216,348127232~46137344,402653184~12582912] 2024-08-17T17:51:26.980 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 14 diff [0~100663296,109051904~20971520,146800640~29360128,180355072~83886080,272629760~46137344,322961408~16777216,348127232~41943040,402653184~12582912] 2024-08-17T17:51:26.980 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~100663296,109051904~20971520,146800640~29360128,180355072~83886080,272629760~46137344,322961408~16777216,348127232~41943040,402653184~12582912] 2024-08-17T17:51:26.986 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~25165824,146800640~117440512,272629760~67108864,348127232~46137344,402653184~12582912] 2024-08-17T17:51:26.986 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 15 diff [0~100663296,109051904~29360128,146800640~29360128,180355072~83886080,272629760~46137344,322961408~16777216,348127232~41943040,402653184~12582912] 2024-08-17T17:51:26.986 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~100663296,109051904~29360128,146800640~29360128,180355072~83886080,272629760~46137344,322961408~16777216,348127232~41943040,402653184~12582912] 2024-08-17T17:51:26.991 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~33554432,146800640~121634816,272629760~71303168,348127232~46137344,402653184~12582912] 2024-08-17T17:51:26.991 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 16 diff [0~100663296,109051904~29360128,146800640~29360128,180355072~83886080,272629760~46137344,322961408~16777216,348127232~41943040,402653184~12582912] 2024-08-17T17:51:26.991 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~100663296,109051904~29360128,146800640~29360128,180355072~83886080,272629760~46137344,322961408~16777216,348127232~41943040,402653184~12582912] 2024-08-17T17:51:26.997 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~33554432,146800640~121634816,272629760~71303168,348127232~46137344,402653184~12582912] 2024-08-17T17:51:26.997 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 17 diff [0~100663296,109051904~29360128,146800640~29360128,180355072~88080384,272629760~50331648,327155712~12582912,348127232~41943040,402653184~12582912] 2024-08-17T17:51:26.997 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~100663296,109051904~29360128,146800640~29360128,180355072~88080384,272629760~50331648,327155712~12582912,348127232~41943040,402653184~12582912] 2024-08-17T17:51:27.003 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~33554432,146800640~197132288,348127232~46137344,402653184~12582912] 2024-08-17T17:51:27.003 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 18 diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,398458880~16777216] 2024-08-17T17:51:27.003 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,398458880~16777216] 2024-08-17T17:51:27.010 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~46137344,398458880~16777216] 2024-08-17T17:51:27.010 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to 19 diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:27.010 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:27.016 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:27.016 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 3 diff [46137344~8388608,67108864~12582912,121634816~4194304,146800640~8388608,201326592~8388608,222298112~8388608,289406976~4194304,310378496~8388608,373293056~4194304] 2024-08-17T17:51:27.016 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [46137344~8388608,67108864~12582912,121634816~4194304,146800640~8388608,201326592~8388608,222298112~8388608,289406976~4194304,310378496~8388608,373293056~4194304] 2024-08-17T17:51:27.017 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [46137344~8388608,67108864~12582912,121634816~8388608,146800640~8388608,201326592~12582912,222298112~12582912,289406976~8388608,310378496~8388608,373293056~4194304] 2024-08-17T17:51:27.017 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 4 diff [29360128~8388608,46137344~8388608,58720256~4194304,67108864~12582912,109051904~8388608,121634816~4194304,146800640~12582912,192937984~16777216,218103808~12582912,289406976~4194304,310378496~8388608,348127232~8388608,364904448~4194304,373293056~4194304,406847488~4194304] 2024-08-17T17:51:27.018 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [29360128~8388608,46137344~8388608,58720256~4194304,67108864~12582912,109051904~8388608,121634816~4194304,146800640~12582912,192937984~16777216,218103808~12582912,289406976~4194304,310378496~8388608,348127232~8388608,364904448~4194304,373293056~4194304,406847488~4194304] 2024-08-17T17:51:27.019 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [29360128~8388608,46137344~8388608,58720256~4194304,67108864~12582912,109051904~20971520,146800640~12582912,192937984~20971520,218103808~16777216,289406976~8388608,310378496~8388608,348127232~8388608,364904448~12582912,406847488~4194304] 2024-08-17T17:51:27.019 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 5 diff [12582912~8388608,29360128~8388608,46137344~16777216,67108864~12582912,83886080~12582912,109051904~8388608,121634816~4194304,146800640~12582912,180355072~4194304,192937984~16777216,218103808~16777216,289406976~4194304,310378496~8388608,348127232~12582912,364904448~4194304,373293056~4194304,406847488~4194304] 2024-08-17T17:51:27.019 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [12582912~8388608,29360128~8388608,46137344~16777216,67108864~12582912,83886080~12582912,109051904~8388608,121634816~4194304,146800640~12582912,180355072~4194304,192937984~16777216,218103808~16777216,289406976~4194304,310378496~8388608,348127232~12582912,364904448~4194304,373293056~4194304,406847488~4194304] 2024-08-17T17:51:27.020 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [12582912~8388608,29360128~8388608,46137344~16777216,67108864~12582912,83886080~16777216,109051904~20971520,146800640~12582912,180355072~4194304,192937984~20971520,218103808~16777216,289406976~8388608,310378496~8388608,348127232~12582912,364904448~12582912,406847488~4194304] 2024-08-17T17:51:27.021 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 6 diff [4194304~12582912,29360128~8388608,46137344~16777216,67108864~12582912,83886080~12582912,109051904~20971520,146800640~12582912,180355072~4194304,192937984~16777216,218103808~16777216,276824064~8388608,289406976~4194304,310378496~8388608,335544320~4194304,348127232~12582912,364904448~12582912,402653184~8388608] 2024-08-17T17:51:27.021 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4194304~12582912,29360128~8388608,46137344~16777216,67108864~12582912,83886080~12582912,109051904~20971520,146800640~12582912,180355072~4194304,192937984~16777216,218103808~16777216,276824064~8388608,289406976~4194304,310378496~8388608,335544320~4194304,348127232~12582912,364904448~12582912,402653184~8388608] 2024-08-17T17:51:27.023 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~16777216,29360128~8388608,46137344~16777216,67108864~12582912,83886080~16777216,109051904~20971520,146800640~12582912,180355072~4194304,192937984~20971520,218103808~20971520,276824064~20971520,310378496~8388608,335544320~4194304,348127232~12582912,364904448~16777216,402653184~8388608] 2024-08-17T17:51:27.023 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 7 diff [4194304~12582912,29360128~8388608,41943040~20971520,67108864~12582912,83886080~16777216,109051904~20971520,146800640~12582912,167772160~4194304,180355072~4194304,192937984~16777216,218103808~16777216,276824064~8388608,289406976~4194304,297795584~8388608,310378496~8388608,335544320~4194304,348127232~12582912,364904448~16777216,402653184~8388608] 2024-08-17T17:51:27.023 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4194304~12582912,29360128~8388608,41943040~20971520,67108864~12582912,83886080~16777216,109051904~20971520,146800640~12582912,167772160~4194304,180355072~4194304,192937984~16777216,218103808~16777216,276824064~8388608,289406976~4194304,297795584~8388608,310378496~8388608,335544320~4194304,348127232~12582912,364904448~16777216,402653184~8388608] 2024-08-17T17:51:27.025 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~16777216,29360128~8388608,41943040~20971520,67108864~12582912,83886080~20971520,109051904~25165824,146800640~12582912,167772160~4194304,180355072~4194304,192937984~20971520,218103808~20971520,276824064~41943040,335544320~4194304,348127232~12582912,364904448~16777216,402653184~8388608] 2024-08-17T17:51:27.025 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 8 diff [4194304~12582912,29360128~33554432,67108864~12582912,83886080~16777216,109051904~20971520,146800640~16777216,167772160~4194304,180355072~8388608,192937984~16777216,213909504~20971520,276824064~8388608,289406976~16777216,310378496~8388608,327155712~4194304,335544320~4194304,348127232~12582912,364904448~16777216,402653184~8388608] 2024-08-17T17:51:27.025 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4194304~12582912,29360128~33554432,67108864~12582912,83886080~16777216,109051904~20971520,146800640~16777216,167772160~4194304,180355072~8388608,192937984~16777216,213909504~20971520,276824064~8388608,289406976~16777216,310378496~8388608,327155712~4194304,335544320~4194304,348127232~12582912,364904448~16777216,402653184~8388608] 2024-08-17T17:51:27.028 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~16777216,29360128~33554432,67108864~12582912,83886080~20971520,109051904~25165824,146800640~25165824,180355072~58720256,276824064~41943040,327155712~4194304,335544320~4194304,348127232~12582912,364904448~16777216,402653184~8388608] 2024-08-17T17:51:27.028 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 9 diff [0~16777216,25165824~37748736,67108864~33554432,109051904~20971520,146800640~16777216,167772160~4194304,180355072~8388608,192937984~16777216,213909504~20971520,276824064~8388608,289406976~16777216,310378496~8388608,327155712~4194304,335544320~4194304,348127232~12582912,364904448~16777216,402653184~8388608] 2024-08-17T17:51:27.028 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~16777216,25165824~37748736,67108864~33554432,109051904~20971520,146800640~16777216,167772160~4194304,180355072~8388608,192937984~16777216,213909504~20971520,276824064~8388608,289406976~16777216,310378496~8388608,327155712~4194304,335544320~4194304,348127232~12582912,364904448~16777216,402653184~8388608] 2024-08-17T17:51:27.031 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~20971520,25165824~37748736,67108864~37748736,109051904~25165824,146800640~25165824,180355072~58720256,276824064~41943040,327155712~4194304,335544320~4194304,348127232~12582912,364904448~16777216,402653184~8388608] 2024-08-17T17:51:27.031 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 10 diff [0~100663296,109051904~20971520,146800640~16777216,167772160~4194304,180355072~8388608,192937984~16777216,213909504~20971520,276824064~8388608,289406976~16777216,310378496~8388608,322961408~16777216,348127232~12582912,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.031 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~100663296,109051904~20971520,146800640~16777216,167772160~4194304,180355072~8388608,192937984~16777216,213909504~20971520,276824064~8388608,289406976~16777216,310378496~8388608,322961408~16777216,348127232~12582912,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.034 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~25165824,146800640~25165824,180355072~58720256,276824064~41943040,322961408~16777216,348127232~12582912,364904448~29360128,402653184~8388608] 2024-08-17T17:51:27.034 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 11 diff [0~100663296,109051904~20971520,146800640~16777216,167772160~4194304,180355072~29360128,213909504~20971520,247463936~8388608,272629760~12582912,289406976~29360128,322961408~16777216,348127232~12582912,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.034 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~100663296,109051904~20971520,146800640~16777216,167772160~4194304,180355072~29360128,213909504~20971520,247463936~8388608,272629760~12582912,289406976~29360128,322961408~16777216,348127232~12582912,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.038 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~25165824,146800640~25165824,180355072~58720256,247463936~8388608,272629760~46137344,322961408~16777216,348127232~12582912,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.038 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 12 diff [0~100663296,109051904~20971520,146800640~25165824,180355072~29360128,213909504~25165824,247463936~8388608,272629760~12582912,289406976~29360128,322961408~16777216,348127232~12582912,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.038 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~100663296,109051904~20971520,146800640~25165824,180355072~29360128,213909504~25165824,247463936~8388608,272629760~12582912,289406976~29360128,322961408~16777216,348127232~12582912,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.042 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~25165824,146800640~25165824,180355072~62914560,247463936~8388608,272629760~46137344,322961408~16777216,348127232~12582912,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.042 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 13 diff [0~100663296,109051904~20971520,146800640~29360128,180355072~29360128,213909504~25165824,247463936~8388608,272629760~12582912,289406976~29360128,322961408~16777216,348127232~12582912,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.042 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~100663296,109051904~20971520,146800640~29360128,180355072~29360128,213909504~25165824,247463936~8388608,272629760~12582912,289406976~29360128,322961408~16777216,348127232~12582912,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.046 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~25165824,146800640~96468992,247463936~8388608,272629760~46137344,322961408~16777216,348127232~12582912,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.047 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 14 diff [0~100663296,109051904~20971520,146800640~29360128,180355072~58720256,243269632~20971520,272629760~46137344,322961408~16777216,348127232~12582912,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.047 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~100663296,109051904~20971520,146800640~29360128,180355072~58720256,243269632~20971520,272629760~46137344,322961408~16777216,348127232~12582912,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.051 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~25165824,146800640~117440512,272629760~67108864,348127232~12582912,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.051 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 15 diff [0~100663296,109051904~29360128,146800640~29360128,180355072~58720256,243269632~20971520,272629760~46137344,322961408~16777216,348127232~12582912,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.051 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~100663296,109051904~29360128,146800640~29360128,180355072~58720256,243269632~20971520,272629760~46137344,322961408~16777216,348127232~12582912,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.056 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~33554432,146800640~121634816,272629760~71303168,348127232~12582912,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.056 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 16 diff [0~100663296,109051904~29360128,146800640~29360128,180355072~58720256,243269632~20971520,272629760~46137344,322961408~16777216,348127232~12582912,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.056 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~100663296,109051904~29360128,146800640~29360128,180355072~58720256,243269632~20971520,272629760~46137344,322961408~16777216,348127232~12582912,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.062 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~33554432,146800640~121634816,272629760~71303168,348127232~12582912,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.062 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 17 diff [0~100663296,109051904~29360128,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~12582912,348127232~12582912,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.062 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~100663296,109051904~29360128,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~12582912,348127232~12582912,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.068 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~33554432,146800640~197132288,348127232~12582912,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.068 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 18 diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,398458880~16777216] 2024-08-17T17:51:27.068 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,398458880~16777216] 2024-08-17T17:51:27.074 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~29360128,398458880~16777216] 2024-08-17T17:51:27.074 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to 19 diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.074 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.081 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:27.081 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 4 diff [29360128~8388608,58720256~4194304,109051904~8388608,150994944~8388608,192937984~8388608,218103808~8388608,348127232~8388608,364904448~4194304,406847488~4194304] 2024-08-17T17:51:27.081 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [29360128~8388608,58720256~4194304,109051904~8388608,150994944~8388608,192937984~8388608,218103808~8388608,348127232~8388608,364904448~4194304,406847488~4194304] 2024-08-17T17:51:27.082 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [29360128~8388608,58720256~4194304,109051904~12582912,150994944~8388608,192937984~8388608,218103808~12582912,348127232~8388608,364904448~8388608,406847488~4194304] 2024-08-17T17:51:27.082 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 5 diff [12582912~8388608,29360128~8388608,54525952~8388608,83886080~12582912,109051904~8388608,150994944~8388608,180355072~4194304,192937984~8388608,218103808~8388608,230686720~4194304,310378496~8388608,348127232~12582912,364904448~4194304,406847488~4194304] 2024-08-17T17:51:27.082 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [12582912~8388608,29360128~8388608,54525952~8388608,83886080~12582912,109051904~8388608,150994944~8388608,180355072~4194304,192937984~8388608,218103808~8388608,230686720~4194304,310378496~8388608,348127232~12582912,364904448~4194304,406847488~4194304] 2024-08-17T17:51:27.084 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [12582912~8388608,29360128~8388608,54525952~8388608,83886080~16777216,109051904~12582912,150994944~8388608,180355072~4194304,192937984~8388608,218103808~16777216,310378496~8388608,348127232~12582912,364904448~8388608,406847488~4194304] 2024-08-17T17:51:27.084 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 6 diff [4194304~12582912,29360128~8388608,54525952~8388608,83886080~12582912,109051904~16777216,150994944~8388608,180355072~4194304,192937984~8388608,218103808~16777216,276824064~8388608,310378496~8388608,335544320~4194304,348127232~12582912,364904448~12582912,402653184~8388608] 2024-08-17T17:51:27.084 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4194304~12582912,29360128~8388608,54525952~8388608,83886080~12582912,109051904~16777216,150994944~8388608,180355072~4194304,192937984~8388608,218103808~16777216,276824064~8388608,310378496~8388608,335544320~4194304,348127232~12582912,364904448~12582912,402653184~8388608] 2024-08-17T17:51:27.085 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~16777216,29360128~8388608,54525952~8388608,83886080~16777216,109051904~20971520,150994944~8388608,180355072~4194304,192937984~8388608,218103808~20971520,276824064~12582912,310378496~8388608,335544320~4194304,348127232~12582912,364904448~16777216,402653184~8388608] 2024-08-17T17:51:27.085 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 7 diff [4194304~12582912,29360128~8388608,41943040~8388608,54525952~8388608,83886080~16777216,109051904~20971520,150994944~8388608,167772160~4194304,180355072~4194304,192937984~8388608,218103808~16777216,276824064~8388608,297795584~8388608,310378496~8388608,335544320~4194304,348127232~12582912,364904448~16777216,402653184~8388608] 2024-08-17T17:51:27.086 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4194304~12582912,29360128~8388608,41943040~8388608,54525952~8388608,83886080~16777216,109051904~20971520,150994944~8388608,167772160~4194304,180355072~4194304,192937984~8388608,218103808~16777216,276824064~8388608,297795584~8388608,310378496~8388608,335544320~4194304,348127232~12582912,364904448~16777216,402653184~8388608] 2024-08-17T17:51:27.087 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~16777216,29360128~8388608,41943040~8388608,54525952~8388608,83886080~20971520,109051904~25165824,150994944~8388608,167772160~4194304,180355072~4194304,192937984~8388608,218103808~20971520,276824064~12582912,297795584~20971520,335544320~4194304,348127232~12582912,364904448~16777216,402653184~8388608] 2024-08-17T17:51:27.087 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 8 diff [4194304~12582912,29360128~20971520,54525952~8388608,83886080~16777216,109051904~20971520,150994944~12582912,167772160~4194304,180355072~8388608,192937984~8388608,213909504~20971520,276824064~8388608,293601280~12582912,310378496~8388608,327155712~4194304,335544320~4194304,348127232~12582912,364904448~16777216,402653184~8388608] 2024-08-17T17:51:27.087 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4194304~12582912,29360128~20971520,54525952~8388608,83886080~16777216,109051904~20971520,150994944~12582912,167772160~4194304,180355072~8388608,192937984~8388608,213909504~20971520,276824064~8388608,293601280~12582912,310378496~8388608,327155712~4194304,335544320~4194304,348127232~12582912,364904448~16777216,402653184~8388608] 2024-08-17T17:51:27.090 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~16777216,29360128~20971520,54525952~8388608,83886080~20971520,109051904~25165824,150994944~20971520,180355072~20971520,213909504~25165824,276824064~12582912,293601280~25165824,327155712~4194304,335544320~4194304,348127232~12582912,364904448~16777216,402653184~8388608] 2024-08-17T17:51:27.090 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 9 diff [0~16777216,25165824~25165824,54525952~8388608,79691776~20971520,109051904~20971520,150994944~12582912,167772160~4194304,180355072~8388608,192937984~16777216,213909504~20971520,276824064~8388608,293601280~12582912,310378496~8388608,327155712~4194304,335544320~4194304,348127232~12582912,364904448~16777216,402653184~8388608] 2024-08-17T17:51:27.090 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~16777216,25165824~25165824,54525952~8388608,79691776~20971520,109051904~20971520,150994944~12582912,167772160~4194304,180355072~8388608,192937984~16777216,213909504~20971520,276824064~8388608,293601280~12582912,310378496~8388608,327155712~4194304,335544320~4194304,348127232~12582912,364904448~16777216,402653184~8388608] 2024-08-17T17:51:27.092 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~20971520,25165824~25165824,54525952~8388608,79691776~25165824,109051904~25165824,150994944~20971520,180355072~29360128,213909504~25165824,276824064~12582912,293601280~25165824,327155712~4194304,335544320~4194304,348127232~12582912,364904448~16777216,402653184~8388608] 2024-08-17T17:51:27.092 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 10 diff [0~50331648,54525952~12582912,79691776~20971520,109051904~20971520,150994944~12582912,167772160~4194304,180355072~8388608,192937984~16777216,213909504~20971520,276824064~8388608,293601280~12582912,310378496~8388608,322961408~16777216,348127232~12582912,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.093 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~50331648,54525952~12582912,79691776~20971520,109051904~20971520,150994944~12582912,167772160~4194304,180355072~8388608,192937984~16777216,213909504~20971520,276824064~8388608,293601280~12582912,310378496~8388608,322961408~16777216,348127232~12582912,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.095 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~50331648,54525952~12582912,79691776~25165824,109051904~25165824,150994944~20971520,180355072~29360128,213909504~25165824,276824064~12582912,293601280~25165824,322961408~16777216,348127232~12582912,364904448~29360128,402653184~8388608] 2024-08-17T17:51:27.095 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 11 diff [0~50331648,54525952~16777216,79691776~20971520,109051904~20971520,150994944~12582912,167772160~4194304,180355072~29360128,213909504~20971520,247463936~8388608,272629760~12582912,293601280~25165824,322961408~16777216,348127232~12582912,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.095 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~50331648,54525952~16777216,79691776~20971520,109051904~20971520,150994944~12582912,167772160~4194304,180355072~29360128,213909504~20971520,247463936~8388608,272629760~12582912,293601280~25165824,322961408~16777216,348127232~12582912,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.099 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~50331648,54525952~16777216,79691776~25165824,109051904~25165824,150994944~20971520,180355072~29360128,213909504~25165824,247463936~8388608,272629760~16777216,293601280~25165824,322961408~16777216,348127232~12582912,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.099 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 12 diff [0~46137344,54525952~20971520,79691776~20971520,109051904~20971520,150994944~20971520,180355072~29360128,213909504~25165824,247463936~8388608,272629760~12582912,289406976~29360128,322961408~16777216,348127232~12582912,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.099 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~46137344,54525952~20971520,79691776~20971520,109051904~20971520,150994944~20971520,180355072~29360128,213909504~25165824,247463936~8388608,272629760~12582912,289406976~29360128,322961408~16777216,348127232~12582912,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.102 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~50331648,54525952~20971520,79691776~25165824,109051904~25165824,150994944~20971520,180355072~29360128,213909504~29360128,247463936~8388608,272629760~46137344,322961408~16777216,348127232~12582912,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.102 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 13 diff [0~67108864,71303168~29360128,109051904~20971520,150994944~25165824,180355072~29360128,213909504~25165824,247463936~8388608,272629760~12582912,289406976~29360128,322961408~16777216,348127232~12582912,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.103 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~29360128,109051904~20971520,150994944~25165824,180355072~29360128,213909504~25165824,247463936~8388608,272629760~12582912,289406976~29360128,322961408~16777216,348127232~12582912,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.107 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~25165824,150994944~58720256,213909504~29360128,247463936~8388608,272629760~46137344,322961408~16777216,348127232~12582912,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.107 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 14 diff [0~67108864,71303168~29360128,109051904~20971520,150994944~25165824,180355072~58720256,243269632~20971520,272629760~67108864,348127232~12582912,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.107 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~29360128,109051904~20971520,150994944~25165824,180355072~58720256,243269632~20971520,272629760~67108864,348127232~12582912,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.112 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~25165824,150994944~113246208,272629760~67108864,348127232~12582912,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.112 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 15 diff [0~67108864,71303168~29360128,109051904~29360128,146800640~29360128,180355072~58720256,243269632~20971520,272629760~67108864,348127232~12582912,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.112 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~29360128,109051904~29360128,146800640~29360128,180355072~58720256,243269632~20971520,272629760~67108864,348127232~12582912,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.117 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~33554432,146800640~121634816,272629760~71303168,348127232~12582912,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.117 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 16 diff [0~67108864,71303168~29360128,109051904~29360128,146800640~29360128,180355072~58720256,243269632~20971520,272629760~67108864,348127232~12582912,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.117 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~29360128,109051904~29360128,146800640~29360128,180355072~58720256,243269632~20971520,272629760~67108864,348127232~12582912,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.122 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~33554432,146800640~121634816,272629760~71303168,348127232~12582912,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.122 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 17 diff [0~67108864,71303168~29360128,109051904~29360128,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~12582912,348127232~12582912,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.125 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~29360128,109051904~29360128,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~12582912,348127232~12582912,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.127 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~33554432,146800640~197132288,348127232~12582912,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.128 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 18 diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,398458880~16777216] 2024-08-17T17:51:27.128 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,398458880~16777216] 2024-08-17T17:51:27.133 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~29360128,398458880~16777216] 2024-08-17T17:51:27.133 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to 19 diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.133 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.139 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:27.139 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 5 diff [12582912~8388608,33554432~4194304,54525952~8388608,83886080~12582912,180355072~4194304,230686720~4194304,310378496~8388608,352321536~8388608] 2024-08-17T17:51:27.139 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [12582912~8388608,33554432~4194304,54525952~8388608,83886080~12582912,180355072~4194304,230686720~4194304,310378496~8388608,352321536~8388608] 2024-08-17T17:51:27.140 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [12582912~8388608,33554432~4194304,54525952~8388608,83886080~16777216,180355072~4194304,230686720~4194304,310378496~8388608,352321536~8388608] 2024-08-17T17:51:27.140 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 6 diff [4194304~12582912,33554432~4194304,54525952~8388608,83886080~12582912,117440512~8388608,180355072~4194304,226492416~8388608,276824064~8388608,310378496~8388608,335544320~4194304,352321536~8388608,369098752~8388608,402653184~4194304] 2024-08-17T17:51:27.140 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4194304~12582912,33554432~4194304,54525952~8388608,83886080~12582912,117440512~8388608,180355072~4194304,226492416~8388608,276824064~8388608,310378496~8388608,335544320~4194304,352321536~8388608,369098752~8388608,402653184~4194304] 2024-08-17T17:51:27.141 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~16777216,33554432~4194304,54525952~8388608,83886080~16777216,117440512~12582912,180355072~4194304,226492416~12582912,276824064~12582912,310378496~8388608,335544320~4194304,352321536~8388608,369098752~12582912,402653184~4194304] 2024-08-17T17:51:27.141 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 7 diff [4194304~12582912,33554432~4194304,41943040~8388608,54525952~8388608,83886080~16777216,117440512~12582912,167772160~4194304,180355072~4194304,226492416~8388608,276824064~8388608,297795584~8388608,310378496~8388608,335544320~4194304,352321536~8388608,369098752~12582912,402653184~4194304] 2024-08-17T17:51:27.141 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4194304~12582912,33554432~4194304,41943040~8388608,54525952~8388608,83886080~16777216,117440512~12582912,167772160~4194304,180355072~4194304,226492416~8388608,276824064~8388608,297795584~8388608,310378496~8388608,335544320~4194304,352321536~8388608,369098752~12582912,402653184~4194304] 2024-08-17T17:51:27.143 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~16777216,33554432~4194304,41943040~8388608,54525952~8388608,83886080~20971520,117440512~16777216,167772160~4194304,180355072~4194304,226492416~12582912,276824064~12582912,297795584~20971520,335544320~4194304,352321536~8388608,369098752~12582912,402653184~4194304] 2024-08-17T17:51:27.143 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 8 diff [4194304~12582912,33554432~16777216,54525952~8388608,83886080~16777216,117440512~12582912,155189248~8388608,167772160~4194304,180355072~8388608,213909504~4194304,226492416~8388608,276824064~8388608,293601280~12582912,310378496~8388608,327155712~4194304,335544320~4194304,352321536~8388608,369098752~12582912,402653184~4194304] 2024-08-17T17:51:27.143 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4194304~12582912,33554432~16777216,54525952~8388608,83886080~16777216,117440512~12582912,155189248~8388608,167772160~4194304,180355072~8388608,213909504~4194304,226492416~8388608,276824064~8388608,293601280~12582912,310378496~8388608,327155712~4194304,335544320~4194304,352321536~8388608,369098752~12582912,402653184~4194304] 2024-08-17T17:51:27.145 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~16777216,33554432~16777216,54525952~8388608,83886080~20971520,117440512~16777216,155189248~16777216,180355072~12582912,213909504~8388608,226492416~12582912,276824064~12582912,293601280~25165824,327155712~4194304,335544320~4194304,352321536~8388608,369098752~12582912,402653184~4194304] 2024-08-17T17:51:27.145 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 9 diff [0~16777216,25165824~25165824,54525952~8388608,79691776~20971520,117440512~12582912,155189248~8388608,167772160~4194304,180355072~8388608,201326592~8388608,213909504~4194304,226492416~8388608,276824064~8388608,293601280~12582912,310378496~8388608,327155712~4194304,335544320~4194304,348127232~12582912,364904448~16777216,402653184~4194304] 2024-08-17T17:51:27.145 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~16777216,25165824~25165824,54525952~8388608,79691776~20971520,117440512~12582912,155189248~8388608,167772160~4194304,180355072~8388608,201326592~8388608,213909504~4194304,226492416~8388608,276824064~8388608,293601280~12582912,310378496~8388608,327155712~4194304,335544320~4194304,348127232~12582912,364904448~16777216,402653184~4194304] 2024-08-17T17:51:27.147 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~20971520,25165824~25165824,54525952~8388608,79691776~25165824,117440512~16777216,155189248~16777216,180355072~12582912,201326592~8388608,213909504~8388608,226492416~12582912,276824064~12582912,293601280~25165824,327155712~4194304,335544320~4194304,348127232~12582912,364904448~16777216,402653184~4194304] 2024-08-17T17:51:27.147 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 10 diff [0~50331648,54525952~12582912,79691776~20971520,117440512~12582912,155189248~8388608,167772160~4194304,180355072~8388608,201326592~8388608,213909504~4194304,226492416~8388608,276824064~8388608,293601280~12582912,310378496~8388608,322961408~16777216,348127232~12582912,364904448~25165824,402653184~4194304] 2024-08-17T17:51:27.147 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~50331648,54525952~12582912,79691776~20971520,117440512~12582912,155189248~8388608,167772160~4194304,180355072~8388608,201326592~8388608,213909504~4194304,226492416~8388608,276824064~8388608,293601280~12582912,310378496~8388608,322961408~16777216,348127232~12582912,364904448~25165824,402653184~4194304] 2024-08-17T17:51:27.150 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~50331648,54525952~12582912,79691776~25165824,117440512~16777216,155189248~16777216,180355072~12582912,201326592~8388608,213909504~8388608,226492416~12582912,276824064~12582912,293601280~25165824,322961408~16777216,348127232~12582912,364904448~29360128,402653184~4194304] 2024-08-17T17:51:27.150 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 11 diff [0~50331648,54525952~16777216,79691776~20971520,109051904~20971520,155189248~8388608,167772160~4194304,180355072~12582912,201326592~8388608,213909504~4194304,226492416~8388608,247463936~8388608,272629760~12582912,293601280~25165824,322961408~16777216,348127232~12582912,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.150 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~50331648,54525952~16777216,79691776~20971520,109051904~20971520,155189248~8388608,167772160~4194304,180355072~12582912,201326592~8388608,213909504~4194304,226492416~8388608,247463936~8388608,272629760~12582912,293601280~25165824,322961408~16777216,348127232~12582912,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.153 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~50331648,54525952~16777216,79691776~25165824,109051904~25165824,155189248~16777216,180355072~12582912,201326592~8388608,213909504~8388608,226492416~12582912,247463936~8388608,272629760~16777216,293601280~25165824,322961408~16777216,348127232~12582912,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.154 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 12 diff [0~46137344,54525952~20971520,79691776~20971520,109051904~20971520,155189248~16777216,180355072~16777216,201326592~8388608,213909504~4194304,226492416~12582912,247463936~8388608,272629760~12582912,289406976~29360128,322961408~16777216,348127232~12582912,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.154 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~46137344,54525952~20971520,79691776~20971520,109051904~20971520,155189248~16777216,180355072~16777216,201326592~8388608,213909504~4194304,226492416~12582912,247463936~8388608,272629760~12582912,289406976~29360128,322961408~16777216,348127232~12582912,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.157 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~50331648,54525952~20971520,79691776~25165824,109051904~25165824,155189248~16777216,180355072~29360128,213909504~8388608,226492416~16777216,247463936~8388608,272629760~46137344,322961408~16777216,348127232~12582912,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.157 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 13 diff [0~67108864,71303168~29360128,109051904~20971520,155189248~20971520,180355072~16777216,201326592~8388608,213909504~8388608,226492416~12582912,247463936~8388608,272629760~12582912,289406976~29360128,322961408~16777216,348127232~12582912,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.157 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~29360128,109051904~20971520,155189248~20971520,180355072~16777216,201326592~8388608,213909504~8388608,226492416~12582912,247463936~8388608,272629760~12582912,289406976~29360128,322961408~16777216,348127232~12582912,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.161 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~25165824,155189248~54525952,213909504~8388608,226492416~16777216,247463936~8388608,272629760~46137344,322961408~16777216,348127232~12582912,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.161 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 14 diff [0~67108864,71303168~29360128,109051904~20971520,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~20971520,272629760~67108864,348127232~12582912,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.161 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~29360128,109051904~20971520,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~20971520,272629760~67108864,348127232~12582912,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.165 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~25165824,155189248~67108864,226492416~37748736,272629760~67108864,348127232~12582912,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.166 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 15 diff [0~67108864,71303168~29360128,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~20971520,272629760~67108864,348127232~12582912,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.166 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~29360128,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~20971520,272629760~67108864,348127232~12582912,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.170 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~33554432,146800640~75497472,226492416~41943040,272629760~71303168,348127232~12582912,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.170 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 16 diff [0~67108864,71303168~29360128,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~20971520,272629760~67108864,348127232~12582912,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.170 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~29360128,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~20971520,272629760~67108864,348127232~12582912,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.175 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~33554432,146800640~75497472,226492416~41943040,272629760~71303168,348127232~12582912,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.175 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 17 diff [0~67108864,71303168~29360128,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~25165824,272629760~50331648,327155712~12582912,348127232~12582912,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.175 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~29360128,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~25165824,272629760~50331648,327155712~12582912,348127232~12582912,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.180 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~33554432,146800640~75497472,226492416~117440512,348127232~12582912,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.180 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 18 diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,398458880~16777216] 2024-08-17T17:51:27.180 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,398458880~16777216] 2024-08-17T17:51:27.185 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~75497472,226492416~117440512,348127232~12582912,364904448~29360128,398458880~16777216] 2024-08-17T17:51:27.185 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to 19 diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.185 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.191 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:27.191 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 6 diff [4194304~8388608,92274688~4194304,117440512~8388608,226492416~8388608,276824064~8388608,335544320~4194304,369098752~8388608,402653184~4194304] 2024-08-17T17:51:27.191 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4194304~8388608,92274688~4194304,117440512~8388608,226492416~8388608,276824064~8388608,335544320~4194304,369098752~8388608,402653184~4194304] 2024-08-17T17:51:27.192 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~12582912,92274688~4194304,117440512~12582912,226492416~12582912,276824064~12582912,335544320~4194304,369098752~12582912,402653184~4194304] 2024-08-17T17:51:27.192 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 7 diff [4194304~12582912,33554432~4194304,41943040~8388608,92274688~8388608,117440512~12582912,167772160~4194304,226492416~8388608,276824064~8388608,297795584~8388608,310378496~4194304,335544320~4194304,369098752~12582912,402653184~4194304] 2024-08-17T17:51:27.192 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4194304~12582912,33554432~4194304,41943040~8388608,92274688~8388608,117440512~12582912,167772160~4194304,226492416~8388608,276824064~8388608,297795584~8388608,310378496~4194304,335544320~4194304,369098752~12582912,402653184~4194304] 2024-08-17T17:51:27.193 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~16777216,33554432~4194304,41943040~8388608,92274688~12582912,117440512~16777216,167772160~4194304,226492416~12582912,276824064~12582912,297795584~16777216,335544320~4194304,369098752~12582912,402653184~4194304] 2024-08-17T17:51:27.193 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 8 diff [4194304~12582912,33554432~16777216,54525952~4194304,88080384~12582912,117440512~12582912,155189248~8388608,167772160~4194304,180355072~8388608,213909504~4194304,226492416~8388608,276824064~8388608,293601280~12582912,310378496~4194304,327155712~4194304,335544320~4194304,369098752~12582912,402653184~4194304] 2024-08-17T17:51:27.193 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [4194304~12582912,33554432~16777216,54525952~4194304,88080384~12582912,117440512~12582912,155189248~8388608,167772160~4194304,180355072~8388608,213909504~4194304,226492416~8388608,276824064~8388608,293601280~12582912,310378496~4194304,327155712~4194304,335544320~4194304,369098752~12582912,402653184~4194304] 2024-08-17T17:51:27.195 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [4194304~16777216,33554432~16777216,54525952~8388608,88080384~16777216,117440512~16777216,155189248~16777216,180355072~12582912,213909504~8388608,226492416~12582912,276824064~12582912,293601280~20971520,327155712~4194304,335544320~4194304,369098752~12582912,402653184~4194304] 2024-08-17T17:51:27.195 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 9 diff [0~16777216,25165824~25165824,54525952~4194304,79691776~4194304,88080384~12582912,117440512~12582912,155189248~8388608,167772160~4194304,180355072~8388608,201326592~8388608,213909504~4194304,226492416~8388608,276824064~8388608,293601280~12582912,310378496~4194304,327155712~4194304,335544320~4194304,348127232~8388608,364904448~16777216,402653184~4194304] 2024-08-17T17:51:27.195 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~16777216,25165824~25165824,54525952~4194304,79691776~4194304,88080384~12582912,117440512~12582912,155189248~8388608,167772160~4194304,180355072~8388608,201326592~8388608,213909504~4194304,226492416~8388608,276824064~8388608,293601280~12582912,310378496~4194304,327155712~4194304,335544320~4194304,348127232~8388608,364904448~16777216,402653184~4194304] 2024-08-17T17:51:27.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:26 smithi136 ceph-mon[109861]: osdmap e1275: 8 total, 8 up, 8 in 2024-08-17T17:51:27.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:26 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:51:27.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:26 smithi136 ceph-mon[109861]: pgmap v1993: 105 pgs: 105 active+clean; 1.1 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 129 KiB/s rd, 24 MiB/s wr, 185 op/s 2024-08-17T17:51:27.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:26 smithi136 ceph-mon[115078]: osdmap e1275: 8 total, 8 up, 8 in 2024-08-17T17:51:27.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:26 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:51:27.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:26 smithi136 ceph-mon[115078]: pgmap v1993: 105 pgs: 105 active+clean; 1.1 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 129 KiB/s rd, 24 MiB/s wr, 185 op/s 2024-08-17T17:51:27.197 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~20971520,25165824~25165824,54525952~8388608,79691776~25165824,117440512~16777216,155189248~16777216,180355072~12582912,201326592~8388608,213909504~8388608,226492416~12582912,276824064~12582912,293601280~20971520,327155712~4194304,335544320~4194304,348127232~8388608,364904448~16777216,402653184~4194304] 2024-08-17T17:51:27.197 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 10 diff [0~50331648,54525952~12582912,79691776~4194304,88080384~12582912,117440512~12582912,155189248~8388608,167772160~4194304,180355072~8388608,201326592~8388608,213909504~4194304,226492416~8388608,276824064~8388608,293601280~12582912,310378496~4194304,322961408~16777216,348127232~8388608,364904448~25165824,402653184~4194304] 2024-08-17T17:51:27.197 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~50331648,54525952~12582912,79691776~4194304,88080384~12582912,117440512~12582912,155189248~8388608,167772160~4194304,180355072~8388608,201326592~8388608,213909504~4194304,226492416~8388608,276824064~8388608,293601280~12582912,310378496~4194304,322961408~16777216,348127232~8388608,364904448~25165824,402653184~4194304] 2024-08-17T17:51:27.199 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~50331648,54525952~12582912,79691776~25165824,117440512~16777216,155189248~16777216,180355072~12582912,201326592~8388608,213909504~8388608,226492416~12582912,276824064~12582912,293601280~20971520,322961408~16777216,348127232~8388608,364904448~29360128,402653184~4194304] 2024-08-17T17:51:27.199 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 11 diff [0~50331648,54525952~16777216,79691776~4194304,88080384~12582912,109051904~20971520,155189248~8388608,167772160~4194304,180355072~12582912,201326592~8388608,213909504~4194304,226492416~8388608,247463936~8388608,272629760~12582912,293601280~20971520,322961408~16777216,348127232~8388608,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.199 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~50331648,54525952~16777216,79691776~4194304,88080384~12582912,109051904~20971520,155189248~8388608,167772160~4194304,180355072~12582912,201326592~8388608,213909504~4194304,226492416~8388608,247463936~8388608,272629760~12582912,293601280~20971520,322961408~16777216,348127232~8388608,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.202 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~50331648,54525952~16777216,79691776~25165824,109051904~25165824,155189248~16777216,180355072~12582912,201326592~8388608,213909504~8388608,226492416~12582912,247463936~8388608,272629760~16777216,293601280~20971520,322961408~16777216,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.202 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 12 diff [0~46137344,54525952~20971520,79691776~4194304,88080384~12582912,109051904~20971520,155189248~16777216,180355072~16777216,201326592~8388608,213909504~4194304,226492416~12582912,247463936~8388608,272629760~12582912,289406976~25165824,322961408~16777216,348127232~8388608,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.202 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~46137344,54525952~20971520,79691776~4194304,88080384~12582912,109051904~20971520,155189248~16777216,180355072~16777216,201326592~8388608,213909504~4194304,226492416~12582912,247463936~8388608,272629760~12582912,289406976~25165824,322961408~16777216,348127232~8388608,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.205 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~50331648,54525952~20971520,79691776~25165824,109051904~25165824,155189248~16777216,180355072~29360128,213909504~8388608,226492416~16777216,247463936~8388608,272629760~41943040,322961408~16777216,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.206 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 13 diff [0~67108864,71303168~29360128,109051904~20971520,155189248~20971520,180355072~16777216,201326592~8388608,213909504~8388608,226492416~12582912,247463936~8388608,272629760~12582912,289406976~25165824,322961408~16777216,348127232~8388608,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.206 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~29360128,109051904~20971520,155189248~20971520,180355072~16777216,201326592~8388608,213909504~8388608,226492416~12582912,247463936~8388608,272629760~12582912,289406976~25165824,322961408~16777216,348127232~8388608,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.209 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~25165824,155189248~54525952,213909504~8388608,226492416~16777216,247463936~8388608,272629760~41943040,322961408~16777216,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.209 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 14 diff [0~67108864,71303168~29360128,109051904~20971520,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~20971520,272629760~46137344,322961408~16777216,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.209 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~29360128,109051904~20971520,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~20971520,272629760~46137344,322961408~16777216,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.213 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~25165824,155189248~67108864,226492416~37748736,272629760~67108864,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.213 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 15 diff [0~67108864,71303168~29360128,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~20971520,272629760~46137344,322961408~16777216,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.213 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~29360128,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~20971520,272629760~46137344,322961408~16777216,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.218 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~33554432,146800640~75497472,226492416~41943040,272629760~71303168,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.218 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 16 diff [0~67108864,71303168~29360128,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~20971520,272629760~46137344,322961408~16777216,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.218 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~29360128,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~20971520,272629760~46137344,322961408~16777216,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.222 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~33554432,146800640~75497472,226492416~41943040,272629760~71303168,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.223 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 17 diff [0~67108864,71303168~29360128,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~25165824,272629760~50331648,327155712~12582912,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.223 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~29360128,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~25165824,272629760~50331648,327155712~12582912,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.228 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~33554432,146800640~75497472,226492416~117440512,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.228 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 18 diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,398458880~16777216] 2024-08-17T17:51:27.228 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,398458880~16777216] 2024-08-17T17:51:27.233 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~75497472,226492416~117440512,348127232~12582912,364904448~29360128,398458880~16777216] 2024-08-17T17:51:27.233 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to 19 diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.234 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.238 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:27.238 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 7 diff [12582912~4194304,33554432~4194304,41943040~8388608,96468992~4194304,121634816~8388608,167772160~4194304,297795584~8388608,310378496~4194304,373293056~8388608] 2024-08-17T17:51:27.239 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [12582912~4194304,33554432~4194304,41943040~8388608,96468992~4194304,121634816~8388608,167772160~4194304,297795584~8388608,310378496~4194304,373293056~8388608] 2024-08-17T17:51:27.239 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [12582912~8388608,33554432~4194304,41943040~8388608,96468992~8388608,121634816~12582912,167772160~4194304,297795584~16777216,373293056~8388608] 2024-08-17T17:51:27.239 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 8 diff [8388608~8388608,33554432~16777216,54525952~4194304,88080384~12582912,121634816~8388608,155189248~8388608,167772160~4194304,180355072~8388608,213909504~4194304,293601280~12582912,310378496~4194304,327155712~4194304,373293056~8388608] 2024-08-17T17:51:27.240 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [8388608~8388608,33554432~16777216,54525952~4194304,88080384~12582912,121634816~8388608,155189248~8388608,167772160~4194304,180355072~8388608,213909504~4194304,293601280~12582912,310378496~4194304,327155712~4194304,373293056~8388608] 2024-08-17T17:51:27.241 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [8388608~12582912,33554432~16777216,54525952~8388608,88080384~16777216,121634816~12582912,155189248~16777216,180355072~12582912,213909504~8388608,293601280~20971520,327155712~4194304,373293056~8388608] 2024-08-17T17:51:27.241 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 9 diff [0~16777216,25165824~25165824,54525952~4194304,79691776~4194304,88080384~12582912,121634816~8388608,155189248~8388608,167772160~4194304,180355072~8388608,201326592~8388608,213909504~4194304,293601280~12582912,310378496~4194304,327155712~4194304,348127232~8388608,364904448~16777216,402653184~4194304] 2024-08-17T17:51:27.241 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~16777216,25165824~25165824,54525952~4194304,79691776~4194304,88080384~12582912,121634816~8388608,155189248~8388608,167772160~4194304,180355072~8388608,201326592~8388608,213909504~4194304,293601280~12582912,310378496~4194304,327155712~4194304,348127232~8388608,364904448~16777216,402653184~4194304] 2024-08-17T17:51:27.242 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~20971520,25165824~25165824,54525952~8388608,79691776~25165824,121634816~12582912,155189248~16777216,180355072~12582912,201326592~8388608,213909504~8388608,293601280~20971520,327155712~4194304,348127232~8388608,364904448~16777216,402653184~4194304] 2024-08-17T17:51:27.243 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 10 diff [0~50331648,54525952~12582912,79691776~4194304,88080384~12582912,121634816~8388608,155189248~8388608,167772160~4194304,180355072~8388608,201326592~8388608,213909504~4194304,293601280~12582912,310378496~4194304,322961408~16777216,348127232~8388608,364904448~25165824,402653184~4194304] 2024-08-17T17:51:27.243 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~50331648,54525952~12582912,79691776~4194304,88080384~12582912,121634816~8388608,155189248~8388608,167772160~4194304,180355072~8388608,201326592~8388608,213909504~4194304,293601280~12582912,310378496~4194304,322961408~16777216,348127232~8388608,364904448~25165824,402653184~4194304] 2024-08-17T17:51:27.245 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~50331648,54525952~12582912,79691776~25165824,121634816~12582912,155189248~16777216,180355072~12582912,201326592~8388608,213909504~8388608,293601280~20971520,322961408~16777216,348127232~8388608,364904448~29360128,402653184~4194304] 2024-08-17T17:51:27.245 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 11 diff [0~50331648,54525952~16777216,79691776~4194304,88080384~12582912,109051904~8388608,121634816~8388608,155189248~8388608,167772160~4194304,180355072~12582912,201326592~8388608,213909504~4194304,230686720~4194304,247463936~8388608,272629760~4194304,293601280~20971520,322961408~16777216,348127232~8388608,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.245 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~50331648,54525952~16777216,79691776~4194304,88080384~12582912,109051904~8388608,121634816~8388608,155189248~8388608,167772160~4194304,180355072~12582912,201326592~8388608,213909504~4194304,230686720~4194304,247463936~8388608,272629760~4194304,293601280~20971520,322961408~16777216,348127232~8388608,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.247 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~50331648,54525952~16777216,79691776~25165824,109051904~25165824,155189248~16777216,180355072~12582912,201326592~8388608,213909504~8388608,230686720~4194304,247463936~8388608,272629760~8388608,293601280~20971520,322961408~16777216,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.247 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 12 diff [0~46137344,54525952~20971520,79691776~4194304,88080384~12582912,109051904~20971520,155189248~16777216,180355072~16777216,201326592~8388608,213909504~4194304,230686720~8388608,247463936~8388608,272629760~4194304,289406976~25165824,322961408~16777216,348127232~8388608,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.247 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~46137344,54525952~20971520,79691776~4194304,88080384~12582912,109051904~20971520,155189248~16777216,180355072~16777216,201326592~8388608,213909504~4194304,230686720~8388608,247463936~8388608,272629760~4194304,289406976~25165824,322961408~16777216,348127232~8388608,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.250 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~50331648,54525952~20971520,79691776~25165824,109051904~25165824,155189248~16777216,180355072~29360128,213909504~8388608,230686720~12582912,247463936~8388608,272629760~8388608,289406976~25165824,322961408~16777216,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.250 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 13 diff [0~67108864,71303168~29360128,109051904~20971520,155189248~20971520,180355072~16777216,201326592~8388608,213909504~8388608,230686720~8388608,247463936~8388608,272629760~4194304,289406976~25165824,322961408~16777216,348127232~8388608,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.250 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~29360128,109051904~20971520,155189248~20971520,180355072~16777216,201326592~8388608,213909504~8388608,230686720~8388608,247463936~8388608,272629760~4194304,289406976~25165824,322961408~16777216,348127232~8388608,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.253 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~25165824,155189248~54525952,213909504~8388608,230686720~12582912,247463936~8388608,272629760~8388608,289406976~25165824,322961408~16777216,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.253 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 14 diff [0~67108864,71303168~29360128,109051904~20971520,155189248~20971520,180355072~16777216,201326592~20971520,230686720~8388608,243269632~20971520,272629760~4194304,281018368~37748736,322961408~16777216,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.254 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~29360128,109051904~20971520,155189248~20971520,180355072~16777216,201326592~20971520,230686720~8388608,243269632~20971520,272629760~4194304,281018368~37748736,322961408~16777216,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.257 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~25165824,155189248~67108864,230686720~33554432,272629760~67108864,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.257 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 15 diff [0~67108864,71303168~29360128,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,230686720~8388608,243269632~20971520,272629760~4194304,281018368~37748736,322961408~16777216,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.257 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~29360128,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,230686720~8388608,243269632~20971520,272629760~4194304,281018368~37748736,322961408~16777216,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:26 smithi165 ceph-mon[105517]: osdmap e1275: 8 total, 8 up, 8 in 2024-08-17T17:51:27.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:26 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:51:27.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:26 smithi165 ceph-mon[105517]: pgmap v1993: 105 pgs: 105 active+clean; 1.1 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 129 KiB/s rd, 24 MiB/s wr, 185 op/s 2024-08-17T17:51:27.261 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~33554432,146800640~75497472,230686720~37748736,272629760~71303168,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.261 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 16 diff [0~67108864,71303168~29360128,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~20971520,272629760~46137344,322961408~16777216,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.261 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~29360128,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~20971520,272629760~46137344,322961408~16777216,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.266 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~33554432,146800640~75497472,226492416~41943040,272629760~71303168,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.266 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 17 diff [0~67108864,71303168~29360128,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~25165824,272629760~50331648,327155712~12582912,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.266 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~29360128,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~25165824,272629760~50331648,327155712~12582912,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.271 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~104857600,109051904~33554432,146800640~75497472,226492416~117440512,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.271 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 18 diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,398458880~16777216] 2024-08-17T17:51:27.271 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,398458880~16777216] 2024-08-17T17:51:27.276 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~75497472,226492416~117440512,348127232~12582912,364904448~29360128,398458880~16777216] 2024-08-17T17:51:27.276 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to 19 diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.277 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.283 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:27.283 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to 8 diff [8388608~8388608,37748736~4194304,54525952~4194304,88080384~8388608,155189248~8388608,180355072~8388608,213909504~4194304,293601280~4194304,310378496~4194304,327155712~4194304] 2024-08-17T17:51:27.283 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [8388608~8388608,37748736~4194304,54525952~4194304,88080384~8388608,155189248~8388608,180355072~8388608,213909504~4194304,293601280~4194304,310378496~4194304,327155712~4194304] 2024-08-17T17:51:27.284 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [8388608~8388608,37748736~4194304,54525952~8388608,88080384~12582912,155189248~12582912,180355072~12582912,213909504~8388608,293601280~8388608,310378496~4194304,327155712~4194304] 2024-08-17T17:51:27.284 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to 9 diff [0~12582912,25165824~8388608,37748736~4194304,54525952~4194304,79691776~4194304,88080384~8388608,155189248~8388608,180355072~8388608,201326592~8388608,213909504~4194304,293601280~4194304,310378496~4194304,327155712~4194304,348127232~8388608,364904448~8388608,402653184~4194304] 2024-08-17T17:51:27.284 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~12582912,25165824~8388608,37748736~4194304,54525952~4194304,79691776~4194304,88080384~8388608,155189248~8388608,180355072~8388608,201326592~8388608,213909504~4194304,293601280~4194304,310378496~4194304,327155712~4194304,348127232~8388608,364904448~8388608,402653184~4194304] 2024-08-17T17:51:27.286 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~16777216,25165824~16777216,54525952~8388608,79691776~20971520,155189248~12582912,180355072~12582912,201326592~8388608,213909504~8388608,293601280~8388608,310378496~4194304,327155712~4194304,348127232~8388608,364904448~8388608,402653184~4194304] 2024-08-17T17:51:27.286 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to 10 diff [0~33554432,37748736~4194304,54525952~12582912,79691776~4194304,88080384~8388608,155189248~8388608,180355072~8388608,201326592~8388608,213909504~4194304,293601280~8388608,310378496~4194304,322961408~16777216,348127232~8388608,364904448~25165824,402653184~4194304] 2024-08-17T17:51:27.286 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~33554432,37748736~4194304,54525952~12582912,79691776~4194304,88080384~8388608,155189248~8388608,180355072~8388608,201326592~8388608,213909504~4194304,293601280~8388608,310378496~4194304,322961408~16777216,348127232~8388608,364904448~25165824,402653184~4194304] 2024-08-17T17:51:27.288 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~41943040,54525952~12582912,79691776~20971520,155189248~12582912,180355072~12582912,201326592~8388608,213909504~8388608,293601280~8388608,310378496~4194304,322961408~16777216,348127232~8388608,364904448~29360128,402653184~4194304] 2024-08-17T17:51:27.288 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to 11 diff [0~33554432,37748736~4194304,54525952~16777216,79691776~4194304,88080384~8388608,109051904~8388608,155189248~8388608,180355072~12582912,201326592~8388608,213909504~4194304,230686720~4194304,247463936~8388608,272629760~4194304,293601280~8388608,306184192~8388608,322961408~16777216,348127232~8388608,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.288 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~33554432,37748736~4194304,54525952~16777216,79691776~4194304,88080384~8388608,109051904~8388608,155189248~8388608,180355072~12582912,201326592~8388608,213909504~4194304,230686720~4194304,247463936~8388608,272629760~4194304,293601280~8388608,306184192~8388608,322961408~16777216,348127232~8388608,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.290 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~41943040,54525952~16777216,79691776~20971520,109051904~12582912,155189248~12582912,180355072~12582912,201326592~8388608,213909504~8388608,230686720~4194304,247463936~8388608,272629760~8388608,293601280~8388608,306184192~8388608,322961408~16777216,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.290 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to 12 diff [0~33554432,37748736~4194304,54525952~20971520,79691776~4194304,88080384~8388608,109051904~16777216,155189248~12582912,180355072~16777216,201326592~8388608,213909504~4194304,230686720~8388608,247463936~8388608,272629760~4194304,289406976~12582912,306184192~8388608,322961408~16777216,348127232~8388608,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.290 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~33554432,37748736~4194304,54525952~20971520,79691776~4194304,88080384~8388608,109051904~16777216,155189248~12582912,180355072~16777216,201326592~8388608,213909504~4194304,230686720~8388608,247463936~8388608,272629760~4194304,289406976~12582912,306184192~8388608,322961408~16777216,348127232~8388608,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.293 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~46137344,54525952~20971520,79691776~20971520,109051904~20971520,155189248~16777216,180355072~29360128,213909504~8388608,230686720~12582912,247463936~8388608,272629760~8388608,289406976~12582912,306184192~8388608,322961408~16777216,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.293 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to 13 diff [0~41943040,46137344~20971520,71303168~25165824,109051904~16777216,155189248~20971520,180355072~16777216,201326592~8388608,213909504~8388608,230686720~8388608,247463936~8388608,272629760~4194304,289406976~12582912,306184192~8388608,322961408~16777216,348127232~8388608,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.293 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~41943040,46137344~20971520,71303168~25165824,109051904~16777216,155189248~20971520,180355072~16777216,201326592~8388608,213909504~8388608,230686720~8388608,247463936~8388608,272629760~4194304,289406976~12582912,306184192~8388608,322961408~16777216,348127232~8388608,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.296 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~100663296,109051904~20971520,155189248~54525952,213909504~8388608,230686720~12582912,247463936~8388608,272629760~8388608,289406976~12582912,306184192~8388608,322961408~16777216,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.296 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to 14 diff [0~41943040,46137344~20971520,71303168~25165824,109051904~16777216,155189248~20971520,180355072~16777216,201326592~20971520,230686720~8388608,243269632~20971520,272629760~4194304,281018368~20971520,306184192~12582912,322961408~16777216,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.296 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~41943040,46137344~20971520,71303168~25165824,109051904~16777216,155189248~20971520,180355072~16777216,201326592~20971520,230686720~8388608,243269632~20971520,272629760~4194304,281018368~20971520,306184192~12582912,322961408~16777216,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.299 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~100663296,109051904~20971520,155189248~67108864,230686720~33554432,272629760~29360128,306184192~33554432,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.299 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to 15 diff [0~67108864,71303168~25165824,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,230686720~8388608,243269632~20971520,272629760~4194304,281018368~37748736,322961408~16777216,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.299 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,230686720~8388608,243269632~20971520,272629760~4194304,281018368~37748736,322961408~16777216,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.302 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~100663296,109051904~33554432,146800640~75497472,230686720~37748736,272629760~71303168,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.303 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to 16 diff [0~67108864,71303168~25165824,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~20971520,272629760~46137344,322961408~16777216,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.303 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~20971520,272629760~46137344,322961408~16777216,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.306 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~100663296,109051904~33554432,146800640~75497472,226492416~41943040,272629760~71303168,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.306 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to 17 diff [0~67108864,71303168~25165824,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~25165824,272629760~50331648,327155712~12582912,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.306 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~25165824,272629760~50331648,327155712~12582912,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.310 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~100663296,109051904~33554432,146800640~75497472,226492416~117440512,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.310 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to 18 diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,398458880~16777216] 2024-08-17T17:51:27.310 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,398458880~16777216] 2024-08-17T17:51:27.315 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~75497472,226492416~117440512,348127232~12582912,364904448~29360128,398458880~16777216] 2024-08-17T17:51:27.315 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to 19 diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.315 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.320 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:27.321 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to 9 diff [0~8388608,25165824~8388608,79691776~4194304,180355072~8388608,201326592~8388608,348127232~8388608,364904448~8388608,402653184~4194304] 2024-08-17T17:51:27.321 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~8388608,25165824~8388608,79691776~4194304,180355072~8388608,201326592~8388608,348127232~8388608,364904448~8388608,402653184~4194304] 2024-08-17T17:51:27.322 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~12582912,25165824~12582912,79691776~8388608,180355072~8388608,201326592~8388608,348127232~8388608,364904448~8388608,402653184~4194304] 2024-08-17T17:51:27.322 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to 10 diff [0~33554432,54525952~8388608,79691776~4194304,180355072~8388608,201326592~8388608,293601280~8388608,322961408~4194304,331350016~8388608,348127232~8388608,364904448~25165824,402653184~4194304] 2024-08-17T17:51:27.322 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~33554432,54525952~8388608,79691776~4194304,180355072~8388608,201326592~8388608,293601280~8388608,322961408~4194304,331350016~8388608,348127232~8388608,364904448~25165824,402653184~4194304] 2024-08-17T17:51:27.323 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~37748736,54525952~12582912,79691776~8388608,180355072~8388608,201326592~8388608,293601280~8388608,322961408~4194304,331350016~8388608,348127232~8388608,364904448~29360128,402653184~4194304] 2024-08-17T17:51:27.323 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to 11 diff [0~33554432,54525952~12582912,79691776~4194304,109051904~8388608,180355072~8388608,201326592~8388608,230686720~4194304,247463936~8388608,272629760~4194304,293601280~8388608,306184192~8388608,322961408~4194304,331350016~8388608,348127232~8388608,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.323 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~33554432,54525952~12582912,79691776~4194304,109051904~8388608,180355072~8388608,201326592~8388608,230686720~4194304,247463936~8388608,272629760~4194304,293601280~8388608,306184192~8388608,322961408~4194304,331350016~8388608,348127232~8388608,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.325 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~37748736,54525952~16777216,79691776~8388608,109051904~12582912,180355072~8388608,201326592~8388608,230686720~4194304,247463936~8388608,272629760~8388608,293601280~8388608,306184192~8388608,322961408~4194304,331350016~8388608,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.325 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to 12 diff [0~33554432,37748736~4194304,54525952~12582912,71303168~4194304,79691776~4194304,109051904~16777216,159383552~8388608,180355072~16777216,201326592~8388608,230686720~8388608,247463936~8388608,272629760~4194304,289406976~12582912,306184192~8388608,322961408~4194304,331350016~8388608,348127232~8388608,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.325 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~33554432,37748736~4194304,54525952~12582912,71303168~4194304,79691776~4194304,109051904~16777216,159383552~8388608,180355072~16777216,201326592~8388608,230686720~8388608,247463936~8388608,272629760~4194304,289406976~12582912,306184192~8388608,322961408~4194304,331350016~8388608,348127232~8388608,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.327 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~46137344,54525952~20971520,79691776~8388608,109051904~20971520,159383552~12582912,180355072~29360128,230686720~12582912,247463936~8388608,272629760~8388608,289406976~12582912,306184192~8388608,322961408~4194304,331350016~8388608,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.327 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to 13 diff [0~41943040,46137344~20971520,71303168~16777216,109051904~16777216,159383552~16777216,180355072~16777216,201326592~8388608,213909504~8388608,230686720~8388608,247463936~8388608,272629760~4194304,289406976~12582912,306184192~8388608,322961408~4194304,331350016~8388608,348127232~8388608,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.327 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~41943040,46137344~20971520,71303168~16777216,109051904~16777216,159383552~16777216,180355072~16777216,201326592~8388608,213909504~8388608,230686720~8388608,247463936~8388608,272629760~4194304,289406976~12582912,306184192~8388608,322961408~4194304,331350016~8388608,348127232~8388608,364904448~25165824,402653184~8388608] 2024-08-17T17:51:27.329 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~88080384,109051904~20971520,159383552~50331648,213909504~8388608,230686720~12582912,247463936~8388608,272629760~8388608,289406976~12582912,306184192~8388608,322961408~4194304,331350016~8388608,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.329 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to 14 diff [0~41943040,46137344~20971520,71303168~16777216,92274688~4194304,109051904~16777216,155189248~20971520,180355072~16777216,201326592~20971520,230686720~8388608,243269632~20971520,272629760~4194304,281018368~20971520,306184192~12582912,322961408~4194304,331350016~8388608,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.329 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~41943040,46137344~20971520,71303168~16777216,92274688~4194304,109051904~16777216,155189248~20971520,180355072~16777216,201326592~20971520,230686720~8388608,243269632~20971520,272629760~4194304,281018368~20971520,306184192~12582912,322961408~4194304,331350016~8388608,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.332 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~88080384,92274688~4194304,109051904~20971520,155189248~67108864,230686720~33554432,272629760~29360128,306184192~20971520,331350016~8388608,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.332 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to 15 diff [0~67108864,71303168~16777216,92274688~4194304,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,230686720~8388608,243269632~20971520,272629760~4194304,281018368~37748736,322961408~4194304,331350016~8388608,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.332 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~16777216,92274688~4194304,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,230686720~8388608,243269632~20971520,272629760~4194304,281018368~37748736,322961408~4194304,331350016~8388608,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.334 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~88080384,92274688~4194304,109051904~33554432,146800640~75497472,230686720~37748736,272629760~54525952,331350016~12582912,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.334 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to 16 diff [0~67108864,71303168~16777216,92274688~4194304,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~20971520,272629760~46137344,322961408~16777216,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.334 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~16777216,92274688~4194304,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~20971520,272629760~46137344,322961408~16777216,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.337 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~88080384,92274688~4194304,109051904~33554432,146800640~75497472,226492416~41943040,272629760~71303168,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.337 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to 17 diff [0~67108864,71303168~25165824,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~25165824,272629760~50331648,327155712~12582912,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.337 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~25165824,272629760~50331648,327155712~12582912,348127232~8388608,364904448~25165824,402653184~12582912] 2024-08-17T17:51:27.341 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~100663296,109051904~33554432,146800640~75497472,226492416~117440512,348127232~8388608,364904448~29360128,402653184~12582912] 2024-08-17T17:51:27.341 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to 18 diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,398458880~16777216] 2024-08-17T17:51:27.341 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~16777216,201326592~20971520,226492416~12582912,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,398458880~16777216] 2024-08-17T17:51:27.345 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~75497472,226492416~117440512,348127232~12582912,364904448~29360128,398458880~16777216] 2024-08-17T17:51:27.345 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to 19 diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.345 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.349 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:27.349 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to 10 diff [0~25165824,54525952~8388608,293601280~8388608,322961408~4194304,331350016~8388608,348127232~4194304,373293056~16777216] 2024-08-17T17:51:27.349 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~25165824,54525952~8388608,293601280~8388608,322961408~4194304,331350016~8388608,348127232~4194304,373293056~16777216] 2024-08-17T17:51:27.350 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~25165824,54525952~12582912,293601280~8388608,322961408~4194304,331350016~8388608,348127232~8388608,373293056~20971520] 2024-08-17T17:51:27.350 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to 11 diff [0~29360128,54525952~12582912,109051904~8388608,180355072~8388608,230686720~4194304,247463936~8388608,272629760~4194304,293601280~8388608,306184192~8388608,322961408~4194304,331350016~8388608,348127232~4194304,364904448~4194304,373293056~16777216,402653184~8388608] 2024-08-17T17:51:27.350 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~29360128,54525952~12582912,109051904~8388608,180355072~8388608,230686720~4194304,247463936~8388608,272629760~4194304,293601280~8388608,306184192~8388608,322961408~4194304,331350016~8388608,348127232~4194304,364904448~4194304,373293056~16777216,402653184~8388608] 2024-08-17T17:51:27.351 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~29360128,54525952~16777216,109051904~12582912,180355072~8388608,230686720~4194304,247463936~8388608,272629760~8388608,293601280~8388608,306184192~8388608,322961408~4194304,331350016~8388608,348127232~8388608,364904448~4194304,373293056~20971520,402653184~12582912] 2024-08-17T17:51:27.351 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to 12 diff [0~29360128,37748736~4194304,54525952~12582912,71303168~4194304,109051904~16777216,159383552~8388608,180355072~16777216,230686720~8388608,247463936~8388608,272629760~4194304,289406976~12582912,306184192~8388608,322961408~4194304,331350016~8388608,348127232~4194304,364904448~4194304,373293056~16777216,402653184~8388608] 2024-08-17T17:51:27.352 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~29360128,37748736~4194304,54525952~12582912,71303168~4194304,109051904~16777216,159383552~8388608,180355072~16777216,230686720~8388608,247463936~8388608,272629760~4194304,289406976~12582912,306184192~8388608,322961408~4194304,331350016~8388608,348127232~4194304,364904448~4194304,373293056~16777216,402653184~8388608] 2024-08-17T17:51:27.353 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~33554432,37748736~8388608,54525952~20971520,109051904~20971520,159383552~12582912,180355072~20971520,230686720~12582912,247463936~8388608,272629760~8388608,289406976~12582912,306184192~8388608,322961408~4194304,331350016~8388608,348127232~8388608,364904448~4194304,373293056~20971520,402653184~12582912] 2024-08-17T17:51:27.353 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to 13 diff [0~29360128,33554432~8388608,46137344~20971520,71303168~16777216,109051904~16777216,159383552~16777216,180355072~16777216,213909504~8388608,230686720~8388608,247463936~8388608,272629760~4194304,289406976~12582912,306184192~8388608,322961408~4194304,331350016~8388608,348127232~4194304,364904448~4194304,373293056~16777216,402653184~8388608] 2024-08-17T17:51:27.353 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~29360128,33554432~8388608,46137344~20971520,71303168~16777216,109051904~16777216,159383552~16777216,180355072~16777216,213909504~8388608,230686720~8388608,247463936~8388608,272629760~4194304,289406976~12582912,306184192~8388608,322961408~4194304,331350016~8388608,348127232~4194304,364904448~4194304,373293056~16777216,402653184~8388608] 2024-08-17T17:51:27.355 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~88080384,109051904~20971520,159383552~41943040,213909504~8388608,230686720~12582912,247463936~8388608,272629760~8388608,289406976~12582912,306184192~8388608,322961408~4194304,331350016~8388608,348127232~8388608,364904448~4194304,373293056~20971520,402653184~12582912] 2024-08-17T17:51:27.355 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to 14 diff [0~29360128,33554432~8388608,46137344~20971520,71303168~16777216,92274688~4194304,109051904~16777216,155189248~20971520,180355072~16777216,209715200~12582912,230686720~8388608,243269632~20971520,272629760~4194304,281018368~20971520,306184192~12582912,322961408~4194304,331350016~8388608,348127232~4194304,364904448~4194304,373293056~16777216,402653184~12582912] 2024-08-17T17:51:27.356 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~29360128,33554432~8388608,46137344~20971520,71303168~16777216,92274688~4194304,109051904~16777216,155189248~20971520,180355072~16777216,209715200~12582912,230686720~8388608,243269632~20971520,272629760~4194304,281018368~20971520,306184192~12582912,322961408~4194304,331350016~8388608,348127232~4194304,364904448~4194304,373293056~16777216,402653184~12582912] 2024-08-17T17:51:27.358 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~88080384,92274688~4194304,109051904~20971520,155189248~46137344,209715200~12582912,230686720~33554432,272629760~29360128,306184192~20971520,331350016~8388608,348127232~8388608,364904448~4194304,373293056~20971520,402653184~12582912] 2024-08-17T17:51:27.358 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to 15 diff [0~29360128,33554432~33554432,71303168~16777216,92274688~4194304,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,209715200~12582912,230686720~8388608,243269632~20971520,272629760~4194304,281018368~37748736,322961408~4194304,331350016~8388608,348127232~4194304,364904448~4194304,373293056~16777216,402653184~12582912] 2024-08-17T17:51:27.358 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~29360128,33554432~33554432,71303168~16777216,92274688~4194304,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,209715200~12582912,230686720~8388608,243269632~20971520,272629760~4194304,281018368~37748736,322961408~4194304,331350016~8388608,348127232~4194304,364904448~4194304,373293056~16777216,402653184~12582912] 2024-08-17T17:51:27.360 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~88080384,92274688~4194304,109051904~33554432,146800640~54525952,209715200~12582912,230686720~37748736,272629760~54525952,331350016~12582912,348127232~8388608,364904448~4194304,373293056~20971520,402653184~12582912] 2024-08-17T17:51:27.360 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to 16 diff [0~29360128,33554432~33554432,71303168~16777216,92274688~4194304,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,209715200~12582912,226492416~12582912,243269632~20971520,272629760~46137344,322961408~16777216,348127232~4194304,364904448~4194304,373293056~16777216,402653184~12582912] 2024-08-17T17:51:27.360 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~29360128,33554432~33554432,71303168~16777216,92274688~4194304,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,209715200~12582912,226492416~12582912,243269632~20971520,272629760~46137344,322961408~16777216,348127232~4194304,364904448~4194304,373293056~16777216,402653184~12582912] 2024-08-17T17:51:27.363 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~88080384,92274688~4194304,109051904~33554432,146800640~54525952,209715200~12582912,226492416~41943040,272629760~71303168,348127232~8388608,364904448~4194304,373293056~20971520,402653184~12582912] 2024-08-17T17:51:27.363 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to 17 diff [0~29360128,33554432~33554432,71303168~25165824,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,209715200~12582912,226492416~12582912,243269632~25165824,272629760~50331648,327155712~12582912,348127232~4194304,364904448~4194304,373293056~16777216,402653184~12582912] 2024-08-17T17:51:27.364 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~29360128,33554432~33554432,71303168~25165824,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,209715200~12582912,226492416~12582912,243269632~25165824,272629760~50331648,327155712~12582912,348127232~4194304,364904448~4194304,373293056~16777216,402653184~12582912] 2024-08-17T17:51:27.367 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~100663296,109051904~33554432,146800640~54525952,209715200~12582912,226492416~117440512,348127232~8388608,364904448~4194304,373293056~20971520,402653184~12582912] 2024-08-17T17:51:27.367 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to 18 diff [0~29360128,33554432~33554432,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~16777216,205520896~16777216,226492416~12582912,243269632~25165824,272629760~50331648,327155712~16777216,348127232~8388608,364904448~4194304,373293056~16777216,398458880~16777216] 2024-08-17T17:51:27.367 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~29360128,33554432~33554432,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~16777216,205520896~16777216,226492416~12582912,243269632~25165824,272629760~50331648,327155712~16777216,348127232~8388608,364904448~4194304,373293056~16777216,398458880~16777216] 2024-08-17T17:51:27.371 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~54525952,205520896~16777216,226492416~117440512,348127232~12582912,364904448~4194304,373293056~20971520,398458880~16777216] 2024-08-17T17:51:27.371 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to 19 diff [0~29360128,33554432~33554432,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~8388608,364904448~4194304,373293056~16777216,394264576~25165824] 2024-08-17T17:51:27.371 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~29360128,33554432~33554432,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~8388608,364904448~4194304,373293056~16777216,394264576~25165824] 2024-08-17T17:51:27.375 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~54525952,205520896~138412032,348127232~12582912,364904448~4194304,373293056~46137344] 2024-08-17T17:51:27.375 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to 11 diff [20971520~8388608,62914560~8388608,109051904~8388608,180355072~8388608,230686720~4194304,247463936~8388608,272629760~4194304,306184192~8388608,364904448~4194304,402653184~8388608] 2024-08-17T17:51:27.375 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [20971520~8388608,62914560~8388608,109051904~8388608,180355072~8388608,230686720~4194304,247463936~8388608,272629760~4194304,306184192~8388608,364904448~4194304,402653184~8388608] 2024-08-17T17:51:27.376 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [20971520~8388608,62914560~8388608,109051904~12582912,180355072~8388608,230686720~4194304,247463936~8388608,272629760~8388608,306184192~8388608,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.376 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to 12 diff [20971520~8388608,37748736~4194304,62914560~12582912,109051904~16777216,159383552~8388608,180355072~16777216,230686720~8388608,247463936~8388608,272629760~4194304,289406976~4194304,306184192~8388608,364904448~4194304,402653184~8388608] 2024-08-17T17:51:27.376 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [20971520~8388608,37748736~4194304,62914560~12582912,109051904~16777216,159383552~8388608,180355072~16777216,230686720~8388608,247463936~8388608,272629760~4194304,289406976~4194304,306184192~8388608,364904448~4194304,402653184~8388608] 2024-08-17T17:51:27.378 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [20971520~12582912,37748736~8388608,62914560~12582912,109051904~20971520,159383552~12582912,180355072~20971520,230686720~12582912,247463936~8388608,272629760~8388608,289406976~8388608,306184192~8388608,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.378 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to 13 diff [20971520~8388608,33554432~8388608,46137344~12582912,62914560~25165824,109051904~16777216,159383552~16777216,180355072~16777216,213909504~8388608,230686720~8388608,247463936~8388608,272629760~4194304,289406976~4194304,306184192~8388608,335544320~4194304,364904448~4194304,402653184~8388608] 2024-08-17T17:51:27.378 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [20971520~8388608,33554432~8388608,46137344~12582912,62914560~25165824,109051904~16777216,159383552~16777216,180355072~16777216,213909504~8388608,230686720~8388608,247463936~8388608,272629760~4194304,289406976~4194304,306184192~8388608,335544320~4194304,364904448~4194304,402653184~8388608] 2024-08-17T17:51:27.379 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [20971520~67108864,109051904~20971520,159383552~41943040,213909504~8388608,230686720~12582912,247463936~8388608,272629760~8388608,289406976~8388608,306184192~8388608,335544320~4194304,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.379 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to 14 diff [20971520~8388608,33554432~8388608,46137344~12582912,62914560~25165824,92274688~4194304,109051904~16777216,155189248~20971520,180355072~16777216,209715200~12582912,230686720~8388608,243269632~20971520,272629760~4194304,281018368~12582912,306184192~12582912,335544320~4194304,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.379 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [20971520~8388608,33554432~8388608,46137344~12582912,62914560~25165824,92274688~4194304,109051904~16777216,155189248~20971520,180355072~16777216,209715200~12582912,230686720~8388608,243269632~20971520,272629760~4194304,281018368~12582912,306184192~12582912,335544320~4194304,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.381 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [20971520~67108864,92274688~4194304,109051904~20971520,155189248~46137344,209715200~12582912,230686720~33554432,272629760~25165824,306184192~16777216,335544320~4194304,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.381 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to 15 diff [20971520~8388608,33554432~25165824,62914560~25165824,92274688~4194304,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,209715200~12582912,230686720~8388608,243269632~20971520,272629760~4194304,281018368~12582912,301989888~16777216,331350016~8388608,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.381 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [20971520~8388608,33554432~25165824,62914560~25165824,92274688~4194304,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,209715200~12582912,230686720~8388608,243269632~20971520,272629760~4194304,281018368~12582912,301989888~16777216,331350016~8388608,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.384 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [20971520~67108864,92274688~4194304,109051904~33554432,146800640~54525952,209715200~12582912,230686720~37748736,272629760~25165824,301989888~20971520,331350016~12582912,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.384 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to 16 diff [20971520~8388608,33554432~25165824,62914560~25165824,92274688~4194304,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,209715200~12582912,226492416~12582912,243269632~20971520,272629760~20971520,301989888~16777216,327155712~12582912,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.384 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [20971520~8388608,33554432~25165824,62914560~25165824,92274688~4194304,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,209715200~12582912,226492416~12582912,243269632~20971520,272629760~20971520,301989888~16777216,327155712~12582912,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.386 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [20971520~67108864,92274688~4194304,109051904~33554432,146800640~54525952,209715200~12582912,226492416~41943040,272629760~25165824,301989888~20971520,327155712~16777216,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.386 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to 17 diff [12582912~16777216,33554432~25165824,62914560~33554432,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,209715200~12582912,226492416~12582912,243269632~25165824,272629760~20971520,301989888~20971520,327155712~12582912,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.386 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [12582912~16777216,33554432~25165824,62914560~33554432,109051904~29360128,146800640~4194304,155189248~20971520,180355072~16777216,209715200~12582912,226492416~12582912,243269632~25165824,272629760~20971520,301989888~20971520,327155712~12582912,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.389 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [12582912~88080384,109051904~33554432,146800640~54525952,209715200~12582912,226492416~71303168,301989888~41943040,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.389 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to 18 diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~16777216,205520896~16777216,226492416~12582912,243269632~25165824,272629760~20971520,301989888~20971520,327155712~16777216,348127232~8388608,364904448~4194304,398458880~16777216] 2024-08-17T17:51:27.389 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~16777216,205520896~16777216,226492416~12582912,243269632~25165824,272629760~20971520,301989888~20971520,327155712~16777216,348127232~8388608,364904448~4194304,398458880~16777216] 2024-08-17T17:51:27.393 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~130023424,146800640~54525952,205520896~16777216,226492416~71303168,301989888~41943040,348127232~12582912,364904448~4194304,398458880~16777216] 2024-08-17T17:51:27.393 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to 19 diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~4194304,301989888~20971520,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.393 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~4194304,301989888~20971520,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.396 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~130023424,146800640~54525952,205520896~92274688,301989888~41943040,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.396 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to 12 diff [20971520~8388608,37748736~4194304,71303168~4194304,117440512~8388608,159383552~8388608,180355072~8388608,192937984~4194304,234881024~4194304,289406976~4194304] 2024-08-17T17:51:27.396 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [20971520~8388608,37748736~4194304,71303168~4194304,117440512~8388608,159383552~8388608,180355072~8388608,192937984~4194304,234881024~4194304,289406976~4194304] 2024-08-17T17:51:27.397 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [20971520~12582912,37748736~8388608,71303168~4194304,117440512~12582912,159383552~12582912,180355072~20971520,234881024~8388608,289406976~8388608] 2024-08-17T17:51:27.397 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to 13 diff [20971520~8388608,33554432~8388608,46137344~12582912,71303168~16777216,117440512~8388608,159383552~16777216,180355072~8388608,192937984~4194304,213909504~8388608,234881024~4194304,289406976~4194304,335544320~4194304] 2024-08-17T17:51:27.397 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [20971520~8388608,33554432~8388608,46137344~12582912,71303168~16777216,117440512~8388608,159383552~16777216,180355072~8388608,192937984~4194304,213909504~8388608,234881024~4194304,289406976~4194304,335544320~4194304] 2024-08-17T17:51:27.399 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [20971520~41943040,71303168~16777216,117440512~12582912,159383552~41943040,213909504~8388608,234881024~8388608,289406976~8388608,335544320~4194304] 2024-08-17T17:51:27.399 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to 14 diff [20971520~8388608,33554432~8388608,46137344~12582912,71303168~16777216,92274688~4194304,117440512~8388608,155189248~20971520,180355072~8388608,192937984~4194304,209715200~12582912,234881024~4194304,243269632~4194304,255852544~8388608,281018368~12582912,314572800~4194304,335544320~4194304,402653184~12582912] 2024-08-17T17:51:27.399 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [20971520~8388608,33554432~8388608,46137344~12582912,71303168~16777216,92274688~4194304,117440512~8388608,155189248~20971520,180355072~8388608,192937984~4194304,209715200~12582912,234881024~4194304,243269632~4194304,255852544~8388608,281018368~12582912,314572800~4194304,335544320~4194304,402653184~12582912] 2024-08-17T17:51:27.400 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [20971520~41943040,71303168~16777216,92274688~4194304,117440512~12582912,155189248~46137344,209715200~12582912,234881024~12582912,255852544~8388608,281018368~16777216,314572800~8388608,335544320~4194304,402653184~12582912] 2024-08-17T17:51:27.400 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to 15 diff [20971520~8388608,33554432~25165824,71303168~16777216,92274688~4194304,117440512~20971520,146800640~4194304,155189248~20971520,180355072~8388608,192937984~4194304,209715200~12582912,234881024~4194304,243269632~4194304,251658240~12582912,281018368~12582912,301989888~8388608,314572800~4194304,331350016~8388608,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.400 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [20971520~8388608,33554432~25165824,71303168~16777216,92274688~4194304,117440512~20971520,146800640~4194304,155189248~20971520,180355072~8388608,192937984~4194304,209715200~12582912,234881024~4194304,243269632~4194304,251658240~12582912,281018368~12582912,301989888~8388608,314572800~4194304,331350016~8388608,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.402 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [20971520~41943040,71303168~16777216,92274688~4194304,117440512~25165824,146800640~54525952,209715200~12582912,234881024~12582912,251658240~16777216,281018368~16777216,301989888~8388608,314572800~8388608,331350016~12582912,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.402 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to 16 diff [20971520~8388608,33554432~25165824,67108864~20971520,92274688~4194304,109051904~4194304,117440512~20971520,146800640~4194304,155189248~20971520,180355072~8388608,192937984~4194304,209715200~12582912,226492416~12582912,243269632~4194304,251658240~12582912,272629760~4194304,281018368~12582912,301989888~8388608,314572800~4194304,327155712~12582912,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.402 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [20971520~8388608,33554432~25165824,67108864~20971520,92274688~4194304,109051904~4194304,117440512~20971520,146800640~4194304,155189248~20971520,180355072~8388608,192937984~4194304,209715200~12582912,226492416~12582912,243269632~4194304,251658240~12582912,272629760~4194304,281018368~12582912,301989888~8388608,314572800~4194304,327155712~12582912,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.405 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [20971520~41943040,67108864~20971520,92274688~4194304,109051904~4194304,117440512~25165824,146800640~54525952,209715200~12582912,226492416~20971520,251658240~16777216,272629760~4194304,281018368~16777216,301989888~8388608,314572800~8388608,327155712~16777216,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.405 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to 17 diff [12582912~16777216,33554432~25165824,62914560~33554432,109051904~29360128,146800640~4194304,155189248~20971520,180355072~8388608,192937984~4194304,209715200~12582912,226492416~12582912,243269632~4194304,251658240~16777216,272629760~4194304,281018368~12582912,301989888~8388608,314572800~8388608,327155712~12582912,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.405 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [12582912~16777216,33554432~25165824,62914560~33554432,109051904~29360128,146800640~4194304,155189248~20971520,180355072~8388608,192937984~4194304,209715200~12582912,226492416~12582912,243269632~4194304,251658240~16777216,272629760~4194304,281018368~12582912,301989888~8388608,314572800~8388608,327155712~12582912,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.407 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [12582912~88080384,109051904~33554432,146800640~54525952,209715200~12582912,226492416~20971520,251658240~25165824,281018368~16777216,301989888~8388608,314572800~29360128,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.407 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to 18 diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~8388608,192937984~4194304,205520896~16777216,226492416~12582912,243269632~4194304,251658240~16777216,272629760~4194304,281018368~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,398458880~16777216] 2024-08-17T17:51:27.407 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~8388608,192937984~4194304,205520896~16777216,226492416~12582912,243269632~4194304,251658240~16777216,272629760~4194304,281018368~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,398458880~16777216] 2024-08-17T17:51:27.410 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~130023424,146800640~54525952,205520896~16777216,226492416~20971520,251658240~25165824,281018368~16777216,301989888~8388608,314572800~29360128,348127232~12582912,364904448~4194304,398458880~16777216] 2024-08-17T17:51:27.410 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to 19 diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~8388608,192937984~8388608,205520896~33554432,243269632~4194304,251658240~16777216,272629760~12582912,289406976~4194304,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.410 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~8388608,192937984~8388608,205520896~33554432,243269632~4194304,251658240~16777216,272629760~12582912,289406976~4194304,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.414 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~130023424,146800640~54525952,205520896~41943040,251658240~46137344,301989888~8388608,314572800~29360128,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.414 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to 13 diff [20971520~4194304,33554432~4194304,46137344~12582912,71303168~16777216,163577856~12582912,213909504~8388608,335544320~4194304] 2024-08-17T17:51:27.414 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [20971520~4194304,33554432~4194304,46137344~12582912,71303168~16777216,163577856~12582912,213909504~8388608,335544320~4194304] 2024-08-17T17:51:27.414 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [20971520~8388608,33554432~4194304,46137344~16777216,71303168~16777216,163577856~16777216,213909504~8388608,335544320~4194304] 2024-08-17T17:51:27.415 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to 14 diff [20971520~4194304,33554432~4194304,46137344~12582912,71303168~16777216,92274688~4194304,155189248~4194304,163577856~12582912,209715200~12582912,243269632~4194304,255852544~8388608,281018368~8388608,314572800~4194304,335544320~4194304,402653184~12582912] 2024-08-17T17:51:27.415 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [20971520~4194304,33554432~4194304,46137344~12582912,71303168~16777216,92274688~4194304,155189248~4194304,163577856~12582912,209715200~12582912,243269632~4194304,255852544~8388608,281018368~8388608,314572800~4194304,335544320~4194304,402653184~12582912] 2024-08-17T17:51:27.416 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [20971520~8388608,33554432~4194304,46137344~16777216,71303168~16777216,92274688~4194304,155189248~25165824,209715200~12582912,243269632~4194304,255852544~8388608,281018368~8388608,314572800~8388608,335544320~4194304,402653184~12582912] 2024-08-17T17:51:27.416 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to 15 diff [20971520~4194304,33554432~4194304,41943040~16777216,71303168~16777216,92274688~4194304,121634816~4194304,130023424~8388608,146800640~4194304,155189248~4194304,163577856~12582912,209715200~12582912,243269632~4194304,251658240~12582912,281018368~8388608,301989888~8388608,314572800~4194304,331350016~8388608,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.416 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [20971520~4194304,33554432~4194304,41943040~16777216,71303168~16777216,92274688~4194304,121634816~4194304,130023424~8388608,146800640~4194304,155189248~4194304,163577856~12582912,209715200~12582912,243269632~4194304,251658240~12582912,281018368~8388608,301989888~8388608,314572800~4194304,331350016~8388608,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.417 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [20971520~8388608,33554432~4194304,41943040~20971520,71303168~16777216,92274688~4194304,121634816~20971520,146800640~33554432,209715200~12582912,243269632~4194304,251658240~16777216,281018368~8388608,301989888~8388608,314572800~8388608,331350016~12582912,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.417 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to 16 diff [20971520~16777216,41943040~16777216,67108864~20971520,92274688~4194304,109051904~4194304,121634816~4194304,130023424~8388608,146800640~4194304,155189248~4194304,163577856~12582912,209715200~12582912,226492416~8388608,243269632~4194304,251658240~12582912,272629760~4194304,281018368~8388608,301989888~8388608,314572800~4194304,327155712~12582912,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.417 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [20971520~16777216,41943040~16777216,67108864~20971520,92274688~4194304,109051904~4194304,121634816~4194304,130023424~8388608,146800640~4194304,155189248~4194304,163577856~12582912,209715200~12582912,226492416~8388608,243269632~4194304,251658240~12582912,272629760~4194304,281018368~8388608,301989888~8388608,314572800~4194304,327155712~12582912,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.419 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [20971520~16777216,41943040~20971520,67108864~20971520,92274688~4194304,109051904~4194304,121634816~20971520,146800640~33554432,209715200~12582912,226492416~12582912,243269632~4194304,251658240~16777216,272629760~4194304,281018368~8388608,301989888~8388608,314572800~8388608,327155712~16777216,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.419 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to 17 diff [12582912~25165824,41943040~16777216,62914560~33554432,109051904~8388608,121634816~4194304,130023424~8388608,146800640~4194304,155189248~4194304,163577856~12582912,209715200~12582912,226492416~8388608,243269632~4194304,251658240~16777216,272629760~4194304,281018368~8388608,301989888~8388608,314572800~8388608,327155712~12582912,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.419 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [12582912~25165824,41943040~16777216,62914560~33554432,109051904~8388608,121634816~4194304,130023424~8388608,146800640~4194304,155189248~4194304,163577856~12582912,209715200~12582912,226492416~8388608,243269632~4194304,251658240~16777216,272629760~4194304,281018368~8388608,301989888~8388608,314572800~8388608,327155712~12582912,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.422 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [12582912~25165824,41943040~58720256,109051904~8388608,121634816~20971520,146800640~33554432,209715200~12582912,226492416~12582912,243269632~4194304,251658240~25165824,281018368~8388608,301989888~8388608,314572800~29360128,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.422 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to 18 diff [0~4194304,12582912~25165824,41943040~16777216,62914560~33554432,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~20971520,205520896~16777216,226492416~8388608,243269632~4194304,251658240~16777216,272629760~4194304,281018368~8388608,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,398458880~16777216] 2024-08-17T17:51:27.422 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~25165824,41943040~16777216,62914560~33554432,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~20971520,205520896~16777216,226492416~8388608,243269632~4194304,251658240~16777216,272629760~4194304,281018368~8388608,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,398458880~16777216] 2024-08-17T17:51:27.425 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~25165824,41943040~75497472,121634816~20971520,146800640~33554432,205520896~16777216,226492416~12582912,243269632~4194304,251658240~25165824,281018368~8388608,301989888~8388608,314572800~29360128,348127232~12582912,364904448~4194304,398458880~16777216] 2024-08-17T17:51:27.425 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to 19 diff [0~4194304,12582912~25165824,41943040~16777216,62914560~33554432,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~20971520,192937984~4194304,205520896~29360128,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.425 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~25165824,41943040~16777216,62914560~33554432,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~20971520,192937984~4194304,205520896~29360128,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.427 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~25165824,41943040~75497472,121634816~20971520,146800640~33554432,192937984~8388608,205520896~33554432,243269632~4194304,251658240~37748736,301989888~8388608,314572800~29360128,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.428 INFO:tasks.workunit.client.0.smithi136.stdout:from 13 to 14 diff [92274688~4194304,155189248~4194304,209715200~4194304,243269632~4194304,255852544~8388608,281018368~8388608,314572800~4194304,402653184~12582912] 2024-08-17T17:51:27.428 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [92274688~4194304,155189248~4194304,209715200~4194304,243269632~4194304,255852544~8388608,281018368~8388608,314572800~4194304,402653184~12582912] 2024-08-17T17:51:27.428 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [92274688~4194304,155189248~8388608,209715200~4194304,243269632~4194304,255852544~8388608,281018368~8388608,314572800~8388608,402653184~12582912] 2024-08-17T17:51:27.428 INFO:tasks.workunit.client.0.smithi136.stdout:from 13 to 15 diff [41943040~4194304,92274688~4194304,121634816~4194304,130023424~8388608,146800640~4194304,155189248~4194304,209715200~4194304,243269632~4194304,251658240~12582912,281018368~8388608,301989888~8388608,314572800~4194304,331350016~8388608,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.428 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [41943040~4194304,92274688~4194304,121634816~4194304,130023424~8388608,146800640~4194304,155189248~4194304,209715200~4194304,243269632~4194304,251658240~12582912,281018368~8388608,301989888~8388608,314572800~4194304,331350016~8388608,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.429 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [41943040~8388608,92274688~4194304,121634816~20971520,146800640~16777216,209715200~4194304,243269632~4194304,251658240~16777216,281018368~8388608,301989888~8388608,314572800~8388608,331350016~12582912,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.430 INFO:tasks.workunit.client.0.smithi136.stdout:from 13 to 16 diff [20971520~4194304,29360128~4194304,41943040~4194304,67108864~12582912,92274688~4194304,109051904~4194304,121634816~4194304,130023424~8388608,146800640~4194304,155189248~4194304,209715200~4194304,226492416~8388608,243269632~4194304,251658240~12582912,272629760~4194304,281018368~8388608,301989888~8388608,314572800~4194304,327155712~12582912,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.430 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [20971520~4194304,29360128~4194304,41943040~4194304,67108864~12582912,92274688~4194304,109051904~4194304,121634816~4194304,130023424~8388608,146800640~4194304,155189248~4194304,209715200~4194304,226492416~8388608,243269632~4194304,251658240~12582912,272629760~4194304,281018368~8388608,301989888~8388608,314572800~4194304,327155712~12582912,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.431 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [20971520~12582912,41943040~8388608,67108864~12582912,92274688~4194304,109051904~4194304,121634816~20971520,146800640~16777216,209715200~4194304,226492416~12582912,243269632~4194304,251658240~16777216,272629760~4194304,281018368~8388608,301989888~8388608,314572800~8388608,327155712~16777216,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.431 INFO:tasks.workunit.client.0.smithi136.stdout:from 13 to 17 diff [12582912~12582912,29360128~4194304,41943040~4194304,62914560~16777216,88080384~8388608,109051904~8388608,121634816~4194304,130023424~8388608,146800640~4194304,155189248~4194304,209715200~8388608,226492416~8388608,243269632~4194304,251658240~16777216,272629760~4194304,281018368~8388608,301989888~8388608,314572800~8388608,327155712~12582912,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.431 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [12582912~12582912,29360128~4194304,41943040~4194304,62914560~16777216,88080384~8388608,109051904~8388608,121634816~4194304,130023424~8388608,146800640~4194304,155189248~4194304,209715200~8388608,226492416~8388608,243269632~4194304,251658240~16777216,272629760~4194304,281018368~8388608,301989888~8388608,314572800~8388608,327155712~12582912,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.433 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [12582912~20971520,41943040~8388608,62914560~16777216,88080384~12582912,109051904~8388608,121634816~20971520,146800640~16777216,209715200~12582912,226492416~12582912,243269632~4194304,251658240~25165824,281018368~8388608,301989888~8388608,314572800~29360128,364904448~4194304,402653184~12582912] 2024-08-17T17:51:27.433 INFO:tasks.workunit.client.0.smithi136.stdout:from 13 to 18 diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,205520896~12582912,226492416~8388608,243269632~4194304,251658240~16777216,272629760~4194304,281018368~8388608,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,398458880~16777216] 2024-08-17T17:51:27.433 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,205520896~12582912,226492416~8388608,243269632~4194304,251658240~16777216,272629760~4194304,281018368~8388608,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,398458880~16777216] 2024-08-17T17:51:27.435 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~20971520,41943040~8388608,62914560~54525952,121634816~20971520,146800640~25165824,205520896~16777216,226492416~12582912,243269632~4194304,251658240~25165824,281018368~8388608,301989888~8388608,314572800~29360128,348127232~12582912,364904448~4194304,398458880~16777216] 2024-08-17T17:51:27.435 INFO:tasks.workunit.client.0.smithi136.stdout:from 13 to 19 diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,192937984~4194304,205520896~12582912,222298112~12582912,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.436 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,192937984~4194304,205520896~12582912,222298112~12582912,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.438 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~20971520,41943040~8388608,62914560~54525952,121634816~20971520,146800640~25165824,192937984~8388608,205520896~33554432,243269632~4194304,251658240~37748736,301989888~8388608,314572800~29360128,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.438 INFO:tasks.workunit.client.0.smithi136.stdout:from 14 to 15 diff [41943040~4194304,121634816~4194304,130023424~8388608,146800640~4194304,155189248~4194304,251658240~12582912,301989888~8388608,331350016~8388608,364904448~4194304] 2024-08-17T17:51:27.438 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [41943040~4194304,121634816~4194304,130023424~8388608,146800640~4194304,155189248~4194304,251658240~12582912,301989888~8388608,331350016~8388608,364904448~4194304] 2024-08-17T17:51:27.439 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [41943040~8388608,121634816~20971520,146800640~16777216,251658240~16777216,301989888~8388608,331350016~12582912,364904448~4194304] 2024-08-17T17:51:27.439 INFO:tasks.workunit.client.0.smithi136.stdout:from 14 to 16 diff [20971520~4194304,29360128~4194304,41943040~4194304,67108864~12582912,109051904~4194304,121634816~4194304,130023424~8388608,146800640~4194304,155189248~4194304,226492416~8388608,251658240~12582912,272629760~4194304,301989888~8388608,327155712~12582912,364904448~4194304] 2024-08-17T17:51:27.439 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [20971520~4194304,29360128~4194304,41943040~4194304,67108864~12582912,109051904~4194304,121634816~4194304,130023424~8388608,146800640~4194304,155189248~4194304,226492416~8388608,251658240~12582912,272629760~4194304,301989888~8388608,327155712~12582912,364904448~4194304] 2024-08-17T17:51:27.440 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [20971520~12582912,41943040~8388608,67108864~12582912,109051904~4194304,121634816~20971520,146800640~16777216,226492416~12582912,251658240~16777216,272629760~4194304,301989888~8388608,327155712~16777216,364904448~4194304] 2024-08-17T17:51:27.440 INFO:tasks.workunit.client.0.smithi136.stdout:from 14 to 17 diff [12582912~12582912,29360128~4194304,41943040~4194304,62914560~16777216,88080384~8388608,109051904~8388608,121634816~4194304,130023424~8388608,146800640~4194304,155189248~4194304,213909504~4194304,226492416~8388608,251658240~16777216,272629760~4194304,301989888~8388608,318767104~20971520,364904448~4194304] 2024-08-17T17:51:27.440 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [12582912~12582912,29360128~4194304,41943040~4194304,62914560~16777216,88080384~8388608,109051904~8388608,121634816~4194304,130023424~8388608,146800640~4194304,155189248~4194304,213909504~4194304,226492416~8388608,251658240~16777216,272629760~4194304,301989888~8388608,318767104~20971520,364904448~4194304] 2024-08-17T17:51:27.442 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [12582912~20971520,41943040~8388608,62914560~16777216,88080384~12582912,109051904~8388608,121634816~20971520,146800640~16777216,213909504~8388608,226492416~12582912,251658240~25165824,301989888~8388608,318767104~25165824,364904448~4194304] 2024-08-17T17:51:27.442 INFO:tasks.workunit.client.0.smithi136.stdout:from 14 to 18 diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,205520896~4194304,213909504~4194304,226492416~8388608,251658240~16777216,272629760~4194304,301989888~8388608,318767104~25165824,348127232~8388608,364904448~4194304,398458880~8388608] 2024-08-17T17:51:27.442 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,205520896~4194304,213909504~4194304,226492416~8388608,251658240~16777216,272629760~4194304,301989888~8388608,318767104~25165824,348127232~8388608,364904448~4194304,398458880~8388608] 2024-08-17T17:51:27.444 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~20971520,41943040~8388608,62914560~54525952,121634816~20971520,146800640~25165824,205520896~4194304,213909504~8388608,226492416~12582912,251658240~25165824,301989888~8388608,318767104~25165824,348127232~12582912,364904448~4194304,398458880~12582912] 2024-08-17T17:51:27.444 INFO:tasks.workunit.client.0.smithi136.stdout:from 14 to 19 diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,192937984~4194304,205520896~4194304,213909504~4194304,222298112~12582912,251658240~16777216,272629760~8388608,301989888~8388608,318767104~25165824,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.444 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,192937984~4194304,205520896~4194304,213909504~4194304,222298112~12582912,251658240~16777216,272629760~8388608,301989888~8388608,318767104~25165824,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.446 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~20971520,41943040~8388608,62914560~54525952,121634816~20971520,146800640~25165824,192937984~8388608,205520896~4194304,213909504~25165824,251658240~33554432,301989888~8388608,318767104~25165824,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.446 INFO:tasks.workunit.client.0.smithi136.stdout:from 15 to 16 diff [20971520~4194304,29360128~4194304,67108864~12582912,109051904~4194304,134217728~4194304,226492416~8388608,272629760~4194304,327155712~8388608] 2024-08-17T17:51:27.447 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [20971520~4194304,29360128~4194304,67108864~12582912,109051904~4194304,134217728~4194304,226492416~8388608,272629760~4194304,327155712~8388608] 2024-08-17T17:51:27.447 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [20971520~12582912,67108864~12582912,109051904~4194304,134217728~4194304,226492416~12582912,272629760~4194304,327155712~8388608] 2024-08-17T17:51:27.447 INFO:tasks.workunit.client.0.smithi136.stdout:from 15 to 17 diff [12582912~12582912,29360128~4194304,62914560~16777216,88080384~8388608,109051904~8388608,134217728~4194304,146800640~4194304,213909504~4194304,226492416~8388608,251658240~16777216,272629760~4194304,318767104~16777216] 2024-08-17T17:51:27.447 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [12582912~12582912,29360128~4194304,62914560~16777216,88080384~8388608,109051904~8388608,134217728~4194304,146800640~4194304,213909504~4194304,226492416~8388608,251658240~16777216,272629760~4194304,318767104~16777216] 2024-08-17T17:51:27.449 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [12582912~20971520,62914560~16777216,88080384~12582912,109051904~8388608,134217728~4194304,146800640~8388608,213909504~8388608,226492416~12582912,251658240~25165824,318767104~16777216] 2024-08-17T17:51:27.449 INFO:tasks.workunit.client.0.smithi136.stdout:from 15 to 18 diff [0~4194304,12582912~12582912,29360128~4194304,62914560~20971520,88080384~8388608,100663296~16777216,134217728~4194304,146800640~4194304,159383552~8388608,205520896~4194304,213909504~4194304,226492416~8388608,251658240~16777216,272629760~4194304,318767104~25165824,348127232~8388608,398458880~8388608] 2024-08-17T17:51:27.449 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~12582912,29360128~4194304,62914560~20971520,88080384~8388608,100663296~16777216,134217728~4194304,146800640~4194304,159383552~8388608,205520896~4194304,213909504~4194304,226492416~8388608,251658240~16777216,272629760~4194304,318767104~25165824,348127232~8388608,398458880~8388608] 2024-08-17T17:51:27.450 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~20971520,62914560~54525952,134217728~4194304,146800640~8388608,159383552~12582912,205520896~4194304,213909504~8388608,226492416~12582912,251658240~25165824,318767104~25165824,348127232~12582912,398458880~12582912] 2024-08-17T17:51:27.450 INFO:tasks.workunit.client.0.smithi136.stdout:from 15 to 19 diff [0~4194304,12582912~12582912,29360128~4194304,62914560~20971520,88080384~8388608,100663296~16777216,134217728~4194304,146800640~4194304,159383552~8388608,192937984~4194304,205520896~4194304,213909504~4194304,222298112~12582912,251658240~16777216,272629760~8388608,318767104~25165824,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.451 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~12582912,29360128~4194304,62914560~20971520,88080384~8388608,100663296~16777216,134217728~4194304,146800640~4194304,159383552~8388608,192937984~4194304,205520896~4194304,213909504~4194304,222298112~12582912,251658240~16777216,272629760~8388608,318767104~25165824,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.452 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~20971520,62914560~54525952,134217728~4194304,146800640~8388608,159383552~12582912,192937984~8388608,205520896~4194304,213909504~25165824,251658240~33554432,318767104~25165824,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.453 INFO:tasks.workunit.client.0.smithi136.stdout:from 16 to 17 diff [12582912~8388608,62914560~8388608,88080384~8388608,113246208~4194304,146800640~4194304,213909504~4194304,251658240~16777216,318767104~8388608] 2024-08-17T17:51:27.453 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [12582912~8388608,62914560~8388608,88080384~8388608,113246208~4194304,146800640~4194304,213909504~4194304,251658240~16777216,318767104~8388608] 2024-08-17T17:51:27.454 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [12582912~12582912,62914560~8388608,88080384~12582912,113246208~4194304,146800640~8388608,213909504~8388608,251658240~20971520,318767104~8388608] 2024-08-17T17:51:27.454 INFO:tasks.workunit.client.0.smithi136.stdout:from 16 to 18 diff [0~4194304,12582912~8388608,62914560~8388608,79691776~4194304,88080384~8388608,100663296~8388608,113246208~4194304,146800640~4194304,159383552~8388608,205520896~4194304,213909504~4194304,251658240~16777216,318767104~12582912,335544320~8388608,348127232~8388608,398458880~8388608] 2024-08-17T17:51:27.454 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~8388608,62914560~8388608,79691776~4194304,88080384~8388608,100663296~8388608,113246208~4194304,146800640~4194304,159383552~8388608,205520896~4194304,213909504~4194304,251658240~16777216,318767104~12582912,335544320~8388608,348127232~8388608,398458880~8388608] 2024-08-17T17:51:27.455 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~12582912,62914560~8388608,79691776~37748736,146800640~8388608,159383552~12582912,205520896~4194304,213909504~8388608,251658240~20971520,318767104~25165824,348127232~12582912,398458880~12582912] 2024-08-17T17:51:27.455 INFO:tasks.workunit.client.0.smithi136.stdout:from 16 to 19 diff [0~4194304,12582912~8388608,62914560~8388608,79691776~4194304,88080384~8388608,100663296~8388608,113246208~4194304,146800640~4194304,159383552~8388608,192937984~4194304,205520896~4194304,213909504~4194304,222298112~12582912,251658240~16777216,276824064~4194304,318767104~12582912,335544320~8388608,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.455 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~8388608,62914560~8388608,79691776~4194304,88080384~8388608,100663296~8388608,113246208~4194304,146800640~4194304,159383552~8388608,192937984~4194304,205520896~4194304,213909504~4194304,222298112~12582912,251658240~16777216,276824064~4194304,318767104~12582912,335544320~8388608,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.457 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~12582912,62914560~8388608,79691776~37748736,146800640~8388608,159383552~12582912,192937984~8388608,205520896~4194304,213909504~20971520,251658240~20971520,276824064~8388608,318767104~25165824,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.457 INFO:tasks.workunit.client.0.smithi136.stdout:from 17 to 18 diff [0~4194304,79691776~4194304,100663296~8388608,159383552~8388608,205520896~4194304,327155712~4194304,335544320~8388608,348127232~8388608,398458880~8388608] 2024-08-17T17:51:27.457 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,79691776~4194304,100663296~8388608,159383552~8388608,205520896~4194304,327155712~4194304,335544320~8388608,348127232~8388608,398458880~8388608] 2024-08-17T17:51:27.457 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,79691776~8388608,100663296~12582912,159383552~12582912,205520896~4194304,327155712~16777216,348127232~12582912,398458880~12582912] 2024-08-17T17:51:27.458 INFO:tasks.workunit.client.0.smithi136.stdout:from 17 to 19 diff [0~4194304,79691776~4194304,100663296~8388608,159383552~8388608,192937984~4194304,205520896~4194304,222298112~12582912,255852544~4194304,276824064~4194304,327155712~4194304,335544320~8388608,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.458 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,79691776~4194304,100663296~8388608,159383552~8388608,192937984~4194304,205520896~4194304,222298112~12582912,255852544~4194304,276824064~4194304,327155712~4194304,335544320~8388608,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.459 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,79691776~8388608,100663296~12582912,159383552~12582912,192937984~8388608,205520896~4194304,222298112~12582912,255852544~8388608,276824064~8388608,327155712~16777216,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.459 INFO:tasks.workunit.client.0.smithi136.stdout:from 18 to 19 diff [192937984~4194304,222298112~12582912,255852544~4194304,276824064~4194304,364904448~4194304,394264576~8388608,406847488~12582912] 2024-08-17T17:51:27.459 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [192937984~4194304,222298112~12582912,255852544~4194304,276824064~4194304,364904448~4194304,394264576~8388608,406847488~12582912] 2024-08-17T17:51:27.460 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [192937984~8388608,222298112~12582912,255852544~8388608,276824064~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.718 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:51:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:51:27] "GET /metrics HTTP/1.1" 200 36493 "" "Prometheus/2.51.0" 2024-08-17T17:51:27.846 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:27.846 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:27.863 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:27.863 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:27.863 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:27.869 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:27.869 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.869 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.876 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:27.876 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.876 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.882 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:27.882 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.882 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.888 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:27.888 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.888 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.893 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:27.893 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.893 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.897 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:27.897 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to HEAD diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.897 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.902 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:27.902 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to HEAD diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.902 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:27.906 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:27.906 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to HEAD diff [0~29360128,33554432~33554432,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~8388608,364904448~4194304,373293056~16777216,394264576~25165824] 2024-08-17T17:51:27.907 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~29360128,33554432~33554432,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~8388608,364904448~4194304,373293056~16777216,394264576~25165824] 2024-08-17T17:51:27.910 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~54525952,205520896~138412032,348127232~12582912,364904448~4194304,373293056~46137344] 2024-08-17T17:51:27.910 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to HEAD diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~4194304,301989888~20971520,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.910 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~4194304,301989888~20971520,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.914 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~130023424,146800640~54525952,205520896~92274688,301989888~41943040,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.914 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to HEAD diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~8388608,192937984~8388608,205520896~33554432,243269632~4194304,251658240~16777216,272629760~12582912,289406976~4194304,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.914 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~8388608,192937984~8388608,205520896~33554432,243269632~4194304,251658240~16777216,272629760~12582912,289406976~4194304,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.917 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~130023424,146800640~54525952,205520896~41943040,251658240~46137344,301989888~8388608,314572800~29360128,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.917 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to HEAD diff [0~4194304,12582912~25165824,41943040~16777216,62914560~33554432,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~20971520,192937984~4194304,205520896~29360128,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.917 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~25165824,41943040~16777216,62914560~33554432,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~20971520,192937984~4194304,205520896~29360128,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.919 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~25165824,41943040~75497472,121634816~20971520,146800640~33554432,192937984~8388608,205520896~33554432,243269632~4194304,251658240~37748736,301989888~8388608,314572800~29360128,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.919 INFO:tasks.workunit.client.0.smithi136.stdout:from 13 to HEAD diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,192937984~4194304,205520896~12582912,222298112~12582912,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.920 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,192937984~4194304,205520896~12582912,222298112~12582912,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.921 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~20971520,41943040~8388608,62914560~54525952,121634816~20971520,146800640~25165824,192937984~8388608,205520896~33554432,243269632~4194304,251658240~37748736,301989888~8388608,314572800~29360128,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.921 INFO:tasks.workunit.client.0.smithi136.stdout:from 14 to HEAD diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,192937984~4194304,205520896~4194304,213909504~4194304,222298112~12582912,251658240~16777216,272629760~8388608,301989888~8388608,318767104~25165824,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.921 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,192937984~4194304,205520896~4194304,213909504~4194304,222298112~12582912,251658240~16777216,272629760~8388608,301989888~8388608,318767104~25165824,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.923 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~20971520,41943040~8388608,62914560~54525952,121634816~20971520,146800640~25165824,192937984~8388608,205520896~4194304,213909504~25165824,251658240~33554432,301989888~8388608,318767104~25165824,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.923 INFO:tasks.workunit.client.0.smithi136.stdout:from 15 to HEAD diff [0~4194304,12582912~12582912,29360128~4194304,62914560~20971520,88080384~8388608,100663296~16777216,134217728~4194304,146800640~4194304,159383552~8388608,192937984~4194304,205520896~4194304,213909504~4194304,222298112~12582912,251658240~16777216,272629760~8388608,318767104~25165824,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.924 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~12582912,29360128~4194304,62914560~20971520,88080384~8388608,100663296~16777216,134217728~4194304,146800640~4194304,159383552~8388608,192937984~4194304,205520896~4194304,213909504~4194304,222298112~12582912,251658240~16777216,272629760~8388608,318767104~25165824,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.925 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~20971520,62914560~54525952,134217728~4194304,146800640~8388608,159383552~12582912,192937984~8388608,205520896~4194304,213909504~25165824,251658240~33554432,318767104~25165824,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.925 INFO:tasks.workunit.client.0.smithi136.stdout:from 16 to HEAD diff [0~4194304,12582912~8388608,62914560~8388608,79691776~4194304,88080384~8388608,100663296~8388608,113246208~4194304,146800640~4194304,159383552~8388608,192937984~4194304,205520896~4194304,213909504~4194304,222298112~12582912,251658240~16777216,276824064~4194304,318767104~12582912,335544320~8388608,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.925 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~8388608,62914560~8388608,79691776~4194304,88080384~8388608,100663296~8388608,113246208~4194304,146800640~4194304,159383552~8388608,192937984~4194304,205520896~4194304,213909504~4194304,222298112~12582912,251658240~16777216,276824064~4194304,318767104~12582912,335544320~8388608,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.926 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~12582912,62914560~8388608,79691776~37748736,146800640~8388608,159383552~12582912,192937984~8388608,205520896~4194304,213909504~20971520,251658240~20971520,276824064~8388608,318767104~25165824,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.926 INFO:tasks.workunit.client.0.smithi136.stdout:from 17 to HEAD diff [0~4194304,79691776~4194304,100663296~8388608,159383552~8388608,192937984~4194304,205520896~4194304,222298112~12582912,255852544~4194304,276824064~4194304,327155712~4194304,335544320~8388608,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.927 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,79691776~4194304,100663296~8388608,159383552~8388608,192937984~4194304,205520896~4194304,222298112~12582912,255852544~4194304,276824064~4194304,327155712~4194304,335544320~8388608,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:27.927 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,79691776~8388608,100663296~12582912,159383552~12582912,192937984~8388608,205520896~4194304,222298112~12582912,255852544~8388608,276824064~8388608,327155712~16777216,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:28.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:27 smithi165 ceph-mon[105517]: osdmap e1276: 8 total, 8 up, 8 in 2024-08-17T17:51:28.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:27 smithi136 ceph-mon[109861]: osdmap e1276: 8 total, 8 up, 8 in 2024-08-17T17:51:28.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:27 smithi136 ceph-mon[115078]: osdmap e1276: 8 total, 8 up, 8 in 2024-08-17T17:51:28.856 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:28.856 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:28.873 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:28.873 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:28.873 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:28.879 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:28.879 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:28.879 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:28.886 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:28.886 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:28.886 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:28.891 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:28.891 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:28.891 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:28.897 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:28.897 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:28.897 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:28.903 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:28.903 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:28.903 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:28.907 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:28.907 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to HEAD diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:28.907 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:28.912 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:28.912 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to HEAD diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:28.912 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:28.915 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:28.916 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to HEAD diff [0~29360128,33554432~33554432,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~8388608,364904448~4194304,373293056~16777216,394264576~25165824] 2024-08-17T17:51:28.916 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~29360128,33554432~33554432,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~8388608,364904448~4194304,373293056~16777216,394264576~25165824] 2024-08-17T17:51:28.919 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~54525952,205520896~138412032,348127232~12582912,364904448~4194304,373293056~46137344] 2024-08-17T17:51:28.919 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to HEAD diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~4194304,301989888~20971520,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:28.919 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~4194304,301989888~20971520,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:28.922 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~130023424,146800640~54525952,205520896~92274688,301989888~41943040,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:28.922 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to HEAD diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~8388608,192937984~8388608,205520896~33554432,243269632~4194304,251658240~16777216,272629760~12582912,289406976~4194304,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:28.922 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~8388608,192937984~8388608,205520896~33554432,243269632~4194304,251658240~16777216,272629760~12582912,289406976~4194304,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:28.924 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~130023424,146800640~54525952,205520896~41943040,251658240~46137344,301989888~8388608,314572800~29360128,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:28.925 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to HEAD diff [0~4194304,12582912~25165824,41943040~16777216,62914560~33554432,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~20971520,192937984~4194304,205520896~29360128,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:28.925 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~25165824,41943040~16777216,62914560~33554432,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~20971520,192937984~4194304,205520896~29360128,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:28.927 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~25165824,41943040~75497472,121634816~20971520,146800640~33554432,192937984~8388608,205520896~33554432,243269632~4194304,251658240~37748736,301989888~8388608,314572800~29360128,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:28.927 INFO:tasks.workunit.client.0.smithi136.stdout:from 13 to HEAD diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,192937984~4194304,205520896~12582912,222298112~12582912,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:28.927 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,192937984~4194304,205520896~12582912,222298112~12582912,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:28.929 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~20971520,41943040~8388608,62914560~54525952,121634816~20971520,146800640~25165824,192937984~8388608,205520896~33554432,243269632~4194304,251658240~37748736,301989888~8388608,314572800~29360128,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:28.929 INFO:tasks.workunit.client.0.smithi136.stdout:from 14 to HEAD diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,192937984~4194304,205520896~4194304,213909504~4194304,222298112~12582912,251658240~16777216,272629760~8388608,301989888~8388608,318767104~25165824,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:28.929 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,192937984~4194304,205520896~4194304,213909504~4194304,222298112~12582912,251658240~16777216,272629760~8388608,301989888~8388608,318767104~25165824,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:28.931 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~20971520,41943040~8388608,62914560~54525952,121634816~20971520,146800640~25165824,192937984~8388608,205520896~4194304,213909504~25165824,251658240~33554432,301989888~8388608,318767104~25165824,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:28.931 INFO:tasks.workunit.client.0.smithi136.stdout:from 15 to HEAD diff [0~4194304,12582912~12582912,29360128~4194304,62914560~20971520,88080384~8388608,100663296~16777216,134217728~4194304,146800640~4194304,159383552~8388608,192937984~4194304,205520896~4194304,213909504~4194304,222298112~12582912,251658240~16777216,272629760~8388608,318767104~25165824,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:28.931 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~12582912,29360128~4194304,62914560~20971520,88080384~8388608,100663296~16777216,134217728~4194304,146800640~4194304,159383552~8388608,192937984~4194304,205520896~4194304,213909504~4194304,222298112~12582912,251658240~16777216,272629760~8388608,318767104~25165824,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:28.932 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~20971520,62914560~54525952,134217728~4194304,146800640~8388608,159383552~12582912,192937984~8388608,205520896~4194304,213909504~25165824,251658240~33554432,318767104~25165824,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:28.932 INFO:tasks.workunit.client.0.smithi136.stdout:from 16 to HEAD diff [0~4194304,12582912~8388608,62914560~8388608,79691776~4194304,88080384~8388608,100663296~8388608,113246208~4194304,146800640~4194304,159383552~8388608,192937984~4194304,205520896~4194304,213909504~4194304,222298112~12582912,251658240~16777216,276824064~4194304,318767104~12582912,335544320~8388608,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:28.933 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~8388608,62914560~8388608,79691776~4194304,88080384~8388608,100663296~8388608,113246208~4194304,146800640~4194304,159383552~8388608,192937984~4194304,205520896~4194304,213909504~4194304,222298112~12582912,251658240~16777216,276824064~4194304,318767104~12582912,335544320~8388608,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:28.933 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~12582912,62914560~8388608,79691776~37748736,146800640~8388608,159383552~12582912,192937984~8388608,205520896~4194304,213909504~20971520,251658240~20971520,276824064~8388608,318767104~25165824,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:29.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:28 smithi165 ceph-mon[105517]: osdmap e1277: 8 total, 8 up, 8 in 2024-08-17T17:51:29.101 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:28 smithi165 ceph-mon[105517]: pgmap v1996: 105 pgs: 105 active+clean; 1.2 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 223 KiB/s rd, 42 MiB/s wr, 347 op/s 2024-08-17T17:51:29.161 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:28 smithi136 ceph-mon[109861]: osdmap e1277: 8 total, 8 up, 8 in 2024-08-17T17:51:29.161 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:28 smithi136 ceph-mon[109861]: pgmap v1996: 105 pgs: 105 active+clean; 1.2 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 223 KiB/s rd, 42 MiB/s wr, 347 op/s 2024-08-17T17:51:29.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:28 smithi136 ceph-mon[115078]: osdmap e1277: 8 total, 8 up, 8 in 2024-08-17T17:51:29.161 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:28 smithi136 ceph-mon[115078]: pgmap v1996: 105 pgs: 105 active+clean; 1.2 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 223 KiB/s rd, 42 MiB/s wr, 347 op/s 2024-08-17T17:51:29.864 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:29.864 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:29.880 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:29.880 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:29.880 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:29.886 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:29.886 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:29.886 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:29.892 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:29.892 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:29.892 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:29.897 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:29.897 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:29.897 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:29.901 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:29.901 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:29.901 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:29.906 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:29.906 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:29.906 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:29.910 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:29.910 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to HEAD diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:29.910 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:29.913 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:29.913 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to HEAD diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:29.913 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:29.916 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:29.916 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to HEAD diff [0~29360128,33554432~33554432,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~8388608,364904448~4194304,373293056~16777216,394264576~25165824] 2024-08-17T17:51:29.917 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~29360128,33554432~33554432,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~8388608,364904448~4194304,373293056~16777216,394264576~25165824] 2024-08-17T17:51:29.920 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~54525952,205520896~138412032,348127232~12582912,364904448~4194304,373293056~46137344] 2024-08-17T17:51:29.920 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to HEAD diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~4194304,301989888~20971520,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:29.920 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~4194304,301989888~20971520,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:29.923 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~130023424,146800640~54525952,205520896~92274688,301989888~41943040,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:29.923 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to HEAD diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~8388608,192937984~8388608,205520896~33554432,243269632~4194304,251658240~16777216,272629760~12582912,289406976~4194304,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:29.923 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~8388608,192937984~8388608,205520896~33554432,243269632~4194304,251658240~16777216,272629760~12582912,289406976~4194304,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:29.926 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~130023424,146800640~54525952,205520896~41943040,251658240~46137344,301989888~8388608,314572800~29360128,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:29.926 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to HEAD diff [0~4194304,12582912~25165824,41943040~16777216,62914560~33554432,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~20971520,192937984~4194304,205520896~29360128,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:29.926 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~25165824,41943040~16777216,62914560~33554432,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~20971520,192937984~4194304,205520896~29360128,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:29.928 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~25165824,41943040~75497472,121634816~20971520,146800640~33554432,192937984~8388608,205520896~33554432,243269632~4194304,251658240~37748736,301989888~8388608,314572800~29360128,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:29.928 INFO:tasks.workunit.client.0.smithi136.stdout:from 13 to HEAD diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,192937984~4194304,205520896~12582912,222298112~12582912,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:29.928 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,192937984~4194304,205520896~12582912,222298112~12582912,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:29.929 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~20971520,41943040~8388608,62914560~54525952,121634816~20971520,146800640~25165824,192937984~8388608,205520896~33554432,243269632~4194304,251658240~37748736,301989888~8388608,314572800~29360128,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:29.930 INFO:tasks.workunit.client.0.smithi136.stdout:from 14 to HEAD diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,192937984~4194304,205520896~4194304,213909504~4194304,222298112~12582912,251658240~16777216,272629760~8388608,301989888~8388608,318767104~25165824,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:29.930 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,192937984~4194304,205520896~4194304,213909504~4194304,222298112~12582912,251658240~16777216,272629760~8388608,301989888~8388608,318767104~25165824,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:29.931 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~20971520,41943040~8388608,62914560~54525952,121634816~20971520,146800640~25165824,192937984~8388608,205520896~4194304,213909504~25165824,251658240~33554432,301989888~8388608,318767104~25165824,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:29.931 INFO:tasks.workunit.client.0.smithi136.stdout:from 15 to HEAD diff [0~4194304,12582912~12582912,29360128~4194304,62914560~20971520,88080384~8388608,100663296~16777216,134217728~4194304,146800640~4194304,159383552~8388608,192937984~4194304,205520896~4194304,213909504~4194304,222298112~12582912,251658240~16777216,272629760~8388608,318767104~25165824,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:29.931 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~12582912,29360128~4194304,62914560~20971520,88080384~8388608,100663296~16777216,134217728~4194304,146800640~4194304,159383552~8388608,192937984~4194304,205520896~4194304,213909504~4194304,222298112~12582912,251658240~16777216,272629760~8388608,318767104~25165824,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:29.931 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~20971520,62914560~54525952,134217728~4194304,146800640~8388608,159383552~12582912,192937984~8388608,205520896~4194304,213909504~25165824,251658240~33554432,318767104~25165824,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:29 smithi165 ceph-mon[105517]: osdmap e1278: 8 total, 8 up, 8 in 2024-08-17T17:51:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:29 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:51:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:29 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:51:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:29 smithi136 ceph-mon[109861]: osdmap e1278: 8 total, 8 up, 8 in 2024-08-17T17:51:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:29 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:51:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:29 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:51:30.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:29 smithi136 ceph-mon[115078]: osdmap e1278: 8 total, 8 up, 8 in 2024-08-17T17:51:30.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:29 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:51:30.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:29 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:51:30.873 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:30.873 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:30.891 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:30.891 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:30.891 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:30.897 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:30.897 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:30.897 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:30.902 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:30.902 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:30.902 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:30.908 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:30.908 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:30.908 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:30.912 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:30.912 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:30.913 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:30.916 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:30.916 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:30.917 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:30.920 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:30.920 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to HEAD diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:30.921 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:30.924 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:30.924 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to HEAD diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:30.925 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:30.928 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:30.928 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to HEAD diff [0~29360128,33554432~33554432,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~8388608,364904448~4194304,373293056~16777216,394264576~25165824] 2024-08-17T17:51:30.928 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~29360128,33554432~33554432,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~8388608,364904448~4194304,373293056~16777216,394264576~25165824] 2024-08-17T17:51:30.930 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~54525952,205520896~138412032,348127232~12582912,364904448~4194304,373293056~46137344] 2024-08-17T17:51:30.931 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to HEAD diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~4194304,301989888~20971520,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:30.931 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~4194304,301989888~20971520,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:30.933 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~130023424,146800640~54525952,205520896~92274688,301989888~41943040,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:30.933 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to HEAD diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~8388608,192937984~8388608,205520896~33554432,243269632~4194304,251658240~16777216,272629760~12582912,289406976~4194304,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:30.933 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~8388608,192937984~8388608,205520896~33554432,243269632~4194304,251658240~16777216,272629760~12582912,289406976~4194304,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:30.935 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~130023424,146800640~54525952,205520896~41943040,251658240~46137344,301989888~8388608,314572800~29360128,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:30.935 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to HEAD diff [0~4194304,12582912~25165824,41943040~16777216,62914560~33554432,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~20971520,192937984~4194304,205520896~29360128,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:30.935 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~25165824,41943040~16777216,62914560~33554432,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~20971520,192937984~4194304,205520896~29360128,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:30.937 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~25165824,41943040~75497472,121634816~20971520,146800640~33554432,192937984~8388608,205520896~33554432,243269632~4194304,251658240~37748736,301989888~8388608,314572800~29360128,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:30.937 INFO:tasks.workunit.client.0.smithi136.stdout:from 13 to HEAD diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,192937984~4194304,205520896~12582912,222298112~12582912,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:30.937 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,192937984~4194304,205520896~12582912,222298112~12582912,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:30.939 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~20971520,41943040~8388608,62914560~54525952,121634816~20971520,146800640~25165824,192937984~8388608,205520896~33554432,243269632~4194304,251658240~37748736,301989888~8388608,314572800~29360128,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:30.939 INFO:tasks.workunit.client.0.smithi136.stdout:from 14 to HEAD diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,192937984~4194304,205520896~4194304,213909504~4194304,222298112~12582912,251658240~16777216,272629760~8388608,301989888~8388608,318767104~25165824,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:30.939 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,192937984~4194304,205520896~4194304,213909504~4194304,222298112~12582912,251658240~16777216,272629760~8388608,301989888~8388608,318767104~25165824,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:30.940 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~20971520,41943040~8388608,62914560~54525952,121634816~20971520,146800640~25165824,192937984~8388608,205520896~4194304,213909504~25165824,251658240~33554432,301989888~8388608,318767104~25165824,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:31.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:30 smithi136 ceph-mon[109861]: osdmap e1279: 8 total, 8 up, 8 in 2024-08-17T17:51:31.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:51:31.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:51:31.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:51:31.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:51:31.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:51:31.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:30 smithi136 ceph-mon[109861]: pgmap v1999: 105 pgs: 105 active+clean; 1.2 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 95 KiB/s rd, 18 MiB/s wr, 162 op/s 2024-08-17T17:51:31.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:30 smithi136 ceph-mon[115078]: osdmap e1279: 8 total, 8 up, 8 in 2024-08-17T17:51:31.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:51:31.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:51:31.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:51:31.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:51:31.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:51:31.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:30 smithi136 ceph-mon[115078]: pgmap v1999: 105 pgs: 105 active+clean; 1.2 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 95 KiB/s rd, 18 MiB/s wr, 162 op/s 2024-08-17T17:51:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:30 smithi165 ceph-mon[105517]: osdmap e1279: 8 total, 8 up, 8 in 2024-08-17T17:51:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:30 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:51:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:30 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:51:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:30 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:51:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:30 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:51:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:30 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:51:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:30 smithi165 ceph-mon[105517]: pgmap v1999: 105 pgs: 105 active+clean; 1.2 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 95 KiB/s rd, 18 MiB/s wr, 162 op/s 2024-08-17T17:51:31.878 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:31.878 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:31.901 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:31.901 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:31.901 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:31.906 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:31.906 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:31.906 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:31.911 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:31.911 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:31.911 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:31.916 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:31.916 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:31.916 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:31.921 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:31.921 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:31.921 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:31.924 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:31.924 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:31.925 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:31.928 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:31.928 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to HEAD diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:31.928 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:31.931 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:31.931 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to HEAD diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:31.931 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:31.934 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:31.934 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to HEAD diff [0~29360128,33554432~33554432,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~8388608,364904448~4194304,373293056~16777216,394264576~25165824] 2024-08-17T17:51:31.934 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~29360128,33554432~33554432,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~8388608,364904448~4194304,373293056~16777216,394264576~25165824] 2024-08-17T17:51:31.937 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~54525952,205520896~138412032,348127232~12582912,364904448~4194304,373293056~46137344] 2024-08-17T17:51:31.937 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to HEAD diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~4194304,301989888~20971520,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:31.937 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~4194304,301989888~20971520,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:31.940 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~130023424,146800640~54525952,205520896~92274688,301989888~41943040,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:31.940 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to HEAD diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~8388608,192937984~8388608,205520896~33554432,243269632~4194304,251658240~16777216,272629760~12582912,289406976~4194304,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:31.940 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~8388608,192937984~8388608,205520896~33554432,243269632~4194304,251658240~16777216,272629760~12582912,289406976~4194304,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:31.942 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~130023424,146800640~54525952,205520896~41943040,251658240~46137344,301989888~8388608,314572800~29360128,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:31.942 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to HEAD diff [0~4194304,12582912~25165824,41943040~16777216,62914560~33554432,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~20971520,192937984~4194304,205520896~29360128,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:31.942 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~25165824,41943040~16777216,62914560~33554432,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~20971520,192937984~4194304,205520896~29360128,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:31.944 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~25165824,41943040~75497472,121634816~20971520,146800640~33554432,192937984~8388608,205520896~33554432,243269632~4194304,251658240~37748736,301989888~8388608,314572800~29360128,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:31.944 INFO:tasks.workunit.client.0.smithi136.stdout:from 13 to HEAD diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,192937984~4194304,205520896~12582912,222298112~12582912,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:31.944 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~12582912,29360128~4194304,41943040~4194304,62914560~20971520,88080384~8388608,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~12582912,192937984~4194304,205520896~12582912,222298112~12582912,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:31.945 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~20971520,41943040~8388608,62914560~54525952,121634816~20971520,146800640~25165824,192937984~8388608,205520896~33554432,243269632~4194304,251658240~37748736,301989888~8388608,314572800~29360128,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:32.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:31 smithi136 ceph-mon[115078]: osdmap e1280: 8 total, 8 up, 8 in 2024-08-17T17:51:32.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:31 smithi136 ceph-mon[109861]: osdmap e1280: 8 total, 8 up, 8 in 2024-08-17T17:51:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:31 smithi165 ceph-mon[105517]: osdmap e1280: 8 total, 8 up, 8 in 2024-08-17T17:51:32.889 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:32.889 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:32.909 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:32.909 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:32.909 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:32.914 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:32.914 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:32.915 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:32.920 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:32.920 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:32.920 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:32.925 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:32.926 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:32.926 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:32.929 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:32.929 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:32.929 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:32.933 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:32.933 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:32.934 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:32.937 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:32.937 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to HEAD diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:32.937 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:32.940 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:32.940 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to HEAD diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:32.940 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:32.943 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:32.943 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to HEAD diff [0~29360128,33554432~33554432,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~8388608,364904448~4194304,373293056~16777216,394264576~25165824] 2024-08-17T17:51:32.943 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~29360128,33554432~33554432,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~8388608,364904448~4194304,373293056~16777216,394264576~25165824] 2024-08-17T17:51:32.946 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~54525952,205520896~138412032,348127232~12582912,364904448~4194304,373293056~46137344] 2024-08-17T17:51:32.946 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to HEAD diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~4194304,301989888~20971520,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:32.946 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~4194304,301989888~20971520,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:32.948 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~130023424,146800640~54525952,205520896~92274688,301989888~41943040,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:32.948 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to HEAD diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~8388608,192937984~8388608,205520896~33554432,243269632~4194304,251658240~16777216,272629760~12582912,289406976~4194304,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:32.948 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~8388608,192937984~8388608,205520896~33554432,243269632~4194304,251658240~16777216,272629760~12582912,289406976~4194304,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:32.950 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~130023424,146800640~54525952,205520896~41943040,251658240~46137344,301989888~8388608,314572800~29360128,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:32.950 INFO:tasks.workunit.client.0.smithi136.stdout:from 12 to HEAD diff [0~4194304,12582912~25165824,41943040~16777216,62914560~33554432,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~20971520,192937984~4194304,205520896~29360128,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:32.950 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~25165824,41943040~16777216,62914560~33554432,100663296~16777216,121634816~4194304,130023424~8388608,146800640~4194304,155189248~20971520,192937984~4194304,205520896~29360128,243269632~4194304,251658240~16777216,272629760~12582912,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:32.951 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~25165824,41943040~75497472,121634816~20971520,146800640~33554432,192937984~8388608,205520896~33554432,243269632~4194304,251658240~37748736,301989888~8388608,314572800~29360128,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:33.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:32 smithi136 ceph-mon[109861]: osdmap e1281: 8 total, 8 up, 8 in 2024-08-17T17:51:33.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:32 smithi136 ceph-mon[115078]: osdmap e1281: 8 total, 8 up, 8 in 2024-08-17T17:51:33.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:32 smithi165 ceph-mon[105517]: osdmap e1281: 8 total, 8 up, 8 in 2024-08-17T17:51:33.896 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:33.896 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:33.920 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:33.920 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:33.920 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:33.924 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:33.924 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:33.924 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:33.928 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:33.928 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:33.929 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:33.933 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:33.933 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:33.934 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:33.937 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:33.937 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:33.938 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:33.941 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:33.941 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:33.941 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:33.944 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:33.944 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to HEAD diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:33.944 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:33.946 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:33.946 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to HEAD diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:33.946 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:33.948 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:33.949 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to HEAD diff [0~29360128,33554432~33554432,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~8388608,364904448~4194304,373293056~16777216,394264576~25165824] 2024-08-17T17:51:33.949 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~29360128,33554432~33554432,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~8388608,364904448~4194304,373293056~16777216,394264576~25165824] 2024-08-17T17:51:33.951 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~54525952,205520896~138412032,348127232~12582912,364904448~4194304,373293056~46137344] 2024-08-17T17:51:33.951 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to HEAD diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~4194304,301989888~20971520,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:33.951 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~4194304,301989888~20971520,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:33.952 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~130023424,146800640~54525952,205520896~92274688,301989888~41943040,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:33.953 INFO:tasks.workunit.client.0.smithi136.stdout:from 11 to HEAD diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~8388608,192937984~8388608,205520896~33554432,243269632~4194304,251658240~16777216,272629760~12582912,289406976~4194304,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:33.953 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~8388608,192937984~8388608,205520896~33554432,243269632~4194304,251658240~16777216,272629760~12582912,289406976~4194304,301989888~8388608,314572800~8388608,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:33.954 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~130023424,146800640~54525952,205520896~41943040,251658240~46137344,301989888~8388608,314572800~29360128,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:34.162 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:33 smithi165 ceph-mon[105517]: pgmap v2002: 105 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 1.2 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 741 KiB/s rd, 3.2 MiB/s wr, 1.39k op/s 2024-08-17T17:51:34.162 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:33 smithi165 ceph-mon[105517]: osdmap e1282: 8 total, 8 up, 8 in 2024-08-17T17:51:34.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:33 smithi136 ceph-mon[109861]: pgmap v2002: 105 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 1.2 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 741 KiB/s rd, 3.2 MiB/s wr, 1.39k op/s 2024-08-17T17:51:34.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:33 smithi136 ceph-mon[109861]: osdmap e1282: 8 total, 8 up, 8 in 2024-08-17T17:51:34.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:33 smithi136 ceph-mon[115078]: pgmap v2002: 105 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 1.2 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 741 KiB/s rd, 3.2 MiB/s wr, 1.39k op/s 2024-08-17T17:51:34.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:33 smithi136 ceph-mon[115078]: osdmap e1282: 8 total, 8 up, 8 in 2024-08-17T17:51:34.229 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:34.229 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:34.251 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:34.251 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:34.252 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:34.257 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:34.257 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:34.257 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:34.261 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:34.261 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:34.261 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:34.265 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:34.265 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:34.265 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:34.269 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:34.269 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:34.269 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:34.272 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:34.272 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:34.273 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:34.275 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:34.275 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to HEAD diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:34.275 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:34.278 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:34.278 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to HEAD diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:34.278 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:34.282 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:34.282 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to HEAD diff [0~29360128,33554432~33554432,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~8388608,364904448~4194304,373293056~16777216,394264576~25165824] 2024-08-17T17:51:34.282 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~29360128,33554432~33554432,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~8388608,364904448~4194304,373293056~16777216,394264576~25165824] 2024-08-17T17:51:34.282 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~54525952,205520896~138412032,348127232~12582912,364904448~4194304,373293056~46137344] 2024-08-17T17:51:34.283 INFO:tasks.workunit.client.0.smithi136.stdout:from 10 to HEAD diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~4194304,301989888~20971520,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:34.283 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~4194304,12582912~16777216,33554432~25165824,62914560~33554432,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~4194304,301989888~20971520,327155712~16777216,348127232~8388608,364904448~4194304,394264576~25165824] 2024-08-17T17:51:34.283 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~8388608,12582912~130023424,146800640~54525952,205520896~92274688,301989888~41943040,348127232~12582912,364904448~4194304,394264576~25165824] 2024-08-17T17:51:35.182 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:34 smithi165 ceph-mon[105517]: osdmap e1283: 8 total, 8 up, 8 in 2024-08-17T17:51:35.182 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:34 smithi165 ceph-mon[105517]: osdmap e1284: 8 total, 8 up, 8 in 2024-08-17T17:51:35.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:34 smithi136 ceph-mon[109861]: osdmap e1283: 8 total, 8 up, 8 in 2024-08-17T17:51:35.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:34 smithi136 ceph-mon[109861]: osdmap e1284: 8 total, 8 up, 8 in 2024-08-17T17:51:35.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:34 smithi136 ceph-mon[115078]: osdmap e1283: 8 total, 8 up, 8 in 2024-08-17T17:51:35.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:34 smithi136 ceph-mon[115078]: osdmap e1284: 8 total, 8 up, 8 in 2024-08-17T17:51:35.239 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:35.239 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:35.263 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:35.264 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:35.264 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:35.268 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:35.268 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:35.268 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:35.271 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:35.272 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:35.272 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:35.275 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:35.275 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:35.275 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:35.277 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:35.277 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:35.278 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:35.280 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:35.280 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:35.280 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:35.282 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:35.282 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to HEAD diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:35.282 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:35.284 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:35.284 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to HEAD diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:35.284 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:35.285 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:35.286 INFO:tasks.workunit.client.0.smithi136.stdout:from 9 to HEAD diff [0~29360128,33554432~33554432,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~8388608,364904448~4194304,373293056~16777216,394264576~25165824] 2024-08-17T17:51:35.286 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~29360128,33554432~33554432,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~20971520,205520896~33554432,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~8388608,364904448~4194304,373293056~16777216,394264576~25165824] 2024-08-17T17:51:35.286 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~54525952,205520896~138412032,348127232~12582912,364904448~4194304,373293056~46137344] 2024-08-17T17:51:36.250 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:36.250 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:36.266 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:36.266 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:36.266 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:36.272 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:36.272 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:36.272 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:36.275 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:36.275 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:36.275 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:36.278 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:36.278 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:36.278 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:36.281 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:36.281 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:36.281 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:36.283 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:36.283 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:36.283 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:36.285 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:36.285 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to HEAD diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:36.285 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:36.287 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:36.287 INFO:tasks.workunit.client.0.smithi136.stdout:from 8 to HEAD diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:36.287 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:36.288 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:36.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:36 smithi165 ceph-mon[105517]: pgmap v2006: 105 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 1.2 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 345 KiB/s rd, 1.4 KiB/s wr, 691 op/s 2024-08-17T17:51:36.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:36 smithi165 ceph-mon[105517]: osdmap e1285: 8 total, 8 up, 8 in 2024-08-17T17:51:36.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:51:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:36 smithi136 ceph-mon[109861]: pgmap v2006: 105 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 1.2 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 345 KiB/s rd, 1.4 KiB/s wr, 691 op/s 2024-08-17T17:51:36.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:36 smithi136 ceph-mon[109861]: osdmap e1285: 8 total, 8 up, 8 in 2024-08-17T17:51:36.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:51:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:36 smithi136 ceph-mon[115078]: pgmap v2006: 105 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 1.2 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 345 KiB/s rd, 1.4 KiB/s wr, 691 op/s 2024-08-17T17:51:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:36 smithi136 ceph-mon[115078]: osdmap e1285: 8 total, 8 up, 8 in 2024-08-17T17:51:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:51:37.259 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:37.259 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:37.274 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:37.274 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:37.274 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:37.280 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:37.280 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:37.280 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:37.284 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:37.284 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:37.284 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:37.288 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:37.288 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:37.288 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:37.290 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:37.290 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:37.290 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:37.292 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:37.292 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:37.292 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:37.294 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:37.294 INFO:tasks.workunit.client.0.smithi136.stdout:from 7 to HEAD diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:37.294 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~25165824,100663296~37748736,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:37.295 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:37 smithi165 ceph-mon[105517]: osdmap e1286: 8 total, 8 up, 8 in 2024-08-17T17:51:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:37 smithi136 ceph-mon[109861]: osdmap e1286: 8 total, 8 up, 8 in 2024-08-17T17:51:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:37 smithi136 ceph-mon[115078]: osdmap e1286: 8 total, 8 up, 8 in 2024-08-17T17:51:37.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:51:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:51:37] "GET /metrics HTTP/1.1" 200 36496 "" "Prometheus/2.51.0" 2024-08-17T17:51:38.255 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:38.256 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:38.281 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:38.281 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:38.281 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:38.286 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:38.287 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:38.287 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:38.290 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:38.290 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:38.290 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:38.292 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:38.292 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:38.293 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:38.294 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:38.294 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:38.294 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:38.296 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:38.296 INFO:tasks.workunit.client.0.smithi136.stdout:from 6 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:38.296 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~12582912,289406976~33554432,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:38.297 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:38.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:38 smithi165 ceph-mon[105517]: pgmap v2009: 105 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 1.1 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 244 KiB/s rd, 6.5 KiB/s wr, 390 op/s 2024-08-17T17:51:38.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:38 smithi165 ceph-mon[105517]: osdmap e1287: 8 total, 8 up, 8 in 2024-08-17T17:51:38.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:38 smithi136 ceph-mon[109861]: pgmap v2009: 105 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 1.1 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 244 KiB/s rd, 6.5 KiB/s wr, 390 op/s 2024-08-17T17:51:38.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:38 smithi136 ceph-mon[109861]: osdmap e1287: 8 total, 8 up, 8 in 2024-08-17T17:51:38.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:38 smithi136 ceph-mon[115078]: pgmap v2009: 105 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 1.1 GiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 244 KiB/s rd, 6.5 KiB/s wr, 390 op/s 2024-08-17T17:51:38.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:38 smithi136 ceph-mon[115078]: osdmap e1287: 8 total, 8 up, 8 in 2024-08-17T17:51:39.238 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:39.238 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:39.262 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:39.262 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:39.262 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:39.270 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:39.270 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:39.270 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:39.272 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:39.272 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:39.272 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:39.274 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:39.274 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:39.274 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:39.276 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:39.276 INFO:tasks.workunit.client.0.smithi136.stdout:from 5 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:39.276 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:39.277 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:39.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:39 smithi165 ceph-mon[105517]: osdmap e1288: 8 total, 8 up, 8 in 2024-08-17T17:51:39.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:39 smithi165 ceph-mon[105517]: osdmap e1289: 8 total, 8 up, 8 in 2024-08-17T17:51:39.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:39 smithi136 ceph-mon[115078]: osdmap e1288: 8 total, 8 up, 8 in 2024-08-17T17:51:39.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:39 smithi136 ceph-mon[115078]: osdmap e1289: 8 total, 8 up, 8 in 2024-08-17T17:51:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:39 smithi136 ceph-mon[109861]: osdmap e1288: 8 total, 8 up, 8 in 2024-08-17T17:51:39.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:39 smithi136 ceph-mon[109861]: osdmap e1289: 8 total, 8 up, 8 in 2024-08-17T17:51:40.288 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:40.288 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:40.305 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:40.305 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:40.305 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:40.308 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:40.308 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:40.308 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:40.310 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:40.311 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:40.311 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:40.312 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:40.312 INFO:tasks.workunit.client.0.smithi136.stdout:from 4 to HEAD diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:40.312 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~4194304,155189248~20971520,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:40.313 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:40.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:40 smithi136 ceph-mon[109861]: pgmap v2012: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 296 KiB/s rd, 7.5 KiB/s wr, 503 op/s 2024-08-17T17:51:40.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:40 smithi136 ceph-mon[115078]: pgmap v2012: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 296 KiB/s rd, 7.5 KiB/s wr, 503 op/s 2024-08-17T17:51:40.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:40 smithi165 ceph-mon[105517]: pgmap v2012: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 296 KiB/s rd, 7.5 KiB/s wr, 503 op/s 2024-08-17T17:51:41.288 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:41.289 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:41.303 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:41.303 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:41.303 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:41.307 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:41.307 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:41.307 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:41.310 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:41.311 INFO:tasks.workunit.client.0.smithi136.stdout:from 3 to HEAD diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:41.311 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~67108864,71303168~67108864,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:41.311 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:41.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:41 smithi136 ceph-mon[109861]: osdmap e1290: 8 total, 8 up, 8 in 2024-08-17T17:51:41.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:41 smithi136 ceph-mon[115078]: osdmap e1290: 8 total, 8 up, 8 in 2024-08-17T17:51:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:41 smithi165 ceph-mon[105517]: osdmap e1290: 8 total, 8 up, 8 in 2024-08-17T17:51:42.296 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:42.296 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:42.304 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:42.304 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:42.304 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:42.307 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:42.307 INFO:tasks.workunit.client.0.smithi136.stdout:from 2 to HEAD diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:42.307 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~58720256,243269632~25165824,272629760~50331648,327155712~16777216,348127232~12582912,364904448~25165824,394264576~25165824] 2024-08-17T17:51:42.311 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~12582912,364904448~54525952] 2024-08-17T17:51:42.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:42 smithi136 ceph-mon[109861]: pgmap v2015: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 72 KiB/s rd, 1.5 KiB/s wr, 144 op/s 2024-08-17T17:51:42.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:42 smithi136 ceph-mon[109861]: osdmap e1291: 8 total, 8 up, 8 in 2024-08-17T17:51:42.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:42 smithi136 ceph-mon[115078]: pgmap v2015: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 72 KiB/s rd, 1.5 KiB/s wr, 144 op/s 2024-08-17T17:51:42.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:42 smithi136 ceph-mon[115078]: osdmap e1291: 8 total, 8 up, 8 in 2024-08-17T17:51:42.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:42 smithi165 ceph-mon[105517]: pgmap v2015: 105 pgs: 105 active+clean; 1.1 GiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 72 KiB/s rd, 1.5 KiB/s wr, 144 op/s 2024-08-17T17:51:42.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:42 smithi165 ceph-mon[105517]: osdmap e1291: 8 total, 8 up, 8 in 2024-08-17T17:51:43.311 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:43.311 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:43.318 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:43.318 INFO:tasks.workunit.client.0.smithi136.stdout:from 1 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:43.318 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~50331648,327155712~16777216,348127232~41943040,394264576~25165824] 2024-08-17T17:51:43.320 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:43.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:43 smithi136 ceph-mon[109861]: osdmap e1292: 8 total, 8 up, 8 in 2024-08-17T17:51:43.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:43 smithi136 ceph-mon[115078]: osdmap e1292: 8 total, 8 up, 8 in 2024-08-17T17:51:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:43 smithi165 ceph-mon[105517]: osdmap e1292: 8 total, 8 up, 8 in 2024-08-17T17:51:44.235 INFO:tasks.workunit.client.0.smithi136.stdout:from 0 to HEAD diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:44.235 INFO:tasks.workunit.client.0.smithi136.stdout: limited diff [0~138412032,146800640~29360128,180355072~88080384,272629760~67108864,348127232~41943040,394264576~25165824] 2024-08-17T17:51:44.246 INFO:tasks.workunit.client.0.smithi136.stdout: actual was [0~142606336,146800640~197132288,348127232~71303168] 2024-08-17T17:51:44.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:44 smithi136 ceph-mon[109861]: pgmap v2018: 105 pgs: 105 active+clean; 949 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 134 KiB/s rd, 5.0 KiB/s wr, 217 op/s 2024-08-17T17:51:44.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:44 smithi136 ceph-mon[109861]: osdmap e1293: 8 total, 8 up, 8 in 2024-08-17T17:51:44.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:44 smithi136 ceph-mon[109861]: osdmap e1294: 8 total, 8 up, 8 in 2024-08-17T17:51:44.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:44 smithi136 ceph-mon[115078]: pgmap v2018: 105 pgs: 105 active+clean; 949 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 134 KiB/s rd, 5.0 KiB/s wr, 217 op/s 2024-08-17T17:51:44.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:44 smithi136 ceph-mon[115078]: osdmap e1293: 8 total, 8 up, 8 in 2024-08-17T17:51:44.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:44 smithi136 ceph-mon[115078]: osdmap e1294: 8 total, 8 up, 8 in 2024-08-17T17:51:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:44 smithi165 ceph-mon[105517]: pgmap v2018: 105 pgs: 105 active+clean; 949 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 134 KiB/s rd, 5.0 KiB/s wr, 217 op/s 2024-08-17T17:51:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:44 smithi165 ceph-mon[105517]: osdmap e1293: 8 total, 8 up, 8 in 2024-08-17T17:51:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:44 smithi165 ceph-mon[105517]: osdmap e1294: 8 total, 8 up, 8 in 2024-08-17T17:51:45.251 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/1.DiffIterateStress (38516 ms) 2024-08-17T17:51:45.251 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/1.DiffIterateParent 2024-08-17T17:51:45.251 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:51:46.412 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/1.DiffIterateParent (1161 ms) 2024-08-17T17:51:46.412 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/1.DiffIterateIgnoreParent 2024-08-17T17:51:46.412 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:51:46.481 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:46 smithi165 ceph-mon[105517]: pgmap v2021: 105 pgs: 105 active+clean; 949 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 134 KiB/s rd, 5.0 KiB/s wr, 217 op/s 2024-08-17T17:51:46.481 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:46 smithi165 ceph-mon[105517]: osdmap e1295: 8 total, 8 up, 8 in 2024-08-17T17:51:46.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:46 smithi136 ceph-mon[109861]: pgmap v2021: 105 pgs: 105 active+clean; 949 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 134 KiB/s rd, 5.0 KiB/s wr, 217 op/s 2024-08-17T17:51:46.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:46 smithi136 ceph-mon[109861]: osdmap e1295: 8 total, 8 up, 8 in 2024-08-17T17:51:46.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:46 smithi136 ceph-mon[115078]: pgmap v2021: 105 pgs: 105 active+clean; 949 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 134 KiB/s rd, 5.0 KiB/s wr, 217 op/s 2024-08-17T17:51:46.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:46 smithi136 ceph-mon[115078]: osdmap e1295: 8 total, 8 up, 8 in 2024-08-17T17:51:47.406 INFO:tasks.workunit.client.0.smithi136.stdout: wrote [0~4194304,8388608~12582912] to clone 2024-08-17T17:51:47.406 INFO:tasks.workunit.client.0.smithi136.stdout: diff was [0~4194304,8388608~12582912] 2024-08-17T17:51:47.410 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/1.DiffIterateIgnoreParent (998 ms) 2024-08-17T17:51:47.410 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/1.DiffIterateCallbackError 2024-08-17T17:51:47.410 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:51:47.427 INFO:tasks.workunit.client.0.smithi136.stdout: wrote [2326638~52913,7574574~8077,9016516~97036,10454940~67658,12016856~36232,14481480~35988,16198321~4648,17492732~25561,17653083~50519,19520931~70164] 2024-08-17T17:51:47.436 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/1.DiffIterateCallbackError (26 ms) 2024-08-17T17:51:47.436 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/1.DiffIterateParentDiscard 2024-08-17T17:51:47.437 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:51:47.597 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:47 smithi136 ceph-mon[109861]: osdmap e1296: 8 total, 8 up, 8 in 2024-08-17T17:51:47.598 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:47 smithi136 ceph-mon[115078]: osdmap e1296: 8 total, 8 up, 8 in 2024-08-17T17:51:47.598 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:51:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:51:47] "GET /metrics HTTP/1.1" 200 36496 "" "Prometheus/2.51.0" 2024-08-17T17:51:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:47 smithi165 ceph-mon[105517]: osdmap e1296: 8 total, 8 up, 8 in 2024-08-17T17:51:48.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:48 smithi136 ceph-mon[109861]: pgmap v2024: 105 pgs: 105 active+clean; 841 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 54 KiB/s rd, 3.0 MiB/s wr, 77 op/s 2024-08-17T17:51:48.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:48 smithi136 ceph-mon[109861]: osdmap e1297: 8 total, 8 up, 8 in 2024-08-17T17:51:48.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:48 smithi136 ceph-mon[115078]: pgmap v2024: 105 pgs: 105 active+clean; 841 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 54 KiB/s rd, 3.0 MiB/s wr, 77 op/s 2024-08-17T17:51:48.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:48 smithi136 ceph-mon[115078]: osdmap e1297: 8 total, 8 up, 8 in 2024-08-17T17:51:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:48 smithi165 ceph-mon[105517]: pgmap v2024: 105 pgs: 105 active+clean; 841 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 54 KiB/s rd, 3.0 MiB/s wr, 77 op/s 2024-08-17T17:51:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:48 smithi165 ceph-mon[105517]: osdmap e1297: 8 total, 8 up, 8 in 2024-08-17T17:51:49.322 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/1.DiffIterateParentDiscard (1886 ms) 2024-08-17T17:51:49.322 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/1.DiffIterateUnalignedSmall 2024-08-17T17:51:49.323 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:51:49.370 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/1.DiffIterateUnalignedSmall (48 ms) 2024-08-17T17:51:49.370 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/1.DiffIterateUnaligned 2024-08-17T17:51:49.370 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:51:49.437 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/1.DiffIterateUnaligned (67 ms) 2024-08-17T17:51:49.437 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/1.DiffIterateTryAcquireLock 2024-08-17T17:51:49.437 INFO:tasks.workunit.client.0.smithi136.stdout:using new format! 2024-08-17T17:51:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:49 smithi136 ceph-mon[109861]: osdmap e1298: 8 total, 8 up, 8 in 2024-08-17T17:51:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:49 smithi136 ceph-mon[109861]: osdmap e1299: 8 total, 8 up, 8 in 2024-08-17T17:51:49.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:49 smithi136 ceph-mon[115078]: osdmap e1298: 8 total, 8 up, 8 in 2024-08-17T17:51:49.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:49 smithi136 ceph-mon[115078]: osdmap e1299: 8 total, 8 up, 8 in 2024-08-17T17:51:49.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:49 smithi165 ceph-mon[105517]: osdmap e1298: 8 total, 8 up, 8 in 2024-08-17T17:51:49.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:49 smithi165 ceph-mon[105517]: osdmap e1299: 8 total, 8 up, 8 in 2024-08-17T17:51:50.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:50 smithi136 ceph-mon[109861]: pgmap v2027: 105 pgs: 105 active+clean; 790 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 6.0 MiB/s wr, 180 op/s 2024-08-17T17:51:50.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:50 smithi136 ceph-mon[115078]: pgmap v2027: 105 pgs: 105 active+clean; 790 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 6.0 MiB/s wr, 180 op/s 2024-08-17T17:51:50.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:50 smithi165 ceph-mon[105517]: pgmap v2027: 105 pgs: 105 active+clean; 790 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 6.0 MiB/s wr, 180 op/s 2024-08-17T17:51:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:51:51.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:51:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:51:52.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:52 smithi136 ceph-mon[109861]: pgmap v2029: 105 pgs: 105 active+clean; 790 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 96 op/s 2024-08-17T17:51:52.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:52 smithi136 ceph-mon[115078]: pgmap v2029: 105 pgs: 105 active+clean; 790 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 96 op/s 2024-08-17T17:51:52.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:52 smithi165 ceph-mon[105517]: pgmap v2029: 105 pgs: 105 active+clean; 790 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 96 op/s 2024-08-17T17:51:54.483 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] DiffIterateTest/1.DiffIterateTryAcquireLock (5046 ms) 2024-08-17T17:51:54.483 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] DiffIterateTest/1.DiffIterateStriping 2024-08-17T17:51:54.483 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/librbd/test_librbd.cc:8958: Skipped 2024-08-17T17:51:54.484 INFO:tasks.workunit.client.0.smithi136.stdout:Skipping due to unmet REQUIRE 2024-08-17T17:51:54.484 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] DiffIterateTest/1.DiffIterateStriping (0 ms) 2024-08-17T17:51:54.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:54 smithi136 ceph-mon[109861]: pgmap v2030: 105 pgs: 105 active+clean; 887 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 3.6 MiB/s rd, 10 MiB/s wr, 267 op/s 2024-08-17T17:51:54.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:54 smithi136 ceph-mon[109861]: osdmap e1300: 8 total, 8 up, 8 in 2024-08-17T17:51:54.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:54 smithi136 ceph-mon[115078]: pgmap v2030: 105 pgs: 105 active+clean; 887 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 3.6 MiB/s rd, 10 MiB/s wr, 267 op/s 2024-08-17T17:51:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:54 smithi136 ceph-mon[115078]: osdmap e1300: 8 total, 8 up, 8 in 2024-08-17T17:51:54.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:54 smithi165 ceph-mon[105517]: pgmap v2030: 105 pgs: 105 active+clean; 887 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 3.6 MiB/s rd, 10 MiB/s wr, 267 op/s 2024-08-17T17:51:54.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:54 smithi165 ceph-mon[105517]: osdmap e1300: 8 total, 8 up, 8 in 2024-08-17T17:51:55.336 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 19 tests from DiffIterateTest/1 (232254 ms total) 2024-08-17T17:51:55.336 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:51:55.336 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] Global test environment tear-down 2024-08-17T17:51:55.336 INFO:tasks.workunit.client.0.smithi136.stdout:[==========] 490 tests from 24 test suites ran. (2435243 ms total) 2024-08-17T17:51:55.336 INFO:tasks.workunit.client.0.smithi136.stdout:[ PASSED ] 444 tests. 2024-08-17T17:51:55.336 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] 46 tests, listed below: 2024-08-17T17:51:55.337 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalStress.DiscardWithPruneWriteOverlap 2024-08-17T17:51:55.337 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.AioDiscardEvent 2024-08-17T17:51:55.337 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.AioWriteEvent 2024-08-17T17:51:55.337 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.AioFlushEvent 2024-08-17T17:51:55.337 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.SnapCreate 2024-08-17T17:51:55.337 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.SnapProtect 2024-08-17T17:51:55.337 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.SnapUnprotect 2024-08-17T17:51:55.337 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.SnapRename 2024-08-17T17:51:55.337 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.SnapRollback 2024-08-17T17:51:55.337 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.SnapRemove 2024-08-17T17:51:55.337 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.Rename 2024-08-17T17:51:55.337 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.Resize 2024-08-17T17:51:55.337 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.Flatten 2024-08-17T17:51:55.338 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.UpdateFeatures 2024-08-17T17:51:55.338 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.MetadataSet 2024-08-17T17:51:55.338 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.MetadataRemove 2024-08-17T17:51:55.338 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalReplay.ObjectPosition 2024-08-17T17:51:55.338 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalEntries.AioWrite 2024-08-17T17:51:55.338 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalEntries.AioDiscard 2024-08-17T17:51:55.338 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalEntries.AioDiscardWithPrune 2024-08-17T17:51:55.338 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestJournalEntries.AioFlush 2024-08-17T17:51:55.338 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestOperations.DisableJournalingCorrupt 2024-08-17T17:51:55.338 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestMigration.MirroringSamePool 2024-08-17T17:51:55.338 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestMigration.MirroringAbort 2024-08-17T17:51:55.338 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestMigration.MirroringOtherPoolDisabled 2024-08-17T17:51:55.338 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestMigration.MirroringOtherPoolEnabled 2024-08-17T17:51:55.339 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestMigration.MirroringPool 2024-08-17T17:51:55.339 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestDeepCopy.NoSnaps_StrippingLargerDstObjSize 2024-08-17T17:51:55.339 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestDeepCopy.Snaps_StrippingLargerDstObjSize 2024-08-17T17:51:55.339 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestDeepCopy.Clone_StrippingLargerDstObjSize 2024-08-17T17:51:55.339 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestDeepCopy.CloneFlatten_StrippingLargerDstObjSize 2024-08-17T17:51:55.339 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestDeepCopy.Stress_StrippingLargerDstObjSize 2024-08-17T17:51:55.339 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestDeepCopy.NoSnaps_StrippingSmallerDstObjSize 2024-08-17T17:51:55.339 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestDeepCopy.Snaps_StrippingSmallerDstObjSize 2024-08-17T17:51:55.339 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestDeepCopy.Clone_StrippingSmallerDstObjSize 2024-08-17T17:51:55.339 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestDeepCopy.CloneFlatten_StrippingSmallerDstObjSize 2024-08-17T17:51:55.339 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestDeepCopy.Stress_StrippingSmallerDstObjSize 2024-08-17T17:51:55.339 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestMirroring.DisableImageMirrorWithPeer 2024-08-17T17:51:55.339 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestMirroring.DisableJournalingWithPeer 2024-08-17T17:51:55.339 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestInternal.DiscardCopyup 2024-08-17T17:51:55.340 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestInternal.TestCoR 2024-08-17T17:51:55.340 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestLibRBD.EncryptionLoadBadStripePattern 2024-08-17T17:51:55.340 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestLibRBD.EnableJournalingViaLockOwner 2024-08-17T17:51:55.340 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] TestLibRBD.ExclusiveLockReadTransition 2024-08-17T17:51:55.340 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] DiffIterateTest/0.DiffIterateStriping 2024-08-17T17:51:55.340 INFO:tasks.workunit.client.0.smithi136.stdout:[ SKIPPED ] DiffIterateTest/1.DiffIterateStriping 2024-08-17T17:51:55.340 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:51:55.340 INFO:tasks.workunit.client.0.smithi136.stdout: YOU HAVE 2 DISABLED TESTS 2024-08-17T17:51:55.340 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T17:51:55.346 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-08-17T17:51:55.346 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-17T17:51:55.419 INFO:tasks.workunit:Stopping ['rbd/test_librbd.sh'] on client.0... 2024-08-17T17:51:55.419 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-08-17T17:51:55.681 DEBUG:teuthology.parallel:result is None 2024-08-17T17:51:55.681 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-08-17T17:51:55.706 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-08-17T17:51:55.706 DEBUG:teuthology.orchestra.run.smithi136:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-08-17T17:51:55.771 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-08-17T17:51:55.771 INFO:teuthology.task.sequential:In sequential, running task print... 2024-08-17T17:51:55.771 INFO:teuthology.task.print:**** done rbd/test_librbd.sh 7-workload 2024-08-17T17:51:55.772 DEBUG:teuthology.parallel:result is None 2024-08-17T17:51:55.772 INFO:teuthology.run_tasks:Running task print... 2024-08-17T17:51:55.781 INFO:teuthology.task.print:**** done upgrading first half of cluster **** 2024-08-17T17:51:55.781 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-08-17T17:51:55.796 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-08-17T17:51:55.796 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-08-17T17:51:55.796 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph mgr dump --format=json 2024-08-17T17:51:56.012 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:51:56.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:56 smithi136 ceph-mon[109861]: pgmap v2032: 105 pgs: 105 active+clean; 887 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 7.7 MiB/s wr, 190 op/s 2024-08-17T17:51:56.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:56 smithi136 ceph-mon[109861]: osdmap e1301: 8 total, 8 up, 8 in 2024-08-17T17:51:56.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:56 smithi136 ceph-mon[115078]: pgmap v2032: 105 pgs: 105 active+clean; 887 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 7.7 MiB/s wr, 190 op/s 2024-08-17T17:51:56.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:56 smithi136 ceph-mon[115078]: osdmap e1301: 8 total, 8 up, 8 in 2024-08-17T17:51:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:56 smithi165 ceph-mon[105517]: pgmap v2032: 105 pgs: 105 active+clean; 887 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s rd, 7.7 MiB/s wr, 190 op/s 2024-08-17T17:51:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:56 smithi165 ceph-mon[105517]: osdmap e1301: 8 total, 8 up, 8 in 2024-08-17T17:51:57.710 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:51:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:51:57] "GET /metrics HTTP/1.1" 200 36497 "" "Prometheus/2.51.0" 2024-08-17T17:51:58.275 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:51:58.644 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:58 smithi136 ceph-mon[109861]: pgmap v2034: 73 pgs: 73 active+clean; 582 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-08-17T17:51:58.644 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:51:58 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1180435021' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-08-17T17:51:58.644 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:58 smithi136 ceph-mon[115078]: pgmap v2034: 73 pgs: 73 active+clean; 582 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-08-17T17:51:58.645 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:51:58 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1180435021' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-08-17T17:51:58.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:58 smithi165 ceph-mon[105517]: pgmap v2034: 73 pgs: 73 active+clean; 582 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-08-17T17:51:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:51:58 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1180435021' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-08-17T17:51:58.990 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":40,"flags":0,"active_gid":34104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":930115007},{"type":"v1","addr":"172.21.15.136:6801","nonce":930115007}]},"active_addr":"172.21.15.136:6801/930115007","active_change":"2024-08-17T17:16:05.797222+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":34107,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.136:8443/","prometheus":"http://172.21.15.136:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":345,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":2097997230}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":4091615359}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":3845461328}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":323475378}]}]} 2024-08-17T17:51:58.994 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-08-17T17:51:58.995 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-08-17T17:51:58.995 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-17T17:51:59.198 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:00.489 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:00 smithi136 ceph-mon[109861]: pgmap v2035: 73 pgs: 73 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:52:00.490 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:00 smithi136 ceph-mon[115078]: pgmap v2035: 73 pgs: 73 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:52:00.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:00 smithi165 ceph-mon[105517]: pgmap v2035: 73 pgs: 73 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:52:01.175 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:52:01.176 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":1301,"fsid":"8270436c-5cba-11ef-bccf-c7b262605968","created":"2024-08-17T17:03:17.682767+0000","modified":"2024-08-17T17:51:55.319388+0000","last_up_change":"2024-08-17T17:24:16.376267+0000","last_in_change":"2024-08-17T17:07:59.960068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":52,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-17T17:06:20.772596+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-17T17:08:47.840093+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-08-17T17:10:04.522913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-08-17T17:10:05.212219+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51eb458d-9f33-4b87-b7f5-8acb076c16ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":653,"up_from":659,"up_thru":1153,"down_at":653,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1285058127},{"type":"v1","addr":"172.21.15.136:6803","nonce":1285058127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1285058127},{"type":"v1","addr":"172.21.15.136:6805","nonce":1285058127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1285058127},{"type":"v1","addr":"172.21.15.136:6809","nonce":1285058127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1285058127},{"type":"v1","addr":"172.21.15.136:6807","nonce":1285058127}]},"public_addr":"172.21.15.136:6803/1285058127","cluster_addr":"172.21.15.136:6805/1285058127","heartbeat_back_addr":"172.21.15.136:6809/1285058127","heartbeat_front_addr":"172.21.15.136:6807/1285058127","state":["exists","up"]},{"osd":1,"uuid":"aa73fa16-8736-478c-b07a-e674c18698ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":662,"up_from":667,"up_thru":1153,"down_at":662,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":717195127},{"type":"v1","addr":"172.21.15.136:6811","nonce":717195127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":717195127},{"type":"v1","addr":"172.21.15.136:6813","nonce":717195127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":717195127},{"type":"v1","addr":"172.21.15.136:6817","nonce":717195127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":717195127},{"type":"v1","addr":"172.21.15.136:6815","nonce":717195127}]},"public_addr":"172.21.15.136:6811/717195127","cluster_addr":"172.21.15.136:6813/717195127","heartbeat_back_addr":"172.21.15.136:6817/717195127","heartbeat_front_addr":"172.21.15.136:6815/717195127","state":["exists","up"]},{"osd":2,"uuid":"4d2612be-7068-48d9-bb0c-992ebccaa091","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":670,"up_from":674,"up_thru":1153,"down_at":670,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":762368443},{"type":"v1","addr":"172.21.15.136:6819","nonce":762368443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":762368443},{"type":"v1","addr":"172.21.15.136:6821","nonce":762368443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":762368443},{"type":"v1","addr":"172.21.15.136:6825","nonce":762368443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":762368443},{"type":"v1","addr":"172.21.15.136:6823","nonce":762368443}]},"public_addr":"172.21.15.136:6819/762368443","cluster_addr":"172.21.15.136:6821/762368443","heartbeat_back_addr":"172.21.15.136:6825/762368443","heartbeat_front_addr":"172.21.15.136:6823/762368443","state":["exists","up"]},{"osd":3,"uuid":"30b8a4f2-be6d-4d5f-8397-06c2e17cfc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":682,"up_from":686,"up_thru":1153,"down_at":682,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":946860336},{"type":"v1","addr":"172.21.15.136:6827","nonce":946860336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":946860336},{"type":"v1","addr":"172.21.15.136:6829","nonce":946860336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":946860336},{"type":"v1","addr":"172.21.15.136:6833","nonce":946860336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":946860336},{"type":"v1","addr":"172.21.15.136:6831","nonce":946860336}]},"public_addr":"172.21.15.136:6827/946860336","cluster_addr":"172.21.15.136:6829/946860336","heartbeat_back_addr":"172.21.15.136:6833/946860336","heartbeat_front_addr":"172.21.15.136:6831/946860336","state":["exists","up"]},{"osd":4,"uuid":"ae4074cf-0069-403f-acc7-74bedab266bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":1153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6801","nonce":842623090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6803","nonce":842623090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6807","nonce":842623090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6805","nonce":842623090}]},"public_addr":"172.21.15.165:6801/842623090","cluster_addr":"172.21.15.165:6803/842623090","heartbeat_back_addr":"172.21.15.165:6807/842623090","heartbeat_front_addr":"172.21.15.165:6805/842623090","state":["exists","up"]},{"osd":5,"uuid":"586c23a7-4831-4b4d-9a06-c1bfcc1c5286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":1153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6809","nonce":3602301990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6811","nonce":3602301990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6815","nonce":3602301990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6813","nonce":3602301990}]},"public_addr":"172.21.15.165:6809/3602301990","cluster_addr":"172.21.15.165:6811/3602301990","heartbeat_back_addr":"172.21.15.165:6815/3602301990","heartbeat_front_addr":"172.21.15.165:6813/3602301990","state":["exists","up"]},{"osd":6,"uuid":"e838c26b-5e4a-41c6-b35b-b438ba64b237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":1153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6817","nonce":1795674907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6819","nonce":1795674907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6823","nonce":1795674907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6821","nonce":1795674907}]},"public_addr":"172.21.15.165:6817/1795674907","cluster_addr":"172.21.15.165:6819/1795674907","heartbeat_back_addr":"172.21.15.165:6823/1795674907","heartbeat_front_addr":"172.21.15.165:6821/1795674907","state":["exists","up"]},{"osd":7,"uuid":"de2fab2e-177a-4eb0-9262-7bbef0d9837f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":1153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6825","nonce":4195921091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6827","nonce":4195921091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6831","nonce":4195921091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6829","nonce":4195921091}]},"public_addr":"172.21.15.165:6825/4195921091","cluster_addr":"172.21.15.165:6827/4195921091","heartbeat_back_addr":"172.21.15.165:6831/4195921091","heartbeat_front_addr":"172.21.15.165:6829/4195921091","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:30.507144+0000","dead_epoch":653},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:54.456854+0000","dead_epoch":662},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:06:17.338746+0000","dead_epoch":669},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:06:42.262947+0000","dead_epoch":682},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:02.983751+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:26.594502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:50.363918+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:08:15.455798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.136:0/3515054633":"2024-08-17T18:28:42.746890+0000","172.21.15.136:0/2190679281":"2024-08-17T18:19:30.927605+0000","172.21.15.136:0/1504550954":"2024-08-17T18:19:29.931500+0000","172.21.15.136:6800/4104535444":"2024-08-18T17:16:05.797025+0000","172.21.15.136:0/3841116499":"2024-08-18T17:16:05.797025+0000","172.21.15.136:0/3340551496":"2024-08-18T17:16:05.797025+0000","172.21.15.136:0/3946052118":"2024-08-18T17:16:05.797025+0000","172.21.15.165:0/3676140772":"2024-08-18T17:13:24.578001+0000","172.21.15.165:6835/7511620":"2024-08-18T17:13:24.578001+0000","172.21.15.136:0/1414082347":"2024-08-18T17:04:10.472088+0000","172.21.15.165:6834/7511620":"2024-08-18T17:13:24.578001+0000","172.21.15.136:6800/803461847":"2024-08-18T17:12:45.319619+0000","172.21.15.136:0/1577563918":"2024-08-18T17:04:10.472088+0000","172.21.15.165:0/300548107":"2024-08-18T17:13:24.578001+0000","172.21.15.165:0/1675517659":"2024-08-18T17:13:24.578001+0000","172.21.15.136:6801/3513268997":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/972617428":"2024-08-18T17:12:45.319619+0000","172.21.15.136:6800/3058734149":"2024-08-18T17:04:10.472088+0000","172.21.15.136:6800/3513268997":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/2887741187":"2024-08-18T17:12:45.319619+0000","172.21.15.136:0/2931810343":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/1381030503":"2024-08-18T17:03:48.417471+0000","172.21.15.136:0/2540800865":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/3933059683":"2024-08-18T17:04:10.472088+0000","172.21.15.165:0/3723748982":"2024-08-18T17:13:24.578001+0000","172.21.15.136:6801/3058734149":"2024-08-18T17:04:10.472088+0000","172.21.15.136:0/3380731497":"2024-08-18T17:03:48.417471+0000","172.21.15.136:0/1905186682":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/1450531984":"2024-08-18T17:08:47.841029+0000","172.21.15.165:0/1694689688":"2024-08-18T17:13:24.578001+0000","172.21.15.136:0/2336022730":"2024-08-18T17:03:48.417471+0000","172.21.15.136:0/2251781402":"2024-08-18T17:16:05.797025+0000","172.21.15.136:0/3166566423":"2024-08-18T17:12:45.319619+0000","172.21.15.136:6801/803461847":"2024-08-18T17:12:45.319619+0000","172.21.15.136:6801/4104535444":"2024-08-18T17:16:05.797025+0000","172.21.15.136:6801/878395364":"2024-08-18T17:03:48.417471+0000","172.21.15.136:0/707907005":"2024-08-18T17:12:45.319619+0000","172.21.15.165:0/2646930649":"2024-08-18T17:13:24.578001+0000","172.21.15.136:6800/878395364":"2024-08-18T17:03:48.417471+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":1}]},{"pool":6,"snaps":[{"begin":2,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":1}]},{"pool":8,"snaps":[{"begin":2,"length":1}]},{"pool":9,"snaps":[{"begin":2,"length":1}]},{"pool":10,"snaps":[{"begin":2,"length":1}]},{"pool":11,"snaps":[{"begin":2,"length":1}]},{"pool":14,"snaps":[{"begin":5,"length":4}]},{"pool":16,"snaps":[{"begin":37,"length":1},{"begin":43,"length":1}]},{"pool":17,"snaps":[{"begin":22,"length":1}]},{"pool":18,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-17T17:52:01.433 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:01 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1961601247' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-17T17:52:01.433 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:01 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1961601247' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-17T17:52:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:01 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1961601247' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-17T17:52:01.814 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-08-17T17:52:01.815 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-17T17:52:02.016 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:02.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:02 smithi136 ceph-mon[109861]: pgmap v2036: 73 pgs: 73 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:52:02.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:02 smithi136 ceph-mon[115078]: pgmap v2036: 73 pgs: 73 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:52:02.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:02 smithi165 ceph-mon[105517]: pgmap v2036: 73 pgs: 73 active+clean; 582 KiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:52:04.036 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:52:04.036 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":1301,"fsid":"8270436c-5cba-11ef-bccf-c7b262605968","created":"2024-08-17T17:03:17.682767+0000","modified":"2024-08-17T17:51:55.319388+0000","last_up_change":"2024-08-17T17:24:16.376267+0000","last_in_change":"2024-08-17T17:07:59.960068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":52,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-17T17:06:20.772596+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-17T17:08:47.840093+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-08-17T17:10:04.522913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-08-17T17:10:05.212219+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51eb458d-9f33-4b87-b7f5-8acb076c16ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":653,"up_from":659,"up_thru":1153,"down_at":653,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1285058127},{"type":"v1","addr":"172.21.15.136:6803","nonce":1285058127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1285058127},{"type":"v1","addr":"172.21.15.136:6805","nonce":1285058127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1285058127},{"type":"v1","addr":"172.21.15.136:6809","nonce":1285058127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1285058127},{"type":"v1","addr":"172.21.15.136:6807","nonce":1285058127}]},"public_addr":"172.21.15.136:6803/1285058127","cluster_addr":"172.21.15.136:6805/1285058127","heartbeat_back_addr":"172.21.15.136:6809/1285058127","heartbeat_front_addr":"172.21.15.136:6807/1285058127","state":["exists","up"]},{"osd":1,"uuid":"aa73fa16-8736-478c-b07a-e674c18698ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":662,"up_from":667,"up_thru":1153,"down_at":662,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":717195127},{"type":"v1","addr":"172.21.15.136:6811","nonce":717195127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":717195127},{"type":"v1","addr":"172.21.15.136:6813","nonce":717195127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":717195127},{"type":"v1","addr":"172.21.15.136:6817","nonce":717195127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":717195127},{"type":"v1","addr":"172.21.15.136:6815","nonce":717195127}]},"public_addr":"172.21.15.136:6811/717195127","cluster_addr":"172.21.15.136:6813/717195127","heartbeat_back_addr":"172.21.15.136:6817/717195127","heartbeat_front_addr":"172.21.15.136:6815/717195127","state":["exists","up"]},{"osd":2,"uuid":"4d2612be-7068-48d9-bb0c-992ebccaa091","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":670,"up_from":674,"up_thru":1153,"down_at":670,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":762368443},{"type":"v1","addr":"172.21.15.136:6819","nonce":762368443}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":762368443},{"type":"v1","addr":"172.21.15.136:6821","nonce":762368443}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":762368443},{"type":"v1","addr":"172.21.15.136:6825","nonce":762368443}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":762368443},{"type":"v1","addr":"172.21.15.136:6823","nonce":762368443}]},"public_addr":"172.21.15.136:6819/762368443","cluster_addr":"172.21.15.136:6821/762368443","heartbeat_back_addr":"172.21.15.136:6825/762368443","heartbeat_front_addr":"172.21.15.136:6823/762368443","state":["exists","up"]},{"osd":3,"uuid":"30b8a4f2-be6d-4d5f-8397-06c2e17cfc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":682,"up_from":686,"up_thru":1153,"down_at":682,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":946860336},{"type":"v1","addr":"172.21.15.136:6827","nonce":946860336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":946860336},{"type":"v1","addr":"172.21.15.136:6829","nonce":946860336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":946860336},{"type":"v1","addr":"172.21.15.136:6833","nonce":946860336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":946860336},{"type":"v1","addr":"172.21.15.136:6831","nonce":946860336}]},"public_addr":"172.21.15.136:6827/946860336","cluster_addr":"172.21.15.136:6829/946860336","heartbeat_back_addr":"172.21.15.136:6833/946860336","heartbeat_front_addr":"172.21.15.136:6831/946860336","state":["exists","up"]},{"osd":4,"uuid":"ae4074cf-0069-403f-acc7-74bedab266bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":1153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6801","nonce":842623090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6803","nonce":842623090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6807","nonce":842623090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6805","nonce":842623090}]},"public_addr":"172.21.15.165:6801/842623090","cluster_addr":"172.21.15.165:6803/842623090","heartbeat_back_addr":"172.21.15.165:6807/842623090","heartbeat_front_addr":"172.21.15.165:6805/842623090","state":["exists","up"]},{"osd":5,"uuid":"586c23a7-4831-4b4d-9a06-c1bfcc1c5286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":1153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6809","nonce":3602301990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6811","nonce":3602301990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6815","nonce":3602301990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6813","nonce":3602301990}]},"public_addr":"172.21.15.165:6809/3602301990","cluster_addr":"172.21.15.165:6811/3602301990","heartbeat_back_addr":"172.21.15.165:6815/3602301990","heartbeat_front_addr":"172.21.15.165:6813/3602301990","state":["exists","up"]},{"osd":6,"uuid":"e838c26b-5e4a-41c6-b35b-b438ba64b237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":1153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6817","nonce":1795674907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6819","nonce":1795674907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6823","nonce":1795674907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":1795674907},{"type":"v1","addr":"172.21.15.165:6821","nonce":1795674907}]},"public_addr":"172.21.15.165:6817/1795674907","cluster_addr":"172.21.15.165:6819/1795674907","heartbeat_back_addr":"172.21.15.165:6823/1795674907","heartbeat_front_addr":"172.21.15.165:6821/1795674907","state":["exists","up"]},{"osd":7,"uuid":"de2fab2e-177a-4eb0-9262-7bbef0d9837f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":1153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6825","nonce":4195921091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6827","nonce":4195921091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6831","nonce":4195921091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6829","nonce":4195921091}]},"public_addr":"172.21.15.165:6825/4195921091","cluster_addr":"172.21.15.165:6827/4195921091","heartbeat_back_addr":"172.21.15.165:6831/4195921091","heartbeat_front_addr":"172.21.15.165:6829/4195921091","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:30.507144+0000","dead_epoch":653},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:54.456854+0000","dead_epoch":662},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:06:17.338746+0000","dead_epoch":669},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:06:42.262947+0000","dead_epoch":682},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:02.983751+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:26.594502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:50.363918+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:08:15.455798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.136:0/3515054633":"2024-08-17T18:28:42.746890+0000","172.21.15.136:0/2190679281":"2024-08-17T18:19:30.927605+0000","172.21.15.136:0/1504550954":"2024-08-17T18:19:29.931500+0000","172.21.15.165:0/3676140772":"2024-08-18T17:13:24.578001+0000","172.21.15.136:6800/878395364":"2024-08-18T17:03:48.417471+0000","172.21.15.165:0/1675517659":"2024-08-18T17:13:24.578001+0000","172.21.15.136:6800/803461847":"2024-08-18T17:12:45.319619+0000","172.21.15.165:6835/7511620":"2024-08-18T17:13:24.578001+0000","172.21.15.165:6834/7511620":"2024-08-18T17:13:24.578001+0000","172.21.15.136:0/1414082347":"2024-08-18T17:04:10.472088+0000","172.21.15.136:6801/803461847":"2024-08-18T17:12:45.319619+0000","172.21.15.136:0/2251781402":"2024-08-18T17:16:05.797025+0000","172.21.15.136:0/3166566423":"2024-08-18T17:12:45.319619+0000","172.21.15.136:0/3946052118":"2024-08-18T17:16:05.797025+0000","172.21.15.136:0/1577563918":"2024-08-18T17:04:10.472088+0000","172.21.15.165:0/300548107":"2024-08-18T17:13:24.578001+0000","172.21.15.136:6801/3513268997":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/972617428":"2024-08-18T17:12:45.319619+0000","172.21.15.136:6800/3058734149":"2024-08-18T17:04:10.472088+0000","172.21.15.136:6800/3513268997":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/2887741187":"2024-08-18T17:12:45.319619+0000","172.21.15.136:0/2931810343":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/1381030503":"2024-08-18T17:03:48.417471+0000","172.21.15.136:0/3841116499":"2024-08-18T17:16:05.797025+0000","172.21.15.136:0/3340551496":"2024-08-18T17:16:05.797025+0000","172.21.15.136:0/1450531984":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/2540800865":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/3933059683":"2024-08-18T17:04:10.472088+0000","172.21.15.165:0/3723748982":"2024-08-18T17:13:24.578001+0000","172.21.15.136:6801/3058734149":"2024-08-18T17:04:10.472088+0000","172.21.15.136:0/3380731497":"2024-08-18T17:03:48.417471+0000","172.21.15.136:0/1905186682":"2024-08-18T17:08:47.841029+0000","172.21.15.136:6800/4104535444":"2024-08-18T17:16:05.797025+0000","172.21.15.136:0/2336022730":"2024-08-18T17:03:48.417471+0000","172.21.15.165:0/1694689688":"2024-08-18T17:13:24.578001+0000","172.21.15.136:6801/878395364":"2024-08-18T17:03:48.417471+0000","172.21.15.165:0/2646930649":"2024-08-18T17:13:24.578001+0000","172.21.15.136:6801/4104535444":"2024-08-18T17:16:05.797025+0000","172.21.15.136:0/707907005":"2024-08-18T17:12:45.319619+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":1}]},{"pool":6,"snaps":[{"begin":2,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":1}]},{"pool":8,"snaps":[{"begin":2,"length":1}]},{"pool":9,"snaps":[{"begin":2,"length":1}]},{"pool":10,"snaps":[{"begin":2,"length":1}]},{"pool":11,"snaps":[{"begin":2,"length":1}]},{"pool":14,"snaps":[{"begin":5,"length":4}]},{"pool":16,"snaps":[{"begin":37,"length":1},{"begin":43,"length":1}]},{"pool":17,"snaps":[{"begin":22,"length":1}]},{"pool":18,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-17T17:52:04.624 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:04 smithi136 ceph-mon[109861]: pgmap v2037: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:04.624 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:04 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1084584333' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-17T17:52:04.625 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:04 smithi136 ceph-mon[115078]: pgmap v2037: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:04.625 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:04 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1084584333' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-17T17:52:04.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-08-17T17:52:04.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-08-17T17:52:04.648 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-08-17T17:52:04.648 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-08-17T17:52:04.648 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-08-17T17:52:04.649 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-08-17T17:52:04.649 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-08-17T17:52:04.650 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-08-17T17:52:04.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:04 smithi165 ceph-mon[105517]: pgmap v2037: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:04.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:04 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1084584333' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-17T17:52:05.154 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:05.154 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:05.157 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:05.157 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:05.160 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:05.161 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:05.164 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:05.172 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:06.671 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:06 smithi136 ceph-mon[109861]: pgmap v2038: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:06.671 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:52:06.672 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:06 smithi136 ceph-mon[115078]: pgmap v2038: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:06.672 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:52:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:06 smithi165 ceph-mon[105517]: pgmap v2038: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:52:07.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:52:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:52:07] "GET /metrics HTTP/1.1" 200 34059 "" "Prometheus/2.51.0" 2024-08-17T17:52:08.591 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:08 smithi136 ceph-mon[109861]: pgmap v2039: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:08.591 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:08 smithi136 ceph-mon[115078]: pgmap v2039: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:08 smithi165 ceph-mon[105517]: pgmap v2039: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:10.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:10 smithi136 ceph-mon[109861]: pgmap v2040: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:10.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:10 smithi136 ceph-mon[115078]: pgmap v2040: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:10 smithi165 ceph-mon[105517]: pgmap v2040: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:12.410 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:12 smithi136 ceph-mon[109861]: pgmap v2041: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:12.410 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:12 smithi136 ceph-mon[115078]: pgmap v2041: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:12 smithi165 ceph-mon[105517]: pgmap v2041: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:14.660 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:14 smithi136 ceph-mon[109861]: pgmap v2042: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:14.660 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:14 smithi136 ceph-mon[115078]: pgmap v2042: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:14 smithi165 ceph-mon[105517]: pgmap v2042: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:15.900 INFO:teuthology.orchestra.run.smithi136.stdout:128849019425 2024-08-17T17:52:15.901 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-08-17T17:52:16.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:16 smithi136 ceph-mon[109861]: pgmap v2043: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:16.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:16 smithi136 ceph-mon[115078]: pgmap v2043: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:16 smithi165 ceph-mon[105517]: pgmap v2043: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:17.474 INFO:teuthology.orchestra.run.smithi136.stdout:2946347565393 2024-08-17T17:52:17.475 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-08-17T17:52:17.716 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:52:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:52:17] "GET /metrics HTTP/1.1" 200 34059 "" "Prometheus/2.51.0" 2024-08-17T17:52:18.399 INFO:teuthology.orchestra.run.smithi136.stdout:197568496152 2024-08-17T17:52:18.399 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-08-17T17:52:18.648 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:18 smithi136 ceph-mon[109861]: pgmap v2044: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:18.648 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:18 smithi136 ceph-mon[115078]: pgmap v2044: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:18.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:18 smithi165 ceph-mon[105517]: pgmap v2044: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:19.659 INFO:teuthology.orchestra.run.smithi136.stdout:2894807957849 2024-08-17T17:52:19.659 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-08-17T17:52:20.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:20 smithi136 ceph-mon[109861]: pgmap v2045: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:20.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:20 smithi136 ceph-mon[115078]: pgmap v2045: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:20.747 INFO:teuthology.orchestra.run.smithi136.stdout:231928234515 2024-08-17T17:52:20.748 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-08-17T17:52:20.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:20 smithi165 ceph-mon[105517]: pgmap v2045: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:21.251 INFO:teuthology.orchestra.run.smithi136.stdout:2864743186784 2024-08-17T17:52:21.252 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-08-17T17:52:21.675 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:52:21.676 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:52:21.696 INFO:teuthology.orchestra.run.smithi136.stdout:2830383448423 2024-08-17T17:52:21.696 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-08-17T17:52:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:52:22.300 INFO:teuthology.orchestra.run.smithi136.stdout:163208757789 2024-08-17T17:52:22.301 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-08-17T17:52:22.318 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:22.350 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:22.351 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:22.581 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:22 smithi136 ceph-mon[109861]: pgmap v2046: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:22.581 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:22 smithi136 ceph-mon[115078]: pgmap v2046: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:22.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:22 smithi165 ceph-mon[105517]: pgmap v2046: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:24.422 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:24.438 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:24.664 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:24 smithi136 ceph-mon[109861]: pgmap v2047: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:24.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:24 smithi136 ceph-mon[115078]: pgmap v2047: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:24.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:24 smithi165 ceph-mon[105517]: pgmap v2047: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:25.129 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:26.557 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:26 smithi136 ceph-mon[109861]: pgmap v2048: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:26.557 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:26 smithi136 ceph-mon[115078]: pgmap v2048: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:26.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:26 smithi165 ceph-mon[105517]: pgmap v2048: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:27.642 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:27 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3368982133' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-08-17T17:52:27.643 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:27 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3368982133' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-08-17T17:52:27.643 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:52:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:52:27] "GET /metrics HTTP/1.1" 200 34056 "" "Prometheus/2.51.0" 2024-08-17T17:52:27.670 INFO:teuthology.orchestra.run.smithi136.stdout:2946347565396 2024-08-17T17:52:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:27 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3368982133' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-08-17T17:52:28.044 INFO:teuthology.orchestra.run.smithi136.stdout:2894807957850 2024-08-17T17:52:28.573 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2894807957849 got 2894807957850 for osd.2 2024-08-17T17:52:28.573 DEBUG:teuthology.parallel:result is None 2024-08-17T17:52:28.665 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:28 smithi136 ceph-mon[109861]: pgmap v2049: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:28.665 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:28 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2530795161' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-08-17T17:52:28.666 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:28 smithi136 ceph-mon[115078]: pgmap v2049: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:28.666 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:28 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2530795161' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-08-17T17:52:28.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:28 smithi165 ceph-mon[105517]: pgmap v2049: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:28.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:28 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2530795161' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-08-17T17:52:29.014 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:29.014 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:29.383 INFO:teuthology.orchestra.run.smithi136.stdout:128849019428 2024-08-17T17:52:30.622 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:30 smithi136 ceph-mon[109861]: pgmap v2050: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:30.622 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:30 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2428352327' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-08-17T17:52:30.622 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:30 smithi136 ceph-mon[115078]: pgmap v2050: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:30.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:30 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2428352327' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-08-17T17:52:30.644 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2946347565393 got 2946347565396 for osd.3 2024-08-17T17:52:30.644 DEBUG:teuthology.parallel:result is None 2024-08-17T17:52:30.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:30 smithi165 ceph-mon[105517]: pgmap v2050: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:30.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:30 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2428352327' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-08-17T17:52:31.280 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849019425 got 128849019428 for osd.4 2024-08-17T17:52:31.280 DEBUG:teuthology.parallel:result is None 2024-08-17T17:52:31.638 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:31 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:52:31.638 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:31 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:52:31.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:31 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:52:32.583 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:32 smithi136 ceph-mon[109861]: pgmap v2051: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:32.583 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:32 smithi136 ceph-mon[115078]: pgmap v2051: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:32.653 INFO:teuthology.orchestra.run.smithi136.stdout:197568496155 2024-08-17T17:52:32.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:32 smithi165 ceph-mon[105517]: pgmap v2051: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:33.470 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:33 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3310536354' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-08-17T17:52:33.471 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:33 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3310536354' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-08-17T17:52:33.494 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568496152 got 197568496155 for osd.6 2024-08-17T17:52:33.494 DEBUG:teuthology.parallel:result is None 2024-08-17T17:52:33.738 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:33 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3310536354' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-08-17T17:52:34.418 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:34 smithi136 ceph-mon[109861]: pgmap v2052: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:34.419 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:34 smithi136 ceph-mon[115078]: pgmap v2052: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:34.650 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:34 smithi165 ceph-mon[105517]: pgmap v2052: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:34.942 INFO:teuthology.orchestra.run.smithi136.stdout:163208757792 2024-08-17T17:52:35.532 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757789 got 163208757792 for osd.5 2024-08-17T17:52:35.532 DEBUG:teuthology.parallel:result is None 2024-08-17T17:52:35.906 INFO:teuthology.orchestra.run.smithi136.stdout:2830383448426 2024-08-17T17:52:36.147 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:35 smithi136 ceph-mon[109861]: pgmap v2053: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:36.147 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:35 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:52:36.148 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:35 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:52:36.148 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:35 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1340626607' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-08-17T17:52:36.148 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:35 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1509103891' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-08-17T17:52:36.148 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:35 smithi136 ceph-mon[115078]: pgmap v2053: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:36.148 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:35 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:52:36.148 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:35 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:52:36.148 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:35 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1340626607' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-08-17T17:52:36.148 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:35 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1509103891' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-08-17T17:52:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:35 smithi165 ceph-mon[105517]: pgmap v2053: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:35 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:52:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:35 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:52:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:35 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1340626607' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-08-17T17:52:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:35 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1509103891' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-08-17T17:52:36.999 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:52:36.999 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:52:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:52:37.469 INFO:teuthology.orchestra.run.smithi136.stdout:231928234518 2024-08-17T17:52:37.548 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:52:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:52:37] "GET /metrics HTTP/1.1" 200 34058 "" "Prometheus/2.51.0" 2024-08-17T17:52:38.034 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2830383448423 got 2830383448426 for osd.0 2024-08-17T17:52:38.035 DEBUG:teuthology.parallel:result is None 2024-08-17T17:52:38.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:37 smithi136 ceph-mon[109861]: pgmap v2054: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:38.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:37 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1690858586' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-08-17T17:52:38.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:37 smithi136 ceph-mon[115078]: pgmap v2054: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:38.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:37 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1690858586' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-08-17T17:52:38.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:37 smithi165 ceph-mon[105517]: pgmap v2054: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:38.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:37 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1690858586' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-08-17T17:52:38.469 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234515 got 231928234518 for osd.7 2024-08-17T17:52:38.469 DEBUG:teuthology.parallel:result is None 2024-08-17T17:52:38.860 INFO:teuthology.orchestra.run.smithi136.stdout:2864743186788 2024-08-17T17:52:39.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:38 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2856875708' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-08-17T17:52:39.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:38 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2856875708' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-08-17T17:52:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:38 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2856875708' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-08-17T17:52:39.585 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2864743186784 got 2864743186788 for osd.1 2024-08-17T17:52:39.585 DEBUG:teuthology.parallel:result is None 2024-08-17T17:52:39.585 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-08-17T17:52:39.585 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:52:40.130 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:40.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:39 smithi136 ceph-mon[109861]: pgmap v2055: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:40.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:39 smithi136 ceph-mon[115078]: pgmap v2055: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:40.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:39 smithi165 ceph-mon[105517]: pgmap v2055: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:41.907 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:52:41.909 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T17:52:42.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:41 smithi136 ceph-mon[109861]: pgmap v2056: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:42.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:41 smithi136 ceph-mon[115078]: pgmap v2056: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:41 smithi165 ceph-mon[105517]: pgmap v2056: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:42.696 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":2056,"stamp":"2024-08-17T17:52:40.567129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595941,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":279,"num_write_kb":2449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":858268,"kb_used_data":131320,"kb_used_omap":295,"kb_used_meta":726552,"kb_avail":749102948,"statfs":{"total":767960285184,"available":767081418752,"internally_reserved":0,"allocated":134471680,"data_stored":83725919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":302302,"internal_metadata":743990050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002667"},"pg_stats":[{"pgid":"3.1f","version":"70'4","reported_seq":1906,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.333168+0000","last_change":"2024-08-17T17:23:42.937520+0000","last_active":"2024-08-17T17:51:55.333168+0000","last_peered":"2024-08-17T17:51:55.333168+0000","last_clean":"2024-08-17T17:51:55.333168+0000","last_became_active":"2024-08-17T17:23:42.937174+0000","last_became_peered":"2024-08-17T17:23:42.937174+0000","last_unstale":"2024-08-17T17:51:55.333168+0000","last_undegraded":"2024-08-17T17:51:55.333168+0000","last_fullsized":"2024-08-17T17:51:55.333168+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:47:56.679612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1271,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.224325+0000","last_change":"2024-08-17T17:23:07.398676+0000","last_active":"2024-08-17T17:51:54.224325+0000","last_peered":"2024-08-17T17:51:54.224325+0000","last_clean":"2024-08-17T17:51:54.224325+0000","last_became_active":"2024-08-17T17:23:07.398369+0000","last_became_peered":"2024-08-17T17:23:07.398369+0000","last_unstale":"2024-08-17T17:51:54.224325+0000","last_undegraded":"2024-08-17T17:51:54.224325+0000","last_fullsized":"2024-08-17T17:51:54.224325+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:55:04.281412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":1890,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.071332+0000","last_change":"2024-08-17T17:24:17.476168+0000","last_active":"2024-08-17T17:51:56.071332+0000","last_peered":"2024-08-17T17:51:56.071332+0000","last_clean":"2024-08-17T17:51:56.071332+0000","last_became_active":"2024-08-17T17:24:17.465845+0000","last_became_peered":"2024-08-17T17:24:17.465845+0000","last_unstale":"2024-08-17T17:51:56.071332+0000","last_undegraded":"2024-08-17T17:51:56.071332+0000","last_fullsized":"2024-08-17T17:51:56.071332+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:15:01.494447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1906,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.071442+0000","last_change":"2024-08-17T17:24:17.461772+0000","last_active":"2024-08-17T17:51:56.071442+0000","last_peered":"2024-08-17T17:51:56.071442+0000","last_clean":"2024-08-17T17:51:56.071442+0000","last_became_active":"2024-08-17T17:24:17.460806+0000","last_became_peered":"2024-08-17T17:24:17.460806+0000","last_unstale":"2024-08-17T17:51:56.071442+0000","last_undegraded":"2024-08-17T17:51:56.071442+0000","last_fullsized":"2024-08-17T17:51:56.071442+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:01:16.453148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":1253,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.532663+0000","last_change":"2024-08-17T17:10:05.965176+0000","last_active":"2024-08-17T17:51:54.532663+0000","last_peered":"2024-08-17T17:51:54.532663+0000","last_clean":"2024-08-17T17:51:54.532663+0000","last_became_active":"2024-08-17T17:10:05.964861+0000","last_became_peered":"2024-08-17T17:10:05.964861+0000","last_unstale":"2024-08-17T17:51:54.532663+0000","last_undegraded":"2024-08-17T17:51:54.532663+0000","last_fullsized":"2024-08-17T17:51:54.532663+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:59:04.277926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1286,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.330974+0000","last_change":"2024-08-17T17:24:17.462527+0000","last_active":"2024-08-17T17:51:55.330974+0000","last_peered":"2024-08-17T17:51:55.330974+0000","last_clean":"2024-08-17T17:51:55.330974+0000","last_became_active":"2024-08-17T17:24:17.462240+0000","last_became_peered":"2024-08-17T17:24:17.462240+0000","last_unstale":"2024-08-17T17:51:55.330974+0000","last_undegraded":"2024-08-17T17:51:55.330974+0000","last_fullsized":"2024-08-17T17:51:55.330974+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:40:38.063717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":1268,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.532713+0000","last_change":"2024-08-17T17:23:07.391038+0000","last_active":"2024-08-17T17:51:54.532713+0000","last_peered":"2024-08-17T17:51:54.532713+0000","last_clean":"2024-08-17T17:51:54.532713+0000","last_became_active":"2024-08-17T17:23:07.390571+0000","last_became_peered":"2024-08-17T17:23:07.390571+0000","last_unstale":"2024-08-17T17:51:54.532713+0000","last_undegraded":"2024-08-17T17:51:54.532713+0000","last_fullsized":"2024-08-17T17:51:54.532713+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:18:41.791740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1285,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.224887+0000","last_change":"2024-08-17T17:24:17.458168+0000","last_active":"2024-08-17T17:51:54.224887+0000","last_peered":"2024-08-17T17:51:54.224887+0000","last_clean":"2024-08-17T17:51:54.224887+0000","last_became_active":"2024-08-17T17:24:17.457847+0000","last_became_peered":"2024-08-17T17:24:17.457847+0000","last_unstale":"2024-08-17T17:51:54.224887+0000","last_undegraded":"2024-08-17T17:51:54.224887+0000","last_fullsized":"2024-08-17T17:51:54.224887+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:28.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":1889,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332046+0000","last_change":"2024-08-17T17:22:32.883756+0000","last_active":"2024-08-17T17:51:55.332046+0000","last_peered":"2024-08-17T17:51:55.332046+0000","last_clean":"2024-08-17T17:51:55.332046+0000","last_became_active":"2024-08-17T17:22:32.874867+0000","last_became_peered":"2024-08-17T17:22:32.874867+0000","last_unstale":"2024-08-17T17:51:55.332046+0000","last_undegraded":"2024-08-17T17:51:55.332046+0000","last_fullsized":"2024-08-17T17:51:55.332046+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:52:27.375231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1908,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.069811+0000","last_change":"2024-08-17T17:24:17.456622+0000","last_active":"2024-08-17T17:51:56.069811+0000","last_peered":"2024-08-17T17:51:56.069811+0000","last_clean":"2024-08-17T17:51:56.069811+0000","last_became_active":"2024-08-17T17:24:17.455396+0000","last_became_peered":"2024-08-17T17:24:17.455396+0000","last_unstale":"2024-08-17T17:51:56.069811+0000","last_undegraded":"2024-08-17T17:51:56.069811+0000","last_fullsized":"2024-08-17T17:51:56.069811+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:09.564331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1288,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.331131+0000","last_change":"2024-08-17T17:23:42.930464+0000","last_active":"2024-08-17T17:51:55.331131+0000","last_peered":"2024-08-17T17:51:55.331131+0000","last_clean":"2024-08-17T17:51:55.331131+0000","last_became_active":"2024-08-17T17:23:42.929949+0000","last_became_peered":"2024-08-17T17:23:42.929949+0000","last_unstale":"2024-08-17T17:51:55.331131+0000","last_undegraded":"2024-08-17T17:51:55.331131+0000","last_fullsized":"2024-08-17T17:51:55.331131+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:48:27.092339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1893,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.070329+0000","last_change":"2024-08-17T17:24:17.452009+0000","last_active":"2024-08-17T17:51:56.070329+0000","last_peered":"2024-08-17T17:51:56.070329+0000","last_clean":"2024-08-17T17:51:56.070329+0000","last_became_active":"2024-08-17T17:24:17.451492+0000","last_became_peered":"2024-08-17T17:24:17.451492+0000","last_unstale":"2024-08-17T17:51:56.070329+0000","last_undegraded":"2024-08-17T17:51:56.070329+0000","last_fullsized":"2024-08-17T17:51:56.070329+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:21:24.699633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1904,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565316+0000","last_change":"2024-08-17T17:24:17.452025+0000","last_active":"2024-08-17T17:51:55.565316+0000","last_peered":"2024-08-17T17:51:55.565316+0000","last_clean":"2024-08-17T17:51:55.565316+0000","last_became_active":"2024-08-17T17:24:17.448029+0000","last_became_peered":"2024-08-17T17:24:17.448029+0000","last_unstale":"2024-08-17T17:51:55.565316+0000","last_undegraded":"2024-08-17T17:51:55.565316+0000","last_fullsized":"2024-08-17T17:51:55.565316+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:45.089346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1899,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332284+0000","last_change":"2024-08-17T17:24:17.469096+0000","last_active":"2024-08-17T17:51:55.332284+0000","last_peered":"2024-08-17T17:51:55.332284+0000","last_clean":"2024-08-17T17:51:55.332284+0000","last_became_active":"2024-08-17T17:24:17.468669+0000","last_became_peered":"2024-08-17T17:24:17.468669+0000","last_unstale":"2024-08-17T17:51:55.332284+0000","last_undegraded":"2024-08-17T17:51:55.332284+0000","last_fullsized":"2024-08-17T17:51:55.332284+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:03:39.427341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1893,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.070083+0000","last_change":"2024-08-17T17:24:17.453946+0000","last_active":"2024-08-17T17:51:56.070083+0000","last_peered":"2024-08-17T17:51:56.070083+0000","last_clean":"2024-08-17T17:51:56.070083+0000","last_became_active":"2024-08-17T17:24:17.445004+0000","last_became_peered":"2024-08-17T17:24:17.445004+0000","last_unstale":"2024-08-17T17:51:56.070083+0000","last_undegraded":"2024-08-17T17:51:56.070083+0000","last_fullsized":"2024-08-17T17:51:56.070083+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:00:19.366629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1270,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.477362+0000","last_change":"2024-08-17T17:23:07.402338+0000","last_active":"2024-08-17T17:51:54.477362+0000","last_peered":"2024-08-17T17:51:54.477362+0000","last_clean":"2024-08-17T17:51:54.477362+0000","last_became_active":"2024-08-17T17:23:07.401996+0000","last_became_peered":"2024-08-17T17:23:07.401996+0000","last_unstale":"2024-08-17T17:51:54.477362+0000","last_undegraded":"2024-08-17T17:51:54.477362+0000","last_fullsized":"2024-08-17T17:51:54.477362+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:39:21.566786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1878,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.070289+0000","last_change":"2024-08-17T17:24:17.442338+0000","last_active":"2024-08-17T17:51:56.070289+0000","last_peered":"2024-08-17T17:51:56.070289+0000","last_clean":"2024-08-17T17:51:56.070289+0000","last_became_active":"2024-08-17T17:24:17.441948+0000","last_became_peered":"2024-08-17T17:24:17.441948+0000","last_unstale":"2024-08-17T17:51:56.070289+0000","last_undegraded":"2024-08-17T17:51:56.070289+0000","last_fullsized":"2024-08-17T17:51:56.070289+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:17.891492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1896,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.333302+0000","last_change":"2024-08-17T17:24:17.459083+0000","last_active":"2024-08-17T17:51:55.333302+0000","last_peered":"2024-08-17T17:51:55.333302+0000","last_clean":"2024-08-17T17:51:55.333302+0000","last_became_active":"2024-08-17T17:24:17.458759+0000","last_became_peered":"2024-08-17T17:24:17.458759+0000","last_unstale":"2024-08-17T17:51:55.333302+0000","last_undegraded":"2024-08-17T17:51:55.333302+0000","last_fullsized":"2024-08-17T17:51:55.333302+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:11:59.301864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1837,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.072348+0000","last_change":"2024-08-17T17:24:17.468086+0000","last_active":"2024-08-17T17:51:56.072348+0000","last_peered":"2024-08-17T17:51:56.072348+0000","last_clean":"2024-08-17T17:51:56.072348+0000","last_became_active":"2024-08-17T17:24:17.467522+0000","last_became_peered":"2024-08-17T17:24:17.467522+0000","last_unstale":"2024-08-17T17:51:56.072348+0000","last_undegraded":"2024-08-17T17:51:56.072348+0000","last_fullsized":"2024-08-17T17:51:56.072348+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:52:55.249649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1267,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.532926+0000","last_change":"2024-08-17T17:23:07.392447+0000","last_active":"2024-08-17T17:51:54.532926+0000","last_peered":"2024-08-17T17:51:54.532926+0000","last_clean":"2024-08-17T17:51:54.532926+0000","last_became_active":"2024-08-17T17:23:07.391765+0000","last_became_peered":"2024-08-17T17:23:07.391765+0000","last_unstale":"2024-08-17T17:51:54.532926+0000","last_undegraded":"2024-08-17T17:51:54.532926+0000","last_fullsized":"2024-08-17T17:51:54.532926+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:59:18.602346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":1268,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.594762+0000","last_change":"2024-08-17T17:24:17.450313+0000","last_active":"2024-08-17T17:51:54.594762+0000","last_peered":"2024-08-17T17:51:54.594762+0000","last_clean":"2024-08-17T17:51:54.594762+0000","last_became_active":"2024-08-17T17:24:17.448803+0000","last_became_peered":"2024-08-17T17:24:17.448803+0000","last_unstale":"2024-08-17T17:51:54.594762+0000","last_undegraded":"2024-08-17T17:51:54.594762+0000","last_fullsized":"2024-08-17T17:51:54.594762+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:33.554836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1903,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.566095+0000","last_change":"2024-08-17T17:23:42.936147+0000","last_active":"2024-08-17T17:51:55.566095+0000","last_peered":"2024-08-17T17:51:55.566095+0000","last_clean":"2024-08-17T17:51:55.566095+0000","last_became_active":"2024-08-17T17:23:42.935859+0000","last_became_peered":"2024-08-17T17:23:42.935859+0000","last_unstale":"2024-08-17T17:51:55.566095+0000","last_undegraded":"2024-08-17T17:51:55.566095+0000","last_fullsized":"2024-08-17T17:51:55.566095+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:47:36.838706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1281,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332334+0000","last_change":"2024-08-17T17:10:05.969131+0000","last_active":"2024-08-17T17:51:55.332334+0000","last_peered":"2024-08-17T17:51:55.332334+0000","last_clean":"2024-08-17T17:51:55.332334+0000","last_became_active":"2024-08-17T17:10:05.968856+0000","last_became_peered":"2024-08-17T17:10:05.968856+0000","last_unstale":"2024-08-17T17:51:55.332334+0000","last_undegraded":"2024-08-17T17:51:55.332334+0000","last_fullsized":"2024-08-17T17:51:55.332334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:25:49.550065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1272,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332594+0000","last_change":"2024-08-17T17:23:07.455919+0000","last_active":"2024-08-17T17:51:55.332594+0000","last_peered":"2024-08-17T17:51:55.332594+0000","last_clean":"2024-08-17T17:51:55.332594+0000","last_became_active":"2024-08-17T17:23:07.455284+0000","last_became_peered":"2024-08-17T17:23:07.455284+0000","last_unstale":"2024-08-17T17:51:55.332594+0000","last_undegraded":"2024-08-17T17:51:55.332594+0000","last_fullsized":"2024-08-17T17:51:55.332594+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:37:46.260775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1259,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.224555+0000","last_change":"2024-08-17T17:10:06.979492+0000","last_active":"2024-08-17T17:51:54.224555+0000","last_peered":"2024-08-17T17:51:54.224555+0000","last_clean":"2024-08-17T17:51:54.224555+0000","last_became_active":"2024-08-17T17:10:06.979158+0000","last_became_peered":"2024-08-17T17:10:06.979158+0000","last_unstale":"2024-08-17T17:51:54.224555+0000","last_undegraded":"2024-08-17T17:51:54.224555+0000","last_fullsized":"2024-08-17T17:51:54.224555+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:12:42.261365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1283,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.526004+0000","last_change":"2024-08-17T17:23:42.935099+0000","last_active":"2024-08-17T17:51:55.526004+0000","last_peered":"2024-08-17T17:51:55.526004+0000","last_clean":"2024-08-17T17:51:55.526004+0000","last_became_active":"2024-08-17T17:23:42.934501+0000","last_became_peered":"2024-08-17T17:23:42.934501+0000","last_unstale":"2024-08-17T17:51:55.526004+0000","last_undegraded":"2024-08-17T17:51:55.526004+0000","last_fullsized":"2024-08-17T17:51:55.526004+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:15:19.095713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":1902,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332593+0000","last_change":"2024-08-17T17:23:07.436763+0000","last_active":"2024-08-17T17:51:55.332593+0000","last_peered":"2024-08-17T17:51:55.332593+0000","last_clean":"2024-08-17T17:51:55.332593+0000","last_became_active":"2024-08-17T17:23:07.434720+0000","last_became_peered":"2024-08-17T17:23:07.434720+0000","last_unstale":"2024-08-17T17:51:55.332593+0000","last_undegraded":"2024-08-17T17:51:55.332593+0000","last_fullsized":"2024-08-17T17:51:55.332593+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:10:20.594115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1873,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.070209+0000","last_change":"2024-08-17T17:24:17.442329+0000","last_active":"2024-08-17T17:51:56.070209+0000","last_peered":"2024-08-17T17:51:56.070209+0000","last_clean":"2024-08-17T17:51:56.070209+0000","last_became_active":"2024-08-17T17:24:17.441666+0000","last_became_peered":"2024-08-17T17:24:17.441666+0000","last_unstale":"2024-08-17T17:51:56.070209+0000","last_undegraded":"2024-08-17T17:51:56.070209+0000","last_fullsized":"2024-08-17T17:51:56.070209+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:43:06.033194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1885,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565043+0000","last_change":"2024-08-17T17:23:07.428296+0000","last_active":"2024-08-17T17:51:55.565043+0000","last_peered":"2024-08-17T17:51:55.565043+0000","last_clean":"2024-08-17T17:51:55.565043+0000","last_became_active":"2024-08-17T17:23:07.428010+0000","last_became_peered":"2024-08-17T17:23:07.428010+0000","last_unstale":"2024-08-17T17:51:55.565043+0000","last_undegraded":"2024-08-17T17:51:55.565043+0000","last_fullsized":"2024-08-17T17:51:55.565043+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:45.459212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":1874,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.072123+0000","last_change":"2024-08-17T17:24:17.481368+0000","last_active":"2024-08-17T17:51:56.072123+0000","last_peered":"2024-08-17T17:51:56.072123+0000","last_clean":"2024-08-17T17:51:56.072123+0000","last_became_active":"2024-08-17T17:24:17.477509+0000","last_became_peered":"2024-08-17T17:24:17.477509+0000","last_unstale":"2024-08-17T17:51:56.072123+0000","last_undegraded":"2024-08-17T17:51:56.072123+0000","last_fullsized":"2024-08-17T17:51:56.072123+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:05:35.558004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1888,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332264+0000","last_change":"2024-08-17T17:22:32.884274+0000","last_active":"2024-08-17T17:51:55.332264+0000","last_peered":"2024-08-17T17:51:55.332264+0000","last_clean":"2024-08-17T17:51:55.332264+0000","last_became_active":"2024-08-17T17:22:32.874667+0000","last_became_peered":"2024-08-17T17:22:32.874667+0000","last_unstale":"2024-08-17T17:51:55.332264+0000","last_undegraded":"2024-08-17T17:51:55.332264+0000","last_fullsized":"2024-08-17T17:51:55.332264+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:22:05.834587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1280,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.594963+0000","last_change":"2024-08-17T17:22:32.870569+0000","last_active":"2024-08-17T17:51:54.594963+0000","last_peered":"2024-08-17T17:51:54.594963+0000","last_clean":"2024-08-17T17:51:54.594963+0000","last_became_active":"2024-08-17T17:22:32.869335+0000","last_became_peered":"2024-08-17T17:22:32.869335+0000","last_unstale":"2024-08-17T17:51:54.594963+0000","last_undegraded":"2024-08-17T17:51:54.594963+0000","last_fullsized":"2024-08-17T17:51:54.594963+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:24.969174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1907,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.566553+0000","last_change":"2024-08-17T17:23:42.935825+0000","last_active":"2024-08-17T17:51:55.566553+0000","last_peered":"2024-08-17T17:51:55.566553+0000","last_clean":"2024-08-17T17:51:55.566553+0000","last_became_active":"2024-08-17T17:23:42.935150+0000","last_became_peered":"2024-08-17T17:23:42.935150+0000","last_unstale":"2024-08-17T17:51:55.566553+0000","last_undegraded":"2024-08-17T17:51:55.566553+0000","last_fullsized":"2024-08-17T17:51:55.566553+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:57:19.577644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1895,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565154+0000","last_change":"2024-08-17T17:23:07.431586+0000","last_active":"2024-08-17T17:51:55.565154+0000","last_peered":"2024-08-17T17:51:55.565154+0000","last_clean":"2024-08-17T17:51:55.565154+0000","last_became_active":"2024-08-17T17:23:07.431301+0000","last_became_peered":"2024-08-17T17:23:07.431301+0000","last_unstale":"2024-08-17T17:51:55.565154+0000","last_undegraded":"2024-08-17T17:51:55.565154+0000","last_fullsized":"2024-08-17T17:51:55.565154+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:28:18.706489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1923,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565354+0000","last_change":"2024-08-17T17:23:07.420550+0000","last_active":"2024-08-17T17:51:55.565354+0000","last_peered":"2024-08-17T17:51:55.565354+0000","last_clean":"2024-08-17T17:51:55.565354+0000","last_became_active":"2024-08-17T17:23:07.420223+0000","last_became_peered":"2024-08-17T17:23:07.420223+0000","last_unstale":"2024-08-17T17:51:55.565354+0000","last_undegraded":"2024-08-17T17:51:55.565354+0000","last_fullsized":"2024-08-17T17:51:55.565354+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:33:46.996371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1285,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.537366+0000","last_change":"2024-08-17T17:24:17.426267+0000","last_active":"2024-08-17T17:51:55.537366+0000","last_peered":"2024-08-17T17:51:55.537366+0000","last_clean":"2024-08-17T17:51:55.537366+0000","last_became_active":"2024-08-17T17:24:17.425925+0000","last_became_peered":"2024-08-17T17:24:17.425925+0000","last_unstale":"2024-08-17T17:51:55.537366+0000","last_undegraded":"2024-08-17T17:51:55.537366+0000","last_fullsized":"2024-08-17T17:51:55.537366+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:24:25.017856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1281,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.537016+0000","last_change":"2024-08-17T17:24:17.474138+0000","last_active":"2024-08-17T17:51:55.537016+0000","last_peered":"2024-08-17T17:51:55.537016+0000","last_clean":"2024-08-17T17:51:55.537016+0000","last_became_active":"2024-08-17T17:24:17.473708+0000","last_became_peered":"2024-08-17T17:24:17.473708+0000","last_unstale":"2024-08-17T17:51:55.537016+0000","last_undegraded":"2024-08-17T17:51:55.537016+0000","last_fullsized":"2024-08-17T17:51:55.537016+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:10:57.768067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1275,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.225254+0000","last_change":"2024-08-17T17:22:32.871533+0000","last_active":"2024-08-17T17:51:54.225254+0000","last_peered":"2024-08-17T17:51:54.225254+0000","last_clean":"2024-08-17T17:51:54.225254+0000","last_became_active":"2024-08-17T17:22:32.868226+0000","last_became_peered":"2024-08-17T17:22:32.868226+0000","last_unstale":"2024-08-17T17:51:54.225254+0000","last_undegraded":"2024-08-17T17:51:54.225254+0000","last_fullsized":"2024-08-17T17:51:54.225254+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:00:02.629455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":1285,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.537019+0000","last_change":"2024-08-17T17:23:07.391021+0000","last_active":"2024-08-17T17:51:55.537019+0000","last_peered":"2024-08-17T17:51:55.537019+0000","last_clean":"2024-08-17T17:51:55.537019+0000","last_became_active":"2024-08-17T17:23:07.390551+0000","last_became_peered":"2024-08-17T17:23:07.390551+0000","last_unstale":"2024-08-17T17:51:55.537019+0000","last_undegraded":"2024-08-17T17:51:55.537019+0000","last_fullsized":"2024-08-17T17:51:55.537019+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:06.239013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001023532,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1906,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565011+0000","last_change":"2024-08-17T17:23:07.406362+0000","last_active":"2024-08-17T17:51:55.565011+0000","last_peered":"2024-08-17T17:51:55.565011+0000","last_clean":"2024-08-17T17:51:55.565011+0000","last_became_active":"2024-08-17T17:23:07.406071+0000","last_became_peered":"2024-08-17T17:23:07.406071+0000","last_unstale":"2024-08-17T17:51:55.565011+0000","last_undegraded":"2024-08-17T17:51:55.565011+0000","last_fullsized":"2024-08-17T17:51:55.565011+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:45:47.383432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1886,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565198+0000","last_change":"2024-08-17T17:23:42.910246+0000","last_active":"2024-08-17T17:51:55.565198+0000","last_peered":"2024-08-17T17:51:55.565198+0000","last_clean":"2024-08-17T17:51:55.565198+0000","last_became_active":"2024-08-17T17:23:42.909929+0000","last_became_peered":"2024-08-17T17:23:42.909929+0000","last_unstale":"2024-08-17T17:51:55.565198+0000","last_undegraded":"2024-08-17T17:51:55.565198+0000","last_fullsized":"2024-08-17T17:51:55.565198+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:14:45.210558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1916,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.070318+0000","last_change":"2024-08-17T17:24:17.438891+0000","last_active":"2024-08-17T17:51:56.070318+0000","last_peered":"2024-08-17T17:51:56.070318+0000","last_clean":"2024-08-17T17:51:56.070318+0000","last_became_active":"2024-08-17T17:24:17.437714+0000","last_became_peered":"2024-08-17T17:24:17.437714+0000","last_unstale":"2024-08-17T17:51:56.070318+0000","last_undegraded":"2024-08-17T17:51:56.070318+0000","last_fullsized":"2024-08-17T17:51:56.070318+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:58.825017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1909,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332494+0000","last_change":"2024-08-17T17:23:42.908634+0000","last_active":"2024-08-17T17:51:55.332494+0000","last_peered":"2024-08-17T17:51:55.332494+0000","last_clean":"2024-08-17T17:51:55.332494+0000","last_became_active":"2024-08-17T17:23:42.907675+0000","last_became_peered":"2024-08-17T17:23:42.907675+0000","last_unstale":"2024-08-17T17:51:55.332494+0000","last_undegraded":"2024-08-17T17:51:55.332494+0000","last_fullsized":"2024-08-17T17:51:55.332494+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:03:37.612222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1899,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.331258+0000","last_change":"2024-08-17T17:24:17.457559+0000","last_active":"2024-08-17T17:51:55.331258+0000","last_peered":"2024-08-17T17:51:55.331258+0000","last_clean":"2024-08-17T17:51:55.331258+0000","last_became_active":"2024-08-17T17:24:17.457262+0000","last_became_peered":"2024-08-17T17:24:17.457262+0000","last_unstale":"2024-08-17T17:51:55.331258+0000","last_undegraded":"2024-08-17T17:51:55.331258+0000","last_fullsized":"2024-08-17T17:51:55.331258+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:53:38.137277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1296,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.594253+0000","last_change":"2024-08-17T17:23:07.450509+0000","last_active":"2024-08-17T17:51:54.594253+0000","last_peered":"2024-08-17T17:51:54.594253+0000","last_clean":"2024-08-17T17:51:54.594253+0000","last_became_active":"2024-08-17T17:23:07.450182+0000","last_became_peered":"2024-08-17T17:23:07.450182+0000","last_unstale":"2024-08-17T17:51:54.594253+0000","last_undegraded":"2024-08-17T17:51:54.594253+0000","last_fullsized":"2024-08-17T17:51:54.594253+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:01:18.656656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1255,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.526524+0000","last_change":"2024-08-17T17:10:06.995023+0000","last_active":"2024-08-17T17:51:55.526524+0000","last_peered":"2024-08-17T17:51:55.526524+0000","last_clean":"2024-08-17T17:51:55.526524+0000","last_became_active":"2024-08-17T17:10:06.994757+0000","last_became_peered":"2024-08-17T17:10:06.994757+0000","last_unstale":"2024-08-17T17:51:55.526524+0000","last_undegraded":"2024-08-17T17:51:55.526524+0000","last_fullsized":"2024-08-17T17:51:55.526524+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:31:52.163518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1839,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.071057+0000","last_change":"2024-08-17T17:24:17.468443+0000","last_active":"2024-08-17T17:51:56.071057+0000","last_peered":"2024-08-17T17:51:56.071057+0000","last_clean":"2024-08-17T17:51:56.071057+0000","last_became_active":"2024-08-17T17:24:17.467394+0000","last_became_peered":"2024-08-17T17:24:17.467394+0000","last_unstale":"2024-08-17T17:51:56.071057+0000","last_undegraded":"2024-08-17T17:51:56.071057+0000","last_fullsized":"2024-08-17T17:51:56.071057+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:13.024253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":1284,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.537434+0000","last_change":"2024-08-17T17:23:42.920552+0000","last_active":"2024-08-17T17:51:55.537434+0000","last_peered":"2024-08-17T17:51:55.537434+0000","last_clean":"2024-08-17T17:51:55.537434+0000","last_became_active":"2024-08-17T17:23:42.915824+0000","last_became_peered":"2024-08-17T17:23:42.915824+0000","last_unstale":"2024-08-17T17:51:55.537434+0000","last_undegraded":"2024-08-17T17:51:55.537434+0000","last_fullsized":"2024-08-17T17:51:55.537434+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:25:18.769504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041737399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":1597,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.556550+0000","last_change":"2024-08-17T17:22:32.867754+0000","last_active":"2024-08-17T17:51:55.556550+0000","last_peered":"2024-08-17T17:51:55.556550+0000","last_clean":"2024-08-17T17:51:55.556550+0000","last_became_active":"2024-08-17T17:22:32.867345+0000","last_became_peered":"2024-08-17T17:22:32.867345+0000","last_unstale":"2024-08-17T17:51:55.556550+0000","last_undegraded":"2024-08-17T17:51:55.556550+0000","last_fullsized":"2024-08-17T17:51:55.556550+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T17:06:21.174471+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:00:42.776037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1885,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565625+0000","last_change":"2024-08-17T17:24:17.455710+0000","last_active":"2024-08-17T17:51:55.565625+0000","last_peered":"2024-08-17T17:51:55.565625+0000","last_clean":"2024-08-17T17:51:55.565625+0000","last_became_active":"2024-08-17T17:24:17.452989+0000","last_became_peered":"2024-08-17T17:24:17.452989+0000","last_unstale":"2024-08-17T17:51:55.565625+0000","last_undegraded":"2024-08-17T17:51:55.565625+0000","last_fullsized":"2024-08-17T17:51:55.565625+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:58:07.894968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1880,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.070034+0000","last_change":"2024-08-17T17:24:17.453639+0000","last_active":"2024-08-17T17:51:56.070034+0000","last_peered":"2024-08-17T17:51:56.070034+0000","last_clean":"2024-08-17T17:51:56.070034+0000","last_became_active":"2024-08-17T17:24:17.443638+0000","last_became_peered":"2024-08-17T17:24:17.443638+0000","last_unstale":"2024-08-17T17:51:56.070034+0000","last_undegraded":"2024-08-17T17:51:56.070034+0000","last_fullsized":"2024-08-17T17:51:56.070034+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:42.107479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1272,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.225392+0000","last_change":"2024-08-17T17:23:07.451866+0000","last_active":"2024-08-17T17:51:54.225392+0000","last_peered":"2024-08-17T17:51:54.225392+0000","last_clean":"2024-08-17T17:51:54.225392+0000","last_became_active":"2024-08-17T17:23:07.451553+0000","last_became_peered":"2024-08-17T17:23:07.451553+0000","last_unstale":"2024-08-17T17:51:54.225392+0000","last_undegraded":"2024-08-17T17:51:54.225392+0000","last_fullsized":"2024-08-17T17:51:54.225392+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:37:23.930651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1274,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.224623+0000","last_change":"2024-08-17T17:23:42.915851+0000","last_active":"2024-08-17T17:51:54.224623+0000","last_peered":"2024-08-17T17:51:54.224623+0000","last_clean":"2024-08-17T17:51:54.224623+0000","last_became_active":"2024-08-17T17:23:42.915552+0000","last_became_peered":"2024-08-17T17:23:42.915552+0000","last_unstale":"2024-08-17T17:51:54.224623+0000","last_undegraded":"2024-08-17T17:51:54.224623+0000","last_fullsized":"2024-08-17T17:51:54.224623+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:45.197876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1287,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.225154+0000","last_change":"2024-08-17T17:23:42.930803+0000","last_active":"2024-08-17T17:51:54.225154+0000","last_peered":"2024-08-17T17:51:54.225154+0000","last_clean":"2024-08-17T17:51:54.225154+0000","last_became_active":"2024-08-17T17:23:42.930458+0000","last_became_peered":"2024-08-17T17:23:42.930458+0000","last_unstale":"2024-08-17T17:51:54.225154+0000","last_undegraded":"2024-08-17T17:51:54.225154+0000","last_fullsized":"2024-08-17T17:51:54.225154+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:40.816706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1270,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.477299+0000","last_change":"2024-08-17T17:23:07.401624+0000","last_active":"2024-08-17T17:51:54.477299+0000","last_peered":"2024-08-17T17:51:54.477299+0000","last_clean":"2024-08-17T17:51:54.477299+0000","last_became_active":"2024-08-17T17:23:07.400222+0000","last_became_peered":"2024-08-17T17:23:07.400222+0000","last_unstale":"2024-08-17T17:51:54.477299+0000","last_undegraded":"2024-08-17T17:51:54.477299+0000","last_fullsized":"2024-08-17T17:51:54.477299+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:07.069989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1272,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.224982+0000","last_change":"2024-08-17T17:24:17.470809+0000","last_active":"2024-08-17T17:51:54.224982+0000","last_peered":"2024-08-17T17:51:54.224982+0000","last_clean":"2024-08-17T17:51:54.224982+0000","last_became_active":"2024-08-17T17:24:17.469753+0000","last_became_peered":"2024-08-17T17:24:17.469753+0000","last_unstale":"2024-08-17T17:51:54.224982+0000","last_undegraded":"2024-08-17T17:51:54.224982+0000","last_fullsized":"2024-08-17T17:51:54.224982+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:00.049425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":1873,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.069486+0000","last_change":"2024-08-17T17:24:17.461258+0000","last_active":"2024-08-17T17:51:56.069486+0000","last_peered":"2024-08-17T17:51:56.069486+0000","last_clean":"2024-08-17T17:51:56.069486+0000","last_became_active":"2024-08-17T17:24:17.460895+0000","last_became_peered":"2024-08-17T17:24:17.460895+0000","last_unstale":"2024-08-17T17:51:56.069486+0000","last_undegraded":"2024-08-17T17:51:56.069486+0000","last_fullsized":"2024-08-17T17:51:56.069486+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:43:24.208259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1912,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.331877+0000","last_change":"2024-08-17T17:23:42.901638+0000","last_active":"2024-08-17T17:51:55.331877+0000","last_peered":"2024-08-17T17:51:55.331877+0000","last_clean":"2024-08-17T17:51:55.331877+0000","last_became_active":"2024-08-17T17:23:42.901335+0000","last_became_peered":"2024-08-17T17:23:42.901335+0000","last_unstale":"2024-08-17T17:51:55.331877+0000","last_undegraded":"2024-08-17T17:51:55.331877+0000","last_fullsized":"2024-08-17T17:51:55.331877+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:25:35.332839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1268,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.532827+0000","last_change":"2024-08-17T17:24:17.473913+0000","last_active":"2024-08-17T17:51:54.532827+0000","last_peered":"2024-08-17T17:51:54.532827+0000","last_clean":"2024-08-17T17:51:54.532827+0000","last_became_active":"2024-08-17T17:24:17.473195+0000","last_became_peered":"2024-08-17T17:24:17.473195+0000","last_unstale":"2024-08-17T17:51:54.532827+0000","last_undegraded":"2024-08-17T17:51:54.532827+0000","last_fullsized":"2024-08-17T17:51:54.532827+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:12.998033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1269,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.477272+0000","last_change":"2024-08-17T17:23:07.402010+0000","last_active":"2024-08-17T17:51:54.477272+0000","last_peered":"2024-08-17T17:51:54.477272+0000","last_clean":"2024-08-17T17:51:54.477272+0000","last_became_active":"2024-08-17T17:23:07.400849+0000","last_became_peered":"2024-08-17T17:23:07.400849+0000","last_unstale":"2024-08-17T17:51:54.477272+0000","last_undegraded":"2024-08-17T17:51:54.477272+0000","last_fullsized":"2024-08-17T17:51:54.477272+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:16:02.382257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":1252,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.594877+0000","last_change":"2024-08-17T17:10:05.973491+0000","last_active":"2024-08-17T17:51:54.594877+0000","last_peered":"2024-08-17T17:51:54.594877+0000","last_clean":"2024-08-17T17:51:54.594877+0000","last_became_active":"2024-08-17T17:10:05.973228+0000","last_became_peered":"2024-08-17T17:10:05.973228+0000","last_unstale":"2024-08-17T17:51:54.594877+0000","last_undegraded":"2024-08-17T17:51:54.594877+0000","last_fullsized":"2024-08-17T17:51:54.594877+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:54.310135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1288,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.224430+0000","last_change":"2024-08-17T17:24:17.460203+0000","last_active":"2024-08-17T17:51:54.224430+0000","last_peered":"2024-08-17T17:51:54.224430+0000","last_clean":"2024-08-17T17:51:54.224430+0000","last_became_active":"2024-08-17T17:24:17.459682+0000","last_became_peered":"2024-08-17T17:24:17.459682+0000","last_unstale":"2024-08-17T17:51:54.224430+0000","last_undegraded":"2024-08-17T17:51:54.224430+0000","last_fullsized":"2024-08-17T17:51:54.224430+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:58.703700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":1268,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.594587+0000","last_change":"2024-08-17T17:23:07.454042+0000","last_active":"2024-08-17T17:51:54.594587+0000","last_peered":"2024-08-17T17:51:54.594587+0000","last_clean":"2024-08-17T17:51:54.594587+0000","last_became_active":"2024-08-17T17:23:07.453712+0000","last_became_peered":"2024-08-17T17:23:07.453712+0000","last_unstale":"2024-08-17T17:51:54.594587+0000","last_undegraded":"2024-08-17T17:51:54.594587+0000","last_fullsized":"2024-08-17T17:51:54.594587+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:03:07.643456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1253,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.532272+0000","last_change":"2024-08-17T17:10:06.994799+0000","last_active":"2024-08-17T17:51:54.532272+0000","last_peered":"2024-08-17T17:51:54.532272+0000","last_clean":"2024-08-17T17:51:54.532272+0000","last_became_active":"2024-08-17T17:10:06.994150+0000","last_became_peered":"2024-08-17T17:10:06.994150+0000","last_unstale":"2024-08-17T17:51:54.532272+0000","last_undegraded":"2024-08-17T17:51:54.532272+0000","last_fullsized":"2024-08-17T17:51:54.532272+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:50:47.293091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":1266,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.594586+0000","last_change":"2024-08-17T17:22:32.873334+0000","last_active":"2024-08-17T17:51:54.594586+0000","last_peered":"2024-08-17T17:51:54.594586+0000","last_clean":"2024-08-17T17:51:54.594586+0000","last_became_active":"2024-08-17T17:22:32.870147+0000","last_became_peered":"2024-08-17T17:22:32.870147+0000","last_unstale":"2024-08-17T17:51:54.594586+0000","last_undegraded":"2024-08-17T17:51:54.594586+0000","last_fullsized":"2024-08-17T17:51:54.594586+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:18:03.750592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1277,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.524676+0000","last_change":"2024-08-17T17:22:32.847989+0000","last_active":"2024-08-17T17:51:55.524676+0000","last_peered":"2024-08-17T17:51:55.524676+0000","last_clean":"2024-08-17T17:51:55.524676+0000","last_became_active":"2024-08-17T17:22:32.846593+0000","last_became_peered":"2024-08-17T17:22:32.846593+0000","last_unstale":"2024-08-17T17:51:55.524676+0000","last_undegraded":"2024-08-17T17:51:55.524676+0000","last_fullsized":"2024-08-17T17:51:55.524676+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:33:50.370124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1877,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.071934+0000","last_change":"2024-08-17T17:24:17.451528+0000","last_active":"2024-08-17T17:51:56.071934+0000","last_peered":"2024-08-17T17:51:56.071934+0000","last_clean":"2024-08-17T17:51:56.071934+0000","last_became_active":"2024-08-17T17:24:17.451246+0000","last_became_peered":"2024-08-17T17:24:17.451246+0000","last_unstale":"2024-08-17T17:51:56.071934+0000","last_undegraded":"2024-08-17T17:51:56.071934+0000","last_fullsized":"2024-08-17T17:51:56.071934+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:38:18.176559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1251,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.557291+0000","last_change":"2024-08-17T17:10:05.975692+0000","last_active":"2024-08-17T17:51:55.557291+0000","last_peered":"2024-08-17T17:51:55.557291+0000","last_clean":"2024-08-17T17:51:55.557291+0000","last_became_active":"2024-08-17T17:10:05.975057+0000","last_became_peered":"2024-08-17T17:10:05.975057+0000","last_unstale":"2024-08-17T17:51:55.557291+0000","last_undegraded":"2024-08-17T17:51:55.557291+0000","last_fullsized":"2024-08-17T17:51:55.557291+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:44:53.106104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1896,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.331881+0000","last_change":"2024-08-17T17:24:17.460371+0000","last_active":"2024-08-17T17:51:55.331881+0000","last_peered":"2024-08-17T17:51:55.331881+0000","last_clean":"2024-08-17T17:51:55.331881+0000","last_became_active":"2024-08-17T17:24:17.460074+0000","last_became_peered":"2024-08-17T17:24:17.460074+0000","last_unstale":"2024-08-17T17:51:55.331881+0000","last_undegraded":"2024-08-17T17:51:55.331881+0000","last_fullsized":"2024-08-17T17:51:55.331881+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:08:48.820168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1896,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332180+0000","last_change":"2024-08-17T17:24:17.457942+0000","last_active":"2024-08-17T17:51:55.332180+0000","last_peered":"2024-08-17T17:51:55.332180+0000","last_clean":"2024-08-17T17:51:55.332180+0000","last_became_active":"2024-08-17T17:24:17.456475+0000","last_became_peered":"2024-08-17T17:24:17.456475+0000","last_unstale":"2024-08-17T17:51:55.332180+0000","last_undegraded":"2024-08-17T17:51:55.332180+0000","last_fullsized":"2024-08-17T17:51:55.332180+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:12.325657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1266,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.532706+0000","last_change":"2024-08-17T17:24:17.451287+0000","last_active":"2024-08-17T17:51:54.532706+0000","last_peered":"2024-08-17T17:51:54.532706+0000","last_clean":"2024-08-17T17:51:54.532706+0000","last_became_active":"2024-08-17T17:24:17.450951+0000","last_became_peered":"2024-08-17T17:24:17.450951+0000","last_unstale":"2024-08-17T17:51:54.532706+0000","last_undegraded":"2024-08-17T17:51:54.532706+0000","last_fullsized":"2024-08-17T17:51:54.532706+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:13:37.590108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":1268,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.556246+0000","last_change":"2024-08-17T17:23:42.907970+0000","last_active":"2024-08-17T17:51:55.556246+0000","last_peered":"2024-08-17T17:51:55.556246+0000","last_clean":"2024-08-17T17:51:55.556246+0000","last_became_active":"2024-08-17T17:23:42.907595+0000","last_became_peered":"2024-08-17T17:23:42.907595+0000","last_unstale":"2024-08-17T17:51:55.556246+0000","last_undegraded":"2024-08-17T17:51:55.556246+0000","last_fullsized":"2024-08-17T17:51:55.556246+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:11:46.130004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1878,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.072237+0000","last_change":"2024-08-17T17:24:17.486591+0000","last_active":"2024-08-17T17:51:56.072237+0000","last_peered":"2024-08-17T17:51:56.072237+0000","last_clean":"2024-08-17T17:51:56.072237+0000","last_became_active":"2024-08-17T17:24:17.486320+0000","last_became_peered":"2024-08-17T17:24:17.486320+0000","last_unstale":"2024-08-17T17:51:56.072237+0000","last_undegraded":"2024-08-17T17:51:56.072237+0000","last_fullsized":"2024-08-17T17:51:56.072237+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:03.249803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234519,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":120032,"kb_used_data":16792,"kb_used_omap":1,"kb_used_meta":103230,"kb_avail":93625120,"statfs":{"total":95995035648,"available":95872122880,"internally_reserved":0,"allocated":17195008,"data_stored":10837596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":105707979},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496157,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":146268,"kb_used_data":16784,"kb_used_omap":1,"kb_used_meta":129470,"kb_avail":93598884,"statfs":{"total":95995035648,"available":95845257216,"internally_reserved":0,"allocated":17186816,"data_stored":10837469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":132577735},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565398,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":92740,"kb_used_data":16188,"kb_used_omap":72,"kb_used_meta":76471,"kb_avail":93652412,"statfs":{"total":95995035648,"available":95900069888,"internally_reserved":0,"allocated":16576512,"data_stored":10242358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":78306798},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":674,"seq":2894807957853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70540,"kb_used_data":16192,"kb_used_omap":74,"kb_used_meta":54261,"kb_avail":93674612,"statfs":{"total":95995035648,"available":95922802688,"internally_reserved":0,"allocated":16580608,"data_stored":10242884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76116,"internal_metadata":55563948},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743186788,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70472,"kb_used_data":16188,"kb_used_omap":66,"kb_used_meta":54205,"kb_avail":93674680,"statfs":{"total":95995035648,"available":95922872320,"internally_reserved":0,"allocated":16576512,"data_stored":10242287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":55506603},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448427,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88916,"kb_used_data":16772,"kb_used_omap":75,"kb_used_meta":72052,"kb_avail":93656236,"statfs":{"total":95995035648,"available":95903985664,"internally_reserved":0,"allocated":17174528,"data_stored":10833240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":73781439},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757793,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":127700,"kb_used_data":16196,"kb_used_omap":1,"kb_used_meta":111486,"kb_avail":93617452,"statfs":{"total":95995035648,"available":95864270848,"internally_reserved":0,"allocated":16584704,"data_stored":10242927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114162118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019430,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":141600,"kb_used_data":16208,"kb_used_omap":1,"kb_used_meta":125374,"kb_avail":93603552,"statfs":{"total":95995035648,"available":95850037248,"internally_reserved":0,"allocated":16596992,"data_stored":10247158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128383430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:52:42.700 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:52:43.124 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:43.247 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:42 smithi136 ceph-mon[115078]: from='client.44927 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:52:43.247 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:42 smithi136 ceph-mon[109861]: from='client.44927 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:52:43.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:42 smithi165 ceph-mon[105517]: from='client.44927 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:52:44.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:43 smithi136 ceph-mon[109861]: pgmap v2057: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:44.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:43 smithi136 ceph-mon[115078]: pgmap v2057: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:44.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:43 smithi165 ceph-mon[105517]: pgmap v2057: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:45.717 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:52:45.718 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T17:52:46.057 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:45 smithi136 ceph-mon[109861]: pgmap v2058: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:46.057 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:45 smithi136 ceph-mon[115078]: pgmap v2058: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:46.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:45 smithi165 ceph-mon[105517]: pgmap v2058: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:46.698 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":2058,"stamp":"2024-08-17T17:52:44.567908+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595941,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":279,"num_write_kb":2449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":858268,"kb_used_data":131320,"kb_used_omap":295,"kb_used_meta":726552,"kb_avail":749102948,"statfs":{"total":767960285184,"available":767081418752,"internally_reserved":0,"allocated":134471680,"data_stored":83725919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":302302,"internal_metadata":743990050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002695"},"pg_stats":[{"pgid":"3.1f","version":"70'4","reported_seq":1906,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.333168+0000","last_change":"2024-08-17T17:23:42.937520+0000","last_active":"2024-08-17T17:51:55.333168+0000","last_peered":"2024-08-17T17:51:55.333168+0000","last_clean":"2024-08-17T17:51:55.333168+0000","last_became_active":"2024-08-17T17:23:42.937174+0000","last_became_peered":"2024-08-17T17:23:42.937174+0000","last_unstale":"2024-08-17T17:51:55.333168+0000","last_undegraded":"2024-08-17T17:51:55.333168+0000","last_fullsized":"2024-08-17T17:51:55.333168+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:47:56.679612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1271,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.224325+0000","last_change":"2024-08-17T17:23:07.398676+0000","last_active":"2024-08-17T17:51:54.224325+0000","last_peered":"2024-08-17T17:51:54.224325+0000","last_clean":"2024-08-17T17:51:54.224325+0000","last_became_active":"2024-08-17T17:23:07.398369+0000","last_became_peered":"2024-08-17T17:23:07.398369+0000","last_unstale":"2024-08-17T17:51:54.224325+0000","last_undegraded":"2024-08-17T17:51:54.224325+0000","last_fullsized":"2024-08-17T17:51:54.224325+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:55:04.281412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":1890,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.071332+0000","last_change":"2024-08-17T17:24:17.476168+0000","last_active":"2024-08-17T17:51:56.071332+0000","last_peered":"2024-08-17T17:51:56.071332+0000","last_clean":"2024-08-17T17:51:56.071332+0000","last_became_active":"2024-08-17T17:24:17.465845+0000","last_became_peered":"2024-08-17T17:24:17.465845+0000","last_unstale":"2024-08-17T17:51:56.071332+0000","last_undegraded":"2024-08-17T17:51:56.071332+0000","last_fullsized":"2024-08-17T17:51:56.071332+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:15:01.494447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1906,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.071442+0000","last_change":"2024-08-17T17:24:17.461772+0000","last_active":"2024-08-17T17:51:56.071442+0000","last_peered":"2024-08-17T17:51:56.071442+0000","last_clean":"2024-08-17T17:51:56.071442+0000","last_became_active":"2024-08-17T17:24:17.460806+0000","last_became_peered":"2024-08-17T17:24:17.460806+0000","last_unstale":"2024-08-17T17:51:56.071442+0000","last_undegraded":"2024-08-17T17:51:56.071442+0000","last_fullsized":"2024-08-17T17:51:56.071442+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:01:16.453148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":1254,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.536641+0000","last_change":"2024-08-17T17:10:05.965176+0000","last_active":"2024-08-17T17:51:55.536641+0000","last_peered":"2024-08-17T17:51:55.536641+0000","last_clean":"2024-08-17T17:51:55.536641+0000","last_became_active":"2024-08-17T17:10:05.964861+0000","last_became_peered":"2024-08-17T17:10:05.964861+0000","last_unstale":"2024-08-17T17:51:55.536641+0000","last_undegraded":"2024-08-17T17:51:55.536641+0000","last_fullsized":"2024-08-17T17:51:55.536641+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:59:04.277926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1286,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.330974+0000","last_change":"2024-08-17T17:24:17.462527+0000","last_active":"2024-08-17T17:51:55.330974+0000","last_peered":"2024-08-17T17:51:55.330974+0000","last_clean":"2024-08-17T17:51:55.330974+0000","last_became_active":"2024-08-17T17:24:17.462240+0000","last_became_peered":"2024-08-17T17:24:17.462240+0000","last_unstale":"2024-08-17T17:51:55.330974+0000","last_undegraded":"2024-08-17T17:51:55.330974+0000","last_fullsized":"2024-08-17T17:51:55.330974+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:40:38.063717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":1269,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.536271+0000","last_change":"2024-08-17T17:23:07.391038+0000","last_active":"2024-08-17T17:51:55.536271+0000","last_peered":"2024-08-17T17:51:55.536271+0000","last_clean":"2024-08-17T17:51:55.536271+0000","last_became_active":"2024-08-17T17:23:07.390571+0000","last_became_peered":"2024-08-17T17:23:07.390571+0000","last_unstale":"2024-08-17T17:51:55.536271+0000","last_undegraded":"2024-08-17T17:51:55.536271+0000","last_fullsized":"2024-08-17T17:51:55.536271+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:18:41.791740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1286,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332513+0000","last_change":"2024-08-17T17:24:17.458168+0000","last_active":"2024-08-17T17:51:55.332513+0000","last_peered":"2024-08-17T17:51:55.332513+0000","last_clean":"2024-08-17T17:51:55.332513+0000","last_became_active":"2024-08-17T17:24:17.457847+0000","last_became_peered":"2024-08-17T17:24:17.457847+0000","last_unstale":"2024-08-17T17:51:55.332513+0000","last_undegraded":"2024-08-17T17:51:55.332513+0000","last_fullsized":"2024-08-17T17:51:55.332513+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:28.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":1889,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332046+0000","last_change":"2024-08-17T17:22:32.883756+0000","last_active":"2024-08-17T17:51:55.332046+0000","last_peered":"2024-08-17T17:51:55.332046+0000","last_clean":"2024-08-17T17:51:55.332046+0000","last_became_active":"2024-08-17T17:22:32.874867+0000","last_became_peered":"2024-08-17T17:22:32.874867+0000","last_unstale":"2024-08-17T17:51:55.332046+0000","last_undegraded":"2024-08-17T17:51:55.332046+0000","last_fullsized":"2024-08-17T17:51:55.332046+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:52:27.375231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1908,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.069811+0000","last_change":"2024-08-17T17:24:17.456622+0000","last_active":"2024-08-17T17:51:56.069811+0000","last_peered":"2024-08-17T17:51:56.069811+0000","last_clean":"2024-08-17T17:51:56.069811+0000","last_became_active":"2024-08-17T17:24:17.455396+0000","last_became_peered":"2024-08-17T17:24:17.455396+0000","last_unstale":"2024-08-17T17:51:56.069811+0000","last_undegraded":"2024-08-17T17:51:56.069811+0000","last_fullsized":"2024-08-17T17:51:56.069811+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:09.564331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1288,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.331131+0000","last_change":"2024-08-17T17:23:42.930464+0000","last_active":"2024-08-17T17:51:55.331131+0000","last_peered":"2024-08-17T17:51:55.331131+0000","last_clean":"2024-08-17T17:51:55.331131+0000","last_became_active":"2024-08-17T17:23:42.929949+0000","last_became_peered":"2024-08-17T17:23:42.929949+0000","last_unstale":"2024-08-17T17:51:55.331131+0000","last_undegraded":"2024-08-17T17:51:55.331131+0000","last_fullsized":"2024-08-17T17:51:55.331131+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:48:27.092339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1893,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.070329+0000","last_change":"2024-08-17T17:24:17.452009+0000","last_active":"2024-08-17T17:51:56.070329+0000","last_peered":"2024-08-17T17:51:56.070329+0000","last_clean":"2024-08-17T17:51:56.070329+0000","last_became_active":"2024-08-17T17:24:17.451492+0000","last_became_peered":"2024-08-17T17:24:17.451492+0000","last_unstale":"2024-08-17T17:51:56.070329+0000","last_undegraded":"2024-08-17T17:51:56.070329+0000","last_fullsized":"2024-08-17T17:51:56.070329+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:21:24.699633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1904,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565316+0000","last_change":"2024-08-17T17:24:17.452025+0000","last_active":"2024-08-17T17:51:55.565316+0000","last_peered":"2024-08-17T17:51:55.565316+0000","last_clean":"2024-08-17T17:51:55.565316+0000","last_became_active":"2024-08-17T17:24:17.448029+0000","last_became_peered":"2024-08-17T17:24:17.448029+0000","last_unstale":"2024-08-17T17:51:55.565316+0000","last_undegraded":"2024-08-17T17:51:55.565316+0000","last_fullsized":"2024-08-17T17:51:55.565316+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:45.089346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1900,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332332+0000","last_change":"2024-08-17T17:24:17.469096+0000","last_active":"2024-08-17T17:51:55.332332+0000","last_peered":"2024-08-17T17:51:55.332332+0000","last_clean":"2024-08-17T17:51:55.332332+0000","last_became_active":"2024-08-17T17:24:17.468669+0000","last_became_peered":"2024-08-17T17:24:17.468669+0000","last_unstale":"2024-08-17T17:51:55.332332+0000","last_undegraded":"2024-08-17T17:51:55.332332+0000","last_fullsized":"2024-08-17T17:51:55.332332+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:03:39.427341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1893,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.070083+0000","last_change":"2024-08-17T17:24:17.453946+0000","last_active":"2024-08-17T17:51:56.070083+0000","last_peered":"2024-08-17T17:51:56.070083+0000","last_clean":"2024-08-17T17:51:56.070083+0000","last_became_active":"2024-08-17T17:24:17.445004+0000","last_became_peered":"2024-08-17T17:24:17.445004+0000","last_unstale":"2024-08-17T17:51:56.070083+0000","last_undegraded":"2024-08-17T17:51:56.070083+0000","last_fullsized":"2024-08-17T17:51:56.070083+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:00:19.366629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1270,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.477362+0000","last_change":"2024-08-17T17:23:07.402338+0000","last_active":"2024-08-17T17:51:54.477362+0000","last_peered":"2024-08-17T17:51:54.477362+0000","last_clean":"2024-08-17T17:51:54.477362+0000","last_became_active":"2024-08-17T17:23:07.401996+0000","last_became_peered":"2024-08-17T17:23:07.401996+0000","last_unstale":"2024-08-17T17:51:54.477362+0000","last_undegraded":"2024-08-17T17:51:54.477362+0000","last_fullsized":"2024-08-17T17:51:54.477362+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:39:21.566786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1878,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.070289+0000","last_change":"2024-08-17T17:24:17.442338+0000","last_active":"2024-08-17T17:51:56.070289+0000","last_peered":"2024-08-17T17:51:56.070289+0000","last_clean":"2024-08-17T17:51:56.070289+0000","last_became_active":"2024-08-17T17:24:17.441948+0000","last_became_peered":"2024-08-17T17:24:17.441948+0000","last_unstale":"2024-08-17T17:51:56.070289+0000","last_undegraded":"2024-08-17T17:51:56.070289+0000","last_fullsized":"2024-08-17T17:51:56.070289+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:17.891492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1896,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.333302+0000","last_change":"2024-08-17T17:24:17.459083+0000","last_active":"2024-08-17T17:51:55.333302+0000","last_peered":"2024-08-17T17:51:55.333302+0000","last_clean":"2024-08-17T17:51:55.333302+0000","last_became_active":"2024-08-17T17:24:17.458759+0000","last_became_peered":"2024-08-17T17:24:17.458759+0000","last_unstale":"2024-08-17T17:51:55.333302+0000","last_undegraded":"2024-08-17T17:51:55.333302+0000","last_fullsized":"2024-08-17T17:51:55.333302+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:11:59.301864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1837,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.072348+0000","last_change":"2024-08-17T17:24:17.468086+0000","last_active":"2024-08-17T17:51:56.072348+0000","last_peered":"2024-08-17T17:51:56.072348+0000","last_clean":"2024-08-17T17:51:56.072348+0000","last_became_active":"2024-08-17T17:24:17.467522+0000","last_became_peered":"2024-08-17T17:24:17.467522+0000","last_unstale":"2024-08-17T17:51:56.072348+0000","last_undegraded":"2024-08-17T17:51:56.072348+0000","last_fullsized":"2024-08-17T17:51:56.072348+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:52:55.249649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1267,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.532926+0000","last_change":"2024-08-17T17:23:07.392447+0000","last_active":"2024-08-17T17:51:54.532926+0000","last_peered":"2024-08-17T17:51:54.532926+0000","last_clean":"2024-08-17T17:51:54.532926+0000","last_became_active":"2024-08-17T17:23:07.391765+0000","last_became_peered":"2024-08-17T17:23:07.391765+0000","last_unstale":"2024-08-17T17:51:54.532926+0000","last_undegraded":"2024-08-17T17:51:54.532926+0000","last_fullsized":"2024-08-17T17:51:54.532926+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:59:18.602346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":1268,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.594762+0000","last_change":"2024-08-17T17:24:17.450313+0000","last_active":"2024-08-17T17:51:54.594762+0000","last_peered":"2024-08-17T17:51:54.594762+0000","last_clean":"2024-08-17T17:51:54.594762+0000","last_became_active":"2024-08-17T17:24:17.448803+0000","last_became_peered":"2024-08-17T17:24:17.448803+0000","last_unstale":"2024-08-17T17:51:54.594762+0000","last_undegraded":"2024-08-17T17:51:54.594762+0000","last_fullsized":"2024-08-17T17:51:54.594762+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:33.554836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1903,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.566095+0000","last_change":"2024-08-17T17:23:42.936147+0000","last_active":"2024-08-17T17:51:55.566095+0000","last_peered":"2024-08-17T17:51:55.566095+0000","last_clean":"2024-08-17T17:51:55.566095+0000","last_became_active":"2024-08-17T17:23:42.935859+0000","last_became_peered":"2024-08-17T17:23:42.935859+0000","last_unstale":"2024-08-17T17:51:55.566095+0000","last_undegraded":"2024-08-17T17:51:55.566095+0000","last_fullsized":"2024-08-17T17:51:55.566095+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:47:36.838706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1281,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332334+0000","last_change":"2024-08-17T17:10:05.969131+0000","last_active":"2024-08-17T17:51:55.332334+0000","last_peered":"2024-08-17T17:51:55.332334+0000","last_clean":"2024-08-17T17:51:55.332334+0000","last_became_active":"2024-08-17T17:10:05.968856+0000","last_became_peered":"2024-08-17T17:10:05.968856+0000","last_unstale":"2024-08-17T17:51:55.332334+0000","last_undegraded":"2024-08-17T17:51:55.332334+0000","last_fullsized":"2024-08-17T17:51:55.332334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:25:49.550065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1272,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332594+0000","last_change":"2024-08-17T17:23:07.455919+0000","last_active":"2024-08-17T17:51:55.332594+0000","last_peered":"2024-08-17T17:51:55.332594+0000","last_clean":"2024-08-17T17:51:55.332594+0000","last_became_active":"2024-08-17T17:23:07.455284+0000","last_became_peered":"2024-08-17T17:23:07.455284+0000","last_unstale":"2024-08-17T17:51:55.332594+0000","last_undegraded":"2024-08-17T17:51:55.332594+0000","last_fullsized":"2024-08-17T17:51:55.332594+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:37:46.260775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1259,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.224555+0000","last_change":"2024-08-17T17:10:06.979492+0000","last_active":"2024-08-17T17:51:54.224555+0000","last_peered":"2024-08-17T17:51:54.224555+0000","last_clean":"2024-08-17T17:51:54.224555+0000","last_became_active":"2024-08-17T17:10:06.979158+0000","last_became_peered":"2024-08-17T17:10:06.979158+0000","last_unstale":"2024-08-17T17:51:54.224555+0000","last_undegraded":"2024-08-17T17:51:54.224555+0000","last_fullsized":"2024-08-17T17:51:54.224555+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:12:42.261365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1283,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.526004+0000","last_change":"2024-08-17T17:23:42.935099+0000","last_active":"2024-08-17T17:51:55.526004+0000","last_peered":"2024-08-17T17:51:55.526004+0000","last_clean":"2024-08-17T17:51:55.526004+0000","last_became_active":"2024-08-17T17:23:42.934501+0000","last_became_peered":"2024-08-17T17:23:42.934501+0000","last_unstale":"2024-08-17T17:51:55.526004+0000","last_undegraded":"2024-08-17T17:51:55.526004+0000","last_fullsized":"2024-08-17T17:51:55.526004+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:15:19.095713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":1902,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332593+0000","last_change":"2024-08-17T17:23:07.436763+0000","last_active":"2024-08-17T17:51:55.332593+0000","last_peered":"2024-08-17T17:51:55.332593+0000","last_clean":"2024-08-17T17:51:55.332593+0000","last_became_active":"2024-08-17T17:23:07.434720+0000","last_became_peered":"2024-08-17T17:23:07.434720+0000","last_unstale":"2024-08-17T17:51:55.332593+0000","last_undegraded":"2024-08-17T17:51:55.332593+0000","last_fullsized":"2024-08-17T17:51:55.332593+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:10:20.594115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1873,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.070209+0000","last_change":"2024-08-17T17:24:17.442329+0000","last_active":"2024-08-17T17:51:56.070209+0000","last_peered":"2024-08-17T17:51:56.070209+0000","last_clean":"2024-08-17T17:51:56.070209+0000","last_became_active":"2024-08-17T17:24:17.441666+0000","last_became_peered":"2024-08-17T17:24:17.441666+0000","last_unstale":"2024-08-17T17:51:56.070209+0000","last_undegraded":"2024-08-17T17:51:56.070209+0000","last_fullsized":"2024-08-17T17:51:56.070209+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:43:06.033194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1885,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565043+0000","last_change":"2024-08-17T17:23:07.428296+0000","last_active":"2024-08-17T17:51:55.565043+0000","last_peered":"2024-08-17T17:51:55.565043+0000","last_clean":"2024-08-17T17:51:55.565043+0000","last_became_active":"2024-08-17T17:23:07.428010+0000","last_became_peered":"2024-08-17T17:23:07.428010+0000","last_unstale":"2024-08-17T17:51:55.565043+0000","last_undegraded":"2024-08-17T17:51:55.565043+0000","last_fullsized":"2024-08-17T17:51:55.565043+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:45.459212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":1874,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.072123+0000","last_change":"2024-08-17T17:24:17.481368+0000","last_active":"2024-08-17T17:51:56.072123+0000","last_peered":"2024-08-17T17:51:56.072123+0000","last_clean":"2024-08-17T17:51:56.072123+0000","last_became_active":"2024-08-17T17:24:17.477509+0000","last_became_peered":"2024-08-17T17:24:17.477509+0000","last_unstale":"2024-08-17T17:51:56.072123+0000","last_undegraded":"2024-08-17T17:51:56.072123+0000","last_fullsized":"2024-08-17T17:51:56.072123+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:05:35.558004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1888,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332264+0000","last_change":"2024-08-17T17:22:32.884274+0000","last_active":"2024-08-17T17:51:55.332264+0000","last_peered":"2024-08-17T17:51:55.332264+0000","last_clean":"2024-08-17T17:51:55.332264+0000","last_became_active":"2024-08-17T17:22:32.874667+0000","last_became_peered":"2024-08-17T17:22:32.874667+0000","last_unstale":"2024-08-17T17:51:55.332264+0000","last_undegraded":"2024-08-17T17:51:55.332264+0000","last_fullsized":"2024-08-17T17:51:55.332264+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:22:05.834587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1280,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.594963+0000","last_change":"2024-08-17T17:22:32.870569+0000","last_active":"2024-08-17T17:51:54.594963+0000","last_peered":"2024-08-17T17:51:54.594963+0000","last_clean":"2024-08-17T17:51:54.594963+0000","last_became_active":"2024-08-17T17:22:32.869335+0000","last_became_peered":"2024-08-17T17:22:32.869335+0000","last_unstale":"2024-08-17T17:51:54.594963+0000","last_undegraded":"2024-08-17T17:51:54.594963+0000","last_fullsized":"2024-08-17T17:51:54.594963+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:24.969174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1907,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.566553+0000","last_change":"2024-08-17T17:23:42.935825+0000","last_active":"2024-08-17T17:51:55.566553+0000","last_peered":"2024-08-17T17:51:55.566553+0000","last_clean":"2024-08-17T17:51:55.566553+0000","last_became_active":"2024-08-17T17:23:42.935150+0000","last_became_peered":"2024-08-17T17:23:42.935150+0000","last_unstale":"2024-08-17T17:51:55.566553+0000","last_undegraded":"2024-08-17T17:51:55.566553+0000","last_fullsized":"2024-08-17T17:51:55.566553+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:57:19.577644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1895,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565154+0000","last_change":"2024-08-17T17:23:07.431586+0000","last_active":"2024-08-17T17:51:55.565154+0000","last_peered":"2024-08-17T17:51:55.565154+0000","last_clean":"2024-08-17T17:51:55.565154+0000","last_became_active":"2024-08-17T17:23:07.431301+0000","last_became_peered":"2024-08-17T17:23:07.431301+0000","last_unstale":"2024-08-17T17:51:55.565154+0000","last_undegraded":"2024-08-17T17:51:55.565154+0000","last_fullsized":"2024-08-17T17:51:55.565154+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:28:18.706489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1923,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565354+0000","last_change":"2024-08-17T17:23:07.420550+0000","last_active":"2024-08-17T17:51:55.565354+0000","last_peered":"2024-08-17T17:51:55.565354+0000","last_clean":"2024-08-17T17:51:55.565354+0000","last_became_active":"2024-08-17T17:23:07.420223+0000","last_became_peered":"2024-08-17T17:23:07.420223+0000","last_unstale":"2024-08-17T17:51:55.565354+0000","last_undegraded":"2024-08-17T17:51:55.565354+0000","last_fullsized":"2024-08-17T17:51:55.565354+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:33:46.996371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1285,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.537366+0000","last_change":"2024-08-17T17:24:17.426267+0000","last_active":"2024-08-17T17:51:55.537366+0000","last_peered":"2024-08-17T17:51:55.537366+0000","last_clean":"2024-08-17T17:51:55.537366+0000","last_became_active":"2024-08-17T17:24:17.425925+0000","last_became_peered":"2024-08-17T17:24:17.425925+0000","last_unstale":"2024-08-17T17:51:55.537366+0000","last_undegraded":"2024-08-17T17:51:55.537366+0000","last_fullsized":"2024-08-17T17:51:55.537366+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:24:25.017856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1281,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.537016+0000","last_change":"2024-08-17T17:24:17.474138+0000","last_active":"2024-08-17T17:51:55.537016+0000","last_peered":"2024-08-17T17:51:55.537016+0000","last_clean":"2024-08-17T17:51:55.537016+0000","last_became_active":"2024-08-17T17:24:17.473708+0000","last_became_peered":"2024-08-17T17:24:17.473708+0000","last_unstale":"2024-08-17T17:51:55.537016+0000","last_undegraded":"2024-08-17T17:51:55.537016+0000","last_fullsized":"2024-08-17T17:51:55.537016+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:10:57.768067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1275,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.225254+0000","last_change":"2024-08-17T17:22:32.871533+0000","last_active":"2024-08-17T17:51:54.225254+0000","last_peered":"2024-08-17T17:51:54.225254+0000","last_clean":"2024-08-17T17:51:54.225254+0000","last_became_active":"2024-08-17T17:22:32.868226+0000","last_became_peered":"2024-08-17T17:22:32.868226+0000","last_unstale":"2024-08-17T17:51:54.225254+0000","last_undegraded":"2024-08-17T17:51:54.225254+0000","last_fullsized":"2024-08-17T17:51:54.225254+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:00:02.629455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":1285,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.537019+0000","last_change":"2024-08-17T17:23:07.391021+0000","last_active":"2024-08-17T17:51:55.537019+0000","last_peered":"2024-08-17T17:51:55.537019+0000","last_clean":"2024-08-17T17:51:55.537019+0000","last_became_active":"2024-08-17T17:23:07.390551+0000","last_became_peered":"2024-08-17T17:23:07.390551+0000","last_unstale":"2024-08-17T17:51:55.537019+0000","last_undegraded":"2024-08-17T17:51:55.537019+0000","last_fullsized":"2024-08-17T17:51:55.537019+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:06.239013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001023532,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1906,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565011+0000","last_change":"2024-08-17T17:23:07.406362+0000","last_active":"2024-08-17T17:51:55.565011+0000","last_peered":"2024-08-17T17:51:55.565011+0000","last_clean":"2024-08-17T17:51:55.565011+0000","last_became_active":"2024-08-17T17:23:07.406071+0000","last_became_peered":"2024-08-17T17:23:07.406071+0000","last_unstale":"2024-08-17T17:51:55.565011+0000","last_undegraded":"2024-08-17T17:51:55.565011+0000","last_fullsized":"2024-08-17T17:51:55.565011+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:45:47.383432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1886,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565198+0000","last_change":"2024-08-17T17:23:42.910246+0000","last_active":"2024-08-17T17:51:55.565198+0000","last_peered":"2024-08-17T17:51:55.565198+0000","last_clean":"2024-08-17T17:51:55.565198+0000","last_became_active":"2024-08-17T17:23:42.909929+0000","last_became_peered":"2024-08-17T17:23:42.909929+0000","last_unstale":"2024-08-17T17:51:55.565198+0000","last_undegraded":"2024-08-17T17:51:55.565198+0000","last_fullsized":"2024-08-17T17:51:55.565198+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:14:45.210558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1916,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.070318+0000","last_change":"2024-08-17T17:24:17.438891+0000","last_active":"2024-08-17T17:51:56.070318+0000","last_peered":"2024-08-17T17:51:56.070318+0000","last_clean":"2024-08-17T17:51:56.070318+0000","last_became_active":"2024-08-17T17:24:17.437714+0000","last_became_peered":"2024-08-17T17:24:17.437714+0000","last_unstale":"2024-08-17T17:51:56.070318+0000","last_undegraded":"2024-08-17T17:51:56.070318+0000","last_fullsized":"2024-08-17T17:51:56.070318+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:58.825017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1909,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332494+0000","last_change":"2024-08-17T17:23:42.908634+0000","last_active":"2024-08-17T17:51:55.332494+0000","last_peered":"2024-08-17T17:51:55.332494+0000","last_clean":"2024-08-17T17:51:55.332494+0000","last_became_active":"2024-08-17T17:23:42.907675+0000","last_became_peered":"2024-08-17T17:23:42.907675+0000","last_unstale":"2024-08-17T17:51:55.332494+0000","last_undegraded":"2024-08-17T17:51:55.332494+0000","last_fullsized":"2024-08-17T17:51:55.332494+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:03:37.612222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1899,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.331258+0000","last_change":"2024-08-17T17:24:17.457559+0000","last_active":"2024-08-17T17:51:55.331258+0000","last_peered":"2024-08-17T17:51:55.331258+0000","last_clean":"2024-08-17T17:51:55.331258+0000","last_became_active":"2024-08-17T17:24:17.457262+0000","last_became_peered":"2024-08-17T17:24:17.457262+0000","last_unstale":"2024-08-17T17:51:55.331258+0000","last_undegraded":"2024-08-17T17:51:55.331258+0000","last_fullsized":"2024-08-17T17:51:55.331258+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:53:38.137277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1296,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.594253+0000","last_change":"2024-08-17T17:23:07.450509+0000","last_active":"2024-08-17T17:51:54.594253+0000","last_peered":"2024-08-17T17:51:54.594253+0000","last_clean":"2024-08-17T17:51:54.594253+0000","last_became_active":"2024-08-17T17:23:07.450182+0000","last_became_peered":"2024-08-17T17:23:07.450182+0000","last_unstale":"2024-08-17T17:51:54.594253+0000","last_undegraded":"2024-08-17T17:51:54.594253+0000","last_fullsized":"2024-08-17T17:51:54.594253+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:01:18.656656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1255,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.526524+0000","last_change":"2024-08-17T17:10:06.995023+0000","last_active":"2024-08-17T17:51:55.526524+0000","last_peered":"2024-08-17T17:51:55.526524+0000","last_clean":"2024-08-17T17:51:55.526524+0000","last_became_active":"2024-08-17T17:10:06.994757+0000","last_became_peered":"2024-08-17T17:10:06.994757+0000","last_unstale":"2024-08-17T17:51:55.526524+0000","last_undegraded":"2024-08-17T17:51:55.526524+0000","last_fullsized":"2024-08-17T17:51:55.526524+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:31:52.163518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1839,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.071057+0000","last_change":"2024-08-17T17:24:17.468443+0000","last_active":"2024-08-17T17:51:56.071057+0000","last_peered":"2024-08-17T17:51:56.071057+0000","last_clean":"2024-08-17T17:51:56.071057+0000","last_became_active":"2024-08-17T17:24:17.467394+0000","last_became_peered":"2024-08-17T17:24:17.467394+0000","last_unstale":"2024-08-17T17:51:56.071057+0000","last_undegraded":"2024-08-17T17:51:56.071057+0000","last_fullsized":"2024-08-17T17:51:56.071057+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:13.024253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":1284,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.537434+0000","last_change":"2024-08-17T17:23:42.920552+0000","last_active":"2024-08-17T17:51:55.537434+0000","last_peered":"2024-08-17T17:51:55.537434+0000","last_clean":"2024-08-17T17:51:55.537434+0000","last_became_active":"2024-08-17T17:23:42.915824+0000","last_became_peered":"2024-08-17T17:23:42.915824+0000","last_unstale":"2024-08-17T17:51:55.537434+0000","last_undegraded":"2024-08-17T17:51:55.537434+0000","last_fullsized":"2024-08-17T17:51:55.537434+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:25:18.769504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041737399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":1597,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.556550+0000","last_change":"2024-08-17T17:22:32.867754+0000","last_active":"2024-08-17T17:51:55.556550+0000","last_peered":"2024-08-17T17:51:55.556550+0000","last_clean":"2024-08-17T17:51:55.556550+0000","last_became_active":"2024-08-17T17:22:32.867345+0000","last_became_peered":"2024-08-17T17:22:32.867345+0000","last_unstale":"2024-08-17T17:51:55.556550+0000","last_undegraded":"2024-08-17T17:51:55.556550+0000","last_fullsized":"2024-08-17T17:51:55.556550+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T17:06:21.174471+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:00:42.776037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1885,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565625+0000","last_change":"2024-08-17T17:24:17.455710+0000","last_active":"2024-08-17T17:51:55.565625+0000","last_peered":"2024-08-17T17:51:55.565625+0000","last_clean":"2024-08-17T17:51:55.565625+0000","last_became_active":"2024-08-17T17:24:17.452989+0000","last_became_peered":"2024-08-17T17:24:17.452989+0000","last_unstale":"2024-08-17T17:51:55.565625+0000","last_undegraded":"2024-08-17T17:51:55.565625+0000","last_fullsized":"2024-08-17T17:51:55.565625+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:58:07.894968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1880,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.070034+0000","last_change":"2024-08-17T17:24:17.453639+0000","last_active":"2024-08-17T17:51:56.070034+0000","last_peered":"2024-08-17T17:51:56.070034+0000","last_clean":"2024-08-17T17:51:56.070034+0000","last_became_active":"2024-08-17T17:24:17.443638+0000","last_became_peered":"2024-08-17T17:24:17.443638+0000","last_unstale":"2024-08-17T17:51:56.070034+0000","last_undegraded":"2024-08-17T17:51:56.070034+0000","last_fullsized":"2024-08-17T17:51:56.070034+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:42.107479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1272,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.225392+0000","last_change":"2024-08-17T17:23:07.451866+0000","last_active":"2024-08-17T17:51:54.225392+0000","last_peered":"2024-08-17T17:51:54.225392+0000","last_clean":"2024-08-17T17:51:54.225392+0000","last_became_active":"2024-08-17T17:23:07.451553+0000","last_became_peered":"2024-08-17T17:23:07.451553+0000","last_unstale":"2024-08-17T17:51:54.225392+0000","last_undegraded":"2024-08-17T17:51:54.225392+0000","last_fullsized":"2024-08-17T17:51:54.225392+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:37:23.930651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1274,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.224623+0000","last_change":"2024-08-17T17:23:42.915851+0000","last_active":"2024-08-17T17:51:54.224623+0000","last_peered":"2024-08-17T17:51:54.224623+0000","last_clean":"2024-08-17T17:51:54.224623+0000","last_became_active":"2024-08-17T17:23:42.915552+0000","last_became_peered":"2024-08-17T17:23:42.915552+0000","last_unstale":"2024-08-17T17:51:54.224623+0000","last_undegraded":"2024-08-17T17:51:54.224623+0000","last_fullsized":"2024-08-17T17:51:54.224623+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:45.197876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1287,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.225154+0000","last_change":"2024-08-17T17:23:42.930803+0000","last_active":"2024-08-17T17:51:54.225154+0000","last_peered":"2024-08-17T17:51:54.225154+0000","last_clean":"2024-08-17T17:51:54.225154+0000","last_became_active":"2024-08-17T17:23:42.930458+0000","last_became_peered":"2024-08-17T17:23:42.930458+0000","last_unstale":"2024-08-17T17:51:54.225154+0000","last_undegraded":"2024-08-17T17:51:54.225154+0000","last_fullsized":"2024-08-17T17:51:54.225154+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:40.816706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1270,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.477299+0000","last_change":"2024-08-17T17:23:07.401624+0000","last_active":"2024-08-17T17:51:54.477299+0000","last_peered":"2024-08-17T17:51:54.477299+0000","last_clean":"2024-08-17T17:51:54.477299+0000","last_became_active":"2024-08-17T17:23:07.400222+0000","last_became_peered":"2024-08-17T17:23:07.400222+0000","last_unstale":"2024-08-17T17:51:54.477299+0000","last_undegraded":"2024-08-17T17:51:54.477299+0000","last_fullsized":"2024-08-17T17:51:54.477299+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:07.069989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1272,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.224982+0000","last_change":"2024-08-17T17:24:17.470809+0000","last_active":"2024-08-17T17:51:54.224982+0000","last_peered":"2024-08-17T17:51:54.224982+0000","last_clean":"2024-08-17T17:51:54.224982+0000","last_became_active":"2024-08-17T17:24:17.469753+0000","last_became_peered":"2024-08-17T17:24:17.469753+0000","last_unstale":"2024-08-17T17:51:54.224982+0000","last_undegraded":"2024-08-17T17:51:54.224982+0000","last_fullsized":"2024-08-17T17:51:54.224982+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:00.049425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":1873,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.069486+0000","last_change":"2024-08-17T17:24:17.461258+0000","last_active":"2024-08-17T17:51:56.069486+0000","last_peered":"2024-08-17T17:51:56.069486+0000","last_clean":"2024-08-17T17:51:56.069486+0000","last_became_active":"2024-08-17T17:24:17.460895+0000","last_became_peered":"2024-08-17T17:24:17.460895+0000","last_unstale":"2024-08-17T17:51:56.069486+0000","last_undegraded":"2024-08-17T17:51:56.069486+0000","last_fullsized":"2024-08-17T17:51:56.069486+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:43:24.208259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1912,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.331877+0000","last_change":"2024-08-17T17:23:42.901638+0000","last_active":"2024-08-17T17:51:55.331877+0000","last_peered":"2024-08-17T17:51:55.331877+0000","last_clean":"2024-08-17T17:51:55.331877+0000","last_became_active":"2024-08-17T17:23:42.901335+0000","last_became_peered":"2024-08-17T17:23:42.901335+0000","last_unstale":"2024-08-17T17:51:55.331877+0000","last_undegraded":"2024-08-17T17:51:55.331877+0000","last_fullsized":"2024-08-17T17:51:55.331877+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:25:35.332839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1268,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.532827+0000","last_change":"2024-08-17T17:24:17.473913+0000","last_active":"2024-08-17T17:51:54.532827+0000","last_peered":"2024-08-17T17:51:54.532827+0000","last_clean":"2024-08-17T17:51:54.532827+0000","last_became_active":"2024-08-17T17:24:17.473195+0000","last_became_peered":"2024-08-17T17:24:17.473195+0000","last_unstale":"2024-08-17T17:51:54.532827+0000","last_undegraded":"2024-08-17T17:51:54.532827+0000","last_fullsized":"2024-08-17T17:51:54.532827+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:12.998033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1269,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.477272+0000","last_change":"2024-08-17T17:23:07.402010+0000","last_active":"2024-08-17T17:51:54.477272+0000","last_peered":"2024-08-17T17:51:54.477272+0000","last_clean":"2024-08-17T17:51:54.477272+0000","last_became_active":"2024-08-17T17:23:07.400849+0000","last_became_peered":"2024-08-17T17:23:07.400849+0000","last_unstale":"2024-08-17T17:51:54.477272+0000","last_undegraded":"2024-08-17T17:51:54.477272+0000","last_fullsized":"2024-08-17T17:51:54.477272+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:16:02.382257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":1252,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.594877+0000","last_change":"2024-08-17T17:10:05.973491+0000","last_active":"2024-08-17T17:51:54.594877+0000","last_peered":"2024-08-17T17:51:54.594877+0000","last_clean":"2024-08-17T17:51:54.594877+0000","last_became_active":"2024-08-17T17:10:05.973228+0000","last_became_peered":"2024-08-17T17:10:05.973228+0000","last_unstale":"2024-08-17T17:51:54.594877+0000","last_undegraded":"2024-08-17T17:51:54.594877+0000","last_fullsized":"2024-08-17T17:51:54.594877+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:54.310135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1288,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.224430+0000","last_change":"2024-08-17T17:24:17.460203+0000","last_active":"2024-08-17T17:51:54.224430+0000","last_peered":"2024-08-17T17:51:54.224430+0000","last_clean":"2024-08-17T17:51:54.224430+0000","last_became_active":"2024-08-17T17:24:17.459682+0000","last_became_peered":"2024-08-17T17:24:17.459682+0000","last_unstale":"2024-08-17T17:51:54.224430+0000","last_undegraded":"2024-08-17T17:51:54.224430+0000","last_fullsized":"2024-08-17T17:51:54.224430+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:58.703700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":1268,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.594587+0000","last_change":"2024-08-17T17:23:07.454042+0000","last_active":"2024-08-17T17:51:54.594587+0000","last_peered":"2024-08-17T17:51:54.594587+0000","last_clean":"2024-08-17T17:51:54.594587+0000","last_became_active":"2024-08-17T17:23:07.453712+0000","last_became_peered":"2024-08-17T17:23:07.453712+0000","last_unstale":"2024-08-17T17:51:54.594587+0000","last_undegraded":"2024-08-17T17:51:54.594587+0000","last_fullsized":"2024-08-17T17:51:54.594587+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:03:07.643456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1253,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.532272+0000","last_change":"2024-08-17T17:10:06.994799+0000","last_active":"2024-08-17T17:51:54.532272+0000","last_peered":"2024-08-17T17:51:54.532272+0000","last_clean":"2024-08-17T17:51:54.532272+0000","last_became_active":"2024-08-17T17:10:06.994150+0000","last_became_peered":"2024-08-17T17:10:06.994150+0000","last_unstale":"2024-08-17T17:51:54.532272+0000","last_undegraded":"2024-08-17T17:51:54.532272+0000","last_fullsized":"2024-08-17T17:51:54.532272+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:50:47.293091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":1266,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.594586+0000","last_change":"2024-08-17T17:22:32.873334+0000","last_active":"2024-08-17T17:51:54.594586+0000","last_peered":"2024-08-17T17:51:54.594586+0000","last_clean":"2024-08-17T17:51:54.594586+0000","last_became_active":"2024-08-17T17:22:32.870147+0000","last_became_peered":"2024-08-17T17:22:32.870147+0000","last_unstale":"2024-08-17T17:51:54.594586+0000","last_undegraded":"2024-08-17T17:51:54.594586+0000","last_fullsized":"2024-08-17T17:51:54.594586+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:18:03.750592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1277,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.524676+0000","last_change":"2024-08-17T17:22:32.847989+0000","last_active":"2024-08-17T17:51:55.524676+0000","last_peered":"2024-08-17T17:51:55.524676+0000","last_clean":"2024-08-17T17:51:55.524676+0000","last_became_active":"2024-08-17T17:22:32.846593+0000","last_became_peered":"2024-08-17T17:22:32.846593+0000","last_unstale":"2024-08-17T17:51:55.524676+0000","last_undegraded":"2024-08-17T17:51:55.524676+0000","last_fullsized":"2024-08-17T17:51:55.524676+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:33:50.370124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1877,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.071934+0000","last_change":"2024-08-17T17:24:17.451528+0000","last_active":"2024-08-17T17:51:56.071934+0000","last_peered":"2024-08-17T17:51:56.071934+0000","last_clean":"2024-08-17T17:51:56.071934+0000","last_became_active":"2024-08-17T17:24:17.451246+0000","last_became_peered":"2024-08-17T17:24:17.451246+0000","last_unstale":"2024-08-17T17:51:56.071934+0000","last_undegraded":"2024-08-17T17:51:56.071934+0000","last_fullsized":"2024-08-17T17:51:56.071934+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:38:18.176559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1251,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.557291+0000","last_change":"2024-08-17T17:10:05.975692+0000","last_active":"2024-08-17T17:51:55.557291+0000","last_peered":"2024-08-17T17:51:55.557291+0000","last_clean":"2024-08-17T17:51:55.557291+0000","last_became_active":"2024-08-17T17:10:05.975057+0000","last_became_peered":"2024-08-17T17:10:05.975057+0000","last_unstale":"2024-08-17T17:51:55.557291+0000","last_undegraded":"2024-08-17T17:51:55.557291+0000","last_fullsized":"2024-08-17T17:51:55.557291+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:44:53.106104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1896,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.331881+0000","last_change":"2024-08-17T17:24:17.460371+0000","last_active":"2024-08-17T17:51:55.331881+0000","last_peered":"2024-08-17T17:51:55.331881+0000","last_clean":"2024-08-17T17:51:55.331881+0000","last_became_active":"2024-08-17T17:24:17.460074+0000","last_became_peered":"2024-08-17T17:24:17.460074+0000","last_unstale":"2024-08-17T17:51:55.331881+0000","last_undegraded":"2024-08-17T17:51:55.331881+0000","last_fullsized":"2024-08-17T17:51:55.331881+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:08:48.820168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1896,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332180+0000","last_change":"2024-08-17T17:24:17.457942+0000","last_active":"2024-08-17T17:51:55.332180+0000","last_peered":"2024-08-17T17:51:55.332180+0000","last_clean":"2024-08-17T17:51:55.332180+0000","last_became_active":"2024-08-17T17:24:17.456475+0000","last_became_peered":"2024-08-17T17:24:17.456475+0000","last_unstale":"2024-08-17T17:51:55.332180+0000","last_undegraded":"2024-08-17T17:51:55.332180+0000","last_fullsized":"2024-08-17T17:51:55.332180+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:12.325657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1266,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.532706+0000","last_change":"2024-08-17T17:24:17.451287+0000","last_active":"2024-08-17T17:51:54.532706+0000","last_peered":"2024-08-17T17:51:54.532706+0000","last_clean":"2024-08-17T17:51:54.532706+0000","last_became_active":"2024-08-17T17:24:17.450951+0000","last_became_peered":"2024-08-17T17:24:17.450951+0000","last_unstale":"2024-08-17T17:51:54.532706+0000","last_undegraded":"2024-08-17T17:51:54.532706+0000","last_fullsized":"2024-08-17T17:51:54.532706+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:13:37.590108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":1268,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.556246+0000","last_change":"2024-08-17T17:23:42.907970+0000","last_active":"2024-08-17T17:51:55.556246+0000","last_peered":"2024-08-17T17:51:55.556246+0000","last_clean":"2024-08-17T17:51:55.556246+0000","last_became_active":"2024-08-17T17:23:42.907595+0000","last_became_peered":"2024-08-17T17:23:42.907595+0000","last_unstale":"2024-08-17T17:51:55.556246+0000","last_undegraded":"2024-08-17T17:51:55.556246+0000","last_fullsized":"2024-08-17T17:51:55.556246+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:11:46.130004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1878,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.072237+0000","last_change":"2024-08-17T17:24:17.486591+0000","last_active":"2024-08-17T17:51:56.072237+0000","last_peered":"2024-08-17T17:51:56.072237+0000","last_clean":"2024-08-17T17:51:56.072237+0000","last_became_active":"2024-08-17T17:24:17.486320+0000","last_became_peered":"2024-08-17T17:24:17.486320+0000","last_unstale":"2024-08-17T17:51:56.072237+0000","last_undegraded":"2024-08-17T17:51:56.072237+0000","last_fullsized":"2024-08-17T17:51:56.072237+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:03.249803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234520,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":120032,"kb_used_data":16792,"kb_used_omap":1,"kb_used_meta":103230,"kb_avail":93625120,"statfs":{"total":95995035648,"available":95872122880,"internally_reserved":0,"allocated":17195008,"data_stored":10837596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":105707979},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496158,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":146268,"kb_used_data":16784,"kb_used_omap":1,"kb_used_meta":129470,"kb_avail":93598884,"statfs":{"total":95995035648,"available":95845257216,"internally_reserved":0,"allocated":17186816,"data_stored":10837469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":132577735},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565399,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":92740,"kb_used_data":16188,"kb_used_omap":72,"kb_used_meta":76471,"kb_avail":93652412,"statfs":{"total":95995035648,"available":95900069888,"internally_reserved":0,"allocated":16576512,"data_stored":10242358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":78306798},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":674,"seq":2894807957854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70540,"kb_used_data":16192,"kb_used_omap":74,"kb_used_meta":54261,"kb_avail":93674612,"statfs":{"total":95995035648,"available":95922802688,"internally_reserved":0,"allocated":16580608,"data_stored":10242884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76116,"internal_metadata":55563948},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743186789,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70472,"kb_used_data":16188,"kb_used_omap":66,"kb_used_meta":54205,"kb_avail":93674680,"statfs":{"total":95995035648,"available":95922872320,"internally_reserved":0,"allocated":16576512,"data_stored":10242287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":55506603},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448428,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88916,"kb_used_data":16772,"kb_used_omap":75,"kb_used_meta":72052,"kb_avail":93656236,"statfs":{"total":95995035648,"available":95903985664,"internally_reserved":0,"allocated":17174528,"data_stored":10833240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":73781439},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757794,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":127700,"kb_used_data":16196,"kb_used_omap":1,"kb_used_meta":111486,"kb_avail":93617452,"statfs":{"total":95995035648,"available":95864270848,"internally_reserved":0,"allocated":16584704,"data_stored":10242927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114162118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019431,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":141600,"kb_used_data":16208,"kb_used_omap":1,"kb_used_meta":125374,"kb_avail":93603552,"statfs":{"total":95995035648,"available":95850037248,"internally_reserved":0,"allocated":16596992,"data_stored":10247158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128383430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:52:46.704 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-08-17T17:52:46.704 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-08-17T17:52:46.704 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-08-17T17:52:46.705 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph health --format=json 2024-08-17T17:52:46.903 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:47.364 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:47 smithi136 ceph-mon[109861]: from='client.35022 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:52:47.364 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:47 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:52:47.364 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:47 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:52:47.364 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:47 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:52:47.364 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:47 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:52:47.364 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:47 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:52:47.364 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:47 smithi136 ceph-mon[115078]: from='client.35022 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:52:47.365 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:47 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:52:47.365 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:47 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:52:47.365 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:47 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:52:47.365 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:47 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:52:47.365 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:47 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:52:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:47 smithi165 ceph-mon[105517]: from='client.35022 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:52:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:47 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:52:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:47 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:52:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:47 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:52:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:47 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:52:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:47 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:52:47.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:52:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:52:47] "GET /metrics HTTP/1.1" 200 34058 "" "Prometheus/2.51.0" 2024-08-17T17:52:48.639 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:48 smithi136 ceph-mon[109861]: pgmap v2059: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:48.640 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:48 smithi136 ceph-mon[115078]: pgmap v2059: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:48.640 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:52:48.640 INFO:teuthology.orchestra.run.smithi136.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-17T17:52:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:48 smithi165 ceph-mon[105517]: pgmap v2059: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:49.325 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-08-17T17:52:49.325 INFO:teuthology.run_tasks:Running task print... 2024-08-17T17:52:49.338 INFO:teuthology.task.print:**** applying stress + thrashing to mixed-version cluster **** 2024-08-17T17:52:49.338 INFO:teuthology.run_tasks:Running task parallel... 2024-08-17T17:52:49.348 INFO:teuthology.task.parallel:starting parallel... 2024-08-17T17:52:49.348 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-08-17T17:52:49.349 INFO:teuthology.task.sequential:In sequential, running task thrashosds... 2024-08-17T17:52:49.351 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_force_recovery': 0, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_thrash_pg_upmap_items': 0, 'disable_objectstore_tool_tests': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-08-17T17:52:49.351 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'mons down', 'mon down', 'MON_DOWN', 'out of quorum', 'PG_AVAILABILITY'], 'sha1': 'cd8bdbeb9fbdab5733aa3aaadc042c965473e453'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'cd8bdbeb9fbdab5733aa3aaadc042c965473e453'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'main', 'sha1': '91f88e4d132ac08dc53355fa3d9717a53236a3a9'}} 2024-08-17T17:52:49.351 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_force_recovery': 0, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_thrash_pg_upmap_items': 0, 'disable_objectstore_tool_tests': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-08-17T17:52:49.351 INFO:tasks.thrashosds:Beginning thrashosds... 2024-08-17T17:52:49.351 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-08-17T17:52:49.352 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:52:49.566 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:49.608 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:49 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2679997556' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-08-17T17:52:49.608 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:49 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2679997556' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-08-17T17:52:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:49 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2679997556' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-08-17T17:52:50.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:50 smithi136 ceph-mon[109861]: pgmap v2060: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:50.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:50 smithi136 ceph-mon[115078]: pgmap v2060: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:50.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:50 smithi165 ceph-mon[105517]: pgmap v2060: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:51.355 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:52:51.356 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T17:52:51.672 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:52:51.672 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:52:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:52:52.045 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":2061,"stamp":"2024-08-17T17:52:50.569183+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595941,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":279,"num_write_kb":2449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":858268,"kb_used_data":131320,"kb_used_omap":295,"kb_used_meta":726552,"kb_avail":749102948,"statfs":{"total":767960285184,"available":767081418752,"internally_reserved":0,"allocated":134471680,"data_stored":83725919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":302302,"internal_metadata":743990050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002629"},"pg_stats":[{"pgid":"3.1f","version":"70'4","reported_seq":1906,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.333168+0000","last_change":"2024-08-17T17:23:42.937520+0000","last_active":"2024-08-17T17:51:55.333168+0000","last_peered":"2024-08-17T17:51:55.333168+0000","last_clean":"2024-08-17T17:51:55.333168+0000","last_became_active":"2024-08-17T17:23:42.937174+0000","last_became_peered":"2024-08-17T17:23:42.937174+0000","last_unstale":"2024-08-17T17:51:55.333168+0000","last_undegraded":"2024-08-17T17:51:55.333168+0000","last_fullsized":"2024-08-17T17:51:55.333168+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:47:56.679612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1271,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.224325+0000","last_change":"2024-08-17T17:23:07.398676+0000","last_active":"2024-08-17T17:51:54.224325+0000","last_peered":"2024-08-17T17:51:54.224325+0000","last_clean":"2024-08-17T17:51:54.224325+0000","last_became_active":"2024-08-17T17:23:07.398369+0000","last_became_peered":"2024-08-17T17:23:07.398369+0000","last_unstale":"2024-08-17T17:51:54.224325+0000","last_undegraded":"2024-08-17T17:51:54.224325+0000","last_fullsized":"2024-08-17T17:51:54.224325+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:55:04.281412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":1890,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.071332+0000","last_change":"2024-08-17T17:24:17.476168+0000","last_active":"2024-08-17T17:51:56.071332+0000","last_peered":"2024-08-17T17:51:56.071332+0000","last_clean":"2024-08-17T17:51:56.071332+0000","last_became_active":"2024-08-17T17:24:17.465845+0000","last_became_peered":"2024-08-17T17:24:17.465845+0000","last_unstale":"2024-08-17T17:51:56.071332+0000","last_undegraded":"2024-08-17T17:51:56.071332+0000","last_fullsized":"2024-08-17T17:51:56.071332+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:15:01.494447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1906,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.071442+0000","last_change":"2024-08-17T17:24:17.461772+0000","last_active":"2024-08-17T17:51:56.071442+0000","last_peered":"2024-08-17T17:51:56.071442+0000","last_clean":"2024-08-17T17:51:56.071442+0000","last_became_active":"2024-08-17T17:24:17.460806+0000","last_became_peered":"2024-08-17T17:24:17.460806+0000","last_unstale":"2024-08-17T17:51:56.071442+0000","last_undegraded":"2024-08-17T17:51:56.071442+0000","last_fullsized":"2024-08-17T17:51:56.071442+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:01:16.453148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":1254,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.536641+0000","last_change":"2024-08-17T17:10:05.965176+0000","last_active":"2024-08-17T17:51:55.536641+0000","last_peered":"2024-08-17T17:51:55.536641+0000","last_clean":"2024-08-17T17:51:55.536641+0000","last_became_active":"2024-08-17T17:10:05.964861+0000","last_became_peered":"2024-08-17T17:10:05.964861+0000","last_unstale":"2024-08-17T17:51:55.536641+0000","last_undegraded":"2024-08-17T17:51:55.536641+0000","last_fullsized":"2024-08-17T17:51:55.536641+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:59:04.277926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1286,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.330974+0000","last_change":"2024-08-17T17:24:17.462527+0000","last_active":"2024-08-17T17:51:55.330974+0000","last_peered":"2024-08-17T17:51:55.330974+0000","last_clean":"2024-08-17T17:51:55.330974+0000","last_became_active":"2024-08-17T17:24:17.462240+0000","last_became_peered":"2024-08-17T17:24:17.462240+0000","last_unstale":"2024-08-17T17:51:55.330974+0000","last_undegraded":"2024-08-17T17:51:55.330974+0000","last_fullsized":"2024-08-17T17:51:55.330974+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:40:38.063717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":1269,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.536271+0000","last_change":"2024-08-17T17:23:07.391038+0000","last_active":"2024-08-17T17:51:55.536271+0000","last_peered":"2024-08-17T17:51:55.536271+0000","last_clean":"2024-08-17T17:51:55.536271+0000","last_became_active":"2024-08-17T17:23:07.390571+0000","last_became_peered":"2024-08-17T17:23:07.390571+0000","last_unstale":"2024-08-17T17:51:55.536271+0000","last_undegraded":"2024-08-17T17:51:55.536271+0000","last_fullsized":"2024-08-17T17:51:55.536271+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:18:41.791740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1286,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332513+0000","last_change":"2024-08-17T17:24:17.458168+0000","last_active":"2024-08-17T17:51:55.332513+0000","last_peered":"2024-08-17T17:51:55.332513+0000","last_clean":"2024-08-17T17:51:55.332513+0000","last_became_active":"2024-08-17T17:24:17.457847+0000","last_became_peered":"2024-08-17T17:24:17.457847+0000","last_unstale":"2024-08-17T17:51:55.332513+0000","last_undegraded":"2024-08-17T17:51:55.332513+0000","last_fullsized":"2024-08-17T17:51:55.332513+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:28.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":1889,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332046+0000","last_change":"2024-08-17T17:22:32.883756+0000","last_active":"2024-08-17T17:51:55.332046+0000","last_peered":"2024-08-17T17:51:55.332046+0000","last_clean":"2024-08-17T17:51:55.332046+0000","last_became_active":"2024-08-17T17:22:32.874867+0000","last_became_peered":"2024-08-17T17:22:32.874867+0000","last_unstale":"2024-08-17T17:51:55.332046+0000","last_undegraded":"2024-08-17T17:51:55.332046+0000","last_fullsized":"2024-08-17T17:51:55.332046+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:52:27.375231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1908,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.069811+0000","last_change":"2024-08-17T17:24:17.456622+0000","last_active":"2024-08-17T17:51:56.069811+0000","last_peered":"2024-08-17T17:51:56.069811+0000","last_clean":"2024-08-17T17:51:56.069811+0000","last_became_active":"2024-08-17T17:24:17.455396+0000","last_became_peered":"2024-08-17T17:24:17.455396+0000","last_unstale":"2024-08-17T17:51:56.069811+0000","last_undegraded":"2024-08-17T17:51:56.069811+0000","last_fullsized":"2024-08-17T17:51:56.069811+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:09.564331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1288,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.331131+0000","last_change":"2024-08-17T17:23:42.930464+0000","last_active":"2024-08-17T17:51:55.331131+0000","last_peered":"2024-08-17T17:51:55.331131+0000","last_clean":"2024-08-17T17:51:55.331131+0000","last_became_active":"2024-08-17T17:23:42.929949+0000","last_became_peered":"2024-08-17T17:23:42.929949+0000","last_unstale":"2024-08-17T17:51:55.331131+0000","last_undegraded":"2024-08-17T17:51:55.331131+0000","last_fullsized":"2024-08-17T17:51:55.331131+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:48:27.092339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1893,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.070329+0000","last_change":"2024-08-17T17:24:17.452009+0000","last_active":"2024-08-17T17:51:56.070329+0000","last_peered":"2024-08-17T17:51:56.070329+0000","last_clean":"2024-08-17T17:51:56.070329+0000","last_became_active":"2024-08-17T17:24:17.451492+0000","last_became_peered":"2024-08-17T17:24:17.451492+0000","last_unstale":"2024-08-17T17:51:56.070329+0000","last_undegraded":"2024-08-17T17:51:56.070329+0000","last_fullsized":"2024-08-17T17:51:56.070329+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:21:24.699633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1904,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565316+0000","last_change":"2024-08-17T17:24:17.452025+0000","last_active":"2024-08-17T17:51:55.565316+0000","last_peered":"2024-08-17T17:51:55.565316+0000","last_clean":"2024-08-17T17:51:55.565316+0000","last_became_active":"2024-08-17T17:24:17.448029+0000","last_became_peered":"2024-08-17T17:24:17.448029+0000","last_unstale":"2024-08-17T17:51:55.565316+0000","last_undegraded":"2024-08-17T17:51:55.565316+0000","last_fullsized":"2024-08-17T17:51:55.565316+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:45.089346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1900,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332332+0000","last_change":"2024-08-17T17:24:17.469096+0000","last_active":"2024-08-17T17:51:55.332332+0000","last_peered":"2024-08-17T17:51:55.332332+0000","last_clean":"2024-08-17T17:51:55.332332+0000","last_became_active":"2024-08-17T17:24:17.468669+0000","last_became_peered":"2024-08-17T17:24:17.468669+0000","last_unstale":"2024-08-17T17:51:55.332332+0000","last_undegraded":"2024-08-17T17:51:55.332332+0000","last_fullsized":"2024-08-17T17:51:55.332332+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:03:39.427341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1893,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.070083+0000","last_change":"2024-08-17T17:24:17.453946+0000","last_active":"2024-08-17T17:51:56.070083+0000","last_peered":"2024-08-17T17:51:56.070083+0000","last_clean":"2024-08-17T17:51:56.070083+0000","last_became_active":"2024-08-17T17:24:17.445004+0000","last_became_peered":"2024-08-17T17:24:17.445004+0000","last_unstale":"2024-08-17T17:51:56.070083+0000","last_undegraded":"2024-08-17T17:51:56.070083+0000","last_fullsized":"2024-08-17T17:51:56.070083+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:00:19.366629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1270,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.477362+0000","last_change":"2024-08-17T17:23:07.402338+0000","last_active":"2024-08-17T17:51:54.477362+0000","last_peered":"2024-08-17T17:51:54.477362+0000","last_clean":"2024-08-17T17:51:54.477362+0000","last_became_active":"2024-08-17T17:23:07.401996+0000","last_became_peered":"2024-08-17T17:23:07.401996+0000","last_unstale":"2024-08-17T17:51:54.477362+0000","last_undegraded":"2024-08-17T17:51:54.477362+0000","last_fullsized":"2024-08-17T17:51:54.477362+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:39:21.566786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1878,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.070289+0000","last_change":"2024-08-17T17:24:17.442338+0000","last_active":"2024-08-17T17:51:56.070289+0000","last_peered":"2024-08-17T17:51:56.070289+0000","last_clean":"2024-08-17T17:51:56.070289+0000","last_became_active":"2024-08-17T17:24:17.441948+0000","last_became_peered":"2024-08-17T17:24:17.441948+0000","last_unstale":"2024-08-17T17:51:56.070289+0000","last_undegraded":"2024-08-17T17:51:56.070289+0000","last_fullsized":"2024-08-17T17:51:56.070289+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:17.891492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1896,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.333302+0000","last_change":"2024-08-17T17:24:17.459083+0000","last_active":"2024-08-17T17:51:55.333302+0000","last_peered":"2024-08-17T17:51:55.333302+0000","last_clean":"2024-08-17T17:51:55.333302+0000","last_became_active":"2024-08-17T17:24:17.458759+0000","last_became_peered":"2024-08-17T17:24:17.458759+0000","last_unstale":"2024-08-17T17:51:55.333302+0000","last_undegraded":"2024-08-17T17:51:55.333302+0000","last_fullsized":"2024-08-17T17:51:55.333302+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:11:59.301864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1837,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.072348+0000","last_change":"2024-08-17T17:24:17.468086+0000","last_active":"2024-08-17T17:51:56.072348+0000","last_peered":"2024-08-17T17:51:56.072348+0000","last_clean":"2024-08-17T17:51:56.072348+0000","last_became_active":"2024-08-17T17:24:17.467522+0000","last_became_peered":"2024-08-17T17:24:17.467522+0000","last_unstale":"2024-08-17T17:51:56.072348+0000","last_undegraded":"2024-08-17T17:51:56.072348+0000","last_fullsized":"2024-08-17T17:51:56.072348+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:52:55.249649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1267,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.532926+0000","last_change":"2024-08-17T17:23:07.392447+0000","last_active":"2024-08-17T17:51:54.532926+0000","last_peered":"2024-08-17T17:51:54.532926+0000","last_clean":"2024-08-17T17:51:54.532926+0000","last_became_active":"2024-08-17T17:23:07.391765+0000","last_became_peered":"2024-08-17T17:23:07.391765+0000","last_unstale":"2024-08-17T17:51:54.532926+0000","last_undegraded":"2024-08-17T17:51:54.532926+0000","last_fullsized":"2024-08-17T17:51:54.532926+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:59:18.602346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":1268,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.594762+0000","last_change":"2024-08-17T17:24:17.450313+0000","last_active":"2024-08-17T17:51:54.594762+0000","last_peered":"2024-08-17T17:51:54.594762+0000","last_clean":"2024-08-17T17:51:54.594762+0000","last_became_active":"2024-08-17T17:24:17.448803+0000","last_became_peered":"2024-08-17T17:24:17.448803+0000","last_unstale":"2024-08-17T17:51:54.594762+0000","last_undegraded":"2024-08-17T17:51:54.594762+0000","last_fullsized":"2024-08-17T17:51:54.594762+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:33.554836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1903,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.566095+0000","last_change":"2024-08-17T17:23:42.936147+0000","last_active":"2024-08-17T17:51:55.566095+0000","last_peered":"2024-08-17T17:51:55.566095+0000","last_clean":"2024-08-17T17:51:55.566095+0000","last_became_active":"2024-08-17T17:23:42.935859+0000","last_became_peered":"2024-08-17T17:23:42.935859+0000","last_unstale":"2024-08-17T17:51:55.566095+0000","last_undegraded":"2024-08-17T17:51:55.566095+0000","last_fullsized":"2024-08-17T17:51:55.566095+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:47:36.838706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1281,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332334+0000","last_change":"2024-08-17T17:10:05.969131+0000","last_active":"2024-08-17T17:51:55.332334+0000","last_peered":"2024-08-17T17:51:55.332334+0000","last_clean":"2024-08-17T17:51:55.332334+0000","last_became_active":"2024-08-17T17:10:05.968856+0000","last_became_peered":"2024-08-17T17:10:05.968856+0000","last_unstale":"2024-08-17T17:51:55.332334+0000","last_undegraded":"2024-08-17T17:51:55.332334+0000","last_fullsized":"2024-08-17T17:51:55.332334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:25:49.550065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1272,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332594+0000","last_change":"2024-08-17T17:23:07.455919+0000","last_active":"2024-08-17T17:51:55.332594+0000","last_peered":"2024-08-17T17:51:55.332594+0000","last_clean":"2024-08-17T17:51:55.332594+0000","last_became_active":"2024-08-17T17:23:07.455284+0000","last_became_peered":"2024-08-17T17:23:07.455284+0000","last_unstale":"2024-08-17T17:51:55.332594+0000","last_undegraded":"2024-08-17T17:51:55.332594+0000","last_fullsized":"2024-08-17T17:51:55.332594+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:37:46.260775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1259,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.224555+0000","last_change":"2024-08-17T17:10:06.979492+0000","last_active":"2024-08-17T17:51:54.224555+0000","last_peered":"2024-08-17T17:51:54.224555+0000","last_clean":"2024-08-17T17:51:54.224555+0000","last_became_active":"2024-08-17T17:10:06.979158+0000","last_became_peered":"2024-08-17T17:10:06.979158+0000","last_unstale":"2024-08-17T17:51:54.224555+0000","last_undegraded":"2024-08-17T17:51:54.224555+0000","last_fullsized":"2024-08-17T17:51:54.224555+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:12:42.261365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1283,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.526004+0000","last_change":"2024-08-17T17:23:42.935099+0000","last_active":"2024-08-17T17:51:55.526004+0000","last_peered":"2024-08-17T17:51:55.526004+0000","last_clean":"2024-08-17T17:51:55.526004+0000","last_became_active":"2024-08-17T17:23:42.934501+0000","last_became_peered":"2024-08-17T17:23:42.934501+0000","last_unstale":"2024-08-17T17:51:55.526004+0000","last_undegraded":"2024-08-17T17:51:55.526004+0000","last_fullsized":"2024-08-17T17:51:55.526004+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:15:19.095713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":1902,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332593+0000","last_change":"2024-08-17T17:23:07.436763+0000","last_active":"2024-08-17T17:51:55.332593+0000","last_peered":"2024-08-17T17:51:55.332593+0000","last_clean":"2024-08-17T17:51:55.332593+0000","last_became_active":"2024-08-17T17:23:07.434720+0000","last_became_peered":"2024-08-17T17:23:07.434720+0000","last_unstale":"2024-08-17T17:51:55.332593+0000","last_undegraded":"2024-08-17T17:51:55.332593+0000","last_fullsized":"2024-08-17T17:51:55.332593+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:10:20.594115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1873,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.070209+0000","last_change":"2024-08-17T17:24:17.442329+0000","last_active":"2024-08-17T17:51:56.070209+0000","last_peered":"2024-08-17T17:51:56.070209+0000","last_clean":"2024-08-17T17:51:56.070209+0000","last_became_active":"2024-08-17T17:24:17.441666+0000","last_became_peered":"2024-08-17T17:24:17.441666+0000","last_unstale":"2024-08-17T17:51:56.070209+0000","last_undegraded":"2024-08-17T17:51:56.070209+0000","last_fullsized":"2024-08-17T17:51:56.070209+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:43:06.033194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1885,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565043+0000","last_change":"2024-08-17T17:23:07.428296+0000","last_active":"2024-08-17T17:51:55.565043+0000","last_peered":"2024-08-17T17:51:55.565043+0000","last_clean":"2024-08-17T17:51:55.565043+0000","last_became_active":"2024-08-17T17:23:07.428010+0000","last_became_peered":"2024-08-17T17:23:07.428010+0000","last_unstale":"2024-08-17T17:51:55.565043+0000","last_undegraded":"2024-08-17T17:51:55.565043+0000","last_fullsized":"2024-08-17T17:51:55.565043+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:45.459212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":1874,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.072123+0000","last_change":"2024-08-17T17:24:17.481368+0000","last_active":"2024-08-17T17:51:56.072123+0000","last_peered":"2024-08-17T17:51:56.072123+0000","last_clean":"2024-08-17T17:51:56.072123+0000","last_became_active":"2024-08-17T17:24:17.477509+0000","last_became_peered":"2024-08-17T17:24:17.477509+0000","last_unstale":"2024-08-17T17:51:56.072123+0000","last_undegraded":"2024-08-17T17:51:56.072123+0000","last_fullsized":"2024-08-17T17:51:56.072123+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:05:35.558004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1888,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332264+0000","last_change":"2024-08-17T17:22:32.884274+0000","last_active":"2024-08-17T17:51:55.332264+0000","last_peered":"2024-08-17T17:51:55.332264+0000","last_clean":"2024-08-17T17:51:55.332264+0000","last_became_active":"2024-08-17T17:22:32.874667+0000","last_became_peered":"2024-08-17T17:22:32.874667+0000","last_unstale":"2024-08-17T17:51:55.332264+0000","last_undegraded":"2024-08-17T17:51:55.332264+0000","last_fullsized":"2024-08-17T17:51:55.332264+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:22:05.834587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1281,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.557870+0000","last_change":"2024-08-17T17:22:32.870569+0000","last_active":"2024-08-17T17:51:55.557870+0000","last_peered":"2024-08-17T17:51:55.557870+0000","last_clean":"2024-08-17T17:51:55.557870+0000","last_became_active":"2024-08-17T17:22:32.869335+0000","last_became_peered":"2024-08-17T17:22:32.869335+0000","last_unstale":"2024-08-17T17:51:55.557870+0000","last_undegraded":"2024-08-17T17:51:55.557870+0000","last_fullsized":"2024-08-17T17:51:55.557870+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:24.969174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1907,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.566553+0000","last_change":"2024-08-17T17:23:42.935825+0000","last_active":"2024-08-17T17:51:55.566553+0000","last_peered":"2024-08-17T17:51:55.566553+0000","last_clean":"2024-08-17T17:51:55.566553+0000","last_became_active":"2024-08-17T17:23:42.935150+0000","last_became_peered":"2024-08-17T17:23:42.935150+0000","last_unstale":"2024-08-17T17:51:55.566553+0000","last_undegraded":"2024-08-17T17:51:55.566553+0000","last_fullsized":"2024-08-17T17:51:55.566553+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:57:19.577644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1895,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565154+0000","last_change":"2024-08-17T17:23:07.431586+0000","last_active":"2024-08-17T17:51:55.565154+0000","last_peered":"2024-08-17T17:51:55.565154+0000","last_clean":"2024-08-17T17:51:55.565154+0000","last_became_active":"2024-08-17T17:23:07.431301+0000","last_became_peered":"2024-08-17T17:23:07.431301+0000","last_unstale":"2024-08-17T17:51:55.565154+0000","last_undegraded":"2024-08-17T17:51:55.565154+0000","last_fullsized":"2024-08-17T17:51:55.565154+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:28:18.706489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1923,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565354+0000","last_change":"2024-08-17T17:23:07.420550+0000","last_active":"2024-08-17T17:51:55.565354+0000","last_peered":"2024-08-17T17:51:55.565354+0000","last_clean":"2024-08-17T17:51:55.565354+0000","last_became_active":"2024-08-17T17:23:07.420223+0000","last_became_peered":"2024-08-17T17:23:07.420223+0000","last_unstale":"2024-08-17T17:51:55.565354+0000","last_undegraded":"2024-08-17T17:51:55.565354+0000","last_fullsized":"2024-08-17T17:51:55.565354+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:33:46.996371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1285,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.537366+0000","last_change":"2024-08-17T17:24:17.426267+0000","last_active":"2024-08-17T17:51:55.537366+0000","last_peered":"2024-08-17T17:51:55.537366+0000","last_clean":"2024-08-17T17:51:55.537366+0000","last_became_active":"2024-08-17T17:24:17.425925+0000","last_became_peered":"2024-08-17T17:24:17.425925+0000","last_unstale":"2024-08-17T17:51:55.537366+0000","last_undegraded":"2024-08-17T17:51:55.537366+0000","last_fullsized":"2024-08-17T17:51:55.537366+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:24:25.017856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1281,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.537016+0000","last_change":"2024-08-17T17:24:17.474138+0000","last_active":"2024-08-17T17:51:55.537016+0000","last_peered":"2024-08-17T17:51:55.537016+0000","last_clean":"2024-08-17T17:51:55.537016+0000","last_became_active":"2024-08-17T17:24:17.473708+0000","last_became_peered":"2024-08-17T17:24:17.473708+0000","last_unstale":"2024-08-17T17:51:55.537016+0000","last_undegraded":"2024-08-17T17:51:55.537016+0000","last_fullsized":"2024-08-17T17:51:55.537016+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:10:57.768067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1275,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.225254+0000","last_change":"2024-08-17T17:22:32.871533+0000","last_active":"2024-08-17T17:51:54.225254+0000","last_peered":"2024-08-17T17:51:54.225254+0000","last_clean":"2024-08-17T17:51:54.225254+0000","last_became_active":"2024-08-17T17:22:32.868226+0000","last_became_peered":"2024-08-17T17:22:32.868226+0000","last_unstale":"2024-08-17T17:51:54.225254+0000","last_undegraded":"2024-08-17T17:51:54.225254+0000","last_fullsized":"2024-08-17T17:51:54.225254+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:00:02.629455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":1285,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.537019+0000","last_change":"2024-08-17T17:23:07.391021+0000","last_active":"2024-08-17T17:51:55.537019+0000","last_peered":"2024-08-17T17:51:55.537019+0000","last_clean":"2024-08-17T17:51:55.537019+0000","last_became_active":"2024-08-17T17:23:07.390551+0000","last_became_peered":"2024-08-17T17:23:07.390551+0000","last_unstale":"2024-08-17T17:51:55.537019+0000","last_undegraded":"2024-08-17T17:51:55.537019+0000","last_fullsized":"2024-08-17T17:51:55.537019+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:06.239013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001023532,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1906,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565011+0000","last_change":"2024-08-17T17:23:07.406362+0000","last_active":"2024-08-17T17:51:55.565011+0000","last_peered":"2024-08-17T17:51:55.565011+0000","last_clean":"2024-08-17T17:51:55.565011+0000","last_became_active":"2024-08-17T17:23:07.406071+0000","last_became_peered":"2024-08-17T17:23:07.406071+0000","last_unstale":"2024-08-17T17:51:55.565011+0000","last_undegraded":"2024-08-17T17:51:55.565011+0000","last_fullsized":"2024-08-17T17:51:55.565011+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:45:47.383432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1886,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565198+0000","last_change":"2024-08-17T17:23:42.910246+0000","last_active":"2024-08-17T17:51:55.565198+0000","last_peered":"2024-08-17T17:51:55.565198+0000","last_clean":"2024-08-17T17:51:55.565198+0000","last_became_active":"2024-08-17T17:23:42.909929+0000","last_became_peered":"2024-08-17T17:23:42.909929+0000","last_unstale":"2024-08-17T17:51:55.565198+0000","last_undegraded":"2024-08-17T17:51:55.565198+0000","last_fullsized":"2024-08-17T17:51:55.565198+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:14:45.210558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1916,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.070318+0000","last_change":"2024-08-17T17:24:17.438891+0000","last_active":"2024-08-17T17:51:56.070318+0000","last_peered":"2024-08-17T17:51:56.070318+0000","last_clean":"2024-08-17T17:51:56.070318+0000","last_became_active":"2024-08-17T17:24:17.437714+0000","last_became_peered":"2024-08-17T17:24:17.437714+0000","last_unstale":"2024-08-17T17:51:56.070318+0000","last_undegraded":"2024-08-17T17:51:56.070318+0000","last_fullsized":"2024-08-17T17:51:56.070318+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:58.825017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1909,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332494+0000","last_change":"2024-08-17T17:23:42.908634+0000","last_active":"2024-08-17T17:51:55.332494+0000","last_peered":"2024-08-17T17:51:55.332494+0000","last_clean":"2024-08-17T17:51:55.332494+0000","last_became_active":"2024-08-17T17:23:42.907675+0000","last_became_peered":"2024-08-17T17:23:42.907675+0000","last_unstale":"2024-08-17T17:51:55.332494+0000","last_undegraded":"2024-08-17T17:51:55.332494+0000","last_fullsized":"2024-08-17T17:51:55.332494+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:03:37.612222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1899,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.331258+0000","last_change":"2024-08-17T17:24:17.457559+0000","last_active":"2024-08-17T17:51:55.331258+0000","last_peered":"2024-08-17T17:51:55.331258+0000","last_clean":"2024-08-17T17:51:55.331258+0000","last_became_active":"2024-08-17T17:24:17.457262+0000","last_became_peered":"2024-08-17T17:24:17.457262+0000","last_unstale":"2024-08-17T17:51:55.331258+0000","last_undegraded":"2024-08-17T17:51:55.331258+0000","last_fullsized":"2024-08-17T17:51:55.331258+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:53:38.137277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1296,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.594253+0000","last_change":"2024-08-17T17:23:07.450509+0000","last_active":"2024-08-17T17:51:54.594253+0000","last_peered":"2024-08-17T17:51:54.594253+0000","last_clean":"2024-08-17T17:51:54.594253+0000","last_became_active":"2024-08-17T17:23:07.450182+0000","last_became_peered":"2024-08-17T17:23:07.450182+0000","last_unstale":"2024-08-17T17:51:54.594253+0000","last_undegraded":"2024-08-17T17:51:54.594253+0000","last_fullsized":"2024-08-17T17:51:54.594253+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:01:18.656656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1255,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.526524+0000","last_change":"2024-08-17T17:10:06.995023+0000","last_active":"2024-08-17T17:51:55.526524+0000","last_peered":"2024-08-17T17:51:55.526524+0000","last_clean":"2024-08-17T17:51:55.526524+0000","last_became_active":"2024-08-17T17:10:06.994757+0000","last_became_peered":"2024-08-17T17:10:06.994757+0000","last_unstale":"2024-08-17T17:51:55.526524+0000","last_undegraded":"2024-08-17T17:51:55.526524+0000","last_fullsized":"2024-08-17T17:51:55.526524+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:31:52.163518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1839,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.071057+0000","last_change":"2024-08-17T17:24:17.468443+0000","last_active":"2024-08-17T17:51:56.071057+0000","last_peered":"2024-08-17T17:51:56.071057+0000","last_clean":"2024-08-17T17:51:56.071057+0000","last_became_active":"2024-08-17T17:24:17.467394+0000","last_became_peered":"2024-08-17T17:24:17.467394+0000","last_unstale":"2024-08-17T17:51:56.071057+0000","last_undegraded":"2024-08-17T17:51:56.071057+0000","last_fullsized":"2024-08-17T17:51:56.071057+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:13.024253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":1284,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.537434+0000","last_change":"2024-08-17T17:23:42.920552+0000","last_active":"2024-08-17T17:51:55.537434+0000","last_peered":"2024-08-17T17:51:55.537434+0000","last_clean":"2024-08-17T17:51:55.537434+0000","last_became_active":"2024-08-17T17:23:42.915824+0000","last_became_peered":"2024-08-17T17:23:42.915824+0000","last_unstale":"2024-08-17T17:51:55.537434+0000","last_undegraded":"2024-08-17T17:51:55.537434+0000","last_fullsized":"2024-08-17T17:51:55.537434+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:25:18.769504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041737399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":1597,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.556550+0000","last_change":"2024-08-17T17:22:32.867754+0000","last_active":"2024-08-17T17:51:55.556550+0000","last_peered":"2024-08-17T17:51:55.556550+0000","last_clean":"2024-08-17T17:51:55.556550+0000","last_became_active":"2024-08-17T17:22:32.867345+0000","last_became_peered":"2024-08-17T17:22:32.867345+0000","last_unstale":"2024-08-17T17:51:55.556550+0000","last_undegraded":"2024-08-17T17:51:55.556550+0000","last_fullsized":"2024-08-17T17:51:55.556550+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T17:06:21.174471+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:00:42.776037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1885,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565625+0000","last_change":"2024-08-17T17:24:17.455710+0000","last_active":"2024-08-17T17:51:55.565625+0000","last_peered":"2024-08-17T17:51:55.565625+0000","last_clean":"2024-08-17T17:51:55.565625+0000","last_became_active":"2024-08-17T17:24:17.452989+0000","last_became_peered":"2024-08-17T17:24:17.452989+0000","last_unstale":"2024-08-17T17:51:55.565625+0000","last_undegraded":"2024-08-17T17:51:55.565625+0000","last_fullsized":"2024-08-17T17:51:55.565625+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:58:07.894968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1880,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.070034+0000","last_change":"2024-08-17T17:24:17.453639+0000","last_active":"2024-08-17T17:51:56.070034+0000","last_peered":"2024-08-17T17:51:56.070034+0000","last_clean":"2024-08-17T17:51:56.070034+0000","last_became_active":"2024-08-17T17:24:17.443638+0000","last_became_peered":"2024-08-17T17:24:17.443638+0000","last_unstale":"2024-08-17T17:51:56.070034+0000","last_undegraded":"2024-08-17T17:51:56.070034+0000","last_fullsized":"2024-08-17T17:51:56.070034+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:42.107479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1272,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.225392+0000","last_change":"2024-08-17T17:23:07.451866+0000","last_active":"2024-08-17T17:51:54.225392+0000","last_peered":"2024-08-17T17:51:54.225392+0000","last_clean":"2024-08-17T17:51:54.225392+0000","last_became_active":"2024-08-17T17:23:07.451553+0000","last_became_peered":"2024-08-17T17:23:07.451553+0000","last_unstale":"2024-08-17T17:51:54.225392+0000","last_undegraded":"2024-08-17T17:51:54.225392+0000","last_fullsized":"2024-08-17T17:51:54.225392+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:37:23.930651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1274,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.224623+0000","last_change":"2024-08-17T17:23:42.915851+0000","last_active":"2024-08-17T17:51:54.224623+0000","last_peered":"2024-08-17T17:51:54.224623+0000","last_clean":"2024-08-17T17:51:54.224623+0000","last_became_active":"2024-08-17T17:23:42.915552+0000","last_became_peered":"2024-08-17T17:23:42.915552+0000","last_unstale":"2024-08-17T17:51:54.224623+0000","last_undegraded":"2024-08-17T17:51:54.224623+0000","last_fullsized":"2024-08-17T17:51:54.224623+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:45.197876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1287,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.225154+0000","last_change":"2024-08-17T17:23:42.930803+0000","last_active":"2024-08-17T17:51:54.225154+0000","last_peered":"2024-08-17T17:51:54.225154+0000","last_clean":"2024-08-17T17:51:54.225154+0000","last_became_active":"2024-08-17T17:23:42.930458+0000","last_became_peered":"2024-08-17T17:23:42.930458+0000","last_unstale":"2024-08-17T17:51:54.225154+0000","last_undegraded":"2024-08-17T17:51:54.225154+0000","last_fullsized":"2024-08-17T17:51:54.225154+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:40.816706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1270,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.477299+0000","last_change":"2024-08-17T17:23:07.401624+0000","last_active":"2024-08-17T17:51:54.477299+0000","last_peered":"2024-08-17T17:51:54.477299+0000","last_clean":"2024-08-17T17:51:54.477299+0000","last_became_active":"2024-08-17T17:23:07.400222+0000","last_became_peered":"2024-08-17T17:23:07.400222+0000","last_unstale":"2024-08-17T17:51:54.477299+0000","last_undegraded":"2024-08-17T17:51:54.477299+0000","last_fullsized":"2024-08-17T17:51:54.477299+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:07.069989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1272,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.224982+0000","last_change":"2024-08-17T17:24:17.470809+0000","last_active":"2024-08-17T17:51:54.224982+0000","last_peered":"2024-08-17T17:51:54.224982+0000","last_clean":"2024-08-17T17:51:54.224982+0000","last_became_active":"2024-08-17T17:24:17.469753+0000","last_became_peered":"2024-08-17T17:24:17.469753+0000","last_unstale":"2024-08-17T17:51:54.224982+0000","last_undegraded":"2024-08-17T17:51:54.224982+0000","last_fullsized":"2024-08-17T17:51:54.224982+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:00.049425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":1873,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.069486+0000","last_change":"2024-08-17T17:24:17.461258+0000","last_active":"2024-08-17T17:51:56.069486+0000","last_peered":"2024-08-17T17:51:56.069486+0000","last_clean":"2024-08-17T17:51:56.069486+0000","last_became_active":"2024-08-17T17:24:17.460895+0000","last_became_peered":"2024-08-17T17:24:17.460895+0000","last_unstale":"2024-08-17T17:51:56.069486+0000","last_undegraded":"2024-08-17T17:51:56.069486+0000","last_fullsized":"2024-08-17T17:51:56.069486+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:43:24.208259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1912,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.331877+0000","last_change":"2024-08-17T17:23:42.901638+0000","last_active":"2024-08-17T17:51:55.331877+0000","last_peered":"2024-08-17T17:51:55.331877+0000","last_clean":"2024-08-17T17:51:55.331877+0000","last_became_active":"2024-08-17T17:23:42.901335+0000","last_became_peered":"2024-08-17T17:23:42.901335+0000","last_unstale":"2024-08-17T17:51:55.331877+0000","last_undegraded":"2024-08-17T17:51:55.331877+0000","last_fullsized":"2024-08-17T17:51:55.331877+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:25:35.332839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1268,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.532827+0000","last_change":"2024-08-17T17:24:17.473913+0000","last_active":"2024-08-17T17:51:54.532827+0000","last_peered":"2024-08-17T17:51:54.532827+0000","last_clean":"2024-08-17T17:51:54.532827+0000","last_became_active":"2024-08-17T17:24:17.473195+0000","last_became_peered":"2024-08-17T17:24:17.473195+0000","last_unstale":"2024-08-17T17:51:54.532827+0000","last_undegraded":"2024-08-17T17:51:54.532827+0000","last_fullsized":"2024-08-17T17:51:54.532827+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:12.998033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1269,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.477272+0000","last_change":"2024-08-17T17:23:07.402010+0000","last_active":"2024-08-17T17:51:54.477272+0000","last_peered":"2024-08-17T17:51:54.477272+0000","last_clean":"2024-08-17T17:51:54.477272+0000","last_became_active":"2024-08-17T17:23:07.400849+0000","last_became_peered":"2024-08-17T17:23:07.400849+0000","last_unstale":"2024-08-17T17:51:54.477272+0000","last_undegraded":"2024-08-17T17:51:54.477272+0000","last_fullsized":"2024-08-17T17:51:54.477272+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:16:02.382257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":1252,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.594877+0000","last_change":"2024-08-17T17:10:05.973491+0000","last_active":"2024-08-17T17:51:54.594877+0000","last_peered":"2024-08-17T17:51:54.594877+0000","last_clean":"2024-08-17T17:51:54.594877+0000","last_became_active":"2024-08-17T17:10:05.973228+0000","last_became_peered":"2024-08-17T17:10:05.973228+0000","last_unstale":"2024-08-17T17:51:54.594877+0000","last_undegraded":"2024-08-17T17:51:54.594877+0000","last_fullsized":"2024-08-17T17:51:54.594877+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:54.310135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1288,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.224430+0000","last_change":"2024-08-17T17:24:17.460203+0000","last_active":"2024-08-17T17:51:54.224430+0000","last_peered":"2024-08-17T17:51:54.224430+0000","last_clean":"2024-08-17T17:51:54.224430+0000","last_became_active":"2024-08-17T17:24:17.459682+0000","last_became_peered":"2024-08-17T17:24:17.459682+0000","last_unstale":"2024-08-17T17:51:54.224430+0000","last_undegraded":"2024-08-17T17:51:54.224430+0000","last_fullsized":"2024-08-17T17:51:54.224430+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:58.703700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":1268,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.594587+0000","last_change":"2024-08-17T17:23:07.454042+0000","last_active":"2024-08-17T17:51:54.594587+0000","last_peered":"2024-08-17T17:51:54.594587+0000","last_clean":"2024-08-17T17:51:54.594587+0000","last_became_active":"2024-08-17T17:23:07.453712+0000","last_became_peered":"2024-08-17T17:23:07.453712+0000","last_unstale":"2024-08-17T17:51:54.594587+0000","last_undegraded":"2024-08-17T17:51:54.594587+0000","last_fullsized":"2024-08-17T17:51:54.594587+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:03:07.643456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1253,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.532272+0000","last_change":"2024-08-17T17:10:06.994799+0000","last_active":"2024-08-17T17:51:54.532272+0000","last_peered":"2024-08-17T17:51:54.532272+0000","last_clean":"2024-08-17T17:51:54.532272+0000","last_became_active":"2024-08-17T17:10:06.994150+0000","last_became_peered":"2024-08-17T17:10:06.994150+0000","last_unstale":"2024-08-17T17:51:54.532272+0000","last_undegraded":"2024-08-17T17:51:54.532272+0000","last_fullsized":"2024-08-17T17:51:54.532272+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:50:47.293091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":1266,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.594586+0000","last_change":"2024-08-17T17:22:32.873334+0000","last_active":"2024-08-17T17:51:54.594586+0000","last_peered":"2024-08-17T17:51:54.594586+0000","last_clean":"2024-08-17T17:51:54.594586+0000","last_became_active":"2024-08-17T17:22:32.870147+0000","last_became_peered":"2024-08-17T17:22:32.870147+0000","last_unstale":"2024-08-17T17:51:54.594586+0000","last_undegraded":"2024-08-17T17:51:54.594586+0000","last_fullsized":"2024-08-17T17:51:54.594586+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:18:03.750592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1277,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.524676+0000","last_change":"2024-08-17T17:22:32.847989+0000","last_active":"2024-08-17T17:51:55.524676+0000","last_peered":"2024-08-17T17:51:55.524676+0000","last_clean":"2024-08-17T17:51:55.524676+0000","last_became_active":"2024-08-17T17:22:32.846593+0000","last_became_peered":"2024-08-17T17:22:32.846593+0000","last_unstale":"2024-08-17T17:51:55.524676+0000","last_undegraded":"2024-08-17T17:51:55.524676+0000","last_fullsized":"2024-08-17T17:51:55.524676+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:33:50.370124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1877,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.071934+0000","last_change":"2024-08-17T17:24:17.451528+0000","last_active":"2024-08-17T17:51:56.071934+0000","last_peered":"2024-08-17T17:51:56.071934+0000","last_clean":"2024-08-17T17:51:56.071934+0000","last_became_active":"2024-08-17T17:24:17.451246+0000","last_became_peered":"2024-08-17T17:24:17.451246+0000","last_unstale":"2024-08-17T17:51:56.071934+0000","last_undegraded":"2024-08-17T17:51:56.071934+0000","last_fullsized":"2024-08-17T17:51:56.071934+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:38:18.176559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1251,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.557291+0000","last_change":"2024-08-17T17:10:05.975692+0000","last_active":"2024-08-17T17:51:55.557291+0000","last_peered":"2024-08-17T17:51:55.557291+0000","last_clean":"2024-08-17T17:51:55.557291+0000","last_became_active":"2024-08-17T17:10:05.975057+0000","last_became_peered":"2024-08-17T17:10:05.975057+0000","last_unstale":"2024-08-17T17:51:55.557291+0000","last_undegraded":"2024-08-17T17:51:55.557291+0000","last_fullsized":"2024-08-17T17:51:55.557291+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:44:53.106104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1896,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.331881+0000","last_change":"2024-08-17T17:24:17.460371+0000","last_active":"2024-08-17T17:51:55.331881+0000","last_peered":"2024-08-17T17:51:55.331881+0000","last_clean":"2024-08-17T17:51:55.331881+0000","last_became_active":"2024-08-17T17:24:17.460074+0000","last_became_peered":"2024-08-17T17:24:17.460074+0000","last_unstale":"2024-08-17T17:51:55.331881+0000","last_undegraded":"2024-08-17T17:51:55.331881+0000","last_fullsized":"2024-08-17T17:51:55.331881+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:08:48.820168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1896,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332180+0000","last_change":"2024-08-17T17:24:17.457942+0000","last_active":"2024-08-17T17:51:55.332180+0000","last_peered":"2024-08-17T17:51:55.332180+0000","last_clean":"2024-08-17T17:51:55.332180+0000","last_became_active":"2024-08-17T17:24:17.456475+0000","last_became_peered":"2024-08-17T17:24:17.456475+0000","last_unstale":"2024-08-17T17:51:55.332180+0000","last_undegraded":"2024-08-17T17:51:55.332180+0000","last_fullsized":"2024-08-17T17:51:55.332180+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:12.325657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1266,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.532706+0000","last_change":"2024-08-17T17:24:17.451287+0000","last_active":"2024-08-17T17:51:54.532706+0000","last_peered":"2024-08-17T17:51:54.532706+0000","last_clean":"2024-08-17T17:51:54.532706+0000","last_became_active":"2024-08-17T17:24:17.450951+0000","last_became_peered":"2024-08-17T17:24:17.450951+0000","last_unstale":"2024-08-17T17:51:54.532706+0000","last_undegraded":"2024-08-17T17:51:54.532706+0000","last_fullsized":"2024-08-17T17:51:54.532706+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:13:37.590108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":1268,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.556246+0000","last_change":"2024-08-17T17:23:42.907970+0000","last_active":"2024-08-17T17:51:55.556246+0000","last_peered":"2024-08-17T17:51:55.556246+0000","last_clean":"2024-08-17T17:51:55.556246+0000","last_became_active":"2024-08-17T17:23:42.907595+0000","last_became_peered":"2024-08-17T17:23:42.907595+0000","last_unstale":"2024-08-17T17:51:55.556246+0000","last_undegraded":"2024-08-17T17:51:55.556246+0000","last_fullsized":"2024-08-17T17:51:55.556246+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:11:46.130004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1878,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.072237+0000","last_change":"2024-08-17T17:24:17.486591+0000","last_active":"2024-08-17T17:51:56.072237+0000","last_peered":"2024-08-17T17:51:56.072237+0000","last_clean":"2024-08-17T17:51:56.072237+0000","last_became_active":"2024-08-17T17:24:17.486320+0000","last_became_peered":"2024-08-17T17:24:17.486320+0000","last_unstale":"2024-08-17T17:51:56.072237+0000","last_undegraded":"2024-08-17T17:51:56.072237+0000","last_fullsized":"2024-08-17T17:51:56.072237+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:03.249803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234521,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":120032,"kb_used_data":16792,"kb_used_omap":1,"kb_used_meta":103230,"kb_avail":93625120,"statfs":{"total":95995035648,"available":95872122880,"internally_reserved":0,"allocated":17195008,"data_stored":10837596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":105707979},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496159,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":146268,"kb_used_data":16784,"kb_used_omap":1,"kb_used_meta":129470,"kb_avail":93598884,"statfs":{"total":95995035648,"available":95845257216,"internally_reserved":0,"allocated":17186816,"data_stored":10837469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":132577735},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565400,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":92740,"kb_used_data":16188,"kb_used_omap":72,"kb_used_meta":76471,"kb_avail":93652412,"statfs":{"total":95995035648,"available":95900069888,"internally_reserved":0,"allocated":16576512,"data_stored":10242358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":78306798},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":674,"seq":2894807957855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70540,"kb_used_data":16192,"kb_used_omap":74,"kb_used_meta":54261,"kb_avail":93674612,"statfs":{"total":95995035648,"available":95922802688,"internally_reserved":0,"allocated":16580608,"data_stored":10242884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76116,"internal_metadata":55563948},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743186790,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70472,"kb_used_data":16188,"kb_used_omap":66,"kb_used_meta":54205,"kb_avail":93674680,"statfs":{"total":95995035648,"available":95922872320,"internally_reserved":0,"allocated":16576512,"data_stored":10242287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":55506603},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448429,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88916,"kb_used_data":16772,"kb_used_omap":75,"kb_used_meta":72052,"kb_avail":93656236,"statfs":{"total":95995035648,"available":95903985664,"internally_reserved":0,"allocated":17174528,"data_stored":10833240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":73781439},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757795,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":127700,"kb_used_data":16196,"kb_used_omap":1,"kb_used_meta":111486,"kb_avail":93617452,"statfs":{"total":95995035648,"available":95864270848,"internally_reserved":0,"allocated":16584704,"data_stored":10242927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114162118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019432,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":141600,"kb_used_data":16208,"kb_used_omap":1,"kb_used_meta":125374,"kb_avail":93603552,"statfs":{"total":95995035648,"available":95850037248,"internally_reserved":0,"allocated":16596992,"data_stored":10247158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128383430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:52:52.048 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:52:52.252 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:52.668 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:52 smithi136 ceph-mon[109861]: pgmap v2061: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:52.668 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:52 smithi136 ceph-mon[109861]: from='client.35034 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:52:52.668 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:52 smithi136 ceph-mon[115078]: pgmap v2061: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:52.669 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:52 smithi136 ceph-mon[115078]: from='client.35034 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:52:52.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:52 smithi165 ceph-mon[105517]: pgmap v2061: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:52.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:52 smithi165 ceph-mon[105517]: from='client.35034 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:52:54.151 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:52:54.152 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T17:52:54.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:54 smithi136 ceph-mon[115078]: pgmap v2062: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:54.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:54 smithi136 ceph-mon[109861]: pgmap v2062: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:54.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:54 smithi165 ceph-mon[105517]: pgmap v2062: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:54.770 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":2062,"stamp":"2024-08-17T17:52:52.569511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595941,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":279,"num_write_kb":2449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":858268,"kb_used_data":131320,"kb_used_omap":295,"kb_used_meta":726552,"kb_avail":749102948,"statfs":{"total":767960285184,"available":767081418752,"internally_reserved":0,"allocated":134471680,"data_stored":83725919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":302302,"internal_metadata":743990050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002381"},"pg_stats":[{"pgid":"3.1f","version":"70'4","reported_seq":1906,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.333168+0000","last_change":"2024-08-17T17:23:42.937520+0000","last_active":"2024-08-17T17:51:55.333168+0000","last_peered":"2024-08-17T17:51:55.333168+0000","last_clean":"2024-08-17T17:51:55.333168+0000","last_became_active":"2024-08-17T17:23:42.937174+0000","last_became_peered":"2024-08-17T17:23:42.937174+0000","last_unstale":"2024-08-17T17:51:55.333168+0000","last_undegraded":"2024-08-17T17:51:55.333168+0000","last_fullsized":"2024-08-17T17:51:55.333168+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:47:56.679612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1271,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.224325+0000","last_change":"2024-08-17T17:23:07.398676+0000","last_active":"2024-08-17T17:51:54.224325+0000","last_peered":"2024-08-17T17:51:54.224325+0000","last_clean":"2024-08-17T17:51:54.224325+0000","last_became_active":"2024-08-17T17:23:07.398369+0000","last_became_peered":"2024-08-17T17:23:07.398369+0000","last_unstale":"2024-08-17T17:51:54.224325+0000","last_undegraded":"2024-08-17T17:51:54.224325+0000","last_fullsized":"2024-08-17T17:51:54.224325+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:55:04.281412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":1890,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.071332+0000","last_change":"2024-08-17T17:24:17.476168+0000","last_active":"2024-08-17T17:51:56.071332+0000","last_peered":"2024-08-17T17:51:56.071332+0000","last_clean":"2024-08-17T17:51:56.071332+0000","last_became_active":"2024-08-17T17:24:17.465845+0000","last_became_peered":"2024-08-17T17:24:17.465845+0000","last_unstale":"2024-08-17T17:51:56.071332+0000","last_undegraded":"2024-08-17T17:51:56.071332+0000","last_fullsized":"2024-08-17T17:51:56.071332+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:15:01.494447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1906,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.071442+0000","last_change":"2024-08-17T17:24:17.461772+0000","last_active":"2024-08-17T17:51:56.071442+0000","last_peered":"2024-08-17T17:51:56.071442+0000","last_clean":"2024-08-17T17:51:56.071442+0000","last_became_active":"2024-08-17T17:24:17.460806+0000","last_became_peered":"2024-08-17T17:24:17.460806+0000","last_unstale":"2024-08-17T17:51:56.071442+0000","last_undegraded":"2024-08-17T17:51:56.071442+0000","last_fullsized":"2024-08-17T17:51:56.071442+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:01:16.453148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":1254,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.536641+0000","last_change":"2024-08-17T17:10:05.965176+0000","last_active":"2024-08-17T17:51:55.536641+0000","last_peered":"2024-08-17T17:51:55.536641+0000","last_clean":"2024-08-17T17:51:55.536641+0000","last_became_active":"2024-08-17T17:10:05.964861+0000","last_became_peered":"2024-08-17T17:10:05.964861+0000","last_unstale":"2024-08-17T17:51:55.536641+0000","last_undegraded":"2024-08-17T17:51:55.536641+0000","last_fullsized":"2024-08-17T17:51:55.536641+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:59:04.277926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1286,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.330974+0000","last_change":"2024-08-17T17:24:17.462527+0000","last_active":"2024-08-17T17:51:55.330974+0000","last_peered":"2024-08-17T17:51:55.330974+0000","last_clean":"2024-08-17T17:51:55.330974+0000","last_became_active":"2024-08-17T17:24:17.462240+0000","last_became_peered":"2024-08-17T17:24:17.462240+0000","last_unstale":"2024-08-17T17:51:55.330974+0000","last_undegraded":"2024-08-17T17:51:55.330974+0000","last_fullsized":"2024-08-17T17:51:55.330974+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:40:38.063717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":1269,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.536271+0000","last_change":"2024-08-17T17:23:07.391038+0000","last_active":"2024-08-17T17:51:55.536271+0000","last_peered":"2024-08-17T17:51:55.536271+0000","last_clean":"2024-08-17T17:51:55.536271+0000","last_became_active":"2024-08-17T17:23:07.390571+0000","last_became_peered":"2024-08-17T17:23:07.390571+0000","last_unstale":"2024-08-17T17:51:55.536271+0000","last_undegraded":"2024-08-17T17:51:55.536271+0000","last_fullsized":"2024-08-17T17:51:55.536271+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:18:41.791740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1286,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332513+0000","last_change":"2024-08-17T17:24:17.458168+0000","last_active":"2024-08-17T17:51:55.332513+0000","last_peered":"2024-08-17T17:51:55.332513+0000","last_clean":"2024-08-17T17:51:55.332513+0000","last_became_active":"2024-08-17T17:24:17.457847+0000","last_became_peered":"2024-08-17T17:24:17.457847+0000","last_unstale":"2024-08-17T17:51:55.332513+0000","last_undegraded":"2024-08-17T17:51:55.332513+0000","last_fullsized":"2024-08-17T17:51:55.332513+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:28.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":1889,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332046+0000","last_change":"2024-08-17T17:22:32.883756+0000","last_active":"2024-08-17T17:51:55.332046+0000","last_peered":"2024-08-17T17:51:55.332046+0000","last_clean":"2024-08-17T17:51:55.332046+0000","last_became_active":"2024-08-17T17:22:32.874867+0000","last_became_peered":"2024-08-17T17:22:32.874867+0000","last_unstale":"2024-08-17T17:51:55.332046+0000","last_undegraded":"2024-08-17T17:51:55.332046+0000","last_fullsized":"2024-08-17T17:51:55.332046+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:52:27.375231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1908,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.069811+0000","last_change":"2024-08-17T17:24:17.456622+0000","last_active":"2024-08-17T17:51:56.069811+0000","last_peered":"2024-08-17T17:51:56.069811+0000","last_clean":"2024-08-17T17:51:56.069811+0000","last_became_active":"2024-08-17T17:24:17.455396+0000","last_became_peered":"2024-08-17T17:24:17.455396+0000","last_unstale":"2024-08-17T17:51:56.069811+0000","last_undegraded":"2024-08-17T17:51:56.069811+0000","last_fullsized":"2024-08-17T17:51:56.069811+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:09.564331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1288,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.331131+0000","last_change":"2024-08-17T17:23:42.930464+0000","last_active":"2024-08-17T17:51:55.331131+0000","last_peered":"2024-08-17T17:51:55.331131+0000","last_clean":"2024-08-17T17:51:55.331131+0000","last_became_active":"2024-08-17T17:23:42.929949+0000","last_became_peered":"2024-08-17T17:23:42.929949+0000","last_unstale":"2024-08-17T17:51:55.331131+0000","last_undegraded":"2024-08-17T17:51:55.331131+0000","last_fullsized":"2024-08-17T17:51:55.331131+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:48:27.092339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1893,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.070329+0000","last_change":"2024-08-17T17:24:17.452009+0000","last_active":"2024-08-17T17:51:56.070329+0000","last_peered":"2024-08-17T17:51:56.070329+0000","last_clean":"2024-08-17T17:51:56.070329+0000","last_became_active":"2024-08-17T17:24:17.451492+0000","last_became_peered":"2024-08-17T17:24:17.451492+0000","last_unstale":"2024-08-17T17:51:56.070329+0000","last_undegraded":"2024-08-17T17:51:56.070329+0000","last_fullsized":"2024-08-17T17:51:56.070329+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:21:24.699633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1904,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565316+0000","last_change":"2024-08-17T17:24:17.452025+0000","last_active":"2024-08-17T17:51:55.565316+0000","last_peered":"2024-08-17T17:51:55.565316+0000","last_clean":"2024-08-17T17:51:55.565316+0000","last_became_active":"2024-08-17T17:24:17.448029+0000","last_became_peered":"2024-08-17T17:24:17.448029+0000","last_unstale":"2024-08-17T17:51:55.565316+0000","last_undegraded":"2024-08-17T17:51:55.565316+0000","last_fullsized":"2024-08-17T17:51:55.565316+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:45.089346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1900,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332332+0000","last_change":"2024-08-17T17:24:17.469096+0000","last_active":"2024-08-17T17:51:55.332332+0000","last_peered":"2024-08-17T17:51:55.332332+0000","last_clean":"2024-08-17T17:51:55.332332+0000","last_became_active":"2024-08-17T17:24:17.468669+0000","last_became_peered":"2024-08-17T17:24:17.468669+0000","last_unstale":"2024-08-17T17:51:55.332332+0000","last_undegraded":"2024-08-17T17:51:55.332332+0000","last_fullsized":"2024-08-17T17:51:55.332332+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:03:39.427341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1893,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.070083+0000","last_change":"2024-08-17T17:24:17.453946+0000","last_active":"2024-08-17T17:51:56.070083+0000","last_peered":"2024-08-17T17:51:56.070083+0000","last_clean":"2024-08-17T17:51:56.070083+0000","last_became_active":"2024-08-17T17:24:17.445004+0000","last_became_peered":"2024-08-17T17:24:17.445004+0000","last_unstale":"2024-08-17T17:51:56.070083+0000","last_undegraded":"2024-08-17T17:51:56.070083+0000","last_fullsized":"2024-08-17T17:51:56.070083+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:00:19.366629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1270,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.477362+0000","last_change":"2024-08-17T17:23:07.402338+0000","last_active":"2024-08-17T17:51:54.477362+0000","last_peered":"2024-08-17T17:51:54.477362+0000","last_clean":"2024-08-17T17:51:54.477362+0000","last_became_active":"2024-08-17T17:23:07.401996+0000","last_became_peered":"2024-08-17T17:23:07.401996+0000","last_unstale":"2024-08-17T17:51:54.477362+0000","last_undegraded":"2024-08-17T17:51:54.477362+0000","last_fullsized":"2024-08-17T17:51:54.477362+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:39:21.566786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1878,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.070289+0000","last_change":"2024-08-17T17:24:17.442338+0000","last_active":"2024-08-17T17:51:56.070289+0000","last_peered":"2024-08-17T17:51:56.070289+0000","last_clean":"2024-08-17T17:51:56.070289+0000","last_became_active":"2024-08-17T17:24:17.441948+0000","last_became_peered":"2024-08-17T17:24:17.441948+0000","last_unstale":"2024-08-17T17:51:56.070289+0000","last_undegraded":"2024-08-17T17:51:56.070289+0000","last_fullsized":"2024-08-17T17:51:56.070289+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:17.891492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1896,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.333302+0000","last_change":"2024-08-17T17:24:17.459083+0000","last_active":"2024-08-17T17:51:55.333302+0000","last_peered":"2024-08-17T17:51:55.333302+0000","last_clean":"2024-08-17T17:51:55.333302+0000","last_became_active":"2024-08-17T17:24:17.458759+0000","last_became_peered":"2024-08-17T17:24:17.458759+0000","last_unstale":"2024-08-17T17:51:55.333302+0000","last_undegraded":"2024-08-17T17:51:55.333302+0000","last_fullsized":"2024-08-17T17:51:55.333302+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:11:59.301864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1837,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.072348+0000","last_change":"2024-08-17T17:24:17.468086+0000","last_active":"2024-08-17T17:51:56.072348+0000","last_peered":"2024-08-17T17:51:56.072348+0000","last_clean":"2024-08-17T17:51:56.072348+0000","last_became_active":"2024-08-17T17:24:17.467522+0000","last_became_peered":"2024-08-17T17:24:17.467522+0000","last_unstale":"2024-08-17T17:51:56.072348+0000","last_undegraded":"2024-08-17T17:51:56.072348+0000","last_fullsized":"2024-08-17T17:51:56.072348+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:52:55.249649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1267,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.532926+0000","last_change":"2024-08-17T17:23:07.392447+0000","last_active":"2024-08-17T17:51:54.532926+0000","last_peered":"2024-08-17T17:51:54.532926+0000","last_clean":"2024-08-17T17:51:54.532926+0000","last_became_active":"2024-08-17T17:23:07.391765+0000","last_became_peered":"2024-08-17T17:23:07.391765+0000","last_unstale":"2024-08-17T17:51:54.532926+0000","last_undegraded":"2024-08-17T17:51:54.532926+0000","last_fullsized":"2024-08-17T17:51:54.532926+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:59:18.602346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":1268,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.594762+0000","last_change":"2024-08-17T17:24:17.450313+0000","last_active":"2024-08-17T17:51:54.594762+0000","last_peered":"2024-08-17T17:51:54.594762+0000","last_clean":"2024-08-17T17:51:54.594762+0000","last_became_active":"2024-08-17T17:24:17.448803+0000","last_became_peered":"2024-08-17T17:24:17.448803+0000","last_unstale":"2024-08-17T17:51:54.594762+0000","last_undegraded":"2024-08-17T17:51:54.594762+0000","last_fullsized":"2024-08-17T17:51:54.594762+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:33.554836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1903,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.566095+0000","last_change":"2024-08-17T17:23:42.936147+0000","last_active":"2024-08-17T17:51:55.566095+0000","last_peered":"2024-08-17T17:51:55.566095+0000","last_clean":"2024-08-17T17:51:55.566095+0000","last_became_active":"2024-08-17T17:23:42.935859+0000","last_became_peered":"2024-08-17T17:23:42.935859+0000","last_unstale":"2024-08-17T17:51:55.566095+0000","last_undegraded":"2024-08-17T17:51:55.566095+0000","last_fullsized":"2024-08-17T17:51:55.566095+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:47:36.838706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1281,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332334+0000","last_change":"2024-08-17T17:10:05.969131+0000","last_active":"2024-08-17T17:51:55.332334+0000","last_peered":"2024-08-17T17:51:55.332334+0000","last_clean":"2024-08-17T17:51:55.332334+0000","last_became_active":"2024-08-17T17:10:05.968856+0000","last_became_peered":"2024-08-17T17:10:05.968856+0000","last_unstale":"2024-08-17T17:51:55.332334+0000","last_undegraded":"2024-08-17T17:51:55.332334+0000","last_fullsized":"2024-08-17T17:51:55.332334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:25:49.550065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1272,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332594+0000","last_change":"2024-08-17T17:23:07.455919+0000","last_active":"2024-08-17T17:51:55.332594+0000","last_peered":"2024-08-17T17:51:55.332594+0000","last_clean":"2024-08-17T17:51:55.332594+0000","last_became_active":"2024-08-17T17:23:07.455284+0000","last_became_peered":"2024-08-17T17:23:07.455284+0000","last_unstale":"2024-08-17T17:51:55.332594+0000","last_undegraded":"2024-08-17T17:51:55.332594+0000","last_fullsized":"2024-08-17T17:51:55.332594+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:37:46.260775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1259,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.224555+0000","last_change":"2024-08-17T17:10:06.979492+0000","last_active":"2024-08-17T17:51:54.224555+0000","last_peered":"2024-08-17T17:51:54.224555+0000","last_clean":"2024-08-17T17:51:54.224555+0000","last_became_active":"2024-08-17T17:10:06.979158+0000","last_became_peered":"2024-08-17T17:10:06.979158+0000","last_unstale":"2024-08-17T17:51:54.224555+0000","last_undegraded":"2024-08-17T17:51:54.224555+0000","last_fullsized":"2024-08-17T17:51:54.224555+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:12:42.261365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1283,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.526004+0000","last_change":"2024-08-17T17:23:42.935099+0000","last_active":"2024-08-17T17:51:55.526004+0000","last_peered":"2024-08-17T17:51:55.526004+0000","last_clean":"2024-08-17T17:51:55.526004+0000","last_became_active":"2024-08-17T17:23:42.934501+0000","last_became_peered":"2024-08-17T17:23:42.934501+0000","last_unstale":"2024-08-17T17:51:55.526004+0000","last_undegraded":"2024-08-17T17:51:55.526004+0000","last_fullsized":"2024-08-17T17:51:55.526004+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:15:19.095713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":1902,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332593+0000","last_change":"2024-08-17T17:23:07.436763+0000","last_active":"2024-08-17T17:51:55.332593+0000","last_peered":"2024-08-17T17:51:55.332593+0000","last_clean":"2024-08-17T17:51:55.332593+0000","last_became_active":"2024-08-17T17:23:07.434720+0000","last_became_peered":"2024-08-17T17:23:07.434720+0000","last_unstale":"2024-08-17T17:51:55.332593+0000","last_undegraded":"2024-08-17T17:51:55.332593+0000","last_fullsized":"2024-08-17T17:51:55.332593+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:10:20.594115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1873,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.070209+0000","last_change":"2024-08-17T17:24:17.442329+0000","last_active":"2024-08-17T17:51:56.070209+0000","last_peered":"2024-08-17T17:51:56.070209+0000","last_clean":"2024-08-17T17:51:56.070209+0000","last_became_active":"2024-08-17T17:24:17.441666+0000","last_became_peered":"2024-08-17T17:24:17.441666+0000","last_unstale":"2024-08-17T17:51:56.070209+0000","last_undegraded":"2024-08-17T17:51:56.070209+0000","last_fullsized":"2024-08-17T17:51:56.070209+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:43:06.033194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1885,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565043+0000","last_change":"2024-08-17T17:23:07.428296+0000","last_active":"2024-08-17T17:51:55.565043+0000","last_peered":"2024-08-17T17:51:55.565043+0000","last_clean":"2024-08-17T17:51:55.565043+0000","last_became_active":"2024-08-17T17:23:07.428010+0000","last_became_peered":"2024-08-17T17:23:07.428010+0000","last_unstale":"2024-08-17T17:51:55.565043+0000","last_undegraded":"2024-08-17T17:51:55.565043+0000","last_fullsized":"2024-08-17T17:51:55.565043+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:45.459212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":1874,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.072123+0000","last_change":"2024-08-17T17:24:17.481368+0000","last_active":"2024-08-17T17:51:56.072123+0000","last_peered":"2024-08-17T17:51:56.072123+0000","last_clean":"2024-08-17T17:51:56.072123+0000","last_became_active":"2024-08-17T17:24:17.477509+0000","last_became_peered":"2024-08-17T17:24:17.477509+0000","last_unstale":"2024-08-17T17:51:56.072123+0000","last_undegraded":"2024-08-17T17:51:56.072123+0000","last_fullsized":"2024-08-17T17:51:56.072123+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:05:35.558004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1888,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332264+0000","last_change":"2024-08-17T17:22:32.884274+0000","last_active":"2024-08-17T17:51:55.332264+0000","last_peered":"2024-08-17T17:51:55.332264+0000","last_clean":"2024-08-17T17:51:55.332264+0000","last_became_active":"2024-08-17T17:22:32.874667+0000","last_became_peered":"2024-08-17T17:22:32.874667+0000","last_unstale":"2024-08-17T17:51:55.332264+0000","last_undegraded":"2024-08-17T17:51:55.332264+0000","last_fullsized":"2024-08-17T17:51:55.332264+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:22:05.834587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1281,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.557870+0000","last_change":"2024-08-17T17:22:32.870569+0000","last_active":"2024-08-17T17:51:55.557870+0000","last_peered":"2024-08-17T17:51:55.557870+0000","last_clean":"2024-08-17T17:51:55.557870+0000","last_became_active":"2024-08-17T17:22:32.869335+0000","last_became_peered":"2024-08-17T17:22:32.869335+0000","last_unstale":"2024-08-17T17:51:55.557870+0000","last_undegraded":"2024-08-17T17:51:55.557870+0000","last_fullsized":"2024-08-17T17:51:55.557870+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:24.969174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1907,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.566553+0000","last_change":"2024-08-17T17:23:42.935825+0000","last_active":"2024-08-17T17:51:55.566553+0000","last_peered":"2024-08-17T17:51:55.566553+0000","last_clean":"2024-08-17T17:51:55.566553+0000","last_became_active":"2024-08-17T17:23:42.935150+0000","last_became_peered":"2024-08-17T17:23:42.935150+0000","last_unstale":"2024-08-17T17:51:55.566553+0000","last_undegraded":"2024-08-17T17:51:55.566553+0000","last_fullsized":"2024-08-17T17:51:55.566553+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:57:19.577644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1895,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565154+0000","last_change":"2024-08-17T17:23:07.431586+0000","last_active":"2024-08-17T17:51:55.565154+0000","last_peered":"2024-08-17T17:51:55.565154+0000","last_clean":"2024-08-17T17:51:55.565154+0000","last_became_active":"2024-08-17T17:23:07.431301+0000","last_became_peered":"2024-08-17T17:23:07.431301+0000","last_unstale":"2024-08-17T17:51:55.565154+0000","last_undegraded":"2024-08-17T17:51:55.565154+0000","last_fullsized":"2024-08-17T17:51:55.565154+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:28:18.706489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1923,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565354+0000","last_change":"2024-08-17T17:23:07.420550+0000","last_active":"2024-08-17T17:51:55.565354+0000","last_peered":"2024-08-17T17:51:55.565354+0000","last_clean":"2024-08-17T17:51:55.565354+0000","last_became_active":"2024-08-17T17:23:07.420223+0000","last_became_peered":"2024-08-17T17:23:07.420223+0000","last_unstale":"2024-08-17T17:51:55.565354+0000","last_undegraded":"2024-08-17T17:51:55.565354+0000","last_fullsized":"2024-08-17T17:51:55.565354+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:33:46.996371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1285,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.537366+0000","last_change":"2024-08-17T17:24:17.426267+0000","last_active":"2024-08-17T17:51:55.537366+0000","last_peered":"2024-08-17T17:51:55.537366+0000","last_clean":"2024-08-17T17:51:55.537366+0000","last_became_active":"2024-08-17T17:24:17.425925+0000","last_became_peered":"2024-08-17T17:24:17.425925+0000","last_unstale":"2024-08-17T17:51:55.537366+0000","last_undegraded":"2024-08-17T17:51:55.537366+0000","last_fullsized":"2024-08-17T17:51:55.537366+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:24:25.017856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1281,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.537016+0000","last_change":"2024-08-17T17:24:17.474138+0000","last_active":"2024-08-17T17:51:55.537016+0000","last_peered":"2024-08-17T17:51:55.537016+0000","last_clean":"2024-08-17T17:51:55.537016+0000","last_became_active":"2024-08-17T17:24:17.473708+0000","last_became_peered":"2024-08-17T17:24:17.473708+0000","last_unstale":"2024-08-17T17:51:55.537016+0000","last_undegraded":"2024-08-17T17:51:55.537016+0000","last_fullsized":"2024-08-17T17:51:55.537016+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:10:57.768067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1275,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.225254+0000","last_change":"2024-08-17T17:22:32.871533+0000","last_active":"2024-08-17T17:51:54.225254+0000","last_peered":"2024-08-17T17:51:54.225254+0000","last_clean":"2024-08-17T17:51:54.225254+0000","last_became_active":"2024-08-17T17:22:32.868226+0000","last_became_peered":"2024-08-17T17:22:32.868226+0000","last_unstale":"2024-08-17T17:51:54.225254+0000","last_undegraded":"2024-08-17T17:51:54.225254+0000","last_fullsized":"2024-08-17T17:51:54.225254+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:00:02.629455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":1285,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.537019+0000","last_change":"2024-08-17T17:23:07.391021+0000","last_active":"2024-08-17T17:51:55.537019+0000","last_peered":"2024-08-17T17:51:55.537019+0000","last_clean":"2024-08-17T17:51:55.537019+0000","last_became_active":"2024-08-17T17:23:07.390551+0000","last_became_peered":"2024-08-17T17:23:07.390551+0000","last_unstale":"2024-08-17T17:51:55.537019+0000","last_undegraded":"2024-08-17T17:51:55.537019+0000","last_fullsized":"2024-08-17T17:51:55.537019+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:06.239013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001023532,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1906,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565011+0000","last_change":"2024-08-17T17:23:07.406362+0000","last_active":"2024-08-17T17:51:55.565011+0000","last_peered":"2024-08-17T17:51:55.565011+0000","last_clean":"2024-08-17T17:51:55.565011+0000","last_became_active":"2024-08-17T17:23:07.406071+0000","last_became_peered":"2024-08-17T17:23:07.406071+0000","last_unstale":"2024-08-17T17:51:55.565011+0000","last_undegraded":"2024-08-17T17:51:55.565011+0000","last_fullsized":"2024-08-17T17:51:55.565011+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:45:47.383432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1886,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565198+0000","last_change":"2024-08-17T17:23:42.910246+0000","last_active":"2024-08-17T17:51:55.565198+0000","last_peered":"2024-08-17T17:51:55.565198+0000","last_clean":"2024-08-17T17:51:55.565198+0000","last_became_active":"2024-08-17T17:23:42.909929+0000","last_became_peered":"2024-08-17T17:23:42.909929+0000","last_unstale":"2024-08-17T17:51:55.565198+0000","last_undegraded":"2024-08-17T17:51:55.565198+0000","last_fullsized":"2024-08-17T17:51:55.565198+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:14:45.210558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1916,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.070318+0000","last_change":"2024-08-17T17:24:17.438891+0000","last_active":"2024-08-17T17:51:56.070318+0000","last_peered":"2024-08-17T17:51:56.070318+0000","last_clean":"2024-08-17T17:51:56.070318+0000","last_became_active":"2024-08-17T17:24:17.437714+0000","last_became_peered":"2024-08-17T17:24:17.437714+0000","last_unstale":"2024-08-17T17:51:56.070318+0000","last_undegraded":"2024-08-17T17:51:56.070318+0000","last_fullsized":"2024-08-17T17:51:56.070318+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:58.825017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1909,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332494+0000","last_change":"2024-08-17T17:23:42.908634+0000","last_active":"2024-08-17T17:51:55.332494+0000","last_peered":"2024-08-17T17:51:55.332494+0000","last_clean":"2024-08-17T17:51:55.332494+0000","last_became_active":"2024-08-17T17:23:42.907675+0000","last_became_peered":"2024-08-17T17:23:42.907675+0000","last_unstale":"2024-08-17T17:51:55.332494+0000","last_undegraded":"2024-08-17T17:51:55.332494+0000","last_fullsized":"2024-08-17T17:51:55.332494+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:03:37.612222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1899,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.331258+0000","last_change":"2024-08-17T17:24:17.457559+0000","last_active":"2024-08-17T17:51:55.331258+0000","last_peered":"2024-08-17T17:51:55.331258+0000","last_clean":"2024-08-17T17:51:55.331258+0000","last_became_active":"2024-08-17T17:24:17.457262+0000","last_became_peered":"2024-08-17T17:24:17.457262+0000","last_unstale":"2024-08-17T17:51:55.331258+0000","last_undegraded":"2024-08-17T17:51:55.331258+0000","last_fullsized":"2024-08-17T17:51:55.331258+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:53:38.137277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1296,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.594253+0000","last_change":"2024-08-17T17:23:07.450509+0000","last_active":"2024-08-17T17:51:54.594253+0000","last_peered":"2024-08-17T17:51:54.594253+0000","last_clean":"2024-08-17T17:51:54.594253+0000","last_became_active":"2024-08-17T17:23:07.450182+0000","last_became_peered":"2024-08-17T17:23:07.450182+0000","last_unstale":"2024-08-17T17:51:54.594253+0000","last_undegraded":"2024-08-17T17:51:54.594253+0000","last_fullsized":"2024-08-17T17:51:54.594253+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:01:18.656656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1255,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.526524+0000","last_change":"2024-08-17T17:10:06.995023+0000","last_active":"2024-08-17T17:51:55.526524+0000","last_peered":"2024-08-17T17:51:55.526524+0000","last_clean":"2024-08-17T17:51:55.526524+0000","last_became_active":"2024-08-17T17:10:06.994757+0000","last_became_peered":"2024-08-17T17:10:06.994757+0000","last_unstale":"2024-08-17T17:51:55.526524+0000","last_undegraded":"2024-08-17T17:51:55.526524+0000","last_fullsized":"2024-08-17T17:51:55.526524+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:31:52.163518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1839,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.071057+0000","last_change":"2024-08-17T17:24:17.468443+0000","last_active":"2024-08-17T17:51:56.071057+0000","last_peered":"2024-08-17T17:51:56.071057+0000","last_clean":"2024-08-17T17:51:56.071057+0000","last_became_active":"2024-08-17T17:24:17.467394+0000","last_became_peered":"2024-08-17T17:24:17.467394+0000","last_unstale":"2024-08-17T17:51:56.071057+0000","last_undegraded":"2024-08-17T17:51:56.071057+0000","last_fullsized":"2024-08-17T17:51:56.071057+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:13.024253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":1284,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.537434+0000","last_change":"2024-08-17T17:23:42.920552+0000","last_active":"2024-08-17T17:51:55.537434+0000","last_peered":"2024-08-17T17:51:55.537434+0000","last_clean":"2024-08-17T17:51:55.537434+0000","last_became_active":"2024-08-17T17:23:42.915824+0000","last_became_peered":"2024-08-17T17:23:42.915824+0000","last_unstale":"2024-08-17T17:51:55.537434+0000","last_undegraded":"2024-08-17T17:51:55.537434+0000","last_fullsized":"2024-08-17T17:51:55.537434+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:25:18.769504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041737399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":1597,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.556550+0000","last_change":"2024-08-17T17:22:32.867754+0000","last_active":"2024-08-17T17:51:55.556550+0000","last_peered":"2024-08-17T17:51:55.556550+0000","last_clean":"2024-08-17T17:51:55.556550+0000","last_became_active":"2024-08-17T17:22:32.867345+0000","last_became_peered":"2024-08-17T17:22:32.867345+0000","last_unstale":"2024-08-17T17:51:55.556550+0000","last_undegraded":"2024-08-17T17:51:55.556550+0000","last_fullsized":"2024-08-17T17:51:55.556550+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T17:06:21.174471+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:00:42.776037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1885,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.565625+0000","last_change":"2024-08-17T17:24:17.455710+0000","last_active":"2024-08-17T17:51:55.565625+0000","last_peered":"2024-08-17T17:51:55.565625+0000","last_clean":"2024-08-17T17:51:55.565625+0000","last_became_active":"2024-08-17T17:24:17.452989+0000","last_became_peered":"2024-08-17T17:24:17.452989+0000","last_unstale":"2024-08-17T17:51:55.565625+0000","last_undegraded":"2024-08-17T17:51:55.565625+0000","last_fullsized":"2024-08-17T17:51:55.565625+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:58:07.894968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1880,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.070034+0000","last_change":"2024-08-17T17:24:17.453639+0000","last_active":"2024-08-17T17:51:56.070034+0000","last_peered":"2024-08-17T17:51:56.070034+0000","last_clean":"2024-08-17T17:51:56.070034+0000","last_became_active":"2024-08-17T17:24:17.443638+0000","last_became_peered":"2024-08-17T17:24:17.443638+0000","last_unstale":"2024-08-17T17:51:56.070034+0000","last_undegraded":"2024-08-17T17:51:56.070034+0000","last_fullsized":"2024-08-17T17:51:56.070034+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:42.107479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1272,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.225392+0000","last_change":"2024-08-17T17:23:07.451866+0000","last_active":"2024-08-17T17:51:54.225392+0000","last_peered":"2024-08-17T17:51:54.225392+0000","last_clean":"2024-08-17T17:51:54.225392+0000","last_became_active":"2024-08-17T17:23:07.451553+0000","last_became_peered":"2024-08-17T17:23:07.451553+0000","last_unstale":"2024-08-17T17:51:54.225392+0000","last_undegraded":"2024-08-17T17:51:54.225392+0000","last_fullsized":"2024-08-17T17:51:54.225392+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:37:23.930651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1274,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.224623+0000","last_change":"2024-08-17T17:23:42.915851+0000","last_active":"2024-08-17T17:51:54.224623+0000","last_peered":"2024-08-17T17:51:54.224623+0000","last_clean":"2024-08-17T17:51:54.224623+0000","last_became_active":"2024-08-17T17:23:42.915552+0000","last_became_peered":"2024-08-17T17:23:42.915552+0000","last_unstale":"2024-08-17T17:51:54.224623+0000","last_undegraded":"2024-08-17T17:51:54.224623+0000","last_fullsized":"2024-08-17T17:51:54.224623+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:45.197876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1287,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.225154+0000","last_change":"2024-08-17T17:23:42.930803+0000","last_active":"2024-08-17T17:51:54.225154+0000","last_peered":"2024-08-17T17:51:54.225154+0000","last_clean":"2024-08-17T17:51:54.225154+0000","last_became_active":"2024-08-17T17:23:42.930458+0000","last_became_peered":"2024-08-17T17:23:42.930458+0000","last_unstale":"2024-08-17T17:51:54.225154+0000","last_undegraded":"2024-08-17T17:51:54.225154+0000","last_fullsized":"2024-08-17T17:51:54.225154+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:40.816706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1270,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.477299+0000","last_change":"2024-08-17T17:23:07.401624+0000","last_active":"2024-08-17T17:51:54.477299+0000","last_peered":"2024-08-17T17:51:54.477299+0000","last_clean":"2024-08-17T17:51:54.477299+0000","last_became_active":"2024-08-17T17:23:07.400222+0000","last_became_peered":"2024-08-17T17:23:07.400222+0000","last_unstale":"2024-08-17T17:51:54.477299+0000","last_undegraded":"2024-08-17T17:51:54.477299+0000","last_fullsized":"2024-08-17T17:51:54.477299+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:07.069989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1272,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.224982+0000","last_change":"2024-08-17T17:24:17.470809+0000","last_active":"2024-08-17T17:51:54.224982+0000","last_peered":"2024-08-17T17:51:54.224982+0000","last_clean":"2024-08-17T17:51:54.224982+0000","last_became_active":"2024-08-17T17:24:17.469753+0000","last_became_peered":"2024-08-17T17:24:17.469753+0000","last_unstale":"2024-08-17T17:51:54.224982+0000","last_undegraded":"2024-08-17T17:51:54.224982+0000","last_fullsized":"2024-08-17T17:51:54.224982+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:00.049425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":1873,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.069486+0000","last_change":"2024-08-17T17:24:17.461258+0000","last_active":"2024-08-17T17:51:56.069486+0000","last_peered":"2024-08-17T17:51:56.069486+0000","last_clean":"2024-08-17T17:51:56.069486+0000","last_became_active":"2024-08-17T17:24:17.460895+0000","last_became_peered":"2024-08-17T17:24:17.460895+0000","last_unstale":"2024-08-17T17:51:56.069486+0000","last_undegraded":"2024-08-17T17:51:56.069486+0000","last_fullsized":"2024-08-17T17:51:56.069486+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:43:24.208259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1912,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.331877+0000","last_change":"2024-08-17T17:23:42.901638+0000","last_active":"2024-08-17T17:51:55.331877+0000","last_peered":"2024-08-17T17:51:55.331877+0000","last_clean":"2024-08-17T17:51:55.331877+0000","last_became_active":"2024-08-17T17:23:42.901335+0000","last_became_peered":"2024-08-17T17:23:42.901335+0000","last_unstale":"2024-08-17T17:51:55.331877+0000","last_undegraded":"2024-08-17T17:51:55.331877+0000","last_fullsized":"2024-08-17T17:51:55.331877+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:25:35.332839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1268,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.532827+0000","last_change":"2024-08-17T17:24:17.473913+0000","last_active":"2024-08-17T17:51:54.532827+0000","last_peered":"2024-08-17T17:51:54.532827+0000","last_clean":"2024-08-17T17:51:54.532827+0000","last_became_active":"2024-08-17T17:24:17.473195+0000","last_became_peered":"2024-08-17T17:24:17.473195+0000","last_unstale":"2024-08-17T17:51:54.532827+0000","last_undegraded":"2024-08-17T17:51:54.532827+0000","last_fullsized":"2024-08-17T17:51:54.532827+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:12.998033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1269,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.477272+0000","last_change":"2024-08-17T17:23:07.402010+0000","last_active":"2024-08-17T17:51:54.477272+0000","last_peered":"2024-08-17T17:51:54.477272+0000","last_clean":"2024-08-17T17:51:54.477272+0000","last_became_active":"2024-08-17T17:23:07.400849+0000","last_became_peered":"2024-08-17T17:23:07.400849+0000","last_unstale":"2024-08-17T17:51:54.477272+0000","last_undegraded":"2024-08-17T17:51:54.477272+0000","last_fullsized":"2024-08-17T17:51:54.477272+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:16:02.382257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":1252,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.594877+0000","last_change":"2024-08-17T17:10:05.973491+0000","last_active":"2024-08-17T17:51:54.594877+0000","last_peered":"2024-08-17T17:51:54.594877+0000","last_clean":"2024-08-17T17:51:54.594877+0000","last_became_active":"2024-08-17T17:10:05.973228+0000","last_became_peered":"2024-08-17T17:10:05.973228+0000","last_unstale":"2024-08-17T17:51:54.594877+0000","last_undegraded":"2024-08-17T17:51:54.594877+0000","last_fullsized":"2024-08-17T17:51:54.594877+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:54.310135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1288,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.224430+0000","last_change":"2024-08-17T17:24:17.460203+0000","last_active":"2024-08-17T17:51:54.224430+0000","last_peered":"2024-08-17T17:51:54.224430+0000","last_clean":"2024-08-17T17:51:54.224430+0000","last_became_active":"2024-08-17T17:24:17.459682+0000","last_became_peered":"2024-08-17T17:24:17.459682+0000","last_unstale":"2024-08-17T17:51:54.224430+0000","last_undegraded":"2024-08-17T17:51:54.224430+0000","last_fullsized":"2024-08-17T17:51:54.224430+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:58.703700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":1268,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.594587+0000","last_change":"2024-08-17T17:23:07.454042+0000","last_active":"2024-08-17T17:51:54.594587+0000","last_peered":"2024-08-17T17:51:54.594587+0000","last_clean":"2024-08-17T17:51:54.594587+0000","last_became_active":"2024-08-17T17:23:07.453712+0000","last_became_peered":"2024-08-17T17:23:07.453712+0000","last_unstale":"2024-08-17T17:51:54.594587+0000","last_undegraded":"2024-08-17T17:51:54.594587+0000","last_fullsized":"2024-08-17T17:51:54.594587+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:03:07.643456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1253,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.532272+0000","last_change":"2024-08-17T17:10:06.994799+0000","last_active":"2024-08-17T17:51:54.532272+0000","last_peered":"2024-08-17T17:51:54.532272+0000","last_clean":"2024-08-17T17:51:54.532272+0000","last_became_active":"2024-08-17T17:10:06.994150+0000","last_became_peered":"2024-08-17T17:10:06.994150+0000","last_unstale":"2024-08-17T17:51:54.532272+0000","last_undegraded":"2024-08-17T17:51:54.532272+0000","last_fullsized":"2024-08-17T17:51:54.532272+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:50:47.293091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":1266,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.594586+0000","last_change":"2024-08-17T17:22:32.873334+0000","last_active":"2024-08-17T17:51:54.594586+0000","last_peered":"2024-08-17T17:51:54.594586+0000","last_clean":"2024-08-17T17:51:54.594586+0000","last_became_active":"2024-08-17T17:22:32.870147+0000","last_became_peered":"2024-08-17T17:22:32.870147+0000","last_unstale":"2024-08-17T17:51:54.594586+0000","last_undegraded":"2024-08-17T17:51:54.594586+0000","last_fullsized":"2024-08-17T17:51:54.594586+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:18:03.750592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1277,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.524676+0000","last_change":"2024-08-17T17:22:32.847989+0000","last_active":"2024-08-17T17:51:55.524676+0000","last_peered":"2024-08-17T17:51:55.524676+0000","last_clean":"2024-08-17T17:51:55.524676+0000","last_became_active":"2024-08-17T17:22:32.846593+0000","last_became_peered":"2024-08-17T17:22:32.846593+0000","last_unstale":"2024-08-17T17:51:55.524676+0000","last_undegraded":"2024-08-17T17:51:55.524676+0000","last_fullsized":"2024-08-17T17:51:55.524676+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:33:50.370124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1877,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.071934+0000","last_change":"2024-08-17T17:24:17.451528+0000","last_active":"2024-08-17T17:51:56.071934+0000","last_peered":"2024-08-17T17:51:56.071934+0000","last_clean":"2024-08-17T17:51:56.071934+0000","last_became_active":"2024-08-17T17:24:17.451246+0000","last_became_peered":"2024-08-17T17:24:17.451246+0000","last_unstale":"2024-08-17T17:51:56.071934+0000","last_undegraded":"2024-08-17T17:51:56.071934+0000","last_fullsized":"2024-08-17T17:51:56.071934+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:38:18.176559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1251,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.557291+0000","last_change":"2024-08-17T17:10:05.975692+0000","last_active":"2024-08-17T17:51:55.557291+0000","last_peered":"2024-08-17T17:51:55.557291+0000","last_clean":"2024-08-17T17:51:55.557291+0000","last_became_active":"2024-08-17T17:10:05.975057+0000","last_became_peered":"2024-08-17T17:10:05.975057+0000","last_unstale":"2024-08-17T17:51:55.557291+0000","last_undegraded":"2024-08-17T17:51:55.557291+0000","last_fullsized":"2024-08-17T17:51:55.557291+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:44:53.106104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1896,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.331881+0000","last_change":"2024-08-17T17:24:17.460371+0000","last_active":"2024-08-17T17:51:55.331881+0000","last_peered":"2024-08-17T17:51:55.331881+0000","last_clean":"2024-08-17T17:51:55.331881+0000","last_became_active":"2024-08-17T17:24:17.460074+0000","last_became_peered":"2024-08-17T17:24:17.460074+0000","last_unstale":"2024-08-17T17:51:55.331881+0000","last_undegraded":"2024-08-17T17:51:55.331881+0000","last_fullsized":"2024-08-17T17:51:55.331881+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:08:48.820168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1896,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.332180+0000","last_change":"2024-08-17T17:24:17.457942+0000","last_active":"2024-08-17T17:51:55.332180+0000","last_peered":"2024-08-17T17:51:55.332180+0000","last_clean":"2024-08-17T17:51:55.332180+0000","last_became_active":"2024-08-17T17:24:17.456475+0000","last_became_peered":"2024-08-17T17:24:17.456475+0000","last_unstale":"2024-08-17T17:51:55.332180+0000","last_undegraded":"2024-08-17T17:51:55.332180+0000","last_fullsized":"2024-08-17T17:51:55.332180+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:12.325657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1266,"reported_epoch":1300,"state":"active+clean","last_fresh":"2024-08-17T17:51:54.532706+0000","last_change":"2024-08-17T17:24:17.451287+0000","last_active":"2024-08-17T17:51:54.532706+0000","last_peered":"2024-08-17T17:51:54.532706+0000","last_clean":"2024-08-17T17:51:54.532706+0000","last_became_active":"2024-08-17T17:24:17.450951+0000","last_became_peered":"2024-08-17T17:24:17.450951+0000","last_unstale":"2024-08-17T17:51:54.532706+0000","last_undegraded":"2024-08-17T17:51:54.532706+0000","last_fullsized":"2024-08-17T17:51:54.532706+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:13:37.590108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":1268,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:55.556246+0000","last_change":"2024-08-17T17:23:42.907970+0000","last_active":"2024-08-17T17:51:55.556246+0000","last_peered":"2024-08-17T17:51:55.556246+0000","last_clean":"2024-08-17T17:51:55.556246+0000","last_became_active":"2024-08-17T17:23:42.907595+0000","last_became_peered":"2024-08-17T17:23:42.907595+0000","last_unstale":"2024-08-17T17:51:55.556246+0000","last_undegraded":"2024-08-17T17:51:55.556246+0000","last_fullsized":"2024-08-17T17:51:55.556246+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:11:46.130004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1878,"reported_epoch":1301,"state":"active+clean","last_fresh":"2024-08-17T17:51:56.072237+0000","last_change":"2024-08-17T17:24:17.486591+0000","last_active":"2024-08-17T17:51:56.072237+0000","last_peered":"2024-08-17T17:51:56.072237+0000","last_clean":"2024-08-17T17:51:56.072237+0000","last_became_active":"2024-08-17T17:24:17.486320+0000","last_became_peered":"2024-08-17T17:24:17.486320+0000","last_unstale":"2024-08-17T17:51:56.072237+0000","last_undegraded":"2024-08-17T17:51:56.072237+0000","last_fullsized":"2024-08-17T17:51:56.072237+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:03.249803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234522,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":120032,"kb_used_data":16792,"kb_used_omap":1,"kb_used_meta":103230,"kb_avail":93625120,"statfs":{"total":95995035648,"available":95872122880,"internally_reserved":0,"allocated":17195008,"data_stored":10837596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":105707979},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496160,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":146268,"kb_used_data":16784,"kb_used_omap":1,"kb_used_meta":129470,"kb_avail":93598884,"statfs":{"total":95995035648,"available":95845257216,"internally_reserved":0,"allocated":17186816,"data_stored":10837469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":132577735},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565401,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":92740,"kb_used_data":16188,"kb_used_omap":72,"kb_used_meta":76471,"kb_avail":93652412,"statfs":{"total":95995035648,"available":95900069888,"internally_reserved":0,"allocated":16576512,"data_stored":10242358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":78306798},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":674,"seq":2894807957856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70540,"kb_used_data":16192,"kb_used_omap":74,"kb_used_meta":54261,"kb_avail":93674612,"statfs":{"total":95995035648,"available":95922802688,"internally_reserved":0,"allocated":16580608,"data_stored":10242884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76116,"internal_metadata":55563948},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743186791,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70472,"kb_used_data":16188,"kb_used_omap":66,"kb_used_meta":54205,"kb_avail":93674680,"statfs":{"total":95995035648,"available":95922872320,"internally_reserved":0,"allocated":16576512,"data_stored":10242287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":55506603},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448430,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88916,"kb_used_data":16772,"kb_used_omap":75,"kb_used_meta":72052,"kb_avail":93656236,"statfs":{"total":95995035648,"available":95903985664,"internally_reserved":0,"allocated":17174528,"data_stored":10833240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":73781439},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757796,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":127700,"kb_used_data":16196,"kb_used_omap":1,"kb_used_meta":111486,"kb_avail":93617452,"statfs":{"total":95995035648,"available":95864270848,"internally_reserved":0,"allocated":16584704,"data_stored":10242927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114162118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019433,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":141600,"kb_used_data":16208,"kb_used_omap":1,"kb_used_meta":125374,"kb_avail":93603552,"statfs":{"total":95995035648,"available":95850037248,"internally_reserved":0,"allocated":16596992,"data_stored":10247158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128383430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:52:54.773 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-08-17T17:52:54.774 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd dump 2024-08-17T17:52:54.978 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:55 smithi136 ceph-mon[109861]: from='client.35040 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:52:55.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:55 smithi136 ceph-mon[115078]: from='client.35040 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:52:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:55 smithi165 ceph-mon[105517]: from='client.35040 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:52:56.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:56 smithi136 ceph-mon[109861]: pgmap v2063: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:56.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:56 smithi136 ceph-mon[115078]: pgmap v2063: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:56 smithi165 ceph-mon[105517]: pgmap v2063: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:56.807 INFO:teuthology.orchestra.run.smithi136.stdout:epoch 1301 2024-08-17T17:52:56.807 INFO:teuthology.orchestra.run.smithi136.stdout:fsid 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T17:52:56.807 INFO:teuthology.orchestra.run.smithi136.stdout:created 2024-08-17T17:03:17.682767+0000 2024-08-17T17:52:56.807 INFO:teuthology.orchestra.run.smithi136.stdout:modified 2024-08-17T17:51:55.319388+0000 2024-08-17T17:52:56.807 INFO:teuthology.orchestra.run.smithi136.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-08-17T17:52:56.807 INFO:teuthology.orchestra.run.smithi136.stdout:crush_version 22 2024-08-17T17:52:56.807 INFO:teuthology.orchestra.run.smithi136.stdout:full_ratio 0.95 2024-08-17T17:52:56.807 INFO:teuthology.orchestra.run.smithi136.stdout:backfillfull_ratio 0.9 2024-08-17T17:52:56.808 INFO:teuthology.orchestra.run.smithi136.stdout:nearfull_ratio 0.85 2024-08-17T17:52:56.808 INFO:teuthology.orchestra.run.smithi136.stdout:require_min_compat_client luminous 2024-08-17T17:52:56.808 INFO:teuthology.orchestra.run.smithi136.stdout:min_compat_client luminous 2024-08-17T17:52:56.808 INFO:teuthology.orchestra.run.smithi136.stdout:require_osd_release reef 2024-08-17T17:52:56.808 INFO:teuthology.orchestra.run.smithi136.stdout:stretch_mode_enabled false 2024-08-17T17:52:56.808 INFO:teuthology.orchestra.run.smithi136.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-08-17T17:52:56.808 INFO:teuthology.orchestra.run.smithi136.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 64 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.00 2024-08-17T17:52:56.808 INFO:teuthology.orchestra.run.smithi136.stdout:pool 3 'cephfs.foo.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 69 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.50 2024-08-17T17:52:56.808 INFO:teuthology.orchestra.run.smithi136.stdout:pool 4 'cephfs.foo.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 70 flags hashpspool,bulk stripe_width 0 application cephfs read_balance_score 2.25 2024-08-17T17:52:56.808 INFO:teuthology.orchestra.run.smithi136.stdout:max_osd 8 2024-08-17T17:52:56.808 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0 up in weight 1 up_from 659 up_thru 1153 down_at 653 last_clean_interval [8,653) [v2:172.21.15.136:6802/1285058127,v1:172.21.15.136:6803/1285058127] [v2:172.21.15.136:6804/1285058127,v1:172.21.15.136:6805/1285058127] exists,up 51eb458d-9f33-4b87-b7f5-8acb076c16ae 2024-08-17T17:52:56.808 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1 up in weight 1 up_from 667 up_thru 1153 down_at 662 last_clean_interval [13,662) [v2:172.21.15.136:6810/717195127,v1:172.21.15.136:6811/717195127] [v2:172.21.15.136:6812/717195127,v1:172.21.15.136:6813/717195127] exists,up aa73fa16-8736-478c-b07a-e674c18698ef 2024-08-17T17:52:56.808 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2 up in weight 1 up_from 674 up_thru 1153 down_at 670 last_clean_interval [18,670) [v2:172.21.15.136:6818/762368443,v1:172.21.15.136:6819/762368443] [v2:172.21.15.136:6820/762368443,v1:172.21.15.136:6821/762368443] exists,up 4d2612be-7068-48d9-bb0c-992ebccaa091 2024-08-17T17:52:56.808 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3 up in weight 1 up_from 686 up_thru 1153 down_at 682 last_clean_interval [25,682) [v2:172.21.15.136:6826/946860336,v1:172.21.15.136:6827/946860336] [v2:172.21.15.136:6828/946860336,v1:172.21.15.136:6829/946860336] exists,up 30b8a4f2-be6d-4d5f-8397-06c2e17cfc2b 2024-08-17T17:52:56.809 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4 up in weight 1 up_from 30 up_thru 1153 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6800/842623090,v1:172.21.15.165:6801/842623090] [v2:172.21.15.165:6802/842623090,v1:172.21.15.165:6803/842623090] exists,up ae4074cf-0069-403f-acc7-74bedab266bd 2024-08-17T17:52:56.809 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5 up in weight 1 up_from 38 up_thru 1153 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6808/3602301990,v1:172.21.15.165:6809/3602301990] [v2:172.21.15.165:6810/3602301990,v1:172.21.15.165:6811/3602301990] exists,up 586c23a7-4831-4b4d-9a06-c1bfcc1c5286 2024-08-17T17:52:56.809 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6 up in weight 1 up_from 46 up_thru 1153 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6816/1795674907,v1:172.21.15.165:6817/1795674907] [v2:172.21.15.165:6818/1795674907,v1:172.21.15.165:6819/1795674907] exists,up e838c26b-5e4a-41c6-b35b-b438ba64b237 2024-08-17T17:52:56.809 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7 up in weight 1 up_from 54 up_thru 1153 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6824/4195921091,v1:172.21.15.165:6825/4195921091] [v2:172.21.15.165:6826/4195921091,v1:172.21.15.165:6827/4195921091] exists,up de2fab2e-177a-4eb0-9262-7bbef0d9837f 2024-08-17T17:52:56.809 INFO:teuthology.orchestra.run.smithi136.stdout:pg_upmap_items 4.4 [1,5] 2024-08-17T17:52:56.809 INFO:teuthology.orchestra.run.smithi136.stdout:pg_upmap_items 4.b [1,2] 2024-08-17T17:52:56.809 INFO:teuthology.orchestra.run.smithi136.stdout:pg_upmap_items 4.1d [1,2] 2024-08-17T17:52:56.809 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:0/3515054633 expires 2024-08-17T18:28:42.746890+0000 2024-08-17T17:52:56.809 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:0/2190679281 expires 2024-08-17T18:19:30.927605+0000 2024-08-17T17:52:56.809 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:0/1504550954 expires 2024-08-17T18:19:29.931500+0000 2024-08-17T17:52:56.809 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:6800/4104535444 expires 2024-08-18T17:16:05.797025+0000 2024-08-17T17:52:56.809 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:0/3841116499 expires 2024-08-18T17:16:05.797025+0000 2024-08-17T17:52:56.809 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:0/3340551496 expires 2024-08-18T17:16:05.797025+0000 2024-08-17T17:52:56.809 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:0/3946052118 expires 2024-08-18T17:16:05.797025+0000 2024-08-17T17:52:56.810 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.165:0/3676140772 expires 2024-08-18T17:13:24.578001+0000 2024-08-17T17:52:56.810 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.165:6835/7511620 expires 2024-08-18T17:13:24.578001+0000 2024-08-17T17:52:56.810 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:0/1414082347 expires 2024-08-18T17:04:10.472088+0000 2024-08-17T17:52:56.810 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.165:6834/7511620 expires 2024-08-18T17:13:24.578001+0000 2024-08-17T17:52:56.810 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:6800/803461847 expires 2024-08-18T17:12:45.319619+0000 2024-08-17T17:52:56.810 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:0/1577563918 expires 2024-08-18T17:04:10.472088+0000 2024-08-17T17:52:56.810 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.165:0/300548107 expires 2024-08-18T17:13:24.578001+0000 2024-08-17T17:52:56.810 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.165:0/1675517659 expires 2024-08-18T17:13:24.578001+0000 2024-08-17T17:52:56.810 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:6801/3513268997 expires 2024-08-18T17:08:47.841029+0000 2024-08-17T17:52:56.810 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:0/972617428 expires 2024-08-18T17:12:45.319619+0000 2024-08-17T17:52:56.810 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:6800/3058734149 expires 2024-08-18T17:04:10.472088+0000 2024-08-17T17:52:56.810 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:6800/3513268997 expires 2024-08-18T17:08:47.841029+0000 2024-08-17T17:52:56.810 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:0/2887741187 expires 2024-08-18T17:12:45.319619+0000 2024-08-17T17:52:56.810 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:0/2931810343 expires 2024-08-18T17:08:47.841029+0000 2024-08-17T17:52:56.811 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:0/1381030503 expires 2024-08-18T17:03:48.417471+0000 2024-08-17T17:52:56.811 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:0/2540800865 expires 2024-08-18T17:08:47.841029+0000 2024-08-17T17:52:56.811 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:0/3933059683 expires 2024-08-18T17:04:10.472088+0000 2024-08-17T17:52:56.811 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.165:0/3723748982 expires 2024-08-18T17:13:24.578001+0000 2024-08-17T17:52:56.811 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:6801/3058734149 expires 2024-08-18T17:04:10.472088+0000 2024-08-17T17:52:56.811 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:0/3380731497 expires 2024-08-18T17:03:48.417471+0000 2024-08-17T17:52:56.811 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:0/1905186682 expires 2024-08-18T17:08:47.841029+0000 2024-08-17T17:52:56.811 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:0/1450531984 expires 2024-08-18T17:08:47.841029+0000 2024-08-17T17:52:56.811 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.165:0/1694689688 expires 2024-08-18T17:13:24.578001+0000 2024-08-17T17:52:56.811 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:0/2336022730 expires 2024-08-18T17:03:48.417471+0000 2024-08-17T17:52:56.811 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:0/2251781402 expires 2024-08-18T17:16:05.797025+0000 2024-08-17T17:52:56.811 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:0/3166566423 expires 2024-08-18T17:12:45.319619+0000 2024-08-17T17:52:56.811 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:6801/803461847 expires 2024-08-18T17:12:45.319619+0000 2024-08-17T17:52:56.811 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:6801/4104535444 expires 2024-08-18T17:16:05.797025+0000 2024-08-17T17:52:56.812 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:6801/878395364 expires 2024-08-18T17:03:48.417471+0000 2024-08-17T17:52:56.812 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:0/707907005 expires 2024-08-18T17:12:45.319619+0000 2024-08-17T17:52:56.812 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.165:0/2646930649 expires 2024-08-18T17:13:24.578001+0000 2024-08-17T17:52:56.812 INFO:teuthology.orchestra.run.smithi136.stdout:blocklist 172.21.15.136:6800/878395364 expires 2024-08-18T17:03:48.417471+0000 2024-08-17T17:52:57.684 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 659 up_thru 1153 down_at 653 last_clean_interval [8,653) [v2:172.21.15.136:6802/1285058127,v1:172.21.15.136:6803/1285058127] [v2:172.21.15.136:6804/1285058127,v1:172.21.15.136:6805/1285058127] exists,up 51eb458d-9f33-4b87-b7f5-8acb076c16ae', 'osd.1 up in weight 1 up_from 667 up_thru 1153 down_at 662 last_clean_interval [13,662) [v2:172.21.15.136:6810/717195127,v1:172.21.15.136:6811/717195127] [v2:172.21.15.136:6812/717195127,v1:172.21.15.136:6813/717195127] exists,up aa73fa16-8736-478c-b07a-e674c18698ef', 'osd.2 up in weight 1 up_from 674 up_thru 1153 down_at 670 last_clean_interval [18,670) [v2:172.21.15.136:6818/762368443,v1:172.21.15.136:6819/762368443] [v2:172.21.15.136:6820/762368443,v1:172.21.15.136:6821/762368443] exists,up 4d2612be-7068-48d9-bb0c-992ebccaa091', 'osd.3 up in weight 1 up_from 686 up_thru 1153 down_at 682 last_clean_interval [25,682) [v2:172.21.15.136:6826/946860336,v1:172.21.15.136:6827/946860336] [v2:172.21.15.136:6828/946860336,v1:172.21.15.136:6829/946860336] exists,up 30b8a4f2-be6d-4d5f-8397-06c2e17cfc2b', 'osd.4 up in weight 1 up_from 30 up_thru 1153 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6800/842623090,v1:172.21.15.165:6801/842623090] [v2:172.21.15.165:6802/842623090,v1:172.21.15.165:6803/842623090] exists,up ae4074cf-0069-403f-acc7-74bedab266bd', 'osd.5 up in weight 1 up_from 38 up_thru 1153 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6808/3602301990,v1:172.21.15.165:6809/3602301990] [v2:172.21.15.165:6810/3602301990,v1:172.21.15.165:6811/3602301990] exists,up 586c23a7-4831-4b4d-9a06-c1bfcc1c5286', 'osd.6 up in weight 1 up_from 46 up_thru 1153 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6816/1795674907,v1:172.21.15.165:6817/1795674907] [v2:172.21.15.165:6818/1795674907,v1:172.21.15.165:6819/1795674907] exists,up e838c26b-5e4a-41c6-b35b-b438ba64b237', 'osd.7 up in weight 1 up_from 54 up_thru 1153 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6824/4195921091,v1:172.21.15.165:6825/4195921091] [v2:172.21.15.165:6826/4195921091,v1:172.21.15.165:6827/4195921091] exists,up de2fab2e-177a-4eb0-9262-7bbef0d9837f'] 2024-08-17T17:52:57.684 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon mon.a config show 2024-08-17T17:52:57.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:57 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1121127056' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-08-17T17:52:57.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:57 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1121127056' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-08-17T17:52:57.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:52:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:52:57] "GET /metrics HTTP/1.1" 200 34058 "" "Prometheus/2.51.0" 2024-08-17T17:52:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:57 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1121127056' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-08-17T17:52:57.886 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:52:58.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:52:58 smithi136 ceph-mon[109861]: pgmap v2064: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:58.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:52:58 smithi136 ceph-mon[115078]: pgmap v2064: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:52:58 smithi165 ceph-mon[105517]: pgmap v2064: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:52:59.610 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:52:59.610 INFO:teuthology.orchestra.run.smithi136.stdout: "name": "mon.a", 2024-08-17T17:52:59.610 INFO:teuthology.orchestra.run.smithi136.stdout: "cluster": "ceph", 2024-08-17T17:52:59.610 INFO:teuthology.orchestra.run.smithi136.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-08-17T17:52:59.610 INFO:teuthology.orchestra.run.smithi136.stdout: "admin_socket_mode": "", 2024-08-17T17:52:59.610 INFO:teuthology.orchestra.run.smithi136.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-08-17T17:52:59.610 INFO:teuthology.orchestra.run.smithi136.stdout: "auth_client_required": "cephx, none", 2024-08-17T17:52:59.610 INFO:teuthology.orchestra.run.smithi136.stdout: "auth_cluster_required": "cephx", 2024-08-17T17:52:59.611 INFO:teuthology.orchestra.run.smithi136.stdout: "auth_debug": "true", 2024-08-17T17:52:59.611 INFO:teuthology.orchestra.run.smithi136.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-08-17T17:52:59.611 INFO:teuthology.orchestra.run.smithi136.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-08-17T17:52:59.611 INFO:teuthology.orchestra.run.smithi136.stdout: "auth_service_required": "cephx", 2024-08-17T17:52:59.611 INFO:teuthology.orchestra.run.smithi136.stdout: "auth_service_ticket_ttl": "240.000000", 2024-08-17T17:52:59.611 INFO:teuthology.orchestra.run.smithi136.stdout: "auth_supported": "", 2024-08-17T17:52:59.611 INFO:teuthology.orchestra.run.smithi136.stdout: "bdev_aio": "true", 2024-08-17T17:52:59.611 INFO:teuthology.orchestra.run.smithi136.stdout: "bdev_aio_max_queue_depth": "1024", 2024-08-17T17:52:59.611 INFO:teuthology.orchestra.run.smithi136.stdout: "bdev_aio_poll_ms": "250", 2024-08-17T17:52:59.611 INFO:teuthology.orchestra.run.smithi136.stdout: "bdev_aio_reap_max": "16", 2024-08-17T17:52:59.611 INFO:teuthology.orchestra.run.smithi136.stdout: "bdev_async_discard_threads": "0", 2024-08-17T17:52:59.611 INFO:teuthology.orchestra.run.smithi136.stdout: "bdev_block_size": "4096", 2024-08-17T17:52:59.612 INFO:teuthology.orchestra.run.smithi136.stdout: "bdev_debug_aio": "false", 2024-08-17T17:52:59.612 INFO:teuthology.orchestra.run.smithi136.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-08-17T17:52:59.612 INFO:teuthology.orchestra.run.smithi136.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-08-17T17:52:59.612 INFO:teuthology.orchestra.run.smithi136.stdout: "bdev_debug_inflight_ios": "false", 2024-08-17T17:52:59.612 INFO:teuthology.orchestra.run.smithi136.stdout: "bdev_enable_discard": "false", 2024-08-17T17:52:59.612 INFO:teuthology.orchestra.run.smithi136.stdout: "bdev_flock_retry": "3", 2024-08-17T17:52:59.612 INFO:teuthology.orchestra.run.smithi136.stdout: "bdev_flock_retry_interval": "0.100000", 2024-08-17T17:52:59.612 INFO:teuthology.orchestra.run.smithi136.stdout: "bdev_inject_crash": "0", 2024-08-17T17:52:59.612 INFO:teuthology.orchestra.run.smithi136.stdout: "bdev_inject_crash_flush_delay": "2", 2024-08-17T17:52:59.612 INFO:teuthology.orchestra.run.smithi136.stdout: "bdev_ioring": "false", 2024-08-17T17:52:59.612 INFO:teuthology.orchestra.run.smithi136.stdout: "bdev_ioring_hipri": "false", 2024-08-17T17:52:59.612 INFO:teuthology.orchestra.run.smithi136.stdout: "bdev_ioring_sqthread_poll": "false", 2024-08-17T17:52:59.612 INFO:teuthology.orchestra.run.smithi136.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-08-17T17:52:59.613 INFO:teuthology.orchestra.run.smithi136.stdout: "bdev_read_buffer_alignment": "4096", 2024-08-17T17:52:59.613 INFO:teuthology.orchestra.run.smithi136.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-08-17T17:52:59.613 INFO:teuthology.orchestra.run.smithi136.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-08-17T17:52:59.613 INFO:teuthology.orchestra.run.smithi136.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-08-17T17:52:59.613 INFO:teuthology.orchestra.run.smithi136.stdout: "bdev_type": "", 2024-08-17T17:52:59.613 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_alloc_size": "1048576", 2024-08-17T17:52:59.613 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_allocator": "hybrid", 2024-08-17T17:52:59.613 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_buffered_io": "true", 2024-08-17T17:52:59.613 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_check_for_zeros": "false", 2024-08-17T17:52:59.613 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_check_volume_selector_often": "false", 2024-08-17T17:52:59.613 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-08-17T17:52:59.613 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_compact_log_sync": "false", 2024-08-17T17:52:59.614 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-08-17T17:52:59.614 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-08-17T17:52:59.614 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_log_compact_min_size": "16777216", 2024-08-17T17:52:59.614 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_log_replay_check_allocations": "true", 2024-08-17T17:52:59.614 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_max_log_runway": "4194304", 2024-08-17T17:52:59.614 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_max_prefetch": "1048576", 2024-08-17T17:52:59.614 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_min_flush_size": "524288", 2024-08-17T17:52:59.614 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_min_log_runway": "1048576", 2024-08-17T17:52:59.614 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_replay_recovery": "false", 2024-08-17T17:52:59.614 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-08-17T17:52:59.614 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_shared_alloc_size": "65536", 2024-08-17T17:52:59.614 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_sync_write": "false", 2024-08-17T17:52:59.614 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-08-17T17:52:59.615 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-08-17T17:52:59.615 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-08-17T17:52:59.615 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_allocation_from_file": "true", 2024-08-17T17:52:59.615 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_allocator": "hybrid", 2024-08-17T17:52:59.615 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_async_db_compaction": "true", 2024-08-17T17:52:59.615 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-08-17T17:52:59.615 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-08-17T17:52:59.615 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-08-17T17:52:59.615 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-08-17T17:52:59.615 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_bdev_label_multi": "true", 2024-08-17T17:52:59.615 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-08-17T17:52:59.615 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_bdev_label_require_all": "true", 2024-08-17T17:52:59.615 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-08-17T17:52:59.615 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-08-17T17:52:59.615 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blobid_prealloc": "10240", 2024-08-17T17:52:59.616 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_block_create": "true", 2024-08-17T17:52:59.616 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_block_db_create": "false", 2024-08-17T17:52:59.616 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_block_db_path": "", 2024-08-17T17:52:59.616 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_block_db_size": "0", 2024-08-17T17:52:59.616 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_block_path": "", 2024-08-17T17:52:59.616 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_block_preallocate_file": "false", 2024-08-17T17:52:59.616 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_block_size": "107374182400", 2024-08-17T17:52:59.616 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_block_wal_create": "false", 2024-08-17T17:52:59.616 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_block_wal_path": "", 2024-08-17T17:52:59.616 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_block_wal_size": "100663296", 2024-08-17T17:52:59.616 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_bluefs": "true", 2024-08-17T17:52:59.616 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-08-17T17:52:59.616 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_bluefs_env_mirror": "false", 2024-08-17T17:52:59.616 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-08-17T17:52:59.617 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-08-17T17:52:59.617 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-08-17T17:52:59.617 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-08-17T17:52:59.617 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-08-17T17:52:59.617 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-08-17T17:52:59.617 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-08-17T17:52:59.617 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_autotune": "true", 2024-08-17T17:52:59.617 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-08-17T17:52:59.617 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-08-17T17:52:59.617 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-08-17T17:52:59.617 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-08-17T17:52:59.617 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_size": "0", 2024-08-17T17:52:59.617 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-08-17T17:52:59.618 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-08-17T17:52:59.618 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-08-17T17:52:59.618 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-08-17T17:52:59.618 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_type": "2q", 2024-08-17T17:52:59.618 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-08-17T17:52:59.618 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_clone_cow": "true", 2024-08-17T17:52:59.618 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_compression_algorithm": "snappy", 2024-08-17T17:52:59.618 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_compression_max_blob_size": "0", 2024-08-17T17:52:59.618 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-08-17T17:52:59.618 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-08-17T17:52:59.618 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_compression_min_blob_size": "0", 2024-08-17T17:52:59.618 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-08-17T17:52:59.618 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-08-17T17:52:59.619 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_compression_mode": "none", 2024-08-17T17:52:59.619 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-08-17T17:52:59.619 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_csum_type": "crc32c", 2024-08-17T17:52:59.619 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-08-17T17:52:59.619 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_debug_enforce_settings": "default", 2024-08-17T17:52:59.619 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_debug_freelist": "false", 2024-08-17T17:52:59.619 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_debug_fsck_abort": "false", 2024-08-17T17:52:59.619 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-08-17T17:52:59.619 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-08-17T17:52:59.619 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_debug_inject_read_err": "false", 2024-08-17T17:52:59.619 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_debug_legacy_omap": "false", 2024-08-17T17:52:59.619 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-08-17T17:52:59.619 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-08-17T17:52:59.619 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-08-17T17:52:59.620 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-08-17T17:52:59.620 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-08-17T17:52:59.620 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-08-17T17:52:59.620 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-08-17T17:52:59.620 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_debug_small_allocations": "0", 2024-08-17T17:52:59.620 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-08-17T17:52:59.620 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_default_buffered_read": "true", 2024-08-17T17:52:59.620 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_default_buffered_write": "false", 2024-08-17T17:52:59.620 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_deferred_batch_ops": "0", 2024-08-17T17:52:59.620 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-08-17T17:52:59.620 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-08-17T17:52:59.620 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_elastic_shared_blobs": "true", 2024-08-17T17:52:59.620 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-08-17T17:52:59.620 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-08-17T17:52:59.620 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-08-17T17:52:59.621 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-08-17T17:52:59.621 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-08-17T17:52:59.621 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fail_eio": "false", 2024-08-17T17:52:59.621 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-08-17T17:52:59.621 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-08-17T17:52:59.621 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-08-17T17:52:59.621 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-08-17T17:52:59.621 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_on_mkfs": "true", 2024-08-17T17:52:59.621 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-08-17T17:52:59.621 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_on_mount": "false", 2024-08-17T17:52:59.621 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-08-17T17:52:59.621 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_on_umount": "false", 2024-08-17T17:52:59.621 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-08-17T17:52:59.621 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-08-17T17:52:59.621 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-08-17T17:52:59.622 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-08-17T17:52:59.622 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-08-17T17:52:59.622 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-08-17T17:52:59.622 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-08-17T17:52:59.622 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-08-17T17:52:59.622 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_ignore_data_csum": "false", 2024-08-17T17:52:59.622 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-08-17T17:52:59.622 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_kvbackend": "rocksdb", 2024-08-17T17:52:59.622 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-08-17T17:52:59.622 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-08-17T17:52:59.622 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_log_op_age": "5.000000", 2024-08-17T17:52:59.622 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_max_alloc_size": "0", 2024-08-17T17:52:59.623 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_max_blob_size": "0", 2024-08-17T17:52:59.623 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-08-17T17:52:59.623 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-08-17T17:52:59.623 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_max_defer_interval": "3.000000", 2024-08-17T17:52:59.623 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_max_deferred_txc": "32", 2024-08-17T17:52:59.623 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_min_alloc_size": "0", 2024-08-17T17:52:59.623 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-08-17T17:52:59.623 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-08-17T17:52:59.623 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_nid_prealloc": "1024", 2024-08-17T17:52:59.623 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_prefer_deferred_size": "0", 2024-08-17T17:52:59.623 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-08-17T17:52:59.623 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-08-17T17:52:59.623 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_qfsck_on_mount": "true", 2024-08-17T17:52:59.624 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_retry_disk_reads": "3", 2024-08-17T17:52:59.624 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_rocksdb_cf": "true", 2024-08-17T17:52:59.624 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-08-17T17:52:59.625 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-08-17T17:52:59.625 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_rocksdb_options_annex": "", 2024-08-17T17:52:59.625 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-08-17T17:52:59.625 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-08-17T17:52:59.625 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_spdk_coremask": "0x1", 2024-08-17T17:52:59.625 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_spdk_io_sleep": "5", 2024-08-17T17:52:59.625 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_spdk_max_io_completion": "0", 2024-08-17T17:52:59.625 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_spdk_mem": "512", 2024-08-17T17:52:59.625 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_sync_submit_transaction": "false", 2024-08-17T17:52:59.625 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_throttle_bytes": "67108864", 2024-08-17T17:52:59.625 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_throttle_cost_per_io": "0", 2024-08-17T17:52:59.625 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-08-17T17:52:59.625 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-08-17T17:52:59.626 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-08-17T17:52:59.626 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-08-17T17:52:59.626 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_tracing": "false", 2024-08-17T17:52:59.626 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-08-17T17:52:59.626 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-08-17T17:52:59.626 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_volume_selection_reserved": "0", 2024-08-17T17:52:59.626 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-08-17T17:52:59.626 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-08-17T17:52:59.626 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-08-17T17:52:59.626 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-08-17T17:52:59.626 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-08-17T17:52:59.626 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-08-17T17:52:59.626 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_write_v2": "false", 2024-08-17T17:52:59.627 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_write_v2_random": "false", 2024-08-17T17:52:59.627 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_zero_block_detection": "false", 2024-08-17T17:52:59.627 INFO:teuthology.orchestra.run.smithi136.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-08-17T17:52:59.627 INFO:teuthology.orchestra.run.smithi136.stdout: "cephfs_mirror_action_update_interval": "2", 2024-08-17T17:52:59.627 INFO:teuthology.orchestra.run.smithi136.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-08-17T17:52:59.627 INFO:teuthology.orchestra.run.smithi136.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-08-17T17:52:59.627 INFO:teuthology.orchestra.run.smithi136.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-08-17T17:52:59.627 INFO:teuthology.orchestra.run.smithi136.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-08-17T17:52:59.627 INFO:teuthology.orchestra.run.smithi136.stdout: "cephfs_mirror_mount_timeout": "10", 2024-08-17T17:52:59.627 INFO:teuthology.orchestra.run.smithi136.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-08-17T17:52:59.627 INFO:teuthology.orchestra.run.smithi136.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-08-17T17:52:59.627 INFO:teuthology.orchestra.run.smithi136.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-08-17T17:52:59.627 INFO:teuthology.orchestra.run.smithi136.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-08-17T17:52:59.627 INFO:teuthology.orchestra.run.smithi136.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-08-17T17:52:59.628 INFO:teuthology.orchestra.run.smithi136.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-08-17T17:52:59.628 INFO:teuthology.orchestra.run.smithi136.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-08-17T17:52:59.628 INFO:teuthology.orchestra.run.smithi136.stdout: "cephx_cluster_require_signatures": "false", 2024-08-17T17:52:59.628 INFO:teuthology.orchestra.run.smithi136.stdout: "cephx_cluster_require_version": "2", 2024-08-17T17:52:59.628 INFO:teuthology.orchestra.run.smithi136.stdout: "cephx_require_signatures": "false", 2024-08-17T17:52:59.628 INFO:teuthology.orchestra.run.smithi136.stdout: "cephx_require_version": "2", 2024-08-17T17:52:59.628 INFO:teuthology.orchestra.run.smithi136.stdout: "cephx_service_require_signatures": "false", 2024-08-17T17:52:59.628 INFO:teuthology.orchestra.run.smithi136.stdout: "cephx_service_require_version": "2", 2024-08-17T17:52:59.628 INFO:teuthology.orchestra.run.smithi136.stdout: "cephx_sign_messages": "true", 2024-08-17T17:52:59.628 INFO:teuthology.orchestra.run.smithi136.stdout: "chdir": "", 2024-08-17T17:52:59.628 INFO:teuthology.orchestra.run.smithi136.stdout: "client_acl_type": "", 2024-08-17T17:52:59.628 INFO:teuthology.orchestra.run.smithi136.stdout: "client_asio_thread_count": "2", 2024-08-17T17:52:59.628 INFO:teuthology.orchestra.run.smithi136.stdout: "client_cache_mid": "0.750000", 2024-08-17T17:52:59.629 INFO:teuthology.orchestra.run.smithi136.stdout: "client_cache_size": "16384", 2024-08-17T17:52:59.629 INFO:teuthology.orchestra.run.smithi136.stdout: "client_caps_release_delay": "5", 2024-08-17T17:52:59.629 INFO:teuthology.orchestra.run.smithi136.stdout: "client_check_pool_perm": "true", 2024-08-17T17:52:59.629 INFO:teuthology.orchestra.run.smithi136.stdout: "client_collect_and_send_global_metrics": "false", 2024-08-17T17:52:59.629 INFO:teuthology.orchestra.run.smithi136.stdout: "client_debug_force_sync_read": "false", 2024-08-17T17:52:59.629 INFO:teuthology.orchestra.run.smithi136.stdout: "client_debug_getattr_caps": "false", 2024-08-17T17:52:59.629 INFO:teuthology.orchestra.run.smithi136.stdout: "client_debug_inject_features": "", 2024-08-17T17:52:59.629 INFO:teuthology.orchestra.run.smithi136.stdout: "client_debug_inject_tick_delay": "0", 2024-08-17T17:52:59.629 INFO:teuthology.orchestra.run.smithi136.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-08-17T17:52:59.630 INFO:teuthology.orchestra.run.smithi136.stdout: "client_die_on_failed_remount": "false", 2024-08-17T17:52:59.630 INFO:teuthology.orchestra.run.smithi136.stdout: "client_dirsize_rbytes": "true", 2024-08-17T17:52:59.630 INFO:teuthology.orchestra.run.smithi136.stdout: "client_force_lazyio": "false", 2024-08-17T17:52:59.630 INFO:teuthology.orchestra.run.smithi136.stdout: "client_fs": "", 2024-08-17T17:52:59.630 INFO:teuthology.orchestra.run.smithi136.stdout: "client_inject_fixed_oldest_tid": "false", 2024-08-17T17:52:59.630 INFO:teuthology.orchestra.run.smithi136.stdout: "client_inject_release_failure": "false", 2024-08-17T17:52:59.630 INFO:teuthology.orchestra.run.smithi136.stdout: "client_max_inline_size": "4096", 2024-08-17T17:52:59.630 INFO:teuthology.orchestra.run.smithi136.stdout: "client_max_retries_on_remount_failure": "5", 2024-08-17T17:52:59.630 INFO:teuthology.orchestra.run.smithi136.stdout: "client_mds_namespace": "", 2024-08-17T17:52:59.630 INFO:teuthology.orchestra.run.smithi136.stdout: "client_metadata": "", 2024-08-17T17:52:59.630 INFO:teuthology.orchestra.run.smithi136.stdout: "client_mount_gid": "-1", 2024-08-17T17:52:59.630 INFO:teuthology.orchestra.run.smithi136.stdout: "client_mount_timeout": "300", 2024-08-17T17:52:59.630 INFO:teuthology.orchestra.run.smithi136.stdout: "client_mount_uid": "-1", 2024-08-17T17:52:59.630 INFO:teuthology.orchestra.run.smithi136.stdout: "client_mountpoint": "/", 2024-08-17T17:52:59.631 INFO:teuthology.orchestra.run.smithi136.stdout: "client_notify_timeout": "10", 2024-08-17T17:52:59.631 INFO:teuthology.orchestra.run.smithi136.stdout: "client_oc": "true", 2024-08-17T17:52:59.631 INFO:teuthology.orchestra.run.smithi136.stdout: "client_oc_max_dirty": "104857600", 2024-08-17T17:52:59.631 INFO:teuthology.orchestra.run.smithi136.stdout: "client_oc_max_dirty_age": "5.000000", 2024-08-17T17:52:59.631 INFO:teuthology.orchestra.run.smithi136.stdout: "client_oc_max_objects": "1000", 2024-08-17T17:52:59.631 INFO:teuthology.orchestra.run.smithi136.stdout: "client_oc_size": "209715200", 2024-08-17T17:52:59.631 INFO:teuthology.orchestra.run.smithi136.stdout: "client_oc_target_dirty": "8388608", 2024-08-17T17:52:59.631 INFO:teuthology.orchestra.run.smithi136.stdout: "client_permissions": "true", 2024-08-17T17:52:59.631 INFO:teuthology.orchestra.run.smithi136.stdout: "client_quota": "true", 2024-08-17T17:52:59.631 INFO:teuthology.orchestra.run.smithi136.stdout: "client_quota_df": "true", 2024-08-17T17:52:59.631 INFO:teuthology.orchestra.run.smithi136.stdout: "client_readahead_max_bytes": "0", 2024-08-17T17:52:59.631 INFO:teuthology.orchestra.run.smithi136.stdout: "client_readahead_max_periods": "4", 2024-08-17T17:52:59.631 INFO:teuthology.orchestra.run.smithi136.stdout: "client_readahead_min": "131072", 2024-08-17T17:52:59.631 INFO:teuthology.orchestra.run.smithi136.stdout: "client_reconnect_stale": "false", 2024-08-17T17:52:59.632 INFO:teuthology.orchestra.run.smithi136.stdout: "client_shutdown_timeout": "30", 2024-08-17T17:52:59.632 INFO:teuthology.orchestra.run.smithi136.stdout: "client_snapdir": ".snap", 2024-08-17T17:52:59.632 INFO:teuthology.orchestra.run.smithi136.stdout: "client_tick_interval": "1", 2024-08-17T17:52:59.632 INFO:teuthology.orchestra.run.smithi136.stdout: "client_trace": "", 2024-08-17T17:52:59.632 INFO:teuthology.orchestra.run.smithi136.stdout: "client_try_dentry_invalidate": "false", 2024-08-17T17:52:59.632 INFO:teuthology.orchestra.run.smithi136.stdout: "client_use_faked_inos": "false", 2024-08-17T17:52:59.632 INFO:teuthology.orchestra.run.smithi136.stdout: "client_use_random_mds": "false", 2024-08-17T17:52:59.632 INFO:teuthology.orchestra.run.smithi136.stdout: "clog_to_graylog": "false", 2024-08-17T17:52:59.632 INFO:teuthology.orchestra.run.smithi136.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-08-17T17:52:59.632 INFO:teuthology.orchestra.run.smithi136.stdout: "clog_to_graylog_port": "12201", 2024-08-17T17:52:59.632 INFO:teuthology.orchestra.run.smithi136.stdout: "clog_to_monitors": "default=true", 2024-08-17T17:52:59.632 INFO:teuthology.orchestra.run.smithi136.stdout: "clog_to_syslog": "false", 2024-08-17T17:52:59.632 INFO:teuthology.orchestra.run.smithi136.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-08-17T17:52:59.633 INFO:teuthology.orchestra.run.smithi136.stdout: "clog_to_syslog_level": "info", 2024-08-17T17:52:59.633 INFO:teuthology.orchestra.run.smithi136.stdout: "cluster_addr": "-", 2024-08-17T17:52:59.633 INFO:teuthology.orchestra.run.smithi136.stdout: "cluster_network": "", 2024-08-17T17:52:59.633 INFO:teuthology.orchestra.run.smithi136.stdout: "cluster_network_interface": "", 2024-08-17T17:52:59.633 INFO:teuthology.orchestra.run.smithi136.stdout: "compressor_zlib_isal": "false", 2024-08-17T17:52:59.633 INFO:teuthology.orchestra.run.smithi136.stdout: "compressor_zlib_level": "5", 2024-08-17T17:52:59.633 INFO:teuthology.orchestra.run.smithi136.stdout: "compressor_zlib_winsize": "-15", 2024-08-17T17:52:59.633 INFO:teuthology.orchestra.run.smithi136.stdout: "compressor_zstd_level": "1", 2024-08-17T17:52:59.633 INFO:teuthology.orchestra.run.smithi136.stdout: "container_image": "quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d", 2024-08-17T17:52:59.633 INFO:teuthology.orchestra.run.smithi136.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-08-17T17:52:59.633 INFO:teuthology.orchestra.run.smithi136.stdout: "crimson_alien_op_num_threads": "6", 2024-08-17T17:52:59.633 INFO:teuthology.orchestra.run.smithi136.stdout: "crimson_alien_thread_cpu_cores": "", 2024-08-17T17:52:59.633 INFO:teuthology.orchestra.run.smithi136.stdout: "crimson_osd_obc_lru_size": "512", 2024-08-17T17:52:59.633 INFO:teuthology.orchestra.run.smithi136.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-08-17T17:52:59.634 INFO:teuthology.orchestra.run.smithi136.stdout: "crimson_osd_stat_interval": "0", 2024-08-17T17:52:59.634 INFO:teuthology.orchestra.run.smithi136.stdout: "crimson_seastar_cpu_cores": "", 2024-08-17T17:52:59.634 INFO:teuthology.orchestra.run.smithi136.stdout: "crimson_seastar_num_threads": "0", 2024-08-17T17:52:59.634 INFO:teuthology.orchestra.run.smithi136.stdout: "crush_location": "", 2024-08-17T17:52:59.634 INFO:teuthology.orchestra.run.smithi136.stdout: "crush_location_hook": "", 2024-08-17T17:52:59.634 INFO:teuthology.orchestra.run.smithi136.stdout: "crush_location_hook_timeout": "10", 2024-08-17T17:52:59.634 INFO:teuthology.orchestra.run.smithi136.stdout: "daemonize": "false", 2024-08-17T17:52:59.634 INFO:teuthology.orchestra.run.smithi136.stdout: "daos_pool": "tank", 2024-08-17T17:52:59.634 INFO:teuthology.orchestra.run.smithi136.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-08-17T17:52:59.634 INFO:teuthology.orchestra.run.smithi136.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-08-17T17:52:59.634 INFO:teuthology.orchestra.run.smithi136.stdout: "dbstore_db_name_prefix": "dbstore", 2024-08-17T17:52:59.634 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_alienstore": "0/5", 2024-08-17T17:52:59.635 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_allow_any_pool_priority": "false", 2024-08-17T17:52:59.635 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_asok": "1/5", 2024-08-17T17:52:59.635 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_asok_assert_abort": "false", 2024-08-17T17:52:59.635 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_asserts_on_shutdown": "true", 2024-08-17T17:52:59.635 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_auth": "1/5", 2024-08-17T17:52:59.635 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_bdev": "1/3", 2024-08-17T17:52:59.635 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_bluefs": "1/5", 2024-08-17T17:52:59.635 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_bluestore": "1/5", 2024-08-17T17:52:59.635 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_buffer": "0/1", 2024-08-17T17:52:59.635 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_ceph_dedup": "0/5", 2024-08-17T17:52:59.635 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_ceph_exporter": "1/5", 2024-08-17T17:52:59.635 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_cephfs_mirror": "0/5", 2024-08-17T17:52:59.635 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_cephsqlite": "0/5", 2024-08-17T17:52:59.635 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_client": "0/5", 2024-08-17T17:52:59.636 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_compressor": "1/5", 2024-08-17T17:52:59.636 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_context": "0/1", 2024-08-17T17:52:59.636 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_crimson_interrupt": "0/5", 2024-08-17T17:52:59.636 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_crush": "1/1", 2024-08-17T17:52:59.636 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_crypto": "1/5", 2024-08-17T17:52:59.636 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_cyanstore": "0/5", 2024-08-17T17:52:59.636 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_deliberately_leak_memory": "false", 2024-08-17T17:52:59.636 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_disable_randomized_ping": "false", 2024-08-17T17:52:59.636 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_dpdk": "1/5", 2024-08-17T17:52:59.636 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_eventtrace": "1/5", 2024-08-17T17:52:59.636 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_filer": "0/1", 2024-08-17T17:52:59.636 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_filestore": "1/3", 2024-08-17T17:52:59.636 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_finisher": "1/1", 2024-08-17T17:52:59.637 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_fuse": "1/5", 2024-08-17T17:52:59.637 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_heartbeat_testing_span": "0", 2024-08-17T17:52:59.637 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_heartbeatmap": "1/5", 2024-08-17T17:52:59.637 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_immutable_obj_cache": "0/5", 2024-08-17T17:52:59.637 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_javaclient": "1/5", 2024-08-17T17:52:59.637 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_journal": "1/3", 2024-08-17T17:52:59.637 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_journaler": "0/5", 2024-08-17T17:52:59.637 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_kstore": "1/5", 2024-08-17T17:52:59.637 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_lockdep": "0/1", 2024-08-17T17:52:59.637 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_mclock": "1/5", 2024-08-17T17:52:59.637 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_mds": "1/5", 2024-08-17T17:52:59.637 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_mds_balancer": "1/5", 2024-08-17T17:52:59.637 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_mds_locker": "1/5", 2024-08-17T17:52:59.637 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_mds_log": "1/5", 2024-08-17T17:52:59.637 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_mds_log_expire": "1/5", 2024-08-17T17:52:59.638 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_mds_migrator": "1/5", 2024-08-17T17:52:59.638 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_mds_quiesce": "3/5", 2024-08-17T17:52:59.638 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_memstore": "1/5", 2024-08-17T17:52:59.638 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_mgr": "2/5", 2024-08-17T17:52:59.638 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_mgrc": "1/5", 2024-08-17T17:52:59.638 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_mon": "20/20", 2024-08-17T17:52:59.638 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_monc": "0/10", 2024-08-17T17:52:59.638 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_ms": "1/1", 2024-08-17T17:52:59.638 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_none": "0/5", 2024-08-17T17:52:59.638 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_objclass": "0/5", 2024-08-17T17:52:59.638 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_objectcacher": "0/5", 2024-08-17T17:52:59.638 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_objecter": "0/1", 2024-08-17T17:52:59.639 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_optracker": "0/5", 2024-08-17T17:52:59.639 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_osd": "1/5", 2024-08-17T17:52:59.639 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_paxos": "20/20", 2024-08-17T17:52:59.639 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_perfcounter": "1/5", 2024-08-17T17:52:59.639 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_prioritycache": "1/5", 2024-08-17T17:52:59.639 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_rados": "0/5", 2024-08-17T17:52:59.639 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_rbd": "0/5", 2024-08-17T17:52:59.639 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_rbd_mirror": "0/5", 2024-08-17T17:52:59.639 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_rbd_pwl": "0/5", 2024-08-17T17:52:59.639 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_rbd_replay": "0/5", 2024-08-17T17:52:59.639 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_refs": "0/0", 2024-08-17T17:52:59.639 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_reserver": "1/1", 2024-08-17T17:52:59.639 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_rgw": "1/5", 2024-08-17T17:52:59.640 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_rgw_access": "1/5", 2024-08-17T17:52:59.640 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_rgw_datacache": "1/5", 2024-08-17T17:52:59.640 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_rgw_dbstore": "1/5", 2024-08-17T17:52:59.640 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_rgw_flight": "1/5", 2024-08-17T17:52:59.640 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_rgw_lifecycle": "1/5", 2024-08-17T17:52:59.640 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_rgw_notification": "1/5", 2024-08-17T17:52:59.640 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_rgw_sync": "1/5", 2024-08-17T17:52:59.640 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_rocksdb": "4/5", 2024-08-17T17:52:59.640 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_seastore": "0/5", 2024-08-17T17:52:59.640 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_seastore_backref": "0/5", 2024-08-17T17:52:59.640 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_seastore_cache": "0/5", 2024-08-17T17:52:59.640 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_seastore_cleaner": "0/5", 2024-08-17T17:52:59.640 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_seastore_device": "0/5", 2024-08-17T17:52:59.640 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_seastore_epm": "0/5", 2024-08-17T17:52:59.640 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-08-17T17:52:59.641 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_seastore_journal": "0/5", 2024-08-17T17:52:59.641 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_seastore_lba": "0/5", 2024-08-17T17:52:59.641 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_seastore_odata": "0/5", 2024-08-17T17:52:59.641 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_seastore_omap": "0/5", 2024-08-17T17:52:59.641 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_seastore_onode": "0/5", 2024-08-17T17:52:59.641 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_seastore_t": "0/5", 2024-08-17T17:52:59.641 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_seastore_tm": "0/5", 2024-08-17T17:52:59.641 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_striper": "0/1", 2024-08-17T17:52:59.641 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_test": "0/5", 2024-08-17T17:52:59.641 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_throttle": "1/1", 2024-08-17T17:52:59.641 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_timer": "0/1", 2024-08-17T17:52:59.641 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_tp": "0/5", 2024-08-17T17:52:59.641 INFO:teuthology.orchestra.run.smithi136.stdout: "debug_trace": "1/5", 2024-08-17T17:52:59.641 INFO:teuthology.orchestra.run.smithi136.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-08-17T17:52:59.642 INFO:teuthology.orchestra.run.smithi136.stdout: "device_failure_prediction_mode": "none", 2024-08-17T17:52:59.642 INFO:teuthology.orchestra.run.smithi136.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-08-17T17:52:59.642 INFO:teuthology.orchestra.run.smithi136.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-08-17T17:52:59.642 INFO:teuthology.orchestra.run.smithi136.stdout: "err_to_graylog": "false", 2024-08-17T17:52:59.642 INFO:teuthology.orchestra.run.smithi136.stdout: "err_to_journald": "false", 2024-08-17T17:52:59.642 INFO:teuthology.orchestra.run.smithi136.stdout: "err_to_stderr": "true", 2024-08-17T17:52:59.642 INFO:teuthology.orchestra.run.smithi136.stdout: "err_to_syslog": "false", 2024-08-17T17:52:59.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event_tracing": "false", 2024-08-17T17:52:59.642 INFO:teuthology.orchestra.run.smithi136.stdout: "exporter_addr": "0.0.0.0", 2024-08-17T17:52:59.642 INFO:teuthology.orchestra.run.smithi136.stdout: "exporter_cert_file": "", 2024-08-17T17:52:59.642 INFO:teuthology.orchestra.run.smithi136.stdout: "exporter_http_port": "9926", 2024-08-17T17:52:59.642 INFO:teuthology.orchestra.run.smithi136.stdout: "exporter_key_file": "", 2024-08-17T17:52:59.643 INFO:teuthology.orchestra.run.smithi136.stdout: "exporter_prio_limit": "5", 2024-08-17T17:52:59.643 INFO:teuthology.orchestra.run.smithi136.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-08-17T17:52:59.643 INFO:teuthology.orchestra.run.smithi136.stdout: "exporter_sort_metrics": "true", 2024-08-17T17:52:59.643 INFO:teuthology.orchestra.run.smithi136.stdout: "exporter_stats_period": "5", 2024-08-17T17:52:59.643 INFO:teuthology.orchestra.run.smithi136.stdout: "fake_statfs_for_testing": "0", 2024-08-17T17:52:59.643 INFO:teuthology.orchestra.run.smithi136.stdout: "fatal_signal_handlers": "true", 2024-08-17T17:52:59.643 INFO:teuthology.orchestra.run.smithi136.stdout: "filer_max_purge_ops": "10", 2024-08-17T17:52:59.643 INFO:teuthology.orchestra.run.smithi136.stdout: "filer_max_truncate_ops": "128", 2024-08-17T17:52:59.643 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_apply_finisher_threads": "1", 2024-08-17T17:52:59.643 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_blackhole": "false", 2024-08-17T17:52:59.643 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_btrfs_clone_range": "true", 2024-08-17T17:52:59.643 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_btrfs_snap": "true", 2024-08-17T17:52:59.643 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_caller_concurrency": "10", 2024-08-17T17:52:59.643 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_collect_device_partition_information": "true", 2024-08-17T17:52:59.643 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_commit_timeout": "600.000000", 2024-08-17T17:52:59.644 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_debug_inject_read_err": "false", 2024-08-17T17:52:59.644 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_debug_omap_check": "false", 2024-08-17T17:52:59.644 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_debug_verify_split": "false", 2024-08-17T17:52:59.644 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_dump_file": "", 2024-08-17T17:52:59.644 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-08-17T17:52:59.644 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-08-17T17:52:59.644 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_fadvise": "true", 2024-08-17T17:52:59.644 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_fail_eio": "true", 2024-08-17T17:52:59.644 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_fd_cache_shards": "16", 2024-08-17T17:52:59.644 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_fd_cache_size": "128", 2024-08-17T17:52:59.644 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_fiemap": "false", 2024-08-17T17:52:59.644 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_fiemap_threshold": "4096", 2024-08-17T17:52:59.645 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-08-17T17:52:59.645 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_index_retry_probability": "0.000000", 2024-08-17T17:52:59.645 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_inject_stall": "0", 2024-08-17T17:52:59.645 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_journal_parallel": "false", 2024-08-17T17:52:59.645 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_journal_trailing": "false", 2024-08-17T17:52:59.645 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_journal_writeahead": "false", 2024-08-17T17:52:59.645 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_kill_at": "0", 2024-08-17T17:52:59.645 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-08-17T17:52:59.645 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_max_inline_xattr_size": "0", 2024-08-17T17:52:59.645 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-08-17T17:52:59.645 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-08-17T17:52:59.645 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-08-17T17:52:59.645 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_max_inline_xattrs": "0", 2024-08-17T17:52:59.645 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-08-17T17:52:59.646 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_max_inline_xattrs_other": "2", 2024-08-17T17:52:59.646 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-08-17T17:52:59.646 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_max_sync_interval": "5.000000", 2024-08-17T17:52:59.646 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_max_xattr_value_size": "0", 2024-08-17T17:52:59.646 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-08-17T17:52:59.646 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-08-17T17:52:59.646 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-08-17T17:52:59.646 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_merge_threshold": "-10", 2024-08-17T17:52:59.646 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_min_sync_interval": "0.010000", 2024-08-17T17:52:59.646 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_odsync_write": "false", 2024-08-17T17:52:59.646 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_omap_backend": "rocksdb", 2024-08-17T17:52:59.646 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_omap_backend_path": "", 2024-08-17T17:52:59.647 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_omap_header_cache_size": "1024", 2024-08-17T17:52:59.647 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_ondisk_finisher_threads": "1", 2024-08-17T17:52:59.647 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-08-17T17:52:59.647 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_op_thread_timeout": "60", 2024-08-17T17:52:59.647 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_op_threads": "2", 2024-08-17T17:52:59.647 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_punch_hole": "false", 2024-08-17T17:52:59.647 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-08-17T17:52:59.647 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-08-17T17:52:59.647 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-08-17T17:52:59.647 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-08-17T17:52:59.647 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-08-17T17:52:59.647 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_queue_max_bytes": "104857600", 2024-08-17T17:52:59.647 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-08-17T17:52:59.648 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-08-17T17:52:59.648 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-08-17T17:52:59.648 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_queue_max_ops": "50", 2024-08-17T17:52:59.648 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-08-17T17:52:59.648 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_seek_data_hole": "false", 2024-08-17T17:52:59.648 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_sloppy_crc": "false", 2024-08-17T17:52:59.648 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-08-17T17:52:59.648 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_splice": "false", 2024-08-17T17:52:59.648 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_split_multiple": "2", 2024-08-17T17:52:59.648 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_split_rand_factor": "20", 2024-08-17T17:52:59.648 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_update_to": "1000", 2024-08-17T17:52:59.648 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-08-17T17:52:59.648 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-08-17T17:52:59.648 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-08-17T17:52:59.648 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-08-17T17:52:59.649 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-08-17T17:52:59.649 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-08-17T17:52:59.649 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_wbthrottle_enable": "true", 2024-08-17T17:52:59.649 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-08-17T17:52:59.649 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-08-17T17:52:59.649 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-08-17T17:52:59.649 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-08-17T17:52:59.649 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-08-17T17:52:59.649 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-08-17T17:52:59.649 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_xfs_extsize": "false", 2024-08-17T17:52:59.649 INFO:teuthology.orchestra.run.smithi136.stdout: "filestore_zfs_snap": "false", 2024-08-17T17:52:59.649 INFO:teuthology.orchestra.run.smithi136.stdout: "fio_dir": "/tmp/fio", 2024-08-17T17:52:59.649 INFO:teuthology.orchestra.run.smithi136.stdout: "fsid": "8270436c-5cba-11ef-bccf-c7b262605968", 2024-08-17T17:52:59.650 INFO:teuthology.orchestra.run.smithi136.stdout: "fuse_allow_other": "true", 2024-08-17T17:52:59.650 INFO:teuthology.orchestra.run.smithi136.stdout: "fuse_atomic_o_trunc": "true", 2024-08-17T17:52:59.650 INFO:teuthology.orchestra.run.smithi136.stdout: "fuse_big_writes": "true", 2024-08-17T17:52:59.650 INFO:teuthology.orchestra.run.smithi136.stdout: "fuse_debug": "false", 2024-08-17T17:52:59.650 INFO:teuthology.orchestra.run.smithi136.stdout: "fuse_default_permissions": "false", 2024-08-17T17:52:59.650 INFO:teuthology.orchestra.run.smithi136.stdout: "fuse_disable_pagecache": "false", 2024-08-17T17:52:59.650 INFO:teuthology.orchestra.run.smithi136.stdout: "fuse_max_write": "0", 2024-08-17T17:52:59.650 INFO:teuthology.orchestra.run.smithi136.stdout: "fuse_multithreaded": "true", 2024-08-17T17:52:59.650 INFO:teuthology.orchestra.run.smithi136.stdout: "fuse_require_active_mds": "true", 2024-08-17T17:52:59.650 INFO:teuthology.orchestra.run.smithi136.stdout: "fuse_set_user_groups": "true", 2024-08-17T17:52:59.650 INFO:teuthology.orchestra.run.smithi136.stdout: "fuse_splice_move": "true", 2024-08-17T17:52:59.650 INFO:teuthology.orchestra.run.smithi136.stdout: "fuse_splice_read": "true", 2024-08-17T17:52:59.650 INFO:teuthology.orchestra.run.smithi136.stdout: "fuse_splice_write": "true", 2024-08-17T17:52:59.651 INFO:teuthology.orchestra.run.smithi136.stdout: "fuse_syncfs_on_mksnap": "true", 2024-08-17T17:52:59.651 INFO:teuthology.orchestra.run.smithi136.stdout: "fuse_use_invalidate_cb": "true", 2024-08-17T17:52:59.651 INFO:teuthology.orchestra.run.smithi136.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-08-17T17:52:59.651 INFO:teuthology.orchestra.run.smithi136.stdout: "gss_target_name": "ceph", 2024-08-17T17:52:59.651 INFO:teuthology.orchestra.run.smithi136.stdout: "heartbeat_file": "", 2024-08-17T17:52:59.651 INFO:teuthology.orchestra.run.smithi136.stdout: "heartbeat_inject_failure": "0", 2024-08-17T17:52:59.651 INFO:teuthology.orchestra.run.smithi136.stdout: "heartbeat_interval": "5", 2024-08-17T17:52:59.651 INFO:teuthology.orchestra.run.smithi136.stdout: "host": "smithi136", 2024-08-17T17:52:59.651 INFO:teuthology.orchestra.run.smithi136.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-08-17T17:52:59.651 INFO:teuthology.orchestra.run.smithi136.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-08-17T17:52:59.651 INFO:teuthology.orchestra.run.smithi136.stdout: "immutable_object_cache_max_size": "1073741824", 2024-08-17T17:52:59.651 INFO:teuthology.orchestra.run.smithi136.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-08-17T17:52:59.651 INFO:teuthology.orchestra.run.smithi136.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-08-17T17:52:59.651 INFO:teuthology.orchestra.run.smithi136.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-08-17T17:52:59.652 INFO:teuthology.orchestra.run.smithi136.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-08-17T17:52:59.652 INFO:teuthology.orchestra.run.smithi136.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-08-17T17:52:59.652 INFO:teuthology.orchestra.run.smithi136.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-08-17T17:52:59.652 INFO:teuthology.orchestra.run.smithi136.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-08-17T17:52:59.652 INFO:teuthology.orchestra.run.smithi136.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-08-17T17:52:59.652 INFO:teuthology.orchestra.run.smithi136.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-08-17T17:52:59.652 INFO:teuthology.orchestra.run.smithi136.stdout: "immutable_object_cache_watermark": "0.900000", 2024-08-17T17:52:59.652 INFO:teuthology.orchestra.run.smithi136.stdout: "inject_early_sigterm": "false", 2024-08-17T17:52:59.652 INFO:teuthology.orchestra.run.smithi136.stdout: "jaeger_agent_port": "6799", 2024-08-17T17:52:59.652 INFO:teuthology.orchestra.run.smithi136.stdout: "jaeger_tracing_enable": "false", 2024-08-17T17:52:59.652 INFO:teuthology.orchestra.run.smithi136.stdout: "journal_aio": "true", 2024-08-17T17:52:59.652 INFO:teuthology.orchestra.run.smithi136.stdout: "journal_align_min_size": "65536", 2024-08-17T17:52:59.652 INFO:teuthology.orchestra.run.smithi136.stdout: "journal_block_align": "true", 2024-08-17T17:52:59.653 INFO:teuthology.orchestra.run.smithi136.stdout: "journal_block_size": "4096", 2024-08-17T17:52:59.653 INFO:teuthology.orchestra.run.smithi136.stdout: "journal_dio": "true", 2024-08-17T17:52:59.653 INFO:teuthology.orchestra.run.smithi136.stdout: "journal_discard": "false", 2024-08-17T17:52:59.653 INFO:teuthology.orchestra.run.smithi136.stdout: "journal_force_aio": "false", 2024-08-17T17:52:59.653 INFO:teuthology.orchestra.run.smithi136.stdout: "journal_ignore_corruption": "false", 2024-08-17T17:52:59.653 INFO:teuthology.orchestra.run.smithi136.stdout: "journal_max_write_bytes": "10485760", 2024-08-17T17:52:59.653 INFO:teuthology.orchestra.run.smithi136.stdout: "journal_max_write_entries": "100", 2024-08-17T17:52:59.653 INFO:teuthology.orchestra.run.smithi136.stdout: "journal_replay_from": "0", 2024-08-17T17:52:59.653 INFO:teuthology.orchestra.run.smithi136.stdout: "journal_throttle_high_multiple": "0.000000", 2024-08-17T17:52:59.653 INFO:teuthology.orchestra.run.smithi136.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-08-17T17:52:59.653 INFO:teuthology.orchestra.run.smithi136.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-08-17T17:52:59.653 INFO:teuthology.orchestra.run.smithi136.stdout: "journal_throttle_max_multiple": "0.000000", 2024-08-17T17:52:59.653 INFO:teuthology.orchestra.run.smithi136.stdout: "journal_write_header_frequency": "0", 2024-08-17T17:52:59.654 INFO:teuthology.orchestra.run.smithi136.stdout: "journal_zero_on_create": "false", 2024-08-17T17:52:59.654 INFO:teuthology.orchestra.run.smithi136.stdout: "journaler_prefetch_periods": "10", 2024-08-17T17:52:59.654 INFO:teuthology.orchestra.run.smithi136.stdout: "journaler_prezero_periods": "5", 2024-08-17T17:52:59.654 INFO:teuthology.orchestra.run.smithi136.stdout: "journaler_write_head_interval": "15", 2024-08-17T17:52:59.654 INFO:teuthology.orchestra.run.smithi136.stdout: "key": "", 2024-08-17T17:52:59.654 INFO:teuthology.orchestra.run.smithi136.stdout: "keyfile": "", 2024-08-17T17:52:59.654 INFO:teuthology.orchestra.run.smithi136.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-08-17T17:52:59.654 INFO:teuthology.orchestra.run.smithi136.stdout: "kstore_backend": "rocksdb", 2024-08-17T17:52:59.654 INFO:teuthology.orchestra.run.smithi136.stdout: "kstore_default_stripe_size": "65536", 2024-08-17T17:52:59.654 INFO:teuthology.orchestra.run.smithi136.stdout: "kstore_fsck_on_mount": "false", 2024-08-17T17:52:59.654 INFO:teuthology.orchestra.run.smithi136.stdout: "kstore_fsck_on_mount_deep": "true", 2024-08-17T17:52:59.654 INFO:teuthology.orchestra.run.smithi136.stdout: "kstore_max_bytes": "67108864", 2024-08-17T17:52:59.654 INFO:teuthology.orchestra.run.smithi136.stdout: "kstore_max_ops": "512", 2024-08-17T17:52:59.655 INFO:teuthology.orchestra.run.smithi136.stdout: "kstore_nid_prealloc": "1024", 2024-08-17T17:52:59.655 INFO:teuthology.orchestra.run.smithi136.stdout: "kstore_onode_map_size": "1024", 2024-08-17T17:52:59.655 INFO:teuthology.orchestra.run.smithi136.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-08-17T17:52:59.655 INFO:teuthology.orchestra.run.smithi136.stdout: "kstore_sync_submit_transaction": "false", 2024-08-17T17:52:59.655 INFO:teuthology.orchestra.run.smithi136.stdout: "kstore_sync_transaction": "false", 2024-08-17T17:52:59.655 INFO:teuthology.orchestra.run.smithi136.stdout: "librados_thread_count": "2", 2024-08-17T17:52:59.655 INFO:teuthology.orchestra.run.smithi136.stdout: "lockdep": "false", 2024-08-17T17:52:59.655 INFO:teuthology.orchestra.run.smithi136.stdout: "lockdep_force_backtrace": "false", 2024-08-17T17:52:59.655 INFO:teuthology.orchestra.run.smithi136.stdout: "log_coarse_timestamps": "true", 2024-08-17T17:52:59.655 INFO:teuthology.orchestra.run.smithi136.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-08-17T17:52:59.655 INFO:teuthology.orchestra.run.smithi136.stdout: "log_flush_on_exit": "false", 2024-08-17T17:52:59.655 INFO:teuthology.orchestra.run.smithi136.stdout: "log_graylog_host": "127.0.0.1", 2024-08-17T17:52:59.655 INFO:teuthology.orchestra.run.smithi136.stdout: "log_graylog_port": "12201", 2024-08-17T17:52:59.656 INFO:teuthology.orchestra.run.smithi136.stdout: "log_max_new": "1000", 2024-08-17T17:52:59.656 INFO:teuthology.orchestra.run.smithi136.stdout: "log_max_recent": "10000", 2024-08-17T17:52:59.656 INFO:teuthology.orchestra.run.smithi136.stdout: "log_stderr_prefix": "", 2024-08-17T17:52:59.656 INFO:teuthology.orchestra.run.smithi136.stdout: "log_stop_at_utilization": "0.970000", 2024-08-17T17:52:59.656 INFO:teuthology.orchestra.run.smithi136.stdout: "log_to_file": "true", 2024-08-17T17:52:59.656 INFO:teuthology.orchestra.run.smithi136.stdout: "log_to_graylog": "false", 2024-08-17T17:52:59.656 INFO:teuthology.orchestra.run.smithi136.stdout: "log_to_journald": "false", 2024-08-17T17:52:59.656 INFO:teuthology.orchestra.run.smithi136.stdout: "log_to_stderr": "false", 2024-08-17T17:52:59.656 INFO:teuthology.orchestra.run.smithi136.stdout: "log_to_syslog": "false", 2024-08-17T17:52:59.656 INFO:teuthology.orchestra.run.smithi136.stdout: "max_rotating_auth_attempts": "10", 2024-08-17T17:52:59.656 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-08-17T17:52:59.656 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_action_on_write_error": "1", 2024-08-17T17:52:59.656 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_alternate_name_max": "8192", 2024-08-17T17:52:59.656 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_asio_thread_count": "2", 2024-08-17T17:52:59.657 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_export_pin": "true", 2024-08-17T17:52:59.657 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_fragment_dirs": "true", 2024-08-17T17:52:59.657 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-08-17T17:52:59.657 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_fragment_interval": "5", 2024-08-17T17:52:59.657 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_fragment_size_max": "100000", 2024-08-17T17:52:59.657 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_idle_threshold": "0.000000", 2024-08-17T17:52:59.657 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_interval": "10", 2024-08-17T17:52:59.657 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_max": "-1", 2024-08-17T17:52:59.657 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_max_until": "-1", 2024-08-17T17:52:59.657 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_merge_size": "50", 2024-08-17T17:52:59.657 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_midchunk": "0.300000", 2024-08-17T17:52:59.657 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_min_rebalance": "0.100000", 2024-08-17T17:52:59.657 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_min_start": "0.200000", 2024-08-17T17:52:59.658 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_minchunk": "0.001000", 2024-08-17T17:52:59.658 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_mode": "0", 2024-08-17T17:52:59.658 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_need_max": "1.200000", 2024-08-17T17:52:59.658 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_need_min": "0.800000", 2024-08-17T17:52:59.658 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_overload_epochs": "2", 2024-08-17T17:52:59.658 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-08-17T17:52:59.658 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_sample_interval": "3.000000", 2024-08-17T17:52:59.658 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_split_bits": "3", 2024-08-17T17:52:59.658 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_split_rd": "25000.000000", 2024-08-17T17:52:59.658 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_split_size": "10000", 2024-08-17T17:52:59.658 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_split_wr": "10000.000000", 2024-08-17T17:52:59.658 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_target_decay": "10.000000", 2024-08-17T17:52:59.658 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-08-17T17:52:59.659 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_beacon_grace": "15.000000", 2024-08-17T17:52:59.659 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_beacon_interval": "4.000000", 2024-08-17T17:52:59.659 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_beacon_mon_down_grace": "60", 2024-08-17T17:52:59.659 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_cache_memory_limit": "4294967296", 2024-08-17T17:52:59.659 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_cache_mid": "0.700000", 2024-08-17T17:52:59.659 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-08-17T17:52:59.659 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_cache_quiesce_delay": "0", 2024-08-17T17:52:59.659 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_cache_quiesce_sleep": "200", 2024-08-17T17:52:59.659 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_cache_quiesce_splitauth": "true", 2024-08-17T17:52:59.659 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_cache_quiesce_threshold": "524288", 2024-08-17T17:52:59.659 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_cache_release_free_interval": "10", 2024-08-17T17:52:59.659 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_cache_reservation": "0.050000", 2024-08-17T17:52:59.660 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-08-17T17:52:59.660 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_cache_trim_interval": "1", 2024-08-17T17:52:59.660 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_cache_trim_threshold": "262144", 2024-08-17T17:52:59.660 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-08-17T17:52:59.660 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-08-17T17:52:59.660 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_client_delegate_inos_pct": "50", 2024-08-17T17:52:59.660 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_client_prealloc_inos": "1000", 2024-08-17T17:52:59.660 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-08-17T17:52:59.660 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_connect_bootstrapping": "false", 2024-08-17T17:52:59.660 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_damage_table_max_entries": "10000", 2024-08-17T17:52:59.660 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-08-17T17:52:59.660 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_debug_auth_pins": "false", 2024-08-17T17:52:59.660 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_debug_frag": "false", 2024-08-17T17:52:59.660 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_debug_scatterstat": "false", 2024-08-17T17:52:59.661 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_debug_subtrees": "false", 2024-08-17T17:52:59.661 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_decay_halflife": "5.000000", 2024-08-17T17:52:59.661 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_default_dir_hash": "2", 2024-08-17T17:52:59.661 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_defer_session_stale": "true", 2024-08-17T17:52:59.661 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_deny_all_reconnect": "false", 2024-08-17T17:52:59.661 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_dir_keys_per_op": "16384", 2024-08-17T17:52:59.661 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_dir_max_commit_size": "10", 2024-08-17T17:52:59.661 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_dir_max_entries": "0", 2024-08-17T17:52:59.661 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_dir_prefetch": "true", 2024-08-17T17:52:59.661 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_dirstat_min_interval": "1.000000", 2024-08-17T17:52:59.661 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_dump_cache_after_rejoin": "false", 2024-08-17T17:52:59.661 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_dump_cache_on_map": "false", 2024-08-17T17:52:59.661 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_dump_cache_threshold_file": "0", 2024-08-17T17:52:59.662 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-08-17T17:52:59.662 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_early_reply": "true", 2024-08-17T17:52:59.662 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_enable_op_tracker": "true", 2024-08-17T17:52:59.662 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_enforce_unique_name": "true", 2024-08-17T17:52:59.662 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_export_ephemeral_distributed": "true", 2024-08-17T17:52:59.662 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-08-17T17:52:59.662 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_export_ephemeral_random": "true", 2024-08-17T17:52:59.662 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-08-17T17:52:59.662 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-08-17T17:52:59.662 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_forward_all_requests_to_auth": "false", 2024-08-17T17:52:59.662 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-08-17T17:52:59.662 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-08-17T17:52:59.662 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-08-17T17:52:59.663 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-08-17T17:52:59.663 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_health_cache_threshold": "1.500000", 2024-08-17T17:52:59.663 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_health_summarize_threshold": "10", 2024-08-17T17:52:59.663 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_heartbeat_grace": "15.000000", 2024-08-17T17:52:59.663 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_heartbeat_reset_grace": "1000", 2024-08-17T17:52:59.663 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_inject_health_dummy": "false", 2024-08-17T17:52:59.663 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-08-17T17:52:59.663 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_inject_migrator_session_race": "false", 2024-08-17T17:52:59.663 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-08-17T17:52:59.663 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-08-17T17:52:59.663 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-08-17T17:52:59.663 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_join_fs": "", 2024-08-17T17:52:59.663 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_journal_format": "1", 2024-08-17T17:52:59.664 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-08-17T17:52:59.664 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_kill_create_at": "0", 2024-08-17T17:52:59.664 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_kill_dirfrag_at": "0", 2024-08-17T17:52:59.664 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_kill_export_at": "0", 2024-08-17T17:52:59.664 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_kill_import_at": "0", 2024-08-17T17:52:59.664 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_kill_journal_at": "0", 2024-08-17T17:52:59.664 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_kill_journal_expire_at": "0", 2024-08-17T17:52:59.664 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_kill_journal_replay_at": "0", 2024-08-17T17:52:59.664 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_kill_link_at": "0", 2024-08-17T17:52:59.664 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_kill_mdstable_at": "0", 2024-08-17T17:52:59.664 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_kill_openc_at": "0", 2024-08-17T17:52:59.664 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_kill_rename_at": "0", 2024-08-17T17:52:59.664 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_kill_shutdown_at": "0", 2024-08-17T17:52:59.664 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_log_event_large_threshold": "524288", 2024-08-17T17:52:59.665 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_log_events_per_segment": "1024", 2024-08-17T17:52:59.665 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_log_major_segment_event_ratio": "12", 2024-08-17T17:52:59.665 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_log_max_events": "-1", 2024-08-17T17:52:59.665 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_log_max_segments": "128", 2024-08-17T17:52:59.665 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_log_pause": "false", 2024-08-17T17:52:59.665 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_log_segment_size": "0", 2024-08-17T17:52:59.665 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_log_skip_corrupt_events": "false", 2024-08-17T17:52:59.665 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_log_skip_unbounded_events": "false", 2024-08-17T17:52:59.665 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-08-17T17:52:59.665 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_log_trim_threshold": "128", 2024-08-17T17:52:59.665 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-08-17T17:52:59.665 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_log_warn_factor": "2.000000", 2024-08-17T17:52:59.665 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_max_caps_per_client": "1048576", 2024-08-17T17:52:59.665 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_max_completed_flushes": "100000", 2024-08-17T17:52:59.666 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_max_completed_requests": "100000", 2024-08-17T17:52:59.666 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_max_export_size": "20971520", 2024-08-17T17:52:59.666 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_max_file_recover": "32", 2024-08-17T17:52:59.666 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_max_purge_files": "64", 2024-08-17T17:52:59.666 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_max_purge_ops": "8192", 2024-08-17T17:52:59.666 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-08-17T17:52:59.666 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-08-17T17:52:59.666 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_max_snaps_per_dir": "100", 2024-08-17T17:52:59.666 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_metrics_update_interval": "2", 2024-08-17T17:52:59.666 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_min_caps_per_client": "100", 2024-08-17T17:52:59.666 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_min_caps_working_set": "10000", 2024-08-17T17:52:59.666 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-08-17T17:52:59.666 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_numa_node": "-1", 2024-08-17T17:52:59.667 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-08-17T17:52:59.667 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_op_complaint_time": "30.000000", 2024-08-17T17:52:59.667 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_op_history_duration": "600", 2024-08-17T17:52:59.667 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_op_history_size": "20", 2024-08-17T17:52:59.667 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_op_history_slow_op_size": "20", 2024-08-17T17:52:59.667 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_op_history_slow_op_threshold": "10", 2024-08-17T17:52:59.667 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_op_log_threshold": "5", 2024-08-17T17:52:59.667 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_ping_grace": "15", 2024-08-17T17:52:59.667 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_ping_interval": "5", 2024-08-17T17:52:59.667 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-08-17T17:52:59.667 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-08-17T17:52:59.667 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_recall_max_caps": "30000", 2024-08-17T17:52:59.667 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-08-17T17:52:59.667 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_recall_max_decay_threshold": "131072", 2024-08-17T17:52:59.668 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-08-17T17:52:59.668 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_recall_warning_threshold": "262144", 2024-08-17T17:52:59.668 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_reconnect_timeout": "45.000000", 2024-08-17T17:52:59.668 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_replay_interval": "1.000000", 2024-08-17T17:52:59.668 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-08-17T17:52:59.668 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-08-17T17:52:59.668 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_root_ino_gid": "0", 2024-08-17T17:52:59.668 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_root_ino_uid": "0", 2024-08-17T17:52:59.668 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-08-17T17:52:59.668 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-08-17T17:52:59.668 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-08-17T17:52:59.668 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_session_blocklist_on_evict": "true", 2024-08-17T17:52:59.668 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_session_blocklist_on_timeout": "true", 2024-08-17T17:52:59.669 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-08-17T17:52:59.669 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-08-17T17:52:59.669 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-08-17T17:52:59.669 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-08-17T17:52:59.669 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-08-17T17:52:59.669 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_session_metadata_threshold": "16777216", 2024-08-17T17:52:59.669 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-08-17T17:52:59.669 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_shutdown_check": "0", 2024-08-17T17:52:59.669 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_skip_ino": "0", 2024-08-17T17:52:59.669 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_sleep_rank_change": "0.000000", 2024-08-17T17:52:59.669 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_snap_max_uid": "4294967294", 2024-08-17T17:52:59.669 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_snap_min_uid": "0", 2024-08-17T17:52:59.669 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_snap_rstat": "false", 2024-08-17T17:52:59.669 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_standby_replay_damaged": "false", 2024-08-17T17:52:59.670 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_symlink_recovery": "true", 2024-08-17T17:52:59.670 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_task_status_update_interval": "2.000000", 2024-08-17T17:52:59.670 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_thrash_exports": "0", 2024-08-17T17:52:59.670 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_thrash_fragments": "0", 2024-08-17T17:52:59.670 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_tick_interval": "5.000000", 2024-08-17T17:52:59.670 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_valgrind_exit": "false", 2024-08-17T17:52:59.670 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_verify_backtrace": "1", 2024-08-17T17:52:59.670 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_verify_scatter": "false", 2024-08-17T17:52:59.670 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_wipe_ino_prealloc": "false", 2024-08-17T17:52:59.670 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_wipe_sessions": "false", 2024-08-17T17:52:59.670 INFO:teuthology.orchestra.run.smithi136.stdout: "mempool_debug": "false", 2024-08-17T17:52:59.670 INFO:teuthology.orchestra.run.smithi136.stdout: "memstore_debug_omit_block_device_write": "false", 2024-08-17T17:52:59.670 INFO:teuthology.orchestra.run.smithi136.stdout: "memstore_device_bytes": "1073741824", 2024-08-17T17:52:59.671 INFO:teuthology.orchestra.run.smithi136.stdout: "memstore_page_set": "false", 2024-08-17T17:52:59.671 INFO:teuthology.orchestra.run.smithi136.stdout: "memstore_page_size": "65536", 2024-08-17T17:52:59.671 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_client_bytes": "134217728", 2024-08-17T17:52:59.671 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_client_messages": "512", 2024-08-17T17:52:59.671 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-08-17T17:52:59.671 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_connect_retry_interval": "1.000000", 2024-08-17T17:52:59.671 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-08-17T17:52:59.671 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-08-17T17:52:59.671 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_disabled_modules": "", 2024-08-17T17:52:59.671 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_enable_op_tracker": "true", 2024-08-17T17:52:59.671 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-08-17T17:52:59.671 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_max_pg_creating": "1024", 2024-08-17T17:52:59.672 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_max_pg_num_change": "128", 2024-08-17T17:52:59.672 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_mds_bytes": "134217728", 2024-08-17T17:52:59.672 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_mds_messages": "128", 2024-08-17T17:52:59.672 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-08-17T17:52:59.672 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_mon_bytes": "134217728", 2024-08-17T17:52:59.672 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_mon_messages": "128", 2024-08-17T17:52:59.672 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_num_op_tracker_shard": "32", 2024-08-17T17:52:59.672 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_op_complaint_time": "30.000000", 2024-08-17T17:52:59.672 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_op_history_duration": "600", 2024-08-17T17:52:59.672 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_op_history_size": "20", 2024-08-17T17:52:59.672 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_op_history_slow_op_size": "20", 2024-08-17T17:52:59.672 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-08-17T17:52:59.672 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_op_log_threshold": "5", 2024-08-17T17:52:59.672 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_osd_bytes": "536870912", 2024-08-17T17:52:59.673 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_osd_messages": "8192", 2024-08-17T17:52:59.673 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_pool": "true", 2024-08-17T17:52:59.673 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_service_beacon_grace": "60.000000", 2024-08-17T17:52:59.673 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_standby_modules": "true", 2024-08-17T17:52:59.673 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_stats_period": "5", 2024-08-17T17:52:59.673 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_stats_threshold": "5", 2024-08-17T17:52:59.673 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_tick_period": "2", 2024-08-17T17:52:59.673 INFO:teuthology.orchestra.run.smithi136.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-08-17T17:52:59.673 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_accept_timeout_factor": "2.000000", 2024-08-17T17:52:59.673 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_allow_pool_delete": "true", 2024-08-17T17:52:59.673 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_allow_pool_size_one": "false", 2024-08-17T17:52:59.673 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_auth_validate_all_caps": "true", 2024-08-17T17:52:59.673 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-08-17T17:52:59.673 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-08-17T17:52:59.673 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_client_bytes": "104857600", 2024-08-17T17:52:59.674 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_client_directed_command_retry": "2", 2024-08-17T17:52:59.674 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_client_hunt_interval": "3.000000", 2024-08-17T17:52:59.674 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-08-17T17:52:59.674 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-08-17T17:52:59.674 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-08-17T17:52:59.674 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_client_hunt_parallel": "3", 2024-08-17T17:52:59.674 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_client_log_interval": "1.000000", 2024-08-17T17:52:59.674 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-08-17T17:52:59.674 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_client_ping_interval": "10.000000", 2024-08-17T17:52:59.674 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_client_ping_timeout": "30.000000", 2024-08-17T17:52:59.674 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_client_target_rank": "-1", 2024-08-17T17:52:59.674 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_clock_drift_allowed": "1.000000", 2024-08-17T17:52:59.674 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-08-17T17:52:59.674 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-08-17T17:52:59.675 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_cluster_log_level": "debug", 2024-08-17T17:52:59.675 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_cluster_log_to_file": "true", 2024-08-17T17:52:59.675 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_cluster_log_to_graylog": "false", 2024-08-17T17:52:59.675 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-08-17T17:52:59.675 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-08-17T17:52:59.675 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_cluster_log_to_journald": "true", 2024-08-17T17:52:59.675 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_cluster_log_to_stderr": "false", 2024-08-17T17:52:59.675 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-08-17T17:52:59.675 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-08-17T17:52:59.675 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_compact_on_bootstrap": "false", 2024-08-17T17:52:59.675 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_compact_on_start": "false", 2024-08-17T17:52:59.675 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_compact_on_trim": "true", 2024-08-17T17:52:59.675 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_con_tracker_persist_interval": "10", 2024-08-17T17:52:59.675 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_con_tracker_score_halflife": "43200", 2024-08-17T17:52:59.676 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_config_key_max_entry_size": "65536", 2024-08-17T17:52:59.676 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_cpu_threads": "4", 2024-08-17T17:52:59.676 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_crush_min_required_version": "hammer", 2024-08-17T17:52:59.676 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_daemon_bytes": "419430400", 2024-08-17T17:52:59.676 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-08-17T17:52:59.676 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_data_avail_crit": "5", 2024-08-17T17:52:59.676 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_data_avail_warn": "5", 2024-08-17T17:52:59.676 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_data_size_warn": "16106127360", 2024-08-17T17:52:59.676 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_debug_block_osdmap_trim": "false", 2024-08-17T17:52:59.676 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-08-17T17:52:59.676 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_debug_dump_json": "false", 2024-08-17T17:52:59.676 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-08-17T17:52:59.676 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_debug_dump_transactions": "false", 2024-08-17T17:52:59.676 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_debug_extra_checks": "false", 2024-08-17T17:52:59.676 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-08-17T17:52:59.677 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-08-17T17:52:59.677 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_debug_no_require_reef": "false", 2024-08-17T17:52:59.677 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_debug_no_require_squid": "false", 2024-08-17T17:52:59.677 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-08-17T17:52:59.677 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_delta_reset_interval": "10.000000", 2024-08-17T17:52:59.677 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_dns_srv_name": "ceph-mon", 2024-08-17T17:52:59.677 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_down_mkfs_grace": "60", 2024-08-17T17:52:59.677 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_down_uptime_grace": "60", 2024-08-17T17:52:59.677 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_election_default_strategy": "1", 2024-08-17T17:52:59.677 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_election_timeout": "5.000000", 2024-08-17T17:52:59.677 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-08-17T17:52:59.677 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_elector_ping_divisor": "2", 2024-08-17T17:52:59.677 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_elector_ping_timeout": "2.000000", 2024-08-17T17:52:59.677 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_enable_op_tracker": "true", 2024-08-17T17:52:59.678 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_fake_pool_delete": "false", 2024-08-17T17:52:59.678 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_force_quorum_join": "false", 2024-08-17T17:52:59.678 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_fsmap_prune_threshold": "300", 2024-08-17T17:52:59.678 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_globalid_prealloc": "10000", 2024-08-17T17:52:59.678 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_health_detail_to_clog": "true", 2024-08-17T17:52:59.678 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_health_log_update_period": "5", 2024-08-17T17:52:59.678 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_health_max_detail": "50", 2024-08-17T17:52:59.678 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_health_to_clog": "true", 2024-08-17T17:52:59.678 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_health_to_clog_interval": "600", 2024-08-17T17:52:59.678 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-08-17T17:52:59.678 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_host": "[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0]", 2024-08-17T17:52:59.678 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_host_override": "", 2024-08-17T17:52:59.678 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_initial_members": "", 2024-08-17T17:52:59.679 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-08-17T17:52:59.679 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-08-17T17:52:59.679 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-08-17T17:52:59.679 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-08-17T17:52:59.679 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_keyvaluedb": "rocksdb", 2024-08-17T17:52:59.679 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_lease": "5.000000", 2024-08-17T17:52:59.679 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-08-17T17:52:59.679 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-08-17T17:52:59.679 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_log_full_interval": "50", 2024-08-17T17:52:59.679 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_log_max": "10000", 2024-08-17T17:52:59.679 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_log_max_summary": "50", 2024-08-17T17:52:59.679 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_max_log_entries_per_event": "4096", 2024-08-17T17:52:59.679 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_max_log_epochs": "500", 2024-08-17T17:52:59.680 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_max_mdsmap_epochs": "500", 2024-08-17T17:52:59.680 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_max_mgrmap_epochs": "500", 2024-08-17T17:52:59.680 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_max_nvmeof_epochs": "500", 2024-08-17T17:52:59.680 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_max_osd": "10000", 2024-08-17T17:52:59.680 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_max_pg_per_osd": "10000", 2024-08-17T17:52:59.680 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_max_pool_pg_num": "65536", 2024-08-17T17:52:59.680 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-08-17T17:52:59.680 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-08-17T17:52:59.680 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_mds_force_trim_to": "0", 2024-08-17T17:52:59.680 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_mds_skip_sanity": "true", 2024-08-17T17:52:59.680 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_memory_autotune": "true", 2024-08-17T17:52:59.680 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_memory_target": "2147483648", 2024-08-17T17:52:59.680 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_mgr_beacon_grace": "30", 2024-08-17T17:52:59.680 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-08-17T17:52:59.681 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_mgr_digest_period": "5", 2024-08-17T17:52:59.681 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_mgr_inactive_grace": "60", 2024-08-17T17:52:59.681 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_mgr_mkfs_grace": "240", 2024-08-17T17:52:59.681 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-08-17T17:52:59.681 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_min_osdmap_epochs": "500", 2024-08-17T17:52:59.681 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-08-17T17:52:59.681 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-08-17T17:52:59.681 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_op_complaint_time": "30", 2024-08-17T17:52:59.681 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_op_history_duration": "600", 2024-08-17T17:52:59.681 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_op_history_size": "20", 2024-08-17T17:52:59.681 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_op_history_slow_op_size": "20", 2024-08-17T17:52:59.681 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_op_history_slow_op_threshold": "10", 2024-08-17T17:52:59.681 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_op_log_threshold": "5", 2024-08-17T17:52:59.681 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-08-17T17:52:59.682 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-08-17T17:52:59.682 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-08-17T17:52:59.682 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_auto_mark_in": "false", 2024-08-17T17:52:59.682 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_auto_mark_new_in": "true", 2024-08-17T17:52:59.682 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-08-17T17:52:59.682 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-08-17T17:52:59.682 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_cache_size": "500", 2024-08-17T17:52:59.682 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_cache_size_min": "134217728", 2024-08-17T17:52:59.682 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_crush_smoke_test": "true", 2024-08-17T17:52:59.682 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_destroyed_out_interval": "600", 2024-08-17T17:52:59.682 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_down_out_interval": "600", 2024-08-17T17:52:59.682 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-08-17T17:52:59.682 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-08-17T17:52:59.682 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_force_trim_to": "0", 2024-08-17T17:52:59.682 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_full_ratio": "0.950000", 2024-08-17T17:52:59.683 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-08-17T17:52:59.683 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_laggy_halflife": "3600", 2024-08-17T17:52:59.683 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_laggy_max_interval": "300", 2024-08-17T17:52:59.683 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_laggy_weight": "0.300000", 2024-08-17T17:52:59.683 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-08-17T17:52:59.683 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_max_initial_pgs": "1024", 2024-08-17T17:52:59.683 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_min_down_reporters": "2", 2024-08-17T17:52:59.683 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-08-17T17:52:59.683 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-08-17T17:52:59.683 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-08-17T17:52:59.683 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_prime_pg_temp": "true", 2024-08-17T17:52:59.683 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-08-17T17:52:59.683 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-08-17T17:52:59.683 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_report_timeout": "900", 2024-08-17T17:52:59.683 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-08-17T17:52:59.685 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-08-17T17:52:59.685 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_warn_num_repaired": "10", 2024-08-17T17:52:59.685 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osd_warn_op_age": "32.000000", 2024-08-17T17:52:59.685 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-08-17T17:52:59.685 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osdmap_full_prune_interval": "10", 2024-08-17T17:52:59.685 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osdmap_full_prune_min": "10000", 2024-08-17T17:52:59.685 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-08-17T17:52:59.685 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-08-17T17:52:59.685 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_pg_stuck_threshold": "60", 2024-08-17T17:52:59.685 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-08-17T17:52:59.686 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_pg_warn_min_objects": "10000", 2024-08-17T17:52:59.686 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_pg_warn_min_per_osd": "0", 2024-08-17T17:52:59.686 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-08-17T17:52:59.686 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_pool_quota_crit_threshold": "0", 2024-08-17T17:52:59.686 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_pool_quota_warn_threshold": "0", 2024-08-17T17:52:59.686 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_probe_timeout": "2.000000", 2024-08-17T17:52:59.686 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_reweight_max_change": "0.050000", 2024-08-17T17:52:59.686 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_reweight_max_osds": "4", 2024-08-17T17:52:59.686 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-08-17T17:52:59.686 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-08-17T17:52:59.686 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-08-17T17:52:59.686 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-08-17T17:52:59.686 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-08-17T17:52:59.687 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_scrub_interval": "86400", 2024-08-17T17:52:59.687 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_scrub_max_keys": "100", 2024-08-17T17:52:59.687 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_scrub_timeout": "300", 2024-08-17T17:52:59.687 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_session_timeout": "300", 2024-08-17T17:52:59.687 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_smart_report_timeout": "5", 2024-08-17T17:52:59.687 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_stat_smooth_intervals": "6", 2024-08-17T17:52:59.687 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-08-17T17:52:59.687 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_stretch_pool_min_size": "2", 2024-08-17T17:52:59.687 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_stretch_pool_size": "4", 2024-08-17T17:52:59.687 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-08-17T17:52:59.687 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_subscribe_interval": "86400.000000", 2024-08-17T17:52:59.687 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_sync_debug": "false", 2024-08-17T17:52:59.687 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_sync_max_payload_keys": "2000", 2024-08-17T17:52:59.687 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_sync_max_payload_size": "1048576", 2024-08-17T17:52:59.687 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_sync_provider_kill_at": "0", 2024-08-17T17:52:59.687 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_sync_requester_kill_at": "0", 2024-08-17T17:52:59.688 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_sync_timeout": "60.000000", 2024-08-17T17:52:59.688 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_target_pg_per_osd": "100", 2024-08-17T17:52:59.688 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_tick_interval": "5", 2024-08-17T17:52:59.688 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_timecheck_interval": "300.000000", 2024-08-17T17:52:59.688 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-08-17T17:52:59.688 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_use_min_delay_socket": "false", 2024-08-17T17:52:59.688 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_warn_older_version_delay": "604800", 2024-08-17T17:52:59.688 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-08-17T17:52:59.688 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-08-17T17:52:59.688 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-08-17T17:52:59.688 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_warn_on_filestore_osds": "true", 2024-08-17T17:52:59.688 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-08-17T17:52:59.689 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-08-17T17:52:59.689 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-08-17T17:52:59.689 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_warn_on_misplaced": "false", 2024-08-17T17:52:59.689 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-08-17T17:52:59.689 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_warn_on_older_version": "true", 2024-08-17T17:52:59.689 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-08-17T17:52:59.689 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_warn_on_pool_no_app": "true", 2024-08-17T17:52:59.689 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-08-17T17:52:59.689 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-08-17T17:52:59.689 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-08-17T17:52:59.689 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-08-17T17:52:59.689 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_warn_on_too_few_osds": "false", 2024-08-17T17:52:59.689 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-08-17T17:52:59.690 INFO:teuthology.orchestra.run.smithi136.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-08-17T17:52:59.690 INFO:teuthology.orchestra.run.smithi136.stdout: "monmap": "", 2024-08-17T17:52:59.690 INFO:teuthology.orchestra.run.smithi136.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-08-17T17:52:59.690 INFO:teuthology.orchestra.run.smithi136.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-08-17T17:52:59.690 INFO:teuthology.orchestra.run.smithi136.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-08-17T17:52:59.690 INFO:teuthology.orchestra.run.smithi136.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-08-17T17:52:59.690 INFO:teuthology.orchestra.run.smithi136.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-08-17T17:52:59.690 INFO:teuthology.orchestra.run.smithi136.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-08-17T17:52:59.690 INFO:teuthology.orchestra.run.smithi136.stdout: "motr_tracing_enabled": "false", 2024-08-17T17:52:59.690 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_async_op_threads": "3", 2024-08-17T17:52:59.690 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_async_rdma_buffer_size": "131072", 2024-08-17T17:52:59.690 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_async_rdma_cm": "false", 2024-08-17T17:52:59.690 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_async_rdma_device_name": "", 2024-08-17T17:52:59.690 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_async_rdma_dscp": "96", 2024-08-17T17:52:59.691 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-08-17T17:52:59.691 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_async_rdma_gid_idx": "0", 2024-08-17T17:52:59.691 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_async_rdma_local_gid": "", 2024-08-17T17:52:59.691 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_async_rdma_polling_us": "1000", 2024-08-17T17:52:59.691 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_async_rdma_port_num": "1", 2024-08-17T17:52:59.691 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-08-17T17:52:59.691 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-08-17T17:52:59.691 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_async_rdma_roce_ver": "1", 2024-08-17T17:52:59.691 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_async_rdma_send_buffers": "1024", 2024-08-17T17:52:59.691 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_async_rdma_sl": "3", 2024-08-17T17:52:59.691 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_async_rdma_support_srq": "true", 2024-08-17T17:52:59.691 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_async_rdma_type": "ib", 2024-08-17T17:52:59.691 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_async_reap_threshold": "5", 2024-08-17T17:52:59.691 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_bind_before_connect": "false", 2024-08-17T17:52:59.691 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_bind_ipv4": "true", 2024-08-17T17:52:59.691 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_bind_ipv6": "false", 2024-08-17T17:52:59.692 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_bind_msgr1": "true", 2024-08-17T17:52:59.692 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_bind_msgr2": "true", 2024-08-17T17:52:59.692 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_bind_port_max": "7568", 2024-08-17T17:52:59.692 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_bind_port_min": "6800", 2024-08-17T17:52:59.692 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_bind_prefer_ipv4": "false", 2024-08-17T17:52:59.692 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_bind_retry_count": "3", 2024-08-17T17:52:59.692 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_bind_retry_delay": "5", 2024-08-17T17:52:59.692 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_blackhole_client": "false", 2024-08-17T17:52:59.692 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_blackhole_mds": "false", 2024-08-17T17:52:59.692 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_blackhole_mgr": "false", 2024-08-17T17:52:59.692 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_blackhole_mon": "false", 2024-08-17T17:52:59.692 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_blackhole_osd": "false", 2024-08-17T17:52:59.692 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_client_mode": "crc secure", 2024-08-17T17:52:59.692 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-08-17T17:52:59.692 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_cluster_mode": "crc secure", 2024-08-17T17:52:59.693 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_cluster_type": "", 2024-08-17T17:52:59.693 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_compress_secure": "false", 2024-08-17T17:52:59.693 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_connection_idle_timeout": "900", 2024-08-17T17:52:59.693 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_connection_ready_timeout": "10", 2024-08-17T17:52:59.693 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_crc_data": "true", 2024-08-17T17:52:59.693 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_crc_header": "true", 2024-08-17T17:52:59.693 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_die_on_bad_msg": "false", 2024-08-17T17:52:59.693 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_die_on_bug": "true", 2024-08-17T17:52:59.693 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_die_on_old_message": "true", 2024-08-17T17:52:59.693 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_die_on_skipped_message": "false", 2024-08-17T17:52:59.693 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_die_on_unhandled_msg": "false", 2024-08-17T17:52:59.693 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-08-17T17:52:59.693 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_dpdk_coremask": "0xF", 2024-08-17T17:52:59.694 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-08-17T17:52:59.694 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_dpdk_devs_allowlist": "", 2024-08-17T17:52:59.694 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_dpdk_enable_tso": "true", 2024-08-17T17:52:59.694 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-08-17T17:52:59.694 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-08-17T17:52:59.694 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_dpdk_hugepages": "", 2024-08-17T17:52:59.694 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_dpdk_hw_flow_control": "true", 2024-08-17T17:52:59.694 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-08-17T17:52:59.694 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_dpdk_lro": "true", 2024-08-17T17:52:59.694 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_dpdk_memory_channel": "4", 2024-08-17T17:52:59.694 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-08-17T17:52:59.694 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_dpdk_pmd": "", 2024-08-17T17:52:59.694 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_dpdk_port_id": "0", 2024-08-17T17:52:59.694 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-08-17T17:52:59.695 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_dump_corrupt_message_level": "1", 2024-08-17T17:52:59.695 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_dump_on_send": "false", 2024-08-17T17:52:59.695 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_initial_backoff": "0.200000", 2024-08-17T17:52:59.695 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_inject_delay_max": "1.000000", 2024-08-17T17:52:59.695 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_inject_delay_probability": "0.000000", 2024-08-17T17:52:59.695 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_inject_delay_type": "", 2024-08-17T17:52:59.695 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_inject_internal_delays": "0.000000", 2024-08-17T17:52:59.695 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_inject_network_congestion": "0", 2024-08-17T17:52:59.695 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_inject_socket_failures": "0", 2024-08-17T17:52:59.695 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_learn_addr_from_peer": "true", 2024-08-17T17:52:59.695 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_max_accept_failures": "4", 2024-08-17T17:52:59.695 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_max_backoff": "15.000000", 2024-08-17T17:52:59.695 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_mon_client_mode": "secure crc", 2024-08-17T17:52:59.695 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_mon_cluster_mode": "secure crc", 2024-08-17T17:52:59.695 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_mon_service_mode": "secure crc", 2024-08-17T17:52:59.696 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_osd_compress_min_size": "1024", 2024-08-17T17:52:59.696 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_osd_compress_mode": "none", 2024-08-17T17:52:59.696 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_osd_compression_algorithm": "snappy", 2024-08-17T17:52:59.696 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-08-17T17:52:59.696 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_pq_min_cost": "65536", 2024-08-17T17:52:59.696 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_public_type": "", 2024-08-17T17:52:59.696 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_service_mode": "crc secure", 2024-08-17T17:52:59.696 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_tcp_listen_backlog": "512", 2024-08-17T17:52:59.696 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_tcp_nodelay": "true", 2024-08-17T17:52:59.696 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-08-17T17:52:59.696 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_tcp_rcvbuf": "0", 2024-08-17T17:52:59.696 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_time_events_min_wait_interval": "1000", 2024-08-17T17:52:59.696 INFO:teuthology.orchestra.run.smithi136.stdout: "ms_type": "async+posix", 2024-08-17T17:52:59.696 INFO:teuthology.orchestra.run.smithi136.stdout: "no_config_file": "false", 2024-08-17T17:52:59.697 INFO:teuthology.orchestra.run.smithi136.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-08-17T17:52:59.697 INFO:teuthology.orchestra.run.smithi136.stdout: "nvmeof_mon_client_tick_period": "2", 2024-08-17T17:52:59.697 INFO:teuthology.orchestra.run.smithi136.stdout: "objecter_completion_locks_per_session": "32", 2024-08-17T17:52:59.697 INFO:teuthology.orchestra.run.smithi136.stdout: "objecter_debug_inject_relock_delay": "false", 2024-08-17T17:52:59.697 INFO:teuthology.orchestra.run.smithi136.stdout: "objecter_inflight_op_bytes": "104857600", 2024-08-17T17:52:59.697 INFO:teuthology.orchestra.run.smithi136.stdout: "objecter_inflight_ops": "1024", 2024-08-17T17:52:59.697 INFO:teuthology.orchestra.run.smithi136.stdout: "objecter_inject_no_watch_ping": "false", 2024-08-17T17:52:59.697 INFO:teuthology.orchestra.run.smithi136.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-08-17T17:52:59.697 INFO:teuthology.orchestra.run.smithi136.stdout: "objecter_tick_interval": "5.000000", 2024-08-17T17:52:59.697 INFO:teuthology.orchestra.run.smithi136.stdout: "objecter_timeout": "10.000000", 2024-08-17T17:52:59.697 INFO:teuthology.orchestra.run.smithi136.stdout: "objectstore_blackhole": "false", 2024-08-17T17:52:59.697 INFO:teuthology.orchestra.run.smithi136.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-08-17T17:52:59.697 INFO:teuthology.orchestra.run.smithi136.stdout: "openssl_engine_opts": "", 2024-08-17T17:52:59.698 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_agent_delay_time": "5.000000", 2024-08-17T17:52:59.698 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_agent_hist_halflife": "1000", 2024-08-17T17:52:59.698 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_agent_max_low_ops": "2", 2024-08-17T17:52:59.698 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_agent_max_ops": "4", 2024-08-17T17:52:59.698 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-08-17T17:52:59.698 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_agent_quantize_effort": "0.100000", 2024-08-17T17:52:59.698 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_agent_slop": "0.020000", 2024-08-17T17:52:59.698 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-08-17T17:52:59.698 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_allow_recovery_below_min_size": "true", 2024-08-17T17:52:59.698 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_asio_thread_count": "2", 2024-08-17T17:52:59.698 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_async_recovery_min_cost": "100", 2024-08-17T17:52:59.698 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_auto_mark_unfound_lost": "false", 2024-08-17T17:52:59.698 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_backfill_retry_interval": "30.000000", 2024-08-17T17:52:59.699 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_backfill_scan_max": "512", 2024-08-17T17:52:59.699 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_backfill_scan_min": "64", 2024-08-17T17:52:59.699 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_backoff_on_degraded": "false", 2024-08-17T17:52:59.699 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_backoff_on_peering": "false", 2024-08-17T17:52:59.699 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_backoff_on_unfound": "true", 2024-08-17T17:52:59.699 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_beacon_report_interval": "300", 2024-08-17T17:52:59.699 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_bench_duration": "30", 2024-08-17T17:52:59.699 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-08-17T17:52:59.699 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_bench_max_block_size": "67108864", 2024-08-17T17:52:59.699 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_bench_small_size_max_iops": "100", 2024-08-17T17:52:59.699 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_blkin_trace_all": "false", 2024-08-17T17:52:59.699 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_blocked_scrub_grace_period": "120", 2024-08-17T17:52:59.699 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-08-17T17:52:59.699 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-08-17T17:52:59.699 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-08-17T17:52:59.700 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_check_for_log_corruption": "false", 2024-08-17T17:52:59.700 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-08-17T17:52:59.700 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-08-17T17:52:59.700 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-08-17T17:52:59.700 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-08-17T17:52:59.700 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_class_update_on_start": "true", 2024-08-17T17:52:59.700 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_client_message_cap": "256", 2024-08-17T17:52:59.700 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_client_message_size_cap": "524288000", 2024-08-17T17:52:59.700 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_client_op_priority": "63", 2024-08-17T17:52:59.700 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_client_watch_timeout": "30", 2024-08-17T17:52:59.700 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_command_max_records": "256", 2024-08-17T17:52:59.700 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_compact_on_start": "false", 2024-08-17T17:52:59.700 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-08-17T17:52:59.700 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_crush_chooseleaf_type": "1", 2024-08-17T17:52:59.700 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_crush_initial_weight": "-1.000000", 2024-08-17T17:52:59.700 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_crush_update_on_start": "true", 2024-08-17T17:52:59.701 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_crush_update_weight_set": "true", 2024-08-17T17:52:59.701 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-08-17T17:52:59.701 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-08-17T17:52:59.701 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-08-17T17:52:59.701 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_debug_drop_ping_duration": "0", 2024-08-17T17:52:59.701 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-08-17T17:52:59.701 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_debug_feed_pullee": "-1", 2024-08-17T17:52:59.701 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-08-17T17:52:59.701 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-08-17T17:52:59.701 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-08-17T17:52:59.701 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_debug_misdirected_ops": "false", 2024-08-17T17:52:59.701 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_debug_no_acting_change": "false", 2024-08-17T17:52:59.701 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_debug_no_purge_strays": "false", 2024-08-17T17:52:59.701 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_debug_op_order": "false", 2024-08-17T17:52:59.702 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_debug_pg_log_writeout": "false", 2024-08-17T17:52:59.702 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_debug_pretend_recovery_active": "false", 2024-08-17T17:52:59.702 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-08-17T17:52:59.702 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-08-17T17:52:59.702 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_debug_shutdown": "false", 2024-08-17T17:52:59.702 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-08-17T17:52:59.702 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-08-17T17:52:59.702 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_debug_trim_objects": "false", 2024-08-17T17:52:59.702 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_debug_verify_cached_snaps": "false", 2024-08-17T17:52:59.702 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_debug_verify_missing_on_start": "false", 2024-08-17T17:52:59.702 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_debug_verify_snaps": "false", 2024-08-17T17:52:59.702 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-08-17T17:52:59.702 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-08-17T17:52:59.703 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_deep_scrub_keys": "1024", 2024-08-17T17:52:59.703 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-08-17T17:52:59.703 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-08-17T17:52:59.703 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-08-17T17:52:59.703 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_deep_scrub_stride": "524288", 2024-08-17T17:52:59.703 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-08-17T17:52:59.703 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_default_data_pool_replay_window": "45", 2024-08-17T17:52:59.703 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_default_notify_timeout": "30", 2024-08-17T17:52:59.703 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_delete_sleep": "0.000000", 2024-08-17T17:52:59.703 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-08-17T17:52:59.703 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-08-17T17:52:59.703 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-08-17T17:52:59.704 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_discard_disconnected_ops": "true", 2024-08-17T17:52:59.704 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_ec_partial_reads": "true", 2024-08-17T17:52:59.704 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_enable_op_tracker": "true", 2024-08-17T17:52:59.704 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-08-17T17:52:59.704 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_extblkdev_plugins": "vdo", 2024-08-17T17:52:59.704 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-08-17T17:52:59.704 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-08-17T17:52:59.704 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_fast_info": "true", 2024-08-17T17:52:59.704 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_fast_shutdown": "true", 2024-08-17T17:52:59.704 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-08-17T17:52:59.704 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_fast_shutdown_timeout": "15", 2024-08-17T17:52:59.704 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-08-17T17:52:59.704 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-08-17T17:52:59.704 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-08-17T17:52:59.704 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_function_tracing": "false", 2024-08-17T17:52:59.705 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_heartbeat_grace": "20", 2024-08-17T17:52:59.705 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_heartbeat_interval": "6", 2024-08-17T17:52:59.705 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-08-17T17:52:59.705 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_heartbeat_min_peers": "10", 2024-08-17T17:52:59.705 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_heartbeat_min_size": "2000", 2024-08-17T17:52:59.705 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_heartbeat_stale": "600", 2024-08-17T17:52:59.705 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-08-17T17:52:59.705 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_hit_set_max_size": "100000", 2024-08-17T17:52:59.705 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_hit_set_min_size": "1000", 2024-08-17T17:52:59.705 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-08-17T17:52:59.705 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-08-17T17:52:59.705 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-08-17T17:52:59.705 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-08-17T17:52:59.705 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-08-17T17:52:59.706 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_journal_flush_on_shutdown": "true", 2024-08-17T17:52:59.706 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_journal_size": "5120", 2024-08-17T17:52:59.706 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_kill_backfill_at": "0", 2024-08-17T17:52:59.706 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_loop_before_reset_tphandle": "64", 2024-08-17T17:52:59.706 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_map_cache_size": "50", 2024-08-17T17:52:59.706 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_map_dedup": "true", 2024-08-17T17:52:59.706 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_map_message_max": "40", 2024-08-17T17:52:59.706 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_map_message_max_bytes": "10485760", 2024-08-17T17:52:59.706 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_map_share_max_epochs": "40", 2024-08-17T17:52:59.706 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_max_attr_name_len": "100", 2024-08-17T17:52:59.706 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_max_attr_size": "0", 2024-08-17T17:52:59.706 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_max_backfills": "1", 2024-08-17T17:52:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_max_markdown_count": "5", 2024-08-17T17:52:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_max_markdown_period": "600", 2024-08-17T17:52:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_max_object_name_len": "2048", 2024-08-17T17:52:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_max_object_namespace_len": "256", 2024-08-17T17:52:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_max_object_size": "134217728", 2024-08-17T17:52:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-08-17T17:52:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_max_omap_entries_per_request": "1024", 2024-08-17T17:52:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_max_pg_blocked_by": "16", 2024-08-17T17:52:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_max_pg_log_entries": "10000", 2024-08-17T17:52:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-08-17T17:52:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_max_pgls": "1024", 2024-08-17T17:52:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_max_push_cost": "8388608", 2024-08-17T17:52:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_max_push_objects": "10", 2024-08-17T17:52:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_max_scrubs": "3", 2024-08-17T17:52:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-08-17T17:52:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_max_trimming_pgs": "2", 2024-08-17T17:52:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_max_write_op_reply_len": "64", 2024-08-17T17:52:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_max_write_size": "90", 2024-08-17T17:52:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-08-17T17:52:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-08-17T17:52:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-08-17T17:52:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-08-17T17:52:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-08-17T17:52:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-08-17T17:52:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-08-17T17:52:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mclock_override_recovery_settings": "false", 2024-08-17T17:52:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mclock_profile": "balanced", 2024-08-17T17:52:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-08-17T17:52:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-08-17T17:52:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-08-17T17:52:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-08-17T17:52:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-08-17T17:52:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-08-17T17:52:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-08-17T17:52:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-08-17T17:52:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-08-17T17:52:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-08-17T17:52:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mclock_skip_benchmark": "false", 2024-08-17T17:52:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_memory_base": "805306368", 2024-08-17T17:52:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_memory_cache_min": "134217728", 2024-08-17T17:52:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-08-17T17:52:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-08-17T17:52:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_memory_target": "4294967296", 2024-08-17T17:52:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_memory_target_autotune": "false", 2024-08-17T17:52:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-08-17T17:52:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_min_pg_log_entries": "250", 2024-08-17T17:52:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_min_recovery_priority": "0", 2024-08-17T17:52:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mon_heartbeat_interval": "30", 2024-08-17T17:52:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-08-17T17:52:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mon_report_interval": "5", 2024-08-17T17:52:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mon_report_max_in_flight": "2", 2024-08-17T17:52:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-08-17T17:52:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_num_cache_shards": "32", 2024-08-17T17:52:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_num_op_tracker_shard": "32", 2024-08-17T17:52:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_numa_auto_affinity": "true", 2024-08-17T17:52:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_numa_node": "-1", 2024-08-17T17:52:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_numa_prefer_iface": "true", 2024-08-17T17:52:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-08-17T17:52:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_objecter_finishers": "1", 2024-08-17T17:52:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_objectstore": "bluestore", 2024-08-17T17:52:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_objectstore_fuse": "false", 2024-08-17T17:52:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_objectstore_ideal_list_max": "64", 2024-08-17T17:52:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_objectstore_tracing": "false", 2024-08-17T17:52:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_op_complaint_time": "30.000000", 2024-08-17T17:52:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_op_history_duration": "600", 2024-08-17T17:52:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_op_history_size": "20", 2024-08-17T17:52:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_op_history_slow_op_size": "20", 2024-08-17T17:52:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-08-17T17:52:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_op_log_threshold": "5", 2024-08-17T17:52:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_op_num_shards": "0", 2024-08-17T17:52:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_op_num_shards_hdd": "5", 2024-08-17T17:52:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_op_num_shards_ssd": "8", 2024-08-17T17:52:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_op_num_threads_per_shard": "0", 2024-08-17T17:52:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-08-17T17:52:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-08-17T17:52:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-08-17T17:52:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_op_pq_min_cost": "65536", 2024-08-17T17:52:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_op_queue": "mclock_scheduler", 2024-08-17T17:52:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_op_queue_cut_off": "high", 2024-08-17T17:52:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_op_thread_suicide_timeout": "150", 2024-08-17T17:52:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_op_thread_timeout": "15", 2024-08-17T17:52:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_open_classes_on_start": "true", 2024-08-17T17:52:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_os_flags": "0", 2024-08-17T17:52:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_peering_op_priority": "255", 2024-08-17T17:52:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pg_delete_cost": "1048576", 2024-08-17T17:52:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pg_delete_priority": "5", 2024-08-17T17:52:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-08-17T17:52:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-08-17T17:52:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pg_log_dups_tracked": "3000", 2024-08-17T17:52:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pg_log_trim_max": "10000", 2024-08-17T17:52:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pg_log_trim_min": "100", 2024-08-17T17:52:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-08-17T17:52:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pg_object_context_cache_count": "64", 2024-08-17T17:52:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-08-17T17:52:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-08-17T17:52:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-08-17T17:52:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-08-17T17:52:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-08-17T17:52:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-08-17T17:52:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-08-17T17:52:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-08-17T17:52:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_crimson": "false", 2024-08-17T17:52:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_crush_rule": "-1", 2024-08-17T17:52:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_ec_fast_read": "false", 2024-08-17T17:52:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-08-17T17:52:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_flag_bulk": "false", 2024-08-17T17:52:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-08-17T17:52:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_flag_nodelete": "false", 2024-08-17T17:52:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-08-17T17:52:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-08-17T17:52:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_flags": "0", 2024-08-17T17:52:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-08-17T17:52:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_min_size": "0", 2024-08-17T17:52:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-08-17T17:52:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_pg_num": "32", 2024-08-17T17:52:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_pgp_num": "0", 2024-08-17T17:52:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-08-17T17:52:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_read_ratio": "70", 2024-08-17T17:52:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_size": "3", 2024-08-17T17:52:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_default_type": "replicated", 2024-08-17T17:52:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-08-17T17:52:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pool_use_gmt_hitset": "true", 2024-08-17T17:52:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_push_per_object_cost": "1000", 2024-08-17T17:52:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_read_ec_check_for_errors": "false", 2024-08-17T17:52:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_recover_clone_overlap": "true", 2024-08-17T17:52:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_recover_clone_overlap_limit": "10", 2024-08-17T17:52:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_recovery_cost": "20971520", 2024-08-17T17:52:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_recovery_delay_start": "0.000000", 2024-08-17T17:52:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_recovery_max_active": "0", 2024-08-17T17:52:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_recovery_max_active_hdd": "3", 2024-08-17T17:52:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_recovery_max_active_ssd": "10", 2024-08-17T17:52:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_recovery_max_chunk": "8388608", 2024-08-17T17:52:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-08-17T17:52:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_recovery_max_single_start": "1", 2024-08-17T17:52:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_recovery_op_priority": "3", 2024-08-17T17:52:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_recovery_op_warn_multiple": "16", 2024-08-17T17:52:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_recovery_priority": "5", 2024-08-17T17:52:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_recovery_retry_interval": "30.000000", 2024-08-17T17:52:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_recovery_sleep": "0.000000", 2024-08-17T17:52:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-08-17T17:52:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-08-17T17:52:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-08-17T17:52:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_repair_during_recovery": "false", 2024-08-17T17:52:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_requested_scrub_priority": "120", 2024-08-17T17:52:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-08-17T17:52:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_rollback_to_cluster_snap": "", 2024-08-17T17:52:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_scrub_auto_repair": "false", 2024-08-17T17:52:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-08-17T17:52:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-08-17T17:52:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_scrub_begin_hour": "0", 2024-08-17T17:52:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_scrub_begin_week_day": "0", 2024-08-17T17:52:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_scrub_chunk_max": "25", 2024-08-17T17:52:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_scrub_chunk_min": "5", 2024-08-17T17:52:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_scrub_cost": "52428800", 2024-08-17T17:52:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-08-17T17:52:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_scrub_during_recovery": "false", 2024-08-17T17:52:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_scrub_end_hour": "0", 2024-08-17T17:52:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_scrub_end_week_day": "0", 2024-08-17T17:52:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_scrub_event_cost": "4096", 2024-08-17T17:52:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-08-17T17:52:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-08-17T17:52:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_scrub_invalid_stats": "true", 2024-08-17T17:52:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_scrub_load_threshold": "0.500000", 2024-08-17T17:52:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_scrub_max_interval": "604800.000000", 2024-08-17T17:52:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_scrub_max_preemptions": "5", 2024-08-17T17:52:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_scrub_min_interval": "86400.000000", 2024-08-17T17:52:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_scrub_priority": "5", 2024-08-17T17:52:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_scrub_sleep": "0.000000", 2024-08-17T17:52:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-08-17T17:52:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-08-17T17:52:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_shutdown_pgref_assert": "false", 2024-08-17T17:52:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-08-17T17:52:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_skip_data_digest": "false", 2024-08-17T17:52:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_smart_report_timeout": "5", 2024-08-17T17:52:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_snap_trim_cost": "1048576", 2024-08-17T17:52:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_snap_trim_priority": "5", 2024-08-17T17:52:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_snap_trim_sleep": "0.000000", 2024-08-17T17:52:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-08-17T17:52:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-08-17T17:52:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-08-17T17:52:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-08-17T17:52:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_stats_update_period_scrubbing": "15", 2024-08-17T17:52:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-08-17T17:52:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_target_transaction_size": "30", 2024-08-17T17:52:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-08-17T17:52:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-08-17T17:52:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-08-17T17:52:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-08-17T17:52:59.720 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-08-17T17:52:59.720 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-08-17T17:52:59.720 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-08-17T17:52:59.720 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_tier_default_cache_mode": "writeback", 2024-08-17T17:52:59.720 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-08-17T17:52:59.720 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-08-17T17:52:59.720 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_tracing": "false", 2024-08-17T17:52:59.720 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_use_stale_snap": "false", 2024-08-17T17:52:59.720 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-08-17T17:52:59.720 INFO:teuthology.orchestra.run.smithi136.stdout: "osdc_blkin_trace_all": "false", 2024-08-17T17:52:59.720 INFO:teuthology.orchestra.run.smithi136.stdout: "paxos_kill_at": "0", 2024-08-17T17:52:59.720 INFO:teuthology.orchestra.run.smithi136.stdout: "paxos_max_join_drift": "10", 2024-08-17T17:52:59.720 INFO:teuthology.orchestra.run.smithi136.stdout: "paxos_min": "500", 2024-08-17T17:52:59.721 INFO:teuthology.orchestra.run.smithi136.stdout: "paxos_min_wait": "0.050000", 2024-08-17T17:52:59.721 INFO:teuthology.orchestra.run.smithi136.stdout: "paxos_propose_interval": "1.000000", 2024-08-17T17:52:59.721 INFO:teuthology.orchestra.run.smithi136.stdout: "paxos_service_trim_max": "500", 2024-08-17T17:52:59.721 INFO:teuthology.orchestra.run.smithi136.stdout: "paxos_service_trim_max_multiplier": "20", 2024-08-17T17:52:59.721 INFO:teuthology.orchestra.run.smithi136.stdout: "paxos_service_trim_min": "250", 2024-08-17T17:52:59.721 INFO:teuthology.orchestra.run.smithi136.stdout: "paxos_stash_full_interval": "25", 2024-08-17T17:52:59.721 INFO:teuthology.orchestra.run.smithi136.stdout: "paxos_trim_max": "500", 2024-08-17T17:52:59.721 INFO:teuthology.orchestra.run.smithi136.stdout: "paxos_trim_min": "250", 2024-08-17T17:52:59.721 INFO:teuthology.orchestra.run.smithi136.stdout: "perf": "true", 2024-08-17T17:52:59.721 INFO:teuthology.orchestra.run.smithi136.stdout: "pid_file": "", 2024-08-17T17:52:59.721 INFO:teuthology.orchestra.run.smithi136.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-08-17T17:52:59.721 INFO:teuthology.orchestra.run.smithi136.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-08-17T17:52:59.721 INFO:teuthology.orchestra.run.smithi136.stdout: "public_addr": "-", 2024-08-17T17:52:59.721 INFO:teuthology.orchestra.run.smithi136.stdout: "public_addrv": "", 2024-08-17T17:52:59.721 INFO:teuthology.orchestra.run.smithi136.stdout: "public_bind_addr": "-", 2024-08-17T17:52:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: "public_network": "172.21.0.0/20", 2024-08-17T17:52:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: "public_network_interface": "", 2024-08-17T17:52:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: "qat_compressor_busy_polling": "false", 2024-08-17T17:52:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: "qat_compressor_enabled": "false", 2024-08-17T17:52:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: "qat_compressor_session_max_number": "256", 2024-08-17T17:52:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: "rados_mon_op_timeout": "0", 2024-08-17T17:52:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: "rados_osd_op_timeout": "0", 2024-08-17T17:52:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: "rados_tracing": "false", 2024-08-17T17:52:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_atime_update_interval": "60", 2024-08-17T17:52:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-08-17T17:52:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_balance_parent_reads": "false", 2024-08-17T17:52:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_balance_snap_reads": "false", 2024-08-17T17:52:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_blkin_trace_all": "false", 2024-08-17T17:52:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_blocklist_expire_seconds": "0", 2024-08-17T17:52:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_blocklist_on_break_lock": "true", 2024-08-17T17:52:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_cache": "true", 2024-08-17T17:52:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_cache_block_writes_upfront": "false", 2024-08-17T17:52:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_cache_max_dirty": "25165824", 2024-08-17T17:52:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-08-17T17:52:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_cache_max_dirty_object": "0", 2024-08-17T17:52:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_cache_policy": "writearound", 2024-08-17T17:52:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_cache_size": "33554432", 2024-08-17T17:52:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_cache_target_dirty": "16777216", 2024-08-17T17:52:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-08-17T17:52:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_clone_copy_on_read": "false", 2024-08-17T17:52:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_compression_hint": "none", 2024-08-17T17:52:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_concurrent_management_ops": "10", 2024-08-17T17:52:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_config_pool_override_update_timestamp": "1723915725", 2024-08-17T17:52:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_default_clone_format": "auto", 2024-08-17T17:52:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_default_data_pool": "", 2024-08-17T17:52:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_default_features": "61", 2024-08-17T17:52:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_default_format": "2", 2024-08-17T17:52:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_default_map_options": "", 2024-08-17T17:52:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_default_order": "22", 2024-08-17T17:52:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_default_pool": "rbd", 2024-08-17T17:52:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-08-17T17:52:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_default_stripe_count": "0", 2024-08-17T17:52:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_default_stripe_unit": "0", 2024-08-17T17:52:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_disable_zero_copy_writes": "true", 2024-08-17T17:52:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_discard_granularity_bytes": "65536", 2024-08-17T17:52:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-08-17T17:52:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_enable_alloc_hint": "true", 2024-08-17T17:52:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-08-17T17:52:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_io_scheduler": "simple", 2024-08-17T17:52:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-08-17T17:52:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_journal_commit_age": "5.000000", 2024-08-17T17:52:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-08-17T17:52:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-08-17T17:52:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-08-17T17:52:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-08-17T17:52:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_journal_object_flush_interval": "0", 2024-08-17T17:52:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-08-17T17:52:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-08-17T17:52:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_journal_order": "24", 2024-08-17T17:52:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_journal_pool": "", 2024-08-17T17:52:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_journal_splay_width": "4", 2024-08-17T17:52:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_localize_parent_reads": "false", 2024-08-17T17:52:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_localize_snap_reads": "false", 2024-08-17T17:52:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-08-17T17:52:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-08-17T17:52:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-08-17T17:52:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-08-17T17:52:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-08-17T17:52:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-08-17T17:52:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirror_image_policy_type": "simple", 2024-08-17T17:52:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-08-17T17:52:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-08-17T17:52:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-08-17T17:52:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-08-17T17:52:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-08-17T17:52:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-08-17T17:52:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-08-17T17:52:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirror_memory_autotune": "true", 2024-08-17T17:52:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirror_memory_base": "805306368", 2024-08-17T17:52:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-08-17T17:52:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-08-17T17:52:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-08-17T17:52:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-08-17T17:52:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirror_memory_target": "4294967296", 2024-08-17T17:52:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-08-17T17:52:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-08-17T17:52:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-08-17T17:52:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirroring_delete_delay": "0", 2024-08-17T17:52:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-08-17T17:52:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirroring_replay_delay": "0", 2024-08-17T17:52:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-08-17T17:52:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-08-17T17:52:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_move_to_trash_on_remove": "false", 2024-08-17T17:52:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-08-17T17:52:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_mtime_update_interval": "60", 2024-08-17T17:52:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_non_blocking_aio": "true", 2024-08-17T17:52:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_op_thread_timeout": "60", 2024-08-17T17:52:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_op_threads": "1", 2024-08-17T17:52:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_parent_cache_enabled": "false", 2024-08-17T17:52:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_persistent_cache_mode": "disabled", 2024-08-17T17:52:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_persistent_cache_path": "/tmp", 2024-08-17T17:52:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_persistent_cache_size": "1073741824", 2024-08-17T17:52:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_plugins": "", 2024-08-17T17:52:59.729 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_qos_bps_burst": "0", 2024-08-17T17:52:59.729 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-08-17T17:52:59.729 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_qos_bps_limit": "0", 2024-08-17T17:52:59.729 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_qos_exclude_ops": "0", 2024-08-17T17:52:59.729 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_qos_iops_burst": "0", 2024-08-17T17:52:59.729 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-08-17T17:52:59.729 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_qos_iops_limit": "0", 2024-08-17T17:52:59.729 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_qos_read_bps_burst": "0", 2024-08-17T17:52:59.729 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-08-17T17:52:59.729 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_qos_read_bps_limit": "0", 2024-08-17T17:52:59.729 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_qos_read_iops_burst": "0", 2024-08-17T17:52:59.729 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-08-17T17:52:59.730 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_qos_read_iops_limit": "0", 2024-08-17T17:52:59.730 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_qos_schedule_tick_min": "50", 2024-08-17T17:52:59.730 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_qos_write_bps_burst": "0", 2024-08-17T17:52:59.730 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-08-17T17:52:59.730 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_qos_write_bps_limit": "0", 2024-08-17T17:52:59.730 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_qos_write_iops_burst": "0", 2024-08-17T17:52:59.730 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-08-17T17:52:59.730 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_qos_write_iops_limit": "0", 2024-08-17T17:52:59.730 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_quiesce_notification_attempts": "10", 2024-08-17T17:52:59.730 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_read_from_replica_policy": "default", 2024-08-17T17:52:59.730 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-08-17T17:52:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_readahead_max_bytes": "524288", 2024-08-17T17:52:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_readahead_trigger_requests": "10", 2024-08-17T17:52:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_request_timed_out_seconds": "30", 2024-08-17T17:52:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_skip_partial_discard": "true", 2024-08-17T17:52:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-08-17T17:52:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_tracing": "false", 2024-08-17T17:52:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_validate_names": "true", 2024-08-17T17:52:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: "rbd_validate_pool": "true", 2024-08-17T17:52:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: "restapi_base_url": "", 2024-08-17T17:52:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: "restapi_log_level": "", 2024-08-17T17:52:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-08-17T17:52:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_account_default_quota_max_size": "-1", 2024-08-17T17:52:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_acl_grants_max_num": "100", 2024-08-17T17:52:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_admin_entry": "admin", 2024-08-17T17:52:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-08-17T17:52:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_asio_assert_yielding": "false", 2024-08-17T17:52:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_backend_store": "rados", 2024-08-17T17:52:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_barbican_url": "", 2024-08-17T17:52:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_beast_enable_async": "true", 2024-08-17T17:52:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_bucket_counters_cache": "false", 2024-08-17T17:52:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-08-17T17:52:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-08-17T17:52:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-08-17T17:52:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_bucket_index_max_aio": "128", 2024-08-17T17:52:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-08-17T17:52:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-08-17T17:52:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_bucket_quota_ttl": "600", 2024-08-17T17:52:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-08-17T17:52:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_cache_enabled": "true", 2024-08-17T17:52:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_cache_expiry_interval": "900", 2024-08-17T17:52:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_cache_lru_size": "25000", 2024-08-17T17:52:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_config_store": "rados", 2024-08-17T17:52:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_content_length_compat": "false", 2024-08-17T17:52:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_copy_obj_progress": "true", 2024-08-17T17:52:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-08-17T17:52:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_cors_rules_max_num": "100", 2024-08-17T17:52:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_cross_domain_policy": "", 2024-08-17T17:52:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_default_encryption_key": "", 2024-08-17T17:52:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_kmip_addr": "", 2024-08-17T17:52:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_kmip_ca_path": "", 2024-08-17T17:52:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_kmip_client_cert": "", 2024-08-17T17:52:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_kmip_client_key": "", 2024-08-17T17:52:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-08-17T17:52:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_kmip_password": "", 2024-08-17T17:52:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-08-17T17:52:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_kmip_username": "", 2024-08-17T17:52:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_require_ssl": "true", 2024-08-17T17:52:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-08-17T17:52:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-08-17T17:52:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-08-17T17:52:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-08-17T17:52:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-08-17T17:52:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-08-17T17:52:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-08-17T17:52:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-08-17T17:52:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-08-17T17:52:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-08-17T17:52:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-08-17T17:52:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-08-17T17:52:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-08-17T17:52:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-08-17T17:52:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_suppress_logs": "true", 2024-08-17T17:52:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_vault_addr": "", 2024-08-17T17:52:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_vault_auth": "token", 2024-08-17T17:52:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_vault_namespace": "", 2024-08-17T17:52:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_vault_prefix": "", 2024-08-17T17:52:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-08-17T17:52:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-08-17T17:52:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-08-17T17:52:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-08-17T17:52:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_vault_token_file": "", 2024-08-17T17:52:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-08-17T17:52:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_curl_buffersize": "524288", 2024-08-17T17:52:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_curl_low_speed_limit": "1024", 2024-08-17T17:52:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_curl_low_speed_time": "300", 2024-08-17T17:52:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_curl_tcp_keepalive": "0", 2024-08-17T17:52:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-08-17T17:52:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-08-17T17:52:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-08-17T17:52:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-08-17T17:52:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-08-17T17:52:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_d3n_l1_fadvise": "4", 2024-08-17T17:52:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-08-17T17:52:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-08-17T17:52:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-08-17T17:52:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-08-17T17:52:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-08-17T17:52:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-08-17T17:52:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-08-17T17:52:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-08-17T17:52:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_d4n_l1_fadvise": "4", 2024-08-17T17:52:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-08-17T17:52:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-08-17T17:52:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-08-17T17:52:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_data_log_changes_size": "1000", 2024-08-17T17:52:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_data_log_num_shards": "128", 2024-08-17T17:52:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-08-17T17:52:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_data_log_window": "30", 2024-08-17T17:52:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_data_notify_interval_msec": "0", 2024-08-17T17:52:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_data_sync_poll_interval": "20", 2024-08-17T17:52:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_data_sync_spawn_window": "20", 2024-08-17T17:52:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-08-17T17:52:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-08-17T17:52:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-08-17T17:52:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_default_data_log_backing": "fifo", 2024-08-17T17:52:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-08-17T17:52:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_default_region_info_oid": "default.region", 2024-08-17T17:52:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-08-17T17:52:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-08-17T17:52:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_defer_to_bucket_acls": "", 2024-08-17T17:52:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-08-17T17:52:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_disable_s3select": "false", 2024-08-17T17:52:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-08-17T17:52:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-08-17T17:52:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-08-17T17:52:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-08-17T17:52:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-08-17T17:52:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-08-17T17:52:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-08-17T17:52:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_dmclock_data_res": "500.000000", 2024-08-17T17:52:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-08-17T17:52:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-08-17T17:52:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-08-17T17:52:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-08-17T17:52:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_dns_name": "", 2024-08-17T17:52:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_dns_s3website_name": "", 2024-08-17T17:52:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_dynamic_resharding": "true", 2024-08-17T17:52:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-08-17T17:52:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-08-17T17:52:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-08-17T17:52:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_enable_gc_threads": "true", 2024-08-17T17:52:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_enable_lc_threads": "true", 2024-08-17T17:52:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_enable_mdsearch": "true", 2024-08-17T17:52:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_enable_ops_log": "false", 2024-08-17T17:52:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_enable_quota_threads": "true", 2024-08-17T17:52:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_enable_static_website": "false", 2024-08-17T17:52:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_enable_usage_log": "false", 2024-08-17T17:52:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_enforce_swift_acls": "true", 2024-08-17T17:52:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_exit_timeout_secs": "120", 2024-08-17T17:52:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_expose_bucket": "false", 2024-08-17T17:52:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_extended_http_attrs": "", 2024-08-17T17:52:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_filter": "none", 2024-08-17T17:52:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-08-17T17:52:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_frontends": "beast port=7480", 2024-08-17T17:52:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_gc_max_concurrent_io": "10", 2024-08-17T17:52:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_gc_max_deferred": "50", 2024-08-17T17:52:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-08-17T17:52:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_gc_max_objs": "32", 2024-08-17T17:52:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_gc_max_queue_size": "134213632", 2024-08-17T17:52:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_gc_max_trim_chunk": "16", 2024-08-17T17:52:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_gc_obj_min_wait": "7200", 2024-08-17T17:52:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_gc_processor_max_time": "3600", 2024-08-17T17:52:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_gc_processor_period": "3600", 2024-08-17T17:52:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-08-17T17:52:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_get_obj_window_size": "16777216", 2024-08-17T17:52:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_healthcheck_disabling_path": "", 2024-08-17T17:52:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_ignore_get_invalid_range": "false", 2024-08-17T17:52:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_init_timeout": "300", 2024-08-17T17:52:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-08-17T17:52:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-08-17T17:52:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_kafka_connection_idle": "300", 2024-08-17T17:52:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_kafka_message_timeout": "5000", 2024-08-17T17:52:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_kafka_sleep_timeout": "10", 2024-08-17T17:52:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-08-17T17:52:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-08-17T17:52:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-08-17T17:52:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_keystone_admin_domain": "", 2024-08-17T17:52:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_keystone_admin_password": "", 2024-08-17T17:52:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_keystone_admin_password_path": "", 2024-08-17T17:52:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_keystone_admin_project": "", 2024-08-17T17:52:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_keystone_admin_tenant": "", 2024-08-17T17:52:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_keystone_admin_token": "", 2024-08-17T17:52:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_keystone_admin_token_path": "", 2024-08-17T17:52:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_keystone_admin_user": "", 2024-08-17T17:52:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_keystone_api_version": "2", 2024-08-17T17:52:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_keystone_barbican_domain": "", 2024-08-17T17:52:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_keystone_barbican_password": "", 2024-08-17T17:52:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_keystone_barbican_project": "", 2024-08-17T17:52:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_keystone_barbican_tenant": "", 2024-08-17T17:52:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_keystone_barbican_user": "", 2024-08-17T17:52:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-08-17T17:52:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_keystone_implicit_tenants": "false", 2024-08-17T17:52:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-08-17T17:52:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_keystone_service_token_enabled": "false", 2024-08-17T17:52:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_keystone_token_cache_size": "10000", 2024-08-17T17:52:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_keystone_url": "", 2024-08-17T17:52:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_keystone_verify_ssl": "true", 2024-08-17T17:52:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_lc_debug_interval": "-1", 2024-08-17T17:52:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_lc_lock_max_time": "90", 2024-08-17T17:52:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_lc_max_objs": "32", 2024-08-17T17:52:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_lc_max_rules": "1000", 2024-08-17T17:52:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_lc_max_worker": "3", 2024-08-17T17:52:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_lc_max_wp_worker": "3", 2024-08-17T17:52:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_lc_thread_delay": "0", 2024-08-17T17:52:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-08-17T17:52:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_ldap_dnattr": "uid", 2024-08-17T17:52:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-08-17T17:52:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_ldap_searchfilter": "", 2024-08-17T17:52:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-08-17T17:52:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_ldap_uri": "", 2024-08-17T17:52:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_lfuda_sync_frequency": "60", 2024-08-17T17:52:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-08-17T17:52:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-08-17T17:52:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-08-17T17:52:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_log_http_headers": "", 2024-08-17T17:52:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_log_nonexistent_bucket": "false", 2024-08-17T17:52:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-08-17T17:52:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_log_object_name_utc": "false", 2024-08-17T17:52:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-08-17T17:52:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-08-17T17:52:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_max_attr_name_len": "0", 2024-08-17T17:52:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_max_attr_size": "0", 2024-08-17T17:52:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_max_attrs_num_in_req": "0", 2024-08-17T17:52:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_max_chunk_size": "4194304", 2024-08-17T17:52:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_max_concurrent_requests": "1024", 2024-08-17T17:52:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-08-17T17:52:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_max_dynamic_shards": "1999", 2024-08-17T17:52:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_max_listing_results": "1000", 2024-08-17T17:52:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_max_notify_retries": "10", 2024-08-17T17:52:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_max_objs_per_shard": "100000", 2024-08-17T17:52:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_max_put_param_size": "1048576", 2024-08-17T17:52:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_max_put_size": "5368709120", 2024-08-17T17:52:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_max_slo_entries": "1000", 2024-08-17T17:52:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_md_log_max_shards": "64", 2024-08-17T17:52:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_md_notify_interval_msec": "200", 2024-08-17T17:52:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_meta_sync_poll_interval": "20", 2024-08-17T17:52:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_meta_sync_spawn_window": "20", 2024-08-17T17:52:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-08-17T17:52:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_mp_lock_max_time": "600", 2024-08-17T17:52:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-08-17T17:52:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_multipart_min_part_size": "5242880", 2024-08-17T17:52:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-08-17T17:52:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-08-17T17:52:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_nfs_fhcache_size": "2017", 2024-08-17T17:52:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-08-17T17:52:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-08-17T17:52:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_nfs_lru_lanes": "5", 2024-08-17T17:52:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_nfs_max_gc": "300", 2024-08-17T17:52:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-08-17T17:52:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_nfs_run_gc_threads": "false", 2024-08-17T17:52:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_nfs_run_lc_threads": "false", 2024-08-17T17:52:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_nfs_run_quota_threads": "false", 2024-08-17T17:52:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_nfs_run_sync_thread": "false", 2024-08-17T17:52:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-08-17T17:52:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-08-17T17:52:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_num_async_rados_threads": "32", 2024-08-17T17:52:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_num_control_oids": "8", 2024-08-17T17:52:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_numa_node": "-1", 2024-08-17T17:52:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_obj_stripe_size": "4194304", 2024-08-17T17:52:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-08-17T17:52:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_objexp_chunk_size": "100", 2024-08-17T17:52:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_objexp_gc_interval": "600", 2024-08-17T17:52:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_objexp_hints_num_shards": "127", 2024-08-17T17:52:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-08-17T17:52:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-08-17T17:52:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_op_thread_timeout": "600", 2024-08-17T17:52:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_op_tracing": "false", 2024-08-17T17:52:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_opa_token": "", 2024-08-17T17:52:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_opa_url": "", 2024-08-17T17:52:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_opa_verify_ssl": "true", 2024-08-17T17:52:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-08-17T17:52:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-08-17T17:52:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_ops_log_rados": "false", 2024-08-17T17:52:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_ops_log_socket_path": "", 2024-08-17T17:52:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-08-17T17:52:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-08-17T17:52:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-08-17T17:52:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_period_push_interval": "2.000000", 2024-08-17T17:52:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_period_push_interval_max": "30.000000", 2024-08-17T17:52:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_period_root_pool": ".rgw.root", 2024-08-17T17:52:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-08-17T17:52:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-08-17T17:52:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_posix_cache_lanes": "3", 2024-08-17T17:52:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-08-17T17:52:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_posix_cache_max_buckets": "100", 2024-08-17T17:52:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_posix_cache_partitions": "3", 2024-08-17T17:52:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-08-17T17:52:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_print_continue": "true", 2024-08-17T17:52:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_print_prohibited_content_length": "false", 2024-08-17T17:52:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-08-17T17:52:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-08-17T17:52:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-08-17T17:52:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-08-17T17:52:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_rados_tracing": "false", 2024-08-17T17:52:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_realm": "", 2024-08-17T17:52:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_realm_id": "", 2024-08-17T17:52:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-08-17T17:52:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_region": "", 2024-08-17T17:52:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_region_root_pool": ".rgw.root", 2024-08-17T17:52:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_relaxed_region_enforcement": "false", 2024-08-17T17:52:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-08-17T17:52:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-08-17T17:52:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_request_uri": "", 2024-08-17T17:52:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_reshard_batch_size": "64", 2024-08-17T17:52:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-08-17T17:52:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_reshard_debug_interval": "-1", 2024-08-17T17:52:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_reshard_max_aio": "128", 2024-08-17T17:52:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_reshard_num_logs": "16", 2024-08-17T17:52:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_reshard_thread_interval": "600", 2024-08-17T17:52:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_resolve_cname": "false", 2024-08-17T17:52:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_rest_getusage_op_compat": "false", 2024-08-17T17:52:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_run_sync_thread": "true", 2024-08-17T17:52:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-08-17T17:52:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-08-17T17:52:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_s3_auth_use_keystone": "false", 2024-08-17T17:52:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_s3_auth_use_ldap": "false", 2024-08-17T17:52:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_s3_auth_use_rados": "true", 2024-08-17T17:52:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_s3_auth_use_sts": "false", 2024-08-17T17:52:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-08-17T17:52:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_s3_success_create_obj_status": "0", 2024-08-17T17:52:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-08-17T17:52:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_scheduler_type": "throttler", 2024-08-17T17:52:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_script_uri": "", 2024-08-17T17:52:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_service_provider_name": "", 2024-08-17T17:52:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-08-17T17:52:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_sts_client_id": "", 2024-08-17T17:52:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_sts_client_secret": "", 2024-08-17T17:52:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_sts_entry": "sts", 2024-08-17T17:52:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_sts_key": "sts", 2024-08-17T17:52:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_sts_max_session_duration": "43200", 2024-08-17T17:52:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_sts_min_session_duration": "900", 2024-08-17T17:52:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_sts_token_introspection_url": "", 2024-08-17T17:52:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_swift_account_in_url": "false", 2024-08-17T17:52:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_swift_auth_entry": "auth", 2024-08-17T17:52:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_swift_auth_url": "", 2024-08-17T17:52:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_swift_custom_header": "", 2024-08-17T17:52:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_swift_enforce_content_length": "false", 2024-08-17T17:52:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_swift_need_stats": "true", 2024-08-17T17:52:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_swift_tenant_name": "", 2024-08-17T17:52:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_swift_token_expiration": "86400", 2024-08-17T17:52:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_swift_url": "", 2024-08-17T17:52:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_swift_url_prefix": "swift", 2024-08-17T17:52:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_swift_versioning_enabled": "false", 2024-08-17T17:52:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-08-17T17:52:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-08-17T17:52:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_sync_lease_period": "120", 2024-08-17T17:52:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-08-17T17:52:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_sync_log_trim_interval": "1200", 2024-08-17T17:52:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-08-17T17:52:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-08-17T17:52:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-08-17T17:52:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_sync_obj_etag_verify": "false", 2024-08-17T17:52:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_sync_trace_history_size": "4096", 2024-08-17T17:52:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-08-17T17:52:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-08-17T17:52:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_thread_pool_size": "512", 2024-08-17T17:52:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_topic_persistency_max_retries": "0", 2024-08-17T17:52:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-08-17T17:52:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-08-17T17:52:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_topic_require_publish_policy": "false", 2024-08-17T17:52:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_torrent_comment": "", 2024-08-17T17:52:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_torrent_createby": "", 2024-08-17T17:52:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_torrent_encoding": "", 2024-08-17T17:52:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_torrent_flag": "false", 2024-08-17T17:52:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_torrent_max_size": "5368709120", 2024-08-17T17:52:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_torrent_origin": "", 2024-08-17T17:52:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_torrent_sha_unit": "524288", 2024-08-17T17:52:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_torrent_tracker": "", 2024-08-17T17:52:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_trust_forwarded_https": "false", 2024-08-17T17:52:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-08-17T17:52:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_usage_log_tick_interval": "30", 2024-08-17T17:52:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_usage_max_shards": "32", 2024-08-17T17:52:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_usage_max_user_shards": "1", 2024-08-17T17:52:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_use_opa_authz": "false", 2024-08-17T17:52:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_user_counters_cache": "false", 2024-08-17T17:52:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_user_counters_cache_size": "10000", 2024-08-17T17:52:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-08-17T17:52:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_user_default_quota_max_size": "-1", 2024-08-17T17:52:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_user_max_buckets": "1000", 2024-08-17T17:52:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_user_policies_max_num": "100", 2024-08-17T17:52:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-08-17T17:52:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-08-17T17:52:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_user_quota_sync_interval": "86400", 2024-08-17T17:52:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-08-17T17:52:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_user_unique_email": "true", 2024-08-17T17:52:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_verify_ssl": "true", 2024-08-17T17:52:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_website_routing_rules_max_num": "50", 2024-08-17T17:52:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_zone": "", 2024-08-17T17:52:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_zone_id": "", 2024-08-17T17:52:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-08-17T17:52:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_zonegroup": "", 2024-08-17T17:52:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_zonegroup_id": "", 2024-08-17T17:52:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-08-17T17:52:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: "rgwlc_auto_session_clear": "true", 2024-08-17T17:52:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: "rgwlc_skip_bucket_step": "false", 2024-08-17T17:52:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: "rocksdb_block_size": "4096", 2024-08-17T17:52:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-08-17T17:52:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-08-17T17:52:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-08-17T17:52:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-08-17T17:52:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: "rocksdb_cache_shard_bits": "4", 2024-08-17T17:52:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: "rocksdb_cache_size": "536870912", 2024-08-17T17:52:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: "rocksdb_cache_type": "binned_lru", 2024-08-17T17:52:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-08-17T17:52:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-08-17T17:52:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-08-17T17:52:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "rocksdb_collect_compaction_stats": "false", 2024-08-17T17:52:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "rocksdb_collect_extended_stats": "false", 2024-08-17T17:52:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "rocksdb_collect_memory_stats": "false", 2024-08-17T17:52:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-08-17T17:52:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "rocksdb_index_type": "binary_search", 2024-08-17T17:52:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "rocksdb_log_to_ceph_log": "true", 2024-08-17T17:52:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "rocksdb_metadata_block_size": "4096", 2024-08-17T17:52:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "rocksdb_partition_filters": "false", 2024-08-17T17:52:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "rocksdb_perf": "false", 2024-08-17T17:52:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-08-17T17:52:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-08-17T17:52:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "rotating_keys_renewal_timeout": "10", 2024-08-17T17:52:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "run_dir": "/var/run/ceph", 2024-08-17T17:52:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: "seastore_block_create": "true", 2024-08-17T17:52:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: "seastore_cache_lru_size": "67108864", 2024-08-17T17:52:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: "seastore_cbjournal_size": "5368709120", 2024-08-17T17:52:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: "seastore_data_delta_based_overwrite": "0", 2024-08-17T17:52:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: "seastore_default_max_object_size": "16777216", 2024-08-17T17:52:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-08-17T17:52:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: "seastore_device_size": "53687091200", 2024-08-17T17:52:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-08-17T17:52:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: "seastore_full_integrity_check": "false", 2024-08-17T17:52:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: "seastore_journal_batch_capacity": "16", 2024-08-17T17:52:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-08-17T17:52:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-08-17T17:52:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: "seastore_journal_iodepth_limit": "5", 2024-08-17T17:52:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: "seastore_main_device_type": "SSD", 2024-08-17T17:52:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: "seastore_max_concurrent_transactions": "8", 2024-08-17T17:52:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: "seastore_max_data_allocation_size": "32768", 2024-08-17T17:52:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-08-17T17:52:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-08-17T17:52:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-08-17T17:52:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-08-17T17:52:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: "seastore_segment_size": "67108864", 2024-08-17T17:52:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: "set_keepcaps": "false", 2024-08-17T17:52:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: "setgroup": "ceph", 2024-08-17T17:52:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: "setuser": "ceph", 2024-08-17T17:52:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: "setuser_match_path": "", 2024-08-17T17:52:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: "target_max_misplaced_ratio": "0.050000", 2024-08-17T17:52:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: "thp": "false", 2024-08-17T17:52:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: "threadpool_default_timeout": "60", 2024-08-17T17:52:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: "threadpool_empty_queue_max_wait": "2", 2024-08-17T17:52:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: "throttler_perf_counter": "true", 2024-08-17T17:52:59.762 INFO:teuthology.orchestra.run.smithi136.stdout: "tmp_dir": "/tmp", 2024-08-17T17:52:59.762 INFO:teuthology.orchestra.run.smithi136.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX" 2024-08-17T17:52:59.762 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:53:00.460 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-08-17T17:53:00.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:00 smithi136 ceph-mon[109861]: pgmap v2065: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:00.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:00 smithi136 ceph-mon[115078]: pgmap v2065: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:00.707 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:53:00.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:00 smithi165 ceph-mon[105517]: pgmap v2065: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:02.373 INFO:teuthology.orchestra.run.smithi136.stdout:mon.a: {} 2024-08-17T17:53:02.374 INFO:teuthology.orchestra.run.smithi136.stdout:mon.b: {} 2024-08-17T17:53:02.374 INFO:teuthology.orchestra.run.smithi136.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-17T17:53:02.374 INFO:teuthology.orchestra.run.smithi136.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-17T17:53:02.376 INFO:teuthology.orchestra.run.smithi136.stdout:mon.c: {} 2024-08-17T17:53:02.377 INFO:teuthology.orchestra.run.smithi136.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-17T17:53:02.677 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:02 smithi136 ceph-mon[109861]: pgmap v2066: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:02.677 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:02 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/518691657' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-17T17:53:02.677 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:02 smithi136 ceph-mon[115078]: pgmap v2066: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:02.678 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:02 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/518691657' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-17T17:53:02.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:02 smithi165 ceph-mon[105517]: pgmap v2066: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:02.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:02 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/518691657' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-17T17:53:02.995 INFO:tasks.thrashosds.thrasher:Unable to test ceph-objectstore-tool, not available on all OSD nodes 2024-08-17T17:53:02.996 INFO:teuthology.task.sequential:In sequential, running task full_sequential... 2024-08-17T17:53:02.997 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-17T17:53:02.998 INFO:tasks.radosbench:Beginning radosbench... 2024-08-17T17:53:02.998 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-08-17T17:53:02.999 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-08-17T17:53:02.999 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-08-17T17:53:02.999 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-17T17:53:02.999 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:53:03.000 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-08-17T17:53:03.000 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:03.000 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-08-17T17:53:03.000 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:53:03.001 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-08-17T17:53:03.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-17T17:53:03.001 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-08-17T17:53:03.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:53:03.027 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:03.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:03.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:03.129 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:03.161 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:03.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:03.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:03.263 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:03.264 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:53:03.266 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:53:03.266 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:53:03.266 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:53:03.286 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:03.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:03.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:03.388 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:03.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:03.409 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:03.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:03.510 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:03.533 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:03.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:03.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:03.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:03.657 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:03.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:03.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:03.759 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:03.780 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:03.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:03.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:03.882 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:03.903 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:03.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:03.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:04.005 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:04.027 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:04.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:04.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:04.129 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:04.150 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:04.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:04.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:04.251 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:04.277 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:04.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:04.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:04.379 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:04.405 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:04.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:04.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:04.507 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:04.528 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:04.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:04.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:04.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:04.653 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:04.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:04.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:04.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:04 smithi136 ceph-mon[109861]: pgmap v2067: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:04.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:04 smithi136 ceph-mon[115078]: pgmap v2067: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:04.755 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:04.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:04 smithi165 ceph-mon[105517]: pgmap v2067: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:04.776 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:04.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:04.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:04.878 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:04.899 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:04.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:04.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:05.001 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:05.022 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:05.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:05.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:05.123 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:05.144 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:05.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:05.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:05.246 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:05.267 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:05.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:05.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:05.369 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:05.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:05.391 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:05.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:05.492 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:05.513 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:05.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:05.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:05.615 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:05.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:05.637 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:05.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:05.738 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:05.759 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:05.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:05.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:05.861 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:05.883 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:05.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:05.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:05.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:06.006 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:06.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:06.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:06.107 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:06.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:06.129 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:06.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:06.230 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:06.252 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:06.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:06.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:06.354 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:06.376 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:06.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:06.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:06.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:06.500 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:06.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:06.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:06.602 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:06.624 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:06.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:06.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:06.726 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:06.748 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:06.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:06.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:06 smithi165 ceph-mon[105517]: pgmap v2068: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:53:06.786 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:06 smithi136 ceph-mon[109861]: pgmap v2068: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:06.786 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:53:06.786 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:06 smithi136 ceph-mon[115078]: pgmap v2068: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:06.786 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:53:06.850 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:06.873 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:06.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:06.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:06.975 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:06.996 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:06.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:06.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:07.098 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:07.120 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:07.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:07.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:07.222 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:07.243 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:07.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:07.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:07.345 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:07.367 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:07.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:07.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:07.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:07.491 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:07.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:07.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:07.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:07.616 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:07.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:07.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:07.710 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:53:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:53:07] "GET /metrics HTTP/1.1" 200 34056 "" "Prometheus/2.51.0" 2024-08-17T17:53:07.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:07.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:07.739 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:07.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:07.840 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:07.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:07.862 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:07.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:07.963 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:07.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:07.985 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:07.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:08.086 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:08.103 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:53:08.104 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T17:53:08.104 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-17T17:53:08.104 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:53:08.110 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:08.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:08.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:08.212 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:08.233 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:08.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:08.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:08.335 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:08.360 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:08.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:08.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:08.461 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T17:53:08.461 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:08.487 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:08.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:08.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:08.589 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:08.615 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:08.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:08.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:08.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:08.738 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:08.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:08.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:08 smithi165 ceph-mon[105517]: pgmap v2069: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:08 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/685988681' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:53:08.798 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:08 smithi136 ceph-mon[109861]: pgmap v2069: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:08.798 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:08 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/685988681' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:53:08.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:08 smithi136 ceph-mon[115078]: pgmap v2069: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:08.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:08 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/685988681' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:53:08.840 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:08.861 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:08.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:08.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:08.963 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:08.989 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:08.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:08.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:09.091 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:09.116 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:09.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:09.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:09.218 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:09.244 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:09.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:09.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:09.346 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:09.368 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:09.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:09.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:09.470 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:09.473 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'unique_pool_0' created 2024-08-17T17:53:09.498 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:09.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:09.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:09.600 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:09.622 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:09.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:09.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:09.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:09 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:53:09.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:09 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/685988681' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:53:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:09 smithi136 ceph-mon[115078]: osdmap e1302: 8 total, 8 up, 8 in 2024-08-17T17:53:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:09 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/360854604' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-08-17T17:53:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:09 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:53:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:09 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/685988681' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:53:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:09 smithi136 ceph-mon[109861]: osdmap e1302: 8 total, 8 up, 8 in 2024-08-17T17:53:09.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:09 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/360854604' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-08-17T17:53:09.723 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:09.745 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:09.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:09.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:09.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:09 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:53:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:09 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/685988681' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:53:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:09 smithi165 ceph-mon[105517]: osdmap e1302: 8 total, 8 up, 8 in 2024-08-17T17:53:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:09 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/360854604' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-08-17T17:53:09.846 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:09.872 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:09.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:09.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:09.974 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:09.996 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:09.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:09.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:10.098 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:10.123 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:10.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:10.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:10.225 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:10.247 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-08-17T17:53:10.248 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:10.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:10.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:10.350 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:10.375 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:10.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:10.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:10.477 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:10.504 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:10.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:10.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:10.606 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:10.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:10.632 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:10.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:10.733 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:10.755 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:10.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:10.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:10 smithi165 ceph-mon[105517]: pgmap v2071: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:10 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/360854604' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-08-17T17:53:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:10 smithi165 ceph-mon[105517]: osdmap e1303: 8 total, 8 up, 8 in 2024-08-17T17:53:10.855 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:10 smithi136 ceph-mon[109861]: pgmap v2071: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:10.855 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:10 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/360854604' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-08-17T17:53:10.855 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:10 smithi136 ceph-mon[109861]: osdmap e1303: 8 total, 8 up, 8 in 2024-08-17T17:53:10.855 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:10 smithi136 ceph-mon[115078]: pgmap v2071: 73 pgs: 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:10.856 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:10 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/360854604' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-08-17T17:53:10.856 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:10 smithi136 ceph-mon[115078]: osdmap e1303: 8 total, 8 up, 8 in 2024-08-17T17:53:10.857 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:10.883 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:10.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:10.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:10.985 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:11.006 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:11.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:11.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:11.107 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:11.133 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:11.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:11.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:11.147 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T17:53:11.148 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T17:53:11.148 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:53:11.148 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:53:11.157 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T17:53:11.158 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:53:11.174 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T17:53:11.174 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:53:11.192 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T17:53:11.192 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:53:11.208 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T17:53:11.208 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:53:11.224 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T17:53:11.225 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:53:11.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:11.241 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T17:53:11.241 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:53:11.260 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:11.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:11.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:11.361 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:11.386 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:11.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:11.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:11.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:11.511 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:11.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:11.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:11.613 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:11.634 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:11.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:11.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:11.706 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-17T17:53:11.736 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:11 smithi165 ceph-mon[105517]: osdmap e1304: 8 total, 8 up, 8 in 2024-08-17T17:53:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:11 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2084145312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:53:11.772 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:11.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:11.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:11.780 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:11 smithi136 ceph-mon[115078]: osdmap e1304: 8 total, 8 up, 8 in 2024-08-17T17:53:11.780 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:11 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2084145312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:53:11.780 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:11 smithi136 ceph-mon[109861]: osdmap e1304: 8 total, 8 up, 8 in 2024-08-17T17:53:11.780 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:11 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2084145312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:53:11.874 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:11.900 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:11.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:11.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:12.001 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:53:12.002 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:12.024 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:12.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:12.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:12.126 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:12.148 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:12.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:12.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:12.250 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:12.271 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:12.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:12.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:12.372 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:12.393 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:12.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:12.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:12.495 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:12.516 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:12.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:12.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:12.618 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:12.644 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:12.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:12.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:12.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:12 smithi136 ceph-mon[109861]: pgmap v2074: 89 pgs: 4 creating+peering, 12 unknown, 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:12.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:12 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:53:12.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:12 smithi136 ceph-mon[109861]: osdmap e1305: 8 total, 8 up, 8 in 2024-08-17T17:53:12.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:12 smithi136 ceph-mon[115078]: pgmap v2074: 89 pgs: 4 creating+peering, 12 unknown, 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:12.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:12 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:53:12.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:12 smithi136 ceph-mon[115078]: osdmap e1305: 8 total, 8 up, 8 in 2024-08-17T17:53:12.746 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:12 smithi165 ceph-mon[105517]: pgmap v2074: 89 pgs: 4 creating+peering, 12 unknown, 73 active+clean; 582 KiB data, 838 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:12 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:53:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:12 smithi165 ceph-mon[105517]: osdmap e1305: 8 total, 8 up, 8 in 2024-08-17T17:53:12.770 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:12.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:12.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:12.873 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:12.876 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:53:12.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:12.898 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:12.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:12.999 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:13.020 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:13.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:13.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:13.122 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:13.144 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:13.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:13.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:13.156 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:53:13.247 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:13.268 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:13.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:13.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:13.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:13.395 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:13.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:13.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:13.497 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:13.519 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:13.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:13.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:13.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:13.646 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:13.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:13.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:13.748 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:13.774 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:13.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:13.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:13.875 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:13.901 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:13.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:13.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:14.002 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:14.024 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:14.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:14.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:14.126 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:14.152 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:14.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:14.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:14.253 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:14.275 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:14.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:14.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:14.377 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:14.402 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:14.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:14.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:14.445 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:53:14.484 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:53:14.504 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:14.528 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:14.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:14.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:14.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:14.656 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:14.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:14.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:14.717 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:14 smithi136 ceph-mon[109861]: pgmap v2076: 89 pgs: 4 creating+peering, 85 active+clean; 582 KiB data, 842 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:14.717 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:14 smithi136 ceph-mon[115078]: pgmap v2076: 89 pgs: 4 creating+peering, 85 active+clean; 582 KiB data, 842 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:14.758 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:14 smithi165 ceph-mon[105517]: pgmap v2076: 89 pgs: 4 creating+peering, 85 active+clean; 582 KiB data, 842 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:14.783 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:14.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:14.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:14.885 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:14.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:14.908 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:14.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:15.009 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:15.030 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:15.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:15.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:15.131 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:15.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:15.158 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:15.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:15.259 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:15.283 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:15.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:15.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:15.341 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:53:15.385 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:15.407 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:15.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:15.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:15.494 INFO:teuthology.orchestra.run.smithi136.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-08-17T17:53:15.508 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:15.531 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:15.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:15.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:15.633 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:15.655 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:15.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:15.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:15.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:15 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3339434259' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:53:15.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:15 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3339434259' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:53:15.757 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:15.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:15 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3339434259' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:53:15.779 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:15.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:15.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:15.880 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:15.902 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:15.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:15.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:16.004 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:16.025 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:16.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:16.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:16.127 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:16.148 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:16.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:16.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:16.250 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:16.270 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:16.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:16.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:16.372 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:16.393 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:16.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:16.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:16.495 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:16.516 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:16.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:16.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:16.581 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:16 smithi136 ceph-mon[109861]: pgmap v2077: 89 pgs: 4 creating+peering, 85 active+clean; 582 KiB data, 842 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:16.581 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:16 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3339434259' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:53:16.581 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:16 smithi136 ceph-mon[109861]: osdmap e1306: 8 total, 8 up, 8 in 2024-08-17T17:53:16.582 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:16 smithi136 ceph-mon[115078]: pgmap v2077: 89 pgs: 4 creating+peering, 85 active+clean; 582 KiB data, 842 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:16.582 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:16 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3339434259' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:53:16.582 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:16 smithi136 ceph-mon[115078]: osdmap e1306: 8 total, 8 up, 8 in 2024-08-17T17:53:16.618 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:16.638 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:16.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:16.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:16.740 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:16.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:16 smithi165 ceph-mon[105517]: pgmap v2077: 89 pgs: 4 creating+peering, 85 active+clean; 582 KiB data, 842 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:16.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:16 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3339434259' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:53:16.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:16 smithi165 ceph-mon[105517]: osdmap e1306: 8 total, 8 up, 8 in 2024-08-17T17:53:16.763 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:16.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:16.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:16.865 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:16.887 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:16.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:16.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:16.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:17.010 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:17.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:17.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:17.051 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T17:53:17.051 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:53:17.112 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:17.133 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:17.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:17.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:17.234 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:17.255 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:17.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:17.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:17.357 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:17.378 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:17.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:17.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:17.480 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:17.502 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:17.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:17.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:17.601 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-17T17:53:17.604 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:17.606 DEBUG:teuthology.orchestra.run.smithi136:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-08-17T17:53:17.606 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-08-17T17:53:17.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:53:17.625 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:17.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:17.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:17.647 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-17T17:53:17.647 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-17T17:53:17.698 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:hints = 1 2024-08-17T17:53:17.698 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-17T17:53:17.698 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Object prefix: benchmark_data_smithi136_235573 2024-08-17T17:53:17.699 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:53:17.699 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 0 0 0 0 0 0 - 0 2024-08-17T17:53:17.727 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:17.765 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:17 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:53:17.765 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:17 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:53:17.766 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:53:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:53:17] "GET /metrics HTTP/1.1" 200 34056 "" "Prometheus/2.51.0" 2024-08-17T17:53:17.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:17.782 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:17.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:17.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:17.920 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:17.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:17.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:17 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:53:18.022 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:18.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:18.064 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:18.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:18.165 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:18.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:18.204 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:18.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:18.305 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:18.344 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:18.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:18.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:18.446 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:18.481 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:18.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:18.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:18.583 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:18.613 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:18.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:18.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:18.644 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:18 smithi136 ceph-mon[109861]: pgmap v2079: 89 pgs: 89 active+clean; 582 KiB data, 840 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:18.645 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:18 smithi136 ceph-mon[115078]: pgmap v2079: 89 pgs: 89 active+clean; 582 KiB data, 840 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:18.698 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 1 16 3665 3649 228.05 228.062 0.00616612 0.00437152 2024-08-17T17:53:18.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:18.746 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:18.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:18.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:18.848 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:18.891 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:18.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:18.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:18.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:19.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:18 smithi165 ceph-mon[105517]: pgmap v2079: 89 pgs: 89 active+clean; 582 KiB data, 840 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:19.025 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:19.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:19.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:19.127 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:19.163 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:19.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:19.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:19.265 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:19.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:19.301 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:19.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:19.402 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:19.438 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:19.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:19.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:19.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:19.569 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:19.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:19.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:19.671 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:19.698 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 2 16 6764 6748 210.855 193.688 0.00376416 0.00473188 2024-08-17T17:53:19.714 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:19.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:19.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:19.816 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:19.847 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:19.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:19.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:19.949 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:19.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:19.986 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:19.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:20.087 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:20.121 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:20.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:20.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:20.223 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:20.257 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:20.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:20.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:20.359 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:20.390 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:20.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:20.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:20.492 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:20.532 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:20.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:20.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:20.557 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T17:53:20.571 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:53:20.629 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T17:53:20.629 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:53:20.633 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:20.665 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T17:53:20.666 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:53:20.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:20.673 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:20.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:20.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:20 smithi136 ceph-mon[109861]: pgmap v2080: 89 pgs: 89 active+clean; 582 KiB data, 835 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:20.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:20 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3415380840' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:53:20.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:20 smithi136 ceph-mon[115078]: pgmap v2080: 89 pgs: 89 active+clean; 582 KiB data, 835 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:20.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:20 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3415380840' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:53:20.698 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 3 16 10005 9989 208.085 202.562 0.00345755 0.004799 2024-08-17T17:53:20.701 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T17:53:20.701 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:53:20.732 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T17:53:20.732 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:53:20.768 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T17:53:20.768 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:53:20.774 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:20.796 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T17:53:20.797 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:53:20.806 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:20.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:20.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:20.830 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T17:53:20.830 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:53:20.864 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T17:53:20.864 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:53:20.908 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:20.945 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:20.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:20.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:21.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:20 smithi165 ceph-mon[105517]: pgmap v2080: 89 pgs: 89 active+clean; 582 KiB data, 835 MiB used, 714 GiB / 715 GiB avail 2024-08-17T17:53:21.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:20 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3415380840' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:53:21.047 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:21.083 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:21.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:21.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:21.185 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:21.217 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:21.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:21.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:21.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:21.352 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:21.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:21.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:21.454 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:21.487 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:21.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:21.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:21.589 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:21.624 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:21.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:21.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:21.699 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 4 16 13301 13285 207.56 206 0.00529935 0.00481066 2024-08-17T17:53:21.726 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:21.759 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:21.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:21.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:21.861 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:21.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:21.892 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:21.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:21 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:53:21.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:21 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3415380840' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:53:21.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:21 smithi136 ceph-mon[109861]: osdmap e1307: 8 total, 8 up, 8 in 2024-08-17T17:53:21.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:21 smithi136 ceph-mon[109861]: pgmap v2082: 89 pgs: 89 active+clean; 140 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s wr, 278 op/s 2024-08-17T17:53:21.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:21 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2201856954' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:53:21.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:53:21.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:53:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:21 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:53:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:21 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3415380840' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:53:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:21 smithi136 ceph-mon[115078]: osdmap e1307: 8 total, 8 up, 8 in 2024-08-17T17:53:21.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:21 smithi136 ceph-mon[115078]: pgmap v2082: 89 pgs: 89 active+clean; 140 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s wr, 278 op/s 2024-08-17T17:53:21.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:21 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2201856954' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:53:21.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:53:21.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:53:21.993 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:22.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:21 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:53:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:21 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3415380840' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:53:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:21 smithi165 ceph-mon[105517]: osdmap e1307: 8 total, 8 up, 8 in 2024-08-17T17:53:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:21 smithi165 ceph-mon[105517]: pgmap v2082: 89 pgs: 89 active+clean; 140 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s wr, 278 op/s 2024-08-17T17:53:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:21 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2201856954' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:53:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:53:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:53:22.030 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:22.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:22.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:22.132 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:22.163 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:22.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:22.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:22.265 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:22.304 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:22.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:22.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:22.363 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:53:22.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:22.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:22.438 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:22.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:22.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:22.568 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:22.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:22.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:22.670 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:22.699 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 5 16 16342 16326 204.058 190.062 0.00404506 0.00489439 2024-08-17T17:53:22.703 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:22.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:22.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:22.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:22.837 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:22.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:22.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:22.939 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:22.972 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:22.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:22.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:23.074 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:23.111 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:23.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:23.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:23.213 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:23.250 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:23.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:23.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:23.332 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:53:23.352 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:23.415 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:23.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:23.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:23.517 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:23.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:23.558 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:23.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:23.658 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:23.691 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:23.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:23.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:23.699 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 6 16 19255 19239 200.39 182.062 0.00591842 0.00498401 2024-08-17T17:53:23.793 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:23.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:23.828 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:23.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:23.887 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:23 smithi136 ceph-mon[109861]: pgmap v2083: 89 pgs: 89 active+clean; 730 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-08-17T17:53:23.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:23 smithi136 ceph-mon[115078]: pgmap v2083: 89 pgs: 89 active+clean; 730 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-08-17T17:53:23.929 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:23.966 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:23.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:23.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:23 smithi165 ceph-mon[105517]: pgmap v2083: 89 pgs: 89 active+clean; 730 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-08-17T17:53:24.067 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:24.100 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:24.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:24.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:24.202 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:24.248 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:24.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:24.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:24.350 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:24.386 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:24.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:24.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:24.401 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:53:24.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:24.532 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:24.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:24.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:24.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:24.667 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:24.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:24.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:24.699 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 7 16 21948 21932 195.8 168.312 0.0027608 0.0051023 2024-08-17T17:53:24.769 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:24.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:24.804 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:24.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:24.905 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:24.937 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:24.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:24.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:25.039 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:25.078 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:25.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:25.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:25.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:25.218 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:25.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:25.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:25.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:25.352 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:25.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:25.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:25.453 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:25.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:25.525 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:25.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:25.626 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:25.672 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:25.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:25.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:25.699 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 8 16 24558 24542 191.714 163.125 0.00513211 0.00521056 2024-08-17T17:53:25.775 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:25.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:25.816 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:25.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:25.917 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:25.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:25.939 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:25.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:25.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:25 smithi136 ceph-mon[109861]: pgmap v2084: 89 pgs: 89 active+clean; 730 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-08-17T17:53:25.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:25 smithi136 ceph-mon[115078]: pgmap v2084: 89 pgs: 89 active+clean; 730 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-08-17T17:53:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:25 smithi165 ceph-mon[105517]: pgmap v2084: 89 pgs: 89 active+clean; 730 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-08-17T17:53:26.040 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:26.076 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:26.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:26.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:26.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:26.213 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:26.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:26.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:26.315 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:26.348 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:26.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:26.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:26.450 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:26.489 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:26.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:26.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:26.591 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:26.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:26.624 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:26.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:26.699 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 9 16 26811 26795 186.057 140.812 0.00907913 0.00536975 2024-08-17T17:53:26.725 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:26.760 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:26.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:26.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:26.862 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:26.901 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:26.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:26.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:27.003 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:27.023 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:27.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:27.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:27.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:27.160 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:27.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:27.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:27.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:27.285 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:27.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:27.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:27.387 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:27.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:27.426 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:27.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:27.527 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:27.558 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:27.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:27.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:27.660 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:27.690 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:27.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:27.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:27.699 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 10 16 28844 28828 180.156 127.062 0.00924583 0.00554543 2024-08-17T17:53:27.779 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:27 smithi136 ceph-mon[109861]: pgmap v2085: 89 pgs: 89 active+clean; 1.2 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-08-17T17:53:27.779 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:53:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:53:27] "GET /metrics HTTP/1.1" 200 36532 "" "Prometheus/2.51.0" 2024-08-17T17:53:27.779 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:27 smithi136 ceph-mon[115078]: pgmap v2085: 89 pgs: 89 active+clean; 1.2 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-08-17T17:53:27.792 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:27.816 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:53:27.816 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:53:27.820 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:27.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:27.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:27.922 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:27.953 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:27.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:27.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:28.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:27 smithi165 ceph-mon[105517]: pgmap v2085: 89 pgs: 89 active+clean; 1.2 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-08-17T17:53:28.055 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:28.089 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:28.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:28.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:28.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:28.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:28.224 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:28.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:28.325 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:28.355 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:28.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:28.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:28.457 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:28.487 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:28.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:28.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:28.589 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:28.624 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:28.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:28.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:28.699 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 11 16 30971 30955 175.863 132.938 0.00414043 0.00568116 2024-08-17T17:53:28.726 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:28.753 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:28.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:28.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:28.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:28.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:28.894 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:28.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:28.995 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:29.025 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:29.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:29.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:29.127 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:29.167 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:29.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:29.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:29.269 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:29.307 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:29.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:29.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:29.409 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:29.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:29.436 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:29.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:29.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:29.570 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:29.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:29.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:29.672 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:29.699 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 12 16 32754 32738 170.493 111.438 0.00234241 0.00585513 2024-08-17T17:53:29.702 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:29.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:29.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:29.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:29.825 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:29.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:29.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:29.926 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:29.954 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:29 smithi136 ceph-mon[109861]: pgmap v2086: 89 pgs: 89 active+clean; 1.6 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 160 MiB/s wr, 2.56k op/s 2024-08-17T17:53:29.954 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:29 smithi136 ceph-mon[115078]: pgmap v2086: 89 pgs: 89 active+clean; 1.6 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 160 MiB/s wr, 2.56k op/s 2024-08-17T17:53:29.955 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:29.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:29.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:30.010 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:53:30.010 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-17T17:53:30.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-17T17:53:30.010 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T17:53:30.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.010 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.44783.0:3271 51.1a 51:5c67c520:::rbd_object_map.aeefc272e7cc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1111)", 2024-08-17T17:53:30.010 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:47:16.692853+0000", 2024-08-17T17:53:30.011 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 372.39285264400002, 2024-08-17T17:53:30.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.051535495000000001, 2024-08-17T17:53:30.011 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:53:30.011 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:53:30.011 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:53:30.011 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:53:30.011 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.44783", 2024-08-17T17:53:30.011 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3691373571", 2024-08-17T17:53:30.011 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 3271 2024-08-17T17:53:30.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.011 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:53:30.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:53:30.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.692853+0000", 2024-08-17T17:53:30.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:53:30.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.692853+0000", 2024-08-17T17:53:30.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:53:30.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.692853+0000", 2024-08-17T17:53:30.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7100000000000001e-07 2024-08-17T17:53:30.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:53:30.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.692858+0000", 2024-08-17T17:53:30.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4859999999999999e-06 2024-08-17T17:53:30.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:53:30.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.692866+0000", 2024-08-17T17:53:30.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1810000000000001e-06 2024-08-17T17:53:30.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:53:30.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.692871+0000", 2024-08-17T17:53:30.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8239999999999999e-06 2024-08-17T17:53:30.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:53:30.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.701337+0000", 2024-08-17T17:53:30.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0084662959999999999 2024-08-17T17:53:30.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:53:30.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.701503+0000", 2024-08-17T17:53:30.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016545500000000001 2024-08-17T17:53:30.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6,7", 2024-08-17T17:53:30.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.701856+0000", 2024-08-17T17:53:30.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000352868 2024-08-17T17:53:30.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:53:30.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.702720+0000", 2024-08-17T17:53:30.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00086482200000000001 2024-08-17T17:53:30.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.728872+0000", 2024-08-17T17:53:30.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.026151490999999999 2024-08-17T17:53:30.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.744229+0000", 2024-08-17T17:53:30.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015357478000000001 2024-08-17T17:53:30.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:53:30.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.744337+0000", 2024-08-17T17:53:30.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000108059 2024-08-17T17:53:30.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:53:30.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.744388+0000", 2024-08-17T17:53:30.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0864e-05 2024-08-17T17:53:30.017 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.017 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:53:30.017 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.017 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.44783.0:3275 51.13 51:c92463f4:::rbd_data.aeefc272e7cc.0000000000000034:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~3817080] snapc 6a=[6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1111)", 2024-08-17T17:53:30.017 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:47:16.696150+0000", 2024-08-17T17:53:30.018 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 372.38955555899997, 2024-08-17T17:53:30.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048283611999999997, 2024-08-17T17:53:30.018 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:53:30.018 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:53:30.018 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:53:30.018 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:53:30.018 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.44783", 2024-08-17T17:53:30.018 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3691373571", 2024-08-17T17:53:30.018 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 3275 2024-08-17T17:53:30.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.018 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:53:30.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:53:30.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.696150+0000", 2024-08-17T17:53:30.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:53:30.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.696150+0000", 2024-08-17T17:53:30.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:53:30.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.696151+0000", 2024-08-17T17:53:30.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3519999999999999e-06 2024-08-17T17:53:30.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:53:30.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.699926+0000", 2024-08-17T17:53:30.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037748270000000001 2024-08-17T17:53:30.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:53:30.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.699940+0000", 2024-08-17T17:53:30.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3644e-05 2024-08-17T17:53:30.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:53:30.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.699948+0000", 2024-08-17T17:53:30.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0700000000000007e-06 2024-08-17T17:53:30.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:53:30.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.700167+0000", 2024-08-17T17:53:30.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000218867 2024-08-17T17:53:30.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:53:30.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.701003+0000", 2024-08-17T17:53:30.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00083655899999999996 2024-08-17T17:53:30.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:53:30.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.701165+0000", 2024-08-17T17:53:30.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000162058 2024-08-17T17:53:30.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:53:30.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.706346+0000", 2024-08-17T17:53:30.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0051803609999999996 2024-08-17T17:53:30.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.709537+0000", 2024-08-17T17:53:30.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031916819999999999 2024-08-17T17:53:30.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.744364+0000", 2024-08-17T17:53:30.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034826374 2024-08-17T17:53:30.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:53:30.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.744395+0000", 2024-08-17T17:53:30.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1146000000000003e-05 2024-08-17T17:53:30.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:53:30.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.744433+0000", 2024-08-17T17:53:30.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8671999999999998e-05 2024-08-17T17:53:30.024 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.024 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:53:30.024 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.024 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.44783.0:3277 51.b 51:d223e004:::rbd_data.aeefc272e7cc.000000000000000d:head [write 0~994428] snapc 6a=[6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1111)", 2024-08-17T17:53:30.024 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:47:16.700427+0000", 2024-08-17T17:53:30.024 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 372.385278836, 2024-08-17T17:53:30.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.043990329000000002, 2024-08-17T17:53:30.025 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:53:30.025 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:53:30.025 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:53:30.025 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:53:30.025 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.44783", 2024-08-17T17:53:30.025 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3691373571", 2024-08-17T17:53:30.025 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 3277 2024-08-17T17:53:30.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.025 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:53:30.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:53:30.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.700427+0000", 2024-08-17T17:53:30.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:53:30.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.700427+0000", 2024-08-17T17:53:30.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:53:30.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.700428+0000", 2024-08-17T17:53:30.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.877e-06 2024-08-17T17:53:30.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:53:30.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.701105+0000", 2024-08-17T17:53:30.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000676485 2024-08-17T17:53:30.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:53:30.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.701117+0000", 2024-08-17T17:53:30.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2513e-05 2024-08-17T17:53:30.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:53:30.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.701127+0000", 2024-08-17T17:53:30.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9459999999999993e-06 2024-08-17T17:53:30.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:53:30.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.701189+0000", 2024-08-17T17:53:30.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1801e-05 2024-08-17T17:53:30.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:53:30.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.701278+0000", 2024-08-17T17:53:30.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8751000000000005e-05 2024-08-17T17:53:30.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,6", 2024-08-17T17:53:30.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.701457+0000", 2024-08-17T17:53:30.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000179254 2024-08-17T17:53:30.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:53:30.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.704569+0000", 2024-08-17T17:53:30.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003111918 2024-08-17T17:53:30.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.728808+0000", 2024-08-17T17:53:30.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.024238919000000001 2024-08-17T17:53:30.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.744324+0000", 2024-08-17T17:53:30.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015516301 2024-08-17T17:53:30.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:53:30.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.744375+0000", 2024-08-17T17:53:30.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0487e-05 2024-08-17T17:53:30.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:53:30.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:16.744417+0000", 2024-08-17T17:53:30.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2076999999999997e-05 2024-08-17T17:53:30.030 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.031 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:53:30.031 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.031 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.44783.0:3394 51.1a 51:5b2b8ec2:::rbd_data.aeefc272e7cc.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1327099~2867205] snapc 6c=[6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1113)", 2024-08-17T17:53:30.031 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:47:18.708097+0000", 2024-08-17T17:53:30.031 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 370.377608275, 2024-08-17T17:53:30.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.043964185000000003, 2024-08-17T17:53:30.031 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:53:30.031 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:53:30.031 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:53:30.031 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:53:30.031 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.44783", 2024-08-17T17:53:30.031 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3691373571", 2024-08-17T17:53:30.032 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 3394 2024-08-17T17:53:30.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.032 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:53:30.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:53:30.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:18.708097+0000", 2024-08-17T17:53:30.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:53:30.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:18.708097+0000", 2024-08-17T17:53:30.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:53:30.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:18.708099+0000", 2024-08-17T17:53:30.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.719e-06 2024-08-17T17:53:30.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:53:30.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:18.711019+0000", 2024-08-17T17:53:30.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0029200150000000002 2024-08-17T17:53:30.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:53:30.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:18.711032+0000", 2024-08-17T17:53:30.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3222e-05 2024-08-17T17:53:30.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:53:30.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:18.711041+0000", 2024-08-17T17:53:30.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7660000000000004e-06 2024-08-17T17:53:30.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:53:30.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:18.716661+0000", 2024-08-17T17:53:30.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0056205489999999999 2024-08-17T17:53:30.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:53:30.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:18.716834+0000", 2024-08-17T17:53:30.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000172356 2024-08-17T17:53:30.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6,7", 2024-08-17T17:53:30.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:18.716987+0000", 2024-08-17T17:53:30.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000152912 2024-08-17T17:53:30.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:18.733282+0000", 2024-08-17T17:53:30.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016295401000000001 2024-08-17T17:53:30.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:53:30.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:18.733294+0000", 2024-08-17T17:53:30.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1858000000000001e-05 2024-08-17T17:53:30.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:18.751984+0000", 2024-08-17T17:53:30.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018689992999999998 2024-08-17T17:53:30.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:53:30.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:18.752022+0000", 2024-08-17T17:53:30.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8000000000000002e-05 2024-08-17T17:53:30.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:53:30.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:47:18.752061+0000", 2024-08-17T17:53:30.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9394000000000003e-05 2024-08-17T17:53:30.037 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.037 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:53:30.037 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.037 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:21636 53.5 53:a7657ccb:::benchmark_data_smithi136_235573_object21635:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:53:30.037 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:24.546494+0000", 2024-08-17T17:53:30.037 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.5392117179999998, 2024-08-17T17:53:30.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.053937172999999998, 2024-08-17T17:53:30.037 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:53:30.038 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:53:30.038 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:53:30.038 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:53:30.038 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:53:30.038 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:53:30.038 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 21636 2024-08-17T17:53:30.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.038 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:53:30.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:53:30.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.546494+0000", 2024-08-17T17:53:30.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:53:30.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.546494+0000", 2024-08-17T17:53:30.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:53:30.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.546495+0000", 2024-08-17T17:53:30.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3120000000000001e-06 2024-08-17T17:53:30.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:53:30.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.546531+0000", 2024-08-17T17:53:30.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6307999999999999e-05 2024-08-17T17:53:30.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:53:30.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.546541+0000", 2024-08-17T17:53:30.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9550000000000004e-06 2024-08-17T17:53:30.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:53:30.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.546548+0000", 2024-08-17T17:53:30.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3330000000000002e-06 2024-08-17T17:53:30.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:53:30.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.547109+0000", 2024-08-17T17:53:30.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00056117099999999996 2024-08-17T17:53:30.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:53:30.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.547224+0000", 2024-08-17T17:53:30.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011476600000000001 2024-08-17T17:53:30.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:53:30.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.547353+0000", 2024-08-17T17:53:30.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129722 2024-08-17T17:53:30.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:53:30.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.551562+0000", 2024-08-17T17:53:30.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0042087740000000002 2024-08-17T17:53:30.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.551742+0000", 2024-08-17T17:53:30.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018017200000000001 2024-08-17T17:53:30.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.600328+0000", 2024-08-17T17:53:30.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048585362 2024-08-17T17:53:30.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:53:30.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.600383+0000", 2024-08-17T17:53:30.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5067e-05 2024-08-17T17:53:30.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:53:30.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.600431+0000", 2024-08-17T17:53:30.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8231e-05 2024-08-17T17:53:30.043 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.043 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:53:30.043 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.043 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:21637 53.5 53:a21d0903:::benchmark_data_smithi136_235573_object21636:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:53:30.043 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:24.546561+0000", 2024-08-17T17:53:30.044 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.5391445050000003, 2024-08-17T17:53:30.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055768690000000003, 2024-08-17T17:53:30.044 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:53:30.044 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:53:30.044 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:53:30.044 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:53:30.044 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:53:30.044 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:53:30.044 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 21637 2024-08-17T17:53:30.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.044 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:53:30.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:53:30.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.546561+0000", 2024-08-17T17:53:30.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:53:30.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.546561+0000", 2024-08-17T17:53:30.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:53:30.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.546561+0000", 2024-08-17T17:53:30.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.75e-07 2024-08-17T17:53:30.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:53:30.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.546594+0000", 2024-08-17T17:53:30.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2356e-05 2024-08-17T17:53:30.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:53:30.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.546600+0000", 2024-08-17T17:53:30.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6449999999999999e-06 2024-08-17T17:53:30.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:53:30.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.546605+0000", 2024-08-17T17:53:30.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4240000000000003e-06 2024-08-17T17:53:30.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:53:30.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.548281+0000", 2024-08-17T17:53:30.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016760830000000001 2024-08-17T17:53:30.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:53:30.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.548396+0000", 2024-08-17T17:53:30.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115466 2024-08-17T17:53:30.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:53:30.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.548512+0000", 2024-08-17T17:53:30.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011539099999999999 2024-08-17T17:53:30.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:53:30.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.551568+0000", 2024-08-17T17:53:30.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0030560679999999999 2024-08-17T17:53:30.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.551780+0000", 2024-08-17T17:53:30.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021221900000000001 2024-08-17T17:53:30.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.602221+0000", 2024-08-17T17:53:30.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050441208000000001 2024-08-17T17:53:30.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:53:30.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.602283+0000", 2024-08-17T17:53:30.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1377000000000005e-05 2024-08-17T17:53:30.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:53:30.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.602330+0000", 2024-08-17T17:53:30.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6978000000000003e-05 2024-08-17T17:53:30.049 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.049 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:53:30.049 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.049 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:21638 53.5 53:adca2665:::benchmark_data_smithi136_235573_object21637:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:53:30.050 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:24.546614+0000", 2024-08-17T17:53:30.050 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.5390916969999999, 2024-08-17T17:53:30.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055850274999999998, 2024-08-17T17:53:30.050 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:53:30.050 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:53:30.050 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:53:30.050 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:53:30.050 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:53:30.050 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:53:30.050 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 21638 2024-08-17T17:53:30.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.050 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:53:30.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:53:30.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.546614+0000", 2024-08-17T17:53:30.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:53:30.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.546614+0000", 2024-08-17T17:53:30.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:53:30.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.546614+0000", 2024-08-17T17:53:30.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1600000000000001e-07 2024-08-17T17:53:30.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:53:30.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.546634+0000", 2024-08-17T17:53:30.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9896000000000001e-05 2024-08-17T17:53:30.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:53:30.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.546642+0000", 2024-08-17T17:53:30.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4440000000000002e-06 2024-08-17T17:53:30.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:53:30.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.546647+0000", 2024-08-17T17:53:30.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6910000000000002e-06 2024-08-17T17:53:30.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:53:30.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.549148+0000", 2024-08-17T17:53:30.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025007969999999999 2024-08-17T17:53:30.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:53:30.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.549247+0000", 2024-08-17T17:53:30.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8907999999999996e-05 2024-08-17T17:53:30.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:53:30.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.549362+0000", 2024-08-17T17:53:30.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115077 2024-08-17T17:53:30.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:53:30.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.551572+0000", 2024-08-17T17:53:30.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0022097559999999998 2024-08-17T17:53:30.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.551807+0000", 2024-08-17T17:53:30.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023563800000000001 2024-08-17T17:53:30.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.602375+0000", 2024-08-17T17:53:30.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050567782999999998 2024-08-17T17:53:30.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:53:30.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.602429+0000", 2024-08-17T17:53:30.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3718999999999998e-05 2024-08-17T17:53:30.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:53:30.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.602464+0000", 2024-08-17T17:53:30.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5049999999999998e-05 2024-08-17T17:53:30.055 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.055 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:53:30.056 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.056 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:21642 53.5 53:aad113b1:::benchmark_data_smithi136_235573_object21641:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:53:30.056 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:24.547004+0000", 2024-08-17T17:53:30.056 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.5387012200000001, 2024-08-17T17:53:30.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055591252000000001, 2024-08-17T17:53:30.056 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:53:30.056 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:53:30.056 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:53:30.056 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:53:30.056 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:53:30.056 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:53:30.056 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 21642 2024-08-17T17:53:30.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.057 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:53:30.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:53:30.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.547004+0000", 2024-08-17T17:53:30.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:53:30.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.547004+0000", 2024-08-17T17:53:30.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:53:30.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.547005+0000", 2024-08-17T17:53:30.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.54e-07 2024-08-17T17:53:30.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:53:30.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.547039+0000", 2024-08-17T17:53:30.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4489999999999997e-05 2024-08-17T17:53:30.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:53:30.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.547059+0000", 2024-08-17T17:53:30.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9531e-05 2024-08-17T17:53:30.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:53:30.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.547068+0000", 2024-08-17T17:53:30.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.346e-06 2024-08-17T17:53:30.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:53:30.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.550316+0000", 2024-08-17T17:53:30.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032481229999999999 2024-08-17T17:53:30.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:53:30.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.550393+0000", 2024-08-17T17:53:30.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6573e-05 2024-08-17T17:53:30.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:53:30.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.550473+0000", 2024-08-17T17:53:30.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9862000000000002e-05 2024-08-17T17:53:30.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:53:30.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.551576+0000", 2024-08-17T17:53:30.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001102708 2024-08-17T17:53:30.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.551986+0000", 2024-08-17T17:53:30.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00041063500000000001 2024-08-17T17:53:30.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.602514+0000", 2024-08-17T17:53:30.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050527508999999998 2024-08-17T17:53:30.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:53:30.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.602559+0000", 2024-08-17T17:53:30.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4863000000000001e-05 2024-08-17T17:53:30.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:53:30.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.602595+0000", 2024-08-17T17:53:30.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6857999999999998e-05 2024-08-17T17:53:30.064 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.064 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:53:30.064 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.064 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:21658 53.5 53:a5d616aa:::benchmark_data_smithi136_235573_object21657:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:53:30.065 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:24.554049+0000", 2024-08-17T17:53:30.065 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.5316560619999997, 2024-08-17T17:53:30.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048673014000000001, 2024-08-17T17:53:30.065 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:53:30.065 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:53:30.065 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:53:30.065 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:53:30.065 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:53:30.065 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:53:30.065 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 21658 2024-08-17T17:53:30.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.065 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:53:30.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:53:30.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.554049+0000", 2024-08-17T17:53:30.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:53:30.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.554049+0000", 2024-08-17T17:53:30.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:53:30.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.554051+0000", 2024-08-17T17:53:30.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.418e-06 2024-08-17T17:53:30.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:53:30.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.554075+0000", 2024-08-17T17:53:30.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4193e-05 2024-08-17T17:53:30.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:53:30.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.554087+0000", 2024-08-17T17:53:30.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1999e-05 2024-08-17T17:53:30.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:53:30.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.554094+0000", 2024-08-17T17:53:30.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0720000000000002e-06 2024-08-17T17:53:30.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:53:30.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.554174+0000", 2024-08-17T17:53:30.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0239000000000002e-05 2024-08-17T17:53:30.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:53:30.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.554295+0000", 2024-08-17T17:53:30.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121062 2024-08-17T17:53:30.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:53:30.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.554425+0000", 2024-08-17T17:53:30.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012963100000000001 2024-08-17T17:53:30.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:53:30.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.555471+0000", 2024-08-17T17:53:30.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010463829999999999 2024-08-17T17:53:30.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.556086+0000", 2024-08-17T17:53:30.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00061486099999999997 2024-08-17T17:53:30.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.602640+0000", 2024-08-17T17:53:30.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.046553612000000001 2024-08-17T17:53:30.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:53:30.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.602680+0000", 2024-08-17T17:53:30.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0605000000000002e-05 2024-08-17T17:53:30.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:53:30.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:24.602722+0000", 2024-08-17T17:53:30.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1938999999999999e-05 2024-08-17T17:53:30.070 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.070 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:53:30.070 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.070 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:25151 53.5 53:a899085d:::benchmark_data_smithi136_235573_object25150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:53:30.071 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:25.914937+0000", 2024-08-17T17:53:30.071 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 3.1707683630000001, 2024-08-17T17:53:30.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.088361351000000005, 2024-08-17T17:53:30.071 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:53:30.071 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:53:30.071 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:53:30.071 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:53:30.071 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:53:30.071 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:53:30.071 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 25151 2024-08-17T17:53:30.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.071 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:53:30.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:53:30.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:25.914937+0000", 2024-08-17T17:53:30.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:53:30.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:25.914937+0000", 2024-08-17T17:53:30.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:53:30.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:25.914939+0000", 2024-08-17T17:53:30.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.641e-06 2024-08-17T17:53:30.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:53:30.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:25.914962+0000", 2024-08-17T17:53:30.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3266999999999999e-05 2024-08-17T17:53:30.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:53:30.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:25.914977+0000", 2024-08-17T17:53:30.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5412000000000001e-05 2024-08-17T17:53:30.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:53:30.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:25.914984+0000", 2024-08-17T17:53:30.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4919999999999998e-06 2024-08-17T17:53:30.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:53:30.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:25.915034+0000", 2024-08-17T17:53:30.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0053999999999999e-05 2024-08-17T17:53:30.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:53:30.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:25.915149+0000", 2024-08-17T17:53:30.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115124 2024-08-17T17:53:30.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:53:30.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:25.915268+0000", 2024-08-17T17:53:30.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011910599999999999 2024-08-17T17:53:30.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:53:30.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:25.916416+0000", 2024-08-17T17:53:30.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011482619999999999 2024-08-17T17:53:30.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:25.916650+0000", 2024-08-17T17:53:30.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000233661 2024-08-17T17:53:30.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:26.003169+0000", 2024-08-17T17:53:30.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.086519150000000003 2024-08-17T17:53:30.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:53:30.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:26.003233+0000", 2024-08-17T17:53:30.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4015000000000003e-05 2024-08-17T17:53:30.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:53:30.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:26.003298+0000", 2024-08-17T17:53:30.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5166999999999995e-05 2024-08-17T17:53:30.076 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.076 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:53:30.076 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.077 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:28079 53.5 53:a0b08868:::benchmark_data_smithi136_235573_object28078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:53:30.077 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:27.301226+0000", 2024-08-17T17:53:30.077 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 1.7844796970000001, 2024-08-17T17:53:30.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.045316169000000003, 2024-08-17T17:53:30.077 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:53:30.077 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:53:30.077 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:53:30.077 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:53:30.077 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:53:30.077 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:53:30.077 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 28079 2024-08-17T17:53:30.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.077 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:53:30.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:53:30.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.301226+0000", 2024-08-17T17:53:30.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:53:30.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.301226+0000", 2024-08-17T17:53:30.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:53:30.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.301227+0000", 2024-08-17T17:53:30.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.331e-06 2024-08-17T17:53:30.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:53:30.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.301245+0000", 2024-08-17T17:53:30.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7866000000000001e-05 2024-08-17T17:53:30.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:53:30.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.301257+0000", 2024-08-17T17:53:30.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2058999999999999e-05 2024-08-17T17:53:30.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:53:30.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.301264+0000", 2024-08-17T17:53:30.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1330000000000001e-06 2024-08-17T17:53:30.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:53:30.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.301303+0000", 2024-08-17T17:53:30.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8435e-05 2024-08-17T17:53:30.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:53:30.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.301381+0000", 2024-08-17T17:53:30.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8339999999999999e-05 2024-08-17T17:53:30.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:53:30.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.301470+0000", 2024-08-17T17:53:30.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8720000000000004e-05 2024-08-17T17:53:30.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:53:30.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.302399+0000", 2024-08-17T17:53:30.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00092891100000000004 2024-08-17T17:53:30.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.302708+0000", 2024-08-17T17:53:30.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00030927899999999998 2024-08-17T17:53:30.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.346421+0000", 2024-08-17T17:53:30.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.043713488000000002 2024-08-17T17:53:30.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:53:30.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.346484+0000", 2024-08-17T17:53:30.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3108999999999995e-05 2024-08-17T17:53:30.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:53:30.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.346542+0000", 2024-08-17T17:53:30.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7497999999999997e-05 2024-08-17T17:53:30.083 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.083 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:53:30.083 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.083 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:28587 53.5 53:ac0070c4:::benchmark_data_smithi136_235573_object28586:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:53:30.083 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:27.543468+0000", 2024-08-17T17:53:30.083 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 1.542237793, 2024-08-17T17:53:30.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.064193380999999994, 2024-08-17T17:53:30.083 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:53:30.083 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:53:30.083 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:53:30.083 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:53:30.084 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:53:30.084 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:53:30.084 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 28587 2024-08-17T17:53:30.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.084 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:53:30.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:53:30.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.543468+0000", 2024-08-17T17:53:30.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:53:30.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.543468+0000", 2024-08-17T17:53:30.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:53:30.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.543469+0000", 2024-08-17T17:53:30.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.421e-06 2024-08-17T17:53:30.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:53:30.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.543491+0000", 2024-08-17T17:53:30.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2169999999999999e-05 2024-08-17T17:53:30.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:53:30.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.543506+0000", 2024-08-17T17:53:30.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5142e-05 2024-08-17T17:53:30.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:53:30.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.543515+0000", 2024-08-17T17:53:30.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6170000000000007e-06 2024-08-17T17:53:30.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:53:30.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.543771+0000", 2024-08-17T17:53:30.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025644699999999998 2024-08-17T17:53:30.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:53:30.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.543895+0000", 2024-08-17T17:53:30.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123774 2024-08-17T17:53:30.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:53:30.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.544028+0000", 2024-08-17T17:53:30.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013240699999999999 2024-08-17T17:53:30.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:53:30.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.547574+0000", 2024-08-17T17:53:30.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003546652 2024-08-17T17:53:30.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.548539+0000", 2024-08-17T17:53:30.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00096468900000000002 2024-08-17T17:53:30.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.607540+0000", 2024-08-17T17:53:30.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.059001285000000001 2024-08-17T17:53:30.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:53:30.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.607612+0000", 2024-08-17T17:53:30.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1313e-05 2024-08-17T17:53:30.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:53:30.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.607661+0000", 2024-08-17T17:53:30.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9463999999999999e-05 2024-08-17T17:53:30.089 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.089 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:53:30.089 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.089 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:28589 53.5 53:a9b52e90:::benchmark_data_smithi136_235573_object28588:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:53:30.089 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:27.543539+0000", 2024-08-17T17:53:30.089 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 1.54216621, 2024-08-17T17:53:30.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.066471214000000001, 2024-08-17T17:53:30.089 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:53:30.089 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:53:30.089 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:53:30.089 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:53:30.089 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:53:30.089 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:53:30.090 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 28589 2024-08-17T17:53:30.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.090 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:53:30.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:53:30.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.543539+0000", 2024-08-17T17:53:30.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:53:30.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.543539+0000", 2024-08-17T17:53:30.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:53:30.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.543540+0000", 2024-08-17T17:53:30.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3099999999999998e-07 2024-08-17T17:53:30.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:53:30.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.543561+0000", 2024-08-17T17:53:30.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1338000000000001e-05 2024-08-17T17:53:30.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:53:30.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.543570+0000", 2024-08-17T17:53:30.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0170000000000002e-06 2024-08-17T17:53:30.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:53:30.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.543576+0000", 2024-08-17T17:53:30.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8209999999999998e-06 2024-08-17T17:53:30.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:53:30.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.546183+0000", 2024-08-17T17:53:30.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0026068409999999999 2024-08-17T17:53:30.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:53:30.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.546371+0000", 2024-08-17T17:53:30.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000188149 2024-08-17T17:53:30.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:53:30.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.546515+0000", 2024-08-17T17:53:30.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014384700000000001 2024-08-17T17:53:30.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:53:30.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.548271+0000", 2024-08-17T17:53:30.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017557829999999999 2024-08-17T17:53:30.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.548877+0000", 2024-08-17T17:53:30.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00060634399999999998 2024-08-17T17:53:30.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.609905+0000", 2024-08-17T17:53:30.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.061027763999999998 2024-08-17T17:53:30.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:53:30.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.609961+0000", 2024-08-17T17:53:30.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6365e-05 2024-08-17T17:53:30.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:53:30.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.610010+0000", 2024-08-17T17:53:30.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9413999999999998e-05 2024-08-17T17:53:30.095 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.095 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:53:30.095 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.095 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:28607 53.5 53:a337b7e6:::benchmark_data_smithi136_235573_object28606:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:53:30.095 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:27.549646+0000", 2024-08-17T17:53:30.095 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 1.5360599000000001, 2024-08-17T17:53:30.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.060565964999999999, 2024-08-17T17:53:30.095 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:53:30.095 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:53:30.095 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:53:30.095 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:53:30.096 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:53:30.096 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:53:30.096 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 28607 2024-08-17T17:53:30.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.096 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:53:30.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:53:30.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.549646+0000", 2024-08-17T17:53:30.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:53:30.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.549646+0000", 2024-08-17T17:53:30.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:53:30.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.549646+0000", 2024-08-17T17:53:30.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4899999999999995e-07 2024-08-17T17:53:30.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:53:30.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.549665+0000", 2024-08-17T17:53:30.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9066e-05 2024-08-17T17:53:30.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:53:30.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.549673+0000", 2024-08-17T17:53:30.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1559999999999995e-06 2024-08-17T17:53:30.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:53:30.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.549679+0000", 2024-08-17T17:53:30.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000004e-06 2024-08-17T17:53:30.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:53:30.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.549737+0000", 2024-08-17T17:53:30.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7807999999999999e-05 2024-08-17T17:53:30.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:53:30.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.549854+0000", 2024-08-17T17:53:30.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011665100000000001 2024-08-17T17:53:30.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:53:30.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.549973+0000", 2024-08-17T17:53:30.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011955600000000001 2024-08-17T17:53:30.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:53:30.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.551096+0000", 2024-08-17T17:53:30.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011233040000000001 2024-08-17T17:53:30.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.603794+0000", 2024-08-17T17:53:30.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.052697932000000003 2024-08-17T17:53:30.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.610102+0000", 2024-08-17T17:53:30.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063076850000000004 2024-08-17T17:53:30.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:53:30.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.610165+0000", 2024-08-17T17:53:30.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2689999999999998e-05 2024-08-17T17:53:30.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:53:30.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.610211+0000", 2024-08-17T17:53:30.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6758000000000002e-05 2024-08-17T17:53:30.101 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.101 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:53:30.101 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.101 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:28611 53.5 53:a2a6911c:::benchmark_data_smithi136_235573_object28610:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:53:30.101 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:27.553535+0000", 2024-08-17T17:53:30.101 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 1.532170445, 2024-08-17T17:53:30.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.06220647, 2024-08-17T17:53:30.101 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:53:30.101 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:53:30.101 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:53:30.101 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:53:30.101 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:53:30.102 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:53:30.102 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 28611 2024-08-17T17:53:30.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.102 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:53:30.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:53:30.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.553535+0000", 2024-08-17T17:53:30.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:53:30.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.553535+0000", 2024-08-17T17:53:30.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:53:30.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.553536+0000", 2024-08-17T17:53:30.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7700000000000003e-07 2024-08-17T17:53:30.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:53:30.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.553560+0000", 2024-08-17T17:53:30.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3980000000000001e-05 2024-08-17T17:53:30.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:53:30.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.553567+0000", 2024-08-17T17:53:30.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.092e-06 2024-08-17T17:53:30.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:53:30.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.553571+0000", 2024-08-17T17:53:30.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3200000000000001e-06 2024-08-17T17:53:30.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:53:30.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.553603+0000", 2024-08-17T17:53:30.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1970000000000001e-05 2024-08-17T17:53:30.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:53:30.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.553679+0000", 2024-08-17T17:53:30.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6135000000000002e-05 2024-08-17T17:53:30.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:53:30.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.553765+0000", 2024-08-17T17:53:30.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5511000000000002e-05 2024-08-17T17:53:30.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:53:30.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.554594+0000", 2024-08-17T17:53:30.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00082866700000000005 2024-08-17T17:53:30.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.604374+0000", 2024-08-17T17:53:30.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.049780469000000001 2024-08-17T17:53:30.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.615640+0000", 2024-08-17T17:53:30.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011266135 2024-08-17T17:53:30.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:53:30.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.615702+0000", 2024-08-17T17:53:30.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1933999999999994e-05 2024-08-17T17:53:30.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:53:30.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.615741+0000", 2024-08-17T17:53:30.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9379999999999999e-05 2024-08-17T17:53:30.107 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.107 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:53:30.107 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.107 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:29174 53.5 53:aa3538a2:::benchmark_data_smithi136_235573_object29173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:53:30.107 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:27.852565+0000", 2024-08-17T17:53:30.107 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 1.23314068, 2024-08-17T17:53:30.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.074549589999999999, 2024-08-17T17:53:30.107 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:53:30.107 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:53:30.108 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:53:30.108 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:53:30.108 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:53:30.108 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:53:30.108 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 29174 2024-08-17T17:53:30.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.108 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:53:30.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:53:30.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.852565+0000", 2024-08-17T17:53:30.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:53:30.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.852565+0000", 2024-08-17T17:53:30.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:53:30.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.852567+0000", 2024-08-17T17:53:30.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9759999999999998e-06 2024-08-17T17:53:30.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:53:30.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.852591+0000", 2024-08-17T17:53:30.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4456000000000001e-05 2024-08-17T17:53:30.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:53:30.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.852616+0000", 2024-08-17T17:53:30.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5282e-05 2024-08-17T17:53:30.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:53:30.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.852635+0000", 2024-08-17T17:53:30.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8184000000000001e-05 2024-08-17T17:53:30.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:53:30.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.852804+0000", 2024-08-17T17:53:30.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000169407 2024-08-17T17:53:30.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:53:30.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.852928+0000", 2024-08-17T17:53:30.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012404000000000001 2024-08-17T17:53:30.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:53:30.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.853094+0000", 2024-08-17T17:53:30.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016632800000000001 2024-08-17T17:53:30.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:53:30.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.856633+0000", 2024-08-17T17:53:30.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035389179999999998 2024-08-17T17:53:30.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.858242+0000", 2024-08-17T17:53:30.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016083149999999999 2024-08-17T17:53:30.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.927025+0000", 2024-08-17T17:53:30.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.068783026999999997 2024-08-17T17:53:30.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:53:30.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.927074+0000", 2024-08-17T17:53:30.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8881999999999997e-05 2024-08-17T17:53:30.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:53:30.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.927114+0000", 2024-08-17T17:53:30.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0775000000000002e-05 2024-08-17T17:53:30.113 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.113 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:53:30.113 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.113 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:29176 53.5 53:a9c72d3f:::benchmark_data_smithi136_235573_object29175:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:53:30.113 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:27.852659+0000", 2024-08-17T17:53:30.113 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 1.2330464560000001, 2024-08-17T17:53:30.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.074628297999999996, 2024-08-17T17:53:30.113 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:53:30.113 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:53:30.113 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:53:30.114 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:53:30.114 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:53:30.114 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:53:30.114 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 29176 2024-08-17T17:53:30.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.114 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:53:30.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:53:30.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.852659+0000", 2024-08-17T17:53:30.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:53:30.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.852659+0000", 2024-08-17T17:53:30.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:53:30.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.852659+0000", 2024-08-17T17:53:30.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1600000000000001e-07 2024-08-17T17:53:30.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:53:30.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.852681+0000", 2024-08-17T17:53:30.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1223e-05 2024-08-17T17:53:30.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:53:30.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.852689+0000", 2024-08-17T17:53:30.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1540000000000002e-06 2024-08-17T17:53:30.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:53:30.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.852710+0000", 2024-08-17T17:53:30.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1280999999999998e-05 2024-08-17T17:53:30.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:53:30.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.854180+0000", 2024-08-17T17:53:30.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014699699999999999 2024-08-17T17:53:30.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:53:30.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.854305+0000", 2024-08-17T17:53:30.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000124671 2024-08-17T17:53:30.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:53:30.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.854430+0000", 2024-08-17T17:53:30.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125463 2024-08-17T17:53:30.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:53:30.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.856638+0000", 2024-08-17T17:53:30.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0022073459999999998 2024-08-17T17:53:30.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.858331+0000", 2024-08-17T17:53:30.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001693478 2024-08-17T17:53:30.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.927195+0000", 2024-08-17T17:53:30.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.068863606999999993 2024-08-17T17:53:30.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:53:30.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.927245+0000", 2024-08-17T17:53:30.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0599999999999997e-05 2024-08-17T17:53:30.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:53:30.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.927287+0000", 2024-08-17T17:53:30.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1989000000000001e-05 2024-08-17T17:53:30.119 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.119 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:53:30.119 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.119 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:29177 53.5 53:adc2085e:::benchmark_data_smithi136_235573_object29176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:53:30.119 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:27.852721+0000", 2024-08-17T17:53:30.119 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 1.2329846790000001, 2024-08-17T17:53:30.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.075662573999999996, 2024-08-17T17:53:30.119 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:53:30.119 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:53:30.119 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:53:30.119 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:53:30.119 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:53:30.119 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:53:30.120 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 29177 2024-08-17T17:53:30.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.120 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:53:30.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:53:30.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.852721+0000", 2024-08-17T17:53:30.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:53:30.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.852721+0000", 2024-08-17T17:53:30.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:53:30.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.852721+0000", 2024-08-17T17:53:30.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5000000000000003e-07 2024-08-17T17:53:30.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:53:30.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.852745+0000", 2024-08-17T17:53:30.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4071e-05 2024-08-17T17:53:30.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:53:30.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.852753+0000", 2024-08-17T17:53:30.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2180000000000002e-06 2024-08-17T17:53:30.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:53:30.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.852758+0000", 2024-08-17T17:53:30.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9989999999999999e-06 2024-08-17T17:53:30.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:53:30.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.855407+0000", 2024-08-17T17:53:30.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0026497579999999999 2024-08-17T17:53:30.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:53:30.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.855516+0000", 2024-08-17T17:53:30.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000108937 2024-08-17T17:53:30.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:53:30.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.855634+0000", 2024-08-17T17:53:30.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117563 2024-08-17T17:53:30.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:53:30.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.856815+0000", 2024-08-17T17:53:30.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011813870000000001 2024-08-17T17:53:30.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.858383+0000", 2024-08-17T17:53:30.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015675870000000001 2024-08-17T17:53:30.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:53:30.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.928296+0000", 2024-08-17T17:53:30.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.069913575000000006 2024-08-17T17:53:30.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:53:30.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.928347+0000", 2024-08-17T17:53:30.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1078000000000001e-05 2024-08-17T17:53:30.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:53:30.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:27.928383+0000", 2024-08-17T17:53:30.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5850999999999999e-05 2024-08-17T17:53:30.124 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.125 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:53:30.125 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.125 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.35091.0:30533 53.8 e1307/1303 53:1f9fa07a:::benchmark_data_smithi136_235573_object30532:head v 1307'1933, mlcod=1307'1928)", 2024-08-17T17:53:30.125 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:28.479671+0000", 2024-08-17T17:53:30.125 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.606034615, 2024-08-17T17:53:30.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.046488044999999999, 2024-08-17T17:53:30.125 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:53:30.125 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:53:30.125 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:53:30.125 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:53:30.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:53:30.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:28.479671+0000", 2024-08-17T17:53:30.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:53:30.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:28.479671+0000", 2024-08-17T17:53:30.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:53:30.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:28.479671+0000", 2024-08-17T17:53:30.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8999999999999996e-07 2024-08-17T17:53:30.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:53:30.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:28.479777+0000", 2024-08-17T17:53:30.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010607000000000001 2024-08-17T17:53:30.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:53:30.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:28.524894+0000", 2024-08-17T17:53:30.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.045117013999999997 2024-08-17T17:53:30.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:53:30.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:28.524915+0000", 2024-08-17T17:53:30.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0934e-05 2024-08-17T17:53:30.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:53:30.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:28.524988+0000", 2024-08-17T17:53:30.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2837999999999996e-05 2024-08-17T17:53:30.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:53:30.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:28.525011+0000", 2024-08-17T17:53:30.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2271000000000001e-05 2024-08-17T17:53:30.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:53:30.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:28.526106+0000", 2024-08-17T17:53:30.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001095908 2024-08-17T17:53:30.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:53:30.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:28.526159+0000", 2024-08-17T17:53:30.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2420000000000003e-05 2024-08-17T17:53:30.129 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.129 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:53:30.129 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.129 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.35091.0:30535 53.2 e1307/1303 53:4bc98f42:::benchmark_data_smithi136_235573_object30534:head v 1307'1927, mlcod=1307'1924)", 2024-08-17T17:53:30.130 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:28.480027+0000", 2024-08-17T17:53:30.130 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.60567798299999998, 2024-08-17T17:53:30.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.046133570999999998, 2024-08-17T17:53:30.130 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:53:30.130 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:53:30.130 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:53:30.130 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:53:30.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:53:30.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:28.480027+0000", 2024-08-17T17:53:30.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:53:30.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:28.480027+0000", 2024-08-17T17:53:30.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:53:30.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:53:30.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:28.480028+0000", 2024-08-17T17:53:30.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0299999999999999e-07 2024-08-17T17:53:30.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:53:30.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:28.480152+0000", 2024-08-17T17:53:30.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000124401 2024-08-17T17:53:30.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:53:30.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:28.524911+0000", 2024-08-17T17:53:30.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.044758924999999998 2024-08-17T17:53:30.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:53:30.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:28.524917+0000", 2024-08-17T17:53:30.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2179999999999996e-06 2024-08-17T17:53:30.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:53:30.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:28.524986+0000", 2024-08-17T17:53:30.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8558000000000003e-05 2024-08-17T17:53:30.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:53:30.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:28.525010+0000", 2024-08-17T17:53:30.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4068999999999999e-05 2024-08-17T17:53:30.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:53:30.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:28.526106+0000", 2024-08-17T17:53:30.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001096278 2024-08-17T17:53:30.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:53:30.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:53:30.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:53:30.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:28.526161+0000", 2024-08-17T17:53:30.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4619e-05 2024-08-17T17:53:30.133 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.133 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:53:30.134 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.134 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:53:30.134 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:53:30.134 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:53:30.134 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:30.135 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:29 smithi165 ceph-mon[105517]: pgmap v2086: 89 pgs: 89 active+clean; 1.6 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 160 MiB/s wr, 2.56k op/s 2024-08-17T17:53:30.168 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:30.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:30.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:30.270 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:30.306 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:30.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:30.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:30.408 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:30.441 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:30.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:30.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:30.542 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:30.579 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:30.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:30.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:30.681 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:30.699 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 13 16 34608 34592 166.291 115.875 0.00612917 0.00600617 2024-08-17T17:53:30.709 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:30.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:30.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:30.810 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:30.834 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:30.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:30.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:30.935 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:30.964 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:30.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:30.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:31.067 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:31.097 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:31.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:31.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:31.199 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:31.236 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:31.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:31.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:31.337 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:31.369 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:31.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:31.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:31.471 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:31.499 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:31.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:31.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:31.601 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:31.634 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:31.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:31.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:31.700 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 14 16 36746 36730 163.957 133.625 0.00884229 0.00609436 2024-08-17T17:53:31.735 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:31.773 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:31.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:31.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:31.875 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:31.902 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:31 smithi136 ceph-mon[109861]: pgmap v2087: 89 pgs: 89 active+clean; 1.7 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 162 MiB/s wr, 2.60k op/s 2024-08-17T17:53:31.903 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:31 smithi136 ceph-mon[115078]: pgmap v2087: 89 pgs: 89 active+clean; 1.7 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 162 MiB/s wr, 2.60k op/s 2024-08-17T17:53:31.910 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:31.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:31.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:31 smithi165 ceph-mon[105517]: pgmap v2087: 89 pgs: 89 active+clean; 1.7 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 162 MiB/s wr, 2.60k op/s 2024-08-17T17:53:32.012 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:32.050 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:32.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:32.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:32.152 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:32.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:32.187 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:32.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:32.288 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:32.324 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:32.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:32.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:32.426 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:32.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:32.462 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:32.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:32.563 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:32.598 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:32.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:32.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:32.700 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 15 16 38922 38906 162.092 136 0.00895136 0.00616326 2024-08-17T17:53:32.700 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:32.716 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-17T17:53:32.728 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:32.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:32.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:32.829 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:32.870 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:32.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:32.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:32.972 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:33.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:33.014 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:33.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:33.115 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:33.145 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:33.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:33.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:33.247 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:33.282 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:33.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:33.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:33.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:33.418 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:33.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:33.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:33.520 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:33.551 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:33.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:33.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:33.653 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:33.700 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 16 16 40943 40927 159.847 126.312 0.00792428 0.00625113 2024-08-17T17:53:33.702 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:33.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:33.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:33.803 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:33.845 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:33.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:33.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:33.866 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:33 smithi136 ceph-mon[109861]: pgmap v2088: 89 pgs: 89 active+clean; 2.2 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 174 MiB/s wr, 2.79k op/s 2024-08-17T17:53:33.866 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:33 smithi136 ceph-mon[115078]: pgmap v2088: 89 pgs: 89 active+clean; 2.2 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 174 MiB/s wr, 2.79k op/s 2024-08-17T17:53:33.948 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:33.987 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:33.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:33.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:33 smithi165 ceph-mon[105517]: pgmap v2088: 89 pgs: 89 active+clean; 2.2 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 174 MiB/s wr, 2.79k op/s 2024-08-17T17:53:34.089 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:34.136 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:34.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:34.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:34.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:34.270 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:34.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:34.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:34.372 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:34.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:34.419 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:34.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:34.508 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:53:34.509 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T17:53:34.520 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:34.562 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:34.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:34.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:34.664 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:34.677 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T17:53:34.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:34 smithi136 ceph-mon[109861]: from='client.35103 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:53:34.746 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:34 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3926869030' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:53:34.746 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:34 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:53:34.746 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:34 smithi136 ceph-mon[115078]: from='client.35103 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:53:34.746 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:34 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3926869030' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:53:34.746 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:34 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:53:34.747 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 17 16 42652 42636 156.727 106.812 0.00841105 0.00637493 2024-08-17T17:53:34.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:34.748 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:34.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:34.849 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:34.884 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:34.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:34.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:34.986 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:34 smithi165 ceph-mon[105517]: from='client.35103 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:53:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:34 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3926869030' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:53:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:34 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:53:35.015 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:35.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:35.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:35.117 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:35.154 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:35.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:35.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:35.256 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:35.292 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:35.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:35.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:35.394 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:35.430 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:35.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:35.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:35.532 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:35.568 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:35.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:35.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:35.670 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:35.701 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 18 16 44655 44639 154.975 125.188 0.00998829 0.00644624 2024-08-17T17:53:35.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:35.705 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:35.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:35.807 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:35.842 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:35.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:35.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:35.889 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:35 smithi136 ceph-mon[109861]: pgmap v2089: 89 pgs: 89 active+clean; 2.2 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-08-17T17:53:35.889 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:35 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:53:35.889 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:35 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:53:35.889 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:35 smithi136 ceph-mon[109861]: osdmap e1308: 8 total, 8 up, 8 in 2024-08-17T17:53:35.889 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:35 smithi136 ceph-mon[115078]: pgmap v2089: 89 pgs: 89 active+clean; 2.2 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-08-17T17:53:35.890 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:35 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:53:35.890 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:35 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:53:35.890 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:35 smithi136 ceph-mon[115078]: osdmap e1308: 8 total, 8 up, 8 in 2024-08-17T17:53:35.944 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:35.967 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T17:53:35.967 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:53:35.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:35.981 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:35.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:36.001 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T17:53:36.001 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:53:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:35 smithi165 ceph-mon[105517]: pgmap v2089: 89 pgs: 89 active+clean; 2.2 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-08-17T17:53:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:35 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:53:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:35 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:53:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:35 smithi165 ceph-mon[105517]: osdmap e1308: 8 total, 8 up, 8 in 2024-08-17T17:53:36.033 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T17:53:36.034 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:53:36.068 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T17:53:36.069 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:53:36.082 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:36.100 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T17:53:36.101 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:53:36.113 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:36.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:36.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:36.139 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T17:53:36.140 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:53:36.173 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T17:53:36.174 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:53:36.208 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T17:53:36.209 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:53:36.216 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:36.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:36.244 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:36.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:36.345 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:36.374 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:36.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:36.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:36.378 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":2088,"stamp":"2024-08-17T17:53:32.580466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2341148645,"num_objects":35740,"num_object_clones":0,"num_object_copies":107220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35740,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":35993,"num_write_kb":2288145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35922,"ondisk_log_size":35922,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7635904,"kb_used_data":6771392,"kb_used_omap":295,"kb_used_meta":864152,"kb_avail":742325312,"statfs":{"total":767960285184,"available":760141119488,"internally_reserved":0,"allocated":6933905408,"data_stored":6889890548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":302302,"internal_metadata":884892450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,4,5,9,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2194276352,"num_objects":33482,"num_object_clones":0,"num_object_copies":100446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33482,"num_write_kb":2142848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004708"},"pg_stats":[{"pgid":"53.f","version":"1307'2354","reported_seq":2367,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:31.856678+0000","last_change":"2024-08-17T17:53:10.489536+0000","last_active":"2024-08-17T17:53:31.856678+0000","last_peered":"2024-08-17T17:53:31.856678+0000","last_clean":"2024-08-17T17:53:31.856678+0000","last_became_active":"2024-08-17T17:53:10.489173+0000","last_became_peered":"2024-08-17T17:53:10.489173+0000","last_unstale":"2024-08-17T17:53:31.856678+0000","last_undegraded":"2024-08-17T17:53:31.856678+0000","last_fullsized":"2024-08-17T17:53:31.856678+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":2354,"log_dups_size":0,"ondisk_log_size":2354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:46:03.457908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154271744,"num_objects":2354,"num_object_clones":0,"num_object_copies":7062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2354,"num_write_kb":150656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.e","version":"1307'2280","reported_seq":2293,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:31.832298+0000","last_change":"2024-08-17T17:53:10.488446+0000","last_active":"2024-08-17T17:53:31.832298+0000","last_peered":"2024-08-17T17:53:31.832298+0000","last_clean":"2024-08-17T17:53:31.832298+0000","last_became_active":"2024-08-17T17:53:10.488058+0000","last_became_peered":"2024-08-17T17:53:10.488058+0000","last_unstale":"2024-08-17T17:53:31.832298+0000","last_undegraded":"2024-08-17T17:53:31.832298+0000","last_fullsized":"2024-08-17T17:53:31.832298+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":2280,"log_dups_size":0,"ondisk_log_size":2280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:00:57.616106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149422080,"num_objects":2280,"num_object_clones":0,"num_object_copies":6840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2280,"num_write_kb":145920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.d","version":"1307'2141","reported_seq":2160,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:30.330101+0000","last_change":"2024-08-17T17:53:10.646982+0000","last_active":"2024-08-17T17:53:30.330101+0000","last_peered":"2024-08-17T17:53:30.330101+0000","last_clean":"2024-08-17T17:53:30.330101+0000","last_became_active":"2024-08-17T17:53:10.646591+0000","last_became_peered":"2024-08-17T17:53:10.646591+0000","last_unstale":"2024-08-17T17:53:30.330101+0000","last_undegraded":"2024-08-17T17:53:30.330101+0000","last_fullsized":"2024-08-17T17:53:30.330101+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":2141,"log_dups_size":0,"ondisk_log_size":2141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:43:53.874191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140312576,"num_objects":2141,"num_object_clones":0,"num_object_copies":6423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2141,"num_write_kb":137024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.c","version":"1307'2263","reported_seq":2276,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:31.873664+0000","last_change":"2024-08-17T17:53:10.559850+0000","last_active":"2024-08-17T17:53:31.873664+0000","last_peered":"2024-08-17T17:53:31.873664+0000","last_clean":"2024-08-17T17:53:31.873664+0000","last_became_active":"2024-08-17T17:53:10.559588+0000","last_became_peered":"2024-08-17T17:53:10.559588+0000","last_unstale":"2024-08-17T17:53:31.873664+0000","last_undegraded":"2024-08-17T17:53:31.873664+0000","last_fullsized":"2024-08-17T17:53:31.873664+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":2263,"log_dups_size":0,"ondisk_log_size":2263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:16:20.553763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148307968,"num_objects":2263,"num_object_clones":0,"num_object_copies":6789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2263,"num_write_kb":144832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"53.b","version":"1307'2331","reported_seq":2344,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:31.876080+0000","last_change":"2024-08-17T17:53:10.490102+0000","last_active":"2024-08-17T17:53:31.876080+0000","last_peered":"2024-08-17T17:53:31.876080+0000","last_clean":"2024-08-17T17:53:31.876080+0000","last_became_active":"2024-08-17T17:53:10.489717+0000","last_became_peered":"2024-08-17T17:53:10.489717+0000","last_unstale":"2024-08-17T17:53:31.876080+0000","last_undegraded":"2024-08-17T17:53:31.876080+0000","last_fullsized":"2024-08-17T17:53:31.876080+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":2331,"log_dups_size":0,"ondisk_log_size":2331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:50:43.448878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152764416,"num_objects":2331,"num_object_clones":0,"num_object_copies":6993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2331,"num_write_kb":149184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"53.a","version":"1307'2162","reported_seq":2181,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:30.666162+0000","last_change":"2024-08-17T17:53:10.489596+0000","last_active":"2024-08-17T17:53:30.666162+0000","last_peered":"2024-08-17T17:53:30.666162+0000","last_clean":"2024-08-17T17:53:30.666162+0000","last_became_active":"2024-08-17T17:53:10.489230+0000","last_became_peered":"2024-08-17T17:53:10.489230+0000","last_unstale":"2024-08-17T17:53:30.666162+0000","last_undegraded":"2024-08-17T17:53:30.666162+0000","last_fullsized":"2024-08-17T17:53:30.666162+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":2162,"log_dups_size":0,"ondisk_log_size":2162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:23:47.226443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141688832,"num_objects":2162,"num_object_clones":0,"num_object_copies":6486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2162,"num_write_kb":138368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"53.9","version":"1307'2175","reported_seq":2194,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:30.793078+0000","last_change":"2024-08-17T17:53:10.487653+0000","last_active":"2024-08-17T17:53:30.793078+0000","last_peered":"2024-08-17T17:53:30.793078+0000","last_clean":"2024-08-17T17:53:30.793078+0000","last_became_active":"2024-08-17T17:53:10.487310+0000","last_became_peered":"2024-08-17T17:53:10.487310+0000","last_unstale":"2024-08-17T17:53:30.793078+0000","last_undegraded":"2024-08-17T17:53:30.793078+0000","last_fullsized":"2024-08-17T17:53:30.793078+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":2175,"log_dups_size":0,"ondisk_log_size":2175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:10:55.120382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142540800,"num_objects":2175,"num_object_clones":0,"num_object_copies":6525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2175,"num_write_kb":139200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.8","version":"1307'2350","reported_seq":2363,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:31.872650+0000","last_change":"2024-08-17T17:53:10.487722+0000","last_active":"2024-08-17T17:53:31.872650+0000","last_peered":"2024-08-17T17:53:31.872650+0000","last_clean":"2024-08-17T17:53:31.872650+0000","last_became_active":"2024-08-17T17:53:10.487354+0000","last_became_peered":"2024-08-17T17:53:10.487354+0000","last_unstale":"2024-08-17T17:53:31.872650+0000","last_undegraded":"2024-08-17T17:53:31.872650+0000","last_fullsized":"2024-08-17T17:53:31.872650+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":2350,"log_dups_size":0,"ondisk_log_size":2350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:34:55.381227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154009600,"num_objects":2350,"num_object_clones":0,"num_object_copies":7050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2350,"num_write_kb":150400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"53.7","version":"1307'2207","reported_seq":2225,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:30.316079+0000","last_change":"2024-08-17T17:53:10.486659+0000","last_active":"2024-08-17T17:53:30.316079+0000","last_peered":"2024-08-17T17:53:30.316079+0000","last_clean":"2024-08-17T17:53:30.316079+0000","last_became_active":"2024-08-17T17:53:10.486213+0000","last_became_peered":"2024-08-17T17:53:10.486213+0000","last_unstale":"2024-08-17T17:53:30.316079+0000","last_undegraded":"2024-08-17T17:53:30.316079+0000","last_fullsized":"2024-08-17T17:53:30.316079+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":2207,"log_dups_size":0,"ondisk_log_size":2207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:04:36.795065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144637952,"num_objects":2207,"num_object_clones":0,"num_object_copies":6621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2207,"num_write_kb":141248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.6","version":"1307'2170","reported_seq":2189,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:30.661787+0000","last_change":"2024-08-17T17:53:10.646981+0000","last_active":"2024-08-17T17:53:30.661787+0000","last_peered":"2024-08-17T17:53:30.661787+0000","last_clean":"2024-08-17T17:53:30.661787+0000","last_became_active":"2024-08-17T17:53:10.646671+0000","last_became_peered":"2024-08-17T17:53:10.646671+0000","last_unstale":"2024-08-17T17:53:30.661787+0000","last_undegraded":"2024-08-17T17:53:30.661787+0000","last_fullsized":"2024-08-17T17:53:30.661787+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":2170,"log_dups_size":0,"ondisk_log_size":2170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:07:35.471542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142213120,"num_objects":2170,"num_object_clones":0,"num_object_copies":6510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2170,"num_write_kb":138880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"53.5","version":"1307'2129","reported_seq":2148,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:30.815018+0000","last_change":"2024-08-17T17:53:10.485818+0000","last_active":"2024-08-17T17:53:30.815018+0000","last_peered":"2024-08-17T17:53:30.815018+0000","last_clean":"2024-08-17T17:53:30.815018+0000","last_became_active":"2024-08-17T17:53:10.485504+0000","last_became_peered":"2024-08-17T17:53:10.485504+0000","last_unstale":"2024-08-17T17:53:30.815018+0000","last_undegraded":"2024-08-17T17:53:30.815018+0000","last_fullsized":"2024-08-17T17:53:30.815018+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":2129,"log_dups_size":0,"ondisk_log_size":2129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:42:28.302125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139526144,"num_objects":2129,"num_object_clones":0,"num_object_copies":6387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2129,"num_write_kb":136256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.4","version":"1307'2108","reported_seq":2127,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:30.292022+0000","last_change":"2024-08-17T17:53:10.647352+0000","last_active":"2024-08-17T17:53:30.292022+0000","last_peered":"2024-08-17T17:53:30.292022+0000","last_clean":"2024-08-17T17:53:30.292022+0000","last_became_active":"2024-08-17T17:53:10.646558+0000","last_became_peered":"2024-08-17T17:53:10.646558+0000","last_unstale":"2024-08-17T17:53:30.292022+0000","last_undegraded":"2024-08-17T17:53:30.292022+0000","last_fullsized":"2024-08-17T17:53:30.292022+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":2108,"log_dups_size":0,"ondisk_log_size":2108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:36:52.581014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138149888,"num_objects":2108,"num_object_clones":0,"num_object_copies":6324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2108,"num_write_kb":134912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.3","version":"1307'2266","reported_seq":2279,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:31.859685+0000","last_change":"2024-08-17T17:53:10.491051+0000","last_active":"2024-08-17T17:53:31.859685+0000","last_peered":"2024-08-17T17:53:31.859685+0000","last_clean":"2024-08-17T17:53:31.859685+0000","last_became_active":"2024-08-17T17:53:10.490705+0000","last_became_peered":"2024-08-17T17:53:10.490705+0000","last_unstale":"2024-08-17T17:53:31.859685+0000","last_undegraded":"2024-08-17T17:53:31.859685+0000","last_fullsized":"2024-08-17T17:53:31.859685+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":2266,"log_dups_size":0,"ondisk_log_size":2266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:10:17.630133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148504576,"num_objects":2266,"num_object_clones":0,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2266,"num_write_kb":145024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.2","version":"1307'2338","reported_seq":2351,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:31.873505+0000","last_change":"2024-08-17T17:53:10.488417+0000","last_active":"2024-08-17T17:53:31.873505+0000","last_peered":"2024-08-17T17:53:31.873505+0000","last_clean":"2024-08-17T17:53:31.873505+0000","last_became_active":"2024-08-17T17:53:10.488065+0000","last_became_peered":"2024-08-17T17:53:10.488065+0000","last_unstale":"2024-08-17T17:53:31.873505+0000","last_undegraded":"2024-08-17T17:53:31.873505+0000","last_fullsized":"2024-08-17T17:53:31.873505+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":2338,"log_dups_size":0,"ondisk_log_size":2338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:37:51.342306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153223168,"num_objects":2338,"num_object_clones":0,"num_object_copies":7014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2338,"num_write_kb":149632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"53.1","version":"1307'2145","reported_seq":2164,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:30.325437+0000","last_change":"2024-08-17T17:53:10.488017+0000","last_active":"2024-08-17T17:53:30.325437+0000","last_peered":"2024-08-17T17:53:30.325437+0000","last_clean":"2024-08-17T17:53:30.325437+0000","last_became_active":"2024-08-17T17:53:10.487717+0000","last_became_peered":"2024-08-17T17:53:10.487717+0000","last_unstale":"2024-08-17T17:53:30.325437+0000","last_undegraded":"2024-08-17T17:53:30.325437+0000","last_fullsized":"2024-08-17T17:53:30.325437+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":2145,"log_dups_size":0,"ondisk_log_size":2145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:19:02.563935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140574720,"num_objects":2145,"num_object_clones":0,"num_object_copies":6435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2145,"num_write_kb":137280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.0","version":"1307'2295","reported_seq":2308,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:31.870356+0000","last_change":"2024-08-17T17:53:10.646902+0000","last_active":"2024-08-17T17:53:31.870356+0000","last_peered":"2024-08-17T17:53:31.870356+0000","last_clean":"2024-08-17T17:53:31.870356+0000","last_became_active":"2024-08-17T17:53:10.646517+0000","last_became_peered":"2024-08-17T17:53:10.646517+0000","last_unstale":"2024-08-17T17:53:31.870356+0000","last_undegraded":"2024-08-17T17:53:31.870356+0000","last_fullsized":"2024-08-17T17:53:31.870356+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":2295,"log_dups_size":0,"ondisk_log_size":2295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:25:17.966365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150405120,"num_objects":2295,"num_object_clones":0,"num_object_copies":6885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2295,"num_write_kb":146880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":1917,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.581018+0000","last_change":"2024-08-17T17:23:42.937520+0000","last_active":"2024-08-17T17:53:20.581018+0000","last_peered":"2024-08-17T17:53:20.581018+0000","last_clean":"2024-08-17T17:53:20.581018+0000","last_became_active":"2024-08-17T17:23:42.937174+0000","last_became_peered":"2024-08-17T17:23:42.937174+0000","last_unstale":"2024-08-17T17:53:20.581018+0000","last_undegraded":"2024-08-17T17:53:20.581018+0000","last_fullsized":"2024-08-17T17:53:20.581018+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:47:56.679612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1276,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.939329+0000","last_change":"2024-08-17T17:23:07.398676+0000","last_active":"2024-08-17T17:53:15.939329+0000","last_peered":"2024-08-17T17:53:15.939329+0000","last_clean":"2024-08-17T17:53:15.939329+0000","last_became_active":"2024-08-17T17:23:07.398369+0000","last_became_peered":"2024-08-17T17:23:07.398369+0000","last_unstale":"2024-08-17T17:53:15.939329+0000","last_undegraded":"2024-08-17T17:53:15.939329+0000","last_fullsized":"2024-08-17T17:53:15.939329+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:55:04.281412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":1901,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.566784+0000","last_change":"2024-08-17T17:24:17.476168+0000","last_active":"2024-08-17T17:53:20.566784+0000","last_peered":"2024-08-17T17:53:20.566784+0000","last_clean":"2024-08-17T17:53:20.566784+0000","last_became_active":"2024-08-17T17:24:17.465845+0000","last_became_peered":"2024-08-17T17:24:17.465845+0000","last_unstale":"2024-08-17T17:53:20.566784+0000","last_undegraded":"2024-08-17T17:53:20.566784+0000","last_fullsized":"2024-08-17T17:53:20.566784+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:15:01.494447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1918,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.561957+0000","last_change":"2024-08-17T17:24:17.461772+0000","last_active":"2024-08-17T17:53:20.561957+0000","last_peered":"2024-08-17T17:53:20.561957+0000","last_clean":"2024-08-17T17:53:20.561957+0000","last_became_active":"2024-08-17T17:24:17.460806+0000","last_became_peered":"2024-08-17T17:24:17.460806+0000","last_unstale":"2024-08-17T17:53:20.561957+0000","last_undegraded":"2024-08-17T17:53:20.561957+0000","last_fullsized":"2024-08-17T17:53:20.561957+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:01:16.453148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":1259,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.496200+0000","last_change":"2024-08-17T17:10:05.965176+0000","last_active":"2024-08-17T17:53:15.496200+0000","last_peered":"2024-08-17T17:53:15.496200+0000","last_clean":"2024-08-17T17:53:15.496200+0000","last_became_active":"2024-08-17T17:10:05.964861+0000","last_became_peered":"2024-08-17T17:10:05.964861+0000","last_unstale":"2024-08-17T17:53:15.496200+0000","last_undegraded":"2024-08-17T17:53:15.496200+0000","last_fullsized":"2024-08-17T17:53:15.496200+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:59:04.277926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1291,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.939447+0000","last_change":"2024-08-17T17:24:17.462527+0000","last_active":"2024-08-17T17:53:15.939447+0000","last_peered":"2024-08-17T17:53:15.939447+0000","last_clean":"2024-08-17T17:53:15.939447+0000","last_became_active":"2024-08-17T17:24:17.462240+0000","last_became_peered":"2024-08-17T17:24:17.462240+0000","last_unstale":"2024-08-17T17:53:15.939447+0000","last_undegraded":"2024-08-17T17:53:15.939447+0000","last_fullsized":"2024-08-17T17:53:15.939447+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:40:38.063717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":1274,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.496083+0000","last_change":"2024-08-17T17:23:07.391038+0000","last_active":"2024-08-17T17:53:15.496083+0000","last_peered":"2024-08-17T17:53:15.496083+0000","last_clean":"2024-08-17T17:53:15.496083+0000","last_became_active":"2024-08-17T17:23:07.390571+0000","last_became_peered":"2024-08-17T17:23:07.390571+0000","last_unstale":"2024-08-17T17:53:15.496083+0000","last_undegraded":"2024-08-17T17:53:15.496083+0000","last_fullsized":"2024-08-17T17:53:15.496083+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:18:41.791740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1291,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.939690+0000","last_change":"2024-08-17T17:24:17.458168+0000","last_active":"2024-08-17T17:53:15.939690+0000","last_peered":"2024-08-17T17:53:15.939690+0000","last_clean":"2024-08-17T17:53:15.939690+0000","last_became_active":"2024-08-17T17:24:17.457847+0000","last_became_peered":"2024-08-17T17:24:17.457847+0000","last_unstale":"2024-08-17T17:53:15.939690+0000","last_undegraded":"2024-08-17T17:53:15.939690+0000","last_fullsized":"2024-08-17T17:53:15.939690+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:28.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":1900,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.580463+0000","last_change":"2024-08-17T17:22:32.883756+0000","last_active":"2024-08-17T17:53:20.580463+0000","last_peered":"2024-08-17T17:53:20.580463+0000","last_clean":"2024-08-17T17:53:20.580463+0000","last_became_active":"2024-08-17T17:22:32.874867+0000","last_became_peered":"2024-08-17T17:22:32.874867+0000","last_unstale":"2024-08-17T17:53:20.580463+0000","last_undegraded":"2024-08-17T17:53:20.580463+0000","last_fullsized":"2024-08-17T17:53:20.580463+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:52:27.375231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1919,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.576455+0000","last_change":"2024-08-17T17:24:17.456622+0000","last_active":"2024-08-17T17:53:20.576455+0000","last_peered":"2024-08-17T17:53:20.576455+0000","last_clean":"2024-08-17T17:53:20.576455+0000","last_became_active":"2024-08-17T17:24:17.455396+0000","last_became_peered":"2024-08-17T17:24:17.455396+0000","last_unstale":"2024-08-17T17:53:20.576455+0000","last_undegraded":"2024-08-17T17:53:20.576455+0000","last_fullsized":"2024-08-17T17:53:20.576455+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:09.564331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1293,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.939536+0000","last_change":"2024-08-17T17:23:42.930464+0000","last_active":"2024-08-17T17:53:15.939536+0000","last_peered":"2024-08-17T17:53:15.939536+0000","last_clean":"2024-08-17T17:53:15.939536+0000","last_became_active":"2024-08-17T17:23:42.929949+0000","last_became_peered":"2024-08-17T17:23:42.929949+0000","last_unstale":"2024-08-17T17:53:15.939536+0000","last_undegraded":"2024-08-17T17:53:15.939536+0000","last_fullsized":"2024-08-17T17:53:15.939536+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:48:27.092339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1904,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.566892+0000","last_change":"2024-08-17T17:24:17.452009+0000","last_active":"2024-08-17T17:53:20.566892+0000","last_peered":"2024-08-17T17:53:20.566892+0000","last_clean":"2024-08-17T17:53:20.566892+0000","last_became_active":"2024-08-17T17:24:17.451492+0000","last_became_peered":"2024-08-17T17:24:17.451492+0000","last_unstale":"2024-08-17T17:53:20.566892+0000","last_undegraded":"2024-08-17T17:53:20.566892+0000","last_fullsized":"2024-08-17T17:53:20.566892+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:21:24.699633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1916,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.571198+0000","last_change":"2024-08-17T17:24:17.452025+0000","last_active":"2024-08-17T17:53:20.571198+0000","last_peered":"2024-08-17T17:53:20.571198+0000","last_clean":"2024-08-17T17:53:20.571198+0000","last_became_active":"2024-08-17T17:24:17.448029+0000","last_became_peered":"2024-08-17T17:24:17.448029+0000","last_unstale":"2024-08-17T17:53:20.571198+0000","last_undegraded":"2024-08-17T17:53:20.571198+0000","last_fullsized":"2024-08-17T17:53:20.571198+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:45.089346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1911,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.581000+0000","last_change":"2024-08-17T17:24:17.469096+0000","last_active":"2024-08-17T17:53:20.581000+0000","last_peered":"2024-08-17T17:53:20.581000+0000","last_clean":"2024-08-17T17:53:20.581000+0000","last_became_active":"2024-08-17T17:24:17.468669+0000","last_became_peered":"2024-08-17T17:24:17.468669+0000","last_unstale":"2024-08-17T17:53:20.581000+0000","last_undegraded":"2024-08-17T17:53:20.581000+0000","last_fullsized":"2024-08-17T17:53:20.581000+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:03:39.427341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1905,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.561890+0000","last_change":"2024-08-17T17:24:17.453946+0000","last_active":"2024-08-17T17:53:20.561890+0000","last_peered":"2024-08-17T17:53:20.561890+0000","last_clean":"2024-08-17T17:53:20.561890+0000","last_became_active":"2024-08-17T17:24:17.445004+0000","last_became_peered":"2024-08-17T17:24:17.445004+0000","last_unstale":"2024-08-17T17:53:20.561890+0000","last_undegraded":"2024-08-17T17:53:20.561890+0000","last_fullsized":"2024-08-17T17:53:20.561890+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:00:19.366629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1276,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.496821+0000","last_change":"2024-08-17T17:23:07.402338+0000","last_active":"2024-08-17T17:53:15.496821+0000","last_peered":"2024-08-17T17:53:15.496821+0000","last_clean":"2024-08-17T17:53:15.496821+0000","last_became_active":"2024-08-17T17:23:07.401996+0000","last_became_peered":"2024-08-17T17:23:07.401996+0000","last_unstale":"2024-08-17T17:53:15.496821+0000","last_undegraded":"2024-08-17T17:53:15.496821+0000","last_fullsized":"2024-08-17T17:53:15.496821+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:39:21.566786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1889,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.561979+0000","last_change":"2024-08-17T17:24:17.442338+0000","last_active":"2024-08-17T17:53:20.561979+0000","last_peered":"2024-08-17T17:53:20.561979+0000","last_clean":"2024-08-17T17:53:20.561979+0000","last_became_active":"2024-08-17T17:24:17.441948+0000","last_became_peered":"2024-08-17T17:24:17.441948+0000","last_unstale":"2024-08-17T17:53:20.561979+0000","last_undegraded":"2024-08-17T17:53:20.561979+0000","last_fullsized":"2024-08-17T17:53:20.561979+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:17.891492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1907,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.581184+0000","last_change":"2024-08-17T17:24:17.459083+0000","last_active":"2024-08-17T17:53:20.581184+0000","last_peered":"2024-08-17T17:53:20.581184+0000","last_clean":"2024-08-17T17:53:20.581184+0000","last_became_active":"2024-08-17T17:24:17.458759+0000","last_became_peered":"2024-08-17T17:24:17.458759+0000","last_unstale":"2024-08-17T17:53:20.581184+0000","last_undegraded":"2024-08-17T17:53:20.581184+0000","last_fullsized":"2024-08-17T17:53:20.581184+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:11:59.301864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1848,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.562472+0000","last_change":"2024-08-17T17:24:17.468086+0000","last_active":"2024-08-17T17:53:20.562472+0000","last_peered":"2024-08-17T17:53:20.562472+0000","last_clean":"2024-08-17T17:53:20.562472+0000","last_became_active":"2024-08-17T17:24:17.467522+0000","last_became_peered":"2024-08-17T17:24:17.467522+0000","last_unstale":"2024-08-17T17:53:20.562472+0000","last_undegraded":"2024-08-17T17:53:20.562472+0000","last_fullsized":"2024-08-17T17:53:20.562472+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:52:55.249649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1272,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.496387+0000","last_change":"2024-08-17T17:23:07.392447+0000","last_active":"2024-08-17T17:53:15.496387+0000","last_peered":"2024-08-17T17:53:15.496387+0000","last_clean":"2024-08-17T17:53:15.496387+0000","last_became_active":"2024-08-17T17:23:07.391765+0000","last_became_peered":"2024-08-17T17:23:07.391765+0000","last_unstale":"2024-08-17T17:53:15.496387+0000","last_undegraded":"2024-08-17T17:53:15.496387+0000","last_fullsized":"2024-08-17T17:53:15.496387+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:59:18.602346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":1274,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.570249+0000","last_change":"2024-08-17T17:24:17.450313+0000","last_active":"2024-08-17T17:53:20.570249+0000","last_peered":"2024-08-17T17:53:20.570249+0000","last_clean":"2024-08-17T17:53:20.570249+0000","last_became_active":"2024-08-17T17:24:17.448803+0000","last_became_peered":"2024-08-17T17:24:17.448803+0000","last_unstale":"2024-08-17T17:53:20.570249+0000","last_undegraded":"2024-08-17T17:53:20.570249+0000","last_fullsized":"2024-08-17T17:53:20.570249+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:33.554836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1916,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.571802+0000","last_change":"2024-08-17T17:23:42.936147+0000","last_active":"2024-08-17T17:53:20.571802+0000","last_peered":"2024-08-17T17:53:20.571802+0000","last_clean":"2024-08-17T17:53:20.571802+0000","last_became_active":"2024-08-17T17:23:42.935859+0000","last_became_peered":"2024-08-17T17:23:42.935859+0000","last_unstale":"2024-08-17T17:53:20.571802+0000","last_undegraded":"2024-08-17T17:53:20.571802+0000","last_fullsized":"2024-08-17T17:53:20.571802+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:47:36.838706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1286,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.939610+0000","last_change":"2024-08-17T17:10:05.969131+0000","last_active":"2024-08-17T17:53:15.939610+0000","last_peered":"2024-08-17T17:53:15.939610+0000","last_clean":"2024-08-17T17:53:15.939610+0000","last_became_active":"2024-08-17T17:10:05.968856+0000","last_became_peered":"2024-08-17T17:10:05.968856+0000","last_unstale":"2024-08-17T17:53:15.939610+0000","last_undegraded":"2024-08-17T17:53:15.939610+0000","last_fullsized":"2024-08-17T17:53:15.939610+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:25:49.550065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1277,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.939187+0000","last_change":"2024-08-17T17:23:07.455919+0000","last_active":"2024-08-17T17:53:15.939187+0000","last_peered":"2024-08-17T17:53:15.939187+0000","last_clean":"2024-08-17T17:53:15.939187+0000","last_became_active":"2024-08-17T17:23:07.455284+0000","last_became_peered":"2024-08-17T17:23:07.455284+0000","last_unstale":"2024-08-17T17:53:15.939187+0000","last_undegraded":"2024-08-17T17:53:15.939187+0000","last_fullsized":"2024-08-17T17:53:15.939187+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:37:46.260775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1264,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.939502+0000","last_change":"2024-08-17T17:10:06.979492+0000","last_active":"2024-08-17T17:53:15.939502+0000","last_peered":"2024-08-17T17:53:15.939502+0000","last_clean":"2024-08-17T17:53:15.939502+0000","last_became_active":"2024-08-17T17:10:06.979158+0000","last_became_peered":"2024-08-17T17:10:06.979158+0000","last_unstale":"2024-08-17T17:53:15.939502+0000","last_undegraded":"2024-08-17T17:53:15.939502+0000","last_fullsized":"2024-08-17T17:53:15.939502+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:12:42.261365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1289,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.496883+0000","last_change":"2024-08-17T17:23:42.935099+0000","last_active":"2024-08-17T17:53:15.496883+0000","last_peered":"2024-08-17T17:53:15.496883+0000","last_clean":"2024-08-17T17:53:15.496883+0000","last_became_active":"2024-08-17T17:23:42.934501+0000","last_became_peered":"2024-08-17T17:23:42.934501+0000","last_unstale":"2024-08-17T17:53:15.496883+0000","last_undegraded":"2024-08-17T17:53:15.496883+0000","last_fullsized":"2024-08-17T17:53:15.496883+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:15:19.095713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":1914,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.580899+0000","last_change":"2024-08-17T17:23:07.436763+0000","last_active":"2024-08-17T17:53:20.580899+0000","last_peered":"2024-08-17T17:53:20.580899+0000","last_clean":"2024-08-17T17:53:20.580899+0000","last_became_active":"2024-08-17T17:23:07.434720+0000","last_became_peered":"2024-08-17T17:23:07.434720+0000","last_unstale":"2024-08-17T17:53:20.580899+0000","last_undegraded":"2024-08-17T17:53:20.580899+0000","last_fullsized":"2024-08-17T17:53:20.580899+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:10:20.594115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1884,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.562028+0000","last_change":"2024-08-17T17:24:17.442329+0000","last_active":"2024-08-17T17:53:20.562028+0000","last_peered":"2024-08-17T17:53:20.562028+0000","last_clean":"2024-08-17T17:53:20.562028+0000","last_became_active":"2024-08-17T17:24:17.441666+0000","last_became_peered":"2024-08-17T17:24:17.441666+0000","last_unstale":"2024-08-17T17:53:20.562028+0000","last_undegraded":"2024-08-17T17:53:20.562028+0000","last_fullsized":"2024-08-17T17:53:20.562028+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:43:06.033194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1898,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.573382+0000","last_change":"2024-08-17T17:23:07.428296+0000","last_active":"2024-08-17T17:53:20.573382+0000","last_peered":"2024-08-17T17:53:20.573382+0000","last_clean":"2024-08-17T17:53:20.573382+0000","last_became_active":"2024-08-17T17:23:07.428010+0000","last_became_peered":"2024-08-17T17:23:07.428010+0000","last_unstale":"2024-08-17T17:53:20.573382+0000","last_undegraded":"2024-08-17T17:53:20.573382+0000","last_fullsized":"2024-08-17T17:53:20.573382+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:45.459212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":1885,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.564462+0000","last_change":"2024-08-17T17:24:17.481368+0000","last_active":"2024-08-17T17:53:20.564462+0000","last_peered":"2024-08-17T17:53:20.564462+0000","last_clean":"2024-08-17T17:53:20.564462+0000","last_became_active":"2024-08-17T17:24:17.477509+0000","last_became_peered":"2024-08-17T17:24:17.477509+0000","last_unstale":"2024-08-17T17:53:20.564462+0000","last_undegraded":"2024-08-17T17:53:20.564462+0000","last_fullsized":"2024-08-17T17:53:20.564462+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:05:35.558004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1899,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.580625+0000","last_change":"2024-08-17T17:22:32.884274+0000","last_active":"2024-08-17T17:53:20.580625+0000","last_peered":"2024-08-17T17:53:20.580625+0000","last_clean":"2024-08-17T17:53:20.580625+0000","last_became_active":"2024-08-17T17:22:32.874667+0000","last_became_peered":"2024-08-17T17:22:32.874667+0000","last_unstale":"2024-08-17T17:53:20.580625+0000","last_undegraded":"2024-08-17T17:53:20.580625+0000","last_fullsized":"2024-08-17T17:53:20.580625+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:22:05.834587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1286,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.583075+0000","last_change":"2024-08-17T17:22:32.870569+0000","last_active":"2024-08-17T17:53:15.583075+0000","last_peered":"2024-08-17T17:53:15.583075+0000","last_clean":"2024-08-17T17:53:15.583075+0000","last_became_active":"2024-08-17T17:22:32.869335+0000","last_became_peered":"2024-08-17T17:22:32.869335+0000","last_unstale":"2024-08-17T17:53:15.583075+0000","last_undegraded":"2024-08-17T17:53:15.583075+0000","last_fullsized":"2024-08-17T17:53:15.583075+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:24.969174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1919,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.572343+0000","last_change":"2024-08-17T17:23:42.935825+0000","last_active":"2024-08-17T17:53:20.572343+0000","last_peered":"2024-08-17T17:53:20.572343+0000","last_clean":"2024-08-17T17:53:20.572343+0000","last_became_active":"2024-08-17T17:23:42.935150+0000","last_became_peered":"2024-08-17T17:23:42.935150+0000","last_unstale":"2024-08-17T17:53:20.572343+0000","last_undegraded":"2024-08-17T17:53:20.572343+0000","last_fullsized":"2024-08-17T17:53:20.572343+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:57:19.577644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1907,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.571248+0000","last_change":"2024-08-17T17:23:07.431586+0000","last_active":"2024-08-17T17:53:20.571248+0000","last_peered":"2024-08-17T17:53:20.571248+0000","last_clean":"2024-08-17T17:53:20.571248+0000","last_became_active":"2024-08-17T17:23:07.431301+0000","last_became_peered":"2024-08-17T17:23:07.431301+0000","last_unstale":"2024-08-17T17:53:20.571248+0000","last_undegraded":"2024-08-17T17:53:20.571248+0000","last_fullsized":"2024-08-17T17:53:20.571248+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:28:18.706489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1935,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.572170+0000","last_change":"2024-08-17T17:23:07.420550+0000","last_active":"2024-08-17T17:53:20.572170+0000","last_peered":"2024-08-17T17:53:20.572170+0000","last_clean":"2024-08-17T17:53:20.572170+0000","last_became_active":"2024-08-17T17:23:07.420223+0000","last_became_peered":"2024-08-17T17:23:07.420223+0000","last_unstale":"2024-08-17T17:53:20.572170+0000","last_undegraded":"2024-08-17T17:53:20.572170+0000","last_fullsized":"2024-08-17T17:53:20.572170+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:33:46.996371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1290,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.496357+0000","last_change":"2024-08-17T17:24:17.426267+0000","last_active":"2024-08-17T17:53:15.496357+0000","last_peered":"2024-08-17T17:53:15.496357+0000","last_clean":"2024-08-17T17:53:15.496357+0000","last_became_active":"2024-08-17T17:24:17.425925+0000","last_became_peered":"2024-08-17T17:24:17.425925+0000","last_unstale":"2024-08-17T17:53:15.496357+0000","last_undegraded":"2024-08-17T17:53:15.496357+0000","last_fullsized":"2024-08-17T17:53:15.496357+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:24:25.017856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1286,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.495978+0000","last_change":"2024-08-17T17:24:17.474138+0000","last_active":"2024-08-17T17:53:15.495978+0000","last_peered":"2024-08-17T17:53:15.495978+0000","last_clean":"2024-08-17T17:53:15.495978+0000","last_became_active":"2024-08-17T17:24:17.473708+0000","last_became_peered":"2024-08-17T17:24:17.473708+0000","last_unstale":"2024-08-17T17:53:15.495978+0000","last_undegraded":"2024-08-17T17:53:15.495978+0000","last_fullsized":"2024-08-17T17:53:15.495978+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:10:57.768067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1280,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.939777+0000","last_change":"2024-08-17T17:22:32.871533+0000","last_active":"2024-08-17T17:53:15.939777+0000","last_peered":"2024-08-17T17:53:15.939777+0000","last_clean":"2024-08-17T17:53:15.939777+0000","last_became_active":"2024-08-17T17:22:32.868226+0000","last_became_peered":"2024-08-17T17:22:32.868226+0000","last_unstale":"2024-08-17T17:53:15.939777+0000","last_undegraded":"2024-08-17T17:53:15.939777+0000","last_fullsized":"2024-08-17T17:53:15.939777+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:00:02.629455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":1290,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.496242+0000","last_change":"2024-08-17T17:23:07.391021+0000","last_active":"2024-08-17T17:53:15.496242+0000","last_peered":"2024-08-17T17:53:15.496242+0000","last_clean":"2024-08-17T17:53:15.496242+0000","last_became_active":"2024-08-17T17:23:07.390551+0000","last_became_peered":"2024-08-17T17:23:07.390551+0000","last_unstale":"2024-08-17T17:53:15.496242+0000","last_undegraded":"2024-08-17T17:53:15.496242+0000","last_fullsized":"2024-08-17T17:53:15.496242+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:06.239013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001023532,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1919,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.572703+0000","last_change":"2024-08-17T17:23:07.406362+0000","last_active":"2024-08-17T17:53:20.572703+0000","last_peered":"2024-08-17T17:53:20.572703+0000","last_clean":"2024-08-17T17:53:20.572703+0000","last_became_active":"2024-08-17T17:23:07.406071+0000","last_became_peered":"2024-08-17T17:23:07.406071+0000","last_unstale":"2024-08-17T17:53:20.572703+0000","last_undegraded":"2024-08-17T17:53:20.572703+0000","last_fullsized":"2024-08-17T17:53:20.572703+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:45:47.383432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1899,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.578499+0000","last_change":"2024-08-17T17:23:42.910246+0000","last_active":"2024-08-17T17:53:20.578499+0000","last_peered":"2024-08-17T17:53:20.578499+0000","last_clean":"2024-08-17T17:53:20.578499+0000","last_became_active":"2024-08-17T17:23:42.909929+0000","last_became_peered":"2024-08-17T17:23:42.909929+0000","last_unstale":"2024-08-17T17:53:20.578499+0000","last_undegraded":"2024-08-17T17:53:20.578499+0000","last_fullsized":"2024-08-17T17:53:20.578499+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:14:45.210558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1928,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.576210+0000","last_change":"2024-08-17T17:24:17.438891+0000","last_active":"2024-08-17T17:53:20.576210+0000","last_peered":"2024-08-17T17:53:20.576210+0000","last_clean":"2024-08-17T17:53:20.576210+0000","last_became_active":"2024-08-17T17:24:17.437714+0000","last_became_peered":"2024-08-17T17:24:17.437714+0000","last_unstale":"2024-08-17T17:53:20.576210+0000","last_undegraded":"2024-08-17T17:53:20.576210+0000","last_fullsized":"2024-08-17T17:53:20.576210+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:58.825017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1921,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.581102+0000","last_change":"2024-08-17T17:23:42.908634+0000","last_active":"2024-08-17T17:53:20.581102+0000","last_peered":"2024-08-17T17:53:20.581102+0000","last_clean":"2024-08-17T17:53:20.581102+0000","last_became_active":"2024-08-17T17:23:42.907675+0000","last_became_peered":"2024-08-17T17:23:42.907675+0000","last_unstale":"2024-08-17T17:53:20.581102+0000","last_undegraded":"2024-08-17T17:53:20.581102+0000","last_fullsized":"2024-08-17T17:53:20.581102+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:03:37.612222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1910,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.582218+0000","last_change":"2024-08-17T17:24:17.457559+0000","last_active":"2024-08-17T17:53:20.582218+0000","last_peered":"2024-08-17T17:53:20.582218+0000","last_clean":"2024-08-17T17:53:20.582218+0000","last_became_active":"2024-08-17T17:24:17.457262+0000","last_became_peered":"2024-08-17T17:24:17.457262+0000","last_unstale":"2024-08-17T17:53:20.582218+0000","last_undegraded":"2024-08-17T17:53:20.582218+0000","last_fullsized":"2024-08-17T17:53:20.582218+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:53:38.137277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1302,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.582876+0000","last_change":"2024-08-17T17:23:07.450509+0000","last_active":"2024-08-17T17:53:15.582876+0000","last_peered":"2024-08-17T17:53:15.582876+0000","last_clean":"2024-08-17T17:53:15.582876+0000","last_became_active":"2024-08-17T17:23:07.450182+0000","last_became_peered":"2024-08-17T17:23:07.450182+0000","last_unstale":"2024-08-17T17:53:15.582876+0000","last_undegraded":"2024-08-17T17:53:15.582876+0000","last_fullsized":"2024-08-17T17:53:15.582876+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:01:18.656656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1261,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.496601+0000","last_change":"2024-08-17T17:10:06.995023+0000","last_active":"2024-08-17T17:53:15.496601+0000","last_peered":"2024-08-17T17:53:15.496601+0000","last_clean":"2024-08-17T17:53:15.496601+0000","last_became_active":"2024-08-17T17:10:06.994757+0000","last_became_peered":"2024-08-17T17:10:06.994757+0000","last_unstale":"2024-08-17T17:53:15.496601+0000","last_undegraded":"2024-08-17T17:53:15.496601+0000","last_fullsized":"2024-08-17T17:53:15.496601+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:31:52.163518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1851,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.564841+0000","last_change":"2024-08-17T17:24:17.468443+0000","last_active":"2024-08-17T17:53:20.564841+0000","last_peered":"2024-08-17T17:53:20.564841+0000","last_clean":"2024-08-17T17:53:20.564841+0000","last_became_active":"2024-08-17T17:24:17.467394+0000","last_became_peered":"2024-08-17T17:24:17.467394+0000","last_unstale":"2024-08-17T17:53:20.564841+0000","last_undegraded":"2024-08-17T17:53:20.564841+0000","last_fullsized":"2024-08-17T17:53:20.564841+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:13.024253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":1289,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.496056+0000","last_change":"2024-08-17T17:23:42.920552+0000","last_active":"2024-08-17T17:53:15.496056+0000","last_peered":"2024-08-17T17:53:15.496056+0000","last_clean":"2024-08-17T17:53:15.496056+0000","last_became_active":"2024-08-17T17:23:42.915824+0000","last_became_peered":"2024-08-17T17:23:42.915824+0000","last_unstale":"2024-08-17T17:53:15.496056+0000","last_undegraded":"2024-08-17T17:53:15.496056+0000","last_fullsized":"2024-08-17T17:53:15.496056+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:25:18.769504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041737399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":1602,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.582755+0000","last_change":"2024-08-17T17:22:32.867754+0000","last_active":"2024-08-17T17:53:15.582755+0000","last_peered":"2024-08-17T17:53:15.582755+0000","last_clean":"2024-08-17T17:53:15.582755+0000","last_became_active":"2024-08-17T17:22:32.867345+0000","last_became_peered":"2024-08-17T17:22:32.867345+0000","last_unstale":"2024-08-17T17:53:15.582755+0000","last_undegraded":"2024-08-17T17:53:15.582755+0000","last_fullsized":"2024-08-17T17:53:15.582755+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T17:06:21.174471+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:00:42.776037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1897,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.572588+0000","last_change":"2024-08-17T17:24:17.455710+0000","last_active":"2024-08-17T17:53:20.572588+0000","last_peered":"2024-08-17T17:53:20.572588+0000","last_clean":"2024-08-17T17:53:20.572588+0000","last_became_active":"2024-08-17T17:24:17.452989+0000","last_became_peered":"2024-08-17T17:24:17.452989+0000","last_unstale":"2024-08-17T17:53:20.572588+0000","last_undegraded":"2024-08-17T17:53:20.572588+0000","last_fullsized":"2024-08-17T17:53:20.572588+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:58:07.894968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1891,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.561534+0000","last_change":"2024-08-17T17:24:17.453639+0000","last_active":"2024-08-17T17:53:20.561534+0000","last_peered":"2024-08-17T17:53:20.561534+0000","last_clean":"2024-08-17T17:53:20.561534+0000","last_became_active":"2024-08-17T17:24:17.443638+0000","last_became_peered":"2024-08-17T17:24:17.443638+0000","last_unstale":"2024-08-17T17:53:20.561534+0000","last_undegraded":"2024-08-17T17:53:20.561534+0000","last_fullsized":"2024-08-17T17:53:20.561534+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:42.107479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1278,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.939215+0000","last_change":"2024-08-17T17:23:07.451866+0000","last_active":"2024-08-17T17:53:15.939215+0000","last_peered":"2024-08-17T17:53:15.939215+0000","last_clean":"2024-08-17T17:53:15.939215+0000","last_became_active":"2024-08-17T17:23:07.451553+0000","last_became_peered":"2024-08-17T17:23:07.451553+0000","last_unstale":"2024-08-17T17:53:15.939215+0000","last_undegraded":"2024-08-17T17:53:15.939215+0000","last_fullsized":"2024-08-17T17:53:15.939215+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:37:23.930651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1280,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.567732+0000","last_change":"2024-08-17T17:23:42.915851+0000","last_active":"2024-08-17T17:53:20.567732+0000","last_peered":"2024-08-17T17:53:20.567732+0000","last_clean":"2024-08-17T17:53:20.567732+0000","last_became_active":"2024-08-17T17:23:42.915552+0000","last_became_peered":"2024-08-17T17:23:42.915552+0000","last_unstale":"2024-08-17T17:53:20.567732+0000","last_undegraded":"2024-08-17T17:53:20.567732+0000","last_fullsized":"2024-08-17T17:53:20.567732+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:45.197876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1293,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.939675+0000","last_change":"2024-08-17T17:23:42.930803+0000","last_active":"2024-08-17T17:53:15.939675+0000","last_peered":"2024-08-17T17:53:15.939675+0000","last_clean":"2024-08-17T17:53:15.939675+0000","last_became_active":"2024-08-17T17:23:42.930458+0000","last_became_peered":"2024-08-17T17:23:42.930458+0000","last_unstale":"2024-08-17T17:53:15.939675+0000","last_undegraded":"2024-08-17T17:53:15.939675+0000","last_fullsized":"2024-08-17T17:53:15.939675+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:40.816706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1276,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.496350+0000","last_change":"2024-08-17T17:23:07.401624+0000","last_active":"2024-08-17T17:53:15.496350+0000","last_peered":"2024-08-17T17:53:15.496350+0000","last_clean":"2024-08-17T17:53:15.496350+0000","last_became_active":"2024-08-17T17:23:07.400222+0000","last_became_peered":"2024-08-17T17:23:07.400222+0000","last_unstale":"2024-08-17T17:53:15.496350+0000","last_undegraded":"2024-08-17T17:53:15.496350+0000","last_fullsized":"2024-08-17T17:53:15.496350+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:07.069989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1278,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.939636+0000","last_change":"2024-08-17T17:24:17.470809+0000","last_active":"2024-08-17T17:53:15.939636+0000","last_peered":"2024-08-17T17:53:15.939636+0000","last_clean":"2024-08-17T17:53:15.939636+0000","last_became_active":"2024-08-17T17:24:17.469753+0000","last_became_peered":"2024-08-17T17:24:17.469753+0000","last_unstale":"2024-08-17T17:53:15.939636+0000","last_undegraded":"2024-08-17T17:53:15.939636+0000","last_fullsized":"2024-08-17T17:53:15.939636+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:00.049425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":1884,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.563577+0000","last_change":"2024-08-17T17:24:17.461258+0000","last_active":"2024-08-17T17:53:20.563577+0000","last_peered":"2024-08-17T17:53:20.563577+0000","last_clean":"2024-08-17T17:53:20.563577+0000","last_became_active":"2024-08-17T17:24:17.460895+0000","last_became_peered":"2024-08-17T17:24:17.460895+0000","last_unstale":"2024-08-17T17:53:20.563577+0000","last_undegraded":"2024-08-17T17:53:20.563577+0000","last_fullsized":"2024-08-17T17:53:20.563577+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:43:24.208259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1923,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.580356+0000","last_change":"2024-08-17T17:23:42.901638+0000","last_active":"2024-08-17T17:53:20.580356+0000","last_peered":"2024-08-17T17:53:20.580356+0000","last_clean":"2024-08-17T17:53:20.580356+0000","last_became_active":"2024-08-17T17:23:42.901335+0000","last_became_peered":"2024-08-17T17:23:42.901335+0000","last_unstale":"2024-08-17T17:53:20.580356+0000","last_undegraded":"2024-08-17T17:53:20.580356+0000","last_fullsized":"2024-08-17T17:53:20.580356+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:25:35.332839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1274,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.496094+0000","last_change":"2024-08-17T17:24:17.473913+0000","last_active":"2024-08-17T17:53:15.496094+0000","last_peered":"2024-08-17T17:53:15.496094+0000","last_clean":"2024-08-17T17:53:15.496094+0000","last_became_active":"2024-08-17T17:24:17.473195+0000","last_became_peered":"2024-08-17T17:24:17.473195+0000","last_unstale":"2024-08-17T17:53:15.496094+0000","last_undegraded":"2024-08-17T17:53:15.496094+0000","last_fullsized":"2024-08-17T17:53:15.496094+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:12.998033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1275,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.495865+0000","last_change":"2024-08-17T17:23:07.402010+0000","last_active":"2024-08-17T17:53:15.495865+0000","last_peered":"2024-08-17T17:53:15.495865+0000","last_clean":"2024-08-17T17:53:15.495865+0000","last_became_active":"2024-08-17T17:23:07.400849+0000","last_became_peered":"2024-08-17T17:23:07.400849+0000","last_unstale":"2024-08-17T17:53:15.495865+0000","last_undegraded":"2024-08-17T17:53:15.495865+0000","last_fullsized":"2024-08-17T17:53:15.495865+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:16:02.382257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":1258,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.583049+0000","last_change":"2024-08-17T17:10:05.973491+0000","last_active":"2024-08-17T17:53:15.583049+0000","last_peered":"2024-08-17T17:53:15.583049+0000","last_clean":"2024-08-17T17:53:15.583049+0000","last_became_active":"2024-08-17T17:10:05.973228+0000","last_became_peered":"2024-08-17T17:10:05.973228+0000","last_unstale":"2024-08-17T17:53:15.583049+0000","last_undegraded":"2024-08-17T17:53:15.583049+0000","last_fullsized":"2024-08-17T17:53:15.583049+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:54.310135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1294,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.567611+0000","last_change":"2024-08-17T17:24:17.460203+0000","last_active":"2024-08-17T17:53:20.567611+0000","last_peered":"2024-08-17T17:53:20.567611+0000","last_clean":"2024-08-17T17:53:20.567611+0000","last_became_active":"2024-08-17T17:24:17.459682+0000","last_became_peered":"2024-08-17T17:24:17.459682+0000","last_unstale":"2024-08-17T17:53:20.567611+0000","last_undegraded":"2024-08-17T17:53:20.567611+0000","last_fullsized":"2024-08-17T17:53:20.567611+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:58.703700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":1274,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.583243+0000","last_change":"2024-08-17T17:23:07.454042+0000","last_active":"2024-08-17T17:53:15.583243+0000","last_peered":"2024-08-17T17:53:15.583243+0000","last_clean":"2024-08-17T17:53:15.583243+0000","last_became_active":"2024-08-17T17:23:07.453712+0000","last_became_peered":"2024-08-17T17:23:07.453712+0000","last_unstale":"2024-08-17T17:53:15.583243+0000","last_undegraded":"2024-08-17T17:53:15.583243+0000","last_fullsized":"2024-08-17T17:53:15.583243+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:03:07.643456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1259,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.566282+0000","last_change":"2024-08-17T17:10:06.994799+0000","last_active":"2024-08-17T17:53:20.566282+0000","last_peered":"2024-08-17T17:53:20.566282+0000","last_clean":"2024-08-17T17:53:20.566282+0000","last_became_active":"2024-08-17T17:10:06.994150+0000","last_became_peered":"2024-08-17T17:10:06.994150+0000","last_unstale":"2024-08-17T17:53:20.566282+0000","last_undegraded":"2024-08-17T17:53:20.566282+0000","last_fullsized":"2024-08-17T17:53:20.566282+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:50:47.293091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":1272,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.583242+0000","last_change":"2024-08-17T17:22:32.873334+0000","last_active":"2024-08-17T17:53:15.583242+0000","last_peered":"2024-08-17T17:53:15.583242+0000","last_clean":"2024-08-17T17:53:15.583242+0000","last_became_active":"2024-08-17T17:22:32.870147+0000","last_became_peered":"2024-08-17T17:22:32.870147+0000","last_unstale":"2024-08-17T17:53:15.583242+0000","last_undegraded":"2024-08-17T17:53:15.583242+0000","last_fullsized":"2024-08-17T17:53:15.583242+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:18:03.750592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1283,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.496083+0000","last_change":"2024-08-17T17:22:32.847989+0000","last_active":"2024-08-17T17:53:15.496083+0000","last_peered":"2024-08-17T17:53:15.496083+0000","last_clean":"2024-08-17T17:53:15.496083+0000","last_became_active":"2024-08-17T17:22:32.846593+0000","last_became_peered":"2024-08-17T17:22:32.846593+0000","last_unstale":"2024-08-17T17:53:15.496083+0000","last_undegraded":"2024-08-17T17:53:15.496083+0000","last_fullsized":"2024-08-17T17:53:15.496083+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:33:50.370124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1889,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.564398+0000","last_change":"2024-08-17T17:24:17.451528+0000","last_active":"2024-08-17T17:53:20.564398+0000","last_peered":"2024-08-17T17:53:20.564398+0000","last_clean":"2024-08-17T17:53:20.564398+0000","last_became_active":"2024-08-17T17:24:17.451246+0000","last_became_peered":"2024-08-17T17:24:17.451246+0000","last_unstale":"2024-08-17T17:53:20.564398+0000","last_undegraded":"2024-08-17T17:53:20.564398+0000","last_fullsized":"2024-08-17T17:53:20.564398+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:38:18.176559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1256,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.582896+0000","last_change":"2024-08-17T17:10:05.975692+0000","last_active":"2024-08-17T17:53:15.582896+0000","last_peered":"2024-08-17T17:53:15.582896+0000","last_clean":"2024-08-17T17:53:15.582896+0000","last_became_active":"2024-08-17T17:10:05.975057+0000","last_became_peered":"2024-08-17T17:10:05.975057+0000","last_unstale":"2024-08-17T17:53:15.582896+0000","last_undegraded":"2024-08-17T17:53:15.582896+0000","last_fullsized":"2024-08-17T17:53:15.582896+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:44:53.106104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1907,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.580809+0000","last_change":"2024-08-17T17:24:17.460371+0000","last_active":"2024-08-17T17:53:20.580809+0000","last_peered":"2024-08-17T17:53:20.580809+0000","last_clean":"2024-08-17T17:53:20.580809+0000","last_became_active":"2024-08-17T17:24:17.460074+0000","last_became_peered":"2024-08-17T17:24:17.460074+0000","last_unstale":"2024-08-17T17:53:20.580809+0000","last_undegraded":"2024-08-17T17:53:20.580809+0000","last_fullsized":"2024-08-17T17:53:20.580809+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:08:48.820168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1908,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.580842+0000","last_change":"2024-08-17T17:24:17.457942+0000","last_active":"2024-08-17T17:53:20.580842+0000","last_peered":"2024-08-17T17:53:20.580842+0000","last_clean":"2024-08-17T17:53:20.580842+0000","last_became_active":"2024-08-17T17:24:17.456475+0000","last_became_peered":"2024-08-17T17:24:17.456475+0000","last_unstale":"2024-08-17T17:53:20.580842+0000","last_undegraded":"2024-08-17T17:53:20.580842+0000","last_fullsized":"2024-08-17T17:53:20.580842+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:12.325657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1272,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.566572+0000","last_change":"2024-08-17T17:24:17.451287+0000","last_active":"2024-08-17T17:53:20.566572+0000","last_peered":"2024-08-17T17:53:20.566572+0000","last_clean":"2024-08-17T17:53:20.566572+0000","last_became_active":"2024-08-17T17:24:17.450951+0000","last_became_peered":"2024-08-17T17:24:17.450951+0000","last_unstale":"2024-08-17T17:53:20.566572+0000","last_undegraded":"2024-08-17T17:53:20.566572+0000","last_fullsized":"2024-08-17T17:53:20.566572+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:13:37.590108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":1273,"reported_epoch":1306,"state":"active+clean","last_fresh":"2024-08-17T17:53:15.582960+0000","last_change":"2024-08-17T17:23:42.907970+0000","last_active":"2024-08-17T17:53:15.582960+0000","last_peered":"2024-08-17T17:53:15.582960+0000","last_clean":"2024-08-17T17:53:15.582960+0000","last_became_active":"2024-08-17T17:23:42.907595+0000","last_became_peered":"2024-08-17T17:23:42.907595+0000","last_unstale":"2024-08-17T17:53:15.582960+0000","last_undegraded":"2024-08-17T17:53:15.582960+0000","last_fullsized":"2024-08-17T17:53:15.582960+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:11:46.130004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1889,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-08-17T17:53:20.564068+0000","last_change":"2024-08-17T17:24:17.486591+0000","last_active":"2024-08-17T17:53:20.564068+0000","last_peered":"2024-08-17T17:53:20.564068+0000","last_clean":"2024-08-17T17:53:20.564068+0000","last_became_active":"2024-08-17T17:24:17.486320+0000","last_became_peered":"2024-08-17T17:24:17.486320+0000","last_unstale":"2024-08-17T17:53:20.564068+0000","last_undegraded":"2024-08-17T17:53:20.564068+0000","last_fullsized":"2024-08-17T17:53:20.564068+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:03.249803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":53,"num_pg":16,"stat_sum":{"num_bytes":2340552704,"num_objects":35714,"num_object_clones":0,"num_object_copies":107142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35714,"num_write_kb":2285696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7029653504,"data_stored":7029653504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35714,"ondisk_log_size":35714,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234530,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":287972,"kb_used_data":158492,"kb_used_omap":1,"kb_used_meta":129470,"kb_avail":93457180,"statfs":{"total":95995035648,"available":95700152320,"internally_reserved":0,"allocated":162295808,"data_stored":157018294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":132577739},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496168,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":704964,"kb_used_data":587772,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":93040188,"statfs":{"total":95995035648,"available":95273152512,"internally_reserved":0,"allocated":601878528,"data_stored":596822935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":119994823},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565409,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":789044,"kb_used_data":706924,"kb_used_omap":72,"kb_used_meta":82039,"kb_avail":92956108,"statfs":{"total":95995035648,"available":95187054592,"internally_reserved":0,"allocated":723890176,"data_stored":718077222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":84008430},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":674,"seq":2894807957864,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":647928,"kb_used_data":557552,"kb_used_omap":74,"kb_used_meta":90293,"kb_avail":93097224,"statfs":{"total":95995035648,"available":95331557376,"internally_reserved":0,"allocated":570933248,"data_stored":565116724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76116,"internal_metadata":92460716},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743186799,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1436164,"kb_used_data":1337532,"kb_used_omap":66,"kb_used_meta":98557,"kb_avail":92308988,"statfs":{"total":95995035648,"available":94524403712,"internally_reserved":0,"allocated":1369632768,"data_stored":1363932396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":100923051},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448438,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":919996,"kb_used_data":833972,"kb_used_omap":75,"kb_used_meta":85940,"kb_avail":92825156,"statfs":{"total":95995035648,"available":95052959744,"internally_reserved":0,"allocated":853987328,"data_stored":848167240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":88002751},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757804,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1150160,"kb_used_data":1028808,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":92594992,"statfs":{"total":95995035648,"available":94817271808,"internally_reserved":0,"allocated":1053499392,"data_stored":1048237513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124254662},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019441,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1699676,"kb_used_data":1560340,"kb_used_omap":1,"kb_used_meta":139326,"kb_avail":92045476,"statfs":{"total":95995035648,"available":94254567424,"internally_reserved":0,"allocated":1597788160,"data_stored":1592518224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":142670278},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":864157696,"data_stored":864157696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1385365504,"data_stored":1385365504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":581238784,"data_stored":581238784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":708902912,"data_stored":708902912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1672871936,"data_stored":1672871936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1063714816,"data_stored":1063714816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":605028352,"data_stored":605028352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":148373504,"data_stored":148373504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:53:36.382 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-17T17:53:36.382 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool set unique_pool_0 pg_num 26 2024-08-17T17:53:36.476 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:36.507 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:36.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:36.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:36.609 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:36.637 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:36.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:36.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:36.702 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 19 16 46502 46486 152.887 115.438 0.0161795 0.00653524 2024-08-17T17:53:36.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:36.774 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:36.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:36.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:36.875 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:36.905 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:36.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:36.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:36 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3482074333' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:53:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:53:36.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:36 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3482074333' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:53:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:53:37.007 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:37.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:36 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3482074333' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:53:37.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:53:37.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:37.052 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:37.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:37.084 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:53:37.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:37.190 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:37.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:37.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:37.266 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:53:37.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:37.321 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:37.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:37.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:37.422 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:37.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:37.466 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:37.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:37.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:37.599 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:37.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:37.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:37.645 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:53:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:53:37] "GET /metrics HTTP/1.1" 200 36536 "" "Prometheus/2.51.0" 2024-08-17T17:53:37.678 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:53:37.701 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:37.702 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:2024-08-17T17:53:37.702321+0000 min lat: 0.00118032 max lat: 0.196833 avg lat: 0.00662033 2024-08-17T17:53:37.702 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:53:37.702 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 20 16 48325 48309 150.939 113.938 0.00869625 0.00662033 2024-08-17T17:53:37.740 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:37.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:37.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:37.842 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:37.889 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:37.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:37.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:37.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:37 smithi136 ceph-mon[109861]: pgmap v2091: 89 pgs: 89 active+clean; 2.5 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 135 MiB/s wr, 2.17k op/s 2024-08-17T17:53:37.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:37 smithi136 ceph-mon[115078]: pgmap v2091: 89 pgs: 89 active+clean; 2.5 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 135 MiB/s wr, 2.17k op/s 2024-08-17T17:53:37.992 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:37 smithi165 ceph-mon[105517]: pgmap v2091: 89 pgs: 89 active+clean; 2.5 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 135 MiB/s wr, 2.17k op/s 2024-08-17T17:53:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:38.028 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:38.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:38.129 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:38.163 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:38.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:38.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:38.265 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:38.294 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:38.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:38.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:38.396 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:38.429 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:38.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:38.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:38.531 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:38.562 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:38.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:38.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:38.664 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:38.702 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 21 16 50156 50140 149.198 114.438 0.0118931 0.00669687 2024-08-17T17:53:38.720 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:38.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:38.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:38.822 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:38.843 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:38.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:38.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:38.945 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:38.980 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:38.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:38.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:39.073 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:53:39.082 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:39.162 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:39.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:39.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:39.265 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:39.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:39.304 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:39.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:39.405 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:39.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:39.448 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:39.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:39.549 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:39.587 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:39.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:39.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:39.688 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:39.702 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 22 16 51697 51681 146.794 96.3125 0.00915975 0.0068075 2024-08-17T17:53:39.724 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:39.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:39.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:39.826 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:39.864 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:39.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:39.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:39.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:39 smithi136 ceph-mon[109861]: pgmap v2092: 89 pgs: 89 active+clean; 2.8 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-08-17T17:53:39.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:39 smithi136 ceph-mon[115078]: pgmap v2092: 89 pgs: 89 active+clean; 2.8 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-08-17T17:53:39.966 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:39.991 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:39.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:39.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:39 smithi165 ceph-mon[105517]: pgmap v2092: 89 pgs: 89 active+clean; 2.8 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-08-17T17:53:40.092 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:40.132 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:40.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:40.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:40.234 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:40.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:40.275 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:40.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:40.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:40.415 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:40.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:40.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:40.517 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:40.554 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:40.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:40.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:40.656 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:40.694 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:40.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:40.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:40.702 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 23 16 53267 53251 144.678 98.125 0.0129395 0.00690686 2024-08-17T17:53:40.796 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:40.830 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:40.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:40.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:40.932 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:40.962 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:40.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:40.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:41.064 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:41.094 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:41.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:41.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:41.196 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:41.223 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:41.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:41.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:41.325 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:41.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:41.365 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:41.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:41.466 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:41.494 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:41.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:41.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:41.596 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:41.644 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:41.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:41.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:41.703 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 24 16 55003 54987 143.169 108.5 0.0113681 0.00697977 2024-08-17T17:53:41.749 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:41.779 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:41.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:41.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:41.780 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:53:41.881 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:41.912 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:41.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:41.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:41.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:41 smithi136 ceph-mon[109861]: pgmap v2093: 89 pgs: 89 active+clean; 2.9 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-08-17T17:53:41.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:41 smithi136 ceph-mon[115078]: pgmap v2093: 89 pgs: 89 active+clean; 2.9 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-08-17T17:53:42.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:41 smithi165 ceph-mon[105517]: pgmap v2093: 89 pgs: 89 active+clean; 2.9 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-08-17T17:53:42.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:42.047 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:42.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:42.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:42.149 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:42.181 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:42.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:42.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:42.283 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:42.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:42.310 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:42.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:42.411 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:42.437 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:42.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:42.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:42.468 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T17:53:42.468 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:53:42.540 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:42.582 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:42.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:42.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:42.684 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:42.701 INFO:teuthology.orchestra.run.smithi136.stderr:set pool 53 pg_num to 26 2024-08-17T17:53:42.705 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 25 16 56531 56515 141.249 95.5 0.0113589 0.00707242 2024-08-17T17:53:42.714 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:42.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:42.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:42.815 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:42.851 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:42.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:42.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:42.954 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:42.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:42.998 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:42.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:43.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:42 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1256715735' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T17:53:43.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:42 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T17:53:43.048 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:42 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1256715735' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T17:53:43.048 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:42 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T17:53:43.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:42 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1256715735' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T17:53:43.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:42 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T17:53:43.099 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:43.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:43.137 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:43.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:43.238 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:43.277 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:43.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:43.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:43.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:43.418 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:43.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:43.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:43.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:43.558 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:43.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:43.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:43.660 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:43.688 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:43.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:43.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:43.705 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 26 16 58210 58194 139.852 104.938 0.0115705 0.00714474 2024-08-17T17:53:43.789 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:43.823 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:43.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:43.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:43.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:43.961 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:43.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:43.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:43 smithi165 ceph-mon[105517]: pgmap v2094: 89 pgs: 89 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-08-17T17:53:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:43 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-08-17T17:53:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:43 smithi165 ceph-mon[105517]: osdmap e1309: 8 total, 8 up, 8 in 2024-08-17T17:53:44.062 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:44.106 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:44.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:44.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:44.184 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:53:44.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:43 smithi136 ceph-mon[109861]: pgmap v2094: 89 pgs: 89 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-08-17T17:53:44.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:43 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-08-17T17:53:44.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:43 smithi136 ceph-mon[109861]: osdmap e1309: 8 total, 8 up, 8 in 2024-08-17T17:53:44.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:43 smithi136 ceph-mon[115078]: pgmap v2094: 89 pgs: 89 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-08-17T17:53:44.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:43 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-08-17T17:53:44.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:43 smithi136 ceph-mon[115078]: osdmap e1309: 8 total, 8 up, 8 in 2024-08-17T17:53:44.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:44.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:44.240 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:44.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:44.341 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:44.372 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:44.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:44.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:44.474 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:44.512 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:44.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:44.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:44.613 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:44.642 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:44.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:44.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:44.652 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-17T17:53:44.705 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 27 16 59873 59857 138.522 103.938 0.00661956 0.00721335 2024-08-17T17:53:44.744 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:44.787 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:44.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:44.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:44.889 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:44.911 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:44.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:44.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:44.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:44 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"} : dispatch 2024-08-17T17:53:44.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:44 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"} : dispatch 2024-08-17T17:53:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:44 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"} : dispatch 2024-08-17T17:53:45.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:45.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:45.035 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:45.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:45.136 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:45.158 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:45.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:45.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:45.260 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:45.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:45.282 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:45.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:45.383 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:45.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:45.409 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:45.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:45.510 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:45.532 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:45.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:45.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:45.634 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:45.655 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:45.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:45.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:45.705 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 28 16 59930 59914 133.703 3.5625 0.181042 0.0072284 2024-08-17T17:53:45.757 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:45.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:45.788 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:45.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:45.889 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:45.910 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:53:45.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:45.929 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:45.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:46.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:45 smithi165 ceph-mon[105517]: pgmap v2096: 89 pgs: 89 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-08-17T17:53:46.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:45 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-17T17:53:46.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:45 smithi165 ceph-mon[105517]: osdmap e1310: 8 total, 8 up, 8 in 2024-08-17T17:53:46.022 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:45 smithi136 ceph-mon[109861]: pgmap v2096: 89 pgs: 89 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-08-17T17:53:46.023 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:45 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-17T17:53:46.023 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:45 smithi136 ceph-mon[109861]: osdmap e1310: 8 total, 8 up, 8 in 2024-08-17T17:53:46.023 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:45 smithi136 ceph-mon[115078]: pgmap v2096: 89 pgs: 89 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-08-17T17:53:46.023 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:45 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-17T17:53:46.023 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:45 smithi136 ceph-mon[115078]: osdmap e1310: 8 total, 8 up, 8 in 2024-08-17T17:53:46.029 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:46.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:46.076 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:46.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:46.177 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:46.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:46.213 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:46.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:46.314 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:46.349 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:46.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:46.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:46.451 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:46.490 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:46.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:46.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:46.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:46.627 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:46.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:46.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:46.705 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 29 16 61605 61589 132.702 104.688 0.00829876 0.00753006 2024-08-17T17:53:46.729 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:46.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:46.770 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:46.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:46.871 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:46.915 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:46.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:46.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:46.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:46 smithi136 ceph-mon[109861]: osdmap e1311: 8 total, 8 up, 8 in 2024-08-17T17:53:46.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:46 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:53:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:46 smithi136 ceph-mon[115078]: osdmap e1311: 8 total, 8 up, 8 in 2024-08-17T17:53:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:46 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:53:47.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:46 smithi165 ceph-mon[105517]: osdmap e1311: 8 total, 8 up, 8 in 2024-08-17T17:53:47.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:46 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:53:47.017 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:47.050 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:47.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:47.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:47.152 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:47.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:47.200 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:47.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:47.301 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:47.326 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:47.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:47.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:47.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:47.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:47.462 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:47.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:47.563 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:47.596 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:47.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:47.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:47.698 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:47.705 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 30 16 62813 62797 130.795 75.5 0.00626851 0.00764117 2024-08-17T17:53:47.732 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:47.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:47.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:47.736 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:53:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:53:47] "GET /metrics HTTP/1.1" 200 36536 "" "Prometheus/2.51.0" 2024-08-17T17:53:47.834 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:47.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:47.866 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:47.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:47.967 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:48.001 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:48.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:48.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:47 smithi165 ceph-mon[105517]: 53.4 scrub starts 2024-08-17T17:53:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:47 smithi165 ceph-mon[105517]: 53.5 scrub starts 2024-08-17T17:53:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:47 smithi165 ceph-mon[105517]: 53.4 scrub ok 2024-08-17T17:53:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:47 smithi165 ceph-mon[105517]: 53.3 scrub starts 2024-08-17T17:53:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:47 smithi165 ceph-mon[105517]: pgmap v2099: 99 pgs: 4 peering, 8 unknown, 87 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-08-17T17:53:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:47 smithi165 ceph-mon[105517]: 53.6 scrub starts 2024-08-17T17:53:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:47 smithi165 ceph-mon[105517]: 53.0 scrub starts 2024-08-17T17:53:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:47 smithi165 ceph-mon[105517]: 53.3 scrub ok 2024-08-17T17:53:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:47 smithi165 ceph-mon[105517]: 53.6 scrub ok 2024-08-17T17:53:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:47 smithi165 ceph-mon[105517]: 53.13 scrub starts 2024-08-17T17:53:48.103 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:48.120 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:47 smithi136 ceph-mon[109861]: 53.4 scrub starts 2024-08-17T17:53:48.120 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:47 smithi136 ceph-mon[109861]: 53.5 scrub starts 2024-08-17T17:53:48.120 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:47 smithi136 ceph-mon[109861]: 53.4 scrub ok 2024-08-17T17:53:48.120 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:47 smithi136 ceph-mon[109861]: 53.3 scrub starts 2024-08-17T17:53:48.120 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:47 smithi136 ceph-mon[109861]: pgmap v2099: 99 pgs: 4 peering, 8 unknown, 87 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-08-17T17:53:48.120 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:47 smithi136 ceph-mon[109861]: 53.6 scrub starts 2024-08-17T17:53:48.120 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:47 smithi136 ceph-mon[109861]: 53.0 scrub starts 2024-08-17T17:53:48.120 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:47 smithi136 ceph-mon[109861]: 53.3 scrub ok 2024-08-17T17:53:48.120 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:47 smithi136 ceph-mon[109861]: 53.6 scrub ok 2024-08-17T17:53:48.121 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:47 smithi136 ceph-mon[109861]: 53.13 scrub starts 2024-08-17T17:53:48.121 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:47 smithi136 ceph-mon[115078]: 53.4 scrub starts 2024-08-17T17:53:48.121 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:47 smithi136 ceph-mon[115078]: 53.5 scrub starts 2024-08-17T17:53:48.121 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:47 smithi136 ceph-mon[115078]: 53.4 scrub ok 2024-08-17T17:53:48.121 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:47 smithi136 ceph-mon[115078]: 53.3 scrub starts 2024-08-17T17:53:48.121 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:47 smithi136 ceph-mon[115078]: pgmap v2099: 99 pgs: 4 peering, 8 unknown, 87 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-08-17T17:53:48.121 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:47 smithi136 ceph-mon[115078]: 53.6 scrub starts 2024-08-17T17:53:48.121 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:47 smithi136 ceph-mon[115078]: 53.0 scrub starts 2024-08-17T17:53:48.121 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:47 smithi136 ceph-mon[115078]: 53.3 scrub ok 2024-08-17T17:53:48.121 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:47 smithi136 ceph-mon[115078]: 53.6 scrub ok 2024-08-17T17:53:48.122 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:47 smithi136 ceph-mon[115078]: 53.13 scrub starts 2024-08-17T17:53:48.129 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:48.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:48.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:48.231 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:48.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:48.271 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:48.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:48.372 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:48.393 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:48.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:48.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:48.495 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:48.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:48.533 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:48.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:48.634 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:48.665 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:48.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:48.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:48.706 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 31 16 64053 64037 129.076 77.5 0.00235901 0.00774227 2024-08-17T17:53:48.765 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-17T17:53:48.765 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:53:48.766 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-17T17:53:48.766 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.566995 2024-08-17T17:53:48.766 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 7 0.5669950826563549 2024-08-17T17:53:48.767 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:48.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:48.812 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:48.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:48.913 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:48.942 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:48.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:48.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:48 smithi165 ceph-mon[105517]: 53.5 scrub ok 2024-08-17T17:53:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:48 smithi165 ceph-mon[105517]: 53.7 scrub starts 2024-08-17T17:53:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:48 smithi165 ceph-mon[105517]: 53.15 scrub starts 2024-08-17T17:53:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:48 smithi165 ceph-mon[105517]: 53.7 scrub ok 2024-08-17T17:53:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:48 smithi165 ceph-mon[105517]: 53.0 scrub ok 2024-08-17T17:53:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:48 smithi165 ceph-mon[105517]: 53.15 scrub ok 2024-08-17T17:53:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:48 smithi165 ceph-mon[105517]: 53.2 scrub starts 2024-08-17T17:53:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:48 smithi165 ceph-mon[105517]: 53.16 scrub starts 2024-08-17T17:53:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:48 smithi165 ceph-mon[105517]: 53.13 scrub ok 2024-08-17T17:53:49.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:48 smithi165 ceph-mon[105517]: 53.16 scrub ok 2024-08-17T17:53:49.043 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:49.065 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:49.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:49.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:49.072 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:48 smithi136 ceph-mon[109861]: 53.5 scrub ok 2024-08-17T17:53:49.072 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:48 smithi136 ceph-mon[109861]: 53.7 scrub starts 2024-08-17T17:53:49.072 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:48 smithi136 ceph-mon[109861]: 53.15 scrub starts 2024-08-17T17:53:49.072 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:48 smithi136 ceph-mon[109861]: 53.7 scrub ok 2024-08-17T17:53:49.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:48 smithi136 ceph-mon[109861]: 53.0 scrub ok 2024-08-17T17:53:49.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:48 smithi136 ceph-mon[109861]: 53.15 scrub ok 2024-08-17T17:53:49.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:48 smithi136 ceph-mon[109861]: 53.2 scrub starts 2024-08-17T17:53:49.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:48 smithi136 ceph-mon[109861]: 53.16 scrub starts 2024-08-17T17:53:49.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:48 smithi136 ceph-mon[109861]: 53.13 scrub ok 2024-08-17T17:53:49.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:48 smithi136 ceph-mon[109861]: 53.16 scrub ok 2024-08-17T17:53:49.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:48 smithi136 ceph-mon[115078]: 53.5 scrub ok 2024-08-17T17:53:49.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:48 smithi136 ceph-mon[115078]: 53.7 scrub starts 2024-08-17T17:53:49.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:48 smithi136 ceph-mon[115078]: 53.15 scrub starts 2024-08-17T17:53:49.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:48 smithi136 ceph-mon[115078]: 53.7 scrub ok 2024-08-17T17:53:49.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:48 smithi136 ceph-mon[115078]: 53.0 scrub ok 2024-08-17T17:53:49.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:48 smithi136 ceph-mon[115078]: 53.15 scrub ok 2024-08-17T17:53:49.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:48 smithi136 ceph-mon[115078]: 53.2 scrub starts 2024-08-17T17:53:49.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:48 smithi136 ceph-mon[115078]: 53.16 scrub starts 2024-08-17T17:53:49.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:48 smithi136 ceph-mon[115078]: 53.13 scrub ok 2024-08-17T17:53:49.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:48 smithi136 ceph-mon[115078]: 53.16 scrub ok 2024-08-17T17:53:49.166 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:49.193 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:49.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:49.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:49.295 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:49.334 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:49.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:49.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:49.436 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:49.465 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:49.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:49.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:49.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:49.592 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:49.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:49.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:49.694 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:49.706 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 32 16 65171 65155 127.225 69.875 0.0152661 0.00785527 2024-08-17T17:53:49.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:49.734 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:49.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:49.781 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T17:53:49.835 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:49.856 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:49.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:49.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:49.906 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:49 smithi136 ceph-mon[109861]: 53.9 scrub starts 2024-08-17T17:53:49.906 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:49 smithi136 ceph-mon[109861]: 53.2 scrub ok 2024-08-17T17:53:49.906 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:49 smithi136 ceph-mon[109861]: pgmap v2100: 99 pgs: 2 active+clean+scrubbing, 4 peering, 4 unknown, 89 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 11 KiB/s, 0 objects/s recovering 2024-08-17T17:53:49.906 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:49 smithi136 ceph-mon[109861]: 53.9 scrub ok 2024-08-17T17:53:49.906 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:49 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4266806505' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:53:49.906 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:49 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:53:49.906 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:49 smithi136 ceph-mon[109861]: 53.19 scrub starts 2024-08-17T17:53:49.907 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:49 smithi136 ceph-mon[115078]: 53.9 scrub starts 2024-08-17T17:53:49.907 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:49 smithi136 ceph-mon[115078]: 53.2 scrub ok 2024-08-17T17:53:49.907 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:49 smithi136 ceph-mon[115078]: pgmap v2100: 99 pgs: 2 active+clean+scrubbing, 4 peering, 4 unknown, 89 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 11 KiB/s, 0 objects/s recovering 2024-08-17T17:53:49.907 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:49 smithi136 ceph-mon[115078]: 53.9 scrub ok 2024-08-17T17:53:49.907 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:49 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4266806505' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:53:49.907 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:49 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:53:49.907 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:49 smithi136 ceph-mon[115078]: 53.19 scrub starts 2024-08-17T17:53:49.958 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:49.990 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:49.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:49.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:49 smithi165 ceph-mon[105517]: 53.9 scrub starts 2024-08-17T17:53:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:49 smithi165 ceph-mon[105517]: 53.2 scrub ok 2024-08-17T17:53:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:49 smithi165 ceph-mon[105517]: pgmap v2100: 99 pgs: 2 active+clean+scrubbing, 4 peering, 4 unknown, 89 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 11 KiB/s, 0 objects/s recovering 2024-08-17T17:53:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:49 smithi165 ceph-mon[105517]: 53.9 scrub ok 2024-08-17T17:53:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:49 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4266806505' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:53:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:49 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:53:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:49 smithi165 ceph-mon[105517]: 53.19 scrub starts 2024-08-17T17:53:50.092 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:50.125 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:50.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:50.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:50.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:50.248 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:50.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:50.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:50.350 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:50.371 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:50.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:50.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:50.473 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:50.508 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:50.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:50.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:50.610 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:50.631 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:50.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:50.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:50.706 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 33 16 66189 66173 125.298 63.625 0.0031082 0.00797659 2024-08-17T17:53:50.734 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:50.763 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:50.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:50.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:50.802 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:50 smithi136 ceph-mon[109861]: 53.14 scrub starts 2024-08-17T17:53:50.802 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:50 smithi136 ceph-mon[109861]: 53.14 scrub ok 2024-08-17T17:53:50.802 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:50 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:53:50.802 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:50 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T17:53:50.802 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:50 smithi136 ceph-mon[109861]: 53.8 scrub starts 2024-08-17T17:53:50.802 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:50 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:53:50.802 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:50 smithi136 ceph-mon[109861]: osdmap e1312: 8 total, 8 up, 8 in 2024-08-17T17:53:50.802 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:50 smithi136 ceph-mon[109861]: 53.19 scrub ok 2024-08-17T17:53:50.803 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:50 smithi136 ceph-mon[115078]: 53.14 scrub starts 2024-08-17T17:53:50.803 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:50 smithi136 ceph-mon[115078]: 53.14 scrub ok 2024-08-17T17:53:50.803 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:50 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:53:50.803 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:50 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T17:53:50.803 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:50 smithi136 ceph-mon[115078]: 53.8 scrub starts 2024-08-17T17:53:50.803 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:50 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:53:50.803 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:50 smithi136 ceph-mon[115078]: osdmap e1312: 8 total, 8 up, 8 in 2024-08-17T17:53:50.803 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:50 smithi136 ceph-mon[115078]: 53.19 scrub ok 2024-08-17T17:53:50.864 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:50.885 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:50.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:50.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:50.987 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:50 smithi165 ceph-mon[105517]: 53.14 scrub starts 2024-08-17T17:53:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:50 smithi165 ceph-mon[105517]: 53.14 scrub ok 2024-08-17T17:53:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:50 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:53:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:50 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T17:53:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:50 smithi165 ceph-mon[105517]: 53.8 scrub starts 2024-08-17T17:53:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:50 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:53:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:50 smithi165 ceph-mon[105517]: osdmap e1312: 8 total, 8 up, 8 in 2024-08-17T17:53:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:50 smithi165 ceph-mon[105517]: 53.19 scrub ok 2024-08-17T17:53:51.024 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:51.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:51.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:51.090 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T17:53:51.091 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:53:51.126 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:51.137 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T17:53:51.137 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:53:51.148 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:51.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:51.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:51.156 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T17:53:51.156 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:53:51.172 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T17:53:51.172 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:53:51.223 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T17:53:51.223 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:53:51.246 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T17:53:51.247 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:53:51.250 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:51.279 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:51.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:51.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:51.280 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T17:53:51.280 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:53:51.314 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T17:53:51.314 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:53:51.380 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:51.416 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:51.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:51.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:51.518 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:51.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:51.543 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:51.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:51.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:51.680 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:51.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:51.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:51.706 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 34 16 67494 67478 124.012 81.5625 0.00418954 0.00805924 2024-08-17T17:53:51.782 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:51.813 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:51.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:51.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:51.914 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:51.936 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:51.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:51.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:52.038 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:52.043 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:51 smithi136 ceph-mon[109861]: 53.17 scrub starts 2024-08-17T17:53:52.044 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:51 smithi136 ceph-mon[109861]: 53.17 scrub ok 2024-08-17T17:53:52.044 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:51 smithi136 ceph-mon[109861]: pgmap v2102: 99 pgs: 2 active+clean+scrubbing, 1 peering, 1 unknown, 95 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 3.1 MiB/s wr, 49 op/s; 21 KiB/s, 0 objects/s recovering 2024-08-17T17:53:52.044 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:51 smithi136 ceph-mon[109861]: 53.8 scrub ok 2024-08-17T17:53:52.044 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:51 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/468633610' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:53:52.044 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:53:52.044 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:53:52.044 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:51 smithi136 ceph-mon[115078]: 53.17 scrub starts 2024-08-17T17:53:52.044 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:51 smithi136 ceph-mon[115078]: 53.17 scrub ok 2024-08-17T17:53:52.044 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:51 smithi136 ceph-mon[115078]: pgmap v2102: 99 pgs: 2 active+clean+scrubbing, 1 peering, 1 unknown, 95 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 3.1 MiB/s wr, 49 op/s; 21 KiB/s, 0 objects/s recovering 2024-08-17T17:53:52.045 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:51 smithi136 ceph-mon[115078]: 53.8 scrub ok 2024-08-17T17:53:52.045 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:51 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/468633610' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:53:52.045 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:53:52.045 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:53:52.068 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:52.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:52.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:52.170 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:52.190 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:52.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:52.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:51 smithi165 ceph-mon[105517]: 53.17 scrub starts 2024-08-17T17:53:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:51 smithi165 ceph-mon[105517]: 53.17 scrub ok 2024-08-17T17:53:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:51 smithi165 ceph-mon[105517]: pgmap v2102: 99 pgs: 2 active+clean+scrubbing, 1 peering, 1 unknown, 95 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 3.1 MiB/s wr, 49 op/s; 21 KiB/s, 0 objects/s recovering 2024-08-17T17:53:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:51 smithi165 ceph-mon[105517]: 53.8 scrub ok 2024-08-17T17:53:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:51 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/468633610' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:53:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:53:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:53:52.293 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:52.314 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:52.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:52.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:52.416 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:52.453 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:52.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:52.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:52.547 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:53:52.555 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:52.577 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:52.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:52.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:52.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:52.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:52.704 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:52.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:52.706 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 35 16 68391 68375 122.07 56.0625 0.00326745 0.00818247 2024-08-17T17:53:52.805 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:52.826 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:52.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:52.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:52.885 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:53:52.885 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T17:53:52.885 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-17T17:53:52.885 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-17T17:53:52.886 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:53:52.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:52 smithi136 ceph-mon[115078]: 53.11 scrub starts 2024-08-17T17:53:52.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:52 smithi136 ceph-mon[115078]: 53.12 scrub starts 2024-08-17T17:53:52.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:52 smithi136 ceph-mon[115078]: 53.11 scrub ok 2024-08-17T17:53:52.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-08-17T17:53:52.886 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:52 smithi136 ceph-mon[109861]: 53.11 scrub starts 2024-08-17T17:53:52.886 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:52 smithi136 ceph-mon[109861]: 53.12 scrub starts 2024-08-17T17:53:52.887 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:52 smithi136 ceph-mon[109861]: 53.11 scrub ok 2024-08-17T17:53:52.887 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-08-17T17:53:52.928 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:52.960 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:52.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:52.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:53.062 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:53.083 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:53.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:53.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:53.185 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:53.217 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:53.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:53.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:53.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:52 smithi165 ceph-mon[105517]: 53.11 scrub starts 2024-08-17T17:53:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:52 smithi165 ceph-mon[105517]: 53.12 scrub starts 2024-08-17T17:53:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:52 smithi165 ceph-mon[105517]: 53.11 scrub ok 2024-08-17T17:53:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-08-17T17:53:53.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:53.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:53.342 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:53.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:53.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:53.465 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:53.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:53.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:53.567 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:53.590 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:53.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:53.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:53.593 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:53:53.692 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:53.706 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 36 16 68962 68946 119.671 35.6875 0.00262861 0.00820154 2024-08-17T17:53:53.713 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:53.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:53.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:53.815 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:53.838 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:53.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:53.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:53.940 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:53.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:53.962 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:53.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:54.034 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:53 smithi136 ceph-mon[115078]: pgmap v2103: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 8.1 KiB/s, 0 objects/s recovering 2024-08-17T17:53:54.034 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "17"}]': finished 2024-08-17T17:53:54.034 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:53 smithi136 ceph-mon[115078]: osdmap e1313: 8 total, 8 up, 8 in 2024-08-17T17:53:54.034 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:53 smithi136 ceph-mon[115078]: 53.12 scrub ok 2024-08-17T17:53:54.035 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:53 smithi136 ceph-mon[109861]: pgmap v2103: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 8.1 KiB/s, 0 objects/s recovering 2024-08-17T17:53:54.035 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "17"}]': finished 2024-08-17T17:53:54.035 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:53 smithi136 ceph-mon[109861]: osdmap e1313: 8 total, 8 up, 8 in 2024-08-17T17:53:54.035 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:53 smithi136 ceph-mon[109861]: 53.12 scrub ok 2024-08-17T17:53:54.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:54.085 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:54.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:54.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:54.186 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:54.208 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:54.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:54.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:54.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:53 smithi165 ceph-mon[105517]: pgmap v2103: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 8.1 KiB/s, 0 objects/s recovering 2024-08-17T17:53:54.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "17"}]': finished 2024-08-17T17:53:54.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:53 smithi165 ceph-mon[105517]: osdmap e1313: 8 total, 8 up, 8 in 2024-08-17T17:53:54.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:53 smithi165 ceph-mon[105517]: 53.12 scrub ok 2024-08-17T17:53:54.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:54.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:54.333 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:54.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:54.434 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:54.441 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-17T17:53:54.458 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:54.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:54.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:54.561 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:54.586 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:54.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:54.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:54.688 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:54.706 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 37 16 68962 68946 116.437 0 - 0.00820154 2024-08-17T17:53:54.714 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:54.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:54.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:54.816 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:54.847 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:54.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:54.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:54.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:54 smithi136 ceph-mon[109861]: 53.1 scrub starts 2024-08-17T17:53:54.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:54 smithi136 ceph-mon[109861]: 53.1 scrub ok 2024-08-17T17:53:54.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:54 smithi136 ceph-mon[109861]: 53.18 scrub starts 2024-08-17T17:53:54.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:54 smithi136 ceph-mon[109861]: osdmap e1314: 8 total, 8 up, 8 in 2024-08-17T17:53:54.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:54 smithi136 ceph-mon[109861]: 53.18 scrub ok 2024-08-17T17:53:54.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:54 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-08-17T17:53:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:54 smithi136 ceph-mon[115078]: 53.1 scrub starts 2024-08-17T17:53:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:54 smithi136 ceph-mon[115078]: 53.1 scrub ok 2024-08-17T17:53:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:54 smithi136 ceph-mon[115078]: 53.18 scrub starts 2024-08-17T17:53:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:54 smithi136 ceph-mon[115078]: osdmap e1314: 8 total, 8 up, 8 in 2024-08-17T17:53:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:54 smithi136 ceph-mon[115078]: 53.18 scrub ok 2024-08-17T17:53:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:54 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-08-17T17:53:54.949 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:54.971 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:54.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:54.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:55.073 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:55.108 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:55.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:55.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:55.208 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:54 smithi165 ceph-mon[105517]: 53.1 scrub starts 2024-08-17T17:53:55.208 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:54 smithi165 ceph-mon[105517]: 53.1 scrub ok 2024-08-17T17:53:55.208 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:54 smithi165 ceph-mon[105517]: 53.18 scrub starts 2024-08-17T17:53:55.209 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:54 smithi165 ceph-mon[105517]: osdmap e1314: 8 total, 8 up, 8 in 2024-08-17T17:53:55.209 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:54 smithi165 ceph-mon[105517]: 53.18 scrub ok 2024-08-17T17:53:55.209 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:54 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-08-17T17:53:55.210 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:55.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:55.242 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:55.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:55.343 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:55.366 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:55.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:55.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:55.467 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:55.489 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:55.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:55.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:55.591 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:55.614 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:55.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:55.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:55.707 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 38 16 69360 69344 114.028 12.4375 0.00218268 0.00863519 2024-08-17T17:53:55.716 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:55.738 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:55.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:55.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:55.840 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:55.862 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:55.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:55.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:55.964 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:55.985 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:55.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:55.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:56.003 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:55 smithi136 ceph-mon[109861]: pgmap v2106: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail 2024-08-17T17:53:56.003 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:55 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "18"}]': finished 2024-08-17T17:53:56.003 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:55 smithi136 ceph-mon[109861]: osdmap e1315: 8 total, 8 up, 8 in 2024-08-17T17:53:56.003 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:55 smithi136 ceph-mon[115078]: pgmap v2106: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail 2024-08-17T17:53:56.003 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:55 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "18"}]': finished 2024-08-17T17:53:56.003 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:55 smithi136 ceph-mon[115078]: osdmap e1315: 8 total, 8 up, 8 in 2024-08-17T17:53:56.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:53:56.087 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:56.120 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:56.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:56.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:56.137 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:55 smithi165 ceph-mon[105517]: pgmap v2106: 99 pgs: 1 active+clean+scrubbing, 98 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail 2024-08-17T17:53:56.137 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:55 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "18"}]': finished 2024-08-17T17:53:56.137 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:55 smithi165 ceph-mon[105517]: osdmap e1315: 8 total, 8 up, 8 in 2024-08-17T17:53:56.221 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:56.243 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:56.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:56.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:56.345 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:56.366 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:56.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:56.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:56.468 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:56.489 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:56.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:56.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:56.591 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:56.609 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:53:56.613 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:56.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:56.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:56.707 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 39 16 69360 69344 111.104 0 - 0.00863519 2024-08-17T17:53:56.714 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:56.736 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:56.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:56.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:56.838 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:56.862 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:56.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:56.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:56.915 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:56 smithi136 ceph-mon[109861]: osdmap e1316: 8 total, 8 up, 8 in 2024-08-17T17:53:56.915 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:56 smithi136 ceph-mon[115078]: osdmap e1316: 8 total, 8 up, 8 in 2024-08-17T17:53:56.965 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:56.989 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:56.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:56.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:57.091 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:57.128 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:57.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:57.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:57.230 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:57.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:56 smithi165 ceph-mon[105517]: osdmap e1316: 8 total, 8 up, 8 in 2024-08-17T17:53:57.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:57.272 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:57.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:57.373 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:57.411 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:57.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:57.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:57.513 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:57.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:57.551 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:57.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:57.652 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:57.691 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:57.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:57.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:57.707 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:2024-08-17T17:53:57.707239+0000 min lat: 0.00118032 max lat: 2.13253 avg lat: 0.00906789 2024-08-17T17:53:57.707 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:53:57.707 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 40 16 70573 70557 110.222 37.9062 0.0111532 0.00906789 2024-08-17T17:53:57.794 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:53:57.820 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:53:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:53:57] "GET /metrics HTTP/1.1" 200 36552 "" "Prometheus/2.51.0" 2024-08-17T17:53:57.821 INFO:teuthology.orchestra.run.smithi136.stderr:set osd.7 primary-affinity to 0.566995 (8371582) 2024-08-17T17:53:57.824 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:57.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:57.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:53:57.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:57.950 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:57.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:57.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:58.000 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:57 smithi165 ceph-mon[105517]: pgmap v2109: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 96 active+clean; 4.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 4130/205716 objects misplaced (2.008%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:53:58.000 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:57 smithi165 ceph-mon[105517]: osdmap e1317: 8 total, 8 up, 8 in 2024-08-17T17:53:58.000 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:57 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1757172585' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.5669950826563549} : dispatch 2024-08-17T17:53:58.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:53:58.087 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:58.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:58.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:53:58.189 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:58.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:57 smithi136 ceph-mon[109861]: pgmap v2109: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 96 active+clean; 4.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 4130/205716 objects misplaced (2.008%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:53:58.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:57 smithi136 ceph-mon[109861]: osdmap e1317: 8 total, 8 up, 8 in 2024-08-17T17:53:58.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:57 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1757172585' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.5669950826563549} : dispatch 2024-08-17T17:53:58.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:57 smithi136 ceph-mon[115078]: pgmap v2109: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 96 active+clean; 4.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 4130/205716 objects misplaced (2.008%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:53:58.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:57 smithi136 ceph-mon[115078]: osdmap e1317: 8 total, 8 up, 8 in 2024-08-17T17:53:58.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:57 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1757172585' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.5669950826563549} : dispatch 2024-08-17T17:53:58.231 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:58.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:58.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:58.333 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:58.366 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:58.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:58.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:58.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:58.507 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:58.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:58.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:58.609 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:58.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:58.647 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:58.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:58.707 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 41 16 71923 71907 109.591 84.375 0.0119042 0.00911961 2024-08-17T17:53:58.748 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:58.794 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:58.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:58.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:58.896 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:53:58.933 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:58.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:58.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:53:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:58 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1757172585' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.5669950826563549}]': finished 2024-08-17T17:53:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:58 smithi136 ceph-mon[109861]: osdmap e1318: 8 total, 8 up, 8 in 2024-08-17T17:53:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:58 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:53:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:58 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:53:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:53:58 smithi136 ceph-mon[109861]: osdmap e1319: 8 total, 8 up, 8 in 2024-08-17T17:53:58.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:58 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1757172585' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.5669950826563549}]': finished 2024-08-17T17:53:58.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:58 smithi136 ceph-mon[115078]: osdmap e1318: 8 total, 8 up, 8 in 2024-08-17T17:53:58.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:58 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:53:58.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:58 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:53:58.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:53:58 smithi136 ceph-mon[115078]: osdmap e1319: 8 total, 8 up, 8 in 2024-08-17T17:53:59.036 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:59.079 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:59.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:59.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:59.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:53:59.212 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:59.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:59.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:53:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:58 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1757172585' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.5669950826563549}]': finished 2024-08-17T17:53:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:58 smithi165 ceph-mon[105517]: osdmap e1318: 8 total, 8 up, 8 in 2024-08-17T17:53:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:58 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:53:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:58 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:53:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:53:58 smithi165 ceph-mon[105517]: osdmap e1319: 8 total, 8 up, 8 in 2024-08-17T17:53:59.314 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:53:59.327 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:53:59.388 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:59.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:59.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:53:59.490 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:53:59.531 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:59.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:53:59.632 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:53:59.672 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:53:59.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:59.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:53:59.707 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 42 16 73373 73357 109.139 90.625 0.00642168 0.00915732 2024-08-17T17:53:59.775 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:53:59.808 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:53:59.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:53:59.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:53:59.910 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:00.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:00.004 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:00.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:00.105 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:00.137 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:00.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:00.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:00.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:00.269 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:00.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:00.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:00.371 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:00.402 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:00.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:00.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:00.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:00.549 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:00.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:00.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:00.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:00.685 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:00.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:00.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:00.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:00 smithi136 ceph-mon[109861]: pgmap v2112: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 97 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 4130/208104 objects misplaced (1.985%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:54:00.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:00 smithi136 ceph-mon[115078]: pgmap v2112: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 97 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 4130/208104 objects misplaced (1.985%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:54:00.707 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 43 16 74844 74828 108.739 91.9375 0.00449399 0.00919016 2024-08-17T17:54:00.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:00 smithi165 ceph-mon[105517]: pgmap v2112: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 97 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 4130/208104 objects misplaced (1.985%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:54:00.787 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:00.820 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:00.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:00.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:00.922 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:00.960 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:00.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:00.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:01.061 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:01.095 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:01.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:01.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:01.197 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:01.233 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:01.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:01.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:01.335 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:01.375 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:01.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:01.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:01.477 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:01.535 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:01.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:01.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:01.637 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:01.678 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:01.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:01.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:01.707 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 44 16 76357 76341 108.417 94.5625 0.00574294 0.00921875 2024-08-17T17:54:01.781 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:01.829 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:01.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:01.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:01.931 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:01.951 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:01.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:01.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:02.053 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:02.091 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:02.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:02.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:02.193 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:02.229 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:02.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:02.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:02.331 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:02.359 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:02.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:02.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:02.462 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:02.491 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:02.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:02.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:02.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:02.623 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:02.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:02.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:02.707 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 45 16 77740 77724 107.928 86.4375 0.0145769 0.00926079 2024-08-17T17:54:02.725 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:02.754 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:54:02.756 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:02.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:02.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:02.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:02 smithi165 ceph-mon[105517]: pgmap v2114: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 97 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 4130/211281 objects misplaced (1.955%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:54:02.858 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:02.892 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:02.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:02.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:02.902 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:02 smithi136 ceph-mon[115078]: pgmap v2114: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 97 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 4130/211281 objects misplaced (1.955%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:54:02.904 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:02 smithi136 ceph-mon[109861]: pgmap v2114: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 97 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 4130/211281 objects misplaced (1.955%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:54:02.993 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:03.016 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:03.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:03.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:03.119 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:03.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:03.154 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:03.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:03.255 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:03.289 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:03.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:03.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:03.391 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:03.419 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:03.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:03.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:03.521 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:03.561 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:03.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:03.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:03.663 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:03.697 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:03.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:03.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:03.707 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 46 16 78957 78941 107.235 76.0625 0.0056689 0.00931098 2024-08-17T17:54:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-08-17T17:54:03.799 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:03.820 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:03.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:03.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:03.922 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-08-17T17:54:03.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-08-17T17:54:03.949 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:03.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:03.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:04.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:04.074 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:04.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:04.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:04.176 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:04.197 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:04.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:04.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:04.298 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:04.325 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:04.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:04.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:04.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:04.448 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:04.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:04.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:04.550 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:04.574 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:04.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:04.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:04.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:04.698 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:04.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:04.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:04.707 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 47 16 79303 79287 105.414 21.625 0.00210377 0.00930401 2024-08-17T17:54:04.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:04 smithi165 ceph-mon[105517]: pgmap v2115: 99 pgs: 2 active+recovering+undersized+remapped, 97 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 29 MiB/s wr, 462 op/s; 7348/226986 objects misplaced (3.237%); 5.3 MiB/s, 84 objects/s recovering 2024-08-17T17:54:04.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:04 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "19"}]': finished 2024-08-17T17:54:04.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:04 smithi165 ceph-mon[105517]: osdmap e1320: 8 total, 8 up, 8 in 2024-08-17T17:54:04.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:04 smithi165 ceph-mon[105517]: osdmap e1321: 8 total, 8 up, 8 in 2024-08-17T17:54:04.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:04.821 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:04.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:04.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:04.912 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-17T17:54:04.912 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 0 1 2024-08-17T17:54:04.923 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:04.938 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:04 smithi136 ceph-mon[115078]: pgmap v2115: 99 pgs: 2 active+recovering+undersized+remapped, 97 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 29 MiB/s wr, 462 op/s; 7348/226986 objects misplaced (3.237%); 5.3 MiB/s, 84 objects/s recovering 2024-08-17T17:54:04.938 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:04 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "19"}]': finished 2024-08-17T17:54:04.938 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:04 smithi136 ceph-mon[115078]: osdmap e1320: 8 total, 8 up, 8 in 2024-08-17T17:54:04.938 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:04 smithi136 ceph-mon[115078]: osdmap e1321: 8 total, 8 up, 8 in 2024-08-17T17:54:04.938 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:04 smithi136 ceph-mon[109861]: pgmap v2115: 99 pgs: 2 active+recovering+undersized+remapped, 97 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 29 MiB/s wr, 462 op/s; 7348/226986 objects misplaced (3.237%); 5.3 MiB/s, 84 objects/s recovering 2024-08-17T17:54:04.938 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:04 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "19"}]': finished 2024-08-17T17:54:04.938 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:04 smithi136 ceph-mon[109861]: osdmap e1320: 8 total, 8 up, 8 in 2024-08-17T17:54:04.939 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:04 smithi136 ceph-mon[109861]: osdmap e1321: 8 total, 8 up, 8 in 2024-08-17T17:54:04.945 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:04.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:04.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:05.047 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:05.073 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:05.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:05.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:05.175 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:05.197 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:05.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:05.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:05.299 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:05.322 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:05.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:05.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:05.423 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:05.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:05.474 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:05.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:05.575 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:05.605 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:05.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:05.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-08-17T17:54:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "20"}]': finished 2024-08-17T17:54:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:05 smithi136 ceph-mon[109861]: osdmap e1322: 8 total, 8 up, 8 in 2024-08-17T17:54:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-08-17T17:54:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "20"}]': finished 2024-08-17T17:54:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:05 smithi136 ceph-mon[115078]: osdmap e1322: 8 total, 8 up, 8 in 2024-08-17T17:54:05.707 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:05.708 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 48 16 79553 79537 103.544 15.625 0.00375975 0.00964271 2024-08-17T17:54:05.739 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:05.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:05.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:05.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-08-17T17:54:05.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "20"}]': finished 2024-08-17T17:54:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:05 smithi165 ceph-mon[105517]: osdmap e1322: 8 total, 8 up, 8 in 2024-08-17T17:54:05.841 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:05.868 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:05.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:05.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:05.969 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:05.991 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:05.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:05.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:06.093 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:06.127 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:06.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:06.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:06.229 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:06.254 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:06.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:06.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:06.356 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:06.383 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:06.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:06.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:06.484 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:06.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:06.509 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:06.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:06.610 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:06.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:06.634 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:06.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:06.708 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 49 16 79977 79961 101.971 26.5 0.00318842 0.0096234 2024-08-17T17:54:06.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:06.758 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:06.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:06.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:06 smithi165 ceph-mon[105517]: pgmap v2118: 99 pgs: 2 active+recovering+undersized+remapped, 97 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s; 7348/226986 objects misplaced (3.237%); 5.3 MiB/s, 84 objects/s recovering 2024-08-17T17:54:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:54:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:06 smithi165 ceph-mon[105517]: osdmap e1323: 8 total, 8 up, 8 in 2024-08-17T17:54:06.762 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:06 smithi136 ceph-mon[109861]: pgmap v2118: 99 pgs: 2 active+recovering+undersized+remapped, 97 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s; 7348/226986 objects misplaced (3.237%); 5.3 MiB/s, 84 objects/s recovering 2024-08-17T17:54:06.762 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:54:06.762 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:06 smithi136 ceph-mon[109861]: osdmap e1323: 8 total, 8 up, 8 in 2024-08-17T17:54:06.762 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:06 smithi136 ceph-mon[115078]: pgmap v2118: 99 pgs: 2 active+recovering+undersized+remapped, 97 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s; 7348/226986 objects misplaced (3.237%); 5.3 MiB/s, 84 objects/s recovering 2024-08-17T17:54:06.763 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:54:06.763 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:06 smithi136 ceph-mon[115078]: osdmap e1323: 8 total, 8 up, 8 in 2024-08-17T17:54:06.860 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:06.882 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:06.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:06.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:06.983 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:07.008 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:07.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:07.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:07.111 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:07.132 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:07.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:07.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:07.234 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:07.259 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:07.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:07.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:07.361 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:07.384 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:07.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:07.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:07.485 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:07.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:07.515 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:07.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:07.616 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:07.646 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:07.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:07.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:07.708 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 50 16 80343 80327 100.39 22.875 0.00723733 0.00995624 2024-08-17T17:54:07.748 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:07.784 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:07.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:07.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:07.788 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:54:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:54:07] "GET /metrics HTTP/1.1" 200 36558 "" "Prometheus/2.51.0" 2024-08-17T17:54:07.886 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:07.928 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:07.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:07.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:08.030 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:08.062 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:08.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:08.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:08.164 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:08.199 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:08.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:08.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:08.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:08.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:08.344 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:08.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:08.445 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:08.484 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:08.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:08.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:08.586 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:08.619 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:08.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:08.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:08.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:08 smithi136 ceph-mon[115078]: pgmap v2121: 99 pgs: 1 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 95 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 34 MiB/s wr, 539 op/s; 4886/227793 objects degraded (2.145%); 8435/227793 objects misplaced (3.703%); 11 MiB/s, 169 objects/s recovering 2024-08-17T17:54:08.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:08 smithi136 ceph-mon[115078]: osdmap e1324: 8 total, 8 up, 8 in 2024-08-17T17:54:08.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:08 smithi136 ceph-mon[115078]: Health check failed: Degraded data redundancy: 4886/227793 objects degraded (2.145%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:54:08.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:08 smithi136 ceph-mon[109861]: pgmap v2121: 99 pgs: 1 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 95 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 34 MiB/s wr, 539 op/s; 4886/227793 objects degraded (2.145%); 8435/227793 objects misplaced (3.703%); 11 MiB/s, 169 objects/s recovering 2024-08-17T17:54:08.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:08 smithi136 ceph-mon[109861]: osdmap e1324: 8 total, 8 up, 8 in 2024-08-17T17:54:08.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:08 smithi136 ceph-mon[109861]: Health check failed: Degraded data redundancy: 4886/227793 objects degraded (2.145%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:54:08.708 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 51 16 81776 81760 100.177 89.5625 0.0147107 0.00997738 2024-08-17T17:54:08.721 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:08.749 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:08.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:08.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:08 smithi165 ceph-mon[105517]: pgmap v2121: 99 pgs: 1 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 95 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 34 MiB/s wr, 539 op/s; 4886/227793 objects degraded (2.145%); 8435/227793 objects misplaced (3.703%); 11 MiB/s, 169 objects/s recovering 2024-08-17T17:54:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:08 smithi165 ceph-mon[105517]: osdmap e1324: 8 total, 8 up, 8 in 2024-08-17T17:54:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:08 smithi165 ceph-mon[105517]: Health check failed: Degraded data redundancy: 4886/227793 objects degraded (2.145%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:54:08.851 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:08.893 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:08.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:08.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:08.995 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:09.031 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:09.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:09.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:09.133 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:09.171 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:09.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:09.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:09.273 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:09.311 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:09.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:09.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:09.413 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:09.450 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:09.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:09.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:09.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:09.574 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:09.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:09.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:09.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:09.703 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:09.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:09.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:09.708 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 52 16 83011 82995 99.7349 77.1875 0.011202 0.0100219 2024-08-17T17:54:09.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:09.831 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:09.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:09.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:09.932 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:09.969 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:09.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:09.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:10.070 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:10.091 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:10.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:10.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:10.193 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:10.234 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:10.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:10.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:10.336 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:10.374 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:10.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:10.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:10.476 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:10.505 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:10.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:10.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:10.608 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:10.647 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:10.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:10.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:10.666 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:10 smithi136 ceph-mon[109861]: pgmap v2123: 99 pgs: 1 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 95 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 51 MiB/s wr, 813 op/s; 4886/231903 objects degraded (2.107%); 8435/231903 objects misplaced (3.637%); 9.8 MiB/s, 156 objects/s recovering 2024-08-17T17:54:10.666 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:10 smithi136 ceph-mon[115078]: pgmap v2123: 99 pgs: 1 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 95 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 51 MiB/s wr, 813 op/s; 4886/231903 objects degraded (2.107%); 8435/231903 objects misplaced (3.637%); 9.8 MiB/s, 156 objects/s recovering 2024-08-17T17:54:10.708 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 53 16 84257 84241 99.3223 77.875 0.00869029 0.0100627 2024-08-17T17:54:10.749 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:10.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:10 smithi165 ceph-mon[105517]: pgmap v2123: 99 pgs: 1 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 95 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 51 MiB/s wr, 813 op/s; 4886/231903 objects degraded (2.107%); 8435/231903 objects misplaced (3.637%); 9.8 MiB/s, 156 objects/s recovering 2024-08-17T17:54:10.799 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:10.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:10.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:10.901 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:10.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:10.949 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:10.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:11.050 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:11.078 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:11.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:11.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:11.145 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:54:11.180 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:11.208 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:11.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:11.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:11.310 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:11.342 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:11.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:11.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:11.444 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:11.471 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:11.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:11.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:11.572 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:11.603 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:11.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:11.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:11.612 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:54:11.612 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-17T17:54:11.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-17T17:54:11.612 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T17:54:11.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.612 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:32847 53.4 53:2654c298:::benchmark_data_smithi136_235573_object32846:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:54:11.612 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:29.753549+0000", 2024-08-17T17:54:11.612 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.687971838999999, 2024-08-17T17:54:11.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18485406800000001, 2024-08-17T17:54:11.612 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:54:11.612 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:54:11.612 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:54:11.613 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:54:11.613 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:54:11.613 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:54:11.613 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 32847 2024-08-17T17:54:11.613 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.613 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:54:11.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.613 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:54:11.613 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.753549+0000", 2024-08-17T17:54:11.613 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.613 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.613 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:54:11.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.753549+0000", 2024-08-17T17:54:11.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:54:11.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.753550+0000", 2024-08-17T17:54:11.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6100000000000001e-07 2024-08-17T17:54:11.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:54:11.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.753566+0000", 2024-08-17T17:54:11.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.647e-05 2024-08-17T17:54:11.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:54:11.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.753573+0000", 2024-08-17T17:54:11.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4899999999999997e-06 2024-08-17T17:54:11.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:54:11.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.753578+0000", 2024-08-17T17:54:11.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7740000000000004e-06 2024-08-17T17:54:11.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.753964+0000", 2024-08-17T17:54:11.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038664400000000001 2024-08-17T17:54:11.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:54:11.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.754124+0000", 2024-08-17T17:54:11.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000160166 2024-08-17T17:54:11.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4", 2024-08-17T17:54:11.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.754268+0000", 2024-08-17T17:54:11.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014335800000000001 2024-08-17T17:54:11.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:54:11.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.755719+0000", 2024-08-17T17:54:11.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014508329999999999 2024-08-17T17:54:11.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.758491+0000", 2024-08-17T17:54:11.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0027723299999999999 2024-08-17T17:54:11.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.938277+0000", 2024-08-17T17:54:11.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17978627599999999 2024-08-17T17:54:11.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:54:11.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.938346+0000", 2024-08-17T17:54:11.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8732000000000005e-05 2024-08-17T17:54:11.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:54:11.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.938403+0000", 2024-08-17T17:54:11.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7433999999999999e-05 2024-08-17T17:54:11.618 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.618 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:54:11.618 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.618 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:32861 53.d 53:b605fed6:::benchmark_data_smithi136_235573_object32860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:54:11.618 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:29.759202+0000", 2024-08-17T17:54:11.618 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.682319395, 2024-08-17T17:54:11.619 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18128428999999999, 2024-08-17T17:54:11.619 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:54:11.619 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:54:11.619 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:54:11.619 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:54:11.619 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:54:11.619 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:54:11.619 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 32861 2024-08-17T17:54:11.619 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.619 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:54:11.619 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.619 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:54:11.620 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.759202+0000", 2024-08-17T17:54:11.620 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.620 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:54:11.620 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.759202+0000", 2024-08-17T17:54:11.620 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.620 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:54:11.620 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.759202+0000", 2024-08-17T17:54:11.620 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.54e-07 2024-08-17T17:54:11.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.620 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:54:11.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.759221+0000", 2024-08-17T17:54:11.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8206000000000002e-05 2024-08-17T17:54:11.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.621 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:54:11.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.759231+0000", 2024-08-17T17:54:11.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0538e-05 2024-08-17T17:54:11.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.621 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:54:11.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.759236+0000", 2024-08-17T17:54:11.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1710000000000001e-06 2024-08-17T17:54:11.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.759287+0000", 2024-08-17T17:54:11.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0822000000000002e-05 2024-08-17T17:54:11.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:54:11.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.759432+0000", 2024-08-17T17:54:11.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014498700000000001 2024-08-17T17:54:11.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:54:11.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.759570+0000", 2024-08-17T17:54:11.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000138011 2024-08-17T17:54:11.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.623 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.623 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:54:11.623 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.761141+0000", 2024-08-17T17:54:11.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015705700000000001 2024-08-17T17:54:11.623 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.623 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.623 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.623 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.764818+0000", 2024-08-17T17:54:11.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036777590000000001 2024-08-17T17:54:11.623 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.623 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.623 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.623 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.940396+0000", 2024-08-17T17:54:11.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17557782499999999 2024-08-17T17:54:11.623 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.624 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:54:11.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.940447+0000", 2024-08-17T17:54:11.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0872999999999999e-05 2024-08-17T17:54:11.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.624 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:54:11.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:29.940486+0000", 2024-08-17T17:54:11.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8773999999999998e-05 2024-08-17T17:54:11.624 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.624 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:54:11.624 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.625 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:67745 53.d 53:bd80ca37:::benchmark_data_smithi136_235573_object67744:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1312)", 2024-08-17T17:54:11.625 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:51.850913+0000", 2024-08-17T17:54:11.625 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.590607550000001, 2024-08-17T17:54:11.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18166998600000001, 2024-08-17T17:54:11.625 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:54:11.625 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:54:11.625 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:54:11.625 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:54:11.625 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:54:11.625 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:54:11.625 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 67745 2024-08-17T17:54:11.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.626 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:54:11.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.626 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:54:11.626 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:51.850913+0000", 2024-08-17T17:54:11.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.626 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.626 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:54:11.626 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:51.850913+0000", 2024-08-17T17:54:11.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.626 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.626 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:54:11.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:51.850915+0000", 2024-08-17T17:54:11.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6810000000000001e-06 2024-08-17T17:54:11.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:54:11.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:51.850945+0000", 2024-08-17T17:54:11.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0110000000000001e-05 2024-08-17T17:54:11.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:54:11.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:51.850964+0000", 2024-08-17T17:54:11.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8224999999999999e-05 2024-08-17T17:54:11.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:54:11.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:51.850969+0000", 2024-08-17T17:54:11.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6860000000000003e-06 2024-08-17T17:54:11.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:51.851019+0000", 2024-08-17T17:54:11.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9931e-05 2024-08-17T17:54:11.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:54:11.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:51.851153+0000", 2024-08-17T17:54:11.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013347399999999999 2024-08-17T17:54:11.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:54:11.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:51.851287+0000", 2024-08-17T17:54:11.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013402299999999999 2024-08-17T17:54:11.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:54:11.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:51.853717+0000", 2024-08-17T17:54:11.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0024303329999999998 2024-08-17T17:54:11.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:51.854101+0000", 2024-08-17T17:54:11.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000383834 2024-08-17T17:54:11.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:52.032466+0000", 2024-08-17T17:54:11.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17836552999999999 2024-08-17T17:54:11.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:54:11.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:52.032528+0000", 2024-08-17T17:54:11.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1834000000000005e-05 2024-08-17T17:54:11.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:54:11.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:52.032583+0000", 2024-08-17T17:54:11.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5325000000000003e-05 2024-08-17T17:54:11.631 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.631 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:54:11.631 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.631 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:67749 53.4 53:25b52b06:::benchmark_data_smithi136_235573_object67748:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1312)", 2024-08-17T17:54:11.631 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:51.852544+0000", 2024-08-17T17:54:11.631 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.588976818999999, 2024-08-17T17:54:11.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.180894468, 2024-08-17T17:54:11.631 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:54:11.631 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:54:11.631 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:54:11.631 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:54:11.632 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:54:11.632 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:54:11.632 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 67749 2024-08-17T17:54:11.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.632 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:54:11.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.632 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:54:11.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:51.852544+0000", 2024-08-17T17:54:11.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.632 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:54:11.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:51.852544+0000", 2024-08-17T17:54:11.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:54:11.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:51.852545+0000", 2024-08-17T17:54:11.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.167e-06 2024-08-17T17:54:11.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:54:11.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:51.852569+0000", 2024-08-17T17:54:11.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4088e-05 2024-08-17T17:54:11.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:54:11.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:51.852610+0000", 2024-08-17T17:54:11.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0525999999999998e-05 2024-08-17T17:54:11.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:54:11.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:51.852618+0000", 2024-08-17T17:54:11.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4570000000000005e-06 2024-08-17T17:54:11.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:51.852940+0000", 2024-08-17T17:54:11.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00032116000000000002 2024-08-17T17:54:11.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:54:11.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:51.853063+0000", 2024-08-17T17:54:11.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123397 2024-08-17T17:54:11.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4", 2024-08-17T17:54:11.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:51.853203+0000", 2024-08-17T17:54:11.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014008799999999999 2024-08-17T17:54:11.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:54:11.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:51.854961+0000", 2024-08-17T17:54:11.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001757619 2024-08-17T17:54:11.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:51.855417+0000", 2024-08-17T17:54:11.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000456556 2024-08-17T17:54:11.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:52.033331+0000", 2024-08-17T17:54:11.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.177913347 2024-08-17T17:54:11.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:54:11.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:52.033390+0000", 2024-08-17T17:54:11.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9437e-05 2024-08-17T17:54:11.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:54:11.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:52.033439+0000", 2024-08-17T17:54:11.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8625999999999998e-05 2024-08-17T17:54:11.638 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.638 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:54:11.638 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.638 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:79307 53.13 53:c97b79e5:::benchmark_data_smithi136_235573_object79306:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-08-17T17:54:11.638 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:54:06.276976+0000", 2024-08-17T17:54:11.639 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.1645450769999997, 2024-08-17T17:54:11.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1455467210000001, 2024-08-17T17:54:11.639 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:54:11.639 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:54:11.639 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:54:11.639 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:54:11.639 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:54:11.639 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:54:11.639 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 79307 2024-08-17T17:54:11.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.640 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:54:11.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:54:11.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.276976+0000", 2024-08-17T17:54:11.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:54:11.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.276976+0000", 2024-08-17T17:54:11.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:54:11.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.276978+0000", 2024-08-17T17:54:11.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0810000000000001e-06 2024-08-17T17:54:11.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:54:11.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277027+0000", 2024-08-17T17:54:11.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8541000000000002e-05 2024-08-17T17:54:11.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:54:11.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277051+0000", 2024-08-17T17:54:11.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4170000000000001e-05 2024-08-17T17:54:11.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:54:11.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277061+0000", 2024-08-17T17:54:11.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0108999999999999e-05 2024-08-17T17:54:11.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277665+0000", 2024-08-17T17:54:11.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00060386800000000003 2024-08-17T17:54:11.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:54:11.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277686+0000", 2024-08-17T17:54:11.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1007999999999999e-05 2024-08-17T17:54:11.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.376199+0000", 2024-08-17T17:54:11.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.098513072 2024-08-17T17:54:11.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:54:11.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.376350+0000", 2024-08-17T17:54:11.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015143599999999999 2024-08-17T17:54:11.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,2,3", 2024-08-17T17:54:11.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.376525+0000", 2024-08-17T17:54:11.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017496499999999999 2024-08-17T17:54:11.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:54:11.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.421092+0000", 2024-08-17T17:54:11.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.044566755 2024-08-17T17:54:11.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.421230+0000", 2024-08-17T17:54:11.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013754899999999999 2024-08-17T17:54:11.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.421274+0000", 2024-08-17T17:54:11.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4211000000000001e-05 2024-08-17T17:54:11.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.421316+0000", 2024-08-17T17:54:11.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2126000000000003e-05 2024-08-17T17:54:11.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:54:11.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.421386+0000", 2024-08-17T17:54:11.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0457000000000007e-05 2024-08-17T17:54:11.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:54:11.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.422523+0000", 2024-08-17T17:54:11.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001136373 2024-08-17T17:54:11.646 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.646 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:54:11.647 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.647 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:79357 53.13 53:cda71420:::benchmark_data_smithi136_235573_object79356:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-08-17T17:54:11.647 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:54:06.277620+0000", 2024-08-17T17:54:11.647 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.1639012739999997, 2024-08-17T17:54:11.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1460402629999999, 2024-08-17T17:54:11.647 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:54:11.647 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:54:11.647 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:54:11.647 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:54:11.647 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:54:11.647 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:54:11.647 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 79357 2024-08-17T17:54:11.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.648 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:54:11.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:54:11.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277620+0000", 2024-08-17T17:54:11.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:54:11.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277620+0000", 2024-08-17T17:54:11.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:54:11.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277621+0000", 2024-08-17T17:54:11.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5799999999999998e-07 2024-08-17T17:54:11.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:54:11.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277663+0000", 2024-08-17T17:54:11.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2011000000000001e-05 2024-08-17T17:54:11.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:54:11.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277688+0000", 2024-08-17T17:54:11.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5548e-05 2024-08-17T17:54:11.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:54:11.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277703+0000", 2024-08-17T17:54:11.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4347e-05 2024-08-17T17:54:11.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277756+0000", 2024-08-17T17:54:11.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3430999999999997e-05 2024-08-17T17:54:11.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:54:11.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277766+0000", 2024-08-17T17:54:11.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0118999999999999e-05 2024-08-17T17:54:11.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.398116+0000", 2024-08-17T17:54:11.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1203494700000001 2024-08-17T17:54:11.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:54:11.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.398292+0000", 2024-08-17T17:54:11.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000176095 2024-08-17T17:54:11.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,2,3", 2024-08-17T17:54:11.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.398487+0000", 2024-08-17T17:54:11.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019578299999999999 2024-08-17T17:54:11.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:54:11.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.421173+0000", 2024-08-17T17:54:11.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.022685539000000001 2024-08-17T17:54:11.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.422711+0000", 2024-08-17T17:54:11.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001537821 2024-08-17T17:54:11.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.422754+0000", 2024-08-17T17:54:11.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3269000000000002e-05 2024-08-17T17:54:11.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.422813+0000", 2024-08-17T17:54:11.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8486999999999998e-05 2024-08-17T17:54:11.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:54:11.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.422867+0000", 2024-08-17T17:54:11.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4707000000000003e-05 2024-08-17T17:54:11.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:54:11.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.423660+0000", 2024-08-17T17:54:11.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00079277799999999995 2024-08-17T17:54:11.654 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.654 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:54:11.654 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.654 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:79415 53.13 53:cd2d7832:::benchmark_data_smithi136_235573_object79414:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-08-17T17:54:11.654 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:54:06.277718+0000", 2024-08-17T17:54:11.655 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.1638025680000004, 2024-08-17T17:54:11.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.157780756, 2024-08-17T17:54:11.655 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:54:11.655 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:54:11.655 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:54:11.655 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:54:11.655 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:54:11.655 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:54:11.655 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 79415 2024-08-17T17:54:11.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.655 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:54:11.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:54:11.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277718+0000", 2024-08-17T17:54:11.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:54:11.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277718+0000", 2024-08-17T17:54:11.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:54:11.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277719+0000", 2024-08-17T17:54:11.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2699999999999999e-07 2024-08-17T17:54:11.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:54:11.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277769+0000", 2024-08-17T17:54:11.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9688000000000003e-05 2024-08-17T17:54:11.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:54:11.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277777+0000", 2024-08-17T17:54:11.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8299999999999996e-06 2024-08-17T17:54:11.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:54:11.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277792+0000", 2024-08-17T17:54:11.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5502000000000002e-05 2024-08-17T17:54:11.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277836+0000", 2024-08-17T17:54:11.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3686000000000002e-05 2024-08-17T17:54:11.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:54:11.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277843+0000", 2024-08-17T17:54:11.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9260000000000003e-06 2024-08-17T17:54:11.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.423710+0000", 2024-08-17T17:54:11.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1458674259999999 2024-08-17T17:54:11.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:54:11.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.423813+0000", 2024-08-17T17:54:11.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010284400000000001 2024-08-17T17:54:11.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,2,3", 2024-08-17T17:54:11.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.423912+0000", 2024-08-17T17:54:11.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9031999999999996e-05 2024-08-17T17:54:11.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.429244+0000", 2024-08-17T17:54:11.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0053320609999999999 2024-08-17T17:54:11.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.429316+0000", 2024-08-17T17:54:11.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1927000000000004e-05 2024-08-17T17:54:11.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:54:11.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.429336+0000", 2024-08-17T17:54:11.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9984999999999999e-05 2024-08-17T17:54:11.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.434047+0000", 2024-08-17T17:54:11.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0047112760000000004 2024-08-17T17:54:11.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:54:11.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.434115+0000", 2024-08-17T17:54:11.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7921000000000002e-05 2024-08-17T17:54:11.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:54:11.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.435499+0000", 2024-08-17T17:54:11.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0013840250000000001 2024-08-17T17:54:11.662 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.662 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:54:11.662 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.662 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:79417 53.13 53:cba2b271:::benchmark_data_smithi136_235573_object79416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-08-17T17:54:11.662 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:54:06.277806+0000", 2024-08-17T17:54:11.663 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.163715067, 2024-08-17T17:54:11.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1673137810000001, 2024-08-17T17:54:11.663 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:54:11.663 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:54:11.663 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:54:11.663 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:54:11.663 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:54:11.663 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:54:11.663 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 79417 2024-08-17T17:54:11.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.663 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:54:11.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:54:11.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277806+0000", 2024-08-17T17:54:11.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:54:11.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277806+0000", 2024-08-17T17:54:11.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:54:11.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277807+0000", 2024-08-17T17:54:11.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8400000000000004e-07 2024-08-17T17:54:11.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:54:11.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277851+0000", 2024-08-17T17:54:11.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4419999999999998e-05 2024-08-17T17:54:11.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:54:11.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277858+0000", 2024-08-17T17:54:11.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0190000000000001e-06 2024-08-17T17:54:11.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:54:11.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277863+0000", 2024-08-17T17:54:11.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4129999999999997e-06 2024-08-17T17:54:11.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277909+0000", 2024-08-17T17:54:11.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5210000000000003e-05 2024-08-17T17:54:11.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:54:11.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277924+0000", 2024-08-17T17:54:11.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5165e-05 2024-08-17T17:54:11.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.425070+0000", 2024-08-17T17:54:11.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.147146695 2024-08-17T17:54:11.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:54:11.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.425226+0000", 2024-08-17T17:54:11.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015576300000000001 2024-08-17T17:54:11.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,2,3", 2024-08-17T17:54:11.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.425401+0000", 2024-08-17T17:54:11.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017507800000000001 2024-08-17T17:54:11.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.435559+0000", 2024-08-17T17:54:11.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010157571000000001 2024-08-17T17:54:11.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.435620+0000", 2024-08-17T17:54:11.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1080999999999993e-05 2024-08-17T17:54:11.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.435661+0000", 2024-08-17T17:54:11.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0608999999999997e-05 2024-08-17T17:54:11.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:54:11.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.443776+0000", 2024-08-17T17:54:11.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0081157199999999999 2024-08-17T17:54:11.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:54:11.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.443839+0000", 2024-08-17T17:54:11.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2460999999999997e-05 2024-08-17T17:54:11.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:54:11.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.445120+0000", 2024-08-17T17:54:11.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012809919999999999 2024-08-17T17:54:11.670 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.670 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:54:11.670 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.670 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:79421 53.13 53:cc533114:::benchmark_data_smithi136_235573_object79420:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-08-17T17:54:11.670 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:54:06.277877+0000", 2024-08-17T17:54:11.670 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.1636439239999996, 2024-08-17T17:54:11.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.173700754, 2024-08-17T17:54:11.670 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:54:11.670 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:54:11.671 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:54:11.671 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:54:11.671 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:54:11.671 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:54:11.671 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 79421 2024-08-17T17:54:11.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.671 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:54:11.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:54:11.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277877+0000", 2024-08-17T17:54:11.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:54:11.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277877+0000", 2024-08-17T17:54:11.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:54:11.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277878+0000", 2024-08-17T17:54:11.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9299999999999998e-07 2024-08-17T17:54:11.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:54:11.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277922+0000", 2024-08-17T17:54:11.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4478999999999999e-05 2024-08-17T17:54:11.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:54:11.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277930+0000", 2024-08-17T17:54:11.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8590000000000005e-06 2024-08-17T17:54:11.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:54:11.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277936+0000", 2024-08-17T17:54:11.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5820000000000001e-06 2024-08-17T17:54:11.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277973+0000", 2024-08-17T17:54:11.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7317999999999997e-05 2024-08-17T17:54:11.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:54:11.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277979+0000", 2024-08-17T17:54:11.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5359999999999998e-06 2024-08-17T17:54:11.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.427210+0000", 2024-08-17T17:54:11.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1492304090000001 2024-08-17T17:54:11.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:54:11.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.427375+0000", 2024-08-17T17:54:11.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000164945 2024-08-17T17:54:11.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,2,3", 2024-08-17T17:54:11.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.427543+0000", 2024-08-17T17:54:11.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016769300000000001 2024-08-17T17:54:11.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.435709+0000", 2024-08-17T17:54:11.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0081668750000000005 2024-08-17T17:54:11.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.435755+0000", 2024-08-17T17:54:11.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5102999999999998e-05 2024-08-17T17:54:11.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:54:11.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.445193+0000", 2024-08-17T17:54:11.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0094383910000000008 2024-08-17T17:54:11.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.450226+0000", 2024-08-17T17:54:11.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005033498 2024-08-17T17:54:11.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:54:11.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.450287+0000", 2024-08-17T17:54:11.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1080000000000005e-05 2024-08-17T17:54:11.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:54:11.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.451578+0000", 2024-08-17T17:54:11.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012903929999999999 2024-08-17T17:54:11.677 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.678 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:54:11.678 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.678 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:79423 53.13 53:cb2249fa:::benchmark_data_smithi136_235573_object79422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-08-17T17:54:11.678 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:54:06.277949+0000", 2024-08-17T17:54:11.678 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.1635721969999997, 2024-08-17T17:54:11.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1822028499999999, 2024-08-17T17:54:11.678 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:54:11.678 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:54:11.678 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:54:11.678 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:54:11.679 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:54:11.679 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:54:11.679 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 79423 2024-08-17T17:54:11.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.679 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:54:11.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:54:11.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277949+0000", 2024-08-17T17:54:11.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:54:11.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277949+0000", 2024-08-17T17:54:11.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:54:11.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.277949+0000", 2024-08-17T17:54:11.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.51e-07 2024-08-17T17:54:11.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:54:11.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278333+0000", 2024-08-17T17:54:11.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000383462 2024-08-17T17:54:11.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:54:11.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278343+0000", 2024-08-17T17:54:11.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8449999999999998e-06 2024-08-17T17:54:11.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:54:11.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278350+0000", 2024-08-17T17:54:11.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8759999999999999e-06 2024-08-17T17:54:11.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278414+0000", 2024-08-17T17:54:11.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4772999999999998e-05 2024-08-17T17:54:11.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:54:11.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278423+0000", 2024-08-17T17:54:11.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6889999999999993e-06 2024-08-17T17:54:11.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.429392+0000", 2024-08-17T17:54:11.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1509690909999999 2024-08-17T17:54:11.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:54:11.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.429550+0000", 2024-08-17T17:54:11.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000158076 2024-08-17T17:54:11.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,2,3", 2024-08-17T17:54:11.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.429730+0000", 2024-08-17T17:54:11.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000179765 2024-08-17T17:54:11.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.440665+0000", 2024-08-17T17:54:11.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010935201 2024-08-17T17:54:11.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.445178+0000", 2024-08-17T17:54:11.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0045131449999999997 2024-08-17T17:54:11.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:54:11.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.445199+0000", 2024-08-17T17:54:11.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0492e-05 2024-08-17T17:54:11.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.458548+0000", 2024-08-17T17:54:11.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013349211 2024-08-17T17:54:11.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:54:11.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.458610+0000", 2024-08-17T17:54:11.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2173000000000003e-05 2024-08-17T17:54:11.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:54:11.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.460152+0000", 2024-08-17T17:54:11.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015414999999999999 2024-08-17T17:54:11.685 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.685 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:54:11.685 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.685 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:79473 53.13 53:cecdd8dd:::benchmark_data_smithi136_235573_object79472:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-08-17T17:54:11.685 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:54:06.278362+0000", 2024-08-17T17:54:11.685 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.1631591920000002, 2024-08-17T17:54:11.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.182773415, 2024-08-17T17:54:11.686 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:54:11.686 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:54:11.686 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:54:11.686 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:54:11.686 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:54:11.686 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:54:11.686 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 79473 2024-08-17T17:54:11.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.686 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:54:11.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:54:11.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278362+0000", 2024-08-17T17:54:11.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:54:11.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278362+0000", 2024-08-17T17:54:11.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:54:11.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278363+0000", 2024-08-17T17:54:11.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8599999999999998e-07 2024-08-17T17:54:11.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:54:11.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278410+0000", 2024-08-17T17:54:11.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7330999999999998e-05 2024-08-17T17:54:11.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:54:11.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278418+0000", 2024-08-17T17:54:11.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4039999999999995e-06 2024-08-17T17:54:11.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:54:11.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278424+0000", 2024-08-17T17:54:11.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0070000000000003e-06 2024-08-17T17:54:11.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278741+0000", 2024-08-17T17:54:11.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00031671999999999999 2024-08-17T17:54:11.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:54:11.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278748+0000", 2024-08-17T17:54:11.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3719999999999999e-06 2024-08-17T17:54:11.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.432343+0000", 2024-08-17T17:54:11.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.153594628 2024-08-17T17:54:11.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:54:11.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.432540+0000", 2024-08-17T17:54:11.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019675000000000001 2024-08-17T17:54:11.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,2,3", 2024-08-17T17:54:11.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.432760+0000", 2024-08-17T17:54:11.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022006 2024-08-17T17:54:11.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.440717+0000", 2024-08-17T17:54:11.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0079574369999999995 2024-08-17T17:54:11.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:54:11.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.445205+0000", 2024-08-17T17:54:11.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0044874850000000003 2024-08-17T17:54:11.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.445275+0000", 2024-08-17T17:54:11.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0528e-05 2024-08-17T17:54:11.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.460202+0000", 2024-08-17T17:54:11.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014927042999999999 2024-08-17T17:54:11.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:54:11.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.460258+0000", 2024-08-17T17:54:11.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5278999999999997e-05 2024-08-17T17:54:11.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:54:11.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.461135+0000", 2024-08-17T17:54:11.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00087768500000000001 2024-08-17T17:54:11.692 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.692 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:54:11.692 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.692 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:79547 53.13 53:c80b7e94:::benchmark_data_smithi136_235573_object79546:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-08-17T17:54:11.692 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:54:06.278467+0000", 2024-08-17T17:54:11.692 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.1630541770000002, 2024-08-17T17:54:11.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.184686999, 2024-08-17T17:54:11.693 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:54:11.693 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:54:11.693 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:54:11.693 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:54:11.693 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:54:11.693 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:54:11.693 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 79547 2024-08-17T17:54:11.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.693 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:54:11.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:54:11.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278467+0000", 2024-08-17T17:54:11.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:54:11.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278467+0000", 2024-08-17T17:54:11.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:54:11.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278467+0000", 2024-08-17T17:54:11.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3900000000000005e-07 2024-08-17T17:54:11.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:54:11.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278513+0000", 2024-08-17T17:54:11.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5541000000000003e-05 2024-08-17T17:54:11.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:54:11.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278521+0000", 2024-08-17T17:54:11.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0029999999999994e-06 2024-08-17T17:54:11.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:54:11.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278527+0000", 2024-08-17T17:54:11.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7389999999999996e-06 2024-08-17T17:54:11.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278774+0000", 2024-08-17T17:54:11.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024720599999999998 2024-08-17T17:54:11.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:54:11.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278780+0000", 2024-08-17T17:54:11.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0249999999999999e-06 2024-08-17T17:54:11.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.435788+0000", 2024-08-17T17:54:11.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.157008091 2024-08-17T17:54:11.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:54:11.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.435962+0000", 2024-08-17T17:54:11.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000174286 2024-08-17T17:54:11.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,2,3", 2024-08-17T17:54:11.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.436151+0000", 2024-08-17T17:54:11.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018874500000000001 2024-08-17T17:54:11.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:54:11.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.445211+0000", 2024-08-17T17:54:11.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0090599800000000005 2024-08-17T17:54:11.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.450174+0000", 2024-08-17T17:54:11.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0049627489999999998 2024-08-17T17:54:11.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.461185+0000", 2024-08-17T17:54:11.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011011514 2024-08-17T17:54:11.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.461554+0000", 2024-08-17T17:54:11.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036872899999999999 2024-08-17T17:54:11.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:54:11.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.461662+0000", 2024-08-17T17:54:11.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107742 2024-08-17T17:54:11.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:54:11.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.463154+0000", 2024-08-17T17:54:11.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014921100000000001 2024-08-17T17:54:11.700 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.700 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:54:11.700 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.700 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:79579 53.13 53:ccf2e78e:::benchmark_data_smithi136_235573_object79578:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-08-17T17:54:11.700 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:54:06.278535+0000", 2024-08-17T17:54:11.700 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.1629863130000002, 2024-08-17T17:54:11.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1877677520000001, 2024-08-17T17:54:11.700 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:54:11.700 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:54:11.700 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:54:11.700 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:54:11.701 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:54:11.701 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:54:11.701 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 79579 2024-08-17T17:54:11.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.701 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:54:11.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:54:11.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278535+0000", 2024-08-17T17:54:11.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:54:11.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278535+0000", 2024-08-17T17:54:11.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:54:11.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278535+0000", 2024-08-17T17:54:11.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.75e-07 2024-08-17T17:54:11.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:54:11.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278575+0000", 2024-08-17T17:54:11.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9688999999999999e-05 2024-08-17T17:54:11.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:54:11.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278582+0000", 2024-08-17T17:54:11.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4059999999999999e-06 2024-08-17T17:54:11.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:54:11.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278588+0000", 2024-08-17T17:54:11.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2429999999999996e-06 2024-08-17T17:54:11.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278803+0000", 2024-08-17T17:54:11.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021531600000000001 2024-08-17T17:54:11.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:54:11.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278809+0000", 2024-08-17T17:54:11.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0340000000000002e-06 2024-08-17T17:54:11.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.437422+0000", 2024-08-17T17:54:11.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.158613248 2024-08-17T17:54:11.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:54:11.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.437585+0000", 2024-08-17T17:54:11.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000162984 2024-08-17T17:54:11.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,2,3", 2024-08-17T17:54:11.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.437769+0000", 2024-08-17T17:54:11.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018401600000000001 2024-08-17T17:54:11.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:54:11.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.445217+0000", 2024-08-17T17:54:11.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0074479380000000003 2024-08-17T17:54:11.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.451634+0000", 2024-08-17T17:54:11.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0064166539999999999 2024-08-17T17:54:11.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.461226+0000", 2024-08-17T17:54:11.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0095920769999999992 2024-08-17T17:54:11.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.464720+0000", 2024-08-17T17:54:11.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034941170000000001 2024-08-17T17:54:11.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:54:11.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.464783+0000", 2024-08-17T17:54:11.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2937999999999999e-05 2024-08-17T17:54:11.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:54:11.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.466302+0000", 2024-08-17T17:54:11.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001519517 2024-08-17T17:54:11.707 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.707 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:54:11.707 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.707 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:79650 53.13 53:cc8ab36f:::benchmark_data_smithi136_235573_object79649:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-08-17T17:54:11.707 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:54:06.278595+0000", 2024-08-17T17:54:11.707 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.162926025, 2024-08-17T17:54:11.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1887913669999999, 2024-08-17T17:54:11.708 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:54:11.708 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:54:11.708 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:54:11.708 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:54:11.708 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:54:11.708 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:54:11.708 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 79650 2024-08-17T17:54:11.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.708 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:54:11.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:54:11.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278595+0000", 2024-08-17T17:54:11.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:54:11.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278595+0000", 2024-08-17T17:54:11.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:54:11.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278596+0000", 2024-08-17T17:54:11.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0800000000000005e-07 2024-08-17T17:54:11.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:54:11.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278643+0000", 2024-08-17T17:54:11.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7268999999999998e-05 2024-08-17T17:54:11.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:54:11.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278651+0000", 2024-08-17T17:54:11.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5010000000000004e-06 2024-08-17T17:54:11.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:54:11.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278657+0000", 2024-08-17T17:54:11.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4589999999999997e-06 2024-08-17T17:54:11.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278831+0000", 2024-08-17T17:54:11.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017468599999999999 2024-08-17T17:54:11.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:54:11.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278838+0000", 2024-08-17T17:54:11.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2060000000000004e-06 2024-08-17T17:54:11.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.439199+0000", 2024-08-17T17:54:11.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.160361537 2024-08-17T17:54:11.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:54:11.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.439348+0000", 2024-08-17T17:54:11.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014882400000000001 2024-08-17T17:54:11.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,2,3", 2024-08-17T17:54:11.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.439489+0000", 2024-08-17T17:54:11.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014063300000000001 2024-08-17T17:54:11.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:54:11.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.445223+0000", 2024-08-17T17:54:11.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0057345510000000001 2024-08-17T17:54:11.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.458180+0000", 2024-08-17T17:54:11.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012956924 2024-08-17T17:54:11.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.463286+0000", 2024-08-17T17:54:11.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0051055509999999998 2024-08-17T17:54:11.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.466357+0000", 2024-08-17T17:54:11.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0030713680000000001 2024-08-17T17:54:11.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:54:11.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.466473+0000", 2024-08-17T17:54:11.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115691 2024-08-17T17:54:11.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:54:11.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.467386+0000", 2024-08-17T17:54:11.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00091365900000000004 2024-08-17T17:54:11.715 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.715 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:54:11.715 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.715 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:79695 53.13 53:cd3c65e8:::benchmark_data_smithi136_235573_object79694:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-08-17T17:54:11.715 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:54:06.278667+0000", 2024-08-17T17:54:11.715 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.1628545189999997, 2024-08-17T17:54:11.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.190385684, 2024-08-17T17:54:11.715 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:54:11.715 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:54:11.715 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:54:11.715 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:54:11.715 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:54:11.716 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:54:11.716 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 79695 2024-08-17T17:54:11.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.716 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:54:11.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:54:11.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278667+0000", 2024-08-17T17:54:11.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:54:11.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278667+0000", 2024-08-17T17:54:11.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:54:11.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278667+0000", 2024-08-17T17:54:11.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8800000000000003e-07 2024-08-17T17:54:11.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:54:11.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278722+0000", 2024-08-17T17:54:11.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5482999999999997e-05 2024-08-17T17:54:11.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:54:11.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278730+0000", 2024-08-17T17:54:11.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5449999999999998e-06 2024-08-17T17:54:11.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:54:11.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278736+0000", 2024-08-17T17:54:11.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.772e-06 2024-08-17T17:54:11.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278861+0000", 2024-08-17T17:54:11.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125666 2024-08-17T17:54:11.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:54:11.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278868+0000", 2024-08-17T17:54:11.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0759999999999999e-06 2024-08-17T17:54:11.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.440749+0000", 2024-08-17T17:54:11.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.161881119 2024-08-17T17:54:11.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:54:11.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.440901+0000", 2024-08-17T17:54:11.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000152039 2024-08-17T17:54:11.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,2,3", 2024-08-17T17:54:11.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.441057+0000", 2024-08-17T17:54:11.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015676399999999999 2024-08-17T17:54:11.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:54:11.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.445229+0000", 2024-08-17T17:54:11.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0041719740000000002 2024-08-17T17:54:11.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.461511+0000", 2024-08-17T17:54:11.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016281393000000002 2024-08-17T17:54:11.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.464666+0000", 2024-08-17T17:54:11.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031547150000000002 2024-08-17T17:54:11.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.467707+0000", 2024-08-17T17:54:11.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003041219 2024-08-17T17:54:11.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:54:11.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.467761+0000", 2024-08-17T17:54:11.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4181999999999997e-05 2024-08-17T17:54:11.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:54:11.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.469052+0000", 2024-08-17T17:54:11.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001291249 2024-08-17T17:54:11.722 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.722 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:54:11.722 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.722 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:79723 53.13 53:c8a8a178:::benchmark_data_smithi136_235573_object79722:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-08-17T17:54:11.722 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:54:06.278744+0000", 2024-08-17T17:54:11.723 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.1627771210000004, 2024-08-17T17:54:11.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.19176302, 2024-08-17T17:54:11.723 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:54:11.723 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:54:11.723 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:54:11.723 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:54:11.723 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:54:11.723 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:54:11.723 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 79723 2024-08-17T17:54:11.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.723 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:54:11.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:54:11.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278744+0000", 2024-08-17T17:54:11.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:54:11.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278744+0000", 2024-08-17T17:54:11.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:54:11.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278744+0000", 2024-08-17T17:54:11.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9200000000000001e-07 2024-08-17T17:54:11.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:54:11.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278788+0000", 2024-08-17T17:54:11.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3528000000000001e-05 2024-08-17T17:54:11.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:54:11.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278795+0000", 2024-08-17T17:54:11.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0979999999999996e-06 2024-08-17T17:54:11.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:54:11.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278800+0000", 2024-08-17T17:54:11.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3820000000000003e-06 2024-08-17T17:54:11.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278890+0000", 2024-08-17T17:54:11.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0037999999999994e-05 2024-08-17T17:54:11.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:54:11.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278896+0000", 2024-08-17T17:54:11.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6110000000000001e-06 2024-08-17T17:54:11.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.442230+0000", 2024-08-17T17:54:11.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.16333369 2024-08-17T17:54:11.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:54:11.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.442424+0000", 2024-08-17T17:54:11.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019412699999999999 2024-08-17T17:54:11.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,2,3", 2024-08-17T17:54:11.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.442621+0000", 2024-08-17T17:54:11.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0001974 2024-08-17T17:54:11.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:54:11.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.445236+0000", 2024-08-17T17:54:11.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0026144060000000001 2024-08-17T17:54:11.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.463210+0000", 2024-08-17T17:54:11.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017973962999999999 2024-08-17T17:54:11.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.469163+0000", 2024-08-17T17:54:11.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0059536220000000004 2024-08-17T17:54:11.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.469260+0000", 2024-08-17T17:54:11.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6892000000000007e-05 2024-08-17T17:54:11.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:54:11.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.469325+0000", 2024-08-17T17:54:11.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4842e-05 2024-08-17T17:54:11.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:54:11.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.470507+0000", 2024-08-17T17:54:11.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001181929 2024-08-17T17:54:11.729 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.729 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:54:11.730 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.730 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:79868 53.13 53:ce21af0b:::benchmark_data_smithi136_235573_object79867:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-08-17T17:54:11.730 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:54:06.278807+0000", 2024-08-17T17:54:11.730 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.1627137310000002, 2024-08-17T17:54:11.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1971192450000001, 2024-08-17T17:54:11.730 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:54:11.730 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:54:11.730 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:54:11.730 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:54:11.730 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:54:11.730 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:54:11.730 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 79868 2024-08-17T17:54:11.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.731 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:54:11.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:54:11.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278807+0000", 2024-08-17T17:54:11.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:54:11.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278807+0000", 2024-08-17T17:54:11.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:54:11.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278808+0000", 2024-08-17T17:54:11.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.89e-07 2024-08-17T17:54:11.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:54:11.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278851+0000", 2024-08-17T17:54:11.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3538999999999998e-05 2024-08-17T17:54:11.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:54:11.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278858+0000", 2024-08-17T17:54:11.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9589999999999998e-06 2024-08-17T17:54:11.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:54:11.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278863+0000", 2024-08-17T17:54:11.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1850000000000003e-06 2024-08-17T17:54:11.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278996+0000", 2024-08-17T17:54:11.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013236300000000001 2024-08-17T17:54:11.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:54:11.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.279002+0000", 2024-08-17T17:54:11.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4080000000000003e-06 2024-08-17T17:54:11.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.445307+0000", 2024-08-17T17:54:11.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1663051449999999 2024-08-17T17:54:11.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:54:11.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.445461+0000", 2024-08-17T17:54:11.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015353799999999999 2024-08-17T17:54:11.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,2,3", 2024-08-17T17:54:11.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.445631+0000", 2024-08-17T17:54:11.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016989800000000001 2024-08-17T17:54:11.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:54:11.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.450128+0000", 2024-08-17T17:54:11.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0044976540000000002 2024-08-17T17:54:11.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.463248+0000", 2024-08-17T17:54:11.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013119789 2024-08-17T17:54:11.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.470575+0000", 2024-08-17T17:54:11.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0073268550000000002 2024-08-17T17:54:11.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.471056+0000", 2024-08-17T17:54:11.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00048128500000000002 2024-08-17T17:54:11.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:54:11.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.471463+0000", 2024-08-17T17:54:11.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00040684599999999999 2024-08-17T17:54:11.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:54:11.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.475927+0000", 2024-08-17T17:54:11.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0044632919999999998 2024-08-17T17:54:11.737 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.737 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:54:11.737 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.737 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:79894 53.13 53:cce0089f:::benchmark_data_smithi136_235573_object79893:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-08-17T17:54:11.737 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:54:06.278870+0000", 2024-08-17T17:54:11.737 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.1626506609999998, 2024-08-17T17:54:11.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1993731190000001, 2024-08-17T17:54:11.737 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:54:11.738 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:54:11.738 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:54:11.738 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:54:11.738 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:54:11.738 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:54:11.738 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 79894 2024-08-17T17:54:11.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.738 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:54:11.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:54:11.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278870+0000", 2024-08-17T17:54:11.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:54:11.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278870+0000", 2024-08-17T17:54:11.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:54:11.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278871+0000", 2024-08-17T17:54:11.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0500000000000004e-07 2024-08-17T17:54:11.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:54:11.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278915+0000", 2024-08-17T17:54:11.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4540999999999999e-05 2024-08-17T17:54:11.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:54:11.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278921+0000", 2024-08-17T17:54:11.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7289999999999997e-06 2024-08-17T17:54:11.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:54:11.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278926+0000", 2024-08-17T17:54:11.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.955e-06 2024-08-17T17:54:11.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.279025+0000", 2024-08-17T17:54:11.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8913999999999995e-05 2024-08-17T17:54:11.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:54:11.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.279031+0000", 2024-08-17T17:54:11.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2929999999999997e-06 2024-08-17T17:54:11.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.446880+0000", 2024-08-17T17:54:11.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1678486969999999 2024-08-17T17:54:11.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:54:11.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.447041+0000", 2024-08-17T17:54:11.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016054200000000001 2024-08-17T17:54:11.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,2,3", 2024-08-17T17:54:11.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.447201+0000", 2024-08-17T17:54:11.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016066100000000001 2024-08-17T17:54:11.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:54:11.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.450188+0000", 2024-08-17T17:54:11.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0029869979999999998 2024-08-17T17:54:11.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.467659+0000", 2024-08-17T17:54:11.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017470511000000001 2024-08-17T17:54:11.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.470621+0000", 2024-08-17T17:54:11.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0029618399999999999 2024-08-17T17:54:11.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.475982+0000", 2024-08-17T17:54:11.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005361106 2024-08-17T17:54:11.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:54:11.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.476038+0000", 2024-08-17T17:54:11.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6076999999999999e-05 2024-08-17T17:54:11.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:54:11.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.478244+0000", 2024-08-17T17:54:11.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0022057499999999998 2024-08-17T17:54:11.744 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.745 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:54:11.745 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.745 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:79902 53.13 53:cb47b7c6:::benchmark_data_smithi136_235573_object79901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1323)", 2024-08-17T17:54:11.745 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:54:06.278937+0000", 2024-08-17T17:54:11.745 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.1625842449999997, 2024-08-17T17:54:11.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2007815980000001, 2024-08-17T17:54:11.745 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:54:11.745 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:54:11.745 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:54:11.745 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:54:11.746 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:54:11.746 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:54:11.746 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 79902 2024-08-17T17:54:11.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.746 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:54:11.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:54:11.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278937+0000", 2024-08-17T17:54:11.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:54:11.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278937+0000", 2024-08-17T17:54:11.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:54:11.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278937+0000", 2024-08-17T17:54:11.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4199999999999996e-07 2024-08-17T17:54:11.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:54:11.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278981+0000", 2024-08-17T17:54:11.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3658000000000001e-05 2024-08-17T17:54:11.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:54:11.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278988+0000", 2024-08-17T17:54:11.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0879999999999997e-06 2024-08-17T17:54:11.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:54:11.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.278994+0000", 2024-08-17T17:54:11.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5890000000000002e-06 2024-08-17T17:54:11.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.279053+0000", 2024-08-17T17:54:11.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9333000000000003e-05 2024-08-17T17:54:11.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:54:11.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.279059+0000", 2024-08-17T17:54:11.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5890000000000002e-06 2024-08-17T17:54:11.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.448402+0000", 2024-08-17T17:54:11.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.169343206 2024-08-17T17:54:11.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:54:11.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.448590+0000", 2024-08-17T17:54:11.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018812700000000001 2024-08-17T17:54:11.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,2,3", 2024-08-17T17:54:11.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.448821+0000", 2024-08-17T17:54:11.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000231556 2024-08-17T17:54:11.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:54:11.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.453788+0000", 2024-08-17T17:54:11.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0049668910000000002 2024-08-17T17:54:11.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.469112+0000", 2024-08-17T17:54:11.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015323363 2024-08-17T17:54:11.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.470671+0000", 2024-08-17T17:54:11.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015592290000000001 2024-08-17T17:54:11.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.478313+0000", 2024-08-17T17:54:11.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0076417250000000003 2024-08-17T17:54:11.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:54:11.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.478489+0000", 2024-08-17T17:54:11.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017635099999999999 2024-08-17T17:54:11.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:54:11.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.479718+0000", 2024-08-17T17:54:11.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012293510000000001 2024-08-17T17:54:11.753 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.753 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:54:11.753 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.753 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:79977 53.13 53:c9f671cf:::benchmark_data_smithi136_235573_object79976:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1323)", 2024-08-17T17:54:11.753 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:54:06.426124+0000", 2024-08-17T17:54:11.753 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.0153969590000003, 2024-08-17T17:54:11.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0547452260000001, 2024-08-17T17:54:11.753 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:54:11.753 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:54:11.754 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:54:11.754 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:54:11.754 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:54:11.754 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:54:11.754 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 79977 2024-08-17T17:54:11.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.754 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:54:11.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:54:11.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.426124+0000", 2024-08-17T17:54:11.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:54:11.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.426124+0000", 2024-08-17T17:54:11.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:11.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:54:11.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.426125+0000", 2024-08-17T17:54:11.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5000000000000001e-07 2024-08-17T17:54:11.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:54:11.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.426144+0000", 2024-08-17T17:54:11.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9343000000000001e-05 2024-08-17T17:54:11.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:54:11.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.426152+0000", 2024-08-17T17:54:11.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9379999999999992e-06 2024-08-17T17:54:11.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:54:11.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.426157+0000", 2024-08-17T17:54:11.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7720000000000002e-06 2024-08-17T17:54:11.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.426203+0000", 2024-08-17T17:54:11.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6196999999999998e-05 2024-08-17T17:54:11.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:54:11.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:06.426209+0000", 2024-08-17T17:54:11.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.314e-06 2024-08-17T17:54:11.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:11.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.451673+0000", 2024-08-17T17:54:11.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.025463754 2024-08-17T17:54:11.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:54:11.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.451830+0000", 2024-08-17T17:54:11.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015686699999999999 2024-08-17T17:54:11.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,2,3", 2024-08-17T17:54:11.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.452000+0000", 2024-08-17T17:54:11.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016954000000000001 2024-08-17T17:54:11.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:54:11.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.454534+0000", 2024-08-17T17:54:11.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025345540000000001 2024-08-17T17:54:11.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.469212+0000", 2024-08-17T17:54:11.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014677492 2024-08-17T17:54:11.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.470804+0000", 2024-08-17T17:54:11.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015921139999999999 2024-08-17T17:54:11.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:11.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.479786+0000", 2024-08-17T17:54:11.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0089824789999999998 2024-08-17T17:54:11.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:54:11.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.479844+0000", 2024-08-17T17:54:11.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7535999999999999e-05 2024-08-17T17:54:11.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:11.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:11.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:54:11.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:07.480869+0000", 2024-08-17T17:54:11.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001025476 2024-08-17T17:54:11.760 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.760 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:54:11.760 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.760 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:11.760 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:54:11.760 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:54:11.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:11.762 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 54 16 85506 85490 98.9285 78.0625 0.026437 0.0101036 2024-08-17T17:54:11.790 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:11.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:11.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:11.892 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:11.930 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:11.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:11.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:12.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:12.063 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:12.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:12.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:12.165 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:12.207 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:12.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:12.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:12.277 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T17:54:12.278 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:54:12.283 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T17:54:12.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:12.322 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:12 smithi136 ceph-mon[109861]: pgmap v2124: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 95 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 54 MiB/s wr, 856 op/s; 9658/242409 objects degraded (3.984%); 10821/242409 objects misplaced (4.464%); 7.1 MiB/s, 112 objects/s recovering 2024-08-17T17:54:12.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:12 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1068388823' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:54:12.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:12 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:54:12.323 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:12 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3742982024' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:54:12.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:12 smithi136 ceph-mon[115078]: pgmap v2124: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 95 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 54 MiB/s wr, 856 op/s; 9658/242409 objects degraded (3.984%); 10821/242409 objects misplaced (4.464%); 7.1 MiB/s, 112 objects/s recovering 2024-08-17T17:54:12.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:12 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1068388823' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:54:12.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:12 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:54:12.323 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:12 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3742982024' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:54:12.324 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T17:54:12.324 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:54:12.331 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:12.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:12.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:12.368 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T17:54:12.368 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:54:12.408 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T17:54:12.408 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:54:12.433 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:12.442 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T17:54:12.443 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:54:12.476 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T17:54:12.476 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:54:12.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:12.488 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:12.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:12.507 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T17:54:12.507 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:54:12.538 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T17:54:12.538 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:54:12.589 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:12.624 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:12.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:12.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:12.708 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 55 15 86748 86733 98.5422 77.6875 0.00881311 0.0101428 2024-08-17T17:54:12.726 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:12 smithi165 ceph-mon[105517]: pgmap v2124: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 95 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 54 MiB/s wr, 856 op/s; 9658/242409 objects degraded (3.984%); 10821/242409 objects misplaced (4.464%); 7.1 MiB/s, 112 objects/s recovering 2024-08-17T17:54:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:12 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1068388823' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:54:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:12 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:54:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:12 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3742982024' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:54:12.768 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:12.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:12.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:12.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:12.913 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:12.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:12.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:13.016 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:13.046 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:13.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:13.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:13.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:13.175 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:13.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:13.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:13.277 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:13.317 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:13.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:13.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:13.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:13.447 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:13.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:13.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:13.549 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:13.595 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:13.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:13.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:13.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:13 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:54:13.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:13 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:54:13.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:13 smithi136 ceph-mon[109861]: osdmap e1325: 8 total, 8 up, 8 in 2024-08-17T17:54:13.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:13 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:54:13.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:13 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:54:13.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:13 smithi136 ceph-mon[115078]: osdmap e1325: 8 total, 8 up, 8 in 2024-08-17T17:54:13.697 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:13.708 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 56 16 88064 88048 98.2501 82.1875 0.0174597 0.0101729 2024-08-17T17:54:13.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:13.729 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:13.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:13.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:13 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:54:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:13 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:54:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:13 smithi165 ceph-mon[105517]: osdmap e1325: 8 total, 8 up, 8 in 2024-08-17T17:54:13.831 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:13.867 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:13.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:13.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:13.969 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:13.995 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:13.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:13.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:14.097 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:14.125 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:14.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:14.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:14.145 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-17T17:54:14.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:14.252 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:14.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:14.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:14.354 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:14.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:14.401 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:14.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:14.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:14 smithi136 ceph-mon[109861]: pgmap v2126: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s; 10789/254601 objects degraded (4.238%); 8524/254601 objects misplaced (3.348%); 5.8 MiB/s, 92 objects/s recovering 2024-08-17T17:54:14.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:14 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 10789/254601 objects degraded (4.238%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:14.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:14 smithi136 ceph-mon[115078]: pgmap v2126: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s; 10789/254601 objects degraded (4.238%); 8524/254601 objects misplaced (3.348%); 5.8 MiB/s, 92 objects/s recovering 2024-08-17T17:54:14.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:14 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 10789/254601 objects degraded (4.238%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:14.502 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:14.533 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:14.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:14.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:14.635 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:14.685 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:14.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:14.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:14.710 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 57 16 89383 89367 97.9698 82.4375 0.016428 0.0102024 2024-08-17T17:54:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:14 smithi165 ceph-mon[105517]: pgmap v2126: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s; 10789/254601 objects degraded (4.238%); 8524/254601 objects misplaced (3.348%); 5.8 MiB/s, 92 objects/s recovering 2024-08-17T17:54:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:14 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 10789/254601 objects degraded (4.238%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:14.788 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:14.811 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:14.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:14.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:14.912 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:14.946 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:14.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:14.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:15.048 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:15.074 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:15.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:15.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:15.177 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:15.210 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:15.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:15.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:15.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:15.338 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:15.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:15.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:15.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:15.481 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:15.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:15.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:15.583 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:15.618 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:15.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:15.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:15.710 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 58 16 90501 90485 97.4852 69.875 0.0106649 0.0102531 2024-08-17T17:54:15.720 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:15.754 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:15.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:15.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:15.856 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:15.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:15.893 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:15.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:15.994 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:16.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:16.039 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:16.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:16.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:16.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:16.174 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:16.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:16.275 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:16.321 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:16.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:16.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:16.422 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:16.461 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:16.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:16.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:16.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:16 smithi165 ceph-mon[105517]: pgmap v2127: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 55 MiB/s wr, 880 op/s; 10789/254601 objects degraded (4.238%); 8524/254601 objects misplaced (3.348%); 4.6 MiB/s, 72 objects/s recovering 2024-08-17T17:54:16.563 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:16.599 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:16.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:16.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:16.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:16 smithi136 ceph-mon[109861]: pgmap v2127: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 55 MiB/s wr, 880 op/s; 10789/254601 objects degraded (4.238%); 8524/254601 objects misplaced (3.348%); 4.6 MiB/s, 72 objects/s recovering 2024-08-17T17:54:16.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:16 smithi136 ceph-mon[115078]: pgmap v2127: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 55 MiB/s wr, 880 op/s; 10789/254601 objects degraded (4.238%); 8524/254601 objects misplaced (3.348%); 4.6 MiB/s, 72 objects/s recovering 2024-08-17T17:54:16.701 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:16.710 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 59 16 91694 91678 97.0966 74.5625 0.0245949 0.0102942 2024-08-17T17:54:16.729 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:16.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:16.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:16.831 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:16.859 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:16.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:16.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:16.961 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:16.982 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:16.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:16.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:17.084 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:17.120 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:17.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:17.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:17.222 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:17.243 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:17.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:17.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:17.345 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:17.376 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:17.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:17.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:17.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:17.520 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:17.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:17.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:17.622 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:17.655 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:17.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:17.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:17.710 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:2024-08-17T17:54:17.711136+0000 min lat: 0.00118032 max lat: 2.13253 avg lat: 0.0103508 2024-08-17T17:54:17.711 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:54:17.711 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 60 16 92739 92723 96.5665 65.3125 0.0055063 0.0103508 2024-08-17T17:54:17.757 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:17.767 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:54:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:54:17] "GET /metrics HTTP/1.1" 200 36558 "" "Prometheus/2.51.0" 2024-08-17T17:54:17.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:17.798 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:17.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:17.899 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:17.938 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:17.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:17.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:18.040 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:18.065 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:18.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:18.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:18.167 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:18.213 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:18.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:18.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:18.315 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:18.366 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:18.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:18.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:18.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:18.511 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:18.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:18.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:18.589 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:18 smithi136 ceph-mon[109861]: pgmap v2128: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 62 MiB/s wr, 997 op/s; 10789/265482 objects degraded (4.064%); 7976/265482 objects misplaced (3.004%); 5.7 MiB/s, 91 objects/s recovering 2024-08-17T17:54:18.590 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:18 smithi136 ceph-mon[115078]: pgmap v2128: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 62 MiB/s wr, 997 op/s; 10789/265482 objects degraded (4.064%); 7976/265482 objects misplaced (3.004%); 5.7 MiB/s, 91 objects/s recovering 2024-08-17T17:54:18.614 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:18.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:18.663 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:18.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:18.711 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 61 16 93834 93818 96.1053 68.4375 0.00749331 0.0103993 2024-08-17T17:54:18.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:18 smithi165 ceph-mon[105517]: pgmap v2128: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 62 MiB/s wr, 997 op/s; 10789/265482 objects degraded (4.064%); 7976/265482 objects misplaced (3.004%); 5.7 MiB/s, 91 objects/s recovering 2024-08-17T17:54:18.765 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:18.793 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:18.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:18.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:18.895 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:18.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:18.942 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:18.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:19.044 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:19.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:19.093 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:19.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:19.194 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:19.239 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:19.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:19.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:19.341 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:19.343 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.0 to 1 (10000) 2024-08-17T17:54:19.368 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:19.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:19.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:19.470 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:19.512 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:19.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:19.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:19.614 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:19.651 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:19.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:19.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:19.672 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:54:19.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:19 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4027140031' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:54:19.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:19 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 10789/265482 objects degraded (4.064%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:19.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:19 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4027140031' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:54:19.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:19 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 10789/265482 objects degraded (4.064%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:19.711 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 62 16 94833 94817 95.562 62.4375 0.0166629 0.0104591 2024-08-17T17:54:19.753 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:19.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:19 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4027140031' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:54:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:19 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 10789/265482 objects degraded (4.064%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:19.780 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:19.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:19.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:19.882 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:19.927 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:19.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:19.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:20.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:20.060 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:20.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:20.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:20.161 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:20.187 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:20.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:20.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:20.289 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:20.338 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:20.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:20.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:20.385 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:54:20.417 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 1 1 2024-08-17T17:54:20.440 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:20.513 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:54:20.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:20.518 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:20.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:20.619 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:20.645 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:20.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:20.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:20.661 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:20 smithi136 ceph-mon[109861]: pgmap v2129: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s; 10789/272976 objects degraded (3.952%); 7976/272976 objects misplaced (2.922%); 5.4 MiB/s, 85 objects/s recovering 2024-08-17T17:54:20.661 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:20 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4027140031' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T17:54:20.661 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:20 smithi136 ceph-mon[109861]: osdmap e1326: 8 total, 8 up, 8 in 2024-08-17T17:54:20.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:20 smithi136 ceph-mon[115078]: pgmap v2129: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s; 10789/272976 objects degraded (3.952%); 7976/272976 objects misplaced (2.922%); 5.4 MiB/s, 85 objects/s recovering 2024-08-17T17:54:20.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:20 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4027140031' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T17:54:20.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:20 smithi136 ceph-mon[115078]: osdmap e1326: 8 total, 8 up, 8 in 2024-08-17T17:54:20.711 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 63 16 95956 95940 95.1591 70.1875 0.0296857 0.0105028 2024-08-17T17:54:20.747 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:20.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:20 smithi165 ceph-mon[105517]: pgmap v2129: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s; 10789/272976 objects degraded (3.952%); 7976/272976 objects misplaced (2.922%); 5.4 MiB/s, 85 objects/s recovering 2024-08-17T17:54:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:20 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4027140031' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T17:54:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:20 smithi165 ceph-mon[105517]: osdmap e1326: 8 total, 8 up, 8 in 2024-08-17T17:54:20.776 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:20.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:20.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:20.878 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:20.922 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:20.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:20.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:21.024 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:21.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:21.070 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:21.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:21.171 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:21.199 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:21.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:21.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:21.301 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:21.347 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:21.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:21.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:21.449 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:21.497 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:21.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:21.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:21.599 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:21.640 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:21.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:21.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:21.711 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 64 16 97169 97153 94.8568 75.8125 0.0127534 0.0105368 2024-08-17T17:54:21.741 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:21.767 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:21.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:21.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:21.869 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:21.895 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:21.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:21.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:21.973 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:54:21.997 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:22.022 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:22.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:22.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:22.124 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:22.157 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:22.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:22.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:22.259 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:22.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:22.290 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:22.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:22.391 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:22.418 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:22.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:22.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:22.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:22 smithi136 ceph-mon[109861]: pgmap v2131: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 62 MiB/s wr, 993 op/s; 10789/279378 objects degraded (3.862%); 7976/279378 objects misplaced (2.855%); 2.1 MiB/s, 32 objects/s recovering 2024-08-17T17:54:22.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:54:22.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:54:22.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:22 smithi136 ceph-mon[115078]: pgmap v2131: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 62 MiB/s wr, 993 op/s; 10789/279378 objects degraded (3.862%); 7976/279378 objects misplaced (2.855%); 2.1 MiB/s, 32 objects/s recovering 2024-08-17T17:54:22.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:54:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:54:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:22 smithi165 ceph-mon[105517]: pgmap v2131: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 62 MiB/s wr, 993 op/s; 10789/279378 objects degraded (3.862%); 7976/279378 objects misplaced (2.855%); 2.1 MiB/s, 32 objects/s recovering 2024-08-17T17:54:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:54:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:54:22.520 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:22.559 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:22.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:22.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:22.661 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:22.689 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:22.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:22.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:22.711 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 65 16 98315 98299 94.4993 71.625 0.00345615 0.0105733 2024-08-17T17:54:22.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:22.820 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:22.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:22.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:22.922 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:22.947 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:22.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:22.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:23.050 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:23.076 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:23.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:23.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:23.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:23.204 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:23.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:23.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:23.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:23.337 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:23.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:23.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:23.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:23.466 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:23.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:23.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:23.568 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:23.592 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:23.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:23.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:23.694 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:23.711 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 66 16 99446 99430 94.1385 70.6875 0.00647591 0.0106177 2024-08-17T17:54:23.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:23.726 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:23.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:23.826 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:23.859 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:23.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:23.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:23.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:23.988 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:23.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:23.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:24.090 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:24.134 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:24.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:24.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:24.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:24.269 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:24.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:24.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:24.371 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:24.402 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:24.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:24.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:24.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:24 smithi136 ceph-mon[109861]: pgmap v2132: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s; 10789/289269 objects degraded (3.730%); 7428/289269 objects misplaced (2.568%); 3.4 MiB/s, 54 objects/s recovering 2024-08-17T17:54:24.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:24 smithi136 ceph-mon[115078]: pgmap v2132: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s; 10789/289269 objects degraded (3.730%); 7428/289269 objects misplaced (2.568%); 3.4 MiB/s, 54 objects/s recovering 2024-08-17T17:54:24.504 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:24.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:24 smithi165 ceph-mon[105517]: pgmap v2132: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s; 10789/289269 objects degraded (3.730%); 7428/289269 objects misplaced (2.568%); 3.4 MiB/s, 54 objects/s recovering 2024-08-17T17:54:24.542 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:24.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:24.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:24.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:24.678 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:24.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:24.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:24.711 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 67 16 100733 100717 93.9339 80.4375 0.00449319 0.0106411 2024-08-17T17:54:24.780 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:24.805 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:24.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:24.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:24.907 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:24.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:24.941 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:24.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:25.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:25.073 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:25.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:25.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:25.174 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:25.210 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:25.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:25.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:25.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:25.352 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:25.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:25.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:25 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 10789/289269 objects degraded (3.730%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:25.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:25 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 10789/289269 objects degraded (3.730%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:25.455 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:25.495 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:25.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:25.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:25.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:25 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 10789/289269 objects degraded (3.730%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:25.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:25.633 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:25.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:25.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:25.712 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 68 16 101939 101923 93.6609 75.375 0.0180365 0.0106716 2024-08-17T17:54:25.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:25.775 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:25.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:25.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:25.877 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:25.915 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:25.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:25.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:26.018 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:26.050 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:26.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:26.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:26.152 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:26.183 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:26.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:26.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:26.285 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:26.309 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:26.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:26.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:26.411 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:26.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:26 smithi136 ceph-mon[109861]: pgmap v2133: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s; 10789/289269 objects degraded (3.730%); 7428/289269 objects misplaced (2.568%); 3.4 MiB/s, 54 objects/s recovering 2024-08-17T17:54:26.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:26 smithi136 ceph-mon[115078]: pgmap v2133: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s; 10789/289269 objects degraded (3.730%); 7428/289269 objects misplaced (2.568%); 3.4 MiB/s, 54 objects/s recovering 2024-08-17T17:54:26.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:26.448 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:26.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:26.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:26 smithi165 ceph-mon[105517]: pgmap v2133: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s; 10789/289269 objects degraded (3.730%); 7428/289269 objects misplaced (2.568%); 3.4 MiB/s, 54 objects/s recovering 2024-08-17T17:54:26.549 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:26.586 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:26.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:26.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:26.688 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:26.712 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 69 16 103054 103038 93.3134 69.6875 0.0142147 0.0107119 2024-08-17T17:54:26.720 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:26.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:26.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:26.822 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:26.860 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:26.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:26.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:26.962 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:26.995 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:26.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:26.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:27.096 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:27.132 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:27.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:27.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:27.234 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:27.268 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:27.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:27.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:27.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:27.391 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:27.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:27.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:27.493 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:27.534 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:27.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:27.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:27.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:27.665 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:27.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:27.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:27.712 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 70 16 104263 104247 93.0597 75.5625 0.00504539 0.0107409 2024-08-17T17:54:27.766 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:27.803 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:27.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:27.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:27.905 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:27.937 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:27.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:27.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:27.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:54:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:54:27] "GET /metrics HTTP/1.1" 200 36566 "" "Prometheus/2.51.0" 2024-08-17T17:54:28.030 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:54:28.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:28.066 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:28.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:28.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:28.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:28.200 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:28.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:28.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:28.302 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:28.325 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:28.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:28.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:28.426 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:28.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:28 smithi136 ceph-mon[109861]: pgmap v2134: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s; 10789/300048 objects degraded (3.596%); 6704/300048 objects misplaced (2.234%); 4.0 MiB/s, 63 objects/s recovering 2024-08-17T17:54:28.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:28 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:54:28.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:28 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:54:28.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:28 smithi136 ceph-mon[115078]: pgmap v2134: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s; 10789/300048 objects degraded (3.596%); 6704/300048 objects misplaced (2.234%); 4.0 MiB/s, 63 objects/s recovering 2024-08-17T17:54:28.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:28 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:54:28.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:28 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:54:28.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:28.464 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:28.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:28.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:28 smithi165 ceph-mon[105517]: pgmap v2134: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s; 10789/300048 objects degraded (3.596%); 6704/300048 objects misplaced (2.234%); 4.0 MiB/s, 63 objects/s recovering 2024-08-17T17:54:28.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:28 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:54:28.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:28 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:54:28.565 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:28.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:28.608 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:28.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:28.709 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:28.713 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 71 16 105342 105326 92.6988 67.4375 0.0125229 0.0107827 2024-08-17T17:54:28.750 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:28.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:28.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:28.852 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:28.893 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:28.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:28.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:28.994 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:29.022 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:29.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:29.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:29.082 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:54:29.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:29.158 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:29.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:29.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:29.260 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:29.288 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:29.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:29.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:29.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:29.417 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:29.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:29.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:29.519 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:29.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:29.557 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:29.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:29.659 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:29.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:29.702 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:29.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:29.712 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 72 16 106451 106435 92.3739 69.3125 0.00782115 0.0108204 2024-08-17T17:54:29.804 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:29.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:29.856 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:29.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:29.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:29.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:29.988 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:29.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:30.089 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:30.116 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:30.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:30.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:30.218 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:30.284 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:30.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:30.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:30.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:30 smithi136 ceph-mon[109861]: pgmap v2135: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s; 10789/307257 objects degraded (3.511%); 6704/307257 objects misplaced (2.182%); 4.0 MiB/s, 63 objects/s recovering 2024-08-17T17:54:30.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:30 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 10789/307257 objects degraded (3.511%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:30.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:54:30.324 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:54:30.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:30 smithi136 ceph-mon[115078]: pgmap v2135: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s; 10789/307257 objects degraded (3.511%); 6704/307257 objects misplaced (2.182%); 4.0 MiB/s, 63 objects/s recovering 2024-08-17T17:54:30.324 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:30 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 10789/307257 objects degraded (3.511%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:54:30.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:54:30.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:30.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:30.413 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:30.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:30.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:30 smithi165 ceph-mon[105517]: pgmap v2135: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s; 10789/307257 objects degraded (3.511%); 6704/307257 objects misplaced (2.182%); 4.0 MiB/s, 63 objects/s recovering 2024-08-17T17:54:30.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:30 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 10789/307257 objects degraded (3.511%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:30.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:30 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:54:30.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:30 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:54:30.514 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:30.541 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:30.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:30.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:30.643 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:30.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:30.680 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:30.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:30.712 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 73 16 107466 107450 91.9775 63.4375 0.0126274 0.0108673 2024-08-17T17:54:30.782 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:30.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:30.814 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:30.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:30.916 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:30.957 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:30.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:30.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:31.059 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:31.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:31.101 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:31.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:31.107 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:54:31.107 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T17:54:31.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:31.107 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:107882 53.10 53:0a08a68d:::benchmark_data_smithi136_235573_object107881:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1326)", 2024-08-17T17:54:31.107 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:54:31.098151+0000", 2024-08-17T17:54:31.107 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0079033899999999997, 2024-08-17T17:54:31.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0079388029999999995, 2024-08-17T17:54:31.108 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:54:31.108 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:54:31.108 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:54:31.108 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:54:31.108 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:54:31.108 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:54:31.108 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 107882 2024-08-17T17:54:31.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:31.108 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:54:31.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:31.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:54:31.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:31.098151+0000", 2024-08-17T17:54:31.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:31.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:31.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:31.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:54:31.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:31.098151+0000", 2024-08-17T17:54:31.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:54:31.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:31.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:31.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:54:31.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:31.098153+0000", 2024-08-17T17:54:31.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5579999999999999e-06 2024-08-17T17:54:31.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:31.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:31.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:54:31.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:31.098178+0000", 2024-08-17T17:54:31.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5062000000000002e-05 2024-08-17T17:54:31.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:31.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:31.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:54:31.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:31.098199+0000", 2024-08-17T17:54:31.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0551e-05 2024-08-17T17:54:31.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:31.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:31.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:54:31.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:31.098208+0000", 2024-08-17T17:54:31.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2779999999999999e-06 2024-08-17T17:54:31.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:31.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:31.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:54:31.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:31.098287+0000", 2024-08-17T17:54:31.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8726000000000005e-05 2024-08-17T17:54:31.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:31.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:31.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:54:31.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:31.098454+0000", 2024-08-17T17:54:31.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016708000000000001 2024-08-17T17:54:31.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:31.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:31.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,5,6", 2024-08-17T17:54:31.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:31.098604+0000", 2024-08-17T17:54:31.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015066 2024-08-17T17:54:31.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:31.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:31.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:54:31.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:31.100960+0000", 2024-08-17T17:54:31.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023556050000000002 2024-08-17T17:54:31.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:31.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:31.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:31.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:31.105161+0000", 2024-08-17T17:54:31.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0042006279999999997 2024-08-17T17:54:31.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:54:31.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:54:31.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:54:31.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:31.105399+0000", 2024-08-17T17:54:31.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023834299999999999 2024-08-17T17:54:31.113 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:31.113 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:54:31.113 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:31.113 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:54:31.113 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-17T17:54:31.113 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 1 2024-08-17T17:54:31.113 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:54:31.160 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T17:54:31.160 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T17:54:31.160 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T17:54:31.160 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T17:54:31.160 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T17:54:31.160 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:54:31.161 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:54:31.161 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:54:31.161 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:54:31.161 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:54:31.161 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:54:31.161 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T17:54:31.161 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T17:54:31.161 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T17:54:31.161 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:54:31.161 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:54:31.202 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:31.246 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:31.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:31.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:31.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:31.376 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:31.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:31.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:31.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:31 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3761355863' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:54:31.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:31 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3761355863' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:54:31.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:31.510 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:31.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:31 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3761355863' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:54:31.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:31.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:31.612 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:31.647 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:31.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:31.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:31.712 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 74 16 108510 108494 91.6163 65.25 0.00883238 0.0108817 2024-08-17T17:54:31.748 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:31.777 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:31.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:31.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:31.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:31.902 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:31.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:31.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:32.004 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:32.027 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:32.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:32.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:32.129 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:32.151 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:32.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:32.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:32.253 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:32.283 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:32.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:32.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:32.385 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:32.423 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:32.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:32.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:32.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:32 smithi165 ceph-mon[105517]: pgmap v2136: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s; 10789/313371 objects degraded (3.443%); 6704/313371 objects misplaced (2.139%); 3.5 MiB/s, 56 objects/s recovering 2024-08-17T17:54:32.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:32 smithi165 ceph-mon[105517]: osdmap e1327: 8 total, 8 up, 8 in 2024-08-17T17:54:32.526 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:32.560 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:32.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:32.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:32.583 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:32 smithi136 ceph-mon[109861]: pgmap v2136: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s; 10789/313371 objects degraded (3.443%); 6704/313371 objects misplaced (2.139%); 3.5 MiB/s, 56 objects/s recovering 2024-08-17T17:54:32.583 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:32 smithi136 ceph-mon[109861]: osdmap e1327: 8 total, 8 up, 8 in 2024-08-17T17:54:32.583 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:32 smithi136 ceph-mon[115078]: pgmap v2136: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s; 10789/313371 objects degraded (3.443%); 6704/313371 objects misplaced (2.139%); 3.5 MiB/s, 56 objects/s recovering 2024-08-17T17:54:32.583 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:32 smithi136 ceph-mon[115078]: osdmap e1327: 8 total, 8 up, 8 in 2024-08-17T17:54:32.663 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:32.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:32.710 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:32.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:32.712 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 75 16 109092 109076 90.8798 36.375 0.00415166 0.0109988 2024-08-17T17:54:32.812 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:32.852 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:32.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:32.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:32.954 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:32.994 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:32.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:32.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:33.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:33.120 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:33.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:33.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:33.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:33.258 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:33.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:33.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:33.360 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:33.381 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:33.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:33.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:33.483 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:33.508 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:33.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:33.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:33.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:33 smithi165 ceph-mon[105517]: osdmap e1328: 8 total, 8 up, 8 in 2024-08-17T17:54:33.609 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:33.633 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:33.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:33.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:33 smithi136 ceph-mon[109861]: osdmap e1328: 8 total, 8 up, 8 in 2024-08-17T17:54:33.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:33 smithi136 ceph-mon[115078]: osdmap e1328: 8 total, 8 up, 8 in 2024-08-17T17:54:33.712 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 76 16 110087 110071 90.5022 62.1875 0.00680315 0.0110443 2024-08-17T17:54:33.736 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:33.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:33.769 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:33.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:33.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:33.905 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:33.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:33.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:34.007 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:34.048 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:34.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:34.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:34.150 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:34.184 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:34.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:34.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:34.286 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:34.329 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:34.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:34.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:34.431 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:34.460 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:34.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:34.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:34.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:34 smithi165 ceph-mon[105517]: pgmap v2139: 99 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 89 MiB/s wr, 1.43k op/s; 10789/323508 objects degraded (3.335%); 5960/323508 objects misplaced (1.842%); 5.7 MiB/s, 91 objects/s recovering 2024-08-17T17:54:34.534 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:34 smithi136 ceph-mon[109861]: pgmap v2139: 99 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 89 MiB/s wr, 1.43k op/s; 10789/323508 objects degraded (3.335%); 5960/323508 objects misplaced (1.842%); 5.7 MiB/s, 91 objects/s recovering 2024-08-17T17:54:34.536 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:34 smithi136 ceph-mon[115078]: pgmap v2139: 99 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 89 MiB/s wr, 1.43k op/s; 10789/323508 objects degraded (3.335%); 5960/323508 objects misplaced (1.842%); 5.7 MiB/s, 91 objects/s recovering 2024-08-17T17:54:34.561 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:34.575 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-17T17:54:34.602 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:34.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:34.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:34.705 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:34.713 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 77 16 111202 111186 90.2309 69.6875 0.0229381 0.0110773 2024-08-17T17:54:34.763 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:34.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:34.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:34.866 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:34.895 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:34.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:34.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:34.997 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:35.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:35.051 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:35.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:35.152 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:35.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:35.195 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:35.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:35.296 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:35.337 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:35.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:35.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:35.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:35.462 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:35.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:35.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:35.543 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:35 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 10789/323508 objects degraded (3.335%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:35.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:35 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 10789/323508 objects degraded (3.335%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:35.564 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:35.610 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:35.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:35.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:35.712 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:35.713 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 78 16 112236 112220 89.9023 64.625 0.00866128 0.0111179 2024-08-17T17:54:35.744 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:35.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:35.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:35.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:35 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 10789/323508 objects degraded (3.335%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:35.836 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:54:35.846 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:35.870 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:35.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:35.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:35.972 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:36.026 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:36.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:36.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:36.128 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:36.178 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:36.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:36.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:36.279 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:36.331 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:36.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:36.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:36.393 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:36 smithi136 ceph-mon[109861]: pgmap v2140: 99 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 61 MiB/s wr, 976 op/s; 10789/323508 objects degraded (3.335%); 5960/323508 objects misplaced (1.842%); 2.9 MiB/s, 46 objects/s recovering 2024-08-17T17:54:36.393 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:54:36.394 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:36 smithi136 ceph-mon[115078]: pgmap v2140: 99 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 61 MiB/s wr, 976 op/s; 10789/323508 objects degraded (3.335%); 5960/323508 objects misplaced (1.842%); 2.9 MiB/s, 46 objects/s recovering 2024-08-17T17:54:36.394 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:54:36.433 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:36.509 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:54:36.511 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:36.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:36.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:36.613 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:36.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:36.671 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:36.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:36.713 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 79 16 113130 113114 89.4716 55.875 0.0124043 0.011171 2024-08-17T17:54:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:36 smithi165 ceph-mon[105517]: pgmap v2140: 99 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 61 MiB/s wr, 976 op/s; 10789/323508 objects degraded (3.335%); 5960/323508 objects misplaced (1.842%); 2.9 MiB/s, 46 objects/s recovering 2024-08-17T17:54:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:54:36.772 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:36.796 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:36.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:36.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:36.898 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:36.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:36.946 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:36.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:37.047 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:37.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:37.090 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:37.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:37.191 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:37.231 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:37.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:37.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:37.294 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T17:54:37.299 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.1 to 1 (10000) 2024-08-17T17:54:37.332 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:37.342 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:37 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/490668312' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:54:37.342 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:37 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:54:37.342 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:37 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3529827521' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:54:37.342 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:37 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:54:37.343 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:37 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/490668312' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:54:37.343 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:37 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:54:37.343 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:37 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3529827521' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:54:37.343 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:37 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:54:37.380 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:37.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:37.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:37.482 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:37.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:37.513 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:37.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:37.614 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:37.641 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:37.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:37.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:37.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:54:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:54:37] "GET /metrics HTTP/1.1" 200 36567 "" "Prometheus/2.51.0" 2024-08-17T17:54:37.714 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:2024-08-17T17:54:37.714170+0000 min lat: 0.00118032 max lat: 2.13253 avg lat: 0.0112065 2024-08-17T17:54:37.714 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:54:37.714 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 80 16 114207 114191 89.1945 67.3125 0.00404904 0.0112065 2024-08-17T17:54:37.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:37 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/490668312' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:54:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:37 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:54:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:37 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3529827521' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:54:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:37 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:54:37.772 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:37.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:37.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:37.874 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:37.913 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:37.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:37.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:38.015 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:38.062 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:38.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:38.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:38.164 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:38.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:38.191 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:38.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:38.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:38.326 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:38.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:38.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:38.398 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:38 smithi136 ceph-mon[115078]: pgmap v2141: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s; 6017/332178 objects degraded (1.811%); 10237/332178 objects misplaced (3.082%); 4.2 MiB/s, 67 objects/s recovering 2024-08-17T17:54:38.398 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:38 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:54:38.398 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:38 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:54:38.399 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:38 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T17:54:38.399 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:38 smithi136 ceph-mon[115078]: osdmap e1329: 8 total, 8 up, 8 in 2024-08-17T17:54:38.399 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:38 smithi136 ceph-mon[109861]: pgmap v2141: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s; 6017/332178 objects degraded (1.811%); 10237/332178 objects misplaced (3.082%); 4.2 MiB/s, 67 objects/s recovering 2024-08-17T17:54:38.399 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:38 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:54:38.399 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:38 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:54:38.399 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:38 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T17:54:38.399 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:38 smithi136 ceph-mon[109861]: osdmap e1329: 8 total, 8 up, 8 in 2024-08-17T17:54:38.429 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:38.470 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:38.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:38.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:38.572 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:38.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:38.606 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:38.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:38.707 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:38.714 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 81 16 115365 115349 88.9868 72.375 0.0160936 0.0112318 2024-08-17T17:54:38.733 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:38.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:38.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:38.754 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 2 1 2024-08-17T17:54:38.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:38 smithi165 ceph-mon[105517]: pgmap v2141: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s; 6017/332178 objects degraded (1.811%); 10237/332178 objects misplaced (3.082%); 4.2 MiB/s, 67 objects/s recovering 2024-08-17T17:54:38.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:38 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:54:38.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:38 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:54:38.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:38 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T17:54:38.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:38 smithi165 ceph-mon[105517]: osdmap e1329: 8 total, 8 up, 8 in 2024-08-17T17:54:38.835 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:38.874 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:38.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:38.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:38.977 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:39.005 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:39.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:39.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:39.107 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:39.135 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:39.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:39.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:39.237 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:39.272 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:39.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:39.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:39.337 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:39 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 6017/332178 objects degraded (1.811%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:39.338 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:39 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 6017/332178 objects degraded (1.811%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:39.374 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:39.432 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:39.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:39.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:39.533 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:39.583 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:39.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:39.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:39.685 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:39.714 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 82 16 116438 116422 88.7194 67.0625 0.00748109 0.0112667 2024-08-17T17:54:39.716 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:39.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:39.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:39.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:39 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 6017/332178 objects degraded (1.811%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:39.818 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:39.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:39.851 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:39.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:39.952 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:39.988 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:39.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:39.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:40.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:40.114 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:40.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:40.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:40.216 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:40.245 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:40.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:40.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:40.347 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:40.373 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:40.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:40.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:40.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:40.507 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:40.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:40.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:40.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:40.638 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:40.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:40.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:40.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:40 smithi136 ceph-mon[109861]: pgmap v2143: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 59 MiB/s wr, 945 op/s; 6017/337902 objects degraded (1.781%); 10237/337902 objects misplaced (3.030%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:54:40.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:40 smithi136 ceph-mon[115078]: pgmap v2143: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 59 MiB/s wr, 945 op/s; 6017/337902 objects degraded (1.781%); 10237/337902 objects misplaced (3.030%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:54:40.714 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 83 16 117418 117402 88.3884 61.25 0.00780801 0.0113073 2024-08-17T17:54:40.739 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:40.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:40 smithi165 ceph-mon[105517]: pgmap v2143: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 59 MiB/s wr, 945 op/s; 6017/337902 objects degraded (1.781%); 10237/337902 objects misplaced (3.030%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:54:40.779 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:40.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:40.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:40.881 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:40.925 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:40.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:40.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:41.027 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:41.061 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:41.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:41.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:41.163 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:41.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:41.190 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:41.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:41.291 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:41.331 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:41.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:41.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:41.389 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:54:41.433 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:41.474 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:41.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:41.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:41.576 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:41.600 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:41.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:41.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:41.702 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:41.717 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 84 16 118448 118432 88.0998 64.375 0.0131384 0.0113455 2024-08-17T17:54:41.732 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:41.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:41.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:41.834 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:41.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:41.864 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:41.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:41.965 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:42.002 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:42.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:42.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:42.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:42.130 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:42.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:42.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:42.232 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:42.265 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:42.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:42.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:42.366 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:42.396 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:42.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:42.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:42.498 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:42.542 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:42.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:42.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:42.644 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:42.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:42.678 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:42.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:42.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:42 smithi136 ceph-mon[109861]: pgmap v2144: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 50 MiB/s wr, 805 op/s; 6017/343725 objects degraded (1.751%); 9835/343725 objects misplaced (2.861%); 2.2 MiB/s, 35 objects/s recovering 2024-08-17T17:54:42.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:42 smithi136 ceph-mon[115078]: pgmap v2144: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 50 MiB/s wr, 805 op/s; 6017/343725 objects degraded (1.751%); 9835/343725 objects misplaced (2.861%); 2.2 MiB/s, 35 objects/s recovering 2024-08-17T17:54:42.717 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 85 16 119426 119410 87.7824 61.125 0.0096812 0.0113866 2024-08-17T17:54:42.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:42 smithi165 ceph-mon[105517]: pgmap v2144: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 50 MiB/s wr, 805 op/s; 6017/343725 objects degraded (1.751%); 9835/343725 objects misplaced (2.861%); 2.2 MiB/s, 35 objects/s recovering 2024-08-17T17:54:42.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:42.806 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:42.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:42.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:42.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:42.946 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:42.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:42.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:43.048 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:43.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:43.088 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:43.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:43.190 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:43.210 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:43.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:43.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:43.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:43.348 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:43.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:43.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:43.450 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:43.485 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:43.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:43.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:43.587 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:43.611 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:43.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:43.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:43.713 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:43.717 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 86 16 120481 120465 87.5283 65.9375 0.020742 0.0114194 2024-08-17T17:54:43.749 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:43.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:43.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:43.851 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:43.900 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:43.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:43.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:44.002 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:44.036 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:44.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:44.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:44.138 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:44.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:44.160 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:44.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:44.261 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:44.311 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:44.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:44.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:44.413 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:44.442 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:44.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:44.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:44.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:44.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:44.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:44.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:44.675 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:44.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:44 smithi136 ceph-mon[115078]: pgmap v2145: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s; 6017/353667 objects degraded (1.701%); 9835/353667 objects misplaced (2.781%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T17:54:44.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:44 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 6017/353667 objects degraded (1.701%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:44.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:44 smithi136 ceph-mon[109861]: pgmap v2145: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s; 6017/353667 objects degraded (1.701%); 9835/353667 objects misplaced (2.781%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T17:54:44.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:44 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 6017/353667 objects degraded (1.701%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:44.706 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:44.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:44.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:44.717 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 87 16 121293 121277 87.1056 50.75 0.00776391 0.0114754 2024-08-17T17:54:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:44 smithi165 ceph-mon[105517]: pgmap v2145: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s; 6017/353667 objects degraded (1.701%); 9835/353667 objects misplaced (2.781%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T17:54:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:44 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 6017/353667 objects degraded (1.701%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:44.808 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:44.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:44.848 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:44.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:44.949 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:44.988 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:44.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:44.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:45.091 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:45.121 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:45.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:45.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:45.226 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:45.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:45.261 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:45.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:45.362 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:45.407 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:45.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:45.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:45.490 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:54:45.493 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:54:45.509 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:45.533 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:45.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:45.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:45.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:45.663 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:45.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:45.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:45.717 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 88 16 122303 122287 86.8331 63.125 0.0112745 0.0115115 2024-08-17T17:54:45.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:45.810 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:45.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:45.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:45.912 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:45.964 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:45.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:45.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:46.066 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:46.093 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:46.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:46.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:46.195 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:46.237 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:46.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:46.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:46.339 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:46.368 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:46.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:46.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:46.470 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:46.497 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:46.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:46.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:46.599 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:46 smithi136 ceph-mon[109861]: pgmap v2146: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s; 6017/353667 objects degraded (1.701%); 9835/353667 objects misplaced (2.781%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T17:54:46.599 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:46 smithi136 ceph-mon[115078]: pgmap v2146: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s; 6017/353667 objects degraded (1.701%); 9835/353667 objects misplaced (2.781%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T17:54:46.599 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:46.649 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:46.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:46.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:46.717 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 89 16 123329 123313 86.5779 64.125 0.0148322 0.0115448 2024-08-17T17:54:46.751 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:46.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:46 smithi165 ceph-mon[105517]: pgmap v2146: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s; 6017/353667 objects degraded (1.701%); 9835/353667 objects misplaced (2.781%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T17:54:46.779 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:46.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:46.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:46.881 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:46.911 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:46.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:46.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:47.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:47.038 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:47.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:47.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:47.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:47.172 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:47.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:47.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:47.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:47.305 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:47.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:47.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:47.407 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:47.434 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:47.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:47.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:47.467 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:54:47.468 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T17:54:47.468 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-17T17:54:47.468 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-17T17:54:47.468 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:54:47.536 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:47.559 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:47.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:47.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:47.661 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:47.676 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:54:47.685 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:47.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:47.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:47.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:54:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:54:47] "GET /metrics HTTP/1.1" 200 36567 "" "Prometheus/2.51.0" 2024-08-17T17:54:47.717 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Total time run: 90.014 2024-08-17T17:54:47.717 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Total writes made: 124283 2024-08-17T17:54:47.717 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Write size: 65536 2024-08-17T17:54:47.718 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Object size: 65536 2024-08-17T17:54:47.718 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Bandwidth (MB/sec): 86.2942 2024-08-17T17:54:47.718 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Stddev Bandwidth: 45.1017 2024-08-17T17:54:47.718 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Max bandwidth (MB/sec): 228.062 2024-08-17T17:54:47.718 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Min bandwidth (MB/sec): 0 2024-08-17T17:54:47.718 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Average IOPS: 1380 2024-08-17T17:54:47.718 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Stddev IOPS: 721.634 2024-08-17T17:54:47.718 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Max IOPS: 3649 2024-08-17T17:54:47.718 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Min IOPS: 0 2024-08-17T17:54:47.718 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Average Latency(s): 0.0115836 2024-08-17T17:54:47.718 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Stddev Latency(s): 0.0441449 2024-08-17T17:54:47.718 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Max latency(s): 2.13253 2024-08-17T17:54:47.719 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Min latency(s): 0.00118032 2024-08-17T17:54:47.724 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Cleaning up (deleting benchmark objects) 2024-08-17T17:54:47.787 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:47.812 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:47.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:47.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:47.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:47.957 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:47.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:47.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:48.059 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:48.090 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:48.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:48.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:48.186 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-17T17:54:48.192 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:48.282 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:48.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:48.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:48.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:48.411 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:48.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:48.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:48.444 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:48 smithi136 ceph-mon[109861]: pgmap v2147: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 6017/362415 objects degraded (1.660%); 9448/362415 objects misplaced (2.607%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:54:48.445 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:48 smithi136 ceph-mon[115078]: pgmap v2147: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 6017/362415 objects degraded (1.660%); 9448/362415 objects misplaced (2.607%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:54:48.513 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:48.541 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:48.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:48.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:48.643 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:48.667 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:48.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:48.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:48 smithi165 ceph-mon[105517]: pgmap v2147: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 6017/362415 objects degraded (1.660%); 9448/362415 objects misplaced (2.607%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:54:48.769 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:48.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:48.825 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:48.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:48.926 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:48.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:48.977 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:48.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:49.078 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:49.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:49.126 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:49.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:49.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:49.253 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:49.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:49.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:49.355 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:49.378 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:49.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:49.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:49.480 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:49.508 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:49.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:49.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:49.610 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:49.654 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:49.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:49.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:49 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 6017/362415 objects degraded (1.660%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:49.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:49 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 6017/362415 objects degraded (1.660%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:49.756 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:49.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:49 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 6017/362415 objects degraded (1.660%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:49.802 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:49.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:49.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:49.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:49.928 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:49.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:49.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:50.030 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:50.057 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:50.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:50.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:50.159 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:50.183 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:50.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:50.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:50.284 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:50.312 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:50.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:50.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:50.414 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:50.440 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:50.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:50.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:50.542 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:50.570 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:50.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:50.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:50.672 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:50.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:50 smithi136 ceph-mon[109861]: pgmap v2148: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 56 MiB/s wr, 894 op/s; 6017/368307 objects degraded (1.634%); 9448/368307 objects misplaced (2.565%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:54:50.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:50 smithi136 ceph-mon[115078]: pgmap v2148: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 56 MiB/s wr, 894 op/s; 6017/368307 objects degraded (1.634%); 9448/368307 objects misplaced (2.565%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:54:50.699 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:50.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:50.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:50.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:50 smithi165 ceph-mon[105517]: pgmap v2148: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 56 MiB/s wr, 894 op/s; 6017/368307 objects degraded (1.634%); 9448/368307 objects misplaced (2.565%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:54:50.800 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:50.841 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:50.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:50.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:50.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:50.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:50.980 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:50.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:51.082 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:51.119 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:51.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:51.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:51.222 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:51.248 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:51.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:51.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:51.350 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:51.393 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:51.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:51.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:51.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:54:51.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:54:51.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:54:51.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:54:51.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:54:51.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:54:51.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:54:51.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:54:51.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:54:51.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:54:51.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:54:51.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:54:51.495 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:51.523 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:51.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:51.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:51.625 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:51.669 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:51.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:51.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:54:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:54:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:54:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:54:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:54:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:54:51.775 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:51.803 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:51.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:51.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:51.906 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:51.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:51.934 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:51.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:52.035 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:52.062 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:52.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:52.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:52.164 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:52.198 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:52.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:52.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:52.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:52.337 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:52.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:52.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:52.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:52.464 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:52.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:52.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:52.566 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:52.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:52.596 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:52.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:52.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:52 smithi136 ceph-mon[109861]: pgmap v2149: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 57 MiB/s wr, 996 op/s; 6017/367821 objects degraded (1.636%); 9028/367821 objects misplaced (2.454%); 2.1 MiB/s, 33 objects/s recovering 2024-08-17T17:54:52.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:52 smithi136 ceph-mon[115078]: pgmap v2149: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 57 MiB/s wr, 996 op/s; 6017/367821 objects degraded (1.636%); 9028/367821 objects misplaced (2.454%); 2.1 MiB/s, 33 objects/s recovering 2024-08-17T17:54:52.697 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:52.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:52.731 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:52.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:52.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:52 smithi165 ceph-mon[105517]: pgmap v2149: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 57 MiB/s wr, 996 op/s; 6017/367821 objects degraded (1.636%); 9028/367821 objects misplaced (2.454%); 2.1 MiB/s, 33 objects/s recovering 2024-08-17T17:54:52.832 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:52.859 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:52.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:52.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:52.962 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:52.995 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:52.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:52.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:53.097 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:53.127 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:53.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:53.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:53.229 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:53.253 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:53.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:53.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:53.262 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:54:53.354 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:53.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:53.387 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:53.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:53.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:53.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:53.537 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:53.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:53.638 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:53.683 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:53.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:53.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:53.785 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:53.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:53.829 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:53.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:53.930 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:53.954 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:53.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:53.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:54.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:54.079 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:54.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:54.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:54.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:54.205 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:54.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:54.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:54.208 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T17:54:54.208 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:54:54.271 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T17:54:54.272 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:54:54.307 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:54.317 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T17:54:54.317 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:54:54.345 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:54.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:54.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:54.358 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T17:54:54.358 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:54:54.395 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T17:54:54.395 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:54:54.430 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T17:54:54.430 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:54:54.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:54.464 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T17:54:54.464 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:54:54.473 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:54.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:54.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:54.493 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T17:54:54.494 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:54:54.575 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:54.615 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:54.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:54.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:54.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:54 smithi136 ceph-mon[109861]: pgmap v2150: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 51 MiB/s wr, 1.08k op/s; 5843/363042 objects degraded (1.609%); 8941/363042 objects misplaced (2.463%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T17:54:54.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:54 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3104184674' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:54:54.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:54 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5843/363042 objects degraded (1.609%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:54 smithi136 ceph-mon[115078]: pgmap v2150: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 51 MiB/s wr, 1.08k op/s; 5843/363042 objects degraded (1.609%); 8941/363042 objects misplaced (2.463%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T17:54:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:54 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3104184674' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:54:54.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:54 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5843/363042 objects degraded (1.609%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:54.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:54.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:54.761 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:54.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:54.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:54 smithi165 ceph-mon[105517]: pgmap v2150: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 51 MiB/s wr, 1.08k op/s; 5843/363042 objects degraded (1.609%); 8941/363042 objects misplaced (2.463%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T17:54:54.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:54 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3104184674' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:54:54.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:54 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5843/363042 objects degraded (1.609%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:54:54.862 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:54.899 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:54.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:54.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:55.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:55.046 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:55.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:55.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:55.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:55.172 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:55.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:55.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:55.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:55.308 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:55.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:55.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:55.392 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.2 to 1 (10000) 2024-08-17T17:54:55.409 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:55.461 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:55.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:55.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:55.563 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:55.588 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:55.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:55.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:55.690 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:55 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1398500627' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:54:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:55 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:54:55.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:55 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1398500627' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:54:55.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:55 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:54:55.734 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:55.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:55.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:55 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1398500627' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:54:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:55 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:54:55.836 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:55.881 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:55.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:55.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:55.983 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:56.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:56.026 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:56.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:56.127 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:56.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:56.156 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:56.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:56.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:56.261 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 3 1 2024-08-17T17:54:56.295 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:56.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:56.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:56.396 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:56.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:56.444 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:56.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:56.479 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:56 smithi136 ceph-mon[109861]: pgmap v2151: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 33 MiB/s wr, 805 op/s; 5843/363042 objects degraded (1.609%); 8941/363042 objects misplaced (2.463%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T17:54:56.479 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:56 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T17:54:56.479 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:56 smithi136 ceph-mon[109861]: osdmap e1330: 8 total, 8 up, 8 in 2024-08-17T17:54:56.479 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:56 smithi136 ceph-mon[115078]: pgmap v2151: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 33 MiB/s wr, 805 op/s; 5843/363042 objects degraded (1.609%); 8941/363042 objects misplaced (2.463%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T17:54:56.480 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:56 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T17:54:56.480 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:56 smithi136 ceph-mon[115078]: osdmap e1330: 8 total, 8 up, 8 in 2024-08-17T17:54:56.545 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:56.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:56.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:56.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:56.675 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:56.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:56.728 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:56.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:56 smithi165 ceph-mon[105517]: pgmap v2151: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 33 MiB/s wr, 805 op/s; 5843/363042 objects degraded (1.609%); 8941/363042 objects misplaced (2.463%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T17:54:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:56 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T17:54:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:56 smithi165 ceph-mon[105517]: osdmap e1330: 8 total, 8 up, 8 in 2024-08-17T17:54:56.828 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:56.861 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:56.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:56.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:54:56.962 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:56.992 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:56.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:56.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:57.094 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:57.120 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:57.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:57.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:57.222 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:57.248 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:57.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:57.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:57.350 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:57.377 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:57.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:57.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:57.402 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T17:54:57.422 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:57 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1223686399' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:54:57.422 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:57 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:54:57.422 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:57 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1223686399' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:54:57.423 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:57 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:54:57.479 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:57.505 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:54:57.520 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:57.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:57.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:57.622 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:57.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:57.661 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:57.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:57.703 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:54:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:54:57] "GET /metrics HTTP/1.1" 200 36571 "" "Prometheus/2.51.0" 2024-08-17T17:54:57.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:57 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1223686399' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:54:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:57 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:54:57.763 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:57.829 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:57.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:57.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:57.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:57.964 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:57.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:57.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:58.066 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:58.106 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:58.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:58.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:58.154 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T17:54:58.154 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:54:58.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:54:58.237 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:58.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:58.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:54:58.339 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:54:58.382 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:58.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:58.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:54:58.484 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:58.512 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:58.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:58.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:58.614 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:58.642 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:58.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:58.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:58.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:58 smithi136 ceph-mon[109861]: pgmap v2153: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 102 B/s rd, 22 MiB/s wr, 969 op/s; 5527/354228 objects degraded (1.560%); 8315/354228 objects misplaced (2.347%); 3.4 MiB/s, 54 objects/s recovering 2024-08-17T17:54:58.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:58 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:54:58.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:58 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:54:58.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:58 smithi136 ceph-mon[109861]: osdmap e1331: 8 total, 8 up, 8 in 2024-08-17T17:54:58.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:58 smithi136 ceph-mon[115078]: pgmap v2153: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 102 B/s rd, 22 MiB/s wr, 969 op/s; 5527/354228 objects degraded (1.560%); 8315/354228 objects misplaced (2.347%); 3.4 MiB/s, 54 objects/s recovering 2024-08-17T17:54:58.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:58 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:54:58.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:58 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:54:58.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:58 smithi136 ceph-mon[115078]: osdmap e1331: 8 total, 8 up, 8 in 2024-08-17T17:54:58.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:58.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:58 smithi165 ceph-mon[105517]: pgmap v2153: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 102 B/s rd, 22 MiB/s wr, 969 op/s; 5527/354228 objects degraded (1.560%); 8315/354228 objects misplaced (2.347%); 3.4 MiB/s, 54 objects/s recovering 2024-08-17T17:54:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:58 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:54:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:58 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:54:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:58 smithi165 ceph-mon[105517]: osdmap e1331: 8 total, 8 up, 8 in 2024-08-17T17:54:58.769 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:58.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:58.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:58.871 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:58.899 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:58.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:58.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:59.002 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:59.029 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:59.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:59.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:59.131 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:54:59.167 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:59.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:59.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:54:59.258 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:54:59.269 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:59.291 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:59.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:59.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:59.314 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-17T17:54:59.351 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:54:59.393 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:54:59.449 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:59.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:59.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:54:59.551 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:54:59.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:59.581 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:59.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:54:59.599 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:54:59 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5527/354228 objects degraded (1.560%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:54:59.599 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:54:59 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5527/354228 objects degraded (1.560%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:54:59.682 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:54:59.709 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:59.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:59.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:54:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:54:59 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5527/354228 objects degraded (1.560%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:54:59.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:54:59.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:59.847 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:54:59.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:54:59.948 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:54:59.997 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:54:59.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:54:59.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:00.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:00.123 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:00.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:00.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:00.225 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:00.259 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:00.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:00.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:00.361 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:00.391 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:00.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:00.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:00.493 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:00.534 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:00.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:00.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:00.577 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:55:00.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:00.662 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:00.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:00.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:00.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:00 smithi136 ceph-mon[109861]: pgmap v2155: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 5.3 MiB/s wr, 987 op/s; 5527/348210 objects degraded (1.587%); 8315/348210 objects misplaced (2.388%); 3.1 MiB/s, 50 objects/s recovering 2024-08-17T17:55:00.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:00 smithi136 ceph-mon[115078]: pgmap v2155: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 5.3 MiB/s wr, 987 op/s; 5527/348210 objects degraded (1.587%); 8315/348210 objects misplaced (2.388%); 3.1 MiB/s, 50 objects/s recovering 2024-08-17T17:55:00.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:00 smithi165 ceph-mon[105517]: pgmap v2155: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 5.3 MiB/s wr, 987 op/s; 5527/348210 objects degraded (1.587%); 8315/348210 objects misplaced (2.388%); 3.1 MiB/s, 50 objects/s recovering 2024-08-17T17:55:00.764 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:00.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:00.814 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:00.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:00.915 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:00.951 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:00.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:00.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:01.053 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:01.082 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:01.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:01.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:01.184 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:01.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:01.223 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:01.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:01.324 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:01.349 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:01.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:01.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:01.451 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:01.487 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:01.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:01.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:01.589 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:01.623 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:01.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:01.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:01.724 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:01.783 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:01.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:01.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:01.885 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:01.930 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:01.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:01.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:02.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:02.059 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:02.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:02.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:02.161 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:02.189 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:02.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:02.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:02.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:02.293 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:55:02.318 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:02.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:02.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:02.420 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:02.444 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:02.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:02.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:02.545 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:02.593 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:02.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:02.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:02.696 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:02.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:02 smithi136 ceph-mon[115078]: pgmap v2156: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 690 op/s; 5527/346452 objects degraded (1.595%); 8315/346452 objects misplaced (2.400%); 2.5 MiB/s, 39 objects/s recovering 2024-08-17T17:55:02.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:02 smithi136 ceph-mon[109861]: pgmap v2156: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 690 op/s; 5527/346452 objects degraded (1.595%); 8315/346452 objects misplaced (2.400%); 2.5 MiB/s, 39 objects/s recovering 2024-08-17T17:55:02.720 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:02.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:02.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:02.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:02 smithi165 ceph-mon[105517]: pgmap v2156: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 690 op/s; 5527/346452 objects degraded (1.595%); 8315/346452 objects misplaced (2.400%); 2.5 MiB/s, 39 objects/s recovering 2024-08-17T17:55:02.823 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:02.850 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:02.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:02.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:02.952 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:02.976 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:02.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:02.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:03.078 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:03.117 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:03.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:03.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:03.219 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:03.241 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:03.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:03.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:03.343 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:03.402 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:03.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:03.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:03.504 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:03.536 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:03.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:03.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:03.638 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:03.667 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:03.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:03.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:03.769 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:03.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:03.811 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:03.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:03.837 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:55:03.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:03.937 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:03.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:03.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:04.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:04.065 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:04.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:04.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:04.167 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:04.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:04.217 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:04.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:04.318 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:04.362 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:04.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:04.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:04.448 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.3 to 1 (10000) 2024-08-17T17:55:04.464 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:04.504 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:04.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:04.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:04.606 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:04.644 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:04.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:04.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:04.651 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:04 smithi136 ceph-mon[109861]: pgmap v2157: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s; 5222/333657 objects degraded (1.565%); 7659/333657 objects misplaced (2.295%); 5.0 MiB/s, 79 objects/s recovering 2024-08-17T17:55:04.651 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:04 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3498267710' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:55:04.651 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:04 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:55:04.651 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:04 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5222/333657 objects degraded (1.565%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:55:04.652 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:04 smithi136 ceph-mon[115078]: pgmap v2157: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s; 5222/333657 objects degraded (1.565%); 7659/333657 objects misplaced (2.295%); 5.0 MiB/s, 79 objects/s recovering 2024-08-17T17:55:04.652 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:04 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3498267710' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:55:04.652 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:04 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:55:04.652 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:04 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5222/333657 objects degraded (1.565%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:55:04.746 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:04.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:04 smithi165 ceph-mon[105517]: pgmap v2157: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s; 5222/333657 objects degraded (1.565%); 7659/333657 objects misplaced (2.295%); 5.0 MiB/s, 79 objects/s recovering 2024-08-17T17:55:04.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:04 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3498267710' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:55:04.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:04 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:55:04.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:04 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5222/333657 objects degraded (1.565%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:55:04.775 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:04.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:04.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:04.877 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:04.911 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:04.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:04.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:05.014 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:05.041 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:05.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:05.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:05.098 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T17:55:05.098 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T17:55:05.098 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T17:55:05.099 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T17:55:05.099 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:55:05.099 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:55:05.099 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:55:05.099 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:55:05.099 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T17:55:05.099 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T17:55:05.100 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T17:55:05.100 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T17:55:05.100 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:55:05.100 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:55:05.100 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:55:05.100 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:55:05.144 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:05.185 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:05.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:05.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:05.287 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:05.332 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:05.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:05.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:05.434 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:05.461 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:05.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:05.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:05.563 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:05.601 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:05.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:05.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:05.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:05.730 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:05.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:05.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:05.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:05 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T17:55:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:05 smithi165 ceph-mon[105517]: osdmap e1332: 8 total, 8 up, 8 in 2024-08-17T17:55:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:05 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/673924557' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:55:05.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:05 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T17:55:05.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:05 smithi136 ceph-mon[109861]: osdmap e1332: 8 total, 8 up, 8 in 2024-08-17T17:55:05.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:05 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/673924557' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:55:05.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:05 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T17:55:05.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:05 smithi136 ceph-mon[115078]: osdmap e1332: 8 total, 8 up, 8 in 2024-08-17T17:55:05.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:05 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/673924557' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:55:05.832 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:05.858 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:05.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:05.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:05.960 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:05.986 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:05.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:05.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:06.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:06.126 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:06.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:06.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:06.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:06.268 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:06.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:06.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:06.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:06.396 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:06.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:06.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:06.498 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:06.546 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:06.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:06.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:06.648 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:06.686 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:06.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:06.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:06 smithi165 ceph-mon[105517]: pgmap v2159: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 856 op/s; 5222/333657 objects degraded (1.565%); 7659/333657 objects misplaced (2.295%); 2.5 MiB/s, 39 objects/s recovering 2024-08-17T17:55:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:55:06.787 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:06.829 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:06.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:06.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:06.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:06 smithi136 ceph-mon[109861]: pgmap v2159: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 856 op/s; 5222/333657 objects degraded (1.565%); 7659/333657 objects misplaced (2.295%); 2.5 MiB/s, 39 objects/s recovering 2024-08-17T17:55:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:55:06.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:06 smithi136 ceph-mon[115078]: pgmap v2159: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 856 op/s; 5222/333657 objects degraded (1.565%); 7659/333657 objects misplaced (2.295%); 2.5 MiB/s, 39 objects/s recovering 2024-08-17T17:55:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:55:06.959 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:06.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:06.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:07.060 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:07.091 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:07.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:07.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:07.193 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:07.229 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:07.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:07.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:07.331 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:07.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:07.358 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:07.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:07.459 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:07.501 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:07.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:07.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:07.604 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:07.628 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:07.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:07.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:07.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:07.759 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:07.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:07.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:07.860 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:07.896 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:07.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:07.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:07.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:55:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:55:07] "GET /metrics HTTP/1.1" 200 36564 "" "Prometheus/2.51.0" 2024-08-17T17:55:07.998 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:08.025 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:08.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:08.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:08.127 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:08.148 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:08.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:08.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:08.250 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:08.283 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:08.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:08.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:08.385 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:08.412 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:08.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:08.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:08.514 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:08.542 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:08.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:08.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:08.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:08.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:08.665 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:08.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:08.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:08 smithi136 ceph-mon[109861]: pgmap v2160: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 841 op/s; 4907/324963 objects degraded (1.510%); 6994/324963 objects misplaced (2.152%); 4.4 MiB/s, 70 objects/s recovering 2024-08-17T17:55:08.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:08 smithi136 ceph-mon[115078]: pgmap v2160: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 841 op/s; 4907/324963 objects degraded (1.510%); 6994/324963 objects misplaced (2.152%); 4.4 MiB/s, 70 objects/s recovering 2024-08-17T17:55:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:08 smithi165 ceph-mon[105517]: pgmap v2160: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 841 op/s; 4907/324963 objects degraded (1.510%); 6994/324963 objects misplaced (2.152%); 4.4 MiB/s, 70 objects/s recovering 2024-08-17T17:55:08.766 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:08.800 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:08.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:08.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:08.902 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:08.927 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:08.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:08.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:09.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:09.060 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:09.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:09.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:09.162 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:09.195 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:09.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:09.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:09.297 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:09.338 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:09.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:09.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:09.440 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:09.478 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:09.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:09.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:09.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:09.612 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:09.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:09.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:09.714 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:09.747 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:09.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:09.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:09.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:09 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 4907/324963 objects degraded (1.510%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:09.849 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:09.892 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:09.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:09.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:09 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 4907/324963 objects degraded (1.510%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:09.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:09 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 4907/324963 objects degraded (1.510%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:09.994 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:10.032 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:10.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:10.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:10.134 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:10.173 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:10.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:10.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:10.275 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:10.313 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:10.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:10.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:10.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:10.446 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:10.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:10.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:10.548 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:10.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:10.587 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:10.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:10.688 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:10.726 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:10.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:10.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:10.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:10 smithi165 ceph-mon[105517]: pgmap v2161: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 971 op/s; 4907/319047 objects degraded (1.538%); 6994/319047 objects misplaced (2.192%); 4.0 MiB/s, 64 objects/s recovering 2024-08-17T17:55:10.827 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:10.860 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:10.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:10.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:10.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:10 smithi136 ceph-mon[109861]: pgmap v2161: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 971 op/s; 4907/319047 objects degraded (1.538%); 6994/319047 objects misplaced (2.192%); 4.0 MiB/s, 64 objects/s recovering 2024-08-17T17:55:10.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:10 smithi136 ceph-mon[115078]: pgmap v2161: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 971 op/s; 4907/319047 objects degraded (1.538%); 6994/319047 objects misplaced (2.192%); 4.0 MiB/s, 64 objects/s recovering 2024-08-17T17:55:10.962 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:11.001 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:11.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:11.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:11.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:11.125 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:11.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:11.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:11.226 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:11.266 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:11.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:11.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:11.368 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:11.408 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:11.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:11.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:11.510 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:11.550 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:11.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:11.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:11.652 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:11.699 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:11.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:11.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:11.801 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:11.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:11.823 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:11.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:11.924 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:11.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:11.948 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:11.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:12.049 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:12.080 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:12.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:12.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:12.181 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:12.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:12.204 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:12.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:12.306 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:12.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:12.343 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:12.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:12.444 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:12.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:12.476 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:12.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:12.579 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:12.600 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:12.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:12.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:12.702 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:12.730 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:12.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:12.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:12 smithi165 ceph-mon[105517]: pgmap v2162: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 971 op/s; 4907/317295 objects degraded (1.547%); 6994/317295 objects misplaced (2.204%); 4.0 MiB/s, 64 objects/s recovering 2024-08-17T17:55:12.832 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:12.868 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:12.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:12.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:12.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:12 smithi136 ceph-mon[109861]: pgmap v2162: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 971 op/s; 4907/317295 objects degraded (1.547%); 6994/317295 objects misplaced (2.204%); 4.0 MiB/s, 64 objects/s recovering 2024-08-17T17:55:12.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:12 smithi136 ceph-mon[115078]: pgmap v2162: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 971 op/s; 4907/317295 objects degraded (1.547%); 6994/317295 objects misplaced (2.204%); 4.0 MiB/s, 64 objects/s recovering 2024-08-17T17:55:12.970 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:12.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:12.992 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:12.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:13.093 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:13.133 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:13.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:13.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:13.234 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:13.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:13.257 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:13.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:13.358 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:13.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:13.401 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:13.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:13.435 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 4 1 2024-08-17T17:55:13.502 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:13.529 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:13.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:13.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:13.631 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:13.660 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:13.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:13.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:13.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:13.806 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:13.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:13.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:13.909 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:13.946 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:13.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:13.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:14.047 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:14.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:14.086 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:14.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:14.187 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:14.217 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:14.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:14.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:14.319 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:14.349 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:14.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:14.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:14.451 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:14.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:14.479 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:14.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:14.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:14.605 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:14.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:14.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:14.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:14 smithi136 ceph-mon[109861]: pgmap v2163: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 952 op/s; 4637/305079 objects degraded (1.520%); 6352/305079 objects misplaced (2.082%); 3.9 MiB/s, 63 objects/s recovering 2024-08-17T17:55:14.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:14 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 4637/305079 objects degraded (1.520%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:14.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:14 smithi136 ceph-mon[115078]: pgmap v2163: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 952 op/s; 4637/305079 objects degraded (1.520%); 6352/305079 objects misplaced (2.082%); 3.9 MiB/s, 63 objects/s recovering 2024-08-17T17:55:14.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:14 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 4637/305079 objects degraded (1.520%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:14.707 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:14.736 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:14.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:14.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:14 smithi165 ceph-mon[105517]: pgmap v2163: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 952 op/s; 4637/305079 objects degraded (1.520%); 6352/305079 objects misplaced (2.082%); 3.9 MiB/s, 63 objects/s recovering 2024-08-17T17:55:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:14 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 4637/305079 objects degraded (1.520%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:14.837 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:14.864 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:14.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:14.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:14.966 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:14.992 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:14.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:14.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:15.094 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:15.125 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:15.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:15.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:15.226 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:15.269 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:15.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:15.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:15.371 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:15.392 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:15.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:15.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:15.494 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:15.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:55:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=cleanup t=2024-08-17T17:55:15.063051307Z level=info msg="Completed cleanup jobs" duration=21.714521ms 2024-08-17T17:55:15.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:55:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=plugins.update.checker t=2024-08-17T17:55:15.183633629Z level=info msg="Update check succeeded" duration=61.77063ms 2024-08-17T17:55:15.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 17:55:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=grafana.update.checker t=2024-08-17T17:55:15.200358838Z level=info msg="Update check succeeded" duration=70.201987ms 2024-08-17T17:55:15.535 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:15.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:15.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:15.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:15.663 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:15.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:15.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:15.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:15.800 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:15.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:15.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:15.902 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:15.929 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:15.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:15.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:16.031 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:16.056 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:16.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:16.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:16.158 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:16.191 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:16.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:16.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:16.292 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:16.332 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:16.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:16.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:16.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:16.460 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:16.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:16.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:16.561 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:16.590 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:16.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:16.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:16.692 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:16.734 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:16.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:16.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:16.753 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:16 smithi136 ceph-mon[109861]: pgmap v2164: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 935 op/s; 4637/305079 objects degraded (1.520%); 6352/305079 objects misplaced (2.082%); 3.9 MiB/s, 61 objects/s recovering 2024-08-17T17:55:16.753 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:16 smithi136 ceph-mon[115078]: pgmap v2164: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 935 op/s; 4637/305079 objects degraded (1.520%); 6352/305079 objects misplaced (2.082%); 3.9 MiB/s, 61 objects/s recovering 2024-08-17T17:55:16.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:16 smithi165 ceph-mon[105517]: pgmap v2164: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 935 op/s; 4637/305079 objects degraded (1.520%); 6352/305079 objects misplaced (2.082%); 3.9 MiB/s, 61 objects/s recovering 2024-08-17T17:55:16.818 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:55:16.819 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T17:55:16.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:55:16.819 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.35091.0:151594 53.17 e1332/1310 53:ea284c93:::benchmark_data_smithi136_235573_object27308:head v 1332'6758, mlcod=1332'6755)", 2024-08-17T17:55:16.819 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:55:16.814498+0000", 2024-08-17T17:55:16.819 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.001669441, 2024-08-17T17:55:16.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016967169999999999, 2024-08-17T17:55:16.819 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:55:16.819 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:55:16.819 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T17:55:16.819 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:55:16.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:55:16.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:55:16.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:55:16.814498+0000", 2024-08-17T17:55:16.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:55:16.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:55:16.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:55:16.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:55:16.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:55:16.814498+0000", 2024-08-17T17:55:16.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:55:16.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:55:16.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:55:16.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:55:16.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:55:16.814499+0000", 2024-08-17T17:55:16.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5400000000000001e-07 2024-08-17T17:55:16.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:55:16.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:55:16.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:55:16.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:55:16.814505+0000", 2024-08-17T17:55:16.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.455e-06 2024-08-17T17:55:16.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:55:16.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:55:16.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:55:16.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:55:16.814515+0000", 2024-08-17T17:55:16.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3219999999999999e-06 2024-08-17T17:55:16.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:55:16.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:55:16.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:55:16.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:55:16.814520+0000", 2024-08-17T17:55:16.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5550000000000002e-06 2024-08-17T17:55:16.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:55:16.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:55:16.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:55:16.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:55:16.814577+0000", 2024-08-17T17:55:16.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6484000000000003e-05 2024-08-17T17:55:16.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:55:16.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:55:16.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:55:16.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:55:16.814599+0000", 2024-08-17T17:55:16.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2211000000000001e-05 2024-08-17T17:55:16.823 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:55:16.823 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:55:16.823 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:55:16.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:55:16.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:55:16.823 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:151601 53.a 53:569bf495:::benchmark_data_smithi136_235573_object27315:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1332)", 2024-08-17T17:55:16.823 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:55:16.815860+0000", 2024-08-17T17:55:16.823 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.00030799000000000001, 2024-08-17T17:55:16.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00040161599999999998, 2024-08-17T17:55:16.823 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:55:16.823 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:55:16.824 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T17:55:16.824 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:55:16.824 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:55:16.824 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:55:16.824 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 151601 2024-08-17T17:55:16.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:55:16.824 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:55:16.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:55:16.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:55:16.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:55:16.815860+0000", 2024-08-17T17:55:16.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:55:16.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:55:16.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:55:16.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:55:16.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:55:16.815860+0000", 2024-08-17T17:55:16.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:55:16.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:55:16.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:55:16.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:55:16.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:55:16.815861+0000", 2024-08-17T17:55:16.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.37e-06 2024-08-17T17:55:16.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:55:16.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:55:16.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:55:16.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:55:16.815866+0000", 2024-08-17T17:55:16.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9259999999999999e-06 2024-08-17T17:55:16.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:55:16.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:55:16.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:55:16.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:55:16.815883+0000", 2024-08-17T17:55:16.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6523000000000001e-05 2024-08-17T17:55:16.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:55:16.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:55:16.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:55:16.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:55:16.815891+0000", 2024-08-17T17:55:16.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1899999999999995e-06 2024-08-17T17:55:16.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:55:16.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:55:16.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:55:16.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:55:16.816042+0000", 2024-08-17T17:55:16.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015105300000000001 2024-08-17T17:55:16.827 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:55:16.827 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:55:16.828 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:55:16.828 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:55:16.828 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-17T17:55:16.828 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 2 2024-08-17T17:55:16.828 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:55:16.836 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:16.863 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:16.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:16.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:16.965 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:16.993 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:16.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:16.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:17.095 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:17.122 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:17.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:17.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:17.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:17.250 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:17.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:17.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:17.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:17.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:17.378 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:17.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:17.479 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:17.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:17.522 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:17.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:17.603 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:55:17.623 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:17.676 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:17.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:17.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:17.741 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:55:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:55:17] "GET /metrics HTTP/1.1" 200 36564 "" "Prometheus/2.51.0" 2024-08-17T17:55:17.778 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:17.807 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:17.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:17.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:17.909 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:17.950 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:17.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:17.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:17.980 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:55:18.052 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:18.103 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-17T17:55:18.122 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:18.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:18.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:18.224 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:18.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:18.272 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:18.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:18.373 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:18.397 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:18.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:18.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:18.499 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:18.518 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T17:55:18.527 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:18.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:18.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:18.629 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:18.676 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:18.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:18.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:18.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:18 smithi136 ceph-mon[109861]: pgmap v2165: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 4401/297516 objects degraded (1.479%); 5748/297516 objects misplaced (1.932%); 4.7 MiB/s, 75 objects/s recovering 2024-08-17T17:55:18.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:18 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4182857503' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:55:18.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:18 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:55:18.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:18 smithi136 ceph-mon[115078]: pgmap v2165: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 4401/297516 objects degraded (1.479%); 5748/297516 objects misplaced (1.932%); 4.7 MiB/s, 75 objects/s recovering 2024-08-17T17:55:18.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:18 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4182857503' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:55:18.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:18 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:55:18.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:18 smithi165 ceph-mon[105517]: pgmap v2165: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 4401/297516 objects degraded (1.479%); 5748/297516 objects misplaced (1.932%); 4.7 MiB/s, 75 objects/s recovering 2024-08-17T17:55:18.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:18 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4182857503' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:55:18.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:18 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:55:18.778 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:18.827 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:18.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:18.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:18.929 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:18.953 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:18.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:18.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:19.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:19.094 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:19.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:19.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:19.195 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:19.241 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:19.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:19.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:19.343 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:19.370 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:19.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:19.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:19.472 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:19.505 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:19.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:19.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:19.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:19.638 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:19.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:19.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:19.740 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:19 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:55:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:19 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:55:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:19 smithi165 ceph-mon[105517]: osdmap e1333: 8 total, 8 up, 8 in 2024-08-17T17:55:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:19 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 4401/297516 objects degraded (1.479%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:19.766 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:19.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:19.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:19.868 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:19.901 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:19.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:19.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:19.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:19 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:55:19.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:19 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:55:19.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:19 smithi136 ceph-mon[109861]: osdmap e1333: 8 total, 8 up, 8 in 2024-08-17T17:55:19.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:19 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 4401/297516 objects degraded (1.479%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:19.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:19 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:55:19.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:19 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:55:19.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:19 smithi136 ceph-mon[115078]: osdmap e1333: 8 total, 8 up, 8 in 2024-08-17T17:55:19.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:19 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 4401/297516 objects degraded (1.479%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:20.002 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:20.036 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:20.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:20.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:20.139 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:20.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:20.184 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:20.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:20.285 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:20.323 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:20.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:20.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:20.424 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:20.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:20.454 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:20.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:20.558 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:20.583 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:20.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:20.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:20.684 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:20.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:20 smithi136 ceph-mon[109861]: pgmap v2167: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 883 op/s; 4401/292527 objects degraded (1.504%); 5748/292527 objects misplaced (1.965%); 3.6 MiB/s, 58 objects/s recovering 2024-08-17T17:55:20.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:20 smithi136 ceph-mon[115078]: pgmap v2167: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 883 op/s; 4401/292527 objects degraded (1.504%); 5748/292527 objects misplaced (1.965%); 3.6 MiB/s, 58 objects/s recovering 2024-08-17T17:55:20.711 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:20.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:20.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:20.813 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:20.841 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:20.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:20.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:20.942 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:20.983 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:20.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:20.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:21.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:20 smithi165 ceph-mon[105517]: pgmap v2167: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 883 op/s; 4401/292527 objects degraded (1.504%); 5748/292527 objects misplaced (1.965%); 3.6 MiB/s, 58 objects/s recovering 2024-08-17T17:55:21.085 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:21.119 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:21.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:21.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:21.221 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:21.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:21.260 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:21.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:21.361 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:21.382 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:21.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:21.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:21.484 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:21.512 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:21.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:21.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:21.613 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:21.640 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:21.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:21.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:21.742 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:21.769 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:21.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:21.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:21.871 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:21.893 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:21.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:21.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:21.994 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:22.035 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:22.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:22.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:22.082 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:55:22.138 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:22.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:22.165 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:22.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:22.266 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:22.288 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:22.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:22.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:22.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:22.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:22.426 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:22.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:22.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:22 smithi136 ceph-mon[109861]: pgmap v2168: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 878 op/s; 4401/290934 objects degraded (1.513%); 5748/290934 objects misplaced (1.976%); 3.6 MiB/s, 58 objects/s recovering 2024-08-17T17:55:22.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:55:22.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:55:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:22 smithi136 ceph-mon[115078]: pgmap v2168: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 878 op/s; 4401/290934 objects degraded (1.513%); 5748/290934 objects misplaced (1.976%); 3.6 MiB/s, 58 objects/s recovering 2024-08-17T17:55:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:55:22.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:55:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:22 smithi165 ceph-mon[105517]: pgmap v2168: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 878 op/s; 4401/290934 objects degraded (1.513%); 5748/290934 objects misplaced (1.976%); 3.6 MiB/s, 58 objects/s recovering 2024-08-17T17:55:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:55:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:55:22.527 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:22.559 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:22.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:22.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:22.661 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:22.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:22.697 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:22.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:22.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:22.801 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:55:22.806 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:55:22.823 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:22.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:22.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:22.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:22.953 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:22.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:22.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:23.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:23.107 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:23.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:23.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:23.209 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:23.258 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:23.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:23.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:23.359 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:23.386 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:23.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:23.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:23.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:23.525 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:23.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:23.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:23.627 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:23.655 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:23.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:23.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:23.757 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:23.790 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:23.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:23.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:23.892 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:23.932 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:23.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:23.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:24.034 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:24.082 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:24.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:24.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:24.183 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:24.215 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.4 to 1 (10000) 2024-08-17T17:55:24.221 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:24.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:24.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:24.323 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:24.374 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:24.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:24.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:24.475 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:24.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:24 smithi165 ceph-mon[105517]: pgmap v2169: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 837 op/s; 4117/279930 objects degraded (1.471%); 5027/279930 objects misplaced (1.796%); 3.8 MiB/s, 61 objects/s recovering 2024-08-17T17:55:24.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:24 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2075748530' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:55:24.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:24 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:55:24.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:24.525 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:24.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:24.626 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:24.661 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:24.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:24.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:24.667 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:24 smithi136 ceph-mon[109861]: pgmap v2169: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 837 op/s; 4117/279930 objects degraded (1.471%); 5027/279930 objects misplaced (1.796%); 3.8 MiB/s, 61 objects/s recovering 2024-08-17T17:55:24.667 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:24 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2075748530' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:55:24.667 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:24 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:55:24.667 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:24 smithi136 ceph-mon[115078]: pgmap v2169: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 837 op/s; 4117/279930 objects degraded (1.471%); 5027/279930 objects misplaced (1.796%); 3.8 MiB/s, 61 objects/s recovering 2024-08-17T17:55:24.667 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:24 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2075748530' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:55:24.667 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:24 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:55:24.763 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:24.795 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:24.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:24.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:24.897 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:24.923 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:24.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:24.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:25.025 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:25.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:25.069 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:25.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:25.170 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:25.209 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:25.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:25.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:25.310 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:25.353 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:25.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:25.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:25 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-17T17:55:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:25 smithi136 ceph-mon[109861]: osdmap e1334: 8 total, 8 up, 8 in 2024-08-17T17:55:25.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:25 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 4117/279930 objects degraded (1.471%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:25.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:25 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-17T17:55:25.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:25 smithi136 ceph-mon[115078]: osdmap e1334: 8 total, 8 up, 8 in 2024-08-17T17:55:25.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:25 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 4117/279930 objects degraded (1.471%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:25.455 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:25.480 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:25.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:25.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:25.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:25 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-17T17:55:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:25 smithi165 ceph-mon[105517]: osdmap e1334: 8 total, 8 up, 8 in 2024-08-17T17:55:25.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:25 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 4117/279930 objects degraded (1.471%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:25.581 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:25.625 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:25.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:25.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:25.727 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:25.763 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:25.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:25.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:25.865 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:25.922 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:25.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:25.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:26.024 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:26.050 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:26.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:26.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:26.151 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:26.179 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:26.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:26.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:26.280 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:26.310 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:26.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:26.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:26.313 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:26 smithi136 ceph-mon[115078]: pgmap v2171: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 732 op/s; 4117/279930 objects degraded (1.471%); 5027/279930 objects misplaced (1.796%); 2.6 MiB/s, 41 objects/s recovering 2024-08-17T17:55:26.313 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:26 smithi136 ceph-mon[109861]: pgmap v2171: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 732 op/s; 4117/279930 objects degraded (1.471%); 5027/279930 objects misplaced (1.796%); 2.6 MiB/s, 41 objects/s recovering 2024-08-17T17:55:26.314 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 5 1 2024-08-17T17:55:26.412 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:26.452 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:26.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:26.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:26.554 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:26.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:26.625 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:26.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:26.726 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:26.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:26 smithi165 ceph-mon[105517]: pgmap v2171: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 732 op/s; 4117/279930 objects degraded (1.471%); 5027/279930 objects misplaced (1.796%); 2.6 MiB/s, 41 objects/s recovering 2024-08-17T17:55:26.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:26.769 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:26.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:26.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:26.899 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:26.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:26.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:27.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:27.042 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:27.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:27.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:27.143 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:27.185 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:27.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:27.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:27.287 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:27.342 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:27.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:27.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:27.444 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:27.484 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:55:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:55:27] "GET /metrics HTTP/1.1" 200 36564 "" "Prometheus/2.51.0" 2024-08-17T17:55:27.485 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:27.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:27.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:27.492 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:55:27.585 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T17:55:27.585 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:55:27.586 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:27.625 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T17:55:27.626 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:55:27.647 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:27.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:27.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:27.662 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T17:55:27.662 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:55:27.692 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T17:55:27.693 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:55:27.732 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T17:55:27.732 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:55:27.748 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:27.776 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T17:55:27.776 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:55:27.786 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:27.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:27.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:27.814 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T17:55:27.814 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:55:27.843 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T17:55:27.843 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:55:27.888 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:27.925 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:27.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:27.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:28.027 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:28.073 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:28.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:28.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:28.174 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:28.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:28.215 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:28.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:28.315 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:28.354 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:28.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:28.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:28.385 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:28 smithi136 ceph-mon[109861]: pgmap v2172: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 835 op/s; 3869/272193 objects degraded (1.421%); 4288/272193 objects misplaced (1.575%); 5.1 MiB/s, 81 objects/s recovering 2024-08-17T17:55:28.386 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:28 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1948062821' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:55:28.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:28 smithi136 ceph-mon[115078]: pgmap v2172: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 835 op/s; 3869/272193 objects degraded (1.421%); 4288/272193 objects misplaced (1.575%); 5.1 MiB/s, 81 objects/s recovering 2024-08-17T17:55:28.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:28 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1948062821' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:55:28.457 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:28.497 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:28.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:28.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:28.599 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:28.624 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:28.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:28.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:28.726 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:28.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:28 smithi165 ceph-mon[105517]: pgmap v2172: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 835 op/s; 3869/272193 objects degraded (1.421%); 4288/272193 objects misplaced (1.575%); 5.1 MiB/s, 81 objects/s recovering 2024-08-17T17:55:28.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:28 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1948062821' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:55:28.769 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:28.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:28.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:28.822 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:55:28.871 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:28.886 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T17:55:28.886 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:55:28.928 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:28.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:28.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:29.030 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:29.063 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:29.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:29.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:29.165 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:29.202 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:29.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:29.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:29.303 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:29.341 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:29.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:29.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:29.442 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:29.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:29.476 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:29.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:29.577 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:29.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:29.613 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:29.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:29.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:29 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 3869/272193 objects degraded (1.421%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:29.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:29 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 3869/272193 objects degraded (1.421%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:29.714 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:29.740 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:29.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:29.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:29.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:29 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 3869/272193 objects degraded (1.421%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:29.842 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:29.883 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:29.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:29.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:29.984 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:30.009 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:30.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:30.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:30.110 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:30.150 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:30.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:30.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:30.252 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:30.295 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:30.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:30.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:30.397 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:30.422 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:30 smithi136 ceph-mon[109861]: pgmap v2173: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 868 op/s; 3869/266463 objects degraded (1.452%); 4288/266463 objects misplaced (1.609%); 4.1 MiB/s, 66 objects/s recovering 2024-08-17T17:55:30.423 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:30 smithi136 ceph-mon[115078]: pgmap v2173: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 868 op/s; 3869/266463 objects degraded (1.452%); 4288/266463 objects misplaced (1.609%); 4.1 MiB/s, 66 objects/s recovering 2024-08-17T17:55:30.444 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:30.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:30.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:30.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:30.583 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:30.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:30.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:30.685 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:30.723 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:30.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:30.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:30.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:30 smithi165 ceph-mon[105517]: pgmap v2173: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 868 op/s; 3869/266463 objects degraded (1.452%); 4288/266463 objects misplaced (1.609%); 4.1 MiB/s, 66 objects/s recovering 2024-08-17T17:55:30.824 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:30.852 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:30.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:30.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:30.953 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:30.997 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:30.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:30.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:31.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:31.122 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:31.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:31.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:31.143 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-17T17:55:31.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:31.253 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:31.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:31.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:31.355 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:31.386 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:31.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:31.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:31.488 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:31.516 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:31.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:31.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:31.618 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:31.662 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:31.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:31.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:31.763 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:31.811 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:31.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:31.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:31.914 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:31.947 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:31.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:31.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:32.049 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:32.077 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:32.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:32.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:32.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:32.180 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:55:32.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:32.228 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:32.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:32.316 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T17:55:32.329 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:32.353 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:32.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:32.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:32.454 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:32.505 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:32.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:32.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:32.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:32.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:32.647 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:32.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:32.687 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:32 smithi136 ceph-mon[115078]: pgmap v2174: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 873 op/s; 3869/264723 objects degraded (1.462%); 4288/264723 objects misplaced (1.620%); 4.1 MiB/s, 66 objects/s recovering 2024-08-17T17:55:32.687 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:32 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/341453520' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:55:32.687 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:32 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:55:32.688 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:32 smithi136 ceph-mon[109861]: pgmap v2174: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 873 op/s; 3869/264723 objects degraded (1.462%); 4288/264723 objects misplaced (1.620%); 4.1 MiB/s, 66 objects/s recovering 2024-08-17T17:55:32.688 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:32 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/341453520' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:55:32.688 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:32 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:55:32.749 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:32.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:32 smithi165 ceph-mon[105517]: pgmap v2174: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 873 op/s; 3869/264723 objects degraded (1.462%); 4288/264723 objects misplaced (1.620%); 4.1 MiB/s, 66 objects/s recovering 2024-08-17T17:55:32.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:32 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/341453520' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:55:32.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:32 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:55:32.802 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:32.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:32.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:32.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:32.931 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:32.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:32.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:32.977 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:55:33.033 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:33.073 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:33.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:33.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:33.175 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:33.199 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:33.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:33.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:33.301 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:33.343 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:33.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:33.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:33.445 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:33.469 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:33.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:33.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:33.571 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:33.600 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:33.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:33.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:33.605 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:33 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:55:33.605 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:33 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:55:33.605 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:33 smithi136 ceph-mon[109861]: osdmap e1335: 8 total, 8 up, 8 in 2024-08-17T17:55:33.605 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:33 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:55:33.605 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:33 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:55:33.605 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:33 smithi136 ceph-mon[115078]: osdmap e1335: 8 total, 8 up, 8 in 2024-08-17T17:55:33.702 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:33.751 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:33.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:33.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:33.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:33 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:55:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:33 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:55:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:33 smithi165 ceph-mon[105517]: osdmap e1335: 8 total, 8 up, 8 in 2024-08-17T17:55:33.852 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:33.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:33.878 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:33.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:33.979 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:34.002 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:34.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:34.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:34.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:34.132 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:34.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:34.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:34.233 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:34.268 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:34.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:34.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:34.370 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:34.409 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:34.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:34.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:34.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:34 smithi136 ceph-mon[109861]: pgmap v2176: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 3631/252708 objects degraded (1.437%); 3464/252708 objects misplaced (1.371%); 5.1 MiB/s, 81 objects/s recovering 2024-08-17T17:55:34.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:34 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 3631/252708 objects degraded (1.437%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:34.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:34 smithi136 ceph-mon[115078]: pgmap v2176: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 3631/252708 objects degraded (1.437%); 3464/252708 objects misplaced (1.371%); 5.1 MiB/s, 81 objects/s recovering 2024-08-17T17:55:34.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:34 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 3631/252708 objects degraded (1.437%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:34.511 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:34.551 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:34.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:34.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:34.652 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:34.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:34 smithi165 ceph-mon[105517]: pgmap v2176: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 3631/252708 objects degraded (1.437%); 3464/252708 objects misplaced (1.371%); 5.1 MiB/s, 81 objects/s recovering 2024-08-17T17:55:34.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:34 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 3631/252708 objects degraded (1.437%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:34.686 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:55:34.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:34.689 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:34.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:34.790 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:34.837 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:34.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:34.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:34.939 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:34.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:34.970 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:34.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:35.070 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:35.100 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:35.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:35.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:35.202 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:35.242 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:35.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:35.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:35.344 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:35.354 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.5 to 1 (10000) 2024-08-17T17:55:35.383 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:35.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:35.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:35.412 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:55:35.485 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:35.535 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:35.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:35.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:35.637 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:35.645 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:35 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/101241564' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:55:35.645 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:35 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:55:35.647 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:35 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/101241564' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:55:35.647 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:35 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:55:35.698 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:35.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:35.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:35.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:35 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/101241564' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:55:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:35 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:55:35.799 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:35.845 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:35.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:35.946 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:35.983 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:35.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:35.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:36.085 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:36.114 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:36.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:36.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:36.216 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:36.243 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:36.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:36.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:36.345 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:36.360 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 6 1 2024-08-17T17:55:36.383 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:36.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:36.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:36.485 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:36.538 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:36.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:36.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:36.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:36 smithi136 ceph-mon[109861]: pgmap v2177: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 906 op/s; 3631/252708 objects degraded (1.437%); 3464/252708 objects misplaced (1.371%); 4.3 MiB/s, 68 objects/s recovering 2024-08-17T17:55:36.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:36 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T17:55:36.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:36 smithi136 ceph-mon[109861]: osdmap e1336: 8 total, 8 up, 8 in 2024-08-17T17:55:36.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:55:36.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:55:36.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:36 smithi136 ceph-mon[115078]: pgmap v2177: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 906 op/s; 3631/252708 objects degraded (1.437%); 3464/252708 objects misplaced (1.371%); 4.3 MiB/s, 68 objects/s recovering 2024-08-17T17:55:36.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:36 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T17:55:36.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:36 smithi136 ceph-mon[115078]: osdmap e1336: 8 total, 8 up, 8 in 2024-08-17T17:55:36.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:55:36.574 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:55:36.639 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:36.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:36.688 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:36.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:36 smithi165 ceph-mon[105517]: pgmap v2177: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 906 op/s; 3631/252708 objects degraded (1.437%); 3464/252708 objects misplaced (1.371%); 4.3 MiB/s, 68 objects/s recovering 2024-08-17T17:55:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:36 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T17:55:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:36 smithi165 ceph-mon[105517]: osdmap e1336: 8 total, 8 up, 8 in 2024-08-17T17:55:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:55:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:55:36.788 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:36.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:36.853 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:36.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:36.954 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:36.998 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:36.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:36.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:37.100 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:37.138 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:37.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:37.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:37.240 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:37.288 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:37.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:37.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:37.390 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:37.433 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:37.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:37.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:37.535 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:37.559 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:37.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:37.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:37.661 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:37.688 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:37.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:37.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:37.790 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:37.814 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:37.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:37.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:37.916 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:37.937 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:37.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:37.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:37.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:55:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:55:37] "GET /metrics HTTP/1.1" 200 36569 "" "Prometheus/2.51.0" 2024-08-17T17:55:38.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:38.061 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:38.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:38.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:38.165 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:38.187 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:38.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:38.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:38.288 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:38.310 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:38.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:38.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:38.411 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:38.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:38 smithi136 ceph-mon[109861]: pgmap v2179: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 3319/244590 objects degraded (1.357%); 2498/244590 objects misplaced (1.021%); 6.1 MiB/s, 97 objects/s recovering 2024-08-17T17:55:38.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:38 smithi136 ceph-mon[109861]: osdmap e1337: 8 total, 8 up, 8 in 2024-08-17T17:55:38.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:38 smithi136 ceph-mon[115078]: pgmap v2179: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 3319/244590 objects degraded (1.357%); 2498/244590 objects misplaced (1.021%); 6.1 MiB/s, 97 objects/s recovering 2024-08-17T17:55:38.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:38 smithi136 ceph-mon[115078]: osdmap e1337: 8 total, 8 up, 8 in 2024-08-17T17:55:38.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:38.450 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:38.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:38.551 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:38.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:38.577 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:38.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:38.678 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:38.717 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:38.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:38.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:38.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:38 smithi165 ceph-mon[105517]: pgmap v2179: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 3319/244590 objects degraded (1.357%); 2498/244590 objects misplaced (1.021%); 6.1 MiB/s, 97 objects/s recovering 2024-08-17T17:55:38.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:38 smithi165 ceph-mon[105517]: osdmap e1337: 8 total, 8 up, 8 in 2024-08-17T17:55:38.818 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:38.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:38.860 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:38.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:38.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:38.983 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:38.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:38.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:39.019 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T17:55:39.019 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:55:39.086 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:39.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:39.133 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:39.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:39.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:39.274 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:55:39.275 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:39.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:39.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:39.346 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:55:39.377 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:39.393 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:39 smithi136 ceph-mon[109861]: osdmap e1338: 8 total, 8 up, 8 in 2024-08-17T17:55:39.393 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:39 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 3319/244590 objects degraded (1.357%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:39.400 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:39.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:39.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:39.502 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:39.545 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:39.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:39.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:39.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:39.675 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:39.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:39.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:39.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:39 smithi136 ceph-mon[115078]: osdmap e1338: 8 total, 8 up, 8 in 2024-08-17T17:55:39.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:39 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 3319/244590 objects degraded (1.357%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:39.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:39 smithi165 ceph-mon[105517]: osdmap e1338: 8 total, 8 up, 8 in 2024-08-17T17:55:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:39 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 3319/244590 objects degraded (1.357%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:39.777 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:39.819 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:39.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:39.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:39.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:39.949 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:39.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:39.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:40.051 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:40.095 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:40.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:40.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:40.196 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:40.224 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:40.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:40.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:40.326 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:40.355 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:40.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:40.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:40.457 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:40.493 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:40.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:40.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:40.595 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:40.638 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:40.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:40.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:40.685 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:40 smithi136 ceph-mon[109861]: pgmap v2182: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 752 op/s; 3319/239163 objects degraded (1.388%); 2498/239163 objects misplaced (1.044%); 4.4 MiB/s, 70 objects/s recovering 2024-08-17T17:55:40.686 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:40 smithi136 ceph-mon[115078]: pgmap v2182: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 752 op/s; 3319/239163 objects degraded (1.388%); 2498/239163 objects misplaced (1.044%); 4.4 MiB/s, 70 objects/s recovering 2024-08-17T17:55:40.719 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-17T17:55:40.740 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:40.744 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:40 smithi165 ceph-mon[105517]: pgmap v2182: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 752 op/s; 3319/239163 objects degraded (1.388%); 2498/239163 objects misplaced (1.044%); 4.4 MiB/s, 70 objects/s recovering 2024-08-17T17:55:40.783 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:40.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:40.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:40.887 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:40.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:40.919 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:40.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:41.020 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:41.037 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:55:41.066 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:41.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:41.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:41.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:41.198 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:41.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:41.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:41.300 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:41.330 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:41.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:41.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:41.432 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:41.463 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:41.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:41.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:41.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:41.598 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:41.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:41.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:41.700 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:41.737 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:41.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:41.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:41.839 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:41.875 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:41.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:41.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:41.977 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:42.011 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:42.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:42.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:42.113 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:42.148 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:42.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:42.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:42.250 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:42.282 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:42.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:42.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:42.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:42.412 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:42.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:42.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:42.513 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:42.544 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:42.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:42.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:42.606 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:42 smithi136 ceph-mon[109861]: pgmap v2183: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 836 op/s; 3319/237651 objects degraded (1.397%); 2498/237651 objects misplaced (1.051%); 4.4 MiB/s, 70 objects/s recovering 2024-08-17T17:55:42.607 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:42 smithi136 ceph-mon[115078]: pgmap v2183: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 836 op/s; 3319/237651 objects degraded (1.397%); 2498/237651 objects misplaced (1.051%); 4.4 MiB/s, 70 objects/s recovering 2024-08-17T17:55:42.646 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:42.660 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T17:55:42.660 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:55:42.687 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:42.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:42.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:42.706 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T17:55:42.706 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:55:42.738 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T17:55:42.739 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:55:42.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:42 smithi165 ceph-mon[105517]: pgmap v2183: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 836 op/s; 3319/237651 objects degraded (1.397%); 2498/237651 objects misplaced (1.051%); 4.4 MiB/s, 70 objects/s recovering 2024-08-17T17:55:42.769 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T17:55:42.769 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:55:42.789 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:42.812 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T17:55:42.812 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:55:42.823 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:42.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:42.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:42.844 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T17:55:42.845 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:55:42.877 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T17:55:42.878 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:55:42.906 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T17:55:42.906 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:55:42.925 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:42.964 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:42.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:42.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:43.066 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:43.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:43.099 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:43.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:43.200 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:43.238 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:43.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:43.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:43.341 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:43.390 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:43.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:43.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:43.492 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:43.521 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:43.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:43.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:43.623 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:43.662 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:43.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:43.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:43.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:43 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4112694349' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:55:43.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:43 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4112694349' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:55:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:43 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4112694349' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:55:43.764 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:43.838 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:43.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:43.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:43.940 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:43.974 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:43.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:43.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:44.076 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:44.107 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:44.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:44.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:44.209 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:44.240 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:44.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:44.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:44.264 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T17:55:44.265 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T17:55:44.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:44.265 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.35091.0:175661 53.f e1338/1303 53:f2c61d05:::benchmark_data_smithi136_235573_object51375:head v 1338'11292, mlcod=1338'11290)", 2024-08-17T17:55:44.265 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:55:44.260244+0000", 2024-08-17T17:55:44.265 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0035328740000000001, 2024-08-17T17:55:44.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035701040000000002, 2024-08-17T17:55:44.265 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:55:44.265 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T17:55:44.265 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:55:44.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:44.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:55:44.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:44.260244+0000", 2024-08-17T17:55:44.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:44.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:44.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:44.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:55:44.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:44.260244+0000", 2024-08-17T17:55:44.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:44.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:44.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:44.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:55:44.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:44.260243+0000", 2024-08-17T17:55:44.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:55:44.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:44.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:44.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:55:44.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:44.260250+0000", 2024-08-17T17:55:44.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1110000000000002e-06 2024-08-17T17:55:44.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:44.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:44.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:55:44.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:44.260265+0000", 2024-08-17T17:55:44.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4628e-05 2024-08-17T17:55:44.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:44.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:44.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:55:44.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:44.260271+0000", 2024-08-17T17:55:44.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5919999999999997e-06 2024-08-17T17:55:44.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:44.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:44.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:55:44.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:44.260423+0000", 2024-08-17T17:55:44.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015214499999999999 2024-08-17T17:55:44.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:44.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:44.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:55:44.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:44.260468+0000", 2024-08-17T17:55:44.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4354000000000003e-05 2024-08-17T17:55:44.268 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:44.268 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:44.268 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:44.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:44.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:44.269 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.35091.0:175662 53.14 e1338/1310 53:2b2e6659:::benchmark_data_smithi136_235573_object51376:head v 1338'7278, mlcod=1338'7275)", 2024-08-17T17:55:44.269 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:55:44.262300+0000", 2024-08-17T17:55:44.269 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0014770110000000001, 2024-08-17T17:55:44.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001575251, 2024-08-17T17:55:44.269 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:55:44.269 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T17:55:44.269 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:55:44.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:44.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:55:44.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:44.262300+0000", 2024-08-17T17:55:44.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:44.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:44.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:44.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:55:44.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:44.262300+0000", 2024-08-17T17:55:44.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:44.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:44.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:44.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:55:44.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:44.262299+0000", 2024-08-17T17:55:44.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:55:44.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:44.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:44.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:55:44.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:44.262306+0000", 2024-08-17T17:55:44.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7139999999999996e-06 2024-08-17T17:55:44.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:44.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:44.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:55:44.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:44.262318+0000", 2024-08-17T17:55:44.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2799999999999999e-05 2024-08-17T17:55:44.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:44.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:44.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:55:44.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:44.262325+0000", 2024-08-17T17:55:44.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0840000000000003e-06 2024-08-17T17:55:44.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:44.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:44.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:55:44.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:44.262360+0000", 2024-08-17T17:55:44.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4405e-05 2024-08-17T17:55:44.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:44.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:44.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:55:44.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:44.262373+0000", 2024-08-17T17:55:44.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3444e-05 2024-08-17T17:55:44.272 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:44.272 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:44.272 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:44.272 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:44.272 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-08-17T17:55:44.272 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 2 2024-08-17T17:55:44.272 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T17:55:44.342 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:44.379 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:44.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:44.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:44.481 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:44.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:44.506 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:44.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:44.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:44.648 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:44.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:44.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:44.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:44 smithi136 ceph-mon[109861]: pgmap v2184: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 97 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.17k op/s; 2322/227208 objects degraded (1.022%); 2294/227208 objects misplaced (1.010%); 7.9 MiB/s, 126 objects/s recovering 2024-08-17T17:55:44.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:44 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 2322/227208 objects degraded (1.022%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:44.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:44 smithi136 ceph-mon[115078]: pgmap v2184: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 97 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.17k op/s; 2322/227208 objects degraded (1.022%); 2294/227208 objects misplaced (1.010%); 7.9 MiB/s, 126 objects/s recovering 2024-08-17T17:55:44.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:44 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 2322/227208 objects degraded (1.022%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:44.750 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:44 smithi165 ceph-mon[105517]: pgmap v2184: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 97 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.17k op/s; 2322/227208 objects degraded (1.022%); 2294/227208 objects misplaced (1.010%); 7.9 MiB/s, 126 objects/s recovering 2024-08-17T17:55:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:44 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 2322/227208 objects degraded (1.022%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:44.781 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:44.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:44.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:44.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:44.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:44.927 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:44.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:44.941 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-17T17:55:45.028 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:45.067 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:45.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:45.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:45.169 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:45.206 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:45.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:45.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:45.257 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:55:45.308 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:45.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:45.341 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:45.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:45.442 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:45.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:45.480 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:45.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:45.582 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:45.617 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:45.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:45.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:45.719 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:45.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:45.746 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:45.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:45.815 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:55:45.847 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:45.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:45.902 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:45.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:46.003 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:46.026 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:46.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:46.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:46.128 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:46.182 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:46.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:46.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:46.284 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:46.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:46.315 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:46.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:46.416 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:46.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:46.456 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:46.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:46.492 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:46 smithi165 ceph-mon[105517]: pgmap v2185: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 97 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 723 op/s; 2322/227208 objects degraded (1.022%); 2294/227208 objects misplaced (1.010%); 3.8 MiB/s, 61 objects/s recovering 2024-08-17T17:55:46.557 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:46.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:46.599 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:46.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:46.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:46 smithi136 ceph-mon[109861]: pgmap v2185: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 97 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 723 op/s; 2322/227208 objects degraded (1.022%); 2294/227208 objects misplaced (1.010%); 3.8 MiB/s, 61 objects/s recovering 2024-08-17T17:55:46.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:46 smithi136 ceph-mon[115078]: pgmap v2185: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 97 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 723 op/s; 2322/227208 objects degraded (1.022%); 2294/227208 objects misplaced (1.010%); 3.8 MiB/s, 61 objects/s recovering 2024-08-17T17:55:46.701 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:46.729 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:46.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:46.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:46.831 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:46.855 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:46.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:46.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:46.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:46.992 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:46.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:46.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:47.094 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:47.127 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:47.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:47.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:47.147 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T17:55:47.148 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T17:55:47.148 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-08-17T17:55:47.148 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-08-17T17:55:47.148 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T17:55:47.229 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:47.251 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:47.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:47.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:47.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:47.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:47.384 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:47.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:47.485 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:47.525 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:47.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:47.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:47.627 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:47.671 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:47.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:47.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:47.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:55:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:55:47] "GET /metrics HTTP/1.1" 200 36569 "" "Prometheus/2.51.0" 2024-08-17T17:55:47.774 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:47.812 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:47.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:47.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:47.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:47.956 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:47.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:47.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:48.057 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:48.088 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:48.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:48.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:48.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:48.222 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:48.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:48.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:48.324 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:48.365 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:48.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:48.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:48.467 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:48.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:48.497 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:48.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:48.569 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.6 to 1 (10000) 2024-08-17T17:55:48.570 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:48 smithi136 ceph-mon[109861]: pgmap v2186: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 97 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 730 op/s; 2053/218865 objects degraded (0.938%); 1632/218865 objects misplaced (0.746%); 6.0 MiB/s, 96 objects/s recovering 2024-08-17T17:55:48.570 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:48 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4000047458' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:55:48.571 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:48 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:55:48.571 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:48 smithi136 ceph-mon[115078]: pgmap v2186: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 97 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 730 op/s; 2053/218865 objects degraded (0.938%); 1632/218865 objects misplaced (0.746%); 6.0 MiB/s, 96 objects/s recovering 2024-08-17T17:55:48.571 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:48 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4000047458' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:55:48.571 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:48 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:55:48.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:48.635 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:48.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:48.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:48.727 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:48 smithi165 ceph-mon[105517]: pgmap v2186: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 97 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 730 op/s; 2053/218865 objects degraded (0.938%); 1632/218865 objects misplaced (0.746%); 6.0 MiB/s, 96 objects/s recovering 2024-08-17T17:55:48.727 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:48 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4000047458' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:55:48.727 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:48 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:55:48.737 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:48.759 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:48.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:48.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:48.861 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:48.904 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:48.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:48.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:49.006 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:49.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:49.032 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:49.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:49.046 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-17T17:55:49.134 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:49.176 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:49.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:49.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:49.278 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:49.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:49.313 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:49.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:49.335 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:55:49.413 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:49.452 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:49.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:49.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:49.554 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:49.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:49.590 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:49.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:49.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:49 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T17:55:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:49 smithi136 ceph-mon[109861]: osdmap e1339: 8 total, 8 up, 8 in 2024-08-17T17:55:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:49 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 2053/218865 objects degraded (0.938%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:49.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:49 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T17:55:49.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:49 smithi136 ceph-mon[115078]: osdmap e1339: 8 total, 8 up, 8 in 2024-08-17T17:55:49.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:49 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 2053/218865 objects degraded (0.938%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:49.699 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:49 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T17:55:49.699 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:49 smithi165 ceph-mon[105517]: osdmap e1339: 8 total, 8 up, 8 in 2024-08-17T17:55:49.699 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:49 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 2053/218865 objects degraded (0.938%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:49.713 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:49.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:49.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:49.815 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:49.853 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:49.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:49.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:49.954 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:49.995 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:49.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:49.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:50.098 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:50.142 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:50.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:50.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:50.244 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:50.286 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:50.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:50.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:50.388 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:50.415 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:50.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:50.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:50.517 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:50.544 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:50.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:50.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:50.646 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:50.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:50 smithi165 ceph-mon[105517]: pgmap v2188: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 97 active+clean; 4.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 860 op/s; 2053/213333 objects degraded (0.962%); 1632/213333 objects misplaced (0.765%); 5.6 MiB/s, 89 objects/s recovering 2024-08-17T17:55:50.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:50.683 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:50.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:50.784 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:50.823 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:50.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:50.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:50.924 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:50.940 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:50 smithi136 ceph-mon[109861]: pgmap v2188: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 97 active+clean; 4.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 860 op/s; 2053/213333 objects degraded (0.962%); 1632/213333 objects misplaced (0.765%); 5.6 MiB/s, 89 objects/s recovering 2024-08-17T17:55:50.940 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:50 smithi136 ceph-mon[115078]: pgmap v2188: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 97 active+clean; 4.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 860 op/s; 2053/213333 objects degraded (0.962%); 1632/213333 objects misplaced (0.765%); 5.6 MiB/s, 89 objects/s recovering 2024-08-17T17:55:50.958 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:50.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:50.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:50.977 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 7 1 2024-08-17T17:55:51.060 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:51.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:51.112 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:51.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:51.213 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:51.278 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:51.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:51.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:51.351 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T17:55:51.351 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T17:55:51.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T17:55:51.351 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T17:55:51.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.351 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:32836 53.0 53:02eb864f:::benchmark_data_smithi136_235573_object32835:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:55:51.351 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:53:29.747897+0000", 2024-08-17T17:55:51.352 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 141.59653553199999, 2024-08-17T17:55:51.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.19520247499999999, 2024-08-17T17:55:51.352 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:55:51.352 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:55:51.352 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:55:51.352 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:55:51.352 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:55:51.352 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 32836 2024-08-17T17:55:51.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.352 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:55:51.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:55:51.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.747897+0000", 2024-08-17T17:55:51.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:55:51.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.747897+0000", 2024-08-17T17:55:51.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:55:51.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.747896+0000", 2024-08-17T17:55:51.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:55:51.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:55:51.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.748244+0000", 2024-08-17T17:55:51.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034770800000000001 2024-08-17T17:55:51.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:55:51.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.748254+0000", 2024-08-17T17:55:51.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0686999999999999e-05 2024-08-17T17:55:51.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:55:51.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.748261+0000", 2024-08-17T17:55:51.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2820000000000002e-06 2024-08-17T17:55:51.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:55:51.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.754505+0000", 2024-08-17T17:55:51.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062446990000000003 2024-08-17T17:55:51.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:55:51.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.754613+0000", 2024-08-17T17:55:51.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010746 2024-08-17T17:55:51.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:55:51.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.754898+0000", 2024-08-17T17:55:51.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028547700000000002 2024-08-17T17:55:51.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.760976+0000", 2024-08-17T17:55:51.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060777560000000001 2024-08-17T17:55:51.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:55:51.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.760982+0000", 2024-08-17T17:55:51.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9359999999999996e-06 2024-08-17T17:55:51.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.943006+0000", 2024-08-17T17:55:51.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18202437599999999 2024-08-17T17:55:51.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:55:51.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.943054+0000", 2024-08-17T17:55:51.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7157000000000003e-05 2024-08-17T17:55:51.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:55:51.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.943099+0000", 2024-08-17T17:55:51.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5803999999999997e-05 2024-08-17T17:55:51.358 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.358 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:51.358 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.358 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:32849 53.0 53:09f7d5d1:::benchmark_data_smithi136_235573_object32848:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:55:51.358 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:53:29.755912+0000", 2024-08-17T17:55:51.358 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 141.58852014300001, 2024-08-17T17:55:51.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.187285812, 2024-08-17T17:55:51.358 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:55:51.358 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:55:51.359 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:55:51.359 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:55:51.359 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:55:51.359 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 32849 2024-08-17T17:55:51.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.359 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:55:51.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:55:51.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.755912+0000", 2024-08-17T17:55:51.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:55:51.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.755912+0000", 2024-08-17T17:55:51.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:55:51.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.755911+0000", 2024-08-17T17:55:51.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:55:51.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:55:51.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.755943+0000", 2024-08-17T17:55:51.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2493000000000002e-05 2024-08-17T17:55:51.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:55:51.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.755958+0000", 2024-08-17T17:55:51.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4537e-05 2024-08-17T17:55:51.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:55:51.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.755965+0000", 2024-08-17T17:55:51.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7279999999999998e-06 2024-08-17T17:55:51.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:55:51.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.756270+0000", 2024-08-17T17:55:51.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00030566799999999999 2024-08-17T17:55:51.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:55:51.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.756614+0000", 2024-08-17T17:55:51.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034377799999999999 2024-08-17T17:55:51.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:55:51.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.756799+0000", 2024-08-17T17:55:51.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000184833 2024-08-17T17:55:51.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:55:51.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.760987+0000", 2024-08-17T17:55:51.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041880830000000004 2024-08-17T17:55:51.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.761020+0000", 2024-08-17T17:55:51.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3182000000000002e-05 2024-08-17T17:55:51.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.943134+0000", 2024-08-17T17:55:51.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18211419400000001 2024-08-17T17:55:51.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:55:51.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.943169+0000", 2024-08-17T17:55:51.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4922999999999998e-05 2024-08-17T17:55:51.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:55:51.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.943198+0000", 2024-08-17T17:55:51.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8822999999999998e-05 2024-08-17T17:55:51.364 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.364 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:51.364 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.365 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:32851 53.0 53:0f637703:::benchmark_data_smithi136_235573_object32850:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:55:51.365 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:53:29.755976+0000", 2024-08-17T17:55:51.365 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 141.58845617, 2024-08-17T17:55:51.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18731230600000001, 2024-08-17T17:55:51.365 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:55:51.365 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:55:51.365 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:55:51.365 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:55:51.365 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:55:51.365 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 32851 2024-08-17T17:55:51.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.366 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:55:51.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:55:51.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.755976+0000", 2024-08-17T17:55:51.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:55:51.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.755976+0000", 2024-08-17T17:55:51.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:55:51.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.755976+0000", 2024-08-17T17:55:51.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T17:55:51.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:55:51.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.756148+0000", 2024-08-17T17:55:51.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017265600000000001 2024-08-17T17:55:51.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:55:51.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.756160+0000", 2024-08-17T17:55:51.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2099999999999999e-05 2024-08-17T17:55:51.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:55:51.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.756167+0000", 2024-08-17T17:55:51.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5200000000000003e-06 2024-08-17T17:55:51.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:55:51.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.758191+0000", 2024-08-17T17:55:51.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020242179999999999 2024-08-17T17:55:51.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:55:51.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.758294+0000", 2024-08-17T17:55:51.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102617 2024-08-17T17:55:51.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:55:51.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.758408+0000", 2024-08-17T17:55:51.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113979 2024-08-17T17:55:51.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:55:51.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.760996+0000", 2024-08-17T17:55:51.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002588202 2024-08-17T17:55:51.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.761047+0000", 2024-08-17T17:55:51.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1462000000000003e-05 2024-08-17T17:55:51.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.943228+0000", 2024-08-17T17:55:51.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18218055699999999 2024-08-17T17:55:51.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:55:51.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.943261+0000", 2024-08-17T17:55:51.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2750999999999999e-05 2024-08-17T17:55:51.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:55:51.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.943289+0000", 2024-08-17T17:55:51.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7807999999999999e-05 2024-08-17T17:55:51.371 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.371 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:51.371 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.371 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:32877 53.0 53:0264ef60:::benchmark_data_smithi136_235573_object32876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:55:51.371 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:53:29.766396+0000", 2024-08-17T17:55:51.371 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 141.57803663799999, 2024-08-17T17:55:51.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17698629099999999, 2024-08-17T17:55:51.371 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:55:51.371 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:55:51.372 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:55:51.372 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:55:51.372 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:55:51.372 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 32877 2024-08-17T17:55:51.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.372 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:55:51.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:55:51.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.766396+0000", 2024-08-17T17:55:51.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:55:51.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.766396+0000", 2024-08-17T17:55:51.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:55:51.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.766394+0000", 2024-08-17T17:55:51.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:55:51.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:55:51.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.766454+0000", 2024-08-17T17:55:51.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9061999999999998e-05 2024-08-17T17:55:51.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:55:51.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.766468+0000", 2024-08-17T17:55:51.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4667e-05 2024-08-17T17:55:51.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:55:51.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.766475+0000", 2024-08-17T17:55:51.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3779999999999998e-06 2024-08-17T17:55:51.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:55:51.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.766518+0000", 2024-08-17T17:55:51.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3046000000000001e-05 2024-08-17T17:55:51.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:55:51.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.766634+0000", 2024-08-17T17:55:51.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116704 2024-08-17T17:55:51.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:55:51.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.766766+0000", 2024-08-17T17:55:51.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013154700000000001 2024-08-17T17:55:51.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:55:51.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.768094+0000", 2024-08-17T17:55:51.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013283500000000001 2024-08-17T17:55:51.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.768433+0000", 2024-08-17T17:55:51.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033874900000000001 2024-08-17T17:55:51.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.943322+0000", 2024-08-17T17:55:51.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.174889086 2024-08-17T17:55:51.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:55:51.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.943355+0000", 2024-08-17T17:55:51.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2938000000000002e-05 2024-08-17T17:55:51.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:55:51.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.943382+0000", 2024-08-17T17:55:51.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7086000000000001e-05 2024-08-17T17:55:51.377 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.377 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:51.377 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.377 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:32879 53.0 53:06564bb1:::benchmark_data_smithi136_235573_object32878:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:55:51.377 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:53:29.768466+0000", 2024-08-17T17:55:51.377 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 141.57596618700001, 2024-08-17T17:55:51.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17666860500000001, 2024-08-17T17:55:51.377 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:55:51.377 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:55:51.377 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:55:51.378 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:55:51.378 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:55:51.378 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 32879 2024-08-17T17:55:51.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.378 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:55:51.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:55:51.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.768466+0000", 2024-08-17T17:55:51.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:55:51.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.768466+0000", 2024-08-17T17:55:51.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:55:51.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.768465+0000", 2024-08-17T17:55:51.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:55:51.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:55:51.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.768519+0000", 2024-08-17T17:55:51.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4565999999999999e-05 2024-08-17T17:55:51.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:55:51.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.768532+0000", 2024-08-17T17:55:51.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2354999999999999e-05 2024-08-17T17:55:51.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:55:51.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.768537+0000", 2024-08-17T17:55:51.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9400000000000001e-06 2024-08-17T17:55:51.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:55:51.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.768565+0000", 2024-08-17T17:55:51.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8663e-05 2024-08-17T17:55:51.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:55:51.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.768660+0000", 2024-08-17T17:55:51.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5052999999999999e-05 2024-08-17T17:55:51.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:55:51.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.768752+0000", 2024-08-17T17:55:51.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1154000000000002e-05 2024-08-17T17:55:51.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:55:51.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.769702+0000", 2024-08-17T17:55:51.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00095040700000000001 2024-08-17T17:55:51.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.769963+0000", 2024-08-17T17:55:51.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026142599999999999 2024-08-17T17:55:51.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.945036+0000", 2024-08-17T17:55:51.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.175072333 2024-08-17T17:55:51.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:55:51.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.945099+0000", 2024-08-17T17:55:51.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2903000000000004e-05 2024-08-17T17:55:51.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:55:51.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.945135+0000", 2024-08-17T17:55:51.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6208999999999998e-05 2024-08-17T17:55:51.384 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.384 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:51.384 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.384 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.35091.0:67749 53.4 e1312/1310 53:25b52b06:::benchmark_data_smithi136_235573_object67748:head v 1312'3986, mlcod=1312'3984)", 2024-08-17T17:55:51.384 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:53:51.853324+0000", 2024-08-17T17:55:51.384 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 119.49110868299999, 2024-08-17T17:55:51.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17928466400000001, 2024-08-17T17:55:51.384 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:55:51.384 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:55:51.384 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:55:51.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:55:51.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:51.853324+0000", 2024-08-17T17:55:51.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:55:51.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:51.853324+0000", 2024-08-17T17:55:51.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:55:51.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:51.853323+0000", 2024-08-17T17:55:51.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:55:51.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:55:51.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:51.853376+0000", 2024-08-17T17:55:51.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3260999999999997e-05 2024-08-17T17:55:51.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:55:51.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:51.853385+0000", 2024-08-17T17:55:51.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9700000000000005e-06 2024-08-17T17:55:51.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:55:51.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:51.853391+0000", 2024-08-17T17:55:51.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.105e-06 2024-08-17T17:55:51.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:55:51.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:51.853430+0000", 2024-08-17T17:55:51.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8838999999999999e-05 2024-08-17T17:55:51.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:55:51.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:51.853444+0000", 2024-08-17T17:55:51.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3525e-05 2024-08-17T17:55:51.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:55:51.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.032579+0000", 2024-08-17T17:55:51.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17913513 2024-08-17T17:55:51.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:55:51.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.032608+0000", 2024-08-17T17:55:51.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9703999999999999e-05 2024-08-17T17:55:51.388 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.388 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:51.388 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.388 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.35091.0:67753 53.a e1312/1303 53:596cd44f:::benchmark_data_smithi136_235573_object67752:head v 1312'4291, mlcod=1312'4288)", 2024-08-17T17:55:51.388 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:53:51.856118+0000", 2024-08-17T17:55:51.389 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 119.488314055, 2024-08-17T17:55:51.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.181175419, 2024-08-17T17:55:51.389 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:55:51.389 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:55:51.389 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:55:51.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:55:51.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:51.856118+0000", 2024-08-17T17:55:51.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:55:51.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:51.856118+0000", 2024-08-17T17:55:51.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:55:51.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:51.856118+0000", 2024-08-17T17:55:51.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:55:51.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:55:51.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:51.856239+0000", 2024-08-17T17:55:51.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012084899999999999 2024-08-17T17:55:51.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:55:51.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.025085+0000", 2024-08-17T17:55:51.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16884622899999999 2024-08-17T17:55:51.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:55:51.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.025104+0000", 2024-08-17T17:55:51.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9629000000000001e-05 2024-08-17T17:55:51.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:55:51.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.025449+0000", 2024-08-17T17:55:51.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034467699999999999 2024-08-17T17:55:51.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:55:51.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.025473+0000", 2024-08-17T17:55:51.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3714e-05 2024-08-17T17:55:51.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:55:51.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.037231+0000", 2024-08-17T17:55:51.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011758608 2024-08-17T17:55:51.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:55:51.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.037294+0000", 2024-08-17T17:55:51.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2428999999999995e-05 2024-08-17T17:55:51.393 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.393 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:51.393 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.393 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:68027 53.10 53:0fbede35:::benchmark_data_smithi136_235573_object68026:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1312)", 2024-08-17T17:55:51.393 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:53:52.222814+0000", 2024-08-17T17:55:51.393 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 119.121618044, 2024-08-17T17:55:51.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.122685277, 2024-08-17T17:55:51.393 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:55:51.393 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:55:51.393 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:55:51.393 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:55:51.393 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:55:51.393 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 68027 2024-08-17T17:55:51.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.394 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:55:51.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:55:51.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.222814+0000", 2024-08-17T17:55:51.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:55:51.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.222814+0000", 2024-08-17T17:55:51.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:55:51.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.222810+0000", 2024-08-17T17:55:51.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999957 2024-08-17T17:55:51.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:55:51.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.222856+0000", 2024-08-17T17:55:51.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5636000000000001e-05 2024-08-17T17:55:51.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:55:51.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.222869+0000", 2024-08-17T17:55:51.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3363000000000001e-05 2024-08-17T17:55:51.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:55:51.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.222876+0000", 2024-08-17T17:55:51.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8310000000000003e-06 2024-08-17T17:55:51.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:55:51.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.222925+0000", 2024-08-17T17:55:51.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8693000000000003e-05 2024-08-17T17:55:51.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:55:51.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.223051+0000", 2024-08-17T17:55:51.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012587300000000001 2024-08-17T17:55:51.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:55:51.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.223153+0000", 2024-08-17T17:55:51.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010232 2024-08-17T17:55:51.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.224617+0000", 2024-08-17T17:55:51.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014645210000000001 2024-08-17T17:55:51.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:55:51.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.224630+0000", 2024-08-17T17:55:51.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2160999999999999e-05 2024-08-17T17:55:51.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.345416+0000", 2024-08-17T17:55:51.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.120786504 2024-08-17T17:55:51.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:55:51.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.345466+0000", 2024-08-17T17:55:51.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0312000000000002e-05 2024-08-17T17:55:51.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:55:51.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.345500+0000", 2024-08-17T17:55:51.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3244000000000002e-05 2024-08-17T17:55:51.399 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.399 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:51.399 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.399 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:68288 53.0 53:02be0f21:::benchmark_data_smithi136_235573_object68287:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1312)", 2024-08-17T17:55:51.399 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:53:52.477753+0000", 2024-08-17T17:55:51.399 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 118.866679023, 2024-08-17T17:55:51.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.157531381, 2024-08-17T17:55:51.400 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:55:51.400 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:55:51.400 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:55:51.400 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:55:51.400 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:55:51.400 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 68288 2024-08-17T17:55:51.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.400 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:55:51.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:55:51.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.477753+0000", 2024-08-17T17:55:51.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:55:51.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.477753+0000", 2024-08-17T17:55:51.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:55:51.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.477752+0000", 2024-08-17T17:55:51.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:55:51.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:55:51.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.477804+0000", 2024-08-17T17:55:51.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1996000000000002e-05 2024-08-17T17:55:51.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:55:51.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.477813+0000", 2024-08-17T17:55:51.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9579999999999996e-06 2024-08-17T17:55:51.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:55:51.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.477818+0000", 2024-08-17T17:55:51.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4689999999999999e-06 2024-08-17T17:55:51.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:55:51.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.477847+0000", 2024-08-17T17:55:51.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9728999999999999e-05 2024-08-17T17:55:51.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:55:51.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.477926+0000", 2024-08-17T17:55:51.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8639000000000004e-05 2024-08-17T17:55:51.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:55:51.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.478017+0000", 2024-08-17T17:55:51.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0755999999999996e-05 2024-08-17T17:55:51.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:55:51.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.479273+0000", 2024-08-17T17:55:51.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012560679999999999 2024-08-17T17:55:51.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.480317+0000", 2024-08-17T17:55:51.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010437949999999999 2024-08-17T17:55:51.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.635132+0000", 2024-08-17T17:55:51.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.154814902 2024-08-17T17:55:51.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:55:51.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.635245+0000", 2024-08-17T17:55:51.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113453 2024-08-17T17:55:51.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:55:51.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.635285+0000", 2024-08-17T17:55:51.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9687999999999997e-05 2024-08-17T17:55:51.405 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.405 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:51.405 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.406 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:68294 53.0 53:06d875bf:::benchmark_data_smithi136_235573_object68293:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1312)", 2024-08-17T17:55:51.406 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:53:52.481453+0000", 2024-08-17T17:55:51.406 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 118.86297954699999, 2024-08-17T17:55:51.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15402068199999999, 2024-08-17T17:55:51.406 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:55:51.406 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:55:51.406 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:55:51.406 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:55:51.406 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:55:51.406 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 68294 2024-08-17T17:55:51.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.407 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:55:51.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:55:51.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.481453+0000", 2024-08-17T17:55:51.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:55:51.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.481453+0000", 2024-08-17T17:55:51.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:55:51.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.481451+0000", 2024-08-17T17:55:51.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999976 2024-08-17T17:55:51.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:55:51.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.481498+0000", 2024-08-17T17:55:51.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7360000000000001e-05 2024-08-17T17:55:51.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:55:51.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.481520+0000", 2024-08-17T17:55:51.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1605999999999999e-05 2024-08-17T17:55:51.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:55:51.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.481532+0000", 2024-08-17T17:55:51.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2476e-05 2024-08-17T17:55:51.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:55:51.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.481600+0000", 2024-08-17T17:55:51.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7772000000000001e-05 2024-08-17T17:55:51.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:55:51.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.481798+0000", 2024-08-17T17:55:51.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019790399999999999 2024-08-17T17:55:51.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:55:51.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.481877+0000", 2024-08-17T17:55:51.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9126000000000001e-05 2024-08-17T17:55:51.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:55:51.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.482761+0000", 2024-08-17T17:55:51.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00088436099999999996 2024-08-17T17:55:51.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.483461+0000", 2024-08-17T17:55:51.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00070001799999999995 2024-08-17T17:55:51.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.635378+0000", 2024-08-17T17:55:51.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15191692700000001 2024-08-17T17:55:51.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:55:51.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.635443+0000", 2024-08-17T17:55:51.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4700000000000001e-05 2024-08-17T17:55:51.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:55:51.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.635474+0000", 2024-08-17T17:55:51.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0582e-05 2024-08-17T17:55:51.412 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.412 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:51.412 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.412 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:120896 53.18 53:1f17b9c4:::benchmark_data_smithi136_235573_object120895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-08-17T17:55:51.412 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:54:44.099773+0000", 2024-08-17T17:55:51.412 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 67.244659260000006, 2024-08-17T17:55:51.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15623999999999999, 2024-08-17T17:55:51.412 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:55:51.412 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:55:51.413 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:55:51.413 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:55:51.413 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:55:51.413 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 120896 2024-08-17T17:55:51.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.413 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:55:51.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:55:51.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.099773+0000", 2024-08-17T17:55:51.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:55:51.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.099773+0000", 2024-08-17T17:55:51.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:55:51.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.099772+0000", 2024-08-17T17:55:51.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:55:51.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:55:51.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.099817+0000", 2024-08-17T17:55:51.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5210000000000003e-05 2024-08-17T17:55:51.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:55:51.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.099836+0000", 2024-08-17T17:55:51.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9207000000000001e-05 2024-08-17T17:55:51.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:55:51.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.099843+0000", 2024-08-17T17:55:51.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0430000000000002e-06 2024-08-17T17:55:51.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:55:51.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.099889+0000", 2024-08-17T17:55:51.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5683000000000003e-05 2024-08-17T17:55:51.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:55:51.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.100009+0000", 2024-08-17T17:55:51.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012064400000000001 2024-08-17T17:55:51.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:55:51.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.100135+0000", 2024-08-17T17:55:51.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012608000000000001 2024-08-17T17:55:51.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:55:51.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.102051+0000", 2024-08-17T17:55:51.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001915871 2024-08-17T17:55:51.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.102467+0000", 2024-08-17T17:55:51.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00041593199999999999 2024-08-17T17:55:51.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.255920+0000", 2024-08-17T17:55:51.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15345252100000001 2024-08-17T17:55:51.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:55:51.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.255975+0000", 2024-08-17T17:55:51.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5290000000000001e-05 2024-08-17T17:55:51.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:55:51.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.256013+0000", 2024-08-17T17:55:51.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8129e-05 2024-08-17T17:55:51.418 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.418 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:51.418 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.419 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:121084 53.18 53:1abe089f:::benchmark_data_smithi136_235573_object121083:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-08-17T17:55:51.419 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:54:44.420430+0000", 2024-08-17T17:55:51.419 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.924002204999994, 2024-08-17T17:55:51.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12166052199999999, 2024-08-17T17:55:51.419 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:55:51.419 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:55:51.419 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:55:51.419 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:55:51.419 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:55:51.419 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 121084 2024-08-17T17:55:51.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.419 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:55:51.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:55:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.420430+0000", 2024-08-17T17:55:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:55:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.420430+0000", 2024-08-17T17:55:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:55:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.420429+0000", 2024-08-17T17:55:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:55:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:55:51.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.420508+0000", 2024-08-17T17:55:51.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9254999999999999e-05 2024-08-17T17:55:51.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:55:51.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.420520+0000", 2024-08-17T17:55:51.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1915e-05 2024-08-17T17:55:51.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.421 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:51.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:55:51.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.420527+0000", 2024-08-17T17:55:51.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1600000000000001e-06 2024-08-17T17:55:51.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:55:51.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.420580+0000", 2024-08-17T17:55:51.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2692999999999998e-05 2024-08-17T17:55:51.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:55:51.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.420736+0000", 2024-08-17T17:55:51.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015599599999999999 2024-08-17T17:55:51.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:55:51.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.420894+0000", 2024-08-17T17:55:51.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015825400000000001 2024-08-17T17:55:51.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:55:51.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.424214+0000", 2024-08-17T17:55:51.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00332037 2024-08-17T17:55:51.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.428495+0000", 2024-08-17T17:55:51.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00428101 2024-08-17T17:55:51.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.541993+0000", 2024-08-17T17:55:51.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.113497368 2024-08-17T17:55:51.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:55:51.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.542038+0000", 2024-08-17T17:55:51.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5334000000000003e-05 2024-08-17T17:55:51.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:55:51.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.542091+0000", 2024-08-17T17:55:51.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2670000000000002e-05 2024-08-17T17:55:51.426 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.426 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:51.426 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.426 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:121095 53.18 53:1837058b:::benchmark_data_smithi136_235573_object121094:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-08-17T17:55:51.426 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:54:44.428572+0000", 2024-08-17T17:55:51.426 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.915860648000006, 2024-08-17T17:55:51.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12182949999999999, 2024-08-17T17:55:51.427 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:55:51.427 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:55:51.427 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:55:51.427 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:55:51.427 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:55:51.427 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 121095 2024-08-17T17:55:51.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.427 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:55:51.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:55:51.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.428572+0000", 2024-08-17T17:55:51.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:55:51.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.428572+0000", 2024-08-17T17:55:51.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:55:51.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.428564+0000", 2024-08-17T17:55:51.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999924 2024-08-17T17:55:51.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:55:51.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.428607+0000", 2024-08-17T17:55:51.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2509000000000002e-05 2024-08-17T17:55:51.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:55:51.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.428617+0000", 2024-08-17T17:55:51.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8430000000000005e-06 2024-08-17T17:55:51.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:55:51.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.428623+0000", 2024-08-17T17:55:51.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.913e-06 2024-08-17T17:55:51.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:55:51.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.428660+0000", 2024-08-17T17:55:51.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7285e-05 2024-08-17T17:55:51.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:55:51.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.428780+0000", 2024-08-17T17:55:51.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120491 2024-08-17T17:55:51.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:55:51.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.428908+0000", 2024-08-17T17:55:51.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012789 2024-08-17T17:55:51.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:55:51.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.433371+0000", 2024-08-17T17:55:51.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044628369999999999 2024-08-17T17:55:51.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.433419+0000", 2024-08-17T17:55:51.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8337000000000002e-05 2024-08-17T17:55:51.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.550313+0000", 2024-08-17T17:55:51.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.116893414 2024-08-17T17:55:51.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:55:51.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.550363+0000", 2024-08-17T17:55:51.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0127999999999998e-05 2024-08-17T17:55:51.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:55:51.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.550401+0000", 2024-08-17T17:55:51.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8316999999999999e-05 2024-08-17T17:55:51.432 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.432 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:51.432 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.432 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:121099 53.8 53:133c9910:::benchmark_data_smithi136_235573_object121098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-08-17T17:55:51.432 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:54:44.431033+0000", 2024-08-17T17:55:51.433 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 66.913399456999997, 2024-08-17T17:55:51.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.121192772, 2024-08-17T17:55:51.433 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:55:51.433 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:55:51.433 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:55:51.433 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:55:51.433 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:55:51.433 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 121099 2024-08-17T17:55:51.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.433 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:55:51.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:55:51.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.431033+0000", 2024-08-17T17:55:51.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:55:51.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.431033+0000", 2024-08-17T17:55:51.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:55:51.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.431032+0000", 2024-08-17T17:55:51.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T17:55:51.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:55:51.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.431063+0000", 2024-08-17T17:55:51.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0479000000000001e-05 2024-08-17T17:55:51.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:55:51.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.431080+0000", 2024-08-17T17:55:51.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6687000000000001e-05 2024-08-17T17:55:51.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:55:51.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.431083+0000", 2024-08-17T17:55:51.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.839e-06 2024-08-17T17:55:51.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:55:51.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.433395+0000", 2024-08-17T17:55:51.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023116809999999999 2024-08-17T17:55:51.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:55:51.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.433495+0000", 2024-08-17T17:55:51.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9617999999999994e-05 2024-08-17T17:55:51.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:55:51.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.433584+0000", 2024-08-17T17:55:51.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9190000000000005e-05 2024-08-17T17:55:51.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.435033+0000", 2024-08-17T17:55:51.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014488979999999999 2024-08-17T17:55:51.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:55:51.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.435038+0000", 2024-08-17T17:55:51.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3260000000000003e-06 2024-08-17T17:55:51.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.552113+0000", 2024-08-17T17:55:51.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.117075243 2024-08-17T17:55:51.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:55:51.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.552190+0000", 2024-08-17T17:55:51.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6416999999999997e-05 2024-08-17T17:55:51.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:55:51.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.552226+0000", 2024-08-17T17:55:51.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6018e-05 2024-08-17T17:55:51.438 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.438 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:51.438 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.439 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:147656 53.18 53:1c9ded58:::benchmark_data_smithi136_235573_object23370:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1332)", 2024-08-17T17:55:51.439 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:55:12.129405+0000", 2024-08-17T17:55:51.439 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.215027640999999, 2024-08-17T17:55:51.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.169596569, 2024-08-17T17:55:51.439 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:55:51.439 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:55:51.439 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:55:51.439 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:55:51.439 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:55:51.439 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 147656 2024-08-17T17:55:51.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.439 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:55:51.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:55:51.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.129405+0000", 2024-08-17T17:55:51.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:55:51.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.129405+0000", 2024-08-17T17:55:51.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:55:51.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.129404+0000", 2024-08-17T17:55:51.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:55:51.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:55:51.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.129409+0000", 2024-08-17T17:55:51.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1309999999999997e-06 2024-08-17T17:55:51.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:55:51.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.129418+0000", 2024-08-17T17:55:51.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5049999999999996e-06 2024-08-17T17:55:51.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:55:51.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.129424+0000", 2024-08-17T17:55:51.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.835e-06 2024-08-17T17:55:51.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:55:51.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.129603+0000", 2024-08-17T17:55:51.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000179249 2024-08-17T17:55:51.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:55:51.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.129717+0000", 2024-08-17T17:55:51.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113615 2024-08-17T17:55:51.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:55:51.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.130110+0000", 2024-08-17T17:55:51.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039272100000000001 2024-08-17T17:55:51.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:55:51.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.132566+0000", 2024-08-17T17:55:51.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024558900000000001 2024-08-17T17:55:51.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.138376+0000", 2024-08-17T17:55:51.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058102040000000002 2024-08-17T17:55:51.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.298920+0000", 2024-08-17T17:55:51.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16054427199999999 2024-08-17T17:55:51.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:55:51.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.298961+0000", 2024-08-17T17:55:51.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0667999999999998e-05 2024-08-17T17:55:51.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:55:51.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.299001+0000", 2024-08-17T17:55:51.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0710999999999997e-05 2024-08-17T17:55:51.444 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.444 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:147907 53.18 53:1f47ed04:::benchmark_data_smithi136_235573_object23621:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1332)", 2024-08-17T17:55:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:55:12.533549+0000", 2024-08-17T17:55:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.810883220999997, 2024-08-17T17:55:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.137997445, 2024-08-17T17:55:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:55:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:55:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:55:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:55:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:55:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 147907 2024-08-17T17:55:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:55:51.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:55:51.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.533549+0000", 2024-08-17T17:55:51.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:55:51.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.533549+0000", 2024-08-17T17:55:51.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:55:51.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.533548+0000", 2024-08-17T17:55:51.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:55:51.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:55:51.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.533554+0000", 2024-08-17T17:55:51.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9780000000000002e-06 2024-08-17T17:55:51.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:55:51.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.533568+0000", 2024-08-17T17:55:51.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.448e-05 2024-08-17T17:55:51.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:55:51.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.533578+0000", 2024-08-17T17:55:51.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8230000000000006e-06 2024-08-17T17:55:51.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:55:51.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.533615+0000", 2024-08-17T17:55:51.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6698e-05 2024-08-17T17:55:51.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:55:51.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.533695+0000", 2024-08-17T17:55:51.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0012000000000006e-05 2024-08-17T17:55:51.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:55:51.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.534054+0000", 2024-08-17T17:55:51.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00035874399999999998 2024-08-17T17:55:51.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:55:51.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.535689+0000", 2024-08-17T17:55:51.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016348090000000001 2024-08-17T17:55:51.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.663816+0000", 2024-08-17T17:55:51.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12812737299999999 2024-08-17T17:55:51.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.671467+0000", 2024-08-17T17:55:51.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0076514649999999997 2024-08-17T17:55:51.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:55:51.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.671507+0000", 2024-08-17T17:55:51.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9885999999999997e-05 2024-08-17T17:55:51.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:55:51.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.671547+0000", 2024-08-17T17:55:51.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9357999999999998e-05 2024-08-17T17:55:51.451 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.451 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:51.451 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.451 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:148115 53.8 53:14b82a5c:::benchmark_data_smithi136_235573_object23829:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1332)", 2024-08-17T17:55:51.451 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:55:12.859699+0000", 2024-08-17T17:55:51.451 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.484733611999999, 2024-08-17T17:55:51.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.14147837399999999, 2024-08-17T17:55:51.451 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:55:51.451 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:55:51.451 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:55:51.451 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:55:51.452 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:55:51.452 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 148115 2024-08-17T17:55:51.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.452 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:55:51.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:55:51.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.859699+0000", 2024-08-17T17:55:51.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:55:51.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.859699+0000", 2024-08-17T17:55:51.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:55:51.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.859697+0000", 2024-08-17T17:55:51.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:55:51.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:55:51.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.859704+0000", 2024-08-17T17:55:51.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8619999999999997e-06 2024-08-17T17:55:51.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:55:51.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.859717+0000", 2024-08-17T17:55:51.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2544000000000001e-05 2024-08-17T17:55:51.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:55:51.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.859724+0000", 2024-08-17T17:55:51.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3640000000000001e-06 2024-08-17T17:55:51.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:55:51.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.859776+0000", 2024-08-17T17:55:51.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2345000000000001e-05 2024-08-17T17:55:51.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:55:51.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.859878+0000", 2024-08-17T17:55:51.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010153999999999999 2024-08-17T17:55:51.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:55:51.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.860252+0000", 2024-08-17T17:55:51.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037417899999999998 2024-08-17T17:55:51.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:55:51.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.861970+0000", 2024-08-17T17:55:51.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00171743 2024-08-17T17:55:51.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.862503+0000", 2024-08-17T17:55:51.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00053330499999999998 2024-08-17T17:55:51.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:13.001115+0000", 2024-08-17T17:55:51.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13861174400000001 2024-08-17T17:55:51.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:55:51.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:13.001152+0000", 2024-08-17T17:55:51.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7342999999999998e-05 2024-08-17T17:55:51.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:55:51.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:13.001177+0000", 2024-08-17T17:55:51.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5258000000000001e-05 2024-08-17T17:55:51.457 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.457 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:51.457 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.458 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:177232 53.18 53:197a93a2:::benchmark_data_smithi136_235573_object52946:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1338)", 2024-08-17T17:55:51.458 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:55:46.009266+0000", 2024-08-17T17:55:51.458 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.3351662710000003, 2024-08-17T17:55:51.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12895083299999999, 2024-08-17T17:55:51.458 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:55:51.458 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:55:51.458 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:55:51.458 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:55:51.458 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:55:51.458 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 177232 2024-08-17T17:55:51.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.459 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:55:51.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:55:51.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.009266+0000", 2024-08-17T17:55:51.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:55:51.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.009266+0000", 2024-08-17T17:55:51.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:55:51.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.009265+0000", 2024-08-17T17:55:51.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:55:51.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:55:51.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.009270+0000", 2024-08-17T17:55:51.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6580000000000001e-06 2024-08-17T17:55:51.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:55:51.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.009283+0000", 2024-08-17T17:55:51.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.296e-05 2024-08-17T17:55:51.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:55:51.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.009288+0000", 2024-08-17T17:55:51.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5389999999999998e-06 2024-08-17T17:55:51.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:55:51.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.009343+0000", 2024-08-17T17:55:51.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5002e-05 2024-08-17T17:55:51.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:55:51.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.009416+0000", 2024-08-17T17:55:51.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2316000000000004e-05 2024-08-17T17:55:51.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:55:51.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.009836+0000", 2024-08-17T17:55:51.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042075800000000003 2024-08-17T17:55:51.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:55:51.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.012485+0000", 2024-08-17T17:55:51.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002648882 2024-08-17T17:55:51.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.134059+0000", 2024-08-17T17:55:51.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.121573854 2024-08-17T17:55:51.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.138121+0000", 2024-08-17T17:55:51.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040616669999999997 2024-08-17T17:55:51.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:55:51.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.138153+0000", 2024-08-17T17:55:51.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2357000000000002e-05 2024-08-17T17:55:51.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:55:51.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.138217+0000", 2024-08-17T17:55:51.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3931999999999997e-05 2024-08-17T17:55:51.464 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.464 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:51.464 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.464 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:177808 53.8 53:12038db4:::benchmark_data_smithi136_235573_object53522:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1338)", 2024-08-17T17:55:51.464 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:55:46.716653+0000", 2024-08-17T17:55:51.464 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.6277795099999999, 2024-08-17T17:55:51.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13229261, 2024-08-17T17:55:51.465 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:55:51.465 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:55:51.465 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:55:51.465 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:55:51.465 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:55:51.465 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 177808 2024-08-17T17:55:51.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.465 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:55:51.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:55:51.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.716653+0000", 2024-08-17T17:55:51.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:55:51.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.716653+0000", 2024-08-17T17:55:51.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:55:51.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.716653+0000", 2024-08-17T17:55:51.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T17:55:51.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:55:51.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.716654+0000", 2024-08-17T17:55:51.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8369999999999999e-06 2024-08-17T17:55:51.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:55:51.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.716658+0000", 2024-08-17T17:55:51.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9360000000000001e-06 2024-08-17T17:55:51.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:55:51.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.716661+0000", 2024-08-17T17:55:51.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8600000000000001e-06 2024-08-17T17:55:51.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:55:51.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.718104+0000", 2024-08-17T17:55:51.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001442664 2024-08-17T17:55:51.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:55:51.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.718247+0000", 2024-08-17T17:55:51.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014332900000000001 2024-08-17T17:55:51.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:55:51.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.718740+0000", 2024-08-17T17:55:51.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00049313500000000001 2024-08-17T17:55:51.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.723492+0000", 2024-08-17T17:55:51.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0047519459999999999 2024-08-17T17:55:51.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:55:51.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.724644+0000", 2024-08-17T17:55:51.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001151944 2024-08-17T17:55:51.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.848800+0000", 2024-08-17T17:55:51.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.124155319 2024-08-17T17:55:51.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:55:51.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.848892+0000", 2024-08-17T17:55:51.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1922999999999994e-05 2024-08-17T17:55:51.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:55:51.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.848946+0000", 2024-08-17T17:55:51.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3989000000000001e-05 2024-08-17T17:55:51.470 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.470 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:51.470 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.470 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:177811 53.8 53:17773269:::benchmark_data_smithi136_235573_object53525:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1338)", 2024-08-17T17:55:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:55:46.727055+0000", 2024-08-17T17:55:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.6173778370000003, 2024-08-17T17:55:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.1220001, 2024-08-17T17:55:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:55:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:55:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:55:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:55:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:55:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 177811 2024-08-17T17:55:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:55:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:55:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.727055+0000", 2024-08-17T17:55:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:55:51.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.727055+0000", 2024-08-17T17:55:51.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:51.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:55:51.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.727053+0000", 2024-08-17T17:55:51.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:55:51.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:55:51.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.727060+0000", 2024-08-17T17:55:51.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6370000000000002e-06 2024-08-17T17:55:51.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:55:51.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.727082+0000", 2024-08-17T17:55:51.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2619000000000001e-05 2024-08-17T17:55:51.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:55:51.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.727090+0000", 2024-08-17T17:55:51.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2599999999999999e-06 2024-08-17T17:55:51.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:55:51.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.727130+0000", 2024-08-17T17:55:51.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0802e-05 2024-08-17T17:55:51.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:55:51.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.727254+0000", 2024-08-17T17:55:51.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012341500000000001 2024-08-17T17:55:51.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:55:51.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.727718+0000", 2024-08-17T17:55:51.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000464313 2024-08-17T17:55:51.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:55:51.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.730166+0000", 2024-08-17T17:55:51.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024474850000000001 2024-08-17T17:55:51.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.730334+0000", 2024-08-17T17:55:51.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016850799999999999 2024-08-17T17:55:51.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:55:51.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.848980+0000", 2024-08-17T17:55:51.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.118645719 2024-08-17T17:55:51.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:55:51.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.849022+0000", 2024-08-17T17:55:51.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2126999999999998e-05 2024-08-17T17:55:51.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:51.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:51.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:55:51.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:46.849055+0000", 2024-08-17T17:55:51.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2753999999999998e-05 2024-08-17T17:55:51.476 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.476 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:51.476 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.476 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:51.477 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:51.477 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T17:55:51.508 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:51.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:51.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:51.585 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:55:51.610 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:51.631 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:55:51.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:51.635 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:51.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:51.733 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:55:51.733 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:55:51.736 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:55:51.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:51.765 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:51.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:51.866 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:51.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:51.931 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:51.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:52.031 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:52.079 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:52.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:52.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:52.181 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:52.232 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:52.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:52.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:52.334 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:52.337 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-17T17:55:52.378 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:52.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:52.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:52.477 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T17:55:52.479 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:52.520 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:52.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:52.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:52.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:52.651 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:52.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:52.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:52.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:52 smithi136 ceph-mon[109861]: pgmap v2189: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 97 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 863 op/s; 2053/211722 objects degraded (0.970%); 1632/211722 objects misplaced (0.771%); 5.6 MiB/s, 89 objects/s recovering 2024-08-17T17:55:52.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:55:52.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:52 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3321935168' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:55:52.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:52 smithi136 ceph-mon[115078]: pgmap v2189: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 97 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 863 op/s; 2053/211722 objects degraded (0.970%); 1632/211722 objects misplaced (0.771%); 5.6 MiB/s, 89 objects/s recovering 2024-08-17T17:55:52.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:55:52.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:52 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3321935168' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:55:52.752 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:52.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:52 smithi165 ceph-mon[105517]: pgmap v2189: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 97 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 863 op/s; 2053/211722 objects degraded (0.970%); 1632/211722 objects misplaced (0.771%); 5.6 MiB/s, 89 objects/s recovering 2024-08-17T17:55:52.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:55:52.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:52 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3321935168' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:55:52.789 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:52.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:52.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:52.891 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:52.930 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:52.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:52.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:52.993 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:55:53.032 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:53.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:53.085 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:53.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:53.186 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:53.229 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:53.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:53.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:53.332 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:53.369 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:53.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:53.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:53.471 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:53.510 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:53.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:53.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:53.612 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:53.640 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:53.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:53.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:53 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:55:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:53 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3321935168' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:55:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:53 smithi136 ceph-mon[109861]: osdmap e1340: 8 total, 8 up, 8 in 2024-08-17T17:55:53.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:53 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:55:53.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:53 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3321935168' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:55:53.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:53 smithi136 ceph-mon[115078]: osdmap e1340: 8 total, 8 up, 8 in 2024-08-17T17:55:53.742 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:53 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:55:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:53 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3321935168' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:55:53.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:53 smithi165 ceph-mon[105517]: osdmap e1340: 8 total, 8 up, 8 in 2024-08-17T17:55:53.778 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:53.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:53.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:53.880 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:53.920 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:53.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:53.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:54.022 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:54.059 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:54.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:54.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:54.160 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:54.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:54.205 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:54.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:54.306 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:54.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:54.375 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:54.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:54.476 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:54.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:54.534 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:54.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:54.565 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:54 smithi136 ceph-mon[109861]: pgmap v2191: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 97 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 1772/201372 objects degraded (0.880%); 1054/201372 objects misplaced (0.523%); 5.6 MiB/s, 96 objects/s recovering 2024-08-17T17:55:54.565 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:54 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 1772/201372 objects degraded (0.880%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:54.565 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:54 smithi136 ceph-mon[115078]: pgmap v2191: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 97 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 1772/201372 objects degraded (0.880%); 1054/201372 objects misplaced (0.523%); 5.6 MiB/s, 96 objects/s recovering 2024-08-17T17:55:54.565 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:54 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 1772/201372 objects degraded (0.880%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:54.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:54.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:54.667 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:54.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:54.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:54 smithi165 ceph-mon[105517]: pgmap v2191: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 97 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 1772/201372 objects degraded (0.880%); 1054/201372 objects misplaced (0.523%); 5.6 MiB/s, 96 objects/s recovering 2024-08-17T17:55:54.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:54 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 1772/201372 objects degraded (0.880%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:54.767 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:54.798 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:54.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:54.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:54.900 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:54.936 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:54.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:54.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:55.037 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:55.077 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:55.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:55.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:55.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:55.222 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:55.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:55.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:55.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:55.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:55.349 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:55.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:55.450 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:55.485 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:55:55.498 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:55.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:55.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:55.602 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:55.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:55.662 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:55.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:55.763 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:55.793 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:55.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:55.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:55.895 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:55.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:55.939 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:55.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:56.040 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:56.072 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:56.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:56.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:56.174 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:56.214 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:56.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:56.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:56.316 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:56.345 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:56.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:56.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:56.359 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T17:55:56.359 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T17:55:56.359 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:55:56.359 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:55:56.373 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T17:55:56.374 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:55:56.409 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T17:55:56.409 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:55:56.440 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T17:55:56.440 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:55:56.446 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T17:55:56.446 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T17:55:56.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:56.446 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.35091.0:186105 53.a e1340/1303 53:566cbc9f:::benchmark_data_smithi136_235573_object61819:head v 1340'11735, mlcod=1340'11730)", 2024-08-17T17:55:56.446 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:55:56.441090+0000", 2024-08-17T17:55:56.446 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0032080949999999998, 2024-08-17T17:55:56.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003244388, 2024-08-17T17:55:56.446 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:55:56.446 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T17:55:56.446 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:55:56.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:56.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:55:56.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:56.441090+0000", 2024-08-17T17:55:56.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:56.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:56.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:56.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:55:56.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:56.441090+0000", 2024-08-17T17:55:56.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:56.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:56.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:56.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:55:56.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:56.441088+0000", 2024-08-17T17:55:56.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:55:56.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:56.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:56.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:55:56.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:56.441095+0000", 2024-08-17T17:55:56.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9450000000000004e-06 2024-08-17T17:55:56.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:56.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:56.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:55:56.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:56.441102+0000", 2024-08-17T17:55:56.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0110000000000003e-06 2024-08-17T17:55:56.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:56.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:56.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:55:56.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:56.441107+0000", 2024-08-17T17:55:56.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6999999999999999e-06 2024-08-17T17:55:56.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:56.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:56.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:55:56.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:56.441142+0000", 2024-08-17T17:55:56.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4759999999999999e-05 2024-08-17T17:55:56.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:56.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:56.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:55:56.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:56.441156+0000", 2024-08-17T17:55:56.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4334e-05 2024-08-17T17:55:56.450 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:56.450 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:56.450 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:56.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:56.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:56.450 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.35091.0:186086 53.c e1340/1303 ondisk, result = 0)", 2024-08-17T17:55:56.450 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:55:56.443916+0000", 2024-08-17T17:55:56.450 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00038145900000000001, 2024-08-17T17:55:56.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00047975300000000003, 2024-08-17T17:55:56.450 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:55:56.450 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T17:55:56.450 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:55:56.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:56.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:55:56.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:56.443916+0000", 2024-08-17T17:55:56.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:56.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:56.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:56.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:55:56.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:56.443916+0000", 2024-08-17T17:55:56.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:55:56.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:56.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:56.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:55:56.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:56.443916+0000", 2024-08-17T17:55:56.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967296 2024-08-17T17:55:56.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:56.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:56.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:55:56.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:56.443918+0000", 2024-08-17T17:55:56.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.897e-06 2024-08-17T17:55:56.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:56.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:56.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:55:56.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:56.443923+0000", 2024-08-17T17:55:56.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4220000000000002e-06 2024-08-17T17:55:56.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:56.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:56.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:55:56.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:56.443926+0000", 2024-08-17T17:55:56.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4879999999999999e-06 2024-08-17T17:55:56.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:56.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:56.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:55:56.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:56.444320+0000", 2024-08-17T17:55:56.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039415799999999998 2024-08-17T17:55:56.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:55:56.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:55:56.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:55:56.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:56.444326+0000", 2024-08-17T17:55:56.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5899999999999998e-06 2024-08-17T17:55:56.454 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:56.454 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:55:56.454 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:56.454 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:55:56.454 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-08-17T17:55:56.454 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 2 2024-08-17T17:55:56.454 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T17:55:56.455 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:56.474 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T17:55:56.474 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:55:56.487 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:56.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:56.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:56.509 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T17:55:56.509 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:55:56.543 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T17:55:56.543 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:55:56.589 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:56.631 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:56.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:56.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:56.643 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:56 smithi136 ceph-mon[109861]: pgmap v2192: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 97 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 728 op/s; 1772/201372 objects degraded (0.880%); 1054/201372 objects misplaced (0.523%); 2.5 MiB/s, 46 objects/s recovering 2024-08-17T17:55:56.644 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:56 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3864932249' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:55:56.644 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:56 smithi136 ceph-mon[115078]: pgmap v2192: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 97 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 728 op/s; 1772/201372 objects degraded (0.880%); 1054/201372 objects misplaced (0.523%); 2.5 MiB/s, 46 objects/s recovering 2024-08-17T17:55:56.644 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:56 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3864932249' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:55:56.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:56 smithi165 ceph-mon[105517]: pgmap v2192: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 97 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 728 op/s; 1772/201372 objects degraded (0.880%); 1054/201372 objects misplaced (0.523%); 2.5 MiB/s, 46 objects/s recovering 2024-08-17T17:55:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:56 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3864932249' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:55:56.765 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:56.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:56.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:56.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:55:56.897 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:56.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:56.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:55:56.999 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:57.026 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:57.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:57.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:57.128 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:57.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:57.157 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:57.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:57.247 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-17T17:55:57.258 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:57.314 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:57.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:57.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:57.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:57.442 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:57.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:57.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:57.544 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:57.581 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:57.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:57.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:57.684 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:57.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:57 smithi136 ceph-mon[109861]: osdmap e1341: 8 total, 8 up, 8 in 2024-08-17T17:55:57.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:57 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-08-17T17:55:57.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:55:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:55:57] "GET /metrics HTTP/1.1" 200 36564 "" "Prometheus/2.51.0" 2024-08-17T17:55:57.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:57 smithi136 ceph-mon[115078]: osdmap e1341: 8 total, 8 up, 8 in 2024-08-17T17:55:57.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:57 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-08-17T17:55:57.715 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:57.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:57.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:57.817 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:57.846 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:57.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:57.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:57.879 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:57 smithi165 ceph-mon[105517]: osdmap e1341: 8 total, 8 up, 8 in 2024-08-17T17:55:57.879 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:57 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-08-17T17:55:57.948 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:57.981 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:57.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:57.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:58.083 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:58.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:58.120 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:58.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:58.175 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:55:58.221 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:58.246 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:58.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:58.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:58.347 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:58.373 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:58.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:58.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:58.475 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:58.503 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:58.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:58.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:58.521 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.7 to 1 (10000) 2024-08-17T17:55:58.524 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:55:58.606 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:58.629 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:58.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:58.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:58.730 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:58.752 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:58.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:58.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:58 smithi165 ceph-mon[105517]: pgmap v2194: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 97 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 835 op/s; 1641/193272 objects misplaced (0.849%); 3.9 MiB/s, 109 objects/s recovering 2024-08-17T17:55:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:58 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "21"}]': finished 2024-08-17T17:55:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:58 smithi165 ceph-mon[105517]: osdmap e1342: 8 total, 8 up, 8 in 2024-08-17T17:55:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:58 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3229481809' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:55:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:58 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:55:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:58 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-17T17:55:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:58 smithi165 ceph-mon[105517]: osdmap e1343: 8 total, 8 up, 8 in 2024-08-17T17:55:58.854 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:55:58.877 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:58.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:58.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:55:58.886 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:58 smithi136 ceph-mon[109861]: pgmap v2194: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 97 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 835 op/s; 1641/193272 objects misplaced (0.849%); 3.9 MiB/s, 109 objects/s recovering 2024-08-17T17:55:58.887 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:58 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "21"}]': finished 2024-08-17T17:55:58.887 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:58 smithi136 ceph-mon[109861]: osdmap e1342: 8 total, 8 up, 8 in 2024-08-17T17:55:58.887 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:58 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3229481809' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:55:58.887 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:58 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:55:58.887 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:58 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-17T17:55:58.887 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:58 smithi136 ceph-mon[109861]: osdmap e1343: 8 total, 8 up, 8 in 2024-08-17T17:55:58.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:58 smithi136 ceph-mon[115078]: pgmap v2194: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 97 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 835 op/s; 1641/193272 objects misplaced (0.849%); 3.9 MiB/s, 109 objects/s recovering 2024-08-17T17:55:58.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:58 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "21"}]': finished 2024-08-17T17:55:58.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:58 smithi136 ceph-mon[115078]: osdmap e1342: 8 total, 8 up, 8 in 2024-08-17T17:55:58.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:58 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3229481809' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:55:58.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:58 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:55:58.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:58 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-17T17:55:58.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:58 smithi136 ceph-mon[115078]: osdmap e1343: 8 total, 8 up, 8 in 2024-08-17T17:55:58.978 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:55:59.002 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:59.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:59.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:55:59.104 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:59.127 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:59.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:59.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:59.229 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:59.251 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:59.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:59.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:59.353 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:59.377 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:55:59.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:59.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:55:59.479 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:55:59.501 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:59.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:59.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:55:59.558 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:59 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-08-17T17:55:59.558 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:59 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:59.558 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:59 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "22"}]': finished 2024-08-17T17:55:59.559 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:55:59 smithi136 ceph-mon[109861]: osdmap e1344: 8 total, 8 up, 8 in 2024-08-17T17:55:59.559 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:59 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-08-17T17:55:59.559 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:59 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:59.559 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:59 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "22"}]': finished 2024-08-17T17:55:59.559 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:55:59 smithi136 ceph-mon[115078]: osdmap e1344: 8 total, 8 up, 8 in 2024-08-17T17:55:59.602 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:55:59.624 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:59.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:59.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:55:59.651 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-08-17T17:55:59.651 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:55:59.726 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:55:59.748 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:59.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:59.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:55:59.850 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:55:59.874 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:55:59.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:55:59.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:55:59.976 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:55:59.995 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:59 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-08-17T17:55:59.996 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:59 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:55:59.996 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:59 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "22"}]': finished 2024-08-17T17:55:59.996 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:55:59 smithi165 ceph-mon[105517]: osdmap e1344: 8 total, 8 up, 8 in 2024-08-17T17:56:00.009 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:00.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:00.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:00.110 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:00.136 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:00.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:00.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:00.238 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:00.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:00.275 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:00.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:00.376 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:00.397 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:00.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:00.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:00.499 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:00.506 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:56:00.521 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:00.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:00.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:00.622 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:00.644 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:00.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:00.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:00.745 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:00.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:00 smithi165 ceph-mon[105517]: pgmap v2197: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 97 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 1641/187635 objects misplaced (0.875%); 1.8 MiB/s, 84 objects/s recovering 2024-08-17T17:56:00.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:00 smithi165 ceph-mon[105517]: osdmap e1345: 8 total, 8 up, 8 in 2024-08-17T17:56:00.766 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:00.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:00.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:00.864 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:00 smithi136 ceph-mon[115078]: pgmap v2197: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 97 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 1641/187635 objects misplaced (0.875%); 1.8 MiB/s, 84 objects/s recovering 2024-08-17T17:56:00.864 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:00 smithi136 ceph-mon[115078]: osdmap e1345: 8 total, 8 up, 8 in 2024-08-17T17:56:00.864 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:00 smithi136 ceph-mon[109861]: pgmap v2197: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 97 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 1641/187635 objects misplaced (0.875%); 1.8 MiB/s, 84 objects/s recovering 2024-08-17T17:56:00.864 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:00 smithi136 ceph-mon[109861]: osdmap e1345: 8 total, 8 up, 8 in 2024-08-17T17:56:00.868 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:00.883 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:56:00.886 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T17:56:00.886 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:56:00.889 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:00.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:00.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:00.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:01.011 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:01.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:01.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:01.113 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:01.135 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:01.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:01.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:01.237 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:01.259 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:01.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:01.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:01.361 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:01.382 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:01.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:01.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:01.484 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:01.506 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:01.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:01.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:01.522 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T17:56:01.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:01.634 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:01.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:01.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:01.672 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:01 smithi136 ceph-mon[109861]: 53.10 scrub starts 2024-08-17T17:56:01.673 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:01 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-08-17T17:56:01.673 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:01 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1806906614' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:56:01.673 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:01 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:56:01.673 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:01 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:56:01.673 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:01 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:56:01.673 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:01 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:56:01.673 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:01 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "23"}]': finished 2024-08-17T17:56:01.673 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:01 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:56:01.673 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:01 smithi136 ceph-mon[109861]: osdmap e1346: 8 total, 8 up, 8 in 2024-08-17T17:56:01.674 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:01 smithi136 ceph-mon[115078]: 53.10 scrub starts 2024-08-17T17:56:01.674 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:01 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-08-17T17:56:01.674 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:01 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1806906614' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:56:01.674 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:01 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:56:01.674 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:01 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:56:01.674 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:01 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:56:01.674 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:01 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:56:01.674 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:01 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "23"}]': finished 2024-08-17T17:56:01.674 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:01 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:56:01.674 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:01 smithi136 ceph-mon[115078]: osdmap e1346: 8 total, 8 up, 8 in 2024-08-17T17:56:01.736 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:01.762 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:01.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:01.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:01.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:01.879 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:01 smithi165 ceph-mon[105517]: 53.10 scrub starts 2024-08-17T17:56:01.879 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:01 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-08-17T17:56:01.879 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:01 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1806906614' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:56:01.879 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:01 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:56:01.879 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:01 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:56:01.879 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:01 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:56:01.879 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:01 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:56:01.879 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:01 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "23"}]': finished 2024-08-17T17:56:01.879 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:01 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:56:01.879 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:01 smithi165 ceph-mon[105517]: osdmap e1346: 8 total, 8 up, 8 in 2024-08-17T17:56:01.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:01.896 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:01.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:01.922 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-17T17:56:01.997 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:02.033 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:02.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:02.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:02.135 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:02.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:02.171 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:02.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:02.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:02.276 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:56:02.301 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:02.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:02.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:02.402 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:02.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:02.430 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:02.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:02.531 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:02.559 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:02.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:02.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:02.661 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:02.684 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:02.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:02.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:02.728 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:02 smithi165 ceph-mon[105517]: pgmap v2200: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 97 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 541 op/s; 1641/186777 objects misplaced (0.879%) 2024-08-17T17:56:02.728 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:02 smithi165 ceph-mon[105517]: osdmap e1347: 8 total, 8 up, 8 in 2024-08-17T17:56:02.786 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:02.809 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:02.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:02.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:02.911 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:02.936 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:02.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:02.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:02.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:02 smithi136 ceph-mon[109861]: pgmap v2200: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 97 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 541 op/s; 1641/186777 objects misplaced (0.879%) 2024-08-17T17:56:02.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:02 smithi136 ceph-mon[109861]: osdmap e1347: 8 total, 8 up, 8 in 2024-08-17T17:56:02.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:02 smithi136 ceph-mon[115078]: pgmap v2200: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 97 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 541 op/s; 1641/186777 objects misplaced (0.879%) 2024-08-17T17:56:02.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:02 smithi136 ceph-mon[115078]: osdmap e1347: 8 total, 8 up, 8 in 2024-08-17T17:56:03.037 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:03.058 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:03.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:03.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:03.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:03.181 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:03.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:03.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:03.365 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:03.388 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:03.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:03.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:03.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:03.512 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:03.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:03.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:03.613 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:03.640 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:03.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:03.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:03.741 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:03.765 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:03.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:03.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:03.867 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:03.904 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:03.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:03.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:04.005 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:04.046 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:04.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:04.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:04.113 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T17:56:04.114 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:56:04.147 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:04.185 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:04.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:04.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:04.287 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:04.327 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:04.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:04.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:04.429 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:04.454 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:04.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:04.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:04.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:04.598 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:04.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:04.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:04.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:04.737 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:04.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:04.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:04.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:04 smithi136 ceph-mon[109861]: pgmap v2203: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 97 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 425 op/s; 3672/182526 objects misplaced (2.012%); 9.7 MiB/s, 154 objects/s recovering 2024-08-17T17:56:04.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:04 smithi136 ceph-mon[109861]: osdmap e1348: 8 total, 8 up, 8 in 2024-08-17T17:56:04.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:04 smithi136 ceph-mon[109861]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2 pgs undersized) 2024-08-17T17:56:04.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:04 smithi136 ceph-mon[115078]: pgmap v2203: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 97 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 425 op/s; 3672/182526 objects misplaced (2.012%); 9.7 MiB/s, 154 objects/s recovering 2024-08-17T17:56:04.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:04 smithi136 ceph-mon[115078]: osdmap e1348: 8 total, 8 up, 8 in 2024-08-17T17:56:04.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:04 smithi136 ceph-mon[115078]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2 pgs undersized) 2024-08-17T17:56:04.840 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:04.876 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:04.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:04.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:04.978 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:04 smithi165 ceph-mon[105517]: pgmap v2203: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 97 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 425 op/s; 3672/182526 objects misplaced (2.012%); 9.7 MiB/s, 154 objects/s recovering 2024-08-17T17:56:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:04 smithi165 ceph-mon[105517]: osdmap e1348: 8 total, 8 up, 8 in 2024-08-17T17:56:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:04 smithi165 ceph-mon[105517]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2 pgs undersized) 2024-08-17T17:56:05.023 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:05.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:05.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:05.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:05.161 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:05.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:05.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:05.263 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:05.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:05.301 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:05.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:05.402 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:05.430 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:56:05.456 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-17T17:56:05.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:05.470 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:05.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:05.571 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:05.604 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:05.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:05.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:05.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:05.750 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:05.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:05.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:05.809 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:56:05.852 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:05.887 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:05.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:05.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:05.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:06.025 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:06.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:06.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:06.127 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:06.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:06.186 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:06.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:06.287 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:06.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:06.314 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:06.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:06.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:06.443 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:06.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:06.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:06.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:06.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:06.584 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:06.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:06.685 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:06.721 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:06.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:06.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:06.781 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:06 smithi136 ceph-mon[109861]: pgmap v2205: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 97 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 412 op/s; 3672/182526 objects misplaced (2.012%); 9.4 MiB/s, 149 objects/s recovering 2024-08-17T17:56:06.781 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:56:06.781 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:56:06.781 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:06 smithi136 ceph-mon[115078]: pgmap v2205: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 97 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 412 op/s; 3672/182526 objects misplaced (2.012%); 9.4 MiB/s, 149 objects/s recovering 2024-08-17T17:56:06.782 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:56:06.782 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:56:06.823 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:06.852 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:06.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:06.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:06.863 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:56:06.954 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:07.000 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:07.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:07.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:06 smithi165 ceph-mon[105517]: pgmap v2205: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 97 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 412 op/s; 3672/182526 objects misplaced (2.012%); 9.4 MiB/s, 149 objects/s recovering 2024-08-17T17:56:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:56:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:56:07.102 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:07.136 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:07.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:07.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:07.238 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:07.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:07.268 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:07.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:07.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:07.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:07.428 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:07.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:07.529 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:07.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:07.569 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:07.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:07.670 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:07.683 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:07 smithi165 ceph-mon[105517]: Health check failed: Degraded data redundancy: 7758/178566 objects degraded (4.345%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:07.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:07 smithi136 ceph-mon[109861]: Health check failed: Degraded data redundancy: 7758/178566 objects degraded (4.345%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:07.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:07 smithi136 ceph-mon[115078]: Health check failed: Degraded data redundancy: 7758/178566 objects degraded (4.345%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:07.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:56:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:56:07] "GET /metrics HTTP/1.1" 200 36563 "" "Prometheus/2.51.0" 2024-08-17T17:56:07.713 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:07.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:07.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:07.719 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T17:56:07.719 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:56:07.815 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:07.849 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:07.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:07.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:07.951 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:07.992 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:07.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:07.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:08.094 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:08.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:08.129 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:08.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:08.230 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:08.256 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:08.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:08.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:08.357 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:08.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-17T17:56:08.419 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:08.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:08.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:08.521 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:08.551 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:08.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:08.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:08.653 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:08.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:08.691 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:08.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:08.725 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:56:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:08 smithi165 ceph-mon[105517]: pgmap v2206: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 456 op/s; 7758/178566 objects degraded (4.345%); 10644/178566 objects misplaced (5.961%); 12 MiB/s, 186 objects/s recovering 2024-08-17T17:56:08.792 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:08.842 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:08.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:08.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:08.848 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:08 smithi136 ceph-mon[109861]: pgmap v2206: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 456 op/s; 7758/178566 objects degraded (4.345%); 10644/178566 objects misplaced (5.961%); 12 MiB/s, 186 objects/s recovering 2024-08-17T17:56:08.849 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:08 smithi136 ceph-mon[115078]: pgmap v2206: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 456 op/s; 7758/178566 objects degraded (4.345%); 10644/178566 objects misplaced (5.961%); 12 MiB/s, 186 objects/s recovering 2024-08-17T17:56:08.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:09.004 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:09.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:09.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:09.106 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:09.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:09.160 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:09.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:09.261 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:09.290 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:09.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:09.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:09.392 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:09.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:09.425 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:09.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:09.426 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:56:09.432 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T17:56:09.526 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:09.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:09.574 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:09.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:09.675 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:09.708 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:09.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:09.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:09.723 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:56:09.809 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:09.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:09.881 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:09.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:09.982 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:10.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:10.016 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:10.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:10.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:10.146 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:10.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:10.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:10.248 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:10.273 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:10.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:10.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:10.375 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:10.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:10.417 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:10.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:10.518 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:10.561 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:10.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:10.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:10.663 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:10.667 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":2207,"stamp":"2024-08-17T17:56:08.650089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3818821629,"num_objects":58289,"num_object_clones":0,"num_object_copies":174867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7758,"num_objects_misplaced":10644,"num_objects_unfound":0,"num_objects_dirty":58289,"num_whiteouts":0,"num_read":379,"num_read_kb":823,"num_write":152630,"num_write_kb":5528946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9684,"num_bytes_recovered":610993760,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205754,"ondisk_log_size":205754,"up":297,"acting":294,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":307,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":13576464,"kb_used_data":11839940,"kb_used_omap":295,"kb_used_meta":1735960,"kb_avail":736384752,"statfs":{"total":767960285184,"available":754057986048,"internally_reserved":0,"allocated":12124098560,"data_stored":12091272103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":302302,"internal_metadata":1777623842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-203358208,"num_objects":-3103,"num_object_clones":0,"num_object_copies":-9309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7758,"num_objects_misplaced":5331,"num_objects_unfound":0,"num_objects_dirty":-3103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3103,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1046,"num_bytes_recovered":68550656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.127574"},"pg_stats":[{"pgid":"53.19","version":"1348'7640","reported_seq":7784,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:06.114295+0000","last_change":"2024-08-17T17:53:49.813139+0000","last_active":"2024-08-17T17:56:06.114295+0000","last_peered":"2024-08-17T17:56:06.114295+0000","last_clean":"2024-08-17T17:56:06.114295+0000","last_became_active":"2024-08-17T17:53:45.801709+0000","last_became_peered":"2024-08-17T17:53:45.801709+0000","last_unstale":"2024-08-17T17:56:06.114295+0000","last_undegraded":"2024-08-17T17:56:06.114295+0000","last_fullsized":"2024-08-17T17:56:06.114295+0000","mapping_epoch":1310,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":5,"last_scrub":"1311'3869","last_scrub_stamp":"2024-08-17T17:53:49.812836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:49.812836+0000","objects_scrubbed":1929,"log_size":7640,"log_dups_size":0,"ondisk_log_size":7640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:36.206484+0000","scrub_duration":554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117833728,"num_objects":1798,"num_object_clones":0,"num_object_copies":5394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3781,"num_write_kb":116800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.18","version":"1348'7971","reported_seq":8089,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:07.047822+0000","last_change":"2024-08-17T17:53:53.987643+0000","last_active":"2024-08-17T17:56:07.047822+0000","last_peered":"2024-08-17T17:56:07.047822+0000","last_clean":"2024-08-17T17:56:07.047822+0000","last_became_active":"2024-08-17T17:53:45.788000+0000","last_became_peered":"2024-08-17T17:53:45.788000+0000","last_unstale":"2024-08-17T17:56:07.047822+0000","last_undegraded":"2024-08-17T17:56:07.047822+0000","last_fullsized":"2024-08-17T17:56:07.047822+0000","mapping_epoch":1310,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":5,"last_scrub":"1313'4093","last_scrub_stamp":"2024-08-17T17:53:53.987508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:53.987508+0000","objects_scrubbed":2233,"log_size":7971,"log_dups_size":0,"ondisk_log_size":7971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:14:47.493287+0000","scrub_duration":0.21921206200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116850688,"num_objects":1783,"num_object_clones":0,"num_object_copies":5349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3878,"num_write_kb":109696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"53.17","version":"1348'7946","reported_seq":8092,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:05.755068+0000","last_change":"2024-08-17T17:53:50.287990+0000","last_active":"2024-08-17T17:56:05.755068+0000","last_peered":"2024-08-17T17:56:05.755068+0000","last_clean":"2024-08-17T17:56:05.755068+0000","last_became_active":"2024-08-17T17:53:45.802369+0000","last_became_peered":"2024-08-17T17:53:45.802369+0000","last_unstale":"2024-08-17T17:56:05.755068+0000","last_undegraded":"2024-08-17T17:56:05.755068+0000","last_fullsized":"2024-08-17T17:56:05.755068+0000","mapping_epoch":1310,"log_start":"1306'159","ondisk_log_start":"1306'159","created":1310,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":5,"last_scrub":"1312'3969","last_scrub_stamp":"2024-08-17T17:53:50.287885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:50.287885+0000","objects_scrubbed":2090,"log_size":7787,"log_dups_size":0,"ondisk_log_size":7787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:29:18.190184+0000","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124125184,"num_objects":1894,"num_object_clones":0,"num_object_copies":5682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3990,"num_write_kb":121408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.16","version":"1348'7941","reported_seq":8365,"reported_epoch":1348,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:56:05.992318+0000","last_change":"2024-08-17T17:56:03.772184+0000","last_active":"2024-08-17T17:56:05.992318+0000","last_peered":"2024-08-17T17:56:05.992318+0000","last_clean":"2024-08-17T17:56:00.271068+0000","last_became_active":"2024-08-17T17:56:03.726122+0000","last_became_peered":"2024-08-17T17:56:03.726122+0000","last_unstale":"2024-08-17T17:56:05.992318+0000","last_undegraded":"2024-08-17T17:56:03.680764+0000","last_fullsized":"2024-08-17T17:56:03.528461+0000","mapping_epoch":1347,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":5,"last_scrub":"1311'3927","last_scrub_stamp":"2024-08-17T17:53:48.318097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:48.318097+0000","objects_scrubbed":2014,"log_size":7941,"log_dups_size":0,"ondisk_log_size":7941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121831424,"num_objects":1859,"num_object_clones":0,"num_object_copies":5577,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7758,"num_objects_misplaced":3879,"num_objects_unfound":0,"num_objects_dirty":1859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":123776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":4521984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[2,1],"avail_no_missing":["2","3","4"],"object_location_counts":[{"shards":"1,2","objects":1},{"shards":"2,3,4","objects":1858}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"53.15","version":"1348'7734","reported_seq":8423,"reported_epoch":1348,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-17T17:56:06.217174+0000","last_change":"2024-08-17T17:56:01.744916+0000","last_active":"2024-08-17T17:56:06.217174+0000","last_peered":"2024-08-17T17:56:06.217174+0000","last_clean":"2024-08-17T17:55:58.529149+0000","last_became_active":"2024-08-17T17:56:01.715968+0000","last_became_peered":"2024-08-17T17:56:01.715968+0000","last_unstale":"2024-08-17T17:56:06.217174+0000","last_undegraded":"2024-08-17T17:56:06.217174+0000","last_fullsized":"2024-08-17T17:56:01.526888+0000","mapping_epoch":1345,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":5,"last_scrub":"1311'3700","last_scrub_stamp":"2024-08-17T17:53:47.592482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:47.592482+0000","objects_scrubbed":1931,"log_size":7734,"log_dups_size":0,"ondisk_log_size":7734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123011072,"num_objects":1877,"num_object_clones":0,"num_object_copies":5631,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3591,"num_objects_unfound":0,"num_objects_dirty":1877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4048,"num_write_kb":127808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":11862016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[0,4],"avail_no_missing":["0","1","3"],"object_location_counts":[{"shards":"0,1,3","objects":1795},{"shards":"0,1,3,4","objects":1},{"shards":"0,4","objects":81}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.14","version":"1348'7701","reported_seq":8806,"reported_epoch":1348,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-17T17:56:05.847522+0000","last_change":"2024-08-17T17:55:59.811423+0000","last_active":"2024-08-17T17:56:05.847522+0000","last_peered":"2024-08-17T17:56:05.847522+0000","last_clean":"2024-08-17T17:55:57.307497+0000","last_became_active":"2024-08-17T17:55:59.771920+0000","last_became_peered":"2024-08-17T17:55:59.771920+0000","last_unstale":"2024-08-17T17:56:05.847522+0000","last_undegraded":"2024-08-17T17:56:05.847522+0000","last_fullsized":"2024-08-17T17:55:59.520008+0000","mapping_epoch":1343,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":5,"last_scrub":"1311'3909","last_scrub_stamp":"2024-08-17T17:53:49.696328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:49.696328+0000","objects_scrubbed":1954,"log_size":7701,"log_dups_size":0,"ondisk_log_size":7701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118292480,"num_objects":1805,"num_object_clones":0,"num_object_copies":5415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3174,"num_objects_unfound":0,"num_objects_dirty":1805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3811,"num_write_kb":117184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":320,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[1,3],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":1587},{"shards":"1,3","objects":218}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.13","version":"1348'7544","reported_seq":17242,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:06.161528+0000","last_change":"2024-08-17T17:55:38.393304+0000","last_active":"2024-08-17T17:56:06.161528+0000","last_peered":"2024-08-17T17:56:06.161528+0000","last_clean":"2024-08-17T17:56:06.161528+0000","last_became_active":"2024-08-17T17:55:38.383298+0000","last_became_peered":"2024-08-17T17:55:38.383298+0000","last_unstale":"2024-08-17T17:56:06.161528+0000","last_undegraded":"2024-08-17T17:56:06.161528+0000","last_fullsized":"2024-08-17T17:56:06.161528+0000","mapping_epoch":1337,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1338,"parent":"0.0","parent_split_bits":5,"last_scrub":"1311'3687","last_scrub_stamp":"2024-08-17T17:53:48.035372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:48.035372+0000","objects_scrubbed":1888,"log_size":7544,"log_dups_size":0,"ondisk_log_size":7544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:04:23.248001+0000","scrub_duration":0.55651981800000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117374976,"num_objects":1791,"num_object_clones":0,"num_object_copies":5373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3869,"num_write_kb":120704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2386,"num_bytes_recovered":156368896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.12","version":"1348'7769","reported_seq":17699,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:06.746980+0000","last_change":"2024-08-17T17:56:01.548247+0000","last_active":"2024-08-17T17:56:06.746980+0000","last_peered":"2024-08-17T17:56:06.746980+0000","last_clean":"2024-08-17T17:56:06.746980+0000","last_became_active":"2024-08-17T17:56:01.535252+0000","last_became_peered":"2024-08-17T17:56:01.535252+0000","last_unstale":"2024-08-17T17:56:06.746980+0000","last_undegraded":"2024-08-17T17:56:06.746980+0000","last_fullsized":"2024-08-17T17:56:06.746980+0000","mapping_epoch":1345,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1346,"parent":"0.0","parent_split_bits":5,"last_scrub":"1312'4010","last_scrub_stamp":"2024-08-17T17:53:52.788196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:52.788196+0000","objects_scrubbed":2082,"log_size":7769,"log_dups_size":0,"ondisk_log_size":7769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:06:14.839811+0000","scrub_duration":1.1488786010000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117178368,"num_objects":1788,"num_object_clones":0,"num_object_copies":5364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3780,"num_write_kb":111552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2443,"num_bytes_recovered":160104448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.11","version":"1348'7801","reported_seq":14310,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:06.149323+0000","last_change":"2024-08-17T17:54:32.250770+0000","last_active":"2024-08-17T17:56:06.149323+0000","last_peered":"2024-08-17T17:56:06.149323+0000","last_clean":"2024-08-17T17:56:06.149323+0000","last_became_active":"2024-08-17T17:54:32.246225+0000","last_became_peered":"2024-08-17T17:54:32.246225+0000","last_unstale":"2024-08-17T17:56:06.149323+0000","last_undegraded":"2024-08-17T17:56:06.149323+0000","last_fullsized":"2024-08-17T17:56:06.149323+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"1312'3987","last_scrub_stamp":"2024-08-17T17:53:51.667290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:51.667290+0000","objects_scrubbed":2071,"log_size":7801,"log_dups_size":0,"ondisk_log_size":7801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:59:10.324504+0000","scrub_duration":708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118751232,"num_objects":1812,"num_object_clones":0,"num_object_copies":5436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3831,"num_write_kb":114304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":138805248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.10","version":"1348'7792","reported_seq":14399,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:06.057596+0000","last_change":"2024-08-17T17:56:02.524184+0000","last_active":"2024-08-17T17:56:06.057596+0000","last_peered":"2024-08-17T17:56:06.057596+0000","last_clean":"2024-08-17T17:56:06.057596+0000","last_became_active":"2024-08-17T17:55:57.520262+0000","last_became_peered":"2024-08-17T17:55:57.520262+0000","last_unstale":"2024-08-17T17:56:06.057596+0000","last_undegraded":"2024-08-17T17:56:06.057596+0000","last_fullsized":"2024-08-17T17:56:06.057596+0000","mapping_epoch":1341,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":7792,"log_dups_size":0,"ondisk_log_size":7792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T17:56:07.493977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116686848,"num_objects":1781,"num_object_clones":0,"num_object_copies":5343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5937,"num_write_kb":246944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2161,"num_bytes_recovered":116588544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.f","version":"1348'12285","reported_seq":12340,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:06.965311+0000","last_change":"2024-08-17T17:53:10.489536+0000","last_active":"2024-08-17T17:56:06.965311+0000","last_peered":"2024-08-17T17:56:06.965311+0000","last_clean":"2024-08-17T17:56:06.965311+0000","last_became_active":"2024-08-17T17:53:10.489173+0000","last_became_peered":"2024-08-17T17:53:10.489173+0000","last_unstale":"2024-08-17T17:56:06.965311+0000","last_undegraded":"2024-08-17T17:56:06.965311+0000","last_fullsized":"2024-08-17T17:56:06.965311+0000","mapping_epoch":1303,"log_start":"1309'3657","ondisk_log_start":"1309'3657","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":8628,"log_dups_size":0,"ondisk_log_size":8628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:46:03.457908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242417664,"num_objects":3699,"num_object_clones":0,"num_object_copies":11097,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12285,"num_write_kb":511488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.e","version":"1348'11808","reported_seq":11863,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:06.910150+0000","last_change":"2024-08-17T17:53:10.488446+0000","last_active":"2024-08-17T17:56:06.910150+0000","last_peered":"2024-08-17T17:56:06.910150+0000","last_clean":"2024-08-17T17:56:06.910150+0000","last_became_active":"2024-08-17T17:53:10.488058+0000","last_became_peered":"2024-08-17T17:53:10.488058+0000","last_unstale":"2024-08-17T17:56:06.910150+0000","last_undegraded":"2024-08-17T17:56:06.910150+0000","last_fullsized":"2024-08-17T17:56:06.910150+0000","mapping_epoch":1303,"log_start":"1308'3208","ondisk_log_start":"1308'3208","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":8600,"log_dups_size":0,"ondisk_log_size":8600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:00:57.616106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240779264,"num_objects":3674,"num_object_clones":0,"num_object_copies":11022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11808,"num_write_kb":495424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.d","version":"1348'12125","reported_seq":12237,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:05.844415+0000","last_change":"2024-08-17T17:53:10.646982+0000","last_active":"2024-08-17T17:56:05.844415+0000","last_peered":"2024-08-17T17:56:05.844415+0000","last_clean":"2024-08-17T17:56:05.844415+0000","last_became_active":"2024-08-17T17:53:10.646591+0000","last_became_peered":"2024-08-17T17:53:10.646591+0000","last_unstale":"2024-08-17T17:56:05.844415+0000","last_undegraded":"2024-08-17T17:56:05.844415+0000","last_fullsized":"2024-08-17T17:56:05.844415+0000","mapping_epoch":1303,"log_start":"1326'6073","ondisk_log_start":"1326'6073","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":6052,"log_dups_size":0,"ondisk_log_size":6052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:06:34.821249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239992856,"num_objects":3663,"num_object_clones":0,"num_object_copies":10989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3663,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12125,"num_write_kb":505153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.c","version":"1348'11832","reported_seq":11884,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:07.102736+0000","last_change":"2024-08-17T17:53:10.559850+0000","last_active":"2024-08-17T17:56:07.102736+0000","last_peered":"2024-08-17T17:56:07.102736+0000","last_clean":"2024-08-17T17:56:07.102736+0000","last_became_active":"2024-08-17T17:53:10.559588+0000","last_became_peered":"2024-08-17T17:53:10.559588+0000","last_unstale":"2024-08-17T17:56:07.102736+0000","last_undegraded":"2024-08-17T17:56:07.102736+0000","last_fullsized":"2024-08-17T17:56:07.102736+0000","mapping_epoch":1303,"log_start":"1307'2100","ondisk_log_start":"1307'2100","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":9732,"log_dups_size":0,"ondisk_log_size":9732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:16:20.553763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241827840,"num_objects":3690,"num_object_clones":0,"num_object_copies":11070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11832,"num_write_kb":496704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"53.b","version":"1348'11839","reported_seq":11894,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:07.076374+0000","last_change":"2024-08-17T17:53:10.490102+0000","last_active":"2024-08-17T17:56:07.076374+0000","last_peered":"2024-08-17T17:56:07.076374+0000","last_clean":"2024-08-17T17:56:07.076374+0000","last_became_active":"2024-08-17T17:53:10.489717+0000","last_became_peered":"2024-08-17T17:53:10.489717+0000","last_unstale":"2024-08-17T17:56:07.076374+0000","last_undegraded":"2024-08-17T17:56:07.076374+0000","last_fullsized":"2024-08-17T17:56:07.076374+0000","mapping_epoch":1303,"log_start":"1307'2100","ondisk_log_start":"1307'2100","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":9739,"log_dups_size":0,"ondisk_log_size":9739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:50:43.448878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234815488,"num_objects":3583,"num_object_clones":0,"num_object_copies":10749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11839,"num_write_kb":493504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"53.a","version":"1348'11954","reported_seq":12065,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:05.556885+0000","last_change":"2024-08-17T17:53:10.489596+0000","last_active":"2024-08-17T17:56:05.556885+0000","last_peered":"2024-08-17T17:56:05.556885+0000","last_clean":"2024-08-17T17:56:05.556885+0000","last_became_active":"2024-08-17T17:53:10.489230+0000","last_became_peered":"2024-08-17T17:53:10.489230+0000","last_unstale":"2024-08-17T17:56:05.556885+0000","last_undegraded":"2024-08-17T17:56:05.556885+0000","last_fullsized":"2024-08-17T17:56:05.556885+0000","mapping_epoch":1303,"log_start":"1318'4558","ondisk_log_start":"1318'4558","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":7396,"log_dups_size":0,"ondisk_log_size":7396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:50.891777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242352128,"num_objects":3698,"num_object_clones":0,"num_object_copies":11094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":500864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"53.9","version":"1348'7885","reported_seq":8031,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:05.945793+0000","last_change":"2024-08-17T17:53:48.914243+0000","last_active":"2024-08-17T17:56:05.945793+0000","last_peered":"2024-08-17T17:56:05.945793+0000","last_clean":"2024-08-17T17:56:05.945793+0000","last_became_active":"2024-08-17T17:53:45.787626+0000","last_became_peered":"2024-08-17T17:53:45.787626+0000","last_unstale":"2024-08-17T17:56:05.945793+0000","last_undegraded":"2024-08-17T17:56:05.945793+0000","last_fullsized":"2024-08-17T17:56:05.945793+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3875","last_scrub_stamp":"2024-08-17T17:53:48.913906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:48.913906+0000","objects_scrubbed":2075,"log_size":7885,"log_dups_size":0,"ondisk_log_size":7885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:19:31.071940+0000","scrub_duration":692,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119668736,"num_objects":1826,"num_object_clones":0,"num_object_copies":5478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":120768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.8","version":"1348'7902","reported_seq":8025,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:07.101851+0000","last_change":"2024-08-17T17:53:50.732592+0000","last_active":"2024-08-17T17:56:07.101851+0000","last_peered":"2024-08-17T17:56:07.101851+0000","last_clean":"2024-08-17T17:56:07.101851+0000","last_became_active":"2024-08-17T17:53:45.777386+0000","last_became_peered":"2024-08-17T17:53:45.777386+0000","last_unstale":"2024-08-17T17:56:07.101851+0000","last_undegraded":"2024-08-17T17:56:07.101851+0000","last_fullsized":"2024-08-17T17:56:07.101851+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1312'4030","last_scrub_stamp":"2024-08-17T17:53:50.732148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:50.732148+0000","objects_scrubbed":2054,"log_size":7902,"log_dups_size":0,"ondisk_log_size":7902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:42:36.666939+0000","scrub_duration":0.96559516700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116588544,"num_objects":1779,"num_object_clones":0,"num_object_copies":5337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3889,"num_write_kb":115648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"53.7","version":"1348'7782","reported_seq":7927,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:05.750225+0000","last_change":"2024-08-17T17:53:47.346759+0000","last_active":"2024-08-17T17:56:05.750225+0000","last_peered":"2024-08-17T17:56:05.750225+0000","last_clean":"2024-08-17T17:56:05.750225+0000","last_became_active":"2024-08-17T17:53:45.779914+0000","last_became_peered":"2024-08-17T17:53:45.779914+0000","last_unstale":"2024-08-17T17:56:05.750225+0000","last_undegraded":"2024-08-17T17:56:05.750225+0000","last_fullsized":"2024-08-17T17:56:05.750225+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3847","last_scrub_stamp":"2024-08-17T17:53:47.346625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:47.346625+0000","objects_scrubbed":1924,"log_size":7782,"log_dups_size":0,"ondisk_log_size":7782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:44:59.835054+0000","scrub_duration":430,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118226944,"num_objects":1804,"num_object_clones":0,"num_object_copies":5412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3940,"num_write_kb":122240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.6","version":"1348'7983","reported_seq":8131,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:05.491143+0000","last_change":"2024-08-17T17:53:47.249481+0000","last_active":"2024-08-17T17:56:05.491143+0000","last_peered":"2024-08-17T17:56:05.491143+0000","last_clean":"2024-08-17T17:56:05.491143+0000","last_became_active":"2024-08-17T17:53:45.758286+0000","last_became_peered":"2024-08-17T17:53:45.758286+0000","last_unstale":"2024-08-17T17:56:05.491143+0000","last_undegraded":"2024-08-17T17:56:05.491143+0000","last_fullsized":"2024-08-17T17:56:05.491143+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3922","last_scrub_stamp":"2024-08-17T17:53:47.248989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:47.248989+0000","objects_scrubbed":1980,"log_size":7983,"log_dups_size":0,"ondisk_log_size":7983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:43:54.442535+0000","scrub_duration":555,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123469824,"num_objects":1884,"num_object_clones":0,"num_object_copies":5652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4070,"num_write_kb":127168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"53.5","version":"1348'7524","reported_seq":7667,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:06.172822+0000","last_change":"2024-08-17T17:53:46.758091+0000","last_active":"2024-08-17T17:56:06.172822+0000","last_peered":"2024-08-17T17:56:06.172822+0000","last_clean":"2024-08-17T17:56:06.172822+0000","last_became_active":"2024-08-17T17:53:45.771578+0000","last_became_peered":"2024-08-17T17:53:45.771578+0000","last_unstale":"2024-08-17T17:56:06.172822+0000","last_undegraded":"2024-08-17T17:56:06.172822+0000","last_fullsized":"2024-08-17T17:56:06.172822+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3661","last_scrub_stamp":"2024-08-17T17:53:46.757894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:46.757894+0000","objects_scrubbed":1802,"log_size":7524,"log_dups_size":0,"ondisk_log_size":7524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:27:03.974414+0000","scrub_duration":509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116391936,"num_objects":1776,"num_object_clones":0,"num_object_copies":5328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3868,"num_write_kb":122944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.4","version":"1348'7897","reported_seq":8043,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:05.846286+0000","last_change":"2024-08-17T17:53:46.426863+0000","last_active":"2024-08-17T17:56:05.846286+0000","last_peered":"2024-08-17T17:56:05.846286+0000","last_clean":"2024-08-17T17:56:05.846286+0000","last_became_active":"2024-08-17T17:53:45.764066+0000","last_became_peered":"2024-08-17T17:53:45.764066+0000","last_unstale":"2024-08-17T17:56:05.846286+0000","last_undegraded":"2024-08-17T17:56:05.846286+0000","last_fullsized":"2024-08-17T17:56:05.846286+0000","mapping_epoch":1303,"log_start":"1306'180","ondisk_log_start":"1306'180","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3774","last_scrub_stamp":"2024-08-17T17:53:46.426732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:46.426732+0000","objects_scrubbed":1958,"log_size":7717,"log_dups_size":0,"ondisk_log_size":7717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:49:46.480003+0000","scrub_duration":541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121372672,"num_objects":1852,"num_object_clones":0,"num_object_copies":5556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4138,"num_write_kb":129024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.3","version":"1348'7516","reported_seq":7629,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:06.922321+0000","last_change":"2024-08-17T17:53:46.968605+0000","last_active":"2024-08-17T17:56:06.922321+0000","last_peered":"2024-08-17T17:56:06.922321+0000","last_clean":"2024-08-17T17:56:06.922321+0000","last_became_active":"2024-08-17T17:53:45.764323+0000","last_became_peered":"2024-08-17T17:53:45.764323+0000","last_unstale":"2024-08-17T17:56:06.922321+0000","last_undegraded":"2024-08-17T17:56:06.922321+0000","last_fullsized":"2024-08-17T17:56:06.922321+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3658","last_scrub_stamp":"2024-08-17T17:53:46.967846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:46.967846+0000","objects_scrubbed":1838,"log_size":7516,"log_dups_size":0,"ondisk_log_size":7516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:09:25.913293+0000","scrub_duration":0.48413113800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115212288,"num_objects":1758,"num_object_clones":0,"num_object_copies":5274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3866,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.2","version":"1348'7855","reported_seq":7974,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:07.066949+0000","last_change":"2024-08-17T17:53:48.304744+0000","last_active":"2024-08-17T17:56:07.066949+0000","last_peered":"2024-08-17T17:56:07.066949+0000","last_clean":"2024-08-17T17:56:07.066949+0000","last_became_active":"2024-08-17T17:53:45.780923+0000","last_became_peered":"2024-08-17T17:53:45.780923+0000","last_unstale":"2024-08-17T17:56:07.066949+0000","last_undegraded":"2024-08-17T17:56:07.066949+0000","last_fullsized":"2024-08-17T17:56:07.066949+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3850","last_scrub_stamp":"2024-08-17T17:53:48.304649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:48.304649+0000","objects_scrubbed":2003,"log_size":7855,"log_dups_size":0,"ondisk_log_size":7855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:01.087711+0000","scrub_duration":0.70259554000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117833728,"num_objects":1798,"num_object_clones":0,"num_object_copies":5394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":122176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"53.1","version":"1348'7842","reported_seq":7987,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:05.685627+0000","last_change":"2024-08-17T17:53:53.382389+0000","last_active":"2024-08-17T17:56:05.685627+0000","last_peered":"2024-08-17T17:56:05.685627+0000","last_clean":"2024-08-17T17:56:05.685627+0000","last_became_active":"2024-08-17T17:53:45.782234+0000","last_became_peered":"2024-08-17T17:53:45.782234+0000","last_unstale":"2024-08-17T17:56:05.685627+0000","last_undegraded":"2024-08-17T17:56:05.685627+0000","last_fullsized":"2024-08-17T17:56:05.685627+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1313'4031","last_scrub_stamp":"2024-08-17T17:53:53.382269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:53.382269+0000","objects_scrubbed":2178,"log_size":7842,"log_dups_size":0,"ondisk_log_size":7842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:43.716721+0000","scrub_duration":410,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121307136,"num_objects":1851,"num_object_clones":0,"num_object_copies":5553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3813,"num_write_kb":111552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.0","version":"1348'7713","reported_seq":7829,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:06.998919+0000","last_change":"2024-08-17T17:53:47.447388+0000","last_active":"2024-08-17T17:56:06.998919+0000","last_peered":"2024-08-17T17:56:06.998919+0000","last_clean":"2024-08-17T17:56:06.998919+0000","last_became_active":"2024-08-17T17:53:45.741805+0000","last_became_peered":"2024-08-17T17:53:45.741805+0000","last_unstale":"2024-08-17T17:56:06.998919+0000","last_undegraded":"2024-08-17T17:56:06.998919+0000","last_fullsized":"2024-08-17T17:56:06.998919+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3799","last_scrub_stamp":"2024-08-17T17:53:47.447203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:47.447203+0000","objects_scrubbed":1904,"log_size":7713,"log_dups_size":0,"ondisk_log_size":7713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:52:36.467443+0000","scrub_duration":0.60964741,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114032640,"num_objects":1740,"num_object_clones":0,"num_object_copies":5220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3930,"num_write_kb":120512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":2001,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.549460+0000","last_change":"2024-08-17T17:23:42.937520+0000","last_active":"2024-08-17T17:56:03.549460+0000","last_peered":"2024-08-17T17:56:03.549460+0000","last_clean":"2024-08-17T17:56:03.549460+0000","last_became_active":"2024-08-17T17:23:42.937174+0000","last_became_peered":"2024-08-17T17:23:42.937174+0000","last_unstale":"2024-08-17T17:56:03.549460+0000","last_undegraded":"2024-08-17T17:56:03.549460+0000","last_fullsized":"2024-08-17T17:56:03.549460+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:47:56.679612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1319,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.684220+0000","last_change":"2024-08-17T17:23:07.398676+0000","last_active":"2024-08-17T17:56:03.684220+0000","last_peered":"2024-08-17T17:56:03.684220+0000","last_clean":"2024-08-17T17:56:03.684220+0000","last_became_active":"2024-08-17T17:23:07.398369+0000","last_became_peered":"2024-08-17T17:23:07.398369+0000","last_unstale":"2024-08-17T17:56:03.684220+0000","last_undegraded":"2024-08-17T17:56:03.684220+0000","last_fullsized":"2024-08-17T17:56:03.684220+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:55:04.281412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":1984,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.547283+0000","last_change":"2024-08-17T17:24:17.476168+0000","last_active":"2024-08-17T17:56:03.547283+0000","last_peered":"2024-08-17T17:56:03.547283+0000","last_clean":"2024-08-17T17:56:03.547283+0000","last_became_active":"2024-08-17T17:24:17.465845+0000","last_became_peered":"2024-08-17T17:24:17.465845+0000","last_unstale":"2024-08-17T17:56:03.547283+0000","last_undegraded":"2024-08-17T17:56:03.547283+0000","last_fullsized":"2024-08-17T17:56:03.547283+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:15:01.494447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2001,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.546081+0000","last_change":"2024-08-17T17:24:17.461772+0000","last_active":"2024-08-17T17:56:03.546081+0000","last_peered":"2024-08-17T17:56:03.546081+0000","last_clean":"2024-08-17T17:56:03.546081+0000","last_became_active":"2024-08-17T17:24:17.460806+0000","last_became_peered":"2024-08-17T17:24:17.460806+0000","last_unstale":"2024-08-17T17:56:03.546081+0000","last_undegraded":"2024-08-17T17:56:03.546081+0000","last_fullsized":"2024-08-17T17:56:03.546081+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:01:16.453148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":1301,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.526875+0000","last_change":"2024-08-17T17:10:05.965176+0000","last_active":"2024-08-17T17:56:02.526875+0000","last_peered":"2024-08-17T17:56:02.526875+0000","last_clean":"2024-08-17T17:56:02.526875+0000","last_became_active":"2024-08-17T17:10:05.964861+0000","last_became_peered":"2024-08-17T17:10:05.964861+0000","last_unstale":"2024-08-17T17:56:02.526875+0000","last_undegraded":"2024-08-17T17:56:02.526875+0000","last_fullsized":"2024-08-17T17:56:02.526875+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:59:04.277926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1333,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.524995+0000","last_change":"2024-08-17T17:24:17.462527+0000","last_active":"2024-08-17T17:56:02.524995+0000","last_peered":"2024-08-17T17:56:02.524995+0000","last_clean":"2024-08-17T17:56:02.524995+0000","last_became_active":"2024-08-17T17:24:17.462240+0000","last_became_peered":"2024-08-17T17:24:17.462240+0000","last_unstale":"2024-08-17T17:56:02.524995+0000","last_undegraded":"2024-08-17T17:56:02.524995+0000","last_fullsized":"2024-08-17T17:56:02.524995+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:40:38.063717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":1316,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.527390+0000","last_change":"2024-08-17T17:23:07.391038+0000","last_active":"2024-08-17T17:56:02.527390+0000","last_peered":"2024-08-17T17:56:02.527390+0000","last_clean":"2024-08-17T17:56:02.527390+0000","last_became_active":"2024-08-17T17:23:07.390571+0000","last_became_peered":"2024-08-17T17:23:07.390571+0000","last_unstale":"2024-08-17T17:56:02.527390+0000","last_undegraded":"2024-08-17T17:56:02.527390+0000","last_fullsized":"2024-08-17T17:56:02.527390+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:18:41.791740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1333,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.525102+0000","last_change":"2024-08-17T17:24:17.458168+0000","last_active":"2024-08-17T17:56:02.525102+0000","last_peered":"2024-08-17T17:56:02.525102+0000","last_clean":"2024-08-17T17:56:02.525102+0000","last_became_active":"2024-08-17T17:24:17.457847+0000","last_became_peered":"2024-08-17T17:24:17.457847+0000","last_unstale":"2024-08-17T17:56:02.525102+0000","last_undegraded":"2024-08-17T17:56:02.525102+0000","last_fullsized":"2024-08-17T17:56:02.525102+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:28.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":1983,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.548781+0000","last_change":"2024-08-17T17:22:32.883756+0000","last_active":"2024-08-17T17:56:03.548781+0000","last_peered":"2024-08-17T17:56:03.548781+0000","last_clean":"2024-08-17T17:56:03.548781+0000","last_became_active":"2024-08-17T17:22:32.874867+0000","last_became_peered":"2024-08-17T17:22:32.874867+0000","last_unstale":"2024-08-17T17:56:03.548781+0000","last_undegraded":"2024-08-17T17:56:03.548781+0000","last_fullsized":"2024-08-17T17:56:03.548781+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:52:27.375231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":2002,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.529577+0000","last_change":"2024-08-17T17:24:17.456622+0000","last_active":"2024-08-17T17:56:03.529577+0000","last_peered":"2024-08-17T17:56:03.529577+0000","last_clean":"2024-08-17T17:56:03.529577+0000","last_became_active":"2024-08-17T17:24:17.455396+0000","last_became_peered":"2024-08-17T17:24:17.455396+0000","last_unstale":"2024-08-17T17:56:03.529577+0000","last_undegraded":"2024-08-17T17:56:03.529577+0000","last_fullsized":"2024-08-17T17:56:03.529577+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:09.564331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1335,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.525133+0000","last_change":"2024-08-17T17:23:42.930464+0000","last_active":"2024-08-17T17:56:02.525133+0000","last_peered":"2024-08-17T17:56:02.525133+0000","last_clean":"2024-08-17T17:56:02.525133+0000","last_became_active":"2024-08-17T17:23:42.929949+0000","last_became_peered":"2024-08-17T17:23:42.929949+0000","last_unstale":"2024-08-17T17:56:02.525133+0000","last_undegraded":"2024-08-17T17:56:02.525133+0000","last_fullsized":"2024-08-17T17:56:02.525133+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:48:27.092339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1987,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.546852+0000","last_change":"2024-08-17T17:24:17.452009+0000","last_active":"2024-08-17T17:56:03.546852+0000","last_peered":"2024-08-17T17:56:03.546852+0000","last_clean":"2024-08-17T17:56:03.546852+0000","last_became_active":"2024-08-17T17:24:17.451492+0000","last_became_peered":"2024-08-17T17:24:17.451492+0000","last_unstale":"2024-08-17T17:56:03.546852+0000","last_undegraded":"2024-08-17T17:56:03.546852+0000","last_fullsized":"2024-08-17T17:56:03.546852+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:21:24.699633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1999,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.526077+0000","last_change":"2024-08-17T17:24:17.452025+0000","last_active":"2024-08-17T17:56:03.526077+0000","last_peered":"2024-08-17T17:56:03.526077+0000","last_clean":"2024-08-17T17:56:03.526077+0000","last_became_active":"2024-08-17T17:24:17.448029+0000","last_became_peered":"2024-08-17T17:24:17.448029+0000","last_unstale":"2024-08-17T17:56:03.526077+0000","last_undegraded":"2024-08-17T17:56:03.526077+0000","last_fullsized":"2024-08-17T17:56:03.526077+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:45.089346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1994,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.549222+0000","last_change":"2024-08-17T17:24:17.469096+0000","last_active":"2024-08-17T17:56:03.549222+0000","last_peered":"2024-08-17T17:56:03.549222+0000","last_clean":"2024-08-17T17:56:03.549222+0000","last_became_active":"2024-08-17T17:24:17.468669+0000","last_became_peered":"2024-08-17T17:24:17.468669+0000","last_unstale":"2024-08-17T17:56:03.549222+0000","last_undegraded":"2024-08-17T17:56:03.549222+0000","last_fullsized":"2024-08-17T17:56:03.549222+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:03:39.427341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1988,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.545710+0000","last_change":"2024-08-17T17:24:17.453946+0000","last_active":"2024-08-17T17:56:03.545710+0000","last_peered":"2024-08-17T17:56:03.545710+0000","last_clean":"2024-08-17T17:56:03.545710+0000","last_became_active":"2024-08-17T17:24:17.445004+0000","last_became_peered":"2024-08-17T17:24:17.445004+0000","last_unstale":"2024-08-17T17:56:03.545710+0000","last_undegraded":"2024-08-17T17:56:03.545710+0000","last_fullsized":"2024-08-17T17:56:03.545710+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:00:19.366629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1319,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.531012+0000","last_change":"2024-08-17T17:23:07.402338+0000","last_active":"2024-08-17T17:56:02.531012+0000","last_peered":"2024-08-17T17:56:02.531012+0000","last_clean":"2024-08-17T17:56:02.531012+0000","last_became_active":"2024-08-17T17:23:07.401996+0000","last_became_peered":"2024-08-17T17:23:07.401996+0000","last_unstale":"2024-08-17T17:56:02.531012+0000","last_undegraded":"2024-08-17T17:56:02.531012+0000","last_fullsized":"2024-08-17T17:56:02.531012+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:39:21.566786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1972,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.545933+0000","last_change":"2024-08-17T17:24:17.442338+0000","last_active":"2024-08-17T17:56:03.545933+0000","last_peered":"2024-08-17T17:56:03.545933+0000","last_clean":"2024-08-17T17:56:03.545933+0000","last_became_active":"2024-08-17T17:24:17.441948+0000","last_became_peered":"2024-08-17T17:24:17.441948+0000","last_unstale":"2024-08-17T17:56:03.545933+0000","last_undegraded":"2024-08-17T17:56:03.545933+0000","last_fullsized":"2024-08-17T17:56:03.545933+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:17.891492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1990,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.549599+0000","last_change":"2024-08-17T17:24:17.459083+0000","last_active":"2024-08-17T17:56:03.549599+0000","last_peered":"2024-08-17T17:56:03.549599+0000","last_clean":"2024-08-17T17:56:03.549599+0000","last_became_active":"2024-08-17T17:24:17.458759+0000","last_became_peered":"2024-08-17T17:24:17.458759+0000","last_unstale":"2024-08-17T17:56:03.549599+0000","last_undegraded":"2024-08-17T17:56:03.549599+0000","last_fullsized":"2024-08-17T17:56:03.549599+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:11:59.301864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1931,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.546166+0000","last_change":"2024-08-17T17:24:17.468086+0000","last_active":"2024-08-17T17:56:03.546166+0000","last_peered":"2024-08-17T17:56:03.546166+0000","last_clean":"2024-08-17T17:56:03.546166+0000","last_became_active":"2024-08-17T17:24:17.467522+0000","last_became_peered":"2024-08-17T17:24:17.467522+0000","last_unstale":"2024-08-17T17:56:03.546166+0000","last_undegraded":"2024-08-17T17:56:03.546166+0000","last_fullsized":"2024-08-17T17:56:03.546166+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:52:55.249649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1315,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.780586+0000","last_change":"2024-08-17T17:23:07.392447+0000","last_active":"2024-08-17T17:56:03.780586+0000","last_peered":"2024-08-17T17:56:03.780586+0000","last_clean":"2024-08-17T17:56:03.780586+0000","last_became_active":"2024-08-17T17:23:07.391765+0000","last_became_peered":"2024-08-17T17:23:07.391765+0000","last_unstale":"2024-08-17T17:56:03.780586+0000","last_undegraded":"2024-08-17T17:56:03.780586+0000","last_fullsized":"2024-08-17T17:56:03.780586+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:59:18.602346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":83,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.546918+0000","last_change":"2024-08-17T17:53:58.839889+0000","last_active":"2024-08-17T17:56:03.546918+0000","last_peered":"2024-08-17T17:56:03.546918+0000","last_clean":"2024-08-17T17:56:03.546918+0000","last_became_active":"2024-08-17T17:53:58.839537+0000","last_became_peered":"2024-08-17T17:53:58.839537+0000","last_unstale":"2024-08-17T17:56:03.546918+0000","last_undegraded":"2024-08-17T17:56:03.546918+0000","last_fullsized":"2024-08-17T17:56:03.546918+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:47.169954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1999,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.526246+0000","last_change":"2024-08-17T17:23:42.936147+0000","last_active":"2024-08-17T17:56:03.526246+0000","last_peered":"2024-08-17T17:56:03.526246+0000","last_clean":"2024-08-17T17:56:03.526246+0000","last_became_active":"2024-08-17T17:23:42.935859+0000","last_became_peered":"2024-08-17T17:23:42.935859+0000","last_unstale":"2024-08-17T17:56:03.526246+0000","last_undegraded":"2024-08-17T17:56:03.526246+0000","last_fullsized":"2024-08-17T17:56:03.526246+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:47:36.838706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1328,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.525286+0000","last_change":"2024-08-17T17:10:05.969131+0000","last_active":"2024-08-17T17:56:02.525286+0000","last_peered":"2024-08-17T17:56:02.525286+0000","last_clean":"2024-08-17T17:56:02.525286+0000","last_became_active":"2024-08-17T17:10:05.968856+0000","last_became_peered":"2024-08-17T17:10:05.968856+0000","last_unstale":"2024-08-17T17:56:02.525286+0000","last_undegraded":"2024-08-17T17:56:02.525286+0000","last_fullsized":"2024-08-17T17:56:02.525286+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:25:49.550065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1319,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.525141+0000","last_change":"2024-08-17T17:23:07.455919+0000","last_active":"2024-08-17T17:56:02.525141+0000","last_peered":"2024-08-17T17:56:02.525141+0000","last_clean":"2024-08-17T17:56:02.525141+0000","last_became_active":"2024-08-17T17:23:07.455284+0000","last_became_peered":"2024-08-17T17:23:07.455284+0000","last_unstale":"2024-08-17T17:56:02.525141+0000","last_undegraded":"2024-08-17T17:56:02.525141+0000","last_fullsized":"2024-08-17T17:56:02.525141+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:37:46.260775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1307,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.524805+0000","last_change":"2024-08-17T17:10:06.979492+0000","last_active":"2024-08-17T17:56:02.524805+0000","last_peered":"2024-08-17T17:56:02.524805+0000","last_clean":"2024-08-17T17:56:02.524805+0000","last_became_active":"2024-08-17T17:10:06.979158+0000","last_became_peered":"2024-08-17T17:10:06.979158+0000","last_unstale":"2024-08-17T17:56:02.524805+0000","last_undegraded":"2024-08-17T17:56:02.524805+0000","last_fullsized":"2024-08-17T17:56:02.524805+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:12:42.261365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1331,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.531078+0000","last_change":"2024-08-17T17:23:42.935099+0000","last_active":"2024-08-17T17:56:02.531078+0000","last_peered":"2024-08-17T17:56:02.531078+0000","last_clean":"2024-08-17T17:56:02.531078+0000","last_became_active":"2024-08-17T17:23:42.934501+0000","last_became_peered":"2024-08-17T17:23:42.934501+0000","last_unstale":"2024-08-17T17:56:02.531078+0000","last_undegraded":"2024-08-17T17:56:02.531078+0000","last_fullsized":"2024-08-17T17:56:02.531078+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:15:19.095713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":1997,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.549151+0000","last_change":"2024-08-17T17:23:07.436763+0000","last_active":"2024-08-17T17:56:03.549151+0000","last_peered":"2024-08-17T17:56:03.549151+0000","last_clean":"2024-08-17T17:56:03.549151+0000","last_became_active":"2024-08-17T17:23:07.434720+0000","last_became_peered":"2024-08-17T17:23:07.434720+0000","last_unstale":"2024-08-17T17:56:03.549151+0000","last_undegraded":"2024-08-17T17:56:03.549151+0000","last_fullsized":"2024-08-17T17:56:03.549151+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:10:20.594115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1967,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.545846+0000","last_change":"2024-08-17T17:24:17.442329+0000","last_active":"2024-08-17T17:56:03.545846+0000","last_peered":"2024-08-17T17:56:03.545846+0000","last_clean":"2024-08-17T17:56:03.545846+0000","last_became_active":"2024-08-17T17:24:17.441666+0000","last_became_peered":"2024-08-17T17:24:17.441666+0000","last_unstale":"2024-08-17T17:56:03.545846+0000","last_undegraded":"2024-08-17T17:56:03.545846+0000","last_fullsized":"2024-08-17T17:56:03.545846+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:43:06.033194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1981,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.526943+0000","last_change":"2024-08-17T17:23:07.428296+0000","last_active":"2024-08-17T17:56:03.526943+0000","last_peered":"2024-08-17T17:56:03.526943+0000","last_clean":"2024-08-17T17:56:03.526943+0000","last_became_active":"2024-08-17T17:23:07.428010+0000","last_became_peered":"2024-08-17T17:23:07.428010+0000","last_unstale":"2024-08-17T17:56:03.526943+0000","last_undegraded":"2024-08-17T17:56:03.526943+0000","last_fullsized":"2024-08-17T17:56:03.526943+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:45.459212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":1968,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.547240+0000","last_change":"2024-08-17T17:24:17.481368+0000","last_active":"2024-08-17T17:56:03.547240+0000","last_peered":"2024-08-17T17:56:03.547240+0000","last_clean":"2024-08-17T17:56:03.547240+0000","last_became_active":"2024-08-17T17:24:17.477509+0000","last_became_peered":"2024-08-17T17:24:17.477509+0000","last_unstale":"2024-08-17T17:56:03.547240+0000","last_undegraded":"2024-08-17T17:56:03.547240+0000","last_fullsized":"2024-08-17T17:56:03.547240+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:05:35.558004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1983,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.548900+0000","last_change":"2024-08-17T17:22:32.884274+0000","last_active":"2024-08-17T17:56:03.548900+0000","last_peered":"2024-08-17T17:56:03.548900+0000","last_clean":"2024-08-17T17:56:03.548900+0000","last_became_active":"2024-08-17T17:22:32.874667+0000","last_became_peered":"2024-08-17T17:22:32.874667+0000","last_unstale":"2024-08-17T17:56:03.548900+0000","last_undegraded":"2024-08-17T17:56:03.548900+0000","last_fullsized":"2024-08-17T17:56:03.548900+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:22:05.834587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1328,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.530638+0000","last_change":"2024-08-17T17:22:32.870569+0000","last_active":"2024-08-17T17:56:02.530638+0000","last_peered":"2024-08-17T17:56:02.530638+0000","last_clean":"2024-08-17T17:56:02.530638+0000","last_became_active":"2024-08-17T17:22:32.869335+0000","last_became_peered":"2024-08-17T17:22:32.869335+0000","last_unstale":"2024-08-17T17:56:02.530638+0000","last_undegraded":"2024-08-17T17:56:02.530638+0000","last_fullsized":"2024-08-17T17:56:02.530638+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:24.969174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2002,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.527341+0000","last_change":"2024-08-17T17:23:42.935825+0000","last_active":"2024-08-17T17:56:03.527341+0000","last_peered":"2024-08-17T17:56:03.527341+0000","last_clean":"2024-08-17T17:56:03.527341+0000","last_became_active":"2024-08-17T17:23:42.935150+0000","last_became_peered":"2024-08-17T17:23:42.935150+0000","last_unstale":"2024-08-17T17:56:03.527341+0000","last_undegraded":"2024-08-17T17:56:03.527341+0000","last_fullsized":"2024-08-17T17:56:03.527341+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:57:19.577644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1990,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.526190+0000","last_change":"2024-08-17T17:23:07.431586+0000","last_active":"2024-08-17T17:56:03.526190+0000","last_peered":"2024-08-17T17:56:03.526190+0000","last_clean":"2024-08-17T17:56:03.526190+0000","last_became_active":"2024-08-17T17:23:07.431301+0000","last_became_peered":"2024-08-17T17:23:07.431301+0000","last_unstale":"2024-08-17T17:56:03.526190+0000","last_undegraded":"2024-08-17T17:56:03.526190+0000","last_fullsized":"2024-08-17T17:56:03.526190+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:28:18.706489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2018,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.527179+0000","last_change":"2024-08-17T17:23:07.420550+0000","last_active":"2024-08-17T17:56:03.527179+0000","last_peered":"2024-08-17T17:56:03.527179+0000","last_clean":"2024-08-17T17:56:03.527179+0000","last_became_active":"2024-08-17T17:23:07.420223+0000","last_became_peered":"2024-08-17T17:23:07.420223+0000","last_unstale":"2024-08-17T17:56:03.527179+0000","last_undegraded":"2024-08-17T17:56:03.527179+0000","last_fullsized":"2024-08-17T17:56:03.527179+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:33:46.996371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1333,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.780909+0000","last_change":"2024-08-17T17:24:17.426267+0000","last_active":"2024-08-17T17:56:03.780909+0000","last_peered":"2024-08-17T17:56:03.780909+0000","last_clean":"2024-08-17T17:56:03.780909+0000","last_became_active":"2024-08-17T17:24:17.425925+0000","last_became_peered":"2024-08-17T17:24:17.425925+0000","last_unstale":"2024-08-17T17:56:03.780909+0000","last_undegraded":"2024-08-17T17:56:03.780909+0000","last_fullsized":"2024-08-17T17:56:03.780909+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:24:25.017856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1328,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.527268+0000","last_change":"2024-08-17T17:24:17.474138+0000","last_active":"2024-08-17T17:56:02.527268+0000","last_peered":"2024-08-17T17:56:02.527268+0000","last_clean":"2024-08-17T17:56:02.527268+0000","last_became_active":"2024-08-17T17:24:17.473708+0000","last_became_peered":"2024-08-17T17:24:17.473708+0000","last_unstale":"2024-08-17T17:56:02.527268+0000","last_undegraded":"2024-08-17T17:56:02.527268+0000","last_fullsized":"2024-08-17T17:56:02.527268+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:10:57.768067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1323,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.525548+0000","last_change":"2024-08-17T17:22:32.871533+0000","last_active":"2024-08-17T17:56:02.525548+0000","last_peered":"2024-08-17T17:56:02.525548+0000","last_clean":"2024-08-17T17:56:02.525548+0000","last_became_active":"2024-08-17T17:22:32.868226+0000","last_became_peered":"2024-08-17T17:22:32.868226+0000","last_unstale":"2024-08-17T17:56:02.525548+0000","last_undegraded":"2024-08-17T17:56:02.525548+0000","last_fullsized":"2024-08-17T17:56:02.525548+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:00:02.629455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":1332,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.526441+0000","last_change":"2024-08-17T17:23:07.391021+0000","last_active":"2024-08-17T17:56:02.526441+0000","last_peered":"2024-08-17T17:56:02.526441+0000","last_clean":"2024-08-17T17:56:02.526441+0000","last_became_active":"2024-08-17T17:23:07.390551+0000","last_became_peered":"2024-08-17T17:23:07.390551+0000","last_unstale":"2024-08-17T17:56:02.526441+0000","last_undegraded":"2024-08-17T17:56:02.526441+0000","last_fullsized":"2024-08-17T17:56:02.526441+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:06.239013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001023532,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2002,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.527162+0000","last_change":"2024-08-17T17:23:07.406362+0000","last_active":"2024-08-17T17:56:03.527162+0000","last_peered":"2024-08-17T17:56:03.527162+0000","last_clean":"2024-08-17T17:56:03.527162+0000","last_became_active":"2024-08-17T17:23:07.406071+0000","last_became_peered":"2024-08-17T17:23:07.406071+0000","last_unstale":"2024-08-17T17:56:03.527162+0000","last_undegraded":"2024-08-17T17:56:03.527162+0000","last_fullsized":"2024-08-17T17:56:03.527162+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:45:47.383432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1982,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.525875+0000","last_change":"2024-08-17T17:23:42.910246+0000","last_active":"2024-08-17T17:56:03.525875+0000","last_peered":"2024-08-17T17:56:03.525875+0000","last_clean":"2024-08-17T17:56:03.525875+0000","last_became_active":"2024-08-17T17:23:42.909929+0000","last_became_peered":"2024-08-17T17:23:42.909929+0000","last_unstale":"2024-08-17T17:56:03.525875+0000","last_undegraded":"2024-08-17T17:56:03.525875+0000","last_fullsized":"2024-08-17T17:56:03.525875+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:14:45.210558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2011,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.528558+0000","last_change":"2024-08-17T17:24:17.438891+0000","last_active":"2024-08-17T17:56:03.528558+0000","last_peered":"2024-08-17T17:56:03.528558+0000","last_clean":"2024-08-17T17:56:03.528558+0000","last_became_active":"2024-08-17T17:24:17.437714+0000","last_became_peered":"2024-08-17T17:24:17.437714+0000","last_unstale":"2024-08-17T17:56:03.528558+0000","last_undegraded":"2024-08-17T17:56:03.528558+0000","last_fullsized":"2024-08-17T17:56:03.528558+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:58.825017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2004,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.549283+0000","last_change":"2024-08-17T17:23:42.908634+0000","last_active":"2024-08-17T17:56:03.549283+0000","last_peered":"2024-08-17T17:56:03.549283+0000","last_clean":"2024-08-17T17:56:03.549283+0000","last_became_active":"2024-08-17T17:23:42.907675+0000","last_became_peered":"2024-08-17T17:23:42.907675+0000","last_unstale":"2024-08-17T17:56:03.549283+0000","last_undegraded":"2024-08-17T17:56:03.549283+0000","last_fullsized":"2024-08-17T17:56:03.549283+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:03:37.612222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1994,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.548509+0000","last_change":"2024-08-17T17:24:17.457559+0000","last_active":"2024-08-17T17:56:03.548509+0000","last_peered":"2024-08-17T17:56:03.548509+0000","last_clean":"2024-08-17T17:56:03.548509+0000","last_became_active":"2024-08-17T17:24:17.457262+0000","last_became_peered":"2024-08-17T17:24:17.457262+0000","last_unstale":"2024-08-17T17:56:03.548509+0000","last_undegraded":"2024-08-17T17:56:03.548509+0000","last_fullsized":"2024-08-17T17:56:03.548509+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:53:38.137277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":716,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.526007+0000","last_change":"2024-08-17T17:53:58.844594+0000","last_active":"2024-08-17T17:56:03.526007+0000","last_peered":"2024-08-17T17:56:03.526007+0000","last_clean":"2024-08-17T17:56:03.526007+0000","last_became_active":"2024-08-17T17:53:58.844085+0000","last_became_peered":"2024-08-17T17:53:58.844085+0000","last_unstale":"2024-08-17T17:56:03.526007+0000","last_undegraded":"2024-08-17T17:56:03.526007+0000","last_fullsized":"2024-08-17T17:56:03.526007+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:58:50.195797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1304,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.525858+0000","last_change":"2024-08-17T17:10:06.995023+0000","last_active":"2024-08-17T17:56:03.525858+0000","last_peered":"2024-08-17T17:56:03.525858+0000","last_clean":"2024-08-17T17:56:03.525858+0000","last_became_active":"2024-08-17T17:10:06.994757+0000","last_became_peered":"2024-08-17T17:10:06.994757+0000","last_unstale":"2024-08-17T17:56:03.525858+0000","last_undegraded":"2024-08-17T17:56:03.525858+0000","last_fullsized":"2024-08-17T17:56:03.525858+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:31:52.163518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1934,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.546351+0000","last_change":"2024-08-17T17:24:17.468443+0000","last_active":"2024-08-17T17:56:03.546351+0000","last_peered":"2024-08-17T17:56:03.546351+0000","last_clean":"2024-08-17T17:56:03.546351+0000","last_became_active":"2024-08-17T17:24:17.467394+0000","last_became_peered":"2024-08-17T17:24:17.467394+0000","last_unstale":"2024-08-17T17:56:03.546351+0000","last_undegraded":"2024-08-17T17:56:03.546351+0000","last_fullsized":"2024-08-17T17:56:03.546351+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:13.024253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":1331,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.526936+0000","last_change":"2024-08-17T17:23:42.920552+0000","last_active":"2024-08-17T17:56:02.526936+0000","last_peered":"2024-08-17T17:56:02.526936+0000","last_clean":"2024-08-17T17:56:02.526936+0000","last_became_active":"2024-08-17T17:23:42.915824+0000","last_became_peered":"2024-08-17T17:23:42.915824+0000","last_unstale":"2024-08-17T17:56:02.526936+0000","last_undegraded":"2024-08-17T17:56:02.526936+0000","last_fullsized":"2024-08-17T17:56:02.526936+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:25:18.769504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041737399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":686,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.547961+0000","last_change":"2024-08-17T17:53:58.844782+0000","last_active":"2024-08-17T17:56:03.547961+0000","last_peered":"2024-08-17T17:56:03.547961+0000","last_clean":"2024-08-17T17:56:03.547961+0000","last_became_active":"2024-08-17T17:53:58.843133+0000","last_became_peered":"2024-08-17T17:53:58.843133+0000","last_unstale":"2024-08-17T17:56:03.547961+0000","last_undegraded":"2024-08-17T17:56:03.547961+0000","last_fullsized":"2024-08-17T17:56:03.547961+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T17:06:21.174471+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:30:14.411310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1980,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.527087+0000","last_change":"2024-08-17T17:24:17.455710+0000","last_active":"2024-08-17T17:56:03.527087+0000","last_peered":"2024-08-17T17:56:03.527087+0000","last_clean":"2024-08-17T17:56:03.527087+0000","last_became_active":"2024-08-17T17:24:17.452989+0000","last_became_peered":"2024-08-17T17:24:17.452989+0000","last_unstale":"2024-08-17T17:56:03.527087+0000","last_undegraded":"2024-08-17T17:56:03.527087+0000","last_fullsized":"2024-08-17T17:56:03.527087+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:58:07.894968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1974,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.545688+0000","last_change":"2024-08-17T17:24:17.453639+0000","last_active":"2024-08-17T17:56:03.545688+0000","last_peered":"2024-08-17T17:56:03.545688+0000","last_clean":"2024-08-17T17:56:03.545688+0000","last_became_active":"2024-08-17T17:24:17.443638+0000","last_became_peered":"2024-08-17T17:24:17.443638+0000","last_unstale":"2024-08-17T17:56:03.545688+0000","last_undegraded":"2024-08-17T17:56:03.545688+0000","last_fullsized":"2024-08-17T17:56:03.545688+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:42.107479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1320,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.525410+0000","last_change":"2024-08-17T17:23:07.451866+0000","last_active":"2024-08-17T17:56:02.525410+0000","last_peered":"2024-08-17T17:56:02.525410+0000","last_clean":"2024-08-17T17:56:02.525410+0000","last_became_active":"2024-08-17T17:23:07.451553+0000","last_became_peered":"2024-08-17T17:23:07.451553+0000","last_unstale":"2024-08-17T17:56:02.525410+0000","last_undegraded":"2024-08-17T17:56:02.525410+0000","last_fullsized":"2024-08-17T17:56:02.525410+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:37:23.930651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1322,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.524889+0000","last_change":"2024-08-17T17:23:42.915851+0000","last_active":"2024-08-17T17:56:02.524889+0000","last_peered":"2024-08-17T17:56:02.524889+0000","last_clean":"2024-08-17T17:56:02.524889+0000","last_became_active":"2024-08-17T17:23:42.915552+0000","last_became_peered":"2024-08-17T17:23:42.915552+0000","last_unstale":"2024-08-17T17:56:02.524889+0000","last_undegraded":"2024-08-17T17:56:02.524889+0000","last_fullsized":"2024-08-17T17:56:02.524889+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:45.197876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1335,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.525399+0000","last_change":"2024-08-17T17:23:42.930803+0000","last_active":"2024-08-17T17:56:02.525399+0000","last_peered":"2024-08-17T17:56:02.525399+0000","last_clean":"2024-08-17T17:56:02.525399+0000","last_became_active":"2024-08-17T17:23:42.930458+0000","last_became_peered":"2024-08-17T17:23:42.930458+0000","last_unstale":"2024-08-17T17:56:02.525399+0000","last_undegraded":"2024-08-17T17:56:02.525399+0000","last_fullsized":"2024-08-17T17:56:02.525399+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:40.816706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1319,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.528754+0000","last_change":"2024-08-17T17:23:07.401624+0000","last_active":"2024-08-17T17:56:02.528754+0000","last_peered":"2024-08-17T17:56:02.528754+0000","last_clean":"2024-08-17T17:56:02.528754+0000","last_became_active":"2024-08-17T17:23:07.400222+0000","last_became_peered":"2024-08-17T17:23:07.400222+0000","last_unstale":"2024-08-17T17:56:02.528754+0000","last_undegraded":"2024-08-17T17:56:02.528754+0000","last_fullsized":"2024-08-17T17:56:02.528754+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:07.069989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1320,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.525200+0000","last_change":"2024-08-17T17:24:17.470809+0000","last_active":"2024-08-17T17:56:02.525200+0000","last_peered":"2024-08-17T17:56:02.525200+0000","last_clean":"2024-08-17T17:56:02.525200+0000","last_became_active":"2024-08-17T17:24:17.469753+0000","last_became_peered":"2024-08-17T17:24:17.469753+0000","last_unstale":"2024-08-17T17:56:02.525200+0000","last_undegraded":"2024-08-17T17:56:02.525200+0000","last_fullsized":"2024-08-17T17:56:02.525200+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:00.049425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":1968,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.546551+0000","last_change":"2024-08-17T17:24:17.461258+0000","last_active":"2024-08-17T17:56:03.546551+0000","last_peered":"2024-08-17T17:56:03.546551+0000","last_clean":"2024-08-17T17:56:03.546551+0000","last_became_active":"2024-08-17T17:24:17.460895+0000","last_became_peered":"2024-08-17T17:24:17.460895+0000","last_unstale":"2024-08-17T17:56:03.546551+0000","last_undegraded":"2024-08-17T17:56:03.546551+0000","last_fullsized":"2024-08-17T17:56:03.546551+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:43:24.208259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2006,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.548705+0000","last_change":"2024-08-17T17:23:42.901638+0000","last_active":"2024-08-17T17:56:03.548705+0000","last_peered":"2024-08-17T17:56:03.548705+0000","last_clean":"2024-08-17T17:56:03.548705+0000","last_became_active":"2024-08-17T17:23:42.901335+0000","last_became_peered":"2024-08-17T17:23:42.901335+0000","last_unstale":"2024-08-17T17:56:03.548705+0000","last_undegraded":"2024-08-17T17:56:03.548705+0000","last_fullsized":"2024-08-17T17:56:03.548705+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:25:35.332839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1316,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.527480+0000","last_change":"2024-08-17T17:24:17.473913+0000","last_active":"2024-08-17T17:56:02.527480+0000","last_peered":"2024-08-17T17:56:02.527480+0000","last_clean":"2024-08-17T17:56:02.527480+0000","last_became_active":"2024-08-17T17:24:17.473195+0000","last_became_peered":"2024-08-17T17:24:17.473195+0000","last_unstale":"2024-08-17T17:56:02.527480+0000","last_undegraded":"2024-08-17T17:56:02.527480+0000","last_fullsized":"2024-08-17T17:56:02.527480+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:12.998033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1318,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.528649+0000","last_change":"2024-08-17T17:23:07.402010+0000","last_active":"2024-08-17T17:56:02.528649+0000","last_peered":"2024-08-17T17:56:02.528649+0000","last_clean":"2024-08-17T17:56:02.528649+0000","last_became_active":"2024-08-17T17:23:07.400849+0000","last_became_peered":"2024-08-17T17:23:07.400849+0000","last_unstale":"2024-08-17T17:56:02.528649+0000","last_undegraded":"2024-08-17T17:56:02.528649+0000","last_fullsized":"2024-08-17T17:56:02.528649+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:16:02.382257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":53,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.526799+0000","last_change":"2024-08-17T17:53:58.836811+0000","last_active":"2024-08-17T17:56:02.526799+0000","last_peered":"2024-08-17T17:56:02.526799+0000","last_clean":"2024-08-17T17:56:02.526799+0000","last_became_active":"2024-08-17T17:53:58.836139+0000","last_became_peered":"2024-08-17T17:53:58.836139+0000","last_unstale":"2024-08-17T17:56:02.526799+0000","last_undegraded":"2024-08-17T17:56:02.526799+0000","last_fullsized":"2024-08-17T17:56:02.526799+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:09:18.260164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1336,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.524729+0000","last_change":"2024-08-17T17:24:17.460203+0000","last_active":"2024-08-17T17:56:02.524729+0000","last_peered":"2024-08-17T17:56:02.524729+0000","last_clean":"2024-08-17T17:56:02.524729+0000","last_became_active":"2024-08-17T17:24:17.459682+0000","last_became_peered":"2024-08-17T17:24:17.459682+0000","last_unstale":"2024-08-17T17:56:02.524729+0000","last_undegraded":"2024-08-17T17:56:02.524729+0000","last_fullsized":"2024-08-17T17:56:02.524729+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:58.703700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":1316,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.530764+0000","last_change":"2024-08-17T17:23:07.454042+0000","last_active":"2024-08-17T17:56:02.530764+0000","last_peered":"2024-08-17T17:56:02.530764+0000","last_clean":"2024-08-17T17:56:02.530764+0000","last_became_active":"2024-08-17T17:23:07.453712+0000","last_became_peered":"2024-08-17T17:23:07.453712+0000","last_unstale":"2024-08-17T17:56:02.530764+0000","last_undegraded":"2024-08-17T17:56:02.530764+0000","last_fullsized":"2024-08-17T17:56:02.530764+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:03:07.643456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1301,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.526189+0000","last_change":"2024-08-17T17:10:06.994799+0000","last_active":"2024-08-17T17:56:02.526189+0000","last_peered":"2024-08-17T17:56:02.526189+0000","last_clean":"2024-08-17T17:56:02.526189+0000","last_became_active":"2024-08-17T17:10:06.994150+0000","last_became_peered":"2024-08-17T17:10:06.994150+0000","last_unstale":"2024-08-17T17:56:02.526189+0000","last_undegraded":"2024-08-17T17:56:02.526189+0000","last_fullsized":"2024-08-17T17:56:02.526189+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:50:47.293091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":1314,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.530800+0000","last_change":"2024-08-17T17:22:32.873334+0000","last_active":"2024-08-17T17:56:02.530800+0000","last_peered":"2024-08-17T17:56:02.530800+0000","last_clean":"2024-08-17T17:56:02.530800+0000","last_became_active":"2024-08-17T17:22:32.870147+0000","last_became_peered":"2024-08-17T17:22:32.870147+0000","last_unstale":"2024-08-17T17:56:02.530800+0000","last_undegraded":"2024-08-17T17:56:02.530800+0000","last_fullsized":"2024-08-17T17:56:02.530800+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:18:03.750592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1325,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.528534+0000","last_change":"2024-08-17T17:22:32.847989+0000","last_active":"2024-08-17T17:56:02.528534+0000","last_peered":"2024-08-17T17:56:02.528534+0000","last_clean":"2024-08-17T17:56:02.528534+0000","last_became_active":"2024-08-17T17:22:32.846593+0000","last_became_peered":"2024-08-17T17:22:32.846593+0000","last_unstale":"2024-08-17T17:56:02.528534+0000","last_undegraded":"2024-08-17T17:56:02.528534+0000","last_fullsized":"2024-08-17T17:56:02.528534+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:33:50.370124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1972,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.547171+0000","last_change":"2024-08-17T17:24:17.451528+0000","last_active":"2024-08-17T17:56:03.547171+0000","last_peered":"2024-08-17T17:56:03.547171+0000","last_clean":"2024-08-17T17:56:03.547171+0000","last_became_active":"2024-08-17T17:24:17.451246+0000","last_became_peered":"2024-08-17T17:24:17.451246+0000","last_unstale":"2024-08-17T17:56:03.547171+0000","last_undegraded":"2024-08-17T17:56:03.547171+0000","last_fullsized":"2024-08-17T17:56:03.547171+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:38:18.176559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":41,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.524780+0000","last_change":"2024-08-17T17:53:58.838279+0000","last_active":"2024-08-17T17:56:02.524780+0000","last_peered":"2024-08-17T17:56:02.524780+0000","last_clean":"2024-08-17T17:56:02.524780+0000","last_became_active":"2024-08-17T17:53:58.837978+0000","last_became_peered":"2024-08-17T17:53:58.837978+0000","last_unstale":"2024-08-17T17:56:02.524780+0000","last_undegraded":"2024-08-17T17:56:02.524780+0000","last_fullsized":"2024-08-17T17:56:02.524780+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:06.019142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1990,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.549086+0000","last_change":"2024-08-17T17:24:17.460371+0000","last_active":"2024-08-17T17:56:03.549086+0000","last_peered":"2024-08-17T17:56:03.549086+0000","last_clean":"2024-08-17T17:56:03.549086+0000","last_became_active":"2024-08-17T17:24:17.460074+0000","last_became_peered":"2024-08-17T17:24:17.460074+0000","last_unstale":"2024-08-17T17:56:03.549086+0000","last_undegraded":"2024-08-17T17:56:03.549086+0000","last_fullsized":"2024-08-17T17:56:03.549086+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:08:48.820168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1991,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.548517+0000","last_change":"2024-08-17T17:24:17.457942+0000","last_active":"2024-08-17T17:56:03.548517+0000","last_peered":"2024-08-17T17:56:03.548517+0000","last_clean":"2024-08-17T17:56:03.548517+0000","last_became_active":"2024-08-17T17:24:17.456475+0000","last_became_peered":"2024-08-17T17:24:17.456475+0000","last_unstale":"2024-08-17T17:56:03.548517+0000","last_undegraded":"2024-08-17T17:56:03.548517+0000","last_fullsized":"2024-08-17T17:56:03.548517+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:12.325657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1314,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.526943+0000","last_change":"2024-08-17T17:24:17.451287+0000","last_active":"2024-08-17T17:56:02.526943+0000","last_peered":"2024-08-17T17:56:02.526943+0000","last_clean":"2024-08-17T17:56:02.526943+0000","last_became_active":"2024-08-17T17:24:17.450951+0000","last_became_peered":"2024-08-17T17:24:17.450951+0000","last_unstale":"2024-08-17T17:56:02.526943+0000","last_undegraded":"2024-08-17T17:56:02.526943+0000","last_fullsized":"2024-08-17T17:56:02.526943+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:13:37.590108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":1315,"reported_epoch":1347,"state":"active+clean","last_fresh":"2024-08-17T17:56:02.530386+0000","last_change":"2024-08-17T17:23:42.907970+0000","last_active":"2024-08-17T17:56:02.530386+0000","last_peered":"2024-08-17T17:56:02.530386+0000","last_clean":"2024-08-17T17:56:02.530386+0000","last_became_active":"2024-08-17T17:23:42.907595+0000","last_became_peered":"2024-08-17T17:23:42.907595+0000","last_unstale":"2024-08-17T17:56:02.530386+0000","last_undegraded":"2024-08-17T17:56:02.530386+0000","last_fullsized":"2024-08-17T17:56:02.530386+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:11:46.130004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1972,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.546767+0000","last_change":"2024-08-17T17:24:17.486591+0000","last_active":"2024-08-17T17:56:03.546767+0000","last_peered":"2024-08-17T17:56:03.546767+0000","last_clean":"2024-08-17T17:56:03.546767+0000","last_became_active":"2024-08-17T17:24:17.486320+0000","last_became_peered":"2024-08-17T17:24:17.486320+0000","last_unstale":"2024-08-17T17:56:03.546767+0000","last_undegraded":"2024-08-17T17:56:03.546767+0000","last_fullsized":"2024-08-17T17:56:03.546767+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:03.249803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":53,"num_pg":26,"stat_sum":{"num_bytes":3818225688,"num_objects":58263,"num_object_clones":0,"num_object_copies":174789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7758,"num_objects_misplaced":10644,"num_objects_unfound":0,"num_objects_dirty":58263,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152351,"num_write_kb":5526497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9678,"num_bytes_recovered":609222656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11974356992,"data_stored":11974344776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205546,"ondisk_log_size":205546,"up":78,"acting":75,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234561,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":526004,"kb_used_data":378472,"kb_used_omap":1,"kb_used_meta":147518,"kb_avail":93219148,"statfs":{"total":95995035648,"available":95456407552,"internally_reserved":0,"allocated":387555328,"data_stored":383439356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":151058891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496199,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1280040,"kb_used_data":1068512,"kb_used_omap":1,"kb_used_meta":211518,"kb_avail":92465112,"statfs":{"total":95995035648,"available":94684274688,"internally_reserved":0,"allocated":1094156288,"data_stored":1090048381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":216594887},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565440,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1501652,"kb_used_data":1347276,"kb_used_omap":72,"kb_used_meta":154231,"kb_avail":92243500,"statfs":{"total":95995035648,"available":94457344000,"internally_reserved":0,"allocated":1379610624,"data_stored":1375517910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":157933038},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":674,"seq":2894807957895,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1380828,"kb_used_data":1190484,"kb_used_omap":74,"kb_used_meta":190261,"kb_avail":92364324,"statfs":{"total":95995035648,"available":94581067776,"internally_reserved":0,"allocated":1219055616,"data_stored":1214955260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76116,"internal_metadata":194827948},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743186830,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2712856,"kb_used_data":2432336,"kb_used_omap":66,"kb_used_meta":280445,"kb_avail":91032296,"statfs":{"total":95995035648,"available":93217071104,"internally_reserved":0,"allocated":2490712064,"data_stored":2486615207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":287176363},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448469,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1851424,"kb_used_data":1648788,"kb_used_omap":75,"kb_used_meta":202548,"kb_avail":91893728,"statfs":{"total":95995035648,"available":94099177472,"internally_reserved":0,"allocated":1688358912,"data_stored":1684259064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":207409343},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757835,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1873380,"kb_used_data":1665880,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":91871772,"statfs":{"total":95995035648,"available":94076694528,"internally_reserved":0,"allocated":1705861120,"data_stored":1701756711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":212466118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019472,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2450280,"kb_used_data":2108192,"kb_used_omap":1,"kb_used_meta":341950,"kb_avail":91294872,"statfs":{"total":95995035648,"available":93485948928,"internally_reserved":0,"allocated":2158788608,"data_stored":2154680214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":350157254},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1661075456,"data_stored":1661075456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2479427584,"data_stored":2479423512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1201803264,"data_stored":1201799192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1359020032,"data_stored":1359020032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2144665600,"data_stored":2144665600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1680543744,"data_stored":1680539672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1073283072,"data_stored":1073283072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":374538240,"data_stored":374538240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:56:10.673 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:56:10.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:10.701 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:10.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:10.802 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:10.823 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:10 smithi165 ceph-mon[105517]: pgmap v2207: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 435 op/s; 7758/174867 objects degraded (4.437%); 10644/174867 objects misplaced (6.087%); 9.2 MiB/s, 146 objects/s recovering 2024-08-17T17:56:10.823 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:10 smithi165 ceph-mon[105517]: from='client.35247 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:56:10.824 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:56:10.824 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:56:10.824 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:56:10.824 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:56:10.824 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:56:10.828 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:10.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:10.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:10.930 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:10.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:10 smithi136 ceph-mon[109861]: pgmap v2207: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 435 op/s; 7758/174867 objects degraded (4.437%); 10644/174867 objects misplaced (6.087%); 9.2 MiB/s, 146 objects/s recovering 2024-08-17T17:56:10.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:10 smithi136 ceph-mon[109861]: from='client.35247 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:56:10.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:56:10.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:56:10.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:56:10.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:56:10.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:56:10.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:10 smithi136 ceph-mon[115078]: pgmap v2207: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 435 op/s; 7758/174867 objects degraded (4.437%); 10644/174867 objects misplaced (6.087%); 9.2 MiB/s, 146 objects/s recovering 2024-08-17T17:56:10.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:10 smithi136 ceph-mon[115078]: from='client.35247 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:56:10.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:56:10.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:56:10.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:56:10.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:56:10.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:56:10.994 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:10.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:10.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:11.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:11.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:11.130 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:11.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:11.231 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:11.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:11.271 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:11.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:11.372 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:11.394 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:11.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:11.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:11.495 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T17:56:11.495 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:56:11.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:11.530 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:11.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:11.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:11.632 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:11.668 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:11.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:11.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:11.770 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:11.809 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:11.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:11.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:11.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:11.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:11.948 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:11.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:12.049 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:12.082 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:12.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:12.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:12.184 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:12.217 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:12.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:12.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:12.222 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-17T17:56:12.321 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:12.360 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:12.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:12.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:12.462 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:12.497 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:12.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:12.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:12.537 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:56:12.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:12.642 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:12.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:12.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:12.744 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:12.787 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:12.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:12.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:12.812 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:12 smithi165 ceph-mon[105517]: pgmap v2208: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 358 op/s; 7758/173790 objects degraded (4.464%); 10644/173790 objects misplaced (6.125%); 3.8 MiB/s, 61 objects/s recovering 2024-08-17T17:56:12.889 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:12.930 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:12.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:12.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:12.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:12 smithi136 ceph-mon[109861]: pgmap v2208: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 358 op/s; 7758/173790 objects degraded (4.464%); 10644/173790 objects misplaced (6.125%); 3.8 MiB/s, 61 objects/s recovering 2024-08-17T17:56:12.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:12 smithi136 ceph-mon[115078]: pgmap v2208: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 358 op/s; 7758/173790 objects degraded (4.464%); 10644/173790 objects misplaced (6.125%); 3.8 MiB/s, 61 objects/s recovering 2024-08-17T17:56:13.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:13.068 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:13.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:13.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:13.169 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:13.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:13.214 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:13.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:13.315 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:13.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:13.352 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:13.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:13.453 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:13.474 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:13.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:13.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:13.576 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:13.608 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:13.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:13.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:13.709 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:13.744 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:13.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:13.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:13.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:13 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 7512/163125 objects degraded (4.605%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:13.846 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:13.889 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:13.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:13.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:13 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 7512/163125 objects degraded (4.605%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:13.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:13 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 7512/163125 objects degraded (4.605%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:13.991 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:14.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:14.012 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:14.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:14.113 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:14.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:14.154 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:14.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:14.255 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:14.295 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:14.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:14.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:14.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:14.405 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T17:56:14.405 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:56:14.430 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:14.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:14.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:14.532 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:14.574 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:14.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:14.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:14.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:14.704 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:14.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:14.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:14.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:14 smithi165 ceph-mon[105517]: pgmap v2209: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 646 op/s; 7512/163125 objects degraded (4.605%); 10004/163125 objects misplaced (6.133%); 5.5 MiB/s, 88 objects/s recovering 2024-08-17T17:56:14.806 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:14.828 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:14 smithi136 ceph-mon[109861]: pgmap v2209: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 646 op/s; 7512/163125 objects degraded (4.605%); 10004/163125 objects misplaced (6.133%); 5.5 MiB/s, 88 objects/s recovering 2024-08-17T17:56:14.828 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:14 smithi136 ceph-mon[115078]: pgmap v2209: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 646 op/s; 7512/163125 objects degraded (4.605%); 10004/163125 objects misplaced (6.133%); 5.5 MiB/s, 88 objects/s recovering 2024-08-17T17:56:14.843 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:14.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:14.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:14.946 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:14.973 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:14.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:14.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:15.074 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:15.106 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:15.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:15.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:15.208 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:15.246 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:15.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:15.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:15.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:15.381 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:15.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:15.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:15.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:15.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:15.522 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:15.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:15.624 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:15.650 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:15.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:15.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:15.752 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:15.782 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-17T17:56:15.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:15.785 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:15.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:15.787 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T17:56:15.787 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T17:56:15.788 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T17:56:15.788 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T17:56:15.788 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T17:56:15.788 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T17:56:15.788 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T17:56:15.788 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T17:56:15.788 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:56:15.788 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:56:15.788 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:56:15.788 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:56:15.789 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:56:15.789 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:56:15.789 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:56:15.789 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:56:15.886 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:15.914 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:15.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:15.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:15.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:15 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2529963422' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:56:15.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:15 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2529963422' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:56:16.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:15 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2529963422' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:56:16.016 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:16.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:16.073 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:16.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:16.109 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:56:16.174 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:16.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:16.209 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:16.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:16.310 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:16.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:16.346 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:16.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:16.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:16.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:16.482 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:16.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:16.583 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:16.604 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:16.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:16.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:16.664 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:16 smithi165 ceph-mon[105517]: pgmap v2210: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 581 op/s; 7512/163125 objects degraded (4.605%); 10004/163125 objects misplaced (6.133%); 4.9 MiB/s, 79 objects/s recovering 2024-08-17T17:56:16.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:16.736 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:16.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:16.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:16.838 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:16.859 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:16.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:16.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:16.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:16 smithi136 ceph-mon[109861]: pgmap v2210: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 581 op/s; 7512/163125 objects degraded (4.605%); 10004/163125 objects misplaced (6.133%); 4.9 MiB/s, 79 objects/s recovering 2024-08-17T17:56:16.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:16 smithi136 ceph-mon[115078]: pgmap v2210: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 581 op/s; 7512/163125 objects degraded (4.605%); 10004/163125 objects misplaced (6.133%); 4.9 MiB/s, 79 objects/s recovering 2024-08-17T17:56:16.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:16.992 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:16.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:16.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:17.094 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:17.142 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:17.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:17.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:17.244 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:17.282 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:17.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:17.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:17.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:17.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:17.429 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:17.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:17.530 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:17.570 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:17.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:17.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:17.672 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:17.713 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:17.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:17.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:17.815 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:17.852 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:56:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:56:17] "GET /metrics HTTP/1.1" 200 36563 "" "Prometheus/2.51.0" 2024-08-17T17:56:17.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:17.863 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:17.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:17.964 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:18.003 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:18.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:18.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:18.104 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:18.148 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T17:56:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T17:56:18.148 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-08-17T17:56:18.148 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T17:56:18.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:18.149 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:18.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:18.250 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:18.285 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:18.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:18.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:18.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:18.413 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:18.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:18.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:18.515 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:18.580 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:18.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:18.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:18.681 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:18.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:18.710 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:18.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:18.715 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:18 smithi136 ceph-mon[109861]: pgmap v2211: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 715 op/s; 7253/156759 objects degraded (4.627%); 9402/156759 objects misplaced (5.998%); 6.5 MiB/s, 103 objects/s recovering 2024-08-17T17:56:18.715 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:18 smithi136 ceph-mon[115078]: pgmap v2211: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 715 op/s; 7253/156759 objects degraded (4.627%); 9402/156759 objects misplaced (5.998%); 6.5 MiB/s, 103 objects/s recovering 2024-08-17T17:56:18.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:18.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:18.846 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:18.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:18.890 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:18 smithi165 ceph-mon[105517]: pgmap v2211: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 715 op/s; 7253/156759 objects degraded (4.627%); 9402/156759 objects misplaced (5.998%); 6.5 MiB/s, 103 objects/s recovering 2024-08-17T17:56:18.919 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-17T17:56:18.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:18.947 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:56:18.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:18.993 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:18.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:19.094 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:19.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:19.145 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:19.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:19.225 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:56:19.246 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:19.279 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:19.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:19.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:19.380 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:19.425 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:19.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:19.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:19.526 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:19.566 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:19.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:19.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:19.668 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:19.708 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:19.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:19.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:19.809 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:19.837 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:19.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:19.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:19.939 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:19.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:19 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 7253/156759 objects degraded (4.627%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:19.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:19 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 7253/156759 objects degraded (4.627%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:19.985 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:19.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:19.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:20.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:19 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 7253/156759 objects degraded (4.627%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:20.088 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:20.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:20.122 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:20.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:20.223 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:20.249 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:20.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:20.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:20.350 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:20.386 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:20.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:20.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:20.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:20.532 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:20.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:20.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:20.631 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T17:56:20.635 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:20.668 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:20.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:20.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:20.715 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:20 smithi165 ceph-mon[105517]: pgmap v2212: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 736 op/s; 7253/152022 objects degraded (4.771%); 9402/152022 objects misplaced (6.185%); 3.9 MiB/s, 62 objects/s recovering 2024-08-17T17:56:20.715 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:20 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3086809089' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:56:20.770 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:20.812 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:20.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:20.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:20.882 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:20 smithi136 ceph-mon[109861]: pgmap v2212: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 736 op/s; 7253/152022 objects degraded (4.771%); 9402/152022 objects misplaced (6.185%); 3.9 MiB/s, 62 objects/s recovering 2024-08-17T17:56:20.882 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:20 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3086809089' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:56:20.882 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:20 smithi136 ceph-mon[115078]: pgmap v2212: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 736 op/s; 7253/152022 objects degraded (4.771%); 9402/152022 objects misplaced (6.185%); 3.9 MiB/s, 62 objects/s recovering 2024-08-17T17:56:20.882 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:20 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3086809089' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:56:20.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:20.950 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:56:20.953 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:20.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:20.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:21.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:21.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:21.096 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:21.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:21.196 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:21.235 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:21.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:21.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:21.339 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:21.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:21.377 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:21.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:21.383 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T17:56:21.383 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T17:56:21.383 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-08-17T17:56:21.383 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-08-17T17:56:21.383 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T17:56:21.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:21.515 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:21.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:21.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:21.617 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:21.665 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:21.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:21.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:21.766 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:21.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:21.810 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:21.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:21.911 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:21.944 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:21.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:21.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:21 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:56:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:21 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3086809089' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:56:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:21 smithi136 ceph-mon[109861]: osdmap e1349: 8 total, 8 up, 8 in 2024-08-17T17:56:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:21 smithi136 ceph-mon[109861]: pgmap v2214: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 758 op/s; 7253/151020 objects degraded (4.803%); 9402/151020 objects misplaced (6.226%); 4.7 MiB/s, 74 objects/s recovering 2024-08-17T17:56:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:56:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:56:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:21 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:56:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:21 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3086809089' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:56:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:21 smithi136 ceph-mon[115078]: osdmap e1349: 8 total, 8 up, 8 in 2024-08-17T17:56:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:21 smithi136 ceph-mon[115078]: pgmap v2214: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 758 op/s; 7253/151020 objects degraded (4.803%); 9402/151020 objects misplaced (6.226%); 4.7 MiB/s, 74 objects/s recovering 2024-08-17T17:56:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:56:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:56:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:21 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:56:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:21 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3086809089' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:56:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:21 smithi165 ceph-mon[105517]: osdmap e1349: 8 total, 8 up, 8 in 2024-08-17T17:56:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:21 smithi165 ceph-mon[105517]: pgmap v2214: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 758 op/s; 7253/151020 objects degraded (4.803%); 9402/151020 objects misplaced (6.226%); 4.7 MiB/s, 74 objects/s recovering 2024-08-17T17:56:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:56:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:56:22.046 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:22.076 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:22.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:22.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:22.093 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-17T17:56:22.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:22.248 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:22.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:22.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:22.350 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:22.380 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:22.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:22.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:22.451 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:56:22.482 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:22.525 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:22.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:22.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:22.626 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:22.657 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:22.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:22.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:22.758 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:22.789 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:22.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:22.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:22.891 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:22.923 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:22.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:22.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:23.025 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:23.061 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:23.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:23.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:23.163 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:23.187 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:23.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:23.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:23.289 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:23.328 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:23.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:23.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:23.429 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:23.465 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:23.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:23.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:23.566 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:23.611 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:23.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:23.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:23.713 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:23.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:23.753 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:23.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:23.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:23.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:23.893 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:23.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:23.977 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:23 smithi165 ceph-mon[105517]: pgmap v2215: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 753 op/s; 7037/140508 objects degraded (5.008%); 8788/140508 objects misplaced (6.254%); 4.5 MiB/s, 72 objects/s recovering 2024-08-17T17:56:23.995 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:24.025 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:24.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:24.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:24.126 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:24.164 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:24.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:24.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:24.168 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:23 smithi136 ceph-mon[109861]: pgmap v2215: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 753 op/s; 7037/140508 objects degraded (5.008%); 8788/140508 objects misplaced (6.254%); 4.5 MiB/s, 72 objects/s recovering 2024-08-17T17:56:24.169 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:23 smithi136 ceph-mon[115078]: pgmap v2215: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 753 op/s; 7037/140508 objects degraded (5.008%); 8788/140508 objects misplaced (6.254%); 4.5 MiB/s, 72 objects/s recovering 2024-08-17T17:56:24.266 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:24.298 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:24.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:24.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:24.400 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:24.411 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T17:56:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T17:56:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T17:56:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T17:56:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.411 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44783.0:3272 51.8 51:169c93f6:::rbd_data.aeefc272e7cc.000000000000005c:head [set-alloc-hint object_size 4194304 write_size 4194304,write 602761~3591543] snapc 6a=[6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1111)", 2024-08-17T17:56:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:47:16.694179+0000", 2024-08-17T17:56:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 547.71338984700003, 2024-08-17T17:56:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.050001372000000002, 2024-08-17T17:56:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:56:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:56:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:56:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44783", 2024-08-17T17:56:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3691373571", 2024-08-17T17:56:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3272 2024-08-17T17:56:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:56:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:56:24.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:47:16.694179+0000", 2024-08-17T17:56:24.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:56:24.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:47:16.694179+0000", 2024-08-17T17:56:24.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:56:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:47:16.694177+0000", 2024-08-17T17:56:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T17:56:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:56:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:47:16.699442+0000", 2024-08-17T17:56:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052647609999999997 2024-08-17T17:56:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:56:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:47:16.699457+0000", 2024-08-17T17:56:24.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.518e-05 2024-08-17T17:56:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:56:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:47:16.699465+0000", 2024-08-17T17:56:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3590000000000008e-06 2024-08-17T17:56:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:56:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:47:16.699525+0000", 2024-08-17T17:56:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9239e-05 2024-08-17T17:56:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:56:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:47:16.699679+0000", 2024-08-17T17:56:24.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015479400000000001 2024-08-17T17:56:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,6", 2024-08-17T17:56:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:47:16.699808+0000", 2024-08-17T17:56:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012887200000000001 2024-08-17T17:56:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:56:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:47:16.723373+0000", 2024-08-17T17:56:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023565101000000001 2024-08-17T17:56:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:56:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:47:16.728673+0000", 2024-08-17T17:56:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052992580000000003 2024-08-17T17:56:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:56:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:47:16.744095+0000", 2024-08-17T17:56:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015422046 2024-08-17T17:56:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:56:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:47:16.744143+0000", 2024-08-17T17:56:24.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8442999999999997e-05 2024-08-17T17:56:24.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:56:24.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:47:16.744180+0000", 2024-08-17T17:56:24.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7020000000000001e-05 2024-08-17T17:56:24.418 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.418 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:56:24.418 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.418 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44783.0:40756 51.5 e1146/1004 51:a4faf314:::rbd_data.aeefb29c4b4f.0000000000000000:head v 1146'47, mlcod=1143'44)", 2024-08-17T17:56:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:47:52.303794+0000", 2024-08-17T17:56:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 512.10377460799998, 2024-08-17T17:56:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.054938266999999999, 2024-08-17T17:56:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:56:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:56:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:56:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:56:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:47:52.303794+0000", 2024-08-17T17:56:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:56:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:47:52.303794+0000", 2024-08-17T17:56:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:56:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:47:52.303793+0000", 2024-08-17T17:56:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:56:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:56:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:47:52.318681+0000", 2024-08-17T17:56:24.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014887961999999999 2024-08-17T17:56:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:56:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:47:52.318729+0000", 2024-08-17T17:56:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7843999999999998e-05 2024-08-17T17:56:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:56:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:47:52.318750+0000", 2024-08-17T17:56:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.107e-05 2024-08-17T17:56:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:56:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:47:52.318866+0000", 2024-08-17T17:56:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115908 2024-08-17T17:56:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:56:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:47:52.318900+0000", 2024-08-17T17:56:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3881000000000003e-05 2024-08-17T17:56:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:56:24.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:47:52.358637+0000", 2024-08-17T17:56:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.039737521999999997 2024-08-17T17:56:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:56:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:47:52.358732+0000", 2024-08-17T17:56:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5001999999999996e-05 2024-08-17T17:56:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:56:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.34884.0:1215 52.d e1220/1153 52:b56ef0f6:::rbd_data.88441c375ed9.0000000000000000:head v 1220'21, mlcod=1196'19)", 2024-08-17T17:56:24.423 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:49:50.900443+0000", 2024-08-17T17:56:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 393.50712572399999, 2024-08-17T17:56:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.041317930000000003, 2024-08-17T17:56:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:56:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:56:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:56:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:56:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:49:50.900443+0000", 2024-08-17T17:56:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:56:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:49:50.900443+0000", 2024-08-17T17:56:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:56:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:49:50.900442+0000", 2024-08-17T17:56:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:56:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:56:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:49:50.922760+0000", 2024-08-17T17:56:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022318108 2024-08-17T17:56:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:56:24.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:49:50.922815+0000", 2024-08-17T17:56:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5157000000000001e-05 2024-08-17T17:56:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:56:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:49:50.922837+0000", 2024-08-17T17:56:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1421999999999999e-05 2024-08-17T17:56:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:56:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:49:50.922921+0000", 2024-08-17T17:56:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3752999999999996e-05 2024-08-17T17:56:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:56:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:49:50.922954+0000", 2024-08-17T17:56:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3423999999999997e-05 2024-08-17T17:56:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:56:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:49:50.941718+0000", 2024-08-17T17:56:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018764218999999999 2024-08-17T17:56:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:56:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:49:50.941761+0000", 2024-08-17T17:56:24.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2755e-05 2024-08-17T17:56:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:56:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.34890.0:1530 52.15 52:ab497c70:::rbd_data.884aff6c7779.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1241)", 2024-08-17T17:56:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:50:46.951388+0000", 2024-08-17T17:56:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 337.45618045399999, 2024-08-17T17:56:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.047910422000000001, 2024-08-17T17:56:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:56:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:56:24.428 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:56:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.34890", 2024-08-17T17:56:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3439126247", 2024-08-17T17:56:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1530 2024-08-17T17:56:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:56:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:56:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:50:46.951388+0000", 2024-08-17T17:56:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:56:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:50:46.951388+0000", 2024-08-17T17:56:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:56:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:50:46.951385+0000", 2024-08-17T17:56:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999962 2024-08-17T17:56:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:56:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:50:46.963317+0000", 2024-08-17T17:56:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011932042 2024-08-17T17:56:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:56:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:50:46.963347+0000", 2024-08-17T17:56:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0255000000000001e-05 2024-08-17T17:56:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:56:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:50:46.963362+0000", 2024-08-17T17:56:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4656e-05 2024-08-17T17:56:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:56:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:50:46.963478+0000", 2024-08-17T17:56:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116443 2024-08-17T17:56:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:56:24.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:50:46.963614+0000", 2024-08-17T17:56:24.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013626400000000001 2024-08-17T17:56:24.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4", 2024-08-17T17:56:24.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:50:46.963782+0000", 2024-08-17T17:56:24.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016820100000000001 2024-08-17T17:56:24.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:56:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:50:46.969943+0000", 2024-08-17T17:56:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0061606229999999996 2024-08-17T17:56:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:56:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:50:46.985110+0000", 2024-08-17T17:56:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015166956000000001 2024-08-17T17:56:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:56:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:50:46.999207+0000", 2024-08-17T17:56:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014096984 2024-08-17T17:56:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:56:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:50:46.999261+0000", 2024-08-17T17:56:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4076000000000002e-05 2024-08-17T17:56:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:56:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:50:46.999299+0000", 2024-08-17T17:56:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7527999999999997e-05 2024-08-17T17:56:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:56:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.34890.0:1531 52.d e1241/1153 52:b089ddbc:::rbd_data.884aff6c7779.0000000000000002:head v 1241'26, mlcod=1237'24)", 2024-08-17T17:56:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:50:46.956211+0000", 2024-08-17T17:56:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 337.45135755799998, 2024-08-17T17:56:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.042885597999999997, 2024-08-17T17:56:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:56:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:56:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:56:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:56:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:50:46.956211+0000", 2024-08-17T17:56:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:56:24.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:50:46.956211+0000", 2024-08-17T17:56:24.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:56:24.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:50:46.956210+0000", 2024-08-17T17:56:24.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:56:24.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:56:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:50:46.971230+0000", 2024-08-17T17:56:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015020567 2024-08-17T17:56:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:56:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:50:46.971257+0000", 2024-08-17T17:56:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7083000000000001e-05 2024-08-17T17:56:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:56:24.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:50:46.971268+0000", 2024-08-17T17:56:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1146e-05 2024-08-17T17:56:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:56:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:50:46.971340+0000", 2024-08-17T17:56:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1209000000000002e-05 2024-08-17T17:56:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:56:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:50:46.971366+0000", 2024-08-17T17:56:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6418999999999999e-05 2024-08-17T17:56:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:56:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:50:46.999026+0000", 2024-08-17T17:56:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.027660315000000001 2024-08-17T17:56:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:56:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:50:46.999097+0000", 2024-08-17T17:56:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0355e-05 2024-08-17T17:56:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:56:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.34890.0:2483 52.6 52:67fe8c99:::rbd_data.884a13e23c62.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 6c=[6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1262)", 2024-08-17T17:56:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:51:12.761024+0000", 2024-08-17T17:56:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 311.64654421500001, 2024-08-17T17:56:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.041585455, 2024-08-17T17:56:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:56:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:56:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:56:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.34890", 2024-08-17T17:56:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3439126247", 2024-08-17T17:56:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2483 2024-08-17T17:56:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:56:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:56:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:12.761024+0000", 2024-08-17T17:56:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:56:24.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:12.761024+0000", 2024-08-17T17:56:24.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:56:24.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:12.761023+0000", 2024-08-17T17:56:24.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T17:56:24.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:56:24.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:12.766932+0000", 2024-08-17T17:56:24.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059095349999999996 2024-08-17T17:56:24.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:56:24.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:12.766952+0000", 2024-08-17T17:56:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9519999999999999e-05 2024-08-17T17:56:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:56:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:12.766976+0000", 2024-08-17T17:56:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3941999999999999e-05 2024-08-17T17:56:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:56:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:12.767035+0000", 2024-08-17T17:56:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9670000000000003e-05 2024-08-17T17:56:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:56:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:12.767205+0000", 2024-08-17T17:56:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000169665 2024-08-17T17:56:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,6", 2024-08-17T17:56:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:12.767391+0000", 2024-08-17T17:56:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018578600000000001 2024-08-17T17:56:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:56:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:12.787610+0000", 2024-08-17T17:56:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020219032000000001 2024-08-17T17:56:24.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:56:24.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:12.788908+0000", 2024-08-17T17:56:24.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001298298 2024-08-17T17:56:24.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:56:24.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:12.802555+0000", 2024-08-17T17:56:24.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013647306999999999 2024-08-17T17:56:24.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:56:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:12.802588+0000", 2024-08-17T17:56:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2982999999999999e-05 2024-08-17T17:56:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:56:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:12.802610+0000", 2024-08-17T17:56:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1458e-05 2024-08-17T17:56:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:56:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.34890.0:2480 52.5 e1262/1153 52:a19e36b8:::rbd_data.884a13e23c62.0000000000000037:head v 1262'29, mlcod=1261'25)", 2024-08-17T17:56:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:51:12.768059+0000", 2024-08-17T17:56:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 311.63950971200001, 2024-08-17T17:56:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.046453514000000001, 2024-08-17T17:56:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:56:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:56:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:56:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:56:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:12.768059+0000", 2024-08-17T17:56:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:56:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:12.768059+0000", 2024-08-17T17:56:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:56:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:12.768059+0000", 2024-08-17T17:56:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T17:56:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:56:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:12.788975+0000", 2024-08-17T17:56:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020916615 2024-08-17T17:56:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:56:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:12.788992+0000", 2024-08-17T17:56:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6410999999999999e-05 2024-08-17T17:56:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:56:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:12.789000+0000", 2024-08-17T17:56:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1790000000000008e-06 2024-08-17T17:56:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:56:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:12.789119+0000", 2024-08-17T17:56:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119498 2024-08-17T17:56:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:56:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:12.789150+0000", 2024-08-17T17:56:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0911000000000003e-05 2024-08-17T17:56:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:56:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:12.814405+0000", 2024-08-17T17:56:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.025254427999999999 2024-08-17T17:56:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:56:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:12.814512+0000", 2024-08-17T17:56:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000107865 2024-08-17T17:56:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:56:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.34890.0:2591 52.6 52:67fe8c99:::rbd_data.884a13e23c62.0000000000000002:head [write 1220029~2974275] snapc 6e=[6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1264)", 2024-08-17T17:56:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:51:14.776354+0000", 2024-08-17T17:56:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 309.63121486699998, 2024-08-17T17:56:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.043802256999999997, 2024-08-17T17:56:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:56:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:56:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:56:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.34890", 2024-08-17T17:56:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3439126247", 2024-08-17T17:56:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2591 2024-08-17T17:56:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:56:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:56:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:14.776354+0000", 2024-08-17T17:56:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:56:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:14.776354+0000", 2024-08-17T17:56:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:56:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:14.776352+0000", 2024-08-17T17:56:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:56:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:56:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:14.785859+0000", 2024-08-17T17:56:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0095062319999999999 2024-08-17T17:56:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:56:24.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:14.785876+0000", 2024-08-17T17:56:24.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6994999999999999e-05 2024-08-17T17:56:24.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:56:24.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:14.785885+0000", 2024-08-17T17:56:24.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0359999999999995e-06 2024-08-17T17:56:24.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:56:24.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:14.785951+0000", 2024-08-17T17:56:24.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6780999999999996e-05 2024-08-17T17:56:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:56:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:14.786082+0000", 2024-08-17T17:56:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000130817 2024-08-17T17:56:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,6", 2024-08-17T17:56:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:14.786333+0000", 2024-08-17T17:56:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025047699999999998 2024-08-17T17:56:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:56:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:14.816841+0000", 2024-08-17T17:56:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.030507893000000001 2024-08-17T17:56:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:56:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:14.819322+0000", 2024-08-17T17:56:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024811049999999999 2024-08-17T17:56:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:56:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:14.820022+0000", 2024-08-17T17:56:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00070034199999999996 2024-08-17T17:56:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:56:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:14.820097+0000", 2024-08-17T17:56:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4833999999999995e-05 2024-08-17T17:56:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:56:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:14.820156+0000", 2024-08-17T17:56:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9292000000000001e-05 2024-08-17T17:56:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:56:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.34890.0:2592 52.d e1264/1153 52:b44ca37b:::rbd_data.884a13e23c62.0000000000000003:head v 1264'35, mlcod=1262'31)", 2024-08-17T17:56:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:51:14.780337+0000", 2024-08-17T17:56:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 309.62723165199998, 2024-08-17T17:56:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.039559288999999997, 2024-08-17T17:56:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:56:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:56:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:56:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:56:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:14.780337+0000", 2024-08-17T17:56:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:56:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:14.780337+0000", 2024-08-17T17:56:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:56:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:14.780323+0000", 2024-08-17T17:56:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999857 2024-08-17T17:56:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:56:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:14.792606+0000", 2024-08-17T17:56:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012283191000000001 2024-08-17T17:56:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:56:24.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:14.792626+0000", 2024-08-17T17:56:24.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9928999999999998e-05 2024-08-17T17:56:24.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:56:24.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:14.792634+0000", 2024-08-17T17:56:24.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6090000000000001e-06 2024-08-17T17:56:24.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:56:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:14.792692+0000", 2024-08-17T17:56:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.783e-05 2024-08-17T17:56:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:56:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:14.792719+0000", 2024-08-17T17:56:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7064999999999999e-05 2024-08-17T17:56:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:56:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:14.819836+0000", 2024-08-17T17:56:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.027116774999999999 2024-08-17T17:56:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:56:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:14.819896+0000", 2024-08-17T17:56:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0229999999999998e-05 2024-08-17T17:56:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:56:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.34890.0:2720 52.5 e1266/1153 52:a7dff434:::rbd_data.884a13e23c62.000000000000000f:head v 1266'34, mlcod=1263'31)", 2024-08-17T17:56:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:51:16.847607+0000", 2024-08-17T17:56:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 307.55996157800001, 2024-08-17T17:56:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.039889233000000003, 2024-08-17T17:56:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:56:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:56:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:56:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:56:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.847607+0000", 2024-08-17T17:56:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:56:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.847607+0000", 2024-08-17T17:56:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:56:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.847606+0000", 2024-08-17T17:56:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:56:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:56:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.861314+0000", 2024-08-17T17:56:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01370797 2024-08-17T17:56:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:56:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.861340+0000", 2024-08-17T17:56:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6622999999999999e-05 2024-08-17T17:56:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:56:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.861352+0000", 2024-08-17T17:56:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1396000000000001e-05 2024-08-17T17:56:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:56:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.861429+0000", 2024-08-17T17:56:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7109999999999999e-05 2024-08-17T17:56:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:56:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.861476+0000", 2024-08-17T17:56:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6830999999999999e-05 2024-08-17T17:56:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:56:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.887450+0000", 2024-08-17T17:56:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.025974621 2024-08-17T17:56:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:56:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.887496+0000", 2024-08-17T17:56:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6144999999999999e-05 2024-08-17T17:56:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:56:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.34890.0:2722 52.15 52:abae7bd7:::rbd_data.884a13e23c62.0000000000000050:head [write 0~2686597] snapc 70=[70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1266)", 2024-08-17T17:56:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:51:16.848730+0000", 2024-08-17T17:56:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 307.55883904799998, 2024-08-17T17:56:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.044485561999999999, 2024-08-17T17:56:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:56:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:56:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:56:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.34890", 2024-08-17T17:56:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3439126247", 2024-08-17T17:56:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2722 2024-08-17T17:56:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:56:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:56:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.848730+0000", 2024-08-17T17:56:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:56:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.848730+0000", 2024-08-17T17:56:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:56:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.848726+0000", 2024-08-17T17:56:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999962 2024-08-17T17:56:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:56:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.856226+0000", 2024-08-17T17:56:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0075005899999999997 2024-08-17T17:56:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:56:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.856252+0000", 2024-08-17T17:56:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5466e-05 2024-08-17T17:56:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:56:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.856265+0000", 2024-08-17T17:56:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2938999999999999e-05 2024-08-17T17:56:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:56:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.856316+0000", 2024-08-17T17:56:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1594000000000001e-05 2024-08-17T17:56:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:24.472 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:24.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:24.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:56:24.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.856434+0000", 2024-08-17T17:56:24.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000117652 2024-08-17T17:56:24.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4", 2024-08-17T17:56:24.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.856618+0000", 2024-08-17T17:56:24.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018438600000000001 2024-08-17T17:56:24.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:56:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.864052+0000", 2024-08-17T17:56:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0074334839999999998 2024-08-17T17:56:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:56:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.872579+0000", 2024-08-17T17:56:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0085267529999999998 2024-08-17T17:56:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:56:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.893123+0000", 2024-08-17T17:56:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020544732 2024-08-17T17:56:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:56:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.893170+0000", 2024-08-17T17:56:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6147999999999999e-05 2024-08-17T17:56:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:56:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.893215+0000", 2024-08-17T17:56:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5602000000000001e-05 2024-08-17T17:56:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:56:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.34890.0:2723 52.7 52:e0b2af58:::rbd_data.884a13e23c62.0000000000000004:head [write 168060~4026244] snapc 70=[70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1266)", 2024-08-17T17:56:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:51:16.856277+0000", 2024-08-17T17:56:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 307.55129175100001, 2024-08-17T17:56:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.039990518000000003, 2024-08-17T17:56:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:56:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:56:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:56:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.34890", 2024-08-17T17:56:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3439126247", 2024-08-17T17:56:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2723 2024-08-17T17:56:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:56:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:56:24.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.856277+0000", 2024-08-17T17:56:24.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:56:24.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.856277+0000", 2024-08-17T17:56:24.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:56:24.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.856276+0000", 2024-08-17T17:56:24.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:56:24.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:56:24.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.869179+0000", 2024-08-17T17:56:24.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012903135 2024-08-17T17:56:24.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:56:24.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.869198+0000", 2024-08-17T17:56:24.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9131999999999999e-05 2024-08-17T17:56:24.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:56:24.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.869205+0000", 2024-08-17T17:56:24.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9530000000000002e-06 2024-08-17T17:56:24.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:56:24.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.869246+0000", 2024-08-17T17:56:24.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1294000000000001e-05 2024-08-17T17:56:24.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:56:24.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.869327+0000", 2024-08-17T17:56:24.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0860999999999997e-05 2024-08-17T17:56:24.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,5", 2024-08-17T17:56:24.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.869452+0000", 2024-08-17T17:56:24.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000125487 2024-08-17T17:56:24.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:56:24.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.876383+0000", 2024-08-17T17:56:24.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069307040000000002 2024-08-17T17:56:24.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:56:24.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.890603+0000", 2024-08-17T17:56:24.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014220142999999999 2024-08-17T17:56:24.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:56:24.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.896145+0000", 2024-08-17T17:56:24.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055416889999999998 2024-08-17T17:56:24.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:56:24.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.896228+0000", 2024-08-17T17:56:24.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3176000000000005e-05 2024-08-17T17:56:24.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:56:24.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:16.896267+0000", 2024-08-17T17:56:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9371e-05 2024-08-17T17:56:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:56:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.34890.0:2938 52.15 52:af7543c2:::rbd_data.884a13e23c62.000000000000003d:head [set-alloc-hint object_size 4194304 write_size 4194304,write 652237~3542067] snapc 74=[74,73,72,71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1270)", 2024-08-17T17:56:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:51:20.828161+0000", 2024-08-17T17:56:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 303.57940734499999, 2024-08-17T17:56:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.039176067000000002, 2024-08-17T17:56:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:56:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:56:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:56:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.34890", 2024-08-17T17:56:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3439126247", 2024-08-17T17:56:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2938 2024-08-17T17:56:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:56:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:56:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:20.828161+0000", 2024-08-17T17:56:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:56:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:20.828161+0000", 2024-08-17T17:56:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:56:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:20.828160+0000", 2024-08-17T17:56:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:56:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:56:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:20.841972+0000", 2024-08-17T17:56:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0138119 2024-08-17T17:56:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:56:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:20.842007+0000", 2024-08-17T17:56:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5802e-05 2024-08-17T17:56:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:56:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:20.842021+0000", 2024-08-17T17:56:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3713000000000001e-05 2024-08-17T17:56:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:56:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:20.842122+0000", 2024-08-17T17:56:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000101307 2024-08-17T17:56:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:56:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:20.842365+0000", 2024-08-17T17:56:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024266899999999999 2024-08-17T17:56:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4", 2024-08-17T17:56:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:20.842626+0000", 2024-08-17T17:56:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026050500000000001 2024-08-17T17:56:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:56:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:20.853512+0000", 2024-08-17T17:56:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010886343999999999 2024-08-17T17:56:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:56:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:20.866062+0000", 2024-08-17T17:56:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012549543 2024-08-17T17:56:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:56:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:20.867267+0000", 2024-08-17T17:56:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001205228 2024-08-17T17:56:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:56:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:20.867307+0000", 2024-08-17T17:56:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9971000000000001e-05 2024-08-17T17:56:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:56:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:20.867337+0000", 2024-08-17T17:56:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0658999999999999e-05 2024-08-17T17:56:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:56:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.34890.0:3085 52.15 52:af7543c2:::rbd_data.884a13e23c62.000000000000003d:head [write 233752~3960552] snapc 77=[77,76,75,74,73,72,71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1273)", 2024-08-17T17:56:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:51:23.839541+0000", 2024-08-17T17:56:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 300.56802727399997, 2024-08-17T17:56:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038591254999999998, 2024-08-17T17:56:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:56:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:56:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:56:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.34890", 2024-08-17T17:56:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3439126247", 2024-08-17T17:56:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3085 2024-08-17T17:56:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:56:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:56:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:23.839541+0000", 2024-08-17T17:56:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:56:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:23.839541+0000", 2024-08-17T17:56:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:56:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:23.839538+0000", 2024-08-17T17:56:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999967 2024-08-17T17:56:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:56:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:23.850544+0000", 2024-08-17T17:56:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011005549 2024-08-17T17:56:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:56:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:23.850558+0000", 2024-08-17T17:56:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4409e-05 2024-08-17T17:56:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:56:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:23.850575+0000", 2024-08-17T17:56:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7025999999999999e-05 2024-08-17T17:56:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:56:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:23.850643+0000", 2024-08-17T17:56:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7967999999999997e-05 2024-08-17T17:56:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:56:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:23.850739+0000", 2024-08-17T17:56:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5991999999999998e-05 2024-08-17T17:56:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4", 2024-08-17T17:56:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:23.850946+0000", 2024-08-17T17:56:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020708799999999999 2024-08-17T17:56:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:56:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:23.866006+0000", 2024-08-17T17:56:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01505974 2024-08-17T17:56:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:56:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:23.876084+0000", 2024-08-17T17:56:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010078006 2024-08-17T17:56:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:56:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:23.878019+0000", 2024-08-17T17:56:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001935151 2024-08-17T17:56:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:56:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:23.878089+0000", 2024-08-17T17:56:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9729999999999998e-05 2024-08-17T17:56:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:56:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:23.878133+0000", 2024-08-17T17:56:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3893999999999997e-05 2024-08-17T17:56:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:56:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.34890.0:3088 52.5 e1273/1153 52:a7dff434:::rbd_data.884a13e23c62.000000000000000f:head v 1273'45, mlcod=1270'41)", 2024-08-17T17:56:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:51:23.844929+0000", 2024-08-17T17:56:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 300.562639558, 2024-08-17T17:56:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.051925840000000001, 2024-08-17T17:56:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:56:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:56:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:56:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:56:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:23.844929+0000", 2024-08-17T17:56:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:56:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:23.844929+0000", 2024-08-17T17:56:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:56:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:23.844929+0000", 2024-08-17T17:56:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T17:56:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:56:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:23.863146+0000", 2024-08-17T17:56:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018217125000000001 2024-08-17T17:56:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:56:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:23.863173+0000", 2024-08-17T17:56:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6970000000000001e-05 2024-08-17T17:56:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:56:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:23.863190+0000", 2024-08-17T17:56:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7155000000000001e-05 2024-08-17T17:56:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:56:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:23.863268+0000", 2024-08-17T17:56:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8004999999999996e-05 2024-08-17T17:56:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:56:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:23.863305+0000", 2024-08-17T17:56:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7435999999999999e-05 2024-08-17T17:56:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:56:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:23.896783+0000", 2024-08-17T17:56:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.033477938999999998 2024-08-17T17:56:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:56:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:23.896855+0000", 2024-08-17T17:56:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1637000000000005e-05 2024-08-17T17:56:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:56:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.34890.0:3207 52.9 52:977e6342:::rbd_data.884a13e23c62.0000000000000063:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~3460317] snapc 79=[79,78,77,76,75,74,73,72,71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1275)", 2024-08-17T17:56:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:51:25.859039+0000", 2024-08-17T17:56:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 298.54852948500002, 2024-08-17T17:56:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.040899171999999998, 2024-08-17T17:56:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:56:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:56:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:56:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.34890", 2024-08-17T17:56:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3439126247", 2024-08-17T17:56:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3207 2024-08-17T17:56:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:56:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:56:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:25.859039+0000", 2024-08-17T17:56:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:56:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:25.859039+0000", 2024-08-17T17:56:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:56:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:25.859035+0000", 2024-08-17T17:56:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999962 2024-08-17T17:56:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:56:24.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:25.868676+0000", 2024-08-17T17:56:24.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0096405229999999998 2024-08-17T17:56:24.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:56:24.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:25.868721+0000", 2024-08-17T17:56:24.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5238999999999998e-05 2024-08-17T17:56:24.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:56:24.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:25.868734+0000", 2024-08-17T17:56:24.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2520999999999999e-05 2024-08-17T17:56:24.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:56:24.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:25.868821+0000", 2024-08-17T17:56:24.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7348999999999993e-05 2024-08-17T17:56:24.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:56:24.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:25.869004+0000", 2024-08-17T17:56:24.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018320700000000001 2024-08-17T17:56:24.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:56:24.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:25.869336+0000", 2024-08-17T17:56:24.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033147299999999999 2024-08-17T17:56:24.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:56:24.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:25.878806+0000", 2024-08-17T17:56:24.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0094707130000000004 2024-08-17T17:56:24.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:56:24.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:25.898842+0000", 2024-08-17T17:56:24.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020035225 2024-08-17T17:56:24.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:56:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:25.899873+0000", 2024-08-17T17:56:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010318840000000001 2024-08-17T17:56:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:56:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:25.899912+0000", 2024-08-17T17:56:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9042000000000003e-05 2024-08-17T17:56:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:56:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:51:25.899938+0000", 2024-08-17T17:56:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5897999999999998e-05 2024-08-17T17:56:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:56:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.35091.0:197535 53.16 e1348/1347 53:6c43c907:::benchmark_data_smithi136_235573_object73249:head v 1348'8194, mlcod=1319'4245)", 2024-08-17T17:56:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:56:16.039940+0000", 2024-08-17T17:56:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.3676285289999992, 2024-08-17T17:56:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034208332000000001, 2024-08-17T17:56:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:56:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:56:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:56:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:56:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:16.039940+0000", 2024-08-17T17:56:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:56:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:16.039940+0000", 2024-08-17T17:56:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:56:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:16.039938+0000", 2024-08-17T17:56:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T17:56:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:56:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:16.039949+0000", 2024-08-17T17:56:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0808999999999999e-05 2024-08-17T17:56:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:56:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:16.039971+0000", 2024-08-17T17:56:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1903999999999999e-05 2024-08-17T17:56:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:56:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:16.039980+0000", 2024-08-17T17:56:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8789999999999992e-06 2024-08-17T17:56:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:56:24.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:16.040158+0000", 2024-08-17T17:56:24.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017844700000000001 2024-08-17T17:56:24.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:56:24.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:16.040214+0000", 2024-08-17T17:56:24.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5883e-05 2024-08-17T17:56:24.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:56:24.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:16.074013+0000", 2024-08-17T17:56:24.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.033798419000000003 2024-08-17T17:56:24.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:56:24.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:16.074148+0000", 2024-08-17T17:56:24.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013588200000000001 2024-08-17T17:56:24.510 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.510 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:56:24.510 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.35091.0:197539 53.16 e1348/1347 53:6c9ffcde:::benchmark_data_smithi136_235573_object73253:head v 1348'8195, mlcod=1319'4245)", 2024-08-17T17:56:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:56:16.045565+0000", 2024-08-17T17:56:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.3620034190000005, 2024-08-17T17:56:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.029063359, 2024-08-17T17:56:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:56:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:56:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:56:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:56:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:16.045565+0000", 2024-08-17T17:56:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:56:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:16.045565+0000", 2024-08-17T17:56:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:56:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:16.045564+0000", 2024-08-17T17:56:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:56:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:56:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:16.045573+0000", 2024-08-17T17:56:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4810000000000006e-06 2024-08-17T17:56:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:56:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:16.045590+0000", 2024-08-17T17:56:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6665999999999999e-05 2024-08-17T17:56:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:56:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:16.045597+0000", 2024-08-17T17:56:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.622e-06 2024-08-17T17:56:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:56:24.514 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:16.045929+0000", 2024-08-17T17:56:24.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033224900000000001 2024-08-17T17:56:24.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.514 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:56:24.514 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:16.049038+0000", 2024-08-17T17:56:24.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031082670000000001 2024-08-17T17:56:24.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.514 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:56:24.514 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:16.074154+0000", 2024-08-17T17:56:24.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.025115911000000001 2024-08-17T17:56:24.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:56:24.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:16.074629+0000", 2024-08-17T17:56:24.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00047499400000000001 2024-08-17T17:56:24.515 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.515 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:56:24.515 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.516 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.35091.0:200038 53.c e1348/1303 53:339cb278:::benchmark_data_smithi136_235573_object75752:head v 1348'12461, mlcod=1348'12459)", 2024-08-17T17:56:24.516 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:56:19.301525+0000", 2024-08-17T17:56:24.516 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.1060434380000004, 2024-08-17T17:56:24.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022062548000000001, 2024-08-17T17:56:24.516 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:56:24.516 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:56:24.516 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:56:24.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:56:24.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:19.301525+0000", 2024-08-17T17:56:24.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:56:24.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:19.301525+0000", 2024-08-17T17:56:24.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:56:24.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:19.301524+0000", 2024-08-17T17:56:24.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:56:24.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:56:24.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:19.301535+0000", 2024-08-17T17:56:24.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0968000000000001e-05 2024-08-17T17:56:24.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:56:24.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:19.301557+0000", 2024-08-17T17:56:24.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2464e-05 2024-08-17T17:56:24.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:56:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:19.301566+0000", 2024-08-17T17:56:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0350000000000007e-06 2024-08-17T17:56:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:56:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:19.301713+0000", 2024-08-17T17:56:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000146298 2024-08-17T17:56:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:56:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:19.301739+0000", 2024-08-17T17:56:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6613000000000001e-05 2024-08-17T17:56:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:56:24.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:19.323559+0000", 2024-08-17T17:56:24.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021820020999999998 2024-08-17T17:56:24.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:56:24.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:19.323588+0000", 2024-08-17T17:56:24.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8421000000000001e-05 2024-08-17T17:56:24.520 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.520 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:56:24.520 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.35091.0:201580 53.16 e1349/1347 53:6c623184:::benchmark_data_smithi136_235573_object77294:head v 1349'8306, mlcod=1319'4357)", 2024-08-17T17:56:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:56:21.345588+0000", 2024-08-17T17:56:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.061980894, 2024-08-17T17:56:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021474521, 2024-08-17T17:56:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:56:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:56:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:56:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:56:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:21.345588+0000", 2024-08-17T17:56:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:56:24.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:21.345588+0000", 2024-08-17T17:56:24.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:56:24.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:56:24.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:21.345586+0000", 2024-08-17T17:56:24.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:56:24.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:56:24.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:21.345596+0000", 2024-08-17T17:56:24.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7550000000000006e-06 2024-08-17T17:56:24.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:56:24.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:21.345619+0000", 2024-08-17T17:56:24.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2997999999999999e-05 2024-08-17T17:56:24.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:56:24.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:21.345627+0000", 2024-08-17T17:56:24.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2209999999999997e-06 2024-08-17T17:56:24.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:56:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:21.345721+0000", 2024-08-17T17:56:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4264000000000004e-05 2024-08-17T17:56:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:56:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:21.345753+0000", 2024-08-17T17:56:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1884999999999997e-05 2024-08-17T17:56:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:56:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:21.366986+0000", 2024-08-17T17:56:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021232228999999998 2024-08-17T17:56:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:56:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:56:24.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:56:24.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:56:21.367062+0000", 2024-08-17T17:56:24.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6809000000000003e-05 2024-08-17T17:56:24.525 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.525 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:56:24.525 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.525 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:56:24.525 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:56:24.525 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T17:56:24.573 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:24.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:24.617 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:24.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:24.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:24.741 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:24.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:24.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:24.843 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:24.887 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:24.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:24.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:24.986 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:24 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 7037/140508 objects degraded (5.008%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:24.989 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:25.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:25.023 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:25.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:25.124 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:25.164 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:25.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:25.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:25.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:24 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 7037/140508 objects degraded (5.008%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:25.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:24 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 7037/140508 objects degraded (5.008%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:25.267 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:25.307 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:25.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:25.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:25.313 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-17T17:56:25.409 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:25.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:25.455 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:25.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:25.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:25.582 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:25.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:25.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:25.684 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:25.722 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:25.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:25.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:25.824 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:25.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:25.864 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:25.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:25.965 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:26.001 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:26.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:26.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:26.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:25 smithi165 ceph-mon[105517]: pgmap v2216: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 753 op/s; 7037/140508 objects degraded (5.008%); 8788/140508 objects misplaced (6.254%); 4.5 MiB/s, 72 objects/s recovering 2024-08-17T17:56:26.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:26.136 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:26.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:26.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:26.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:25 smithi136 ceph-mon[109861]: pgmap v2216: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 753 op/s; 7037/140508 objects degraded (5.008%); 8788/140508 objects misplaced (6.254%); 4.5 MiB/s, 72 objects/s recovering 2024-08-17T17:56:26.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:25 smithi136 ceph-mon[115078]: pgmap v2216: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 753 op/s; 7037/140508 objects degraded (5.008%); 8788/140508 objects misplaced (6.254%); 4.5 MiB/s, 72 objects/s recovering 2024-08-17T17:56:26.237 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:26.263 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:26.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:26.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:26.365 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:26.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:26.412 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:26.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:26.513 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:26.546 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:26.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:26.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:26.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:26.684 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:26.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:26.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:26.786 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:26.811 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:26.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:26.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:26.859 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:56:26.913 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:26.942 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:26.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:26.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:27.044 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:27.087 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:27.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:27.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:27.090 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:56:27.189 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:27.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:27.233 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:27.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:27.334 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:27.377 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:27.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:27.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:27.479 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:27.510 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:27.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:27.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:27.556 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:56:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:56:27] "GET /metrics HTTP/1.1" 200 36570 "" "Prometheus/2.51.0" 2024-08-17T17:56:27.612 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:27.641 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:27.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:27.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:27.742 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:27.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:27.778 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:27.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:27.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:27.919 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:27.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:27.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:27.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:27 smithi136 ceph-mon[109861]: pgmap v2217: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 759 op/s; 6820/133965 objects degraded (5.091%); 8104/133965 objects misplaced (6.049%); 4.7 MiB/s, 75 objects/s recovering 2024-08-17T17:56:27.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:27 smithi136 ceph-mon[115078]: pgmap v2217: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 759 op/s; 6820/133965 objects degraded (5.091%); 8104/133965 objects misplaced (6.049%); 4.7 MiB/s, 75 objects/s recovering 2024-08-17T17:56:28.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:27 smithi165 ceph-mon[105517]: pgmap v2217: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 759 op/s; 6820/133965 objects degraded (5.091%); 8104/133965 objects misplaced (6.049%); 4.7 MiB/s, 75 objects/s recovering 2024-08-17T17:56:28.021 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:28.060 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:28.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:28.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:28.162 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:28.189 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:56:28.192 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T17:56:28.197 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:28.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:28.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:28.270 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:56:28.301 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:28.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:28.360 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:28.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:28.461 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:28.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:28.492 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:28.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:28.593 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:28.632 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:28.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:28.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:28.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:28.757 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:28.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:28.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:28.859 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:28.898 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:28.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:28.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:29.000 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:29.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:28 smithi165 ceph-mon[105517]: from='client.45164 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:56:29.040 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:29.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:29.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:29.043 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:28 smithi136 ceph-mon[109861]: from='client.45164 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:56:29.043 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:28 smithi136 ceph-mon[115078]: from='client.45164 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:56:29.142 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:29.171 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:29.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:29.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:29.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:29.302 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:29.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:29.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:29.404 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:29.442 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:29.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:29.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:29.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:29.570 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:29.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:29.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:29.672 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:29.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:29.710 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:29.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:29.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:29.837 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:29.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:29.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:29.939 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:29.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:29.973 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:29.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:29 smithi165 ceph-mon[105517]: pgmap v2218: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 776 op/s; 6820/128721 objects degraded (5.298%); 8104/128721 objects misplaced (6.296%); 4.7 MiB/s, 75 objects/s recovering 2024-08-17T17:56:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:29 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 6820/128721 objects degraded (5.298%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:30.020 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:29 smithi136 ceph-mon[109861]: pgmap v2218: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 776 op/s; 6820/128721 objects degraded (5.298%); 8104/128721 objects misplaced (6.296%); 4.7 MiB/s, 75 objects/s recovering 2024-08-17T17:56:30.020 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:29 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 6820/128721 objects degraded (5.298%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:30.020 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:29 smithi136 ceph-mon[115078]: pgmap v2218: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 776 op/s; 6820/128721 objects degraded (5.298%); 8104/128721 objects misplaced (6.296%); 4.7 MiB/s, 75 objects/s recovering 2024-08-17T17:56:30.021 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:29 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 6820/128721 objects degraded (5.298%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:30.074 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:30.114 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:30.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:30.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:30.216 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:30.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:30.256 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:30.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:30.359 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:30.403 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:30.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:30.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:30.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:30.546 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:30.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:30.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:30.583 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":2217,"stamp":"2024-08-17T17:56:26.656288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2925303805,"num_objects":44655,"num_object_clones":0,"num_object_copies":133965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6820,"num_objects_misplaced":8104,"num_objects_unfound":0,"num_objects_dirty":44655,"num_whiteouts":0,"num_read":379,"num_read_kb":823,"num_write":166264,"num_write_kb":5528946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11189,"num_bytes_recovered":709625440,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":209451,"ondisk_log_size":209451,"up":297,"acting":294,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":308,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11087952,"kb_used_data":9271108,"kb_used_omap":295,"kb_used_meta":1816408,"kb_avail":738873264,"statfs":{"total":767960285184,"available":756606222336,"internally_reserved":0,"allocated":9493614592,"data_stored":9460727463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":302302,"internal_metadata":1860002594},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,9,3,6,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-497942528,"num_objects":-7598,"num_object_clones":0,"num_object_copies":-22794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-433,"num_objects_misplaced":-1298,"num_objects_unfound":0,"num_objects_dirty":-7598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7598,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":758,"num_bytes_recovered":49676288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003405"},"pg_stats":[{"pgid":"53.19","version":"1349'8126","reported_seq":8273,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:26.287240+0000","last_change":"2024-08-17T17:53:49.813139+0000","last_active":"2024-08-17T17:56:26.287240+0000","last_peered":"2024-08-17T17:56:26.287240+0000","last_clean":"2024-08-17T17:56:26.287240+0000","last_became_active":"2024-08-17T17:53:45.801709+0000","last_became_peered":"2024-08-17T17:53:45.801709+0000","last_unstale":"2024-08-17T17:56:26.287240+0000","last_undegraded":"2024-08-17T17:56:26.287240+0000","last_fullsized":"2024-08-17T17:56:26.287240+0000","mapping_epoch":1310,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":5,"last_scrub":"1311'3869","last_scrub_stamp":"2024-08-17T17:53:49.812836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:49.812836+0000","objects_scrubbed":1929,"log_size":8126,"log_dups_size":0,"ondisk_log_size":8126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:36.206484+0000","scrub_duration":554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85983232,"num_objects":1312,"num_object_clones":0,"num_object_copies":3936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4267,"num_write_kb":116800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.18","version":"1349'8314","reported_seq":8434,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:22.136270+0000","last_change":"2024-08-17T17:53:53.987643+0000","last_active":"2024-08-17T17:56:22.136270+0000","last_peered":"2024-08-17T17:56:22.136270+0000","last_clean":"2024-08-17T17:56:22.136270+0000","last_became_active":"2024-08-17T17:53:45.788000+0000","last_became_peered":"2024-08-17T17:53:45.788000+0000","last_unstale":"2024-08-17T17:56:22.136270+0000","last_undegraded":"2024-08-17T17:56:22.136270+0000","last_fullsized":"2024-08-17T17:56:22.136270+0000","mapping_epoch":1310,"log_start":"1307'656","ondisk_log_start":"1307'656","created":1310,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":5,"last_scrub":"1313'4093","last_scrub_stamp":"2024-08-17T17:53:53.987508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:53.987508+0000","objects_scrubbed":2233,"log_size":7658,"log_dups_size":0,"ondisk_log_size":7658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:14:47.493287+0000","scrub_duration":0.21921206200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94371840,"num_objects":1440,"num_object_clones":0,"num_object_copies":4320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4221,"num_write_kb":109696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"53.17","version":"1349'8430","reported_seq":8579,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:25.968997+0000","last_change":"2024-08-17T17:53:50.287990+0000","last_active":"2024-08-17T17:56:25.968997+0000","last_peered":"2024-08-17T17:56:25.968997+0000","last_clean":"2024-08-17T17:56:25.968997+0000","last_became_active":"2024-08-17T17:53:45.802369+0000","last_became_peered":"2024-08-17T17:53:45.802369+0000","last_unstale":"2024-08-17T17:56:25.968997+0000","last_undegraded":"2024-08-17T17:56:25.968997+0000","last_fullsized":"2024-08-17T17:56:25.968997+0000","mapping_epoch":1310,"log_start":"1307'1112","ondisk_log_start":"1307'1112","created":1310,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":5,"last_scrub":"1312'3969","last_scrub_stamp":"2024-08-17T17:53:50.287885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:50.287885+0000","objects_scrubbed":2090,"log_size":7318,"log_dups_size":0,"ondisk_log_size":7318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:29:18.190184+0000","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92405760,"num_objects":1410,"num_object_clones":0,"num_object_copies":4230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4474,"num_write_kb":121408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.16","version":"1349'8411","reported_seq":10714,"reported_epoch":1349,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:56:25.983569+0000","last_change":"2024-08-17T17:56:03.772184+0000","last_active":"2024-08-17T17:56:25.983569+0000","last_peered":"2024-08-17T17:56:25.983569+0000","last_clean":"2024-08-17T17:56:00.271068+0000","last_became_active":"2024-08-17T17:56:03.726122+0000","last_became_peered":"2024-08-17T17:56:03.726122+0000","last_unstale":"2024-08-17T17:56:25.983569+0000","last_undegraded":"2024-08-17T17:56:03.680764+0000","last_fullsized":"2024-08-17T17:56:03.528461+0000","mapping_epoch":1347,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":5,"last_scrub":"1311'3927","last_scrub_stamp":"2024-08-17T17:53:48.318097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:48.318097+0000","objects_scrubbed":2014,"log_size":8411,"log_dups_size":0,"ondisk_log_size":8411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91029504,"num_objects":1389,"num_object_clones":0,"num_object_copies":4167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6820,"num_objects_misplaced":3410,"num_objects_unfound":0,"num_objects_dirty":1389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4493,"num_write_kb":123776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":538,"num_bytes_recovered":35258368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[2,1],"avail_no_missing":["2","3","4"],"object_location_counts":[{"shards":"2,3,4","objects":1389}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"53.15","version":"1349'8192","reported_seq":10518,"reported_epoch":1349,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-17T17:56:26.287347+0000","last_change":"2024-08-17T17:56:01.744916+0000","last_active":"2024-08-17T17:56:26.287347+0000","last_peered":"2024-08-17T17:56:26.287347+0000","last_clean":"2024-08-17T17:55:58.529149+0000","last_became_active":"2024-08-17T17:56:01.715968+0000","last_became_peered":"2024-08-17T17:56:01.715968+0000","last_unstale":"2024-08-17T17:56:26.287347+0000","last_undegraded":"2024-08-17T17:56:26.287347+0000","last_fullsized":"2024-08-17T17:56:01.526888+0000","mapping_epoch":1345,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":5,"last_scrub":"1311'3700","last_scrub_stamp":"2024-08-17T17:53:47.592482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:47.592482+0000","objects_scrubbed":1931,"log_size":8192,"log_dups_size":0,"ondisk_log_size":8192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92995584,"num_objects":1419,"num_object_clones":0,"num_object_copies":4257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2502,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4506,"num_write_kb":127808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":726,"num_bytes_recovered":47579136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[0,4],"avail_no_missing":["0","1","3"],"object_location_counts":[{"shards":"0,1,3","objects":1251},{"shards":"0,4","objects":168}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.14","version":"1349'8185","reported_seq":10766,"reported_epoch":1349,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-17T17:56:26.030491+0000","last_change":"2024-08-17T17:55:59.811423+0000","last_active":"2024-08-17T17:56:26.030491+0000","last_peered":"2024-08-17T17:56:26.030491+0000","last_clean":"2024-08-17T17:55:57.307497+0000","last_became_active":"2024-08-17T17:55:59.771920+0000","last_became_peered":"2024-08-17T17:55:59.771920+0000","last_unstale":"2024-08-17T17:56:26.030491+0000","last_undegraded":"2024-08-17T17:56:26.030491+0000","last_fullsized":"2024-08-17T17:55:59.520008+0000","mapping_epoch":1343,"log_start":"1306'501","ondisk_log_start":"1306'501","created":1310,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":5,"last_scrub":"1311'3909","last_scrub_stamp":"2024-08-17T17:53:49.696328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:49.696328+0000","objects_scrubbed":1954,"log_size":7684,"log_dups_size":0,"ondisk_log_size":7684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86573056,"num_objects":1321,"num_object_clones":0,"num_object_copies":3963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2192,"num_objects_unfound":0,"num_objects_dirty":1321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4295,"num_write_kb":117184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":811,"num_bytes_recovered":53149696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[1,3],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":1096},{"shards":"1,3","objects":225}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.13","version":"1349'7998","reported_seq":17699,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:26.300376+0000","last_change":"2024-08-17T17:55:38.393304+0000","last_active":"2024-08-17T17:56:26.300376+0000","last_peered":"2024-08-17T17:56:26.300376+0000","last_clean":"2024-08-17T17:56:26.300376+0000","last_became_active":"2024-08-17T17:55:38.383298+0000","last_became_peered":"2024-08-17T17:55:38.383298+0000","last_unstale":"2024-08-17T17:56:26.300376+0000","last_undegraded":"2024-08-17T17:56:26.300376+0000","last_fullsized":"2024-08-17T17:56:26.300376+0000","mapping_epoch":1337,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1338,"parent":"0.0","parent_split_bits":5,"last_scrub":"1311'3687","last_scrub_stamp":"2024-08-17T17:53:48.035372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:48.035372+0000","objects_scrubbed":1888,"log_size":7998,"log_dups_size":0,"ondisk_log_size":7998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:04:23.248001+0000","scrub_duration":0.55651981800000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87621632,"num_objects":1337,"num_object_clones":0,"num_object_copies":4011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4323,"num_write_kb":120704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2386,"num_bytes_recovered":156368896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.12","version":"1349'8154","reported_seq":18085,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:21.959643+0000","last_change":"2024-08-17T17:56:01.548247+0000","last_active":"2024-08-17T17:56:21.959643+0000","last_peered":"2024-08-17T17:56:21.959643+0000","last_clean":"2024-08-17T17:56:21.959643+0000","last_became_active":"2024-08-17T17:56:01.535252+0000","last_became_peered":"2024-08-17T17:56:01.535252+0000","last_unstale":"2024-08-17T17:56:21.959643+0000","last_undegraded":"2024-08-17T17:56:21.959643+0000","last_fullsized":"2024-08-17T17:56:21.959643+0000","mapping_epoch":1345,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1346,"parent":"0.0","parent_split_bits":5,"last_scrub":"1312'4010","last_scrub_stamp":"2024-08-17T17:53:52.788196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:52.788196+0000","objects_scrubbed":2082,"log_size":8154,"log_dups_size":0,"ondisk_log_size":8154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:06:14.839811+0000","scrub_duration":1.1488786010000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91947008,"num_objects":1403,"num_object_clones":0,"num_object_copies":4209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4165,"num_write_kb":111552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2443,"num_bytes_recovered":160104448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.11","version":"1349'8246","reported_seq":14758,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:26.208508+0000","last_change":"2024-08-17T17:54:32.250770+0000","last_active":"2024-08-17T17:56:26.208508+0000","last_peered":"2024-08-17T17:56:26.208508+0000","last_clean":"2024-08-17T17:56:26.208508+0000","last_became_active":"2024-08-17T17:54:32.246225+0000","last_became_peered":"2024-08-17T17:54:32.246225+0000","last_unstale":"2024-08-17T17:56:26.208508+0000","last_undegraded":"2024-08-17T17:56:26.208508+0000","last_fullsized":"2024-08-17T17:56:26.208508+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"1312'3987","last_scrub_stamp":"2024-08-17T17:53:51.667290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:51.667290+0000","objects_scrubbed":2071,"log_size":8246,"log_dups_size":0,"ondisk_log_size":8246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:59:10.324504+0000","scrub_duration":708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89587712,"num_objects":1367,"num_object_clones":0,"num_object_copies":4101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4276,"num_write_kb":114304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":138805248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.10","version":"1349'8263","reported_seq":14876,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:26.221310+0000","last_change":"2024-08-17T17:56:23.553836+0000","last_active":"2024-08-17T17:56:26.221310+0000","last_peered":"2024-08-17T17:56:26.221310+0000","last_clean":"2024-08-17T17:56:26.221310+0000","last_became_active":"2024-08-17T17:55:57.520262+0000","last_became_peered":"2024-08-17T17:55:57.520262+0000","last_unstale":"2024-08-17T17:56:26.221310+0000","last_undegraded":"2024-08-17T17:56:26.221310+0000","last_fullsized":"2024-08-17T17:56:26.221310+0000","mapping_epoch":1341,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":8263,"log_dups_size":0,"ondisk_log_size":8263,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T17:56:28.527270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85819392,"num_objects":1310,"num_object_clones":0,"num_object_copies":3930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6408,"num_write_kb":246944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2161,"num_bytes_recovered":116588544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.f","version":"1349'12989","reported_seq":13044,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:21.954548+0000","last_change":"2024-08-17T17:53:10.489536+0000","last_active":"2024-08-17T17:56:21.954548+0000","last_peered":"2024-08-17T17:56:21.954548+0000","last_clean":"2024-08-17T17:56:21.954548+0000","last_became_active":"2024-08-17T17:53:10.489173+0000","last_became_peered":"2024-08-17T17:53:10.489173+0000","last_unstale":"2024-08-17T17:56:21.954548+0000","last_undegraded":"2024-08-17T17:56:21.954548+0000","last_fullsized":"2024-08-17T17:56:21.954548+0000","mapping_epoch":1303,"log_start":"1312'4357","ondisk_log_start":"1312'4357","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":8632,"log_dups_size":0,"ondisk_log_size":8632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:46:03.457908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196280320,"num_objects":2995,"num_object_clones":0,"num_object_copies":8985,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12989,"num_write_kb":511488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.e","version":"1349'12556","reported_seq":12612,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:21.981558+0000","last_change":"2024-08-17T17:53:10.488446+0000","last_active":"2024-08-17T17:56:21.981558+0000","last_peered":"2024-08-17T17:56:21.981558+0000","last_clean":"2024-08-17T17:56:21.981558+0000","last_became_active":"2024-08-17T17:53:10.488058+0000","last_became_peered":"2024-08-17T17:53:10.488058+0000","last_unstale":"2024-08-17T17:56:21.981558+0000","last_undegraded":"2024-08-17T17:56:21.981558+0000","last_fullsized":"2024-08-17T17:56:21.981558+0000","mapping_epoch":1303,"log_start":"1311'3908","ondisk_log_start":"1311'3908","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":8648,"log_dups_size":0,"ondisk_log_size":8648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:00:57.616106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191758336,"num_objects":2926,"num_object_clones":0,"num_object_copies":8778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12556,"num_write_kb":495424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.d","version":"1349'13109","reported_seq":13223,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:25.980135+0000","last_change":"2024-08-17T17:53:10.646982+0000","last_active":"2024-08-17T17:56:25.980135+0000","last_peered":"2024-08-17T17:56:25.980135+0000","last_clean":"2024-08-17T17:56:25.980135+0000","last_became_active":"2024-08-17T17:53:10.646591+0000","last_became_peered":"2024-08-17T17:53:10.646591+0000","last_unstale":"2024-08-17T17:56:25.980135+0000","last_undegraded":"2024-08-17T17:56:25.980135+0000","last_fullsized":"2024-08-17T17:56:25.980135+0000","mapping_epoch":1303,"log_start":"1328'7073","ondisk_log_start":"1328'7073","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":6036,"log_dups_size":0,"ondisk_log_size":6036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:06:34.821249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175505432,"num_objects":2679,"num_object_clones":0,"num_object_copies":8037,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2679,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13109,"num_write_kb":505153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.c","version":"1349'12589","reported_seq":12645,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:22.067389+0000","last_change":"2024-08-17T17:53:10.559850+0000","last_active":"2024-08-17T17:56:22.067389+0000","last_peered":"2024-08-17T17:56:22.067389+0000","last_clean":"2024-08-17T17:56:22.067389+0000","last_became_active":"2024-08-17T17:53:10.559588+0000","last_became_peered":"2024-08-17T17:53:10.559588+0000","last_unstale":"2024-08-17T17:56:22.067389+0000","last_undegraded":"2024-08-17T17:56:22.067389+0000","last_fullsized":"2024-08-17T17:56:22.067389+0000","mapping_epoch":1303,"log_start":"1308'2900","ondisk_log_start":"1308'2900","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":9689,"log_dups_size":0,"ondisk_log_size":9689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:16:20.553763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192217088,"num_objects":2933,"num_object_clones":0,"num_object_copies":8799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12589,"num_write_kb":496704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"53.b","version":"1349'12545","reported_seq":12601,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:22.117936+0000","last_change":"2024-08-17T17:53:10.490102+0000","last_active":"2024-08-17T17:56:22.117936+0000","last_peered":"2024-08-17T17:56:22.117936+0000","last_clean":"2024-08-17T17:56:22.117936+0000","last_became_active":"2024-08-17T17:53:10.489717+0000","last_became_peered":"2024-08-17T17:53:10.489717+0000","last_unstale":"2024-08-17T17:56:22.117936+0000","last_undegraded":"2024-08-17T17:56:22.117936+0000","last_fullsized":"2024-08-17T17:56:22.117936+0000","mapping_epoch":1303,"log_start":"1308'2800","ondisk_log_start":"1308'2800","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":9745,"log_dups_size":0,"ondisk_log_size":9745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:50:43.448878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188547072,"num_objects":2877,"num_object_clones":0,"num_object_copies":8631,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12545,"num_write_kb":493504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"53.a","version":"1349'12919","reported_seq":13031,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:25.511847+0000","last_change":"2024-08-17T17:53:10.489596+0000","last_active":"2024-08-17T17:56:25.511847+0000","last_peered":"2024-08-17T17:56:25.511847+0000","last_clean":"2024-08-17T17:56:25.511847+0000","last_became_active":"2024-08-17T17:53:10.489230+0000","last_became_peered":"2024-08-17T17:53:10.489230+0000","last_unstale":"2024-08-17T17:56:25.511847+0000","last_undegraded":"2024-08-17T17:56:25.511847+0000","last_fullsized":"2024-08-17T17:56:25.511847+0000","mapping_epoch":1303,"log_start":"1325'5558","ondisk_log_start":"1325'5558","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":7361,"log_dups_size":0,"ondisk_log_size":7361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:50.891777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179109888,"num_objects":2733,"num_object_clones":0,"num_object_copies":8199,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12919,"num_write_kb":500864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"53.9","version":"1349'8393","reported_seq":8541,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:26.285653+0000","last_change":"2024-08-17T17:53:48.914243+0000","last_active":"2024-08-17T17:56:26.285653+0000","last_peered":"2024-08-17T17:56:26.285653+0000","last_clean":"2024-08-17T17:56:26.285653+0000","last_became_active":"2024-08-17T17:53:45.787626+0000","last_became_peered":"2024-08-17T17:53:45.787626+0000","last_unstale":"2024-08-17T17:56:26.285653+0000","last_undegraded":"2024-08-17T17:56:26.285653+0000","last_fullsized":"2024-08-17T17:56:26.285653+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3875","last_scrub_stamp":"2024-08-17T17:53:48.913906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:48.913906+0000","objects_scrubbed":2075,"log_size":8393,"log_dups_size":0,"ondisk_log_size":8393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:19:31.071940+0000","scrub_duration":692,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86376448,"num_objects":1318,"num_object_clones":0,"num_object_copies":3954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":120768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.8","version":"1349'8255","reported_seq":8379,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:21.973258+0000","last_change":"2024-08-17T17:53:50.732592+0000","last_active":"2024-08-17T17:56:21.973258+0000","last_peered":"2024-08-17T17:56:21.973258+0000","last_clean":"2024-08-17T17:56:21.973258+0000","last_became_active":"2024-08-17T17:53:45.777386+0000","last_became_peered":"2024-08-17T17:53:45.777386+0000","last_unstale":"2024-08-17T17:56:21.973258+0000","last_undegraded":"2024-08-17T17:56:21.973258+0000","last_fullsized":"2024-08-17T17:56:21.973258+0000","mapping_epoch":1303,"log_start":"1306'325","ondisk_log_start":"1306'325","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1312'4030","last_scrub_stamp":"2024-08-17T17:53:50.732148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:50.732148+0000","objects_scrubbed":2054,"log_size":7930,"log_dups_size":0,"ondisk_log_size":7930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:42:36.666939+0000","scrub_duration":0.96559516700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93454336,"num_objects":1426,"num_object_clones":0,"num_object_copies":4278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4242,"num_write_kb":115648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"53.7","version":"1349'8264","reported_seq":8411,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:26.014245+0000","last_change":"2024-08-17T17:53:47.346759+0000","last_active":"2024-08-17T17:56:26.014245+0000","last_peered":"2024-08-17T17:56:26.014245+0000","last_clean":"2024-08-17T17:56:26.014245+0000","last_became_active":"2024-08-17T17:53:45.779914+0000","last_became_peered":"2024-08-17T17:53:45.779914+0000","last_unstale":"2024-08-17T17:56:26.014245+0000","last_undegraded":"2024-08-17T17:56:26.014245+0000","last_fullsized":"2024-08-17T17:56:26.014245+0000","mapping_epoch":1303,"log_start":"1307'692","ondisk_log_start":"1307'692","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3847","last_scrub_stamp":"2024-08-17T17:53:47.346625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:47.346625+0000","objects_scrubbed":1924,"log_size":7572,"log_dups_size":0,"ondisk_log_size":7572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:44:59.835054+0000","scrub_duration":430,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86638592,"num_objects":1322,"num_object_clones":0,"num_object_copies":3966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4422,"num_write_kb":122240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.6","version":"1349'8450","reported_seq":8600,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:25.515155+0000","last_change":"2024-08-17T17:53:47.249481+0000","last_active":"2024-08-17T17:56:25.515155+0000","last_peered":"2024-08-17T17:56:25.515155+0000","last_clean":"2024-08-17T17:56:25.515155+0000","last_became_active":"2024-08-17T17:53:45.758286+0000","last_became_peered":"2024-08-17T17:53:45.758286+0000","last_unstale":"2024-08-17T17:56:25.515155+0000","last_undegraded":"2024-08-17T17:56:25.515155+0000","last_fullsized":"2024-08-17T17:56:25.515155+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3922","last_scrub_stamp":"2024-08-17T17:53:47.248989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:47.248989+0000","objects_scrubbed":1980,"log_size":8450,"log_dups_size":0,"ondisk_log_size":8450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:43:54.442535+0000","scrub_duration":555,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92864512,"num_objects":1417,"num_object_clones":0,"num_object_copies":4251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4537,"num_write_kb":127168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"53.5","version":"1349'7988","reported_seq":8134,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:26.284223+0000","last_change":"2024-08-17T17:53:46.758091+0000","last_active":"2024-08-17T17:56:26.284223+0000","last_peered":"2024-08-17T17:56:26.284223+0000","last_clean":"2024-08-17T17:56:26.284223+0000","last_became_active":"2024-08-17T17:53:45.771578+0000","last_became_peered":"2024-08-17T17:53:45.771578+0000","last_unstale":"2024-08-17T17:56:26.284223+0000","last_undegraded":"2024-08-17T17:56:26.284223+0000","last_fullsized":"2024-08-17T17:56:26.284223+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3661","last_scrub_stamp":"2024-08-17T17:53:46.757894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:46.757894+0000","objects_scrubbed":1802,"log_size":7988,"log_dups_size":0,"ondisk_log_size":7988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:27:03.974414+0000","scrub_duration":509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85983232,"num_objects":1312,"num_object_clones":0,"num_object_copies":3936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4332,"num_write_kb":122944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.4","version":"1349'8383","reported_seq":8531,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:25.999124+0000","last_change":"2024-08-17T17:53:46.426863+0000","last_active":"2024-08-17T17:56:25.999124+0000","last_peered":"2024-08-17T17:56:25.999124+0000","last_clean":"2024-08-17T17:56:25.999124+0000","last_became_active":"2024-08-17T17:53:45.764066+0000","last_became_peered":"2024-08-17T17:53:45.764066+0000","last_unstale":"2024-08-17T17:56:25.999124+0000","last_undegraded":"2024-08-17T17:56:25.999124+0000","last_fullsized":"2024-08-17T17:56:25.999124+0000","mapping_epoch":1303,"log_start":"1307'1132","ondisk_log_start":"1307'1132","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3774","last_scrub_stamp":"2024-08-17T17:53:46.426732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:46.426732+0000","objects_scrubbed":1958,"log_size":7251,"log_dups_size":0,"ondisk_log_size":7251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:49:46.480003+0000","scrub_duration":541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89522176,"num_objects":1366,"num_object_clones":0,"num_object_copies":4098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4624,"num_write_kb":129024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.3","version":"1349'7869","reported_seq":7983,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:21.890960+0000","last_change":"2024-08-17T17:53:46.968605+0000","last_active":"2024-08-17T17:56:21.890960+0000","last_peered":"2024-08-17T17:56:21.890960+0000","last_clean":"2024-08-17T17:56:21.890960+0000","last_became_active":"2024-08-17T17:53:45.764323+0000","last_became_peered":"2024-08-17T17:53:45.764323+0000","last_unstale":"2024-08-17T17:56:21.890960+0000","last_undegraded":"2024-08-17T17:56:21.890960+0000","last_fullsized":"2024-08-17T17:56:21.890960+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3658","last_scrub_stamp":"2024-08-17T17:53:46.967846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:46.967846+0000","objects_scrubbed":1838,"log_size":7869,"log_dups_size":0,"ondisk_log_size":7869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:09:25.913293+0000","scrub_duration":0.48413113800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92078080,"num_objects":1405,"num_object_clones":0,"num_object_copies":4215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4219,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.2","version":"1349'8230","reported_seq":8350,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:22.062731+0000","last_change":"2024-08-17T17:53:48.304744+0000","last_active":"2024-08-17T17:56:22.062731+0000","last_peered":"2024-08-17T17:56:22.062731+0000","last_clean":"2024-08-17T17:56:22.062731+0000","last_became_active":"2024-08-17T17:53:45.780923+0000","last_became_peered":"2024-08-17T17:53:45.780923+0000","last_unstale":"2024-08-17T17:56:22.062731+0000","last_undegraded":"2024-08-17T17:56:22.062731+0000","last_fullsized":"2024-08-17T17:56:22.062731+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3850","last_scrub_stamp":"2024-08-17T17:53:48.304649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:48.304649+0000","objects_scrubbed":2003,"log_size":8230,"log_dups_size":0,"ondisk_log_size":8230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:01.087711+0000","scrub_duration":0.70259554000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93257728,"num_objects":1423,"num_object_clones":0,"num_object_copies":4269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4398,"num_write_kb":122176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"53.1","version":"1349'8317","reported_seq":8464,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:26.013257+0000","last_change":"2024-08-17T17:53:53.382389+0000","last_active":"2024-08-17T17:56:26.013257+0000","last_peered":"2024-08-17T17:56:26.013257+0000","last_clean":"2024-08-17T17:56:26.013257+0000","last_became_active":"2024-08-17T17:53:45.782234+0000","last_became_peered":"2024-08-17T17:53:45.782234+0000","last_unstale":"2024-08-17T17:56:26.013257+0000","last_undegraded":"2024-08-17T17:56:26.013257+0000","last_fullsized":"2024-08-17T17:56:26.013257+0000","mapping_epoch":1303,"log_start":"1307'886","ondisk_log_start":"1307'886","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1313'4031","last_scrub_stamp":"2024-08-17T17:53:53.382269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:53.382269+0000","objects_scrubbed":2178,"log_size":7431,"log_dups_size":0,"ondisk_log_size":7431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:43.716721+0000","scrub_duration":410,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90177536,"num_objects":1376,"num_object_clones":0,"num_object_copies":4128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4288,"num_write_kb":111552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.0","version":"1349'8040","reported_seq":8157,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:22.057258+0000","last_change":"2024-08-17T17:53:47.447388+0000","last_active":"2024-08-17T17:56:22.057258+0000","last_peered":"2024-08-17T17:56:22.057258+0000","last_clean":"2024-08-17T17:56:22.057258+0000","last_became_active":"2024-08-17T17:53:45.741805+0000","last_became_peered":"2024-08-17T17:53:45.741805+0000","last_unstale":"2024-08-17T17:56:22.057258+0000","last_undegraded":"2024-08-17T17:56:22.057258+0000","last_fullsized":"2024-08-17T17:56:22.057258+0000","mapping_epoch":1303,"log_start":"1306'72","ondisk_log_start":"1306'72","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3799","last_scrub_stamp":"2024-08-17T17:53:47.447203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:47.447203+0000","objects_scrubbed":1904,"log_size":7968,"log_dups_size":0,"ondisk_log_size":7968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:52:36.467443+0000","scrub_duration":0.60964741,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92602368,"num_objects":1413,"num_object_clones":0,"num_object_copies":4239,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4257,"num_write_kb":120512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":2003,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.654961+0000","last_change":"2024-08-17T17:23:42.937520+0000","last_active":"2024-08-17T17:56:20.654961+0000","last_peered":"2024-08-17T17:56:20.654961+0000","last_clean":"2024-08-17T17:56:20.654961+0000","last_became_active":"2024-08-17T17:23:42.937174+0000","last_became_peered":"2024-08-17T17:23:42.937174+0000","last_unstale":"2024-08-17T17:56:20.654961+0000","last_undegraded":"2024-08-17T17:56:20.654961+0000","last_fullsized":"2024-08-17T17:56:20.654961+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:47:56.679612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1320,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:07.129956+0000","last_change":"2024-08-17T17:23:07.398676+0000","last_active":"2024-08-17T17:56:07.129956+0000","last_peered":"2024-08-17T17:56:07.129956+0000","last_clean":"2024-08-17T17:56:07.129956+0000","last_became_active":"2024-08-17T17:23:07.398369+0000","last_became_peered":"2024-08-17T17:23:07.398369+0000","last_unstale":"2024-08-17T17:56:07.129956+0000","last_undegraded":"2024-08-17T17:56:07.129956+0000","last_fullsized":"2024-08-17T17:56:07.129956+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:55:04.281412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":1987,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.641078+0000","last_change":"2024-08-17T17:24:17.476168+0000","last_active":"2024-08-17T17:56:20.641078+0000","last_peered":"2024-08-17T17:56:20.641078+0000","last_clean":"2024-08-17T17:56:20.641078+0000","last_became_active":"2024-08-17T17:24:17.465845+0000","last_became_peered":"2024-08-17T17:24:17.465845+0000","last_unstale":"2024-08-17T17:56:20.641078+0000","last_undegraded":"2024-08-17T17:56:20.641078+0000","last_fullsized":"2024-08-17T17:56:20.641078+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:15:01.494447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2003,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.638941+0000","last_change":"2024-08-17T17:24:17.461772+0000","last_active":"2024-08-17T17:56:20.638941+0000","last_peered":"2024-08-17T17:56:20.638941+0000","last_clean":"2024-08-17T17:56:20.638941+0000","last_became_active":"2024-08-17T17:24:17.460806+0000","last_became_peered":"2024-08-17T17:24:17.460806+0000","last_unstale":"2024-08-17T17:56:20.638941+0000","last_undegraded":"2024-08-17T17:56:20.638941+0000","last_fullsized":"2024-08-17T17:56:20.638941+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:01:16.453148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":1302,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.780709+0000","last_change":"2024-08-17T17:10:05.965176+0000","last_active":"2024-08-17T17:56:03.780709+0000","last_peered":"2024-08-17T17:56:03.780709+0000","last_clean":"2024-08-17T17:56:03.780709+0000","last_became_active":"2024-08-17T17:10:05.964861+0000","last_became_peered":"2024-08-17T17:10:05.964861+0000","last_unstale":"2024-08-17T17:56:03.780709+0000","last_undegraded":"2024-08-17T17:56:03.780709+0000","last_fullsized":"2024-08-17T17:56:03.780709+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:59:04.277926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1334,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.684451+0000","last_change":"2024-08-17T17:24:17.462527+0000","last_active":"2024-08-17T17:56:03.684451+0000","last_peered":"2024-08-17T17:56:03.684451+0000","last_clean":"2024-08-17T17:56:03.684451+0000","last_became_active":"2024-08-17T17:24:17.462240+0000","last_became_peered":"2024-08-17T17:24:17.462240+0000","last_unstale":"2024-08-17T17:56:03.684451+0000","last_undegraded":"2024-08-17T17:56:03.684451+0000","last_fullsized":"2024-08-17T17:56:03.684451+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:40:38.063717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":1317,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.780665+0000","last_change":"2024-08-17T17:23:07.391038+0000","last_active":"2024-08-17T17:56:03.780665+0000","last_peered":"2024-08-17T17:56:03.780665+0000","last_clean":"2024-08-17T17:56:03.780665+0000","last_became_active":"2024-08-17T17:23:07.390571+0000","last_became_peered":"2024-08-17T17:23:07.390571+0000","last_unstale":"2024-08-17T17:56:03.780665+0000","last_undegraded":"2024-08-17T17:56:03.780665+0000","last_fullsized":"2024-08-17T17:56:03.780665+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:18:41.791740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1334,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.684783+0000","last_change":"2024-08-17T17:24:17.458168+0000","last_active":"2024-08-17T17:56:03.684783+0000","last_peered":"2024-08-17T17:56:03.684783+0000","last_clean":"2024-08-17T17:56:03.684783+0000","last_became_active":"2024-08-17T17:24:17.457847+0000","last_became_peered":"2024-08-17T17:24:17.457847+0000","last_unstale":"2024-08-17T17:56:03.684783+0000","last_undegraded":"2024-08-17T17:56:03.684783+0000","last_fullsized":"2024-08-17T17:56:03.684783+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:28.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":1985,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.645460+0000","last_change":"2024-08-17T17:22:32.883756+0000","last_active":"2024-08-17T17:56:20.645460+0000","last_peered":"2024-08-17T17:56:20.645460+0000","last_clean":"2024-08-17T17:56:20.645460+0000","last_became_active":"2024-08-17T17:22:32.874867+0000","last_became_peered":"2024-08-17T17:22:32.874867+0000","last_unstale":"2024-08-17T17:56:20.645460+0000","last_undegraded":"2024-08-17T17:56:20.645460+0000","last_fullsized":"2024-08-17T17:56:20.645460+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:52:27.375231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":2005,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.648990+0000","last_change":"2024-08-17T17:24:17.456622+0000","last_active":"2024-08-17T17:56:20.648990+0000","last_peered":"2024-08-17T17:56:20.648990+0000","last_clean":"2024-08-17T17:56:20.648990+0000","last_became_active":"2024-08-17T17:24:17.455396+0000","last_became_peered":"2024-08-17T17:24:17.455396+0000","last_unstale":"2024-08-17T17:56:20.648990+0000","last_undegraded":"2024-08-17T17:56:20.648990+0000","last_fullsized":"2024-08-17T17:56:20.648990+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:09.564331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1336,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.684578+0000","last_change":"2024-08-17T17:23:42.930464+0000","last_active":"2024-08-17T17:56:03.684578+0000","last_peered":"2024-08-17T17:56:03.684578+0000","last_clean":"2024-08-17T17:56:03.684578+0000","last_became_active":"2024-08-17T17:23:42.929949+0000","last_became_peered":"2024-08-17T17:23:42.929949+0000","last_unstale":"2024-08-17T17:56:03.684578+0000","last_undegraded":"2024-08-17T17:56:03.684578+0000","last_fullsized":"2024-08-17T17:56:03.684578+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:48:27.092339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1989,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.639196+0000","last_change":"2024-08-17T17:24:17.452009+0000","last_active":"2024-08-17T17:56:20.639196+0000","last_peered":"2024-08-17T17:56:20.639196+0000","last_clean":"2024-08-17T17:56:20.639196+0000","last_became_active":"2024-08-17T17:24:17.451492+0000","last_became_peered":"2024-08-17T17:24:17.451492+0000","last_unstale":"2024-08-17T17:56:20.639196+0000","last_undegraded":"2024-08-17T17:56:20.639196+0000","last_fullsized":"2024-08-17T17:56:20.639196+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:21:24.699633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2001,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.657327+0000","last_change":"2024-08-17T17:24:17.452025+0000","last_active":"2024-08-17T17:56:20.657327+0000","last_peered":"2024-08-17T17:56:20.657327+0000","last_clean":"2024-08-17T17:56:20.657327+0000","last_became_active":"2024-08-17T17:24:17.448029+0000","last_became_peered":"2024-08-17T17:24:17.448029+0000","last_unstale":"2024-08-17T17:56:20.657327+0000","last_undegraded":"2024-08-17T17:56:20.657327+0000","last_fullsized":"2024-08-17T17:56:20.657327+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:45.089346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1996,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.646002+0000","last_change":"2024-08-17T17:24:17.469096+0000","last_active":"2024-08-17T17:56:20.646002+0000","last_peered":"2024-08-17T17:56:20.646002+0000","last_clean":"2024-08-17T17:56:20.646002+0000","last_became_active":"2024-08-17T17:24:17.468669+0000","last_became_peered":"2024-08-17T17:24:17.468669+0000","last_unstale":"2024-08-17T17:56:20.646002+0000","last_undegraded":"2024-08-17T17:56:20.646002+0000","last_fullsized":"2024-08-17T17:56:20.646002+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:03:39.427341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1990,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.639013+0000","last_change":"2024-08-17T17:24:17.453946+0000","last_active":"2024-08-17T17:56:20.639013+0000","last_peered":"2024-08-17T17:56:20.639013+0000","last_clean":"2024-08-17T17:56:20.639013+0000","last_became_active":"2024-08-17T17:24:17.445004+0000","last_became_peered":"2024-08-17T17:24:17.445004+0000","last_unstale":"2024-08-17T17:56:20.639013+0000","last_undegraded":"2024-08-17T17:56:20.639013+0000","last_fullsized":"2024-08-17T17:56:20.639013+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:00:19.366629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1320,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.526138+0000","last_change":"2024-08-17T17:23:07.402338+0000","last_active":"2024-08-17T17:56:03.526138+0000","last_peered":"2024-08-17T17:56:03.526138+0000","last_clean":"2024-08-17T17:56:03.526138+0000","last_became_active":"2024-08-17T17:23:07.401996+0000","last_became_peered":"2024-08-17T17:23:07.401996+0000","last_unstale":"2024-08-17T17:56:03.526138+0000","last_undegraded":"2024-08-17T17:56:03.526138+0000","last_fullsized":"2024-08-17T17:56:03.526138+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:39:21.566786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1974,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.639178+0000","last_change":"2024-08-17T17:24:17.442338+0000","last_active":"2024-08-17T17:56:20.639178+0000","last_peered":"2024-08-17T17:56:20.639178+0000","last_clean":"2024-08-17T17:56:20.639178+0000","last_became_active":"2024-08-17T17:24:17.441948+0000","last_became_peered":"2024-08-17T17:24:17.441948+0000","last_unstale":"2024-08-17T17:56:20.639178+0000","last_undegraded":"2024-08-17T17:56:20.639178+0000","last_fullsized":"2024-08-17T17:56:20.639178+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:17.891492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1993,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.655162+0000","last_change":"2024-08-17T17:24:17.459083+0000","last_active":"2024-08-17T17:56:20.655162+0000","last_peered":"2024-08-17T17:56:20.655162+0000","last_clean":"2024-08-17T17:56:20.655162+0000","last_became_active":"2024-08-17T17:24:17.458759+0000","last_became_peered":"2024-08-17T17:24:17.458759+0000","last_unstale":"2024-08-17T17:56:20.655162+0000","last_undegraded":"2024-08-17T17:56:20.655162+0000","last_fullsized":"2024-08-17T17:56:20.655162+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:11:59.301864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1933,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.639155+0000","last_change":"2024-08-17T17:24:17.468086+0000","last_active":"2024-08-17T17:56:20.639155+0000","last_peered":"2024-08-17T17:56:20.639155+0000","last_clean":"2024-08-17T17:56:20.639155+0000","last_became_active":"2024-08-17T17:24:17.467522+0000","last_became_peered":"2024-08-17T17:24:17.467522+0000","last_unstale":"2024-08-17T17:56:20.639155+0000","last_undegraded":"2024-08-17T17:56:20.639155+0000","last_fullsized":"2024-08-17T17:56:20.639155+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:52:55.249649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1316,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:07.112460+0000","last_change":"2024-08-17T17:23:07.392447+0000","last_active":"2024-08-17T17:56:07.112460+0000","last_peered":"2024-08-17T17:56:07.112460+0000","last_clean":"2024-08-17T17:56:07.112460+0000","last_became_active":"2024-08-17T17:23:07.391765+0000","last_became_peered":"2024-08-17T17:23:07.391765+0000","last_unstale":"2024-08-17T17:56:07.112460+0000","last_undegraded":"2024-08-17T17:56:07.112460+0000","last_fullsized":"2024-08-17T17:56:07.112460+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:59:18.602346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":86,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.639332+0000","last_change":"2024-08-17T17:53:58.839889+0000","last_active":"2024-08-17T17:56:20.639332+0000","last_peered":"2024-08-17T17:56:20.639332+0000","last_clean":"2024-08-17T17:56:20.639332+0000","last_became_active":"2024-08-17T17:53:58.839537+0000","last_became_peered":"2024-08-17T17:53:58.839537+0000","last_unstale":"2024-08-17T17:56:20.639332+0000","last_undegraded":"2024-08-17T17:56:20.639332+0000","last_fullsized":"2024-08-17T17:56:20.639332+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:47.169954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2001,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.661556+0000","last_change":"2024-08-17T17:23:42.936147+0000","last_active":"2024-08-17T17:56:20.661556+0000","last_peered":"2024-08-17T17:56:20.661556+0000","last_clean":"2024-08-17T17:56:20.661556+0000","last_became_active":"2024-08-17T17:23:42.935859+0000","last_became_peered":"2024-08-17T17:23:42.935859+0000","last_unstale":"2024-08-17T17:56:20.661556+0000","last_undegraded":"2024-08-17T17:56:20.661556+0000","last_fullsized":"2024-08-17T17:56:20.661556+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:47:36.838706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1329,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.684780+0000","last_change":"2024-08-17T17:10:05.969131+0000","last_active":"2024-08-17T17:56:03.684780+0000","last_peered":"2024-08-17T17:56:03.684780+0000","last_clean":"2024-08-17T17:56:03.684780+0000","last_became_active":"2024-08-17T17:10:05.968856+0000","last_became_peered":"2024-08-17T17:10:05.968856+0000","last_unstale":"2024-08-17T17:56:03.684780+0000","last_undegraded":"2024-08-17T17:56:03.684780+0000","last_fullsized":"2024-08-17T17:56:03.684780+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:25:49.550065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1320,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.684010+0000","last_change":"2024-08-17T17:23:07.455919+0000","last_active":"2024-08-17T17:56:03.684010+0000","last_peered":"2024-08-17T17:56:03.684010+0000","last_clean":"2024-08-17T17:56:03.684010+0000","last_became_active":"2024-08-17T17:23:07.455284+0000","last_became_peered":"2024-08-17T17:23:07.455284+0000","last_unstale":"2024-08-17T17:56:03.684010+0000","last_undegraded":"2024-08-17T17:56:03.684010+0000","last_fullsized":"2024-08-17T17:56:03.684010+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:37:46.260775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1308,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.684482+0000","last_change":"2024-08-17T17:10:06.979492+0000","last_active":"2024-08-17T17:56:03.684482+0000","last_peered":"2024-08-17T17:56:03.684482+0000","last_clean":"2024-08-17T17:56:03.684482+0000","last_became_active":"2024-08-17T17:10:06.979158+0000","last_became_peered":"2024-08-17T17:10:06.979158+0000","last_unstale":"2024-08-17T17:56:03.684482+0000","last_undegraded":"2024-08-17T17:56:03.684482+0000","last_fullsized":"2024-08-17T17:56:03.684482+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:12:42.261365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1332,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.526201+0000","last_change":"2024-08-17T17:23:42.935099+0000","last_active":"2024-08-17T17:56:03.526201+0000","last_peered":"2024-08-17T17:56:03.526201+0000","last_clean":"2024-08-17T17:56:03.526201+0000","last_became_active":"2024-08-17T17:23:42.934501+0000","last_became_peered":"2024-08-17T17:23:42.934501+0000","last_unstale":"2024-08-17T17:56:03.526201+0000","last_undegraded":"2024-08-17T17:56:03.526201+0000","last_fullsized":"2024-08-17T17:56:03.526201+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:15:19.095713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":1999,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.645885+0000","last_change":"2024-08-17T17:23:07.436763+0000","last_active":"2024-08-17T17:56:20.645885+0000","last_peered":"2024-08-17T17:56:20.645885+0000","last_clean":"2024-08-17T17:56:20.645885+0000","last_became_active":"2024-08-17T17:23:07.434720+0000","last_became_peered":"2024-08-17T17:23:07.434720+0000","last_unstale":"2024-08-17T17:56:20.645885+0000","last_undegraded":"2024-08-17T17:56:20.645885+0000","last_fullsized":"2024-08-17T17:56:20.645885+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:10:20.594115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1970,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.639128+0000","last_change":"2024-08-17T17:24:17.442329+0000","last_active":"2024-08-17T17:56:20.639128+0000","last_peered":"2024-08-17T17:56:20.639128+0000","last_clean":"2024-08-17T17:56:20.639128+0000","last_became_active":"2024-08-17T17:24:17.441666+0000","last_became_peered":"2024-08-17T17:24:17.441666+0000","last_unstale":"2024-08-17T17:56:20.639128+0000","last_undegraded":"2024-08-17T17:56:20.639128+0000","last_fullsized":"2024-08-17T17:56:20.639128+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:43:06.033194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1983,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.657016+0000","last_change":"2024-08-17T17:23:07.428296+0000","last_active":"2024-08-17T17:56:20.657016+0000","last_peered":"2024-08-17T17:56:20.657016+0000","last_clean":"2024-08-17T17:56:20.657016+0000","last_became_active":"2024-08-17T17:23:07.428010+0000","last_became_peered":"2024-08-17T17:23:07.428010+0000","last_unstale":"2024-08-17T17:56:20.657016+0000","last_undegraded":"2024-08-17T17:56:20.657016+0000","last_fullsized":"2024-08-17T17:56:20.657016+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:45.459212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":1970,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.650963+0000","last_change":"2024-08-17T17:24:17.481368+0000","last_active":"2024-08-17T17:56:20.650963+0000","last_peered":"2024-08-17T17:56:20.650963+0000","last_clean":"2024-08-17T17:56:20.650963+0000","last_became_active":"2024-08-17T17:24:17.477509+0000","last_became_peered":"2024-08-17T17:24:17.477509+0000","last_unstale":"2024-08-17T17:56:20.650963+0000","last_undegraded":"2024-08-17T17:56:20.650963+0000","last_fullsized":"2024-08-17T17:56:20.650963+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:05:35.558004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1985,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.645605+0000","last_change":"2024-08-17T17:22:32.884274+0000","last_active":"2024-08-17T17:56:20.645605+0000","last_peered":"2024-08-17T17:56:20.645605+0000","last_clean":"2024-08-17T17:56:20.645605+0000","last_became_active":"2024-08-17T17:22:32.874667+0000","last_became_peered":"2024-08-17T17:22:32.874667+0000","last_unstale":"2024-08-17T17:56:20.645605+0000","last_undegraded":"2024-08-17T17:56:20.645605+0000","last_fullsized":"2024-08-17T17:56:20.645605+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:22:05.834587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1329,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.531597+0000","last_change":"2024-08-17T17:22:32.870569+0000","last_active":"2024-08-17T17:56:03.531597+0000","last_peered":"2024-08-17T17:56:03.531597+0000","last_clean":"2024-08-17T17:56:03.531597+0000","last_became_active":"2024-08-17T17:22:32.869335+0000","last_became_peered":"2024-08-17T17:22:32.869335+0000","last_unstale":"2024-08-17T17:56:03.531597+0000","last_undegraded":"2024-08-17T17:56:03.531597+0000","last_fullsized":"2024-08-17T17:56:03.531597+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:24.969174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2004,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.663210+0000","last_change":"2024-08-17T17:23:42.935825+0000","last_active":"2024-08-17T17:56:20.663210+0000","last_peered":"2024-08-17T17:56:20.663210+0000","last_clean":"2024-08-17T17:56:20.663210+0000","last_became_active":"2024-08-17T17:23:42.935150+0000","last_became_peered":"2024-08-17T17:23:42.935150+0000","last_unstale":"2024-08-17T17:56:20.663210+0000","last_undegraded":"2024-08-17T17:56:20.663210+0000","last_fullsized":"2024-08-17T17:56:20.663210+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:57:19.577644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1992,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.661412+0000","last_change":"2024-08-17T17:23:07.431586+0000","last_active":"2024-08-17T17:56:20.661412+0000","last_peered":"2024-08-17T17:56:20.661412+0000","last_clean":"2024-08-17T17:56:20.661412+0000","last_became_active":"2024-08-17T17:23:07.431301+0000","last_became_peered":"2024-08-17T17:23:07.431301+0000","last_unstale":"2024-08-17T17:56:20.661412+0000","last_undegraded":"2024-08-17T17:56:20.661412+0000","last_fullsized":"2024-08-17T17:56:20.661412+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:28:18.706489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2020,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.662966+0000","last_change":"2024-08-17T17:23:07.420550+0000","last_active":"2024-08-17T17:56:20.662966+0000","last_peered":"2024-08-17T17:56:20.662966+0000","last_clean":"2024-08-17T17:56:20.662966+0000","last_became_active":"2024-08-17T17:23:07.420223+0000","last_became_peered":"2024-08-17T17:23:07.420223+0000","last_unstale":"2024-08-17T17:56:20.662966+0000","last_undegraded":"2024-08-17T17:56:20.662966+0000","last_fullsized":"2024-08-17T17:56:20.662966+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:33:46.996371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1334,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:07.112439+0000","last_change":"2024-08-17T17:24:17.426267+0000","last_active":"2024-08-17T17:56:07.112439+0000","last_peered":"2024-08-17T17:56:07.112439+0000","last_clean":"2024-08-17T17:56:07.112439+0000","last_became_active":"2024-08-17T17:24:17.425925+0000","last_became_peered":"2024-08-17T17:24:17.425925+0000","last_unstale":"2024-08-17T17:56:07.112439+0000","last_undegraded":"2024-08-17T17:56:07.112439+0000","last_fullsized":"2024-08-17T17:56:07.112439+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:24:25.017856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1329,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.780535+0000","last_change":"2024-08-17T17:24:17.474138+0000","last_active":"2024-08-17T17:56:03.780535+0000","last_peered":"2024-08-17T17:56:03.780535+0000","last_clean":"2024-08-17T17:56:03.780535+0000","last_became_active":"2024-08-17T17:24:17.473708+0000","last_became_peered":"2024-08-17T17:24:17.473708+0000","last_unstale":"2024-08-17T17:56:03.780535+0000","last_undegraded":"2024-08-17T17:56:03.780535+0000","last_fullsized":"2024-08-17T17:56:03.780535+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:10:57.768067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1324,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.684860+0000","last_change":"2024-08-17T17:22:32.871533+0000","last_active":"2024-08-17T17:56:03.684860+0000","last_peered":"2024-08-17T17:56:03.684860+0000","last_clean":"2024-08-17T17:56:03.684860+0000","last_became_active":"2024-08-17T17:22:32.868226+0000","last_became_peered":"2024-08-17T17:22:32.868226+0000","last_unstale":"2024-08-17T17:56:03.684860+0000","last_undegraded":"2024-08-17T17:56:03.684860+0000","last_fullsized":"2024-08-17T17:56:03.684860+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:00:02.629455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":1333,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.780681+0000","last_change":"2024-08-17T17:23:07.391021+0000","last_active":"2024-08-17T17:56:03.780681+0000","last_peered":"2024-08-17T17:56:03.780681+0000","last_clean":"2024-08-17T17:56:03.780681+0000","last_became_active":"2024-08-17T17:23:07.390551+0000","last_became_peered":"2024-08-17T17:23:07.390551+0000","last_unstale":"2024-08-17T17:56:03.780681+0000","last_undegraded":"2024-08-17T17:56:03.780681+0000","last_fullsized":"2024-08-17T17:56:03.780681+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:06.239013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001023532,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2004,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.659031+0000","last_change":"2024-08-17T17:23:07.406362+0000","last_active":"2024-08-17T17:56:20.659031+0000","last_peered":"2024-08-17T17:56:20.659031+0000","last_clean":"2024-08-17T17:56:20.659031+0000","last_became_active":"2024-08-17T17:23:07.406071+0000","last_became_peered":"2024-08-17T17:23:07.406071+0000","last_unstale":"2024-08-17T17:56:20.659031+0000","last_undegraded":"2024-08-17T17:56:20.659031+0000","last_fullsized":"2024-08-17T17:56:20.659031+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:45:47.383432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1984,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.656852+0000","last_change":"2024-08-17T17:23:42.910246+0000","last_active":"2024-08-17T17:56:20.656852+0000","last_peered":"2024-08-17T17:56:20.656852+0000","last_clean":"2024-08-17T17:56:20.656852+0000","last_became_active":"2024-08-17T17:23:42.909929+0000","last_became_peered":"2024-08-17T17:23:42.909929+0000","last_unstale":"2024-08-17T17:56:20.656852+0000","last_undegraded":"2024-08-17T17:56:20.656852+0000","last_fullsized":"2024-08-17T17:56:20.656852+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:14:45.210558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2013,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.647438+0000","last_change":"2024-08-17T17:24:17.438891+0000","last_active":"2024-08-17T17:56:20.647438+0000","last_peered":"2024-08-17T17:56:20.647438+0000","last_clean":"2024-08-17T17:56:20.647438+0000","last_became_active":"2024-08-17T17:24:17.437714+0000","last_became_peered":"2024-08-17T17:24:17.437714+0000","last_unstale":"2024-08-17T17:56:20.647438+0000","last_undegraded":"2024-08-17T17:56:20.647438+0000","last_fullsized":"2024-08-17T17:56:20.647438+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:58.825017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2006,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.646085+0000","last_change":"2024-08-17T17:23:42.908634+0000","last_active":"2024-08-17T17:56:20.646085+0000","last_peered":"2024-08-17T17:56:20.646085+0000","last_clean":"2024-08-17T17:56:20.646085+0000","last_became_active":"2024-08-17T17:23:42.907675+0000","last_became_peered":"2024-08-17T17:23:42.907675+0000","last_unstale":"2024-08-17T17:56:20.646085+0000","last_undegraded":"2024-08-17T17:56:20.646085+0000","last_fullsized":"2024-08-17T17:56:20.646085+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:03:37.612222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1996,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.645094+0000","last_change":"2024-08-17T17:24:17.457559+0000","last_active":"2024-08-17T17:56:20.645094+0000","last_peered":"2024-08-17T17:56:20.645094+0000","last_clean":"2024-08-17T17:56:20.645094+0000","last_became_active":"2024-08-17T17:24:17.457262+0000","last_became_peered":"2024-08-17T17:24:17.457262+0000","last_unstale":"2024-08-17T17:56:20.645094+0000","last_undegraded":"2024-08-17T17:56:20.645094+0000","last_fullsized":"2024-08-17T17:56:20.645094+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:53:38.137277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":718,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.657107+0000","last_change":"2024-08-17T17:53:58.844594+0000","last_active":"2024-08-17T17:56:20.657107+0000","last_peered":"2024-08-17T17:56:20.657107+0000","last_clean":"2024-08-17T17:56:20.657107+0000","last_became_active":"2024-08-17T17:53:58.844085+0000","last_became_peered":"2024-08-17T17:53:58.844085+0000","last_unstale":"2024-08-17T17:56:20.657107+0000","last_undegraded":"2024-08-17T17:56:20.657107+0000","last_fullsized":"2024-08-17T17:56:20.657107+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:58:50.195797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1305,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:06.981637+0000","last_change":"2024-08-17T17:10:06.995023+0000","last_active":"2024-08-17T17:56:06.981637+0000","last_peered":"2024-08-17T17:56:06.981637+0000","last_clean":"2024-08-17T17:56:06.981637+0000","last_became_active":"2024-08-17T17:10:06.994757+0000","last_became_peered":"2024-08-17T17:10:06.994757+0000","last_unstale":"2024-08-17T17:56:06.981637+0000","last_undegraded":"2024-08-17T17:56:06.981637+0000","last_fullsized":"2024-08-17T17:56:06.981637+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:31:52.163518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1936,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.639548+0000","last_change":"2024-08-17T17:24:17.468443+0000","last_active":"2024-08-17T17:56:20.639548+0000","last_peered":"2024-08-17T17:56:20.639548+0000","last_clean":"2024-08-17T17:56:20.639548+0000","last_became_active":"2024-08-17T17:24:17.467394+0000","last_became_peered":"2024-08-17T17:24:17.467394+0000","last_unstale":"2024-08-17T17:56:20.639548+0000","last_undegraded":"2024-08-17T17:56:20.639548+0000","last_fullsized":"2024-08-17T17:56:20.639548+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:13.024253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":1332,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.780558+0000","last_change":"2024-08-17T17:23:42.920552+0000","last_active":"2024-08-17T17:56:03.780558+0000","last_peered":"2024-08-17T17:56:03.780558+0000","last_clean":"2024-08-17T17:56:03.780558+0000","last_became_active":"2024-08-17T17:23:42.915824+0000","last_became_peered":"2024-08-17T17:23:42.915824+0000","last_unstale":"2024-08-17T17:56:03.780558+0000","last_undegraded":"2024-08-17T17:56:03.780558+0000","last_fullsized":"2024-08-17T17:56:03.780558+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:25:18.769504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041737399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":688,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.648855+0000","last_change":"2024-08-17T17:53:58.844782+0000","last_active":"2024-08-17T17:56:20.648855+0000","last_peered":"2024-08-17T17:56:20.648855+0000","last_clean":"2024-08-17T17:56:20.648855+0000","last_became_active":"2024-08-17T17:53:58.843133+0000","last_became_peered":"2024-08-17T17:53:58.843133+0000","last_unstale":"2024-08-17T17:56:20.648855+0000","last_undegraded":"2024-08-17T17:56:20.648855+0000","last_fullsized":"2024-08-17T17:56:20.648855+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T17:06:21.174471+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:30:14.411310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1982,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.658967+0000","last_change":"2024-08-17T17:24:17.455710+0000","last_active":"2024-08-17T17:56:20.658967+0000","last_peered":"2024-08-17T17:56:20.658967+0000","last_clean":"2024-08-17T17:56:20.658967+0000","last_became_active":"2024-08-17T17:24:17.452989+0000","last_became_peered":"2024-08-17T17:24:17.452989+0000","last_unstale":"2024-08-17T17:56:20.658967+0000","last_undegraded":"2024-08-17T17:56:20.658967+0000","last_fullsized":"2024-08-17T17:56:20.658967+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:58:07.894968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1976,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.638730+0000","last_change":"2024-08-17T17:24:17.453639+0000","last_active":"2024-08-17T17:56:20.638730+0000","last_peered":"2024-08-17T17:56:20.638730+0000","last_clean":"2024-08-17T17:56:20.638730+0000","last_became_active":"2024-08-17T17:24:17.443638+0000","last_became_peered":"2024-08-17T17:24:17.443638+0000","last_unstale":"2024-08-17T17:56:20.638730+0000","last_undegraded":"2024-08-17T17:56:20.638730+0000","last_fullsized":"2024-08-17T17:56:20.638730+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:42.107479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1321,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.684058+0000","last_change":"2024-08-17T17:23:07.451866+0000","last_active":"2024-08-17T17:56:03.684058+0000","last_peered":"2024-08-17T17:56:03.684058+0000","last_clean":"2024-08-17T17:56:03.684058+0000","last_became_active":"2024-08-17T17:23:07.451553+0000","last_became_peered":"2024-08-17T17:23:07.451553+0000","last_unstale":"2024-08-17T17:56:03.684058+0000","last_undegraded":"2024-08-17T17:56:03.684058+0000","last_fullsized":"2024-08-17T17:56:03.684058+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:37:23.930651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1323,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.684548+0000","last_change":"2024-08-17T17:23:42.915851+0000","last_active":"2024-08-17T17:56:03.684548+0000","last_peered":"2024-08-17T17:56:03.684548+0000","last_clean":"2024-08-17T17:56:03.684548+0000","last_became_active":"2024-08-17T17:23:42.915552+0000","last_became_peered":"2024-08-17T17:23:42.915552+0000","last_unstale":"2024-08-17T17:56:03.684548+0000","last_undegraded":"2024-08-17T17:56:03.684548+0000","last_fullsized":"2024-08-17T17:56:03.684548+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:45.197876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1336,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.684944+0000","last_change":"2024-08-17T17:23:42.930803+0000","last_active":"2024-08-17T17:56:03.684944+0000","last_peered":"2024-08-17T17:56:03.684944+0000","last_clean":"2024-08-17T17:56:03.684944+0000","last_became_active":"2024-08-17T17:23:42.930458+0000","last_became_peered":"2024-08-17T17:23:42.930458+0000","last_unstale":"2024-08-17T17:56:03.684944+0000","last_undegraded":"2024-08-17T17:56:03.684944+0000","last_fullsized":"2024-08-17T17:56:03.684944+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:40.816706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1320,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.525527+0000","last_change":"2024-08-17T17:23:07.401624+0000","last_active":"2024-08-17T17:56:03.525527+0000","last_peered":"2024-08-17T17:56:03.525527+0000","last_clean":"2024-08-17T17:56:03.525527+0000","last_became_active":"2024-08-17T17:23:07.400222+0000","last_became_peered":"2024-08-17T17:23:07.400222+0000","last_unstale":"2024-08-17T17:56:03.525527+0000","last_undegraded":"2024-08-17T17:56:03.525527+0000","last_fullsized":"2024-08-17T17:56:03.525527+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:07.069989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1321,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.684712+0000","last_change":"2024-08-17T17:24:17.470809+0000","last_active":"2024-08-17T17:56:03.684712+0000","last_peered":"2024-08-17T17:56:03.684712+0000","last_clean":"2024-08-17T17:56:03.684712+0000","last_became_active":"2024-08-17T17:24:17.469753+0000","last_became_peered":"2024-08-17T17:24:17.469753+0000","last_unstale":"2024-08-17T17:56:03.684712+0000","last_undegraded":"2024-08-17T17:56:03.684712+0000","last_fullsized":"2024-08-17T17:56:03.684712+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:00.049425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":1970,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.638926+0000","last_change":"2024-08-17T17:24:17.461258+0000","last_active":"2024-08-17T17:56:20.638926+0000","last_peered":"2024-08-17T17:56:20.638926+0000","last_clean":"2024-08-17T17:56:20.638926+0000","last_became_active":"2024-08-17T17:24:17.460895+0000","last_became_peered":"2024-08-17T17:24:17.460895+0000","last_unstale":"2024-08-17T17:56:20.638926+0000","last_undegraded":"2024-08-17T17:56:20.638926+0000","last_fullsized":"2024-08-17T17:56:20.638926+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:43:24.208259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2009,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.645413+0000","last_change":"2024-08-17T17:23:42.901638+0000","last_active":"2024-08-17T17:56:20.645413+0000","last_peered":"2024-08-17T17:56:20.645413+0000","last_clean":"2024-08-17T17:56:20.645413+0000","last_became_active":"2024-08-17T17:23:42.901335+0000","last_became_peered":"2024-08-17T17:23:42.901335+0000","last_unstale":"2024-08-17T17:56:20.645413+0000","last_undegraded":"2024-08-17T17:56:20.645413+0000","last_fullsized":"2024-08-17T17:56:20.645413+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:25:35.332839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1317,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.780675+0000","last_change":"2024-08-17T17:24:17.473913+0000","last_active":"2024-08-17T17:56:03.780675+0000","last_peered":"2024-08-17T17:56:03.780675+0000","last_clean":"2024-08-17T17:56:03.780675+0000","last_became_active":"2024-08-17T17:24:17.473195+0000","last_became_peered":"2024-08-17T17:24:17.473195+0000","last_unstale":"2024-08-17T17:56:03.780675+0000","last_undegraded":"2024-08-17T17:56:03.780675+0000","last_fullsized":"2024-08-17T17:56:03.780675+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:12.998033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1319,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.525344+0000","last_change":"2024-08-17T17:23:07.402010+0000","last_active":"2024-08-17T17:56:03.525344+0000","last_peered":"2024-08-17T17:56:03.525344+0000","last_clean":"2024-08-17T17:56:03.525344+0000","last_became_active":"2024-08-17T17:23:07.400849+0000","last_became_peered":"2024-08-17T17:23:07.400849+0000","last_unstale":"2024-08-17T17:56:03.525344+0000","last_undegraded":"2024-08-17T17:56:03.525344+0000","last_fullsized":"2024-08-17T17:56:03.525344+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:16:02.382257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":55,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:17.120886+0000","last_change":"2024-08-17T17:53:58.836811+0000","last_active":"2024-08-17T17:56:17.120886+0000","last_peered":"2024-08-17T17:56:17.120886+0000","last_clean":"2024-08-17T17:56:17.120886+0000","last_became_active":"2024-08-17T17:53:58.836139+0000","last_became_peered":"2024-08-17T17:53:58.836139+0000","last_unstale":"2024-08-17T17:56:17.120886+0000","last_undegraded":"2024-08-17T17:56:17.120886+0000","last_fullsized":"2024-08-17T17:56:17.120886+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:09:18.260164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1337,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.684403+0000","last_change":"2024-08-17T17:24:17.460203+0000","last_active":"2024-08-17T17:56:03.684403+0000","last_peered":"2024-08-17T17:56:03.684403+0000","last_clean":"2024-08-17T17:56:03.684403+0000","last_became_active":"2024-08-17T17:24:17.459682+0000","last_became_peered":"2024-08-17T17:24:17.459682+0000","last_unstale":"2024-08-17T17:56:03.684403+0000","last_undegraded":"2024-08-17T17:56:03.684403+0000","last_fullsized":"2024-08-17T17:56:03.684403+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:58.703700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":1317,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.531160+0000","last_change":"2024-08-17T17:23:07.454042+0000","last_active":"2024-08-17T17:56:03.531160+0000","last_peered":"2024-08-17T17:56:03.531160+0000","last_clean":"2024-08-17T17:56:03.531160+0000","last_became_active":"2024-08-17T17:23:07.453712+0000","last_became_peered":"2024-08-17T17:23:07.453712+0000","last_unstale":"2024-08-17T17:56:03.531160+0000","last_undegraded":"2024-08-17T17:56:03.531160+0000","last_fullsized":"2024-08-17T17:56:03.531160+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:03:07.643456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1302,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.780557+0000","last_change":"2024-08-17T17:10:06.994799+0000","last_active":"2024-08-17T17:56:03.780557+0000","last_peered":"2024-08-17T17:56:03.780557+0000","last_clean":"2024-08-17T17:56:03.780557+0000","last_became_active":"2024-08-17T17:10:06.994150+0000","last_became_peered":"2024-08-17T17:10:06.994150+0000","last_unstale":"2024-08-17T17:56:03.780557+0000","last_undegraded":"2024-08-17T17:56:03.780557+0000","last_fullsized":"2024-08-17T17:56:03.780557+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:50:47.293091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":1315,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.531703+0000","last_change":"2024-08-17T17:22:32.873334+0000","last_active":"2024-08-17T17:56:03.531703+0000","last_peered":"2024-08-17T17:56:03.531703+0000","last_clean":"2024-08-17T17:56:03.531703+0000","last_became_active":"2024-08-17T17:22:32.870147+0000","last_became_peered":"2024-08-17T17:22:32.870147+0000","last_unstale":"2024-08-17T17:56:03.531703+0000","last_undegraded":"2024-08-17T17:56:03.531703+0000","last_fullsized":"2024-08-17T17:56:03.531703+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:18:03.750592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1326,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.525520+0000","last_change":"2024-08-17T17:22:32.847989+0000","last_active":"2024-08-17T17:56:03.525520+0000","last_peered":"2024-08-17T17:56:03.525520+0000","last_clean":"2024-08-17T17:56:03.525520+0000","last_became_active":"2024-08-17T17:22:32.846593+0000","last_became_peered":"2024-08-17T17:22:32.846593+0000","last_unstale":"2024-08-17T17:56:03.525520+0000","last_undegraded":"2024-08-17T17:56:03.525520+0000","last_fullsized":"2024-08-17T17:56:03.525520+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:33:50.370124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1974,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.650882+0000","last_change":"2024-08-17T17:24:17.451528+0000","last_active":"2024-08-17T17:56:20.650882+0000","last_peered":"2024-08-17T17:56:20.650882+0000","last_clean":"2024-08-17T17:56:20.650882+0000","last_became_active":"2024-08-17T17:24:17.451246+0000","last_became_peered":"2024-08-17T17:24:17.451246+0000","last_unstale":"2024-08-17T17:56:20.650882+0000","last_undegraded":"2024-08-17T17:56:20.650882+0000","last_fullsized":"2024-08-17T17:56:20.650882+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:38:18.176559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":43,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.637743+0000","last_change":"2024-08-17T17:53:58.838279+0000","last_active":"2024-08-17T17:56:20.637743+0000","last_peered":"2024-08-17T17:56:20.637743+0000","last_clean":"2024-08-17T17:56:20.637743+0000","last_became_active":"2024-08-17T17:53:58.837978+0000","last_became_peered":"2024-08-17T17:53:58.837978+0000","last_unstale":"2024-08-17T17:56:20.637743+0000","last_undegraded":"2024-08-17T17:56:20.637743+0000","last_fullsized":"2024-08-17T17:56:20.637743+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:06.019142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1993,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.645835+0000","last_change":"2024-08-17T17:24:17.460371+0000","last_active":"2024-08-17T17:56:20.645835+0000","last_peered":"2024-08-17T17:56:20.645835+0000","last_clean":"2024-08-17T17:56:20.645835+0000","last_became_active":"2024-08-17T17:24:17.460074+0000","last_became_peered":"2024-08-17T17:24:17.460074+0000","last_unstale":"2024-08-17T17:56:20.645835+0000","last_undegraded":"2024-08-17T17:56:20.645835+0000","last_fullsized":"2024-08-17T17:56:20.645835+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:08:48.820168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1993,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.645139+0000","last_change":"2024-08-17T17:24:17.457942+0000","last_active":"2024-08-17T17:56:20.645139+0000","last_peered":"2024-08-17T17:56:20.645139+0000","last_clean":"2024-08-17T17:56:20.645139+0000","last_became_active":"2024-08-17T17:24:17.456475+0000","last_became_peered":"2024-08-17T17:24:17.456475+0000","last_unstale":"2024-08-17T17:56:20.645139+0000","last_undegraded":"2024-08-17T17:56:20.645139+0000","last_fullsized":"2024-08-17T17:56:20.645139+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:12.325657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1315,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.780819+0000","last_change":"2024-08-17T17:24:17.451287+0000","last_active":"2024-08-17T17:56:03.780819+0000","last_peered":"2024-08-17T17:56:03.780819+0000","last_clean":"2024-08-17T17:56:03.780819+0000","last_became_active":"2024-08-17T17:24:17.450951+0000","last_became_peered":"2024-08-17T17:24:17.450951+0000","last_unstale":"2024-08-17T17:56:03.780819+0000","last_undegraded":"2024-08-17T17:56:03.780819+0000","last_fullsized":"2024-08-17T17:56:03.780819+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:13:37.590108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":1316,"reported_epoch":1348,"state":"active+clean","last_fresh":"2024-08-17T17:56:03.531401+0000","last_change":"2024-08-17T17:23:42.907970+0000","last_active":"2024-08-17T17:56:03.531401+0000","last_peered":"2024-08-17T17:56:03.531401+0000","last_clean":"2024-08-17T17:56:03.531401+0000","last_became_active":"2024-08-17T17:23:42.907595+0000","last_became_peered":"2024-08-17T17:23:42.907595+0000","last_unstale":"2024-08-17T17:56:03.531401+0000","last_undegraded":"2024-08-17T17:56:03.531401+0000","last_fullsized":"2024-08-17T17:56:03.531401+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:11:46.130004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1974,"reported_epoch":1349,"state":"active+clean","last_fresh":"2024-08-17T17:56:20.644772+0000","last_change":"2024-08-17T17:24:17.486591+0000","last_active":"2024-08-17T17:56:20.644772+0000","last_peered":"2024-08-17T17:56:20.644772+0000","last_clean":"2024-08-17T17:56:20.644772+0000","last_became_active":"2024-08-17T17:24:17.486320+0000","last_became_peered":"2024-08-17T17:24:17.486320+0000","last_unstale":"2024-08-17T17:56:20.644772+0000","last_undegraded":"2024-08-17T17:56:20.644772+0000","last_fullsized":"2024-08-17T17:56:20.644772+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:03.249803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":53,"num_pg":26,"stat_sum":{"num_bytes":2924707864,"num_objects":44629,"num_object_clones":0,"num_object_copies":133887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6820,"num_objects_misplaced":8104,"num_objects_unfound":0,"num_objects_dirty":44629,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165985,"num_write_kb":5526497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11183,"num_bytes_recovered":707854336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9335091200,"data_stored":9335078984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":209243,"ondisk_log_size":209243,"up":78,"acting":75,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234564,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":454404,"kb_used_data":310968,"kb_used_omap":1,"kb_used_meta":143422,"kb_avail":93290748,"statfs":{"total":95995035648,"available":95529725952,"internally_reserved":0,"allocated":318431232,"data_stored":314307676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":146864587},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496202,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1076092,"kb_used_data":864560,"kb_used_omap":1,"kb_used_meta":211518,"kb_avail":92669060,"statfs":{"total":95995035648,"available":94893117440,"internally_reserved":0,"allocated":885309440,"data_stored":881193949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":216594887},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565444,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1251108,"kb_used_data":1074588,"kb_used_omap":72,"kb_used_meta":176439,"kb_avail":92494044,"statfs":{"total":95995035648,"available":94713901056,"internally_reserved":0,"allocated":1100378112,"data_stored":1096277814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":180674030},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":674,"seq":2894807957899,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1107820,"kb_used_data":917476,"kb_used_omap":74,"kb_used_meta":190261,"kb_avail":92637332,"statfs":{"total":95995035648,"available":94860627968,"internally_reserved":0,"allocated":939495424,"data_stored":935387484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76116,"internal_metadata":194827948},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743186834,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2111720,"kb_used_data":1800608,"kb_used_omap":66,"kb_used_meta":310973,"kb_avail":91633432,"statfs":{"total":95995035648,"available":93832634368,"internally_reserved":0,"allocated":1843822592,"data_stored":1839718151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":318437035},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448473,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1443628,"kb_used_data":1217380,"kb_used_omap":75,"kb_used_meta":226164,"kb_avail":92301524,"statfs":{"total":95995035648,"available":94516760576,"internally_reserved":0,"allocated":1246597120,"data_stored":1242489688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":231592127},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757838,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1548148,"kb_used_data":1340648,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":92197004,"statfs":{"total":95995035648,"available":94409732096,"internally_reserved":0,"allocated":1372823552,"data_stored":1368711559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":212466118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019475,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2095032,"kb_used_data":1744880,"kb_used_omap":1,"kb_used_meta":350142,"kb_avail":91650120,"statfs":{"total":95995035648,"available":93849722880,"internally_reserved":0,"allocated":1786757120,"data_stored":1782641142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":358545862},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1219297280,"data_stored":1219297280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1828065280,"data_stored":1828061208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":924323840,"data_stored":924319768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1078394880,"data_stored":1078394880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1775435776,"data_stored":1775435776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1343885312,"data_stored":1343881240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":861798400,"data_stored":861798400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":303890432,"data_stored":303890432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:56:30.588 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph status --format=json 2024-08-17T17:56:30.648 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:30.678 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:30.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:30.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:30.780 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:30.810 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:30.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:30.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:30.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:30.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:30.942 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:30.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:31.043 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:31.086 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:31.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:31.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:31.188 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:31.249 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:31.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:31.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:31.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:31.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:31.393 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:31.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:31.494 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:31.532 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:31.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:31.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:31.633 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:31.675 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:31.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:31.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:31.777 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:31.806 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:31.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:31.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:31.851 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:56:31.851 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T17:56:31.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:56:31.851 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:208746 53.13 53:c9abd666:::benchmark_data_smithi136_235573_object84460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1349)", 2024-08-17T17:56:31.851 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:56:30.355278+0000", 2024-08-17T17:56:31.851 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.01534834, 2024-08-17T17:56:31.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015387511, 2024-08-17T17:56:31.851 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:56:31.852 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:56:31.852 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:56:31.852 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:56:31.852 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:56:31.852 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:56:31.852 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 208746 2024-08-17T17:56:31.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:56:31.852 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:56:31.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:56:31.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:56:31.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:56:30.355278+0000", 2024-08-17T17:56:31.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:56:31.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:56:31.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:56:31.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:56:31.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:56:30.355278+0000", 2024-08-17T17:56:31.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:56:31.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:56:31.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:56:31.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:56:31.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:56:30.355280+0000", 2024-08-17T17:56:31.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.48e-06 2024-08-17T17:56:31.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:56:31.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:56:31.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:56:31.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:56:30.355284+0000", 2024-08-17T17:56:31.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7319999999999998e-06 2024-08-17T17:56:31.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:56:31.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:56:31.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:56:31.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:56:30.355297+0000", 2024-08-17T17:56:31.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2959000000000001e-05 2024-08-17T17:56:31.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:56:31.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:56:31.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:56:31.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:56:30.355305+0000", 2024-08-17T17:56:31.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1769999999999998e-06 2024-08-17T17:56:31.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:56:31.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:56:31.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:56:31.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:56:30.358298+0000", 2024-08-17T17:56:31.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002992303 2024-08-17T17:56:31.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:56:31.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:56:31.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:56:31.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:56:30.358393+0000", 2024-08-17T17:56:31.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4826000000000003e-05 2024-08-17T17:56:31.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:56:31.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:56:31.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,3", 2024-08-17T17:56:31.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:56:30.358855+0000", 2024-08-17T17:56:31.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00046229099999999998 2024-08-17T17:56:31.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:56:31.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:56:31.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:56:31.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:56:30.366145+0000", 2024-08-17T17:56:31.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0072900999999999999 2024-08-17T17:56:31.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:56:31.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:56:31.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:56:31.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:56:30.369674+0000", 2024-08-17T17:56:31.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035290360000000002 2024-08-17T17:56:31.856 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:56:31.856 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:56:31.856 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:56:31.856 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:56:31.856 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-17T17:56:31.856 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 1 2024-08-17T17:56:31.857 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:56:31.909 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:31.940 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:31.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:31.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:31 smithi165 ceph-mon[105517]: pgmap v2219: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 780 op/s; 6820/127542 objects degraded (5.347%); 8104/127542 objects misplaced (6.354%); 4.7 MiB/s, 75 objects/s recovering 2024-08-17T17:56:32.042 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:32.081 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:32.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:32.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:32.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:32.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:31 smithi136 ceph-mon[109861]: pgmap v2219: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 780 op/s; 6820/127542 objects degraded (5.347%); 8104/127542 objects misplaced (6.354%); 4.7 MiB/s, 75 objects/s recovering 2024-08-17T17:56:32.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:31 smithi136 ceph-mon[115078]: pgmap v2219: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 780 op/s; 6820/127542 objects degraded (5.347%); 8104/127542 objects misplaced (6.354%); 4.7 MiB/s, 75 objects/s recovering 2024-08-17T17:56:32.212 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:32.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:32.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:32.232 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:56:32.314 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:32.342 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:32.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:32.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:32.445 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:32.479 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:32.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:32.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:32.581 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:32.615 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:32.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:32.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:32.709 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-17T17:56:32.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:32.744 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:32.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:32.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:32.846 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:32.899 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:32.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:32.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:33.005 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:33.032 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:33.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:33.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:33.134 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:33.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:33.169 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:33.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:33.270 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:33.308 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:33.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:33.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:33.410 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:33.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:33.445 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:33.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:33.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:33.588 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:33.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:33.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:33.600 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T17:56:33.600 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:56:33.638 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T17:56:33.639 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:56:33.674 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T17:56:33.674 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:56:33.689 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:33.713 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T17:56:33.713 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:56:33.718 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:33.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:33.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:33.742 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T17:56:33.742 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:56:33.778 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T17:56:33.779 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:56:33.807 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T17:56:33.807 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:56:33.820 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:33.840 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T17:56:33.840 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:56:33.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:33.864 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:33.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:33.965 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:34.115 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:34.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:34.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:34.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:33 smithi136 ceph-mon[109861]: pgmap v2220: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 953 op/s; 6560/116667 objects degraded (5.623%); 7397/116667 objects misplaced (6.340%); 6.1 MiB/s, 97 objects/s recovering 2024-08-17T17:56:34.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:33 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3285782053' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:56:34.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:33 smithi136 ceph-mon[115078]: pgmap v2220: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 953 op/s; 6560/116667 objects degraded (5.623%); 7397/116667 objects misplaced (6.340%); 6.1 MiB/s, 97 objects/s recovering 2024-08-17T17:56:34.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:33 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3285782053' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:56:34.218 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:34.244 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:34.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:34.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:33 smithi165 ceph-mon[105517]: pgmap v2220: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 953 op/s; 6560/116667 objects degraded (5.623%); 7397/116667 objects misplaced (6.340%); 6.1 MiB/s, 97 objects/s recovering 2024-08-17T17:56:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:33 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3285782053' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:56:34.345 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:34.379 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:34.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:34.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:34.481 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:34.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:34.513 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:34.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:34.613 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:34.656 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:34.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:34.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:34.758 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:34.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:34.802 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:34.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:34.903 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:34.943 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:34.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:34.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:35.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:35.079 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:35.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:35.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:35.161 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:34 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 6560/116667 objects degraded (5.623%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:35.181 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:35.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:34 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 6560/116667 objects degraded (5.623%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:35.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:34 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 6560/116667 objects degraded (5.623%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:35.216 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:35.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:35.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:35.318 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:35.344 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:35.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:35.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:35.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:35.488 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:35.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:35.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:35.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:35.622 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:35.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:35.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:35.724 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:35.765 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:35.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:35.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:35.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:35.897 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:35.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:35.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:35.999 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:36.040 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:36.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:36.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:36.142 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:36.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:36.178 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:36.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:36.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:35 smithi136 ceph-mon[109861]: pgmap v2221: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 661 op/s; 6560/116667 objects degraded (5.623%); 7397/116667 objects misplaced (6.340%); 4.2 MiB/s, 67 objects/s recovering 2024-08-17T17:56:36.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:35 smithi136 ceph-mon[115078]: pgmap v2221: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 661 op/s; 6560/116667 objects degraded (5.623%); 7397/116667 objects misplaced (6.340%); 4.2 MiB/s, 67 objects/s recovering 2024-08-17T17:56:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:35 smithi165 ceph-mon[105517]: pgmap v2221: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 661 op/s; 6560/116667 objects degraded (5.623%); 7397/116667 objects misplaced (6.340%); 4.2 MiB/s, 67 objects/s recovering 2024-08-17T17:56:36.279 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:36.300 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:36.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:36.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:36.402 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:36.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:36.440 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:36.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:36.541 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:36.576 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:36.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:36.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:36.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:36.709 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:36.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:36.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:36.810 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:36.849 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:36.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:36.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:36.951 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:36.976 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:36.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:36.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:37.078 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:37.110 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:56:37.110 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:56:37.116 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:37.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:37.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:37.218 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:37.242 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:37.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:37.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:56:37.343 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:37.389 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:37.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:37.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:37.491 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:37.517 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:37.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:37.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:37.619 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:37.653 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:37.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:37.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:37.756 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:37.788 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:56:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:56:37] "GET /metrics HTTP/1.1" 200 36561 "" "Prometheus/2.51.0" 2024-08-17T17:56:37.791 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:37.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:37.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:37.893 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:37.936 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:37.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:37.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:38.037 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:38.067 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:38.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:38.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:38.169 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:38.194 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:38.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:38.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:38.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:37 smithi136 ceph-mon[109861]: pgmap v2222: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.2 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 851 op/s; 6294/109824 objects degraded (5.731%); 6653/109824 objects misplaced (6.058%); 6.5 MiB/s, 104 objects/s recovering 2024-08-17T17:56:38.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:37 smithi136 ceph-mon[115078]: pgmap v2222: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.2 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 851 op/s; 6294/109824 objects degraded (5.731%); 6653/109824 objects misplaced (6.058%); 6.5 MiB/s, 104 objects/s recovering 2024-08-17T17:56:38.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:37 smithi165 ceph-mon[105517]: pgmap v2222: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.2 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 851 op/s; 6294/109824 objects degraded (5.731%); 6653/109824 objects misplaced (6.058%); 6.5 MiB/s, 104 objects/s recovering 2024-08-17T17:56:38.297 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:38.324 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:38.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:38.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:38.426 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:38.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:38.464 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:38.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:38.565 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:38.607 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:38.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:38.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:38.708 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:38.747 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:38.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:38.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:38.850 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:38.871 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:38.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:38.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:38.973 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:39.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:39.012 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:39.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:39.114 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:39.156 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:39.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:39.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:39.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:39.299 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:39.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:39.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:39.401 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:39.440 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:39.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:39.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:39.542 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:39.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:39.583 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:39.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:39.684 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:39.721 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:39.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:39.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:39.808 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:56:39.823 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:39.851 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:39.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:39.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:39.866 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:56:39.953 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:40.000 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:40.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:40.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:40.102 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:40.123 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:39 smithi136 ceph-mon[109861]: pgmap v2223: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 6294/104730 objects degraded (6.010%); 6653/104730 objects misplaced (6.353%); 4.5 MiB/s, 71 objects/s recovering 2024-08-17T17:56:40.123 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:39 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 6294/104730 objects degraded (6.010%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:40.123 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:39 smithi136 ceph-mon[115078]: pgmap v2223: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 6294/104730 objects degraded (6.010%); 6653/104730 objects misplaced (6.353%); 4.5 MiB/s, 71 objects/s recovering 2024-08-17T17:56:40.123 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:39 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 6294/104730 objects degraded (6.010%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:40.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:40.162 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:40.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:40.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:39 smithi165 ceph-mon[105517]: pgmap v2223: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 6294/104730 objects degraded (6.010%); 6653/104730 objects misplaced (6.353%); 4.5 MiB/s, 71 objects/s recovering 2024-08-17T17:56:40.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:39 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 6294/104730 objects degraded (6.010%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:40.263 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:40.295 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:40.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:40.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:40.397 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:40.435 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:40.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:40.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:40.536 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:40.580 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:40.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:40.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:40.682 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:40.712 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:40.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:40.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:40.814 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:40.858 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:40.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:40.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:40.960 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:40.993 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:40.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:40.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:41.095 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:41.136 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:41.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:41.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:41.146 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:56:41.238 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:41.264 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:41.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:41.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:41.366 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:41.421 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:41.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:41.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:41.523 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:41.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:41.552 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:41.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:41.653 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:41.688 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:41.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:41.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:41.790 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:41.804 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T17:56:41.815 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:41.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:41.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:41.917 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:41.947 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:41.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:41.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:42.049 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:42.089 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:42.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:42.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:42.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:42.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:41 smithi136 ceph-mon[109861]: pgmap v2224: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 6294/104730 objects degraded (6.010%); 6653/104730 objects misplaced (6.353%); 4.5 MiB/s, 71 objects/s recovering 2024-08-17T17:56:42.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:41 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3707775140' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:56:42.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:41 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:56:42.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:41 smithi136 ceph-mon[115078]: pgmap v2224: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 6294/104730 objects degraded (6.010%); 6653/104730 objects misplaced (6.353%); 4.5 MiB/s, 71 objects/s recovering 2024-08-17T17:56:42.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:41 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3707775140' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:56:42.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:41 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:56:42.218 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:42.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:42.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:41 smithi165 ceph-mon[105517]: pgmap v2224: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 2.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 6294/104730 objects degraded (6.010%); 6653/104730 objects misplaced (6.353%); 4.5 MiB/s, 71 objects/s recovering 2024-08-17T17:56:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:41 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3707775140' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:56:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:41 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:56:42.320 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:42.359 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:42.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:42.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:42.460 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:42.496 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:42.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:42.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:42.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:42.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:42.635 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:42.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:42.736 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:42.772 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:42.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:42.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:42.874 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:42.912 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:42.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:42.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:43.014 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:43.054 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:43.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:43.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:43.156 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:43.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:42 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:56:43.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:42 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:56:43.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:42 smithi136 ceph-mon[109861]: osdmap e1350: 8 total, 8 up, 8 in 2024-08-17T17:56:43.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:42 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:56:43.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:42 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:56:43.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:42 smithi136 ceph-mon[115078]: osdmap e1350: 8 total, 8 up, 8 in 2024-08-17T17:56:43.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:43.198 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:43.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:43.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:42 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:56:43.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:42 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:56:43.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:42 smithi165 ceph-mon[105517]: osdmap e1350: 8 total, 8 up, 8 in 2024-08-17T17:56:43.298 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:43.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:43.332 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:43.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:43.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:43.464 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:43.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:43.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:43.566 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:43.603 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:43.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:43.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:43.705 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:43.742 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:43.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:43.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:43.844 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:43.884 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:43.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:43.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:43.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:44.029 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:44.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:44.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:44.130 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:44.175 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:44.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:44.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:44.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:43 smithi136 ceph-mon[109861]: pgmap v2226: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.9 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 798 op/s; 6048/92694 objects degraded (6.525%); 5908/92694 objects misplaced (6.374%); 5.5 MiB/s, 87 objects/s recovering 2024-08-17T17:56:44.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:43 smithi136 ceph-mon[115078]: pgmap v2226: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.9 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 798 op/s; 6048/92694 objects degraded (6.525%); 5908/92694 objects misplaced (6.374%); 5.5 MiB/s, 87 objects/s recovering 2024-08-17T17:56:44.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:43 smithi165 ceph-mon[105517]: pgmap v2226: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.9 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 798 op/s; 6048/92694 objects degraded (6.525%); 5908/92694 objects misplaced (6.374%); 5.5 MiB/s, 87 objects/s recovering 2024-08-17T17:56:44.277 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:44.312 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:44.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:44.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:44.414 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:44.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:44.457 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:44.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:44.558 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:44.597 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:44.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:44.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:44.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:44.743 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:44.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:44.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:44.845 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:44.873 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:44.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:44.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:44.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:44 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 6048/92694 objects degraded (6.525%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:44.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:44 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 6048/92694 objects degraded (6.525%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:44.978 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:44.999 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:44.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:45.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:45.101 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:45.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:45.134 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:45.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:45.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:45.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:44 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 6048/92694 objects degraded (6.525%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:45.268 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:45.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:45.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:45.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:45.397 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:45.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:45.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:45.499 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:45.522 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:45.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:45.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:45.624 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:45.660 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:45.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:45.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:45.762 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:45.796 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:45.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:45.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:45.898 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:45.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:45.933 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:45.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:46.034 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:46.054 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:46.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:46.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:46.156 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:46.179 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:45 smithi165 ceph-mon[105517]: pgmap v2227: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.9 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 798 op/s; 6048/92694 objects degraded (6.525%); 5908/92694 objects misplaced (6.374%); 5.5 MiB/s, 87 objects/s recovering 2024-08-17T17:56:46.193 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:46.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:46.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:46.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:45 smithi136 ceph-mon[109861]: pgmap v2227: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.9 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 798 op/s; 6048/92694 objects degraded (6.525%); 5908/92694 objects misplaced (6.374%); 5.5 MiB/s, 87 objects/s recovering 2024-08-17T17:56:46.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:45 smithi136 ceph-mon[115078]: pgmap v2227: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.9 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 798 op/s; 6048/92694 objects degraded (6.525%); 5908/92694 objects misplaced (6.374%); 5.5 MiB/s, 87 objects/s recovering 2024-08-17T17:56:46.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:46.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:46.324 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:46.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:46.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:46.460 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:46.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:46.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:46.562 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:46.602 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:46.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:46.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:46.704 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:46.732 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:46.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:46.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:46.834 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:46.876 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:46.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:46.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:46.978 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:47.014 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:47.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:47.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:47.116 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:47.151 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:47.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:47.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:47.253 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:47.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:47.275 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:47.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:47.376 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:47.411 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:47.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:47.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:47.513 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:47.546 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:47.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:47.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:47.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:47.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:47.672 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:47.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:47.773 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:47.815 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:47.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:47.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:47.836 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:56:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:56:47] "GET /metrics HTTP/1.1" 200 36561 "" "Prometheus/2.51.0" 2024-08-17T17:56:47.917 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:47.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:47.947 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:47.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:48.048 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:48.091 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:48.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:48.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:48.193 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:48.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:47 smithi136 ceph-mon[115078]: pgmap v2228: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.7 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 797 op/s; 5798/85893 objects degraded (6.750%); 5133/85893 objects misplaced (5.976%); 5.5 MiB/s, 88 objects/s recovering 2024-08-17T17:56:48.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:47 smithi136 ceph-mon[109861]: pgmap v2228: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.7 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 797 op/s; 5798/85893 objects degraded (6.750%); 5133/85893 objects misplaced (5.976%); 5.5 MiB/s, 88 objects/s recovering 2024-08-17T17:56:48.238 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:48.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:48.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:48.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:47 smithi165 ceph-mon[105517]: pgmap v2228: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.7 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 797 op/s; 5798/85893 objects degraded (6.750%); 5133/85893 objects misplaced (5.976%); 5.5 MiB/s, 88 objects/s recovering 2024-08-17T17:56:48.340 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:48.394 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:48.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:48.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:48.496 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:48.545 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:48.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:48.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:48.648 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:48.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:48.679 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:48.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:48.780 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:48.810 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:48.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:48.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:48.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:48.939 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:48.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:48.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:49.041 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:49.083 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:49.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:49.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:49.185 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:49.223 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:49.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:49.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:49.325 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:49.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:49.380 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:49.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:49.482 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:49.527 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:49.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:49.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:49.628 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:49.654 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:49.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:49.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:49.733 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:56:49.756 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:49.780 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:49.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:49.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:49.882 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:49.886 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:49 smithi136 ceph-mon[109861]: pgmap v2229: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.6 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 805 op/s; 5798/80559 objects degraded (7.197%); 5133/80559 objects misplaced (6.372%); 5.5 MiB/s, 88 objects/s recovering 2024-08-17T17:56:49.886 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:49 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5798/80559 objects degraded (7.197%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:49.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:49 smithi136 ceph-mon[115078]: pgmap v2229: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.6 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 805 op/s; 5798/80559 objects degraded (7.197%); 5133/80559 objects misplaced (6.372%); 5.5 MiB/s, 88 objects/s recovering 2024-08-17T17:56:49.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:49 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5798/80559 objects degraded (7.197%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:49.917 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:49.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:49.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:50.019 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:50.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:50.068 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:50.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:50.169 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:50.203 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:50.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:50.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:50.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:49 smithi165 ceph-mon[105517]: pgmap v2229: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.6 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 805 op/s; 5798/80559 objects degraded (7.197%); 5133/80559 objects misplaced (6.372%); 5.5 MiB/s, 88 objects/s recovering 2024-08-17T17:56:50.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:49 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5798/80559 objects degraded (7.197%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:50.296 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:56:50.296 INFO:teuthology.orchestra.run.smithi136.stdout:{"fsid":"8270436c-5cba-11ef-bccf-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5798/80559 objects degraded (7.197%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":2306,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":1350,"num_osds":8,"num_up_osds":8,"osd_up_since":1723915456,"num_in_osds":8,"osd_in_since":1723914479,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":96},{"state_name":"active+recovering+undersized+remapped","count":2},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":99,"num_pools":5,"num_objects":26853,"data_bytes":1758631933,"bytes_used":8057917440,"bytes_avail":759902367744,"bytes_total":767960285184,"degraded_objects":5798,"degraded_total":80559,"degraded_ratio":0.071972094986283347,"misplaced_objects":5133,"misplaced_total":80559,"misplaced_ratio":0.063717275537183926,"recovering_objects_per_sec":88,"recovering_bytes_per_sec":5789326,"recovering_keys_per_sec":0,"num_objects_recovered":884,"num_bytes_recovered":57933824,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":805},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi136.ltsuqd","status":"up:active","gid":24557}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.136:8443/","prometheus":"http://172.21.15.136:9283/"}},"servicemap":{"epoch":33,"modified":"2024-08-17T17:55:16.625533+0000","services":{}},"progress_events":{}} 2024-08-17T17:56:50.304 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:50.333 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:50.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:50.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:50.435 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:50.473 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:50.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:50.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:50.557 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:56:50.575 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:50.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:50.656 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:50.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:50.757 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:50.795 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:50.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:50.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:50.897 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:50.928 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:50.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:50.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:50.944 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T17:56:50.944 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:56:51.030 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:51.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:51.074 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:51.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:51.175 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:51.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:50 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3611606715' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:56:51.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:50 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3611606715' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:56:51.226 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:51.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:51.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:51.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:50 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3611606715' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:56:51.328 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:51.366 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:51.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:51.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:51.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:51.506 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:51.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:51.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:51.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:51.635 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:51.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:51.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:51.736 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:51.770 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:51.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:51.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:51.871 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:51.903 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:51.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:51.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:52.005 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:52.008 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-17T17:56:52.051 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:52.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:52.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:52.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:52.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:52.191 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:52.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:52.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:52.320 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:52.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:52.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:52.420 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:52 smithi136 ceph-mon[109861]: pgmap v2230: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.6 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 805 op/s; 5798/80559 objects degraded (7.197%); 5133/80559 objects misplaced (6.372%); 5.5 MiB/s, 88 objects/s recovering 2024-08-17T17:56:52.420 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:56:52.420 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:56:52.420 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:52 smithi136 ceph-mon[115078]: pgmap v2230: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.6 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 805 op/s; 5798/80559 objects degraded (7.197%); 5133/80559 objects misplaced (6.372%); 5.5 MiB/s, 88 objects/s recovering 2024-08-17T17:56:52.420 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:56:52.420 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:56:52.423 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:52.453 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:52.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:52.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:52 smithi165 ceph-mon[105517]: pgmap v2230: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.6 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 805 op/s; 5798/80559 objects degraded (7.197%); 5133/80559 objects misplaced (6.372%); 5.5 MiB/s, 88 objects/s recovering 2024-08-17T17:56:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:56:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:56:52.554 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:52.596 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:52.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:52.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:52.698 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:52.730 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:52.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:52.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:52.802 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-08-17T17:56:52.802 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:56:52.832 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:52.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:52.917 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:52.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:53.018 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:53.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:53.047 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:53.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:53.148 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:53.184 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:53.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:53.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:53.286 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:53.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:53.355 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:53.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:53.357 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:56:53.409 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:56:53.457 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:53.484 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:53.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:53.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:53.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:53.628 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:53.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:53.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:53.729 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:53.758 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:53.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:53.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:53.860 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:53.899 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:53.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:53.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:54.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:54.047 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:54.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:54.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:54.149 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:54.179 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:54.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:54.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:54.280 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:54.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:54.324 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:54.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:54.417 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:54 smithi136 ceph-mon[109861]: pgmap v2231: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.4 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 880 op/s; 5530/69147 objects degraded (7.997%); 4351/69147 objects misplaced (6.292%); 5.2 MiB/s, 83 objects/s recovering 2024-08-17T17:56:54.417 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:54 smithi136 ceph-mon[115078]: pgmap v2231: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.4 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 880 op/s; 5530/69147 objects degraded (7.997%); 4351/69147 objects misplaced (6.292%); 5.2 MiB/s, 83 objects/s recovering 2024-08-17T17:56:54.425 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:54.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:54.474 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:54.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:54.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:54 smithi165 ceph-mon[105517]: pgmap v2231: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.4 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 880 op/s; 5530/69147 objects degraded (7.997%); 4351/69147 objects misplaced (6.292%); 5.2 MiB/s, 83 objects/s recovering 2024-08-17T17:56:54.575 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:54.605 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:54.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:54.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:54.707 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:54.737 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T17:56:54.738 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:56:54.738 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:54.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:54.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:54.774 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T17:56:54.775 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:56:54.817 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T17:56:54.817 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:56:54.840 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:54.854 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T17:56:54.854 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:56:54.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:54.872 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:54.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:54.890 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T17:56:54.890 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:56:54.921 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T17:56:54.921 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:56:54.951 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T17:56:54.951 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:56:54.974 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:54.982 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T17:56:54.982 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:56:55.005 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:55.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:55.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:55.107 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:55.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:55.144 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:55.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:55.245 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:55.258 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:55 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5530/69147 objects degraded (7.997%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:55.258 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:55 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/668083226' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:56:55.258 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:55 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5530/69147 objects degraded (7.997%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:55.258 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:55 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/668083226' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:56:55.272 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:55.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:55.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:55.374 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:55.410 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:55.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:55.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:55 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5530/69147 objects degraded (7.997%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:56:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:55 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/668083226' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:56:55.512 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:55.547 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:55.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:55.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:55.649 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:55.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:55.687 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:55.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:55.788 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:55.833 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:55.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:55.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:55.934 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:55.966 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:55.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:55.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:56.067 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:56.088 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:56.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:56.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:56.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:56.221 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:56.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:56.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:56.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:56.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:56.366 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:56.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:56.467 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:56:56.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:56.492 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:56.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:56:56.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:56 smithi165 ceph-mon[105517]: pgmap v2232: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.4 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 653 op/s; 5530/69147 objects degraded (7.997%); 4351/69147 objects misplaced (6.292%); 4.7 MiB/s, 75 objects/s recovering 2024-08-17T17:56:56.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:56.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:56.632 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:56.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:56.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:56 smithi136 ceph-mon[109861]: pgmap v2232: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.4 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 653 op/s; 5530/69147 objects degraded (7.997%); 4351/69147 objects misplaced (6.292%); 4.7 MiB/s, 75 objects/s recovering 2024-08-17T17:56:56.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:56 smithi136 ceph-mon[115078]: pgmap v2232: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.4 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 653 op/s; 5530/69147 objects degraded (7.997%); 4351/69147 objects misplaced (6.292%); 4.7 MiB/s, 75 objects/s recovering 2024-08-17T17:56:56.733 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:56.753 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:56.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:56.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:56.855 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:56.893 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:56.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:56.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:56.995 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:57.030 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:57.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:57.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:57.132 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:57.158 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:57.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:57.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:57.260 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:57.297 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:57.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:57.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:57.399 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:57.435 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:57.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:57.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:57.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:57.571 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:57.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:57.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:57.673 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:57.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:56:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:56:57] "GET /metrics HTTP/1.1" 200 36562 "" "Prometheus/2.51.0" 2024-08-17T17:56:57.710 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:57.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:57.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:57.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:56:57.845 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:57.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:57.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:56:57.948 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:57.986 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:57.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:57.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:58.088 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:56:58.124 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:58.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:58.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:56:58.226 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:58.261 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:58.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:58.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:58.363 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:58.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:58.399 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:58.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:58.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:56:58.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:56:58 smithi165 ceph-mon[105517]: pgmap v2233: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.3 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 844 op/s; 5252/62289 objects degraded (8.432%); 3468/62289 objects misplaced (5.568%); 7.4 MiB/s, 118 objects/s recovering 2024-08-17T17:56:58.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:58.530 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:58.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:56:58.631 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:58.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:58.663 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:58.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:58.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:56:58 smithi136 ceph-mon[109861]: pgmap v2233: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.3 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 844 op/s; 5252/62289 objects degraded (8.432%); 3468/62289 objects misplaced (5.568%); 7.4 MiB/s, 118 objects/s recovering 2024-08-17T17:56:58.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:56:58 smithi136 ceph-mon[115078]: pgmap v2233: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.3 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 844 op/s; 5252/62289 objects degraded (8.432%); 3468/62289 objects misplaced (5.568%); 7.4 MiB/s, 118 objects/s recovering 2024-08-17T17:56:58.764 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:58.785 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:58.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:58.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:58.887 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:58.926 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:58.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:58.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:59.028 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:59.065 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:59.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:59.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:59.167 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:56:59.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:59.210 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:59.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:56:59.310 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:56:59.352 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:59.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:59.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:56:59.454 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:59.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:59.501 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:59.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:59.602 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:59.623 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:59.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:59.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:56:59.724 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:56:59.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:59.761 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:56:59.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:56:59.862 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:56:59.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:56:59.911 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:56:59.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:00.012 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:00.032 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:00.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:00.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:00.134 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:00.175 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:00.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:00.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:00.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:00.302 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:00.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:00.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:00.404 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:00.445 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:00.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:00.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:00.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:00 smithi136 ceph-mon[109861]: pgmap v2234: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 798 op/s; 5252/57120 objects degraded (9.195%); 3468/57120 objects misplaced (6.071%); 5.0 MiB/s, 80 objects/s recovering 2024-08-17T17:57:00.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:00 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5252/57120 objects degraded (9.195%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:57:00.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:00 smithi136 ceph-mon[115078]: pgmap v2234: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 798 op/s; 5252/57120 objects degraded (9.195%); 3468/57120 objects misplaced (6.071%); 5.0 MiB/s, 80 objects/s recovering 2024-08-17T17:57:00.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:00 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5252/57120 objects degraded (9.195%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:57:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:00 smithi165 ceph-mon[105517]: pgmap v2234: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 798 op/s; 5252/57120 objects degraded (9.195%); 3468/57120 objects misplaced (6.071%); 5.0 MiB/s, 80 objects/s recovering 2024-08-17T17:57:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:00 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5252/57120 objects degraded (9.195%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:57:00.547 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:00.585 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:00.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:00.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:00.687 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:00.719 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:00.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:00.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:00.820 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:00.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:00.858 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:00.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:00.959 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:01.005 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:01.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:01.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:01.106 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:01.139 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:01.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:01.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:01.241 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:01.280 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:01.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:01.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:01.382 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:01.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:01.428 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:01.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:01.529 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:01.553 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:01.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:01.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:01.655 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:01.687 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:01.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:01.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:01.789 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:01.820 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:01.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:01.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:01.922 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:01.964 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:01.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:01.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:02.066 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:02.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:02.103 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:02.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:02.204 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:02.238 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:02.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:02.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:02.340 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:02.380 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:02.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:02.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:02.482 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:02.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:02 smithi165 ceph-mon[105517]: pgmap v2235: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 650 op/s; 5252/57120 objects degraded (9.195%); 3468/57120 objects misplaced (6.071%); 5.0 MiB/s, 80 objects/s recovering 2024-08-17T17:57:02.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:02.514 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:02.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:02.558 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:02 smithi136 ceph-mon[115078]: pgmap v2235: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 650 op/s; 5252/57120 objects degraded (9.195%); 3468/57120 objects misplaced (6.071%); 5.0 MiB/s, 80 objects/s recovering 2024-08-17T17:57:02.558 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:02 smithi136 ceph-mon[109861]: pgmap v2235: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 1.2 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 650 op/s; 5252/57120 objects degraded (9.195%); 3468/57120 objects misplaced (6.071%); 5.0 MiB/s, 80 objects/s recovering 2024-08-17T17:57:02.615 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:02.650 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:02.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:02.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:02.752 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:02.783 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:02.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:02.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:02.885 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:02.923 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:02.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:02.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:03.025 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:03.067 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:03.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:03.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:03.168 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:03.212 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:03.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:03.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:03.314 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:03.344 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:03.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:03.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:03.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:03.479 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:03.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:03.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:03.582 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:03.621 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:03.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:03.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:03.723 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:03.751 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:03.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:03.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:03.853 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:03.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:03.907 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:03.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:04.008 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:04.043 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:04.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:04.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:04.145 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:04.175 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:04.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:04.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:04.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:04.308 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:04.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:04.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:04.410 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:04.449 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:04.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:04.514 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:04 smithi165 ceph-mon[105517]: pgmap v2236: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 955 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 963 op/s; 5011/45873 objects degraded (10.924%); 2602/45873 objects misplaced (5.672%); 7.6 MiB/s, 121 objects/s recovering 2024-08-17T17:57:04.551 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:04.587 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:04.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:04.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:04.689 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:04.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:04 smithi136 ceph-mon[109861]: pgmap v2236: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 955 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 963 op/s; 5011/45873 objects degraded (10.924%); 2602/45873 objects misplaced (5.672%); 7.6 MiB/s, 121 objects/s recovering 2024-08-17T17:57:04.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:04 smithi136 ceph-mon[115078]: pgmap v2236: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 955 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 963 op/s; 5011/45873 objects degraded (10.924%); 2602/45873 objects misplaced (5.672%); 7.6 MiB/s, 121 objects/s recovering 2024-08-17T17:57:04.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:04.723 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:04.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:04.824 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:04.865 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:04.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:04.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:04.968 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:05.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:05.001 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:05.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:05.102 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:05.123 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:05.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:05.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:05.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:05.255 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:05.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:05.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:05.357 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:05.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:05.380 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:05.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:05.481 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:05.506 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:05.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:05.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:05.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:05 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5011/45873 objects degraded (10.924%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:57:05.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:05.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:05.643 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:05.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:05 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5011/45873 objects degraded (10.924%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:57:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:05 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5011/45873 objects degraded (10.924%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:57:05.744 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:05.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:05.766 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:05.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:05.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:05.913 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:05.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:05.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:06.015 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:06.046 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:06.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:06.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:06.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:06.180 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:06.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:06.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:06.252 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:57:06.282 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:06.317 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:06.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:06.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:06.366 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:06 smithi136 ceph-mon[109861]: pgmap v2237: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 955 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 646 op/s; 5011/45873 objects degraded (10.924%); 2602/45873 objects misplaced (5.672%); 5.2 MiB/s, 83 objects/s recovering 2024-08-17T17:57:06.366 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:57:06.366 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:06 smithi136 ceph-mon[115078]: pgmap v2237: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 955 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 646 op/s; 5011/45873 objects degraded (10.924%); 2602/45873 objects misplaced (5.672%); 5.2 MiB/s, 83 objects/s recovering 2024-08-17T17:57:06.366 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:57:06.419 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:06.456 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:06.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:06.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:06.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:06 smithi165 ceph-mon[105517]: pgmap v2237: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 96 active+clean; 955 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 646 op/s; 5011/45873 objects degraded (10.924%); 2602/45873 objects misplaced (5.672%); 5.2 MiB/s, 83 objects/s recovering 2024-08-17T17:57:06.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:57:06.557 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:06.595 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:06.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:06.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:06.696 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:06.741 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:06.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:06.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:06.842 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:06.896 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:06.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:06.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:06.998 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:07.029 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:07.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:07.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:07.131 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:07.180 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:07.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:07.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:07.282 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:07.306 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:07.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:07.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:07.408 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:07.453 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:07.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:07.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:07.555 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:07.589 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:07.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:07.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:07.691 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:07.733 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:07.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:07.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:07.834 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:07.873 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:07.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:07.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:07.937 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:57:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:57:07] "GET /metrics HTTP/1.1" 200 36562 "" "Prometheus/2.51.0" 2024-08-17T17:57:07.976 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:08.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:08.038 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:08.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:08.139 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:08.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:08.175 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:08.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:08.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:08.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:08.316 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:08.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:08.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:08.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:08 smithi136 ceph-mon[109861]: pgmap v2238: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 96 active+clean; 816 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 831 op/s; 4772/39210 objects degraded (12.170%); 2387/39210 objects misplaced (6.088%); 6.1 MiB/s, 128 objects/s recovering 2024-08-17T17:57:08.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:08 smithi136 ceph-mon[115078]: pgmap v2238: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 96 active+clean; 816 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 831 op/s; 4772/39210 objects degraded (12.170%); 2387/39210 objects misplaced (6.088%); 6.1 MiB/s, 128 objects/s recovering 2024-08-17T17:57:08.450 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:08.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:08.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:08.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:08 smithi165 ceph-mon[105517]: pgmap v2238: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 96 active+clean; 816 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 831 op/s; 4772/39210 objects degraded (12.170%); 2387/39210 objects misplaced (6.088%); 6.1 MiB/s, 128 objects/s recovering 2024-08-17T17:57:08.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:08.599 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:08.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:08.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:08.701 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:08.732 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:08.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:08.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:08.833 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:08.860 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:08.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:08.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:08.888 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:57:08.962 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:09.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:09.011 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:09.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:09.112 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:09.141 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:09.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:09.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:09.243 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:09.259 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T17:57:09.282 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:09.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:09.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:09.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:09.406 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:09.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:09.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:09.509 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:09.531 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:09.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:09.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:09.632 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:09.655 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:09.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:09.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:09.690 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:57:09.690 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-17T17:57:09.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-17T17:57:09.690 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T17:57:09.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.690 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:68969 53.11 53:8c790e22:::benchmark_data_smithi136_235573_object68968:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1315)", 2024-08-17T17:57:09.691 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:55.004301+0000", 2024-08-17T17:57:09.691 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 193.91528258100001, 2024-08-17T17:57:09.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.916112381, 2024-08-17T17:57:09.691 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:57:09.691 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:57:09.691 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:57:09.691 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:57:09.691 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:57:09.691 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:57:09.691 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 68969 2024-08-17T17:57:09.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.691 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:57:09.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:57:09.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.004301+0000", 2024-08-17T17:57:09.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:57:09.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.004301+0000", 2024-08-17T17:57:09.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:57:09.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.004303+0000", 2024-08-17T17:57:09.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.556e-06 2024-08-17T17:57:09.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:57:09.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.004325+0000", 2024-08-17T17:57:09.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1933000000000002e-05 2024-08-17T17:57:09.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:57:09.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.004338+0000", 2024-08-17T17:57:09.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3397999999999999e-05 2024-08-17T17:57:09.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:57:09.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.004345+0000", 2024-08-17T17:57:09.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8410000000000002e-06 2024-08-17T17:57:09.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.004392+0000", 2024-08-17T17:57:09.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7027000000000002e-05 2024-08-17T17:57:09.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.004400+0000", 2024-08-17T17:57:09.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8199999999999997e-06 2024-08-17T17:57:09.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.816867+0000", 2024-08-17T17:57:09.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.812467202 2024-08-17T17:57:09.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.816878+0000", 2024-08-17T17:57:09.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1053000000000001e-05 2024-08-17T17:57:09.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.883671+0000", 2024-08-17T17:57:09.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.066792427 2024-08-17T17:57:09.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:57:09.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.883802+0000", 2024-08-17T17:57:09.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000131641 2024-08-17T17:57:09.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:57:09.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.883912+0000", 2024-08-17T17:57:09.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010993999999999999 2024-08-17T17:57:09.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:57:09.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.919417+0000", 2024-08-17T17:57:09.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035505038000000003 2024-08-17T17:57:09.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.919484+0000", 2024-08-17T17:57:09.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6679000000000003e-05 2024-08-17T17:57:09.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.919528+0000", 2024-08-17T17:57:09.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3924000000000002e-05 2024-08-17T17:57:09.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:57:09.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.919588+0000", 2024-08-17T17:57:09.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0452999999999999e-05 2024-08-17T17:57:09.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:57:09.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.920414+0000", 2024-08-17T17:57:09.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00082544900000000002 2024-08-17T17:57:09.699 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.699 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:57:09.699 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.699 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:68984 53.11 53:89631696:::benchmark_data_smithi136_235573_object68983:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1315)", 2024-08-17T17:57:09.699 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:55.019611+0000", 2024-08-17T17:57:09.699 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 193.89997268400001, 2024-08-17T17:57:09.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.907546974, 2024-08-17T17:57:09.699 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:57:09.699 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:57:09.700 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:57:09.700 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:57:09.700 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:57:09.700 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:57:09.700 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 68984 2024-08-17T17:57:09.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.700 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:57:09.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:57:09.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.019611+0000", 2024-08-17T17:57:09.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:57:09.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.019611+0000", 2024-08-17T17:57:09.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:57:09.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.019612+0000", 2024-08-17T17:57:09.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2359999999999999e-06 2024-08-17T17:57:09.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:57:09.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.019630+0000", 2024-08-17T17:57:09.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.785e-05 2024-08-17T17:57:09.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:57:09.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.019644+0000", 2024-08-17T17:57:09.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4027e-05 2024-08-17T17:57:09.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:57:09.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.019664+0000", 2024-08-17T17:57:09.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0176000000000001e-05 2024-08-17T17:57:09.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.019745+0000", 2024-08-17T17:57:09.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0463000000000005e-05 2024-08-17T17:57:09.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.019759+0000", 2024-08-17T17:57:09.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3716999999999999e-05 2024-08-17T17:57:09.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817260+0000", 2024-08-17T17:57:09.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.79750139099999995 2024-08-17T17:57:09.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817268+0000", 2024-08-17T17:57:09.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2749999999999995e-06 2024-08-17T17:57:09.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.920465+0000", 2024-08-17T17:57:09.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.103196316 2024-08-17T17:57:09.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:57:09.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.920611+0000", 2024-08-17T17:57:09.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014624000000000001 2024-08-17T17:57:09.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:57:09.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.920791+0000", 2024-08-17T17:57:09.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018059400000000001 2024-08-17T17:57:09.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.924968+0000", 2024-08-17T17:57:09.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0041761309999999996 2024-08-17T17:57:09.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.926259+0000", 2024-08-17T17:57:09.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012915820000000001 2024-08-17T17:57:09.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:57:09.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.926273+0000", 2024-08-17T17:57:09.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4302e-05 2024-08-17T17:57:09.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:57:09.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.926309+0000", 2024-08-17T17:57:09.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5055999999999998e-05 2024-08-17T17:57:09.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:57:09.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.927158+0000", 2024-08-17T17:57:09.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00084961800000000001 2024-08-17T17:57:09.708 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.708 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:57:09.708 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.708 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:68988 53.11 53:8b834f85:::benchmark_data_smithi136_235573_object68987:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1315)", 2024-08-17T17:57:09.708 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:55.023364+0000", 2024-08-17T17:57:09.708 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 193.89621938900001, 2024-08-17T17:57:09.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.904550363, 2024-08-17T17:57:09.708 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:57:09.708 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:57:09.708 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:57:09.709 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:57:09.709 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:57:09.709 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:57:09.709 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 68988 2024-08-17T17:57:09.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.709 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:57:09.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:57:09.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.023364+0000", 2024-08-17T17:57:09.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:57:09.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.023364+0000", 2024-08-17T17:57:09.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:57:09.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.023366+0000", 2024-08-17T17:57:09.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.719e-06 2024-08-17T17:57:09.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:57:09.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.023389+0000", 2024-08-17T17:57:09.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2989e-05 2024-08-17T17:57:09.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:57:09.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.023402+0000", 2024-08-17T17:57:09.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2473999999999999e-05 2024-08-17T17:57:09.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:57:09.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.023409+0000", 2024-08-17T17:57:09.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7710000000000006e-06 2024-08-17T17:57:09.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.023491+0000", 2024-08-17T17:57:09.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1546999999999997e-05 2024-08-17T17:57:09.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.023601+0000", 2024-08-17T17:57:09.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109702 2024-08-17T17:57:09.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817294+0000", 2024-08-17T17:57:09.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.7936936 2024-08-17T17:57:09.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817302+0000", 2024-08-17T17:57:09.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6399999999999997e-06 2024-08-17T17:57:09.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.922098+0000", 2024-08-17T17:57:09.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.104796092 2024-08-17T17:57:09.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:57:09.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.922231+0000", 2024-08-17T17:57:09.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013339800000000001 2024-08-17T17:57:09.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:57:09.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.922397+0000", 2024-08-17T17:57:09.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016586699999999999 2024-08-17T17:57:09.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:57:09.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.927167+0000", 2024-08-17T17:57:09.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0047701080000000003 2024-08-17T17:57:09.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.927205+0000", 2024-08-17T17:57:09.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7141000000000002e-05 2024-08-17T17:57:09.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.927277+0000", 2024-08-17T17:57:09.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2736000000000003e-05 2024-08-17T17:57:09.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:57:09.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.927319+0000", 2024-08-17T17:57:09.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1539000000000003e-05 2024-08-17T17:57:09.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:57:09.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.927915+0000", 2024-08-17T17:57:09.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00059604000000000005 2024-08-17T17:57:09.719 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.719 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:57:09.719 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.720 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:69008 53.11 53:8e2f4718:::benchmark_data_smithi136_235573_object69007:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1315)", 2024-08-17T17:57:09.720 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:55.038169+0000", 2024-08-17T17:57:09.720 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 193.881414842, 2024-08-17T17:57:09.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8907584499999999, 2024-08-17T17:57:09.721 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:57:09.721 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:57:09.721 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:57:09.721 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:57:09.722 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:57:09.722 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:57:09.722 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69008 2024-08-17T17:57:09.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.722 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:57:09.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:57:09.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.038169+0000", 2024-08-17T17:57:09.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:57:09.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.038169+0000", 2024-08-17T17:57:09.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:57:09.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.038170+0000", 2024-08-17T17:57:09.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8899999999999998e-07 2024-08-17T17:57:09.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:57:09.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.038195+0000", 2024-08-17T17:57:09.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4936e-05 2024-08-17T17:57:09.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:57:09.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.038211+0000", 2024-08-17T17:57:09.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5767999999999999e-05 2024-08-17T17:57:09.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:57:09.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.038219+0000", 2024-08-17T17:57:09.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4419999999999998e-06 2024-08-17T17:57:09.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.039102+0000", 2024-08-17T17:57:09.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00088293200000000001 2024-08-17T17:57:09.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.039111+0000", 2024-08-17T17:57:09.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8869999999999998e-06 2024-08-17T17:57:09.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817328+0000", 2024-08-17T17:57:09.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.77821703399999997 2024-08-17T17:57:09.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817336+0000", 2024-08-17T17:57:09.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8250000000000005e-06 2024-08-17T17:57:09.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.923528+0000", 2024-08-17T17:57:09.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1061920650000001 2024-08-17T17:57:09.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:57:09.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.923668+0000", 2024-08-17T17:57:09.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014017000000000001 2024-08-17T17:57:09.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:57:09.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.923845+0000", 2024-08-17T17:57:09.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000177059 2024-08-17T17:57:09.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.927238+0000", 2024-08-17T17:57:09.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033931819999999998 2024-08-17T17:57:09.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:57:09.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.927247+0000", 2024-08-17T17:57:09.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3809999999999999e-06 2024-08-17T17:57:09.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.928080+0000", 2024-08-17T17:57:09.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00083379199999999998 2024-08-17T17:57:09.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:57:09.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.928203+0000", 2024-08-17T17:57:09.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012258700000000001 2024-08-17T17:57:09.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:57:09.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.928927+0000", 2024-08-17T17:57:09.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00072440099999999995 2024-08-17T17:57:09.739 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.739 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:57:09.740 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.740 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:69026 53.11 53:8f20bb15:::benchmark_data_smithi136_235573_object69025:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1315)", 2024-08-17T17:57:09.740 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:55.048011+0000", 2024-08-17T17:57:09.741 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 193.87157325000001, 2024-08-17T17:57:09.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9082420760000001, 2024-08-17T17:57:09.741 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:57:09.741 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:57:09.742 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:57:09.742 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:57:09.742 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:57:09.742 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:57:09.742 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69026 2024-08-17T17:57:09.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.743 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:57:09.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:57:09.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.048011+0000", 2024-08-17T17:57:09.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:57:09.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.048011+0000", 2024-08-17T17:57:09.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:57:09.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.048012+0000", 2024-08-17T17:57:09.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2139999999999999e-06 2024-08-17T17:57:09.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:57:09.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.048036+0000", 2024-08-17T17:57:09.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4275000000000001e-05 2024-08-17T17:57:09.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:57:09.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.048048+0000", 2024-08-17T17:57:09.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2351000000000001e-05 2024-08-17T17:57:09.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:57:09.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.048055+0000", 2024-08-17T17:57:09.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8419999999999999e-06 2024-08-17T17:57:09.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.048116+0000", 2024-08-17T17:57:09.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0266000000000002e-05 2024-08-17T17:57:09.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.048125+0000", 2024-08-17T17:57:09.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5130000000000002e-06 2024-08-17T17:57:09.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817361+0000", 2024-08-17T17:57:09.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.76923631999999997 2024-08-17T17:57:09.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817369+0000", 2024-08-17T17:57:09.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3409999999999997e-06 2024-08-17T17:57:09.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.928970+0000", 2024-08-17T17:57:09.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.111600948 2024-08-17T17:57:09.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:57:09.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.929116+0000", 2024-08-17T17:57:09.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014680499999999999 2024-08-17T17:57:09.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:57:09.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.929275+0000", 2024-08-17T17:57:09.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000158997 2024-08-17T17:57:09.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.934818+0000", 2024-08-17T17:57:09.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0055429850000000003 2024-08-17T17:57:09.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:57:09.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.940734+0000", 2024-08-17T17:57:09.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005915862 2024-08-17T17:57:09.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.955006+0000", 2024-08-17T17:57:09.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014272079999999999 2024-08-17T17:57:09.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:57:09.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.955057+0000", 2024-08-17T17:57:09.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1023000000000003e-05 2024-08-17T17:57:09.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:57:09.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.956253+0000", 2024-08-17T17:57:09.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001195254 2024-08-17T17:57:09.760 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.760 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:57:09.760 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.761 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:69044 53.11 53:8f858b2a:::benchmark_data_smithi136_235573_object69043:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1315)", 2024-08-17T17:57:09.761 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:55.059783+0000", 2024-08-17T17:57:09.761 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 193.859800831, 2024-08-17T17:57:09.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8975963199999999, 2024-08-17T17:57:09.761 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:57:09.762 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:57:09.762 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:57:09.762 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:57:09.762 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:57:09.762 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:57:09.763 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69044 2024-08-17T17:57:09.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.763 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:57:09.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:57:09.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.059783+0000", 2024-08-17T17:57:09.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:57:09.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.059783+0000", 2024-08-17T17:57:09.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:57:09.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.059785+0000", 2024-08-17T17:57:09.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8640000000000001e-06 2024-08-17T17:57:09.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:57:09.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.059807+0000", 2024-08-17T17:57:09.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2005999999999999e-05 2024-08-17T17:57:09.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:57:09.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.059822+0000", 2024-08-17T17:57:09.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5396999999999998e-05 2024-08-17T17:57:09.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:57:09.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.059830+0000", 2024-08-17T17:57:09.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2980000000000003e-06 2024-08-17T17:57:09.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.059932+0000", 2024-08-17T17:57:09.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010234900000000001 2024-08-17T17:57:09.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.059944+0000", 2024-08-17T17:57:09.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2306999999999999e-05 2024-08-17T17:57:09.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817396+0000", 2024-08-17T17:57:09.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.75745137500000004 2024-08-17T17:57:09.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817403+0000", 2024-08-17T17:57:09.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3379999999999999e-06 2024-08-17T17:57:09.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.930494+0000", 2024-08-17T17:57:09.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1130907059999999 2024-08-17T17:57:09.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:57:09.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.930706+0000", 2024-08-17T17:57:09.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000212497 2024-08-17T17:57:09.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:57:09.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.930860+0000", 2024-08-17T17:57:09.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015366800000000001 2024-08-17T17:57:09.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.940774+0000", 2024-08-17T17:57:09.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0099136910000000005 2024-08-17T17:57:09.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:57:09.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.940785+0000", 2024-08-17T17:57:09.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1488000000000001e-05 2024-08-17T17:57:09.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.956381+0000", 2024-08-17T17:57:09.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015596102000000001 2024-08-17T17:57:09.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:57:09.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.956440+0000", 2024-08-17T17:57:09.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8777000000000003e-05 2024-08-17T17:57:09.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:57:09.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.957379+0000", 2024-08-17T17:57:09.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00093945699999999999 2024-08-17T17:57:09.777 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.777 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:57:09.777 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.777 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:69086 53.11 53:8e8ebd26:::benchmark_data_smithi136_235573_object69085:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1315)", 2024-08-17T17:57:09.777 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:55.086561+0000", 2024-08-17T17:57:09.777 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 193.833022516, 2024-08-17T17:57:09.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8721604119999999, 2024-08-17T17:57:09.778 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:57:09.778 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:57:09.778 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:57:09.778 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:57:09.778 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:57:09.778 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:57:09.778 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69086 2024-08-17T17:57:09.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.778 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:57:09.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:57:09.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.086561+0000", 2024-08-17T17:57:09.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:57:09.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.086561+0000", 2024-08-17T17:57:09.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:57:09.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.086562+0000", 2024-08-17T17:57:09.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0699999999999997e-07 2024-08-17T17:57:09.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:57:09.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.086580+0000", 2024-08-17T17:57:09.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7907999999999999e-05 2024-08-17T17:57:09.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:57:09.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.086588+0000", 2024-08-17T17:57:09.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1650000000000006e-06 2024-08-17T17:57:09.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:57:09.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.086594+0000", 2024-08-17T17:57:09.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5960000000000003e-06 2024-08-17T17:57:09.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.086659+0000", 2024-08-17T17:57:09.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5699999999999998e-05 2024-08-17T17:57:09.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.086668+0000", 2024-08-17T17:57:09.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7800000000000006e-06 2024-08-17T17:57:09.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817429+0000", 2024-08-17T17:57:09.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.73076094400000002 2024-08-17T17:57:09.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817436+0000", 2024-08-17T17:57:09.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.554e-06 2024-08-17T17:57:09.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.932078+0000", 2024-08-17T17:57:09.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.114641496 2024-08-17T17:57:09.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:57:09.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.932220+0000", 2024-08-17T17:57:09.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014168500000000001 2024-08-17T17:57:09.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:57:09.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.932400+0000", 2024-08-17T17:57:09.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017982799999999999 2024-08-17T17:57:09.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:57:09.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.940792+0000", 2024-08-17T17:57:09.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0083922670000000001 2024-08-17T17:57:09.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.940853+0000", 2024-08-17T17:57:09.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1643000000000006e-05 2024-08-17T17:57:09.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.957587+0000", 2024-08-17T17:57:09.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016733843000000002 2024-08-17T17:57:09.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:57:09.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.957742+0000", 2024-08-17T17:57:09.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015436099999999999 2024-08-17T17:57:09.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:57:09.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.958722+0000", 2024-08-17T17:57:09.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00098013500000000003 2024-08-17T17:57:09.785 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.785 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:57:09.785 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.785 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:69132 53.11 53:8bfa44b7:::benchmark_data_smithi136_235573_object69131:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1315)", 2024-08-17T17:57:09.785 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:55.116580+0000", 2024-08-17T17:57:09.785 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 193.80300433400001, 2024-08-17T17:57:09.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.843158788, 2024-08-17T17:57:09.786 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:57:09.786 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:57:09.786 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:57:09.786 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:57:09.786 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:57:09.786 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:57:09.786 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69132 2024-08-17T17:57:09.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.786 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:57:09.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:57:09.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.116580+0000", 2024-08-17T17:57:09.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:57:09.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.116580+0000", 2024-08-17T17:57:09.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:57:09.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.116581+0000", 2024-08-17T17:57:09.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4219999999999999e-06 2024-08-17T17:57:09.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:57:09.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.116604+0000", 2024-08-17T17:57:09.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3434e-05 2024-08-17T17:57:09.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:57:09.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.116618+0000", 2024-08-17T17:57:09.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3808000000000001e-05 2024-08-17T17:57:09.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:57:09.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.116625+0000", 2024-08-17T17:57:09.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0099999999999998e-06 2024-08-17T17:57:09.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.116704+0000", 2024-08-17T17:57:09.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9270000000000005e-05 2024-08-17T17:57:09.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.116715+0000", 2024-08-17T17:57:09.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0285000000000001e-05 2024-08-17T17:57:09.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817461+0000", 2024-08-17T17:57:09.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.70074646699999998 2024-08-17T17:57:09.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817469+0000", 2024-08-17T17:57:09.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5390000000000002e-06 2024-08-17T17:57:09.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.933506+0000", 2024-08-17T17:57:09.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1160373290000001 2024-08-17T17:57:09.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:57:09.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.933624+0000", 2024-08-17T17:57:09.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011785 2024-08-17T17:57:09.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:57:09.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.933771+0000", 2024-08-17T17:57:09.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014699099999999999 2024-08-17T17:57:09.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:57:09.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.940799+0000", 2024-08-17T17:57:09.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0070276549999999998 2024-08-17T17:57:09.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.940899+0000", 2024-08-17T17:57:09.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000100616 2024-08-17T17:57:09.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.958817+0000", 2024-08-17T17:57:09.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017917756999999999 2024-08-17T17:57:09.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:57:09.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.958874+0000", 2024-08-17T17:57:09.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6832000000000001e-05 2024-08-17T17:57:09.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:57:09.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.959738+0000", 2024-08-17T17:57:09.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00086452299999999996 2024-08-17T17:57:09.793 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.793 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:57:09.793 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.793 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:69198 53.11 53:8fdb5866:::benchmark_data_smithi136_235573_object69197:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1315)", 2024-08-17T17:57:09.793 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:55.153390+0000", 2024-08-17T17:57:09.793 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 193.76619395200001, 2024-08-17T17:57:09.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8072508969999999, 2024-08-17T17:57:09.793 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:57:09.793 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:57:09.793 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:57:09.794 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:57:09.794 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:57:09.794 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:57:09.794 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69198 2024-08-17T17:57:09.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.794 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:57:09.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:57:09.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.153390+0000", 2024-08-17T17:57:09.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:57:09.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.153390+0000", 2024-08-17T17:57:09.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:57:09.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.153392+0000", 2024-08-17T17:57:09.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.669e-06 2024-08-17T17:57:09.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:57:09.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.153417+0000", 2024-08-17T17:57:09.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4950999999999999e-05 2024-08-17T17:57:09.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:57:09.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.153437+0000", 2024-08-17T17:57:09.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0849e-05 2024-08-17T17:57:09.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:57:09.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.153446+0000", 2024-08-17T17:57:09.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9209999999999998e-06 2024-08-17T17:57:09.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.153563+0000", 2024-08-17T17:57:09.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116761 2024-08-17T17:57:09.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.153577+0000", 2024-08-17T17:57:09.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4243999999999999e-05 2024-08-17T17:57:09.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817495+0000", 2024-08-17T17:57:09.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.66391758499999998 2024-08-17T17:57:09.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817507+0000", 2024-08-17T17:57:09.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1693e-05 2024-08-17T17:57:09.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.934850+0000", 2024-08-17T17:57:09.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1173437639999999 2024-08-17T17:57:09.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:57:09.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.934987+0000", 2024-08-17T17:57:09.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013709200000000001 2024-08-17T17:57:09.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:57:09.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.935160+0000", 2024-08-17T17:57:09.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000172781 2024-08-17T17:57:09.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:57:09.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.940805+0000", 2024-08-17T17:57:09.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0056452179999999996 2024-08-17T17:57:09.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.940977+0000", 2024-08-17T17:57:09.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017190599999999999 2024-08-17T17:57:09.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.959842+0000", 2024-08-17T17:57:09.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018864328 2024-08-17T17:57:09.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:57:09.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.959903+0000", 2024-08-17T17:57:09.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0942e-05 2024-08-17T17:57:09.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:57:09.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.960641+0000", 2024-08-17T17:57:09.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00073819300000000005 2024-08-17T17:57:09.800 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.801 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:57:09.801 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.801 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:69210 53.11 53:88140cf8:::benchmark_data_smithi136_235573_object69209:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1315)", 2024-08-17T17:57:09.801 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:55.160564+0000", 2024-08-17T17:57:09.801 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 193.75901986, 2024-08-17T17:57:09.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.801046972, 2024-08-17T17:57:09.801 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:57:09.801 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:57:09.801 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:57:09.801 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:57:09.801 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:57:09.801 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:57:09.802 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69210 2024-08-17T17:57:09.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.802 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:57:09.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:57:09.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.160564+0000", 2024-08-17T17:57:09.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:57:09.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.160564+0000", 2024-08-17T17:57:09.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:57:09.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.160566+0000", 2024-08-17T17:57:09.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.001e-06 2024-08-17T17:57:09.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:57:09.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.160592+0000", 2024-08-17T17:57:09.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5874999999999999e-05 2024-08-17T17:57:09.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:57:09.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.160611+0000", 2024-08-17T17:57:09.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9426999999999999e-05 2024-08-17T17:57:09.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:57:09.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.160620+0000", 2024-08-17T17:57:09.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1609999999999992e-06 2024-08-17T17:57:09.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.160710+0000", 2024-08-17T17:57:09.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9456999999999994e-05 2024-08-17T17:57:09.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.160722+0000", 2024-08-17T17:57:09.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2465e-05 2024-08-17T17:57:09.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817532+0000", 2024-08-17T17:57:09.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.65680985400000003 2024-08-17T17:57:09.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817539+0000", 2024-08-17T17:57:09.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0910000000000004e-06 2024-08-17T17:57:09.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.936542+0000", 2024-08-17T17:57:09.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1190022230000001 2024-08-17T17:57:09.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:57:09.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.936711+0000", 2024-08-17T17:57:09.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000169488 2024-08-17T17:57:09.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:57:09.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.936888+0000", 2024-08-17T17:57:09.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000176835 2024-08-17T17:57:09.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:57:09.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.940811+0000", 2024-08-17T17:57:09.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039228209999999999 2024-08-17T17:57:09.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.947437+0000", 2024-08-17T17:57:09.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0066259919999999998 2024-08-17T17:57:09.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.960747+0000", 2024-08-17T17:57:09.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.01331033 2024-08-17T17:57:09.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:57:09.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.960803+0000", 2024-08-17T17:57:09.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5597999999999999e-05 2024-08-17T17:57:09.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:57:09.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.961611+0000", 2024-08-17T17:57:09.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00080835399999999995 2024-08-17T17:57:09.808 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.808 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:57:09.808 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.809 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:69271 53.11 53:8ce5bb14:::benchmark_data_smithi136_235573_object69270:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1315)", 2024-08-17T17:57:09.809 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:55.211366+0000", 2024-08-17T17:57:09.809 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 193.70821745800001, 2024-08-17T17:57:09.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.751121581, 2024-08-17T17:57:09.809 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:57:09.809 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:57:09.809 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:57:09.809 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:57:09.809 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:57:09.809 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:57:09.809 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69271 2024-08-17T17:57:09.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.810 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:57:09.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.810 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:57:09.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.211366+0000", 2024-08-17T17:57:09.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.810 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:57:09.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.211366+0000", 2024-08-17T17:57:09.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:57:09.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.211368+0000", 2024-08-17T17:57:09.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6440000000000001e-06 2024-08-17T17:57:09.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:57:09.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.211390+0000", 2024-08-17T17:57:09.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1974999999999999e-05 2024-08-17T17:57:09.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:57:09.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.211404+0000", 2024-08-17T17:57:09.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4215e-05 2024-08-17T17:57:09.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:57:09.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.211411+0000", 2024-08-17T17:57:09.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8850000000000001e-06 2024-08-17T17:57:09.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.211461+0000", 2024-08-17T17:57:09.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0043000000000002e-05 2024-08-17T17:57:09.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.211468+0000", 2024-08-17T17:57:09.813 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:09 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1974874804' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:57:09.813 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:09 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:57:09.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1180000000000003e-06 2024-08-17T17:57:09.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817565+0000", 2024-08-17T17:57:09.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60609648299999996 2024-08-17T17:57:09.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817572+0000", 2024-08-17T17:57:09.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2119999999999997e-06 2024-08-17T17:57:09.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.938061+0000", 2024-08-17T17:57:09.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1204888129999999 2024-08-17T17:57:09.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:57:09.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.938185+0000", 2024-08-17T17:57:09.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012426900000000001 2024-08-17T17:57:09.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:57:09.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.938325+0000", 2024-08-17T17:57:09.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013990000000000001 2024-08-17T17:57:09.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:57:09.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.940816+0000", 2024-08-17T17:57:09.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002491165 2024-08-17T17:57:09.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.954923+0000", 2024-08-17T17:57:09.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014106466 2024-08-17T17:57:09.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.961662+0000", 2024-08-17T17:57:09.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0067390109999999996 2024-08-17T17:57:09.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:57:09.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.961705+0000", 2024-08-17T17:57:09.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3813999999999998e-05 2024-08-17T17:57:09.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:57:09.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.962488+0000", 2024-08-17T17:57:09.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00078256799999999998 2024-08-17T17:57:09.817 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.817 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:57:09.817 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.817 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:69273 53.11 53:8df1eb89:::benchmark_data_smithi136_235573_object69272:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1315)", 2024-08-17T17:57:09.818 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:55.212496+0000", 2024-08-17T17:57:09.818 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 193.707088333, 2024-08-17T17:57:09.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.750955126, 2024-08-17T17:57:09.818 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:57:09.818 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:57:09.818 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:57:09.818 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:57:09.818 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:57:09.818 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:57:09.818 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69273 2024-08-17T17:57:09.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.818 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:57:09.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:57:09.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.212496+0000", 2024-08-17T17:57:09.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:57:09.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.212496+0000", 2024-08-17T17:57:09.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:57:09.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.212497+0000", 2024-08-17T17:57:09.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.063e-06 2024-08-17T17:57:09.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:57:09.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.212519+0000", 2024-08-17T17:57:09.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2107e-05 2024-08-17T17:57:09.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:57:09.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.212535+0000", 2024-08-17T17:57:09.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6290000000000002e-05 2024-08-17T17:57:09.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:57:09.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.212543+0000", 2024-08-17T17:57:09.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.875e-06 2024-08-17T17:57:09.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.212605+0000", 2024-08-17T17:57:09.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2398999999999997e-05 2024-08-17T17:57:09.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.212615+0000", 2024-08-17T17:57:09.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7249999999999992e-06 2024-08-17T17:57:09.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817597+0000", 2024-08-17T17:57:09.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60498199399999997 2024-08-17T17:57:09.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817604+0000", 2024-08-17T17:57:09.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3599999999999998e-06 2024-08-17T17:57:09.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.939436+0000", 2024-08-17T17:57:09.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1218320500000001 2024-08-17T17:57:09.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:57:09.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.939630+0000", 2024-08-17T17:57:09.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000193396 2024-08-17T17:57:09.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:57:09.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.939811+0000", 2024-08-17T17:57:09.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000181368 2024-08-17T17:57:09.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:57:09.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.942382+0000", 2024-08-17T17:57:09.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002571201 2024-08-17T17:57:09.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.956307+0000", 2024-08-17T17:57:09.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013924348 2024-08-17T17:57:09.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.962542+0000", 2024-08-17T17:57:09.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0062349199999999997 2024-08-17T17:57:09.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:57:09.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.962590+0000", 2024-08-17T17:57:09.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8458000000000003e-05 2024-08-17T17:57:09.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:57:09.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.963451+0000", 2024-08-17T17:57:09.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00086057199999999999 2024-08-17T17:57:09.825 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.825 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:57:09.825 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.825 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:69281 53.11 53:8c292b23:::benchmark_data_smithi136_235573_object69280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1315)", 2024-08-17T17:57:09.825 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:55.220092+0000", 2024-08-17T17:57:09.825 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 193.699492303, 2024-08-17T17:57:09.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7441553829999998, 2024-08-17T17:57:09.826 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:57:09.826 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:57:09.826 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:57:09.826 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:57:09.826 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:57:09.826 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:57:09.826 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69281 2024-08-17T17:57:09.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.826 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:57:09.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:57:09.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.220092+0000", 2024-08-17T17:57:09.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:57:09.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.220092+0000", 2024-08-17T17:57:09.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:57:09.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.220093+0000", 2024-08-17T17:57:09.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2470000000000001e-06 2024-08-17T17:57:09.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:57:09.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.220117+0000", 2024-08-17T17:57:09.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3886000000000001e-05 2024-08-17T17:57:09.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:57:09.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.220160+0000", 2024-08-17T17:57:09.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3520999999999999e-05 2024-08-17T17:57:09.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:57:09.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.220170+0000", 2024-08-17T17:57:09.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4909999999999994e-06 2024-08-17T17:57:09.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.830 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.220244+0000", 2024-08-17T17:57:09.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4002000000000001e-05 2024-08-17T17:57:09.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.830 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.220255+0000", 2024-08-17T17:57:09.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1022000000000001e-05 2024-08-17T17:57:09.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817631+0000", 2024-08-17T17:57:09.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59737596900000001 2024-08-17T17:57:09.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817638+0000", 2024-08-17T17:57:09.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.204e-06 2024-08-17T17:57:09.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.941025+0000", 2024-08-17T17:57:09.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.123386934 2024-08-17T17:57:09.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:57:09.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.941186+0000", 2024-08-17T17:57:09.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000161421 2024-08-17T17:57:09.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:57:09.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.941373+0000", 2024-08-17T17:57:09.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000186727 2024-08-17T17:57:09.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:57:09.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.947399+0000", 2024-08-17T17:57:09.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0060264990000000003 2024-08-17T17:57:09.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.957501+0000", 2024-08-17T17:57:09.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.01010118 2024-08-17T17:57:09.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.963499+0000", 2024-08-17T17:57:09.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0059988250000000002 2024-08-17T17:57:09.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:57:09.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.963560+0000", 2024-08-17T17:57:09.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0177999999999999e-05 2024-08-17T17:57:09.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:57:09.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.964247+0000", 2024-08-17T17:57:09.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00068727700000000003 2024-08-17T17:57:09.834 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.834 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:57:09.834 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.835 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:69306 53.11 53:8c2d2c08:::benchmark_data_smithi136_235573_object69305:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1315)", 2024-08-17T17:57:09.835 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:55.248260+0000", 2024-08-17T17:57:09.835 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 193.67132379099999, 2024-08-17T17:57:09.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7167232129999999, 2024-08-17T17:57:09.835 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:57:09.835 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:57:09.835 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:57:09.835 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:57:09.835 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:57:09.835 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:57:09.835 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69306 2024-08-17T17:57:09.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.836 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:57:09.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:57:09.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.248260+0000", 2024-08-17T17:57:09.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:57:09.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.248260+0000", 2024-08-17T17:57:09.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:57:09.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.248262+0000", 2024-08-17T17:57:09.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.513e-06 2024-08-17T17:57:09.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:57:09.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.248283+0000", 2024-08-17T17:57:09.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1653000000000001e-05 2024-08-17T17:57:09.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:57:09.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.248297+0000", 2024-08-17T17:57:09.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3835000000000001e-05 2024-08-17T17:57:09.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:57:09.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.248304+0000", 2024-08-17T17:57:09.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1620000000000002e-06 2024-08-17T17:57:09.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.248362+0000", 2024-08-17T17:57:09.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8131999999999998e-05 2024-08-17T17:57:09.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.248373+0000", 2024-08-17T17:57:09.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0866000000000001e-05 2024-08-17T17:57:09.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817663+0000", 2024-08-17T17:57:09.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.56928986800000003 2024-08-17T17:57:09.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817669+0000", 2024-08-17T17:57:09.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2809999999999997e-06 2024-08-17T17:57:09.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.942412+0000", 2024-08-17T17:57:09.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.124742382 2024-08-17T17:57:09.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:57:09.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.942545+0000", 2024-08-17T17:57:09.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013318999999999999 2024-08-17T17:57:09.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:57:09.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.942707+0000", 2024-08-17T17:57:09.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000161564 2024-08-17T17:57:09.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:57:09.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.947450+0000", 2024-08-17T17:57:09.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0047430720000000001 2024-08-17T17:57:09.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.958773+0000", 2024-08-17T17:57:09.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011323497 2024-08-17T17:57:09.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.964294+0000", 2024-08-17T17:57:09.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0055210470000000003 2024-08-17T17:57:09.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:57:09.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.964345+0000", 2024-08-17T17:57:09.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0855e-05 2024-08-17T17:57:09.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:57:09.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.964983+0000", 2024-08-17T17:57:09.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000638296 2024-08-17T17:57:09.842 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.842 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:57:09.842 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.843 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:09 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1974874804' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:57:09.843 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:09 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:57:09.843 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:69332 53.11 53:88359284:::benchmark_data_smithi136_235573_object69331:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1315)", 2024-08-17T17:57:09.843 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:55.278084+0000", 2024-08-17T17:57:09.844 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 193.64150001799999, 2024-08-17T17:57:09.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6876464869999999, 2024-08-17T17:57:09.844 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:57:09.844 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:57:09.844 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:57:09.844 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:57:09.844 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:57:09.844 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:57:09.844 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69332 2024-08-17T17:57:09.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.844 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:57:09.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:57:09.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.278084+0000", 2024-08-17T17:57:09.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:57:09.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.278084+0000", 2024-08-17T17:57:09.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:57:09.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.278085+0000", 2024-08-17T17:57:09.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2079999999999999e-06 2024-08-17T17:57:09.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:57:09.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.278104+0000", 2024-08-17T17:57:09.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9432e-05 2024-08-17T17:57:09.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:57:09.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.278115+0000", 2024-08-17T17:57:09.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0716e-05 2024-08-17T17:57:09.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:57:09.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.278122+0000", 2024-08-17T17:57:09.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.686e-06 2024-08-17T17:57:09.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.278175+0000", 2024-08-17T17:57:09.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2942999999999998e-05 2024-08-17T17:57:09.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.278183+0000", 2024-08-17T17:57:09.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9389999999999997e-06 2024-08-17T17:57:09.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817700+0000", 2024-08-17T17:57:09.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.53951727000000005 2024-08-17T17:57:09.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817708+0000", 2024-08-17T17:57:09.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8010000000000004e-06 2024-08-17T17:57:09.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.943786+0000", 2024-08-17T17:57:09.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.126078439 2024-08-17T17:57:09.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:57:09.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.943939+0000", 2024-08-17T17:57:09.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015258099999999999 2024-08-17T17:57:09.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:57:09.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.944150+0000", 2024-08-17T17:57:09.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000210819 2024-08-17T17:57:09.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:57:09.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.947455+0000", 2024-08-17T17:57:09.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033054080000000001 2024-08-17T17:57:09.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.959797+0000", 2024-08-17T17:57:09.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012341935 2024-08-17T17:57:09.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.965025+0000", 2024-08-17T17:57:09.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0052282509999999997 2024-08-17T17:57:09.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:57:09.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.965079+0000", 2024-08-17T17:57:09.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3964e-05 2024-08-17T17:57:09.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:57:09.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.965730+0000", 2024-08-17T17:57:09.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00065109499999999997 2024-08-17T17:57:09.851 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.851 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:57:09.851 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.851 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:69360 53.11 53:8e27d54b:::benchmark_data_smithi136_235573_object69359:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1315)", 2024-08-17T17:57:09.852 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:53:55.350806+0000", 2024-08-17T17:57:09.852 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 193.56877761300001, 2024-08-17T17:57:09.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.617292435, 2024-08-17T17:57:09.852 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:57:09.852 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:57:09.852 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:57:09.852 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:57:09.852 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:57:09.852 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:57:09.852 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69360 2024-08-17T17:57:09.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.852 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:57:09.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:57:09.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.350806+0000", 2024-08-17T17:57:09.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:57:09.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.350806+0000", 2024-08-17T17:57:09.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:57:09.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.350807+0000", 2024-08-17T17:57:09.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0729999999999999e-06 2024-08-17T17:57:09.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:57:09.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.350825+0000", 2024-08-17T17:57:09.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7771e-05 2024-08-17T17:57:09.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:57:09.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.350835+0000", 2024-08-17T17:57:09.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9850000000000001e-06 2024-08-17T17:57:09.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:57:09.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.350840+0000", 2024-08-17T17:57:09.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2240000000000003e-06 2024-08-17T17:57:09.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.350889+0000", 2024-08-17T17:57:09.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8968999999999998e-05 2024-08-17T17:57:09.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.350896+0000", 2024-08-17T17:57:09.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1930000000000004e-06 2024-08-17T17:57:09.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817880+0000", 2024-08-17T17:57:09.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.46698344899999999 2024-08-17T17:57:09.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:55.817889+0000", 2024-08-17T17:57:09.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.918e-06 2024-08-17T17:57:09.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.945815+0000", 2024-08-17T17:57:09.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1279261730000001 2024-08-17T17:57:09.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:57:09.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.946063+0000", 2024-08-17T17:57:09.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024845299999999999 2024-08-17T17:57:09.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:57:09.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.946224+0000", 2024-08-17T17:57:09.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016054700000000001 2024-08-17T17:57:09.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:57:09.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.954823+0000", 2024-08-17T17:57:09.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0085987040000000004 2024-08-17T17:57:09.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.960701+0000", 2024-08-17T17:57:09.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005878164 2024-08-17T17:57:09.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.966798+0000", 2024-08-17T17:57:09.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0060971319999999999 2024-08-17T17:57:09.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:57:09.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.966835+0000", 2024-08-17T17:57:09.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7421e-05 2024-08-17T17:57:09.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:57:09.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:53:56.968099+0000", 2024-08-17T17:57:09.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012632590000000001 2024-08-17T17:57:09.859 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.859 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:57:09.859 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.859 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:79212 53.12 53:49e34d1a:::benchmark_data_smithi136_235573_object79211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1321)", 2024-08-17T17:57:09.859 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:54:04.316050+0000", 2024-08-17T17:57:09.859 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 184.603533908, 2024-08-17T17:57:09.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.220309858, 2024-08-17T17:57:09.859 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:57:09.860 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:57:09.860 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:57:09.860 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:57:09.860 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:57:09.860 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:57:09.860 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 79212 2024-08-17T17:57:09.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.860 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:57:09.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:57:09.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316050+0000", 2024-08-17T17:57:09.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:57:09.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316050+0000", 2024-08-17T17:57:09.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:57:09.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316050+0000", 2024-08-17T17:57:09.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5299999999999999e-07 2024-08-17T17:57:09.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:57:09.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316075+0000", 2024-08-17T17:57:09.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4808999999999999e-05 2024-08-17T17:57:09.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.862 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:57:09.862 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316082+0000", 2024-08-17T17:57:09.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3709999999999996e-06 2024-08-17T17:57:09.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.862 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:57:09.862 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316087+0000", 2024-08-17T17:57:09.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9579999999999998e-06 2024-08-17T17:57:09.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.863 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316463+0000", 2024-08-17T17:57:09.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000376914 2024-08-17T17:57:09.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T17:57:09.863 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316467+0000", 2024-08-17T17:57:09.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.659e-06 2024-08-17T17:57:09.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.863 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.319685+0000", 2024-08-17T17:57:09.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003217934 2024-08-17T17:57:09.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.319760+0000", 2024-08-17T17:57:09.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5344000000000002e-05 2024-08-17T17:57:09.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.447969+0000", 2024-08-17T17:57:09.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1282087270000001 2024-08-17T17:57:09.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:57:09.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.448133+0000", 2024-08-17T17:57:09.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000163707 2024-08-17T17:57:09.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,2,6", 2024-08-17T17:57:09.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.448306+0000", 2024-08-17T17:57:09.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000173468 2024-08-17T17:57:09.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:57:09.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.451465+0000", 2024-08-17T17:57:09.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031585150000000002 2024-08-17T17:57:09.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.520836+0000", 2024-08-17T17:57:09.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.069371438999999993 2024-08-17T17:57:09.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.527591+0000", 2024-08-17T17:57:09.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0067549070000000001 2024-08-17T17:57:09.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.535026+0000", 2024-08-17T17:57:09.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0074348169999999998 2024-08-17T17:57:09.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.867 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:57:09.867 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.535084+0000", 2024-08-17T17:57:09.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7587999999999998e-05 2024-08-17T17:57:09.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.867 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:57:09.867 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.536360+0000", 2024-08-17T17:57:09.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001276248 2024-08-17T17:57:09.867 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.867 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:57:09.867 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.867 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:79221 53.12 53:4c24bda3:::benchmark_data_smithi136_235573_object79220:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1321)", 2024-08-17T17:57:09.867 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:54:04.316132+0000", 2024-08-17T17:57:09.868 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 184.60345173600001, 2024-08-17T17:57:09.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.221225362, 2024-08-17T17:57:09.868 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:57:09.868 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:57:09.868 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:57:09.868 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:57:09.868 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:57:09.868 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:57:09.868 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 79221 2024-08-17T17:57:09.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.868 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:57:09.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.868 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:57:09.868 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316132+0000", 2024-08-17T17:57:09.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:57:09.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316132+0000", 2024-08-17T17:57:09.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:57:09.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316133+0000", 2024-08-17T17:57:09.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4799999999999999e-07 2024-08-17T17:57:09.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:57:09.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316160+0000", 2024-08-17T17:57:09.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7008e-05 2024-08-17T17:57:09.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:57:09.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316167+0000", 2024-08-17T17:57:09.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2779999999999996e-06 2024-08-17T17:57:09.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:57:09.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316173+0000", 2024-08-17T17:57:09.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7749999999999998e-06 2024-08-17T17:57:09.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316487+0000", 2024-08-17T17:57:09.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00031422999999999998 2024-08-17T17:57:09.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T17:57:09.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316491+0000", 2024-08-17T17:57:09.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1740000000000002e-06 2024-08-17T17:57:09.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.872 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.319790+0000", 2024-08-17T17:57:09.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003299246 2024-08-17T17:57:09.872 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.872 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.872 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.872 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.319805+0000", 2024-08-17T17:57:09.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5068999999999999e-05 2024-08-17T17:57:09.872 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.872 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.872 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.872 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.449825+0000", 2024-08-17T17:57:09.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.130019302 2024-08-17T17:57:09.872 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.872 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:57:09.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.449976+0000", 2024-08-17T17:57:09.873 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015088300000000001 2024-08-17T17:57:09.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,2,6", 2024-08-17T17:57:09.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.450154+0000", 2024-08-17T17:57:09.873 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000178927 2024-08-17T17:57:09.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:57:09.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.453311+0000", 2024-08-17T17:57:09.873 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031561029999999999 2024-08-17T17:57:09.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.522123+0000", 2024-08-17T17:57:09.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.068812203000000002 2024-08-17T17:57:09.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.529410+0000", 2024-08-17T17:57:09.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0072869579999999996 2024-08-17T17:57:09.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.536412+0000", 2024-08-17T17:57:09.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0070023289999999998 2024-08-17T17:57:09.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:57:09.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.536483+0000", 2024-08-17T17:57:09.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0938000000000004e-05 2024-08-17T17:57:09.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:57:09.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.537357+0000", 2024-08-17T17:57:09.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00087449100000000003 2024-08-17T17:57:09.875 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.875 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:57:09.875 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.876 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:79224 53.12 53:488406dc:::benchmark_data_smithi136_235573_object79223:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1321)", 2024-08-17T17:57:09.876 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:54:04.316361+0000", 2024-08-17T17:57:09.876 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 184.60322269100001, 2024-08-17T17:57:09.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.222025747, 2024-08-17T17:57:09.876 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:57:09.876 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:57:09.876 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:57:09.876 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:57:09.876 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:57:09.876 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:57:09.876 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 79224 2024-08-17T17:57:09.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.877 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:57:09.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.877 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:57:09.877 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316361+0000", 2024-08-17T17:57:09.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.877 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:57:09.877 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316361+0000", 2024-08-17T17:57:09.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.877 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:57:09.877 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316362+0000", 2024-08-17T17:57:09.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9100000000000004e-07 2024-08-17T17:57:09.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:57:09.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316515+0000", 2024-08-17T17:57:09.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000153666 2024-08-17T17:57:09.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:57:09.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316527+0000", 2024-08-17T17:57:09.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1304000000000001e-05 2024-08-17T17:57:09.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:57:09.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316534+0000", 2024-08-17T17:57:09.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4739999999999999e-06 2024-08-17T17:57:09.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316667+0000", 2024-08-17T17:57:09.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013290599999999999 2024-08-17T17:57:09.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T17:57:09.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316672+0000", 2024-08-17T17:57:09.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6410000000000001e-06 2024-08-17T17:57:09.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.319829+0000", 2024-08-17T17:57:09.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031570809999999999 2024-08-17T17:57:09.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.319834+0000", 2024-08-17T17:57:09.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5620000000000003e-06 2024-08-17T17:57:09.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.451584+0000", 2024-08-17T17:57:09.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1317491310000001 2024-08-17T17:57:09.881 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.881 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:57:09.881 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.451743+0000", 2024-08-17T17:57:09.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015900899999999999 2024-08-17T17:57:09.881 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.881 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,2,6", 2024-08-17T17:57:09.881 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.451907+0000", 2024-08-17T17:57:09.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000164346 2024-08-17T17:57:09.881 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.881 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:57:09.881 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.520750+0000", 2024-08-17T17:57:09.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.068842642999999995 2024-08-17T17:57:09.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.882 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.523158+0000", 2024-08-17T17:57:09.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0024084850000000001 2024-08-17T17:57:09.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.882 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.531772+0000", 2024-08-17T17:57:09.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0086144570000000007 2024-08-17T17:57:09.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.537407+0000", 2024-08-17T17:57:09.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0056344309999999996 2024-08-17T17:57:09.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:57:09.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.537464+0000", 2024-08-17T17:57:09.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6759999999999999e-05 2024-08-17T17:57:09.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:57:09.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.538387+0000", 2024-08-17T17:57:09.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00092325999999999997 2024-08-17T17:57:09.883 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.883 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:57:09.884 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.884 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:79265 53.12 53:4b451fc7:::benchmark_data_smithi136_235573_object79264:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1321)", 2024-08-17T17:57:09.884 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:54:04.316641+0000", 2024-08-17T17:57:09.884 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 184.60294299, 2024-08-17T17:57:09.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2226974859999999, 2024-08-17T17:57:09.884 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:57:09.884 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:57:09.884 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:57:09.884 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:57:09.884 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T17:57:09.884 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:57:09.885 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 79265 2024-08-17T17:57:09.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.885 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:57:09.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:57:09.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316641+0000", 2024-08-17T17:57:09.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:57:09.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316641+0000", 2024-08-17T17:57:09.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:57:09.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:57:09.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316641+0000", 2024-08-17T17:57:09.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2099999999999996e-07 2024-08-17T17:57:09.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:57:09.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316673+0000", 2024-08-17T17:57:09.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1702999999999998e-05 2024-08-17T17:57:09.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:57:09.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316686+0000", 2024-08-17T17:57:09.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.322e-05 2024-08-17T17:57:09.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:57:09.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316721+0000", 2024-08-17T17:57:09.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4375999999999997e-05 2024-08-17T17:57:09.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316779+0000", 2024-08-17T17:57:09.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8548000000000003e-05 2024-08-17T17:57:09.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T17:57:09.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.316785+0000", 2024-08-17T17:57:09.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2639999999999999e-06 2024-08-17T17:57:09.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.319963+0000", 2024-08-17T17:57:09.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031783100000000002 2024-08-17T17:57:09.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T17:57:09.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:04.319969+0000", 2024-08-17T17:57:09.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.874e-06 2024-08-17T17:57:09.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:57:09.889 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.453433+0000", 2024-08-17T17:57:09.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1334640309999999 2024-08-17T17:57:09.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:57:09.889 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.453589+0000", 2024-08-17T17:57:09.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0001561 2024-08-17T17:57:09.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,2,6", 2024-08-17T17:57:09.889 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.453770+0000", 2024-08-17T17:57:09.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000181373 2024-08-17T17:57:09.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.890 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T17:57:09.890 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.520757+0000", 2024-08-17T17:57:09.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.066986757999999993 2024-08-17T17:57:09.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.890 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.890 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.525754+0000", 2024-08-17T17:57:09.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0049973910000000003 2024-08-17T17:57:09.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.890 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.890 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.533372+0000", 2024-08-17T17:57:09.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0076174340000000002 2024-08-17T17:57:09.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:57:09.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.538439+0000", 2024-08-17T17:57:09.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0050671780000000003 2024-08-17T17:57:09.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T17:57:09.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.538494+0000", 2024-08-17T17:57:09.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4639999999999999e-05 2024-08-17T17:57:09.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:57:09.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:57:09.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T17:57:09.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:54:05.539338+0000", 2024-08-17T17:57:09.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00084466499999999998 2024-08-17T17:57:09.892 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.892 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:57:09.892 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.892 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:57:09.892 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:57:09.892 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:57:09.892 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:09.893 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:09 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1974874804' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:57:09.893 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:09 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:57:09.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:09.927 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:09.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:10.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:10.064 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:10.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:10.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:10.166 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:10.199 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:10.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:10.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:10.301 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:10.337 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:10.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:10.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:10.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:10.474 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:10.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:10.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:10.576 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:10.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:10.619 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:10.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:10.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:10 smithi136 ceph-mon[109861]: pgmap v2239: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 96 active+clean; 710 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 781 op/s; 4772/34134 objects degraded (13.980%); 2387/34134 objects misplaced (6.993%); 3.4 MiB/s, 85 objects/s recovering 2024-08-17T17:57:10.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:10 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:57:10.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:10 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:57:10.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:10 smithi136 ceph-mon[109861]: osdmap e1351: 8 total, 8 up, 8 in 2024-08-17T17:57:10.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:10 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 4772/34134 objects degraded (13.980%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:57:10.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:10 smithi136 ceph-mon[115078]: pgmap v2239: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 96 active+clean; 710 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 781 op/s; 4772/34134 objects degraded (13.980%); 2387/34134 objects misplaced (6.993%); 3.4 MiB/s, 85 objects/s recovering 2024-08-17T17:57:10.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:10 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:57:10.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:10 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:57:10.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:10 smithi136 ceph-mon[115078]: osdmap e1351: 8 total, 8 up, 8 in 2024-08-17T17:57:10.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:10 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 4772/34134 objects degraded (13.980%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:57:10.719 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:10.750 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:10.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:10.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:10 smithi165 ceph-mon[105517]: pgmap v2239: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 96 active+clean; 710 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 781 op/s; 4772/34134 objects degraded (13.980%); 2387/34134 objects misplaced (6.993%); 3.4 MiB/s, 85 objects/s recovering 2024-08-17T17:57:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:10 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:57:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:10 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:57:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:10 smithi165 ceph-mon[105517]: osdmap e1351: 8 total, 8 up, 8 in 2024-08-17T17:57:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:10 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 4772/34134 objects degraded (13.980%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:57:10.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:10.902 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:10.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:10.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:11.004 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:11.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:11.059 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:11.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:11.160 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:11.196 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:11.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:11.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:11.298 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:11.331 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:11.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:11.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:11.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:11.479 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:11.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:11.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:11.581 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:11.640 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:11.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:11.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:57:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:57:11.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:57:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:11 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:57:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:11 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:57:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:11 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:57:11.743 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:11 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:57:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:11 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:57:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:11 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:57:11.778 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:11.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:11.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:11.880 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:11.908 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:11.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:11.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:12.010 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:12.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:12.057 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:12.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:12.158 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:12.189 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:12.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:12.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:12.291 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:12.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:12.339 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:12.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:12.440 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:12.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:12.506 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:12.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:12.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:12 smithi165 ceph-mon[105517]: pgmap v2241: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 96 active+clean; 710 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 765 op/s; 4772/34134 objects degraded (13.980%); 2387/34134 objects misplaced (6.993%); 4.1 MiB/s, 102 objects/s recovering 2024-08-17T17:57:12.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:12 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:57:12.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:12 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:57:12.608 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:12.646 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:12.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:12.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:12.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:12 smithi136 ceph-mon[109861]: pgmap v2241: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 96 active+clean; 710 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 765 op/s; 4772/34134 objects degraded (13.980%); 2387/34134 objects misplaced (6.993%); 4.1 MiB/s, 102 objects/s recovering 2024-08-17T17:57:12.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:57:12.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:57:12.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:12 smithi136 ceph-mon[115078]: pgmap v2241: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 96 active+clean; 710 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 765 op/s; 4772/34134 objects degraded (13.980%); 2387/34134 objects misplaced (6.993%); 4.1 MiB/s, 102 objects/s recovering 2024-08-17T17:57:12.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:57:12.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:57:12.751 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:12.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:12.795 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:12.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:12.896 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:12.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:12.936 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:12.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:13.038 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:13.077 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:13.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:13.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:13.179 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:13.213 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:13.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:13.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:13.315 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:13.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:13.353 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:13.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:13.454 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:13.479 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:13.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:13.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:13.581 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:13.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:13.620 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:13.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:13.721 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:13.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:13.783 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:13.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:13.884 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:13.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:13.925 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:13.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:14.026 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:14.067 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:14.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:14.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:14.169 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:14.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:14.207 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:14.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:14.308 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:14.356 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:14.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:14.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:14.455 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-17T17:57:14.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:14.493 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:14.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:14.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:14.573 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:14 smithi136 ceph-mon[109861]: pgmap v2242: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 96 active+clean; 482 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 755 op/s; 4540/23187 objects degraded (19.580%); 2270/23187 objects misplaced (9.790%); 1.8 MiB/s, 118 objects/s recovering 2024-08-17T17:57:14.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:14 smithi136 ceph-mon[115078]: pgmap v2242: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 96 active+clean; 482 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 755 op/s; 4540/23187 objects degraded (19.580%); 2270/23187 objects misplaced (9.790%); 1.8 MiB/s, 118 objects/s recovering 2024-08-17T17:57:14.595 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:14.622 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:14.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:14.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:14.724 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:14.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:14.760 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:14.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:14 smithi165 ceph-mon[105517]: pgmap v2242: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 96 active+clean; 482 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 755 op/s; 4540/23187 objects degraded (19.580%); 2270/23187 objects misplaced (9.790%); 1.8 MiB/s, 118 objects/s recovering 2024-08-17T17:57:14.862 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:14.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:14.920 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:14.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:14.963 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:57:15.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:15.050 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:15.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:15.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:15.152 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:15.194 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:15.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:15.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:15.296 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:15.328 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:15.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:15.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:15.430 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:15.461 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:15.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:15.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:15 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 4540/23187 objects degraded (19.580%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:57:15.563 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:15.605 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:15.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:15.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:15.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:15 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 4540/23187 objects degraded (19.580%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:57:15.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:15 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 4540/23187 objects degraded (19.580%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:57:15.707 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:15.754 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:15.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:15.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:15.856 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:15.883 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:15.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:15.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:15.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:16.032 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:16.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:16.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:16.134 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:16.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:16.170 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:16.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:16.271 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:16.308 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:16.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:16.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:16.410 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:16.471 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:16.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:16.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:16.573 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:16.613 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:16.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:16.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:16.621 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:57:16.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:16 smithi136 ceph-mon[109861]: pgmap v2243: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 96 active+clean; 482 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 755 op/s; 4540/23187 objects degraded (19.580%); 2270/23187 objects misplaced (9.790%); 1.8 MiB/s, 118 objects/s recovering 2024-08-17T17:57:16.622 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:16 smithi136 ceph-mon[115078]: pgmap v2243: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 96 active+clean; 482 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 755 op/s; 4540/23187 objects degraded (19.580%); 2270/23187 objects misplaced (9.790%); 1.8 MiB/s, 118 objects/s recovering 2024-08-17T17:57:16.625 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T17:57:16.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:16 smithi165 ceph-mon[105517]: pgmap v2243: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 96 active+clean; 482 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 755 op/s; 4540/23187 objects degraded (19.580%); 2270/23187 objects misplaced (9.790%); 1.8 MiB/s, 118 objects/s recovering 2024-08-17T17:57:16.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:16.749 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:16.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:16.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:16.850 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:16.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:16.898 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:16.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:16.999 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:17.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:17.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:17.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:17.145 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:17.181 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:17.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:17.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:17.283 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:17.317 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:17.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:17.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:17.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:17.453 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:17.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:17.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:17.555 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:17.597 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:17.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:17.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:17.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:17 smithi136 ceph-mon[109861]: from='client.35295 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:57:17.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:17 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:57:17.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:17 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:57:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:17 smithi136 ceph-mon[115078]: from='client.35295 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:57:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:17 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:57:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:17 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:57:17.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:57:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:57:17] "GET /metrics HTTP/1.1" 200 36562 "" "Prometheus/2.51.0" 2024-08-17T17:57:17.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:17.731 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:17.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:17.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:17 smithi165 ceph-mon[105517]: from='client.35295 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:57:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:17 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:57:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:17 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:57:17.833 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:17.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:17.871 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:17.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:17.973 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:18.010 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:18.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:18.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:18.112 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:18.151 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:18.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:18.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:18.253 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:18.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:18.293 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:18.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:18.394 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:18.431 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:18.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:18.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:18.533 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:18.569 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:18.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:18.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:18.671 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:18.677 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:57:18.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:18 smithi136 ceph-mon[115078]: pgmap v2244: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 96 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 759 op/s; 4288/16410 objects degraded (26.130%); 2144/16410 objects misplaced (13.065%); 1.5 MiB/s, 127 objects/s recovering 2024-08-17T17:57:18.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:18 smithi136 ceph-mon[109861]: pgmap v2244: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 96 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 759 op/s; 4288/16410 objects degraded (26.130%); 2144/16410 objects misplaced (13.065%); 1.5 MiB/s, 127 objects/s recovering 2024-08-17T17:57:18.736 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:18.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:18.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:18.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:18 smithi165 ceph-mon[105517]: pgmap v2244: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 96 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 759 op/s; 4288/16410 objects degraded (26.130%); 2144/16410 objects misplaced (13.065%); 1.5 MiB/s, 127 objects/s recovering 2024-08-17T17:57:18.838 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:18.859 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:18.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:18.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:18.961 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:18.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:18.985 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:18.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:19.086 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:19.108 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:19.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:19.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:19.210 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:19.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:19.233 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:19.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:19.334 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:19.382 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:19.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:19.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:19.484 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:19.528 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:19.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:19.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:19.630 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:19.671 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:19.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:19.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:19.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:19.803 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:19.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:19.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:19.905 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:19.943 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:19.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:19.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:20.045 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:20.083 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:20.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:20.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:20.185 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:20.221 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:20.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:20.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:20.284 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:57:20.317 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":2243,"stamp":"2024-08-17T17:57:14.678320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":505321469,"num_objects":7729,"num_object_clones":0,"num_object_copies":23187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4540,"num_objects_misplaced":2270,"num_objects_unfound":0,"num_objects_dirty":7729,"num_whiteouts":0,"num_read":379,"num_read_kb":823,"num_write":203190,"num_write_kb":5528946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15579,"num_bytes_recovered":938149472,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":215774,"ondisk_log_size":215774,"up":297,"acting":294,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":305,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4357836,"kb_used_data":2374020,"kb_used_omap":295,"kb_used_meta":1982872,"kb_avail":745603380,"statfs":{"total":767960285184,"available":763497861120,"internally_reserved":0,"allocated":2430996480,"data_stored":2397988007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":302302,"internal_metadata":2030461730},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,7,3,6,11,6,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-495583232,"num_objects":-7562,"num_object_clones":0,"num_object_copies":-22686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-471,"num_objects_misplaced":-332,"num_objects_unfound":0,"num_objects_dirty":-7562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7562,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1187,"num_bytes_recovered":18612224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003068"},"pg_stats":[{"pgid":"53.19","version":"1351'9203","reported_seq":9354,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:11.474924+0000","last_change":"2024-08-17T17:53:49.813139+0000","last_active":"2024-08-17T17:57:11.474924+0000","last_peered":"2024-08-17T17:57:11.474924+0000","last_clean":"2024-08-17T17:57:11.474924+0000","last_became_active":"2024-08-17T17:53:45.801709+0000","last_became_peered":"2024-08-17T17:53:45.801709+0000","last_unstale":"2024-08-17T17:57:11.474924+0000","last_undegraded":"2024-08-17T17:57:11.474924+0000","last_fullsized":"2024-08-17T17:57:11.474924+0000","mapping_epoch":1310,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":5,"last_scrub":"1311'3869","last_scrub_stamp":"2024-08-17T17:53:49.812836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:49.812836+0000","objects_scrubbed":1929,"log_size":9203,"log_dups_size":0,"ondisk_log_size":9203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:36.206484+0000","scrub_duration":554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15400960,"num_objects":235,"num_object_clones":0,"num_object_copies":705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5344,"num_write_kb":116800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.18","version":"1351'9525","reported_seq":9647,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:12.137339+0000","last_change":"2024-08-17T17:53:53.987643+0000","last_active":"2024-08-17T17:57:12.137339+0000","last_peered":"2024-08-17T17:57:12.137339+0000","last_clean":"2024-08-17T17:57:12.137339+0000","last_became_active":"2024-08-17T17:53:45.788000+0000","last_became_peered":"2024-08-17T17:53:45.788000+0000","last_unstale":"2024-08-17T17:57:12.137339+0000","last_undegraded":"2024-08-17T17:57:12.137339+0000","last_fullsized":"2024-08-17T17:57:12.137339+0000","mapping_epoch":1310,"log_start":"1308'3014","ondisk_log_start":"1308'3014","created":1310,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":5,"last_scrub":"1313'4093","last_scrub_stamp":"2024-08-17T17:53:53.987508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:53.987508+0000","objects_scrubbed":2233,"log_size":6511,"log_dups_size":0,"ondisk_log_size":6511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:14:47.493287+0000","scrub_duration":0.21921206200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15007744,"num_objects":229,"num_object_clones":0,"num_object_copies":687,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5432,"num_write_kb":109696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"53.17","version":"1351'9595","reported_seq":9747,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:11.261470+0000","last_change":"2024-08-17T17:53:50.287990+0000","last_active":"2024-08-17T17:57:11.261470+0000","last_peered":"2024-08-17T17:57:11.261470+0000","last_clean":"2024-08-17T17:57:11.261470+0000","last_became_active":"2024-08-17T17:53:45.802369+0000","last_became_peered":"2024-08-17T17:53:45.802369+0000","last_unstale":"2024-08-17T17:57:11.261470+0000","last_undegraded":"2024-08-17T17:57:11.261470+0000","last_fullsized":"2024-08-17T17:57:11.261470+0000","mapping_epoch":1310,"log_start":"1308'2904","ondisk_log_start":"1308'2904","created":1310,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":5,"last_scrub":"1312'3969","last_scrub_stamp":"2024-08-17T17:53:50.287885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:50.287885+0000","objects_scrubbed":2090,"log_size":6691,"log_dups_size":0,"ondisk_log_size":6691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:29:18.190184+0000","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16056320,"num_objects":245,"num_object_clones":0,"num_object_copies":735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5639,"num_write_kb":121408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.16","version":"1351'9551","reported_seq":16417,"reported_epoch":1351,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:57:11.004078+0000","last_change":"2024-08-17T17:56:03.772184+0000","last_active":"2024-08-17T17:57:11.004078+0000","last_peered":"2024-08-17T17:57:11.004078+0000","last_clean":"2024-08-17T17:56:00.271068+0000","last_became_active":"2024-08-17T17:56:03.726122+0000","last_became_peered":"2024-08-17T17:56:03.726122+0000","last_unstale":"2024-08-17T17:57:11.004078+0000","last_undegraded":"2024-08-17T17:56:03.680764+0000","last_fullsized":"2024-08-17T17:56:03.528461+0000","mapping_epoch":1347,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":5,"last_scrub":"1311'3927","last_scrub_stamp":"2024-08-17T17:53:48.318097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:48.318097+0000","objects_scrubbed":2014,"log_size":9551,"log_dups_size":0,"ondisk_log_size":9551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16318464,"num_objects":249,"num_object_clones":0,"num_object_copies":747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4540,"num_objects_misplaced":2270,"num_objects_unfound":0,"num_objects_dirty":249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5633,"num_write_kb":123776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1678,"num_bytes_recovered":109969408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[2,1],"avail_no_missing":["2","3","4"],"object_location_counts":[{"shards":"2,3,4","objects":249}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"53.15","version":"1351'9325","reported_seq":16950,"reported_epoch":1351,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T17:57:11.491956+0000","last_change":"2024-08-17T17:57:11.487840+0000","last_active":"2024-08-17T17:57:11.491956+0000","last_peered":"2024-08-17T17:57:11.491956+0000","last_clean":"2024-08-17T17:55:58.529149+0000","last_became_active":"2024-08-17T17:56:01.715968+0000","last_became_peered":"2024-08-17T17:56:01.715968+0000","last_unstale":"2024-08-17T17:57:11.491956+0000","last_undegraded":"2024-08-17T17:57:11.487357+0000","last_fullsized":"2024-08-17T17:56:01.526888+0000","mapping_epoch":1345,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":5,"last_scrub":"1311'3700","last_scrub_stamp":"2024-08-17T17:53:47.592482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:47.592482+0000","objects_scrubbed":1931,"log_size":9325,"log_dups_size":0,"ondisk_log_size":9325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18743296,"num_objects":286,"num_object_clones":0,"num_object_copies":858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5639,"num_write_kb":127808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2491,"num_bytes_recovered":129564672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[0,4],"avail_no_missing":["0","1","3"],"object_location_counts":[{"shards":"0,4","objects":286}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.14","version":"1351'9276","reported_seq":16316,"reported_epoch":1351,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T17:57:11.250433+0000","last_change":"2024-08-17T17:57:11.251417+0000","last_active":"2024-08-17T17:57:11.250433+0000","last_peered":"2024-08-17T17:57:11.250433+0000","last_clean":"2024-08-17T17:55:57.307497+0000","last_became_active":"2024-08-17T17:55:59.771920+0000","last_became_peered":"2024-08-17T17:55:59.771920+0000","last_unstale":"2024-08-17T17:57:11.250433+0000","last_undegraded":"2024-08-17T17:57:11.250433+0000","last_fullsized":"2024-08-17T17:55:59.520008+0000","mapping_epoch":1343,"log_start":"1307'2191","ondisk_log_start":"1307'2191","created":1310,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":5,"last_scrub":"1311'3909","last_scrub_stamp":"2024-08-17T17:53:49.696328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:49.696328+0000","objects_scrubbed":1954,"log_size":7085,"log_dups_size":0,"ondisk_log_size":7085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15073280,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5386,"num_write_kb":117184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2296,"num_bytes_recovered":124977152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[1,3],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,3","objects":230}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.13","version":"1351'9093","reported_seq":18798,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:11.405643+0000","last_change":"2024-08-17T17:55:38.393304+0000","last_active":"2024-08-17T17:57:11.405643+0000","last_peered":"2024-08-17T17:57:11.405643+0000","last_clean":"2024-08-17T17:57:11.405643+0000","last_became_active":"2024-08-17T17:55:38.383298+0000","last_became_peered":"2024-08-17T17:55:38.383298+0000","last_unstale":"2024-08-17T17:57:11.405643+0000","last_undegraded":"2024-08-17T17:57:11.405643+0000","last_fullsized":"2024-08-17T17:57:11.405643+0000","mapping_epoch":1337,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1338,"parent":"0.0","parent_split_bits":5,"last_scrub":"1311'3687","last_scrub_stamp":"2024-08-17T17:53:48.035372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:48.035372+0000","objects_scrubbed":1888,"log_size":9093,"log_dups_size":0,"ondisk_log_size":9093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:04:23.248001+0000","scrub_duration":0.55651981800000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15859712,"num_objects":242,"num_object_clones":0,"num_object_copies":726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5418,"num_write_kb":120704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2386,"num_bytes_recovered":156368896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.12","version":"1351'9344","reported_seq":19277,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:11.997494+0000","last_change":"2024-08-17T17:56:01.548247+0000","last_active":"2024-08-17T17:57:11.997494+0000","last_peered":"2024-08-17T17:57:11.997494+0000","last_clean":"2024-08-17T17:57:11.997494+0000","last_became_active":"2024-08-17T17:56:01.535252+0000","last_became_peered":"2024-08-17T17:56:01.535252+0000","last_unstale":"2024-08-17T17:57:11.997494+0000","last_undegraded":"2024-08-17T17:57:11.997494+0000","last_fullsized":"2024-08-17T17:57:11.997494+0000","mapping_epoch":1345,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1346,"parent":"0.0","parent_split_bits":5,"last_scrub":"1312'4010","last_scrub_stamp":"2024-08-17T17:53:52.788196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:52.788196+0000","objects_scrubbed":2082,"log_size":9344,"log_dups_size":0,"ondisk_log_size":9344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:06:14.839811+0000","scrub_duration":1.1488786010000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13959168,"num_objects":213,"num_object_clones":0,"num_object_copies":639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5355,"num_write_kb":111552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2443,"num_bytes_recovered":160104448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.11","version":"1351'9374","reported_seq":15890,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:11.412775+0000","last_change":"2024-08-17T17:54:32.250770+0000","last_active":"2024-08-17T17:57:11.412775+0000","last_peered":"2024-08-17T17:57:11.412775+0000","last_clean":"2024-08-17T17:57:11.412775+0000","last_became_active":"2024-08-17T17:54:32.246225+0000","last_became_peered":"2024-08-17T17:54:32.246225+0000","last_unstale":"2024-08-17T17:57:11.412775+0000","last_undegraded":"2024-08-17T17:57:11.412775+0000","last_fullsized":"2024-08-17T17:57:11.412775+0000","mapping_epoch":1327,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"1312'3987","last_scrub_stamp":"2024-08-17T17:53:51.667290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:51.667290+0000","objects_scrubbed":2071,"log_size":9374,"log_dups_size":0,"ondisk_log_size":9374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:59:10.324504+0000","scrub_duration":708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15663104,"num_objects":239,"num_object_clones":0,"num_object_copies":717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5404,"num_write_kb":114304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":138805248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.10","version":"1351'9328","reported_seq":15953,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:11.407405+0000","last_change":"2024-08-17T17:57:06.836097+0000","last_active":"2024-08-17T17:57:11.407405+0000","last_peered":"2024-08-17T17:57:11.407405+0000","last_clean":"2024-08-17T17:57:11.407405+0000","last_became_active":"2024-08-17T17:55:57.520262+0000","last_became_peered":"2024-08-17T17:55:57.520262+0000","last_unstale":"2024-08-17T17:57:11.407405+0000","last_undegraded":"2024-08-17T17:57:11.407405+0000","last_fullsized":"2024-08-17T17:57:11.407405+0000","mapping_epoch":1341,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":9328,"log_dups_size":0,"ondisk_log_size":9328,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T17:57:11.831355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16023552,"num_objects":245,"num_object_clones":0,"num_object_copies":735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7473,"num_write_kb":246944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2161,"num_bytes_recovered":116588544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.f","version":"1351'15528","reported_seq":15585,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:12.010082+0000","last_change":"2024-08-17T17:53:10.489536+0000","last_active":"2024-08-17T17:57:12.010082+0000","last_peered":"2024-08-17T17:57:12.010082+0000","last_clean":"2024-08-17T17:57:12.010082+0000","last_became_active":"2024-08-17T17:53:10.489173+0000","last_became_peered":"2024-08-17T17:53:10.489173+0000","last_unstale":"2024-08-17T17:57:12.010082+0000","last_undegraded":"2024-08-17T17:57:12.010082+0000","last_fullsized":"2024-08-17T17:57:12.010082+0000","mapping_epoch":1303,"log_start":"1326'6857","ondisk_log_start":"1326'6857","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":8671,"log_dups_size":0,"ondisk_log_size":8671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:46:03.457908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29884416,"num_objects":456,"num_object_clones":0,"num_object_copies":1368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15528,"num_write_kb":511488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.e","version":"1351'15025","reported_seq":15083,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:12.024514+0000","last_change":"2024-08-17T17:53:10.488446+0000","last_active":"2024-08-17T17:57:12.024514+0000","last_peered":"2024-08-17T17:57:12.024514+0000","last_clean":"2024-08-17T17:57:12.024514+0000","last_became_active":"2024-08-17T17:53:10.488058+0000","last_became_peered":"2024-08-17T17:53:10.488058+0000","last_unstale":"2024-08-17T17:57:12.024514+0000","last_undegraded":"2024-08-17T17:57:12.024514+0000","last_fullsized":"2024-08-17T17:57:12.024514+0000","mapping_epoch":1303,"log_start":"1326'6408","ondisk_log_start":"1326'6408","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":8617,"log_dups_size":0,"ondisk_log_size":8617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:00:57.616106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29949952,"num_objects":457,"num_object_clones":0,"num_object_copies":1371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15025,"num_write_kb":495424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.d","version":"1351'15287","reported_seq":15404,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:11.271900+0000","last_change":"2024-08-17T17:53:10.646982+0000","last_active":"2024-08-17T17:57:11.271900+0000","last_peered":"2024-08-17T17:57:11.271900+0000","last_clean":"2024-08-17T17:57:11.271900+0000","last_became_active":"2024-08-17T17:53:10.646591+0000","last_became_peered":"2024-08-17T17:53:10.646591+0000","last_unstale":"2024-08-17T17:57:11.271900+0000","last_undegraded":"2024-08-17T17:57:11.271900+0000","last_fullsized":"2024-08-17T17:57:11.271900+0000","mapping_epoch":1303,"log_start":"1332'8973","ondisk_log_start":"1332'8973","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":6314,"log_dups_size":0,"ondisk_log_size":6314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:06:34.821249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32768024,"num_objects":501,"num_object_clones":0,"num_object_copies":1503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":501,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15287,"num_write_kb":505153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.c","version":"1351'15049","reported_seq":15107,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:12.163453+0000","last_change":"2024-08-17T17:53:10.559850+0000","last_active":"2024-08-17T17:57:12.163453+0000","last_peered":"2024-08-17T17:57:12.163453+0000","last_clean":"2024-08-17T17:57:12.163453+0000","last_became_active":"2024-08-17T17:53:10.559588+0000","last_became_peered":"2024-08-17T17:53:10.559588+0000","last_unstale":"2024-08-17T17:57:12.163453+0000","last_undegraded":"2024-08-17T17:57:12.163453+0000","last_fullsized":"2024-08-17T17:57:12.163453+0000","mapping_epoch":1303,"log_start":"1324'5300","ondisk_log_start":"1324'5300","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":9749,"log_dups_size":0,"ondisk_log_size":9749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:16:20.553763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30998528,"num_objects":473,"num_object_clones":0,"num_object_copies":1419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15049,"num_write_kb":496704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"53.b","version":"1351'14963","reported_seq":15021,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:12.132228+0000","last_change":"2024-08-17T17:53:10.490102+0000","last_active":"2024-08-17T17:57:12.132228+0000","last_peered":"2024-08-17T17:57:12.132228+0000","last_clean":"2024-08-17T17:57:12.132228+0000","last_became_active":"2024-08-17T17:53:10.489717+0000","last_became_peered":"2024-08-17T17:53:10.489717+0000","last_unstale":"2024-08-17T17:57:12.132228+0000","last_undegraded":"2024-08-17T17:57:12.132228+0000","last_fullsized":"2024-08-17T17:57:12.132228+0000","mapping_epoch":1303,"log_start":"1324'5200","ondisk_log_start":"1324'5200","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":9763,"log_dups_size":0,"ondisk_log_size":9763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:50:43.448878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30081024,"num_objects":459,"num_object_clones":0,"num_object_copies":1377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14963,"num_write_kb":493504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"53.a","version":"1351'15101","reported_seq":15218,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:10.676718+0000","last_change":"2024-08-17T17:53:10.489596+0000","last_active":"2024-08-17T17:57:10.676718+0000","last_peered":"2024-08-17T17:57:10.676718+0000","last_clean":"2024-08-17T17:57:10.676718+0000","last_became_active":"2024-08-17T17:53:10.489230+0000","last_became_peered":"2024-08-17T17:53:10.489230+0000","last_unstale":"2024-08-17T17:57:10.676718+0000","last_undegraded":"2024-08-17T17:57:10.676718+0000","last_fullsized":"2024-08-17T17:57:10.676718+0000","mapping_epoch":1303,"log_start":"1329'7758","ondisk_log_start":"1329'7758","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":7343,"log_dups_size":0,"ondisk_log_size":7343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:50.891777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36110336,"num_objects":551,"num_object_clones":0,"num_object_copies":1653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15101,"num_write_kb":500864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"53.9","version":"1351'9454","reported_seq":9606,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:11.438015+0000","last_change":"2024-08-17T17:53:48.914243+0000","last_active":"2024-08-17T17:57:11.438015+0000","last_peered":"2024-08-17T17:57:11.438015+0000","last_clean":"2024-08-17T17:57:11.438015+0000","last_became_active":"2024-08-17T17:53:45.787626+0000","last_became_peered":"2024-08-17T17:53:45.787626+0000","last_unstale":"2024-08-17T17:57:11.438015+0000","last_undegraded":"2024-08-17T17:57:11.438015+0000","last_fullsized":"2024-08-17T17:57:11.438015+0000","mapping_epoch":1303,"log_start":"1306'15","ondisk_log_start":"1306'15","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3875","last_scrub_stamp":"2024-08-17T17:53:48.913906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:48.913906+0000","objects_scrubbed":2075,"log_size":9439,"log_dups_size":0,"ondisk_log_size":9439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:19:31.071940+0000","scrub_duration":692,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16842752,"num_objects":257,"num_object_clones":0,"num_object_copies":771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5592,"num_write_kb":120768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.8","version":"1351'9482","reported_seq":9608,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:12.054791+0000","last_change":"2024-08-17T17:53:50.732592+0000","last_active":"2024-08-17T17:57:12.054791+0000","last_peered":"2024-08-17T17:57:12.054791+0000","last_clean":"2024-08-17T17:57:12.054791+0000","last_became_active":"2024-08-17T17:53:45.777386+0000","last_became_peered":"2024-08-17T17:53:45.777386+0000","last_unstale":"2024-08-17T17:57:12.054791+0000","last_undegraded":"2024-08-17T17:57:12.054791+0000","last_fullsized":"2024-08-17T17:57:12.054791+0000","mapping_epoch":1303,"log_start":"1308'2873","ondisk_log_start":"1308'2873","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1312'4030","last_scrub_stamp":"2024-08-17T17:53:50.732148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:50.732148+0000","objects_scrubbed":2054,"log_size":6609,"log_dups_size":0,"ondisk_log_size":6609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:42:36.666939+0000","scrub_duration":0.96559516700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13041664,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5469,"num_write_kb":115648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"53.7","version":"1351'9363","reported_seq":9514,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:11.191380+0000","last_change":"2024-08-17T17:53:47.346759+0000","last_active":"2024-08-17T17:57:11.191380+0000","last_peered":"2024-08-17T17:57:11.191380+0000","last_clean":"2024-08-17T17:57:11.191380+0000","last_became_active":"2024-08-17T17:53:45.779914+0000","last_became_peered":"2024-08-17T17:53:45.779914+0000","last_unstale":"2024-08-17T17:57:11.191380+0000","last_undegraded":"2024-08-17T17:57:11.191380+0000","last_fullsized":"2024-08-17T17:57:11.191380+0000","mapping_epoch":1303,"log_start":"1307'2422","ondisk_log_start":"1307'2422","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3847","last_scrub_stamp":"2024-08-17T17:53:47.346625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:47.346625+0000","objects_scrubbed":1924,"log_size":6941,"log_dups_size":0,"ondisk_log_size":6941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:44:59.835054+0000","scrub_duration":430,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14614528,"num_objects":223,"num_object_clones":0,"num_object_copies":669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5521,"num_write_kb":122240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.6","version":"1351'9601","reported_seq":9755,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:10.644837+0000","last_change":"2024-08-17T17:53:47.249481+0000","last_active":"2024-08-17T17:57:10.644837+0000","last_peered":"2024-08-17T17:57:10.644837+0000","last_clean":"2024-08-17T17:57:10.644837+0000","last_became_active":"2024-08-17T17:53:45.758286+0000","last_became_peered":"2024-08-17T17:53:45.758286+0000","last_unstale":"2024-08-17T17:57:10.644837+0000","last_undegraded":"2024-08-17T17:57:10.644837+0000","last_fullsized":"2024-08-17T17:57:10.644837+0000","mapping_epoch":1303,"log_start":"1307'761","ondisk_log_start":"1307'761","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3922","last_scrub_stamp":"2024-08-17T17:53:47.248989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:47.248989+0000","objects_scrubbed":1980,"log_size":8840,"log_dups_size":0,"ondisk_log_size":8840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:43:54.442535+0000","scrub_duration":555,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17432576,"num_objects":266,"num_object_clones":0,"num_object_copies":798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5688,"num_write_kb":127168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"53.5","version":"1351'9065","reported_seq":9215,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:11.401292+0000","last_change":"2024-08-17T17:53:46.758091+0000","last_active":"2024-08-17T17:57:11.401292+0000","last_peered":"2024-08-17T17:57:11.401292+0000","last_clean":"2024-08-17T17:57:11.401292+0000","last_became_active":"2024-08-17T17:53:45.771578+0000","last_became_peered":"2024-08-17T17:53:45.771578+0000","last_unstale":"2024-08-17T17:57:11.401292+0000","last_undegraded":"2024-08-17T17:57:11.401292+0000","last_fullsized":"2024-08-17T17:57:11.401292+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3661","last_scrub_stamp":"2024-08-17T17:53:46.757894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:46.757894+0000","objects_scrubbed":1802,"log_size":9065,"log_dups_size":0,"ondisk_log_size":9065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:27:03.974414+0000","scrub_duration":509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15400960,"num_objects":235,"num_object_clones":0,"num_object_copies":705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5409,"num_write_kb":122944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.4","version":"1351'9493","reported_seq":9645,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:11.112294+0000","last_change":"2024-08-17T17:53:46.426863+0000","last_active":"2024-08-17T17:57:11.112294+0000","last_peered":"2024-08-17T17:57:11.112294+0000","last_clean":"2024-08-17T17:57:11.112294+0000","last_became_active":"2024-08-17T17:53:45.764066+0000","last_became_peered":"2024-08-17T17:53:45.764066+0000","last_unstale":"2024-08-17T17:57:11.112294+0000","last_undegraded":"2024-08-17T17:57:11.112294+0000","last_fullsized":"2024-08-17T17:57:11.112294+0000","mapping_epoch":1303,"log_start":"1308'2833","ondisk_log_start":"1308'2833","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3774","last_scrub_stamp":"2024-08-17T17:53:46.426732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:46.426732+0000","objects_scrubbed":1958,"log_size":6660,"log_dups_size":0,"ondisk_log_size":6660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:49:46.480003+0000","scrub_duration":541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":256,"num_object_clones":0,"num_object_copies":768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5734,"num_write_kb":129024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.3","version":"1351'9050","reported_seq":9166,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:11.971302+0000","last_change":"2024-08-17T17:53:46.968605+0000","last_active":"2024-08-17T17:57:11.971302+0000","last_peered":"2024-08-17T17:57:11.971302+0000","last_clean":"2024-08-17T17:57:11.971302+0000","last_became_active":"2024-08-17T17:53:45.764323+0000","last_became_peered":"2024-08-17T17:53:45.764323+0000","last_unstale":"2024-08-17T17:57:11.971302+0000","last_undegraded":"2024-08-17T17:57:11.971302+0000","last_fullsized":"2024-08-17T17:57:11.971302+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3658","last_scrub_stamp":"2024-08-17T17:53:46.967846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:46.967846+0000","objects_scrubbed":1838,"log_size":9050,"log_dups_size":0,"ondisk_log_size":9050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:09:25.913293+0000","scrub_duration":0.48413113800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14680064,"num_objects":224,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5400,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.2","version":"1351'9412","reported_seq":9534,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:12.097170+0000","last_change":"2024-08-17T17:53:48.304744+0000","last_active":"2024-08-17T17:57:12.097170+0000","last_peered":"2024-08-17T17:57:12.097170+0000","last_clean":"2024-08-17T17:57:12.097170+0000","last_became_active":"2024-08-17T17:53:45.780923+0000","last_became_peered":"2024-08-17T17:53:45.780923+0000","last_unstale":"2024-08-17T17:57:12.097170+0000","last_undegraded":"2024-08-17T17:57:12.097170+0000","last_fullsized":"2024-08-17T17:57:12.097170+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3850","last_scrub_stamp":"2024-08-17T17:53:48.304649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:48.304649+0000","objects_scrubbed":2003,"log_size":9412,"log_dups_size":0,"ondisk_log_size":9412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:01.087711+0000","scrub_duration":0.70259554000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15794176,"num_objects":241,"num_object_clones":0,"num_object_copies":723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5580,"num_write_kb":122176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"53.1","version":"1351'9434","reported_seq":9585,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:10.959557+0000","last_change":"2024-08-17T17:53:53.382389+0000","last_active":"2024-08-17T17:57:10.959557+0000","last_peered":"2024-08-17T17:57:10.959557+0000","last_clean":"2024-08-17T17:57:10.959557+0000","last_became_active":"2024-08-17T17:53:45.782234+0000","last_became_peered":"2024-08-17T17:53:45.782234+0000","last_unstale":"2024-08-17T17:57:10.959557+0000","last_undegraded":"2024-08-17T17:57:10.959557+0000","last_fullsized":"2024-08-17T17:57:10.959557+0000","mapping_epoch":1303,"log_start":"1307'2599","ondisk_log_start":"1307'2599","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1313'4031","last_scrub_stamp":"2024-08-17T17:53:53.382269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:53.382269+0000","objects_scrubbed":2178,"log_size":6835,"log_dups_size":0,"ondisk_log_size":6835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:43.716721+0000","scrub_duration":410,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16973824,"num_objects":259,"num_object_clones":0,"num_object_copies":777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5405,"num_write_kb":111552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.0","version":"1351'9220","reported_seq":9339,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:12.176236+0000","last_change":"2024-08-17T17:53:47.447388+0000","last_active":"2024-08-17T17:57:12.176236+0000","last_peered":"2024-08-17T17:57:12.176236+0000","last_clean":"2024-08-17T17:57:12.176236+0000","last_became_active":"2024-08-17T17:53:45.741805+0000","last_became_peered":"2024-08-17T17:53:45.741805+0000","last_unstale":"2024-08-17T17:57:12.176236+0000","last_undegraded":"2024-08-17T17:57:12.176236+0000","last_fullsized":"2024-08-17T17:57:12.176236+0000","mapping_epoch":1303,"log_start":"1307'2467","ondisk_log_start":"1307'2467","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3799","last_scrub_stamp":"2024-08-17T17:53:47.447203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:47.447203+0000","objects_scrubbed":1904,"log_size":6753,"log_dups_size":0,"ondisk_log_size":6753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:52:36.467443+0000","scrub_duration":0.60964741,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15269888,"num_objects":233,"num_object_clones":0,"num_object_copies":699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5437,"num_write_kb":120512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":2007,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.269675+0000","last_change":"2024-08-17T17:23:42.937520+0000","last_active":"2024-08-17T17:57:09.269675+0000","last_peered":"2024-08-17T17:57:09.269675+0000","last_clean":"2024-08-17T17:57:09.269675+0000","last_became_active":"2024-08-17T17:23:42.937174+0000","last_became_peered":"2024-08-17T17:23:42.937174+0000","last_unstale":"2024-08-17T17:57:09.269675+0000","last_undegraded":"2024-08-17T17:57:09.269675+0000","last_fullsized":"2024-08-17T17:57:09.269675+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:47:56.679612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1322,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.804391+0000","last_change":"2024-08-17T17:23:07.398676+0000","last_active":"2024-08-17T17:56:41.804391+0000","last_peered":"2024-08-17T17:56:41.804391+0000","last_clean":"2024-08-17T17:56:41.804391+0000","last_became_active":"2024-08-17T17:23:07.398369+0000","last_became_peered":"2024-08-17T17:23:07.398369+0000","last_unstale":"2024-08-17T17:56:41.804391+0000","last_undegraded":"2024-08-17T17:56:41.804391+0000","last_fullsized":"2024-08-17T17:56:41.804391+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:55:04.281412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":1991,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.789775+0000","last_change":"2024-08-17T17:24:17.476168+0000","last_active":"2024-08-17T17:57:09.789775+0000","last_peered":"2024-08-17T17:57:09.789775+0000","last_clean":"2024-08-17T17:57:09.789775+0000","last_became_active":"2024-08-17T17:24:17.465845+0000","last_became_peered":"2024-08-17T17:24:17.465845+0000","last_unstale":"2024-08-17T17:57:09.789775+0000","last_undegraded":"2024-08-17T17:57:09.789775+0000","last_fullsized":"2024-08-17T17:57:09.789775+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:15:01.494447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2007,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.263699+0000","last_change":"2024-08-17T17:24:17.461772+0000","last_active":"2024-08-17T17:57:09.263699+0000","last_peered":"2024-08-17T17:57:09.263699+0000","last_clean":"2024-08-17T17:57:09.263699+0000","last_became_active":"2024-08-17T17:24:17.460806+0000","last_became_peered":"2024-08-17T17:24:17.460806+0000","last_unstale":"2024-08-17T17:57:09.263699+0000","last_undegraded":"2024-08-17T17:57:09.263699+0000","last_fullsized":"2024-08-17T17:57:09.263699+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:01:16.453148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":1304,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.814355+0000","last_change":"2024-08-17T17:10:05.965176+0000","last_active":"2024-08-17T17:56:41.814355+0000","last_peered":"2024-08-17T17:56:41.814355+0000","last_clean":"2024-08-17T17:56:41.814355+0000","last_became_active":"2024-08-17T17:10:05.964861+0000","last_became_peered":"2024-08-17T17:10:05.964861+0000","last_unstale":"2024-08-17T17:56:41.814355+0000","last_undegraded":"2024-08-17T17:56:41.814355+0000","last_fullsized":"2024-08-17T17:56:41.814355+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:59:04.277926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1336,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.803203+0000","last_change":"2024-08-17T17:24:17.462527+0000","last_active":"2024-08-17T17:56:41.803203+0000","last_peered":"2024-08-17T17:56:41.803203+0000","last_clean":"2024-08-17T17:56:41.803203+0000","last_became_active":"2024-08-17T17:24:17.462240+0000","last_became_peered":"2024-08-17T17:24:17.462240+0000","last_unstale":"2024-08-17T17:56:41.803203+0000","last_undegraded":"2024-08-17T17:56:41.803203+0000","last_fullsized":"2024-08-17T17:56:41.803203+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:40:38.063717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":1319,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.814442+0000","last_change":"2024-08-17T17:23:07.391038+0000","last_active":"2024-08-17T17:56:41.814442+0000","last_peered":"2024-08-17T17:56:41.814442+0000","last_clean":"2024-08-17T17:56:41.814442+0000","last_became_active":"2024-08-17T17:23:07.390571+0000","last_became_peered":"2024-08-17T17:23:07.390571+0000","last_unstale":"2024-08-17T17:56:41.814442+0000","last_undegraded":"2024-08-17T17:56:41.814442+0000","last_fullsized":"2024-08-17T17:56:41.814442+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:18:41.791740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1336,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.803063+0000","last_change":"2024-08-17T17:24:17.458168+0000","last_active":"2024-08-17T17:56:41.803063+0000","last_peered":"2024-08-17T17:56:41.803063+0000","last_clean":"2024-08-17T17:56:41.803063+0000","last_became_active":"2024-08-17T17:24:17.457847+0000","last_became_peered":"2024-08-17T17:24:17.457847+0000","last_unstale":"2024-08-17T17:56:41.803063+0000","last_undegraded":"2024-08-17T17:56:41.803063+0000","last_fullsized":"2024-08-17T17:56:41.803063+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:28.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":1989,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.270718+0000","last_change":"2024-08-17T17:22:32.883756+0000","last_active":"2024-08-17T17:57:09.270718+0000","last_peered":"2024-08-17T17:57:09.270718+0000","last_clean":"2024-08-17T17:57:09.270718+0000","last_became_active":"2024-08-17T17:22:32.874867+0000","last_became_peered":"2024-08-17T17:22:32.874867+0000","last_unstale":"2024-08-17T17:57:09.270718+0000","last_undegraded":"2024-08-17T17:57:09.270718+0000","last_fullsized":"2024-08-17T17:57:09.270718+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:52:27.375231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":2009,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.263889+0000","last_change":"2024-08-17T17:24:17.456622+0000","last_active":"2024-08-17T17:57:09.263889+0000","last_peered":"2024-08-17T17:57:09.263889+0000","last_clean":"2024-08-17T17:57:09.263889+0000","last_became_active":"2024-08-17T17:24:17.455396+0000","last_became_peered":"2024-08-17T17:24:17.455396+0000","last_unstale":"2024-08-17T17:57:09.263889+0000","last_undegraded":"2024-08-17T17:57:09.263889+0000","last_fullsized":"2024-08-17T17:57:09.263889+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:09.564331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1338,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.803266+0000","last_change":"2024-08-17T17:23:42.930464+0000","last_active":"2024-08-17T17:56:41.803266+0000","last_peered":"2024-08-17T17:56:41.803266+0000","last_clean":"2024-08-17T17:56:41.803266+0000","last_became_active":"2024-08-17T17:23:42.929949+0000","last_became_peered":"2024-08-17T17:23:42.929949+0000","last_unstale":"2024-08-17T17:56:41.803266+0000","last_undegraded":"2024-08-17T17:56:41.803266+0000","last_fullsized":"2024-08-17T17:56:41.803266+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:48:27.092339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1994,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.264389+0000","last_change":"2024-08-17T17:24:17.452009+0000","last_active":"2024-08-17T17:57:09.264389+0000","last_peered":"2024-08-17T17:57:09.264389+0000","last_clean":"2024-08-17T17:57:09.264389+0000","last_became_active":"2024-08-17T17:24:17.451492+0000","last_became_peered":"2024-08-17T17:24:17.451492+0000","last_unstale":"2024-08-17T17:57:09.264389+0000","last_undegraded":"2024-08-17T17:57:09.264389+0000","last_fullsized":"2024-08-17T17:57:09.264389+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:21:24.699633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2006,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.261635+0000","last_change":"2024-08-17T17:24:17.452025+0000","last_active":"2024-08-17T17:57:09.261635+0000","last_peered":"2024-08-17T17:57:09.261635+0000","last_clean":"2024-08-17T17:57:09.261635+0000","last_became_active":"2024-08-17T17:24:17.448029+0000","last_became_peered":"2024-08-17T17:24:17.448029+0000","last_unstale":"2024-08-17T17:57:09.261635+0000","last_undegraded":"2024-08-17T17:57:09.261635+0000","last_fullsized":"2024-08-17T17:57:09.261635+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:45.089346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2001,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.269390+0000","last_change":"2024-08-17T17:24:17.469096+0000","last_active":"2024-08-17T17:57:09.269390+0000","last_peered":"2024-08-17T17:57:09.269390+0000","last_clean":"2024-08-17T17:57:09.269390+0000","last_became_active":"2024-08-17T17:24:17.468669+0000","last_became_peered":"2024-08-17T17:24:17.468669+0000","last_unstale":"2024-08-17T17:57:09.269390+0000","last_undegraded":"2024-08-17T17:57:09.269390+0000","last_fullsized":"2024-08-17T17:57:09.269390+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:03:39.427341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1994,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.263399+0000","last_change":"2024-08-17T17:24:17.453946+0000","last_active":"2024-08-17T17:57:09.263399+0000","last_peered":"2024-08-17T17:57:09.263399+0000","last_clean":"2024-08-17T17:57:09.263399+0000","last_became_active":"2024-08-17T17:24:17.445004+0000","last_became_peered":"2024-08-17T17:24:17.445004+0000","last_unstale":"2024-08-17T17:57:09.263399+0000","last_undegraded":"2024-08-17T17:57:09.263399+0000","last_fullsized":"2024-08-17T17:57:09.263399+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:00:19.366629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1322,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.809585+0000","last_change":"2024-08-17T17:23:07.402338+0000","last_active":"2024-08-17T17:56:41.809585+0000","last_peered":"2024-08-17T17:56:41.809585+0000","last_clean":"2024-08-17T17:56:41.809585+0000","last_became_active":"2024-08-17T17:23:07.401996+0000","last_became_peered":"2024-08-17T17:23:07.401996+0000","last_unstale":"2024-08-17T17:56:41.809585+0000","last_undegraded":"2024-08-17T17:56:41.809585+0000","last_fullsized":"2024-08-17T17:56:41.809585+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:39:21.566786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1979,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.263566+0000","last_change":"2024-08-17T17:24:17.442338+0000","last_active":"2024-08-17T17:57:09.263566+0000","last_peered":"2024-08-17T17:57:09.263566+0000","last_clean":"2024-08-17T17:57:09.263566+0000","last_became_active":"2024-08-17T17:24:17.441948+0000","last_became_peered":"2024-08-17T17:24:17.441948+0000","last_unstale":"2024-08-17T17:57:09.263566+0000","last_undegraded":"2024-08-17T17:57:09.263566+0000","last_fullsized":"2024-08-17T17:57:09.263566+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:17.891492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1997,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.269836+0000","last_change":"2024-08-17T17:24:17.459083+0000","last_active":"2024-08-17T17:57:09.269836+0000","last_peered":"2024-08-17T17:57:09.269836+0000","last_clean":"2024-08-17T17:57:09.269836+0000","last_became_active":"2024-08-17T17:24:17.458759+0000","last_became_peered":"2024-08-17T17:24:17.458759+0000","last_unstale":"2024-08-17T17:57:09.269836+0000","last_undegraded":"2024-08-17T17:57:09.269836+0000","last_fullsized":"2024-08-17T17:57:09.269836+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:11:59.301864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1938,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.263884+0000","last_change":"2024-08-17T17:24:17.468086+0000","last_active":"2024-08-17T17:57:09.263884+0000","last_peered":"2024-08-17T17:57:09.263884+0000","last_clean":"2024-08-17T17:57:09.263884+0000","last_became_active":"2024-08-17T17:24:17.467522+0000","last_became_peered":"2024-08-17T17:24:17.467522+0000","last_unstale":"2024-08-17T17:57:09.263884+0000","last_undegraded":"2024-08-17T17:57:09.263884+0000","last_fullsized":"2024-08-17T17:57:09.263884+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:52:55.249649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1318,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.814325+0000","last_change":"2024-08-17T17:23:07.392447+0000","last_active":"2024-08-17T17:56:41.814325+0000","last_peered":"2024-08-17T17:56:41.814325+0000","last_clean":"2024-08-17T17:56:41.814325+0000","last_became_active":"2024-08-17T17:23:07.391765+0000","last_became_peered":"2024-08-17T17:23:07.391765+0000","last_unstale":"2024-08-17T17:56:41.814325+0000","last_undegraded":"2024-08-17T17:56:41.814325+0000","last_fullsized":"2024-08-17T17:56:41.814325+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:59:18.602346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":90,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.264480+0000","last_change":"2024-08-17T17:53:58.839889+0000","last_active":"2024-08-17T17:57:09.264480+0000","last_peered":"2024-08-17T17:57:09.264480+0000","last_clean":"2024-08-17T17:57:09.264480+0000","last_became_active":"2024-08-17T17:53:58.839537+0000","last_became_peered":"2024-08-17T17:53:58.839537+0000","last_unstale":"2024-08-17T17:57:09.264480+0000","last_undegraded":"2024-08-17T17:57:09.264480+0000","last_fullsized":"2024-08-17T17:57:09.264480+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:47.169954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2005,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.260606+0000","last_change":"2024-08-17T17:23:42.936147+0000","last_active":"2024-08-17T17:57:09.260606+0000","last_peered":"2024-08-17T17:57:09.260606+0000","last_clean":"2024-08-17T17:57:09.260606+0000","last_became_active":"2024-08-17T17:23:42.935859+0000","last_became_peered":"2024-08-17T17:23:42.935859+0000","last_unstale":"2024-08-17T17:57:09.260606+0000","last_undegraded":"2024-08-17T17:57:09.260606+0000","last_fullsized":"2024-08-17T17:57:09.260606+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:47:36.838706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1332,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:57.214450+0000","last_change":"2024-08-17T17:10:05.969131+0000","last_active":"2024-08-17T17:56:57.214450+0000","last_peered":"2024-08-17T17:56:57.214450+0000","last_clean":"2024-08-17T17:56:57.214450+0000","last_became_active":"2024-08-17T17:10:05.968856+0000","last_became_peered":"2024-08-17T17:10:05.968856+0000","last_unstale":"2024-08-17T17:56:57.214450+0000","last_undegraded":"2024-08-17T17:56:57.214450+0000","last_fullsized":"2024-08-17T17:56:57.214450+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:25:49.550065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1323,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:52.209183+0000","last_change":"2024-08-17T17:23:07.455919+0000","last_active":"2024-08-17T17:56:52.209183+0000","last_peered":"2024-08-17T17:56:52.209183+0000","last_clean":"2024-08-17T17:56:52.209183+0000","last_became_active":"2024-08-17T17:23:07.455284+0000","last_became_peered":"2024-08-17T17:23:07.455284+0000","last_unstale":"2024-08-17T17:56:52.209183+0000","last_undegraded":"2024-08-17T17:56:52.209183+0000","last_fullsized":"2024-08-17T17:56:52.209183+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:37:46.260775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1310,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.803014+0000","last_change":"2024-08-17T17:10:06.979492+0000","last_active":"2024-08-17T17:56:41.803014+0000","last_peered":"2024-08-17T17:56:41.803014+0000","last_clean":"2024-08-17T17:56:41.803014+0000","last_became_active":"2024-08-17T17:10:06.979158+0000","last_became_peered":"2024-08-17T17:10:06.979158+0000","last_unstale":"2024-08-17T17:56:41.803014+0000","last_undegraded":"2024-08-17T17:56:41.803014+0000","last_fullsized":"2024-08-17T17:56:41.803014+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:12:42.261365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1335,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.809650+0000","last_change":"2024-08-17T17:23:42.935099+0000","last_active":"2024-08-17T17:56:41.809650+0000","last_peered":"2024-08-17T17:56:41.809650+0000","last_clean":"2024-08-17T17:56:41.809650+0000","last_became_active":"2024-08-17T17:23:42.934501+0000","last_became_peered":"2024-08-17T17:23:42.934501+0000","last_unstale":"2024-08-17T17:56:41.809650+0000","last_undegraded":"2024-08-17T17:56:41.809650+0000","last_fullsized":"2024-08-17T17:56:41.809650+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:15:19.095713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":2003,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.269303+0000","last_change":"2024-08-17T17:23:07.436763+0000","last_active":"2024-08-17T17:57:09.269303+0000","last_peered":"2024-08-17T17:57:09.269303+0000","last_clean":"2024-08-17T17:57:09.269303+0000","last_became_active":"2024-08-17T17:23:07.434720+0000","last_became_peered":"2024-08-17T17:23:07.434720+0000","last_unstale":"2024-08-17T17:57:09.269303+0000","last_undegraded":"2024-08-17T17:57:09.269303+0000","last_fullsized":"2024-08-17T17:57:09.269303+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:10:20.594115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1974,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.263494+0000","last_change":"2024-08-17T17:24:17.442329+0000","last_active":"2024-08-17T17:57:09.263494+0000","last_peered":"2024-08-17T17:57:09.263494+0000","last_clean":"2024-08-17T17:57:09.263494+0000","last_became_active":"2024-08-17T17:24:17.441666+0000","last_became_peered":"2024-08-17T17:24:17.441666+0000","last_unstale":"2024-08-17T17:57:09.263494+0000","last_undegraded":"2024-08-17T17:57:09.263494+0000","last_fullsized":"2024-08-17T17:57:09.263494+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:43:06.033194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1987,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.265004+0000","last_change":"2024-08-17T17:23:07.428296+0000","last_active":"2024-08-17T17:57:09.265004+0000","last_peered":"2024-08-17T17:57:09.265004+0000","last_clean":"2024-08-17T17:57:09.265004+0000","last_became_active":"2024-08-17T17:23:07.428010+0000","last_became_peered":"2024-08-17T17:23:07.428010+0000","last_unstale":"2024-08-17T17:57:09.265004+0000","last_undegraded":"2024-08-17T17:57:09.265004+0000","last_fullsized":"2024-08-17T17:57:09.265004+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:45.459212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":1975,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.264465+0000","last_change":"2024-08-17T17:24:17.481368+0000","last_active":"2024-08-17T17:57:09.264465+0000","last_peered":"2024-08-17T17:57:09.264465+0000","last_clean":"2024-08-17T17:57:09.264465+0000","last_became_active":"2024-08-17T17:24:17.477509+0000","last_became_peered":"2024-08-17T17:24:17.477509+0000","last_unstale":"2024-08-17T17:57:09.264465+0000","last_undegraded":"2024-08-17T17:57:09.264465+0000","last_fullsized":"2024-08-17T17:57:09.264465+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:05:35.558004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1989,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.270905+0000","last_change":"2024-08-17T17:22:32.884274+0000","last_active":"2024-08-17T17:57:09.270905+0000","last_peered":"2024-08-17T17:57:09.270905+0000","last_clean":"2024-08-17T17:57:09.270905+0000","last_became_active":"2024-08-17T17:22:32.874667+0000","last_became_peered":"2024-08-17T17:22:32.874667+0000","last_unstale":"2024-08-17T17:57:09.270905+0000","last_undegraded":"2024-08-17T17:57:09.270905+0000","last_fullsized":"2024-08-17T17:57:09.270905+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:22:05.834587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1332,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:56.930706+0000","last_change":"2024-08-17T17:22:32.870569+0000","last_active":"2024-08-17T17:56:56.930706+0000","last_peered":"2024-08-17T17:56:56.930706+0000","last_clean":"2024-08-17T17:56:56.930706+0000","last_became_active":"2024-08-17T17:22:32.869335+0000","last_became_peered":"2024-08-17T17:22:32.869335+0000","last_unstale":"2024-08-17T17:56:56.930706+0000","last_undegraded":"2024-08-17T17:56:56.930706+0000","last_fullsized":"2024-08-17T17:56:56.930706+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:24.969174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2009,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.262507+0000","last_change":"2024-08-17T17:23:42.935825+0000","last_active":"2024-08-17T17:57:09.262507+0000","last_peered":"2024-08-17T17:57:09.262507+0000","last_clean":"2024-08-17T17:57:09.262507+0000","last_became_active":"2024-08-17T17:23:42.935150+0000","last_became_peered":"2024-08-17T17:23:42.935150+0000","last_unstale":"2024-08-17T17:57:09.262507+0000","last_undegraded":"2024-08-17T17:57:09.262507+0000","last_fullsized":"2024-08-17T17:57:09.262507+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:57:19.577644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1997,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.255902+0000","last_change":"2024-08-17T17:23:07.431586+0000","last_active":"2024-08-17T17:57:09.255902+0000","last_peered":"2024-08-17T17:57:09.255902+0000","last_clean":"2024-08-17T17:57:09.255902+0000","last_became_active":"2024-08-17T17:23:07.431301+0000","last_became_peered":"2024-08-17T17:23:07.431301+0000","last_unstale":"2024-08-17T17:57:09.255902+0000","last_undegraded":"2024-08-17T17:57:09.255902+0000","last_fullsized":"2024-08-17T17:57:09.255902+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:28:18.706489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2025,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.262275+0000","last_change":"2024-08-17T17:23:07.420550+0000","last_active":"2024-08-17T17:57:09.262275+0000","last_peered":"2024-08-17T17:57:09.262275+0000","last_clean":"2024-08-17T17:57:09.262275+0000","last_became_active":"2024-08-17T17:23:07.420223+0000","last_became_peered":"2024-08-17T17:23:07.420223+0000","last_unstale":"2024-08-17T17:57:09.262275+0000","last_undegraded":"2024-08-17T17:57:09.262275+0000","last_fullsized":"2024-08-17T17:57:09.262275+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:33:46.996371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1336,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.814480+0000","last_change":"2024-08-17T17:24:17.426267+0000","last_active":"2024-08-17T17:56:41.814480+0000","last_peered":"2024-08-17T17:56:41.814480+0000","last_clean":"2024-08-17T17:56:41.814480+0000","last_became_active":"2024-08-17T17:24:17.425925+0000","last_became_peered":"2024-08-17T17:24:17.425925+0000","last_unstale":"2024-08-17T17:56:41.814480+0000","last_undegraded":"2024-08-17T17:56:41.814480+0000","last_fullsized":"2024-08-17T17:56:41.814480+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:24:25.017856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1332,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:42.136712+0000","last_change":"2024-08-17T17:24:17.474138+0000","last_active":"2024-08-17T17:56:42.136712+0000","last_peered":"2024-08-17T17:56:42.136712+0000","last_clean":"2024-08-17T17:56:42.136712+0000","last_became_active":"2024-08-17T17:24:17.473708+0000","last_became_peered":"2024-08-17T17:24:17.473708+0000","last_unstale":"2024-08-17T17:56:42.136712+0000","last_undegraded":"2024-08-17T17:56:42.136712+0000","last_fullsized":"2024-08-17T17:56:42.136712+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:10:57.768067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1326,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.803306+0000","last_change":"2024-08-17T17:22:32.871533+0000","last_active":"2024-08-17T17:56:41.803306+0000","last_peered":"2024-08-17T17:56:41.803306+0000","last_clean":"2024-08-17T17:56:41.803306+0000","last_became_active":"2024-08-17T17:22:32.868226+0000","last_became_peered":"2024-08-17T17:22:32.868226+0000","last_unstale":"2024-08-17T17:56:41.803306+0000","last_undegraded":"2024-08-17T17:56:41.803306+0000","last_fullsized":"2024-08-17T17:56:41.803306+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:00:02.629455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":1336,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:42.136679+0000","last_change":"2024-08-17T17:23:07.391021+0000","last_active":"2024-08-17T17:56:42.136679+0000","last_peered":"2024-08-17T17:56:42.136679+0000","last_clean":"2024-08-17T17:56:42.136679+0000","last_became_active":"2024-08-17T17:23:07.390551+0000","last_became_peered":"2024-08-17T17:23:07.390551+0000","last_unstale":"2024-08-17T17:56:42.136679+0000","last_undegraded":"2024-08-17T17:56:42.136679+0000","last_fullsized":"2024-08-17T17:56:42.136679+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:06.239013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001023532,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2008,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.261328+0000","last_change":"2024-08-17T17:23:07.406362+0000","last_active":"2024-08-17T17:57:09.261328+0000","last_peered":"2024-08-17T17:57:09.261328+0000","last_clean":"2024-08-17T17:57:09.261328+0000","last_became_active":"2024-08-17T17:23:07.406071+0000","last_became_peered":"2024-08-17T17:23:07.406071+0000","last_unstale":"2024-08-17T17:57:09.261328+0000","last_undegraded":"2024-08-17T17:57:09.261328+0000","last_fullsized":"2024-08-17T17:57:09.261328+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:45:47.383432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1988,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.255300+0000","last_change":"2024-08-17T17:23:42.910246+0000","last_active":"2024-08-17T17:57:09.255300+0000","last_peered":"2024-08-17T17:57:09.255300+0000","last_clean":"2024-08-17T17:57:09.255300+0000","last_became_active":"2024-08-17T17:23:42.909929+0000","last_became_peered":"2024-08-17T17:23:42.909929+0000","last_unstale":"2024-08-17T17:57:09.255300+0000","last_undegraded":"2024-08-17T17:57:09.255300+0000","last_fullsized":"2024-08-17T17:57:09.255300+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:14:45.210558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2018,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.262960+0000","last_change":"2024-08-17T17:24:17.438891+0000","last_active":"2024-08-17T17:57:09.262960+0000","last_peered":"2024-08-17T17:57:09.262960+0000","last_clean":"2024-08-17T17:57:09.262960+0000","last_became_active":"2024-08-17T17:24:17.437714+0000","last_became_peered":"2024-08-17T17:24:17.437714+0000","last_unstale":"2024-08-17T17:57:09.262960+0000","last_undegraded":"2024-08-17T17:57:09.262960+0000","last_fullsized":"2024-08-17T17:57:09.262960+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:58.825017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2010,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.269472+0000","last_change":"2024-08-17T17:23:42.908634+0000","last_active":"2024-08-17T17:57:09.269472+0000","last_peered":"2024-08-17T17:57:09.269472+0000","last_clean":"2024-08-17T17:57:09.269472+0000","last_became_active":"2024-08-17T17:23:42.907675+0000","last_became_peered":"2024-08-17T17:23:42.907675+0000","last_unstale":"2024-08-17T17:57:09.269472+0000","last_undegraded":"2024-08-17T17:57:09.269472+0000","last_fullsized":"2024-08-17T17:57:09.269472+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:03:37.612222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2000,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.405182+0000","last_change":"2024-08-17T17:24:17.457559+0000","last_active":"2024-08-17T17:57:09.405182+0000","last_peered":"2024-08-17T17:57:09.405182+0000","last_clean":"2024-08-17T17:57:09.405182+0000","last_became_active":"2024-08-17T17:24:17.457262+0000","last_became_peered":"2024-08-17T17:24:17.457262+0000","last_unstale":"2024-08-17T17:57:09.405182+0000","last_undegraded":"2024-08-17T17:57:09.405182+0000","last_fullsized":"2024-08-17T17:57:09.405182+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:53:38.137277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":722,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.255561+0000","last_change":"2024-08-17T17:53:58.844594+0000","last_active":"2024-08-17T17:57:09.255561+0000","last_peered":"2024-08-17T17:57:09.255561+0000","last_clean":"2024-08-17T17:57:09.255561+0000","last_became_active":"2024-08-17T17:53:58.844085+0000","last_became_peered":"2024-08-17T17:53:58.844085+0000","last_unstale":"2024-08-17T17:57:09.255561+0000","last_undegraded":"2024-08-17T17:57:09.255561+0000","last_fullsized":"2024-08-17T17:57:09.255561+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:58:50.195797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1307,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.809424+0000","last_change":"2024-08-17T17:10:06.995023+0000","last_active":"2024-08-17T17:56:41.809424+0000","last_peered":"2024-08-17T17:56:41.809424+0000","last_clean":"2024-08-17T17:56:41.809424+0000","last_became_active":"2024-08-17T17:10:06.994757+0000","last_became_peered":"2024-08-17T17:10:06.994757+0000","last_unstale":"2024-08-17T17:56:41.809424+0000","last_undegraded":"2024-08-17T17:56:41.809424+0000","last_fullsized":"2024-08-17T17:56:41.809424+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:31:52.163518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1940,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.789538+0000","last_change":"2024-08-17T17:24:17.468443+0000","last_active":"2024-08-17T17:57:09.789538+0000","last_peered":"2024-08-17T17:57:09.789538+0000","last_clean":"2024-08-17T17:57:09.789538+0000","last_became_active":"2024-08-17T17:24:17.467394+0000","last_became_peered":"2024-08-17T17:24:17.467394+0000","last_unstale":"2024-08-17T17:57:09.789538+0000","last_undegraded":"2024-08-17T17:57:09.789538+0000","last_fullsized":"2024-08-17T17:57:09.789538+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:13.024253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":1335,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:47.138475+0000","last_change":"2024-08-17T17:23:42.920552+0000","last_active":"2024-08-17T17:56:47.138475+0000","last_peered":"2024-08-17T17:56:47.138475+0000","last_clean":"2024-08-17T17:56:47.138475+0000","last_became_active":"2024-08-17T17:23:42.915824+0000","last_became_peered":"2024-08-17T17:23:42.915824+0000","last_unstale":"2024-08-17T17:56:47.138475+0000","last_undegraded":"2024-08-17T17:56:47.138475+0000","last_fullsized":"2024-08-17T17:56:47.138475+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:25:18.769504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041737399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":692,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.269110+0000","last_change":"2024-08-17T17:53:58.844782+0000","last_active":"2024-08-17T17:57:09.269110+0000","last_peered":"2024-08-17T17:57:09.269110+0000","last_clean":"2024-08-17T17:57:09.269110+0000","last_became_active":"2024-08-17T17:53:58.843133+0000","last_became_peered":"2024-08-17T17:53:58.843133+0000","last_unstale":"2024-08-17T17:57:09.269110+0000","last_undegraded":"2024-08-17T17:57:09.269110+0000","last_fullsized":"2024-08-17T17:57:09.269110+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T17:06:21.174471+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:30:14.411310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1987,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.261251+0000","last_change":"2024-08-17T17:24:17.455710+0000","last_active":"2024-08-17T17:57:09.261251+0000","last_peered":"2024-08-17T17:57:09.261251+0000","last_clean":"2024-08-17T17:57:09.261251+0000","last_became_active":"2024-08-17T17:24:17.452989+0000","last_became_peered":"2024-08-17T17:24:17.452989+0000","last_unstale":"2024-08-17T17:57:09.261251+0000","last_undegraded":"2024-08-17T17:57:09.261251+0000","last_fullsized":"2024-08-17T17:57:09.261251+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:58:07.894968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1981,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.264587+0000","last_change":"2024-08-17T17:24:17.453639+0000","last_active":"2024-08-17T17:57:09.264587+0000","last_peered":"2024-08-17T17:57:09.264587+0000","last_clean":"2024-08-17T17:57:09.264587+0000","last_became_active":"2024-08-17T17:24:17.443638+0000","last_became_peered":"2024-08-17T17:24:17.443638+0000","last_unstale":"2024-08-17T17:57:09.264587+0000","last_undegraded":"2024-08-17T17:57:09.264587+0000","last_fullsized":"2024-08-17T17:57:09.264587+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:42.107479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1323,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.802999+0000","last_change":"2024-08-17T17:23:07.451866+0000","last_active":"2024-08-17T17:56:41.802999+0000","last_peered":"2024-08-17T17:56:41.802999+0000","last_clean":"2024-08-17T17:56:41.802999+0000","last_became_active":"2024-08-17T17:23:07.451553+0000","last_became_peered":"2024-08-17T17:23:07.451553+0000","last_unstale":"2024-08-17T17:56:41.802999+0000","last_undegraded":"2024-08-17T17:56:41.802999+0000","last_fullsized":"2024-08-17T17:56:41.802999+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:37:23.930651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1325,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.803107+0000","last_change":"2024-08-17T17:23:42.915851+0000","last_active":"2024-08-17T17:56:41.803107+0000","last_peered":"2024-08-17T17:56:41.803107+0000","last_clean":"2024-08-17T17:56:41.803107+0000","last_became_active":"2024-08-17T17:23:42.915552+0000","last_became_peered":"2024-08-17T17:23:42.915552+0000","last_unstale":"2024-08-17T17:56:41.803107+0000","last_undegraded":"2024-08-17T17:56:41.803107+0000","last_fullsized":"2024-08-17T17:56:41.803107+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:45.197876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1338,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.803043+0000","last_change":"2024-08-17T17:23:42.930803+0000","last_active":"2024-08-17T17:56:41.803043+0000","last_peered":"2024-08-17T17:56:41.803043+0000","last_clean":"2024-08-17T17:56:41.803043+0000","last_became_active":"2024-08-17T17:23:42.930458+0000","last_became_peered":"2024-08-17T17:23:42.930458+0000","last_unstale":"2024-08-17T17:56:41.803043+0000","last_undegraded":"2024-08-17T17:56:41.803043+0000","last_fullsized":"2024-08-17T17:56:41.803043+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:40.816706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1322,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.809047+0000","last_change":"2024-08-17T17:23:07.401624+0000","last_active":"2024-08-17T17:56:41.809047+0000","last_peered":"2024-08-17T17:56:41.809047+0000","last_clean":"2024-08-17T17:56:41.809047+0000","last_became_active":"2024-08-17T17:23:07.400222+0000","last_became_peered":"2024-08-17T17:23:07.400222+0000","last_unstale":"2024-08-17T17:56:41.809047+0000","last_undegraded":"2024-08-17T17:56:41.809047+0000","last_fullsized":"2024-08-17T17:56:41.809047+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:07.069989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1323,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.802844+0000","last_change":"2024-08-17T17:24:17.470809+0000","last_active":"2024-08-17T17:56:41.802844+0000","last_peered":"2024-08-17T17:56:41.802844+0000","last_clean":"2024-08-17T17:56:41.802844+0000","last_became_active":"2024-08-17T17:24:17.469753+0000","last_became_peered":"2024-08-17T17:24:17.469753+0000","last_unstale":"2024-08-17T17:56:41.802844+0000","last_undegraded":"2024-08-17T17:56:41.802844+0000","last_fullsized":"2024-08-17T17:56:41.802844+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:00.049425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":1974,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.262371+0000","last_change":"2024-08-17T17:24:17.461258+0000","last_active":"2024-08-17T17:57:09.262371+0000","last_peered":"2024-08-17T17:57:09.262371+0000","last_clean":"2024-08-17T17:57:09.262371+0000","last_became_active":"2024-08-17T17:24:17.460895+0000","last_became_peered":"2024-08-17T17:24:17.460895+0000","last_unstale":"2024-08-17T17:57:09.262371+0000","last_undegraded":"2024-08-17T17:57:09.262371+0000","last_fullsized":"2024-08-17T17:57:09.262371+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:43:24.208259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2013,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.270673+0000","last_change":"2024-08-17T17:23:42.901638+0000","last_active":"2024-08-17T17:57:09.270673+0000","last_peered":"2024-08-17T17:57:09.270673+0000","last_clean":"2024-08-17T17:57:09.270673+0000","last_became_active":"2024-08-17T17:23:42.901335+0000","last_became_peered":"2024-08-17T17:23:42.901335+0000","last_unstale":"2024-08-17T17:57:09.270673+0000","last_undegraded":"2024-08-17T17:57:09.270673+0000","last_fullsized":"2024-08-17T17:57:09.270673+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:25:35.332839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1319,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.814497+0000","last_change":"2024-08-17T17:24:17.473913+0000","last_active":"2024-08-17T17:56:41.814497+0000","last_peered":"2024-08-17T17:56:41.814497+0000","last_clean":"2024-08-17T17:56:41.814497+0000","last_became_active":"2024-08-17T17:24:17.473195+0000","last_became_peered":"2024-08-17T17:24:17.473195+0000","last_unstale":"2024-08-17T17:56:41.814497+0000","last_undegraded":"2024-08-17T17:56:41.814497+0000","last_fullsized":"2024-08-17T17:56:41.814497+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:12.998033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1321,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.808884+0000","last_change":"2024-08-17T17:23:07.402010+0000","last_active":"2024-08-17T17:56:41.808884+0000","last_peered":"2024-08-17T17:56:41.808884+0000","last_clean":"2024-08-17T17:56:41.808884+0000","last_became_active":"2024-08-17T17:23:07.400849+0000","last_became_peered":"2024-08-17T17:23:07.400849+0000","last_unstale":"2024-08-17T17:56:41.808884+0000","last_undegraded":"2024-08-17T17:56:41.808884+0000","last_fullsized":"2024-08-17T17:56:41.808884+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:16:02.382257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":57,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.814226+0000","last_change":"2024-08-17T17:53:58.836811+0000","last_active":"2024-08-17T17:56:41.814226+0000","last_peered":"2024-08-17T17:56:41.814226+0000","last_clean":"2024-08-17T17:56:41.814226+0000","last_became_active":"2024-08-17T17:53:58.836139+0000","last_became_peered":"2024-08-17T17:53:58.836139+0000","last_unstale":"2024-08-17T17:56:41.814226+0000","last_undegraded":"2024-08-17T17:56:41.814226+0000","last_fullsized":"2024-08-17T17:56:41.814226+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:09:18.260164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1339,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.802811+0000","last_change":"2024-08-17T17:24:17.460203+0000","last_active":"2024-08-17T17:56:41.802811+0000","last_peered":"2024-08-17T17:56:41.802811+0000","last_clean":"2024-08-17T17:56:41.802811+0000","last_became_active":"2024-08-17T17:24:17.459682+0000","last_became_peered":"2024-08-17T17:24:17.459682+0000","last_unstale":"2024-08-17T17:56:41.802811+0000","last_undegraded":"2024-08-17T17:56:41.802811+0000","last_fullsized":"2024-08-17T17:56:41.802811+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:58.703700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":1319,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.812894+0000","last_change":"2024-08-17T17:23:07.454042+0000","last_active":"2024-08-17T17:56:41.812894+0000","last_peered":"2024-08-17T17:56:41.812894+0000","last_clean":"2024-08-17T17:56:41.812894+0000","last_became_active":"2024-08-17T17:23:07.453712+0000","last_became_peered":"2024-08-17T17:23:07.453712+0000","last_unstale":"2024-08-17T17:56:41.812894+0000","last_undegraded":"2024-08-17T17:56:41.812894+0000","last_fullsized":"2024-08-17T17:56:41.812894+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:03:07.643456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1304,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.813981+0000","last_change":"2024-08-17T17:10:06.994799+0000","last_active":"2024-08-17T17:56:41.813981+0000","last_peered":"2024-08-17T17:56:41.813981+0000","last_clean":"2024-08-17T17:56:41.813981+0000","last_became_active":"2024-08-17T17:10:06.994150+0000","last_became_peered":"2024-08-17T17:10:06.994150+0000","last_unstale":"2024-08-17T17:56:41.813981+0000","last_undegraded":"2024-08-17T17:56:41.813981+0000","last_fullsized":"2024-08-17T17:56:41.813981+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:50:47.293091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":1317,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.812569+0000","last_change":"2024-08-17T17:22:32.873334+0000","last_active":"2024-08-17T17:56:41.812569+0000","last_peered":"2024-08-17T17:56:41.812569+0000","last_clean":"2024-08-17T17:56:41.812569+0000","last_became_active":"2024-08-17T17:22:32.870147+0000","last_became_peered":"2024-08-17T17:22:32.870147+0000","last_unstale":"2024-08-17T17:56:41.812569+0000","last_undegraded":"2024-08-17T17:56:41.812569+0000","last_fullsized":"2024-08-17T17:56:41.812569+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:18:03.750592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1329,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:42.020292+0000","last_change":"2024-08-17T17:22:32.847989+0000","last_active":"2024-08-17T17:56:42.020292+0000","last_peered":"2024-08-17T17:56:42.020292+0000","last_clean":"2024-08-17T17:56:42.020292+0000","last_became_active":"2024-08-17T17:22:32.846593+0000","last_became_peered":"2024-08-17T17:22:32.846593+0000","last_unstale":"2024-08-17T17:56:42.020292+0000","last_undegraded":"2024-08-17T17:56:42.020292+0000","last_fullsized":"2024-08-17T17:56:42.020292+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:33:50.370124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1978,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.264381+0000","last_change":"2024-08-17T17:24:17.451528+0000","last_active":"2024-08-17T17:57:09.264381+0000","last_peered":"2024-08-17T17:57:09.264381+0000","last_clean":"2024-08-17T17:57:09.264381+0000","last_became_active":"2024-08-17T17:24:17.451246+0000","last_became_peered":"2024-08-17T17:24:17.451246+0000","last_unstale":"2024-08-17T17:57:09.264381+0000","last_undegraded":"2024-08-17T17:57:09.264381+0000","last_fullsized":"2024-08-17T17:57:09.264381+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:38:18.176559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":45,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.802776+0000","last_change":"2024-08-17T17:53:58.838279+0000","last_active":"2024-08-17T17:56:41.802776+0000","last_peered":"2024-08-17T17:56:41.802776+0000","last_clean":"2024-08-17T17:56:41.802776+0000","last_became_active":"2024-08-17T17:53:58.837978+0000","last_became_peered":"2024-08-17T17:53:58.837978+0000","last_unstale":"2024-08-17T17:56:41.802776+0000","last_undegraded":"2024-08-17T17:56:41.802776+0000","last_fullsized":"2024-08-17T17:56:41.802776+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:06.019142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1997,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.269216+0000","last_change":"2024-08-17T17:24:17.460371+0000","last_active":"2024-08-17T17:57:09.269216+0000","last_peered":"2024-08-17T17:57:09.269216+0000","last_clean":"2024-08-17T17:57:09.269216+0000","last_became_active":"2024-08-17T17:24:17.460074+0000","last_became_peered":"2024-08-17T17:24:17.460074+0000","last_unstale":"2024-08-17T17:57:09.269216+0000","last_undegraded":"2024-08-17T17:57:09.269216+0000","last_fullsized":"2024-08-17T17:57:09.269216+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:08:48.820168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1997,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.400074+0000","last_change":"2024-08-17T17:24:17.457942+0000","last_active":"2024-08-17T17:57:09.400074+0000","last_peered":"2024-08-17T17:57:09.400074+0000","last_clean":"2024-08-17T17:57:09.400074+0000","last_became_active":"2024-08-17T17:24:17.456475+0000","last_became_peered":"2024-08-17T17:24:17.456475+0000","last_unstale":"2024-08-17T17:57:09.400074+0000","last_undegraded":"2024-08-17T17:57:09.400074+0000","last_fullsized":"2024-08-17T17:57:09.400074+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:12.325657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1317,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:41.814427+0000","last_change":"2024-08-17T17:24:17.451287+0000","last_active":"2024-08-17T17:56:41.814427+0000","last_peered":"2024-08-17T17:56:41.814427+0000","last_clean":"2024-08-17T17:56:41.814427+0000","last_became_active":"2024-08-17T17:24:17.450951+0000","last_became_peered":"2024-08-17T17:24:17.450951+0000","last_unstale":"2024-08-17T17:56:41.814427+0000","last_undegraded":"2024-08-17T17:56:41.814427+0000","last_fullsized":"2024-08-17T17:56:41.814427+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:13:37.590108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":1319,"reported_epoch":1350,"state":"active+clean","last_fresh":"2024-08-17T17:56:51.929244+0000","last_change":"2024-08-17T17:23:42.907970+0000","last_active":"2024-08-17T17:56:51.929244+0000","last_peered":"2024-08-17T17:56:51.929244+0000","last_clean":"2024-08-17T17:56:51.929244+0000","last_became_active":"2024-08-17T17:23:42.907595+0000","last_became_peered":"2024-08-17T17:23:42.907595+0000","last_unstale":"2024-08-17T17:56:51.929244+0000","last_undegraded":"2024-08-17T17:56:51.929244+0000","last_fullsized":"2024-08-17T17:56:51.929244+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:11:46.130004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1979,"reported_epoch":1351,"state":"active+clean","last_fresh":"2024-08-17T17:57:09.266290+0000","last_change":"2024-08-17T17:24:17.486591+0000","last_active":"2024-08-17T17:57:09.266290+0000","last_peered":"2024-08-17T17:57:09.266290+0000","last_clean":"2024-08-17T17:57:09.266290+0000","last_became_active":"2024-08-17T17:24:17.486320+0000","last_became_peered":"2024-08-17T17:24:17.486320+0000","last_unstale":"2024-08-17T17:57:09.266290+0000","last_undegraded":"2024-08-17T17:57:09.266290+0000","last_fullsized":"2024-08-17T17:57:09.266290+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:03.249803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":53,"num_pg":26,"stat_sum":{"num_bytes":504725528,"num_objects":7703,"num_object_clones":0,"num_object_copies":23109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4540,"num_objects_misplaced":2270,"num_objects_unfound":0,"num_objects_dirty":7703,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":202911,"num_write_kb":5526497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15573,"num_bytes_recovered":936378368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2269065216,"data_stored":2269053000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":215566,"ondisk_log_size":215566,"up":78,"acting":75,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234574,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":258528,"kb_used_data":74904,"kb_used_omap":1,"kb_used_meta":183614,"kb_avail":93486624,"statfs":{"total":95995035648,"available":95730302976,"internally_reserved":0,"allocated":76701696,"data_stored":72562972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":188021195},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496212,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":411672,"kb_used_data":159888,"kb_used_omap":1,"kb_used_meta":251710,"kb_avail":93333480,"statfs":{"total":95995035648,"available":95573483520,"internally_reserved":0,"allocated":163725312,"data_stored":159594653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":257751495},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,0,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565453,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":611848,"kb_used_data":439356,"kb_used_omap":72,"kb_used_meta":172279,"kb_avail":93133304,"statfs":{"total":95995035648,"available":95368503296,"internally_reserved":0,"allocated":449900544,"data_stored":445785078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":176414190},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":674,"seq":2894807957908,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":497868,"kb_used_data":285252,"kb_used_omap":74,"kb_used_meta":212469,"kb_avail":93247284,"statfs":{"total":95995035648,"available":95485218816,"internally_reserved":0,"allocated":292098048,"data_stored":287974940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76116,"internal_metadata":217568940},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743186843,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":756556,"kb_used_data":435584,"kb_used_omap":66,"kb_used_meta":320701,"kb_avail":92988596,"statfs":{"total":95995035648,"available":95220322304,"internally_reserved":0,"allocated":446038016,"data_stored":441918407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":328398507},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448482,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":475788,"kb_used_data":227332,"kb_used_omap":75,"kb_used_meta":248372,"kb_avail":93269364,"statfs":{"total":95995035648,"available":95507828736,"internally_reserved":0,"allocated":232787968,"data_stored":228665368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":254333119},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757848,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":484368,"kb_used_data":246408,"kb_used_omap":1,"kb_used_meta":237822,"kb_avail":93260784,"statfs":{"total":95995035648,"available":95499042816,"internally_reserved":0,"allocated":252321792,"data_stored":248194631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":243530182},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019485,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":861208,"kb_used_data":505296,"kb_used_omap":1,"kb_used_meta":355902,"kb_avail":92883944,"statfs":{"total":95995035648,"available":95113158656,"internally_reserved":0,"allocated":517423104,"data_stored":513291958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":364444102},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":209125376,"data_stored":209125376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":418451456,"data_stored":418447384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":275189760,"data_stored":275185688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":425263104,"data_stored":425263104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":507445248,"data_stored":507445248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":219549696,"data_stored":219545624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":153354240,"data_stored":153354240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":60686336,"data_stored":60686336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:57:20.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:20.339 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:57:20.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:20.349 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:20.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:20.450 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:20.477 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:20.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:20.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:20.543 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:20 smithi136 ceph-mon[109861]: pgmap v2245: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 96 active+clean; 232 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 764 op/s; 4288/11199 objects degraded (38.289%); 2144/11199 objects misplaced (19.145%); 1.5 MiB/s, 127 objects/s recovering 2024-08-17T17:57:20.543 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:20 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 4288/11199 objects degraded (38.289%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:57:20.544 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:20 smithi136 ceph-mon[115078]: pgmap v2245: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 96 active+clean; 232 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 764 op/s; 4288/11199 objects degraded (38.289%); 2144/11199 objects misplaced (19.145%); 1.5 MiB/s, 127 objects/s recovering 2024-08-17T17:57:20.544 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:20 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 4288/11199 objects degraded (38.289%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:57:20.579 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:20.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:20.627 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:20.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:20.728 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:20 smithi165 ceph-mon[105517]: pgmap v2245: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 96 active+clean; 232 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 764 op/s; 4288/11199 objects degraded (38.289%); 2144/11199 objects misplaced (19.145%); 1.5 MiB/s, 127 objects/s recovering 2024-08-17T17:57:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:20 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 4288/11199 objects degraded (38.289%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:57:20.783 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:20.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:20.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:20.885 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:20.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:20.923 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:20.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:21.024 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:21.056 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:21.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:21.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:21.157 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:21.204 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:21.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:21.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:21.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:21.338 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:21.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:21.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:21.440 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:21.491 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:21.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:21.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:21.565 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Removed 124283 objects 2024-08-17T17:57:21.565 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Clean up completed and total clean up time :153.841 2024-08-17T17:57:21.593 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:21.615 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:21.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:21.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:21.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:57:21.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:57:21.699 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-08-17T17:57:21.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-08-17T17:57:21.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:57:21.766 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:21.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:21.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:21.868 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:21.893 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:21.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:21.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:21.995 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:22.017 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:22.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:22.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:22.119 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:22.140 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:22.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:22.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:22.241 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:22.264 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:22.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:22.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:22.366 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:22.388 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:22.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:22.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:22.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:22.513 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:22.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:22.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:22.614 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:22.636 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:22.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:22.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:22.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:22 smithi136 ceph-mon[109861]: pgmap v2246: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 96 active+clean; 232 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 668 op/s; 4288/11199 objects degraded (38.289%); 2144/11199 objects misplaced (19.145%); 1.3 MiB/s, 111 objects/s recovering 2024-08-17T17:57:22.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:22 smithi136 ceph-mon[115078]: pgmap v2246: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 96 active+clean; 232 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 668 op/s; 4288/11199 objects degraded (38.289%); 2144/11199 objects misplaced (19.145%); 1.3 MiB/s, 111 objects/s recovering 2024-08-17T17:57:22.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:22.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:22 smithi165 ceph-mon[105517]: pgmap v2246: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 96 active+clean; 232 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 668 op/s; 4288/11199 objects degraded (38.289%); 2144/11199 objects misplaced (19.145%); 1.3 MiB/s, 111 objects/s recovering 2024-08-17T17:57:22.761 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:22.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:22.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:22.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:22.884 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:22.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:22.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:22.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:23.010 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:23.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:23.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:23.111 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:23.133 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:23.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:23.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:23.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:23.257 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:23.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:23.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:23.323 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:57:23.359 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:23.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:23.382 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:23.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:23.483 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:23.506 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:23.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:23.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:23.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:23.631 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:23.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:23.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:23.732 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:23.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:23.756 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:23.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:23.857 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:23.880 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:23.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:23.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:23.981 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:24.005 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:24.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:24.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:24.107 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:24.130 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:24.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:24.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:24.232 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:24.257 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:24.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:24.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:24.359 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:24.381 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:24.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:24.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:24.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:24.507 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:24.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:24.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:24.609 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:24.631 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:24.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:24.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:24.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:24 smithi136 ceph-mon[109861]: pgmap v2247: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 96 active+clean; 7.9 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 937 op/s; 4068/435 objects degraded (935.172%); 2034/435 objects misplaced (467.586%); 1.8 MiB/s, 163 objects/s recovering 2024-08-17T17:57:24.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:24 smithi136 ceph-mon[115078]: pgmap v2247: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 96 active+clean; 7.9 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 937 op/s; 4068/435 objects degraded (935.172%); 2034/435 objects misplaced (467.586%); 1.8 MiB/s, 163 objects/s recovering 2024-08-17T17:57:24.733 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:24.754 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:24.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:24.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:24.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:24 smithi165 ceph-mon[105517]: pgmap v2247: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 96 active+clean; 7.9 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 937 op/s; 4068/435 objects degraded (935.172%); 2034/435 objects misplaced (467.586%); 1.8 MiB/s, 163 objects/s recovering 2024-08-17T17:57:24.856 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:24.877 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:24.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:24.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:24.979 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:25.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:25.003 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:25.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:25.104 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:25.125 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:25.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:25.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:25.227 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:25.249 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:25.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:25.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:25.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:25.374 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:25.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:25.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:25.475 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:25.498 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:25.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:25.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:25.600 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:25.621 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:25.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:25.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:25.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:25 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 4068/435 objects degraded (935.172%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:57:25.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:25 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 4068/435 objects degraded (935.172%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:57:25.722 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:25.744 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:25.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:25.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:25.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:25 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 4068/435 objects degraded (935.172%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:57:25.846 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:25.869 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:25.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:25.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:25.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:25.992 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:25.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:25.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:26.094 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:26.117 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:26.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:26.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:26.218 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:26.240 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:26.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:26.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:26.342 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:26.364 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:26.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:26.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:26.465 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:26.487 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:26.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:26.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:26.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:26.610 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:26.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:26.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:26.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:26 smithi136 ceph-mon[109861]: pgmap v2248: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 96 active+clean; 7.9 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 633 op/s; 4068/435 objects degraded (935.172%); 2034/435 objects misplaced (467.586%); 1.2 MiB/s, 109 objects/s recovering 2024-08-17T17:57:26.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:26 smithi136 ceph-mon[109861]: osdmap e1352: 8 total, 8 up, 8 in 2024-08-17T17:57:26.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:26 smithi136 ceph-mon[115078]: pgmap v2248: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 96 active+clean; 7.9 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 633 op/s; 4068/435 objects degraded (935.172%); 2034/435 objects misplaced (467.586%); 1.2 MiB/s, 109 objects/s recovering 2024-08-17T17:57:26.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:26 smithi136 ceph-mon[115078]: osdmap e1352: 8 total, 8 up, 8 in 2024-08-17T17:57:26.712 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:26.734 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:26.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:26.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:26.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:26 smithi165 ceph-mon[105517]: pgmap v2248: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 96 active+clean; 7.9 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 633 op/s; 4068/435 objects degraded (935.172%); 2034/435 objects misplaced (467.586%); 1.2 MiB/s, 109 objects/s recovering 2024-08-17T17:57:26.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:26 smithi165 ceph-mon[105517]: osdmap e1352: 8 total, 8 up, 8 in 2024-08-17T17:57:26.836 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:26.858 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:26.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:26.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:26.960 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:26.982 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:26.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:26.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:27.084 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:27.105 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:27.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:27.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:27.207 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:27.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:27.229 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:27.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:27.330 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:27.351 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:27.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:27.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:27.453 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:27.476 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:27.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:27.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:27.577 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:27.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:27.600 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:27.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:27.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:27 smithi136 ceph-mon[109861]: osdmap e1353: 8 total, 8 up, 8 in 2024-08-17T17:57:27.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:27 smithi136 ceph-mon[115078]: osdmap e1353: 8 total, 8 up, 8 in 2024-08-17T17:57:27.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:57:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:57:27] "GET /metrics HTTP/1.1" 200 36565 "" "Prometheus/2.51.0" 2024-08-17T17:57:27.701 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:27.723 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:27.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:27.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:27.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:27 smithi165 ceph-mon[105517]: osdmap e1353: 8 total, 8 up, 8 in 2024-08-17T17:57:27.825 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:27.846 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:27.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:27.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:27.948 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:27.971 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:27.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:27.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:28.072 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:28.095 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:28.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:28.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:28.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:28.219 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:28.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:28.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:28.320 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:28.342 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:28.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:28.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:28.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:28.465 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:28.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:28.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:28.567 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:28.589 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:28.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:28.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:28.691 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:28.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:28 smithi136 ceph-mon[109861]: pgmap v2251: 99 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 96 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 466 op/s; 2/78 objects degraded (2.564%); 984 KiB/s, 106 objects/s recovering 2024-08-17T17:57:28.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:28 smithi136 ceph-mon[115078]: pgmap v2251: 99 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 96 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 466 op/s; 2/78 objects degraded (2.564%); 984 KiB/s, 106 objects/s recovering 2024-08-17T17:57:28.713 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:28.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:28.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:28.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:28 smithi165 ceph-mon[105517]: pgmap v2251: 99 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 96 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 466 op/s; 2/78 objects degraded (2.564%); 984 KiB/s, 106 objects/s recovering 2024-08-17T17:57:28.814 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:28.836 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:28.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:28.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:28.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:28.960 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:28.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:28.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:29.061 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:29.083 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:29.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:29.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:29.184 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:29.205 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:29.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:29.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:29.307 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:29.329 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:29.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:29.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:29.431 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:29.452 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:29.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:29.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:29.554 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:29.575 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:29.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:29.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:29.677 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:29.698 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:29.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:29.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:29.800 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:29.822 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:29.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:29.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:29.924 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:29.945 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:29.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:29.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:30.047 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:30.069 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:30.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:30.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:30.171 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:30.193 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:30.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:30.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:30.294 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:30.316 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:30.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:30.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:30.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:30.439 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:30.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:30.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:30.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:30.563 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:30.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:30.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:30.664 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:30.685 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:30.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:30.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:30.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:30 smithi136 ceph-mon[115078]: pgmap v2252: 99 pgs: 1 active+recovering+undersized+remapped, 1 peering, 97 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 466 op/s; 2/78 objects degraded (2.564%); 984 KiB/s, 210 objects/s recovering 2024-08-17T17:57:30.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:30 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 2/78 objects degraded (2.564%), 1 pg undersized (PG_DEGRADED) 2024-08-17T17:57:30.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:30 smithi136 ceph-mon[109861]: pgmap v2252: 99 pgs: 1 active+recovering+undersized+remapped, 1 peering, 97 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 466 op/s; 2/78 objects degraded (2.564%); 984 KiB/s, 210 objects/s recovering 2024-08-17T17:57:30.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:30 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 2/78 objects degraded (2.564%), 1 pg undersized (PG_DEGRADED) 2024-08-17T17:57:30.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:30 smithi165 ceph-mon[105517]: pgmap v2252: 99 pgs: 1 active+recovering+undersized+remapped, 1 peering, 97 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 466 op/s; 2/78 objects degraded (2.564%); 984 KiB/s, 210 objects/s recovering 2024-08-17T17:57:30.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:30 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 2/78 objects degraded (2.564%), 1 pg undersized (PG_DEGRADED) 2024-08-17T17:57:30.787 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:30.808 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:30.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:30.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:30.910 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:30.931 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:30.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:30.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:31.033 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:31.056 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:31.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:31.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:31.157 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:31.179 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:31.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:31.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:31.280 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:31.302 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:31.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:31.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:31.404 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:31.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:31.426 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:31.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:31.527 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:31.549 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:31.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:31.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:31.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:31.673 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:31.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:31.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:31.775 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:31.797 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:31.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:31.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:31.899 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:31.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:31.921 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:31.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:32.022 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:32.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:32.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:32.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:32.145 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:32.168 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:32.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:32.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:32.269 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:32.291 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:32.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:32.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:32.393 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:32.415 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:32.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:32.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:32.517 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:32.538 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:32.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:32.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:32.640 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:32.661 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:32.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:32.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:32.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:32 smithi136 ceph-mon[109861]: pgmap v2253: 99 pgs: 1 active+recovering+undersized+remapped, 1 peering, 97 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 15 op/s; 2/78 objects degraded (2.564%); 104 KiB/s, 125 objects/s recovering 2024-08-17T17:57:32.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:32 smithi136 ceph-mon[115078]: pgmap v2253: 99 pgs: 1 active+recovering+undersized+remapped, 1 peering, 97 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 15 op/s; 2/78 objects degraded (2.564%); 104 KiB/s, 125 objects/s recovering 2024-08-17T17:57:32.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:32 smithi165 ceph-mon[105517]: pgmap v2253: 99 pgs: 1 active+recovering+undersized+remapped, 1 peering, 97 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 15 op/s; 2/78 objects degraded (2.564%); 104 KiB/s, 125 objects/s recovering 2024-08-17T17:57:32.763 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:32.785 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:32.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:32.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:32.887 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:32.910 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:32.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:32.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:33.012 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:33.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:33.034 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:33.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:33.135 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:33.157 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:33.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:33.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:33.259 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:33.281 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:33.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:33.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:33.383 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:33.407 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:33.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:33.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:33.508 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:33.532 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:33.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:33.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:33.634 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:33.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:33.659 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:33.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:33 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-08-17T17:57:33.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:33 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-08-17T17:57:33.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:33 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-08-17T17:57:33.783 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:33.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:33.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:33.885 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:33.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:33.908 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:33.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:34.008 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:34.031 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:34.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:34.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:34.133 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:34.155 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:34.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:34.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:34.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:34.280 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:34.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:34.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:34.382 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:34.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:34.408 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:34.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:34.509 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:34.531 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:34.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:34.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:34.633 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:34.655 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:34.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:34.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:34.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:34 smithi136 ceph-mon[109861]: pgmap v2254: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 15 op/s; 104 KiB/s, 227 objects/s recovering 2024-08-17T17:57:34.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:34 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-08-17T17:57:34.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:34 smithi136 ceph-mon[109861]: osdmap e1354: 8 total, 8 up, 8 in 2024-08-17T17:57:34.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:34 smithi136 ceph-mon[115078]: pgmap v2254: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 15 op/s; 104 KiB/s, 227 objects/s recovering 2024-08-17T17:57:34.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:34 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-08-17T17:57:34.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:34 smithi136 ceph-mon[115078]: osdmap e1354: 8 total, 8 up, 8 in 2024-08-17T17:57:34.757 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:34.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:34 smithi165 ceph-mon[105517]: pgmap v2254: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 15 op/s; 104 KiB/s, 227 objects/s recovering 2024-08-17T17:57:34.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:34 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-08-17T17:57:34.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:34 smithi165 ceph-mon[105517]: osdmap e1354: 8 total, 8 up, 8 in 2024-08-17T17:57:34.779 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:34.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:34.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:34.881 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:34.902 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:34.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:34.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:35.004 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:35.025 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:35.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:35.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:35.127 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:35.149 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:35.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:35.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:35.251 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:35.272 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:35.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:35.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:35.373 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:35.400 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:35.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:35.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:35.502 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:35.522 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:35.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:35.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:35.624 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:35.646 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:35.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:35.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:35.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:35 smithi165 ceph-mon[105517]: osdmap e1355: 8 total, 8 up, 8 in 2024-08-17T17:57:35.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:35 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:57:35.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:35 smithi136 ceph-mon[109861]: osdmap e1355: 8 total, 8 up, 8 in 2024-08-17T17:57:35.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:35 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:57:35.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:35 smithi136 ceph-mon[115078]: osdmap e1355: 8 total, 8 up, 8 in 2024-08-17T17:57:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:35 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:57:35.748 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:35.770 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:35.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:35.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:35.872 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:35.894 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:35.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:35.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:35.995 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:36.017 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:36.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:36.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:36.119 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:36.141 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:36.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:36.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:36.243 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:36.264 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:36.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:36.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:36.366 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:36.388 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:36.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:36.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:36.490 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:36.512 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:36.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:36.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:36.614 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:36.637 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:36.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:36.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:36 smithi136 ceph-mon[109861]: pgmap v2257: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 206 objects/s recovering 2024-08-17T17:57:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:36 smithi136 ceph-mon[109861]: osdmap e1356: 8 total, 8 up, 8 in 2024-08-17T17:57:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:36 smithi136 ceph-mon[109861]: 53.18 starting backfill to osd.3 from (0'0,0'0] MAX to 1351'9754 2024-08-17T17:57:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:36 smithi136 ceph-mon[109861]: 53.17 starting backfill to osd.4 from (0'0,0'0] MAX to 1351'9840 2024-08-17T17:57:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:36 smithi136 ceph-mon[109861]: 53.18 starting backfill to osd.5 from (0'0,0'0] MAX to 1351'9754 2024-08-17T17:57:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:36 smithi136 ceph-mon[109861]: 53.17 starting backfill to osd.5 from (0'0,0'0] MAX to 1351'9840 2024-08-17T17:57:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:36 smithi136 ceph-mon[109861]: 53.17 starting backfill to osd.7 from (0'0,0'0] MAX to 1351'9840 2024-08-17T17:57:36.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:57:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:36 smithi136 ceph-mon[115078]: pgmap v2257: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 206 objects/s recovering 2024-08-17T17:57:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:36 smithi136 ceph-mon[115078]: osdmap e1356: 8 total, 8 up, 8 in 2024-08-17T17:57:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:36 smithi136 ceph-mon[115078]: 53.18 starting backfill to osd.3 from (0'0,0'0] MAX to 1351'9754 2024-08-17T17:57:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:36 smithi136 ceph-mon[115078]: 53.17 starting backfill to osd.4 from (0'0,0'0] MAX to 1351'9840 2024-08-17T17:57:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:36 smithi136 ceph-mon[115078]: 53.18 starting backfill to osd.5 from (0'0,0'0] MAX to 1351'9754 2024-08-17T17:57:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:36 smithi136 ceph-mon[115078]: 53.17 starting backfill to osd.5 from (0'0,0'0] MAX to 1351'9840 2024-08-17T17:57:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:36 smithi136 ceph-mon[115078]: 53.17 starting backfill to osd.7 from (0'0,0'0] MAX to 1351'9840 2024-08-17T17:57:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:57:36.739 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:36 smithi165 ceph-mon[105517]: pgmap v2257: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 206 objects/s recovering 2024-08-17T17:57:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:36 smithi165 ceph-mon[105517]: osdmap e1356: 8 total, 8 up, 8 in 2024-08-17T17:57:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:36 smithi165 ceph-mon[105517]: 53.18 starting backfill to osd.3 from (0'0,0'0] MAX to 1351'9754 2024-08-17T17:57:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:36 smithi165 ceph-mon[105517]: 53.17 starting backfill to osd.4 from (0'0,0'0] MAX to 1351'9840 2024-08-17T17:57:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:36 smithi165 ceph-mon[105517]: 53.18 starting backfill to osd.5 from (0'0,0'0] MAX to 1351'9754 2024-08-17T17:57:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:36 smithi165 ceph-mon[105517]: 53.17 starting backfill to osd.5 from (0'0,0'0] MAX to 1351'9840 2024-08-17T17:57:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:36 smithi165 ceph-mon[105517]: 53.17 starting backfill to osd.7 from (0'0,0'0] MAX to 1351'9840 2024-08-17T17:57:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:57:36.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:36.762 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:36.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:36.863 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:36.885 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:36.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:36.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:36.987 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:37.008 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:37.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:37.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:37.110 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:37.133 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:37.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:37.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:37.234 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:37.256 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:37.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:37.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:37.358 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:37.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:37.380 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:37.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:37.482 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:37.503 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:37.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:37.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:37.605 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:37.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:37.626 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:37.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:37 smithi136 ceph-mon[109861]: osdmap e1357: 8 total, 8 up, 8 in 2024-08-17T17:57:37.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:37 smithi136 ceph-mon[115078]: osdmap e1357: 8 total, 8 up, 8 in 2024-08-17T17:57:37.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:57:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:57:37] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-08-17T17:57:37.727 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:37.748 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:37.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:37.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:37 smithi165 ceph-mon[105517]: osdmap e1357: 8 total, 8 up, 8 in 2024-08-17T17:57:37.851 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:37.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:37.873 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:37.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:37.974 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:37.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:37.996 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:37.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:38.097 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:38.118 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:38.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:38.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:38.220 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:38.247 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:38.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:38.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:38.349 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:38.370 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:38.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:38.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:38.472 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:38.493 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:38.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:38.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:38.594 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:38.615 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:38.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:38.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:38.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:38 smithi136 ceph-mon[109861]: pgmap v2260: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 96 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:57:38.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:38 smithi136 ceph-mon[109861]: osdmap e1358: 8 total, 8 up, 8 in 2024-08-17T17:57:38.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:38 smithi136 ceph-mon[115078]: pgmap v2260: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 96 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:57:38.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:38 smithi136 ceph-mon[115078]: osdmap e1358: 8 total, 8 up, 8 in 2024-08-17T17:57:38.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:38.738 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:38.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:38.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:38.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:38 smithi165 ceph-mon[105517]: pgmap v2260: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 96 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:57:38.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:38 smithi165 ceph-mon[105517]: osdmap e1358: 8 total, 8 up, 8 in 2024-08-17T17:57:38.840 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:38.861 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:38.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:38.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:38.963 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:38.986 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:38.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:38.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:39.071 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:57:39.071 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T17:57:39.071 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-17T17:57:39.071 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:57:39.089 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:39.115 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:39.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:39.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:39.217 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:39.239 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:39.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:39.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:39.341 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:39.364 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:39.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:39.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:39.466 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:39.489 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:39.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:39.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:39.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:39.613 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:39.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:39.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:39.714 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:39.737 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:39.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:39.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:39 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-08-17T17:57:39.838 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:39.862 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:39.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:39.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:39.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:39 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-08-17T17:57:39.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:39 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-08-17T17:57:39.964 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:39.986 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:39.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:39.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:40.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:40.113 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:40.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:40.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:40.214 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:40.239 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:40.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:40.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:40.341 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:40.367 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:40.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:40.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:40.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:40.491 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:40.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:40.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:40.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:40.618 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:40.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:40.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:40.720 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:40.745 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:40.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:40.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:40 smithi165 ceph-mon[105517]: pgmap v2262: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 3 peering, 95 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:57:40.847 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:40.872 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:40.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:40.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:40.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:40 smithi136 ceph-mon[109861]: pgmap v2262: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 3 peering, 95 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:57:40.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:40 smithi136 ceph-mon[115078]: pgmap v2262: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 3 peering, 95 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:57:40.974 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:40.998 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:40.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:40.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:41.100 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:41.125 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:41.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:41.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:41.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:41.253 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:41.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:41.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:41.355 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:41.382 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:41.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:41.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:41.484 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:41.509 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:41.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:41.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:41.611 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:41.635 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:41.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:41.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:41.737 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:41.760 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:41.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:41.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:41 smithi165 ceph-mon[105517]: 53.10 scrub starts 2024-08-17T17:57:41.862 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:41.884 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:41.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:41.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:41.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:41 smithi136 ceph-mon[109861]: 53.10 scrub starts 2024-08-17T17:57:41.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:41 smithi136 ceph-mon[115078]: 53.10 scrub starts 2024-08-17T17:57:41.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:42.008 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:42.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:42.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:42.110 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:42.136 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:42.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:42.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:42.238 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:42.267 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:42.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:42.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:42.368 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:42.391 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:42.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:42.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:42.492 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:42.515 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:42.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:42.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:42.616 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:42.638 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:42.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:42.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:42.740 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:42.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:42 smithi165 ceph-mon[105517]: pgmap v2263: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 3 peering, 95 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:57:42.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:42.762 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:42.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:42.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:42.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:42.886 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:42.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:42.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:42 smithi136 ceph-mon[109861]: pgmap v2263: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 3 peering, 95 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:57:42.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:42 smithi136 ceph-mon[115078]: pgmap v2263: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 3 peering, 95 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:57:42.987 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:43.009 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:43.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:43.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:43.111 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:43.132 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:43.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:43.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:43.234 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:43.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:43.260 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:43.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:43.361 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:43.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:43.383 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:43.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:43.484 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:43.505 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:43.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:43.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:43.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:43.629 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:43.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:43.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:43.731 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:43.753 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:43.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:43.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:43.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:43.877 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:43.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:43.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:43.978 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:44.001 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:44.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:44.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:44.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:44.125 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:44.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:44.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:44.230 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:44.253 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:44.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:44.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:44.355 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:44.377 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:44.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:44.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:44.479 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:44.502 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:44.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:44.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:44.604 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:44.628 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:44.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:44.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:44.729 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:44.751 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:44.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:44.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:44 smithi165 ceph-mon[105517]: pgmap v2264: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:57:44.853 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:44.875 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:44.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:44.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:44.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:44 smithi136 ceph-mon[109861]: pgmap v2264: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:57:44.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:44 smithi136 ceph-mon[115078]: pgmap v2264: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:57:44.977 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:45.002 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:45.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:45.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:45.104 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:45.126 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:45.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:45.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:45.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:45.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:45.251 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:45.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:45.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:45.378 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:45.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:45.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:45.480 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:45.502 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:45.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:45.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:45.603 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:45.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:45.627 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:45.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:45.728 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:45.751 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:45.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:45.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:45.852 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:45.876 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:45.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:45.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:45.977 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:46.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:46.004 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:46.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:46.105 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:46.131 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:46.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:46.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:46.233 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:46.254 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:46.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:46.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:46.356 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:46.379 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:46.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:46.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:46.480 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:46.504 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:46.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:46.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:46.605 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:46.629 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:46.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:46.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:46.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:46.753 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:46.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:46.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:46.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:46 smithi165 ceph-mon[105517]: pgmap v2265: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 84 objects/s recovering 2024-08-17T17:57:46.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:46 smithi165 ceph-mon[105517]: 53.10 scrub starts 2024-08-17T17:57:46.855 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:46.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:46.879 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:46.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:46.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:46 smithi136 ceph-mon[109861]: pgmap v2265: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 84 objects/s recovering 2024-08-17T17:57:46.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:46 smithi136 ceph-mon[109861]: 53.10 scrub starts 2024-08-17T17:57:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:46 smithi136 ceph-mon[115078]: pgmap v2265: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 84 objects/s recovering 2024-08-17T17:57:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:46 smithi136 ceph-mon[115078]: 53.10 scrub starts 2024-08-17T17:57:46.980 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:47.003 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:47.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:47.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:47.105 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:47.128 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:47.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:47.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:47.229 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:47.251 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:47.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:47.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:47.353 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:47.377 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:47.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:47.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:47.479 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:47.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:47.502 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:47.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:47.603 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:47.626 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:47.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:47.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:47.727 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:47.750 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:47.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:47.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:47.847 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:57:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:57:47] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-08-17T17:57:47.851 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:47.872 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:47.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:47.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:47.974 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:47.996 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:47.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:47.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:48.098 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:48.121 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:48.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:48.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:48.223 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:48.247 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:48.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:48.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:48.349 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:48.373 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:48.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:48.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:48.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:48.499 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:48.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:48.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:48.601 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:48.624 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:48.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:48.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:48.726 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:48.750 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:48.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:48.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:48 smithi165 ceph-mon[105517]: pgmap v2266: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 124 objects/s recovering 2024-08-17T17:57:48.852 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:48.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:48.876 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:48.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:48.943 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:48 smithi136 ceph-mon[109861]: pgmap v2266: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 124 objects/s recovering 2024-08-17T17:57:48.944 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:48 smithi136 ceph-mon[115078]: pgmap v2266: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 124 objects/s recovering 2024-08-17T17:57:48.977 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:49.003 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:49.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:49.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:49.105 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:49.129 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:49.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:49.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:49.231 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:49.254 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:49.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:49.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:49.356 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:49.378 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:49.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:49.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:49.480 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:49.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:49.504 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:49.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:49.605 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:49.631 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:49.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:49.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:49.719 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-17T17:57:49.733 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:49.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:49.759 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:49.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:49.860 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:49.885 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:49.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:49.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:49.987 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:50.012 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:50.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:50.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:50.114 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:50.137 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:50.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:50.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:50.240 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:50.266 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:50.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:50.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:50.368 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:50.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:50.391 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:50.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:50.492 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:50.515 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:50.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:50.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:50.531 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:57:50.616 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:50.642 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:50.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:50.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:50.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:50.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:50 smithi165 ceph-mon[105517]: pgmap v2267: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 110 objects/s recovering 2024-08-17T17:57:50.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:50.772 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:50.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:50.873 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:50.897 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:50.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:50.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:50.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:50 smithi136 ceph-mon[109861]: pgmap v2267: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 110 objects/s recovering 2024-08-17T17:57:50.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:50 smithi136 ceph-mon[115078]: pgmap v2267: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 110 objects/s recovering 2024-08-17T17:57:51.000 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:51.023 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:51.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:51.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:51.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:51.149 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:51.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:51.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:51.251 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:51.275 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:51.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:51.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:51.376 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:51.400 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:51.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:51.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:51.502 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:51.525 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:51.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:51.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:51.627 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:51.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:51.651 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:51.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:51.752 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:51.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:57:51.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:51.777 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:51.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:51.848 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:57:51.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:57:51.878 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:51.901 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:51.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:51.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:52.003 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:52.027 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:52.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:52.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:52.128 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:52.152 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:52.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:52.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:52.253 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:52.276 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:52.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:52.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:52.378 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:52.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:52.404 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:52.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:52.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:52.532 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:52.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:52.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:52.634 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:52.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:52.656 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:52.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:52.757 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:52.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:52 smithi165 ceph-mon[105517]: pgmap v2268: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 103 objects/s recovering 2024-08-17T17:57:52.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:52 smithi165 ceph-mon[105517]: 53.10 scrub starts 2024-08-17T17:57:52.782 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:52.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:52.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:52.884 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:52.909 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:52.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:52.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:52.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:52 smithi136 ceph-mon[115078]: pgmap v2268: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 103 objects/s recovering 2024-08-17T17:57:52.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:52 smithi136 ceph-mon[115078]: 53.10 scrub starts 2024-08-17T17:57:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:52 smithi136 ceph-mon[109861]: pgmap v2268: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 103 objects/s recovering 2024-08-17T17:57:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:52 smithi136 ceph-mon[109861]: 53.10 scrub starts 2024-08-17T17:57:53.014 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:53.038 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:53.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:53.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:53.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:53.164 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:53.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:53.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:53.266 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:53.289 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:53.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:53.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:53.391 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:53.415 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:53.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:53.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:53.517 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:53.539 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:53.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:53.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:53.641 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:53.663 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:53.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:53.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:53.764 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:53.790 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:53.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:53.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:53.891 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:53.915 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:53.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:53.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:54.016 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:54.040 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:54.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:54.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:54.142 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:54.164 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:54.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:54.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:54.266 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:54.289 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:54.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:54.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:54.391 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:54.414 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:54.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:54.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:54.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:54.538 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:54.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:54.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:54.640 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:54.665 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:54.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:54.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:54.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:54 smithi165 ceph-mon[105517]: pgmap v2269: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 164 objects/s recovering 2024-08-17T17:57:54.767 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:54.789 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:54.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:54.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:54.891 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:54.914 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:54.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:54.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:54.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:54 smithi136 ceph-mon[109861]: pgmap v2269: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 164 objects/s recovering 2024-08-17T17:57:54.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:54 smithi136 ceph-mon[115078]: pgmap v2269: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 164 objects/s recovering 2024-08-17T17:57:55.016 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:55.041 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:55.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:55.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:55.143 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:55.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:55.166 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:55.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:55.267 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:55.290 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:55.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:55.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:55.392 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:55.416 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:55.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:55.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:55.507 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T17:57:55.518 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:55.546 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:55.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:55.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:55.648 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:55.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:55.673 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:55.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:55 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2688516962' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:57:55.775 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:55.797 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:55.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:55.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:55.899 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:55.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:55.924 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:55.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:55.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:55 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2688516962' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:57:55.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:55 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2688516962' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:57:56.025 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:56.048 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:56.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:56.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:56.150 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:56.175 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:56.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:56.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:56.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:56.303 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:56.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:56.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:56.405 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:56.426 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:56.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:56.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:56.528 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:56.551 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:56.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:56.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:56.653 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:56.674 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:56.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:56.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:56.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:56 smithi136 ceph-mon[109861]: pgmap v2270: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 112 objects/s recovering 2024-08-17T17:57:56.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:56 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:57:56.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:56 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2688516962' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:57:56.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:56 smithi136 ceph-mon[109861]: osdmap e1359: 8 total, 8 up, 8 in 2024-08-17T17:57:56.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:56 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1193837625' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:57:56.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:56 smithi136 ceph-mon[115078]: pgmap v2270: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 112 objects/s recovering 2024-08-17T17:57:56.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:56 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:57:56.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:56 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2688516962' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:57:56.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:56 smithi136 ceph-mon[115078]: osdmap e1359: 8 total, 8 up, 8 in 2024-08-17T17:57:56.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:56 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1193837625' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:57:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:56 smithi165 ceph-mon[105517]: pgmap v2270: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 97 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s, 112 objects/s recovering 2024-08-17T17:57:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:56 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:57:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:56 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2688516962' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:57:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:56 smithi165 ceph-mon[105517]: osdmap e1359: 8 total, 8 up, 8 in 2024-08-17T17:57:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:56 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1193837625' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:57:56.774 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T17:57:56.774 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T17:57:56.774 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T17:57:56.774 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T17:57:56.774 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T17:57:56.775 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:57:56.775 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:57:56.775 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:57:56.775 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:57:56.775 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:57:56.775 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T17:57:56.775 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T17:57:56.776 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T17:57:56.776 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:57:56.776 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:57:56.776 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:57:56.776 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:56.799 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:56.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:56.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:56.900 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:56.927 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:56.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:56.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:57.028 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:57.052 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:57.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:57.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:57.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:57.176 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:57.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:57.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:57.278 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:57.302 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:57.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:57.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:57.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:57.429 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:57.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:57.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:57.531 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:57.556 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:57.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:57.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:57.658 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:57.681 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:57.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:57.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:57 smithi165 ceph-mon[105517]: 53.10 scrub starts 2024-08-17T17:57:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:57 smithi165 ceph-mon[105517]: 53.10 scrub ok 2024-08-17T17:57:57.783 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:57:57.805 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:57.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:57.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:57:57.906 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:57.928 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:57.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:57.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:57.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:57 smithi136 ceph-mon[115078]: 53.10 scrub starts 2024-08-17T17:57:57.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:57 smithi136 ceph-mon[115078]: 53.10 scrub ok 2024-08-17T17:57:57.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:57 smithi136 ceph-mon[109861]: 53.10 scrub starts 2024-08-17T17:57:57.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:57 smithi136 ceph-mon[109861]: 53.10 scrub ok 2024-08-17T17:57:57.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:57:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:57:57] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-08-17T17:57:58.030 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:57:58.056 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:58.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:58.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:57:58.158 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:57:58.184 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:58.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:58.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:57:58.285 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:58.308 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:58.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:58.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:58.409 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:58.435 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:58.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:58.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:58.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:58.561 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:58.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:58.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:58.663 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:58.687 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:58.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:58.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:57:58 smithi165 ceph-mon[105517]: pgmap v2272: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 98 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:57:58.789 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:58.810 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:58.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:58.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:58.911 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:57:58.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:58.934 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:58.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:57:58.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:57:58 smithi136 ceph-mon[115078]: pgmap v2272: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 98 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:57:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:57:58 smithi136 ceph-mon[109861]: pgmap v2272: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 98 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:57:59.035 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:59.056 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:59.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:59.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:59.158 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:59.179 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:59.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:59.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:59.280 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:57:59.302 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:59.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:59.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:57:59.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:59.431 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:59.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:59.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:59.532 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:57:59.557 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:59.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:59.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:57:59.659 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:57:59.680 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:57:59.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:59.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:57:59.782 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:59.808 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:59.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:59.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:57:59.909 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:57:59.934 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:57:59.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:57:59.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:00.036 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:00.062 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:00.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:00.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:00.163 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:00.187 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:00.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:00.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:00.191 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:58:00.289 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:00.314 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:00.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:00.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:00.416 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:00.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:00.442 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:00.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:00.543 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:00.566 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:00.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:00.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:00.667 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:00.692 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:00.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:00.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:00.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:00 smithi136 ceph-mon[109861]: pgmap v2273: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 98 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:00.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:00 smithi136 ceph-mon[109861]: osdmap e1360: 8 total, 8 up, 8 in 2024-08-17T17:58:00.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:00 smithi136 ceph-mon[115078]: pgmap v2273: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 98 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:00.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:00 smithi136 ceph-mon[115078]: osdmap e1360: 8 total, 8 up, 8 in 2024-08-17T17:58:00.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:00 smithi165 ceph-mon[105517]: pgmap v2273: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 98 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:00.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:00 smithi165 ceph-mon[105517]: osdmap e1360: 8 total, 8 up, 8 in 2024-08-17T17:58:00.794 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:00.815 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:00.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:00.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:00.916 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:00.942 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:00.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:00.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:01.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:01.069 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:01.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:01.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:01.170 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:01.172 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:58:01.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:01.197 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:01.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:01.298 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:01.319 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:01.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:01.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:01.420 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:01.446 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:01.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:01.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:01.548 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:01.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:01.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:01.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:01.674 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:01.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:01 smithi136 ceph-mon[115078]: osdmap e1361: 8 total, 8 up, 8 in 2024-08-17T17:58:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:01 smithi136 ceph-mon[109861]: osdmap e1361: 8 total, 8 up, 8 in 2024-08-17T17:58:01.699 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:01.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:01.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:01.737 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:58:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:01 smithi165 ceph-mon[105517]: osdmap e1361: 8 total, 8 up, 8 in 2024-08-17T17:58:01.801 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:01.822 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:01.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:01.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:01.924 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:01.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:01.947 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:01.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:02.047 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:02.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:02.074 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:02.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:02.175 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:02.196 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:02.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:02.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:02.298 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:02.323 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:02.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:02.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:02.424 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:02.450 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:02.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:02.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:02.551 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:02.573 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:02.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:02.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:02.675 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:02.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:02 smithi136 ceph-mon[115078]: pgmap v2276: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 98 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:02.696 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:02.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:02 smithi136 ceph-mon[109861]: pgmap v2276: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 98 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:02.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:02.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:02.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:02 smithi165 ceph-mon[105517]: pgmap v2276: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 98 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:02.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:02.825 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:02.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:02.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:02.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:02.951 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:02.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:02.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:03.053 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:03.078 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:03.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:03.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:03.180 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:03.185 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:58:03.202 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:03.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:03.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:03.303 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:03.325 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:03.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:03.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:03.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:03.453 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:03.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:03.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:03.554 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:03.578 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:03.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:03.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:03.680 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:03.701 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:03.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:03.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:03 smithi165 ceph-mon[105517]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-08-17T17:58:03.787 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:03 smithi136 ceph-mon[109861]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-08-17T17:58:03.787 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:03 smithi136 ceph-mon[115078]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-08-17T17:58:03.803 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:03.828 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:03.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:03.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:03.929 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:03.951 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:03.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:03.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:04.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:04.076 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:04.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:04.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:04.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:04.202 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:04.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:04.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:04.265 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:58:04.304 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:04.325 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:04.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:04.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:04.411 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:58:04.412 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T17:58:04.426 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:04.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:04.449 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:04.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:04.497 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'unique_pool_0' removed 2024-08-17T17:58:04.551 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:04.576 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:04.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:04.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:04.677 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:04.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:04 smithi136 ceph-mon[109861]: pgmap v2277: 99 pgs: 99 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:04.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:04 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1166807886' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:58:04.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:04 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:58:04.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:04 smithi136 ceph-mon[115078]: pgmap v2277: 99 pgs: 99 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:04.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:04 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1166807886' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:58:04.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:04 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:58:04.700 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:04.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:04.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:04.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:04 smithi165 ceph-mon[105517]: pgmap v2277: 99 pgs: 99 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:04.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:04 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1166807886' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:58:04.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:04 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:58:04.802 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:04.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:04.824 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:04.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:04.925 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:04.947 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:04.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:04.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:05.049 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:05.070 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:05.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:05.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:05.172 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:05.193 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:05.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:05.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:05.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:05.316 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:05.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:05.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:05.418 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:05.439 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:05.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:05.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:05.541 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:05.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:05.564 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:05.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:05.664 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:05.686 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:05.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:05.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:05.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:05 smithi136 ceph-mon[115078]: from='client.45215 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:58:05.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:05 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T17:58:05.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:05 smithi136 ceph-mon[115078]: osdmap e1362: 8 total, 8 up, 8 in 2024-08-17T17:58:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:05 smithi136 ceph-mon[109861]: from='client.45215 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:58:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:05 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T17:58:05.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:05 smithi136 ceph-mon[109861]: osdmap e1362: 8 total, 8 up, 8 in 2024-08-17T17:58:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:05 smithi165 ceph-mon[105517]: from='client.45215 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:58:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:05 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T17:58:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:05 smithi165 ceph-mon[105517]: osdmap e1362: 8 total, 8 up, 8 in 2024-08-17T17:58:05.787 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:05.808 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:05.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:05.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:05.910 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:05.932 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:05.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:05.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:06.034 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:06.057 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:06.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:06.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:06.201 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:06.222 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:06.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:06.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:06.324 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:06.346 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:06.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:06.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:06.448 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:06.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:06.473 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:06.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:06.573 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:06.595 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:06.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:06.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:06.697 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:06.718 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:06.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:06.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:06 smithi165 ceph-mon[105517]: pgmap v2279: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:58:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:58:06.820 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:06.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:06.842 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:06.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:06.908 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:06 smithi136 ceph-mon[109861]: pgmap v2279: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:06.909 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:58:06.909 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:58:06.909 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:06 smithi136 ceph-mon[115078]: pgmap v2279: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:06.909 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:58:06.909 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:58:06.929 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":2277,"stamp":"2024-08-17T17:58:02.691779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595941,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":379,"num_read_kb":823,"num_write":203175,"num_write_kb":5282002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19343,"num_bytes_recovered":837879392,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":215265,"ondisk_log_size":215265,"up":297,"acting":297,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":301,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2388168,"kb_used_data":277336,"kb_used_omap":295,"kb_used_meta":2110424,"kb_avail":747573048,"statfs":{"total":767960285184,"available":765514801152,"internally_reserved":0,"allocated":283992064,"data_stored":250406045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":302302,"internal_metadata":2161074978},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-7741,"num_write_kb":-246944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-463,"num_bytes_recovered":-116588544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.181016"},"pg_stats":[{"pgid":"53.19","version":"1351'9438","reported_seq":9608,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.455670+0000","last_change":"2024-08-17T17:58:00.455362+0000","last_active":"2024-08-17T17:58:00.455670+0000","last_peered":"2024-08-17T17:58:00.455670+0000","last_clean":"2024-08-17T17:58:00.455670+0000","last_became_active":"2024-08-17T17:58:00.430892+0000","last_became_peered":"2024-08-17T17:58:00.430892+0000","last_unstale":"2024-08-17T17:58:00.455670+0000","last_undegraded":"2024-08-17T17:58:00.455670+0000","last_fullsized":"2024-08-17T17:58:00.455670+0000","mapping_epoch":1360,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1361,"parent":"0.0","parent_split_bits":5,"last_scrub":"1311'3869","last_scrub_stamp":"2024-08-17T17:53:49.812836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:49.812836+0000","objects_scrubbed":1929,"log_size":9438,"log_dups_size":0,"ondisk_log_size":9438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:00:50.298273+0000","scrub_duration":554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5579,"num_write_kb":116800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3754,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"53.18","version":"1351'9754","reported_seq":9910,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.876599+0000","last_change":"2024-08-17T17:57:37.445618+0000","last_active":"2024-08-17T17:58:00.876599+0000","last_peered":"2024-08-17T17:58:00.876599+0000","last_clean":"2024-08-17T17:58:00.876599+0000","last_became_active":"2024-08-17T17:57:37.422404+0000","last_became_peered":"2024-08-17T17:57:37.422404+0000","last_unstale":"2024-08-17T17:58:00.876599+0000","last_undegraded":"2024-08-17T17:58:00.876599+0000","last_fullsized":"2024-08-17T17:58:00.876599+0000","mapping_epoch":1357,"log_start":"1308'3456","ondisk_log_start":"1308'3456","created":1310,"last_epoch_clean":1358,"parent":"0.0","parent_split_bits":5,"last_scrub":"1313'4093","last_scrub_stamp":"2024-08-17T17:53:53.987508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:53.987508+0000","objects_scrubbed":2233,"log_size":6298,"log_dups_size":0,"ondisk_log_size":6298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:31:50.466157+0000","scrub_duration":0.21921206200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5661,"num_write_kb":109696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.17","version":"1351'9840","reported_seq":10019,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.717785+0000","last_change":"2024-08-17T17:57:37.447165+0000","last_active":"2024-08-17T17:57:59.717785+0000","last_peered":"2024-08-17T17:57:59.717785+0000","last_clean":"2024-08-17T17:57:59.717785+0000","last_became_active":"2024-08-17T17:57:37.422891+0000","last_became_peered":"2024-08-17T17:57:37.422891+0000","last_unstale":"2024-08-17T17:57:59.717785+0000","last_undegraded":"2024-08-17T17:57:59.717785+0000","last_fullsized":"2024-08-17T17:57:59.717785+0000","mapping_epoch":1357,"log_start":"1308'3386","ondisk_log_start":"1308'3386","created":1310,"last_epoch_clean":1358,"parent":"0.0","parent_split_bits":5,"last_scrub":"1312'3969","last_scrub_stamp":"2024-08-17T17:53:50.287885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:50.287885+0000","objects_scrubbed":2090,"log_size":6454,"log_dups_size":0,"ondisk_log_size":6454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:48:58.321534+0000","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5884,"num_write_kb":121408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"53.16","version":"1351'9800","reported_seq":17681,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.428900+0000","last_change":"2024-08-17T17:57:36.438605+0000","last_active":"2024-08-17T17:58:00.428900+0000","last_peered":"2024-08-17T17:58:00.428900+0000","last_clean":"2024-08-17T17:58:00.428900+0000","last_became_active":"2024-08-17T17:57:36.411057+0000","last_became_peered":"2024-08-17T17:57:36.411057+0000","last_unstale":"2024-08-17T17:58:00.428900+0000","last_undegraded":"2024-08-17T17:58:00.428900+0000","last_fullsized":"2024-08-17T17:58:00.428900+0000","mapping_epoch":1356,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1357,"parent":"0.0","parent_split_bits":5,"last_scrub":"1311'3927","last_scrub_stamp":"2024-08-17T17:53:48.318097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:48.318097+0000","objects_scrubbed":2014,"log_size":9800,"log_dups_size":0,"ondisk_log_size":9800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:51:05.646006+0000","scrub_duration":593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5882,"num_write_kb":123776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1927,"num_bytes_recovered":126287872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.15","version":"1351'9611","reported_seq":19038,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.717464+0000","last_change":"2024-08-17T17:57:26.401893+0000","last_active":"2024-08-17T17:57:59.717464+0000","last_peered":"2024-08-17T17:57:59.717464+0000","last_clean":"2024-08-17T17:57:59.717464+0000","last_became_active":"2024-08-17T17:57:26.369456+0000","last_became_peered":"2024-08-17T17:57:26.369456+0000","last_unstale":"2024-08-17T17:57:59.717464+0000","last_undegraded":"2024-08-17T17:57:59.717464+0000","last_fullsized":"2024-08-17T17:57:59.717464+0000","mapping_epoch":1352,"log_start":"1306'324","ondisk_log_start":"1306'324","created":1310,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":5,"last_scrub":"1311'3700","last_scrub_stamp":"2024-08-17T17:53:47.592482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:47.592482+0000","objects_scrubbed":1931,"log_size":9287,"log_dups_size":0,"ondisk_log_size":9287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:12:32.436447+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5925,"num_write_kb":127808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3928,"num_bytes_recovered":129564672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.14","version":"1351'9506","reported_seq":18027,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.716775+0000","last_change":"2024-08-17T17:57:26.396808+0000","last_active":"2024-08-17T17:57:59.716775+0000","last_peered":"2024-08-17T17:57:59.716775+0000","last_clean":"2024-08-17T17:57:59.716775+0000","last_became_active":"2024-08-17T17:57:26.367549+0000","last_became_peered":"2024-08-17T17:57:26.367549+0000","last_unstale":"2024-08-17T17:57:59.716775+0000","last_undegraded":"2024-08-17T17:57:59.716775+0000","last_fullsized":"2024-08-17T17:57:59.716775+0000","mapping_epoch":1352,"log_start":"1308'2672","ondisk_log_start":"1308'2672","created":1310,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":5,"last_scrub":"1311'3909","last_scrub_stamp":"2024-08-17T17:53:49.696328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:49.696328+0000","objects_scrubbed":1954,"log_size":6834,"log_dups_size":0,"ondisk_log_size":6834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:25:45.213397+0000","scrub_duration":810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5616,"num_write_kb":117184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2781,"num_bytes_recovered":124977152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"53.13","version":"1351'9335","reported_seq":19061,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.876761+0000","last_change":"2024-08-17T17:55:38.393304+0000","last_active":"2024-08-17T17:58:00.876761+0000","last_peered":"2024-08-17T17:58:00.876761+0000","last_clean":"2024-08-17T17:58:00.876761+0000","last_became_active":"2024-08-17T17:55:38.383298+0000","last_became_peered":"2024-08-17T17:55:38.383298+0000","last_unstale":"2024-08-17T17:58:00.876761+0000","last_undegraded":"2024-08-17T17:58:00.876761+0000","last_fullsized":"2024-08-17T17:58:00.876761+0000","mapping_epoch":1337,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1338,"parent":"0.0","parent_split_bits":5,"last_scrub":"1311'3687","last_scrub_stamp":"2024-08-17T17:53:48.035372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:48.035372+0000","objects_scrubbed":1888,"log_size":9335,"log_dups_size":0,"ondisk_log_size":9335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:28:58.174221+0000","scrub_duration":0.55651981800000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5660,"num_write_kb":120704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2386,"num_bytes_recovered":156368896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.12","version":"1351'9557","reported_seq":19500,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.717268+0000","last_change":"2024-08-17T17:56:01.548247+0000","last_active":"2024-08-17T17:57:59.717268+0000","last_peered":"2024-08-17T17:57:59.717268+0000","last_clean":"2024-08-17T17:57:59.717268+0000","last_became_active":"2024-08-17T17:56:01.535252+0000","last_became_peered":"2024-08-17T17:56:01.535252+0000","last_unstale":"2024-08-17T17:57:59.717268+0000","last_undegraded":"2024-08-17T17:57:59.717268+0000","last_fullsized":"2024-08-17T17:57:59.717268+0000","mapping_epoch":1345,"log_start":"0'0","ondisk_log_start":"0'0","created":1310,"last_epoch_clean":1346,"parent":"0.0","parent_split_bits":5,"last_scrub":"1312'4010","last_scrub_stamp":"2024-08-17T17:53:52.788196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:52.788196+0000","objects_scrubbed":2082,"log_size":9557,"log_dups_size":0,"ondisk_log_size":9557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:06:14.839811+0000","scrub_duration":1.1488786010000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5568,"num_write_kb":111552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2443,"num_bytes_recovered":160104448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.11","version":"1351'9613","reported_seq":16150,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.876671+0000","last_change":"2024-08-17T17:54:32.250770+0000","last_active":"2024-08-17T17:58:00.876671+0000","last_peered":"2024-08-17T17:58:00.876671+0000","last_clean":"2024-08-17T17:58:00.876671+0000","last_became_active":"2024-08-17T17:54:32.246225+0000","last_became_peered":"2024-08-17T17:54:32.246225+0000","last_unstale":"2024-08-17T17:58:00.876671+0000","last_undegraded":"2024-08-17T17:58:00.876671+0000","last_fullsized":"2024-08-17T17:58:00.876671+0000","mapping_epoch":1327,"log_start":"1306'69","ondisk_log_start":"1306'69","created":1310,"last_epoch_clean":1328,"parent":"0.0","parent_split_bits":5,"last_scrub":"1312'3987","last_scrub_stamp":"2024-08-17T17:53:51.667290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:51.667290+0000","objects_scrubbed":2071,"log_size":9544,"log_dups_size":0,"ondisk_log_size":9544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:25:31.945001+0000","scrub_duration":708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5643,"num_write_kb":114304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":138805248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.10","version":"1351'9596","reported_seq":16264,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.876563+0000","last_change":"2024-08-17T17:57:55.863913+0000","last_active":"2024-08-17T17:58:00.876563+0000","last_peered":"2024-08-17T17:58:00.876563+0000","last_clean":"2024-08-17T17:58:00.876563+0000","last_became_active":"2024-08-17T17:55:57.520262+0000","last_became_peered":"2024-08-17T17:55:57.520262+0000","last_unstale":"2024-08-17T17:58:00.876563+0000","last_undegraded":"2024-08-17T17:58:00.876563+0000","last_fullsized":"2024-08-17T17:58:00.876563+0000","mapping_epoch":1341,"log_start":"1306'135","ondisk_log_start":"1306'135","created":1310,"last_epoch_clean":1342,"parent":"0.0","parent_split_bits":5,"last_scrub":"1351'9596","last_scrub_stamp":"2024-08-17T17:57:55.863784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:57:55.863784+0000","objects_scrubbed":0,"log_size":9461,"log_dups_size":0,"ondisk_log_size":9461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:00:02.129897+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.f","version":"1351'15984","reported_seq":16053,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.717974+0000","last_change":"2024-08-17T17:53:10.489536+0000","last_active":"2024-08-17T17:57:59.717974+0000","last_peered":"2024-08-17T17:57:59.717974+0000","last_clean":"2024-08-17T17:57:59.717974+0000","last_became_active":"2024-08-17T17:53:10.489173+0000","last_became_peered":"2024-08-17T17:53:10.489173+0000","last_unstale":"2024-08-17T17:57:59.717974+0000","last_undegraded":"2024-08-17T17:57:59.717974+0000","last_fullsized":"2024-08-17T17:57:59.717974+0000","mapping_epoch":1303,"log_start":"1329'7357","ondisk_log_start":"1329'7357","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":8627,"log_dups_size":0,"ondisk_log_size":8627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:46:03.457908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15984,"num_write_kb":511488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.e","version":"1351'15482","reported_seq":15551,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.717780+0000","last_change":"2024-08-17T17:53:10.488446+0000","last_active":"2024-08-17T17:57:59.717780+0000","last_peered":"2024-08-17T17:57:59.717780+0000","last_clean":"2024-08-17T17:57:59.717780+0000","last_became_active":"2024-08-17T17:53:10.488058+0000","last_became_peered":"2024-08-17T17:53:10.488058+0000","last_unstale":"2024-08-17T17:57:59.717780+0000","last_undegraded":"2024-08-17T17:57:59.717780+0000","last_fullsized":"2024-08-17T17:57:59.717780+0000","mapping_epoch":1303,"log_start":"1328'6908","ondisk_log_start":"1328'6908","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":8574,"log_dups_size":0,"ondisk_log_size":8574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:00:57.616106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15482,"num_write_kb":495424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.d","version":"1351'15788","reported_seq":15928,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.429025+0000","last_change":"2024-08-17T17:53:10.646982+0000","last_active":"2024-08-17T17:58:00.429025+0000","last_peered":"2024-08-17T17:58:00.429025+0000","last_clean":"2024-08-17T17:58:00.429025+0000","last_became_active":"2024-08-17T17:53:10.646591+0000","last_became_peered":"2024-08-17T17:53:10.646591+0000","last_unstale":"2024-08-17T17:58:00.429025+0000","last_undegraded":"2024-08-17T17:58:00.429025+0000","last_fullsized":"2024-08-17T17:58:00.429025+0000","mapping_epoch":1303,"log_start":"1332'9473","ondisk_log_start":"1332'9473","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":6315,"log_dups_size":0,"ondisk_log_size":6315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:23:01.174079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15788,"num_write_kb":505153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.c","version":"1351'15522","reported_seq":15591,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.427927+0000","last_change":"2024-08-17T17:53:10.559850+0000","last_active":"2024-08-17T17:57:59.427927+0000","last_peered":"2024-08-17T17:57:59.427927+0000","last_clean":"2024-08-17T17:57:59.427927+0000","last_became_active":"2024-08-17T17:53:10.559588+0000","last_became_peered":"2024-08-17T17:53:10.559588+0000","last_unstale":"2024-08-17T17:57:59.427927+0000","last_undegraded":"2024-08-17T17:57:59.427927+0000","last_fullsized":"2024-08-17T17:57:59.427927+0000","mapping_epoch":1303,"log_start":"1325'5800","ondisk_log_start":"1325'5800","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":9722,"log_dups_size":0,"ondisk_log_size":9722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:16:20.553763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15522,"num_write_kb":496704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"53.b","version":"1351'15422","reported_seq":15491,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.427367+0000","last_change":"2024-08-17T17:53:10.490102+0000","last_active":"2024-08-17T17:57:59.427367+0000","last_peered":"2024-08-17T17:57:59.427367+0000","last_clean":"2024-08-17T17:57:59.427367+0000","last_became_active":"2024-08-17T17:53:10.489717+0000","last_became_peered":"2024-08-17T17:53:10.489717+0000","last_unstale":"2024-08-17T17:57:59.427367+0000","last_undegraded":"2024-08-17T17:57:59.427367+0000","last_fullsized":"2024-08-17T17:57:59.427367+0000","mapping_epoch":1303,"log_start":"1325'5700","ondisk_log_start":"1325'5700","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":9722,"log_dups_size":0,"ondisk_log_size":9722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:50:43.448878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15422,"num_write_kb":493504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"53.a","version":"1351'15652","reported_seq":15789,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.716557+0000","last_change":"2024-08-17T17:53:10.489596+0000","last_active":"2024-08-17T17:57:59.716557+0000","last_peered":"2024-08-17T17:57:59.716557+0000","last_clean":"2024-08-17T17:57:59.716557+0000","last_became_active":"2024-08-17T17:53:10.489230+0000","last_became_peered":"2024-08-17T17:53:10.489230+0000","last_unstale":"2024-08-17T17:57:59.716557+0000","last_undegraded":"2024-08-17T17:57:59.716557+0000","last_fullsized":"2024-08-17T17:57:59.716557+0000","mapping_epoch":1303,"log_start":"1329'8258","ondisk_log_start":"1329'8258","created":1303,"last_epoch_clean":1304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:09.466177+0000","objects_scrubbed":0,"log_size":7394,"log_dups_size":0,"ondisk_log_size":7394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:34:48.833586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15652,"num_write_kb":500864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"53.9","version":"1351'9711","reported_seq":9885,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.876779+0000","last_change":"2024-08-17T17:53:48.914243+0000","last_active":"2024-08-17T17:58:00.876779+0000","last_peered":"2024-08-17T17:58:00.876779+0000","last_clean":"2024-08-17T17:58:00.876779+0000","last_became_active":"2024-08-17T17:53:45.787626+0000","last_became_peered":"2024-08-17T17:53:45.787626+0000","last_unstale":"2024-08-17T17:58:00.876779+0000","last_undegraded":"2024-08-17T17:58:00.876779+0000","last_fullsized":"2024-08-17T17:58:00.876779+0000","mapping_epoch":1303,"log_start":"1306'452","ondisk_log_start":"1306'452","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3875","last_scrub_stamp":"2024-08-17T17:53:48.913906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:48.913906+0000","objects_scrubbed":2075,"log_size":9259,"log_dups_size":0,"ondisk_log_size":9259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:40:39.216043+0000","scrub_duration":692,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5849,"num_write_kb":120768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.8","version":"1351'9681","reported_seq":9818,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.425689+0000","last_change":"2024-08-17T17:53:50.732592+0000","last_active":"2024-08-17T17:57:59.425689+0000","last_peered":"2024-08-17T17:57:59.425689+0000","last_clean":"2024-08-17T17:57:59.425689+0000","last_became_active":"2024-08-17T17:53:45.777386+0000","last_became_peered":"2024-08-17T17:53:45.777386+0000","last_unstale":"2024-08-17T17:57:59.425689+0000","last_undegraded":"2024-08-17T17:57:59.425689+0000","last_fullsized":"2024-08-17T17:57:59.425689+0000","mapping_epoch":1303,"log_start":"1308'3277","ondisk_log_start":"1308'3277","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1312'4030","last_scrub_stamp":"2024-08-17T17:53:50.732148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:50.732148+0000","objects_scrubbed":2054,"log_size":6404,"log_dups_size":0,"ondisk_log_size":6404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:42:36.666939+0000","scrub_duration":0.96559516700000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5668,"num_write_kb":115648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"53.7","version":"1351'9586","reported_seq":9759,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.429286+0000","last_change":"2024-08-17T17:53:47.346759+0000","last_active":"2024-08-17T17:58:00.429286+0000","last_peered":"2024-08-17T17:58:00.429286+0000","last_clean":"2024-08-17T17:58:00.429286+0000","last_became_active":"2024-08-17T17:53:45.779914+0000","last_became_peered":"2024-08-17T17:53:45.779914+0000","last_unstale":"2024-08-17T17:58:00.429286+0000","last_undegraded":"2024-08-17T17:58:00.429286+0000","last_fullsized":"2024-08-17T17:58:00.429286+0000","mapping_epoch":1303,"log_start":"1308'2887","ondisk_log_start":"1308'2887","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3847","last_scrub_stamp":"2024-08-17T17:53:47.346625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:47.346625+0000","objects_scrubbed":1924,"log_size":6699,"log_dups_size":0,"ondisk_log_size":6699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:56:13.541749+0000","scrub_duration":430,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5744,"num_write_kb":122240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.6","version":"1351'9867","reported_seq":10041,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.716990+0000","last_change":"2024-08-17T17:53:47.249481+0000","last_active":"2024-08-17T17:57:59.716990+0000","last_peered":"2024-08-17T17:57:59.716990+0000","last_clean":"2024-08-17T17:57:59.716990+0000","last_became_active":"2024-08-17T17:53:45.758286+0000","last_became_peered":"2024-08-17T17:53:45.758286+0000","last_unstale":"2024-08-17T17:57:59.716990+0000","last_undegraded":"2024-08-17T17:57:59.716990+0000","last_fullsized":"2024-08-17T17:57:59.716990+0000","mapping_epoch":1303,"log_start":"1307'1307","ondisk_log_start":"1307'1307","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3922","last_scrub_stamp":"2024-08-17T17:53:47.248989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:47.248989+0000","objects_scrubbed":1980,"log_size":8560,"log_dups_size":0,"ondisk_log_size":8560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:51.725815+0000","scrub_duration":555,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5954,"num_write_kb":127168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"53.5","version":"1351'9300","reported_seq":9472,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.876961+0000","last_change":"2024-08-17T17:53:46.758091+0000","last_active":"2024-08-17T17:58:00.876961+0000","last_peered":"2024-08-17T17:58:00.876961+0000","last_clean":"2024-08-17T17:58:00.876961+0000","last_became_active":"2024-08-17T17:53:45.771578+0000","last_became_peered":"2024-08-17T17:53:45.771578+0000","last_unstale":"2024-08-17T17:58:00.876961+0000","last_undegraded":"2024-08-17T17:58:00.876961+0000","last_fullsized":"2024-08-17T17:58:00.876961+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3661","last_scrub_stamp":"2024-08-17T17:53:46.757894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:46.757894+0000","objects_scrubbed":1802,"log_size":9300,"log_dups_size":0,"ondisk_log_size":9300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:35:47.654567+0000","scrub_duration":509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5644,"num_write_kb":122944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"53.4","version":"1351'9749","reported_seq":9923,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.429052+0000","last_change":"2024-08-17T17:53:46.426863+0000","last_active":"2024-08-17T17:58:00.429052+0000","last_peered":"2024-08-17T17:58:00.429052+0000","last_clean":"2024-08-17T17:58:00.429052+0000","last_became_active":"2024-08-17T17:53:45.764066+0000","last_became_peered":"2024-08-17T17:53:45.764066+0000","last_unstale":"2024-08-17T17:58:00.429052+0000","last_undegraded":"2024-08-17T17:58:00.429052+0000","last_fullsized":"2024-08-17T17:58:00.429052+0000","mapping_epoch":1303,"log_start":"1308'3296","ondisk_log_start":"1308'3296","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3774","last_scrub_stamp":"2024-08-17T17:53:46.426732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:46.426732+0000","objects_scrubbed":1958,"log_size":6453,"log_dups_size":0,"ondisk_log_size":6453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:57.590628+0000","scrub_duration":541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5990,"num_write_kb":129024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.3","version":"1351'9274","reported_seq":9401,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.717573+0000","last_change":"2024-08-17T17:53:46.968605+0000","last_active":"2024-08-17T17:57:59.717573+0000","last_peered":"2024-08-17T17:57:59.717573+0000","last_clean":"2024-08-17T17:57:59.717573+0000","last_became_active":"2024-08-17T17:53:45.764323+0000","last_became_peered":"2024-08-17T17:53:45.764323+0000","last_unstale":"2024-08-17T17:57:59.717573+0000","last_undegraded":"2024-08-17T17:57:59.717573+0000","last_fullsized":"2024-08-17T17:57:59.717573+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3658","last_scrub_stamp":"2024-08-17T17:53:46.967846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:46.967846+0000","objects_scrubbed":1838,"log_size":9274,"log_dups_size":0,"ondisk_log_size":9274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:09:25.913293+0000","scrub_duration":0.48413113800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5624,"num_write_kb":121152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"53.2","version":"1351'9653","reported_seq":9786,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.427410+0000","last_change":"2024-08-17T17:53:48.304744+0000","last_active":"2024-08-17T17:57:59.427410+0000","last_peered":"2024-08-17T17:57:59.427410+0000","last_clean":"2024-08-17T17:57:59.427410+0000","last_became_active":"2024-08-17T17:53:45.780923+0000","last_became_peered":"2024-08-17T17:53:45.780923+0000","last_unstale":"2024-08-17T17:57:59.427410+0000","last_undegraded":"2024-08-17T17:57:59.427410+0000","last_fullsized":"2024-08-17T17:57:59.427410+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3850","last_scrub_stamp":"2024-08-17T17:53:48.304649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:48.304649+0000","objects_scrubbed":2003,"log_size":9653,"log_dups_size":0,"ondisk_log_size":9653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:01.087711+0000","scrub_duration":0.70259554000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5821,"num_write_kb":122176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"53.1","version":"1351'9693","reported_seq":9866,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.428965+0000","last_change":"2024-08-17T17:53:53.382389+0000","last_active":"2024-08-17T17:58:00.428965+0000","last_peered":"2024-08-17T17:58:00.428965+0000","last_clean":"2024-08-17T17:58:00.428965+0000","last_became_active":"2024-08-17T17:53:45.782234+0000","last_became_peered":"2024-08-17T17:53:45.782234+0000","last_unstale":"2024-08-17T17:58:00.428965+0000","last_undegraded":"2024-08-17T17:58:00.428965+0000","last_fullsized":"2024-08-17T17:58:00.428965+0000","mapping_epoch":1303,"log_start":"1308'3124","ondisk_log_start":"1308'3124","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1313'4031","last_scrub_stamp":"2024-08-17T17:53:53.382269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:53.382269+0000","objects_scrubbed":2178,"log_size":6569,"log_dups_size":0,"ondisk_log_size":6569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:01:49.792387+0000","scrub_duration":410,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5664,"num_write_kb":111552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"53.0","version":"1351'9453","reported_seq":9583,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.425668+0000","last_change":"2024-08-17T17:53:47.447388+0000","last_active":"2024-08-17T17:57:59.425668+0000","last_peered":"2024-08-17T17:57:59.425668+0000","last_clean":"2024-08-17T17:57:59.425668+0000","last_became_active":"2024-08-17T17:53:45.741805+0000","last_became_peered":"2024-08-17T17:53:45.741805+0000","last_unstale":"2024-08-17T17:57:59.425668+0000","last_undegraded":"2024-08-17T17:57:59.425668+0000","last_fullsized":"2024-08-17T17:57:59.425668+0000","mapping_epoch":1303,"log_start":"1308'2929","ondisk_log_start":"1308'2929","created":1303,"last_epoch_clean":1311,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'3799","last_scrub_stamp":"2024-08-17T17:53:47.447203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:53:09.466177+0000","last_clean_scrub_stamp":"2024-08-17T17:53:47.447203+0000","objects_scrubbed":1904,"log_size":6524,"log_dups_size":0,"ondisk_log_size":6524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:52:36.467443+0000","scrub_duration":0.60964741,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5670,"num_write_kb":120512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":2028,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.877255+0000","last_change":"2024-08-17T17:23:42.937520+0000","last_active":"2024-08-17T17:58:00.877255+0000","last_peered":"2024-08-17T17:58:00.877255+0000","last_clean":"2024-08-17T17:58:00.877255+0000","last_became_active":"2024-08-17T17:23:42.937174+0000","last_became_peered":"2024-08-17T17:23:42.937174+0000","last_unstale":"2024-08-17T17:58:00.877255+0000","last_undegraded":"2024-08-17T17:58:00.877255+0000","last_fullsized":"2024-08-17T17:58:00.877255+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:47:56.679612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1332,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.425776+0000","last_change":"2024-08-17T17:23:07.398676+0000","last_active":"2024-08-17T17:57:59.425776+0000","last_peered":"2024-08-17T17:57:59.425776+0000","last_clean":"2024-08-17T17:57:59.425776+0000","last_became_active":"2024-08-17T17:23:07.398369+0000","last_became_peered":"2024-08-17T17:23:07.398369+0000","last_unstale":"2024-08-17T17:57:59.425776+0000","last_undegraded":"2024-08-17T17:57:59.425776+0000","last_fullsized":"2024-08-17T17:57:59.425776+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:55:04.281412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":2009,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.717014+0000","last_change":"2024-08-17T17:24:17.476168+0000","last_active":"2024-08-17T17:57:59.717014+0000","last_peered":"2024-08-17T17:57:59.717014+0000","last_clean":"2024-08-17T17:57:59.717014+0000","last_became_active":"2024-08-17T17:24:17.465845+0000","last_became_peered":"2024-08-17T17:24:17.465845+0000","last_unstale":"2024-08-17T17:57:59.717014+0000","last_undegraded":"2024-08-17T17:57:59.717014+0000","last_fullsized":"2024-08-17T17:57:59.717014+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:15:01.494447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2025,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.716502+0000","last_change":"2024-08-17T17:24:17.461772+0000","last_active":"2024-08-17T17:57:59.716502+0000","last_peered":"2024-08-17T17:57:59.716502+0000","last_clean":"2024-08-17T17:57:59.716502+0000","last_became_active":"2024-08-17T17:24:17.460806+0000","last_became_peered":"2024-08-17T17:24:17.460806+0000","last_unstale":"2024-08-17T17:57:59.716502+0000","last_undegraded":"2024-08-17T17:57:59.716502+0000","last_fullsized":"2024-08-17T17:57:59.716502+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:01:16.453148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":1315,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.427635+0000","last_change":"2024-08-17T17:10:05.965176+0000","last_active":"2024-08-17T17:57:59.427635+0000","last_peered":"2024-08-17T17:57:59.427635+0000","last_clean":"2024-08-17T17:57:59.427635+0000","last_became_active":"2024-08-17T17:10:05.964861+0000","last_became_peered":"2024-08-17T17:10:05.964861+0000","last_unstale":"2024-08-17T17:57:59.427635+0000","last_undegraded":"2024-08-17T17:57:59.427635+0000","last_fullsized":"2024-08-17T17:57:59.427635+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:59:04.277926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1347,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.425928+0000","last_change":"2024-08-17T17:24:17.462527+0000","last_active":"2024-08-17T17:57:59.425928+0000","last_peered":"2024-08-17T17:57:59.425928+0000","last_clean":"2024-08-17T17:57:59.425928+0000","last_became_active":"2024-08-17T17:24:17.462240+0000","last_became_peered":"2024-08-17T17:24:17.462240+0000","last_unstale":"2024-08-17T17:57:59.425928+0000","last_undegraded":"2024-08-17T17:57:59.425928+0000","last_fullsized":"2024-08-17T17:57:59.425928+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:40:38.063717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":1330,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.427739+0000","last_change":"2024-08-17T17:23:07.391038+0000","last_active":"2024-08-17T17:57:59.427739+0000","last_peered":"2024-08-17T17:57:59.427739+0000","last_clean":"2024-08-17T17:57:59.427739+0000","last_became_active":"2024-08-17T17:23:07.390571+0000","last_became_peered":"2024-08-17T17:23:07.390571+0000","last_unstale":"2024-08-17T17:57:59.427739+0000","last_undegraded":"2024-08-17T17:57:59.427739+0000","last_fullsized":"2024-08-17T17:57:59.427739+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:18:41.791740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1347,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.426309+0000","last_change":"2024-08-17T17:24:17.458168+0000","last_active":"2024-08-17T17:57:59.426309+0000","last_peered":"2024-08-17T17:57:59.426309+0000","last_clean":"2024-08-17T17:57:59.426309+0000","last_became_active":"2024-08-17T17:24:17.457847+0000","last_became_peered":"2024-08-17T17:24:17.457847+0000","last_unstale":"2024-08-17T17:57:59.426309+0000","last_undegraded":"2024-08-17T17:57:59.426309+0000","last_fullsized":"2024-08-17T17:57:59.426309+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:28.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":2010,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.876943+0000","last_change":"2024-08-17T17:22:32.883756+0000","last_active":"2024-08-17T17:58:00.876943+0000","last_peered":"2024-08-17T17:58:00.876943+0000","last_clean":"2024-08-17T17:58:00.876943+0000","last_became_active":"2024-08-17T17:22:32.874867+0000","last_became_peered":"2024-08-17T17:22:32.874867+0000","last_unstale":"2024-08-17T17:58:00.876943+0000","last_undegraded":"2024-08-17T17:58:00.876943+0000","last_fullsized":"2024-08-17T17:58:00.876943+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:52:27.375231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":2029,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.427047+0000","last_change":"2024-08-17T17:24:17.456622+0000","last_active":"2024-08-17T17:58:00.427047+0000","last_peered":"2024-08-17T17:58:00.427047+0000","last_clean":"2024-08-17T17:58:00.427047+0000","last_became_active":"2024-08-17T17:24:17.455396+0000","last_became_peered":"2024-08-17T17:24:17.455396+0000","last_unstale":"2024-08-17T17:58:00.427047+0000","last_undegraded":"2024-08-17T17:58:00.427047+0000","last_fullsized":"2024-08-17T17:58:00.427047+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:09.564331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1349,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.426039+0000","last_change":"2024-08-17T17:23:42.930464+0000","last_active":"2024-08-17T17:57:59.426039+0000","last_peered":"2024-08-17T17:57:59.426039+0000","last_clean":"2024-08-17T17:57:59.426039+0000","last_became_active":"2024-08-17T17:23:42.929949+0000","last_became_peered":"2024-08-17T17:23:42.929949+0000","last_unstale":"2024-08-17T17:57:59.426039+0000","last_undegraded":"2024-08-17T17:57:59.426039+0000","last_fullsized":"2024-08-17T17:57:59.426039+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:48:27.092339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2012,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.717061+0000","last_change":"2024-08-17T17:24:17.452009+0000","last_active":"2024-08-17T17:57:59.717061+0000","last_peered":"2024-08-17T17:57:59.717061+0000","last_clean":"2024-08-17T17:57:59.717061+0000","last_became_active":"2024-08-17T17:24:17.451492+0000","last_became_peered":"2024-08-17T17:24:17.451492+0000","last_unstale":"2024-08-17T17:57:59.717061+0000","last_undegraded":"2024-08-17T17:57:59.717061+0000","last_fullsized":"2024-08-17T17:57:59.717061+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:21:24.699633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2026,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.429095+0000","last_change":"2024-08-17T17:24:17.452025+0000","last_active":"2024-08-17T17:58:00.429095+0000","last_peered":"2024-08-17T17:58:00.429095+0000","last_clean":"2024-08-17T17:58:00.429095+0000","last_became_active":"2024-08-17T17:24:17.448029+0000","last_became_peered":"2024-08-17T17:24:17.448029+0000","last_unstale":"2024-08-17T17:58:00.429095+0000","last_undegraded":"2024-08-17T17:58:00.429095+0000","last_fullsized":"2024-08-17T17:58:00.429095+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:45.089346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2021,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.877081+0000","last_change":"2024-08-17T17:24:17.469096+0000","last_active":"2024-08-17T17:58:00.877081+0000","last_peered":"2024-08-17T17:58:00.877081+0000","last_clean":"2024-08-17T17:58:00.877081+0000","last_became_active":"2024-08-17T17:24:17.468669+0000","last_became_peered":"2024-08-17T17:24:17.468669+0000","last_unstale":"2024-08-17T17:58:00.877081+0000","last_undegraded":"2024-08-17T17:58:00.877081+0000","last_fullsized":"2024-08-17T17:58:00.877081+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:03:39.427341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2012,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.716480+0000","last_change":"2024-08-17T17:24:17.453946+0000","last_active":"2024-08-17T17:57:59.716480+0000","last_peered":"2024-08-17T17:57:59.716480+0000","last_clean":"2024-08-17T17:57:59.716480+0000","last_became_active":"2024-08-17T17:24:17.445004+0000","last_became_peered":"2024-08-17T17:24:17.445004+0000","last_unstale":"2024-08-17T17:57:59.716480+0000","last_undegraded":"2024-08-17T17:57:59.716480+0000","last_fullsized":"2024-08-17T17:57:59.716480+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:00:19.366629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1333,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.718056+0000","last_change":"2024-08-17T17:23:07.402338+0000","last_active":"2024-08-17T17:57:59.718056+0000","last_peered":"2024-08-17T17:57:59.718056+0000","last_clean":"2024-08-17T17:57:59.718056+0000","last_became_active":"2024-08-17T17:23:07.401996+0000","last_became_peered":"2024-08-17T17:23:07.401996+0000","last_unstale":"2024-08-17T17:57:59.718056+0000","last_undegraded":"2024-08-17T17:57:59.718056+0000","last_fullsized":"2024-08-17T17:57:59.718056+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:39:21.566786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1997,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.716557+0000","last_change":"2024-08-17T17:24:17.442338+0000","last_active":"2024-08-17T17:57:59.716557+0000","last_peered":"2024-08-17T17:57:59.716557+0000","last_clean":"2024-08-17T17:57:59.716557+0000","last_became_active":"2024-08-17T17:24:17.441948+0000","last_became_peered":"2024-08-17T17:24:17.441948+0000","last_unstale":"2024-08-17T17:57:59.716557+0000","last_undegraded":"2024-08-17T17:57:59.716557+0000","last_fullsized":"2024-08-17T17:57:59.716557+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:17.891492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2017,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.877290+0000","last_change":"2024-08-17T17:24:17.459083+0000","last_active":"2024-08-17T17:58:00.877290+0000","last_peered":"2024-08-17T17:58:00.877290+0000","last_clean":"2024-08-17T17:58:00.877290+0000","last_became_active":"2024-08-17T17:24:17.458759+0000","last_became_peered":"2024-08-17T17:24:17.458759+0000","last_unstale":"2024-08-17T17:58:00.877290+0000","last_undegraded":"2024-08-17T17:58:00.877290+0000","last_fullsized":"2024-08-17T17:58:00.877290+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:11:59.301864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1956,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.716637+0000","last_change":"2024-08-17T17:24:17.468086+0000","last_active":"2024-08-17T17:57:59.716637+0000","last_peered":"2024-08-17T17:57:59.716637+0000","last_clean":"2024-08-17T17:57:59.716637+0000","last_became_active":"2024-08-17T17:24:17.467522+0000","last_became_peered":"2024-08-17T17:24:17.467522+0000","last_unstale":"2024-08-17T17:57:59.716637+0000","last_undegraded":"2024-08-17T17:57:59.716637+0000","last_fullsized":"2024-08-17T17:57:59.716637+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:52:55.249649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1328,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.427707+0000","last_change":"2024-08-17T17:23:07.392447+0000","last_active":"2024-08-17T17:57:59.427707+0000","last_peered":"2024-08-17T17:57:59.427707+0000","last_clean":"2024-08-17T17:57:59.427707+0000","last_became_active":"2024-08-17T17:23:07.391765+0000","last_became_peered":"2024-08-17T17:23:07.391765+0000","last_unstale":"2024-08-17T17:57:59.427707+0000","last_undegraded":"2024-08-17T17:57:59.427707+0000","last_fullsized":"2024-08-17T17:57:59.427707+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:59:18.602346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":108,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.716986+0000","last_change":"2024-08-17T17:53:58.839889+0000","last_active":"2024-08-17T17:57:59.716986+0000","last_peered":"2024-08-17T17:57:59.716986+0000","last_clean":"2024-08-17T17:57:59.716986+0000","last_became_active":"2024-08-17T17:53:58.839537+0000","last_became_peered":"2024-08-17T17:53:58.839537+0000","last_unstale":"2024-08-17T17:57:59.716986+0000","last_undegraded":"2024-08-17T17:57:59.716986+0000","last_fullsized":"2024-08-17T17:57:59.716986+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:47.169954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2025,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.428996+0000","last_change":"2024-08-17T17:23:42.936147+0000","last_active":"2024-08-17T17:58:00.428996+0000","last_peered":"2024-08-17T17:58:00.428996+0000","last_clean":"2024-08-17T17:58:00.428996+0000","last_became_active":"2024-08-17T17:23:42.935859+0000","last_became_peered":"2024-08-17T17:23:42.935859+0000","last_unstale":"2024-08-17T17:58:00.428996+0000","last_undegraded":"2024-08-17T17:58:00.428996+0000","last_fullsized":"2024-08-17T17:58:00.428996+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:47:36.838706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1342,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.426271+0000","last_change":"2024-08-17T17:10:05.969131+0000","last_active":"2024-08-17T17:57:59.426271+0000","last_peered":"2024-08-17T17:57:59.426271+0000","last_clean":"2024-08-17T17:57:59.426271+0000","last_became_active":"2024-08-17T17:10:05.968856+0000","last_became_peered":"2024-08-17T17:10:05.968856+0000","last_unstale":"2024-08-17T17:57:59.426271+0000","last_undegraded":"2024-08-17T17:57:59.426271+0000","last_fullsized":"2024-08-17T17:57:59.426271+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:25:49.550065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1333,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.426428+0000","last_change":"2024-08-17T17:23:07.455919+0000","last_active":"2024-08-17T17:57:59.426428+0000","last_peered":"2024-08-17T17:57:59.426428+0000","last_clean":"2024-08-17T17:57:59.426428+0000","last_became_active":"2024-08-17T17:23:07.455284+0000","last_became_peered":"2024-08-17T17:23:07.455284+0000","last_unstale":"2024-08-17T17:57:59.426428+0000","last_undegraded":"2024-08-17T17:57:59.426428+0000","last_fullsized":"2024-08-17T17:57:59.426428+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:37:46.260775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1321,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.426654+0000","last_change":"2024-08-17T17:10:06.979492+0000","last_active":"2024-08-17T17:57:59.426654+0000","last_peered":"2024-08-17T17:57:59.426654+0000","last_clean":"2024-08-17T17:57:59.426654+0000","last_became_active":"2024-08-17T17:10:06.979158+0000","last_became_peered":"2024-08-17T17:10:06.979158+0000","last_unstale":"2024-08-17T17:57:59.426654+0000","last_undegraded":"2024-08-17T17:57:59.426654+0000","last_fullsized":"2024-08-17T17:57:59.426654+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:12:42.261365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1345,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.718111+0000","last_change":"2024-08-17T17:23:42.935099+0000","last_active":"2024-08-17T17:57:59.718111+0000","last_peered":"2024-08-17T17:57:59.718111+0000","last_clean":"2024-08-17T17:57:59.718111+0000","last_became_active":"2024-08-17T17:23:42.934501+0000","last_became_peered":"2024-08-17T17:23:42.934501+0000","last_unstale":"2024-08-17T17:57:59.718111+0000","last_undegraded":"2024-08-17T17:57:59.718111+0000","last_fullsized":"2024-08-17T17:57:59.718111+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:15:19.095713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":2024,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.877056+0000","last_change":"2024-08-17T17:23:07.436763+0000","last_active":"2024-08-17T17:58:00.877056+0000","last_peered":"2024-08-17T17:58:00.877056+0000","last_clean":"2024-08-17T17:58:00.877056+0000","last_became_active":"2024-08-17T17:23:07.434720+0000","last_became_peered":"2024-08-17T17:23:07.434720+0000","last_unstale":"2024-08-17T17:58:00.877056+0000","last_undegraded":"2024-08-17T17:58:00.877056+0000","last_fullsized":"2024-08-17T17:58:00.877056+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:10:20.594115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1992,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.716554+0000","last_change":"2024-08-17T17:24:17.442329+0000","last_active":"2024-08-17T17:57:59.716554+0000","last_peered":"2024-08-17T17:57:59.716554+0000","last_clean":"2024-08-17T17:57:59.716554+0000","last_became_active":"2024-08-17T17:24:17.441666+0000","last_became_peered":"2024-08-17T17:24:17.441666+0000","last_unstale":"2024-08-17T17:57:59.716554+0000","last_undegraded":"2024-08-17T17:57:59.716554+0000","last_fullsized":"2024-08-17T17:57:59.716554+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:43:06.033194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2008,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.429403+0000","last_change":"2024-08-17T17:23:07.428296+0000","last_active":"2024-08-17T17:58:00.429403+0000","last_peered":"2024-08-17T17:58:00.429403+0000","last_clean":"2024-08-17T17:58:00.429403+0000","last_became_active":"2024-08-17T17:23:07.428010+0000","last_became_peered":"2024-08-17T17:23:07.428010+0000","last_unstale":"2024-08-17T17:58:00.429403+0000","last_undegraded":"2024-08-17T17:58:00.429403+0000","last_fullsized":"2024-08-17T17:58:00.429403+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:45.459212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":1993,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.717107+0000","last_change":"2024-08-17T17:24:17.481368+0000","last_active":"2024-08-17T17:57:59.717107+0000","last_peered":"2024-08-17T17:57:59.717107+0000","last_clean":"2024-08-17T17:57:59.717107+0000","last_became_active":"2024-08-17T17:24:17.477509+0000","last_became_peered":"2024-08-17T17:24:17.477509+0000","last_unstale":"2024-08-17T17:57:59.717107+0000","last_undegraded":"2024-08-17T17:57:59.717107+0000","last_fullsized":"2024-08-17T17:57:59.717107+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:05:35.558004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2010,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.877082+0000","last_change":"2024-08-17T17:22:32.884274+0000","last_active":"2024-08-17T17:58:00.877082+0000","last_peered":"2024-08-17T17:58:00.877082+0000","last_clean":"2024-08-17T17:58:00.877082+0000","last_became_active":"2024-08-17T17:22:32.874667+0000","last_became_peered":"2024-08-17T17:22:32.874667+0000","last_unstale":"2024-08-17T17:58:00.877082+0000","last_undegraded":"2024-08-17T17:58:00.877082+0000","last_fullsized":"2024-08-17T17:58:00.877082+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:22:05.834587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1342,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.718142+0000","last_change":"2024-08-17T17:22:32.870569+0000","last_active":"2024-08-17T17:57:59.718142+0000","last_peered":"2024-08-17T17:57:59.718142+0000","last_clean":"2024-08-17T17:57:59.718142+0000","last_became_active":"2024-08-17T17:22:32.869335+0000","last_became_peered":"2024-08-17T17:22:32.869335+0000","last_unstale":"2024-08-17T17:57:59.718142+0000","last_undegraded":"2024-08-17T17:57:59.718142+0000","last_fullsized":"2024-08-17T17:57:59.718142+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:24.969174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2029,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.429280+0000","last_change":"2024-08-17T17:23:42.935825+0000","last_active":"2024-08-17T17:58:00.429280+0000","last_peered":"2024-08-17T17:58:00.429280+0000","last_clean":"2024-08-17T17:58:00.429280+0000","last_became_active":"2024-08-17T17:23:42.935150+0000","last_became_peered":"2024-08-17T17:23:42.935150+0000","last_unstale":"2024-08-17T17:58:00.429280+0000","last_undegraded":"2024-08-17T17:58:00.429280+0000","last_fullsized":"2024-08-17T17:58:00.429280+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:57:19.577644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2017,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.428803+0000","last_change":"2024-08-17T17:23:07.431586+0000","last_active":"2024-08-17T17:58:00.428803+0000","last_peered":"2024-08-17T17:58:00.428803+0000","last_clean":"2024-08-17T17:58:00.428803+0000","last_became_active":"2024-08-17T17:23:07.431301+0000","last_became_peered":"2024-08-17T17:23:07.431301+0000","last_unstale":"2024-08-17T17:58:00.428803+0000","last_undegraded":"2024-08-17T17:58:00.428803+0000","last_fullsized":"2024-08-17T17:58:00.428803+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:28:18.706489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2045,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.429118+0000","last_change":"2024-08-17T17:23:07.420550+0000","last_active":"2024-08-17T17:58:00.429118+0000","last_peered":"2024-08-17T17:58:00.429118+0000","last_clean":"2024-08-17T17:58:00.429118+0000","last_became_active":"2024-08-17T17:23:07.420223+0000","last_became_peered":"2024-08-17T17:23:07.420223+0000","last_unstale":"2024-08-17T17:58:00.429118+0000","last_undegraded":"2024-08-17T17:58:00.429118+0000","last_fullsized":"2024-08-17T17:58:00.429118+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:33:46.996371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1346,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.427783+0000","last_change":"2024-08-17T17:24:17.426267+0000","last_active":"2024-08-17T17:57:59.427783+0000","last_peered":"2024-08-17T17:57:59.427783+0000","last_clean":"2024-08-17T17:57:59.427783+0000","last_became_active":"2024-08-17T17:24:17.425925+0000","last_became_peered":"2024-08-17T17:24:17.425925+0000","last_unstale":"2024-08-17T17:57:59.427783+0000","last_undegraded":"2024-08-17T17:57:59.427783+0000","last_fullsized":"2024-08-17T17:57:59.427783+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:24:25.017856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1342,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.427559+0000","last_change":"2024-08-17T17:24:17.474138+0000","last_active":"2024-08-17T17:57:59.427559+0000","last_peered":"2024-08-17T17:57:59.427559+0000","last_clean":"2024-08-17T17:57:59.427559+0000","last_became_active":"2024-08-17T17:24:17.473708+0000","last_became_peered":"2024-08-17T17:24:17.473708+0000","last_unstale":"2024-08-17T17:57:59.427559+0000","last_undegraded":"2024-08-17T17:57:59.427559+0000","last_fullsized":"2024-08-17T17:57:59.427559+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:10:57.768067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1336,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.426588+0000","last_change":"2024-08-17T17:22:32.871533+0000","last_active":"2024-08-17T17:57:59.426588+0000","last_peered":"2024-08-17T17:57:59.426588+0000","last_clean":"2024-08-17T17:57:59.426588+0000","last_became_active":"2024-08-17T17:22:32.868226+0000","last_became_peered":"2024-08-17T17:22:32.868226+0000","last_unstale":"2024-08-17T17:57:59.426588+0000","last_undegraded":"2024-08-17T17:57:59.426588+0000","last_fullsized":"2024-08-17T17:57:59.426588+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:00:02.629455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":1346,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.427598+0000","last_change":"2024-08-17T17:23:07.391021+0000","last_active":"2024-08-17T17:57:59.427598+0000","last_peered":"2024-08-17T17:57:59.427598+0000","last_clean":"2024-08-17T17:57:59.427598+0000","last_became_active":"2024-08-17T17:23:07.390551+0000","last_became_peered":"2024-08-17T17:23:07.390551+0000","last_unstale":"2024-08-17T17:57:59.427598+0000","last_undegraded":"2024-08-17T17:57:59.427598+0000","last_fullsized":"2024-08-17T17:57:59.427598+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:06.239013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001023532,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2029,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.429398+0000","last_change":"2024-08-17T17:23:07.406362+0000","last_active":"2024-08-17T17:58:00.429398+0000","last_peered":"2024-08-17T17:58:00.429398+0000","last_clean":"2024-08-17T17:58:00.429398+0000","last_became_active":"2024-08-17T17:23:07.406071+0000","last_became_peered":"2024-08-17T17:23:07.406071+0000","last_unstale":"2024-08-17T17:58:00.429398+0000","last_undegraded":"2024-08-17T17:58:00.429398+0000","last_fullsized":"2024-08-17T17:58:00.429398+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:45:47.383432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2009,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.428639+0000","last_change":"2024-08-17T17:23:42.910246+0000","last_active":"2024-08-17T17:58:00.428639+0000","last_peered":"2024-08-17T17:58:00.428639+0000","last_clean":"2024-08-17T17:58:00.428639+0000","last_became_active":"2024-08-17T17:23:42.909929+0000","last_became_peered":"2024-08-17T17:23:42.909929+0000","last_unstale":"2024-08-17T17:58:00.428639+0000","last_undegraded":"2024-08-17T17:58:00.428639+0000","last_fullsized":"2024-08-17T17:58:00.428639+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:14:45.210558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2038,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.426825+0000","last_change":"2024-08-17T17:24:17.438891+0000","last_active":"2024-08-17T17:58:00.426825+0000","last_peered":"2024-08-17T17:58:00.426825+0000","last_clean":"2024-08-17T17:58:00.426825+0000","last_became_active":"2024-08-17T17:24:17.437714+0000","last_became_peered":"2024-08-17T17:24:17.437714+0000","last_unstale":"2024-08-17T17:58:00.426825+0000","last_undegraded":"2024-08-17T17:58:00.426825+0000","last_fullsized":"2024-08-17T17:58:00.426825+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:58.825017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":2031,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.877162+0000","last_change":"2024-08-17T17:23:42.908634+0000","last_active":"2024-08-17T17:58:00.877162+0000","last_peered":"2024-08-17T17:58:00.877162+0000","last_clean":"2024-08-17T17:58:00.877162+0000","last_became_active":"2024-08-17T17:23:42.907675+0000","last_became_peered":"2024-08-17T17:23:42.907675+0000","last_unstale":"2024-08-17T17:58:00.877162+0000","last_undegraded":"2024-08-17T17:58:00.877162+0000","last_fullsized":"2024-08-17T17:58:00.877162+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:03:37.612222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2021,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.876957+0000","last_change":"2024-08-17T17:24:17.457559+0000","last_active":"2024-08-17T17:58:00.876957+0000","last_peered":"2024-08-17T17:58:00.876957+0000","last_clean":"2024-08-17T17:58:00.876957+0000","last_became_active":"2024-08-17T17:24:17.457262+0000","last_became_peered":"2024-08-17T17:24:17.457262+0000","last_unstale":"2024-08-17T17:58:00.876957+0000","last_undegraded":"2024-08-17T17:58:00.876957+0000","last_fullsized":"2024-08-17T17:58:00.876957+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:53:38.137277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":743,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.428854+0000","last_change":"2024-08-17T17:53:58.844594+0000","last_active":"2024-08-17T17:58:00.428854+0000","last_peered":"2024-08-17T17:58:00.428854+0000","last_clean":"2024-08-17T17:58:00.428854+0000","last_became_active":"2024-08-17T17:53:58.844085+0000","last_became_peered":"2024-08-17T17:53:58.844085+0000","last_unstale":"2024-08-17T17:58:00.428854+0000","last_undegraded":"2024-08-17T17:58:00.428854+0000","last_fullsized":"2024-08-17T17:58:00.428854+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:58:50.195797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1317,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.717717+0000","last_change":"2024-08-17T17:10:06.995023+0000","last_active":"2024-08-17T17:57:59.717717+0000","last_peered":"2024-08-17T17:57:59.717717+0000","last_clean":"2024-08-17T17:57:59.717717+0000","last_became_active":"2024-08-17T17:10:06.994757+0000","last_became_peered":"2024-08-17T17:10:06.994757+0000","last_unstale":"2024-08-17T17:57:59.717717+0000","last_undegraded":"2024-08-17T17:57:59.717717+0000","last_fullsized":"2024-08-17T17:57:59.717717+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:31:52.163518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1959,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.716651+0000","last_change":"2024-08-17T17:24:17.468443+0000","last_active":"2024-08-17T17:57:59.716651+0000","last_peered":"2024-08-17T17:57:59.716651+0000","last_clean":"2024-08-17T17:57:59.716651+0000","last_became_active":"2024-08-17T17:24:17.467394+0000","last_became_peered":"2024-08-17T17:24:17.467394+0000","last_unstale":"2024-08-17T17:57:59.716651+0000","last_undegraded":"2024-08-17T17:57:59.716651+0000","last_fullsized":"2024-08-17T17:57:59.716651+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:13.024253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":1345,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.427495+0000","last_change":"2024-08-17T17:23:42.920552+0000","last_active":"2024-08-17T17:57:59.427495+0000","last_peered":"2024-08-17T17:57:59.427495+0000","last_clean":"2024-08-17T17:57:59.427495+0000","last_became_active":"2024-08-17T17:23:42.915824+0000","last_became_peered":"2024-08-17T17:23:42.915824+0000","last_unstale":"2024-08-17T17:57:59.427495+0000","last_undegraded":"2024-08-17T17:57:59.427495+0000","last_fullsized":"2024-08-17T17:57:59.427495+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:25:18.769504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041737399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":713,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.876703+0000","last_change":"2024-08-17T17:53:58.844782+0000","last_active":"2024-08-17T17:58:00.876703+0000","last_peered":"2024-08-17T17:58:00.876703+0000","last_clean":"2024-08-17T17:58:00.876703+0000","last_became_active":"2024-08-17T17:53:58.843133+0000","last_became_peered":"2024-08-17T17:53:58.843133+0000","last_unstale":"2024-08-17T17:58:00.876703+0000","last_undegraded":"2024-08-17T17:58:00.876703+0000","last_fullsized":"2024-08-17T17:58:00.876703+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T17:06:21.174471+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:30:14.411310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2007,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.429310+0000","last_change":"2024-08-17T17:24:17.455710+0000","last_active":"2024-08-17T17:58:00.429310+0000","last_peered":"2024-08-17T17:58:00.429310+0000","last_clean":"2024-08-17T17:58:00.429310+0000","last_became_active":"2024-08-17T17:24:17.452989+0000","last_became_peered":"2024-08-17T17:24:17.452989+0000","last_unstale":"2024-08-17T17:58:00.429310+0000","last_undegraded":"2024-08-17T17:58:00.429310+0000","last_fullsized":"2024-08-17T17:58:00.429310+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:58:07.894968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1999,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.716451+0000","last_change":"2024-08-17T17:24:17.453639+0000","last_active":"2024-08-17T17:57:59.716451+0000","last_peered":"2024-08-17T17:57:59.716451+0000","last_clean":"2024-08-17T17:57:59.716451+0000","last_became_active":"2024-08-17T17:24:17.443638+0000","last_became_peered":"2024-08-17T17:24:17.443638+0000","last_unstale":"2024-08-17T17:57:59.716451+0000","last_undegraded":"2024-08-17T17:57:59.716451+0000","last_fullsized":"2024-08-17T17:57:59.716451+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:42.107479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1334,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.426616+0000","last_change":"2024-08-17T17:23:07.451866+0000","last_active":"2024-08-17T17:57:59.426616+0000","last_peered":"2024-08-17T17:57:59.426616+0000","last_clean":"2024-08-17T17:57:59.426616+0000","last_became_active":"2024-08-17T17:23:07.451553+0000","last_became_peered":"2024-08-17T17:23:07.451553+0000","last_unstale":"2024-08-17T17:57:59.426616+0000","last_undegraded":"2024-08-17T17:57:59.426616+0000","last_fullsized":"2024-08-17T17:57:59.426616+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:37:23.930651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1336,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.426696+0000","last_change":"2024-08-17T17:23:42.915851+0000","last_active":"2024-08-17T17:57:59.426696+0000","last_peered":"2024-08-17T17:57:59.426696+0000","last_clean":"2024-08-17T17:57:59.426696+0000","last_became_active":"2024-08-17T17:23:42.915552+0000","last_became_peered":"2024-08-17T17:23:42.915552+0000","last_unstale":"2024-08-17T17:57:59.426696+0000","last_undegraded":"2024-08-17T17:57:59.426696+0000","last_fullsized":"2024-08-17T17:57:59.426696+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:45.197876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1349,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.426771+0000","last_change":"2024-08-17T17:23:42.930803+0000","last_active":"2024-08-17T17:57:59.426771+0000","last_peered":"2024-08-17T17:57:59.426771+0000","last_clean":"2024-08-17T17:57:59.426771+0000","last_became_active":"2024-08-17T17:23:42.930458+0000","last_became_peered":"2024-08-17T17:23:42.930458+0000","last_unstale":"2024-08-17T17:57:59.426771+0000","last_undegraded":"2024-08-17T17:57:59.426771+0000","last_fullsized":"2024-08-17T17:57:59.426771+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:40.816706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1333,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.717444+0000","last_change":"2024-08-17T17:23:07.401624+0000","last_active":"2024-08-17T17:57:59.717444+0000","last_peered":"2024-08-17T17:57:59.717444+0000","last_clean":"2024-08-17T17:57:59.717444+0000","last_became_active":"2024-08-17T17:23:07.400222+0000","last_became_peered":"2024-08-17T17:23:07.400222+0000","last_unstale":"2024-08-17T17:57:59.717444+0000","last_undegraded":"2024-08-17T17:57:59.717444+0000","last_fullsized":"2024-08-17T17:57:59.717444+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:07.069989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1334,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.426155+0000","last_change":"2024-08-17T17:24:17.470809+0000","last_active":"2024-08-17T17:57:59.426155+0000","last_peered":"2024-08-17T17:57:59.426155+0000","last_clean":"2024-08-17T17:57:59.426155+0000","last_became_active":"2024-08-17T17:24:17.469753+0000","last_became_peered":"2024-08-17T17:24:17.469753+0000","last_unstale":"2024-08-17T17:57:59.426155+0000","last_undegraded":"2024-08-17T17:57:59.426155+0000","last_fullsized":"2024-08-17T17:57:59.426155+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:00.049425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":1992,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.716718+0000","last_change":"2024-08-17T17:24:17.461258+0000","last_active":"2024-08-17T17:57:59.716718+0000","last_peered":"2024-08-17T17:57:59.716718+0000","last_clean":"2024-08-17T17:57:59.716718+0000","last_became_active":"2024-08-17T17:24:17.460895+0000","last_became_peered":"2024-08-17T17:24:17.460895+0000","last_unstale":"2024-08-17T17:57:59.716718+0000","last_undegraded":"2024-08-17T17:57:59.716718+0000","last_fullsized":"2024-08-17T17:57:59.716718+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:43:24.208259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2033,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.876834+0000","last_change":"2024-08-17T17:23:42.901638+0000","last_active":"2024-08-17T17:58:00.876834+0000","last_peered":"2024-08-17T17:58:00.876834+0000","last_clean":"2024-08-17T17:58:00.876834+0000","last_became_active":"2024-08-17T17:23:42.901335+0000","last_became_peered":"2024-08-17T17:23:42.901335+0000","last_unstale":"2024-08-17T17:58:00.876834+0000","last_undegraded":"2024-08-17T17:58:00.876834+0000","last_fullsized":"2024-08-17T17:58:00.876834+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:25:35.332839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1330,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.427831+0000","last_change":"2024-08-17T17:24:17.473913+0000","last_active":"2024-08-17T17:57:59.427831+0000","last_peered":"2024-08-17T17:57:59.427831+0000","last_clean":"2024-08-17T17:57:59.427831+0000","last_became_active":"2024-08-17T17:24:17.473195+0000","last_became_peered":"2024-08-17T17:24:17.473195+0000","last_unstale":"2024-08-17T17:57:59.427831+0000","last_undegraded":"2024-08-17T17:57:59.427831+0000","last_fullsized":"2024-08-17T17:57:59.427831+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:12.998033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1332,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.717308+0000","last_change":"2024-08-17T17:23:07.402010+0000","last_active":"2024-08-17T17:57:59.717308+0000","last_peered":"2024-08-17T17:57:59.717308+0000","last_clean":"2024-08-17T17:57:59.717308+0000","last_became_active":"2024-08-17T17:23:07.400849+0000","last_became_peered":"2024-08-17T17:23:07.400849+0000","last_unstale":"2024-08-17T17:57:59.717308+0000","last_undegraded":"2024-08-17T17:57:59.717308+0000","last_fullsized":"2024-08-17T17:57:59.717308+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:16:02.382257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":67,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.427787+0000","last_change":"2024-08-17T17:53:58.836811+0000","last_active":"2024-08-17T17:57:59.427787+0000","last_peered":"2024-08-17T17:57:59.427787+0000","last_clean":"2024-08-17T17:57:59.427787+0000","last_became_active":"2024-08-17T17:53:58.836139+0000","last_became_peered":"2024-08-17T17:53:58.836139+0000","last_unstale":"2024-08-17T17:57:59.427787+0000","last_undegraded":"2024-08-17T17:57:59.427787+0000","last_fullsized":"2024-08-17T17:57:59.427787+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:09:18.260164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1350,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.426451+0000","last_change":"2024-08-17T17:24:17.460203+0000","last_active":"2024-08-17T17:57:59.426451+0000","last_peered":"2024-08-17T17:57:59.426451+0000","last_clean":"2024-08-17T17:57:59.426451+0000","last_became_active":"2024-08-17T17:24:17.459682+0000","last_became_peered":"2024-08-17T17:24:17.459682+0000","last_unstale":"2024-08-17T17:57:59.426451+0000","last_undegraded":"2024-08-17T17:57:59.426451+0000","last_fullsized":"2024-08-17T17:57:59.426451+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:58.703700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":1330,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.718221+0000","last_change":"2024-08-17T17:23:07.454042+0000","last_active":"2024-08-17T17:57:59.718221+0000","last_peered":"2024-08-17T17:57:59.718221+0000","last_clean":"2024-08-17T17:57:59.718221+0000","last_became_active":"2024-08-17T17:23:07.453712+0000","last_became_peered":"2024-08-17T17:23:07.453712+0000","last_unstale":"2024-08-17T17:57:59.718221+0000","last_undegraded":"2024-08-17T17:57:59.718221+0000","last_fullsized":"2024-08-17T17:57:59.718221+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:03:07.643456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1315,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.427475+0000","last_change":"2024-08-17T17:10:06.994799+0000","last_active":"2024-08-17T17:57:59.427475+0000","last_peered":"2024-08-17T17:57:59.427475+0000","last_clean":"2024-08-17T17:57:59.427475+0000","last_became_active":"2024-08-17T17:10:06.994150+0000","last_became_peered":"2024-08-17T17:10:06.994150+0000","last_unstale":"2024-08-17T17:57:59.427475+0000","last_undegraded":"2024-08-17T17:57:59.427475+0000","last_fullsized":"2024-08-17T17:57:59.427475+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:50:47.293091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":1328,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.717819+0000","last_change":"2024-08-17T17:22:32.873334+0000","last_active":"2024-08-17T17:57:59.717819+0000","last_peered":"2024-08-17T17:57:59.717819+0000","last_clean":"2024-08-17T17:57:59.717819+0000","last_became_active":"2024-08-17T17:22:32.870147+0000","last_became_peered":"2024-08-17T17:22:32.870147+0000","last_unstale":"2024-08-17T17:57:59.717819+0000","last_undegraded":"2024-08-17T17:57:59.717819+0000","last_fullsized":"2024-08-17T17:57:59.717819+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:18:03.750592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1339,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.717112+0000","last_change":"2024-08-17T17:22:32.847989+0000","last_active":"2024-08-17T17:57:59.717112+0000","last_peered":"2024-08-17T17:57:59.717112+0000","last_clean":"2024-08-17T17:57:59.717112+0000","last_became_active":"2024-08-17T17:22:32.846593+0000","last_became_peered":"2024-08-17T17:22:32.846593+0000","last_unstale":"2024-08-17T17:57:59.717112+0000","last_undegraded":"2024-08-17T17:57:59.717112+0000","last_fullsized":"2024-08-17T17:57:59.717112+0000","mapping_epoch":659,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":660,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:33:50.370124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1996,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.717098+0000","last_change":"2024-08-17T17:24:17.451528+0000","last_active":"2024-08-17T17:57:59.717098+0000","last_peered":"2024-08-17T17:57:59.717098+0000","last_clean":"2024-08-17T17:57:59.717098+0000","last_became_active":"2024-08-17T17:24:17.451246+0000","last_became_peered":"2024-08-17T17:24:17.451246+0000","last_unstale":"2024-08-17T17:57:59.717098+0000","last_undegraded":"2024-08-17T17:57:59.717098+0000","last_fullsized":"2024-08-17T17:57:59.717098+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:38:18.176559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":55,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.426552+0000","last_change":"2024-08-17T17:53:58.838279+0000","last_active":"2024-08-17T17:57:59.426552+0000","last_peered":"2024-08-17T17:57:59.426552+0000","last_clean":"2024-08-17T17:57:59.426552+0000","last_became_active":"2024-08-17T17:53:58.837978+0000","last_became_peered":"2024-08-17T17:53:58.837978+0000","last_unstale":"2024-08-17T17:57:59.426552+0000","last_undegraded":"2024-08-17T17:57:59.426552+0000","last_fullsized":"2024-08-17T17:57:59.426552+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:06.019142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2017,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.876988+0000","last_change":"2024-08-17T17:24:17.460371+0000","last_active":"2024-08-17T17:58:00.876988+0000","last_peered":"2024-08-17T17:58:00.876988+0000","last_clean":"2024-08-17T17:58:00.876988+0000","last_became_active":"2024-08-17T17:24:17.460074+0000","last_became_peered":"2024-08-17T17:24:17.460074+0000","last_unstale":"2024-08-17T17:58:00.876988+0000","last_undegraded":"2024-08-17T17:58:00.876988+0000","last_fullsized":"2024-08-17T17:58:00.876988+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:08:48.820168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":2018,"reported_epoch":1361,"state":"active+clean","last_fresh":"2024-08-17T17:58:00.876809+0000","last_change":"2024-08-17T17:24:17.457942+0000","last_active":"2024-08-17T17:58:00.876809+0000","last_peered":"2024-08-17T17:58:00.876809+0000","last_clean":"2024-08-17T17:58:00.876809+0000","last_became_active":"2024-08-17T17:24:17.456475+0000","last_became_peered":"2024-08-17T17:24:17.456475+0000","last_unstale":"2024-08-17T17:58:00.876809+0000","last_undegraded":"2024-08-17T17:58:00.876809+0000","last_fullsized":"2024-08-17T17:58:00.876809+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:12.325657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1327,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.427707+0000","last_change":"2024-08-17T17:24:17.451287+0000","last_active":"2024-08-17T17:57:59.427707+0000","last_peered":"2024-08-17T17:57:59.427707+0000","last_clean":"2024-08-17T17:57:59.427707+0000","last_became_active":"2024-08-17T17:24:17.450951+0000","last_became_peered":"2024-08-17T17:24:17.450951+0000","last_unstale":"2024-08-17T17:57:59.427707+0000","last_undegraded":"2024-08-17T17:57:59.427707+0000","last_fullsized":"2024-08-17T17:57:59.427707+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:13:37.590108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":1329,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.717386+0000","last_change":"2024-08-17T17:23:42.907970+0000","last_active":"2024-08-17T17:57:59.717386+0000","last_peered":"2024-08-17T17:57:59.717386+0000","last_clean":"2024-08-17T17:57:59.717386+0000","last_became_active":"2024-08-17T17:23:42.907595+0000","last_became_peered":"2024-08-17T17:23:42.907595+0000","last_unstale":"2024-08-17T17:57:59.717386+0000","last_undegraded":"2024-08-17T17:57:59.717386+0000","last_fullsized":"2024-08-17T17:57:59.717386+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:11:46.130004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1997,"reported_epoch":1360,"state":"active+clean","last_fresh":"2024-08-17T17:57:59.716880+0000","last_change":"2024-08-17T17:24:17.486591+0000","last_active":"2024-08-17T17:57:59.716880+0000","last_peered":"2024-08-17T17:57:59.716880+0000","last_clean":"2024-08-17T17:57:59.716880+0000","last_became_active":"2024-08-17T17:24:17.486320+0000","last_became_peered":"2024-08-17T17:24:17.486320+0000","last_unstale":"2024-08-17T17:57:59.716880+0000","last_undegraded":"2024-08-17T17:57:59.716880+0000","last_fullsized":"2024-08-17T17:57:59.716880+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:03.249803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":53,"num_pg":26,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":202896,"num_write_kb":5279553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19337,"num_bytes_recovered":836108288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":198836224,"data_stored":198836224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":215057,"ondisk_log_size":215057,"up":78,"acting":78,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234584,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":193924,"kb_used_data":10296,"kb_used_omap":1,"kb_used_meta":183614,"kb_avail":93551228,"statfs":{"total":95995035648,"available":95796457472,"internally_reserved":0,"allocated":10543104,"data_stored":6328744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":188021195},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496222,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":284160,"kb_used_data":10288,"kb_used_omap":1,"kb_used_meta":273854,"kb_avail":93460992,"statfs":{"total":95995035648,"available":95704055808,"internally_reserved":0,"allocated":10534912,"data_stored":6328617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":280426951},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565463,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":365080,"kb_used_data":148300,"kb_used_omap":72,"kb_used_meta":216695,"kb_avail":93380072,"statfs":{"total":95995035648,"available":95621193728,"internally_reserved":0,"allocated":151859200,"data_stored":147675761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":221896174},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":674,"seq":2894807957918,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":230576,"kb_used_data":9696,"kb_used_omap":74,"kb_used_meta":220789,"kb_avail":93514576,"statfs":{"total":95995035648,"available":95758925824,"internally_reserved":0,"allocated":9928704,"data_stored":5734032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76116,"internal_metadata":226088620},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743186853,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":407912,"kb_used_data":69084,"kb_used_omap":66,"kb_used_meta":338749,"kb_avail":93337240,"statfs":{"total":95995035648,"available":95577333760,"internally_reserved":0,"allocated":70742016,"data_stored":66550843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":346879659},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448492,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":271200,"kb_used_data":10260,"kb_used_omap":75,"kb_used_meta":260852,"kb_avail":93473952,"statfs":{"total":95995035648,"available":95717326848,"internally_reserved":0,"allocated":10506240,"data_stored":6315667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":267112639},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757858,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":243444,"kb_used_data":9700,"kb_used_omap":1,"kb_used_meta":233726,"kb_avail":93501708,"statfs":{"total":95995035648,"available":95745748992,"internally_reserved":0,"allocated":9932800,"data_stored":5734075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":239335878},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019495,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":391872,"kb_used_data":9712,"kb_used_omap":1,"kb_used_meta":382142,"kb_avail":93353280,"statfs":{"total":95995035648,"available":95593758720,"internally_reserved":0,"allocated":9945088,"data_stored":5738306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":391313862},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":60817408,"data_stored":60817408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":138018816,"data_stored":138018816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":53,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:58:06.934 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-08-17T17:58:06.934 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:58:06.935 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-17T17:58:06.935 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd out 0 2024-08-17T17:58:06.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:06.999 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:06.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:07.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:07.101 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:07.123 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:07.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:07.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:07.224 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:07.245 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:07.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:07.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:07.347 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:07.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:07.369 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:07.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:07.470 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:07.491 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:07.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:07.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:07.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:07.615 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:07.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:07.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:07.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:07.737 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:07.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:07.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:07.764 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:58:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:58:07] "GET /metrics HTTP/1.1" 200 34129 "" "Prometheus/2.51.0" 2024-08-17T17:58:07.839 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:07.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:07.862 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:07.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:07.962 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:07.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:07.985 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:07.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:08.062 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-17T17:58:08.063 INFO:tasks.radosbench:Beginning radosbench... 2024-08-17T17:58:08.063 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-08-17T17:58:08.063 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-08-17T17:58:08.086 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:08.108 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:08.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:08.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:08.198 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T17:58:08.198 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:58:08.210 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:08.231 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:08.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:08.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:08.332 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:08.354 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:08.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:08.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:08.456 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:08.478 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:08.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:08.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:08.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:08.602 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:08.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:08.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:08.704 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:08.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:08.726 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:08.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:08 smithi165 ceph-mon[105517]: pgmap v2280: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:08.827 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:08.849 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:08.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:08.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:08.940 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:08 smithi136 ceph-mon[109861]: pgmap v2280: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:08.940 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:08 smithi136 ceph-mon[115078]: pgmap v2280: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:08.951 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:08.961 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-17T17:58:08.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:08.973 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:08.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:09.074 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:09.096 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:09.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:09.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:09.197 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:09.219 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:09.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:09.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:09.320 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:09.342 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:09.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:09.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:09.432 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:58:09.444 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:09.465 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:09.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:09.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:09.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:09.588 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:09.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:09.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:09.690 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:09.711 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:09.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:09.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:09.813 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:09.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:09.835 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:09.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:09.936 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:09.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:09.959 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:09.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:10.061 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:10.082 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:10.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:10.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:10.184 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:10.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:10.207 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:10.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:10.308 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:10.309 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:58:10.309 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:58:10.309 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:58:10.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:10.332 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:10.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:10.433 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:10.453 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:10.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:10.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:10.555 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:10.578 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:10.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:10.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:10.680 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:10.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:10 smithi136 ceph-mon[115078]: pgmap v2281: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:10.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:10 smithi136 ceph-mon[109861]: pgmap v2281: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:10.701 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:10.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:10.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:10.803 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:10.829 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:10.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:10.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:10.930 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:10.954 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:10.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:10.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:11.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:10 smithi165 ceph-mon[105517]: pgmap v2281: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:11.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:11.080 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:11.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:11.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:11.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:11.206 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:11.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:11.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:11.308 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:11.332 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:11.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:11.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:11.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:11.457 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:11.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:11.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:11.559 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:11.582 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:11.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:11.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:11.684 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:11.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:11.708 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:11.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:11.809 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:11.834 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:11.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:11.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:11.937 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:11.959 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:11.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:11.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:12.061 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:12.082 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:12.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:12.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:12.185 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:12.211 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:12.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:12.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:12.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:12.333 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:12.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:12.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:12.435 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:12.456 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:12.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:12.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:12.521 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T17:58:12.558 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:12.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:12.580 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:12.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:12.681 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:12.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:12 smithi136 ceph-mon[109861]: pgmap v2282: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:12.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:12 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1536674186' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:58:12.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:12 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:58:12.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:58:12.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:58:12.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:12 smithi136 ceph-mon[115078]: pgmap v2282: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:12.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:12 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1536674186' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:58:12.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:12 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:58:12.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:58:12.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:58:12.703 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:12.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:12.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:12.805 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:12.826 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:12.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:12.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:12.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:12.954 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:12.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:12.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:12 smithi165 ceph-mon[105517]: pgmap v2282: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:12 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1536674186' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:58:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:12 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:58:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:12 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:58:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:12 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:58:13.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:13.082 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:13.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:13.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:13.184 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:13.208 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:13.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:13.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:13.310 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:13.334 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:13.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:13.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:13.436 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:13.457 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:13.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:13.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:13.559 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:13.581 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:13.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:13.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:13.683 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:13.708 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:13.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:13.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:13.810 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:13.816 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:13 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:58:13.816 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:13 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T17:58:13.816 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:13 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:58:13.817 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:13 smithi136 ceph-mon[109861]: osdmap e1363: 8 total, 8 up, 8 in 2024-08-17T17:58:13.817 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:13 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:58:13.817 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:13 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T17:58:13.817 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:13 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:58:13.817 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:13 smithi136 ceph-mon[115078]: osdmap e1363: 8 total, 8 up, 8 in 2024-08-17T17:58:13.833 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:13.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:13.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:13.935 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:13.957 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:13.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:13.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:13 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:58:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:13 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T17:58:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:13 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:58:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:13 smithi165 ceph-mon[105517]: osdmap e1363: 8 total, 8 up, 8 in 2024-08-17T17:58:14.059 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:14.080 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:14.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:14.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:14.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:14.206 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:14.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:14.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:14.308 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:14.329 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:14.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:14.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:14.431 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:14.453 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:14.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:14.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:14.555 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:14.577 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:14.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:14.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:14.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:14.705 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:14.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:14.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:14.806 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:14.832 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:14.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:14.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:14.934 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:14.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:14 smithi136 ceph-mon[109861]: pgmap v2284: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:14.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:14 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:58:14.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:14 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:58:14.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:14 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:58:14.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:14 smithi136 ceph-mon[115078]: pgmap v2284: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:14.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:14 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:58:14.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:14 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:58:14.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:14 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:58:14.955 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:14.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:14.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:14 smithi165 ceph-mon[105517]: pgmap v2284: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:14 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:58:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:14 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:58:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:14 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:58:15.057 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:15.078 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:15.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:15.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:15.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:15.201 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:15.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:15.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:15.302 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:15.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:15.325 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:15.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:15.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:15.451 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:15.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:15.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:15.553 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:15.577 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:15.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:15.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:15.679 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:15.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:15.702 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:15.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:15.803 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:15.824 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:15.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:15.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:15.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:15.952 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:15.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:15.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:16.053 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:16.076 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:16.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:16.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:16.178 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:16.199 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:16.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:16.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:16.301 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:16.327 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:16.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:16.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:16.428 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:16.452 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:16.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:16.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:16.553 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:16.576 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:16.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:16.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:16.678 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:16.687 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:16 smithi136 ceph-mon[115078]: pgmap v2285: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:16.687 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:16 smithi136 ceph-mon[109861]: pgmap v2285: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:16.701 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:16.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:16.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:16 smithi165 ceph-mon[105517]: pgmap v2285: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:16.802 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:16.824 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:16.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:16.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:16.926 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:16.947 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:16.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:16.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:17.049 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:17.072 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:17.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:17.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:17.174 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:17.196 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:17.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:17.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:17.298 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:17.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:17.321 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:17.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:17.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:17.445 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:17.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:17.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:17.465 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:58:17.526 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T17:58:17.527 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:58:17.547 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:17.569 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:17.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:17.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:17.670 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:17.692 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:17.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:17.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:17.794 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:17.817 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:17.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:17.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:17.878 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:58:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:58:17] "GET /metrics HTTP/1.1" 200 34129 "" "Prometheus/2.51.0" 2024-08-17T17:58:17.919 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:17.941 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:17.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:17.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:18.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:18.064 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:18.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:18.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:18.165 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:18.187 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:18.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:18.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:18.289 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:18.310 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:18.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:18.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:18.378 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-17T17:58:18.412 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:18.441 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:18.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:18.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:18.535 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'unique_pool_1' created 2024-08-17T17:58:18.543 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:18.564 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:18.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:18.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:18.619 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:18 smithi136 ceph-mon[109861]: pgmap v2286: 73 pgs: 73 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:18.619 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:18 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4276431402' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-08-17T17:58:18.619 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:18 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-08-17T17:58:18.619 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:18 smithi136 ceph-mon[115078]: pgmap v2286: 73 pgs: 73 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:18.619 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:18 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4276431402' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-08-17T17:58:18.619 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:18 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-08-17T17:58:18.666 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:18.687 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:18.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:18.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:18.789 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:18.810 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:18.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:18.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:18.912 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:18.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:18.935 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:18.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:18 smithi165 ceph-mon[105517]: pgmap v2286: 73 pgs: 73 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:18 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4276431402' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-08-17T17:58:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:18 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-08-17T17:58:19.012 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T17:58:19.013 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:58:19.034 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T17:58:19.035 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:58:19.036 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:19.053 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T17:58:19.053 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:58:19.057 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:19.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:19.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:19.069 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T17:58:19.069 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:58:19.085 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T17:58:19.086 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:58:19.101 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T17:58:19.101 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:58:19.117 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T17:58:19.117 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:58:19.135 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T17:58:19.135 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:58:19.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:19.180 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:19.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:19.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:19.282 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:19.305 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:19.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:19.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:19.407 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:19.428 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:19.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:19.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:19.530 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:19.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:19.556 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:19.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:19.657 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:19.678 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:19.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:19.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:19.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:19.796 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:19 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-08-17T17:58:19.796 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:19 smithi136 ceph-mon[109861]: osdmap e1364: 8 total, 8 up, 8 in 2024-08-17T17:58:19.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:19 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2924891802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:58:19.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:19 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-08-17T17:58:19.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:19 smithi136 ceph-mon[115078]: osdmap e1364: 8 total, 8 up, 8 in 2024-08-17T17:58:19.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:19 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2924891802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:58:19.803 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:19.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:19.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:19.905 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:19.927 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:19.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:19.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:19 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-08-17T17:58:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:19 smithi165 ceph-mon[105517]: osdmap e1364: 8 total, 8 up, 8 in 2024-08-17T17:58:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:19 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2924891802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:58:20.028 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:20.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:20.050 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:20.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:20.115 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:58:20.151 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:20.172 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:20.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:20.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:20.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:20.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:20.296 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:20.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:20.396 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:20.417 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:20.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:20.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:20.518 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:20.540 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:20.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:20.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:20.627 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-08-17T17:58:20.639 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:58:20.642 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:20.663 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:20.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:20.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:20.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:20.786 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:20.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:20.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:20.871 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:20 smithi136 ceph-mon[109861]: pgmap v2288: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:20.871 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:20 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:58:20.871 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:20 smithi136 ceph-mon[109861]: osdmap e1365: 8 total, 8 up, 8 in 2024-08-17T17:58:20.871 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:20 smithi136 ceph-mon[115078]: pgmap v2288: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:20.872 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:20 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:58:20.872 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:20 smithi136 ceph-mon[115078]: osdmap e1365: 8 total, 8 up, 8 in 2024-08-17T17:58:20.888 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:20.910 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:20.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:20.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:21.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:20 smithi165 ceph-mon[105517]: pgmap v2288: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:21.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:20 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:58:21.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:20 smithi165 ceph-mon[105517]: osdmap e1365: 8 total, 8 up, 8 in 2024-08-17T17:58:21.011 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:21.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:21.033 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:21.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:21.134 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:21.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:21.156 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:21.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:21.257 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:21.278 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:21.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:21.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:21.380 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:21.404 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:21.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:21.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:21.506 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:21.527 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:21.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:21.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:21.629 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:21.651 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:21.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:21.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:21.752 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:21.774 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:21.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:21.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:21.876 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:21.897 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:21.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:21.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:21.924 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:21 smithi136 ceph-mon[109861]: osdmap e1366: 8 total, 8 up, 8 in 2024-08-17T17:58:21.924 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:58:21.924 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:58:21.925 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:21 smithi136 ceph-mon[115078]: osdmap e1366: 8 total, 8 up, 8 in 2024-08-17T17:58:21.925 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:58:21.925 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:58:21.999 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:21 smithi165 ceph-mon[105517]: osdmap e1366: 8 total, 8 up, 8 in 2024-08-17T17:58:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:58:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:58:22.020 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:22.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:22.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:22.122 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:22.136 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:58:22.143 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:22.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:22.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:22.244 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:22.266 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:22.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:22.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:22.367 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:22.389 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:22.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:22.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:22.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:22 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a[109838]: 2024-08-17T17:58:22.201+0000 7f5b8e77d640 -1 mon.a@0(leader).osd e1366 definitely_dead 0 2024-08-17T17:58:22.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:22.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:22.513 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:22.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:22.565 INFO:teuthology.orchestra.run.smithi136.stderr:marked out osd.0. 2024-08-17T17:58:22.614 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:22.635 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:22.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:22.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:22.737 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:22.758 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:22.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:22.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:22.860 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:22.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:22.883 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:22.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:22.913 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:22 smithi136 ceph-mon[109861]: pgmap v2291: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:22.913 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:22 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3525704507' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-08-17T17:58:22.913 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:22 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-08-17T17:58:22.913 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:22 smithi136 ceph-mon[109861]: Client client.admin marked osd.0 out, while it was still marked up 2024-08-17T17:58:22.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:22 smithi136 ceph-mon[115078]: pgmap v2291: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:22.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:22 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3525704507' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-08-17T17:58:22.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:22 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-08-17T17:58:22.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:22 smithi136 ceph-mon[115078]: Client client.admin marked osd.0 out, while it was still marked up 2024-08-17T17:58:22.984 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:23.006 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:23.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:23.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:22 smithi165 ceph-mon[105517]: pgmap v2291: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:58:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:22 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3525704507' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-08-17T17:58:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:22 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-08-17T17:58:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:22 smithi165 ceph-mon[105517]: Client client.admin marked osd.0 out, while it was still marked up 2024-08-17T17:58:23.107 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:23.128 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:23.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:23.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:23.230 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:23.251 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:23.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:23.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:23.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:23.374 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:23.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:23.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:23.476 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:23.499 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:23.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:23.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:23.601 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:23.623 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:23.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:23.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:23.726 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:23.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:23.748 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:23.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:23.749 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:58:23.849 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:23.861 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:23 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-08-17T17:58:23.861 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:23 smithi136 ceph-mon[109861]: osdmap e1367: 8 total, 8 up, 7 in 2024-08-17T17:58:23.861 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:23 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-08-17T17:58:23.862 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:23 smithi136 ceph-mon[115078]: osdmap e1367: 8 total, 8 up, 7 in 2024-08-17T17:58:23.870 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:23.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:23.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:23.972 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:23.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:23.993 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:23.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:23 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-08-17T17:58:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:23 smithi165 ceph-mon[105517]: osdmap e1367: 8 total, 8 up, 7 in 2024-08-17T17:58:24.095 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:24.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:24.117 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:24.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:24.218 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:24.239 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:24.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:24.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:24.341 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:24.364 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:24.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:24.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:24.466 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:24.487 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:24.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:24.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:24.536 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:58:24.536 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T17:58:24.536 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-17T17:58:24.536 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:58:24.589 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:24.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:24.612 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:24.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:24.712 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:24.733 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:24.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:24.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:24.835 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:24.856 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:24.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:24.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:24.862 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:24 smithi136 ceph-mon[109861]: pgmap v2293: 89 pgs: 89 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail 2024-08-17T17:58:24.862 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:24 smithi136 ceph-mon[109861]: osdmap e1368: 8 total, 8 up, 7 in 2024-08-17T17:58:24.863 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:24 smithi136 ceph-mon[115078]: pgmap v2293: 89 pgs: 89 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail 2024-08-17T17:58:24.863 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:24 smithi136 ceph-mon[115078]: osdmap e1368: 8 total, 8 up, 7 in 2024-08-17T17:58:24.958 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:24.980 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:24.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:24.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:25.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:24 smithi165 ceph-mon[105517]: pgmap v2293: 89 pgs: 89 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail 2024-08-17T17:58:25.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:24 smithi165 ceph-mon[105517]: osdmap e1368: 8 total, 8 up, 7 in 2024-08-17T17:58:25.082 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:25.104 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:25.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:25.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:25.151 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-17T17:58:25.206 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:25.229 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:25.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:25.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:25.331 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:25.352 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:25.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:25.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:25.454 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:25.476 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:25.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:25.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:25.577 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:25.602 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:25.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:25.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:25.620 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:58:25.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:25 smithi136 ceph-mon[115078]: osdmap e1369: 8 total, 8 up, 7 in 2024-08-17T17:58:25.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:25 smithi136 ceph-mon[109861]: osdmap e1369: 8 total, 8 up, 7 in 2024-08-17T17:58:25.705 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:25.726 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:25.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:25.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:25.828 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:25.853 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:25.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:25.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:25.954 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:25.976 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:25.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:25.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:25 smithi165 ceph-mon[105517]: osdmap e1369: 8 total, 8 up, 7 in 2024-08-17T17:58:26.077 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:26.103 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:26.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:26.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:26.205 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:26.227 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:26.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:26.328 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:26.349 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:26.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:26.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:26.451 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:26.472 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:26.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:26.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:26.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:26.596 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T17:58:26.602 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:26.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:26.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:26.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:26.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:26.730 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:26.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:26.831 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:26.853 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:26.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:26.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:26.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:26 smithi136 ceph-mon[115078]: pgmap v2296: 89 pgs: 89 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail 2024-08-17T17:58:26.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:26 smithi136 ceph-mon[115078]: osdmap e1370: 8 total, 8 up, 7 in 2024-08-17T17:58:26.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:26 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3145009400' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:58:26.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:26 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:58:26.886 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:26 smithi136 ceph-mon[109861]: pgmap v2296: 89 pgs: 89 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail 2024-08-17T17:58:26.886 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:26 smithi136 ceph-mon[109861]: osdmap e1370: 8 total, 8 up, 7 in 2024-08-17T17:58:26.886 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:26 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3145009400' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:58:26.886 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:26 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:58:26.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:26.980 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:26.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:26.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:26 smithi165 ceph-mon[105517]: pgmap v2296: 89 pgs: 89 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail 2024-08-17T17:58:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:26 smithi165 ceph-mon[105517]: osdmap e1370: 8 total, 8 up, 7 in 2024-08-17T17:58:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:26 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3145009400' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:58:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:26 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:58:27.082 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:27.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:27.105 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:27.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:27.206 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:27.228 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:27.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:27.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:27.330 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:27.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:27.357 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:27.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:27.457 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:27.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:27.480 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:27.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:27.542 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:58:27.581 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:27.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:27.603 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:27.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:27.704 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:27.725 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:27.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:27.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:27.776 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:27 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:58:27.776 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:27 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:58:27.776 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:27 smithi136 ceph-mon[109861]: osdmap e1371: 8 total, 8 up, 7 in 2024-08-17T17:58:27.776 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:58:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:58:27] "GET /metrics HTTP/1.1" 200 36472 "" "Prometheus/2.51.0" 2024-08-17T17:58:27.776 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:27 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:58:27.776 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:27 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:58:27.777 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:27 smithi136 ceph-mon[115078]: osdmap e1371: 8 total, 8 up, 7 in 2024-08-17T17:58:27.827 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:27.848 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:27.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:27.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:27.949 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:27.975 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:27.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:27.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:28.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:27 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:58:28.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:27 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:58:28.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:27 smithi165 ceph-mon[105517]: osdmap e1371: 8 total, 8 up, 7 in 2024-08-17T17:58:28.077 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:28.099 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:28.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:28.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:28.200 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:28.227 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:28.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:28.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:28.328 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:28.350 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:28.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:28.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:28.451 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:28.473 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:28.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:28.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:28.577 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:28.602 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:28.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:28.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:28.676 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:28 smithi136 ceph-mon[109861]: pgmap v2299: 89 pgs: 89 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-08-17T17:58:28.676 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:28 smithi136 ceph-mon[115078]: pgmap v2299: 89 pgs: 89 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-08-17T17:58:28.704 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:28.725 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:28.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:28.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:28.827 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:28.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:28.849 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:28.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:28.950 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:28.976 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:28.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:28.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:28 smithi165 ceph-mon[105517]: pgmap v2299: 89 pgs: 89 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-08-17T17:58:29.078 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:29.099 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:29.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:29.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:29.201 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:29.227 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:29.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:29.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:29.329 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:29.345 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-17T17:58:29.345 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:58:29.355 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:29.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:29.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:29.456 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:29.482 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:29.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:29.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:29.528 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T17:58:29.529 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:58:29.545 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T17:58:29.545 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:58:29.561 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T17:58:29.561 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:58:29.577 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T17:58:29.577 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:58:29.585 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:29.594 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T17:58:29.594 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:58:29.610 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:29.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:29.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:29.612 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T17:58:29.612 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:58:29.627 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T17:58:29.628 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:58:29.645 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T17:58:29.645 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:58:29.713 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:29.738 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:29.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:29.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:29.833 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:29 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2784416303' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:58:29.833 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:29 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2784416303' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:58:29.840 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:29.865 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:29.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:29.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:29.966 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:29.989 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:58:29.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:29.991 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:29.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:29 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2784416303' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:58:30.092 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:30.113 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:30.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:30.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:30.215 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:30.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:30.238 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:30.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:30.339 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:30.360 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:30.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:30.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:30.461 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:30.487 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:30.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:30.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:30.589 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:30.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:30.615 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:30.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:30.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:30.741 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:30.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:30.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:30.816 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:30 smithi136 ceph-mon[109861]: pgmap v2300: 89 pgs: 89 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 113 KiB/s, 2 objects/s recovering 2024-08-17T17:58:30.817 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:30 smithi136 ceph-mon[115078]: pgmap v2300: 89 pgs: 89 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 113 KiB/s, 2 objects/s recovering 2024-08-17T17:58:30.842 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:30.867 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:30.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:30.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:30.968 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:30.992 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:30.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:30.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:30 smithi165 ceph-mon[105517]: pgmap v2300: 89 pgs: 89 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 113 KiB/s, 2 objects/s recovering 2024-08-17T17:58:31.094 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:31.116 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:31.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:31.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:31.218 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:31.239 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:31.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:31.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:31.341 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:31.366 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:31.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:31.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:31.467 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:31.492 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:31.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:31.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:31.594 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:31.611 INFO:teuthology.orchestra.run.smithi136.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-08-17T17:58:31.612 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T17:58:31.612 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:58:31.617 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:31.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:31.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:31.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:31 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4142061405' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:58:31.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:31 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:58:31.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:31 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4142061405' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:58:31.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:31 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:58:31.719 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:31.743 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:31.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:31.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:31.845 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:31.867 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:31.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:31.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:31.969 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:31.994 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:31.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:31.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:31 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4142061405' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:58:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:31 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:58:32.095 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:32.120 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:32.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:32.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:32.221 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:32.242 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:32.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:32.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:32.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:32.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:32.371 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:32.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:32.397 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-17T17:58:32.472 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:32.493 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:32.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:32.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:32.595 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:32.621 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:32.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:32.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:32.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:32 smithi136 ceph-mon[109861]: pgmap v2301: 89 pgs: 89 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 94 KiB/s, 1 objects/s recovering 2024-08-17T17:58:32.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:32 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:58:32.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:32 smithi136 ceph-mon[109861]: osdmap e1372: 8 total, 8 up, 7 in 2024-08-17T17:58:32.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:32 smithi136 ceph-mon[115078]: pgmap v2301: 89 pgs: 89 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 94 KiB/s, 1 objects/s recovering 2024-08-17T17:58:32.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:32 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:58:32.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:32 smithi136 ceph-mon[115078]: osdmap e1372: 8 total, 8 up, 7 in 2024-08-17T17:58:32.722 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:32.748 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:32.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:32.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:32.850 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:32.878 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:32.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:32.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:32.980 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:33.002 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:33.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:33.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:32 smithi165 ceph-mon[105517]: pgmap v2301: 89 pgs: 89 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 94 KiB/s, 1 objects/s recovering 2024-08-17T17:58:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:32 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:58:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:32 smithi165 ceph-mon[105517]: osdmap e1372: 8 total, 8 up, 7 in 2024-08-17T17:58:33.103 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:33.126 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:33.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:33.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:33.154 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:58:33.193 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:58:33.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:33.250 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:33.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:33.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:33.352 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:33.374 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:33.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:33.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:33.468 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:58:33.476 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:33.500 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:33.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:33.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:33.602 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:33.628 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:33.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:33.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:33.729 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:33.735 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:33 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:58:33.735 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:33 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:58:33.754 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:33.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:33.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:33.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:33.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:33.878 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:33.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:33.979 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:33.989 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:58:34.004 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:34.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:34.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:33 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:58:34.106 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:34.128 DEBUG:teuthology.orchestra.run.smithi136:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-08-17T17:58:34.128 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-08-17T17:58:34.129 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:58:34.130 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:34.131 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-17T17:58:34.131 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-17T17:58:34.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:34.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:34.233 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:34.258 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:34.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:34.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:34.360 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:34.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:34.386 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:34.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:34.487 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:34.510 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:34.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:34.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:34.611 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:34.637 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:34.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:34.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:34.702 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:hints = 1 2024-08-17T17:58:34.702 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-17T17:58:34.703 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Object prefix: benchmark_data_smithi136_273969 2024-08-17T17:58:34.703 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:58:34.703 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 0 0 0 0 0 0 - 0 2024-08-17T17:58:34.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:34.778 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:34.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:34.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:34.880 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:34.914 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:34.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:34.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:34.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:34 smithi136 ceph-mon[109861]: pgmap v2303: 89 pgs: 89 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 81 KiB/s, 1 objects/s recovering 2024-08-17T17:58:34.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:34 smithi136 ceph-mon[115078]: pgmap v2303: 89 pgs: 89 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 81 KiB/s, 1 objects/s recovering 2024-08-17T17:58:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:34 smithi165 ceph-mon[105517]: pgmap v2303: 89 pgs: 89 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 81 KiB/s, 1 objects/s recovering 2024-08-17T17:58:35.016 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:35.051 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:35.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:35.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:35.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:35.190 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:35.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:35.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:35.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:35.328 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:35.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:35.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:35.430 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:35.465 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:35.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:35.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:35.567 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:35.599 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:35.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:35.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:35.701 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:35.702 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 1 16 4139 4123 257.684 257.688 0.00310843 0.00386862 2024-08-17T17:58:35.735 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:35.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:35.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:35.837 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:35.873 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:35.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:35.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:35.975 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:36.011 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:36.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:36.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:36.113 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:36.147 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:36.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:36.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:36.250 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:36.287 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:36.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:36.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:36.389 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:36.427 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:36.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:36.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:36.529 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:36.567 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:36.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:36.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:36.669 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:36.702 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 2 16 7982 7966 248.925 240.188 0.00509676 0.00400965 2024-08-17T17:58:36.703 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:36.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:36.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:36.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:36.843 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:36.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:36.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:36.945 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:36.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:36 smithi136 ceph-mon[115078]: pgmap v2304: 89 pgs: 89 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 71 KiB/s, 1 objects/s recovering 2024-08-17T17:58:36.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:58:36.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:58:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:36 smithi136 ceph-mon[109861]: pgmap v2304: 89 pgs: 89 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 71 KiB/s, 1 objects/s recovering 2024-08-17T17:58:36.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:58:36.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:58:36.982 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:36.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:36.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:36 smithi165 ceph-mon[105517]: pgmap v2304: 89 pgs: 89 active+clean; 582 KiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 71 KiB/s, 1 objects/s recovering 2024-08-17T17:58:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:58:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:58:37.084 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:37.117 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:37.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:37.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:37.219 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:37.253 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:37.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:37.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:37.355 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:37.389 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:37.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:37.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:37.490 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:37.528 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:37.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:37.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:37.630 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:37.640 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:58:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:58:37] "GET /metrics HTTP/1.1" 200 36488 "" "Prometheus/2.51.0" 2024-08-17T17:58:37.668 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:37.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:37.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:37.702 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 3 16 11365 11349 236.423 211.438 0.00516898 0.00422241 2024-08-17T17:58:37.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:37.806 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:37.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:37.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:37.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:37.941 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:37.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:37.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:38.042 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:38.077 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:38.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:38.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:38.179 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:38.213 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:38.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:38.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:38.315 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:38.348 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:38.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:38.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:38.450 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:38.487 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:38.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:38.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:38.589 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:38.625 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:38.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:38.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:38.676 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T17:58:38.700 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:58:38.703 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 4 16 14627 14611 228.281 203.875 0.0123552 0.0043748 2024-08-17T17:58:38.727 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:38.751 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T17:58:38.751 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:58:38.761 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:38.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:38.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:38.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:38.898 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:38.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:38.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:38.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:38 smithi136 ceph-mon[109861]: pgmap v2305: 89 pgs: 89 active+clean; 133 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s wr, 211 op/s; 58 KiB/s, 0 objects/s recovering 2024-08-17T17:58:38.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:38 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2402447737' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:58:38.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:38 smithi136 ceph-mon[115078]: pgmap v2305: 89 pgs: 89 active+clean; 133 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s wr, 211 op/s; 58 KiB/s, 0 objects/s recovering 2024-08-17T17:58:38.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:38 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2402447737' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:58:39.000 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:39.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:38 smithi165 ceph-mon[105517]: pgmap v2305: 89 pgs: 89 active+clean; 133 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s wr, 211 op/s; 58 KiB/s, 0 objects/s recovering 2024-08-17T17:58:39.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:38 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2402447737' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:58:39.034 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:39.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:39.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:39.136 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:39.170 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:39.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:39.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:39.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:39.305 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:39.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:39.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:39.407 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:39.442 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:39.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:39.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:39.543 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:39.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:39.579 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:39.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:39.680 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:39.703 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 5 16 17687 17671 220.872 191.25 0.00266671 0.00452114 2024-08-17T17:58:39.713 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:39.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:39.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:39.815 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:39.850 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:39.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:39.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:39.945 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:39 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:58:39.945 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:39 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2402447737' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:58:39.945 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:39 smithi136 ceph-mon[109861]: osdmap e1373: 8 total, 8 up, 7 in 2024-08-17T17:58:39.945 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:39 smithi136 ceph-mon[109861]: pgmap v2307: 89 pgs: 89 active+clean; 466 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 58 MiB/s wr, 931 op/s 2024-08-17T17:58:39.945 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:39 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:58:39.945 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:39 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2402447737' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:58:39.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:39 smithi136 ceph-mon[115078]: osdmap e1373: 8 total, 8 up, 7 in 2024-08-17T17:58:39.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:39 smithi136 ceph-mon[115078]: pgmap v2307: 89 pgs: 89 active+clean; 466 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 58 MiB/s wr, 931 op/s 2024-08-17T17:58:39.951 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:39.990 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-17T17:58:39.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:39.991 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:39.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:39 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:58:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:39 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2402447737' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:58:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:39 smithi165 ceph-mon[105517]: osdmap e1373: 8 total, 8 up, 7 in 2024-08-17T17:58:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:39 smithi165 ceph-mon[105517]: pgmap v2307: 89 pgs: 89 active+clean; 466 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 58 MiB/s wr, 931 op/s 2024-08-17T17:58:40.092 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:40.124 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:40.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:40.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:40.226 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:40.250 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T17:58:40.251 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:58:40.265 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:40.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:40.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:40.283 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T17:58:40.283 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:58:40.316 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T17:58:40.317 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:58:40.347 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T17:58:40.347 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:58:40.367 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:40.380 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T17:58:40.380 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:58:40.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:40.404 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:40.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:40.414 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T17:58:40.414 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:58:40.452 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T17:58:40.453 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:58:40.485 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T17:58:40.485 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:58:40.505 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:40.540 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:40.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:40.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:40.642 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:40.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:40.677 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:40.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:40.703 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 6 16 20513 20497 213.494 176.625 0.00427703 0.00467803 2024-08-17T17:58:40.778 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:40.811 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:40.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:40.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:40.913 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:40.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:40 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3760909306' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:58:40.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:40 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3760909306' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:58:40.950 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:40.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:40.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:40 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3760909306' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:58:41.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:41.087 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:41.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:41.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:41.189 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:41.224 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:41.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:41.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:41.326 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:41.362 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:41.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:41.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:41.464 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:41.498 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:41.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:41.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:41.600 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:41.634 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:41.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:41.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:41.703 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 7 15 23269 23254 207.608 172.312 0.00899335 0.00481126 2024-08-17T17:58:41.736 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:41.775 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:41.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:41.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:41.877 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:41.911 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:41.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:41.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:41.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:41 smithi136 ceph-mon[109861]: pgmap v2308: 89 pgs: 89 active+clean; 466 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 51 MiB/s wr, 819 op/s 2024-08-17T17:58:41.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:41 smithi136 ceph-mon[115078]: pgmap v2308: 89 pgs: 89 active+clean; 466 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 51 MiB/s wr, 819 op/s 2024-08-17T17:58:42.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:41 smithi165 ceph-mon[105517]: pgmap v2308: 89 pgs: 89 active+clean; 466 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 51 MiB/s wr, 819 op/s 2024-08-17T17:58:42.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:42.049 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:42.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:42.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:42.149 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:58:42.152 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:42.188 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:42.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:42.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:42.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:42.323 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:42.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:42.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:42.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:42.464 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:42.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:42.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:42.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:42.603 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:42.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:42.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:42.703 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 8 16 25880 25864 202.046 163.125 0.0056092 0.00494361 2024-08-17T17:58:42.705 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:42.709 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:58:42.742 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:42.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:42.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:42.844 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:42.879 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:42.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:42.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:42.981 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:43.015 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:43.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:43.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:43.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:43.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:43.158 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:43.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:43.259 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:43.294 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:43.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:43.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:43.396 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:43.433 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:43.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:43.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:43.534 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:43.566 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:43.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:43.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:43.668 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:43.703 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 9 16 28306 28290 196.442 151.625 0.0105678 0.00508486 2024-08-17T17:58:43.706 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:43.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:43.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:43.737 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:58:43.808 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:43.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:43.848 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:43.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:43.949 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:43.984 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:43.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:43.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:44.086 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:44.114 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:44.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:44.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:44.153 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:43 smithi136 ceph-mon[109861]: pgmap v2309: 89 pgs: 89 active+clean; 1.4 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-08-17T17:58:44.154 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:43 smithi136 ceph-mon[115078]: pgmap v2309: 89 pgs: 89 active+clean; 1.4 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-08-17T17:58:44.216 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:44.254 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:44.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:44.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:44.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:43 smithi165 ceph-mon[105517]: pgmap v2309: 89 pgs: 89 active+clean; 1.4 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-08-17T17:58:44.356 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:44.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:44.399 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:44.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:44.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:44.531 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:44.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:44.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:44.633 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:44.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:44.665 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:44.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:44.703 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 10 16 30447 30431 190.178 133.812 0.00195077 0.00522968 2024-08-17T17:58:44.766 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:44.803 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:44.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:44.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:44.904 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:44.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:44.937 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:44.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:45.037 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:45.071 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:45.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:45.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:45.173 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:45.210 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:45.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:45.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:45.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:45.348 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:45.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:45.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:45.450 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:45.487 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:45.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:45.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:45.589 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:45.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:45.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:45.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:45.703 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 11 16 32544 32528 184.803 131.062 0.0123465 0.00540584 2024-08-17T17:58:45.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:45.767 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:45.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:45.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:45.869 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:45.907 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:45.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:45.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:46.008 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:46.039 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:46.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:46.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:46.113 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:45 smithi136 ceph-mon[109861]: pgmap v2310: 89 pgs: 89 active+clean; 1.4 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-08-17T17:58:46.113 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:45 smithi136 ceph-mon[115078]: pgmap v2310: 89 pgs: 89 active+clean; 1.4 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-08-17T17:58:46.141 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:46.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:46.162 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:46.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:46.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:45 smithi165 ceph-mon[105517]: pgmap v2310: 89 pgs: 89 active+clean; 1.4 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-08-17T17:58:46.263 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:46.299 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:46.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:46.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:46.402 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:46.427 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:46.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:46.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:46.528 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:46.552 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:46.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:46.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:46.654 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:46.690 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:46.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:46.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:46.703 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 12 16 34147 34131 177.751 100.188 0.00767265 0.00562039 2024-08-17T17:58:46.792 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:46.826 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:46.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:46.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:46.928 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:46.959 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:46.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:46.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:47.060 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:47.081 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:47.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:47.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:47.182 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:47.212 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:47.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:47.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:47.314 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:47.347 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:47.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:47.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:47.449 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:47.480 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:47.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:47.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:47.582 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:47.618 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:47.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:47.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:47.703 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 13 16 35864 35848 172.332 107.312 0.00759289 0.00579739 2024-08-17T17:58:47.720 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:47.756 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:47.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:47.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:47.767 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:58:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:58:47] "GET /metrics HTTP/1.1" 200 36488 "" "Prometheus/2.51.0" 2024-08-17T17:58:47.857 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:47.891 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:47.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:47.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:47.993 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:48.026 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:48.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:48.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:48.128 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:48.163 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:48.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:48.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:48.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:47 smithi136 ceph-mon[109861]: pgmap v2311: 89 pgs: 89 active+clean; 1.7 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 163 MiB/s wr, 2.60k op/s 2024-08-17T17:58:48.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:47 smithi136 ceph-mon[115078]: pgmap v2311: 89 pgs: 89 active+clean; 1.7 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 163 MiB/s wr, 2.60k op/s 2024-08-17T17:58:48.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:47 smithi165 ceph-mon[105517]: pgmap v2311: 89 pgs: 89 active+clean; 1.7 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 163 MiB/s wr, 2.60k op/s 2024-08-17T17:58:48.265 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:48.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:48.299 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:48.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:48.400 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:48.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:48.423 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:48.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:48.523 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:48.545 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:48.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:48.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:48.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:48.678 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:48.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:48.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:48.704 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 14 15 37401 37386 166.888 96.125 0.00720447 0.005987 2024-08-17T17:58:48.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:48.803 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:48.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:48.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:48.905 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:48.936 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:48.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:48.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:49.038 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:49.073 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:49.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:49.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:49.175 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:49.209 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:49.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:49.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:49.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:49.345 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:49.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:49.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:49.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:49.485 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:49.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:49.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:49.587 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:49.614 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:49.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:49.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:49.704 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 15 16 39285 39269 163.607 117.688 0.0128114 0.00610724 2024-08-17T17:58:49.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:49.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:49.759 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:49.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:49.774 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:58:49.775 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T17:58:49.860 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:49.897 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:49.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:49.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:49.999 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:50.033 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:50.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:50.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:50.135 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:50.174 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:50.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:50.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:50.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:49 smithi136 ceph-mon[109861]: pgmap v2312: 89 pgs: 89 active+clean; 2.1 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 166 MiB/s wr, 2.65k op/s 2024-08-17T17:58:50.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:49 smithi136 ceph-mon[109861]: from='client.35388 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:58:50.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:49 smithi136 ceph-mon[115078]: pgmap v2312: 89 pgs: 89 active+clean; 2.1 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 166 MiB/s wr, 2.65k op/s 2024-08-17T17:58:50.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:49 smithi136 ceph-mon[115078]: from='client.35388 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:58:50.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:49 smithi165 ceph-mon[105517]: pgmap v2312: 89 pgs: 89 active+clean; 2.1 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 166 MiB/s wr, 2.65k op/s 2024-08-17T17:58:50.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:49 smithi165 ceph-mon[105517]: from='client.35388 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:58:50.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:50.311 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:50.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:50.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:50.413 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:50.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:50.451 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:50.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:50.552 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:50.591 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:50.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:50.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:50.693 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:50.704 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 16 16 41173 41157 160.756 118 0.00692303 0.00621514 2024-08-17T17:58:50.726 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:50.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:50.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:50.777 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:58:50.828 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:50.863 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:50.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:50.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:50.964 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:50.997 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:50.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:50.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:51.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:51.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:51.138 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:51.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:51.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:51.282 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:51.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:51.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:51.375 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:58:51.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:51.411 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":2312,"stamp":"2024-08-17T17:58:48.705236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2232358885,"num_objects":34080,"num_object_clones":0,"num_object_copies":102240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34080,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":34333,"num_write_kb":2181905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":2362076,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34262,"ondisk_log_size":34262,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":8516668,"kb_used_data":6426436,"kb_used_omap":219,"kb_used_meta":2089956,"kb_avail":647699396,"statfs":{"total":671965249536,"available":663244181504,"internally_reserved":0,"allocated":6580670464,"data_stored":6550651387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224669,"internal_metadata":2140115555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[13,6,9,9,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1743257600,"num_objects":26600,"num_object_clones":0,"num_object_copies":79800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26600,"num_write_kb":1702400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.026815"},"pg_stats":[{"pgid":"54.f","version":"1373'2077","reported_seq":2106,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:45.719027+0000","last_change":"2024-08-17T17:58:19.551058+0000","last_active":"2024-08-17T17:58:45.719027+0000","last_peered":"2024-08-17T17:58:45.719027+0000","last_clean":"2024-08-17T17:58:45.719027+0000","last_became_active":"2024-08-17T17:58:19.550748+0000","last_became_peered":"2024-08-17T17:58:19.550748+0000","last_unstale":"2024-08-17T17:58:45.719027+0000","last_undegraded":"2024-08-17T17:58:45.719027+0000","last_fullsized":"2024-08-17T17:58:45.719027+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":2077,"log_dups_size":0,"ondisk_log_size":2077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:43:13.383674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136118272,"num_objects":2077,"num_object_clones":0,"num_object_copies":6231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2077,"num_write_kb":132928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.e","version":"1373'2173","reported_seq":2191,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:47.250593+0000","last_change":"2024-08-17T17:58:19.593455+0000","last_active":"2024-08-17T17:58:47.250593+0000","last_peered":"2024-08-17T17:58:47.250593+0000","last_clean":"2024-08-17T17:58:47.250593+0000","last_became_active":"2024-08-17T17:58:19.593144+0000","last_became_peered":"2024-08-17T17:58:19.593144+0000","last_unstale":"2024-08-17T17:58:47.250593+0000","last_undegraded":"2024-08-17T17:58:47.250593+0000","last_fullsized":"2024-08-17T17:58:47.250593+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":2173,"log_dups_size":0,"ondisk_log_size":2173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:08:45.701487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142409728,"num_objects":2173,"num_object_clones":0,"num_object_copies":6519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2173,"num_write_kb":139072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"54.d","version":"1373'2163","reported_seq":2190,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:47.019571+0000","last_change":"2024-08-17T17:58:23.582595+0000","last_active":"2024-08-17T17:58:47.019571+0000","last_peered":"2024-08-17T17:58:47.019571+0000","last_clean":"2024-08-17T17:58:47.019571+0000","last_became_active":"2024-08-17T17:58:23.582234+0000","last_became_peered":"2024-08-17T17:58:23.582234+0000","last_unstale":"2024-08-17T17:58:47.019571+0000","last_undegraded":"2024-08-17T17:58:47.019571+0000","last_fullsized":"2024-08-17T17:58:47.019571+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":2163,"log_dups_size":0,"ondisk_log_size":2163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:10:06.366767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141754368,"num_objects":2163,"num_object_clones":0,"num_object_copies":6489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2163,"num_write_kb":138432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"54.c","version":"1373'2022","reported_seq":2051,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:45.709145+0000","last_change":"2024-08-17T17:58:19.549839+0000","last_active":"2024-08-17T17:58:45.709145+0000","last_peered":"2024-08-17T17:58:45.709145+0000","last_clean":"2024-08-17T17:58:45.709145+0000","last_became_active":"2024-08-17T17:58:19.549588+0000","last_became_peered":"2024-08-17T17:58:19.549588+0000","last_unstale":"2024-08-17T17:58:45.709145+0000","last_undegraded":"2024-08-17T17:58:45.709145+0000","last_fullsized":"2024-08-17T17:58:45.709145+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":2022,"log_dups_size":0,"ondisk_log_size":2022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:48:07.771191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132513792,"num_objects":2022,"num_object_clones":0,"num_object_copies":6066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2022,"num_write_kb":129408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.b","version":"1373'2050","reported_seq":2086,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:45.688936+0000","last_change":"2024-08-17T17:58:23.585551+0000","last_active":"2024-08-17T17:58:45.688936+0000","last_peered":"2024-08-17T17:58:45.688936+0000","last_clean":"2024-08-17T17:58:45.688936+0000","last_became_active":"2024-08-17T17:58:23.581936+0000","last_became_peered":"2024-08-17T17:58:23.581936+0000","last_unstale":"2024-08-17T17:58:45.688936+0000","last_undegraded":"2024-08-17T17:58:45.688936+0000","last_fullsized":"2024-08-17T17:58:45.688936+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":2050,"log_dups_size":0,"ondisk_log_size":2050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:07:29.720931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134348800,"num_objects":2050,"num_object_clones":0,"num_object_copies":6150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2050,"num_write_kb":131200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.a","version":"1373'2202","reported_seq":2219,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:47.009034+0000","last_change":"2024-08-17T17:58:19.551465+0000","last_active":"2024-08-17T17:58:47.009034+0000","last_peered":"2024-08-17T17:58:47.009034+0000","last_clean":"2024-08-17T17:58:47.009034+0000","last_became_active":"2024-08-17T17:58:19.551177+0000","last_became_peered":"2024-08-17T17:58:19.551177+0000","last_unstale":"2024-08-17T17:58:47.009034+0000","last_undegraded":"2024-08-17T17:58:47.009034+0000","last_fullsized":"2024-08-17T17:58:47.009034+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":2202,"log_dups_size":0,"ondisk_log_size":2202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:20:45.172117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144310272,"num_objects":2202,"num_object_clones":0,"num_object_copies":6606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2202,"num_write_kb":140928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"54.9","version":"1373'2170","reported_seq":2199,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:46.340375+0000","last_change":"2024-08-17T17:58:19.547271+0000","last_active":"2024-08-17T17:58:46.340375+0000","last_peered":"2024-08-17T17:58:46.340375+0000","last_clean":"2024-08-17T17:58:46.340375+0000","last_became_active":"2024-08-17T17:58:19.547003+0000","last_became_peered":"2024-08-17T17:58:19.547003+0000","last_unstale":"2024-08-17T17:58:46.340375+0000","last_undegraded":"2024-08-17T17:58:46.340375+0000","last_fullsized":"2024-08-17T17:58:46.340375+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":2170,"log_dups_size":0,"ondisk_log_size":2170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:08.876694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142213120,"num_objects":2170,"num_object_clones":0,"num_object_copies":6510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2170,"num_write_kb":138880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"54.8","version":"1373'2187","reported_seq":2205,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:47.242147+0000","last_change":"2024-08-17T17:58:19.549974+0000","last_active":"2024-08-17T17:58:47.242147+0000","last_peered":"2024-08-17T17:58:47.242147+0000","last_clean":"2024-08-17T17:58:47.242147+0000","last_became_active":"2024-08-17T17:58:19.549673+0000","last_became_peered":"2024-08-17T17:58:19.549673+0000","last_unstale":"2024-08-17T17:58:47.242147+0000","last_undegraded":"2024-08-17T17:58:47.242147+0000","last_fullsized":"2024-08-17T17:58:47.242147+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":2187,"log_dups_size":0,"ondisk_log_size":2187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:34:59.904300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143327232,"num_objects":2187,"num_object_clones":0,"num_object_copies":6561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2187,"num_write_kb":139968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"54.7","version":"1373'1998","reported_seq":2033,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:45.723575+0000","last_change":"2024-08-17T17:58:23.583414+0000","last_active":"2024-08-17T17:58:45.723575+0000","last_peered":"2024-08-17T17:58:45.723575+0000","last_clean":"2024-08-17T17:58:45.723575+0000","last_became_active":"2024-08-17T17:58:23.582261+0000","last_became_peered":"2024-08-17T17:58:23.582261+0000","last_unstale":"2024-08-17T17:58:45.723575+0000","last_undegraded":"2024-08-17T17:58:45.723575+0000","last_fullsized":"2024-08-17T17:58:45.723575+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":1998,"log_dups_size":0,"ondisk_log_size":1998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:32:05.634094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130940928,"num_objects":1998,"num_object_clones":0,"num_object_copies":5994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1998,"num_write_kb":127872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.6","version":"1373'2188","reported_seq":2204,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:46.947035+0000","last_change":"2024-08-17T17:58:19.593645+0000","last_active":"2024-08-17T17:58:46.947035+0000","last_peered":"2024-08-17T17:58:46.947035+0000","last_clean":"2024-08-17T17:58:46.947035+0000","last_became_active":"2024-08-17T17:58:19.593265+0000","last_became_peered":"2024-08-17T17:58:19.593265+0000","last_unstale":"2024-08-17T17:58:46.947035+0000","last_undegraded":"2024-08-17T17:58:46.947035+0000","last_fullsized":"2024-08-17T17:58:46.947035+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":2188,"log_dups_size":0,"ondisk_log_size":2188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:52:46.137037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143392768,"num_objects":2188,"num_object_clones":0,"num_object_copies":6564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2188,"num_write_kb":140032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"54.5","version":"1373'2146","reported_seq":2173,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:46.996523+0000","last_change":"2024-08-17T17:58:23.582940+0000","last_active":"2024-08-17T17:58:46.996523+0000","last_peered":"2024-08-17T17:58:46.996523+0000","last_clean":"2024-08-17T17:58:46.996523+0000","last_became_active":"2024-08-17T17:58:23.582647+0000","last_became_peered":"2024-08-17T17:58:23.582647+0000","last_unstale":"2024-08-17T17:58:46.996523+0000","last_undegraded":"2024-08-17T17:58:46.996523+0000","last_fullsized":"2024-08-17T17:58:46.996523+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":2146,"log_dups_size":0,"ondisk_log_size":2146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:32:22.678243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140640256,"num_objects":2146,"num_object_clones":0,"num_object_copies":6438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2146,"num_write_kb":137344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"54.4","version":"1373'2043","reported_seq":2072,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:45.705938+0000","last_change":"2024-08-17T17:58:19.549870+0000","last_active":"2024-08-17T17:58:45.705938+0000","last_peered":"2024-08-17T17:58:45.705938+0000","last_clean":"2024-08-17T17:58:45.705938+0000","last_became_active":"2024-08-17T17:58:19.549683+0000","last_became_peered":"2024-08-17T17:58:19.549683+0000","last_unstale":"2024-08-17T17:58:45.705938+0000","last_undegraded":"2024-08-17T17:58:45.705938+0000","last_fullsized":"2024-08-17T17:58:45.705938+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":2043,"log_dups_size":0,"ondisk_log_size":2043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:27:17.205900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133890048,"num_objects":2043,"num_object_clones":0,"num_object_copies":6129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2043,"num_write_kb":130752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.3","version":"1373'2150","reported_seq":2186,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:46.330292+0000","last_change":"2024-08-17T17:58:23.585059+0000","last_active":"2024-08-17T17:58:46.330292+0000","last_peered":"2024-08-17T17:58:46.330292+0000","last_clean":"2024-08-17T17:58:46.330292+0000","last_became_active":"2024-08-17T17:58:23.584745+0000","last_became_peered":"2024-08-17T17:58:23.584745+0000","last_unstale":"2024-08-17T17:58:46.330292+0000","last_undegraded":"2024-08-17T17:58:46.330292+0000","last_fullsized":"2024-08-17T17:58:46.330292+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":2150,"log_dups_size":0,"ondisk_log_size":2150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:39:09.866950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140902400,"num_objects":2150,"num_object_clones":0,"num_object_copies":6450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2150,"num_write_kb":137600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"54.2","version":"1373'2174","reported_seq":2191,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:47.180654+0000","last_change":"2024-08-17T17:58:23.582529+0000","last_active":"2024-08-17T17:58:47.180654+0000","last_peered":"2024-08-17T17:58:47.180654+0000","last_clean":"2024-08-17T17:58:47.180654+0000","last_became_active":"2024-08-17T17:58:23.582203+0000","last_became_peered":"2024-08-17T17:58:23.582203+0000","last_unstale":"2024-08-17T17:58:47.180654+0000","last_undegraded":"2024-08-17T17:58:47.180654+0000","last_fullsized":"2024-08-17T17:58:47.180654+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":2174,"log_dups_size":0,"ondisk_log_size":2174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:29:48.813331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142475264,"num_objects":2174,"num_object_clones":0,"num_object_copies":6522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2174,"num_write_kb":139136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"54.1","version":"1373'2186","reported_seq":2204,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:47.162776+0000","last_change":"2024-08-17T17:58:19.592487+0000","last_active":"2024-08-17T17:58:47.162776+0000","last_peered":"2024-08-17T17:58:47.162776+0000","last_clean":"2024-08-17T17:58:47.162776+0000","last_became_active":"2024-08-17T17:58:19.592298+0000","last_became_peered":"2024-08-17T17:58:19.592298+0000","last_unstale":"2024-08-17T17:58:47.162776+0000","last_undegraded":"2024-08-17T17:58:47.162776+0000","last_fullsized":"2024-08-17T17:58:47.162776+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":2186,"log_dups_size":0,"ondisk_log_size":2186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:04:19.878612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143261696,"num_objects":2186,"num_object_clones":0,"num_object_copies":6558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2186,"num_write_kb":139904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"54.0","version":"1373'2125","reported_seq":2143,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:47.022707+0000","last_change":"2024-08-17T17:58:19.548035+0000","last_active":"2024-08-17T17:58:47.022707+0000","last_peered":"2024-08-17T17:58:47.022707+0000","last_clean":"2024-08-17T17:58:47.022707+0000","last_became_active":"2024-08-17T17:58:19.547741+0000","last_became_peered":"2024-08-17T17:58:19.547741+0000","last_unstale":"2024-08-17T17:58:47.022707+0000","last_undegraded":"2024-08-17T17:58:47.022707+0000","last_fullsized":"2024-08-17T17:58:47.022707+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":2125,"log_dups_size":0,"ondisk_log_size":2125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:43:17.339560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139264000,"num_objects":2125,"num_object_clones":0,"num_object_copies":6375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2125,"num_write_kb":136000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":47,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.994506+0000","last_change":"2024-08-17T17:58:23.586005+0000","last_active":"2024-08-17T17:58:32.994506+0000","last_peered":"2024-08-17T17:58:32.994506+0000","last_clean":"2024-08-17T17:58:32.994506+0000","last_became_active":"2024-08-17T17:58:23.580706+0000","last_became_peered":"2024-08-17T17:58:23.580706+0000","last_unstale":"2024-08-17T17:58:32.994506+0000","last_undegraded":"2024-08-17T17:58:32.994506+0000","last_fullsized":"2024-08-17T17:58:32.994506+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:00:46.147301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1345,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.385421+0000","last_change":"2024-08-17T17:23:07.398676+0000","last_active":"2024-08-17T17:58:32.385421+0000","last_peered":"2024-08-17T17:58:32.385421+0000","last_clean":"2024-08-17T17:58:32.385421+0000","last_became_active":"2024-08-17T17:23:07.398369+0000","last_became_peered":"2024-08-17T17:23:07.398369+0000","last_unstale":"2024-08-17T17:58:32.385421+0000","last_undegraded":"2024-08-17T17:58:32.385421+0000","last_fullsized":"2024-08-17T17:58:32.385421+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:55:04.281412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":2036,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.689821+0000","last_change":"2024-08-17T17:24:17.476168+0000","last_active":"2024-08-17T17:58:38.689821+0000","last_peered":"2024-08-17T17:58:38.689821+0000","last_clean":"2024-08-17T17:58:38.689821+0000","last_became_active":"2024-08-17T17:24:17.465845+0000","last_became_peered":"2024-08-17T17:24:17.465845+0000","last_unstale":"2024-08-17T17:58:38.689821+0000","last_undegraded":"2024-08-17T17:58:38.689821+0000","last_fullsized":"2024-08-17T17:58:38.689821+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:15:01.494447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2059,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.690222+0000","last_change":"2024-08-17T17:58:23.585542+0000","last_active":"2024-08-17T17:58:38.690222+0000","last_peered":"2024-08-17T17:58:38.690222+0000","last_clean":"2024-08-17T17:58:38.690222+0000","last_became_active":"2024-08-17T17:58:23.584976+0000","last_became_peered":"2024-08-17T17:58:23.584976+0000","last_unstale":"2024-08-17T17:58:38.690222+0000","last_undegraded":"2024-08-17T17:58:38.690222+0000","last_fullsized":"2024-08-17T17:58:38.690222+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:46:31.440698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":1327,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.994288+0000","last_change":"2024-08-17T17:10:05.965176+0000","last_active":"2024-08-17T17:58:32.994288+0000","last_peered":"2024-08-17T17:58:32.994288+0000","last_clean":"2024-08-17T17:58:32.994288+0000","last_became_active":"2024-08-17T17:10:05.964861+0000","last_became_peered":"2024-08-17T17:10:05.964861+0000","last_unstale":"2024-08-17T17:58:32.994288+0000","last_undegraded":"2024-08-17T17:58:32.994288+0000","last_fullsized":"2024-08-17T17:58:32.994288+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:59:04.277926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1368,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.385423+0000","last_change":"2024-08-17T17:58:23.582593+0000","last_active":"2024-08-17T17:58:32.385423+0000","last_peered":"2024-08-17T17:58:32.385423+0000","last_clean":"2024-08-17T17:58:32.385423+0000","last_became_active":"2024-08-17T17:58:23.582276+0000","last_became_peered":"2024-08-17T17:58:23.582276+0000","last_unstale":"2024-08-17T17:58:32.385423+0000","last_undegraded":"2024-08-17T17:58:32.385423+0000","last_fullsized":"2024-08-17T17:58:32.385423+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:04:27.806072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":1342,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.994244+0000","last_change":"2024-08-17T17:23:07.391038+0000","last_active":"2024-08-17T17:58:32.994244+0000","last_peered":"2024-08-17T17:58:32.994244+0000","last_clean":"2024-08-17T17:58:32.994244+0000","last_became_active":"2024-08-17T17:23:07.390571+0000","last_became_peered":"2024-08-17T17:23:07.390571+0000","last_unstale":"2024-08-17T17:58:32.994244+0000","last_undegraded":"2024-08-17T17:58:32.994244+0000","last_fullsized":"2024-08-17T17:58:32.994244+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:18:41.791740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1359,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.385621+0000","last_change":"2024-08-17T17:24:17.458168+0000","last_active":"2024-08-17T17:58:32.385621+0000","last_peered":"2024-08-17T17:58:32.385621+0000","last_clean":"2024-08-17T17:58:32.385621+0000","last_became_active":"2024-08-17T17:24:17.457847+0000","last_became_peered":"2024-08-17T17:24:17.457847+0000","last_unstale":"2024-08-17T17:58:32.385621+0000","last_undegraded":"2024-08-17T17:58:32.385621+0000","last_fullsized":"2024-08-17T17:58:32.385621+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:28.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":29,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.385581+0000","last_change":"2024-08-17T17:58:23.579378+0000","last_active":"2024-08-17T17:58:32.385581+0000","last_peered":"2024-08-17T17:58:32.385581+0000","last_clean":"2024-08-17T17:58:32.385581+0000","last_became_active":"2024-08-17T17:58:23.579006+0000","last_became_peered":"2024-08-17T17:58:23.579006+0000","last_unstale":"2024-08-17T17:58:32.385581+0000","last_undegraded":"2024-08-17T17:58:32.385581+0000","last_fullsized":"2024-08-17T17:58:32.385581+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:45:42.529376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":2053,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.696099+0000","last_change":"2024-08-17T17:24:17.456622+0000","last_active":"2024-08-17T17:58:38.696099+0000","last_peered":"2024-08-17T17:58:38.696099+0000","last_clean":"2024-08-17T17:58:38.696099+0000","last_became_active":"2024-08-17T17:24:17.455396+0000","last_became_peered":"2024-08-17T17:24:17.455396+0000","last_unstale":"2024-08-17T17:58:38.696099+0000","last_undegraded":"2024-08-17T17:58:38.696099+0000","last_fullsized":"2024-08-17T17:58:38.696099+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:09.564331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1361,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.385549+0000","last_change":"2024-08-17T17:23:42.930464+0000","last_active":"2024-08-17T17:58:32.385549+0000","last_peered":"2024-08-17T17:58:32.385549+0000","last_clean":"2024-08-17T17:58:32.385549+0000","last_became_active":"2024-08-17T17:23:42.929949+0000","last_became_peered":"2024-08-17T17:23:42.929949+0000","last_unstale":"2024-08-17T17:58:32.385549+0000","last_undegraded":"2024-08-17T17:58:32.385549+0000","last_fullsized":"2024-08-17T17:58:32.385549+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:48:27.092339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2038,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.692601+0000","last_change":"2024-08-17T17:24:17.452009+0000","last_active":"2024-08-17T17:58:38.692601+0000","last_peered":"2024-08-17T17:58:38.692601+0000","last_clean":"2024-08-17T17:58:38.692601+0000","last_became_active":"2024-08-17T17:24:17.451492+0000","last_became_peered":"2024-08-17T17:24:17.451492+0000","last_unstale":"2024-08-17T17:58:38.692601+0000","last_undegraded":"2024-08-17T17:58:38.692601+0000","last_fullsized":"2024-08-17T17:58:38.692601+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:21:24.699633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2050,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.686682+0000","last_change":"2024-08-17T17:24:17.452025+0000","last_active":"2024-08-17T17:58:38.686682+0000","last_peered":"2024-08-17T17:58:38.686682+0000","last_clean":"2024-08-17T17:58:38.686682+0000","last_became_active":"2024-08-17T17:24:17.448029+0000","last_became_peered":"2024-08-17T17:24:17.448029+0000","last_unstale":"2024-08-17T17:58:38.686682+0000","last_undegraded":"2024-08-17T17:58:38.686682+0000","last_fullsized":"2024-08-17T17:58:38.686682+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:45.089346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":676,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.690016+0000","last_change":"2024-08-17T17:58:23.586641+0000","last_active":"2024-08-17T17:58:38.690016+0000","last_peered":"2024-08-17T17:58:38.690016+0000","last_clean":"2024-08-17T17:58:38.690016+0000","last_became_active":"2024-08-17T17:58:23.585843+0000","last_became_peered":"2024-08-17T17:58:23.585843+0000","last_unstale":"2024-08-17T17:58:38.690016+0000","last_undegraded":"2024-08-17T17:58:38.690016+0000","last_fullsized":"2024-08-17T17:58:38.690016+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:34:00.228118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2046,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.689684+0000","last_change":"2024-08-17T17:58:23.584638+0000","last_active":"2024-08-17T17:58:38.689684+0000","last_peered":"2024-08-17T17:58:38.689684+0000","last_clean":"2024-08-17T17:58:38.689684+0000","last_became_active":"2024-08-17T17:58:23.583535+0000","last_became_peered":"2024-08-17T17:58:23.583535+0000","last_unstale":"2024-08-17T17:58:38.689684+0000","last_undegraded":"2024-08-17T17:58:38.689684+0000","last_fullsized":"2024-08-17T17:58:38.689684+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:00:25.957667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1345,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:31.610897+0000","last_change":"2024-08-17T17:23:07.402338+0000","last_active":"2024-08-17T17:58:31.610897+0000","last_peered":"2024-08-17T17:58:31.610897+0000","last_clean":"2024-08-17T17:58:31.610897+0000","last_became_active":"2024-08-17T17:23:07.401996+0000","last_became_peered":"2024-08-17T17:23:07.401996+0000","last_unstale":"2024-08-17T17:58:31.610897+0000","last_undegraded":"2024-08-17T17:58:31.610897+0000","last_fullsized":"2024-08-17T17:58:31.610897+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:39:21.566786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2023,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.689882+0000","last_change":"2024-08-17T17:24:17.442338+0000","last_active":"2024-08-17T17:58:38.689882+0000","last_peered":"2024-08-17T17:58:38.689882+0000","last_clean":"2024-08-17T17:58:38.689882+0000","last_became_active":"2024-08-17T17:24:17.441948+0000","last_became_peered":"2024-08-17T17:24:17.441948+0000","last_unstale":"2024-08-17T17:58:38.689882+0000","last_undegraded":"2024-08-17T17:58:38.689882+0000","last_fullsized":"2024-08-17T17:58:38.689882+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:17.891492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":669,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.994529+0000","last_change":"2024-08-17T17:58:23.589701+0000","last_active":"2024-08-17T17:58:32.994529+0000","last_peered":"2024-08-17T17:58:32.994529+0000","last_clean":"2024-08-17T17:58:32.994529+0000","last_became_active":"2024-08-17T17:58:23.589393+0000","last_became_peered":"2024-08-17T17:58:23.589393+0000","last_unstale":"2024-08-17T17:58:32.994529+0000","last_undegraded":"2024-08-17T17:58:32.994529+0000","last_fullsized":"2024-08-17T17:58:32.994529+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:41:03.356477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1982,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.690751+0000","last_change":"2024-08-17T17:24:17.468086+0000","last_active":"2024-08-17T17:58:38.690751+0000","last_peered":"2024-08-17T17:58:38.690751+0000","last_clean":"2024-08-17T17:58:38.690751+0000","last_became_active":"2024-08-17T17:24:17.467522+0000","last_became_peered":"2024-08-17T17:24:17.467522+0000","last_unstale":"2024-08-17T17:58:38.690751+0000","last_undegraded":"2024-08-17T17:58:38.690751+0000","last_fullsized":"2024-08-17T17:58:38.690751+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:52:55.249649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1341,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.994256+0000","last_change":"2024-08-17T17:23:07.392447+0000","last_active":"2024-08-17T17:58:32.994256+0000","last_peered":"2024-08-17T17:58:32.994256+0000","last_clean":"2024-08-17T17:58:32.994256+0000","last_became_active":"2024-08-17T17:23:07.391765+0000","last_became_peered":"2024-08-17T17:23:07.391765+0000","last_unstale":"2024-08-17T17:58:32.994256+0000","last_undegraded":"2024-08-17T17:58:32.994256+0000","last_fullsized":"2024-08-17T17:58:32.994256+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:59:18.602346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":134,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.692703+0000","last_change":"2024-08-17T17:53:58.839889+0000","last_active":"2024-08-17T17:58:38.692703+0000","last_peered":"2024-08-17T17:58:38.692703+0000","last_clean":"2024-08-17T17:58:38.692703+0000","last_became_active":"2024-08-17T17:53:58.839537+0000","last_became_peered":"2024-08-17T17:53:58.839537+0000","last_unstale":"2024-08-17T17:58:38.692703+0000","last_undegraded":"2024-08-17T17:58:38.692703+0000","last_fullsized":"2024-08-17T17:58:38.692703+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:47.169954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2050,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.687702+0000","last_change":"2024-08-17T17:23:42.936147+0000","last_active":"2024-08-17T17:58:38.687702+0000","last_peered":"2024-08-17T17:58:38.687702+0000","last_clean":"2024-08-17T17:58:38.687702+0000","last_became_active":"2024-08-17T17:23:42.935859+0000","last_became_peered":"2024-08-17T17:23:42.935859+0000","last_unstale":"2024-08-17T17:58:38.687702+0000","last_undegraded":"2024-08-17T17:58:38.687702+0000","last_fullsized":"2024-08-17T17:58:38.687702+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:47:36.838706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1354,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.385719+0000","last_change":"2024-08-17T17:10:05.969131+0000","last_active":"2024-08-17T17:58:32.385719+0000","last_peered":"2024-08-17T17:58:32.385719+0000","last_clean":"2024-08-17T17:58:32.385719+0000","last_became_active":"2024-08-17T17:10:05.968856+0000","last_became_peered":"2024-08-17T17:10:05.968856+0000","last_unstale":"2024-08-17T17:58:32.385719+0000","last_undegraded":"2024-08-17T17:58:32.385719+0000","last_fullsized":"2024-08-17T17:58:32.385719+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:25:49.550065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1345,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.385670+0000","last_change":"2024-08-17T17:23:07.455919+0000","last_active":"2024-08-17T17:58:32.385670+0000","last_peered":"2024-08-17T17:58:32.385670+0000","last_clean":"2024-08-17T17:58:32.385670+0000","last_became_active":"2024-08-17T17:23:07.455284+0000","last_became_peered":"2024-08-17T17:23:07.455284+0000","last_unstale":"2024-08-17T17:58:32.385670+0000","last_undegraded":"2024-08-17T17:58:32.385670+0000","last_fullsized":"2024-08-17T17:58:32.385670+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:37:46.260775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1333,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.385668+0000","last_change":"2024-08-17T17:10:06.979492+0000","last_active":"2024-08-17T17:58:32.385668+0000","last_peered":"2024-08-17T17:58:32.385668+0000","last_clean":"2024-08-17T17:58:32.385668+0000","last_became_active":"2024-08-17T17:10:06.979158+0000","last_became_peered":"2024-08-17T17:10:06.979158+0000","last_unstale":"2024-08-17T17:58:32.385668+0000","last_undegraded":"2024-08-17T17:58:32.385668+0000","last_fullsized":"2024-08-17T17:58:32.385668+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:12:42.261365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1358,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:37.060889+0000","last_change":"2024-08-17T17:23:42.935099+0000","last_active":"2024-08-17T17:58:37.060889+0000","last_peered":"2024-08-17T17:58:37.060889+0000","last_clean":"2024-08-17T17:58:37.060889+0000","last_became_active":"2024-08-17T17:23:42.934501+0000","last_became_peered":"2024-08-17T17:23:42.934501+0000","last_unstale":"2024-08-17T17:58:37.060889+0000","last_undegraded":"2024-08-17T17:58:37.060889+0000","last_fullsized":"2024-08-17T17:58:37.060889+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:15:19.095713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":51,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.687388+0000","last_change":"2024-08-17T17:58:23.586651+0000","last_active":"2024-08-17T17:58:38.687388+0000","last_peered":"2024-08-17T17:58:38.687388+0000","last_clean":"2024-08-17T17:58:38.687388+0000","last_became_active":"2024-08-17T17:58:23.580712+0000","last_became_peered":"2024-08-17T17:58:23.580712+0000","last_unstale":"2024-08-17T17:58:38.687388+0000","last_undegraded":"2024-08-17T17:58:38.687388+0000","last_fullsized":"2024-08-17T17:58:38.687388+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:40:33.674413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2026,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.689791+0000","last_change":"2024-08-17T17:58:23.584201+0000","last_active":"2024-08-17T17:58:38.689791+0000","last_peered":"2024-08-17T17:58:38.689791+0000","last_clean":"2024-08-17T17:58:38.689791+0000","last_became_active":"2024-08-17T17:58:23.583863+0000","last_became_peered":"2024-08-17T17:58:23.583863+0000","last_unstale":"2024-08-17T17:58:38.689791+0000","last_undegraded":"2024-08-17T17:58:38.689791+0000","last_fullsized":"2024-08-17T17:58:38.689791+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:36:16.400418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2032,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.687520+0000","last_change":"2024-08-17T17:23:07.428296+0000","last_active":"2024-08-17T17:58:38.687520+0000","last_peered":"2024-08-17T17:58:38.687520+0000","last_clean":"2024-08-17T17:58:38.687520+0000","last_became_active":"2024-08-17T17:23:07.428010+0000","last_became_peered":"2024-08-17T17:23:07.428010+0000","last_unstale":"2024-08-17T17:58:38.687520+0000","last_undegraded":"2024-08-17T17:58:38.687520+0000","last_fullsized":"2024-08-17T17:58:38.687520+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:45.459212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":2030,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.691375+0000","last_change":"2024-08-17T17:58:23.594285+0000","last_active":"2024-08-17T17:58:38.691375+0000","last_peered":"2024-08-17T17:58:38.691375+0000","last_clean":"2024-08-17T17:58:38.691375+0000","last_became_active":"2024-08-17T17:58:23.588147+0000","last_became_peered":"2024-08-17T17:58:23.588147+0000","last_unstale":"2024-08-17T17:58:38.691375+0000","last_undegraded":"2024-08-17T17:58:38.691375+0000","last_fullsized":"2024-08-17T17:58:38.691375+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:12:40.216512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":604,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.994084+0000","last_change":"2024-08-17T17:58:23.579834+0000","last_active":"2024-08-17T17:58:32.994084+0000","last_peered":"2024-08-17T17:58:32.994084+0000","last_clean":"2024-08-17T17:58:32.994084+0000","last_became_active":"2024-08-17T17:58:23.579507+0000","last_became_peered":"2024-08-17T17:58:23.579507+0000","last_unstale":"2024-08-17T17:58:32.994084+0000","last_undegraded":"2024-08-17T17:58:32.994084+0000","last_fullsized":"2024-08-17T17:58:32.994084+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:43:24.177069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1362,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.329769+0000","last_change":"2024-08-17T17:58:23.582011+0000","last_active":"2024-08-17T17:58:32.329769+0000","last_peered":"2024-08-17T17:58:32.329769+0000","last_clean":"2024-08-17T17:58:32.329769+0000","last_became_active":"2024-08-17T17:58:23.581649+0000","last_became_peered":"2024-08-17T17:58:23.581649+0000","last_unstale":"2024-08-17T17:58:32.329769+0000","last_undegraded":"2024-08-17T17:58:32.329769+0000","last_fullsized":"2024-08-17T17:58:32.329769+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:38:55.478552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2054,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.687131+0000","last_change":"2024-08-17T17:23:42.935825+0000","last_active":"2024-08-17T17:58:38.687131+0000","last_peered":"2024-08-17T17:58:38.687131+0000","last_clean":"2024-08-17T17:58:38.687131+0000","last_became_active":"2024-08-17T17:23:42.935150+0000","last_became_peered":"2024-08-17T17:23:42.935150+0000","last_unstale":"2024-08-17T17:58:38.687131+0000","last_undegraded":"2024-08-17T17:58:38.687131+0000","last_fullsized":"2024-08-17T17:58:38.687131+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:57:19.577644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2042,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.686520+0000","last_change":"2024-08-17T17:23:07.431586+0000","last_active":"2024-08-17T17:58:38.686520+0000","last_peered":"2024-08-17T17:58:38.686520+0000","last_clean":"2024-08-17T17:58:38.686520+0000","last_became_active":"2024-08-17T17:23:07.431301+0000","last_became_peered":"2024-08-17T17:23:07.431301+0000","last_unstale":"2024-08-17T17:58:38.686520+0000","last_undegraded":"2024-08-17T17:58:38.686520+0000","last_fullsized":"2024-08-17T17:58:38.686520+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:28:18.706489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2077,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.686865+0000","last_change":"2024-08-17T17:58:23.581351+0000","last_active":"2024-08-17T17:58:38.686865+0000","last_peered":"2024-08-17T17:58:38.686865+0000","last_clean":"2024-08-17T17:58:38.686865+0000","last_became_active":"2024-08-17T17:58:23.580717+0000","last_became_peered":"2024-08-17T17:58:23.580717+0000","last_unstale":"2024-08-17T17:58:38.686865+0000","last_undegraded":"2024-08-17T17:58:38.686865+0000","last_fullsized":"2024-08-17T17:58:38.686865+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:02:47.291227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1359,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.994415+0000","last_change":"2024-08-17T17:24:17.426267+0000","last_active":"2024-08-17T17:58:32.994415+0000","last_peered":"2024-08-17T17:58:32.994415+0000","last_clean":"2024-08-17T17:58:32.994415+0000","last_became_active":"2024-08-17T17:24:17.425925+0000","last_became_peered":"2024-08-17T17:24:17.425925+0000","last_unstale":"2024-08-17T17:58:32.994415+0000","last_undegraded":"2024-08-17T17:58:32.994415+0000","last_fullsized":"2024-08-17T17:58:32.994415+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:24:25.017856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1355,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.691977+0000","last_change":"2024-08-17T17:24:17.474138+0000","last_active":"2024-08-17T17:58:38.691977+0000","last_peered":"2024-08-17T17:58:38.691977+0000","last_clean":"2024-08-17T17:58:38.691977+0000","last_became_active":"2024-08-17T17:24:17.473708+0000","last_became_peered":"2024-08-17T17:24:17.473708+0000","last_unstale":"2024-08-17T17:58:38.691977+0000","last_undegraded":"2024-08-17T17:58:38.691977+0000","last_fullsized":"2024-08-17T17:58:38.691977+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:10:57.768067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1358,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.385823+0000","last_change":"2024-08-17T17:58:23.575900+0000","last_active":"2024-08-17T17:58:32.385823+0000","last_peered":"2024-08-17T17:58:32.385823+0000","last_clean":"2024-08-17T17:58:32.385823+0000","last_became_active":"2024-08-17T17:58:23.575606+0000","last_became_peered":"2024-08-17T17:58:23.575606+0000","last_unstale":"2024-08-17T17:58:32.385823+0000","last_undegraded":"2024-08-17T17:58:32.385823+0000","last_fullsized":"2024-08-17T17:58:32.385823+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:17:46.057413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":1359,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.692292+0000","last_change":"2024-08-17T17:23:07.391021+0000","last_active":"2024-08-17T17:58:38.692292+0000","last_peered":"2024-08-17T17:58:38.692292+0000","last_clean":"2024-08-17T17:58:38.692292+0000","last_became_active":"2024-08-17T17:23:07.390551+0000","last_became_peered":"2024-08-17T17:23:07.390551+0000","last_unstale":"2024-08-17T17:58:38.692292+0000","last_undegraded":"2024-08-17T17:58:38.692292+0000","last_fullsized":"2024-08-17T17:58:38.692292+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:06.239013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001023532,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2060,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.687725+0000","last_change":"2024-08-17T17:58:23.580871+0000","last_active":"2024-08-17T17:58:38.687725+0000","last_peered":"2024-08-17T17:58:38.687725+0000","last_clean":"2024-08-17T17:58:38.687725+0000","last_became_active":"2024-08-17T17:58:23.580542+0000","last_became_peered":"2024-08-17T17:58:23.580542+0000","last_unstale":"2024-08-17T17:58:38.687725+0000","last_undegraded":"2024-08-17T17:58:38.687725+0000","last_fullsized":"2024-08-17T17:58:38.687725+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:51:14.019944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2033,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.686017+0000","last_change":"2024-08-17T17:23:42.910246+0000","last_active":"2024-08-17T17:58:38.686017+0000","last_peered":"2024-08-17T17:58:38.686017+0000","last_clean":"2024-08-17T17:58:38.686017+0000","last_became_active":"2024-08-17T17:23:42.909929+0000","last_became_peered":"2024-08-17T17:23:42.909929+0000","last_unstale":"2024-08-17T17:58:38.686017+0000","last_undegraded":"2024-08-17T17:58:38.686017+0000","last_fullsized":"2024-08-17T17:58:38.686017+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:14:45.210558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2068,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.697271+0000","last_change":"2024-08-17T17:58:23.586426+0000","last_active":"2024-08-17T17:58:38.697271+0000","last_peered":"2024-08-17T17:58:38.697271+0000","last_clean":"2024-08-17T17:58:38.697271+0000","last_became_active":"2024-08-17T17:58:23.585776+0000","last_became_peered":"2024-08-17T17:58:23.585776+0000","last_unstale":"2024-08-17T17:58:38.697271+0000","last_undegraded":"2024-08-17T17:58:38.697271+0000","last_fullsized":"2024-08-17T17:58:38.697271+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:39:33.064885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":663,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.687372+0000","last_change":"2024-08-17T17:58:23.582327+0000","last_active":"2024-08-17T17:58:38.687372+0000","last_peered":"2024-08-17T17:58:38.687372+0000","last_clean":"2024-08-17T17:58:38.687372+0000","last_became_active":"2024-08-17T17:58:23.581984+0000","last_became_peered":"2024-08-17T17:58:23.581984+0000","last_unstale":"2024-08-17T17:58:38.687372+0000","last_undegraded":"2024-08-17T17:58:38.687372+0000","last_fullsized":"2024-08-17T17:58:38.687372+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:10:23.678071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":670,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.385324+0000","last_change":"2024-08-17T17:58:23.583104+0000","last_active":"2024-08-17T17:58:32.385324+0000","last_peered":"2024-08-17T17:58:32.385324+0000","last_clean":"2024-08-17T17:58:32.385324+0000","last_became_active":"2024-08-17T17:58:23.582784+0000","last_became_peered":"2024-08-17T17:58:23.582784+0000","last_unstale":"2024-08-17T17:58:32.385324+0000","last_undegraded":"2024-08-17T17:58:32.385324+0000","last_fullsized":"2024-08-17T17:58:32.385324+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:39:37.921577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":774,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.686273+0000","last_change":"2024-08-17T17:58:23.585368+0000","last_active":"2024-08-17T17:58:38.686273+0000","last_peered":"2024-08-17T17:58:38.686273+0000","last_clean":"2024-08-17T17:58:38.686273+0000","last_became_active":"2024-08-17T17:58:23.584864+0000","last_became_peered":"2024-08-17T17:58:23.584864+0000","last_unstale":"2024-08-17T17:58:38.686273+0000","last_undegraded":"2024-08-17T17:58:38.686273+0000","last_fullsized":"2024-08-17T17:58:38.686273+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:51.734726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1330,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:31.610797+0000","last_change":"2024-08-17T17:10:06.995023+0000","last_active":"2024-08-17T17:58:31.610797+0000","last_peered":"2024-08-17T17:58:31.610797+0000","last_clean":"2024-08-17T17:58:31.610797+0000","last_became_active":"2024-08-17T17:10:06.994757+0000","last_became_peered":"2024-08-17T17:10:06.994757+0000","last_unstale":"2024-08-17T17:58:31.610797+0000","last_undegraded":"2024-08-17T17:58:31.610797+0000","last_fullsized":"2024-08-17T17:58:31.610797+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:31:52.163518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1985,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.691185+0000","last_change":"2024-08-17T17:24:17.468443+0000","last_active":"2024-08-17T17:58:38.691185+0000","last_peered":"2024-08-17T17:58:38.691185+0000","last_clean":"2024-08-17T17:58:38.691185+0000","last_became_active":"2024-08-17T17:24:17.467394+0000","last_became_peered":"2024-08-17T17:24:17.467394+0000","last_unstale":"2024-08-17T17:58:38.691185+0000","last_undegraded":"2024-08-17T17:58:38.691185+0000","last_fullsized":"2024-08-17T17:58:38.691185+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:13.024253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":1357,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.994159+0000","last_change":"2024-08-17T17:23:42.920552+0000","last_active":"2024-08-17T17:58:32.994159+0000","last_peered":"2024-08-17T17:58:32.994159+0000","last_clean":"2024-08-17T17:58:32.994159+0000","last_became_active":"2024-08-17T17:23:42.915824+0000","last_became_peered":"2024-08-17T17:23:42.915824+0000","last_unstale":"2024-08-17T17:58:32.994159+0000","last_undegraded":"2024-08-17T17:58:32.994159+0000","last_fullsized":"2024-08-17T17:58:32.994159+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:25:18.769504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041737399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":645,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:31.610478+0000","last_change":"2024-08-17T17:58:26.598586+0000","last_active":"2024-08-17T17:58:31.610478+0000","last_peered":"2024-08-17T17:58:31.610478+0000","last_clean":"2024-08-17T17:58:31.610478+0000","last_became_active":"2024-08-17T17:58:26.598163+0000","last_became_peered":"2024-08-17T17:58:26.598163+0000","last_unstale":"2024-08-17T17:58:31.610478+0000","last_undegraded":"2024-08-17T17:58:31.610478+0000","last_fullsized":"2024-08-17T17:58:31.610478+0000","mapping_epoch":1370,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T17:06:21.174471+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:36:48.182691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2031,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.687630+0000","last_change":"2024-08-17T17:24:17.455710+0000","last_active":"2024-08-17T17:58:38.687630+0000","last_peered":"2024-08-17T17:58:38.687630+0000","last_clean":"2024-08-17T17:58:38.687630+0000","last_became_active":"2024-08-17T17:24:17.452989+0000","last_became_peered":"2024-08-17T17:24:17.452989+0000","last_unstale":"2024-08-17T17:58:38.687630+0000","last_undegraded":"2024-08-17T17:58:38.687630+0000","last_fullsized":"2024-08-17T17:58:38.687630+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:58:07.894968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2025,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.689584+0000","last_change":"2024-08-17T17:24:17.453639+0000","last_active":"2024-08-17T17:58:38.689584+0000","last_peered":"2024-08-17T17:58:38.689584+0000","last_clean":"2024-08-17T17:58:38.689584+0000","last_became_active":"2024-08-17T17:24:17.443638+0000","last_became_peered":"2024-08-17T17:24:17.443638+0000","last_unstale":"2024-08-17T17:58:38.689584+0000","last_undegraded":"2024-08-17T17:58:38.689584+0000","last_fullsized":"2024-08-17T17:58:38.689584+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:42.107479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1346,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.386000+0000","last_change":"2024-08-17T17:23:07.451866+0000","last_active":"2024-08-17T17:58:32.386000+0000","last_peered":"2024-08-17T17:58:32.386000+0000","last_clean":"2024-08-17T17:58:32.386000+0000","last_became_active":"2024-08-17T17:23:07.451553+0000","last_became_peered":"2024-08-17T17:23:07.451553+0000","last_unstale":"2024-08-17T17:58:32.386000+0000","last_undegraded":"2024-08-17T17:58:32.386000+0000","last_fullsized":"2024-08-17T17:58:32.386000+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:37:23.930651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1348,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.385702+0000","last_change":"2024-08-17T17:23:42.915851+0000","last_active":"2024-08-17T17:58:32.385702+0000","last_peered":"2024-08-17T17:58:32.385702+0000","last_clean":"2024-08-17T17:58:32.385702+0000","last_became_active":"2024-08-17T17:23:42.915552+0000","last_became_peered":"2024-08-17T17:23:42.915552+0000","last_unstale":"2024-08-17T17:58:32.385702+0000","last_undegraded":"2024-08-17T17:58:32.385702+0000","last_fullsized":"2024-08-17T17:58:32.385702+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:45.197876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1361,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.385776+0000","last_change":"2024-08-17T17:23:42.930803+0000","last_active":"2024-08-17T17:58:32.385776+0000","last_peered":"2024-08-17T17:58:32.385776+0000","last_clean":"2024-08-17T17:58:32.385776+0000","last_became_active":"2024-08-17T17:23:42.930458+0000","last_became_peered":"2024-08-17T17:23:42.930458+0000","last_unstale":"2024-08-17T17:58:32.385776+0000","last_undegraded":"2024-08-17T17:58:32.385776+0000","last_fullsized":"2024-08-17T17:58:32.385776+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:40.816706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1345,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:31.610637+0000","last_change":"2024-08-17T17:23:07.401624+0000","last_active":"2024-08-17T17:58:31.610637+0000","last_peered":"2024-08-17T17:58:31.610637+0000","last_clean":"2024-08-17T17:58:31.610637+0000","last_became_active":"2024-08-17T17:23:07.400222+0000","last_became_peered":"2024-08-17T17:23:07.400222+0000","last_unstale":"2024-08-17T17:58:31.610637+0000","last_undegraded":"2024-08-17T17:58:31.610637+0000","last_fullsized":"2024-08-17T17:58:31.610637+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:07.069989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1346,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.385670+0000","last_change":"2024-08-17T17:24:17.470809+0000","last_active":"2024-08-17T17:58:32.385670+0000","last_peered":"2024-08-17T17:58:32.385670+0000","last_clean":"2024-08-17T17:58:32.385670+0000","last_became_active":"2024-08-17T17:24:17.469753+0000","last_became_peered":"2024-08-17T17:24:17.469753+0000","last_unstale":"2024-08-17T17:58:32.385670+0000","last_undegraded":"2024-08-17T17:58:32.385670+0000","last_fullsized":"2024-08-17T17:58:32.385670+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:00.049425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":2030,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.690961+0000","last_change":"2024-08-17T17:58:23.592069+0000","last_active":"2024-08-17T17:58:38.690961+0000","last_peered":"2024-08-17T17:58:38.690961+0000","last_clean":"2024-08-17T17:58:38.690961+0000","last_became_active":"2024-08-17T17:58:23.584024+0000","last_became_peered":"2024-08-17T17:58:23.584024+0000","last_unstale":"2024-08-17T17:58:38.690961+0000","last_undegraded":"2024-08-17T17:58:38.690961+0000","last_fullsized":"2024-08-17T17:58:38.690961+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:37:17.459578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":666,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.695836+0000","last_change":"2024-08-17T17:58:23.581996+0000","last_active":"2024-08-17T17:58:38.695836+0000","last_peered":"2024-08-17T17:58:38.695836+0000","last_clean":"2024-08-17T17:58:38.695836+0000","last_became_active":"2024-08-17T17:58:23.580469+0000","last_became_peered":"2024-08-17T17:58:23.580469+0000","last_unstale":"2024-08-17T17:58:38.695836+0000","last_undegraded":"2024-08-17T17:58:38.695836+0000","last_fullsized":"2024-08-17T17:58:38.695836+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:01:28.612769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1342,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.994327+0000","last_change":"2024-08-17T17:24:17.473913+0000","last_active":"2024-08-17T17:58:32.994327+0000","last_peered":"2024-08-17T17:58:32.994327+0000","last_clean":"2024-08-17T17:58:32.994327+0000","last_became_active":"2024-08-17T17:24:17.473195+0000","last_became_peered":"2024-08-17T17:24:17.473195+0000","last_unstale":"2024-08-17T17:58:32.994327+0000","last_undegraded":"2024-08-17T17:58:32.994327+0000","last_fullsized":"2024-08-17T17:58:32.994327+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:12.998033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1344,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:31.610300+0000","last_change":"2024-08-17T17:23:07.402010+0000","last_active":"2024-08-17T17:58:31.610300+0000","last_peered":"2024-08-17T17:58:31.610300+0000","last_clean":"2024-08-17T17:58:31.610300+0000","last_became_active":"2024-08-17T17:23:07.400849+0000","last_became_peered":"2024-08-17T17:23:07.400849+0000","last_unstale":"2024-08-17T17:58:31.610300+0000","last_undegraded":"2024-08-17T17:58:31.610300+0000","last_fullsized":"2024-08-17T17:58:31.610300+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:16:02.382257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":80,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.994272+0000","last_change":"2024-08-17T17:53:58.836811+0000","last_active":"2024-08-17T17:58:32.994272+0000","last_peered":"2024-08-17T17:58:32.994272+0000","last_clean":"2024-08-17T17:58:32.994272+0000","last_became_active":"2024-08-17T17:53:58.836139+0000","last_became_peered":"2024-08-17T17:53:58.836139+0000","last_unstale":"2024-08-17T17:58:32.994272+0000","last_undegraded":"2024-08-17T17:58:32.994272+0000","last_fullsized":"2024-08-17T17:58:32.994272+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:09:18.260164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1362,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.385326+0000","last_change":"2024-08-17T17:24:17.460203+0000","last_active":"2024-08-17T17:58:32.385326+0000","last_peered":"2024-08-17T17:58:32.385326+0000","last_clean":"2024-08-17T17:58:32.385326+0000","last_became_active":"2024-08-17T17:24:17.459682+0000","last_became_peered":"2024-08-17T17:24:17.459682+0000","last_unstale":"2024-08-17T17:58:32.385326+0000","last_undegraded":"2024-08-17T17:58:32.385326+0000","last_fullsized":"2024-08-17T17:58:32.385326+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:58.703700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":1341,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.330006+0000","last_change":"2024-08-17T17:23:07.454042+0000","last_active":"2024-08-17T17:58:32.330006+0000","last_peered":"2024-08-17T17:58:32.330006+0000","last_clean":"2024-08-17T17:58:32.330006+0000","last_became_active":"2024-08-17T17:23:07.453712+0000","last_became_peered":"2024-08-17T17:23:07.453712+0000","last_unstale":"2024-08-17T17:58:32.330006+0000","last_undegraded":"2024-08-17T17:58:32.330006+0000","last_fullsized":"2024-08-17T17:58:32.330006+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:03:07.643456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1327,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.993765+0000","last_change":"2024-08-17T17:10:06.994799+0000","last_active":"2024-08-17T17:58:32.993765+0000","last_peered":"2024-08-17T17:58:32.993765+0000","last_clean":"2024-08-17T17:58:32.993765+0000","last_became_active":"2024-08-17T17:10:06.994150+0000","last_became_peered":"2024-08-17T17:10:06.994150+0000","last_unstale":"2024-08-17T17:58:32.993765+0000","last_undegraded":"2024-08-17T17:58:32.993765+0000","last_fullsized":"2024-08-17T17:58:32.993765+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:50:47.293091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":1352,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.329937+0000","last_change":"2024-08-17T17:58:23.589073+0000","last_active":"2024-08-17T17:58:32.329937+0000","last_peered":"2024-08-17T17:58:32.329937+0000","last_clean":"2024-08-17T17:58:32.329937+0000","last_became_active":"2024-08-17T17:58:23.583387+0000","last_became_peered":"2024-08-17T17:58:23.583387+0000","last_unstale":"2024-08-17T17:58:32.329937+0000","last_undegraded":"2024-08-17T17:58:32.329937+0000","last_fullsized":"2024-08-17T17:58:32.329937+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:23:27.036125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1370,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:31.610444+0000","last_change":"2024-08-17T17:58:23.592768+0000","last_active":"2024-08-17T17:58:31.610444+0000","last_peered":"2024-08-17T17:58:31.610444+0000","last_clean":"2024-08-17T17:58:31.610444+0000","last_became_active":"2024-08-17T17:58:23.582210+0000","last_became_peered":"2024-08-17T17:58:23.582210+0000","last_unstale":"2024-08-17T17:58:31.610444+0000","last_undegraded":"2024-08-17T17:58:31.610444+0000","last_fullsized":"2024-08-17T17:58:31.610444+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:31.713201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2023,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.691293+0000","last_change":"2024-08-17T17:24:17.451528+0000","last_active":"2024-08-17T17:58:38.691293+0000","last_peered":"2024-08-17T17:58:38.691293+0000","last_clean":"2024-08-17T17:58:38.691293+0000","last_became_active":"2024-08-17T17:24:17.451246+0000","last_became_peered":"2024-08-17T17:24:17.451246+0000","last_unstale":"2024-08-17T17:58:38.691293+0000","last_undegraded":"2024-08-17T17:58:38.691293+0000","last_fullsized":"2024-08-17T17:58:38.691293+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:38:18.176559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":68,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.682121+0000","last_change":"2024-08-17T17:53:58.838279+0000","last_active":"2024-08-17T17:58:38.682121+0000","last_peered":"2024-08-17T17:58:38.682121+0000","last_clean":"2024-08-17T17:58:38.682121+0000","last_became_active":"2024-08-17T17:53:58.837978+0000","last_became_peered":"2024-08-17T17:53:58.837978+0000","last_unstale":"2024-08-17T17:58:38.682121+0000","last_undegraded":"2024-08-17T17:58:38.682121+0000","last_fullsized":"2024-08-17T17:58:38.682121+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:06.019142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":675,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.689933+0000","last_change":"2024-08-17T17:58:23.586407+0000","last_active":"2024-08-17T17:58:38.689933+0000","last_peered":"2024-08-17T17:58:38.689933+0000","last_clean":"2024-08-17T17:58:38.689933+0000","last_became_active":"2024-08-17T17:58:23.584967+0000","last_became_peered":"2024-08-17T17:58:23.584967+0000","last_unstale":"2024-08-17T17:58:38.689933+0000","last_undegraded":"2024-08-17T17:58:38.689933+0000","last_fullsized":"2024-08-17T17:58:38.689933+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:58:08.040251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":675,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.692474+0000","last_change":"2024-08-17T17:58:23.584794+0000","last_active":"2024-08-17T17:58:38.692474+0000","last_peered":"2024-08-17T17:58:38.692474+0000","last_clean":"2024-08-17T17:58:38.692474+0000","last_became_active":"2024-08-17T17:58:23.584485+0000","last_became_peered":"2024-08-17T17:58:23.584485+0000","last_unstale":"2024-08-17T17:58:38.692474+0000","last_undegraded":"2024-08-17T17:58:38.692474+0000","last_fullsized":"2024-08-17T17:58:38.692474+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:26:11.516977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1340,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.994385+0000","last_change":"2024-08-17T17:24:17.451287+0000","last_active":"2024-08-17T17:58:32.994385+0000","last_peered":"2024-08-17T17:58:32.994385+0000","last_clean":"2024-08-17T17:58:32.994385+0000","last_became_active":"2024-08-17T17:24:17.450951+0000","last_became_peered":"2024-08-17T17:24:17.450951+0000","last_unstale":"2024-08-17T17:58:32.994385+0000","last_undegraded":"2024-08-17T17:58:32.994385+0000","last_fullsized":"2024-08-17T17:58:32.994385+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:13:37.590108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":1340,"reported_epoch":1372,"state":"active+clean","last_fresh":"2024-08-17T17:58:32.329734+0000","last_change":"2024-08-17T17:23:42.907970+0000","last_active":"2024-08-17T17:58:32.329734+0000","last_peered":"2024-08-17T17:58:32.329734+0000","last_clean":"2024-08-17T17:58:32.329734+0000","last_became_active":"2024-08-17T17:23:42.907595+0000","last_became_peered":"2024-08-17T17:23:42.907595+0000","last_unstale":"2024-08-17T17:58:32.329734+0000","last_undegraded":"2024-08-17T17:58:32.329734+0000","last_fullsized":"2024-08-17T17:58:32.329734+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:11:46.130004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2023,"reported_epoch":1373,"state":"active+clean","last_fresh":"2024-08-17T17:58:38.689843+0000","last_change":"2024-08-17T17:24:17.486591+0000","last_active":"2024-08-17T17:58:38.689843+0000","last_peered":"2024-08-17T17:58:38.689843+0000","last_clean":"2024-08-17T17:58:38.689843+0000","last_became_active":"2024-08-17T17:24:17.486320+0000","last_became_peered":"2024-08-17T17:24:17.486320+0000","last_unstale":"2024-08-17T17:58:38.689843+0000","last_undegraded":"2024-08-17T17:58:38.689843+0000","last_fullsized":"2024-08-17T17:58:38.689843+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:03.249803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":54,"num_pg":16,"stat_sum":{"num_bytes":2231762944,"num_objects":34054,"num_object_clones":0,"num_object_copies":102162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34054,"num_write_kb":2179456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6704201728,"data_stored":6704201728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34054,"ondisk_log_size":34054,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":604,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":17266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234593,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1285244,"kb_used_data":1079476,"kb_used_omap":1,"kb_used_meta":205758,"kb_avail":92459908,"statfs":{"total":95995035648,"available":94678945792,"internally_reserved":0,"allocated":1105383424,"data_stored":1101082660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":210696651},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,5,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496231,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1396408,"kb_used_data":1098736,"kb_used_omap":1,"kb_used_meta":297662,"kb_avail":92348744,"statfs":{"total":95995035648,"available":94565113856,"internally_reserved":0,"allocated":1125105664,"data_stored":1120809451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":304806343},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565472,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1785184,"kb_used_data":1486488,"kb_used_omap":72,"kb_used_meta":298615,"kb_avail":91959968,"statfs":{"total":95995035648,"available":94167007232,"internally_reserved":0,"allocated":1522163712,"data_stored":1517886206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":305782254},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":674,"seq":2894807957927,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":763108,"kb_used_data":524188,"kb_used_omap":74,"kb_used_meta":238837,"kb_avail":92982044,"statfs":{"total":95995035648,"available":95213613056,"internally_reserved":0,"allocated":536768512,"data_stored":532487436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76116,"internal_metadata":244569772},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743186862,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":767528,"kb_used_data":402336,"kb_used_omap":66,"kb_used_meta":365117,"kb_avail":92977624,"statfs":{"total":95995035648,"available":95209086976,"internally_reserved":0,"allocated":411992064,"data_stored":407706861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":373880491},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":2830383448501,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757867,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1576296,"kb_used_data":1274464,"kb_used_omap":1,"kb_used_meta":301822,"kb_avail":92168856,"statfs":{"total":95995035648,"available":94380908544,"internally_reserved":0,"allocated":1305051136,"data_stored":1300766007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":309066182},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019504,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":942900,"kb_used_data":560748,"kb_used_omap":1,"kb_used_meta":382142,"kb_avail":92802252,"statfs":{"total":95995035648,"available":95029506048,"internally_reserved":0,"allocated":574205952,"data_stored":569912766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":391313862},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":421003264,"data_stored":421003264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":543883264,"data_stored":543883264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1603928064,"data_stored":1603928064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":568721408,"data_stored":568721408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1296367616,"data_stored":1296367616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1137704960,"data_stored":1137704960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1132593152,"data_stored":1132593152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:58:51.416 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pgp_num 1 2024-08-17T17:58:51.419 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:51.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:51.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:51.522 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:51.557 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:51.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:51.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:51.658 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:51.690 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:51.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:51.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:51.704 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 17 16 43104 43088 158.398 120.688 0.00582442 0.00630799 2024-08-17T17:58:51.792 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:51.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:51.826 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:51.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:51.927 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:51.960 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:51.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:51.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:52.022 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:51 smithi136 ceph-mon[109861]: pgmap v2313: 89 pgs: 89 active+clean; 2.1 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-08-17T17:58:52.022 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:58:52.022 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:51 smithi136 ceph-mon[115078]: pgmap v2313: 89 pgs: 89 active+clean; 2.1 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-08-17T17:58:52.022 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:58:52.062 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:52.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:52.099 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:52.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:52.200 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:52.221 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:52.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:52.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:51 smithi165 ceph-mon[105517]: pgmap v2313: 89 pgs: 89 active+clean; 2.1 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-08-17T17:58:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:58:52.323 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:52.355 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:52.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:52.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:52.457 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:52.493 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:52.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:52.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:52.595 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:52.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:52.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:52.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:52.704 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 18 16 44820 44804 155.556 107.25 0.00952131 0.00642328 2024-08-17T17:58:52.732 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:52.768 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:52.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:52.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:52.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:52.906 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:52.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:52.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:53.008 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:53.045 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:53.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:53.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:53.147 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:53.185 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:53.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:53.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:53.286 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:53.325 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:53.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:53.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:53.427 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:53.461 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:53.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:53.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:53.563 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:53.607 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:53.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:53.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:53.704 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 19 16 46681 46665 153.49 116.312 0.00737212 0.00651008 2024-08-17T17:58:53.709 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:53.739 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:53.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:53.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:53.842 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:53.876 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:53.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:53.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:53.979 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:54.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:54.016 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:54.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:54.036 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:53 smithi136 ceph-mon[109861]: pgmap v2314: 89 pgs: 89 active+clean; 2.6 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 184 MiB/s wr, 2.95k op/s 2024-08-17T17:58:54.036 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:53 smithi136 ceph-mon[115078]: pgmap v2314: 89 pgs: 89 active+clean; 2.6 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 184 MiB/s wr, 2.95k op/s 2024-08-17T17:58:54.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:54.151 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:54.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:54.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:54.253 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:54.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:53 smithi165 ceph-mon[105517]: pgmap v2314: 89 pgs: 89 active+clean; 2.6 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 184 MiB/s wr, 2.95k op/s 2024-08-17T17:58:54.291 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:54.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:54.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:54.393 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:54.442 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:54.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:54.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:54.545 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:54.580 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:54.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:54.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:54.682 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:54.704 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:2024-08-17T17:58:54.704677+0000 min lat: 0.0012629 max lat: 0.286626 avg lat: 0.00659798 2024-08-17T17:58:54.704 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:58:54.704 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 20 16 48481 48465 151.44 112.5 0.00593852 0.00659798 2024-08-17T17:58:54.716 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:54.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:54.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:54.819 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:54.849 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:54.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:54.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:54.951 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:54.972 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:54.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:54.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:55.074 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:55.100 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:55.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:55.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:55.201 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:55.223 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:55.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:55.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:55.325 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:55.353 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:55.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:55.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:55.456 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:55.491 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:55.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:55.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:55.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:55.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:55.615 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:55.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:55.704 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 21 16 49489 49473 147.228 63 0.0183494 0.00678669 2024-08-17T17:58:55.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:55.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:55.754 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:55.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:55.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:55.881 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:55.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:55.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:55.983 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:56.019 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:56.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:56.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:56.047 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:55 smithi136 ceph-mon[109861]: pgmap v2315: 89 pgs: 89 active+clean; 2.6 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-08-17T17:58:56.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:55 smithi136 ceph-mon[115078]: pgmap v2315: 89 pgs: 89 active+clean; 2.6 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-08-17T17:58:56.121 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:56.155 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:56.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:56.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:56.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:56.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:55 smithi165 ceph-mon[105517]: pgmap v2315: 89 pgs: 89 active+clean; 2.6 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-08-17T17:58:56.287 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:56.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:56.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:56.389 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:56.419 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:56.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:56.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:56.520 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:56.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:56.542 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:56.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:56.643 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:56.683 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:56.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:56.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:56.704 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 22 16 50962 50946 144.72 92.0625 0.0102438 0.00690476 2024-08-17T17:58:56.785 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:56.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:56.807 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:56.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:56.909 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:58:56.925 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:58:56.926 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T17:58:56.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.926 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.35370.0:50060 54.a e1373/1364 54:5346a245:::benchmark_data_smithi136_273969_object50059:head v 1373'3170, mlcod=1373'3168)", 2024-08-17T17:58:56.926 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:58:56.163775+0000", 2024-08-17T17:58:56.926 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0027917710000000002, 2024-08-17T17:58:56.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002820417, 2024-08-17T17:58:56.926 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:58:56.926 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:58:56.926 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T17:58:56.927 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:58:56.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:58:56.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.163775+0000", 2024-08-17T17:58:56.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:58:56.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:58:56.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.163775+0000", 2024-08-17T17:58:56.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:58:56.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.928 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:58:56.928 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.163776+0000", 2024-08-17T17:58:56.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3499999999999995e-07 2024-08-17T17:58:56.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.928 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:58:56.928 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.163810+0000", 2024-08-17T17:58:56.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4116999999999999e-05 2024-08-17T17:58:56.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.928 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:58:56.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.163824+0000", 2024-08-17T17:58:56.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3448999999999999e-05 2024-08-17T17:58:56.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:58:56.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.163831+0000", 2024-08-17T17:58:56.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3019999999999997e-06 2024-08-17T17:58:56.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:58:56.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.163887+0000", 2024-08-17T17:58:56.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5687999999999999e-05 2024-08-17T17:58:56.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:58:56.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.163909+0000", 2024-08-17T17:58:56.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1878999999999998e-05 2024-08-17T17:58:56.930 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:58:56.930 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:58:56.930 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:58:56.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.930 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.35370.0:50053 54.5 e1373/1367 54:a78b2343:::benchmark_data_smithi136_273969_object50052:head v 1373'3104, mlcod=1373'3101)", 2024-08-17T17:58:56.930 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:58:56.164424+0000", 2024-08-17T17:58:56.930 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0021433239999999998, 2024-08-17T17:58:56.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0022297969999999999, 2024-08-17T17:58:56.931 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:58:56.931 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:58:56.931 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T17:58:56.931 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:58:56.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:58:56.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.164424+0000", 2024-08-17T17:58:56.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:58:56.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:58:56.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.164424+0000", 2024-08-17T17:58:56.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:58:56.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:58:56.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.164424+0000", 2024-08-17T17:58:56.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8099999999999999e-07 2024-08-17T17:58:56.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:58:56.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.164456+0000", 2024-08-17T17:58:56.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2477000000000001e-05 2024-08-17T17:58:56.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:58:56.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.164464+0000", 2024-08-17T17:58:56.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5399999999999998e-06 2024-08-17T17:58:56.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:58:56.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.164469+0000", 2024-08-17T17:58:56.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9649999999999999e-06 2024-08-17T17:58:56.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:58:56.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.164523+0000", 2024-08-17T17:58:56.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4248e-05 2024-08-17T17:58:56.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:58:56.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.164542+0000", 2024-08-17T17:58:56.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9245999999999998e-05 2024-08-17T17:58:56.934 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:58:56.934 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:58:56.934 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:58:56.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.935 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:50065 54.f 54:f8887e0c:::benchmark_data_smithi136_273969_object50064:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1373)", 2024-08-17T17:58:56.935 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:58:56.164740+0000", 2024-08-17T17:58:56.935 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0018273320000000001, 2024-08-17T17:58:56.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0019833939999999999, 2024-08-17T17:58:56.935 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:58:56.935 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:58:56.935 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:58:56.935 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:58:56.935 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T17:58:56.935 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:58:56.935 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 50065 2024-08-17T17:58:56.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.936 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:58:56.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:58:56.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.164740+0000", 2024-08-17T17:58:56.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:58:56.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:58:56.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.164740+0000", 2024-08-17T17:58:56.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:58:56.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:58:56.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.164741+0000", 2024-08-17T17:58:56.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1340000000000001e-06 2024-08-17T17:58:56.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:58:56.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.164763+0000", 2024-08-17T17:58:56.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1994e-05 2024-08-17T17:58:56.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:58:56.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.164774+0000", 2024-08-17T17:58:56.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0764999999999999e-05 2024-08-17T17:58:56.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:58:56.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.164778+0000", 2024-08-17T17:58:56.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.758e-06 2024-08-17T17:58:56.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:58:56.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.164825+0000", 2024-08-17T17:58:56.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6780999999999998e-05 2024-08-17T17:58:56.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:58:56.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.164969+0000", 2024-08-17T17:58:56.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000143922 2024-08-17T17:58:56.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:58:56.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.165118+0000", 2024-08-17T17:58:56.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000148682 2024-08-17T17:58:56.939 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:58:56.939 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:58:56.939 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:58:56.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.939 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:50066 54.f 54.76b0181f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1373)", 2024-08-17T17:58:56.939 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:58:56.165427+0000", 2024-08-17T17:58:56.939 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.001140496, 2024-08-17T17:58:56.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001353904, 2024-08-17T17:58:56.940 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:58:56.940 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:58:56.940 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-08-17T17:58:56.940 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:58:56.940 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T17:58:56.940 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:58:56.940 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 50066 2024-08-17T17:58:56.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.940 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:58:56.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:58:56.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.165427+0000", 2024-08-17T17:58:56.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:58:56.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:58:56.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.165427+0000", 2024-08-17T17:58:56.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:58:56.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:58:56.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.165428+0000", 2024-08-17T17:58:56.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6199999999999996e-07 2024-08-17T17:58:56.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:58:56.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.165447+0000", 2024-08-17T17:58:56.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9939999999999999e-05 2024-08-17T17:58:56.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:58:56.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.165460+0000", 2024-08-17T17:58:56.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2269000000000001e-05 2024-08-17T17:58:56.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:58:56.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.165466+0000", 2024-08-17T17:58:56.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4130000000000003e-06 2024-08-17T17:58:56.944 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:58:56.944 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:58:56.944 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:58:56.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.944 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.35370.0:50063 54.d e1373/1367 54:b98c76b2:::benchmark_data_smithi136_273969_object50062:head v 1373'3142, mlcod=1373'3140)", 2024-08-17T17:58:56.944 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:58:56.165794+0000", 2024-08-17T17:58:56.944 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.000773184, 2024-08-17T17:58:56.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010301609999999999, 2024-08-17T17:58:56.945 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:58:56.945 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:58:56.945 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T17:58:56.945 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:58:56.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:58:56.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.165794+0000", 2024-08-17T17:58:56.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:58:56.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:58:56.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.165794+0000", 2024-08-17T17:58:56.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:58:56.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:58:56.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.165794+0000", 2024-08-17T17:58:56.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4299999999999998e-07 2024-08-17T17:58:56.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:58:56.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.165878+0000", 2024-08-17T17:58:56.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3120000000000004e-05 2024-08-17T17:58:56.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:58:56.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.165935+0000", 2024-08-17T17:58:56.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7664000000000002e-05 2024-08-17T17:58:56.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:58:56.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.165941+0000", 2024-08-17T17:58:56.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7930000000000003e-06 2024-08-17T17:58:56.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:58:56.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.165994+0000", 2024-08-17T17:58:56.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3482000000000001e-05 2024-08-17T17:58:56.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:58:56.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.166022+0000", 2024-08-17T17:58:56.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7358000000000001e-05 2024-08-17T17:58:56.949 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:58:56.949 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:58:56.949 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:58:56.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.949 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:50072 54.c 54:35ed2485:::benchmark_data_smithi136_273969_object50071:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1373)", 2024-08-17T17:58:56.949 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:58:56.165993+0000", 2024-08-17T17:58:56.949 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.00057440200000000001, 2024-08-17T17:58:56.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00087962200000000004, 2024-08-17T17:58:56.950 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:58:56.950 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:58:56.950 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:58:56.950 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T17:58:56.950 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T17:58:56.950 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:58:56.950 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 50072 2024-08-17T17:58:56.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.950 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:58:56.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:58:56.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.165993+0000", 2024-08-17T17:58:56.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:58:56.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:58:56.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.165993+0000", 2024-08-17T17:58:56.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:58:56.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:58:56.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.165994+0000", 2024-08-17T17:58:56.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9800000000000003e-07 2024-08-17T17:58:56.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:58:56.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.166014+0000", 2024-08-17T17:58:56.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0686000000000002e-05 2024-08-17T17:58:56.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:58:56.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.166023+0000", 2024-08-17T17:58:56.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8449999999999992e-06 2024-08-17T17:58:56.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:58:56.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.166029+0000", 2024-08-17T17:58:56.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.728e-06 2024-08-17T17:58:56.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:58:56.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.166210+0000", 2024-08-17T17:58:56.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000181316 2024-08-17T17:58:56.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:58:56.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.166353+0000", 2024-08-17T17:58:56.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014296900000000001 2024-08-17T17:58:56.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T17:58:56.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.166471+0000", 2024-08-17T17:58:56.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117483 2024-08-17T17:58:56.954 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:58:56.954 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:58:56.954 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:58:56.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.954 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.35370.0:50070 54.3 e1373/1367 54:c88024c4:::benchmark_data_smithi136_273969_object50069:head v 1373'3202, mlcod=1373'3200)", 2024-08-17T17:58:56.954 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:58:56.166270+0000", 2024-08-17T17:58:56.955 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.00029666300000000002, 2024-08-17T17:58:56.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00065656600000000003, 2024-08-17T17:58:56.955 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:58:56.955 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:58:56.955 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T17:58:56.955 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:58:56.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:58:56.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.166270+0000", 2024-08-17T17:58:56.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:58:56.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:58:56.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.166270+0000", 2024-08-17T17:58:56.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:58:56.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:58:56.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.166271+0000", 2024-08-17T17:58:56.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9200000000000001e-07 2024-08-17T17:58:56.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:58:56.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.166370+0000", 2024-08-17T17:58:56.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8914999999999997e-05 2024-08-17T17:58:56.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:58:56.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.166382+0000", 2024-08-17T17:58:56.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2072000000000001e-05 2024-08-17T17:58:56.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:58:56.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.166389+0000", 2024-08-17T17:58:56.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7000000000000002e-06 2024-08-17T17:58:56.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:58:56.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.166445+0000", 2024-08-17T17:58:56.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6400999999999998e-05 2024-08-17T17:58:56.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:58:56.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.166465+0000", 2024-08-17T17:58:56.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9388000000000001e-05 2024-08-17T17:58:56.958 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:58:56.958 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:58:56.958 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:58:56.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.958 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.35370.0:50068 54.a e1373/1364 54:5cefed5e:::benchmark_data_smithi136_273969_object50067:head v 1373'3171, mlcod=1373'3168)", 2024-08-17T17:58:56.958 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:58:56.166763+0000", 2024-08-17T17:58:56.958 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4294967295.999804, 2024-08-17T17:58:56.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021263599999999999, 2024-08-17T17:58:56.959 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T17:58:56.959 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T17:58:56.959 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T17:58:56.959 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T17:58:56.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T17:58:56.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.166763+0000", 2024-08-17T17:58:56.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:58:56.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T17:58:56.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.166763+0000", 2024-08-17T17:58:56.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T17:58:56.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T17:58:56.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.166764+0000", 2024-08-17T17:58:56.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4000000000000001e-07 2024-08-17T17:58:56.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T17:58:56.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.166793+0000", 2024-08-17T17:58:56.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8968000000000001e-05 2024-08-17T17:58:56.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T17:58:56.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.166805+0000", 2024-08-17T17:58:56.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2601e-05 2024-08-17T17:58:56.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T17:58:56.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.166811+0000", 2024-08-17T17:58:56.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7370000000000003e-06 2024-08-17T17:58:56.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T17:58:56.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.166917+0000", 2024-08-17T17:58:56.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105783 2024-08-17T17:58:56.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T17:58:56.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T17:58:56.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T17:58:56.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:58:56.166933+0000", 2024-08-17T17:58:56.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6518999999999999e-05 2024-08-17T17:58:56.962 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:58:56.962 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T17:58:56.962 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:58:56.962 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T17:58:56.962 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-17T17:58:56.962 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 8 2024-08-17T17:58:56.962 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:58:56.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:56.963 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:56.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:58:57.064 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:57.094 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:57.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:57.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:57.196 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:57.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:57.235 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:57.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:57.336 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:57.369 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:57.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:57.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:57.472 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:57.510 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:57.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:57.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:57.612 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:57.648 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:57.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:57.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:57.698 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:58:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:58:57] "GET /metrics HTTP/1.1" 200 36533 "" "Prometheus/2.51.0" 2024-08-17T17:58:57.704 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 23 16 52420 52404 142.389 91.125 0.00813578 0.00701797 2024-08-17T17:58:57.751 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:57.791 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:57.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:57.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:57.893 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:57.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:57.932 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:57.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:58.014 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:58:58.034 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:58.076 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:57 smithi136 ceph-mon[109861]: pgmap v2316: 89 pgs: 89 active+clean; 2.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 122 MiB/s wr, 1.96k op/s 2024-08-17T17:58:58.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:57 smithi136 ceph-mon[115078]: pgmap v2316: 89 pgs: 89 active+clean; 2.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 122 MiB/s wr, 1.96k op/s 2024-08-17T17:58:58.077 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T17:58:58.077 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:58:58.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:58.079 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:58.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:58.107 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T17:58:58.107 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:58:58.133 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T17:58:58.133 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:58:58.164 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T17:58:58.164 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:58:58.181 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:58.187 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T17:58:58.187 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:58:58.212 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:58.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:58.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:58.218 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T17:58:58.218 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:58:58.248 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T17:58:58.248 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:58:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:57 smithi165 ceph-mon[105517]: pgmap v2316: 89 pgs: 89 active+clean; 2.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 122 MiB/s wr, 1.96k op/s 2024-08-17T17:58:58.279 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T17:58:58.279 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:58:58.315 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:58.352 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:58.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:58.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:58.453 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:58:58.485 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:58.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:58.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:58:58.587 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:58.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:58.625 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:58.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:58.705 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 24 16 54068 54052 140.748 103 0.0100704 0.00709978 2024-08-17T17:58:58.726 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:58.767 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:58.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:58.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:58.868 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:58.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:58.908 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:58.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:59.009 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:58:59.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:59.052 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:59.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:58:59.153 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:59.191 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:59.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:59.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:59.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:58 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1695855142' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:58:59.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:58 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1695855142' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:58:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:58 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1695855142' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:58:59.293 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:58:59.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:59.331 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:59.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:58:59.432 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:58:59.475 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:59.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:59.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:58:59.577 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:58:59.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:59.619 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:59.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:58:59.705 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 25 16 55699 55683 139.195 101.938 0.00490757 0.00717847 2024-08-17T17:58:59.719 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:58:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:59.753 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:58:59.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:58:59.823 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T17:58:59.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:58:59.891 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:58:59.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:58:59.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:58:59.993 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:00.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:00.027 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:00.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:00.128 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:00.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:00.162 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:00.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:00.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:59 smithi136 ceph-mon[109861]: pgmap v2317: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 118 MiB/s wr, 1.88k op/s 2024-08-17T17:59:00.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:59 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/292052018' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:59:00.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:58:59 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:59:00.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:59 smithi136 ceph-mon[115078]: pgmap v2317: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 118 MiB/s wr, 1.88k op/s 2024-08-17T17:59:00.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:59 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/292052018' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:59:00.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:58:59 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:59:00.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:59 smithi165 ceph-mon[105517]: pgmap v2317: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 118 MiB/s wr, 1.88k op/s 2024-08-17T17:59:00.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:59 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/292052018' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:59:00.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:58:59 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:59:00.264 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:00.297 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:00.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:00.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:00.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:00.444 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:00.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:00.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:00.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:00.575 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:00.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:00.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:00.677 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:00.705 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 26 16 57343 57327 137.793 102.75 0.0199407 0.00725172 2024-08-17T17:59:00.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:00.715 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:00.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:00.816 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:00.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:00.854 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:00.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:00.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:00 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:59:00.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:00 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:59:00.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:00 smithi136 ceph-mon[109861]: osdmap e1374: 8 total, 8 up, 7 in 2024-08-17T17:59:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:00 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:59:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:00 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:59:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:00 smithi136 ceph-mon[115078]: osdmap e1374: 8 total, 8 up, 7 in 2024-08-17T17:59:00.955 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:00.985 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:00.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:00.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:01.087 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:01.128 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:01.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:01.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:01.230 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:00 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:59:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:00 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:59:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:00 smithi165 ceph-mon[105517]: osdmap e1374: 8 total, 8 up, 7 in 2024-08-17T17:59:01.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:01.268 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:01.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:01.280 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-17T17:59:01.369 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:01.406 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:01.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:01.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:01.508 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:01.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:01.548 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:01.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:01.650 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:01.693 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:01.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:01.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:01.705 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 27 16 58906 58890 136.307 97.6875 0.0161914 0.00733141 2024-08-17T17:59:01.795 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:01.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:01.833 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:01.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:01.934 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:01.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:01 smithi136 ceph-mon[109861]: pgmap v2319: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-08-17T17:59:01.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:01 smithi136 ceph-mon[115078]: pgmap v2319: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-08-17T17:59:01.969 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:01.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:01.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:02.071 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:02.104 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:02.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:02.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:02.198 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:59:02.206 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:02.250 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:02.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:02.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:02.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:01 smithi165 ceph-mon[105517]: pgmap v2319: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-08-17T17:59:02.352 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:02.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:02.385 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:02.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:02.486 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:02.517 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:02.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:02.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:02.619 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:02.655 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:02.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:02.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:02.705 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 28 16 60467 60451 134.923 97.5625 0.0106752 0.00740688 2024-08-17T17:59:02.725 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:59:02.757 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:02.804 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:02.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:02.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:02.906 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:02.942 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:02.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:02.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:03.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:03.079 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:03.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:03.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:03.181 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:03.215 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:03.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:03.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:03.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:03.347 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:03.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:03.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:03.449 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:03.487 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:03.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:03.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:03.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:03.625 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:03.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:03.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:03.678 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:59:03.705 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 29 16 61948 61932 133.462 92.5625 0.00527508 0.00748728 2024-08-17T17:59:03.726 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:03.768 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:03.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:03.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:03.835 INFO:teuthology.orchestra.run.smithi136.stderr:set pool 1 pgp_num to 1 2024-08-17T17:59:03.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:03.905 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:03.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:03.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:04.006 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:04.037 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:04.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:04.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:04.139 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:04.170 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:04.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:04.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:04.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:03 smithi136 ceph-mon[109861]: pgmap v2320: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-08-17T17:59:04.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:03 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2724622605' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-08-17T17:59:04.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:03 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-08-17T17:59:04.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:03 smithi136 ceph-mon[115078]: pgmap v2320: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-08-17T17:59:04.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:03 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2724622605' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-08-17T17:59:04.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:03 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-08-17T17:59:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:03 smithi165 ceph-mon[105517]: pgmap v2320: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-08-17T17:59:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:03 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2724622605' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-08-17T17:59:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:03 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-08-17T17:59:04.271 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:04.298 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:04.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:04.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:04.400 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:04.430 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:04.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:04.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:04.531 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:04.561 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:04.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:04.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:04.663 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:04.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:04.694 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:04.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:04.705 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 30 16 63329 63313 131.89 86.3125 0.00432369 0.00757661 2024-08-17T17:59:04.795 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:04.831 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:04.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:04.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:04.934 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:04.968 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:04.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:04.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:05.070 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:05.102 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:05.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:05.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:05.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:04 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-08-17T17:59:05.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:04 smithi136 ceph-mon[109861]: osdmap e1375: 8 total, 8 up, 7 in 2024-08-17T17:59:05.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:04 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-08-17T17:59:05.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:04 smithi136 ceph-mon[115078]: osdmap e1375: 8 total, 8 up, 7 in 2024-08-17T17:59:05.204 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:05.234 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:05.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:05.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:04 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-08-17T17:59:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:04 smithi165 ceph-mon[105517]: osdmap e1375: 8 total, 8 up, 7 in 2024-08-17T17:59:05.335 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:05.378 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:05.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:05.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:05.480 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:05.519 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:05.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:05.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:05.622 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:05.652 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:05.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:05.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:05.705 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 31 16 64634 64618 130.266 81.5625 0.00737101 0.00767141 2024-08-17T17:59:05.755 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:05.786 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:05.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:05.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:05.888 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:05.924 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:05.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:05.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:06.025 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:06.059 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:06.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:06.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:06.161 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:06.162 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:05 smithi165 ceph-mon[105517]: pgmap v2322: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-08-17T17:59:06.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:05 smithi136 ceph-mon[109861]: pgmap v2322: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-08-17T17:59:06.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:05 smithi136 ceph-mon[115078]: pgmap v2322: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-08-17T17:59:06.200 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:06.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:06.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:06.302 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:06.344 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:06.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:06.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:06.446 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:06.477 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:06.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:06.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:06.579 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:06.608 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:06.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:06.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:06.705 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 32 16 66084 66068 129.027 90.625 0.0152607 0.00774524 2024-08-17T17:59:06.710 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:06.741 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:06.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:06.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:06.843 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:06.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:06.883 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:06.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:06.938 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:59:06.938 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:59:06.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:07.028 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:07.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:07.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:07.130 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:07.166 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:07.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:07.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:07.211 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:59:07.256 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:59:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:59:07.268 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:07.274 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T17:59:07.274 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:59:07.304 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:07.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:07.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:07.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:07.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:07.443 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:07.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:07.521 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:59:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:59:07] "GET /metrics HTTP/1.1" 200 36542 "" "Prometheus/2.51.0" 2024-08-17T17:59:07.562 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:07.589 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:07.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:07.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:07.691 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:07.705 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 33 16 67474 67458 127.75 86.875 0.0169409 0.00782253 2024-08-17T17:59:07.721 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:07.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:07.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:07.823 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:07.852 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:07.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:07.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:07.954 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:07.984 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:07.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:07.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:08.086 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:08.134 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:08.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:08.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:08.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:07 smithi136 ceph-mon[109861]: pgmap v2323: 89 pgs: 89 active+clean; 3.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-08-17T17:59:08.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:07 smithi136 ceph-mon[115078]: pgmap v2323: 89 pgs: 89 active+clean; 3.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-08-17T17:59:08.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:07 smithi165 ceph-mon[105517]: pgmap v2323: 89 pgs: 89 active+clean; 3.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-08-17T17:59:08.269 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:08.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:08.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:08.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:08.409 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:08.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:08.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:08.511 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:08.549 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:08.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:08.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:08.651 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:08.680 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:08.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:08.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:08.706 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 34 15 68864 68849 126.549 86.9375 0.00946086 0.00789669 2024-08-17T17:59:08.782 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:08.784 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-17T17:59:08.820 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:08.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:08.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:08.922 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:08.955 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:08.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:08.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:09.057 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:09.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:09.090 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:09.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:09.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:09.227 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:09.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:09.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:09.329 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:09.369 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:09.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:09.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:09.472 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:09.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:09.510 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:09.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:09.611 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:09.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:09.647 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:09.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:09.706 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 35 16 70258 70242 125.421 87.0625 0.013308 0.00796834 2024-08-17T17:59:09.748 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:09.780 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:09.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:09.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:09.882 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:09.923 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:09.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:09.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:10.025 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:10.051 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:10.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:10.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:10.153 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:10.174 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:10.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:10.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:10.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:09 smithi136 ceph-mon[109861]: pgmap v2324: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-08-17T17:59:10.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:09 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 5]} : dispatch 2024-08-17T17:59:10.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:09 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]} : dispatch 2024-08-17T17:59:10.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:09 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "54.c", "id": [3, 1]} : dispatch 2024-08-17T17:59:10.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:09 smithi136 ceph-mon[115078]: pgmap v2324: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-08-17T17:59:10.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:09 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 5]} : dispatch 2024-08-17T17:59:10.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:09 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]} : dispatch 2024-08-17T17:59:10.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:09 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "54.c", "id": [3, 1]} : dispatch 2024-08-17T17:59:10.247 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:59:10.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:09 smithi165 ceph-mon[105517]: pgmap v2324: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-08-17T17:59:10.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:09 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 5]} : dispatch 2024-08-17T17:59:10.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:09 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]} : dispatch 2024-08-17T17:59:10.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:09 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "54.c", "id": [3, 1]} : dispatch 2024-08-17T17:59:10.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:10.296 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:10.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:10.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:10.400 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:10.421 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:10.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:10.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:10.522 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:10.549 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:10.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:10.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:10.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:10.676 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:10.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:10.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:10.706 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 36 16 70760 70744 122.808 31.375 0.00356116 0.00797427 2024-08-17T17:59:10.778 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:10.800 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:10.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:10.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:10.902 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:10.928 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:10.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:10.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:11.030 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:11.056 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:11.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:11.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:11.158 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:11.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:11.181 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:11.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:11.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 5]}]': finished 2024-08-17T17:59:11.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-08-17T17:59:11.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "54.c", "id": [3, 1]}]': finished 2024-08-17T17:59:11.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:10 smithi136 ceph-mon[115078]: osdmap e1376: 8 total, 8 up, 7 in 2024-08-17T17:59:11.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 5]}]': finished 2024-08-17T17:59:11.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-08-17T17:59:11.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "54.c", "id": [3, 1]}]': finished 2024-08-17T17:59:11.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:10 smithi136 ceph-mon[109861]: osdmap e1376: 8 total, 8 up, 7 in 2024-08-17T17:59:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 5]}]': finished 2024-08-17T17:59:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-08-17T17:59:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "54.c", "id": [3, 1]}]': finished 2024-08-17T17:59:11.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:10 smithi165 ceph-mon[105517]: osdmap e1376: 8 total, 8 up, 7 in 2024-08-17T17:59:11.282 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:11.303 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:11.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:11.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:11.405 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:11.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:11.427 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:11.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:11.528 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:11.554 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:11.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:11.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:11.656 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:11.678 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:11.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:11.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:11.706 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 37 16 70760 70744 119.489 0 - 0.00797427 2024-08-17T17:59:11.715 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-17T17:59:11.715 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:59:11.716 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_1 2024-08-17T17:59:11.716 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:59:11.782 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:11.808 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:11.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:11.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:11.886 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T17:59:11.909 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:11.933 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:11.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:11.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:11.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:11 smithi136 ceph-mon[109861]: pgmap v2326: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 56 MiB/s wr, 896 op/s 2024-08-17T17:59:11.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:11 smithi136 ceph-mon[109861]: osdmap e1377: 8 total, 8 up, 7 in 2024-08-17T17:59:11.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:11 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3949341110' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:59:11.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:11 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:59:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:11 smithi136 ceph-mon[115078]: pgmap v2326: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 56 MiB/s wr, 896 op/s 2024-08-17T17:59:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:11 smithi136 ceph-mon[115078]: osdmap e1377: 8 total, 8 up, 7 in 2024-08-17T17:59:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:11 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3949341110' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:59:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:11 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:59:12.036 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:12.064 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:12.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:12.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:12.166 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:12.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:12.202 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:12.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:12.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:11 smithi165 ceph-mon[105517]: pgmap v2326: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 56 MiB/s wr, 896 op/s 2024-08-17T17:59:12.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:11 smithi165 ceph-mon[105517]: osdmap e1377: 8 total, 8 up, 7 in 2024-08-17T17:59:12.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:11 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3949341110' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:59:12.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:11 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:59:12.304 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:12.344 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:12.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:12.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:12.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:12.481 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:12.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:12.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:12.583 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:12.621 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:12.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:12.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:12.706 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 38 16 71623 71607 117.762 26.9688 0.00772968 0.00848595 2024-08-17T17:59:12.722 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:12.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:12.757 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:12.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:12.858 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:12.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:12.896 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:12.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:12.997 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:13.035 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:13.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:13.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:13.137 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:13.175 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:13.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:13.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:13.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:12 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:59:13.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:12 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T17:59:13.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:12 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:59:13.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:12 smithi136 ceph-mon[109861]: osdmap e1378: 8 total, 8 up, 7 in 2024-08-17T17:59:13.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:12 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:59:13.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:12 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T17:59:13.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:12 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:59:13.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:12 smithi136 ceph-mon[115078]: osdmap e1378: 8 total, 8 up, 7 in 2024-08-17T17:59:13.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:12 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:59:13.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:12 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T17:59:13.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:12 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:59:13.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:12 smithi165 ceph-mon[105517]: osdmap e1378: 8 total, 8 up, 7 in 2024-08-17T17:59:13.278 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:13.312 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:13.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:13.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:13.414 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:13.455 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:13.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:13.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:13.557 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:13.595 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:13.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:13.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:13.697 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:13.707 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 39 16 72929 72913 116.836 81.625 0.00921206 0.00855421 2024-08-17T17:59:13.728 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:13.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:13.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:13.830 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:13.861 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:13.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:13.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:13.963 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:13.964 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T17:59:13.964 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:59:14.009 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:14.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:14.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:14.110 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:14.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:14.147 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:14.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:14.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:13 smithi136 ceph-mon[109861]: pgmap v2329: 89 pgs: 1 peering, 88 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-08-17T17:59:14.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:13 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:59:14.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:13 smithi136 ceph-mon[115078]: pgmap v2329: 89 pgs: 1 peering, 88 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-08-17T17:59:14.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:13 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:59:14.249 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:14.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:13 smithi165 ceph-mon[105517]: pgmap v2329: 89 pgs: 1 peering, 88 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-08-17T17:59:14.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:13 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:59:14.293 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:14.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:14.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:14.395 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:14.431 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:14.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:14.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:14.533 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:14.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:14.575 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:14.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:14.676 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:14.707 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:2024-08-17T17:59:14.707054+0000 min lat: 0.0012629 max lat: 2.15603 avg lat: 0.00863544 2024-08-17T17:59:14.707 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:59:14.707 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 40 16 74082 74066 115.716 72.0625 0.0121599 0.00863544 2024-08-17T17:59:14.713 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:59:14.720 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:14.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:14.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:14.778 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T17:59:14.779 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:59:14.812 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T17:59:14.812 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:59:14.823 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:14.846 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T17:59:14.846 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:59:14.865 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:14.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:14.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:14.877 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T17:59:14.877 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:59:14.909 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T17:59:14.909 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:59:14.946 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T17:59:14.946 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:59:14.968 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:14.978 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T17:59:14.979 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:59:14.979 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:14 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1081737038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:59:14.979 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:14 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1081737038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:59:15.009 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T17:59:15.009 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:59:15.023 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:15.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:15.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:15.125 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:15.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:15.160 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:15.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:15.203 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-17T17:59:15.239 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:14 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1081737038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:59:15.261 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:15.288 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:15.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:15.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:15.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:15.430 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:59:15.442 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:15.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:15.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:15.543 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:15.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:15.601 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:15.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:15.702 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:15.707 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 41 16 75175 75159 114.56 68.3125 0.0083897 0.00872383 2024-08-17T17:59:15.732 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:15.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:15.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:15.834 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:15.882 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:15.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:15.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:15.945 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:59:15.984 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:16.017 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:16.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:16.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:16.118 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:16.149 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:16.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:16.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:16.173 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:15 smithi136 ceph-mon[109861]: pgmap v2330: 89 pgs: 1 peering, 88 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 47 MiB/s wr, 756 op/s 2024-08-17T17:59:16.173 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:15 smithi136 ceph-mon[115078]: pgmap v2330: 89 pgs: 1 peering, 88 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 47 MiB/s wr, 756 op/s 2024-08-17T17:59:16.252 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:16.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:15 smithi165 ceph-mon[105517]: pgmap v2330: 89 pgs: 1 peering, 88 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 47 MiB/s wr, 756 op/s 2024-08-17T17:59:16.284 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:16.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:16.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:16.386 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:16.415 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:16.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:16.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:16.517 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:16.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:16.554 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:16.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:16.655 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:16.697 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:16.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:16.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:16.707 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 42 16 76372 76356 113.613 74.8125 0.0131689 0.00879545 2024-08-17T17:59:16.723 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:59:16.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:16.842 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:16.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:16.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:16.944 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:16.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:16.982 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:16.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:17.083 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:17.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:17.116 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:17.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:17.217 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:17.258 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:17.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:17.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:17.360 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:17.399 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:17.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:17.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:17.500 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:17.532 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:17.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:17.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:17.634 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:17.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:17.669 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:17.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:17.707 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 43 16 77520 77504 112.639 71.75 0.0112419 0.00887231 2024-08-17T17:59:17.771 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:17.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:17.802 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:17.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:17.903 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:17.916 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:59:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:59:17] "GET /metrics HTTP/1.1" 200 36542 "" "Prometheus/2.51.0" 2024-08-17T17:59:17.919 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:59:17.938 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:17.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:17.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:17 smithi165 ceph-mon[105517]: pgmap v2331: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s; 4227/218736 objects misplaced (1.932%); 955 KiB/s, 14 objects/s recovering 2024-08-17T17:59:18.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:18.080 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:18.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:18.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:18.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:18.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:17 smithi136 ceph-mon[109861]: pgmap v2331: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s; 4227/218736 objects misplaced (1.932%); 955 KiB/s, 14 objects/s recovering 2024-08-17T17:59:18.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:17 smithi136 ceph-mon[115078]: pgmap v2331: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s; 4227/218736 objects misplaced (1.932%); 955 KiB/s, 14 objects/s recovering 2024-08-17T17:59:18.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:18.221 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:18.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:18.322 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:18.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:18.360 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:18.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:18.461 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:18.488 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:18.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:18.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:18.589 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:18.624 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:18.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:18.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:18.628 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T17:59:18.629 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T17:59:18.629 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.629 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:78407 54.e 54:738fbf0a:::benchmark_data_smithi136_273969_object78406:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1378)", 2024-08-17T17:59:18.629 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:18.618629+0000", 2024-08-17T17:59:18.629 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0083548779999999996, 2024-08-17T17:59:18.629 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0083917460000000003, 2024-08-17T17:59:18.629 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:18.629 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:59:18.629 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:18.629 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:18.629 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:18.630 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78407 2024-08-17T17:59:18.630 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.630 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:18.630 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:18.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.618629+0000", 2024-08-17T17:59:18.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:18.630 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.630 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:18.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.618629+0000", 2024-08-17T17:59:18.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:18.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:18.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.618621+0000", 2024-08-17T17:59:18.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999919 2024-08-17T17:59:18.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:18.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.618662+0000", 2024-08-17T17:59:18.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1455999999999997e-05 2024-08-17T17:59:18.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:18.632 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.618680+0000", 2024-08-17T17:59:18.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7904e-05 2024-08-17T17:59:18.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.632 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.632 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:18.632 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.618688+0000", 2024-08-17T17:59:18.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9489999999999996e-06 2024-08-17T17:59:18.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.632 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.632 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:18.632 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.618741+0000", 2024-08-17T17:59:18.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3103000000000003e-05 2024-08-17T17:59:18.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.633 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.633 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:18.633 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.618868+0000", 2024-08-17T17:59:18.633 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000126957 2024-08-17T17:59:18.633 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.633 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.633 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:59:18.633 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.619007+0000", 2024-08-17T17:59:18.633 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0001384 2024-08-17T17:59:18.633 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.633 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.633 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:18.634 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.622460+0000", 2024-08-17T17:59:18.634 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003453607 2024-08-17T17:59:18.634 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.634 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.634 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:59:18.634 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.622475+0000", 2024-08-17T17:59:18.634 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4949999999999999e-05 2024-08-17T17:59:18.634 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:18.634 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:18.634 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:18.634 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.634 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.634 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:78415 54.e 54:7e7f36e4:::benchmark_data_smithi136_273969_object78414:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1378)", 2024-08-17T17:59:18.635 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:18.622536+0000", 2024-08-17T17:59:18.635 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0044475649999999997, 2024-08-17T17:59:18.635 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045888869999999998, 2024-08-17T17:59:18.635 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:18.635 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T17:59:18.635 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:18.635 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:18.635 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:18.635 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78415 2024-08-17T17:59:18.635 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.635 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:18.635 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.635 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:18.636 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.622536+0000", 2024-08-17T17:59:18.636 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:18.636 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.636 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.636 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:18.636 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.622536+0000", 2024-08-17T17:59:18.636 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:18.636 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.636 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.636 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:18.637 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.622535+0000", 2024-08-17T17:59:18.637 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:59:18.637 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.637 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.637 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:18.637 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.622580+0000", 2024-08-17T17:59:18.637 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5237000000000001e-05 2024-08-17T17:59:18.637 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.637 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.637 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:18.637 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.622593+0000", 2024-08-17T17:59:18.637 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3293e-05 2024-08-17T17:59:18.638 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.638 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.638 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:18.638 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.622600+0000", 2024-08-17T17:59:18.638 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7800000000000003e-06 2024-08-17T17:59:18.638 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.638 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.638 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:18.638 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.622737+0000", 2024-08-17T17:59:18.638 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013680699999999999 2024-08-17T17:59:18.638 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:18.638 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:18.639 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:18.639 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.639 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.639 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:78416 54.8 54:162998e9:::benchmark_data_smithi136_273969_object78415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1378)", 2024-08-17T17:59:18.639 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:18.622925+0000", 2024-08-17T17:59:18.639 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.004058318, 2024-08-17T17:59:18.639 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042577379999999996, 2024-08-17T17:59:18.639 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:18.639 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:59:18.639 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:18.639 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:18.639 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:18.639 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78416 2024-08-17T17:59:18.639 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.640 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:18.640 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.640 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:18.640 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.622925+0000", 2024-08-17T17:59:18.640 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:18.640 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.640 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.640 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:18.640 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.622925+0000", 2024-08-17T17:59:18.640 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:18.640 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.640 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.640 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:18.641 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.622925+0000", 2024-08-17T17:59:18.641 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:59:18.641 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.641 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.641 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:18.641 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.622984+0000", 2024-08-17T17:59:18.641 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9191000000000003e-05 2024-08-17T17:59:18.641 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.641 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.641 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:18.641 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.622997+0000", 2024-08-17T17:59:18.641 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2773e-05 2024-08-17T17:59:18.641 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.641 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.642 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:18.642 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.623003+0000", 2024-08-17T17:59:18.642 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3169999999999998e-06 2024-08-17T17:59:18.642 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.642 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.642 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:18.642 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.623135+0000", 2024-08-17T17:59:18.642 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013211000000000001 2024-08-17T17:59:18.642 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.642 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.642 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:18.642 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.623395+0000", 2024-08-17T17:59:18.642 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025976299999999999 2024-08-17T17:59:18.643 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.643 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.643 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-08-17T17:59:18.643 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.623766+0000", 2024-08-17T17:59:18.643 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037073599999999998 2024-08-17T17:59:18.643 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:18.643 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:18.643 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:18.643 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.643 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.643 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.35370.0:78414 54.0 e1378/1364 54:01ca071d:::benchmark_data_smithi136_273969_object78413:head v 1378'4897, mlcod=1378'4895)", 2024-08-17T17:59:18.643 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:18.624847+0000", 2024-08-17T17:59:18.644 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0021365490000000002, 2024-08-17T17:59:18.644 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024008969999999999, 2024-08-17T17:59:18.644 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:18.644 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T17:59:18.644 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:18.644 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.644 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:18.644 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.624847+0000", 2024-08-17T17:59:18.644 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:18.644 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.644 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.644 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:18.644 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.624847+0000", 2024-08-17T17:59:18.644 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:18.645 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.645 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.645 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:18.645 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.624846+0000", 2024-08-17T17:59:18.645 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:59:18.645 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.645 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.645 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:18.645 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.624872+0000", 2024-08-17T17:59:18.645 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5536000000000001e-05 2024-08-17T17:59:18.645 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.645 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.645 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:18.646 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.624889+0000", 2024-08-17T17:59:18.646 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.77e-05 2024-08-17T17:59:18.646 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.646 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.646 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:18.646 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.624898+0000", 2024-08-17T17:59:18.646 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2509999999999994e-06 2024-08-17T17:59:18.646 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.646 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.646 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:18.646 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.625074+0000", 2024-08-17T17:59:18.646 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017686899999999999 2024-08-17T17:59:18.646 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:18.647 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:18.647 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:18.647 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:18.625688+0000", 2024-08-17T17:59:18.647 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00061350100000000002 2024-08-17T17:59:18.647 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:18.647 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:18.647 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:18.647 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:18.647 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-08-17T17:59:18.647 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 4 2024-08-17T17:59:18.647 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T17:59:18.707 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 44 16 78504 78488 111.477 61.5 0.00998179 0.00896448 2024-08-17T17:59:18.727 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:18.761 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:18.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:18.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:18.862 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:18.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:18.899 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:18.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:19.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:19.037 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:19.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:19.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:19.139 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:19.170 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:19.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:19.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:19.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:19.308 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:19.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:19.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:19.411 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:19.451 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:19.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:19.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:19.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:19.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:19.578 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:19.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:19.679 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:19.707 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 45 16 79564 79548 110.472 66.25 0.010842 0.00904673 2024-08-17T17:59:19.710 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:19.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:19.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:19.813 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:19.848 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:19.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:19.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:19.950 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:19.994 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:19.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:19.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:20.096 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:20.118 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:20.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:20.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:20.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:19 smithi136 ceph-mon[109861]: pgmap v2332: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 81 MiB/s wr, 1.30k op/s; 4227/228921 objects misplaced (1.846%); 816 KiB/s, 12 objects/s recovering 2024-08-17T17:59:20.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:19 smithi136 ceph-mon[115078]: pgmap v2332: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 81 MiB/s wr, 1.30k op/s; 4227/228921 objects misplaced (1.846%); 816 KiB/s, 12 objects/s recovering 2024-08-17T17:59:20.220 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:20.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:20.254 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:20.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:19 smithi165 ceph-mon[105517]: pgmap v2332: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 81 MiB/s wr, 1.30k op/s; 4227/228921 objects misplaced (1.846%); 816 KiB/s, 12 objects/s recovering 2024-08-17T17:59:20.354 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:20.394 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:20.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:20.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:20.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:20.530 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:20.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:20.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:20.632 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:20.634 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-17T17:59:20.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:20.686 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:20.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:20.707 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 46 16 80623 80607 109.509 66.1875 0.0108516 0.00912521 2024-08-17T17:59:20.787 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:20.832 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:20.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:20.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:20.934 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:20.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:20.994 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:20.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:21.095 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:21.126 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:21.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:21.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:21.207 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:59:21.227 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:21.254 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:21.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:21.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:21.356 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:21.389 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:21.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:21.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:21.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:21.527 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:21.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:21.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:21.628 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:21.664 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:21.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:21.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:21.707 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 47 16 81606 81590 108.486 61.4375 0.0140342 0.00921269 2024-08-17T17:59:21.767 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:21.797 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:21.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:21.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:21.898 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:21.920 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:21.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:21.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:22.022 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:22.060 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:22.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:22.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:22.162 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:22.188 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:22.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:22.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:22.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:22.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:22.332 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:22.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:22.433 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:22.462 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:22.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:22.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:22 smithi165 ceph-mon[105517]: pgmap v2333: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 56 MiB/s wr, 901 op/s; 4227/228921 objects misplaced (1.846%); 664 KiB/s, 10 objects/s recovering 2024-08-17T17:59:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:59:22.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:59:22.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:22.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:22.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:22.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:22.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:22 smithi136 ceph-mon[109861]: pgmap v2333: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 56 MiB/s wr, 901 op/s; 4227/228921 objects misplaced (1.846%); 664 KiB/s, 10 objects/s recovering 2024-08-17T17:59:22.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:59:22.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:59:22.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:22 smithi136 ceph-mon[115078]: pgmap v2333: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 56 MiB/s wr, 901 op/s; 4227/228921 objects misplaced (1.846%); 664 KiB/s, 10 objects/s recovering 2024-08-17T17:59:22.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:59:22.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:59:22.707 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 48 16 82499 82483 107.389 55.8125 0.0110347 0.00930708 2024-08-17T17:59:22.731 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:22.760 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:22.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:22.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:22.862 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:22.892 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:22.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:22.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:22.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:23.035 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:23.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:23.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:23.137 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:23.179 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:23.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:23.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:23.280 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:23.312 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:23.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:23.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:23.413 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:23.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:23.463 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:23.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:23.564 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:23.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:23.603 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:23.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:23.704 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:23.708 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 49 16 83615 83599 106.62 69.75 0.00586377 0.00937392 2024-08-17T17:59:23.744 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:23.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:23.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:23.846 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:23.877 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:23.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:23.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:23.979 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:24.020 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:24.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:24.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:24.122 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:24.153 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:24.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:24.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:24.255 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:24.304 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:24.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:24.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:24.406 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:24.435 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:59:24.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:24.448 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:24.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:24.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:24 smithi165 ceph-mon[105517]: pgmap v2334: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 77 MiB/s wr, 1.24k op/s; 4090/244524 objects misplaced (1.673%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:59:24.541 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:24 smithi136 ceph-mon[109861]: pgmap v2334: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 77 MiB/s wr, 1.24k op/s; 4090/244524 objects misplaced (1.673%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:59:24.541 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:24 smithi136 ceph-mon[115078]: pgmap v2334: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 77 MiB/s wr, 1.24k op/s; 4090/244524 objects misplaced (1.673%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:59:24.549 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:24.577 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:24.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:24.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:24.679 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:24.708 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 50 16 84642 84626 105.771 64.1875 0.00664979 0.00944805 2024-08-17T17:59:24.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:24.713 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:24.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:24.814 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:24.838 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T17:59:24.844 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T17:59:24.854 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:24.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:24.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:24.956 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:24.992 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:24.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:24.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:25.094 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:25.139 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:25.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:25.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:25.240 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:25.274 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:25.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:25.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:25.289 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T17:59:25.289 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T17:59:25.289 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-08-17T17:59:25.290 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-08-17T17:59:25.290 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T17:59:25.375 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:25.406 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:25.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:25.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:25.508 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:25.547 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:25.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:25.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:25.649 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:25.683 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:25.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:25.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:25.708 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 51 16 85758 85742 105.064 69.75 0.0100033 0.00951233 2024-08-17T17:59:25.785 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:25.806 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:25.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:25.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:25.908 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:25.941 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:25.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:25.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:26.043 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:26.080 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:26.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:26.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:26.094 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-17T17:59:26.182 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:26.212 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:26.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:26.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:26.314 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:26.339 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:26.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:26.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:26.442 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:26.513 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:59:26.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:26.514 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:26.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:26.615 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:26.665 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:26.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:26.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:26.708 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 52 16 86619 86603 104.079 53.8125 0.0163363 0.00960254 2024-08-17T17:59:26.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:26 smithi165 ceph-mon[105517]: pgmap v2335: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 57 MiB/s wr, 919 op/s; 4090/244524 objects misplaced (1.673%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:59:26.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:26 smithi165 ceph-mon[105517]: from='client.45314 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:59:26.766 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:26.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:26 smithi136 ceph-mon[109861]: pgmap v2335: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 57 MiB/s wr, 919 op/s; 4090/244524 objects misplaced (1.673%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:59:26.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:26 smithi136 ceph-mon[109861]: from='client.45314 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:59:26.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:26 smithi136 ceph-mon[115078]: pgmap v2335: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 57 MiB/s wr, 919 op/s; 4090/244524 objects misplaced (1.673%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:59:26.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:26 smithi136 ceph-mon[115078]: from='client.45314 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:59:26.817 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:26.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:26.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:26.827 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":2335,"stamp":"2024-08-17T17:59:24.715669+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5340600293,"num_objects":81508,"num_object_clones":0,"num_object_copies":244524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4090,"num_objects_unfound":0,"num_objects_dirty":81508,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":81761,"num_write_kb":5217297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":18025180,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81690,"ondisk_log_size":81690,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":268,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":17974828,"kb_used_data":15636212,"kb_used_omap":219,"kb_used_meta":2338340,"kb_avail":638241236,"statfs":{"total":671965249536,"available":653559025664,"internally_reserved":0,"allocated":16011481088,"data_stored":15981199469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224669,"internal_metadata":2394460771},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,8,6,6,6,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":723058688,"num_objects":11033,"num_object_clones":0,"num_object_copies":33099,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4090,"num_objects_unfound":0,"num_objects_dirty":11033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11033,"num_write_kb":706112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":239,"num_bytes_recovered":15663104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003423"},"pg_stats":[{"pgid":"54.f","version":"1378'5038","reported_seq":5077,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:20.737090+0000","last_change":"2024-08-17T17:58:19.551058+0000","last_active":"2024-08-17T17:59:20.737090+0000","last_peered":"2024-08-17T17:59:20.737090+0000","last_clean":"2024-08-17T17:59:20.737090+0000","last_became_active":"2024-08-17T17:58:19.550748+0000","last_became_peered":"2024-08-17T17:58:19.550748+0000","last_unstale":"2024-08-17T17:59:20.737090+0000","last_undegraded":"2024-08-17T17:59:20.737090+0000","last_fullsized":"2024-08-17T17:59:20.737090+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":5038,"log_dups_size":0,"ondisk_log_size":5038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:43:13.383674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330170368,"num_objects":5038,"num_object_clones":0,"num_object_copies":15114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5038,"num_write_kb":322432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.e","version":"1378'5181","reported_seq":5204,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:22.323271+0000","last_change":"2024-08-17T17:58:19.593455+0000","last_active":"2024-08-17T17:59:22.323271+0000","last_peered":"2024-08-17T17:59:22.323271+0000","last_clean":"2024-08-17T17:59:22.323271+0000","last_became_active":"2024-08-17T17:58:19.593144+0000","last_became_peered":"2024-08-17T17:58:19.593144+0000","last_unstale":"2024-08-17T17:59:22.323271+0000","last_undegraded":"2024-08-17T17:59:22.323271+0000","last_fullsized":"2024-08-17T17:59:22.323271+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":5181,"log_dups_size":0,"ondisk_log_size":5181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:08:45.701487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339542016,"num_objects":5181,"num_object_clones":0,"num_object_copies":15543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5181,"num_write_kb":331584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"54.d","version":"1378'5137","reported_seq":5169,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:22.097784+0000","last_change":"2024-08-17T17:58:23.582595+0000","last_active":"2024-08-17T17:59:22.097784+0000","last_peered":"2024-08-17T17:59:22.097784+0000","last_clean":"2024-08-17T17:59:22.097784+0000","last_became_active":"2024-08-17T17:58:23.582234+0000","last_became_peered":"2024-08-17T17:58:23.582234+0000","last_unstale":"2024-08-17T17:59:22.097784+0000","last_undegraded":"2024-08-17T17:59:22.097784+0000","last_fullsized":"2024-08-17T17:59:22.097784+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":5137,"log_dups_size":0,"ondisk_log_size":5137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:10:06.366767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336658432,"num_objects":5137,"num_object_clones":0,"num_object_copies":15411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5137,"num_write_kb":328768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"54.c","version":"1378'4929","reported_seq":5450,"reported_epoch":1378,"state":"active+recovering+remapped","last_fresh":"2024-08-17T17:59:20.780194+0000","last_change":"2024-08-17T17:59:12.058065+0000","last_active":"2024-08-17T17:59:20.780194+0000","last_peered":"2024-08-17T17:59:20.780194+0000","last_clean":"2024-08-17T17:59:09.884394+0000","last_became_active":"2024-08-17T17:59:11.993002+0000","last_became_peered":"2024-08-17T17:59:11.993002+0000","last_unstale":"2024-08-17T17:59:20.780194+0000","last_undegraded":"2024-08-17T17:59:20.780194+0000","last_fullsized":"2024-08-17T17:59:20.780194+0000","mapping_epoch":1377,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":4929,"log_dups_size":0,"ondisk_log_size":4929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323026944,"num_objects":4929,"num_object_clones":0,"num_object_copies":14787,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4090,"num_objects_unfound":0,"num_objects_dirty":4929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4929,"num_write_kb":315456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":239,"num_bytes_recovered":15663104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[3,5,4],"avail_no_missing":["3","4","5"],"object_location_counts":[{"shards":"3,4,5","objects":839}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.b","version":"1378'4983","reported_seq":5029,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:20.763688+0000","last_change":"2024-08-17T17:58:23.585551+0000","last_active":"2024-08-17T17:59:20.763688+0000","last_peered":"2024-08-17T17:59:20.763688+0000","last_clean":"2024-08-17T17:59:20.763688+0000","last_became_active":"2024-08-17T17:58:23.581936+0000","last_became_peered":"2024-08-17T17:58:23.581936+0000","last_unstale":"2024-08-17T17:59:20.763688+0000","last_undegraded":"2024-08-17T17:59:20.763688+0000","last_fullsized":"2024-08-17T17:59:20.763688+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":4983,"log_dups_size":0,"ondisk_log_size":4983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:07:29.720931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326565888,"num_objects":4983,"num_object_clones":0,"num_object_copies":14949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4983,"num_write_kb":318912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.a","version":"1378'5171","reported_seq":5194,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:22.081959+0000","last_change":"2024-08-17T17:58:19.551465+0000","last_active":"2024-08-17T17:59:22.081959+0000","last_peered":"2024-08-17T17:59:22.081959+0000","last_clean":"2024-08-17T17:59:22.081959+0000","last_became_active":"2024-08-17T17:58:19.551177+0000","last_became_peered":"2024-08-17T17:58:19.551177+0000","last_unstale":"2024-08-17T17:59:22.081959+0000","last_undegraded":"2024-08-17T17:59:22.081959+0000","last_fullsized":"2024-08-17T17:59:22.081959+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":5171,"log_dups_size":0,"ondisk_log_size":5171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:20:45.172117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338886656,"num_objects":5171,"num_object_clones":0,"num_object_copies":15513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5171,"num_write_kb":330944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"54.9","version":"1378'5212","reported_seq":5251,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:21.304682+0000","last_change":"2024-08-17T17:58:19.547271+0000","last_active":"2024-08-17T17:59:21.304682+0000","last_peered":"2024-08-17T17:59:21.304682+0000","last_clean":"2024-08-17T17:59:21.304682+0000","last_became_active":"2024-08-17T17:58:19.547003+0000","last_became_peered":"2024-08-17T17:58:19.547003+0000","last_unstale":"2024-08-17T17:59:21.304682+0000","last_undegraded":"2024-08-17T17:59:21.304682+0000","last_fullsized":"2024-08-17T17:59:21.304682+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":5212,"log_dups_size":0,"ondisk_log_size":5212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:08.876694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341573632,"num_objects":5212,"num_object_clones":0,"num_object_copies":15636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5212,"num_write_kb":333568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"54.8","version":"1378'5044","reported_seq":5067,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:22.311184+0000","last_change":"2024-08-17T17:58:19.549974+0000","last_active":"2024-08-17T17:59:22.311184+0000","last_peered":"2024-08-17T17:59:22.311184+0000","last_clean":"2024-08-17T17:59:22.311184+0000","last_became_active":"2024-08-17T17:58:19.549673+0000","last_became_peered":"2024-08-17T17:58:19.549673+0000","last_unstale":"2024-08-17T17:59:22.311184+0000","last_undegraded":"2024-08-17T17:59:22.311184+0000","last_fullsized":"2024-08-17T17:59:22.311184+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":5044,"log_dups_size":0,"ondisk_log_size":5044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:34:59.904300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330563584,"num_objects":5044,"num_object_clones":0,"num_object_copies":15132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5044,"num_write_kb":322816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"54.7","version":"1378'5067","reported_seq":5112,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:20.792145+0000","last_change":"2024-08-17T17:58:23.583414+0000","last_active":"2024-08-17T17:59:20.792145+0000","last_peered":"2024-08-17T17:59:20.792145+0000","last_clean":"2024-08-17T17:59:20.792145+0000","last_became_active":"2024-08-17T17:58:23.582261+0000","last_became_peered":"2024-08-17T17:58:23.582261+0000","last_unstale":"2024-08-17T17:59:20.792145+0000","last_undegraded":"2024-08-17T17:59:20.792145+0000","last_fullsized":"2024-08-17T17:59:20.792145+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":5067,"log_dups_size":0,"ondisk_log_size":5067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:32:05.634094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332070912,"num_objects":5067,"num_object_clones":0,"num_object_copies":15201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5067,"num_write_kb":324288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.6","version":"1378'5084","reported_seq":5106,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:21.779090+0000","last_change":"2024-08-17T17:58:19.593645+0000","last_active":"2024-08-17T17:59:21.779090+0000","last_peered":"2024-08-17T17:59:21.779090+0000","last_clean":"2024-08-17T17:59:21.779090+0000","last_became_active":"2024-08-17T17:58:19.593265+0000","last_became_peered":"2024-08-17T17:58:19.593265+0000","last_unstale":"2024-08-17T17:59:21.779090+0000","last_undegraded":"2024-08-17T17:59:21.779090+0000","last_fullsized":"2024-08-17T17:59:21.779090+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":5084,"log_dups_size":0,"ondisk_log_size":5084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:52:46.137037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333185024,"num_objects":5084,"num_object_clones":0,"num_object_copies":15252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5084,"num_write_kb":325376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"54.5","version":"1378'5105","reported_seq":5137,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:22.100759+0000","last_change":"2024-08-17T17:58:23.582940+0000","last_active":"2024-08-17T17:59:22.100759+0000","last_peered":"2024-08-17T17:59:22.100759+0000","last_clean":"2024-08-17T17:59:22.100759+0000","last_became_active":"2024-08-17T17:58:23.582647+0000","last_became_peered":"2024-08-17T17:58:23.582647+0000","last_unstale":"2024-08-17T17:59:22.100759+0000","last_undegraded":"2024-08-17T17:59:22.100759+0000","last_fullsized":"2024-08-17T17:59:22.100759+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":5105,"log_dups_size":0,"ondisk_log_size":5105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:32:22.678243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334561280,"num_objects":5105,"num_object_clones":0,"num_object_copies":15315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5105,"num_write_kb":326720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"54.4","version":"1378'5004","reported_seq":5043,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:20.702783+0000","last_change":"2024-08-17T17:58:19.549870+0000","last_active":"2024-08-17T17:59:20.702783+0000","last_peered":"2024-08-17T17:59:20.702783+0000","last_clean":"2024-08-17T17:59:20.702783+0000","last_became_active":"2024-08-17T17:58:19.549683+0000","last_became_peered":"2024-08-17T17:58:19.549683+0000","last_unstale":"2024-08-17T17:59:20.702783+0000","last_undegraded":"2024-08-17T17:59:20.702783+0000","last_fullsized":"2024-08-17T17:59:20.702783+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":5004,"log_dups_size":0,"ondisk_log_size":5004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:27:17.205900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327942144,"num_objects":5004,"num_object_clones":0,"num_object_copies":15012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5004,"num_write_kb":320256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.3","version":"1378'5188","reported_seq":5234,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:21.337500+0000","last_change":"2024-08-17T17:58:23.585059+0000","last_active":"2024-08-17T17:59:21.337500+0000","last_peered":"2024-08-17T17:59:21.337500+0000","last_clean":"2024-08-17T17:59:21.337500+0000","last_became_active":"2024-08-17T17:58:23.584745+0000","last_became_peered":"2024-08-17T17:58:23.584745+0000","last_unstale":"2024-08-17T17:59:21.337500+0000","last_undegraded":"2024-08-17T17:59:21.337500+0000","last_fullsized":"2024-08-17T17:59:21.337500+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":5188,"log_dups_size":0,"ondisk_log_size":5188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:39:09.866950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340000768,"num_objects":5188,"num_object_clones":0,"num_object_copies":15564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5188,"num_write_kb":332032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"54.2","version":"1378'5155","reported_seq":5177,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:22.198213+0000","last_change":"2024-08-17T17:58:23.582529+0000","last_active":"2024-08-17T17:59:22.198213+0000","last_peered":"2024-08-17T17:59:22.198213+0000","last_clean":"2024-08-17T17:59:22.198213+0000","last_became_active":"2024-08-17T17:58:23.582203+0000","last_became_peered":"2024-08-17T17:58:23.582203+0000","last_unstale":"2024-08-17T17:59:22.198213+0000","last_undegraded":"2024-08-17T17:59:22.198213+0000","last_fullsized":"2024-08-17T17:59:22.198213+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":5155,"log_dups_size":0,"ondisk_log_size":5155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:29:48.813331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337838080,"num_objects":5155,"num_object_clones":0,"num_object_copies":15465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5155,"num_write_kb":329920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"54.1","version":"1378'5079","reported_seq":5102,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:22.210829+0000","last_change":"2024-08-17T17:58:19.592487+0000","last_active":"2024-08-17T17:59:22.210829+0000","last_peered":"2024-08-17T17:59:22.210829+0000","last_clean":"2024-08-17T17:59:22.210829+0000","last_became_active":"2024-08-17T17:58:19.592298+0000","last_became_peered":"2024-08-17T17:58:19.592298+0000","last_unstale":"2024-08-17T17:59:22.210829+0000","last_undegraded":"2024-08-17T17:59:22.210829+0000","last_fullsized":"2024-08-17T17:59:22.210829+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":5079,"log_dups_size":0,"ondisk_log_size":5079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:04:19.878612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332857344,"num_objects":5079,"num_object_clones":0,"num_object_copies":15237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5079,"num_write_kb":325056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"54.0","version":"1378'5105","reported_seq":5128,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:22.050389+0000","last_change":"2024-08-17T17:58:19.548035+0000","last_active":"2024-08-17T17:59:22.050389+0000","last_peered":"2024-08-17T17:59:22.050389+0000","last_clean":"2024-08-17T17:59:22.050389+0000","last_became_active":"2024-08-17T17:58:19.547741+0000","last_became_peered":"2024-08-17T17:58:19.547741+0000","last_unstale":"2024-08-17T17:59:22.050389+0000","last_undegraded":"2024-08-17T17:59:22.050389+0000","last_fullsized":"2024-08-17T17:59:22.050389+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":5105,"log_dups_size":0,"ondisk_log_size":5105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:43:17.339560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334561280,"num_objects":5105,"num_object_clones":0,"num_object_copies":15315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5105,"num_write_kb":326720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":52,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.887032+0000","last_change":"2024-08-17T17:58:23.586005+0000","last_active":"2024-08-17T17:59:10.887032+0000","last_peered":"2024-08-17T17:59:10.887032+0000","last_clean":"2024-08-17T17:59:10.887032+0000","last_became_active":"2024-08-17T17:58:23.580706+0000","last_became_peered":"2024-08-17T17:58:23.580706+0000","last_unstale":"2024-08-17T17:59:10.887032+0000","last_undegraded":"2024-08-17T17:59:10.887032+0000","last_fullsized":"2024-08-17T17:59:10.887032+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:00:46.147301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1350,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.885103+0000","last_change":"2024-08-17T17:23:07.398676+0000","last_active":"2024-08-17T17:59:10.885103+0000","last_peered":"2024-08-17T17:59:10.885103+0000","last_clean":"2024-08-17T17:59:10.885103+0000","last_became_active":"2024-08-17T17:23:07.398369+0000","last_became_peered":"2024-08-17T17:23:07.398369+0000","last_unstale":"2024-08-17T17:59:10.885103+0000","last_undegraded":"2024-08-17T17:59:10.885103+0000","last_fullsized":"2024-08-17T17:59:10.885103+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:55:04.281412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":2046,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.886470+0000","last_change":"2024-08-17T17:24:17.476168+0000","last_active":"2024-08-17T17:59:11.886470+0000","last_peered":"2024-08-17T17:59:11.886470+0000","last_clean":"2024-08-17T17:59:11.886470+0000","last_became_active":"2024-08-17T17:24:17.465845+0000","last_became_peered":"2024-08-17T17:24:17.465845+0000","last_unstale":"2024-08-17T17:59:11.886470+0000","last_undegraded":"2024-08-17T17:59:11.886470+0000","last_fullsized":"2024-08-17T17:59:11.886470+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:15:01.494447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2069,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.885957+0000","last_change":"2024-08-17T17:58:23.585542+0000","last_active":"2024-08-17T17:59:11.885957+0000","last_peered":"2024-08-17T17:59:11.885957+0000","last_clean":"2024-08-17T17:59:11.885957+0000","last_became_active":"2024-08-17T17:58:23.584976+0000","last_became_peered":"2024-08-17T17:58:23.584976+0000","last_unstale":"2024-08-17T17:59:11.885957+0000","last_undegraded":"2024-08-17T17:59:11.885957+0000","last_fullsized":"2024-08-17T17:59:11.885957+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:46:31.440698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":1332,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.886959+0000","last_change":"2024-08-17T17:10:05.965176+0000","last_active":"2024-08-17T17:59:10.886959+0000","last_peered":"2024-08-17T17:59:10.886959+0000","last_clean":"2024-08-17T17:59:10.886959+0000","last_became_active":"2024-08-17T17:10:05.964861+0000","last_became_peered":"2024-08-17T17:10:05.964861+0000","last_unstale":"2024-08-17T17:59:10.886959+0000","last_undegraded":"2024-08-17T17:59:10.886959+0000","last_fullsized":"2024-08-17T17:59:10.886959+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:59:04.277926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1373,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.885333+0000","last_change":"2024-08-17T17:58:23.582593+0000","last_active":"2024-08-17T17:59:10.885333+0000","last_peered":"2024-08-17T17:59:10.885333+0000","last_clean":"2024-08-17T17:59:10.885333+0000","last_became_active":"2024-08-17T17:58:23.582276+0000","last_became_peered":"2024-08-17T17:58:23.582276+0000","last_unstale":"2024-08-17T17:59:10.885333+0000","last_undegraded":"2024-08-17T17:59:10.885333+0000","last_fullsized":"2024-08-17T17:59:10.885333+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:04:27.806072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":1347,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.886784+0000","last_change":"2024-08-17T17:23:07.391038+0000","last_active":"2024-08-17T17:59:10.886784+0000","last_peered":"2024-08-17T17:59:10.886784+0000","last_clean":"2024-08-17T17:59:10.886784+0000","last_became_active":"2024-08-17T17:23:07.390571+0000","last_became_peered":"2024-08-17T17:23:07.390571+0000","last_unstale":"2024-08-17T17:59:10.886784+0000","last_undegraded":"2024-08-17T17:59:10.886784+0000","last_fullsized":"2024-08-17T17:59:10.886784+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:18:41.791740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1364,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.885500+0000","last_change":"2024-08-17T17:24:17.458168+0000","last_active":"2024-08-17T17:59:10.885500+0000","last_peered":"2024-08-17T17:59:10.885500+0000","last_clean":"2024-08-17T17:59:10.885500+0000","last_became_active":"2024-08-17T17:24:17.457847+0000","last_became_peered":"2024-08-17T17:24:17.457847+0000","last_unstale":"2024-08-17T17:59:10.885500+0000","last_undegraded":"2024-08-17T17:59:10.885500+0000","last_fullsized":"2024-08-17T17:59:10.885500+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:28.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":34,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.885391+0000","last_change":"2024-08-17T17:58:23.579378+0000","last_active":"2024-08-17T17:59:10.885391+0000","last_peered":"2024-08-17T17:59:10.885391+0000","last_clean":"2024-08-17T17:59:10.885391+0000","last_became_active":"2024-08-17T17:58:23.579006+0000","last_became_peered":"2024-08-17T17:58:23.579006+0000","last_unstale":"2024-08-17T17:59:10.885391+0000","last_undegraded":"2024-08-17T17:59:10.885391+0000","last_fullsized":"2024-08-17T17:59:10.885391+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:45:42.529376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":2063,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.883126+0000","last_change":"2024-08-17T17:24:17.456622+0000","last_active":"2024-08-17T17:59:11.883126+0000","last_peered":"2024-08-17T17:59:11.883126+0000","last_clean":"2024-08-17T17:59:11.883126+0000","last_became_active":"2024-08-17T17:24:17.455396+0000","last_became_peered":"2024-08-17T17:24:17.455396+0000","last_unstale":"2024-08-17T17:59:11.883126+0000","last_undegraded":"2024-08-17T17:59:11.883126+0000","last_fullsized":"2024-08-17T17:59:11.883126+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:09.564331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1366,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.885465+0000","last_change":"2024-08-17T17:23:42.930464+0000","last_active":"2024-08-17T17:59:10.885465+0000","last_peered":"2024-08-17T17:59:10.885465+0000","last_clean":"2024-08-17T17:59:10.885465+0000","last_became_active":"2024-08-17T17:23:42.929949+0000","last_became_peered":"2024-08-17T17:23:42.929949+0000","last_unstale":"2024-08-17T17:59:10.885465+0000","last_undegraded":"2024-08-17T17:59:10.885465+0000","last_fullsized":"2024-08-17T17:59:10.885465+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:48:27.092339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2049,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.886571+0000","last_change":"2024-08-17T17:24:17.452009+0000","last_active":"2024-08-17T17:59:11.886571+0000","last_peered":"2024-08-17T17:59:11.886571+0000","last_clean":"2024-08-17T17:59:11.886571+0000","last_became_active":"2024-08-17T17:24:17.451492+0000","last_became_peered":"2024-08-17T17:24:17.451492+0000","last_unstale":"2024-08-17T17:59:11.886571+0000","last_undegraded":"2024-08-17T17:59:11.886571+0000","last_fullsized":"2024-08-17T17:59:11.886571+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:21:24.699633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2060,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.883309+0000","last_change":"2024-08-17T17:24:17.452025+0000","last_active":"2024-08-17T17:59:11.883309+0000","last_peered":"2024-08-17T17:59:11.883309+0000","last_clean":"2024-08-17T17:59:11.883309+0000","last_became_active":"2024-08-17T17:24:17.448029+0000","last_became_peered":"2024-08-17T17:24:17.448029+0000","last_unstale":"2024-08-17T17:59:11.883309+0000","last_undegraded":"2024-08-17T17:59:11.883309+0000","last_fullsized":"2024-08-17T17:59:11.883309+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:45.089346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":686,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.886565+0000","last_change":"2024-08-17T17:58:23.586641+0000","last_active":"2024-08-17T17:59:11.886565+0000","last_peered":"2024-08-17T17:59:11.886565+0000","last_clean":"2024-08-17T17:59:11.886565+0000","last_became_active":"2024-08-17T17:58:23.585843+0000","last_became_peered":"2024-08-17T17:58:23.585843+0000","last_unstale":"2024-08-17T17:59:11.886565+0000","last_undegraded":"2024-08-17T17:59:11.886565+0000","last_fullsized":"2024-08-17T17:59:11.886565+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:34:00.228118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2056,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.885670+0000","last_change":"2024-08-17T17:58:23.584638+0000","last_active":"2024-08-17T17:59:11.885670+0000","last_peered":"2024-08-17T17:59:11.885670+0000","last_clean":"2024-08-17T17:59:11.885670+0000","last_became_active":"2024-08-17T17:58:23.583535+0000","last_became_peered":"2024-08-17T17:58:23.583535+0000","last_unstale":"2024-08-17T17:59:11.885670+0000","last_undegraded":"2024-08-17T17:59:11.885670+0000","last_fullsized":"2024-08-17T17:59:11.885670+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:00:25.957667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1350,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.567562+0000","last_change":"2024-08-17T17:23:07.402338+0000","last_active":"2024-08-17T17:59:11.567562+0000","last_peered":"2024-08-17T17:59:11.567562+0000","last_clean":"2024-08-17T17:59:11.567562+0000","last_became_active":"2024-08-17T17:23:07.401996+0000","last_became_peered":"2024-08-17T17:23:07.401996+0000","last_unstale":"2024-08-17T17:59:11.567562+0000","last_undegraded":"2024-08-17T17:59:11.567562+0000","last_fullsized":"2024-08-17T17:59:11.567562+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:39:21.566786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2034,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.885808+0000","last_change":"2024-08-17T17:24:17.442338+0000","last_active":"2024-08-17T17:59:11.885808+0000","last_peered":"2024-08-17T17:59:11.885808+0000","last_clean":"2024-08-17T17:59:11.885808+0000","last_became_active":"2024-08-17T17:24:17.441948+0000","last_became_peered":"2024-08-17T17:24:17.441948+0000","last_unstale":"2024-08-17T17:59:11.885808+0000","last_undegraded":"2024-08-17T17:59:11.885808+0000","last_fullsized":"2024-08-17T17:59:11.885808+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:17.891492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":674,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.887115+0000","last_change":"2024-08-17T17:58:23.589701+0000","last_active":"2024-08-17T17:59:10.887115+0000","last_peered":"2024-08-17T17:59:10.887115+0000","last_clean":"2024-08-17T17:59:10.887115+0000","last_became_active":"2024-08-17T17:58:23.589393+0000","last_became_peered":"2024-08-17T17:58:23.589393+0000","last_unstale":"2024-08-17T17:59:10.887115+0000","last_undegraded":"2024-08-17T17:59:10.887115+0000","last_fullsized":"2024-08-17T17:59:10.887115+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:41:03.356477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1993,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.886135+0000","last_change":"2024-08-17T17:24:17.468086+0000","last_active":"2024-08-17T17:59:11.886135+0000","last_peered":"2024-08-17T17:59:11.886135+0000","last_clean":"2024-08-17T17:59:11.886135+0000","last_became_active":"2024-08-17T17:24:17.467522+0000","last_became_peered":"2024-08-17T17:24:17.467522+0000","last_unstale":"2024-08-17T17:59:11.886135+0000","last_undegraded":"2024-08-17T17:59:11.886135+0000","last_fullsized":"2024-08-17T17:59:11.886135+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:52:55.249649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1346,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.886794+0000","last_change":"2024-08-17T17:23:07.392447+0000","last_active":"2024-08-17T17:59:10.886794+0000","last_peered":"2024-08-17T17:59:10.886794+0000","last_clean":"2024-08-17T17:59:10.886794+0000","last_became_active":"2024-08-17T17:23:07.391765+0000","last_became_peered":"2024-08-17T17:23:07.391765+0000","last_unstale":"2024-08-17T17:59:10.886794+0000","last_undegraded":"2024-08-17T17:59:10.886794+0000","last_fullsized":"2024-08-17T17:59:10.886794+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:59:18.602346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":145,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.886619+0000","last_change":"2024-08-17T17:53:58.839889+0000","last_active":"2024-08-17T17:59:11.886619+0000","last_peered":"2024-08-17T17:59:11.886619+0000","last_clean":"2024-08-17T17:59:11.886619+0000","last_became_active":"2024-08-17T17:53:58.839537+0000","last_became_peered":"2024-08-17T17:53:58.839537+0000","last_unstale":"2024-08-17T17:59:11.886619+0000","last_undegraded":"2024-08-17T17:59:11.886619+0000","last_fullsized":"2024-08-17T17:59:11.886619+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:47.169954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2060,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.883707+0000","last_change":"2024-08-17T17:23:42.936147+0000","last_active":"2024-08-17T17:59:11.883707+0000","last_peered":"2024-08-17T17:59:11.883707+0000","last_clean":"2024-08-17T17:59:11.883707+0000","last_became_active":"2024-08-17T17:23:42.935859+0000","last_became_peered":"2024-08-17T17:23:42.935859+0000","last_unstale":"2024-08-17T17:59:11.883707+0000","last_undegraded":"2024-08-17T17:59:11.883707+0000","last_fullsized":"2024-08-17T17:59:11.883707+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:47:36.838706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1360,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.909249+0000","last_change":"2024-08-17T17:10:05.969131+0000","last_active":"2024-08-17T17:59:11.909249+0000","last_peered":"2024-08-17T17:59:11.909249+0000","last_clean":"2024-08-17T17:59:11.909249+0000","last_became_active":"2024-08-17T17:10:05.968856+0000","last_became_peered":"2024-08-17T17:10:05.968856+0000","last_unstale":"2024-08-17T17:59:11.909249+0000","last_undegraded":"2024-08-17T17:59:11.909249+0000","last_fullsized":"2024-08-17T17:59:11.909249+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:25:49.550065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1351,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.908646+0000","last_change":"2024-08-17T17:23:07.455919+0000","last_active":"2024-08-17T17:59:11.908646+0000","last_peered":"2024-08-17T17:59:11.908646+0000","last_clean":"2024-08-17T17:59:11.908646+0000","last_became_active":"2024-08-17T17:23:07.455284+0000","last_became_peered":"2024-08-17T17:23:07.455284+0000","last_unstale":"2024-08-17T17:59:11.908646+0000","last_undegraded":"2024-08-17T17:59:11.908646+0000","last_fullsized":"2024-08-17T17:59:11.908646+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:37:46.260775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1338,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.885605+0000","last_change":"2024-08-17T17:10:06.979492+0000","last_active":"2024-08-17T17:59:10.885605+0000","last_peered":"2024-08-17T17:59:10.885605+0000","last_clean":"2024-08-17T17:59:10.885605+0000","last_became_active":"2024-08-17T17:10:06.979158+0000","last_became_peered":"2024-08-17T17:10:06.979158+0000","last_unstale":"2024-08-17T17:59:10.885605+0000","last_undegraded":"2024-08-17T17:59:10.885605+0000","last_fullsized":"2024-08-17T17:59:10.885605+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:12:42.261365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1363,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.566816+0000","last_change":"2024-08-17T17:23:42.935099+0000","last_active":"2024-08-17T17:59:11.566816+0000","last_peered":"2024-08-17T17:59:11.566816+0000","last_clean":"2024-08-17T17:59:11.566816+0000","last_became_active":"2024-08-17T17:23:42.934501+0000","last_became_peered":"2024-08-17T17:23:42.934501+0000","last_unstale":"2024-08-17T17:59:11.566816+0000","last_undegraded":"2024-08-17T17:59:11.566816+0000","last_fullsized":"2024-08-17T17:59:11.566816+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:15:19.095713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":61,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.883912+0000","last_change":"2024-08-17T17:58:23.586651+0000","last_active":"2024-08-17T17:59:11.883912+0000","last_peered":"2024-08-17T17:59:11.883912+0000","last_clean":"2024-08-17T17:59:11.883912+0000","last_became_active":"2024-08-17T17:58:23.580712+0000","last_became_peered":"2024-08-17T17:58:23.580712+0000","last_unstale":"2024-08-17T17:59:11.883912+0000","last_undegraded":"2024-08-17T17:59:11.883912+0000","last_fullsized":"2024-08-17T17:59:11.883912+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:40:33.674413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2043,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.885723+0000","last_change":"2024-08-17T17:59:10.888817+0000","last_active":"2024-08-17T17:59:11.885723+0000","last_peered":"2024-08-17T17:59:11.885723+0000","last_clean":"2024-08-17T17:59:11.885723+0000","last_became_active":"2024-08-17T17:59:10.888483+0000","last_became_peered":"2024-08-17T17:59:10.888483+0000","last_unstale":"2024-08-17T17:59:11.885723+0000","last_undegraded":"2024-08-17T17:59:11.885723+0000","last_fullsized":"2024-08-17T17:59:11.885723+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:26.369367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2042,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.884012+0000","last_change":"2024-08-17T17:23:07.428296+0000","last_active":"2024-08-17T17:59:11.884012+0000","last_peered":"2024-08-17T17:59:11.884012+0000","last_clean":"2024-08-17T17:59:11.884012+0000","last_became_active":"2024-08-17T17:23:07.428010+0000","last_became_peered":"2024-08-17T17:23:07.428010+0000","last_unstale":"2024-08-17T17:59:11.884012+0000","last_undegraded":"2024-08-17T17:59:11.884012+0000","last_fullsized":"2024-08-17T17:59:11.884012+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:45.459212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":2041,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.886759+0000","last_change":"2024-08-17T17:58:23.594285+0000","last_active":"2024-08-17T17:59:11.886759+0000","last_peered":"2024-08-17T17:59:11.886759+0000","last_clean":"2024-08-17T17:59:11.886759+0000","last_became_active":"2024-08-17T17:58:23.588147+0000","last_became_peered":"2024-08-17T17:58:23.588147+0000","last_unstale":"2024-08-17T17:59:11.886759+0000","last_undegraded":"2024-08-17T17:59:11.886759+0000","last_fullsized":"2024-08-17T17:59:11.886759+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:12:40.216512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":609,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.886878+0000","last_change":"2024-08-17T17:58:23.579834+0000","last_active":"2024-08-17T17:59:10.886878+0000","last_peered":"2024-08-17T17:59:10.886878+0000","last_clean":"2024-08-17T17:59:10.886878+0000","last_became_active":"2024-08-17T17:58:23.579507+0000","last_became_peered":"2024-08-17T17:58:23.579507+0000","last_unstale":"2024-08-17T17:59:10.886878+0000","last_undegraded":"2024-08-17T17:59:10.886878+0000","last_fullsized":"2024-08-17T17:59:10.886878+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:43:24.177069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1368,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.882974+0000","last_change":"2024-08-17T17:58:23.582011+0000","last_active":"2024-08-17T17:59:10.882974+0000","last_peered":"2024-08-17T17:59:10.882974+0000","last_clean":"2024-08-17T17:59:10.882974+0000","last_became_active":"2024-08-17T17:58:23.581649+0000","last_became_peered":"2024-08-17T17:58:23.581649+0000","last_unstale":"2024-08-17T17:59:10.882974+0000","last_undegraded":"2024-08-17T17:59:10.882974+0000","last_fullsized":"2024-08-17T17:59:10.882974+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:38:55.478552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2064,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.884030+0000","last_change":"2024-08-17T17:23:42.935825+0000","last_active":"2024-08-17T17:59:11.884030+0000","last_peered":"2024-08-17T17:59:11.884030+0000","last_clean":"2024-08-17T17:59:11.884030+0000","last_became_active":"2024-08-17T17:23:42.935150+0000","last_became_peered":"2024-08-17T17:23:42.935150+0000","last_unstale":"2024-08-17T17:59:11.884030+0000","last_undegraded":"2024-08-17T17:59:11.884030+0000","last_fullsized":"2024-08-17T17:59:11.884030+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:57:19.577644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.883186+0000","last_change":"2024-08-17T17:59:10.888598+0000","last_active":"2024-08-17T17:59:11.883186+0000","last_peered":"2024-08-17T17:59:11.883186+0000","last_clean":"2024-08-17T17:59:11.883186+0000","last_became_active":"2024-08-17T17:59:10.888289+0000","last_became_peered":"2024-08-17T17:59:10.888289+0000","last_unstale":"2024-08-17T17:59:11.883186+0000","last_undegraded":"2024-08-17T17:59:11.883186+0000","last_fullsized":"2024-08-17T17:59:11.883186+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:34:39.372046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2087,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.883866+0000","last_change":"2024-08-17T17:58:23.581351+0000","last_active":"2024-08-17T17:59:11.883866+0000","last_peered":"2024-08-17T17:59:11.883866+0000","last_clean":"2024-08-17T17:59:11.883866+0000","last_became_active":"2024-08-17T17:58:23.580717+0000","last_became_peered":"2024-08-17T17:58:23.580717+0000","last_unstale":"2024-08-17T17:59:11.883866+0000","last_undegraded":"2024-08-17T17:59:11.883866+0000","last_fullsized":"2024-08-17T17:59:11.883866+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:02:47.291227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1364,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.887090+0000","last_change":"2024-08-17T17:24:17.426267+0000","last_active":"2024-08-17T17:59:10.887090+0000","last_peered":"2024-08-17T17:59:10.887090+0000","last_clean":"2024-08-17T17:59:10.887090+0000","last_became_active":"2024-08-17T17:24:17.425925+0000","last_became_peered":"2024-08-17T17:24:17.425925+0000","last_unstale":"2024-08-17T17:59:10.887090+0000","last_undegraded":"2024-08-17T17:59:10.887090+0000","last_fullsized":"2024-08-17T17:59:10.887090+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:24:25.017856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1360,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.886662+0000","last_change":"2024-08-17T17:24:17.474138+0000","last_active":"2024-08-17T17:59:10.886662+0000","last_peered":"2024-08-17T17:59:10.886662+0000","last_clean":"2024-08-17T17:59:10.886662+0000","last_became_active":"2024-08-17T17:24:17.473708+0000","last_became_peered":"2024-08-17T17:24:17.473708+0000","last_unstale":"2024-08-17T17:59:10.886662+0000","last_undegraded":"2024-08-17T17:59:10.886662+0000","last_fullsized":"2024-08-17T17:59:10.886662+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:10:57.768067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1363,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.885645+0000","last_change":"2024-08-17T17:58:23.575900+0000","last_active":"2024-08-17T17:59:10.885645+0000","last_peered":"2024-08-17T17:59:10.885645+0000","last_clean":"2024-08-17T17:59:10.885645+0000","last_became_active":"2024-08-17T17:58:23.575606+0000","last_became_peered":"2024-08-17T17:58:23.575606+0000","last_unstale":"2024-08-17T17:59:10.885645+0000","last_undegraded":"2024-08-17T17:59:10.885645+0000","last_fullsized":"2024-08-17T17:59:10.885645+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:17:46.057413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":1364,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.886578+0000","last_change":"2024-08-17T17:23:07.391021+0000","last_active":"2024-08-17T17:59:10.886578+0000","last_peered":"2024-08-17T17:59:10.886578+0000","last_clean":"2024-08-17T17:59:10.886578+0000","last_became_active":"2024-08-17T17:23:07.390551+0000","last_became_peered":"2024-08-17T17:23:07.390551+0000","last_unstale":"2024-08-17T17:59:10.886578+0000","last_undegraded":"2024-08-17T17:59:10.886578+0000","last_fullsized":"2024-08-17T17:59:10.886578+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:06.239013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001023532,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2070,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.884285+0000","last_change":"2024-08-17T17:58:23.580871+0000","last_active":"2024-08-17T17:59:11.884285+0000","last_peered":"2024-08-17T17:59:11.884285+0000","last_clean":"2024-08-17T17:59:11.884285+0000","last_became_active":"2024-08-17T17:58:23.580542+0000","last_became_peered":"2024-08-17T17:58:23.580542+0000","last_unstale":"2024-08-17T17:59:11.884285+0000","last_undegraded":"2024-08-17T17:59:11.884285+0000","last_fullsized":"2024-08-17T17:59:11.884285+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:51:14.019944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2044,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.883135+0000","last_change":"2024-08-17T17:23:42.910246+0000","last_active":"2024-08-17T17:59:11.883135+0000","last_peered":"2024-08-17T17:59:11.883135+0000","last_clean":"2024-08-17T17:59:11.883135+0000","last_became_active":"2024-08-17T17:23:42.909929+0000","last_became_peered":"2024-08-17T17:23:42.909929+0000","last_unstale":"2024-08-17T17:59:11.883135+0000","last_undegraded":"2024-08-17T17:59:11.883135+0000","last_fullsized":"2024-08-17T17:59:11.883135+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:14:45.210558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2079,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.883067+0000","last_change":"2024-08-17T17:58:23.586426+0000","last_active":"2024-08-17T17:59:11.883067+0000","last_peered":"2024-08-17T17:59:11.883067+0000","last_clean":"2024-08-17T17:59:11.883067+0000","last_became_active":"2024-08-17T17:58:23.585776+0000","last_became_peered":"2024-08-17T17:58:23.585776+0000","last_unstale":"2024-08-17T17:59:11.883067+0000","last_undegraded":"2024-08-17T17:59:11.883067+0000","last_fullsized":"2024-08-17T17:59:11.883067+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:39:33.064885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":673,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.883868+0000","last_change":"2024-08-17T17:58:23.582327+0000","last_active":"2024-08-17T17:59:11.883868+0000","last_peered":"2024-08-17T17:59:11.883868+0000","last_clean":"2024-08-17T17:59:11.883868+0000","last_became_active":"2024-08-17T17:58:23.581984+0000","last_became_peered":"2024-08-17T17:58:23.581984+0000","last_unstale":"2024-08-17T17:59:11.883868+0000","last_undegraded":"2024-08-17T17:59:11.883868+0000","last_fullsized":"2024-08-17T17:59:11.883868+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:10:23.678071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":675,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.885044+0000","last_change":"2024-08-17T17:58:23.583104+0000","last_active":"2024-08-17T17:59:10.885044+0000","last_peered":"2024-08-17T17:59:10.885044+0000","last_clean":"2024-08-17T17:59:10.885044+0000","last_became_active":"2024-08-17T17:58:23.582784+0000","last_became_peered":"2024-08-17T17:58:23.582784+0000","last_unstale":"2024-08-17T17:59:10.885044+0000","last_undegraded":"2024-08-17T17:59:10.885044+0000","last_fullsized":"2024-08-17T17:59:10.885044+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:39:37.921577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":784,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.883295+0000","last_change":"2024-08-17T17:58:23.585368+0000","last_active":"2024-08-17T17:59:11.883295+0000","last_peered":"2024-08-17T17:59:11.883295+0000","last_clean":"2024-08-17T17:59:11.883295+0000","last_became_active":"2024-08-17T17:58:23.584864+0000","last_became_peered":"2024-08-17T17:58:23.584864+0000","last_unstale":"2024-08-17T17:59:11.883295+0000","last_undegraded":"2024-08-17T17:59:11.883295+0000","last_fullsized":"2024-08-17T17:59:11.883295+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:51.734726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1335,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.566787+0000","last_change":"2024-08-17T17:10:06.995023+0000","last_active":"2024-08-17T17:59:11.566787+0000","last_peered":"2024-08-17T17:59:11.566787+0000","last_clean":"2024-08-17T17:59:11.566787+0000","last_became_active":"2024-08-17T17:10:06.994757+0000","last_became_peered":"2024-08-17T17:10:06.994757+0000","last_unstale":"2024-08-17T17:59:11.566787+0000","last_undegraded":"2024-08-17T17:59:11.566787+0000","last_fullsized":"2024-08-17T17:59:11.566787+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:31:52.163518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1995,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.886223+0000","last_change":"2024-08-17T17:24:17.468443+0000","last_active":"2024-08-17T17:59:11.886223+0000","last_peered":"2024-08-17T17:59:11.886223+0000","last_clean":"2024-08-17T17:59:11.886223+0000","last_became_active":"2024-08-17T17:24:17.467394+0000","last_became_peered":"2024-08-17T17:24:17.467394+0000","last_unstale":"2024-08-17T17:59:11.886223+0000","last_undegraded":"2024-08-17T17:59:11.886223+0000","last_fullsized":"2024-08-17T17:59:11.886223+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:13.024253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":1363,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.887000+0000","last_change":"2024-08-17T17:23:42.920552+0000","last_active":"2024-08-17T17:59:10.887000+0000","last_peered":"2024-08-17T17:59:10.887000+0000","last_clean":"2024-08-17T17:59:10.887000+0000","last_became_active":"2024-08-17T17:23:42.915824+0000","last_became_peered":"2024-08-17T17:23:42.915824+0000","last_unstale":"2024-08-17T17:59:10.887000+0000","last_undegraded":"2024-08-17T17:59:10.887000+0000","last_fullsized":"2024-08-17T17:59:10.887000+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:25:18.769504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041737399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":650,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.567199+0000","last_change":"2024-08-17T17:58:26.598586+0000","last_active":"2024-08-17T17:59:11.567199+0000","last_peered":"2024-08-17T17:59:11.567199+0000","last_clean":"2024-08-17T17:59:11.567199+0000","last_became_active":"2024-08-17T17:58:26.598163+0000","last_became_peered":"2024-08-17T17:58:26.598163+0000","last_unstale":"2024-08-17T17:59:11.567199+0000","last_undegraded":"2024-08-17T17:59:11.567199+0000","last_fullsized":"2024-08-17T17:59:11.567199+0000","mapping_epoch":1370,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T17:06:21.174471+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:36:48.182691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2042,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.884310+0000","last_change":"2024-08-17T17:24:17.455710+0000","last_active":"2024-08-17T17:59:11.884310+0000","last_peered":"2024-08-17T17:59:11.884310+0000","last_clean":"2024-08-17T17:59:11.884310+0000","last_became_active":"2024-08-17T17:24:17.452989+0000","last_became_peered":"2024-08-17T17:24:17.452989+0000","last_unstale":"2024-08-17T17:59:11.884310+0000","last_undegraded":"2024-08-17T17:59:11.884310+0000","last_fullsized":"2024-08-17T17:59:11.884310+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:58:07.894968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2036,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.885607+0000","last_change":"2024-08-17T17:24:17.453639+0000","last_active":"2024-08-17T17:59:11.885607+0000","last_peered":"2024-08-17T17:59:11.885607+0000","last_clean":"2024-08-17T17:59:11.885607+0000","last_became_active":"2024-08-17T17:24:17.443638+0000","last_became_peered":"2024-08-17T17:24:17.443638+0000","last_unstale":"2024-08-17T17:59:11.885607+0000","last_undegraded":"2024-08-17T17:59:11.885607+0000","last_fullsized":"2024-08-17T17:59:11.885607+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:42.107479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1351,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.886141+0000","last_change":"2024-08-17T17:23:07.451866+0000","last_active":"2024-08-17T17:59:10.886141+0000","last_peered":"2024-08-17T17:59:10.886141+0000","last_clean":"2024-08-17T17:59:10.886141+0000","last_became_active":"2024-08-17T17:23:07.451553+0000","last_became_peered":"2024-08-17T17:23:07.451553+0000","last_unstale":"2024-08-17T17:59:10.886141+0000","last_undegraded":"2024-08-17T17:59:10.886141+0000","last_fullsized":"2024-08-17T17:59:10.886141+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:37:23.930651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1353,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.885693+0000","last_change":"2024-08-17T17:23:42.915851+0000","last_active":"2024-08-17T17:59:10.885693+0000","last_peered":"2024-08-17T17:59:10.885693+0000","last_clean":"2024-08-17T17:59:10.885693+0000","last_became_active":"2024-08-17T17:23:42.915552+0000","last_became_peered":"2024-08-17T17:23:42.915552+0000","last_unstale":"2024-08-17T17:59:10.885693+0000","last_undegraded":"2024-08-17T17:59:10.885693+0000","last_fullsized":"2024-08-17T17:59:10.885693+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:45.197876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1366,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.885449+0000","last_change":"2024-08-17T17:23:42.930803+0000","last_active":"2024-08-17T17:59:10.885449+0000","last_peered":"2024-08-17T17:59:10.885449+0000","last_clean":"2024-08-17T17:59:10.885449+0000","last_became_active":"2024-08-17T17:23:42.930458+0000","last_became_peered":"2024-08-17T17:23:42.930458+0000","last_unstale":"2024-08-17T17:59:10.885449+0000","last_undegraded":"2024-08-17T17:59:10.885449+0000","last_fullsized":"2024-08-17T17:59:10.885449+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:40.816706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1350,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.567380+0000","last_change":"2024-08-17T17:23:07.401624+0000","last_active":"2024-08-17T17:59:11.567380+0000","last_peered":"2024-08-17T17:59:11.567380+0000","last_clean":"2024-08-17T17:59:11.567380+0000","last_became_active":"2024-08-17T17:23:07.400222+0000","last_became_peered":"2024-08-17T17:23:07.400222+0000","last_unstale":"2024-08-17T17:59:11.567380+0000","last_undegraded":"2024-08-17T17:59:11.567380+0000","last_fullsized":"2024-08-17T17:59:11.567380+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:07.069989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1351,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.885922+0000","last_change":"2024-08-17T17:24:17.470809+0000","last_active":"2024-08-17T17:59:10.885922+0000","last_peered":"2024-08-17T17:59:10.885922+0000","last_clean":"2024-08-17T17:59:10.885922+0000","last_became_active":"2024-08-17T17:24:17.469753+0000","last_became_peered":"2024-08-17T17:24:17.469753+0000","last_unstale":"2024-08-17T17:59:10.885922+0000","last_undegraded":"2024-08-17T17:59:10.885922+0000","last_fullsized":"2024-08-17T17:59:10.885922+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:00.049425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":2040,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.886452+0000","last_change":"2024-08-17T17:58:23.592069+0000","last_active":"2024-08-17T17:59:11.886452+0000","last_peered":"2024-08-17T17:59:11.886452+0000","last_clean":"2024-08-17T17:59:11.886452+0000","last_became_active":"2024-08-17T17:58:23.584024+0000","last_became_peered":"2024-08-17T17:58:23.584024+0000","last_unstale":"2024-08-17T17:59:11.886452+0000","last_undegraded":"2024-08-17T17:59:11.886452+0000","last_fullsized":"2024-08-17T17:59:11.886452+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:37:17.459578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":676,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.883069+0000","last_change":"2024-08-17T17:58:23.581996+0000","last_active":"2024-08-17T17:59:11.883069+0000","last_peered":"2024-08-17T17:59:11.883069+0000","last_clean":"2024-08-17T17:59:11.883069+0000","last_became_active":"2024-08-17T17:58:23.580469+0000","last_became_peered":"2024-08-17T17:58:23.580469+0000","last_unstale":"2024-08-17T17:59:11.883069+0000","last_undegraded":"2024-08-17T17:59:11.883069+0000","last_fullsized":"2024-08-17T17:59:11.883069+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:01:28.612769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1347,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.886829+0000","last_change":"2024-08-17T17:24:17.473913+0000","last_active":"2024-08-17T17:59:10.886829+0000","last_peered":"2024-08-17T17:59:10.886829+0000","last_clean":"2024-08-17T17:59:10.886829+0000","last_became_active":"2024-08-17T17:24:17.473195+0000","last_became_peered":"2024-08-17T17:24:17.473195+0000","last_unstale":"2024-08-17T17:59:10.886829+0000","last_undegraded":"2024-08-17T17:59:10.886829+0000","last_fullsized":"2024-08-17T17:59:10.886829+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:12.998033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1349,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.567149+0000","last_change":"2024-08-17T17:23:07.402010+0000","last_active":"2024-08-17T17:59:11.567149+0000","last_peered":"2024-08-17T17:59:11.567149+0000","last_clean":"2024-08-17T17:59:11.567149+0000","last_became_active":"2024-08-17T17:23:07.400849+0000","last_became_peered":"2024-08-17T17:23:07.400849+0000","last_unstale":"2024-08-17T17:59:11.567149+0000","last_undegraded":"2024-08-17T17:59:11.567149+0000","last_fullsized":"2024-08-17T17:59:11.567149+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:16:02.382257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":85,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.886928+0000","last_change":"2024-08-17T17:53:58.836811+0000","last_active":"2024-08-17T17:59:10.886928+0000","last_peered":"2024-08-17T17:59:10.886928+0000","last_clean":"2024-08-17T17:59:10.886928+0000","last_became_active":"2024-08-17T17:53:58.836139+0000","last_became_peered":"2024-08-17T17:53:58.836139+0000","last_unstale":"2024-08-17T17:59:10.886928+0000","last_undegraded":"2024-08-17T17:59:10.886928+0000","last_fullsized":"2024-08-17T17:59:10.886928+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:09:18.260164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1367,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.885016+0000","last_change":"2024-08-17T17:24:17.460203+0000","last_active":"2024-08-17T17:59:10.885016+0000","last_peered":"2024-08-17T17:59:10.885016+0000","last_clean":"2024-08-17T17:59:10.885016+0000","last_became_active":"2024-08-17T17:24:17.459682+0000","last_became_peered":"2024-08-17T17:24:17.459682+0000","last_unstale":"2024-08-17T17:59:10.885016+0000","last_undegraded":"2024-08-17T17:59:10.885016+0000","last_fullsized":"2024-08-17T17:59:10.885016+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:58.703700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":1346,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.883114+0000","last_change":"2024-08-17T17:23:07.454042+0000","last_active":"2024-08-17T17:59:10.883114+0000","last_peered":"2024-08-17T17:59:10.883114+0000","last_clean":"2024-08-17T17:59:10.883114+0000","last_became_active":"2024-08-17T17:23:07.453712+0000","last_became_peered":"2024-08-17T17:23:07.453712+0000","last_unstale":"2024-08-17T17:59:10.883114+0000","last_undegraded":"2024-08-17T17:59:10.883114+0000","last_fullsized":"2024-08-17T17:59:10.883114+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:03:07.643456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1332,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.886478+0000","last_change":"2024-08-17T17:10:06.994799+0000","last_active":"2024-08-17T17:59:10.886478+0000","last_peered":"2024-08-17T17:59:10.886478+0000","last_clean":"2024-08-17T17:59:10.886478+0000","last_became_active":"2024-08-17T17:10:06.994150+0000","last_became_peered":"2024-08-17T17:10:06.994150+0000","last_unstale":"2024-08-17T17:59:10.886478+0000","last_undegraded":"2024-08-17T17:59:10.886478+0000","last_fullsized":"2024-08-17T17:59:10.886478+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:50:47.293091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":1357,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.883107+0000","last_change":"2024-08-17T17:58:23.589073+0000","last_active":"2024-08-17T17:59:10.883107+0000","last_peered":"2024-08-17T17:59:10.883107+0000","last_clean":"2024-08-17T17:59:10.883107+0000","last_became_active":"2024-08-17T17:58:23.583387+0000","last_became_peered":"2024-08-17T17:58:23.583387+0000","last_unstale":"2024-08-17T17:59:10.883107+0000","last_undegraded":"2024-08-17T17:59:10.883107+0000","last_fullsized":"2024-08-17T17:59:10.883107+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:23:27.036125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1376,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.567193+0000","last_change":"2024-08-17T17:58:23.592768+0000","last_active":"2024-08-17T17:59:11.567193+0000","last_peered":"2024-08-17T17:59:11.567193+0000","last_clean":"2024-08-17T17:59:11.567193+0000","last_became_active":"2024-08-17T17:58:23.582210+0000","last_became_peered":"2024-08-17T17:58:23.582210+0000","last_unstale":"2024-08-17T17:59:11.567193+0000","last_undegraded":"2024-08-17T17:59:11.567193+0000","last_fullsized":"2024-08-17T17:59:11.567193+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:31.713201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2033,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.886736+0000","last_change":"2024-08-17T17:24:17.451528+0000","last_active":"2024-08-17T17:59:11.886736+0000","last_peered":"2024-08-17T17:59:11.886736+0000","last_clean":"2024-08-17T17:59:11.886736+0000","last_became_active":"2024-08-17T17:24:17.451246+0000","last_became_peered":"2024-08-17T17:24:17.451246+0000","last_unstale":"2024-08-17T17:59:11.886736+0000","last_undegraded":"2024-08-17T17:59:11.886736+0000","last_fullsized":"2024-08-17T17:59:11.886736+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:38:18.176559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":73,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.885178+0000","last_change":"2024-08-17T17:53:58.838279+0000","last_active":"2024-08-17T17:59:10.885178+0000","last_peered":"2024-08-17T17:59:10.885178+0000","last_clean":"2024-08-17T17:59:10.885178+0000","last_became_active":"2024-08-17T17:53:58.837978+0000","last_became_peered":"2024-08-17T17:53:58.837978+0000","last_unstale":"2024-08-17T17:59:10.885178+0000","last_undegraded":"2024-08-17T17:59:10.885178+0000","last_fullsized":"2024-08-17T17:59:10.885178+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:06.019142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":685,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.886478+0000","last_change":"2024-08-17T17:58:23.586407+0000","last_active":"2024-08-17T17:59:11.886478+0000","last_peered":"2024-08-17T17:59:11.886478+0000","last_clean":"2024-08-17T17:59:11.886478+0000","last_became_active":"2024-08-17T17:58:23.584967+0000","last_became_peered":"2024-08-17T17:58:23.584967+0000","last_unstale":"2024-08-17T17:59:11.886478+0000","last_undegraded":"2024-08-17T17:59:11.886478+0000","last_fullsized":"2024-08-17T17:59:11.886478+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:58:08.040251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":685,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.886273+0000","last_change":"2024-08-17T17:58:23.584794+0000","last_active":"2024-08-17T17:59:11.886273+0000","last_peered":"2024-08-17T17:59:11.886273+0000","last_clean":"2024-08-17T17:59:11.886273+0000","last_became_active":"2024-08-17T17:58:23.584485+0000","last_became_peered":"2024-08-17T17:58:23.584485+0000","last_unstale":"2024-08-17T17:59:11.886273+0000","last_undegraded":"2024-08-17T17:59:11.886273+0000","last_fullsized":"2024-08-17T17:59:11.886273+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:26:11.516977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1345,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:10.887046+0000","last_change":"2024-08-17T17:24:17.451287+0000","last_active":"2024-08-17T17:59:10.887046+0000","last_peered":"2024-08-17T17:59:10.887046+0000","last_clean":"2024-08-17T17:59:10.887046+0000","last_became_active":"2024-08-17T17:24:17.450951+0000","last_became_peered":"2024-08-17T17:24:17.450951+0000","last_unstale":"2024-08-17T17:59:10.887046+0000","last_undegraded":"2024-08-17T17:59:10.887046+0000","last_fullsized":"2024-08-17T17:59:10.887046+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:13:37.590108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":1346,"reported_epoch":1377,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.962448+0000","last_change":"2024-08-17T17:23:42.907970+0000","last_active":"2024-08-17T17:59:11.962448+0000","last_peered":"2024-08-17T17:59:11.962448+0000","last_clean":"2024-08-17T17:59:11.962448+0000","last_became_active":"2024-08-17T17:23:42.907595+0000","last_became_peered":"2024-08-17T17:23:42.907595+0000","last_unstale":"2024-08-17T17:59:11.962448+0000","last_undegraded":"2024-08-17T17:59:11.962448+0000","last_fullsized":"2024-08-17T17:59:11.962448+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:11:46.130004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2033,"reported_epoch":1378,"state":"active+clean","last_fresh":"2024-08-17T17:59:11.906723+0000","last_change":"2024-08-17T17:24:17.486591+0000","last_active":"2024-08-17T17:59:11.906723+0000","last_peered":"2024-08-17T17:59:11.906723+0000","last_clean":"2024-08-17T17:59:11.906723+0000","last_became_active":"2024-08-17T17:24:17.486320+0000","last_became_peered":"2024-08-17T17:24:17.486320+0000","last_unstale":"2024-08-17T17:59:11.906723+0000","last_undegraded":"2024-08-17T17:59:11.906723+0000","last_fullsized":"2024-08-17T17:59:11.906723+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:03.249803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":54,"num_pg":16,"stat_sum":{"num_bytes":5340004352,"num_objects":81482,"num_object_clones":0,"num_object_copies":244446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4090,"num_objects_unfound":0,"num_objects_dirty":81482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81482,"num_write_kb":5214848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":239,"num_bytes_recovered":15663104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16079126528,"data_stored":16079126528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81482,"ondisk_log_size":81482,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":604,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":17266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234600,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2873356,"kb_used_data":2599492,"kb_used_omap":1,"kb_used_meta":273854,"kb_avail":90871796,"statfs":{"total":95995035648,"available":93052719104,"internally_reserved":0,"allocated":2661879808,"data_stored":2657541538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":280426955},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496238,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2973512,"kb_used_data":2627456,"kb_used_omap":1,"kb_used_meta":346046,"kb_avail":90771640,"statfs":{"total":95995035648,"available":92950159360,"internally_reserved":0,"allocated":2690514944,"data_stored":2686181225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":354351559},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565479,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4174512,"kb_used_data":3835560,"kb_used_omap":72,"kb_used_meta":338871,"kb_avail":89570640,"statfs":{"total":95995035648,"available":91720335360,"internally_reserved":0,"allocated":3927613440,"data_stored":3923298428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":347004398},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,2,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":674,"seq":2894807957934,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1546292,"kb_used_data":1293484,"kb_used_omap":74,"kb_used_meta":252725,"kb_avail":92198860,"statfs":{"total":95995035648,"available":94411632640,"internally_reserved":0,"allocated":1324527616,"data_stored":1320209034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76116,"internal_metadata":258791084},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743186869,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1414456,"kb_used_data":1043696,"kb_used_omap":66,"kb_used_meta":370685,"kb_avail":92330696,"statfs":{"total":95995035648,"available":94546632704,"internally_reserved":0,"allocated":1068744704,"data_stored":1064421995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":379582123},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":2830383448508,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757874,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3270456,"kb_used_data":2928432,"kb_used_omap":1,"kb_used_meta":342014,"kb_avail":90474696,"statfs":{"total":95995035648,"available":92646088704,"internally_reserved":0,"allocated":2998714368,"data_stored":2994391733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":350222790},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019511,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1722244,"kb_used_data":1308092,"kb_used_omap":1,"kb_used_meta":414142,"kb_avail":92022908,"statfs":{"total":95995035648,"available":94231457792,"internally_reserved":0,"allocated":1339486208,"data_stored":1335155516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":424081862},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1070399488,"data_stored":1070399488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1326317568,"data_stored":1326317568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3966697472,"data_stored":3966697472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1333919744,"data_stored":1333919744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3018194944,"data_stored":3018194944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2695561216,"data_stored":2695561216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2668036096,"data_stored":2668036096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:59:26.884 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-17T17:59:26.885 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool set unique_pool_1 pg_num 26 2024-08-17T17:59:26.918 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:26.960 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:26.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:26.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:27.061 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:27.094 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:27.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:27.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:27.196 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:27.226 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:27.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:27.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:27.328 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:27.361 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:27.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:27.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:27.463 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:27.495 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:27.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:27.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:27.597 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:27.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:27.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:27.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:27.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:27 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:59:27.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:27 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:59:27.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:27 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:59:27.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:27 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:59:27.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:59:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:59:27] "GET /metrics HTTP/1.1" 200 36558 "" "Prometheus/2.51.0" 2024-08-17T17:59:27.708 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 53 16 87652 87636 103.333 64.5625 0.0080907 0.00967197 2024-08-17T17:59:27.732 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:27.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:27.768 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:27.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:27.869 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:27.904 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:27.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:27.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:28.006 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:28.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:27 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:59:28.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:27 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:59:28.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:28.043 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:28.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:28.144 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:28.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:28.182 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:28.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:28.283 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:28.313 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:28.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:28.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:28.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:28.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:28.446 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:28.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:28.485 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:59:28.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:28.555 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T17:59:28.594 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:28.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:28.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:28.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:28 smithi136 ceph-mon[109861]: pgmap v2336: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s; 3930/251175 objects misplaced (1.565%); 2.1 MiB/s, 33 objects/s recovering 2024-08-17T17:59:28.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:28 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2943467975' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:59:28.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:28 smithi136 ceph-mon[115078]: pgmap v2336: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s; 3930/251175 objects misplaced (1.565%); 2.1 MiB/s, 33 objects/s recovering 2024-08-17T17:59:28.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:28 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2943467975' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:59:28.697 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:28.708 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 54 16 88598 88582 102.514 59.125 0.068973 0.00974858 2024-08-17T17:59:28.729 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:28.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:28.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:28.830 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:28.867 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:28.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:28.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:28.969 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:28.986 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:28 smithi165 ceph-mon[105517]: pgmap v2336: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s; 3930/251175 objects misplaced (1.565%); 2.1 MiB/s, 33 objects/s recovering 2024-08-17T17:59:28.986 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:28 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2943467975' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:59:29.012 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:29.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:29.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:29.114 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:29.150 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:29.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:29.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:29.252 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:29.282 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:29.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:29.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:29.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:29.414 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:29.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:29.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:29.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:29.557 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:29.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:29.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:29.631 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:29 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:59:29.632 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:29 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2943467975' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:59:29.632 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:29 smithi165 ceph-mon[105517]: osdmap e1379: 8 total, 8 up, 7 in 2024-08-17T17:59:29.659 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:29.688 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:29.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:29.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:29.689 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T17:59:29.690 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T17:59:29.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T17:59:29.690 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T17:59:29.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.690 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:32836 53.0 53:02eb864f:::benchmark_data_smithi136_235573_object32835:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:59:29.690 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:53:29.747897+0000", 2024-08-17T17:59:29.690 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 359.93644586599999, 2024-08-17T17:59:29.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.19520247499999999, 2024-08-17T17:59:29.690 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:29.690 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:59:29.690 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:29.690 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:59:29.691 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:59:29.691 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 32836 2024-08-17T17:59:29.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.691 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:29.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:29.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.747897+0000", 2024-08-17T17:59:29.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:29.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.747897+0000", 2024-08-17T17:59:29.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:29.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.747896+0000", 2024-08-17T17:59:29.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:59:29.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:29.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.748244+0000", 2024-08-17T17:59:29.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034770800000000001 2024-08-17T17:59:29.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:29.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.748254+0000", 2024-08-17T17:59:29.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0686999999999999e-05 2024-08-17T17:59:29.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:29.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.748261+0000", 2024-08-17T17:59:29.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2820000000000002e-06 2024-08-17T17:59:29.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:29.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.754505+0000", 2024-08-17T17:59:29.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062446990000000003 2024-08-17T17:59:29.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:29.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.754613+0000", 2024-08-17T17:59:29.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010746 2024-08-17T17:59:29.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:59:29.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.754898+0000", 2024-08-17T17:59:29.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028547700000000002 2024-08-17T17:59:29.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.760976+0000", 2024-08-17T17:59:29.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060777560000000001 2024-08-17T17:59:29.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:59:29.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.760982+0000", 2024-08-17T17:59:29.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9359999999999996e-06 2024-08-17T17:59:29.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.943006+0000", 2024-08-17T17:59:29.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18202437599999999 2024-08-17T17:59:29.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:59:29.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.943054+0000", 2024-08-17T17:59:29.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7157000000000003e-05 2024-08-17T17:59:29.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:29.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.943099+0000", 2024-08-17T17:59:29.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5803999999999997e-05 2024-08-17T17:59:29.696 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.696 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:29.696 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.696 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:32849 53.0 53:09f7d5d1:::benchmark_data_smithi136_235573_object32848:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:59:29.697 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:53:29.755912+0000", 2024-08-17T17:59:29.697 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 359.92843047700001, 2024-08-17T17:59:29.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.187285812, 2024-08-17T17:59:29.697 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:29.697 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:59:29.697 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:29.697 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:59:29.697 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:59:29.697 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 32849 2024-08-17T17:59:29.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.697 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:29.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:29.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.755912+0000", 2024-08-17T17:59:29.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:29.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.755912+0000", 2024-08-17T17:59:29.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:29.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.755911+0000", 2024-08-17T17:59:29.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:59:29.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:29.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.755943+0000", 2024-08-17T17:59:29.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2493000000000002e-05 2024-08-17T17:59:29.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:29.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.755958+0000", 2024-08-17T17:59:29.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4537e-05 2024-08-17T17:59:29.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:29.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.755965+0000", 2024-08-17T17:59:29.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7279999999999998e-06 2024-08-17T17:59:29.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:29.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.756270+0000", 2024-08-17T17:59:29.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00030566799999999999 2024-08-17T17:59:29.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:29.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.756614+0000", 2024-08-17T17:59:29.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034377799999999999 2024-08-17T17:59:29.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:59:29.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.756799+0000", 2024-08-17T17:59:29.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000184833 2024-08-17T17:59:29.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:59:29.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.760987+0000", 2024-08-17T17:59:29.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041880830000000004 2024-08-17T17:59:29.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.761020+0000", 2024-08-17T17:59:29.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3182000000000002e-05 2024-08-17T17:59:29.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.943134+0000", 2024-08-17T17:59:29.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18211419400000001 2024-08-17T17:59:29.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:59:29.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.943169+0000", 2024-08-17T17:59:29.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4922999999999998e-05 2024-08-17T17:59:29.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:29.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.943198+0000", 2024-08-17T17:59:29.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8822999999999998e-05 2024-08-17T17:59:29.703 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.703 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:29.703 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.703 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:32851 53.0 53:0f637703:::benchmark_data_smithi136_235573_object32850:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:59:29.703 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:53:29.755976+0000", 2024-08-17T17:59:29.703 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 359.928366504, 2024-08-17T17:59:29.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18731230600000001, 2024-08-17T17:59:29.703 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:29.703 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:59:29.703 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:29.703 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:59:29.703 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:59:29.704 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 32851 2024-08-17T17:59:29.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.704 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:29.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:29.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.755976+0000", 2024-08-17T17:59:29.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:29.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.755976+0000", 2024-08-17T17:59:29.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:29.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.755976+0000", 2024-08-17T17:59:29.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T17:59:29.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:29.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.756148+0000", 2024-08-17T17:59:29.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017265600000000001 2024-08-17T17:59:29.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:29.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.756160+0000", 2024-08-17T17:59:29.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2099999999999999e-05 2024-08-17T17:59:29.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:29.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.756167+0000", 2024-08-17T17:59:29.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5200000000000003e-06 2024-08-17T17:59:29.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:29.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.758191+0000", 2024-08-17T17:59:29.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020242179999999999 2024-08-17T17:59:29.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:29.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.758294+0000", 2024-08-17T17:59:29.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102617 2024-08-17T17:59:29.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:59:29.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.758408+0000", 2024-08-17T17:59:29.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113979 2024-08-17T17:59:29.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:59:29.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.760996+0000", 2024-08-17T17:59:29.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002588202 2024-08-17T17:59:29.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.761047+0000", 2024-08-17T17:59:29.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1462000000000003e-05 2024-08-17T17:59:29.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.943228+0000", 2024-08-17T17:59:29.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18218055699999999 2024-08-17T17:59:29.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:59:29.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.943261+0000", 2024-08-17T17:59:29.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2750999999999999e-05 2024-08-17T17:59:29.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:29.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.943289+0000", 2024-08-17T17:59:29.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7807999999999999e-05 2024-08-17T17:59:29.709 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.709 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:29.709 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.709 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:32877 53.0 53:0264ef60:::benchmark_data_smithi136_235573_object32876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:59:29.709 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:53:29.766396+0000", 2024-08-17T17:59:29.709 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 359.91794697199998, 2024-08-17T17:59:29.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17698629099999999, 2024-08-17T17:59:29.709 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:29.709 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:59:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:59:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:59:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 32877 2024-08-17T17:59:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.766396+0000", 2024-08-17T17:59:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.766396+0000", 2024-08-17T17:59:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.766394+0000", 2024-08-17T17:59:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:59:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.766454+0000", 2024-08-17T17:59:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9061999999999998e-05 2024-08-17T17:59:29.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:29.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.766468+0000", 2024-08-17T17:59:29.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4667e-05 2024-08-17T17:59:29.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:29.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.766475+0000", 2024-08-17T17:59:29.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3779999999999998e-06 2024-08-17T17:59:29.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:29.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.766518+0000", 2024-08-17T17:59:29.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3046000000000001e-05 2024-08-17T17:59:29.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:29.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.766634+0000", 2024-08-17T17:59:29.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116704 2024-08-17T17:59:29.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:59:29.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.766766+0000", 2024-08-17T17:59:29.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013154700000000001 2024-08-17T17:59:29.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:59:29.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.768094+0000", 2024-08-17T17:59:29.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013283500000000001 2024-08-17T17:59:29.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.768433+0000", 2024-08-17T17:59:29.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033874900000000001 2024-08-17T17:59:29.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.943322+0000", 2024-08-17T17:59:29.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.174889086 2024-08-17T17:59:29.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:59:29.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.943355+0000", 2024-08-17T17:59:29.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2938000000000002e-05 2024-08-17T17:59:29.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:29.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.943382+0000", 2024-08-17T17:59:29.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7086000000000001e-05 2024-08-17T17:59:29.716 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.716 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:29.716 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.716 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:32879 53.0 53:06564bb1:::benchmark_data_smithi136_235573_object32878:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1307)", 2024-08-17T17:59:29.716 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:53:29.768466+0000", 2024-08-17T17:59:29.716 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 359.91587652099997, 2024-08-17T17:59:29.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17666860500000001, 2024-08-17T17:59:29.716 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:29.716 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:59:29.716 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:29.717 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:59:29.717 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:59:29.717 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 32879 2024-08-17T17:59:29.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.717 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:29.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:29.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.768466+0000", 2024-08-17T17:59:29.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:29.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.768466+0000", 2024-08-17T17:59:29.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:29.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.768465+0000", 2024-08-17T17:59:29.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:59:29.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:29.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.768519+0000", 2024-08-17T17:59:29.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4565999999999999e-05 2024-08-17T17:59:29.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:29.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.768532+0000", 2024-08-17T17:59:29.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2354999999999999e-05 2024-08-17T17:59:29.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:29.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.768537+0000", 2024-08-17T17:59:29.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9400000000000001e-06 2024-08-17T17:59:29.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:29.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.768565+0000", 2024-08-17T17:59:29.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8663e-05 2024-08-17T17:59:29.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:29.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.768660+0000", 2024-08-17T17:59:29.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5052999999999999e-05 2024-08-17T17:59:29.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:59:29.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.768752+0000", 2024-08-17T17:59:29.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1154000000000002e-05 2024-08-17T17:59:29.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:59:29.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.769702+0000", 2024-08-17T17:59:29.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00095040700000000001 2024-08-17T17:59:29.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.769963+0000", 2024-08-17T17:59:29.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026142599999999999 2024-08-17T17:59:29.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.945036+0000", 2024-08-17T17:59:29.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.175072333 2024-08-17T17:59:29.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:59:29.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.945099+0000", 2024-08-17T17:59:29.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2903000000000004e-05 2024-08-17T17:59:29.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:29.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:29.945135+0000", 2024-08-17T17:59:29.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6208999999999998e-05 2024-08-17T17:59:29.722 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.722 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:29.722 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.723 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.35091.0:67749 53.4 e1312/1310 53:25b52b06:::benchmark_data_smithi136_235573_object67748:head v 1312'3986, mlcod=1312'3984)", 2024-08-17T17:59:29.723 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:53:51.853324+0000", 2024-08-17T17:59:29.723 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 337.83101901700002, 2024-08-17T17:59:29.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17928466400000001, 2024-08-17T17:59:29.723 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:29.723 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:59:29.723 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:29.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:29.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:51.853324+0000", 2024-08-17T17:59:29.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:29.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:51.853324+0000", 2024-08-17T17:59:29.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:29.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:51.853323+0000", 2024-08-17T17:59:29.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:59:29.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:29.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:51.853376+0000", 2024-08-17T17:59:29.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3260999999999997e-05 2024-08-17T17:59:29.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:29.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:51.853385+0000", 2024-08-17T17:59:29.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9700000000000005e-06 2024-08-17T17:59:29.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:29.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:51.853391+0000", 2024-08-17T17:59:29.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.105e-06 2024-08-17T17:59:29.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:29.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:51.853430+0000", 2024-08-17T17:59:29.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8838999999999999e-05 2024-08-17T17:59:29.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:29.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:51.853444+0000", 2024-08-17T17:59:29.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3525e-05 2024-08-17T17:59:29.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:59:29.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.032579+0000", 2024-08-17T17:59:29.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17913513 2024-08-17T17:59:29.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:29.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.032608+0000", 2024-08-17T17:59:29.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9703999999999999e-05 2024-08-17T17:59:29.727 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.727 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:29.727 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.727 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.35091.0:67753 53.a e1312/1303 53:596cd44f:::benchmark_data_smithi136_235573_object67752:head v 1312'4291, mlcod=1312'4288)", 2024-08-17T17:59:29.727 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:53:51.856118+0000", 2024-08-17T17:59:29.727 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 337.82822438900001, 2024-08-17T17:59:29.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.181175419, 2024-08-17T17:59:29.728 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:29.728 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:59:29.728 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:29.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:29.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:51.856118+0000", 2024-08-17T17:59:29.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:29.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:51.856118+0000", 2024-08-17T17:59:29.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:29.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:51.856118+0000", 2024-08-17T17:59:29.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:59:29.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:29.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:51.856239+0000", 2024-08-17T17:59:29.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012084899999999999 2024-08-17T17:59:29.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:29.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.025085+0000", 2024-08-17T17:59:29.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16884622899999999 2024-08-17T17:59:29.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:29.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.025104+0000", 2024-08-17T17:59:29.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9629000000000001e-05 2024-08-17T17:59:29.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:29.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.025449+0000", 2024-08-17T17:59:29.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034467699999999999 2024-08-17T17:59:29.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:29.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.025473+0000", 2024-08-17T17:59:29.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3714e-05 2024-08-17T17:59:29.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:59:29.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.037231+0000", 2024-08-17T17:59:29.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011758608 2024-08-17T17:59:29.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:29.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.037294+0000", 2024-08-17T17:59:29.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2428999999999995e-05 2024-08-17T17:59:29.731 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.731 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:29.731 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.732 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:68288 53.0 53:02be0f21:::benchmark_data_smithi136_235573_object68287:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1312)", 2024-08-17T17:59:29.732 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:53:52.477753+0000", 2024-08-17T17:59:29.732 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 337.20658935699998, 2024-08-17T17:59:29.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.157531381, 2024-08-17T17:59:29.732 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:29.732 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:59:29.733 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 55 16 89689 89673 101.89 68.1875 0.0211169 0.00980849 2024-08-17T17:59:29.733 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:29.733 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:59:29.733 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:59:29.733 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 68288 2024-08-17T17:59:29.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.733 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:29.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:29.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.477753+0000", 2024-08-17T17:59:29.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:29.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.477753+0000", 2024-08-17T17:59:29.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:29.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.477752+0000", 2024-08-17T17:59:29.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:59:29.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:29.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.477804+0000", 2024-08-17T17:59:29.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1996000000000002e-05 2024-08-17T17:59:29.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:29.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.477813+0000", 2024-08-17T17:59:29.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9579999999999996e-06 2024-08-17T17:59:29.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:29.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.477818+0000", 2024-08-17T17:59:29.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4689999999999999e-06 2024-08-17T17:59:29.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:29.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.477847+0000", 2024-08-17T17:59:29.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9728999999999999e-05 2024-08-17T17:59:29.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:29.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.477926+0000", 2024-08-17T17:59:29.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8639000000000004e-05 2024-08-17T17:59:29.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:59:29.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.478017+0000", 2024-08-17T17:59:29.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0755999999999996e-05 2024-08-17T17:59:29.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:59:29.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.479273+0000", 2024-08-17T17:59:29.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012560679999999999 2024-08-17T17:59:29.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.480317+0000", 2024-08-17T17:59:29.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010437949999999999 2024-08-17T17:59:29.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.635132+0000", 2024-08-17T17:59:29.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.154814902 2024-08-17T17:59:29.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:59:29.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.635245+0000", 2024-08-17T17:59:29.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113453 2024-08-17T17:59:29.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:29.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:53:52.635285+0000", 2024-08-17T17:59:29.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9687999999999997e-05 2024-08-17T17:59:29.739 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.739 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:29.739 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.739 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:120896 53.18 53:1f17b9c4:::benchmark_data_smithi136_235573_object120895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1329)", 2024-08-17T17:59:29.739 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:54:44.099773+0000", 2024-08-17T17:59:29.739 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 285.58456959400002, 2024-08-17T17:59:29.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15623999999999999, 2024-08-17T17:59:29.739 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:29.740 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:59:29.740 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:29.740 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:59:29.740 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:59:29.740 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 120896 2024-08-17T17:59:29.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.740 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:29.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:29.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.099773+0000", 2024-08-17T17:59:29.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:29.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.099773+0000", 2024-08-17T17:59:29.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:29.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.099772+0000", 2024-08-17T17:59:29.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:59:29.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:29.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.099817+0000", 2024-08-17T17:59:29.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5210000000000003e-05 2024-08-17T17:59:29.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:29.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.099836+0000", 2024-08-17T17:59:29.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9207000000000001e-05 2024-08-17T17:59:29.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:29.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.099843+0000", 2024-08-17T17:59:29.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0430000000000002e-06 2024-08-17T17:59:29.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:29.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.099889+0000", 2024-08-17T17:59:29.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5683000000000003e-05 2024-08-17T17:59:29.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:29.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.100009+0000", 2024-08-17T17:59:29.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012064400000000001 2024-08-17T17:59:29.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:59:29.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.100135+0000", 2024-08-17T17:59:29.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012608000000000001 2024-08-17T17:59:29.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:59:29.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.102051+0000", 2024-08-17T17:59:29.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001915871 2024-08-17T17:59:29.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.102467+0000", 2024-08-17T17:59:29.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00041593199999999999 2024-08-17T17:59:29.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.255920+0000", 2024-08-17T17:59:29.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15345252100000001 2024-08-17T17:59:29.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:59:29.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.255975+0000", 2024-08-17T17:59:29.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5290000000000001e-05 2024-08-17T17:59:29.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:29.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:44.256013+0000", 2024-08-17T17:59:29.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8129e-05 2024-08-17T17:59:29.745 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.745 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:29.745 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.746 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:147656 53.18 53:1c9ded58:::benchmark_data_smithi136_235573_object23370:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1332)", 2024-08-17T17:59:29.746 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:55:12.129405+0000", 2024-08-17T17:59:29.746 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 257.55493797499997, 2024-08-17T17:59:29.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.169596569, 2024-08-17T17:59:29.746 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:29.746 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:59:29.746 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:29.746 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:59:29.746 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:59:29.746 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 147656 2024-08-17T17:59:29.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.747 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:29.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:29.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.129405+0000", 2024-08-17T17:59:29.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:29.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.129405+0000", 2024-08-17T17:59:29.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:29.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.129404+0000", 2024-08-17T17:59:29.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:59:29.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:29.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.129409+0000", 2024-08-17T17:59:29.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1309999999999997e-06 2024-08-17T17:59:29.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:29.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.129418+0000", 2024-08-17T17:59:29.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5049999999999996e-06 2024-08-17T17:59:29.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:29.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.129424+0000", 2024-08-17T17:59:29.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.835e-06 2024-08-17T17:59:29.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:29.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.129603+0000", 2024-08-17T17:59:29.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000179249 2024-08-17T17:59:29.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:29.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.129717+0000", 2024-08-17T17:59:29.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113615 2024-08-17T17:59:29.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:59:29.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.130110+0000", 2024-08-17T17:59:29.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039272100000000001 2024-08-17T17:59:29.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:59:29.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.132566+0000", 2024-08-17T17:59:29.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024558900000000001 2024-08-17T17:59:29.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.138376+0000", 2024-08-17T17:59:29.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058102040000000002 2024-08-17T17:59:29.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.298920+0000", 2024-08-17T17:59:29.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16054427199999999 2024-08-17T17:59:29.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:59:29.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.298961+0000", 2024-08-17T17:59:29.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0667999999999998e-05 2024-08-17T17:59:29.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:29.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:55:12.299001+0000", 2024-08-17T17:59:29.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0710999999999997e-05 2024-08-17T17:59:29.752 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.752 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:29.752 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.752 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:37171 54.8 54:125218ab:::benchmark_data_smithi136_273969_object37170:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1373)", 2024-08-17T17:59:29.752 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:58:48.388504+0000", 2024-08-17T17:59:29.752 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 41.295839196999999, 2024-08-17T17:59:29.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.21517143899999999, 2024-08-17T17:59:29.752 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:29.753 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:59:29.753 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:29.753 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:29.753 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:29.753 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 37171 2024-08-17T17:59:29.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.753 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:29.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:29.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:48.388504+0000", 2024-08-17T17:59:29.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:29.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:48.388504+0000", 2024-08-17T17:59:29.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:29.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:48.388502+0000", 2024-08-17T17:59:29.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:59:29.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:29.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:48.388595+0000", 2024-08-17T17:59:29.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2256999999999994e-05 2024-08-17T17:59:29.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:29.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:48.388612+0000", 2024-08-17T17:59:29.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7609999999999999e-05 2024-08-17T17:59:29.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:29.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:48.388620+0000", 2024-08-17T17:59:29.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4139999999999997e-06 2024-08-17T17:59:29.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:29.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:48.388662+0000", 2024-08-17T17:59:29.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2175999999999997e-05 2024-08-17T17:59:29.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:29.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:48.388785+0000", 2024-08-17T17:59:29.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012320000000000001 2024-08-17T17:59:29.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-08-17T17:59:29.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:48.388914+0000", 2024-08-17T17:59:29.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012923599999999999 2024-08-17T17:59:29.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:59:29.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:48.390534+0000", 2024-08-17T17:59:29.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001620008 2024-08-17T17:59:29.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:48.391611+0000", 2024-08-17T17:59:29.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010769410000000001 2024-08-17T17:59:29.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:48.603549+0000", 2024-08-17T17:59:29.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.211938035 2024-08-17T17:59:29.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:59:29.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:48.603635+0000", 2024-08-17T17:59:29.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6026999999999995e-05 2024-08-17T17:59:29.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:29.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:48.603675+0000", 2024-08-17T17:59:29.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9802000000000003e-05 2024-08-17T17:59:29.758 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.758 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:29.758 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.759 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:48976 54.e 54:72c1b419:::benchmark_data_smithi136_273969_object48975:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1373)", 2024-08-17T17:59:29.759 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:58:55.054293+0000", 2024-08-17T17:59:29.759 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.630049757999998, 2024-08-17T17:59:29.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.28328446200000001, 2024-08-17T17:59:29.759 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:29.759 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:59:29.759 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:29.759 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:29.759 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:29.759 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 48976 2024-08-17T17:59:29.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.759 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:29.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:29.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.054293+0000", 2024-08-17T17:59:29.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:29.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.054293+0000", 2024-08-17T17:59:29.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:29.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.054292+0000", 2024-08-17T17:59:29.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T17:59:29.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:29.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.054438+0000", 2024-08-17T17:59:29.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014573799999999999 2024-08-17T17:59:29.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:29.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.054446+0000", 2024-08-17T17:59:29.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7370000000000006e-06 2024-08-17T17:59:29.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:29.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.054451+0000", 2024-08-17T17:59:29.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1429999999999997e-06 2024-08-17T17:59:29.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:29.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.054497+0000", 2024-08-17T17:59:29.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6199000000000003e-05 2024-08-17T17:59:29.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:29.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.054619+0000", 2024-08-17T17:59:29.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012221000000000001 2024-08-17T17:59:29.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:59:29.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.054760+0000", 2024-08-17T17:59:29.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000140545 2024-08-17T17:59:29.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:59:29.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.056601+0000", 2024-08-17T17:59:29.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018411980000000001 2024-08-17T17:59:29.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.063185+0000", 2024-08-17T17:59:29.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0065833970000000004 2024-08-17T17:59:29.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.337417+0000", 2024-08-17T17:59:29.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27423292799999999 2024-08-17T17:59:29.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:59:29.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.337473+0000", 2024-08-17T17:59:29.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5892e-05 2024-08-17T17:59:29.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:29.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.337577+0000", 2024-08-17T17:59:29.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104141 2024-08-17T17:59:29.765 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.765 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:29.765 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.765 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:48986 54.e 54:76ccfb2f:::benchmark_data_smithi136_273969_object48985:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1373)", 2024-08-17T17:59:29.765 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:58:55.059290+0000", 2024-08-17T17:59:29.765 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.625053186000002, 2024-08-17T17:59:29.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27839962899999998, 2024-08-17T17:59:29.765 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:29.765 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:59:29.765 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:29.766 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:29.766 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:29.766 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 48986 2024-08-17T17:59:29.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.766 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:29.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:29.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.059290+0000", 2024-08-17T17:59:29.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:29.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.059290+0000", 2024-08-17T17:59:29.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:29.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.059285+0000", 2024-08-17T17:59:29.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999957 2024-08-17T17:59:29.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:29.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.059353+0000", 2024-08-17T17:59:29.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8039000000000004e-05 2024-08-17T17:59:29.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:29.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.059378+0000", 2024-08-17T17:59:29.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4366999999999999e-05 2024-08-17T17:59:29.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:29.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.059387+0000", 2024-08-17T17:59:29.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7429999999999997e-06 2024-08-17T17:59:29.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:29.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.059461+0000", 2024-08-17T17:59:29.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4062999999999999e-05 2024-08-17T17:59:29.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:29.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.059585+0000", 2024-08-17T17:59:29.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012356100000000001 2024-08-17T17:59:29.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:59:29.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.059706+0000", 2024-08-17T17:59:29.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120938 2024-08-17T17:59:29.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:59:29.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.062726+0000", 2024-08-17T17:59:29.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030205190000000002 2024-08-17T17:59:29.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.063709+0000", 2024-08-17T17:59:29.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00098208600000000007 2024-08-17T17:59:29.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.337620+0000", 2024-08-17T17:59:29.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27391174600000001 2024-08-17T17:59:29.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:59:29.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.337658+0000", 2024-08-17T17:59:29.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7561999999999997e-05 2024-08-17T17:59:29.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:29.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.337689+0000", 2024-08-17T17:59:29.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1321e-05 2024-08-17T17:59:29.771 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.771 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:29.771 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.772 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:48993 54.e 54:7eea629e:::benchmark_data_smithi136_273969_object48992:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1373)", 2024-08-17T17:59:29.772 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:58:55.064172+0000", 2024-08-17T17:59:29.772 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.620171186999997, 2024-08-17T17:59:29.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27757973000000002, 2024-08-17T17:59:29.772 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:29.772 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:59:29.772 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:29.772 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:29.772 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:29.772 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 48993 2024-08-17T17:59:29.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.772 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:29.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:29.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.064172+0000", 2024-08-17T17:59:29.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:29.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.064172+0000", 2024-08-17T17:59:29.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:29.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.064171+0000", 2024-08-17T17:59:29.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:59:29.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:29.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.064252+0000", 2024-08-17T17:59:29.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1594999999999994e-05 2024-08-17T17:59:29.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:29.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.064265+0000", 2024-08-17T17:59:29.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2877999999999999e-05 2024-08-17T17:59:29.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:29.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.064272+0000", 2024-08-17T17:59:29.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.109e-06 2024-08-17T17:59:29.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:29.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.064326+0000", 2024-08-17T17:59:29.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3572999999999997e-05 2024-08-17T17:59:29.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:29.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.064458+0000", 2024-08-17T17:59:29.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013180399999999999 2024-08-17T17:59:29.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:59:29.776 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.064575+0000", 2024-08-17T17:59:29.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000117138 2024-08-17T17:59:29.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:59:29.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.069254+0000", 2024-08-17T17:59:29.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046794239999999997 2024-08-17T17:59:29.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.069840+0000", 2024-08-17T17:59:29.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00058567700000000005 2024-08-17T17:59:29.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.341611+0000", 2024-08-17T17:59:29.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.271770599 2024-08-17T17:59:29.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:59:29.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.341704+0000", 2024-08-17T17:59:29.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3536999999999996e-05 2024-08-17T17:59:29.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:29.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.341751+0000", 2024-08-17T17:59:29.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7256000000000003e-05 2024-08-17T17:59:29.778 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.778 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:29.778 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.779 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:48995 54.e 54:7a39b2b0:::benchmark_data_smithi136_273969_object48994:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1373)", 2024-08-17T17:59:29.779 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:58:55.067015+0000", 2024-08-17T17:59:29.779 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.617327875999997, 2024-08-17T17:59:29.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.274878228, 2024-08-17T17:59:29.779 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:29.779 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:59:29.779 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:29.779 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:29.779 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:29.779 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 48995 2024-08-17T17:59:29.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.779 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:29.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:29.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.067015+0000", 2024-08-17T17:59:29.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:29.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.067015+0000", 2024-08-17T17:59:29.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:29.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.067014+0000", 2024-08-17T17:59:29.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T17:59:29.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:29.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.067046+0000", 2024-08-17T17:59:29.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2100000000000001e-05 2024-08-17T17:59:29.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:29.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.067060+0000", 2024-08-17T17:59:29.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3439e-05 2024-08-17T17:59:29.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:29.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.067071+0000", 2024-08-17T17:59:29.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1717e-05 2024-08-17T17:59:29.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:29.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.067145+0000", 2024-08-17T17:59:29.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3882999999999997e-05 2024-08-17T17:59:29.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:29.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.067268+0000", 2024-08-17T17:59:29.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012262300000000001 2024-08-17T17:59:29.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:59:29.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.067348+0000", 2024-08-17T17:59:29.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0414000000000006e-05 2024-08-17T17:59:29.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:59:29.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.069263+0000", 2024-08-17T17:59:29.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001914841 2024-08-17T17:59:29.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.069873+0000", 2024-08-17T17:59:29.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00060947900000000001 2024-08-17T17:59:29.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.341788+0000", 2024-08-17T17:59:29.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27191507599999998 2024-08-17T17:59:29.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:59:29.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.341859+0000", 2024-08-17T17:59:29.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0803000000000006e-05 2024-08-17T17:59:29.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:29.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.341893+0000", 2024-08-17T17:59:29.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4548999999999997e-05 2024-08-17T17:59:29.784 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.785 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:29.785 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.785 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:49012 54.e 54:7d14b5c5:::benchmark_data_smithi136_273969_object49011:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1373)", 2024-08-17T17:59:29.785 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:58:55.077010+0000", 2024-08-17T17:59:29.785 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.607332315000001, 2024-08-17T17:59:29.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.26644016500000001, 2024-08-17T17:59:29.785 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:29.785 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:59:29.785 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:29.785 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:29.785 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:29.786 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 49012 2024-08-17T17:59:29.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.786 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:29.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:29.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.077010+0000", 2024-08-17T17:59:29.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:29.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.077010+0000", 2024-08-17T17:59:29.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:29.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.077009+0000", 2024-08-17T17:59:29.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T17:59:29.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:29.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.077060+0000", 2024-08-17T17:59:29.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1520000000000001e-05 2024-08-17T17:59:29.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:29.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.077083+0000", 2024-08-17T17:59:29.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2768e-05 2024-08-17T17:59:29.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:29.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.077090+0000", 2024-08-17T17:59:29.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2790000000000001e-06 2024-08-17T17:59:29.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:29.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.077146+0000", 2024-08-17T17:59:29.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6011000000000003e-05 2024-08-17T17:59:29.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:29.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.077309+0000", 2024-08-17T17:59:29.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016266300000000001 2024-08-17T17:59:29.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:59:29.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.077444+0000", 2024-08-17T17:59:29.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013501999999999999 2024-08-17T17:59:29.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:59:29.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.079480+0000", 2024-08-17T17:59:29.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002036286 2024-08-17T17:59:29.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.079641+0000", 2024-08-17T17:59:29.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016083800000000001 2024-08-17T17:59:29.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.343341+0000", 2024-08-17T17:59:29.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.26370031999999999 2024-08-17T17:59:29.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:59:29.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.343414+0000", 2024-08-17T17:59:29.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2756999999999995e-05 2024-08-17T17:59:29.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:29.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.343451+0000", 2024-08-17T17:59:29.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6599e-05 2024-08-17T17:59:29.792 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.792 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:29.792 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.793 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:49017 54.e 54:7457b514:::benchmark_data_smithi136_273969_object49016:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1373)", 2024-08-17T17:59:29.793 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:58:55.082603+0000", 2024-08-17T17:59:29.793 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.60174026, 2024-08-17T17:59:29.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.260950336, 2024-08-17T17:59:29.793 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:29.793 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:59:29.793 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:29.793 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:29.793 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:29.793 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 49017 2024-08-17T17:59:29.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.793 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:29.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:29.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.082603+0000", 2024-08-17T17:59:29.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:29.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.082603+0000", 2024-08-17T17:59:29.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:29.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.082602+0000", 2024-08-17T17:59:29.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:59:29.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:29.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.082658+0000", 2024-08-17T17:59:29.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6688999999999999e-05 2024-08-17T17:59:29.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:29.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.082673+0000", 2024-08-17T17:59:29.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4970999999999999e-05 2024-08-17T17:59:29.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:29.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.082678+0000", 2024-08-17T17:59:29.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0250000000000002e-06 2024-08-17T17:59:29.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:29.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.082720+0000", 2024-08-17T17:59:29.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1545999999999998e-05 2024-08-17T17:59:29.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:29.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.082865+0000", 2024-08-17T17:59:29.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014499599999999999 2024-08-17T17:59:29.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:59:29.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.083015+0000", 2024-08-17T17:59:29.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000150352 2024-08-17T17:59:29.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:59:29.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.084920+0000", 2024-08-17T17:59:29.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001905038 2024-08-17T17:59:29.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.085278+0000", 2024-08-17T17:59:29.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00035742100000000002 2024-08-17T17:59:29.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.343487+0000", 2024-08-17T17:59:29.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25820974899999999 2024-08-17T17:59:29.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:59:29.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.343524+0000", 2024-08-17T17:59:29.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6548999999999999e-05 2024-08-17T17:59:29.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:29.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:58:55.343553+0000", 2024-08-17T17:59:29.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9003999999999999e-05 2024-08-17T17:59:29.799 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.799 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:29.799 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.799 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:81674 54.e 54:75eefd73:::benchmark_data_smithi136_273969_object81673:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1378)", 2024-08-17T17:59:29.799 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:21.793290+0000", 2024-08-17T17:59:29.799 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.8910528840000005, 2024-08-17T17:59:29.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18330375700000001, 2024-08-17T17:59:29.799 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:29.800 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:59:29.800 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:29.800 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:29.800 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:29.800 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 81674 2024-08-17T17:59:29.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.800 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:29.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:29.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.793290+0000", 2024-08-17T17:59:29.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:29.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.793290+0000", 2024-08-17T17:59:29.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:29.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.793289+0000", 2024-08-17T17:59:29.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:59:29.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:29.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.793355+0000", 2024-08-17T17:59:29.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5939999999999995e-05 2024-08-17T17:59:29.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:29.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.793368+0000", 2024-08-17T17:59:29.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3453e-05 2024-08-17T17:59:29.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:29.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.793374+0000", 2024-08-17T17:59:29.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8209999999999998e-06 2024-08-17T17:59:29.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:29.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.797637+0000", 2024-08-17T17:59:29.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004263579 2024-08-17T17:59:29.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:29.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.797776+0000", 2024-08-17T17:59:29.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013874699999999999 2024-08-17T17:59:29.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:59:29.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.797916+0000", 2024-08-17T17:59:29.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013956999999999999 2024-08-17T17:59:29.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:59:29.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.804624+0000", 2024-08-17T17:59:29.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0067082660000000001 2024-08-17T17:59:29.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.807398+0000", 2024-08-17T17:59:29.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027737059999999999 2024-08-17T17:59:29.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.976461+0000", 2024-08-17T17:59:29.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16906359200000001 2024-08-17T17:59:29.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:59:29.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.976550+0000", 2024-08-17T17:59:29.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8443999999999995e-05 2024-08-17T17:59:29.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:29.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.976594+0000", 2024-08-17T17:59:29.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3822999999999997e-05 2024-08-17T17:59:29.805 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.805 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:29.805 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.806 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:81678 54.e 54:738ce7e5:::benchmark_data_smithi136_273969_object81677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1378)", 2024-08-17T17:59:29.806 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:21.794445+0000", 2024-08-17T17:59:29.806 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.8898973100000003, 2024-08-17T17:59:29.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18601032300000001, 2024-08-17T17:59:29.806 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:29.806 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:59:29.806 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:29.806 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:29.806 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:29.806 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 81678 2024-08-17T17:59:29.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.806 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:29.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:29.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.794445+0000", 2024-08-17T17:59:29.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:29.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.794445+0000", 2024-08-17T17:59:29.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:29.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.794445+0000", 2024-08-17T17:59:29.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T17:59:29.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:29.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.794499+0000", 2024-08-17T17:59:29.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3981999999999999e-05 2024-08-17T17:59:29.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:29.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.794508+0000", 2024-08-17T17:59:29.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9579999999999996e-06 2024-08-17T17:59:29.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:29.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.794513+0000", 2024-08-17T17:59:29.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2089999999999996e-06 2024-08-17T17:59:29.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:29.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.800198+0000", 2024-08-17T17:59:29.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056851799999999997 2024-08-17T17:59:29.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:29.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.800472+0000", 2024-08-17T17:59:29.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027395700000000002 2024-08-17T17:59:29.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:59:29.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.800610+0000", 2024-08-17T17:59:29.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013741900000000001 2024-08-17T17:59:29.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:59:29.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.809960+0000", 2024-08-17T17:59:29.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0093503349999999995 2024-08-17T17:59:29.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.812096+0000", 2024-08-17T17:59:29.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002135886 2024-08-17T17:59:29.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.980399+0000", 2024-08-17T17:59:29.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16830341400000001 2024-08-17T17:59:29.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:59:29.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.980432+0000", 2024-08-17T17:59:29.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3027000000000001e-05 2024-08-17T17:59:29.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:29.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.980456+0000", 2024-08-17T17:59:29.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3419000000000001e-05 2024-08-17T17:59:29.812 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.812 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:29.812 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.812 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:81714 54.e 54:7d13a236:::benchmark_data_smithi136_273969_object81713:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1378)", 2024-08-17T17:59:29.812 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:21.825318+0000", 2024-08-17T17:59:29.812 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.8590246449999999, 2024-08-17T17:59:29.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16144593600000001, 2024-08-17T17:59:29.812 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:29.812 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:59:29.812 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:29.813 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:29.813 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:29.813 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 81714 2024-08-17T17:59:29.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.813 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:29.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:29.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.825318+0000", 2024-08-17T17:59:29.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:29.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.825318+0000", 2024-08-17T17:59:29.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:29.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:29.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.825316+0000", 2024-08-17T17:59:29.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T17:59:29.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:29.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.825356+0000", 2024-08-17T17:59:29.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9576000000000002e-05 2024-08-17T17:59:29.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:29.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.825378+0000", 2024-08-17T17:59:29.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2529999999999999e-05 2024-08-17T17:59:29.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:29.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.825388+0000", 2024-08-17T17:59:29.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4429999999999992e-06 2024-08-17T17:59:29.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:29.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.825442+0000", 2024-08-17T17:59:29.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4489e-05 2024-08-17T17:59:29.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:29.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.825569+0000", 2024-08-17T17:59:29.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000127076 2024-08-17T17:59:29.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:59:29.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.825702+0000", 2024-08-17T17:59:29.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013242899999999999 2024-08-17T17:59:29.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T17:59:29.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.828822+0000", 2024-08-17T17:59:29.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031202370000000001 2024-08-17T17:59:29.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.832214+0000", 2024-08-17T17:59:29.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033920489999999998 2024-08-17T17:59:29.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:59:29.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.986671+0000", 2024-08-17T17:59:29.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15445710700000001 2024-08-17T17:59:29.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:59:29.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.986723+0000", 2024-08-17T17:59:29.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2544999999999999e-05 2024-08-17T17:59:29.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:29.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:29.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:29.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:21.986764+0000", 2024-08-17T17:59:29.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0599999999999998e-05 2024-08-17T17:59:29.818 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.818 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:29.818 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.818 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:29.818 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:29.818 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T17:59:29.819 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:29.860 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:29.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:29.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:29.862 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:29 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:59:29.862 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:29 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2943467975' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:59:29.862 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:29 smithi136 ceph-mon[109861]: osdmap e1379: 8 total, 8 up, 7 in 2024-08-17T17:59:29.862 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:29 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:59:29.862 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:29 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2943467975' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:59:29.862 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:29 smithi136 ceph-mon[115078]: osdmap e1379: 8 total, 8 up, 7 in 2024-08-17T17:59:29.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:30.001 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:30.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:30.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:30.103 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:30.133 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:30.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:30.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:30.234 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:30.267 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:30.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:30.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:30.369 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:30.404 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:30.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:30.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:30.506 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:30.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:30.550 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:30.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:30.609 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-17T17:59:30.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:30.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:30.690 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:30.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:30.708 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 56 16 90738 90722 101.241 65.5625 0.015896 0.0098716 2024-08-17T17:59:30.791 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:30.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:30.825 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:30.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:30.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:30.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:30 smithi136 ceph-mon[109861]: pgmap v2338: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 64 MiB/s wr, 1.03k op/s; 3930/259809 objects misplaced (1.513%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T17:59:30.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:30 smithi136 ceph-mon[115078]: pgmap v2338: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 64 MiB/s wr, 1.03k op/s; 3930/259809 objects misplaced (1.513%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T17:59:30.968 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:30.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:30.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:30.977 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:59:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:30 smithi165 ceph-mon[105517]: pgmap v2338: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 64 MiB/s wr, 1.03k op/s; 3930/259809 objects misplaced (1.513%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T17:59:31.069 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:31.109 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:31.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:31.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:31.211 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:31.243 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:31.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:31.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:31.344 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:31.366 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:31.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:31.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:31.467 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:31.508 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:31.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:31.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:31.609 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:31.645 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:31.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:31.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:31.708 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 57 16 91672 91656 100.489 58.375 0.0127797 0.00994589 2024-08-17T17:59:31.747 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:31.784 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:31.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:31.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:31.888 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:31.898 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:59:31.926 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:31.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:31.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:32.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:32.065 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:32.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:32.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:32.167 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:32.203 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:32.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:32.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:32.305 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:32.334 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:32.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:32.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:32.437 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:32.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:32.478 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:32.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:32.579 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:32.626 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:32.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:32.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:32.666 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:32 smithi136 ceph-mon[109861]: pgmap v2339: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 64 MiB/s wr, 1.03k op/s; 3930/259809 objects misplaced (1.513%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T17:59:32.666 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:32 smithi136 ceph-mon[115078]: pgmap v2339: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 64 MiB/s wr, 1.03k op/s; 3930/259809 objects misplaced (1.513%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T17:59:32.708 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 58 16 92719 92703 99.8849 65.4375 0.0163776 0.0100065 2024-08-17T17:59:32.728 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:32.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:32.766 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:32.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:32.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:32 smithi165 ceph-mon[105517]: pgmap v2339: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 64 MiB/s wr, 1.03k op/s; 3930/259809 objects misplaced (1.513%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T17:59:32.868 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:32.910 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:32.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:32.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:32.941 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T17:59:32.941 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:59:32.976 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T17:59:32.976 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:59:33.006 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T17:59:33.006 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:59:33.011 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:33.032 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T17:59:33.032 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:59:33.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:33.052 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:33.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:33.064 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T17:59:33.064 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:59:33.096 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T17:59:33.097 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:59:33.124 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T17:59:33.125 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:59:33.152 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T17:59:33.152 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:59:33.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:33.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:33.188 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:33.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:33.289 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:33.322 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:33.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:33.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:33.423 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:33.434 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T17:59:33.434 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:59:33.458 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:33.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:33.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:33.559 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:33.598 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:33.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:33.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:33.700 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:33.709 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 59 16 93735 93719 99.2681 63.5 0.0205548 0.0100686 2024-08-17T17:59:33.743 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:33.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:33.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:33 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2552431142' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:59:33.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:33 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2552431142' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:59:33.799 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:33 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2552431142' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:59:33.845 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:33.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:33.892 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:33.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:33.930 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:59:33.993 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:34.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:34.040 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:34.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:34.141 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:34.178 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:34.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:34.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:34.200 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-17T17:59:34.280 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:34.312 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:34.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:34.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:34.414 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:34.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:34.465 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:34.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:34.565 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:34.583 INFO:teuthology.orchestra.run.smithi136.stderr:set pool 54 pg_num to 26 2024-08-17T17:59:34.606 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:34.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:34.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:34.622 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:59:34.709 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:34.710 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:2024-08-17T17:59:34.709222+0000 min lat: 0.0012629 max lat: 2.15603 avg lat: 0.0101331 2024-08-17T17:59:34.710 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:59:34.710 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 60 16 94718 94702 98.6375 61.4375 0.0131521 0.0101331 2024-08-17T17:59:34.744 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:34.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:34.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:34.785 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:34 smithi136 ceph-mon[109861]: pgmap v2340: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s; 3769/275028 objects misplaced (1.370%); 2.0 MiB/s, 32 objects/s recovering 2024-08-17T17:59:34.786 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:34 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1650282591' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T17:59:34.786 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:34 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T17:59:34.786 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:34 smithi136 ceph-mon[115078]: pgmap v2340: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s; 3769/275028 objects misplaced (1.370%); 2.0 MiB/s, 32 objects/s recovering 2024-08-17T17:59:34.786 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:34 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1650282591' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T17:59:34.786 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:34 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T17:59:34.847 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:34.882 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:34.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:34.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:34.910 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:34 smithi165 ceph-mon[105517]: pgmap v2340: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s; 3769/275028 objects misplaced (1.370%); 2.0 MiB/s, 32 objects/s recovering 2024-08-17T17:59:34.910 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:34 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1650282591' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T17:59:34.910 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:34 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T17:59:34.984 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:35.021 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:35.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:35.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:35.123 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:35.159 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:35.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:35.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:35.261 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:35.292 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:35.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:35.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:35.394 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:35.429 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:35.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:35.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:35.531 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:35.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:35.567 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:35.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:35.668 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:35.709 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 61 16 95559 95543 97.8821 52.5625 0.0398388 0.0101953 2024-08-17T17:59:35.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:35.733 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:35.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:35.834 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:35.846 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:35 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-08-17T17:59:35.847 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:35 smithi165 ceph-mon[105517]: osdmap e1380: 8 total, 8 up, 7 in 2024-08-17T17:59:35.847 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:35 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"} : dispatch 2024-08-17T17:59:35.881 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:35.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:35.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:35.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:35 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-08-17T17:59:35.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:35 smithi136 ceph-mon[109861]: osdmap e1380: 8 total, 8 up, 7 in 2024-08-17T17:59:35.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:35 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"} : dispatch 2024-08-17T17:59:35.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:35 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-08-17T17:59:35.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:35 smithi136 ceph-mon[115078]: osdmap e1380: 8 total, 8 up, 7 in 2024-08-17T17:59:35.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:35 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"} : dispatch 2024-08-17T17:59:35.983 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:36.004 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:36.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:36.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:36.106 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:36.128 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:36.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:36.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:36.230 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:36.253 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:36.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:36.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:36.355 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:36.377 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:36.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:36.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:36.479 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:36.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:36.502 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:36.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:36.543 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:59:36.603 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:36.640 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:36.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:36.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:36.709 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 62 16 95611 95595 96.3559 3.25 0.00419404 0.0103736 2024-08-17T17:59:36.742 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:36.774 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:36 smithi165 ceph-mon[105517]: pgmap v2342: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 62 MiB/s wr, 993 op/s; 3769/275028 objects misplaced (1.370%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:59:36.774 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-17T17:59:36.774 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:36 smithi165 ceph-mon[105517]: osdmap e1381: 8 total, 8 up, 7 in 2024-08-17T17:59:36.775 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:59:36.775 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:59:36.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:36.786 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:36.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:36.887 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:36.932 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:36.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:36.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:36 smithi136 ceph-mon[109861]: pgmap v2342: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 62 MiB/s wr, 993 op/s; 3769/275028 objects misplaced (1.370%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:59:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-17T17:59:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:36 smithi136 ceph-mon[109861]: osdmap e1381: 8 total, 8 up, 7 in 2024-08-17T17:59:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:59:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:59:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:36 smithi136 ceph-mon[115078]: pgmap v2342: 89 pgs: 1 active+recovering+remapped, 88 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 62 MiB/s wr, 993 op/s; 3769/275028 objects misplaced (1.370%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:59:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-17T17:59:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:36 smithi136 ceph-mon[115078]: osdmap e1381: 8 total, 8 up, 7 in 2024-08-17T17:59:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:59:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:59:37.035 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:37.074 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:37.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:37.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:37.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:37.223 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:37.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:37.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:37.229 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T17:59:37.229 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:59:37.325 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:37.364 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:37.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:37.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:37.465 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:37.504 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:37.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:37.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:37.582 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:59:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:59:37] "GET /metrics HTTP/1.1" 200 36565 "" "Prometheus/2.51.0" 2024-08-17T17:59:37.606 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:37.647 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:37.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:37.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:37.709 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 63 16 96858 96842 96.0634 77.9375 0.0147733 0.0104047 2024-08-17T17:59:37.750 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:37.785 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:37.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:37.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:37.887 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:37.908 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:37 smithi165 ceph-mon[105517]: osdmap e1382: 8 total, 8 up, 7 in 2024-08-17T17:59:37.908 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:37 smithi165 ceph-mon[105517]: 54.7 scrub starts 2024-08-17T17:59:37.921 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:37.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:37.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:37.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:37 smithi136 ceph-mon[109861]: osdmap e1382: 8 total, 8 up, 7 in 2024-08-17T17:59:37.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:37 smithi136 ceph-mon[109861]: 54.7 scrub starts 2024-08-17T17:59:37.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:37 smithi136 ceph-mon[115078]: osdmap e1382: 8 total, 8 up, 7 in 2024-08-17T17:59:37.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:37 smithi136 ceph-mon[115078]: 54.7 scrub starts 2024-08-17T17:59:38.023 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:38.054 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:38.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:38.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:38.083 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-17T17:59:38.157 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:38.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:38.195 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:38.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:38.296 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:38.324 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:38.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:38.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:38.426 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:38.451 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:59:38.478 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:38.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:38.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:38.500 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:59:38.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:38.616 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:38.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:38.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:38.710 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 64 16 98038 98022 95.7144 73.75 0.0108215 0.0104426 2024-08-17T17:59:38.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:38.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:38.758 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:38.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:38.859 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:38.890 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:38.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:38.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:38.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:38 smithi136 ceph-mon[109861]: pgmap v2345: 99 pgs: 1 active+recovering+remapped, 4 peering, 10 unknown, 84 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s; 3605/280947 objects misplaced (1.283%); 3.4 MiB/s, 54 objects/s recovering 2024-08-17T17:59:38.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:38 smithi136 ceph-mon[109861]: 54.9 scrub starts 2024-08-17T17:59:38.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:38 smithi136 ceph-mon[109861]: 54.4 scrub starts 2024-08-17T17:59:38.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:38 smithi136 ceph-mon[115078]: pgmap v2345: 99 pgs: 1 active+recovering+remapped, 4 peering, 10 unknown, 84 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s; 3605/280947 objects misplaced (1.283%); 3.4 MiB/s, 54 objects/s recovering 2024-08-17T17:59:38.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:38 smithi136 ceph-mon[115078]: 54.9 scrub starts 2024-08-17T17:59:38.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:38 smithi136 ceph-mon[115078]: 54.4 scrub starts 2024-08-17T17:59:38.979 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:38 smithi165 ceph-mon[105517]: pgmap v2345: 99 pgs: 1 active+recovering+remapped, 4 peering, 10 unknown, 84 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s; 3605/280947 objects misplaced (1.283%); 3.4 MiB/s, 54 objects/s recovering 2024-08-17T17:59:38.979 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:38 smithi165 ceph-mon[105517]: 54.9 scrub starts 2024-08-17T17:59:38.979 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:38 smithi165 ceph-mon[105517]: 54.4 scrub starts 2024-08-17T17:59:38.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:39.039 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:39.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:39.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:39.142 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:39.174 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:39.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:39.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:39.276 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:39.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:39.315 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:39.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:39.416 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:39.451 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:39.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:39.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:39.553 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:39.585 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:39.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:39.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:39.625 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T17:59:39.688 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:39.710 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 65 16 99306 99290 95.4602 79.25 0.00621514 0.0104705 2024-08-17T17:59:39.722 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:39.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:39.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:39 smithi165 ceph-mon[105517]: 54.17 scrub starts 2024-08-17T17:59:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:39 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3773636322' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:59:39.824 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:39.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:39.855 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:39.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:39.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:39 smithi136 ceph-mon[109861]: 54.17 scrub starts 2024-08-17T17:59:39.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:39 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3773636322' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:59:39.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:39 smithi136 ceph-mon[115078]: 54.17 scrub starts 2024-08-17T17:59:39.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:39 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3773636322' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:59:39.956 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:39.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:39.991 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:39.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:40.092 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:40.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:40.134 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:40.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:40.235 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:40.268 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:40.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:40.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:40.370 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:40.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:40.407 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:40.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:40.496 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T17:59:40.496 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:59:40.508 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:40.542 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:40.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:40.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:40.644 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:40.677 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:40.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:40.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:40.691 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:40 smithi136 ceph-mon[109861]: pgmap v2346: 99 pgs: 1 active+recovering+remapped, 4 peering, 4 unknown, 90 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 45 MiB/s wr, 713 op/s; 3605/287871 objects misplaced (1.252%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:59:40.691 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:40 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:59:40.691 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:40 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3773636322' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:59:40.691 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:40 smithi136 ceph-mon[109861]: osdmap e1383: 8 total, 8 up, 7 in 2024-08-17T17:59:40.691 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:40 smithi136 ceph-mon[109861]: 54.14 scrub starts 2024-08-17T17:59:40.691 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:40 smithi136 ceph-mon[109861]: 54.19 scrub starts 2024-08-17T17:59:40.691 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:40 smithi136 ceph-mon[115078]: pgmap v2346: 99 pgs: 1 active+recovering+remapped, 4 peering, 4 unknown, 90 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 45 MiB/s wr, 713 op/s; 3605/287871 objects misplaced (1.252%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:59:40.692 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:40 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:59:40.692 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:40 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3773636322' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:59:40.692 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:40 smithi136 ceph-mon[115078]: osdmap e1383: 8 total, 8 up, 7 in 2024-08-17T17:59:40.692 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:40 smithi136 ceph-mon[115078]: 54.14 scrub starts 2024-08-17T17:59:40.692 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:40 smithi136 ceph-mon[115078]: 54.19 scrub starts 2024-08-17T17:59:40.710 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 66 16 100663 100647 95.2988 84.8125 0.00920818 0.010488 2024-08-17T17:59:40.779 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:40.809 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:40.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:40.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:40.911 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:40.946 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:40.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:40.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:40 smithi165 ceph-mon[105517]: pgmap v2346: 99 pgs: 1 active+recovering+remapped, 4 peering, 4 unknown, 90 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 45 MiB/s wr, 713 op/s; 3605/287871 objects misplaced (1.252%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:59:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:40 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:59:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:40 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3773636322' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:59:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:40 smithi165 ceph-mon[105517]: osdmap e1383: 8 total, 8 up, 7 in 2024-08-17T17:59:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:40 smithi165 ceph-mon[105517]: 54.14 scrub starts 2024-08-17T17:59:41.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:40 smithi165 ceph-mon[105517]: 54.19 scrub starts 2024-08-17T17:59:41.048 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:41.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:41.083 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:41.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:41.184 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:41.220 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:41.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:41.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:41.321 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:41.356 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:41.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:41.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:41.458 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:41.480 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-17T17:59:41.481 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:59:41.481 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-08-17T17:59:41.481 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-17T17:59:41.481 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-17T17:59:41.481 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-08-17T17:59:41.492 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:41.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:41.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:41.594 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:41.629 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:41.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:41.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:41.710 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 67 16 101979 101963 95.1039 82.25 0.00918211 0.0105098 2024-08-17T17:59:41.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-17T17:59:41.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:41.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:41.802 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:41.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:41.903 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:41.936 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:41.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:41.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:41.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:41 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3376129345' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:59:41.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:41 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3376129345' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:59:42.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:41 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3376129345' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:59:42.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:42.083 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:59:42.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:42.088 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:42.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:42.189 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:42.223 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:42.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:42.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:42.326 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:42.360 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:42.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:42.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:42.427 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T17:59:42.428 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T17:59:42.428 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T17:59:42.428 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T17:59:42.428 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T17:59:42.428 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:59:42.428 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:59:42.428 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:59:42.428 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:59:42.429 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:59:42.429 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:59:42.429 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T17:59:42.429 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T17:59:42.429 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T17:59:42.429 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:59:42.430 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:59:42.462 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:42.491 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:42.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:42.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:42.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:42.634 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:42.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:42.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:42.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:42 smithi136 ceph-mon[109861]: pgmap v2348: 99 pgs: 1 active+recovering+remapped, 4 peering, 4 unknown, 90 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 45 MiB/s wr, 713 op/s; 3605/287871 objects misplaced (1.252%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:59:42.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:42 smithi136 ceph-mon[109861]: 54.7 scrub starts 2024-08-17T17:59:42.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:42 smithi136 ceph-mon[115078]: pgmap v2348: 99 pgs: 1 active+recovering+remapped, 4 peering, 4 unknown, 90 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 45 MiB/s wr, 713 op/s; 3605/287871 objects misplaced (1.252%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:59:42.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:42 smithi136 ceph-mon[115078]: 54.7 scrub starts 2024-08-17T17:59:42.710 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 68 16 103204 103188 94.8312 76.5625 0.00296676 0.0105375 2024-08-17T17:59:42.736 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:42.767 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:42.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:42.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:42.869 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:42.904 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:42.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:42.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:43.006 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:43.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:42 smithi165 ceph-mon[105517]: pgmap v2348: 99 pgs: 1 active+recovering+remapped, 4 peering, 4 unknown, 90 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 45 MiB/s wr, 713 op/s; 3605/287871 objects misplaced (1.252%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:59:43.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:42 smithi165 ceph-mon[105517]: 54.7 scrub starts 2024-08-17T17:59:43.038 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:43.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:43.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:43.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:43.179 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:43.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:43.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:43.281 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:43.317 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:43.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:43.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:43.419 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:43.442 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:43.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:43.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:43.544 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:43.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:43.579 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:43.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:43.680 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:43.710 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 69 16 104438 104422 94.5745 77.125 0.0104925 0.0105692 2024-08-17T17:59:43.734 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:43.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:43.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:43.836 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:43.869 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:43.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:43.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:43.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:43 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-08-17T17:59:43.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:43 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-08-17T17:59:43.947 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:43 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-08-17T17:59:43.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:43.996 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:43.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:43.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:44.098 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:44.121 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:44.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:44.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:44.194 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T17:59:44.194 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T17:59:44.194 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-08-17T17:59:44.194 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T17:59:44.222 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:44.243 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:44.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:44.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:44.345 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:44.367 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:44.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:44.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:44.469 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:44.490 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:44.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:44.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:44.592 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:44.617 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:44.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:44.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:44.710 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 70 16 104808 104792 93.5537 23.125 0.00263338 0.010559 2024-08-17T17:59:44.719 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:44.744 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:44.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:44.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:44.846 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:44.871 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:44.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:44.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:44.874 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:44 smithi136 ceph-mon[115078]: pgmap v2349: 99 pgs: 1 active+recovering+remapped, 98 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s; 3379/305721 objects misplaced (1.105%); 3.4 MiB/s, 55 objects/s recovering 2024-08-17T17:59:44.875 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:44 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "17"}]': finished 2024-08-17T17:59:44.875 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:44 smithi136 ceph-mon[115078]: osdmap e1384: 8 total, 8 up, 7 in 2024-08-17T17:59:44.875 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:44 smithi136 ceph-mon[115078]: 54.14 scrub starts 2024-08-17T17:59:44.875 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:44 smithi136 ceph-mon[115078]: 54.19 scrub starts 2024-08-17T17:59:44.875 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:44 smithi136 ceph-mon[115078]: osdmap e1385: 8 total, 8 up, 7 in 2024-08-17T17:59:44.875 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:44 smithi136 ceph-mon[109861]: pgmap v2349: 99 pgs: 1 active+recovering+remapped, 98 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s; 3379/305721 objects misplaced (1.105%); 3.4 MiB/s, 55 objects/s recovering 2024-08-17T17:59:44.875 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:44 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "17"}]': finished 2024-08-17T17:59:44.875 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:44 smithi136 ceph-mon[109861]: osdmap e1384: 8 total, 8 up, 7 in 2024-08-17T17:59:44.875 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:44 smithi136 ceph-mon[109861]: 54.14 scrub starts 2024-08-17T17:59:44.875 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:44 smithi136 ceph-mon[109861]: 54.19 scrub starts 2024-08-17T17:59:44.875 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:44 smithi136 ceph-mon[109861]: osdmap e1385: 8 total, 8 up, 7 in 2024-08-17T17:59:44.973 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:44.996 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:44.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:44.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:44 smithi165 ceph-mon[105517]: pgmap v2349: 99 pgs: 1 active+recovering+remapped, 98 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s; 3379/305721 objects misplaced (1.105%); 3.4 MiB/s, 55 objects/s recovering 2024-08-17T17:59:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:44 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "17"}]': finished 2024-08-17T17:59:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:44 smithi165 ceph-mon[105517]: osdmap e1384: 8 total, 8 up, 7 in 2024-08-17T17:59:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:44 smithi165 ceph-mon[105517]: 54.14 scrub starts 2024-08-17T17:59:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:44 smithi165 ceph-mon[105517]: 54.19 scrub starts 2024-08-17T17:59:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:44 smithi165 ceph-mon[105517]: osdmap e1385: 8 total, 8 up, 7 in 2024-08-17T17:59:45.098 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:45.120 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:45.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:45.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:45.126 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-17T17:59:45.222 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:45.244 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:45.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:45.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:45.340 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:59:45.346 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:45.369 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:45.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:45.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:45.471 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:45.502 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:45.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:45.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:45.604 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:45.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:45.634 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:45.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:45.710 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 71 16 104896 104880 92.3136 5.5 0.0058418 0.0108256 2024-08-17T17:59:45.735 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:45.745 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:45 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-08-17T17:59:45.746 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:45 smithi165 ceph-mon[105517]: 54.17 scrub starts 2024-08-17T17:59:45.746 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:45 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:59:45.746 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:45 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:59:45.746 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:45 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "18"}]': finished 2024-08-17T17:59:45.746 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:45 smithi165 ceph-mon[105517]: osdmap e1386: 8 total, 8 up, 7 in 2024-08-17T17:59:45.772 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:45.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:45.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:45.814 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:45 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-08-17T17:59:45.814 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:45 smithi136 ceph-mon[115078]: 54.17 scrub starts 2024-08-17T17:59:45.814 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:45 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:59:45.814 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:45 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:59:45.814 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:45 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "18"}]': finished 2024-08-17T17:59:45.814 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:45 smithi136 ceph-mon[115078]: osdmap e1386: 8 total, 8 up, 7 in 2024-08-17T17:59:45.815 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:45 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-08-17T17:59:45.815 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:45 smithi136 ceph-mon[109861]: 54.17 scrub starts 2024-08-17T17:59:45.815 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:45 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:59:45.815 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:45 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:59:45.815 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:45 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "18"}]': finished 2024-08-17T17:59:45.815 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:45 smithi136 ceph-mon[109861]: osdmap e1386: 8 total, 8 up, 7 in 2024-08-17T17:59:45.866 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:59:45.873 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:45.896 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:45.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:45.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:45.998 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:46.023 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:46.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:46.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:46.125 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:46.147 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:46.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:46.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:46.249 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:46.270 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:46.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:46.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:46.372 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:46.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:46.395 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:46.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:46.496 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:46.519 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:46.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:46.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:46.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:59:46.622 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:46.672 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:46.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:46.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:46.710 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 72 16 105106 105090 91.2137 13.125 0.0032293 0.0108168 2024-08-17T17:59:46.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:46 smithi165 ceph-mon[105517]: pgmap v2352: 99 pgs: 1 active+recovering+remapped, 98 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 62 MiB/s wr, 991 op/s; 3379/305721 objects misplaced (1.105%); 2.4 MiB/s, 37 objects/s recovering 2024-08-17T17:59:46.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:46 smithi165 ceph-mon[105517]: 54.17 scrub starts 2024-08-17T17:59:46.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:46 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:59:46.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:46 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:59:46.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:46 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:59:46.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:46 smithi165 ceph-mon[105517]: osdmap e1387: 8 total, 8 up, 7 in 2024-08-17T17:59:46.774 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:46.796 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:46.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:46.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:46.898 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:46.923 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:46.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:46.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:46.934 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:59:46.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:46 smithi136 ceph-mon[109861]: pgmap v2352: 99 pgs: 1 active+recovering+remapped, 98 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 62 MiB/s wr, 991 op/s; 3379/305721 objects misplaced (1.105%); 2.4 MiB/s, 37 objects/s recovering 2024-08-17T17:59:46.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:46 smithi136 ceph-mon[109861]: 54.17 scrub starts 2024-08-17T17:59:46.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:46 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:59:46.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:46 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:59:46.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:46 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:59:46.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:46 smithi136 ceph-mon[109861]: osdmap e1387: 8 total, 8 up, 7 in 2024-08-17T17:59:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:46 smithi136 ceph-mon[115078]: pgmap v2352: 99 pgs: 1 active+recovering+remapped, 98 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 62 MiB/s wr, 991 op/s; 3379/305721 objects misplaced (1.105%); 2.4 MiB/s, 37 objects/s recovering 2024-08-17T17:59:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:46 smithi136 ceph-mon[115078]: 54.17 scrub starts 2024-08-17T17:59:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:46 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:59:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:46 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:59:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:46 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:59:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:46 smithi136 ceph-mon[115078]: osdmap e1387: 8 total, 8 up, 7 in 2024-08-17T17:59:47.025 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:47.037 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T17:59:47.037 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T17:59:47.037 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-08-17T17:59:47.037 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-08-17T17:59:47.037 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T17:59:47.046 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:47.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:47.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:47.149 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:47.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:47.172 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:47.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:47.273 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:47.295 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:47.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:47.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:47.397 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:47.418 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:47.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:47.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:47.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:47.542 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:47.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:47.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:47.645 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:47.667 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:47.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:47.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:47.707 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:47 smithi136 ceph-mon[109861]: 54.19 scrub starts 2024-08-17T17:59:47.707 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:47 smithi136 ceph-mon[109861]: 54.7 scrub starts 2024-08-17T17:59:47.707 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:47 smithi136 ceph-mon[109861]: osdmap e1388: 8 total, 8 up, 7 in 2024-08-17T17:59:47.707 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:59:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:59:47] "GET /metrics HTTP/1.1" 200 36565 "" "Prometheus/2.51.0" 2024-08-17T17:59:47.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:47 smithi136 ceph-mon[115078]: 54.19 scrub starts 2024-08-17T17:59:47.708 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:47 smithi136 ceph-mon[115078]: 54.7 scrub starts 2024-08-17T17:59:47.708 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:47 smithi136 ceph-mon[115078]: osdmap e1388: 8 total, 8 up, 7 in 2024-08-17T17:59:47.710 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 73 16 105106 105090 89.9643 0 - 0.0108168 2024-08-17T17:59:47.724 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:47 smithi165 ceph-mon[105517]: 54.19 scrub starts 2024-08-17T17:59:47.724 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:47 smithi165 ceph-mon[105517]: 54.7 scrub starts 2024-08-17T17:59:47.724 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:47 smithi165 ceph-mon[105517]: osdmap e1388: 8 total, 8 up, 7 in 2024-08-17T17:59:47.747 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-17T17:59:47.768 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:47.797 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:47.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:47.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:47.856 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:59:47.898 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:47.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:47.941 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:47.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:48.042 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:48.068 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:59:48.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:48.108 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:48.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:48.209 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:48.252 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:48.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:48.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:48.352 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T17:59:48.352 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-17T17:59:48.352 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T17:59:48.354 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:48.391 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:48.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:48.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:48.493 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:48.530 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:48.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:48.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:48.634 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:48.666 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:48.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:48.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:48.711 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 74 16 106063 106047 89.5567 29.9062 0.0444973 0.0111602 2024-08-17T17:59:48.767 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:48.799 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:48.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:48.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:48.901 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:48.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:48 smithi136 ceph-mon[109861]: pgmap v2355: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 97 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 29 MiB/s wr, 456 op/s; 6368/311196 objects degraded (2.046%); 3086/311196 objects misplaced (0.992%); 4.6 MiB/s, 73 objects/s recovering 2024-08-17T17:59:48.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:48 smithi136 ceph-mon[109861]: Health check failed: Degraded data redundancy: 6368/311196 objects degraded (2.046%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:59:48.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:48 smithi136 ceph-mon[109861]: 54.14 scrub starts 2024-08-17T17:59:48.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:48 smithi136 ceph-mon[115078]: pgmap v2355: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 97 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 29 MiB/s wr, 456 op/s; 6368/311196 objects degraded (2.046%); 3086/311196 objects misplaced (0.992%); 4.6 MiB/s, 73 objects/s recovering 2024-08-17T17:59:48.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:48 smithi136 ceph-mon[115078]: Health check failed: Degraded data redundancy: 6368/311196 objects degraded (2.046%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:59:48.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:48 smithi136 ceph-mon[115078]: 54.14 scrub starts 2024-08-17T17:59:48.948 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:48.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:48.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:48 smithi165 ceph-mon[105517]: pgmap v2355: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 97 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 29 MiB/s wr, 456 op/s; 6368/311196 objects degraded (2.046%); 3086/311196 objects misplaced (0.992%); 4.6 MiB/s, 73 objects/s recovering 2024-08-17T17:59:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:48 smithi165 ceph-mon[105517]: Health check failed: Degraded data redundancy: 6368/311196 objects degraded (2.046%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:59:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:48 smithi165 ceph-mon[105517]: 54.14 scrub starts 2024-08-17T17:59:49.050 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:49.096 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:49.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:49.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:49.198 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:49.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:49.233 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:49.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:49.334 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:49.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:49.373 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:49.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:49.474 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:49.511 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:49.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:49.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:49.613 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:49.651 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:49.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:49.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:49.711 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 75 16 107191 107175 89.3026 70.5 0.0261154 0.0111926 2024-08-17T17:59:49.752 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:49.790 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:49.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:49.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:49.892 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:49.928 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:49.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:49.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:50.030 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:50.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:50.066 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:50.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:50.167 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:50.203 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:50.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:50.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:50.305 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:50.341 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:50.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:50.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:50.443 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:50.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:50.484 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:50.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:50.585 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:50.611 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:50.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:50.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:50.711 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 76 16 108333 108317 89.0666 71.375 0.00966399 0.0112223 2024-08-17T17:59:50.713 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:50.752 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:50.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:50.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:50.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:50.878 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:50.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:50.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:50.925 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:50 smithi165 ceph-mon[105517]: pgmap v2357: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 97 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 46 MiB/s wr, 732 op/s; 6368/315099 objects degraded (2.021%); 3086/315099 objects misplaced (0.979%); 4.3 MiB/s, 68 objects/s recovering 2024-08-17T17:59:50.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:50 smithi136 ceph-mon[109861]: pgmap v2357: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 97 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 46 MiB/s wr, 732 op/s; 6368/315099 objects degraded (2.021%); 3086/315099 objects misplaced (0.979%); 4.3 MiB/s, 68 objects/s recovering 2024-08-17T17:59:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:50 smithi136 ceph-mon[115078]: pgmap v2357: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 97 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 46 MiB/s wr, 732 op/s; 6368/315099 objects degraded (2.021%); 3086/315099 objects misplaced (0.979%); 4.3 MiB/s, 68 objects/s recovering 2024-08-17T17:59:50.980 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:51.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:51.018 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:51.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:51.119 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:51.154 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:51.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:51.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:51.206 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T17:59:51.207 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T17:59:51.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T17:59:51.207 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T17:59:51.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.207 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:78749 53.12 53.68f6f672 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1320)", 2024-08-17T17:59:51.207 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:54:03.555049+0000", 2024-08-17T17:59:51.207 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 347.64905197500002, 2024-08-17T17:59:51.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71013271, 2024-08-17T17:59:51.207 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:51.207 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T17:59:51.208 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:51.208 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:59:51.208 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:59:51.208 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78749 2024-08-17T17:59:51.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.208 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:51.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:51.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.555049+0000", 2024-08-17T17:59:51.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:51.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.555049+0000", 2024-08-17T17:59:51.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:51.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.555049+0000", 2024-08-17T17:59:51.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:59:51.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:51.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.555093+0000", 2024-08-17T17:59:51.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4663999999999998e-05 2024-08-17T17:59:51.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:51.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.555108+0000", 2024-08-17T17:59:51.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4554e-05 2024-08-17T17:59:51.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:51.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.555114+0000", 2024-08-17T17:59:51.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.725e-06 2024-08-17T17:59:51.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.555160+0000", 2024-08-17T17:59:51.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5015000000000002e-05 2024-08-17T17:59:51.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T17:59:51.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.555167+0000", 2024-08-17T17:59:51.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.374e-06 2024-08-17T17:59:51.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:04.265172+0000", 2024-08-17T17:59:51.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71000474999999996 2024-08-17T17:59:51.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:51.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:04.265182+0000", 2024-08-17T17:59:51.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0518e-05 2024-08-17T17:59:51.212 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.212 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:51.212 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.212 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:78773 53.12 53.ee1fedf2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1320)", 2024-08-17T17:59:51.212 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:54:03.569417+0000", 2024-08-17T17:59:51.213 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 347.63468441399999, 2024-08-17T17:59:51.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.69579845299999998, 2024-08-17T17:59:51.213 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:51.213 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T17:59:51.213 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:51.213 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:59:51.214 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:59:51.214 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78773 2024-08-17T17:59:51.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.214 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:51.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:51.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.569417+0000", 2024-08-17T17:59:51.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:51.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.569417+0000", 2024-08-17T17:59:51.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:51.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.569416+0000", 2024-08-17T17:59:51.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:59:51.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:51.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.569483+0000", 2024-08-17T17:59:51.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7681000000000005e-05 2024-08-17T17:59:51.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:51.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.569494+0000", 2024-08-17T17:59:51.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0657e-05 2024-08-17T17:59:51.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:51.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.569501+0000", 2024-08-17T17:59:51.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2450000000000001e-06 2024-08-17T17:59:51.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.569545+0000", 2024-08-17T17:59:51.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3380999999999997e-05 2024-08-17T17:59:51.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T17:59:51.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.569551+0000", 2024-08-17T17:59:51.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8939999999999998e-06 2024-08-17T17:59:51.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:04.265209+0000", 2024-08-17T17:59:51.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.69565834500000001 2024-08-17T17:59:51.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:51.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:04.265215+0000", 2024-08-17T17:59:51.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5640000000000002e-06 2024-08-17T17:59:51.218 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.218 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:51.218 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.218 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:78804 53.12 53.84050552 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1320)", 2024-08-17T17:59:51.218 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:54:03.590164+0000", 2024-08-17T17:59:51.218 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 347.61393750000002, 2024-08-17T17:59:51.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.67506941600000003, 2024-08-17T17:59:51.219 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:51.219 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T17:59:51.219 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:51.219 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:59:51.219 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:59:51.219 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78804 2024-08-17T17:59:51.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.219 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:51.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:51.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.590164+0000", 2024-08-17T17:59:51.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:51.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.590164+0000", 2024-08-17T17:59:51.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:51.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.590163+0000", 2024-08-17T17:59:51.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:59:51.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:51.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.590218+0000", 2024-08-17T17:59:51.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5081999999999999e-05 2024-08-17T17:59:51.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:51.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.590230+0000", 2024-08-17T17:59:51.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2591e-05 2024-08-17T17:59:51.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:51.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.590257+0000", 2024-08-17T17:59:51.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7005999999999999e-05 2024-08-17T17:59:51.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.590308+0000", 2024-08-17T17:59:51.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0448999999999998e-05 2024-08-17T17:59:51.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T17:59:51.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.590316+0000", 2024-08-17T17:59:51.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1929999999999993e-06 2024-08-17T17:59:51.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:04.265228+0000", 2024-08-17T17:59:51.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.67491174300000001 2024-08-17T17:59:51.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:51.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:04.265233+0000", 2024-08-17T17:59:51.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.711e-06 2024-08-17T17:59:51.225 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.225 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:51.225 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.225 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:78812 53.12 53.4f4a012 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1320)", 2024-08-17T17:59:51.225 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:54:03.594679+0000", 2024-08-17T17:59:51.225 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 347.60942276200001, 2024-08-17T17:59:51.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.67057970200000006, 2024-08-17T17:59:51.225 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:51.225 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T17:59:51.225 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:51.225 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:59:51.226 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:59:51.226 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78812 2024-08-17T17:59:51.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.226 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:51.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:51.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.594679+0000", 2024-08-17T17:59:51.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:51.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.594679+0000", 2024-08-17T17:59:51.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:51.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.594677+0000", 2024-08-17T17:59:51.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:59:51.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:51.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.594755+0000", 2024-08-17T17:59:51.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7089999999999995e-05 2024-08-17T17:59:51.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:51.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.594767+0000", 2024-08-17T17:59:51.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2189e-05 2024-08-17T17:59:51.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:51.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.594774+0000", 2024-08-17T17:59:51.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7739999999999999e-06 2024-08-17T17:59:51.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.594829+0000", 2024-08-17T17:59:51.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5646000000000002e-05 2024-08-17T17:59:51.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T17:59:51.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.594837+0000", 2024-08-17T17:59:51.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1969999999999996e-06 2024-08-17T17:59:51.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:04.265249+0000", 2024-08-17T17:59:51.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.67041208600000002 2024-08-17T17:59:51.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:51.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:04.265258+0000", 2024-08-17T17:59:51.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8999999999999995e-06 2024-08-17T17:59:51.230 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.230 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:51.230 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.230 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:78832 53.12 53.c1e02812 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1320)", 2024-08-17T17:59:51.230 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:54:03.608949+0000", 2024-08-17T17:59:51.230 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 347.59515280900001, 2024-08-17T17:59:51.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.656335632, 2024-08-17T17:59:51.230 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:51.231 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T17:59:51.231 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:51.231 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:59:51.231 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:59:51.231 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78832 2024-08-17T17:59:51.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.233 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:51.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:51.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.608949+0000", 2024-08-17T17:59:51.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:51.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.608949+0000", 2024-08-17T17:59:51.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:51.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.608947+0000", 2024-08-17T17:59:51.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T17:59:51.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:51.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.609003+0000", 2024-08-17T17:59:51.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6137000000000001e-05 2024-08-17T17:59:51.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:51.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.609016+0000", 2024-08-17T17:59:51.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3413e-05 2024-08-17T17:59:51.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:51.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.609038+0000", 2024-08-17T17:59:51.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1614999999999999e-05 2024-08-17T17:59:51.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.609155+0000", 2024-08-17T17:59:51.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000117253 2024-08-17T17:59:51.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T17:59:51.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.609165+0000", 2024-08-17T17:59:51.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7010000000000008e-06 2024-08-17T17:59:51.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:04.265279+0000", 2024-08-17T17:59:51.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.65611418099999996 2024-08-17T17:59:51.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:51.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:04.265284+0000", 2024-08-17T17:59:51.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0019999999999997e-06 2024-08-17T17:59:51.237 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.237 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:51.237 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.238 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35091.0:78833 53.12 53.24fedfd2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1320)", 2024-08-17T17:59:51.238 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:54:03.609056+0000", 2024-08-17T17:59:51.238 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 347.59504549500002, 2024-08-17T17:59:51.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.656251056, 2024-08-17T17:59:51.238 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:51.238 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T17:59:51.238 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:51.238 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35091", 2024-08-17T17:59:51.238 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T17:59:51.238 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 78833 2024-08-17T17:59:51.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.238 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:51.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:51.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.609056+0000", 2024-08-17T17:59:51.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:51.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.609056+0000", 2024-08-17T17:59:51.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:51.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.609055+0000", 2024-08-17T17:59:51.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T17:59:51.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:51.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.609092+0000", 2024-08-17T17:59:51.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6779999999999997e-05 2024-08-17T17:59:51.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:51.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.609100+0000", 2024-08-17T17:59:51.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3760000000000002e-06 2024-08-17T17:59:51.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:51.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.609105+0000", 2024-08-17T17:59:51.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4269999999999999e-06 2024-08-17T17:59:51.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.609186+0000", 2024-08-17T17:59:51.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1038999999999994e-05 2024-08-17T17:59:51.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T17:59:51.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:03.609192+0000", 2024-08-17T17:59:51.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.66e-06 2024-08-17T17:59:51.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:04.265302+0000", 2024-08-17T17:59:51.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.65611039599999998 2024-08-17T17:59:51.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:51.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:54:04.265307+0000", 2024-08-17T17:59:51.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8489999999999997e-06 2024-08-17T17:59:51.242 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.242 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:51.242 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.242 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:104813 54.11 54.64e93111 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T17:59:51.242 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.630215+0000", 2024-08-17T17:59:51.243 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.5738867809999997, 2024-08-17T17:59:51.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.83036598800000005, 2024-08-17T17:59:51.243 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:51.243 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T17:59:51.243 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:51.243 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:51.243 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:51.243 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104813 2024-08-17T17:59:51.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.243 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:51.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:51.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.630215+0000", 2024-08-17T17:59:51.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:51.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.630215+0000", 2024-08-17T17:59:51.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:51.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.630213+0000", 2024-08-17T17:59:51.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:59:51.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:51.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.630324+0000", 2024-08-17T17:59:51.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011047 2024-08-17T17:59:51.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:51.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.630338+0000", 2024-08-17T17:59:51.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.452e-05 2024-08-17T17:59:51.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:51.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.630344+0000", 2024-08-17T17:59:51.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3029999999999999e-06 2024-08-17T17:59:51.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.630399+0000", 2024-08-17T17:59:51.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5204999999999997e-05 2024-08-17T17:59:51.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T17:59:51.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.630409+0000", 2024-08-17T17:59:51.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0363999999999999e-05 2024-08-17T17:59:51.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460567+0000", 2024-08-17T17:59:51.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.83015739499999996 2024-08-17T17:59:51.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:51.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460581+0000", 2024-08-17T17:59:51.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4001e-05 2024-08-17T17:59:51.247 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.247 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:51.247 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.247 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:104818 54.11 54.1190c0f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T17:59:51.247 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.637872+0000", 2024-08-17T17:59:51.247 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.5662295799999999, 2024-08-17T17:59:51.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.82274040599999998, 2024-08-17T17:59:51.247 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:51.248 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T17:59:51.248 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:51.248 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:51.248 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:51.248 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104818 2024-08-17T17:59:51.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.248 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:51.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:51.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.637872+0000", 2024-08-17T17:59:51.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:51.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.637872+0000", 2024-08-17T17:59:51.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:51.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.637870+0000", 2024-08-17T17:59:51.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:59:51.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:51.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.637930+0000", 2024-08-17T17:59:51.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9210999999999999e-05 2024-08-17T17:59:51.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:51.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.637943+0000", 2024-08-17T17:59:51.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3332e-05 2024-08-17T17:59:51.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:51.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.637948+0000", 2024-08-17T17:59:51.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4659999999999998e-06 2024-08-17T17:59:51.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.637992+0000", 2024-08-17T17:59:51.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3899000000000001e-05 2024-08-17T17:59:51.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T17:59:51.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.638000+0000", 2024-08-17T17:59:51.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9039999999999992e-06 2024-08-17T17:59:51.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460604+0000", 2024-08-17T17:59:51.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.82260419100000004 2024-08-17T17:59:51.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:51.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460612+0000", 2024-08-17T17:59:51.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7729999999999999e-06 2024-08-17T17:59:51.253 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.253 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:51.253 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.253 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:104840 54.11 54.d5652851 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T17:59:51.253 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.667423+0000", 2024-08-17T17:59:51.253 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.5366788539999998, 2024-08-17T17:59:51.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.79322487200000003, 2024-08-17T17:59:51.254 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:51.254 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T17:59:51.254 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:51.254 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:51.254 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:51.254 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104840 2024-08-17T17:59:51.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.254 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:51.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:51.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.667423+0000", 2024-08-17T17:59:51.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:51.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.667423+0000", 2024-08-17T17:59:51.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:51.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.667422+0000", 2024-08-17T17:59:51.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T17:59:51.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:51.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.667485+0000", 2024-08-17T17:59:51.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2741000000000001e-05 2024-08-17T17:59:51.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:51.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.667491+0000", 2024-08-17T17:59:51.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8699999999999997e-06 2024-08-17T17:59:51.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:51.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.667494+0000", 2024-08-17T17:59:51.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5930000000000002e-06 2024-08-17T17:59:51.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.667535+0000", 2024-08-17T17:59:51.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1273999999999998e-05 2024-08-17T17:59:51.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T17:59:51.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.667543+0000", 2024-08-17T17:59:51.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1659999999999997e-06 2024-08-17T17:59:51.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460636+0000", 2024-08-17T17:59:51.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.79309297400000001 2024-08-17T17:59:51.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:51.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460647+0000", 2024-08-17T17:59:51.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1863e-05 2024-08-17T17:59:51.258 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.258 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:51.258 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.258 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:104844 54.11 54.e06561d1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T17:59:51.258 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.672468+0000", 2024-08-17T17:59:51.259 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.5316334869999997, 2024-08-17T17:59:51.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.78822546199999999, 2024-08-17T17:59:51.259 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:51.259 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T17:59:51.259 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:51.259 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:51.259 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:51.259 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104844 2024-08-17T17:59:51.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.259 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:51.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:51.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.672468+0000", 2024-08-17T17:59:51.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:51.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.672468+0000", 2024-08-17T17:59:51.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:51.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.672467+0000", 2024-08-17T17:59:51.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:59:51.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:51.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.672534+0000", 2024-08-17T17:59:51.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6926000000000003e-05 2024-08-17T17:59:51.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:51.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.672544+0000", 2024-08-17T17:59:51.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.022e-05 2024-08-17T17:59:51.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:51.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.672549+0000", 2024-08-17T17:59:51.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.809e-06 2024-08-17T17:59:51.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.672586+0000", 2024-08-17T17:59:51.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6585999999999998e-05 2024-08-17T17:59:51.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T17:59:51.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.672593+0000", 2024-08-17T17:59:51.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8269999999999998e-06 2024-08-17T17:59:51.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460683+0000", 2024-08-17T17:59:51.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.78808987600000002 2024-08-17T17:59:51.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:51.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460693+0000", 2024-08-17T17:59:51.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0138e-05 2024-08-17T17:59:51.263 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.263 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:51.263 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.263 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:104848 54.11 54.d7e6d951 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T17:59:51.264 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.673087+0000", 2024-08-17T17:59:51.264 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.5310148410000002, 2024-08-17T17:59:51.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.78763660400000002, 2024-08-17T17:59:51.264 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:51.264 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T17:59:51.264 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:51.264 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:51.264 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:51.264 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104848 2024-08-17T17:59:51.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.264 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:51.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:51.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.673087+0000", 2024-08-17T17:59:51.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:51.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.673087+0000", 2024-08-17T17:59:51.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:51.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.673086+0000", 2024-08-17T17:59:51.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:59:51.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:51.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.673524+0000", 2024-08-17T17:59:51.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000438591 2024-08-17T17:59:51.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:51.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.673539+0000", 2024-08-17T17:59:51.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4791e-05 2024-08-17T17:59:51.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:51.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.673546+0000", 2024-08-17T17:59:51.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2200000000000003e-06 2024-08-17T17:59:51.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.673598+0000", 2024-08-17T17:59:51.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1418000000000001e-05 2024-08-17T17:59:51.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T17:59:51.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.673605+0000", 2024-08-17T17:59:51.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.866e-06 2024-08-17T17:59:51.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460712+0000", 2024-08-17T17:59:51.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.78710790600000002 2024-08-17T17:59:51.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:51.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460723+0000", 2024-08-17T17:59:51.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0771000000000001e-05 2024-08-17T17:59:51.269 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.269 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:51.269 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.269 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:104917 54.11 54.eda9ea11 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T17:59:51.269 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.725191+0000", 2024-08-17T17:59:51.270 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.4789099380000001, 2024-08-17T17:59:51.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.73558475400000001, 2024-08-17T17:59:51.270 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:51.270 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T17:59:51.270 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:51.270 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:51.270 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:51.270 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104917 2024-08-17T17:59:51.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.270 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:51.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:51.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.725191+0000", 2024-08-17T17:59:51.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:51.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.725191+0000", 2024-08-17T17:59:51.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:51.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.725191+0000", 2024-08-17T17:59:51.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T17:59:51.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:51.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.725274+0000", 2024-08-17T17:59:51.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3373000000000003e-05 2024-08-17T17:59:51.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:51.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.725287+0000", 2024-08-17T17:59:51.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3025e-05 2024-08-17T17:59:51.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:51.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.725294+0000", 2024-08-17T17:59:51.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6270000000000002e-06 2024-08-17T17:59:51.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.725350+0000", 2024-08-17T17:59:51.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5897999999999999e-05 2024-08-17T17:59:51.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T17:59:51.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.725358+0000", 2024-08-17T17:59:51.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8220000000000007e-06 2024-08-17T17:59:51.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460768+0000", 2024-08-17T17:59:51.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.73540999399999996 2024-08-17T17:59:51.274 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:51.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:51.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460776+0000", 2024-08-17T17:59:51.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6920000000000008e-06 2024-08-17T17:59:51.275 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.276 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:51.276 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.276 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:104940 54.11 54.f8fdfd11 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T17:59:51.276 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.741796+0000", 2024-08-17T17:59:51.276 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.4623058540000002, 2024-08-17T17:59:51.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71900983799999996, 2024-08-17T17:59:51.276 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:51.276 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T17:59:51.276 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:51.276 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:51.276 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:51.277 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104940 2024-08-17T17:59:51.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.277 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:51.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:51.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.741796+0000", 2024-08-17T17:59:51.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:51.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.741796+0000", 2024-08-17T17:59:51.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:51.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.741794+0000", 2024-08-17T17:59:51.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T17:59:51.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:51.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.741827+0000", 2024-08-17T17:59:51.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2960000000000003e-05 2024-08-17T17:59:51.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:51.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.741844+0000", 2024-08-17T17:59:51.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7703e-05 2024-08-17T17:59:51.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:51.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.741852+0000", 2024-08-17T17:59:51.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6040000000000004e-06 2024-08-17T17:59:51.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.741897+0000", 2024-08-17T17:59:51.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4941999999999998e-05 2024-08-17T17:59:51.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T17:59:51.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.741905+0000", 2024-08-17T17:59:51.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1289999999999996e-06 2024-08-17T17:59:51.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460796+0000", 2024-08-17T17:59:51.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71889108999999995 2024-08-17T17:59:51.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:51.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460805+0000", 2024-08-17T17:59:51.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1630000000000002e-06 2024-08-17T17:59:51.281 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.281 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:51.281 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.281 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:104958 54.11 54.e8b0e6d1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T17:59:51.281 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.760286+0000", 2024-08-17T17:59:51.281 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.4438152500000001, 2024-08-17T17:59:51.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.70054623599999999, 2024-08-17T17:59:51.281 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:51.281 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T17:59:51.281 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:51.282 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:51.282 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:51.282 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104958 2024-08-17T17:59:51.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.282 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:51.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:51.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.760286+0000", 2024-08-17T17:59:51.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:51.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.760286+0000", 2024-08-17T17:59:51.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:51.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.760285+0000", 2024-08-17T17:59:51.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T17:59:51.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:51.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.760326+0000", 2024-08-17T17:59:51.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0945000000000002e-05 2024-08-17T17:59:51.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:51.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.760341+0000", 2024-08-17T17:59:51.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5248999999999999e-05 2024-08-17T17:59:51.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:51.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.760347+0000", 2024-08-17T17:59:51.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6919999999999999e-06 2024-08-17T17:59:51.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.760399+0000", 2024-08-17T17:59:51.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2293999999999998e-05 2024-08-17T17:59:51.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T17:59:51.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.760407+0000", 2024-08-17T17:59:51.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4150000000000002e-06 2024-08-17T17:59:51.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460824+0000", 2024-08-17T17:59:51.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.70041780799999998 2024-08-17T17:59:51.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:51.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460832+0000", 2024-08-17T17:59:51.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0450000000000001e-06 2024-08-17T17:59:51.286 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.286 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:51.286 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.286 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:105002 54.11 54.3957dd31 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T17:59:51.286 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.797619+0000", 2024-08-17T17:59:51.286 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.4064828939999998, 2024-08-17T17:59:51.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.66324079499999999, 2024-08-17T17:59:51.286 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:51.287 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T17:59:51.287 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:51.287 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:51.287 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:51.287 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 105002 2024-08-17T17:59:51.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.287 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:51.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:51.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.797619+0000", 2024-08-17T17:59:51.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:51.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.797619+0000", 2024-08-17T17:59:51.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:51.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.797617+0000", 2024-08-17T17:59:51.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:59:51.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:51.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.797682+0000", 2024-08-17T17:59:51.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4725000000000001e-05 2024-08-17T17:59:51.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:51.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.797696+0000", 2024-08-17T17:59:51.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4202999999999999e-05 2024-08-17T17:59:51.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:51.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.797703+0000", 2024-08-17T17:59:51.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4350000000000002e-06 2024-08-17T17:59:51.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.797745+0000", 2024-08-17T17:59:51.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2514e-05 2024-08-17T17:59:51.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T17:59:51.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.797753+0000", 2024-08-17T17:59:51.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4769999999999997e-06 2024-08-17T17:59:51.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460851+0000", 2024-08-17T17:59:51.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.66309843099999999 2024-08-17T17:59:51.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:51.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460859+0000", 2024-08-17T17:59:51.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1589999999999993e-06 2024-08-17T17:59:51.290 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.290 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:51.291 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.291 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:105025 54.11 54.479d4851 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T17:59:51.291 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.820146+0000", 2024-08-17T17:59:51.291 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.3839555199999998, 2024-08-17T17:59:51.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.64075398500000003, 2024-08-17T17:59:51.291 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:51.291 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T17:59:51.291 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:51.291 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:51.291 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:51.291 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 105025 2024-08-17T17:59:51.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.292 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:51.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:51.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820146+0000", 2024-08-17T17:59:51.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:51.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820146+0000", 2024-08-17T17:59:51.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:51.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820145+0000", 2024-08-17T17:59:51.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:59:51.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:51.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820212+0000", 2024-08-17T17:59:51.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6554e-05 2024-08-17T17:59:51.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:51.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820222+0000", 2024-08-17T17:59:51.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0061000000000001e-05 2024-08-17T17:59:51.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:51.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820226+0000", 2024-08-17T17:59:51.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8289999999999998e-06 2024-08-17T17:59:51.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820269+0000", 2024-08-17T17:59:51.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2595000000000001e-05 2024-08-17T17:59:51.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T17:59:51.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820278+0000", 2024-08-17T17:59:51.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8189999999999998e-06 2024-08-17T17:59:51.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460890+0000", 2024-08-17T17:59:51.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.64061211900000004 2024-08-17T17:59:51.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:51.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460900+0000", 2024-08-17T17:59:51.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9019999999999994e-06 2024-08-17T17:59:51.296 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.296 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:51.296 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.296 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:105028 54.11 54.30ac10b1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T17:59:51.296 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.820327+0000", 2024-08-17T17:59:51.296 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.3837745999999997, 2024-08-17T17:59:51.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.64060198099999999, 2024-08-17T17:59:51.297 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:51.297 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T17:59:51.297 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:51.297 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:51.297 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:51.297 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 105028 2024-08-17T17:59:51.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.297 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:51.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:51.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820327+0000", 2024-08-17T17:59:51.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:51.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820327+0000", 2024-08-17T17:59:51.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:51.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820326+0000", 2024-08-17T17:59:51.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T17:59:51.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:51.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820386+0000", 2024-08-17T17:59:51.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9802000000000001e-05 2024-08-17T17:59:51.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:51.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820395+0000", 2024-08-17T17:59:51.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9719999999999998e-06 2024-08-17T17:59:51.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:51.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820400+0000", 2024-08-17T17:59:51.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3319999999999999e-06 2024-08-17T17:59:51.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820440+0000", 2024-08-17T17:59:51.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9567999999999998e-05 2024-08-17T17:59:51.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T17:59:51.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820447+0000", 2024-08-17T17:59:51.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7510000000000002e-06 2024-08-17T17:59:51.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460921+0000", 2024-08-17T17:59:51.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.64047475200000004 2024-08-17T17:59:51.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:51.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460929+0000", 2024-08-17T17:59:51.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3089999999999998e-06 2024-08-17T17:59:51.301 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.301 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:51.301 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.301 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:105029 54.11 54.123b9a31 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T17:59:51.301 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.820409+0000", 2024-08-17T17:59:51.301 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.3836926119999999, 2024-08-17T17:59:51.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.64055096199999995, 2024-08-17T17:59:51.301 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:51.302 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T17:59:51.302 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:51.302 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:51.302 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:51.302 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 105029 2024-08-17T17:59:51.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.302 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:51.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:51.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820409+0000", 2024-08-17T17:59:51.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:51.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820409+0000", 2024-08-17T17:59:51.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:51.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820408+0000", 2024-08-17T17:59:51.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T17:59:51.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:51.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820474+0000", 2024-08-17T17:59:51.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6168999999999996e-05 2024-08-17T17:59:51.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:51.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820483+0000", 2024-08-17T17:59:51.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6859999999999995e-06 2024-08-17T17:59:51.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:51.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820488+0000", 2024-08-17T17:59:51.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2009999999999998e-06 2024-08-17T17:59:51.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820524+0000", 2024-08-17T17:59:51.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5641e-05 2024-08-17T17:59:51.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T17:59:51.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820530+0000", 2024-08-17T17:59:51.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9220000000000002e-06 2024-08-17T17:59:51.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460952+0000", 2024-08-17T17:59:51.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.64042236799999996 2024-08-17T17:59:51.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:51.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460960+0000", 2024-08-17T17:59:51.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5109999999999997e-06 2024-08-17T17:59:51.306 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.306 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:51.306 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.307 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:105032 54.11 54.79e941d1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T17:59:51.307 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.824271+0000", 2024-08-17T17:59:51.307 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.3798306309999999, 2024-08-17T17:59:51.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.63671803000000005, 2024-08-17T17:59:51.307 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:51.307 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T17:59:51.307 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:51.307 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:51.308 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:51.308 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 105032 2024-08-17T17:59:51.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.308 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:51.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:51.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.824271+0000", 2024-08-17T17:59:51.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:51.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.824271+0000", 2024-08-17T17:59:51.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:51.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.824270+0000", 2024-08-17T17:59:51.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:59:51.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:51.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.824331+0000", 2024-08-17T17:59:51.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0705000000000002e-05 2024-08-17T17:59:51.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:51.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.824341+0000", 2024-08-17T17:59:51.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0609e-05 2024-08-17T17:59:51.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:51.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.824346+0000", 2024-08-17T17:59:51.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0320000000000003e-06 2024-08-17T17:59:51.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.824391+0000", 2024-08-17T17:59:51.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5136000000000003e-05 2024-08-17T17:59:51.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T17:59:51.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.824400+0000", 2024-08-17T17:59:51.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0409999999999998e-06 2024-08-17T17:59:51.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460981+0000", 2024-08-17T17:59:51.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.63658124699999996 2024-08-17T17:59:51.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:51.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460989+0000", 2024-08-17T17:59:51.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0660000000000004e-06 2024-08-17T17:59:51.312 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.312 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:51.313 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.313 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:105048 54.11 54.b0d54f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T17:59:51.313 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.850427+0000", 2024-08-17T17:59:51.313 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.3536740600000003, 2024-08-17T17:59:51.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61059117200000002, 2024-08-17T17:59:51.313 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:51.313 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T17:59:51.313 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T17:59:51.313 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T17:59:51.313 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T17:59:51.314 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 105048 2024-08-17T17:59:51.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.314 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:51.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:51.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.850427+0000", 2024-08-17T17:59:51.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:51.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.850427+0000", 2024-08-17T17:59:51.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:51.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:51.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.850414+0000", 2024-08-17T17:59:51.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999871 2024-08-17T17:59:51.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:51.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.850494+0000", 2024-08-17T17:59:51.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9821999999999996e-05 2024-08-17T17:59:51.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:51.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.850507+0000", 2024-08-17T17:59:51.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2481e-05 2024-08-17T17:59:51.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:51.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.850514+0000", 2024-08-17T17:59:51.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9959999999999996e-06 2024-08-17T17:59:51.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.850550+0000", 2024-08-17T17:59:51.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6668999999999997e-05 2024-08-17T17:59:51.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T17:59:51.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.850556+0000", 2024-08-17T17:59:51.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5829999999999997e-06 2024-08-17T17:59:51.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:51.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.461011+0000", 2024-08-17T17:59:51.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61045490099999999 2024-08-17T17:59:51.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:51.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:51.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T17:59:51.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.461019+0000", 2024-08-17T17:59:51.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7030000000000006e-06 2024-08-17T17:59:51.318 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.318 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:51.318 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.318 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:51.318 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:51.318 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T17:59:51.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:51.320 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:51.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:51.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:51.458 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:51.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:51.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:51.560 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:51.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:51.592 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:51.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:51.694 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:51.711 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 77 16 109429 109413 88.7993 68.5 0.00777827 0.0112563 2024-08-17T17:59:51.731 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:51.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:51.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:51.833 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:51.870 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:51.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:51.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:51.972 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:52.010 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:52.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:52.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:52.112 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:52.144 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:52.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:52.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:52.246 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:52.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:52.270 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:52.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:52.371 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:52.410 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:52.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:52.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:52.473 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-17T17:59:52.473 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:52 smithi165 ceph-mon[105517]: pgmap v2358: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 97 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 33 MiB/s wr, 520 op/s; 6368/315099 objects degraded (2.021%); 3086/315099 objects misplaced (0.979%); 3.1 MiB/s, 48 objects/s recovering 2024-08-17T17:59:52.474 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:59:52.474 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:59:52.512 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:52.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:52.535 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:52.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:52.636 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:52.671 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:52.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:52.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:52.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:52 smithi136 ceph-mon[109861]: pgmap v2358: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 97 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 33 MiB/s wr, 520 op/s; 6368/315099 objects degraded (2.021%); 3086/315099 objects misplaced (0.979%); 3.1 MiB/s, 48 objects/s recovering 2024-08-17T17:59:52.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:59:52.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:59:52.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:52 smithi136 ceph-mon[115078]: pgmap v2358: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 97 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 33 MiB/s wr, 520 op/s; 6368/315099 objects degraded (2.021%); 3086/315099 objects misplaced (0.979%); 3.1 MiB/s, 48 objects/s recovering 2024-08-17T17:59:52.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T17:59:52.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:59:52.711 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 78 16 110409 110393 88.4461 61.25 0.00503054 0.0113013 2024-08-17T17:59:52.773 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:52.801 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:59:52.812 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:52.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:52.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:52.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:52.952 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:52.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:52.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:53.054 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:53.076 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:53.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:53.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:53.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:53.212 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:53.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:53.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:53.314 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:53.348 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:53.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:53.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:53.450 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:53.485 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:53.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:53.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:53.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:53 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 6368/328023 objects degraded (1.941%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:59:53.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:53.624 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:53.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:53.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:53 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 6368/328023 objects degraded (1.941%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:59:53.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:53 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 6368/328023 objects degraded (1.941%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:59:53.711 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 79 16 111456 111440 88.1547 65.4375 0.0137375 0.0113384 2024-08-17T17:59:53.726 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:53.760 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:53.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:53.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:53.861 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:53.899 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:53.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:53.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:54.001 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:54.038 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:54.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:54.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:54.140 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:54.175 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:54.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:54.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:54.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:59:54.310 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:54.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:54.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:59:54.397 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:54 smithi165 ceph-mon[105517]: pgmap v2359: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s; 6368/328023 objects degraded (1.941%); 9014/328023 objects misplaced (2.748%); 6.2 MiB/s, 98 objects/s recovering 2024-08-17T17:59:54.412 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:54.445 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:54.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:54.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:54.547 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:54.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:54.575 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:54.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:54.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:54.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:54 smithi136 ceph-mon[109861]: pgmap v2359: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s; 6368/328023 objects degraded (1.941%); 9014/328023 objects misplaced (2.748%); 6.2 MiB/s, 98 objects/s recovering 2024-08-17T17:59:54.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:54 smithi136 ceph-mon[115078]: pgmap v2359: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s; 6368/328023 objects degraded (1.941%); 9014/328023 objects misplaced (2.748%); 6.2 MiB/s, 98 objects/s recovering 2024-08-17T17:59:54.710 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:54.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:54.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:54.712 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:2024-08-17T17:59:54.711789+0000 min lat: 0.0012629 max lat: 2.15603 avg lat: 0.0113854 2024-08-17T17:59:54.712 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:59:54.712 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 80 16 112400 112384 87.7902 59 0.0177785 0.0113854 2024-08-17T17:59:54.812 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:54.852 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:54.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:54.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:54.953 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:54.992 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:54.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:54.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:55.094 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:55.127 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:55.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:55.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:55.229 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:55.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:55.270 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:55.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:55.371 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:55.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:55.409 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:55.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:55.510 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:55.534 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:55.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:55.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:55.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:55.671 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:55.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:55.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:55.711 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 81 16 113355 113339 87.4432 59.6875 0.00247436 0.0114287 2024-08-17T17:59:55.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:55.794 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:55.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:55.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:55.851 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T17:59:55.851 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T17:59:55.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.851 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.35370.0:112803 54.b e1388/1367 54:d02a5c1e:::benchmark_data_smithi136_273969_object112802:head v 1388'6934, mlcod=1388'6932)", 2024-08-17T17:59:55.851 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:55.149612+0000", 2024-08-17T17:59:55.851 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0061600049999999996, 2024-08-17T17:59:55.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006190442, 2024-08-17T17:59:55.851 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:55.851 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T17:59:55.851 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:55.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:55.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:55.149612+0000", 2024-08-17T17:59:55.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:55.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:55.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:55.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:55.149612+0000", 2024-08-17T17:59:55.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:55.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:55.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:55.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:55.149611+0000", 2024-08-17T17:59:55.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:59:55.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:55.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:55.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:55.149666+0000", 2024-08-17T17:59:55.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4756000000000002e-05 2024-08-17T17:59:55.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:55.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:55.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:55.149676+0000", 2024-08-17T17:59:55.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0492000000000001e-05 2024-08-17T17:59:55.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:55.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:55.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:55.149682+0000", 2024-08-17T17:59:55.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8680000000000004e-06 2024-08-17T17:59:55.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:55.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:55.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:55.150290+0000", 2024-08-17T17:59:55.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00060827500000000003 2024-08-17T17:59:55.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:55.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:55.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:55.150310+0000", 2024-08-17T17:59:55.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0134999999999999e-05 2024-08-17T17:59:55.854 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:55.855 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:55.855 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:55.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:55.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.855 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.35370.0:112807 54.5 e1388/1381 54:a04fca8a:::benchmark_data_smithi136_273969_object112806:head v 1388'6442, mlcod=1388'6440)", 2024-08-17T17:59:55.855 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:55.153855+0000", 2024-08-17T17:59:55.855 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.001916264, 2024-08-17T17:59:55.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020194639999999999, 2024-08-17T17:59:55.855 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:55.855 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:59:55.855 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:55.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:55.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:55.153855+0000", 2024-08-17T17:59:55.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:55.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:55.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:55.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:55.153855+0000", 2024-08-17T17:59:55.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:55.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:55.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:55.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:55.153855+0000", 2024-08-17T17:59:55.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T17:59:55.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:55.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:55.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:55.153877+0000", 2024-08-17T17:59:55.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1562000000000001e-05 2024-08-17T17:59:55.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:55.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:55.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:55.153884+0000", 2024-08-17T17:59:55.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0570000000000004e-06 2024-08-17T17:59:55.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:55.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:55.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:55.153888+0000", 2024-08-17T17:59:55.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4220000000000002e-06 2024-08-17T17:59:55.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:55.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T17:59:55.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:55.153957+0000", 2024-08-17T17:59:55.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8688000000000004e-05 2024-08-17T17:59:55.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:55.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T17:59:55.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:55.153976+0000", 2024-08-17T17:59:55.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8902e-05 2024-08-17T17:59:55.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:55.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T17:59:55.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:55.155859+0000", 2024-08-17T17:59:55.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018835029999999999 2024-08-17T17:59:55.858 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:55.858 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:55.859 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:55.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:55.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.859 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.35370.0:112808 54.b e1388/1367)", 2024-08-17T17:59:55.859 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:55.155368+0000", 2024-08-17T17:59:55.859 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00040365600000000001, 2024-08-17T17:59:55.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00057416300000000004, 2024-08-17T17:59:55.859 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T17:59:55.859 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-08-17T17:59:55.859 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T17:59:55.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T17:59:55.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:55.155368+0000", 2024-08-17T17:59:55.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:55.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:55.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T17:59:55.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:55.155368+0000", 2024-08-17T17:59:55.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T17:59:55.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:55.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T17:59:55.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:55.155367+0000", 2024-08-17T17:59:55.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T17:59:55.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:55.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T17:59:55.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:55.155400+0000", 2024-08-17T17:59:55.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2530999999999997e-05 2024-08-17T17:59:55.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:55.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T17:59:55.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:55.155411+0000", 2024-08-17T17:59:55.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0917999999999999e-05 2024-08-17T17:59:55.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T17:59:55.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T17:59:55.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T17:59:55.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:55.155417+0000", 2024-08-17T17:59:55.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0140000000000004e-06 2024-08-17T17:59:55.861 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:55.862 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T17:59:55.862 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:55.862 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T17:59:55.862 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-08-17T17:59:55.862 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 3 2024-08-17T17:59:55.862 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T17:59:55.896 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:55.929 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:55.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:55.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:56.031 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:56.072 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:56.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:56.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:56.174 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:56.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:56.218 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:56.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:56.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:56.333 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:59:56.349 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:56.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:56.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:56.451 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:56.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:56.494 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:56.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:56.586 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:56 smithi136 ceph-mon[109861]: pgmap v2360: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 42 MiB/s wr, 678 op/s; 6368/328023 objects degraded (1.941%); 9014/328023 objects misplaced (2.748%); 3.2 MiB/s, 51 objects/s recovering 2024-08-17T17:59:56.586 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:56 smithi136 ceph-mon[115078]: pgmap v2360: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 42 MiB/s wr, 678 op/s; 6368/328023 objects degraded (1.941%); 9014/328023 objects misplaced (2.748%); 3.2 MiB/s, 51 objects/s recovering 2024-08-17T17:59:56.594 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:56.631 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:56.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:56.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:56.712 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 82 15 114381 114366 87.1596 64.1875 0.0183446 0.0114677 2024-08-17T17:59:56.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:56 smithi165 ceph-mon[105517]: pgmap v2360: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 42 MiB/s wr, 678 op/s; 6368/328023 objects degraded (1.941%); 9014/328023 objects misplaced (2.748%); 3.2 MiB/s, 51 objects/s recovering 2024-08-17T17:59:56.770 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:56.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:56.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:56.820 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-17T17:59:56.872 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:56.905 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:56.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:56.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:57.007 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:57.040 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:57.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:57.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:57.142 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:57.185 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:59:57.190 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:57.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:57.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:57.293 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:57.332 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:57.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:57.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:57.434 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:57.465 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:57.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:57.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:57.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:57.605 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:57.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:57.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:59:57.707 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:57.711 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 83 16 115436 115420 86.9031 65.875 0.0173338 0.0115011 2024-08-17T17:59:57.730 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:57.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:57.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:57.832 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:57.871 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:57.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:57.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:57.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 17:59:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:17:59:57] "GET /metrics HTTP/1.1" 200 36569 "" "Prometheus/2.51.0" 2024-08-17T17:59:57.973 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:58.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:58.011 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:58.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:58.112 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:58.149 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:58.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:58.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:58.250 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:59:58.298 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:58.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:58.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:59:58.400 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:58.441 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:58.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:58.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:58.527 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:58 smithi136 ceph-mon[109861]: pgmap v2361: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 51 MiB/s wr, 808 op/s; 6368/335445 objects degraded (1.898%); 8547/335445 objects misplaced (2.548%); 4.6 MiB/s, 73 objects/s recovering 2024-08-17T17:59:58.527 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:58 smithi136 ceph-mon[115078]: pgmap v2361: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 51 MiB/s wr, 808 op/s; 6368/335445 objects degraded (1.898%); 8547/335445 objects misplaced (2.548%); 4.6 MiB/s, 73 objects/s recovering 2024-08-17T17:59:58.544 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:58.581 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:58.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:58.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:58.627 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:58 smithi165 ceph-mon[105517]: pgmap v2361: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 51 MiB/s wr, 808 op/s; 6368/335445 objects degraded (1.898%); 8547/335445 objects misplaced (2.548%); 4.6 MiB/s, 73 objects/s recovering 2024-08-17T17:59:58.682 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:58.712 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 84 16 116609 116593 86.7412 73.3125 0.0139048 0.0115232 2024-08-17T17:59:58.716 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:58.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:58.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:58.818 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:59:58.853 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:58.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:58.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:59:58.955 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:58.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:58.993 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:58.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:59.094 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:59.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:59.136 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:59.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:59.237 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:59.273 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:59.273 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T17:59:59.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:59.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:59.311 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T17:59:59.311 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T17:59:59.311 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-08-17T17:59:59.311 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-08-17T17:59:59.311 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T17:59:59.375 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:59:59.412 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:59.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:59.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:59:59.427 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:59 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/30157985' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:59:59.427 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 17:59:59 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:59:59.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:59 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/30157985' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:59:59.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 17:59:59 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:59:59.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:59 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/30157985' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:59:59.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 17:59:59 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:59:59.514 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:59:59.554 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:59.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:59.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:59:59.656 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:59:59.693 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:59.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:59.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:59:59.712 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 85 16 117736 117720 86.5493 70.4375 0.00929117 0.011549 2024-08-17T17:59:59.795 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:59:59.835 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T17:59:59.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:59.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:59:59.937 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:59:59.967 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T17:59:59.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:59:59.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:00.070 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:00.084 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-17T18:00:00.112 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:00.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:00.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:00.214 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:00.248 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:00.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:00.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:00.350 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:00.388 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:00.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:00.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:00.398 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:00 smithi136 ceph-mon[109861]: pgmap v2362: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 51 MiB/s wr, 821 op/s; 6368/342870 objects degraded (1.857%); 8547/342870 objects misplaced (2.493%); 4.1 MiB/s, 65 objects/s recovering 2024-08-17T18:00:00.398 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:00 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:00:00.399 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:00 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 6368/342870 objects degraded (1.857%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:00.399 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:00 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:00:00.399 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:00 smithi136 ceph-mon[109861]: osdmap e1389: 8 total, 8 up, 7 in 2024-08-17T18:00:00.399 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:00 smithi136 ceph-mon[109861]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 6368/342870 objects degraded (1.857%), 1 pg degraded 2024-08-17T18:00:00.399 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:00 smithi136 ceph-mon[109861]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-08-17T18:00:00.399 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:00 smithi136 ceph-mon[109861]: [WRN] PG_DEGRADED: Degraded data redundancy: 6368/342870 objects degraded (1.857%), 1 pg degraded 2024-08-17T18:00:00.399 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:00 smithi136 ceph-mon[109861]: pg 54.10 is active+recovery_wait+undersized+degraded+remapped, acting [3,2] 2024-08-17T18:00:00.399 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:00 smithi136 ceph-mon[115078]: pgmap v2362: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 51 MiB/s wr, 821 op/s; 6368/342870 objects degraded (1.857%); 8547/342870 objects misplaced (2.493%); 4.1 MiB/s, 65 objects/s recovering 2024-08-17T18:00:00.399 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:00 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:00:00.399 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:00 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 6368/342870 objects degraded (1.857%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:00.399 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:00 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:00:00.400 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:00 smithi136 ceph-mon[115078]: osdmap e1389: 8 total, 8 up, 7 in 2024-08-17T18:00:00.400 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:00 smithi136 ceph-mon[115078]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 6368/342870 objects degraded (1.857%), 1 pg degraded 2024-08-17T18:00:00.400 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:00 smithi136 ceph-mon[115078]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-08-17T18:00:00.400 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:00 smithi136 ceph-mon[115078]: [WRN] PG_DEGRADED: Degraded data redundancy: 6368/342870 objects degraded (1.857%), 1 pg degraded 2024-08-17T18:00:00.400 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:00 smithi136 ceph-mon[115078]: pg 54.10 is active+recovery_wait+undersized+degraded+remapped, acting [3,2] 2024-08-17T18:00:00.429 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:00:00.429 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-17T18:00:00.429 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:00:00.429 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_1 2024-08-17T18:00:00.429 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T18:00:00.490 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:00 smithi165 ceph-mon[105517]: pgmap v2362: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 51 MiB/s wr, 821 op/s; 6368/342870 objects degraded (1.857%); 8547/342870 objects misplaced (2.493%); 4.1 MiB/s, 65 objects/s recovering 2024-08-17T18:00:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:00 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:00:00.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:00 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 6368/342870 objects degraded (1.857%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:00.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:00 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:00:00.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:00 smithi165 ceph-mon[105517]: osdmap e1389: 8 total, 8 up, 7 in 2024-08-17T18:00:00.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:00 smithi165 ceph-mon[105517]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 6368/342870 objects degraded (1.857%), 1 pg degraded 2024-08-17T18:00:00.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:00 smithi165 ceph-mon[105517]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-08-17T18:00:00.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:00 smithi165 ceph-mon[105517]: [WRN] PG_DEGRADED: Degraded data redundancy: 6368/342870 objects degraded (1.857%), 1 pg degraded 2024-08-17T18:00:00.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:00 smithi165 ceph-mon[105517]: pg 54.10 is active+recovery_wait+undersized+degraded+remapped, acting [3,2] 2024-08-17T18:00:00.527 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:00.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:00.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:00.628 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:00.660 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:00.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:00.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:00.714 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 86 16 118839 118823 86.3428 68.9375 0.0110858 0.0115766 2024-08-17T18:00:00.762 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:00.798 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:00.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:00.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:00.901 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:00.935 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:00.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:00.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:01.037 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:01.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:01.099 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:01.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:01.200 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:01.229 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:01.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:01.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:01.258 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:00:01.259 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:00:01.259 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:00:01.259 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:00:01.259 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:00:01.259 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:00:01.259 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:00:01.259 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:00:01.259 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:00:01.259 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:00:01.259 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:00:01.260 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:00:01.260 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:00:01.260 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:00:01.260 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:00:01.260 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:00:01.297 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:01 smithi136 ceph-mon[109861]: 54.6 scrub starts 2024-08-17T18:00:01.297 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:01 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1728763846' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:00:01.331 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:01.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:01.370 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:01.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:01.471 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:01.502 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:01.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:01.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:01.604 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:01.630 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:01.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:01.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:01.689 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:01 smithi136 ceph-mon[115078]: 54.6 scrub starts 2024-08-17T18:00:01.689 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:01 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1728763846' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:00:01.714 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 87 16 119903 119887 86.1147 66.5 0.0099889 0.0116057 2024-08-17T18:00:01.731 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:01.754 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:01.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:01.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:01 smithi165 ceph-mon[105517]: 54.6 scrub starts 2024-08-17T18:00:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:01 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1728763846' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:00:01.856 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:01.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:01.891 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:01.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:01.992 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:02.028 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:02.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:02.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:02.130 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:02.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:02.169 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:02.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:02.270 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:02.306 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:02.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:02.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:02.408 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:02.450 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:02.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:02.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:02.552 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:02.590 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:02.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:02.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:02.692 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:02.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:02 smithi136 ceph-mon[109861]: pgmap v2364: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 58 MiB/s wr, 925 op/s; 6368/342870 objects degraded (1.857%); 8547/342870 objects misplaced (2.493%); 4.6 MiB/s, 73 objects/s recovering 2024-08-17T18:00:02.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:02 smithi136 ceph-mon[115078]: pgmap v2364: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 58 MiB/s wr, 925 op/s; 6368/342870 objects degraded (1.857%); 8547/342870 objects misplaced (2.493%); 4.6 MiB/s, 73 objects/s recovering 2024-08-17T18:00:02.714 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 88 16 121031 121015 85.937 70.5 0.0186998 0.0116314 2024-08-17T18:00:02.734 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:02.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:02.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:02.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:02 smithi165 ceph-mon[105517]: pgmap v2364: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 58 MiB/s wr, 925 op/s; 6368/342870 objects degraded (1.857%); 8547/342870 objects misplaced (2.493%); 4.6 MiB/s, 73 objects/s recovering 2024-08-17T18:00:02.836 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:02.880 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:02.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:02.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:02.984 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:03.020 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:03.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:03.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:03.047 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:00:03.122 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:03.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:03.158 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:03.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:03.259 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:03.314 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:03.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:03.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:03.416 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:03.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:03.459 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:03.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:03.514 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:00:03.515 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:00:03.560 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:03.599 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:03.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:03.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:03.701 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:03.714 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 89 16 122043 122027 85.682 63.25 0.00327695 0.0116596 2024-08-17T18:00:03.722 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:03.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:03.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:03.824 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:03.864 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:03.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:03.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:03.966 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:04.000 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:04.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:04.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:04.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:04.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:04.144 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:04.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:04.191 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-17T18:00:04.246 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:04.282 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:04.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:04.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:04.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:04.442 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:04.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:04.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:04.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:04 smithi165 ceph-mon[105517]: pgmap v2365: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s; 6368/359721 objects degraded (1.770%); 8070/359721 objects misplaced (2.243%); 3.9 MiB/s, 63 objects/s recovering 2024-08-17T18:00:04.544 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:04.565 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:04.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:04.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:04.667 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:04.688 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:04.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:04.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:04.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:04 smithi136 ceph-mon[109861]: pgmap v2365: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s; 6368/359721 objects degraded (1.770%); 8070/359721 objects misplaced (2.243%); 3.9 MiB/s, 63 objects/s recovering 2024-08-17T18:00:04.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:04 smithi136 ceph-mon[115078]: pgmap v2365: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s; 6368/359721 objects degraded (1.770%); 8070/359721 objects misplaced (2.243%); 3.9 MiB/s, 63 objects/s recovering 2024-08-17T18:00:04.714 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 90 3 122882 122879 85.3216 53.25 0.18577 0.0117113 2024-08-17T18:00:04.790 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:04.811 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:04.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:04.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:04.913 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:04.937 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:04.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:04.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:04.953 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:00:05.039 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:05.062 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:05.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:05.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:05.165 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:05.184 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:00:05.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:05.192 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:05.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:05.293 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:05.318 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:05.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:05.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:05.420 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:05.442 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:05.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:05.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:05.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:05.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:05.566 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:05.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:05.667 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:05.690 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:05.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:05.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:05 smithi136 ceph-mon[109861]: 54.16 scrub starts 2024-08-17T18:00:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:05 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 6368/359721 objects degraded (1.770%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:05.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:05 smithi136 ceph-mon[115078]: 54.16 scrub starts 2024-08-17T18:00:05.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:05 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 6368/359721 objects degraded (1.770%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:05.714 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Total time run: 90.0169 2024-08-17T18:00:05.714 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Total writes made: 122882 2024-08-17T18:00:05.714 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Write size: 65536 2024-08-17T18:00:05.715 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Object size: 65536 2024-08-17T18:00:05.715 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Bandwidth (MB/sec): 85.3187 2024-08-17T18:00:05.715 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Stddev Bandwidth: 47.1004 2024-08-17T18:00:05.715 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Max bandwidth (MB/sec): 257.688 2024-08-17T18:00:05.715 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Min bandwidth (MB/sec): 0 2024-08-17T18:00:05.715 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Average IOPS: 1365 2024-08-17T18:00:05.715 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Stddev IOPS: 753.621 2024-08-17T18:00:05.715 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Max IOPS: 4123 2024-08-17T18:00:05.715 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Min IOPS: 0 2024-08-17T18:00:05.715 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Average Latency(s): 0.0117158 2024-08-17T18:00:05.715 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Stddev Latency(s): 0.0419753 2024-08-17T18:00:05.715 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Max latency(s): 2.15603 2024-08-17T18:00:05.715 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Min latency(s): 0.0012629 2024-08-17T18:00:05.720 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Cleaning up (deleting benchmark objects) 2024-08-17T18:00:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:05 smithi165 ceph-mon[105517]: 54.16 scrub starts 2024-08-17T18:00:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:05 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 6368/359721 objects degraded (1.770%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:05.792 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:05.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:05.835 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:05.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:05.936 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:05.969 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:05.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:05.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:06.071 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:06.109 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:06.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:06.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:06.211 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:06.235 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:06.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:06.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:06.337 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:06.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:06.372 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:06.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:06.402 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:06 smithi136 ceph-mon[109861]: pgmap v2366: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s; 6368/359721 objects degraded (1.770%); 8070/359721 objects misplaced (2.243%); 3.9 MiB/s, 63 objects/s recovering 2024-08-17T18:00:06.402 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:06 smithi136 ceph-mon[109861]: 54.6 scrub starts 2024-08-17T18:00:06.402 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:00:06.402 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:06 smithi136 ceph-mon[115078]: pgmap v2366: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s; 6368/359721 objects degraded (1.770%); 8070/359721 objects misplaced (2.243%); 3.9 MiB/s, 63 objects/s recovering 2024-08-17T18:00:06.403 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:06 smithi136 ceph-mon[115078]: 54.6 scrub starts 2024-08-17T18:00:06.403 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:00:06.431 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:00:06.473 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:06.517 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:06.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:06.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:06.619 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:06.660 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:06.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:06.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:06.662 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:06 smithi165 ceph-mon[105517]: pgmap v2366: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s; 6368/359721 objects degraded (1.770%); 8070/359721 objects misplaced (2.243%); 3.9 MiB/s, 63 objects/s recovering 2024-08-17T18:00:06.662 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:06 smithi165 ceph-mon[105517]: 54.6 scrub starts 2024-08-17T18:00:06.662 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:00:06.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:06.804 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:06.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:06.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:06.906 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:06.945 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:06.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:06.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:06.966 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:00:07.046 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:07.079 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:07.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:07.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:07.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:07.217 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:07.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:07.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:07.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:07.355 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:07.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:07.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:07.456 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:07.491 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:07.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:07.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:07.592 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:07.624 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:07.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:07.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:07.650 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:00:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:00:07] "GET /metrics HTTP/1.1" 200 36564 "" "Prometheus/2.51.0" 2024-08-17T18:00:07.726 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:07.761 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:07.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:07.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:07.862 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:07.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:07.897 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:07.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:07.998 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:08.042 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:08.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:08.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:08.143 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:08.184 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:08.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:08.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:08.286 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:08.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:08.325 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:08.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:08.426 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:08.466 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:08.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:08.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:08.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:08.607 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:08.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:08.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:08.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:08 smithi136 ceph-mon[115078]: pgmap v2367: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 61 MiB/s wr, 1.01k op/s; 6350/364119 objects degraded (1.744%); 7382/364119 objects misplaced (2.027%); 5.0 MiB/s, 79 objects/s recovering 2024-08-17T18:00:08.665 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:08 smithi136 ceph-mon[115078]: 54.16 scrub starts 2024-08-17T18:00:08.667 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:08 smithi136 ceph-mon[109861]: pgmap v2367: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 61 MiB/s wr, 1.01k op/s; 6350/364119 objects degraded (1.744%); 7382/364119 objects misplaced (2.027%); 5.0 MiB/s, 79 objects/s recovering 2024-08-17T18:00:08.667 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:08 smithi136 ceph-mon[109861]: 54.16 scrub starts 2024-08-17T18:00:08.709 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:08.744 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:08.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:08.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:08 smithi165 ceph-mon[105517]: pgmap v2367: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 61 MiB/s wr, 1.01k op/s; 6350/364119 objects degraded (1.744%); 7382/364119 objects misplaced (2.027%); 5.0 MiB/s, 79 objects/s recovering 2024-08-17T18:00:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:08 smithi165 ceph-mon[105517]: 54.16 scrub starts 2024-08-17T18:00:08.845 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:08.887 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:08.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:08.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:08.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:09.030 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:09.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:09.132 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:09.169 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:09.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:09.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:09.271 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:09.309 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:09.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:09.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:09.411 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:09.442 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:09.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:09.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:09.544 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:09.580 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:09.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:09.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:09.595 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:09 smithi136 ceph-mon[109861]: 54.6 scrub starts 2024-08-17T18:00:09.595 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:09 smithi136 ceph-mon[115078]: 54.6 scrub starts 2024-08-17T18:00:09.682 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:09.720 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:09.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:09.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:09 smithi165 ceph-mon[105517]: 54.6 scrub starts 2024-08-17T18:00:09.821 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:09.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:09.877 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:09.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:09.978 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:10.013 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:10.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:10.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:10.115 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:10.170 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:10.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:10.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:10.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:10.306 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:10.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:10.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:10.408 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:10.445 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:10.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:10.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:10.521 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:10 smithi136 ceph-mon[109861]: pgmap v2368: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.4 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 102 B/s rd, 54 MiB/s wr, 973 op/s; 6350/365388 objects degraded (1.738%); 7382/365388 objects misplaced (2.020%); 5.0 MiB/s, 79 objects/s recovering 2024-08-17T18:00:10.522 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:10 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 6350/365388 objects degraded (1.738%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:10.522 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:10 smithi136 ceph-mon[115078]: pgmap v2368: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.4 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 102 B/s rd, 54 MiB/s wr, 973 op/s; 6350/365388 objects degraded (1.738%); 7382/365388 objects misplaced (2.020%); 5.0 MiB/s, 79 objects/s recovering 2024-08-17T18:00:10.522 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:10 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 6350/365388 objects degraded (1.738%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:10.547 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:10.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:10.585 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:10.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:10.686 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:10.713 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:10.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:10.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:10 smithi165 ceph-mon[105517]: pgmap v2368: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.4 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 102 B/s rd, 54 MiB/s wr, 973 op/s; 6350/365388 objects degraded (1.738%); 7382/365388 objects misplaced (2.020%); 5.0 MiB/s, 79 objects/s recovering 2024-08-17T18:00:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:10 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 6350/365388 objects degraded (1.738%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:10.815 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:10.831 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:00:10.864 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:10.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:10.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:10.966 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:11.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:11.006 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:11.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:11.107 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:11.137 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:11.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:11.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:11.226 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:00:11.228 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T18:00:11.239 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:11.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:11.285 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:11.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:11.350 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:00:11.386 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:11.423 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:11.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:11.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:11.524 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:11.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:11.567 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:11.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:11.645 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:11 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/228643664' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:00:11.645 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:11 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/228643664' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:00:11.668 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:11.702 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:11.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:11.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:11 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/228643664' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:00:11.803 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:11.838 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:11.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:11.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:11.940 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:11.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:11.978 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:11.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:12.079 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:12.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:12.115 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:12.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:12.216 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:12.250 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:12.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:12.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:12.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:12.386 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:12.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:12.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:12.488 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:12.521 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:12.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:12.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:12.623 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:12.633 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:12 smithi136 ceph-mon[109861]: pgmap v2369: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.4 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 89 B/s rd, 47 MiB/s wr, 849 op/s; 6350/365388 objects degraded (1.738%); 7382/365388 objects misplaced (2.020%); 4.4 MiB/s, 69 objects/s recovering 2024-08-17T18:00:12.633 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:12 smithi136 ceph-mon[109861]: from='client.45356 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:00:12.633 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:12 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:00:12.633 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:12 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/228643664' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:00:12.633 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:12 smithi136 ceph-mon[109861]: osdmap e1390: 8 total, 8 up, 7 in 2024-08-17T18:00:12.633 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:12 smithi136 ceph-mon[109861]: 54.16 scrub starts 2024-08-17T18:00:12.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:12 smithi136 ceph-mon[115078]: pgmap v2369: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.4 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 89 B/s rd, 47 MiB/s wr, 849 op/s; 6350/365388 objects degraded (1.738%); 7382/365388 objects misplaced (2.020%); 4.4 MiB/s, 69 objects/s recovering 2024-08-17T18:00:12.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:12 smithi136 ceph-mon[115078]: from='client.45356 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:00:12.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:12 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:00:12.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:12 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/228643664' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:00:12.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:12 smithi136 ceph-mon[115078]: osdmap e1390: 8 total, 8 up, 7 in 2024-08-17T18:00:12.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:12 smithi136 ceph-mon[115078]: 54.16 scrub starts 2024-08-17T18:00:12.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:12.652 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:12.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:12.752 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:12 smithi165 ceph-mon[105517]: pgmap v2369: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.4 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 89 B/s rd, 47 MiB/s wr, 849 op/s; 6350/365388 objects degraded (1.738%); 7382/365388 objects misplaced (2.020%); 4.4 MiB/s, 69 objects/s recovering 2024-08-17T18:00:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:12 smithi165 ceph-mon[105517]: from='client.45356 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:00:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:12 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:00:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:12 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/228643664' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:00:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:12 smithi165 ceph-mon[105517]: osdmap e1390: 8 total, 8 up, 7 in 2024-08-17T18:00:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:12 smithi165 ceph-mon[105517]: 54.16 scrub starts 2024-08-17T18:00:12.797 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:12.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:12.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:12.899 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:12.933 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:12.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:12.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:12.939 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:00:12.939 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:00:13.035 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:13.068 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:13.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:13.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:13.170 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:13.209 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:13.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:13.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:13.311 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:13.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:13.347 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:13.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:13.448 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:13.484 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:13.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:13.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:13.586 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:13.625 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:13.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:13.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:13.728 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:13.759 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:13.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:13.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:13.860 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:13.907 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:13.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:13.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:13.910 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":2369,"stamp":"2024-08-17T18:00:10.728439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7980849149,"num_objects":121796,"num_object_clones":0,"num_object_copies":365388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6350,"num_objects_misplaced":7382,"num_objects_unfound":0,"num_objects_dirty":121796,"num_whiteouts":0,"num_read":379,"num_read_kb":823,"num_write":124275,"num_write_kb":7866898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2799,"num_bytes_recovered":184617692,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":175898,"ondisk_log_size":175898,"up":297,"acting":295,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":302,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":26614280,"kb_used_data":23981072,"kb_used_omap":219,"kb_used_meta":2632868,"kb_avail":629601784,"statfs":{"total":671965249536,"available":644712226816,"internally_reserved":0,"allocated":24556617728,"data_stored":24525754632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":224669,"internal_metadata":2696057443},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":491847704,"num_objects":7506,"num_object_clones":0,"num_object_copies":22518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-18,"num_objects_misplaced":-1165,"num_objects_unfound":0,"num_objects_dirty":7506,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9732,"num_write_kb":551553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":798,"num_bytes_recovered":52297728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.460347"},"pg_stats":[{"pgid":"54.19","version":"1389'7018","reported_seq":7101,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T18:00:06.383242+0000","last_change":"2024-08-17T18:00:05.727733+0000","last_active":"2024-08-17T18:00:06.383242+0000","last_peered":"2024-08-17T18:00:06.383242+0000","last_clean":"2024-08-17T18:00:06.383242+0000","last_became_active":"2024-08-17T17:59:36.681900+0000","last_became_peered":"2024-08-17T17:59:36.681900+0000","last_unstale":"2024-08-17T18:00:06.383242+0000","last_undegraded":"2024-08-17T18:00:06.383242+0000","last_fullsized":"2024-08-17T18:00:06.383242+0000","mapping_epoch":1381,"log_start":"0'0","ondisk_log_start":"0'0","created":1381,"last_epoch_clean":1382,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":7018,"log_dups_size":0,"ondisk_log_size":7018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T18:00:09.712370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254935040,"num_objects":3890,"num_object_clones":0,"num_object_copies":11670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3942,"num_write_kb":250624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"54.18","version":"1389'6865","reported_seq":6907,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T18:00:07.310656+0000","last_change":"2024-08-17T17:59:36.677193+0000","last_active":"2024-08-17T18:00:07.310656+0000","last_peered":"2024-08-17T18:00:07.310656+0000","last_clean":"2024-08-17T18:00:07.310656+0000","last_became_active":"2024-08-17T17:59:36.674940+0000","last_became_peered":"2024-08-17T17:59:36.674940+0000","last_unstale":"2024-08-17T18:00:07.310656+0000","last_undegraded":"2024-08-17T18:00:07.310656+0000","last_fullsized":"2024-08-17T18:00:07.310656+0000","mapping_epoch":1381,"log_start":"0'0","ondisk_log_start":"0'0","created":1381,"last_epoch_clean":1382,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":6865,"log_dups_size":0,"ondisk_log_size":6865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246874112,"num_objects":3767,"num_object_clones":0,"num_object_copies":11301,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3895,"num_write_kb":245184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"54.17","version":"1389'6822","reported_seq":6913,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T18:00:05.936589+0000","last_change":"2024-08-17T18:00:01.784443+0000","last_active":"2024-08-17T18:00:05.936589+0000","last_peered":"2024-08-17T18:00:05.936589+0000","last_clean":"2024-08-17T18:00:05.936589+0000","last_became_active":"2024-08-17T17:59:36.688212+0000","last_became_peered":"2024-08-17T17:59:36.688212+0000","last_unstale":"2024-08-17T18:00:05.936589+0000","last_undegraded":"2024-08-17T18:00:05.936589+0000","last_fullsized":"2024-08-17T18:00:05.936589+0000","mapping_epoch":1381,"log_start":"0'0","ondisk_log_start":"0'0","created":1381,"last_epoch_clean":1382,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":6822,"log_dups_size":0,"ondisk_log_size":6822,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T18:00:06.781601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249102336,"num_objects":3801,"num_object_clones":0,"num_object_copies":11403,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3823,"num_write_kb":243968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.16","version":"1389'6847","reported_seq":6891,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T18:00:06.957516+0000","last_change":"2024-08-17T18:00:04.455220+0000","last_active":"2024-08-17T18:00:06.957516+0000","last_peered":"2024-08-17T18:00:06.957516+0000","last_clean":"2024-08-17T18:00:06.957516+0000","last_became_active":"2024-08-17T17:59:36.653312+0000","last_became_peered":"2024-08-17T17:59:36.653312+0000","last_unstale":"2024-08-17T18:00:06.957516+0000","last_undegraded":"2024-08-17T18:00:06.957516+0000","last_fullsized":"2024-08-17T18:00:06.957516+0000","mapping_epoch":1381,"log_start":"0'0","ondisk_log_start":"0'0","created":1381,"last_epoch_clean":1382,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":6847,"log_dups_size":0,"ondisk_log_size":6847,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248381440,"num_objects":3790,"num_object_clones":0,"num_object_copies":11370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3880,"num_write_kb":245440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"54.15","version":"1389'6816","reported_seq":6867,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T18:00:06.942960+0000","last_change":"2024-08-17T17:59:36.696935+0000","last_active":"2024-08-17T18:00:06.942960+0000","last_peered":"2024-08-17T18:00:06.942960+0000","last_clean":"2024-08-17T18:00:06.942960+0000","last_became_active":"2024-08-17T17:59:36.693194+0000","last_became_peered":"2024-08-17T17:59:36.693194+0000","last_unstale":"2024-08-17T18:00:06.942960+0000","last_undegraded":"2024-08-17T18:00:06.942960+0000","last_fullsized":"2024-08-17T18:00:06.942960+0000","mapping_epoch":1381,"log_start":"0'0","ondisk_log_start":"0'0","created":1381,"last_epoch_clean":1382,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":6816,"log_dups_size":0,"ondisk_log_size":6816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246939648,"num_objects":3768,"num_object_clones":0,"num_object_copies":11304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3858,"num_write_kb":244032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"54.14","version":"1389'6793","reported_seq":6873,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T18:00:05.931709+0000","last_change":"2024-08-17T18:00:03.837012+0000","last_active":"2024-08-17T18:00:05.931709+0000","last_peered":"2024-08-17T18:00:05.931709+0000","last_clean":"2024-08-17T18:00:05.931709+0000","last_became_active":"2024-08-17T17:59:36.693778+0000","last_became_peered":"2024-08-17T17:59:36.693778+0000","last_unstale":"2024-08-17T18:00:05.931709+0000","last_undegraded":"2024-08-17T18:00:05.931709+0000","last_fullsized":"2024-08-17T18:00:05.931709+0000","mapping_epoch":1381,"log_start":"0'0","ondisk_log_start":"0'0","created":1381,"last_epoch_clean":1382,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":6793,"log_dups_size":0,"ondisk_log_size":6793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T18:00:08.820266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249298944,"num_objects":3804,"num_object_clones":0,"num_object_copies":11412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3832,"num_write_kb":244352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.13","version":"1389'6943","reported_seq":7028,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T18:00:06.407064+0000","last_change":"2024-08-17T18:00:02.659166+0000","last_active":"2024-08-17T18:00:06.407064+0000","last_peered":"2024-08-17T18:00:06.407064+0000","last_clean":"2024-08-17T18:00:06.407064+0000","last_became_active":"2024-08-17T17:59:36.674258+0000","last_became_peered":"2024-08-17T17:59:36.674258+0000","last_unstale":"2024-08-17T18:00:06.407064+0000","last_undegraded":"2024-08-17T18:00:06.407064+0000","last_fullsized":"2024-08-17T18:00:06.407064+0000","mapping_epoch":1381,"log_start":"0'0","ondisk_log_start":"0'0","created":1381,"last_epoch_clean":1382,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":6943,"log_dups_size":0,"ondisk_log_size":6943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T18:00:07.648472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252706816,"num_objects":3856,"num_object_clones":0,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3922,"num_write_kb":248896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"54.12","version":"1389'6887","reported_seq":6933,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T18:00:07.235882+0000","last_change":"2024-08-17T17:59:36.668085+0000","last_active":"2024-08-17T18:00:07.235882+0000","last_peered":"2024-08-17T18:00:07.235882+0000","last_clean":"2024-08-17T18:00:07.235882+0000","last_became_active":"2024-08-17T17:59:36.661096+0000","last_became_peered":"2024-08-17T17:59:36.661096+0000","last_unstale":"2024-08-17T18:00:07.235882+0000","last_undegraded":"2024-08-17T18:00:07.235882+0000","last_fullsized":"2024-08-17T18:00:07.235882+0000","mapping_epoch":1381,"log_start":"0'0","ondisk_log_start":"0'0","created":1381,"last_epoch_clean":1382,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":6887,"log_dups_size":0,"ondisk_log_size":6887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248774656,"num_objects":3796,"num_object_clones":0,"num_object_copies":11388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3904,"num_write_kb":246400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"54.11","version":"1389'6831","reported_seq":8804,"reported_epoch":1389,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-17T18:00:06.436461+0000","last_change":"2024-08-17T17:59:47.800196+0000","last_active":"2024-08-17T18:00:06.436461+0000","last_peered":"2024-08-17T18:00:06.436461+0000","last_clean":"2024-08-17T17:59:44.103800+0000","last_became_active":"2024-08-17T17:59:47.766919+0000","last_became_peered":"2024-08-17T17:59:47.766919+0000","last_unstale":"2024-08-17T18:00:06.436461+0000","last_undegraded":"2024-08-17T18:00:06.436461+0000","last_fullsized":"2024-08-17T17:59:47.463059+0000","mapping_epoch":1387,"log_start":"0'0","ondisk_log_start":"0'0","created":1381,"last_epoch_clean":1382,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":6831,"log_dups_size":0,"ondisk_log_size":6831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249757696,"num_objects":3811,"num_object_clones":0,"num_object_copies":11433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5182,"num_objects_unfound":0,"num_objects_dirty":3811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3857,"num_write_kb":245376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":42008576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[1,2],"avail_no_missing":["1","2","5"],"object_location_counts":[{"shards":"1,2","objects":1220},{"shards":"1,2,5","objects":2591}],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[]},{"pgid":"54.10","version":"1389'6800","reported_seq":6877,"reported_epoch":1389,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:00:05.914933+0000","last_change":"2024-08-17T17:59:45.609935+0000","last_active":"2024-08-17T18:00:05.914933+0000","last_peered":"2024-08-17T18:00:05.914933+0000","last_clean":"2024-08-17T17:59:43.603263+0000","last_became_active":"2024-08-17T17:59:45.579984+0000","last_became_peered":"2024-08-17T17:59:45.579984+0000","last_unstale":"2024-08-17T18:00:05.914933+0000","last_undegraded":"2024-08-17T17:59:45.538281+0000","last_fullsized":"2024-08-17T17:59:45.458305+0000","mapping_epoch":1385,"log_start":"0'0","ondisk_log_start":"0'0","created":1381,"last_epoch_clean":1382,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":6800,"log_dups_size":0,"ondisk_log_size":6800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249724928,"num_objects":3811,"num_object_clones":0,"num_object_copies":11433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6350,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3829,"num_write_kb":244448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":589824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[3,2],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"2,3","objects":636},{"shards":"3,4,6","objects":3175}],"blocked_by":[],"up_primary":5,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.f","version":"1389'7773","reported_seq":7838,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T18:00:05.998045+0000","last_change":"2024-08-17T17:58:19.551058+0000","last_active":"2024-08-17T18:00:05.998045+0000","last_peered":"2024-08-17T18:00:05.998045+0000","last_clean":"2024-08-17T18:00:05.998045+0000","last_became_active":"2024-08-17T17:58:19.550748+0000","last_became_peered":"2024-08-17T17:58:19.550748+0000","last_unstale":"2024-08-17T18:00:05.998045+0000","last_undegraded":"2024-08-17T18:00:05.998045+0000","last_fullsized":"2024-08-17T18:00:05.998045+0000","mapping_epoch":1364,"log_start":"1373'2215","ondisk_log_start":"1373'2215","created":1364,"last_epoch_clean":1365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:52:38.089541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":506920960,"num_objects":7735,"num_object_clones":0,"num_object_copies":23205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7773,"num_write_kb":496256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.e","version":"1389'7858","reported_seq":7889,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T18:00:07.342171+0000","last_change":"2024-08-17T17:58:19.593455+0000","last_active":"2024-08-17T18:00:07.342171+0000","last_peered":"2024-08-17T18:00:07.342171+0000","last_clean":"2024-08-17T18:00:07.342171+0000","last_became_active":"2024-08-17T17:58:19.593144+0000","last_became_peered":"2024-08-17T17:58:19.593144+0000","last_unstale":"2024-08-17T18:00:07.342171+0000","last_undegraded":"2024-08-17T18:00:07.342171+0000","last_fullsized":"2024-08-17T18:00:07.342171+0000","mapping_epoch":1364,"log_start":"1372'100","ondisk_log_start":"1372'100","created":1364,"last_epoch_clean":1365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":7758,"log_dups_size":0,"ondisk_log_size":7758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:08:45.701487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499777536,"num_objects":7626,"num_object_clones":0,"num_object_copies":22878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7858,"num_write_kb":495488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"54.d","version":"1389'7714","reported_seq":7758,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T18:00:07.069729+0000","last_change":"2024-08-17T17:58:23.582595+0000","last_active":"2024-08-17T18:00:07.069729+0000","last_peered":"2024-08-17T18:00:07.069729+0000","last_clean":"2024-08-17T18:00:07.069729+0000","last_became_active":"2024-08-17T17:58:23.582234+0000","last_became_peered":"2024-08-17T17:58:23.582234+0000","last_unstale":"2024-08-17T18:00:07.069729+0000","last_undegraded":"2024-08-17T18:00:07.069729+0000","last_fullsized":"2024-08-17T18:00:07.069729+0000","mapping_epoch":1367,"log_start":"1372'800","ondisk_log_start":"1372'800","created":1364,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":6914,"log_dups_size":0,"ondisk_log_size":6914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:10:06.366767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491454488,"num_objects":7500,"num_object_clones":0,"num_object_copies":22500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7500,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7714,"num_write_kb":486785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"54.c","version":"1389'7593","reported_seq":11914,"reported_epoch":1389,"state":"active+recovering+remapped","last_fresh":"2024-08-17T18:00:05.994087+0000","last_change":"2024-08-17T17:59:12.058065+0000","last_active":"2024-08-17T18:00:05.994087+0000","last_peered":"2024-08-17T18:00:05.994087+0000","last_clean":"2024-08-17T17:59:09.884394+0000","last_became_active":"2024-08-17T17:59:11.993002+0000","last_became_peered":"2024-08-17T17:59:11.993002+0000","last_unstale":"2024-08-17T18:00:05.994087+0000","last_undegraded":"2024-08-17T18:00:05.994087+0000","last_fullsized":"2024-08-17T18:00:05.994087+0000","mapping_epoch":1377,"log_start":"1373'2026","ondisk_log_start":"1373'2026","created":1364,"last_epoch_clean":1365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":5567,"log_dups_size":0,"ondisk_log_size":5567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":494206976,"num_objects":7541,"num_object_clones":0,"num_object_copies":22623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2200,"num_objects_unfound":0,"num_objects_dirty":7541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7593,"num_write_kb":484288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2129,"num_bytes_recovered":139526144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[3,5,4],"avail_no_missing":["3","4","5"],"object_location_counts":[{"shards":"3,4,5","objects":5341}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.b","version":"1389'7562","reported_seq":7634,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T18:00:05.970806+0000","last_change":"2024-08-17T17:58:23.585551+0000","last_active":"2024-08-17T18:00:05.970806+0000","last_peered":"2024-08-17T18:00:05.970806+0000","last_clean":"2024-08-17T18:00:05.970806+0000","last_became_active":"2024-08-17T17:58:23.581936+0000","last_became_peered":"2024-08-17T17:58:23.581936+0000","last_unstale":"2024-08-17T18:00:05.970806+0000","last_undegraded":"2024-08-17T18:00:05.970806+0000","last_fullsized":"2024-08-17T18:00:05.970806+0000","mapping_epoch":1367,"log_start":"1373'1926","ondisk_log_start":"1373'1926","created":1364,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":5636,"log_dups_size":0,"ondisk_log_size":5636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:39:18.876782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":493486080,"num_objects":7530,"num_object_clones":0,"num_object_copies":22590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7562,"num_write_kb":482944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.a","version":"1389'7871","reported_seq":7905,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T18:00:07.086937+0000","last_change":"2024-08-17T17:58:19.551465+0000","last_active":"2024-08-17T18:00:07.086937+0000","last_peered":"2024-08-17T18:00:07.086937+0000","last_clean":"2024-08-17T18:00:07.086937+0000","last_became_active":"2024-08-17T17:58:19.551177+0000","last_became_peered":"2024-08-17T17:58:19.551177+0000","last_unstale":"2024-08-17T18:00:07.086937+0000","last_undegraded":"2024-08-17T18:00:07.086937+0000","last_fullsized":"2024-08-17T18:00:07.086937+0000","mapping_epoch":1364,"log_start":"1373'1000","ondisk_log_start":"1373'1000","created":1364,"last_epoch_clean":1365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":6871,"log_dups_size":0,"ondisk_log_size":6871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:20:45.172117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":501809152,"num_objects":7657,"num_object_clones":0,"num_object_copies":22971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7871,"num_write_kb":496896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"54.9","version":"1389'7054","reported_seq":7131,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T18:00:06.401721+0000","last_change":"2024-08-17T18:00:00.670918+0000","last_active":"2024-08-17T18:00:06.401721+0000","last_peered":"2024-08-17T18:00:06.401721+0000","last_clean":"2024-08-17T18:00:06.401721+0000","last_became_active":"2024-08-17T17:59:36.654557+0000","last_became_peered":"2024-08-17T17:59:36.654557+0000","last_unstale":"2024-08-17T18:00:06.401721+0000","last_undegraded":"2024-08-17T18:00:06.401721+0000","last_fullsized":"2024-08-17T18:00:06.401721+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":7054,"log_dups_size":0,"ondisk_log_size":7054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257556480,"num_objects":3930,"num_object_clones":0,"num_object_copies":11790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3978,"num_write_kb":253056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"54.8","version":"1389'6797","reported_seq":6839,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T18:00:07.356473+0000","last_change":"2024-08-17T17:59:36.663446+0000","last_active":"2024-08-17T18:00:07.356473+0000","last_peered":"2024-08-17T18:00:07.356473+0000","last_clean":"2024-08-17T18:00:07.356473+0000","last_became_active":"2024-08-17T17:59:36.660417+0000","last_became_peered":"2024-08-17T17:59:36.660417+0000","last_unstale":"2024-08-17T18:00:07.356473+0000","last_undegraded":"2024-08-17T18:00:07.356473+0000","last_fullsized":"2024-08-17T18:00:07.356473+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":6797,"log_dups_size":0,"ondisk_log_size":6797,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243990528,"num_objects":3723,"num_object_clones":0,"num_object_copies":11169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3827,"num_write_kb":241600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"54.7","version":"1389'6870","reported_seq":6961,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T18:00:05.997280+0000","last_change":"2024-08-17T18:00:02.748936+0000","last_active":"2024-08-17T18:00:05.997280+0000","last_peered":"2024-08-17T18:00:05.997280+0000","last_clean":"2024-08-17T18:00:05.997280+0000","last_became_active":"2024-08-17T17:59:36.654489+0000","last_became_peered":"2024-08-17T17:59:36.654489+0000","last_unstale":"2024-08-17T18:00:05.997280+0000","last_undegraded":"2024-08-17T18:00:05.997280+0000","last_fullsized":"2024-08-17T18:00:05.997280+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":6870,"log_dups_size":0,"ondisk_log_size":6870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T18:00:07.737033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252379136,"num_objects":3851,"num_object_clones":0,"num_object_copies":11553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3871,"num_write_kb":247104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.6","version":"1389'6844","reported_seq":6889,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T18:00:06.951268+0000","last_change":"2024-08-17T18:00:05.737529+0000","last_active":"2024-08-17T18:00:06.951268+0000","last_peered":"2024-08-17T18:00:06.951268+0000","last_clean":"2024-08-17T18:00:06.951268+0000","last_became_active":"2024-08-17T17:59:36.650178+0000","last_became_peered":"2024-08-17T17:59:36.650178+0000","last_unstale":"2024-08-17T18:00:06.951268+0000","last_undegraded":"2024-08-17T18:00:06.951268+0000","last_fullsized":"2024-08-17T18:00:06.951268+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":6844,"log_dups_size":0,"ondisk_log_size":6844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248709120,"num_objects":3795,"num_object_clones":0,"num_object_copies":11385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3877,"num_write_kb":245504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"54.5","version":"1389'6844","reported_seq":6900,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T18:00:07.020297+0000","last_change":"2024-08-17T17:59:36.697642+0000","last_active":"2024-08-17T18:00:07.020297+0000","last_peered":"2024-08-17T18:00:07.020297+0000","last_clean":"2024-08-17T18:00:07.020297+0000","last_became_active":"2024-08-17T17:59:36.681835+0000","last_became_peered":"2024-08-17T17:59:36.681835+0000","last_unstale":"2024-08-17T18:00:07.020297+0000","last_undegraded":"2024-08-17T18:00:07.020297+0000","last_fullsized":"2024-08-17T18:00:07.020297+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":6844,"log_dups_size":0,"ondisk_log_size":6844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249561088,"num_objects":3808,"num_object_clones":0,"num_object_copies":11424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3886,"num_write_kb":246208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"54.4","version":"1389'6789","reported_seq":6864,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T18:00:05.953462+0000","last_change":"2024-08-17T18:00:05.757088+0000","last_active":"2024-08-17T18:00:05.953462+0000","last_peered":"2024-08-17T18:00:05.953462+0000","last_clean":"2024-08-17T18:00:05.953462+0000","last_became_active":"2024-08-17T17:59:36.659081+0000","last_became_peered":"2024-08-17T17:59:36.659081+0000","last_unstale":"2024-08-17T18:00:05.953462+0000","last_undegraded":"2024-08-17T18:00:05.953462+0000","last_fullsized":"2024-08-17T18:00:05.953462+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":6789,"log_dups_size":0,"ondisk_log_size":6789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T18:00:09.827576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249036800,"num_objects":3800,"num_object_clones":0,"num_object_copies":11400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3828,"num_write_kb":244096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.3","version":"1389'6886","reported_seq":6971,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T18:00:06.396532+0000","last_change":"2024-08-17T18:00:05.749447+0000","last_active":"2024-08-17T18:00:06.396532+0000","last_peered":"2024-08-17T18:00:06.396532+0000","last_clean":"2024-08-17T18:00:06.396532+0000","last_became_active":"2024-08-17T17:59:36.627614+0000","last_became_peered":"2024-08-17T17:59:36.627614+0000","last_unstale":"2024-08-17T18:00:06.396532+0000","last_undegraded":"2024-08-17T18:00:06.396532+0000","last_fullsized":"2024-08-17T18:00:06.396532+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":6886,"log_dups_size":0,"ondisk_log_size":6886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T18:00:10.631398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249626624,"num_objects":3809,"num_object_clones":0,"num_object_copies":11427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3865,"num_write_kb":245568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"54.2","version":"1389'6931","reported_seq":6973,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T18:00:07.269390+0000","last_change":"2024-08-17T17:59:36.652281+0000","last_active":"2024-08-17T18:00:07.269390+0000","last_peered":"2024-08-17T18:00:07.269390+0000","last_clean":"2024-08-17T18:00:07.269390+0000","last_became_active":"2024-08-17T17:59:36.649201+0000","last_became_peered":"2024-08-17T17:59:36.649201+0000","last_unstale":"2024-08-17T18:00:07.269390+0000","last_undegraded":"2024-08-17T18:00:07.269390+0000","last_fullsized":"2024-08-17T18:00:07.269390+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":6931,"log_dups_size":0,"ondisk_log_size":6931,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250085376,"num_objects":3816,"num_object_clones":0,"num_object_copies":11448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3948,"num_write_kb":248448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"54.1","version":"1389'6892","reported_seq":6936,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T18:00:07.250115+0000","last_change":"2024-08-17T17:59:36.642298+0000","last_active":"2024-08-17T18:00:07.250115+0000","last_peered":"2024-08-17T18:00:07.250115+0000","last_clean":"2024-08-17T18:00:07.250115+0000","last_became_active":"2024-08-17T17:59:36.639233+0000","last_became_peered":"2024-08-17T17:59:36.639233+0000","last_unstale":"2024-08-17T18:00:07.250115+0000","last_undegraded":"2024-08-17T18:00:07.250115+0000","last_fullsized":"2024-08-17T18:00:07.250115+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":6892,"log_dups_size":0,"ondisk_log_size":6892,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247988224,"num_objects":3784,"num_object_clones":0,"num_object_copies":11352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3918,"num_write_kb":246464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"54.0","version":"1389'6857","reported_seq":6900,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T18:00:07.071836+0000","last_change":"2024-08-17T17:59:36.643268+0000","last_active":"2024-08-17T18:00:07.071836+0000","last_peered":"2024-08-17T18:00:07.071836+0000","last_clean":"2024-08-17T18:00:07.071836+0000","last_became_active":"2024-08-17T17:59:36.633136+0000","last_became_peered":"2024-08-17T17:59:36.633136+0000","last_unstale":"2024-08-17T18:00:07.071836+0000","last_undegraded":"2024-08-17T18:00:07.071836+0000","last_fullsized":"2024-08-17T18:00:07.071836+0000","mapping_epoch":1364,"log_start":"0'0","ondisk_log_start":"0'0","created":1364,"last_epoch_clean":1382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:58:18.524591+0000","last_clean_scrub_stamp":"2024-08-17T17:58:18.524591+0000","objects_scrubbed":0,"log_size":6857,"log_dups_size":0,"ondisk_log_size":6857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247169024,"num_objects":3771,"num_object_clones":0,"num_object_copies":11313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3885,"num_write_kb":245024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":63,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.573936+0000","last_change":"2024-08-17T17:58:23.586005+0000","last_active":"2024-08-17T17:59:47.573936+0000","last_peered":"2024-08-17T17:59:47.573936+0000","last_clean":"2024-08-17T17:59:47.573936+0000","last_became_active":"2024-08-17T17:58:23.580706+0000","last_became_peered":"2024-08-17T17:58:23.580706+0000","last_unstale":"2024-08-17T17:59:47.573936+0000","last_undegraded":"2024-08-17T17:59:47.573936+0000","last_fullsized":"2024-08-17T17:59:47.573936+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:00:46.147301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1361,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.599411+0000","last_change":"2024-08-17T17:23:07.398676+0000","last_active":"2024-08-17T17:59:47.599411+0000","last_peered":"2024-08-17T17:59:47.599411+0000","last_clean":"2024-08-17T17:59:47.599411+0000","last_became_active":"2024-08-17T17:23:07.398369+0000","last_became_peered":"2024-08-17T17:23:07.398369+0000","last_unstale":"2024-08-17T17:59:47.599411+0000","last_undegraded":"2024-08-17T17:59:47.599411+0000","last_fullsized":"2024-08-17T17:59:47.599411+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:55:04.281412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":2068,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.284850+0000","last_change":"2024-08-17T17:24:17.476168+0000","last_active":"2024-08-17T17:59:59.284850+0000","last_peered":"2024-08-17T17:59:59.284850+0000","last_clean":"2024-08-17T17:59:59.284850+0000","last_became_active":"2024-08-17T17:24:17.465845+0000","last_became_peered":"2024-08-17T17:24:17.465845+0000","last_unstale":"2024-08-17T17:59:59.284850+0000","last_undegraded":"2024-08-17T17:59:59.284850+0000","last_fullsized":"2024-08-17T17:59:59.284850+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:15:01.494447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2091,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.283150+0000","last_change":"2024-08-17T17:58:23.585542+0000","last_active":"2024-08-17T17:59:59.283150+0000","last_peered":"2024-08-17T17:59:59.283150+0000","last_clean":"2024-08-17T17:59:59.283150+0000","last_became_active":"2024-08-17T17:58:23.584976+0000","last_became_peered":"2024-08-17T17:58:23.584976+0000","last_unstale":"2024-08-17T17:59:59.283150+0000","last_undegraded":"2024-08-17T17:59:59.283150+0000","last_fullsized":"2024-08-17T17:59:59.283150+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:46:31.440698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":1343,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.573634+0000","last_change":"2024-08-17T17:10:05.965176+0000","last_active":"2024-08-17T17:59:47.573634+0000","last_peered":"2024-08-17T17:59:47.573634+0000","last_clean":"2024-08-17T17:59:47.573634+0000","last_became_active":"2024-08-17T17:10:05.964861+0000","last_became_peered":"2024-08-17T17:10:05.964861+0000","last_unstale":"2024-08-17T17:59:47.573634+0000","last_undegraded":"2024-08-17T17:59:47.573634+0000","last_fullsized":"2024-08-17T17:59:47.573634+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:59:04.277926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1385,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.287884+0000","last_change":"2024-08-17T17:58:23.582593+0000","last_active":"2024-08-17T17:59:59.287884+0000","last_peered":"2024-08-17T17:59:59.287884+0000","last_clean":"2024-08-17T17:59:59.287884+0000","last_became_active":"2024-08-17T17:58:23.582276+0000","last_became_peered":"2024-08-17T17:58:23.582276+0000","last_unstale":"2024-08-17T17:59:59.287884+0000","last_undegraded":"2024-08-17T17:59:59.287884+0000","last_fullsized":"2024-08-17T17:59:59.287884+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:04:27.806072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":1358,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.573230+0000","last_change":"2024-08-17T17:23:07.391038+0000","last_active":"2024-08-17T17:59:47.573230+0000","last_peered":"2024-08-17T17:59:47.573230+0000","last_clean":"2024-08-17T17:59:47.573230+0000","last_became_active":"2024-08-17T17:23:07.390571+0000","last_became_peered":"2024-08-17T17:23:07.390571+0000","last_unstale":"2024-08-17T17:59:47.573230+0000","last_undegraded":"2024-08-17T17:59:47.573230+0000","last_fullsized":"2024-08-17T17:59:47.573230+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:18:41.791740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1375,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.599751+0000","last_change":"2024-08-17T17:24:17.458168+0000","last_active":"2024-08-17T17:59:47.599751+0000","last_peered":"2024-08-17T17:59:47.599751+0000","last_clean":"2024-08-17T17:59:47.599751+0000","last_became_active":"2024-08-17T17:24:17.457847+0000","last_became_peered":"2024-08-17T17:24:17.457847+0000","last_unstale":"2024-08-17T17:59:47.599751+0000","last_undegraded":"2024-08-17T17:59:47.599751+0000","last_fullsized":"2024-08-17T17:59:47.599751+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:28.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":45,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.599757+0000","last_change":"2024-08-17T17:58:23.579378+0000","last_active":"2024-08-17T17:59:47.599757+0000","last_peered":"2024-08-17T17:59:47.599757+0000","last_clean":"2024-08-17T17:59:47.599757+0000","last_became_active":"2024-08-17T17:58:23.579006+0000","last_became_peered":"2024-08-17T17:58:23.579006+0000","last_unstale":"2024-08-17T17:59:47.599757+0000","last_undegraded":"2024-08-17T17:59:47.599757+0000","last_fullsized":"2024-08-17T17:59:47.599757+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:45:42.529376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":2085,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.294453+0000","last_change":"2024-08-17T17:24:17.456622+0000","last_active":"2024-08-17T17:59:59.294453+0000","last_peered":"2024-08-17T17:59:59.294453+0000","last_clean":"2024-08-17T17:59:59.294453+0000","last_became_active":"2024-08-17T17:24:17.455396+0000","last_became_peered":"2024-08-17T17:24:17.455396+0000","last_unstale":"2024-08-17T17:59:59.294453+0000","last_undegraded":"2024-08-17T17:59:59.294453+0000","last_fullsized":"2024-08-17T17:59:59.294453+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:09.564331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1378,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.287996+0000","last_change":"2024-08-17T17:23:42.930464+0000","last_active":"2024-08-17T17:59:59.287996+0000","last_peered":"2024-08-17T17:59:59.287996+0000","last_clean":"2024-08-17T17:59:59.287996+0000","last_became_active":"2024-08-17T17:23:42.929949+0000","last_became_peered":"2024-08-17T17:23:42.929949+0000","last_unstale":"2024-08-17T17:59:59.287996+0000","last_undegraded":"2024-08-17T17:59:59.287996+0000","last_fullsized":"2024-08-17T17:59:59.287996+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:48:27.092339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2071,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.288658+0000","last_change":"2024-08-17T17:24:17.452009+0000","last_active":"2024-08-17T17:59:59.288658+0000","last_peered":"2024-08-17T17:59:59.288658+0000","last_clean":"2024-08-17T17:59:59.288658+0000","last_became_active":"2024-08-17T17:24:17.451492+0000","last_became_peered":"2024-08-17T17:24:17.451492+0000","last_unstale":"2024-08-17T17:59:59.288658+0000","last_undegraded":"2024-08-17T17:59:59.288658+0000","last_fullsized":"2024-08-17T17:59:59.288658+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:21:24.699633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2083,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.284902+0000","last_change":"2024-08-17T17:24:17.452025+0000","last_active":"2024-08-17T17:59:59.284902+0000","last_peered":"2024-08-17T17:59:59.284902+0000","last_clean":"2024-08-17T17:59:59.284902+0000","last_became_active":"2024-08-17T17:24:17.448029+0000","last_became_peered":"2024-08-17T17:24:17.448029+0000","last_unstale":"2024-08-17T17:59:59.284902+0000","last_undegraded":"2024-08-17T17:59:59.284902+0000","last_fullsized":"2024-08-17T17:59:59.284902+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:45.089346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":708,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.285031+0000","last_change":"2024-08-17T17:58:23.586641+0000","last_active":"2024-08-17T17:59:59.285031+0000","last_peered":"2024-08-17T17:59:59.285031+0000","last_clean":"2024-08-17T17:59:59.285031+0000","last_became_active":"2024-08-17T17:58:23.585843+0000","last_became_peered":"2024-08-17T17:58:23.585843+0000","last_unstale":"2024-08-17T17:59:59.285031+0000","last_undegraded":"2024-08-17T17:59:59.285031+0000","last_fullsized":"2024-08-17T17:59:59.285031+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:34:00.228118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2078,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.284246+0000","last_change":"2024-08-17T17:58:23.584638+0000","last_active":"2024-08-17T17:59:59.284246+0000","last_peered":"2024-08-17T17:59:59.284246+0000","last_clean":"2024-08-17T17:59:59.284246+0000","last_became_active":"2024-08-17T17:58:23.583535+0000","last_became_peered":"2024-08-17T17:58:23.583535+0000","last_unstale":"2024-08-17T17:59:59.284246+0000","last_undegraded":"2024-08-17T17:59:59.284246+0000","last_fullsized":"2024-08-17T17:59:59.284246+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:00:25.957667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1362,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.285896+0000","last_change":"2024-08-17T17:23:07.402338+0000","last_active":"2024-08-17T17:59:59.285896+0000","last_peered":"2024-08-17T17:59:59.285896+0000","last_clean":"2024-08-17T17:59:59.285896+0000","last_became_active":"2024-08-17T17:23:07.401996+0000","last_became_peered":"2024-08-17T17:23:07.401996+0000","last_unstale":"2024-08-17T17:59:59.285896+0000","last_undegraded":"2024-08-17T17:59:59.285896+0000","last_fullsized":"2024-08-17T17:59:59.285896+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:39:21.566786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2056,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.285338+0000","last_change":"2024-08-17T17:24:17.442338+0000","last_active":"2024-08-17T17:59:59.285338+0000","last_peered":"2024-08-17T17:59:59.285338+0000","last_clean":"2024-08-17T17:59:59.285338+0000","last_became_active":"2024-08-17T17:24:17.441948+0000","last_became_peered":"2024-08-17T17:24:17.441948+0000","last_unstale":"2024-08-17T17:59:59.285338+0000","last_undegraded":"2024-08-17T17:59:59.285338+0000","last_fullsized":"2024-08-17T17:59:59.285338+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:17.891492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":685,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.574017+0000","last_change":"2024-08-17T17:58:23.589701+0000","last_active":"2024-08-17T17:59:47.574017+0000","last_peered":"2024-08-17T17:59:47.574017+0000","last_clean":"2024-08-17T17:59:47.574017+0000","last_became_active":"2024-08-17T17:58:23.589393+0000","last_became_peered":"2024-08-17T17:58:23.589393+0000","last_unstale":"2024-08-17T17:59:47.574017+0000","last_undegraded":"2024-08-17T17:59:47.574017+0000","last_fullsized":"2024-08-17T17:59:47.574017+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:41:03.356477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2015,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.283346+0000","last_change":"2024-08-17T17:24:17.468086+0000","last_active":"2024-08-17T17:59:59.283346+0000","last_peered":"2024-08-17T17:59:59.283346+0000","last_clean":"2024-08-17T17:59:59.283346+0000","last_became_active":"2024-08-17T17:24:17.467522+0000","last_became_peered":"2024-08-17T17:24:17.467522+0000","last_unstale":"2024-08-17T17:59:59.283346+0000","last_undegraded":"2024-08-17T17:59:59.283346+0000","last_fullsized":"2024-08-17T17:59:59.283346+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:52:55.249649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1357,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.573239+0000","last_change":"2024-08-17T17:23:07.392447+0000","last_active":"2024-08-17T17:59:47.573239+0000","last_peered":"2024-08-17T17:59:47.573239+0000","last_clean":"2024-08-17T17:59:47.573239+0000","last_became_active":"2024-08-17T17:23:07.391765+0000","last_became_peered":"2024-08-17T17:23:07.391765+0000","last_unstale":"2024-08-17T17:59:47.573239+0000","last_undegraded":"2024-08-17T17:59:47.573239+0000","last_fullsized":"2024-08-17T17:59:47.573239+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:59:18.602346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":167,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.288746+0000","last_change":"2024-08-17T17:53:58.839889+0000","last_active":"2024-08-17T17:59:59.288746+0000","last_peered":"2024-08-17T17:59:59.288746+0000","last_clean":"2024-08-17T17:59:59.288746+0000","last_became_active":"2024-08-17T17:53:58.839537+0000","last_became_peered":"2024-08-17T17:53:58.839537+0000","last_unstale":"2024-08-17T17:59:59.288746+0000","last_undegraded":"2024-08-17T17:59:59.288746+0000","last_fullsized":"2024-08-17T17:59:59.288746+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:47.169954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2082,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.273940+0000","last_change":"2024-08-17T17:23:42.936147+0000","last_active":"2024-08-17T17:59:59.273940+0000","last_peered":"2024-08-17T17:59:59.273940+0000","last_clean":"2024-08-17T17:59:59.273940+0000","last_became_active":"2024-08-17T17:23:42.935859+0000","last_became_peered":"2024-08-17T17:23:42.935859+0000","last_unstale":"2024-08-17T17:59:59.273940+0000","last_undegraded":"2024-08-17T17:59:59.273940+0000","last_fullsized":"2024-08-17T17:59:59.273940+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:47:36.838706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1371,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.599945+0000","last_change":"2024-08-17T17:10:05.969131+0000","last_active":"2024-08-17T17:59:47.599945+0000","last_peered":"2024-08-17T17:59:47.599945+0000","last_clean":"2024-08-17T17:59:47.599945+0000","last_became_active":"2024-08-17T17:10:05.968856+0000","last_became_peered":"2024-08-17T17:10:05.968856+0000","last_unstale":"2024-08-17T17:59:47.599945+0000","last_undegraded":"2024-08-17T17:59:47.599945+0000","last_fullsized":"2024-08-17T17:59:47.599945+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:25:49.550065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1362,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.599422+0000","last_change":"2024-08-17T17:23:07.455919+0000","last_active":"2024-08-17T17:59:47.599422+0000","last_peered":"2024-08-17T17:59:47.599422+0000","last_clean":"2024-08-17T17:59:47.599422+0000","last_became_active":"2024-08-17T17:23:07.455284+0000","last_became_peered":"2024-08-17T17:23:07.455284+0000","last_unstale":"2024-08-17T17:59:47.599422+0000","last_undegraded":"2024-08-17T17:59:47.599422+0000","last_fullsized":"2024-08-17T17:59:47.599422+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:37:46.260775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1350,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.287694+0000","last_change":"2024-08-17T17:10:06.979492+0000","last_active":"2024-08-17T17:59:59.287694+0000","last_peered":"2024-08-17T17:59:59.287694+0000","last_clean":"2024-08-17T17:59:59.287694+0000","last_became_active":"2024-08-17T17:10:06.979158+0000","last_became_peered":"2024-08-17T17:10:06.979158+0000","last_unstale":"2024-08-17T17:59:59.287694+0000","last_undegraded":"2024-08-17T17:59:59.287694+0000","last_fullsized":"2024-08-17T17:59:59.287694+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:12:42.261365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1374,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.703773+0000","last_change":"2024-08-17T17:23:42.935099+0000","last_active":"2024-08-17T17:59:47.703773+0000","last_peered":"2024-08-17T17:59:47.703773+0000","last_clean":"2024-08-17T17:59:47.703773+0000","last_became_active":"2024-08-17T17:23:42.934501+0000","last_became_peered":"2024-08-17T17:23:42.934501+0000","last_unstale":"2024-08-17T17:59:47.703773+0000","last_undegraded":"2024-08-17T17:59:47.703773+0000","last_fullsized":"2024-08-17T17:59:47.703773+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:15:19.095713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":83,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.270059+0000","last_change":"2024-08-17T17:58:23.586651+0000","last_active":"2024-08-17T17:59:59.270059+0000","last_peered":"2024-08-17T17:59:59.270059+0000","last_clean":"2024-08-17T17:59:59.270059+0000","last_became_active":"2024-08-17T17:58:23.580712+0000","last_became_peered":"2024-08-17T17:58:23.580712+0000","last_unstale":"2024-08-17T17:59:59.270059+0000","last_undegraded":"2024-08-17T17:59:59.270059+0000","last_fullsized":"2024-08-17T17:59:59.270059+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:40:33.674413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2065,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.285242+0000","last_change":"2024-08-17T17:59:10.888817+0000","last_active":"2024-08-17T17:59:59.285242+0000","last_peered":"2024-08-17T17:59:59.285242+0000","last_clean":"2024-08-17T17:59:59.285242+0000","last_became_active":"2024-08-17T17:59:10.888483+0000","last_became_peered":"2024-08-17T17:59:10.888483+0000","last_unstale":"2024-08-17T17:59:59.285242+0000","last_undegraded":"2024-08-17T17:59:59.285242+0000","last_fullsized":"2024-08-17T17:59:59.285242+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:26.369367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2065,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.270202+0000","last_change":"2024-08-17T17:23:07.428296+0000","last_active":"2024-08-17T17:59:59.270202+0000","last_peered":"2024-08-17T17:59:59.270202+0000","last_clean":"2024-08-17T17:59:59.270202+0000","last_became_active":"2024-08-17T17:23:07.428010+0000","last_became_peered":"2024-08-17T17:23:07.428010+0000","last_unstale":"2024-08-17T17:59:59.270202+0000","last_undegraded":"2024-08-17T17:59:59.270202+0000","last_fullsized":"2024-08-17T17:59:59.270202+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:45.459212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":2063,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.285467+0000","last_change":"2024-08-17T17:58:23.594285+0000","last_active":"2024-08-17T17:59:59.285467+0000","last_peered":"2024-08-17T17:59:59.285467+0000","last_clean":"2024-08-17T17:59:59.285467+0000","last_became_active":"2024-08-17T17:58:23.588147+0000","last_became_peered":"2024-08-17T17:58:23.588147+0000","last_unstale":"2024-08-17T17:59:59.285467+0000","last_undegraded":"2024-08-17T17:59:59.285467+0000","last_fullsized":"2024-08-17T17:59:59.285467+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:12:40.216512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":620,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.573606+0000","last_change":"2024-08-17T17:58:23.579834+0000","last_active":"2024-08-17T17:59:47.573606+0000","last_peered":"2024-08-17T17:59:47.573606+0000","last_clean":"2024-08-17T17:59:47.573606+0000","last_became_active":"2024-08-17T17:58:23.579507+0000","last_became_peered":"2024-08-17T17:58:23.579507+0000","last_unstale":"2024-08-17T17:59:47.573606+0000","last_undegraded":"2024-08-17T17:59:47.573606+0000","last_fullsized":"2024-08-17T17:59:47.573606+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:43:24.177069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1379,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.705446+0000","last_change":"2024-08-17T17:58:23.582011+0000","last_active":"2024-08-17T17:59:47.705446+0000","last_peered":"2024-08-17T17:59:47.705446+0000","last_clean":"2024-08-17T17:59:47.705446+0000","last_became_active":"2024-08-17T17:58:23.581649+0000","last_became_peered":"2024-08-17T17:58:23.581649+0000","last_unstale":"2024-08-17T17:59:47.705446+0000","last_undegraded":"2024-08-17T17:59:47.705446+0000","last_fullsized":"2024-08-17T17:59:47.705446+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:38:55.478552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2086,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.270574+0000","last_change":"2024-08-17T17:23:42.935825+0000","last_active":"2024-08-17T17:59:59.270574+0000","last_peered":"2024-08-17T17:59:59.270574+0000","last_clean":"2024-08-17T17:59:59.270574+0000","last_became_active":"2024-08-17T17:23:42.935150+0000","last_became_peered":"2024-08-17T17:23:42.935150+0000","last_unstale":"2024-08-17T17:59:59.270574+0000","last_undegraded":"2024-08-17T17:59:59.270574+0000","last_fullsized":"2024-08-17T17:59:59.270574+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:57:19.577644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":36,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.293992+0000","last_change":"2024-08-17T17:59:10.888598+0000","last_active":"2024-08-17T17:59:59.293992+0000","last_peered":"2024-08-17T17:59:59.293992+0000","last_clean":"2024-08-17T17:59:59.293992+0000","last_became_active":"2024-08-17T17:59:10.888289+0000","last_became_peered":"2024-08-17T17:59:10.888289+0000","last_unstale":"2024-08-17T17:59:59.293992+0000","last_undegraded":"2024-08-17T17:59:59.293992+0000","last_fullsized":"2024-08-17T17:59:59.293992+0000","mapping_epoch":1376,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1377,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:34:39.372046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2109,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.270379+0000","last_change":"2024-08-17T17:58:23.581351+0000","last_active":"2024-08-17T17:59:59.270379+0000","last_peered":"2024-08-17T17:59:59.270379+0000","last_clean":"2024-08-17T17:59:59.270379+0000","last_became_active":"2024-08-17T17:58:23.580717+0000","last_became_peered":"2024-08-17T17:58:23.580717+0000","last_unstale":"2024-08-17T17:59:59.270379+0000","last_undegraded":"2024-08-17T17:59:59.270379+0000","last_fullsized":"2024-08-17T17:59:59.270379+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:02:47.291227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1375,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.573749+0000","last_change":"2024-08-17T17:24:17.426267+0000","last_active":"2024-08-17T17:59:47.573749+0000","last_peered":"2024-08-17T17:59:47.573749+0000","last_clean":"2024-08-17T17:59:47.573749+0000","last_became_active":"2024-08-17T17:24:17.425925+0000","last_became_peered":"2024-08-17T17:24:17.425925+0000","last_unstale":"2024-08-17T17:59:47.573749+0000","last_undegraded":"2024-08-17T17:59:47.573749+0000","last_fullsized":"2024-08-17T17:59:47.573749+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:24:25.017856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1371,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.573590+0000","last_change":"2024-08-17T17:24:17.474138+0000","last_active":"2024-08-17T17:59:47.573590+0000","last_peered":"2024-08-17T17:59:47.573590+0000","last_clean":"2024-08-17T17:59:47.573590+0000","last_became_active":"2024-08-17T17:24:17.473708+0000","last_became_peered":"2024-08-17T17:24:17.473708+0000","last_unstale":"2024-08-17T17:59:47.573590+0000","last_undegraded":"2024-08-17T17:59:47.573590+0000","last_fullsized":"2024-08-17T17:59:47.573590+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:10:57.768067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1374,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.599860+0000","last_change":"2024-08-17T17:58:23.575900+0000","last_active":"2024-08-17T17:59:47.599860+0000","last_peered":"2024-08-17T17:59:47.599860+0000","last_clean":"2024-08-17T17:59:47.599860+0000","last_became_active":"2024-08-17T17:58:23.575606+0000","last_became_peered":"2024-08-17T17:58:23.575606+0000","last_unstale":"2024-08-17T17:59:47.599860+0000","last_undegraded":"2024-08-17T17:59:47.599860+0000","last_fullsized":"2024-08-17T17:59:47.599860+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:17:46.057413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":1375,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.573210+0000","last_change":"2024-08-17T17:23:07.391021+0000","last_active":"2024-08-17T17:59:47.573210+0000","last_peered":"2024-08-17T17:59:47.573210+0000","last_clean":"2024-08-17T17:59:47.573210+0000","last_became_active":"2024-08-17T17:23:07.390551+0000","last_became_peered":"2024-08-17T17:23:07.390551+0000","last_unstale":"2024-08-17T17:59:47.573210+0000","last_undegraded":"2024-08-17T17:59:47.573210+0000","last_fullsized":"2024-08-17T17:59:47.573210+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:06.239013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001023532,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2093,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.270674+0000","last_change":"2024-08-17T17:58:23.580871+0000","last_active":"2024-08-17T17:59:59.270674+0000","last_peered":"2024-08-17T17:59:59.270674+0000","last_clean":"2024-08-17T17:59:59.270674+0000","last_became_active":"2024-08-17T17:58:23.580542+0000","last_became_peered":"2024-08-17T17:58:23.580542+0000","last_unstale":"2024-08-17T17:59:59.270674+0000","last_undegraded":"2024-08-17T17:59:59.270674+0000","last_fullsized":"2024-08-17T17:59:59.270674+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:51:14.019944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2066,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.273369+0000","last_change":"2024-08-17T17:23:42.910246+0000","last_active":"2024-08-17T17:59:59.273369+0000","last_peered":"2024-08-17T17:59:59.273369+0000","last_clean":"2024-08-17T17:59:59.273369+0000","last_became_active":"2024-08-17T17:23:42.909929+0000","last_became_peered":"2024-08-17T17:23:42.909929+0000","last_unstale":"2024-08-17T17:59:59.273369+0000","last_undegraded":"2024-08-17T17:59:59.273369+0000","last_fullsized":"2024-08-17T17:59:59.273369+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:14:45.210558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2101,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.295206+0000","last_change":"2024-08-17T17:58:23.586426+0000","last_active":"2024-08-17T17:59:59.295206+0000","last_peered":"2024-08-17T17:59:59.295206+0000","last_clean":"2024-08-17T17:59:59.295206+0000","last_became_active":"2024-08-17T17:58:23.585776+0000","last_became_peered":"2024-08-17T17:58:23.585776+0000","last_unstale":"2024-08-17T17:59:59.295206+0000","last_undegraded":"2024-08-17T17:59:59.295206+0000","last_fullsized":"2024-08-17T17:59:59.295206+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:39:33.064885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":695,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.269976+0000","last_change":"2024-08-17T17:58:23.582327+0000","last_active":"2024-08-17T17:59:59.269976+0000","last_peered":"2024-08-17T17:59:59.269976+0000","last_clean":"2024-08-17T17:59:59.269976+0000","last_became_active":"2024-08-17T17:58:23.581984+0000","last_became_peered":"2024-08-17T17:58:23.581984+0000","last_unstale":"2024-08-17T17:59:59.269976+0000","last_undegraded":"2024-08-17T17:59:59.269976+0000","last_fullsized":"2024-08-17T17:59:59.269976+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:10:23.678071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":686,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.599552+0000","last_change":"2024-08-17T17:58:23.583104+0000","last_active":"2024-08-17T17:59:47.599552+0000","last_peered":"2024-08-17T17:59:47.599552+0000","last_clean":"2024-08-17T17:59:47.599552+0000","last_became_active":"2024-08-17T17:58:23.582784+0000","last_became_peered":"2024-08-17T17:58:23.582784+0000","last_unstale":"2024-08-17T17:59:47.599552+0000","last_undegraded":"2024-08-17T17:59:47.599552+0000","last_fullsized":"2024-08-17T17:59:47.599552+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:39:37.921577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":807,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.273599+0000","last_change":"2024-08-17T17:58:23.585368+0000","last_active":"2024-08-17T17:59:59.273599+0000","last_peered":"2024-08-17T17:59:59.273599+0000","last_clean":"2024-08-17T17:59:59.273599+0000","last_became_active":"2024-08-17T17:58:23.584864+0000","last_became_peered":"2024-08-17T17:58:23.584864+0000","last_unstale":"2024-08-17T17:59:59.273599+0000","last_undegraded":"2024-08-17T17:59:59.273599+0000","last_fullsized":"2024-08-17T17:59:59.273599+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:51.734726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1346,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.703617+0000","last_change":"2024-08-17T17:10:06.995023+0000","last_active":"2024-08-17T17:59:47.703617+0000","last_peered":"2024-08-17T17:59:47.703617+0000","last_clean":"2024-08-17T17:59:47.703617+0000","last_became_active":"2024-08-17T17:10:06.994757+0000","last_became_peered":"2024-08-17T17:10:06.994757+0000","last_unstale":"2024-08-17T17:59:47.703617+0000","last_undegraded":"2024-08-17T17:59:47.703617+0000","last_fullsized":"2024-08-17T17:59:47.703617+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:31:52.163518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2018,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.283582+0000","last_change":"2024-08-17T17:24:17.468443+0000","last_active":"2024-08-17T17:59:59.283582+0000","last_peered":"2024-08-17T17:59:59.283582+0000","last_clean":"2024-08-17T17:59:59.283582+0000","last_became_active":"2024-08-17T17:24:17.467394+0000","last_became_peered":"2024-08-17T17:24:17.467394+0000","last_unstale":"2024-08-17T17:59:59.283582+0000","last_undegraded":"2024-08-17T17:59:59.283582+0000","last_fullsized":"2024-08-17T17:59:59.283582+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:13.024253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":1374,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.573691+0000","last_change":"2024-08-17T17:23:42.920552+0000","last_active":"2024-08-17T17:59:47.573691+0000","last_peered":"2024-08-17T17:59:47.573691+0000","last_clean":"2024-08-17T17:59:47.573691+0000","last_became_active":"2024-08-17T17:23:42.915824+0000","last_became_peered":"2024-08-17T17:23:42.915824+0000","last_unstale":"2024-08-17T17:59:47.573691+0000","last_undegraded":"2024-08-17T17:59:47.573691+0000","last_fullsized":"2024-08-17T17:59:47.573691+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:25:18.769504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041737399999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":661,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.703408+0000","last_change":"2024-08-17T17:58:26.598586+0000","last_active":"2024-08-17T17:59:47.703408+0000","last_peered":"2024-08-17T17:59:47.703408+0000","last_clean":"2024-08-17T17:59:47.703408+0000","last_became_active":"2024-08-17T17:58:26.598163+0000","last_became_peered":"2024-08-17T17:58:26.598163+0000","last_unstale":"2024-08-17T17:59:47.703408+0000","last_undegraded":"2024-08-17T17:59:47.703408+0000","last_fullsized":"2024-08-17T17:59:47.703408+0000","mapping_epoch":1370,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T17:06:21.174471+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:36:48.182691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2064,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.270539+0000","last_change":"2024-08-17T17:24:17.455710+0000","last_active":"2024-08-17T17:59:59.270539+0000","last_peered":"2024-08-17T17:59:59.270539+0000","last_clean":"2024-08-17T17:59:59.270539+0000","last_became_active":"2024-08-17T17:24:17.452989+0000","last_became_peered":"2024-08-17T17:24:17.452989+0000","last_unstale":"2024-08-17T17:59:59.270539+0000","last_undegraded":"2024-08-17T17:59:59.270539+0000","last_fullsized":"2024-08-17T17:59:59.270539+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:58:07.894968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2058,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.284164+0000","last_change":"2024-08-17T17:24:17.453639+0000","last_active":"2024-08-17T17:59:59.284164+0000","last_peered":"2024-08-17T17:59:59.284164+0000","last_clean":"2024-08-17T17:59:59.284164+0000","last_became_active":"2024-08-17T17:24:17.443638+0000","last_became_peered":"2024-08-17T17:24:17.443638+0000","last_unstale":"2024-08-17T17:59:59.284164+0000","last_undegraded":"2024-08-17T17:59:59.284164+0000","last_fullsized":"2024-08-17T17:59:59.284164+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:42.107479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1363,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.599429+0000","last_change":"2024-08-17T17:23:07.451866+0000","last_active":"2024-08-17T17:59:47.599429+0000","last_peered":"2024-08-17T17:59:47.599429+0000","last_clean":"2024-08-17T17:59:47.599429+0000","last_became_active":"2024-08-17T17:23:07.451553+0000","last_became_peered":"2024-08-17T17:23:07.451553+0000","last_unstale":"2024-08-17T17:59:47.599429+0000","last_undegraded":"2024-08-17T17:59:47.599429+0000","last_fullsized":"2024-08-17T17:59:47.599429+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:37:23.930651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1365,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.287759+0000","last_change":"2024-08-17T17:23:42.915851+0000","last_active":"2024-08-17T17:59:59.287759+0000","last_peered":"2024-08-17T17:59:59.287759+0000","last_clean":"2024-08-17T17:59:59.287759+0000","last_became_active":"2024-08-17T17:23:42.915552+0000","last_became_peered":"2024-08-17T17:23:42.915552+0000","last_unstale":"2024-08-17T17:59:59.287759+0000","last_undegraded":"2024-08-17T17:59:59.287759+0000","last_fullsized":"2024-08-17T17:59:59.287759+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:45.197876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1378,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.599925+0000","last_change":"2024-08-17T17:23:42.930803+0000","last_active":"2024-08-17T17:59:47.599925+0000","last_peered":"2024-08-17T17:59:47.599925+0000","last_clean":"2024-08-17T17:59:47.599925+0000","last_became_active":"2024-08-17T17:23:42.930458+0000","last_became_peered":"2024-08-17T17:23:42.930458+0000","last_unstale":"2024-08-17T17:59:47.599925+0000","last_undegraded":"2024-08-17T17:59:47.599925+0000","last_fullsized":"2024-08-17T17:59:47.599925+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:40.816706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1362,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.285761+0000","last_change":"2024-08-17T17:23:07.401624+0000","last_active":"2024-08-17T17:59:59.285761+0000","last_peered":"2024-08-17T17:59:59.285761+0000","last_clean":"2024-08-17T17:59:59.285761+0000","last_became_active":"2024-08-17T17:23:07.400222+0000","last_became_peered":"2024-08-17T17:23:07.400222+0000","last_unstale":"2024-08-17T17:59:59.285761+0000","last_undegraded":"2024-08-17T17:59:59.285761+0000","last_fullsized":"2024-08-17T17:59:59.285761+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:07.069989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1363,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.599919+0000","last_change":"2024-08-17T17:24:17.470809+0000","last_active":"2024-08-17T17:59:47.599919+0000","last_peered":"2024-08-17T17:59:47.599919+0000","last_clean":"2024-08-17T17:59:47.599919+0000","last_became_active":"2024-08-17T17:24:17.469753+0000","last_became_peered":"2024-08-17T17:24:17.469753+0000","last_unstale":"2024-08-17T17:59:47.599919+0000","last_undegraded":"2024-08-17T17:59:47.599919+0000","last_fullsized":"2024-08-17T17:59:47.599919+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:00.049425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":2062,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.285209+0000","last_change":"2024-08-17T17:58:23.592069+0000","last_active":"2024-08-17T17:59:59.285209+0000","last_peered":"2024-08-17T17:59:59.285209+0000","last_clean":"2024-08-17T17:59:59.285209+0000","last_became_active":"2024-08-17T17:58:23.584024+0000","last_became_peered":"2024-08-17T17:58:23.584024+0000","last_unstale":"2024-08-17T17:59:59.285209+0000","last_undegraded":"2024-08-17T17:59:59.285209+0000","last_fullsized":"2024-08-17T17:59:59.285209+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:37:17.459578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":698,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.294936+0000","last_change":"2024-08-17T17:58:23.581996+0000","last_active":"2024-08-17T17:59:59.294936+0000","last_peered":"2024-08-17T17:59:59.294936+0000","last_clean":"2024-08-17T17:59:59.294936+0000","last_became_active":"2024-08-17T17:58:23.580469+0000","last_became_peered":"2024-08-17T17:58:23.580469+0000","last_unstale":"2024-08-17T17:59:59.294936+0000","last_undegraded":"2024-08-17T17:59:59.294936+0000","last_fullsized":"2024-08-17T17:59:59.294936+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:01:28.612769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1359,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.573635+0000","last_change":"2024-08-17T17:24:17.473913+0000","last_active":"2024-08-17T17:59:47.573635+0000","last_peered":"2024-08-17T17:59:47.573635+0000","last_clean":"2024-08-17T17:59:47.573635+0000","last_became_active":"2024-08-17T17:24:17.473195+0000","last_became_peered":"2024-08-17T17:24:17.473195+0000","last_unstale":"2024-08-17T17:59:47.573635+0000","last_undegraded":"2024-08-17T17:59:47.573635+0000","last_fullsized":"2024-08-17T17:59:47.573635+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:12.998033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1361,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.285477+0000","last_change":"2024-08-17T17:23:07.402010+0000","last_active":"2024-08-17T17:59:59.285477+0000","last_peered":"2024-08-17T17:59:59.285477+0000","last_clean":"2024-08-17T17:59:59.285477+0000","last_became_active":"2024-08-17T17:23:07.400849+0000","last_became_peered":"2024-08-17T17:23:07.400849+0000","last_unstale":"2024-08-17T17:59:59.285477+0000","last_undegraded":"2024-08-17T17:59:59.285477+0000","last_fullsized":"2024-08-17T17:59:59.285477+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:16:02.382257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":96,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.573546+0000","last_change":"2024-08-17T17:53:58.836811+0000","last_active":"2024-08-17T17:59:47.573546+0000","last_peered":"2024-08-17T17:59:47.573546+0000","last_clean":"2024-08-17T17:59:47.573546+0000","last_became_active":"2024-08-17T17:53:58.836139+0000","last_became_peered":"2024-08-17T17:53:58.836139+0000","last_unstale":"2024-08-17T17:59:47.573546+0000","last_undegraded":"2024-08-17T17:59:47.573546+0000","last_fullsized":"2024-08-17T17:59:47.573546+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:09:18.260164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1379,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.287523+0000","last_change":"2024-08-17T17:24:17.460203+0000","last_active":"2024-08-17T17:59:59.287523+0000","last_peered":"2024-08-17T17:59:59.287523+0000","last_clean":"2024-08-17T17:59:59.287523+0000","last_became_active":"2024-08-17T17:24:17.459682+0000","last_became_peered":"2024-08-17T17:24:17.459682+0000","last_unstale":"2024-08-17T17:59:59.287523+0000","last_undegraded":"2024-08-17T17:59:59.287523+0000","last_fullsized":"2024-08-17T17:59:59.287523+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:58.703700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":1358,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.704680+0000","last_change":"2024-08-17T17:23:07.454042+0000","last_active":"2024-08-17T17:59:47.704680+0000","last_peered":"2024-08-17T17:59:47.704680+0000","last_clean":"2024-08-17T17:59:47.704680+0000","last_became_active":"2024-08-17T17:23:07.453712+0000","last_became_peered":"2024-08-17T17:23:07.453712+0000","last_unstale":"2024-08-17T17:59:47.704680+0000","last_undegraded":"2024-08-17T17:59:47.704680+0000","last_fullsized":"2024-08-17T17:59:47.704680+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:03:07.643456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1344,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.284012+0000","last_change":"2024-08-17T17:10:06.994799+0000","last_active":"2024-08-17T17:59:59.284012+0000","last_peered":"2024-08-17T17:59:59.284012+0000","last_clean":"2024-08-17T17:59:59.284012+0000","last_became_active":"2024-08-17T17:10:06.994150+0000","last_became_peered":"2024-08-17T17:10:06.994150+0000","last_unstale":"2024-08-17T17:59:59.284012+0000","last_undegraded":"2024-08-17T17:59:59.284012+0000","last_fullsized":"2024-08-17T17:59:59.284012+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:50:47.293091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":1369,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.705625+0000","last_change":"2024-08-17T17:58:23.589073+0000","last_active":"2024-08-17T17:59:47.705625+0000","last_peered":"2024-08-17T17:59:47.705625+0000","last_clean":"2024-08-17T17:59:47.705625+0000","last_became_active":"2024-08-17T17:58:23.583387+0000","last_became_peered":"2024-08-17T17:58:23.583387+0000","last_unstale":"2024-08-17T17:59:47.705625+0000","last_undegraded":"2024-08-17T17:59:47.705625+0000","last_fullsized":"2024-08-17T17:59:47.705625+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:23:27.036125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1387,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.703338+0000","last_change":"2024-08-17T17:58:23.592768+0000","last_active":"2024-08-17T17:59:47.703338+0000","last_peered":"2024-08-17T17:59:47.703338+0000","last_clean":"2024-08-17T17:59:47.703338+0000","last_became_active":"2024-08-17T17:58:23.582210+0000","last_became_peered":"2024-08-17T17:58:23.582210+0000","last_unstale":"2024-08-17T17:59:47.703338+0000","last_undegraded":"2024-08-17T17:59:47.703338+0000","last_fullsized":"2024-08-17T17:59:47.703338+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:31.713201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2055,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.285435+0000","last_change":"2024-08-17T17:24:17.451528+0000","last_active":"2024-08-17T17:59:59.285435+0000","last_peered":"2024-08-17T17:59:59.285435+0000","last_clean":"2024-08-17T17:59:59.285435+0000","last_became_active":"2024-08-17T17:24:17.451246+0000","last_became_peered":"2024-08-17T17:24:17.451246+0000","last_unstale":"2024-08-17T17:59:59.285435+0000","last_undegraded":"2024-08-17T17:59:59.285435+0000","last_fullsized":"2024-08-17T17:59:59.285435+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:38:18.176559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":84,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.599641+0000","last_change":"2024-08-17T17:53:58.838279+0000","last_active":"2024-08-17T17:59:47.599641+0000","last_peered":"2024-08-17T17:59:47.599641+0000","last_clean":"2024-08-17T17:59:47.599641+0000","last_became_active":"2024-08-17T17:53:58.837978+0000","last_became_peered":"2024-08-17T17:53:58.837978+0000","last_unstale":"2024-08-17T17:59:47.599641+0000","last_undegraded":"2024-08-17T17:59:47.599641+0000","last_fullsized":"2024-08-17T17:59:47.599641+0000","mapping_epoch":1318,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:06.019142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":707,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.284950+0000","last_change":"2024-08-17T17:58:23.586407+0000","last_active":"2024-08-17T17:59:59.284950+0000","last_peered":"2024-08-17T17:59:59.284950+0000","last_clean":"2024-08-17T17:59:59.284950+0000","last_became_active":"2024-08-17T17:58:23.584967+0000","last_became_peered":"2024-08-17T17:58:23.584967+0000","last_unstale":"2024-08-17T17:59:59.284950+0000","last_undegraded":"2024-08-17T17:59:59.284950+0000","last_fullsized":"2024-08-17T17:59:59.284950+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:58:08.040251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":707,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.283664+0000","last_change":"2024-08-17T17:58:23.584794+0000","last_active":"2024-08-17T17:59:59.283664+0000","last_peered":"2024-08-17T17:59:59.283664+0000","last_clean":"2024-08-17T17:59:59.283664+0000","last_became_active":"2024-08-17T17:58:23.584485+0000","last_became_peered":"2024-08-17T17:58:23.584485+0000","last_unstale":"2024-08-17T17:59:59.283664+0000","last_undegraded":"2024-08-17T17:59:59.283664+0000","last_fullsized":"2024-08-17T17:59:59.283664+0000","mapping_epoch":1367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:26:11.516977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1356,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.573710+0000","last_change":"2024-08-17T17:24:17.451287+0000","last_active":"2024-08-17T17:59:47.573710+0000","last_peered":"2024-08-17T17:59:47.573710+0000","last_clean":"2024-08-17T17:59:47.573710+0000","last_became_active":"2024-08-17T17:24:17.450951+0000","last_became_peered":"2024-08-17T17:24:17.450951+0000","last_unstale":"2024-08-17T17:59:47.573710+0000","last_undegraded":"2024-08-17T17:59:47.573710+0000","last_fullsized":"2024-08-17T17:59:47.573710+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:13:37.590108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":1357,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-08-17T17:59:47.705579+0000","last_change":"2024-08-17T17:23:42.907970+0000","last_active":"2024-08-17T17:59:47.705579+0000","last_peered":"2024-08-17T17:59:47.705579+0000","last_clean":"2024-08-17T17:59:47.705579+0000","last_became_active":"2024-08-17T17:23:42.907595+0000","last_became_peered":"2024-08-17T17:23:42.907595+0000","last_unstale":"2024-08-17T17:59:47.705579+0000","last_undegraded":"2024-08-17T17:59:47.705579+0000","last_fullsized":"2024-08-17T17:59:47.705579+0000","mapping_epoch":674,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:11:46.130004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2056,"reported_epoch":1389,"state":"active+clean","last_fresh":"2024-08-17T17:59:59.287959+0000","last_change":"2024-08-17T17:24:17.486591+0000","last_active":"2024-08-17T17:59:59.287959+0000","last_peered":"2024-08-17T17:59:59.287959+0000","last_clean":"2024-08-17T17:59:59.287959+0000","last_became_active":"2024-08-17T17:24:17.486320+0000","last_became_peered":"2024-08-17T17:24:17.486320+0000","last_unstale":"2024-08-17T17:59:59.287959+0000","last_undegraded":"2024-08-17T17:59:59.287959+0000","last_fullsized":"2024-08-17T17:59:59.287959+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:03.249803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":54,"num_pg":26,"stat_sum":{"num_bytes":7980253208,"num_objects":121770,"num_object_clones":0,"num_object_copies":365310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6350,"num_objects_misplaced":7382,"num_objects_unfound":0,"num_objects_dirty":121770,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123996,"num_write_kb":7864449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2781,"num_bytes_recovered":182255616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24405749760,"data_stored":24405737544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":175690,"ondisk_log_size":175690,"up":78,"acting":76,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":604,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":17266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234609,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4207228,"kb_used_data":3905460,"kb_used_omap":1,"kb_used_meta":301758,"kb_avail":89537924,"statfs":{"total":95995035648,"available":91686834176,"internally_reserved":0,"allocated":3999191040,"data_stored":3994771447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":309000651},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496247,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4336892,"kb_used_data":3962932,"kb_used_omap":1,"kb_used_meta":373950,"kb_avail":89408260,"statfs":{"total":95995035648,"available":91554058240,"internally_reserved":0,"allocated":4058042368,"data_stored":4053623254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":382925255},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565488,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6387556,"kb_used_data":5972252,"kb_used_omap":72,"kb_used_meta":415223,"kb_avail":87357596,"statfs":{"total":95995035648,"available":89454178304,"internally_reserved":0,"allocated":6115586048,"data_stored":6111185641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":425188846},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":674,"seq":2894807957943,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2341732,"kb_used_data":2020892,"kb_used_omap":74,"kb_used_meta":320757,"kb_avail":91403420,"statfs":{"total":95995035648,"available":93597102080,"internally_reserved":0,"allocated":2069393408,"data_stored":2064993503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76116,"internal_metadata":328455852},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743186878,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2231720,"kb_used_data":1842912,"kb_used_omap":66,"kb_used_meta":388733,"kb_avail":91513432,"statfs":{"total":95995035648,"available":93709754368,"internally_reserved":0,"allocated":1887141888,"data_stored":1882737856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":398063275},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":2830383448517,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757883,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4767788,"kb_used_data":4371556,"kb_used_omap":1,"kb_used_meta":396158,"kb_avail":88977364,"statfs":{"total":95995035648,"available":91112820736,"internally_reserved":0,"allocated":4476473344,"data_stored":4472065314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":405666246},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019520,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2341364,"kb_used_data":1905068,"kb_used_omap":1,"kb_used_meta":436286,"kb_avail":91403788,"statfs":{"total":95995035648,"available":93597478912,"internally_reserved":0,"allocated":1950789632,"data_stored":1946377617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":446757318},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1864695808,"data_stored":1864695808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2050686976,"data_stored":2050686976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6092427264,"data_stored":6092423192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1938161664,"data_stored":1938161664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4464381952,"data_stored":4464377880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4025683968,"data_stored":4025679896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3969712128,"data_stored":3969712128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T18:00:13.915 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool set unique_pool_1 pgp_num 26 2024-08-17T18:00:14.010 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:14.041 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:14.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:14.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:14.143 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:14.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:14.179 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:14.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:14.279 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:14.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:14.314 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:14.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:14.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:14.443 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:14.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:14.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:14.545 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:14.554 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:14 smithi136 ceph-mon[109861]: pgmap v2371: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.1 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 19 MiB/s wr, 969 op/s; 5974/348642 objects degraded (1.714%); 6812/348642 objects misplaced (1.954%); 6.6 MiB/s, 105 objects/s recovering 2024-08-17T18:00:14.554 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:14 smithi136 ceph-mon[115078]: pgmap v2371: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.1 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 19 MiB/s wr, 969 op/s; 5974/348642 objects degraded (1.714%); 6812/348642 objects misplaced (1.954%); 6.6 MiB/s, 105 objects/s recovering 2024-08-17T18:00:14.582 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:14.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:14.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:14.592 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-17T18:00:14.670 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:00:14.684 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:14 smithi165 ceph-mon[105517]: pgmap v2371: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.1 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 19 MiB/s wr, 969 op/s; 5974/348642 objects degraded (1.714%); 6812/348642 objects misplaced (1.954%); 6.6 MiB/s, 105 objects/s recovering 2024-08-17T18:00:14.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:14.771 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:14.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:14.872 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:14.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:14.899 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:14.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:15.000 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:15.040 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:15.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:15.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:15.142 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:15.215 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:00:15.215 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:00:15.231 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:15.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:15.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:15.237 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:00:15.255 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:00:15.256 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:00:15.267 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:00:15.291 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:00:15.292 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:00:15.299 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:00:15.333 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:15.337 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:00:15.337 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:00:15.368 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:00:15.368 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:00:15.381 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:15.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:15.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:15.393 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:15 smithi136 ceph-mon[115078]: 54.6 scrub starts 2024-08-17T18:00:15.393 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:15 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5974/348642 objects degraded (1.714%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:15.394 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:15 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2840266005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:00:15.395 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:15 smithi136 ceph-mon[109861]: 54.6 scrub starts 2024-08-17T18:00:15.395 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:15 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5974/348642 objects degraded (1.714%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:15.395 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:15 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2840266005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:00:15.400 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:00:15.401 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:00:15.433 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:00:15.433 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:00:15.470 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:00:15.470 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:00:15.483 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:15.512 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:15.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:15.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:15.614 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:15.644 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:15.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:15.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:15.746 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:15 smithi165 ceph-mon[105517]: 54.6 scrub starts 2024-08-17T18:00:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:15 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5974/348642 objects degraded (1.714%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:15 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2840266005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:00:15.777 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:15.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:15.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:15.879 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:15.911 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:15.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:15.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:16.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:16.048 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:16.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:16.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:16.151 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:16.195 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:16.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:16.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:16.296 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:16.329 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:16.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:16.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:16.431 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:16.471 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:16.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:16.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:16.573 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:16.608 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:16.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:16.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:16.628 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:16 smithi136 ceph-mon[109861]: pgmap v2372: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.1 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 19 MiB/s wr, 969 op/s; 5974/348642 objects degraded (1.714%); 6812/348642 objects misplaced (1.954%); 6.6 MiB/s, 105 objects/s recovering 2024-08-17T18:00:16.629 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:16 smithi136 ceph-mon[115078]: pgmap v2372: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.1 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 19 MiB/s wr, 969 op/s; 5974/348642 objects degraded (1.714%); 6812/348642 objects misplaced (1.954%); 6.6 MiB/s, 105 objects/s recovering 2024-08-17T18:00:16.709 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:16.750 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:16.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:16.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:16 smithi165 ceph-mon[105517]: pgmap v2372: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 7.1 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 19 MiB/s wr, 969 op/s; 5974/348642 objects degraded (1.714%); 6812/348642 objects misplaced (1.954%); 6.6 MiB/s, 105 objects/s recovering 2024-08-17T18:00:16.852 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:16.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:16.895 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:16.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:16.996 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:17.038 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:17.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:17.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:17.140 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:17.181 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:17.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:17.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:17.283 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:17.322 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:17.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:17.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:17.424 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:17.454 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:17.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:17.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:17.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:17.594 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:17.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:17.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:17.696 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:17.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:00:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:00:17] "GET /metrics HTTP/1.1" 200 36564 "" "Prometheus/2.51.0" 2024-08-17T18:00:17.733 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:17.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:17.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:17.835 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:17.871 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:17.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:17.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:17.973 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:18.013 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:18.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:18.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:18.115 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:18.149 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:18.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:18.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:18.251 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:18.284 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:18.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:18.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:18.367 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:00:18.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:18.412 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:18.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:18.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:18.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:18 smithi165 ceph-mon[105517]: pgmap v2373: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 102 B/s rd, 8.0 MiB/s wr, 1.07k op/s; 5642/339813 objects degraded (1.660%); 6261/339813 objects misplaced (1.842%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T18:00:18.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:18 smithi165 ceph-mon[105517]: 54.16 scrub starts 2024-08-17T18:00:18.514 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:18.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:18.570 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:18.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:18.671 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:18.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:18 smithi136 ceph-mon[109861]: pgmap v2373: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 102 B/s rd, 8.0 MiB/s wr, 1.07k op/s; 5642/339813 objects degraded (1.660%); 6261/339813 objects misplaced (1.842%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T18:00:18.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:18 smithi136 ceph-mon[109861]: 54.16 scrub starts 2024-08-17T18:00:18.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:18 smithi136 ceph-mon[115078]: pgmap v2373: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 102 B/s rd, 8.0 MiB/s wr, 1.07k op/s; 5642/339813 objects degraded (1.660%); 6261/339813 objects misplaced (1.842%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T18:00:18.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:18 smithi136 ceph-mon[115078]: 54.16 scrub starts 2024-08-17T18:00:18.713 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:18.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:18.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:18.815 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:18.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:18.854 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:18.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:18.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:18.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:18.994 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:18.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:19.095 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:19.139 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:19.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:19.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:19.241 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:19.283 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:19.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:19.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:19.388 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:19.425 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:19.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:19.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:19.527 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:19.558 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:19 smithi136 ceph-mon[109861]: 54.6 scrub starts 2024-08-17T18:00:19.558 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:19 smithi136 ceph-mon[115078]: 54.6 scrub starts 2024-08-17T18:00:19.565 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:19.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:19.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:19.666 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:19.703 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:19.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:19.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:19 smithi165 ceph-mon[105517]: 54.6 scrub starts 2024-08-17T18:00:19.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:19.837 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:19.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:19.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:19.939 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:19.976 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:19.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:19.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:20.078 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:20.106 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:20.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:20.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:20.208 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:20.246 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:20.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:20.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:20.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:20.374 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:20.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:20.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:20.476 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:20.533 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:20.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:20.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:20.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:20.641 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:20 smithi136 ceph-mon[109861]: pgmap v2374: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.7 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 1.14k op/s; 5642/331269 objects degraded (1.703%); 6261/331269 objects misplaced (1.890%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T18:00:20.641 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:20 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5642/331269 objects degraded (1.703%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:20.641 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:20 smithi136 ceph-mon[115078]: pgmap v2374: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.7 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 1.14k op/s; 5642/331269 objects degraded (1.703%); 6261/331269 objects misplaced (1.890%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T18:00:20.641 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:20 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5642/331269 objects degraded (1.703%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:20.662 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:20.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:20.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:20.691 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:00:20.691 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T18:00:20.691 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-17T18:00:20.691 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-17T18:00:20.691 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:00:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:20 smithi165 ceph-mon[105517]: pgmap v2374: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.7 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 1.14k op/s; 5642/331269 objects degraded (1.703%); 6261/331269 objects misplaced (1.890%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T18:00:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:20 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5642/331269 objects degraded (1.703%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:20.764 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:20.801 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:20.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:20.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:20.903 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:20.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:20.947 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:20.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:21.047 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:21.074 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:21.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:21.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:21.176 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:21.215 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:21.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:21.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:21.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:21.347 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:21.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:21.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:21.391 INFO:teuthology.orchestra.run.smithi136.stderr:set pool 54 pgp_num to 26 2024-08-17T18:00:21.449 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:21.476 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:21.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:21.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:21.578 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:21.613 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:21.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:21.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:21 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/369155926' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "26"} : dispatch 2024-08-17T18:00:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:21 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "26"} : dispatch 2024-08-17T18:00:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:00:21.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:00:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:21 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/369155926' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "26"} : dispatch 2024-08-17T18:00:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:21 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "26"} : dispatch 2024-08-17T18:00:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:00:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:00:21.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:21.750 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:21.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:21.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:21 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/369155926' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "26"} : dispatch 2024-08-17T18:00:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:21 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "26"} : dispatch 2024-08-17T18:00:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:00:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:00:21.852 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:21.878 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:21.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:21.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:21.980 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:22.009 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:22.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:22.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:22.110 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:22.136 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:22.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:22.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:22.237 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:22.268 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:22.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:22.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:22.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-17T18:00:22.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:22.405 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:22.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:22.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:22.507 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:22.539 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:22.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:22.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:22.641 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:22.687 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:22.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:22.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:22.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:22 smithi136 ceph-mon[109861]: pgmap v2375: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.7 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 1.14k op/s; 5642/331269 objects degraded (1.703%); 6261/331269 objects misplaced (1.890%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T18:00:22.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:22 smithi136 ceph-mon[109861]: 54.16 deep-scrub starts 2024-08-17T18:00:22.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:22 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "26"}]': finished 2024-08-17T18:00:22.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:22 smithi136 ceph-mon[109861]: osdmap e1391: 8 total, 8 up, 7 in 2024-08-17T18:00:22.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:22 smithi136 ceph-mon[115078]: pgmap v2375: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.7 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 1.14k op/s; 5642/331269 objects degraded (1.703%); 6261/331269 objects misplaced (1.890%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T18:00:22.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:22 smithi136 ceph-mon[115078]: 54.16 deep-scrub starts 2024-08-17T18:00:22.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:22 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "26"}]': finished 2024-08-17T18:00:22.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:22 smithi136 ceph-mon[115078]: osdmap e1391: 8 total, 8 up, 7 in 2024-08-17T18:00:22.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:22 smithi165 ceph-mon[105517]: pgmap v2375: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.7 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 1.14k op/s; 5642/331269 objects degraded (1.703%); 6261/331269 objects misplaced (1.890%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T18:00:22.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:22 smithi165 ceph-mon[105517]: 54.16 deep-scrub starts 2024-08-17T18:00:22.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:22 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "26"}]': finished 2024-08-17T18:00:22.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:22 smithi165 ceph-mon[105517]: osdmap e1391: 8 total, 8 up, 7 in 2024-08-17T18:00:22.788 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:22.823 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:22.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:22.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:22.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:22.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:22.962 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:22.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:23.062 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:23.083 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:23.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:23.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:23.185 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:23.217 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:23.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:23.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:23.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:23.353 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:23.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:23.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:23.455 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:23.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:23.491 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:23.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:23.592 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:23.627 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:23.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:23.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:23.729 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:23.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:23.762 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:23.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:23.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:23.902 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:23.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:23.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:24.004 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:24.041 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:24.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:24.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:24.143 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:24.178 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:24.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:24.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:24.280 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:24.321 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:24.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:24.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:24.423 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:24.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:24.465 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:24.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:24.566 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:24.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:24.603 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:24.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:24.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:24 smithi136 ceph-mon[109861]: pgmap v2377: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+clean+scrubbing+deep, 95 active+clean; 6.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.13k op/s; 5334/314805 objects degraded (1.694%); 5706/314805 objects misplaced (1.813%); 6.7 MiB/s, 106 objects/s recovering 2024-08-17T18:00:24.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:24 smithi136 ceph-mon[115078]: pgmap v2377: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+clean+scrubbing+deep, 95 active+clean; 6.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.13k op/s; 5334/314805 objects degraded (1.694%); 5706/314805 objects misplaced (1.813%); 6.7 MiB/s, 106 objects/s recovering 2024-08-17T18:00:24.704 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:24.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:24.743 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:24.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:24.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:24 smithi165 ceph-mon[105517]: pgmap v2377: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+clean+scrubbing+deep, 95 active+clean; 6.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.13k op/s; 5334/314805 objects degraded (1.694%); 5706/314805 objects misplaced (1.813%); 6.7 MiB/s, 106 objects/s recovering 2024-08-17T18:00:24.844 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:24.883 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:24.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:24.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:24.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:25.017 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:25.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:25.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:25.119 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:25.156 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:25.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:25.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:25.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:25.293 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:25.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:25.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:25.394 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:25.431 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:25.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:25.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:25.532 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:25.556 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:25.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:25.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:25.658 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:25.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:25 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5334/314805 objects degraded (1.694%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:25.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:25 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5334/314805 objects degraded (1.694%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:25.698 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:25.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:25.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:25 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5334/314805 objects degraded (1.694%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:25.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:25.835 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:25.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:25.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:25.937 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:25.969 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:25.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:25.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:26.071 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:26.104 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:26.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:26.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:26.205 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:26.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:26.233 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:26.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:26.334 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:26.375 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:26.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:26.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:26.477 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:26.510 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:26.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:26.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:26.612 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:26.645 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:26.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:26.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:26.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:26 smithi136 ceph-mon[109861]: pgmap v2378: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+clean+scrubbing+deep, 95 active+clean; 6.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.13k op/s; 5334/314805 objects degraded (1.694%); 5706/314805 objects misplaced (1.813%); 6.7 MiB/s, 106 objects/s recovering 2024-08-17T18:00:26.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:26 smithi136 ceph-mon[115078]: pgmap v2378: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+clean+scrubbing+deep, 95 active+clean; 6.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.13k op/s; 5334/314805 objects degraded (1.694%); 5706/314805 objects misplaced (1.813%); 6.7 MiB/s, 106 objects/s recovering 2024-08-17T18:00:26.747 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:26.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:26 smithi165 ceph-mon[105517]: pgmap v2378: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+clean+scrubbing+deep, 95 active+clean; 6.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.13k op/s; 5334/314805 objects degraded (1.694%); 5706/314805 objects misplaced (1.813%); 6.7 MiB/s, 106 objects/s recovering 2024-08-17T18:00:26.768 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:26.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:26.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:26.871 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:26.902 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:26.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:26.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:26.930 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:00:27.004 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:27.058 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:27.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:27.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:27.160 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:27.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:27.198 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:27.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:27.299 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:27.334 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:27.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:27.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:27.436 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:27.479 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:27.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:27.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:27.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:27.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:27.602 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:27.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:27.702 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:27.739 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:27.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:27.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:27.783 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:00:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:00:27] "GET /metrics HTTP/1.1" 200 36570 "" "Prometheus/2.51.0" 2024-08-17T18:00:27.841 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:27.878 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:27.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:27.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:27.980 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:28.013 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:28.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:28.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:28.115 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:28.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:28.152 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:28.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:28.253 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:28.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:28.291 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:28.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:28.392 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:28.427 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:28.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:28.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:28.529 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:28.552 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:28.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:28.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:28.654 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:28.692 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:28.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:28.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:28.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:28 smithi136 ceph-mon[109861]: pgmap v2379: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.11k op/s; 4970/306585 objects degraded (1.621%); 5107/306585 objects misplaced (1.666%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T18:00:28.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:28 smithi136 ceph-mon[115078]: pgmap v2379: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.11k op/s; 4970/306585 objects degraded (1.621%); 5107/306585 objects misplaced (1.666%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T18:00:28.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:28 smithi165 ceph-mon[105517]: pgmap v2379: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.11k op/s; 4970/306585 objects degraded (1.621%); 5107/306585 objects misplaced (1.666%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T18:00:28.794 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:28.821 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:28.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:28.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:28.923 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:28.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:28.953 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:28.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:29.054 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:29.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:29.079 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:29.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:29.180 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:29.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:29.222 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:29.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:29.323 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:29.356 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:29.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:29.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:29.458 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:29.531 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:29.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:29.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:29.633 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:29.671 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:29.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:29.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:29.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:29 smithi136 ceph-mon[109861]: 54.6 scrub starts 2024-08-17T18:00:29.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:29 smithi136 ceph-mon[115078]: 54.6 scrub starts 2024-08-17T18:00:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:29 smithi165 ceph-mon[105517]: 54.6 scrub starts 2024-08-17T18:00:29.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:29.806 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:29.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:29.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:29.907 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:29.947 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:29.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:29.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:30.048 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:30.083 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:30.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:30.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:30.185 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:30.225 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:30.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:30.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:30.327 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:30.362 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:30.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:30.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:30.464 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:30.489 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:30.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:30.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:30.591 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:30.627 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:30.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:30.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:30.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:30 smithi136 ceph-mon[109861]: pgmap v2380: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.09k op/s; 4970/298620 objects degraded (1.664%); 5107/298620 objects misplaced (1.710%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T18:00:30.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:30 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 4970/298620 objects degraded (1.664%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:30.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:30 smithi136 ceph-mon[115078]: pgmap v2380: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.09k op/s; 4970/298620 objects degraded (1.664%); 5107/298620 objects misplaced (1.710%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T18:00:30.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:30 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 4970/298620 objects degraded (1.664%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:30.729 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:30.750 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:30.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:30.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:30.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:30 smithi165 ceph-mon[105517]: pgmap v2380: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.09k op/s; 4970/298620 objects degraded (1.664%); 5107/298620 objects misplaced (1.710%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T18:00:30.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:30 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 4970/298620 objects degraded (1.664%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:30.852 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:30.888 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:30.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:30.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:30.990 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:31.027 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:31.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:31.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:31.129 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:31.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:31.151 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:31.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:31.252 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:31.274 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:31.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:31.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:31.375 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:31.410 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:31.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:31.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:31.512 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:31.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:31.539 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:31.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:31.640 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:31.679 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:31.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:31.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:31.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:31 smithi136 ceph-mon[109861]: 54.16 scrub starts 2024-08-17T18:00:31.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:31 smithi136 ceph-mon[115078]: 54.16 scrub starts 2024-08-17T18:00:31.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:31 smithi165 ceph-mon[105517]: 54.16 scrub starts 2024-08-17T18:00:31.781 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:31.811 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:31.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:31.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:31.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:31.953 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:31.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:31.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:32.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:32.091 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:32.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:32.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:32.192 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:32.233 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:32.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:32.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:32.334 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:32.368 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:32.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:32.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:32.470 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:32.493 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:32.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:32.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:32.595 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:32.625 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:32.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:32.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:32.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:32 smithi136 ceph-mon[109861]: pgmap v2381: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.09k op/s; 4970/298620 objects degraded (1.664%); 5107/298620 objects misplaced (1.710%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T18:00:32.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:32 smithi136 ceph-mon[109861]: 54.6 scrub starts 2024-08-17T18:00:32.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:32 smithi136 ceph-mon[115078]: pgmap v2381: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.09k op/s; 4970/298620 objects degraded (1.664%); 5107/298620 objects misplaced (1.710%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T18:00:32.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:32 smithi136 ceph-mon[115078]: 54.6 scrub starts 2024-08-17T18:00:32.727 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:32.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:32 smithi165 ceph-mon[105517]: pgmap v2381: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.09k op/s; 4970/298620 objects degraded (1.664%); 5107/298620 objects misplaced (1.710%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T18:00:32.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:32 smithi165 ceph-mon[105517]: 54.6 scrub starts 2024-08-17T18:00:32.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:32.763 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:32.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:32.864 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:32.899 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:32.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:32.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:33.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:33.041 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:33.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:33.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:33.143 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:33.181 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:33.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:33.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:33.282 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:33.318 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:33.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:33.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:33.420 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:33.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:33.455 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:33.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:33.556 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:33.589 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:33.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:33.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:33.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:33.726 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:33.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:33.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:33.828 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:33.850 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:33.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:33.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:33.952 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:33.993 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:33.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:33.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:34.095 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:34.116 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:34.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:34.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:34.218 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:34.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:34.253 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:34.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:34.354 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:34.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:34.391 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:34.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:34.492 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:34.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:34.531 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:34.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:34.632 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:34.672 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:34.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:34.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:34.693 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:00:34.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:34 smithi136 ceph-mon[109861]: pgmap v2382: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.24k op/s; 4674/283350 objects degraded (1.650%); 4494/283350 objects misplaced (1.586%); 7.4 MiB/s, 118 objects/s recovering 2024-08-17T18:00:34.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:34 smithi136 ceph-mon[109861]: 54.16 scrub starts 2024-08-17T18:00:34.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:34 smithi136 ceph-mon[115078]: pgmap v2382: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.24k op/s; 4674/283350 objects degraded (1.650%); 4494/283350 objects misplaced (1.586%); 7.4 MiB/s, 118 objects/s recovering 2024-08-17T18:00:34.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:34 smithi136 ceph-mon[115078]: 54.16 scrub starts 2024-08-17T18:00:34.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:34 smithi165 ceph-mon[105517]: pgmap v2382: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.24k op/s; 4674/283350 objects degraded (1.650%); 4494/283350 objects misplaced (1.586%); 7.4 MiB/s, 118 objects/s recovering 2024-08-17T18:00:34.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:34 smithi165 ceph-mon[105517]: 54.16 scrub starts 2024-08-17T18:00:34.774 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:34.804 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:34.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:34.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:34.906 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:34.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:34.932 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:34.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:35.033 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:35.069 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:35.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:35.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:35.170 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:35.211 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:35.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:35.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:35.313 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:35.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:35.348 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:35.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:35.449 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:35.479 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:35.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:35.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:35.581 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:35.617 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:35.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:35.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:35.719 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:35.758 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:35.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:35.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:35 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 4674/283350 objects degraded (1.650%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:35.859 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:35.861 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:35 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 4674/283350 objects degraded (1.650%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:35.861 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:35 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 4674/283350 objects degraded (1.650%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:35.899 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:35.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:35.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:36.001 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:36.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:36.033 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:36.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:36.134 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:36.168 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:36.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:36.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:36.270 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:36.296 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:36.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:36.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:36.398 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:36.433 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:36.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:36.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:36.534 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:36.566 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:36.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:36.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:36.668 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:36.704 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:36.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:36.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:36 smithi165 ceph-mon[105517]: pgmap v2383: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 873 op/s; 4674/283350 objects degraded (1.650%); 4494/283350 objects misplaced (1.586%); 6.1 MiB/s, 96 objects/s recovering 2024-08-17T18:00:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:00:36.806 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:36.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:36.833 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:36.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:36.934 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:00:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:36 smithi136 ceph-mon[109861]: pgmap v2383: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 873 op/s; 4674/283350 objects degraded (1.650%); 4494/283350 objects misplaced (1.586%); 6.1 MiB/s, 96 objects/s recovering 2024-08-17T18:00:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:00:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:36 smithi136 ceph-mon[115078]: pgmap v2383: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 873 op/s; 4674/283350 objects degraded (1.650%); 4494/283350 objects misplaced (1.586%); 6.1 MiB/s, 96 objects/s recovering 2024-08-17T18:00:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:00:36.971 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:36.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:36.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:00:37.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:37.119 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:37.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:37.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:37.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:37.252 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:37.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:37.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:37.311 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-17T18:00:37.311 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:00:37.312 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-17T18:00:37.312 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-08-17T18:00:37.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.2 2024-08-17T18:00:37.354 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:37.388 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:37.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:37.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:37.490 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:37.531 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:37.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:37.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:37.633 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:37.655 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:37.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:37.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:37.696 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:00:37 smithi136 systemd[1]: Stopping Ceph osd.2 for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T18:00:37.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:00:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:00:37] "GET /metrics HTTP/1.1" 200 36567 "" "Prometheus/2.51.0" 2024-08-17T18:00:37.756 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:37.795 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:37.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:37.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:37.897 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:37.941 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:37.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:37.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:38.043 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:38.082 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:38.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:38.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:38.184 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:38.221 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:38.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:38.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:38.322 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:38.356 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:38.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:38.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:38.458 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:38.483 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:38.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:38.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:38.585 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:38.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:38.620 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:38.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:38.721 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:38.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:38.755 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:38.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:38.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:38 smithi165 ceph-mon[105517]: pgmap v2384: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s; 4404/275877 objects degraded (1.596%); 3822/275877 objects misplaced (1.385%); 9.2 MiB/s, 146 objects/s recovering 2024-08-17T18:00:38.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:38 smithi165 ceph-mon[105517]: 54.6 scrub starts 2024-08-17T18:00:38.856 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:38.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:38.878 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:38.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:38.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:38 smithi136 ceph-mon[109861]: pgmap v2384: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s; 4404/275877 objects degraded (1.596%); 3822/275877 objects misplaced (1.385%); 9.2 MiB/s, 146 objects/s recovering 2024-08-17T18:00:38.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:38 smithi136 ceph-mon[109861]: 54.6 scrub starts 2024-08-17T18:00:38.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:38 smithi136 ceph-mon[115078]: pgmap v2384: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s; 4404/275877 objects degraded (1.596%); 3822/275877 objects misplaced (1.385%); 9.2 MiB/s, 146 objects/s recovering 2024-08-17T18:00:38.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:38 smithi136 ceph-mon[115078]: 54.6 scrub starts 2024-08-17T18:00:38.979 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:39.014 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:39.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:39.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:39.116 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:39.158 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:39.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:39.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:39.260 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:39.289 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:39.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:39.391 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:39.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:39.425 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:39.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:39.526 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:39.571 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:39.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:39.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:39.673 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:39.693 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:39.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:39.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:39 smithi165 ceph-mon[105517]: 54.16 scrub starts 2024-08-17T18:00:39.796 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:39.837 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:39.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:39.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:39.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:39.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:39 smithi136 ceph-mon[109861]: 54.16 scrub starts 2024-08-17T18:00:39.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:39 smithi136 ceph-mon[115078]: 54.16 scrub starts 2024-08-17T18:00:39.972 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:39.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:39.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:40.074 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:40.111 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:40.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:40.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:40.213 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:40.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:40.248 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:40.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:40.349 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:40.370 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:40.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:40.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:40.472 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:40.504 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:40.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:40.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:40.606 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:40.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:40.648 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:40.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:40.749 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:40 smithi165 ceph-mon[105517]: pgmap v2385: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s; 4404/268290 objects degraded (1.642%); 3822/268290 objects misplaced (1.425%); 6.1 MiB/s, 97 objects/s recovering 2024-08-17T18:00:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:40 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 4404/268290 objects degraded (1.642%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:40.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:40 smithi165 ceph-mon[105517]: 54.6 scrub starts 2024-08-17T18:00:40.775 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:40.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:40.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:40.877 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:40.911 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:40.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:40.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:40.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:40 smithi136 ceph-mon[109861]: pgmap v2385: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s; 4404/268290 objects degraded (1.642%); 3822/268290 objects misplaced (1.425%); 6.1 MiB/s, 97 objects/s recovering 2024-08-17T18:00:40.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:40 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 4404/268290 objects degraded (1.642%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:40.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:40 smithi136 ceph-mon[109861]: 54.6 scrub starts 2024-08-17T18:00:40.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:40 smithi136 ceph-mon[115078]: pgmap v2385: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s; 4404/268290 objects degraded (1.642%); 3822/268290 objects misplaced (1.425%); 6.1 MiB/s, 97 objects/s recovering 2024-08-17T18:00:40.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:40 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 4404/268290 objects degraded (1.642%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:40.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:40 smithi136 ceph-mon[115078]: 54.6 scrub starts 2024-08-17T18:00:41.013 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:41.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:41.059 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:41.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:41.160 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:41.197 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:41.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:41.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:41.299 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:41.340 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:41.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:41.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:41.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:41.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:41.481 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:41.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:41.582 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:41.615 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:41.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:41.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:41.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:41.750 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:41.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:41.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:41.852 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:41.879 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:41.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:41.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:41.981 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:42.017 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:42.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:42.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:42.119 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:42.160 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:42.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:42.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:42.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:42.295 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:42.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:42.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:42.398 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:42.439 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:42.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:42.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:42.482 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:00:42.542 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:42.583 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:42.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:42.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:42.685 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:42.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:42 smithi136 ceph-mon[109861]: pgmap v2386: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 842 op/s; 4404/268290 objects degraded (1.642%); 3822/268290 objects misplaced (1.425%); 6.1 MiB/s, 97 objects/s recovering 2024-08-17T18:00:42.698 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:42 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1088403084' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:00:42.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:42 smithi136 ceph-mon[115078]: pgmap v2386: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 842 op/s; 4404/268290 objects degraded (1.642%); 3822/268290 objects misplaced (1.425%); 6.1 MiB/s, 97 objects/s recovering 2024-08-17T18:00:42.699 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:42 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1088403084' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:00:42.718 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:42.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:42.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:42.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:42 smithi165 ceph-mon[105517]: pgmap v2386: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 842 op/s; 4404/268290 objects degraded (1.642%); 3822/268290 objects misplaced (1.425%); 6.1 MiB/s, 97 objects/s recovering 2024-08-17T18:00:42.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:42 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1088403084' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:00:42.820 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:42.870 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:42.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:42.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:42.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:43.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:43.015 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:43.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:43.116 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:43.158 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:43.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:43.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:43.259 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:43.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:43.297 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:43.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:43.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:43.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:43.437 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:43.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:43.538 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:43.568 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:43.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:43.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:43.667 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:43 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:00:43.667 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:43 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1088403084' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:00:43.667 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:43 smithi136 ceph-mon[109861]: osdmap e1392: 8 total, 8 up, 7 in 2024-08-17T18:00:43.667 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:43 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:00:43.667 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:43 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1088403084' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:00:43.667 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:43 smithi136 ceph-mon[115078]: osdmap e1392: 8 total, 8 up, 7 in 2024-08-17T18:00:43.670 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:43.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:43.708 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:43.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:43 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:00:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:43 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1088403084' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:00:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:43 smithi165 ceph-mon[105517]: osdmap e1392: 8 total, 8 up, 7 in 2024-08-17T18:00:43.809 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:43.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:43.847 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:43.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:43.847 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:00:43.848 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:00:43.881 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:00:43.882 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:00:43.921 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:00:43.921 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:00:43.947 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:43.958 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:00:43.958 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:00:44.001 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:44.002 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:00:44.002 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:00:44.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:44.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:44.037 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:00:44.037 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:00:44.078 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:00:44.079 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:00:44.089 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:00:44.089 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:00:44.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:44.115 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:00:44.116 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:00:44.138 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:44.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:44.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:44.196 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:00:43 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2[142367]: 2024-08-17T18:00:43.983+0000 7f20280b3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T18:00:44.196 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:00:43 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2[142367]: 2024-08-17T18:00:43.983+0000 7f20280b3640 -1 osd.2 1392 *** Got signal Terminated *** 2024-08-17T18:00:44.196 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:00:43 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2[142367]: 2024-08-17T18:00:43.983+0000 7f20280b3640 -1 osd.2 1392 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T18:00:44.241 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:44.275 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:44.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:44.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:44.377 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:44.418 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:44.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:44.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:44.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:44.560 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:44.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:44.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:44.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:44.683 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:44.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:44.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:44.785 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:44.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:44.807 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:44.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:44.871 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:44 smithi136 ceph-mon[109861]: pgmap v2388: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 966 op/s; 4128/254340 objects degraded (1.623%); 3036/254340 objects misplaced (1.194%); 8.1 MiB/s, 129 objects/s recovering 2024-08-17T18:00:44.871 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:44 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/581015461' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:00:44.871 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:44 smithi136 ceph-mon[109861]: osd.2 marked itself down and dead 2024-08-17T18:00:44.871 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:44 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 4128/254340 objects degraded (1.623%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:44.871 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:44 smithi136 ceph-mon[115078]: pgmap v2388: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 966 op/s; 4128/254340 objects degraded (1.623%); 3036/254340 objects misplaced (1.194%); 8.1 MiB/s, 129 objects/s recovering 2024-08-17T18:00:44.872 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:44 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/581015461' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:00:44.872 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:44 smithi136 ceph-mon[115078]: osd.2 marked itself down and dead 2024-08-17T18:00:44.872 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:44 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 4128/254340 objects degraded (1.623%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:44.894 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-17T18:00:44.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:44.930 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:44.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:44.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:44 smithi165 ceph-mon[105517]: pgmap v2388: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 96 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 966 op/s; 4128/254340 objects degraded (1.623%); 3036/254340 objects misplaced (1.194%); 8.1 MiB/s, 129 objects/s recovering 2024-08-17T18:00:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:44 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/581015461' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:00:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:44 smithi165 ceph-mon[105517]: osd.2 marked itself down and dead 2024-08-17T18:00:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:44 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 4128/254340 objects degraded (1.623%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:00:45.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:45.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:45.058 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:45.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:45.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:45.184 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:45.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:45.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:45.286 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:45.312 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:45.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:45.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:45.414 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:45.440 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:45.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:45.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:45.541 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:45.574 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:45.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:45.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:45.671 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:45 smithi136 ceph-mon[109861]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T18:00:45.671 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:45 smithi136 ceph-mon[109861]: osdmap e1393: 8 total, 7 up, 7 in 2024-08-17T18:00:45.671 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:45 smithi136 ceph-mon[115078]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T18:00:45.672 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:45 smithi136 ceph-mon[115078]: osdmap e1393: 8 total, 7 up, 7 in 2024-08-17T18:00:45.672 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:00:45 smithi136 podman[287921]: 2024-08-17 18:00:45.355929435 +0000 UTC m=+1.664492925 container died 933eab0a702a51e3fa99de6e373a49e883425ed23ba22ee45d4aace6e1a3eda9 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD) 2024-08-17T18:00:45.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:45.699 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:45.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:45.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:45.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:45.823 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:45.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:45.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:45.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:45.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:45.949 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:45.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:46.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:45 smithi165 ceph-mon[105517]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T18:00:46.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:45 smithi165 ceph-mon[105517]: osdmap e1393: 8 total, 7 up, 7 in 2024-08-17T18:00:46.050 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:46.074 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:46.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:46.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:46.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:46.198 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:46.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:46.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:46.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:46.321 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:46.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:46.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:46.423 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:46.446 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:46.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:46.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:46.548 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:46.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:46.570 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:46.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:46.671 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:46.694 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:46.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:46.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:46.796 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:46.817 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:46.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:46.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:46.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:46.942 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:46.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:46.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:46.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:46 smithi136 ceph-mon[109861]: pgmap v2390: 99 pgs: 4 stale+active+clean, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 92 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 897 op/s; 4128/254340 objects degraded (1.623%); 3036/254340 objects misplaced (1.194%); 5.4 MiB/s, 86 objects/s recovering 2024-08-17T18:00:46.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:46 smithi136 ceph-mon[109861]: 54.17 scrub starts 2024-08-17T18:00:46.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:46 smithi136 ceph-mon[109861]: osdmap e1394: 8 total, 7 up, 7 in 2024-08-17T18:00:46.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:46 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:00:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:46 smithi136 ceph-mon[115078]: pgmap v2390: 99 pgs: 4 stale+active+clean, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 92 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 897 op/s; 4128/254340 objects degraded (1.623%); 3036/254340 objects misplaced (1.194%); 5.4 MiB/s, 86 objects/s recovering 2024-08-17T18:00:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:46 smithi136 ceph-mon[115078]: 54.17 scrub starts 2024-08-17T18:00:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:46 smithi136 ceph-mon[115078]: osdmap e1394: 8 total, 7 up, 7 in 2024-08-17T18:00:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:46 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:00:47.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:46 smithi165 ceph-mon[105517]: pgmap v2390: 99 pgs: 4 stale+active+clean, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 92 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 897 op/s; 4128/254340 objects degraded (1.623%); 3036/254340 objects misplaced (1.194%); 5.4 MiB/s, 86 objects/s recovering 2024-08-17T18:00:47.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:46 smithi165 ceph-mon[105517]: 54.17 scrub starts 2024-08-17T18:00:47.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:46 smithi165 ceph-mon[105517]: osdmap e1394: 8 total, 7 up, 7 in 2024-08-17T18:00:47.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:46 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:00:47.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:47.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:47.065 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:47.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:47.166 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:47.192 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:47.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:47.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:47.294 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:47.319 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:47.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:47.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:47.364 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:00:47.387 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:00:47 smithi136 podman[287921]: 2024-08-17 18:00:47.122139586 +0000 UTC m=+3.430703060 container cleanup 933eab0a702a51e3fa99de6e373a49e883425ed23ba22ee45d4aace6e1a3eda9 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=main-cd8bdbe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0) 2024-08-17T18:00:47.387 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:00:47 smithi136 bash[287921]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2 2024-08-17T18:00:47.421 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:47.444 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:47.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:47.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:47.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:47.578 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:47.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:47.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:47.652 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:47 smithi136 ceph-mon[109861]: 54.19 scrub starts 2024-08-17T18:00:47.652 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:47 smithi136 ceph-mon[109861]: 54.7 scrub starts 2024-08-17T18:00:47.652 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:47 smithi136 ceph-mon[109861]: osdmap e1395: 8 total, 7 up, 7 in 2024-08-17T18:00:47.652 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:00:47 smithi136 podman[288082]: 2024-08-17 18:00:47.448897397 +0000 UTC m=+2.318568514 container remove 933eab0a702a51e3fa99de6e373a49e883425ed23ba22ee45d4aace6e1a3eda9 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-08-17T18:00:47.652 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:47 smithi136 ceph-mon[115078]: 54.19 scrub starts 2024-08-17T18:00:47.652 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:47 smithi136 ceph-mon[115078]: 54.7 scrub starts 2024-08-17T18:00:47.653 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:47 smithi136 ceph-mon[115078]: osdmap e1395: 8 total, 7 up, 7 in 2024-08-17T18:00:47.653 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:00:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:00:47] "GET /metrics HTTP/1.1" 200 36567 "" "Prometheus/2.51.0" 2024-08-17T18:00:47.681 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:47.708 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:47.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:47.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:47 smithi165 ceph-mon[105517]: 54.19 scrub starts 2024-08-17T18:00:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:47 smithi165 ceph-mon[105517]: 54.7 scrub starts 2024-08-17T18:00:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:47 smithi165 ceph-mon[105517]: osdmap e1395: 8 total, 7 up, 7 in 2024-08-17T18:00:47.810 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:47.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:47.833 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:47.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:47.934 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:47.955 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:47.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:47.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:48.057 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:48.058 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:00:48.085 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:48.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:48.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:48.108 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:00:48.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:48.209 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:48.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:48.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:48.310 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:48.331 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:48.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:48.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:48.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:48.454 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:48.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:48.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:48.556 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:48.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:48.579 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:48.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:48.637 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:48 smithi136 ceph-mon[109861]: pgmap v2393: 99 pgs: 12 active+undersized, 2 stale+active+clean, 1 active+recovering+undersized+remapped, 1 creating+peering, 2 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 376 op/s; 4128/249534 objects degraded (1.654%); 2472/249534 objects misplaced (0.991%); 8.3 MiB/s, 132 objects/s recovering 2024-08-17T18:00:48.637 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:48 smithi136 ceph-mon[109861]: 54.14 scrub starts 2024-08-17T18:00:48.637 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:48 smithi136 ceph-mon[115078]: pgmap v2393: 99 pgs: 12 active+undersized, 2 stale+active+clean, 1 active+recovering+undersized+remapped, 1 creating+peering, 2 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 376 op/s; 4128/249534 objects degraded (1.654%); 2472/249534 objects misplaced (0.991%); 8.3 MiB/s, 132 objects/s recovering 2024-08-17T18:00:48.637 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:48 smithi136 ceph-mon[115078]: 54.14 scrub starts 2024-08-17T18:00:48.680 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:48.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:48.706 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:48.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:48.807 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:48.828 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:48.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:48.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:48.929 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:48.951 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:48.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:48.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:48 smithi165 ceph-mon[105517]: pgmap v2393: 99 pgs: 12 active+undersized, 2 stale+active+clean, 1 active+recovering+undersized+remapped, 1 creating+peering, 2 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 376 op/s; 4128/249534 objects degraded (1.654%); 2472/249534 objects misplaced (0.991%); 8.3 MiB/s, 132 objects/s recovering 2024-08-17T18:00:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:48 smithi165 ceph-mon[105517]: 54.14 scrub starts 2024-08-17T18:00:49.053 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:49.075 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:49.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:49.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:49.177 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:49.198 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:49.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:49.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:49.299 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:49.320 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:49.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:49.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:49.421 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:49.444 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:49.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:49.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:49.546 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:49.571 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:49.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:49.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:49.673 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:49.694 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:49.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:49.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:49.795 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:49.817 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:49.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:49.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:49.919 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:49.928 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:49 smithi165 ceph-mon[105517]: 54.13 scrub starts 2024-08-17T18:00:49.928 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:49 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 4128/249534 objects degraded (1.654%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:00:49.944 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:49.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:49.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:49.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:49 smithi136 ceph-mon[109861]: 54.13 scrub starts 2024-08-17T18:00:49.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:49 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 4128/249534 objects degraded (1.654%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:00:49.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:49 smithi136 ceph-mon[115078]: 54.13 scrub starts 2024-08-17T18:00:49.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:49 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 4128/249534 objects degraded (1.654%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:00:50.010 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:00:50.046 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:50.068 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:50.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:50.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:50.170 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:50.191 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:50.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:50.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:50.293 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:50.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:50.314 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:50.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:50.415 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:50.437 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:50.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:50.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:50.538 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:50.559 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:50.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:50.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:50.661 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:50.682 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:50.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:50.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:50.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:50 smithi136 ceph-mon[109861]: pgmap v2394: 99 pgs: 1 active+recovery_wait+degraded+remapped, 18 active+undersized, 1 stale+active+clean, 1 creating+peering, 2 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+undersized+degraded, 71 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 467 op/s; 16542/245916 objects degraded (6.727%); 2593/245916 objects misplaced (1.054%); 7.4 MiB/s, 117 objects/s recovering 2024-08-17T18:00:50.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:50 smithi136 ceph-mon[115078]: pgmap v2394: 99 pgs: 1 active+recovery_wait+degraded+remapped, 18 active+undersized, 1 stale+active+clean, 1 creating+peering, 2 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+undersized+degraded, 71 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 467 op/s; 16542/245916 objects degraded (6.727%); 2593/245916 objects misplaced (1.054%); 7.4 MiB/s, 117 objects/s recovering 2024-08-17T18:00:50.697 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:00:50 smithi136 podman[288606]: 2024-08-17 18:00:50.281960195 +0000 UTC m=+1.651952408 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T18:00:50.697 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:00:50 smithi136 podman[288606]: 2024-08-17 18:00:50.412442442 +0000 UTC m=+1.782434657 container create 2369ac8f713dcce33ceac1bbeb92f53a376ffec5d986da5d5ebb89f05f220e45 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=main-cd8bdbe, org.label-schema.vendor=CentOS) 2024-08-17T18:00:50.785 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:50.807 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:50.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:50.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:50.909 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:50.918 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:50 smithi165 ceph-mon[105517]: pgmap v2394: 99 pgs: 1 active+recovery_wait+degraded+remapped, 18 active+undersized, 1 stale+active+clean, 1 creating+peering, 2 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+undersized+degraded, 71 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 467 op/s; 16542/245916 objects degraded (6.727%); 2593/245916 objects misplaced (1.054%); 7.4 MiB/s, 117 objects/s recovering 2024-08-17T18:00:50.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:50.932 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:50.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:51.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:51.053 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:51.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:51.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:51.155 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:51.177 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:51.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:51.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:51.278 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:51.301 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:51.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:51.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:51.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:51.427 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:51.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:51.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:51.529 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:51.551 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:51.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:51.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:51.653 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:51.674 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:51.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:51.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:51.776 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:51.799 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:51.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:51.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:51.864 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:00:51 smithi136 podman[288606]: 2024-08-17 18:00:51.697077544 +0000 UTC m=+3.067069771 container init 2369ac8f713dcce33ceac1bbeb92f53a376ffec5d986da5d5ebb89f05f220e45 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-cd8bdbe, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-08-17T18:00:51.864 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:00:51 smithi136 podman[288606]: 2024-08-17 18:00:51.703389148 +0000 UTC m=+3.073381362 container start 2369ac8f713dcce33ceac1bbeb92f53a376ffec5d986da5d5ebb89f05f220e45 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-08-17T18:00:51.865 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:00:51 smithi136 podman[288606]: 2024-08-17 18:00:51.864104137 +0000 UTC m=+3.234096365 container attach 2369ac8f713dcce33ceac1bbeb92f53a376ffec5d986da5d5ebb89f05f220e45 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-08-17T18:00:51.900 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:51.925 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:51.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:51.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:52.027 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:52.050 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:52.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:52.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:52.094 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:00:52.152 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:52.174 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:52.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:52.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:52.276 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:52.304 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:52.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:52.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:52.405 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:52.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:52.432 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:52.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:52.486 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:52 smithi136 ceph-mon[115078]: pgmap v2395: 99 pgs: 1 activating+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 21 active+undersized, 1 creating+peering, 2 active+clean+scrubbing, 9 active+undersized+degraded, 64 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 514 op/s; 23873/244716 objects degraded (9.755%); 3327/244716 objects misplaced (1.360%); 8.4 MiB/s, 134 objects/s recovering 2024-08-17T18:00:52.487 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:52 smithi136 ceph-mon[115078]: 54.3 scrub starts 2024-08-17T18:00:52.487 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:00:52.487 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:00:52.487 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:00:52.487 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:00:52.487 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:52 smithi136 ceph-mon[109861]: pgmap v2395: 99 pgs: 1 activating+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 21 active+undersized, 1 creating+peering, 2 active+clean+scrubbing, 9 active+undersized+degraded, 64 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 514 op/s; 23873/244716 objects degraded (9.755%); 3327/244716 objects misplaced (1.360%); 8.4 MiB/s, 134 objects/s recovering 2024-08-17T18:00:52.487 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:52 smithi136 ceph-mon[109861]: 54.3 scrub starts 2024-08-17T18:00:52.487 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:00:52.487 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:00:52.488 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:00:52.488 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:00:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:52 smithi165 ceph-mon[105517]: pgmap v2395: 99 pgs: 1 activating+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 21 active+undersized, 1 creating+peering, 2 active+clean+scrubbing, 9 active+undersized+degraded, 64 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 514 op/s; 23873/244716 objects degraded (9.755%); 3327/244716 objects misplaced (1.360%); 8.4 MiB/s, 134 objects/s recovering 2024-08-17T18:00:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:52 smithi165 ceph-mon[105517]: 54.3 scrub starts 2024-08-17T18:00:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:00:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:00:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:00:52.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:00:52.533 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:52.560 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:52.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:52.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:52.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:52.692 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:52.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:52.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:52.794 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:52.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:52.831 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:52.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:52.932 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:52.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:52.959 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:52.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:53.061 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:53.087 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:53.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:53.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:53.189 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:53.217 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:53.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:53.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:53.318 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:53.343 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:53.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:53.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:53.445 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:53.476 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:53.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:53.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:53.579 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:53.602 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:53.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:53.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:53.704 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:53.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:53.731 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:53.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:53.831 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:00:53 smithi136 podman[288606]: 2024-08-17 18:00:53.519573873 +0000 UTC m=+4.889566095 container died 2369ac8f713dcce33ceac1bbeb92f53a376ffec5d986da5d5ebb89f05f220e45 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0) 2024-08-17T18:00:53.833 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:53.866 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:53.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:53.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:53.968 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:53.993 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:53.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:53.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:54.095 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:54.127 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:54.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:54.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:54.229 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:54.239 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.2.service' 2024-08-17T18:00:54.253 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:54.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:54.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:54.316 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T18:00:54.317 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-08-17T18:00:54.355 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:54.383 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:54.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:54.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:54.390 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:00:54.390 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:00:54.485 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:54.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:54 smithi165 ceph-mon[105517]: pgmap v2396: 99 pgs: 1 activating+degraded+remapped, 21 active+undersized, 1 active+recovering+degraded+remapped, 5 active+clean+scrubbing, 9 active+undersized+degraded, 62 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 424 op/s; 24252/244143 objects degraded (9.934%); 2385/244143 objects misplaced (0.977%); 8.6 MiB/s, 137 objects/s recovering 2024-08-17T18:00:54.511 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:54.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:54.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:54.613 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:54.635 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:54.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:54.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:54.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:54 smithi136 ceph-mon[109861]: pgmap v2396: 99 pgs: 1 activating+degraded+remapped, 21 active+undersized, 1 active+recovering+degraded+remapped, 5 active+clean+scrubbing, 9 active+undersized+degraded, 62 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 424 op/s; 24252/244143 objects degraded (9.934%); 2385/244143 objects misplaced (0.977%); 8.6 MiB/s, 137 objects/s recovering 2024-08-17T18:00:54.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:54 smithi136 ceph-mon[115078]: pgmap v2396: 99 pgs: 1 activating+degraded+remapped, 21 active+undersized, 1 active+recovering+degraded+remapped, 5 active+clean+scrubbing, 9 active+undersized+degraded, 62 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 424 op/s; 24252/244143 objects degraded (9.934%); 2385/244143 objects misplaced (0.977%); 8.6 MiB/s, 137 objects/s recovering 2024-08-17T18:00:54.737 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:54.762 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:54.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:54.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:54.864 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:54.886 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:54.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:54.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:54.988 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:55.010 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:55.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:55.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:55.112 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:55.134 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:55.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:55.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:55.152 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-17T18:00:55.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:55.261 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:55.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:55.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:55.363 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:55.385 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:55.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:55.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:55.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:55 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 24252/244143 objects degraded (9.934%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T18:00:55.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:55 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 24252/244143 objects degraded (9.934%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T18:00:55.487 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:55 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 24252/244143 objects degraded (9.934%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T18:00:55.517 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:55.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:55.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:55.618 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:55.643 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:55.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:55.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:55.746 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:55.772 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:55.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:55.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:55.874 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:55.896 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:55.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:55.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:55.997 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:56.019 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:56.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:56.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:56.121 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:56.143 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:56.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:56.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:56.245 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:56.272 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:56.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:56.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:56.374 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:56.402 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:56.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:56.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:56.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:56.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:56 smithi165 ceph-mon[105517]: pgmap v2397: 99 pgs: 1 activating+degraded+remapped, 21 active+undersized, 1 active+recovering+degraded+remapped, 5 active+clean+scrubbing, 9 active+undersized+degraded, 62 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 368 op/s; 24252/244143 objects degraded (9.934%); 2385/244143 objects misplaced (0.977%); 7.5 MiB/s, 119 objects/s recovering 2024-08-17T18:00:56.518 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:56 smithi136 ceph-mon[109861]: pgmap v2397: 99 pgs: 1 activating+degraded+remapped, 21 active+undersized, 1 active+recovering+degraded+remapped, 5 active+clean+scrubbing, 9 active+undersized+degraded, 62 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 368 op/s; 24252/244143 objects degraded (9.934%); 2385/244143 objects misplaced (0.977%); 7.5 MiB/s, 119 objects/s recovering 2024-08-17T18:00:56.518 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:56 smithi136 ceph-mon[115078]: pgmap v2397: 99 pgs: 1 activating+degraded+remapped, 21 active+undersized, 1 active+recovering+degraded+remapped, 5 active+clean+scrubbing, 9 active+undersized+degraded, 62 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 368 op/s; 24252/244143 objects degraded (9.934%); 2385/244143 objects misplaced (0.977%); 7.5 MiB/s, 119 objects/s recovering 2024-08-17T18:00:56.519 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:00:56.519 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:00:56.528 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:56.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:56.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:56.538 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:00:56.538 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:00:56.538 INFO:teuthology.orchestra.run.smithi136.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-17T18:00:56.557 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:00:56.557 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:00:56.575 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:00:56.575 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:00:56.593 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:00:56.594 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:00:56.612 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:00:56.613 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:00:56.629 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:56.630 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:00:56.630 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:00:56.657 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:56.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:56.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:56.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:56.785 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:56.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:56.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:56.887 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:56.917 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:56.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:56.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:57.019 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:57.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:57.044 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:57.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:57.145 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:57.162 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:00:57.173 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:57.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:57.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:57.228 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:00:57.275 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:57.301 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:57.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:57.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:57.403 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:57.426 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:57.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:57.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:57.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:57 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/83928559' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:00:57.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:57 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3408375645' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:00:57.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:57 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:00:57.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:57 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/83928559' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:00:57.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:57 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3408375645' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:00:57.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:57 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:00:57.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:57 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/83928559' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:00:57.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:57 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3408375645' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:00:57.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:57 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:00:57.529 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:57.553 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:57.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:57.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:57.655 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:57.678 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:57.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:57.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:57.702 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:00:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:00:57] "GET /metrics HTTP/1.1" 200 36571 "" "Prometheus/2.51.0" 2024-08-17T18:00:57.730 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-17T18:00:57.730 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-17T18:00:57.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:57.804 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:57.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:57.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:57.906 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:57.930 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:57.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:57.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:58.032 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:00:58.054 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:58.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:58.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:00:58.156 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:58.178 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:58.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:58.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:58.219 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:58 smithi136 ceph-mon[115078]: pgmap v2398: 99 pgs: 1 active+recovery_wait+degraded+remapped, 21 active+undersized, 1 active+recovering+degraded+remapped, 9 active+undersized+degraded, 67 active+clean; 4.9 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 283 op/s; 24029/240918 objects degraded (9.974%); 2342/240918 objects misplaced (0.972%); 3.6 MiB/s, 57 objects/s recovering 2024-08-17T18:00:58.219 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:58 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:00:58.219 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:58 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:00:58.219 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:00:58 smithi136 ceph-mon[115078]: osdmap e1396: 8 total, 7 up, 7 in 2024-08-17T18:00:58.220 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:58 smithi136 ceph-mon[109861]: pgmap v2398: 99 pgs: 1 active+recovery_wait+degraded+remapped, 21 active+undersized, 1 active+recovering+degraded+remapped, 9 active+undersized+degraded, 67 active+clean; 4.9 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 283 op/s; 24029/240918 objects degraded (9.974%); 2342/240918 objects misplaced (0.972%); 3.6 MiB/s, 57 objects/s recovering 2024-08-17T18:00:58.220 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:58 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:00:58.220 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:58 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:00:58.220 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:00:58 smithi136 ceph-mon[109861]: osdmap e1396: 8 total, 7 up, 7 in 2024-08-17T18:00:58.280 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:58.302 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:58.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:58.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:58.404 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:58.430 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:58.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:58.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:58.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:58 smithi165 ceph-mon[105517]: pgmap v2398: 99 pgs: 1 active+recovery_wait+degraded+remapped, 21 active+undersized, 1 active+recovering+degraded+remapped, 9 active+undersized+degraded, 67 active+clean; 4.9 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 283 op/s; 24029/240918 objects degraded (9.974%); 2342/240918 objects misplaced (0.972%); 3.6 MiB/s, 57 objects/s recovering 2024-08-17T18:00:58.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:58 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:00:58.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:58 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:00:58.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:00:58 smithi165 ceph-mon[105517]: osdmap e1396: 8 total, 7 up, 7 in 2024-08-17T18:00:58.532 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:00:58.555 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:58.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:58.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:00:58.657 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:58.686 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:58.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:58.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:00:58.788 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:58.813 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:58.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:58.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:58.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:58.942 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:58.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:58.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:59.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:59.069 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:59.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:59.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:59.170 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:59.196 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:59.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:59.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:59.298 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:59.318 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-08-17T18:00:59.318 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:00:59.318 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-08-17T18:00:59.318 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2024-08-17T18:00:59.318 DEBUG:teuthology.orchestra.run.smithi136:osd.2> sudo journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.2.service 2024-08-17T18:00:59.321 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl start ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.2 2024-08-17T18:00:59.323 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:59.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:59.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:59.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:00:59.448 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:59.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:59.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:00:59.550 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:00:59.574 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:00:59.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:59.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:00:59.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:00:59.696 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:00:59 smithi136 systemd[1]: Starting Ceph osd.2 for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T18:00:59.699 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:59.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:59.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:00:59.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:00:59.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:00:59.824 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:59.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:59.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:00:59.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:00:59.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:00:59.954 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:00:59.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:00.055 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:00.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:00.083 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:00.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:00.184 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:00.208 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:00.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:00.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:00.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:00.336 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:00.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:00.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:00.371 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:00 smithi136 ceph-mon[109861]: pgmap v2400: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 270 op/s; 23707/237804 objects degraded (9.969%); 1840/237804 objects misplaced (0.774%); 4.3 MiB/s, 68 objects/s recovering 2024-08-17T18:01:00.371 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:00 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 23707/237804 objects degraded (9.969%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T18:01:00.371 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:00 smithi136 ceph-mon[115078]: pgmap v2400: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 270 op/s; 23707/237804 objects degraded (9.969%); 1840/237804 objects misplaced (0.774%); 4.3 MiB/s, 68 objects/s recovering 2024-08-17T18:01:00.371 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:00 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 23707/237804 objects degraded (9.969%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T18:01:00.438 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:00.460 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:00.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:00.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:00 smithi165 ceph-mon[105517]: pgmap v2400: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 270 op/s; 23707/237804 objects degraded (9.969%); 1840/237804 objects misplaced (0.774%); 4.3 MiB/s, 68 objects/s recovering 2024-08-17T18:01:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:00 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 23707/237804 objects degraded (9.969%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T18:01:00.562 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:00.589 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:00.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:00.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:00.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:00.719 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:00.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:00.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:00.821 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:00.845 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:00.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:00.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:00.947 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:00.973 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:00.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:00.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:01.050 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:01:01.050 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T18:01:01.050 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-17T18:01:01.050 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-17T18:01:01.050 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:01:01.059 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:01:01.074 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:01.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:01.103 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:01.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:01.196 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:01 smithi136 podman[290209]: 2024-08-17 18:01:01.056356133 +0000 UTC m=+0.202130543 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T18:01:01.204 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:01.227 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:01.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:01.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:01.328 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:01.352 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:01.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:01.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:01.453 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:01.483 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:01.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:01.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:01.584 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:01.611 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:01.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:01.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:01.713 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:01.736 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:01.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:01.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:01.838 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:01.861 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:01.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:01.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:01.882 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:01:01.963 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:01.991 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:01.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:01.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:02.093 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:02.126 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:02.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:02.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:02.167 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-17T18:01:02.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:02.251 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:02.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:02.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:02.353 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:02.375 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:02.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:02.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:02.477 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:02.506 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:02.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:02.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:02.508 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:02 smithi136 ceph-mon[109861]: pgmap v2401: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 259 op/s; 23419/236937 objects degraded (9.884%); 1759/236937 objects misplaced (0.742%); 4.0 MiB/s, 64 objects/s recovering 2024-08-17T18:01:02.508 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:02 smithi136 ceph-mon[115078]: pgmap v2401: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 259 op/s; 23419/236937 objects degraded (9.884%); 1759/236937 objects misplaced (0.742%); 4.0 MiB/s, 64 objects/s recovering 2024-08-17T18:01:02.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:02 smithi165 ceph-mon[105517]: pgmap v2401: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 259 op/s; 23419/236937 objects degraded (9.884%); 1759/236937 objects misplaced (0.742%); 4.0 MiB/s, 64 objects/s recovering 2024-08-17T18:01:02.608 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:02.631 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:02.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:02.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:02.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:02.755 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:02.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:02.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:02.808 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:02 smithi136 podman[290209]: 2024-08-17 18:01:02.556667624 +0000 UTC m=+1.702442005 container create 4a4fe7c5a31182ef829a44fe6a43ae47c13c07ea322bc06df2df596609828c19 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T18:01:02.857 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:02.880 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:02.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:02.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:02.982 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:03.003 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:03.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:03.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:03.105 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:03.133 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:03.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:03.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:03.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:03.268 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:03.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:03.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:03.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:03.399 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:03.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:03.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:03.501 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:03.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:03.527 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:03.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:03.628 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:03.653 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:03.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:03.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:03.755 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:03.777 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:03.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:03.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:03.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:03.907 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:03.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:03.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:04.009 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:04.032 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:04.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:04.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:04.134 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:04.159 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:04.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:04.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:04.261 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:04.292 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:04.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:04.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:04.294 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:01:04.394 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:04.395 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:04 smithi136 ceph-mon[109861]: pgmap v2402: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 441 op/s; 23027/230898 objects degraded (9.973%); 1118/230898 objects misplaced (0.484%); 4.3 MiB/s, 69 objects/s recovering 2024-08-17T18:01:04.395 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:04 smithi136 ceph-mon[115078]: pgmap v2402: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 441 op/s; 23027/230898 objects degraded (9.973%); 1118/230898 objects misplaced (0.484%); 4.3 MiB/s, 69 objects/s recovering 2024-08-17T18:01:04.396 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:04 smithi136 podman[290209]: 2024-08-17 18:01:04.143595805 +0000 UTC m=+3.289370189 container init 4a4fe7c5a31182ef829a44fe6a43ae47c13c07ea322bc06df2df596609828c19 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-cd8bdbe, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T18:01:04.396 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:04 smithi136 podman[290209]: 2024-08-17 18:01:04.151602363 +0000 UTC m=+3.297376748 container start 4a4fe7c5a31182ef829a44fe6a43ae47c13c07ea322bc06df2df596609828c19 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate, maintainer=Guillaume Abrioux , RELEASE=main-cd8bdbe, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-08-17T18:01:04.396 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:04 smithi136 podman[290209]: 2024-08-17 18:01:04.251885392 +0000 UTC m=+3.397659775 container attach 4a4fe7c5a31182ef829a44fe6a43ae47c13c07ea322bc06df2df596609828c19 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-08-17T18:01:04.396 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:04 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate[290618]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T18:01:04.396 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:04 smithi136 bash[290209]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T18:01:04.396 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:04 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate[290618]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T18:01:04.396 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:04 smithi136 bash[290209]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T18:01:04.396 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:04 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate[290618]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-17T18:01:04.396 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:04 smithi136 bash[290209]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-17T18:01:04.396 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:04 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate[290618]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T18:01:04.396 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:04 smithi136 bash[290209]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T18:01:04.422 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:04.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:04.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:04.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:04 smithi165 ceph-mon[105517]: pgmap v2402: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 441 op/s; 23027/230898 objects degraded (9.973%); 1118/230898 objects misplaced (0.484%); 4.3 MiB/s, 69 objects/s recovering 2024-08-17T18:01:04.523 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:04.552 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:04.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:04.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:04.653 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:04.681 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:04.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:04.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:04.696 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:04 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate[290618]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T18:01:04.696 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:04 smithi136 bash[290209]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T18:01:04.696 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:04 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate[290618]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-17T18:01:04.696 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:04 smithi136 bash[290209]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-17T18:01:04.696 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:04 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate[290618]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2024-08-17T18:01:04.697 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:04 smithi136 bash[290209]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2024-08-17T18:01:04.783 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:04.807 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:04.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:04.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:04.909 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:04.936 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:04.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:04.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:04.965 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:01:04.966 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:01:04.966 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:01:04.966 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:01:04.966 INFO:teuthology.orchestra.run.smithi136.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-17T18:01:04.971 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:01:04.971 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:01:04.992 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:01:04.992 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:01:05.020 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:01:05.020 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:01:05.038 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:05.039 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:01:05.039 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:01:05.058 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:01:05.058 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:01:05.063 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:05.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:05.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:05.165 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:05.190 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:05.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:05.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:05.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:05.310 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:05 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 23027/230898 objects degraded (9.973%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T18:01:05.311 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:05 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3580878433' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:01:05.311 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:05 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 23027/230898 objects degraded (9.973%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T18:01:05.311 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:05 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3580878433' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:01:05.319 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:05.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:05.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:05.421 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:05.442 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:05.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:05.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:05.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:05 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 23027/230898 objects degraded (9.973%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T18:01:05.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:05 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3580878433' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:01:05.543 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:05.566 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:05.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:05.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:05.619 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:05 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate[290618]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2024-08-17T18:01:05.620 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:05 smithi136 bash[290209]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2024-08-17T18:01:05.620 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:05 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate[290618]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2024-08-17T18:01:05.620 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:05 smithi136 bash[290209]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2024-08-17T18:01:05.620 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:05 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate[290618]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-08-17T18:01:05.620 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:05 smithi136 bash[290209]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-08-17T18:01:05.620 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:05 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate[290618]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-17T18:01:05.620 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:05 smithi136 bash[290209]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-17T18:01:05.620 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:05 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate[290618]: --> ceph-volume lvm activate successful for osd ID: 2 2024-08-17T18:01:05.620 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:05 smithi136 bash[290209]: --> ceph-volume lvm activate successful for osd ID: 2 2024-08-17T18:01:05.668 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:05.699 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:05.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:05.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:05.801 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:05.826 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:05.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:05.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:05.928 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:05.946 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:05 smithi136 podman[290209]: 2024-08-17 18:01:05.619309342 +0000 UTC m=+4.765083731 container died 4a4fe7c5a31182ef829a44fe6a43ae47c13c07ea322bc06df2df596609828c19 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-cd8bdbe) 2024-08-17T18:01:05.953 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:05.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:05.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:06.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:06.078 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:06.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:06.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:06.180 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:06.207 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:06.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:06.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:06.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:06.332 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:06.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:06.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:06.434 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:06.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:06 smithi136 ceph-mon[109861]: pgmap v2403: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 441 op/s; 23027/230898 objects degraded (9.973%); 1118/230898 objects misplaced (0.484%); 4.3 MiB/s, 69 objects/s recovering 2024-08-17T18:01:06.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:06 smithi136 ceph-mon[109861]: 54.4 scrub starts 2024-08-17T18:01:06.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:01:06.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:06 smithi136 ceph-mon[115078]: pgmap v2403: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 441 op/s; 23027/230898 objects degraded (9.973%); 1118/230898 objects misplaced (0.484%); 4.3 MiB/s, 69 objects/s recovering 2024-08-17T18:01:06.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:06 smithi136 ceph-mon[115078]: 54.4 scrub starts 2024-08-17T18:01:06.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:01:06.458 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:06.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:06.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:06.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:06 smithi165 ceph-mon[105517]: pgmap v2403: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 441 op/s; 23027/230898 objects degraded (9.973%); 1118/230898 objects misplaced (0.484%); 4.3 MiB/s, 69 objects/s recovering 2024-08-17T18:01:06.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:06 smithi165 ceph-mon[105517]: 54.4 scrub starts 2024-08-17T18:01:06.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:01:06.560 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:06.586 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:06.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:06.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:06.688 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:06.710 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:06.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:06.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:06.812 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:06.835 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:06.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:06.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:06.937 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:06.959 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:06.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:06.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:07.061 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:07.095 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:07.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:07.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:07.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:07.219 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:07.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:07.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:07.321 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:07.343 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:07.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:07.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:07.445 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:07.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:01:07.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:01:07.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:01:07.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:01:07.467 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:07.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:07.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:07.569 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:07.581 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-17T18:01:07.582 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-17T18:01:07.594 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:07.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:07.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:07.696 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:07.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:07.734 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:07.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:07.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:01:07.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:01:07.798 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:01:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:01:07] "GET /metrics HTTP/1.1" 200 36570 "" "Prometheus/2.51.0" 2024-08-17T18:01:07.835 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:07.869 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:07.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:07.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:07.970 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:07.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:07.999 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:07.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:08.080 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:07 smithi136 podman[291149]: 2024-08-17 18:01:07.953946308 +0000 UTC m=+2.456523404 container remove 4a4fe7c5a31182ef829a44fe6a43ae47c13c07ea322bc06df2df596609828c19 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-activate, ceph=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-cd8bdbe, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-08-17T18:01:08.100 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:08.120 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:08.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:08.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:08.222 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:08.244 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:08.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:08.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:08.346 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:08.372 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:08.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:08.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:08.474 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:08.501 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:08.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:08.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:08.603 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:08.628 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:08.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:08.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:08.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:08 smithi136 ceph-mon[109861]: pgmap v2404: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 496 op/s; 22694/226029 objects degraded (10.040%); 1039/226029 objects misplaced (0.460%); 4.6 MiB/s, 73 objects/s recovering 2024-08-17T18:01:08.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:08 smithi136 ceph-mon[109861]: 54.17 scrub starts 2024-08-17T18:01:08.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:01:08.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:01:08.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:01:08.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:08 smithi136 ceph-mon[115078]: pgmap v2404: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 496 op/s; 22694/226029 objects degraded (10.040%); 1039/226029 objects misplaced (0.460%); 4.6 MiB/s, 73 objects/s recovering 2024-08-17T18:01:08.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:08 smithi136 ceph-mon[115078]: 54.17 scrub starts 2024-08-17T18:01:08.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:01:08.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:01:08.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:01:08.730 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:08.752 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:08.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:08.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:08 smithi165 ceph-mon[105517]: pgmap v2404: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 496 op/s; 22694/226029 objects degraded (10.040%); 1039/226029 objects misplaced (0.460%); 4.6 MiB/s, 73 objects/s recovering 2024-08-17T18:01:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:08 smithi165 ceph-mon[105517]: 54.17 scrub starts 2024-08-17T18:01:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:01:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:01:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:01:08.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:08.874 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:08.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:08.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:08.884 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:01:08.976 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:09.003 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:09.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:09.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:09.105 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:09.136 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:09.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:09.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:09.237 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:09.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:09.260 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:09.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:09.361 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:09.383 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:09.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:09.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:09.485 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:09.513 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:09.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:09.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:09.583 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:01:09.615 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:09.647 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:09.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:09.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:09.693 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:09 smithi136 ceph-mon[109861]: pgmap v2405: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.5 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 575 op/s; 18568/222033 objects degraded (8.363%); 4044/222033 objects misplaced (1.821%); 5.9 MiB/s, 94 objects/s recovering 2024-08-17T18:01:09.694 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:09 smithi136 ceph-mon[109861]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-08-17T18:01:09.694 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:09 smithi136 ceph-mon[115078]: pgmap v2405: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.5 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 575 op/s; 18568/222033 objects degraded (8.363%); 4044/222033 objects misplaced (1.821%); 5.9 MiB/s, 94 objects/s recovering 2024-08-17T18:01:09.694 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:09 smithi136 ceph-mon[115078]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-08-17T18:01:09.748 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:09 smithi165 ceph-mon[105517]: pgmap v2405: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.5 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 575 op/s; 18568/222033 objects degraded (8.363%); 4044/222033 objects misplaced (1.821%); 5.9 MiB/s, 94 objects/s recovering 2024-08-17T18:01:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:09 smithi165 ceph-mon[105517]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-08-17T18:01:09.770 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:09.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:09.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:09.872 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:09.893 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:09.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:09.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:09.995 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:10.022 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:10.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:10.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:10.068 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:01:10.125 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:10.149 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:10.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:10.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:10.251 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:10.273 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:10.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:10.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:10.374 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:10.395 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:10.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:10.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:10.427 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:01:10.498 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:10.524 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:10.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:10.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:10.617 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:10 smithi136 ceph-mon[109861]: 54.19 scrub starts 2024-08-17T18:01:10.617 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:10 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 18568/222033 objects degraded (8.363%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:01:10.617 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:10 smithi136 ceph-mon[109861]: 54.7 scrub starts 2024-08-17T18:01:10.617 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:10 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2432722018' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:01:10.617 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:10 smithi136 podman[291846]: 2024-08-17 18:01:10.101859784 +0000 UTC m=+0.025969947 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T18:01:10.618 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:10 smithi136 ceph-mon[115078]: 54.19 scrub starts 2024-08-17T18:01:10.618 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:10 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 18568/222033 objects degraded (8.363%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:01:10.618 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:10 smithi136 ceph-mon[115078]: 54.7 scrub starts 2024-08-17T18:01:10.618 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:10 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2432722018' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:01:10.626 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:10.651 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:10.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:10.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:10.753 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:10 smithi165 ceph-mon[105517]: 54.19 scrub starts 2024-08-17T18:01:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:10 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 18568/222033 objects degraded (8.363%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:01:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:10 smithi165 ceph-mon[105517]: 54.7 scrub starts 2024-08-17T18:01:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:10 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2432722018' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:01:10.780 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:10.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:10.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:10.881 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:10.907 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:10.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:10.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:11.009 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:11.032 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:11.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:11.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:11.134 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:11.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:11.156 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:11.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:11.196 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:11 smithi136 podman[291846]: 2024-08-17 18:01:11.003060599 +0000 UTC m=+0.927170743 container create 697e0eb06edb12bcdfb22c6cd6ebdb7b4731fdcc3914ff0fa671d3c37cc55ea5 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-cd8bdbe, GIT_CLEAN=True) 2024-08-17T18:01:11.257 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:11.280 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:11.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:11.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:11.382 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:11.406 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:11.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:11.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:11.508 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:11.534 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:11.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:11.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:11.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:11 smithi136 ceph-mon[109861]: 54.9 scrub starts 2024-08-17T18:01:11.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:11 smithi136 ceph-mon[109861]: pgmap v2406: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.5 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 468 op/s; 18568/222033 objects degraded (8.363%); 4044/222033 objects misplaced (1.821%); 4.2 MiB/s, 67 objects/s recovering 2024-08-17T18:01:11.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:11 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:01:11.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:11 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2432722018' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:01:11.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:11 smithi136 ceph-mon[109861]: osdmap e1397: 8 total, 7 up, 7 in 2024-08-17T18:01:11.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:11 smithi136 ceph-mon[109861]: 54.14 scrub starts 2024-08-17T18:01:11.577 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:11 smithi136 ceph-mon[109861]: 54.0 scrub starts 2024-08-17T18:01:11.578 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:11 smithi136 ceph-mon[115078]: 54.9 scrub starts 2024-08-17T18:01:11.578 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:11 smithi136 ceph-mon[115078]: pgmap v2406: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.5 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 468 op/s; 18568/222033 objects degraded (8.363%); 4044/222033 objects misplaced (1.821%); 4.2 MiB/s, 67 objects/s recovering 2024-08-17T18:01:11.578 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:11 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:01:11.578 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:11 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2432722018' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:01:11.578 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:11 smithi136 ceph-mon[115078]: osdmap e1397: 8 total, 7 up, 7 in 2024-08-17T18:01:11.578 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:11 smithi136 ceph-mon[115078]: 54.14 scrub starts 2024-08-17T18:01:11.578 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:11 smithi136 ceph-mon[115078]: 54.0 scrub starts 2024-08-17T18:01:11.636 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:11.658 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:11.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:11.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:11.759 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:11 smithi165 ceph-mon[105517]: 54.9 scrub starts 2024-08-17T18:01:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:11 smithi165 ceph-mon[105517]: pgmap v2406: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.5 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 468 op/s; 18568/222033 objects degraded (8.363%); 4044/222033 objects misplaced (1.821%); 4.2 MiB/s, 67 objects/s recovering 2024-08-17T18:01:11.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:11 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:01:11.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:11 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2432722018' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:01:11.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:11 smithi165 ceph-mon[105517]: osdmap e1397: 8 total, 7 up, 7 in 2024-08-17T18:01:11.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:11 smithi165 ceph-mon[105517]: 54.14 scrub starts 2024-08-17T18:01:11.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:11 smithi165 ceph-mon[105517]: 54.0 scrub starts 2024-08-17T18:01:11.784 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:11.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:11.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:11.860 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:11 smithi136 podman[291846]: 2024-08-17 18:01:11.57639464 +0000 UTC m=+1.500504790 container init 697e0eb06edb12bcdfb22c6cd6ebdb7b4731fdcc3914ff0fa671d3c37cc55ea5 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, RELEASE=main-cd8bdbe, maintainer=Guillaume Abrioux ) 2024-08-17T18:01:11.860 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:11 smithi136 podman[291846]: 2024-08-17 18:01:11.582451854 +0000 UTC m=+1.506561998 container start 697e0eb06edb12bcdfb22c6cd6ebdb7b4731fdcc3914ff0fa671d3c37cc55ea5 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=main-cd8bdbe, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-08-17T18:01:11.860 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:11 smithi136 ceph-osd[292061]: -- 172.21.15.136:0/1334695961 <== mon.2 v2:172.21.15.136:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560029cd9a40 con 0x56002aaa6400 2024-08-17T18:01:11.886 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:11.912 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:11.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:11.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:12.013 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:12.039 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:12.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:12.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:12.123 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:01:12.123 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:01:12.141 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:12.164 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:12.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:12.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:12.220 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-17T18:01:12.266 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:12.289 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:12.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:12.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:12.391 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:12.418 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:12.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:12.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:12.434 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:12 smithi136 bash[291846]: 697e0eb06edb12bcdfb22c6cd6ebdb7b4731fdcc3914ff0fa671d3c37cc55ea5 2024-08-17T18:01:12.434 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:12 smithi136 systemd[1]: Started Ceph osd.2 for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T18:01:12.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:12.541 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:12.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:12.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:12.643 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:12.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:12.666 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:12.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:12.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:12 smithi136 ceph-mon[109861]: 54.14 scrub ok 2024-08-17T18:01:12.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:12 smithi136 ceph-mon[109861]: 54.4 scrub starts 2024-08-17T18:01:12.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:12 smithi136 ceph-mon[115078]: 54.14 scrub ok 2024-08-17T18:01:12.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:12 smithi136 ceph-mon[115078]: 54.4 scrub starts 2024-08-17T18:01:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:12 smithi165 ceph-mon[105517]: 54.14 scrub ok 2024-08-17T18:01:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:12 smithi165 ceph-mon[105517]: 54.4 scrub starts 2024-08-17T18:01:12.766 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:12.791 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:12.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:12.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:12.893 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:12.920 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:12.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:12.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:13.022 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:13.043 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:13.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:13.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:13.145 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:13.167 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:13.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:13.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:13.269 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:13.291 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:13.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:13.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:13.393 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:13.417 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:13.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:13.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:13.518 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:13.541 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:13.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:13.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:13.643 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:13.667 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:13.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:13.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:13.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:13 smithi136 ceph-mon[109861]: pgmap v2408: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 18073/215631 objects degraded (8.381%); 3890/215631 objects misplaced (1.804%); 3.0 MiB/s, 48 objects/s recovering 2024-08-17T18:01:13.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:13 smithi136 ceph-mon[109861]: 54.4 scrub ok 2024-08-17T18:01:13.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:13 smithi136 ceph-mon[109861]: 54.17 scrub starts 2024-08-17T18:01:13.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:13 smithi136 ceph-mon[115078]: pgmap v2408: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 18073/215631 objects degraded (8.381%); 3890/215631 objects misplaced (1.804%); 3.0 MiB/s, 48 objects/s recovering 2024-08-17T18:01:13.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:13 smithi136 ceph-mon[115078]: 54.4 scrub ok 2024-08-17T18:01:13.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:13 smithi136 ceph-mon[115078]: 54.17 scrub starts 2024-08-17T18:01:13.697 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:13 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2[291983]: 2024-08-17T18:01:13.322+0000 7f0b62061640 -1 Falling back to public interface 2024-08-17T18:01:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:13 smithi165 ceph-mon[105517]: pgmap v2408: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 18073/215631 objects degraded (8.381%); 3890/215631 objects misplaced (1.804%); 3.0 MiB/s, 48 objects/s recovering 2024-08-17T18:01:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:13 smithi165 ceph-mon[105517]: 54.4 scrub ok 2024-08-17T18:01:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:13 smithi165 ceph-mon[105517]: 54.17 scrub starts 2024-08-17T18:01:13.769 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:13.793 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:13.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:13.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:13.895 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:13.919 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:13.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:13.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:14.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:14.045 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:14.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:14.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:14.147 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:14.171 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:14.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:14.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:14.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:14.297 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:14.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:14.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:14.400 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:14.429 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:14.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:14.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:14.531 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:14.565 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:14.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:14.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:14.667 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:14.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:14 smithi136 ceph-mon[109861]: 54.17 scrub ok 2024-08-17T18:01:14.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:14 smithi136 ceph-mon[115078]: 54.17 scrub ok 2024-08-17T18:01:14.702 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:14.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:14.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:14.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:14 smithi165 ceph-mon[105517]: 54.17 scrub ok 2024-08-17T18:01:14.804 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:14.831 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:14.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:14.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:14.933 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:14.957 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:14.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:14.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:15.059 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:15.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:15.087 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:15.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:15.188 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:15.219 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:15.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:15.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:15.320 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:15.342 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:15.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:15.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:15.444 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:15.465 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:15.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:15.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:15.567 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:15.588 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:15.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:15.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:15.690 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:15.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:15 smithi136 ceph-mon[109861]: pgmap v2409: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 17915/214728 objects degraded (8.343%); 3890/214728 objects misplaced (1.812%); 3.0 MiB/s, 48 objects/s recovering 2024-08-17T18:01:15.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:15 smithi136 ceph-mon[109861]: 54.19 scrub starts 2024-08-17T18:01:15.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:15 smithi136 ceph-mon[109861]: 54.19 scrub ok 2024-08-17T18:01:15.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:15 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 17915/214728 objects degraded (8.343%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:01:15.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:15 smithi136 ceph-mon[109861]: 54.7 scrub starts 2024-08-17T18:01:15.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:15 smithi136 ceph-mon[115078]: pgmap v2409: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 17915/214728 objects degraded (8.343%); 3890/214728 objects misplaced (1.812%); 3.0 MiB/s, 48 objects/s recovering 2024-08-17T18:01:15.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:15 smithi136 ceph-mon[115078]: 54.19 scrub starts 2024-08-17T18:01:15.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:15 smithi136 ceph-mon[115078]: 54.19 scrub ok 2024-08-17T18:01:15.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:15 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 17915/214728 objects degraded (8.343%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:01:15.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:15 smithi136 ceph-mon[115078]: 54.7 scrub starts 2024-08-17T18:01:15.719 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:15.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:15.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:15.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-17T18:01:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:15 smithi165 ceph-mon[105517]: pgmap v2409: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 17915/214728 objects degraded (8.343%); 3890/214728 objects misplaced (1.812%); 3.0 MiB/s, 48 objects/s recovering 2024-08-17T18:01:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:15 smithi165 ceph-mon[105517]: 54.19 scrub starts 2024-08-17T18:01:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:15 smithi165 ceph-mon[105517]: 54.19 scrub ok 2024-08-17T18:01:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:15 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 17915/214728 objects degraded (8.343%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:01:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:15 smithi165 ceph-mon[105517]: 54.7 scrub starts 2024-08-17T18:01:15.821 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:15.847 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:15.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:15.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:15.949 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:15.981 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:15.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:15.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:16.083 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:16.109 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:16.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:16.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:16.211 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:16.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:16.236 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:16.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:16.337 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:16.361 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:16.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:16.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:16.463 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:16.486 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:16.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:16.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:16.588 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:16.611 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:16.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:16.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:16.672 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:16 smithi136 ceph-mon[109861]: 54.9 scrub starts 2024-08-17T18:01:16.672 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:16 smithi136 ceph-mon[109861]: 54.7 scrub ok 2024-08-17T18:01:16.672 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:16 smithi136 ceph-mon[109861]: 54.9 scrub ok 2024-08-17T18:01:16.672 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:16 smithi136 ceph-mon[109861]: 54.5 scrub starts 2024-08-17T18:01:16.672 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:16 smithi136 ceph-mon[115078]: 54.9 scrub starts 2024-08-17T18:01:16.672 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:16 smithi136 ceph-mon[115078]: 54.7 scrub ok 2024-08-17T18:01:16.672 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:16 smithi136 ceph-mon[115078]: 54.9 scrub ok 2024-08-17T18:01:16.672 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:16 smithi136 ceph-mon[115078]: 54.5 scrub starts 2024-08-17T18:01:16.713 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:16.736 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:16.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:16.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:16 smithi165 ceph-mon[105517]: 54.9 scrub starts 2024-08-17T18:01:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:16 smithi165 ceph-mon[105517]: 54.7 scrub ok 2024-08-17T18:01:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:16 smithi165 ceph-mon[105517]: 54.9 scrub ok 2024-08-17T18:01:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:16 smithi165 ceph-mon[105517]: 54.5 scrub starts 2024-08-17T18:01:16.839 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:16.864 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:16.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:16.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:16.966 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:16.992 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:16.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:16.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:17.094 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:17.117 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:17.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:17.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:17.167 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:01:17.220 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:17.250 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:17.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:17.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:17.352 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:17.375 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:17.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:17.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:17.477 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:17.499 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:17.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:17.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:17.601 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:17.622 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:17.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:17.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:17.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:17 smithi136 ceph-mon[109861]: pgmap v2410: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 17915/214728 objects degraded (8.343%); 3890/214728 objects misplaced (1.812%); 2.5 MiB/s, 39 objects/s recovering 2024-08-17T18:01:17.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:17 smithi136 ceph-mon[109861]: 54.5 scrub ok 2024-08-17T18:01:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:17 smithi136 ceph-mon[115078]: pgmap v2410: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 17915/214728 objects degraded (8.343%); 3890/214728 objects misplaced (1.812%); 2.5 MiB/s, 39 objects/s recovering 2024-08-17T18:01:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:17 smithi136 ceph-mon[115078]: 54.5 scrub ok 2024-08-17T18:01:17.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:01:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:01:17] "GET /metrics HTTP/1.1" 200 36570 "" "Prometheus/2.51.0" 2024-08-17T18:01:17.727 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:17.750 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:17.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:17.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:17 smithi165 ceph-mon[105517]: pgmap v2410: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 17915/214728 objects degraded (8.343%); 3890/214728 objects misplaced (1.812%); 2.5 MiB/s, 39 objects/s recovering 2024-08-17T18:01:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:17 smithi165 ceph-mon[105517]: 54.5 scrub ok 2024-08-17T18:01:17.852 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:17.879 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:17.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:17.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:17.981 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:18.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:18.010 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:18.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:18.056 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:01:18.056 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:01:18.082 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:01:18.082 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:01:18.082 INFO:teuthology.orchestra.run.smithi136.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-17T18:01:18.112 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:18.117 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:01:18.117 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:01:18.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:18.143 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:18.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:18.152 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:01:18.152 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:01:18.186 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:01:18.187 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:01:18.219 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:01:18.219 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:01:18.244 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:18.254 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:01:18.254 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:01:18.274 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:18.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:18.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:18.364 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:01:18.375 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:18.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:18.407 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:18.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:18.508 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:18.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:18.542 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:18.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:18.643 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:18.671 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:18.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:18.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:18.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:18 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/166922310' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:01:18.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:18 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/166922310' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:01:18.703 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:01:18.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:18 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/166922310' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:01:18.772 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:18.807 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:18.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:18.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:18.909 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:18.946 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:18.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:18.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:19.048 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:19.078 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:19.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:19.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:19.180 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:19.204 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:19.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:19.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:19.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:19.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:19.328 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:19.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:19.429 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:19.449 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:19.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:19.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:19.551 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:19.579 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:19.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:19.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:19.681 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:19.711 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:19.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:19.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:19 smithi165 ceph-mon[105517]: pgmap v2411: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 17254/206673 objects degraded (8.348%); 3715/206673 objects misplaced (1.798%); 1.0 MiB/s, 16 objects/s recovering 2024-08-17T18:01:19.812 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:19.843 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:19.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:19.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:19.945 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:19.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:19 smithi136 ceph-mon[109861]: pgmap v2411: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 17254/206673 objects degraded (8.348%); 3715/206673 objects misplaced (1.798%); 1.0 MiB/s, 16 objects/s recovering 2024-08-17T18:01:19.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:19 smithi136 ceph-mon[115078]: pgmap v2411: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 17254/206673 objects degraded (8.348%); 3715/206673 objects misplaced (1.798%); 1.0 MiB/s, 16 objects/s recovering 2024-08-17T18:01:19.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:19.979 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:19.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:20.080 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:20.109 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:20.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:20.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:20.211 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:20.249 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:20.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:20.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:20.351 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:20.377 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:20.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:20.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:20.478 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:20.508 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:20.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:20.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:20.610 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:20.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:20.637 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:20.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:20.738 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:20.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:20.761 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:20.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:20 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 17254/206673 objects degraded (8.348%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:01:20.862 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:20.887 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:20.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:20.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:20.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:20 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 17254/206673 objects degraded (8.348%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:01:20.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:20 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 17254/206673 objects degraded (8.348%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:01:20.947 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:20 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2[291983]: 2024-08-17T18:01:20.614+0000 7f0b62061640 -1 osd.2 1392 log_to_monitors true 2024-08-17T18:01:20.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:20.994 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-17T18:01:20.995 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-17T18:01:21.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:21.024 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:21.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:21.125 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:21.149 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:21.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:21.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:21.251 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:21.289 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:21.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:21.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:21.391 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:21.419 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:21.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:21.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:21.498 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:21 smithi136 ceph-mon[109861]: pgmap v2412: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 17254/206673 objects degraded (8.348%); 3715/206673 objects misplaced (1.798%); 1.0 MiB/s, 16 objects/s recovering 2024-08-17T18:01:21.499 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:21 smithi136 ceph-mon[109861]: from='osd.2 [v2:172.21.15.136:6818/809598197,v1:172.21.15.136:6819/809598197]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-17T18:01:21.499 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:21 smithi136 ceph-mon[109861]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-17T18:01:21.499 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:21 smithi136 ceph-mon[109861]: 54.15 scrub starts 2024-08-17T18:01:21.499 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:01:21.499 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:01:21.499 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:21 smithi136 ceph-mon[115078]: pgmap v2412: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 17254/206673 objects degraded (8.348%); 3715/206673 objects misplaced (1.798%); 1.0 MiB/s, 16 objects/s recovering 2024-08-17T18:01:21.499 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:21 smithi136 ceph-mon[115078]: from='osd.2 [v2:172.21.15.136:6818/809598197,v1:172.21.15.136:6819/809598197]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-17T18:01:21.499 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:21 smithi136 ceph-mon[115078]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-17T18:01:21.499 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:21 smithi136 ceph-mon[115078]: 54.15 scrub starts 2024-08-17T18:01:21.499 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:01:21.500 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:01:21.521 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:21.542 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:21.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:21.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:21.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:21.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:21.666 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:21.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:21 smithi165 ceph-mon[105517]: pgmap v2412: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 17254/206673 objects degraded (8.348%); 3715/206673 objects misplaced (1.798%); 1.0 MiB/s, 16 objects/s recovering 2024-08-17T18:01:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:21 smithi165 ceph-mon[105517]: from='osd.2 [v2:172.21.15.136:6818/809598197,v1:172.21.15.136:6819/809598197]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-17T18:01:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:21 smithi165 ceph-mon[105517]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-17T18:01:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:21 smithi165 ceph-mon[105517]: 54.15 scrub starts 2024-08-17T18:01:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:01:21.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:01:21.768 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:21.792 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:21.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:21.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:21.866 INFO:teuthology.orchestra.run.smithi136.stderr:no valid command found; 10 closest matches: 2024-08-17T18:01:21.866 INFO:teuthology.orchestra.run.smithi136.stderr:0 2024-08-17T18:01:21.866 INFO:teuthology.orchestra.run.smithi136.stderr:1 2024-08-17T18:01:21.866 INFO:teuthology.orchestra.run.smithi136.stderr:2 2024-08-17T18:01:21.866 INFO:teuthology.orchestra.run.smithi136.stderr:abort 2024-08-17T18:01:21.866 INFO:teuthology.orchestra.run.smithi136.stderr:assert 2024-08-17T18:01:21.867 INFO:teuthology.orchestra.run.smithi136.stderr:bluefs debug_inject_read_zeros 2024-08-17T18:01:21.867 INFO:teuthology.orchestra.run.smithi136.stderr:bluefs files list 2024-08-17T18:01:21.867 INFO:teuthology.orchestra.run.smithi136.stderr:bluefs stats 2024-08-17T18:01:21.867 INFO:teuthology.orchestra.run.smithi136.stderr:bluestore allocator dump block 2024-08-17T18:01:21.867 INFO:teuthology.orchestra.run.smithi136.stderr:bluestore allocator fragmentation block 2024-08-17T18:01:21.867 INFO:teuthology.orchestra.run.smithi136.stderr:admin_socket: invalid command 2024-08-17T18:01:21.867 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:01:21 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2[291983]: 2024-08-17T18:01:21.497+0000 7f0b55e03640 -1 osd.2 1392 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T18:01:21.895 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:21.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:21.924 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:21.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:22.025 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:22.047 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:22.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:22.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:22.149 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:22.171 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:22.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:22.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:22.273 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:22.297 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:22.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:22.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:22.399 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:22.425 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:22.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:22.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:22.527 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:22.535 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:01:22.550 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:22.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:22.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:22.652 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:22.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:22.674 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:22.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:22.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:22 smithi165 ceph-mon[105517]: 54.15 scrub ok 2024-08-17T18:01:22.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:22 smithi165 ceph-mon[105517]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-17T18:01:22.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:22 smithi165 ceph-mon[105517]: osdmap e1398: 8 total, 7 up, 7 in 2024-08-17T18:01:22.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:22 smithi165 ceph-mon[105517]: from='osd.2 [v2:172.21.15.136:6818/809598197,v1:172.21.15.136:6819/809598197]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-17T18:01:22.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:22 smithi165 ceph-mon[105517]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-17T18:01:22.772 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:22 smithi136 ceph-mon[109861]: 54.15 scrub ok 2024-08-17T18:01:22.772 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:22 smithi136 ceph-mon[109861]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-17T18:01:22.772 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:22 smithi136 ceph-mon[109861]: osdmap e1398: 8 total, 7 up, 7 in 2024-08-17T18:01:22.772 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:22 smithi136 ceph-mon[109861]: from='osd.2 [v2:172.21.15.136:6818/809598197,v1:172.21.15.136:6819/809598197]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-17T18:01:22.772 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:22 smithi136 ceph-mon[109861]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-17T18:01:22.772 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:22 smithi136 ceph-mon[115078]: 54.15 scrub ok 2024-08-17T18:01:22.773 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:22 smithi136 ceph-mon[115078]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-17T18:01:22.773 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:22 smithi136 ceph-mon[115078]: osdmap e1398: 8 total, 7 up, 7 in 2024-08-17T18:01:22.773 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:22 smithi136 ceph-mon[115078]: from='osd.2 [v2:172.21.15.136:6818/809598197,v1:172.21.15.136:6819/809598197]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-17T18:01:22.773 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:22 smithi136 ceph-mon[115078]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-17T18:01:22.775 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:22.798 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:22.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:22.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:22.900 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:22.922 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:22.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:22.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:22.996 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:01:23.024 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:23.046 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:23.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:23.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:23.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:23.172 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:23.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:23.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:23.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:23.298 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:23.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:23.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:23.302 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:01:23.302 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T18:01:23.302 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-17T18:01:23.302 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:01:23.399 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:23.422 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:23.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:23.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:23.524 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:23.545 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:23.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:23.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:23.640 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:23 smithi136 ceph-mon[109861]: pgmap v2414: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 16701/198525 objects degraded (8.413%); 3555/198525 objects misplaced (1.791%); 1.0 MiB/s, 16 objects/s recovering 2024-08-17T18:01:23.641 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:23 smithi136 ceph-mon[109861]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T18:01:23.641 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:23 smithi136 ceph-mon[109861]: osd.2 [v2:172.21.15.136:6818/809598197,v1:172.21.15.136:6819/809598197] boot 2024-08-17T18:01:23.641 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:23 smithi136 ceph-mon[109861]: osdmap e1399: 8 total, 8 up, 7 in 2024-08-17T18:01:23.641 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:23 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-17T18:01:23.641 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:23 smithi136 ceph-mon[109861]: osdmap e1400: 8 total, 8 up, 7 in 2024-08-17T18:01:23.641 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:23 smithi136 ceph-mon[115078]: pgmap v2414: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 16701/198525 objects degraded (8.413%); 3555/198525 objects misplaced (1.791%); 1.0 MiB/s, 16 objects/s recovering 2024-08-17T18:01:23.641 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:23 smithi136 ceph-mon[115078]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T18:01:23.641 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:23 smithi136 ceph-mon[115078]: osd.2 [v2:172.21.15.136:6818/809598197,v1:172.21.15.136:6819/809598197] boot 2024-08-17T18:01:23.641 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:23 smithi136 ceph-mon[115078]: osdmap e1399: 8 total, 8 up, 7 in 2024-08-17T18:01:23.641 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:23 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-17T18:01:23.641 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:23 smithi136 ceph-mon[115078]: osdmap e1400: 8 total, 8 up, 7 in 2024-08-17T18:01:23.647 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:23.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:23.671 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:23.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:23.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:23 smithi165 ceph-mon[105517]: pgmap v2414: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 16701/198525 objects degraded (8.413%); 3555/198525 objects misplaced (1.791%); 1.0 MiB/s, 16 objects/s recovering 2024-08-17T18:01:23.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:23 smithi165 ceph-mon[105517]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T18:01:23.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:23 smithi165 ceph-mon[105517]: osd.2 [v2:172.21.15.136:6818/809598197,v1:172.21.15.136:6819/809598197] boot 2024-08-17T18:01:23.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:23 smithi165 ceph-mon[105517]: osdmap e1399: 8 total, 8 up, 7 in 2024-08-17T18:01:23.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:23 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-17T18:01:23.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:23 smithi165 ceph-mon[105517]: osdmap e1400: 8 total, 8 up, 7 in 2024-08-17T18:01:23.772 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:23.794 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:23.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:23.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:23.896 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:23.919 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:23.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:23.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:24.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:24.043 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:24.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:24.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:24.145 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:24.166 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:24.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:24.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:24.271 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:24.274 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-17T18:01:24.275 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-08-17T18:01:24.292 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:24.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:24.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:24.394 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:24.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:24.416 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:24.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:24.517 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:24.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:24.550 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:24.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:24.605 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:24 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 16701/198525 objects degraded (8.413%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:01:24.606 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:24 smithi136 ceph-mon[109861]: osdmap e1401: 8 total, 8 up, 7 in 2024-08-17T18:01:24.606 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:24 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 16701/198525 objects degraded (8.413%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:01:24.606 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:24 smithi136 ceph-mon[115078]: osdmap e1401: 8 total, 8 up, 7 in 2024-08-17T18:01:24.639 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-17T18:01:24.651 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:24.715 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:24.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:24.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:24.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:24 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 16701/198525 objects degraded (8.413%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:01:24.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:24 smithi165 ceph-mon[105517]: osdmap e1401: 8 total, 8 up, 7 in 2024-08-17T18:01:24.817 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:24.847 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:24.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:24.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:24.949 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:24.973 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:24.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:24.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:25.075 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:25.097 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:25.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:25.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:25.199 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:25.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:25.223 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:25.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:25.324 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:25.347 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:25.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:25.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:25.402 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:01:25.449 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:25.470 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:25.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:25.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:25.572 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:25.594 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:25.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:25.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:25.640 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:25 smithi136 ceph-mon[109861]: pgmap v2417: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 16493/197322 objects degraded (8.358%); 3555/197322 objects misplaced (1.802%); 874 KiB/s, 13 objects/s recovering 2024-08-17T18:01:25.640 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:25 smithi136 ceph-mon[109861]: 54.3 scrub starts 2024-08-17T18:01:25.640 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:25 smithi136 ceph-mon[109861]: 54.3 scrub ok 2024-08-17T18:01:25.640 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:25 smithi136 ceph-mon[109861]: 54.8 scrub starts 2024-08-17T18:01:25.640 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:25 smithi136 ceph-mon[109861]: 54.8 scrub ok 2024-08-17T18:01:25.641 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:25 smithi136 ceph-mon[115078]: pgmap v2417: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 16493/197322 objects degraded (8.358%); 3555/197322 objects misplaced (1.802%); 874 KiB/s, 13 objects/s recovering 2024-08-17T18:01:25.641 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:25 smithi136 ceph-mon[115078]: 54.3 scrub starts 2024-08-17T18:01:25.641 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:25 smithi136 ceph-mon[115078]: 54.3 scrub ok 2024-08-17T18:01:25.641 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:25 smithi136 ceph-mon[115078]: 54.8 scrub starts 2024-08-17T18:01:25.641 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:25 smithi136 ceph-mon[115078]: 54.8 scrub ok 2024-08-17T18:01:25.697 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:25.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:25.719 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:25.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:25 smithi165 ceph-mon[105517]: pgmap v2417: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 16493/197322 objects degraded (8.358%); 3555/197322 objects misplaced (1.802%); 874 KiB/s, 13 objects/s recovering 2024-08-17T18:01:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:25 smithi165 ceph-mon[105517]: 54.3 scrub starts 2024-08-17T18:01:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:25 smithi165 ceph-mon[105517]: 54.3 scrub ok 2024-08-17T18:01:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:25 smithi165 ceph-mon[105517]: 54.8 scrub starts 2024-08-17T18:01:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:25 smithi165 ceph-mon[105517]: 54.8 scrub ok 2024-08-17T18:01:25.821 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:25.842 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:25.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:25.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:25.944 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:25.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:25.967 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:25.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:26.068 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:26.090 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:26.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:26.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:26.192 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:26.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:26.213 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:26.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:26.314 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:26.336 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:26.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:26.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:26.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:26.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:26.460 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:26.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:26.562 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:26.584 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:26.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:26.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:26.686 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:26.702 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:01:26.713 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:26.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:26.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:26.815 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:26.837 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:26.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:26.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:26.939 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:26.961 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:26.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:26.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:27.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:27.088 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:27.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:27.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:27.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:27.213 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:27.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:27.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:27.315 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:27.337 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:27.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:27.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:27.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:27.462 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:27.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:27.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:27.509 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:01:27.564 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:27.604 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:27.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:27.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:27.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:27.742 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:27.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:27.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:27 smithi165 ceph-mon[105517]: pgmap v2419: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 16493/197322 objects degraded (8.358%); 3555/197322 objects misplaced (1.802%); 1.1 MiB/s, 18 objects/s recovering 2024-08-17T18:01:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:27 smithi165 ceph-mon[105517]: osdmap e1402: 8 total, 8 up, 7 in 2024-08-17T18:01:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:27 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2801595876' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:01:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:27 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:01:27.844 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:27.871 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:27.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:27.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:27.901 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:27 smithi136 ceph-mon[109861]: pgmap v2419: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 16493/197322 objects degraded (8.358%); 3555/197322 objects misplaced (1.802%); 1.1 MiB/s, 18 objects/s recovering 2024-08-17T18:01:27.901 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:27 smithi136 ceph-mon[109861]: osdmap e1402: 8 total, 8 up, 7 in 2024-08-17T18:01:27.901 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:27 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2801595876' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:01:27.901 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:27 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:01:27.902 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:27 smithi136 ceph-mon[115078]: pgmap v2419: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 21 active+undersized, 9 active+undersized+degraded, 67 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 16493/197322 objects degraded (8.358%); 3555/197322 objects misplaced (1.802%); 1.1 MiB/s, 18 objects/s recovering 2024-08-17T18:01:27.902 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:27 smithi136 ceph-mon[115078]: osdmap e1402: 8 total, 8 up, 7 in 2024-08-17T18:01:27.902 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:27 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2801595876' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:01:27.902 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:27 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:01:27.902 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:01:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:01:27] "GET /metrics HTTP/1.1" 200 36647 "" "Prometheus/2.51.0" 2024-08-17T18:01:27.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:28.003 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:28.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:28.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:28.105 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:28.128 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:28.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:28.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:28.230 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:28.263 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:28.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:28.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:28.365 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:28.391 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:28.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:28.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:28.493 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:28.521 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:28.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:28.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:28.623 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:28.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:28.654 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:28.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:28.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:28 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:01:28.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:28 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:01:28.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:28 smithi136 ceph-mon[115078]: osdmap e1403: 8 total, 8 up, 7 in 2024-08-17T18:01:28.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:28 smithi136 ceph-mon[115078]: 54.11 scrub starts 2024-08-17T18:01:28.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:28 smithi136 ceph-mon[115078]: osdmap e1404: 8 total, 8 up, 7 in 2024-08-17T18:01:28.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:28 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:01:28.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:28 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:01:28.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:28 smithi136 ceph-mon[109861]: osdmap e1403: 8 total, 8 up, 7 in 2024-08-17T18:01:28.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:28 smithi136 ceph-mon[109861]: 54.11 scrub starts 2024-08-17T18:01:28.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:28 smithi136 ceph-mon[109861]: osdmap e1404: 8 total, 8 up, 7 in 2024-08-17T18:01:28.756 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:28.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:28.788 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:28.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:28.889 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:28.914 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:28.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:28.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:28 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:01:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:28 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:01:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:28 smithi165 ceph-mon[105517]: osdmap e1403: 8 total, 8 up, 7 in 2024-08-17T18:01:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:28 smithi165 ceph-mon[105517]: 54.11 scrub starts 2024-08-17T18:01:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:28 smithi165 ceph-mon[105517]: osdmap e1404: 8 total, 8 up, 7 in 2024-08-17T18:01:29.017 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:29.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:29.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:29.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:29.145 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:29.172 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:29.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:29.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:29.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:29.275 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-17T18:01:29.305 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:29.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:29.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:29.407 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:29.431 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:29.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:29.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:29.533 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:29.558 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:29.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:29.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:29.589 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:29 smithi136 ceph-mon[109861]: pgmap v2422: 99 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 4.0 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 5923/194523 objects degraded (3.045%); 243 KiB/s, 242 objects/s recovering 2024-08-17T18:01:29.589 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:29 smithi136 ceph-mon[109861]: 54.11 scrub ok 2024-08-17T18:01:29.590 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:29 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5923/194523 objects degraded (3.045%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:01:29.590 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:29 smithi136 ceph-mon[115078]: pgmap v2422: 99 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 4.0 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 5923/194523 objects degraded (3.045%); 243 KiB/s, 242 objects/s recovering 2024-08-17T18:01:29.590 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:29 smithi136 ceph-mon[115078]: 54.11 scrub ok 2024-08-17T18:01:29.590 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:29 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5923/194523 objects degraded (3.045%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:01:29.661 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:29.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:29.696 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:29.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:29.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:29.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:29.832 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:29.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:29.933 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:29.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:29.973 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:29.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:29 smithi165 ceph-mon[105517]: pgmap v2422: 99 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 4.0 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 5923/194523 objects degraded (3.045%); 243 KiB/s, 242 objects/s recovering 2024-08-17T18:01:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:29 smithi165 ceph-mon[105517]: 54.11 scrub ok 2024-08-17T18:01:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:29 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5923/194523 objects degraded (3.045%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:01:30.071 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:01:30.072 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:01:30.074 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:30.097 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:01:30.098 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:01:30.100 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:30.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:30.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:30.121 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:01:30.121 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-08-17T18:01:30.127 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:01:30.128 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T18:01:30.128 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-17T18:01:30.128 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-17T18:01:30.128 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:01:30.146 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:01:30.146 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:01:30.166 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:01:30.167 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:01:30.190 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:01:30.190 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:01:30.201 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:30.208 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:01:30.208 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:01:30.225 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:30.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:30.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:30.227 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:01:30.228 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:01:30.327 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:30.328 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:01:30.353 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:30.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:30.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:30.455 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:30.480 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:30.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:30.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:30.582 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:30.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:30.607 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:30.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:30.708 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:30.735 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:30.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:30.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:30.837 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:30.862 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:30.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:30.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:30.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:30 smithi136 ceph-mon[109861]: osdmap e1405: 8 total, 8 up, 7 in 2024-08-17T18:01:30.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:30 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2295362834' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:01:30.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:30 smithi136 ceph-mon[115078]: osdmap e1405: 8 total, 8 up, 7 in 2024-08-17T18:01:30.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:30 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2295362834' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:01:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:30 smithi165 ceph-mon[105517]: osdmap e1405: 8 total, 8 up, 7 in 2024-08-17T18:01:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:30 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2295362834' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:01:31.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:31.069 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:31.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:31.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:31.171 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:31.197 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:31.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:31.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:31.300 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:31.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:31.335 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:31.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:31.436 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:31.463 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:31.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:31.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:31.564 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:31.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:31.591 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:31.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:31.692 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:31.728 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:31.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:31.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:31.730 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-17T18:01:31.747 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:01:31.830 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:31.856 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:31.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:31.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:31.880 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:31 smithi136 ceph-mon[109861]: pgmap v2425: 99 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 4.0 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 5923/194523 objects degraded (3.045%); 272 KiB/s, 271 objects/s recovering 2024-08-17T18:01:31.881 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:31 smithi136 ceph-mon[115078]: pgmap v2425: 99 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 4.0 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 5923/194523 objects degraded (3.045%); 272 KiB/s, 271 objects/s recovering 2024-08-17T18:01:31.958 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:31.986 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:31.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:31.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:31 smithi165 ceph-mon[105517]: pgmap v2425: 99 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 4.0 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 5923/194523 objects degraded (3.045%); 272 KiB/s, 271 objects/s recovering 2024-08-17T18:01:32.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:32.116 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:32.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:32.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:32.218 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:32.243 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:32.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:32.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:32.345 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:32.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:32.371 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:32.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:32.473 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:32.500 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:32.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:32.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:32.602 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:32.626 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:32.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:32.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:32.728 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:32.742 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:01:32.742 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:01:32.757 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:32.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:32.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:32.852 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:32 smithi136 ceph-mon[109861]: 54.0 scrub starts 2024-08-17T18:01:32.852 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:32 smithi136 ceph-mon[115078]: 54.0 scrub starts 2024-08-17T18:01:32.860 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:32.882 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:32.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:32.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:32.984 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:32 smithi165 ceph-mon[105517]: 54.0 scrub starts 2024-08-17T18:01:33.011 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:33.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:33.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:33.113 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:33.124 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:01:33.149 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:33.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:33.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:33.251 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:33.282 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:33.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:33.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:33.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:33.418 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:33.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:33.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:33.520 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:33.549 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:33.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:33.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:33.650 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:33.673 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:33.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:33.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:33.691 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:33 smithi136 ceph-mon[109861]: pgmap v2426: 99 pgs: 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 5923/188571 objects degraded (3.141%) 2024-08-17T18:01:33.691 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:33 smithi136 ceph-mon[115078]: pgmap v2426: 99 pgs: 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 5923/188571 objects degraded (3.141%) 2024-08-17T18:01:33.775 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:33.801 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:33.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:33.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:33.903 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:33.930 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:33.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:33.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:33 smithi165 ceph-mon[105517]: pgmap v2426: 99 pgs: 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 5923/188571 objects degraded (3.141%) 2024-08-17T18:01:34.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:34.055 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:34.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:34.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:34.157 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:34.178 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:34.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:34.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:34.280 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:34.301 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:34.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:34.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:34.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:34.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:34.425 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:34.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:34.526 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:34.547 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:34.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:34.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:34.649 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:34.679 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:34.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:34.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:34.781 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:34.798 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:34 smithi136 ceph-mon[109861]: osdmap e1406: 8 total, 8 up, 7 in 2024-08-17T18:01:34.799 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:34 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5923/188571 objects degraded (3.141%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:01:34.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:34 smithi136 ceph-mon[115078]: osdmap e1406: 8 total, 8 up, 7 in 2024-08-17T18:01:34.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:34 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5923/188571 objects degraded (3.141%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:01:34.808 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:34.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:34.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:34.909 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:34.935 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:34.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:34.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:34 smithi165 ceph-mon[105517]: osdmap e1406: 8 total, 8 up, 7 in 2024-08-17T18:01:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:34 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5923/188571 objects degraded (3.141%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:01:35.032 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:01:35.037 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:35.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:35.064 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:35.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:35.165 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:35.193 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:35.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:35.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:35.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:35.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:35.320 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:35.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:35.421 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:35.447 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:35.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:35.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:35.548 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:35.569 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:01:35.574 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:35.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:35.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:35.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:35.724 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:35.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:35.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:35.790 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:35 smithi136 ceph-mon[109861]: pgmap v2428: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 4668/186360 objects degraded (2.505%) 2024-08-17T18:01:35.790 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:35 smithi136 ceph-mon[109861]: 54.18 deep-scrub starts 2024-08-17T18:01:35.790 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:35 smithi136 ceph-mon[109861]: osdmap e1407: 8 total, 8 up, 7 in 2024-08-17T18:01:35.790 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:35 smithi136 ceph-mon[109861]: 54.0 scrub starts 2024-08-17T18:01:35.790 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:35 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/147583998' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:01:35.790 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:35 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:01:35.791 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:35 smithi136 ceph-mon[115078]: pgmap v2428: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 4668/186360 objects degraded (2.505%) 2024-08-17T18:01:35.791 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:35 smithi136 ceph-mon[115078]: 54.18 deep-scrub starts 2024-08-17T18:01:35.791 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:35 smithi136 ceph-mon[115078]: osdmap e1407: 8 total, 8 up, 7 in 2024-08-17T18:01:35.791 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:35 smithi136 ceph-mon[115078]: 54.0 scrub starts 2024-08-17T18:01:35.791 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:35 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/147583998' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:01:35.791 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:35 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:01:35.826 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:35.856 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:35.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:35.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:35.958 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:35.994 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:35.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:35.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:35 smithi165 ceph-mon[105517]: pgmap v2428: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 4668/186360 objects degraded (2.505%) 2024-08-17T18:01:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:35 smithi165 ceph-mon[105517]: 54.18 deep-scrub starts 2024-08-17T18:01:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:35 smithi165 ceph-mon[105517]: osdmap e1407: 8 total, 8 up, 7 in 2024-08-17T18:01:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:35 smithi165 ceph-mon[105517]: 54.0 scrub starts 2024-08-17T18:01:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:35 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/147583998' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:01:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:35 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:01:36.096 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:36.123 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:36.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:36.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:36.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:36.259 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:36.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:36.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:36.361 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:36.393 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:36.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:36.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:36.495 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:36.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:36.527 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:36.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:36.628 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:36.651 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:36.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:36.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:36.753 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:36.778 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:36.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:36.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:36.880 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:36.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:36.910 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:36.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:36.912 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:36 smithi136 ceph-mon[109861]: 54.0 scrub ok 2024-08-17T18:01:36.912 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:36 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:01:36.912 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:36 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:01:36.912 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:36 smithi136 ceph-mon[109861]: osdmap e1408: 8 total, 8 up, 7 in 2024-08-17T18:01:36.912 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:01:36.912 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:01:36.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:36 smithi136 ceph-mon[115078]: 54.0 scrub ok 2024-08-17T18:01:36.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:36 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:01:36.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:36 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:01:36.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:36 smithi136 ceph-mon[115078]: osdmap e1408: 8 total, 8 up, 7 in 2024-08-17T18:01:36.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:01:36.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:01:36.913 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:36 smithi165 ceph-mon[105517]: 54.0 scrub ok 2024-08-17T18:01:36.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:36 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:01:36.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:36 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:01:36.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:36 smithi165 ceph-mon[105517]: osdmap e1408: 8 total, 8 up, 7 in 2024-08-17T18:01:36.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:01:36.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:01:37.011 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:37.042 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:37.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:37.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:37.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:37.170 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:37.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:37.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:37.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:37.294 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:37.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:37.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:37.396 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:37.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:37.432 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:37.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:37.533 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:37.565 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:37.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:37.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:37.567 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:01:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:01:37] "GET /metrics HTTP/1.1" 200 36623 "" "Prometheus/2.51.0" 2024-08-17T18:01:37.666 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:37.687 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:37.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:37.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:37 smithi165 ceph-mon[105517]: pgmap v2431: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 4668/186360 objects degraded (2.505%) 2024-08-17T18:01:37.789 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:37.813 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:37.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:37.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:37.915 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:37.940 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:37.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:37.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:37.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:37 smithi136 ceph-mon[109861]: pgmap v2431: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 4668/186360 objects degraded (2.505%) 2024-08-17T18:01:37.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:37 smithi136 ceph-mon[115078]: pgmap v2431: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 4668/186360 objects degraded (2.505%) 2024-08-17T18:01:38.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:38.069 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:38.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:38.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:38.171 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:38.194 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:38.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:38.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:38.296 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:38.324 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:38.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:38.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:38.426 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:38.453 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:38.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:38.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:38.555 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:38.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:38.579 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:38.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:38.679 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:38.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:38.717 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:38.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:38.818 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:38.847 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:38.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:38.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:38.927 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:38 smithi136 ceph-mon[109861]: osdmap e1409: 8 total, 8 up, 7 in 2024-08-17T18:01:38.928 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:38 smithi136 ceph-mon[115078]: osdmap e1409: 8 total, 8 up, 7 in 2024-08-17T18:01:38.949 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:38.988 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:38.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:38.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:39.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:38 smithi165 ceph-mon[105517]: osdmap e1409: 8 total, 8 up, 7 in 2024-08-17T18:01:39.091 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:39.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:39.123 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:39.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:39.224 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:39.249 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:39.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:39.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:39.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:39.383 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:39.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:39.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:39.485 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:39.513 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:39.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:39.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:39.615 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:39.652 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:39.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:39.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:39.679 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:01:39.679 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:01:39.753 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:39.788 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:39.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:39.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:39.868 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:39 smithi136 ceph-mon[109861]: pgmap v2433: 99 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 2972/177432 objects degraded (1.675%); 1.3 MiB/s, 272 objects/s recovering 2024-08-17T18:01:39.868 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:39 smithi136 ceph-mon[109861]: osdmap e1410: 8 total, 8 up, 7 in 2024-08-17T18:01:39.868 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:39 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 2972/177432 objects degraded (1.675%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:01:39.868 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:39 smithi136 ceph-mon[115078]: pgmap v2433: 99 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 2972/177432 objects degraded (1.675%); 1.3 MiB/s, 272 objects/s recovering 2024-08-17T18:01:39.869 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:39 smithi136 ceph-mon[115078]: osdmap e1410: 8 total, 8 up, 7 in 2024-08-17T18:01:39.869 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:39 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 2972/177432 objects degraded (1.675%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:01:39.890 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:39.904 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:01:39.904 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:01:39.918 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:39.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:39.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:39.931 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:01:39.931 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:01:39.953 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:01:39.953 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:01:39.976 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:01:39.976 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:01:39.999 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:01:39.999 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:01:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:39 smithi165 ceph-mon[105517]: pgmap v2433: 99 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 2972/177432 objects degraded (1.675%); 1.3 MiB/s, 272 objects/s recovering 2024-08-17T18:01:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:39 smithi165 ceph-mon[105517]: osdmap e1410: 8 total, 8 up, 7 in 2024-08-17T18:01:40.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:39 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 2972/177432 objects degraded (1.675%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:01:40.019 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:01:40.019 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:01:40.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:40.037 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:01:40.037 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:01:40.045 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:40.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:40.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:40.056 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:01:40.056 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:01:40.147 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:40.176 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:40.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:40.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:40.278 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:40.303 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:01:40.303 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-17T18:01:40.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-17T18:01:40.303 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:01:40.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.303 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35091.0:239322 53.a 53:578761b4:::benchmark_data_smithi136_235573_object115036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1350)", 2024-08-17T18:01:40.303 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:57:09.250286+0000", 2024-08-17T18:01:40.303 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 271.04954912699998, 2024-08-17T18:01:40.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.53959296199999995, 2024-08-17T18:01:40.304 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:01:40.304 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:01:40.304 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:01:40.304 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:01:40.304 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35091", 2024-08-17T18:01:40.304 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/1137886352", 2024-08-17T18:01:40.304 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 239322 2024-08-17T18:01:40.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.304 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:01:40.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:01:40.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.250286+0000", 2024-08-17T18:01:40.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:01:40.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.250286+0000", 2024-08-17T18:01:40.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:01:40.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.250288+0000", 2024-08-17T18:01:40.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8780000000000001e-06 2024-08-17T18:01:40.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:01:40.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.250294+0000", 2024-08-17T18:01:40.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3629999999999999e-06 2024-08-17T18:01:40.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:01:40.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.250311+0000", 2024-08-17T18:01:40.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7132000000000001e-05 2024-08-17T18:01:40.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:01:40.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.250319+0000", 2024-08-17T18:01:40.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4649999999999994e-06 2024-08-17T18:01:40.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.250485+0000", 2024-08-17T18:01:40.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016534899999999999 2024-08-17T18:01:40.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:01:40.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.250789+0000", 2024-08-17T18:01:40.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000304223 2024-08-17T18:01:40.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:01:40.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.251126+0000", 2024-08-17T18:01:40.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000336466 2024-08-17T18:01:40.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:01:40.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.256481+0000", 2024-08-17T18:01:40.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0053555640000000002 2024-08-17T18:01:40.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.789721+0000", 2024-08-17T18:01:40.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.53323979799999999 2024-08-17T18:01:40.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.789788+0000", 2024-08-17T18:01:40.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6706999999999997e-05 2024-08-17T18:01:40.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:01:40.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.789840+0000", 2024-08-17T18:01:40.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2133999999999999e-05 2024-08-17T18:01:40.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:01:40.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.789879+0000", 2024-08-17T18:01:40.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8883e-05 2024-08-17T18:01:40.310 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.310 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:01:40.310 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.310 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.35091.0:239319 53.e e1350/1303 53:71154b78:::benchmark_data_smithi136_235573_object115033:head v 1350'14914, mlcod=1350'14912)", 2024-08-17T18:01:40.310 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:57:09.254480+0000", 2024-08-17T18:01:40.310 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 271.04535464499997, 2024-08-17T18:01:40.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.53549093800000003, 2024-08-17T18:01:40.310 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:01:40.310 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:01:40.310 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:01:40.310 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:01:40.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:01:40.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.254480+0000", 2024-08-17T18:01:40.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:01:40.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.254480+0000", 2024-08-17T18:01:40.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:01:40.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.254481+0000", 2024-08-17T18:01:40.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4900000000000005e-07 2024-08-17T18:01:40.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:01:40.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.254488+0000", 2024-08-17T18:01:40.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0090000000000001e-06 2024-08-17T18:01:40.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:01:40.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.254498+0000", 2024-08-17T18:01:40.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0454e-05 2024-08-17T18:01:40.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:01:40.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.254505+0000", 2024-08-17T18:01:40.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3450000000000002e-06 2024-08-17T18:01:40.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.255916+0000", 2024-08-17T18:01:40.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014116409999999999 2024-08-17T18:01:40.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:01:40.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.255936+0000", 2024-08-17T18:01:40.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9865999999999999e-05 2024-08-17T18:01:40.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:01:40.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.789909+0000", 2024-08-17T18:01:40.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.53397283799999995 2024-08-17T18:01:40.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:01:40.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.789971+0000", 2024-08-17T18:01:40.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2036e-05 2024-08-17T18:01:40.314 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.314 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:01:40.314 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.315 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.35091.0:239325 53.e e1350/1303 53:74094b29:::benchmark_data_smithi136_235573_object115039:head v 1350'14915, mlcod=1350'14912)", 2024-08-17T18:01:40.315 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:57:09.259988+0000", 2024-08-17T18:01:40.315 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 271.039846775, 2024-08-17T18:01:40.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.53000947399999998, 2024-08-17T18:01:40.315 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:01:40.315 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:01:40.315 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:01:40.315 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:01:40.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:01:40.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.259988+0000", 2024-08-17T18:01:40.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:01:40.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.259988+0000", 2024-08-17T18:01:40.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:01:40.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.259990+0000", 2024-08-17T18:01:40.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.505e-06 2024-08-17T18:01:40.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:01:40.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.260001+0000", 2024-08-17T18:01:40.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1049e-05 2024-08-17T18:01:40.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:01:40.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.260015+0000", 2024-08-17T18:01:40.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4559000000000001e-05 2024-08-17T18:01:40.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:01:40.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.260025+0000", 2024-08-17T18:01:40.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9939999999999995e-06 2024-08-17T18:01:40.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.260927+0000", 2024-08-17T18:01:40.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00090168699999999997 2024-08-17T18:01:40.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:01:40.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.260946+0000", 2024-08-17T18:01:40.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9142000000000001e-05 2024-08-17T18:01:40.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:01:40.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.789974+0000", 2024-08-17T18:01:40.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.52902803899999995 2024-08-17T18:01:40.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:01:40.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.789997+0000", 2024-08-17T18:01:40.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3499e-05 2024-08-17T18:01:40.321 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.321 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:01:40.321 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.321 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.35091.0:239321 53.14 e1351/1343 53:2dd71a88:::benchmark_data_smithi136_235573_object115035:head v 1351'9247, mlcod=1331'6033)", 2024-08-17T18:01:40.322 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:57:09.265744+0000", 2024-08-17T18:01:40.322 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 271.03409119899999, 2024-08-17T18:01:40.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.53070054799999999, 2024-08-17T18:01:40.322 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:01:40.322 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:01:40.322 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:01:40.322 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:01:40.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:01:40.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.265744+0000", 2024-08-17T18:01:40.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:01:40.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.265744+0000", 2024-08-17T18:01:40.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:01:40.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.265744+0000", 2024-08-17T18:01:40.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0499999999999999e-07 2024-08-17T18:01:40.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:01:40.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.265749+0000", 2024-08-17T18:01:40.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3940000000000004e-06 2024-08-17T18:01:40.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:01:40.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.265760+0000", 2024-08-17T18:01:40.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0794e-05 2024-08-17T18:01:40.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:01:40.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.265768+0000", 2024-08-17T18:01:40.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.43e-06 2024-08-17T18:01:40.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.789784+0000", 2024-08-17T18:01:40.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.52401636200000001 2024-08-17T18:01:40.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:01:40.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.789808+0000", 2024-08-17T18:01:40.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3966000000000001e-05 2024-08-17T18:01:40.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:01:40.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.796413+0000", 2024-08-17T18:01:40.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0066049059999999998 2024-08-17T18:01:40.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:01:40.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:57:09.796444+0000", 2024-08-17T18:01:40.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1390999999999998e-05 2024-08-17T18:01:40.327 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.327 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:01:40.327 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.327 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104426 54.10 54:0c08f48e:::benchmark_data_smithi136_273969_object104425:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:01:40.327 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.460498+0000", 2024-08-17T18:01:40.328 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 115.839336586, 2024-08-17T18:01:40.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.15692383, 2024-08-17T18:01:40.328 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:01:40.328 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:01:40.328 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:01:40.328 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:01:40.328 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:01:40.328 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:01:40.328 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104426 2024-08-17T18:01:40.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.329 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:01:40.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:01:40.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460498+0000", 2024-08-17T18:01:40.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:01:40.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460498+0000", 2024-08-17T18:01:40.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:01:40.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460500+0000", 2024-08-17T18:01:40.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.562e-06 2024-08-17T18:01:40.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:01:40.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460526+0000", 2024-08-17T18:01:40.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6466e-05 2024-08-17T18:01:40.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:01:40.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460535+0000", 2024-08-17T18:01:40.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0189999999999995e-06 2024-08-17T18:01:40.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:01:40.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460542+0000", 2024-08-17T18:01:40.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8730000000000001e-06 2024-08-17T18:01:40.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460590+0000", 2024-08-17T18:01:40.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7673999999999998e-05 2024-08-17T18:01:40.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-08-17T18:01:40.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460595+0000", 2024-08-17T18:01:40.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0130000000000001e-06 2024-08-17T18:01:40.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.462127+0000", 2024-08-17T18:01:40.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015317869999999999 2024-08-17T18:01:40.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:01:40.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.462136+0000", 2024-08-17T18:01:40.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5820000000000008e-06 2024-08-17T18:01:40.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.582918+0000", 2024-08-17T18:01:40.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1207821739999999 2024-08-17T18:01:40.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:01:40.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.583070+0000", 2024-08-17T18:01:40.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015153000000000001 2024-08-17T18:01:40.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:01:40.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.583305+0000", 2024-08-17T18:01:40.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023529200000000001 2024-08-17T18:01:40.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.611591+0000", 2024-08-17T18:01:40.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.028286170999999999 2024-08-17T18:01:40.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.611639+0000", 2024-08-17T18:01:40.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7868000000000003e-05 2024-08-17T18:01:40.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:01:40.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.615716+0000", 2024-08-17T18:01:40.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040763889999999997 2024-08-17T18:01:40.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:01:40.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.615749+0000", 2024-08-17T18:01:40.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3045e-05 2024-08-17T18:01:40.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:01:40.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.617422+0000", 2024-08-17T18:01:40.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016733850000000001 2024-08-17T18:01:40.335 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.335 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:01:40.335 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.336 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104434 54.10 54:0a03e580:::benchmark_data_smithi136_273969_object104433:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:01:40.336 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.460552+0000", 2024-08-17T18:01:40.336 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 115.839282399, 2024-08-17T18:01:40.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.17111078, 2024-08-17T18:01:40.336 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:01:40.336 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:01:40.336 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:01:40.336 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:01:40.336 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:01:40.337 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:01:40.337 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104434 2024-08-17T18:01:40.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.337 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:01:40.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:01:40.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460552+0000", 2024-08-17T18:01:40.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:01:40.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460552+0000", 2024-08-17T18:01:40.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:01:40.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460553+0000", 2024-08-17T18:01:40.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.82e-07 2024-08-17T18:01:40.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:01:40.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460577+0000", 2024-08-17T18:01:40.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4414999999999999e-05 2024-08-17T18:01:40.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.339 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:01:40.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460584+0000", 2024-08-17T18:01:40.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7100000000000001e-06 2024-08-17T18:01:40.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.339 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:01:40.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460588+0000", 2024-08-17T18:01:40.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8449999999999996e-06 2024-08-17T18:01:40.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.339 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460647+0000", 2024-08-17T18:01:40.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9329000000000001e-05 2024-08-17T18:01:40.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:01:40.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460652+0000", 2024-08-17T18:01:40.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5499999999999996e-06 2024-08-17T18:01:40.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.462838+0000", 2024-08-17T18:01:40.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021861739999999999 2024-08-17T18:01:40.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:01:40.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.462843+0000", 2024-08-17T18:01:40.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0309999999999998e-06 2024-08-17T18:01:40.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.341 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.341 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.611677+0000", 2024-08-17T18:01:40.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.148834299 2024-08-17T18:01:40.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.341 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:01:40.341 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.611880+0000", 2024-08-17T18:01:40.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020264600000000001 2024-08-17T18:01:40.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:01:40.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.612660+0000", 2024-08-17T18:01:40.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00077971400000000004 2024-08-17T18:01:40.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.617632+0000", 2024-08-17T18:01:40.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0049720750000000003 2024-08-17T18:01:40.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:01:40.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.617649+0000", 2024-08-17T18:01:40.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7370000000000001e-05 2024-08-17T18:01:40.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.343 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.343 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.630089+0000", 2024-08-17T18:01:40.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012439725 2024-08-17T18:01:40.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.343 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:01:40.343 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.630128+0000", 2024-08-17T18:01:40.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9240999999999999e-05 2024-08-17T18:01:40.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.343 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:01:40.344 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.631663+0000", 2024-08-17T18:01:40.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001535174 2024-08-17T18:01:40.344 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.344 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:01:40.344 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.344 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104447 54.10 54:083450d8:::benchmark_data_smithi136_273969_object104446:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:01:40.344 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.460594+0000", 2024-08-17T18:01:40.344 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 115.839240449, 2024-08-17T18:01:40.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1723181149999999, 2024-08-17T18:01:40.344 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:01:40.344 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:01:40.344 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:01:40.345 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:01:40.345 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:01:40.345 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:01:40.345 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104447 2024-08-17T18:01:40.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.345 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:01:40.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:01:40.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460594+0000", 2024-08-17T18:01:40.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:01:40.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460594+0000", 2024-08-17T18:01:40.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.346 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:01:40.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460595+0000", 2024-08-17T18:01:40.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8100000000000003e-07 2024-08-17T18:01:40.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.346 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:01:40.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460631+0000", 2024-08-17T18:01:40.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6387999999999998e-05 2024-08-17T18:01:40.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:01:40.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460639+0000", 2024-08-17T18:01:40.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3130000000000001e-06 2024-08-17T18:01:40.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:01:40.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460644+0000", 2024-08-17T18:01:40.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6840000000000001e-06 2024-08-17T18:01:40.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460675+0000", 2024-08-17T18:01:40.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1220999999999998e-05 2024-08-17T18:01:40.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.348 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:01:40.348 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460680+0000", 2024-08-17T18:01:40.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9099999999999996e-06 2024-08-17T18:01:40.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.462873+0000", 2024-08-17T18:01:40.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021922389999999999 2024-08-17T18:01:40.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:01:40.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.462880+0000", 2024-08-17T18:01:40.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2259999999999999e-06 2024-08-17T18:01:40.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.613949+0000", 2024-08-17T18:01:40.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.151069353 2024-08-17T18:01:40.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:01:40.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.614143+0000", 2024-08-17T18:01:40.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019340500000000001 2024-08-17T18:01:40.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:01:40.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.614467+0000", 2024-08-17T18:01:40.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00032422700000000001 2024-08-17T18:01:40.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:01:40.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.617653+0000", 2024-08-17T18:01:40.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003186024 2024-08-17T18:01:40.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.629973+0000", 2024-08-17T18:01:40.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012319719 2024-08-17T18:01:40.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.631722+0000", 2024-08-17T18:01:40.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001749404 2024-08-17T18:01:40.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:01:40.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.631759+0000", 2024-08-17T18:01:40.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6616999999999999e-05 2024-08-17T18:01:40.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:01:40.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.632912+0000", 2024-08-17T18:01:40.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011539040000000001 2024-08-17T18:01:40.353 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.353 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:01:40.353 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.354 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104557 54.10 54:08b8b66a:::benchmark_data_smithi136_273969_object104556:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:01:40.354 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.460654+0000", 2024-08-17T18:01:40.354 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 115.839180473, 2024-08-17T18:01:40.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1798764369999999, 2024-08-17T18:01:40.354 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:01:40.354 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:01:40.354 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:01:40.354 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:01:40.354 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:01:40.354 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:01:40.354 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104557 2024-08-17T18:01:40.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.355 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:01:40.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:01:40.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460654+0000", 2024-08-17T18:01:40.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:01:40.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460654+0000", 2024-08-17T18:01:40.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:01:40.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460655+0000", 2024-08-17T18:01:40.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0900000000000002e-07 2024-08-17T18:01:40.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:01:40.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460687+0000", 2024-08-17T18:01:40.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1955999999999997e-05 2024-08-17T18:01:40.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:01:40.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460709+0000", 2024-08-17T18:01:40.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1759999999999998e-05 2024-08-17T18:01:40.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:01:40.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460714+0000", 2024-08-17T18:01:40.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8429999999999998e-06 2024-08-17T18:01:40.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460752+0000", 2024-08-17T18:01:40.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8022000000000003e-05 2024-08-17T18:01:40.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:01:40.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460757+0000", 2024-08-17T18:01:40.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8040000000000001e-06 2024-08-17T18:01:40.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.462914+0000", 2024-08-17T18:01:40.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021564689999999998 2024-08-17T18:01:40.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:01:40.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.462919+0000", 2024-08-17T18:01:40.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9660000000000004e-06 2024-08-17T18:01:40.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.617674+0000", 2024-08-17T18:01:40.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1547550950000001 2024-08-17T18:01:40.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:01:40.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.617834+0000", 2024-08-17T18:01:40.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000159798 2024-08-17T18:01:40.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:01:40.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.618189+0000", 2024-08-17T18:01:40.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00035543499999999997 2024-08-17T18:01:40.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:01:40.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.621993+0000", 2024-08-17T18:01:40.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003803716 2024-08-17T18:01:40.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.630024+0000", 2024-08-17T18:01:40.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0080315070000000002 2024-08-17T18:01:40.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.638621+0000", 2024-08-17T18:01:40.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.008596467 2024-08-17T18:01:40.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:01:40.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.638679+0000", 2024-08-17T18:01:40.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7896999999999998e-05 2024-08-17T18:01:40.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:01:40.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.640531+0000", 2024-08-17T18:01:40.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001852193 2024-08-17T18:01:40.362 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.362 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:01:40.362 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.363 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104561 54.10 54:080c6153:::benchmark_data_smithi136_273969_object104560:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:01:40.363 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.460728+0000", 2024-08-17T18:01:40.363 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 115.83910695500001, 2024-08-17T18:01:40.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.181197896, 2024-08-17T18:01:40.363 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:01:40.363 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:01:40.363 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:01:40.363 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:01:40.363 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:01:40.363 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:01:40.363 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104561 2024-08-17T18:01:40.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.363 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:01:40.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:01:40.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460728+0000", 2024-08-17T18:01:40.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:01:40.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460728+0000", 2024-08-17T18:01:40.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:01:40.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460728+0000", 2024-08-17T18:01:40.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8400000000000005e-07 2024-08-17T18:01:40.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:01:40.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460772+0000", 2024-08-17T18:01:40.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3789999999999999e-05 2024-08-17T18:01:40.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:01:40.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460782+0000", 2024-08-17T18:01:40.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5759999999999995e-06 2024-08-17T18:01:40.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:01:40.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460787+0000", 2024-08-17T18:01:40.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1830000000000002e-06 2024-08-17T18:01:40.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460831+0000", 2024-08-17T18:01:40.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4511000000000001e-05 2024-08-17T18:01:40.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:40.368 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:40.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:40.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:01:40.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460837+0000", 2024-08-17T18:01:40.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1340000000000003e-06 2024-08-17T18:01:40.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.462940+0000", 2024-08-17T18:01:40.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021034130000000002 2024-08-17T18:01:40.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:01:40.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.462944+0000", 2024-08-17T18:01:40.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7639999999999999e-06 2024-08-17T18:01:40.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.619595+0000", 2024-08-17T18:01:40.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1566512929999999 2024-08-17T18:01:40.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:01:40.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.619752+0000", 2024-08-17T18:01:40.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015728199999999999 2024-08-17T18:01:40.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:01:40.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.619997+0000", 2024-08-17T18:01:40.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024468899999999997 2024-08-17T18:01:40.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.372 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:01:40.372 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.625016+0000", 2024-08-17T18:01:40.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0050195350000000003 2024-08-17T18:01:40.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.372 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.372 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.636204+0000", 2024-08-17T18:01:40.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011187289 2024-08-17T18:01:40.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.640603+0000", 2024-08-17T18:01:40.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.004399699 2024-08-17T18:01:40.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:01:40.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.640669+0000", 2024-08-17T18:01:40.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5543999999999995e-05 2024-08-17T18:01:40.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:01:40.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.641926+0000", 2024-08-17T18:01:40.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012567100000000001 2024-08-17T18:01:40.373 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.374 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:01:40.374 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.374 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104584 54.10 54:0a7acb30:::benchmark_data_smithi136_273969_object104583:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:01:40.374 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.460796+0000", 2024-08-17T18:01:40.374 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 115.839038621, 2024-08-17T18:01:40.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1872988900000001, 2024-08-17T18:01:40.374 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:01:40.374 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:01:40.374 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:01:40.374 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:01:40.374 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:01:40.375 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:01:40.375 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104584 2024-08-17T18:01:40.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.375 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:01:40.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:01:40.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460796+0000", 2024-08-17T18:01:40.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:01:40.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460796+0000", 2024-08-17T18:01:40.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:01:40.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460797+0000", 2024-08-17T18:01:40.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.99e-07 2024-08-17T18:01:40.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:01:40.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460824+0000", 2024-08-17T18:01:40.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7625000000000001e-05 2024-08-17T18:01:40.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:01:40.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460832+0000", 2024-08-17T18:01:40.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4329999999999998e-06 2024-08-17T18:01:40.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:01:40.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460837+0000", 2024-08-17T18:01:40.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9869999999999999e-06 2024-08-17T18:01:40.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460883+0000", 2024-08-17T18:01:40.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6065e-05 2024-08-17T18:01:40.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:01:40.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460888+0000", 2024-08-17T18:01:40.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3730000000000001e-06 2024-08-17T18:01:40.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.378 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.462961+0000", 2024-08-17T18:01:40.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020730280000000002 2024-08-17T18:01:40.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.378 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:01:40.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.462965+0000", 2024-08-17T18:01:40.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5659999999999998e-06 2024-08-17T18:01:40.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.620849+0000", 2024-08-17T18:01:40.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1578842899999999 2024-08-17T18:01:40.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:01:40.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.620984+0000", 2024-08-17T18:01:40.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000134871 2024-08-17T18:01:40.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:01:40.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.621197+0000", 2024-08-17T18:01:40.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021278000000000001 2024-08-17T18:01:40.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:01:40.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.629901+0000", 2024-08-17T18:01:40.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0087040629999999997 2024-08-17T18:01:40.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.636245+0000", 2024-08-17T18:01:40.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063444590000000002 2024-08-17T18:01:40.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.646207+0000", 2024-08-17T18:01:40.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0099618659999999998 2024-08-17T18:01:40.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:01:40.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.646266+0000", 2024-08-17T18:01:40.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9225999999999998e-05 2024-08-17T18:01:40.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:01:40.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.648095+0000", 2024-08-17T18:01:40.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018286590000000001 2024-08-17T18:01:40.381 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.382 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:01:40.382 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.382 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104598 54.10 54:098f3f77:::benchmark_data_smithi136_273969_object104597:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:01:40.382 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.460846+0000", 2024-08-17T18:01:40.382 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 115.838989043, 2024-08-17T18:01:40.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1893474550000001, 2024-08-17T18:01:40.382 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:01:40.382 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:01:40.382 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:01:40.382 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:01:40.382 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:01:40.382 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:01:40.383 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104598 2024-08-17T18:01:40.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.383 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:01:40.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:01:40.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460846+0000", 2024-08-17T18:01:40.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:01:40.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460846+0000", 2024-08-17T18:01:40.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:01:40.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460846+0000", 2024-08-17T18:01:40.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4000000000000002e-07 2024-08-17T18:01:40.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:01:40.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460879+0000", 2024-08-17T18:01:40.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2716000000000003e-05 2024-08-17T18:01:40.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:01:40.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460886+0000", 2024-08-17T18:01:40.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3050000000000004e-06 2024-08-17T18:01:40.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:01:40.385 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460891+0000", 2024-08-17T18:01:40.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1150000000000002e-06 2024-08-17T18:01:40.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.385 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460943+0000", 2024-08-17T18:01:40.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2011000000000001e-05 2024-08-17T18:01:40.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:01:40.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460948+0000", 2024-08-17T18:01:40.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8790000000000002e-06 2024-08-17T18:01:40.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463002+0000", 2024-08-17T18:01:40.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020533389999999999 2024-08-17T18:01:40.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:01:40.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463009+0000", 2024-08-17T18:01:40.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5660000000000001e-06 2024-08-17T18:01:40.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.622040+0000", 2024-08-17T18:01:40.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.159031304 2024-08-17T18:01:40.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:01:40.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.622192+0000", 2024-08-17T18:01:40.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015120300000000001 2024-08-17T18:01:40.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:01:40.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.622441+0000", 2024-08-17T18:01:40.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024948 2024-08-17T18:01:40.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:01:40.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.629911+0000", 2024-08-17T18:01:40.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0074695719999999998 2024-08-17T18:01:40.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.638518+0000", 2024-08-17T18:01:40.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.008607356 2024-08-17T18:01:40.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.648168+0000", 2024-08-17T18:01:40.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0096494809999999997 2024-08-17T18:01:40.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:01:40.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.648286+0000", 2024-08-17T18:01:40.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118375 2024-08-17T18:01:40.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:01:40.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.650193+0000", 2024-08-17T18:01:40.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0019073130000000001 2024-08-17T18:01:40.390 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.390 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:01:40.390 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.391 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104715 54.10 54:0b603cab:::benchmark_data_smithi136_273969_object104714:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:01:40.391 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.460911+0000", 2024-08-17T18:01:40.391 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 115.83892373800001, 2024-08-17T18:01:40.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.195364366, 2024-08-17T18:01:40.391 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:01:40.391 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:01:40.391 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:01:40.391 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:01:40.391 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:01:40.391 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:01:40.391 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104715 2024-08-17T18:01:40.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.392 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:01:40.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:01:40.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460911+0000", 2024-08-17T18:01:40.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:01:40.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460911+0000", 2024-08-17T18:01:40.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:01:40.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460912+0000", 2024-08-17T18:01:40.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1600000000000001e-07 2024-08-17T18:01:40.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:01:40.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460940+0000", 2024-08-17T18:01:40.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7946999999999999e-05 2024-08-17T18:01:40.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:01:40.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460946+0000", 2024-08-17T18:01:40.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9750000000000001e-06 2024-08-17T18:01:40.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:01:40.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460952+0000", 2024-08-17T18:01:40.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.135e-06 2024-08-17T18:01:40.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460983+0000", 2024-08-17T18:01:40.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1780999999999999e-05 2024-08-17T18:01:40.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:01:40.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460987+0000", 2024-08-17T18:01:40.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.67e-06 2024-08-17T18:01:40.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463044+0000", 2024-08-17T18:01:40.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020564450000000001 2024-08-17T18:01:40.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:01:40.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463050+0000", 2024-08-17T18:01:40.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4989999999999999e-06 2024-08-17T18:01:40.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.623583+0000", 2024-08-17T18:01:40.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1605328690000001 2024-08-17T18:01:40.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:01:40.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.623753+0000", 2024-08-17T18:01:40.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016986900000000001 2024-08-17T18:01:40.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:01:40.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.623997+0000", 2024-08-17T18:01:40.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000244233 2024-08-17T18:01:40.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:01:40.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.630035+0000", 2024-08-17T18:01:40.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00603828 2024-08-17T18:01:40.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.638568+0000", 2024-08-17T18:01:40.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0085331780000000006 2024-08-17T18:01:40.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.654197+0000", 2024-08-17T18:01:40.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015628764 2024-08-17T18:01:40.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:01:40.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.654267+0000", 2024-08-17T18:01:40.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9865999999999998e-05 2024-08-17T18:01:40.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:01:40.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.656275+0000", 2024-08-17T18:01:40.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002008339 2024-08-17T18:01:40.400 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.400 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:01:40.400 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.400 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104717 54.10 54:0f40c48e:::benchmark_data_smithi136_273969_object104716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:01:40.400 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.460961+0000", 2024-08-17T18:01:40.400 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 115.83887424, 2024-08-17T18:01:40.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1975942019999999, 2024-08-17T18:01:40.400 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:01:40.400 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:01:40.401 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:01:40.401 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:01:40.401 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:01:40.401 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:01:40.401 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104717 2024-08-17T18:01:40.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.401 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:01:40.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:01:40.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460961+0000", 2024-08-17T18:01:40.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.402 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:01:40.402 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460961+0000", 2024-08-17T18:01:40.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.402 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:01:40.402 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460961+0000", 2024-08-17T18:01:40.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7100000000000002e-07 2024-08-17T18:01:40.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.402 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:01:40.402 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460980+0000", 2024-08-17T18:01:40.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.861e-05 2024-08-17T18:01:40.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:01:40.403 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460985+0000", 2024-08-17T18:01:40.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7629999999999997e-06 2024-08-17T18:01:40.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:01:40.403 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460989+0000", 2024-08-17T18:01:40.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5760000000000002e-06 2024-08-17T18:01:40.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.403 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461015+0000", 2024-08-17T18:01:40.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5573999999999999e-05 2024-08-17T18:01:40.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:01:40.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461017+0000", 2024-08-17T18:01:40.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.537e-06 2024-08-17T18:01:40.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463076+0000", 2024-08-17T18:01:40.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002058926 2024-08-17T18:01:40.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:01:40.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463080+0000", 2024-08-17T18:01:40.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7409999999999998e-06 2024-08-17T18:01:40.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.625066+0000", 2024-08-17T18:01:40.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1619860850000001 2024-08-17T18:01:40.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:01:40.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.625197+0000", 2024-08-17T18:01:40.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000131514 2024-08-17T18:01:40.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:01:40.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.625353+0000", 2024-08-17T18:01:40.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015604799999999999 2024-08-17T18:01:40.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:01:40.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.630043+0000", 2024-08-17T18:01:40.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0046895219999999998 2024-08-17T18:01:40.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.644315+0000", 2024-08-17T18:01:40.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014271905 2024-08-17T18:01:40.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.656409+0000", 2024-08-17T18:01:40.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012094194000000001 2024-08-17T18:01:40.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:01:40.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.656471+0000", 2024-08-17T18:01:40.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1897999999999996e-05 2024-08-17T18:01:40.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:01:40.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.658555+0000", 2024-08-17T18:01:40.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020838380000000002 2024-08-17T18:01:40.408 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.408 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:01:40.408 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.408 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104721 54.10 54:09fb12db:::benchmark_data_smithi136_273969_object104720:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:01:40.408 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.460995+0000", 2024-08-17T18:01:40.408 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 115.83883960999999, 2024-08-17T18:01:40.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1996424779999999, 2024-08-17T18:01:40.408 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:01:40.409 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:01:40.409 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:01:40.409 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:01:40.409 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:01:40.409 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:01:40.409 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104721 2024-08-17T18:01:40.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.409 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:01:40.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.409 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:01:40.409 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460995+0000", 2024-08-17T18:01:40.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.410 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:01:40.410 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460995+0000", 2024-08-17T18:01:40.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.410 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:01:40.410 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460996+0000", 2024-08-17T18:01:40.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7399999999999998e-07 2024-08-17T18:01:40.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.410 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:01:40.410 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461018+0000", 2024-08-17T18:01:40.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2667000000000002e-05 2024-08-17T18:01:40.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.411 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:01:40.411 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461023+0000", 2024-08-17T18:01:40.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8040000000000001e-06 2024-08-17T18:01:40.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.411 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:01:40.411 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461026+0000", 2024-08-17T18:01:40.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.914e-06 2024-08-17T18:01:40.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.411 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.411 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461056+0000", 2024-08-17T18:01:40.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0443999999999999e-05 2024-08-17T18:01:40.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:01:40.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461060+0000", 2024-08-17T18:01:40.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3780000000000001e-06 2024-08-17T18:01:40.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463107+0000", 2024-08-17T18:01:40.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020471180000000001 2024-08-17T18:01:40.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:01:40.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463113+0000", 2024-08-17T18:01:40.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0100000000000001e-06 2024-08-17T18:01:40.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.626172+0000", 2024-08-17T18:01:40.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.163058996 2024-08-17T18:01:40.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:01:40.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.626319+0000", 2024-08-17T18:01:40.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014732800000000001 2024-08-17T18:01:40.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:01:40.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.626554+0000", 2024-08-17T18:01:40.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023483 2024-08-17T18:01:40.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:01:40.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.630050+0000", 2024-08-17T18:01:40.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034959349999999999 2024-08-17T18:01:40.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.650268+0000", 2024-08-17T18:01:40.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020217681000000001 2024-08-17T18:01:40.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.415 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.415 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.658757+0000", 2024-08-17T18:01:40.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0084896079999999992 2024-08-17T18:01:40.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.415 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:01:40.415 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.658818+0000", 2024-08-17T18:01:40.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0832000000000003e-05 2024-08-17T18:01:40.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.415 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:01:40.415 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.660638+0000", 2024-08-17T18:01:40.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018194590000000001 2024-08-17T18:01:40.416 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.416 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:01:40.416 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.416 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104722 54.10 54:0b2c511b:::benchmark_data_smithi136_273969_object104721:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:01:40.416 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.461032+0000", 2024-08-17T18:01:40.416 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 115.83880232999999, 2024-08-17T18:01:40.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2024628589999999, 2024-08-17T18:01:40.416 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:01:40.416 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:01:40.416 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:01:40.416 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:01:40.417 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:01:40.417 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:01:40.417 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104722 2024-08-17T18:01:40.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.417 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:01:40.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:01:40.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461032+0000", 2024-08-17T18:01:40.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:01:40.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461032+0000", 2024-08-17T18:01:40.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:01:40.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461033+0000", 2024-08-17T18:01:40.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.51e-07 2024-08-17T18:01:40.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:01:40.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461052+0000", 2024-08-17T18:01:40.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9491999999999999e-05 2024-08-17T18:01:40.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:01:40.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461059+0000", 2024-08-17T18:01:40.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4799999999999998e-06 2024-08-17T18:01:40.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.419 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.419 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:01:40.419 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461064+0000", 2024-08-17T18:01:40.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7199999999999997e-06 2024-08-17T18:01:40.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.419 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.419 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461119+0000", 2024-08-17T18:01:40.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5528000000000001e-05 2024-08-17T18:01:40.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:01:40.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461122+0000", 2024-08-17T18:01:40.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5859999999999999e-06 2024-08-17T18:01:40.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463137+0000", 2024-08-17T18:01:40.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002015775 2024-08-17T18:01:40.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:01:40.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463141+0000", 2024-08-17T18:01:40.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4300000000000002e-06 2024-08-17T18:01:40.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.627394+0000", 2024-08-17T18:01:40.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1642535629999999 2024-08-17T18:01:40.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:01:40.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.627555+0000", 2024-08-17T18:01:40.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000160416 2024-08-17T18:01:40.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:01:40.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.627730+0000", 2024-08-17T18:01:40.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017476300000000001 2024-08-17T18:01:40.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:01:40.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.630057+0000", 2024-08-17T18:01:40.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023274659999999998 2024-08-17T18:01:40.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.650326+0000", 2024-08-17T18:01:40.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020268412999999999 2024-08-17T18:01:40.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.423 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.423 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.661579+0000", 2024-08-17T18:01:40.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011253275 2024-08-17T18:01:40.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.423 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:01:40.423 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.661631+0000", 2024-08-17T18:01:40.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2290000000000002e-05 2024-08-17T18:01:40.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.423 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:01:40.423 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.663495+0000", 2024-08-17T18:01:40.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018642109999999999 2024-08-17T18:01:40.424 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.424 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:01:40.424 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.424 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104725 54.10 54:0a958ad6:::benchmark_data_smithi136_273969_object104724:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:01:40.424 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.461101+0000", 2024-08-17T18:01:40.424 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 115.838733701, 2024-08-17T18:01:40.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2042328259999999, 2024-08-17T18:01:40.424 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:01:40.424 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:01:40.424 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:01:40.425 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:01:40.425 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:01:40.425 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:01:40.425 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104725 2024-08-17T18:01:40.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.425 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:01:40.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:01:40.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461101+0000", 2024-08-17T18:01:40.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:01:40.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461101+0000", 2024-08-17T18:01:40.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:01:40.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461102+0000", 2024-08-17T18:01:40.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9800000000000004e-07 2024-08-17T18:01:40.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:01:40.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461121+0000", 2024-08-17T18:01:40.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9491e-05 2024-08-17T18:01:40.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:01:40.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461128+0000", 2024-08-17T18:01:40.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2089999999999999e-06 2024-08-17T18:01:40.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:01:40.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461134+0000", 2024-08-17T18:01:40.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8209999999999998e-06 2024-08-17T18:01:40.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461160+0000", 2024-08-17T18:01:40.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5414000000000001e-05 2024-08-17T18:01:40.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:01:40.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461163+0000", 2024-08-17T18:01:40.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.219e-06 2024-08-17T18:01:40.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463168+0000", 2024-08-17T18:01:40.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020057539999999998 2024-08-17T18:01:40.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:01:40.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463175+0000", 2024-08-17T18:01:40.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2319999999999998e-06 2024-08-17T18:01:40.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.628473+0000", 2024-08-17T18:01:40.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1652981410000001 2024-08-17T18:01:40.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:01:40.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.628622+0000", 2024-08-17T18:01:40.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014901000000000001 2024-08-17T18:01:40.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:01:40.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.628863+0000", 2024-08-17T18:01:40.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024113300000000001 2024-08-17T18:01:40.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:01:40.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.632928+0000", 2024-08-17T18:01:40.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040654350000000001 2024-08-17T18:01:40.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.656352+0000", 2024-08-17T18:01:40.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023423247000000001 2024-08-17T18:01:40.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.432 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.432 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.663605+0000", 2024-08-17T18:01:40.432 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0072530839999999999 2024-08-17T18:01:40.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.432 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:01:40.432 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.663664+0000", 2024-08-17T18:01:40.432 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9172000000000002e-05 2024-08-17T18:01:40.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:01:40.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.665334+0000", 2024-08-17T18:01:40.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016699659999999999 2024-08-17T18:01:40.433 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.433 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:01:40.433 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.433 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104730 54.10 54:0f4303c7:::benchmark_data_smithi136_273969_object104729:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:01:40.433 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.461143+0000", 2024-08-17T18:01:40.434 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 115.838692034, 2024-08-17T18:01:40.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.207791611, 2024-08-17T18:01:40.434 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:01:40.434 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:01:40.434 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:01:40.434 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:01:40.434 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:01:40.434 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:01:40.434 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104730 2024-08-17T18:01:40.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.435 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:01:40.435 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.435 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:01:40.435 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461143+0000", 2024-08-17T18:01:40.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.435 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.435 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:01:40.435 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461143+0000", 2024-08-17T18:01:40.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:01:40.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461143+0000", 2024-08-17T18:01:40.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3700000000000001e-07 2024-08-17T18:01:40.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:01:40.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461162+0000", 2024-08-17T18:01:40.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8580000000000002e-05 2024-08-17T18:01:40.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:01:40.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461168+0000", 2024-08-17T18:01:40.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2770000000000002e-06 2024-08-17T18:01:40.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:01:40.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461173+0000", 2024-08-17T18:01:40.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9269999999999996e-06 2024-08-17T18:01:40.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461198+0000", 2024-08-17T18:01:40.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4581000000000001e-05 2024-08-17T18:01:40.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:01:40.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461200+0000", 2024-08-17T18:01:40.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6010000000000002e-06 2024-08-17T18:01:40.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463205+0000", 2024-08-17T18:01:40.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020046320000000001 2024-08-17T18:01:40.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:01:40.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463211+0000", 2024-08-17T18:01:40.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2890000000000003e-06 2024-08-17T18:01:40.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.632975+0000", 2024-08-17T18:01:40.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.169764158 2024-08-17T18:01:40.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:01:40.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.633138+0000", 2024-08-17T18:01:40.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000162607 2024-08-17T18:01:40.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:01:40.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.633455+0000", 2024-08-17T18:01:40.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00031745299999999998 2024-08-17T18:01:40.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:01:40.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.636257+0000", 2024-08-17T18:01:40.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0028021090000000001 2024-08-17T18:01:40.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.658654+0000", 2024-08-17T18:01:40.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.022396322 2024-08-17T18:01:40.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.667073+0000", 2024-08-17T18:01:40.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0084193099999999993 2024-08-17T18:01:40.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:01:40.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.667123+0000", 2024-08-17T18:01:40.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0209e-05 2024-08-17T18:01:40.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:01:40.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.668934+0000", 2024-08-17T18:01:40.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018111189999999999 2024-08-17T18:01:40.442 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.442 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:01:40.442 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.442 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104740 54.10 54:0bf65440:::benchmark_data_smithi136_273969_object104739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:01:40.442 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.461181+0000", 2024-08-17T18:01:40.443 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 115.838653815, 2024-08-17T18:01:40.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2141290300000001, 2024-08-17T18:01:40.443 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:01:40.443 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:01:40.443 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:01:40.443 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:01:40.443 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:01:40.443 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:01:40.443 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104740 2024-08-17T18:01:40.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.443 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:01:40.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:01:40.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461181+0000", 2024-08-17T18:01:40.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:01:40.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461181+0000", 2024-08-17T18:01:40.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:01:40.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461181+0000", 2024-08-17T18:01:40.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.46e-07 2024-08-17T18:01:40.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:01:40.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461200+0000", 2024-08-17T18:01:40.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8284999999999999e-05 2024-08-17T18:01:40.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:01:40.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461206+0000", 2024-08-17T18:01:40.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2299999999999996e-06 2024-08-17T18:01:40.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:01:40.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461211+0000", 2024-08-17T18:01:40.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0520000000000001e-06 2024-08-17T18:01:40.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461236+0000", 2024-08-17T18:01:40.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5131e-05 2024-08-17T18:01:40.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:01:40.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461239+0000", 2024-08-17T18:01:40.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8710000000000001e-06 2024-08-17T18:01:40.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463249+0000", 2024-08-17T18:01:40.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002010267 2024-08-17T18:01:40.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:01:40.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463257+0000", 2024-08-17T18:01:40.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7270000000000007e-06 2024-08-17T18:01:40.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.635018+0000", 2024-08-17T18:01:40.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1717606540000001 2024-08-17T18:01:40.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:01:40.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.635123+0000", 2024-08-17T18:01:40.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010581699999999999 2024-08-17T18:01:40.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:01:40.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.635324+0000", 2024-08-17T18:01:40.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000201036 2024-08-17T18:01:40.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:01:40.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.640550+0000", 2024-08-17T18:01:40.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0052251800000000003 2024-08-17T18:01:40.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.658714+0000", 2024-08-17T18:01:40.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018164235000000001 2024-08-17T18:01:40.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.673662+0000", 2024-08-17T18:01:40.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014948527 2024-08-17T18:01:40.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:01:40.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.673730+0000", 2024-08-17T18:01:40.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7206999999999995e-05 2024-08-17T18:01:40.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:01:40.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.675310+0000", 2024-08-17T18:01:40.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001580365 2024-08-17T18:01:40.450 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.450 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:01:40.450 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.450 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104765 54.10 54:0fe4eb9c:::benchmark_data_smithi136_273969_object104764:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:01:40.450 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.461219+0000", 2024-08-17T18:01:40.451 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 115.838615745, 2024-08-17T18:01:40.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2158349980000001, 2024-08-17T18:01:40.451 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:01:40.451 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:01:40.451 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:01:40.451 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:01:40.451 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:01:40.451 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:01:40.451 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104765 2024-08-17T18:01:40.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.451 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:01:40.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:01:40.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461219+0000", 2024-08-17T18:01:40.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:01:40.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461219+0000", 2024-08-17T18:01:40.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:01:40.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461219+0000", 2024-08-17T18:01:40.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1100000000000001e-07 2024-08-17T18:01:40.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:01:40.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461237+0000", 2024-08-17T18:01:40.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7504e-05 2024-08-17T18:01:40.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:01:40.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461243+0000", 2024-08-17T18:01:40.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9540000000000001e-06 2024-08-17T18:01:40.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:01:40.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461248+0000", 2024-08-17T18:01:40.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3920000000000002e-06 2024-08-17T18:01:40.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463290+0000", 2024-08-17T18:01:40.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020416470000000002 2024-08-17T18:01:40.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:01:40.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463296+0000", 2024-08-17T18:01:40.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4740000000000002e-06 2024-08-17T18:01:40.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.636283+0000", 2024-08-17T18:01:40.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1729864000000001 2024-08-17T18:01:40.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:01:40.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.636379+0000", 2024-08-17T18:01:40.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6189999999999999e-05 2024-08-17T18:01:40.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:01:40.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.636537+0000", 2024-08-17T18:01:40.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015755500000000001 2024-08-17T18:01:40.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:01:40.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.640558+0000", 2024-08-17T18:01:40.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040215609999999999 2024-08-17T18:01:40.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.661538+0000", 2024-08-17T18:01:40.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020980144999999999 2024-08-17T18:01:40.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.675377+0000", 2024-08-17T18:01:40.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013839179 2024-08-17T18:01:40.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:01:40.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.675428+0000", 2024-08-17T18:01:40.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0639000000000001e-05 2024-08-17T18:01:40.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:01:40.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.677054+0000", 2024-08-17T18:01:40.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001625947 2024-08-17T18:01:40.458 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.458 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:01:40.458 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.458 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104808 54.10 54:0caf74eb:::benchmark_data_smithi136_273969_object104807:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:01:40.458 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.461256+0000", 2024-08-17T18:01:40.458 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 115.838578595, 2024-08-17T18:01:40.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2175662680000001, 2024-08-17T18:01:40.458 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:01:40.459 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:01:40.459 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:01:40.459 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:01:40.459 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:01:40.459 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:01:40.459 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104808 2024-08-17T18:01:40.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.459 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:01:40.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:01:40.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461256+0000", 2024-08-17T18:01:40.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:01:40.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461256+0000", 2024-08-17T18:01:40.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:01:40.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:01:40.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461257+0000", 2024-08-17T18:01:40.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5999999999999999e-07 2024-08-17T18:01:40.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:01:40.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461278+0000", 2024-08-17T18:01:40.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1223999999999999e-05 2024-08-17T18:01:40.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:01:40.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461284+0000", 2024-08-17T18:01:40.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5379999999999999e-06 2024-08-17T18:01:40.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:01:40.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461290+0000", 2024-08-17T18:01:40.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7130000000000002e-06 2024-08-17T18:01:40.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463320+0000", 2024-08-17T18:01:40.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020298809999999999 2024-08-17T18:01:40.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:01:40.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463326+0000", 2024-08-17T18:01:40.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4339999999999997e-06 2024-08-17T18:01:40.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:01:40.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.637362+0000", 2024-08-17T18:01:40.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.174035218 2024-08-17T18:01:40.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:01:40.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.637517+0000", 2024-08-17T18:01:40.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015547400000000001 2024-08-17T18:01:40.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:01:40.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.637667+0000", 2024-08-17T18:01:40.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014947699999999999 2024-08-17T18:01:40.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:01:40.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.642461+0000", 2024-08-17T18:01:40.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0047943869999999998 2024-08-17T18:01:40.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.663548+0000", 2024-08-17T18:01:40.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.021087257000000002 2024-08-17T18:01:40.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:01:40.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.677112+0000", 2024-08-17T18:01:40.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013563685000000001 2024-08-17T18:01:40.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:01:40.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.677175+0000", 2024-08-17T18:01:40.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3504999999999996e-05 2024-08-17T18:01:40.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:01:40.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:01:40.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:01:40.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.678822+0000", 2024-08-17T18:01:40.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001647115 2024-08-17T18:01:40.466 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.466 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:01:40.466 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.466 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:01:40.466 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:01:40.466 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:01:40.469 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:40.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:40.497 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:40.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:40.598 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:40.620 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:40.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:40.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:40.722 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:40.743 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:40.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:40.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:40.845 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:40.871 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:40.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:40.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:40.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:40 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2346865229' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:01:40.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:40 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2346865229' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:01:40.961 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:01:40.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:41.001 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:41.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:41.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:40 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2346865229' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:01:41.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:41.126 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:41.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:41.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:41.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:41.255 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:41.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:41.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:41.357 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:41.382 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:41.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:41.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:41.484 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:41.505 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:41.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:41.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:41.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:41.631 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:41.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:41.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:41.733 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:41.768 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:41.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:41.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:41.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:41.899 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:41.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:41.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:41.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:41 smithi136 ceph-mon[109861]: pgmap v2435: 99 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 2972/177432 objects degraded (1.675%); 1.0 MiB/s, 222 objects/s recovering 2024-08-17T18:01:41.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:41 smithi136 ceph-mon[109861]: osdmap e1411: 8 total, 8 up, 7 in 2024-08-17T18:01:41.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:41 smithi136 ceph-mon[115078]: pgmap v2435: 99 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 2972/177432 objects degraded (1.675%); 1.0 MiB/s, 222 objects/s recovering 2024-08-17T18:01:41.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:41 smithi136 ceph-mon[115078]: osdmap e1411: 8 total, 8 up, 7 in 2024-08-17T18:01:42.001 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:42.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:41 smithi165 ceph-mon[105517]: pgmap v2435: 99 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 2972/177432 objects degraded (1.675%); 1.0 MiB/s, 222 objects/s recovering 2024-08-17T18:01:42.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:41 smithi165 ceph-mon[105517]: osdmap e1411: 8 total, 8 up, 7 in 2024-08-17T18:01:42.037 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:42.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:42.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:42.139 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:42.166 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:42.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:42.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:42.268 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:42.305 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:42.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:42.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:42.408 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:42.437 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:42.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:42.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:42.539 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:42.564 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:42.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:42.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:42.666 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:42.693 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:42.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:42.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:42.795 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:42.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:42.823 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:42.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:42.924 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:42.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:42 smithi136 ceph-mon[109861]: osdmap e1412: 8 total, 8 up, 7 in 2024-08-17T18:01:42.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:42 smithi136 ceph-mon[115078]: osdmap e1412: 8 total, 8 up, 7 in 2024-08-17T18:01:42.950 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:42.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:42.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:43.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:42 smithi165 ceph-mon[105517]: osdmap e1412: 8 total, 8 up, 7 in 2024-08-17T18:01:43.053 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:43.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:43.087 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:43.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:43.188 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:43.222 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:43.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:43.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:43.325 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:43.351 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:43.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:43.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:43.453 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:43.476 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:43.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:43.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:43.577 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:43.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:43.600 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:43.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:43.701 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:43.728 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:43.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:43.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:43.830 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:43.855 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:43.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:43.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:43.883 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:43 smithi136 ceph-mon[109861]: pgmap v2438: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 560 op/s; 2972/170052 objects degraded (1.748%) 2024-08-17T18:01:43.883 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:43 smithi136 ceph-mon[115078]: pgmap v2438: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 560 op/s; 2972/170052 objects degraded (1.748%) 2024-08-17T18:01:43.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:43.985 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:43.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:43.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:43 smithi165 ceph-mon[105517]: pgmap v2438: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 560 op/s; 2972/170052 objects degraded (1.748%) 2024-08-17T18:01:44.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:44.117 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:44.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:44.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:44.219 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:44.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:44.248 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:44.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:44.347 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-17T18:01:44.349 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:44.373 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:44.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:44.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:44.436 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:01:44.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:44.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:44.510 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:44.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:44.611 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:44.637 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:44.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:44.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:44.663 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:01:44.739 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:44 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-08-17T18:01:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:44 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 2972/170052 objects degraded (1.748%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:01:44.764 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:44.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:44.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:44.840 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:44 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-08-17T18:01:44.841 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:44 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 2972/170052 objects degraded (1.748%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:01:44.841 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:44 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-08-17T18:01:44.841 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:44 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 2972/170052 objects degraded (1.748%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:01:44.865 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:01:44.889 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:44.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:44.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:01:44.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:45.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:45.017 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:45.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:45.118 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:45.143 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:45.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:45.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:45.245 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:45.267 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:45.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:45.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:45.369 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:01:45.390 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:45.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:45.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:01:45.492 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:45.514 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:45.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:45.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:45.615 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:45.640 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:45.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:45.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:45.742 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:45.747 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:45 smithi165 ceph-mon[105517]: pgmap v2439: 99 pgs: 1 active+recovering+undersized+remapped, 98 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 548 op/s; 2266/167562 objects misplaced (1.352%); 1.2 MiB/s, 103 objects/s recovering 2024-08-17T18:01:45.747 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:45 smithi165 ceph-mon[105517]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2972/170052 objects degraded (1.748%), 1 pg degraded) 2024-08-17T18:01:45.747 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:45 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "19"}]': finished 2024-08-17T18:01:45.747 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:45 smithi165 ceph-mon[105517]: osdmap e1413: 8 total, 8 up, 7 in 2024-08-17T18:01:45.764 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:45.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:45.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:45.866 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:01:45.887 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:45.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:45.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:01:45.915 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:45 smithi136 ceph-mon[109861]: pgmap v2439: 99 pgs: 1 active+recovering+undersized+remapped, 98 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 548 op/s; 2266/167562 objects misplaced (1.352%); 1.2 MiB/s, 103 objects/s recovering 2024-08-17T18:01:45.915 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:45 smithi136 ceph-mon[109861]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2972/170052 objects degraded (1.748%), 1 pg degraded) 2024-08-17T18:01:45.915 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:45 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "19"}]': finished 2024-08-17T18:01:45.915 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:45 smithi136 ceph-mon[109861]: osdmap e1413: 8 total, 8 up, 7 in 2024-08-17T18:01:45.915 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:45 smithi136 ceph-mon[115078]: pgmap v2439: 99 pgs: 1 active+recovering+undersized+remapped, 98 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 548 op/s; 2266/167562 objects misplaced (1.352%); 1.2 MiB/s, 103 objects/s recovering 2024-08-17T18:01:45.915 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:45 smithi136 ceph-mon[115078]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2972/170052 objects degraded (1.748%), 1 pg degraded) 2024-08-17T18:01:45.916 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:45 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "19"}]': finished 2024-08-17T18:01:45.916 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:45 smithi136 ceph-mon[115078]: osdmap e1413: 8 total, 8 up, 7 in 2024-08-17T18:01:45.954 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:01:45.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:46.010 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:46.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:46.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:46.112 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:46.136 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:46.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:46.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:46.218 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:01:46.238 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:46.259 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:46.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:46.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:46.361 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:46.382 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:46.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:46.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:46.484 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:46.505 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:46.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:46.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:46.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:46.619 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:01:46.633 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:46.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:46.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:46.729 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:01:46.729 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:01:46.729 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-08-17T18:01:46.729 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:01:46.735 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:46.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:46 smithi165 ceph-mon[105517]: osdmap e1414: 8 total, 8 up, 7 in 2024-08-17T18:01:46.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:46 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-08-17T18:01:46.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:46 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2397430742' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:01:46.764 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:46.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:46.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:46.866 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:46.871 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:46 smithi136 ceph-mon[109861]: osdmap e1414: 8 total, 8 up, 7 in 2024-08-17T18:01:46.872 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:46 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-08-17T18:01:46.872 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:46 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2397430742' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:01:46.872 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:46 smithi136 ceph-mon[115078]: osdmap e1414: 8 total, 8 up, 7 in 2024-08-17T18:01:46.872 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:46 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-08-17T18:01:46.872 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:46 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2397430742' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:01:46.888 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:46.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:46.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:46.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:47.033 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:47.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:47.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:47.135 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:47.171 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:47.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:47.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:47.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:47.311 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:47.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:47.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:47.413 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:47.416 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-17T18:01:47.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:47.440 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:47.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:47.541 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:47.566 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:47.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:47.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:47.645 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:01:47.668 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:47.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:47.691 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:47.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:47.734 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:01:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:01:47] "GET /metrics HTTP/1.1" 200 36623 "" "Prometheus/2.51.0" 2024-08-17T18:01:47.734 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:47 smithi136 ceph-mon[115078]: pgmap v2442: 99 pgs: 1 active+recovering+undersized+remapped, 98 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 612 op/s; 2266/167562 objects misplaced (1.352%); 1.3 MiB/s, 116 objects/s recovering 2024-08-17T18:01:47.734 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:47 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:01:47.734 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:47 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "20"}]': finished 2024-08-17T18:01:47.735 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:47 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2397430742' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:01:47.735 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:47 smithi136 ceph-mon[115078]: osdmap e1415: 8 total, 8 up, 7 in 2024-08-17T18:01:47.735 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:47 smithi136 ceph-mon[109861]: pgmap v2442: 99 pgs: 1 active+recovering+undersized+remapped, 98 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 612 op/s; 2266/167562 objects misplaced (1.352%); 1.3 MiB/s, 116 objects/s recovering 2024-08-17T18:01:47.735 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:47 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:01:47.735 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:47 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "20"}]': finished 2024-08-17T18:01:47.735 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:47 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2397430742' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:01:47.735 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:47 smithi136 ceph-mon[109861]: osdmap e1415: 8 total, 8 up, 7 in 2024-08-17T18:01:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:47 smithi165 ceph-mon[105517]: pgmap v2442: 99 pgs: 1 active+recovering+undersized+remapped, 98 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 612 op/s; 2266/167562 objects misplaced (1.352%); 1.3 MiB/s, 116 objects/s recovering 2024-08-17T18:01:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:47 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:01:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:47 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "20"}]': finished 2024-08-17T18:01:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:47 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2397430742' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:01:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:47 smithi165 ceph-mon[105517]: osdmap e1415: 8 total, 8 up, 7 in 2024-08-17T18:01:47.793 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:47.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:47.815 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:47.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:47.916 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:47.937 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:47.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:48.039 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:48.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:48.062 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:48.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:48.163 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:48.187 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:48.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:48.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:48.289 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:48.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:48.311 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:48.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:48.412 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:01:48.434 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:48.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:48.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:01:48.536 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:48.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:48.558 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:48.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:48.659 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:48.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:48.682 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:48.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:48.783 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:01:48.806 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:48.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:48.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:01:48.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:48 smithi165 ceph-mon[105517]: osdmap e1416: 8 total, 8 up, 7 in 2024-08-17T18:01:48.843 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:48 smithi136 ceph-mon[109861]: osdmap e1416: 8 total, 8 up, 7 in 2024-08-17T18:01:48.843 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:48 smithi136 ceph-mon[115078]: osdmap e1416: 8 total, 8 up, 7 in 2024-08-17T18:01:48.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:48.912 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:01:48.912 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:01:48.912 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-17T18:01:48.912 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.957956 2024-08-17T18:01:48.912 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 4 0.9579558637883395 2024-08-17T18:01:48.963 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:48.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:48.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:49.021 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:01:49.022 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:01:49.061 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:01:49.061 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:01:49.065 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:49.098 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:49.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:49.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:49.102 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:01:49.102 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:01:49.133 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:01:49.133 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:01:49.169 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:01:49.169 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:01:49.201 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:49.208 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:01:49.208 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:01:49.239 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:01:49.240 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:01:49.246 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:49.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:49.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:49.274 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:01:49.275 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:01:49.348 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:49.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:49.379 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:49.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:49.480 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:49.514 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:01:49.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:49.525 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:49.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:49.567 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:01:49.567 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:01:49.627 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:49.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:49.664 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:49.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:49 smithi165 ceph-mon[105517]: pgmap v2445: 99 pgs: 1 active+recovering+undersized+remapped, 1 unknown, 1 remapped+peering, 96 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1947/154986 objects misplaced (1.256%); 2.5 MiB/s, 39 objects/s recovering 2024-08-17T18:01:49.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:49 smithi165 ceph-mon[105517]: osdmap e1417: 8 total, 8 up, 7 in 2024-08-17T18:01:49.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:49 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2001871371' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:01:49.765 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:49.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:49.810 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:49.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:49.912 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:01:49.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:49.945 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:49.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:01:49.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:49 smithi136 ceph-mon[109861]: pgmap v2445: 99 pgs: 1 active+recovering+undersized+remapped, 1 unknown, 1 remapped+peering, 96 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1947/154986 objects misplaced (1.256%); 2.5 MiB/s, 39 objects/s recovering 2024-08-17T18:01:49.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:49 smithi136 ceph-mon[109861]: osdmap e1417: 8 total, 8 up, 7 in 2024-08-17T18:01:49.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:49 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2001871371' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:01:49.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:49 smithi136 ceph-mon[115078]: pgmap v2445: 99 pgs: 1 active+recovering+undersized+remapped, 1 unknown, 1 remapped+peering, 96 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1947/154986 objects misplaced (1.256%); 2.5 MiB/s, 39 objects/s recovering 2024-08-17T18:01:49.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:49 smithi136 ceph-mon[115078]: osdmap e1417: 8 total, 8 up, 7 in 2024-08-17T18:01:49.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:49 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2001871371' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:01:50.033 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:01:50.046 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:50.080 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:50.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:50.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:50.182 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:50.219 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:50.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:50.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:50.322 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:50.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:50.360 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:50.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:50.461 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:50.494 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:50.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:50.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:50.596 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:01:50.624 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:50.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:50.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:01:50.726 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:50.764 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:50.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:50.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:50.798 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:01:50.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-17T18:01:50.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:50.892 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:50.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:50.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:50.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:51.040 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:51.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:51.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:51.142 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:51.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:51.189 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:51.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:51.257 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:01:51.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:51.331 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:51.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:51.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:51.432 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:51.471 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:51.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:51.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:51.573 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:51.610 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:51.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:51.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:51.712 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:51.737 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:51.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:51.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:51.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:51 smithi165 ceph-mon[105517]: pgmap v2447: 99 pgs: 1 active+recovering+undersized+remapped, 1 unknown, 1 remapped+peering, 96 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1947/154986 objects misplaced (1.256%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T18:01:51.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:01:51.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:01:51.840 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:51.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:51.874 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:51.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:51.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:51 smithi136 ceph-mon[109861]: pgmap v2447: 99 pgs: 1 active+recovering+undersized+remapped, 1 unknown, 1 remapped+peering, 96 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1947/154986 objects misplaced (1.256%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T18:01:51.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:01:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:01:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:51 smithi136 ceph-mon[115078]: pgmap v2447: 99 pgs: 1 active+recovering+undersized+remapped, 1 unknown, 1 remapped+peering, 96 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1947/154986 objects misplaced (1.256%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T18:01:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:01:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:01:51.974 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:52.005 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:52.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:52.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:52.033 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:01:52.106 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:52.146 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:52.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:52.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:52.247 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:52.292 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:52.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:52.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:52.394 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:01:52.425 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:52.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:52.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:01:52.526 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:52.569 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:52.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:52.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:52.671 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:52.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:52.693 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:52.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:52.794 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:01:52.829 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:52.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:52.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:01:52.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:52.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:52 smithi136 ceph-mon[109861]: Health check failed: Degraded data redundancy: 7386/152946 objects degraded (4.829%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:01:52.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:52 smithi136 ceph-mon[115078]: Health check failed: Degraded data redundancy: 7386/152946 objects degraded (4.829%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:01:52.965 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:52.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:52.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:52 smithi165 ceph-mon[105517]: Health check failed: Degraded data redundancy: 7386/152946 objects degraded (4.829%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:01:53.067 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:53.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:53.109 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:53.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:53.211 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:01:53.237 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:53.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:53.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:01:53.339 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:53.384 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:53.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:53.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:53.486 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:01:53.517 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:53.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:53.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:01:53.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:53.643 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:53.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:53.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:53.746 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:53.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:53 smithi165 ceph-mon[105517]: pgmap v2448: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 96 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 811 op/s; 7386/152946 objects degraded (4.829%); 9599/152946 objects misplaced (6.276%); 2.7 MiB/s, 42 objects/s recovering 2024-08-17T18:01:53.782 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:53.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:53.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:53.801 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:01:53.801 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:01:53.885 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:01:53.915 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:53.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:53.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:01:53.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:53 smithi136 ceph-mon[109861]: pgmap v2448: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 96 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 811 op/s; 7386/152946 objects degraded (4.829%); 9599/152946 objects misplaced (6.276%); 2.7 MiB/s, 42 objects/s recovering 2024-08-17T18:01:53.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:53 smithi136 ceph-mon[115078]: pgmap v2448: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 96 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 811 op/s; 7386/152946 objects degraded (4.829%); 9599/152946 objects misplaced (6.276%); 2.7 MiB/s, 42 objects/s recovering 2024-08-17T18:01:54.017 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:54.065 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:54.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:54.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:54.167 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:01:54.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:54.195 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:54.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:01:54.296 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:54.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:54.329 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:54.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:54.431 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:54.466 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:54.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:54.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:54.498 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:01:54.532 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-17T18:01:54.568 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:54.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:54.606 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:54.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:54.707 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:54.710 INFO:teuthology.orchestra.run.smithi136.stderr:set osd.4 primary-affinity to 0.957956 (8627802) 2024-08-17T18:01:54.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:54 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/926277895' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.9579558637883395} : dispatch 2024-08-17T18:01:54.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:54.766 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:54.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:54.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:01:54.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:54.891 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:54.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:01:54.902 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:01:54.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:54 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/926277895' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.9579558637883395} : dispatch 2024-08-17T18:01:54.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:54 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/926277895' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.9579558637883395} : dispatch 2024-08-17T18:01:54.992 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:55.038 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:55.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:55.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:55.140 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:55.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:55.167 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:55.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:55.268 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:55.303 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:55.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:55.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:55.405 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:01:55.427 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:55.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:55.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:01:55.529 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:55.563 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:55.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:55.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:55.665 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:55.694 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:55.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:55.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:55.716 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:01:55.796 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:55.824 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:55.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:55.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:55.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:55.931 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:55 smithi136 ceph-mon[109861]: pgmap v2449: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 590 op/s; 7386/150588 objects degraded (4.905%); 9380/150588 objects misplaced (6.229%); 3.1 MiB/s, 50 objects/s recovering 2024-08-17T18:01:55.932 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:55 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/926277895' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.9579558637883395}]': finished 2024-08-17T18:01:55.932 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:55 smithi136 ceph-mon[109861]: osdmap e1418: 8 total, 8 up, 7 in 2024-08-17T18:01:55.932 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:55 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/577515803' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:01:55.932 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:55 smithi136 ceph-mon[115078]: pgmap v2449: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 590 op/s; 7386/150588 objects degraded (4.905%); 9380/150588 objects misplaced (6.229%); 3.1 MiB/s, 50 objects/s recovering 2024-08-17T18:01:55.932 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:55 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/926277895' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.9579558637883395}]': finished 2024-08-17T18:01:55.932 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:55 smithi136 ceph-mon[115078]: osdmap e1418: 8 total, 8 up, 7 in 2024-08-17T18:01:55.932 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:55 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/577515803' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:01:55.963 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:55.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:55.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:55 smithi165 ceph-mon[105517]: pgmap v2449: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 590 op/s; 7386/150588 objects degraded (4.905%); 9380/150588 objects misplaced (6.229%); 3.1 MiB/s, 50 objects/s recovering 2024-08-17T18:01:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:55 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/926277895' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.9579558637883395}]': finished 2024-08-17T18:01:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:55 smithi165 ceph-mon[105517]: osdmap e1418: 8 total, 8 up, 7 in 2024-08-17T18:01:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:55 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/577515803' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:01:56.064 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:56.101 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:56.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:56.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:56.203 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:56.233 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:56.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:56.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:56.338 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:56.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:56.370 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:56.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:56.471 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:56.501 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:56.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:56.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:56.603 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:56.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:56.655 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:56.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:56.756 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:56.798 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:56.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:56.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:56.900 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:56.915 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:56 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:01:56.916 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:56 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/577515803' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:01:56.916 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:56 smithi136 ceph-mon[109861]: osdmap e1419: 8 total, 8 up, 7 in 2024-08-17T18:01:56.916 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:56 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:01:56.916 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:56 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/577515803' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:01:56.916 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:56 smithi136 ceph-mon[115078]: osdmap e1419: 8 total, 8 up, 7 in 2024-08-17T18:01:56.939 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:56.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:56.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:56 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:01:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:56 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/577515803' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:01:57.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:56 smithi165 ceph-mon[105517]: osdmap e1419: 8 total, 8 up, 7 in 2024-08-17T18:01:57.041 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:57.069 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:57.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:57.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:57.171 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:01:57.201 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:57.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:57.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:01:57.303 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:57.343 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:57.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:57.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:57.445 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:01:57.480 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:57.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:57.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:01:57.503 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:01:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:01:57] "GET /metrics HTTP/1.1" 200 36638 "" "Prometheus/2.51.0" 2024-08-17T18:01:57.538 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:01:57.538 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:01:57.556 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:01:57.557 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:01:57.577 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:01:57.577 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:01:57.582 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:57.596 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:01:57.596 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:01:57.616 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:57.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:57.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:57.619 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:01:57.619 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:01:57.647 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:01:57.648 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:01:57.673 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:01:57.673 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:01:57.698 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:01:57.698 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:01:57.718 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:57.753 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:57.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:57.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:57.855 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:57.878 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:01:57.878 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:01:57.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.879 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.35370.0:200996 54.c e1419/1394 54:371e82bb:::benchmark_data_smithi136_273969_object78111:head v 1419'12349, mlcod=1419'12346)", 2024-08-17T18:01:57.879 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:57.719891+0000", 2024-08-17T18:01:57.879 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0052771160000000001, 2024-08-17T18:01:57.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0053068580000000002, 2024-08-17T18:01:57.879 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:01:57.879 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:01:57.879 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:01:57.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:01:57.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.719891+0000", 2024-08-17T18:01:57.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:01:57.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:01:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.719891+0000", 2024-08-17T18:01:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:01:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:01:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.719889+0000", 2024-08-17T18:01:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:01:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:01:57.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.719897+0000", 2024-08-17T18:01:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7230000000000004e-06 2024-08-17T18:01:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:01:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.719910+0000", 2024-08-17T18:01:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2683e-05 2024-08-17T18:01:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:01:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.719919+0000", 2024-08-17T18:01:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7429999999999997e-06 2024-08-17T18:01:57.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:01:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.722346+0000", 2024-08-17T18:01:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002426334 2024-08-17T18:01:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:01:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.722377+0000", 2024-08-17T18:01:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0867000000000001e-05 2024-08-17T18:01:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:01:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:01:57.882 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:01:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:200999 54.2 54:410c7a3a:::benchmark_data_smithi136_273969_object78114:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1419)", 2024-08-17T18:01:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:57.723205+0000", 2024-08-17T18:01:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0019622699999999999, 2024-08-17T18:01:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020564630000000001, 2024-08-17T18:01:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:01:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-08-17T18:01:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:01:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:01:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:01:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 200999 2024-08-17T18:01:57.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:01:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:01:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.723205+0000", 2024-08-17T18:01:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:01:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:01:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.723205+0000", 2024-08-17T18:01:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:01:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:01:57.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.723204+0000", 2024-08-17T18:01:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:01:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:01:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.723210+0000", 2024-08-17T18:01:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0780000000000001e-06 2024-08-17T18:01:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:01:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.723221+0000", 2024-08-17T18:01:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1382e-05 2024-08-17T18:01:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:01:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.723228+0000", 2024-08-17T18:01:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9079999999999998e-06 2024-08-17T18:01:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:01:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.723281+0000", 2024-08-17T18:01:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2839e-05 2024-08-17T18:01:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:01:57.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.723398+0000", 2024-08-17T18:01:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011708499999999999 2024-08-17T18:01:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,6", 2024-08-17T18:01:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.723955+0000", 2024-08-17T18:01:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00055691200000000001 2024-08-17T18:01:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:01:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:01:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:01:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.35370.0:200806 54.10 e1419/1400)", 2024-08-17T18:01:57.887 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:57.725018+0000", 2024-08-17T18:01:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.000149147, 2024-08-17T18:01:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00030652499999999998, 2024-08-17T18:01:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:01:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-08-17T18:01:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:01:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:01:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.725018+0000", 2024-08-17T18:01:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:01:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:01:57.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.725018+0000", 2024-08-17T18:01:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:01:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:01:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.725018+0000", 2024-08-17T18:01:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:01:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:01:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.725024+0000", 2024-08-17T18:01:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2219999999999999e-06 2024-08-17T18:01:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:01:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.725036+0000", 2024-08-17T18:01:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2228999999999999e-05 2024-08-17T18:01:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:01:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.725043+0000", 2024-08-17T18:01:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2089999999999999e-06 2024-08-17T18:01:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:01:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:01:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:01:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPushReply(54.10 1419/1400 [PushReplyOp(54:086aee18:::benchmark_data_smithi136_273969_object77953:head)])", 2024-08-17T18:01:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:57.708347+0000", 2024-08-17T18:01:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.016821117, 2024-08-17T18:01:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017021107000000001, 2024-08-17T18:01:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:01:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-08-17T18:01:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:01:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:01:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.708347+0000", 2024-08-17T18:01:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:01:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:01:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.708347+0000", 2024-08-17T18:01:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:01:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:01:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.708345+0000", 2024-08-17T18:01:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:01:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:01:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.708355+0000", 2024-08-17T18:01:57.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8039999999999996e-06 2024-08-17T18:01:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:01:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.708369+0000", 2024-08-17T18:01:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4188999999999999e-05 2024-08-17T18:01:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:01:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.708378+0000", 2024-08-17T18:01:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3389999999999999e-06 2024-08-17T18:01:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:01:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:01:57.893 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:01:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(54.10 1419/1400 [PushOp(54:0f0dc14d:::benchmark_data_smithi136_273969_object78065:head, version: 1378'4872, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(54:0f0dc14d:::benchmark_data_smithi136_273969_object78065:head@1378'4872, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-17T18:01:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:57.713802+0000", 2024-08-17T18:01:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.011365207, 2024-08-17T18:01:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011630876, 2024-08-17T18:01:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:01:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-08-17T18:01:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:01:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:01:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.713802+0000", 2024-08-17T18:01:57.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:01:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:01:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.713802+0000", 2024-08-17T18:01:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:01:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:01:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.713802+0000", 2024-08-17T18:01:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:01:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:01:57.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.713836+0000", 2024-08-17T18:01:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4017000000000003e-05 2024-08-17T18:01:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:01:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.713852+0000", 2024-08-17T18:01:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5951999999999999e-05 2024-08-17T18:01:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:01:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.713865+0000", 2024-08-17T18:01:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3616e-05 2024-08-17T18:01:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:01:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:01:57.896 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:01:57.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.897 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.35370.0:200993 54.c e1419/1394 54:360d5485:::benchmark_data_smithi136_273969_object78108:head v 1419'12348, mlcod=1419'12346)", 2024-08-17T18:01:57.897 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:57.717981+0000", 2024-08-17T18:01:57.897 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0071862899999999997, 2024-08-17T18:01:57.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0074954080000000003, 2024-08-17T18:01:57.897 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:01:57.897 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:01:57.897 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:01:57.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:01:57.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.717981+0000", 2024-08-17T18:01:57.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:01:57.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:01:57.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.717981+0000", 2024-08-17T18:01:57.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:01:57.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:01:57.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.717980+0000", 2024-08-17T18:01:57.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:01:57.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:01:57.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.717988+0000", 2024-08-17T18:01:57.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6869999999999994e-06 2024-08-17T18:01:57.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:01:57.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.717998+0000", 2024-08-17T18:01:57.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0189e-05 2024-08-17T18:01:57.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:01:57.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.718005+0000", 2024-08-17T18:01:57.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4899999999999997e-06 2024-08-17T18:01:57.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:01:57.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.718043+0000", 2024-08-17T18:01:57.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8515e-05 2024-08-17T18:01:57.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:01:57.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:01:57.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:01:57.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:57.718062+0000", 2024-08-17T18:01:57.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9082000000000001e-05 2024-08-17T18:01:57.900 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:01:57.900 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:01:57.900 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:01:57.901 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:01:57.901 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-08-17T18:01:57.901 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 6 2024-08-17T18:01:57.901 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:01:57.901 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:57 smithi165 ceph-mon[105517]: pgmap v2452: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 33 MiB/s wr, 1.26k op/s; 7386/150588 objects degraded (4.905%); 9380/150588 objects misplaced (6.229%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T18:01:57.901 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:57 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4191489583' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:01:57.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:57 smithi165 ceph-mon[105517]: 54.18 scrub starts 2024-08-17T18:01:57.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:57.903 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:57.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:57.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:57 smithi136 ceph-mon[109861]: pgmap v2452: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 33 MiB/s wr, 1.26k op/s; 7386/150588 objects degraded (4.905%); 9380/150588 objects misplaced (6.229%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T18:01:57.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:57 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4191489583' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:01:57.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:57 smithi136 ceph-mon[109861]: 54.18 scrub starts 2024-08-17T18:01:57.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:57 smithi136 ceph-mon[115078]: pgmap v2452: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 33 MiB/s wr, 1.26k op/s; 7386/150588 objects degraded (4.905%); 9380/150588 objects misplaced (6.229%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T18:01:57.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:57 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4191489583' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:01:57.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:57 smithi136 ceph-mon[115078]: 54.18 scrub starts 2024-08-17T18:01:58.003 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:58.025 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:58.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:58.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:58.127 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:58.171 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:58.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:58.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:58.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:58.307 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:58.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:58.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:58.409 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:58.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:58.439 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:58.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:58.539 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:58.569 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:58.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:58.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:58.671 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:01:58.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:58.705 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:58.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:01:58.806 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:01:58.836 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:58.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:58.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:01:58.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:58.950 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:01:58.969 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:58.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:58.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:58.976 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:58 smithi136 ceph-mon[109861]: 54.18 scrub ok 2024-08-17T18:01:58.976 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:58 smithi136 ceph-mon[115078]: 54.18 scrub ok 2024-08-17T18:01:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:58 smithi165 ceph-mon[105517]: 54.18 scrub ok 2024-08-17T18:01:59.071 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:01:59.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:59.106 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:59.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:01:59.122 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-17T18:01:59.207 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:01:59.249 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:01:59.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:59.253 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:59.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:01:59.354 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:59.382 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:59.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:59.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:59.484 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:59.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:59.534 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:59.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:01:59.536 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:01:59.635 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:01:59.666 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:59.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:59.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:01:59.768 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:01:59.794 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:01:59.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:59.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:01:59.896 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:01:59.929 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:01:59.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:01:59.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:59 smithi165 ceph-mon[105517]: pgmap v2453: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 96 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 30 MiB/s wr, 1.72k op/s; 7086/137112 objects degraded (5.168%); 9103/137112 objects misplaced (6.639%); 3.3 MiB/s, 52 objects/s recovering 2024-08-17T18:02:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:01:59 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 7086/137112 objects degraded (5.168%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:02:00.031 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:00.055 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:00.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:00.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:00.157 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:00.192 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:00.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:00.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:00.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:59 smithi136 ceph-mon[109861]: pgmap v2453: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 96 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 30 MiB/s wr, 1.72k op/s; 7086/137112 objects degraded (5.168%); 9103/137112 objects misplaced (6.639%); 3.3 MiB/s, 52 objects/s recovering 2024-08-17T18:02:00.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:01:59 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 7086/137112 objects degraded (5.168%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:02:00.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:59 smithi136 ceph-mon[115078]: pgmap v2453: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 96 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 30 MiB/s wr, 1.72k op/s; 7086/137112 objects degraded (5.168%); 9103/137112 objects misplaced (6.639%); 3.3 MiB/s, 52 objects/s recovering 2024-08-17T18:02:00.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:01:59 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 7086/137112 objects degraded (5.168%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:02:00.294 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:00.323 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:00.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:00.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:00.425 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:00.454 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:00.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:00.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:00.477 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:02:00.556 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:00.578 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:00.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:00.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:00.680 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:00.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:00.719 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:00.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:00.820 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:00.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:00.856 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:00.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:00.957 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:00.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:00.989 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:00.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:00 smithi165 ceph-mon[105517]: pgmap v2454: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 96 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 659 op/s; 7086/137112 objects degraded (5.168%); 9103/137112 objects misplaced (6.639%); 2.5 MiB/s, 40 objects/s recovering 2024-08-17T18:02:01.091 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:01.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:01.133 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:01.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:01.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:00 smithi136 ceph-mon[109861]: pgmap v2454: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 96 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 659 op/s; 7086/137112 objects degraded (5.168%); 9103/137112 objects misplaced (6.639%); 2.5 MiB/s, 40 objects/s recovering 2024-08-17T18:02:01.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:00 smithi136 ceph-mon[115078]: pgmap v2454: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 96 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 659 op/s; 7086/137112 objects degraded (5.168%); 9103/137112 objects misplaced (6.639%); 2.5 MiB/s, 40 objects/s recovering 2024-08-17T18:02:01.234 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:01.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:01.271 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:01.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:01.372 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:01.397 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:02:01.397 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:02:01.397 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [1, 2, 3, 4, 5, 6, 7] 2024-08-17T18:02:01.397 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd out 5 2024-08-17T18:02:01.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:01.412 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:01.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:01.512 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:01.548 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:01.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:01.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:01.650 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:01.691 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:01.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:01.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:01.744 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:02:01.744 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:02:01.744 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-08-17T18:02:01.744 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-08-17T18:02:01.744 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:02:01.794 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:01.833 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:02.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:02.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:02.039 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:01 smithi165 ceph-mon[105517]: 54.6 scrub starts 2024-08-17T18:02:02.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:02.174 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:02.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:02.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:02.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:01 smithi136 ceph-mon[109861]: 54.6 scrub starts 2024-08-17T18:02:02.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:01 smithi136 ceph-mon[115078]: 54.6 scrub starts 2024-08-17T18:02:02.276 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:02.315 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:02.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:02.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:02.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:02.432 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-17T18:02:02.448 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:02.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:02.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:02.549 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:02.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:02.597 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:02.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:02.611 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:02:02.668 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:02:02.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:02.728 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:02.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:02.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:02.825 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:02:02.829 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:02.858 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:02.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:02.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:02.883 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:02 smithi136 ceph-mon[109861]: pgmap v2455: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 96 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 964 op/s; 6846/127440 objects degraded (5.372%); 8982/127440 objects misplaced (7.048%); 2.6 MiB/s, 41 objects/s recovering 2024-08-17T18:02:02.883 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:02 smithi136 ceph-mon[115078]: pgmap v2455: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 96 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 964 op/s; 6846/127440 objects degraded (5.372%); 8982/127440 objects misplaced (7.048%); 2.6 MiB/s, 41 objects/s recovering 2024-08-17T18:02:02.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:02.996 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:02.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:02.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:03.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:02 smithi165 ceph-mon[105517]: pgmap v2455: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 96 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 964 op/s; 6846/127440 objects degraded (5.372%); 8982/127440 objects misplaced (7.048%); 2.6 MiB/s, 41 objects/s recovering 2024-08-17T18:02:03.099 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:03.127 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:03.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:03.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:03.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:03.260 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:03.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:03.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:03.362 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:03.391 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:03.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:03.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:03.494 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:03.528 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:03.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:03.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:03.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:03.662 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:03.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:03.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:03.764 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:03.806 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:03.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:03.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:03.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:03.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:03.940 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:03.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:04.041 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:04.069 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:04.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:04.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:04.171 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:04.176 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:04 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 6846/124524 objects degraded (5.498%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:02:04.198 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:04.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:04.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:04.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:04.329 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:04.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:04.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:04.331 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:04 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 6846/124524 objects degraded (5.498%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:02:04.332 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:04 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 6846/124524 objects degraded (5.498%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:02:04.432 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:04.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:04.474 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:04.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:04.575 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:04.618 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:04.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:04.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:04.720 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:04.750 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:04.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:04.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:04.852 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:04.884 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:04.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:04.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:04.915 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:02:04.915 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T18:02:04.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T18:02:04.916 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:02:04.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.916 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:179375 54.10 54:09c4e419:::benchmark_data_smithi136_273969_object56490:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1397)", 2024-08-17T18:02:04.916 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:20.619242+0000", 2024-08-17T18:02:04.916 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 44.291500583999998, 2024-08-17T18:02:04.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.080588077, 2024-08-17T18:02:04.916 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:02:04.916 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:02:04.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:02:04.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:02:04.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:02:04.916 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 179375 2024-08-17T18:02:04.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.917 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:02:04.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:02:04.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:20.619242+0000", 2024-08-17T18:02:04.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:04.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:02:04.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:20.619242+0000", 2024-08-17T18:02:04.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:04.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:02:04.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:20.619240+0000", 2024-08-17T18:02:04.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:02:04.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:02:04.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:20.619247+0000", 2024-08-17T18:02:04.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9110000000000004e-06 2024-08-17T18:02:04.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:02:04.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:20.619264+0000", 2024-08-17T18:02:04.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6855999999999999e-05 2024-08-17T18:02:04.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:02:04.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:20.619271+0000", 2024-08-17T18:02:04.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0809999999999996e-06 2024-08-17T18:02:04.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:20.643243+0000", 2024-08-17T18:02:04.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023972284 2024-08-17T18:02:04.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:02:04.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:20.645016+0000", 2024-08-17T18:02:04.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017727769999999999 2024-08-17T18:02:04.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:20.799073+0000", 2024-08-17T18:02:04.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.154057 2024-08-17T18:02:04.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:02:04.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:20.802366+0000", 2024-08-17T18:02:04.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003292835 2024-08-17T18:02:04.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.638766+0000", 2024-08-17T18:02:04.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.83639995199999995 2024-08-17T18:02:04.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:02:04.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.641264+0000", 2024-08-17T18:02:04.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024977879999999999 2024-08-17T18:02:04.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,4", 2024-08-17T18:02:04.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.646497+0000", 2024-08-17T18:02:04.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00523385 2024-08-17T18:02:04.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:04.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.672883+0000", 2024-08-17T18:02:04.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.026385545 2024-08-17T18:02:04.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:04.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.674771+0000", 2024-08-17T18:02:04.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001888327 2024-08-17T18:02:04.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:02:04.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.691467+0000", 2024-08-17T18:02:04.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016695695 2024-08-17T18:02:04.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:02:04.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.692933+0000", 2024-08-17T18:02:04.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014657450000000001 2024-08-17T18:02:04.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:02:04.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.699830+0000", 2024-08-17T18:02:04.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068969269999999997 2024-08-17T18:02:04.924 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:04.924 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:02:04.924 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:04.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.924 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180116 54.10 54:0f9da3af:::benchmark_data_smithi136_273969_object57231:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:02:04.924 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:21.847960+0000", 2024-08-17T18:02:04.925 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.062781948000001, 2024-08-17T18:02:04.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9097264840000001, 2024-08-17T18:02:04.925 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:02:04.925 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:02:04.925 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:02:04.925 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:02:04.925 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:02:04.925 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180116 2024-08-17T18:02:04.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.925 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:02:04.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:02:04.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.847960+0000", 2024-08-17T18:02:04.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:04.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:02:04.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.847960+0000", 2024-08-17T18:02:04.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:04.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:02:04.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.847959+0000", 2024-08-17T18:02:04.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:02:04.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:02:04.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.847965+0000", 2024-08-17T18:02:04.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.596e-06 2024-08-17T18:02:04.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:02:04.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.847980+0000", 2024-08-17T18:02:04.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4819e-05 2024-08-17T18:02:04.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:02:04.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.847988+0000", 2024-08-17T18:02:04.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8110000000000003e-06 2024-08-17T18:02:04.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.933300+0000", 2024-08-17T18:02:04.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.085312084999999996 2024-08-17T18:02:04.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:02:04.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.936616+0000", 2024-08-17T18:02:04.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033163709999999998 2024-08-17T18:02:04.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.966460+0000", 2024-08-17T18:02:04.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.029843103999999999 2024-08-17T18:02:04.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:02:04.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.970083+0000", 2024-08-17T18:02:04.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036235529999999998 2024-08-17T18:02:04.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.508777+0000", 2024-08-17T18:02:04.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.53869427000000003 2024-08-17T18:02:04.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:02:04.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.511313+0000", 2024-08-17T18:02:04.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025361260000000001 2024-08-17T18:02:04.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,4", 2024-08-17T18:02:04.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.516630+0000", 2024-08-17T18:02:04.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0053168260000000002 2024-08-17T18:02:04.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542130+0000", 2024-08-17T18:02:04.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.025500109999999999 2024-08-17T18:02:04.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:02:04.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542142+0000", 2024-08-17T18:02:04.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.204e-05 2024-08-17T18:02:04.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479434+0000", 2024-08-17T18:02:04.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93729177399999997 2024-08-17T18:02:04.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:02:04.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479449+0000", 2024-08-17T18:02:04.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4654000000000001e-05 2024-08-17T18:02:04.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.540870+0000", 2024-08-17T18:02:04.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0614213989999999 2024-08-17T18:02:04.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:02:04.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.543825+0000", 2024-08-17T18:02:04.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029547509999999998 2024-08-17T18:02:04.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.756247+0000", 2024-08-17T18:02:04.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.212422074 2024-08-17T18:02:04.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:02:04.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.757687+0000", 2024-08-17T18:02:04.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001439631 2024-08-17T18:02:04.935 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:04.935 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:02:04.935 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:04.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.935 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180126 54.10 54:0ae91374:::benchmark_data_smithi136_273969_object57241:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:02:04.935 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:21.861391+0000", 2024-08-17T18:02:04.936 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.049351131999998, 2024-08-17T18:02:04.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9929762970000002, 2024-08-17T18:02:04.936 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:02:04.936 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:02:04.936 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:02:04.936 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:02:04.936 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:02:04.936 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180126 2024-08-17T18:02:04.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.936 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:02:04.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:02:04.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.861391+0000", 2024-08-17T18:02:04.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:04.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:02:04.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.861391+0000", 2024-08-17T18:02:04.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:04.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:02:04.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.861390+0000", 2024-08-17T18:02:04.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:02:04.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:02:04.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.861396+0000", 2024-08-17T18:02:04.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0059999999999998e-06 2024-08-17T18:02:04.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:02:04.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.861414+0000", 2024-08-17T18:02:04.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8851999999999998e-05 2024-08-17T18:02:04.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:02:04.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.861422+0000", 2024-08-17T18:02:04.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3220000000000004e-06 2024-08-17T18:02:04.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.970423+0000", 2024-08-17T18:02:04.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.109001084 2024-08-17T18:02:04.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:02:04.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.973780+0000", 2024-08-17T18:02:04.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033568019999999999 2024-08-17T18:02:04.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.995958+0000", 2024-08-17T18:02:04.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022178531000000001 2024-08-17T18:02:04.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:02:04.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.999891+0000", 2024-08-17T18:02:04.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039327559999999999 2024-08-17T18:02:04.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542792+0000", 2024-08-17T18:02:04.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.54290070599999996 2024-08-17T18:02:04.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:02:04.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542797+0000", 2024-08-17T18:02:04.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9849999999999997e-06 2024-08-17T18:02:04.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479690+0000", 2024-08-17T18:02:04.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93689339000000005 2024-08-17T18:02:04.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:02:04.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479700+0000", 2024-08-17T18:02:04.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5610000000000004e-06 2024-08-17T18:02:04.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.769574+0000", 2024-08-17T18:02:04.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.289874223 2024-08-17T18:02:04.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:02:04.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.775343+0000", 2024-08-17T18:02:04.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057693670000000001 2024-08-17T18:02:04.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.824122+0000", 2024-08-17T18:02:04.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.048778909000000002 2024-08-17T18:02:04.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:02:04.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.826036+0000", 2024-08-17T18:02:04.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019139980000000001 2024-08-17T18:02:04.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:02:04.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.830279+0000", 2024-08-17T18:02:04.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042426240000000004 2024-08-17T18:02:04.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:02:04.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.847119+0000", 2024-08-17T18:02:04.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016840368000000001 2024-08-17T18:02:04.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:04.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.847762+0000", 2024-08-17T18:02:04.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064298700000000003 2024-08-17T18:02:04.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:04.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.848426+0000", 2024-08-17T18:02:04.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000663925 2024-08-17T18:02:04.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:02:04.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.849470+0000", 2024-08-17T18:02:04.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001043719 2024-08-17T18:02:04.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:02:04.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.854367+0000", 2024-08-17T18:02:04.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004897658 2024-08-17T18:02:04.945 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:04.946 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:02:04.946 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:04.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.946 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180174 54.10 54:08915b2c:::benchmark_data_smithi136_273969_object57289:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:02:04.946 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:21.930733+0000", 2024-08-17T18:02:04.946 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.980009473999999, 2024-08-17T18:02:04.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1011982060000003, 2024-08-17T18:02:04.946 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:02:04.946 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:02:04.946 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:02:04.946 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:02:04.946 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:02:04.947 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180174 2024-08-17T18:02:04.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.947 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:02:04.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:02:04.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.930733+0000", 2024-08-17T18:02:04.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:04.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:02:04.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.930733+0000", 2024-08-17T18:02:04.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:04.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:02:04.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.930731+0000", 2024-08-17T18:02:04.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:02:04.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:02:04.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.930738+0000", 2024-08-17T18:02:04.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3450000000000002e-06 2024-08-17T18:02:04.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:02:04.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.930750+0000", 2024-08-17T18:02:04.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2476e-05 2024-08-17T18:02:04.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:02:04.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.930757+0000", 2024-08-17T18:02:04.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8719999999999996e-06 2024-08-17T18:02:04.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.974198+0000", 2024-08-17T18:02:04.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.043441133 2024-08-17T18:02:04.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:02:04.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.977482+0000", 2024-08-17T18:02:04.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032840030000000002 2024-08-17T18:02:04.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.032292+0000", 2024-08-17T18:02:04.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.054810287999999999 2024-08-17T18:02:04.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:02:04.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.035548+0000", 2024-08-17T18:02:04.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032551569999999998 2024-08-17T18:02:04.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542945+0000", 2024-08-17T18:02:04.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50739734999999997 2024-08-17T18:02:04.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:02:04.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542951+0000", 2024-08-17T18:02:04.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8069999999999996e-06 2024-08-17T18:02:04.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479903+0000", 2024-08-17T18:02:04.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93695215499999995 2024-08-17T18:02:04.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:02:04.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479912+0000", 2024-08-17T18:02:04.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7020000000000007e-06 2024-08-17T18:02:04.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.954749+0000", 2024-08-17T18:02:04.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4748370450000001 2024-08-17T18:02:04.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:02:04.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.960359+0000", 2024-08-17T18:02:04.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056101390000000001 2024-08-17T18:02:04.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.992310+0000", 2024-08-17T18:02:04.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.031950854000000001 2024-08-17T18:02:04.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:02:04.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.994150+0000", 2024-08-17T18:02:04.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001840752 2024-08-17T18:02:04.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:02:04.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.997815+0000", 2024-08-17T18:02:04.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036646690000000002 2024-08-17T18:02:04.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:02:04.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.010247+0000", 2024-08-17T18:02:04.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012432284 2024-08-17T18:02:04.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:04.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.025714+0000", 2024-08-17T18:02:04.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015466680999999999 2024-08-17T18:02:04.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:04.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.026527+0000", 2024-08-17T18:02:04.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00081259799999999999 2024-08-17T18:02:04.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:02:04.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.027532+0000", 2024-08-17T18:02:04.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001005159 2024-08-17T18:02:04.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:02:04.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.031931+0000", 2024-08-17T18:02:04.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043991380000000004 2024-08-17T18:02:04.956 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:04.956 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:02:04.957 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:04.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.957 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180181 54.10 54:0a355281:::benchmark_data_smithi136_273969_object57296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:02:04.957 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:21.940096+0000", 2024-08-17T18:02:04.957 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.970645814999997, 2024-08-17T18:02:04.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0013956070000001, 2024-08-17T18:02:04.957 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:02:04.957 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:02:04.957 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:02:04.957 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:02:04.957 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:02:04.957 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180181 2024-08-17T18:02:04.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.958 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:02:04.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:02:04.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.940096+0000", 2024-08-17T18:02:04.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:04.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:02:04.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.940096+0000", 2024-08-17T18:02:04.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:04.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:02:04.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.940094+0000", 2024-08-17T18:02:04.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:02:04.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:02:04.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.940102+0000", 2024-08-17T18:02:04.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3370000000000002e-06 2024-08-17T18:02:04.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:02:04.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.940118+0000", 2024-08-17T18:02:04.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5713999999999999e-05 2024-08-17T18:02:04.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:02:04.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.940126+0000", 2024-08-17T18:02:04.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7350000000000002e-06 2024-08-17T18:02:04.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.978817+0000", 2024-08-17T18:02:04.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038691019 2024-08-17T18:02:04.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:02:04.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.982796+0000", 2024-08-17T18:02:04.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039788150000000001 2024-08-17T18:02:04.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.014670+0000", 2024-08-17T18:02:04.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.031873637000000003 2024-08-17T18:02:04.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:02:04.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.018063+0000", 2024-08-17T18:02:04.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033937350000000002 2024-08-17T18:02:04.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542851+0000", 2024-08-17T18:02:04.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.52478792900000004 2024-08-17T18:02:04.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:02:04.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542857+0000", 2024-08-17T18:02:04.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6289999999999998e-06 2024-08-17T18:02:04.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479777+0000", 2024-08-17T18:02:04.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93691995400000005 2024-08-17T18:02:04.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:02:04.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479786+0000", 2024-08-17T18:02:04.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8529999999999998e-06 2024-08-17T18:02:04.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.860711+0000", 2024-08-17T18:02:04.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.38092492 2024-08-17T18:02:04.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:02:04.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.867150+0000", 2024-08-17T18:02:04.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064388700000000002 2024-08-17T18:02:04.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.900225+0000", 2024-08-17T18:02:04.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.03307529 2024-08-17T18:02:04.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:02:04.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.902023+0000", 2024-08-17T18:02:04.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001798231 2024-08-17T18:02:04.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:02:04.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.906038+0000", 2024-08-17T18:02:04.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040151620000000001 2024-08-17T18:02:04.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:02:04.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.919715+0000", 2024-08-17T18:02:04.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013676364999999999 2024-08-17T18:02:04.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:04.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.935483+0000", 2024-08-17T18:02:04.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015768372999999999 2024-08-17T18:02:04.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:04.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.936289+0000", 2024-08-17T18:02:04.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00080628999999999998 2024-08-17T18:02:04.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:02:04.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.937359+0000", 2024-08-17T18:02:04.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001069293 2024-08-17T18:02:04.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:02:04.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.941492+0000", 2024-08-17T18:02:04.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041333460000000004 2024-08-17T18:02:04.968 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:04.968 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:02:04.968 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:04.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.968 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180198 54.10 54:0a39a81d:::benchmark_data_smithi136_273969_object57313:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:02:04.969 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:21.961999+0000", 2024-08-17T18:02:04.969 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.948742926000001, 2024-08-17T18:02:04.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9727821460000001, 2024-08-17T18:02:04.969 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:02:04.969 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:02:04.969 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:02:04.969 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:02:04.969 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:02:04.969 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180198 2024-08-17T18:02:04.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.969 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:02:04.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:02:04.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.961999+0000", 2024-08-17T18:02:04.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:04.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:02:04.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.961999+0000", 2024-08-17T18:02:04.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:04.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:02:04.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.961998+0000", 2024-08-17T18:02:04.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:02:04.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:02:04.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.962003+0000", 2024-08-17T18:02:04.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3430000000000003e-06 2024-08-17T18:02:04.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:02:04.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.962017+0000", 2024-08-17T18:02:04.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3643e-05 2024-08-17T18:02:04.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:02:04.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.962024+0000", 2024-08-17T18:02:04.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.849e-06 2024-08-17T18:02:04.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.035869+0000", 2024-08-17T18:02:04.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.073845659999999994 2024-08-17T18:02:04.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:02:04.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.039312+0000", 2024-08-17T18:02:04.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003442905 2024-08-17T18:02:04.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.058190+0000", 2024-08-17T18:02:04.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018877535000000001 2024-08-17T18:02:04.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:02:04.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.061806+0000", 2024-08-17T18:02:04.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003616452 2024-08-17T18:02:04.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542821+0000", 2024-08-17T18:02:04.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.48101451699999997 2024-08-17T18:02:04.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:02:04.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542827+0000", 2024-08-17T18:02:04.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9739999999999999e-06 2024-08-17T18:02:04.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479734+0000", 2024-08-17T18:02:04.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93690761600000005 2024-08-17T18:02:04.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:02:04.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479743+0000", 2024-08-17T18:02:04.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9619999999999999e-06 2024-08-17T18:02:04.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.854680+0000", 2024-08-17T18:02:04.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3749364169999998 2024-08-17T18:02:04.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:02:04.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.860366+0000", 2024-08-17T18:02:04.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056863520000000004 2024-08-17T18:02:04.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.882033+0000", 2024-08-17T18:02:04.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021666920999999999 2024-08-17T18:02:04.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:02:04.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.883991+0000", 2024-08-17T18:02:04.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001958146 2024-08-17T18:02:04.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:02:04.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.888022+0000", 2024-08-17T18:02:04.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040313129999999999 2024-08-17T18:02:04.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:02:04.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.909504+0000", 2024-08-17T18:02:04.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021482015 2024-08-17T18:02:04.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:04.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.910210+0000", 2024-08-17T18:02:04.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00070519400000000002 2024-08-17T18:02:04.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:04.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.929264+0000", 2024-08-17T18:02:04.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019054252000000001 2024-08-17T18:02:04.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:02:04.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.930507+0000", 2024-08-17T18:02:04.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012434029999999999 2024-08-17T18:02:04.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:02:04.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.934781+0000", 2024-08-17T18:02:04.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042741080000000004 2024-08-17T18:02:04.979 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:04.979 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:02:04.979 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:04.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.979 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180233 54.10 54:08986526:::benchmark_data_smithi136_273969_object57348:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:02:04.979 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:22.029556+0000", 2024-08-17T18:02:04.980 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.881185969000001, 2024-08-17T18:02:04.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9954107949999997, 2024-08-17T18:02:04.980 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:02:04.980 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:02:04.980 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:02:04.980 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:02:04.980 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:02:04.980 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180233 2024-08-17T18:02:04.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.980 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:02:04.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:02:04.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.029556+0000", 2024-08-17T18:02:04.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:04.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:02:04.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.029556+0000", 2024-08-17T18:02:04.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:04.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:02:04.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.029555+0000", 2024-08-17T18:02:04.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:02:04.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:02:04.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.029561+0000", 2024-08-17T18:02:04.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0349999999999998e-06 2024-08-17T18:02:04.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:02:04.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.029575+0000", 2024-08-17T18:02:04.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4566000000000001e-05 2024-08-17T18:02:04.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:02:04.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.029583+0000", 2024-08-17T18:02:04.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9330000000000001e-06 2024-08-17T18:02:04.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.040674+0000", 2024-08-17T18:02:04.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011090546999999999 2024-08-17T18:02:04.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:02:04.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.044717+0000", 2024-08-17T18:02:04.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040431340000000003 2024-08-17T18:02:04.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.075210+0000", 2024-08-17T18:02:04.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.030493089000000001 2024-08-17T18:02:04.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:02:04.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.078485+0000", 2024-08-17T18:02:04.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032751820000000002 2024-08-17T18:02:04.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542915+0000", 2024-08-17T18:02:04.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.46442982199999999 2024-08-17T18:02:04.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:02:04.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542921+0000", 2024-08-17T18:02:04.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7710000000000003e-06 2024-08-17T18:02:04.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479864+0000", 2024-08-17T18:02:04.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93694376899999998 2024-08-17T18:02:04.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:02:04.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479873+0000", 2024-08-17T18:02:04.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7870000000000007e-06 2024-08-17T18:02:04.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.948551+0000", 2024-08-17T18:02:04.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4686778700000001 2024-08-17T18:02:04.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:02:04.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.954417+0000", 2024-08-17T18:02:04.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058657630000000004 2024-08-17T18:02:04.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.974711+0000", 2024-08-17T18:02:04.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.02029392 2024-08-17T18:02:04.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:02:04.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.976548+0000", 2024-08-17T18:02:04.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018375240000000001 2024-08-17T18:02:04.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:02:04.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.980434+0000", 2024-08-17T18:02:04.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038862089999999998 2024-08-17T18:02:04.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:02:04.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.000907+0000", 2024-08-17T18:02:04.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020472305999999999 2024-08-17T18:02:04.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:04.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.001622+0000", 2024-08-17T18:02:04.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00071567799999999997 2024-08-17T18:02:04.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:04.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.019920+0000", 2024-08-17T18:02:04.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018297686000000001 2024-08-17T18:02:04.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:02:04.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.020902+0000", 2024-08-17T18:02:04.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00098207099999999994 2024-08-17T18:02:04.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:02:04.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.024967+0000", 2024-08-17T18:02:04.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040647970000000002 2024-08-17T18:02:04.989 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:04.990 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:02:04.990 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:04.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.990 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180273 54.10 54:0f06273e:::benchmark_data_smithi136_273969_object57388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:02:04.990 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:22.143352+0000", 2024-08-17T18:02:04.990 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.767390636000002, 2024-08-17T18:02:04.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7034936619999996, 2024-08-17T18:02:04.990 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:02:04.990 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:02:04.990 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:02:04.990 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:02:04.991 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:02:04.991 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180273 2024-08-17T18:02:04.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.991 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:02:04.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:02:04.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.143352+0000", 2024-08-17T18:02:04.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:04.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:02:04.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.143352+0000", 2024-08-17T18:02:04.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:04.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:02:04.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.143350+0000", 2024-08-17T18:02:04.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:02:04.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:02:04.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.143356+0000", 2024-08-17T18:02:04.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.575e-06 2024-08-17T18:02:04.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:02:04.993 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:04.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.143367+0000", 2024-08-17T18:02:04.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.116e-05 2024-08-17T18:02:04.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:02:04.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.143373+0000", 2024-08-17T18:02:04.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5450000000000003e-06 2024-08-17T18:02:04.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.163881+0000", 2024-08-17T18:02:04.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020508683 2024-08-17T18:02:04.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:02:04.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.167050+0000", 2024-08-17T18:02:04.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031688659999999998 2024-08-17T18:02:04.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.192396+0000", 2024-08-17T18:02:04.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.025345660999999998 2024-08-17T18:02:04.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:02:04.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.196588+0000", 2024-08-17T18:02:04.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041921679999999996 2024-08-17T18:02:04.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542759+0000", 2024-08-17T18:02:04.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.34617123100000002 2024-08-17T18:02:04.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:02:04.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542766+0000", 2024-08-17T18:02:04.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0999999999999998e-06 2024-08-17T18:02:04.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479644+0000", 2024-08-17T18:02:04.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93687755900000003 2024-08-17T18:02:04.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:02:04.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479655+0000", 2024-08-17T18:02:04.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1167e-05 2024-08-17T18:02:04.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.763829+0000", 2024-08-17T18:02:04.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2841733739999999 2024-08-17T18:02:04.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:02:04.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.769270+0000", 2024-08-17T18:02:04.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005441085 2024-08-17T18:02:04.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:04.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.804951+0000", 2024-08-17T18:02:04.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035681601 2024-08-17T18:02:04.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:04.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:04.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:02:05.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.806790+0000", 2024-08-17T18:02:05.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018387869999999999 2024-08-17T18:02:05.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:02:05.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.810598+0000", 2024-08-17T18:02:05.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038084780000000001 2024-08-17T18:02:05.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:02:05.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.823793+0000", 2024-08-17T18:02:05.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013195014 2024-08-17T18:02:05.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:05.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.840554+0000", 2024-08-17T18:02:05.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016760048999999999 2024-08-17T18:02:05.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:05.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.841188+0000", 2024-08-17T18:02:05.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063486499999999997 2024-08-17T18:02:05.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:02:05.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.842214+0000", 2024-08-17T18:02:05.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001025417 2024-08-17T18:02:05.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:02:05.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.846845+0000", 2024-08-17T18:02:05.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00463139 2024-08-17T18:02:05.002 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.002 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:02:05.002 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.002 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180362 54.10 54:0f5741ac:::benchmark_data_smithi136_273969_object57477:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:02:05.003 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:22.267742+0000", 2024-08-17T18:02:05.003 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.642999906999997, 2024-08-17T18:02:05.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5718885580000004, 2024-08-17T18:02:05.003 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:02:05.003 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:02:05.003 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:02:05.003 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:02:05.003 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:02:05.003 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180362 2024-08-17T18:02:05.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.003 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:02:05.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:02:05.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.267742+0000", 2024-08-17T18:02:05.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:05.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:02:05.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.267742+0000", 2024-08-17T18:02:05.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:05.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:02:05.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.267741+0000", 2024-08-17T18:02:05.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:02:05.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:02:05.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.267745+0000", 2024-08-17T18:02:05.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6969999999999999e-06 2024-08-17T18:02:05.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:02:05.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.267753+0000", 2024-08-17T18:02:05.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2099999999999993e-06 2024-08-17T18:02:05.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:02:05.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.267759+0000", 2024-08-17T18:02:05.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.293e-06 2024-08-17T18:02:05.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.297122+0000", 2024-08-17T18:02:05.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.029363607999999999 2024-08-17T18:02:05.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:02:05.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.300774+0000", 2024-08-17T18:02:05.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036522759999999999 2024-08-17T18:02:05.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.328199+0000", 2024-08-17T18:02:05.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.027424474000000001 2024-08-17T18:02:05.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:02:05.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.332945+0000", 2024-08-17T18:02:05.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004745567 2024-08-17T18:02:05.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542721+0000", 2024-08-17T18:02:05.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20977694799999999 2024-08-17T18:02:05.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:02:05.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542728+0000", 2024-08-17T18:02:05.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7900000000000002e-06 2024-08-17T18:02:05.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479480+0000", 2024-08-17T18:02:05.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93675217300000002 2024-08-17T18:02:05.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:02:05.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479490+0000", 2024-08-17T18:02:05.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2070000000000001e-06 2024-08-17T18:02:05.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.757950+0000", 2024-08-17T18:02:05.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2784603570000002 2024-08-17T18:02:05.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:02:05.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.763478+0000", 2024-08-17T18:02:05.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055275100000000002 2024-08-17T18:02:05.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.784742+0000", 2024-08-17T18:02:05.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021264387999999999 2024-08-17T18:02:05.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:02:05.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.787023+0000", 2024-08-17T18:02:05.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022810869999999998 2024-08-17T18:02:05.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:02:05.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.791977+0000", 2024-08-17T18:02:05.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049536820000000001 2024-08-17T18:02:05.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:02:05.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.804625+0000", 2024-08-17T18:02:05.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012648453 2024-08-17T18:02:05.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:05.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.814553+0000", 2024-08-17T18:02:05.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0099276380000000008 2024-08-17T18:02:05.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:05.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.834303+0000", 2024-08-17T18:02:05.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01975037 2024-08-17T18:02:05.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:02:05.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.835353+0000", 2024-08-17T18:02:05.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010493869999999999 2024-08-17T18:02:05.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:02:05.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.839631+0000", 2024-08-17T18:02:05.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042783090000000001 2024-08-17T18:02:05.013 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.013 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:02:05.013 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.013 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180378 54.10 54:08c0d9ad:::benchmark_data_smithi136_273969_object57493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:02:05.013 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:22.286706+0000", 2024-08-17T18:02:05.013 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.624036347000001, 2024-08-17T18:02:05.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.661461214, 2024-08-17T18:02:05.014 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:02:05.014 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:02:05.014 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:02:05.014 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:02:05.014 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:02:05.014 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180378 2024-08-17T18:02:05.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.014 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:02:05.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:02:05.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.286706+0000", 2024-08-17T18:02:05.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:05.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:02:05.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.286706+0000", 2024-08-17T18:02:05.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:05.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:02:05.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.286704+0000", 2024-08-17T18:02:05.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:02:05.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:02:05.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.286712+0000", 2024-08-17T18:02:05.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7980000000000006e-06 2024-08-17T18:02:05.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:02:05.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.286727+0000", 2024-08-17T18:02:05.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4842e-05 2024-08-17T18:02:05.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:02:05.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.286735+0000", 2024-08-17T18:02:05.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6899999999999992e-06 2024-08-17T18:02:05.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.358257+0000", 2024-08-17T18:02:05.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071522756000000007 2024-08-17T18:02:05.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:02:05.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.362217+0000", 2024-08-17T18:02:05.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039599190000000001 2024-08-17T18:02:05.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.417476+0000", 2024-08-17T18:02:05.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.055258920000000003 2024-08-17T18:02:05.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:02:05.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.421718+0000", 2024-08-17T18:02:05.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042413160000000002 2024-08-17T18:02:05.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542885+0000", 2024-08-17T18:02:05.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.121167504 2024-08-17T18:02:05.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:02:05.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542890+0000", 2024-08-17T18:02:05.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8679999999999998e-06 2024-08-17T18:02:05.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479822+0000", 2024-08-17T18:02:05.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93693204200000002 2024-08-17T18:02:05.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:02:05.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479834+0000", 2024-08-17T18:02:05.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2006e-05 2024-08-17T18:02:05.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.867421+0000", 2024-08-17T18:02:05.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3875873570000001 2024-08-17T18:02:05.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:02:05.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.873105+0000", 2024-08-17T18:02:05.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056837179999999999 2024-08-17T18:02:05.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.920008+0000", 2024-08-17T18:02:05.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.046902417000000002 2024-08-17T18:02:05.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:02:05.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.921823+0000", 2024-08-17T18:02:05.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001815821 2024-08-17T18:02:05.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:02:05.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.925572+0000", 2024-08-17T18:02:05.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037488589999999998 2024-08-17T18:02:05.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:02:05.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.935695+0000", 2024-08-17T18:02:05.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010123036 2024-08-17T18:02:05.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:05.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.942233+0000", 2024-08-17T18:02:05.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0065380569999999999 2024-08-17T18:02:05.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:05.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.942843+0000", 2024-08-17T18:02:05.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00060995600000000004 2024-08-17T18:02:05.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:02:05.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.943910+0000", 2024-08-17T18:02:05.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010668450000000001 2024-08-17T18:02:05.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:02:05.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.948167+0000", 2024-08-17T18:02:05.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042569469999999996 2024-08-17T18:02:05.023 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.023 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:02:05.024 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.024 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180538 54.2 54:4084e074:::benchmark_data_smithi136_273969_object57653:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:02:05.024 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:22.472724+0000", 2024-08-17T18:02:05.024 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.438018206000002, 2024-08-17T18:02:05.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.083849345, 2024-08-17T18:02:05.024 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:02:05.024 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:02:05.024 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:02:05.024 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:02:05.024 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:02:05.024 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180538 2024-08-17T18:02:05.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.025 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:02:05.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:02:05.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.472724+0000", 2024-08-17T18:02:05.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:05.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:02:05.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.472724+0000", 2024-08-17T18:02:05.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:05.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:02:05.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.472723+0000", 2024-08-17T18:02:05.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:02:05.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:02:05.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.472729+0000", 2024-08-17T18:02:05.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3400000000000003e-06 2024-08-17T18:02:05.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:02:05.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.472742+0000", 2024-08-17T18:02:05.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2495e-05 2024-08-17T18:02:05.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:02:05.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.472749+0000", 2024-08-17T18:02:05.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9909999999999996e-06 2024-08-17T18:02:05.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.472789+0000", 2024-08-17T18:02:05.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9968000000000001e-05 2024-08-17T18:02:05.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:02:05.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.472896+0000", 2024-08-17T18:02:05.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010768900000000001 2024-08-17T18:02:05.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-08-17T18:02:05.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.473437+0000", 2024-08-17T18:02:05.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00054072199999999999 2024-08-17T18:02:05.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.483366+0000", 2024-08-17T18:02:05.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0099294789999999997 2024-08-17T18:02:05.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:02:05.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.483382+0000", 2024-08-17T18:02:05.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5379999999999998e-05 2024-08-17T18:02:05.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.480196+0000", 2024-08-17T18:02:05.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99681430500000001 2024-08-17T18:02:05.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:02:05.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.480206+0000", 2024-08-17T18:02:05.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0399e-05 2024-08-17T18:02:05.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.552647+0000", 2024-08-17T18:02:05.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0724403659999999 2024-08-17T18:02:05.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:02:05.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.552777+0000", 2024-08-17T18:02:05.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013025999999999999 2024-08-17T18:02:05.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.556429+0000", 2024-08-17T18:02:05.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003651886 2024-08-17T18:02:05.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:02:05.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.556573+0000", 2024-08-17T18:02:05.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000144325 2024-08-17T18:02:05.031 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.031 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:02:05.031 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.031 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180549 54.10 54:0ef7c369:::benchmark_data_smithi136_273969_object57664:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1399)", 2024-08-17T18:02:05.032 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:22.501329+0000", 2024-08-17T18:02:05.032 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.409413315000002, 2024-08-17T18:02:05.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5377924729999997, 2024-08-17T18:02:05.032 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:02:05.032 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:02:05.032 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:02:05.032 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:02:05.032 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:02:05.032 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180549 2024-08-17T18:02:05.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.032 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:02:05.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:02:05.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.501329+0000", 2024-08-17T18:02:05.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:05.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:02:05.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.501329+0000", 2024-08-17T18:02:05.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:05.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:02:05.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.501327+0000", 2024-08-17T18:02:05.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:02:05.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:02:05.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.501334+0000", 2024-08-17T18:02:05.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.764e-06 2024-08-17T18:02:05.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:02:05.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.501347+0000", 2024-08-17T18:02:05.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2947e-05 2024-08-17T18:02:05.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:02:05.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.501354+0000", 2024-08-17T18:02:05.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8469999999999998e-06 2024-08-17T18:02:05.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542971+0000", 2024-08-17T18:02:05.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.041617526000000002 2024-08-17T18:02:05.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:02:05.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542977+0000", 2024-08-17T18:02:05.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.288e-06 2024-08-17T18:02:05.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479935+0000", 2024-08-17T18:02:05.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93695844900000003 2024-08-17T18:02:05.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:02:05.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479943+0000", 2024-08-17T18:02:05.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6909999999999997e-06 2024-08-17T18:02:05.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.960597+0000", 2024-08-17T18:02:05.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.480653856 2024-08-17T18:02:05.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:02:05.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.965790+0000", 2024-08-17T18:02:05.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051937190000000003 2024-08-17T18:02:05.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.010563+0000", 2024-08-17T18:02:05.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.044773043999999998 2024-08-17T18:02:05.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:02:05.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.012451+0000", 2024-08-17T18:02:05.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001887108 2024-08-17T18:02:05.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:02:05.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.016301+0000", 2024-08-17T18:02:05.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038504619999999998 2024-08-17T18:02:05.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:02:05.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.032206+0000", 2024-08-17T18:02:05.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015904795999999999 2024-08-17T18:02:05.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:05.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.032839+0000", 2024-08-17T18:02:05.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063271599999999996 2024-08-17T18:02:05.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:05.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.033465+0000", 2024-08-17T18:02:05.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062640899999999997 2024-08-17T18:02:05.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:02:05.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.034490+0000", 2024-08-17T18:02:05.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001025443 2024-08-17T18:02:05.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:02:05.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.039121+0000", 2024-08-17T18:02:05.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046309389999999997 2024-08-17T18:02:05.042 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.042 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:02:05.042 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.042 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180580 54.10 54:08b7bb9d:::benchmark_data_smithi136_273969_object57695:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1401)", 2024-08-17T18:02:05.043 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:24.596431+0000", 2024-08-17T18:02:05.043 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 40.314311259999997, 2024-08-17T18:02:05.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6123190539999999, 2024-08-17T18:02:05.043 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:02:05.043 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:02:05.043 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:02:05.043 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:02:05.043 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:02:05.043 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180580 2024-08-17T18:02:05.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.043 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:02:05.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:02:05.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.596431+0000", 2024-08-17T18:02:05.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:05.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:02:05.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.596431+0000", 2024-08-17T18:02:05.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:05.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:02:05.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.596429+0000", 2024-08-17T18:02:05.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:02:05.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:02:05.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.596436+0000", 2024-08-17T18:02:05.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2740000000000004e-06 2024-08-17T18:02:05.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:02:05.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.596449+0000", 2024-08-17T18:02:05.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3514e-05 2024-08-17T18:02:05.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:02:05.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.596460+0000", 2024-08-17T18:02:05.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1038e-05 2024-08-17T18:02:05.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.039419+0000", 2024-08-17T18:02:05.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.44295865 2024-08-17T18:02:05.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:02:05.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.042353+0000", 2024-08-17T18:02:05.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002934641 2024-08-17T18:02:05.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.060793+0000", 2024-08-17T18:02:05.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018439779 2024-08-17T18:02:05.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:02:05.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.062127+0000", 2024-08-17T18:02:05.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001334212 2024-08-17T18:02:05.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.189141+0000", 2024-08-17T18:02:05.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.127013878 2024-08-17T18:02:05.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:02:05.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.190968+0000", 2024-08-17T18:02:05.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001826261 2024-08-17T18:02:05.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:02:05.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.194933+0000", 2024-08-17T18:02:05.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039653300000000004 2024-08-17T18:02:05.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:05.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.201866+0000", 2024-08-17T18:02:05.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069329919999999998 2024-08-17T18:02:05.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:05.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.202483+0000", 2024-08-17T18:02:05.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00061715800000000003 2024-08-17T18:02:05.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:02:05.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.202683+0000", 2024-08-17T18:02:05.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000199507 2024-08-17T18:02:05.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:02:05.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.203739+0000", 2024-08-17T18:02:05.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010565570000000001 2024-08-17T18:02:05.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:02:05.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.208750+0000", 2024-08-17T18:02:05.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005010848 2024-08-17T18:02:05.050 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.050 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:02:05.050 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.051 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180607 54.10 54:0f7a5843:::benchmark_data_smithi136_273969_object57722:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1401)", 2024-08-17T18:02:05.051 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:24.627171+0000", 2024-08-17T18:02:05.051 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 40.283571571000003, 2024-08-17T18:02:05.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6376815360000001, 2024-08-17T18:02:05.051 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:02:05.051 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:02:05.051 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:02:05.051 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:02:05.051 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:02:05.051 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180607 2024-08-17T18:02:05.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.051 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:02:05.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:02:05.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.627171+0000", 2024-08-17T18:02:05.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:05.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:02:05.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.627171+0000", 2024-08-17T18:02:05.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:05.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:02:05.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.627169+0000", 2024-08-17T18:02:05.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:02:05.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:02:05.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.627175+0000", 2024-08-17T18:02:05.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3569999999999997e-06 2024-08-17T18:02:05.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:02:05.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.627185+0000", 2024-08-17T18:02:05.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0281e-05 2024-08-17T18:02:05.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:02:05.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.627192+0000", 2024-08-17T18:02:05.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4899999999999997e-06 2024-08-17T18:02:05.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.042641+0000", 2024-08-17T18:02:05.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4154490289999999 2024-08-17T18:02:05.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:02:05.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.045434+0000", 2024-08-17T18:02:05.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027932120000000002 2024-08-17T18:02:05.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.084159+0000", 2024-08-17T18:02:05.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038724959000000003 2024-08-17T18:02:05.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:02:05.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.085413+0000", 2024-08-17T18:02:05.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001254072 2024-08-17T18:02:05.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.239127+0000", 2024-08-17T18:02:05.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15371395600000001 2024-08-17T18:02:05.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:02:05.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.240998+0000", 2024-08-17T18:02:05.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001870752 2024-08-17T18:02:05.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:02:05.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.245523+0000", 2024-08-17T18:02:05.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045257980000000001 2024-08-17T18:02:05.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:02:05.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.257833+0000", 2024-08-17T18:02:05.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012309672000000001 2024-08-17T18:02:05.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:05.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.258606+0000", 2024-08-17T18:02:05.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00077326900000000004 2024-08-17T18:02:05.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:05.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.259327+0000", 2024-08-17T18:02:05.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00072101700000000001 2024-08-17T18:02:05.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:02:05.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.260405+0000", 2024-08-17T18:02:05.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001077176 2024-08-17T18:02:05.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:02:05.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.264852+0000", 2024-08-17T18:02:05.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044476170000000001 2024-08-17T18:02:05.058 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.058 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:02:05.058 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.059 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180664 54.10 54:0d9859e3:::benchmark_data_smithi136_273969_object57779:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1401)", 2024-08-17T18:02:05.059 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:24.698340+0000", 2024-08-17T18:02:05.059 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 40.212401999000001, 2024-08-17T18:02:05.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5592268169999999, 2024-08-17T18:02:05.059 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:02:05.059 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:02:05.059 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:02:05.059 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:02:05.059 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:02:05.059 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180664 2024-08-17T18:02:05.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.060 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:02:05.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:02:05.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.698340+0000", 2024-08-17T18:02:05.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:05.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:02:05.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.698340+0000", 2024-08-17T18:02:05.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:05.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:02:05.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.698339+0000", 2024-08-17T18:02:05.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:02:05.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:02:05.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.698345+0000", 2024-08-17T18:02:05.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3289999999999999e-06 2024-08-17T18:02:05.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:02:05.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.698359+0000", 2024-08-17T18:02:05.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3645999999999999e-05 2024-08-17T18:02:05.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:02:05.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.698365+0000", 2024-08-17T18:02:05.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7120000000000003e-06 2024-08-17T18:02:05.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.046290+0000", 2024-08-17T18:02:05.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.347924678 2024-08-17T18:02:05.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:02:05.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.048759+0000", 2024-08-17T18:02:05.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002469097 2024-08-17T18:02:05.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.072793+0000", 2024-08-17T18:02:05.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.024033968999999999 2024-08-17T18:02:05.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:02:05.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.074098+0000", 2024-08-17T18:02:05.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001304454 2024-08-17T18:02:05.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.220487+0000", 2024-08-17T18:02:05.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.14638969499999999 2024-08-17T18:02:05.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:02:05.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.222607+0000", 2024-08-17T18:02:05.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021193789999999998 2024-08-17T18:02:05.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:02:05.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.226749+0000", 2024-08-17T18:02:05.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041426019999999996 2024-08-17T18:02:05.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:05.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.249844+0000", 2024-08-17T18:02:05.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023095075 2024-08-17T18:02:05.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:02:05.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.250114+0000", 2024-08-17T18:02:05.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027006099999999999 2024-08-17T18:02:05.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:05.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.250948+0000", 2024-08-17T18:02:05.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00083311500000000005 2024-08-17T18:02:05.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:02:05.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.252283+0000", 2024-08-17T18:02:05.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013353950000000001 2024-08-17T18:02:05.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:02:05.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.257567+0000", 2024-08-17T18:02:05.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052840910000000003 2024-08-17T18:02:05.067 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.067 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:02:05.067 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.067 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180787 54.10 54:0e855b4b:::benchmark_data_smithi136_273969_object57902:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1401)", 2024-08-17T18:02:05.067 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:24.891973+0000", 2024-08-17T18:02:05.067 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 40.018769075000002, 2024-08-17T18:02:05.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4466802329999999, 2024-08-17T18:02:05.067 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:02:05.067 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:02:05.068 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:02:05.068 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:02:05.068 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:02:05.068 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180787 2024-08-17T18:02:05.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.068 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:02:05.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:02:05.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.891973+0000", 2024-08-17T18:02:05.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:05.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:02:05.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.891973+0000", 2024-08-17T18:02:05.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:05.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:02:05.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.891972+0000", 2024-08-17T18:02:05.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:02:05.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:02:05.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.891978+0000", 2024-08-17T18:02:05.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.489e-06 2024-08-17T18:02:05.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:02:05.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.891989+0000", 2024-08-17T18:02:05.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0814e-05 2024-08-17T18:02:05.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:02:05.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.891994+0000", 2024-08-17T18:02:05.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5849999999999999e-06 2024-08-17T18:02:05.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.086323+0000", 2024-08-17T18:02:05.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1943286199999998 2024-08-17T18:02:05.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:02:05.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.088618+0000", 2024-08-17T18:02:05.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022949540000000001 2024-08-17T18:02:05.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.103459+0000", 2024-08-17T18:02:05.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014841294 2024-08-17T18:02:05.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:02:05.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.104663+0000", 2024-08-17T18:02:05.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001203204 2024-08-17T18:02:05.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.281808+0000", 2024-08-17T18:02:05.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17714501499999999 2024-08-17T18:02:05.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:02:05.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.284980+0000", 2024-08-17T18:02:05.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031723739999999999 2024-08-17T18:02:05.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:02:05.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.289946+0000", 2024-08-17T18:02:05.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049663190000000003 2024-08-17T18:02:05.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:05.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.313491+0000", 2024-08-17T18:02:05.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023544820000000001 2024-08-17T18:02:05.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:02:05.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.313720+0000", 2024-08-17T18:02:05.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022937200000000001 2024-08-17T18:02:05.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:05.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.333229+0000", 2024-08-17T18:02:05.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019508584999999998 2024-08-17T18:02:05.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:02:05.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.334230+0000", 2024-08-17T18:02:05.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010011379999999999 2024-08-17T18:02:05.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:02:05.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.338653+0000", 2024-08-17T18:02:05.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044231590000000003 2024-08-17T18:02:05.075 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.075 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:02:05.075 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.075 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180832 54.10 54:0c4a64fd:::benchmark_data_smithi136_273969_object57947:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1401)", 2024-08-17T18:02:05.075 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:24.976914+0000", 2024-08-17T18:02:05.075 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.933828484999999, 2024-08-17T18:02:05.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3778381899999999, 2024-08-17T18:02:05.076 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:02:05.076 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:02:05.076 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:02:05.076 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:02:05.076 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:02:05.076 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180832 2024-08-17T18:02:05.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.076 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:02:05.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:02:05.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.976914+0000", 2024-08-17T18:02:05.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:05.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:02:05.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.976914+0000", 2024-08-17T18:02:05.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:05.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:02:05.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.976912+0000", 2024-08-17T18:02:05.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:02:05.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:02:05.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.976918+0000", 2024-08-17T18:02:05.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9540000000000001e-06 2024-08-17T18:02:05.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:02:05.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.976932+0000", 2024-08-17T18:02:05.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3726999999999999e-05 2024-08-17T18:02:05.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:02:05.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.976939+0000", 2024-08-17T18:02:05.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6499999999999999e-06 2024-08-17T18:02:05.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.088884+0000", 2024-08-17T18:02:05.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.111945757 2024-08-17T18:02:05.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:02:05.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.091124+0000", 2024-08-17T18:02:05.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022394630000000001 2024-08-17T18:02:05.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.123802+0000", 2024-08-17T18:02:05.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.032678076 2024-08-17T18:02:05.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:02:05.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.125022+0000", 2024-08-17T18:02:05.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012200049999999999 2024-08-17T18:02:05.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.322996+0000", 2024-08-17T18:02:05.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.19797377199999999 2024-08-17T18:02:05.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:02:05.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.324908+0000", 2024-08-17T18:02:05.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019125119999999999 2024-08-17T18:02:05.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:02:05.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.328896+0000", 2024-08-17T18:02:05.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039873460000000001 2024-08-17T18:02:05.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:02:05.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.339470+0000", 2024-08-17T18:02:05.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010574225 2024-08-17T18:02:05.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:05.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.346638+0000", 2024-08-17T18:02:05.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071680119999999996 2024-08-17T18:02:05.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:05.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.347438+0000", 2024-08-17T18:02:05.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00079986399999999996 2024-08-17T18:02:05.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:02:05.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.348684+0000", 2024-08-17T18:02:05.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012465530000000001 2024-08-17T18:02:05.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:02:05.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.354752+0000", 2024-08-17T18:02:05.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060676560000000003 2024-08-17T18:02:05.083 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.083 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:02:05.084 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.084 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180936 54.10 54:0b5f349a:::benchmark_data_smithi136_273969_object58051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1401)", 2024-08-17T18:02:05.084 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:25.343460+0000", 2024-08-17T18:02:05.084 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.567281856000001, 2024-08-17T18:02:05.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0025335470000001, 2024-08-17T18:02:05.084 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:02:05.084 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:02:05.084 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:02:05.084 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:02:05.084 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:02:05.084 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180936 2024-08-17T18:02:05.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.085 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:02:05.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:02:05.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.343460+0000", 2024-08-17T18:02:05.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:05.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:02:05.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.343460+0000", 2024-08-17T18:02:05.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:05.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:02:05.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.343459+0000", 2024-08-17T18:02:05.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:02:05.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:02:05.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.343466+0000", 2024-08-17T18:02:05.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8820000000000001e-06 2024-08-17T18:02:05.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:02:05.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.343479+0000", 2024-08-17T18:02:05.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.306e-05 2024-08-17T18:02:05.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:02:05.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.343488+0000", 2024-08-17T18:02:05.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6330000000000002e-06 2024-08-17T18:02:05.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.091459+0000", 2024-08-17T18:02:05.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7479711390000001 2024-08-17T18:02:05.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:02:05.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.093708+0000", 2024-08-17T18:02:05.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022491490000000002 2024-08-17T18:02:05.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.113505+0000", 2024-08-17T18:02:05.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019796971999999999 2024-08-17T18:02:05.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:02:05.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.114725+0000", 2024-08-17T18:02:05.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012199940000000001 2024-08-17T18:02:05.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.303446+0000", 2024-08-17T18:02:05.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18872046000000001 2024-08-17T18:02:05.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:02:05.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.305293+0000", 2024-08-17T18:02:05.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018468219999999999 2024-08-17T18:02:05.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:02:05.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.309465+0000", 2024-08-17T18:02:05.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041719280000000001 2024-08-17T18:02:05.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:02:05.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.332589+0000", 2024-08-17T18:02:05.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023124150999999999 2024-08-17T18:02:05.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:05.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.339278+0000", 2024-08-17T18:02:05.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006689632 2024-08-17T18:02:05.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:05.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.340076+0000", 2024-08-17T18:02:05.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00079814199999999995 2024-08-17T18:02:05.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:02:05.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.341136+0000", 2024-08-17T18:02:05.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010596169999999999 2024-08-17T18:02:05.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:02:05.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.345994+0000", 2024-08-17T18:02:05.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004857765 2024-08-17T18:02:05.092 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.092 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:02:05.092 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.092 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGRecoveryDeleteReply(54.10 e1401,1400 54:0f9da3af:::benchmark_data_smithi136_273969_object57231:head,1398'8529)", 2024-08-17T18:02:05.092 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:25.545611+0000", 2024-08-17T18:02:05.092 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.365131640000001, 2024-08-17T18:02:05.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.201322319, 2024-08-17T18:02:05.093 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:02:05.093 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:02:05.093 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:02:05.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:02:05.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.545611+0000", 2024-08-17T18:02:05.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:05.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:02:05.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.545611+0000", 2024-08-17T18:02:05.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:05.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:02:05.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.545610+0000", 2024-08-17T18:02:05.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:02:05.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:02:05.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.545620+0000", 2024-08-17T18:02:05.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0526000000000001e-05 2024-08-17T18:02:05.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:02:05.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.545632+0000", 2024-08-17T18:02:05.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1771e-05 2024-08-17T18:02:05.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:02:05.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.545641+0000", 2024-08-17T18:02:05.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.568e-06 2024-08-17T18:02:05.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.743396+0000", 2024-08-17T18:02:05.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.197755677 2024-08-17T18:02:05.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:02:05.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.746933+0000", 2024-08-17T18:02:05.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035364770000000001 2024-08-17T18:02:05.096 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.096 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:02:05.096 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.096 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGRecoveryDeleteReply(54.10 e1401,1400 54:0f9da3af:::benchmark_data_smithi136_273969_object57231:head,1398'8529)", 2024-08-17T18:02:05.096 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:25.546013+0000", 2024-08-17T18:02:05.096 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.364729095000001, 2024-08-17T18:02:05.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.209840958, 2024-08-17T18:02:05.097 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:02:05.097 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:02:05.097 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:02:05.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:02:05.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.546013+0000", 2024-08-17T18:02:05.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:05.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:02:05.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.546013+0000", 2024-08-17T18:02:05.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:02:05.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:02:05.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.546012+0000", 2024-08-17T18:02:05.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:02:05.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:02:05.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.546017+0000", 2024-08-17T18:02:05.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5430000000000004e-06 2024-08-17T18:02:05.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:02:05.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.546028+0000", 2024-08-17T18:02:05.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1343e-05 2024-08-17T18:02:05.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:02:05.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.546034+0000", 2024-08-17T18:02:05.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.215e-06 2024-08-17T18:02:05.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:02:05.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.747175+0000", 2024-08-17T18:02:05.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.201141062 2024-08-17T18:02:05.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:02:05.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:02:05.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:02:05.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.755854+0000", 2024-08-17T18:02:05.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0086793800000000004 2024-08-17T18:02:05.100 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.100 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:02:05.100 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.100 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:02:05.100 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:02:05.101 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:02:05.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:05.101 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:05.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:05.203 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:05.242 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:05.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:05.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:05 smithi165 ceph-mon[105517]: pgmap v2456: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 264 op/s; 6846/124524 objects degraded (5.498%); 8753/124524 objects misplaced (7.029%); 3.0 MiB/s, 48 objects/s recovering 2024-08-17T18:02:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:05 smithi165 ceph-mon[105517]: 54.16 scrub starts 2024-08-17T18:02:05.344 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:05.372 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:05.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:05.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:05.393 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:05 smithi136 ceph-mon[109861]: pgmap v2456: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 264 op/s; 6846/124524 objects degraded (5.498%); 8753/124524 objects misplaced (7.029%); 3.0 MiB/s, 48 objects/s recovering 2024-08-17T18:02:05.393 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:05 smithi136 ceph-mon[109861]: 54.16 scrub starts 2024-08-17T18:02:05.393 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:05 smithi136 ceph-mon[115078]: pgmap v2456: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 264 op/s; 6846/124524 objects degraded (5.498%); 8753/124524 objects misplaced (7.029%); 3.0 MiB/s, 48 objects/s recovering 2024-08-17T18:02:05.394 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:05 smithi136 ceph-mon[115078]: 54.16 scrub starts 2024-08-17T18:02:05.474 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:05.503 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:05.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:05.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:05.605 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:05.638 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:05.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:05.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:05.740 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:05.772 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:05.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:05.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:05.873 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:05.900 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:05.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:05.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:06.002 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:06.036 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:06.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:06.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:06.086 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:02:06.115 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-17T18:02:06.138 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:06.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:06.200 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:06.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:06 smithi165 ceph-mon[105517]: 54.6 scrub starts 2024-08-17T18:02:06.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:06 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1132990472' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:02:06.289 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:06 smithi136 ceph-mon[109861]: 54.6 scrub starts 2024-08-17T18:02:06.289 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:06 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1132990472' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:02:06.289 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:06 smithi136 ceph-mon[115078]: 54.6 scrub starts 2024-08-17T18:02:06.289 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:06 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1132990472' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:02:06.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:06.327 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:06.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:06.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:06.429 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:06.475 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:06.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:06.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:06.487 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:02:06.577 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:06.598 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:06 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a[109838]: 2024-08-17T18:02:06.483+0000 7f5b8e77d640 -1 mon.a@0(leader).osd e1420 definitely_dead 0 2024-08-17T18:02:06.608 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:06.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:06.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:06.710 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:06.738 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:06.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:06.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:06.840 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:06.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:06.892 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:06.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:06.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:07.043 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:07.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:07.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:07.145 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:07.179 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:07.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:07.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:07.210 INFO:teuthology.orchestra.run.smithi136.stderr:marked out osd.5. 2024-08-17T18:02:07.281 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:07.333 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:07.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:07.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:07.435 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:07 smithi136 ceph-mon[109861]: pgmap v2457: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 239 op/s; 6846/124524 objects degraded (5.498%); 8753/124524 objects misplaced (7.029%); 2.7 MiB/s, 43 objects/s recovering 2024-08-17T18:02:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:07 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:02:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:07 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1132990472' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:02:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:07 smithi136 ceph-mon[109861]: osdmap e1420: 8 total, 8 up, 7 in 2024-08-17T18:02:07.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:02:07.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:02:07.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:07 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3391993328' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-08-17T18:02:07.448 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:07 smithi136 ceph-mon[109861]: Client client.admin marked osd.5 out, while it was still marked up 2024-08-17T18:02:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:07 smithi136 ceph-mon[115078]: pgmap v2457: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 239 op/s; 6846/124524 objects degraded (5.498%); 8753/124524 objects misplaced (7.029%); 2.7 MiB/s, 43 objects/s recovering 2024-08-17T18:02:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:07 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:02:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:07 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1132990472' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:02:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:07 smithi136 ceph-mon[115078]: osdmap e1420: 8 total, 8 up, 7 in 2024-08-17T18:02:07.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:02:07.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:02:07.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:07 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3391993328' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-08-17T18:02:07.449 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:07 smithi136 ceph-mon[115078]: Client client.admin marked osd.5 out, while it was still marked up 2024-08-17T18:02:07.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:07.468 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:07.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:07.481 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:07 smithi165 ceph-mon[105517]: pgmap v2457: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 239 op/s; 6846/124524 objects degraded (5.498%); 8753/124524 objects misplaced (7.029%); 2.7 MiB/s, 43 objects/s recovering 2024-08-17T18:02:07.481 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:07 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:02:07.481 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:07 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1132990472' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:02:07.481 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:07 smithi165 ceph-mon[105517]: osdmap e1420: 8 total, 8 up, 7 in 2024-08-17T18:02:07.482 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:02:07.482 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:02:07.482 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:07 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3391993328' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-08-17T18:02:07.482 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:07 smithi165 ceph-mon[105517]: Client client.admin marked osd.5 out, while it was still marked up 2024-08-17T18:02:07.569 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:07.597 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:07.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:07.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:07.688 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:02:07.688 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:02:07.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:07.706 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:02:07.707 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:02:07.707 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:02:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:02:07] "GET /metrics HTTP/1.1" 200 36643 "" "Prometheus/2.51.0" 2024-08-17T18:02:07.726 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:02:07.727 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:02:07.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:07.728 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:07.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:07.744 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:02:07.745 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:02:07.764 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:02:07.764 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:02:07.782 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:02:07.783 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:02:07.799 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:02:07.800 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:02:07.820 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:02:07.821 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:02:07.829 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:07.852 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:07.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:07.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:07.954 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:07.977 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:07.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:07.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:08.078 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:08.102 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:08.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:08.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:08.204 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:08.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:08.246 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:08.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:08.346 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:08.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:08 smithi165 ceph-mon[105517]: 54.16 scrub starts 2024-08-17T18:02:08.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:08 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3391993328' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-08-17T18:02:08.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:08 smithi165 ceph-mon[105517]: osdmap e1421: 8 total, 8 up, 6 in 2024-08-17T18:02:08.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:08 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2164243257' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:02:08.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:02:08.369 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:02:08.369 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:02:08.384 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:08.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:08.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:08.487 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:08.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:08.508 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:08.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:08.609 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:08.632 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:08.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:08.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:08.660 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:08 smithi136 ceph-mon[109861]: 54.16 scrub starts 2024-08-17T18:02:08.660 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:08 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3391993328' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-08-17T18:02:08.660 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:08 smithi136 ceph-mon[109861]: osdmap e1421: 8 total, 8 up, 6 in 2024-08-17T18:02:08.660 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:08 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2164243257' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:02:08.660 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:02:08.660 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:08 smithi136 ceph-mon[115078]: 54.16 scrub starts 2024-08-17T18:02:08.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:08 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3391993328' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-08-17T18:02:08.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:08 smithi136 ceph-mon[115078]: osdmap e1421: 8 total, 8 up, 6 in 2024-08-17T18:02:08.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:08 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2164243257' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:02:08.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:02:08.733 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:08.755 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:08.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:08.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:08.856 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:08.879 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:08.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:08.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:08.981 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:09.001 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:09.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:09.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:09.038 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-17T18:02:09.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:09.144 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:09.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:09.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:09.246 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:09.254 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:02:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:09 smithi165 ceph-mon[105517]: pgmap v2460: 99 pgs: 1 remapped+peering, 4 peering, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 2.3 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 237 op/s; 6591/112695 objects degraded (5.849%); 7255/112695 objects misplaced (6.438%); 3.8 MiB/s, 61 objects/s recovering 2024-08-17T18:02:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:09 smithi165 ceph-mon[105517]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-08-17T18:02:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:09 smithi165 ceph-mon[105517]: osdmap e1422: 8 total, 8 up, 6 in 2024-08-17T18:02:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:09 smithi165 ceph-mon[105517]: osdmap e1423: 8 total, 8 up, 6 in 2024-08-17T18:02:09.271 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:09.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:09.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:09.373 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:09.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:09.455 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:09.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:09.487 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:09 smithi136 ceph-mon[115078]: pgmap v2460: 99 pgs: 1 remapped+peering, 4 peering, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 2.3 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 237 op/s; 6591/112695 objects degraded (5.849%); 7255/112695 objects misplaced (6.438%); 3.8 MiB/s, 61 objects/s recovering 2024-08-17T18:02:09.487 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:09 smithi136 ceph-mon[115078]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-08-17T18:02:09.487 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:09 smithi136 ceph-mon[115078]: osdmap e1422: 8 total, 8 up, 6 in 2024-08-17T18:02:09.487 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:09 smithi136 ceph-mon[115078]: osdmap e1423: 8 total, 8 up, 6 in 2024-08-17T18:02:09.487 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:09 smithi136 ceph-mon[109861]: pgmap v2460: 99 pgs: 1 remapped+peering, 4 peering, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 2.3 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 237 op/s; 6591/112695 objects degraded (5.849%); 7255/112695 objects misplaced (6.438%); 3.8 MiB/s, 61 objects/s recovering 2024-08-17T18:02:09.487 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:09 smithi136 ceph-mon[109861]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-08-17T18:02:09.487 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:09 smithi136 ceph-mon[109861]: osdmap e1422: 8 total, 8 up, 6 in 2024-08-17T18:02:09.487 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:09 smithi136 ceph-mon[109861]: osdmap e1423: 8 total, 8 up, 6 in 2024-08-17T18:02:09.556 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:09.580 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:09.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:09.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:09.682 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:09.705 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:09.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:09.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:09.807 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:09.828 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:09.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:09.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:09.930 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:09.953 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:09.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:09.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:09.994 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:02:10.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:10.078 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:10.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:10.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:10.180 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:10.202 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:10.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:10.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:10.304 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:10.327 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:10.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:10.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:10.415 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:02:10.429 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:10.429 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:10 smithi165 ceph-mon[105517]: 54.b starting backfill to osd.6 from (0'0,0'0] MAX to 1420'12830 2024-08-17T18:02:10.429 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:10 smithi165 ceph-mon[105517]: 54.f starting backfill to osd.6 from (0'0,0'0] MAX to 1420'13129 2024-08-17T18:02:10.429 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:10 smithi165 ceph-mon[105517]: 54.d starting backfill to osd.2 from (0'0,0'0] MAX to 1420'12971 2024-08-17T18:02:10.430 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:10 smithi165 ceph-mon[105517]: 54.a starting backfill to osd.2 from (0'0,0'0] MAX to 1420'13207 2024-08-17T18:02:10.430 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:10 smithi165 ceph-mon[105517]: 54.3 starting backfill to osd.2 from (0'0,0'0] MAX to 1420'9537 2024-08-17T18:02:10.430 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:10 smithi165 ceph-mon[105517]: 54.c starting backfill to osd.6 from (0'0,0'0] MAX to 1420'12828 2024-08-17T18:02:10.430 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:10 smithi165 ceph-mon[105517]: 54.13 starting backfill to osd.6 from (0'0,0'0] MAX to 1420'9712 2024-08-17T18:02:10.430 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:10 smithi165 ceph-mon[105517]: 54.6 starting backfill to osd.6 from (0'0,0'0] MAX to 1420'9508 2024-08-17T18:02:10.430 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:10 smithi165 ceph-mon[105517]: 54.16 starting backfill to osd.6 from (0'0,0'0] MAX to 1420'9457 2024-08-17T18:02:10.430 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:10 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 6591/112695 objects degraded (5.849%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:02:10.459 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:10.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:10.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:10.546 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:10 smithi136 ceph-mon[109861]: 54.b starting backfill to osd.6 from (0'0,0'0] MAX to 1420'12830 2024-08-17T18:02:10.546 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:10 smithi136 ceph-mon[109861]: 54.f starting backfill to osd.6 from (0'0,0'0] MAX to 1420'13129 2024-08-17T18:02:10.546 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:10 smithi136 ceph-mon[109861]: 54.d starting backfill to osd.2 from (0'0,0'0] MAX to 1420'12971 2024-08-17T18:02:10.546 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:10 smithi136 ceph-mon[109861]: 54.a starting backfill to osd.2 from (0'0,0'0] MAX to 1420'13207 2024-08-17T18:02:10.546 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:10 smithi136 ceph-mon[109861]: 54.3 starting backfill to osd.2 from (0'0,0'0] MAX to 1420'9537 2024-08-17T18:02:10.546 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:10 smithi136 ceph-mon[109861]: 54.c starting backfill to osd.6 from (0'0,0'0] MAX to 1420'12828 2024-08-17T18:02:10.546 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:10 smithi136 ceph-mon[109861]: 54.13 starting backfill to osd.6 from (0'0,0'0] MAX to 1420'9712 2024-08-17T18:02:10.546 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:10 smithi136 ceph-mon[109861]: 54.6 starting backfill to osd.6 from (0'0,0'0] MAX to 1420'9508 2024-08-17T18:02:10.546 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:10 smithi136 ceph-mon[109861]: 54.16 starting backfill to osd.6 from (0'0,0'0] MAX to 1420'9457 2024-08-17T18:02:10.547 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:10 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 6591/112695 objects degraded (5.849%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:02:10.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:10 smithi136 ceph-mon[115078]: 54.b starting backfill to osd.6 from (0'0,0'0] MAX to 1420'12830 2024-08-17T18:02:10.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:10 smithi136 ceph-mon[115078]: 54.f starting backfill to osd.6 from (0'0,0'0] MAX to 1420'13129 2024-08-17T18:02:10.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:10 smithi136 ceph-mon[115078]: 54.d starting backfill to osd.2 from (0'0,0'0] MAX to 1420'12971 2024-08-17T18:02:10.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:10 smithi136 ceph-mon[115078]: 54.a starting backfill to osd.2 from (0'0,0'0] MAX to 1420'13207 2024-08-17T18:02:10.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:10 smithi136 ceph-mon[115078]: 54.3 starting backfill to osd.2 from (0'0,0'0] MAX to 1420'9537 2024-08-17T18:02:10.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:10 smithi136 ceph-mon[115078]: 54.c starting backfill to osd.6 from (0'0,0'0] MAX to 1420'12828 2024-08-17T18:02:10.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:10 smithi136 ceph-mon[115078]: 54.13 starting backfill to osd.6 from (0'0,0'0] MAX to 1420'9712 2024-08-17T18:02:10.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:10 smithi136 ceph-mon[115078]: 54.6 starting backfill to osd.6 from (0'0,0'0] MAX to 1420'9508 2024-08-17T18:02:10.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:10 smithi136 ceph-mon[115078]: 54.16 starting backfill to osd.6 from (0'0,0'0] MAX to 1420'9457 2024-08-17T18:02:10.548 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:10 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 6591/112695 objects degraded (5.849%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:02:10.561 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:10.585 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:10.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:10.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:10.687 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:10.714 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:10.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:10.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:10.815 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:10.839 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:10.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:10.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:10.941 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:10.967 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:10.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:10.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:11.068 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:11.097 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:11.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:11.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:11.198 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:11.222 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:11.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:11.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:11.324 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:11.347 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:11.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:11.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:11.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:11 smithi136 ceph-mon[109861]: pgmap v2463: 99 pgs: 1 remapped+peering, 4 peering, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 2.3 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 985 op/s; 6591/112695 objects degraded (5.849%); 7255/112695 objects misplaced (6.438%); 4.0 MiB/s, 63 objects/s recovering 2024-08-17T18:02:11.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:11 smithi136 ceph-mon[115078]: pgmap v2463: 99 pgs: 1 remapped+peering, 4 peering, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 2.3 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 985 op/s; 6591/112695 objects degraded (5.849%); 7255/112695 objects misplaced (6.438%); 4.0 MiB/s, 63 objects/s recovering 2024-08-17T18:02:11.449 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:11.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:11.475 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:11.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:11 smithi165 ceph-mon[105517]: pgmap v2463: 99 pgs: 1 remapped+peering, 4 peering, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 2.3 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 985 op/s; 6591/112695 objects degraded (5.849%); 7255/112695 objects misplaced (6.438%); 4.0 MiB/s, 63 objects/s recovering 2024-08-17T18:02:11.575 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:11.599 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:11.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:11.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:11.701 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:11.728 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:11.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:11.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:11.829 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:11.854 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:11.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:11.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:11.956 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:11.980 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:11.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:11.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:12.081 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:12.105 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:12.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:12.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:12.206 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:12.230 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:12.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:12.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:12.332 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:12.356 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:12.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:12.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:12.375 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:02:12.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:12.487 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:12.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:12.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:12.589 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:12.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:12.631 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:12.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:12.732 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:12.752 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:02:12.752 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:02:12.774 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:12.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:12.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:12.876 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:12.899 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:12.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:12.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:13.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:13.025 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:13.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:13.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:13.127 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:13.150 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:13.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:13.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:13.252 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:13.276 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:13.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:13.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:13.378 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:13.400 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:13.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:13.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:13.502 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:13 smithi165 ceph-mon[105517]: pgmap v2464: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 7 active+remapped+backfill_wait, 1 remapped+peering, 4 peering, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.2 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 797 op/s; 7819/110442 objects degraded (7.080%); 20656/110442 objects misplaced (18.703%); 15 MiB/s, 239 objects/s recovering 2024-08-17T18:02:13.526 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:13.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:13.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:13.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-17T18:02:13.628 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:13.659 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:13.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:13.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:13.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:13 smithi136 ceph-mon[109861]: pgmap v2464: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 7 active+remapped+backfill_wait, 1 remapped+peering, 4 peering, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.2 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 797 op/s; 7819/110442 objects degraded (7.080%); 20656/110442 objects misplaced (18.703%); 15 MiB/s, 239 objects/s recovering 2024-08-17T18:02:13.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:13 smithi136 ceph-mon[115078]: pgmap v2464: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 7 active+remapped+backfill_wait, 1 remapped+peering, 4 peering, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.2 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 797 op/s; 7819/110442 objects degraded (7.080%); 20656/110442 objects misplaced (18.703%); 15 MiB/s, 239 objects/s recovering 2024-08-17T18:02:13.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:13.785 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:13.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:13.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:13.887 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:13.908 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:13.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:13.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:14.010 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:14.033 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:14.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:14.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:14.135 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:14.161 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:14.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:14.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:14.263 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:14.281 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7] out_osds: [0, 5] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:02:14.281 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:02:14.281 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-08-17T18:02:14.281 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-08-17T18:02:14.288 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:14.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:14.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:14.390 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:14.417 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:14.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:14.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:14.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:14 smithi165 ceph-mon[105517]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-08-17T18:02:14.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:14 smithi165 ceph-mon[105517]: osdmap e1424: 8 total, 8 up, 6 in 2024-08-17T18:02:14.519 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:14.541 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:14.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:14.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:14.643 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:14.665 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:14.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:14.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:14.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:14 smithi136 ceph-mon[109861]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-08-17T18:02:14.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:14 smithi136 ceph-mon[109861]: osdmap e1424: 8 total, 8 up, 6 in 2024-08-17T18:02:14.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:14 smithi136 ceph-mon[115078]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-08-17T18:02:14.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:14 smithi136 ceph-mon[115078]: osdmap e1424: 8 total, 8 up, 6 in 2024-08-17T18:02:14.697 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:02:14.767 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:14.789 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:14.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:14.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:14.891 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:14.916 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:14.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:14.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:15.017 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:15.048 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:15.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:15.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:15.150 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:15.179 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:15.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:15.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:15.281 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:15.309 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:15.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:15.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:15.392 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:15 smithi165 ceph-mon[105517]: pgmap v2465: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 7 active+remapped+backfill_wait, 1 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 207 op/s; 8907/110334 objects degraded (8.073%); 20654/110334 objects misplaced (18.720%); 12 MiB/s, 189 objects/s recovering 2024-08-17T18:02:15.393 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:15 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 8907/110334 objects degraded (8.073%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T18:02:15.411 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:15.440 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:15.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:15.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:15.542 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:15.568 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:15.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:15.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:15.667 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:15 smithi136 ceph-mon[109861]: pgmap v2465: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 7 active+remapped+backfill_wait, 1 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 207 op/s; 8907/110334 objects degraded (8.073%); 20654/110334 objects misplaced (18.720%); 12 MiB/s, 189 objects/s recovering 2024-08-17T18:02:15.667 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:15 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 8907/110334 objects degraded (8.073%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T18:02:15.667 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:15 smithi136 ceph-mon[115078]: pgmap v2465: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 7 active+remapped+backfill_wait, 1 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 207 op/s; 8907/110334 objects degraded (8.073%); 20654/110334 objects misplaced (18.720%); 12 MiB/s, 189 objects/s recovering 2024-08-17T18:02:15.667 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:15 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 8907/110334 objects degraded (8.073%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T18:02:15.671 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:15.725 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:15.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:15.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:15.769 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:02:15.773 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:02:15.827 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:15.859 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:15.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:15.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:15.961 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:15.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:15.985 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:15.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:16.086 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:16.109 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:16.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:16.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:16.210 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:16.240 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:16.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:16.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:16.341 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:16.364 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:16.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:16.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:16.466 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:16.491 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:16.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:16.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:16.593 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:16.622 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:16.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:16.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:16.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:16 smithi136 ceph-mon[109861]: osdmap e1425: 8 total, 8 up, 6 in 2024-08-17T18:02:16.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:16 smithi136 ceph-mon[115078]: osdmap e1425: 8 total, 8 up, 6 in 2024-08-17T18:02:16.724 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:16.753 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:16.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:16.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:16 smithi165 ceph-mon[105517]: osdmap e1425: 8 total, 8 up, 6 in 2024-08-17T18:02:16.856 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:16.878 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:16.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:16.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:16.980 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:17.019 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:17.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:17.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:17.121 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:17.150 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:17.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:17.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:17.252 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:17.278 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:17.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:17.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:17.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:17.413 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:17.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:17.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:17.515 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:17.545 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:17.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:17.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:17.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:17.673 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:17.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:17.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:17.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:17 smithi136 ceph-mon[109861]: pgmap v2468: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 7 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 116 op/s; 10508/110334 objects degraded (9.524%); 23871/110334 objects misplaced (21.635%); 11 MiB/s, 170 objects/s recovering 2024-08-17T18:02:17.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:17 smithi136 ceph-mon[115078]: pgmap v2468: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 7 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 116 op/s; 10508/110334 objects degraded (9.524%); 23871/110334 objects misplaced (21.635%); 11 MiB/s, 170 objects/s recovering 2024-08-17T18:02:17.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:02:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:02:17] "GET /metrics HTTP/1.1" 200 36643 "" "Prometheus/2.51.0" 2024-08-17T18:02:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:17 smithi165 ceph-mon[105517]: pgmap v2468: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 7 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 116 op/s; 10508/110334 objects degraded (9.524%); 23871/110334 objects misplaced (21.635%); 11 MiB/s, 170 objects/s recovering 2024-08-17T18:02:17.775 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:17.800 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:17.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:17.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:17.902 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:17.930 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:17.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:17.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:18.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:18.055 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:18.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:18.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:18.157 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:18.184 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:18.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:18.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:18.285 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:18.320 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:18.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:18.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:18.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:18.456 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:18.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:18.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:18.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:18 smithi165 ceph-mon[105517]: osdmap e1426: 8 total, 8 up, 6 in 2024-08-17T18:02:18.558 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:18.585 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:18 smithi136 ceph-mon[109861]: osdmap e1426: 8 total, 8 up, 6 in 2024-08-17T18:02:18.585 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:18 smithi136 ceph-mon[115078]: osdmap e1426: 8 total, 8 up, 6 in 2024-08-17T18:02:18.589 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:18.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:18.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:18.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:18.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:18.734 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:18.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:18.835 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:18.864 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:18.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:18.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:18.967 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:18.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:18.995 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:18.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:19.067 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:02:19.068 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:02:19.096 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:19.125 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:19.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:19.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:19.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:19.255 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:19.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:19.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:19.335 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:02:19.357 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:19.383 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:19.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:19.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:19.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:19 smithi136 ceph-mon[109861]: pgmap v2470: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 6 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.2 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 152 op/s; 10472/107697 objects degraded (9.724%); 20066/107697 objects misplaced (18.632%); 30 MiB/s, 485 objects/s recovering 2024-08-17T18:02:19.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:19 smithi136 ceph-mon[109861]: osdmap e1427: 8 total, 8 up, 6 in 2024-08-17T18:02:19.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:19 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3991876238' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:02:19.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:19 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:02:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:19 smithi136 ceph-mon[115078]: pgmap v2470: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 6 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.2 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 152 op/s; 10472/107697 objects degraded (9.724%); 20066/107697 objects misplaced (18.632%); 30 MiB/s, 485 objects/s recovering 2024-08-17T18:02:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:19 smithi136 ceph-mon[115078]: osdmap e1427: 8 total, 8 up, 6 in 2024-08-17T18:02:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:19 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3991876238' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:02:19.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:19 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:02:19.485 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:19.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:19 smithi165 ceph-mon[105517]: pgmap v2470: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 6 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.2 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 152 op/s; 10472/107697 objects degraded (9.724%); 20066/107697 objects misplaced (18.632%); 30 MiB/s, 485 objects/s recovering 2024-08-17T18:02:19.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:19 smithi165 ceph-mon[105517]: osdmap e1427: 8 total, 8 up, 6 in 2024-08-17T18:02:19.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:19 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3991876238' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:02:19.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:19 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:02:19.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:19.516 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:19.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:19.617 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:19.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:19.648 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:19.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:19.749 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:19.775 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:19.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:19.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:19.876 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:19.905 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:19.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:19.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:20.007 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:20.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:20.043 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:20.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:20.144 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:20.175 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:20.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:20.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:20.278 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:20.309 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:20.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:20.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:20.411 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:20.436 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:20.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:20.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:20.538 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:20.575 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:20.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:20.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:20.584 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:20 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:02:20.584 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:20 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:02:20.584 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:20 smithi136 ceph-mon[109861]: osdmap e1428: 8 total, 8 up, 6 in 2024-08-17T18:02:20.584 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:20 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 10472/107697 objects degraded (9.724%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:02:20.584 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:20 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:02:20.584 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:20 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:02:20.584 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:20 smithi136 ceph-mon[115078]: osdmap e1428: 8 total, 8 up, 6 in 2024-08-17T18:02:20.584 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:20 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 10472/107697 objects degraded (9.724%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:02:20.677 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:20.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:20.710 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:20.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:20 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:02:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:20 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:02:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:20 smithi165 ceph-mon[105517]: osdmap e1428: 8 total, 8 up, 6 in 2024-08-17T18:02:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:20 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 10472/107697 objects degraded (9.724%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:02:20.811 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:20.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:20.834 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:20.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:20.935 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:20.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:20.969 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:20.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:21.070 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:21.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:21.099 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:21.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:21.200 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:21.235 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:21.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:21.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:21.336 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:21.342 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-17T18:02:21.380 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:21.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:21.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:21.482 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:21.513 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:21.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:21.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:21.615 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:21.641 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:21.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:21.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:21 smithi136 ceph-mon[109861]: pgmap v2473: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 6 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.2 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 204 op/s; 10314/107469 objects degraded (9.597%); 20071/107469 objects misplaced (18.676%); 40 MiB/s, 638 objects/s recovering 2024-08-17T18:02:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:02:21.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:02:21.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:02:21.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:02:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:21 smithi136 ceph-mon[115078]: pgmap v2473: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 6 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.2 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 204 op/s; 10314/107469 objects degraded (9.597%); 20071/107469 objects misplaced (18.676%); 40 MiB/s, 638 objects/s recovering 2024-08-17T18:02:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:02:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:02:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:02:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:02:21.698 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:02:21.743 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:21 smithi165 ceph-mon[105517]: pgmap v2473: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 6 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.2 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 204 op/s; 10314/107469 objects degraded (9.597%); 20071/107469 objects misplaced (18.676%); 40 MiB/s, 638 objects/s recovering 2024-08-17T18:02:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:02:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:02:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:02:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:02:21.771 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:21.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:21.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:21.873 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:21.897 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:21.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:21.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:21.999 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:22.026 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:22.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:22.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:22.128 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:22.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:22.159 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:22.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:22.260 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:22.285 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:22.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:22.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:22.387 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:22.411 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:22.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:22.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:22.513 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:22.540 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:22.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:22.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:22.641 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:22.670 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:22.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:22.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:22.772 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:22.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:22.803 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:22.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:22.905 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:22.932 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:22.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:22.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:23.034 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:23.071 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:23.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:23.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:23.174 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:23.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:23.212 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:23.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:23.313 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:23.342 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:23.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:23.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:23.444 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:23.469 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:23.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:23.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:23.496 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:23 smithi165 ceph-mon[105517]: pgmap v2474: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 2.1 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 447 op/s; 10314/102279 objects degraded (10.084%); 17228/102279 objects misplaced (16.844%); 49 MiB/s, 776 objects/s recovering 2024-08-17T18:02:23.571 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:23.601 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:23.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:23.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:23 smithi136 ceph-mon[109861]: pgmap v2474: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 2.1 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 447 op/s; 10314/102279 objects degraded (10.084%); 17228/102279 objects misplaced (16.844%); 49 MiB/s, 776 objects/s recovering 2024-08-17T18:02:23.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:23 smithi136 ceph-mon[115078]: pgmap v2474: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 2.1 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 447 op/s; 10314/102279 objects degraded (10.084%); 17228/102279 objects misplaced (16.844%); 49 MiB/s, 776 objects/s recovering 2024-08-17T18:02:23.703 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:23.732 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:23.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:23.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:23.796 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:02:23.796 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:02:23.834 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:23.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:23.857 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:23.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:23.958 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:23.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:23.998 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:23.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:24.099 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:24.125 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:24.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:24.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:24.227 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:24.265 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:24.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:24.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:24.367 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:24.409 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:24.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:24.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:24.511 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:24.532 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:24.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:24.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:24.634 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:24.663 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:24.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:24.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:24.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:24.787 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:24.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:24.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:24.889 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:24.917 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:24.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:24.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:25.019 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:25.043 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:25.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:25.045 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-17T18:02:25.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:25.146 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:25.169 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:25.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:25.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:25.270 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:25.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:25.294 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:25.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:25.310 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:02:25.395 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:25.426 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:25.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:25.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:25.528 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:25.571 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:25.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:25.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:25.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:25 smithi165 ceph-mon[105517]: pgmap v2475: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 2.1 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 355 op/s; 10240/101217 objects degraded (10.117%); 17153/101217 objects misplaced (16.947%); 17 MiB/s, 279 objects/s recovering 2024-08-17T18:02:25.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:25 smithi165 ceph-mon[105517]: osdmap e1429: 8 total, 8 up, 6 in 2024-08-17T18:02:25.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:25 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 10240/101217 objects degraded (10.117%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:02:25.673 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:25.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:25 smithi136 ceph-mon[109861]: pgmap v2475: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 2.1 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 355 op/s; 10240/101217 objects degraded (10.117%); 17153/101217 objects misplaced (16.947%); 17 MiB/s, 279 objects/s recovering 2024-08-17T18:02:25.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:25 smithi136 ceph-mon[109861]: osdmap e1429: 8 total, 8 up, 6 in 2024-08-17T18:02:25.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:25 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 10240/101217 objects degraded (10.117%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:02:25.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:25 smithi136 ceph-mon[115078]: pgmap v2475: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 2.1 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 355 op/s; 10240/101217 objects degraded (10.117%); 17153/101217 objects misplaced (16.947%); 17 MiB/s, 279 objects/s recovering 2024-08-17T18:02:25.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:25 smithi136 ceph-mon[115078]: osdmap e1429: 8 total, 8 up, 6 in 2024-08-17T18:02:25.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:25 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 10240/101217 objects degraded (10.117%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:02:25.698 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:25.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:25.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:25.801 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:25.828 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:25.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:25.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:25.930 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:25.965 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:25.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:25.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:26.067 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:26.095 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:26.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:26.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:26.197 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:26.219 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:26.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:26.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:26.321 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:26.349 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:26.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:26.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:26.451 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:26.486 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:26.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:26.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:26.586 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:26 smithi165 ceph-mon[105517]: osdmap e1430: 8 total, 8 up, 6 in 2024-08-17T18:02:26.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:26.635 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:26.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:26.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:26.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:26 smithi136 ceph-mon[109861]: osdmap e1430: 8 total, 8 up, 6 in 2024-08-17T18:02:26.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:26 smithi136 ceph-mon[115078]: osdmap e1430: 8 total, 8 up, 6 in 2024-08-17T18:02:26.737 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:26.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:26.776 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:26.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:26.877 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:26.906 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:26.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:26.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:27.008 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:27.038 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:27.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:27.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:27.140 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:27.168 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:27.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:27.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:27.269 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:27.282 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:02:27.282 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:02:27.295 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:27.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:27.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:27.397 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:27.437 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:27.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:27.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:27.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:27 smithi165 ceph-mon[105517]: pgmap v2478: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 2.1 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 326 op/s; 10080/100971 objects degraded (9.983%); 17149/100971 objects misplaced (16.984%); 17 MiB/s, 275 objects/s recovering 2024-08-17T18:02:27.540 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:27.563 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:27.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:27.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:27.665 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:27.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:27 smithi136 ceph-mon[109861]: pgmap v2478: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 2.1 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 326 op/s; 10080/100971 objects degraded (9.983%); 17149/100971 objects misplaced (16.984%); 17 MiB/s, 275 objects/s recovering 2024-08-17T18:02:27.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:27 smithi136 ceph-mon[115078]: pgmap v2478: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 2.1 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 326 op/s; 10080/100971 objects degraded (9.983%); 17149/100971 objects misplaced (16.984%); 17 MiB/s, 275 objects/s recovering 2024-08-17T18:02:27.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:02:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:02:27] "GET /metrics HTTP/1.1" 200 36641 "" "Prometheus/2.51.0" 2024-08-17T18:02:27.701 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:27.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:27.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:27.803 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:27.831 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:27.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:27.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:27.932 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:27.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:27.962 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:27.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:28.019 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-17T18:02:28.066 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:28.098 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:28.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:28.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:28.201 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:28.229 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:28.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:28.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:28.331 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:28.339 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:02:28.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:28.358 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:28.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:28.460 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:28.483 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:28.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:28.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:28.585 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:28.612 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:28.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:28.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:28.714 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:28.739 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:28.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:28.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:28.840 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:28.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:28.884 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:28.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:28.985 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:29.013 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:29.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:29.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:29.115 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:29.139 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:29.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:29.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:29.241 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:29.270 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:29.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:29.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:29.372 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:29.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:29.398 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:29.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:29.499 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:29.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:29 smithi165 ceph-mon[105517]: pgmap v2479: 99 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 1.9 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 573 op/s; 10002/93699 objects degraded (10.675%); 14048/93699 objects misplaced (14.993%); 30 MiB/s, 473 objects/s recovering 2024-08-17T18:02:29.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:29 smithi165 ceph-mon[105517]: osdmap e1431: 8 total, 8 up, 6 in 2024-08-17T18:02:29.531 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:29.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:29.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:29.633 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:29.659 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:29.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:29.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:29.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:29 smithi136 ceph-mon[109861]: pgmap v2479: 99 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 1.9 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 573 op/s; 10002/93699 objects degraded (10.675%); 14048/93699 objects misplaced (14.993%); 30 MiB/s, 473 objects/s recovering 2024-08-17T18:02:29.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:29 smithi136 ceph-mon[109861]: osdmap e1431: 8 total, 8 up, 6 in 2024-08-17T18:02:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:29 smithi136 ceph-mon[115078]: pgmap v2479: 99 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 1.9 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 573 op/s; 10002/93699 objects degraded (10.675%); 14048/93699 objects misplaced (14.993%); 30 MiB/s, 473 objects/s recovering 2024-08-17T18:02:29.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:29 smithi136 ceph-mon[115078]: osdmap e1431: 8 total, 8 up, 6 in 2024-08-17T18:02:29.761 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:29.787 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:29.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:29.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:29.889 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:29.919 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:29.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:29.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:30.021 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:30.048 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:30.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:30.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:30.150 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:30.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:30.193 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:30.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:30.293 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:02:30.293 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:02:30.294 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:30.334 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:30.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:30.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:30.436 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:30.467 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:30.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:30.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:30.569 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:30.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:30.614 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:30.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:30.692 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:30 smithi165 ceph-mon[105517]: osdmap e1432: 8 total, 8 up, 6 in 2024-08-17T18:02:30.692 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:30 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 10002/93699 objects degraded (10.675%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:02:30.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:30 smithi136 ceph-mon[109861]: osdmap e1432: 8 total, 8 up, 6 in 2024-08-17T18:02:30.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:30 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 10002/93699 objects degraded (10.675%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:02:30.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:30 smithi136 ceph-mon[115078]: osdmap e1432: 8 total, 8 up, 6 in 2024-08-17T18:02:30.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:30 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 10002/93699 objects degraded (10.675%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:02:30.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:30.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:30.756 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:30.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:30.857 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:30.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:30.905 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:30.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:31.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:31.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:31.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:31.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:31.046 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-17T18:02:31.146 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:31.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:31.179 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:31.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:31.280 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:31.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:31.312 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:31.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:31.413 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:31.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:31.446 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:31.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:31.547 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:31.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:31.585 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:31.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:31.685 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:31 smithi136 ceph-mon[109861]: pgmap v2482: 99 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 1.9 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 443 op/s; 9866/93489 objects degraded (10.553%); 14044/93489 objects misplaced (15.022%); 23 MiB/s, 360 objects/s recovering 2024-08-17T18:02:31.685 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:31 smithi136 ceph-mon[115078]: pgmap v2482: 99 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 1.9 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 443 op/s; 9866/93489 objects degraded (10.553%); 14044/93489 objects misplaced (15.022%); 23 MiB/s, 360 objects/s recovering 2024-08-17T18:02:31.685 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:31.716 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:31.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:31.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:31.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:31 smithi165 ceph-mon[105517]: pgmap v2482: 99 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 1.9 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 443 op/s; 9866/93489 objects degraded (10.553%); 14044/93489 objects misplaced (15.022%); 23 MiB/s, 360 objects/s recovering 2024-08-17T18:02:31.818 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:31.846 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:31.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:31.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:31.948 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:31.979 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:31.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:31.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:32.081 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:32.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:32.111 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:32.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:32.212 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:32.244 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:32.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:32.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:32.346 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:32.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:32.376 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:32.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:32.477 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:32.505 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:32.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:32.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:32.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:32.642 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:32.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:32.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:32.744 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:32.774 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:32.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:32.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:32.876 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:32.910 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:32.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:32.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:33.012 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:33.038 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:33.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:33.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:33.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:33.175 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:33.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:33.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:33.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:33.311 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:33.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:33.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:33.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:33.512 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:33.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:33.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:33.614 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:33.648 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:33.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:33.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:33 smithi136 ceph-mon[109861]: pgmap v2483: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.8 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 744 op/s; 9868/86178 objects degraded (11.451%); 13018/86178 objects misplaced (15.106%); 22 MiB/s, 384 objects/s recovering 2024-08-17T18:02:33.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:33 smithi136 ceph-mon[115078]: pgmap v2483: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.8 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 744 op/s; 9868/86178 objects degraded (11.451%); 13018/86178 objects misplaced (15.106%); 22 MiB/s, 384 objects/s recovering 2024-08-17T18:02:33.716 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:02:33.750 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:33 smithi165 ceph-mon[105517]: pgmap v2483: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.8 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 744 op/s; 9868/86178 objects degraded (11.451%); 13018/86178 objects misplaced (15.106%); 22 MiB/s, 384 objects/s recovering 2024-08-17T18:02:33.792 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:33.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:33.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:33.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:33.925 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:33.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:33.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:34.027 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:34.056 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:34.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:34.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:34.158 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:34.188 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:34.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:34.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:34.289 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:34.333 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:34.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:34.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:34.435 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:34.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:34.469 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:34.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:34.570 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:34.609 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:34.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:34.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:34.711 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:34.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:34.748 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:34.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:34.775 INFO:teuthology.orchestra.run.smithi136.stderr:no valid command found; 10 closest matches: 2024-08-17T18:02:34.775 INFO:teuthology.orchestra.run.smithi136.stderr:0 2024-08-17T18:02:34.775 INFO:teuthology.orchestra.run.smithi136.stderr:1 2024-08-17T18:02:34.775 INFO:teuthology.orchestra.run.smithi136.stderr:2 2024-08-17T18:02:34.775 INFO:teuthology.orchestra.run.smithi136.stderr:abort 2024-08-17T18:02:34.775 INFO:teuthology.orchestra.run.smithi136.stderr:assert 2024-08-17T18:02:34.776 INFO:teuthology.orchestra.run.smithi136.stderr:bench [] [] [] [] 2024-08-17T18:02:34.776 INFO:teuthology.orchestra.run.smithi136.stderr:bluefs debug_inject_read_zeros 2024-08-17T18:02:34.776 INFO:teuthology.orchestra.run.smithi136.stderr:bluefs files list 2024-08-17T18:02:34.776 INFO:teuthology.orchestra.run.smithi136.stderr:bluefs stats 2024-08-17T18:02:34.776 INFO:teuthology.orchestra.run.smithi136.stderr:bluestore allocator dump block 2024-08-17T18:02:34.776 INFO:teuthology.orchestra.run.smithi136.stderr:Error EINVAL: invalid command 2024-08-17T18:02:34.850 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:34.886 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:34.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:34.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:34.988 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:35.014 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:35.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:35.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:35.116 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:35.144 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:35.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:35.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:35.246 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:35.275 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:35.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:35.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:35.377 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:35.406 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:35.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:35.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:35.508 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:35.518 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:35 smithi136 ceph-mon[109861]: pgmap v2484: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.7 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 678 op/s; 9782/84684 objects degraded (11.551%); 12934/84684 objects misplaced (15.273%); 19 MiB/s, 329 objects/s recovering 2024-08-17T18:02:35.518 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:35 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 9782/84684 objects degraded (11.551%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:02:35.518 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:35 smithi136 ceph-mon[115078]: pgmap v2484: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.7 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 678 op/s; 9782/84684 objects degraded (11.551%); 12934/84684 objects misplaced (15.273%); 19 MiB/s, 329 objects/s recovering 2024-08-17T18:02:35.518 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:35 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 9782/84684 objects degraded (11.551%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:02:35.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:35.539 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:35.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:35.640 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:35.651 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-17T18:02:35.653 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-17T18:02:35.653 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-08-17T18:02:35.672 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:35.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:35.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:35 smithi165 ceph-mon[105517]: pgmap v2484: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.7 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 678 op/s; 9782/84684 objects degraded (11.551%); 12934/84684 objects misplaced (15.273%); 19 MiB/s, 329 objects/s recovering 2024-08-17T18:02:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:35 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 9782/84684 objects degraded (11.551%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:02:35.774 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:35.804 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:35.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:35.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:35.906 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:35.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:35.942 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:35.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:36.043 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:36.073 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:36.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:36.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:36.175 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:36.200 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:36.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:36.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:36.302 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:36.333 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:36.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:36.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:36.435 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:36.467 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:36.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:36.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:36.570 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:36.604 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:36.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:36.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:36.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:02:36.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:02:36.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:02:36.707 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:36.746 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:36.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:36.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:36.848 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:36.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:36.890 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:36.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:36.991 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:37.019 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:37.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:37.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:37.121 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:37.148 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:37.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:37.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:37.250 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:37.277 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:37.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:37.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:37.379 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:37.411 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:37.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:37.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:37.513 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:37.535 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:37.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:37.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:37.637 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:37.671 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:37.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:37.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:37.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:37.808 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:37.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:37.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:37.910 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:37.913 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:37 smithi136 ceph-mon[109861]: pgmap v2485: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.7 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 385 op/s; 9615/84441 objects degraded (11.387%); 12940/84441 objects misplaced (15.324%); 3.8 MiB/s, 94 objects/s recovering 2024-08-17T18:02:37.913 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:02:37.913 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:02:37.913 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:02:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:02:37] "GET /metrics HTTP/1.1" 200 36638 "" "Prometheus/2.51.0" 2024-08-17T18:02:37.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:37 smithi136 ceph-mon[115078]: pgmap v2485: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.7 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 385 op/s; 9615/84441 objects degraded (11.387%); 12940/84441 objects misplaced (15.324%); 3.8 MiB/s, 94 objects/s recovering 2024-08-17T18:02:37.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:02:37.914 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:02:37.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:37.952 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:37.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:37 smithi165 ceph-mon[105517]: pgmap v2485: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.7 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 385 op/s; 9615/84441 objects degraded (11.387%); 12940/84441 objects misplaced (15.324%); 3.8 MiB/s, 94 objects/s recovering 2024-08-17T18:02:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:02:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:02:38.053 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:38.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:38.081 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:38.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:38.182 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:38.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:38.216 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:38.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:38.317 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:38.347 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:38.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:38.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:38.448 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:38.489 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:38.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:38.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:38.591 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:38.629 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:38.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:38.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:38.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:38.771 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:38.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:38.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:38.873 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:38.902 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:38.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:38.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:38.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:02:38.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:02:38.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:02:38.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:02:38.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:02:38.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:02:39.004 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:39.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:38 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:02:39.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:38 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:02:39.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:38 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:02:39.042 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:39.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:39.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:39.144 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:39.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:39.171 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:39.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:39.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:39.302 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:39.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:39.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:39.404 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:39.438 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:39.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:39.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:39.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:39.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:39.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:39.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:39.674 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:39.712 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:39.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:39.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:39.813 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:39.839 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:39.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:39.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:39.941 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:39.969 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:39.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:39.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:39 smithi165 ceph-mon[105517]: pgmap v2486: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.5 GiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 648 op/s; 9514/75018 objects degraded (12.682%); 12125/75018 objects misplaced (16.163%); 3.8 MiB/s, 127 objects/s recovering 2024-08-17T18:02:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:39 smithi165 ceph-mon[105517]: pgmap v2487: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.5 GiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 699 op/s; 9514/75018 objects degraded (12.682%); 12125/75018 objects misplaced (16.163%); 3.7 MiB/s, 131 objects/s recovering 2024-08-17T18:02:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:39 smithi165 ceph-mon[105517]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-08-17T18:02:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:39 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 9514/75018 objects degraded (12.682%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:02:40.070 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:40.105 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:40.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:40.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:40.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:39 smithi136 ceph-mon[109861]: pgmap v2486: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.5 GiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 648 op/s; 9514/75018 objects degraded (12.682%); 12125/75018 objects misplaced (16.163%); 3.8 MiB/s, 127 objects/s recovering 2024-08-17T18:02:40.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:39 smithi136 ceph-mon[109861]: pgmap v2487: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.5 GiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 699 op/s; 9514/75018 objects degraded (12.682%); 12125/75018 objects misplaced (16.163%); 3.7 MiB/s, 131 objects/s recovering 2024-08-17T18:02:40.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:39 smithi136 ceph-mon[109861]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-08-17T18:02:40.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:39 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 9514/75018 objects degraded (12.682%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:02:40.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:39 smithi136 ceph-mon[115078]: pgmap v2486: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.5 GiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 648 op/s; 9514/75018 objects degraded (12.682%); 12125/75018 objects misplaced (16.163%); 3.8 MiB/s, 127 objects/s recovering 2024-08-17T18:02:40.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:39 smithi136 ceph-mon[115078]: pgmap v2487: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.5 GiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 699 op/s; 9514/75018 objects degraded (12.682%); 12125/75018 objects misplaced (16.163%); 3.7 MiB/s, 131 objects/s recovering 2024-08-17T18:02:40.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:39 smithi136 ceph-mon[115078]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-08-17T18:02:40.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:39 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 9514/75018 objects degraded (12.682%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:02:40.207 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:40.233 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:40.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:40.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:40.335 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:40.363 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:40.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:40.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:40.465 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:40.499 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:40.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:40.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:40.600 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:40.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:40.632 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:40.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:40.733 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:40.761 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:40.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:40.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:40.863 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:40.893 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:40.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:40.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:40.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:40 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4091132797' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:02:40.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:40 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4091132797' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:02:40.995 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:40 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4091132797' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:02:41.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:41.032 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:41.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:41.052 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:02:41.053 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:02:41.053 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:02:41.053 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:02:41.054 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:02:41.054 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:02:41.054 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:02:41.054 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:02:41.054 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:02:41.054 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:02:41.054 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:02:41.055 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:02:41.055 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:02:41.055 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:02:41.055 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:02:41.055 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:02:41.134 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:41.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:41.160 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:41.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:41.262 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:41.285 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:41.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:41.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:41.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:41.415 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:41.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:41.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:41.517 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:41.545 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:41.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:41.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:41.646 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:41.674 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:41.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:41.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:41.775 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:41.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:41.809 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:41.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:41.910 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:41.937 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:41.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:41.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:41.973 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:41 smithi136 ceph-mon[109861]: pgmap v2488: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.5 GiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 611 op/s; 9333/74751 objects degraded (12.485%); 12127/74751 objects misplaced (16.223%); 3.7 MiB/s, 122 objects/s recovering 2024-08-17T18:02:41.974 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:41 smithi136 ceph-mon[115078]: pgmap v2488: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.5 GiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 611 op/s; 9333/74751 objects degraded (12.485%); 12127/74751 objects misplaced (16.223%); 3.7 MiB/s, 122 objects/s recovering 2024-08-17T18:02:42.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:41 smithi165 ceph-mon[105517]: pgmap v2488: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.5 GiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 611 op/s; 9333/74751 objects degraded (12.485%); 12127/74751 objects misplaced (16.223%); 3.7 MiB/s, 122 objects/s recovering 2024-08-17T18:02:42.038 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:42.071 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:42.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:42.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:42.172 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:42.205 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:42.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:42.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:42.307 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:42.335 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:42.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:42.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:42.437 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:42.478 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:42.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:42.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:42.481 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:02:42.579 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:42.609 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:42.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:42.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:42.637 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:02:42.686 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:02:42.711 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:42.725 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:42 smithi136 ceph-mon[109861]: pgmap v2489: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.4 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 623 op/s; 9333/67071 objects degraded (13.915%); 11441/67071 objects misplaced (17.058%); 2.2 MiB/s, 108 objects/s recovering 2024-08-17T18:02:42.745 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:42.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:42.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:42.847 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:42.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:42.885 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:42.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:42.986 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:43.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:42 smithi165 ceph-mon[105517]: pgmap v2489: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.4 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 623 op/s; 9333/67071 objects degraded (13.915%); 11441/67071 objects misplaced (17.058%); 2.2 MiB/s, 108 objects/s recovering 2024-08-17T18:02:43.015 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:43.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:43.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:43.028 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:42 smithi136 ceph-mon[115078]: pgmap v2489: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.4 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 623 op/s; 9333/67071 objects degraded (13.915%); 11441/67071 objects misplaced (17.058%); 2.2 MiB/s, 108 objects/s recovering 2024-08-17T18:02:43.117 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:43.145 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:43.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:43.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:43.246 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:43.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:43.278 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:43.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:43.379 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:43.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:43.414 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:43.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:43.515 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:43.542 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:43.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:43.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:43.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:43.683 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:43.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:43.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:43.785 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:43.825 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:43.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:43.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:43.927 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:43.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:43.954 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:43.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:44.055 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:44.084 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:44.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:44.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:44.185 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:44.215 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:44.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:44.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:44.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:44.340 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:44.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:44.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:44.442 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:44.464 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:44.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:44.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:44.499 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:02:44.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:44.594 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:44.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:44.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:44.696 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:44.721 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:44.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:44.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:44.823 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:44.859 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:44.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:44.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:44.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:44.996 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:44.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:44.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:45.098 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:45.128 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:45.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:45.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:45.230 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:45.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:45.265 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:45.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:45.366 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:45.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:45.395 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:45.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:45.482 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:45 smithi136 ceph-mon[115078]: pgmap v2490: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.3 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 620 op/s; 9237/65685 objects degraded (14.063%); 11345/65685 objects misplaced (17.272%); 2.2 MiB/s, 109 objects/s recovering 2024-08-17T18:02:45.482 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:45 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 9237/65685 objects degraded (14.063%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:02:45.482 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:45 smithi136 ceph-mon[109861]: pgmap v2490: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.3 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 620 op/s; 9237/65685 objects degraded (14.063%); 11345/65685 objects misplaced (17.272%); 2.2 MiB/s, 109 objects/s recovering 2024-08-17T18:02:45.482 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:45 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 9237/65685 objects degraded (14.063%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:02:45.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:45.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:45 smithi165 ceph-mon[105517]: pgmap v2490: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.3 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 620 op/s; 9237/65685 objects degraded (14.063%); 11345/65685 objects misplaced (17.272%); 2.2 MiB/s, 109 objects/s recovering 2024-08-17T18:02:45.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:45 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 9237/65685 objects degraded (14.063%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:02:45.528 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:45.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:45.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:45.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:45.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:45.666 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:45.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:45.767 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:45.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:45.803 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:45.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:45.904 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:45.933 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:45.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:45.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:46.035 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:46.063 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:46.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:46.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:46.164 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:46.194 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:46.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:46.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:46.296 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:46.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:46.334 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:46.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:46.434 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:46.465 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:46.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:46.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:46.567 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:46.602 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:46.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:46.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:46.704 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:46.718 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:02:46.737 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:46.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:46.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:46.838 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:46.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:46.872 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:46.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:46.973 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:46.982 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:02:46.982 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:02:46.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.982 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:227296 54.12 54:48899318:::benchmark_data_smithi136_273969_object104411:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1432)", 2024-08-17T18:02:46.982 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:02:46.967921+0000", 2024-08-17T18:02:46.982 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012507436, 2024-08-17T18:02:46.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012538641, 2024-08-17T18:02:46.983 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:02:46.983 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:02:46.983 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-17T18:02:46.983 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:02:46.983 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:02:46.983 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:02:46.983 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 227296 2024-08-17T18:02:46.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.983 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:02:46.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:02:46.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.967921+0000", 2024-08-17T18:02:46.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:02:46.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:02:46.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.967921+0000", 2024-08-17T18:02:46.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:02:46.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:02:46.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.967923+0000", 2024-08-17T18:02:46.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4649999999999999e-06 2024-08-17T18:02:46.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:02:46.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.967927+0000", 2024-08-17T18:02:46.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.878e-06 2024-08-17T18:02:46.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:02:46.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.967936+0000", 2024-08-17T18:02:46.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7569999999999999e-06 2024-08-17T18:02:46.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:02:46.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.967943+0000", 2024-08-17T18:02:46.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1e-06 2024-08-17T18:02:46.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:02:46.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.968086+0000", 2024-08-17T18:02:46.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000143005 2024-08-17T18:02:46.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:02:46.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.968224+0000", 2024-08-17T18:02:46.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000138854 2024-08-17T18:02:46.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:02:46.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.968699+0000", 2024-08-17T18:02:46.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00047432399999999998 2024-08-17T18:02:46.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:02:46.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.975537+0000", 2024-08-17T18:02:46.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0068380539999999997 2024-08-17T18:02:46.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:46.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.975934+0000", 2024-08-17T18:02:46.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00039680100000000001 2024-08-17T18:02:46.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:02:46.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.977546+0000", 2024-08-17T18:02:46.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001612397 2024-08-17T18:02:46.988 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:02:46.988 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:02:46.988 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:02:46.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.989 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.35370.0:227204 54.10 e1432/1421 54:0f5e8f4e:::benchmark_data_smithi136_273969_object104319:head v 1432'9974, mlcod=1383'6193)", 2024-08-17T18:02:46.989 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:02:46.978128+0000", 2024-08-17T18:02:46.989 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0023010499999999998, 2024-08-17T18:02:46.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0024140379999999999, 2024-08-17T18:02:46.989 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:02:46.989 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:02:46.989 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T18:02:46.989 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:02:46.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:02:46.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978128+0000", 2024-08-17T18:02:46.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:02:46.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:02:46.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978128+0000", 2024-08-17T18:02:46.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:02:46.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:02:46.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978129+0000", 2024-08-17T18:02:46.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0419999999999999e-06 2024-08-17T18:02:46.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:02:46.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978136+0000", 2024-08-17T18:02:46.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4240000000000004e-06 2024-08-17T18:02:46.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:02:46.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978149+0000", 2024-08-17T18:02:46.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2942000000000001e-05 2024-08-17T18:02:46.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:02:46.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978157+0000", 2024-08-17T18:02:46.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.362e-06 2024-08-17T18:02:46.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:02:46.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978222+0000", 2024-08-17T18:02:46.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5450000000000005e-05 2024-08-17T18:02:46.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:02:46.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978249+0000", 2024-08-17T18:02:46.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7206e-05 2024-08-17T18:02:46.992 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:02:46.992 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:02:46.992 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:02:46.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.993 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:227303 54.12 54:4de5095d:::benchmark_data_smithi136_273969_object104418:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1432)", 2024-08-17T18:02:46.993 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:02:46.978294+0000", 2024-08-17T18:02:46.993 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.002134581, 2024-08-17T18:02:46.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0022968860000000002, 2024-08-17T18:02:46.993 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:02:46.993 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:02:46.993 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-17T18:02:46.993 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:02:46.993 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:02:46.993 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:02:46.993 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 227303 2024-08-17T18:02:46.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.994 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:02:46.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:02:46.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978294+0000", 2024-08-17T18:02:46.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:02:46.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:02:46.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978294+0000", 2024-08-17T18:02:46.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:02:46.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:02:46.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978296+0000", 2024-08-17T18:02:46.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4559999999999999e-06 2024-08-17T18:02:46.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:02:46.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978300+0000", 2024-08-17T18:02:46.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3889999999999998e-06 2024-08-17T18:02:46.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:02:46.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978312+0000", 2024-08-17T18:02:46.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2187999999999999e-05 2024-08-17T18:02:46.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:02:46.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978319+0000", 2024-08-17T18:02:46.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5939999999999999e-06 2024-08-17T18:02:46.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:02:46.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978389+0000", 2024-08-17T18:02:46.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0090000000000001e-05 2024-08-17T18:02:46.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:02:46.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978551+0000", 2024-08-17T18:02:46.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016176799999999999 2024-08-17T18:02:46.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:02:46.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.979188+0000", 2024-08-17T18:02:46.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00063745399999999997 2024-08-17T18:02:46.997 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:02:46.997 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:02:46.997 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:02:46.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.998 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.35370.0:227296 54.12 e1432/1414)", 2024-08-17T18:02:46.998 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:02:46.978581+0000", 2024-08-17T18:02:46.998 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.001847558, 2024-08-17T18:02:46.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002065225, 2024-08-17T18:02:46.998 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:02:46.998 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:02:46.998 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-08-17T18:02:46.998 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:02:46.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:02:46.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978581+0000", 2024-08-17T18:02:46.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:02:46.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:02:46.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978581+0000", 2024-08-17T18:02:46.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:02:46.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:46.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:02:46.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978582+0000", 2024-08-17T18:02:46.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6899999999999996e-07 2024-08-17T18:02:46.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:46.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:47.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:02:47.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978587+0000", 2024-08-17T18:02:47.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4380000000000003e-06 2024-08-17T18:02:47.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:47.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:47.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:02:47.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978596+0000", 2024-08-17T18:02:47.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8810000000000002e-06 2024-08-17T18:02:47.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:47.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:47.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:02:47.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978603+0000", 2024-08-17T18:02:47.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3099999999999997e-06 2024-08-17T18:02:47.001 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:02:47.001 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:02:47.001 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:02:47.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:47.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:47.001 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.35370.0:227302 54.11 e1432/1402 54:8de3ef48:::benchmark_data_smithi136_273969_object104417:head v 1432'10024, mlcod=1432'10022)", 2024-08-17T18:02:47.001 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:02:46.978863+0000", 2024-08-17T18:02:47.002 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0015655059999999999, 2024-08-17T18:02:47.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001822115, 2024-08-17T18:02:47.002 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:02:47.002 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:02:47.002 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T18:02:47.002 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:02:47.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:47.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:02:47.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978863+0000", 2024-08-17T18:02:47.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:02:47.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:47.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:47.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:02:47.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978863+0000", 2024-08-17T18:02:47.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:02:47.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:47.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:47.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:02:47.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978864+0000", 2024-08-17T18:02:47.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9999999999999996e-07 2024-08-17T18:02:47.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:47.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:47.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:02:47.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978869+0000", 2024-08-17T18:02:47.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2390000000000002e-06 2024-08-17T18:02:47.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:47.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:47.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:02:47.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978881+0000", 2024-08-17T18:02:47.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.202e-05 2024-08-17T18:02:47.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:47.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:47.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:02:47.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978887+0000", 2024-08-17T18:02:47.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6230000000000002e-06 2024-08-17T18:02:47.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:47.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:47.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:02:47.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978938+0000", 2024-08-17T18:02:47.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1016000000000001e-05 2024-08-17T18:02:47.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:02:47.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:02:47.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:02:47.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:46.978957+0000", 2024-08-17T18:02:47.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9293e-05 2024-08-17T18:02:47.006 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:02:47.006 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:02:47.006 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:02:47.006 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:02:47.007 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-17T18:02:47.007 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 5 2024-08-17T18:02:47.007 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:02:47.014 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:47.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:47.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:47.116 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:47.154 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:47.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:47.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:47.249 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:02:47.256 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:47.290 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:47.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:47.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:47.392 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:47.432 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:47.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:47.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:47 smithi165 ceph-mon[105517]: pgmap v2491: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.3 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 622 op/s; 9042/65379 objects degraded (13.830%); 11336/65379 objects misplaced (17.339%); 2.4 MiB/s, 111 objects/s recovering 2024-08-17T18:02:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:47 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/782481971' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:02:47.533 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:47.568 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:47.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:47.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:47.632 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:47 smithi136 ceph-mon[109861]: pgmap v2491: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.3 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 622 op/s; 9042/65379 objects degraded (13.830%); 11336/65379 objects misplaced (17.339%); 2.4 MiB/s, 111 objects/s recovering 2024-08-17T18:02:47.633 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:47 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/782481971' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:02:47.633 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:47 smithi136 ceph-mon[115078]: pgmap v2491: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.3 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 622 op/s; 9042/65379 objects degraded (13.830%); 11336/65379 objects misplaced (17.339%); 2.4 MiB/s, 111 objects/s recovering 2024-08-17T18:02:47.633 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:47 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/782481971' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:02:47.633 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:02:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:02:47] "GET /metrics HTTP/1.1" 200 36638 "" "Prometheus/2.51.0" 2024-08-17T18:02:47.671 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:47.710 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:47.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:47.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:47.812 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:47.845 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:47.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:47.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:47.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:47.988 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:47.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:47.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:48.090 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:48.120 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:48.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:48.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:48.222 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:48.267 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:48.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:48.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:48.369 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:48.403 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:48.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:48.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:48.505 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:48.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:48 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:02:48.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:48 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/782481971' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:02:48.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:48 smithi165 ceph-mon[105517]: osdmap e1433: 8 total, 8 up, 6 in 2024-08-17T18:02:48.542 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:48.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:48.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:48.557 INFO:teuthology.orchestra.run.smithi136.stderr:no valid command found; 10 closest matches: 2024-08-17T18:02:48.558 INFO:teuthology.orchestra.run.smithi136.stderr:0 2024-08-17T18:02:48.558 INFO:teuthology.orchestra.run.smithi136.stderr:1 2024-08-17T18:02:48.558 INFO:teuthology.orchestra.run.smithi136.stderr:2 2024-08-17T18:02:48.558 INFO:teuthology.orchestra.run.smithi136.stderr:abort 2024-08-17T18:02:48.558 INFO:teuthology.orchestra.run.smithi136.stderr:assert 2024-08-17T18:02:48.558 INFO:teuthology.orchestra.run.smithi136.stderr:bench [] [] [] [] 2024-08-17T18:02:48.558 INFO:teuthology.orchestra.run.smithi136.stderr:bluefs debug_inject_read_zeros 2024-08-17T18:02:48.558 INFO:teuthology.orchestra.run.smithi136.stderr:bluefs files list 2024-08-17T18:02:48.558 INFO:teuthology.orchestra.run.smithi136.stderr:bluefs stats 2024-08-17T18:02:48.558 INFO:teuthology.orchestra.run.smithi136.stderr:bluestore allocator dump block 2024-08-17T18:02:48.559 INFO:teuthology.orchestra.run.smithi136.stderr:Error EINVAL: invalid command 2024-08-17T18:02:48.559 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:48 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:02:48.559 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:48 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/782481971' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:02:48.559 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:48 smithi136 ceph-mon[109861]: osdmap e1433: 8 total, 8 up, 6 in 2024-08-17T18:02:48.559 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:48 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:02:48.559 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:48 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/782481971' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:02:48.560 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:48 smithi136 ceph-mon[115078]: osdmap e1433: 8 total, 8 up, 6 in 2024-08-17T18:02:48.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:48.681 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:48.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:48.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:48.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:48.816 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:48.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:48.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:48.834 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-17T18:02:48.918 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:48.954 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:48.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:48.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:49.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:49.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:49.114 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:49.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:49.215 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:49.259 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:49.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:49.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:49.361 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:49.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:49.397 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:49.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:49.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:49 smithi136 ceph-mon[109861]: pgmap v2493: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.1 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 639 op/s; 8941/55812 objects degraded (16.020%); 10543/55812 objects misplaced (18.890%); 2.4 MiB/s, 128 objects/s recovering 2024-08-17T18:02:49.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:49 smithi136 ceph-mon[115078]: pgmap v2493: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.1 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 639 op/s; 8941/55812 objects degraded (16.020%); 10543/55812 objects misplaced (18.890%); 2.4 MiB/s, 128 objects/s recovering 2024-08-17T18:02:49.498 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:49.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:49.528 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:49.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:49.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:49.674 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:49.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:49.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:49 smithi165 ceph-mon[105517]: pgmap v2493: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.1 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 639 op/s; 8941/55812 objects degraded (16.020%); 10543/55812 objects misplaced (18.890%); 2.4 MiB/s, 128 objects/s recovering 2024-08-17T18:02:49.778 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:49.816 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:49.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:49.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:49.919 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:49.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:49.955 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:49.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:50.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:50.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:50.091 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:50.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:50.193 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:50.239 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:50.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:50.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:50.341 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:50.375 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:50.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:50.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:50.425 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:50 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 8941/55812 objects degraded (16.020%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:02:50.426 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:50 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 8941/55812 objects degraded (16.020%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:02:50.477 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:50.513 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:50.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:50.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:50.616 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:50.648 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:50.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:50.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:50.750 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:50.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:50 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 8941/55812 objects degraded (16.020%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:02:50.781 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:50.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:50.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:50.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:50.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:50.914 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:50.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:51.015 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:51.048 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:51.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:51.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:51.150 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:51.196 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:51.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:51.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:51.298 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:51.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:51.334 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:51.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:51.435 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:51.477 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:51.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:51.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:51.579 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:51.621 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:51.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:51.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:51.626 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:51 smithi136 ceph-mon[109861]: pgmap v2494: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.1 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 644 op/s; 8825/55413 objects degraded (15.926%); 10543/55413 objects misplaced (19.026%); 2.2 MiB/s, 125 objects/s recovering 2024-08-17T18:02:51.626 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:02:51.627 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:02:51.627 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:51 smithi136 ceph-mon[115078]: pgmap v2494: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.1 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 644 op/s; 8825/55413 objects degraded (15.926%); 10543/55413 objects misplaced (19.026%); 2.2 MiB/s, 125 objects/s recovering 2024-08-17T18:02:51.627 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:02:51.627 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:02:51.724 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:51 smithi165 ceph-mon[105517]: pgmap v2494: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.1 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 644 op/s; 8825/55413 objects degraded (15.926%); 10543/55413 objects misplaced (19.026%); 2.2 MiB/s, 125 objects/s recovering 2024-08-17T18:02:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:02:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:02:51.769 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:51.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:51.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:51.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:51.903 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:51.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:51.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:52.005 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:52.029 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-17T18:02:52.029 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-17T18:02:52.029 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-08-17T18:02:52.045 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:52.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:52.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:52.070 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:02:52.147 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:52.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:52.181 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:52.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:52.282 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:52.318 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:52.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:52.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:52.420 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:52.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:52.453 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:52.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:52.464 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:02:52.555 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:52.596 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:52.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:52.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:52.688 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:02:52.698 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:52.719 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:02:52.720 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:02:52.749 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:52.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:52.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:52.753 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:02:52.754 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:02:52.785 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:02:52.786 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:02:52.819 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:02:52.820 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:02:52.851 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:52.855 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:02:52.855 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:02:52.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:52.882 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:52.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:52.887 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:02:52.887 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:02:52.923 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:02:52.924 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:02:52.954 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:02:52.955 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:02:52.983 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:53.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:53.026 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:53.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:53.127 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:53.162 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:53.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:53.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:53.264 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:53.301 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:53.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:53.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:53.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:53.433 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:53.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:53.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:53.535 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:53.560 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:53.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:53.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:53.617 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:53 smithi136 ceph-mon[109861]: pgmap v2495: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 931 MiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 743 op/s; 8827/44763 objects degraded (19.719%); 9644/44763 objects misplaced (21.545%); 2.2 MiB/s, 128 objects/s recovering 2024-08-17T18:02:53.617 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:53 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2188845938' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:02:53.617 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:53 smithi136 ceph-mon[115078]: pgmap v2495: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 931 MiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 743 op/s; 8827/44763 objects degraded (19.719%); 9644/44763 objects misplaced (21.545%); 2.2 MiB/s, 128 objects/s recovering 2024-08-17T18:02:53.617 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:53 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2188845938' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:02:53.662 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:53.689 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:53.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:53.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:53 smithi165 ceph-mon[105517]: pgmap v2495: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 931 MiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 743 op/s; 8827/44763 objects degraded (19.719%); 9644/44763 objects misplaced (21.545%); 2.2 MiB/s, 128 objects/s recovering 2024-08-17T18:02:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:53 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2188845938' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:02:53.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:53.821 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:53.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:53.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:53.922 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:53.947 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:53.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:53.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:54.049 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:54.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:54.083 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:54.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:54.184 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:54.208 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:54.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:54.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:54.310 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:54.338 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:54.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:54.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:54.440 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:54.463 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:54.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:54.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:54.565 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:54.589 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:54.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:54.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:54.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:54.718 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:54.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:54.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:54.820 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:54.842 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:54.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:54.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:54.944 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:54.966 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:54.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:54.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:55.068 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:55.091 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:55.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:55.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:55.193 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:55.215 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:55.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:55.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:55.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:55.341 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:55.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:55.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:55.443 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:55.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:55 smithi136 ceph-mon[109861]: pgmap v2496: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 891 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 761 op/s; 8697/42825 objects degraded (20.308%); 9514/42825 objects misplaced (22.216%); 2.4 MiB/s, 131 objects/s recovering 2024-08-17T18:02:55.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:55 smithi136 ceph-mon[109861]: osdmap e1434: 8 total, 8 up, 6 in 2024-08-17T18:02:55.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:55 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 8697/42825 objects degraded (20.308%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:02:55.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:55 smithi136 ceph-mon[115078]: pgmap v2496: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 891 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 761 op/s; 8697/42825 objects degraded (20.308%); 9514/42825 objects misplaced (22.216%); 2.4 MiB/s, 131 objects/s recovering 2024-08-17T18:02:55.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:55 smithi136 ceph-mon[115078]: osdmap e1434: 8 total, 8 up, 6 in 2024-08-17T18:02:55.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:55 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 8697/42825 objects degraded (20.308%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:02:55.472 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:55.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:55.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:55.574 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:55.605 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:55.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:55.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:55.707 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:55.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:55.747 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:55.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:55 smithi165 ceph-mon[105517]: pgmap v2496: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 891 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 761 op/s; 8697/42825 objects degraded (20.308%); 9514/42825 objects misplaced (22.216%); 2.4 MiB/s, 131 objects/s recovering 2024-08-17T18:02:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:55 smithi165 ceph-mon[105517]: osdmap e1434: 8 total, 8 up, 6 in 2024-08-17T18:02:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:55 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 8697/42825 objects degraded (20.308%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:02:55.848 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:55.890 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:55.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:55.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:55.992 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:56.031 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:56.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:56.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:56.128 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:02:56.133 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:02:56.173 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:56.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:56.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:02:56.275 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:56.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:56.317 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:56.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:56.418 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:56.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:56 smithi136 ceph-mon[109861]: osdmap e1435: 8 total, 8 up, 6 in 2024-08-17T18:02:56.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:56 smithi136 ceph-mon[115078]: osdmap e1435: 8 total, 8 up, 6 in 2024-08-17T18:02:56.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:56.462 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:56.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:56.564 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:56.593 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:56.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:56.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:56.695 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:56.723 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:56.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:56.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:56 smithi165 ceph-mon[105517]: osdmap e1435: 8 total, 8 up, 6 in 2024-08-17T18:02:56.784 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:02:56.827 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:56.872 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:56.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:56.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:56.974 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:57.008 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:57.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:57.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:57.079 INFO:teuthology.orchestra.run.smithi136.stderr:no valid command found; 10 closest matches: 2024-08-17T18:02:57.079 INFO:teuthology.orchestra.run.smithi136.stderr:0 2024-08-17T18:02:57.079 INFO:teuthology.orchestra.run.smithi136.stderr:1 2024-08-17T18:02:57.079 INFO:teuthology.orchestra.run.smithi136.stderr:2 2024-08-17T18:02:57.079 INFO:teuthology.orchestra.run.smithi136.stderr:abort 2024-08-17T18:02:57.079 INFO:teuthology.orchestra.run.smithi136.stderr:assert 2024-08-17T18:02:57.080 INFO:teuthology.orchestra.run.smithi136.stderr:bench [] [] [] [] 2024-08-17T18:02:57.080 INFO:teuthology.orchestra.run.smithi136.stderr:bluefs debug_inject_read_zeros 2024-08-17T18:02:57.080 INFO:teuthology.orchestra.run.smithi136.stderr:bluefs files list 2024-08-17T18:02:57.080 INFO:teuthology.orchestra.run.smithi136.stderr:bluefs stats 2024-08-17T18:02:57.080 INFO:teuthology.orchestra.run.smithi136.stderr:bluestore allocator dump block 2024-08-17T18:02:57.080 INFO:teuthology.orchestra.run.smithi136.stderr:Error EINVAL: invalid command 2024-08-17T18:02:57.110 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:02:57.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:57.143 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:57.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:02:57.244 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:57.271 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:57.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:57.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:57.373 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:57.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:57.410 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:57.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:57.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:57 smithi136 ceph-mon[109861]: pgmap v2499: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 882 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 559 op/s; 8697/42372 objects degraded (20.525%); 9514/42372 objects misplaced (22.454%); 1.5 MiB/s, 74 objects/s recovering 2024-08-17T18:02:57.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:57 smithi136 ceph-mon[115078]: pgmap v2499: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 882 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 559 op/s; 8697/42372 objects degraded (20.525%); 9514/42372 objects misplaced (22.454%); 1.5 MiB/s, 74 objects/s recovering 2024-08-17T18:02:57.511 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:57.540 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:57.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:57.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:57.642 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:02:57.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:57.685 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:57.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:02:57.756 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:02:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:02:57] "GET /metrics HTTP/1.1" 200 36637 "" "Prometheus/2.51.0" 2024-08-17T18:02:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:57 smithi165 ceph-mon[105517]: pgmap v2499: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 882 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 559 op/s; 8697/42372 objects degraded (20.525%); 9514/42372 objects misplaced (22.454%); 1.5 MiB/s, 74 objects/s recovering 2024-08-17T18:02:57.788 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:57.816 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:57.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:57.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:57.918 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:57.945 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:57.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:57.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:58.047 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:58.077 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:58.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:58.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:58.179 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:58.220 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:58.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:58.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:58.322 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:58.352 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:58.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:58.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:58.455 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:58.486 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:58.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:58.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:58.588 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:58.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:58.617 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:58.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:58.718 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:02:58.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:58.761 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:58.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:02:58.862 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:58.900 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:58.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:58.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:59.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:59.031 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-17T18:02:59.031 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-17T18:02:59.032 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-08-17T18:02:59.035 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:59.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:59.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:59.137 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:59.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:59.173 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:59.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:59.274 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:02:59.300 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:59.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:59.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:02:59.402 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:59.429 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:59.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:59.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:59.531 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:59.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:59.562 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:59.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:59.664 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:02:59.692 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:59.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:59.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:02:59.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:02:59 smithi136 ceph-mon[109861]: pgmap v2500: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 652 MiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 1.00k op/s; 5123/31359 objects degraded (16.337%); 6196/31359 objects misplaced (19.758%); 4.2 MiB/s, 140 objects/s recovering 2024-08-17T18:02:59.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:02:59 smithi136 ceph-mon[115078]: pgmap v2500: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 652 MiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 1.00k op/s; 5123/31359 objects degraded (16.337%); 6196/31359 objects misplaced (19.758%); 4.2 MiB/s, 140 objects/s recovering 2024-08-17T18:02:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:02:59 smithi165 ceph-mon[105517]: pgmap v2500: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 652 MiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 1.00k op/s; 5123/31359 objects degraded (16.337%); 6196/31359 objects misplaced (19.758%); 4.2 MiB/s, 140 objects/s recovering 2024-08-17T18:02:59.794 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:02:59.838 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:02:59.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:59.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:02:59.940 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:02:59.975 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:02:59.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:02:59.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:02:59.988 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:02:59.988 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:03:00.080 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:00.119 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:00.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:00.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:00.223 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:00.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:00.265 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:00.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:00.366 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:00.396 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:00.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:00.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:00.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:00 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5123/31359 objects degraded (16.337%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T18:03:00.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:00 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5123/31359 objects degraded (16.337%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T18:03:00.498 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:00.531 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:00.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:00.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:00.634 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:00.678 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:00.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:00.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:00.741 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:03:00.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:00 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5123/31359 objects degraded (16.337%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T18:03:00.781 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:00.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:00.831 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:00.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:00.932 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:00.963 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:00.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:00.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:01.065 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:01.092 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:01.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:01.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:01.194 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:01.236 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:01.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:01.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:01.337 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:01.370 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:01.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:01.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:01.472 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:01.502 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:01.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:01.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:01.603 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:01.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:01.633 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:01.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:01 smithi136 ceph-mon[109861]: pgmap v2501: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 643 MiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 576 op/s; 5123/30933 objects degraded (16.562%); 6196/30933 objects misplaced (20.030%); 4.2 MiB/s, 89 objects/s recovering 2024-08-17T18:03:01.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:01 smithi136 ceph-mon[115078]: pgmap v2501: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 643 MiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 576 op/s; 5123/30933 objects degraded (16.562%); 6196/30933 objects misplaced (20.030%); 4.2 MiB/s, 89 objects/s recovering 2024-08-17T18:03:01.735 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:01 smithi165 ceph-mon[105517]: pgmap v2501: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 643 MiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 576 op/s; 5123/30933 objects degraded (16.562%); 6196/30933 objects misplaced (20.030%); 4.2 MiB/s, 89 objects/s recovering 2024-08-17T18:03:01.765 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:01.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:01.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:01.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:01.895 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:01.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:01.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:01.938 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-17T18:03:01.997 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:02.042 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:02.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:02.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:02.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:02.172 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:03:02.182 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:02.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:02.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:02.284 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:02.319 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:02.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:02.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:02.421 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:02.457 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:02.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:02.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:02.559 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:02.606 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:02.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:02.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:02.708 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:02.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:02.756 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:02.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:02.857 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:02.886 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:02.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:02.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:02.988 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:03.015 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:03.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:03.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:03.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:03.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:03.155 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:03.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:03.256 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:03.286 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:03.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:03.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:03.391 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:03.412 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:03.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:03.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:03.514 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:03.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:03.542 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:03.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:03.643 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:03.690 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:03.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:03.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:03 smithi136 ceph-mon[109861]: pgmap v2502: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 400 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 981 op/s; 5123/19263 objects degraded (26.595%); 4974/19263 objects misplaced (25.822%); 4.6 MiB/s, 96 objects/s recovering 2024-08-17T18:03:03.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:03 smithi136 ceph-mon[115078]: pgmap v2502: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 400 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 981 op/s; 5123/19263 objects degraded (26.595%); 4974/19263 objects misplaced (25.822%); 4.6 MiB/s, 96 objects/s recovering 2024-08-17T18:03:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:03 smithi165 ceph-mon[105517]: pgmap v2502: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 400 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 981 op/s; 5123/19263 objects degraded (26.595%); 4974/19263 objects misplaced (25.822%); 4.6 MiB/s, 96 objects/s recovering 2024-08-17T18:03:03.792 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:03.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:03.820 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:03.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:03.921 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:03.947 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:03.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:03.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:04.049 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:04.078 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:04.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:04.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:04.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:04.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:04.211 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:04.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:04.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:04.530 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:04.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:04.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:04.632 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:04.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:04.662 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:04.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:04.763 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:04.801 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:04.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:04.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:04.902 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:04.927 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:04.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:04.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:05.031 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:05.087 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:05.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:05.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:05.189 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:05.224 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:05.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:05.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:05.326 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:05.358 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:05.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:05.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:05.460 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:05.487 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:05.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:05.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:05.588 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:05.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:05.616 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:05.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:05.674 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:03:05.675 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:05 smithi136 ceph-mon[109861]: pgmap v2503: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 353 MiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 868 op/s; 5123/17022 objects degraded (30.096%); 4635/17022 objects misplaced (27.229%); 4.8 MiB/s, 94 objects/s recovering 2024-08-17T18:03:05.675 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:05 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5123/17022 objects degraded (30.096%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T18:03:05.676 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:05 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2586009285' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:03:05.676 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:05 smithi136 ceph-mon[115078]: pgmap v2503: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 353 MiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 868 op/s; 5123/17022 objects degraded (30.096%); 4635/17022 objects misplaced (27.229%); 4.8 MiB/s, 94 objects/s recovering 2024-08-17T18:03:05.676 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:05 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5123/17022 objects degraded (30.096%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T18:03:05.676 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:05 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2586009285' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:03:05.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:05.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:05.746 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:05.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:05 smithi165 ceph-mon[105517]: pgmap v2503: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 353 MiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 868 op/s; 5123/17022 objects degraded (30.096%); 4635/17022 objects misplaced (27.229%); 4.8 MiB/s, 94 objects/s recovering 2024-08-17T18:03:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:05 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5123/17022 objects degraded (30.096%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T18:03:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:05 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2586009285' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:03:05.848 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:05.869 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:05.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:05.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:05.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:06.008 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:06.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:06.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:06.110 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:06.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:06.151 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:06.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:06.252 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:06.302 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:06.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:06.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:06.404 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:06.447 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:06.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:06.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:06.549 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:06.579 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:06.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:06.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:06.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:06 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:03:06.662 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:06 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2586009285' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:03:06.662 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:06 smithi165 ceph-mon[105517]: osdmap e1436: 8 total, 8 up, 6 in 2024-08-17T18:03:06.662 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:03:06.681 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:06.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:06 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:03:06.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:06 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2586009285' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:03:06.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:06 smithi136 ceph-mon[115078]: osdmap e1436: 8 total, 8 up, 6 in 2024-08-17T18:03:06.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:03:06.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:06 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:03:06.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:06 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2586009285' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:03:06.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:06 smithi136 ceph-mon[109861]: osdmap e1436: 8 total, 8 up, 6 in 2024-08-17T18:03:06.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:03:06.723 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:06.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:06.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:06.825 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:06.851 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:06.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:06.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:06.953 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:06.990 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:06.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:06.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:07.092 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:07.132 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:07.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:07.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:07.235 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:07.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:07.264 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:07.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:07.365 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:07.370 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:03:07.401 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:07.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:07.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:07.503 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:07.535 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:07.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:07.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:07.617 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:07 smithi136 ceph-mon[109861]: pgmap v2505: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 346 MiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 857 op/s; 5123/16644 objects degraded (30.780%); 4635/16644 objects misplaced (27.848%); 4.8 MiB/s, 93 objects/s recovering 2024-08-17T18:03:07.617 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:07 smithi136 ceph-mon[115078]: pgmap v2505: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 346 MiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 857 op/s; 5123/16644 objects degraded (30.780%); 4635/16644 objects misplaced (27.848%); 4.8 MiB/s, 93 objects/s recovering 2024-08-17T18:03:07.617 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:03:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:03:07] "GET /metrics HTTP/1.1" 200 36638 "" "Prometheus/2.51.0" 2024-08-17T18:03:07.637 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:07.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:07.680 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:07.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:07.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:07 smithi165 ceph-mon[105517]: pgmap v2505: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 346 MiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 857 op/s; 5123/16644 objects degraded (30.780%); 4635/16644 objects misplaced (27.848%); 4.8 MiB/s, 93 objects/s recovering 2024-08-17T18:03:07.782 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:07.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:07.811 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:07.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:07.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:07.952 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:07.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:07.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:08.035 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Removed 122882 objects 2024-08-17T18:03:08.036 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Clean up completed and total clean up time :182.315 2024-08-17T18:03:08.054 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:08.076 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:08.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:08.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:08.177 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:08.199 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:08.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:08.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:08.301 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:08.323 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:08.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:08.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:08.424 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:08.446 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:08.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:08.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:08.548 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:08.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:08.571 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:08.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:08.672 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:08.688 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-08-17T18:03:08.688 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-08-17T18:03:08.695 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:08.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:08.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:08.793 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:03:08.793 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:03:08.793 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:03:08.793 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:03:08.797 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:08.806 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:03:08.807 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:03:08.818 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:08.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:08.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:08.823 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:03:08.823 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:03:08.841 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:03:08.841 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:03:08.858 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:03:08.858 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:03:08.874 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:03:08.874 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:03:08.890 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:03:08.890 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:03:08.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:08.942 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:08.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:08.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:09.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:09.067 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:09.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:09.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:09.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:09.191 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:09.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:09.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:09.293 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:09.315 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:09.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:09.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:09.417 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:09.440 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:09.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:09.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:09.542 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:09.566 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:09.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:09.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:09.668 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:09.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:09.695 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:09.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:09 smithi136 ceph-mon[109861]: pgmap v2506: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 76 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 931 op/s; 5123/3693 objects degraded (138.722%); 3460/3693 objects misplaced (93.691%); 3.0 MiB/s, 86 objects/s recovering 2024-08-17T18:03:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:09 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/395915233' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:03:09.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:09 smithi136 ceph-mon[115078]: pgmap v2506: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 76 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 931 op/s; 5123/3693 objects degraded (138.722%); 3460/3693 objects misplaced (93.691%); 3.0 MiB/s, 86 objects/s recovering 2024-08-17T18:03:09.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:09 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/395915233' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:03:09.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:09 smithi165 ceph-mon[105517]: pgmap v2506: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 76 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 931 op/s; 5123/3693 objects degraded (138.722%); 3460/3693 objects misplaced (93.691%); 3.0 MiB/s, 86 objects/s recovering 2024-08-17T18:03:09.770 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:09 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/395915233' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:03:09.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:09.837 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:09.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:09.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:09.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:09.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:09.987 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:09.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:10.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:10.119 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:10.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:10.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:10.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:10.245 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:10.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:10.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:10.347 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:10.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:10.373 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:10.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:10.474 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:10.496 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:10.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:10.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:10.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:10.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:10.623 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:10.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:10.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:10 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5123/3693 objects degraded (138.722%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T18:03:10.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:10 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5123/3693 objects degraded (138.722%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T18:03:10.724 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:10.747 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:10.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:10.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:10 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5123/3693 objects degraded (138.722%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T18:03:10.849 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:10.873 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:10.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:10.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:10.974 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:10.998 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:10.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:10.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:11.100 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:11.122 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:11.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:11.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:11.224 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:11.249 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:11.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:11.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:11.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:11.373 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:11.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:11.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:11.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:11.498 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:11.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:11.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:11.556 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:11 smithi136 ceph-mon[109861]: pgmap v2507: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 70 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 926 op/s; 5123/3408 objects degraded (150.323%); 3460/3408 objects misplaced (101.526%); 3.0 MiB/s, 86 objects/s recovering 2024-08-17T18:03:11.557 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:11 smithi136 ceph-mon[115078]: pgmap v2507: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 70 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 926 op/s; 5123/3408 objects degraded (150.323%); 3460/3408 objects misplaced (101.526%); 3.0 MiB/s, 86 objects/s recovering 2024-08-17T18:03:11.600 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:11.622 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:11.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:11.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:11.724 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:11.748 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:11.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:11.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:11 smithi165 ceph-mon[105517]: pgmap v2507: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 70 MiB data, 3.4 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 926 op/s; 5123/3408 objects degraded (150.323%); 3460/3408 objects misplaced (101.526%); 3.0 MiB/s, 86 objects/s recovering 2024-08-17T18:03:11.850 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:11.873 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:11.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:11.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:11.974 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:11.997 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:11.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:11.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:12.099 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:12.123 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:12.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:12.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:12.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:12.250 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:12.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:12.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:12.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:12.375 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:12.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:12.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:12.425 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:03:12.426 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-17T18:03:12.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-17T18:03:12.426 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:03:12.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.426 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:192150 54.12 54:49a14c33:::benchmark_data_smithi136_273969_object69265:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1414)", 2024-08-17T18:03:12.426 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:01:45.619181+0000", 2024-08-17T18:03:12.426 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 86.802597347000003, 2024-08-17T18:03:12.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.276965009, 2024-08-17T18:03:12.426 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:03:12.426 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:03:12.426 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:03:12.426 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:03:12.427 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:03:12.427 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:03:12.427 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 192150 2024-08-17T18:03:12.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.427 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:03:12.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:03:12.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619181+0000", 2024-08-17T18:03:12.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:03:12.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619181+0000", 2024-08-17T18:03:12.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:03:12.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619182+0000", 2024-08-17T18:03:12.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4670000000000001e-06 2024-08-17T18:03:12.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:03:12.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619189+0000", 2024-08-17T18:03:12.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0900000000000001e-06 2024-08-17T18:03:12.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:03:12.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619204+0000", 2024-08-17T18:03:12.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5814000000000001e-05 2024-08-17T18:03:12.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:03:12.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619215+0000", 2024-08-17T18:03:12.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1055e-05 2024-08-17T18:03:12.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619307+0000", 2024-08-17T18:03:12.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1644000000000006e-05 2024-08-17T18:03:12.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:03:12.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619317+0000", 2024-08-17T18:03:12.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0399e-05 2024-08-17T18:03:12.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.854325+0000", 2024-08-17T18:03:12.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2350079519999999 2024-08-17T18:03:12.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:03:12.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.855087+0000", 2024-08-17T18:03:12.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00076172700000000002 2024-08-17T18:03:12.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.864942+0000", 2024-08-17T18:03:12.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0098545830000000001 2024-08-17T18:03:12.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:03:12.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.865043+0000", 2024-08-17T18:03:12.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010126 2024-08-17T18:03:12.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.432 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:03:12.432 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.865180+0000", 2024-08-17T18:03:12.432 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000136728 2024-08-17T18:03:12.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.432 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:03:12.432 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.870130+0000", 2024-08-17T18:03:12.432 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.004950538 2024-08-17T18:03:12.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.432 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.432 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.871248+0000", 2024-08-17T18:03:12.432 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00111798 2024-08-17T18:03:12.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.882468+0000", 2024-08-17T18:03:12.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011219688 2024-08-17T18:03:12.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.895546+0000", 2024-08-17T18:03:12.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013078485000000001 2024-08-17T18:03:12.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:03:12.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.895627+0000", 2024-08-17T18:03:12.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0935999999999999e-05 2024-08-17T18:03:12.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:03:12.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.896146+0000", 2024-08-17T18:03:12.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00051866300000000005 2024-08-17T18:03:12.434 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.434 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:03:12.434 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.435 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:192163 54.12 54:4f23e758:::benchmark_data_smithi136_273969_object69278:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1414)", 2024-08-17T18:03:12.435 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:01:45.619331+0000", 2024-08-17T18:03:12.435 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 86.802447309000001, 2024-08-17T18:03:12.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.306504149, 2024-08-17T18:03:12.435 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:03:12.435 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:03:12.435 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:03:12.435 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:03:12.435 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:03:12.435 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:03:12.435 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 192163 2024-08-17T18:03:12.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.435 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:03:12.435 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:03:12.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619331+0000", 2024-08-17T18:03:12.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:03:12.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619331+0000", 2024-08-17T18:03:12.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:03:12.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619332+0000", 2024-08-17T18:03:12.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3099999999999997e-07 2024-08-17T18:03:12.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:03:12.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619335+0000", 2024-08-17T18:03:12.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7050000000000001e-06 2024-08-17T18:03:12.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:03:12.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619343+0000", 2024-08-17T18:03:12.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3100000000000003e-06 2024-08-17T18:03:12.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:03:12.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619349+0000", 2024-08-17T18:03:12.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9020000000000004e-06 2024-08-17T18:03:12.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619466+0000", 2024-08-17T18:03:12.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011772 2024-08-17T18:03:12.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:03:12.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619475+0000", 2024-08-17T18:03:12.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4279999999999996e-06 2024-08-17T18:03:12.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.856035+0000", 2024-08-17T18:03:12.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.236560772 2024-08-17T18:03:12.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:03:12.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.857073+0000", 2024-08-17T18:03:12.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001037786 2024-08-17T18:03:12.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.876475+0000", 2024-08-17T18:03:12.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019401277000000001 2024-08-17T18:03:12.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:03:12.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.876558+0000", 2024-08-17T18:03:12.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3465000000000001e-05 2024-08-17T18:03:12.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:03:12.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.876647+0000", 2024-08-17T18:03:12.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9367e-05 2024-08-17T18:03:12.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:03:12.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.883470+0000", 2024-08-17T18:03:12.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0068226509999999999 2024-08-17T18:03:12.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.895353+0000", 2024-08-17T18:03:12.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011882762999999999 2024-08-17T18:03:12.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.918556+0000", 2024-08-17T18:03:12.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023202837 2024-08-17T18:03:12.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.924734+0000", 2024-08-17T18:03:12.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0061787539999999998 2024-08-17T18:03:12.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:03:12.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.924825+0000", 2024-08-17T18:03:12.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0402000000000006e-05 2024-08-17T18:03:12.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:03:12.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.925835+0000", 2024-08-17T18:03:12.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010103790000000001 2024-08-17T18:03:12.443 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.443 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:03:12.444 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.444 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:192171 54.12 54:4ea3b2a6:::benchmark_data_smithi136_273969_object69286:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1414)", 2024-08-17T18:03:12.444 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:01:45.619430+0000", 2024-08-17T18:03:12.444 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 86.802347842000003, 2024-08-17T18:03:12.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3073286230000001, 2024-08-17T18:03:12.444 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:03:12.444 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:03:12.444 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:03:12.444 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:03:12.444 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:03:12.444 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:03:12.445 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 192171 2024-08-17T18:03:12.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.445 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:03:12.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:03:12.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619430+0000", 2024-08-17T18:03:12.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:03:12.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619430+0000", 2024-08-17T18:03:12.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:03:12.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619431+0000", 2024-08-17T18:03:12.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-07 2024-08-17T18:03:12.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:03:12.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619434+0000", 2024-08-17T18:03:12.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.942e-06 2024-08-17T18:03:12.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:03:12.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619445+0000", 2024-08-17T18:03:12.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0859000000000001e-05 2024-08-17T18:03:12.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:03:12.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619451+0000", 2024-08-17T18:03:12.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8699999999999997e-06 2024-08-17T18:03:12.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619504+0000", 2024-08-17T18:03:12.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3680000000000001e-05 2024-08-17T18:03:12.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:03:12.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619512+0000", 2024-08-17T18:03:12.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.187e-06 2024-08-17T18:03:12.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.857133+0000", 2024-08-17T18:03:12.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2376217789999999 2024-08-17T18:03:12.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:03:12.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.857953+0000", 2024-08-17T18:03:12.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00081922199999999998 2024-08-17T18:03:12.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.879821+0000", 2024-08-17T18:03:12.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.021868581000000002 2024-08-17T18:03:12.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:03:12.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.879940+0000", 2024-08-17T18:03:12.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119096 2024-08-17T18:03:12.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:03:12.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.880101+0000", 2024-08-17T18:03:12.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000161137 2024-08-17T18:03:12.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:03:12.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.884587+0000", 2024-08-17T18:03:12.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0044860020000000002 2024-08-17T18:03:12.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.895435+0000", 2024-08-17T18:03:12.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0108475 2024-08-17T18:03:12.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.918629+0000", 2024-08-17T18:03:12.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023194105999999999 2024-08-17T18:03:12.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.925895+0000", 2024-08-17T18:03:12.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0072663399999999996 2024-08-17T18:03:12.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:03:12.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.925953+0000", 2024-08-17T18:03:12.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7711999999999999e-05 2024-08-17T18:03:12.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:03:12.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.926759+0000", 2024-08-17T18:03:12.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00080602400000000002 2024-08-17T18:03:12.452 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.452 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:03:12.452 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.452 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:192180 54.12 54:4e71d729:::benchmark_data_smithi136_273969_object69295:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1414)", 2024-08-17T18:03:12.452 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:01:45.619479+0000", 2024-08-17T18:03:12.452 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 86.802299043000005, 2024-08-17T18:03:12.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.309031767, 2024-08-17T18:03:12.452 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:03:12.452 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:03:12.453 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:03:12.453 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:03:12.453 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:03:12.453 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:03:12.453 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 192180 2024-08-17T18:03:12.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.453 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:03:12.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:03:12.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619479+0000", 2024-08-17T18:03:12.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:03:12.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619479+0000", 2024-08-17T18:03:12.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:03:12.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619480+0000", 2024-08-17T18:03:12.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2e-07 2024-08-17T18:03:12.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:03:12.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619506+0000", 2024-08-17T18:03:12.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5979999999999999e-05 2024-08-17T18:03:12.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:03:12.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619539+0000", 2024-08-17T18:03:12.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2929000000000003e-05 2024-08-17T18:03:12.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:03:12.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619545+0000", 2024-08-17T18:03:12.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1759999999999998e-06 2024-08-17T18:03:12.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619595+0000", 2024-08-17T18:03:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0432000000000001e-05 2024-08-17T18:03:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:03:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619603+0000", 2024-08-17T18:03:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6899999999999992e-06 2024-08-17T18:03:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.859671+0000", 2024-08-17T18:03:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.240067867 2024-08-17T18:03:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:03:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.860629+0000", 2024-08-17T18:03:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00095811900000000001 2024-08-17T18:03:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.884642+0000", 2024-08-17T18:03:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.024012782999999999 2024-08-17T18:03:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:03:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.884767+0000", 2024-08-17T18:03:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012548 2024-08-17T18:03:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:03:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.884913+0000", 2024-08-17T18:03:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000146067 2024-08-17T18:03:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:03:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.890573+0000", 2024-08-17T18:03:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005659191 2024-08-17T18:03:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.895490+0000", 2024-08-17T18:03:12.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0049179740000000003 2024-08-17T18:03:12.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.918689+0000", 2024-08-17T18:03:12.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023198212999999999 2024-08-17T18:03:12.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.926833+0000", 2024-08-17T18:03:12.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0081445130000000008 2024-08-17T18:03:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:03:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.926898+0000", 2024-08-17T18:03:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4667999999999998e-05 2024-08-17T18:03:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:03:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.928511+0000", 2024-08-17T18:03:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001613165 2024-08-17T18:03:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:03:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:192215 54.12 54:4e9684f7:::benchmark_data_smithi136_273969_object69330:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1414)", 2024-08-17T18:03:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:01:45.619613+0000", 2024-08-17T18:03:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 86.802165778000003, 2024-08-17T18:03:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3110728090000001, 2024-08-17T18:03:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:03:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:03:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:03:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:03:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:03:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:03:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 192215 2024-08-17T18:03:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.462 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:03:12.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:03:12.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619613+0000", 2024-08-17T18:03:12.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:03:12.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619613+0000", 2024-08-17T18:03:12.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:03:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619613+0000", 2024-08-17T18:03:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2200000000000004e-07 2024-08-17T18:03:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:03:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619624+0000", 2024-08-17T18:03:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0586e-05 2024-08-17T18:03:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:03:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619631+0000", 2024-08-17T18:03:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4429999999999997e-06 2024-08-17T18:03:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:03:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619636+0000", 2024-08-17T18:03:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0610000000000003e-06 2024-08-17T18:03:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619709+0000", 2024-08-17T18:03:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3028999999999995e-05 2024-08-17T18:03:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:03:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619718+0000", 2024-08-17T18:03:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5960000000000004e-06 2024-08-17T18:03:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.860832+0000", 2024-08-17T18:03:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.241114388 2024-08-17T18:03:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:03:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.861514+0000", 2024-08-17T18:03:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00068133600000000005 2024-08-17T18:03:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.890605+0000", 2024-08-17T18:03:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.029091479999999999 2024-08-17T18:03:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:03:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.890686+0000", 2024-08-17T18:03:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0698999999999994e-05 2024-08-17T18:03:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:03:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.890795+0000", 2024-08-17T18:03:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109188 2024-08-17T18:03:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:03:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.895300+0000", 2024-08-17T18:03:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0045045459999999999 2024-08-17T18:03:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.899478+0000", 2024-08-17T18:03:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0041783979999999998 2024-08-17T18:03:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.928599+0000", 2024-08-17T18:03:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.029121266999999999 2024-08-17T18:03:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.929785+0000", 2024-08-17T18:03:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001185495 2024-08-17T18:03:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:03:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.929825+0000", 2024-08-17T18:03:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0224e-05 2024-08-17T18:03:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:03:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.930685+0000", 2024-08-17T18:03:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00086045100000000003 2024-08-17T18:03:12.469 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.469 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:03:12.469 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.469 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:192263 54.12 54:4c7fca38:::benchmark_data_smithi136_273969_object69378:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1414)", 2024-08-17T18:03:12.469 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:01:45.619653+0000", 2024-08-17T18:03:12.469 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 86.802124904999999, 2024-08-17T18:03:12.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.312035579, 2024-08-17T18:03:12.469 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:03:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:03:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:03:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:03:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:03:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:03:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 192263 2024-08-17T18:03:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:03:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:03:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619653+0000", 2024-08-17T18:03:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:03:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619653+0000", 2024-08-17T18:03:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:03:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619654+0000", 2024-08-17T18:03:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4000000000000002e-07 2024-08-17T18:03:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:03:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619656+0000", 2024-08-17T18:03:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2189999999999998e-06 2024-08-17T18:03:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:03:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619668+0000", 2024-08-17T18:03:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1948e-05 2024-08-17T18:03:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:03:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619677+0000", 2024-08-17T18:03:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4420000000000004e-06 2024-08-17T18:03:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619742+0000", 2024-08-17T18:03:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4486000000000006e-05 2024-08-17T18:03:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:03:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619748+0000", 2024-08-17T18:03:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7219999999999996e-06 2024-08-17T18:03:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.871280+0000", 2024-08-17T18:03:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2515325820000001 2024-08-17T18:03:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:03:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.872233+0000", 2024-08-17T18:03:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00095263599999999998 2024-08-17T18:03:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.897356+0000", 2024-08-17T18:03:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025122924000000001 2024-08-17T18:03:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:03:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.897467+0000", 2024-08-17T18:03:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110763 2024-08-17T18:03:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:03:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.897634+0000", 2024-08-17T18:03:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000167178 2024-08-17T18:03:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:03:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.904590+0000", 2024-08-17T18:03:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.006956472 2024-08-17T18:03:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.904660+0000", 2024-08-17T18:03:12.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0111999999999996e-05 2024-08-17T18:03:12.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.928654+0000", 2024-08-17T18:03:12.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023993976 2024-08-17T18:03:12.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.930772+0000", 2024-08-17T18:03:12.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021179940000000002 2024-08-17T18:03:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:03:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.930844+0000", 2024-08-17T18:03:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1367999999999998e-05 2024-08-17T18:03:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:03:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.931689+0000", 2024-08-17T18:03:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00084531700000000005 2024-08-17T18:03:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:03:12.478 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.478 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.478 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.478 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:192270 54.12 54:48b908b4:::benchmark_data_smithi136_273969_object69385:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1414)", 2024-08-17T18:03:12.478 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:01:45.619724+0000", 2024-08-17T18:03:12.478 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 86.802054166999994, 2024-08-17T18:03:12.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.343520584, 2024-08-17T18:03:12.478 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:03:12.478 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:03:12.478 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:03:12.478 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:03:12.478 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:03:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:03:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 192270 2024-08-17T18:03:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:03:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:03:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619724+0000", 2024-08-17T18:03:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:03:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619724+0000", 2024-08-17T18:03:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:03:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619725+0000", 2024-08-17T18:03:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7000000000000005e-07 2024-08-17T18:03:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:03:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619727+0000", 2024-08-17T18:03:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4880000000000001e-06 2024-08-17T18:03:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:03:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619735+0000", 2024-08-17T18:03:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4510000000000003e-06 2024-08-17T18:03:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:03:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619740+0000", 2024-08-17T18:03:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7200000000000003e-06 2024-08-17T18:03:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619769+0000", 2024-08-17T18:03:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8399999999999999e-05 2024-08-17T18:03:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:03:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619773+0000", 2024-08-17T18:03:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7090000000000001e-06 2024-08-17T18:03:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.872277+0000", 2024-08-17T18:03:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2525032359999999 2024-08-17T18:03:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:03:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.873109+0000", 2024-08-17T18:03:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00083258500000000005 2024-08-17T18:03:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.900982+0000", 2024-08-17T18:03:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.027872309000000001 2024-08-17T18:03:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:03:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.901757+0000", 2024-08-17T18:03:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00077577500000000003 2024-08-17T18:03:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:03:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.901950+0000", 2024-08-17T18:03:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192339 2024-08-17T18:03:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.907034+0000", 2024-08-17T18:03:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0050840160000000002 2024-08-17T18:03:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:03:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.907056+0000", 2024-08-17T18:03:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1812000000000001e-05 2024-08-17T18:03:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.928710+0000", 2024-08-17T18:03:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.021654718999999999 2024-08-17T18:03:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.961596+0000", 2024-08-17T18:03:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.032885494000000001 2024-08-17T18:03:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:03:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.961659+0000", 2024-08-17T18:03:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3176999999999995e-05 2024-08-17T18:03:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:03:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.963245+0000", 2024-08-17T18:03:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001585784 2024-08-17T18:03:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:03:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:192338 54.12 54:49abe991:::benchmark_data_smithi136_273969_object69453:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1414)", 2024-08-17T18:03:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:01:45.619750+0000", 2024-08-17T18:03:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 86.802028112000002, 2024-08-17T18:03:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3444434219999999, 2024-08-17T18:03:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:03:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:03:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:03:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:03:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:03:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:03:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 192338 2024-08-17T18:03:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:03:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:03:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619750+0000", 2024-08-17T18:03:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:03:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619750+0000", 2024-08-17T18:03:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:03:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619751+0000", 2024-08-17T18:03:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7300000000000002e-07 2024-08-17T18:03:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:03:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619753+0000", 2024-08-17T18:03:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.903e-06 2024-08-17T18:03:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:03:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619761+0000", 2024-08-17T18:03:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0870000000000001e-06 2024-08-17T18:03:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:03:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619765+0000", 2024-08-17T18:03:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6999999999999999e-06 2024-08-17T18:03:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619792+0000", 2024-08-17T18:03:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6590000000000001e-05 2024-08-17T18:03:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:03:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619796+0000", 2024-08-17T18:03:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6260000000000003e-06 2024-08-17T18:03:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.873170+0000", 2024-08-17T18:03:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2533736200000001 2024-08-17T18:03:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:03:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.874081+0000", 2024-08-17T18:03:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00091100000000000003 2024-08-17T18:03:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.914796+0000", 2024-08-17T18:03:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040714988000000001 2024-08-17T18:03:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:03:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.914876+0000", 2024-08-17T18:03:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9779999999999998e-05 2024-08-17T18:03:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:03:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.914977+0000", 2024-08-17T18:03:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000101032 2024-08-17T18:03:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:03:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.918710+0000", 2024-08-17T18:03:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037327689999999999 2024-08-17T18:03:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.936854+0000", 2024-08-17T18:03:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018144113 2024-08-17T18:03:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.953835+0000", 2024-08-17T18:03:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016981581999999999 2024-08-17T18:03:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.963340+0000", 2024-08-17T18:03:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0095045040000000004 2024-08-17T18:03:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:03:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.963384+0000", 2024-08-17T18:03:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4509999999999999e-05 2024-08-17T18:03:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:03:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.964194+0000", 2024-08-17T18:03:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00080924499999999999 2024-08-17T18:03:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:03:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:192350 54.12 54:4da9c63d:::benchmark_data_smithi136_273969_object69465:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1414)", 2024-08-17T18:03:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:01:45.619776+0000", 2024-08-17T18:03:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 86.802002719000001, 2024-08-17T18:03:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.345182713, 2024-08-17T18:03:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:03:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:03:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:03:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:03:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:03:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:03:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 192350 2024-08-17T18:03:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:03:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:03:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619776+0000", 2024-08-17T18:03:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:03:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619776+0000", 2024-08-17T18:03:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:03:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619776+0000", 2024-08-17T18:03:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0200000000000003e-07 2024-08-17T18:03:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:03:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619778+0000", 2024-08-17T18:03:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9630000000000001e-06 2024-08-17T18:03:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:03:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619785+0000", 2024-08-17T18:03:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4760000000000001e-06 2024-08-17T18:03:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:03:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619791+0000", 2024-08-17T18:03:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1370000000000001e-06 2024-08-17T18:03:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.498 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:12.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619816+0000", 2024-08-17T18:03:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5332000000000001e-05 2024-08-17T18:03:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:03:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619821+0000", 2024-08-17T18:03:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0849999999999996e-06 2024-08-17T18:03:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.874145+0000", 2024-08-17T18:03:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2543244840000001 2024-08-17T18:03:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:03:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.874910+0000", 2024-08-17T18:03:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00076478500000000003 2024-08-17T18:03:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.922173+0000", 2024-08-17T18:03:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.047263067999999998 2024-08-17T18:03:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:03:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.922280+0000", 2024-08-17T18:03:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106425 2024-08-17T18:03:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:03:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.922423+0000", 2024-08-17T18:03:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014353200000000001 2024-08-17T18:03:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:03:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.929745+0000", 2024-08-17T18:03:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.007321312 2024-08-17T18:03:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.947063+0000", 2024-08-17T18:03:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017318170000000001 2024-08-17T18:03:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.953888+0000", 2024-08-17T18:03:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0068247769999999998 2024-08-17T18:03:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.964252+0000", 2024-08-17T18:03:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010364363999999999 2024-08-17T18:03:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:03:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.964289+0000", 2024-08-17T18:03:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6832999999999998e-05 2024-08-17T18:03:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:03:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.964958+0000", 2024-08-17T18:03:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00066956800000000005 2024-08-17T18:03:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:03:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:192355 54.12 54:4d9fafac:::benchmark_data_smithi136_273969_object69470:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1414)", 2024-08-17T18:03:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:01:45.619798+0000", 2024-08-17T18:03:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 86.801980482999994, 2024-08-17T18:03:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.345945814, 2024-08-17T18:03:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:03:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:03:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:03:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:03:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:03:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:03:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 192355 2024-08-17T18:03:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:03:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:03:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619798+0000", 2024-08-17T18:03:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:03:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619798+0000", 2024-08-17T18:03:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:03:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619798+0000", 2024-08-17T18:03:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3300000000000003e-07 2024-08-17T18:03:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:03:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619800+0000", 2024-08-17T18:03:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.787e-06 2024-08-17T18:03:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:03:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619806+0000", 2024-08-17T18:03:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8819999999999998e-06 2024-08-17T18:03:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:03:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619811+0000", 2024-08-17T18:03:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9219999999999996e-06 2024-08-17T18:03:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619840+0000", 2024-08-17T18:03:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9329999999999999e-05 2024-08-17T18:03:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:03:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619845+0000", 2024-08-17T18:03:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7759999999999997e-06 2024-08-17T18:03:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.906031+0000", 2024-08-17T18:03:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2861855470000001 2024-08-17T18:03:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:03:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.906929+0000", 2024-08-17T18:03:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00089853600000000002 2024-08-17T18:03:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.933411+0000", 2024-08-17T18:03:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.026482265000000001 2024-08-17T18:03:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:03:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.933518+0000", 2024-08-17T18:03:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010658999999999999 2024-08-17T18:03:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:03:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.933650+0000", 2024-08-17T18:03:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013180599999999999 2024-08-17T18:03:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:03:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.938261+0000", 2024-08-17T18:03:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0046110190000000001 2024-08-17T18:03:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.963297+0000", 2024-08-17T18:03:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025036520999999999 2024-08-17T18:03:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.965015+0000", 2024-08-17T18:03:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017182059999999999 2024-08-17T18:03:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.965096+0000", 2024-08-17T18:03:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0345000000000004e-05 2024-08-17T18:03:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:03:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.965146+0000", 2024-08-17T18:03:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0518e-05 2024-08-17T18:03:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:03:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.965744+0000", 2024-08-17T18:03:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00059733099999999999 2024-08-17T18:03:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:03:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:192366 54.12 54:4eebfbda:::benchmark_data_smithi136_273969_object69481:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1414)", 2024-08-17T18:03:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:01:45.619816+0000", 2024-08-17T18:03:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 86.801962134000007, 2024-08-17T18:03:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.346838848, 2024-08-17T18:03:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:03:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:03:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:03:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:03:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:03:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:03:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 192366 2024-08-17T18:03:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:03:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:03:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619816+0000", 2024-08-17T18:03:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:03:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619816+0000", 2024-08-17T18:03:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:03:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619817+0000", 2024-08-17T18:03:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2599999999999998e-07 2024-08-17T18:03:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:03:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619819+0000", 2024-08-17T18:03:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0439999999999998e-06 2024-08-17T18:03:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:03:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619824+0000", 2024-08-17T18:03:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6359999999999999e-06 2024-08-17T18:03:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:03:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619829+0000", 2024-08-17T18:03:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8749999999999999e-06 2024-08-17T18:03:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619864+0000", 2024-08-17T18:03:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4660999999999999e-05 2024-08-17T18:03:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:03:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619869+0000", 2024-08-17T18:03:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9960000000000001e-06 2024-08-17T18:03:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.907090+0000", 2024-08-17T18:03:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2872208949999999 2024-08-17T18:03:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:03:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.908038+0000", 2024-08-17T18:03:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00094804000000000004 2024-08-17T18:03:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.938314+0000", 2024-08-17T18:03:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030276356000000001 2024-08-17T18:03:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:03:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.938425+0000", 2024-08-17T18:03:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111422 2024-08-17T18:03:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:03:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.938516+0000", 2024-08-17T18:03:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0457999999999993e-05 2024-08-17T18:03:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:03:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.947010+0000", 2024-08-17T18:03:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0084945860000000002 2024-08-17T18:03:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.965789+0000", 2024-08-17T18:03:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018778197 2024-08-17T18:03:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.965865+0000", 2024-08-17T18:03:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5878999999999996e-05 2024-08-17T18:03:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.965979+0000", 2024-08-17T18:03:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011421699999999999 2024-08-17T18:03:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:03:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.966024+0000", 2024-08-17T18:03:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5491000000000002e-05 2024-08-17T18:03:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:03:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.966655+0000", 2024-08-17T18:03:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00063076899999999999 2024-08-17T18:03:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:03:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:192380 54.12 54:4fae2bcc:::benchmark_data_smithi136_273969_object69495:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1414)", 2024-08-17T18:03:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:01:45.619835+0000", 2024-08-17T18:03:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 86.801942890999996, 2024-08-17T18:03:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3475054950000001, 2024-08-17T18:03:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:03:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:03:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:03:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:03:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:03:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:03:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 192380 2024-08-17T18:03:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:03:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:03:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619835+0000", 2024-08-17T18:03:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:03:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619835+0000", 2024-08-17T18:03:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:03:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619836+0000", 2024-08-17T18:03:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6899999999999998e-07 2024-08-17T18:03:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:03:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619842+0000", 2024-08-17T18:03:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.959e-06 2024-08-17T18:03:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:03:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619848+0000", 2024-08-17T18:03:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6789999999999999e-06 2024-08-17T18:03:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:03:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619854+0000", 2024-08-17T18:03:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1760000000000001e-06 2024-08-17T18:03:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619895+0000", 2024-08-17T18:03:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1482e-05 2024-08-17T18:03:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:03:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619900+0000", 2024-08-17T18:03:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1939999999999997e-06 2024-08-17T18:03:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.908086+0000", 2024-08-17T18:03:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.288185849 2024-08-17T18:03:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:03:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.909055+0000", 2024-08-17T18:03:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00096864499999999997 2024-08-17T18:03:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.943157+0000", 2024-08-17T18:03:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034102463 2024-08-17T18:03:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:03:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.943225+0000", 2024-08-17T18:03:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8181000000000003e-05 2024-08-17T18:03:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:03:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.943339+0000", 2024-08-17T18:03:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114021 2024-08-17T18:03:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:03:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.947078+0000", 2024-08-17T18:03:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037382309999999998 2024-08-17T18:03:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.965825+0000", 2024-08-17T18:03:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018747526 2024-08-17T18:03:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.965902+0000", 2024-08-17T18:03:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7014000000000005e-05 2024-08-17T18:03:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.966705+0000", 2024-08-17T18:03:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00080280499999999997 2024-08-17T18:03:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:03:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.966751+0000", 2024-08-17T18:03:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6113999999999999e-05 2024-08-17T18:03:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:03:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.967341+0000", 2024-08-17T18:03:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00058978700000000004 2024-08-17T18:03:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:03:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:192422 54.12 54:4d52f0f7:::benchmark_data_smithi136_273969_object69537:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1414)", 2024-08-17T18:03:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:01:45.619860+0000", 2024-08-17T18:03:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 86.801917934000002, 2024-08-17T18:03:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3487727460000001, 2024-08-17T18:03:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:03:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:03:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:03:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:03:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:03:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:03:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 192422 2024-08-17T18:03:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:03:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:03:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619860+0000", 2024-08-17T18:03:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:03:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619860+0000", 2024-08-17T18:03:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:03:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619861+0000", 2024-08-17T18:03:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8099999999999998e-07 2024-08-17T18:03:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:03:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619862+0000", 2024-08-17T18:03:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.708e-06 2024-08-17T18:03:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:03:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619868+0000", 2024-08-17T18:03:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6690000000000003e-06 2024-08-17T18:03:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:03:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619873+0000", 2024-08-17T18:03:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6539999999999998e-06 2024-08-17T18:03:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619928+0000", 2024-08-17T18:03:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4738999999999999e-05 2024-08-17T18:03:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:03:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619936+0000", 2024-08-17T18:03:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3999999999999992e-06 2024-08-17T18:03:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.909101+0000", 2024-08-17T18:03:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2891654189999999 2024-08-17T18:03:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:03:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.909988+0000", 2024-08-17T18:03:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00088621800000000003 2024-08-17T18:03:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.949085+0000", 2024-08-17T18:03:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039097911999999999 2024-08-17T18:03:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:03:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.949205+0000", 2024-08-17T18:03:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011924799999999999 2024-08-17T18:03:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:03:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.949352+0000", 2024-08-17T18:03:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000147267 2024-08-17T18:03:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:03:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.959270+0000", 2024-08-17T18:03:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0099177729999999995 2024-08-17T18:03:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.965941+0000", 2024-08-17T18:03:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0066713639999999999 2024-08-17T18:03:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.967423+0000", 2024-08-17T18:03:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014821599999999999 2024-08-17T18:03:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.967526+0000", 2024-08-17T18:03:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010317700000000001 2024-08-17T18:03:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:03:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.967879+0000", 2024-08-17T18:03:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00035270100000000002 2024-08-17T18:03:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:03:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.968633+0000", 2024-08-17T18:03:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00075395599999999996 2024-08-17T18:03:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:03:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:192448 54.12 54:4c9f983c:::benchmark_data_smithi136_273969_object69563:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1414)", 2024-08-17T18:03:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:01:45.619892+0000", 2024-08-17T18:03:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 86.801886354000004, 2024-08-17T18:03:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.354143925, 2024-08-17T18:03:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:03:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:03:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:03:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:03:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:03:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:03:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 192448 2024-08-17T18:03:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:03:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:03:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619892+0000", 2024-08-17T18:03:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:03:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619892+0000", 2024-08-17T18:03:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:03:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619892+0000", 2024-08-17T18:03:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-08-17T18:03:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:03:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619894+0000", 2024-08-17T18:03:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7460000000000001e-06 2024-08-17T18:03:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:03:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619898+0000", 2024-08-17T18:03:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.284e-06 2024-08-17T18:03:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:03:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619902+0000", 2024-08-17T18:03:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1879999999999998e-06 2024-08-17T18:03:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619967+0000", 2024-08-17T18:03:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5599999999999995e-05 2024-08-17T18:03:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:03:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619973+0000", 2024-08-17T18:03:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4090000000000002e-06 2024-08-17T18:03:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.910038+0000", 2024-08-17T18:03:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2900656100000001 2024-08-17T18:03:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:03:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.910968+0000", 2024-08-17T18:03:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00092942299999999997 2024-08-17T18:03:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.955198+0000", 2024-08-17T18:03:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.044230250999999998 2024-08-17T18:03:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:03:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.955318+0000", 2024-08-17T18:03:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011972999999999999 2024-08-17T18:03:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:03:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.955480+0000", 2024-08-17T18:03:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016240499999999999 2024-08-17T18:03:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:03:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.961511+0000", 2024-08-17T18:03:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0060310759999999998 2024-08-17T18:03:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.967489+0000", 2024-08-17T18:03:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0059778540000000003 2024-08-17T18:03:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.968686+0000", 2024-08-17T18:03:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011974959999999999 2024-08-17T18:03:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.972800+0000", 2024-08-17T18:03:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.004113444 2024-08-17T18:03:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:03:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.972847+0000", 2024-08-17T18:03:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6999000000000002e-05 2024-08-17T18:03:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:03:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.974036+0000", 2024-08-17T18:03:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001189025 2024-08-17T18:03:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:03:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:192479 54.12 54:4a6669ca:::benchmark_data_smithi136_273969_object69594:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1414)", 2024-08-17T18:03:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:01:45.619918+0000", 2024-08-17T18:03:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 86.801860696000006, 2024-08-17T18:03:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3552398139999999, 2024-08-17T18:03:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:03:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:03:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:03:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:03:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:03:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:03:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 192479 2024-08-17T18:03:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:03:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:03:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619918+0000", 2024-08-17T18:03:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:03:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619918+0000", 2024-08-17T18:03:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:03:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619918+0000", 2024-08-17T18:03:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9499999999999998e-07 2024-08-17T18:03:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:03:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619919+0000", 2024-08-17T18:03:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.308e-06 2024-08-17T18:03:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:03:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619925+0000", 2024-08-17T18:03:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8429999999999998e-06 2024-08-17T18:03:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:03:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619929+0000", 2024-08-17T18:03:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6490000000000001e-06 2024-08-17T18:03:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.620011+0000", 2024-08-17T18:03:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1954999999999997e-05 2024-08-17T18:03:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:03:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.620019+0000", 2024-08-17T18:03:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7029999999999995e-06 2024-08-17T18:03:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.911012+0000", 2024-08-17T18:03:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.290992693 2024-08-17T18:03:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:03:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.911934+0000", 2024-08-17T18:03:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00092230800000000005 2024-08-17T18:03:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.959312+0000", 2024-08-17T18:03:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.047377895000000003 2024-08-17T18:03:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:03:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.959430+0000", 2024-08-17T18:03:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117564 2024-08-17T18:03:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:03:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.959581+0000", 2024-08-17T18:03:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015072800000000001 2024-08-17T18:03:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:03:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.964974+0000", 2024-08-17T18:03:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0053936230000000002 2024-08-17T18:03:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.974156+0000", 2024-08-17T18:03:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0091822620000000001 2024-08-17T18:03:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.974209+0000", 2024-08-17T18:03:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2967999999999998e-05 2024-08-17T18:03:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.974260+0000", 2024-08-17T18:03:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0090999999999999e-05 2024-08-17T18:03:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:03:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.974327+0000", 2024-08-17T18:03:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7335999999999994e-05 2024-08-17T18:03:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:03:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.975157+0000", 2024-08-17T18:03:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00083059300000000002 2024-08-17T18:03:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:03:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:192486 54.12 54:482c6c30:::benchmark_data_smithi136_273969_object69601:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1414)", 2024-08-17T18:03:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:01:45.619933+0000", 2024-08-17T18:03:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 86.801845626000002, 2024-08-17T18:03:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3790528040000001, 2024-08-17T18:03:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:03:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:03:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:03:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:03:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:03:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:03:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 192486 2024-08-17T18:03:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:03:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:03:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619933+0000", 2024-08-17T18:03:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:03:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619933+0000", 2024-08-17T18:03:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:03:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619933+0000", 2024-08-17T18:03:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2499999999999999e-07 2024-08-17T18:03:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:03:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619940+0000", 2024-08-17T18:03:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6889999999999998e-06 2024-08-17T18:03:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:03:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619943+0000", 2024-08-17T18:03:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3850000000000002e-06 2024-08-17T18:03:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:03:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.619946+0000", 2024-08-17T18:03:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6740000000000001e-06 2024-08-17T18:03:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.620042+0000", 2024-08-17T18:03:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6349999999999997e-05 2024-08-17T18:03:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:03:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.620048+0000", 2024-08-17T18:03:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.998e-06 2024-08-17T18:03:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.968728+0000", 2024-08-17T18:03:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.348679572 2024-08-17T18:03:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:03:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.971659+0000", 2024-08-17T18:03:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0029317050000000002 2024-08-17T18:03:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.983457+0000", 2024-08-17T18:03:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.01179798 2024-08-17T18:03:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:03:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.983561+0000", 2024-08-17T18:03:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103348 2024-08-17T18:03:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:03:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.983719+0000", 2024-08-17T18:03:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000158708 2024-08-17T18:03:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:03:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.991586+0000", 2024-08-17T18:03:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0078671439999999995 2024-08-17T18:03:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.998045+0000", 2024-08-17T18:03:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0064581889999999996 2024-08-17T18:03:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.998089+0000", 2024-08-17T18:03:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4499999999999997e-05 2024-08-17T18:03:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.998128+0000", 2024-08-17T18:03:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9084000000000001e-05 2024-08-17T18:03:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:03:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.998178+0000", 2024-08-17T18:03:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9496999999999997e-05 2024-08-17T18:03:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:03:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:46.998986+0000", 2024-08-17T18:03:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00080775599999999997 2024-08-17T18:03:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:03:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:203217 54.12 54:49799a76:::benchmark_data_smithi136_273969_object80332:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1419)", 2024-08-17T18:03:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:02:00.442081+0000", 2024-08-17T18:03:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 71.979697174999998, 2024-08-17T18:03:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.084169029000000006, 2024-08-17T18:03:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:03:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:03:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:03:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:03:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:03:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:03:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 203217 2024-08-17T18:03:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:03:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:03:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.442081+0000", 2024-08-17T18:03:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:03:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.442081+0000", 2024-08-17T18:03:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:03:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.442083+0000", 2024-08-17T18:03:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.53e-06 2024-08-17T18:03:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:03:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.442089+0000", 2024-08-17T18:03:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9580000000000004e-06 2024-08-17T18:03:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:03:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.442104+0000", 2024-08-17T18:03:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5447999999999998e-05 2024-08-17T18:03:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:03:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.442113+0000", 2024-08-17T18:03:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8699999999999998e-06 2024-08-17T18:03:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.442183+0000", 2024-08-17T18:03:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9942000000000002e-05 2024-08-17T18:03:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:03:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.443890+0000", 2024-08-17T18:03:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001707301 2024-08-17T18:03:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.450665+0000", 2024-08-17T18:03:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0067743760000000004 2024-08-17T18:03:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:03:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.450809+0000", 2024-08-17T18:03:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014455499999999999 2024-08-17T18:03:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:03:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.450936+0000", 2024-08-17T18:03:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012640000000000001 2024-08-17T18:03:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:03:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.454147+0000", 2024-08-17T18:03:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003211382 2024-08-17T18:03:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.456252+0000", 2024-08-17T18:03:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021055800000000001 2024-08-17T18:03:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.456697+0000", 2024-08-17T18:03:12.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044479799999999999 2024-08-17T18:03:12.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.525263+0000", 2024-08-17T18:03:12.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.068565223999999994 2024-08-17T18:03:12.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:03:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.525321+0000", 2024-08-17T18:03:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8539000000000003e-05 2024-08-17T18:03:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:03:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.526250+0000", 2024-08-17T18:03:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00092912599999999995 2024-08-17T18:03:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:03:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:203572 54.12 54:4abbfe40:::benchmark_data_smithi136_273969_object80687:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1419)", 2024-08-17T18:03:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:02:00.947800+0000", 2024-08-17T18:03:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 71.473977965000003, 2024-08-17T18:03:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.12528927100000001, 2024-08-17T18:03:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:03:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:03:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:03:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:03:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:03:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:03:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 203572 2024-08-17T18:03:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:03:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:03:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.947800+0000", 2024-08-17T18:03:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:03:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.947800+0000", 2024-08-17T18:03:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:03:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.947802+0000", 2024-08-17T18:03:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.595e-06 2024-08-17T18:03:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:03:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.947807+0000", 2024-08-17T18:03:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5530000000000003e-06 2024-08-17T18:03:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:03:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.947819+0000", 2024-08-17T18:03:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2551999999999999e-05 2024-08-17T18:03:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:03:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.947827+0000", 2024-08-17T18:03:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7340000000000008e-06 2024-08-17T18:03:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.953729+0000", 2024-08-17T18:03:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005902165 2024-08-17T18:03:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:03:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.954779+0000", 2024-08-17T18:03:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010503649999999999 2024-08-17T18:03:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.978480+0000", 2024-08-17T18:03:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023701048999999998 2024-08-17T18:03:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:03:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.978553+0000", 2024-08-17T18:03:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3052000000000005e-05 2024-08-17T18:03:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:03:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.978649+0000", 2024-08-17T18:03:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5266000000000006e-05 2024-08-17T18:03:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:03:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.989444+0000", 2024-08-17T18:03:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010795492 2024-08-17T18:03:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.991615+0000", 2024-08-17T18:03:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021708930000000001 2024-08-17T18:03:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.993896+0000", 2024-08-17T18:03:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002281244 2024-08-17T18:03:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:01.071939+0000", 2024-08-17T18:03:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078043161 2024-08-17T18:03:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:03:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:01.072043+0000", 2024-08-17T18:03:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103704 2024-08-17T18:03:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:03:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:01.073090+0000", 2024-08-17T18:03:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010464459999999999 2024-08-17T18:03:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:03:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:203594 54.12 54:4a5ccc73:::benchmark_data_smithi136_273969_object80709:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1419)", 2024-08-17T18:03:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:02:00.966922+0000", 2024-08-17T18:03:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 71.454856277000005, 2024-08-17T18:03:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.109366246, 2024-08-17T18:03:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:03:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:03:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:03:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:03:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:03:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:03:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 203594 2024-08-17T18:03:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:03:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:03:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.966922+0000", 2024-08-17T18:03:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:03:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.966922+0000", 2024-08-17T18:03:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:03:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.966924+0000", 2024-08-17T18:03:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5209999999999999e-06 2024-08-17T18:03:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:03:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.966929+0000", 2024-08-17T18:03:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2530000000000004e-06 2024-08-17T18:03:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:03:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.966941+0000", 2024-08-17T18:03:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1724e-05 2024-08-17T18:03:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:03:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.966948+0000", 2024-08-17T18:03:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6389999999999992e-06 2024-08-17T18:03:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.967406+0000", 2024-08-17T18:03:12.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00045780499999999998 2024-08-17T18:03:12.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:03:12.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.968500+0000", 2024-08-17T18:03:12.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010936699999999999 2024-08-17T18:03:12.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.983344+0000", 2024-08-17T18:03:12.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014844507 2024-08-17T18:03:12.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:03:12.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.983452+0000", 2024-08-17T18:03:12.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000108217 2024-08-17T18:03:12.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:03:12.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.983610+0000", 2024-08-17T18:03:12.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000157236 2024-08-17T18:03:12.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:03:12.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.989454+0000", 2024-08-17T18:03:12.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0058440310000000004 2024-08-17T18:03:12.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.991705+0000", 2024-08-17T18:03:12.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0022510199999999998 2024-08-17T18:03:12.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.994192+0000", 2024-08-17T18:03:12.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0024873970000000001 2024-08-17T18:03:12.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:01.075237+0000", 2024-08-17T18:03:12.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.081044448000000005 2024-08-17T18:03:12.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:03:12.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:01.075294+0000", 2024-08-17T18:03:12.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7188999999999997e-05 2024-08-17T18:03:12.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:03:12.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:01.076288+0000", 2024-08-17T18:03:12.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00099458899999999993 2024-08-17T18:03:12.585 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.586 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:03:12.586 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.586 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:203612 54.12 54:4fdcd645:::benchmark_data_smithi136_273969_object80727:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1419)", 2024-08-17T18:03:12.586 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:02:00.992884+0000", 2024-08-17T18:03:12.586 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 71.428894783000004, 2024-08-17T18:03:12.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.10038431, 2024-08-17T18:03:12.586 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:03:12.586 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:03:12.586 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:03:12.586 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:03:12.586 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:03:12.586 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:03:12.587 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 203612 2024-08-17T18:03:12.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.587 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:03:12.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:03:12.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.992884+0000", 2024-08-17T18:03:12.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:03:12.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.992884+0000", 2024-08-17T18:03:12.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:03:12.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.588 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:03:12.588 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.992885+0000", 2024-08-17T18:03:12.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.302e-06 2024-08-17T18:03:12.588 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.588 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:03:12.588 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.992890+0000", 2024-08-17T18:03:12.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6979999999999997e-06 2024-08-17T18:03:12.588 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.588 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:03:12.588 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.992901+0000", 2024-08-17T18:03:12.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1864e-05 2024-08-17T18:03:12.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:03:12.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.992907+0000", 2024-08-17T18:03:12.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4360000000000001e-06 2024-08-17T18:03:12.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.992962+0000", 2024-08-17T18:03:12.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5260999999999998e-05 2024-08-17T18:03:12.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:03:12.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:00.993730+0000", 2024-08-17T18:03:12.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00076774999999999999 2024-08-17T18:03:12.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:03:12.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:01.078480+0000", 2024-08-17T18:03:12.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.084750184000000006 2024-08-17T18:03:12.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:03:12.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:01.078577+0000", 2024-08-17T18:03:12.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6837999999999997e-05 2024-08-17T18:03:12.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:03:12.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:01.078708+0000", 2024-08-17T18:03:12.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000131419 2024-08-17T18:03:12.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:03:12.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:01.082021+0000", 2024-08-17T18:03:12.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003312316 2024-08-17T18:03:12.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:01.084097+0000", 2024-08-17T18:03:12.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002076535 2024-08-17T18:03:12.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.592 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.592 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:01.090480+0000", 2024-08-17T18:03:12.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063833429999999997 2024-08-17T18:03:12.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.592 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:03:12.592 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:01.092106+0000", 2024-08-17T18:03:12.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016254660000000001 2024-08-17T18:03:12.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.592 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:03:12.592 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:01.092282+0000", 2024-08-17T18:03:12.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017652600000000001 2024-08-17T18:03:12.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:03:12.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:03:12.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:03:12.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:02:01.093268+0000", 2024-08-17T18:03:12.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00098537199999999998 2024-08-17T18:03:12.593 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.593 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:03:12.593 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.593 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:03:12.593 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:03:12.593 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:03:12.614 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:12.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:12.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:12.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:12.738 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:12.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:12.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:12.839 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:12.840 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:03:12.862 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:12.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:12.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:12.964 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:12.984 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:12.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:12.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:13.086 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:13.109 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:13.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:13.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:13.144 INFO:teuthology.orchestra.run.smithi136.stderr:no valid command found; 10 closest matches: 2024-08-17T18:03:13.144 INFO:teuthology.orchestra.run.smithi136.stderr:0 2024-08-17T18:03:13.144 INFO:teuthology.orchestra.run.smithi136.stderr:1 2024-08-17T18:03:13.144 INFO:teuthology.orchestra.run.smithi136.stderr:2 2024-08-17T18:03:13.144 INFO:teuthology.orchestra.run.smithi136.stderr:abort 2024-08-17T18:03:13.144 INFO:teuthology.orchestra.run.smithi136.stderr:assert 2024-08-17T18:03:13.145 INFO:teuthology.orchestra.run.smithi136.stderr:bench [] [] [] [] 2024-08-17T18:03:13.145 INFO:teuthology.orchestra.run.smithi136.stderr:bluefs debug_inject_read_zeros 2024-08-17T18:03:13.145 INFO:teuthology.orchestra.run.smithi136.stderr:bluefs files list 2024-08-17T18:03:13.145 INFO:teuthology.orchestra.run.smithi136.stderr:bluefs stats 2024-08-17T18:03:13.145 INFO:teuthology.orchestra.run.smithi136.stderr:bluestore allocator dump block 2024-08-17T18:03:13.145 INFO:teuthology.orchestra.run.smithi136.stderr:Error EINVAL: invalid command 2024-08-17T18:03:13.211 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:13.234 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:13.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:13.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:13.337 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:13.360 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:13.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:13.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:13.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:13 smithi136 ceph-mon[109861]: pgmap v2508: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 9.9 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 639 op/s; 5123/528 objects degraded (970.265%); 3213/528 objects misplaced (608.523%); 1.8 MiB/s, 140 objects/s recovering 2024-08-17T18:03:13.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:13 smithi136 ceph-mon[115078]: pgmap v2508: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 9.9 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 639 op/s; 5123/528 objects degraded (970.265%); 3213/528 objects misplaced (608.523%); 1.8 MiB/s, 140 objects/s recovering 2024-08-17T18:03:13.462 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:13.486 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:13.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:13.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:13.581 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:03:13.587 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:13.609 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:13.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:13.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:13.711 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:13.733 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:13.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:13.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:13 smithi165 ceph-mon[105517]: pgmap v2508: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 9.9 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 639 op/s; 5123/528 objects degraded (970.265%); 3213/528 objects misplaced (608.523%); 1.8 MiB/s, 140 objects/s recovering 2024-08-17T18:03:13.836 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:13.858 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:13.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:13.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:13.960 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:13.982 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:13.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:13.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:14.079 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-17T18:03:14.080 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-17T18:03:14.080 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-08-17T18:03:14.084 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:14.143 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:14.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:14.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:14.245 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:14.270 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:14.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:14.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:14.372 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:14.382 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:03:14.395 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:14.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:14.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:14.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:14.519 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:14.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:14.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:14.621 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:14.644 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:14.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:14.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:14.746 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:14.767 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:14.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:14.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:14.869 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:14.893 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:14.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:14.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:14.995 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:15.017 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:15.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:15.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:15.079 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-17T18:03:15.119 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:15.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:15.151 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:15.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:15.252 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:15.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:15.274 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:15.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:15.376 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:15.399 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:15.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:15.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:15.501 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:15.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:15.523 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:15.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:15.624 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:15.645 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:15.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:15.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:15.661 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:15 smithi136 ceph-mon[109861]: pgmap v2509: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 7.5 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 569 op/s; 5123/414 objects degraded (1237.440%); 3213/414 objects misplaced (776.087%); 793 KiB/s, 227 objects/s recovering 2024-08-17T18:03:15.661 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:15 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5123/414 objects degraded (1237.440%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T18:03:15.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:15 smithi136 ceph-mon[115078]: pgmap v2509: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 7.5 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 569 op/s; 5123/414 objects degraded (1237.440%); 3213/414 objects misplaced (776.087%); 793 KiB/s, 227 objects/s recovering 2024-08-17T18:03:15.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:15 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5123/414 objects degraded (1237.440%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T18:03:15.747 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:15.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:15 smithi165 ceph-mon[105517]: pgmap v2509: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 7.5 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 569 op/s; 5123/414 objects degraded (1237.440%); 3213/414 objects misplaced (776.087%); 793 KiB/s, 227 objects/s recovering 2024-08-17T18:03:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:15 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5123/414 objects degraded (1237.440%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T18:03:15.771 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:15.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:15.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:15.873 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:15.894 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:15.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:15.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:15.996 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:16.014 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:03:16.014 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:03:16.024 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:16.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:16.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:16.040 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:03:16.126 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:16.147 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:16.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:16.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:16.249 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:16.272 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:16.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:16.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:16.374 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:16.396 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:16.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:16.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:16.498 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:16.520 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:16.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:16.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:16.622 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:16.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:16.645 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:16.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:16.746 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:16.768 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:16.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:16.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:16.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:16.891 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:16.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:16.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:16.994 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:17.017 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:17.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:17.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:17.094 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:03:17.120 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:17.141 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:17.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:17.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:17.243 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:17.265 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:17.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:17.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:17.367 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:17.392 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:17.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:17.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:17.493 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:17.516 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:17.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:17.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:17.618 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:17.621 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:17 smithi136 ceph-mon[109861]: pgmap v2510: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 7.5 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 514 op/s; 5123/414 objects degraded (1237.440%); 3213/414 objects misplaced (776.087%); 732 KiB/s, 209 objects/s recovering 2024-08-17T18:03:17.622 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:03:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:03:17] "GET /metrics HTTP/1.1" 200 36638 "" "Prometheus/2.51.0" 2024-08-17T18:03:17.622 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:17 smithi136 ceph-mon[115078]: pgmap v2510: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 7.5 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 514 op/s; 5123/414 objects degraded (1237.440%); 3213/414 objects misplaced (776.087%); 732 KiB/s, 209 objects/s recovering 2024-08-17T18:03:17.642 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:17.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:17.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:17.744 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:17 smithi165 ceph-mon[105517]: pgmap v2510: 99 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 7.5 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 514 op/s; 5123/414 objects degraded (1237.440%); 3213/414 objects misplaced (776.087%); 732 KiB/s, 209 objects/s recovering 2024-08-17T18:03:17.765 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:17.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:17.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:17.867 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:17.889 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:17.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:17.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:17.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:18.014 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:18.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:18.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:18.116 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:18.138 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:18.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:18.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:18.240 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:18.264 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:18.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:18.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:18.366 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:18.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:18.395 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:18.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:18.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:18.519 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:18.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:18.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:18.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:18.644 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:18.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:18.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:18.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:18 smithi136 ceph-mon[109861]: osdmap e1437: 8 total, 8 up, 6 in 2024-08-17T18:03:18.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:18 smithi136 ceph-mon[115078]: osdmap e1437: 8 total, 8 up, 6 in 2024-08-17T18:03:18.746 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:18.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:18 smithi165 ceph-mon[105517]: osdmap e1437: 8 total, 8 up, 6 in 2024-08-17T18:03:18.768 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:18.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:18.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:18.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:18.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:18.892 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:18.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:18.994 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:19.015 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:19.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:19.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:19.067 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:03:19.067 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:03:19.078 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:03:19.078 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:03:19.094 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:03:19.294 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:03:19.294 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:03:19.294 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:03:19.295 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:03:19.295 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:03:19.295 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:03:19.295 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:03:19.296 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:03:19.296 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:03:19.296 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:03:19.296 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:03:19.297 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:19.320 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:19.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:19.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:19.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:19.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:19.451 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:19.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:19.552 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:19.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:19.577 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:19.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:19.630 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:19 smithi136 ceph-mon[109861]: pgmap v2512: 99 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 7.5 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 116 op/s; 5123/414 objects degraded (1237.440%); 3213/414 objects misplaced (776.087%); 0 B/s, 443 objects/s recovering 2024-08-17T18:03:19.630 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:19 smithi136 ceph-mon[109861]: osdmap e1438: 8 total, 8 up, 6 in 2024-08-17T18:03:19.630 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:19 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3927397712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:03:19.631 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:19 smithi136 ceph-mon[115078]: pgmap v2512: 99 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 7.5 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 116 op/s; 5123/414 objects degraded (1237.440%); 3213/414 objects misplaced (776.087%); 0 B/s, 443 objects/s recovering 2024-08-17T18:03:19.631 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:19 smithi136 ceph-mon[115078]: osdmap e1438: 8 total, 8 up, 6 in 2024-08-17T18:03:19.631 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:19 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3927397712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:03:19.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:19.702 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:19.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:19.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:19 smithi165 ceph-mon[105517]: pgmap v2512: 99 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 7.5 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 0 B/s wr, 116 op/s; 5123/414 objects degraded (1237.440%); 3213/414 objects misplaced (776.087%); 0 B/s, 443 objects/s recovering 2024-08-17T18:03:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:19 smithi165 ceph-mon[105517]: osdmap e1438: 8 total, 8 up, 6 in 2024-08-17T18:03:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:19 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3927397712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:03:19.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:19.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:19.829 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:19.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:19.930 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:19.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:19.953 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:19.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:20.054 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:20.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:20.079 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:20.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:20.180 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:20.204 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:20.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:20.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:20.306 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:20.327 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:20.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:20.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:20.429 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:20.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:20.452 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:20.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:20.553 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:20.577 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:20.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:20.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:20.642 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:20 smithi136 ceph-mon[109861]: osdmap e1439: 8 total, 8 up, 6 in 2024-08-17T18:03:20.643 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:20 smithi136 ceph-mon[109861]: osdmap e1440: 8 total, 8 up, 6 in 2024-08-17T18:03:20.643 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:20 smithi136 ceph-mon[115078]: osdmap e1439: 8 total, 8 up, 6 in 2024-08-17T18:03:20.643 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:20 smithi136 ceph-mon[115078]: osdmap e1440: 8 total, 8 up, 6 in 2024-08-17T18:03:20.680 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:20.703 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:20.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:20.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:20 smithi165 ceph-mon[105517]: osdmap e1439: 8 total, 8 up, 6 in 2024-08-17T18:03:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:20 smithi165 ceph-mon[105517]: osdmap e1440: 8 total, 8 up, 6 in 2024-08-17T18:03:20.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:20.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:20.828 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:20.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:20.929 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:20.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:20.954 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:20.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:21.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:21.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:21.079 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:21.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:21.180 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:21.203 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:21.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:21.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:21.305 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:21.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:21.328 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:21.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:21.429 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:21.451 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:21.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:21.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:21.553 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:21.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:21.577 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:21.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:21.679 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:21.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:21.702 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:21.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:21 smithi165 ceph-mon[105517]: pgmap v2516: 99 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 7.5 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 5123/414 objects degraded (1237.440%); 3213/414 objects misplaced (776.087%); 0 B/s, 667 objects/s recovering 2024-08-17T18:03:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:21 smithi165 ceph-mon[105517]: osdmap e1441: 8 total, 8 up, 6 in 2024-08-17T18:03:21.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:03:21.790 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:21 smithi136 ceph-mon[109861]: pgmap v2516: 99 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 7.5 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 5123/414 objects degraded (1237.440%); 3213/414 objects misplaced (776.087%); 0 B/s, 667 objects/s recovering 2024-08-17T18:03:21.791 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:21 smithi136 ceph-mon[109861]: osdmap e1441: 8 total, 8 up, 6 in 2024-08-17T18:03:21.791 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:03:21.791 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:21 smithi136 ceph-mon[115078]: pgmap v2516: 99 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 7.5 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 5123/414 objects degraded (1237.440%); 3213/414 objects misplaced (776.087%); 0 B/s, 667 objects/s recovering 2024-08-17T18:03:21.791 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:21 smithi136 ceph-mon[115078]: osdmap e1441: 8 total, 8 up, 6 in 2024-08-17T18:03:21.791 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:03:21.803 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:21.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:21.826 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:21.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:21.929 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:21.950 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:21.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:21.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:22.052 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:22.076 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:22.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:22.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:22.178 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:22.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:22.201 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:22.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:22.302 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:22.328 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:22.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:22.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:22.430 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:22.452 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:22.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:22.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:22.517 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'unique_pool_1' removed 2024-08-17T18:03:22.553 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:22.601 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:22.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:22.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:22.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:22.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:22.740 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:22.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:22.841 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:22.863 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:22.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:22.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:22.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:22 smithi136 ceph-mon[109861]: osdmap e1442: 8 total, 8 up, 6 in 2024-08-17T18:03:22.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:22 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3188120303' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T18:03:22.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:22 smithi136 ceph-mon[115078]: osdmap e1442: 8 total, 8 up, 6 in 2024-08-17T18:03:22.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:22 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3188120303' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T18:03:22.964 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:22.987 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:22.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:22.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:22 smithi165 ceph-mon[105517]: osdmap e1442: 8 total, 8 up, 6 in 2024-08-17T18:03:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:22 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3188120303' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T18:03:23.089 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:23.110 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:23.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:23.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:23.212 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:23.233 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:23.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:23.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:23.335 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:23.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:23.358 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:23.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:23.459 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:23.469 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-17T18:03:23.470 INFO:tasks.radosbench:Beginning radosbench... 2024-08-17T18:03:23.470 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-08-17T18:03:23.471 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-08-17T18:03:23.480 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:23.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:23.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:23.582 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:23.606 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:23.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:23.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:23.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:23 smithi136 ceph-mon[109861]: 54.13 scrub starts 2024-08-17T18:03:23.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:23 smithi136 ceph-mon[109861]: pgmap v2519: 99 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 7.5 MiB data, 3.1 GiB used, 533 GiB / 536 GiB avail; 1666/414 objects degraded (402.415%); 3213/414 objects misplaced (776.087%); 0 B/s, 467 objects/s recovering 2024-08-17T18:03:23.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:23 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 1666/414 objects degraded (402.415%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:03:23.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:23 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3188120303' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T18:03:23.700 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:23 smithi136 ceph-mon[109861]: osdmap e1443: 8 total, 8 up, 6 in 2024-08-17T18:03:23.700 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:23 smithi136 ceph-mon[115078]: 54.13 scrub starts 2024-08-17T18:03:23.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:23 smithi136 ceph-mon[115078]: pgmap v2519: 99 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 7.5 MiB data, 3.1 GiB used, 533 GiB / 536 GiB avail; 1666/414 objects degraded (402.415%); 3213/414 objects misplaced (776.087%); 0 B/s, 467 objects/s recovering 2024-08-17T18:03:23.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:23 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 1666/414 objects degraded (402.415%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:03:23.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:23 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3188120303' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T18:03:23.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:23 smithi136 ceph-mon[115078]: osdmap e1443: 8 total, 8 up, 6 in 2024-08-17T18:03:23.704 INFO:teuthology.orchestra.run.smithi136.stderr:no valid command found; 10 closest matches: 2024-08-17T18:03:23.704 INFO:teuthology.orchestra.run.smithi136.stderr:0 2024-08-17T18:03:23.704 INFO:teuthology.orchestra.run.smithi136.stderr:1 2024-08-17T18:03:23.704 INFO:teuthology.orchestra.run.smithi136.stderr:2 2024-08-17T18:03:23.704 INFO:teuthology.orchestra.run.smithi136.stderr:abort 2024-08-17T18:03:23.704 INFO:teuthology.orchestra.run.smithi136.stderr:assert 2024-08-17T18:03:23.704 INFO:teuthology.orchestra.run.smithi136.stderr:bench [] [] [] [] 2024-08-17T18:03:23.704 INFO:teuthology.orchestra.run.smithi136.stderr:bluefs debug_inject_read_zeros 2024-08-17T18:03:23.704 INFO:teuthology.orchestra.run.smithi136.stderr:bluefs files list 2024-08-17T18:03:23.705 INFO:teuthology.orchestra.run.smithi136.stderr:bluefs stats 2024-08-17T18:03:23.705 INFO:teuthology.orchestra.run.smithi136.stderr:bluestore allocator dump block 2024-08-17T18:03:23.705 INFO:teuthology.orchestra.run.smithi136.stderr:Error EINVAL: invalid command 2024-08-17T18:03:23.708 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:23.730 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:23.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:23.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:23.832 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:23.854 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:23.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:23.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:23.956 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:23.978 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:23.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:23.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:23 smithi165 ceph-mon[105517]: 54.13 scrub starts 2024-08-17T18:03:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:23 smithi165 ceph-mon[105517]: pgmap v2519: 99 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 7.5 MiB data, 3.1 GiB used, 533 GiB / 536 GiB avail; 1666/414 objects degraded (402.415%); 3213/414 objects misplaced (776.087%); 0 B/s, 467 objects/s recovering 2024-08-17T18:03:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:23 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 1666/414 objects degraded (402.415%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:03:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:23 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3188120303' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T18:03:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:23 smithi165 ceph-mon[105517]: osdmap e1443: 8 total, 8 up, 6 in 2024-08-17T18:03:24.080 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:24.102 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:24.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:24.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:24.204 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:24.207 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:03:24.225 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:24.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:24.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:24.327 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:24.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:24.350 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:24.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:24.451 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:24.473 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:24.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:24.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:24.519 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:03:24.575 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:24.597 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:24.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:24.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:24.600 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-17T18:03:24.601 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-17T18:03:24.601 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-08-17T18:03:24.698 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:24.720 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:24.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:24.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:24.822 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:24.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:24.845 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:24.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:24.901 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:24 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2870317897' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:03:24.902 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:24 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:03:24.902 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:24 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2870317897' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:03:24.902 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:24 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:03:24.946 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:24.967 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:24.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:24.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:25.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:24 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2870317897' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:03:25.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:24 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:03:25.069 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:25.092 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:25.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:25.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:25.194 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:25.215 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:25.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:25.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:25.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:25.338 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:25.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:25.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:25.440 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:25.462 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:25.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:25.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:25.563 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:25.585 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:25.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:25.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:25.687 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:25.709 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:25.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:25.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:25.814 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:25.816 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:25 smithi136 ceph-mon[109861]: pgmap v2521: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 533 GiB / 536 GiB avail 2024-08-17T18:03:25.816 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:25 smithi136 ceph-mon[109861]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1666/414 objects degraded (402.415%), 2 pgs degraded, 2 pgs undersized) 2024-08-17T18:03:25.816 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:25 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:03:25.816 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:25 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:03:25.816 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:25 smithi136 ceph-mon[109861]: osdmap e1444: 8 total, 8 up, 6 in 2024-08-17T18:03:25.817 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:25 smithi136 ceph-mon[115078]: pgmap v2521: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 533 GiB / 536 GiB avail 2024-08-17T18:03:25.817 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:25 smithi136 ceph-mon[115078]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1666/414 objects degraded (402.415%), 2 pgs degraded, 2 pgs undersized) 2024-08-17T18:03:25.817 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:25 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:03:25.817 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:25 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:03:25.817 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:25 smithi136 ceph-mon[115078]: osdmap e1444: 8 total, 8 up, 6 in 2024-08-17T18:03:25.837 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:25.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:25.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:25.939 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:25.962 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:25.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:25.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:25 smithi165 ceph-mon[105517]: pgmap v2521: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 533 GiB / 536 GiB avail 2024-08-17T18:03:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:25 smithi165 ceph-mon[105517]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1666/414 objects degraded (402.415%), 2 pgs degraded, 2 pgs undersized) 2024-08-17T18:03:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:25 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:03:26.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:25 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:03:26.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:25 smithi165 ceph-mon[105517]: osdmap e1444: 8 total, 8 up, 6 in 2024-08-17T18:03:26.044 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:03:26.044 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:03:26.065 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:26.088 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:26.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:26.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:26.190 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:26.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:26.213 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:26.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:26.314 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:26.338 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:26.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:26.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:26.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:26.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:26.462 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:26.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:26.563 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:26.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:26.585 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:26.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:26.686 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:26.709 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:26.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:26.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:26.740 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-17T18:03:26.795 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:03:26.802 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:03:26.811 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:26.834 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:03:26.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:26.839 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:26.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:26.940 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:26.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:26.963 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:26.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:27.064 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:27.085 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:27.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:27.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:27.187 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:27.209 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:27.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:27.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:27.311 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:27.332 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:27.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:27.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:27.434 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:27.457 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:27.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:27.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:27.559 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:27.580 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:27.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:27.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:27.611 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:03:27.643 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:27 smithi136 ceph-mon[109861]: pgmap v2523: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 533 GiB / 536 GiB avail 2024-08-17T18:03:27.643 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:27 smithi136 ceph-mon[115078]: pgmap v2523: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 533 GiB / 536 GiB avail 2024-08-17T18:03:27.644 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:03:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:03:27] "GET /metrics HTTP/1.1" 200 36622 "" "Prometheus/2.51.0" 2024-08-17T18:03:27.682 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:27.709 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:27.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:27.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:27.810 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:27.835 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:27.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:27.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:27.937 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:27.959 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:27.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:27.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:28.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:27 smithi165 ceph-mon[105517]: pgmap v2523: 73 pgs: 73 active+clean; 582 KiB data, 3.1 GiB used, 533 GiB / 536 GiB avail 2024-08-17T18:03:28.062 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:28.084 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:28.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:28.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:28.185 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:28.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:28.208 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:28.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:28.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:28.336 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:28.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:28.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:28.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:28.463 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:28.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:28.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:28.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:28.589 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:28.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:28.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:28.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:28.716 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:28.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:28.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:28.818 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:28.843 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:28.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:28.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:28.945 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:28.970 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:28.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:28.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:29.072 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:29.095 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:29.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:29.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:29.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:29.223 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:29.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:29.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:29.325 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:29.350 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:29.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:29.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:29.452 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:29.478 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:29.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:29.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:29.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:29.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:29.606 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:29.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:29.707 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:29.732 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:29.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:29.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:29.834 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:29.841 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:29 smithi136 ceph-mon[109861]: pgmap v2524: 73 pgs: 73 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:29.841 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:29 smithi136 ceph-mon[115078]: pgmap v2524: 73 pgs: 73 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:29.861 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:29.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:29.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:29.962 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:29.987 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:29.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:29.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:29 smithi165 ceph-mon[105517]: pgmap v2524: 73 pgs: 73 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:30.089 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:30.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:30.115 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:30.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:30.216 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:30.239 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:30.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:30.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:30.340 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:30.362 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:30.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:30.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:30.464 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:30.489 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:30.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:30.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:30.591 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:30.616 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:30.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:30.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:30.718 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:30.739 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:30.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:30.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:30.841 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:30.867 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:30.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:30.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:30.909 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:03:30.923 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:03:30.969 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:30.994 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:30.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:30.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:31.096 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:31.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:31.119 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:31.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:31.220 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:31.241 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:31.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:31.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:31.343 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:31.369 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:31.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:31.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:31.471 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:31.493 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:31.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:31.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:31.595 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:31.616 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:31.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:31.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:31.619 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:31 smithi136 ceph-mon[109861]: pgmap v2525: 73 pgs: 73 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:31.620 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:31 smithi136 ceph-mon[115078]: pgmap v2525: 73 pgs: 73 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:31.718 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:31.740 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:31.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:31.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:31.842 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:31.868 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:31.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:31.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:31.970 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:31.995 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:31.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:31.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:31 smithi165 ceph-mon[105517]: pgmap v2525: 73 pgs: 73 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:32.097 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:32.122 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:32.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:32.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:32.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:32.249 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:32.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:32.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:32.351 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:32.372 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:32.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:32.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:32.474 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:32.496 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:32.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:32.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:32.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:32.619 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:32.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:32.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:32.721 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:32.742 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:32.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:32.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:32.844 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:32.870 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:32.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:32.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:32.972 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:32.997 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:32.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:32.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:33.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:33.123 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:33.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:33.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:33.225 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:33.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:33.247 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:33.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:33.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:33.371 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:33.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:33.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:33.473 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:33.496 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:33.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:33.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:33.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:33.622 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:33.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:33.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:33.652 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:33 smithi136 ceph-mon[109861]: pgmap v2526: 73 pgs: 73 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:33.653 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:33 smithi136 ceph-mon[115078]: pgmap v2526: 73 pgs: 73 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:33.724 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:33.746 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:33.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:33.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:33.848 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:33.869 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:33.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:33.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:33.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:33.997 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:33.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:33.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:33 smithi165 ceph-mon[105517]: pgmap v2526: 73 pgs: 73 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:34.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:34.125 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:34.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:34.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:34.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:34.251 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:34.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:34.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:34.301 INFO:teuthology.orchestra.run.smithi136.stderr:no valid command found; 10 closest matches: 2024-08-17T18:03:34.301 INFO:teuthology.orchestra.run.smithi136.stderr:0 2024-08-17T18:03:34.301 INFO:teuthology.orchestra.run.smithi136.stderr:1 2024-08-17T18:03:34.301 INFO:teuthology.orchestra.run.smithi136.stderr:2 2024-08-17T18:03:34.301 INFO:teuthology.orchestra.run.smithi136.stderr:abort 2024-08-17T18:03:34.301 INFO:teuthology.orchestra.run.smithi136.stderr:assert 2024-08-17T18:03:34.301 INFO:teuthology.orchestra.run.smithi136.stderr:bench [] [] [] [] 2024-08-17T18:03:34.301 INFO:teuthology.orchestra.run.smithi136.stderr:bluefs debug_inject_read_zeros 2024-08-17T18:03:34.301 INFO:teuthology.orchestra.run.smithi136.stderr:bluefs files list 2024-08-17T18:03:34.301 INFO:teuthology.orchestra.run.smithi136.stderr:bluefs stats 2024-08-17T18:03:34.301 INFO:teuthology.orchestra.run.smithi136.stderr:bluestore allocator dump block 2024-08-17T18:03:34.301 INFO:teuthology.orchestra.run.smithi136.stderr:Error EINVAL: invalid command 2024-08-17T18:03:34.354 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:34.379 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:34.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:34.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:34.481 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:34.505 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:34.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:34.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:34.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:34.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:34.631 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:34.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:34.642 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:03:34.643 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:03:34.659 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:03:34.659 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:03:34.674 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:03:34.675 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:03:34.691 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:03:34.691 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:03:34.708 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:03:34.708 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:03:34.724 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:03:34.725 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:03:34.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:34.740 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:03:34.741 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:03:34.753 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:34.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:34.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:34.757 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:03:34.757 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:03:34.855 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:34.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:34.879 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:34.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:34.980 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:35.003 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:35.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:35.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:35.104 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:35.126 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:35.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:35.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:35.228 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:35.251 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:35.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:35.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:35.353 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:35.375 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:35.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:35.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:35.476 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:35.499 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:35.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:35.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:35.601 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:35.623 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:35.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:35.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:35.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:35 smithi136 ceph-mon[109861]: pgmap v2527: 73 pgs: 73 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:35.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:35 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2840329645' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:03:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:35 smithi136 ceph-mon[115078]: pgmap v2527: 73 pgs: 73 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:35.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:35 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2840329645' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:03:35.724 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:35.748 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:35.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:35.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:35.850 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:35.861 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:03:35.861 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T18:03:35.862 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-17T18:03:35.862 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-17T18:03:35.862 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:03:35.873 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:35.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:35.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:35.975 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:35.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:35.999 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:35.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:35 smithi165 ceph-mon[105517]: pgmap v2527: 73 pgs: 73 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:35 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2840329645' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:03:36.101 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:36.124 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:36.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:36.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:36.226 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:36.252 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:36.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:36.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:36.354 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:36.376 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:36.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:36.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:36.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:36.500 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:36.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:36.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:36.602 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:36.623 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:36.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:36.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:36.725 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:36.750 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:36.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:36.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:36.851 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:36.875 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:36.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:36.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:36.977 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:36.999 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:37.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:37.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:37.101 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:37.125 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:37.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:37.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:37.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:37.236 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:37 smithi136 ceph-mon[115078]: pgmap v2528: 73 pgs: 73 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:37.236 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:03:37.236 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:03:37.237 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:37 smithi136 ceph-mon[109861]: pgmap v2528: 73 pgs: 73 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:37.237 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:03:37.237 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:03:37.251 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:37.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:37.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:37.353 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:37.376 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:37.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:37.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:37.478 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:37.503 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:37.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:37.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:37 smithi165 ceph-mon[105517]: pgmap v2528: 73 pgs: 73 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:03:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:03:37.543 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:03:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:03:37] "GET /metrics HTTP/1.1" 200 34201 "" "Prometheus/2.51.0" 2024-08-17T18:03:37.604 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:37.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:37.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:37.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:37.731 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:37.754 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:37.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:37.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:37.856 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-17T18:03:37.856 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:37.882 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:37.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:37.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:37.984 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:38.009 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:38.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:38.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:38.110 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:38.136 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:38.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:38.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:38.191 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-17T18:03:38.191 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-17T18:03:38.212 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:03:38.238 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:38.262 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:38.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:38.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:38.364 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:38.384 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:38.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:38.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:38.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:38 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2888952779' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:03:38.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:38 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2888952779' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:03:38.460 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:03:38.487 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:38.508 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:38.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:38.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:38.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:38 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2888952779' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:03:38.610 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:38.636 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:38.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:38.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:38.723 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:03:38.737 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:38.759 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:38.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:38.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:38.861 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:38.882 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:38.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:38.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:38.984 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:39.006 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:03:39.007 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:39.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:39.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:39.109 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:39.131 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:39.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:39.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:39.214 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'unique_pool_2' created 2024-08-17T18:03:39.233 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:39.270 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:39.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:39.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:39.372 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:39.393 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:39.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:39.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:39.460 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:39 smithi136 ceph-mon[109861]: pgmap v2529: 73 pgs: 73 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:39.461 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:39 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:03:39.461 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:39 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2888952779' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:03:39.461 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:39 smithi136 ceph-mon[109861]: osdmap e1445: 8 total, 8 up, 6 in 2024-08-17T18:03:39.461 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:39 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3576447878' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-08-17T18:03:39.461 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:39 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:03:39.461 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:39 smithi136 ceph-mon[115078]: pgmap v2529: 73 pgs: 73 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:39.461 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:39 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:03:39.461 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:39 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2888952779' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:03:39.461 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:39 smithi136 ceph-mon[115078]: osdmap e1445: 8 total, 8 up, 6 in 2024-08-17T18:03:39.461 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:39 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3576447878' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-08-17T18:03:39.461 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:39 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:03:39.495 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:39.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:39 smithi165 ceph-mon[105517]: pgmap v2529: 73 pgs: 73 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:39.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:39 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:03:39.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:39 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2888952779' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:03:39.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:39 smithi165 ceph-mon[105517]: osdmap e1445: 8 total, 8 up, 6 in 2024-08-17T18:03:39.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:39 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3576447878' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-08-17T18:03:39.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:39 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:03:39.516 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:39.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:39.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:39.618 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:39.640 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:39.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:39.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:39.741 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:39.763 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:39.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:39.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:39.865 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:39.887 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:39.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:39.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:39.988 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:40.011 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:40.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:40.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:40.113 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:40.134 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:40.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:40.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:40.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:40.256 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:40.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:40.358 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:40.380 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:40.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:40.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:40.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:40 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3576447878' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-08-17T18:03:40.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:40 smithi136 ceph-mon[115078]: osdmap e1446: 8 total, 8 up, 6 in 2024-08-17T18:03:40.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:40 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3576447878' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-08-17T18:03:40.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:40 smithi136 ceph-mon[109861]: osdmap e1446: 8 total, 8 up, 6 in 2024-08-17T18:03:40.481 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:40.504 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:40.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:40.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:40.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:40 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3576447878' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-08-17T18:03:40.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:40 smithi165 ceph-mon[105517]: osdmap e1446: 8 total, 8 up, 6 in 2024-08-17T18:03:40.581 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-08-17T18:03:40.606 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:40.645 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:40.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:40.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:40.747 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:40.769 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:40.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:40.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:40.889 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:40.911 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:40.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:40.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:40.957 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:03:41.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:41.034 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:41.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:41.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:41.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:41.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:41.189 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:41.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:41.290 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:41.312 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:41.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:41.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:41.414 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:41.435 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:41.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:41.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:41.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:41 smithi136 ceph-mon[109861]: pgmap v2532: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:41.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:41 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T18:03:41.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:41 smithi136 ceph-mon[109861]: osdmap e1447: 8 total, 8 up, 6 in 2024-08-17T18:03:41.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:41 smithi136 ceph-mon[115078]: pgmap v2532: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:41.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:41 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T18:03:41.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:41 smithi136 ceph-mon[115078]: osdmap e1447: 8 total, 8 up, 6 in 2024-08-17T18:03:41.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:41 smithi165 ceph-mon[105517]: pgmap v2532: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:41.544 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:41 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T18:03:41.544 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:41 smithi165 ceph-mon[105517]: osdmap e1447: 8 total, 8 up, 6 in 2024-08-17T18:03:41.544 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:41.566 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:41.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:41.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:41.668 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:41.690 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:41.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:41.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:41.792 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:41.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:41.815 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:41.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:41.916 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:41.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:41.938 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:41.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:42.039 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:42.061 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:42.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:42.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:42.163 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:42.184 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:42.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:42.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:42.287 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:42.308 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:42.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:42.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:42.410 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:42.432 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:42.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:42.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:42.458 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:42 smithi136 ceph-mon[109861]: osdmap e1448: 8 total, 8 up, 6 in 2024-08-17T18:03:42.458 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:42 smithi136 ceph-mon[115078]: osdmap e1448: 8 total, 8 up, 6 in 2024-08-17T18:03:42.476 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:03:42.477 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:03:42.493 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:03:42.493 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:03:42.509 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:03:42.509 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:03:42.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:42 smithi165 ceph-mon[105517]: osdmap e1448: 8 total, 8 up, 6 in 2024-08-17T18:03:42.525 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:03:42.525 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:03:42.533 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:42.541 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:03:42.541 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:03:42.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:42.555 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:42.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:42.557 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:03:42.557 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:03:42.574 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:03:42.574 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:03:42.592 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:03:42.592 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:03:42.657 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:42.719 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:42.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:42.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:42.821 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:42.843 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:42.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:42.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:42.945 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:42.967 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:42.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:42.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:43.069 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:43.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:43.091 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:43.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:43.105 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:03:43.105 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:03:43.192 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:43.192 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 7] out_osds: [0, 5] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:03:43.192 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:03:43.193 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [1, 2, 3, 4, 6, 7] 2024-08-17T18:03:43.193 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd out 7 2024-08-17T18:03:43.214 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:43.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:43.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:43.316 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:43.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:43.338 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:43.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:43.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:43.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:43 smithi136 ceph-mon[109861]: pgmap v2535: 89 pgs: 89 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:43.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:43 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4149690505' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:03:43.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:43 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:03:43.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:43 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:03:43.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:43 smithi136 ceph-mon[115078]: pgmap v2535: 89 pgs: 89 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:43.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:43 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4149690505' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:03:43.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:43 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:03:43.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:43 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:03:43.460 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:43.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:43.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:43.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:43 smithi165 ceph-mon[105517]: pgmap v2535: 89 pgs: 89 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:43.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:43 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4149690505' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:03:43.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:43 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:03:43.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:43 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:03:43.562 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:43.583 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:43.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:43.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:43.685 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:43.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:43.742 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:43.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:43.843 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:43.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:43.866 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:43.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:43.967 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:43.987 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-17T18:03:43.990 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:43.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:43.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:44.091 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:44.113 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:44.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:44.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:44.215 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:44.242 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:44.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:44.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:44.344 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:44.365 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:44.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:44.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:44.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:44.490 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:44.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:44.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:44.592 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:44.613 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:44.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:44.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:44.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:44.741 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:44.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:44.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:44.843 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:44.870 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:44.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:44.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:44.939 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:03:44.940 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:03:44.941 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:03:44.942 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:03:44.972 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:44.998 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:44.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:44.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:45.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:45.125 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:45.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:45.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:45.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:45.255 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:45.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:45.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:45.357 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:45.362 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:45 smithi136 ceph-mon[109861]: pgmap v2536: 89 pgs: 89 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:45.363 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:45 smithi136 ceph-mon[115078]: pgmap v2536: 89 pgs: 89 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:45.382 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:45.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:45.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:45.484 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:45.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:45.507 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:45.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:45.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:45 smithi165 ceph-mon[105517]: pgmap v2536: 89 pgs: 89 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:45.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:45.634 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:45.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:45.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:45.736 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:45.761 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:45.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:45.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:45.863 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:45.886 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:45.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:45.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:45.988 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:46.010 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:46.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:46.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:46.111 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:46.138 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:46.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:46.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:46.239 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:46.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:46.263 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:46.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:46.364 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:46.386 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:03:46.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:46.387 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:46.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:46.489 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:46.515 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:46.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:46.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:46.617 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:46.640 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:46.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:46.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:46.742 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:46.764 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:46.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:46.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:46.866 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:46.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:46.889 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:46.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:46.990 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:47.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:47.012 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:47.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:47.113 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:47.135 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:47.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:47.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:47.237 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:47.259 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:47.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:47.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:47.361 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:47.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:47.384 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:47.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:47.485 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:47.507 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:47.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:47.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:47.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:47 smithi165 ceph-mon[105517]: pgmap v2537: 89 pgs: 89 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:47.609 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:47.635 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:47.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:47.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:47.686 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:47 smithi136 ceph-mon[109861]: pgmap v2537: 89 pgs: 89 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:47.686 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:47 smithi136 ceph-mon[115078]: pgmap v2537: 89 pgs: 89 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:47.686 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:03:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:03:47] "GET /metrics HTTP/1.1" 200 34201 "" "Prometheus/2.51.0" 2024-08-17T18:03:47.737 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:47.759 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:47.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:47.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:47.861 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:47.887 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:47.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:47.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:47.989 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:48.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:48.012 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:48.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:48.113 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:48.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:48.138 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:48.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:48.240 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:48.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:48.269 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:48.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:48.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:48.396 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:48.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:48.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:48.497 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:48.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:48.522 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:48.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:48.624 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:48.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:48.647 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:48.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:48.749 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:48.770 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:48.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:48.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:48.872 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:48.898 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:48.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:48.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:49.000 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:49.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:49.022 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:49.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:49.123 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:49.149 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:49.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:49.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:49.251 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:49.277 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:49.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:49.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:49.379 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:49.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:49.404 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:49.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:49.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:49 smithi136 ceph-mon[109861]: pgmap v2538: 89 pgs: 89 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:49.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:49 smithi136 ceph-mon[115078]: pgmap v2538: 89 pgs: 89 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:49.505 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:49.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:49 smithi165 ceph-mon[105517]: pgmap v2538: 89 pgs: 89 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:49.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:49.530 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:49.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:49.631 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:49.657 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:49.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:49.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:49.759 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:49.785 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:49.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:49.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:49.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:49.912 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:49.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:49.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:50.014 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:50.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:50.042 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:50.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:50.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:50.169 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:50.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:50.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:50.251 INFO:teuthology.orchestra.run.smithi136.stderr:marked out osd.7. 2024-08-17T18:03:50.251 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:03:50.255 INFO:teuthology.orchestra.run.smithi136.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-08-17T18:03:50.270 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:50.277 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:50 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3999838101' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:03:50.277 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:50 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3930696559' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T18:03:50.277 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:50 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T18:03:50.277 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:50 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2943263441' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-08-17T18:03:50.277 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:50 smithi136 ceph-mon[115078]: Client client.admin marked osd.7 out, while it was still marked up 2024-08-17T18:03:50.277 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:50 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a[109838]: 2024-08-17T18:03:50.235+0000 7f5b8e77d640 -1 mon.a@0(leader).osd e1448 definitely_dead 0 2024-08-17T18:03:50.278 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:50 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3999838101' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:03:50.278 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:50 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3930696559' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T18:03:50.278 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:50 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T18:03:50.278 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:50 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2943263441' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-08-17T18:03:50.278 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:50 smithi136 ceph-mon[109861]: Client client.admin marked osd.7 out, while it was still marked up 2024-08-17T18:03:50.294 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:50.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:50.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:50.395 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:50.419 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:50.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:50.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:50.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:50 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3999838101' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:03:50.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:50 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3930696559' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T18:03:50.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:50 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T18:03:50.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:50 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2943263441' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-08-17T18:03:50.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:50 smithi165 ceph-mon[105517]: Client client.admin marked osd.7 out, while it was still marked up 2024-08-17T18:03:50.521 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:50.546 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:50.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:50.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:50.648 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:50.670 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:50.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:50.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:50.772 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:50.793 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:50.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:50.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:50.895 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:50.917 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:50.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:50.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:51.019 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:51.045 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:51.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:51.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:51.147 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:51.172 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:51.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:51.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:51.274 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:51.299 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:51.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:51.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:51.392 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:51 smithi136 ceph-mon[109861]: pgmap v2539: 89 pgs: 89 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:51.392 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:51 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:03:51.392 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:51 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3999838101' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:03:51.392 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:51 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T18:03:51.392 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:51 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2943263441' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-08-17T18:03:51.392 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:51 smithi136 ceph-mon[109861]: osdmap e1449: 8 total, 8 up, 5 in 2024-08-17T18:03:51.392 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:03:51.392 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:03:51.393 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:51 smithi136 ceph-mon[115078]: pgmap v2539: 89 pgs: 89 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:51.393 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:51 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:03:51.393 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:51 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3999838101' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:03:51.393 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:51 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T18:03:51.393 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:51 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2943263441' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-08-17T18:03:51.393 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:51 smithi136 ceph-mon[115078]: osdmap e1449: 8 total, 8 up, 5 in 2024-08-17T18:03:51.393 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:03:51.393 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:03:51.401 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:51.422 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:51.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:51.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:51.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:51 smithi165 ceph-mon[105517]: pgmap v2539: 89 pgs: 89 active+clean; 582 KiB data, 2.7 GiB used, 534 GiB / 536 GiB avail 2024-08-17T18:03:51.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:51 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:03:51.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:51 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3999838101' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:03:51.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:51 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T18:03:51.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:51 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2943263441' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-08-17T18:03:51.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:51 smithi165 ceph-mon[105517]: osdmap e1449: 8 total, 8 up, 5 in 2024-08-17T18:03:51.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:03:51.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:03:51.524 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:51.545 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:51.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:51.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:51.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:51.668 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:51.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:51.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:51.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:51.792 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:51.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:51.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:51.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:51.916 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:51.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:51.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:52.017 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:52.039 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:52.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:52.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:52.141 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:52.162 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:52.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:52.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:52.264 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:52.286 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:52.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:52.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:52.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:52.410 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:52.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:52.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:52.412 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:52 smithi136 ceph-mon[109861]: osdmap e1450: 8 total, 8 up, 5 in 2024-08-17T18:03:52.413 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:52 smithi136 ceph-mon[115078]: osdmap e1450: 8 total, 8 up, 5 in 2024-08-17T18:03:52.418 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:03:52.512 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:52.533 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:52.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:52.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:52.635 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:52.657 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:52.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:52.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:52.758 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:52.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:52 smithi165 ceph-mon[105517]: osdmap e1450: 8 total, 8 up, 5 in 2024-08-17T18:03:52.780 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:52.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:52.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:52.882 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:52.904 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:52.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:52.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:53.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:53.027 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:53.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:53.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:53.129 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:53.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:53.154 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:53.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:53.255 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:53.278 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:53.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:53.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:53.380 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:53.402 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:53.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:53.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:53.504 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:53.526 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:53.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:53.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:53.628 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:53.649 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:53.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:53.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:53.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:53 smithi136 ceph-mon[115078]: pgmap v2542: 89 pgs: 1 remapped+peering, 88 active+clean; 582 KiB data, 2.3 GiB used, 445 GiB / 447 GiB avail; 19 B/s, 0 objects/s recovering 2024-08-17T18:03:53.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:53 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T18:03:53.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:53 smithi136 ceph-mon[115078]: osdmap e1451: 8 total, 8 up, 5 in 2024-08-17T18:03:53.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:53 smithi136 ceph-mon[109861]: pgmap v2542: 89 pgs: 1 remapped+peering, 88 active+clean; 582 KiB data, 2.3 GiB used, 445 GiB / 447 GiB avail; 19 B/s, 0 objects/s recovering 2024-08-17T18:03:53.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:53 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T18:03:53.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:53 smithi136 ceph-mon[109861]: osdmap e1451: 8 total, 8 up, 5 in 2024-08-17T18:03:53.712 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:03:53.751 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:53 smithi165 ceph-mon[105517]: pgmap v2542: 89 pgs: 1 remapped+peering, 88 active+clean; 582 KiB data, 2.3 GiB used, 445 GiB / 447 GiB avail; 19 B/s, 0 objects/s recovering 2024-08-17T18:03:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:53 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T18:03:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:53 smithi165 ceph-mon[105517]: osdmap e1451: 8 total, 8 up, 5 in 2024-08-17T18:03:53.776 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:53.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:53.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:53.878 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:53.901 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:53.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:53.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:54.002 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:54.024 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:54.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:54.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:54.126 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:54.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:54.150 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:54.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:54.251 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:54.275 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:54.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:54.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:54.309 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:03:54.309 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:03:54.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:54.398 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:54.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:54.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:54.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:54 smithi136 ceph-mon[109861]: osdmap e1452: 8 total, 8 up, 5 in 2024-08-17T18:03:54.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:54 smithi136 ceph-mon[115078]: osdmap e1452: 8 total, 8 up, 5 in 2024-08-17T18:03:54.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:54.523 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:54.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:54.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:54.625 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:54.648 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:54.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:54.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:54.750 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:54.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:54 smithi165 ceph-mon[105517]: osdmap e1452: 8 total, 8 up, 5 in 2024-08-17T18:03:54.773 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:54.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:54.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:54.875 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:54.896 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:54.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:54.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:54.999 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:55.020 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:55.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:55.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:55.099 DEBUG:teuthology.orchestra.run.smithi136:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-08-17T18:03:55.102 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-17T18:03:55.102 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-17T18:03:55.123 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:55.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:55.144 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:55.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:55.159 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:hints = 1 2024-08-17T18:03:55.159 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-17T18:03:55.159 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Object prefix: benchmark_data_smithi136_313476 2024-08-17T18:03:55.159 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T18:03:55.159 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 0 0 0 0 0 0 - 0 2024-08-17T18:03:55.245 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:55.285 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:55.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:55.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:55.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:55.416 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:55.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:55.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:55.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:55.546 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:55.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:55.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:55.613 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:55 smithi136 ceph-mon[109861]: pgmap v2545: 89 pgs: 1 remapped+peering, 88 active+clean; 582 KiB data, 2.3 GiB used, 445 GiB / 447 GiB avail; 1.2 KiB/s, 2 keys/s, 3 objects/s recovering 2024-08-17T18:03:55.613 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:55 smithi136 ceph-mon[109861]: osdmap e1453: 8 total, 8 up, 5 in 2024-08-17T18:03:55.613 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:55 smithi136 ceph-mon[115078]: pgmap v2545: 89 pgs: 1 remapped+peering, 88 active+clean; 582 KiB data, 2.3 GiB used, 445 GiB / 447 GiB avail; 1.2 KiB/s, 2 keys/s, 3 objects/s recovering 2024-08-17T18:03:55.613 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:55 smithi136 ceph-mon[115078]: osdmap e1453: 8 total, 8 up, 5 in 2024-08-17T18:03:55.620 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:03:55.648 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:55.661 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-17T18:03:55.694 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:55.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:55.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:55 smithi165 ceph-mon[105517]: pgmap v2545: 89 pgs: 1 remapped+peering, 88 active+clean; 582 KiB data, 2.3 GiB used, 445 GiB / 447 GiB avail; 1.2 KiB/s, 2 keys/s, 3 objects/s recovering 2024-08-17T18:03:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:55 smithi165 ceph-mon[105517]: osdmap e1453: 8 total, 8 up, 5 in 2024-08-17T18:03:55.796 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:55.823 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:55.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:55.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:55.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:55.949 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:55.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:55.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:56.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:56.079 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:56.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:56.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:56.125 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6] out_osds: [0, 5, 7] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:03:56.125 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:03:56.125 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-08-17T18:03:56.126 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd in 0 2024-08-17T18:03:56.159 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 1 16 3293 3277 204.8 204.812 0.00771805 0.00486597 2024-08-17T18:03:56.181 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:56.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:56.230 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:56.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:56.331 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:56.356 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:56.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:56.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:56.458 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:56.497 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:56.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:56.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:56.522 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:56 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4027580409' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:03:56.522 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:56 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4027580409' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:03:56.600 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:56.635 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:56.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:56.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:56.636 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:03:56.637 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:03:56.637 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:03:56.637 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:03:56.637 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:03:56.637 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:03:56.637 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:03:56.637 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:03:56.637 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:03:56.637 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:03:56.637 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:03:56.637 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:03:56.638 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:03:56.638 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:03:56.638 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:03:56.638 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:03:56.737 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:56 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4027580409' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:03:56.774 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:56.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:56.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:56.876 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:56.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:56.918 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:56.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:57.019 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:57.047 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:57.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:57.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:57.149 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:57.159 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 2 16 6320 6304 196.984 189.188 0.009503 0.00506807 2024-08-17T18:03:57.187 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:57.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:57.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:57.289 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:57.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:57.318 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:57.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:57.419 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:03:57.460 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:57.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:57.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:03:57.562 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:57.590 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:57.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:57.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:57.599 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:57 smithi136 ceph-mon[109861]: pgmap v2547: 89 pgs: 1 remapped+peering, 88 active+clean; 582 KiB data, 2.3 GiB used, 445 GiB / 447 GiB avail; 1000 B/s, 2 keys/s, 2 objects/s recovering 2024-08-17T18:03:57.599 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:03:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:03:57] "GET /metrics HTTP/1.1" 200 36547 "" "Prometheus/2.51.0" 2024-08-17T18:03:57.599 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:57 smithi136 ceph-mon[115078]: pgmap v2547: 89 pgs: 1 remapped+peering, 88 active+clean; 582 KiB data, 2.3 GiB used, 445 GiB / 447 GiB avail; 1000 B/s, 2 keys/s, 2 objects/s recovering 2024-08-17T18:03:57.692 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:57.731 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:57.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:57.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:57 smithi165 ceph-mon[105517]: pgmap v2547: 89 pgs: 1 remapped+peering, 88 active+clean; 582 KiB data, 2.3 GiB used, 445 GiB / 447 GiB avail; 1000 B/s, 2 keys/s, 2 objects/s recovering 2024-08-17T18:03:57.833 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:57.871 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:57.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:57.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:57.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:58.000 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:58.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:58.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:58.102 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:58.140 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:58.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:58.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:58.162 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 3 16 9386 9370 195.192 191.625 0.00841246 0.00511347 2024-08-17T18:03:58.242 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:58.267 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:58.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:58.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:58.369 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:03:58.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:58.395 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:58.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:03:58.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:03:58.521 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:58.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:58.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:03:58.623 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:58.662 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:58.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:58.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:58.764 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:03:58.790 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:58.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:58.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:03:58.892 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:58.929 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:58.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:58.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:59.031 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:03:59.070 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:59.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:59.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:03:59.160 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 4 16 11946 11930 186.39 160 0.00156047 0.0053414 2024-08-17T18:03:59.171 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:59.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:59.211 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:59.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:59.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:59.341 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:03:59.349 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:59.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:59.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:59.451 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:59.479 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:59.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:59.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:59.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:03:59.621 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:59.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:59.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:03:59.648 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:59 smithi136 ceph-mon[109861]: pgmap v2548: 89 pgs: 89 active+clean; 299 MiB data, 2.9 GiB used, 444 GiB / 447 GiB avail; 50 MiB/s wr, 794 op/s; 97 KiB/s, 1 keys/s, 1 objects/s recovering 2024-08-17T18:03:59.649 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:59 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/750820803' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:03:59.649 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:03:59 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:03:59.649 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:59 smithi136 ceph-mon[115078]: pgmap v2548: 89 pgs: 89 active+clean; 299 MiB data, 2.9 GiB used, 444 GiB / 447 GiB avail; 50 MiB/s wr, 794 op/s; 97 KiB/s, 1 keys/s, 1 objects/s recovering 2024-08-17T18:03:59.649 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:59 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/750820803' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:03:59.649 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:03:59 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:03:59.724 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:03:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:59 smithi165 ceph-mon[105517]: pgmap v2548: 89 pgs: 89 active+clean; 299 MiB data, 2.9 GiB used, 444 GiB / 447 GiB avail; 50 MiB/s wr, 794 op/s; 97 KiB/s, 1 keys/s, 1 objects/s recovering 2024-08-17T18:03:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:59 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/750820803' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:03:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:03:59 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:03:59.763 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:03:59.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:59.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:03:59.866 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:03:59.890 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:03:59.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:03:59.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:03:59.927 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:03:59.992 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:00.038 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:00.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:00.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:00.140 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:00.160 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 5 16 14443 14427 180.302 156.062 0.0103219 0.00553848 2024-08-17T18:04:00.188 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:00.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:00.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:00.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:00.318 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:00.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:00.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:00.420 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:00.445 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:00.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:00.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:00.547 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:00.576 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:00.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:00.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:00.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:00.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:00 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:04:00.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:00 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:04:00.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:00 smithi136 ceph-mon[109861]: osdmap e1454: 8 total, 8 up, 5 in 2024-08-17T18:04:00.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:00 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:04:00.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:00 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:04:00.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:00 smithi136 ceph-mon[115078]: osdmap e1454: 8 total, 8 up, 5 in 2024-08-17T18:04:00.704 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:00.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:00.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:00.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:00 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:04:00.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:00 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:04:00.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:00 smithi165 ceph-mon[105517]: osdmap e1454: 8 total, 8 up, 5 in 2024-08-17T18:04:00.806 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:00.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:00.828 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:00.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:00.930 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:00.966 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:00.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:00.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:01.068 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:01.103 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:01.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:01.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:01.160 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 6 16 16662 16646 173.365 138.688 0.0037992 0.00576236 2024-08-17T18:04:01.206 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:01.245 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:01.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:01.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:01.347 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:01.384 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:01.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:01.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:01.486 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:01.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:01.523 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:01.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:01.624 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:01.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:01.656 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:01.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:01 smithi136 ceph-mon[109861]: pgmap v2550: 89 pgs: 89 active+clean; 299 MiB data, 2.9 GiB used, 444 GiB / 447 GiB avail; 43 MiB/s wr, 687 op/s; 83 KiB/s, 0 objects/s recovering 2024-08-17T18:04:01.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:01 smithi136 ceph-mon[115078]: pgmap v2550: 89 pgs: 89 active+clean; 299 MiB data, 2.9 GiB used, 444 GiB / 447 GiB avail; 43 MiB/s wr, 687 op/s; 83 KiB/s, 0 objects/s recovering 2024-08-17T18:04:01.757 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:01 smithi165 ceph-mon[105517]: pgmap v2550: 89 pgs: 89 active+clean; 299 MiB data, 2.9 GiB used, 444 GiB / 447 GiB avail; 43 MiB/s wr, 687 op/s; 83 KiB/s, 0 objects/s recovering 2024-08-17T18:04:01.783 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:01.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:01.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:01.885 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:01.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:01.907 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:01.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:02.008 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:02.034 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:02.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:02.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:02.136 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:02.160 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 7 16 18372 18356 163.866 106.875 0.00239152 0.00609617 2024-08-17T18:04:02.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:02.166 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:02.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:02.267 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:02.293 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:02.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:02.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:02.395 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:02.421 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:02.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:02.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:02.523 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:02.552 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:02.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:02.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:02.654 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:02.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:02.684 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:02.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:02.786 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:02.822 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:02.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:02.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:02.924 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:02.948 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:02.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:02.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:03.050 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:03.078 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:03.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:03.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:03.160 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 8 16 20485 20469 159.889 132.062 0.0050778 0.00624684 2024-08-17T18:04:03.180 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:03.209 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:03.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:03.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:03.224 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:04:03.273 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:04:03.311 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:03.371 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:03.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:03.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:03.473 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:03.517 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:03.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:03.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:03.540 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:03 smithi136 ceph-mon[109861]: pgmap v2551: 89 pgs: 89 active+clean; 1.0 GiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 133 MiB/s wr, 2.13k op/s; 72 KiB/s, 0 objects/s recovering 2024-08-17T18:04:03.540 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:03 smithi136 ceph-mon[115078]: pgmap v2551: 89 pgs: 89 active+clean; 1.0 GiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 133 MiB/s wr, 2.13k op/s; 72 KiB/s, 0 objects/s recovering 2024-08-17T18:04:03.619 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:03.658 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:03.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:03.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:03.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:03 smithi165 ceph-mon[105517]: pgmap v2551: 89 pgs: 89 active+clean; 1.0 GiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 133 MiB/s wr, 2.13k op/s; 72 KiB/s, 0 objects/s recovering 2024-08-17T18:04:03.784 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:03.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:03.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:03.886 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:03.923 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:03.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:03.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:04.025 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:04.051 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:04.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:04.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:04.153 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:04.160 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 9 16 22633 22617 157.039 134.25 0.00390335 0.00636121 2024-08-17T18:04:04.179 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:04.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:04.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:04.281 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:04.309 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:04.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:04.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:04.411 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:04.454 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:04.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:04.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:04.557 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:04.593 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:04.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:04.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:04.665 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:04:04.695 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:04.750 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:04.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:04.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:04.851 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:04.877 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:04.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:04.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:04.979 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:05.004 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:05.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:05.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:05.106 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:05.132 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:05.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:05.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:05.161 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 10 16 24766 24750 154.665 133.312 0.00567278 0.00646004 2024-08-17T18:04:05.234 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:05.259 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:04:05.261 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:05.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:05.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:05.363 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:05.409 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:05.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:05.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:05.512 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:05.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:05.554 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:05.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:05.655 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:05.694 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:05.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:05.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:05 smithi136 ceph-mon[109861]: pgmap v2552: 89 pgs: 89 active+clean; 1.1 GiB data, 5.4 GiB used, 442 GiB / 447 GiB avail; 112 MiB/s wr, 1.78k op/s; 58 KiB/s, 0 objects/s recovering 2024-08-17T18:04:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:05 smithi136 ceph-mon[115078]: pgmap v2552: 89 pgs: 89 active+clean; 1.1 GiB data, 5.4 GiB used, 442 GiB / 447 GiB avail; 112 MiB/s wr, 1.78k op/s; 58 KiB/s, 0 objects/s recovering 2024-08-17T18:04:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:05 smithi165 ceph-mon[105517]: pgmap v2552: 89 pgs: 89 active+clean; 1.1 GiB data, 5.4 GiB used, 442 GiB / 447 GiB avail; 112 MiB/s wr, 1.78k op/s; 58 KiB/s, 0 objects/s recovering 2024-08-17T18:04:05.795 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:05.816 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:05.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:05.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:05.918 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:05.955 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:05.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:05.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:06.057 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:06.077 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:06.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:06.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:06.161 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 11 16 26360 26344 149.661 99.625 0.00696396 0.0066759 2024-08-17T18:04:06.179 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:06.206 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:06.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:06.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:06.308 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:06.335 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:06.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:06.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:06.437 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:06.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:06.473 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:06.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:06.574 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:06.600 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:06.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:06.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:06.701 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:06.739 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:06.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:06.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:06.841 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:06.865 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:06.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:06.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:06.967 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:07.004 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:07.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:07.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:07.106 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:07.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:07.132 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:07.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:07.161 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 12 16 28227 28211 146.912 116.688 0.0105152 0.00678162 2024-08-17T18:04:07.232 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:07.259 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:07.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:07.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:07.361 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:07.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:07.388 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:07.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:07.489 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:07 smithi165 ceph-mon[105517]: pgmap v2553: 89 pgs: 89 active+clean; 1.1 GiB data, 5.4 GiB used, 442 GiB / 447 GiB avail; 111 MiB/s wr, 1.77k op/s; 58 KiB/s, 0 objects/s recovering 2024-08-17T18:04:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:04:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:04:07.515 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:07.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:07.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:07.616 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:07.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:07.644 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:07.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:07.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:07 smithi136 ceph-mon[109861]: pgmap v2553: 89 pgs: 89 active+clean; 1.1 GiB data, 5.4 GiB used, 442 GiB / 447 GiB avail; 111 MiB/s wr, 1.77k op/s; 58 KiB/s, 0 objects/s recovering 2024-08-17T18:04:07.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:04:07.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:04:07.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:04:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:04:07] "GET /metrics HTTP/1.1" 200 36618 "" "Prometheus/2.51.0" 2024-08-17T18:04:07.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:07 smithi136 ceph-mon[115078]: pgmap v2553: 89 pgs: 89 active+clean; 1.1 GiB data, 5.4 GiB used, 442 GiB / 447 GiB avail; 111 MiB/s wr, 1.77k op/s; 58 KiB/s, 0 objects/s recovering 2024-08-17T18:04:07.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:04:07.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:04:07.744 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:07.782 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:07.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:07.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:07.884 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:07.905 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:07.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:07.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:08.007 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:08.027 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:08.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:08.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:08.129 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:08.158 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:08.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:08.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:08.161 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 13 15 29592 29577 142.178 85.375 0.00306452 0.00702708 2024-08-17T18:04:08.260 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:08.297 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:08.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:08.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:08.399 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:08.424 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:08.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:08.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:08.526 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:08.567 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:08.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:08.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:08.669 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:08.694 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:08.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:08.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:08.796 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:08.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:08.833 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:08.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:08.934 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:08.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:08.983 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:08.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:09.084 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:09.124 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:09.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:09.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:09.161 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 14 16 31357 31341 139.897 110.25 0.00734098 0.00714236 2024-08-17T18:04:09.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:09.259 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:09.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:09.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:09 smithi165 ceph-mon[105517]: pgmap v2554: 89 pgs: 89 active+clean; 1.7 GiB data, 7.2 GiB used, 440 GiB / 447 GiB avail; 141 MiB/s wr, 2.25k op/s 2024-08-17T18:04:09.361 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:09.399 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:09.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:09.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:09.501 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:09.528 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:09.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:09.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:09.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:09.654 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:09.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:09.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:09 smithi136 ceph-mon[109861]: pgmap v2554: 89 pgs: 89 active+clean; 1.7 GiB data, 7.2 GiB used, 440 GiB / 447 GiB avail; 141 MiB/s wr, 2.25k op/s 2024-08-17T18:04:09.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:09 smithi136 ceph-mon[115078]: pgmap v2554: 89 pgs: 89 active+clean; 1.7 GiB data, 7.2 GiB used, 440 GiB / 447 GiB avail; 141 MiB/s wr, 2.25k op/s 2024-08-17T18:04:09.756 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:09.783 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:09.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:09.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:09.884 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:09.920 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:09.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:09.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:10.022 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:10.049 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:10.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:10.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:10.151 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:10.161 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 15 16 32977 32961 137.32 101.25 0.0103329 0.00727648 2024-08-17T18:04:10.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:10.189 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:10.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:10.290 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:10.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:10.318 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:10.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:10.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:10.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]} : dispatch 2024-08-17T18:04:10.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [1, 6]} : dispatch 2024-08-17T18:04:10.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 2]} : dispatch 2024-08-17T18:04:10.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]} : dispatch 2024-08-17T18:04:10.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [1, 6]} : dispatch 2024-08-17T18:04:10.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 2]} : dispatch 2024-08-17T18:04:10.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:10.448 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:10.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:10.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]} : dispatch 2024-08-17T18:04:10.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [1, 6]} : dispatch 2024-08-17T18:04:10.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 2]} : dispatch 2024-08-17T18:04:10.550 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:10.587 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:10.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:10.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:10.689 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:10.716 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:10.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:10.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:10.818 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:10.854 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:10.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:10.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:10.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:10.982 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:11.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:11.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:11.128 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:11.162 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 16 16 34692 34676 135.431 107.188 0.0122672 0.0073773 2024-08-17T18:04:11.163 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:11.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:11.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:11.265 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:11.287 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:11.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:11.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:11.389 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:11.412 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:11.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:11.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:11 smithi165 ceph-mon[105517]: pgmap v2555: 89 pgs: 89 active+clean; 1.7 GiB data, 7.2 GiB used, 440 GiB / 447 GiB avail; 130 MiB/s wr, 2.07k op/s 2024-08-17T18:04:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:11 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]': finished 2024-08-17T18:04:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:11 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [1, 6]}]': finished 2024-08-17T18:04:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:11 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 2]}]': finished 2024-08-17T18:04:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:11 smithi165 ceph-mon[105517]: osdmap e1455: 8 total, 8 up, 5 in 2024-08-17T18:04:11.515 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:11.550 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:11.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:11.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:11.652 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:11.674 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:11.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:11.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:11.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:11 smithi136 ceph-mon[115078]: pgmap v2555: 89 pgs: 89 active+clean; 1.7 GiB data, 7.2 GiB used, 440 GiB / 447 GiB avail; 130 MiB/s wr, 2.07k op/s 2024-08-17T18:04:11.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:11 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]': finished 2024-08-17T18:04:11.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:11 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [1, 6]}]': finished 2024-08-17T18:04:11.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:11 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 2]}]': finished 2024-08-17T18:04:11.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:11 smithi136 ceph-mon[115078]: osdmap e1455: 8 total, 8 up, 5 in 2024-08-17T18:04:11.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:11 smithi136 ceph-mon[109861]: pgmap v2555: 89 pgs: 89 active+clean; 1.7 GiB data, 7.2 GiB used, 440 GiB / 447 GiB avail; 130 MiB/s wr, 2.07k op/s 2024-08-17T18:04:11.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]': finished 2024-08-17T18:04:11.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [1, 6]}]': finished 2024-08-17T18:04:11.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 2]}]': finished 2024-08-17T18:04:11.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:11 smithi136 ceph-mon[109861]: osdmap e1455: 8 total, 8 up, 5 in 2024-08-17T18:04:11.776 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:11.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:11.811 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:11.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:11.912 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:11.952 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:11.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:11.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:12.054 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:12.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:12.096 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:12.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:12.162 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 17 16 36227 36211 133.107 95.9375 0.0159906 0.00750697 2024-08-17T18:04:12.198 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:12.224 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:12.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:12.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:12.326 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:12.351 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:12.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:12.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:12.453 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:12.478 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:12.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:12.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:12.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:12 smithi136 ceph-mon[109861]: osdmap e1456: 8 total, 8 up, 5 in 2024-08-17T18:04:12.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:12 smithi136 ceph-mon[115078]: osdmap e1456: 8 total, 8 up, 5 in 2024-08-17T18:04:12.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:12.618 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:12.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:12.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:12.720 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:12.745 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:12.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:12.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:12.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:12 smithi165 ceph-mon[105517]: osdmap e1456: 8 total, 8 up, 5 in 2024-08-17T18:04:12.847 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:12.874 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:12.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:12.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:12.976 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:13.001 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:13.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:13.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:13.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:13.129 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:13.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:13.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:13.162 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 18 16 37906 37890 131.542 104.938 0.010821 0.00759599 2024-08-17T18:04:13.231 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:13.256 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:13.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:13.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:13.358 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:13.383 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:13.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:13.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:13.485 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:13.528 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:13.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:13.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:13.629 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:13.668 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:13.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:13.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:13.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:13 smithi136 ceph-mon[109861]: pgmap v2558: 89 pgs: 89 active+clean; 2.1 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 134 MiB/s wr, 2.15k op/s 2024-08-17T18:04:13.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:13 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a[109838]: 2024-08-17T18:04:13.473+0000 7f5b8e77d640 -1 mon.a@0(leader).osd e1456 definitely_dead 0 2024-08-17T18:04:13.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:13 smithi136 ceph-mon[115078]: pgmap v2558: 89 pgs: 89 active+clean; 2.1 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 134 MiB/s wr, 2.15k op/s 2024-08-17T18:04:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:13 smithi165 ceph-mon[105517]: pgmap v2558: 89 pgs: 89 active+clean; 2.1 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 134 MiB/s wr, 2.15k op/s 2024-08-17T18:04:13.769 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:13.795 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:13.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:13.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:13.897 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:13.947 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:13.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:13.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:14.048 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:14.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:14.099 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:14.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:14.162 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 19 16 39519 39503 129.924 100.812 0.0049301 0.0076912 2024-08-17T18:04:14.200 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:14.227 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:14.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:14.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:14.329 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:14.337 INFO:teuthology.orchestra.run.smithi136.stderr:marked in osd.0. 2024-08-17T18:04:14.354 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:14.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:14.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:14.457 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:14.479 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:14.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:14.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:14.582 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:14.604 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:14.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:14.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:14.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:14 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2560326725' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-08-17T18:04:14.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:14 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-08-17T18:04:14.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:14 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2560326725' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-08-17T18:04:14.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:14 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-08-17T18:04:14.705 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:14.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:14.729 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:14.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:14 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2560326725' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-08-17T18:04:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:14 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-08-17T18:04:14.829 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:14.853 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:14.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:14.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:14.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:14.979 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:14.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:14.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:15.081 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:15.107 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:15.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:15.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:15.162 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:2024-08-17T18:04:15.162682+0000 min lat: 0.00131667 max lat: 0.166925 avg lat: 0.00773281 2024-08-17T18:04:15.163 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T18:04:15.163 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 20 16 39707 39691 124.016 11.75 0.00248614 0.00773281 2024-08-17T18:04:15.209 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:15.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:15.233 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:15.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:15.334 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:15.356 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:15.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:15.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:15.410 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:15 smithi136 ceph-mon[109861]: pgmap v2559: 89 pgs: 89 active+clean; 2.2 GiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-08-17T18:04:15.410 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:15 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-08-17T18:04:15.410 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:15 smithi136 ceph-mon[109861]: osdmap e1457: 8 total, 8 up, 6 in 2024-08-17T18:04:15.410 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:15 smithi136 ceph-mon[109861]: osdmap e1458: 8 total, 8 up, 6 in 2024-08-17T18:04:15.410 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:15 smithi136 ceph-mon[115078]: pgmap v2559: 89 pgs: 89 active+clean; 2.2 GiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-08-17T18:04:15.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:15 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-08-17T18:04:15.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:15 smithi136 ceph-mon[115078]: osdmap e1457: 8 total, 8 up, 6 in 2024-08-17T18:04:15.411 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:15 smithi136 ceph-mon[115078]: osdmap e1458: 8 total, 8 up, 6 in 2024-08-17T18:04:15.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:15.480 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:15.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:15.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:15.582 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:15.604 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:15.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:15.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:15.705 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:15.727 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:15.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:15.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:15 smithi165 ceph-mon[105517]: pgmap v2559: 89 pgs: 89 active+clean; 2.2 GiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-08-17T18:04:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:15 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-08-17T18:04:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:15 smithi165 ceph-mon[105517]: osdmap e1457: 8 total, 8 up, 6 in 2024-08-17T18:04:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:15 smithi165 ceph-mon[105517]: osdmap e1458: 8 total, 8 up, 6 in 2024-08-17T18:04:15.829 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:15.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:15.852 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:15.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:15.953 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:15.961 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:04:15.962 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:04:15.988 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:15.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:15.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:15.995 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:04:15.995 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:04:16.029 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:04:16.029 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:04:16.069 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:04:16.069 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:04:16.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:16.099 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:04:16.099 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:04:16.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:16.117 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:16.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:16.131 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:04:16.131 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:04:16.162 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:04:16.162 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:04:16.162 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 21 16 39916 39900 118.732 13.0625 0.0205118 0.0084169 2024-08-17T18:04:16.194 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:04:16.195 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:04:16.218 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:16.243 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:16.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:16.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:16.346 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:16.370 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:16.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:16.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:16.471 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:16.524 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:16.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:16.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:16.626 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:16.650 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:16.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:16.754 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:16.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:16 smithi136 ceph-mon[109861]: osdmap e1459: 8 total, 8 up, 6 in 2024-08-17T18:04:16.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:16 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/127731113' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:04:16.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:16 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:04:16.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:16 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:04:16.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:16 smithi136 ceph-mon[115078]: osdmap e1459: 8 total, 8 up, 6 in 2024-08-17T18:04:16.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:16 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/127731113' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:04:16.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:16 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:04:16.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:16 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:04:16.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:16.799 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:16.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:16.899 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:16.922 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:16.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:16.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:16 smithi165 ceph-mon[105517]: osdmap e1459: 8 total, 8 up, 6 in 2024-08-17T18:04:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:16 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/127731113' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:04:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:16 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:04:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:16 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:04:17.023 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:17.049 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:17.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:17.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:17.151 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:17.162 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 22 16 41001 40985 116.417 67.8125 0.0082501 0.00858248 2024-08-17T18:04:17.173 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:17.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:17.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:17.209 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-08-17T18:04:17.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:17.300 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:17.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:17.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:17.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:17.426 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:17.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:17.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:17.467 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:04:17.528 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:17.554 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:17.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:17.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:17.656 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:17.679 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:17.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:17.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:17.734 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:17 smithi136 ceph-mon[109861]: pgmap v2563: 89 pgs: 89 active+clean; 2.2 GiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-08-17T18:04:17.734 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:17 smithi136 ceph-mon[109861]: osdmap e1460: 8 total, 8 up, 6 in 2024-08-17T18:04:17.735 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:04:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:04:17] "GET /metrics HTTP/1.1" 200 36618 "" "Prometheus/2.51.0" 2024-08-17T18:04:17.735 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:17 smithi136 ceph-mon[115078]: pgmap v2563: 89 pgs: 89 active+clean; 2.2 GiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-08-17T18:04:17.735 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:17 smithi136 ceph-mon[115078]: osdmap e1460: 8 total, 8 up, 6 in 2024-08-17T18:04:17.781 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:17.803 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:17.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:17.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:17.860 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:04:17.860 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:04:17.905 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:17.946 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:17.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:17.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:17 smithi165 ceph-mon[105517]: pgmap v2563: 89 pgs: 89 active+clean; 2.2 GiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-08-17T18:04:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:17 smithi165 ceph-mon[105517]: osdmap e1460: 8 total, 8 up, 6 in 2024-08-17T18:04:18.047 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:18.071 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:18.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:18.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:18.163 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 23 15 42062 42047 114.241 66.375 0.0152681 0.00874629 2024-08-17T18:04:18.172 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:18.195 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:18.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:18.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:18.297 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:18.320 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:18.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:18.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:18.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:18.447 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:18.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:18.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:18.549 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:18.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:18.596 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:18.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:18.697 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:18.720 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:18.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:18.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:18.814 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:18 smithi136 ceph-mon[109861]: osdmap e1461: 8 total, 8 up, 6 in 2024-08-17T18:04:18.814 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:04:18.814 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:04:18.814 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:04:18.814 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:18 smithi136 ceph-mon[115078]: osdmap e1461: 8 total, 8 up, 6 in 2024-08-17T18:04:18.815 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:04:18.815 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:04:18.815 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:04:18.822 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:18.846 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:18.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:18.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:18.948 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:18.991 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:18.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:18.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:18 smithi165 ceph-mon[105517]: osdmap e1461: 8 total, 8 up, 6 in 2024-08-17T18:04:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:04:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:04:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:04:19.093 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:19.139 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:19.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:19.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:19.163 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 24 16 43092 43076 112.161 64.3125 0.0290252 0.00890816 2024-08-17T18:04:19.241 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:19.265 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:19.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:19.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:19.367 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:19.390 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:19.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:19.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:19.492 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:19.533 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:19.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:19.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:19.635 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:19.691 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:19.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:19.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:19 smithi165 ceph-mon[105517]: pgmap v2566: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.5 GiB data, 9.9 GiB used, 526 GiB / 536 GiB avail; 81 MiB/s wr, 1.29k op/s; 12491/121380 objects degraded (10.291%); 2371/121380 objects misplaced (1.953%); 972 KiB/s, 20 objects/s recovering 2024-08-17T18:04:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:19 smithi165 ceph-mon[105517]: Health check failed: Degraded data redundancy: 12491/121380 objects degraded (10.291%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:04:19.792 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:19.816 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:19.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:19.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:19.918 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:19.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:19 smithi136 ceph-mon[109861]: pgmap v2566: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.5 GiB data, 9.9 GiB used, 526 GiB / 536 GiB avail; 81 MiB/s wr, 1.29k op/s; 12491/121380 objects degraded (10.291%); 2371/121380 objects misplaced (1.953%); 972 KiB/s, 20 objects/s recovering 2024-08-17T18:04:19.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:19 smithi136 ceph-mon[109861]: Health check failed: Degraded data redundancy: 12491/121380 objects degraded (10.291%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:04:19.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:19 smithi136 ceph-mon[115078]: pgmap v2566: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.5 GiB data, 9.9 GiB used, 526 GiB / 536 GiB avail; 81 MiB/s wr, 1.29k op/s; 12491/121380 objects degraded (10.291%); 2371/121380 objects misplaced (1.953%); 972 KiB/s, 20 objects/s recovering 2024-08-17T18:04:19.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:19 smithi136 ceph-mon[115078]: Health check failed: Degraded data redundancy: 12491/121380 objects degraded (10.291%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:04:19.960 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:19.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:19.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:20.062 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:20.093 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-17T18:04:20.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:20.099 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:20.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:20.163 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 25 16 44131 44115 110.272 64.9375 0.0205234 0.00906256 2024-08-17T18:04:20.200 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:20.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:20.252 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:20.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:20.353 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:20.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:20.402 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:20.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:20.503 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:20.547 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:20.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:20.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:20.649 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:20.673 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:20.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:20.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:20.749 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:04:20.775 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:20.839 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:20.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:20.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:20.941 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:20.966 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:20.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:20.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:21.067 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:04:21.067 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:21.089 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:21.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:21.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:21.163 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 26 16 45005 44989 108.132 54.625 0.0491525 0.00923752 2024-08-17T18:04:21.191 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:21.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:21.242 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:21.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:21.343 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:21.387 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:21.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:21.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:21.489 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:21.513 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:21.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:21.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:21.561 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:04:21.615 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:21.660 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:21.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:21.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:21.762 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:21.788 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:21.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:21.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:21.890 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:21.926 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:21.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:21.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:21.940 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:21 smithi136 ceph-mon[109861]: pgmap v2567: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.5 GiB data, 9.9 GiB used, 526 GiB / 536 GiB avail; 55 MiB/s wr, 878 op/s; 12491/121380 objects degraded (10.291%); 2371/121380 objects misplaced (1.953%); 663 KiB/s, 14 objects/s recovering 2024-08-17T18:04:21.940 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:21 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1986021774' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:04:21.940 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:21 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:04:21.940 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:04:21.940 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:04:21.941 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:21 smithi136 ceph-mon[115078]: pgmap v2567: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.5 GiB data, 9.9 GiB used, 526 GiB / 536 GiB avail; 55 MiB/s wr, 878 op/s; 12491/121380 objects degraded (10.291%); 2371/121380 objects misplaced (1.953%); 663 KiB/s, 14 objects/s recovering 2024-08-17T18:04:21.941 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:21 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1986021774' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:04:21.941 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:21 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:04:21.941 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:04:21.941 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:04:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:21 smithi165 ceph-mon[105517]: pgmap v2567: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.5 GiB data, 9.9 GiB used, 526 GiB / 536 GiB avail; 55 MiB/s wr, 878 op/s; 12491/121380 objects degraded (10.291%); 2371/121380 objects misplaced (1.953%); 663 KiB/s, 14 objects/s recovering 2024-08-17T18:04:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:21 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1986021774' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:04:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:21 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:04:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:04:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:04:22.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:22.054 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:22.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:22.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:22.156 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:22.163 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 27 16 46009 45993 106.451 62.75 0.0111807 0.00938645 2024-08-17T18:04:22.179 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:22.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:22.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:22.210 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 0] out_osds: [5, 7] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:04:22.210 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:04:22.211 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-08-17T18:04:22.211 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-17T18:04:22.211 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-17T18:04:22.211 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-08-17T18:04:22.281 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:22.305 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:22.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:22.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:22.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:22.470 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:22.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:22.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:22.572 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:22.596 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:22.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:22.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:22.698 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:22.736 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:22.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:22.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:22.744 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:04:22.838 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:22.854 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:22 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:04:22.854 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:22 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:04:22.854 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:22 smithi136 ceph-mon[115078]: osdmap e1462: 8 total, 8 up, 6 in 2024-08-17T18:04:22.857 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:22 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:04:22.857 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:22 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:04:22.857 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:22 smithi136 ceph-mon[109861]: osdmap e1462: 8 total, 8 up, 6 in 2024-08-17T18:04:22.862 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:22.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:22.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:22.871 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:22 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:04:22.871 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:22 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:04:22.871 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:22 smithi165 ceph-mon[105517]: osdmap e1462: 8 total, 8 up, 6 in 2024-08-17T18:04:22.964 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:23.024 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:23.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:23.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:23.125 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:23.163 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 28 16 47048 47032 104.967 64.9375 0.0397863 0.00951947 2024-08-17T18:04:23.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:23.172 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:23.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:23.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:23.297 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:23.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:23.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:23.399 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:23.446 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:23.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:23.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:23.548 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:23.551 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:04:23.591 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:23.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:23.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:23.605 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:04:23.605 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:04:23.693 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:23.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:23 smithi136 ceph-mon[109861]: pgmap v2569: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 102 MiB/s wr, 1.62k op/s; 12491/135615 objects degraded (9.211%); 2017/135615 objects misplaced (1.487%); 4.3 MiB/s, 72 objects/s recovering 2024-08-17T18:04:23.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:23 smithi136 ceph-mon[115078]: pgmap v2569: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 102 MiB/s wr, 1.62k op/s; 12491/135615 objects degraded (9.211%); 2017/135615 objects misplaced (1.487%); 4.3 MiB/s, 72 objects/s recovering 2024-08-17T18:04:23.722 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:23.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:23.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:23.823 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:23.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:23.865 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:23.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:23.966 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:23.990 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:23.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:23.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:23 smithi165 ceph-mon[105517]: pgmap v2569: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 102 MiB/s wr, 1.62k op/s; 12491/135615 objects degraded (9.211%); 2017/135615 objects misplaced (1.487%); 4.3 MiB/s, 72 objects/s recovering 2024-08-17T18:04:24.092 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:24.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:24.119 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:24.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:24.163 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 29 16 48088 48072 103.589 65 0.00644138 0.00964402 2024-08-17T18:04:24.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:24.248 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:24.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:24.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:24.350 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:24.373 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:04:24.375 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:24.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:24.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:24.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:24.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:24.521 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:24.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:24.622 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:24.662 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:24.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:24.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:24.764 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:24.793 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:24.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:24.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:24.829 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:04:24.829 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-17T18:04:24.829 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:04:24.830 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:24 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 12491/135615 objects degraded (9.211%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:04:24.897 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:24.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:24.942 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:24.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:24.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:24 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 12491/135615 objects degraded (9.211%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:04:24.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:24 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 12491/135615 objects degraded (9.211%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:04:25.043 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:25.086 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:25.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:25.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:25.163 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 30 16 49129 49113 102.305 65.0625 0.0214477 0.00976862 2024-08-17T18:04:25.189 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:25.229 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:25.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:25.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:25.331 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:25.355 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:25.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:25.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:25.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-17T18:04:25.457 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:25.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:25.505 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:25.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:25.606 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:25.615 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:25 smithi136 ceph-mon[109861]: pgmap v2570: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 66 MiB/s wr, 1.05k op/s; 12491/136518 objects degraded (9.150%); 2017/136518 objects misplaced (1.477%); 3.4 MiB/s, 56 objects/s recovering 2024-08-17T18:04:25.632 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:25.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:25.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:25.734 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:25.758 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:25.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:25.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:25.859 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:25.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:25.883 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:25.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:25.911 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:04:25.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:25 smithi136 ceph-mon[115078]: pgmap v2570: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 66 MiB/s wr, 1.05k op/s; 12491/136518 objects degraded (9.150%); 2017/136518 objects misplaced (1.477%); 3.4 MiB/s, 56 objects/s recovering 2024-08-17T18:04:25.984 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:26.008 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:26.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:26.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:25 smithi165 ceph-mon[105517]: pgmap v2570: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 66 MiB/s wr, 1.05k op/s; 12491/136518 objects degraded (9.150%); 2017/136518 objects misplaced (1.477%); 3.4 MiB/s, 56 objects/s recovering 2024-08-17T18:04:26.109 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:26.148 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:26.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:26.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:26.163 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 31 16 50111 50095 100.984 61.375 0.0111133 0.00989699 2024-08-17T18:04:26.250 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:26.275 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:26.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:26.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:26.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:26.425 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:26.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:26.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:26.445 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:04:26.527 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:26.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:26.578 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:26.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:26.680 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:26.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:26.724 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:26.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:26.825 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:26.851 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:26.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:26.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:26.953 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:26.994 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:26.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:26.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:27.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:27.120 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:27.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:27.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:27.164 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 32 16 51138 51122 99.8342 64.1875 0.00993715 0.01001 2024-08-17T18:04:27.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:27.243 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:27.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:27.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:27.345 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:27.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:27.388 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:27.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:27.489 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:27.512 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:27.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:27.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:27.615 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:27.639 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:27.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:27.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:27.741 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:27.766 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:27.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:27.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:27.868 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:27.893 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:27.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:27.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:27.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:27 smithi136 ceph-mon[109861]: pgmap v2571: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 38 MiB/s wr, 613 op/s; 12491/136518 objects degraded (9.150%); 2017/136518 objects misplaced (1.477%); 2.6 MiB/s, 40 objects/s recovering 2024-08-17T18:04:27.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:27 smithi136 ceph-mon[115078]: pgmap v2571: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 38 MiB/s wr, 613 op/s; 12491/136518 objects degraded (9.150%); 2017/136518 objects misplaced (1.477%); 2.6 MiB/s, 40 objects/s recovering 2024-08-17T18:04:27.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:04:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:04:27] "GET /metrics HTTP/1.1" 200 36635 "" "Prometheus/2.51.0" 2024-08-17T18:04:27.994 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:28.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:27 smithi165 ceph-mon[105517]: pgmap v2571: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 38 MiB/s wr, 613 op/s; 12491/136518 objects degraded (9.150%); 2017/136518 objects misplaced (1.477%); 2.6 MiB/s, 40 objects/s recovering 2024-08-17T18:04:28.039 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:28.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:28.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:28.141 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:28.163 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 33 16 52123 52107 98.6743 61.5625 0.0207818 0.0101272 2024-08-17T18:04:28.164 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:28.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:28.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:28.266 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:28.289 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:28.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:28.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:28.391 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:28.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:28.434 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:28.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:28.535 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:28.576 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:28.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:28.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:28.678 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:28.718 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:28.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:28.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:28.821 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:28.845 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:28.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:28.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:28.924 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:04:28.925 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:04:28.925 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:04:28.925 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:04:28.925 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:04:28.925 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:04:28.925 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:04:28.925 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:04:28.925 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:04:28.925 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:04:28.925 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:04:28.926 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:04:28.926 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:04:28.926 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:04:28.926 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:04:28.926 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:04:28.926 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:28 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/404942183' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:04:28.927 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:28 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/404942183' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:04:28.947 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:28.993 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:28.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:28.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:28 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/404942183' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:04:29.094 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:29.119 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:29.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:29.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:29.164 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 34 16 53102 53086 97.5716 61.1875 0.00637964 0.0102409 2024-08-17T18:04:29.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:29.242 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:29.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:29.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:29.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:29.367 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:29.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:29.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:29.469 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:29.495 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:29.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:29.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:29.597 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:29.619 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:29.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:29.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:29.721 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:29.746 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:29.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:29.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:29.848 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:29.891 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:29.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:29.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:29.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:29 smithi136 ceph-mon[109861]: pgmap v2572: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 64 MiB/s wr, 1.02k op/s; 12491/151992 objects degraded (8.218%); 1661/151992 objects misplaced (1.093%); 4.4 MiB/s, 70 objects/s recovering 2024-08-17T18:04:29.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:29 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 12491/151992 objects degraded (8.218%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:04:29.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:29 smithi136 ceph-mon[115078]: pgmap v2572: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 64 MiB/s wr, 1.02k op/s; 12491/151992 objects degraded (8.218%); 1661/151992 objects misplaced (1.093%); 4.4 MiB/s, 70 objects/s recovering 2024-08-17T18:04:29.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:29 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 12491/151992 objects degraded (8.218%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:04:29.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:29 smithi165 ceph-mon[105517]: pgmap v2572: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 64 MiB/s wr, 1.02k op/s; 12491/151992 objects degraded (8.218%); 1661/151992 objects misplaced (1.093%); 4.4 MiB/s, 70 objects/s recovering 2024-08-17T18:04:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:29 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 12491/151992 objects degraded (8.218%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:04:30.017 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:30.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:30.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:30.119 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:30.156 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:30.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:30.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:30.164 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 35 16 54042 54026 96.4623 58.75 0.00511997 0.01036 2024-08-17T18:04:30.259 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:30.297 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:30.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:30.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:30.399 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:30.442 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:30.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:30.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:30.543 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:30.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:30.570 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:30.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:30.649 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 0] out_osds: [5, 7] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:04:30.649 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:04:30.649 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-08-17T18:04:30.650 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd in 5 2024-08-17T18:04:30.671 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:30.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:30.739 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:30.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:30.840 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:30.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:30.881 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:30.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:30.982 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:31.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:31.032 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:31.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:31.133 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:31.164 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 36 16 55033 55017 95.5032 61.9375 0.0197818 0.010464 2024-08-17T18:04:31.178 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:31.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:31.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:31.282 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:31.307 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:31.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:31.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:31.409 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:31.455 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:31.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:31.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:31.560 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:31.584 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:31.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:31.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:31.686 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:31.710 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:31.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:31.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:31.812 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:31.837 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:31.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:31.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:31.892 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:31 smithi136 ceph-mon[109861]: pgmap v2573: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 64 MiB/s wr, 1.02k op/s; 12491/151992 objects degraded (8.218%); 1661/151992 objects misplaced (1.093%); 4.4 MiB/s, 70 objects/s recovering 2024-08-17T18:04:31.892 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:31 smithi136 ceph-mon[115078]: pgmap v2573: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 64 MiB/s wr, 1.02k op/s; 12491/151992 objects degraded (8.218%); 1661/151992 objects misplaced (1.093%); 4.4 MiB/s, 70 objects/s recovering 2024-08-17T18:04:31.940 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:31.962 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:31.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:31.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:31 smithi165 ceph-mon[105517]: pgmap v2573: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 64 MiB/s wr, 1.02k op/s; 12491/151992 objects degraded (8.218%); 1661/151992 objects misplaced (1.093%); 4.4 MiB/s, 70 objects/s recovering 2024-08-17T18:04:32.064 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:32.108 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:32.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:32.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:32.164 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 37 16 56039 56023 94.6212 62.875 0.00963407 0.0105621 2024-08-17T18:04:32.211 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:32.253 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:32.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:32.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:32.355 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:32.394 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:32.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:32.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:32.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:32.520 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:32.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:32.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:32.622 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:32.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:32.647 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:32.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:32.748 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:32.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:32.774 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:32.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:32.875 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:32.918 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:32.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:32.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:33.020 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:33.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:33.068 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:33.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:33.164 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 38 16 56994 56978 93.7018 59.6875 0.00762486 0.0106641 2024-08-17T18:04:33.170 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:33.192 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:33.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:33.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:33.294 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:33.318 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:33.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:33.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:33.420 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:33.465 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:33.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:33.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:33.567 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:33.606 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:33.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:33.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:33.667 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:04:33.708 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:33.763 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:33.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:33.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:33.865 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:33.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:33.911 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:33.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:33.939 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:04:33.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:33 smithi136 ceph-mon[109861]: pgmap v2574: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.4 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 75 MiB/s wr, 1.21k op/s; 12491/165867 objects degraded (7.531%); 1239/165867 objects misplaced (0.747%); 6.6 MiB/s, 106 objects/s recovering 2024-08-17T18:04:33.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:33 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/951371666' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:04:33.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:33 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:04:33.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:33 smithi136 ceph-mon[115078]: pgmap v2574: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.4 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 75 MiB/s wr, 1.21k op/s; 12491/165867 objects degraded (7.531%); 1239/165867 objects misplaced (0.747%); 6.6 MiB/s, 106 objects/s recovering 2024-08-17T18:04:33.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:33 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/951371666' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:04:33.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:33 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:04:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:33 smithi165 ceph-mon[105517]: pgmap v2574: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.4 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 75 MiB/s wr, 1.21k op/s; 12491/165867 objects degraded (7.531%); 1239/165867 objects misplaced (0.747%); 6.6 MiB/s, 106 objects/s recovering 2024-08-17T18:04:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:33 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/951371666' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:04:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:33 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:04:34.012 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:34.037 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:34.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:34.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:34.139 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:34.164 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 39 16 57860 57844 92.6863 54.125 0.0171999 0.0107824 2024-08-17T18:04:34.186 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:34.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:34.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:34.288 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:34.312 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:34.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:34.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:34.413 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:34.452 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:34.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:34.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:34.554 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:34.577 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:34.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:34.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:34.679 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:34.704 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:34.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:34.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:34.806 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:34.844 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:34.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:34.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:34.946 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:34.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:34.984 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:34.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:34 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:04:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:34 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:04:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:34 smithi165 ceph-mon[105517]: osdmap e1463: 8 total, 8 up, 6 in 2024-08-17T18:04:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:34 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 12491/165867 objects degraded (7.531%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:04:35.085 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:35.110 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:35.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:35.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:35.165 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:2024-08-17T18:04:35.164943+0000 min lat: 0.00131667 max lat: 1.66948 avg lat: 0.0108869 2024-08-17T18:04:35.165 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T18:04:35.165 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 40 16 58777 58761 91.8019 57.3125 0.00940925 0.0108869 2024-08-17T18:04:35.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:34 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:04:35.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:34 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:04:35.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:34 smithi136 ceph-mon[109861]: osdmap e1463: 8 total, 8 up, 6 in 2024-08-17T18:04:35.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:34 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 12491/165867 objects degraded (7.531%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:04:35.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:34 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:04:35.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:34 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:04:35.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:34 smithi136 ceph-mon[115078]: osdmap e1463: 8 total, 8 up, 6 in 2024-08-17T18:04:35.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:34 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 12491/165867 objects degraded (7.531%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:04:35.212 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:35.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:35.234 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:35.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:35.335 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:35.360 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:35.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:35.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:35.462 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:35.499 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:35.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:35.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:35.601 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:35.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:35.640 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:35.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:35.741 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:35.775 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:35.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:35.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:35.877 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:35.916 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:35.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:35.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:35 smithi165 ceph-mon[105517]: pgmap v2576: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.4 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 63 MiB/s wr, 1.01k op/s; 12491/166764 objects degraded (7.490%); 1239/166764 objects misplaced (0.743%); 4.9 MiB/s, 77 objects/s recovering 2024-08-17T18:04:36.019 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:36.043 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:36.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:36.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:36.145 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:36.164 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 41 16 59529 59513 90.709 47 0.106444 0.0110067 2024-08-17T18:04:36.172 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:36.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:36.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:36.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:35 smithi136 ceph-mon[109861]: pgmap v2576: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.4 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 63 MiB/s wr, 1.01k op/s; 12491/166764 objects degraded (7.490%); 1239/166764 objects misplaced (0.743%); 4.9 MiB/s, 77 objects/s recovering 2024-08-17T18:04:36.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:35 smithi136 ceph-mon[115078]: pgmap v2576: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.4 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 63 MiB/s wr, 1.01k op/s; 12491/166764 objects degraded (7.490%); 1239/166764 objects misplaced (0.743%); 4.9 MiB/s, 77 objects/s recovering 2024-08-17T18:04:36.274 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:36.305 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:36.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:36.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:36.407 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:36.428 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:36.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:36.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:36.530 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:36.552 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:36.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:36.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:36.654 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:36.677 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:36.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:36.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:36.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:36.804 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:36.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:36.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:36.905 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:36.929 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:36.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:36.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:04:37.028 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:04:37.029 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:04:37.031 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:37.070 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:37.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:37.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:37.104 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:04:37.104 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:04:37.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.104 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.45626.0:60185 55.9 55.78d3f689 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1463)", 2024-08-17T18:04:37.104 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:04:37.088554+0000", 2024-08-17T18:04:37.104 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013707763, 2024-08-17T18:04:37.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013732873, 2024-08-17T18:04:37.104 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:37.104 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:37.104 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-08-17T18:04:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:04:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.45626", 2024-08-17T18:04:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:04:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60185 2024-08-17T18:04:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.088554+0000", 2024-08-17T18:04:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:37.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.088554+0000", 2024-08-17T18:04:37.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:37.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:37.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.088555+0000", 2024-08-17T18:04:37.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.198e-06 2024-08-17T18:04:37.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:37.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.088579+0000", 2024-08-17T18:04:37.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4185e-05 2024-08-17T18:04:37.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:37.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.088592+0000", 2024-08-17T18:04:37.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2438e-05 2024-08-17T18:04:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.088599+0000", 2024-08-17T18:04:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1049999999999997e-06 2024-08-17T18:04:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.45626.0:60174 55.1 e1463/1458 ondisk, result = 0)", 2024-08-17T18:04:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:04:37.097689+0000", 2024-08-17T18:04:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0045722169999999999, 2024-08-17T18:04:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0046446669999999999, 2024-08-17T18:04:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T18:04:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.097689+0000", 2024-08-17T18:04:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.097689+0000", 2024-08-17T18:04:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.097699+0000", 2024-08-17T18:04:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.217e-06 2024-08-17T18:04:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.097704+0000", 2024-08-17T18:04:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4240000000000001e-06 2024-08-17T18:04:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.097714+0000", 2024-08-17T18:04:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0299000000000001e-05 2024-08-17T18:04:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.097721+0000", 2024-08-17T18:04:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5899999999999996e-06 2024-08-17T18:04:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.100277+0000", 2024-08-17T18:04:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025560779999999998 2024-08-17T18:04:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:04:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.100284+0000", 2024-08-17T18:04:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9870000000000002e-06 2024-08-17T18:04:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.45626.0:60191 55.a 55:594d85aa:::benchmark_data_smithi136_313476_object60190:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1463)", 2024-08-17T18:04:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:04:37.098122+0000", 2024-08-17T18:04:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.004139813, 2024-08-17T18:04:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0042619329999999999, 2024-08-17T18:04:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-17T18:04:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:04:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.45626", 2024-08-17T18:04:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:04:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60191 2024-08-17T18:04:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.098122+0000", 2024-08-17T18:04:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.098122+0000", 2024-08-17T18:04:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.098123+0000", 2024-08-17T18:04:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.327e-06 2024-08-17T18:04:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.098143+0000", 2024-08-17T18:04:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9842000000000001e-05 2024-08-17T18:04:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.098154+0000", 2024-08-17T18:04:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1187e-05 2024-08-17T18:04:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.098161+0000", 2024-08-17T18:04:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4860000000000002e-06 2024-08-17T18:04:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.098207+0000", 2024-08-17T18:04:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6884e-05 2024-08-17T18:04:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:04:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.098358+0000", 2024-08-17T18:04:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015047199999999999 2024-08-17T18:04:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4", 2024-08-17T18:04:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.098471+0000", 2024-08-17T18:04:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011356199999999999 2024-08-17T18:04:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:04:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.100671+0000", 2024-08-17T18:04:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021993350000000002 2024-08-17T18:04:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.45626.0:60174 55.1 55:83a16602:::benchmark_data_smithi136_313476_object60173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1463)", 2024-08-17T18:04:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:04:37.075271+0000", 2024-08-17T18:04:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.026990362, 2024-08-17T18:04:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.027173696000000001, 2024-08-17T18:04:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:04:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:04:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.45626", 2024-08-17T18:04:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:04:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60174 2024-08-17T18:04:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.075271+0000", 2024-08-17T18:04:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.075271+0000", 2024-08-17T18:04:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.075272+0000", 2024-08-17T18:04:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9500000000000002e-07 2024-08-17T18:04:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.075291+0000", 2024-08-17T18:04:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8719000000000002e-05 2024-08-17T18:04:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.075299+0000", 2024-08-17T18:04:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2930000000000001e-06 2024-08-17T18:04:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.075304+0000", 2024-08-17T18:04:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2700000000000004e-06 2024-08-17T18:04:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.084002+0000", 2024-08-17T18:04:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0086981929999999999 2024-08-17T18:04:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:04:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.084166+0000", 2024-08-17T18:04:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016341500000000001 2024-08-17T18:04:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4", 2024-08-17T18:04:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.084319+0000", 2024-08-17T18:04:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015306299999999999 2024-08-17T18:04:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:04:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.097977+0000", 2024-08-17T18:04:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013657938999999999 2024-08-17T18:04:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.100245+0000", 2024-08-17T18:04:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002268384 2024-08-17T18:04:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.100289+0000", 2024-08-17T18:04:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4302999999999999e-05 2024-08-17T18:04:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:04:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.101380+0000", 2024-08-17T18:04:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010904179999999999 2024-08-17T18:04:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.45626.0:60180 55.1 55.b824fbf1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1463)", 2024-08-17T18:04:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:04:37.086260+0000", 2024-08-17T18:04:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.016001909000000002, 2024-08-17T18:04:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016259862, 2024-08-17T18:04:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-08-17T18:04:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:04:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.45626", 2024-08-17T18:04:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:04:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60180 2024-08-17T18:04:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.086260+0000", 2024-08-17T18:04:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.086260+0000", 2024-08-17T18:04:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.086261+0000", 2024-08-17T18:04:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.595e-06 2024-08-17T18:04:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.086284+0000", 2024-08-17T18:04:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3226000000000001e-05 2024-08-17T18:04:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.086406+0000", 2024-08-17T18:04:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121321 2024-08-17T18:04:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:37.086421+0000", 2024-08-17T18:04:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5257e-05 2024-08-17T18:04:37.127 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:37.127 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:37.127 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:37.127 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:37.127 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-17T18:04:37.127 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 5 2024-08-17T18:04:37.127 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:04:37.165 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 42 16 60245 60229 89.6147 44.75 0.00825187 0.0111517 2024-08-17T18:04:37.172 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:37.195 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:37.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:37.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:37.297 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:37.337 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:37.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:37.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:37.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:37.464 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:37.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:37.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:37.566 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:37.590 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:37.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:37.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:37.692 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:37.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:04:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:04:37] "GET /metrics HTTP/1.1" 200 36639 "" "Prometheus/2.51.0" 2024-08-17T18:04:37.716 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:37.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:37.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:37.818 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:37.903 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:37.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:37.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:37.998 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:37 smithi136 ceph-mon[115078]: pgmap v2577: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.4 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 63 MiB/s wr, 1.01k op/s; 12491/166764 objects degraded (7.490%); 1239/166764 objects misplaced (0.743%); 4.9 MiB/s, 77 objects/s recovering 2024-08-17T18:04:37.999 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:37 smithi136 ceph-mon[109861]: pgmap v2577: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.4 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 63 MiB/s wr, 1.01k op/s; 12491/166764 objects degraded (7.490%); 1239/166764 objects misplaced (0.743%); 4.9 MiB/s, 77 objects/s recovering 2024-08-17T18:04:38.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:37 smithi165 ceph-mon[105517]: pgmap v2577: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.4 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 63 MiB/s wr, 1.01k op/s; 12491/166764 objects degraded (7.490%); 1239/166764 objects misplaced (0.743%); 4.9 MiB/s, 77 objects/s recovering 2024-08-17T18:04:38.033 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:38.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:38.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:38.135 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:38.160 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:38.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:38.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:38.165 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 43 16 61203 61187 88.923 59.875 0.0176569 0.0112389 2024-08-17T18:04:38.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:38.287 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:38.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:38.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:38.376 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:04:38.379 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:04:38.389 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:38.413 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:38.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:38.416 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-17T18:04:38.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:38.518 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:38.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:38.587 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:38.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:38.688 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:38.736 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:38.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:38.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:38.838 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:38.861 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:38.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:38.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:38.963 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:38.986 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:38.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:38.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:39.088 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:39.126 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:39.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:39.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:39.165 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 44 16 62068 62052 88.1307 54.0625 0.0226526 0.0113404 2024-08-17T18:04:39.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:39.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:39.267 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:39.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:39.369 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:39.411 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:39.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:39.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:39.513 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:39.557 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:39.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:39.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:39.660 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:39.686 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:39.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:39.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:39.788 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:39.832 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:39.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:39.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:39.886 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:04:39.933 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:39.937 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:39 smithi136 ceph-mon[115078]: pgmap v2578: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 58 MiB/s wr, 920 op/s; 12491/179628 objects degraded (6.954%); 713/179628 objects misplaced (0.397%); 5.9 MiB/s, 94 objects/s recovering 2024-08-17T18:04:39.938 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:39 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 12491/179628 objects degraded (6.954%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:04:39.938 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:39 smithi136 ceph-mon[109861]: pgmap v2578: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 58 MiB/s wr, 920 op/s; 12491/179628 objects degraded (6.954%); 713/179628 objects misplaced (0.397%); 5.9 MiB/s, 94 objects/s recovering 2024-08-17T18:04:39.938 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:39 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 12491/179628 objects degraded (6.954%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:04:39.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:39.981 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:39.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:39 smithi165 ceph-mon[105517]: pgmap v2578: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 58 MiB/s wr, 920 op/s; 12491/179628 objects degraded (6.954%); 713/179628 objects misplaced (0.397%); 5.9 MiB/s, 94 objects/s recovering 2024-08-17T18:04:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:39 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 12491/179628 objects degraded (6.954%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:04:40.084 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:40.146 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:40.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:40.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:40.166 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 45 16 62986 62970 87.445 57.375 0.0399676 0.0114277 2024-08-17T18:04:40.249 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:40.273 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:40.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:40.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:40.375 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:40.398 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:40.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:40.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:40.500 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:40.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:40.557 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:40.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:40.658 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:40.681 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:40.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:40.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:40.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:40.795 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:04:40.805 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:40.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:40.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:40.906 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:40.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:40.959 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:40.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:41.060 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:41.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:41.103 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:41.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:41.166 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 46 16 63895 63879 86.779 56.8125 0.0178988 0.0115169 2024-08-17T18:04:41.204 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:41.226 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:41.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:41.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:41.328 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:41.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:41.385 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:41.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:41.486 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:41.509 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:41.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:41.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:41.612 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:41.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:41.639 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:41.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:41.740 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:41.741 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:41 smithi136 ceph-mon[115078]: pgmap v2579: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 58 MiB/s wr, 920 op/s; 12491/179628 objects degraded (6.954%); 713/179628 objects misplaced (0.397%); 5.9 MiB/s, 94 objects/s recovering 2024-08-17T18:04:41.763 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:41.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:41.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:41.865 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:41.952 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:41.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:41.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:42.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:41 smithi165 ceph-mon[105517]: pgmap v2579: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 58 MiB/s wr, 920 op/s; 12491/179628 objects degraded (6.954%); 713/179628 objects misplaced (0.397%); 5.9 MiB/s, 94 objects/s recovering 2024-08-17T18:04:42.054 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:42.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:42.093 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:42.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:42.166 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 47 16 64724 64708 86.035 51.8125 0.0140668 0.0116166 2024-08-17T18:04:42.195 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:42.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:41 smithi136 ceph-mon[109861]: pgmap v2579: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 58 MiB/s wr, 920 op/s; 12491/179628 objects degraded (6.954%); 713/179628 objects misplaced (0.397%); 5.9 MiB/s, 94 objects/s recovering 2024-08-17T18:04:42.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:41 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a[109838]: 2024-08-17T18:04:41.916+0000 7f5b8e77d640 -1 mon.a@0(leader).osd e1463 definitely_dead 0 2024-08-17T18:04:42.218 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:42.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:42.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:42.320 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:42.342 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:42.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:42.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:42.444 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:42.468 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:42.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:42.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:42.569 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:42.614 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:42.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:42.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:42.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:42.739 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:42.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:42.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:42.757 INFO:teuthology.orchestra.run.smithi136.stderr:marked in osd.5. 2024-08-17T18:04:42.841 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:42.863 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:42.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:42.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:42.943 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:42 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2636123036' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-08-17T18:04:42.943 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:42 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2636123036' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-08-17T18:04:42.965 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:42.987 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:42.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:42.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:43.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:42 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2636123036' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-08-17T18:04:43.089 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:43.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:43.111 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:43.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:43.166 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 48 16 65218 65202 84.8858 30.875 0.00324871 0.0116788 2024-08-17T18:04:43.212 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:43.234 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:43.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:43.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:43.335 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:43.357 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:43.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:43.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:43.459 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:43.481 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:43.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:43.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:43.583 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:43.605 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:43.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:43.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:43.705 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:04:43.705 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:04:43.707 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:43.716 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:04:43.716 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:04:43.728 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:43.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:43.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:43.732 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:04:43.732 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:04:43.750 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:04:43.750 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:04:43.774 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:04:43.774 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:04:43.774 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:43 smithi136 ceph-mon[109861]: pgmap v2580: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 55 MiB/s wr, 878 op/s; 12491/192243 objects degraded (6.498%); 228/192243 objects misplaced (0.119%); 6.3 MiB/s, 101 objects/s recovering 2024-08-17T18:04:43.774 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:43 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2636123036' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-08-17T18:04:43.774 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:43 smithi136 ceph-mon[109861]: osdmap e1464: 8 total, 8 up, 7 in 2024-08-17T18:04:43.774 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:43 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1809861360' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:04:43.775 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:43 smithi136 ceph-mon[115078]: pgmap v2580: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 55 MiB/s wr, 878 op/s; 12491/192243 objects degraded (6.498%); 228/192243 objects misplaced (0.119%); 6.3 MiB/s, 101 objects/s recovering 2024-08-17T18:04:43.775 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:43 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2636123036' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-08-17T18:04:43.775 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:43 smithi136 ceph-mon[115078]: osdmap e1464: 8 total, 8 up, 7 in 2024-08-17T18:04:43.775 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:43 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1809861360' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:04:43.807 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:04:43.807 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:04:43.830 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:43.838 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:04:43.838 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:04:43.855 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:43.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:43.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:43.858 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:04:43.858 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:04:43.945 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:04:43.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:43.978 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:43.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:43.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:43 smithi165 ceph-mon[105517]: pgmap v2580: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 55 MiB/s wr, 878 op/s; 12491/192243 objects degraded (6.498%); 228/192243 objects misplaced (0.119%); 6.3 MiB/s, 101 objects/s recovering 2024-08-17T18:04:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:43 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2636123036' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-08-17T18:04:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:43 smithi165 ceph-mon[105517]: osdmap e1464: 8 total, 8 up, 7 in 2024-08-17T18:04:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:43 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1809861360' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:04:44.080 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:44.103 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:44.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:44.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:44.166 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 49 16 65224 65208 83.1612 0.375 1.1318 0.0117487 2024-08-17T18:04:44.205 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:44.227 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:44.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:44.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:44.329 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:44.351 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:44.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:44.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:44.453 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:44.475 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:44.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:44.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:44.577 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:44.599 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:44.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:44.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:44.606 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-08-17T18:04:44.701 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:44.722 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:44.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:44.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:44.824 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:44.846 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:44.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:44.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:44.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:44 smithi136 ceph-mon[109861]: osdmap e1465: 8 total, 8 up, 7 in 2024-08-17T18:04:44.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:44 smithi136 ceph-mon[109861]: pgmap v2583: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 68 MiB/s wr, 1.09k op/s; 12491/193044 objects degraded (6.471%); 228/193044 objects misplaced (0.118%); 7.9 MiB/s, 126 objects/s recovering 2024-08-17T18:04:44.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:44 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 12491/192243 objects degraded (6.498%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:04:44.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:44 smithi136 ceph-mon[115078]: osdmap e1465: 8 total, 8 up, 7 in 2024-08-17T18:04:44.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:44 smithi136 ceph-mon[115078]: pgmap v2583: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 68 MiB/s wr, 1.09k op/s; 12491/193044 objects degraded (6.471%); 228/193044 objects misplaced (0.118%); 7.9 MiB/s, 126 objects/s recovering 2024-08-17T18:04:44.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:44 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 12491/192243 objects degraded (6.498%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:04:44.948 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:44.995 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:44.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:44.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:45.097 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:45.131 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:45.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:45.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:45.166 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 50 16 65355 65339 81.6618 8.1875 0.0109289 0.0122386 2024-08-17T18:04:45.233 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:45.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:44 smithi165 ceph-mon[105517]: osdmap e1465: 8 total, 8 up, 7 in 2024-08-17T18:04:45.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:44 smithi165 ceph-mon[105517]: pgmap v2583: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 68 MiB/s wr, 1.09k op/s; 12491/193044 objects degraded (6.471%); 228/193044 objects misplaced (0.118%); 7.9 MiB/s, 126 objects/s recovering 2024-08-17T18:04:45.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:44 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 12491/192243 objects degraded (6.498%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:04:45.271 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:45.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:45.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:45.373 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:45.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:45.423 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:45.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:45.524 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:45.560 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:45.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:45.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:45.662 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:45.700 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:45.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:45.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:45.802 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:45.836 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:45.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:45.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:45.869 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:45 smithi136 ceph-mon[109861]: osdmap e1466: 8 total, 8 up, 7 in 2024-08-17T18:04:45.869 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:45 smithi136 ceph-mon[115078]: osdmap e1466: 8 total, 8 up, 7 in 2024-08-17T18:04:45.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:45.969 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:45.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:45.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:46.071 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:46.107 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:46.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:46.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:46.167 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 51 16 66266 66250 81.1769 56.9375 0.0138559 0.012312 2024-08-17T18:04:46.209 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:46.248 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:46.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:46.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:46.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:45 smithi165 ceph-mon[105517]: osdmap e1466: 8 total, 8 up, 7 in 2024-08-17T18:04:46.350 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:46.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:46.389 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:46.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:46.490 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:46.525 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:46.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:46.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:46.627 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:46.664 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:46.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:46.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:46.766 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:46.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:46.805 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:46.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:46.906 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:46.952 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:46.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:46.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:47.035 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:04:47.055 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:47.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:46 smithi136 ceph-mon[109861]: pgmap v2585: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 47 MiB/s wr, 745 op/s; 12491/193044 objects degraded (6.471%); 228/193044 objects misplaced (0.118%); 5.1 MiB/s, 80 objects/s recovering 2024-08-17T18:04:47.080 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:46 smithi136 ceph-mon[115078]: pgmap v2585: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 47 MiB/s wr, 745 op/s; 12491/193044 objects degraded (6.471%); 228/193044 objects misplaced (0.118%); 5.1 MiB/s, 80 objects/s recovering 2024-08-17T18:04:47.085 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:47.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:47.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:47.167 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 52 16 67174 67158 80.7071 56.75 0.0560759 0.0123831 2024-08-17T18:04:47.188 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:47.195 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:04:47.195 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:04:47.259 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:47.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:47.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:47.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:46 smithi165 ceph-mon[105517]: pgmap v2585: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 47 MiB/s wr, 745 op/s; 12491/193044 objects degraded (6.471%); 228/193044 objects misplaced (0.118%); 5.1 MiB/s, 80 objects/s recovering 2024-08-17T18:04:47.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:47.398 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:47.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:47.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:47.500 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:47.540 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:47.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:47.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:47.642 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:47.671 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:47.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:47.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:47.773 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:47.812 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:47.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:47.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:47.855 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:04:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:04:47] "GET /metrics HTTP/1.1" 200 36639 "" "Prometheus/2.51.0" 2024-08-17T18:04:47.914 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:47.951 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:47.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:47.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:48.053 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:48.084 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:48.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:48.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:48.167 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 53 16 68029 68013 80.1925 53.4375 0.0142238 0.0124595 2024-08-17T18:04:48.186 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:48.217 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:48.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:48.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:48.319 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:48.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:48.361 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:48.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:48.463 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:48.494 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:48.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:48.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:48.596 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:48.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:48.642 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:48.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:48.743 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:48.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:48.788 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:48.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:48.889 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:48.932 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:48.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:48.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:49.034 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:49.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:49.064 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:49.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:49.165 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:49.167 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 54 16 68674 68658 79.4539 40.3125 0.0294337 0.0125611 2024-08-17T18:04:49.186 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:49.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:49.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:49.289 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:49.332 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:49.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:49.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:49.434 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:49.457 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:49.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:49.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:49.558 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:49 smithi136 ceph-mon[109861]: pgmap v2586: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 28 MiB/s wr, 453 op/s; 25438/200412 objects degraded (12.693%); 10251/200412 objects misplaced (5.115%); 6.8 MiB/s, 109 objects/s recovering 2024-08-17T18:04:49.558 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:49 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3117274448' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:04:49.559 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:49 smithi136 ceph-mon[115078]: pgmap v2586: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 28 MiB/s wr, 453 op/s; 25438/200412 objects degraded (12.693%); 10251/200412 objects misplaced (5.115%); 6.8 MiB/s, 109 objects/s recovering 2024-08-17T18:04:49.559 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:49 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3117274448' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:04:49.559 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:49.589 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:49.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:49.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:49.607 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 0, 5] out_osds: [7] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:04:49.607 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:04:49.607 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-08-17T18:04:49.608 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-17T18:04:49.608 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-17T18:04:49.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-08-17T18:04:49.631 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:49 smithi165 ceph-mon[105517]: pgmap v2586: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 28 MiB/s wr, 453 op/s; 25438/200412 objects degraded (12.693%); 10251/200412 objects misplaced (5.115%); 6.8 MiB/s, 109 objects/s recovering 2024-08-17T18:04:49.631 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:49 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3117274448' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:04:49.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:49.712 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:49.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:49.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:49.814 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:49.838 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:49.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:49.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:49.929 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:04:49.940 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:49.970 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:49.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:49.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:50.072 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:50.113 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:50.114 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:04:50.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:50.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:50.167 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 55 16 69158 69142 78.5593 30.25 0.0339422 0.0127226 2024-08-17T18:04:50.215 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:50.239 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:50.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:50.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:50.341 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:50.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:50 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:04:50.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:50 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3117274448' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:04:50.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:50 smithi165 ceph-mon[105517]: osdmap e1467: 8 total, 8 up, 7 in 2024-08-17T18:04:50.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:50 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 25438/200412 objects degraded (12.693%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:04:50.377 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:50.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:50.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:50.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:50 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:04:50.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:50 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3117274448' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:04:50.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:50 smithi136 ceph-mon[109861]: osdmap e1467: 8 total, 8 up, 7 in 2024-08-17T18:04:50.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:50 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 25438/200412 objects degraded (12.693%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:04:50.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:50 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:04:50.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:50 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3117274448' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:04:50.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:50 smithi136 ceph-mon[115078]: osdmap e1467: 8 total, 8 up, 7 in 2024-08-17T18:04:50.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:50 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 25438/200412 objects degraded (12.693%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:04:50.479 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:50.506 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:50.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:50.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:50.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:50.640 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:50.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:50.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:50.742 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:50.774 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:50.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:50.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:50.876 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:50.908 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:50.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:50.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:51.010 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:51.038 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:51.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:51.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:51.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:51.167 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 56 16 69836 69820 77.9131 42.375 0.019164 0.0128285 2024-08-17T18:04:51.173 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:51.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:51.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:51.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:51.295 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:51.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:51.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:51.397 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:51.428 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:51.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:51.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:51.530 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:51.553 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:51.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:51.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:51.655 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:51.677 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:51.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:51.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:51 smithi136 ceph-mon[109861]: pgmap v2588: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 24 MiB/s wr, 380 op/s; 25438/200412 objects degraded (12.693%); 10251/200412 objects misplaced (5.115%); 6.3 MiB/s, 101 objects/s recovering 2024-08-17T18:04:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:04:51.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:51 smithi136 ceph-mon[115078]: pgmap v2588: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 24 MiB/s wr, 380 op/s; 25438/200412 objects degraded (12.693%); 10251/200412 objects misplaced (5.115%); 6.3 MiB/s, 101 objects/s recovering 2024-08-17T18:04:51.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:04:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:51 smithi165 ceph-mon[105517]: pgmap v2588: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 24 MiB/s wr, 380 op/s; 25438/200412 objects degraded (12.693%); 10251/200412 objects misplaced (5.115%); 6.3 MiB/s, 101 objects/s recovering 2024-08-17T18:04:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:04:51.778 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:51.809 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:51.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:51.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:51.911 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:51.952 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:51.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:51.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:52.054 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:52.086 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:52.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:52.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:52.167 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 57 16 70519 70503 77.2951 42.6875 0.00738703 0.0129304 2024-08-17T18:04:52.188 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:52.218 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:52.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:52.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:52.320 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:52.357 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:52.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:52.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:52.459 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:52.492 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:52.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:52.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:52.594 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:52.631 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:52.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:52.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:52.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:52.760 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:52.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:52.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:52.862 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:52.895 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:52.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:52.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:52.997 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:53.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:53.031 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:53.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:53.132 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:53.164 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:53.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:53.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:53.167 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 58 16 71425 71409 76.9387 56.625 0.0107488 0.0129912 2024-08-17T18:04:53.266 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:53.301 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:53.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:53.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:53.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:53.436 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:53.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:53.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:53.538 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:53.575 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:53.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:53.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:53.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:53 smithi136 ceph-mon[109861]: pgmap v2589: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 44 MiB/s wr, 709 op/s; 25438/210072 objects degraded (12.109%); 9735/210072 objects misplaced (4.634%); 9.1 MiB/s, 146 objects/s recovering 2024-08-17T18:04:53.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:53 smithi136 ceph-mon[115078]: pgmap v2589: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 44 MiB/s wr, 709 op/s; 25438/210072 objects degraded (12.109%); 9735/210072 objects misplaced (4.634%); 9.1 MiB/s, 146 objects/s recovering 2024-08-17T18:04:53.704 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:53.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:53.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:53 smithi165 ceph-mon[105517]: pgmap v2589: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 44 MiB/s wr, 709 op/s; 25438/210072 objects degraded (12.109%); 9735/210072 objects misplaced (4.634%); 9.1 MiB/s, 146 objects/s recovering 2024-08-17T18:04:53.806 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:53.839 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:53.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:53.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:53.941 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:53.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:53.972 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:53.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:54.073 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:54.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:54.103 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:54.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:54.167 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 59 16 72260 72244 76.5191 52.1875 0.0208182 0.0130605 2024-08-17T18:04:54.204 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:54.250 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:54.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:54.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:54.352 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:54.391 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:54.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:54.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:54.493 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:54.534 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:54.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:54.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:54.636 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:54.660 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-17T18:04:54.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:54.673 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:54.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:54.774 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:54.807 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:54.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:54.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:54.909 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:54.943 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:54.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:54.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:55.045 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:55.050 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:04:55.050 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-17T18:04:55.050 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:04:55.091 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:55.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:55.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:55.170 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:2024-08-17T18:04:55.169670+0000 min lat: 0.00131667 max lat: 2.29878 avg lat: 0.0131333 2024-08-17T18:04:55.170 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T18:04:55.170 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 60 16 73089 73073 76.105 51.8125 0.0399815 0.0131333 2024-08-17T18:04:55.193 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:55.209 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:04:55.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:55.222 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:55.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:55.241 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:04:55.324 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:55.361 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:55.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:55.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:55.407 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:55 smithi136 ceph-mon[109861]: pgmap v2590: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 39 MiB/s wr, 620 op/s; 25438/210648 objects degraded (12.076%); 9735/210648 objects misplaced (4.621%); 7.7 MiB/s, 123 objects/s recovering 2024-08-17T18:04:55.407 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:55 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 25438/210648 objects degraded (12.076%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:04:55.407 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:55 smithi136 ceph-mon[115078]: pgmap v2590: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 39 MiB/s wr, 620 op/s; 25438/210648 objects degraded (12.076%); 9735/210648 objects misplaced (4.621%); 7.7 MiB/s, 123 objects/s recovering 2024-08-17T18:04:55.407 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:55 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 25438/210648 objects degraded (12.076%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:04:55.463 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:55.501 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:55.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:55.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:55.603 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:55.644 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:55.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:55.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:55.746 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:55 smithi165 ceph-mon[105517]: pgmap v2590: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 39 MiB/s wr, 620 op/s; 25438/210648 objects degraded (12.076%); 9735/210648 objects misplaced (4.621%); 7.7 MiB/s, 123 objects/s recovering 2024-08-17T18:04:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:55 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 25438/210648 objects degraded (12.076%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:04:55.777 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:55.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:55.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:55.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:55.921 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:55.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:55.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:56.023 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:56.052 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:56.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:56.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:56.154 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:56.169 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 61 16 73919 73903 75.7077 51.875 0.0158057 0.0132023 2024-08-17T18:04:56.181 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:56.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:56.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:56.284 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:56.309 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:56.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:56.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:56.411 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:56.451 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:56.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:56.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:56.553 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:56.608 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:56.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:56.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:56.710 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:56.739 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:56.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:56.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:56.841 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:56.869 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:56.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:56.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:56.971 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:57.012 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:57.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:57.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:57.114 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:57.159 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:57.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:57.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:57.169 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 62 16 74771 74755 75.3454 53.25 0.0174146 0.0132656 2024-08-17T18:04:57.172 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:04:57.261 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:57.291 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:57.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:57.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:57.393 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:57.420 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:57.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:57.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:57.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:57 smithi136 ceph-mon[109861]: pgmap v2591: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 37 MiB/s wr, 586 op/s; 25438/210648 objects degraded (12.076%); 9735/210648 objects misplaced (4.621%); 7.3 MiB/s, 117 objects/s recovering 2024-08-17T18:04:57.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:57 smithi136 ceph-mon[115078]: pgmap v2591: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 37 MiB/s wr, 586 op/s; 25438/210648 objects degraded (12.076%); 9735/210648 objects misplaced (4.621%); 7.3 MiB/s, 117 objects/s recovering 2024-08-17T18:04:57.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:57.544 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:57.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:57.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:57.646 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:57.678 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:57.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:57.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:57 smithi165 ceph-mon[105517]: pgmap v2591: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 37 MiB/s wr, 586 op/s; 25438/210648 objects degraded (12.076%); 9735/210648 objects misplaced (4.621%); 7.3 MiB/s, 117 objects/s recovering 2024-08-17T18:04:57.780 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:57.813 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:57.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:57.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:57.814 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:04:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:04:57] "GET /metrics HTTP/1.1" 200 36644 "" "Prometheus/2.51.0" 2024-08-17T18:04:57.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:57.943 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:57.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:57.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:58.045 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:04:58.085 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:58.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:58.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:04:58.170 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 63 16 75483 75467 74.8558 44.5 0.0171269 0.0133528 2024-08-17T18:04:58.187 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:58.225 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:58.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:58.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:58.327 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:04:58.363 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:58.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:04:58.464 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:58.493 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:58.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:58.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:58.498 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:04:58.594 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:04:58.627 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:58.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:58.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:04:58.729 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:04:58.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:58.760 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:58.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:04:58.861 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:58.899 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:58.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:58.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:59.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:04:59.042 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:59.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:59.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:04:59.144 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:59.170 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 64 16 76307 76291 74.4907 51.5 0.00653914 0.013416 2024-08-17T18:04:59.188 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:59.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:59.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:59.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:04:59.318 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:04:59.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:59.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:04:59.420 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:59.458 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:59.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:59.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:59.498 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:04:59.498 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:04:59.533 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:04:59.534 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:04:59.560 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:04:59.587 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:04:59 smithi136 ceph-mon[109861]: pgmap v2592: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 48 MiB/s wr, 760 op/s; 25438/223221 objects degraded (11.396%); 9310/223221 objects misplaced (4.171%); 5.9 MiB/s, 94 objects/s recovering 2024-08-17T18:04:59.587 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:04:59.587 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:04:59.588 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:04:59 smithi136 ceph-mon[115078]: pgmap v2592: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 48 MiB/s wr, 760 op/s; 25438/223221 objects degraded (11.396%); 9310/223221 objects misplaced (4.171%); 5.9 MiB/s, 94 objects/s recovering 2024-08-17T18:04:59.611 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:59.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:59.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:04:59.631 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:04:59.631 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:04:59.668 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:04:59.669 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:04:59.704 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:04:59.704 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:04:59.705 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:04:59.705 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-17T18:04:59.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-17T18:04:59.705 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:04:59.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.706 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104426 54.10 54:0c08f48e:::benchmark_data_smithi136_273969_object104425:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:04:59.706 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.460498+0000", 2024-08-17T18:04:59.706 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 315.24161949500001, 2024-08-17T18:04:59.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.15692383, 2024-08-17T18:04:59.706 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:59.706 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:59.706 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:04:59.706 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:04:59.706 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:04:59.706 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:04:59.706 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104426 2024-08-17T18:04:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460498+0000", 2024-08-17T18:04:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460498+0000", 2024-08-17T18:04:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460500+0000", 2024-08-17T18:04:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.562e-06 2024-08-17T18:04:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460526+0000", 2024-08-17T18:04:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6466e-05 2024-08-17T18:04:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:59.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460535+0000", 2024-08-17T18:04:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0189999999999995e-06 2024-08-17T18:04:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460542+0000", 2024-08-17T18:04:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8730000000000001e-06 2024-08-17T18:04:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460590+0000", 2024-08-17T18:04:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7673999999999998e-05 2024-08-17T18:04:59.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-08-17T18:04:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460595+0000", 2024-08-17T18:04:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0130000000000001e-06 2024-08-17T18:04:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.462127+0000", 2024-08-17T18:04:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015317869999999999 2024-08-17T18:04:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:04:59.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.462136+0000", 2024-08-17T18:04:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5820000000000008e-06 2024-08-17T18:04:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.582918+0000", 2024-08-17T18:04:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1207821739999999 2024-08-17T18:04:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:04:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.583070+0000", 2024-08-17T18:04:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015153000000000001 2024-08-17T18:04:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:04:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.583305+0000", 2024-08-17T18:04:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023529200000000001 2024-08-17T18:04:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.611591+0000", 2024-08-17T18:04:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.028286170999999999 2024-08-17T18:04:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.611639+0000", 2024-08-17T18:04:59.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7868000000000003e-05 2024-08-17T18:04:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:04:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.615716+0000", 2024-08-17T18:04:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040763889999999997 2024-08-17T18:04:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:04:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.615749+0000", 2024-08-17T18:04:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3045e-05 2024-08-17T18:04:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:04:59.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.617422+0000", 2024-08-17T18:04:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016733850000000001 2024-08-17T18:04:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104434 54.10 54:0a03e580:::benchmark_data_smithi136_273969_object104433:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:04:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.460552+0000", 2024-08-17T18:04:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 315.24156530800002, 2024-08-17T18:04:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.17111078, 2024-08-17T18:04:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:59.714 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:04:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:04:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:04:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:04:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104434 2024-08-17T18:04:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460552+0000", 2024-08-17T18:04:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460552+0000", 2024-08-17T18:04:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460553+0000", 2024-08-17T18:04:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.82e-07 2024-08-17T18:04:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460577+0000", 2024-08-17T18:04:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4414999999999999e-05 2024-08-17T18:04:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460584+0000", 2024-08-17T18:04:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7100000000000001e-06 2024-08-17T18:04:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460588+0000", 2024-08-17T18:04:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8449999999999996e-06 2024-08-17T18:04:59.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460647+0000", 2024-08-17T18:04:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9329000000000001e-05 2024-08-17T18:04:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:04:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460652+0000", 2024-08-17T18:04:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5499999999999996e-06 2024-08-17T18:04:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.462838+0000", 2024-08-17T18:04:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021861739999999999 2024-08-17T18:04:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:04:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.462843+0000", 2024-08-17T18:04:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0309999999999998e-06 2024-08-17T18:04:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.611677+0000", 2024-08-17T18:04:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.148834299 2024-08-17T18:04:59.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:04:59.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.611880+0000", 2024-08-17T18:04:59.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020264600000000001 2024-08-17T18:04:59.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:04:59.720 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:04:59.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.612660+0000", 2024-08-17T18:04:59.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00077971400000000004 2024-08-17T18:04:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.617632+0000", 2024-08-17T18:04:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0049720750000000003 2024-08-17T18:04:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:04:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.617649+0000", 2024-08-17T18:04:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7370000000000001e-05 2024-08-17T18:04:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.630089+0000", 2024-08-17T18:04:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012439725 2024-08-17T18:04:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:04:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.630128+0000", 2024-08-17T18:04:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9240999999999999e-05 2024-08-17T18:04:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:04:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.631663+0000", 2024-08-17T18:04:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001535174 2024-08-17T18:04:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:59.723 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104447 54.10 54:083450d8:::benchmark_data_smithi136_273969_object104446:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:04:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.460594+0000", 2024-08-17T18:04:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 315.24152335799999, 2024-08-17T18:04:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1723181149999999, 2024-08-17T18:04:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:04:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:04:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:04:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:04:59.724 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104447 2024-08-17T18:04:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460594+0000", 2024-08-17T18:04:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460594+0000", 2024-08-17T18:04:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460595+0000", 2024-08-17T18:04:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8100000000000003e-07 2024-08-17T18:04:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460631+0000", 2024-08-17T18:04:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6387999999999998e-05 2024-08-17T18:04:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460639+0000", 2024-08-17T18:04:59.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3130000000000001e-06 2024-08-17T18:04:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460644+0000", 2024-08-17T18:04:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6840000000000001e-06 2024-08-17T18:04:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460675+0000", 2024-08-17T18:04:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1220999999999998e-05 2024-08-17T18:04:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:04:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460680+0000", 2024-08-17T18:04:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9099999999999996e-06 2024-08-17T18:04:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.462873+0000", 2024-08-17T18:04:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021922389999999999 2024-08-17T18:04:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:04:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.462880+0000", 2024-08-17T18:04:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2259999999999999e-06 2024-08-17T18:04:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.613949+0000", 2024-08-17T18:04:59.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.151069353 2024-08-17T18:04:59.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:04:59.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.614143+0000", 2024-08-17T18:04:59.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019340500000000001 2024-08-17T18:04:59.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:04:59.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.614467+0000", 2024-08-17T18:04:59.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00032422700000000001 2024-08-17T18:04:59.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:04:59.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.617653+0000", 2024-08-17T18:04:59.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003186024 2024-08-17T18:04:59.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.629973+0000", 2024-08-17T18:04:59.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012319719 2024-08-17T18:04:59.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.631722+0000", 2024-08-17T18:04:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001749404 2024-08-17T18:04:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:04:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.631759+0000", 2024-08-17T18:04:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6616999999999999e-05 2024-08-17T18:04:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:04:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.632912+0000", 2024-08-17T18:04:59.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011539040000000001 2024-08-17T18:04:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104557 54.10 54:08b8b66a:::benchmark_data_smithi136_273969_object104556:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:04:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.460654+0000", 2024-08-17T18:04:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 315.24146338200001, 2024-08-17T18:04:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1798764369999999, 2024-08-17T18:04:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:04:59.732 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:04:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:04:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:04:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104557 2024-08-17T18:04:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460654+0000", 2024-08-17T18:04:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460654+0000", 2024-08-17T18:04:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460655+0000", 2024-08-17T18:04:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0900000000000002e-07 2024-08-17T18:04:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460687+0000", 2024-08-17T18:04:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1955999999999997e-05 2024-08-17T18:04:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460709+0000", 2024-08-17T18:04:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1759999999999998e-05 2024-08-17T18:04:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460714+0000", 2024-08-17T18:04:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8429999999999998e-06 2024-08-17T18:04:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460752+0000", 2024-08-17T18:04:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8022000000000003e-05 2024-08-17T18:04:59.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:04:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460757+0000", 2024-08-17T18:04:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8040000000000001e-06 2024-08-17T18:04:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.462914+0000", 2024-08-17T18:04:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021564689999999998 2024-08-17T18:04:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:04:59.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.462919+0000", 2024-08-17T18:04:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9660000000000004e-06 2024-08-17T18:04:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.617674+0000", 2024-08-17T18:04:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1547550950000001 2024-08-17T18:04:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:04:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.617834+0000", 2024-08-17T18:04:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000159798 2024-08-17T18:04:59.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:04:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.618189+0000", 2024-08-17T18:04:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00035543499999999997 2024-08-17T18:04:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:04:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.621993+0000", 2024-08-17T18:04:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003803716 2024-08-17T18:04:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.630024+0000", 2024-08-17T18:04:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0080315070000000002 2024-08-17T18:04:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.638621+0000", 2024-08-17T18:04:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.008596467 2024-08-17T18:04:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:04:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.638679+0000", 2024-08-17T18:04:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7896999999999998e-05 2024-08-17T18:04:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:04:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.640531+0000", 2024-08-17T18:04:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001852193 2024-08-17T18:04:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104561 54.10 54:080c6153:::benchmark_data_smithi136_273969_object104560:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:04:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.460728+0000", 2024-08-17T18:04:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 315.24138986399998, 2024-08-17T18:04:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.181197896, 2024-08-17T18:04:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:59.740 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:04:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:04:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:04:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:04:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104561 2024-08-17T18:04:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460728+0000", 2024-08-17T18:04:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460728+0000", 2024-08-17T18:04:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460728+0000", 2024-08-17T18:04:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8400000000000005e-07 2024-08-17T18:04:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460772+0000", 2024-08-17T18:04:59.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3789999999999999e-05 2024-08-17T18:04:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460782+0000", 2024-08-17T18:04:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5759999999999995e-06 2024-08-17T18:04:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460787+0000", 2024-08-17T18:04:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1830000000000002e-06 2024-08-17T18:04:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460831+0000", 2024-08-17T18:04:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4511000000000001e-05 2024-08-17T18:04:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:04:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460837+0000", 2024-08-17T18:04:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1340000000000003e-06 2024-08-17T18:04:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.462940+0000", 2024-08-17T18:04:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021034130000000002 2024-08-17T18:04:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:04:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.462944+0000", 2024-08-17T18:04:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7639999999999999e-06 2024-08-17T18:04:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.619595+0000", 2024-08-17T18:04:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1566512929999999 2024-08-17T18:04:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:04:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.619752+0000", 2024-08-17T18:04:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015728199999999999 2024-08-17T18:04:59.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:04:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.619997+0000", 2024-08-17T18:04:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024468899999999997 2024-08-17T18:04:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:04:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.625016+0000", 2024-08-17T18:04:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0050195350000000003 2024-08-17T18:04:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.636204+0000", 2024-08-17T18:04:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011187289 2024-08-17T18:04:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.640603+0000", 2024-08-17T18:04:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.004399699 2024-08-17T18:04:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:04:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.640669+0000", 2024-08-17T18:04:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5543999999999995e-05 2024-08-17T18:04:59.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:04:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.641926+0000", 2024-08-17T18:04:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012567100000000001 2024-08-17T18:04:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104584 54.10 54:0a7acb30:::benchmark_data_smithi136_273969_object104583:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:04:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.460796+0000", 2024-08-17T18:04:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 315.24132152999999, 2024-08-17T18:04:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1872988900000001, 2024-08-17T18:04:59.748 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:04:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:04:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:04:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:04:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104584 2024-08-17T18:04:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460796+0000", 2024-08-17T18:04:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460796+0000", 2024-08-17T18:04:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460797+0000", 2024-08-17T18:04:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.99e-07 2024-08-17T18:04:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460824+0000", 2024-08-17T18:04:59.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7625000000000001e-05 2024-08-17T18:04:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460832+0000", 2024-08-17T18:04:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4329999999999998e-06 2024-08-17T18:04:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460837+0000", 2024-08-17T18:04:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9869999999999999e-06 2024-08-17T18:04:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460883+0000", 2024-08-17T18:04:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6065e-05 2024-08-17T18:04:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:04:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460888+0000", 2024-08-17T18:04:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3730000000000001e-06 2024-08-17T18:04:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.462961+0000", 2024-08-17T18:04:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020730280000000002 2024-08-17T18:04:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:04:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.462965+0000", 2024-08-17T18:04:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5659999999999998e-06 2024-08-17T18:04:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.620849+0000", 2024-08-17T18:04:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1578842899999999 2024-08-17T18:04:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:04:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.620984+0000", 2024-08-17T18:04:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000134871 2024-08-17T18:04:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:04:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.621197+0000", 2024-08-17T18:04:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021278000000000001 2024-08-17T18:04:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:04:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.629901+0000", 2024-08-17T18:04:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0087040629999999997 2024-08-17T18:04:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.636245+0000", 2024-08-17T18:04:59.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063444590000000002 2024-08-17T18:04:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.646207+0000", 2024-08-17T18:04:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0099618659999999998 2024-08-17T18:04:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:04:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.646266+0000", 2024-08-17T18:04:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9225999999999998e-05 2024-08-17T18:04:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:04:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.648095+0000", 2024-08-17T18:04:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018286590000000001 2024-08-17T18:04:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104598 54.10 54:098f3f77:::benchmark_data_smithi136_273969_object104597:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:04:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.460846+0000", 2024-08-17T18:04:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 315.24127195199998, 2024-08-17T18:04:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1893474550000001, 2024-08-17T18:04:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:59.756 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:04:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:04:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:04:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:04:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104598 2024-08-17T18:04:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460846+0000", 2024-08-17T18:04:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460846+0000", 2024-08-17T18:04:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460846+0000", 2024-08-17T18:04:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4000000000000002e-07 2024-08-17T18:04:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460879+0000", 2024-08-17T18:04:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2716000000000003e-05 2024-08-17T18:04:59.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460886+0000", 2024-08-17T18:04:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3050000000000004e-06 2024-08-17T18:04:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460891+0000", 2024-08-17T18:04:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1150000000000002e-06 2024-08-17T18:04:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460943+0000", 2024-08-17T18:04:59.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2011000000000001e-05 2024-08-17T18:04:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:04:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460948+0000", 2024-08-17T18:04:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8790000000000002e-06 2024-08-17T18:04:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463002+0000", 2024-08-17T18:04:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020533389999999999 2024-08-17T18:04:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:04:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463009+0000", 2024-08-17T18:04:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5660000000000001e-06 2024-08-17T18:04:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.622040+0000", 2024-08-17T18:04:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.159031304 2024-08-17T18:04:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:04:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.622192+0000", 2024-08-17T18:04:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015120300000000001 2024-08-17T18:04:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:04:59.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.622441+0000", 2024-08-17T18:04:59.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024948 2024-08-17T18:04:59.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:04:59.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.629911+0000", 2024-08-17T18:04:59.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0074695719999999998 2024-08-17T18:04:59.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.638518+0000", 2024-08-17T18:04:59.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.008607356 2024-08-17T18:04:59.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.648168+0000", 2024-08-17T18:04:59.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0096494809999999997 2024-08-17T18:04:59.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:04:59.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.648286+0000", 2024-08-17T18:04:59.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118375 2024-08-17T18:04:59.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:04:59 smithi165 ceph-mon[105517]: pgmap v2592: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 48 MiB/s wr, 760 op/s; 25438/223221 objects degraded (11.396%); 9310/223221 objects misplaced (4.171%); 5.9 MiB/s, 94 objects/s recovering 2024-08-17T18:04:59.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:04:59.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.650193+0000", 2024-08-17T18:04:59.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0019073130000000001 2024-08-17T18:04:59.765 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.765 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:59.765 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.765 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104715 54.10 54:0b603cab:::benchmark_data_smithi136_273969_object104714:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:04:59.765 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.460911+0000", 2024-08-17T18:04:59.765 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 315.24120664700001, 2024-08-17T18:04:59.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.195364366, 2024-08-17T18:04:59.765 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:59.765 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:59.766 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:04:59.766 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:04:59.766 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:04:59.766 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:04:59.766 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104715 2024-08-17T18:04:59.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.766 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:59.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:59.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460911+0000", 2024-08-17T18:04:59.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:59.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460911+0000", 2024-08-17T18:04:59.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:59.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460912+0000", 2024-08-17T18:04:59.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1600000000000001e-07 2024-08-17T18:04:59.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:59.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460940+0000", 2024-08-17T18:04:59.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7946999999999999e-05 2024-08-17T18:04:59.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:59.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460946+0000", 2024-08-17T18:04:59.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9750000000000001e-06 2024-08-17T18:04:59.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:59.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460952+0000", 2024-08-17T18:04:59.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.135e-06 2024-08-17T18:04:59.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460983+0000", 2024-08-17T18:04:59.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1780999999999999e-05 2024-08-17T18:04:59.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:04:59.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460987+0000", 2024-08-17T18:04:59.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.67e-06 2024-08-17T18:04:59.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463044+0000", 2024-08-17T18:04:59.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020564450000000001 2024-08-17T18:04:59.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:04:59.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463050+0000", 2024-08-17T18:04:59.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4989999999999999e-06 2024-08-17T18:04:59.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.623583+0000", 2024-08-17T18:04:59.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1605328690000001 2024-08-17T18:04:59.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:04:59.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.623753+0000", 2024-08-17T18:04:59.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016986900000000001 2024-08-17T18:04:59.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:04:59.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.623997+0000", 2024-08-17T18:04:59.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000244233 2024-08-17T18:04:59.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:04:59.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.630035+0000", 2024-08-17T18:04:59.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00603828 2024-08-17T18:04:59.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.638568+0000", 2024-08-17T18:04:59.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0085331780000000006 2024-08-17T18:04:59.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.654197+0000", 2024-08-17T18:04:59.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015628764 2024-08-17T18:04:59.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:04:59.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.654267+0000", 2024-08-17T18:04:59.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9865999999999998e-05 2024-08-17T18:04:59.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:04:59.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.656275+0000", 2024-08-17T18:04:59.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002008339 2024-08-17T18:04:59.773 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.773 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:59.773 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.773 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104717 54.10 54:0f40c48e:::benchmark_data_smithi136_273969_object104716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:04:59.773 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.460961+0000", 2024-08-17T18:04:59.773 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 315.241157149, 2024-08-17T18:04:59.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1975942019999999, 2024-08-17T18:04:59.774 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:59.774 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:59.774 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:04:59.774 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:04:59.774 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:04:59.774 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:04:59.774 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104717 2024-08-17T18:04:59.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.774 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:59.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:59.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460961+0000", 2024-08-17T18:04:59.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:59.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460961+0000", 2024-08-17T18:04:59.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:59.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460961+0000", 2024-08-17T18:04:59.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7100000000000002e-07 2024-08-17T18:04:59.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:59.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460980+0000", 2024-08-17T18:04:59.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.861e-05 2024-08-17T18:04:59.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:59.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460985+0000", 2024-08-17T18:04:59.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7629999999999997e-06 2024-08-17T18:04:59.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:59.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460989+0000", 2024-08-17T18:04:59.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5760000000000002e-06 2024-08-17T18:04:59.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461015+0000", 2024-08-17T18:04:59.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5573999999999999e-05 2024-08-17T18:04:59.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:04:59.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461017+0000", 2024-08-17T18:04:59.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.537e-06 2024-08-17T18:04:59.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463076+0000", 2024-08-17T18:04:59.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002058926 2024-08-17T18:04:59.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:04:59.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463080+0000", 2024-08-17T18:04:59.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7409999999999998e-06 2024-08-17T18:04:59.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.625066+0000", 2024-08-17T18:04:59.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1619860850000001 2024-08-17T18:04:59.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:04:59.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.625197+0000", 2024-08-17T18:04:59.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000131514 2024-08-17T18:04:59.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.779 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:04:59.779 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:04:59.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:04:59.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.625353+0000", 2024-08-17T18:04:59.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015604799999999999 2024-08-17T18:04:59.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:04:59.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.630043+0000", 2024-08-17T18:04:59.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0046895219999999998 2024-08-17T18:04:59.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.644315+0000", 2024-08-17T18:04:59.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014271905 2024-08-17T18:04:59.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.656409+0000", 2024-08-17T18:04:59.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012094194000000001 2024-08-17T18:04:59.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:04:59.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.656471+0000", 2024-08-17T18:04:59.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1897999999999996e-05 2024-08-17T18:04:59.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:04:59.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.658555+0000", 2024-08-17T18:04:59.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020838380000000002 2024-08-17T18:04:59.782 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.782 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:59.782 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.782 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104721 54.10 54:09fb12db:::benchmark_data_smithi136_273969_object104720:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:04:59.782 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.460995+0000", 2024-08-17T18:04:59.783 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 315.24112251899999, 2024-08-17T18:04:59.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1996424779999999, 2024-08-17T18:04:59.783 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:59.783 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:59.783 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:04:59.783 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:04:59.783 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:04:59.783 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:04:59.783 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104721 2024-08-17T18:04:59.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.783 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:59.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:59.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460995+0000", 2024-08-17T18:04:59.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:59.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460995+0000", 2024-08-17T18:04:59.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:59.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.460996+0000", 2024-08-17T18:04:59.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7399999999999998e-07 2024-08-17T18:04:59.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:59.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461018+0000", 2024-08-17T18:04:59.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2667000000000002e-05 2024-08-17T18:04:59.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:59.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461023+0000", 2024-08-17T18:04:59.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8040000000000001e-06 2024-08-17T18:04:59.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:59.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461026+0000", 2024-08-17T18:04:59.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.914e-06 2024-08-17T18:04:59.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461056+0000", 2024-08-17T18:04:59.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0443999999999999e-05 2024-08-17T18:04:59.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:04:59.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461060+0000", 2024-08-17T18:04:59.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3780000000000001e-06 2024-08-17T18:04:59.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463107+0000", 2024-08-17T18:04:59.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020471180000000001 2024-08-17T18:04:59.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:04:59.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463113+0000", 2024-08-17T18:04:59.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0100000000000001e-06 2024-08-17T18:04:59.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.626172+0000", 2024-08-17T18:04:59.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.163058996 2024-08-17T18:04:59.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:04:59.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.626319+0000", 2024-08-17T18:04:59.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014732800000000001 2024-08-17T18:04:59.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:04:59.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.626554+0000", 2024-08-17T18:04:59.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023483 2024-08-17T18:04:59.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:04:59.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.630050+0000", 2024-08-17T18:04:59.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034959349999999999 2024-08-17T18:04:59.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.650268+0000", 2024-08-17T18:04:59.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020217681000000001 2024-08-17T18:04:59.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.658757+0000", 2024-08-17T18:04:59.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0084896079999999992 2024-08-17T18:04:59.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:04:59.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.658818+0000", 2024-08-17T18:04:59.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0832000000000003e-05 2024-08-17T18:04:59.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:04:59.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.660638+0000", 2024-08-17T18:04:59.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018194590000000001 2024-08-17T18:04:59.791 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.791 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:59.791 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.791 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104722 54.10 54:0b2c511b:::benchmark_data_smithi136_273969_object104721:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:04:59.791 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.461032+0000", 2024-08-17T18:04:59.791 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 315.24108523899997, 2024-08-17T18:04:59.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2024628589999999, 2024-08-17T18:04:59.791 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:59.791 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:59.791 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:04:59.791 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:04:59.792 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:04:59.792 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:04:59.792 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104722 2024-08-17T18:04:59.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.792 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:59.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:59.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461032+0000", 2024-08-17T18:04:59.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:59.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461032+0000", 2024-08-17T18:04:59.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:59.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461033+0000", 2024-08-17T18:04:59.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.51e-07 2024-08-17T18:04:59.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:59.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461052+0000", 2024-08-17T18:04:59.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9491999999999999e-05 2024-08-17T18:04:59.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:59.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461059+0000", 2024-08-17T18:04:59.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4799999999999998e-06 2024-08-17T18:04:59.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:59.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461064+0000", 2024-08-17T18:04:59.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7199999999999997e-06 2024-08-17T18:04:59.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461119+0000", 2024-08-17T18:04:59.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5528000000000001e-05 2024-08-17T18:04:59.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:04:59.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461122+0000", 2024-08-17T18:04:59.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5859999999999999e-06 2024-08-17T18:04:59.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463137+0000", 2024-08-17T18:04:59.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002015775 2024-08-17T18:04:59.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:04:59.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463141+0000", 2024-08-17T18:04:59.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4300000000000002e-06 2024-08-17T18:04:59.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.627394+0000", 2024-08-17T18:04:59.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1642535629999999 2024-08-17T18:04:59.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:04:59.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.627555+0000", 2024-08-17T18:04:59.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000160416 2024-08-17T18:04:59.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:04:59.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.627730+0000", 2024-08-17T18:04:59.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017476300000000001 2024-08-17T18:04:59.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:04:59.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.630057+0000", 2024-08-17T18:04:59.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023274659999999998 2024-08-17T18:04:59.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.650326+0000", 2024-08-17T18:04:59.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020268412999999999 2024-08-17T18:04:59.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.661579+0000", 2024-08-17T18:04:59.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011253275 2024-08-17T18:04:59.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:04:59.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.661631+0000", 2024-08-17T18:04:59.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2290000000000002e-05 2024-08-17T18:04:59.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:04:59.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.663495+0000", 2024-08-17T18:04:59.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018642109999999999 2024-08-17T18:04:59.799 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.799 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:59.799 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.799 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104725 54.10 54:0a958ad6:::benchmark_data_smithi136_273969_object104724:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:04:59.799 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.461101+0000", 2024-08-17T18:04:59.799 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 315.24101660999997, 2024-08-17T18:04:59.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2042328259999999, 2024-08-17T18:04:59.799 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:59.799 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:59.799 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:04:59.800 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:04:59.800 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:04:59.800 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:04:59.800 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104725 2024-08-17T18:04:59.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.800 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:59.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:59.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461101+0000", 2024-08-17T18:04:59.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:59.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461101+0000", 2024-08-17T18:04:59.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:59.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461102+0000", 2024-08-17T18:04:59.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9800000000000004e-07 2024-08-17T18:04:59.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:59.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461121+0000", 2024-08-17T18:04:59.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9491e-05 2024-08-17T18:04:59.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:59.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461128+0000", 2024-08-17T18:04:59.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2089999999999999e-06 2024-08-17T18:04:59.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:59.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461134+0000", 2024-08-17T18:04:59.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8209999999999998e-06 2024-08-17T18:04:59.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461160+0000", 2024-08-17T18:04:59.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5414000000000001e-05 2024-08-17T18:04:59.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:04:59.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461163+0000", 2024-08-17T18:04:59.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.219e-06 2024-08-17T18:04:59.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463168+0000", 2024-08-17T18:04:59.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020057539999999998 2024-08-17T18:04:59.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:04:59.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463175+0000", 2024-08-17T18:04:59.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2319999999999998e-06 2024-08-17T18:04:59.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.628473+0000", 2024-08-17T18:04:59.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1652981410000001 2024-08-17T18:04:59.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:04:59.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.628622+0000", 2024-08-17T18:04:59.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014901000000000001 2024-08-17T18:04:59.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:04:59.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.628863+0000", 2024-08-17T18:04:59.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024113300000000001 2024-08-17T18:04:59.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:04:59.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.632928+0000", 2024-08-17T18:04:59.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040654350000000001 2024-08-17T18:04:59.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.656352+0000", 2024-08-17T18:04:59.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023423247000000001 2024-08-17T18:04:59.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.663605+0000", 2024-08-17T18:04:59.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0072530839999999999 2024-08-17T18:04:59.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:04:59.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.663664+0000", 2024-08-17T18:04:59.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9172000000000002e-05 2024-08-17T18:04:59.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:04:59.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.665334+0000", 2024-08-17T18:04:59.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016699659999999999 2024-08-17T18:04:59.807 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.807 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:59.807 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.807 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104730 54.10 54:0f4303c7:::benchmark_data_smithi136_273969_object104729:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:04:59.807 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.461143+0000", 2024-08-17T18:04:59.807 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 315.24097494300003, 2024-08-17T18:04:59.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.207791611, 2024-08-17T18:04:59.808 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:59.808 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:59.808 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:04:59.808 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:04:59.808 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:04:59.808 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:04:59.808 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104730 2024-08-17T18:04:59.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.808 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:59.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:59.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461143+0000", 2024-08-17T18:04:59.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.810 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:59.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461143+0000", 2024-08-17T18:04:59.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.810 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:59.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461143+0000", 2024-08-17T18:04:59.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3700000000000001e-07 2024-08-17T18:04:59.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:59.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461162+0000", 2024-08-17T18:04:59.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8580000000000002e-05 2024-08-17T18:04:59.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:59.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461168+0000", 2024-08-17T18:04:59.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2770000000000002e-06 2024-08-17T18:04:59.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:59.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461173+0000", 2024-08-17T18:04:59.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9269999999999996e-06 2024-08-17T18:04:59.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461198+0000", 2024-08-17T18:04:59.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4581000000000001e-05 2024-08-17T18:04:59.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:04:59.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461200+0000", 2024-08-17T18:04:59.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6010000000000002e-06 2024-08-17T18:04:59.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463205+0000", 2024-08-17T18:04:59.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020046320000000001 2024-08-17T18:04:59.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:04:59.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463211+0000", 2024-08-17T18:04:59.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2890000000000003e-06 2024-08-17T18:04:59.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.632975+0000", 2024-08-17T18:04:59.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.169764158 2024-08-17T18:04:59.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:04:59.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.633138+0000", 2024-08-17T18:04:59.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000162607 2024-08-17T18:04:59.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:04:59.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.633455+0000", 2024-08-17T18:04:59.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00031745299999999998 2024-08-17T18:04:59.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:04:59.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.636257+0000", 2024-08-17T18:04:59.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0028021090000000001 2024-08-17T18:04:59.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.658654+0000", 2024-08-17T18:04:59.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.022396322 2024-08-17T18:04:59.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.667073+0000", 2024-08-17T18:04:59.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0084193099999999993 2024-08-17T18:04:59.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:04:59.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.667123+0000", 2024-08-17T18:04:59.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0209e-05 2024-08-17T18:04:59.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:04:59.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.668934+0000", 2024-08-17T18:04:59.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018111189999999999 2024-08-17T18:04:59.816 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.816 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:59.816 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.816 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104740 54.10 54:0bf65440:::benchmark_data_smithi136_273969_object104739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:04:59.816 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.461181+0000", 2024-08-17T18:04:59.816 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 315.24093672399999, 2024-08-17T18:04:59.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2141290300000001, 2024-08-17T18:04:59.816 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:59.817 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:59.817 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:04:59.817 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:04:59.817 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:04:59.817 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:04:59.817 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104740 2024-08-17T18:04:59.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.817 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:59.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:59.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461181+0000", 2024-08-17T18:04:59.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:59.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461181+0000", 2024-08-17T18:04:59.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:59.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461181+0000", 2024-08-17T18:04:59.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.46e-07 2024-08-17T18:04:59.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:59.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461200+0000", 2024-08-17T18:04:59.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8284999999999999e-05 2024-08-17T18:04:59.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:59.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461206+0000", 2024-08-17T18:04:59.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2299999999999996e-06 2024-08-17T18:04:59.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:59.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461211+0000", 2024-08-17T18:04:59.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0520000000000001e-06 2024-08-17T18:04:59.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461236+0000", 2024-08-17T18:04:59.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5131e-05 2024-08-17T18:04:59.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:04:59.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461239+0000", 2024-08-17T18:04:59.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8710000000000001e-06 2024-08-17T18:04:59.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463249+0000", 2024-08-17T18:04:59.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002010267 2024-08-17T18:04:59.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:04:59.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463257+0000", 2024-08-17T18:04:59.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7270000000000007e-06 2024-08-17T18:04:59.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.635018+0000", 2024-08-17T18:04:59.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1717606540000001 2024-08-17T18:04:59.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:04:59.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.635123+0000", 2024-08-17T18:04:59.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010581699999999999 2024-08-17T18:04:59.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:04:59.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.635324+0000", 2024-08-17T18:04:59.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000201036 2024-08-17T18:04:59.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:04:59.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.640550+0000", 2024-08-17T18:04:59.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0052251800000000003 2024-08-17T18:04:59.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.658714+0000", 2024-08-17T18:04:59.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018164235000000001 2024-08-17T18:04:59.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.673662+0000", 2024-08-17T18:04:59.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014948527 2024-08-17T18:04:59.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:04:59.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.673730+0000", 2024-08-17T18:04:59.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7206999999999995e-05 2024-08-17T18:04:59.824 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:04:59.824 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:04:59.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:04:59.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.675310+0000", 2024-08-17T18:04:59.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001580365 2024-08-17T18:04:59.825 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.825 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:59.825 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.825 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104765 54.10 54:0fe4eb9c:::benchmark_data_smithi136_273969_object104764:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:04:59.825 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.461219+0000", 2024-08-17T18:04:59.825 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 315.24089865399998, 2024-08-17T18:04:59.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2158349980000001, 2024-08-17T18:04:59.826 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:59.826 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:59.826 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:04:59.826 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:04:59.826 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:04:59.826 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:04:59.826 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104765 2024-08-17T18:04:59.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.826 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:59.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:59.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461219+0000", 2024-08-17T18:04:59.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:59.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461219+0000", 2024-08-17T18:04:59.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:59.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461219+0000", 2024-08-17T18:04:59.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1100000000000001e-07 2024-08-17T18:04:59.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:59.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461237+0000", 2024-08-17T18:04:59.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7504e-05 2024-08-17T18:04:59.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:59.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461243+0000", 2024-08-17T18:04:59.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9540000000000001e-06 2024-08-17T18:04:59.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:59.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461248+0000", 2024-08-17T18:04:59.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3920000000000002e-06 2024-08-17T18:04:59.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463290+0000", 2024-08-17T18:04:59.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020416470000000002 2024-08-17T18:04:59.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:04:59.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463296+0000", 2024-08-17T18:04:59.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4740000000000002e-06 2024-08-17T18:04:59.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.636283+0000", 2024-08-17T18:04:59.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1729864000000001 2024-08-17T18:04:59.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.830 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:04:59.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.636379+0000", 2024-08-17T18:04:59.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6189999999999999e-05 2024-08-17T18:04:59.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.830 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:04:59.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.636537+0000", 2024-08-17T18:04:59.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015755500000000001 2024-08-17T18:04:59.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.830 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:04:59.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.640558+0000", 2024-08-17T18:04:59.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040215609999999999 2024-08-17T18:04:59.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.661538+0000", 2024-08-17T18:04:59.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020980144999999999 2024-08-17T18:04:59.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.675377+0000", 2024-08-17T18:04:59.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013839179 2024-08-17T18:04:59.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:04:59.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.675428+0000", 2024-08-17T18:04:59.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0639000000000001e-05 2024-08-17T18:04:59.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:04:59.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.677054+0000", 2024-08-17T18:04:59.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001625947 2024-08-17T18:04:59.832 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.832 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:59.832 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.833 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:104808 54.10 54:0caf74eb:::benchmark_data_smithi136_273969_object104807:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1385)", 2024-08-17T18:04:59.833 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T17:59:44.461256+0000", 2024-08-17T18:04:59.833 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 315.24086150400001, 2024-08-17T18:04:59.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2175662680000001, 2024-08-17T18:04:59.833 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:59.833 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:59.833 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:04:59.833 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:04:59.833 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:04:59.833 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:04:59.834 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 104808 2024-08-17T18:04:59.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.834 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:59.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:59.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461256+0000", 2024-08-17T18:04:59.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:59.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461256+0000", 2024-08-17T18:04:59.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:59.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461257+0000", 2024-08-17T18:04:59.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5999999999999999e-07 2024-08-17T18:04:59.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:59.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461278+0000", 2024-08-17T18:04:59.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1223999999999999e-05 2024-08-17T18:04:59.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:59.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461284+0000", 2024-08-17T18:04:59.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5379999999999999e-06 2024-08-17T18:04:59.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:59.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.461290+0000", 2024-08-17T18:04:59.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7130000000000002e-06 2024-08-17T18:04:59.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463320+0000", 2024-08-17T18:04:59.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020298809999999999 2024-08-17T18:04:59.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:04:59.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:44.463326+0000", 2024-08-17T18:04:59.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4339999999999997e-06 2024-08-17T18:04:59.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.637362+0000", 2024-08-17T18:04:59.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.174035218 2024-08-17T18:04:59.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:04:59.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.637517+0000", 2024-08-17T18:04:59.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015547400000000001 2024-08-17T18:04:59.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:04:59.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.637667+0000", 2024-08-17T18:04:59.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014947699999999999 2024-08-17T18:04:59.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:04:59.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.642461+0000", 2024-08-17T18:04:59.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0047943869999999998 2024-08-17T18:04:59.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.663548+0000", 2024-08-17T18:04:59.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.021087257000000002 2024-08-17T18:04:59.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:04:59.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.677112+0000", 2024-08-17T18:04:59.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013563685000000001 2024-08-17T18:04:59.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:04:59.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.677175+0000", 2024-08-17T18:04:59.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3504999999999996e-05 2024-08-17T18:04:59.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:04:59.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T17:59:45.678822+0000", 2024-08-17T18:04:59.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001647115 2024-08-17T18:04:59.840 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.840 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:59.840 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.840 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:192150 54.12 54.cc328592 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1413)", 2024-08-17T18:04:59.840 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:01:44.638075+0000", 2024-08-17T18:04:59.840 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 195.06404232700001, 2024-08-17T18:04:59.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97759919100000003, 2024-08-17T18:04:59.840 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:59.840 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:59.841 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:04:59.841 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:04:59.841 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:04:59.841 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:04:59.841 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 192150 2024-08-17T18:04:59.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.841 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:59.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:59.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.638075+0000", 2024-08-17T18:04:59.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:59.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.638075+0000", 2024-08-17T18:04:59.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:59.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.638076+0000", 2024-08-17T18:04:59.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.71e-07 2024-08-17T18:04:59.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:59.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.638080+0000", 2024-08-17T18:04:59.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5880000000000002e-06 2024-08-17T18:04:59.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:59.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.638088+0000", 2024-08-17T18:04:59.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9389999999999997e-06 2024-08-17T18:04:59.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:59.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.638094+0000", 2024-08-17T18:04:59.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1090000000000003e-06 2024-08-17T18:04:59.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.638139+0000", 2024-08-17T18:04:59.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5617e-05 2024-08-17T18:04:59.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:04:59.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.638147+0000", 2024-08-17T18:04:59.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1960000000000002e-06 2024-08-17T18:04:59.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.615665+0000", 2024-08-17T18:04:59.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97751804900000006 2024-08-17T18:04:59.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:04:59.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.615675+0000", 2024-08-17T18:04:59.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8220000000000002e-06 2024-08-17T18:04:59.845 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.845 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:59.845 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.845 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:192163 54.12 54.1ae7c4f2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1413)", 2024-08-17T18:04:59.845 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:01:44.656303+0000", 2024-08-17T18:04:59.845 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 195.04581515300001, 2024-08-17T18:04:59.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.95941970899999995, 2024-08-17T18:04:59.845 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:59.845 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:59.846 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:04:59.846 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:04:59.846 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:04:59.846 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:04:59.846 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 192163 2024-08-17T18:04:59.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.846 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:59.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:59.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.656303+0000", 2024-08-17T18:04:59.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:59.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.656303+0000", 2024-08-17T18:04:59.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:59.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.656304+0000", 2024-08-17T18:04:59.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.344e-06 2024-08-17T18:04:59.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:59.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.656309+0000", 2024-08-17T18:04:59.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1309999999999997e-06 2024-08-17T18:04:59.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:59.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.656321+0000", 2024-08-17T18:04:59.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1812e-05 2024-08-17T18:04:59.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:59.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.656327+0000", 2024-08-17T18:04:59.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0260000000000004e-06 2024-08-17T18:04:59.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.656370+0000", 2024-08-17T18:04:59.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3486999999999999e-05 2024-08-17T18:04:59.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:04:59.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.656378+0000", 2024-08-17T18:04:59.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6299999999999998e-06 2024-08-17T18:04:59.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.615716+0000", 2024-08-17T18:04:59.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.95933753499999996 2024-08-17T18:04:59.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:04:59.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.615722+0000", 2024-08-17T18:04:59.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7440000000000001e-06 2024-08-17T18:04:59.850 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.850 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:59.850 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.850 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:192171 54.12 54.654dc572 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1413)", 2024-08-17T18:04:59.850 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:01:44.666722+0000", 2024-08-17T18:04:59.850 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 195.03539538999999, 2024-08-17T18:04:59.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.94976133699999998, 2024-08-17T18:04:59.850 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:59.850 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:59.851 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:04:59.851 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:04:59.851 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:04:59.851 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:04:59.851 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 192171 2024-08-17T18:04:59.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.851 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:59.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:59.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.666722+0000", 2024-08-17T18:04:59.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:59.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.666722+0000", 2024-08-17T18:04:59.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:59.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.666724+0000", 2024-08-17T18:04:59.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4580000000000001e-06 2024-08-17T18:04:59.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:59.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.666728+0000", 2024-08-17T18:04:59.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1150000000000004e-06 2024-08-17T18:04:59.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:59.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.666738+0000", 2024-08-17T18:04:59.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0009e-05 2024-08-17T18:04:59.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:59.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.666743+0000", 2024-08-17T18:04:59.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.271e-06 2024-08-17T18:04:59.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.666784+0000", 2024-08-17T18:04:59.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1316000000000002e-05 2024-08-17T18:04:59.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:04:59.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.666790+0000", 2024-08-17T18:04:59.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5419999999999996e-06 2024-08-17T18:04:59.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.616467+0000", 2024-08-17T18:04:59.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.94967747599999996 2024-08-17T18:04:59.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:04:59.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.616484+0000", 2024-08-17T18:04:59.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.615e-05 2024-08-17T18:04:59.856 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.856 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:59.856 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.856 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:192180 54.12 54.94eb8e72 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1413)", 2024-08-17T18:04:59.856 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:01:44.681882+0000", 2024-08-17T18:04:59.857 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 195.02023565799999, 2024-08-17T18:04:59.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93464022499999999, 2024-08-17T18:04:59.857 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:04:59.857 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:04:59.857 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:04:59.857 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:04:59.857 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:04:59.857 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:04:59.857 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 192180 2024-08-17T18:04:59.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.858 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:04:59.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:04:59.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.681882+0000", 2024-08-17T18:04:59.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:04:59.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.681882+0000", 2024-08-17T18:04:59.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:04:59.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:04:59.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.681883+0000", 2024-08-17T18:04:59.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0800000000000003e-07 2024-08-17T18:04:59.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:04:59.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.681887+0000", 2024-08-17T18:04:59.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4920000000000004e-06 2024-08-17T18:04:59.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:04:59.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.681895+0000", 2024-08-17T18:04:59.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0509999999999997e-06 2024-08-17T18:04:59.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:04:59.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.681900+0000", 2024-08-17T18:04:59.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5469999999999998e-06 2024-08-17T18:04:59.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.681940+0000", 2024-08-17T18:04:59.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0092999999999997e-05 2024-08-17T18:04:59.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:04:59.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:44.681946+0000", 2024-08-17T18:04:59.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.728e-06 2024-08-17T18:04:59.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:04:59.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.616513+0000", 2024-08-17T18:04:59.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93456702800000002 2024-08-17T18:04:59.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:04:59.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:04:59.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:04:59.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:45.616522+0000", 2024-08-17T18:04:59.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3780000000000007e-06 2024-08-17T18:04:59.862 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.862 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:59.862 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.862 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:04:59.862 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:04:59.862 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:04:59.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:04:59.863 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:04:59.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:04:59.964 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:00.005 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:00.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:00.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:00.107 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:00.135 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:00.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:00.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:00.170 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 65 16 77065 77049 74.0735 47.375 0.0345359 0.0134938 2024-08-17T18:05:00.237 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:00.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:00.269 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:00.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:00.370 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:00.410 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:00.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:00.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:00.512 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:00.532 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:00 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2279746627' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:05:00.532 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:00 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 25438/223221 objects degraded (11.396%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:00.533 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:00 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2279746627' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:05:00.533 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:00 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 25438/223221 objects degraded (11.396%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:00.553 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:00.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:00.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:00.655 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:00.687 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:00.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:00.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:00.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:00 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2279746627' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:05:00.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:00 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 25438/223221 objects degraded (11.396%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:00.788 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:00.835 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:00.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:00.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:00.894 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 0, 5] out_osds: [7] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:05:00.894 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:05:00.895 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_2 2024-08-17T18:05:00.895 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T18:05:00.937 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:00.988 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:00.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:00.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:01.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:01.117 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:01.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:01.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:01.170 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 66 16 77850 77834 73.6945 49.0625 0.0194233 0.0135616 2024-08-17T18:05:01.219 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:01.250 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:01.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:01.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:01.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:01.384 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:01.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:01.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:01.486 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:01.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:01.519 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:01.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:01.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:01.663 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:01.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:01.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:01 smithi136 ceph-mon[109861]: pgmap v2593: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 43 MiB/s wr, 694 op/s; 25438/223221 objects degraded (11.396%); 9310/223221 objects misplaced (4.171%); 5.4 MiB/s, 85 objects/s recovering 2024-08-17T18:05:01.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:01 smithi136 ceph-mon[115078]: pgmap v2593: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 43 MiB/s wr, 694 op/s; 25438/223221 objects degraded (11.396%); 9310/223221 objects misplaced (4.171%); 5.4 MiB/s, 85 objects/s recovering 2024-08-17T18:05:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:01 smithi165 ceph-mon[105517]: pgmap v2593: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 43 MiB/s wr, 694 op/s; 25438/223221 objects degraded (11.396%); 9310/223221 objects misplaced (4.171%); 5.4 MiB/s, 85 objects/s recovering 2024-08-17T18:05:01.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:01.814 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:01.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:01.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:01.916 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:01.953 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:01.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:01.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:02.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:02.093 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:02.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:02.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:02.170 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 67 16 78710 78694 73.3967 53.75 0.0137129 0.013617 2024-08-17T18:05:02.195 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:02.230 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:02.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:02.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:02.332 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:02.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:02.367 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:02.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:02.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:02.523 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:02.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:02.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:02.625 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:02.689 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:02.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:02.690 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-17T18:05:02.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:02.792 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:02.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:02.832 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:02.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:02.933 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:02.975 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:02.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:02.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:02.990 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:05:03.077 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:03.108 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:03.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:03.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:03.170 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 68 16 79501 79485 73.0444 49.4375 0.0126016 0.0136835 2024-08-17T18:05:03.210 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:03.242 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:03.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:03.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:03.344 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:03.389 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:03.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:03.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:03.436 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:03 smithi165 ceph-mon[105517]: pgmap v2594: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 59 MiB/s wr, 942 op/s; 25438/234354 objects degraded (10.855%); 8837/234354 objects misplaced (3.771%); 7.4 MiB/s, 117 objects/s recovering 2024-08-17T18:05:03.490 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:03.523 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:03 smithi136 ceph-mon[109861]: pgmap v2594: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 59 MiB/s wr, 942 op/s; 25438/234354 objects degraded (10.855%); 8837/234354 objects misplaced (3.771%); 7.4 MiB/s, 117 objects/s recovering 2024-08-17T18:05:03.524 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:03 smithi136 ceph-mon[115078]: pgmap v2594: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 59 MiB/s wr, 942 op/s; 25438/234354 objects degraded (10.855%); 8837/234354 objects misplaced (3.771%); 7.4 MiB/s, 117 objects/s recovering 2024-08-17T18:05:03.538 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:03.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:03.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:03.640 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:03.641 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:05:03.668 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:03.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:03.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:03.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:03.795 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:03.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:03.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:03.810 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:05:03.897 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:03.965 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:03.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:03.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:04.067 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:04.114 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:04.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:04.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:04.170 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 69 16 80241 80225 72.6558 46.25 0.0158279 0.0137567 2024-08-17T18:05:04.217 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:04.249 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:04.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:04.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:04.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:04.353 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:05:04.393 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:04.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:04.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:04.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:04 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3026187102' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:05:04.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:04 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3026187102' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:05:04.495 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:04.527 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:04.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:04.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:04.629 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:04.664 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:04.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:04.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:04.671 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:04 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3026187102' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:05:04.766 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:04.795 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:04.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:04.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:04.897 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:04.932 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:04.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:04.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:04.970 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:05:04.970 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:05:04.970 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-08-17T18:05:04.970 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:05:05.034 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:05.065 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:05.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:05.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:05.167 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:05.170 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 70 16 81050 81034 72.3402 50.5625 0.0591356 0.0138142 2024-08-17T18:05:05.208 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:05.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:05.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:05.310 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:05.358 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:05.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:05.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:05.460 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:05.504 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:05.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:05.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:05.606 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:05.646 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:05.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:05.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:05 smithi136 ceph-mon[109861]: pgmap v2595: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 43 MiB/s wr, 694 op/s; 25438/235098 objects degraded (10.820%); 8837/235098 objects misplaced (3.759%); 4.7 MiB/s, 74 objects/s recovering 2024-08-17T18:05:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:05 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:05:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:05 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3026187102' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:05:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:05 smithi136 ceph-mon[109861]: osdmap e1468: 8 total, 8 up, 7 in 2024-08-17T18:05:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:05 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 25438/235098 objects degraded (10.820%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:05 smithi136 ceph-mon[115078]: pgmap v2595: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 43 MiB/s wr, 694 op/s; 25438/235098 objects degraded (10.820%); 8837/235098 objects misplaced (3.759%); 4.7 MiB/s, 74 objects/s recovering 2024-08-17T18:05:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:05 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:05:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:05 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3026187102' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:05:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:05 smithi136 ceph-mon[115078]: osdmap e1468: 8 total, 8 up, 7 in 2024-08-17T18:05:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:05 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 25438/235098 objects degraded (10.820%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:05.747 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:05.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:05 smithi165 ceph-mon[105517]: pgmap v2595: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 43 MiB/s wr, 694 op/s; 25438/235098 objects degraded (10.820%); 8837/235098 objects misplaced (3.759%); 4.7 MiB/s, 74 objects/s recovering 2024-08-17T18:05:05.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:05 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:05:05.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:05 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3026187102' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:05:05.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:05 smithi165 ceph-mon[105517]: osdmap e1468: 8 total, 8 up, 7 in 2024-08-17T18:05:05.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:05 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 25438/235098 objects degraded (10.820%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:05.787 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:05.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:05.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:05.889 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:05.916 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:05.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:05.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:06.018 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:06.059 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:06.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:06.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:06.161 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:06.170 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 71 16 81879 81863 72.0509 51.8125 0.0185075 0.0138726 2024-08-17T18:05:06.204 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:06.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:06.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:06.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:06.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:06.342 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:06.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:06.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:06.471 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:06.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:06.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:06.573 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:06.596 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:06.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:06.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:06.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:05:06.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:05:06.698 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:06.721 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:06.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:06.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:05:06.823 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:06.856 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:06.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:06.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:06.958 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:06.980 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:06.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:06.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:07.082 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:07.116 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:07.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:07.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:07.171 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 72 16 82588 82572 71.6656 44.3125 0.00699303 0.0139471 2024-08-17T18:05:07.217 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:07.246 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:07.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:07.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:07.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:07.377 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:07.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:07.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:07.479 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:07.509 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:07.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:07.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:07.610 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:07.637 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:07.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:07.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:07.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:07 smithi165 ceph-mon[105517]: pgmap v2597: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 51 MiB/s wr, 814 op/s; 25438/235098 objects degraded (10.820%); 8837/235098 objects misplaced (3.759%); 5.6 MiB/s, 89 objects/s recovering 2024-08-17T18:05:07.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:07 smithi136 ceph-mon[109861]: pgmap v2597: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 51 MiB/s wr, 814 op/s; 25438/235098 objects degraded (10.820%); 8837/235098 objects misplaced (3.759%); 5.6 MiB/s, 89 objects/s recovering 2024-08-17T18:05:07.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:07 smithi136 ceph-mon[115078]: pgmap v2597: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 51 MiB/s wr, 814 op/s; 25438/235098 objects degraded (10.820%); 8837/235098 objects misplaced (3.759%); 5.6 MiB/s, 89 objects/s recovering 2024-08-17T18:05:07.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:05:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:05:07] "GET /metrics HTTP/1.1" 200 36639 "" "Prometheus/2.51.0" 2024-08-17T18:05:07.739 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:07.752 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-17T18:05:07.782 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:07.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:07.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:07.884 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:07.938 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:07.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:07.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:08.040 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:08.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:08.086 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:08.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:08.153 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:05:08.171 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 73 16 83377 83361 71.3593 49.3125 0.0342414 0.0140067 2024-08-17T18:05:08.187 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:08.221 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:08.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:08.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:08.325 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:08.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:08.360 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:08.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:08.461 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:08.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:08.498 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:08.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:08.569 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:05:08.600 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:08.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:08.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:08.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:08.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:08.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:08.788 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:08.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:08.889 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:08.918 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:08.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:08.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:09.020 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:09.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:09.063 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:09.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:09.164 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:09.171 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 74 16 84150 84134 71.0479 48.3125 0.0171031 0.014068 2024-08-17T18:05:09.195 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:09.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:09.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:09.297 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:09.348 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:09.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:09.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:09.450 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:09.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:09.479 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:09.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:09.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:09.607 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:09.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:09.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:09 smithi136 ceph-mon[109861]: pgmap v2598: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 50 MiB/s wr, 792 op/s; 25438/247005 objects degraded (10.299%); 8322/247005 objects misplaced (3.369%); 6.2 MiB/s, 98 objects/s recovering 2024-08-17T18:05:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:09 smithi136 ceph-mon[115078]: pgmap v2598: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 50 MiB/s wr, 792 op/s; 25438/247005 objects degraded (10.299%); 8322/247005 objects misplaced (3.369%); 6.2 MiB/s, 98 objects/s recovering 2024-08-17T18:05:09.709 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:09.733 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:09.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:09.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:09 smithi165 ceph-mon[105517]: pgmap v2598: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 50 MiB/s wr, 792 op/s; 25438/247005 objects degraded (10.299%); 8322/247005 objects misplaced (3.369%); 6.2 MiB/s, 98 objects/s recovering 2024-08-17T18:05:09.785 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:05:09.786 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T18:05:09.836 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:09.877 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:09.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:09.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:09.979 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:10.025 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:10.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:10.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:10.127 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:10.171 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 75 16 84872 84856 70.702 45.125 0.00621269 0.0141362 2024-08-17T18:05:10.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:10.172 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:10.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:10.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:10.307 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:10.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:10.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:10.404 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:05:10.408 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:10.434 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:10.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:10.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:10.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:10 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 25438/247005 objects degraded (10.299%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:10.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:10 smithi136 ceph-mon[109861]: from='client.45686 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:05:10.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:10 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 25438/247005 objects degraded (10.299%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:10.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:10 smithi136 ceph-mon[115078]: from='client.45686 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:05:10.536 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:10.564 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:10.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:10.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:10.571 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":2598,"stamp":"2024-08-17T18:05:08.225813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5394798565,"num_objects":82335,"num_object_clones":0,"num_object_copies":247005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25438,"num_objects_misplaced":8322,"num_objects_unfound":0,"num_objects_dirty":82335,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":82588,"num_write_kb":5270225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4852,"num_bytes_recovered":316285780,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82517,"ondisk_log_size":82517,"up":267,"acting":260,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":280,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":19667724,"kb_used_data":16156236,"kb_used_omap":350,"kb_used_meta":3511073,"kb_avail":636548340,"statfs":{"total":671965249536,"available":651825500160,"internally_reserved":0,"allocated":16543985664,"data_stored":16509139964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":358888,"internal_metadata":3595339288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,6,6,6,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":519569408,"num_objects":7928,"num_object_clones":0,"num_object_copies":23784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-988,"num_objects_unfound":0,"num_objects_dirty":7928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7928,"num_write_kb":507392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":988,"num_bytes_recovered":64749568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004324"},"pg_stats":[{"pgid":"55.f","version":"1468'5126","reported_seq":9782,"reported_epoch":1468,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:05:07.058204+0000","last_change":"2024-08-17T18:04:45.033662+0000","last_active":"2024-08-17T18:05:07.058204+0000","last_peered":"2024-08-17T18:05:07.058204+0000","last_clean":"2024-08-17T18:04:14.223726+0000","last_became_active":"2024-08-17T18:04:45.023531+0000","last_became_peered":"2024-08-17T18:04:45.023531+0000","last_unstale":"2024-08-17T18:05:07.058204+0000","last_undegraded":"2024-08-17T18:04:44.883222+0000","last_fullsized":"2024-08-17T18:04:44.762504+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5126,"log_dups_size":0,"ondisk_log_size":5126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:34:46.944210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335937536,"num_objects":5126,"num_object_clones":0,"num_object_copies":15378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5126,"num_write_kb":328064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2300,"num_bytes_recovered":150732800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[6,0],"avail_no_missing":["6","1","2"],"object_location_counts":[{"shards":"0,1,2,6","objects":2301},{"shards":"0,2,6","objects":1624},{"shards":"0,6","objects":1063},{"shards":"1,2,6","objects":138}],"blocked_by":[],"up_primary":5,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.e","version":"1468'5045","reported_seq":5100,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:06.508950+0000","last_change":"2024-08-17T18:03:40.237084+0000","last_active":"2024-08-17T18:05:06.508950+0000","last_peered":"2024-08-17T18:05:06.508950+0000","last_clean":"2024-08-17T18:05:06.508950+0000","last_became_active":"2024-08-17T18:03:40.236802+0000","last_became_peered":"2024-08-17T18:03:40.236802+0000","last_unstale":"2024-08-17T18:05:06.508950+0000","last_undegraded":"2024-08-17T18:05:06.508950+0000","last_fullsized":"2024-08-17T18:05:06.508950+0000","mapping_epoch":1446,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5045,"log_dups_size":0,"ondisk_log_size":5045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:20:34.770426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330629120,"num_objects":5045,"num_object_clones":0,"num_object_copies":15135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5045,"num_write_kb":322880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.d","version":"1468'5209","reported_seq":5255,"reported_epoch":1468,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:05:07.683604+0000","last_change":"2024-08-17T18:04:45.034492+0000","last_active":"2024-08-17T18:05:07.683604+0000","last_peered":"2024-08-17T18:05:07.683604+0000","last_clean":"2024-08-17T18:04:14.320037+0000","last_became_active":"2024-08-17T18:04:45.022484+0000","last_became_peered":"2024-08-17T18:04:45.022484+0000","last_unstale":"2024-08-17T18:05:07.683604+0000","last_undegraded":"2024-08-17T18:04:44.883245+0000","last_fullsized":"2024-08-17T18:04:44.763637+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5209,"log_dups_size":0,"ondisk_log_size":5209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:12:45.255851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341377024,"num_objects":5209,"num_object_clones":0,"num_object_copies":15627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6662,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5209,"num_write_kb":333376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0],"avail_no_missing":["4","3","6"],"object_location_counts":[{"shards":"0,3,4","objects":1604},{"shards":"0,4","objects":1076},{"shards":"3,4,6","objects":2529}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"55.c","version":"1468'5127","reported_seq":5187,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:06.515064+0000","last_change":"2024-08-17T18:03:51.280566+0000","last_active":"2024-08-17T18:05:06.515064+0000","last_peered":"2024-08-17T18:05:06.515064+0000","last_clean":"2024-08-17T18:05:06.515064+0000","last_became_active":"2024-08-17T18:03:51.278719+0000","last_became_peered":"2024-08-17T18:03:51.278719+0000","last_unstale":"2024-08-17T18:05:06.515064+0000","last_undegraded":"2024-08-17T18:05:06.515064+0000","last_fullsized":"2024-08-17T18:05:06.515064+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5127,"log_dups_size":0,"ondisk_log_size":5127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:48:22.670685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336003072,"num_objects":5127,"num_object_clones":0,"num_object_copies":15381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5127,"num_write_kb":328128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.b","version":"1468'5084","reported_seq":5124,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:07.050533+0000","last_change":"2024-08-17T18:03:51.272459+0000","last_active":"2024-08-17T18:05:07.050533+0000","last_peered":"2024-08-17T18:05:07.050533+0000","last_clean":"2024-08-17T18:05:07.050533+0000","last_became_active":"2024-08-17T18:03:51.272155+0000","last_became_peered":"2024-08-17T18:03:51.272155+0000","last_unstale":"2024-08-17T18:05:07.050533+0000","last_undegraded":"2024-08-17T18:05:07.050533+0000","last_fullsized":"2024-08-17T18:05:07.050533+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5084,"log_dups_size":0,"ondisk_log_size":5084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:17:38.013046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333185024,"num_objects":5084,"num_object_clones":0,"num_object_copies":15252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5084,"num_write_kb":325376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.a","version":"1468'5063","reported_seq":5124,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:06.463460+0000","last_change":"2024-08-17T18:03:51.276612+0000","last_active":"2024-08-17T18:05:06.463460+0000","last_peered":"2024-08-17T18:05:06.463460+0000","last_clean":"2024-08-17T18:05:06.463460+0000","last_became_active":"2024-08-17T18:03:51.276295+0000","last_became_peered":"2024-08-17T18:03:51.276295+0000","last_unstale":"2024-08-17T18:05:06.463460+0000","last_undegraded":"2024-08-17T18:05:06.463460+0000","last_fullsized":"2024-08-17T18:05:06.463460+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5063,"log_dups_size":0,"ondisk_log_size":5063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:13.305911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331808768,"num_objects":5063,"num_object_clones":0,"num_object_copies":15189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5063,"num_write_kb":324032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.9","version":"1468'5093","reported_seq":5151,"reported_epoch":1468,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:05:06.494658+0000","last_change":"2024-08-17T18:04:45.035995+0000","last_active":"2024-08-17T18:05:06.494658+0000","last_peered":"2024-08-17T18:05:06.494658+0000","last_clean":"2024-08-17T18:04:42.666472+0000","last_became_active":"2024-08-17T18:04:45.022949+0000","last_became_peered":"2024-08-17T18:04:45.022949+0000","last_unstale":"2024-08-17T18:05:06.494658+0000","last_undegraded":"2024-08-17T18:04:44.786308+0000","last_fullsized":"2024-08-17T18:04:44.764956+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5093,"log_dups_size":0,"ondisk_log_size":5093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333774848,"num_objects":5093,"num_object_clones":0,"num_object_copies":15279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5093,"num_write_kb":325952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,2],"avail_no_missing":["3","2","4"],"object_location_counts":[{"shards":"2,3","objects":1103},{"shards":"2,3,4","objects":3990}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.8","version":"1468'5148","reported_seq":5202,"reported_epoch":1468,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:05:06.670189+0000","last_change":"2024-08-17T18:04:44.983210+0000","last_active":"2024-08-17T18:05:06.670189+0000","last_peered":"2024-08-17T18:05:06.670189+0000","last_clean":"2024-08-17T18:04:14.322941+0000","last_became_active":"2024-08-17T18:04:44.978630+0000","last_became_peered":"2024-08-17T18:04:44.978630+0000","last_unstale":"2024-08-17T18:05:06.670189+0000","last_undegraded":"2024-08-17T18:04:44.881409+0000","last_fullsized":"2024-08-17T18:05:06.670189+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337379328,"num_objects":5148,"num_object_clones":0,"num_object_copies":15444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2470,"num_objects_misplaced":4074,"num_objects_unfound":0,"num_objects_dirty":5148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5148,"num_write_kb":329472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[2,0,6],"avail_no_missing":["2","3","6"],"object_location_counts":[{"shards":"0,2,6","objects":1074},{"shards":"2,3,6","objects":2470}],"blocked_by":[],"up_primary":5,"acting_primary":2,"purged_snaps":[]},{"pgid":"55.7","version":"1468'5187","reported_seq":5236,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:06.662336+0000","last_change":"2024-08-17T18:03:51.270104+0000","last_active":"2024-08-17T18:05:06.662336+0000","last_peered":"2024-08-17T18:05:06.662336+0000","last_clean":"2024-08-17T18:05:06.662336+0000","last_became_active":"2024-08-17T18:03:51.269273+0000","last_became_peered":"2024-08-17T18:03:51.269273+0000","last_unstale":"2024-08-17T18:05:06.662336+0000","last_undegraded":"2024-08-17T18:05:06.662336+0000","last_fullsized":"2024-08-17T18:05:06.662336+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5187,"log_dups_size":0,"ondisk_log_size":5187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:24:22.958394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339935232,"num_objects":5187,"num_object_clones":0,"num_object_copies":15561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5187,"num_write_kb":331968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.6","version":"1468'5166","reported_seq":5230,"reported_epoch":1468,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:05:06.675025+0000","last_change":"2024-08-17T18:04:45.018098+0000","last_active":"2024-08-17T18:05:06.675025+0000","last_peered":"2024-08-17T18:05:06.675025+0000","last_clean":"2024-08-17T18:04:42.635203+0000","last_became_active":"2024-08-17T18:04:45.007335+0000","last_became_peered":"2024-08-17T18:04:45.007335+0000","last_unstale":"2024-08-17T18:05:06.675025+0000","last_undegraded":"2024-08-17T18:04:44.790789+0000","last_fullsized":"2024-08-17T18:04:44.763542+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5166,"log_dups_size":0,"ondisk_log_size":5166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338558976,"num_objects":5166,"num_object_clones":0,"num_object_copies":15498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5166,"num_write_kb":330624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,3],"avail_no_missing":["2","1","3"],"object_location_counts":[{"shards":"1,2,3","objects":4099},{"shards":"2,3","objects":1067}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"55.5","version":"1468'5226","reported_seq":9328,"reported_epoch":1468,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-17T18:05:07.057172+0000","last_change":"2024-08-17T18:04:43.827565+0000","last_active":"2024-08-17T18:05:07.057172+0000","last_peered":"2024-08-17T18:05:07.057172+0000","last_clean":"2024-08-17T18:04:14.320565+0000","last_became_active":"2024-08-17T18:04:15.959091+0000","last_became_peered":"2024-08-17T18:04:15.959091+0000","last_unstale":"2024-08-17T18:05:07.057172+0000","last_undegraded":"2024-08-17T18:05:07.057172+0000","last_fullsized":"2024-08-17T18:04:15.518054+0000","mapping_epoch":1458,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5226,"log_dups_size":0,"ondisk_log_size":5226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:53:10.870728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342491136,"num_objects":5226,"num_object_clones":0,"num_object_copies":15678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":515,"num_objects_unfound":0,"num_objects_dirty":5226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5226,"num_write_kb":334464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":132775936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,1],"avail_no_missing":["6","1","3"],"object_location_counts":[{"shards":"1,3,6","objects":515},{"shards":"1,6","objects":4711}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.4","version":"1468'5209","reported_seq":5256,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:06.678173+0000","last_change":"2024-08-17T18:03:51.272772+0000","last_active":"2024-08-17T18:05:06.678173+0000","last_peered":"2024-08-17T18:05:06.678173+0000","last_clean":"2024-08-17T18:05:06.678173+0000","last_became_active":"2024-08-17T18:03:51.272464+0000","last_became_peered":"2024-08-17T18:03:51.272464+0000","last_unstale":"2024-08-17T18:05:06.678173+0000","last_undegraded":"2024-08-17T18:05:06.678173+0000","last_fullsized":"2024-08-17T18:05:06.678173+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5209,"log_dups_size":0,"ondisk_log_size":5209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:32.419414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341377024,"num_objects":5209,"num_object_clones":0,"num_object_copies":15627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5209,"num_write_kb":333376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"55.3","version":"1468'5126","reported_seq":5193,"reported_epoch":1468,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:05:06.661258+0000","last_change":"2024-08-17T18:04:43.918567+0000","last_active":"2024-08-17T18:05:06.661258+0000","last_peered":"2024-08-17T18:05:06.661258+0000","last_clean":"2024-08-17T18:04:14.321026+0000","last_became_active":"2024-08-17T18:04:43.910305+0000","last_became_peered":"2024-08-17T18:04:43.910305+0000","last_unstale":"2024-08-17T18:05:06.661258+0000","last_undegraded":"2024-08-17T18:04:43.799287+0000","last_fullsized":"2024-08-17T18:04:43.767830+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5126,"log_dups_size":0,"ondisk_log_size":5126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335937536,"num_objects":5126,"num_object_clones":0,"num_object_copies":15378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5126,"num_write_kb":328064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,3],"avail_no_missing":["1","3","6"],"object_location_counts":[{"shards":"0,1,3","objects":1574},{"shards":"1,3","objects":1110},{"shards":"1,3,6","objects":2442}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.2","version":"1468'5218","reported_seq":5273,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:06.577076+0000","last_change":"2024-08-17T18:03:40.236296+0000","last_active":"2024-08-17T18:05:06.577076+0000","last_peered":"2024-08-17T18:05:06.577076+0000","last_clean":"2024-08-17T18:05:06.577076+0000","last_became_active":"2024-08-17T18:03:40.235983+0000","last_became_peered":"2024-08-17T18:03:40.235983+0000","last_unstale":"2024-08-17T18:05:06.577076+0000","last_undegraded":"2024-08-17T18:05:06.577076+0000","last_fullsized":"2024-08-17T18:05:06.577076+0000","mapping_epoch":1446,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5218,"log_dups_size":0,"ondisk_log_size":5218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:04:22.402922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341966848,"num_objects":5218,"num_object_clones":0,"num_object_copies":15654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5218,"num_write_kb":333952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.1","version":"1468'5239","reported_seq":6227,"reported_epoch":1468,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-17T18:05:06.534592+0000","last_change":"2024-08-17T18:04:43.930436+0000","last_active":"2024-08-17T18:05:06.534592+0000","last_peered":"2024-08-17T18:05:06.534592+0000","last_clean":"2024-08-17T18:04:14.334099+0000","last_became_active":"2024-08-17T18:04:43.906419+0000","last_became_peered":"2024-08-17T18:04:43.906419+0000","last_unstale":"2024-08-17T18:05:06.534592+0000","last_undegraded":"2024-08-17T18:05:06.534592+0000","last_fullsized":"2024-08-17T18:04:43.761824+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5239,"log_dups_size":0,"ondisk_log_size":5239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343343104,"num_objects":5239,"num_object_clones":0,"num_object_copies":15717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3733,"num_objects_unfound":0,"num_objects_dirty":5239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5239,"num_write_kb":335296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":462,"num_bytes_recovered":30277632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4],"avail_no_missing":["3","2","4"],"object_location_counts":[{"shards":"0,3,4","objects":1686},{"shards":"2,3,4","objects":2047},{"shards":"3,4","objects":1506}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.0","version":"1468'5043","reported_seq":5104,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:06.468251+0000","last_change":"2024-08-17T18:03:51.276703+0000","last_active":"2024-08-17T18:05:06.468251+0000","last_peered":"2024-08-17T18:05:06.468251+0000","last_clean":"2024-08-17T18:05:06.468251+0000","last_became_active":"2024-08-17T18:03:51.276392+0000","last_became_peered":"2024-08-17T18:03:51.276392+0000","last_unstale":"2024-08-17T18:05:06.468251+0000","last_undegraded":"2024-08-17T18:05:06.468251+0000","last_fullsized":"2024-08-17T18:05:06.468251+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5043,"log_dups_size":0,"ondisk_log_size":5043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:27:57.384279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330498048,"num_objects":5043,"num_object_clones":0,"num_object_copies":15129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5043,"num_write_kb":322752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":51,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.378314+0000","last_change":"2024-08-17T18:04:43.798908+0000","last_active":"2024-08-17T18:05:04.378314+0000","last_peered":"2024-08-17T18:05:04.378314+0000","last_clean":"2024-08-17T18:05:04.378314+0000","last_became_active":"2024-08-17T18:04:43.779951+0000","last_became_peered":"2024-08-17T18:04:43.779951+0000","last_unstale":"2024-08-17T18:05:04.378314+0000","last_undegraded":"2024-08-17T18:05:04.378314+0000","last_fullsized":"2024-08-17T18:05:04.378314+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:40.663566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1442,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.283889+0000","last_change":"2024-08-17T17:23:07.398676+0000","last_active":"2024-08-17T18:04:49.283889+0000","last_peered":"2024-08-17T18:04:49.283889+0000","last_clean":"2024-08-17T18:04:49.283889+0000","last_became_active":"2024-08-17T17:23:07.398369+0000","last_became_peered":"2024-08-17T17:23:07.398369+0000","last_unstale":"2024-08-17T18:04:49.283889+0000","last_undegraded":"2024-08-17T18:04:49.283889+0000","last_fullsized":"2024-08-17T18:04:49.283889+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:55:04.281412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":2239,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.359398+0000","last_change":"2024-08-17T18:01:23.501829+0000","last_active":"2024-08-17T18:05:04.359398+0000","last_peered":"2024-08-17T18:05:04.359398+0000","last_clean":"2024-08-17T18:05:04.359398+0000","last_became_active":"2024-08-17T18:01:23.501392+0000","last_became_peered":"2024-08-17T18:01:23.501392+0000","last_unstale":"2024-08-17T18:05:04.359398+0000","last_undegraded":"2024-08-17T18:05:04.359398+0000","last_fullsized":"2024-08-17T18:05:04.359398+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:10:37.662647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2275,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.387815+0000","last_change":"2024-08-17T18:04:14.540774+0000","last_active":"2024-08-17T18:05:04.387815+0000","last_peered":"2024-08-17T18:05:04.387815+0000","last_clean":"2024-08-17T18:05:04.387815+0000","last_became_active":"2024-08-17T18:04:14.539778+0000","last_became_peered":"2024-08-17T18:04:14.539778+0000","last_unstale":"2024-08-17T18:05:04.387815+0000","last_undegraded":"2024-08-17T18:05:04.387815+0000","last_fullsized":"2024-08-17T18:05:04.387815+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:55:02.863360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":18,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.293147+0000","last_change":"2024-08-17T18:04:43.777705+0000","last_active":"2024-08-17T18:04:49.293147+0000","last_peered":"2024-08-17T18:04:49.293147+0000","last_clean":"2024-08-17T18:04:49.293147+0000","last_became_active":"2024-08-17T18:04:43.771283+0000","last_became_peered":"2024-08-17T18:04:43.771283+0000","last_unstale":"2024-08-17T18:04:49.293147+0000","last_undegraded":"2024-08-17T18:04:49.293147+0000","last_fullsized":"2024-08-17T18:04:49.293147+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:13.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1520,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.361306+0000","last_change":"2024-08-17T18:04:14.534963+0000","last_active":"2024-08-17T18:05:04.361306+0000","last_peered":"2024-08-17T18:05:04.361306+0000","last_clean":"2024-08-17T18:05:04.361306+0000","last_became_active":"2024-08-17T18:04:14.534634+0000","last_became_peered":"2024-08-17T18:04:14.534634+0000","last_unstale":"2024-08-17T18:05:04.361306+0000","last_undegraded":"2024-08-17T18:05:04.361306+0000","last_fullsized":"2024-08-17T18:05:04.361306+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:33:27.287860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":18,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.294473+0000","last_change":"2024-08-17T18:04:43.773534+0000","last_active":"2024-08-17T18:04:49.294473+0000","last_peered":"2024-08-17T18:04:49.294473+0000","last_clean":"2024-08-17T18:04:49.294473+0000","last_became_active":"2024-08-17T18:04:43.768507+0000","last_became_peered":"2024-08-17T18:04:43.768507+0000","last_unstale":"2024-08-17T18:04:49.294473+0000","last_undegraded":"2024-08-17T18:04:49.294473+0000","last_fullsized":"2024-08-17T18:04:49.294473+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:50:57.647351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1456,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.283927+0000","last_change":"2024-08-17T17:24:17.458168+0000","last_active":"2024-08-17T18:04:49.283927+0000","last_peered":"2024-08-17T18:04:49.283927+0000","last_clean":"2024-08-17T18:04:49.283927+0000","last_became_active":"2024-08-17T17:24:17.457847+0000","last_became_peered":"2024-08-17T17:24:17.457847+0000","last_unstale":"2024-08-17T18:04:49.283927+0000","last_undegraded":"2024-08-17T18:04:49.283927+0000","last_fullsized":"2024-08-17T18:04:49.283927+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:28.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":32,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.375860+0000","last_change":"2024-08-17T18:04:14.531445+0000","last_active":"2024-08-17T18:05:04.375860+0000","last_peered":"2024-08-17T18:05:04.375860+0000","last_clean":"2024-08-17T18:05:04.375860+0000","last_became_active":"2024-08-17T18:04:14.531099+0000","last_became_peered":"2024-08-17T18:04:14.531099+0000","last_unstale":"2024-08-17T18:05:04.375860+0000","last_undegraded":"2024-08-17T18:05:04.375860+0000","last_fullsized":"2024-08-17T18:05:04.375860+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:43.335123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":2228,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.366043+0000","last_change":"2024-08-17T18:01:23.493148+0000","last_active":"2024-08-17T18:05:04.366043+0000","last_peered":"2024-08-17T18:05:04.366043+0000","last_clean":"2024-08-17T18:05:04.366043+0000","last_became_active":"2024-08-17T18:01:23.491909+0000","last_became_peered":"2024-08-17T18:01:23.491909+0000","last_unstale":"2024-08-17T18:05:04.366043+0000","last_undegraded":"2024-08-17T18:05:04.366043+0000","last_fullsized":"2024-08-17T18:05:04.366043+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:01:40.187357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1474,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:05:02.728471+0000","last_change":"2024-08-17T18:01:23.519890+0000","last_active":"2024-08-17T18:05:02.728471+0000","last_peered":"2024-08-17T18:05:02.728471+0000","last_clean":"2024-08-17T18:05:02.728471+0000","last_became_active":"2024-08-17T18:01:23.519526+0000","last_became_peered":"2024-08-17T18:01:23.519526+0000","last_unstale":"2024-08-17T18:05:02.728471+0000","last_undegraded":"2024-08-17T18:05:02.728471+0000","last_fullsized":"2024-08-17T18:05:02.728471+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:04:52.662528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2241,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.360275+0000","last_change":"2024-08-17T18:01:23.491836+0000","last_active":"2024-08-17T18:05:04.360275+0000","last_peered":"2024-08-17T18:05:04.360275+0000","last_clean":"2024-08-17T18:05:04.360275+0000","last_became_active":"2024-08-17T18:01:23.491209+0000","last_became_peered":"2024-08-17T18:01:23.491209+0000","last_unstale":"2024-08-17T18:05:04.360275+0000","last_undegraded":"2024-08-17T18:05:04.360275+0000","last_fullsized":"2024-08-17T18:05:04.360275+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:35:30.767005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2243,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.369603+0000","last_change":"2024-08-17T17:24:17.452025+0000","last_active":"2024-08-17T18:05:04.369603+0000","last_peered":"2024-08-17T18:05:04.369603+0000","last_clean":"2024-08-17T18:05:04.369603+0000","last_became_active":"2024-08-17T17:24:17.448029+0000","last_became_peered":"2024-08-17T17:24:17.448029+0000","last_unstale":"2024-08-17T18:05:04.369603+0000","last_undegraded":"2024-08-17T18:05:04.369603+0000","last_fullsized":"2024-08-17T18:05:04.369603+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:45.089346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":32,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.377713+0000","last_change":"2024-08-17T18:04:14.541444+0000","last_active":"2024-08-17T18:05:04.377713+0000","last_peered":"2024-08-17T18:05:04.377713+0000","last_clean":"2024-08-17T18:05:04.377713+0000","last_became_active":"2024-08-17T18:04:14.541074+0000","last_became_peered":"2024-08-17T18:04:14.541074+0000","last_unstale":"2024-08-17T18:05:04.377713+0000","last_undegraded":"2024-08-17T18:05:04.377713+0000","last_fullsized":"2024-08-17T18:05:04.377713+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:04:04.979761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2255,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.359207+0000","last_change":"2024-08-17T18:04:14.539660+0000","last_active":"2024-08-17T18:05:04.359207+0000","last_peered":"2024-08-17T18:05:04.359207+0000","last_clean":"2024-08-17T18:05:04.359207+0000","last_became_active":"2024-08-17T18:04:14.537852+0000","last_became_peered":"2024-08-17T18:04:14.537852+0000","last_unstale":"2024-08-17T18:05:04.359207+0000","last_undegraded":"2024-08-17T18:05:04.359207+0000","last_fullsized":"2024-08-17T18:05:04.359207+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:47:45.026997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1470,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.285222+0000","last_change":"2024-08-17T18:04:43.769892+0000","last_active":"2024-08-17T18:04:49.285222+0000","last_peered":"2024-08-17T18:04:49.285222+0000","last_clean":"2024-08-17T18:04:49.285222+0000","last_became_active":"2024-08-17T18:04:43.769702+0000","last_became_peered":"2024-08-17T18:04:43.769702+0000","last_unstale":"2024-08-17T18:04:49.285222+0000","last_undegraded":"2024-08-17T18:04:49.285222+0000","last_fullsized":"2024-08-17T18:04:49.285222+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:39.373904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2215,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.359435+0000","last_change":"2024-08-17T17:24:17.442338+0000","last_active":"2024-08-17T18:05:04.359435+0000","last_peered":"2024-08-17T18:05:04.359435+0000","last_clean":"2024-08-17T18:05:04.359435+0000","last_became_active":"2024-08-17T17:24:17.441948+0000","last_became_peered":"2024-08-17T17:24:17.441948+0000","last_unstale":"2024-08-17T18:05:04.359435+0000","last_undegraded":"2024-08-17T18:05:04.359435+0000","last_fullsized":"2024-08-17T18:05:04.359435+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:17.891492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":39,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.378407+0000","last_change":"2024-08-17T18:04:43.777736+0000","last_active":"2024-08-17T18:05:04.378407+0000","last_peered":"2024-08-17T18:05:04.378407+0000","last_clean":"2024-08-17T18:05:04.378407+0000","last_became_active":"2024-08-17T18:04:43.777438+0000","last_became_peered":"2024-08-17T18:04:43.777438+0000","last_unstale":"2024-08-17T18:05:04.378407+0000","last_undegraded":"2024-08-17T18:05:04.378407+0000","last_fullsized":"2024-08-17T18:05:04.378407+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:24:01.255927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2195,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.388397+0000","last_change":"2024-08-17T18:04:43.915398+0000","last_active":"2024-08-17T18:05:04.388397+0000","last_peered":"2024-08-17T18:05:04.388397+0000","last_clean":"2024-08-17T18:05:04.388397+0000","last_became_active":"2024-08-17T18:04:43.915212+0000","last_became_peered":"2024-08-17T18:04:43.915212+0000","last_unstale":"2024-08-17T18:05:04.388397+0000","last_undegraded":"2024-08-17T18:05:04.388397+0000","last_fullsized":"2024-08-17T18:05:04.388397+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:43:26.901830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.294111+0000","last_change":"2024-08-17T18:04:43.773363+0000","last_active":"2024-08-17T18:04:49.294111+0000","last_peered":"2024-08-17T18:04:49.294111+0000","last_clean":"2024-08-17T18:04:49.294111+0000","last_became_active":"2024-08-17T18:04:43.773005+0000","last_became_peered":"2024-08-17T18:04:43.773005+0000","last_unstale":"2024-08-17T18:04:49.294111+0000","last_undegraded":"2024-08-17T18:04:49.294111+0000","last_fullsized":"2024-08-17T18:04:49.294111+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:03:13.813096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":352,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.360362+0000","last_change":"2024-08-17T18:04:14.555729+0000","last_active":"2024-08-17T18:05:04.360362+0000","last_peered":"2024-08-17T18:05:04.360362+0000","last_clean":"2024-08-17T18:05:04.360362+0000","last_became_active":"2024-08-17T18:04:14.536678+0000","last_became_peered":"2024-08-17T18:04:14.536678+0000","last_unstale":"2024-08-17T18:05:04.360362+0000","last_undegraded":"2024-08-17T18:05:04.360362+0000","last_fullsized":"2024-08-17T18:05:04.360362+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:47:29.424778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2253,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.371461+0000","last_change":"2024-08-17T18:01:23.494237+0000","last_active":"2024-08-17T18:05:04.371461+0000","last_peered":"2024-08-17T18:05:04.371461+0000","last_clean":"2024-08-17T18:05:04.371461+0000","last_became_active":"2024-08-17T18:01:23.493978+0000","last_became_peered":"2024-08-17T18:01:23.493978+0000","last_unstale":"2024-08-17T18:05:04.371461+0000","last_undegraded":"2024-08-17T18:05:04.371461+0000","last_fullsized":"2024-08-17T18:05:04.371461+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:05:41.724192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1471,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.284014+0000","last_change":"2024-08-17T18:03:51.292044+0000","last_active":"2024-08-17T18:04:49.284014+0000","last_peered":"2024-08-17T18:04:49.284014+0000","last_clean":"2024-08-17T18:04:49.284014+0000","last_became_active":"2024-08-17T18:03:51.268271+0000","last_became_peered":"2024-08-17T18:03:51.268271+0000","last_unstale":"2024-08-17T18:04:49.284014+0000","last_undegraded":"2024-08-17T18:04:49.284014+0000","last_fullsized":"2024-08-17T18:04:49.284014+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:00.892410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4756,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1443,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.284031+0000","last_change":"2024-08-17T17:23:07.455919+0000","last_active":"2024-08-17T18:04:49.284031+0000","last_peered":"2024-08-17T18:04:49.284031+0000","last_clean":"2024-08-17T18:04:49.284031+0000","last_became_active":"2024-08-17T17:23:07.455284+0000","last_became_peered":"2024-08-17T17:23:07.455284+0000","last_unstale":"2024-08-17T18:04:49.284031+0000","last_undegraded":"2024-08-17T18:04:49.284031+0000","last_fullsized":"2024-08-17T18:04:49.284031+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:37:46.260775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1467,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.284393+0000","last_change":"2024-08-17T18:04:43.766724+0000","last_active":"2024-08-17T18:04:49.284393+0000","last_peered":"2024-08-17T18:04:49.284393+0000","last_clean":"2024-08-17T18:04:49.284393+0000","last_became_active":"2024-08-17T18:04:43.766428+0000","last_became_peered":"2024-08-17T18:04:43.766428+0000","last_unstale":"2024-08-17T18:04:49.284393+0000","last_undegraded":"2024-08-17T18:04:49.284393+0000","last_fullsized":"2024-08-17T18:04:49.284393+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:09:49.102940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1480,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.369818+0000","last_change":"2024-08-17T18:03:51.279963+0000","last_active":"2024-08-17T18:05:04.369818+0000","last_peered":"2024-08-17T18:05:04.369818+0000","last_clean":"2024-08-17T18:05:04.369818+0000","last_became_active":"2024-08-17T18:03:51.279506+0000","last_became_peered":"2024-08-17T18:03:51.279506+0000","last_unstale":"2024-08-17T18:05:04.369818+0000","last_undegraded":"2024-08-17T18:05:04.369818+0000","last_fullsized":"2024-08-17T18:05:04.369818+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:20:07.304042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":36,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.377361+0000","last_change":"2024-08-17T18:04:14.542589+0000","last_active":"2024-08-17T18:05:04.377361+0000","last_peered":"2024-08-17T18:05:04.377361+0000","last_clean":"2024-08-17T18:05:04.377361+0000","last_became_active":"2024-08-17T18:04:14.531540+0000","last_became_peered":"2024-08-17T18:04:14.531540+0000","last_unstale":"2024-08-17T18:05:04.377361+0000","last_undegraded":"2024-08-17T18:05:04.377361+0000","last_fullsized":"2024-08-17T18:05:04.377361+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:15:33.014782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2246,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.359358+0000","last_change":"2024-08-17T18:04:14.540337+0000","last_active":"2024-08-17T18:05:04.359358+0000","last_peered":"2024-08-17T18:05:04.359358+0000","last_clean":"2024-08-17T18:05:04.359358+0000","last_became_active":"2024-08-17T18:04:14.538363+0000","last_became_peered":"2024-08-17T18:04:14.538363+0000","last_unstale":"2024-08-17T18:05:04.359358+0000","last_undegraded":"2024-08-17T18:05:04.359358+0000","last_fullsized":"2024-08-17T18:05:04.359358+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:51.621443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2225,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.372210+0000","last_change":"2024-08-17T17:23:07.428296+0000","last_active":"2024-08-17T18:05:04.372210+0000","last_peered":"2024-08-17T18:05:04.372210+0000","last_clean":"2024-08-17T18:05:04.372210+0000","last_became_active":"2024-08-17T17:23:07.428010+0000","last_became_peered":"2024-08-17T17:23:07.428010+0000","last_unstale":"2024-08-17T18:05:04.372210+0000","last_undegraded":"2024-08-17T18:05:04.372210+0000","last_fullsized":"2024-08-17T18:05:04.372210+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:45.459212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":2266,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.359532+0000","last_change":"2024-08-17T18:04:43.800633+0000","last_active":"2024-08-17T18:05:04.359532+0000","last_peered":"2024-08-17T18:05:04.359532+0000","last_clean":"2024-08-17T18:05:04.359532+0000","last_became_active":"2024-08-17T18:04:43.784612+0000","last_became_peered":"2024-08-17T18:04:43.784612+0000","last_unstale":"2024-08-17T18:05:04.359532+0000","last_undegraded":"2024-08-17T18:05:04.359532+0000","last_fullsized":"2024-08-17T18:05:04.359532+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:11.405039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":39,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.376247+0000","last_change":"2024-08-17T18:04:43.805055+0000","last_active":"2024-08-17T18:05:04.376247+0000","last_peered":"2024-08-17T18:05:04.376247+0000","last_clean":"2024-08-17T18:05:04.376247+0000","last_became_active":"2024-08-17T18:04:43.804708+0000","last_became_peered":"2024-08-17T18:04:43.804708+0000","last_unstale":"2024-08-17T18:05:04.376247+0000","last_undegraded":"2024-08-17T18:05:04.376247+0000","last_fullsized":"2024-08-17T18:05:04.376247+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:47:22.253073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":32,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.377523+0000","last_change":"2024-08-17T18:04:14.529605+0000","last_active":"2024-08-17T18:05:04.377523+0000","last_peered":"2024-08-17T18:05:04.377523+0000","last_clean":"2024-08-17T18:05:04.377523+0000","last_became_active":"2024-08-17T18:04:14.529024+0000","last_became_peered":"2024-08-17T18:04:14.529024+0000","last_unstale":"2024-08-17T18:05:04.377523+0000","last_undegraded":"2024-08-17T18:05:04.377523+0000","last_fullsized":"2024-08-17T18:05:04.377523+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:38:00.485543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2279,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.370614+0000","last_change":"2024-08-17T18:04:43.773667+0000","last_active":"2024-08-17T18:05:04.370614+0000","last_peered":"2024-08-17T18:05:04.370614+0000","last_clean":"2024-08-17T18:05:04.370614+0000","last_became_active":"2024-08-17T18:04:43.773247+0000","last_became_peered":"2024-08-17T18:04:43.773247+0000","last_unstale":"2024-08-17T18:05:04.370614+0000","last_undegraded":"2024-08-17T18:05:04.370614+0000","last_fullsized":"2024-08-17T18:05:04.370614+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:35:48.964631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":103,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.366940+0000","last_change":"2024-08-17T18:04:43.771589+0000","last_active":"2024-08-17T18:05:04.366940+0000","last_peered":"2024-08-17T18:05:04.366940+0000","last_clean":"2024-08-17T18:05:04.366940+0000","last_became_active":"2024-08-17T18:04:43.769601+0000","last_became_peered":"2024-08-17T18:04:43.769601+0000","last_unstale":"2024-08-17T18:05:04.366940+0000","last_undegraded":"2024-08-17T18:05:04.366940+0000","last_fullsized":"2024-08-17T18:05:04.366940+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:42:49.228921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2295,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.370279+0000","last_change":"2024-08-17T18:04:14.537136+0000","last_active":"2024-08-17T18:05:04.370279+0000","last_peered":"2024-08-17T18:05:04.370279+0000","last_clean":"2024-08-17T18:05:04.370279+0000","last_became_active":"2024-08-17T18:04:14.536815+0000","last_became_peered":"2024-08-17T18:04:14.536815+0000","last_unstale":"2024-08-17T18:05:04.370279+0000","last_undegraded":"2024-08-17T18:05:04.370279+0000","last_fullsized":"2024-08-17T18:05:04.370279+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:02.706133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.293183+0000","last_change":"2024-08-17T18:04:43.773480+0000","last_active":"2024-08-17T18:04:49.293183+0000","last_peered":"2024-08-17T18:04:49.293183+0000","last_clean":"2024-08-17T18:04:49.293183+0000","last_became_active":"2024-08-17T18:04:43.773160+0000","last_became_peered":"2024-08-17T18:04:43.773160+0000","last_unstale":"2024-08-17T18:04:49.293183+0000","last_undegraded":"2024-08-17T18:04:49.293183+0000","last_fullsized":"2024-08-17T18:04:49.293183+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:07:55.468132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":105,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.370201+0000","last_change":"2024-08-17T18:02:08.224928+0000","last_active":"2024-08-17T18:05:04.370201+0000","last_peered":"2024-08-17T18:05:04.370201+0000","last_clean":"2024-08-17T18:05:04.370201+0000","last_became_active":"2024-08-17T18:02:08.224604+0000","last_became_peered":"2024-08-17T18:02:08.224604+0000","last_unstale":"2024-08-17T18:05:04.370201+0000","last_undegraded":"2024-08-17T18:05:04.370201+0000","last_fullsized":"2024-08-17T18:05:04.370201+0000","mapping_epoch":1421,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1422,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:55.716971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1465,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.284167+0000","last_change":"2024-08-17T18:04:14.526890+0000","last_active":"2024-08-17T18:04:49.284167+0000","last_peered":"2024-08-17T18:04:49.284167+0000","last_clean":"2024-08-17T18:04:49.284167+0000","last_became_active":"2024-08-17T18:04:14.526564+0000","last_became_peered":"2024-08-17T18:04:14.526564+0000","last_unstale":"2024-08-17T18:04:49.284167+0000","last_undegraded":"2024-08-17T18:04:49.284167+0000","last_fullsized":"2024-08-17T18:04:49.284167+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:10:13.229827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":18,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.293289+0000","last_change":"2024-08-17T18:04:43.791749+0000","last_active":"2024-08-17T18:04:49.293289+0000","last_peered":"2024-08-17T18:04:49.293289+0000","last_clean":"2024-08-17T18:04:49.293289+0000","last_became_active":"2024-08-17T18:04:43.774474+0000","last_became_peered":"2024-08-17T18:04:43.774474+0000","last_unstale":"2024-08-17T18:04:49.293289+0000","last_undegraded":"2024-08-17T18:04:49.293289+0000","last_fullsized":"2024-08-17T18:04:49.293289+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:19:08.097560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2281,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.372568+0000","last_change":"2024-08-17T18:04:43.781133+0000","last_active":"2024-08-17T18:05:04.372568+0000","last_peered":"2024-08-17T18:05:04.372568+0000","last_clean":"2024-08-17T18:05:04.372568+0000","last_became_active":"2024-08-17T18:04:43.780787+0000","last_became_peered":"2024-08-17T18:04:43.780787+0000","last_unstale":"2024-08-17T18:05:04.372568+0000","last_undegraded":"2024-08-17T18:05:04.372568+0000","last_fullsized":"2024-08-17T18:05:04.372568+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:38:34.901162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2237,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.369615+0000","last_change":"2024-08-17T18:01:23.493896+0000","last_active":"2024-08-17T18:05:04.369615+0000","last_peered":"2024-08-17T18:05:04.369615+0000","last_clean":"2024-08-17T18:05:04.369615+0000","last_became_active":"2024-08-17T18:01:23.493551+0000","last_became_peered":"2024-08-17T18:01:23.493551+0000","last_unstale":"2024-08-17T18:05:04.369615+0000","last_undegraded":"2024-08-17T18:05:04.369615+0000","last_fullsized":"2024-08-17T18:05:04.369615+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:18:47.648483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2215,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.365108+0000","last_change":"2024-08-17T18:04:14.543532+0000","last_active":"2024-08-17T18:05:04.365108+0000","last_peered":"2024-08-17T18:05:04.365108+0000","last_clean":"2024-08-17T18:05:04.365108+0000","last_became_active":"2024-08-17T18:04:14.543131+0000","last_became_peered":"2024-08-17T18:04:14.543131+0000","last_unstale":"2024-08-17T18:05:04.365108+0000","last_undegraded":"2024-08-17T18:05:04.365108+0000","last_fullsized":"2024-08-17T18:05:04.365108+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:51:44.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":32,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.376939+0000","last_change":"2024-08-17T18:04:14.536309+0000","last_active":"2024-08-17T18:05:04.376939+0000","last_peered":"2024-08-17T18:05:04.376939+0000","last_clean":"2024-08-17T18:05:04.376939+0000","last_became_active":"2024-08-17T18:04:14.536000+0000","last_became_peered":"2024-08-17T18:04:14.536000+0000","last_unstale":"2024-08-17T18:05:04.376939+0000","last_undegraded":"2024-08-17T18:05:04.376939+0000","last_fullsized":"2024-08-17T18:05:04.376939+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:37:18.733611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":32,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.374497+0000","last_change":"2024-08-17T18:04:14.543835+0000","last_active":"2024-08-17T18:05:04.374497+0000","last_peered":"2024-08-17T18:05:04.374497+0000","last_clean":"2024-08-17T18:05:04.374497+0000","last_became_active":"2024-08-17T18:04:14.542511+0000","last_became_peered":"2024-08-17T18:04:14.542511+0000","last_unstale":"2024-08-17T18:05:04.374497+0000","last_undegraded":"2024-08-17T18:05:04.374497+0000","last_fullsized":"2024-08-17T18:05:04.374497+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:47:15.826245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":981,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.369773+0000","last_change":"2024-08-17T18:04:14.537308+0000","last_active":"2024-08-17T18:05:04.369773+0000","last_peered":"2024-08-17T18:05:04.369773+0000","last_clean":"2024-08-17T18:05:04.369773+0000","last_became_active":"2024-08-17T18:04:14.535521+0000","last_became_peered":"2024-08-17T18:04:14.535521+0000","last_unstale":"2024-08-17T18:05:04.369773+0000","last_undegraded":"2024-08-17T18:05:04.369773+0000","last_fullsized":"2024-08-17T18:05:04.369773+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:40:00.384936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1446,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.283477+0000","last_change":"2024-08-17T18:04:14.527639+0000","last_active":"2024-08-17T18:04:49.283477+0000","last_peered":"2024-08-17T18:04:49.283477+0000","last_clean":"2024-08-17T18:04:49.283477+0000","last_became_active":"2024-08-17T18:04:14.527117+0000","last_became_peered":"2024-08-17T18:04:14.527117+0000","last_unstale":"2024-08-17T18:04:49.283477+0000","last_undegraded":"2024-08-17T18:04:49.283477+0000","last_fullsized":"2024-08-17T18:04:49.283477+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:01:04.047871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":43,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.366740+0000","last_change":"2024-08-17T18:04:43.771309+0000","last_active":"2024-08-17T18:05:04.366740+0000","last_peered":"2024-08-17T18:05:04.366740+0000","last_clean":"2024-08-17T18:05:04.366740+0000","last_became_active":"2024-08-17T18:04:43.770922+0000","last_became_peered":"2024-08-17T18:04:43.770922+0000","last_unstale":"2024-08-17T18:05:04.366740+0000","last_undegraded":"2024-08-17T18:05:04.366740+0000","last_fullsized":"2024-08-17T18:05:04.366740+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:10.173251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":18,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.293253+0000","last_change":"2024-08-17T18:04:43.786228+0000","last_active":"2024-08-17T18:04:49.293253+0000","last_peered":"2024-08-17T18:04:49.293253+0000","last_clean":"2024-08-17T18:04:49.293253+0000","last_became_active":"2024-08-17T18:04:43.770976+0000","last_became_peered":"2024-08-17T18:04:43.770976+0000","last_unstale":"2024-08-17T18:04:49.293253+0000","last_undegraded":"2024-08-17T18:04:49.293253+0000","last_fullsized":"2024-08-17T18:04:49.293253+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:20:10.131137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":648,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.374309+0000","last_change":"2024-08-17T18:04:17.566582+0000","last_active":"2024-08-17T18:05:04.374309+0000","last_peered":"2024-08-17T18:05:04.374309+0000","last_clean":"2024-08-17T18:05:04.374309+0000","last_became_active":"2024-08-17T18:04:17.562526+0000","last_became_peered":"2024-08-17T18:04:17.562526+0000","last_unstale":"2024-08-17T18:05:04.374309+0000","last_undegraded":"2024-08-17T18:05:04.374309+0000","last_fullsized":"2024-08-17T18:05:04.374309+0000","mapping_epoch":1460,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T17:06:21.174471+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:27:13.722171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2224,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.372482+0000","last_change":"2024-08-17T17:24:17.455710+0000","last_active":"2024-08-17T18:05:04.372482+0000","last_peered":"2024-08-17T18:05:04.372482+0000","last_clean":"2024-08-17T18:05:04.372482+0000","last_became_active":"2024-08-17T17:24:17.452989+0000","last_became_peered":"2024-08-17T17:24:17.452989+0000","last_unstale":"2024-08-17T18:05:04.372482+0000","last_undegraded":"2024-08-17T18:05:04.372482+0000","last_fullsized":"2024-08-17T18:05:04.372482+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:58:07.894968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2231,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.359109+0000","last_change":"2024-08-17T18:04:11.319527+0000","last_active":"2024-08-17T18:05:04.359109+0000","last_peered":"2024-08-17T18:05:04.359109+0000","last_clean":"2024-08-17T18:05:04.359109+0000","last_became_active":"2024-08-17T18:04:11.319104+0000","last_became_peered":"2024-08-17T18:04:11.319104+0000","last_unstale":"2024-08-17T18:05:04.359109+0000","last_undegraded":"2024-08-17T18:05:04.359109+0000","last_fullsized":"2024-08-17T18:05:04.359109+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:09:13.581842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1444,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.284245+0000","last_change":"2024-08-17T17:23:07.451866+0000","last_active":"2024-08-17T18:04:49.284245+0000","last_peered":"2024-08-17T18:04:49.284245+0000","last_clean":"2024-08-17T18:04:49.284245+0000","last_became_active":"2024-08-17T17:23:07.451553+0000","last_became_peered":"2024-08-17T17:23:07.451553+0000","last_unstale":"2024-08-17T18:04:49.284245+0000","last_undegraded":"2024-08-17T18:04:49.284245+0000","last_fullsized":"2024-08-17T18:04:49.284245+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:37:23.930651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1470,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.284474+0000","last_change":"2024-08-17T18:03:51.270293+0000","last_active":"2024-08-17T18:04:49.284474+0000","last_peered":"2024-08-17T18:04:49.284474+0000","last_clean":"2024-08-17T18:04:49.284474+0000","last_became_active":"2024-08-17T18:03:51.269970+0000","last_became_peered":"2024-08-17T18:03:51.269970+0000","last_unstale":"2024-08-17T18:04:49.284474+0000","last_undegraded":"2024-08-17T18:04:49.284474+0000","last_fullsized":"2024-08-17T18:04:49.284474+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:02:30.016517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1474,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.284860+0000","last_change":"2024-08-17T18:01:23.497110+0000","last_active":"2024-08-17T18:04:49.284860+0000","last_peered":"2024-08-17T18:04:49.284860+0000","last_clean":"2024-08-17T18:04:49.284860+0000","last_became_active":"2024-08-17T18:01:23.496800+0000","last_became_peered":"2024-08-17T18:01:23.496800+0000","last_unstale":"2024-08-17T18:04:49.284860+0000","last_undegraded":"2024-08-17T18:04:49.284860+0000","last_fullsized":"2024-08-17T18:04:49.284860+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:32:29.892739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1443,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.284448+0000","last_change":"2024-08-17T17:23:07.401624+0000","last_active":"2024-08-17T18:04:49.284448+0000","last_peered":"2024-08-17T18:04:49.284448+0000","last_clean":"2024-08-17T18:04:49.284448+0000","last_became_active":"2024-08-17T17:23:07.400222+0000","last_became_peered":"2024-08-17T17:23:07.400222+0000","last_unstale":"2024-08-17T18:04:49.284448+0000","last_undegraded":"2024-08-17T18:04:49.284448+0000","last_fullsized":"2024-08-17T18:04:49.284448+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:07.069989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1444,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.283798+0000","last_change":"2024-08-17T17:24:17.470809+0000","last_active":"2024-08-17T18:04:49.283798+0000","last_peered":"2024-08-17T18:04:49.283798+0000","last_clean":"2024-08-17T18:04:49.283798+0000","last_became_active":"2024-08-17T17:24:17.469753+0000","last_became_peered":"2024-08-17T17:24:17.469753+0000","last_unstale":"2024-08-17T18:04:49.283798+0000","last_undegraded":"2024-08-17T18:04:49.283798+0000","last_fullsized":"2024-08-17T18:04:49.283798+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:00.049425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":2233,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.358876+0000","last_change":"2024-08-17T18:04:14.552806+0000","last_active":"2024-08-17T18:05:04.358876+0000","last_peered":"2024-08-17T18:05:04.358876+0000","last_clean":"2024-08-17T18:05:04.358876+0000","last_became_active":"2024-08-17T18:04:14.535546+0000","last_became_peered":"2024-08-17T18:04:14.535546+0000","last_unstale":"2024-08-17T18:05:04.358876+0000","last_undegraded":"2024-08-17T18:05:04.358876+0000","last_fullsized":"2024-08-17T18:05:04.358876+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:44:45.702272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":32,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.375943+0000","last_change":"2024-08-17T18:04:14.539500+0000","last_active":"2024-08-17T18:05:04.375943+0000","last_peered":"2024-08-17T18:05:04.375943+0000","last_clean":"2024-08-17T18:05:04.375943+0000","last_became_active":"2024-08-17T18:04:14.538738+0000","last_became_peered":"2024-08-17T18:04:14.538738+0000","last_unstale":"2024-08-17T18:05:04.375943+0000","last_undegraded":"2024-08-17T18:05:04.375943+0000","last_fullsized":"2024-08-17T18:05:04.375943+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:31.439947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.294499+0000","last_change":"2024-08-17T18:04:43.771107+0000","last_active":"2024-08-17T18:04:49.294499+0000","last_peered":"2024-08-17T18:04:49.294499+0000","last_clean":"2024-08-17T18:04:49.294499+0000","last_became_active":"2024-08-17T18:04:43.770799+0000","last_became_peered":"2024-08-17T18:04:43.770799+0000","last_unstale":"2024-08-17T18:04:49.294499+0000","last_undegraded":"2024-08-17T18:04:49.294499+0000","last_fullsized":"2024-08-17T18:04:49.294499+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:09:04.350798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1469,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.282791+0000","last_change":"2024-08-17T18:04:43.772808+0000","last_active":"2024-08-17T18:04:49.282791+0000","last_peered":"2024-08-17T18:04:49.282791+0000","last_clean":"2024-08-17T18:04:49.282791+0000","last_became_active":"2024-08-17T18:04:43.772550+0000","last_became_peered":"2024-08-17T18:04:43.772550+0000","last_unstale":"2024-08-17T18:04:49.282791+0000","last_undegraded":"2024-08-17T18:04:49.282791+0000","last_fullsized":"2024-08-17T18:04:49.282791+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:04.052168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":18,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.294039+0000","last_change":"2024-08-17T18:04:43.781111+0000","last_active":"2024-08-17T18:04:49.294039+0000","last_peered":"2024-08-17T18:04:49.294039+0000","last_clean":"2024-08-17T18:04:49.294039+0000","last_became_active":"2024-08-17T18:04:43.771839+0000","last_became_peered":"2024-08-17T18:04:43.771839+0000","last_unstale":"2024-08-17T18:04:49.294039+0000","last_undegraded":"2024-08-17T18:04:49.294039+0000","last_fullsized":"2024-08-17T18:04:49.294039+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:52:45.973049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1460,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.283774+0000","last_change":"2024-08-17T17:24:17.460203+0000","last_active":"2024-08-17T18:04:49.283774+0000","last_peered":"2024-08-17T18:04:49.283774+0000","last_clean":"2024-08-17T18:04:49.283774+0000","last_became_active":"2024-08-17T17:24:17.459682+0000","last_became_peered":"2024-08-17T17:24:17.459682+0000","last_unstale":"2024-08-17T18:04:49.283774+0000","last_undegraded":"2024-08-17T18:04:49.283774+0000","last_fullsized":"2024-08-17T18:04:49.283774+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:58.703700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":51,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.373812+0000","last_change":"2024-08-17T18:04:14.551061+0000","last_active":"2024-08-17T18:05:04.373812+0000","last_peered":"2024-08-17T18:05:04.373812+0000","last_clean":"2024-08-17T18:05:04.373812+0000","last_became_active":"2024-08-17T18:04:14.543296+0000","last_became_peered":"2024-08-17T18:04:14.543296+0000","last_unstale":"2024-08-17T18:05:04.373812+0000","last_undegraded":"2024-08-17T18:05:04.373812+0000","last_fullsized":"2024-08-17T18:05:04.373812+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:39:45.294265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.293099+0000","last_change":"2024-08-17T18:04:43.768583+0000","last_active":"2024-08-17T18:04:49.293099+0000","last_peered":"2024-08-17T18:04:49.293099+0000","last_clean":"2024-08-17T18:04:49.293099+0000","last_became_active":"2024-08-17T18:04:43.768278+0000","last_became_peered":"2024-08-17T18:04:43.768278+0000","last_unstale":"2024-08-17T18:04:49.293099+0000","last_undegraded":"2024-08-17T18:04:49.293099+0000","last_fullsized":"2024-08-17T18:04:49.293099+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:00:37.073591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":59,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.284386+0000","last_change":"2024-08-17T18:04:14.547798+0000","last_active":"2024-08-17T18:04:49.284386+0000","last_peered":"2024-08-17T18:04:49.284386+0000","last_clean":"2024-08-17T18:04:49.284386+0000","last_became_active":"2024-08-17T18:04:14.527496+0000","last_became_peered":"2024-08-17T18:04:14.527496+0000","last_unstale":"2024-08-17T18:04:49.284386+0000","last_undegraded":"2024-08-17T18:04:49.284386+0000","last_fullsized":"2024-08-17T18:04:49.284386+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:25:54.338846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1525,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.283220+0000","last_change":"2024-08-17T18:04:43.813536+0000","last_active":"2024-08-17T18:04:49.283220+0000","last_peered":"2024-08-17T18:04:49.283220+0000","last_clean":"2024-08-17T18:04:49.283220+0000","last_became_active":"2024-08-17T18:04:43.799823+0000","last_became_peered":"2024-08-17T18:04:43.799823+0000","last_unstale":"2024-08-17T18:04:49.283220+0000","last_undegraded":"2024-08-17T18:04:49.283220+0000","last_fullsized":"2024-08-17T18:04:49.283220+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:04:10.837113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2221,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.359430+0000","last_change":"2024-08-17T18:03:51.277803+0000","last_active":"2024-08-17T18:05:04.359430+0000","last_peered":"2024-08-17T18:05:04.359430+0000","last_clean":"2024-08-17T18:05:04.359430+0000","last_became_active":"2024-08-17T18:03:51.277408+0000","last_became_peered":"2024-08-17T18:03:51.277408+0000","last_unstale":"2024-08-17T18:05:04.359430+0000","last_undegraded":"2024-08-17T18:05:04.359430+0000","last_fullsized":"2024-08-17T18:05:04.359430+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:12.683553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":183,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.284311+0000","last_change":"2024-08-17T18:04:14.527380+0000","last_active":"2024-08-17T18:04:49.284311+0000","last_peered":"2024-08-17T18:04:49.284311+0000","last_clean":"2024-08-17T18:04:49.284311+0000","last_became_active":"2024-08-17T18:04:14.527076+0000","last_became_peered":"2024-08-17T18:04:14.527076+0000","last_unstale":"2024-08-17T18:04:49.284311+0000","last_undegraded":"2024-08-17T18:04:49.284311+0000","last_fullsized":"2024-08-17T18:04:49.284311+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:36:35.583709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":32,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.377793+0000","last_change":"2024-08-17T18:04:14.541814+0000","last_active":"2024-08-17T18:05:04.377793+0000","last_peered":"2024-08-17T18:05:04.377793+0000","last_clean":"2024-08-17T18:05:04.377793+0000","last_became_active":"2024-08-17T18:04:14.541403+0000","last_became_peered":"2024-08-17T18:04:14.541403+0000","last_unstale":"2024-08-17T18:05:04.377793+0000","last_undegraded":"2024-08-17T18:05:04.377793+0000","last_fullsized":"2024-08-17T18:05:04.377793+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:07:45.294924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":32,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.375681+0000","last_change":"2024-08-17T18:04:14.539149+0000","last_active":"2024-08-17T18:05:04.375681+0000","last_peered":"2024-08-17T18:05:04.375681+0000","last_clean":"2024-08-17T18:05:04.375681+0000","last_became_active":"2024-08-17T18:04:14.538852+0000","last_became_peered":"2024-08-17T18:04:14.538852+0000","last_unstale":"2024-08-17T18:05:04.375681+0000","last_undegraded":"2024-08-17T18:05:04.375681+0000","last_fullsized":"2024-08-17T18:05:04.375681+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:02:55.704464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.293386+0000","last_change":"2024-08-17T18:04:43.771720+0000","last_active":"2024-08-17T18:04:49.293386+0000","last_peered":"2024-08-17T18:04:49.293386+0000","last_clean":"2024-08-17T18:04:49.293386+0000","last_became_active":"2024-08-17T18:04:43.771137+0000","last_became_peered":"2024-08-17T18:04:43.771137+0000","last_unstale":"2024-08-17T18:04:49.293386+0000","last_undegraded":"2024-08-17T18:04:49.293386+0000","last_fullsized":"2024-08-17T18:04:49.293386+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:57.647909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":48,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-08-17T18:04:49.283742+0000","last_change":"2024-08-17T18:03:51.283872+0000","last_active":"2024-08-17T18:04:49.283742+0000","last_peered":"2024-08-17T18:04:49.283742+0000","last_clean":"2024-08-17T18:04:49.283742+0000","last_became_active":"2024-08-17T18:03:51.267197+0000","last_became_peered":"2024-08-17T18:03:51.267197+0000","last_unstale":"2024-08-17T18:04:49.283742+0000","last_undegraded":"2024-08-17T18:04:49.283742+0000","last_fullsized":"2024-08-17T18:04:49.283742+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:18:59.157786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2222,"reported_epoch":1468,"state":"active+clean","last_fresh":"2024-08-17T18:05:04.359766+0000","last_change":"2024-08-17T18:03:51.277491+0000","last_active":"2024-08-17T18:05:04.359766+0000","last_peered":"2024-08-17T18:05:04.359766+0000","last_clean":"2024-08-17T18:05:04.359766+0000","last_became_active":"2024-08-17T18:03:51.277178+0000","last_became_peered":"2024-08-17T18:03:51.277178+0000","last_unstale":"2024-08-17T18:05:04.359766+0000","last_undegraded":"2024-08-17T18:05:04.359766+0000","last_fullsized":"2024-08-17T18:05:04.359766+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:21:35.319085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":55,"num_pg":16,"stat_sum":{"num_bytes":5394202624,"num_objects":82309,"num_object_clones":0,"num_object_copies":246927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25438,"num_objects_misplaced":8322,"num_objects_unfound":0,"num_objects_dirty":82309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82309,"num_write_kb":5267776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4790,"num_bytes_recovered":313917440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16517300224,"data_stored":16517300224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82309,"ondisk_log_size":82309,"up":48,"acting":41,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":6830,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234669,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496307,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3927328,"kb_used_data":3288408,"kb_used_omap":1,"kb_used_meta":638910,"kb_avail":89817824,"statfs":{"total":95995035648,"available":91973451776,"internally_reserved":0,"allocated":3367329792,"data_stored":3362342462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":654244295},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565548,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4253772,"kb_used_data":3545540,"kb_used_omap":72,"kb_used_meta":708151,"kb_avail":89491380,"statfs":{"total":95995035648,"available":91639173120,"internally_reserved":0,"allocated":3630632960,"data_stored":3625660859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":725147118},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,4,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":1399,"seq":6008659247150,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3875228,"kb_used_data":3694544,"kb_used_omap":131,"kb_used_meta":180540,"kb_avail":89869924,"statfs":{"total":95995035648,"available":92026802176,"internally_reserved":0,"allocated":3783213056,"data_stored":3778233879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134291,"internal_metadata":184873837},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743186938,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2991056,"kb_used_data":2405128,"kb_used_omap":66,"kb_used_meta":585853,"kb_avail":90754096,"statfs":{"total":95995035648,"available":92932194304,"internally_reserved":0,"allocated":2462851072,"data_stored":2457875300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":599914155},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448577,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1514392,"kb_used_data":1167372,"kb_used_omap":75,"kb_used_meta":346932,"kb_avail":92230760,"statfs":{"total":95995035648,"available":94444298240,"internally_reserved":0,"allocated":1195388928,"data_stored":1190409713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":355258559},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757942,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":944576,"kb_used_data":466360,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":92800576,"statfs":{"total":95995035648,"available":95027789824,"internally_reserved":0,"allocated":477552640,"data_stored":472584290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":489683398},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019580,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2161372,"kb_used_data":1588884,"kb_used_omap":1,"kb_used_meta":572478,"kb_avail":91583780,"statfs":{"total":95995035648,"available":93781790720,"internally_reserved":0,"allocated":1627017216,"data_stored":1622033461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":586217926},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1191706624,"data_stored":1191706624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2451177472,"data_stored":2451177472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3795517440,"data_stored":3795517440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3619815424,"data_stored":3619815424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1624965120,"data_stored":1624965120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":468189184,"data_stored":468189184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3365928960,"data_stored":3365928960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T18:05:10.575 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-17T18:05:10.575 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool set unique_pool_2 pg_num 26 2024-08-17T18:05:10.576 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:10 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 25438/247005 objects degraded (10.299%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:10.576 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:10 smithi165 ceph-mon[105517]: from='client.45686 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:05:10.666 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:10.717 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:10.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:10.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:10.818 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:10.871 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:10.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:10.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:10.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:11.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:11.007 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:11.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:11.109 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:11.174 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 76 16 85605 85589 70.3722 45.8125 0.0622286 0.0142016 2024-08-17T18:05:11.180 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:11.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:11.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:11.262 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:05:11.262 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:05:11.262 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-08-17T18:05:11.262 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-08-17T18:05:11.262 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:05:11.282 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:11.337 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:11.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:11.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:11.424 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:05:11.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:11.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:11.507 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:11.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:11 smithi165 ceph-mon[105517]: pgmap v2599: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 50 MiB/s wr, 792 op/s; 25438/247005 objects degraded (10.299%); 8322/247005 objects misplaced (3.369%); 6.2 MiB/s, 98 objects/s recovering 2024-08-17T18:05:11.608 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:11.648 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:11.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:11.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:11.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:11 smithi136 ceph-mon[115078]: pgmap v2599: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 50 MiB/s wr, 792 op/s; 25438/247005 objects degraded (10.299%); 8322/247005 objects misplaced (3.369%); 6.2 MiB/s, 98 objects/s recovering 2024-08-17T18:05:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:11 smithi136 ceph-mon[109861]: pgmap v2599: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 50 MiB/s wr, 792 op/s; 25438/247005 objects degraded (10.299%); 8322/247005 objects misplaced (3.369%); 6.2 MiB/s, 98 objects/s recovering 2024-08-17T18:05:11.750 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:11.795 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:11.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:11.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:11.897 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:11.943 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:11.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:11.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:11.982 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-17T18:05:12.029 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:05:12.029 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:05:12.045 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:12.063 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:05:12.063 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:05:12.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:12.082 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:12.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:12.095 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:05:12.095 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:05:12.124 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:05:12.124 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:05:12.156 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:05:12.156 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:05:12.174 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 77 16 86302 86286 70.024 43.5625 0.0233713 0.0142743 2024-08-17T18:05:12.183 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:12.192 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:05:12.192 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:05:12.228 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:05:12.228 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:05:12.233 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:12.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:12.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:12.263 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:05:12.263 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:05:12.314 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:05:12.335 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:12.372 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:12.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:12.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:12.474 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:12.504 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:12.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:12.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:12.606 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:12.820 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:12 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3945488152' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:05:12.821 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:12 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3945488152' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:05:12.821 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:12 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3945488152' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:05:12.843 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:12.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:12.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:12.945 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:12.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:12.968 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:12.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:13.069 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:13.090 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:13.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:13.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:13.174 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 78 16 86696 86680 69.442 24.625 0.00516581 0.0142799 2024-08-17T18:05:13.193 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:13.214 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:13.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:13.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:13.316 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:13.338 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:13.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:13.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:13.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:13.466 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:13.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:13.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:13 smithi165 ceph-mon[105517]: pgmap v2600: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.2 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 48 MiB/s wr, 773 op/s; 25438/257553 objects degraded (9.877%); 7729/257553 objects misplaced (3.001%); 6.9 MiB/s, 110 objects/s recovering 2024-08-17T18:05:13.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:13 smithi165 ceph-mon[105517]: osdmap e1469: 8 total, 8 up, 7 in 2024-08-17T18:05:13.568 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:13.604 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:13.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:13.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:13.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:13 smithi136 ceph-mon[109861]: pgmap v2600: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.2 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 48 MiB/s wr, 773 op/s; 25438/257553 objects degraded (9.877%); 7729/257553 objects misplaced (3.001%); 6.9 MiB/s, 110 objects/s recovering 2024-08-17T18:05:13.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:13 smithi136 ceph-mon[109861]: osdmap e1469: 8 total, 8 up, 7 in 2024-08-17T18:05:13.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:13 smithi136 ceph-mon[115078]: pgmap v2600: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.2 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 48 MiB/s wr, 773 op/s; 25438/257553 objects degraded (9.877%); 7729/257553 objects misplaced (3.001%); 6.9 MiB/s, 110 objects/s recovering 2024-08-17T18:05:13.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:13 smithi136 ceph-mon[115078]: osdmap e1469: 8 total, 8 up, 7 in 2024-08-17T18:05:13.705 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:13.726 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:13.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:13.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:13.828 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:13.876 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:13.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:13.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:13.977 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:14.012 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:14.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:14.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:14.114 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:14.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:14.143 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:14.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:14.174 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 79 16 87269 87253 69.0163 35.8125 0.00627798 0.0144836 2024-08-17T18:05:14.245 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:14.281 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:14.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:14.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:14.367 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:05:14.368 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:05:14.383 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:14.419 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:14.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:14.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:14.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:14 smithi165 ceph-mon[105517]: osdmap e1470: 8 total, 8 up, 7 in 2024-08-17T18:05:14.521 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:14.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:14.554 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:14.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:14.655 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:14.695 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:14.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:14.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:14.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:14 smithi136 ceph-mon[109861]: osdmap e1470: 8 total, 8 up, 7 in 2024-08-17T18:05:14.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:14 smithi136 ceph-mon[115078]: osdmap e1470: 8 total, 8 up, 7 in 2024-08-17T18:05:14.797 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:14.835 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:14.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:14.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:14.937 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:14.971 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:14.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:14.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:15.051 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:05:15.072 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:15.132 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:15.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:15.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:15.174 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:2024-08-17T18:05:15.174595+0000 min lat: 0.00131667 max lat: 2.29878 avg lat: 0.0145259 2024-08-17T18:05:15.174 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T18:05:15.174 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 80 16 88112 88096 68.8121 52.6875 0.0106745 0.0145259 2024-08-17T18:05:15.235 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:15.272 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:15.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:15.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:15.293 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-17T18:05:15.374 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:15.405 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:15.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:15.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:15.506 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:15 smithi165 ceph-mon[105517]: pgmap v2603: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 60 MiB/s wr, 961 op/s; 25438/258177 objects degraded (9.853%); 7729/258177 objects misplaced (2.994%); 8.7 MiB/s, 138 objects/s recovering 2024-08-17T18:05:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:15 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 25438/258177 objects degraded (9.853%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:15.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 18:05:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=plugins.update.checker t=2024-08-17T18:05:15.191378857Z level=info msg="Update check succeeded" duration=69.002335ms 2024-08-17T18:05:15.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 18:05:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=grafana.update.checker t=2024-08-17T18:05:15.197592278Z level=info msg="Update check succeeded" duration=66.889544ms 2024-08-17T18:05:15.512 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 18:05:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=cleanup t=2024-08-17T18:05:15.249612948Z level=info msg="Completed cleanup jobs" duration=211.714239ms 2024-08-17T18:05:15.545 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:15.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:15.621 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:05:15.646 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:15.689 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:15.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:15.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:15.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:15 smithi136 ceph-mon[109861]: pgmap v2603: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 60 MiB/s wr, 961 op/s; 25438/258177 objects degraded (9.853%); 7729/258177 objects misplaced (2.994%); 8.7 MiB/s, 138 objects/s recovering 2024-08-17T18:05:15.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:15 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 25438/258177 objects degraded (9.853%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:15.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:15 smithi136 ceph-mon[115078]: pgmap v2603: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 60 MiB/s wr, 961 op/s; 25438/258177 objects degraded (9.853%); 7729/258177 objects misplaced (2.994%); 8.7 MiB/s, 138 objects/s recovering 2024-08-17T18:05:15.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:15 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 25438/258177 objects degraded (9.853%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:15.791 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:15.839 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:15.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:15.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:15.941 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:15.980 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:15.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:15.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:16.082 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:16.114 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:16.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:16.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:16.174 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 81 15 88938 88923 68.6006 51.6875 0.0317359 0.0145712 2024-08-17T18:05:16.216 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:16.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:16.264 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:16.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:16.365 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:16.403 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:16.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:16.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:16.505 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:16.556 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:16.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:16.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:16.658 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:16.695 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:16.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:16.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:16.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:16.822 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:16.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:16.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:16.924 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:16.953 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:16.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:16.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:17.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:17.089 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:05:17.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:17.093 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:17.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:17.174 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 82 16 89717 89701 68.357 48.625 0.0272477 0.0146221 2024-08-17T18:05:17.195 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:17.246 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:17.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:17.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:17.348 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:17.383 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:17.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:17.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:17.436 INFO:teuthology.orchestra.run.smithi136.stderr:set pool 55 pg_num to 26 2024-08-17T18:05:17.486 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:17 smithi165 ceph-mon[105517]: pgmap v2604: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 29 MiB/s wr, 465 op/s; 25438/258177 objects degraded (9.853%); 7729/258177 objects misplaced (2.994%); 4.6 MiB/s, 74 objects/s recovering 2024-08-17T18:05:17.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:17 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2716808663' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T18:05:17.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:17.515 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:17.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:17.616 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:17.662 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:17.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:17.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:17.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:17 smithi136 ceph-mon[109861]: pgmap v2604: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 29 MiB/s wr, 465 op/s; 25438/258177 objects degraded (9.853%); 7729/258177 objects misplaced (2.994%); 4.6 MiB/s, 74 objects/s recovering 2024-08-17T18:05:17.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:17 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2716808663' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T18:05:17.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:05:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:05:17] "GET /metrics HTTP/1.1" 200 36639 "" "Prometheus/2.51.0" 2024-08-17T18:05:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:17 smithi136 ceph-mon[115078]: pgmap v2604: 89 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 29 MiB/s wr, 465 op/s; 25438/258177 objects degraded (9.853%); 7729/258177 objects misplaced (2.994%); 4.6 MiB/s, 74 objects/s recovering 2024-08-17T18:05:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:17 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2716808663' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T18:05:17.764 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:17.801 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:17.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:17.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:17.895 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:05:17.895 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:05:17.903 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:17.941 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:17.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:17.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:18.043 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:18.112 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:18.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:18.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:18.175 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 83 16 90434 90418 68.0727 44.8125 0.0084438 0.0146827 2024-08-17T18:05:18.214 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:18.271 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:18.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:18.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:18.373 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:18.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:18.421 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:18.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:18.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:18 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2716808663' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-08-17T18:05:18.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:18 smithi165 ceph-mon[105517]: osdmap e1471: 8 total, 8 up, 7 in 2024-08-17T18:05:18.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:05:18.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"} : dispatch 2024-08-17T18:05:18.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:18.611 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:18.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:18.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:18.642 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:18 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2716808663' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-08-17T18:05:18.643 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:18 smithi136 ceph-mon[109861]: osdmap e1471: 8 total, 8 up, 7 in 2024-08-17T18:05:18.643 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:05:18.643 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"} : dispatch 2024-08-17T18:05:18.643 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:18 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2716808663' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-08-17T18:05:18.643 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:18 smithi136 ceph-mon[115078]: osdmap e1471: 8 total, 8 up, 7 in 2024-08-17T18:05:18.643 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:05:18.643 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"} : dispatch 2024-08-17T18:05:18.714 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:18.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:18.767 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:18.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:18.867 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:18.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-17T18:05:18.892 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:18.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:18.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:18.995 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:19.018 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:19.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:19.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:19.120 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:19.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:19.145 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:19.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:19.175 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 84 16 90607 90591 67.3911 10.8125 0.185512 0.0147191 2024-08-17T18:05:19.246 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:19.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:19.268 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:19.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:19.369 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:19.390 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:19.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:19.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:19.492 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:19.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:19.531 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:19.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:19.627 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:05:19.631 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:19.657 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:19 smithi136 ceph-mon[109861]: pgmap v2606: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 38 MiB/s wr, 600 op/s; 25438/268371 objects degraded (9.479%); 7551/268371 objects misplaced (2.814%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T18:05:19.657 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:19 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-17T18:05:19.657 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:19 smithi136 ceph-mon[109861]: osdmap e1472: 8 total, 8 up, 7 in 2024-08-17T18:05:19.658 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:19 smithi136 ceph-mon[115078]: pgmap v2606: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 38 MiB/s wr, 600 op/s; 25438/268371 objects degraded (9.479%); 7551/268371 objects misplaced (2.814%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T18:05:19.658 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:19 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-17T18:05:19.658 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:19 smithi136 ceph-mon[115078]: osdmap e1472: 8 total, 8 up, 7 in 2024-08-17T18:05:19.660 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:19.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:19.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:19 smithi165 ceph-mon[105517]: pgmap v2606: 89 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 38 MiB/s wr, 600 op/s; 25438/268371 objects degraded (9.479%); 7551/268371 objects misplaced (2.814%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T18:05:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:19 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-17T18:05:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:19 smithi165 ceph-mon[105517]: osdmap e1472: 8 total, 8 up, 7 in 2024-08-17T18:05:19.761 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:19.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:19.795 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:19.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:19.896 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:19.924 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:19.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:19.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:20.026 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:20.048 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:20.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:20.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:20.150 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:20.172 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:20.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:20.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:20.175 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 85 16 90759 90743 66.71 9.5 0.0060419 0.0148953 2024-08-17T18:05:20.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:20.301 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:20.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:20.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:20.403 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:20.424 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:20.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:20.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:20.526 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:20.551 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:20.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:20.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:20.653 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:20.676 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:20.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:20.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:20 smithi165 ceph-mon[105517]: osdmap e1473: 8 total, 8 up, 7 in 2024-08-17T18:05:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:20 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 25438/268371 objects degraded (9.479%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:20 smithi165 ceph-mon[105517]: osdmap e1474: 8 total, 8 up, 7 in 2024-08-17T18:05:20.778 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:20.804 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:20.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:20.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:20.906 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:20.928 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:20.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:20.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:20.942 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:20 smithi136 ceph-mon[109861]: osdmap e1473: 8 total, 8 up, 7 in 2024-08-17T18:05:20.942 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:20 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 25438/268371 objects degraded (9.479%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:20.943 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:20 smithi136 ceph-mon[109861]: osdmap e1474: 8 total, 8 up, 7 in 2024-08-17T18:05:20.943 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:20 smithi136 ceph-mon[115078]: osdmap e1473: 8 total, 8 up, 7 in 2024-08-17T18:05:20.943 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:20 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 25438/268371 objects degraded (9.479%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:20.943 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:20 smithi136 ceph-mon[115078]: osdmap e1474: 8 total, 8 up, 7 in 2024-08-17T18:05:21.030 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:21.051 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:21.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:21.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:21.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:21.174 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:21.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:21.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:21.175 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 86 16 90800 90784 65.9642 2.5625 0.00358722 0.0149545 2024-08-17T18:05:21.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:21.298 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:21.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:21.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:21.400 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:21.422 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:21.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:21.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:21.456 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:05:21.528 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:21.550 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:21.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:21.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:21.652 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:21.673 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:21.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:21.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:21.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:21 smithi165 ceph-mon[105517]: pgmap v2610: 99 pgs: 1 peering, 9 unknown, 1 active+recovering+undersized+remapped, 1 creating+peering, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 54 MiB/s wr, 870 op/s; 22968/268629 objects degraded (8.550%); 3477/268629 objects misplaced (1.294%); 2.8 MiB/s, 44 objects/s recovering 2024-08-17T18:05:21.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:21 smithi165 ceph-mon[105517]: osdmap e1475: 8 total, 8 up, 7 in 2024-08-17T18:05:21.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:05:21.775 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:21.797 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:21.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:21.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:21.883 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:21 smithi136 ceph-mon[109861]: pgmap v2610: 99 pgs: 1 peering, 9 unknown, 1 active+recovering+undersized+remapped, 1 creating+peering, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 54 MiB/s wr, 870 op/s; 22968/268629 objects degraded (8.550%); 3477/268629 objects misplaced (1.294%); 2.8 MiB/s, 44 objects/s recovering 2024-08-17T18:05:21.883 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:21 smithi136 ceph-mon[109861]: osdmap e1475: 8 total, 8 up, 7 in 2024-08-17T18:05:21.883 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:05:21.883 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:21 smithi136 ceph-mon[115078]: pgmap v2610: 99 pgs: 1 peering, 9 unknown, 1 active+recovering+undersized+remapped, 1 creating+peering, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 54 MiB/s wr, 870 op/s; 22968/268629 objects degraded (8.550%); 3477/268629 objects misplaced (1.294%); 2.8 MiB/s, 44 objects/s recovering 2024-08-17T18:05:21.883 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:21 smithi136 ceph-mon[115078]: osdmap e1475: 8 total, 8 up, 7 in 2024-08-17T18:05:21.883 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:05:21.899 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:21.921 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:21.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:21.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:22.024 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:22.050 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:22.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:22.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:22.152 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:22.174 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:22.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:22.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:22.175 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 87 16 90800 90784 65.2061 0 - 0.0149545 2024-08-17T18:05:22.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:22.297 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:22.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:22.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:22.387 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:05:22.387 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:05:22.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:22.421 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:22.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:22.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:22.523 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:22.545 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:22.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:22.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:22.546 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:05:22.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:22.668 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:22.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:22.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:22.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:22 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3919752552' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:05:22.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:22 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3919752552' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:05:22.700 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:22 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3919752552' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:05:22.769 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:22.791 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:22.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:22.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:22.893 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:22.916 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:22.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:22.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:23.018 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:23.040 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:23.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:23.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:23.061 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-17T18:05:23.141 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:23.174 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:23.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:23.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:23.176 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 88 16 90802 90786 64.4666 0.0625 0.00487374 0.0150056 2024-08-17T18:05:23.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:23.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:23.312 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:23.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:23.413 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:23.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:23.445 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:23.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:23.546 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:23.580 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:23.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:23.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:23.681 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:23 smithi136 ceph-mon[109861]: pgmap v2612: 99 pgs: 6 active+recovery_wait+degraded, 1 peering, 2 active+recovery_wait, 1 active+clean+scrubbing, 1 creating+peering, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 21 MiB/s wr, 328 op/s; 26410/272316 objects degraded (9.698%); 608 KiB/s, 9 objects/s recovering 2024-08-17T18:05:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:23 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:05:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:23 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3919752552' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:05:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:23 smithi136 ceph-mon[109861]: osdmap e1476: 8 total, 8 up, 7 in 2024-08-17T18:05:23.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:23 smithi136 ceph-mon[115078]: pgmap v2612: 99 pgs: 6 active+recovery_wait+degraded, 1 peering, 2 active+recovery_wait, 1 active+clean+scrubbing, 1 creating+peering, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 21 MiB/s wr, 328 op/s; 26410/272316 objects degraded (9.698%); 608 KiB/s, 9 objects/s recovering 2024-08-17T18:05:23.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:23 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:05:23.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:23 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3919752552' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:05:23.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:23 smithi136 ceph-mon[115078]: osdmap e1476: 8 total, 8 up, 7 in 2024-08-17T18:05:23.715 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:23.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:23.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:23.771 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:05:23.796 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:23 smithi165 ceph-mon[105517]: pgmap v2612: 99 pgs: 6 active+recovery_wait+degraded, 1 peering, 2 active+recovery_wait, 1 active+clean+scrubbing, 1 creating+peering, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 21 MiB/s wr, 328 op/s; 26410/272316 objects degraded (9.698%); 608 KiB/s, 9 objects/s recovering 2024-08-17T18:05:23.796 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:23 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:05:23.796 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:23 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3919752552' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:05:23.796 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:23 smithi165 ceph-mon[105517]: osdmap e1476: 8 total, 8 up, 7 in 2024-08-17T18:05:23.817 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:23.846 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:23.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:23.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:23.948 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:23.978 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:23.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:23.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:24.080 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:24.109 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:24.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:24.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:24.176 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 89 16 91398 91382 64.1608 37.25 0.00482643 0.0155639 2024-08-17T18:05:24.211 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:24.238 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:24.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:24.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:24.325 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:05:24.325 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:05:24.340 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:24.347 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:05:24.348 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:05:24.365 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:24.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:24.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:24.369 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:05:24.369 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:05:24.386 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:05:24.387 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:05:24.411 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:05:24.412 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:05:24.436 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:05:24.436 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:05:24.461 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:05:24.461 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:05:24.467 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:24.489 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:05:24.489 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:05:24.495 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:24.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:24.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:24.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:24.624 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:24.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:24.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:24.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:24 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4080535850' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:05:24.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:24 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 26410/272316 objects degraded (9.698%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:24.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:24 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4080535850' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:05:24.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:24 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 26410/272316 objects degraded (9.698%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:24.726 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:24.748 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:24.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:24.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:24.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:24 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4080535850' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:05:24.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:24 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 26410/272316 objects degraded (9.698%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:24.850 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:24.879 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:24.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:24.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:24.980 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:25.011 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:25.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:25.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:25.113 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:25.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:25.141 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:25.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:25.176 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 90 11 91983 91972 63.8576 36.875 0.211717 0.0156401 2024-08-17T18:05:25.242 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:25.265 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:25.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:25.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:25.367 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:25.389 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:25.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:25.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:25.491 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:25.513 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:25.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:25.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:25.615 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:25.637 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:25.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:25.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:25.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:25 smithi165 ceph-mon[105517]: pgmap v2614: 99 pgs: 6 active+recovery_wait+degraded, 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 1 creating+peering, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 18 MiB/s wr, 283 op/s; 19748/272430 objects degraded (7.249%); 6174/272430 objects misplaced (2.266%); 3.7 MiB/s, 59 objects/s recovering 2024-08-17T18:05:25.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:25.762 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:25.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:25.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:25.885 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:25.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:25.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:25.893 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:25 smithi136 ceph-mon[109861]: pgmap v2614: 99 pgs: 6 active+recovery_wait+degraded, 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 1 creating+peering, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 18 MiB/s wr, 283 op/s; 19748/272430 objects degraded (7.249%); 6174/272430 objects misplaced (2.266%); 3.7 MiB/s, 59 objects/s recovering 2024-08-17T18:05:25.894 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:25 smithi136 ceph-mon[115078]: pgmap v2614: 99 pgs: 6 active+recovery_wait+degraded, 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 1 creating+peering, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 18 MiB/s wr, 283 op/s; 19748/272430 objects degraded (7.249%); 6174/272430 objects misplaced (2.266%); 3.7 MiB/s, 59 objects/s recovering 2024-08-17T18:05:25.971 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 0, 5] out_osds: [7] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:05:25.971 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:05:25.972 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_2 2024-08-17T18:05:25.972 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T18:05:25.987 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:26.008 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:26.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:26.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:26.110 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:26.132 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:26.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:26.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:26.176 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Total time run: 90.2042 2024-08-17T18:05:26.176 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Total writes made: 91983 2024-08-17T18:05:26.177 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Write size: 65536 2024-08-17T18:05:26.177 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Object size: 65536 2024-08-17T18:05:26.177 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Bandwidth (MB/sec): 63.7325 2024-08-17T18:05:26.177 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Stddev Bandwidth: 41.0501 2024-08-17T18:05:26.177 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Max bandwidth (MB/sec): 204.812 2024-08-17T18:05:26.177 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Min bandwidth (MB/sec): 0 2024-08-17T18:05:26.177 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Average IOPS: 1019 2024-08-17T18:05:26.177 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Stddev IOPS: 656.801 2024-08-17T18:05:26.177 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Max IOPS: 3277 2024-08-17T18:05:26.177 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Min IOPS: 0 2024-08-17T18:05:26.177 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Average Latency(s): 0.0156662 2024-08-17T18:05:26.177 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Stddev Latency(s): 0.0616576 2024-08-17T18:05:26.177 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Max latency(s): 4.66319 2024-08-17T18:05:26.177 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Min latency(s): 0.00131667 2024-08-17T18:05:26.196 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Cleaning up (deleting benchmark objects) 2024-08-17T18:05:26.233 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:26.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:26.270 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:26.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:26.371 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:26.409 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:26.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:26.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:26.459 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:05:26.512 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:26.536 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:26.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:26.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:26.638 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:26.670 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:26.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:26.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:26.772 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:26.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:05:26.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:26.796 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:26.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:26.898 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:26.923 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:26.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:26.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:27.025 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:27.050 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:27.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:27.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:27.153 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:27.180 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:27.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:27.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:27.255 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:05:27.255 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T18:05:27.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T18:05:27.255 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:05:27.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.255 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:179375 54.10 54:09c4e419:::benchmark_data_smithi136_273969_object56490:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1397)", 2024-08-17T18:05:27.255 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:20.619242+0000", 2024-08-17T18:05:27.256 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 246.631596325, 2024-08-17T18:05:27.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.080588077, 2024-08-17T18:05:27.256 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:27.256 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:05:27.256 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:27.256 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:27.256 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:27.256 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 179375 2024-08-17T18:05:27.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.256 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:27.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:27.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:20.619242+0000", 2024-08-17T18:05:27.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:27.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:20.619242+0000", 2024-08-17T18:05:27.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:27.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:20.619240+0000", 2024-08-17T18:05:27.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:05:27.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:27.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:20.619247+0000", 2024-08-17T18:05:27.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9110000000000004e-06 2024-08-17T18:05:27.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:27.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:20.619264+0000", 2024-08-17T18:05:27.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6855999999999999e-05 2024-08-17T18:05:27.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:27.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:20.619271+0000", 2024-08-17T18:05:27.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0809999999999996e-06 2024-08-17T18:05:27.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:20.643243+0000", 2024-08-17T18:05:27.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023972284 2024-08-17T18:05:27.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:05:27.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:20.645016+0000", 2024-08-17T18:05:27.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017727769999999999 2024-08-17T18:05:27.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:20.799073+0000", 2024-08-17T18:05:27.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.154057 2024-08-17T18:05:27.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:05:27.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:20.802366+0000", 2024-08-17T18:05:27.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003292835 2024-08-17T18:05:27.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.638766+0000", 2024-08-17T18:05:27.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.83639995199999995 2024-08-17T18:05:27.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:27.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.641264+0000", 2024-08-17T18:05:27.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024977879999999999 2024-08-17T18:05:27.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,4", 2024-08-17T18:05:27.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.646497+0000", 2024-08-17T18:05:27.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00523385 2024-08-17T18:05:27.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.672883+0000", 2024-08-17T18:05:27.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.026385545 2024-08-17T18:05:27.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.674771+0000", 2024-08-17T18:05:27.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001888327 2024-08-17T18:05:27.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:27.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.691467+0000", 2024-08-17T18:05:27.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016695695 2024-08-17T18:05:27.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:05:27.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.692933+0000", 2024-08-17T18:05:27.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014657450000000001 2024-08-17T18:05:27.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:27.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.699830+0000", 2024-08-17T18:05:27.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068969269999999997 2024-08-17T18:05:27.263 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.263 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:27.263 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.264 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180116 54.10 54:0f9da3af:::benchmark_data_smithi136_273969_object57231:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:05:27.264 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:21.847960+0000", 2024-08-17T18:05:27.264 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 245.40287768900001, 2024-08-17T18:05:27.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9097264840000001, 2024-08-17T18:05:27.264 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:27.264 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:05:27.264 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:27.264 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:27.264 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:27.264 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180116 2024-08-17T18:05:27.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.264 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:27.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:27.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.847960+0000", 2024-08-17T18:05:27.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:27.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.847960+0000", 2024-08-17T18:05:27.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:27.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.847959+0000", 2024-08-17T18:05:27.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:05:27.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:27.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.847965+0000", 2024-08-17T18:05:27.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.596e-06 2024-08-17T18:05:27.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:27.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.847980+0000", 2024-08-17T18:05:27.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4819e-05 2024-08-17T18:05:27.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:27.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.847988+0000", 2024-08-17T18:05:27.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8110000000000003e-06 2024-08-17T18:05:27.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.933300+0000", 2024-08-17T18:05:27.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.085312084999999996 2024-08-17T18:05:27.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:05:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.936616+0000", 2024-08-17T18:05:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033163709999999998 2024-08-17T18:05:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.966460+0000", 2024-08-17T18:05:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.029843103999999999 2024-08-17T18:05:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:05:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.970083+0000", 2024-08-17T18:05:27.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036235529999999998 2024-08-17T18:05:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.508777+0000", 2024-08-17T18:05:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.53869427000000003 2024-08-17T18:05:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.511313+0000", 2024-08-17T18:05:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025361260000000001 2024-08-17T18:05:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,4", 2024-08-17T18:05:27.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.516630+0000", 2024-08-17T18:05:27.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0053168260000000002 2024-08-17T18:05:27.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542130+0000", 2024-08-17T18:05:27.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.025500109999999999 2024-08-17T18:05:27.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:27.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542142+0000", 2024-08-17T18:05:27.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.204e-05 2024-08-17T18:05:27.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479434+0000", 2024-08-17T18:05:27.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93729177399999997 2024-08-17T18:05:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479449+0000", 2024-08-17T18:05:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4654000000000001e-05 2024-08-17T18:05:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.540870+0000", 2024-08-17T18:05:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0614213989999999 2024-08-17T18:05:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:05:27.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.543825+0000", 2024-08-17T18:05:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029547509999999998 2024-08-17T18:05:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.756247+0000", 2024-08-17T18:05:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.212422074 2024-08-17T18:05:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.757687+0000", 2024-08-17T18:05:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001439631 2024-08-17T18:05:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:27.273 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180126 54.10 54:0ae91374:::benchmark_data_smithi136_273969_object57241:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:05:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:21.861391+0000", 2024-08-17T18:05:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 245.389446873, 2024-08-17T18:05:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9929762970000002, 2024-08-17T18:05:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:05:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180126 2024-08-17T18:05:27.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.861391+0000", 2024-08-17T18:05:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.861391+0000", 2024-08-17T18:05:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.861390+0000", 2024-08-17T18:05:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:05:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.861396+0000", 2024-08-17T18:05:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0059999999999998e-06 2024-08-17T18:05:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.861414+0000", 2024-08-17T18:05:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8851999999999998e-05 2024-08-17T18:05:27.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.861422+0000", 2024-08-17T18:05:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3220000000000004e-06 2024-08-17T18:05:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.970423+0000", 2024-08-17T18:05:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.109001084 2024-08-17T18:05:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:05:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.973780+0000", 2024-08-17T18:05:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033568019999999999 2024-08-17T18:05:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.995958+0000", 2024-08-17T18:05:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022178531000000001 2024-08-17T18:05:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:05:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.999891+0000", 2024-08-17T18:05:27.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039327559999999999 2024-08-17T18:05:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542792+0000", 2024-08-17T18:05:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.54290070599999996 2024-08-17T18:05:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542797+0000", 2024-08-17T18:05:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9849999999999997e-06 2024-08-17T18:05:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479690+0000", 2024-08-17T18:05:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93689339000000005 2024-08-17T18:05:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479700+0000", 2024-08-17T18:05:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5610000000000004e-06 2024-08-17T18:05:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.769574+0000", 2024-08-17T18:05:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.289874223 2024-08-17T18:05:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:05:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.775343+0000", 2024-08-17T18:05:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057693670000000001 2024-08-17T18:05:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.824122+0000", 2024-08-17T18:05:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.048778909000000002 2024-08-17T18:05:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.826036+0000", 2024-08-17T18:05:27.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019139980000000001 2024-08-17T18:05:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:05:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.830279+0000", 2024-08-17T18:05:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042426240000000004 2024-08-17T18:05:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.847119+0000", 2024-08-17T18:05:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016840368000000001 2024-08-17T18:05:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.847762+0000", 2024-08-17T18:05:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064298700000000003 2024-08-17T18:05:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.848426+0000", 2024-08-17T18:05:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000663925 2024-08-17T18:05:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:05:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.849470+0000", 2024-08-17T18:05:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001043719 2024-08-17T18:05:27.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.854367+0000", 2024-08-17T18:05:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004897658 2024-08-17T18:05:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180174 54.10 54:08915b2c:::benchmark_data_smithi136_273969_object57289:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:05:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:21.930733+0000", 2024-08-17T18:05:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 245.32010521500001, 2024-08-17T18:05:27.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1011982060000003, 2024-08-17T18:05:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:05:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180174 2024-08-17T18:05:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.930733+0000", 2024-08-17T18:05:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.930733+0000", 2024-08-17T18:05:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.930731+0000", 2024-08-17T18:05:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:05:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:27.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.930738+0000", 2024-08-17T18:05:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3450000000000002e-06 2024-08-17T18:05:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.930750+0000", 2024-08-17T18:05:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2476e-05 2024-08-17T18:05:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.930757+0000", 2024-08-17T18:05:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8719999999999996e-06 2024-08-17T18:05:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.974198+0000", 2024-08-17T18:05:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.043441133 2024-08-17T18:05:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:05:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.977482+0000", 2024-08-17T18:05:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032840030000000002 2024-08-17T18:05:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.032292+0000", 2024-08-17T18:05:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.054810287999999999 2024-08-17T18:05:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:05:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.035548+0000", 2024-08-17T18:05:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032551569999999998 2024-08-17T18:05:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542945+0000", 2024-08-17T18:05:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50739734999999997 2024-08-17T18:05:27.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:27.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542951+0000", 2024-08-17T18:05:27.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8069999999999996e-06 2024-08-17T18:05:27.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479903+0000", 2024-08-17T18:05:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93695215499999995 2024-08-17T18:05:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479912+0000", 2024-08-17T18:05:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7020000000000007e-06 2024-08-17T18:05:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.954749+0000", 2024-08-17T18:05:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4748370450000001 2024-08-17T18:05:27.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:05:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.960359+0000", 2024-08-17T18:05:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056101390000000001 2024-08-17T18:05:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.992310+0000", 2024-08-17T18:05:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.031950854000000001 2024-08-17T18:05:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.994150+0000", 2024-08-17T18:05:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001840752 2024-08-17T18:05:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:05:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.997815+0000", 2024-08-17T18:05:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036646690000000002 2024-08-17T18:05:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.010247+0000", 2024-08-17T18:05:27.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012432284 2024-08-17T18:05:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.025714+0000", 2024-08-17T18:05:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015466680999999999 2024-08-17T18:05:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.026527+0000", 2024-08-17T18:05:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00081259799999999999 2024-08-17T18:05:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:05:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.027532+0000", 2024-08-17T18:05:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001005159 2024-08-17T18:05:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.031931+0000", 2024-08-17T18:05:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043991380000000004 2024-08-17T18:05:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180181 54.10 54:0a355281:::benchmark_data_smithi136_273969_object57296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:05:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:21.940096+0000", 2024-08-17T18:05:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 245.31074155600001, 2024-08-17T18:05:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0013956070000001, 2024-08-17T18:05:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:05:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180181 2024-08-17T18:05:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:27.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.940096+0000", 2024-08-17T18:05:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.940096+0000", 2024-08-17T18:05:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:27.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.940094+0000", 2024-08-17T18:05:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:05:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.940102+0000", 2024-08-17T18:05:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3370000000000002e-06 2024-08-17T18:05:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.940118+0000", 2024-08-17T18:05:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5713999999999999e-05 2024-08-17T18:05:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.940126+0000", 2024-08-17T18:05:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7350000000000002e-06 2024-08-17T18:05:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.978817+0000", 2024-08-17T18:05:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038691019 2024-08-17T18:05:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:05:27.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.982796+0000", 2024-08-17T18:05:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039788150000000001 2024-08-17T18:05:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.014670+0000", 2024-08-17T18:05:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.031873637000000003 2024-08-17T18:05:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:05:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.018063+0000", 2024-08-17T18:05:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033937350000000002 2024-08-17T18:05:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542851+0000", 2024-08-17T18:05:27.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.52478792900000004 2024-08-17T18:05:27.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:27.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542857+0000", 2024-08-17T18:05:27.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6289999999999998e-06 2024-08-17T18:05:27.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479777+0000", 2024-08-17T18:05:27.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93691995400000005 2024-08-17T18:05:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479786+0000", 2024-08-17T18:05:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8529999999999998e-06 2024-08-17T18:05:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.860711+0000", 2024-08-17T18:05:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.38092492 2024-08-17T18:05:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:05:27.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.867150+0000", 2024-08-17T18:05:27.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064388700000000002 2024-08-17T18:05:27.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.900225+0000", 2024-08-17T18:05:27.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.03307529 2024-08-17T18:05:27.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.902023+0000", 2024-08-17T18:05:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001798231 2024-08-17T18:05:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:05:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.906038+0000", 2024-08-17T18:05:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040151620000000001 2024-08-17T18:05:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.919715+0000", 2024-08-17T18:05:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013676364999999999 2024-08-17T18:05:27.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.935483+0000", 2024-08-17T18:05:27.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015768372999999999 2024-08-17T18:05:27.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.936289+0000", 2024-08-17T18:05:27.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00080628999999999998 2024-08-17T18:05:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:05:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.937359+0000", 2024-08-17T18:05:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001069293 2024-08-17T18:05:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.941492+0000", 2024-08-17T18:05:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041333460000000004 2024-08-17T18:05:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.307 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180198 54.10 54:0a39a81d:::benchmark_data_smithi136_273969_object57313:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:05:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:21.961999+0000", 2024-08-17T18:05:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 245.28883866699999, 2024-08-17T18:05:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9727821460000001, 2024-08-17T18:05:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:05:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:27.308 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180198 2024-08-17T18:05:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.961999+0000", 2024-08-17T18:05:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:27.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.961999+0000", 2024-08-17T18:05:27.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:27.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.961998+0000", 2024-08-17T18:05:27.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:05:27.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:27.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.962003+0000", 2024-08-17T18:05:27.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3430000000000003e-06 2024-08-17T18:05:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.962017+0000", 2024-08-17T18:05:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3643e-05 2024-08-17T18:05:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.962024+0000", 2024-08-17T18:05:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.849e-06 2024-08-17T18:05:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.035869+0000", 2024-08-17T18:05:27.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.073845659999999994 2024-08-17T18:05:27.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:05:27.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.039312+0000", 2024-08-17T18:05:27.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003442905 2024-08-17T18:05:27.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.058190+0000", 2024-08-17T18:05:27.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018877535000000001 2024-08-17T18:05:27.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:05:27.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.061806+0000", 2024-08-17T18:05:27.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003616452 2024-08-17T18:05:27.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542821+0000", 2024-08-17T18:05:27.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.48101451699999997 2024-08-17T18:05:27.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:27.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542827+0000", 2024-08-17T18:05:27.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9739999999999999e-06 2024-08-17T18:05:27.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479734+0000", 2024-08-17T18:05:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93690761600000005 2024-08-17T18:05:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479743+0000", 2024-08-17T18:05:27.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9619999999999999e-06 2024-08-17T18:05:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.854680+0000", 2024-08-17T18:05:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3749364169999998 2024-08-17T18:05:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:05:27.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.860366+0000", 2024-08-17T18:05:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056863520000000004 2024-08-17T18:05:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.882033+0000", 2024-08-17T18:05:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021666920999999999 2024-08-17T18:05:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.883991+0000", 2024-08-17T18:05:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001958146 2024-08-17T18:05:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:05:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.888022+0000", 2024-08-17T18:05:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040313129999999999 2024-08-17T18:05:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.909504+0000", 2024-08-17T18:05:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021482015 2024-08-17T18:05:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.910210+0000", 2024-08-17T18:05:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00070519400000000002 2024-08-17T18:05:27.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.319 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.929264+0000", 2024-08-17T18:05:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019054252000000001 2024-08-17T18:05:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:05:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.930507+0000", 2024-08-17T18:05:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012434029999999999 2024-08-17T18:05:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.934781+0000", 2024-08-17T18:05:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042741080000000004 2024-08-17T18:05:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180233 54.10 54:08986526:::benchmark_data_smithi136_273969_object57348:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:05:27.321 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:22.029556+0000", 2024-08-17T18:05:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 245.22128171, 2024-08-17T18:05:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9954107949999997, 2024-08-17T18:05:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:05:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180233 2024-08-17T18:05:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.029556+0000", 2024-08-17T18:05:27.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.029556+0000", 2024-08-17T18:05:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.029555+0000", 2024-08-17T18:05:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:05:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.029561+0000", 2024-08-17T18:05:27.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0349999999999998e-06 2024-08-17T18:05:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.029575+0000", 2024-08-17T18:05:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4566000000000001e-05 2024-08-17T18:05:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.029583+0000", 2024-08-17T18:05:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9330000000000001e-06 2024-08-17T18:05:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.040674+0000", 2024-08-17T18:05:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011090546999999999 2024-08-17T18:05:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:05:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.044717+0000", 2024-08-17T18:05:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040431340000000003 2024-08-17T18:05:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.075210+0000", 2024-08-17T18:05:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.030493089000000001 2024-08-17T18:05:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:05:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.078485+0000", 2024-08-17T18:05:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032751820000000002 2024-08-17T18:05:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542915+0000", 2024-08-17T18:05:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.46442982199999999 2024-08-17T18:05:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542921+0000", 2024-08-17T18:05:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7710000000000003e-06 2024-08-17T18:05:27.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479864+0000", 2024-08-17T18:05:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93694376899999998 2024-08-17T18:05:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479873+0000", 2024-08-17T18:05:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7870000000000007e-06 2024-08-17T18:05:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.948551+0000", 2024-08-17T18:05:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4686778700000001 2024-08-17T18:05:27.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:05:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.954417+0000", 2024-08-17T18:05:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058657630000000004 2024-08-17T18:05:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.974711+0000", 2024-08-17T18:05:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.02029392 2024-08-17T18:05:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.976548+0000", 2024-08-17T18:05:27.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018375240000000001 2024-08-17T18:05:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:05:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.980434+0000", 2024-08-17T18:05:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038862089999999998 2024-08-17T18:05:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.000907+0000", 2024-08-17T18:05:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020472305999999999 2024-08-17T18:05:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.001622+0000", 2024-08-17T18:05:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00071567799999999997 2024-08-17T18:05:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.019920+0000", 2024-08-17T18:05:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018297686000000001 2024-08-17T18:05:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:05:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.020902+0000", 2024-08-17T18:05:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00098207099999999994 2024-08-17T18:05:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.024967+0000", 2024-08-17T18:05:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040647970000000002 2024-08-17T18:05:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180273 54.10 54:0f06273e:::benchmark_data_smithi136_273969_object57388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:05:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:22.143352+0000", 2024-08-17T18:05:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 245.10748637699999, 2024-08-17T18:05:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7034936619999996, 2024-08-17T18:05:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:05:27.331 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180273 2024-08-17T18:05:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.143352+0000", 2024-08-17T18:05:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.143352+0000", 2024-08-17T18:05:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.143350+0000", 2024-08-17T18:05:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:05:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.143356+0000", 2024-08-17T18:05:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.575e-06 2024-08-17T18:05:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.143367+0000", 2024-08-17T18:05:27.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.116e-05 2024-08-17T18:05:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.143373+0000", 2024-08-17T18:05:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5450000000000003e-06 2024-08-17T18:05:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.163881+0000", 2024-08-17T18:05:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020508683 2024-08-17T18:05:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:05:27.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.167050+0000", 2024-08-17T18:05:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031688659999999998 2024-08-17T18:05:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.192396+0000", 2024-08-17T18:05:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.025345660999999998 2024-08-17T18:05:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:05:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.196588+0000", 2024-08-17T18:05:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041921679999999996 2024-08-17T18:05:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542759+0000", 2024-08-17T18:05:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.34617123100000002 2024-08-17T18:05:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542766+0000", 2024-08-17T18:05:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0999999999999998e-06 2024-08-17T18:05:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479644+0000", 2024-08-17T18:05:27.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93687755900000003 2024-08-17T18:05:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479655+0000", 2024-08-17T18:05:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1167e-05 2024-08-17T18:05:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.763829+0000", 2024-08-17T18:05:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2841733739999999 2024-08-17T18:05:27.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:05:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.769270+0000", 2024-08-17T18:05:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005441085 2024-08-17T18:05:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.804951+0000", 2024-08-17T18:05:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035681601 2024-08-17T18:05:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:27.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.806790+0000", 2024-08-17T18:05:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018387869999999999 2024-08-17T18:05:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:05:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.810598+0000", 2024-08-17T18:05:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038084780000000001 2024-08-17T18:05:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.823793+0000", 2024-08-17T18:05:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013195014 2024-08-17T18:05:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.840554+0000", 2024-08-17T18:05:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016760048999999999 2024-08-17T18:05:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.841188+0000", 2024-08-17T18:05:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063486499999999997 2024-08-17T18:05:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:05:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.842214+0000", 2024-08-17T18:05:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001025417 2024-08-17T18:05:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.846845+0000", 2024-08-17T18:05:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00463139 2024-08-17T18:05:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.341 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180362 54.10 54:0f5741ac:::benchmark_data_smithi136_273969_object57477:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:05:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:22.267742+0000", 2024-08-17T18:05:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 244.98309564799999, 2024-08-17T18:05:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5718885580000004, 2024-08-17T18:05:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:05:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:27.342 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180362 2024-08-17T18:05:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.267742+0000", 2024-08-17T18:05:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.267742+0000", 2024-08-17T18:05:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.267741+0000", 2024-08-17T18:05:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:05:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.267745+0000", 2024-08-17T18:05:27.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6969999999999999e-06 2024-08-17T18:05:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.267753+0000", 2024-08-17T18:05:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2099999999999993e-06 2024-08-17T18:05:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.267759+0000", 2024-08-17T18:05:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.293e-06 2024-08-17T18:05:27.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.297122+0000", 2024-08-17T18:05:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.029363607999999999 2024-08-17T18:05:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:05:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.300774+0000", 2024-08-17T18:05:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036522759999999999 2024-08-17T18:05:27.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.328199+0000", 2024-08-17T18:05:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.027424474000000001 2024-08-17T18:05:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:05:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.332945+0000", 2024-08-17T18:05:27.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004745567 2024-08-17T18:05:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542721+0000", 2024-08-17T18:05:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.20977694799999999 2024-08-17T18:05:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:27.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542728+0000", 2024-08-17T18:05:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7900000000000002e-06 2024-08-17T18:05:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479480+0000", 2024-08-17T18:05:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93675217300000002 2024-08-17T18:05:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:27.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479490+0000", 2024-08-17T18:05:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2070000000000001e-06 2024-08-17T18:05:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.757950+0000", 2024-08-17T18:05:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2784603570000002 2024-08-17T18:05:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:05:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.763478+0000", 2024-08-17T18:05:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055275100000000002 2024-08-17T18:05:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.784742+0000", 2024-08-17T18:05:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021264387999999999 2024-08-17T18:05:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.787023+0000", 2024-08-17T18:05:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022810869999999998 2024-08-17T18:05:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:05:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.791977+0000", 2024-08-17T18:05:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049536820000000001 2024-08-17T18:05:27.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.804625+0000", 2024-08-17T18:05:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012648453 2024-08-17T18:05:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.814553+0000", 2024-08-17T18:05:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0099276380000000008 2024-08-17T18:05:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.834303+0000", 2024-08-17T18:05:27.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01975037 2024-08-17T18:05:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:05:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.835353+0000", 2024-08-17T18:05:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010493869999999999 2024-08-17T18:05:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.839631+0000", 2024-08-17T18:05:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042783090000000001 2024-08-17T18:05:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180378 54.10 54:08c0d9ad:::benchmark_data_smithi136_273969_object57493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:05:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:22.286706+0000", 2024-08-17T18:05:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 244.96413208800001, 2024-08-17T18:05:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.661461214, 2024-08-17T18:05:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:05:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180378 2024-08-17T18:05:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.354 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.286706+0000", 2024-08-17T18:05:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.286706+0000", 2024-08-17T18:05:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.286704+0000", 2024-08-17T18:05:27.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:05:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.286712+0000", 2024-08-17T18:05:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7980000000000006e-06 2024-08-17T18:05:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.286727+0000", 2024-08-17T18:05:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4842e-05 2024-08-17T18:05:27.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.286735+0000", 2024-08-17T18:05:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6899999999999992e-06 2024-08-17T18:05:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.358257+0000", 2024-08-17T18:05:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.071522756000000007 2024-08-17T18:05:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:05:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.362217+0000", 2024-08-17T18:05:27.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039599190000000001 2024-08-17T18:05:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.417476+0000", 2024-08-17T18:05:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.055258920000000003 2024-08-17T18:05:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:05:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.421718+0000", 2024-08-17T18:05:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042413160000000002 2024-08-17T18:05:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542885+0000", 2024-08-17T18:05:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.121167504 2024-08-17T18:05:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542890+0000", 2024-08-17T18:05:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8679999999999998e-06 2024-08-17T18:05:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479822+0000", 2024-08-17T18:05:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93693204200000002 2024-08-17T18:05:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479834+0000", 2024-08-17T18:05:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2006e-05 2024-08-17T18:05:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.867421+0000", 2024-08-17T18:05:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3875873570000001 2024-08-17T18:05:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:05:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.873105+0000", 2024-08-17T18:05:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056837179999999999 2024-08-17T18:05:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.920008+0000", 2024-08-17T18:05:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.046902417000000002 2024-08-17T18:05:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.921823+0000", 2024-08-17T18:05:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001815821 2024-08-17T18:05:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:05:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.925572+0000", 2024-08-17T18:05:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037488589999999998 2024-08-17T18:05:27.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.935695+0000", 2024-08-17T18:05:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010123036 2024-08-17T18:05:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.942233+0000", 2024-08-17T18:05:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0065380569999999999 2024-08-17T18:05:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.942843+0000", 2024-08-17T18:05:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00060995600000000004 2024-08-17T18:05:27.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:05:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.943910+0000", 2024-08-17T18:05:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010668450000000001 2024-08-17T18:05:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.948167+0000", 2024-08-17T18:05:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042569469999999996 2024-08-17T18:05:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180538 54.2 54:4084e074:::benchmark_data_smithi136_273969_object57653:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:05:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:22.472724+0000", 2024-08-17T18:05:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 244.77811394700001, 2024-08-17T18:05:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.083849345, 2024-08-17T18:05:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:05:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180538 2024-08-17T18:05:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:27.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.472724+0000", 2024-08-17T18:05:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.472724+0000", 2024-08-17T18:05:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.472723+0000", 2024-08-17T18:05:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:05:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.472729+0000", 2024-08-17T18:05:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3400000000000003e-06 2024-08-17T18:05:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.472742+0000", 2024-08-17T18:05:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2495e-05 2024-08-17T18:05:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.472749+0000", 2024-08-17T18:05:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9909999999999996e-06 2024-08-17T18:05:27.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.472789+0000", 2024-08-17T18:05:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9968000000000001e-05 2024-08-17T18:05:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.472896+0000", 2024-08-17T18:05:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010768900000000001 2024-08-17T18:05:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 6", 2024-08-17T18:05:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.473437+0000", 2024-08-17T18:05:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00054072199999999999 2024-08-17T18:05:27.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.483366+0000", 2024-08-17T18:05:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0099294789999999997 2024-08-17T18:05:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.483382+0000", 2024-08-17T18:05:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5379999999999998e-05 2024-08-17T18:05:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.480196+0000", 2024-08-17T18:05:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99681430500000001 2024-08-17T18:05:27.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.480206+0000", 2024-08-17T18:05:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0399e-05 2024-08-17T18:05:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.552647+0000", 2024-08-17T18:05:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0724403659999999 2024-08-17T18:05:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:05:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.552777+0000", 2024-08-17T18:05:27.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013025999999999999 2024-08-17T18:05:27.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.556429+0000", 2024-08-17T18:05:27.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003651886 2024-08-17T18:05:27.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:27.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.556573+0000", 2024-08-17T18:05:27.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000144325 2024-08-17T18:05:27.372 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.372 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:27.372 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.372 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180549 54.10 54:0ef7c369:::benchmark_data_smithi136_273969_object57664:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1399)", 2024-08-17T18:05:27.372 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:22.501329+0000", 2024-08-17T18:05:27.372 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 244.74950905599999, 2024-08-17T18:05:27.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5377924729999997, 2024-08-17T18:05:27.372 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:27.373 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:05:27.373 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:27.373 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:27.373 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:27.373 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180549 2024-08-17T18:05:27.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.373 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:27.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:27.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.501329+0000", 2024-08-17T18:05:27.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:27.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.501329+0000", 2024-08-17T18:05:27.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:27.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.501327+0000", 2024-08-17T18:05:27.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:05:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.501334+0000", 2024-08-17T18:05:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.764e-06 2024-08-17T18:05:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.501347+0000", 2024-08-17T18:05:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2947e-05 2024-08-17T18:05:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:27.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.501354+0000", 2024-08-17T18:05:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8469999999999998e-06 2024-08-17T18:05:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542971+0000", 2024-08-17T18:05:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.041617526000000002 2024-08-17T18:05:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542977+0000", 2024-08-17T18:05:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.288e-06 2024-08-17T18:05:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479935+0000", 2024-08-17T18:05:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93695844900000003 2024-08-17T18:05:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479943+0000", 2024-08-17T18:05:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6909999999999997e-06 2024-08-17T18:05:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.960597+0000", 2024-08-17T18:05:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.480653856 2024-08-17T18:05:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:05:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.965790+0000", 2024-08-17T18:05:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051937190000000003 2024-08-17T18:05:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.010563+0000", 2024-08-17T18:05:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.044773043999999998 2024-08-17T18:05:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.012451+0000", 2024-08-17T18:05:27.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001887108 2024-08-17T18:05:27.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:05:27.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.016301+0000", 2024-08-17T18:05:27.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038504619999999998 2024-08-17T18:05:27.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:27.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.032206+0000", 2024-08-17T18:05:27.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015904795999999999 2024-08-17T18:05:27.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.032839+0000", 2024-08-17T18:05:27.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063271599999999996 2024-08-17T18:05:27.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.033465+0000", 2024-08-17T18:05:27.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062640899999999997 2024-08-17T18:05:27.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:05:27.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.034490+0000", 2024-08-17T18:05:27.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001025443 2024-08-17T18:05:27.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:27.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.039121+0000", 2024-08-17T18:05:27.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046309389999999997 2024-08-17T18:05:27.381 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.381 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:27.381 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.381 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180580 54.10 54:08b7bb9d:::benchmark_data_smithi136_273969_object57695:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1401)", 2024-08-17T18:05:27.381 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:24.596431+0000", 2024-08-17T18:05:27.381 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 242.65440700100001, 2024-08-17T18:05:27.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6123190539999999, 2024-08-17T18:05:27.382 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:27.382 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:05:27.382 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:27.382 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:27.382 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:27.382 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180580 2024-08-17T18:05:27.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.596431+0000", 2024-08-17T18:05:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.596431+0000", 2024-08-17T18:05:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.596429+0000", 2024-08-17T18:05:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:05:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.596436+0000", 2024-08-17T18:05:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2740000000000004e-06 2024-08-17T18:05:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:27.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.596449+0000", 2024-08-17T18:05:27.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3514e-05 2024-08-17T18:05:27.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:27.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.596460+0000", 2024-08-17T18:05:27.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1038e-05 2024-08-17T18:05:27.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.039419+0000", 2024-08-17T18:05:27.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.44295865 2024-08-17T18:05:27.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:05:27.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.042353+0000", 2024-08-17T18:05:27.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002934641 2024-08-17T18:05:27.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.060793+0000", 2024-08-17T18:05:27.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018439779 2024-08-17T18:05:27.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:05:27.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.062127+0000", 2024-08-17T18:05:27.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001334212 2024-08-17T18:05:27.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.189141+0000", 2024-08-17T18:05:27.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.127013878 2024-08-17T18:05:27.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:27.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.190968+0000", 2024-08-17T18:05:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001826261 2024-08-17T18:05:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:05:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.194933+0000", 2024-08-17T18:05:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039653300000000004 2024-08-17T18:05:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.201866+0000", 2024-08-17T18:05:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069329919999999998 2024-08-17T18:05:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.202483+0000", 2024-08-17T18:05:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00061715800000000003 2024-08-17T18:05:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.202683+0000", 2024-08-17T18:05:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000199507 2024-08-17T18:05:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:05:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.203739+0000", 2024-08-17T18:05:27.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010565570000000001 2024-08-17T18:05:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.208750+0000", 2024-08-17T18:05:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005010848 2024-08-17T18:05:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180607 54.10 54:0f7a5843:::benchmark_data_smithi136_273969_object57722:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1401)", 2024-08-17T18:05:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:24.627171+0000", 2024-08-17T18:05:27.391 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 242.62366731200001, 2024-08-17T18:05:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6376815360000001, 2024-08-17T18:05:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:05:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180607 2024-08-17T18:05:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.627171+0000", 2024-08-17T18:05:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.627171+0000", 2024-08-17T18:05:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.627169+0000", 2024-08-17T18:05:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:05:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.627175+0000", 2024-08-17T18:05:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3569999999999997e-06 2024-08-17T18:05:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.627185+0000", 2024-08-17T18:05:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0281e-05 2024-08-17T18:05:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.627192+0000", 2024-08-17T18:05:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4899999999999997e-06 2024-08-17T18:05:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.042641+0000", 2024-08-17T18:05:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4154490289999999 2024-08-17T18:05:27.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:05:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.045434+0000", 2024-08-17T18:05:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027932120000000002 2024-08-17T18:05:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.084159+0000", 2024-08-17T18:05:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038724959000000003 2024-08-17T18:05:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:05:27.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.085413+0000", 2024-08-17T18:05:27.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001254072 2024-08-17T18:05:27.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.239127+0000", 2024-08-17T18:05:27.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15371395600000001 2024-08-17T18:05:27.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:27.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.240998+0000", 2024-08-17T18:05:27.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001870752 2024-08-17T18:05:27.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:05:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.245523+0000", 2024-08-17T18:05:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045257980000000001 2024-08-17T18:05:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.257833+0000", 2024-08-17T18:05:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012309672000000001 2024-08-17T18:05:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.258606+0000", 2024-08-17T18:05:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00077326900000000004 2024-08-17T18:05:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.259327+0000", 2024-08-17T18:05:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00072101700000000001 2024-08-17T18:05:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:05:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.260405+0000", 2024-08-17T18:05:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001077176 2024-08-17T18:05:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:27.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.264852+0000", 2024-08-17T18:05:27.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044476170000000001 2024-08-17T18:05:27.400 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.400 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:27.400 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.400 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180664 54.10 54:0d9859e3:::benchmark_data_smithi136_273969_object57779:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1401)", 2024-08-17T18:05:27.400 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:24.698340+0000", 2024-08-17T18:05:27.400 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 242.55249774000001, 2024-08-17T18:05:27.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5592268169999999, 2024-08-17T18:05:27.400 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:27.400 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:05:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180664 2024-08-17T18:05:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.698340+0000", 2024-08-17T18:05:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.698340+0000", 2024-08-17T18:05:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.698339+0000", 2024-08-17T18:05:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:05:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.698345+0000", 2024-08-17T18:05:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3289999999999999e-06 2024-08-17T18:05:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.698359+0000", 2024-08-17T18:05:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3645999999999999e-05 2024-08-17T18:05:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.698365+0000", 2024-08-17T18:05:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7120000000000003e-06 2024-08-17T18:05:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.046290+0000", 2024-08-17T18:05:27.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.347924678 2024-08-17T18:05:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:05:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.048759+0000", 2024-08-17T18:05:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002469097 2024-08-17T18:05:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.072793+0000", 2024-08-17T18:05:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.024033968999999999 2024-08-17T18:05:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:05:27.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.074098+0000", 2024-08-17T18:05:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001304454 2024-08-17T18:05:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.220487+0000", 2024-08-17T18:05:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.14638969499999999 2024-08-17T18:05:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.222607+0000", 2024-08-17T18:05:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021193789999999998 2024-08-17T18:05:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:05:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.226749+0000", 2024-08-17T18:05:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041426019999999996 2024-08-17T18:05:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.249844+0000", 2024-08-17T18:05:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023095075 2024-08-17T18:05:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.250114+0000", 2024-08-17T18:05:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027006099999999999 2024-08-17T18:05:27.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.250948+0000", 2024-08-17T18:05:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00083311500000000005 2024-08-17T18:05:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:05:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.252283+0000", 2024-08-17T18:05:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013353950000000001 2024-08-17T18:05:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:27.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.257567+0000", 2024-08-17T18:05:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052840910000000003 2024-08-17T18:05:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180787 54.10 54:0e855b4b:::benchmark_data_smithi136_273969_object57902:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1401)", 2024-08-17T18:05:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:24.891973+0000", 2024-08-17T18:05:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 242.35886481599999, 2024-08-17T18:05:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4466802329999999, 2024-08-17T18:05:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:05:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:27.408 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:27.409 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:27.409 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180787 2024-08-17T18:05:27.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.409 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:27.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:27.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.891973+0000", 2024-08-17T18:05:27.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.891973+0000", 2024-08-17T18:05:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.891972+0000", 2024-08-17T18:05:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:05:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.891978+0000", 2024-08-17T18:05:27.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.489e-06 2024-08-17T18:05:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.891989+0000", 2024-08-17T18:05:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0814e-05 2024-08-17T18:05:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.891994+0000", 2024-08-17T18:05:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5849999999999999e-06 2024-08-17T18:05:27.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.086323+0000", 2024-08-17T18:05:27.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1943286199999998 2024-08-17T18:05:27.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:05:27.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.088618+0000", 2024-08-17T18:05:27.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022949540000000001 2024-08-17T18:05:27.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.103459+0000", 2024-08-17T18:05:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014841294 2024-08-17T18:05:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:05:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.104663+0000", 2024-08-17T18:05:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001203204 2024-08-17T18:05:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.281808+0000", 2024-08-17T18:05:27.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17714501499999999 2024-08-17T18:05:27.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:27.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.284980+0000", 2024-08-17T18:05:27.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031723739999999999 2024-08-17T18:05:27.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:05:27.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.289946+0000", 2024-08-17T18:05:27.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049663190000000003 2024-08-17T18:05:27.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.313491+0000", 2024-08-17T18:05:27.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023544820000000001 2024-08-17T18:05:27.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:27.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.313720+0000", 2024-08-17T18:05:27.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022937200000000001 2024-08-17T18:05:27.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.333229+0000", 2024-08-17T18:05:27.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019508584999999998 2024-08-17T18:05:27.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:05:27.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.334230+0000", 2024-08-17T18:05:27.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010011379999999999 2024-08-17T18:05:27.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:27.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.338653+0000", 2024-08-17T18:05:27.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044231590000000003 2024-08-17T18:05:27.417 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.417 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:27.417 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.417 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180832 54.10 54:0c4a64fd:::benchmark_data_smithi136_273969_object57947:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1401)", 2024-08-17T18:05:27.417 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:24.976914+0000", 2024-08-17T18:05:27.417 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 242.27392422599999, 2024-08-17T18:05:27.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3778381899999999, 2024-08-17T18:05:27.418 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:27.418 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:05:27.418 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:27.418 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:27.418 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:27.418 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180832 2024-08-17T18:05:27.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.418 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:27.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:27.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.976914+0000", 2024-08-17T18:05:27.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:27.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.976914+0000", 2024-08-17T18:05:27.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:27.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.976912+0000", 2024-08-17T18:05:27.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:05:27.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:27.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.976918+0000", 2024-08-17T18:05:27.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9540000000000001e-06 2024-08-17T18:05:27.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:27.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.976932+0000", 2024-08-17T18:05:27.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3726999999999999e-05 2024-08-17T18:05:27.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:27.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.976939+0000", 2024-08-17T18:05:27.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6499999999999999e-06 2024-08-17T18:05:27.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.088884+0000", 2024-08-17T18:05:27.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.111945757 2024-08-17T18:05:27.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:05:27.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.091124+0000", 2024-08-17T18:05:27.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022394630000000001 2024-08-17T18:05:27.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.123802+0000", 2024-08-17T18:05:27.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.032678076 2024-08-17T18:05:27.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:05:27.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.125022+0000", 2024-08-17T18:05:27.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012200049999999999 2024-08-17T18:05:27.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.322996+0000", 2024-08-17T18:05:27.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.19797377199999999 2024-08-17T18:05:27.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:27.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.324908+0000", 2024-08-17T18:05:27.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019125119999999999 2024-08-17T18:05:27.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:05:27.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.328896+0000", 2024-08-17T18:05:27.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039873460000000001 2024-08-17T18:05:27.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:27.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.339470+0000", 2024-08-17T18:05:27.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010574225 2024-08-17T18:05:27.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.346638+0000", 2024-08-17T18:05:27.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071680119999999996 2024-08-17T18:05:27.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.347438+0000", 2024-08-17T18:05:27.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00079986399999999996 2024-08-17T18:05:27.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:05:27.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.348684+0000", 2024-08-17T18:05:27.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012465530000000001 2024-08-17T18:05:27.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:27.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.354752+0000", 2024-08-17T18:05:27.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060676560000000003 2024-08-17T18:05:27.426 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.426 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:27.426 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.426 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180936 54.10 54:0b5f349a:::benchmark_data_smithi136_273969_object58051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1401)", 2024-08-17T18:05:27.426 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:25.343460+0000", 2024-08-17T18:05:27.426 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 241.90737759699999, 2024-08-17T18:05:27.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0025335470000001, 2024-08-17T18:05:27.426 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:27.426 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:05:27.426 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:27.427 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:27.427 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:27.427 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180936 2024-08-17T18:05:27.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.427 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:27.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:27.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.343460+0000", 2024-08-17T18:05:27.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:27.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.343460+0000", 2024-08-17T18:05:27.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:27.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.343459+0000", 2024-08-17T18:05:27.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:05:27.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:27.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.343466+0000", 2024-08-17T18:05:27.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8820000000000001e-06 2024-08-17T18:05:27.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:27.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.343479+0000", 2024-08-17T18:05:27.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.306e-05 2024-08-17T18:05:27.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:27.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.343488+0000", 2024-08-17T18:05:27.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6330000000000002e-06 2024-08-17T18:05:27.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.091459+0000", 2024-08-17T18:05:27.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7479711390000001 2024-08-17T18:05:27.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:05:27.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.093708+0000", 2024-08-17T18:05:27.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022491490000000002 2024-08-17T18:05:27.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.113505+0000", 2024-08-17T18:05:27.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019796971999999999 2024-08-17T18:05:27.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:05:27.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.114725+0000", 2024-08-17T18:05:27.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012199940000000001 2024-08-17T18:05:27.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.303446+0000", 2024-08-17T18:05:27.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18872046000000001 2024-08-17T18:05:27.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:27.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.305293+0000", 2024-08-17T18:05:27.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018468219999999999 2024-08-17T18:05:27.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:05:27.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.309465+0000", 2024-08-17T18:05:27.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041719280000000001 2024-08-17T18:05:27.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:27.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.332589+0000", 2024-08-17T18:05:27.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023124150999999999 2024-08-17T18:05:27.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.339278+0000", 2024-08-17T18:05:27.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006689632 2024-08-17T18:05:27.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:27.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.340076+0000", 2024-08-17T18:05:27.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00079814199999999995 2024-08-17T18:05:27.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:05:27.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.341136+0000", 2024-08-17T18:05:27.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010596169999999999 2024-08-17T18:05:27.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:27.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.345994+0000", 2024-08-17T18:05:27.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004857765 2024-08-17T18:05:27.433 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.433 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:27.433 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.434 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGRecoveryDeleteReply(54.10 e1401,1400 54:0f9da3af:::benchmark_data_smithi136_273969_object57231:head,1398'8529)", 2024-08-17T18:05:27.434 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:25.545611+0000", 2024-08-17T18:05:27.434 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 241.70522738099999, 2024-08-17T18:05:27.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.201322319, 2024-08-17T18:05:27.434 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:27.434 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:05:27.434 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:27.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:27.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.545611+0000", 2024-08-17T18:05:27.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:27.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.545611+0000", 2024-08-17T18:05:27.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:27.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.545610+0000", 2024-08-17T18:05:27.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:05:27.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:27.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.545620+0000", 2024-08-17T18:05:27.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0526000000000001e-05 2024-08-17T18:05:27.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:27.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.545632+0000", 2024-08-17T18:05:27.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1771e-05 2024-08-17T18:05:27.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:27.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.545641+0000", 2024-08-17T18:05:27.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.568e-06 2024-08-17T18:05:27.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.743396+0000", 2024-08-17T18:05:27.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.197755677 2024-08-17T18:05:27.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:27.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.746933+0000", 2024-08-17T18:05:27.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035364770000000001 2024-08-17T18:05:27.437 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.437 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:27.437 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.437 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGRecoveryDeleteReply(54.10 e1401,1400 54:0f9da3af:::benchmark_data_smithi136_273969_object57231:head,1398'8529)", 2024-08-17T18:05:27.437 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:25.546013+0000", 2024-08-17T18:05:27.437 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 241.704824836, 2024-08-17T18:05:27.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.209840958, 2024-08-17T18:05:27.437 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:27.437 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:05:27.437 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:27.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:27.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.546013+0000", 2024-08-17T18:05:27.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:27.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.546013+0000", 2024-08-17T18:05:27.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:27.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:27.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.546012+0000", 2024-08-17T18:05:27.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:05:27.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:27.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.546017+0000", 2024-08-17T18:05:27.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5430000000000004e-06 2024-08-17T18:05:27.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:27.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.546028+0000", 2024-08-17T18:05:27.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1343e-05 2024-08-17T18:05:27.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:27.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:25.546034+0000", 2024-08-17T18:05:27.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.215e-06 2024-08-17T18:05:27.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:27.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.747175+0000", 2024-08-17T18:05:27.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.201141062 2024-08-17T18:05:27.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:27.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:27.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:27.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.755854+0000", 2024-08-17T18:05:27.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0086793800000000004 2024-08-17T18:05:27.440 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.440 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:27.440 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.440 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:27.440 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:27.441 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:05:27.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:27.442 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:27.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:27.537 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:05:27.543 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:27.552 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:05:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:05:27] "GET /metrics HTTP/1.1" 200 36619 "" "Prometheus/2.51.0" 2024-08-17T18:05:27.572 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:27.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:27.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:27.674 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:27.695 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:27.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:27.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:27.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:27.821 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:27.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:27.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:27.919 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:27 smithi136 ceph-mon[109861]: pgmap v2615: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 12 MiB/s wr, 199 op/s; 26292/272631 objects degraded (9.644%); 6174/272631 objects misplaced (2.265%); 2.6 MiB/s, 42 objects/s recovering 2024-08-17T18:05:27.920 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:27 smithi136 ceph-mon[115078]: pgmap v2615: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 12 MiB/s wr, 199 op/s; 26292/272631 objects degraded (9.644%); 6174/272631 objects misplaced (2.265%); 2.6 MiB/s, 42 objects/s recovering 2024-08-17T18:05:27.923 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:27.948 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:27.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:27.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:28.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:27 smithi165 ceph-mon[105517]: pgmap v2615: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 12 MiB/s wr, 199 op/s; 26292/272631 objects degraded (9.644%); 6174/272631 objects misplaced (2.265%); 2.6 MiB/s, 42 objects/s recovering 2024-08-17T18:05:28.050 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:28.073 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:28.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:28.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:28.175 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:28.200 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:28.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:28.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:28.294 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:05:28.301 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:28.332 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:28.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:28.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:28.433 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:28.461 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:28.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:28.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:28.563 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:28.589 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:28.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:28.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:28.691 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:28.713 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:28.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:28.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:28.816 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:28.841 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:28.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:28.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:28.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:28.968 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:28.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:28.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:29.070 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:29.093 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:29.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:29.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:29.195 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:29.221 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:05:29.221 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:29.222 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T18:05:29.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:29.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:29.326 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:29.347 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:29.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:29.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:29.448 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:29.457 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-17T18:05:29.470 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:29.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:29.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:29.572 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:29.611 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:29.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:29.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:29.713 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:29.738 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:29.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:29.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:29.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:29 smithi165 ceph-mon[105517]: pgmap v2616: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 127 B/s rd, 19 MiB/s wr, 342 op/s; 26229/274977 objects degraded (9.539%); 5675/274977 objects misplaced (2.064%); 4.6 MiB/s, 74 objects/s recovering 2024-08-17T18:05:29.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:29 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:05:29.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:29 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:05:29.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:29 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 26229/274977 objects degraded (9.539%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:29.774 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:05:29.840 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:29.866 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:29.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:29.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:29.944 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:29 smithi136 ceph-mon[115078]: pgmap v2616: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 127 B/s rd, 19 MiB/s wr, 342 op/s; 26229/274977 objects degraded (9.539%); 5675/274977 objects misplaced (2.064%); 4.6 MiB/s, 74 objects/s recovering 2024-08-17T18:05:29.945 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:29 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:05:29.945 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:29 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:05:29.945 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:29 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 26229/274977 objects degraded (9.539%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:29.945 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:29 smithi136 ceph-mon[109861]: pgmap v2616: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 127 B/s rd, 19 MiB/s wr, 342 op/s; 26229/274977 objects degraded (9.539%); 5675/274977 objects misplaced (2.064%); 4.6 MiB/s, 74 objects/s recovering 2024-08-17T18:05:29.945 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:29 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:05:29.945 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:29 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:05:29.945 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:29 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 26229/274977 objects degraded (9.539%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:29.968 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:29.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:29.994 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:29.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:30.095 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:30.118 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:30.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:30.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:30.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:30.260 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:30.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:30.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:30.362 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:30.392 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:30.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:30.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:30.493 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:30.523 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:30.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:30.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:30.618 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":2616,"stamp":"2024-08-17T18:05:28.231496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6005790717,"num_objects":91659,"num_object_clones":0,"num_object_copies":274977,"num_objects_missing_on_primary":4074,"num_objects_missing":4074,"num_objects_degraded":26229,"num_objects_misplaced":5675,"num_objects_unfound":0,"num_objects_dirty":91659,"num_whiteouts":0,"num_read":379,"num_read_kb":823,"num_write":92536,"num_write_kb":5886866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6217,"num_bytes_recovered":405742420,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148904,"ondisk_log_size":148904,"up":297,"acting":293,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":319,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":21926496,"kb_used_data":18223268,"kb_used_omap":350,"kb_used_meta":3702689,"kb_avail":634289568,"statfs":{"total":671965249536,"available":649512517632,"internally_reserved":0,"allocated":18660626432,"data_stored":18625355747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":358888,"internal_metadata":3791554072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":138608664,"num_objects":2116,"num_object_clones":0,"num_object_copies":6348,"num_objects_missing_on_primary":4074,"num_objects_missing":4074,"num_objects_degraded":3261,"num_objects_misplaced":2198,"num_objects_unfound":0,"num_objects_dirty":2116,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2740,"num_write_kb":155329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":594,"num_bytes_recovered":38928384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001990"},"pg_stats":[{"pgid":"55.19","version":"1476'5654","reported_seq":5748,"reported_epoch":1476,"state":"active+recovery_wait+degraded","last_fresh":"2024-08-17T18:05:26.522525+0000","last_change":"2024-08-17T18:05:19.768391+0000","last_active":"2024-08-17T18:05:26.522525+0000","last_peered":"2024-08-17T18:05:26.522525+0000","last_clean":"2024-08-17T18:04:42.666472+0000","last_became_active":"2024-08-17T18:05:19.664475+0000","last_became_peered":"2024-08-17T18:05:19.664475+0000","last_unstale":"2024-08-17T18:05:26.522525+0000","last_undegraded":"2024-08-17T18:05:19.597318+0000","last_fullsized":"2024-08-17T18:05:26.522525+0000","mapping_epoch":1472,"log_start":"0'0","ondisk_log_start":"0'0","created":1472,"last_epoch_clean":1447,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5654,"log_dups_size":0,"ondisk_log_size":5654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185860096,"num_objects":2836,"num_object_clones":0,"num_object_copies":8508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2854,"num_write_kb":182080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":589824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":["3","2","4"],"object_location_counts":[{"shards":"2,3,4","objects":1974},{"shards":"2,3,5","objects":862}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.18","version":"1476'5743","reported_seq":5815,"reported_epoch":1476,"state":"active+recovery_wait+degraded","last_fresh":"2024-08-17T18:05:24.557450+0000","last_change":"2024-08-17T18:05:23.137482+0000","last_active":"2024-08-17T18:05:24.557450+0000","last_peered":"2024-08-17T18:05:24.557450+0000","last_clean":"2024-08-17T18:04:14.322941+0000","last_became_active":"2024-08-17T18:05:23.106041+0000","last_became_peered":"2024-08-17T18:05:23.106041+0000","last_unstale":"2024-08-17T18:05:24.557450+0000","last_undegraded":"2024-08-17T18:05:20.630708+0000","last_fullsized":"2024-08-17T18:05:24.557450+0000","mapping_epoch":1472,"log_start":"0'0","ondisk_log_start":"0'0","created":1472,"last_epoch_clean":1447,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5743,"log_dups_size":0,"ondisk_log_size":5743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189399040,"num_objects":2890,"num_object_clones":0,"num_object_copies":8670,"num_objects_missing_on_primary":2036,"num_objects_missing":2036,"num_objects_degraded":3276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2890,"num_write_kb":184960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":["2","3","6"],"object_location_counts":[{"shards":"0,2,6","objects":796},{"shards":"0,5,6","objects":854},{"shards":"2,3,6","objects":1240}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"55.17","version":"1476'5772","reported_seq":5847,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:26.424763+0000","last_change":"2024-08-17T18:05:22.551819+0000","last_active":"2024-08-17T18:05:26.424763+0000","last_peered":"2024-08-17T18:05:26.424763+0000","last_clean":"2024-08-17T18:05:26.424763+0000","last_became_active":"2024-08-17T18:05:19.527486+0000","last_became_peered":"2024-08-17T18:05:19.527486+0000","last_unstale":"2024-08-17T18:05:26.424763+0000","last_undegraded":"2024-08-17T18:05:26.424763+0000","last_fullsized":"2024-08-17T18:05:26.424763+0000","mapping_epoch":1472,"log_start":"0'0","ondisk_log_start":"0'0","created":1472,"last_epoch_clean":1473,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5772,"log_dups_size":0,"ondisk_log_size":5772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T18:05:26.785724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189726720,"num_objects":2895,"num_object_clones":0,"num_object_copies":8685,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2915,"num_write_kb":185920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.16","version":"1476'5738","reported_seq":5859,"reported_epoch":1476,"state":"active+recovery_wait+degraded","last_fresh":"2024-08-17T18:05:26.767659+0000","last_change":"2024-08-17T18:05:19.590720+0000","last_active":"2024-08-17T18:05:26.767659+0000","last_peered":"2024-08-17T18:05:26.767659+0000","last_clean":"2024-08-17T18:04:42.635203+0000","last_became_active":"2024-08-17T18:05:19.562594+0000","last_became_peered":"2024-08-17T18:05:19.562594+0000","last_unstale":"2024-08-17T18:05:26.767659+0000","last_undegraded":"2024-08-17T18:05:19.526021+0000","last_fullsized":"2024-08-17T18:05:26.767659+0000","mapping_epoch":1472,"log_start":"0'0","ondisk_log_start":"0'0","created":1472,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5738,"log_dups_size":0,"ondisk_log_size":5738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187564032,"num_objects":2862,"num_object_clones":0,"num_object_copies":8586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2021,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":184256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":["2","1","3"],"object_location_counts":[{"shards":"1,2,3","objects":2021},{"shards":"2,3,5","objects":841}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"55.15","version":"1476'5790","reported_seq":10950,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:26.846512+0000","last_change":"2024-08-17T18:05:19.510829+0000","last_active":"2024-08-17T18:05:26.846512+0000","last_peered":"2024-08-17T18:05:26.846512+0000","last_clean":"2024-08-17T18:05:26.846512+0000","last_became_active":"2024-08-17T18:05:19.508016+0000","last_became_peered":"2024-08-17T18:05:19.508016+0000","last_unstale":"2024-08-17T18:05:26.846512+0000","last_undegraded":"2024-08-17T18:05:26.846512+0000","last_fullsized":"2024-08-17T18:05:26.846512+0000","mapping_epoch":1472,"log_start":"0'0","ondisk_log_start":"0'0","created":1472,"last_epoch_clean":1473,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5790,"log_dups_size":0,"ondisk_log_size":5790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189333504,"num_objects":2889,"num_object_clones":0,"num_object_copies":8667,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2919,"num_write_kb":185856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1271,"num_bytes_recovered":83263488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.14","version":"1476'5803","reported_seq":5878,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:26.665940+0000","last_change":"2024-08-17T18:05:21.765581+0000","last_active":"2024-08-17T18:05:26.665940+0000","last_peered":"2024-08-17T18:05:26.665940+0000","last_clean":"2024-08-17T18:05:26.665940+0000","last_became_active":"2024-08-17T18:05:19.539993+0000","last_became_peered":"2024-08-17T18:05:19.539993+0000","last_unstale":"2024-08-17T18:05:26.665940+0000","last_undegraded":"2024-08-17T18:05:26.665940+0000","last_fullsized":"2024-08-17T18:05:26.665940+0000","mapping_epoch":1472,"log_start":"0'0","ondisk_log_start":"0'0","created":1472,"last_epoch_clean":1473,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5803,"log_dups_size":0,"ondisk_log_size":5803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190480384,"num_objects":2907,"num_object_clones":0,"num_object_copies":8721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2927,"num_write_kb":186656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"55.13","version":"1476'5681","reported_seq":5787,"reported_epoch":1476,"state":"active+recovery_wait+degraded","last_fresh":"2024-08-17T18:05:26.703733+0000","last_change":"2024-08-17T18:05:19.644622+0000","last_active":"2024-08-17T18:05:26.703733+0000","last_peered":"2024-08-17T18:05:26.703733+0000","last_clean":"2024-08-17T18:04:14.321026+0000","last_became_active":"2024-08-17T18:05:19.620346+0000","last_became_peered":"2024-08-17T18:05:19.620346+0000","last_unstale":"2024-08-17T18:05:26.703733+0000","last_undegraded":"2024-08-17T18:05:19.538333+0000","last_fullsized":"2024-08-17T18:05:26.703733+0000","mapping_epoch":1472,"log_start":"0'0","ondisk_log_start":"0'0","created":1472,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5681,"log_dups_size":0,"ondisk_log_size":5681,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186482688,"num_objects":2846,"num_object_clones":0,"num_object_copies":8538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2001,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2864,"num_write_kb":182688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":589824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":["1","3","6"],"object_location_counts":[{"shards":"0,1,3","objects":795},{"shards":"1,3,5","objects":845},{"shards":"1,3,6","objects":1206}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.12","version":"1476'5781","reported_seq":5862,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:26.378380+0000","last_change":"2024-08-17T18:05:23.208413+0000","last_active":"2024-08-17T18:05:26.378380+0000","last_peered":"2024-08-17T18:05:26.378380+0000","last_clean":"2024-08-17T18:05:26.378380+0000","last_became_active":"2024-08-17T18:05:19.540941+0000","last_became_peered":"2024-08-17T18:05:19.540941+0000","last_unstale":"2024-08-17T18:05:26.378380+0000","last_undegraded":"2024-08-17T18:05:26.378380+0000","last_fullsized":"2024-08-17T18:05:26.378380+0000","mapping_epoch":1472,"log_start":"0'0","ondisk_log_start":"0'0","created":1472,"last_epoch_clean":1474,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5781,"log_dups_size":0,"ondisk_log_size":5781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T18:05:27.771894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190382080,"num_objects":2905,"num_object_clones":0,"num_object_copies":8715,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2913,"num_write_kb":186176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.11","version":"1476'5831","reported_seq":7422,"reported_epoch":1476,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:05:26.364182+0000","last_change":"2024-08-17T18:05:20.590739+0000","last_active":"2024-08-17T18:05:26.364182+0000","last_peered":"2024-08-17T18:05:26.364182+0000","last_clean":"2024-08-17T18:04:14.334099+0000","last_became_active":"2024-08-17T18:05:20.571496+0000","last_became_peered":"2024-08-17T18:05:20.571496+0000","last_unstale":"2024-08-17T18:05:26.364182+0000","last_undegraded":"2024-08-17T18:05:20.551636+0000","last_fullsized":"2024-08-17T18:05:20.545771+0000","mapping_epoch":1474,"log_start":"0'0","ondisk_log_start":"0'0","created":1472,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5831,"log_dups_size":0,"ondisk_log_size":5831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192151552,"num_objects":2932,"num_object_clones":0,"num_object_copies":8796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1673,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2944,"num_write_kb":188032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":376,"num_bytes_recovered":24641536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4],"avail_no_missing":["3","2","4"],"object_location_counts":[{"shards":"0,3,4","objects":831},{"shards":"2,3,4","objects":842},{"shards":"3,4","objects":1259}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.10","version":"1476'5637","reported_seq":5719,"reported_epoch":1476,"state":"active+recovery_wait","last_fresh":"2024-08-17T18:05:26.483652+0000","last_change":"2024-08-17T18:05:19.785766+0000","last_active":"2024-08-17T18:05:26.483652+0000","last_peered":"2024-08-17T18:05:26.483652+0000","last_clean":"2024-08-17T18:05:18.429605+0000","last_became_active":"2024-08-17T18:05:19.555661+0000","last_became_peered":"2024-08-17T18:05:19.555661+0000","last_unstale":"2024-08-17T18:05:26.483652+0000","last_undegraded":"2024-08-17T18:05:26.483652+0000","last_fullsized":"2024-08-17T18:05:26.483652+0000","mapping_epoch":1472,"log_start":"0'0","ondisk_log_start":"0'0","created":1472,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5637,"log_dups_size":0,"ondisk_log_size":5637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185335808,"num_objects":2828,"num_object_clones":0,"num_object_copies":8484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2842,"num_write_kb":181440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":["3","2","6"],"object_location_counts":[{"shards":"2,3,6","objects":2828}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.f","version":"1476'5752","reported_seq":10416,"reported_epoch":1476,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:05:26.936200+0000","last_change":"2024-08-17T18:04:45.033662+0000","last_active":"2024-08-17T18:05:26.936200+0000","last_peered":"2024-08-17T18:05:26.936200+0000","last_clean":"2024-08-17T18:04:14.223726+0000","last_became_active":"2024-08-17T18:04:45.023531+0000","last_became_peered":"2024-08-17T18:04:45.023531+0000","last_unstale":"2024-08-17T18:05:26.936200+0000","last_undegraded":"2024-08-17T18:04:44.883222+0000","last_fullsized":"2024-08-17T18:04:44.762504+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5752,"log_dups_size":0,"ondisk_log_size":5752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:34:46.944210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374079488,"num_objects":5708,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5752,"num_write_kb":366720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2300,"num_bytes_recovered":150732800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[6,0],"avail_no_missing":["6","1","2"],"object_location_counts":[{"shards":"0,1,2,6","objects":2279},{"shards":"0,2,6","objects":1624},{"shards":"0,6","objects":1645},{"shards":"1,2,6","objects":138}],"blocked_by":[],"up_primary":5,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.e","version":"1476'5663","reported_seq":5736,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:26.518567+0000","last_change":"2024-08-17T18:03:40.237084+0000","last_active":"2024-08-17T18:05:26.518567+0000","last_peered":"2024-08-17T18:05:26.518567+0000","last_clean":"2024-08-17T18:05:26.518567+0000","last_became_active":"2024-08-17T18:03:40.236802+0000","last_became_peered":"2024-08-17T18:03:40.236802+0000","last_unstale":"2024-08-17T18:05:26.518567+0000","last_undegraded":"2024-08-17T18:05:26.518567+0000","last_fullsized":"2024-08-17T18:05:26.518567+0000","mapping_epoch":1446,"log_start":"1453'113","ondisk_log_start":"1453'113","created":1446,"last_epoch_clean":1447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5550,"log_dups_size":0,"ondisk_log_size":5550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:58:01.523805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368771072,"num_objects":5627,"num_object_clones":0,"num_object_copies":16881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5663,"num_write_kb":361280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.d","version":"1476'5784","reported_seq":7319,"reported_epoch":1476,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-17T18:05:27.744724+0000","last_change":"2024-08-17T18:05:18.574111+0000","last_active":"2024-08-17T18:05:27.744724+0000","last_peered":"2024-08-17T18:05:27.744724+0000","last_clean":"2024-08-17T18:04:14.320037+0000","last_became_active":"2024-08-17T18:04:45.022484+0000","last_became_peered":"2024-08-17T18:04:45.022484+0000","last_unstale":"2024-08-17T18:05:27.744724+0000","last_undegraded":"2024-08-17T18:05:27.744724+0000","last_fullsized":"2024-08-17T18:04:44.763637+0000","mapping_epoch":1465,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5784,"log_dups_size":0,"ondisk_log_size":5784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:12:45.255851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372834328,"num_objects":5690,"num_object_clones":0,"num_object_copies":17070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5675,"num_objects_unfound":0,"num_objects_dirty":5690,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5784,"num_write_kb":367105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":493,"num_bytes_recovered":32309248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0],"avail_no_missing":["4","3","6"],"object_location_counts":[{"shards":"0,3,4","objects":1604},{"shards":"0,3,4,6","objects":1},{"shards":"0,4","objects":2050},{"shards":"3,4,6","objects":2035}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"55.c","version":"1476'5713","reported_seq":5792,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:26.471867+0000","last_change":"2024-08-17T18:03:51.280566+0000","last_active":"2024-08-17T18:05:26.471867+0000","last_peered":"2024-08-17T18:05:26.471867+0000","last_clean":"2024-08-17T18:05:26.471867+0000","last_became_active":"2024-08-17T18:03:51.278719+0000","last_became_peered":"2024-08-17T18:03:51.278719+0000","last_unstale":"2024-08-17T18:05:26.471867+0000","last_undegraded":"2024-08-17T18:05:26.471867+0000","last_fullsized":"2024-08-17T18:05:26.471867+0000","mapping_epoch":1449,"log_start":"1453'167","ondisk_log_start":"1453'167","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5546,"log_dups_size":0,"ondisk_log_size":5546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:15:14.360127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372572160,"num_objects":5685,"num_object_clones":0,"num_object_copies":17055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5713,"num_write_kb":364736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.b","version":"1476'5726","reported_seq":5774,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:27.059787+0000","last_change":"2024-08-17T18:03:51.272459+0000","last_active":"2024-08-17T18:05:27.059787+0000","last_peered":"2024-08-17T18:05:27.059787+0000","last_clean":"2024-08-17T18:05:27.059787+0000","last_became_active":"2024-08-17T18:03:51.272155+0000","last_became_peered":"2024-08-17T18:03:51.272155+0000","last_unstale":"2024-08-17T18:05:27.059787+0000","last_undegraded":"2024-08-17T18:05:27.059787+0000","last_fullsized":"2024-08-17T18:05:27.059787+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5726,"log_dups_size":0,"ondisk_log_size":5726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:17:38.013046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371589120,"num_objects":5670,"num_object_clones":0,"num_object_copies":17010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5726,"num_write_kb":364672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.a","version":"1476'5696","reported_seq":5775,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:26.511340+0000","last_change":"2024-08-17T18:03:51.276612+0000","last_active":"2024-08-17T18:05:26.511340+0000","last_peered":"2024-08-17T18:05:26.511340+0000","last_clean":"2024-08-17T18:05:26.511340+0000","last_became_active":"2024-08-17T18:03:51.276295+0000","last_became_peered":"2024-08-17T18:03:51.276295+0000","last_unstale":"2024-08-17T18:05:26.511340+0000","last_undegraded":"2024-08-17T18:05:26.511340+0000","last_fullsized":"2024-08-17T18:05:26.511340+0000","mapping_epoch":1449,"log_start":"1453'215","ondisk_log_start":"1453'215","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5481,"log_dups_size":0,"ondisk_log_size":5481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:47:27.609199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371195904,"num_objects":5664,"num_object_clones":0,"num_object_copies":16992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5696,"num_write_kb":363520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.9","version":"1476'5658","reported_seq":5756,"reported_epoch":1476,"state":"active+recovery_wait+degraded","last_fresh":"2024-08-17T18:05:26.476579+0000","last_change":"2024-08-17T18:05:19.766797+0000","last_active":"2024-08-17T18:05:26.476579+0000","last_peered":"2024-08-17T18:05:26.476579+0000","last_clean":"2024-08-17T18:04:42.666472+0000","last_became_active":"2024-08-17T18:05:19.666114+0000","last_became_peered":"2024-08-17T18:05:19.666114+0000","last_unstale":"2024-08-17T18:05:26.476579+0000","last_undegraded":"2024-08-17T18:05:19.638313+0000","last_fullsized":"2024-08-17T18:05:26.476579+0000","mapping_epoch":1473,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5658,"log_dups_size":0,"ondisk_log_size":5658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186122240,"num_objects":2840,"num_object_clones":0,"num_object_copies":8520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2858,"num_write_kb":182336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":["3","2","4"],"object_location_counts":[{"shards":"2,3,4","objects":1998},{"shards":"2,3,5","objects":842}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.8","version":"1476'5736","reported_seq":5808,"reported_epoch":1476,"state":"active+recovery_wait+degraded","last_fresh":"2024-08-17T18:05:24.522552+0000","last_change":"2024-08-17T18:05:23.125409+0000","last_active":"2024-08-17T18:05:24.522552+0000","last_peered":"2024-08-17T18:05:24.522552+0000","last_clean":"2024-08-17T18:04:14.322941+0000","last_became_active":"2024-08-17T18:05:22.474572+0000","last_became_peered":"2024-08-17T18:05:22.474572+0000","last_unstale":"2024-08-17T18:05:24.522552+0000","last_undegraded":"2024-08-17T18:05:20.510795+0000","last_fullsized":"2024-08-17T18:05:24.522552+0000","mapping_epoch":1473,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5736,"log_dups_size":0,"ondisk_log_size":5736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188940288,"num_objects":2883,"num_object_clones":0,"num_object_copies":8649,"num_objects_missing_on_primary":2038,"num_objects_missing":2038,"num_objects_degraded":3268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2883,"num_write_kb":184512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":["2","3","6"],"object_location_counts":[{"shards":"0,2,6","objects":808},{"shards":"0,5,6","objects":845},{"shards":"2,3,6","objects":1230}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"55.7","version":"1476'5772","reported_seq":5848,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:26.532555+0000","last_change":"2024-08-17T18:05:26.213174+0000","last_active":"2024-08-17T18:05:26.532555+0000","last_peered":"2024-08-17T18:05:26.532555+0000","last_clean":"2024-08-17T18:05:26.532555+0000","last_became_active":"2024-08-17T18:05:19.505876+0000","last_became_peered":"2024-08-17T18:05:19.505876+0000","last_unstale":"2024-08-17T18:05:26.532555+0000","last_undegraded":"2024-08-17T18:05:26.532555+0000","last_fullsized":"2024-08-17T18:05:26.532555+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5772,"log_dups_size":0,"ondisk_log_size":5772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T18:05:30.800050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189726720,"num_objects":2895,"num_object_clones":0,"num_object_copies":8685,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2915,"num_write_kb":185920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.6","version":"1476'5738","reported_seq":5834,"reported_epoch":1476,"state":"active+recovery_wait+degraded","last_fresh":"2024-08-17T18:05:26.708536+0000","last_change":"2024-08-17T18:05:19.623260+0000","last_active":"2024-08-17T18:05:26.708536+0000","last_peered":"2024-08-17T18:05:26.708536+0000","last_clean":"2024-08-17T18:04:42.635203+0000","last_became_active":"2024-08-17T18:05:19.619233+0000","last_became_peered":"2024-08-17T18:05:19.619233+0000","last_unstale":"2024-08-17T18:05:26.708536+0000","last_undegraded":"2024-08-17T18:05:19.556587+0000","last_fullsized":"2024-08-17T18:05:26.708536+0000","mapping_epoch":1473,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5738,"log_dups_size":0,"ondisk_log_size":5738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189005824,"num_objects":2884,"num_object_clones":0,"num_object_copies":8652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":184960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":["2","1","3"],"object_location_counts":[{"shards":"1,2,3","objects":2054},{"shards":"2,3,5","objects":830}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"55.5","version":"1476'5791","reported_seq":10951,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:26.877057+0000","last_change":"2024-08-17T18:05:19.495627+0000","last_active":"2024-08-17T18:05:26.877057+0000","last_peered":"2024-08-17T18:05:26.877057+0000","last_clean":"2024-08-17T18:05:26.877057+0000","last_became_active":"2024-08-17T18:05:19.491945+0000","last_became_peered":"2024-08-17T18:05:19.491945+0000","last_unstale":"2024-08-17T18:05:26.877057+0000","last_undegraded":"2024-08-17T18:05:26.877057+0000","last_fullsized":"2024-08-17T18:05:26.877057+0000","mapping_epoch":1469,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5791,"log_dups_size":0,"ondisk_log_size":5791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190185472,"num_objects":2902,"num_object_clones":0,"num_object_copies":8706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2920,"num_write_kb":186304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1270,"num_bytes_recovered":83263488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.4","version":"1476'5801","reported_seq":5877,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:26.507860+0000","last_change":"2024-08-17T18:05:26.205309+0000","last_active":"2024-08-17T18:05:26.507860+0000","last_peered":"2024-08-17T18:05:26.507860+0000","last_clean":"2024-08-17T18:05:26.507860+0000","last_became_active":"2024-08-17T18:05:19.476060+0000","last_became_peered":"2024-08-17T18:05:19.476060+0000","last_unstale":"2024-08-17T18:05:26.507860+0000","last_undegraded":"2024-08-17T18:05:26.507860+0000","last_fullsized":"2024-08-17T18:05:26.507860+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5801,"log_dups_size":0,"ondisk_log_size":5801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T18:05:30.814253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190611456,"num_objects":2908,"num_object_clones":0,"num_object_copies":8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2924,"num_write_kb":186656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"55.3","version":"1476'5685","reported_seq":5796,"reported_epoch":1476,"state":"active+recovery_wait+degraded","last_fresh":"2024-08-17T18:05:26.722268+0000","last_change":"2024-08-17T18:05:19.677581+0000","last_active":"2024-08-17T18:05:26.722268+0000","last_peered":"2024-08-17T18:05:26.722268+0000","last_clean":"2024-08-17T18:04:14.321026+0000","last_became_active":"2024-08-17T18:05:19.672336+0000","last_became_peered":"2024-08-17T18:05:19.672336+0000","last_unstale":"2024-08-17T18:05:26.722268+0000","last_undegraded":"2024-08-17T18:05:19.565509+0000","last_fullsized":"2024-08-17T18:05:26.722268+0000","mapping_epoch":1473,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5685,"log_dups_size":0,"ondisk_log_size":5685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186351616,"num_objects":2843,"num_object_clones":0,"num_object_copies":8529,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1994,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2867,"num_write_kb":182752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":["1","3","6"],"object_location_counts":[{"shards":"0,1,3","objects":779},{"shards":"1,3,5","objects":849},{"shards":"1,3,6","objects":1215}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.2","version":"1476'5778","reported_seq":5859,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:26.337906+0000","last_change":"2024-08-17T18:05:22.550675+0000","last_active":"2024-08-17T18:05:26.337906+0000","last_peered":"2024-08-17T18:05:26.337906+0000","last_clean":"2024-08-17T18:05:26.337906+0000","last_became_active":"2024-08-17T18:05:19.523549+0000","last_became_peered":"2024-08-17T18:05:19.523549+0000","last_unstale":"2024-08-17T18:05:26.337906+0000","last_undegraded":"2024-08-17T18:05:26.337906+0000","last_fullsized":"2024-08-17T18:05:26.337906+0000","mapping_epoch":1446,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5778,"log_dups_size":0,"ondisk_log_size":5778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189399040,"num_objects":2890,"num_object_clones":0,"num_object_copies":8670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2910,"num_write_kb":185600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.1","version":"1476'5821","reported_seq":7414,"reported_epoch":1476,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:05:26.466927+0000","last_change":"2024-08-17T18:05:19.488162+0000","last_active":"2024-08-17T18:05:26.466927+0000","last_peered":"2024-08-17T18:05:26.466927+0000","last_clean":"2024-08-17T18:04:14.334099+0000","last_became_active":"2024-08-17T18:05:19.481133+0000","last_became_peered":"2024-08-17T18:05:19.481133+0000","last_unstale":"2024-08-17T18:05:26.466927+0000","last_undegraded":"2024-08-17T18:05:19.473015+0000","last_fullsized":"2024-08-17T18:05:19.469190+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5821,"log_dups_size":0,"ondisk_log_size":5821,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191365120,"num_objects":2920,"num_object_clones":0,"num_object_copies":8760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2934,"num_write_kb":187328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":376,"num_bytes_recovered":24641536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4],"avail_no_missing":["3","2","4"],"object_location_counts":[{"shards":"0,3,4","objects":855},{"shards":"2,3,4","objects":914},{"shards":"3,4","objects":1151}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.0","version":"1476'5647","reported_seq":5729,"reported_epoch":1476,"state":"active+recovery_wait","last_fresh":"2024-08-17T18:05:26.443126+0000","last_change":"2024-08-17T18:05:19.548112+0000","last_active":"2024-08-17T18:05:26.443126+0000","last_peered":"2024-08-17T18:05:26.443126+0000","last_clean":"2024-08-17T18:05:18.429605+0000","last_became_active":"2024-08-17T18:05:19.528008+0000","last_became_peered":"2024-08-17T18:05:19.528008+0000","last_unstale":"2024-08-17T18:05:26.443126+0000","last_undegraded":"2024-08-17T18:05:26.443126+0000","last_fullsized":"2024-08-17T18:05:26.443126+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5647,"log_dups_size":0,"ondisk_log_size":5647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185729024,"num_objects":2834,"num_object_clones":0,"num_object_copies":8502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2852,"num_write_kb":181952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":["3","2","6"],"object_location_counts":[{"shards":"2,3,6","objects":2834}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":67,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.790521+0000","last_change":"2024-08-17T18:04:43.798908+0000","last_active":"2024-08-17T18:05:22.790521+0000","last_peered":"2024-08-17T18:05:22.790521+0000","last_clean":"2024-08-17T18:05:22.790521+0000","last_became_active":"2024-08-17T18:04:43.779951+0000","last_became_peered":"2024-08-17T18:04:43.779951+0000","last_unstale":"2024-08-17T18:05:22.790521+0000","last_undegraded":"2024-08-17T18:05:22.790521+0000","last_fullsized":"2024-08-17T18:05:22.790521+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:40.663566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1450,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.544214+0000","last_change":"2024-08-17T17:23:07.398676+0000","last_active":"2024-08-17T18:05:20.544214+0000","last_peered":"2024-08-17T18:05:20.544214+0000","last_clean":"2024-08-17T18:05:20.544214+0000","last_became_active":"2024-08-17T17:23:07.398369+0000","last_became_peered":"2024-08-17T17:23:07.398369+0000","last_unstale":"2024-08-17T18:05:20.544214+0000","last_undegraded":"2024-08-17T18:05:20.544214+0000","last_fullsized":"2024-08-17T18:05:20.544214+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:55:04.281412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":2255,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.904370+0000","last_change":"2024-08-17T18:01:23.501829+0000","last_active":"2024-08-17T18:05:22.904370+0000","last_peered":"2024-08-17T18:05:22.904370+0000","last_clean":"2024-08-17T18:05:22.904370+0000","last_became_active":"2024-08-17T18:01:23.501392+0000","last_became_peered":"2024-08-17T18:01:23.501392+0000","last_unstale":"2024-08-17T18:05:22.904370+0000","last_undegraded":"2024-08-17T18:05:22.904370+0000","last_fullsized":"2024-08-17T18:05:22.904370+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:10:37.662647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2291,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.909355+0000","last_change":"2024-08-17T18:04:14.540774+0000","last_active":"2024-08-17T18:05:22.909355+0000","last_peered":"2024-08-17T18:05:22.909355+0000","last_clean":"2024-08-17T18:05:22.909355+0000","last_became_active":"2024-08-17T18:04:14.539778+0000","last_became_peered":"2024-08-17T18:04:14.539778+0000","last_unstale":"2024-08-17T18:05:22.909355+0000","last_undegraded":"2024-08-17T18:05:22.909355+0000","last_fullsized":"2024-08-17T18:05:22.909355+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:55:02.863360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":26,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.547457+0000","last_change":"2024-08-17T18:04:43.777705+0000","last_active":"2024-08-17T18:05:20.547457+0000","last_peered":"2024-08-17T18:05:20.547457+0000","last_clean":"2024-08-17T18:05:20.547457+0000","last_became_active":"2024-08-17T18:04:43.771283+0000","last_became_peered":"2024-08-17T18:04:43.771283+0000","last_unstale":"2024-08-17T18:05:20.547457+0000","last_undegraded":"2024-08-17T18:05:20.547457+0000","last_fullsized":"2024-08-17T18:05:20.547457+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:13.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1536,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.903774+0000","last_change":"2024-08-17T18:04:14.534963+0000","last_active":"2024-08-17T18:05:22.903774+0000","last_peered":"2024-08-17T18:05:22.903774+0000","last_clean":"2024-08-17T18:05:22.903774+0000","last_became_active":"2024-08-17T18:04:14.534634+0000","last_became_peered":"2024-08-17T18:04:14.534634+0000","last_unstale":"2024-08-17T18:05:22.903774+0000","last_undegraded":"2024-08-17T18:05:22.903774+0000","last_fullsized":"2024-08-17T18:05:22.903774+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:33:27.287860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":26,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.547350+0000","last_change":"2024-08-17T18:04:43.773534+0000","last_active":"2024-08-17T18:05:20.547350+0000","last_peered":"2024-08-17T18:05:20.547350+0000","last_clean":"2024-08-17T18:05:20.547350+0000","last_became_active":"2024-08-17T18:04:43.768507+0000","last_became_peered":"2024-08-17T18:04:43.768507+0000","last_unstale":"2024-08-17T18:05:20.547350+0000","last_undegraded":"2024-08-17T18:05:20.547350+0000","last_fullsized":"2024-08-17T18:05:20.547350+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:50:57.647351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1465,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.544392+0000","last_change":"2024-08-17T17:24:17.458168+0000","last_active":"2024-08-17T18:05:20.544392+0000","last_peered":"2024-08-17T18:05:20.544392+0000","last_clean":"2024-08-17T18:05:20.544392+0000","last_became_active":"2024-08-17T17:24:17.457847+0000","last_became_peered":"2024-08-17T17:24:17.457847+0000","last_unstale":"2024-08-17T18:05:20.544392+0000","last_undegraded":"2024-08-17T18:05:20.544392+0000","last_fullsized":"2024-08-17T18:05:20.544392+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:28.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":48,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.790087+0000","last_change":"2024-08-17T18:04:14.531445+0000","last_active":"2024-08-17T18:05:22.790087+0000","last_peered":"2024-08-17T18:05:22.790087+0000","last_clean":"2024-08-17T18:05:22.790087+0000","last_became_active":"2024-08-17T18:04:14.531099+0000","last_became_peered":"2024-08-17T18:04:14.531099+0000","last_unstale":"2024-08-17T18:05:22.790087+0000","last_undegraded":"2024-08-17T18:05:22.790087+0000","last_fullsized":"2024-08-17T18:05:22.790087+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:43.335123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":2244,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.771558+0000","last_change":"2024-08-17T18:01:23.493148+0000","last_active":"2024-08-17T18:05:22.771558+0000","last_peered":"2024-08-17T18:05:22.771558+0000","last_clean":"2024-08-17T18:05:22.771558+0000","last_became_active":"2024-08-17T18:01:23.491909+0000","last_became_peered":"2024-08-17T18:01:23.491909+0000","last_unstale":"2024-08-17T18:05:22.771558+0000","last_undegraded":"2024-08-17T18:05:22.771558+0000","last_fullsized":"2024-08-17T18:05:22.771558+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:01:40.187357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1482,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.544427+0000","last_change":"2024-08-17T18:01:23.519890+0000","last_active":"2024-08-17T18:05:20.544427+0000","last_peered":"2024-08-17T18:05:20.544427+0000","last_clean":"2024-08-17T18:05:20.544427+0000","last_became_active":"2024-08-17T18:01:23.519526+0000","last_became_peered":"2024-08-17T18:01:23.519526+0000","last_unstale":"2024-08-17T18:05:20.544427+0000","last_undegraded":"2024-08-17T18:05:20.544427+0000","last_fullsized":"2024-08-17T18:05:20.544427+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:04:52.662528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2257,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.904524+0000","last_change":"2024-08-17T18:01:23.491836+0000","last_active":"2024-08-17T18:05:22.904524+0000","last_peered":"2024-08-17T18:05:22.904524+0000","last_clean":"2024-08-17T18:05:22.904524+0000","last_became_active":"2024-08-17T18:01:23.491209+0000","last_became_peered":"2024-08-17T18:01:23.491209+0000","last_unstale":"2024-08-17T18:05:22.904524+0000","last_undegraded":"2024-08-17T18:05:22.904524+0000","last_fullsized":"2024-08-17T18:05:22.904524+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:35:30.767005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2259,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.550031+0000","last_change":"2024-08-17T17:24:17.452025+0000","last_active":"2024-08-17T18:05:22.550031+0000","last_peered":"2024-08-17T18:05:22.550031+0000","last_clean":"2024-08-17T18:05:22.550031+0000","last_became_active":"2024-08-17T17:24:17.448029+0000","last_became_peered":"2024-08-17T17:24:17.448029+0000","last_unstale":"2024-08-17T18:05:22.550031+0000","last_undegraded":"2024-08-17T18:05:22.550031+0000","last_fullsized":"2024-08-17T18:05:22.550031+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:45.089346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":48,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.790395+0000","last_change":"2024-08-17T18:04:14.541444+0000","last_active":"2024-08-17T18:05:22.790395+0000","last_peered":"2024-08-17T18:05:22.790395+0000","last_clean":"2024-08-17T18:05:22.790395+0000","last_became_active":"2024-08-17T18:04:14.541074+0000","last_became_peered":"2024-08-17T18:04:14.541074+0000","last_unstale":"2024-08-17T18:05:22.790395+0000","last_undegraded":"2024-08-17T18:05:22.790395+0000","last_fullsized":"2024-08-17T18:05:22.790395+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:04:04.979761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2272,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.904015+0000","last_change":"2024-08-17T18:04:14.539660+0000","last_active":"2024-08-17T18:05:22.904015+0000","last_peered":"2024-08-17T18:05:22.904015+0000","last_clean":"2024-08-17T18:05:22.904015+0000","last_became_active":"2024-08-17T18:04:14.537852+0000","last_became_peered":"2024-08-17T18:04:14.537852+0000","last_unstale":"2024-08-17T18:05:22.904015+0000","last_undegraded":"2024-08-17T18:05:22.904015+0000","last_fullsized":"2024-08-17T18:05:22.904015+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:47:45.026997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1478,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.546275+0000","last_change":"2024-08-17T18:04:43.769892+0000","last_active":"2024-08-17T18:05:20.546275+0000","last_peered":"2024-08-17T18:05:20.546275+0000","last_clean":"2024-08-17T18:05:20.546275+0000","last_became_active":"2024-08-17T18:04:43.769702+0000","last_became_peered":"2024-08-17T18:04:43.769702+0000","last_unstale":"2024-08-17T18:05:20.546275+0000","last_undegraded":"2024-08-17T18:05:20.546275+0000","last_fullsized":"2024-08-17T18:05:20.546275+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:39.373904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2231,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.904179+0000","last_change":"2024-08-17T17:24:17.442338+0000","last_active":"2024-08-17T18:05:22.904179+0000","last_peered":"2024-08-17T18:05:22.904179+0000","last_clean":"2024-08-17T18:05:22.904179+0000","last_became_active":"2024-08-17T17:24:17.441948+0000","last_became_peered":"2024-08-17T17:24:17.441948+0000","last_unstale":"2024-08-17T18:05:22.904179+0000","last_undegraded":"2024-08-17T18:05:22.904179+0000","last_fullsized":"2024-08-17T18:05:22.904179+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:17.891492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":55,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.790538+0000","last_change":"2024-08-17T18:04:43.777736+0000","last_active":"2024-08-17T18:05:22.790538+0000","last_peered":"2024-08-17T18:05:22.790538+0000","last_clean":"2024-08-17T18:05:22.790538+0000","last_became_active":"2024-08-17T18:04:43.777438+0000","last_became_peered":"2024-08-17T18:04:43.777438+0000","last_unstale":"2024-08-17T18:05:22.790538+0000","last_undegraded":"2024-08-17T18:05:22.790538+0000","last_fullsized":"2024-08-17T18:05:22.790538+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:24:01.255927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2211,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.909481+0000","last_change":"2024-08-17T18:04:43.915398+0000","last_active":"2024-08-17T18:05:22.909481+0000","last_peered":"2024-08-17T18:05:22.909481+0000","last_clean":"2024-08-17T18:05:22.909481+0000","last_became_active":"2024-08-17T18:04:43.915212+0000","last_became_peered":"2024-08-17T18:04:43.915212+0000","last_unstale":"2024-08-17T18:05:22.909481+0000","last_undegraded":"2024-08-17T18:05:22.909481+0000","last_fullsized":"2024-08-17T18:05:22.909481+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:43:26.901830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.547583+0000","last_change":"2024-08-17T18:04:43.773363+0000","last_active":"2024-08-17T18:05:20.547583+0000","last_peered":"2024-08-17T18:05:20.547583+0000","last_clean":"2024-08-17T18:05:20.547583+0000","last_became_active":"2024-08-17T18:04:43.773005+0000","last_became_peered":"2024-08-17T18:04:43.773005+0000","last_unstale":"2024-08-17T18:05:20.547583+0000","last_undegraded":"2024-08-17T18:05:20.547583+0000","last_fullsized":"2024-08-17T18:05:20.547583+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:03:13.813096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":368,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.904578+0000","last_change":"2024-08-17T18:04:14.555729+0000","last_active":"2024-08-17T18:05:22.904578+0000","last_peered":"2024-08-17T18:05:22.904578+0000","last_clean":"2024-08-17T18:05:22.904578+0000","last_became_active":"2024-08-17T18:04:14.536678+0000","last_became_peered":"2024-08-17T18:04:14.536678+0000","last_unstale":"2024-08-17T18:05:22.904578+0000","last_undegraded":"2024-08-17T18:05:22.904578+0000","last_fullsized":"2024-08-17T18:05:22.904578+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:47:29.424778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2270,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.550888+0000","last_change":"2024-08-17T18:01:23.494237+0000","last_active":"2024-08-17T18:05:22.550888+0000","last_peered":"2024-08-17T18:05:22.550888+0000","last_clean":"2024-08-17T18:05:22.550888+0000","last_became_active":"2024-08-17T18:01:23.493978+0000","last_became_peered":"2024-08-17T18:01:23.493978+0000","last_unstale":"2024-08-17T18:05:22.550888+0000","last_undegraded":"2024-08-17T18:05:22.550888+0000","last_fullsized":"2024-08-17T18:05:22.550888+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:05:41.724192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1479,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.544516+0000","last_change":"2024-08-17T18:03:51.292044+0000","last_active":"2024-08-17T18:05:20.544516+0000","last_peered":"2024-08-17T18:05:20.544516+0000","last_clean":"2024-08-17T18:05:20.544516+0000","last_became_active":"2024-08-17T18:03:51.268271+0000","last_became_peered":"2024-08-17T18:03:51.268271+0000","last_unstale":"2024-08-17T18:05:20.544516+0000","last_undegraded":"2024-08-17T18:05:20.544516+0000","last_fullsized":"2024-08-17T18:05:20.544516+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:00.892410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4756,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1451,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.544467+0000","last_change":"2024-08-17T17:23:07.455919+0000","last_active":"2024-08-17T18:05:20.544467+0000","last_peered":"2024-08-17T18:05:20.544467+0000","last_clean":"2024-08-17T18:05:20.544467+0000","last_became_active":"2024-08-17T17:23:07.455284+0000","last_became_peered":"2024-08-17T17:23:07.455284+0000","last_unstale":"2024-08-17T18:05:20.544467+0000","last_undegraded":"2024-08-17T18:05:20.544467+0000","last_fullsized":"2024-08-17T18:05:20.544467+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:37:46.260775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1475,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.545139+0000","last_change":"2024-08-17T18:04:43.766724+0000","last_active":"2024-08-17T18:05:20.545139+0000","last_peered":"2024-08-17T18:05:20.545139+0000","last_clean":"2024-08-17T18:05:20.545139+0000","last_became_active":"2024-08-17T18:04:43.766428+0000","last_became_peered":"2024-08-17T18:04:43.766428+0000","last_unstale":"2024-08-17T18:05:20.545139+0000","last_undegraded":"2024-08-17T18:05:20.545139+0000","last_fullsized":"2024-08-17T18:05:20.545139+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:09:49.102940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1488,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.546340+0000","last_change":"2024-08-17T18:03:51.279963+0000","last_active":"2024-08-17T18:05:20.546340+0000","last_peered":"2024-08-17T18:05:20.546340+0000","last_clean":"2024-08-17T18:05:20.546340+0000","last_became_active":"2024-08-17T18:03:51.279506+0000","last_became_peered":"2024-08-17T18:03:51.279506+0000","last_unstale":"2024-08-17T18:05:20.546340+0000","last_undegraded":"2024-08-17T18:05:20.546340+0000","last_fullsized":"2024-08-17T18:05:20.546340+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:20:07.304042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":52,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.790452+0000","last_change":"2024-08-17T18:04:14.542589+0000","last_active":"2024-08-17T18:05:22.790452+0000","last_peered":"2024-08-17T18:05:22.790452+0000","last_clean":"2024-08-17T18:05:22.790452+0000","last_became_active":"2024-08-17T18:04:14.531540+0000","last_became_peered":"2024-08-17T18:04:14.531540+0000","last_unstale":"2024-08-17T18:05:22.790452+0000","last_undegraded":"2024-08-17T18:05:22.790452+0000","last_fullsized":"2024-08-17T18:05:22.790452+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:15:33.014782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2262,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.904101+0000","last_change":"2024-08-17T18:04:14.540337+0000","last_active":"2024-08-17T18:05:22.904101+0000","last_peered":"2024-08-17T18:05:22.904101+0000","last_clean":"2024-08-17T18:05:22.904101+0000","last_became_active":"2024-08-17T18:04:14.538363+0000","last_became_peered":"2024-08-17T18:04:14.538363+0000","last_unstale":"2024-08-17T18:05:22.904101+0000","last_undegraded":"2024-08-17T18:05:22.904101+0000","last_fullsized":"2024-08-17T18:05:22.904101+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:51.621443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2241,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.551145+0000","last_change":"2024-08-17T17:23:07.428296+0000","last_active":"2024-08-17T18:05:22.551145+0000","last_peered":"2024-08-17T18:05:22.551145+0000","last_clean":"2024-08-17T18:05:22.551145+0000","last_became_active":"2024-08-17T17:23:07.428010+0000","last_became_peered":"2024-08-17T17:23:07.428010+0000","last_unstale":"2024-08-17T18:05:22.551145+0000","last_undegraded":"2024-08-17T18:05:22.551145+0000","last_fullsized":"2024-08-17T18:05:22.551145+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:45.459212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":2282,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.904470+0000","last_change":"2024-08-17T18:04:43.800633+0000","last_active":"2024-08-17T18:05:22.904470+0000","last_peered":"2024-08-17T18:05:22.904470+0000","last_clean":"2024-08-17T18:05:22.904470+0000","last_became_active":"2024-08-17T18:04:43.784612+0000","last_became_peered":"2024-08-17T18:04:43.784612+0000","last_unstale":"2024-08-17T18:05:22.904470+0000","last_undegraded":"2024-08-17T18:05:22.904470+0000","last_fullsized":"2024-08-17T18:05:22.904470+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:11.405039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":55,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.790207+0000","last_change":"2024-08-17T18:04:43.805055+0000","last_active":"2024-08-17T18:05:22.790207+0000","last_peered":"2024-08-17T18:05:22.790207+0000","last_clean":"2024-08-17T18:05:22.790207+0000","last_became_active":"2024-08-17T18:04:43.804708+0000","last_became_peered":"2024-08-17T18:04:43.804708+0000","last_unstale":"2024-08-17T18:05:22.790207+0000","last_undegraded":"2024-08-17T18:05:22.790207+0000","last_fullsized":"2024-08-17T18:05:22.790207+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:47:22.253073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":48,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.791353+0000","last_change":"2024-08-17T18:04:14.529605+0000","last_active":"2024-08-17T18:05:22.791353+0000","last_peered":"2024-08-17T18:05:22.791353+0000","last_clean":"2024-08-17T18:05:22.791353+0000","last_became_active":"2024-08-17T18:04:14.529024+0000","last_became_peered":"2024-08-17T18:04:14.529024+0000","last_unstale":"2024-08-17T18:05:22.791353+0000","last_undegraded":"2024-08-17T18:05:22.791353+0000","last_fullsized":"2024-08-17T18:05:22.791353+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:38:00.485543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2295,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.550510+0000","last_change":"2024-08-17T18:04:43.773667+0000","last_active":"2024-08-17T18:05:22.550510+0000","last_peered":"2024-08-17T18:05:22.550510+0000","last_clean":"2024-08-17T18:05:22.550510+0000","last_became_active":"2024-08-17T18:04:43.773247+0000","last_became_peered":"2024-08-17T18:04:43.773247+0000","last_unstale":"2024-08-17T18:05:22.550510+0000","last_undegraded":"2024-08-17T18:05:22.550510+0000","last_fullsized":"2024-08-17T18:05:22.550510+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:35:48.964631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":119,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.771448+0000","last_change":"2024-08-17T18:04:43.771589+0000","last_active":"2024-08-17T18:05:22.771448+0000","last_peered":"2024-08-17T18:05:22.771448+0000","last_clean":"2024-08-17T18:05:22.771448+0000","last_became_active":"2024-08-17T18:04:43.769601+0000","last_became_peered":"2024-08-17T18:04:43.769601+0000","last_unstale":"2024-08-17T18:05:22.771448+0000","last_undegraded":"2024-08-17T18:05:22.771448+0000","last_fullsized":"2024-08-17T18:05:22.771448+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:42:49.228921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2311,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.550336+0000","last_change":"2024-08-17T18:04:14.537136+0000","last_active":"2024-08-17T18:05:22.550336+0000","last_peered":"2024-08-17T18:05:22.550336+0000","last_clean":"2024-08-17T18:05:22.550336+0000","last_became_active":"2024-08-17T18:04:14.536815+0000","last_became_peered":"2024-08-17T18:04:14.536815+0000","last_unstale":"2024-08-17T18:05:22.550336+0000","last_undegraded":"2024-08-17T18:05:22.550336+0000","last_fullsized":"2024-08-17T18:05:22.550336+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:02.706133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.547514+0000","last_change":"2024-08-17T18:04:43.773480+0000","last_active":"2024-08-17T18:05:20.547514+0000","last_peered":"2024-08-17T18:05:20.547514+0000","last_clean":"2024-08-17T18:05:20.547514+0000","last_became_active":"2024-08-17T18:04:43.773160+0000","last_became_peered":"2024-08-17T18:04:43.773160+0000","last_unstale":"2024-08-17T18:05:20.547514+0000","last_undegraded":"2024-08-17T18:05:20.547514+0000","last_fullsized":"2024-08-17T18:05:20.547514+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:07:55.468132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":121,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.550244+0000","last_change":"2024-08-17T18:02:08.224928+0000","last_active":"2024-08-17T18:05:22.550244+0000","last_peered":"2024-08-17T18:05:22.550244+0000","last_clean":"2024-08-17T18:05:22.550244+0000","last_became_active":"2024-08-17T18:02:08.224604+0000","last_became_peered":"2024-08-17T18:02:08.224604+0000","last_unstale":"2024-08-17T18:05:22.550244+0000","last_undegraded":"2024-08-17T18:05:22.550244+0000","last_fullsized":"2024-08-17T18:05:22.550244+0000","mapping_epoch":1421,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1422,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:55.716971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1473,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.544550+0000","last_change":"2024-08-17T18:04:14.526890+0000","last_active":"2024-08-17T18:05:20.544550+0000","last_peered":"2024-08-17T18:05:20.544550+0000","last_clean":"2024-08-17T18:05:20.544550+0000","last_became_active":"2024-08-17T18:04:14.526564+0000","last_became_peered":"2024-08-17T18:04:14.526564+0000","last_unstale":"2024-08-17T18:05:20.544550+0000","last_undegraded":"2024-08-17T18:05:20.544550+0000","last_fullsized":"2024-08-17T18:05:20.544550+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:10:13.229827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":26,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.547172+0000","last_change":"2024-08-17T18:04:43.791749+0000","last_active":"2024-08-17T18:05:20.547172+0000","last_peered":"2024-08-17T18:05:20.547172+0000","last_clean":"2024-08-17T18:05:20.547172+0000","last_became_active":"2024-08-17T18:04:43.774474+0000","last_became_peered":"2024-08-17T18:04:43.774474+0000","last_unstale":"2024-08-17T18:05:20.547172+0000","last_undegraded":"2024-08-17T18:05:20.547172+0000","last_fullsized":"2024-08-17T18:05:20.547172+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:19:08.097560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2297,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.551899+0000","last_change":"2024-08-17T18:04:43.781133+0000","last_active":"2024-08-17T18:05:22.551899+0000","last_peered":"2024-08-17T18:05:22.551899+0000","last_clean":"2024-08-17T18:05:22.551899+0000","last_became_active":"2024-08-17T18:04:43.780787+0000","last_became_peered":"2024-08-17T18:04:43.780787+0000","last_unstale":"2024-08-17T18:05:22.551899+0000","last_undegraded":"2024-08-17T18:05:22.551899+0000","last_fullsized":"2024-08-17T18:05:22.551899+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:38:34.901162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2254,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.550038+0000","last_change":"2024-08-17T18:01:23.493896+0000","last_active":"2024-08-17T18:05:22.550038+0000","last_peered":"2024-08-17T18:05:22.550038+0000","last_clean":"2024-08-17T18:05:22.550038+0000","last_became_active":"2024-08-17T18:01:23.493551+0000","last_became_peered":"2024-08-17T18:01:23.493551+0000","last_unstale":"2024-08-17T18:05:22.550038+0000","last_undegraded":"2024-08-17T18:05:22.550038+0000","last_fullsized":"2024-08-17T18:05:22.550038+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:18:47.648483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2231,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.771094+0000","last_change":"2024-08-17T18:04:14.543532+0000","last_active":"2024-08-17T18:05:22.771094+0000","last_peered":"2024-08-17T18:05:22.771094+0000","last_clean":"2024-08-17T18:05:22.771094+0000","last_became_active":"2024-08-17T18:04:14.543131+0000","last_became_peered":"2024-08-17T18:04:14.543131+0000","last_unstale":"2024-08-17T18:05:22.771094+0000","last_undegraded":"2024-08-17T18:05:22.771094+0000","last_fullsized":"2024-08-17T18:05:22.771094+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:51:44.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":48,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.790360+0000","last_change":"2024-08-17T18:04:14.536309+0000","last_active":"2024-08-17T18:05:22.790360+0000","last_peered":"2024-08-17T18:05:22.790360+0000","last_clean":"2024-08-17T18:05:22.790360+0000","last_became_active":"2024-08-17T18:04:14.536000+0000","last_became_peered":"2024-08-17T18:04:14.536000+0000","last_unstale":"2024-08-17T18:05:22.790360+0000","last_undegraded":"2024-08-17T18:05:22.790360+0000","last_fullsized":"2024-08-17T18:05:22.790360+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:37:18.733611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":48,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.790031+0000","last_change":"2024-08-17T18:04:14.543835+0000","last_active":"2024-08-17T18:05:22.790031+0000","last_peered":"2024-08-17T18:05:22.790031+0000","last_clean":"2024-08-17T18:05:22.790031+0000","last_became_active":"2024-08-17T18:04:14.542511+0000","last_became_peered":"2024-08-17T18:04:14.542511+0000","last_unstale":"2024-08-17T18:05:22.790031+0000","last_undegraded":"2024-08-17T18:05:22.790031+0000","last_fullsized":"2024-08-17T18:05:22.790031+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:47:15.826245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":997,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.550257+0000","last_change":"2024-08-17T18:04:14.537308+0000","last_active":"2024-08-17T18:05:22.550257+0000","last_peered":"2024-08-17T18:05:22.550257+0000","last_clean":"2024-08-17T18:05:22.550257+0000","last_became_active":"2024-08-17T18:04:14.535521+0000","last_became_peered":"2024-08-17T18:04:14.535521+0000","last_unstale":"2024-08-17T18:05:22.550257+0000","last_undegraded":"2024-08-17T18:05:22.550257+0000","last_fullsized":"2024-08-17T18:05:22.550257+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:40:00.384936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1454,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.546245+0000","last_change":"2024-08-17T18:04:14.527639+0000","last_active":"2024-08-17T18:05:20.546245+0000","last_peered":"2024-08-17T18:05:20.546245+0000","last_clean":"2024-08-17T18:05:20.546245+0000","last_became_active":"2024-08-17T18:04:14.527117+0000","last_became_peered":"2024-08-17T18:04:14.527117+0000","last_unstale":"2024-08-17T18:05:20.546245+0000","last_undegraded":"2024-08-17T18:05:20.546245+0000","last_fullsized":"2024-08-17T18:05:20.546245+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:01:04.047871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":59,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.771097+0000","last_change":"2024-08-17T18:04:43.771309+0000","last_active":"2024-08-17T18:05:22.771097+0000","last_peered":"2024-08-17T18:05:22.771097+0000","last_clean":"2024-08-17T18:05:22.771097+0000","last_became_active":"2024-08-17T18:04:43.770922+0000","last_became_peered":"2024-08-17T18:04:43.770922+0000","last_unstale":"2024-08-17T18:05:22.771097+0000","last_undegraded":"2024-08-17T18:05:22.771097+0000","last_fullsized":"2024-08-17T18:05:22.771097+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:10.173251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":26,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.547234+0000","last_change":"2024-08-17T18:04:43.786228+0000","last_active":"2024-08-17T18:05:20.547234+0000","last_peered":"2024-08-17T18:05:20.547234+0000","last_clean":"2024-08-17T18:05:20.547234+0000","last_became_active":"2024-08-17T18:04:43.770976+0000","last_became_peered":"2024-08-17T18:04:43.770976+0000","last_unstale":"2024-08-17T18:05:20.547234+0000","last_undegraded":"2024-08-17T18:05:20.547234+0000","last_fullsized":"2024-08-17T18:05:20.547234+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:20:10.131137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":664,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.789870+0000","last_change":"2024-08-17T18:04:17.566582+0000","last_active":"2024-08-17T18:05:22.789870+0000","last_peered":"2024-08-17T18:05:22.789870+0000","last_clean":"2024-08-17T18:05:22.789870+0000","last_became_active":"2024-08-17T18:04:17.562526+0000","last_became_peered":"2024-08-17T18:04:17.562526+0000","last_unstale":"2024-08-17T18:05:22.789870+0000","last_undegraded":"2024-08-17T18:05:22.789870+0000","last_fullsized":"2024-08-17T18:05:22.789870+0000","mapping_epoch":1460,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T17:06:21.174471+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:27:13.722171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2240,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.551739+0000","last_change":"2024-08-17T17:24:17.455710+0000","last_active":"2024-08-17T18:05:22.551739+0000","last_peered":"2024-08-17T18:05:22.551739+0000","last_clean":"2024-08-17T18:05:22.551739+0000","last_became_active":"2024-08-17T17:24:17.452989+0000","last_became_peered":"2024-08-17T17:24:17.452989+0000","last_unstale":"2024-08-17T18:05:22.551739+0000","last_undegraded":"2024-08-17T18:05:22.551739+0000","last_fullsized":"2024-08-17T18:05:22.551739+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:58:07.894968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2247,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.903889+0000","last_change":"2024-08-17T18:04:11.319527+0000","last_active":"2024-08-17T18:05:22.903889+0000","last_peered":"2024-08-17T18:05:22.903889+0000","last_clean":"2024-08-17T18:05:22.903889+0000","last_became_active":"2024-08-17T18:04:11.319104+0000","last_became_peered":"2024-08-17T18:04:11.319104+0000","last_unstale":"2024-08-17T18:05:22.903889+0000","last_undegraded":"2024-08-17T18:05:22.903889+0000","last_fullsized":"2024-08-17T18:05:22.903889+0000","mapping_epoch":1455,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:09:13.581842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1452,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.545751+0000","last_change":"2024-08-17T17:23:07.451866+0000","last_active":"2024-08-17T18:05:20.545751+0000","last_peered":"2024-08-17T18:05:20.545751+0000","last_clean":"2024-08-17T18:05:20.545751+0000","last_became_active":"2024-08-17T17:23:07.451553+0000","last_became_peered":"2024-08-17T17:23:07.451553+0000","last_unstale":"2024-08-17T18:05:20.545751+0000","last_undegraded":"2024-08-17T18:05:20.545751+0000","last_fullsized":"2024-08-17T18:05:20.545751+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:37:23.930651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1478,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.545177+0000","last_change":"2024-08-17T18:03:51.270293+0000","last_active":"2024-08-17T18:05:20.545177+0000","last_peered":"2024-08-17T18:05:20.545177+0000","last_clean":"2024-08-17T18:05:20.545177+0000","last_became_active":"2024-08-17T18:03:51.269970+0000","last_became_peered":"2024-08-17T18:03:51.269970+0000","last_unstale":"2024-08-17T18:05:20.545177+0000","last_undegraded":"2024-08-17T18:05:20.545177+0000","last_fullsized":"2024-08-17T18:05:20.545177+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:02:30.016517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1482,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.554960+0000","last_change":"2024-08-17T18:01:23.497110+0000","last_active":"2024-08-17T18:05:20.554960+0000","last_peered":"2024-08-17T18:05:20.554960+0000","last_clean":"2024-08-17T18:05:20.554960+0000","last_became_active":"2024-08-17T18:01:23.496800+0000","last_became_peered":"2024-08-17T18:01:23.496800+0000","last_unstale":"2024-08-17T18:05:20.554960+0000","last_undegraded":"2024-08-17T18:05:20.554960+0000","last_fullsized":"2024-08-17T18:05:20.554960+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:32:29.892739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1451,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.545501+0000","last_change":"2024-08-17T17:23:07.401624+0000","last_active":"2024-08-17T18:05:20.545501+0000","last_peered":"2024-08-17T18:05:20.545501+0000","last_clean":"2024-08-17T18:05:20.545501+0000","last_became_active":"2024-08-17T17:23:07.400222+0000","last_became_peered":"2024-08-17T17:23:07.400222+0000","last_unstale":"2024-08-17T18:05:20.545501+0000","last_undegraded":"2024-08-17T18:05:20.545501+0000","last_fullsized":"2024-08-17T18:05:20.545501+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:07.069989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1452,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.546160+0000","last_change":"2024-08-17T17:24:17.470809+0000","last_active":"2024-08-17T18:05:20.546160+0000","last_peered":"2024-08-17T18:05:20.546160+0000","last_clean":"2024-08-17T18:05:20.546160+0000","last_became_active":"2024-08-17T17:24:17.469753+0000","last_became_peered":"2024-08-17T17:24:17.469753+0000","last_unstale":"2024-08-17T18:05:20.546160+0000","last_undegraded":"2024-08-17T18:05:20.546160+0000","last_fullsized":"2024-08-17T18:05:20.546160+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:00.049425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":2249,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.904083+0000","last_change":"2024-08-17T18:04:14.552806+0000","last_active":"2024-08-17T18:05:22.904083+0000","last_peered":"2024-08-17T18:05:22.904083+0000","last_clean":"2024-08-17T18:05:22.904083+0000","last_became_active":"2024-08-17T18:04:14.535546+0000","last_became_peered":"2024-08-17T18:04:14.535546+0000","last_unstale":"2024-08-17T18:05:22.904083+0000","last_undegraded":"2024-08-17T18:05:22.904083+0000","last_fullsized":"2024-08-17T18:05:22.904083+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:44:45.702272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":48,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.790129+0000","last_change":"2024-08-17T18:04:14.539500+0000","last_active":"2024-08-17T18:05:22.790129+0000","last_peered":"2024-08-17T18:05:22.790129+0000","last_clean":"2024-08-17T18:05:22.790129+0000","last_became_active":"2024-08-17T18:04:14.538738+0000","last_became_peered":"2024-08-17T18:04:14.538738+0000","last_unstale":"2024-08-17T18:05:22.790129+0000","last_undegraded":"2024-08-17T18:05:22.790129+0000","last_fullsized":"2024-08-17T18:05:22.790129+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:31.439947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.547359+0000","last_change":"2024-08-17T18:04:43.771107+0000","last_active":"2024-08-17T18:05:20.547359+0000","last_peered":"2024-08-17T18:05:20.547359+0000","last_clean":"2024-08-17T18:05:20.547359+0000","last_became_active":"2024-08-17T18:04:43.770799+0000","last_became_peered":"2024-08-17T18:04:43.770799+0000","last_unstale":"2024-08-17T18:05:20.547359+0000","last_undegraded":"2024-08-17T18:05:20.547359+0000","last_fullsized":"2024-08-17T18:05:20.547359+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:09:04.350798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1477,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.545334+0000","last_change":"2024-08-17T18:04:43.772808+0000","last_active":"2024-08-17T18:05:20.545334+0000","last_peered":"2024-08-17T18:05:20.545334+0000","last_clean":"2024-08-17T18:05:20.545334+0000","last_became_active":"2024-08-17T18:04:43.772550+0000","last_became_peered":"2024-08-17T18:04:43.772550+0000","last_unstale":"2024-08-17T18:05:20.545334+0000","last_undegraded":"2024-08-17T18:05:20.545334+0000","last_fullsized":"2024-08-17T18:05:20.545334+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:04.052168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":26,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.547628+0000","last_change":"2024-08-17T18:04:43.781111+0000","last_active":"2024-08-17T18:05:20.547628+0000","last_peered":"2024-08-17T18:05:20.547628+0000","last_clean":"2024-08-17T18:05:20.547628+0000","last_became_active":"2024-08-17T18:04:43.771839+0000","last_became_peered":"2024-08-17T18:04:43.771839+0000","last_unstale":"2024-08-17T18:05:20.547628+0000","last_undegraded":"2024-08-17T18:05:20.547628+0000","last_fullsized":"2024-08-17T18:05:20.547628+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:52:45.973049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1468,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.544943+0000","last_change":"2024-08-17T17:24:17.460203+0000","last_active":"2024-08-17T18:05:20.544943+0000","last_peered":"2024-08-17T18:05:20.544943+0000","last_clean":"2024-08-17T18:05:20.544943+0000","last_became_active":"2024-08-17T17:24:17.459682+0000","last_became_peered":"2024-08-17T17:24:17.459682+0000","last_unstale":"2024-08-17T18:05:20.544943+0000","last_undegraded":"2024-08-17T18:05:20.544943+0000","last_fullsized":"2024-08-17T18:05:20.544943+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:58.703700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":67,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.772684+0000","last_change":"2024-08-17T18:04:14.551061+0000","last_active":"2024-08-17T18:05:22.772684+0000","last_peered":"2024-08-17T18:05:22.772684+0000","last_clean":"2024-08-17T18:05:22.772684+0000","last_became_active":"2024-08-17T18:04:14.543296+0000","last_became_peered":"2024-08-17T18:04:14.543296+0000","last_unstale":"2024-08-17T18:05:22.772684+0000","last_undegraded":"2024-08-17T18:05:22.772684+0000","last_fullsized":"2024-08-17T18:05:22.772684+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:39:45.294265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:21.825622+0000","last_change":"2024-08-17T18:04:43.768583+0000","last_active":"2024-08-17T18:05:21.825622+0000","last_peered":"2024-08-17T18:05:21.825622+0000","last_clean":"2024-08-17T18:05:21.825622+0000","last_became_active":"2024-08-17T18:04:43.768278+0000","last_became_peered":"2024-08-17T18:04:43.768278+0000","last_unstale":"2024-08-17T18:05:21.825622+0000","last_undegraded":"2024-08-17T18:05:21.825622+0000","last_fullsized":"2024-08-17T18:05:21.825622+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:00:37.073591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":67,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.545878+0000","last_change":"2024-08-17T18:04:14.547798+0000","last_active":"2024-08-17T18:05:20.545878+0000","last_peered":"2024-08-17T18:05:20.545878+0000","last_clean":"2024-08-17T18:05:20.545878+0000","last_became_active":"2024-08-17T18:04:14.527496+0000","last_became_peered":"2024-08-17T18:04:14.527496+0000","last_unstale":"2024-08-17T18:05:20.545878+0000","last_undegraded":"2024-08-17T18:05:20.545878+0000","last_fullsized":"2024-08-17T18:05:20.545878+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:25:54.338846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1533,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.545119+0000","last_change":"2024-08-17T18:04:43.813536+0000","last_active":"2024-08-17T18:05:20.545119+0000","last_peered":"2024-08-17T18:05:20.545119+0000","last_clean":"2024-08-17T18:05:20.545119+0000","last_became_active":"2024-08-17T18:04:43.799823+0000","last_became_peered":"2024-08-17T18:04:43.799823+0000","last_unstale":"2024-08-17T18:05:20.545119+0000","last_undegraded":"2024-08-17T18:05:20.545119+0000","last_fullsized":"2024-08-17T18:05:20.545119+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:04:10.837113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2238,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.904465+0000","last_change":"2024-08-17T18:03:51.277803+0000","last_active":"2024-08-17T18:05:22.904465+0000","last_peered":"2024-08-17T18:05:22.904465+0000","last_clean":"2024-08-17T18:05:22.904465+0000","last_became_active":"2024-08-17T18:03:51.277408+0000","last_became_peered":"2024-08-17T18:03:51.277408+0000","last_unstale":"2024-08-17T18:05:22.904465+0000","last_undegraded":"2024-08-17T18:05:22.904465+0000","last_fullsized":"2024-08-17T18:05:22.904465+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:12.683553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":191,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.545099+0000","last_change":"2024-08-17T18:04:14.527380+0000","last_active":"2024-08-17T18:05:20.545099+0000","last_peered":"2024-08-17T18:05:20.545099+0000","last_clean":"2024-08-17T18:05:20.545099+0000","last_became_active":"2024-08-17T18:04:14.527076+0000","last_became_peered":"2024-08-17T18:04:14.527076+0000","last_unstale":"2024-08-17T18:05:20.545099+0000","last_undegraded":"2024-08-17T18:05:20.545099+0000","last_fullsized":"2024-08-17T18:05:20.545099+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:36:35.583709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":48,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.790476+0000","last_change":"2024-08-17T18:04:14.541814+0000","last_active":"2024-08-17T18:05:22.790476+0000","last_peered":"2024-08-17T18:05:22.790476+0000","last_clean":"2024-08-17T18:05:22.790476+0000","last_became_active":"2024-08-17T18:04:14.541403+0000","last_became_peered":"2024-08-17T18:04:14.541403+0000","last_unstale":"2024-08-17T18:05:22.790476+0000","last_undegraded":"2024-08-17T18:05:22.790476+0000","last_fullsized":"2024-08-17T18:05:22.790476+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:07:45.294924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":48,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.789915+0000","last_change":"2024-08-17T18:04:14.539149+0000","last_active":"2024-08-17T18:05:22.789915+0000","last_peered":"2024-08-17T18:05:22.789915+0000","last_clean":"2024-08-17T18:05:22.789915+0000","last_became_active":"2024-08-17T18:04:14.538852+0000","last_became_peered":"2024-08-17T18:04:14.538852+0000","last_unstale":"2024-08-17T18:05:22.789915+0000","last_undegraded":"2024-08-17T18:05:22.789915+0000","last_fullsized":"2024-08-17T18:05:22.789915+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:02:55.704464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.547543+0000","last_change":"2024-08-17T18:04:43.771720+0000","last_active":"2024-08-17T18:05:20.547543+0000","last_peered":"2024-08-17T18:05:20.547543+0000","last_clean":"2024-08-17T18:05:20.547543+0000","last_became_active":"2024-08-17T18:04:43.771137+0000","last_became_peered":"2024-08-17T18:04:43.771137+0000","last_unstale":"2024-08-17T18:05:20.547543+0000","last_undegraded":"2024-08-17T18:05:20.547543+0000","last_fullsized":"2024-08-17T18:05:20.547543+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:57.647909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":56,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-08-17T18:05:20.544203+0000","last_change":"2024-08-17T18:03:51.283872+0000","last_active":"2024-08-17T18:05:20.544203+0000","last_peered":"2024-08-17T18:05:20.544203+0000","last_clean":"2024-08-17T18:05:20.544203+0000","last_became_active":"2024-08-17T18:03:51.267197+0000","last_became_peered":"2024-08-17T18:03:51.267197+0000","last_unstale":"2024-08-17T18:05:20.544203+0000","last_undegraded":"2024-08-17T18:05:20.544203+0000","last_fullsized":"2024-08-17T18:05:20.544203+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:18:59.157786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2238,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-08-17T18:05:22.904154+0000","last_change":"2024-08-17T18:03:51.277491+0000","last_active":"2024-08-17T18:05:22.904154+0000","last_peered":"2024-08-17T18:05:22.904154+0000","last_clean":"2024-08-17T18:05:22.904154+0000","last_became_active":"2024-08-17T18:03:51.277178+0000","last_became_peered":"2024-08-17T18:03:51.277178+0000","last_unstale":"2024-08-17T18:05:22.904154+0000","last_undegraded":"2024-08-17T18:05:22.904154+0000","last_fullsized":"2024-08-17T18:05:22.904154+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:21:35.319085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":55,"num_pg":26,"stat_sum":{"num_bytes":6005194776,"num_objects":91633,"num_object_clones":0,"num_object_copies":274899,"num_objects_missing_on_primary":4074,"num_objects_missing":4074,"num_objects_degraded":26229,"num_objects_misplaced":5675,"num_objects_unfound":0,"num_objects_dirty":91633,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":92257,"num_write_kb":5884417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6155,"num_bytes_recovered":403374080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18541846528,"data_stored":18541838384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148696,"ondisk_log_size":148696,"up":78,"acting":74,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":6830,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234673,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496311,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4283488,"kb_used_data":3638808,"kb_used_omap":1,"kb_used_meta":644670,"kb_avail":89461664,"statfs":{"total":95995035648,"available":91608743936,"internally_reserved":0,"allocated":3726139392,"data_stored":3721093563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":660142535},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565552,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4611212,"kb_used_data":3897412,"kb_used_omap":72,"kb_used_meta":713719,"kb_avail":89133940,"statfs":{"total":95995035648,"available":91273154560,"internally_reserved":0,"allocated":3990949888,"data_stored":3985919288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":730848750},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":1399,"seq":6008659247154,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4312664,"kb_used_data":4104208,"kb_used_omap":131,"kb_used_meta":208316,"kb_avail":89432488,"statfs":{"total":95995035648,"available":91578867712,"internally_reserved":0,"allocated":4202708992,"data_stored":4197671316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134291,"internal_metadata":213316461},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743186942,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3237520,"kb_used_data":2637704,"kb_used_omap":66,"kb_used_meta":599741,"kb_avail":90507632,"statfs":{"total":95995035648,"available":92679815168,"internally_reserved":0,"allocated":2701008896,"data_stored":2695974625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":614135467},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448581,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1779864,"kb_used_data":1396688,"kb_used_omap":75,"kb_used_meta":383028,"kb_avail":91965288,"statfs":{"total":95995035648,"available":94172454912,"internally_reserved":0,"allocated":1430208512,"data_stored":1425166726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":392220863},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757946,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1390164,"kb_used_data":817544,"kb_used_omap":1,"kb_used_meta":572542,"kb_avail":92354988,"statfs":{"total":95995035648,"available":94571507712,"internally_reserved":0,"allocated":837165056,"data_stored":832130859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":586283462},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019584,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2311584,"kb_used_data":1730904,"kb_used_omap":1,"kb_used_meta":580670,"kb_avail":91433568,"statfs":{"total":95995035648,"available":93627973632,"internally_reserved":0,"allocated":1772445696,"data_stored":1767399370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":594606534},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1416105984,"data_stored":1416101912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2681274368,"data_stored":2681274368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4179755008,"data_stored":4179755008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3969515520,"data_stored":3969515520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1757024256,"data_stored":1757020184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":829554688,"data_stored":829554688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3708616704,"data_stored":3708616704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T18:05:30.623 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool set unique_pool_2 pgp_num 26 2024-08-17T18:05:30.624 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:30.648 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:30.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:30.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:30.750 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:30.772 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:30.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:30.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:30.843 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:05:30.852 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:30 smithi136 ceph-mon[109861]: from='client.45707 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:05:30.852 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:30 smithi136 ceph-mon[115078]: from='client.45707 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:05:30.874 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:30.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:30.902 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:30.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:31.004 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:30 smithi165 ceph-mon[105517]: from='client.45707 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:05:31.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:31.031 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:31.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:31.132 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:31.154 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:31.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:31.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:31.256 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:31.282 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:31.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:31.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:31.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:31.413 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:31.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:31.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:31.517 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:31.544 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:31.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:31.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:31.646 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:31.659 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:31 smithi165 ceph-mon[105517]: pgmap v2617: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 105 B/s rd, 16 MiB/s wr, 293 op/s; 26089/274902 objects degraded (9.490%); 5675/274902 objects misplaced (2.064%); 4.3 MiB/s, 68 objects/s recovering 2024-08-17T18:05:31.668 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:31.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:31.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:31.681 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:05:31.682 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:05:31.682 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-08-17T18:05:31.682 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:05:31.770 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:31.799 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:31.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:31.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:31.822 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:31 smithi136 ceph-mon[109861]: pgmap v2617: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 105 B/s rd, 16 MiB/s wr, 293 op/s; 26089/274902 objects degraded (9.490%); 5675/274902 objects misplaced (2.064%); 4.3 MiB/s, 68 objects/s recovering 2024-08-17T18:05:31.822 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:31 smithi136 ceph-mon[115078]: pgmap v2617: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 105 B/s rd, 16 MiB/s wr, 293 op/s; 26089/274902 objects degraded (9.490%); 5675/274902 objects misplaced (2.064%); 4.3 MiB/s, 68 objects/s recovering 2024-08-17T18:05:31.902 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:31.931 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:31.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:31.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:32.033 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:32.057 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:32.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:32.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:32.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:32.185 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:32.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:32.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:32.278 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:05:32.287 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:32.310 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:32.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:32.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:32.364 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-17T18:05:32.413 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:32.439 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:32.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:32.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:32.541 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:32.567 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:32.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:32.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:32.640 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:05:32.669 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:32.693 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:32.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:32.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:32.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:05:32.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:05:32.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:05:32.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:05:32.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:32 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:05:32.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:05:32.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:05:32.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:05:32.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:05:32.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:32 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:05:32.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:05:32.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:05:32.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:05:32.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:05:32.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:32 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:05:32.795 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:32.821 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:32.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:32.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:32.924 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:32.945 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:32.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:32.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:33.047 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:33.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:33.074 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:33.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:33.175 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:33.198 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:33.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:33.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:33.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:33.323 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:33.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:33.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:33.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:33.449 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:33.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:33.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:33.550 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:33.576 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:33.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:33.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:33.588 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:05:33.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:33.704 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:33.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:33.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:33.806 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:33.829 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:33.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:33.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:33.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:33.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:33 smithi136 ceph-mon[109861]: pgmap v2618: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 102 B/s rd, 7.7 MiB/s wr, 311 op/s; 25775/270399 objects degraded (9.532%); 5675/270399 objects misplaced (2.099%); 5.8 MiB/s, 93 objects/s recovering 2024-08-17T18:05:33.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:33 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3387318126' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:05:33.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:33 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:05:33.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:33 smithi136 ceph-mon[115078]: pgmap v2618: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 102 B/s rd, 7.7 MiB/s wr, 311 op/s; 25775/270399 objects degraded (9.532%); 5675/270399 objects misplaced (2.099%); 5.8 MiB/s, 93 objects/s recovering 2024-08-17T18:05:33.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:33 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3387318126' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:05:33.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:33 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:05:33.955 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:33.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:33.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:33 smithi165 ceph-mon[105517]: pgmap v2618: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 102 B/s rd, 7.7 MiB/s wr, 311 op/s; 25775/270399 objects degraded (9.532%); 5675/270399 objects misplaced (2.099%); 5.8 MiB/s, 93 objects/s recovering 2024-08-17T18:05:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:33 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3387318126' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:05:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:33 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:05:34.057 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:34.080 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:34.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:34.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:34.182 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:34.205 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:34.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:34.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:34.307 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:34.331 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:34.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:34.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:34.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:34.458 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:34.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:34.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:34.560 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:34.590 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:34.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:34.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:34.692 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:34.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:34 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:05:34.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:34 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:05:34.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:34 smithi136 ceph-mon[109861]: osdmap e1477: 8 total, 8 up, 7 in 2024-08-17T18:05:34.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:34 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 25775/270399 objects degraded (9.532%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:34.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:34 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:05:34.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:34 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:05:34.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:34 smithi136 ceph-mon[115078]: osdmap e1477: 8 total, 8 up, 7 in 2024-08-17T18:05:34.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:34 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 25775/270399 objects degraded (9.532%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:34.717 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:34.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:34.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:34.819 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:34.847 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:34.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:34.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:34.949 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:34.975 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:34.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:34.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:34 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:05:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:34 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:05:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:34 smithi165 ceph-mon[105517]: osdmap e1477: 8 total, 8 up, 7 in 2024-08-17T18:05:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:34 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 25775/270399 objects degraded (9.532%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:35.076 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:35.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:35.109 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:35.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:35.210 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:35.241 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:35.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:35.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:35.343 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:35.368 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:35.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:35.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:35.470 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:35.495 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:35.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:35.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:35.596 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:05:35.596 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:05:35.596 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:35.624 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:05:35.624 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:05:35.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:35.625 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:35.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:35.643 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:05:35.644 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:05:35.644 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:35 smithi136 ceph-mon[109861]: pgmap v2620: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 102 B/s rd, 7.5 MiB/s wr, 318 op/s; 25775/270072 objects degraded (9.544%); 5136/270072 objects misplaced (1.902%); 6.0 MiB/s, 96 objects/s recovering 2024-08-17T18:05:35.644 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:35 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3969632850' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:05:35.645 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:35 smithi136 ceph-mon[115078]: pgmap v2620: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 102 B/s rd, 7.5 MiB/s wr, 318 op/s; 25775/270072 objects degraded (9.544%); 5136/270072 objects misplaced (1.902%); 6.0 MiB/s, 96 objects/s recovering 2024-08-17T18:05:35.645 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:35 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3969632850' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:05:35.667 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:05:35.668 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:05:35.691 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:05:35.692 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:05:35.713 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:05:35.713 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:05:35.726 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:35.739 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:05:35.739 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:05:35.757 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:35.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:35.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:35.769 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:05:35.769 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:05:35.860 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:35.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:35.884 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:35.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:35.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:35 smithi165 ceph-mon[105517]: pgmap v2620: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 102 B/s rd, 7.5 MiB/s wr, 318 op/s; 25775/270072 objects degraded (9.544%); 5136/270072 objects misplaced (1.902%); 6.0 MiB/s, 96 objects/s recovering 2024-08-17T18:05:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:35 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3969632850' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:05:36.012 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:36.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:36.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:36.114 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:36.136 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:36.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:36.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:36.238 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:36.259 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:36.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:36.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:36.361 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:36.386 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:36.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:36.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:36.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:36.519 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:36.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:36.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:36.608 INFO:teuthology.orchestra.run.smithi136.stderr:set pool 55 pgp_num to 26 2024-08-17T18:05:36.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:36.645 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:36.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:36.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:36.746 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:36.770 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:36.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:36.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:36.865 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:05:36.871 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:36.888 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:36 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3470476065' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "26"} : dispatch 2024-08-17T18:05:36.888 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:36 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "26"} : dispatch 2024-08-17T18:05:36.888 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:05:36.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:36 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3470476065' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "26"} : dispatch 2024-08-17T18:05:36.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:36 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "26"} : dispatch 2024-08-17T18:05:36.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:05:36.894 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:36.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:36.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:36.966 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:36 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3470476065' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "26"} : dispatch 2024-08-17T18:05:36.966 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:36 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "26"} : dispatch 2024-08-17T18:05:36.966 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:05:36.996 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:37.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:37.027 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:37.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:37.128 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:37.159 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:37.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:37.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:37.261 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:37.289 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:37.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:37.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:37.391 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:37.414 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:37.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:37.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:37.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:37.518 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:05:37.539 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:37.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:37.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:37.643 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:37.652 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:37 smithi136 ceph-mon[109861]: pgmap v2621: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 102 B/s rd, 7.1 MiB/s wr, 323 op/s; 25541/269724 objects degraded (9.469%); 5136/269724 objects misplaced (1.904%); 6.7 MiB/s, 107 objects/s recovering 2024-08-17T18:05:37.652 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:37 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "26"}]': finished 2024-08-17T18:05:37.652 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:37 smithi136 ceph-mon[109861]: osdmap e1478: 8 total, 8 up, 7 in 2024-08-17T18:05:37.652 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:05:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:05:37] "GET /metrics HTTP/1.1" 200 36642 "" "Prometheus/2.51.0" 2024-08-17T18:05:37.652 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:37 smithi136 ceph-mon[115078]: pgmap v2621: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 102 B/s rd, 7.1 MiB/s wr, 323 op/s; 25541/269724 objects degraded (9.469%); 5136/269724 objects misplaced (1.904%); 6.7 MiB/s, 107 objects/s recovering 2024-08-17T18:05:37.652 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:37 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "26"}]': finished 2024-08-17T18:05:37.653 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:37 smithi136 ceph-mon[115078]: osdmap e1478: 8 total, 8 up, 7 in 2024-08-17T18:05:37.670 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:37.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:37.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:37.771 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:37.798 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:37.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:37.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:37.900 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:37.924 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:37.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:37.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:37.959 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:37 smithi165 ceph-mon[105517]: pgmap v2621: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 102 B/s rd, 7.1 MiB/s wr, 323 op/s; 25541/269724 objects degraded (9.469%); 5136/269724 objects misplaced (1.904%); 6.7 MiB/s, 107 objects/s recovering 2024-08-17T18:05:37.960 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:37 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "26"}]': finished 2024-08-17T18:05:37.960 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:37 smithi165 ceph-mon[105517]: osdmap e1478: 8 total, 8 up, 7 in 2024-08-17T18:05:38.026 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:38.050 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:38.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:38.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:38.152 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:38.174 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:38.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:38.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:38.276 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:38.305 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:38.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:38.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:38.408 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:38.434 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:38.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:38.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:38.533 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:05:38.536 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:38.600 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:38.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:38.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:38.701 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:38.728 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:38.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:38.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:38.784 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:05:38.830 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:38.862 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:38.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:38.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:38.964 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:38.986 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:38.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:38.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:39.087 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:39.111 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:39.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:39.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:39.213 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:39.245 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:39.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:39.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:39.347 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:39.377 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:39.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:39.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:39.479 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:39.503 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:39.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:39.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:39.605 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:39.606 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:05:39.635 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:39.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:39.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:39.737 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:39.761 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:39.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:39.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:39.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:39.889 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:39.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:39.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:39.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:39 smithi165 ceph-mon[105517]: pgmap v2623: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 414 op/s; 25208/264948 objects degraded (9.514%); 4524/264948 objects misplaced (1.708%); 10 MiB/s, 167 objects/s recovering 2024-08-17T18:05:39.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:39 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2331476367' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:05:39.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:39 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:05:39.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:39 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 25208/264948 objects degraded (9.514%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:39.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:39 smithi136 ceph-mon[115078]: pgmap v2623: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 414 op/s; 25208/264948 objects degraded (9.514%); 4524/264948 objects misplaced (1.708%); 10 MiB/s, 167 objects/s recovering 2024-08-17T18:05:39.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:39 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2331476367' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:05:39.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:39 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:05:39.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:39 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 25208/264948 objects degraded (9.514%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:39.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:39 smithi136 ceph-mon[109861]: pgmap v2623: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 414 op/s; 25208/264948 objects degraded (9.514%); 4524/264948 objects misplaced (1.708%); 10 MiB/s, 167 objects/s recovering 2024-08-17T18:05:39.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:39 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2331476367' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:05:39.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:39 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:05:39.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:39 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 25208/264948 objects degraded (9.514%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:39.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:40.023 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:40.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:40.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:40.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:40.161 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:40.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:40.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:40.263 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:40.291 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:40.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:40.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:40.393 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:40.423 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:40.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:40.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:40.525 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:40.547 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:40.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:40.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:40.588 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:05:40.588 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:05:40.649 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:40.676 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:40.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:40.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:40.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:40 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:05:40.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:40 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:05:40.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:40 smithi136 ceph-mon[109861]: osdmap e1479: 8 total, 8 up, 7 in 2024-08-17T18:05:40.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:40 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:05:40.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:40 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:05:40.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:40 smithi136 ceph-mon[115078]: osdmap e1479: 8 total, 8 up, 7 in 2024-08-17T18:05:40.779 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:40.805 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:40.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:40.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:40.907 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:40.929 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:40.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:40.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:40 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:05:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:40 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:05:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:40 smithi165 ceph-mon[105517]: osdmap e1479: 8 total, 8 up, 7 in 2024-08-17T18:05:41.031 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:41.055 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:41.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:41.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:41.157 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:41.184 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:41.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:41.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:41.285 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:41.311 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:41.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:41.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:41.413 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:41.437 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:41.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:41.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:41.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:41.568 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:41.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:41.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:41.669 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:41.700 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:41.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:41.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:41.802 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:41.833 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:41.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:41.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:41.936 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:41.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:41 smithi136 ceph-mon[115078]: pgmap v2625: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 274 op/s; 24978/264600 objects degraded (9.440%); 4524/264600 objects misplaced (1.710%); 8.2 MiB/s, 130 objects/s recovering 2024-08-17T18:05:41.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:41 smithi136 ceph-mon[109861]: pgmap v2625: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 274 op/s; 24978/264600 objects degraded (9.440%); 4524/264600 objects misplaced (1.710%); 8.2 MiB/s, 130 objects/s recovering 2024-08-17T18:05:41.957 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:41.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:41.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:42.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:41 smithi165 ceph-mon[105517]: pgmap v2625: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 274 op/s; 24978/264600 objects degraded (9.440%); 4524/264600 objects misplaced (1.710%); 8.2 MiB/s, 130 objects/s recovering 2024-08-17T18:05:42.059 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:42.082 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:42.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:42.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:42.184 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:42.210 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:42.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:42.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:42.311 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:42.339 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:42.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:42.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:42.419 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 0, 5] out_osds: [7] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:05:42.419 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:05:42.419 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-08-17T18:05:42.419 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight-by-utilization 1000 3.0 2024-08-17T18:05:42.440 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:42.488 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:42.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:42.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:42.514 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-17T18:05:42.560 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:05:42.561 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:05:42.587 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:05:42.587 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:05:42.590 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:42.608 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:05:42.608 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:05:42.617 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:42.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:42.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:42.632 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:05:42.632 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:05:42.657 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:05:42.657 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:05:42.676 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:05:42.677 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:05:42.679 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:05:42.697 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:05:42.697 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:05:42.720 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:42.721 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:05:42.721 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:05:42.745 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:42.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:42.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:42.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:42 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2274598895' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:05:42.804 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:42 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2274598895' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:05:42.805 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:42 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2274598895' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:05:42.808 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:05:42.847 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:42.870 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:42.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:42.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:42.972 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:42.994 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:42.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:42.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:43.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:43.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:43.130 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:43.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:43.231 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:43.260 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:43.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:43.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:43.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:43.388 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:43.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:43.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:43.489 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:43.519 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:43.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:43.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:43.763 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:43.788 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:43.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:43.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:43.887 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:43 smithi136 ceph-mon[109861]: pgmap v2626: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 439 op/s; 24628/259521 objects degraded (9.490%); 4524/259521 objects misplaced (1.743%); 9.5 MiB/s, 152 objects/s recovering 2024-08-17T18:05:43.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:43 smithi136 ceph-mon[115078]: pgmap v2626: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 439 op/s; 24628/259521 objects degraded (9.490%); 4524/259521 objects misplaced (1.743%); 9.5 MiB/s, 152 objects/s recovering 2024-08-17T18:05:43.890 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:43.915 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:43.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:43.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:43 smithi165 ceph-mon[105517]: pgmap v2626: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 439 op/s; 24628/259521 objects degraded (9.490%); 4524/259521 objects misplaced (1.743%); 9.5 MiB/s, 152 objects/s recovering 2024-08-17T18:05:44.017 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:44.040 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:44.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:44.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:44.057 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:05:44.142 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:44.175 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:44.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:44.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:44.276 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:44.309 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:44.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:44.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:44.411 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:44.436 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:44.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:44.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:44.538 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:44.567 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:44.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:44.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:44.669 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:44.691 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:44.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:44.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:44.793 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:44.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:44.816 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:44.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:44.917 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:44.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:44 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 24628/259521 objects degraded (9.490%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:44.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:44 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 24628/259521 objects degraded (9.490%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:44.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:44.948 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:44.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:44 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 24628/259521 objects degraded (9.490%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T18:05:45.048 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:45.080 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:45.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:45.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:45.093 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:05:45.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:45.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:45.218 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:45.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:45.320 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:45.345 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:45.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:45.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:45.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:45.472 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:45.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:45.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:45.497 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:05:45.497 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T18:05:45.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T18:05:45.498 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:05:45.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.498 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:104813 54.11 54.64e93111 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T18:05:45.498 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.630215+0000", 2024-08-17T18:05:45.498 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 359.86245002300001, 2024-08-17T18:05:45.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.83036598800000005, 2024-08-17T18:05:45.498 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:45.498 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:05:45.498 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:45.498 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:45.498 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:45.498 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104813 2024-08-17T18:05:45.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.499 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:45.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:45.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.630215+0000", 2024-08-17T18:05:45.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:45.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.630215+0000", 2024-08-17T18:05:45.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:45.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.630213+0000", 2024-08-17T18:05:45.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:05:45.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:45.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.630324+0000", 2024-08-17T18:05:45.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011047 2024-08-17T18:05:45.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:45.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.630338+0000", 2024-08-17T18:05:45.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.452e-05 2024-08-17T18:05:45.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:45.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.630344+0000", 2024-08-17T18:05:45.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3029999999999999e-06 2024-08-17T18:05:45.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.630399+0000", 2024-08-17T18:05:45.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5204999999999997e-05 2024-08-17T18:05:45.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.630409+0000", 2024-08-17T18:05:45.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0363999999999999e-05 2024-08-17T18:05:45.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460567+0000", 2024-08-17T18:05:45.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.83015739499999996 2024-08-17T18:05:45.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:45.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460581+0000", 2024-08-17T18:05:45.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4001e-05 2024-08-17T18:05:45.503 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.503 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:45.503 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.503 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:104818 54.11 54.1190c0f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T18:05:45.503 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.637872+0000", 2024-08-17T18:05:45.503 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 359.85479282199998, 2024-08-17T18:05:45.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.82274040599999998, 2024-08-17T18:05:45.503 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:45.504 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:05:45.504 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:45.504 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:45.504 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:45.504 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104818 2024-08-17T18:05:45.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.504 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:45.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:45.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.637872+0000", 2024-08-17T18:05:45.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:45.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.637872+0000", 2024-08-17T18:05:45.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:45.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.637870+0000", 2024-08-17T18:05:45.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:05:45.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:45.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.637930+0000", 2024-08-17T18:05:45.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9210999999999999e-05 2024-08-17T18:05:45.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:45.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.637943+0000", 2024-08-17T18:05:45.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3332e-05 2024-08-17T18:05:45.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:45.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.637948+0000", 2024-08-17T18:05:45.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4659999999999998e-06 2024-08-17T18:05:45.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.637992+0000", 2024-08-17T18:05:45.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3899000000000001e-05 2024-08-17T18:05:45.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.638000+0000", 2024-08-17T18:05:45.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9039999999999992e-06 2024-08-17T18:05:45.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460604+0000", 2024-08-17T18:05:45.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.82260419100000004 2024-08-17T18:05:45.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:45.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460612+0000", 2024-08-17T18:05:45.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7729999999999999e-06 2024-08-17T18:05:45.508 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.508 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:45.508 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.508 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:104840 54.11 54.d5652851 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T18:05:45.508 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.667423+0000", 2024-08-17T18:05:45.508 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 359.82524209600001, 2024-08-17T18:05:45.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.79322487200000003, 2024-08-17T18:05:45.508 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:45.509 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:05:45.509 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:45.509 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:45.509 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:45.509 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104840 2024-08-17T18:05:45.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.509 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:45.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:45.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.667423+0000", 2024-08-17T18:05:45.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:45.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.667423+0000", 2024-08-17T18:05:45.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:45.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.667422+0000", 2024-08-17T18:05:45.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:05:45.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:45.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.667485+0000", 2024-08-17T18:05:45.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2741000000000001e-05 2024-08-17T18:05:45.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:45.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.667491+0000", 2024-08-17T18:05:45.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8699999999999997e-06 2024-08-17T18:05:45.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:45.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.667494+0000", 2024-08-17T18:05:45.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5930000000000002e-06 2024-08-17T18:05:45.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.667535+0000", 2024-08-17T18:05:45.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1273999999999998e-05 2024-08-17T18:05:45.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.667543+0000", 2024-08-17T18:05:45.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1659999999999997e-06 2024-08-17T18:05:45.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460636+0000", 2024-08-17T18:05:45.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.79309297400000001 2024-08-17T18:05:45.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:45.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460647+0000", 2024-08-17T18:05:45.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1863e-05 2024-08-17T18:05:45.513 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.513 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:45.513 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.513 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:104844 54.11 54.e06561d1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T18:05:45.513 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.672468+0000", 2024-08-17T18:05:45.513 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 359.82019672899997, 2024-08-17T18:05:45.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.78822546199999999, 2024-08-17T18:05:45.513 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:45.513 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:05:45.513 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:45.514 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:45.514 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:45.514 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104844 2024-08-17T18:05:45.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.514 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:45.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.514 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:45.514 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.672468+0000", 2024-08-17T18:05:45.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.514 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:45.514 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.672468+0000", 2024-08-17T18:05:45.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:45.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.672467+0000", 2024-08-17T18:05:45.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:05:45.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:45.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.672534+0000", 2024-08-17T18:05:45.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6926000000000003e-05 2024-08-17T18:05:45.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:45.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.672544+0000", 2024-08-17T18:05:45.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.022e-05 2024-08-17T18:05:45.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:45.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.672549+0000", 2024-08-17T18:05:45.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.809e-06 2024-08-17T18:05:45.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.672586+0000", 2024-08-17T18:05:45.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6585999999999998e-05 2024-08-17T18:05:45.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.672593+0000", 2024-08-17T18:05:45.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8269999999999998e-06 2024-08-17T18:05:45.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460683+0000", 2024-08-17T18:05:45.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.78808987600000002 2024-08-17T18:05:45.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:45.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460693+0000", 2024-08-17T18:05:45.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0138e-05 2024-08-17T18:05:45.517 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.518 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:45.518 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.518 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:104848 54.11 54.d7e6d951 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T18:05:45.518 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.673087+0000", 2024-08-17T18:05:45.518 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 359.81957808300001, 2024-08-17T18:05:45.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.78763660400000002, 2024-08-17T18:05:45.518 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:45.518 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:05:45.518 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:45.518 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:45.518 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:45.519 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104848 2024-08-17T18:05:45.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.519 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:45.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:45.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.673087+0000", 2024-08-17T18:05:45.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:45.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.673087+0000", 2024-08-17T18:05:45.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:45.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.673086+0000", 2024-08-17T18:05:45.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:05:45.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:45.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.673524+0000", 2024-08-17T18:05:45.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000438591 2024-08-17T18:05:45.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:45.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.673539+0000", 2024-08-17T18:05:45.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4791e-05 2024-08-17T18:05:45.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:45.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.673546+0000", 2024-08-17T18:05:45.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2200000000000003e-06 2024-08-17T18:05:45.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.673598+0000", 2024-08-17T18:05:45.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1418000000000001e-05 2024-08-17T18:05:45.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.673605+0000", 2024-08-17T18:05:45.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.866e-06 2024-08-17T18:05:45.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460712+0000", 2024-08-17T18:05:45.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.78710790600000002 2024-08-17T18:05:45.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:45.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460723+0000", 2024-08-17T18:05:45.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0771000000000001e-05 2024-08-17T18:05:45.522 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.522 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:45.522 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.523 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:104917 54.11 54.eda9ea11 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T18:05:45.523 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.725191+0000", 2024-08-17T18:05:45.523 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 359.76747318000002, 2024-08-17T18:05:45.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.73558475400000001, 2024-08-17T18:05:45.523 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:45.523 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:05:45.523 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:45.523 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:45.523 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:45.523 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104917 2024-08-17T18:05:45.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.523 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:45.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:45.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.725191+0000", 2024-08-17T18:05:45.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:45.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.725191+0000", 2024-08-17T18:05:45.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:45.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.725191+0000", 2024-08-17T18:05:45.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:05:45.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:45.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.725274+0000", 2024-08-17T18:05:45.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3373000000000003e-05 2024-08-17T18:05:45.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:45.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.725287+0000", 2024-08-17T18:05:45.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3025e-05 2024-08-17T18:05:45.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:45.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.725294+0000", 2024-08-17T18:05:45.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6270000000000002e-06 2024-08-17T18:05:45.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.725350+0000", 2024-08-17T18:05:45.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5897999999999999e-05 2024-08-17T18:05:45.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.725358+0000", 2024-08-17T18:05:45.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8220000000000007e-06 2024-08-17T18:05:45.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460768+0000", 2024-08-17T18:05:45.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.73540999399999996 2024-08-17T18:05:45.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:45.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460776+0000", 2024-08-17T18:05:45.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6920000000000008e-06 2024-08-17T18:05:45.528 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.528 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:45.528 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.528 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:104940 54.11 54.f8fdfd11 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T18:05:45.529 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.741796+0000", 2024-08-17T18:05:45.529 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 359.75086909599997, 2024-08-17T18:05:45.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71900983799999996, 2024-08-17T18:05:45.529 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:45.529 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:05:45.529 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:45.529 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:45.529 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:45.529 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104940 2024-08-17T18:05:45.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.529 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:45.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:45.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.741796+0000", 2024-08-17T18:05:45.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:45.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.741796+0000", 2024-08-17T18:05:45.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:45.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.741794+0000", 2024-08-17T18:05:45.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:05:45.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:45.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.741827+0000", 2024-08-17T18:05:45.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2960000000000003e-05 2024-08-17T18:05:45.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:45.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.741844+0000", 2024-08-17T18:05:45.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7703e-05 2024-08-17T18:05:45.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:45.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.741852+0000", 2024-08-17T18:05:45.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6040000000000004e-06 2024-08-17T18:05:45.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.741897+0000", 2024-08-17T18:05:45.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4941999999999998e-05 2024-08-17T18:05:45.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.741905+0000", 2024-08-17T18:05:45.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1289999999999996e-06 2024-08-17T18:05:45.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460796+0000", 2024-08-17T18:05:45.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71889108999999995 2024-08-17T18:05:45.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:45.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460805+0000", 2024-08-17T18:05:45.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1630000000000002e-06 2024-08-17T18:05:45.533 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.533 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:45.533 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.534 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:104958 54.11 54.e8b0e6d1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T18:05:45.534 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.760286+0000", 2024-08-17T18:05:45.534 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 359.73237849200001, 2024-08-17T18:05:45.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.70054623599999999, 2024-08-17T18:05:45.534 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:45.534 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:05:45.534 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:45.534 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:45.534 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:45.534 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104958 2024-08-17T18:05:45.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.535 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:45.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:45.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.760286+0000", 2024-08-17T18:05:45.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:45.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.760286+0000", 2024-08-17T18:05:45.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:45.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.760285+0000", 2024-08-17T18:05:45.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:05:45.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:45.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.760326+0000", 2024-08-17T18:05:45.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0945000000000002e-05 2024-08-17T18:05:45.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:45.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.760341+0000", 2024-08-17T18:05:45.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5248999999999999e-05 2024-08-17T18:05:45.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:45.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.760347+0000", 2024-08-17T18:05:45.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6919999999999999e-06 2024-08-17T18:05:45.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.760399+0000", 2024-08-17T18:05:45.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2293999999999998e-05 2024-08-17T18:05:45.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.760407+0000", 2024-08-17T18:05:45.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4150000000000002e-06 2024-08-17T18:05:45.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460824+0000", 2024-08-17T18:05:45.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.70041780799999998 2024-08-17T18:05:45.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:45.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460832+0000", 2024-08-17T18:05:45.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0450000000000001e-06 2024-08-17T18:05:45.540 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.540 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:45.540 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.540 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:105002 54.11 54.3957dd31 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T18:05:45.541 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.797619+0000", 2024-08-17T18:05:45.541 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 359.69504613599997, 2024-08-17T18:05:45.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.66324079499999999, 2024-08-17T18:05:45.541 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:45.541 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:05:45.541 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:45.541 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:45.541 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:45.541 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 105002 2024-08-17T18:05:45.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.541 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:45.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:45.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.797619+0000", 2024-08-17T18:05:45.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:45.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.797619+0000", 2024-08-17T18:05:45.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:45.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.797617+0000", 2024-08-17T18:05:45.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:05:45.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:45.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.797682+0000", 2024-08-17T18:05:45.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4725000000000001e-05 2024-08-17T18:05:45.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:45.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.797696+0000", 2024-08-17T18:05:45.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4202999999999999e-05 2024-08-17T18:05:45.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:45.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.797703+0000", 2024-08-17T18:05:45.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4350000000000002e-06 2024-08-17T18:05:45.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.797745+0000", 2024-08-17T18:05:45.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2514e-05 2024-08-17T18:05:45.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.797753+0000", 2024-08-17T18:05:45.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4769999999999997e-06 2024-08-17T18:05:45.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460851+0000", 2024-08-17T18:05:45.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.66309843099999999 2024-08-17T18:05:45.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:45.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460859+0000", 2024-08-17T18:05:45.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1589999999999993e-06 2024-08-17T18:05:45.545 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.545 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:45.545 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.546 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:105025 54.11 54.479d4851 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T18:05:45.546 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.820146+0000", 2024-08-17T18:05:45.546 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 359.67251876199998, 2024-08-17T18:05:45.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.64075398500000003, 2024-08-17T18:05:45.546 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:45.546 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:05:45.546 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:45.546 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:45.546 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:45.546 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 105025 2024-08-17T18:05:45.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.546 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:45.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:45.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820146+0000", 2024-08-17T18:05:45.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:45.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820146+0000", 2024-08-17T18:05:45.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:45.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820145+0000", 2024-08-17T18:05:45.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:05:45.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:45.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820212+0000", 2024-08-17T18:05:45.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6554e-05 2024-08-17T18:05:45.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:45.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820222+0000", 2024-08-17T18:05:45.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0061000000000001e-05 2024-08-17T18:05:45.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:45.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820226+0000", 2024-08-17T18:05:45.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8289999999999998e-06 2024-08-17T18:05:45.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820269+0000", 2024-08-17T18:05:45.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2595000000000001e-05 2024-08-17T18:05:45.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820278+0000", 2024-08-17T18:05:45.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8189999999999998e-06 2024-08-17T18:05:45.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460890+0000", 2024-08-17T18:05:45.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.64061211900000004 2024-08-17T18:05:45.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:45.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460900+0000", 2024-08-17T18:05:45.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9019999999999994e-06 2024-08-17T18:05:45.550 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.550 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:45.550 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.550 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:105028 54.11 54.30ac10b1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T18:05:45.551 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.820327+0000", 2024-08-17T18:05:45.551 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 359.67233784199999, 2024-08-17T18:05:45.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.64060198099999999, 2024-08-17T18:05:45.551 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:45.551 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:05:45.551 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:45.551 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:45.551 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:45.551 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 105028 2024-08-17T18:05:45.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.551 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:45.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:45.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820327+0000", 2024-08-17T18:05:45.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:45.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820327+0000", 2024-08-17T18:05:45.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:45.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820326+0000", 2024-08-17T18:05:45.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:05:45.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:45.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820386+0000", 2024-08-17T18:05:45.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9802000000000001e-05 2024-08-17T18:05:45.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:45.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820395+0000", 2024-08-17T18:05:45.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9719999999999998e-06 2024-08-17T18:05:45.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:45.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820400+0000", 2024-08-17T18:05:45.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3319999999999999e-06 2024-08-17T18:05:45.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820440+0000", 2024-08-17T18:05:45.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9567999999999998e-05 2024-08-17T18:05:45.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.820447+0000", 2024-08-17T18:05:45.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7510000000000002e-06 2024-08-17T18:05:45.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460921+0000", 2024-08-17T18:05:45.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.64047475200000004 2024-08-17T18:05:45.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:45.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460929+0000", 2024-08-17T18:05:45.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3089999999999998e-06 2024-08-17T18:05:45.555 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.555 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:45.555 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.555 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180945 54.11 54:8e76512c:::benchmark_data_smithi136_273969_object58060:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1402)", 2024-08-17T18:05:45.555 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:26.787333+0000", 2024-08-17T18:05:45.555 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 258.70533125700001, 2024-08-17T18:05:45.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.74425201600000002, 2024-08-17T18:05:45.555 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:45.556 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:05:45.556 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:45.556 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:45.556 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:45.556 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180945 2024-08-17T18:05:45.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.556 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:45.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:45.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.787333+0000", 2024-08-17T18:05:45.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:45.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.787333+0000", 2024-08-17T18:05:45.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:45.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.787333+0000", 2024-08-17T18:05:45.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:05:45.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:45.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.787337+0000", 2024-08-17T18:05:45.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7210000000000002e-06 2024-08-17T18:05:45.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:45.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.787348+0000", 2024-08-17T18:05:45.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0429e-05 2024-08-17T18:05:45.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:45.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.787352+0000", 2024-08-17T18:05:45.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7659999999999998e-06 2024-08-17T18:05:45.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.787393+0000", 2024-08-17T18:05:45.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0114000000000003e-05 2024-08-17T18:05:45.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.787400+0000", 2024-08-17T18:05:45.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1940000000000001e-06 2024-08-17T18:05:45.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.525317+0000", 2024-08-17T18:05:45.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.73791703900000005 2024-08-17T18:05:45.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:45.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.525447+0000", 2024-08-17T18:05:45.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013059599999999999 2024-08-17T18:05:45.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:05:45.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.525925+0000", 2024-08-17T18:05:45.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00047725399999999998 2024-08-17T18:05:45.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:45.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.531415+0000", 2024-08-17T18:05:45.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054903060000000004 2024-08-17T18:05:45.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:45.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.531482+0000", 2024-08-17T18:05:45.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7330999999999996e-05 2024-08-17T18:05:45.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:45.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.531494+0000", 2024-08-17T18:05:45.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.171e-05 2024-08-17T18:05:45.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:05:45.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.531542+0000", 2024-08-17T18:05:45.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8192999999999998e-05 2024-08-17T18:05:45.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:45.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.531585+0000", 2024-08-17T18:05:45.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3237e-05 2024-08-17T18:05:45.562 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.562 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:45.562 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.562 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180963 54.11 54:894161ea:::benchmark_data_smithi136_273969_object58078:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1402)", 2024-08-17T18:05:45.562 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:26.865872+0000", 2024-08-17T18:05:45.563 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 258.62679308200001, 2024-08-17T18:05:45.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.66872560999999997, 2024-08-17T18:05:45.563 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:45.563 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:05:45.563 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:45.563 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:05:45.563 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:05:45.563 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180963 2024-08-17T18:05:45.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.563 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:45.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:45.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.865872+0000", 2024-08-17T18:05:45.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:45.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.865872+0000", 2024-08-17T18:05:45.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:45.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.865870+0000", 2024-08-17T18:05:45.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:05:45.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:45.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.865877+0000", 2024-08-17T18:05:45.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5060000000000001e-06 2024-08-17T18:05:45.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:45.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.865891+0000", 2024-08-17T18:05:45.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3947999999999999e-05 2024-08-17T18:05:45.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:45.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.865897+0000", 2024-08-17T18:05:45.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1539999999999999e-06 2024-08-17T18:05:45.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.865956+0000", 2024-08-17T18:05:45.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8879999999999999e-05 2024-08-17T18:05:45.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.865968+0000", 2024-08-17T18:05:45.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2449000000000001e-05 2024-08-17T18:05:45.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.528773+0000", 2024-08-17T18:05:45.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.66280509200000004 2024-08-17T18:05:45.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:45.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.528902+0000", 2024-08-17T18:05:45.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000128236 2024-08-17T18:05:45.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:05:45.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.529261+0000", 2024-08-17T18:05:45.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00035910800000000002 2024-08-17T18:05:45.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:45.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.534390+0000", 2024-08-17T18:05:45.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051290379999999998 2024-08-17T18:05:45.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:45.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.534494+0000", 2024-08-17T18:05:45.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103898 2024-08-17T18:05:45.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:45.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.534536+0000", 2024-08-17T18:05:45.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2039999999999997e-05 2024-08-17T18:05:45.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:05:45.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.534562+0000", 2024-08-17T18:05:45.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6624000000000001e-05 2024-08-17T18:05:45.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:45.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.534597+0000", 2024-08-17T18:05:45.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4937000000000001e-05 2024-08-17T18:05:45.569 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.569 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:45.569 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.569 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:39669 55.5 55:a211053c:::benchmark_data_smithi136_313476_object39668:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1456)", 2024-08-17T18:05:45.569 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:04:14.321096+0000", 2024-08-17T18:05:45.569 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 91.171568305999998, 2024-08-17T18:05:45.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6412118919999998, 2024-08-17T18:05:45.570 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:45.570 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:05:45.570 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:45.570 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:05:45.570 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:05:45.570 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39669 2024-08-17T18:05:45.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.570 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:45.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:45.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.321096+0000", 2024-08-17T18:05:45.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:45.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.321096+0000", 2024-08-17T18:05:45.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:45.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.321096+0000", 2024-08-17T18:05:45.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:05:45.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:45.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.321152+0000", 2024-08-17T18:05:45.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6632000000000003e-05 2024-08-17T18:05:45.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:45.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.321163+0000", 2024-08-17T18:05:45.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0351e-05 2024-08-17T18:05:45.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:45.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.321168+0000", 2024-08-17T18:05:45.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6570000000000002e-06 2024-08-17T18:05:45.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.321206+0000", 2024-08-17T18:05:45.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7873999999999997e-05 2024-08-17T18:05:45.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:45.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.321326+0000", 2024-08-17T18:05:45.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119917 2024-08-17T18:05:45.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-08-17T18:05:45.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.321456+0000", 2024-08-17T18:05:45.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000129775 2024-08-17T18:05:45.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:45.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.325896+0000", 2024-08-17T18:05:45.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044400120000000001 2024-08-17T18:05:45.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:45.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.327927+0000", 2024-08-17T18:05:45.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00203119 2024-08-17T18:05:45.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.338413+0000", 2024-08-17T18:05:45.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010486182 2024-08-17T18:05:45.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.338419+0000", 2024-08-17T18:05:45.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2029999999999997e-06 2024-08-17T18:05:45.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521368+0000", 2024-08-17T18:05:45.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18294843 2024-08-17T18:05:45.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521381+0000", 2024-08-17T18:05:45.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.305e-05 2024-08-17T18:05:45.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.962240+0000", 2024-08-17T18:05:45.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.440859237 2024-08-17T18:05:45.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:45.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.962308+0000", 2024-08-17T18:05:45.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8058000000000005e-05 2024-08-17T18:05:45.576 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.576 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:45.576 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.577 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:39677 55.5 55:a4f0b823:::benchmark_data_smithi136_313476_object39676:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1456)", 2024-08-17T18:05:45.577 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:04:14.326500+0000", 2024-08-17T18:05:45.577 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 91.166164785000007, 2024-08-17T18:05:45.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.636704041, 2024-08-17T18:05:45.577 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:45.577 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:05:45.577 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:45.577 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:05:45.577 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:05:45.577 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39677 2024-08-17T18:05:45.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.577 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:45.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:45.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.326500+0000", 2024-08-17T18:05:45.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:45.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.326500+0000", 2024-08-17T18:05:45.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:45.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.326499+0000", 2024-08-17T18:05:45.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:05:45.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:45.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.326530+0000", 2024-08-17T18:05:45.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1267999999999999e-05 2024-08-17T18:05:45.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:45.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.326541+0000", 2024-08-17T18:05:45.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1061e-05 2024-08-17T18:05:45.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:45.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.326546+0000", 2024-08-17T18:05:45.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0170000000000004e-06 2024-08-17T18:05:45.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.326573+0000", 2024-08-17T18:05:45.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6808000000000002e-05 2024-08-17T18:05:45.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:45.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.326671+0000", 2024-08-17T18:05:45.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7682000000000004e-05 2024-08-17T18:05:45.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-08-17T18:05:45.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.326760+0000", 2024-08-17T18:05:45.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9067999999999995e-05 2024-08-17T18:05:45.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:45.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.328093+0000", 2024-08-17T18:05:45.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013328789999999999 2024-08-17T18:05:45.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:45.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.329318+0000", 2024-08-17T18:05:45.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012251180000000001 2024-08-17T18:05:45.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.338516+0000", 2024-08-17T18:05:45.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0091985169999999998 2024-08-17T18:05:45.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.338523+0000", 2024-08-17T18:05:45.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4049999999999997e-06 2024-08-17T18:05:45.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521415+0000", 2024-08-17T18:05:45.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18289263 2024-08-17T18:05:45.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521425+0000", 2024-08-17T18:05:45.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2180000000000005e-06 2024-08-17T18:05:45.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.963136+0000", 2024-08-17T18:05:45.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4417117310000001 2024-08-17T18:05:45.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:45.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.963204+0000", 2024-08-17T18:05:45.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7597999999999999e-05 2024-08-17T18:05:45.585 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.585 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:45.585 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.586 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:39689 55.5 55:a653d258:::benchmark_data_smithi136_313476_object39688:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1457)", 2024-08-17T18:05:45.586 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:04:14.360299+0000", 2024-08-17T18:05:45.586 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 91.132365930000006, 2024-08-17T18:05:45.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6189520910000001, 2024-08-17T18:05:45.586 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:45.586 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:05:45.586 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:45.586 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:05:45.586 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:05:45.586 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39689 2024-08-17T18:05:45.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.586 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:45.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:45.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.360299+0000", 2024-08-17T18:05:45.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:45.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.360299+0000", 2024-08-17T18:05:45.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:45.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.360298+0000", 2024-08-17T18:05:45.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:05:45.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:45.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.360356+0000", 2024-08-17T18:05:45.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8545000000000003e-05 2024-08-17T18:05:45.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:45.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.360367+0000", 2024-08-17T18:05:45.588 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:45.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0529e-05 2024-08-17T18:05:45.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:45.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.360372+0000", 2024-08-17T18:05:45.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4589999999999997e-06 2024-08-17T18:05:45.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.360408+0000", 2024-08-17T18:05:45.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5410000000000001e-05 2024-08-17T18:05:45.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.360415+0000", 2024-08-17T18:05:45.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9419999999999998e-06 2024-08-17T18:05:45.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521452+0000", 2024-08-17T18:05:45.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16103684600000001 2024-08-17T18:05:45.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521459+0000", 2024-08-17T18:05:45.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7530000000000001e-06 2024-08-17T18:05:45.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.963231+0000", 2024-08-17T18:05:45.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.441771315 2024-08-17T18:05:45.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:45.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.963334+0000", 2024-08-17T18:05:45.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103631 2024-08-17T18:05:45.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-08-17T18:05:45.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.963451+0000", 2024-08-17T18:05:45.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116422 2024-08-17T18:05:45.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:45.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.967463+0000", 2024-08-17T18:05:45.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040121000000000002 2024-08-17T18:05:45.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:45.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.972526+0000", 2024-08-17T18:05:45.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050626539999999998 2024-08-17T18:05:45.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:45.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.977600+0000", 2024-08-17T18:05:45.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050747709999999996 2024-08-17T18:05:45.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:05:45.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.977662+0000", 2024-08-17T18:05:45.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1323999999999998e-05 2024-08-17T18:05:45.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:45.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.979251+0000", 2024-08-17T18:05:45.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001589175 2024-08-17T18:05:45.595 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.595 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:45.595 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.595 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:39692 55.f 55:f8af78c3:::benchmark_data_smithi136_313476_object39691:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1457)", 2024-08-17T18:05:45.595 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:04:14.362183+0000", 2024-08-17T18:05:45.595 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 91.130481437, 2024-08-17T18:05:45.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.610310522, 2024-08-17T18:05:45.595 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:45.595 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:05:45.595 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:45.596 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:05:45.596 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:05:45.596 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39692 2024-08-17T18:05:45.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.596 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:45.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.596 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:45.596 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.362183+0000", 2024-08-17T18:05:45.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.596 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:45.596 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.362183+0000", 2024-08-17T18:05:45.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:45.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.362182+0000", 2024-08-17T18:05:45.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:05:45.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:45.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.362240+0000", 2024-08-17T18:05:45.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7550000000000003e-05 2024-08-17T18:05:45.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:45.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.362249+0000", 2024-08-17T18:05:45.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8689999999999993e-06 2024-08-17T18:05:45.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:45.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.362253+0000", 2024-08-17T18:05:45.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2150000000000003e-06 2024-08-17T18:05:45.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.362293+0000", 2024-08-17T18:05:45.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0093999999999999e-05 2024-08-17T18:05:45.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.362301+0000", 2024-08-17T18:05:45.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2649999999999996e-06 2024-08-17T18:05:45.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521423+0000", 2024-08-17T18:05:45.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.159121755 2024-08-17T18:05:45.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521435+0000", 2024-08-17T18:05:45.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1963e-05 2024-08-17T18:05:45.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.945301+0000", 2024-08-17T18:05:45.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.423865945 2024-08-17T18:05:45.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:45.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.945438+0000", 2024-08-17T18:05:45.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000136804 2024-08-17T18:05:45.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-08-17T18:05:45.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.945560+0000", 2024-08-17T18:05:45.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122545 2024-08-17T18:05:45.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:45.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.947846+0000", 2024-08-17T18:05:45.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022854910000000002 2024-08-17T18:05:45.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:45.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.949142+0000", 2024-08-17T18:05:45.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012963650000000001 2024-08-17T18:05:45.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:45.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.970816+0000", 2024-08-17T18:05:45.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021673504999999999 2024-08-17T18:05:45.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:05:45.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.970860+0000", 2024-08-17T18:05:45.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4206999999999999e-05 2024-08-17T18:05:45.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:45.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.972494+0000", 2024-08-17T18:05:45.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016336860000000001 2024-08-17T18:05:45.602 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.602 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:45.602 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.603 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:39695 55.5 55:a6056f3b:::benchmark_data_smithi136_313476_object39694:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1457)", 2024-08-17T18:05:45.603 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:04:14.365533+0000", 2024-08-17T18:05:45.603 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 91.127131562000002, 2024-08-17T18:05:45.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6147471759999998, 2024-08-17T18:05:45.603 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:45.603 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:05:45.603 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:45.603 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:05:45.603 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:05:45.603 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39695 2024-08-17T18:05:45.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.604 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:45.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:45.604 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.365533+0000", 2024-08-17T18:05:45.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:45.604 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.365533+0000", 2024-08-17T18:05:45.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:45.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.365532+0000", 2024-08-17T18:05:45.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:05:45.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:45.606 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.365600+0000", 2024-08-17T18:05:45.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7230999999999994e-05 2024-08-17T18:05:45.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.606 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:45.606 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.365611+0000", 2024-08-17T18:05:45.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0963e-05 2024-08-17T18:05:45.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.606 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:45.606 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.365616+0000", 2024-08-17T18:05:45.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4659999999999998e-06 2024-08-17T18:05:45.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.365645+0000", 2024-08-17T18:05:45.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8711e-05 2024-08-17T18:05:45.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.365651+0000", 2024-08-17T18:05:45.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9490000000000001e-06 2024-08-17T18:05:45.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521504+0000", 2024-08-17T18:05:45.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15585368199999999 2024-08-17T18:05:45.608 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.608 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.608 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.608 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521512+0000", 2024-08-17T18:05:45.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9759999999999995e-06 2024-08-17T18:05:45.608 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.608 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.608 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.608 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.964836+0000", 2024-08-17T18:05:45.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4433231790000001 2024-08-17T18:05:45.608 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.608 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.608 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:45.608 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.964974+0000", 2024-08-17T18:05:45.609 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000137997 2024-08-17T18:05:45.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-08-17T18:05:45.609 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.965152+0000", 2024-08-17T18:05:45.609 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017841600000000001 2024-08-17T18:05:45.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:45.609 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.967467+0000", 2024-08-17T18:05:45.609 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023148420000000001 2024-08-17T18:05:45.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:45.610 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.972579+0000", 2024-08-17T18:05:45.610 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051124680000000002 2024-08-17T18:05:45.610 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.610 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.610 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:45.610 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.979303+0000", 2024-08-17T18:05:45.610 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0067240920000000001 2024-08-17T18:05:45.610 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.610 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.610 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:05:45.610 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.979344+0000", 2024-08-17T18:05:45.610 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0309000000000003e-05 2024-08-17T18:05:45.610 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.611 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:45.611 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.980280+0000", 2024-08-17T18:05:45.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00093661600000000003 2024-08-17T18:05:45.611 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.611 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:45.611 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.611 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:39702 55.f 55:f182b387:::benchmark_data_smithi136_313476_object39701:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1457)", 2024-08-17T18:05:45.611 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:04:14.371865+0000", 2024-08-17T18:05:45.611 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 91.120800109000001, 2024-08-17T18:05:45.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6018566779999999, 2024-08-17T18:05:45.611 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:45.611 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:05:45.612 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:45.612 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:05:45.612 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:05:45.612 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39702 2024-08-17T18:05:45.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.612 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:45.612 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.612 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:45.612 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.371865+0000", 2024-08-17T18:05:45.612 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.612 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.612 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:45.612 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.371865+0000", 2024-08-17T18:05:45.613 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.613 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.613 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.613 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:45.613 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.371863+0000", 2024-08-17T18:05:45.613 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:05:45.613 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.613 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.613 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:45.613 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.371922+0000", 2024-08-17T18:05:45.613 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8881000000000001e-05 2024-08-17T18:05:45.613 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.613 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.613 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:45.614 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.371930+0000", 2024-08-17T18:05:45.614 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5279999999999998e-06 2024-08-17T18:05:45.614 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.614 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.614 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:45.614 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.371933+0000", 2024-08-17T18:05:45.614 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6200000000000001e-06 2024-08-17T18:05:45.614 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.614 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.614 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.614 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.371970+0000", 2024-08-17T18:05:45.614 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6269000000000001e-05 2024-08-17T18:05:45.614 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.614 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.615 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.615 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.371978+0000", 2024-08-17T18:05:45.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9249999999999995e-06 2024-08-17T18:05:45.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.615 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.615 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521462+0000", 2024-08-17T18:05:45.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.14948477499999999 2024-08-17T18:05:45.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.615 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.615 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521472+0000", 2024-08-17T18:05:45.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0064000000000001e-05 2024-08-17T18:05:45.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.616 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.616 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.947893+0000", 2024-08-17T18:05:45.616 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4264202830000001 2024-08-17T18:05:45.616 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.616 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:45.616 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.947992+0000", 2024-08-17T18:05:45.616 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9685000000000005e-05 2024-08-17T18:05:45.616 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.616 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-08-17T18:05:45.616 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.948102+0000", 2024-08-17T18:05:45.616 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010922200000000001 2024-08-17T18:05:45.617 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:45.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.949236+0000", 2024-08-17T18:05:45.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001134564 2024-08-17T18:05:45.617 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:45.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.950395+0000", 2024-08-17T18:05:45.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001158346 2024-08-17T18:05:45.617 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:45.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.972549+0000", 2024-08-17T18:05:45.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022154396999999999 2024-08-17T18:05:45.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:05:45.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.972624+0000", 2024-08-17T18:05:45.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4575999999999999e-05 2024-08-17T18:05:45.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:45.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.973721+0000", 2024-08-17T18:05:45.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001097677 2024-08-17T18:05:45.619 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.619 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:45.619 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.619 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:39707 55.5 55:a39d6cb7:::benchmark_data_smithi136_313476_object39706:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1457)", 2024-08-17T18:05:45.619 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:04:14.374950+0000", 2024-08-17T18:05:45.619 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 91.117714907999996, 2024-08-17T18:05:45.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.606183025, 2024-08-17T18:05:45.619 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:05:45.620 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:05:45.620 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:05:45.620 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:05:45.620 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:05:45.620 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39707 2024-08-17T18:05:45.620 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.620 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:05:45.620 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.620 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:05:45.620 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.374950+0000", 2024-08-17T18:05:45.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.621 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.621 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.621 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:05:45.621 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.374950+0000", 2024-08-17T18:05:45.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:05:45.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:05:45.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.374949+0000", 2024-08-17T18:05:45.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:05:45.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:05:45.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.374988+0000", 2024-08-17T18:05:45.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9433e-05 2024-08-17T18:05:45.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.623 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:05:45.623 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.374999+0000", 2024-08-17T18:05:45.623 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0883999999999999e-05 2024-08-17T18:05:45.623 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.623 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:05:45.623 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.375004+0000", 2024-08-17T18:05:45.623 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8840000000000002e-06 2024-08-17T18:05:45.623 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.623 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.623 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.375038+0000", 2024-08-17T18:05:45.623 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3470000000000003e-05 2024-08-17T18:05:45.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.624 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.624 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.624 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.375043+0000", 2024-08-17T18:05:45.624 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8760000000000002e-06 2024-08-17T18:05:45.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.624 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.624 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.624 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521535+0000", 2024-08-17T18:05:45.624 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.146491867 2024-08-17T18:05:45.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.624 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.624 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:05:45.625 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521543+0000", 2024-08-17T18:05:45.625 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5020000000000003e-06 2024-08-17T18:05:45.625 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.625 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.625 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:05:45.625 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.966334+0000", 2024-08-17T18:05:45.625 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.444790883 2024-08-17T18:05:45.625 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.625 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.625 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:05:45.625 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.966428+0000", 2024-08-17T18:05:45.625 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4215999999999994e-05 2024-08-17T18:05:45.625 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.625 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.625 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-08-17T18:05:45.626 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.966533+0000", 2024-08-17T18:05:45.626 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105398 2024-08-17T18:05:45.626 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.626 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.626 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:05:45.626 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.967589+0000", 2024-08-17T18:05:45.626 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010561629999999999 2024-08-17T18:05:45.626 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.626 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.626 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:45.626 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.972623+0000", 2024-08-17T18:05:45.626 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050332479999999997 2024-08-17T18:05:45.626 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.627 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:05:45.627 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.980333+0000", 2024-08-17T18:05:45.627 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0077104850000000004 2024-08-17T18:05:45.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.627 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:05:45.627 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.980389+0000", 2024-08-17T18:05:45.627 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5748999999999998e-05 2024-08-17T18:05:45.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:05:45.627 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:05:45.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:05:45.627 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.981133+0000", 2024-08-17T18:05:45.628 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00074385100000000002 2024-08-17T18:05:45.628 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.628 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:45.628 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.628 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:05:45.628 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:05:45.628 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:05:45.654 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:45.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:45.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:45.666 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:45 smithi136 ceph-mon[109861]: pgmap v2627: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 444 op/s; 24628/259047 objects degraded (9.507%); 3964/259047 objects misplaced (1.530%); 11 MiB/s, 173 objects/s recovering 2024-08-17T18:05:45.667 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:45 smithi136 ceph-mon[115078]: pgmap v2627: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 444 op/s; 24628/259047 objects degraded (9.507%); 3964/259047 objects misplaced (1.530%); 11 MiB/s, 173 objects/s recovering 2024-08-17T18:05:45.740 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:05:45.756 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:45.819 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:45.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:45.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:45.920 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:45.944 INFO:teuthology.orchestra.run.smithi136.stdout:moved 0 / 297 (0%) 2024-08-17T18:05:45.944 INFO:teuthology.orchestra.run.smithi136.stdout:avg 42.4286 2024-08-17T18:05:45.944 INFO:teuthology.orchestra.run.smithi136.stdout:stddev 5.72855 -> 5.72855 (expected baseline 6.03053) 2024-08-17T18:05:45.944 INFO:teuthology.orchestra.run.smithi136.stdout:min osd.5 with 35 -> 35 pgs (0.824916 -> 0.824916 * mean) 2024-08-17T18:05:45.944 INFO:teuthology.orchestra.run.smithi136.stdout:max osd.1 with 50 -> 50 pgs (1.17845 -> 1.17845 * mean) 2024-08-17T18:05:45.944 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:05:45.944 INFO:teuthology.orchestra.run.smithi136.stdout:oload 1000 2024-08-17T18:05:45.944 INFO:teuthology.orchestra.run.smithi136.stdout:max_change 3 2024-08-17T18:05:45.945 INFO:teuthology.orchestra.run.smithi136.stdout:max_change_osds 4 2024-08-17T18:05:45.945 INFO:teuthology.orchestra.run.smithi136.stdout:average_utilization 0.0323 2024-08-17T18:05:45.945 INFO:teuthology.orchestra.run.smithi136.stdout:overload_utilization 0.3225 2024-08-17T18:05:45.945 INFO:teuthology.orchestra.run.smithi136.stderr:no change 2024-08-17T18:05:45.953 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:45.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:45.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:46.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:45 smithi165 ceph-mon[105517]: pgmap v2627: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 444 op/s; 24628/259047 objects degraded (9.507%); 3964/259047 objects misplaced (1.530%); 11 MiB/s, 173 objects/s recovering 2024-08-17T18:05:46.055 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:46.083 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:46.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:46.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:46.186 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:46.218 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:46.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:46.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:46.320 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:46.345 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:46.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:46.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:46.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:46.469 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:46.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:46.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:46.571 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:46.594 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:46.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:46.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:46.696 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:46.719 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:46.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:46.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:46.821 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:46.845 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:46.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:46.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:46.948 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:46.977 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:46.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:46.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:47.079 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:47.101 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:47.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:47.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:47.203 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:47.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:47.228 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:47.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:47.329 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:47.356 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:47.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:47.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:47.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:47.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:47.487 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:47.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:47.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:47.618 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:47.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:47.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:47.719 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:47.746 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:47.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:47.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:47.849 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:47.876 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:47.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:47.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:47.927 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:47 smithi136 ceph-mon[109861]: from='client.45743 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:05:47.927 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:47 smithi136 ceph-mon[109861]: pgmap v2628: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 381 op/s; 24398/258702 objects degraded (9.431%); 3964/258702 objects misplaced (1.532%); 9.7 MiB/s, 155 objects/s recovering 2024-08-17T18:05:47.927 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:47 smithi136 ceph-mon[115078]: from='client.45743 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:05:47.927 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:47 smithi136 ceph-mon[115078]: pgmap v2628: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 381 op/s; 24398/258702 objects degraded (9.431%); 3964/258702 objects misplaced (1.532%); 9.7 MiB/s, 155 objects/s recovering 2024-08-17T18:05:47.928 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:05:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:05:47] "GET /metrics HTTP/1.1" 200 36642 "" "Prometheus/2.51.0" 2024-08-17T18:05:47.979 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:48.004 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:48.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:48.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:47 smithi165 ceph-mon[105517]: from='client.45743 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:05:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:47 smithi165 ceph-mon[105517]: pgmap v2628: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 381 op/s; 24398/258702 objects degraded (9.431%); 3964/258702 objects misplaced (1.532%); 9.7 MiB/s, 155 objects/s recovering 2024-08-17T18:05:48.106 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:48.133 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:48.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:48.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:48.163 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-17T18:05:48.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:48.285 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:48.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:48.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:48.343 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-08-17T18:05:48.343 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight-by-utilization 110 1.0 2024-08-17T18:05:48.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:48.414 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:48.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:48.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:48.482 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:05:48.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:48.539 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:48.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:48.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:48.641 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:48.667 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:48.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:48.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:48.769 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:48.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:48.795 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:48.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:48.864 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:05:48.896 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:48.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:48.927 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:48.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:49.028 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:49.054 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:49.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:49.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:49.156 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:49.183 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:49.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:49.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:49.285 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:49.308 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:49.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:49.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:49.410 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:49.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:49.448 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:49.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:49.549 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:49.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:49.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:49.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:49.675 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:49.702 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:49.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:49.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:49.749 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:49 smithi165 ceph-mon[105517]: pgmap v2629: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 399 op/s; 24012/252951 objects degraded (9.493%); 3294/252951 objects misplaced (1.302%); 9.9 MiB/s, 158 objects/s recovering 2024-08-17T18:05:49.749 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:49 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 24012/252951 objects degraded (9.493%), 11 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:05:49.805 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:49.827 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:49.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:49.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:49.929 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:49.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:49 smithi136 ceph-mon[109861]: pgmap v2629: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 399 op/s; 24012/252951 objects degraded (9.493%); 3294/252951 objects misplaced (1.302%); 9.9 MiB/s, 158 objects/s recovering 2024-08-17T18:05:49.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:49 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 24012/252951 objects degraded (9.493%), 11 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:05:49.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:49 smithi136 ceph-mon[115078]: pgmap v2629: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 399 op/s; 24012/252951 objects degraded (9.493%); 3294/252951 objects misplaced (1.302%); 9.9 MiB/s, 158 objects/s recovering 2024-08-17T18:05:49.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:49 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 24012/252951 objects degraded (9.493%), 11 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:05:49.951 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:49.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:49.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:50.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:50.083 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:50.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:50.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:50.185 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:50.208 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:50.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:50.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:50.310 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:50.334 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:50.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:50.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:50.358 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:05:50.358 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:05:50.358 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-08-17T18:05:50.358 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:05:50.436 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:50.460 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:50.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:50.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:50.562 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:50.589 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:50.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:50.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:50.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:50.715 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:50.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:50.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:50.817 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:50.843 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:50.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:50.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:50.945 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:50.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:50.973 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:50.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:51.001 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-17T18:05:51.074 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:51.097 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:51.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:51.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:51.199 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:51.229 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:51.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:51.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:51.326 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:05:51.332 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:51.361 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:51.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:51.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:51.362 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:05:51.463 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:51.486 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:51.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:51.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:51.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:51.612 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:51.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:51.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:51.714 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:51.741 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:51.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:51.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:51 smithi165 ceph-mon[105517]: pgmap v2630: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 387 op/s; 23763/252582 objects degraded (9.408%); 3294/252582 objects misplaced (1.304%); 10 MiB/s, 160 objects/s recovering 2024-08-17T18:05:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:05:51.811 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:51 smithi136 ceph-mon[109861]: pgmap v2630: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 387 op/s; 23763/252582 objects degraded (9.408%); 3294/252582 objects misplaced (1.304%); 10 MiB/s, 160 objects/s recovering 2024-08-17T18:05:51.811 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:05:51.812 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:51 smithi136 ceph-mon[115078]: pgmap v2630: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 387 op/s; 23763/252582 objects degraded (9.408%); 3294/252582 objects misplaced (1.304%); 10 MiB/s, 160 objects/s recovering 2024-08-17T18:05:51.812 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:05:51.843 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:51.877 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:51.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:51.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:51.979 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:52.003 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:52.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:52.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:52.105 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:52.131 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:52.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:52.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:52.233 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:52.256 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:52.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:52.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:52.359 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:52.383 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:52.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:52.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:52.484 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:52.507 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:52.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:52.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:52.610 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:52.643 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:52.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:52.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:52.651 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:05:52.744 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:52.774 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:52.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:52.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:52.876 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:52.888 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:52 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2121343235' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:05:52.889 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:52 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:05:52.889 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:52 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2121343235' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:05:52.889 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:52 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:05:52.904 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:52.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:52.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:52.918 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:05:52.919 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:05:52.938 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:05:52.939 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:05:52.960 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:05:52.960 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:05:52.983 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:05:52.983 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:05:53.004 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:05:53.004 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:05:53.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:53.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:52 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2121343235' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:05:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:52 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:05:53.025 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:05:53.025 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:05:53.030 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:53.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:53.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:53.044 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:05:53.044 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:05:53.064 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:05:53.064 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:05:53.131 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:53.161 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:53.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:53.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:53.263 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:53.290 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:53.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:53.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:53.392 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:53.419 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:53.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:53.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:53.521 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:53.552 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:53.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:53.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:53.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:53 smithi165 ceph-mon[105517]: pgmap v2631: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 467 op/s; 23415/247764 objects degraded (9.451%); 3294/247764 objects misplaced (1.329%); 10 MiB/s, 161 objects/s recovering 2024-08-17T18:05:53.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:53 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:05:53.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:53 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:05:53.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:53 smithi165 ceph-mon[105517]: osdmap e1480: 8 total, 8 up, 7 in 2024-08-17T18:05:53.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:53 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3983176019' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:05:53.654 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:53.657 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:05:53.657 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:05:53.676 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:53.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:53.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:53.777 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:53.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:53.806 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:53.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:53.907 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:53.930 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:53.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:53.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:53.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:53 smithi136 ceph-mon[109861]: pgmap v2631: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 467 op/s; 23415/247764 objects degraded (9.451%); 3294/247764 objects misplaced (1.329%); 10 MiB/s, 161 objects/s recovering 2024-08-17T18:05:53.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:53 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:05:53.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:53 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:05:53.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:53 smithi136 ceph-mon[109861]: osdmap e1480: 8 total, 8 up, 7 in 2024-08-17T18:05:53.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:53 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3983176019' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:05:53.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:53 smithi136 ceph-mon[115078]: pgmap v2631: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 467 op/s; 23415/247764 objects degraded (9.451%); 3294/247764 objects misplaced (1.329%); 10 MiB/s, 161 objects/s recovering 2024-08-17T18:05:53.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:53 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:05:53.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:53 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:05:53.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:53 smithi136 ceph-mon[115078]: osdmap e1480: 8 total, 8 up, 7 in 2024-08-17T18:05:53.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:53 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3983176019' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:05:54.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:54.057 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:54.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:54.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:54.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:54.182 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:54.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:54.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:54.283 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:54.305 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:54.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:54.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:54.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-17T18:05:54.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:54.429 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:54.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:54.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:54.531 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:54.555 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:54.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:54.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:54.619 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:05:54.657 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:54.689 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:54.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:54.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:54.790 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:54.821 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:54.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:54.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:54.917 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:54 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 23415/247764 objects degraded (9.451%), 11 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:05:54.923 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:54.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:54 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 23415/247764 objects degraded (9.451%), 11 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:05:54.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:54 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 23415/247764 objects degraded (9.451%), 11 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:05:54.952 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:54.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:54.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:55.054 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:55.077 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:55.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:55.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:55.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:55.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:55.204 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:55.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:55.305 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:55.333 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:55.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:55.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:55.435 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:55.459 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:55.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:55.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:55.560 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:55.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:55.589 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:55.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:55.690 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:55.715 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:55.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:55.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:55 smithi165 ceph-mon[105517]: pgmap v2633: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 386 op/s; 23415/247458 objects degraded (9.462%); 2486/247458 objects misplaced (1.005%); 11 MiB/s, 171 objects/s recovering 2024-08-17T18:05:55.787 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:55 smithi136 ceph-mon[109861]: pgmap v2633: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 386 op/s; 23415/247458 objects degraded (9.462%); 2486/247458 objects misplaced (1.005%); 11 MiB/s, 171 objects/s recovering 2024-08-17T18:05:55.787 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:55 smithi136 ceph-mon[115078]: pgmap v2633: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 386 op/s; 23415/247458 objects degraded (9.462%); 2486/247458 objects misplaced (1.005%); 11 MiB/s, 171 objects/s recovering 2024-08-17T18:05:55.817 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:55.849 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:55.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:55.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:55.950 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:55.982 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:55.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:55.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:56.084 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:56.113 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:56.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:56.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:56.215 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:56.247 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:56.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:56.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:56.349 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:56.375 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:56.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:56.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:56.477 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:56.500 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:56.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:56.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:56.602 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:56.626 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:56.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:56.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:56.660 INFO:teuthology.orchestra.run.smithi136.stdout:moved 31 / 297 (10.4377%) 2024-08-17T18:05:56.661 INFO:teuthology.orchestra.run.smithi136.stdout:avg 42.4286 2024-08-17T18:05:56.661 INFO:teuthology.orchestra.run.smithi136.stdout:stddev 5.72855 -> 5.0669 (expected baseline 6.03053) 2024-08-17T18:05:56.661 INFO:teuthology.orchestra.run.smithi136.stdout:min osd.5 with 35 -> 46 pgs (0.824916 -> 1.08418 * mean) 2024-08-17T18:05:56.661 INFO:teuthology.orchestra.run.smithi136.stdout:max osd.1 with 50 -> 53 pgs (1.17845 -> 1.24916 * mean) 2024-08-17T18:05:56.661 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:05:56.661 INFO:teuthology.orchestra.run.smithi136.stdout:oload 110 2024-08-17T18:05:56.661 INFO:teuthology.orchestra.run.smithi136.stdout:max_change 1 2024-08-17T18:05:56.661 INFO:teuthology.orchestra.run.smithi136.stdout:max_change_osds 4 2024-08-17T18:05:56.661 INFO:teuthology.orchestra.run.smithi136.stdout:average_utilization 0.0315 2024-08-17T18:05:56.661 INFO:teuthology.orchestra.run.smithi136.stdout:overload_utilization 0.0346 2024-08-17T18:05:56.661 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3 weight 1.0000 -> 0.6855 2024-08-17T18:05:56.661 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2 weight 1.0000 -> 0.7409 2024-08-17T18:05:56.661 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6 weight 1.0000 -> 0.7423 2024-08-17T18:05:56.681 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:05:56.728 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:56.750 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:56.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:56.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:56.852 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:56.877 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:56.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:56.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:56.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:56 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'2':'48556','3':'44926','6':'48646'}"} : dispatch 2024-08-17T18:05:56.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:56 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'2':'48556','3':'44926','6':'48646'}"} : dispatch 2024-08-17T18:05:56.978 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:56.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:05:57.001 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:57.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:57.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:57.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:56 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'2':'48556','3':'44926','6':'48646'}"} : dispatch 2024-08-17T18:05:57.104 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:57.128 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:57.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:57.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:57.230 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:57.251 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:57.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:57.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:57.258 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:05:57.258 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:05:57.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:57.375 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:57.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:57.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:57.385 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-08-17T18:05:57.385 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-08-17T18:05:57.477 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:57.489 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:05:57.501 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:57.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:57.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:57.532 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:05:57.603 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:05:57.619 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:05:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:05:57] "GET /metrics HTTP/1.1" 200 36649 "" "Prometheus/2.51.0" 2024-08-17T18:05:57.624 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:57.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:57.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:05:57.725 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:57.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:57.748 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:57.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:57.849 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:57.872 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:57.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:57.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:57.884 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:57 smithi136 ceph-mon[109861]: pgmap v2634: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 386 op/s; 23173/247089 objects degraded (9.378%); 2486/247089 objects misplaced (1.006%); 11 MiB/s, 171 objects/s recovering 2024-08-17T18:05:57.884 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:57 smithi136 ceph-mon[109861]: from='client.45761 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:05:57.885 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:57 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'48556','3':'44926','6':'48646'}"}]': finished 2024-08-17T18:05:57.885 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:57 smithi136 ceph-mon[109861]: osdmap e1481: 8 total, 8 up, 7 in 2024-08-17T18:05:57.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:57 smithi136 ceph-mon[115078]: pgmap v2634: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 386 op/s; 23173/247089 objects degraded (9.378%); 2486/247089 objects misplaced (1.006%); 11 MiB/s, 171 objects/s recovering 2024-08-17T18:05:57.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:57 smithi136 ceph-mon[115078]: from='client.45761 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:05:57.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:57 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'48556','3':'44926','6':'48646'}"}]': finished 2024-08-17T18:05:57.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:57 smithi136 ceph-mon[115078]: osdmap e1481: 8 total, 8 up, 7 in 2024-08-17T18:05:57.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:57 smithi165 ceph-mon[105517]: pgmap v2634: 99 pgs: 8 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 2 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 386 op/s; 23173/247089 objects degraded (9.378%); 2486/247089 objects misplaced (1.006%); 11 MiB/s, 171 objects/s recovering 2024-08-17T18:05:57.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:57 smithi165 ceph-mon[105517]: from='client.45761 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:05:57.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:57 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'48556','3':'44926','6':'48646'}"}]': finished 2024-08-17T18:05:57.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:57 smithi165 ceph-mon[105517]: osdmap e1481: 8 total, 8 up, 7 in 2024-08-17T18:05:57.974 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:57.975 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-17T18:05:57.994 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:57.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:57.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:58.096 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:58.118 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:58.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:58.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:58.220 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:58.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:58.242 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:58.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:58.343 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:58.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:58.365 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:58.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:58.466 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:58.487 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:58.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:58.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:58.589 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:58.612 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:58.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:58.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:58.714 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:58.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:58.753 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:58.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:58.854 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:58.879 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:58.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:58.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:58 smithi136 ceph-mon[109861]: osdmap e1482: 8 total, 8 up, 7 in 2024-08-17T18:05:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:58 smithi136 ceph-mon[109861]: osdmap e1483: 8 total, 8 up, 7 in 2024-08-17T18:05:58.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:58 smithi136 ceph-mon[115078]: osdmap e1482: 8 total, 8 up, 7 in 2024-08-17T18:05:58.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:58 smithi136 ceph-mon[115078]: osdmap e1483: 8 total, 8 up, 7 in 2024-08-17T18:05:58.981 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:05:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:58 smithi165 ceph-mon[105517]: osdmap e1482: 8 total, 8 up, 7 in 2024-08-17T18:05:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:58 smithi165 ceph-mon[105517]: osdmap e1483: 8 total, 8 up, 7 in 2024-08-17T18:05:59.014 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:59.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:59.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:05:59.116 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:59.145 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:59.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:59.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:59.246 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:05:59.272 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:59.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:59.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:05:59.373 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:05:59.396 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:59.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:59.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:05:59.498 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:05:59.520 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:59.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:59.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:05:59.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:05:59.645 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:59.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:59.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:05:59.747 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:05:59.767 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:05:59.777 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:05:59.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:59.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:05:59.879 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:05:59.902 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:05:59.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:05:59.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:05:59.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:59 smithi136 ceph-mon[109861]: pgmap v2637: 99 pgs: 2 remapped+peering, 3 unknown, 13 peering, 8 active+recovery_wait+degraded, 2 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 68 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 334 op/s; 21046/241737 objects degraded (8.706%); 13 MiB/s, 209 objects/s recovering 2024-08-17T18:05:59.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:59 smithi136 ceph-mon[109861]: 55.c starting backfill to osd.5 from (0'0,0'0] MAX to 1480'6465 2024-08-17T18:05:59.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:59 smithi136 ceph-mon[109861]: 55.a starting backfill to osd.6 from (0'0,0'0] MAX to 1480'6365 2024-08-17T18:05:59.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:59 smithi136 ceph-mon[109861]: 55.e starting backfill to osd.4 from (0'0,0'0] MAX to 1480'6330 2024-08-17T18:05:59.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:59 smithi136 ceph-mon[109861]: Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-08-17T18:05:59.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:05:59 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 21046/241737 objects degraded (8.706%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:05:59.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:59 smithi136 ceph-mon[115078]: pgmap v2637: 99 pgs: 2 remapped+peering, 3 unknown, 13 peering, 8 active+recovery_wait+degraded, 2 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 68 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 334 op/s; 21046/241737 objects degraded (8.706%); 13 MiB/s, 209 objects/s recovering 2024-08-17T18:05:59.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:59 smithi136 ceph-mon[115078]: 55.c starting backfill to osd.5 from (0'0,0'0] MAX to 1480'6465 2024-08-17T18:05:59.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:59 smithi136 ceph-mon[115078]: 55.a starting backfill to osd.6 from (0'0,0'0] MAX to 1480'6365 2024-08-17T18:05:59.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:59 smithi136 ceph-mon[115078]: 55.e starting backfill to osd.4 from (0'0,0'0] MAX to 1480'6330 2024-08-17T18:05:59.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:59 smithi136 ceph-mon[115078]: Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-08-17T18:05:59.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:05:59 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 21046/241737 objects degraded (8.706%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:00.004 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:59 smithi165 ceph-mon[105517]: pgmap v2637: 99 pgs: 2 remapped+peering, 3 unknown, 13 peering, 8 active+recovery_wait+degraded, 2 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 68 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 334 op/s; 21046/241737 objects degraded (8.706%); 13 MiB/s, 209 objects/s recovering 2024-08-17T18:06:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:59 smithi165 ceph-mon[105517]: 55.c starting backfill to osd.5 from (0'0,0'0] MAX to 1480'6465 2024-08-17T18:06:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:59 smithi165 ceph-mon[105517]: 55.a starting backfill to osd.6 from (0'0,0'0] MAX to 1480'6365 2024-08-17T18:06:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:59 smithi165 ceph-mon[105517]: 55.e starting backfill to osd.4 from (0'0,0'0] MAX to 1480'6330 2024-08-17T18:06:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:59 smithi165 ceph-mon[105517]: Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-08-17T18:06:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:05:59 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 21046/241737 objects degraded (8.706%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:00.027 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:00.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:00.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:00.129 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:00.152 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:00.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:00.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:00.254 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:00.279 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:00.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:00.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:00.381 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:00.404 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:00.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:00.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:00.505 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:00.530 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:00.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:00.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:00.632 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:00.656 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:00.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:00.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:00.675 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:06:00.758 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:00.781 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:00.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:00.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:00.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:00.908 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:00.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:00.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:01.009 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:01.034 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:01.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:01.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:01.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:01.161 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:01.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:01.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:01.263 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:01.288 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:01.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:01.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:01.390 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:01.413 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:01.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:01.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:01.515 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:01.539 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:01.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:01.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:01.641 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:01.665 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:01.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:01.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:01.767 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:01.792 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:01.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:01.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:01.894 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:01.918 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:01.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:01.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:01.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:01 smithi136 ceph-mon[109861]: pgmap v2639: 99 pgs: 2 remapped+peering, 3 unknown, 13 peering, 8 active+recovery_wait+degraded, 2 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 68 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 317 op/s; 21046/241737 objects degraded (8.706%); 8.9 MiB/s, 141 objects/s recovering 2024-08-17T18:06:01.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:01 smithi136 ceph-mon[115078]: pgmap v2639: 99 pgs: 2 remapped+peering, 3 unknown, 13 peering, 8 active+recovery_wait+degraded, 2 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 68 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 317 op/s; 21046/241737 objects degraded (8.706%); 8.9 MiB/s, 141 objects/s recovering 2024-08-17T18:06:01.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:01 smithi165 ceph-mon[105517]: pgmap v2639: 99 pgs: 2 remapped+peering, 3 unknown, 13 peering, 8 active+recovery_wait+degraded, 2 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 68 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 317 op/s; 21046/241737 objects degraded (8.706%); 8.9 MiB/s, 141 objects/s recovering 2024-08-17T18:06:02.021 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:02.046 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:02.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:02.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:02.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:02.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:02.173 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:02.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:02.274 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:02.297 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:02.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:02.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:02.399 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:02.422 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:02.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:02.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:02.524 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:02.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:02.550 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:02.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:02.651 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:02.676 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:02.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:02.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:02.778 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:02.802 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:02.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:02.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:02.904 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:02.927 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:02.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:02.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:02.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:02 smithi136 ceph-mon[109861]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2024-08-17T18:06:02.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:02 smithi136 ceph-mon[115078]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2024-08-17T18:06:03.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:02 smithi165 ceph-mon[105517]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2024-08-17T18:06:03.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:03.052 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:03.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:03.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:03.154 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:03.179 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:03.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:03.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:03.281 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:03.301 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:06:03.301 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:06:03.301 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:06:03.301 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:06:03.302 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:06:03.302 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:06:03.304 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:03.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:03.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:03.306 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:06:03.306 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:06:03.327 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:06:03.327 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:06:03.357 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:06:03.357 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:06:03.388 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:06:03.388 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:06:03.406 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:03.412 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:06:03.412 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:06:03.433 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:03.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:03.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:03.534 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:03.559 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:03.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:03.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:03.661 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:03.685 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:03.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:03.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:03.760 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:06:03.787 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:03.812 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:03.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:03.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:03.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:03.936 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:03.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:03.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:03 smithi165 ceph-mon[105517]: pgmap v2640: 99 pgs: 1 active+remapped+backfilling, 2 remapped+peering, 2 active+remapped+backfill_wait, 7 active+recovery_wait+degraded, 2 active+recovery_wait, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 78 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 321 op/s; 31005/241293 objects degraded (12.850%); 13979/241293 objects misplaced (5.793%); 34 MiB/s, 543 objects/s recovering 2024-08-17T18:06:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:03 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2705164804' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:06:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:03 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4124540356' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:06:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:03 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:06:04.038 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:04.061 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:04.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:04.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:04.095 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:03 smithi136 ceph-mon[109861]: pgmap v2640: 99 pgs: 1 active+remapped+backfilling, 2 remapped+peering, 2 active+remapped+backfill_wait, 7 active+recovery_wait+degraded, 2 active+recovery_wait, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 78 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 321 op/s; 31005/241293 objects degraded (12.850%); 13979/241293 objects misplaced (5.793%); 34 MiB/s, 543 objects/s recovering 2024-08-17T18:06:04.095 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:03 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2705164804' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:06:04.095 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:03 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4124540356' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:06:04.095 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:03 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:06:04.095 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:03 smithi136 ceph-mon[115078]: pgmap v2640: 99 pgs: 1 active+remapped+backfilling, 2 remapped+peering, 2 active+remapped+backfill_wait, 7 active+recovery_wait+degraded, 2 active+recovery_wait, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 78 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 321 op/s; 31005/241293 objects degraded (12.850%); 13979/241293 objects misplaced (5.793%); 34 MiB/s, 543 objects/s recovering 2024-08-17T18:06:04.096 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:03 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2705164804' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:06:04.096 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:03 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4124540356' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:06:04.096 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:03 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:06:04.162 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:04.186 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:04.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:04.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:04.288 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:04.312 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:04.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:04.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:04.414 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:04.438 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:04.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:04.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:04.541 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:04.564 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:04.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:04.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:04.666 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:04.691 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:04.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:04.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:04.794 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:04.823 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:04.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:04.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:04.925 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:04.955 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:04.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:04.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:04 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:06:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:04 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:06:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:04 smithi165 ceph-mon[105517]: osdmap e1484: 8 total, 8 up, 7 in 2024-08-17T18:06:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:04 smithi165 ceph-mon[105517]: pgmap v2642: 99 pgs: 2 activating+undersized+degraded+remapped, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 2 active+recovery_wait, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 78 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 75 op/s; 46518/240630 objects degraded (19.332%); 13979/240630 objects misplaced (5.809%); 31 MiB/s, 491 objects/s recovering 2024-08-17T18:06:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:04 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 31005/241293 objects degraded (12.850%), 13 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:05.057 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:05.084 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:05.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:05.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:05.126 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:04 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:06:05.126 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:04 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:06:05.126 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:04 smithi136 ceph-mon[109861]: osdmap e1484: 8 total, 8 up, 7 in 2024-08-17T18:06:05.126 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:04 smithi136 ceph-mon[109861]: pgmap v2642: 99 pgs: 2 activating+undersized+degraded+remapped, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 2 active+recovery_wait, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 78 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 75 op/s; 46518/240630 objects degraded (19.332%); 13979/240630 objects misplaced (5.809%); 31 MiB/s, 491 objects/s recovering 2024-08-17T18:06:05.126 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:04 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 31005/241293 objects degraded (12.850%), 13 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:05.126 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:04 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:06:05.126 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:04 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:06:05.126 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:04 smithi136 ceph-mon[115078]: osdmap e1484: 8 total, 8 up, 7 in 2024-08-17T18:06:05.127 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:04 smithi136 ceph-mon[115078]: pgmap v2642: 99 pgs: 2 activating+undersized+degraded+remapped, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 2 active+recovery_wait, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 78 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 75 op/s; 46518/240630 objects degraded (19.332%); 13979/240630 objects misplaced (5.809%); 31 MiB/s, 491 objects/s recovering 2024-08-17T18:06:05.127 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:04 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 31005/241293 objects degraded (12.850%), 13 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:05.185 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:05.214 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:05.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:05.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:05.316 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:05.340 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:05.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:05.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:05.442 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:05.468 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:05.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:05.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:05.521 INFO:teuthology.orchestra.run.smithi136.stdout:moved 0 / 297 (0%) 2024-08-17T18:06:05.521 INFO:teuthology.orchestra.run.smithi136.stdout:avg 42.4286 2024-08-17T18:06:05.521 INFO:teuthology.orchestra.run.smithi136.stdout:stddev 5.0669 -> 5.0669 (expected baseline 6.03053) 2024-08-17T18:06:05.521 INFO:teuthology.orchestra.run.smithi136.stdout:min osd.6 with 37 -> 37 pgs (0.872054 -> 0.872054 * mean) 2024-08-17T18:06:05.521 INFO:teuthology.orchestra.run.smithi136.stdout:max osd.1 with 53 -> 53 pgs (1.24916 -> 1.24916 * mean) 2024-08-17T18:06:05.522 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:06:05.522 INFO:teuthology.orchestra.run.smithi136.stdout:oload 1000 2024-08-17T18:06:05.522 INFO:teuthology.orchestra.run.smithi136.stdout:max_change 3 2024-08-17T18:06:05.522 INFO:teuthology.orchestra.run.smithi136.stdout:max_change_osds 4 2024-08-17T18:06:05.522 INFO:teuthology.orchestra.run.smithi136.stdout:average_utilization 0.0312 2024-08-17T18:06:05.522 INFO:teuthology.orchestra.run.smithi136.stdout:overload_utilization 0.3125 2024-08-17T18:06:05.522 INFO:teuthology.orchestra.run.smithi136.stderr:no change 2024-08-17T18:06:05.571 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:05.601 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:05.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:05.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:05.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:05.732 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:05.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:05.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:05.834 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:05.861 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:05.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:05.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:05 smithi136 ceph-mon[109861]: from='client.45779 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:06:05.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:05 smithi136 ceph-mon[115078]: from='client.45779 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:06:05.963 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:05.989 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:05.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:05.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:05 smithi165 ceph-mon[105517]: from='client.45779 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:06:06.091 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:06.116 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:06.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:06.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:06.218 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:06.246 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:06.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:06.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:06.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:06.380 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:06.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:06.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:06.481 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:06.509 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:06.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:06.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:06.611 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:06.646 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:06.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:06.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:06.748 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:06.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:06.783 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:06.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:06.884 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:06.919 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:06.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:06.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:07.020 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:07.053 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:07.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:07.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:07.158 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:07.192 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:07.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:07.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:07.294 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:07.318 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:07.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:07.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:07.419 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:07.445 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:07.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:07.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:06:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:06:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:07 smithi165 ceph-mon[105517]: pgmap v2643: 99 pgs: 2 activating+undersized+degraded+remapped, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 2 active+recovery_wait, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 78 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 46 op/s; 46518/240630 objects degraded (19.332%); 13979/240630 objects misplaced (5.809%); 21 MiB/s, 342 objects/s recovering 2024-08-17T18:06:07.547 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:07.584 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:07.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:07.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:07.676 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:06:07.676 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:06:07.676 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:07 smithi136 ceph-mon[109861]: pgmap v2643: 99 pgs: 2 activating+undersized+degraded+remapped, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 2 active+recovery_wait, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 78 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 46 op/s; 46518/240630 objects degraded (19.332%); 13979/240630 objects misplaced (5.809%); 21 MiB/s, 342 objects/s recovering 2024-08-17T18:06:07.676 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:06:07.677 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:06:07.677 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:07 smithi136 ceph-mon[115078]: pgmap v2643: 99 pgs: 2 activating+undersized+degraded+remapped, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 2 active+recovery_wait, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 78 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 46 op/s; 46518/240630 objects degraded (19.332%); 13979/240630 objects misplaced (5.809%); 21 MiB/s, 342 objects/s recovering 2024-08-17T18:06:07.677 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:06:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:06:07] "GET /metrics HTTP/1.1" 200 36695 "" "Prometheus/2.51.0" 2024-08-17T18:06:07.685 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:07.714 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:07.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:07.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:07.816 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:07.845 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:07.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:07.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:07.947 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:07.971 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:07.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:07.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:08.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:08.101 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:08.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:08.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:08.202 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:08.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:08.230 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:08.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:08.331 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:08.367 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:08.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:08.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:08.469 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:08.491 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:08.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:08.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:08.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:08.624 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:08.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:08.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:08.726 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:08.728 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-08-17T18:06:08.728 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-08-17T18:06:08.790 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:08.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:08.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:08.892 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:08.919 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:08.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:08.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:09.021 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:09.046 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:09.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:09.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:09.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:09.172 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:09.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:09.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:09.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:09.282 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:06:09.313 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:09.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:09.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:09.320 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:06:09.320 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T18:06:09.321 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-17T18:06:09.321 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:06:09.321 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:09 smithi136 ceph-mon[109861]: osdmap e1485: 8 total, 8 up, 7 in 2024-08-17T18:06:09.321 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:09 smithi136 ceph-mon[109861]: pgmap v2645: 99 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 81 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 138 op/s; 44491/238401 objects degraded (18.662%); 11463/238401 objects misplaced (4.808%); 55 MiB/s, 876 objects/s recovering 2024-08-17T18:06:09.321 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:09 smithi136 ceph-mon[115078]: osdmap e1485: 8 total, 8 up, 7 in 2024-08-17T18:06:09.321 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:09 smithi136 ceph-mon[115078]: pgmap v2645: 99 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 81 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 138 op/s; 44491/238401 objects degraded (18.662%); 11463/238401 objects misplaced (4.808%); 55 MiB/s, 876 objects/s recovering 2024-08-17T18:06:09.336 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:06:09.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:09.447 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:09.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:09.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:09.503 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:09 smithi165 ceph-mon[105517]: osdmap e1485: 8 total, 8 up, 7 in 2024-08-17T18:06:09.503 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:09 smithi165 ceph-mon[105517]: pgmap v2645: 99 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 81 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 138 op/s; 44491/238401 objects degraded (18.662%); 11463/238401 objects misplaced (4.808%); 55 MiB/s, 876 objects/s recovering 2024-08-17T18:06:09.549 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:09.577 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:09.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:09.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:09.679 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:09.706 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:09.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:09.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:09.808 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:09.836 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:09.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:09.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:09.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:09.968 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:09.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:09.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:10.070 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:10.100 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:10.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:10.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:10.131 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:06:10.202 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:10.230 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:10.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:10.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:10.332 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:10.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:10.356 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:10.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:10.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:10 smithi136 ceph-mon[109861]: osdmap e1486: 8 total, 8 up, 7 in 2024-08-17T18:06:10.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:10 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 44491/238401 objects degraded (18.662%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:10.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:10 smithi136 ceph-mon[115078]: osdmap e1486: 8 total, 8 up, 7 in 2024-08-17T18:06:10.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:10 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 44491/238401 objects degraded (18.662%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:10.457 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:10.488 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:10.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:10.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:10.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:10 smithi165 ceph-mon[105517]: osdmap e1486: 8 total, 8 up, 7 in 2024-08-17T18:06:10.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:10 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 44491/238401 objects degraded (18.662%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:10.589 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:10.619 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:10.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:10.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:10.722 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:10.753 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:10.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:10.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:10.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:10.886 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:10.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:10.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:10.988 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:11.012 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:11.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:11.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:11.114 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:11.136 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:11.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:11.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:11.238 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:11.267 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:11.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:11.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:11.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:11.397 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:11.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:11.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:11.499 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:11 smithi165 ceph-mon[105517]: pgmap v2647: 99 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 81 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 114 op/s; 44491/238401 objects degraded (18.662%); 11463/238401 objects misplaced (4.808%); 41 MiB/s, 658 objects/s recovering 2024-08-17T18:06:11.523 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:11.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:11.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:11.581 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:11 smithi136 ceph-mon[109861]: pgmap v2647: 99 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 81 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 114 op/s; 44491/238401 objects degraded (18.662%); 11463/238401 objects misplaced (4.808%); 41 MiB/s, 658 objects/s recovering 2024-08-17T18:06:11.581 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:11 smithi136 ceph-mon[115078]: pgmap v2647: 99 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 10 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 81 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 114 op/s; 44491/238401 objects degraded (18.662%); 11463/238401 objects misplaced (4.808%); 41 MiB/s, 658 objects/s recovering 2024-08-17T18:06:11.625 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:11.656 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:11.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:11.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:11.758 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:11.792 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:11.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:11.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:11.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:11.919 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:11.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:11.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:12.021 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:12.044 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:12.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:12.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:12.146 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:12.171 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:12.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:12.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:12.273 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:12.297 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:12.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:12.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:12.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:12.432 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:12.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:12.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:12.535 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:12.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:12.560 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:12.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:12.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:12.693 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:12.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:12.795 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:12.823 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:12.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:12.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:12.925 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:12.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:12.948 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:12.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:13.049 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:13.071 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:13.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:13.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:13.173 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:13.201 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:13.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:13.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:13.303 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:13.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:13.336 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:13.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:13.437 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:13.472 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:13.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:13.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:13.574 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:13.599 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:13.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:13.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:13.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:13 smithi136 ceph-mon[109861]: pgmap v2648: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 82 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 238 op/s; 41044/234900 objects degraded (17.473%); 12983/234900 objects misplaced (5.527%); 47 MiB/s, 748 objects/s recovering 2024-08-17T18:06:13.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:13 smithi136 ceph-mon[115078]: pgmap v2648: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 82 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 238 op/s; 41044/234900 objects degraded (17.473%); 12983/234900 objects misplaced (5.527%); 47 MiB/s, 748 objects/s recovering 2024-08-17T18:06:13.702 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:13.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:13.736 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:13.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:13 smithi165 ceph-mon[105517]: pgmap v2648: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 82 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 238 op/s; 41044/234900 objects degraded (17.473%); 12983/234900 objects misplaced (5.527%); 47 MiB/s, 748 objects/s recovering 2024-08-17T18:06:13.837 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:13.868 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:13.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:13.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:13.969 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:14.001 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:14.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:14.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:14.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:14.140 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:14.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:14.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:14.242 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:14.271 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:14.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:14.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:14.373 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:14.402 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:14.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:14.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:14.499 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-17T18:06:14.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:14.779 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:14.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:14.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:14.881 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:14.912 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:14.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:14.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:15.014 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:15.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:15.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:15.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:15.146 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:15.174 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:15.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:15.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:15.252 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:06:15.276 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:15.302 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:15.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:15.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:15.404 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:15.428 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:15.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:15.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:15.530 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:15.561 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:15.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:15.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:15.663 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:15.690 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:15.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:15.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:15.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:15 smithi136 ceph-mon[109861]: pgmap v2649: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 82 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 282 op/s; 40524/233841 objects degraded (17.330%); 12983/233841 objects misplaced (5.552%); 49 MiB/s, 781 objects/s recovering 2024-08-17T18:06:15.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:15 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 40524/233841 objects degraded (17.330%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:15.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:15 smithi136 ceph-mon[115078]: pgmap v2649: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 82 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 282 op/s; 40524/233841 objects degraded (17.330%); 12983/233841 objects misplaced (5.552%); 49 MiB/s, 781 objects/s recovering 2024-08-17T18:06:15.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:15 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 40524/233841 objects degraded (17.330%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:15 smithi165 ceph-mon[105517]: pgmap v2649: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 82 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 282 op/s; 40524/233841 objects degraded (17.330%); 12983/233841 objects misplaced (5.552%); 49 MiB/s, 781 objects/s recovering 2024-08-17T18:06:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:15 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 40524/233841 objects degraded (17.330%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:15.792 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:15.824 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:15.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:15.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:15.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:15.953 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:15.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:15.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:16.043 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:06:16.044 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:06:16.055 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:16.087 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:16.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:16.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:16.189 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:16.213 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:16.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:16.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:16.315 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:16.342 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:16.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:16.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:16.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:16.468 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:16.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:16.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:16.570 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:16.595 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:16.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:16.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:16.697 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:16.719 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:16.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:16.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:16.821 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:16.848 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:16.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:16.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:16.950 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:16.976 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:16.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:16.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:17.078 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:17.111 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:17.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:17.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:17.213 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:17.250 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:17.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:17.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:17.352 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:17.376 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:17.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:17.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:17.478 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:17.510 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:17.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:17.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:17.611 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:17.633 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:17.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:17.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:17.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:17 smithi136 ceph-mon[109861]: pgmap v2650: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 82 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 189 op/s; 40524/233841 objects degraded (17.330%); 12983/233841 objects misplaced (5.552%); 15 MiB/s, 246 objects/s recovering 2024-08-17T18:06:17.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:17 smithi136 ceph-mon[115078]: pgmap v2650: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 82 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 189 op/s; 40524/233841 objects degraded (17.330%); 12983/233841 objects misplaced (5.552%); 15 MiB/s, 246 objects/s recovering 2024-08-17T18:06:17.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:06:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:06:17] "GET /metrics HTTP/1.1" 200 36695 "" "Prometheus/2.51.0" 2024-08-17T18:06:17.735 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:17 smithi165 ceph-mon[105517]: pgmap v2650: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 82 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 189 op/s; 40524/233841 objects degraded (17.330%); 12983/233841 objects misplaced (5.552%); 15 MiB/s, 246 objects/s recovering 2024-08-17T18:06:17.761 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:17.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:17.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:17.863 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:17.885 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:17.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:17.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:17.987 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:18.010 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:18.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:18.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:18.112 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:18.135 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:18.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:18.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:18.237 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:18.259 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:18.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:18.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:18.361 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:18.388 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:18.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:18.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:18.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:18.513 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:18.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:18.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:18.615 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:18.644 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:18.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:18.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:18.746 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:18.782 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:18.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:18.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:18.884 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:18.908 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:18.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:18.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:19.010 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:19.043 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:19.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:19.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:19.145 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:19.172 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:19.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:19.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:19.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:19.306 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:19.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:19.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:19.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:19.429 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:19.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:19.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:19.531 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:19.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:19.566 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:19.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:19.619 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:19 smithi136 ceph-mon[109861]: pgmap v2651: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 310 op/s; 39864/229098 objects degraded (17.400%); 12282/229098 objects misplaced (5.361%); 17 MiB/s, 269 objects/s recovering 2024-08-17T18:06:19.620 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:19 smithi136 ceph-mon[115078]: pgmap v2651: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 310 op/s; 39864/229098 objects degraded (17.400%); 12282/229098 objects misplaced (5.361%); 17 MiB/s, 269 objects/s recovering 2024-08-17T18:06:19.644 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:19 smithi165 ceph-mon[105517]: pgmap v2651: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 310 op/s; 39864/229098 objects degraded (17.400%); 12282/229098 objects misplaced (5.361%); 17 MiB/s, 269 objects/s recovering 2024-08-17T18:06:19.667 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:19.699 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:19.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:19.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:19.801 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:19.829 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:19.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:19.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:19.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:19.962 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:19.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:19.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:20.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:20.097 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:20.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:20.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:20.199 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:20.203 INFO:teuthology.orchestra.run.smithi136.stdout:moved 0 / 297 (0%) 2024-08-17T18:06:20.204 INFO:teuthology.orchestra.run.smithi136.stdout:avg 42.4286 2024-08-17T18:06:20.204 INFO:teuthology.orchestra.run.smithi136.stdout:stddev 5.0669 -> 5.0669 (expected baseline 6.03053) 2024-08-17T18:06:20.204 INFO:teuthology.orchestra.run.smithi136.stdout:min osd.6 with 37 -> 37 pgs (0.872054 -> 0.872054 * mean) 2024-08-17T18:06:20.204 INFO:teuthology.orchestra.run.smithi136.stdout:max osd.1 with 53 -> 53 pgs (1.24916 -> 1.24916 * mean) 2024-08-17T18:06:20.204 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:06:20.204 INFO:teuthology.orchestra.run.smithi136.stdout:oload 1000 2024-08-17T18:06:20.204 INFO:teuthology.orchestra.run.smithi136.stdout:max_change 1 2024-08-17T18:06:20.204 INFO:teuthology.orchestra.run.smithi136.stdout:max_change_osds 4 2024-08-17T18:06:20.204 INFO:teuthology.orchestra.run.smithi136.stdout:average_utilization 0.0311 2024-08-17T18:06:20.204 INFO:teuthology.orchestra.run.smithi136.stdout:overload_utilization 0.3113 2024-08-17T18:06:20.204 INFO:teuthology.orchestra.run.smithi136.stderr:no change 2024-08-17T18:06:20.226 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:20.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:20.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:20.327 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:20.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:20.358 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:20.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:20.459 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:20.489 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:20.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:20.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:20.501 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:20 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 39864/229098 objects degraded (17.400%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:20.502 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:20 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2947876950' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:06:20.502 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:20 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 39864/229098 objects degraded (17.400%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:20.502 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:20 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2947876950' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:06:20.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:20.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:20.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:20.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:20.700 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:06:20.700 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:06:20.700 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:06:20.700 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:06:20.700 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:06:20.700 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:06:20.701 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:06:20.701 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:06:20.701 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:06:20.701 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:06:20.701 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:06:20.701 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:06:20.701 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:06:20.701 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:06:20.702 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:06:20.702 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:06:20.732 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:20 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 39864/229098 objects degraded (17.400%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:20 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2947876950' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:06:20.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:20.762 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:20.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:20.863 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:20.887 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:20.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:20.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:20.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:21.016 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:21.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:21.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:21.117 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:21.144 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:21.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:21.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:21.246 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:21.271 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:21.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:21.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:21.372 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:21.396 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:21.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:21.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:21.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:21 smithi136 ceph-mon[109861]: from='client.35886 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:06:21.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:21 smithi136 ceph-mon[109861]: pgmap v2652: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 281 op/s; 39864/229098 objects degraded (17.400%); 12282/229098 objects misplaced (5.361%); 15 MiB/s, 244 objects/s recovering 2024-08-17T18:06:21.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:06:21.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:21 smithi136 ceph-mon[115078]: from='client.35886 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:06:21.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:21 smithi136 ceph-mon[115078]: pgmap v2652: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 281 op/s; 39864/229098 objects degraded (17.400%); 12282/229098 objects misplaced (5.361%); 15 MiB/s, 244 objects/s recovering 2024-08-17T18:06:21.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:06:21.498 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:21.521 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:21.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:21.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:21.623 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:21.650 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:21.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:21.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:21.751 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:21 smithi165 ceph-mon[105517]: from='client.35886 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:06:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:21 smithi165 ceph-mon[105517]: pgmap v2652: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 281 op/s; 39864/229098 objects degraded (17.400%); 12282/229098 objects misplaced (5.361%); 15 MiB/s, 244 objects/s recovering 2024-08-17T18:06:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:06:21.783 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:21.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:21.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:21.885 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:21.908 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:21.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:21.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:22.010 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:22.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:22.046 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:22.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:22.147 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:22.170 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:22.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:22.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:22.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:22.293 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:22.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:22.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:22.326 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:06:22.395 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:22.417 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:22.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:22.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:22.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:22.546 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:22.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:22.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:22.648 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:22.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:22.678 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:22.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:22.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:22 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2985258568' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:06:22.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:22 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:06:22.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:22 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2985258568' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:06:22.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:22 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:06:22.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:22 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2985258568' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:06:22.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:22 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:06:22.778 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:22.801 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:22.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:22.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:22.903 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:22.929 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:22.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:22.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:23.031 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:23.052 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:23.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:23.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:23.154 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:23.178 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:23.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:23.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:23.280 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:23.303 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:23.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:23.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:23.405 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:23.436 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:23.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:23.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:23.538 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:23.560 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:23.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:23.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:23.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:23.690 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:23.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:23.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:23 smithi136 ceph-mon[109861]: pgmap v2653: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 362 op/s; 39742/225339 objects degraded (17.637%); 11811/225339 objects misplaced (5.241%); 16 MiB/s, 257 objects/s recovering 2024-08-17T18:06:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:23 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:06:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:23 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:06:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:23 smithi136 ceph-mon[109861]: osdmap e1487: 8 total, 8 up, 7 in 2024-08-17T18:06:23.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:23 smithi136 ceph-mon[115078]: pgmap v2653: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 362 op/s; 39742/225339 objects degraded (17.637%); 11811/225339 objects misplaced (5.241%); 16 MiB/s, 257 objects/s recovering 2024-08-17T18:06:23.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:23 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:06:23.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:23 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:06:23.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:23 smithi136 ceph-mon[115078]: osdmap e1487: 8 total, 8 up, 7 in 2024-08-17T18:06:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:23 smithi165 ceph-mon[105517]: pgmap v2653: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 362 op/s; 39742/225339 objects degraded (17.637%); 11811/225339 objects misplaced (5.241%); 16 MiB/s, 257 objects/s recovering 2024-08-17T18:06:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:23 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:06:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:23 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:06:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:23 smithi165 ceph-mon[105517]: osdmap e1487: 8 total, 8 up, 7 in 2024-08-17T18:06:23.792 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:23.823 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:23.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:23.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:23.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:23.950 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:23.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:23.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:24.052 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:24.075 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:24.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:24.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:24.131 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-08-17T18:06:24.131 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-08-17T18:06:24.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:24.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:24.212 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:24.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:24.313 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:24.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:24.352 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:24.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:24.376 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:06:24.376 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:06:24.453 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:24.465 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:06:24.486 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:24.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:24.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:24.588 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:24.593 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:06:24.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:24.617 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:24.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:24.718 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:24.740 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:24.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:24.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:24.841 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:24.864 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:24.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:24.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:24.966 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:24.997 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:24.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:24.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:25.006 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-17T18:06:25.100 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:25.120 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:25.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:25.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:25.222 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:25.248 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:25.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:25.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:25.350 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:25.373 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:25.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:25.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:25.458 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:06:25.474 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:25.500 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:25.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:25.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:25.518 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:25 smithi136 ceph-mon[109861]: pgmap v2655: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 321 op/s; 39145/224190 objects degraded (17.461%); 11811/224190 objects misplaced (5.268%); 8.8 MiB/s, 141 objects/s recovering 2024-08-17T18:06:25.518 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:25 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 39145/224190 objects degraded (17.461%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:25.518 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:25 smithi136 ceph-mon[115078]: pgmap v2655: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 321 op/s; 39145/224190 objects degraded (17.461%); 11811/224190 objects misplaced (5.268%); 8.8 MiB/s, 141 objects/s recovering 2024-08-17T18:06:25.518 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:25 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 39145/224190 objects degraded (17.461%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:25.602 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:25.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:25.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:25.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:25.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:25.755 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:25.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:25.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:25 smithi165 ceph-mon[105517]: pgmap v2655: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 321 op/s; 39145/224190 objects degraded (17.461%); 11811/224190 objects misplaced (5.268%); 8.8 MiB/s, 141 objects/s recovering 2024-08-17T18:06:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:25 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 39145/224190 objects degraded (17.461%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:25.785 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:06:25.787 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:06:25.858 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:25.879 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:25.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:25.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:25.981 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:26.002 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:26.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:26.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:26.104 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:26.125 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:26.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:26.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:26.227 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:26.257 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:26.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:26.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:26.359 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:26.393 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:26.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:26.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:26.495 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:26.528 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:26.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:26.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:26.630 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:26.653 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:26.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:26.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:26.755 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:26.788 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:26.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:26.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:26.890 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:26.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:26.923 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:26.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:27.024 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:27.047 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:27.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:27.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:27.062 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:06:27.149 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:27.172 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:27.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:27.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:27.273 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:27.296 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:27.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:27.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:27.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:27.429 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:27.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:27.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:27.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:27 smithi136 ceph-mon[109861]: pgmap v2656: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 321 op/s; 39145/224190 objects degraded (17.461%); 11811/224190 objects misplaced (5.268%); 8.8 MiB/s, 141 objects/s recovering 2024-08-17T18:06:27.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:27 smithi136 ceph-mon[115078]: pgmap v2656: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 321 op/s; 39145/224190 objects degraded (17.461%); 11811/224190 objects misplaced (5.268%); 8.8 MiB/s, 141 objects/s recovering 2024-08-17T18:06:27.532 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:27.558 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:27.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:27.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:27.660 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:27.686 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:27.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:27.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:27.703 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:06:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:06:27] "GET /metrics HTTP/1.1" 200 36692 "" "Prometheus/2.51.0" 2024-08-17T18:06:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:27 smithi165 ceph-mon[105517]: pgmap v2656: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 321 op/s; 39145/224190 objects degraded (17.461%); 11811/224190 objects misplaced (5.268%); 8.8 MiB/s, 141 objects/s recovering 2024-08-17T18:06:27.788 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:27.815 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:27.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:27.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:27.917 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:27.938 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:27.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:27.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:28.040 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:28.066 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:28.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:28.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:28.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:28.192 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:28.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:28.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:28.294 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:28.324 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:28.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:28.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:28.426 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:28.450 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:28.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:28.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:28.551 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:28.576 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:28.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:28.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:28.678 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:28.702 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:28.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:28.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:28.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:28.833 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:28.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:28.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:28.935 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:28.961 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:28.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:28.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:29.063 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:29.085 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:29.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:29.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:29.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:29.215 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:29.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:29.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:29.316 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:29.340 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:29.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:29.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:29.442 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:29.467 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:29.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:29.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:29.568 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:29.601 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:29.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:29.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:29.633 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:29 smithi136 ceph-mon[109861]: pgmap v2657: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 324 op/s; 38454/219372 objects degraded (17.529%); 11306/219372 objects misplaced (5.154%); 8.7 MiB/s, 138 objects/s recovering 2024-08-17T18:06:29.633 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:29 smithi136 ceph-mon[115078]: pgmap v2657: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 324 op/s; 38454/219372 objects degraded (17.529%); 11306/219372 objects misplaced (5.154%); 8.7 MiB/s, 138 objects/s recovering 2024-08-17T18:06:29.702 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:29.723 INFO:teuthology.orchestra.run.smithi136.stdout:moved 0 / 297 (0%) 2024-08-17T18:06:29.723 INFO:teuthology.orchestra.run.smithi136.stdout:avg 42.4286 2024-08-17T18:06:29.723 INFO:teuthology.orchestra.run.smithi136.stdout:stddev 5.0669 -> 5.0669 (expected baseline 6.03053) 2024-08-17T18:06:29.723 INFO:teuthology.orchestra.run.smithi136.stdout:min osd.6 with 37 -> 37 pgs (0.872054 -> 0.872054 * mean) 2024-08-17T18:06:29.723 INFO:teuthology.orchestra.run.smithi136.stdout:max osd.1 with 53 -> 53 pgs (1.24916 -> 1.24916 * mean) 2024-08-17T18:06:29.723 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:06:29.723 INFO:teuthology.orchestra.run.smithi136.stdout:oload 1000 2024-08-17T18:06:29.724 INFO:teuthology.orchestra.run.smithi136.stdout:max_change 3 2024-08-17T18:06:29.724 INFO:teuthology.orchestra.run.smithi136.stdout:max_change_osds 4 2024-08-17T18:06:29.724 INFO:teuthology.orchestra.run.smithi136.stdout:average_utilization 0.0304 2024-08-17T18:06:29.724 INFO:teuthology.orchestra.run.smithi136.stdout:overload_utilization 0.3042 2024-08-17T18:06:29.724 INFO:teuthology.orchestra.run.smithi136.stderr:no change 2024-08-17T18:06:29.728 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:29.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:29.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:29 smithi165 ceph-mon[105517]: pgmap v2657: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 324 op/s; 38454/219372 objects degraded (17.529%); 11306/219372 objects misplaced (5.154%); 8.7 MiB/s, 138 objects/s recovering 2024-08-17T18:06:29.829 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:29.856 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:29.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:29.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:29.958 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:29.984 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:29.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:29.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:30.086 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:30.116 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:30.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:30.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:30.218 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:30.242 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:30.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:30.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:30.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:30.369 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:30.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:30.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:30.471 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:30.498 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:30.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:30.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:30.572 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:30 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 38454/219372 objects degraded (17.529%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:30.572 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:30 smithi136 ceph-mon[109861]: from='client.45803 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:06:30.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:30 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 38454/219372 objects degraded (17.529%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:30.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:30 smithi136 ceph-mon[115078]: from='client.45803 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:06:30.600 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:30.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:30.622 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:30.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:30.723 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:30.745 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:30.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:30.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:30.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:30 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 38454/219372 objects degraded (17.529%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:30.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:30 smithi165 ceph-mon[105517]: from='client.45803 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:06:30.847 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:30.868 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:30.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:30.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:30.970 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:30.991 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:30.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:30.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:31.093 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:31.117 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:31.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:31.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:31.218 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:31.241 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:31.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:31.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:31.343 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:31.372 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:31.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:31.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:31.474 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:31.495 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:31.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:31.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:31.575 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:31 smithi136 ceph-mon[109861]: pgmap v2658: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 324 op/s; 38454/219372 objects degraded (17.529%); 11306/219372 objects misplaced (5.154%); 8.7 MiB/s, 138 objects/s recovering 2024-08-17T18:06:31.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:31 smithi136 ceph-mon[115078]: pgmap v2658: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 324 op/s; 38454/219372 objects degraded (17.529%); 11306/219372 objects misplaced (5.154%); 8.7 MiB/s, 138 objects/s recovering 2024-08-17T18:06:31.597 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:31.624 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:31.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:31.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:31.725 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:31.752 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:31.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:31.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:31.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:31 smithi165 ceph-mon[105517]: pgmap v2658: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 324 op/s; 38454/219372 objects degraded (17.529%); 11306/219372 objects misplaced (5.154%); 8.7 MiB/s, 138 objects/s recovering 2024-08-17T18:06:31.854 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:31.883 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:31.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:31.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:31.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:32.012 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:32.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:32.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:32.114 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:32.146 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:32.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:32.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:32.184 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:06:32.184 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:06:32.247 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:32.269 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:32.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:32.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:32.340 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:06:32.371 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:32.402 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:32.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:32.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:32.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:32 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2884968482' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:06:32.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:32 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:06:32.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:32 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2884968482' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:06:32.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:32 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:06:32.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:32.526 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:32.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:32.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:32.628 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:32.649 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:32.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:32.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:32.750 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:32.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:32 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2884968482' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:06:32.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:32 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:06:32.777 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:32.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:32.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:32.878 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:32.901 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:32.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:32.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:33.002 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:33.041 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:33.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:33.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:33.073 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:06:33.073 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:06:33.094 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:06:33.095 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:06:33.112 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:06:33.112 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:06:33.131 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:06:33.131 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:06:33.143 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:33.153 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:06:33.153 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:06:33.175 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:06:33.176 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:06:33.181 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:33.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:33.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:33.195 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:06:33.195 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:06:33.216 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:06:33.217 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:06:33.282 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:33.316 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:33.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:33.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:33.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:33.449 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:33.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:33.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:33.550 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:33.580 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:33.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:33.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:33.596 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:33 smithi136 ceph-mon[109861]: pgmap v2659: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 301 op/s; 38371/216279 objects degraded (17.741%); 10775/216279 objects misplaced (4.982%); 9.0 MiB/s, 144 objects/s recovering 2024-08-17T18:06:33.596 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:33 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:06:33.596 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:33 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:06:33.596 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:33 smithi136 ceph-mon[109861]: osdmap e1488: 8 total, 8 up, 7 in 2024-08-17T18:06:33.596 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:33 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:06:33.596 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:33 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3843005010' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:06:33.596 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:33 smithi136 ceph-mon[115078]: pgmap v2659: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 301 op/s; 38371/216279 objects degraded (17.741%); 10775/216279 objects misplaced (4.982%); 9.0 MiB/s, 144 objects/s recovering 2024-08-17T18:06:33.596 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:33 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:06:33.597 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:33 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:06:33.597 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:33 smithi136 ceph-mon[115078]: osdmap e1488: 8 total, 8 up, 7 in 2024-08-17T18:06:33.597 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:33 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:06:33.597 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:33 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3843005010' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:06:33.675 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:33 smithi165 ceph-mon[105517]: pgmap v2659: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 301 op/s; 38371/216279 objects degraded (17.741%); 10775/216279 objects misplaced (4.982%); 9.0 MiB/s, 144 objects/s recovering 2024-08-17T18:06:33.675 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:33 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:06:33.675 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:33 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:06:33.675 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:33 smithi165 ceph-mon[105517]: osdmap e1488: 8 total, 8 up, 7 in 2024-08-17T18:06:33.675 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:33 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:06:33.675 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:33 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3843005010' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:06:33.682 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:33.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:33.705 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:33.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:33.807 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:33.840 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:33.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:33.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:33.941 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:33.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:33.971 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:33.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:34.072 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:34.075 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-17T18:06:34.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:34.094 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:34.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:34.196 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:34.237 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:34.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:34.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:34.339 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:34.362 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:34.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:34.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:34.465 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:34.519 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:34.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:34.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:34.620 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:34.659 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:34.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:34.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:34.761 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:34.785 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:34.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:34.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:34.826 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:06:34.886 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:34.909 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:34.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:34.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:35.011 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:35.051 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:35.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:35.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:35.152 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:35.182 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:35.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:35.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:35.284 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:35.307 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:35.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:35.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:35.338 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:35 smithi136 ceph-mon[109861]: pgmap v2661: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 296 op/s; 37869/215292 objects degraded (17.590%); 10775/215292 objects misplaced (5.005%); 8.7 MiB/s, 139 objects/s recovering 2024-08-17T18:06:35.338 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:35 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 37869/215292 objects degraded (17.590%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:35.339 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:35 smithi136 ceph-mon[115078]: pgmap v2661: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 296 op/s; 37869/215292 objects degraded (17.590%); 10775/215292 objects misplaced (5.005%); 8.7 MiB/s, 139 objects/s recovering 2024-08-17T18:06:35.339 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:35 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 37869/215292 objects degraded (17.590%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:35.409 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:35.440 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:35.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:35.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:35.542 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:35.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:35.575 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:35.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:35.619 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:06:35.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:35 smithi165 ceph-mon[105517]: pgmap v2661: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 296 op/s; 37869/215292 objects degraded (17.590%); 10775/215292 objects misplaced (5.005%); 8.7 MiB/s, 139 objects/s recovering 2024-08-17T18:06:35.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:35 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 37869/215292 objects degraded (17.590%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:35.676 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:35.700 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:35.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:35.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:35.802 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:35.826 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:35.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:35.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:35.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:35.942 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 0, 5] out_osds: [7] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:06:35.942 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:06:35.942 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-08-17T18:06:35.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T18:06:35.952 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:35.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:35.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:36.054 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:36.085 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:36.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:36.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:36.186 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:36.209 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:36.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:36.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:36.310 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:36.339 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:36.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:36.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:36.440 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:36.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:36.463 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:36.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:36.564 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:36.602 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:36.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:36.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:36.704 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:36.735 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:36.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:36.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:36.813 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:06:36.837 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:36.884 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:36.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:36.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:36.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:37.011 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:37.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:37.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:37.113 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:37.145 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:37.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:37.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:37.247 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:37.270 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:37.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:37.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:37.372 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:37.394 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:37.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:37.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:37.465 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:06:37.465 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:06:37.465 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:37 smithi165 ceph-mon[105517]: pgmap v2662: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 296 op/s; 37869/215292 objects degraded (17.590%); 10775/215292 objects misplaced (5.005%); 8.7 MiB/s, 139 objects/s recovering 2024-08-17T18:06:37.496 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:37.516 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:06:37.516 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:06:37.516 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:37 smithi136 ceph-mon[109861]: pgmap v2662: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 296 op/s; 37869/215292 objects degraded (17.590%); 10775/215292 objects misplaced (5.005%); 8.7 MiB/s, 139 objects/s recovering 2024-08-17T18:06:37.517 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:06:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:06:37] "GET /metrics HTTP/1.1" 200 36690 "" "Prometheus/2.51.0" 2024-08-17T18:06:37.517 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:06:37.517 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:06:37.517 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:37 smithi136 ceph-mon[115078]: pgmap v2662: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 296 op/s; 37869/215292 objects degraded (17.590%); 10775/215292 objects misplaced (5.005%); 8.7 MiB/s, 139 objects/s recovering 2024-08-17T18:06:37.525 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:37.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:37.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:37.531 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:06:37.531 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:06:37.536 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:06:37.536 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T18:06:37.536 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-17T18:06:37.536 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:06:37.628 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:37.651 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:37.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:37.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:37.753 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:37.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:37.775 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:37.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:37.806 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:06:37.877 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:37.916 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:37.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:37.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:38.018 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:38.042 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:38.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:38.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:38.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:38.181 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:38.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:38.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:38.283 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:38.305 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:38.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:38.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:38.406 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:38.429 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:38.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:38.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:38.531 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:38.563 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:38.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:38.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:38.665 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:38.702 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:38.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:38.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:38.803 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:38.836 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:38.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:38.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:38.938 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:38.962 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:38.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:38.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:39.064 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:39.085 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:39.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:39.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:39.120 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:06:39.120 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:06:39.121 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:38 smithi136 ceph-mon[109861]: pgmap v2663: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 280 op/s; 37223/210954 objects degraded (17.645%); 10227/210954 objects misplaced (4.848%); 8.9 MiB/s, 143 objects/s recovering 2024-08-17T18:06:39.121 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:06:39.121 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:06:39.121 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:38 smithi136 ceph-mon[115078]: pgmap v2663: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 280 op/s; 37223/210954 objects degraded (17.645%); 10227/210954 objects misplaced (4.848%); 8.9 MiB/s, 143 objects/s recovering 2024-08-17T18:06:39.142 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-17T18:06:39.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:39.216 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:39.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:39.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:38 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:06:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:38 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:06:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:38 smithi165 ceph-mon[105517]: pgmap v2663: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 280 op/s; 37223/210954 objects degraded (17.645%); 10227/210954 objects misplaced (4.848%); 8.9 MiB/s, 143 objects/s recovering 2024-08-17T18:06:39.317 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:39.342 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:39.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:39.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:39.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:39.474 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:39.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:39.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:39.576 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:39.606 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:39.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:39.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:39.708 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:39.738 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:39.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:39.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:39.840 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:39.863 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:39.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:39.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:39.965 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:40.003 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:40.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:40.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:40.105 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:40.127 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:40.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:40.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:40.229 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:40.251 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:40.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:40.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:40.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:39 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 37223/210954 objects degraded (17.645%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:40.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:40.378 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:40.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:40.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:40.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:39 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 37223/210954 objects degraded (17.645%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:40.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:39 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 37223/210954 objects degraded (17.645%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:40.481 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:40.511 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:40.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:40.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:40.612 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:40.647 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:40.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:40.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:40.748 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:40.775 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:40.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:40.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:40.876 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:40.901 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:40.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:40.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:41.002 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:41.003 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:40 smithi136 ceph-mon[109861]: pgmap v2664: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 280 op/s; 37223/210954 objects degraded (17.645%); 10227/210954 objects misplaced (4.848%); 8.9 MiB/s, 143 objects/s recovering 2024-08-17T18:06:41.003 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:40 smithi136 ceph-mon[115078]: pgmap v2664: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 280 op/s; 37223/210954 objects degraded (17.645%); 10227/210954 objects misplaced (4.848%); 8.9 MiB/s, 143 objects/s recovering 2024-08-17T18:06:41.028 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:41.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:41.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:41.130 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:41.154 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:41.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:41.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:41.256 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:40 smithi165 ceph-mon[105517]: pgmap v2664: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 280 op/s; 37223/210954 objects degraded (17.645%); 10227/210954 objects misplaced (4.848%); 8.9 MiB/s, 143 objects/s recovering 2024-08-17T18:06:41.289 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:41.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:41.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:41.390 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:41.413 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:41.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:41.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:41.515 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:41.539 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:41.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:41.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:41.640 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:41.674 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:41.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:41.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:41.776 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:41.799 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:41.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:41.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:41.901 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:41.926 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:41.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:41.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:42.027 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:42.054 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:42.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:42.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:42.155 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:42.185 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:42.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:42.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:42.287 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:42.322 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:42.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:42.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:42.424 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:42.448 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:42.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:42.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:42.549 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:42.571 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:42.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:42.673 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:42.697 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:42.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:42.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:42.799 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:42.824 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:42.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:42.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:42.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:42.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:42.959 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:42.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:43.060 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:43.082 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:43.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:43.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:43.184 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:43.209 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:43.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:43.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:43.311 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:43.333 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:43.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:43.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:43.435 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:43.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:43 smithi136 ceph-mon[109861]: pgmap v2665: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 289 op/s; 37142/207594 objects degraded (17.892%); 9637/207594 objects misplaced (4.642%); 9.2 MiB/s, 147 objects/s recovering 2024-08-17T18:06:43.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:43 smithi136 ceph-mon[115078]: pgmap v2665: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 289 op/s; 37142/207594 objects degraded (17.892%); 9637/207594 objects misplaced (4.642%); 9.2 MiB/s, 147 objects/s recovering 2024-08-17T18:06:43.460 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:43.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:43.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:43.562 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:43.589 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:43.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:43.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:43.690 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:43.712 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:43.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:43.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:43 smithi165 ceph-mon[105517]: pgmap v2665: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 289 op/s; 37142/207594 objects degraded (17.892%); 9637/207594 objects misplaced (4.642%); 9.2 MiB/s, 147 objects/s recovering 2024-08-17T18:06:43.814 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:43.837 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:43.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:43.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:43.939 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:43.961 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:43.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:43.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:44.063 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:44.088 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:44.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:44.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:44.189 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:44.219 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:44.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:44.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:44.320 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:44.343 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:06:44.344 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T18:06:44.348 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:44.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:44.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:44.450 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:44.475 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:44.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:44.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:44.560 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:44 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2952356274' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:06:44.560 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:44 smithi136 ceph-mon[109861]: from='client.45827 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:06:44.560 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:44 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2952356274' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:06:44.561 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:44 smithi136 ceph-mon[115078]: from='client.45827 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:06:44.577 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:44.603 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:44.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:44.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:44.705 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:44.729 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:44.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:44.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:44 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2952356274' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:06:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:44 smithi165 ceph-mon[105517]: from='client.45827 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:06:44.830 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:44.856 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:44.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:44.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:44.958 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:44.988 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:44.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:44.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:45.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:45.117 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:45.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:45.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:45.218 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:45.269 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:45.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:45.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:45.371 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:45.396 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:45.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:45.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:45.498 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:45.523 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:45.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:45.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:45.616 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:45 smithi136 ceph-mon[109861]: pgmap v2666: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 275 op/s; 36590/206436 objects degraded (17.725%); 9637/206436 objects misplaced (4.668%); 9.2 MiB/s, 146 objects/s recovering 2024-08-17T18:06:45.616 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:45 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:06:45.616 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:45 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2952356274' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:06:45.616 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:45 smithi136 ceph-mon[109861]: osdmap e1489: 8 total, 8 up, 7 in 2024-08-17T18:06:45.616 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:45 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 36590/206436 objects degraded (17.725%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:45.617 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:45 smithi136 ceph-mon[115078]: pgmap v2666: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 275 op/s; 36590/206436 objects degraded (17.725%); 9637/206436 objects misplaced (4.668%); 9.2 MiB/s, 146 objects/s recovering 2024-08-17T18:06:45.617 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:45 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:06:45.617 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:45 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2952356274' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:06:45.617 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:45 smithi136 ceph-mon[115078]: osdmap e1489: 8 total, 8 up, 7 in 2024-08-17T18:06:45.617 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:45 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 36590/206436 objects degraded (17.725%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:45.624 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:45.645 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":2665,"stamp":"2024-08-17T18:06:42.252204+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4533786621,"num_objects":69198,"num_object_clones":0,"num_object_copies":207594,"num_objects_missing_on_primary":5474,"num_objects_missing":5474,"num_objects_degraded":37142,"num_objects_misplaced":9637,"num_objects_unfound":0,"num_objects_dirty":69198,"num_whiteouts":0,"num_read":379,"num_read_kb":823,"num_write":115075,"num_write_kb":5889362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23958,"num_bytes_recovered":1568286106,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":164451,"ondisk_log_size":164451,"up":297,"acting":287,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":343,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":19396320,"kb_used_data":15489128,"kb_used_omap":350,"kb_used_meta":3906785,"kb_avail":636819744,"statfs":{"total":671965249536,"available":652103417856,"internally_reserved":0,"allocated":15860867072,"data_stored":15824987277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":358888,"internal_metadata":4000548376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,3,0,1,0,1,3,2,8],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-189726720,"num_objects":-2895,"num_object_clones":0,"num_object_copies":-8685,"num_objects_missing_on_primary":-359,"num_objects_missing":-359,"num_objects_degraded":-1229,"num_objects_misplaced":-1138,"num_objects_unfound":0,"num_objects_dirty":-2895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2895,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1471,"num_bytes_recovered":96403456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002941"},"pg_stats":[{"pgid":"55.19","version":"1488'6338","reported_seq":7824,"reported_epoch":1488,"state":"active+recovery_wait+degraded","last_fresh":"2024-08-17T18:06:41.587346+0000","last_change":"2024-08-17T18:05:19.768391+0000","last_active":"2024-08-17T18:06:41.587346+0000","last_peered":"2024-08-17T18:06:41.587346+0000","last_clean":"2024-08-17T18:04:42.666472+0000","last_became_active":"2024-08-17T18:05:19.664475+0000","last_became_peered":"2024-08-17T18:05:19.664475+0000","last_unstale":"2024-08-17T18:06:41.587346+0000","last_undegraded":"2024-08-17T18:05:19.597318+0000","last_fullsized":"2024-08-17T18:06:41.587346+0000","mapping_epoch":1472,"log_start":"0'0","ondisk_log_start":"0'0","created":1472,"last_epoch_clean":1447,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":6338,"log_dups_size":0,"ondisk_log_size":6338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141033472,"num_objects":2152,"num_object_clones":0,"num_object_copies":6456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3538,"num_write_kb":182080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":693,"num_bytes_recovered":45416448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":["3","2","4"],"object_location_counts":[{"shards":"2,3,4","objects":1290},{"shards":"2,3,5","objects":862}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.18","version":"1488'6431","reported_seq":8529,"reported_epoch":1488,"state":"active+recovery_wait+degraded","last_fresh":"2024-08-17T18:06:37.534570+0000","last_change":"2024-08-17T18:05:23.137482+0000","last_active":"2024-08-17T18:06:37.534570+0000","last_peered":"2024-08-17T18:06:37.534570+0000","last_clean":"2024-08-17T18:04:14.322941+0000","last_became_active":"2024-08-17T18:05:23.106041+0000","last_became_peered":"2024-08-17T18:05:23.106041+0000","last_unstale":"2024-08-17T18:06:37.534570+0000","last_undegraded":"2024-08-17T18:05:20.630708+0000","last_fullsized":"2024-08-17T18:06:37.534570+0000","mapping_epoch":1472,"log_start":"0'0","ondisk_log_start":"0'0","created":1472,"last_epoch_clean":1447,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":6431,"log_dups_size":0,"ondisk_log_size":6431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146538496,"num_objects":2236,"num_object_clones":0,"num_object_copies":6708,"num_objects_missing_on_primary":1365,"num_objects_missing":1365,"num_objects_degraded":1934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3578,"num_write_kb":186048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":672,"num_bytes_recovered":44040192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":["2","3","6"],"object_location_counts":[{"shards":"0,2,6","objects":796},{"shards":"0,5,6","objects":871},{"shards":"2,3,6","objects":569}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"55.17","version":"1488'6480","reported_seq":6594,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:41.423955+0000","last_change":"2024-08-17T18:06:40.837521+0000","last_active":"2024-08-17T18:06:41.423955+0000","last_peered":"2024-08-17T18:06:41.423955+0000","last_clean":"2024-08-17T18:06:41.423955+0000","last_became_active":"2024-08-17T18:05:19.527486+0000","last_became_peered":"2024-08-17T18:05:19.527486+0000","last_unstale":"2024-08-17T18:06:41.423955+0000","last_undegraded":"2024-08-17T18:06:41.423955+0000","last_fullsized":"2024-08-17T18:06:41.423955+0000","mapping_epoch":1472,"log_start":"0'0","ondisk_log_start":"0'0","created":1472,"last_epoch_clean":1473,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":6480,"log_dups_size":0,"ondisk_log_size":6480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T18:06:45.736379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143327232,"num_objects":2187,"num_object_clones":0,"num_object_copies":6561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3623,"num_write_kb":185920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.16","version":"1488'6404","reported_seq":8195,"reported_epoch":1488,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:06:37.526728+0000","last_change":"2024-08-17T18:06:04.225881+0000","last_active":"2024-08-17T18:06:37.526728+0000","last_peered":"2024-08-17T18:06:37.526728+0000","last_clean":"2024-08-17T18:04:42.635203+0000","last_became_active":"2024-08-17T18:06:03.122249+0000","last_became_peered":"2024-08-17T18:06:03.122249+0000","last_unstale":"2024-08-17T18:06:37.526728+0000","last_undegraded":"2024-08-17T18:06:01.776689+0000","last_fullsized":"2024-08-17T18:05:58.672883+0000","mapping_epoch":1482,"log_start":"0'0","ondisk_log_start":"0'0","created":1472,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":6404,"log_dups_size":0,"ondisk_log_size":6404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143917056,"num_objects":2196,"num_object_clones":0,"num_object_copies":6588,"num_objects_missing_on_primary":1349,"num_objects_missing":1349,"num_objects_degraded":5105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3562,"num_write_kb":184256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":683,"num_bytes_recovered":44761088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":1349},{"shards":"1,2,3,5","objects":7},{"shards":"1,5","objects":15},{"shards":"2,3,5","objects":825}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"55.15","version":"1488'6543","reported_seq":11726,"reported_epoch":1488,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:06:41.526568+0000","last_change":"2024-08-17T18:05:58.945848+0000","last_active":"2024-08-17T18:06:41.526568+0000","last_peered":"2024-08-17T18:06:41.526568+0000","last_clean":"2024-08-17T18:05:56.594392+0000","last_became_active":"2024-08-17T18:05:58.935575+0000","last_became_peered":"2024-08-17T18:05:58.935575+0000","last_unstale":"2024-08-17T18:06:41.526568+0000","last_undegraded":"2024-08-17T18:05:58.707928+0000","last_fullsized":"2024-08-17T18:05:58.671950+0000","mapping_epoch":1482,"log_start":"0'0","ondisk_log_start":"0'0","created":1472,"last_epoch_clean":1473,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":6543,"log_dups_size":0,"ondisk_log_size":6543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139984896,"num_objects":2136,"num_object_clones":0,"num_object_copies":6408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2943,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3672,"num_write_kb":185856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1271,"num_bytes_recovered":83263488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1],"avail_no_missing":["0","1","3","6"],"object_location_counts":[{"shards":"0,1","objects":-430},{"shards":"0,1,3,6","objects":531},{"shards":"0,1,6","objects":1644}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"55.14","version":"1488'6472","reported_seq":6586,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:41.417574+0000","last_change":"2024-08-17T18:06:37.962189+0000","last_active":"2024-08-17T18:06:41.417574+0000","last_peered":"2024-08-17T18:06:41.417574+0000","last_clean":"2024-08-17T18:06:41.417574+0000","last_became_active":"2024-08-17T18:05:19.539993+0000","last_became_peered":"2024-08-17T18:05:19.539993+0000","last_unstale":"2024-08-17T18:06:41.417574+0000","last_undegraded":"2024-08-17T18:06:41.417574+0000","last_fullsized":"2024-08-17T18:06:41.417574+0000","mapping_epoch":1472,"log_start":"0'0","ondisk_log_start":"0'0","created":1472,"last_epoch_clean":1473,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":6472,"log_dups_size":0,"ondisk_log_size":6472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T18:06:42.908533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146636800,"num_objects":2238,"num_object_clones":0,"num_object_copies":6714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3596,"num_write_kb":186656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"55.13","version":"1488'6385","reported_seq":10528,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:41.602358+0000","last_change":"2024-08-17T18:06:39.804230+0000","last_active":"2024-08-17T18:06:41.602358+0000","last_peered":"2024-08-17T18:06:41.602358+0000","last_clean":"2024-08-17T18:06:41.602358+0000","last_became_active":"2024-08-17T18:05:19.620346+0000","last_became_peered":"2024-08-17T18:05:19.620346+0000","last_unstale":"2024-08-17T18:06:41.602358+0000","last_undegraded":"2024-08-17T18:06:41.602358+0000","last_fullsized":"2024-08-17T18:06:41.602358+0000","mapping_epoch":1472,"log_start":"0'0","ondisk_log_start":"0'0","created":1472,"last_epoch_clean":1486,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":6385,"log_dups_size":0,"ondisk_log_size":6385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T18:06:44.752443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140345344,"num_objects":2142,"num_object_clones":0,"num_object_copies":6426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3568,"num_write_kb":182688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":131727360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.12","version":"1488'6541","reported_seq":6656,"reported_epoch":1488,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:06:41.248801+0000","last_change":"2024-08-17T18:05:59.022497+0000","last_active":"2024-08-17T18:06:41.248801+0000","last_peered":"2024-08-17T18:06:41.248801+0000","last_clean":"2024-08-17T18:05:56.647831+0000","last_became_active":"2024-08-17T18:05:59.015639+0000","last_became_peered":"2024-08-17T18:05:59.015639+0000","last_unstale":"2024-08-17T18:06:41.248801+0000","last_undegraded":"2024-08-17T18:05:58.699418+0000","last_fullsized":"2024-08-17T18:05:58.674411+0000","mapping_epoch":1482,"log_start":"0'0","ondisk_log_start":"0'0","created":1472,"last_epoch_clean":1474,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":6541,"log_dups_size":0,"ondisk_log_size":6541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140574720,"num_objects":2145,"num_object_clones":0,"num_object_copies":6435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3673,"num_write_kb":186176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2],"avail_no_missing":["1","2","6"],"object_location_counts":[{"shards":"1,2","objects":-462},{"shards":"1,2,6","objects":2199}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.11","version":"1488'6474","reported_seq":8975,"reported_epoch":1488,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:06:37.792791+0000","last_change":"2024-08-17T18:05:59.001166+0000","last_active":"2024-08-17T18:06:37.792791+0000","last_peered":"2024-08-17T18:06:37.792791+0000","last_clean":"2024-08-17T18:04:14.334099+0000","last_became_active":"2024-08-17T18:05:58.948003+0000","last_became_peered":"2024-08-17T18:05:58.948003+0000","last_unstale":"2024-08-17T18:06:37.792791+0000","last_undegraded":"2024-08-17T18:05:58.781533+0000","last_fullsized":"2024-08-17T18:05:58.672733+0000","mapping_epoch":1482,"log_start":"0'0","ondisk_log_start":"0'0","created":1472,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":6474,"log_dups_size":0,"ondisk_log_size":6474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150011904,"num_objects":2289,"num_object_clones":0,"num_object_copies":6867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3587,"num_write_kb":188032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":672,"num_bytes_recovered":44040192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5],"avail_no_missing":["4","2","3"],"object_location_counts":[{"shards":"0,3,4","objects":831},{"shards":"2,3,4","objects":546},{"shards":"3,4,5","objects":847},{"shards":"4,5","objects":64}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"55.10","version":"1488'6333","reported_seq":6451,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:41.134778+0000","last_change":"2024-08-17T18:06:41.098506+0000","last_active":"2024-08-17T18:06:41.134778+0000","last_peered":"2024-08-17T18:06:41.134778+0000","last_clean":"2024-08-17T18:06:41.134778+0000","last_became_active":"2024-08-17T18:05:19.555661+0000","last_became_peered":"2024-08-17T18:05:19.555661+0000","last_unstale":"2024-08-17T18:06:41.134778+0000","last_undegraded":"2024-08-17T18:06:41.134778+0000","last_fullsized":"2024-08-17T18:06:41.134778+0000","mapping_epoch":1472,"log_start":"0'0","ondisk_log_start":"0'0","created":1472,"last_epoch_clean":1484,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":6333,"log_dups_size":0,"ondisk_log_size":6333,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T18:06:45.876487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139722752,"num_objects":2132,"num_object_clones":0,"num_object_copies":6396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3538,"num_write_kb":181440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.f","version":"1488'7181","reported_seq":15917,"reported_epoch":1488,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T18:06:42.143669+0000","last_change":"2024-08-17T18:06:11.977656+0000","last_active":"2024-08-17T18:06:42.143669+0000","last_peered":"2024-08-17T18:06:42.143669+0000","last_clean":"2024-08-17T18:04:14.223726+0000","last_became_active":"2024-08-17T18:04:45.023531+0000","last_became_peered":"2024-08-17T18:04:45.023531+0000","last_unstale":"2024-08-17T18:06:42.143669+0000","last_undegraded":"2024-08-17T18:04:44.883222+0000","last_fullsized":"2024-08-17T18:04:44.762504+0000","mapping_epoch":1465,"log_start":"1454'1353","ondisk_log_start":"1454'1353","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5828,"log_dups_size":0,"ondisk_log_size":5828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:34:46.944210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280428544,"num_objects":4279,"num_object_clones":0,"num_object_copies":12837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":912,"num_objects_misplaced":1191,"num_objects_unfound":0,"num_objects_dirty":4279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7181,"num_write_kb":366720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4260,"num_bytes_recovered":279183360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[6,0],"avail_no_missing":["6","0","1","2"],"object_location_counts":[{"shards":"0,2,6","objects":1191},{"shards":"0,6","objects":2176}],"blocked_by":[],"up_primary":5,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.e","version":"1488'7050","reported_seq":7153,"reported_epoch":1488,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:06:41.527112+0000","last_change":"2024-08-17T18:05:58.941125+0000","last_active":"2024-08-17T18:06:41.527112+0000","last_peered":"2024-08-17T18:06:41.527112+0000","last_clean":"2024-08-17T18:05:56.610743+0000","last_became_active":"2024-08-17T18:05:58.928501+0000","last_became_peered":"2024-08-17T18:05:58.928501+0000","last_unstale":"2024-08-17T18:06:41.527112+0000","last_undegraded":"2024-08-17T18:06:41.527112+0000","last_fullsized":"2024-08-17T18:06:41.527112+0000","mapping_epoch":1482,"log_start":"1454'1767","ondisk_log_start":"1454'1767","created":1446,"last_epoch_clean":1447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5283,"log_dups_size":0,"ondisk_log_size":5283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277872640,"num_objects":4240,"num_object_clones":0,"num_object_copies":12720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4240,"num_objects_unfound":0,"num_objects_dirty":4240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7050,"num_write_kb":361280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,6],"avail_no_missing":["3","1","6"],"object_location_counts":[{"shards":"1,3,6","objects":4240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.d","version":"1488'7130","reported_seq":14892,"reported_epoch":1488,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:06:37.496175+0000","last_change":"2024-08-17T18:05:57.699059+0000","last_active":"2024-08-17T18:06:37.496175+0000","last_peered":"2024-08-17T18:06:37.496175+0000","last_clean":"2024-08-17T18:04:14.320037+0000","last_became_active":"2024-08-17T18:04:45.022484+0000","last_became_peered":"2024-08-17T18:04:45.022484+0000","last_unstale":"2024-08-17T18:06:37.496175+0000","last_undegraded":"2024-08-17T18:05:57.697933+0000","last_fullsized":"2024-08-17T18:04:44.763637+0000","mapping_epoch":1465,"log_start":"1453'300","ondisk_log_start":"1453'300","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":6830,"log_dups_size":0,"ondisk_log_size":6830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:12:45.255851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284622872,"num_objects":4344,"num_object_clones":0,"num_object_copies":13032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1552,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4344,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7130,"num_write_kb":367105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2581,"num_bytes_recovered":169148416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0],"avail_no_missing":["4","0","3","6"],"object_location_counts":[{"shards":"0,3,4","objects":1552},{"shards":"0,4","objects":2792}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"55.c","version":"1488'7192","reported_seq":7301,"reported_epoch":1488,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:06:41.526832+0000","last_change":"2024-08-17T18:05:58.860915+0000","last_active":"2024-08-17T18:06:41.526832+0000","last_peered":"2024-08-17T18:06:41.526832+0000","last_clean":"2024-08-17T18:05:56.632675+0000","last_became_active":"2024-08-17T18:05:58.847790+0000","last_became_peered":"2024-08-17T18:05:58.847790+0000","last_unstale":"2024-08-17T18:06:41.526832+0000","last_undegraded":"2024-08-17T18:06:41.526832+0000","last_fullsized":"2024-08-17T18:06:41.526832+0000","mapping_epoch":1482,"log_start":"1454'1902","ondisk_log_start":"1454'1902","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5290,"log_dups_size":0,"ondisk_log_size":5290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275644416,"num_objects":4206,"num_object_clones":0,"num_object_copies":12618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4206,"num_objects_unfound":0,"num_objects_dirty":4206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7192,"num_write_kb":364736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,2],"avail_no_missing":["3","2","4"],"object_location_counts":[{"shards":"2,3,4","objects":4206}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.b","version":"1488'7140","reported_seq":7201,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:41.973429+0000","last_change":"2024-08-17T18:03:51.272459+0000","last_active":"2024-08-17T18:06:41.973429+0000","last_peered":"2024-08-17T18:06:41.973429+0000","last_clean":"2024-08-17T18:06:41.973429+0000","last_became_active":"2024-08-17T18:03:51.272155+0000","last_became_peered":"2024-08-17T18:03:51.272155+0000","last_unstale":"2024-08-17T18:06:41.973429+0000","last_undegraded":"2024-08-17T18:06:41.973429+0000","last_fullsized":"2024-08-17T18:06:41.973429+0000","mapping_epoch":1449,"log_start":"1454'1350","ondisk_log_start":"1454'1350","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":5790,"log_dups_size":0,"ondisk_log_size":5790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:17:38.013046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278921216,"num_objects":4256,"num_object_clones":0,"num_object_copies":12768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7140,"num_write_kb":364672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"55.a","version":"1488'7066","reported_seq":17146,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:41.553090+0000","last_change":"2024-08-17T18:06:09.243119+0000","last_active":"2024-08-17T18:06:41.553090+0000","last_peered":"2024-08-17T18:06:41.553090+0000","last_clean":"2024-08-17T18:06:41.553090+0000","last_became_active":"2024-08-17T18:06:09.230324+0000","last_became_peered":"2024-08-17T18:06:09.230324+0000","last_unstale":"2024-08-17T18:06:41.553090+0000","last_undegraded":"2024-08-17T18:06:41.553090+0000","last_fullsized":"2024-08-17T18:06:41.553090+0000","mapping_epoch":1485,"log_start":"1454'815","ondisk_log_start":"1454'815","created":1446,"last_epoch_clean":1486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":6251,"log_dups_size":0,"ondisk_log_size":6251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:21:50.416560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281411584,"num_objects":4294,"num_object_clones":0,"num_object_copies":12882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7066,"num_write_kb":363520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4976,"num_bytes_recovered":326107136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"55.9","version":"1488'6328","reported_seq":10459,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:41.336619+0000","last_change":"2024-08-17T18:06:35.968204+0000","last_active":"2024-08-17T18:06:41.336619+0000","last_peered":"2024-08-17T18:06:41.336619+0000","last_clean":"2024-08-17T18:06:41.336619+0000","last_became_active":"2024-08-17T18:05:19.666114+0000","last_became_peered":"2024-08-17T18:05:19.666114+0000","last_unstale":"2024-08-17T18:06:41.336619+0000","last_undegraded":"2024-08-17T18:06:41.336619+0000","last_fullsized":"2024-08-17T18:06:41.336619+0000","mapping_epoch":1473,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":6328,"log_dups_size":0,"ondisk_log_size":6328,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142213120,"num_objects":2170,"num_object_clones":0,"num_object_copies":6510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3528,"num_write_kb":182336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2009,"num_bytes_recovered":131661824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"55.8","version":"1488'6424","reported_seq":8507,"reported_epoch":1488,"state":"active+recovery_wait+degraded","last_fresh":"2024-08-17T18:06:37.431144+0000","last_change":"2024-08-17T18:05:23.125409+0000","last_active":"2024-08-17T18:06:37.431144+0000","last_peered":"2024-08-17T18:06:37.431144+0000","last_clean":"2024-08-17T18:04:14.322941+0000","last_became_active":"2024-08-17T18:05:22.474572+0000","last_became_peered":"2024-08-17T18:05:22.474572+0000","last_unstale":"2024-08-17T18:06:37.431144+0000","last_undegraded":"2024-08-17T18:05:20.510795+0000","last_fullsized":"2024-08-17T18:06:37.431144+0000","mapping_epoch":1473,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":6424,"log_dups_size":0,"ondisk_log_size":6424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146735104,"num_objects":2239,"num_object_clones":0,"num_object_copies":6717,"num_objects_missing_on_primary":1371,"num_objects_missing":1371,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3571,"num_write_kb":185920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":667,"num_bytes_recovered":43712512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":["2","3","6"],"object_location_counts":[{"shards":"0,2,6","objects":808},{"shards":"0,5,6","objects":867},{"shards":"2,3,5,6","objects":1},{"shards":"2,3,6","objects":563}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"55.7","version":"1488'6514","reported_seq":6628,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:41.208503+0000","last_change":"2024-08-17T18:06:36.893794+0000","last_active":"2024-08-17T18:06:41.208503+0000","last_peered":"2024-08-17T18:06:41.208503+0000","last_clean":"2024-08-17T18:06:41.208503+0000","last_became_active":"2024-08-17T18:05:19.505876+0000","last_became_peered":"2024-08-17T18:05:19.505876+0000","last_unstale":"2024-08-17T18:06:41.208503+0000","last_undegraded":"2024-08-17T18:06:41.208503+0000","last_fullsized":"2024-08-17T18:06:41.208503+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":6514,"log_dups_size":0,"ondisk_log_size":6514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T18:06:41.760761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141099008,"num_objects":2153,"num_object_clones":0,"num_object_copies":6459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3657,"num_write_kb":185920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.6","version":"1488'6398","reported_seq":8151,"reported_epoch":1488,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:06:37.576750+0000","last_change":"2024-08-17T18:06:04.210465+0000","last_active":"2024-08-17T18:06:37.576750+0000","last_peered":"2024-08-17T18:06:37.576750+0000","last_clean":"2024-08-17T18:04:42.635203+0000","last_became_active":"2024-08-17T18:06:03.125266+0000","last_became_peered":"2024-08-17T18:06:03.125266+0000","last_unstale":"2024-08-17T18:06:37.576750+0000","last_undegraded":"2024-08-17T18:06:02.258700+0000","last_fullsized":"2024-08-17T18:05:58.672912+0000","mapping_epoch":1482,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":6398,"log_dups_size":0,"ondisk_log_size":6398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145752064,"num_objects":2224,"num_object_clones":0,"num_object_copies":6672,"num_objects_missing_on_primary":1389,"num_objects_missing":1389,"num_objects_degraded":5193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3556,"num_write_kb":184960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":666,"num_bytes_recovered":43646976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"1,2,3","objects":1389},{"shards":"1,2,3,5","objects":5},{"shards":"1,5","objects":-15},{"shards":"2,3,5","objects":845}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"55.5","version":"1488'6516","reported_seq":11700,"reported_epoch":1488,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:06:41.618823+0000","last_change":"2024-08-17T18:05:58.956803+0000","last_active":"2024-08-17T18:06:41.618823+0000","last_peered":"2024-08-17T18:06:41.618823+0000","last_clean":"2024-08-17T18:05:56.572151+0000","last_became_active":"2024-08-17T18:05:58.949192+0000","last_became_peered":"2024-08-17T18:05:58.949192+0000","last_unstale":"2024-08-17T18:06:41.618823+0000","last_undegraded":"2024-08-17T18:05:58.714152+0000","last_fullsized":"2024-08-17T18:05:58.671310+0000","mapping_epoch":1482,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":6516,"log_dups_size":0,"ondisk_log_size":6516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142671872,"num_objects":2177,"num_object_clones":0,"num_object_copies":6531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2872,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3645,"num_write_kb":186304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1270,"num_bytes_recovered":83263488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1],"avail_no_missing":["0","1","3","6"],"object_location_counts":[{"shards":"0,1","objects":-340},{"shards":"0,1,3,6","objects":508},{"shards":"0,1,6","objects":1630}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"55.4","version":"1488'6523","reported_seq":6639,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:41.558331+0000","last_change":"2024-08-17T18:06:40.903548+0000","last_active":"2024-08-17T18:06:41.558331+0000","last_peered":"2024-08-17T18:06:41.558331+0000","last_clean":"2024-08-17T18:06:41.558331+0000","last_became_active":"2024-08-17T18:05:19.476060+0000","last_became_peered":"2024-08-17T18:05:19.476060+0000","last_unstale":"2024-08-17T18:06:41.558331+0000","last_undegraded":"2024-08-17T18:06:41.558331+0000","last_fullsized":"2024-08-17T18:06:41.558331+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":6523,"log_dups_size":0,"ondisk_log_size":6523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T18:06:45.897110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143294464,"num_objects":2186,"num_object_clones":0,"num_object_copies":6558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3646,"num_write_kb":186656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"55.3","version":"1488'6407","reported_seq":7987,"reported_epoch":1488,"state":"active+recovery_wait+degraded","last_fresh":"2024-08-17T18:06:41.617050+0000","last_change":"2024-08-17T18:05:19.677581+0000","last_active":"2024-08-17T18:06:41.617050+0000","last_peered":"2024-08-17T18:06:41.617050+0000","last_clean":"2024-08-17T18:04:14.321026+0000","last_became_active":"2024-08-17T18:05:19.672336+0000","last_became_peered":"2024-08-17T18:05:19.672336+0000","last_unstale":"2024-08-17T18:06:41.617050+0000","last_undegraded":"2024-08-17T18:05:19.565509+0000","last_fullsized":"2024-08-17T18:06:41.617050+0000","mapping_epoch":1473,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":6407,"log_dups_size":0,"ondisk_log_size":6407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139034624,"num_objects":2121,"num_object_clones":0,"num_object_copies":6363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3589,"num_write_kb":182752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":734,"num_bytes_recovered":48103424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":["1","3","6"],"object_location_counts":[{"shards":"0,1,3","objects":779},{"shards":"1,3,5","objects":849},{"shards":"1,3,6","objects":493}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.2","version":"1488'6501","reported_seq":6617,"reported_epoch":1488,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:06:41.301227+0000","last_change":"2024-08-17T18:05:59.022524+0000","last_active":"2024-08-17T18:06:41.301227+0000","last_peered":"2024-08-17T18:06:41.301227+0000","last_clean":"2024-08-17T18:05:56.654332+0000","last_became_active":"2024-08-17T18:05:59.015527+0000","last_became_peered":"2024-08-17T18:05:59.015527+0000","last_unstale":"2024-08-17T18:06:41.301227+0000","last_undegraded":"2024-08-17T18:05:58.708960+0000","last_fullsized":"2024-08-17T18:05:58.674102+0000","mapping_epoch":1482,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":6501,"log_dups_size":0,"ondisk_log_size":6501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142016512,"num_objects":2167,"num_object_clones":0,"num_object_copies":6501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2846,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3633,"num_write_kb":185600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2],"avail_no_missing":["1","2","6"],"object_location_counts":[{"shards":"1,2","objects":-298},{"shards":"1,2,6","objects":2113}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"55.1","version":"1488'6565","reported_seq":9237,"reported_epoch":1488,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:06:37.619777+0000","last_change":"2024-08-17T18:05:59.002705+0000","last_active":"2024-08-17T18:06:37.619777+0000","last_peered":"2024-08-17T18:06:37.619777+0000","last_clean":"2024-08-17T18:04:14.334099+0000","last_became_active":"2024-08-17T18:05:58.989050+0000","last_became_peered":"2024-08-17T18:05:58.989050+0000","last_unstale":"2024-08-17T18:06:37.619777+0000","last_undegraded":"2024-08-17T18:05:58.801840+0000","last_fullsized":"2024-08-17T18:05:58.672713+0000","mapping_epoch":1482,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":6565,"log_dups_size":0,"ondisk_log_size":6565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142606336,"num_objects":2176,"num_object_clones":0,"num_object_copies":6528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3678,"num_write_kb":187328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":47710208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5],"avail_no_missing":["4","2","3"],"object_location_counts":[{"shards":"0,3,4","objects":855},{"shards":"2,3,4","objects":562},{"shards":"3,4,5","objects":823},{"shards":"4,5","objects":-66}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"55.0","version":"1488'6394","reported_seq":6513,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:41.401779+0000","last_change":"2024-08-17T18:06:38.115877+0000","last_active":"2024-08-17T18:06:41.401779+0000","last_peered":"2024-08-17T18:06:41.401779+0000","last_clean":"2024-08-17T18:06:41.401779+0000","last_became_active":"2024-08-17T18:05:19.528008+0000","last_became_peered":"2024-08-17T18:05:19.528008+0000","last_unstale":"2024-08-17T18:06:41.401779+0000","last_undegraded":"2024-08-17T18:06:41.401779+0000","last_fullsized":"2024-08-17T18:06:41.401779+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":1446,"last_epoch_clean":1484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:03:39.211813+0000","last_clean_scrub_stamp":"2024-08-17T18:03:39.211813+0000","objects_scrubbed":0,"log_size":6394,"log_dups_size":0,"ondisk_log_size":6394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T18:06:42.957420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136773632,"num_objects":2087,"num_object_clones":0,"num_object_copies":6261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3599,"num_write_kb":181952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":105,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.375265+0000","last_change":"2024-08-17T18:05:57.692914+0000","last_active":"2024-08-17T18:06:32.375265+0000","last_peered":"2024-08-17T18:06:32.375265+0000","last_clean":"2024-08-17T18:06:32.375265+0000","last_became_active":"2024-08-17T18:05:57.686218+0000","last_became_peered":"2024-08-17T18:05:57.686218+0000","last_unstale":"2024-08-17T18:06:32.375265+0000","last_undegraded":"2024-08-17T18:06:32.375265+0000","last_fullsized":"2024-08-17T18:06:32.375265+0000","mapping_epoch":1481,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:03:33.068321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2328,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1472,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:22.334169+0000","last_change":"2024-08-17T18:05:57.677218+0000","last_active":"2024-08-17T18:06:22.334169+0000","last_peered":"2024-08-17T18:06:22.334169+0000","last_clean":"2024-08-17T18:06:22.334169+0000","last_became_active":"2024-08-17T18:05:57.676851+0000","last_became_peered":"2024-08-17T18:05:57.676851+0000","last_unstale":"2024-08-17T18:06:22.334169+0000","last_undegraded":"2024-08-17T18:06:22.334169+0000","last_fullsized":"2024-08-17T18:06:22.334169+0000","mapping_epoch":1481,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:43:27.698615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":2287,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.341282+0000","last_change":"2024-08-17T18:05:57.683530+0000","last_active":"2024-08-17T18:06:32.341282+0000","last_peered":"2024-08-17T18:06:32.341282+0000","last_clean":"2024-08-17T18:06:32.341282+0000","last_became_active":"2024-08-17T18:05:57.683149+0000","last_became_peered":"2024-08-17T18:05:57.683149+0000","last_unstale":"2024-08-17T18:06:32.341282+0000","last_undegraded":"2024-08-17T18:06:32.341282+0000","last_fullsized":"2024-08-17T18:06:32.341282+0000","mapping_epoch":1481,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:03:54.522969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2316,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.340464+0000","last_change":"2024-08-17T18:04:14.540774+0000","last_active":"2024-08-17T18:06:32.340464+0000","last_peered":"2024-08-17T18:06:32.340464+0000","last_clean":"2024-08-17T18:06:32.340464+0000","last_became_active":"2024-08-17T18:04:14.539778+0000","last_became_peered":"2024-08-17T18:04:14.539778+0000","last_unstale":"2024-08-17T18:06:32.340464+0000","last_undegraded":"2024-08-17T18:06:32.340464+0000","last_fullsized":"2024-08-17T18:06:32.340464+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:55:02.863360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":38,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:22.330968+0000","last_change":"2024-08-17T18:04:43.777705+0000","last_active":"2024-08-17T18:06:22.330968+0000","last_peered":"2024-08-17T18:06:22.330968+0000","last_clean":"2024-08-17T18:06:22.330968+0000","last_became_active":"2024-08-17T18:04:43.771283+0000","last_became_peered":"2024-08-17T18:04:43.771283+0000","last_unstale":"2024-08-17T18:06:22.330968+0000","last_undegraded":"2024-08-17T18:06:22.330968+0000","last_fullsized":"2024-08-17T18:06:22.330968+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:13.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1560,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.340522+0000","last_change":"2024-08-17T18:04:14.534963+0000","last_active":"2024-08-17T18:06:32.340522+0000","last_peered":"2024-08-17T18:06:32.340522+0000","last_clean":"2024-08-17T18:06:32.340522+0000","last_became_active":"2024-08-17T18:04:14.534634+0000","last_became_peered":"2024-08-17T18:04:14.534634+0000","last_unstale":"2024-08-17T18:06:32.340522+0000","last_undegraded":"2024-08-17T18:06:32.340522+0000","last_fullsized":"2024-08-17T18:06:32.340522+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:33:27.287860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":38,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:22.327865+0000","last_change":"2024-08-17T18:04:43.773534+0000","last_active":"2024-08-17T18:06:22.327865+0000","last_peered":"2024-08-17T18:06:22.327865+0000","last_clean":"2024-08-17T18:06:22.327865+0000","last_became_active":"2024-08-17T18:04:43.768507+0000","last_became_peered":"2024-08-17T18:04:43.768507+0000","last_unstale":"2024-08-17T18:06:22.327865+0000","last_undegraded":"2024-08-17T18:06:22.327865+0000","last_fullsized":"2024-08-17T18:06:22.327865+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:50:57.647351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1477,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:22.334655+0000","last_change":"2024-08-17T17:24:17.458168+0000","last_active":"2024-08-17T18:06:22.334655+0000","last_peered":"2024-08-17T18:06:22.334655+0000","last_clean":"2024-08-17T18:06:22.334655+0000","last_became_active":"2024-08-17T17:24:17.457847+0000","last_became_peered":"2024-08-17T17:24:17.457847+0000","last_unstale":"2024-08-17T18:06:22.334655+0000","last_undegraded":"2024-08-17T18:06:22.334655+0000","last_fullsized":"2024-08-17T18:06:22.334655+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:28.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":72,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.373942+0000","last_change":"2024-08-17T18:04:14.531445+0000","last_active":"2024-08-17T18:06:32.373942+0000","last_peered":"2024-08-17T18:06:32.373942+0000","last_clean":"2024-08-17T18:06:32.373942+0000","last_became_active":"2024-08-17T18:04:14.531099+0000","last_became_peered":"2024-08-17T18:04:14.531099+0000","last_unstale":"2024-08-17T18:06:32.373942+0000","last_undegraded":"2024-08-17T18:06:32.373942+0000","last_fullsized":"2024-08-17T18:06:32.373942+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:43.335123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":2107,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.365573+0000","last_change":"2024-08-17T18:05:57.684489+0000","last_active":"2024-08-17T18:06:32.365573+0000","last_peered":"2024-08-17T18:06:32.365573+0000","last_clean":"2024-08-17T18:06:32.365573+0000","last_became_active":"2024-08-17T18:05:57.684182+0000","last_became_peered":"2024-08-17T18:05:57.684182+0000","last_unstale":"2024-08-17T18:06:32.365573+0000","last_undegraded":"2024-08-17T18:06:32.365573+0000","last_fullsized":"2024-08-17T18:06:32.365573+0000","mapping_epoch":1481,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:33:51.573523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1494,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:22.335752+0000","last_change":"2024-08-17T18:01:23.519890+0000","last_active":"2024-08-17T18:06:22.335752+0000","last_peered":"2024-08-17T18:06:22.335752+0000","last_clean":"2024-08-17T18:06:22.335752+0000","last_became_active":"2024-08-17T18:01:23.519526+0000","last_became_peered":"2024-08-17T18:01:23.519526+0000","last_unstale":"2024-08-17T18:06:22.335752+0000","last_undegraded":"2024-08-17T18:06:22.335752+0000","last_fullsized":"2024-08-17T18:06:22.335752+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:04:52.662528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2281,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.341286+0000","last_change":"2024-08-17T18:01:23.491836+0000","last_active":"2024-08-17T18:06:32.341286+0000","last_peered":"2024-08-17T18:06:32.341286+0000","last_clean":"2024-08-17T18:06:32.341286+0000","last_became_active":"2024-08-17T18:01:23.491209+0000","last_became_peered":"2024-08-17T18:01:23.491209+0000","last_unstale":"2024-08-17T18:06:32.341286+0000","last_undegraded":"2024-08-17T18:06:32.341286+0000","last_fullsized":"2024-08-17T18:06:32.341286+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:35:30.767005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2283,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.365379+0000","last_change":"2024-08-17T17:24:17.452025+0000","last_active":"2024-08-17T18:06:32.365379+0000","last_peered":"2024-08-17T18:06:32.365379+0000","last_clean":"2024-08-17T18:06:32.365379+0000","last_became_active":"2024-08-17T17:24:17.448029+0000","last_became_peered":"2024-08-17T17:24:17.448029+0000","last_unstale":"2024-08-17T18:06:32.365379+0000","last_undegraded":"2024-08-17T18:06:32.365379+0000","last_fullsized":"2024-08-17T18:06:32.365379+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:45.089346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":80,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.374797+0000","last_change":"2024-08-17T18:05:57.683555+0000","last_active":"2024-08-17T18:06:32.374797+0000","last_peered":"2024-08-17T18:06:32.374797+0000","last_clean":"2024-08-17T18:06:32.374797+0000","last_became_active":"2024-08-17T18:05:57.683250+0000","last_became_peered":"2024-08-17T18:05:57.683250+0000","last_unstale":"2024-08-17T18:06:32.374797+0000","last_undegraded":"2024-08-17T18:06:32.374797+0000","last_fullsized":"2024-08-17T18:06:32.374797+0000","mapping_epoch":1481,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:22:10.216651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2296,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.340211+0000","last_change":"2024-08-17T18:04:14.539660+0000","last_active":"2024-08-17T18:06:32.340211+0000","last_peered":"2024-08-17T18:06:32.340211+0000","last_clean":"2024-08-17T18:06:32.340211+0000","last_became_active":"2024-08-17T18:04:14.537852+0000","last_became_peered":"2024-08-17T18:04:14.537852+0000","last_unstale":"2024-08-17T18:06:32.340211+0000","last_undegraded":"2024-08-17T18:06:32.340211+0000","last_fullsized":"2024-08-17T18:06:32.340211+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:47:45.026997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1491,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:22.338847+0000","last_change":"2024-08-17T18:04:43.769892+0000","last_active":"2024-08-17T18:06:22.338847+0000","last_peered":"2024-08-17T18:06:22.338847+0000","last_clean":"2024-08-17T18:06:22.338847+0000","last_became_active":"2024-08-17T18:04:43.769702+0000","last_became_peered":"2024-08-17T18:04:43.769702+0000","last_unstale":"2024-08-17T18:06:22.338847+0000","last_undegraded":"2024-08-17T18:06:22.338847+0000","last_fullsized":"2024-08-17T18:06:22.338847+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:39.373904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2256,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.340340+0000","last_change":"2024-08-17T17:24:17.442338+0000","last_active":"2024-08-17T18:06:32.340340+0000","last_peered":"2024-08-17T18:06:32.340340+0000","last_clean":"2024-08-17T18:06:32.340340+0000","last_became_active":"2024-08-17T17:24:17.441948+0000","last_became_peered":"2024-08-17T17:24:17.441948+0000","last_unstale":"2024-08-17T18:06:32.340340+0000","last_undegraded":"2024-08-17T18:06:32.340340+0000","last_fullsized":"2024-08-17T18:06:32.340340+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:17.891492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":80,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.375495+0000","last_change":"2024-08-17T18:04:43.777736+0000","last_active":"2024-08-17T18:06:32.375495+0000","last_peered":"2024-08-17T18:06:32.375495+0000","last_clean":"2024-08-17T18:06:32.375495+0000","last_became_active":"2024-08-17T18:04:43.777438+0000","last_became_peered":"2024-08-17T18:04:43.777438+0000","last_unstale":"2024-08-17T18:06:32.375495+0000","last_undegraded":"2024-08-17T18:06:32.375495+0000","last_fullsized":"2024-08-17T18:06:32.375495+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:24:01.255927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2236,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.340985+0000","last_change":"2024-08-17T18:04:43.915398+0000","last_active":"2024-08-17T18:06:32.340985+0000","last_peered":"2024-08-17T18:06:32.340985+0000","last_clean":"2024-08-17T18:06:32.340985+0000","last_became_active":"2024-08-17T18:04:43.915212+0000","last_became_peered":"2024-08-17T18:04:43.915212+0000","last_unstale":"2024-08-17T18:06:32.340985+0000","last_undegraded":"2024-08-17T18:06:32.340985+0000","last_fullsized":"2024-08-17T18:06:32.340985+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:43:26.901830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":43,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:22.352623+0000","last_change":"2024-08-17T18:05:57.680418+0000","last_active":"2024-08-17T18:06:22.352623+0000","last_peered":"2024-08-17T18:06:22.352623+0000","last_clean":"2024-08-17T18:06:22.352623+0000","last_became_active":"2024-08-17T18:05:57.680120+0000","last_became_peered":"2024-08-17T18:05:57.680120+0000","last_unstale":"2024-08-17T18:06:22.352623+0000","last_undegraded":"2024-08-17T18:06:22.352623+0000","last_fullsized":"2024-08-17T18:06:22.352623+0000","mapping_epoch":1481,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:20:29.004997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":393,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.341324+0000","last_change":"2024-08-17T18:04:14.555729+0000","last_active":"2024-08-17T18:06:32.341324+0000","last_peered":"2024-08-17T18:06:32.341324+0000","last_clean":"2024-08-17T18:06:32.341324+0000","last_became_active":"2024-08-17T18:04:14.536678+0000","last_became_peered":"2024-08-17T18:04:14.536678+0000","last_unstale":"2024-08-17T18:06:32.341324+0000","last_undegraded":"2024-08-17T18:06:32.341324+0000","last_fullsized":"2024-08-17T18:06:32.341324+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:47:29.424778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2301,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.369726+0000","last_change":"2024-08-17T18:05:57.685781+0000","last_active":"2024-08-17T18:06:32.369726+0000","last_peered":"2024-08-17T18:06:32.369726+0000","last_clean":"2024-08-17T18:06:32.369726+0000","last_became_active":"2024-08-17T18:05:57.685475+0000","last_became_peered":"2024-08-17T18:05:57.685475+0000","last_unstale":"2024-08-17T18:06:32.369726+0000","last_undegraded":"2024-08-17T18:06:32.369726+0000","last_fullsized":"2024-08-17T18:06:32.369726+0000","mapping_epoch":1481,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:01:34.861912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1492,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:22.336752+0000","last_change":"2024-08-17T18:03:51.292044+0000","last_active":"2024-08-17T18:06:22.336752+0000","last_peered":"2024-08-17T18:06:22.336752+0000","last_clean":"2024-08-17T18:06:22.336752+0000","last_became_active":"2024-08-17T18:03:51.268271+0000","last_became_peered":"2024-08-17T18:03:51.268271+0000","last_unstale":"2024-08-17T18:06:22.336752+0000","last_undegraded":"2024-08-17T18:06:22.336752+0000","last_fullsized":"2024-08-17T18:06:22.336752+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:00.892410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4756,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1464,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:22.334714+0000","last_change":"2024-08-17T17:23:07.455919+0000","last_active":"2024-08-17T18:06:22.334714+0000","last_peered":"2024-08-17T18:06:22.334714+0000","last_clean":"2024-08-17T18:06:22.334714+0000","last_became_active":"2024-08-17T17:23:07.455284+0000","last_became_peered":"2024-08-17T17:23:07.455284+0000","last_unstale":"2024-08-17T18:06:22.334714+0000","last_undegraded":"2024-08-17T18:06:22.334714+0000","last_fullsized":"2024-08-17T18:06:22.334714+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:37:46.260775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1488,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:22.337731+0000","last_change":"2024-08-17T18:04:43.766724+0000","last_active":"2024-08-17T18:06:22.337731+0000","last_peered":"2024-08-17T18:06:22.337731+0000","last_clean":"2024-08-17T18:06:22.337731+0000","last_became_active":"2024-08-17T18:04:43.766428+0000","last_became_peered":"2024-08-17T18:04:43.766428+0000","last_unstale":"2024-08-17T18:06:22.337731+0000","last_undegraded":"2024-08-17T18:06:22.337731+0000","last_fullsized":"2024-08-17T18:06:22.337731+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:09:49.102940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1500,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:22.338934+0000","last_change":"2024-08-17T18:03:51.279963+0000","last_active":"2024-08-17T18:06:22.338934+0000","last_peered":"2024-08-17T18:06:22.338934+0000","last_clean":"2024-08-17T18:06:22.338934+0000","last_became_active":"2024-08-17T18:03:51.279506+0000","last_became_peered":"2024-08-17T18:03:51.279506+0000","last_unstale":"2024-08-17T18:06:22.338934+0000","last_undegraded":"2024-08-17T18:06:22.338934+0000","last_fullsized":"2024-08-17T18:06:22.338934+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:20:07.304042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":77,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.374884+0000","last_change":"2024-08-17T18:04:14.542589+0000","last_active":"2024-08-17T18:06:32.374884+0000","last_peered":"2024-08-17T18:06:32.374884+0000","last_clean":"2024-08-17T18:06:32.374884+0000","last_became_active":"2024-08-17T18:04:14.531540+0000","last_became_peered":"2024-08-17T18:04:14.531540+0000","last_unstale":"2024-08-17T18:06:32.374884+0000","last_undegraded":"2024-08-17T18:06:32.374884+0000","last_fullsized":"2024-08-17T18:06:32.374884+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:15:33.014782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2240,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.373597+0000","last_change":"2024-08-17T18:05:57.688872+0000","last_active":"2024-08-17T18:06:32.373597+0000","last_peered":"2024-08-17T18:06:32.373597+0000","last_clean":"2024-08-17T18:06:32.373597+0000","last_became_active":"2024-08-17T18:05:57.688515+0000","last_became_peered":"2024-08-17T18:05:57.688515+0000","last_unstale":"2024-08-17T18:06:32.373597+0000","last_undegraded":"2024-08-17T18:06:32.373597+0000","last_fullsized":"2024-08-17T18:06:32.373597+0000","mapping_epoch":1481,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:57:45.942922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2266,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.367367+0000","last_change":"2024-08-17T17:23:07.428296+0000","last_active":"2024-08-17T18:06:32.367367+0000","last_peered":"2024-08-17T18:06:32.367367+0000","last_clean":"2024-08-17T18:06:32.367367+0000","last_became_active":"2024-08-17T17:23:07.428010+0000","last_became_peered":"2024-08-17T17:23:07.428010+0000","last_unstale":"2024-08-17T18:06:32.367367+0000","last_undegraded":"2024-08-17T18:06:32.367367+0000","last_fullsized":"2024-08-17T18:06:32.367367+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:45.459212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":2307,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.340920+0000","last_change":"2024-08-17T18:04:43.800633+0000","last_active":"2024-08-17T18:06:32.340920+0000","last_peered":"2024-08-17T18:06:32.340920+0000","last_clean":"2024-08-17T18:06:32.340920+0000","last_became_active":"2024-08-17T18:04:43.784612+0000","last_became_peered":"2024-08-17T18:04:43.784612+0000","last_unstale":"2024-08-17T18:06:32.340920+0000","last_undegraded":"2024-08-17T18:06:32.340920+0000","last_fullsized":"2024-08-17T18:06:32.340920+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:11.405039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":80,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.374125+0000","last_change":"2024-08-17T18:04:43.805055+0000","last_active":"2024-08-17T18:06:32.374125+0000","last_peered":"2024-08-17T18:06:32.374125+0000","last_clean":"2024-08-17T18:06:32.374125+0000","last_became_active":"2024-08-17T18:04:43.804708+0000","last_became_peered":"2024-08-17T18:04:43.804708+0000","last_unstale":"2024-08-17T18:06:32.374125+0000","last_undegraded":"2024-08-17T18:06:32.374125+0000","last_fullsized":"2024-08-17T18:06:32.374125+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:47:22.253073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":73,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.379430+0000","last_change":"2024-08-17T18:04:14.529605+0000","last_active":"2024-08-17T18:06:32.379430+0000","last_peered":"2024-08-17T18:06:32.379430+0000","last_clean":"2024-08-17T18:06:32.379430+0000","last_became_active":"2024-08-17T18:04:14.529024+0000","last_became_peered":"2024-08-17T18:04:14.529024+0000","last_unstale":"2024-08-17T18:06:32.379430+0000","last_undegraded":"2024-08-17T18:06:32.379430+0000","last_fullsized":"2024-08-17T18:06:32.379430+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:38:00.485543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2319,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.365477+0000","last_change":"2024-08-17T18:04:43.773667+0000","last_active":"2024-08-17T18:06:32.365477+0000","last_peered":"2024-08-17T18:06:32.365477+0000","last_clean":"2024-08-17T18:06:32.365477+0000","last_became_active":"2024-08-17T18:04:43.773247+0000","last_became_peered":"2024-08-17T18:04:43.773247+0000","last_unstale":"2024-08-17T18:06:32.365477+0000","last_undegraded":"2024-08-17T18:06:32.365477+0000","last_fullsized":"2024-08-17T18:06:32.365477+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:35:48.964631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":144,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.360640+0000","last_change":"2024-08-17T18:04:43.771589+0000","last_active":"2024-08-17T18:06:32.360640+0000","last_peered":"2024-08-17T18:06:32.360640+0000","last_clean":"2024-08-17T18:06:32.360640+0000","last_became_active":"2024-08-17T18:04:43.769601+0000","last_became_peered":"2024-08-17T18:04:43.769601+0000","last_unstale":"2024-08-17T18:06:32.360640+0000","last_undegraded":"2024-08-17T18:06:32.360640+0000","last_fullsized":"2024-08-17T18:06:32.360640+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:42:49.228921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2335,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.365364+0000","last_change":"2024-08-17T18:04:14.537136+0000","last_active":"2024-08-17T18:06:32.365364+0000","last_peered":"2024-08-17T18:06:32.365364+0000","last_clean":"2024-08-17T18:06:32.365364+0000","last_became_active":"2024-08-17T18:04:14.536815+0000","last_became_peered":"2024-08-17T18:04:14.536815+0000","last_unstale":"2024-08-17T18:06:32.365364+0000","last_undegraded":"2024-08-17T18:06:32.365364+0000","last_fullsized":"2024-08-17T18:06:32.365364+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:02.706133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":34,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:22.331395+0000","last_change":"2024-08-17T18:04:43.773480+0000","last_active":"2024-08-17T18:06:22.331395+0000","last_peered":"2024-08-17T18:06:22.331395+0000","last_clean":"2024-08-17T18:06:22.331395+0000","last_became_active":"2024-08-17T18:04:43.773160+0000","last_became_peered":"2024-08-17T18:04:43.773160+0000","last_unstale":"2024-08-17T18:06:22.331395+0000","last_undegraded":"2024-08-17T18:06:22.331395+0000","last_fullsized":"2024-08-17T18:06:22.331395+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:07:55.468132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":146,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.365311+0000","last_change":"2024-08-17T18:02:08.224928+0000","last_active":"2024-08-17T18:06:32.365311+0000","last_peered":"2024-08-17T18:06:32.365311+0000","last_clean":"2024-08-17T18:06:32.365311+0000","last_became_active":"2024-08-17T18:02:08.224604+0000","last_became_peered":"2024-08-17T18:02:08.224604+0000","last_unstale":"2024-08-17T18:06:32.365311+0000","last_undegraded":"2024-08-17T18:06:32.365311+0000","last_fullsized":"2024-08-17T18:06:32.365311+0000","mapping_epoch":1421,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1422,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:55.716971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1486,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:27.789836+0000","last_change":"2024-08-17T18:04:14.526890+0000","last_active":"2024-08-17T18:06:27.789836+0000","last_peered":"2024-08-17T18:06:27.789836+0000","last_clean":"2024-08-17T18:06:27.789836+0000","last_became_active":"2024-08-17T18:04:14.526564+0000","last_became_peered":"2024-08-17T18:04:14.526564+0000","last_unstale":"2024-08-17T18:06:27.789836+0000","last_undegraded":"2024-08-17T18:06:27.789836+0000","last_fullsized":"2024-08-17T18:06:27.789836+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:10:13.229827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":38,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:22.327832+0000","last_change":"2024-08-17T18:04:43.791749+0000","last_active":"2024-08-17T18:06:22.327832+0000","last_peered":"2024-08-17T18:06:22.327832+0000","last_clean":"2024-08-17T18:06:22.327832+0000","last_became_active":"2024-08-17T18:04:43.774474+0000","last_became_peered":"2024-08-17T18:04:43.774474+0000","last_unstale":"2024-08-17T18:06:22.327832+0000","last_undegraded":"2024-08-17T18:06:22.327832+0000","last_fullsized":"2024-08-17T18:06:22.327832+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:19:08.097560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2322,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.366024+0000","last_change":"2024-08-17T18:04:43.781133+0000","last_active":"2024-08-17T18:06:32.366024+0000","last_peered":"2024-08-17T18:06:32.366024+0000","last_clean":"2024-08-17T18:06:32.366024+0000","last_became_active":"2024-08-17T18:04:43.780787+0000","last_became_peered":"2024-08-17T18:04:43.780787+0000","last_unstale":"2024-08-17T18:06:32.366024+0000","last_undegraded":"2024-08-17T18:06:32.366024+0000","last_fullsized":"2024-08-17T18:06:32.366024+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:38:34.901162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2278,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.365129+0000","last_change":"2024-08-17T18:01:23.493896+0000","last_active":"2024-08-17T18:06:32.365129+0000","last_peered":"2024-08-17T18:06:32.365129+0000","last_clean":"2024-08-17T18:06:32.365129+0000","last_became_active":"2024-08-17T18:01:23.493551+0000","last_became_peered":"2024-08-17T18:01:23.493551+0000","last_unstale":"2024-08-17T18:06:32.365129+0000","last_undegraded":"2024-08-17T18:06:32.365129+0000","last_fullsized":"2024-08-17T18:06:32.365129+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:18:47.648483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2256,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.360354+0000","last_change":"2024-08-17T18:04:14.543532+0000","last_active":"2024-08-17T18:06:32.360354+0000","last_peered":"2024-08-17T18:06:32.360354+0000","last_clean":"2024-08-17T18:06:32.360354+0000","last_became_active":"2024-08-17T18:04:14.543131+0000","last_became_peered":"2024-08-17T18:04:14.543131+0000","last_unstale":"2024-08-17T18:06:32.360354+0000","last_undegraded":"2024-08-17T18:06:32.360354+0000","last_fullsized":"2024-08-17T18:06:32.360354+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:51:44.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":80,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.374685+0000","last_change":"2024-08-17T18:05:57.686025+0000","last_active":"2024-08-17T18:06:32.374685+0000","last_peered":"2024-08-17T18:06:32.374685+0000","last_clean":"2024-08-17T18:06:32.374685+0000","last_became_active":"2024-08-17T18:05:57.685715+0000","last_became_peered":"2024-08-17T18:05:57.685715+0000","last_unstale":"2024-08-17T18:06:32.374685+0000","last_undegraded":"2024-08-17T18:06:32.374685+0000","last_fullsized":"2024-08-17T18:06:32.374685+0000","mapping_epoch":1481,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:51:01.641238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":80,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.373663+0000","last_change":"2024-08-17T18:05:57.692696+0000","last_active":"2024-08-17T18:06:32.373663+0000","last_peered":"2024-08-17T18:06:32.373663+0000","last_clean":"2024-08-17T18:06:32.373663+0000","last_became_active":"2024-08-17T18:05:57.692415+0000","last_became_peered":"2024-08-17T18:05:57.692415+0000","last_unstale":"2024-08-17T18:06:32.373663+0000","last_undegraded":"2024-08-17T18:06:32.373663+0000","last_fullsized":"2024-08-17T18:06:32.373663+0000","mapping_epoch":1481,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:28:16.944544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1022,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.365210+0000","last_change":"2024-08-17T18:04:14.537308+0000","last_active":"2024-08-17T18:06:32.365210+0000","last_peered":"2024-08-17T18:06:32.365210+0000","last_clean":"2024-08-17T18:06:32.365210+0000","last_became_active":"2024-08-17T18:04:14.535521+0000","last_became_peered":"2024-08-17T18:04:14.535521+0000","last_unstale":"2024-08-17T18:06:32.365210+0000","last_undegraded":"2024-08-17T18:06:32.365210+0000","last_fullsized":"2024-08-17T18:06:32.365210+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:40:00.384936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1444,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:22.336121+0000","last_change":"2024-08-17T18:05:57.698447+0000","last_active":"2024-08-17T18:06:22.336121+0000","last_peered":"2024-08-17T18:06:22.336121+0000","last_clean":"2024-08-17T18:06:22.336121+0000","last_became_active":"2024-08-17T18:05:57.697990+0000","last_became_peered":"2024-08-17T18:05:57.697990+0000","last_unstale":"2024-08-17T18:06:22.336121+0000","last_undegraded":"2024-08-17T18:06:22.336121+0000","last_fullsized":"2024-08-17T18:06:22.336121+0000","mapping_epoch":1481,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:37:44.240165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":84,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.360525+0000","last_change":"2024-08-17T18:04:43.771309+0000","last_active":"2024-08-17T18:06:32.360525+0000","last_peered":"2024-08-17T18:06:32.360525+0000","last_clean":"2024-08-17T18:06:32.360525+0000","last_became_active":"2024-08-17T18:04:43.770922+0000","last_became_peered":"2024-08-17T18:04:43.770922+0000","last_unstale":"2024-08-17T18:06:32.360525+0000","last_undegraded":"2024-08-17T18:06:32.360525+0000","last_fullsized":"2024-08-17T18:06:32.360525+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:10.173251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":38,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:22.327712+0000","last_change":"2024-08-17T18:04:43.786228+0000","last_active":"2024-08-17T18:06:22.327712+0000","last_peered":"2024-08-17T18:06:22.327712+0000","last_clean":"2024-08-17T18:06:22.327712+0000","last_became_active":"2024-08-17T18:04:43.770976+0000","last_became_peered":"2024-08-17T18:04:43.770976+0000","last_unstale":"2024-08-17T18:06:22.327712+0000","last_undegraded":"2024-08-17T18:06:22.327712+0000","last_fullsized":"2024-08-17T18:06:22.327712+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:20:10.131137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":689,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.373568+0000","last_change":"2024-08-17T18:04:17.566582+0000","last_active":"2024-08-17T18:06:32.373568+0000","last_peered":"2024-08-17T18:06:32.373568+0000","last_clean":"2024-08-17T18:06:32.373568+0000","last_became_active":"2024-08-17T18:04:17.562526+0000","last_became_peered":"2024-08-17T18:04:17.562526+0000","last_unstale":"2024-08-17T18:06:32.373568+0000","last_undegraded":"2024-08-17T18:06:32.373568+0000","last_fullsized":"2024-08-17T18:06:32.373568+0000","mapping_epoch":1460,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T17:06:21.174471+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:27:13.722171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2265,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.365822+0000","last_change":"2024-08-17T17:24:17.455710+0000","last_active":"2024-08-17T18:06:32.365822+0000","last_peered":"2024-08-17T18:06:32.365822+0000","last_clean":"2024-08-17T18:06:32.365822+0000","last_became_active":"2024-08-17T17:24:17.452989+0000","last_became_peered":"2024-08-17T17:24:17.452989+0000","last_unstale":"2024-08-17T18:06:32.365822+0000","last_undegraded":"2024-08-17T18:06:32.365822+0000","last_fullsized":"2024-08-17T18:06:32.365822+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:58:07.894968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2215,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:22.330308+0000","last_change":"2024-08-17T18:05:57.685543+0000","last_active":"2024-08-17T18:06:22.330308+0000","last_peered":"2024-08-17T18:06:22.330308+0000","last_clean":"2024-08-17T18:06:22.330308+0000","last_became_active":"2024-08-17T18:05:57.685043+0000","last_became_peered":"2024-08-17T18:05:57.685043+0000","last_unstale":"2024-08-17T18:06:22.330308+0000","last_undegraded":"2024-08-17T18:06:22.330308+0000","last_fullsized":"2024-08-17T18:06:22.330308+0000","mapping_epoch":1481,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:58:07.467314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1474,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:27.789932+0000","last_change":"2024-08-17T18:05:57.683121+0000","last_active":"2024-08-17T18:06:27.789932+0000","last_peered":"2024-08-17T18:06:27.789932+0000","last_clean":"2024-08-17T18:06:27.789932+0000","last_became_active":"2024-08-17T18:05:57.682499+0000","last_became_peered":"2024-08-17T18:05:57.682499+0000","last_unstale":"2024-08-17T18:06:27.789932+0000","last_undegraded":"2024-08-17T18:06:27.789932+0000","last_fullsized":"2024-08-17T18:06:27.789932+0000","mapping_epoch":1481,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:32:51.810858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1491,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.366676+0000","last_change":"2024-08-17T18:03:51.270293+0000","last_active":"2024-08-17T18:06:32.366676+0000","last_peered":"2024-08-17T18:06:32.366676+0000","last_clean":"2024-08-17T18:06:32.366676+0000","last_became_active":"2024-08-17T18:03:51.269970+0000","last_became_peered":"2024-08-17T18:03:51.269970+0000","last_unstale":"2024-08-17T18:06:32.366676+0000","last_undegraded":"2024-08-17T18:06:32.366676+0000","last_fullsized":"2024-08-17T18:06:32.366676+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:02:30.016517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1495,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:27.789904+0000","last_change":"2024-08-17T18:01:23.497110+0000","last_active":"2024-08-17T18:06:27.789904+0000","last_peered":"2024-08-17T18:06:27.789904+0000","last_clean":"2024-08-17T18:06:27.789904+0000","last_became_active":"2024-08-17T18:01:23.496800+0000","last_became_peered":"2024-08-17T18:01:23.496800+0000","last_unstale":"2024-08-17T18:06:27.789904+0000","last_undegraded":"2024-08-17T18:06:27.789904+0000","last_fullsized":"2024-08-17T18:06:27.789904+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:32:29.892739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1464,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.364415+0000","last_change":"2024-08-17T17:23:07.401624+0000","last_active":"2024-08-17T18:06:32.364415+0000","last_peered":"2024-08-17T18:06:32.364415+0000","last_clean":"2024-08-17T18:06:32.364415+0000","last_became_active":"2024-08-17T17:23:07.400222+0000","last_became_peered":"2024-08-17T17:23:07.400222+0000","last_unstale":"2024-08-17T18:06:32.364415+0000","last_undegraded":"2024-08-17T18:06:32.364415+0000","last_fullsized":"2024-08-17T18:06:32.364415+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:07.069989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1465,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:27.789868+0000","last_change":"2024-08-17T17:24:17.470809+0000","last_active":"2024-08-17T18:06:27.789868+0000","last_peered":"2024-08-17T18:06:27.789868+0000","last_clean":"2024-08-17T18:06:27.789868+0000","last_became_active":"2024-08-17T17:24:17.469753+0000","last_became_peered":"2024-08-17T17:24:17.469753+0000","last_unstale":"2024-08-17T18:06:27.789868+0000","last_undegraded":"2024-08-17T18:06:27.789868+0000","last_fullsized":"2024-08-17T18:06:27.789868+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:00.049425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":2273,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.340927+0000","last_change":"2024-08-17T18:04:14.552806+0000","last_active":"2024-08-17T18:06:32.340927+0000","last_peered":"2024-08-17T18:06:32.340927+0000","last_clean":"2024-08-17T18:06:32.340927+0000","last_became_active":"2024-08-17T18:04:14.535546+0000","last_became_peered":"2024-08-17T18:04:14.535546+0000","last_unstale":"2024-08-17T18:06:32.340927+0000","last_undegraded":"2024-08-17T18:06:32.340927+0000","last_fullsized":"2024-08-17T18:06:32.340927+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:44:45.702272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":73,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.374043+0000","last_change":"2024-08-17T18:04:14.539500+0000","last_active":"2024-08-17T18:06:32.374043+0000","last_peered":"2024-08-17T18:06:32.374043+0000","last_clean":"2024-08-17T18:06:32.374043+0000","last_became_active":"2024-08-17T18:04:14.538738+0000","last_became_peered":"2024-08-17T18:04:14.538738+0000","last_unstale":"2024-08-17T18:06:32.374043+0000","last_undegraded":"2024-08-17T18:06:32.374043+0000","last_fullsized":"2024-08-17T18:06:32.374043+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:31.439947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":34,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:22.327979+0000","last_change":"2024-08-17T18:04:43.771107+0000","last_active":"2024-08-17T18:06:22.327979+0000","last_peered":"2024-08-17T18:06:22.327979+0000","last_clean":"2024-08-17T18:06:22.327979+0000","last_became_active":"2024-08-17T18:04:43.770799+0000","last_became_peered":"2024-08-17T18:04:43.770799+0000","last_unstale":"2024-08-17T18:06:22.327979+0000","last_undegraded":"2024-08-17T18:06:22.327979+0000","last_fullsized":"2024-08-17T18:06:22.327979+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:09:04.350798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1482,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.369773+0000","last_change":"2024-08-17T18:05:57.685616+0000","last_active":"2024-08-17T18:06:32.369773+0000","last_peered":"2024-08-17T18:06:32.369773+0000","last_clean":"2024-08-17T18:06:32.369773+0000","last_became_active":"2024-08-17T18:05:57.685285+0000","last_became_peered":"2024-08-17T18:05:57.685285+0000","last_unstale":"2024-08-17T18:06:32.369773+0000","last_undegraded":"2024-08-17T18:06:32.369773+0000","last_fullsized":"2024-08-17T18:06:32.369773+0000","mapping_epoch":1481,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:19:58.776593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":38,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:22.331554+0000","last_change":"2024-08-17T18:04:43.781111+0000","last_active":"2024-08-17T18:06:22.331554+0000","last_peered":"2024-08-17T18:06:22.331554+0000","last_clean":"2024-08-17T18:06:22.331554+0000","last_became_active":"2024-08-17T18:04:43.771839+0000","last_became_peered":"2024-08-17T18:04:43.771839+0000","last_unstale":"2024-08-17T18:06:22.331554+0000","last_undegraded":"2024-08-17T18:06:22.331554+0000","last_fullsized":"2024-08-17T18:06:22.331554+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:52:45.973049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1490,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.363796+0000","last_change":"2024-08-17T18:05:57.680961+0000","last_active":"2024-08-17T18:06:32.363796+0000","last_peered":"2024-08-17T18:06:32.363796+0000","last_clean":"2024-08-17T18:06:32.363796+0000","last_became_active":"2024-08-17T18:05:57.680691+0000","last_became_peered":"2024-08-17T18:05:57.680691+0000","last_unstale":"2024-08-17T18:06:32.363796+0000","last_undegraded":"2024-08-17T18:06:32.363796+0000","last_fullsized":"2024-08-17T18:06:32.363796+0000","mapping_epoch":1481,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:18:53.999988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":92,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.361063+0000","last_change":"2024-08-17T18:04:14.551061+0000","last_active":"2024-08-17T18:06:32.361063+0000","last_peered":"2024-08-17T18:06:32.361063+0000","last_clean":"2024-08-17T18:06:32.361063+0000","last_became_active":"2024-08-17T18:04:14.543296+0000","last_became_peered":"2024-08-17T18:04:14.543296+0000","last_unstale":"2024-08-17T18:06:32.361063+0000","last_undegraded":"2024-08-17T18:06:32.361063+0000","last_fullsized":"2024-08-17T18:06:32.361063+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:39:45.294265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":43,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:22.349502+0000","last_change":"2024-08-17T18:05:57.685365+0000","last_active":"2024-08-17T18:06:22.349502+0000","last_peered":"2024-08-17T18:06:22.349502+0000","last_clean":"2024-08-17T18:06:22.349502+0000","last_became_active":"2024-08-17T18:05:57.683922+0000","last_became_peered":"2024-08-17T18:05:57.683922+0000","last_unstale":"2024-08-17T18:06:22.349502+0000","last_undegraded":"2024-08-17T18:06:22.349502+0000","last_fullsized":"2024-08-17T18:06:22.349502+0000","mapping_epoch":1481,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:24:38.120397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":80,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:22.335093+0000","last_change":"2024-08-17T18:04:14.547798+0000","last_active":"2024-08-17T18:06:22.335093+0000","last_peered":"2024-08-17T18:06:22.335093+0000","last_clean":"2024-08-17T18:06:22.335093+0000","last_became_active":"2024-08-17T18:04:14.527496+0000","last_became_peered":"2024-08-17T18:04:14.527496+0000","last_unstale":"2024-08-17T18:06:22.335093+0000","last_undegraded":"2024-08-17T18:06:22.335093+0000","last_fullsized":"2024-08-17T18:06:22.335093+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:25:54.338846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1546,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:22.330778+0000","last_change":"2024-08-17T18:04:43.813536+0000","last_active":"2024-08-17T18:06:22.330778+0000","last_peered":"2024-08-17T18:06:22.330778+0000","last_clean":"2024-08-17T18:06:22.330778+0000","last_became_active":"2024-08-17T18:04:43.799823+0000","last_became_peered":"2024-08-17T18:04:43.799823+0000","last_unstale":"2024-08-17T18:06:22.330778+0000","last_undegraded":"2024-08-17T18:06:22.330778+0000","last_fullsized":"2024-08-17T18:06:22.330778+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:04:10.837113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2200,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.366352+0000","last_change":"2024-08-17T18:05:57.677642+0000","last_active":"2024-08-17T18:06:32.366352+0000","last_peered":"2024-08-17T18:06:32.366352+0000","last_clean":"2024-08-17T18:06:32.366352+0000","last_became_active":"2024-08-17T18:05:57.677389+0000","last_became_peered":"2024-08-17T18:05:57.677389+0000","last_unstale":"2024-08-17T18:06:32.366352+0000","last_undegraded":"2024-08-17T18:06:32.366352+0000","last_fullsized":"2024-08-17T18:06:32.366352+0000","mapping_epoch":1481,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:56:28.758678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":204,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:22.337692+0000","last_change":"2024-08-17T18:04:14.527380+0000","last_active":"2024-08-17T18:06:22.337692+0000","last_peered":"2024-08-17T18:06:22.337692+0000","last_clean":"2024-08-17T18:06:22.337692+0000","last_became_active":"2024-08-17T18:04:14.527076+0000","last_became_peered":"2024-08-17T18:04:14.527076+0000","last_unstale":"2024-08-17T18:06:22.337692+0000","last_undegraded":"2024-08-17T18:06:22.337692+0000","last_fullsized":"2024-08-17T18:06:22.337692+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:36:35.583709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":73,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.375026+0000","last_change":"2024-08-17T18:04:14.541814+0000","last_active":"2024-08-17T18:06:32.375026+0000","last_peered":"2024-08-17T18:06:32.375026+0000","last_clean":"2024-08-17T18:06:32.375026+0000","last_became_active":"2024-08-17T18:04:14.541403+0000","last_became_peered":"2024-08-17T18:04:14.541403+0000","last_unstale":"2024-08-17T18:06:32.375026+0000","last_undegraded":"2024-08-17T18:06:32.375026+0000","last_fullsized":"2024-08-17T18:06:32.375026+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:07:45.294924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":80,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.374453+0000","last_change":"2024-08-17T18:05:57.686937+0000","last_active":"2024-08-17T18:06:32.374453+0000","last_peered":"2024-08-17T18:06:32.374453+0000","last_clean":"2024-08-17T18:06:32.374453+0000","last_became_active":"2024-08-17T18:05:57.686609+0000","last_became_peered":"2024-08-17T18:05:57.686609+0000","last_unstale":"2024-08-17T18:06:32.374453+0000","last_undegraded":"2024-08-17T18:06:32.374453+0000","last_fullsized":"2024-08-17T18:06:32.374453+0000","mapping_epoch":1481,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:46:14.769622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":34,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:22.331471+0000","last_change":"2024-08-17T18:04:43.771720+0000","last_active":"2024-08-17T18:06:22.331471+0000","last_peered":"2024-08-17T18:06:22.331471+0000","last_clean":"2024-08-17T18:06:22.331471+0000","last_became_active":"2024-08-17T18:04:43.771137+0000","last_became_peered":"2024-08-17T18:04:43.771137+0000","last_unstale":"2024-08-17T18:06:22.331471+0000","last_undegraded":"2024-08-17T18:06:22.331471+0000","last_fullsized":"2024-08-17T18:06:22.331471+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:57.647909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":69,"reported_epoch":1487,"state":"active+clean","last_fresh":"2024-08-17T18:06:22.334492+0000","last_change":"2024-08-17T18:03:51.283872+0000","last_active":"2024-08-17T18:06:22.334492+0000","last_peered":"2024-08-17T18:06:22.334492+0000","last_clean":"2024-08-17T18:06:22.334492+0000","last_became_active":"2024-08-17T18:03:51.267197+0000","last_became_peered":"2024-08-17T18:03:51.267197+0000","last_unstale":"2024-08-17T18:06:22.334492+0000","last_undegraded":"2024-08-17T18:06:22.334492+0000","last_fullsized":"2024-08-17T18:06:22.334492+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:18:59.157786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2263,"reported_epoch":1488,"state":"active+clean","last_fresh":"2024-08-17T18:06:32.340867+0000","last_change":"2024-08-17T18:03:51.277491+0000","last_active":"2024-08-17T18:06:32.340867+0000","last_peered":"2024-08-17T18:06:32.340867+0000","last_clean":"2024-08-17T18:06:32.340867+0000","last_became_active":"2024-08-17T18:03:51.277178+0000","last_became_peered":"2024-08-17T18:03:51.277178+0000","last_unstale":"2024-08-17T18:06:32.340867+0000","last_undegraded":"2024-08-17T18:06:32.340867+0000","last_fullsized":"2024-08-17T18:06:32.340867+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:21:35.319085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":55,"num_pg":26,"stat_sum":{"num_bytes":4533190680,"num_objects":69172,"num_object_clones":0,"num_object_copies":207516,"num_objects_missing_on_primary":5474,"num_objects_missing":5474,"num_objects_degraded":37142,"num_objects_misplaced":9637,"num_objects_unfound":0,"num_objects_dirty":69172,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114796,"num_write_kb":5886913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23894,"num_bytes_recovered":1565917184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15750082560,"data_stored":15750070344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":164243,"ondisk_log_size":164243,"up":78,"acting":68,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":7412,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234688,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496326,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3893476,"kb_used_data":3186460,"kb_used_omap":1,"kb_used_meta":707006,"kb_avail":89851676,"statfs":{"total":95995035648,"available":92008116224,"internally_reserved":0,"allocated":3262935040,"data_stored":3257799263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":723974599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565567,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3950732,"kb_used_data":3210564,"kb_used_omap":72,"kb_used_meta":740087,"kb_avail":89794420,"statfs":{"total":95995035648,"available":91949486080,"internally_reserved":0,"allocated":3287617536,"data_stored":3282500502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":757849582},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":1399,"seq":6008659247169,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3536148,"kb_used_data":3327692,"kb_used_omap":131,"kb_used_meta":208316,"kb_avail":90209004,"statfs":{"total":95995035648,"available":92374020096,"internally_reserved":0,"allocated":3407556608,"data_stored":3402436012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134291,"internal_metadata":213316461},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743186957,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2638160,"kb_used_data":2032776,"kb_used_omap":66,"kb_used_meta":605309,"kb_avail":91106992,"statfs":{"total":95995035648,"available":93293559808,"internally_reserved":0,"allocated":2081562624,"data_stored":2076441919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":619837099},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448596,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1672472,"kb_used_data":1261584,"kb_used_omap":75,"kb_used_meta":410804,"kb_avail":92072680,"statfs":{"total":95995035648,"available":94282424320,"internally_reserved":0,"allocated":1291862016,"data_stored":1286733796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":420663487},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757960,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1707220,"kb_used_data":1084620,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":92037932,"statfs":{"total":95995035648,"available":94246842368,"internally_reserved":0,"allocated":1110650880,"data_stored":1105526177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":637532614},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,0,1,0,1,3,2,8],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019598,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1998112,"kb_used_data":1385432,"kb_used_omap":1,"kb_used_meta":612670,"kb_avail":91747040,"statfs":{"total":95995035648,"available":93948968960,"internally_reserved":0,"allocated":1418682368,"data_stored":1413549608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":627374534},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1276383232,"data_stored":1276379160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2068905984,"data_stored":2068905984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3387359232,"data_stored":3387359232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3269459968,"data_stored":3269459968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1404637184,"data_stored":1404633112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1098977280,"data_stored":1098973208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3244359680,"data_stored":3244359680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":55,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T18:06:45.650 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-17T18:06:45.650 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-08-17T18:06:45.662 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:45.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:45.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:45.725 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:06:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:45 smithi165 ceph-mon[105517]: pgmap v2666: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 275 op/s; 36590/206436 objects degraded (17.725%); 9637/206436 objects misplaced (4.668%); 9.2 MiB/s, 146 objects/s recovering 2024-08-17T18:06:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:45 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:06:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:45 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2952356274' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:06:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:45 smithi165 ceph-mon[105517]: osdmap e1489: 8 total, 8 up, 7 in 2024-08-17T18:06:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:45 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 36590/206436 objects degraded (17.725%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:45.763 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:45.786 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:45.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:45.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:45.887 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:45.909 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:45.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:45.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:46.011 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:46.039 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:46.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:46.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:46.141 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:46.162 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:46.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:46.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:46.264 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:46.286 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:46.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:46.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:46.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:46.414 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:46.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:46.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:46.516 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:46.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:46.548 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:46.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:46.649 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:46.672 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:46.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:46.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:46.774 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:46.797 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:46.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:46.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:46.898 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:46.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:46.927 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:46.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:47.028 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:47.053 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:47.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:47.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:47.155 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:47.179 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:47.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:47.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:47.241 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:06:47.242 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:06:47.258 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:06:47.259 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:06:47.278 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:06:47.278 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:06:47.280 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:47.299 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:06:47.299 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:06:47.305 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:47.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:47.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:47.320 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:06:47.320 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:06:47.338 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:06:47.338 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:06:47.357 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:06:47.357 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:06:47.381 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:06:47.381 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:06:47.381 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:47 smithi136 ceph-mon[109861]: pgmap v2668: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 295 op/s; 36590/206436 objects degraded (17.725%); 9637/206436 objects misplaced (4.668%); 9.4 MiB/s, 149 objects/s recovering 2024-08-17T18:06:47.381 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:47 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1727286349' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:06:47.381 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:47 smithi136 ceph-mon[115078]: pgmap v2668: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 295 op/s; 36590/206436 objects degraded (17.725%); 9637/206436 objects misplaced (4.668%); 9.4 MiB/s, 149 objects/s recovering 2024-08-17T18:06:47.381 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:47 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1727286349' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:06:47.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:47.430 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:47.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:47.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:47.532 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:47.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:47.559 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:47.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:47.577 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:06:47.661 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:47.686 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:47.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:47.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:47.694 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:06:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:06:47] "GET /metrics HTTP/1.1" 200 36690 "" "Prometheus/2.51.0" 2024-08-17T18:06:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:47 smithi165 ceph-mon[105517]: pgmap v2668: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 295 op/s; 36590/206436 objects degraded (17.725%); 9637/206436 objects misplaced (4.668%); 9.4 MiB/s, 149 objects/s recovering 2024-08-17T18:06:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:47 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1727286349' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:06:47.788 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:47.818 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:47.819 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:06:47.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:47.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:47.921 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:47.948 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:47.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:47.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:48.050 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:48.079 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:48.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:48.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:48.180 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:48.203 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:48.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:48.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:48.305 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:48.331 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:48.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:48.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:48.433 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:48.455 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:48.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:48.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:48.470 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:06:48.557 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:48.584 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:48.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:48.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:48.685 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:48.707 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:48.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:48.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:48.809 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:48.831 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:48.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:48.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:48.933 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:48.957 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:48.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:48.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:49.059 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:49.090 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:49.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:49.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:49.191 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:49.214 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:49.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:49.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:49.315 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:49.340 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:49.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:49.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:49.442 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:49.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:49 smithi136 ceph-mon[109861]: pgmap v2669: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 1 active+clean+scrubbing, 9 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 316 op/s; 35876/201471 objects degraded (17.807%); 8920/201471 objects misplaced (4.427%); 11 MiB/s, 168 objects/s recovering 2024-08-17T18:06:49.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:49 smithi136 ceph-mon[115078]: pgmap v2669: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 1 active+clean+scrubbing, 9 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 316 op/s; 35876/201471 objects degraded (17.807%); 8920/201471 objects misplaced (4.427%); 11 MiB/s, 168 objects/s recovering 2024-08-17T18:06:49.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:49.474 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:49.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:49.575 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:49.597 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:49.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:49.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:49.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:49.723 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:49.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:49.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:49 smithi165 ceph-mon[105517]: pgmap v2669: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 1 active+clean+scrubbing, 9 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 316 op/s; 35876/201471 objects degraded (17.807%); 8920/201471 objects misplaced (4.427%); 11 MiB/s, 168 objects/s recovering 2024-08-17T18:06:49.824 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:49.848 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:49.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:49.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:49.949 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:49.988 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:49.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:49.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:50.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:50.113 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:50.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:50.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:50.215 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:50.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:50.238 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:50.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:50.340 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:50.361 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:50.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:50.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:50.463 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:50.492 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:50.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:50.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:50.587 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:50 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 35876/201471 objects degraded (17.807%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:50.587 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:50 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 35876/201471 objects degraded (17.807%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:50.594 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:50.624 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:50.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:50.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:50.725 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:50.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:50.749 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:50.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:50.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:50 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 35876/201471 objects degraded (17.807%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:50.850 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:50.872 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:50.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:50.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:50.974 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:50.997 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:50.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:50.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:51.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:51.132 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:51.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:51.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:51.233 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:51.261 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:51.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:51.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:51.351 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:06:51.351 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:06:51.363 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:51.396 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:51.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:51.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:51.498 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:51.520 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:51.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:51.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:51.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:51.648 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:51.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:51.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:51 smithi136 ceph-mon[109861]: pgmap v2670: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 1 active+clean+scrubbing, 9 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 316 op/s; 35876/201471 objects degraded (17.807%); 8920/201471 objects misplaced (4.427%); 11 MiB/s, 168 objects/s recovering 2024-08-17T18:06:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:06:51.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:51 smithi136 ceph-mon[115078]: pgmap v2670: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 1 active+clean+scrubbing, 9 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 316 op/s; 35876/201471 objects degraded (17.807%); 8920/201471 objects misplaced (4.427%); 11 MiB/s, 168 objects/s recovering 2024-08-17T18:06:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:06:51.750 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:51 smithi165 ceph-mon[105517]: pgmap v2670: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 1 active+clean+scrubbing, 9 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 316 op/s; 35876/201471 objects degraded (17.807%); 8920/201471 objects misplaced (4.427%); 11 MiB/s, 168 objects/s recovering 2024-08-17T18:06:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:06:51.773 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:51.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:51.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:51.874 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:51.905 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:51.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:51.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:52.007 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:52.030 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:52.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:52.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:52.118 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:06:52.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:52.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:52.165 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:52.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:52.266 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:52.287 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:52.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:52.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:52.389 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:52.427 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:52.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:52.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:52.529 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:52.552 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:52.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:52.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:52.653 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:52.677 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:52.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:52.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:52.778 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:52.808 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:52.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:52.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:52.909 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:52.934 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:52.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:52.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:53.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-17T18:06:53.036 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:53.067 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:53.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:53.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:53.169 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:53.191 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:53.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:53.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:53.293 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:53.322 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:53.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:53.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:53.424 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:53.447 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:53.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:53.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:53.495 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:53 smithi136 ceph-mon[109861]: pgmap v2671: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 321 op/s; 35777/197937 objects degraded (18.075%); 8099/197937 objects misplaced (4.092%); 12 MiB/s, 190 objects/s recovering 2024-08-17T18:06:53.495 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:53 smithi136 ceph-mon[115078]: pgmap v2671: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 321 op/s; 35777/197937 objects degraded (18.075%); 8099/197937 objects misplaced (4.092%); 12 MiB/s, 190 objects/s recovering 2024-08-17T18:06:53.550 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:53.581 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:53.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:53.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:53.683 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:53.707 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:53.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:53.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:53 smithi165 ceph-mon[105517]: pgmap v2671: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 321 op/s; 35777/197937 objects degraded (18.075%); 8099/197937 objects misplaced (4.092%); 12 MiB/s, 190 objects/s recovering 2024-08-17T18:06:53.810 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:53.834 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:53.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:53.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:53.936 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:53.963 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:53.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:53.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:54.065 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:54.095 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:54.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:54.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:54.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:54.222 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:54.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:54.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:54.324 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:54.353 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:54.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:54.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:54.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:54 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1404896834' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-08-17T18:06:54.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:54 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-08-17T18:06:54.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:54 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1404896834' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-08-17T18:06:54.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:54 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-08-17T18:06:54.455 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:54.484 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:54.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:54.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:54.525 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:06:54.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:54.614 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:54.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:54.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:54.716 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:54.742 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:54.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:54.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:54.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:54 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1404896834' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-08-17T18:06:54.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:54 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-08-17T18:06:54.843 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:54.870 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:54.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:54.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:54.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:06:54.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:54.997 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:54.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:06:55.098 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:55.119 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:55.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:55.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:55.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:55.246 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:55.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:55.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:55.348 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:55.370 INFO:teuthology.orchestra.run.smithi136.stderr:set pool 1 pg_num to 11 2024-08-17T18:06:55.374 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:55.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:55.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:55.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:55 smithi136 ceph-mon[109861]: pgmap v2672: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 327 op/s; 35110/196611 objects degraded (17.858%); 8099/196611 objects misplaced (4.119%); 12 MiB/s, 196 objects/s recovering 2024-08-17T18:06:55.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:55 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-08-17T18:06:55.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:55 smithi136 ceph-mon[109861]: osdmap e1490: 8 total, 8 up, 7 in 2024-08-17T18:06:55.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:55 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 35110/196611 objects degraded (17.858%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:55.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:55 smithi136 ceph-mon[115078]: pgmap v2672: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 327 op/s; 35110/196611 objects degraded (17.858%); 8099/196611 objects misplaced (4.119%); 12 MiB/s, 196 objects/s recovering 2024-08-17T18:06:55.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:55 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-08-17T18:06:55.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:55 smithi136 ceph-mon[115078]: osdmap e1490: 8 total, 8 up, 7 in 2024-08-17T18:06:55.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:55 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 35110/196611 objects degraded (17.858%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:55.476 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:55.498 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:55.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:55.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:55.599 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:55.623 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:55.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:55.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:55.725 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:55.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:55.746 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:55.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:55 smithi165 ceph-mon[105517]: pgmap v2672: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 327 op/s; 35110/196611 objects degraded (17.858%); 8099/196611 objects misplaced (4.119%); 12 MiB/s, 196 objects/s recovering 2024-08-17T18:06:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:55 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-08-17T18:06:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:55 smithi165 ceph-mon[105517]: osdmap e1490: 8 total, 8 up, 7 in 2024-08-17T18:06:55.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:55 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 35110/196611 objects degraded (17.858%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:06:55.847 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:55.875 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:55.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:55.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:55.977 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:56.007 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:56.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:56.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:56.109 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:56.171 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:56.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:56.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:56.273 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:56.298 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:56.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:56.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:56.341 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:06:56.400 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:56.430 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:56.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:56.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:56.532 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:56.564 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:56.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:56.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:56.666 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:56.687 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:56.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:56.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:56.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:56 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2651001428' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:06:56.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:56 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-08-17T18:06:56.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:56 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2651001428' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:06:56.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:56 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-08-17T18:06:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:56 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2651001428' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:06:56.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:56 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-08-17T18:06:56.789 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:56.814 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:56.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:56.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:56.916 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:56.943 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:56.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:56.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:57.045 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:57.069 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:57.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:57.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:57.171 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:57.194 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:57.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:57.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:57.296 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:57.323 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:57.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:57.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:57.379 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:57 smithi136 ceph-mon[109861]: pgmap v2674: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 327 op/s; 35110/196611 objects degraded (17.858%); 8099/196611 objects misplaced (4.119%); 12 MiB/s, 196 objects/s recovering 2024-08-17T18:06:57.379 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:57 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:06:57.379 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:57 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2651001428' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:06:57.379 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:57 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-08-17T18:06:57.380 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:57 smithi136 ceph-mon[109861]: osdmap e1491: 8 total, 8 up, 7 in 2024-08-17T18:06:57.380 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:57 smithi136 ceph-mon[115078]: pgmap v2674: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 327 op/s; 35110/196611 objects degraded (17.858%); 8099/196611 objects misplaced (4.119%); 12 MiB/s, 196 objects/s recovering 2024-08-17T18:06:57.380 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:57 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:06:57.380 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:57 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2651001428' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:06:57.380 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:57 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-08-17T18:06:57.380 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:57 smithi136 ceph-mon[115078]: osdmap e1491: 8 total, 8 up, 7 in 2024-08-17T18:06:57.426 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:57.449 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:57.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:57.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:57.551 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:57.576 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:57.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:57.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:57.645 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:06:57.646 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:06:57.679 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:06:57.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:06:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:06:57] "GET /metrics HTTP/1.1" 200 36694 "" "Prometheus/2.51.0" 2024-08-17T18:06:57.706 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:57.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:57.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:06:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:57 smithi165 ceph-mon[105517]: pgmap v2674: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+degraded, 9 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 327 op/s; 35110/196611 objects degraded (17.858%); 8099/196611 objects misplaced (4.119%); 12 MiB/s, 196 objects/s recovering 2024-08-17T18:06:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:57 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:06:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:57 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2651001428' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:06:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:57 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-08-17T18:06:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:57 smithi165 ceph-mon[105517]: osdmap e1491: 8 total, 8 up, 7 in 2024-08-17T18:06:57.809 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:57.830 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:57.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:57.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:57.932 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:57.960 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:57.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:57.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:58.062 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:58.087 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:58.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:58.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:58.189 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:06:58.214 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:58.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:58.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:06:58.316 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:58.339 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:58.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:58.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:58.440 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:58.475 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:58.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:58.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:58.576 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:58.600 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:58.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:58.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:58.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:58 smithi136 ceph-mon[109861]: osdmap e1492: 8 total, 8 up, 7 in 2024-08-17T18:06:58.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:58 smithi136 ceph-mon[115078]: osdmap e1492: 8 total, 8 up, 7 in 2024-08-17T18:06:58.702 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:58.729 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:58.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:58.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:58 smithi165 ceph-mon[105517]: osdmap e1492: 8 total, 8 up, 7 in 2024-08-17T18:06:58.830 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:58.854 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:58.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:58.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:58.956 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:58.982 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:58.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:58.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:59.084 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:59.109 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:59.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:59.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:59.133 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:06:59.211 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:06:59.232 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:59.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:59.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:06:59.334 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:59.368 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:59.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:59.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:59.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:59 smithi136 ceph-mon[109861]: pgmap v2677: 109 pgs: 1 peering, 10 unknown, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 1 creating+peering, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 82 active+clean; 3.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 363 op/s; 31903/191400 objects degraded (16.668%); 9352/191400 objects misplaced (4.886%); 8.8 MiB/s, 292 objects/s recovering 2024-08-17T18:06:59.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:59 smithi136 ceph-mon[109861]: osdmap e1493: 8 total, 8 up, 7 in 2024-08-17T18:06:59.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:59 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:06:59.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:06:59 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:06:59.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:59 smithi136 ceph-mon[115078]: pgmap v2677: 109 pgs: 1 peering, 10 unknown, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 1 creating+peering, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 82 active+clean; 3.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 363 op/s; 31903/191400 objects degraded (16.668%); 9352/191400 objects misplaced (4.886%); 8.8 MiB/s, 292 objects/s recovering 2024-08-17T18:06:59.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:59 smithi136 ceph-mon[115078]: osdmap e1493: 8 total, 8 up, 7 in 2024-08-17T18:06:59.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:59 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:06:59.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:06:59 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:06:59.470 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:06:59.496 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:59.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:59.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:06:59.598 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:06:59.625 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:06:59.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:59.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:06:59.727 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:06:59.755 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:59.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:59.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:06:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:59 smithi165 ceph-mon[105517]: pgmap v2677: 109 pgs: 1 peering, 10 unknown, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 1 creating+peering, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 82 active+clean; 3.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 363 op/s; 31903/191400 objects degraded (16.668%); 9352/191400 objects misplaced (4.886%); 8.8 MiB/s, 292 objects/s recovering 2024-08-17T18:06:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:59 smithi165 ceph-mon[105517]: osdmap e1493: 8 total, 8 up, 7 in 2024-08-17T18:06:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:59 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:06:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:06:59 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:06:59.856 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:06:59.881 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:06:59.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:06:59.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:06:59.983 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:00.013 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:00.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:00.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:00.115 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:00.146 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:00.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:00.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:00.247 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:00.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:00.270 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:00.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:00.371 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:00.393 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:00.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:00.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:00.495 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:00.524 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:00.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:00.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:00.626 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:00.654 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:00.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:00.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:00.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:00 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 31903/191400 objects degraded (16.668%), 12 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:07:00.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:00 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 31903/191400 objects degraded (16.668%), 12 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:07:00.756 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:00.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:00 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 31903/191400 objects degraded (16.668%), 12 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:07:00.779 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:00.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:00.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:00.881 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:00.906 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:00.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:00.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:01.008 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:01.035 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:01.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:01.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:01.111 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 0, 5] out_osds: [7] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:07:01.111 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:07:01.111 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-08-17T18:07:01.112 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-17T18:07:01.112 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-17T18:07:01.112 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-08-17T18:07:01.137 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:01.160 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:01.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:01.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:01.262 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:01.294 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:01.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:01.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:01.395 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:01.418 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:01.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:01.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:01.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:01 smithi136 ceph-mon[109861]: pgmap v2679: 109 pgs: 1 peering, 10 unknown, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 1 creating+peering, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 82 active+clean; 3.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 293 op/s; 31903/191400 objects degraded (16.668%); 9352/191400 objects misplaced (4.886%); 5.3 MiB/s, 239 objects/s recovering 2024-08-17T18:07:01.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:01 smithi136 ceph-mon[115078]: pgmap v2679: 109 pgs: 1 peering, 10 unknown, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 1 creating+peering, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 82 active+clean; 3.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 293 op/s; 31903/191400 objects degraded (16.668%); 9352/191400 objects misplaced (4.886%); 5.3 MiB/s, 239 objects/s recovering 2024-08-17T18:07:01.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:01.544 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:01.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:01.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:01.646 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:01.670 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:01.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:01.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:01 smithi165 ceph-mon[105517]: pgmap v2679: 109 pgs: 1 peering, 10 unknown, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 1 creating+peering, 9 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 82 active+clean; 3.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 293 op/s; 31903/191400 objects degraded (16.668%); 9352/191400 objects misplaced (4.886%); 5.3 MiB/s, 239 objects/s recovering 2024-08-17T18:07:01.772 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:01.799 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:01.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:01.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:01.901 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:01.927 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:01.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:01.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:02.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:02.051 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:02.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:02.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:02.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:02.181 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:02.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:02.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:02.282 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:02.307 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:02.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:02.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:02.409 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:02.441 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:02.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:02.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:02.543 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:02.577 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:02.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:02.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:02.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:02.706 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:02.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:02.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:02.808 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:02.832 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:02.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:02.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:02.934 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:02.955 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:02.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:02.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:03.057 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:03.081 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:03.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:03.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:03.183 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:03.209 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:03.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:03.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:03.310 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:03.345 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:03.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:03.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:03.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:03 smithi136 ceph-mon[109861]: pgmap v2680: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 3 active+clean+scrubbing, 1 creating+peering, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 90 active+clean; 3.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 477 op/s; 29415/188010 objects degraded (15.645%); 10667/188010 objects misplaced (5.674%); 11 MiB/s, 335 objects/s recovering 2024-08-17T18:07:03.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:03 smithi136 ceph-mon[115078]: pgmap v2680: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 3 active+clean+scrubbing, 1 creating+peering, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 90 active+clean; 3.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 477 op/s; 29415/188010 objects degraded (15.645%); 10667/188010 objects misplaced (5.674%); 11 MiB/s, 335 objects/s recovering 2024-08-17T18:07:03.447 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:03.471 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:03.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:03.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:03.573 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:03.599 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:03.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:03.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:03.701 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:03.728 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:03.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:03.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:03 smithi165 ceph-mon[105517]: pgmap v2680: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 3 active+clean+scrubbing, 1 creating+peering, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 90 active+clean; 3.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 477 op/s; 29415/188010 objects degraded (15.645%); 10667/188010 objects misplaced (5.674%); 11 MiB/s, 335 objects/s recovering 2024-08-17T18:07:03.829 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:03.859 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:03.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:03.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:03.960 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:03.987 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:03.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:03.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:04.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:04.115 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:04.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:04.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:04.146 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:07:04.217 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:04.242 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:04.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:04.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:04.345 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:04.368 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:04.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:04.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:04.470 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:04.501 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:04.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:04.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:04.603 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:04.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:04.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:04.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:04.731 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:04.760 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:04.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:04.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:04.862 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:04.888 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:04.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:04.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:04.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:05.019 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:05.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:05.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:05.121 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:05.152 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:05.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:05.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:05.254 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:05.282 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:05.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:05.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:05.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:05.409 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:05.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:05.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:05.511 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:05.535 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:05.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:05.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:05.637 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:05.662 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:05.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:05.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:05 smithi136 ceph-mon[109861]: pgmap v2681: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 3 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 3.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 434 op/s; 28829/186291 objects degraded (15.475%); 10550/186291 objects misplaced (5.663%); 11 MiB/s, 298 objects/s recovering 2024-08-17T18:07:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:05 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 28829/186291 objects degraded (15.475%), 12 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:07:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:07:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:07:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:05 smithi136 ceph-mon[115078]: pgmap v2681: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 3 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 3.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 434 op/s; 28829/186291 objects degraded (15.475%); 10550/186291 objects misplaced (5.663%); 11 MiB/s, 298 objects/s recovering 2024-08-17T18:07:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:05 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 28829/186291 objects degraded (15.475%), 12 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:07:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:07:05.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:07:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:05 smithi165 ceph-mon[105517]: pgmap v2681: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 3 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 3.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 434 op/s; 28829/186291 objects degraded (15.475%); 10550/186291 objects misplaced (5.663%); 11 MiB/s, 298 objects/s recovering 2024-08-17T18:07:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:05 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 28829/186291 objects degraded (15.475%), 12 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:07:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:07:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:07:05.764 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:05.786 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:05.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:05.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:05.888 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:05.888 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:07:05.889 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:07:05.914 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:05.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:05.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:05.919 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:07:05.919 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:07:05.945 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:07:05.946 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:07:05.967 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:07:05.967 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:07:05.990 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:07:05.991 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:07:06.013 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:07:06.013 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:07:06.016 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:06.035 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:07:06.036 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:07:06.042 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:06.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:06.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:06.058 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:07:06.058 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:07:06.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:06.175 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:06.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:06.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:06.277 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:06.305 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:06.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:06.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:06.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:06.429 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:06.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:06.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:06.531 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:06.562 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:06.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:06.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:06.609 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:07:06.609 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-17T18:07:06.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-17T18:07:06.609 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:07:06.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.609 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.45626.0:39670 55.8 55.d367ed68 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e1457)", 2024-08-17T18:07:06.609 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:04:14.339556+0000", 2024-08-17T18:07:06.609 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 172.267720137, 2024-08-17T18:07:06.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18556566499999999, 2024-08-17T18:07:06.610 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:07:06.610 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:07:06.610 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:07:06.610 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:07:06.610 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.45626", 2024-08-17T18:07:06.610 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:07:06.610 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 39670 2024-08-17T18:07:06.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.610 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:07:06.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.610 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:07:06.610 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.339556+0000", 2024-08-17T18:07:06.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:07:06.611 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.339556+0000", 2024-08-17T18:07:06.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:07:06.611 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.339558+0000", 2024-08-17T18:07:06.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6080000000000001e-06 2024-08-17T18:07:06.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:07:06.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.339605+0000", 2024-08-17T18:07:06.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7055999999999998e-05 2024-08-17T18:07:06.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:07:06.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.339621+0000", 2024-08-17T18:07:06.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6192999999999999e-05 2024-08-17T18:07:06.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:07:06.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.339630+0000", 2024-08-17T18:07:06.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.833e-06 2024-08-17T18:07:06.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.613 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.613 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.343620+0000", 2024-08-17T18:07:06.613 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039901529999999998 2024-08-17T18:07:06.613 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.613 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:07:06.613 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.343629+0000", 2024-08-17T18:07:06.613 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8449999999999992e-06 2024-08-17T18:07:06.613 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.613 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.613 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.525111+0000", 2024-08-17T18:07:06.613 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.181482634 2024-08-17T18:07:06.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:07:06.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.525122+0000", 2024-08-17T18:07:06.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0343000000000001e-05 2024-08-17T18:07:06.614 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.614 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:07:06.614 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.614 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.45626.0:39672 55.8 55.2a64ef28 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e1457)", 2024-08-17T18:07:06.614 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:04:14.339702+0000", 2024-08-17T18:07:06.614 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 172.26757379099999, 2024-08-17T18:07:06.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18545436600000001, 2024-08-17T18:07:06.615 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:07:06.615 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:07:06.615 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:07:06.615 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:07:06.615 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.45626", 2024-08-17T18:07:06.615 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:07:06.615 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 39672 2024-08-17T18:07:06.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.615 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:07:06.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:07:06.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.339702+0000", 2024-08-17T18:07:06.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:07:06.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.339702+0000", 2024-08-17T18:07:06.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:07:06.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.339703+0000", 2024-08-17T18:07:06.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4000000000000001e-07 2024-08-17T18:07:06.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:07:06.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.339725+0000", 2024-08-17T18:07:06.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1937e-05 2024-08-17T18:07:06.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:07:06.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.339734+0000", 2024-08-17T18:07:06.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5820000000000002e-06 2024-08-17T18:07:06.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:07:06.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.339740+0000", 2024-08-17T18:07:06.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3890000000000002e-06 2024-08-17T18:07:06.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.343650+0000", 2024-08-17T18:07:06.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039103100000000002 2024-08-17T18:07:06.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:07:06.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.343656+0000", 2024-08-17T18:07:06.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4829999999999999e-06 2024-08-17T18:07:06.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.525149+0000", 2024-08-17T18:07:06.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18149356 2024-08-17T18:07:06.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.619 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.619 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:07:06.619 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.525157+0000", 2024-08-17T18:07:06.619 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4649999999999997e-06 2024-08-17T18:07:06.619 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.619 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:07:06.619 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.619 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.619 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.619 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.45626.0:39680 55.8 55.8532c638 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e1457)", 2024-08-17T18:07:06.619 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:04:14.339797+0000", 2024-08-17T18:07:06.619 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 172.26747906599999, 2024-08-17T18:07:06.619 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18539096799999999, 2024-08-17T18:07:06.620 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:07:06.620 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:07:06.620 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:07:06.620 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:07:06.620 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.45626", 2024-08-17T18:07:06.620 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:07:06.620 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 39680 2024-08-17T18:07:06.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.620 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:07:06.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.620 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:07:06.620 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.339797+0000", 2024-08-17T18:07:06.620 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.621 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:07:06.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.339797+0000", 2024-08-17T18:07:06.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.621 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:07:06.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.339798+0000", 2024-08-17T18:07:06.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.75e-07 2024-08-17T18:07:06.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.621 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:07:06.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.339822+0000", 2024-08-17T18:07:06.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4698999999999998e-05 2024-08-17T18:07:06.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:07:06.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.339830+0000", 2024-08-17T18:07:06.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7300000000000005e-06 2024-08-17T18:07:06.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:07:06.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.339836+0000", 2024-08-17T18:07:06.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8860000000000001e-06 2024-08-17T18:07:06.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.623 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.623 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.623 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.343676+0000", 2024-08-17T18:07:06.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038402919999999999 2024-08-17T18:07:06.623 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.623 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.623 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:07:06.623 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.343682+0000", 2024-08-17T18:07:06.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2750000000000003e-06 2024-08-17T18:07:06.623 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.623 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.623 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.623 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.525181+0000", 2024-08-17T18:07:06.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18149923500000001 2024-08-17T18:07:06.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.624 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:07:06.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.525188+0000", 2024-08-17T18:07:06.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2760000000000003e-06 2024-08-17T18:07:06.624 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.624 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:07:06.625 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.625 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.45626.0:39697 55.8 55.1bb19358 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1457)", 2024-08-17T18:07:06.625 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:04:14.367812+0000", 2024-08-17T18:07:06.625 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 172.239463828, 2024-08-17T18:07:06.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15740602000000001, 2024-08-17T18:07:06.625 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:07:06.625 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:07:06.625 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:07:06.625 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:07:06.625 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.45626", 2024-08-17T18:07:06.625 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:07:06.625 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 39697 2024-08-17T18:07:06.626 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.626 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:07:06.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.626 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:07:06.626 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.367812+0000", 2024-08-17T18:07:06.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.626 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.626 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:07:06.626 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.367812+0000", 2024-08-17T18:07:06.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.626 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:07:06.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.367825+0000", 2024-08-17T18:07:06.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2069000000000001e-05 2024-08-17T18:07:06.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:07:06.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.367852+0000", 2024-08-17T18:07:06.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7047999999999999e-05 2024-08-17T18:07:06.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:07:06.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.367874+0000", 2024-08-17T18:07:06.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2269e-05 2024-08-17T18:07:06.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:07:06.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.367883+0000", 2024-08-17T18:07:06.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2459999999999992e-06 2024-08-17T18:07:06.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.367947+0000", 2024-08-17T18:07:06.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4041999999999994e-05 2024-08-17T18:07:06.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:07:06.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.367956+0000", 2024-08-17T18:07:06.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7730000000000005e-06 2024-08-17T18:07:06.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.525211+0000", 2024-08-17T18:07:06.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15725545299999999 2024-08-17T18:07:06.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:07:06.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.525218+0000", 2024-08-17T18:07:06.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1199999999999996e-06 2024-08-17T18:07:06.629 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.629 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:07:06.629 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.630 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.45626.0:39706 55.8 55.17a587e8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1457)", 2024-08-17T18:07:06.630 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:04:14.374686+0000", 2024-08-17T18:07:06.630 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 172.23259009700001, 2024-08-17T18:07:06.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.150639573, 2024-08-17T18:07:06.630 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:07:06.630 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:07:06.630 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:07:06.630 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:07:06.630 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.45626", 2024-08-17T18:07:06.630 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:07:06.630 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 39706 2024-08-17T18:07:06.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.631 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:07:06.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:07:06.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.374686+0000", 2024-08-17T18:07:06.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:07:06.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.374686+0000", 2024-08-17T18:07:06.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:07:06.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.374688+0000", 2024-08-17T18:07:06.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6309999999999999e-06 2024-08-17T18:07:06.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.632 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:07:06.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.374741+0000", 2024-08-17T18:07:06.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2936999999999998e-05 2024-08-17T18:07:06.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.632 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:07:06.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.374758+0000", 2024-08-17T18:07:06.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6888000000000002e-05 2024-08-17T18:07:06.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:07:06.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.374765+0000", 2024-08-17T18:07:06.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2690000000000002e-06 2024-08-17T18:07:06.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.374814+0000", 2024-08-17T18:07:06.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8804999999999998e-05 2024-08-17T18:07:06.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:07:06.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.374821+0000", 2024-08-17T18:07:06.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0199999999999997e-06 2024-08-17T18:07:06.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.525317+0000", 2024-08-17T18:07:06.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15049621099999999 2024-08-17T18:07:06.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:07:06.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:14.525326+0000", 2024-08-17T18:07:06.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8119999999999997e-06 2024-08-17T18:07:06.634 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.634 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:07:06.634 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.635 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.45626.0:54921 55.d e1462/1458 55:b986c78a:::benchmark_data_smithi136_313476_object54920:head v 1462'3483, mlcod=0'0)", 2024-08-17T18:07:06.635 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:04:31.056557+0000", 2024-08-17T18:07:06.635 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 155.55071927399999, 2024-08-17T18:07:06.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.046234578999999998, 2024-08-17T18:07:06.635 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:07:06.635 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:07:06.635 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:07:06.635 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:07:06.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:07:06.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:31.056557+0000", 2024-08-17T18:07:06.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:07:06.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:31.056557+0000", 2024-08-17T18:07:06.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:07:06.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:31.056558+0000", 2024-08-17T18:07:06.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.283e-06 2024-08-17T18:07:06.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:07:06.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:31.056643+0000", 2024-08-17T18:07:06.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4265999999999996e-05 2024-08-17T18:07:06.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:07:06.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:31.056657+0000", 2024-08-17T18:07:06.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4532000000000001e-05 2024-08-17T18:07:06.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:07:06.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:31.056664+0000", 2024-08-17T18:07:06.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3839999999999999e-06 2024-08-17T18:07:06.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:31.063362+0000", 2024-08-17T18:07:06.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0066972200000000003 2024-08-17T18:07:06.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:07:06.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:31.063389+0000", 2024-08-17T18:07:06.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7392000000000001e-05 2024-08-17T18:07:06.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:07:06.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:31.102617+0000", 2024-08-17T18:07:06.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039228088000000001 2024-08-17T18:07:06.639 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:06 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1245859904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:07:06.639 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:07:06.639 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:07:06.639 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:06 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1245859904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:07:06.639 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:07:06.639 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:07:06.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:07:06.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:31.102792+0000", 2024-08-17T18:07:06.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017441399999999999 2024-08-17T18:07:06.640 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.640 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:07:06.640 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.641 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.45626.0:54923 55.d e1462/1458 55:b7b84e14:::benchmark_data_smithi136_313476_object54922:head v 1462'3484, mlcod=0'0)", 2024-08-17T18:07:06.641 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:04:31.057865+0000", 2024-08-17T18:07:06.641 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 155.549410927, 2024-08-17T18:07:06.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.045115553000000003, 2024-08-17T18:07:06.641 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:07:06.641 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:07:06.641 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:07:06.641 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:07:06.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:07:06.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:31.057865+0000", 2024-08-17T18:07:06.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:07:06.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:31.057865+0000", 2024-08-17T18:07:06.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:07:06.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:31.057866+0000", 2024-08-17T18:07:06.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2999999999999999e-07 2024-08-17T18:07:06.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:07:06.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:31.057900+0000", 2024-08-17T18:07:06.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3847999999999998e-05 2024-08-17T18:07:06.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:07:06.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:31.057911+0000", 2024-08-17T18:07:06.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0947999999999999e-05 2024-08-17T18:07:06.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:07:06.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:31.057921+0000", 2024-08-17T18:07:06.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.04e-05 2024-08-17T18:07:06.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:31.068428+0000", 2024-08-17T18:07:06.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010506797 2024-08-17T18:07:06.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:07:06.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:31.068486+0000", 2024-08-17T18:07:06.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7754999999999998e-05 2024-08-17T18:07:06.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:07:06.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:31.102796+0000", 2024-08-17T18:07:06.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034310044999999997 2024-08-17T18:07:06.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:07:06.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:31.102981+0000", 2024-08-17T18:07:06.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018493000000000001 2024-08-17T18:07:06.645 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.645 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:07:06.645 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.645 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.45626.0:58409 55.5 e1463/1458 55:a95c6f08:::benchmark_data_smithi136_313476_object58408:head v 1463'3699, mlcod=0'0)", 2024-08-17T18:07:06.645 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:04:34.797620+0000", 2024-08-17T18:07:06.645 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 151.80965669599999, 2024-08-17T18:07:06.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.044907279000000001, 2024-08-17T18:07:06.645 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:07:06.646 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:07:06.646 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:07:06.646 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:07:06.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:07:06.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:34.797620+0000", 2024-08-17T18:07:06.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:07:06.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:34.797620+0000", 2024-08-17T18:07:06.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:07:06.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:34.797620+0000", 2024-08-17T18:07:06.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8200000000000001e-07 2024-08-17T18:07:06.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:07:06.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:34.797648+0000", 2024-08-17T18:07:06.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8371999999999999e-05 2024-08-17T18:07:06.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:07:06.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:34.797658+0000", 2024-08-17T18:07:06.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9550000000000004e-06 2024-08-17T18:07:06.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:07:06.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:34.797663+0000", 2024-08-17T18:07:06.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4390000000000002e-06 2024-08-17T18:07:06.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:34.816631+0000", 2024-08-17T18:07:06.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018968207000000001 2024-08-17T18:07:06.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:07:06.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:34.816676+0000", 2024-08-17T18:07:06.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4753000000000003e-05 2024-08-17T18:07:06.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:07:06.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:34.842491+0000", 2024-08-17T18:07:06.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025815106000000001 2024-08-17T18:07:06.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:07:06.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:34.842527+0000", 2024-08-17T18:07:06.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6065000000000001e-05 2024-08-17T18:07:06.649 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.650 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:07:06.650 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.650 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.45626.0:61664 55.d e1463/1458 55:b8a18626:::benchmark_data_smithi136_313476_object61663:head v 1463'3935, mlcod=0'0)", 2024-08-17T18:07:06.650 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:04:38.708645+0000", 2024-08-17T18:07:06.650 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 147.89863109699999, 2024-08-17T18:07:06.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.046909171, 2024-08-17T18:07:06.650 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:07:06.650 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:07:06.650 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:07:06.650 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:07:06.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:07:06.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:38.708645+0000", 2024-08-17T18:07:06.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:07:06.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:38.708645+0000", 2024-08-17T18:07:06.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:07:06.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:38.708647+0000", 2024-08-17T18:07:06.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.42e-06 2024-08-17T18:07:06.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:07:06.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:38.708782+0000", 2024-08-17T18:07:06.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013588399999999999 2024-08-17T18:07:06.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:07:06.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:38.708798+0000", 2024-08-17T18:07:06.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5271e-05 2024-08-17T18:07:06.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:07:06.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:38.708805+0000", 2024-08-17T18:07:06.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1690000000000003e-06 2024-08-17T18:07:06.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:38.709005+0000", 2024-08-17T18:07:06.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000199815 2024-08-17T18:07:06.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:07:06.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:38.709058+0000", 2024-08-17T18:07:06.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2911999999999998e-05 2024-08-17T18:07:06.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:07:06.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:38.755477+0000", 2024-08-17T18:07:06.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.046418902999999997 2024-08-17T18:07:06.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:07:06.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:38.755554+0000", 2024-08-17T18:07:06.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7797000000000001e-05 2024-08-17T18:07:06.654 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.654 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:07:06.654 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.655 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.45626.0:62822 55.1 e1463/1458 55:86fc378d:::benchmark_data_smithi136_313476_object62821:head v 1463'4038, mlcod=0'0)", 2024-08-17T18:07:06.655 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:04:39.964772+0000", 2024-08-17T18:07:06.655 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 146.64250475700001, 2024-08-17T18:07:06.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.046176754, 2024-08-17T18:07:06.655 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:07:06.655 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:07:06.655 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:07:06.655 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:07:06.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:07:06.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:39.964772+0000", 2024-08-17T18:07:06.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:07:06.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:39.964772+0000", 2024-08-17T18:07:06.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:07:06.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:39.964772+0000", 2024-08-17T18:07:06.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9900000000000002e-07 2024-08-17T18:07:06.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:07:06.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:39.964792+0000", 2024-08-17T18:07:06.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9962999999999998e-05 2024-08-17T18:07:06.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:07:06.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:39.964798+0000", 2024-08-17T18:07:06.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0240000000000003e-06 2024-08-17T18:07:06.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:07:06.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:39.964802+0000", 2024-08-17T18:07:06.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5499999999999996e-06 2024-08-17T18:07:06.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:39.985184+0000", 2024-08-17T18:07:06.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020381313000000002 2024-08-17T18:07:06.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:07:06.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:39.985234+0000", 2024-08-17T18:07:06.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0034000000000002e-05 2024-08-17T18:07:06.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:07:06.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:40.010905+0000", 2024-08-17T18:07:06.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025671097 2024-08-17T18:07:06.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:07:06.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:40.010948+0000", 2024-08-17T18:07:06.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3473999999999997e-05 2024-08-17T18:07:06.659 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.659 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:07:06.659 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.659 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.45626.0:63284 55.5 e1463/1458 55:af6b4e56:::benchmark_data_smithi136_313476_object63283:head v 1463'3995, mlcod=0'0)", 2024-08-17T18:07:06.659 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:04:40.478755+0000", 2024-08-17T18:07:06.660 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 146.12852090800001, 2024-08-17T18:07:06.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048521729, 2024-08-17T18:07:06.660 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:07:06.660 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:07:06.660 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:07:06.660 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:07:06.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:07:06.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:40.478755+0000", 2024-08-17T18:07:06.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:07:06.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:40.478755+0000", 2024-08-17T18:07:06.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:07:06.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:40.478756+0000", 2024-08-17T18:07:06.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0690000000000001e-06 2024-08-17T18:07:06.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:07:06.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:40.478791+0000", 2024-08-17T18:07:06.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4693999999999997e-05 2024-08-17T18:07:06.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:07:06.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:40.478805+0000", 2024-08-17T18:07:06.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3433e-05 2024-08-17T18:07:06.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:07:06.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:40.478812+0000", 2024-08-17T18:07:06.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1810000000000004e-06 2024-08-17T18:07:06.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:40.496527+0000", 2024-08-17T18:07:06.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017715418 2024-08-17T18:07:06.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:07:06.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:40.496557+0000", 2024-08-17T18:07:06.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0054999999999999e-05 2024-08-17T18:07:06.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:07:06.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:40.527199+0000", 2024-08-17T18:07:06.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030641806000000001 2024-08-17T18:07:06.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:07:06.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:40.527277+0000", 2024-08-17T18:07:06.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8072999999999996e-05 2024-08-17T18:07:06.664 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.664 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:07:06.664 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.664 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.45626.0:63568 55.5 e1463/1458 55:a8734f55:::benchmark_data_smithi136_313476_object63567:head v 1463'4014, mlcod=0'0)", 2024-08-17T18:07:06.664 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:04:40.814758+0000", 2024-08-17T18:07:06.664 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 145.79251823000001, 2024-08-17T18:07:06.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.045356094999999999, 2024-08-17T18:07:06.664 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:07:06.664 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:07:06.664 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:07:06.665 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:07:06.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:07:06.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:40.814758+0000", 2024-08-17T18:07:06.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:07:06.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:40.814758+0000", 2024-08-17T18:07:06.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:07:06.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:40.814758+0000", 2024-08-17T18:07:06.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0699999999999998e-07 2024-08-17T18:07:06.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:07:06.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:40.814793+0000", 2024-08-17T18:07:06.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4122999999999998e-05 2024-08-17T18:07:06.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:07:06.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:40.814800+0000", 2024-08-17T18:07:06.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.729e-06 2024-08-17T18:07:06.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:07:06.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:40.814806+0000", 2024-08-17T18:07:06.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6869999999999999e-06 2024-08-17T18:07:06.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:40.835879+0000", 2024-08-17T18:07:06.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.021072896000000001 2024-08-17T18:07:06.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:07:06.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:40.835905+0000", 2024-08-17T18:07:06.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5891e-05 2024-08-17T18:07:06.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:07:06.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:40.860052+0000", 2024-08-17T18:07:06.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.024146887999999998 2024-08-17T18:07:06.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:07:06.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:04:40.860114+0000", 2024-08-17T18:07:06.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2473999999999999e-05 2024-08-17T18:07:06.668 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.668 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:07:06.668 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.669 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.45626.0:79095 55.d e1467/1465 55:bbef6c2d:::benchmark_data_smithi136_313476_object79094:head v 1467'4978, mlcod=0'0)", 2024-08-17T18:07:06.669 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:05:02.626664+0000", 2024-08-17T18:07:06.669 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 123.980611983, 2024-08-17T18:07:06.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.066232447, 2024-08-17T18:07:06.669 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:07:06.669 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:07:06.669 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:07:06.670 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:07:06.670 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:07:06.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:07:06.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.626664+0000", 2024-08-17T18:07:06.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:07:06.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.626664+0000", 2024-08-17T18:07:06.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:07:06.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.626666+0000", 2024-08-17T18:07:06.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4130000000000001e-06 2024-08-17T18:07:06.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:07:06.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.626714+0000", 2024-08-17T18:07:06.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7991000000000002e-05 2024-08-17T18:07:06.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:07:06.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.626734+0000", 2024-08-17T18:07:06.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0526e-05 2024-08-17T18:07:06.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:07:06.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.626743+0000", 2024-08-17T18:07:06.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4069999999999993e-06 2024-08-17T18:07:06.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.626927+0000", 2024-08-17T18:07:06.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018432100000000001 2024-08-17T18:07:06.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:07:06.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.626979+0000", 2024-08-17T18:07:06.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1789000000000002e-05 2024-08-17T18:07:06.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:07:06.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.692709+0000", 2024-08-17T18:07:06.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.065730190999999993 2024-08-17T18:07:06.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:07:06.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.692897+0000", 2024-08-17T18:07:06.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018780899999999999 2024-08-17T18:07:06.674 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.674 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:07:06.674 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.674 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.45626.0:79099 55.d e1467/1465 55:bbb553f6:::benchmark_data_smithi136_313476_object79098:head v 1467'4979, mlcod=0'0)", 2024-08-17T18:07:06.674 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:05:02.638279+0000", 2024-08-17T18:07:06.674 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 123.968997419, 2024-08-17T18:07:06.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.066079867, 2024-08-17T18:07:06.674 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:07:06.674 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:07:06.675 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:07:06.675 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:07:06.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:07:06.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.638279+0000", 2024-08-17T18:07:06.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:07:06.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.638279+0000", 2024-08-17T18:07:06.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:07:06.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.638279+0000", 2024-08-17T18:07:06.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5999999999999999e-07 2024-08-17T18:07:06.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:07:06.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.638318+0000", 2024-08-17T18:07:06.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8869999999999999e-05 2024-08-17T18:07:06.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:07:06.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.638326+0000", 2024-08-17T18:07:06.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9039999999999992e-06 2024-08-17T18:07:06.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:07:06.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.638331+0000", 2024-08-17T18:07:06.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8709999999999996e-06 2024-08-17T18:07:06.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.643471+0000", 2024-08-17T18:07:06.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0051398440000000002 2024-08-17T18:07:06.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:07:06.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.643495+0000", 2024-08-17T18:07:06.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4638999999999999e-05 2024-08-17T18:07:06.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:07:06.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.704309+0000", 2024-08-17T18:07:06.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.060814013 2024-08-17T18:07:06.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:07:06.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.704359+0000", 2024-08-17T18:07:06.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9265999999999999e-05 2024-08-17T18:07:06.679 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.679 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:07:06.679 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.679 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.45626.0:79101 55.d e1467/1465 55:b161cf93:::benchmark_data_smithi136_313476_object79100:head v 1467'4980, mlcod=0'0)", 2024-08-17T18:07:06.679 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:05:02.638346+0000", 2024-08-17T18:07:06.679 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 123.968930181, 2024-08-17T18:07:06.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.066042322000000001, 2024-08-17T18:07:06.679 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:07:06.679 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:07:06.679 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:07:06.680 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:07:06.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:07:06.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.638346+0000", 2024-08-17T18:07:06.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:07:06.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.638346+0000", 2024-08-17T18:07:06.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:07:06.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.638346+0000", 2024-08-17T18:07:06.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5600000000000002e-07 2024-08-17T18:07:06.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:07:06.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.638387+0000", 2024-08-17T18:07:06.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0957000000000001e-05 2024-08-17T18:07:06.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:07:06.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.638394+0000", 2024-08-17T18:07:06.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5409999999999997e-06 2024-08-17T18:07:06.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:07:06.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.638398+0000", 2024-08-17T18:07:06.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.51e-06 2024-08-17T18:07:06.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.655030+0000", 2024-08-17T18:07:06.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016631459000000001 2024-08-17T18:07:06.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:07:06.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.655091+0000", 2024-08-17T18:07:06.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1285999999999996e-05 2024-08-17T18:07:06.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:07:06.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.704365+0000", 2024-08-17T18:07:06.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.049274279999999997 2024-08-17T18:07:06.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:07:06.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.704388+0000", 2024-08-17T18:07:06.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3033000000000001e-05 2024-08-17T18:07:06.683 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.683 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:07:06.683 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.684 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.45626.0:79108 55.d e1467/1465 55:b85c912f:::benchmark_data_smithi136_313476_object79107:head v 1467'4981, mlcod=0'0)", 2024-08-17T18:07:06.684 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:05:02.653422+0000", 2024-08-17T18:07:06.684 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 123.953853903, 2024-08-17T18:07:06.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056001507999999998, 2024-08-17T18:07:06.684 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:07:06.684 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:07:06.684 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:07:06.684 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:07:06.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:07:06.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.653422+0000", 2024-08-17T18:07:06.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:07:06.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.653422+0000", 2024-08-17T18:07:06.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:07:06.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.653423+0000", 2024-08-17T18:07:06.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8500000000000001e-07 2024-08-17T18:07:06.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:07:06.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.653459+0000", 2024-08-17T18:07:06.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6288000000000002e-05 2024-08-17T18:07:06.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:07:06.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.653474+0000", 2024-08-17T18:07:06.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4345000000000001e-05 2024-08-17T18:07:06.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:07:06.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.653480+0000", 2024-08-17T18:07:06.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1059999999999996e-06 2024-08-17T18:07:06.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.669515+0000", 2024-08-17T18:07:06.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016035132000000001 2024-08-17T18:07:06.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:07:06.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.669587+0000", 2024-08-17T18:07:06.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2291000000000003e-05 2024-08-17T18:07:06.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:07:06.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.709357+0000", 2024-08-17T18:07:06.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039770176999999997 2024-08-17T18:07:06.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:07:06.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.709424+0000", 2024-08-17T18:07:06.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6483999999999996e-05 2024-08-17T18:07:06.688 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.688 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:07:06.688 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.688 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.45626.0:79112 55.5 e1467/1458 55:a5af278c:::benchmark_data_smithi136_313476_object79111:head v 1467'5022, mlcod=1454'1645)", 2024-08-17T18:07:06.688 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:05:02.655482+0000", 2024-08-17T18:07:06.688 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 123.951794675, 2024-08-17T18:07:06.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048936354000000001, 2024-08-17T18:07:06.688 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:07:06.688 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:07:06.688 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:07:06.688 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:07:06.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:07:06.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.655482+0000", 2024-08-17T18:07:06.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:07:06.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.655482+0000", 2024-08-17T18:07:06.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:07:06.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.655484+0000", 2024-08-17T18:07:06.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2029999999999999e-06 2024-08-17T18:07:06.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:07:06.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.655527+0000", 2024-08-17T18:07:06.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3268e-05 2024-08-17T18:07:06.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:07:06.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.655546+0000", 2024-08-17T18:07:06.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8921999999999999e-05 2024-08-17T18:07:06.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:07:06.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.655555+0000", 2024-08-17T18:07:06.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5079999999999994e-06 2024-08-17T18:07:06.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.664728+0000", 2024-08-17T18:07:06.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0091728739999999993 2024-08-17T18:07:06.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:07:06.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.664757+0000", 2024-08-17T18:07:06.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8192e-05 2024-08-17T18:07:06.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:07:06.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.704393+0000", 2024-08-17T18:07:06.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039636738999999997 2024-08-17T18:07:06.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:07:06.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.704418+0000", 2024-08-17T18:07:06.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4647999999999998e-05 2024-08-17T18:07:06.694 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.694 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:07:06.694 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.694 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.45626.0:79110 55.d e1467/1465 55:b1adc305:::benchmark_data_smithi136_313476_object79109:head v 1467'4982, mlcod=0'0)", 2024-08-17T18:07:06.695 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:05:02.655591+0000", 2024-08-17T18:07:06.695 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 123.951685465, 2024-08-17T18:07:06.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.053881749, 2024-08-17T18:07:06.695 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:07:06.695 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:07:06.695 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:07:06.695 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:07:06.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:07:06.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.655591+0000", 2024-08-17T18:07:06.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:07:06.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.655591+0000", 2024-08-17T18:07:06.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:07:06.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.655591+0000", 2024-08-17T18:07:06.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3000000000000001e-07 2024-08-17T18:07:06.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:07:06.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.655623+0000", 2024-08-17T18:07:06.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1608e-05 2024-08-17T18:07:06.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:07:06.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.655633+0000", 2024-08-17T18:07:06.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0061000000000001e-05 2024-08-17T18:07:06.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:07:06.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.655637+0000", 2024-08-17T18:07:06.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3499999999999999e-06 2024-08-17T18:07:06.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.675581+0000", 2024-08-17T18:07:06.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019943901 2024-08-17T18:07:06.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:07:06.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.675606+0000", 2024-08-17T18:07:06.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5157e-05 2024-08-17T18:07:06.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:07:06.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.709428+0000", 2024-08-17T18:07:06.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.033821676000000002 2024-08-17T18:07:06.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:07:06.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.709473+0000", 2024-08-17T18:07:06.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4566e-05 2024-08-17T18:07:06.699 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.699 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:07:06.699 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.699 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.45626.0:79113 55.d e1467/1465 55:b81991c2:::benchmark_data_smithi136_313476_object79112:head v 1467'4983, mlcod=0'0)", 2024-08-17T18:07:06.699 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:05:02.659095+0000", 2024-08-17T18:07:06.699 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 123.94818164599999, 2024-08-17T18:07:06.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.052700964000000003, 2024-08-17T18:07:06.699 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:07:06.699 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:07:06.699 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:07:06.699 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:07:06.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:07:06.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.659095+0000", 2024-08-17T18:07:06.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:07:06.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.659095+0000", 2024-08-17T18:07:06.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:07:06.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.659096+0000", 2024-08-17T18:07:06.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1319999999999999e-06 2024-08-17T18:07:06.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:07:06.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.659138+0000", 2024-08-17T18:07:06.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2009999999999999e-05 2024-08-17T18:07:06.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:07:06.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.660839+0000", 2024-08-17T18:07:06.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017014529999999999 2024-08-17T18:07:06.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:07:06.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.660854+0000", 2024-08-17T18:07:06.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4823e-05 2024-08-17T18:07:06.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.692996+0000", 2024-08-17T18:07:06.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.032142188000000002 2024-08-17T18:07:06.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:07:06.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.693046+0000", 2024-08-17T18:07:06.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9679999999999999e-05 2024-08-17T18:07:06.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:07:06.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.711772+0000", 2024-08-17T18:07:06.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018726573999999999 2024-08-17T18:07:06.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:07:06.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.711796+0000", 2024-08-17T18:07:06.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3104000000000001e-05 2024-08-17T18:07:06.703 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.703 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:07:06.703 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.703 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.45626.0:79115 55.5 e1467/1458 55:af3260de:::benchmark_data_smithi136_313476_object79114:head v 1467'5023, mlcod=1454'1645)", 2024-08-17T18:07:06.703 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:05:02.660902+0000", 2024-08-17T18:07:06.703 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 123.946374676, 2024-08-17T18:07:06.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050842740999999997, 2024-08-17T18:07:06.704 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:07:06.704 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:07:06.704 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:07:06.704 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:07:06.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:07:06.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.660902+0000", 2024-08-17T18:07:06.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:07:06.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.660902+0000", 2024-08-17T18:07:06.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:07:06.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:07:06.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.660902+0000", 2024-08-17T18:07:06.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5099999999999998e-07 2024-08-17T18:07:06.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:07:06.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.660981+0000", 2024-08-17T18:07:06.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8693e-05 2024-08-17T18:07:06.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:07:06.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.660993+0000", 2024-08-17T18:07:06.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1824000000000001e-05 2024-08-17T18:07:06.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:07:06.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.660999+0000", 2024-08-17T18:07:06.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9610000000000002e-06 2024-08-17T18:07:06.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:07:06.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.688861+0000", 2024-08-17T18:07:06.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.027861614 2024-08-17T18:07:06.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:07:06.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.688887+0000", 2024-08-17T18:07:06.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6367999999999999e-05 2024-08-17T18:07:06.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:07:06.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.711705+0000", 2024-08-17T18:07:06.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.022818515000000001 2024-08-17T18:07:06.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:07:06.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:07:06.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:07:06.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:05:02.711744+0000", 2024-08-17T18:07:06.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8915000000000003e-05 2024-08-17T18:07:06.707 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.707 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:07:06.707 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.708 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:07:06.708 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:07:06.708 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:07:06.708 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:06.731 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:06.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:06.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:06 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1245859904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:07:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:07:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:07:06.833 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:06.860 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:06.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:06.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:06.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:06.986 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:06.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:06.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:07.088 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:07.111 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:07.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:07.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:07.213 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:07.243 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:07.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:07.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:07.345 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:07.367 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:07.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:07.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:07.469 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:07.501 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:07.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:07.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:07.603 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:07.632 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:07.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:07.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:07.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:07 smithi165 ceph-mon[105517]: pgmap v2682: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 3 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 3.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 239 op/s; 28829/186291 objects degraded (15.475%); 10550/186291 objects misplaced (5.663%); 8.8 MiB/s, 141 objects/s recovering 2024-08-17T18:07:07.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:07 smithi136 ceph-mon[109861]: pgmap v2682: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 3 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 3.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 239 op/s; 28829/186291 objects degraded (15.475%); 10550/186291 objects misplaced (5.663%); 8.8 MiB/s, 141 objects/s recovering 2024-08-17T18:07:07.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:07 smithi136 ceph-mon[115078]: pgmap v2682: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 3 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 3.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 239 op/s; 28829/186291 objects degraded (15.475%); 10550/186291 objects misplaced (5.663%); 8.8 MiB/s, 141 objects/s recovering 2024-08-17T18:07:07.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:07:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:07:07] "GET /metrics HTTP/1.1" 200 36699 "" "Prometheus/2.51.0" 2024-08-17T18:07:07.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:07.761 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:07.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:07.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:07.862 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:07.893 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:07.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:07.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:07.995 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:08.021 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:08.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:08.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:08.123 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:08.146 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:08.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:08.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:08.248 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:08.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:08.282 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:08.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:08.383 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:08.414 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:08.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:08.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:08.516 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:08.540 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:08.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:08.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:08.642 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:08.665 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:08.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:08.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:08.767 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:08.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:08.789 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:08.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:08.890 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:08.916 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:08.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:08.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:09.018 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:09.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:09.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:09.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:09.146 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:09.170 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:09.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:09.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:09.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:09.296 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:09.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:09.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:09.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:09.425 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:09.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:09.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:09.527 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:09.551 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:09.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:09.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:09.652 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:09.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:09.683 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:09.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:09 smithi136 ceph-mon[109861]: pgmap v2683: 109 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 7 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 86 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 373 op/s; 28056/180192 objects degraded (15.570%); 9604/180192 objects misplaced (5.330%); 13 MiB/s, 202 objects/s recovering 2024-08-17T18:07:09.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:09 smithi136 ceph-mon[115078]: pgmap v2683: 109 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 7 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 86 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 373 op/s; 28056/180192 objects degraded (15.570%); 9604/180192 objects misplaced (5.330%); 13 MiB/s, 202 objects/s recovering 2024-08-17T18:07:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:09 smithi165 ceph-mon[105517]: pgmap v2683: 109 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 7 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 86 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 373 op/s; 28056/180192 objects degraded (15.570%); 9604/180192 objects misplaced (5.330%); 13 MiB/s, 202 objects/s recovering 2024-08-17T18:07:09.784 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:09.811 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:09.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:09.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:09.913 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:09.936 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:09.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:09.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:10.038 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:10.065 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:10.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:10.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:10.167 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:10.194 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:10.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:10.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:10.296 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:10.321 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:10.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:10.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:10.423 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:10.450 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:10.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:10.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:10.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:10.580 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:10.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:10.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:10.682 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:10.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:10 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 28056/180192 objects degraded (15.570%), 12 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:10.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:10 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 28056/180192 objects degraded (15.570%), 12 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:10.706 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:10.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:10.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:10 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 28056/180192 objects degraded (15.570%), 12 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:10.809 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:10.832 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:10.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:10.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:10.968 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:10.996 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:10.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:10.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:11.097 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:11.120 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:11.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:11.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:11.222 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:11.246 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:11.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:11.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:11.348 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:11.375 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:11.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:11.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:11.477 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:11.502 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:11.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:11.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:11.604 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:11.633 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:11.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:11.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:11 smithi136 ceph-mon[109861]: pgmap v2684: 109 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 7 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 86 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 313 op/s; 28056/180192 objects degraded (15.570%); 9604/180192 objects misplaced (5.330%); 11 MiB/s, 170 objects/s recovering 2024-08-17T18:07:11.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:11 smithi136 ceph-mon[115078]: pgmap v2684: 109 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 7 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 86 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 313 op/s; 28056/180192 objects degraded (15.570%); 9604/180192 objects misplaced (5.330%); 11 MiB/s, 170 objects/s recovering 2024-08-17T18:07:11.735 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:11 smithi165 ceph-mon[105517]: pgmap v2684: 109 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 7 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 86 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 313 op/s; 28056/180192 objects degraded (15.570%); 9604/180192 objects misplaced (5.330%); 11 MiB/s, 170 objects/s recovering 2024-08-17T18:07:11.764 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:11.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:11.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:11.866 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:11.897 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:11.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:11.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:11.998 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:12.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:12.032 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:12.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:12.133 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:12.156 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:12.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:12.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:12.258 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:12.282 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:12.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:12.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:12.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:12.390 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:07:12.416 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:12.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:12.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:12.518 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:12.544 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:12.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:12.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:12.646 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:12.678 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:12.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:12.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:12.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:12 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1429296653' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:07:12.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:12 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1429296653' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:07:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:12 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1429296653' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:07:12.779 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:12.802 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:12.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:12.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:12.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:12.936 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:12.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:12.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:12.939 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-17T18:07:13.037 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:13.068 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:13.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:13.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:13.170 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:13.193 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:13.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:13.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:13.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:13.322 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:13.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:13.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:13.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:13.452 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:13.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:13.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:13.554 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:13.570 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:13 smithi136 ceph-mon[109861]: pgmap v2685: 109 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 10 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 83 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 415 op/s; 27950/176448 objects degraded (15.840%); 8631/176448 objects misplaced (4.892%); 15 MiB/s, 239 objects/s recovering 2024-08-17T18:07:13.570 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:13 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:07:13.570 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:13 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1429296653' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:07:13.571 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:13 smithi136 ceph-mon[109861]: osdmap e1494: 8 total, 8 up, 7 in 2024-08-17T18:07:13.571 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:13 smithi136 ceph-mon[115078]: pgmap v2685: 109 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 10 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 83 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 415 op/s; 27950/176448 objects degraded (15.840%); 8631/176448 objects misplaced (4.892%); 15 MiB/s, 239 objects/s recovering 2024-08-17T18:07:13.571 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:13 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:07:13.571 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:13 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1429296653' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:07:13.571 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:13 smithi136 ceph-mon[115078]: osdmap e1494: 8 total, 8 up, 7 in 2024-08-17T18:07:13.586 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:13.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:13.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:13.688 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:13.713 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:13.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:13.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:13 smithi165 ceph-mon[105517]: pgmap v2685: 109 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 10 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 83 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 415 op/s; 27950/176448 objects degraded (15.840%); 8631/176448 objects misplaced (4.892%); 15 MiB/s, 239 objects/s recovering 2024-08-17T18:07:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:13 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:07:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:13 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1429296653' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:07:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:13 smithi165 ceph-mon[105517]: osdmap e1494: 8 total, 8 up, 7 in 2024-08-17T18:07:13.815 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:13.854 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:13.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:13.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:13.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:13.983 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:13.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:13.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:14.085 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:14.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:14.118 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:14.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:14.170 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:07:14.171 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "" 2024-08-17T18:07:14.171 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:07:14.218 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:14.244 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:14.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:14.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:14.345 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:14.377 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:14.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:14.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:14.479 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:14.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:14.505 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:14.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:14.606 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:14.632 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:14.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:14.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:14.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:14.764 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:14.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:14.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:14.865 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:14.889 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:14.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:14.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:14.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:15.014 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:15.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:15.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:15.115 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:15.138 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:15.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:15.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:15.240 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:15.262 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:15.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:15.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:15.364 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:15.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:15.388 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:15.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:15.489 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:15.513 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:15.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:15.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:15.615 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:15.641 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:15.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:15.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:15.682 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:15 smithi136 ceph-mon[109861]: pgmap v2687: 109 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 10 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 83 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 383 op/s; 27393/174774 objects degraded (15.673%); 8518/174774 objects misplaced (4.874%); 14 MiB/s, 226 objects/s recovering 2024-08-17T18:07:15.683 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:15 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 27393/174774 objects degraded (15.673%), 12 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:15.683 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:15 smithi136 ceph-mon[115078]: pgmap v2687: 109 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 10 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 83 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 383 op/s; 27393/174774 objects degraded (15.673%); 8518/174774 objects misplaced (4.874%); 14 MiB/s, 226 objects/s recovering 2024-08-17T18:07:15.683 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:15 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 27393/174774 objects degraded (15.673%), 12 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:15.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:15 smithi165 ceph-mon[105517]: pgmap v2687: 109 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 10 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 83 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 383 op/s; 27393/174774 objects degraded (15.673%); 8518/174774 objects misplaced (4.874%); 14 MiB/s, 226 objects/s recovering 2024-08-17T18:07:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:15 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 27393/174774 objects degraded (15.673%), 12 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:15.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:15.776 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:15.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:15.877 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:15.905 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:15.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:15.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:16.007 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:16.034 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:16.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:16.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:16.136 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:16.162 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:16.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:16.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:16.264 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:16.289 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:16.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:16.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:16.391 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:16.422 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:16.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:16.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:16.524 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:16.546 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:16.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:16.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:16.648 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:16.671 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:16.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:16.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:16.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:16.804 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:16.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:16.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:16.906 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:16.929 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:16.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:16.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:16.939 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:07:17.031 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:17.055 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:17.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:17.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:17.156 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:17.181 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:17.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:17.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:17.282 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:17.308 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:17.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:17.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:17.409 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:17.440 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:17.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:17.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:17.541 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:17.568 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:17.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:17.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:17.670 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:17.695 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:17.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:17.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:17.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:17 smithi136 ceph-mon[109861]: pgmap v2688: 109 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 10 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 83 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 383 op/s; 27393/174774 objects degraded (15.673%); 8518/174774 objects misplaced (4.874%); 14 MiB/s, 226 objects/s recovering 2024-08-17T18:07:17.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:17 smithi136 ceph-mon[109861]: osdmap e1495: 8 total, 8 up, 7 in 2024-08-17T18:07:17.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:17 smithi136 ceph-mon[115078]: pgmap v2688: 109 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 10 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 83 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 383 op/s; 27393/174774 objects degraded (15.673%); 8518/174774 objects misplaced (4.874%); 14 MiB/s, 226 objects/s recovering 2024-08-17T18:07:17.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:17 smithi136 ceph-mon[115078]: osdmap e1495: 8 total, 8 up, 7 in 2024-08-17T18:07:17.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:07:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:07:17] "GET /metrics HTTP/1.1" 200 36699 "" "Prometheus/2.51.0" 2024-08-17T18:07:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:17 smithi165 ceph-mon[105517]: pgmap v2688: 109 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 10 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 83 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 383 op/s; 27393/174774 objects degraded (15.673%); 8518/174774 objects misplaced (4.874%); 14 MiB/s, 226 objects/s recovering 2024-08-17T18:07:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:17 smithi165 ceph-mon[105517]: osdmap e1495: 8 total, 8 up, 7 in 2024-08-17T18:07:17.797 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:17.820 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:17.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:17.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:17.922 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:17.947 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:17.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:17.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:18.049 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:18.072 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:18.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:18.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:18.134 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:07:18.174 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:18.201 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:18.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:18.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:18.303 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:18.326 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:18.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:18.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:18.428 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:18.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:18 smithi136 ceph-mon[109861]: osdmap e1496: 8 total, 8 up, 7 in 2024-08-17T18:07:18.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:18 smithi136 ceph-mon[115078]: osdmap e1496: 8 total, 8 up, 7 in 2024-08-17T18:07:18.452 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:18.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:18.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:18.554 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:18.583 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:18.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:18.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:18.685 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:18.718 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:18.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:18.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:18.738 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:07:18.738 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T18:07:18.739 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-17T18:07:18.739 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:07:18.754 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:18 smithi165 ceph-mon[105517]: osdmap e1496: 8 total, 8 up, 7 in 2024-08-17T18:07:18.820 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:18.846 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:18.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:18.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:18.927 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:07:18.948 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:18.972 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:18.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:18.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:19.074 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:19.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:19.099 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:19.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:19.200 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:19.223 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:19.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:19.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:19.325 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:19.348 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:19.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:19.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:19.378 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-17T18:07:19.416 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:07:19.450 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:19.501 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:19.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:19.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:19.602 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:19.616 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:19 smithi136 ceph-mon[109861]: pgmap v2691: 109 pgs: 1 peering, 2 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 94 active+clean; 3.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 392 op/s; 25287/169383 objects degraded (14.929%); 8775/169383 objects misplaced (5.181%); 11 MiB/s, 334 objects/s recovering 2024-08-17T18:07:19.617 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:19 smithi136 ceph-mon[115078]: pgmap v2691: 109 pgs: 1 peering, 2 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 94 active+clean; 3.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 392 op/s; 25287/169383 objects degraded (14.929%); 8775/169383 objects misplaced (5.181%); 11 MiB/s, 334 objects/s recovering 2024-08-17T18:07:19.637 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:19.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:19.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:19.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:19 smithi165 ceph-mon[105517]: pgmap v2691: 109 pgs: 1 peering, 2 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 94 active+clean; 3.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 392 op/s; 25287/169383 objects degraded (14.929%); 8775/169383 objects misplaced (5.181%); 11 MiB/s, 334 objects/s recovering 2024-08-17T18:07:19.739 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:19.769 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:19.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:19.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:19.871 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:19.899 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:19.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:19.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:20.001 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:20.025 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:20.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:20.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:20.127 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:20.152 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:20.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:20.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:20.254 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:20.265 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:07:20.288 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:20.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:20.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:20.390 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:20.421 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:20.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:20.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:20.523 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:20.546 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:20.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:20.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:20.648 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:20.671 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:20.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:20.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:20.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:20 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 25287/169383 objects degraded (14.929%), 10 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:20.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:20 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 25287/169383 objects degraded (14.929%), 10 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:20 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 25287/169383 objects degraded (14.929%), 10 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:20.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:20.799 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:20.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:20.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:20.904 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:20.928 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:20.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:20.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:21.030 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:21.061 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:21.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:21.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:21.164 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:21.188 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:21.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:21.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:21.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:21.315 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:21.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:21.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:21.347 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:07:21.417 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:21.448 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:21.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:21.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:21.550 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:21.576 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:21.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:21.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:21.592 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:21 smithi136 ceph-mon[109861]: pgmap v2692: 109 pgs: 1 peering, 2 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 94 active+clean; 3.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 299 op/s; 25287/169383 objects degraded (14.929%); 8775/169383 objects misplaced (5.181%); 8.5 MiB/s, 254 objects/s recovering 2024-08-17T18:07:21.592 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:07:21.592 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:07:21.592 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:21 smithi136 ceph-mon[115078]: pgmap v2692: 109 pgs: 1 peering, 2 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 94 active+clean; 3.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 299 op/s; 25287/169383 objects degraded (14.929%); 8775/169383 objects misplaced (5.181%); 8.5 MiB/s, 254 objects/s recovering 2024-08-17T18:07:21.592 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:07:21.592 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:07:21.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:21.704 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:21.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:21.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:21 smithi165 ceph-mon[105517]: pgmap v2692: 109 pgs: 1 peering, 2 active+remapped+backfill_wait, 2 active+recovery_wait+degraded, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 94 active+clean; 3.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 299 op/s; 25287/169383 objects degraded (14.929%); 8775/169383 objects misplaced (5.181%); 8.5 MiB/s, 254 objects/s recovering 2024-08-17T18:07:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:07:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:07:21.789 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 0, 5] out_osds: [7] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:07:21.790 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 1 1 2024-08-17T18:07:21.806 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:21.834 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:21.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:21.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:21.935 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:21.961 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:21.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:21.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:22.063 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:22.086 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:22.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:22.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:22.187 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:22.213 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:22.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:22.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:22.315 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:22.341 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:22.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:22.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:22.415 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:07:22.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:22.472 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:22.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:22.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:22.574 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:22.599 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:22.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:22.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:22.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:22 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1192838984' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:07:22.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:22 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:07:22.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:22 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1192838984' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:07:22.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:22 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:07:22.701 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:22.727 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:22.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:22.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:22.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:22 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1192838984' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:07:22.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:22 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:07:22.828 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:22.857 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:22.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:22.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:22.959 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:22.989 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:22.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:22.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:23.090 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:23.115 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:23.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:23.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:23.217 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:23.240 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:23.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:23.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:23.342 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:23.366 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:23.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:23.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:23.468 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:23.498 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:23.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:23.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:23.600 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:23.621 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:23.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:23.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:23 smithi136 ceph-mon[109861]: pgmap v2693: 109 pgs: 1 active+recovery_wait, 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 4 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 91 active+clean; 3.4 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 366 op/s; 24381/165978 objects degraded (14.689%); 8264/165978 objects misplaced (4.979%); 15 MiB/s, 360 objects/s recovering 2024-08-17T18:07:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:23 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:07:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:23 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:07:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:23 smithi136 ceph-mon[109861]: osdmap e1497: 8 total, 8 up, 7 in 2024-08-17T18:07:23.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:23 smithi136 ceph-mon[115078]: pgmap v2693: 109 pgs: 1 active+recovery_wait, 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 4 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 91 active+clean; 3.4 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 366 op/s; 24381/165978 objects degraded (14.689%); 8264/165978 objects misplaced (4.979%); 15 MiB/s, 360 objects/s recovering 2024-08-17T18:07:23.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:23 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:07:23.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:23 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:07:23.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:23 smithi136 ceph-mon[115078]: osdmap e1497: 8 total, 8 up, 7 in 2024-08-17T18:07:23.723 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:23.748 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:23.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:23.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:23 smithi165 ceph-mon[105517]: pgmap v2693: 109 pgs: 1 active+recovery_wait, 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 4 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 91 active+clean; 3.4 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 366 op/s; 24381/165978 objects degraded (14.689%); 8264/165978 objects misplaced (4.979%); 15 MiB/s, 360 objects/s recovering 2024-08-17T18:07:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:23 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:07:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:23 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:07:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:23 smithi165 ceph-mon[105517]: osdmap e1497: 8 total, 8 up, 7 in 2024-08-17T18:07:23.850 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:23.879 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:23.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:23.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:23.981 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:24.004 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:24.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:24.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:24.106 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:24.135 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:24.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:24.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:24.237 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:24.264 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:24.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:24.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:24.366 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:24.391 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:24.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:24.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:24.493 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:24.522 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:24.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:24.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:24.623 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:24.658 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:24.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:24.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:24.761 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:24.788 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:24.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:24.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:24.791 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:07:24.805 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:07:24.805 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T18:07:24.805 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-17T18:07:24.805 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-17T18:07:24.805 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:07:24.890 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:24.917 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:24.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:24.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:25.019 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:25.044 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:25.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:25.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:25.146 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:25.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:25.177 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:25.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:25.249 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:07:25.278 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:25.306 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:25.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:25.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:25.408 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:25.435 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:25.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:25.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:25.537 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:25.559 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:25.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:25.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:25.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:25.688 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:25.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:25.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:25.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:25 smithi136 ceph-mon[109861]: pgmap v2695: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 91 active+clean; 3.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 447 op/s; 23790/164193 objects degraded (14.489%); 8136/164193 objects misplaced (4.955%); 18 MiB/s, 412 objects/s recovering 2024-08-17T18:07:25.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:25 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 23790/164193 objects degraded (14.489%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:25.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:25 smithi136 ceph-mon[115078]: pgmap v2695: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 91 active+clean; 3.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 447 op/s; 23790/164193 objects degraded (14.489%); 8136/164193 objects misplaced (4.955%); 18 MiB/s, 412 objects/s recovering 2024-08-17T18:07:25.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:25 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 23790/164193 objects degraded (14.489%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:25 smithi165 ceph-mon[105517]: pgmap v2695: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 91 active+clean; 3.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 447 op/s; 23790/164193 objects degraded (14.489%); 8136/164193 objects misplaced (4.955%); 18 MiB/s, 412 objects/s recovering 2024-08-17T18:07:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:25 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 23790/164193 objects degraded (14.489%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:25.789 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:25.818 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:25.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:25.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:25.871 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-17T18:07:25.921 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:25.944 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:25.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:25.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:26.046 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:26.070 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:26.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:26.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:26.172 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:26.201 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:26.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:26.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:26.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:26.335 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:26.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:26.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:26.437 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:26.465 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:26.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:26.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:26.566 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:26.568 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:07:26.569 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:07:26.591 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:07:26.592 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:07:26.595 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:26.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:26.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:26.615 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:07:26.615 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:07:26.633 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:07:26.633 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:07:26.652 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:07:26.652 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:07:26.673 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:07:26.673 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:07:26.690 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:07:26.690 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:07:26.697 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:26.716 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:07:26.716 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:07:26.726 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:26.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:26.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:26.828 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:26.853 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:26.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:26.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:26.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:26.987 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:26.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:26.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:27.089 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:27.115 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:27.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:27.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:27.217 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:27.242 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:27.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:27.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:27.343 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:27.362 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:07:27.371 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:27.372 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:07:27.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:27.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:27.474 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:27.499 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:27.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:27.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:27.601 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:27.630 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:27.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:27.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:27.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:27 smithi136 ceph-mon[109861]: pgmap v2696: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 91 active+clean; 3.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 254 op/s; 23790/164193 objects degraded (14.489%); 8136/164193 objects misplaced (4.955%); 13 MiB/s, 215 objects/s recovering 2024-08-17T18:07:27.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:27 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1098137633' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:07:27.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:27 smithi136 ceph-mon[115078]: pgmap v2696: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 91 active+clean; 3.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 254 op/s; 23790/164193 objects degraded (14.489%); 8136/164193 objects misplaced (4.955%); 13 MiB/s, 215 objects/s recovering 2024-08-17T18:07:27.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:27 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1098137633' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:07:27.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:07:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:07:27] "GET /metrics HTTP/1.1" 200 36695 "" "Prometheus/2.51.0" 2024-08-17T18:07:27.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:27.760 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:27.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:27.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:27 smithi165 ceph-mon[105517]: pgmap v2696: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+clean+scrubbing, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 91 active+clean; 3.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 254 op/s; 23790/164193 objects degraded (14.489%); 8136/164193 objects misplaced (4.955%); 13 MiB/s, 215 objects/s recovering 2024-08-17T18:07:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:27 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1098137633' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:07:27.862 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:27.887 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:27.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:27.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:27.988 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:28.016 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:28.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:28.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:28.118 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:28.149 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:28.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:28.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:28.251 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:28.281 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:28.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:28.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:28.383 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:28.416 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:28.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:28.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:28.517 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:28.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:28.546 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:28.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:28.647 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:28.677 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:28.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:28.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:28.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:28.809 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:28.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:28.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:28.911 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:28.943 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:28.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:28.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:29.045 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:29.073 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:29.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:29.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:29.175 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:29.200 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:29.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:29.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:29.303 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:29.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:29.332 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:29.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:29.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:29.459 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:29.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:29.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:29.563 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:29.587 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:29.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:29.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:29.676 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:29 smithi136 ceph-mon[109861]: pgmap v2697: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+clean+scrubbing, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 92 active+clean; 3.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 352 op/s; 21133/158799 objects degraded (13.308%); 8657/158799 objects misplaced (5.452%); 17 MiB/s, 277 objects/s recovering 2024-08-17T18:07:29.677 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:29 smithi136 ceph-mon[115078]: pgmap v2697: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+clean+scrubbing, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 92 active+clean; 3.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 352 op/s; 21133/158799 objects degraded (13.308%); 8657/158799 objects misplaced (5.452%); 17 MiB/s, 277 objects/s recovering 2024-08-17T18:07:29.689 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:29.715 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:29.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:29.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:29 smithi165 ceph-mon[105517]: pgmap v2697: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+clean+scrubbing, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 92 active+clean; 3.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 352 op/s; 21133/158799 objects degraded (13.308%); 8657/158799 objects misplaced (5.452%); 17 MiB/s, 277 objects/s recovering 2024-08-17T18:07:29.816 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:29.840 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:29.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:29.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:29.942 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:29.971 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:29.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:29.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:30.073 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:30.098 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:30.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:30.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:30.200 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:30.225 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:30.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:30.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:30.327 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:30.355 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:30.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:30.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:30.457 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:30.481 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:30.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:30.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:30.583 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:30.608 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:30.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:30.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:30.610 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:30 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 21133/158799 objects degraded (13.308%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:30.611 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:30 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 21133/158799 objects degraded (13.308%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:30.711 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:30.738 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:30.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:30.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:30.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:30 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 21133/158799 objects degraded (13.308%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:30.840 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:30.868 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:30.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:30.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:30.969 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:30.997 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:30.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:30.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:31.099 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:31.126 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:31.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:31.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:31.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:31.250 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:31.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:31.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:31.352 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:31.375 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:31.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:31.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:31.432 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.1 to 1 (10000) 2024-08-17T18:07:31.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:31.502 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:31.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:31.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:31.604 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:31.626 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:31.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:31.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:31.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:31 smithi136 ceph-mon[109861]: pgmap v2698: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+clean+scrubbing, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 92 active+clean; 3.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 352 op/s; 21133/158799 objects degraded (13.308%); 8657/158799 objects misplaced (5.452%); 17 MiB/s, 277 objects/s recovering 2024-08-17T18:07:31.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:31 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/218282054' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T18:07:31.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:31 smithi136 ceph-mon[115078]: pgmap v2698: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+clean+scrubbing, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 92 active+clean; 3.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 352 op/s; 21133/158799 objects degraded (13.308%); 8657/158799 objects misplaced (5.452%); 17 MiB/s, 277 objects/s recovering 2024-08-17T18:07:31.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:31 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/218282054' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T18:07:31.704 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:07:31.728 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:31.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:31 smithi165 ceph-mon[105517]: pgmap v2698: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+clean+scrubbing, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 92 active+clean; 3.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 352 op/s; 21133/158799 objects degraded (13.308%); 8657/158799 objects misplaced (5.452%); 17 MiB/s, 277 objects/s recovering 2024-08-17T18:07:31.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:31 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/218282054' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T18:07:31.779 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:31.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:31.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:31.881 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:31.908 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:31.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:31.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:32.010 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:32.037 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:32.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:32.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:32.139 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:32.164 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:32.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:32.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:32.266 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:32.291 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:32.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:32.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:32.393 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:32.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:32.421 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:32.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:32.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:32.546 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:32.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:32.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:32.648 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:32.676 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:32.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:32.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:32.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:32 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/218282054' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T18:07:32.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:32 smithi136 ceph-mon[109861]: osdmap e1498: 8 total, 8 up, 7 in 2024-08-17T18:07:32.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:32 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/218282054' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T18:07:32.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:32 smithi136 ceph-mon[115078]: osdmap e1498: 8 total, 8 up, 7 in 2024-08-17T18:07:32.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:32 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/218282054' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T18:07:32.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:32 smithi165 ceph-mon[105517]: osdmap e1498: 8 total, 8 up, 7 in 2024-08-17T18:07:32.778 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:32.801 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:32.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:32.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:32.903 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:32.926 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:32.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:32.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:33.028 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:33.054 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:33.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:33.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:33.156 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:33.178 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:33.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:33.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:33.280 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:33.305 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:33.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:33.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:33.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:33.432 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:33.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:33.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:33.534 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:33.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:33.562 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:33.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:33.663 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:33.688 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:33.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:33.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:33 smithi136 ceph-mon[109861]: pgmap v2700: 109 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 8 active+clean+scrubbing, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 88 active+clean; 3.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 395 op/s; 21133/154275 objects degraded (13.698%); 7913/154275 objects misplaced (5.129%); 13 MiB/s, 206 objects/s recovering 2024-08-17T18:07:33.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:33 smithi136 ceph-mon[115078]: pgmap v2700: 109 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 8 active+clean+scrubbing, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 88 active+clean; 3.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 395 op/s; 21133/154275 objects degraded (13.698%); 7913/154275 objects misplaced (5.129%); 13 MiB/s, 206 objects/s recovering 2024-08-17T18:07:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:33 smithi165 ceph-mon[105517]: pgmap v2700: 109 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 8 active+clean+scrubbing, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 88 active+clean; 3.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 395 op/s; 21133/154275 objects degraded (13.698%); 7913/154275 objects misplaced (5.129%); 13 MiB/s, 206 objects/s recovering 2024-08-17T18:07:33.789 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:33.817 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:33.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:33.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:33.919 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:33.945 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:33.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:33.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:34.047 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:34.070 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:34.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:34.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:34.172 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:34.197 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:34.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:34.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:34.298 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:34.326 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:34.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:34.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:34.428 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:34.454 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:34.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:34.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:34.556 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:34.578 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:34.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:34.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:34.680 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:34.702 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:34.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:34.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:34.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:34.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:34.834 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:34.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:34.935 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:34.959 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:34.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:34.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:35.060 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:35.088 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:35.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:35.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:35.190 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:35.217 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:35.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:35.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:35.319 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:35.342 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:35.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:35.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:35.444 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:35.468 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:35.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:35.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:35.569 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:35.600 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:35.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:35.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:35.702 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:35.729 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:35.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:35.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:35 smithi165 ceph-mon[105517]: pgmap v2701: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 8 active+clean+scrubbing, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 88 active+clean; 3.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 403 op/s; 20589/152091 objects degraded (13.537%); 7837/152091 objects misplaced (5.153%); 13 MiB/s, 211 objects/s recovering 2024-08-17T18:07:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:35 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 20589/152091 objects degraded (13.537%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:35.831 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:35.857 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:35.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:35.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:35.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:35 smithi136 ceph-mon[109861]: pgmap v2701: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 8 active+clean+scrubbing, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 88 active+clean; 3.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 403 op/s; 20589/152091 objects degraded (13.537%); 7837/152091 objects misplaced (5.153%); 13 MiB/s, 211 objects/s recovering 2024-08-17T18:07:35.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:35 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 20589/152091 objects degraded (13.537%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:35.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:35 smithi136 ceph-mon[115078]: pgmap v2701: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 8 active+clean+scrubbing, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 88 active+clean; 3.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 403 op/s; 20589/152091 objects degraded (13.537%); 7837/152091 objects misplaced (5.153%); 13 MiB/s, 211 objects/s recovering 2024-08-17T18:07:35.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:35 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 20589/152091 objects degraded (13.537%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:35.959 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:35.982 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:35.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:35.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:36.084 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:36.111 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:36.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:36.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:36.213 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:36.239 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:36.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:36.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:36.341 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:36.366 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:36.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:36.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:36.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:36.490 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:36.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:36.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:36.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:36.621 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:36.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:36.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:36.723 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:36.747 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:36.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:36.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:36.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:07:36.849 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:36.874 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:36.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:36.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:07:36.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:07:36.976 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:37.002 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:37.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:37.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:37.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:37.129 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:37.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:37.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:37.230 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:37.253 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:37.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:37.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:37.356 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:37.383 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:37.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:37.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:37.485 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:37.508 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:37.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:37.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:37.610 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:37.639 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:37.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:37.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:37.740 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:37.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:37 smithi165 ceph-mon[105517]: pgmap v2702: 109 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+clean+scrubbing, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 92 active+clean; 3.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 418 op/s; 20589/151635 objects degraded (13.578%); 7144/151635 objects misplaced (4.711%); 18 MiB/s, 280 objects/s recovering 2024-08-17T18:07:37.769 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:37.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:37.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:37.871 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:37.900 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:37.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:37.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:37.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:37 smithi136 ceph-mon[109861]: pgmap v2702: 109 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+clean+scrubbing, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 92 active+clean; 3.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 418 op/s; 20589/151635 objects degraded (13.578%); 7144/151635 objects misplaced (4.711%); 18 MiB/s, 280 objects/s recovering 2024-08-17T18:07:37.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:37 smithi136 ceph-mon[115078]: pgmap v2702: 109 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 4 active+clean+scrubbing, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 92 active+clean; 3.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 418 op/s; 20589/151635 objects degraded (13.578%); 7144/151635 objects misplaced (4.711%); 18 MiB/s, 280 objects/s recovering 2024-08-17T18:07:37.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:07:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:07:37] "GET /metrics HTTP/1.1" 200 36691 "" "Prometheus/2.51.0" 2024-08-17T18:07:38.002 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:38.028 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:38.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:38.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:38.130 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:38.159 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:38.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:38.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:38.261 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:38.292 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:38.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:38.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:38.394 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:38.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:38.424 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:38.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:38.525 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:38.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:38.553 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:38.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:38.654 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:38.677 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:38.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:38.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:38.779 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:38.805 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:38.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:38.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:38.907 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:38.943 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:38.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:38.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:39.045 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:39.068 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:39.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:39.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:39.170 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:39.197 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:39.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:39.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:39.299 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:39.322 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:39.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:39.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:39.424 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:39.449 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:39.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:39.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:39.551 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:39.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:39.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:39.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:39.675 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:39 smithi136 ceph-mon[109861]: pgmap v2703: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+clean+scrubbing, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 3.0 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 451 op/s; 20085/145236 objects degraded (13.829%); 6770/145236 objects misplaced (4.661%); 13 MiB/s, 204 objects/s recovering 2024-08-17T18:07:39.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:39 smithi136 ceph-mon[115078]: pgmap v2703: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+clean+scrubbing, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 3.0 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 451 op/s; 20085/145236 objects degraded (13.829%); 6770/145236 objects misplaced (4.661%); 13 MiB/s, 204 objects/s recovering 2024-08-17T18:07:39.702 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:39.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:39.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:39 smithi165 ceph-mon[105517]: pgmap v2703: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+clean+scrubbing, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 3.0 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 451 op/s; 20085/145236 objects degraded (13.829%); 6770/145236 objects misplaced (4.661%); 13 MiB/s, 204 objects/s recovering 2024-08-17T18:07:39.804 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:39.830 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:39.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:39.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:39.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:39.958 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:39.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:39.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:40.060 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:40.083 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:40.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:40.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:40.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 2 1 2024-08-17T18:07:40.186 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:40.212 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:40.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:40.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:40.314 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:40.344 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:40.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:40.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:40.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:40.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:40.479 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:40.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:40.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:40.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:40.616 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:40.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:40.704 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:07:40.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:40.741 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:07:40.741 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:07:40.743 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:40 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 20085/145236 objects degraded (13.829%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:40.743 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:40 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 20085/145236 objects degraded (13.829%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:40.759 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:07:40.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:40 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 20085/145236 objects degraded (13.829%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:40.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:40.774 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:40.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:40.875 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:40.899 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:40.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:40.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:41.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:41.029 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:41.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:41.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:41.131 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:41.160 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:41.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:41.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:41.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:41.287 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:41.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:41.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:41.389 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:41.414 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:41.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:41.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:41.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:41.540 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:41.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:41.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:41.687 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:41.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:41 smithi136 ceph-mon[109861]: pgmap v2704: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+clean+scrubbing, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 3.0 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 451 op/s; 20085/145236 objects degraded (13.829%); 6770/145236 objects misplaced (4.661%); 13 MiB/s, 204 objects/s recovering 2024-08-17T18:07:41.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:41 smithi136 ceph-mon[115078]: pgmap v2704: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+clean+scrubbing, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 3.0 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 451 op/s; 20085/145236 objects degraded (13.829%); 6770/145236 objects misplaced (4.661%); 13 MiB/s, 204 objects/s recovering 2024-08-17T18:07:41.712 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:41.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:41.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:41 smithi165 ceph-mon[105517]: pgmap v2704: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+clean+scrubbing, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 91 active+clean; 3.0 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 451 op/s; 20085/145236 objects degraded (13.829%); 6770/145236 objects misplaced (4.661%); 13 MiB/s, 204 objects/s recovering 2024-08-17T18:07:41.814 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:41.844 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:41.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:41.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:41.945 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:41.973 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:41.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:41.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:42.075 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:42.104 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:42.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:42.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:42.205 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:42.228 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:42.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:42.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:42.330 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:42.355 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:42.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:42.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:42.457 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:42.476 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:07:42.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:42.487 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:42.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:42.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:42.612 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:42.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:42.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:42.714 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:42.739 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:42.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:42.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:42.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:42 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2410688116' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:07:42.842 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:42.867 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:42.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:42.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:42.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:42 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2410688116' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:07:42.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:42 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2410688116' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:07:42.969 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:42.992 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:42.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:42.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:43.093 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:43.116 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:43.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:43.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:43.156 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-17T18:07:43.217 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:43.241 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:43.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:43.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:43.343 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:43.370 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:43.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:43.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:43.472 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:43.500 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:43.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:43.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:43.602 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:43.629 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:43.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:43.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:43.731 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:43.755 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:43.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:43.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:43 smithi165 ceph-mon[105517]: pgmap v2705: 109 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 87 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 555 op/s; 17732/140406 objects degraded (12.629%); 7765/140406 objects misplaced (5.530%); 10 MiB/s, 243 objects/s recovering 2024-08-17T18:07:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:43 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:07:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:43 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2410688116' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:07:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:43 smithi165 ceph-mon[105517]: osdmap e1499: 8 total, 8 up, 7 in 2024-08-17T18:07:43.858 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:43.888 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:43.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:43.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:43.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:43 smithi136 ceph-mon[109861]: pgmap v2705: 109 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 87 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 555 op/s; 17732/140406 objects degraded (12.629%); 7765/140406 objects misplaced (5.530%); 10 MiB/s, 243 objects/s recovering 2024-08-17T18:07:43.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:43 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:07:43.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:43 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2410688116' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:07:43.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:43 smithi136 ceph-mon[109861]: osdmap e1499: 8 total, 8 up, 7 in 2024-08-17T18:07:43.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:43 smithi136 ceph-mon[115078]: pgmap v2705: 109 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 87 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 555 op/s; 17732/140406 objects degraded (12.629%); 7765/140406 objects misplaced (5.530%); 10 MiB/s, 243 objects/s recovering 2024-08-17T18:07:43.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:43 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:07:43.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:43 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2410688116' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:07:43.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:43 smithi136 ceph-mon[115078]: osdmap e1499: 8 total, 8 up, 7 in 2024-08-17T18:07:43.990 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:44.020 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:44.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:44.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:44.122 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:44.148 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:44.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:44.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:44.250 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:44.276 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:44.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:44.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:44.378 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:44.405 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:44.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:44.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:44.507 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:44.539 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:44.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:44.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:44.641 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:44.668 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:44.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:44.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:44.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:44 smithi136 ceph-mon[109861]: osdmap e1500: 8 total, 8 up, 7 in 2024-08-17T18:07:44.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:44 smithi136 ceph-mon[115078]: osdmap e1500: 8 total, 8 up, 7 in 2024-08-17T18:07:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:44 smithi165 ceph-mon[105517]: osdmap e1500: 8 total, 8 up, 7 in 2024-08-17T18:07:44.769 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:44.801 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:44.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:44.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:44.903 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:44.934 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:44.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:44.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:45.036 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:45.063 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:45.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:45.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:45.165 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:45.188 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:45.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:45.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:45.290 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:45.317 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:45.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:45.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:45.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:45.449 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:45.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:45.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:45.551 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:45.579 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:45.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:45.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:45.681 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:45.708 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:45.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:45.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:45 smithi165 ceph-mon[105517]: pgmap v2708: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 87 active+clean; 2.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 557 op/s; 17204/138243 objects degraded (12.445%); 7706/138243 objects misplaced (5.574%); 8.3 MiB/s, 242 objects/s recovering 2024-08-17T18:07:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:45 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 17204/138243 objects degraded (12.445%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:45.809 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:45.837 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:45.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:45.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:45.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:45 smithi136 ceph-mon[109861]: pgmap v2708: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 87 active+clean; 2.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 557 op/s; 17204/138243 objects degraded (12.445%); 7706/138243 objects misplaced (5.574%); 8.3 MiB/s, 242 objects/s recovering 2024-08-17T18:07:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:45 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 17204/138243 objects degraded (12.445%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:45.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:45 smithi136 ceph-mon[115078]: pgmap v2708: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 87 active+clean; 2.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 557 op/s; 17204/138243 objects degraded (12.445%); 7706/138243 objects misplaced (5.574%); 8.3 MiB/s, 242 objects/s recovering 2024-08-17T18:07:45.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:45 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 17204/138243 objects degraded (12.445%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:45.968 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:45.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:45.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:46.070 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:46.101 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:46.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:46.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:46.203 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:46.229 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:46.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:46.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:46.332 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:46.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:46.357 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:46.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:46.459 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:46.484 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:46.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:46.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:46.586 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:46.614 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:46.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:46.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:46.716 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:46.744 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:46.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:46.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:46.753 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:07:46.753 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:07:46.780 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:07:46.780 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:07:46.801 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:07:46.801 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:07:46.821 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:07:46.822 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:07:46.842 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:07:46.842 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:07:46.846 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:46.862 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:07:46.863 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:07:46.874 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:46.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:46.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:46.887 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:07:46.887 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:07:46.914 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:07:46.914 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:07:46.976 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:47.003 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:47.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:47.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:47.106 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:47.137 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:47.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:47.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:47.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:47.264 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:47.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:47.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:47.365 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:47.391 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:47.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:47.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:47.493 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:47.499 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.2 to 1 (10000) 2024-08-17T18:07:47.530 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:47.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:47.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:47.579 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:07:47.632 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:47.657 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:47.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:47.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:47.759 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:47 smithi165 ceph-mon[105517]: pgmap v2709: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 95 active+clean; 2.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 306 op/s; 16899/137871 objects degraded (12.257%); 7706/137871 objects misplaced (5.589%); 4.9 MiB/s, 225 objects/s recovering 2024-08-17T18:07:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:47 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1328502552' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:07:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:47 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2445598585' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T18:07:47.769 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:47 smithi136 ceph-mon[115078]: pgmap v2709: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 95 active+clean; 2.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 306 op/s; 16899/137871 objects degraded (12.257%); 7706/137871 objects misplaced (5.589%); 4.9 MiB/s, 225 objects/s recovering 2024-08-17T18:07:47.769 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:47 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1328502552' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:07:47.770 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:47 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2445598585' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T18:07:47.770 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:47 smithi136 ceph-mon[109861]: pgmap v2709: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 95 active+clean; 2.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 306 op/s; 16899/137871 objects degraded (12.257%); 7706/137871 objects misplaced (5.589%); 4.9 MiB/s, 225 objects/s recovering 2024-08-17T18:07:47.770 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:47 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1328502552' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:07:47.770 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:47 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2445598585' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T18:07:47.770 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:07:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:07:47] "GET /metrics HTTP/1.1" 200 36691 "" "Prometheus/2.51.0" 2024-08-17T18:07:47.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:47.788 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:47.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:47.890 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:47.913 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:47.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:47.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:48.014 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:48.039 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:48.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:48.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:48.142 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:48.169 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:48.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:48.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:48.271 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:48.295 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:48.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:48.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:48.397 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:48.419 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:48.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:48.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:48.501 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:07:48.516 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:07:48.521 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:48.532 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 3 1 2024-08-17T18:07:48.585 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:48.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:48.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:48.609 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:48 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2445598585' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T18:07:48.609 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:48 smithi136 ceph-mon[115078]: osdmap e1501: 8 total, 8 up, 7 in 2024-08-17T18:07:48.609 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:48 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2445598585' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T18:07:48.609 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:48 smithi136 ceph-mon[109861]: osdmap e1501: 8 total, 8 up, 7 in 2024-08-17T18:07:48.686 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:48.711 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:48.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:48.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:48.813 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:48.837 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:48.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:48.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:48.939 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:48.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:48.965 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:48.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:48 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2445598585' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T18:07:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:48 smithi165 ceph-mon[105517]: osdmap e1501: 8 total, 8 up, 7 in 2024-08-17T18:07:49.066 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:49.093 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:49.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:49.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:49.195 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:49.218 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:49.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:49.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:49.320 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:49.344 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:49.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:49.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:49.446 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:49.468 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:49.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:49.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:49.570 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:49.594 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:49.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:49.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:49.696 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:49.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:49 smithi136 ceph-mon[109861]: pgmap v2711: 109 pgs: 2 remapped+peering, 1 active+remapped+backfilling, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 95 active+clean; 2.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 469 op/s; 9556/131949 objects degraded (7.242%); 5542/131949 objects misplaced (4.200%); 28 MiB/s, 502 objects/s recovering 2024-08-17T18:07:49.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:49 smithi136 ceph-mon[109861]: osdmap e1502: 8 total, 8 up, 7 in 2024-08-17T18:07:49.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:49 smithi136 ceph-mon[115078]: pgmap v2711: 109 pgs: 2 remapped+peering, 1 active+remapped+backfilling, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 95 active+clean; 2.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 469 op/s; 9556/131949 objects degraded (7.242%); 5542/131949 objects misplaced (4.200%); 28 MiB/s, 502 objects/s recovering 2024-08-17T18:07:49.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:49 smithi136 ceph-mon[115078]: osdmap e1502: 8 total, 8 up, 7 in 2024-08-17T18:07:49.718 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:49.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:49.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:49.804 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:07:49.820 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:49.843 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:49.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:49.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:49.945 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:49.970 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:49.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:49.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:49 smithi165 ceph-mon[105517]: pgmap v2711: 109 pgs: 2 remapped+peering, 1 active+remapped+backfilling, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 95 active+clean; 2.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 469 op/s; 9556/131949 objects degraded (7.242%); 5542/131949 objects misplaced (4.200%); 28 MiB/s, 502 objects/s recovering 2024-08-17T18:07:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:49 smithi165 ceph-mon[105517]: osdmap e1502: 8 total, 8 up, 7 in 2024-08-17T18:07:50.072 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:50.095 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:50.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:50.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:50.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:50.219 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:50.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:50.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:50.321 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:50.343 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:50.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:50.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:50.445 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:50.467 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:50.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:50.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:50.569 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:50.591 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:50.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:50.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:50.693 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:50.714 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:50.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:50.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:50.816 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:50.837 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:50.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:50.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:50.939 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:50.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:50 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 9556/131949 objects degraded (7.242%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:50.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:50 smithi136 ceph-mon[109861]: osdmap e1503: 8 total, 8 up, 7 in 2024-08-17T18:07:50.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:50 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 9556/131949 objects degraded (7.242%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:50 smithi136 ceph-mon[115078]: osdmap e1503: 8 total, 8 up, 7 in 2024-08-17T18:07:50.961 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:50.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:50.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:50 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 9556/131949 objects degraded (7.242%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:50 smithi165 ceph-mon[105517]: osdmap e1503: 8 total, 8 up, 7 in 2024-08-17T18:07:51.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:51.085 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:51.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:51.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:51.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:51.209 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:51.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:51.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:51.311 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:51.332 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:51.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:51.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:51.434 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:51.460 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:51.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:51.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:51.562 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:51.583 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:51.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:51.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:51.686 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:51.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:51.708 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:51.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:51.809 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:51.832 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:51.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:51.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:51.934 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:51.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:51 smithi136 ceph-mon[115078]: pgmap v2714: 109 pgs: 2 remapped+peering, 1 active+remapped+backfilling, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 95 active+clean; 2.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 349 op/s; 9556/131949 objects degraded (7.242%); 5542/131949 objects misplaced (4.200%); 24 MiB/s, 429 objects/s recovering 2024-08-17T18:07:51.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:51 smithi136 ceph-mon[115078]: osdmap e1504: 8 total, 8 up, 7 in 2024-08-17T18:07:51.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:51 smithi136 ceph-mon[115078]: 55.10 scrub starts 2024-08-17T18:07:51.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:51 smithi136 ceph-mon[115078]: 55.10 scrub ok 2024-08-17T18:07:51.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:07:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:51 smithi136 ceph-mon[109861]: pgmap v2714: 109 pgs: 2 remapped+peering, 1 active+remapped+backfilling, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 95 active+clean; 2.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 349 op/s; 9556/131949 objects degraded (7.242%); 5542/131949 objects misplaced (4.200%); 24 MiB/s, 429 objects/s recovering 2024-08-17T18:07:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:51 smithi136 ceph-mon[109861]: osdmap e1504: 8 total, 8 up, 7 in 2024-08-17T18:07:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:51 smithi136 ceph-mon[109861]: 55.10 scrub starts 2024-08-17T18:07:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:51 smithi136 ceph-mon[109861]: 55.10 scrub ok 2024-08-17T18:07:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:07:51.955 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:51.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:51.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:51 smithi165 ceph-mon[105517]: pgmap v2714: 109 pgs: 2 remapped+peering, 1 active+remapped+backfilling, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 95 active+clean; 2.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 349 op/s; 9556/131949 objects degraded (7.242%); 5542/131949 objects misplaced (4.200%); 24 MiB/s, 429 objects/s recovering 2024-08-17T18:07:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:51 smithi165 ceph-mon[105517]: osdmap e1504: 8 total, 8 up, 7 in 2024-08-17T18:07:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:51 smithi165 ceph-mon[105517]: 55.10 scrub starts 2024-08-17T18:07:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:51 smithi165 ceph-mon[105517]: 55.10 scrub ok 2024-08-17T18:07:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:07:52.057 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:52.078 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:52.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:52.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:52.180 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:52.202 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:52.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:52.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:52.304 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:52.325 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:52.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:52.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:52.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:52.449 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:52.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:52.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:52.550 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:52.577 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:52.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:52.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:52.679 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:52.702 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:52.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:52.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:52.806 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:52.828 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:52.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:52.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:52.929 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:52.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:52 smithi136 ceph-mon[115078]: 55.4 scrub starts 2024-08-17T18:07:52.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:52 smithi136 ceph-mon[115078]: 55.4 scrub ok 2024-08-17T18:07:52.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:52 smithi136 ceph-mon[115078]: 1.7 scrub starts 2024-08-17T18:07:52.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:52 smithi136 ceph-mon[115078]: 1.7 scrub ok 2024-08-17T18:07:52.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:52 smithi136 ceph-mon[115078]: 55.7 scrub starts 2024-08-17T18:07:52.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:52 smithi136 ceph-mon[115078]: 55.7 scrub ok 2024-08-17T18:07:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:52 smithi136 ceph-mon[109861]: 55.4 scrub starts 2024-08-17T18:07:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:52 smithi136 ceph-mon[109861]: 55.4 scrub ok 2024-08-17T18:07:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:52 smithi136 ceph-mon[109861]: 1.7 scrub starts 2024-08-17T18:07:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:52 smithi136 ceph-mon[109861]: 1.7 scrub ok 2024-08-17T18:07:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:52 smithi136 ceph-mon[109861]: 55.7 scrub starts 2024-08-17T18:07:52.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:52 smithi136 ceph-mon[109861]: 55.7 scrub ok 2024-08-17T18:07:52.951 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:52.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:52.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:52 smithi165 ceph-mon[105517]: 55.4 scrub starts 2024-08-17T18:07:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:52 smithi165 ceph-mon[105517]: 55.4 scrub ok 2024-08-17T18:07:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:52 smithi165 ceph-mon[105517]: 1.7 scrub starts 2024-08-17T18:07:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:52 smithi165 ceph-mon[105517]: 1.7 scrub ok 2024-08-17T18:07:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:52 smithi165 ceph-mon[105517]: 55.7 scrub starts 2024-08-17T18:07:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:52 smithi165 ceph-mon[105517]: 55.7 scrub ok 2024-08-17T18:07:53.054 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:53.075 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:53.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:53.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:53.177 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:53.198 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:53.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:53.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:53.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:53.323 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:53.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:53.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:53.425 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:53.446 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:53.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:53.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:53.548 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:53.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:53.570 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:53.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:53.671 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:53.692 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:53.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:53.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:53.794 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:53.815 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:53.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:53.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:53.917 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:53.939 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:53.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:53.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:53.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:53 smithi136 ceph-mon[115078]: 1.9 scrub starts 2024-08-17T18:07:53.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:53 smithi136 ceph-mon[115078]: 1.9 scrub ok 2024-08-17T18:07:53.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:53 smithi136 ceph-mon[115078]: pgmap v2716: 109 pgs: 2 remapped+peering, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 100 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 8818/130932 objects degraded (6.735%); 1302/130932 objects misplaced (0.994%); 36 MiB/s, 582 objects/s recovering 2024-08-17T18:07:53.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:53 smithi136 ceph-mon[109861]: 1.9 scrub starts 2024-08-17T18:07:53.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:53 smithi136 ceph-mon[109861]: 1.9 scrub ok 2024-08-17T18:07:53.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:53 smithi136 ceph-mon[109861]: pgmap v2716: 109 pgs: 2 remapped+peering, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 100 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 8818/130932 objects degraded (6.735%); 1302/130932 objects misplaced (0.994%); 36 MiB/s, 582 objects/s recovering 2024-08-17T18:07:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:53 smithi165 ceph-mon[105517]: 1.9 scrub starts 2024-08-17T18:07:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:53 smithi165 ceph-mon[105517]: 1.9 scrub ok 2024-08-17T18:07:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:53 smithi165 ceph-mon[105517]: pgmap v2716: 109 pgs: 2 remapped+peering, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 100 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 8818/130932 objects degraded (6.735%); 1302/130932 objects misplaced (0.994%); 36 MiB/s, 582 objects/s recovering 2024-08-17T18:07:54.041 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:54.062 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:54.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:54.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:54.164 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:54.185 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:54.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:54.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:54.287 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:54.308 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:54.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:54.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:54.410 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:54.442 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:54.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:54.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:54.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:54.569 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:54.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:54.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:54.670 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:54.695 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:54.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:54.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:54.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:54.822 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:54.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:54.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:54.924 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:54.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:54 smithi136 ceph-mon[109861]: 1.5 scrub starts 2024-08-17T18:07:54.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:54 smithi136 ceph-mon[109861]: 1.5 scrub ok 2024-08-17T18:07:54.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:54 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 8818/130932 objects degraded (6.735%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:54.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:54 smithi136 ceph-mon[115078]: 1.5 scrub starts 2024-08-17T18:07:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:54 smithi136 ceph-mon[115078]: 1.5 scrub ok 2024-08-17T18:07:54.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:54 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 8818/130932 objects degraded (6.735%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:54.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:54.948 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:54.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:54 smithi165 ceph-mon[105517]: 1.5 scrub starts 2024-08-17T18:07:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:54 smithi165 ceph-mon[105517]: 1.5 scrub ok 2024-08-17T18:07:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:54 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 8818/130932 objects degraded (6.735%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:55.048 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:55.071 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:55.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:55.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:55.173 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:55.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:55.199 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:55.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:55.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:55.341 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:55.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:55.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:55.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:55.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:55.467 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:55.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:55.568 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:55.596 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:55.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:55.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:55.698 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:55.725 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:55.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:55.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:55.827 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:55.852 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:55.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:55.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:55.906 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:55 smithi136 ceph-mon[109861]: pgmap v2717: 109 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 14576/130914 objects degraded (11.134%); 826/130914 objects misplaced (0.631%); 29 MiB/s, 464 objects/s recovering 2024-08-17T18:07:55.906 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:55 smithi136 ceph-mon[109861]: 1.0 scrub starts 2024-08-17T18:07:55.906 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:55 smithi136 ceph-mon[109861]: 1.0 scrub ok 2024-08-17T18:07:55.906 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:55 smithi136 ceph-mon[109861]: 55.0 scrub starts 2024-08-17T18:07:55.906 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:55 smithi136 ceph-mon[115078]: pgmap v2717: 109 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 14576/130914 objects degraded (11.134%); 826/130914 objects misplaced (0.631%); 29 MiB/s, 464 objects/s recovering 2024-08-17T18:07:55.907 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:55 smithi136 ceph-mon[115078]: 1.0 scrub starts 2024-08-17T18:07:55.907 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:55 smithi136 ceph-mon[115078]: 1.0 scrub ok 2024-08-17T18:07:55.907 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:55 smithi136 ceph-mon[115078]: 55.0 scrub starts 2024-08-17T18:07:55.954 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:55.980 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:55.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:55.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:55 smithi165 ceph-mon[105517]: pgmap v2717: 109 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 14576/130914 objects degraded (11.134%); 826/130914 objects misplaced (0.631%); 29 MiB/s, 464 objects/s recovering 2024-08-17T18:07:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:55 smithi165 ceph-mon[105517]: 1.0 scrub starts 2024-08-17T18:07:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:55 smithi165 ceph-mon[105517]: 1.0 scrub ok 2024-08-17T18:07:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:55 smithi165 ceph-mon[105517]: 55.0 scrub starts 2024-08-17T18:07:56.082 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:56.106 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:56.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:56.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:56.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:56.234 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:56.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:56.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:56.336 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:56.359 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:56.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:56.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:56.461 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:56.486 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:56.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:56.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:56.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:56.620 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:56.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:56.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:56.722 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:56.751 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:56.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:56.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:56.853 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:56.881 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:56.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:56.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:56.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:56 smithi136 ceph-mon[109861]: 55.14 scrub starts 2024-08-17T18:07:56.969 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:56 smithi136 ceph-mon[109861]: 1.a scrub starts 2024-08-17T18:07:56.969 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:56 smithi136 ceph-mon[109861]: 1.a scrub ok 2024-08-17T18:07:56.969 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:56 smithi136 ceph-mon[109861]: 55.14 scrub ok 2024-08-17T18:07:56.969 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:56 smithi136 ceph-mon[109861]: 55.0 scrub ok 2024-08-17T18:07:56.969 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:56 smithi136 ceph-mon[109861]: 55.17 scrub starts 2024-08-17T18:07:56.969 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:56 smithi136 ceph-mon[109861]: 55.17 scrub ok 2024-08-17T18:07:56.970 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:56 smithi136 ceph-mon[115078]: 55.14 scrub starts 2024-08-17T18:07:56.970 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:56 smithi136 ceph-mon[115078]: 1.a scrub starts 2024-08-17T18:07:56.970 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:56 smithi136 ceph-mon[115078]: 1.a scrub ok 2024-08-17T18:07:56.970 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:56 smithi136 ceph-mon[115078]: 55.14 scrub ok 2024-08-17T18:07:56.970 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:56 smithi136 ceph-mon[115078]: 55.0 scrub ok 2024-08-17T18:07:56.970 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:56 smithi136 ceph-mon[115078]: 55.17 scrub starts 2024-08-17T18:07:56.970 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:56 smithi136 ceph-mon[115078]: 55.17 scrub ok 2024-08-17T18:07:56.983 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:57.007 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:57.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:57.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:56 smithi165 ceph-mon[105517]: 55.14 scrub starts 2024-08-17T18:07:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:56 smithi165 ceph-mon[105517]: 1.a scrub starts 2024-08-17T18:07:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:56 smithi165 ceph-mon[105517]: 1.a scrub ok 2024-08-17T18:07:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:56 smithi165 ceph-mon[105517]: 55.14 scrub ok 2024-08-17T18:07:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:56 smithi165 ceph-mon[105517]: 55.0 scrub ok 2024-08-17T18:07:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:56 smithi165 ceph-mon[105517]: 55.17 scrub starts 2024-08-17T18:07:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:56 smithi165 ceph-mon[105517]: 55.17 scrub ok 2024-08-17T18:07:57.110 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:57.135 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:57.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:57.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:57.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:57.264 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:57.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:57.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:57.366 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:57.390 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:57.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:57.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:57.492 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:57.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:57.519 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:57.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:57.621 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:57.642 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:57.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:57.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:57.744 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:07:57.769 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:57.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:57.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:07:57.871 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:57.898 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:57.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:57.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:57.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:57 smithi136 ceph-mon[109861]: pgmap v2718: 109 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 14576/130797 objects degraded (11.144%); 826/130797 objects misplaced (0.632%); 22 MiB/s, 359 objects/s recovering 2024-08-17T18:07:57.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:57 smithi136 ceph-mon[115078]: pgmap v2718: 109 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 14576/130797 objects degraded (11.144%); 826/130797 objects misplaced (0.632%); 22 MiB/s, 359 objects/s recovering 2024-08-17T18:07:57.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:07:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:07:57] "GET /metrics HTTP/1.1" 200 36673 "" "Prometheus/2.51.0" 2024-08-17T18:07:58.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:58.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:57 smithi165 ceph-mon[105517]: pgmap v2718: 109 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 14576/130797 objects degraded (11.144%); 826/130797 objects misplaced (0.632%); 22 MiB/s, 359 objects/s recovering 2024-08-17T18:07:58.023 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:58.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:58.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:07:58.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:58.146 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:58.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:58.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:58.248 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:58.276 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:58.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:58.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:58.378 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:58.403 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:58.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:58.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:58.505 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:58.529 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:58.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:58.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:58.631 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:58.655 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:58.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:58.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:58.756 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:58.785 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:58.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:58.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:58.887 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:58.912 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:58.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:58.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:58.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:58 smithi136 ceph-mon[115078]: 1.4 scrub starts 2024-08-17T18:07:58.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:58 smithi136 ceph-mon[115078]: 1.4 scrub ok 2024-08-17T18:07:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:58 smithi136 ceph-mon[109861]: 1.4 scrub starts 2024-08-17T18:07:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:58 smithi136 ceph-mon[109861]: 1.4 scrub ok 2024-08-17T18:07:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:58 smithi165 ceph-mon[105517]: 1.4 scrub starts 2024-08-17T18:07:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:58 smithi165 ceph-mon[105517]: 1.4 scrub ok 2024-08-17T18:07:59.014 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:07:59.043 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:59.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:59.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:07:59.145 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:59.173 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:59.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:59.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:59.275 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:07:59.305 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:59.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:59.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:07:59.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:07:59.461 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:59.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:59.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:07:59.563 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:07:59.590 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:07:59.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:59.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:07:59.692 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:07:59.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:59 smithi136 ceph-mon[115078]: pgmap v2719: 109 pgs: 1 active+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 14475/128106 objects degraded (11.299%); 27 MiB/s, 431 objects/s recovering 2024-08-17T18:07:59.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:59 smithi136 ceph-mon[115078]: 1.6 scrub starts 2024-08-17T18:07:59.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:59 smithi136 ceph-mon[115078]: 1.6 scrub ok 2024-08-17T18:07:59.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:07:59 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 14475/128106 objects degraded (11.299%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:59.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:59 smithi136 ceph-mon[109861]: pgmap v2719: 109 pgs: 1 active+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 14475/128106 objects degraded (11.299%); 27 MiB/s, 431 objects/s recovering 2024-08-17T18:07:59.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:59 smithi136 ceph-mon[109861]: 1.6 scrub starts 2024-08-17T18:07:59.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:59 smithi136 ceph-mon[109861]: 1.6 scrub ok 2024-08-17T18:07:59.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:07:59 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 14475/128106 objects degraded (11.299%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:07:59.717 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:59.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:59.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:07:59.819 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:07:59.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:59.848 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:59.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:07:59.949 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:07:59.971 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:07:59.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:07:59.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:59 smithi165 ceph-mon[105517]: pgmap v2719: 109 pgs: 1 active+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 14475/128106 objects degraded (11.299%); 27 MiB/s, 431 objects/s recovering 2024-08-17T18:08:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:59 smithi165 ceph-mon[105517]: 1.6 scrub starts 2024-08-17T18:08:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:59 smithi165 ceph-mon[105517]: 1.6 scrub ok 2024-08-17T18:08:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:07:59 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 14475/128106 objects degraded (11.299%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:08:00.073 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:00.098 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:00.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:00.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:00.199 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:00.226 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:00.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:00.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:00.328 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:00.355 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:00.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:00.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:00.457 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:00.482 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:00.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:00.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:00.584 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:00.610 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:00.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:00.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:00.712 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:00.744 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:00.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:00.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:00.846 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:00.875 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:00.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:00.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:00.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:00 smithi136 ceph-mon[109861]: 1.8 scrub starts 2024-08-17T18:08:00.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:00 smithi136 ceph-mon[109861]: 1.8 scrub ok 2024-08-17T18:08:00.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:00 smithi136 ceph-mon[115078]: 1.8 scrub starts 2024-08-17T18:08:00.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:00 smithi136 ceph-mon[115078]: 1.8 scrub ok 2024-08-17T18:08:00.977 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:01.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:01.003 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:01.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:00 smithi165 ceph-mon[105517]: 1.8 scrub starts 2024-08-17T18:08:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:00 smithi165 ceph-mon[105517]: 1.8 scrub ok 2024-08-17T18:08:01.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:01.131 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:01.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:01.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:01.233 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:01.258 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:01.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:01.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:01.360 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:01.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:01.388 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:01.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:01.489 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:01.514 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:01.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:01.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:01.617 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:01.648 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:01.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:01.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:01.679 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:01 smithi136 ceph-mon[109861]: pgmap v2720: 109 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 14257/126804 objects degraded (11.243%); 24 MiB/s, 396 objects/s recovering 2024-08-17T18:08:01.679 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:01 smithi136 ceph-mon[109861]: 1.1 scrub starts 2024-08-17T18:08:01.679 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:01 smithi136 ceph-mon[109861]: 1.1 scrub ok 2024-08-17T18:08:01.679 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:01 smithi136 ceph-mon[115078]: pgmap v2720: 109 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 14257/126804 objects degraded (11.243%); 24 MiB/s, 396 objects/s recovering 2024-08-17T18:08:01.679 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:01 smithi136 ceph-mon[115078]: 1.1 scrub starts 2024-08-17T18:08:01.680 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:01 smithi136 ceph-mon[115078]: 1.1 scrub ok 2024-08-17T18:08:01.750 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:01.782 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:01.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:01.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:01.884 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:01.912 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:01.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:01.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:02.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:01 smithi165 ceph-mon[105517]: pgmap v2720: 109 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 14257/126804 objects degraded (11.243%); 24 MiB/s, 396 objects/s recovering 2024-08-17T18:08:02.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:01 smithi165 ceph-mon[105517]: 1.1 scrub starts 2024-08-17T18:08:02.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:01 smithi165 ceph-mon[105517]: 1.1 scrub ok 2024-08-17T18:08:02.013 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:02.042 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:02.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:02.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:02.144 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:02.174 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:02.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:02.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:02.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:02.302 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:02.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:02.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:02.309 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:08:02.309 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:08:02.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:08:02.309 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.45626.0:144447 55.4 e1504/1472 55:27f9711f:::benchmark_data_smithi136_313476_object52461:head v 1504'7473, mlcod=1504'7471)", 2024-08-17T18:08:02.309 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:08:02.306776+0000", 2024-08-17T18:08:02.309 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.00084075100000000004, 2024-08-17T18:08:02.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000867583, 2024-08-17T18:08:02.309 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:08:02.310 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:08:02.310 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T18:08:02.310 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:08:02.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:08:02.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:08:02.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:02.306776+0000", 2024-08-17T18:08:02.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:08:02.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:08:02.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:08:02.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:08:02.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:02.306776+0000", 2024-08-17T18:08:02.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:08:02.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:08:02.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:08:02.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:08:02.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:02.306777+0000", 2024-08-17T18:08:02.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.139e-06 2024-08-17T18:08:02.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:08:02.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:08:02.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:08:02.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:02.306788+0000", 2024-08-17T18:08:02.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.042e-05 2024-08-17T18:08:02.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:08:02.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:08:02.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:08:02.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:02.306807+0000", 2024-08-17T18:08:02.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9394000000000001e-05 2024-08-17T18:08:02.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:08:02.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:08:02.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:08:02.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:02.306817+0000", 2024-08-17T18:08:02.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5089999999999999e-06 2024-08-17T18:08:02.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:08:02.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:08:02.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:08:02.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:02.306912+0000", 2024-08-17T18:08:02.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5320000000000002e-05 2024-08-17T18:08:02.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:08:02.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:08:02.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:08:02.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:02.306942+0000", 2024-08-17T18:08:02.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9856e-05 2024-08-17T18:08:02.313 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:08:02.313 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:08:02.313 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:08:02.313 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:08:02.313 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-17T18:08:02.313 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 1 2024-08-17T18:08:02.313 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:08:02.404 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:02.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:02.435 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:02.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:02.536 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:02.561 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:02.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:02.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:02.612 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:08:02.663 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:02.686 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:02.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:02.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:02.788 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:02.813 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:02.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:02.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:02.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:02.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:02.939 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:02.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:02.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:02 smithi136 ceph-mon[109861]: 1.3 scrub starts 2024-08-17T18:08:02.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:02 smithi136 ceph-mon[109861]: 1.3 scrub ok 2024-08-17T18:08:02.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:02 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3011914293' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:08:02.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:02 smithi136 ceph-mon[115078]: 1.3 scrub starts 2024-08-17T18:08:02.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:02 smithi136 ceph-mon[115078]: 1.3 scrub ok 2024-08-17T18:08:02.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:02 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3011914293' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:08:03.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:02 smithi165 ceph-mon[105517]: 1.3 scrub starts 2024-08-17T18:08:03.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:02 smithi165 ceph-mon[105517]: 1.3 scrub ok 2024-08-17T18:08:03.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:02 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3011914293' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:08:03.040 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:03.072 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:03.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:03.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:03.174 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:03.195 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:03.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:03.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:03.297 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:03.320 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:03.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:03.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:03.422 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-17T18:08:03.422 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:03.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:03.458 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:03.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:03.559 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:03.586 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:03.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:03.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:03.689 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:03 smithi136 ceph-mon[109861]: pgmap v2721: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 6 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 13503/121527 objects degraded (11.111%); 21 MiB/s, 431 objects/s recovering 2024-08-17T18:08:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:03 smithi136 ceph-mon[109861]: 1.2 scrub starts 2024-08-17T18:08:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:03 smithi136 ceph-mon[109861]: 1.2 scrub ok 2024-08-17T18:08:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:03 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:08:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:03 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3011914293' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:08:03.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:03 smithi136 ceph-mon[109861]: osdmap e1505: 8 total, 8 up, 7 in 2024-08-17T18:08:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:03 smithi136 ceph-mon[115078]: pgmap v2721: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 6 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 13503/121527 objects degraded (11.111%); 21 MiB/s, 431 objects/s recovering 2024-08-17T18:08:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:03 smithi136 ceph-mon[115078]: 1.2 scrub starts 2024-08-17T18:08:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:03 smithi136 ceph-mon[115078]: 1.2 scrub ok 2024-08-17T18:08:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:03 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:08:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:03 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3011914293' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:08:03.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:03 smithi136 ceph-mon[115078]: osdmap e1505: 8 total, 8 up, 7 in 2024-08-17T18:08:03.710 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:03.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:03.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:03.812 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:03.837 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:03.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:03.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:03.913 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:08:03.913 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:08:03.914 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:08:03.939 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:03.965 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:03.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:03.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:03 smithi165 ceph-mon[105517]: pgmap v2721: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 6 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 13503/121527 objects degraded (11.111%); 21 MiB/s, 431 objects/s recovering 2024-08-17T18:08:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:03 smithi165 ceph-mon[105517]: 1.2 scrub starts 2024-08-17T18:08:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:03 smithi165 ceph-mon[105517]: 1.2 scrub ok 2024-08-17T18:08:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:03 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:08:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:03 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3011914293' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:08:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:03 smithi165 ceph-mon[105517]: osdmap e1505: 8 total, 8 up, 7 in 2024-08-17T18:08:04.067 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:04.089 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:04.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:04.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:04.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:04.218 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:04.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:04.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:04.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:04.344 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:04.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:04.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:04.446 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:04.468 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:04.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:04.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:04.570 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:04.591 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:04.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:04.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:04.693 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:04.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:04 smithi136 ceph-mon[109861]: osdmap e1506: 8 total, 8 up, 7 in 2024-08-17T18:08:04.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:04 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 13503/121527 objects degraded (11.111%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:08:04.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:04 smithi136 ceph-mon[115078]: osdmap e1506: 8 total, 8 up, 7 in 2024-08-17T18:08:04.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:04 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 13503/121527 objects degraded (11.111%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:08:04.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:04.733 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:04.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:04.834 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:04.863 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:04.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:04.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:04.965 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:04.995 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:04.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:04.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:04 smithi165 ceph-mon[105517]: osdmap e1506: 8 total, 8 up, 7 in 2024-08-17T18:08:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:04 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 13503/121527 objects degraded (11.111%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:08:05.097 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:05.124 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:05.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:05.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:05.226 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:05.256 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:05.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:05.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:05.359 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:05.382 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:05.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:05.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:05.484 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:05.510 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:05.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:05.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:05.612 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:05.640 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:05.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:05.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:05.742 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:05.770 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:05.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:05.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:05.859 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:08:05.872 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:05.887 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:05 smithi136 ceph-mon[109861]: pgmap v2724: 109 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 12337/118989 objects degraded (10.368%); 774/118989 objects misplaced (0.650%); 12 MiB/s, 332 objects/s recovering 2024-08-17T18:08:05.887 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:05 smithi136 ceph-mon[109861]: osdmap e1507: 8 total, 8 up, 7 in 2024-08-17T18:08:05.888 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:08:05.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:05 smithi136 ceph-mon[115078]: pgmap v2724: 109 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 12337/118989 objects degraded (10.368%); 774/118989 objects misplaced (0.650%); 12 MiB/s, 332 objects/s recovering 2024-08-17T18:08:05.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:05 smithi136 ceph-mon[115078]: osdmap e1507: 8 total, 8 up, 7 in 2024-08-17T18:08:05.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:08:05.916 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:05.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:05.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:05 smithi165 ceph-mon[105517]: pgmap v2724: 109 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 2.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 12337/118989 objects degraded (10.368%); 774/118989 objects misplaced (0.650%); 12 MiB/s, 332 objects/s recovering 2024-08-17T18:08:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:05 smithi165 ceph-mon[105517]: osdmap e1507: 8 total, 8 up, 7 in 2024-08-17T18:08:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:08:06.018 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:06.042 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:06.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:06.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:06.144 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:06.171 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:06.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:06.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:06.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:06.299 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:06.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:06.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:06.401 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:06.424 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:06.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:06.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:06.526 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:06.552 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:06.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:06.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:06.654 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:06.690 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:06.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:06.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:06.792 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:06.827 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:06.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:06.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:06.929 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:06.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:08:06.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:06 smithi136 ceph-mon[109861]: pgmap v2726: 109 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 102 active+clean; 2.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 496 op/s; 12008/117513 objects degraded (10.218%); 774/117513 objects misplaced (0.659%); 4.0 MiB/s, 302 objects/s recovering 2024-08-17T18:08:06.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:06.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:08:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:06 smithi136 ceph-mon[115078]: pgmap v2726: 109 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 102 active+clean; 2.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 496 op/s; 12008/117513 objects degraded (10.218%); 774/117513 objects misplaced (0.659%); 4.0 MiB/s, 302 objects/s recovering 2024-08-17T18:08:06.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:06.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:06.952 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:06.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:06.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:06.976 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:07.054 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:07.076 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:07.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:07.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:07.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:07.205 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:07.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:07.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:08:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:06 smithi165 ceph-mon[105517]: pgmap v2726: 109 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 102 active+clean; 2.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 496 op/s; 12008/117513 objects degraded (10.218%); 774/117513 objects misplaced (0.659%); 4.0 MiB/s, 302 objects/s recovering 2024-08-17T18:08:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:07.307 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:07.331 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:07.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:07.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:07.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:07.460 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:07.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:07.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:07.526 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:08:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:08:07] "GET /metrics HTTP/1.1" 200 36673 "" "Prometheus/2.51.0" 2024-08-17T18:08:07.562 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:07.591 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:07.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:07.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:07.693 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:07.714 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:07.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:07.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:07.816 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:07.840 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:07.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:07.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:07.941 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:07.970 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:07.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:07.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:08.072 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:08.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:08.095 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:08.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:08.196 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:08.221 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:08.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:08.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:08.322 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:08.351 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:08.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:08.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:08.453 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:08.484 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:08.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:08.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:08.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:08.629 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:08.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:08.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:08.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:08.761 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:08.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:08.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:08.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:08.886 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:08.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:08.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:08.988 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:09.014 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:09.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:09.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:09.116 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:09.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:09.148 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:09.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:09.249 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:09.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:09.289 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:09.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:09.380 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:09 smithi165 ceph-mon[105517]: pgmap v2727: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 102 active+clean; 2.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 437 op/s; 11877/113526 objects degraded (10.462%); 174/113526 objects misplaced (0.153%); 12 MiB/s, 242 objects/s recovering 2024-08-17T18:08:09.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:09.414 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:09.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:09.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:09.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:09 smithi136 ceph-mon[109861]: pgmap v2727: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 102 active+clean; 2.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 437 op/s; 11877/113526 objects degraded (10.462%); 174/113526 objects misplaced (0.153%); 12 MiB/s, 242 objects/s recovering 2024-08-17T18:08:09.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:09 smithi136 ceph-mon[115078]: pgmap v2727: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 102 active+clean; 2.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 437 op/s; 11877/113526 objects degraded (10.462%); 174/113526 objects misplaced (0.153%); 12 MiB/s, 242 objects/s recovering 2024-08-17T18:08:09.516 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:09.542 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:09.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:09.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:09.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:09.678 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:09.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:09.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:09.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:09.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:09.806 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:09.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:09.907 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:09.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:09.936 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:09.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:10.037 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:10.072 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:10.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:10.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:10.174 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:10.202 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:10.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:10.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:10.219 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:08:10.228 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:08:10.228 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T18:08:10.228 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-17T18:08:10.228 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-17T18:08:10.228 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:08:10.304 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:10.328 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:10.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:10.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:10.332 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.3 to 1 (10000) 2024-08-17T18:08:10.353 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:10 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 11877/113526 objects degraded (10.462%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:08:10.353 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:10 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4120629388' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T18:08:10.353 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:10 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 11877/113526 objects degraded (10.462%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:08:10.353 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:10 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4120629388' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T18:08:10.430 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:10.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:10.455 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:10.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:10.556 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:10.577 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:10.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:10.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:10.679 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:10.701 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:10.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:10.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:10 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 11877/113526 objects degraded (10.462%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:08:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:10 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4120629388' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T18:08:10.804 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:10.825 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:10.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:10.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:10.927 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:10.951 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:10.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:10.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:11.034 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:08:11.035 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:08:11.051 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:08:11.051 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:08:11.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:11.067 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:08:11.067 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:08:11.074 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:11.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:11.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:11.083 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:08:11.083 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:08:11.099 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:08:11.099 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:08:11.116 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:08:11.116 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:08:11.132 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:08:11.132 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:08:11.149 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:08:11.149 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:08:11.176 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:11.197 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:11.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:11.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:11.299 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:11.321 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:11.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:11.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:11.423 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:11.445 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:11.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:11.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:11 smithi165 ceph-mon[105517]: pgmap v2728: 109 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 102 active+clean; 2.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 398 op/s; 11662/112248 objects degraded (10.389%); 174/112248 objects misplaced (0.155%); 11 MiB/s, 217 objects/s recovering 2024-08-17T18:08:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:11 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4120629388' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T18:08:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:11 smithi165 ceph-mon[105517]: osdmap e1508: 8 total, 8 up, 7 in 2024-08-17T18:08:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:11 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1389965256' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:08:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:11 smithi165 ceph-mon[105517]: osdmap e1509: 8 total, 8 up, 7 in 2024-08-17T18:08:11.547 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:11.568 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:11.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:11.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:11.670 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:11.692 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:11.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:11.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:11 smithi136 ceph-mon[109861]: pgmap v2728: 109 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 102 active+clean; 2.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 398 op/s; 11662/112248 objects degraded (10.389%); 174/112248 objects misplaced (0.155%); 11 MiB/s, 217 objects/s recovering 2024-08-17T18:08:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:11 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4120629388' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T18:08:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:11 smithi136 ceph-mon[109861]: osdmap e1508: 8 total, 8 up, 7 in 2024-08-17T18:08:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:11 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1389965256' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:08:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:11 smithi136 ceph-mon[109861]: osdmap e1509: 8 total, 8 up, 7 in 2024-08-17T18:08:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:11 smithi136 ceph-mon[115078]: pgmap v2728: 109 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 102 active+clean; 2.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 398 op/s; 11662/112248 objects degraded (10.389%); 174/112248 objects misplaced (0.155%); 11 MiB/s, 217 objects/s recovering 2024-08-17T18:08:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:11 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4120629388' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T18:08:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:11 smithi136 ceph-mon[115078]: osdmap e1508: 8 total, 8 up, 7 in 2024-08-17T18:08:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:11 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1389965256' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:08:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:11 smithi136 ceph-mon[115078]: osdmap e1509: 8 total, 8 up, 7 in 2024-08-17T18:08:11.794 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:11.816 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:11.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:11.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:11.918 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:11.940 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:11.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:11.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:12.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:12.064 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:12.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:12.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:12.165 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:12.187 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:12.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:12.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:12.289 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:12.325 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:12.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:12.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:12.426 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:12.450 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:12.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:12.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:12.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:12.574 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:12.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:12.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:12.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:12.697 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:12.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:12.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:12 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:12 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:12 smithi165 ceph-mon[105517]: osdmap e1510: 8 total, 8 up, 7 in 2024-08-17T18:08:12.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:12.821 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:12.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:12.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:12.923 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:12.945 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:12.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:12.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:12.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:12.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:12.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:12 smithi136 ceph-mon[109861]: osdmap e1510: 8 total, 8 up, 7 in 2024-08-17T18:08:12.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:12.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:12.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:12 smithi136 ceph-mon[115078]: osdmap e1510: 8 total, 8 up, 7 in 2024-08-17T18:08:13.022 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-17T18:08:13.048 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:13.068 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:13.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:13.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:13.170 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:13.191 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:13.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:13.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:13.293 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:13.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:13.315 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:13.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:13.416 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:13.441 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:13.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:13.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:13.543 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:13.564 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:13.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:13.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:13.572 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 4 1 2024-08-17T18:08:13.666 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:13.689 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:13.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:13.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:13 smithi165 ceph-mon[105517]: pgmap v2731: 109 pgs: 1 remapped+peering, 3 peering, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 98 active+clean; 2.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 463 op/s; 11662/108234 objects degraded (10.775%); 174/108234 objects misplaced (0.161%); 7.6 MiB/s, 165 objects/s recovering 2024-08-17T18:08:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:13 smithi165 ceph-mon[105517]: 55.a starting backfill to osd.3 from (1454'815,1484'6440] MIN to 1507'9136 2024-08-17T18:08:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:13 smithi165 ceph-mon[105517]: osdmap e1511: 8 total, 8 up, 7 in 2024-08-17T18:08:13.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:13.822 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:13.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:13.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:13.923 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:13 smithi136 ceph-mon[109861]: pgmap v2731: 109 pgs: 1 remapped+peering, 3 peering, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 98 active+clean; 2.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 463 op/s; 11662/108234 objects degraded (10.775%); 174/108234 objects misplaced (0.161%); 7.6 MiB/s, 165 objects/s recovering 2024-08-17T18:08:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:13 smithi136 ceph-mon[109861]: 55.a starting backfill to osd.3 from (1454'815,1484'6440] MIN to 1507'9136 2024-08-17T18:08:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:13 smithi136 ceph-mon[109861]: osdmap e1511: 8 total, 8 up, 7 in 2024-08-17T18:08:13.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:13 smithi136 ceph-mon[115078]: pgmap v2731: 109 pgs: 1 remapped+peering, 3 peering, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 98 active+clean; 2.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 463 op/s; 11662/108234 objects degraded (10.775%); 174/108234 objects misplaced (0.161%); 7.6 MiB/s, 165 objects/s recovering 2024-08-17T18:08:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:13 smithi136 ceph-mon[115078]: 55.a starting backfill to osd.3 from (1454'815,1484'6440] MIN to 1507'9136 2024-08-17T18:08:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:13 smithi136 ceph-mon[115078]: osdmap e1511: 8 total, 8 up, 7 in 2024-08-17T18:08:13.954 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:13.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:14.055 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:14.089 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:14.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:14.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:14.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:14.230 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:14.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:14.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:14.332 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:14.366 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:14.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:14.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:14.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:14.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:14.512 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:14.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:14.614 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:14.657 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:14.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:14.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:14.759 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:14.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:14.822 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:14.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:14.923 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:14.944 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:14.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:14.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:15.046 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:15.086 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:15.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:15.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:15.188 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:15.219 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:15.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:15.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:15.321 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:15.360 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:15.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:15.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:15.462 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:15.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:15.500 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:15.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:15.601 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:15.634 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:15.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:15.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:15.732 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:15 smithi136 ceph-mon[109861]: pgmap v2734: 109 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 4 peering, 4 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 424 op/s; 8467/107148 objects degraded (7.902%); 259/107148 objects misplaced (0.242%); 10 MiB/s, 354 objects/s recovering 2024-08-17T18:08:15.732 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:15 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 8467/107148 objects degraded (7.902%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:08:15.732 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:15 smithi136 ceph-mon[115078]: pgmap v2734: 109 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 4 peering, 4 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 424 op/s; 8467/107148 objects degraded (7.902%); 259/107148 objects misplaced (0.242%); 10 MiB/s, 354 objects/s recovering 2024-08-17T18:08:15.732 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:15 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 8467/107148 objects degraded (7.902%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:08:15.736 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:15 smithi165 ceph-mon[105517]: pgmap v2734: 109 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 4 peering, 4 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 2.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 424 op/s; 8467/107148 objects degraded (7.902%); 259/107148 objects misplaced (0.242%); 10 MiB/s, 354 objects/s recovering 2024-08-17T18:08:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:15 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 8467/107148 objects degraded (7.902%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:08:15.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:15.795 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:15.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:15.840 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:08:15.863 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:08:15.896 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:15.924 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:15.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:15.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:16.010 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:08:16.025 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:16.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:16.113 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:16.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:16.214 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:16.282 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:16.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:16.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:16.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:16.423 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:16.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:16.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:16.477 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:08:16.524 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:16.559 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:16 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2471509691' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:08:16.559 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:16 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:08:16.559 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:16 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:08:16.559 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:16 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2471509691' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:08:16.560 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:16 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:08:16.560 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:16 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:08:16.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:16.567 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:16.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:16.668 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:16.701 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:16.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:16.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:16 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2471509691' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:08:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:16 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:08:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:16 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:08:16.803 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:16.835 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:16.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:16.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:16.937 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:16.965 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:16.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:16.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:17.067 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:17.103 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:17.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:17.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:17.205 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:17.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:17.235 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:17.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:17.336 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:17.364 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:17.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:17.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:17.467 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:17.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:17.496 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:17.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:17.598 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:17.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:17.632 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:17.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:17.733 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:17 smithi165 ceph-mon[105517]: pgmap v2735: 109 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 102 active+clean; 2.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 288 op/s; 7986/105708 objects degraded (7.555%); 259/105708 objects misplaced (0.245%); 12 MiB/s, 319 objects/s recovering 2024-08-17T18:08:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:17 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:08:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:17 smithi165 ceph-mon[105517]: osdmap e1512: 8 total, 8 up, 7 in 2024-08-17T18:08:17.762 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:17.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:17.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:17.864 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:17.894 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:17.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:17.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:17.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:17 smithi136 ceph-mon[109861]: pgmap v2735: 109 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 102 active+clean; 2.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 288 op/s; 7986/105708 objects degraded (7.555%); 259/105708 objects misplaced (0.245%); 12 MiB/s, 319 objects/s recovering 2024-08-17T18:08:17.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:17 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:08:17.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:17 smithi136 ceph-mon[109861]: osdmap e1512: 8 total, 8 up, 7 in 2024-08-17T18:08:17.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:17 smithi136 ceph-mon[115078]: pgmap v2735: 109 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 102 active+clean; 2.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 288 op/s; 7986/105708 objects degraded (7.555%); 259/105708 objects misplaced (0.245%); 12 MiB/s, 319 objects/s recovering 2024-08-17T18:08:17.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:17 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:08:17.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:17 smithi136 ceph-mon[115078]: osdmap e1512: 8 total, 8 up, 7 in 2024-08-17T18:08:17.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:08:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:08:17] "GET /metrics HTTP/1.1" 200 36673 "" "Prometheus/2.51.0" 2024-08-17T18:08:17.997 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:18.033 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:18.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:18.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:18.135 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:18.177 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:18.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:18.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:18.278 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:18.310 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:18.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:18.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:18.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:18.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:18.444 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:18.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:18.545 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:18.594 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:18.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:18.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:18.696 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:18.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:18.732 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:18.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:18.834 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:18.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:18.869 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:18.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:18.970 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:18.997 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:18.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:18.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:19.099 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:19.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:19.141 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:19.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:19.242 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:19.277 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:19.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:19.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:19.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:19.417 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:19.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:19.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:19.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:19.544 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:19.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:19.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:19.646 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:19.698 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:19.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:19.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:19 smithi165 ceph-mon[105517]: pgmap v2737: 109 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 452 op/s; 6168/100095 objects degraded (6.162%); 2280/100095 objects misplaced (2.278%); 16 MiB/s, 400 objects/s recovering 2024-08-17T18:08:19.773 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:19 smithi136 ceph-mon[109861]: pgmap v2737: 109 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 452 op/s; 6168/100095 objects degraded (6.162%); 2280/100095 objects misplaced (2.278%); 16 MiB/s, 400 objects/s recovering 2024-08-17T18:08:19.773 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:19 smithi136 ceph-mon[115078]: pgmap v2737: 109 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 452 op/s; 6168/100095 objects degraded (6.162%); 2280/100095 objects misplaced (2.278%); 16 MiB/s, 400 objects/s recovering 2024-08-17T18:08:19.800 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:19.836 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:19.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:19.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:19.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:19.963 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:19.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:19.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:20.065 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:20.117 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:20.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:20.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:20.219 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:20.250 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:20.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:20.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:20.296 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:08:20.296 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:08:20.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:20.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:20.385 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:20.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:20.486 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:20.515 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:20.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:20.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:20.616 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:20.658 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:20.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:20.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:20.760 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:20 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 6168/100095 objects degraded (6.162%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:08:20.781 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:20.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:20.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:20.883 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:20.912 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:20.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:20.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:20.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:20 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 6168/100095 objects degraded (6.162%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:08:20.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:20 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 6168/100095 objects degraded (6.162%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:08:20.966 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:08:21.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:21.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:21.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:21.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:21.145 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:21.179 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:21.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:21.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:21.281 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:21.325 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:21.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:21.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:21.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:21.461 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:21.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:21.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:21.562 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:21.607 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:21.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:21.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:21.709 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:21.730 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:21.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:21.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:21 smithi165 ceph-mon[105517]: pgmap v2738: 109 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 461 op/s; 6021/97227 objects degraded (6.193%); 2063/97227 objects misplaced (2.122%); 15 MiB/s, 348 objects/s recovering 2024-08-17T18:08:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:08:21.832 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:21.863 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:21.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:21.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:21 smithi136 ceph-mon[109861]: pgmap v2738: 109 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 461 op/s; 6021/97227 objects degraded (6.193%); 2063/97227 objects misplaced (2.122%); 15 MiB/s, 348 objects/s recovering 2024-08-17T18:08:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:08:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:21 smithi136 ceph-mon[115078]: pgmap v2738: 109 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 461 op/s; 6021/97227 objects degraded (6.193%); 2063/97227 objects misplaced (2.122%); 15 MiB/s, 348 objects/s recovering 2024-08-17T18:08:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:08:21.964 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:22.000 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:22.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:22.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:22.101 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:22.141 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:22.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:22.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:22.243 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:22.275 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:22.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:22.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:22.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:22.399 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:22.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:22.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:22.501 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:22.539 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:22.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:22.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:22.641 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:22.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:22.683 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:22.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:22.784 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:22.812 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:22.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:22.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:22.914 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:22.943 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:22.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:22.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:23.045 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:23.071 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:23.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:23.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:23.104 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:08:23.173 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:23.200 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:23.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:23.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:23.302 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:23.348 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:23.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:23.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:23.450 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:23.476 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:23.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:23.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:23.578 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:23.604 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:23.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:23.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:23.642 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:23 smithi136 ceph-mon[109861]: pgmap v2739: 109 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 770 op/s; 6021/87051 objects degraded (6.917%); 1763/87051 objects misplaced (2.025%); 11 MiB/s, 193 objects/s recovering 2024-08-17T18:08:23.642 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:23 smithi136 ceph-mon[115078]: pgmap v2739: 109 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 770 op/s; 6021/87051 objects degraded (6.917%); 1763/87051 objects misplaced (2.025%); 11 MiB/s, 193 objects/s recovering 2024-08-17T18:08:23.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:23.733 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:23.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:23.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:23.834 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:23.857 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:23.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:23.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:23.959 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:24.006 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:24.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:24.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:24.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:23 smithi165 ceph-mon[105517]: pgmap v2739: 109 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 770 op/s; 6021/87051 objects degraded (6.917%); 1763/87051 objects misplaced (2.025%); 11 MiB/s, 193 objects/s recovering 2024-08-17T18:08:24.108 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:24.156 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:24.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:24.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:24.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:24.295 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:24.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:24.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:24.397 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:24.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-17T18:08:24.455 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:24.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:24.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:24.538 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.4 to 1 (10000) 2024-08-17T18:08:24.557 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:24.588 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:24.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:24.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:24.689 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:24.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:24 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4181218479' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T18:08:24.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:24 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4181218479' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T18:08:24.733 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:24.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:24.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:24.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:24 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4181218479' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T18:08:24.764 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:08:24.835 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:24.870 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:24.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:24.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:24.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:25.000 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:25.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:25.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:25.102 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:25.130 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:25.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:25.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:25.232 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:25.272 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:25.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:25.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:25.374 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:25.407 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:25.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:25.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:25.509 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:25.549 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:25.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:25.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:25.652 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:25.687 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:25.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:25.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:25.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:25 smithi136 ceph-mon[109861]: pgmap v2740: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 824 op/s; 5738/82404 objects degraded (6.963%); 1721/82404 objects misplaced (2.088%); 9.9 MiB/s, 215 objects/s recovering 2024-08-17T18:08:25.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:25 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4181218479' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-17T18:08:25.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:25 smithi136 ceph-mon[109861]: osdmap e1513: 8 total, 8 up, 7 in 2024-08-17T18:08:25.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:25 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5738/82404 objects degraded (6.963%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:08:25.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:25 smithi136 ceph-mon[115078]: pgmap v2740: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 824 op/s; 5738/82404 objects degraded (6.963%); 1721/82404 objects misplaced (2.088%); 9.9 MiB/s, 215 objects/s recovering 2024-08-17T18:08:25.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:25 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4181218479' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-17T18:08:25.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:25 smithi136 ceph-mon[115078]: osdmap e1513: 8 total, 8 up, 7 in 2024-08-17T18:08:25.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:25 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5738/82404 objects degraded (6.963%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:08:25.789 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:25.808 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 6 1 2024-08-17T18:08:25.827 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:25.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:25.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:25.870 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:08:25.929 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:25.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:25.979 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:25.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:25 smithi165 ceph-mon[105517]: pgmap v2740: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 824 op/s; 5738/82404 objects degraded (6.963%); 1721/82404 objects misplaced (2.088%); 9.9 MiB/s, 215 objects/s recovering 2024-08-17T18:08:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:25 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4181218479' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-17T18:08:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:25 smithi165 ceph-mon[105517]: osdmap e1513: 8 total, 8 up, 7 in 2024-08-17T18:08:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:25 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5738/82404 objects degraded (6.963%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:08:26.080 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:26.109 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:26.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:26.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:26.211 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:26.238 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:26.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:26.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:26.339 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:26.371 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:26.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:26.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:26.472 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:26.497 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:26.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:26.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:26.599 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:26.646 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:26.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:26.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:26.748 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:26.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:26.787 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:26.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:26.888 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:26.930 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:26.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:26.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:27.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:27.054 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:27.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:27.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:27.156 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:27.182 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:27.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:27.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:27.284 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:27.321 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:27.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:27.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:27.423 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:27.461 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:27.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:27.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:27.563 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:27.594 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:27.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:27.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:27.696 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:27.723 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:27.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:27.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:27 smithi165 ceph-mon[105517]: pgmap v2742: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 919 op/s; 5738/78657 objects degraded (7.295%); 1721/78657 objects misplaced (2.188%); 7.1 MiB/s, 170 objects/s recovering 2024-08-17T18:08:27.825 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:27.857 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:27.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:27.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:27.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:27 smithi136 ceph-mon[109861]: pgmap v2742: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 919 op/s; 5738/78657 objects degraded (7.295%); 1721/78657 objects misplaced (2.188%); 7.1 MiB/s, 170 objects/s recovering 2024-08-17T18:08:27.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:08:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:08:27] "GET /metrics HTTP/1.1" 200 36648 "" "Prometheus/2.51.0" 2024-08-17T18:08:27.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:27 smithi136 ceph-mon[115078]: pgmap v2742: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 919 op/s; 5738/78657 objects degraded (7.295%); 1721/78657 objects misplaced (2.188%); 7.1 MiB/s, 170 objects/s recovering 2024-08-17T18:08:27.959 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:28.002 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:28.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:28.089 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:08:28.089 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:08:28.104 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:28.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:28.142 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:28.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:28.244 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:28.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:28.272 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:28.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:28.373 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:28.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:28.413 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:28.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:28.514 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:28.543 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:28.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:28.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:28.645 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:28.673 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:28.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:28.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:28.733 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-17T18:08:28.775 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:28.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:28.806 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:28.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:28.907 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:28.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:28.945 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:28.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:29.016 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:08:29.047 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:29.085 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:29.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:29.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:29.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:29.211 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:29.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:29.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:29.313 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:29.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:29.348 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:29.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:29.449 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:29.478 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:29.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:29.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:29.579 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:29.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:29.610 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:29.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:29.619 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:29 smithi136 ceph-mon[115078]: pgmap v2743: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 991 op/s; 5738/70350 objects degraded (8.156%); 1413/70350 objects misplaced (2.009%); 4.3 MiB/s, 151 objects/s recovering 2024-08-17T18:08:29.619 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:29 smithi136 ceph-mon[109861]: pgmap v2743: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 991 op/s; 5738/70350 objects degraded (8.156%); 1413/70350 objects misplaced (2.009%); 4.3 MiB/s, 151 objects/s recovering 2024-08-17T18:08:29.711 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:29.741 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:29.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:29.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:29 smithi165 ceph-mon[105517]: pgmap v2743: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 991 op/s; 5738/70350 objects degraded (8.156%); 1413/70350 objects misplaced (2.009%); 4.3 MiB/s, 151 objects/s recovering 2024-08-17T18:08:29.843 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:29.885 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:29.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:29.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:29.986 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:08:29.987 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:30.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:30.013 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:30.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:30.114 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:30.163 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:30.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:30.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:30.265 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:30.292 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:30.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:30.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:30.394 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:30.460 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:30.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:30.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:30.563 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:30.590 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:30.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:30.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:30.681 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:30 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5738/70350 objects degraded (8.156%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:08:30.692 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:30.723 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:30.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:30.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:30.825 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:30.864 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:30 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5738/70350 objects degraded (8.156%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:08:30.871 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:30 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5738/70350 objects degraded (8.156%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:08:30.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:30.876 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:30.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:30.977 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:31.025 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:31.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:31.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:31.127 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:31.171 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:31.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:31.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:31.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:31.302 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:31.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:31.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:31.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:31.433 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:31.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:31.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:31.534 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:31.564 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:31.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:31.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:31.666 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:31.696 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:31.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:31.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:31.798 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:31.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:31.836 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:31.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:31.876 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:31 smithi136 ceph-mon[109861]: pgmap v2744: 109 pgs: 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 988 op/s; 5738/67563 objects degraded (8.493%); 1413/67563 objects misplaced (2.091%); 2.0 MiB/s, 137 objects/s recovering 2024-08-17T18:08:31.877 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:31 smithi136 ceph-mon[115078]: pgmap v2744: 109 pgs: 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 988 op/s; 5738/67563 objects degraded (8.493%); 1413/67563 objects misplaced (2.091%); 2.0 MiB/s, 137 objects/s recovering 2024-08-17T18:08:31.937 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:31.963 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:31.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:31.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:31 smithi165 ceph-mon[105517]: pgmap v2744: 109 pgs: 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 988 op/s; 5738/67563 objects degraded (8.493%); 1413/67563 objects misplaced (2.091%); 2.0 MiB/s, 137 objects/s recovering 2024-08-17T18:08:32.065 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:32.093 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:32.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:32.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:32.195 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:32.235 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:32.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:32.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:32.337 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:32.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:32.361 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:32.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:32.462 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:32.507 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:32.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:32.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:32.576 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:08:32.609 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:32.634 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:32.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:32.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:32.736 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:32.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:32.780 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:32.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:32.881 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:32.913 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:32 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2224597222' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:08:32.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:32 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:08:32.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:32 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2026246607' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:08:32.920 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:32.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:32.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:32.936 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:32 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2224597222' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:08:32.936 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:32 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:08:32.936 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:32 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2026246607' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:08:32.937 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:32 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2224597222' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:08:32.937 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:32 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:08:32.937 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:32 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2026246607' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:08:33.022 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:33.049 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:33.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:33.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:33.151 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:33.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:33.178 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:33.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:33.244 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:08:33.244 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:08:33.244 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-08-17T18:08:33.244 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-08-17T18:08:33.245 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:08:33.279 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:33.315 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:33.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:33.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:33.416 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:33.447 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:33.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:33.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:33.548 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:33.578 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:33.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:33.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:33.639 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:33 smithi136 ceph-mon[109861]: pgmap v2745: 109 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1.2 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 990 op/s; 5738/57333 objects degraded (10.008%); 1113/57333 objects misplaced (1.941%); 2.0 MiB/s, 137 objects/s recovering 2024-08-17T18:08:33.640 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:33 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:08:33.640 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:33 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:08:33.640 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:33 smithi136 ceph-mon[109861]: osdmap e1514: 8 total, 8 up, 7 in 2024-08-17T18:08:33.640 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:33 smithi136 ceph-mon[115078]: pgmap v2745: 109 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1.2 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 990 op/s; 5738/57333 objects degraded (10.008%); 1113/57333 objects misplaced (1.941%); 2.0 MiB/s, 137 objects/s recovering 2024-08-17T18:08:33.640 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:33 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:08:33.640 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:33 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:08:33.641 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:33 smithi136 ceph-mon[115078]: osdmap e1514: 8 total, 8 up, 7 in 2024-08-17T18:08:33.679 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:33.705 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:33.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:33.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:33.807 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:33.848 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:33.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:33.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:33.916 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:33 smithi165 ceph-mon[105517]: pgmap v2745: 109 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1.2 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 990 op/s; 5738/57333 objects degraded (10.008%); 1113/57333 objects misplaced (1.941%); 2.0 MiB/s, 137 objects/s recovering 2024-08-17T18:08:33.916 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:33 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:08:33.916 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:33 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:08:33.916 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:33 smithi165 ceph-mon[105517]: osdmap e1514: 8 total, 8 up, 7 in 2024-08-17T18:08:33.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-17T18:08:33.951 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:34.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:34.006 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:34.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:34.107 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:34.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:34.149 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:34.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:34.239 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:08:34.250 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:34.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:34.290 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:34.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:34.391 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:34.421 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:34.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:34.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:34.523 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:34.560 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:34.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:34.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:34.662 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:34.700 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:34.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:34.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:34.802 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:34.828 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:34.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:34.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:34.930 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:34.977 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:34.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:34.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:35.049 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:08:35.050 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:08:35.050 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:08:35.050 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:08:35.050 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:08:35.050 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:08:35.050 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:08:35.050 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:08:35.050 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:08:35.050 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:08:35.051 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:08:35.051 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:08:35.051 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:08:35.051 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:08:35.051 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:08:35.051 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:08:35.079 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:35.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:35.129 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:35.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:35.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:35 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:35.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:35 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:35.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:35 smithi136 ceph-mon[109861]: pgmap v2747: 109 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1.1 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s; 5740/52752 objects degraded (10.881%); 1113/52752 objects misplaced (2.110%); 0 B/s, 115 objects/s recovering 2024-08-17T18:08:35.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:35 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5738/57333 objects degraded (10.008%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:08:35.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:35 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:35.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:35 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:35.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:35 smithi136 ceph-mon[115078]: pgmap v2747: 109 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1.1 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s; 5740/52752 objects degraded (10.881%); 1113/52752 objects misplaced (2.110%); 0 B/s, 115 objects/s recovering 2024-08-17T18:08:35.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:35 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5738/57333 objects degraded (10.008%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:08:35.230 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:35.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:35.275 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:35.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:35.376 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:35.402 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:35.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:35.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:35.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:35.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:35 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:35.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:35 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:35.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:35 smithi165 ceph-mon[105517]: pgmap v2747: 109 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1.1 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s; 5740/52752 objects degraded (10.881%); 1113/52752 objects misplaced (2.110%); 0 B/s, 115 objects/s recovering 2024-08-17T18:08:35.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:35 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5738/57333 objects degraded (10.008%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:08:35.550 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:35.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:35.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:35.651 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:35.684 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:35.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:35.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:35.786 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:35.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:35.820 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:35.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:35.921 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:35.968 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:35.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:35.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:36.056 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.6 to 1 (10000) 2024-08-17T18:08:36.071 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:36.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:36.107 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:36.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:36.146 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:08:36.146 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:08:36.146 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:36.146 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:36 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3041231018' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T18:08:36.146 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:36 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T18:08:36.169 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:08:36.169 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T18:08:36.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T18:08:36.169 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:08:36.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.169 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180544 54.e 54:79b4bc87:::benchmark_data_smithi136_273969_object57659:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1399)", 2024-08-17T18:08:36.169 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:22.496156+0000", 2024-08-17T18:08:36.169 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 433.67059121800003, 2024-08-17T18:08:36.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0802525319999998, 2024-08-17T18:08:36.170 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:36.170 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:36.170 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:36.170 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:08:36.170 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:08:36.170 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180544 2024-08-17T18:08:36.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.170 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:36.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:36.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.496156+0000", 2024-08-17T18:08:36.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:36.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.496156+0000", 2024-08-17T18:08:36.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:36.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.496154+0000", 2024-08-17T18:08:36.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:08:36.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:36.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.496160+0000", 2024-08-17T18:08:36.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7019999999999998e-06 2024-08-17T18:08:36.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:36.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.496172+0000", 2024-08-17T18:08:36.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1817000000000001e-05 2024-08-17T18:08:36.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:36.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.496180+0000", 2024-08-17T18:08:36.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4309999999999994e-06 2024-08-17T18:08:36.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.496894+0000", 2024-08-17T18:08:36.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00071417699999999998 2024-08-17T18:08:36.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:36.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.496906+0000", 2024-08-17T18:08:36.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1433999999999999e-05 2024-08-17T18:08:36.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.483332+0000", 2024-08-17T18:08:36.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98642643100000005 2024-08-17T18:08:36.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:36.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.483346+0000", 2024-08-17T18:08:36.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3893999999999999e-05 2024-08-17T18:08:36.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.562868+0000", 2024-08-17T18:08:36.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0795215810000001 2024-08-17T18:08:36.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:36.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.562992+0000", 2024-08-17T18:08:36.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012483399999999999 2024-08-17T18:08:36.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T18:08:36.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.563118+0000", 2024-08-17T18:08:36.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000125649 2024-08-17T18:08:36.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:36.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.571148+0000", 2024-08-17T18:08:36.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080301169999999998 2024-08-17T18:08:36.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.571332+0000", 2024-08-17T18:08:36.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018406299999999999 2024-08-17T18:08:36.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.576190+0000", 2024-08-17T18:08:36.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048579879999999997 2024-08-17T18:08:36.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:36.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.576244+0000", 2024-08-17T18:08:36.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3281e-05 2024-08-17T18:08:36.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:36.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.576408+0000", 2024-08-17T18:08:36.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016452799999999999 2024-08-17T18:08:36.177 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.177 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:36.177 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.178 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180550 54.e 54:7348762f:::benchmark_data_smithi136_273969_object57665:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1399)", 2024-08-17T18:08:36.178 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:22.501734+0000", 2024-08-17T18:08:36.178 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 433.66501299999999, 2024-08-17T18:08:36.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.074907434, 2024-08-17T18:08:36.178 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:36.178 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:36.178 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:36.178 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:08:36.178 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:08:36.178 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180550 2024-08-17T18:08:36.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.178 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:36.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:36.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.501734+0000", 2024-08-17T18:08:36.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:36.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.501734+0000", 2024-08-17T18:08:36.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:36.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.501733+0000", 2024-08-17T18:08:36.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:08:36.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:36.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.501741+0000", 2024-08-17T18:08:36.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3089999999999996e-06 2024-08-17T18:08:36.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:36.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.501753+0000", 2024-08-17T18:08:36.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2536e-05 2024-08-17T18:08:36.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:36.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.501761+0000", 2024-08-17T18:08:36.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6109999999999996e-06 2024-08-17T18:08:36.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.501818+0000", 2024-08-17T18:08:36.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7124999999999999e-05 2024-08-17T18:08:36.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:36.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.501827+0000", 2024-08-17T18:08:36.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1090000000000004e-06 2024-08-17T18:08:36.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.483732+0000", 2024-08-17T18:08:36.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.981904844 2024-08-17T18:08:36.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:36.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.483743+0000", 2024-08-17T18:08:36.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1219e-05 2024-08-17T18:08:36.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.567081+0000", 2024-08-17T18:08:36.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0833372670000001 2024-08-17T18:08:36.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:36.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.567221+0000", 2024-08-17T18:08:36.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000140679 2024-08-17T18:08:36.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T18:08:36.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.567365+0000", 2024-08-17T18:08:36.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014355699999999999 2024-08-17T18:08:36.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:36.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.571365+0000", 2024-08-17T18:08:36.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040001769999999997 2024-08-17T18:08:36.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.571421+0000", 2024-08-17T18:08:36.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5908000000000001e-05 2024-08-17T18:08:36.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.576455+0000", 2024-08-17T18:08:36.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050341129999999998 2024-08-17T18:08:36.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:36.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.576509+0000", 2024-08-17T18:08:36.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3770000000000002e-05 2024-08-17T18:08:36.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:36.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.576641+0000", 2024-08-17T18:08:36.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013238200000000001 2024-08-17T18:08:36.186 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.186 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:36.186 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.186 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:192585 54.13 54.b6293a53 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1415)", 2024-08-17T18:08:36.186 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:47.034029+0000", 2024-08-17T18:08:36.186 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 409.13271813300003, 2024-08-17T18:08:36.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.59444498999999995, 2024-08-17T18:08:36.186 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:36.186 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:08:36.187 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:36.187 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:08:36.187 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:08:36.187 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 192585 2024-08-17T18:08:36.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.187 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:36.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:36.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.034029+0000", 2024-08-17T18:08:36.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:36.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.034029+0000", 2024-08-17T18:08:36.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:36.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.034027+0000", 2024-08-17T18:08:36.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:08:36.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:36.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.034034+0000", 2024-08-17T18:08:36.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2429999999999999e-06 2024-08-17T18:08:36.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:36.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.034044+0000", 2024-08-17T18:08:36.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7319999999999993e-06 2024-08-17T18:08:36.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:36.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.034051+0000", 2024-08-17T18:08:36.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.99e-06 2024-08-17T18:08:36.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.034143+0000", 2024-08-17T18:08:36.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1792999999999993e-05 2024-08-17T18:08:36.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:36.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.034152+0000", 2024-08-17T18:08:36.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5839999999999995e-06 2024-08-17T18:08:36.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.628464+0000", 2024-08-17T18:08:36.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.59431237299999995 2024-08-17T18:08:36.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:36.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.628474+0000", 2024-08-17T18:08:36.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6760000000000002e-06 2024-08-17T18:08:36.191 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.191 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:36.191 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.191 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:192628 54.13 54.495ef113 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1415)", 2024-08-17T18:08:36.191 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:47.073716+0000", 2024-08-17T18:08:36.191 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 409.09303071300002, 2024-08-17T18:08:36.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.55514982599999996, 2024-08-17T18:08:36.191 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:36.191 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:08:36.191 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:36.191 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:08:36.191 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:08:36.192 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 192628 2024-08-17T18:08:36.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.192 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:36.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:36.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.073716+0000", 2024-08-17T18:08:36.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:36.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.073716+0000", 2024-08-17T18:08:36.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:36.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.073715+0000", 2024-08-17T18:08:36.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:08:36.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:36.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.073719+0000", 2024-08-17T18:08:36.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5990000000000002e-06 2024-08-17T18:08:36.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:36.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.073731+0000", 2024-08-17T18:08:36.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2166e-05 2024-08-17T18:08:36.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:36.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.073737+0000", 2024-08-17T18:08:36.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4670000000000003e-06 2024-08-17T18:08:36.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.076255+0000", 2024-08-17T18:08:36.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025182049999999999 2024-08-17T18:08:36.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:36.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.076266+0000", 2024-08-17T18:08:36.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0944000000000001e-05 2024-08-17T18:08:36.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.628858+0000", 2024-08-17T18:08:36.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.55259170000000002 2024-08-17T18:08:36.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:36.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.628866+0000", 2024-08-17T18:08:36.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.32e-06 2024-08-17T18:08:36.196 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.196 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:36.196 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.196 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:192676 54.13 54.c2744833 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1415)", 2024-08-17T18:08:36.196 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:47.124520+0000", 2024-08-17T18:08:36.196 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 409.04222653599999, 2024-08-17T18:08:36.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50437072800000005, 2024-08-17T18:08:36.196 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:36.196 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:08:36.196 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:36.196 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:08:36.196 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:08:36.197 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 192676 2024-08-17T18:08:36.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.197 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:36.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:36.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.124520+0000", 2024-08-17T18:08:36.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:36.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.124520+0000", 2024-08-17T18:08:36.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:36.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.124519+0000", 2024-08-17T18:08:36.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:08:36.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:36.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.124524+0000", 2024-08-17T18:08:36.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.33e-06 2024-08-17T18:08:36.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:36.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.124535+0000", 2024-08-17T18:08:36.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1293e-05 2024-08-17T18:08:36.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:36.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.124541+0000", 2024-08-17T18:08:36.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8279999999999999e-06 2024-08-17T18:08:36.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.124586+0000", 2024-08-17T18:08:36.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4860000000000001e-05 2024-08-17T18:08:36.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:36.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.124593+0000", 2024-08-17T18:08:36.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7270000000000007e-06 2024-08-17T18:08:36.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.628884+0000", 2024-08-17T18:08:36.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50429032299999998 2024-08-17T18:08:36.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:36.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.628891+0000", 2024-08-17T18:08:36.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1389999999999998e-06 2024-08-17T18:08:36.200 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.200 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:36.201 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.201 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:192684 54.13 54.8577ef93 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1415)", 2024-08-17T18:08:36.201 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:47.134015+0000", 2024-08-17T18:08:36.201 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 409.03273181399999, 2024-08-17T18:08:36.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.49489964800000003, 2024-08-17T18:08:36.201 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:36.201 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:08:36.201 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:36.201 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:08:36.201 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:08:36.201 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 192684 2024-08-17T18:08:36.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.202 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:36.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:36.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.134015+0000", 2024-08-17T18:08:36.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:36.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.134015+0000", 2024-08-17T18:08:36.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:36.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.134013+0000", 2024-08-17T18:08:36.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:08:36.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:36.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.134021+0000", 2024-08-17T18:08:36.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6499999999999996e-06 2024-08-17T18:08:36.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:36.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.134039+0000", 2024-08-17T18:08:36.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8229000000000001e-05 2024-08-17T18:08:36.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:36.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.134048+0000", 2024-08-17T18:08:36.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3979999999999999e-06 2024-08-17T18:08:36.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.134628+0000", 2024-08-17T18:08:36.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00058073499999999998 2024-08-17T18:08:36.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:36.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.134637+0000", 2024-08-17T18:08:36.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7280000000000001e-06 2024-08-17T18:08:36.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.628907+0000", 2024-08-17T18:08:36.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.494270286 2024-08-17T18:08:36.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:36.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.628915+0000", 2024-08-17T18:08:36.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2960000000000001e-06 2024-08-17T18:08:36.206 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.206 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:36.206 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.206 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:192732 54.13 54.8359acb3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1415)", 2024-08-17T18:08:36.206 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:47.181840+0000", 2024-08-17T18:08:36.206 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 408.98490690099999, 2024-08-17T18:08:36.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.44709966099999998, 2024-08-17T18:08:36.206 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:36.206 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:08:36.207 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:36.207 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:08:36.207 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:08:36.207 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 192732 2024-08-17T18:08:36.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.207 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:36.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:36.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.181840+0000", 2024-08-17T18:08:36.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:36.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.181840+0000", 2024-08-17T18:08:36.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:36.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.181839+0000", 2024-08-17T18:08:36.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:08:36.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:36.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.181843+0000", 2024-08-17T18:08:36.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2990000000000001e-06 2024-08-17T18:08:36.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:36.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.181849+0000", 2024-08-17T18:08:36.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7880000000000001e-06 2024-08-17T18:08:36.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:36.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.181855+0000", 2024-08-17T18:08:36.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2449999999999998e-06 2024-08-17T18:08:36.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.181913+0000", 2024-08-17T18:08:36.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8576000000000003e-05 2024-08-17T18:08:36.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:36.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.181921+0000", 2024-08-17T18:08:36.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6290000000000001e-06 2024-08-17T18:08:36.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.628932+0000", 2024-08-17T18:08:36.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.44701166799999997 2024-08-17T18:08:36.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:36.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.628940+0000", 2024-08-17T18:08:36.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0360000000000001e-06 2024-08-17T18:08:36.211 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.211 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:36.211 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.211 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:192747 54.13 54.a52b3c53 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1415)", 2024-08-17T18:08:36.211 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:47.197012+0000", 2024-08-17T18:08:36.211 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 408.96973502899999, 2024-08-17T18:08:36.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.43195201, 2024-08-17T18:08:36.212 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:36.212 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:08:36.212 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:36.212 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:08:36.212 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:08:36.212 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 192747 2024-08-17T18:08:36.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.212 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:36.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:36.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.197012+0000", 2024-08-17T18:08:36.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:36.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.197012+0000", 2024-08-17T18:08:36.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:36.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.197011+0000", 2024-08-17T18:08:36.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:08:36.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:36.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.197015+0000", 2024-08-17T18:08:36.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4719999999999997e-06 2024-08-17T18:08:36.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:36.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.197026+0000", 2024-08-17T18:08:36.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0715e-05 2024-08-17T18:08:36.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:36.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.197031+0000", 2024-08-17T18:08:36.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9690000000000002e-06 2024-08-17T18:08:36.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.197103+0000", 2024-08-17T18:08:36.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1870000000000001e-05 2024-08-17T18:08:36.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:36.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.197112+0000", 2024-08-17T18:08:36.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.054e-06 2024-08-17T18:08:36.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.628957+0000", 2024-08-17T18:08:36.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.431844797 2024-08-17T18:08:36.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:36.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.628964+0000", 2024-08-17T18:08:36.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1189999999999999e-06 2024-08-17T18:08:36.216 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.216 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:36.216 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.216 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:192755 54.13 54.127f1cb3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1415)", 2024-08-17T18:08:36.216 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:47.202118+0000", 2024-08-17T18:08:36.216 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 408.96462917100001, 2024-08-17T18:08:36.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.42687145399999998, 2024-08-17T18:08:36.216 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:36.217 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:08:36.217 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:36.217 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:08:36.217 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:08:36.217 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 192755 2024-08-17T18:08:36.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.217 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:36.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:36.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.202118+0000", 2024-08-17T18:08:36.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:36.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.202118+0000", 2024-08-17T18:08:36.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:36.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.202117+0000", 2024-08-17T18:08:36.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:08:36.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:36.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.202121+0000", 2024-08-17T18:08:36.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3200000000000001e-06 2024-08-17T18:08:36.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:36.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.202130+0000", 2024-08-17T18:08:36.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7919999999999998e-06 2024-08-17T18:08:36.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:36.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.202136+0000", 2024-08-17T18:08:36.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.392e-06 2024-08-17T18:08:36.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.202180+0000", 2024-08-17T18:08:36.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3825999999999997e-05 2024-08-17T18:08:36.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:36.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.202186+0000", 2024-08-17T18:08:36.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.344e-06 2024-08-17T18:08:36.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.628981+0000", 2024-08-17T18:08:36.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.42679502800000002 2024-08-17T18:08:36.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:36.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.628989+0000", 2024-08-17T18:08:36.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4619999999999996e-06 2024-08-17T18:08:36.220 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.221 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:36.221 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.221 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:192790 54.13 54.6a7bf693 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1415)", 2024-08-17T18:08:36.221 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:47.238060+0000", 2024-08-17T18:08:36.221 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 408.928687201, 2024-08-17T18:08:36.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.390953512, 2024-08-17T18:08:36.221 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:36.221 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:08:36.221 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:36.221 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:08:36.221 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:08:36.221 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 192790 2024-08-17T18:08:36.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.222 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:36.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:36.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.238060+0000", 2024-08-17T18:08:36.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:36.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.238060+0000", 2024-08-17T18:08:36.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:36.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.238059+0000", 2024-08-17T18:08:36.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:08:36.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:36.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.238063+0000", 2024-08-17T18:08:36.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.515e-06 2024-08-17T18:08:36.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:36.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.238081+0000", 2024-08-17T18:08:36.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8088999999999999e-05 2024-08-17T18:08:36.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:36.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.238091+0000", 2024-08-17T18:08:36.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2730000000000008e-06 2024-08-17T18:08:36.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.238139+0000", 2024-08-17T18:08:36.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8229999999999997e-05 2024-08-17T18:08:36.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:36.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.238147+0000", 2024-08-17T18:08:36.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7000000000000008e-06 2024-08-17T18:08:36.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.629006+0000", 2024-08-17T18:08:36.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39085976500000003 2024-08-17T18:08:36.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:36.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:47.629013+0000", 2024-08-17T18:08:36.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7490000000000001e-06 2024-08-17T18:08:36.225 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.225 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:36.225 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.226 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:225596 54.13 54:cd19ad8a:::benchmark_data_smithi136_273969_object102711:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1432)", 2024-08-17T18:08:36.226 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:02:44.066036+0000", 2024-08-17T18:08:36.226 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 352.10071026600002, 2024-08-17T18:08:36.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.43773147800000001, 2024-08-17T18:08:36.226 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:36.226 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:36.226 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:36.226 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:08:36.226 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:08:36.226 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 225596 2024-08-17T18:08:36.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.227 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:36.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:36.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.066036+0000", 2024-08-17T18:08:36.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:36.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.066036+0000", 2024-08-17T18:08:36.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:36.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.066035+0000", 2024-08-17T18:08:36.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:08:36.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:36.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.066044+0000", 2024-08-17T18:08:36.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.755e-06 2024-08-17T18:08:36.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:36.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.066059+0000", 2024-08-17T18:08:36.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.436e-05 2024-08-17T18:08:36.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:36.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.066074+0000", 2024-08-17T18:08:36.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4979e-05 2024-08-17T18:08:36.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.115216+0000", 2024-08-17T18:08:36.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.049142369999999998 2024-08-17T18:08:36.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:36.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.116943+0000", 2024-08-17T18:08:36.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001727541 2024-08-17T18:08:36.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.148100+0000", 2024-08-17T18:08:36.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.031156461 2024-08-17T18:08:36.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:36.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.150155+0000", 2024-08-17T18:08:36.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020547400000000002 2024-08-17T18:08:36.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.481627+0000", 2024-08-17T18:08:36.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.33147232100000001 2024-08-17T18:08:36.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:36.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.482760+0000", 2024-08-17T18:08:36.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011329160000000001 2024-08-17T18:08:36.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3,6", 2024-08-17T18:08:36.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.485322+0000", 2024-08-17T18:08:36.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002562099 2024-08-17T18:08:36.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.499491+0000", 2024-08-17T18:08:36.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014168912000000001 2024-08-17T18:08:36.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:36.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.499611+0000", 2024-08-17T18:08:36.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120238 2024-08-17T18:08:36.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.500077+0000", 2024-08-17T18:08:36.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00046617400000000003 2024-08-17T18:08:36.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.500485+0000", 2024-08-17T18:08:36.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00040769899999999997 2024-08-17T18:08:36.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:36.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.501117+0000", 2024-08-17T18:08:36.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063198699999999998 2024-08-17T18:08:36.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:36.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.503768+0000", 2024-08-17T18:08:36.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002650992 2024-08-17T18:08:36.234 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.234 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:36.234 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.234 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:225630 54.13 54:c9de4b4e:::benchmark_data_smithi136_273969_object102745:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1432)", 2024-08-17T18:08:36.234 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:02:44.156620+0000", 2024-08-17T18:08:36.234 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 352.010126888, 2024-08-17T18:08:36.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.37701020200000002, 2024-08-17T18:08:36.234 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:36.234 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:36.234 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:36.235 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:08:36.235 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:08:36.235 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 225630 2024-08-17T18:08:36.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.235 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:36.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:36.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.156620+0000", 2024-08-17T18:08:36.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:36.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.156620+0000", 2024-08-17T18:08:36.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:36.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.156618+0000", 2024-08-17T18:08:36.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:08:36.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:36.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.156626+0000", 2024-08-17T18:08:36.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5839999999999997e-06 2024-08-17T18:08:36.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:36.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.156638+0000", 2024-08-17T18:08:36.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2571e-05 2024-08-17T18:08:36.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:36.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.156644+0000", 2024-08-17T18:08:36.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7459999999999997e-06 2024-08-17T18:08:36.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.195120+0000", 2024-08-17T18:08:36.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038475580000000002 2024-08-17T18:08:36.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:36.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.196666+0000", 2024-08-17T18:08:36.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001545942 2024-08-17T18:08:36.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.225928+0000", 2024-08-17T18:08:36.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.029262326000000002 2024-08-17T18:08:36.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:36.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.227723+0000", 2024-08-17T18:08:36.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017948409999999999 2024-08-17T18:08:36.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.511497+0000", 2024-08-17T18:08:36.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.283773732 2024-08-17T18:08:36.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:36.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.512657+0000", 2024-08-17T18:08:36.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001160281 2024-08-17T18:08:36.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3,6", 2024-08-17T18:08:36.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.515816+0000", 2024-08-17T18:08:36.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031589090000000001 2024-08-17T18:08:36.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.521864+0000", 2024-08-17T18:08:36.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00604851 2024-08-17T18:08:36.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.522417+0000", 2024-08-17T18:08:36.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000552573 2024-08-17T18:08:36.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.522853+0000", 2024-08-17T18:08:36.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000435726 2024-08-17T18:08:36.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:36.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.530235+0000", 2024-08-17T18:08:36.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073823810000000004 2024-08-17T18:08:36.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:36.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.530870+0000", 2024-08-17T18:08:36.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063546999999999996 2024-08-17T18:08:36.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:36.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.533630+0000", 2024-08-17T18:08:36.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027595649999999998 2024-08-17T18:08:36.242 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.242 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:36.242 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.242 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:36.243 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:08:36.243 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:08:36.243 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:36.243 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:36 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3041231018' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T18:08:36.243 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:36 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T18:08:36.244 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:08:36.244 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:08:36.244 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:08:36.244 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:36 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3041231018' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T18:08:36.244 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:36 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T18:08:36.245 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:225708 54.13 54:c8de4c3f:::benchmark_data_smithi136_273969_object102823:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1432)", 2024-08-17T18:08:36.245 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:02:44.294046+0000", 2024-08-17T18:08:36.245 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 351.87270119099998, 2024-08-17T18:08:36.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.42225885600000002, 2024-08-17T18:08:36.245 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:36.245 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:36.245 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:36.245 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:08:36.245 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:08:36.245 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 225708 2024-08-17T18:08:36.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.246 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:36.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:36.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.294046+0000", 2024-08-17T18:08:36.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:36.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.294046+0000", 2024-08-17T18:08:36.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:36.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.294044+0000", 2024-08-17T18:08:36.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:08:36.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:36.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.294053+0000", 2024-08-17T18:08:36.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0149999999999992e-06 2024-08-17T18:08:36.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:36.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.294077+0000", 2024-08-17T18:08:36.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4702000000000002e-05 2024-08-17T18:08:36.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:36.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.294086+0000", 2024-08-17T18:08:36.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6580000000000008e-06 2024-08-17T18:08:36.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.338638+0000", 2024-08-17T18:08:36.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.044552302000000002 2024-08-17T18:08:36.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:36.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.340107+0000", 2024-08-17T18:08:36.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014688329999999999 2024-08-17T18:08:36.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.370415+0000", 2024-08-17T18:08:36.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.030307844 2024-08-17T18:08:36.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:36.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.372136+0000", 2024-08-17T18:08:36.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001720638 2024-08-17T18:08:36.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.693850+0000", 2024-08-17T18:08:36.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.32171407000000002 2024-08-17T18:08:36.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:36.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.695281+0000", 2024-08-17T18:08:36.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014316120000000001 2024-08-17T18:08:36.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3,6", 2024-08-17T18:08:36.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.698396+0000", 2024-08-17T18:08:36.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031149699999999999 2024-08-17T18:08:36.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.711946+0000", 2024-08-17T18:08:36.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013549863000000001 2024-08-17T18:08:36.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:36.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.712077+0000", 2024-08-17T18:08:36.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013130100000000001 2024-08-17T18:08:36.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.712686+0000", 2024-08-17T18:08:36.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00060844200000000005 2024-08-17T18:08:36.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.713101+0000", 2024-08-17T18:08:36.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00041545800000000001 2024-08-17T18:08:36.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:36.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.713746+0000", 2024-08-17T18:08:36.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064484600000000003 2024-08-17T18:08:36.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:36.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.716304+0000", 2024-08-17T18:08:36.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00255821 2024-08-17T18:08:36.253 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.253 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:36.253 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.253 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:225726 54.13 54:cffbfbfe:::benchmark_data_smithi136_273969_object102841:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1432)", 2024-08-17T18:08:36.253 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:02:44.373017+0000", 2024-08-17T18:08:36.253 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 351.79372946500001, 2024-08-17T18:08:36.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.37556931799999999, 2024-08-17T18:08:36.254 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:36.254 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:36.254 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:36.254 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:08:36.254 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:08:36.254 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 225726 2024-08-17T18:08:36.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.254 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:36.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:36.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.373017+0000", 2024-08-17T18:08:36.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:36.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.373017+0000", 2024-08-17T18:08:36.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:36.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.373016+0000", 2024-08-17T18:08:36.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:08:36.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:36.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.373023+0000", 2024-08-17T18:08:36.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7259999999999996e-06 2024-08-17T18:08:36.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:36.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.373034+0000", 2024-08-17T18:08:36.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1188e-05 2024-08-17T18:08:36.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:36.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.373041+0000", 2024-08-17T18:08:36.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4989999999999997e-06 2024-08-17T18:08:36.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.377568+0000", 2024-08-17T18:08:36.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045265230000000002 2024-08-17T18:08:36.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:36.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.379031+0000", 2024-08-17T18:08:36.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014636060000000001 2024-08-17T18:08:36.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.399243+0000", 2024-08-17T18:08:36.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020211432000000001 2024-08-17T18:08:36.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:36.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.400699+0000", 2024-08-17T18:08:36.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014560059999999999 2024-08-17T18:08:36.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.720106+0000", 2024-08-17T18:08:36.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31940689900000002 2024-08-17T18:08:36.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:36.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.721491+0000", 2024-08-17T18:08:36.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013853730000000001 2024-08-17T18:08:36.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3,6", 2024-08-17T18:08:36.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.724408+0000", 2024-08-17T18:08:36.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029167329999999999 2024-08-17T18:08:36.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:36.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.742582+0000", 2024-08-17T18:08:36.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018174625999999999 2024-08-17T18:08:36.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.743209+0000", 2024-08-17T18:08:36.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062605199999999999 2024-08-17T18:08:36.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.743694+0000", 2024-08-17T18:08:36.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048579299999999999 2024-08-17T18:08:36.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.744395+0000", 2024-08-17T18:08:36.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00070037500000000004 2024-08-17T18:08:36.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:36.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.745057+0000", 2024-08-17T18:08:36.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00066246800000000004 2024-08-17T18:08:36.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:36.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.748587+0000", 2024-08-17T18:08:36.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035294649999999999 2024-08-17T18:08:36.261 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.261 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:36.261 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.262 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:225736 54.13 54:cb56deac:::benchmark_data_smithi136_273969_object102851:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1432)", 2024-08-17T18:08:36.262 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:02:44.403924+0000", 2024-08-17T18:08:36.262 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 351.76282273100003, 2024-08-17T18:08:36.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.38515698500000001, 2024-08-17T18:08:36.262 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:36.262 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:36.262 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:36.262 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:08:36.262 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:08:36.262 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 225736 2024-08-17T18:08:36.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.262 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:36.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:36.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.403924+0000", 2024-08-17T18:08:36.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:36.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.403924+0000", 2024-08-17T18:08:36.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:36.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.403924+0000", 2024-08-17T18:08:36.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:08:36.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:36.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.403926+0000", 2024-08-17T18:08:36.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8220000000000001e-06 2024-08-17T18:08:36.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:36.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.403930+0000", 2024-08-17T18:08:36.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5789999999999997e-06 2024-08-17T18:08:36.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:36.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.403933+0000", 2024-08-17T18:08:36.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0249999999999998e-06 2024-08-17T18:08:36.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.426304+0000", 2024-08-17T18:08:36.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022370788999999999 2024-08-17T18:08:36.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:36.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.427874+0000", 2024-08-17T18:08:36.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015702019999999999 2024-08-17T18:08:36.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.443230+0000", 2024-08-17T18:08:36.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015356025000000001 2024-08-17T18:08:36.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:36.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.444629+0000", 2024-08-17T18:08:36.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013990039999999999 2024-08-17T18:08:36.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.759686+0000", 2024-08-17T18:08:36.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31505686700000002 2024-08-17T18:08:36.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:36.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.761111+0000", 2024-08-17T18:08:36.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001424841 2024-08-17T18:08:36.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3,6", 2024-08-17T18:08:36.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.764411+0000", 2024-08-17T18:08:36.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033003540000000001 2024-08-17T18:08:36.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.784096+0000", 2024-08-17T18:08:36.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019684994000000001 2024-08-17T18:08:36.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:36.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.784248+0000", 2024-08-17T18:08:36.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000152206 2024-08-17T18:08:36.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.784722+0000", 2024-08-17T18:08:36.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00047387699999999999 2024-08-17T18:08:36.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.785172+0000", 2024-08-17T18:08:36.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00044934499999999997 2024-08-17T18:08:36.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:36.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.785935+0000", 2024-08-17T18:08:36.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00076316899999999996 2024-08-17T18:08:36.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:36.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.789081+0000", 2024-08-17T18:08:36.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031461380000000001 2024-08-17T18:08:36.270 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.270 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:36.270 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.270 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:225741 54.13 54:cf633abf:::benchmark_data_smithi136_273969_object102856:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1432)", 2024-08-17T18:08:36.270 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:02:44.423047+0000", 2024-08-17T18:08:36.270 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 351.743699596, 2024-08-17T18:08:36.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.37510067600000002, 2024-08-17T18:08:36.271 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:36.271 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:36.271 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:36.271 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:08:36.271 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:08:36.271 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 225741 2024-08-17T18:08:36.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.271 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:36.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:36.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.423047+0000", 2024-08-17T18:08:36.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:36.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.423047+0000", 2024-08-17T18:08:36.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:36.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.423045+0000", 2024-08-17T18:08:36.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:08:36.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:36.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.423052+0000", 2024-08-17T18:08:36.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.99e-06 2024-08-17T18:08:36.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:36.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.423065+0000", 2024-08-17T18:08:36.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2388000000000001e-05 2024-08-17T18:08:36.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:36.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.423080+0000", 2024-08-17T18:08:36.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5319999999999999e-05 2024-08-17T18:08:36.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.445280+0000", 2024-08-17T18:08:36.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022200280999999999 2024-08-17T18:08:36.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:36.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.446651+0000", 2024-08-17T18:08:36.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013706230000000001 2024-08-17T18:08:36.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.458226+0000", 2024-08-17T18:08:36.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011575383999999999 2024-08-17T18:08:36.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:36.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.459558+0000", 2024-08-17T18:08:36.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001331392 2024-08-17T18:08:36.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.774045+0000", 2024-08-17T18:08:36.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31448730400000002 2024-08-17T18:08:36.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:36.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.775473+0000", 2024-08-17T18:08:36.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014280289999999999 2024-08-17T18:08:36.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3,6", 2024-08-17T18:08:36.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.778691+0000", 2024-08-17T18:08:36.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032179330000000001 2024-08-17T18:08:36.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.789601+0000", 2024-08-17T18:08:36.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010909785999999999 2024-08-17T18:08:36.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.790045+0000", 2024-08-17T18:08:36.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00044432599999999999 2024-08-17T18:08:36.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.790493+0000", 2024-08-17T18:08:36.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00044791600000000003 2024-08-17T18:08:36.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:36.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.794118+0000", 2024-08-17T18:08:36.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036250330000000002 2024-08-17T18:08:36.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:36.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.794888+0000", 2024-08-17T18:08:36.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00077017100000000005 2024-08-17T18:08:36.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:36.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.798148+0000", 2024-08-17T18:08:36.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032597479999999998 2024-08-17T18:08:36.278 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.278 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:36.278 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.279 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:225746 54.13 54:cd31b559:::benchmark_data_smithi136_273969_object102861:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1432)", 2024-08-17T18:08:36.279 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:02:44.434456+0000", 2024-08-17T18:08:36.279 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 351.73229051999999, 2024-08-17T18:08:36.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39569336100000002, 2024-08-17T18:08:36.279 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:36.279 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:36.279 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:36.279 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:08:36.279 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:08:36.279 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 225746 2024-08-17T18:08:36.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.280 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:36.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:36.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.434456+0000", 2024-08-17T18:08:36.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:36.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.434456+0000", 2024-08-17T18:08:36.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:36.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.434454+0000", 2024-08-17T18:08:36.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:08:36.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:36.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.434463+0000", 2024-08-17T18:08:36.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8680000000000005e-06 2024-08-17T18:08:36.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:36.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.434480+0000", 2024-08-17T18:08:36.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6439e-05 2024-08-17T18:08:36.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:36.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.434491+0000", 2024-08-17T18:08:36.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1313e-05 2024-08-17T18:08:36.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.446806+0000", 2024-08-17T18:08:36.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012314899000000001 2024-08-17T18:08:36.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:36.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.448405+0000", 2024-08-17T18:08:36.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015990850000000001 2024-08-17T18:08:36.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.466879+0000", 2024-08-17T18:08:36.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018474411 2024-08-17T18:08:36.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:36.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.468269+0000", 2024-08-17T18:08:36.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001389248 2024-08-17T18:08:36.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.798379+0000", 2024-08-17T18:08:36.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.33011053800000001 2024-08-17T18:08:36.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:36.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.799956+0000", 2024-08-17T18:08:36.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001576556 2024-08-17T18:08:36.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3,6", 2024-08-17T18:08:36.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.803389+0000", 2024-08-17T18:08:36.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003433251 2024-08-17T18:08:36.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:36.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.820881+0000", 2024-08-17T18:08:36.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017492137000000001 2024-08-17T18:08:36.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.821430+0000", 2024-08-17T18:08:36.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00054860300000000005 2024-08-17T18:08:36.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.824675+0000", 2024-08-17T18:08:36.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032452449999999999 2024-08-17T18:08:36.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.825554+0000", 2024-08-17T18:08:36.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00087920299999999995 2024-08-17T18:08:36.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:36.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.826348+0000", 2024-08-17T18:08:36.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00079352000000000003 2024-08-17T18:08:36.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:36.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.830150+0000", 2024-08-17T18:08:36.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003801914 2024-08-17T18:08:36.286 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.286 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:36.286 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.287 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:225749 54.13 54:ce590af3:::benchmark_data_smithi136_273969_object102864:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1432)", 2024-08-17T18:08:36.287 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:02:44.444581+0000", 2024-08-17T18:08:36.287 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 351.72216558700001, 2024-08-17T18:08:36.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39119509699999999, 2024-08-17T18:08:36.287 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:36.287 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:36.287 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:36.287 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:08:36.287 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:08:36.287 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 225749 2024-08-17T18:08:36.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.287 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:36.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:36.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.444581+0000", 2024-08-17T18:08:36.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:36.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.444581+0000", 2024-08-17T18:08:36.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:36.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.444579+0000", 2024-08-17T18:08:36.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:08:36.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:36.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.444588+0000", 2024-08-17T18:08:36.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0609999999999996e-06 2024-08-17T18:08:36.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:36.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.444603+0000", 2024-08-17T18:08:36.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5238e-05 2024-08-17T18:08:36.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:36.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.444611+0000", 2024-08-17T18:08:36.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.473e-06 2024-08-17T18:08:36.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.449053+0000", 2024-08-17T18:08:36.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044420450000000004 2024-08-17T18:08:36.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:36.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.450581+0000", 2024-08-17T18:08:36.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015280789999999999 2024-08-17T18:08:36.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.475806+0000", 2024-08-17T18:08:36.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.025224628999999998 2024-08-17T18:08:36.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:36.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.477168+0000", 2024-08-17T18:08:36.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013618580000000001 2024-08-17T18:08:36.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.811953+0000", 2024-08-17T18:08:36.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.33478503799999998 2024-08-17T18:08:36.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:36.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.813244+0000", 2024-08-17T18:08:36.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001291267 2024-08-17T18:08:36.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3,6", 2024-08-17T18:08:36.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.816563+0000", 2024-08-17T18:08:36.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033184249999999998 2024-08-17T18:08:36.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:36.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.825055+0000", 2024-08-17T18:08:36.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0084921990000000006 2024-08-17T18:08:36.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.830683+0000", 2024-08-17T18:08:36.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056282499999999996 2024-08-17T18:08:36.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.831176+0000", 2024-08-17T18:08:36.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00049274300000000002 2024-08-17T18:08:36.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.831718+0000", 2024-08-17T18:08:36.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00054253999999999999 2024-08-17T18:08:36.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:36.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.832515+0000", 2024-08-17T18:08:36.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00079617299999999995 2024-08-17T18:08:36.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:36.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.835776+0000", 2024-08-17T18:08:36.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003261758 2024-08-17T18:08:36.294 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.295 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:36.295 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.295 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:225751 54.13 54:cee3306f:::benchmark_data_smithi136_273969_object102866:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1432)", 2024-08-17T18:08:36.295 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:02:44.505436+0000", 2024-08-17T18:08:36.295 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 351.66131080999997, 2024-08-17T18:08:36.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.36635309100000002, 2024-08-17T18:08:36.295 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:36.295 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:36.295 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:36.295 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:08:36.295 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:08:36.296 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 225751 2024-08-17T18:08:36.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.296 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:36.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:36.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.505436+0000", 2024-08-17T18:08:36.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:36.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.505436+0000", 2024-08-17T18:08:36.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:36.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.505435+0000", 2024-08-17T18:08:36.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:08:36.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:36.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.505443+0000", 2024-08-17T18:08:36.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7439999999999996e-06 2024-08-17T18:08:36.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:36.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.505456+0000", 2024-08-17T18:08:36.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2678e-05 2024-08-17T18:08:36.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:36.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.505462+0000", 2024-08-17T18:08:36.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4600000000000002e-06 2024-08-17T18:08:36.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.519881+0000", 2024-08-17T18:08:36.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014419566 2024-08-17T18:08:36.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:36.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.521440+0000", 2024-08-17T18:08:36.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015584170000000001 2024-08-17T18:08:36.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.533833+0000", 2024-08-17T18:08:36.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012393397 2024-08-17T18:08:36.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:36.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.535346+0000", 2024-08-17T18:08:36.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001512858 2024-08-17T18:08:36.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.841962+0000", 2024-08-17T18:08:36.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30661596299999999 2024-08-17T18:08:36.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:36.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.843285+0000", 2024-08-17T18:08:36.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013231060000000001 2024-08-17T18:08:36.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3,6", 2024-08-17T18:08:36.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.847874+0000", 2024-08-17T18:08:36.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045894940000000004 2024-08-17T18:08:36.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.863945+0000", 2024-08-17T18:08:36.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016070404 2024-08-17T18:08:36.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.864429+0000", 2024-08-17T18:08:36.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048449799999999998 2024-08-17T18:08:36.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.864862+0000", 2024-08-17T18:08:36.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00043241500000000001 2024-08-17T18:08:36.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:36.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.867624+0000", 2024-08-17T18:08:36.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002762745 2024-08-17T18:08:36.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:36.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.868495+0000", 2024-08-17T18:08:36.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00087013399999999995 2024-08-17T18:08:36.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:36.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.871789+0000", 2024-08-17T18:08:36.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032944670000000001 2024-08-17T18:08:36.303 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.303 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:36.303 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.303 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:39666 55.d 55:b581e00a:::benchmark_data_smithi136_313476_object39665:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1456)", 2024-08-17T18:08:36.303 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:04:14.320291+0000", 2024-08-17T18:08:36.304 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 261.84645591499998, 2024-08-17T18:08:36.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6548982109999999, 2024-08-17T18:08:36.304 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:36.304 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:08:36.304 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:36.304 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:08:36.304 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:08:36.304 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39666 2024-08-17T18:08:36.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.304 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:36.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:36.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.320291+0000", 2024-08-17T18:08:36.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:36.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.320291+0000", 2024-08-17T18:08:36.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:36.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:36.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.320290+0000", 2024-08-17T18:08:36.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:08:36.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:36.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.320344+0000", 2024-08-17T18:08:36.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3773999999999997e-05 2024-08-17T18:08:36.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:36.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.320353+0000", 2024-08-17T18:08:36.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.73e-06 2024-08-17T18:08:36.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:36.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.320359+0000", 2024-08-17T18:08:36.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9549999999999997e-06 2024-08-17T18:08:36.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.320390+0000", 2024-08-17T18:08:36.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0771000000000001e-05 2024-08-17T18:08:36.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:36.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.320492+0000", 2024-08-17T18:08:36.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010220199999999999 2024-08-17T18:08:36.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-08-17T18:08:36.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.320602+0000", 2024-08-17T18:08:36.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109413 2024-08-17T18:08:36.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:36.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.322056+0000", 2024-08-17T18:08:36.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014548129999999999 2024-08-17T18:08:36.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:36.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.322909+0000", 2024-08-17T18:08:36.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00085237900000000001 2024-08-17T18:08:36.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.336838+0000", 2024-08-17T18:08:36.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013928779000000001 2024-08-17T18:08:36.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:36.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.336873+0000", 2024-08-17T18:08:36.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5371999999999999e-05 2024-08-17T18:08:36.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.518312+0000", 2024-08-17T18:08:36.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.181438549 2024-08-17T18:08:36.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:36.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.518324+0000", 2024-08-17T18:08:36.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.237e-05 2024-08-17T18:08:36.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:36.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.975073+0000", 2024-08-17T18:08:36.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.456749498 2024-08-17T18:08:36.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:36.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:36.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:36.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.975189+0000", 2024-08-17T18:08:36.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011560799999999999 2024-08-17T18:08:36.310 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.310 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:36.310 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.311 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:36.311 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:36.311 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:08:36.331 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:36.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:36.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:36.433 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:36.455 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:36.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:36.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:36.557 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:36.569 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:08:36.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:36.579 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:36.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:36.681 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:36.702 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:36.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:36.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:36.804 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:36.827 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:36.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:36.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:36.929 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:36.951 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:36.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:36.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:36.955 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 0 1 2024-08-17T18:08:37.036 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:08:37.053 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:37.094 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:37.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:37.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:37.195 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:37.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:37 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T18:08:37.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:37 smithi136 ceph-mon[109861]: osdmap e1515: 8 total, 8 up, 7 in 2024-08-17T18:08:37.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:08:37.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:37 smithi136 ceph-mon[109861]: pgmap v2749: 109 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1020 MiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 889 op/s; 5740/48999 objects degraded (11.715%); 1113/48999 objects misplaced (2.271%); 0 B/s, 98 objects/s recovering 2024-08-17T18:08:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:37 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T18:08:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:37 smithi136 ceph-mon[115078]: osdmap e1515: 8 total, 8 up, 7 in 2024-08-17T18:08:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:08:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:37 smithi136 ceph-mon[115078]: pgmap v2749: 109 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1020 MiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 889 op/s; 5740/48999 objects degraded (11.715%); 1113/48999 objects misplaced (2.271%); 0 B/s, 98 objects/s recovering 2024-08-17T18:08:37.212 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-17T18:08:37.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:37.221 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:37.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:37.323 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:37.349 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:37.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:37.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:37.450 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:37.472 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:37.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:37.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:37.485 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:08:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:08:37] "GET /metrics HTTP/1.1" 200 36611 "" "Prometheus/2.51.0" 2024-08-17T18:08:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:37 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T18:08:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:37 smithi165 ceph-mon[105517]: osdmap e1515: 8 total, 8 up, 7 in 2024-08-17T18:08:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:08:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:37 smithi165 ceph-mon[105517]: pgmap v2749: 109 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 1020 MiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 889 op/s; 5740/48999 objects degraded (11.715%); 1113/48999 objects misplaced (2.271%); 0 B/s, 98 objects/s recovering 2024-08-17T18:08:37.574 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:37.597 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:37.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:37.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:37.650 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:08:37.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:37.721 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:37.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:37.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:37.823 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:37.824 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:08:37.845 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:37.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:37.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:37.940 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:08:37.947 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:38.007 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:38.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:38.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:38.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:38.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:38.158 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:38.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:38.259 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:38.284 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:38.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:38.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:38.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:38.415 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:38.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:38.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:38.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:38 smithi136 ceph-mon[109861]: osdmap e1516: 8 total, 8 up, 7 in 2024-08-17T18:08:38.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:38 smithi136 ceph-mon[115078]: osdmap e1516: 8 total, 8 up, 7 in 2024-08-17T18:08:38.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:38 smithi165 ceph-mon[105517]: osdmap e1516: 8 total, 8 up, 7 in 2024-08-17T18:08:38.517 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:38.548 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:38.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:38.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:38.650 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:38.682 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:38.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:38.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:38.784 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:38.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:38.810 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:38.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:38.911 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:38.936 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:38.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:38.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:39.038 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:39.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:39.071 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:39.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:39.172 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:39.210 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:39.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:39.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:39 smithi165 ceph-mon[105517]: osdmap e1517: 8 total, 8 up, 7 in 2024-08-17T18:08:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:39 smithi165 ceph-mon[105517]: 55.2 starting backfill to osd.6 from (0'0,0'0] MAX to 1514'8200 2024-08-17T18:08:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:39 smithi165 ceph-mon[105517]: 55.e starting backfill to osd.6 from (0'0,0'0] MAX to 1514'10416 2024-08-17T18:08:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:39 smithi165 ceph-mon[105517]: pgmap v2752: 109 pgs: 1 active+remapped+backfill_wait, 2 peering, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 891 MiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 847 op/s; 4281/42810 objects degraded (10.000%); 868/42810 objects misplaced (2.028%); 0 B/s, 106 objects/s recovering 2024-08-17T18:08:39.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:39.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:39 smithi136 ceph-mon[109861]: osdmap e1517: 8 total, 8 up, 7 in 2024-08-17T18:08:39.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:39 smithi136 ceph-mon[109861]: 55.2 starting backfill to osd.6 from (0'0,0'0] MAX to 1514'8200 2024-08-17T18:08:39.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:39 smithi136 ceph-mon[109861]: 55.e starting backfill to osd.6 from (0'0,0'0] MAX to 1514'10416 2024-08-17T18:08:39.325 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:39 smithi136 ceph-mon[109861]: pgmap v2752: 109 pgs: 1 active+remapped+backfill_wait, 2 peering, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 891 MiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 847 op/s; 4281/42810 objects degraded (10.000%); 868/42810 objects misplaced (2.028%); 0 B/s, 106 objects/s recovering 2024-08-17T18:08:39.325 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:39 smithi136 ceph-mon[115078]: osdmap e1517: 8 total, 8 up, 7 in 2024-08-17T18:08:39.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:39 smithi136 ceph-mon[115078]: 55.2 starting backfill to osd.6 from (0'0,0'0] MAX to 1514'8200 2024-08-17T18:08:39.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:39 smithi136 ceph-mon[115078]: 55.e starting backfill to osd.6 from (0'0,0'0] MAX to 1514'10416 2024-08-17T18:08:39.326 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:39 smithi136 ceph-mon[115078]: pgmap v2752: 109 pgs: 1 active+remapped+backfill_wait, 2 peering, 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 891 MiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 847 op/s; 4281/42810 objects degraded (10.000%); 868/42810 objects misplaced (2.028%); 0 B/s, 106 objects/s recovering 2024-08-17T18:08:39.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:39.353 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:39.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:39.455 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:39.500 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:39.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:39.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:39.604 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:39.641 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:39.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:39.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:39.647 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:08:39.647 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:08:39.647 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.648 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:171426 55.f 55:f004b9b8:::benchmark_data_smithi136_313476_object79440:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1518)", 2024-08-17T18:08:39.648 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:08:39.631377+0000", 2024-08-17T18:08:39.648 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.013561933999999999, 2024-08-17T18:08:39.648 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013596054999999999, 2024-08-17T18:08:39.648 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:39.648 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-08-17T18:08:39.648 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:39.648 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:08:39.648 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:08:39.648 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 171426 2024-08-17T18:08:39.648 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.648 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:39.649 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.649 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:39.649 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.631377+0000", 2024-08-17T18:08:39.649 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:39.649 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.649 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.649 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:39.649 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.631377+0000", 2024-08-17T18:08:39.649 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:39.649 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.649 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.649 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:39.649 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.631376+0000", 2024-08-17T18:08:39.650 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:08:39.650 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.650 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.650 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:39.650 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.631382+0000", 2024-08-17T18:08:39.650 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9030000000000001e-06 2024-08-17T18:08:39.650 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.650 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.650 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:39.650 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.631392+0000", 2024-08-17T18:08:39.650 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0108e-05 2024-08-17T18:08:39.650 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.651 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.651 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:39.651 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.631399+0000", 2024-08-17T18:08:39.651 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6039999999999998e-06 2024-08-17T18:08:39.651 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.651 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.651 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:39.651 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.631578+0000", 2024-08-17T18:08:39.651 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000179548 2024-08-17T18:08:39.651 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.651 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.651 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:39.652 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.631708+0000", 2024-08-17T18:08:39.652 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012970200000000001 2024-08-17T18:08:39.652 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.652 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.652 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,6", 2024-08-17T18:08:39.652 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.632197+0000", 2024-08-17T18:08:39.652 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048899800000000004 2024-08-17T18:08:39.652 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:39.652 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:39.652 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:39.652 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.652 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.653 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:171433 55.f 55:f5735e22:::benchmark_data_smithi136_313476_object79447:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1518)", 2024-08-17T18:08:39.653 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:08:39.632507+0000", 2024-08-17T18:08:39.653 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.012432161000000001, 2024-08-17T18:08:39.653 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012571064, 2024-08-17T18:08:39.653 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:39.653 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-08-17T18:08:39.653 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:39.653 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:08:39.653 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:08:39.653 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 171433 2024-08-17T18:08:39.653 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.653 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:39.653 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.653 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:39.654 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.632507+0000", 2024-08-17T18:08:39.654 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:39.654 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.654 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.654 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:39.654 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.632507+0000", 2024-08-17T18:08:39.654 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:39.654 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.654 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.654 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:39.654 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.632506+0000", 2024-08-17T18:08:39.654 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:08:39.654 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.655 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.655 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:39.655 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.632511+0000", 2024-08-17T18:08:39.655 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0170000000000004e-06 2024-08-17T18:08:39.655 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.655 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.655 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:39.655 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.632520+0000", 2024-08-17T18:08:39.655 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9360000000000005e-06 2024-08-17T18:08:39.655 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.655 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.655 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:39.655 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.632531+0000", 2024-08-17T18:08:39.655 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0798000000000001e-05 2024-08-17T18:08:39.655 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.656 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:39.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.641057+0000", 2024-08-17T18:08:39.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0085257400000000004 2024-08-17T18:08:39.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.656 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:39.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.641189+0000", 2024-08-17T18:08:39.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013203299999999999 2024-08-17T18:08:39.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.657 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,6", 2024-08-17T18:08:39.657 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.641310+0000", 2024-08-17T18:08:39.657 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121063 2024-08-17T18:08:39.657 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:39.657 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:39.657 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:39.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.657 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.45626.0:171426 55.f e1518/1492)", 2024-08-17T18:08:39.657 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:08:39.640716+0000", 2024-08-17T18:08:39.657 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.004223587, 2024-08-17T18:08:39.657 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044331220000000003, 2024-08-17T18:08:39.657 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:39.658 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-08-17T18:08:39.658 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:39.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.658 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:39.658 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.640716+0000", 2024-08-17T18:08:39.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:39.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.658 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:39.658 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.640716+0000", 2024-08-17T18:08:39.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:39.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.659 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.659 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:39.659 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.640714+0000", 2024-08-17T18:08:39.659 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:08:39.659 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.659 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.659 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:39.659 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.640722+0000", 2024-08-17T18:08:39.659 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3479999999999998e-06 2024-08-17T18:08:39.659 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.659 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.659 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:39.659 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.640737+0000", 2024-08-17T18:08:39.659 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5201999999999999e-05 2024-08-17T18:08:39.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:39.660 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.640745+0000", 2024-08-17T18:08:39.660 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2789999999999998e-06 2024-08-17T18:08:39.660 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:39.660 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:39.660 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:39.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.660 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.45626.0:171438 55.16 e1518/1517 55:6a4d71ee:::benchmark_data_smithi136_313476_object79452:head v 1518'8227, mlcod=1518'8225)", 2024-08-17T18:08:39.660 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:08:39.642302+0000", 2024-08-17T18:08:39.660 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0026376949999999998, 2024-08-17T18:08:39.660 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00289733, 2024-08-17T18:08:39.661 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:39.661 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:08:39.661 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:39.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.661 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:39.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.642302+0000", 2024-08-17T18:08:39.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:39.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.661 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:39.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.642302+0000", 2024-08-17T18:08:39.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:39.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:39.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.642301+0000", 2024-08-17T18:08:39.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:08:39.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:39.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.642308+0000", 2024-08-17T18:08:39.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.776e-06 2024-08-17T18:08:39.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:39.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.642322+0000", 2024-08-17T18:08:39.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4766999999999999e-05 2024-08-17T18:08:39.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.663 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.663 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:39.663 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.642329+0000", 2024-08-17T18:08:39.663 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7739999999999999e-06 2024-08-17T18:08:39.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.663 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.663 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:39.663 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.642397+0000", 2024-08-17T18:08:39.663 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7780999999999994e-05 2024-08-17T18:08:39.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.663 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.663 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:39.663 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.642416+0000", 2024-08-17T18:08:39.664 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9094e-05 2024-08-17T18:08:39.664 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:39.664 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:39.664 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:39.664 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.664 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.664 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.45626.0:171439 55.3 e1518/1473 55:c0451756:::benchmark_data_smithi136_313476_object79453:head v 1518'8145, mlcod=1518'8143)", 2024-08-17T18:08:39.664 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:08:39.642540+0000", 2024-08-17T18:08:39.664 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0023996339999999999, 2024-08-17T18:08:39.664 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027184179999999998, 2024-08-17T18:08:39.664 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:39.664 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:08:39.664 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:39.664 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.664 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:39.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.642540+0000", 2024-08-17T18:08:39.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:39.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:39.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.642540+0000", 2024-08-17T18:08:39.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:39.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:39.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.642539+0000", 2024-08-17T18:08:39.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:08:39.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:39.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.642544+0000", 2024-08-17T18:08:39.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8330000000000001e-06 2024-08-17T18:08:39.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:39.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.642552+0000", 2024-08-17T18:08:39.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3499999999999997e-06 2024-08-17T18:08:39.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:39.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.642558+0000", 2024-08-17T18:08:39.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3580000000000002e-06 2024-08-17T18:08:39.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.667 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:39.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.642610+0000", 2024-08-17T18:08:39.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2818999999999997e-05 2024-08-17T18:08:39.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:39.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:39.667 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:39.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:39.642628+0000", 2024-08-17T18:08:39.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7465e-05 2024-08-17T18:08:39.667 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:39.667 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:39.667 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:39.668 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:39.668 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-08-17T18:08:39.668 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 5 2024-08-17T18:08:39.668 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:08:39.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:39.769 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:39.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:39.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:39.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:39.905 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:39.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:39.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:40.007 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:40.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:40.047 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:40.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:40.148 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:40.190 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:40.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:40.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:40.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:40.315 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:40 smithi136 ceph-mon[109861]: osdmap e1518: 8 total, 8 up, 7 in 2024-08-17T18:08:40.315 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:40 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 4281/42810 objects degraded (10.000%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:08:40.315 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:40 smithi136 ceph-mon[115078]: osdmap e1518: 8 total, 8 up, 7 in 2024-08-17T18:08:40.315 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:40 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 4281/42810 objects degraded (10.000%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:08:40.328 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:40.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:40.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:40.430 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:40.472 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:40.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:40.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:40.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:40 smithi165 ceph-mon[105517]: osdmap e1518: 8 total, 8 up, 7 in 2024-08-17T18:08:40.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:40 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 4281/42810 objects degraded (10.000%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:08:40.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:40.585 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-17T18:08:40.617 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:40.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:40.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:40.719 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:40.753 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:40.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:40.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:40.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:40.888 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:40.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:40.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:40.909 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:08:40.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:41.011 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:08:41.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:41.038 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:41.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:41.139 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:41.168 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:41.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:41.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:41.269 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:41.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:41.301 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:41.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:41.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:41 smithi165 ceph-mon[105517]: pgmap v2754: 109 pgs: 1 active+remapped+backfill_wait, 2 peering, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 865 MiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 582 op/s; 1682/41595 objects degraded (4.044%); 868/41595 objects misplaced (2.087%); 0 B/s, 46 objects/s recovering 2024-08-17T18:08:41.402 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:41.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:41 smithi136 ceph-mon[109861]: pgmap v2754: 109 pgs: 1 active+remapped+backfill_wait, 2 peering, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 865 MiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 582 op/s; 1682/41595 objects degraded (4.044%); 868/41595 objects misplaced (2.087%); 0 B/s, 46 objects/s recovering 2024-08-17T18:08:41.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:41 smithi136 ceph-mon[115078]: pgmap v2754: 109 pgs: 1 active+remapped+backfill_wait, 2 peering, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 865 MiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 582 op/s; 1682/41595 objects degraded (4.044%); 868/41595 objects misplaced (2.087%); 0 B/s, 46 objects/s recovering 2024-08-17T18:08:41.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:41.448 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:41.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:41.549 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:41.580 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:41.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:41.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:41.682 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:41.702 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:41.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:41.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:41.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:41.836 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:41.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:41.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:41.938 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:41.975 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:41.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:41.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:42.077 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:42.104 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:08:42.118 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:42.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:42.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:42.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:42.247 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:42.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:42.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:42 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1339413640' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:08:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:42 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:08:42.329 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:42 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1339413640' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:08:42.329 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:42 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:08:42.329 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:42 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1339413640' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:08:42.329 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:42 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:08:42.350 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:42.392 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:42.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:42.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:42.494 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:42.533 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:42.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:42.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:42.635 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:42.671 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:42.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:42.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:42.773 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:42.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:42.823 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:42.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:42.924 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:42.957 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:42.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:42.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:43.060 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:43.081 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:43.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:43.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:43.102 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:08:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:08:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-08-17T18:08:43.102 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-08-17T18:08:43.102 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:08:43.183 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:43.226 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:43.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:43.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:43.328 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:43.365 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:43.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:43.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:43.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:43 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:08:43.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:43 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:08:43.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:43 smithi136 ceph-mon[109861]: osdmap e1519: 8 total, 8 up, 7 in 2024-08-17T18:08:43.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:43 smithi136 ceph-mon[109861]: pgmap v2756: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 722 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 620 op/s; 5035/34698 objects degraded (14.511%); 1990/34698 objects misplaced (5.735%); 1.9 MiB/s, 211 objects/s recovering 2024-08-17T18:08:43.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:43 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:08:43.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:43 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:08:43.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:43 smithi136 ceph-mon[115078]: osdmap e1519: 8 total, 8 up, 7 in 2024-08-17T18:08:43.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:43 smithi136 ceph-mon[115078]: pgmap v2756: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 722 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 620 op/s; 5035/34698 objects degraded (14.511%); 1990/34698 objects misplaced (5.735%); 1.9 MiB/s, 211 objects/s recovering 2024-08-17T18:08:43.451 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:43 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:08:43.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:43 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:08:43.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:43 smithi165 ceph-mon[105517]: osdmap e1519: 8 total, 8 up, 7 in 2024-08-17T18:08:43.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:43 smithi165 ceph-mon[105517]: pgmap v2756: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 722 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 620 op/s; 5035/34698 objects degraded (14.511%); 1990/34698 objects misplaced (5.735%); 1.9 MiB/s, 211 objects/s recovering 2024-08-17T18:08:43.467 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:43.491 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:43.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:43.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:43.593 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:43.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:43.631 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:43.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:43.732 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:43.759 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:43.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:43.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:43.802 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-17T18:08:43.861 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:43.902 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:43.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:43.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:44.004 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:44.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:44.040 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:44.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:44.136 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:08:44.141 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:44.198 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:44.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:44.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:44.300 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:44.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:44.334 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:44.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:44.435 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:44.474 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:44.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:44.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:44.575 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:44.620 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:44.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:44.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:44.722 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:44.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:44.748 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:44.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:44.849 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:44.880 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:44.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:44.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:44.982 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:45.007 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:45.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:45.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:45.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:45.136 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:45.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:45.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:45.238 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:45.267 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:45.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:45.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:45.369 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:45.410 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:45.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:45.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:45.511 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:45.540 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:45.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:45.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:45.642 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:45.683 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:45.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:45.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:45.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:45 smithi136 ceph-mon[109861]: pgmap v2757: 109 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 655 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 604 op/s; 5035/31506 objects degraded (15.981%); 1990/31506 objects misplaced (6.316%); 1.6 MiB/s, 265 objects/s recovering 2024-08-17T18:08:45.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:45 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5035/31506 objects degraded (15.981%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:08:45.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:45 smithi136 ceph-mon[115078]: pgmap v2757: 109 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 655 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 604 op/s; 5035/31506 objects degraded (15.981%); 1990/31506 objects misplaced (6.316%); 1.6 MiB/s, 265 objects/s recovering 2024-08-17T18:08:45.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:45 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5035/31506 objects degraded (15.981%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:08:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:45 smithi165 ceph-mon[105517]: pgmap v2757: 109 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 655 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 604 op/s; 5035/31506 objects degraded (15.981%); 1990/31506 objects misplaced (6.316%); 1.6 MiB/s, 265 objects/s recovering 2024-08-17T18:08:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:45 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5035/31506 objects degraded (15.981%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:08:45.785 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:45.813 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:45.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:45.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:45.914 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:45.945 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:45.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:45.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:46.047 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:46.076 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:46.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:46.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:46.177 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:46.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:46.217 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:46.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:46.318 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:46.353 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:46.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:46.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:46.455 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:46.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:46.503 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:46.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:46.604 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:46.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:46.641 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:46.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:46.742 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:46.769 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:46.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:46.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:46.871 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:46.911 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:46.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:46.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:46.952 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:08:46.952 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T18:08:46.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T18:08:46.952 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:08:46.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.952 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180126 54.10 54:0ae91374:::benchmark_data_smithi136_273969_object57241:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:08:46.952 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:21.861391+0000", 2024-08-17T18:08:46.953 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 445.08619463000002, 2024-08-17T18:08:46.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9929762970000002, 2024-08-17T18:08:46.953 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:46.953 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:46.953 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:46.953 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:08:46.953 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:08:46.953 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180126 2024-08-17T18:08:46.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.953 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:46.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:46.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.861391+0000", 2024-08-17T18:08:46.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:46.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:46.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.861391+0000", 2024-08-17T18:08:46.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:46.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:46.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.861390+0000", 2024-08-17T18:08:46.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:08:46.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:46.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.861396+0000", 2024-08-17T18:08:46.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0059999999999998e-06 2024-08-17T18:08:46.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:46.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.861414+0000", 2024-08-17T18:08:46.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8851999999999998e-05 2024-08-17T18:08:46.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:46.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.861422+0000", 2024-08-17T18:08:46.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3220000000000004e-06 2024-08-17T18:08:46.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:46.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.970423+0000", 2024-08-17T18:08:46.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.109001084 2024-08-17T18:08:46.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:46.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.973780+0000", 2024-08-17T18:08:46.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033568019999999999 2024-08-17T18:08:46.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:46.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.995958+0000", 2024-08-17T18:08:46.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022178531000000001 2024-08-17T18:08:46.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:46.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.999891+0000", 2024-08-17T18:08:46.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039327559999999999 2024-08-17T18:08:46.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:46.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542792+0000", 2024-08-17T18:08:46.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.54290070599999996 2024-08-17T18:08:46.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:46.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542797+0000", 2024-08-17T18:08:46.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9849999999999997e-06 2024-08-17T18:08:46.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:46.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479690+0000", 2024-08-17T18:08:46.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93689339000000005 2024-08-17T18:08:46.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:46.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479700+0000", 2024-08-17T18:08:46.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5610000000000004e-06 2024-08-17T18:08:46.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:46.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.769574+0000", 2024-08-17T18:08:46.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.289874223 2024-08-17T18:08:46.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:08:46.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.775343+0000", 2024-08-17T18:08:46.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057693670000000001 2024-08-17T18:08:46.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:46.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.824122+0000", 2024-08-17T18:08:46.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.048778909000000002 2024-08-17T18:08:46.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:46.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.826036+0000", 2024-08-17T18:08:46.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019139980000000001 2024-08-17T18:08:46.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:08:46.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.830279+0000", 2024-08-17T18:08:46.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042426240000000004 2024-08-17T18:08:46.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:46.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.847119+0000", 2024-08-17T18:08:46.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016840368000000001 2024-08-17T18:08:46.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:46.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.847762+0000", 2024-08-17T18:08:46.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064298700000000003 2024-08-17T18:08:46.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:46.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.848426+0000", 2024-08-17T18:08:46.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000663925 2024-08-17T18:08:46.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:46.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.849470+0000", 2024-08-17T18:08:46.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001043719 2024-08-17T18:08:46.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:46.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.854367+0000", 2024-08-17T18:08:46.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004897658 2024-08-17T18:08:46.964 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:46.964 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:46.964 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:46.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.964 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180174 54.10 54:08915b2c:::benchmark_data_smithi136_273969_object57289:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:08:46.964 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:21.930733+0000", 2024-08-17T18:08:46.964 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 445.01685297199998, 2024-08-17T18:08:46.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1011982060000003, 2024-08-17T18:08:46.964 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:46.964 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:46.965 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:46.965 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:08:46.965 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:08:46.965 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180174 2024-08-17T18:08:46.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.965 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:46.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:46.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.930733+0000", 2024-08-17T18:08:46.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:46.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:46.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.930733+0000", 2024-08-17T18:08:46.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:46.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:46.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.930731+0000", 2024-08-17T18:08:46.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:08:46.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:46.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.930738+0000", 2024-08-17T18:08:46.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3450000000000002e-06 2024-08-17T18:08:46.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:46.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.930750+0000", 2024-08-17T18:08:46.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2476e-05 2024-08-17T18:08:46.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:46.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.930757+0000", 2024-08-17T18:08:46.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8719999999999996e-06 2024-08-17T18:08:46.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:46.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.974198+0000", 2024-08-17T18:08:46.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.043441133 2024-08-17T18:08:46.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:46.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.977482+0000", 2024-08-17T18:08:46.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032840030000000002 2024-08-17T18:08:46.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:46.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.032292+0000", 2024-08-17T18:08:46.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.054810287999999999 2024-08-17T18:08:46.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:46.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.035548+0000", 2024-08-17T18:08:46.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032551569999999998 2024-08-17T18:08:46.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:46.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542945+0000", 2024-08-17T18:08:46.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.50739734999999997 2024-08-17T18:08:46.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:46.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542951+0000", 2024-08-17T18:08:46.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8069999999999996e-06 2024-08-17T18:08:46.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:46.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479903+0000", 2024-08-17T18:08:46.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93695215499999995 2024-08-17T18:08:46.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:46.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479912+0000", 2024-08-17T18:08:46.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7020000000000007e-06 2024-08-17T18:08:46.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:46.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.954749+0000", 2024-08-17T18:08:46.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4748370450000001 2024-08-17T18:08:46.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:08:46.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.960359+0000", 2024-08-17T18:08:46.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056101390000000001 2024-08-17T18:08:46.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:46.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.992310+0000", 2024-08-17T18:08:46.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.031950854000000001 2024-08-17T18:08:46.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:46.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.994150+0000", 2024-08-17T18:08:46.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001840752 2024-08-17T18:08:46.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:08:46.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.997815+0000", 2024-08-17T18:08:46.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036646690000000002 2024-08-17T18:08:46.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:46.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.010247+0000", 2024-08-17T18:08:46.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012432284 2024-08-17T18:08:46.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:46.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.025714+0000", 2024-08-17T18:08:46.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015466680999999999 2024-08-17T18:08:46.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:46.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.026527+0000", 2024-08-17T18:08:46.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00081259799999999999 2024-08-17T18:08:46.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:46.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.027532+0000", 2024-08-17T18:08:46.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001005159 2024-08-17T18:08:46.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:46.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.031931+0000", 2024-08-17T18:08:46.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043991380000000004 2024-08-17T18:08:46.975 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:46.975 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:46.975 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:46.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.975 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180181 54.10 54:0a355281:::benchmark_data_smithi136_273969_object57296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:08:46.975 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:21.940096+0000", 2024-08-17T18:08:46.975 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 445.00748931300001, 2024-08-17T18:08:46.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0013956070000001, 2024-08-17T18:08:46.976 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:46.976 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:46.976 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:46.976 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:08:46.976 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:08:46.976 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180181 2024-08-17T18:08:46.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.976 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:46.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:46.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.940096+0000", 2024-08-17T18:08:46.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:46.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:46.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.940096+0000", 2024-08-17T18:08:46.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:46.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:46.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.940094+0000", 2024-08-17T18:08:46.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:08:46.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:46.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.940102+0000", 2024-08-17T18:08:46.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3370000000000002e-06 2024-08-17T18:08:46.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:46.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.940118+0000", 2024-08-17T18:08:46.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5713999999999999e-05 2024-08-17T18:08:46.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:46.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.940126+0000", 2024-08-17T18:08:46.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7350000000000002e-06 2024-08-17T18:08:46.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:46.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.978817+0000", 2024-08-17T18:08:46.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038691019 2024-08-17T18:08:46.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:46.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:21.982796+0000", 2024-08-17T18:08:46.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039788150000000001 2024-08-17T18:08:46.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:46.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.014670+0000", 2024-08-17T18:08:46.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.031873637000000003 2024-08-17T18:08:46.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:46.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.018063+0000", 2024-08-17T18:08:46.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033937350000000002 2024-08-17T18:08:46.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:46.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542851+0000", 2024-08-17T18:08:46.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.52478792900000004 2024-08-17T18:08:46.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:46.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542857+0000", 2024-08-17T18:08:46.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6289999999999998e-06 2024-08-17T18:08:46.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:46.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479777+0000", 2024-08-17T18:08:46.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93691995400000005 2024-08-17T18:08:46.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:46.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479786+0000", 2024-08-17T18:08:46.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8529999999999998e-06 2024-08-17T18:08:46.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:46.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.860711+0000", 2024-08-17T18:08:46.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.38092492 2024-08-17T18:08:46.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:08:46.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.867150+0000", 2024-08-17T18:08:46.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064388700000000002 2024-08-17T18:08:46.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:46.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.900225+0000", 2024-08-17T18:08:46.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.03307529 2024-08-17T18:08:46.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:46.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.902023+0000", 2024-08-17T18:08:46.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001798231 2024-08-17T18:08:46.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:08:46.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.906038+0000", 2024-08-17T18:08:46.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040151620000000001 2024-08-17T18:08:46.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:46.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.919715+0000", 2024-08-17T18:08:46.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013676364999999999 2024-08-17T18:08:46.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:46.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.935483+0000", 2024-08-17T18:08:46.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015768372999999999 2024-08-17T18:08:46.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:46.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.936289+0000", 2024-08-17T18:08:46.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00080628999999999998 2024-08-17T18:08:46.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:46.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.937359+0000", 2024-08-17T18:08:46.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001069293 2024-08-17T18:08:46.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:46.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.941492+0000", 2024-08-17T18:08:46.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041333460000000004 2024-08-17T18:08:46.986 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:46.986 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:46.986 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:46.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.986 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180233 54.10 54:08986526:::benchmark_data_smithi136_273969_object57348:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:08:46.986 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:22.029556+0000", 2024-08-17T18:08:46.986 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 444.918029467, 2024-08-17T18:08:46.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9954107949999997, 2024-08-17T18:08:46.986 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:46.987 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:46.987 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:46.987 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:08:46.987 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:08:46.987 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180233 2024-08-17T18:08:46.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.987 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:46.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:46.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.029556+0000", 2024-08-17T18:08:46.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:46.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:46.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.029556+0000", 2024-08-17T18:08:46.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:46.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:46.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.029555+0000", 2024-08-17T18:08:46.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:08:46.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:46.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.029561+0000", 2024-08-17T18:08:46.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0349999999999998e-06 2024-08-17T18:08:46.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:46.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.029575+0000", 2024-08-17T18:08:46.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4566000000000001e-05 2024-08-17T18:08:46.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:46.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.029583+0000", 2024-08-17T18:08:46.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9330000000000001e-06 2024-08-17T18:08:46.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:46.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.040674+0000", 2024-08-17T18:08:46.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011090546999999999 2024-08-17T18:08:46.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:46.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.044717+0000", 2024-08-17T18:08:46.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040431340000000003 2024-08-17T18:08:46.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:46.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.075210+0000", 2024-08-17T18:08:46.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.030493089000000001 2024-08-17T18:08:46.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:46.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.078485+0000", 2024-08-17T18:08:46.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032751820000000002 2024-08-17T18:08:46.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:46.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542915+0000", 2024-08-17T18:08:46.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.46442982199999999 2024-08-17T18:08:46.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:46.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.542921+0000", 2024-08-17T18:08:46.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7710000000000003e-06 2024-08-17T18:08:46.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:46.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479864+0000", 2024-08-17T18:08:46.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93694376899999998 2024-08-17T18:08:46.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:46.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.479873+0000", 2024-08-17T18:08:46.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7870000000000007e-06 2024-08-17T18:08:46.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:46.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.948551+0000", 2024-08-17T18:08:46.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4686778700000001 2024-08-17T18:08:46.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:08:46.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.954417+0000", 2024-08-17T18:08:46.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058657630000000004 2024-08-17T18:08:46.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:46.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.974711+0000", 2024-08-17T18:08:46.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.02029392 2024-08-17T18:08:46.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:46.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.976548+0000", 2024-08-17T18:08:46.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018375240000000001 2024-08-17T18:08:46.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:08:46.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.980434+0000", 2024-08-17T18:08:46.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038862089999999998 2024-08-17T18:08:46.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:46.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.000907+0000", 2024-08-17T18:08:46.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020472305999999999 2024-08-17T18:08:46.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:46.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.001622+0000", 2024-08-17T18:08:46.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00071567799999999997 2024-08-17T18:08:46.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:46.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.019920+0000", 2024-08-17T18:08:46.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018297686000000001 2024-08-17T18:08:46.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:46.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.020902+0000", 2024-08-17T18:08:46.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00098207099999999994 2024-08-17T18:08:46.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:46.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.024967+0000", 2024-08-17T18:08:46.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040647970000000002 2024-08-17T18:08:46.997 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:46.997 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:46.997 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:46.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.997 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140115 55.16 55:686c39f0:::benchmark_data_smithi136_313476_object48129:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1500)", 2024-08-17T18:08:46.997 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.143129+0000", 2024-08-17T18:08:46.997 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.804456270000003, 2024-08-17T18:08:46.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.994428729, 2024-08-17T18:08:46.997 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:46.997 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:46.998 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:46.998 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:08:46.998 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:08:46.998 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140115 2024-08-17T18:08:46.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.998 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:46.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:46.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.143129+0000", 2024-08-17T18:08:46.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:46.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:46.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.143129+0000", 2024-08-17T18:08:46.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:46.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:46.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:46.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.143128+0000", 2024-08-17T18:08:46.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:08:46.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:46.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:47.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.143135+0000", 2024-08-17T18:08:47.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1010000000000003e-06 2024-08-17T18:08:47.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:47.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.143146+0000", 2024-08-17T18:08:47.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0953e-05 2024-08-17T18:08:47.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:47.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.143152+0000", 2024-08-17T18:08:47.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8119999999999996e-06 2024-08-17T18:08:47.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.168312+0000", 2024-08-17T18:08:47.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.025160491 2024-08-17T18:08:47.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:47.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.171272+0000", 2024-08-17T18:08:47.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002959555 2024-08-17T18:08:47.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.190464+0000", 2024-08-17T18:08:47.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019191881000000001 2024-08-17T18:08:47.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:47.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.192704+0000", 2024-08-17T18:08:47.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002239974 2024-08-17T18:08:47.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537367+0000", 2024-08-17T18:08:47.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.34466343999999999 2024-08-17T18:08:47.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:47.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537377+0000", 2024-08-17T18:08:47.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7100000000000002e-06 2024-08-17T18:08:47.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.030542+0000", 2024-08-17T18:08:47.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4931654029999999 2024-08-17T18:08:47.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:47.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.033326+0000", 2024-08-17T18:08:47.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027832360000000001 2024-08-17T18:08:47.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:08:47.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.037511+0000", 2024-08-17T18:08:47.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041850910000000002 2024-08-17T18:08:47.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.041644+0000", 2024-08-17T18:08:47.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041336869999999996 2024-08-17T18:08:47.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:47.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.051430+0000", 2024-08-17T18:08:47.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0097861519999999994 2024-08-17T18:08:47.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.131645+0000", 2024-08-17T18:08:47.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.080214368999999994 2024-08-17T18:08:47.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:47.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.132824+0000", 2024-08-17T18:08:47.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001179399 2024-08-17T18:08:47.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:47.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.137558+0000", 2024-08-17T18:08:47.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0047339340000000004 2024-08-17T18:08:47.006 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.006 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:47.006 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.007 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140118 55.6 55:63934a67:::benchmark_data_smithi136_313476_object48132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1500)", 2024-08-17T18:08:47.007 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.146936+0000", 2024-08-17T18:08:47.007 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.800649618999998, 2024-08-17T18:08:47.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8275733330000001, 2024-08-17T18:08:47.007 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:47.007 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:47.007 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:47.007 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:08:47.007 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:08:47.007 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140118 2024-08-17T18:08:47.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.007 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:47.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:47.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.146936+0000", 2024-08-17T18:08:47.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:47.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.146936+0000", 2024-08-17T18:08:47.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:47.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.146935+0000", 2024-08-17T18:08:47.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:08:47.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:47.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.146940+0000", 2024-08-17T18:08:47.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6169999999999998e-06 2024-08-17T18:08:47.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:47.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.146950+0000", 2024-08-17T18:08:47.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6909999999999992e-06 2024-08-17T18:08:47.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:47.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.146956+0000", 2024-08-17T18:08:47.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7699999999999998e-06 2024-08-17T18:08:47.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.171831+0000", 2024-08-17T18:08:47.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.024875853 2024-08-17T18:08:47.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:47.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.174174+0000", 2024-08-17T18:08:47.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00234293 2024-08-17T18:08:47.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.187094+0000", 2024-08-17T18:08:47.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012919717000000001 2024-08-17T18:08:47.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:47.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.189879+0000", 2024-08-17T18:08:47.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002785142 2024-08-17T18:08:47.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537107+0000", 2024-08-17T18:08:47.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.34722777700000002 2024-08-17T18:08:47.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:47.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537117+0000", 2024-08-17T18:08:47.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0414999999999999e-05 2024-08-17T18:08:47.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.946076+0000", 2024-08-17T18:08:47.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4089589140000003 2024-08-17T18:08:47.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:47.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.949008+0000", 2024-08-17T18:08:47.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029318690000000001 2024-08-17T18:08:47.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:08:47.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.953707+0000", 2024-08-17T18:08:47.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046990069999999998 2024-08-17T18:08:47.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.014 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:47.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.966900+0000", 2024-08-17T18:08:47.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013192812 2024-08-17T18:08:47.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.967853+0000", 2024-08-17T18:08:47.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00095252599999999998 2024-08-17T18:08:47.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:47.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.968105+0000", 2024-08-17T18:08:47.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025275999999999999 2024-08-17T18:08:47.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:47.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.969431+0000", 2024-08-17T18:08:47.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013257869999999999 2024-08-17T18:08:47.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:47.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.974509+0000", 2024-08-17T18:08:47.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050782379999999997 2024-08-17T18:08:47.017 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.017 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:47.017 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.018 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140121 55.6 55:62195bf4:::benchmark_data_smithi136_313476_object48135:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1500)", 2024-08-17T18:08:47.018 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.155517+0000", 2024-08-17T18:08:47.018 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.792068485000001, 2024-08-17T18:08:47.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9475979289999996, 2024-08-17T18:08:47.018 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:47.018 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:47.018 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:47.018 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:08:47.018 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:08:47.018 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140121 2024-08-17T18:08:47.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.018 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:47.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:47.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.155517+0000", 2024-08-17T18:08:47.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:47.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.155517+0000", 2024-08-17T18:08:47.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:47.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.155516+0000", 2024-08-17T18:08:47.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:08:47.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:47.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.155523+0000", 2024-08-17T18:08:47.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7630000000000003e-06 2024-08-17T18:08:47.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:47.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.155537+0000", 2024-08-17T18:08:47.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4782e-05 2024-08-17T18:08:47.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:47.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.155544+0000", 2024-08-17T18:08:47.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8000000000000001e-06 2024-08-17T18:08:47.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.175182+0000", 2024-08-17T18:08:47.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019637852000000001 2024-08-17T18:08:47.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:47.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.177528+0000", 2024-08-17T18:08:47.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002346374 2024-08-17T18:08:47.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.199421+0000", 2024-08-17T18:08:47.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.02189261 2024-08-17T18:08:47.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:47.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.201679+0000", 2024-08-17T18:08:47.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002258164 2024-08-17T18:08:47.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537146+0000", 2024-08-17T18:08:47.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.33546662100000002 2024-08-17T18:08:47.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:47.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537156+0000", 2024-08-17T18:08:47.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8830000000000001e-06 2024-08-17T18:08:47.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.974898+0000", 2024-08-17T18:08:47.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4377422559999999 2024-08-17T18:08:47.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:47.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.978346+0000", 2024-08-17T18:08:47.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003448441 2024-08-17T18:08:47.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:08:47.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.982964+0000", 2024-08-17T18:08:47.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046176430000000003 2024-08-17T18:08:47.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.997825+0000", 2024-08-17T18:08:47.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014860923 2024-08-17T18:08:47.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:47.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.998982+0000", 2024-08-17T18:08:47.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00115672 2024-08-17T18:08:47.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.096437+0000", 2024-08-17T18:08:47.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097455183000000001 2024-08-17T18:08:47.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:47.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.097744+0000", 2024-08-17T18:08:47.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001306794 2024-08-17T18:08:47.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:47.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.103115+0000", 2024-08-17T18:08:47.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00537157 2024-08-17T18:08:47.026 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.026 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:47.026 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.027 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140134 55.6 55:6053b4ec:::benchmark_data_smithi136_313476_object48148:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1500)", 2024-08-17T18:08:47.027 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.187965+0000", 2024-08-17T18:08:47.027 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.759620638999998, 2024-08-17T18:08:47.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9270218950000002, 2024-08-17T18:08:47.027 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:47.027 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:47.027 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:47.027 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:08:47.027 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:08:47.027 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140134 2024-08-17T18:08:47.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.028 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:47.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:47.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.187965+0000", 2024-08-17T18:08:47.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:47.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.187965+0000", 2024-08-17T18:08:47.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:47.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.187964+0000", 2024-08-17T18:08:47.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:08:47.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:47.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.187969+0000", 2024-08-17T18:08:47.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5470000000000004e-06 2024-08-17T18:08:47.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:47.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.187981+0000", 2024-08-17T18:08:47.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.182e-05 2024-08-17T18:08:47.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:47.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.187988+0000", 2024-08-17T18:08:47.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9569999999999996e-06 2024-08-17T18:08:47.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.202404+0000", 2024-08-17T18:08:47.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014416258 2024-08-17T18:08:47.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:47.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.204940+0000", 2024-08-17T18:08:47.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002536061 2024-08-17T18:08:47.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.251179+0000", 2024-08-17T18:08:47.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.046238824999999997 2024-08-17T18:08:47.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:47.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.253521+0000", 2024-08-17T18:08:47.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00234191 2024-08-17T18:08:47.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537183+0000", 2024-08-17T18:08:47.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.28366188799999997 2024-08-17T18:08:47.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:47.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537192+0000", 2024-08-17T18:08:47.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6920000000000008e-06 2024-08-17T18:08:47.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.986670+0000", 2024-08-17T18:08:47.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4494780689999995 2024-08-17T18:08:47.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:47.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.989571+0000", 2024-08-17T18:08:47.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029015310000000002 2024-08-17T18:08:47.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:08:47.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.993999+0000", 2024-08-17T18:08:47.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044280639999999998 2024-08-17T18:08:47.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.998741+0000", 2024-08-17T18:08:47.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0047416170000000001 2024-08-17T18:08:47.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:47.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.999237+0000", 2024-08-17T18:08:47.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00049553300000000004 2024-08-17T18:08:47.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.107406+0000", 2024-08-17T18:08:47.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10816967399999999 2024-08-17T18:08:47.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:47.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.108896+0000", 2024-08-17T18:08:47.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001489654 2024-08-17T18:08:47.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:47.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.114987+0000", 2024-08-17T18:08:47.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060910670000000004 2024-08-17T18:08:47.035 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.035 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:47.036 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.036 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140142 55.16 55:6fe98626:::benchmark_data_smithi136_313476_object48156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1500)", 2024-08-17T18:08:47.036 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.245454+0000", 2024-08-17T18:08:47.036 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.702132014999997, 2024-08-17T18:08:47.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6765945460000005, 2024-08-17T18:08:47.036 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:47.036 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:47.036 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:47.036 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:08:47.036 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:08:47.036 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140142 2024-08-17T18:08:47.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.037 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:47.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:47.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.245454+0000", 2024-08-17T18:08:47.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:47.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.245454+0000", 2024-08-17T18:08:47.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:47.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.245452+0000", 2024-08-17T18:08:47.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:08:47.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:47.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.245509+0000", 2024-08-17T18:08:47.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6351999999999999e-05 2024-08-17T18:08:47.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:47.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.245521+0000", 2024-08-17T18:08:47.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2826000000000001e-05 2024-08-17T18:08:47.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:47.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.245529+0000", 2024-08-17T18:08:47.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4780000000000002e-06 2024-08-17T18:08:47.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.251081+0000", 2024-08-17T18:08:47.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055522970000000003 2024-08-17T18:08:47.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:47.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.253259+0000", 2024-08-17T18:08:47.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002178003 2024-08-17T18:08:47.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.265864+0000", 2024-08-17T18:08:47.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012605269000000001 2024-08-17T18:08:47.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:47.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.267795+0000", 2024-08-17T18:08:47.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001930502 2024-08-17T18:08:47.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537220+0000", 2024-08-17T18:08:47.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.26942472699999997 2024-08-17T18:08:47.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:47.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537229+0000", 2024-08-17T18:08:47.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7729999999999994e-06 2024-08-17T18:08:47.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:50.959941+0000", 2024-08-17T18:08:47.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4227114749999998 2024-08-17T18:08:47.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:47.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:50.962824+0000", 2024-08-17T18:08:47.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028828310000000002 2024-08-17T18:08:47.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:08:47.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:50.967194+0000", 2024-08-17T18:08:47.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043701160000000003 2024-08-17T18:08:47.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:47.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:52.733611+0000", 2024-08-17T18:08:47.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.766416706 2024-08-17T18:08:47.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.914906+0000", 2024-08-17T18:08:47.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1812954549999999 2024-08-17T18:08:47.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.915610+0000", 2024-08-17T18:08:47.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00070412299999999997 2024-08-17T18:08:47.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:47.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.917012+0000", 2024-08-17T18:08:47.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001401577 2024-08-17T18:08:47.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:47.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.922048+0000", 2024-08-17T18:08:47.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050365399999999999 2024-08-17T18:08:47.045 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.045 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:47.045 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.046 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140144 55.16 55:6a89b7a5:::benchmark_data_smithi136_313476_object48158:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1500)", 2024-08-17T18:08:47.046 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.245905+0000", 2024-08-17T18:08:47.046 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.701680787999997, 2024-08-17T18:08:47.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8777299039999997, 2024-08-17T18:08:47.046 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:47.046 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:47.046 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:47.046 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:08:47.046 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:08:47.046 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140144 2024-08-17T18:08:47.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.047 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:47.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:47.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.245905+0000", 2024-08-17T18:08:47.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:47.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.245905+0000", 2024-08-17T18:08:47.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:47.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.245904+0000", 2024-08-17T18:08:47.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:08:47.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:47.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.245909+0000", 2024-08-17T18:08:47.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2739999999999998e-06 2024-08-17T18:08:47.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:47.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.245922+0000", 2024-08-17T18:08:47.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2374999999999999e-05 2024-08-17T18:08:47.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:47.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.245929+0000", 2024-08-17T18:08:47.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1849999999999998e-06 2024-08-17T18:08:47.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.253500+0000", 2024-08-17T18:08:47.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0075709100000000001 2024-08-17T18:08:47.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:47.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.255502+0000", 2024-08-17T18:08:47.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020023689999999999 2024-08-17T18:08:47.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.275777+0000", 2024-08-17T18:08:47.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020275027000000001 2024-08-17T18:08:47.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:47.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.277719+0000", 2024-08-17T18:08:47.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019421569999999999 2024-08-17T18:08:47.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537294+0000", 2024-08-17T18:08:47.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25957475800000002 2024-08-17T18:08:47.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:47.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537303+0000", 2024-08-17T18:08:47.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7760000000000003e-06 2024-08-17T18:08:47.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.008985+0000", 2024-08-17T18:08:47.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4716823100000003 2024-08-17T18:08:47.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:47.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.011852+0000", 2024-08-17T18:08:47.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028672609999999999 2024-08-17T18:08:47.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:08:47.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.016105+0000", 2024-08-17T18:08:47.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004252736 2024-08-17T18:08:47.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.020400+0000", 2024-08-17T18:08:47.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042951789999999997 2024-08-17T18:08:47.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:47.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.050959+0000", 2024-08-17T18:08:47.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.030559118 2024-08-17T18:08:47.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.116967+0000", 2024-08-17T18:08:47.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.066007328000000004 2024-08-17T18:08:47.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:47.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.118335+0000", 2024-08-17T18:08:47.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013681559999999999 2024-08-17T18:08:47.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:47.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.123635+0000", 2024-08-17T18:08:47.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005299893 2024-08-17T18:08:47.056 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.056 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:47.056 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.056 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140146 55.16 55:6ac64e84:::benchmark_data_smithi136_313476_object48160:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1500)", 2024-08-17T18:08:47.056 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.251449+0000", 2024-08-17T18:08:47.056 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.696136289000002, 2024-08-17T18:08:47.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8637902650000004, 2024-08-17T18:08:47.057 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:47.057 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:47.057 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:47.057 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:08:47.057 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:08:47.057 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140146 2024-08-17T18:08:47.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.057 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:47.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:47.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.251449+0000", 2024-08-17T18:08:47.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:47.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.251449+0000", 2024-08-17T18:08:47.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:47.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.251448+0000", 2024-08-17T18:08:47.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:08:47.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:47.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.251455+0000", 2024-08-17T18:08:47.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6250000000000001e-06 2024-08-17T18:08:47.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:47.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.251466+0000", 2024-08-17T18:08:47.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1518999999999999e-05 2024-08-17T18:08:47.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:47.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.251472+0000", 2024-08-17T18:08:47.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6200000000000004e-06 2024-08-17T18:08:47.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.255771+0000", 2024-08-17T18:08:47.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042986459999999997 2024-08-17T18:08:47.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:47.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.257827+0000", 2024-08-17T18:08:47.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020561529999999998 2024-08-17T18:08:47.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.286005+0000", 2024-08-17T18:08:47.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.028178279000000001 2024-08-17T18:08:47.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:47.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.287934+0000", 2024-08-17T18:08:47.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001928527 2024-08-17T18:08:47.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537257+0000", 2024-08-17T18:08:47.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.24932309699999999 2024-08-17T18:08:47.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:47.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537266+0000", 2024-08-17T18:08:47.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5389999999999996e-06 2024-08-17T18:08:47.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.998491+0000", 2024-08-17T18:08:47.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4612248159999996 2024-08-17T18:08:47.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:47.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.001285+0000", 2024-08-17T18:08:47.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027934000000000001 2024-08-17T18:08:47.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:08:47.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.005549+0000", 2024-08-17T18:08:47.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042640719999999998 2024-08-17T18:08:47.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.019664+0000", 2024-08-17T18:08:47.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014115259999999999 2024-08-17T18:08:47.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:47.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.050737+0000", 2024-08-17T18:08:47.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.031073554999999999 2024-08-17T18:08:47.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.108051+0000", 2024-08-17T18:08:47.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.057313192999999998 2024-08-17T18:08:47.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:47.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.109526+0000", 2024-08-17T18:08:47.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014752809999999999 2024-08-17T18:08:47.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:47.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.115240+0000", 2024-08-17T18:08:47.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057137150000000003 2024-08-17T18:08:47.065 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.065 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:47.065 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.065 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140177 55.16 55:69108c58:::benchmark_data_smithi136_313476_object48191:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1500)", 2024-08-17T18:08:47.065 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.409890+0000", 2024-08-17T18:08:47.066 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.537695255000003, 2024-08-17T18:08:47.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7210113929999995, 2024-08-17T18:08:47.066 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:47.066 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:47.066 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:47.066 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:08:47.066 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:08:47.066 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140177 2024-08-17T18:08:47.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.066 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:47.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:47.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.409890+0000", 2024-08-17T18:08:47.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:47.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.409890+0000", 2024-08-17T18:08:47.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:47.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.409889+0000", 2024-08-17T18:08:47.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:08:47.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:47.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.409898+0000", 2024-08-17T18:08:47.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9730000000000003e-06 2024-08-17T18:08:47.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:47.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.409914+0000", 2024-08-17T18:08:47.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.658e-05 2024-08-17T18:08:47.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:47.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.409924+0000", 2024-08-17T18:08:47.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2769999999999994e-06 2024-08-17T18:08:47.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.410911+0000", 2024-08-17T18:08:47.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00098741999999999992 2024-08-17T18:08:47.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:47.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.412914+0000", 2024-08-17T18:08:47.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002003459 2024-08-17T18:08:47.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.429694+0000", 2024-08-17T18:08:47.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016779755 2024-08-17T18:08:47.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:47.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.432264+0000", 2024-08-17T18:08:47.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002569876 2024-08-17T18:08:47.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537330+0000", 2024-08-17T18:08:47.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.105066137 2024-08-17T18:08:47.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:47.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537340+0000", 2024-08-17T18:08:47.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6500000000000008e-06 2024-08-17T18:08:47.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.020664+0000", 2024-08-17T18:08:47.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4833246239999998 2024-08-17T18:08:47.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:47.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.023441+0000", 2024-08-17T18:08:47.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002776868 2024-08-17T18:08:47.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:08:47.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.027597+0000", 2024-08-17T18:08:47.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041561000000000002 2024-08-17T18:08:47.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.040926+0000", 2024-08-17T18:08:47.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013328361 2024-08-17T18:08:47.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:47.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.051205+0000", 2024-08-17T18:08:47.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010279501999999999 2024-08-17T18:08:47.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.124473+0000", 2024-08-17T18:08:47.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.073267551 2024-08-17T18:08:47.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:47.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.125608+0000", 2024-08-17T18:08:47.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011352999999999999 2024-08-17T18:08:47.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:47.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.130902+0000", 2024-08-17T18:08:47.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052936540000000001 2024-08-17T18:08:47.074 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.074 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:47.074 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.074 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140201 55.6 55:667a10e9:::benchmark_data_smithi136_313476_object48215:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1500)", 2024-08-17T18:08:47.074 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.438859+0000", 2024-08-17T18:08:47.074 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.508727157999999, 2024-08-17T18:08:47.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5271923440000004, 2024-08-17T18:08:47.074 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:47.074 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:47.074 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:47.075 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:08:47.075 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:08:47.075 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140201 2024-08-17T18:08:47.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.075 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:47.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:47.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.438859+0000", 2024-08-17T18:08:47.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:47.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.438859+0000", 2024-08-17T18:08:47.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:47.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.438857+0000", 2024-08-17T18:08:47.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:08:47.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:47.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.438863+0000", 2024-08-17T18:08:47.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5699999999999998e-06 2024-08-17T18:08:47.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:47.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.438876+0000", 2024-08-17T18:08:47.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2816999999999999e-05 2024-08-17T18:08:47.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:47.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.438883+0000", 2024-08-17T18:08:47.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6590000000000001e-06 2024-08-17T18:08:47.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.454188+0000", 2024-08-17T18:08:47.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015305309 2024-08-17T18:08:47.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:47.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.456778+0000", 2024-08-17T18:08:47.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025896230000000001 2024-08-17T18:08:47.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.501828+0000", 2024-08-17T18:08:47.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.045050458000000002 2024-08-17T18:08:47.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:47.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.504450+0000", 2024-08-17T18:08:47.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002621974 2024-08-17T18:08:47.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537059+0000", 2024-08-17T18:08:47.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.032608613000000002 2024-08-17T18:08:47.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:47.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537076+0000", 2024-08-17T18:08:47.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7144000000000001e-05 2024-08-17T18:08:47.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.933389+0000", 2024-08-17T18:08:47.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3963127950000001 2024-08-17T18:08:47.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:47.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.936377+0000", 2024-08-17T18:08:47.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029886380000000001 2024-08-17T18:08:47.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:08:47.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.941168+0000", 2024-08-17T18:08:47.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0047900670000000003 2024-08-17T18:08:47.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.958227+0000", 2024-08-17T18:08:47.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017059748 2024-08-17T18:08:47.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.959204+0000", 2024-08-17T18:08:47.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0009770899999999999 2024-08-17T18:08:47.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:47.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.959445+0000", 2024-08-17T18:08:47.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000240974 2024-08-17T18:08:47.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:47.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.960683+0000", 2024-08-17T18:08:47.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012378020000000001 2024-08-17T18:08:47.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:47.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.966051+0000", 2024-08-17T18:08:47.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0053677109999999998 2024-08-17T18:08:47.084 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.084 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:47.084 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.084 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140219 55.6 55:638f9cb2:::benchmark_data_smithi136_313476_object48233:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1500)", 2024-08-17T18:08:47.085 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.475188+0000", 2024-08-17T18:08:47.085 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.472397419000004, 2024-08-17T18:08:47.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8959626580000002, 2024-08-17T18:08:47.085 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:47.085 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:47.085 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:47.085 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:08:47.085 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:08:47.085 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140219 2024-08-17T18:08:47.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.085 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:47.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:47.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.475188+0000", 2024-08-17T18:08:47.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:47.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.475188+0000", 2024-08-17T18:08:47.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:47.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.475187+0000", 2024-08-17T18:08:47.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:08:47.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:47.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.475194+0000", 2024-08-17T18:08:47.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9990000000000002e-06 2024-08-17T18:08:47.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:47.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.475208+0000", 2024-08-17T18:08:47.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4423000000000001e-05 2024-08-17T18:08:47.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:47.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.475216+0000", 2024-08-17T18:08:47.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8639999999999996e-06 2024-08-17T18:08:47.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.512299+0000", 2024-08-17T18:08:47.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037082708999999998 2024-08-17T18:08:47.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:47.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.514754+0000", 2024-08-17T18:08:47.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024551780000000001 2024-08-17T18:08:47.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.535464+0000", 2024-08-17T18:08:47.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020710304999999998 2024-08-17T18:08:47.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:47.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.535476+0000", 2024-08-17T18:08:47.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1453e-05 2024-08-17T18:08:47.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:52.788288+0000", 2024-08-17T18:08:47.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2528118360000002 2024-08-17T18:08:47.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:47.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:52.791022+0000", 2024-08-17T18:08:47.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027340939999999998 2024-08-17T18:08:47.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.930296+0000", 2024-08-17T18:08:47.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1392738250000001 2024-08-17T18:08:47.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:47.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.933061+0000", 2024-08-17T18:08:47.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027656360000000001 2024-08-17T18:08:47.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.180790+0000", 2024-08-17T18:08:47.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.24772897999999999 2024-08-17T18:08:47.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:47.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.183020+0000", 2024-08-17T18:08:47.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002229483 2024-08-17T18:08:47.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:08:47.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.187688+0000", 2024-08-17T18:08:47.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046687259999999998 2024-08-17T18:08:47.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.191766+0000", 2024-08-17T18:08:47.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040775059999999998 2024-08-17T18:08:47.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:47.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.215058+0000", 2024-08-17T18:08:47.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023292335000000001 2024-08-17T18:08:47.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.361167+0000", 2024-08-17T18:08:47.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.14610825799999999 2024-08-17T18:08:47.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:47.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.362649+0000", 2024-08-17T18:08:47.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014826959999999999 2024-08-17T18:08:47.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:47.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.371151+0000", 2024-08-17T18:08:47.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0085017180000000001 2024-08-17T18:08:47.094 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.094 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:47.094 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.094 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140245 55.16 55:6a908b3e:::benchmark_data_smithi136_313476_object48259:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1501)", 2024-08-17T18:08:47.094 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.537627+0000", 2024-08-17T18:08:47.094 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.409958578999998, 2024-08-17T18:08:47.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8502157009999998, 2024-08-17T18:08:47.094 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:47.095 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:47.095 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:47.095 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:08:47.095 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:08:47.095 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140245 2024-08-17T18:08:47.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.095 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:47.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:47.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537627+0000", 2024-08-17T18:08:47.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:47.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537627+0000", 2024-08-17T18:08:47.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:47.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537626+0000", 2024-08-17T18:08:47.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:08:47.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:47.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537631+0000", 2024-08-17T18:08:47.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3120000000000001e-06 2024-08-17T18:08:47.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:47.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537643+0000", 2024-08-17T18:08:47.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2278e-05 2024-08-17T18:08:47.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:47.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537650+0000", 2024-08-17T18:08:47.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8199999999999999e-06 2024-08-17T18:08:47.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537714+0000", 2024-08-17T18:08:47.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3718000000000002e-05 2024-08-17T18:08:47.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:47.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537723+0000", 2024-08-17T18:08:47.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3039999999999994e-06 2024-08-17T18:08:47.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.042001+0000", 2024-08-17T18:08:47.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5042780990000004 2024-08-17T18:08:47.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:47.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.044575+0000", 2024-08-17T18:08:47.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025734820000000002 2024-08-17T18:08:47.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.091971+0000", 2024-08-17T18:08:47.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.047396471000000003 2024-08-17T18:08:47.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:47.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.094508+0000", 2024-08-17T18:08:47.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025367599999999999 2024-08-17T18:08:47.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.232886+0000", 2024-08-17T18:08:47.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.138377737 2024-08-17T18:08:47.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:47.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.234976+0000", 2024-08-17T18:08:47.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020901000000000001 2024-08-17T18:08:47.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:08:47.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.239301+0000", 2024-08-17T18:08:47.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043252610000000004 2024-08-17T18:08:47.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.259472+0000", 2024-08-17T18:08:47.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020170738000000001 2024-08-17T18:08:47.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:47.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.260547+0000", 2024-08-17T18:08:47.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00107544 2024-08-17T18:08:47.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.380530+0000", 2024-08-17T18:08:47.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.119982598 2024-08-17T18:08:47.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:47.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.382034+0000", 2024-08-17T18:08:47.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015040279999999999 2024-08-17T18:08:47.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:47.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.387843+0000", 2024-08-17T18:08:47.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058090759999999998 2024-08-17T18:08:47.103 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.103 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:47.103 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.103 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140251 55.16 55:6fd0bdc9:::benchmark_data_smithi136_313476_object48265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1501)", 2024-08-17T18:08:47.103 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.556377+0000", 2024-08-17T18:08:47.103 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.391209005999997, 2024-08-17T18:08:47.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8137197040000004, 2024-08-17T18:08:47.103 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:47.103 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:47.103 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:47.103 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:08:47.104 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:08:47.104 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140251 2024-08-17T18:08:47.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.104 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:47.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:47.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.556377+0000", 2024-08-17T18:08:47.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:47.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.556377+0000", 2024-08-17T18:08:47.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:47.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.556375+0000", 2024-08-17T18:08:47.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:08:47.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:47.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.556381+0000", 2024-08-17T18:08:47.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4539999999999998e-06 2024-08-17T18:08:47.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:47.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.556396+0000", 2024-08-17T18:08:47.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5579000000000001e-05 2024-08-17T18:08:47.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:47.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.556403+0000", 2024-08-17T18:08:47.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2089999999999999e-06 2024-08-17T18:08:47.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.560709+0000", 2024-08-17T18:08:47.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043056860000000004 2024-08-17T18:08:47.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:47.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.560719+0000", 2024-08-17T18:08:47.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4360000000000008e-06 2024-08-17T18:08:47.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.044919+0000", 2024-08-17T18:08:47.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.484200446 2024-08-17T18:08:47.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:47.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.047552+0000", 2024-08-17T18:08:47.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026333939999999998 2024-08-17T18:08:47.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.064624+0000", 2024-08-17T18:08:47.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017071651 2024-08-17T18:08:47.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:47.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.067296+0000", 2024-08-17T18:08:47.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002672439 2024-08-17T18:08:47.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.198782+0000", 2024-08-17T18:08:47.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13148515799999999 2024-08-17T18:08:47.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:47.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.200937+0000", 2024-08-17T18:08:47.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021558940000000002 2024-08-17T18:08:47.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:08:47.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.205362+0000", 2024-08-17T18:08:47.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044241949999999997 2024-08-17T18:08:47.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.225794+0000", 2024-08-17T18:08:47.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020432344000000002 2024-08-17T18:08:47.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:47.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.226024+0000", 2024-08-17T18:08:47.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022981600000000001 2024-08-17T18:08:47.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.361923+0000", 2024-08-17T18:08:47.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13589936499999999 2024-08-17T18:08:47.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:47.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.363412+0000", 2024-08-17T18:08:47.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014891329999999999 2024-08-17T18:08:47.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:47.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.370096+0000", 2024-08-17T18:08:47.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006684026 2024-08-17T18:08:47.111 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.111 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:47.111 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.112 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140266 55.16 55:6f3984ca:::benchmark_data_smithi136_313476_object48280:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1501)", 2024-08-17T18:08:47.112 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.575512+0000", 2024-08-17T18:08:47.112 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.372074001999998, 2024-08-17T18:08:47.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.804134779, 2024-08-17T18:08:47.112 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:47.112 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:47.112 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:47.112 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:08:47.112 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:08:47.112 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140266 2024-08-17T18:08:47.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.112 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:47.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:47.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.575512+0000", 2024-08-17T18:08:47.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:47.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.575512+0000", 2024-08-17T18:08:47.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:47.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.575511+0000", 2024-08-17T18:08:47.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:08:47.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:47.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.575516+0000", 2024-08-17T18:08:47.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2510000000000002e-06 2024-08-17T18:08:47.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:47.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.575527+0000", 2024-08-17T18:08:47.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0993e-05 2024-08-17T18:08:47.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:47.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.575534+0000", 2024-08-17T18:08:47.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3560000000000004e-06 2024-08-17T18:08:47.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.575580+0000", 2024-08-17T18:08:47.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6164e-05 2024-08-17T18:08:47.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:47.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.575588+0000", 2024-08-17T18:08:47.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4939999999999997e-06 2024-08-17T18:08:47.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.047938+0000", 2024-08-17T18:08:47.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4723502579999996 2024-08-17T18:08:47.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:47.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.050500+0000", 2024-08-17T18:08:47.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025613760000000002 2024-08-17T18:08:47.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.077424+0000", 2024-08-17T18:08:47.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.026924766999999999 2024-08-17T18:08:47.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:47.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.080382+0000", 2024-08-17T18:08:47.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002957627 2024-08-17T18:08:47.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.214958+0000", 2024-08-17T18:08:47.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13457627 2024-08-17T18:08:47.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:47.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.217434+0000", 2024-08-17T18:08:47.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024758839999999998 2024-08-17T18:08:47.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:08:47.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.221763+0000", 2024-08-17T18:08:47.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043288859999999997 2024-08-17T18:08:47.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.226843+0000", 2024-08-17T18:08:47.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050802130000000001 2024-08-17T18:08:47.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:47.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.242789+0000", 2024-08-17T18:08:47.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015945773999999999 2024-08-17T18:08:47.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.372330+0000", 2024-08-17T18:08:47.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12954122500000001 2024-08-17T18:08:47.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:47.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.373797+0000", 2024-08-17T18:08:47.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001466619 2024-08-17T18:08:47.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:47.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.379646+0000", 2024-08-17T18:08:47.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058496149999999998 2024-08-17T18:08:47.120 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.120 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:47.120 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.120 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140287 55.16 55:6c1e48be:::benchmark_data_smithi136_313476_object48301:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1501)", 2024-08-17T18:08:47.120 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.673616+0000", 2024-08-17T18:08:47.120 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 59.273969588, 2024-08-17T18:08:47.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7225061789999998, 2024-08-17T18:08:47.121 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:47.121 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:47.121 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:47.121 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:08:47.121 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:08:47.121 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140287 2024-08-17T18:08:47.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.121 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:47.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:47.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.673616+0000", 2024-08-17T18:08:47.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:47.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.673616+0000", 2024-08-17T18:08:47.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:47.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.673615+0000", 2024-08-17T18:08:47.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:08:47.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:47.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.673620+0000", 2024-08-17T18:08:47.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4380000000000003e-06 2024-08-17T18:08:47.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:47.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.673633+0000", 2024-08-17T18:08:47.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2404e-05 2024-08-17T18:08:47.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:47.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.673640+0000", 2024-08-17T18:08:47.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1180000000000003e-06 2024-08-17T18:08:47.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.673683+0000", 2024-08-17T18:08:47.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3061e-05 2024-08-17T18:08:47.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:08:47.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.673690+0000", 2024-08-17T18:08:47.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.328e-06 2024-08-17T18:08:47.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.051769+0000", 2024-08-17T18:08:47.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3780791890000001 2024-08-17T18:08:47.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:47.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.054411+0000", 2024-08-17T18:08:47.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026415950000000001 2024-08-17T18:08:47.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.137821+0000", 2024-08-17T18:08:47.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.083410350999999994 2024-08-17T18:08:47.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:47.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.140473+0000", 2024-08-17T18:08:47.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002651863 2024-08-17T18:08:47.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.248857+0000", 2024-08-17T18:08:47.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.108384208 2024-08-17T18:08:47.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:47.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.250989+0000", 2024-08-17T18:08:47.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002131431 2024-08-17T18:08:47.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:08:47.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.255463+0000", 2024-08-17T18:08:47.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044744290000000003 2024-08-17T18:08:47.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.260314+0000", 2024-08-17T18:08:47.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048502850000000002 2024-08-17T18:08:47.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:47.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.277247+0000", 2024-08-17T18:08:47.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016933320000000002 2024-08-17T18:08:47.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.388740+0000", 2024-08-17T18:08:47.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11149316300000001 2024-08-17T18:08:47.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:47.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.390215+0000", 2024-08-17T18:08:47.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001475098 2024-08-17T18:08:47.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:47.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.396122+0000", 2024-08-17T18:08:47.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059071169999999999 2024-08-17T18:08:47.129 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.129 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:47.129 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.130 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140290 55.16 55:6b7eed24:::benchmark_data_smithi136_313476_object48304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1502)", 2024-08-17T18:08:47.130 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:48.586146+0000", 2024-08-17T18:08:47.130 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 58.361439373000003, 2024-08-17T18:08:47.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8181273789999999, 2024-08-17T18:08:47.130 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:47.130 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:47.130 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:47.130 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:08:47.130 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:08:47.130 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140290 2024-08-17T18:08:47.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.130 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:47.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:47.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:48.586146+0000", 2024-08-17T18:08:47.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:47.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:48.586146+0000", 2024-08-17T18:08:47.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:47.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:48.586145+0000", 2024-08-17T18:08:47.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:08:47.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:47.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:48.586152+0000", 2024-08-17T18:08:47.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0010000000000004e-06 2024-08-17T18:08:47.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:47.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:48.586167+0000", 2024-08-17T18:08:47.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5792999999999999e-05 2024-08-17T18:08:47.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:47.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:48.586176+0000", 2024-08-17T18:08:47.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2870000000000004e-06 2024-08-17T18:08:47.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.140715+0000", 2024-08-17T18:08:47.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5545393939999999 2024-08-17T18:08:47.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:47.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.143350+0000", 2024-08-17T18:08:47.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026350840000000002 2024-08-17T18:08:47.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.159079+0000", 2024-08-17T18:08:47.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015728829 2024-08-17T18:08:47.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:47.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.161595+0000", 2024-08-17T18:08:47.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002516346 2024-08-17T18:08:47.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.266505+0000", 2024-08-17T18:08:47.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.104909926 2024-08-17T18:08:47.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:47.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.268641+0000", 2024-08-17T18:08:47.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021357619999999998 2024-08-17T18:08:47.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:08:47.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.273043+0000", 2024-08-17T18:08:47.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044022840000000002 2024-08-17T18:08:47.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.276859+0000", 2024-08-17T18:08:47.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038157690000000001 2024-08-17T18:08:47.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:47.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.278368+0000", 2024-08-17T18:08:47.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001508793 2024-08-17T18:08:47.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.396986+0000", 2024-08-17T18:08:47.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.118618009 2024-08-17T18:08:47.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:47.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.398430+0000", 2024-08-17T18:08:47.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014436290000000001 2024-08-17T18:08:47.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:47.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.404274+0000", 2024-08-17T18:08:47.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058441459999999997 2024-08-17T18:08:47.137 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.137 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:47.138 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.138 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140307 55.16 55:6c589f5c:::benchmark_data_smithi136_313476_object48321:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1502)", 2024-08-17T18:08:47.138 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:48.690392+0000", 2024-08-17T18:08:47.138 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 58.257193942000001, 2024-08-17T18:08:47.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7685365280000003, 2024-08-17T18:08:47.138 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:08:47.138 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:08:47.138 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:08:47.138 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:08:47.138 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:08:47.138 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140307 2024-08-17T18:08:47.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.139 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:08:47.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:08:47.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:48.690392+0000", 2024-08-17T18:08:47.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:08:47.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:48.690392+0000", 2024-08-17T18:08:47.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:08:47.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:08:47.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:48.690390+0000", 2024-08-17T18:08:47.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:08:47.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:08:47.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:48.690397+0000", 2024-08-17T18:08:47.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5030000000000002e-06 2024-08-17T18:08:47.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:08:47.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:48.690410+0000", 2024-08-17T18:08:47.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3171e-05 2024-08-17T18:08:47.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:08:47.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:48.690416+0000", 2024-08-17T18:08:47.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5429999999999999e-06 2024-08-17T18:08:47.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.143646+0000", 2024-08-17T18:08:47.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.453229951 2024-08-17T18:08:47.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:08:47.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.146316+0000", 2024-08-17T18:08:47.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002669879 2024-08-17T18:08:47.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.339560+0000", 2024-08-17T18:08:47.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.193243955 2024-08-17T18:08:47.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:08:47.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.342864+0000", 2024-08-17T18:08:47.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033041469999999999 2024-08-17T18:08:47.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:08:47.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.413238+0000", 2024-08-17T18:08:47.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070374183000000007 2024-08-17T18:08:47.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:08:47.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.415827+0000", 2024-08-17T18:08:47.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025888909999999998 2024-08-17T18:08:47.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:08:47.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.423214+0000", 2024-08-17T18:08:47.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073863410000000003 2024-08-17T18:08:47.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.434611+0000", 2024-08-17T18:08:47.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011397025 2024-08-17T18:08:47.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:08:47.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.435576+0000", 2024-08-17T18:08:47.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00096511299999999997 2024-08-17T18:08:47.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:08:47.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.451855+0000", 2024-08-17T18:08:47.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016278678000000001 2024-08-17T18:08:47.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:08:47.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.453176+0000", 2024-08-17T18:08:47.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001320994 2024-08-17T18:08:47.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:08:47.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:08:47.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:08:47.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.458928+0000", 2024-08-17T18:08:47.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057527510000000004 2024-08-17T18:08:47.146 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.146 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:47.146 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.146 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:08:47.146 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:08:47.146 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:08:47.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:47.148 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:47.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:47.249 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:47.282 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:47.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:47.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:47.430 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:47.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:47.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:47.532 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:47.562 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:47.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:47.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:47.664 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:47.695 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:47.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:47.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:47.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:08:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:08:47] "GET /metrics HTTP/1.1" 200 36611 "" "Prometheus/2.51.0" 2024-08-17T18:08:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:47 smithi136 ceph-mon[115078]: pgmap v2758: 109 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 582 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 616 op/s; 5035/28008 objects degraded (17.977%); 1568/28008 objects misplaced (5.598%); 3.4 MiB/s, 241 objects/s recovering 2024-08-17T18:08:47.699 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:47 smithi136 ceph-mon[109861]: pgmap v2758: 109 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 582 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 616 op/s; 5035/28008 objects degraded (17.977%); 1568/28008 objects misplaced (5.598%); 3.4 MiB/s, 241 objects/s recovering 2024-08-17T18:08:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:47 smithi165 ceph-mon[105517]: pgmap v2758: 109 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 582 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 616 op/s; 5035/28008 objects degraded (17.977%); 1568/28008 objects misplaced (5.598%); 3.4 MiB/s, 241 objects/s recovering 2024-08-17T18:08:47.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:47.826 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:47.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:47.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:47.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:47.954 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:47.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:47.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:48.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:48.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-17T18:08:48.094 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:48.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:48.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:48.195 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:48.248 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:48.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:48.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:48.350 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:48.362 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:08:48.383 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:48.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:48.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:48.485 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:48.535 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:48.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:48.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:48.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:08:48.637 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:48.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:48.707 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:48.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:48.808 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:48.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:48.859 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:48.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:48.960 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:48.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:48.995 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:48.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:49.096 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:49.132 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:49.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:49.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:49.234 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:49.271 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:49.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:49.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:49.373 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:49.381 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.0 to 1 (10000) 2024-08-17T18:08:49.425 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:49.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:49.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:49.527 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:49.568 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:49.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:49.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:49.571 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:49 smithi165 ceph-mon[105517]: pgmap v2759: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 409 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 791 op/s; 5035/19707 objects degraded (25.549%); 1010/19707 objects misplaced (5.125%); 3.0 MiB/s, 222 objects/s recovering 2024-08-17T18:08:49.572 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:49 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3897447559' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T18:08:49.572 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:49 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T18:08:49.637 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:49 smithi136 ceph-mon[109861]: pgmap v2759: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 409 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 791 op/s; 5035/19707 objects degraded (25.549%); 1010/19707 objects misplaced (5.125%); 3.0 MiB/s, 222 objects/s recovering 2024-08-17T18:08:49.637 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:49 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3897447559' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T18:08:49.637 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:49 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T18:08:49.637 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:49 smithi136 ceph-mon[115078]: pgmap v2759: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 409 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 791 op/s; 5035/19707 objects degraded (25.549%); 1010/19707 objects misplaced (5.125%); 3.0 MiB/s, 222 objects/s recovering 2024-08-17T18:08:49.637 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:49 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3897447559' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T18:08:49.638 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:49 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T18:08:49.670 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:49.716 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:49.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:49.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:49.818 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:49.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:49.860 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:49.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:49.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:49.971 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:08:49.972 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:08:49.972 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:08:49.972 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:08:49.972 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:08:49.972 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:08:49.972 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:08:49.972 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:08:49.972 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:08:49.972 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:08:49.973 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:08:49.973 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:08:49.973 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:08:49.973 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:08:49.973 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:08:49.973 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:08:49.993 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:49.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:49.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:50.095 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:50.135 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:50.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:50.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:50.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:50.260 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:08:50.260 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:08:50.275 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:50.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:50.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:50.377 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:50.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:50.420 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:50.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:50.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:50 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T18:08:50.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:50 smithi165 ceph-mon[105517]: osdmap e1520: 8 total, 8 up, 7 in 2024-08-17T18:08:50.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:50 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/508921757' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:08:50.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:50 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5035/19707 objects degraded (25.549%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:08:50.521 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:50.566 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:50.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:50.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:50.667 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:50.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:50 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T18:08:50.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:50 smithi136 ceph-mon[109861]: osdmap e1520: 8 total, 8 up, 7 in 2024-08-17T18:08:50.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:50 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/508921757' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:08:50.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:50 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5035/19707 objects degraded (25.549%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:08:50.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:50 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T18:08:50.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:50 smithi136 ceph-mon[115078]: osdmap e1520: 8 total, 8 up, 7 in 2024-08-17T18:08:50.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:50 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/508921757' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:08:50.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:50 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5035/19707 objects degraded (25.549%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:08:50.703 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:50.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:50.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:50.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:50.836 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:50.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:50.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:50.937 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:50.977 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:50.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:50.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:50.998 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-17T18:08:51.079 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:51.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:51.117 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:51.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:51.218 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:51.258 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:51.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:51.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:51.326 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:08:51.360 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:51.391 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 5 1 2024-08-17T18:08:51.404 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:51.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:51.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:51.506 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:51.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:51.560 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:51.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:51.660 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:51.679 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:51 smithi136 ceph-mon[109861]: pgmap v2761: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 364 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 720 op/s; 5035/17538 objects degraded (28.709%); 1010/17538 objects misplaced (5.759%); 2.1 MiB/s, 164 objects/s recovering 2024-08-17T18:08:51.679 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:08:51.679 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:51 smithi136 ceph-mon[115078]: pgmap v2761: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 364 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 720 op/s; 5035/17538 objects degraded (28.709%); 1010/17538 objects misplaced (5.759%); 2.1 MiB/s, 164 objects/s recovering 2024-08-17T18:08:51.679 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:08:51.688 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:51.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:51.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:51 smithi165 ceph-mon[105517]: pgmap v2761: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 364 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 720 op/s; 5035/17538 objects degraded (28.709%); 1010/17538 objects misplaced (5.759%); 2.1 MiB/s, 164 objects/s recovering 2024-08-17T18:08:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:08:51.790 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:51.853 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:08:51.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:51.870 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:51.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:51.921 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:08:51.972 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:52.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:52.016 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:52.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:52.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:52.155 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:52.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:52.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:52.257 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:52.288 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:52.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:52.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:52.389 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:52.420 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:52.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:52.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:52.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:52.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:52.559 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:52.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:52.660 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:52.697 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:52.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:52.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:52.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:52.829 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:52.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:52.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:52.930 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:52.963 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:52.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:52.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:53.065 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:53.104 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:53.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:53.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:53.205 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:53.227 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:53.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:53.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:53.328 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:53.372 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:53.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:53.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:53.473 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:53.498 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:53.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:53.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:53.600 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:53.610 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Removed 91983 objects 2024-08-17T18:08:53.610 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Clean up completed and total clean up time :207.414 2024-08-17T18:08:53.627 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:53.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:53.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:53 smithi136 ceph-mon[109861]: pgmap v2762: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 159 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 900 op/s; 5035/7665 objects degraded (65.688%); 340/7665 objects misplaced (4.436%); 1.7 MiB/s, 167 objects/s recovering 2024-08-17T18:08:53.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:53 smithi136 ceph-mon[115078]: pgmap v2762: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 159 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 900 op/s; 5035/7665 objects degraded (65.688%); 340/7665 objects misplaced (4.436%); 1.7 MiB/s, 167 objects/s recovering 2024-08-17T18:08:53.710 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:08:53.729 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:53.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:53 smithi165 ceph-mon[105517]: pgmap v2762: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 159 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 900 op/s; 5035/7665 objects degraded (65.688%); 340/7665 objects misplaced (4.436%); 1.7 MiB/s, 167 objects/s recovering 2024-08-17T18:08:53.779 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:53.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:53.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:53.881 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:53.903 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:53.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:53.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:54.005 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:54.027 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:54.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:54.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:54.063 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-08-17T18:08:54.063 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-08-17T18:08:54.129 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:54.151 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:54.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:54.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:54.253 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:54.277 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:54.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:54.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:54.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:54.400 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:54.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:54.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:54.502 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:54.524 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:54.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:54.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:54.626 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:54.648 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:54.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:54.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:54.750 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:54.773 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:54.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:54.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:54.875 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:54.897 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:54.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:54.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:54.999 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:55.022 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:55.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:55.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:55.123 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:55.147 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:55.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:55.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:55.248 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:55.271 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:55.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:55.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:55.372 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:55.394 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:55.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:55.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:55.480 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:55 smithi136 ceph-mon[109861]: pgmap v2763: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 97 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 902 op/s; 5035/4698 objects degraded (107.173%); 340/4698 objects misplaced (7.237%); 1.7 MiB/s, 164 objects/s recovering 2024-08-17T18:08:55.481 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:55 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5035/4698 objects degraded (107.173%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:08:55.481 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:55 smithi136 ceph-mon[115078]: pgmap v2763: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 97 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 902 op/s; 5035/4698 objects degraded (107.173%); 340/4698 objects misplaced (7.237%); 1.7 MiB/s, 164 objects/s recovering 2024-08-17T18:08:55.481 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:55 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5035/4698 objects degraded (107.173%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:08:55.495 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:55.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:55 smithi165 ceph-mon[105517]: pgmap v2763: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 97 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 902 op/s; 5035/4698 objects degraded (107.173%); 340/4698 objects misplaced (7.237%); 1.7 MiB/s, 164 objects/s recovering 2024-08-17T18:08:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:55 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5035/4698 objects degraded (107.173%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T18:08:55.517 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:55.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:55.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:55.619 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:55.640 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:55.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:55.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:55.669 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:08:55.718 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:08:55.742 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:55.767 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:55.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:55.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:55.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:55.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:55.893 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:55.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:55.994 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:56.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:56.018 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:56.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:56.119 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:56.143 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:56.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:56.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:56.245 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:56.266 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:56.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:56.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:56.368 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:56.389 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:56.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:56.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:56.428 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:08:56.431 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.5 to 1 (10000) 2024-08-17T18:08:56.491 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:56.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:56 smithi165 ceph-mon[105517]: osdmap e1521: 8 total, 8 up, 7 in 2024-08-17T18:08:56.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:56 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2282660674' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T18:08:56.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:56 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T18:08:56.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:56 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2309842821' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:08:56.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:56.513 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:56.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:56.614 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:56.635 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:56.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:56.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:56.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:56 smithi136 ceph-mon[109861]: osdmap e1521: 8 total, 8 up, 7 in 2024-08-17T18:08:56.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:56 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2282660674' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T18:08:56.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:56 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T18:08:56.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:56 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2309842821' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:08:56.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:56 smithi136 ceph-mon[115078]: osdmap e1521: 8 total, 8 up, 7 in 2024-08-17T18:08:56.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:56 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2282660674' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T18:08:56.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:56 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T18:08:56.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:56 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2309842821' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:08:56.737 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:56.759 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:56.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:56.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:56.861 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:56.882 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:56.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:56.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:56.984 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:57.006 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:57.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:57.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:57.108 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:08:57.129 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:57.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:57.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:08:57.231 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:57.252 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:57.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:57.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:57.354 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:57.376 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:57.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:57.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:57.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:57.499 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:57.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:57.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:57.601 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:57.622 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:08:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:08:57] "GET /metrics HTTP/1.1" 200 36628 "" "Prometheus/2.51.0" 2024-08-17T18:08:57.623 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:57 smithi136 ceph-mon[109861]: pgmap v2765: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 64 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 714 op/s; 5035/3120 objects degraded (161.378%); 269/3120 objects misplaced (8.622%); 0 B/s, 235 objects/s recovering 2024-08-17T18:08:57.623 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:57 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:08:57.623 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:57 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T18:08:57.623 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:57 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2309842821' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:08:57.623 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:57 smithi136 ceph-mon[109861]: osdmap e1522: 8 total, 8 up, 7 in 2024-08-17T18:08:57.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:57 smithi136 ceph-mon[115078]: pgmap v2765: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 64 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 714 op/s; 5035/3120 objects degraded (161.378%); 269/3120 objects misplaced (8.622%); 0 B/s, 235 objects/s recovering 2024-08-17T18:08:57.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:57 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:08:57.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:57 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T18:08:57.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:57 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2309842821' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:08:57.623 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:57 smithi136 ceph-mon[115078]: osdmap e1522: 8 total, 8 up, 7 in 2024-08-17T18:08:57.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:57.624 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:57.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:57.725 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:57.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:57.748 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:57.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:57.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:57 smithi165 ceph-mon[105517]: pgmap v2765: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 64 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 714 op/s; 5035/3120 objects degraded (161.378%); 269/3120 objects misplaced (8.622%); 0 B/s, 235 objects/s recovering 2024-08-17T18:08:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:57 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:08:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:57 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T18:08:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:57 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2309842821' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:08:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:57 smithi165 ceph-mon[105517]: osdmap e1522: 8 total, 8 up, 7 in 2024-08-17T18:08:57.849 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:57.870 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:57.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:57.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:57.972 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:08:57.993 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:57.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:57.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:08:58.095 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:58.117 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:58.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:58.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:58.219 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:58.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:58.241 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:58.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:58.342 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:58.364 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:58.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:58.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:58.466 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:58.487 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:58.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:58.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:58.589 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:58.611 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:58.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:58.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:58.713 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:58.734 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:58.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:58.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:58.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:58 smithi165 ceph-mon[105517]: 55.1 scrub starts 2024-08-17T18:08:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:58 smithi165 ceph-mon[105517]: 55.1 scrub ok 2024-08-17T18:08:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:58 smithi165 ceph-mon[105517]: osdmap e1523: 8 total, 8 up, 7 in 2024-08-17T18:08:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:58 smithi165 ceph-mon[105517]: 55.16 deep-scrub starts 2024-08-17T18:08:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:58 smithi165 ceph-mon[105517]: 55.16 deep-scrub ok 2024-08-17T18:08:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:58 smithi165 ceph-mon[105517]: osdmap e1524: 8 total, 8 up, 7 in 2024-08-17T18:08:58.787 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:58 smithi136 ceph-mon[109861]: 55.1 scrub starts 2024-08-17T18:08:58.787 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:58 smithi136 ceph-mon[109861]: 55.1 scrub ok 2024-08-17T18:08:58.788 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:58 smithi136 ceph-mon[109861]: osdmap e1523: 8 total, 8 up, 7 in 2024-08-17T18:08:58.788 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:58 smithi136 ceph-mon[109861]: 55.16 deep-scrub starts 2024-08-17T18:08:58.788 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:58 smithi136 ceph-mon[109861]: 55.16 deep-scrub ok 2024-08-17T18:08:58.788 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:58 smithi136 ceph-mon[109861]: osdmap e1524: 8 total, 8 up, 7 in 2024-08-17T18:08:58.788 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:58 smithi136 ceph-mon[115078]: 55.1 scrub starts 2024-08-17T18:08:58.788 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:58 smithi136 ceph-mon[115078]: 55.1 scrub ok 2024-08-17T18:08:58.788 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:58 smithi136 ceph-mon[115078]: osdmap e1523: 8 total, 8 up, 7 in 2024-08-17T18:08:58.788 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:58 smithi136 ceph-mon[115078]: 55.16 deep-scrub starts 2024-08-17T18:08:58.788 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:58 smithi136 ceph-mon[115078]: 55.16 deep-scrub ok 2024-08-17T18:08:58.788 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:58 smithi136 ceph-mon[115078]: osdmap e1524: 8 total, 8 up, 7 in 2024-08-17T18:08:58.837 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:58.858 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:58.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:58.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:58.960 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:08:58.983 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:58.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:58.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:08:59.084 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:59.105 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:59.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:59.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:59.207 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:08:59.228 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:59.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:59.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:08:59.330 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:08:59.351 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:59.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:59.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:08:59.453 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:59.477 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:59.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:59.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:59.579 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:08:59.600 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:59.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:59.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:08:59.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:59 smithi136 ceph-mon[109861]: pgmap v2768: 109 pgs: 1 active+clean+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 16 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 5035/804 objects degraded (626.244%); 54/804 objects misplaced (6.716%) 2024-08-17T18:08:59.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:59 smithi136 ceph-mon[109861]: 55.3 scrub starts 2024-08-17T18:08:59.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:59 smithi136 ceph-mon[109861]: 55.3 scrub ok 2024-08-17T18:08:59.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:59 smithi136 ceph-mon[109861]: 55.19 scrub starts 2024-08-17T18:08:59.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:59 smithi136 ceph-mon[109861]: 55.19 scrub ok 2024-08-17T18:08:59.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:59 smithi136 ceph-mon[109861]: 55.6 scrub starts 2024-08-17T18:08:59.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:08:59 smithi136 ceph-mon[109861]: osdmap e1525: 8 total, 8 up, 7 in 2024-08-17T18:08:59.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:59 smithi136 ceph-mon[115078]: pgmap v2768: 109 pgs: 1 active+clean+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 16 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 5035/804 objects degraded (626.244%); 54/804 objects misplaced (6.716%) 2024-08-17T18:08:59.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:59 smithi136 ceph-mon[115078]: 55.3 scrub starts 2024-08-17T18:08:59.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:59 smithi136 ceph-mon[115078]: 55.3 scrub ok 2024-08-17T18:08:59.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:59 smithi136 ceph-mon[115078]: 55.19 scrub starts 2024-08-17T18:08:59.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:59 smithi136 ceph-mon[115078]: 55.19 scrub ok 2024-08-17T18:08:59.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:59 smithi136 ceph-mon[115078]: 55.6 scrub starts 2024-08-17T18:08:59.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:08:59 smithi136 ceph-mon[115078]: osdmap e1525: 8 total, 8 up, 7 in 2024-08-17T18:08:59.702 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:59.705 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:59 smithi165 ceph-mon[105517]: pgmap v2768: 109 pgs: 1 active+clean+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 16 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 5035/804 objects degraded (626.244%); 54/804 objects misplaced (6.716%) 2024-08-17T18:08:59.705 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:59 smithi165 ceph-mon[105517]: 55.3 scrub starts 2024-08-17T18:08:59.705 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:59 smithi165 ceph-mon[105517]: 55.3 scrub ok 2024-08-17T18:08:59.705 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:59 smithi165 ceph-mon[105517]: 55.19 scrub starts 2024-08-17T18:08:59.705 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:59 smithi165 ceph-mon[105517]: 55.19 scrub ok 2024-08-17T18:08:59.706 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:59 smithi165 ceph-mon[105517]: 55.6 scrub starts 2024-08-17T18:08:59.706 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:08:59 smithi165 ceph-mon[105517]: osdmap e1525: 8 total, 8 up, 7 in 2024-08-17T18:08:59.724 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:59.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:59.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:59.826 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:08:59.848 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:08:59.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:59.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:08:59.950 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:08:59.973 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:08:59.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:08:59.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:08:59.980 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:08:59.980 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:08:59.986 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:08:59.986 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:09:00.000 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:09:00.000 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:09:00.028 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:09:00.029 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:09:00.045 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:09:00.045 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:09:00.061 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:09:00.061 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:09:00.075 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:00.078 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:09:00.078 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:09:00.096 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:00.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:00.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:00.097 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:09:00.098 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:09:00.114 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:09:00.114 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:09:00.198 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:00.221 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:00.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:00.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:00.323 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:00.345 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:00.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:00.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:00.447 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:00.470 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:00.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:00.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:00.571 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:00.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:00.594 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:00.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:00.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:00 smithi165 ceph-mon[105517]: 55.6 scrub ok 2024-08-17T18:09:00.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:00 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5035/804 objects degraded (626.244%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T18:09:00.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:00 smithi165 ceph-mon[105517]: 55.11 scrub starts 2024-08-17T18:09:00.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:00 smithi165 ceph-mon[105517]: 55.11 scrub ok 2024-08-17T18:09:00.611 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:00 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3761445162' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:09:00.634 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-17T18:09:00.695 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:00.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:00 smithi136 ceph-mon[115078]: 55.6 scrub ok 2024-08-17T18:09:00.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:00 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5035/804 objects degraded (626.244%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T18:09:00.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:00 smithi136 ceph-mon[115078]: 55.11 scrub starts 2024-08-17T18:09:00.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:00 smithi136 ceph-mon[115078]: 55.11 scrub ok 2024-08-17T18:09:00.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:00 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3761445162' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:09:00.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:00 smithi136 ceph-mon[109861]: 55.6 scrub ok 2024-08-17T18:09:00.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:00 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5035/804 objects degraded (626.244%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T18:09:00.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:00 smithi136 ceph-mon[109861]: 55.11 scrub starts 2024-08-17T18:09:00.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:00 smithi136 ceph-mon[109861]: 55.11 scrub ok 2024-08-17T18:09:00.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:00 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3761445162' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:09:00.718 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:00.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:00.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:00.812 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:09:00.820 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:00.863 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:09:00.877 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:00.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:00.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:00.979 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:01.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:01.001 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:01.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:01.102 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:01.123 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:01.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:01.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:01.225 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:01.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:01.246 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:01.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:01.347 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:01.370 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:01.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:01.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:01.472 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:01.493 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:01.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:01.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:01.594 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:01.616 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:01.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:01.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:01.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:01.739 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:01.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:01.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:01 smithi165 ceph-mon[105517]: pgmap v2771: 109 pgs: 1 active+clean+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 16 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 5035/804 objects degraded (626.244%); 54/804 objects misplaced (6.716%) 2024-08-17T18:09:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:01 smithi165 ceph-mon[105517]: 55.9 scrub starts 2024-08-17T18:09:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:01 smithi165 ceph-mon[105517]: 55.9 scrub ok 2024-08-17T18:09:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:01 smithi165 ceph-mon[105517]: 55.8 scrub starts 2024-08-17T18:09:01.841 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:01.863 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:01.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:01.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:01.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:01 smithi136 ceph-mon[115078]: pgmap v2771: 109 pgs: 1 active+clean+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 16 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 5035/804 objects degraded (626.244%); 54/804 objects misplaced (6.716%) 2024-08-17T18:09:01.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:01 smithi136 ceph-mon[115078]: 55.9 scrub starts 2024-08-17T18:09:01.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:01 smithi136 ceph-mon[115078]: 55.9 scrub ok 2024-08-17T18:09:01.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:01 smithi136 ceph-mon[115078]: 55.8 scrub starts 2024-08-17T18:09:01.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:01 smithi136 ceph-mon[109861]: pgmap v2771: 109 pgs: 1 active+clean+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 16 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 5035/804 objects degraded (626.244%); 54/804 objects misplaced (6.716%) 2024-08-17T18:09:01.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:01 smithi136 ceph-mon[109861]: 55.9 scrub starts 2024-08-17T18:09:01.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:01 smithi136 ceph-mon[109861]: 55.9 scrub ok 2024-08-17T18:09:01.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:01 smithi136 ceph-mon[109861]: 55.8 scrub starts 2024-08-17T18:09:01.965 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:01.986 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:01.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:01.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:02.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:02.109 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:02.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:02.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:02.211 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:02.233 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:02.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:02.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:02.334 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:02.356 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:02.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:02.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:02.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:02.479 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:02.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:02.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:02.581 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:02.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:02.605 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:02.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:02.706 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:02.726 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:02.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:02.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:02.817 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:09:02.817 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T18:09:02.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T18:09:02.817 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:09:02.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.818 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:104813 54.11 54.64e93111 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T18:09:02.818 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.630215+0000", 2024-08-17T18:09:02.818 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 557.18489807000003, 2024-08-17T18:09:02.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.83036598800000005, 2024-08-17T18:09:02.818 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:02.818 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:09:02.818 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:02.818 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:09:02.818 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:09:02.818 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104813 2024-08-17T18:09:02.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.819 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:02.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:02.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.630215+0000", 2024-08-17T18:09:02.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:02.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.630215+0000", 2024-08-17T18:09:02.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:02.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.630213+0000", 2024-08-17T18:09:02.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:09:02.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:02.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.630324+0000", 2024-08-17T18:09:02.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011047 2024-08-17T18:09:02.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:02.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.630338+0000", 2024-08-17T18:09:02.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.452e-05 2024-08-17T18:09:02.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:02.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.630344+0000", 2024-08-17T18:09:02.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3029999999999999e-06 2024-08-17T18:09:02.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.630399+0000", 2024-08-17T18:09:02.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5204999999999997e-05 2024-08-17T18:09:02.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.630409+0000", 2024-08-17T18:09:02.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0363999999999999e-05 2024-08-17T18:09:02.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460567+0000", 2024-08-17T18:09:02.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.83015739499999996 2024-08-17T18:09:02.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:02.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460581+0000", 2024-08-17T18:09:02.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4001e-05 2024-08-17T18:09:02.823 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.823 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:02.823 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.823 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:104818 54.11 54.1190c0f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T18:09:02.823 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.637872+0000", 2024-08-17T18:09:02.823 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 557.177240869, 2024-08-17T18:09:02.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.82274040599999998, 2024-08-17T18:09:02.823 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:02.823 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:09:02.823 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:02.823 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:09:02.824 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:09:02.824 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104818 2024-08-17T18:09:02.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.824 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:02.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:02.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.637872+0000", 2024-08-17T18:09:02.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:02.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.637872+0000", 2024-08-17T18:09:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.637870+0000", 2024-08-17T18:09:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:09:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.637930+0000", 2024-08-17T18:09:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9210999999999999e-05 2024-08-17T18:09:02.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:02.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.637943+0000", 2024-08-17T18:09:02.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3332e-05 2024-08-17T18:09:02.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:02.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.637948+0000", 2024-08-17T18:09:02.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4659999999999998e-06 2024-08-17T18:09:02.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.637992+0000", 2024-08-17T18:09:02.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3899000000000001e-05 2024-08-17T18:09:02.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.638000+0000", 2024-08-17T18:09:02.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9039999999999992e-06 2024-08-17T18:09:02.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460604+0000", 2024-08-17T18:09:02.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.82260419100000004 2024-08-17T18:09:02.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:02.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460612+0000", 2024-08-17T18:09:02.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7729999999999999e-06 2024-08-17T18:09:02.828 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.828 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:02.828 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.828 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:104840 54.11 54.d5652851 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T18:09:02.828 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.667423+0000", 2024-08-17T18:09:02.828 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 557.14769014299998, 2024-08-17T18:09:02.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.79322487200000003, 2024-08-17T18:09:02.828 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:02.829 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:09:02.829 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:02.829 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:09:02.829 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:09:02.829 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104840 2024-08-17T18:09:02.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.829 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:02.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:02.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.667423+0000", 2024-08-17T18:09:02.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:02.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.667423+0000", 2024-08-17T18:09:02.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:02.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.667422+0000", 2024-08-17T18:09:02.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:09:02.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:02.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.667485+0000", 2024-08-17T18:09:02.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2741000000000001e-05 2024-08-17T18:09:02.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:02.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.667491+0000", 2024-08-17T18:09:02.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8699999999999997e-06 2024-08-17T18:09:02.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:02.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.667494+0000", 2024-08-17T18:09:02.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5930000000000002e-06 2024-08-17T18:09:02.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.667535+0000", 2024-08-17T18:09:02.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1273999999999998e-05 2024-08-17T18:09:02.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.667543+0000", 2024-08-17T18:09:02.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1659999999999997e-06 2024-08-17T18:09:02.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460636+0000", 2024-08-17T18:09:02.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.79309297400000001 2024-08-17T18:09:02.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:02.833 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:02.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460647+0000", 2024-08-17T18:09:02.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1863e-05 2024-08-17T18:09:02.834 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.834 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:02.834 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.835 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:104844 54.11 54.e06561d1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T18:09:02.835 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.672468+0000", 2024-08-17T18:09:02.835 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 557.142644776, 2024-08-17T18:09:02.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.78822546199999999, 2024-08-17T18:09:02.835 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:02.835 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:09:02.835 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:02.835 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:09:02.835 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:09:02.835 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104844 2024-08-17T18:09:02.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.835 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:02.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:02.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.672468+0000", 2024-08-17T18:09:02.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:02.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.672468+0000", 2024-08-17T18:09:02.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:02.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.672467+0000", 2024-08-17T18:09:02.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:09:02.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:02.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.672534+0000", 2024-08-17T18:09:02.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6926000000000003e-05 2024-08-17T18:09:02.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:02.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.672544+0000", 2024-08-17T18:09:02.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.022e-05 2024-08-17T18:09:02.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:02.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.672549+0000", 2024-08-17T18:09:02.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.809e-06 2024-08-17T18:09:02.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.672586+0000", 2024-08-17T18:09:02.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6585999999999998e-05 2024-08-17T18:09:02.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.672593+0000", 2024-08-17T18:09:02.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8269999999999998e-06 2024-08-17T18:09:02.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460683+0000", 2024-08-17T18:09:02.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.78808987600000002 2024-08-17T18:09:02.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:02.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460693+0000", 2024-08-17T18:09:02.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0138e-05 2024-08-17T18:09:02.839 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.839 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:02.839 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.840 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:104848 54.11 54.d7e6d951 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T18:09:02.840 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.673087+0000", 2024-08-17T18:09:02.840 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 557.14202612999998, 2024-08-17T18:09:02.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.78763660400000002, 2024-08-17T18:09:02.840 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:02.840 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:09:02.840 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:02.840 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:09:02.840 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:09:02.840 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104848 2024-08-17T18:09:02.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.840 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:02.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:02.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.673087+0000", 2024-08-17T18:09:02.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:02.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.673087+0000", 2024-08-17T18:09:02.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:02.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.673086+0000", 2024-08-17T18:09:02.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:09:02.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:02.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.673524+0000", 2024-08-17T18:09:02.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000438591 2024-08-17T18:09:02.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:02.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.673539+0000", 2024-08-17T18:09:02.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4791e-05 2024-08-17T18:09:02.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:02.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.673546+0000", 2024-08-17T18:09:02.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2200000000000003e-06 2024-08-17T18:09:02.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.673598+0000", 2024-08-17T18:09:02.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1418000000000001e-05 2024-08-17T18:09:02.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.673605+0000", 2024-08-17T18:09:02.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.866e-06 2024-08-17T18:09:02.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460712+0000", 2024-08-17T18:09:02.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.78710790600000002 2024-08-17T18:09:02.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:02.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460723+0000", 2024-08-17T18:09:02.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0771000000000001e-05 2024-08-17T18:09:02.844 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.844 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:02.844 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.844 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:104917 54.11 54.eda9ea11 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T18:09:02.844 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.725191+0000", 2024-08-17T18:09:02.844 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 557.08992122699999, 2024-08-17T18:09:02.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.73558475400000001, 2024-08-17T18:09:02.845 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:02.845 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:09:02.845 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:02.845 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:09:02.845 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:09:02.845 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104917 2024-08-17T18:09:02.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.845 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:02.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:02.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.725191+0000", 2024-08-17T18:09:02.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:02.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.725191+0000", 2024-08-17T18:09:02.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:02.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.725191+0000", 2024-08-17T18:09:02.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:09:02.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:02.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.725274+0000", 2024-08-17T18:09:02.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3373000000000003e-05 2024-08-17T18:09:02.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:02.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.725287+0000", 2024-08-17T18:09:02.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3025e-05 2024-08-17T18:09:02.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:02.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.725294+0000", 2024-08-17T18:09:02.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6270000000000002e-06 2024-08-17T18:09:02.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.725350+0000", 2024-08-17T18:09:02.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5897999999999999e-05 2024-08-17T18:09:02.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.725358+0000", 2024-08-17T18:09:02.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8220000000000007e-06 2024-08-17T18:09:02.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460768+0000", 2024-08-17T18:09:02.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.73540999399999996 2024-08-17T18:09:02.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:02.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460776+0000", 2024-08-17T18:09:02.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6920000000000008e-06 2024-08-17T18:09:02.849 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.849 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:02.849 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.849 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:104940 54.11 54.f8fdfd11 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T18:09:02.849 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.741796+0000", 2024-08-17T18:09:02.849 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 557.07331714300005, 2024-08-17T18:09:02.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71900983799999996, 2024-08-17T18:09:02.849 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:02.849 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:09:02.849 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:02.849 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:09:02.849 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:09:02.850 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104940 2024-08-17T18:09:02.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.850 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:02.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:02.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.741796+0000", 2024-08-17T18:09:02.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:02.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.741796+0000", 2024-08-17T18:09:02.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:02.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.741794+0000", 2024-08-17T18:09:02.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:09:02.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:02.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.741827+0000", 2024-08-17T18:09:02.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2960000000000003e-05 2024-08-17T18:09:02.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:02.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.741844+0000", 2024-08-17T18:09:02.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7703e-05 2024-08-17T18:09:02.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:02.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.741852+0000", 2024-08-17T18:09:02.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6040000000000004e-06 2024-08-17T18:09:02.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.741897+0000", 2024-08-17T18:09:02.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4941999999999998e-05 2024-08-17T18:09:02.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.741905+0000", 2024-08-17T18:09:02.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1289999999999996e-06 2024-08-17T18:09:02.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460796+0000", 2024-08-17T18:09:02.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.71889108999999995 2024-08-17T18:09:02.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:02.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460805+0000", 2024-08-17T18:09:02.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1630000000000002e-06 2024-08-17T18:09:02.853 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.853 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:02.853 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.854 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:104958 54.11 54.e8b0e6d1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-08-17T18:09:02.854 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T17:59:45.760286+0000", 2024-08-17T18:09:02.854 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 557.05482653900003, 2024-08-17T18:09:02.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.70054623599999999, 2024-08-17T18:09:02.854 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:02.854 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:09:02.854 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:02.854 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:09:02.854 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:09:02.854 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 104958 2024-08-17T18:09:02.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.854 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:02.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:02.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.760286+0000", 2024-08-17T18:09:02.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:02.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.760286+0000", 2024-08-17T18:09:02.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:02.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.760285+0000", 2024-08-17T18:09:02.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:09:02.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:02.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.760326+0000", 2024-08-17T18:09:02.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0945000000000002e-05 2024-08-17T18:09:02.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:02.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.760341+0000", 2024-08-17T18:09:02.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5248999999999999e-05 2024-08-17T18:09:02.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:02.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.760347+0000", 2024-08-17T18:09:02.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6919999999999999e-06 2024-08-17T18:09:02.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.760399+0000", 2024-08-17T18:09:02.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2293999999999998e-05 2024-08-17T18:09:02.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:45.760407+0000", 2024-08-17T18:09:02.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4150000000000002e-06 2024-08-17T18:09:02.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460824+0000", 2024-08-17T18:09:02.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.70041780799999998 2024-08-17T18:09:02.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:02.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T17:59:46.460832+0000", 2024-08-17T18:09:02.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0450000000000001e-06 2024-08-17T18:09:02.858 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.858 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:02.858 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.858 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180945 54.11 54:8e76512c:::benchmark_data_smithi136_273969_object58060:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1402)", 2024-08-17T18:09:02.858 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:26.787333+0000", 2024-08-17T18:09:02.858 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 456.02777930399998, 2024-08-17T18:09:02.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.74425201600000002, 2024-08-17T18:09:02.859 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:02.859 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:09:02.859 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:02.859 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:09:02.859 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:09:02.859 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180945 2024-08-17T18:09:02.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.859 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:02.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:02.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.787333+0000", 2024-08-17T18:09:02.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:02.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.787333+0000", 2024-08-17T18:09:02.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:02.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.787333+0000", 2024-08-17T18:09:02.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:09:02.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:02.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.787337+0000", 2024-08-17T18:09:02.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7210000000000002e-06 2024-08-17T18:09:02.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:02.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.787348+0000", 2024-08-17T18:09:02.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0429e-05 2024-08-17T18:09:02.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:02.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.787352+0000", 2024-08-17T18:09:02.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7659999999999998e-06 2024-08-17T18:09:02.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.787393+0000", 2024-08-17T18:09:02.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0114000000000003e-05 2024-08-17T18:09:02.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:26.787400+0000", 2024-08-17T18:09:02.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1940000000000001e-06 2024-08-17T18:09:02.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.525317+0000", 2024-08-17T18:09:02.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.73791703900000005 2024-08-17T18:09:02.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:09:02.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.525447+0000", 2024-08-17T18:09:02.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013059599999999999 2024-08-17T18:09:02.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3", 2024-08-17T18:09:02.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.525925+0000", 2024-08-17T18:09:02.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00047725399999999998 2024-08-17T18:09:02.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:02.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.531415+0000", 2024-08-17T18:09:02.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054903060000000004 2024-08-17T18:09:02.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:02.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.531482+0000", 2024-08-17T18:09:02.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7330999999999996e-05 2024-08-17T18:09:02.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:09:02.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.531494+0000", 2024-08-17T18:09:02.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.171e-05 2024-08-17T18:09:02.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:09:02.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.531542+0000", 2024-08-17T18:09:02.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8192999999999998e-05 2024-08-17T18:09:02.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:02.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:27.531585+0000", 2024-08-17T18:09:02.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3237e-05 2024-08-17T18:09:02.865 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.865 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:02.865 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.865 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:39669 55.5 55:a211053c:::benchmark_data_smithi136_313476_object39668:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1456)", 2024-08-17T18:09:02.866 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:04:14.321096+0000", 2024-08-17T18:09:02.866 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 288.49401635300001, 2024-08-17T18:09:02.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6412118919999998, 2024-08-17T18:09:02.866 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:02.866 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:09:02.866 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:02.866 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:09:02.866 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:09:02.866 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39669 2024-08-17T18:09:02.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.866 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:02.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:02.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.321096+0000", 2024-08-17T18:09:02.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:02.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.321096+0000", 2024-08-17T18:09:02.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:02.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.321096+0000", 2024-08-17T18:09:02.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:09:02.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:02.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.321152+0000", 2024-08-17T18:09:02.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6632000000000003e-05 2024-08-17T18:09:02.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:02.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.321163+0000", 2024-08-17T18:09:02.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0351e-05 2024-08-17T18:09:02.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:02.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.321168+0000", 2024-08-17T18:09:02.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6570000000000002e-06 2024-08-17T18:09:02.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.321206+0000", 2024-08-17T18:09:02.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7873999999999997e-05 2024-08-17T18:09:02.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:09:02.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.321326+0000", 2024-08-17T18:09:02.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119917 2024-08-17T18:09:02.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-08-17T18:09:02.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.321456+0000", 2024-08-17T18:09:02.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000129775 2024-08-17T18:09:02.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:02.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.325896+0000", 2024-08-17T18:09:02.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044400120000000001 2024-08-17T18:09:02.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:09:02.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.327927+0000", 2024-08-17T18:09:02.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00203119 2024-08-17T18:09:02.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.338413+0000", 2024-08-17T18:09:02.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010486182 2024-08-17T18:09:02.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.338419+0000", 2024-08-17T18:09:02.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2029999999999997e-06 2024-08-17T18:09:02.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521368+0000", 2024-08-17T18:09:02.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18294843 2024-08-17T18:09:02.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521381+0000", 2024-08-17T18:09:02.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.305e-05 2024-08-17T18:09:02.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.962240+0000", 2024-08-17T18:09:02.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.440859237 2024-08-17T18:09:02.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:02.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.962308+0000", 2024-08-17T18:09:02.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8058000000000005e-05 2024-08-17T18:09:02.872 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.872 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:02.872 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.873 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:39677 55.5 55:a4f0b823:::benchmark_data_smithi136_313476_object39676:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1456)", 2024-08-17T18:09:02.873 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:04:14.326500+0000", 2024-08-17T18:09:02.873 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 288.488612832, 2024-08-17T18:09:02.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.636704041, 2024-08-17T18:09:02.873 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:02.873 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:09:02.873 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:02.873 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:09:02.873 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:09:02.873 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39677 2024-08-17T18:09:02.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.873 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:02.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:02.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.326500+0000", 2024-08-17T18:09:02.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:02.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.326500+0000", 2024-08-17T18:09:02.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:02.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.326499+0000", 2024-08-17T18:09:02.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:09:02.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:02.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.326530+0000", 2024-08-17T18:09:02.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1267999999999999e-05 2024-08-17T18:09:02.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:02.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.326541+0000", 2024-08-17T18:09:02.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1061e-05 2024-08-17T18:09:02.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:02.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.326546+0000", 2024-08-17T18:09:02.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0170000000000004e-06 2024-08-17T18:09:02.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.326573+0000", 2024-08-17T18:09:02.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6808000000000002e-05 2024-08-17T18:09:02.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:09:02.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.326671+0000", 2024-08-17T18:09:02.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7682000000000004e-05 2024-08-17T18:09:02.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-08-17T18:09:02.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.326760+0000", 2024-08-17T18:09:02.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9067999999999995e-05 2024-08-17T18:09:02.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:09:02.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.328093+0000", 2024-08-17T18:09:02.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013328789999999999 2024-08-17T18:09:02.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:02.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.329318+0000", 2024-08-17T18:09:02.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012251180000000001 2024-08-17T18:09:02.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.338516+0000", 2024-08-17T18:09:02.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0091985169999999998 2024-08-17T18:09:02.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.338523+0000", 2024-08-17T18:09:02.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4049999999999997e-06 2024-08-17T18:09:02.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521415+0000", 2024-08-17T18:09:02.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18289263 2024-08-17T18:09:02.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521425+0000", 2024-08-17T18:09:02.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2180000000000005e-06 2024-08-17T18:09:02.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.963136+0000", 2024-08-17T18:09:02.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4417117310000001 2024-08-17T18:09:02.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:02.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.963204+0000", 2024-08-17T18:09:02.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7597999999999999e-05 2024-08-17T18:09:02.880 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.880 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:02.880 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.880 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:39689 55.5 55:a653d258:::benchmark_data_smithi136_313476_object39688:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1457)", 2024-08-17T18:09:02.880 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:04:14.360299+0000", 2024-08-17T18:09:02.880 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 288.45481397700001, 2024-08-17T18:09:02.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6189520910000001, 2024-08-17T18:09:02.880 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:02.880 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:09:02.881 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:02.881 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:09:02.881 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:09:02.881 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39689 2024-08-17T18:09:02.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.881 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:02.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:02.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.360299+0000", 2024-08-17T18:09:02.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:02.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.360299+0000", 2024-08-17T18:09:02.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:02.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.360298+0000", 2024-08-17T18:09:02.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:09:02.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:02.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.360356+0000", 2024-08-17T18:09:02.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8545000000000003e-05 2024-08-17T18:09:02.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:02.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.360367+0000", 2024-08-17T18:09:02.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0529e-05 2024-08-17T18:09:02.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:02.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.360372+0000", 2024-08-17T18:09:02.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4589999999999997e-06 2024-08-17T18:09:02.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.360408+0000", 2024-08-17T18:09:02.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5410000000000001e-05 2024-08-17T18:09:02.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.360415+0000", 2024-08-17T18:09:02.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9419999999999998e-06 2024-08-17T18:09:02.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521452+0000", 2024-08-17T18:09:02.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16103684600000001 2024-08-17T18:09:02.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521459+0000", 2024-08-17T18:09:02.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7530000000000001e-06 2024-08-17T18:09:02.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.963231+0000", 2024-08-17T18:09:02.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.441771315 2024-08-17T18:09:02.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:09:02.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.963334+0000", 2024-08-17T18:09:02.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103631 2024-08-17T18:09:02.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-08-17T18:09:02.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.963451+0000", 2024-08-17T18:09:02.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116422 2024-08-17T18:09:02.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:09:02.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.967463+0000", 2024-08-17T18:09:02.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040121000000000002 2024-08-17T18:09:02.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:02.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.972526+0000", 2024-08-17T18:09:02.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050626539999999998 2024-08-17T18:09:02.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:02.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.977600+0000", 2024-08-17T18:09:02.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050747709999999996 2024-08-17T18:09:02.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:09:02.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.977662+0000", 2024-08-17T18:09:02.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1323999999999998e-05 2024-08-17T18:09:02.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:02.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.979251+0000", 2024-08-17T18:09:02.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001589175 2024-08-17T18:09:02.887 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.887 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:02.887 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.887 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:39692 55.f 55:f8af78c3:::benchmark_data_smithi136_313476_object39691:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1457)", 2024-08-17T18:09:02.887 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:04:14.362183+0000", 2024-08-17T18:09:02.887 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 288.45292948399998, 2024-08-17T18:09:02.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.610310522, 2024-08-17T18:09:02.888 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:02.888 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:09:02.888 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:02.888 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:09:02.888 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:09:02.888 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39692 2024-08-17T18:09:02.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.888 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:02.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:02.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.362183+0000", 2024-08-17T18:09:02.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:02.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.362183+0000", 2024-08-17T18:09:02.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:02.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.362182+0000", 2024-08-17T18:09:02.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:09:02.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:02.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:02.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.362240+0000", 2024-08-17T18:09:02.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7550000000000003e-05 2024-08-17T18:09:02.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:02.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.362249+0000", 2024-08-17T18:09:02.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8689999999999993e-06 2024-08-17T18:09:02.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:02.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.362253+0000", 2024-08-17T18:09:02.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2150000000000003e-06 2024-08-17T18:09:02.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.362293+0000", 2024-08-17T18:09:02.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0093999999999999e-05 2024-08-17T18:09:02.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.362301+0000", 2024-08-17T18:09:02.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2649999999999996e-06 2024-08-17T18:09:02.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521423+0000", 2024-08-17T18:09:02.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.159121755 2024-08-17T18:09:02.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521435+0000", 2024-08-17T18:09:02.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1963e-05 2024-08-17T18:09:02.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.945301+0000", 2024-08-17T18:09:02.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.423865945 2024-08-17T18:09:02.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:09:02.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.945438+0000", 2024-08-17T18:09:02.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000136804 2024-08-17T18:09:02.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-08-17T18:09:02.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.945560+0000", 2024-08-17T18:09:02.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122545 2024-08-17T18:09:02.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:09:02.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.947846+0000", 2024-08-17T18:09:02.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022854910000000002 2024-08-17T18:09:02.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:02.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.949142+0000", 2024-08-17T18:09:02.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012963650000000001 2024-08-17T18:09:02.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:02.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.970816+0000", 2024-08-17T18:09:02.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021673504999999999 2024-08-17T18:09:02.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:09:02.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.970860+0000", 2024-08-17T18:09:02.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4206999999999999e-05 2024-08-17T18:09:02.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:02.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.972494+0000", 2024-08-17T18:09:02.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016336860000000001 2024-08-17T18:09:02.897 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.897 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:02.897 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.897 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:39695 55.5 55:a6056f3b:::benchmark_data_smithi136_313476_object39694:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1457)", 2024-08-17T18:09:02.897 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:04:14.365533+0000", 2024-08-17T18:09:02.897 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 288.44957960900001, 2024-08-17T18:09:02.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6147471759999998, 2024-08-17T18:09:02.897 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:02.898 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:09:02.898 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:02.898 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:09:02.898 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:09:02.898 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39695 2024-08-17T18:09:02.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.898 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:02.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:02.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.365533+0000", 2024-08-17T18:09:02.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:02.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.365533+0000", 2024-08-17T18:09:02.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:02.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.365532+0000", 2024-08-17T18:09:02.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:09:02.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:02.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.365600+0000", 2024-08-17T18:09:02.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7230999999999994e-05 2024-08-17T18:09:02.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:02.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.365611+0000", 2024-08-17T18:09:02.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0963e-05 2024-08-17T18:09:02.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:02.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.365616+0000", 2024-08-17T18:09:02.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4659999999999998e-06 2024-08-17T18:09:02.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.365645+0000", 2024-08-17T18:09:02.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8711e-05 2024-08-17T18:09:02.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.365651+0000", 2024-08-17T18:09:02.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9490000000000001e-06 2024-08-17T18:09:02.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521504+0000", 2024-08-17T18:09:02.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15585368199999999 2024-08-17T18:09:02.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521512+0000", 2024-08-17T18:09:02.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9759999999999995e-06 2024-08-17T18:09:02.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.964836+0000", 2024-08-17T18:09:02.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4433231790000001 2024-08-17T18:09:02.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:09:02.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.964974+0000", 2024-08-17T18:09:02.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000137997 2024-08-17T18:09:02.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-08-17T18:09:02.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.965152+0000", 2024-08-17T18:09:02.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017841600000000001 2024-08-17T18:09:02.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:09:02.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.967467+0000", 2024-08-17T18:09:02.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023148420000000001 2024-08-17T18:09:02.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:02.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.972579+0000", 2024-08-17T18:09:02.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051124680000000002 2024-08-17T18:09:02.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:02.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.979303+0000", 2024-08-17T18:09:02.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0067240920000000001 2024-08-17T18:09:02.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:09:02.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.979344+0000", 2024-08-17T18:09:02.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0309000000000003e-05 2024-08-17T18:09:02.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:02.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.980280+0000", 2024-08-17T18:09:02.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00093661600000000003 2024-08-17T18:09:02.904 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.905 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:02.905 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.905 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:39702 55.f 55:f182b387:::benchmark_data_smithi136_313476_object39701:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1457)", 2024-08-17T18:09:02.905 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:04:14.371865+0000", 2024-08-17T18:09:02.905 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 288.44324815599998, 2024-08-17T18:09:02.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6018566779999999, 2024-08-17T18:09:02.905 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:02.905 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:09:02.905 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:02.905 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:09:02.905 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:09:02.905 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39702 2024-08-17T18:09:02.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.906 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:02.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:02.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.371865+0000", 2024-08-17T18:09:02.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:02.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.371865+0000", 2024-08-17T18:09:02.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:02.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.371863+0000", 2024-08-17T18:09:02.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:09:02.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:02.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.371922+0000", 2024-08-17T18:09:02.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8881000000000001e-05 2024-08-17T18:09:02.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:02.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.371930+0000", 2024-08-17T18:09:02.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5279999999999998e-06 2024-08-17T18:09:02.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:02.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.371933+0000", 2024-08-17T18:09:02.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6200000000000001e-06 2024-08-17T18:09:02.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.371970+0000", 2024-08-17T18:09:02.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6269000000000001e-05 2024-08-17T18:09:02.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.371978+0000", 2024-08-17T18:09:02.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9249999999999995e-06 2024-08-17T18:09:02.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521462+0000", 2024-08-17T18:09:02.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.14948477499999999 2024-08-17T18:09:02.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521472+0000", 2024-08-17T18:09:02.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0064000000000001e-05 2024-08-17T18:09:02.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.947893+0000", 2024-08-17T18:09:02.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4264202830000001 2024-08-17T18:09:02.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:09:02.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.947992+0000", 2024-08-17T18:09:02.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9685000000000005e-05 2024-08-17T18:09:02.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-08-17T18:09:02.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.948102+0000", 2024-08-17T18:09:02.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010922200000000001 2024-08-17T18:09:02.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:09:02.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.949236+0000", 2024-08-17T18:09:02.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001134564 2024-08-17T18:09:02.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:02.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.950395+0000", 2024-08-17T18:09:02.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001158346 2024-08-17T18:09:02.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:02.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.972549+0000", 2024-08-17T18:09:02.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022154396999999999 2024-08-17T18:09:02.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:09:02.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.972624+0000", 2024-08-17T18:09:02.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4575999999999999e-05 2024-08-17T18:09:02.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:02.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.973721+0000", 2024-08-17T18:09:02.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001097677 2024-08-17T18:09:02.912 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.912 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:02.912 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.912 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:39707 55.5 55:a39d6cb7:::benchmark_data_smithi136_313476_object39706:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1457)", 2024-08-17T18:09:02.912 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:04:14.374950+0000", 2024-08-17T18:09:02.912 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 288.44016295500001, 2024-08-17T18:09:02.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.606183025, 2024-08-17T18:09:02.912 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:02.913 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:09:02.913 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:02.913 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:09:02.913 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:09:02.913 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39707 2024-08-17T18:09:02.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.913 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:02.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:02.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.374950+0000", 2024-08-17T18:09:02.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:02.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.374950+0000", 2024-08-17T18:09:02.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:02.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.374949+0000", 2024-08-17T18:09:02.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:09:02.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:02.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.374988+0000", 2024-08-17T18:09:02.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9433e-05 2024-08-17T18:09:02.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:02.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.374999+0000", 2024-08-17T18:09:02.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0883999999999999e-05 2024-08-17T18:09:02.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:02.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.375004+0000", 2024-08-17T18:09:02.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8840000000000002e-06 2024-08-17T18:09:02.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.375038+0000", 2024-08-17T18:09:02.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3470000000000003e-05 2024-08-17T18:09:02.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.375043+0000", 2024-08-17T18:09:02.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8760000000000002e-06 2024-08-17T18:09:02.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.916 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521535+0000", 2024-08-17T18:09:02.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.146491867 2024-08-17T18:09:02.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.916 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.521543+0000", 2024-08-17T18:09:02.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5020000000000003e-06 2024-08-17T18:09:02.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.916 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.966334+0000", 2024-08-17T18:09:02.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.444790883 2024-08-17T18:09:02.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:09:02.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.966428+0000", 2024-08-17T18:09:02.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4215999999999994e-05 2024-08-17T18:09:02.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-08-17T18:09:02.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.966533+0000", 2024-08-17T18:09:02.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105398 2024-08-17T18:09:02.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:09:02.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.967589+0000", 2024-08-17T18:09:02.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010561629999999999 2024-08-17T18:09:02.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:02.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.972623+0000", 2024-08-17T18:09:02.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050332479999999997 2024-08-17T18:09:02.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:02.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.980333+0000", 2024-08-17T18:09:02.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0077104850000000004 2024-08-17T18:09:02.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:09:02.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.980389+0000", 2024-08-17T18:09:02.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5748999999999998e-05 2024-08-17T18:09:02.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:02.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.981133+0000", 2024-08-17T18:09:02.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00074385100000000002 2024-08-17T18:09:02.919 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.919 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:02.919 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.919 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:170331 55.5 55.9d26b465 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1515)", 2024-08-17T18:09:02.919 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:08:36.079590+0000", 2024-08-17T18:09:02.920 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 26.735523052000001, 2024-08-17T18:09:02.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98240002800000004, 2024-08-17T18:09:02.920 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:02.920 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:09:02.920 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:02.920 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:09:02.920 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:09:02.920 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 170331 2024-08-17T18:09:02.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.920 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:02.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:02.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.079590+0000", 2024-08-17T18:09:02.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:02.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.079590+0000", 2024-08-17T18:09:02.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:02.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.079588+0000", 2024-08-17T18:09:02.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:09:02.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:02.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.079594+0000", 2024-08-17T18:09:02.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9970000000000004e-06 2024-08-17T18:09:02.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:02.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.079610+0000", 2024-08-17T18:09:02.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5937999999999999e-05 2024-08-17T18:09:02.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:02.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.079621+0000", 2024-08-17T18:09:02.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0431999999999999e-05 2024-08-17T18:09:02.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.079670+0000", 2024-08-17T18:09:02.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8851999999999999e-05 2024-08-17T18:09:02.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.079677+0000", 2024-08-17T18:09:02.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7989999999999994e-06 2024-08-17T18:09:02.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:37.061983+0000", 2024-08-17T18:09:02.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.982305082 2024-08-17T18:09:02.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:02.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:37.061990+0000", 2024-08-17T18:09:02.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1679999999999998e-06 2024-08-17T18:09:02.924 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.924 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:02.924 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.924 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:170338 55.15 55.2d910915 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1515)", 2024-08-17T18:09:02.924 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:08:36.081559+0000", 2024-08-17T18:09:02.924 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 26.733553581999999, 2024-08-17T18:09:02.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98040221800000005, 2024-08-17T18:09:02.924 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:02.924 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:09:02.924 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:02.924 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:09:02.925 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:09:02.925 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 170338 2024-08-17T18:09:02.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.925 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:02.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:02.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.081559+0000", 2024-08-17T18:09:02.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:02.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.081559+0000", 2024-08-17T18:09:02.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:02.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.081558+0000", 2024-08-17T18:09:02.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:09:02.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:02.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.081564+0000", 2024-08-17T18:09:02.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7930000000000003e-06 2024-08-17T18:09:02.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:02.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.081575+0000", 2024-08-17T18:09:02.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1744e-05 2024-08-17T18:09:02.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:02.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.081583+0000", 2024-08-17T18:09:02.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9410000000000007e-06 2024-08-17T18:09:02.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.081629+0000", 2024-08-17T18:09:02.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5429000000000002e-05 2024-08-17T18:09:02.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.081636+0000", 2024-08-17T18:09:02.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.503e-06 2024-08-17T18:09:02.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:37.061948+0000", 2024-08-17T18:09:02.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98031135599999997 2024-08-17T18:09:02.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:02.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:37.061961+0000", 2024-08-17T18:09:02.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.382e-05 2024-08-17T18:09:02.928 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.928 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:02.928 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.929 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:170375 55.5 55.2c832fa5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1515)", 2024-08-17T18:09:02.929 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:08:36.129274+0000", 2024-08-17T18:09:02.929 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 26.685839082000001, 2024-08-17T18:09:02.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93273893100000005, 2024-08-17T18:09:02.929 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:02.929 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:09:02.929 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:02.929 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:09:02.929 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:09:02.929 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 170375 2024-08-17T18:09:02.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.929 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:02.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:02.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.129274+0000", 2024-08-17T18:09:02.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:02.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.129274+0000", 2024-08-17T18:09:02.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:02.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.129272+0000", 2024-08-17T18:09:02.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:09:02.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:02.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.129279+0000", 2024-08-17T18:09:02.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6510000000000004e-06 2024-08-17T18:09:02.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:02.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.129291+0000", 2024-08-17T18:09:02.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1983e-05 2024-08-17T18:09:02.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:02.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.129296+0000", 2024-08-17T18:09:02.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5890000000000002e-06 2024-08-17T18:09:02.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.129330+0000", 2024-08-17T18:09:02.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3578999999999998e-05 2024-08-17T18:09:02.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.129336+0000", 2024-08-17T18:09:02.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.579e-06 2024-08-17T18:09:02.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:37.062006+0000", 2024-08-17T18:09:02.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93266979900000002 2024-08-17T18:09:02.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:02.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:37.062013+0000", 2024-08-17T18:09:02.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3319999999999997e-06 2024-08-17T18:09:02.933 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.933 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:02.933 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.933 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:170389 55.15 55.1a6e0d95 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1515)", 2024-08-17T18:09:02.933 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:08:36.147387+0000", 2024-08-17T18:09:02.933 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 26.667725621999999, 2024-08-17T18:09:02.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91465062900000005, 2024-08-17T18:09:02.934 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:02.934 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:09:02.934 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:02.934 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:09:02.934 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:09:02.934 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 170389 2024-08-17T18:09:02.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.934 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:02.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:02.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.147387+0000", 2024-08-17T18:09:02.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:02.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.147387+0000", 2024-08-17T18:09:02.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:02.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:02.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.147386+0000", 2024-08-17T18:09:02.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:09:02.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:02.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.147398+0000", 2024-08-17T18:09:02.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.242e-05 2024-08-17T18:09:02.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:02.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.147413+0000", 2024-08-17T18:09:02.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4997000000000001e-05 2024-08-17T18:09:02.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:02.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.147421+0000", 2024-08-17T18:09:02.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8739999999999995e-06 2024-08-17T18:09:02.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.147460+0000", 2024-08-17T18:09:02.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9412000000000001e-05 2024-08-17T18:09:02.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:02.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:36.147468+0000", 2024-08-17T18:09:02.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7319999999999998e-06 2024-08-17T18:09:02.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:02.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:37.062031+0000", 2024-08-17T18:09:02.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91456266100000005 2024-08-17T18:09:02.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:02.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:02.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:02.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:08:37.062038+0000", 2024-08-17T18:09:02.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0600000000000002e-06 2024-08-17T18:09:02.937 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.938 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:02.938 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.938 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:02.938 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:02.938 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:09:02.938 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:02.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:03.039 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:03.060 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:03.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:03.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:03.162 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:03.183 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:03.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:03.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:03.285 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:03.306 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:03.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:03.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:03.408 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:03.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:03.431 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:03.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:03.532 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:03.554 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:03.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:03.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:03.656 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:03.677 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:03.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:03.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:03.753 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:03 smithi165 ceph-mon[105517]: pgmap v2772: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 104 active+clean; 8.3 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 2872/450 objects degraded (638.222%); 54/450 objects misplaced (12.000%) 2024-08-17T18:09:03.776 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-17T18:09:03.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:03.840 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:03.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:03.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:03.942 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:03 smithi136 ceph-mon[109861]: pgmap v2772: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 104 active+clean; 8.3 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 2872/450 objects degraded (638.222%); 54/450 objects misplaced (12.000%) 2024-08-17T18:09:03.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:03 smithi136 ceph-mon[115078]: pgmap v2772: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 104 active+clean; 8.3 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 2872/450 objects degraded (638.222%); 54/450 objects misplaced (12.000%) 2024-08-17T18:09:03.964 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:03.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:03.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:03.986 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:09:04.066 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:04.087 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:04.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:04.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:04.189 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:04.212 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:04.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:04.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:04.314 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:04.336 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:04.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:04.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:04.460 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:04.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:04.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:04.561 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:04.584 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:04.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:04.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:04.686 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:04.708 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:04.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:04.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:04.736 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:04 smithi136 ceph-mon[109861]: osdmap e1526: 8 total, 8 up, 7 in 2024-08-17T18:09:04.736 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:04 smithi136 ceph-mon[115078]: osdmap e1526: 8 total, 8 up, 7 in 2024-08-17T18:09:04.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:04 smithi165 ceph-mon[105517]: osdmap e1526: 8 total, 8 up, 7 in 2024-08-17T18:09:04.810 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:04.832 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:04.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:04.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:04.933 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:04.954 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:04.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:04.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:05.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:05.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:05.078 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:05.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:05.179 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:05.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:05.201 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:05.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:05.302 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:05.324 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:05.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:05.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:05.426 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:05.448 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:05.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:05.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:05.477 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:05 smithi165 ceph-mon[105517]: pgmap v2774: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 104 active+clean; 8.3 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 2872/450 objects degraded (638.222%); 54/450 objects misplaced (12.000%) 2024-08-17T18:09:05.478 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:05 smithi165 ceph-mon[105517]: osdmap e1527: 8 total, 8 up, 7 in 2024-08-17T18:09:05.549 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:05.571 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:05.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:05.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:05.672 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:05.694 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:05.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:05.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:05.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:05 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 2872/450 objects degraded (638.222%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:09:05.738 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:09:05.738 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:09:05.738 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-08-17T18:09:05.738 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:09:05.796 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:05.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:05.818 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:05.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:05.919 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:05.942 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:05.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:05.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:05 smithi136 ceph-mon[109861]: pgmap v2774: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 104 active+clean; 8.3 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 2872/450 objects degraded (638.222%); 54/450 objects misplaced (12.000%) 2024-08-17T18:09:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:05 smithi136 ceph-mon[109861]: osdmap e1527: 8 total, 8 up, 7 in 2024-08-17T18:09:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:05 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 2872/450 objects degraded (638.222%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:09:05.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:05 smithi136 ceph-mon[115078]: pgmap v2774: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 104 active+clean; 8.3 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 2872/450 objects degraded (638.222%); 54/450 objects misplaced (12.000%) 2024-08-17T18:09:05.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:05 smithi136 ceph-mon[115078]: osdmap e1527: 8 total, 8 up, 7 in 2024-08-17T18:09:05.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:05 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 2872/450 objects degraded (638.222%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:09:06.043 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:06.064 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:06.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:06.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:06.166 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:06.187 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:06.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:06.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:06.290 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:06.311 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:06.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:06.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:06.413 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:06.434 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:06.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:06.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:06.440 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-17T18:09:06.536 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:06.558 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:06.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:06.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:06.653 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:09:06.660 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:06.684 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:06.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:06.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:06.785 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:06.807 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:06.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:06.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:06.909 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:06.930 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:06.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:06.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:07.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:07.054 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:07.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:07.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:07.156 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:07.177 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:07.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:07.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:07.278 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:07.300 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:07.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:07.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:07.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:07.423 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:07.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:07.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:09:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:09:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:07 smithi165 ceph-mon[105517]: pgmap v2776: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 106 active+clean; 8.3 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 403/450 objects degraded (89.556%); 54/450 objects misplaced (12.000%) 2024-08-17T18:09:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:07 smithi165 ceph-mon[105517]: 55.13 scrub starts 2024-08-17T18:09:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:07 smithi165 ceph-mon[105517]: 55.13 scrub ok 2024-08-17T18:09:07.525 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:07.546 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:07.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:07.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:07.648 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:07.669 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:07.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:07.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:07.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:09:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:09:07] "GET /metrics HTTP/1.1" 200 36603 "" "Prometheus/2.51.0" 2024-08-17T18:09:07.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:09:07.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:09:07.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:07 smithi136 ceph-mon[109861]: pgmap v2776: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 106 active+clean; 8.3 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 403/450 objects degraded (89.556%); 54/450 objects misplaced (12.000%) 2024-08-17T18:09:07.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:07 smithi136 ceph-mon[109861]: 55.13 scrub starts 2024-08-17T18:09:07.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:07 smithi136 ceph-mon[109861]: 55.13 scrub ok 2024-08-17T18:09:07.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:09:07.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:09:07.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:07 smithi136 ceph-mon[115078]: pgmap v2776: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 106 active+clean; 8.3 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 403/450 objects degraded (89.556%); 54/450 objects misplaced (12.000%) 2024-08-17T18:09:07.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:07 smithi136 ceph-mon[115078]: 55.13 scrub starts 2024-08-17T18:09:07.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:07 smithi136 ceph-mon[115078]: 55.13 scrub ok 2024-08-17T18:09:07.772 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:07.793 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:07.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:07.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:07.895 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:07.916 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:07.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:07.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:08.018 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:08.039 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:08.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:08.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:08.141 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:08.162 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:08.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:08.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:08.264 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:08.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:08.288 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:08.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:08.388 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:08.410 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:08.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:08.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:08.437 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:09:08.437 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:09:08.437 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-08-17T18:09:08.437 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-08-17T18:09:08.437 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:09:08.512 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:08.533 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:08.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:08.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:08.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:08.658 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:08.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:08.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:08.760 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:08.783 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:08.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:08.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:08.884 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:08.906 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:08.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:08.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:09.007 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:09.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:09.031 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:09.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:09.131 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-17T18:09:09.132 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:09.153 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:09.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:09.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:09.255 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:09.278 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:09.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:09.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:09.376 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:09:09.379 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:09.404 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:09.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:09.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:09.501 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:09 smithi165 ceph-mon[105517]: pgmap v2777: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 106 active+clean; 8.3 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 403/450 objects degraded (89.556%); 54/450 objects misplaced (12.000%) 2024-08-17T18:09:09.507 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:09.529 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:09.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:09.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:09.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:09.654 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:09.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:09.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:09 smithi136 ceph-mon[109861]: pgmap v2777: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 106 active+clean; 8.3 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 403/450 objects degraded (89.556%); 54/450 objects misplaced (12.000%) 2024-08-17T18:09:09.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:09 smithi136 ceph-mon[115078]: pgmap v2777: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 106 active+clean; 8.3 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 403/450 objects degraded (89.556%); 54/450 objects misplaced (12.000%) 2024-08-17T18:09:09.756 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:09.777 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:09.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:09.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:09.879 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:09.901 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:09.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:09.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:10.003 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:10.025 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:10.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:10.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:10.127 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:10.149 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:10.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:10.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:10.251 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:10.273 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:10.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:10.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:10.358 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'unique_pool_2' removed 2024-08-17T18:09:10.375 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:10.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:10.414 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:10.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:10.515 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:10.540 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:10.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:10.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:10.642 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:10.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:10 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 403/450 objects degraded (89.556%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:09:10.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:10 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4045811618' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T18:09:10.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:10 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T18:09:10.665 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:10.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:10.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:10.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:10 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 403/450 objects degraded (89.556%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:09:10.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:10 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4045811618' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T18:09:10.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:10 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T18:09:10.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:10 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 403/450 objects degraded (89.556%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:09:10.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:10 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4045811618' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T18:09:10.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:10 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T18:09:10.767 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:10.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:10.790 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:10.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:10.890 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:10.912 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:10.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:10.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:11.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:11.034 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:11.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:11.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:11.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:11.157 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:11.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:11.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:11.211 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:09:11.211 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T18:09:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T18:09:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:09:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:159609 54.6 54:6784cac7:::benchmark_data_smithi136_273969_object36724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1391)", 2024-08-17T18:09:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:00:40.232786+0000", 2024-08-17T18:09:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 510.97186004299999, 2024-08-17T18:09:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27025203800000003, 2024-08-17T18:09:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:09:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:09:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:09:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 159609 2024-08-17T18:09:11.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.232786+0000", 2024-08-17T18:09:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.232786+0000", 2024-08-17T18:09:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.232784+0000", 2024-08-17T18:09:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:09:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.232791+0000", 2024-08-17T18:09:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7499999999999997e-06 2024-08-17T18:09:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:11.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.232808+0000", 2024-08-17T18:09:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6736e-05 2024-08-17T18:09:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.232816+0000", 2024-08-17T18:09:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7219999999999999e-06 2024-08-17T18:09:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.235631+0000", 2024-08-17T18:09:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028149440000000002 2024-08-17T18:09:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:09:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.235735+0000", 2024-08-17T18:09:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104028 2024-08-17T18:09:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:09:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.237106+0000", 2024-08-17T18:09:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013709740000000001 2024-08-17T18:09:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.243897+0000", 2024-08-17T18:09:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0067915670000000001 2024-08-17T18:09:11.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:09:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.253693+0000", 2024-08-17T18:09:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0097954570000000005 2024-08-17T18:09:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.502972+0000", 2024-08-17T18:09:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.249278992 2024-08-17T18:09:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:09:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.503004+0000", 2024-08-17T18:09:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2363999999999997e-05 2024-08-17T18:09:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.503038+0000", 2024-08-17T18:09:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4196999999999998e-05 2024-08-17T18:09:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:159611 54.16 54:6efd37ec:::benchmark_data_smithi136_273969_object36726:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1391)", 2024-08-17T18:09:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:00:40.234280+0000", 2024-08-17T18:09:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 510.970366189, 2024-08-17T18:09:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.24985114899999999, 2024-08-17T18:09:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:09:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:09:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:09:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 159611 2024-08-17T18:09:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.219 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.234280+0000", 2024-08-17T18:09:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.234280+0000", 2024-08-17T18:09:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:11.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.234278+0000", 2024-08-17T18:09:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:09:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.234285+0000", 2024-08-17T18:09:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1890000000000001e-06 2024-08-17T18:09:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.234298+0000", 2024-08-17T18:09:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2957e-05 2024-08-17T18:09:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:11.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.234307+0000", 2024-08-17T18:09:11.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2069999999999995e-06 2024-08-17T18:09:11.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.235557+0000", 2024-08-17T18:09:11.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012503530000000001 2024-08-17T18:09:11.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:09:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.236018+0000", 2024-08-17T18:09:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00046107599999999999 2024-08-17T18:09:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:09:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.236780+0000", 2024-08-17T18:09:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00076228199999999998 2024-08-17T18:09:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.243861+0000", 2024-08-17T18:09:11.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070807789999999997 2024-08-17T18:09:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:09:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.253683+0000", 2024-08-17T18:09:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0098220019999999998 2024-08-17T18:09:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.484056+0000", 2024-08-17T18:09:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.23037278899999999 2024-08-17T18:09:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:09:11.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.484093+0000", 2024-08-17T18:09:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6900000000000002e-05 2024-08-17T18:09:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:00:40.484131+0000", 2024-08-17T18:09:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8368999999999998e-05 2024-08-17T18:09:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180534 54.6 54:67c0d4fa:::benchmark_data_smithi136_273969_object57649:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1398)", 2024-08-17T18:09:11.225 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:22.470651+0000", 2024-08-17T18:09:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 468.733995069, 2024-08-17T18:09:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.094188945, 2024-08-17T18:09:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:09:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:09:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:09:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180534 2024-08-17T18:09:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:11.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.470651+0000", 2024-08-17T18:09:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.470651+0000", 2024-08-17T18:09:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.470649+0000", 2024-08-17T18:09:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:09:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.470657+0000", 2024-08-17T18:09:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3540000000000002e-06 2024-08-17T18:09:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.470673+0000", 2024-08-17T18:09:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6160000000000001e-05 2024-08-17T18:09:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.470681+0000", 2024-08-17T18:09:11.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8900000000000007e-06 2024-08-17T18:09:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.470740+0000", 2024-08-17T18:09:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9429000000000003e-05 2024-08-17T18:09:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:09:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.470871+0000", 2024-08-17T18:09:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013060200000000001 2024-08-17T18:09:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-08-17T18:09:11.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.471397+0000", 2024-08-17T18:09:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00052645800000000005 2024-08-17T18:09:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:09:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.475397+0000", 2024-08-17T18:09:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039998489999999998 2024-08-17T18:09:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.492146+0000", 2024-08-17T18:09:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016749172999999999 2024-08-17T18:09:11.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.492163+0000", 2024-08-17T18:09:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6566999999999999e-05 2024-08-17T18:09:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.480962+0000", 2024-08-17T18:09:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98879862100000004 2024-08-17T18:09:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.480979+0000", 2024-08-17T18:09:11.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7252999999999999e-05 2024-08-17T18:09:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.545007+0000", 2024-08-17T18:09:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.064027783 2024-08-17T18:09:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:09:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.545364+0000", 2024-08-17T18:09:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00035749700000000002 2024-08-17T18:09:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.564746+0000", 2024-08-17T18:09:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019382160999999998 2024-08-17T18:09:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.564840+0000", 2024-08-17T18:09:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3802999999999996e-05 2024-08-17T18:09:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.233 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180546 54.6 54:654acf96:::benchmark_data_smithi136_273969_object57661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1399)", 2024-08-17T18:09:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:22.496882+0000", 2024-08-17T18:09:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 468.70776459000001, 2024-08-17T18:09:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0682568360000002, 2024-08-17T18:09:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:09:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:09:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:09:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180546 2024-08-17T18:09:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.496882+0000", 2024-08-17T18:09:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.496882+0000", 2024-08-17T18:09:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.496880+0000", 2024-08-17T18:09:11.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:09:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.496888+0000", 2024-08-17T18:09:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.605e-06 2024-08-17T18:09:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.496904+0000", 2024-08-17T18:09:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5965e-05 2024-08-17T18:09:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:11.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.496912+0000", 2024-08-17T18:09:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3890000000000005e-06 2024-08-17T18:09:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.496955+0000", 2024-08-17T18:09:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2945000000000003e-05 2024-08-17T18:09:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.496967+0000", 2024-08-17T18:09:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1804000000000001e-05 2024-08-17T18:09:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.481006+0000", 2024-08-17T18:09:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98403964499999996 2024-08-17T18:09:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.481018+0000", 2024-08-17T18:09:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.17e-05 2024-08-17T18:09:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.547118+0000", 2024-08-17T18:09:11.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.066099849 2024-08-17T18:09:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:09:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.547289+0000", 2024-08-17T18:09:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017115199999999999 2024-08-17T18:09:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:09:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.550740+0000", 2024-08-17T18:09:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034507330000000001 2024-08-17T18:09:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:09:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.564368+0000", 2024-08-17T18:09:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013628049 2024-08-17T18:09:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.564881+0000", 2024-08-17T18:09:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00051272199999999996 2024-08-17T18:09:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.564928+0000", 2024-08-17T18:09:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7463000000000003e-05 2024-08-17T18:09:11.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:09:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.565010+0000", 2024-08-17T18:09:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1538000000000004e-05 2024-08-17T18:09:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.565138+0000", 2024-08-17T18:09:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012886700000000001 2024-08-17T18:09:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.242 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:180552 54.6 54:634ae437:::benchmark_data_smithi136_273969_object57667:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1399)", 2024-08-17T18:09:11.242 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:22.507651+0000", 2024-08-17T18:09:11.242 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 468.696995023, 2024-08-17T18:09:11.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0683632780000001, 2024-08-17T18:09:11.242 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:11.242 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:09:11.242 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:11.242 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:09:11.242 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:09:11.242 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 180552 2024-08-17T18:09:11.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.242 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:11.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:11.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.507651+0000", 2024-08-17T18:09:11.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:11.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.507651+0000", 2024-08-17T18:09:11.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:11.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.507650+0000", 2024-08-17T18:09:11.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:09:11.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:11.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.507657+0000", 2024-08-17T18:09:11.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1409999999999999e-06 2024-08-17T18:09:11.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:11.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.507671+0000", 2024-08-17T18:09:11.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3900000000000001e-05 2024-08-17T18:09:11.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:11.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.507679+0000", 2024-08-17T18:09:11.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1790000000000008e-06 2024-08-17T18:09:11.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.507722+0000", 2024-08-17T18:09:11.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3228e-05 2024-08-17T18:09:11.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:11.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:22.507731+0000", 2024-08-17T18:09:11.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8720000000000008e-06 2024-08-17T18:09:11.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.481042+0000", 2024-08-17T18:09:11.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97331098800000004 2024-08-17T18:09:11.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:11.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:23.481052+0000", 2024-08-17T18:09:11.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3680000000000008e-06 2024-08-17T18:09:11.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.565166+0000", 2024-08-17T18:09:11.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.084114867 2024-08-17T18:09:11.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:09:11.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.565289+0000", 2024-08-17T18:09:11.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122149 2024-08-17T18:09:11.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:09:11.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.565853+0000", 2024-08-17T18:09:11.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00056417499999999998 2024-08-17T18:09:11.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.572028+0000", 2024-08-17T18:09:11.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006175168 2024-08-17T18:09:11.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.572280+0000", 2024-08-17T18:09:11.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025191900000000003 2024-08-17T18:09:11.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:09:11.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.575814+0000", 2024-08-17T18:09:11.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003534354 2024-08-17T18:09:11.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:09:11.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.575868+0000", 2024-08-17T18:09:11.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4044e-05 2024-08-17T18:09:11.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:11.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:24.576014+0000", 2024-08-17T18:09:11.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000146231 2024-08-17T18:09:11.249 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.249 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:11.249 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.250 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:181732 54.6 54:61f036d9:::benchmark_data_smithi136_273969_object58847:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1404)", 2024-08-17T18:09:11.250 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:28.584172+0000", 2024-08-17T18:09:11.250 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 462.62047407099999, 2024-08-17T18:09:11.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97332284800000002, 2024-08-17T18:09:11.250 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:11.250 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:09:11.250 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:11.250 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:09:11.250 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:09:11.250 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 181732 2024-08-17T18:09:11.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.250 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:11.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:11.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.584172+0000", 2024-08-17T18:09:11.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:11.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.584172+0000", 2024-08-17T18:09:11.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:11.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.584171+0000", 2024-08-17T18:09:11.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:09:11.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:11.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.584178+0000", 2024-08-17T18:09:11.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9549999999999992e-06 2024-08-17T18:09:11.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:11.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.584191+0000", 2024-08-17T18:09:11.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2833999999999999e-05 2024-08-17T18:09:11.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:11.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.584200+0000", 2024-08-17T18:09:11.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4300000000000006e-06 2024-08-17T18:09:11.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.584258+0000", 2024-08-17T18:09:11.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8306e-05 2024-08-17T18:09:11.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:11.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.584270+0000", 2024-08-17T18:09:11.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1439e-05 2024-08-17T18:09:11.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.550255+0000", 2024-08-17T18:09:11.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96598526100000004 2024-08-17T18:09:11.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:09:11.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.550397+0000", 2024-08-17T18:09:11.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000141786 2024-08-17T18:09:11.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:09:11.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.550892+0000", 2024-08-17T18:09:11.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00049522499999999996 2024-08-17T18:09:11.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.557345+0000", 2024-08-17T18:09:11.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064536339999999998 2024-08-17T18:09:11.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:09:11.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.557353+0000", 2024-08-17T18:09:11.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4390000000000003e-06 2024-08-17T18:09:11.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.557411+0000", 2024-08-17T18:09:11.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7663e-05 2024-08-17T18:09:11.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:09:11.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.557458+0000", 2024-08-17T18:09:11.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7821999999999997e-05 2024-08-17T18:09:11.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:11.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.557495+0000", 2024-08-17T18:09:11.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6634999999999997e-05 2024-08-17T18:09:11.258 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.258 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:11.258 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.258 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:181748 54.6 54:62195207:::benchmark_data_smithi136_273969_object58863:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1404)", 2024-08-17T18:09:11.258 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:28.599388+0000", 2024-08-17T18:09:11.258 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 462.60525862999998, 2024-08-17T18:09:11.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.968979798, 2024-08-17T18:09:11.259 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:11.259 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:09:11.259 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:11.259 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:09:11.259 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:09:11.259 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 181748 2024-08-17T18:09:11.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.259 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:11.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:11.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.599388+0000", 2024-08-17T18:09:11.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:11.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.599388+0000", 2024-08-17T18:09:11.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:11.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.599385+0000", 2024-08-17T18:09:11.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999976 2024-08-17T18:09:11.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:11.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.599396+0000", 2024-08-17T18:09:11.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0379e-05 2024-08-17T18:09:11.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:11.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.599415+0000", 2024-08-17T18:09:11.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8791999999999999e-05 2024-08-17T18:09:11.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:11.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.599422+0000", 2024-08-17T18:09:11.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7270000000000007e-06 2024-08-17T18:09:11.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.601178+0000", 2024-08-17T18:09:11.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017560519999999999 2024-08-17T18:09:11.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:11.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.601189+0000", 2024-08-17T18:09:11.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1075e-05 2024-08-17T18:09:11.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.557532+0000", 2024-08-17T18:09:11.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.95634238400000005 2024-08-17T18:09:11.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:09:11.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.557618+0000", 2024-08-17T18:09:11.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6446999999999994e-05 2024-08-17T18:09:11.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:09:11.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.557998+0000", 2024-08-17T18:09:11.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038008600000000002 2024-08-17T18:09:11.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.560490+0000", 2024-08-17T18:09:11.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024916109999999999 2024-08-17T18:09:11.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.563471+0000", 2024-08-17T18:09:11.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029810409999999998 2024-08-17T18:09:11.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:09:11.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.568286+0000", 2024-08-17T18:09:11.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048147629999999997 2024-08-17T18:09:11.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:09:11.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.568331+0000", 2024-08-17T18:09:11.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4802000000000002e-05 2024-08-17T18:09:11.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:11.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.568367+0000", 2024-08-17T18:09:11.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6850999999999997e-05 2024-08-17T18:09:11.265 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.265 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:11.265 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.266 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:181769 54.6 54:60a32977:::benchmark_data_smithi136_273969_object58884:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1404)", 2024-08-17T18:09:11.266 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:28.615453+0000", 2024-08-17T18:09:11.266 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 462.58919333900002, 2024-08-17T18:09:11.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.95900557200000003, 2024-08-17T18:09:11.266 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:11.266 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:09:11.266 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:11.266 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:09:11.266 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:09:11.266 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 181769 2024-08-17T18:09:11.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.266 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:11.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:11.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.615453+0000", 2024-08-17T18:09:11.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:11.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.615453+0000", 2024-08-17T18:09:11.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:11.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.615452+0000", 2024-08-17T18:09:11.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:09:11.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:11.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.615460+0000", 2024-08-17T18:09:11.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.593e-06 2024-08-17T18:09:11.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:11.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.615470+0000", 2024-08-17T18:09:11.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9839999999999996e-06 2024-08-17T18:09:11.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:11.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.615477+0000", 2024-08-17T18:09:11.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0230000000000004e-06 2024-08-17T18:09:11.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.618044+0000", 2024-08-17T18:09:11.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002567826 2024-08-17T18:09:11.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:11.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.618075+0000", 2024-08-17T18:09:11.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0349e-05 2024-08-17T18:09:11.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.563485+0000", 2024-08-17T18:09:11.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94540980500000005 2024-08-17T18:09:11.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:09:11.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.563589+0000", 2024-08-17T18:09:11.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103991 2024-08-17T18:09:11.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:09:11.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.563981+0000", 2024-08-17T18:09:11.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039240299999999999 2024-08-17T18:09:11.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.568440+0000", 2024-08-17T18:09:11.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044587860000000002 2024-08-17T18:09:11.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:09:11.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.568506+0000", 2024-08-17T18:09:11.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6635000000000002e-05 2024-08-17T18:09:11.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.574371+0000", 2024-08-17T18:09:11.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058649649999999998 2024-08-17T18:09:11.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:09:11.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.574418+0000", 2024-08-17T18:09:11.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6940000000000001e-05 2024-08-17T18:09:11.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:11.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.574458+0000", 2024-08-17T18:09:11.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0076000000000001e-05 2024-08-17T18:09:11.273 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.273 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:11.273 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.273 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:181770 54.6 54:66140edd:::benchmark_data_smithi136_273969_object58885:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1404)", 2024-08-17T18:09:11.273 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:28.617526+0000", 2024-08-17T18:09:11.273 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 462.58711998799998, 2024-08-17T18:09:11.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96320027900000005, 2024-08-17T18:09:11.273 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:11.273 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:09:11.273 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:11.274 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:09:11.274 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:09:11.274 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 181770 2024-08-17T18:09:11.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.274 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:11.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:11.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.617526+0000", 2024-08-17T18:09:11.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:11.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.617526+0000", 2024-08-17T18:09:11.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:11.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.617525+0000", 2024-08-17T18:09:11.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:09:11.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:11.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.617532+0000", 2024-08-17T18:09:11.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2320000000000004e-06 2024-08-17T18:09:11.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:11.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.617549+0000", 2024-08-17T18:09:11.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6668999999999999e-05 2024-08-17T18:09:11.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:11.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.617557+0000", 2024-08-17T18:09:11.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1019999999999997e-06 2024-08-17T18:09:11.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.618186+0000", 2024-08-17T18:09:11.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062925800000000005 2024-08-17T18:09:11.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:11.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.618207+0000", 2024-08-17T18:09:11.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1035000000000001e-05 2024-08-17T18:09:11.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.565896+0000", 2024-08-17T18:09:11.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94768889700000003 2024-08-17T18:09:11.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:09:11.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.565998+0000", 2024-08-17T18:09:11.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010196600000000001 2024-08-17T18:09:11.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:09:11.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.566323+0000", 2024-08-17T18:09:11.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000324863 2024-08-17T18:09:11.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.574334+0000", 2024-08-17T18:09:11.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080109880000000001 2024-08-17T18:09:11.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:09:11.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.574470+0000", 2024-08-17T18:09:11.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013613699999999999 2024-08-17T18:09:11.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.580652+0000", 2024-08-17T18:09:11.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0061821259999999996 2024-08-17T18:09:11.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:09:11.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.580687+0000", 2024-08-17T18:09:11.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5280000000000001e-05 2024-08-17T18:09:11.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:11.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.580726+0000", 2024-08-17T18:09:11.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9251000000000001e-05 2024-08-17T18:09:11.280 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.280 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:11.280 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.280 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:181849 54.6 54:62024122:::benchmark_data_smithi136_273969_object58964:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1404)", 2024-08-17T18:09:11.280 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:28.710599+0000", 2024-08-17T18:09:11.280 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 462.49404715899999, 2024-08-17T18:09:11.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.870281309, 2024-08-17T18:09:11.280 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:11.280 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:09:11.280 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:11.280 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:09:11.281 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:09:11.281 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 181849 2024-08-17T18:09:11.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.281 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:11.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:11.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.710599+0000", 2024-08-17T18:09:11.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:11.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.710599+0000", 2024-08-17T18:09:11.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:11.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.710597+0000", 2024-08-17T18:09:11.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:09:11.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:11.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.710604+0000", 2024-08-17T18:09:11.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8920000000000003e-06 2024-08-17T18:09:11.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:11.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.710621+0000", 2024-08-17T18:09:11.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7091999999999998e-05 2024-08-17T18:09:11.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:11.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.710629+0000", 2024-08-17T18:09:11.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9300000000000003e-06 2024-08-17T18:09:11.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.710707+0000", 2024-08-17T18:09:11.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8195999999999995e-05 2024-08-17T18:09:11.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:11.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:28.710719+0000", 2024-08-17T18:09:11.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1787000000000001e-05 2024-08-17T18:09:11.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.568535+0000", 2024-08-17T18:09:11.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.85781581699999998 2024-08-17T18:09:11.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:09:11.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.568597+0000", 2024-08-17T18:09:11.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2119000000000006e-05 2024-08-17T18:09:11.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:09:11.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.568907+0000", 2024-08-17T18:09:11.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000310303 2024-08-17T18:09:11.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:09:11.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.574474+0000", 2024-08-17T18:09:11.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055670559999999999 2024-08-17T18:09:11.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.574505+0000", 2024-08-17T18:09:11.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1010999999999998e-05 2024-08-17T18:09:11.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.580806+0000", 2024-08-17T18:09:11.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063007560000000002 2024-08-17T18:09:11.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:09:11.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.580847+0000", 2024-08-17T18:09:11.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0602999999999997e-05 2024-08-17T18:09:11.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:11.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.580880+0000", 2024-08-17T18:09:11.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3587000000000002e-05 2024-08-17T18:09:11.287 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.287 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:11.287 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.287 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:182078 54.6 54:61ee2bb7:::benchmark_data_smithi136_273969_object59193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1404)", 2024-08-17T18:09:11.287 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:29.119925+0000", 2024-08-17T18:09:11.287 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 462.08472092199997, 2024-08-17T18:09:11.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.46105587100000001, 2024-08-17T18:09:11.287 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:11.288 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:09:11.288 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:11.288 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:09:11.288 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:09:11.288 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 182078 2024-08-17T18:09:11.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.288 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:11.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:11.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.119925+0000", 2024-08-17T18:09:11.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:11.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.119925+0000", 2024-08-17T18:09:11.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:11.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.119924+0000", 2024-08-17T18:09:11.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:09:11.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:11.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.119931+0000", 2024-08-17T18:09:11.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2819999999999999e-06 2024-08-17T18:09:11.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:11.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.119945+0000", 2024-08-17T18:09:11.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4620000000000001e-05 2024-08-17T18:09:11.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:11.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.119953+0000", 2024-08-17T18:09:11.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8900000000000007e-06 2024-08-17T18:09:11.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.120024+0000", 2024-08-17T18:09:11.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0846000000000006e-05 2024-08-17T18:09:11.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:11.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.120037+0000", 2024-08-17T18:09:11.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2741999999999999e-05 2024-08-17T18:09:11.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.571351+0000", 2024-08-17T18:09:11.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.45131405699999999 2024-08-17T18:09:11.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:09:11.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.571465+0000", 2024-08-17T18:09:11.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000114088 2024-08-17T18:09:11.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:09:11.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.571840+0000", 2024-08-17T18:09:11.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037458700000000001 2024-08-17T18:09:11.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.574536+0000", 2024-08-17T18:09:11.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026967430000000001 2024-08-17T18:09:11.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:09:11.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.580599+0000", 2024-08-17T18:09:11.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060625599999999998 2024-08-17T18:09:11.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.580915+0000", 2024-08-17T18:09:11.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00031589200000000002 2024-08-17T18:09:11.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:09:11.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.580951+0000", 2024-08-17T18:09:11.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6195000000000002e-05 2024-08-17T18:09:11.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:11.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.580981+0000", 2024-08-17T18:09:11.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0100999999999999e-05 2024-08-17T18:09:11.294 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.294 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:11.294 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.295 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:182090 54.6 54:63127532:::benchmark_data_smithi136_273969_object59205:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1404)", 2024-08-17T18:09:11.295 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:29.143721+0000", 2024-08-17T18:09:11.295 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 462.06092541999999, 2024-08-17T18:09:11.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.44294536200000001, 2024-08-17T18:09:11.295 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:11.295 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:09:11.295 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:11.295 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:09:11.295 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:09:11.295 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 182090 2024-08-17T18:09:11.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.295 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:11.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:11.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.143721+0000", 2024-08-17T18:09:11.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:11.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.143721+0000", 2024-08-17T18:09:11.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:11.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.143719+0000", 2024-08-17T18:09:11.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:09:11.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:11.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.143726+0000", 2024-08-17T18:09:11.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1099999999999997e-06 2024-08-17T18:09:11.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:11.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.143741+0000", 2024-08-17T18:09:11.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4358999999999999e-05 2024-08-17T18:09:11.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:11.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.143750+0000", 2024-08-17T18:09:11.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4320000000000005e-06 2024-08-17T18:09:11.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.143805+0000", 2024-08-17T18:09:11.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5044999999999999e-05 2024-08-17T18:09:11.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:11.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.143818+0000", 2024-08-17T18:09:11.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2279999999999999e-05 2024-08-17T18:09:11.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.574558+0000", 2024-08-17T18:09:11.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.43074056500000002 2024-08-17T18:09:11.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:09:11.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.574662+0000", 2024-08-17T18:09:11.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103529 2024-08-17T18:09:11.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:09:11.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.575113+0000", 2024-08-17T18:09:11.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000451302 2024-08-17T18:09:11.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:09:11.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.580740+0000", 2024-08-17T18:09:11.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00562671 2024-08-17T18:09:11.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.580775+0000", 2024-08-17T18:09:11.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5703e-05 2024-08-17T18:09:11.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.586573+0000", 2024-08-17T18:09:11.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057977080000000004 2024-08-17T18:09:11.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:09:11.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.586625+0000", 2024-08-17T18:09:11.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2104999999999997e-05 2024-08-17T18:09:11.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:11.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.586666+0000", 2024-08-17T18:09:11.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0980999999999999e-05 2024-08-17T18:09:11.301 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.301 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:11.302 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.302 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:182101 54.6 54:61568d88:::benchmark_data_smithi136_273969_object59216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1404)", 2024-08-17T18:09:11.302 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:29.163462+0000", 2024-08-17T18:09:11.302 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 462.04118409, 2024-08-17T18:09:11.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.43511173400000003, 2024-08-17T18:09:11.302 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:11.302 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:09:11.302 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:11.302 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:09:11.302 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:09:11.302 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 182101 2024-08-17T18:09:11.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.303 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:11.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:11.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.163462+0000", 2024-08-17T18:09:11.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:11.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.163462+0000", 2024-08-17T18:09:11.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:11.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.163461+0000", 2024-08-17T18:09:11.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:09:11.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:11.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.163467+0000", 2024-08-17T18:09:11.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6409999999999996e-06 2024-08-17T18:09:11.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:11.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.163480+0000", 2024-08-17T18:09:11.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2959000000000001e-05 2024-08-17T18:09:11.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:11.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.163488+0000", 2024-08-17T18:09:11.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3719999999999999e-06 2024-08-17T18:09:11.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.163545+0000", 2024-08-17T18:09:11.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7404999999999997e-05 2024-08-17T18:09:11.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:11.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.163556+0000", 2024-08-17T18:09:11.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1049999999999999e-05 2024-08-17T18:09:11.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.577624+0000", 2024-08-17T18:09:11.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.41406769500000001 2024-08-17T18:09:11.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:09:11.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.577736+0000", 2024-08-17T18:09:11.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112447 2024-08-17T18:09:11.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:09:11.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.578157+0000", 2024-08-17T18:09:11.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042098 2024-08-17T18:09:11.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.582945+0000", 2024-08-17T18:09:11.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0047872239999999996 2024-08-17T18:09:11.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:09:11.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.593285+0000", 2024-08-17T18:09:11.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010339881 2024-08-17T18:09:11.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.598455+0000", 2024-08-17T18:09:11.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005170231 2024-08-17T18:09:11.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:09:11.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.598539+0000", 2024-08-17T18:09:11.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3864999999999997e-05 2024-08-17T18:09:11.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:11.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.598574+0000", 2024-08-17T18:09:11.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5231000000000002e-05 2024-08-17T18:09:11.309 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.309 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:11.309 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.309 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:182102 54.6 54:65730978:::benchmark_data_smithi136_273969_object59217:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1404)", 2024-08-17T18:09:11.309 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:29.165778+0000", 2024-08-17T18:09:11.310 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 462.03886806399998, 2024-08-17T18:09:11.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.439528425, 2024-08-17T18:09:11.310 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:11.310 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:09:11.310 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:11.310 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:09:11.310 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:09:11.310 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 182102 2024-08-17T18:09:11.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.310 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:11.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:11.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.165778+0000", 2024-08-17T18:09:11.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:11.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.165778+0000", 2024-08-17T18:09:11.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:11.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.165777+0000", 2024-08-17T18:09:11.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:09:11.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:11.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.165786+0000", 2024-08-17T18:09:11.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4400000000000005e-06 2024-08-17T18:09:11.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:11.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.165797+0000", 2024-08-17T18:09:11.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0902e-05 2024-08-17T18:09:11.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:11.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.165804+0000", 2024-08-17T18:09:11.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7719999999999994e-06 2024-08-17T18:09:11.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.165842+0000", 2024-08-17T18:09:11.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7387e-05 2024-08-17T18:09:11.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:11.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.165849+0000", 2024-08-17T18:09:11.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3699999999999997e-06 2024-08-17T18:09:11.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.582971+0000", 2024-08-17T18:09:11.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.41712217099999999 2024-08-17T18:09:11.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:09:11.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.583145+0000", 2024-08-17T18:09:11.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000173318 2024-08-17T18:09:11.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:09:11.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.583632+0000", 2024-08-17T18:09:11.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000487212 2024-08-17T18:09:11.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.593251+0000", 2024-08-17T18:09:11.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0096192840000000005 2024-08-17T18:09:11.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.598634+0000", 2024-08-17T18:09:11.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0053824900000000002 2024-08-17T18:09:11.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:09:11.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.605187+0000", 2024-08-17T18:09:11.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0065531560000000001 2024-08-17T18:09:11.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:09:11.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.605268+0000", 2024-08-17T18:09:11.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1205999999999995e-05 2024-08-17T18:09:11.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:11.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.605307+0000", 2024-08-17T18:09:11.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8643000000000003e-05 2024-08-17T18:09:11.316 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.316 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:11.316 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.316 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:182163 54.6 54:66825484:::benchmark_data_smithi136_273969_object59278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1404)", 2024-08-17T18:09:11.316 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:01:29.346484+0000", 2024-08-17T18:09:11.316 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 461.85816174799999, 2024-08-17T18:09:11.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27071300700000001, 2024-08-17T18:09:11.317 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:09:11.317 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:09:11.317 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:09:11.317 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:09:11.317 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:09:11.317 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 182163 2024-08-17T18:09:11.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.317 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:09:11.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:09:11.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.346484+0000", 2024-08-17T18:09:11.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:09:11.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.346484+0000", 2024-08-17T18:09:11.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:09:11.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:09:11.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.346482+0000", 2024-08-17T18:09:11.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:09:11.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:09:11.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.346491+0000", 2024-08-17T18:09:11.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2770000000000005e-06 2024-08-17T18:09:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:09:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.346506+0000", 2024-08-17T18:09:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5622999999999999e-05 2024-08-17T18:09:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:09:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.346514+0000", 2024-08-17T18:09:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1860000000000003e-06 2024-08-17T18:09:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.346567+0000", 2024-08-17T18:09:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3049e-05 2024-08-17T18:09:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:09:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.346580+0000", 2024-08-17T18:09:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3906e-05 2024-08-17T18:09:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:09:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.586707+0000", 2024-08-17T18:09:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.24012646200000001 2024-08-17T18:09:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:09:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.586826+0000", 2024-08-17T18:09:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118931 2024-08-17T18:09:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:09:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.587317+0000", 2024-08-17T18:09:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00049125099999999995 2024-08-17T18:09:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.593359+0000", 2024-08-17T18:09:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060414209999999999 2024-08-17T18:09:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:09:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.598661+0000", 2024-08-17T18:09:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005302261 2024-08-17T18:09:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:09:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.617115+0000", 2024-08-17T18:09:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018453801999999998 2024-08-17T18:09:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:09:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.617162+0000", 2024-08-17T18:09:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7030000000000002e-05 2024-08-17T18:09:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:09:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:09:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:09:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:01:29.617197+0000", 2024-08-17T18:09:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5814e-05 2024-08-17T18:09:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:09:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:09:11.323 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:09:11.324 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-08-17T18:09:11.324 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T18:09:11.324 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:11.345 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:11.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:11.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:11.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:11.468 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:11.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:11.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:11.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:11 smithi165 ceph-mon[105517]: 55.18 scrub starts 2024-08-17T18:09:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:11 smithi165 ceph-mon[105517]: pgmap v2778: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 106 active+clean; 8.3 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 403/450 objects degraded (89.556%); 54/450 objects misplaced (12.000%); 0 B/s, 265 objects/s recovering 2024-08-17T18:09:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:11 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T18:09:11.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:11 smithi165 ceph-mon[105517]: osdmap e1528: 8 total, 8 up, 7 in 2024-08-17T18:09:11.570 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:11.592 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:11.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:11.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:11.694 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:11.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:11 smithi136 ceph-mon[109861]: 55.18 scrub starts 2024-08-17T18:09:11.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:11 smithi136 ceph-mon[109861]: pgmap v2778: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 106 active+clean; 8.3 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 403/450 objects degraded (89.556%); 54/450 objects misplaced (12.000%); 0 B/s, 265 objects/s recovering 2024-08-17T18:09:11.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:11 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T18:09:11.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:11 smithi136 ceph-mon[109861]: osdmap e1528: 8 total, 8 up, 7 in 2024-08-17T18:09:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:11 smithi136 ceph-mon[115078]: 55.18 scrub starts 2024-08-17T18:09:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:11 smithi136 ceph-mon[115078]: pgmap v2778: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 106 active+clean; 8.3 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 403/450 objects degraded (89.556%); 54/450 objects misplaced (12.000%); 0 B/s, 265 objects/s recovering 2024-08-17T18:09:11.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:11 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T18:09:11.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:11 smithi136 ceph-mon[115078]: osdmap e1528: 8 total, 8 up, 7 in 2024-08-17T18:09:11.716 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:11.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:11.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:11.817 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:11.840 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:11.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:11.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:11.942 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:11.963 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:11.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:11.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:11.987 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-17T18:09:12.066 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:12.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:12.087 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:12.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:12.188 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:12.211 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:12.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:12.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:12.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:12.334 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:12.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:12.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:12.436 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:12.457 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:12.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:12.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:12.559 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:12.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:12.581 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:12.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:12.682 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:12.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-08-17T18:09:12.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-08-17T18:09:12.704 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:12.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:12.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:12.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:12 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-08-17T18:09:12.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:12.828 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:12.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:12.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:12.929 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:12.952 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:12.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:12.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:13.053 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:13.079 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:13.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:13.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:13.181 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:13.207 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:13.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:13.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:13.308 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:13.334 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:13.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:13.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:13.438 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:13.462 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:13.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:13.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:13.564 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:13.587 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:13.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:13.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:13.689 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:13.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:13 smithi136 ceph-mon[109861]: pgmap v2780: 83 pgs: 83 active+clean; 582 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:13.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:13 smithi136 ceph-mon[109861]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 403/450 objects degraded (89.556%), 1 pg degraded) 2024-08-17T18:09:13.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:13 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T18:09:13.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:13 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-08-17T18:09:13.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:13 smithi136 ceph-mon[109861]: osdmap e1529: 8 total, 8 up, 7 in 2024-08-17T18:09:13.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:13 smithi136 ceph-mon[115078]: pgmap v2780: 83 pgs: 83 active+clean; 582 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:13.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:13 smithi136 ceph-mon[115078]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 403/450 objects degraded (89.556%), 1 pg degraded) 2024-08-17T18:09:13.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:13 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T18:09:13.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:13 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-08-17T18:09:13.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:13 smithi136 ceph-mon[115078]: osdmap e1529: 8 total, 8 up, 7 in 2024-08-17T18:09:13.712 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:13.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:13.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:13 smithi165 ceph-mon[105517]: pgmap v2780: 83 pgs: 83 active+clean; 582 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:13 smithi165 ceph-mon[105517]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 403/450 objects degraded (89.556%), 1 pg degraded) 2024-08-17T18:09:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:13 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T18:09:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:13 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-08-17T18:09:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:13 smithi165 ceph-mon[105517]: osdmap e1529: 8 total, 8 up, 7 in 2024-08-17T18:09:13.814 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:13.836 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:13.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:13.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:13.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:13.960 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:13.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:13.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:14.061 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:14.087 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:14.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:14.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:14.189 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:14.214 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:14.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:14.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:14.226 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:09:14.285 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:09:14.316 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:14.337 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:14.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:14.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:14.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:14.463 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:14.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:14.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:14.564 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:14.588 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:14.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:14.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:14.689 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:14.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:14 smithi136 ceph-mon[109861]: osdmap e1530: 8 total, 8 up, 7 in 2024-08-17T18:09:14.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:14 smithi136 ceph-mon[115078]: osdmap e1530: 8 total, 8 up, 7 in 2024-08-17T18:09:14.714 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:14.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:14.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:14 smithi165 ceph-mon[105517]: osdmap e1530: 8 total, 8 up, 7 in 2024-08-17T18:09:14.815 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:14.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:14.838 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:14.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:14.939 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:14.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:14.966 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:14.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:15.067 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:15.092 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:15.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:15.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:15.193 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:15.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:15.220 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:15.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:15.321 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:15.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:15.347 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:15.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:15.448 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:15.474 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:15.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:15.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:15.576 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:15.606 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:15.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:15.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:15.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:15 smithi136 ceph-mon[109861]: pgmap v2783: 83 pgs: 1 unknown, 82 active+clean; 582 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:15.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:15 smithi136 ceph-mon[109861]: osdmap e1531: 8 total, 8 up, 7 in 2024-08-17T18:09:15.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:15 smithi136 ceph-mon[109861]: osdmap e1532: 8 total, 8 up, 7 in 2024-08-17T18:09:15.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:15 smithi136 ceph-mon[115078]: pgmap v2783: 83 pgs: 1 unknown, 82 active+clean; 582 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:15.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:15 smithi136 ceph-mon[115078]: osdmap e1531: 8 total, 8 up, 7 in 2024-08-17T18:09:15.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:15 smithi136 ceph-mon[115078]: osdmap e1532: 8 total, 8 up, 7 in 2024-08-17T18:09:15.708 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:15.734 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:15.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:15.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:15 smithi165 ceph-mon[105517]: pgmap v2783: 83 pgs: 1 unknown, 82 active+clean; 582 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:15 smithi165 ceph-mon[105517]: osdmap e1531: 8 total, 8 up, 7 in 2024-08-17T18:09:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:15 smithi165 ceph-mon[105517]: osdmap e1532: 8 total, 8 up, 7 in 2024-08-17T18:09:15.836 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:15.861 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:15.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:15.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:15.963 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:15.988 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:15.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:15.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:16.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:16.119 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:16.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:16.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:16.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:16.246 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:16.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:16.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:16.347 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:16.374 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:16.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:16.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:16.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:16.487 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:09:16.501 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:16.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:16.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:16.509 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-17T18:09:16.510 INFO:tasks.radosbench:Beginning radosbench... 2024-08-17T18:09:16.510 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-08-17T18:09:16.510 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-08-17T18:09:16.603 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:16.624 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:16.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:16.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:16.725 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:16.738 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:16 smithi136 ceph-mon[109861]: osdmap e1533: 8 total, 8 up, 7 in 2024-08-17T18:09:16.739 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:16 smithi136 ceph-mon[109861]: 1.3 scrub starts 2024-08-17T18:09:16.739 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:16 smithi136 ceph-mon[109861]: 1.3 scrub ok 2024-08-17T18:09:16.739 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:16 smithi136 ceph-mon[115078]: osdmap e1533: 8 total, 8 up, 7 in 2024-08-17T18:09:16.739 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:16 smithi136 ceph-mon[115078]: 1.3 scrub starts 2024-08-17T18:09:16.739 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:16 smithi136 ceph-mon[115078]: 1.3 scrub ok 2024-08-17T18:09:16.751 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:16.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:16.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:16.853 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:16.875 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:16.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:16.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:16.977 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:16.998 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:16.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:16.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:16 smithi165 ceph-mon[105517]: osdmap e1533: 8 total, 8 up, 7 in 2024-08-17T18:09:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:16 smithi165 ceph-mon[105517]: 1.3 scrub starts 2024-08-17T18:09:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:16 smithi165 ceph-mon[105517]: 1.3 scrub ok 2024-08-17T18:09:17.100 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:17.121 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:17.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:17.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:17.223 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:17.248 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:17.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:17.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:17.350 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:17.371 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:17.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:17.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:17.473 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:17.499 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:17.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:17.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:17.600 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:17.622 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:17.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:17.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:17.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:17 smithi136 ceph-mon[115078]: pgmap v2787: 83 pgs: 1 peering, 1 unknown, 81 active+clean; 634 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 19 KiB/s rd, 56 KiB/s wr, 13 op/s; 53 KiB/s, 0 objects/s recovering 2024-08-17T18:09:17.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:17 smithi136 ceph-mon[109861]: pgmap v2787: 83 pgs: 1 peering, 1 unknown, 81 active+clean; 634 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 19 KiB/s rd, 56 KiB/s wr, 13 op/s; 53 KiB/s, 0 objects/s recovering 2024-08-17T18:09:17.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:09:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:09:17] "GET /metrics HTTP/1.1" 200 36603 "" "Prometheus/2.51.0" 2024-08-17T18:09:17.724 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:17.745 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:17.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:17.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:17.847 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:17.868 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:17.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:17.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:17.970 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:17.990 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:17.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:17.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:17 smithi165 ceph-mon[105517]: pgmap v2787: 83 pgs: 1 peering, 1 unknown, 81 active+clean; 634 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 19 KiB/s rd, 56 KiB/s wr, 13 op/s; 53 KiB/s, 0 objects/s recovering 2024-08-17T18:09:18.092 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:18.118 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:18.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:18.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:18.220 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:18.242 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:18.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:18.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:18.343 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:18.369 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:18.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:18.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:18.471 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:18.496 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:18.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:18.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:18.597 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:18.622 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:18.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:18.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:18.723 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:18.746 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:18.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:18.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:18.848 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:18.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:18.870 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:18.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:18.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:18.997 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:18.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:18.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:19.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:19.125 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:19.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:19.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:19.226 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:19.250 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:19.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:19.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:19.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:19.377 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:19.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:19.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:19.479 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:19.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:19.504 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:19.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:19.605 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:19.626 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:19.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:19.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:19.728 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:19.753 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:19.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:19.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:19.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:19 smithi136 ceph-mon[109861]: pgmap v2788: 83 pgs: 1 peering, 82 active+clean; 739 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 46 KiB/s rd, 134 KiB/s wr, 32 op/s; 215 KiB/s, 0 objects/s recovering 2024-08-17T18:09:19.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:19 smithi136 ceph-mon[115078]: pgmap v2788: 83 pgs: 1 peering, 82 active+clean; 739 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 46 KiB/s rd, 134 KiB/s wr, 32 op/s; 215 KiB/s, 0 objects/s recovering 2024-08-17T18:09:19.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:19.875 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:19.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:19.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:19.977 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:19.988 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:09:20.002 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:20.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:20.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:19 smithi165 ceph-mon[105517]: pgmap v2788: 83 pgs: 1 peering, 82 active+clean; 739 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 46 KiB/s rd, 134 KiB/s wr, 32 op/s; 215 KiB/s, 0 objects/s recovering 2024-08-17T18:09:20.101 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:09:20.102 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T18:09:20.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:20.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:20.129 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:20.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:20.230 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:20.253 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:20.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:20.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:20.355 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:20.379 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:20.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:20.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:20.481 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:20.503 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:20.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:20.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:20.601 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:09:20.604 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:20.625 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:20.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:20.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:20.727 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:20.748 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:20.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:20.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:20 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/30235966' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:09:20.850 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:20.872 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:20.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:20.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:20.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:20 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/30235966' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:09:20.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:20 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/30235966' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:09:20.974 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:20.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:20.996 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:20.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:21.097 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:21.118 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:21.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:21.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:21.219 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:21.241 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:21.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:21.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:21.343 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:21.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:21.364 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:21.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:21.396 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":2788,"stamp":"2024-08-17T18:09:18.297134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":756951,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":103,"num_read_kb":225,"num_write":123,"num_write_kb":693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":1087090,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1698,"ondisk_log_size":1698,"up":249,"acting":249,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":266,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":4693300,"kb_used_data":336316,"kb_used_omap":350,"kb_used_meta":4356513,"kb_avail":651522764,"statfs":{"total":671965249536,"available":667159310336,"internally_reserved":0,"allocated":344387584,"data_stored":306312562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":358888,"internal_metadata":4461069848},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":161010,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":225,"num_write":59,"num_write_kb":660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1079058,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.909373"},"pg_stats":[{"pgid":"3.1f","version":"70'4","reported_seq":209,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.594899+0000","last_change":"2024-08-17T18:07:48.546202+0000","last_active":"2024-08-17T18:09:15.594899+0000","last_peered":"2024-08-17T18:09:15.594899+0000","last_clean":"2024-08-17T18:09:15.594899+0000","last_became_active":"2024-08-17T18:07:48.531246+0000","last_became_peered":"2024-08-17T18:07:48.531246+0000","last_unstale":"2024-08-17T18:09:15.594899+0000","last_undegraded":"2024-08-17T18:09:15.594899+0000","last_fullsized":"2024-08-17T18:09:15.594899+0000","mapping_epoch":1501,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:30.944362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1536,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.577367+0000","last_change":"2024-08-17T18:08:37.068490+0000","last_active":"2024-08-17T18:09:14.577367+0000","last_peered":"2024-08-17T18:09:14.577367+0000","last_clean":"2024-08-17T18:09:14.577367+0000","last_became_active":"2024-08-17T18:08:37.068026+0000","last_became_peered":"2024-08-17T18:08:37.068026+0000","last_unstale":"2024-08-17T18:09:14.577367+0000","last_undegraded":"2024-08-17T18:09:14.577367+0000","last_fullsized":"2024-08-17T18:09:14.577367+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:34:27.467963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":2385,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.587009+0000","last_change":"2024-08-17T18:08:37.082356+0000","last_active":"2024-08-17T18:09:15.587009+0000","last_peered":"2024-08-17T18:09:15.587009+0000","last_clean":"2024-08-17T18:09:15.587009+0000","last_became_active":"2024-08-17T18:08:37.082046+0000","last_became_peered":"2024-08-17T18:08:37.082046+0000","last_unstale":"2024-08-17T18:09:15.587009+0000","last_undegraded":"2024-08-17T18:09:15.587009+0000","last_fullsized":"2024-08-17T18:09:15.587009+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:46:13.932917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2407,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.586774+0000","last_change":"2024-08-17T18:04:14.540774+0000","last_active":"2024-08-17T18:09:15.586774+0000","last_peered":"2024-08-17T18:09:15.586774+0000","last_clean":"2024-08-17T18:09:15.586774+0000","last_became_active":"2024-08-17T18:04:14.539778+0000","last_became_peered":"2024-08-17T18:04:14.539778+0000","last_unstale":"2024-08-17T18:09:15.586774+0000","last_undegraded":"2024-08-17T18:09:15.586774+0000","last_fullsized":"2024-08-17T18:09:15.586774+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:55:02.863360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":81,"reported_epoch":1529,"state":"active+clean","last_fresh":"2024-08-17T18:09:12.366338+0000","last_change":"2024-08-17T18:04:43.777705+0000","last_active":"2024-08-17T18:09:12.366338+0000","last_peered":"2024-08-17T18:09:12.366338+0000","last_clean":"2024-08-17T18:09:12.366338+0000","last_became_active":"2024-08-17T18:04:43.771283+0000","last_became_peered":"2024-08-17T18:04:43.771283+0000","last_unstale":"2024-08-17T18:09:12.366338+0000","last_undegraded":"2024-08-17T18:09:12.366338+0000","last_fullsized":"2024-08-17T18:09:12.366338+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:13.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1651,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.587486+0000","last_change":"2024-08-17T18:04:14.534963+0000","last_active":"2024-08-17T18:09:15.587486+0000","last_peered":"2024-08-17T18:09:15.587486+0000","last_clean":"2024-08-17T18:09:15.587486+0000","last_became_active":"2024-08-17T18:04:14.534634+0000","last_became_peered":"2024-08-17T18:04:14.534634+0000","last_unstale":"2024-08-17T18:09:15.587486+0000","last_undegraded":"2024-08-17T18:09:15.587486+0000","last_fullsized":"2024-08-17T18:09:15.587486+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:33:27.287860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":81,"reported_epoch":1529,"state":"active+clean","last_fresh":"2024-08-17T18:09:12.366208+0000","last_change":"2024-08-17T18:04:43.773534+0000","last_active":"2024-08-17T18:09:12.366208+0000","last_peered":"2024-08-17T18:09:12.366208+0000","last_clean":"2024-08-17T18:09:12.366208+0000","last_became_active":"2024-08-17T18:04:43.768507+0000","last_became_peered":"2024-08-17T18:04:43.768507+0000","last_unstale":"2024-08-17T18:09:12.366208+0000","last_undegraded":"2024-08-17T18:09:12.366208+0000","last_fullsized":"2024-08-17T18:09:12.366208+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:50:57.647351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1523,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.578778+0000","last_change":"2024-08-17T17:24:17.458168+0000","last_active":"2024-08-17T18:09:14.578778+0000","last_peered":"2024-08-17T18:09:14.578778+0000","last_clean":"2024-08-17T18:09:14.578778+0000","last_became_active":"2024-08-17T17:24:17.457847+0000","last_became_peered":"2024-08-17T17:24:17.457847+0000","last_unstale":"2024-08-17T18:09:14.578778+0000","last_undegraded":"2024-08-17T18:09:14.578778+0000","last_fullsized":"2024-08-17T18:09:14.578778+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:28.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":164,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.592256+0000","last_change":"2024-08-17T18:04:14.531445+0000","last_active":"2024-08-17T18:09:15.592256+0000","last_peered":"2024-08-17T18:09:15.592256+0000","last_clean":"2024-08-17T18:09:15.592256+0000","last_became_active":"2024-08-17T18:04:14.531099+0000","last_became_peered":"2024-08-17T18:04:14.531099+0000","last_unstale":"2024-08-17T18:09:15.592256+0000","last_undegraded":"2024-08-17T18:09:15.592256+0000","last_fullsized":"2024-08-17T18:09:15.592256+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:43.335123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":74,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.589398+0000","last_change":"2024-08-17T18:07:48.548262+0000","last_active":"2024-08-17T18:09:15.589398+0000","last_peered":"2024-08-17T18:09:15.589398+0000","last_clean":"2024-08-17T18:09:15.589398+0000","last_became_active":"2024-08-17T18:07:48.547846+0000","last_became_peered":"2024-08-17T18:07:48.547846+0000","last_unstale":"2024-08-17T18:09:15.589398+0000","last_undegraded":"2024-08-17T18:09:15.589398+0000","last_fullsized":"2024-08-17T18:09:15.589398+0000","mapping_epoch":1501,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:24:37.513219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1540,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.577812+0000","last_change":"2024-08-17T18:01:23.519890+0000","last_active":"2024-08-17T18:09:14.577812+0000","last_peered":"2024-08-17T18:09:14.577812+0000","last_clean":"2024-08-17T18:09:14.577812+0000","last_became_active":"2024-08-17T18:01:23.519526+0000","last_became_peered":"2024-08-17T18:01:23.519526+0000","last_unstale":"2024-08-17T18:09:14.577812+0000","last_undegraded":"2024-08-17T18:09:14.577812+0000","last_fullsized":"2024-08-17T18:09:14.577812+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:04:52.662528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2372,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.587143+0000","last_change":"2024-08-17T18:01:23.491836+0000","last_active":"2024-08-17T18:09:15.587143+0000","last_peered":"2024-08-17T18:09:15.587143+0000","last_clean":"2024-08-17T18:09:15.587143+0000","last_became_active":"2024-08-17T18:01:23.491209+0000","last_became_peered":"2024-08-17T18:01:23.491209+0000","last_unstale":"2024-08-17T18:09:15.587143+0000","last_undegraded":"2024-08-17T18:09:15.587143+0000","last_fullsized":"2024-08-17T18:09:15.587143+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:35:30.767005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2373,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.626053+0000","last_change":"2024-08-17T17:24:17.452025+0000","last_active":"2024-08-17T18:09:14.626053+0000","last_peered":"2024-08-17T18:09:14.626053+0000","last_clean":"2024-08-17T18:09:14.626053+0000","last_became_active":"2024-08-17T17:24:17.448029+0000","last_became_peered":"2024-08-17T17:24:17.448029+0000","last_unstale":"2024-08-17T18:09:14.626053+0000","last_undegraded":"2024-08-17T18:09:14.626053+0000","last_fullsized":"2024-08-17T18:09:14.626053+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:45.089346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":178,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.593127+0000","last_change":"2024-08-17T18:08:11.358013+0000","last_active":"2024-08-17T18:09:15.593127+0000","last_peered":"2024-08-17T18:09:15.593127+0000","last_clean":"2024-08-17T18:09:15.593127+0000","last_became_active":"2024-08-17T18:08:11.357666+0000","last_became_peered":"2024-08-17T18:08:11.357666+0000","last_unstale":"2024-08-17T18:09:15.593127+0000","last_undegraded":"2024-08-17T18:09:15.593127+0000","last_fullsized":"2024-08-17T18:09:15.593127+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:05:25.290465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2387,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.586489+0000","last_change":"2024-08-17T18:04:14.539660+0000","last_active":"2024-08-17T18:09:15.586489+0000","last_peered":"2024-08-17T18:09:15.586489+0000","last_clean":"2024-08-17T18:09:15.586489+0000","last_became_active":"2024-08-17T18:04:14.537852+0000","last_became_peered":"2024-08-17T18:04:14.537852+0000","last_unstale":"2024-08-17T18:09:15.586489+0000","last_undegraded":"2024-08-17T18:09:15.586489+0000","last_fullsized":"2024-08-17T18:09:15.586489+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:47:45.026997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1537,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.579708+0000","last_change":"2024-08-17T18:04:43.769892+0000","last_active":"2024-08-17T18:09:14.579708+0000","last_peered":"2024-08-17T18:09:14.579708+0000","last_clean":"2024-08-17T18:09:14.579708+0000","last_became_active":"2024-08-17T18:04:43.769702+0000","last_became_peered":"2024-08-17T18:04:43.769702+0000","last_unstale":"2024-08-17T18:09:14.579708+0000","last_undegraded":"2024-08-17T18:09:14.579708+0000","last_fullsized":"2024-08-17T18:09:14.579708+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:39.373904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2347,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.587846+0000","last_change":"2024-08-17T17:24:17.442338+0000","last_active":"2024-08-17T18:09:15.587846+0000","last_peered":"2024-08-17T18:09:15.587846+0000","last_clean":"2024-08-17T18:09:15.587846+0000","last_became_active":"2024-08-17T17:24:17.441948+0000","last_became_peered":"2024-08-17T17:24:17.441948+0000","last_unstale":"2024-08-17T18:09:15.587846+0000","last_undegraded":"2024-08-17T18:09:15.587846+0000","last_fullsized":"2024-08-17T18:09:15.587846+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:17.891492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":171,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.595031+0000","last_change":"2024-08-17T18:04:43.777736+0000","last_active":"2024-08-17T18:09:15.595031+0000","last_peered":"2024-08-17T18:09:15.595031+0000","last_clean":"2024-08-17T18:09:15.595031+0000","last_became_active":"2024-08-17T18:04:43.777438+0000","last_became_peered":"2024-08-17T18:04:43.777438+0000","last_unstale":"2024-08-17T18:09:15.595031+0000","last_undegraded":"2024-08-17T18:09:15.595031+0000","last_fullsized":"2024-08-17T18:09:15.595031+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:24:01.255927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2327,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.587359+0000","last_change":"2024-08-17T18:04:43.915398+0000","last_active":"2024-08-17T18:09:15.587359+0000","last_peered":"2024-08-17T18:09:15.587359+0000","last_clean":"2024-08-17T18:09:15.587359+0000","last_became_active":"2024-08-17T18:04:43.915212+0000","last_became_peered":"2024-08-17T18:04:43.915212+0000","last_unstale":"2024-08-17T18:09:15.587359+0000","last_undegraded":"2024-08-17T18:09:15.587359+0000","last_fullsized":"2024-08-17T18:09:15.587359+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:43:26.901830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":94,"reported_epoch":1529,"state":"active+clean","last_fresh":"2024-08-17T18:09:12.366325+0000","last_change":"2024-08-17T18:08:11.356876+0000","last_active":"2024-08-17T18:09:12.366325+0000","last_peered":"2024-08-17T18:09:12.366325+0000","last_clean":"2024-08-17T18:09:12.366325+0000","last_became_active":"2024-08-17T18:08:11.355662+0000","last_became_peered":"2024-08-17T18:08:11.355662+0000","last_unstale":"2024-08-17T18:09:12.366325+0000","last_undegraded":"2024-08-17T18:09:12.366325+0000","last_fullsized":"2024-08-17T18:09:12.366325+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:55:00.801251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":484,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.587249+0000","last_change":"2024-08-17T18:04:14.555729+0000","last_active":"2024-08-17T18:09:15.587249+0000","last_peered":"2024-08-17T18:09:15.587249+0000","last_clean":"2024-08-17T18:09:15.587249+0000","last_became_active":"2024-08-17T18:04:14.536678+0000","last_became_peered":"2024-08-17T18:04:14.536678+0000","last_unstale":"2024-08-17T18:09:15.587249+0000","last_undegraded":"2024-08-17T18:09:15.587249+0000","last_fullsized":"2024-08-17T18:09:15.587249+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:47:29.424778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2397,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.627200+0000","last_change":"2024-08-17T18:08:37.079571+0000","last_active":"2024-08-17T18:09:14.627200+0000","last_peered":"2024-08-17T18:09:14.627200+0000","last_clean":"2024-08-17T18:09:14.627200+0000","last_became_active":"2024-08-17T18:08:37.079224+0000","last_became_peered":"2024-08-17T18:08:37.079224+0000","last_unstale":"2024-08-17T18:09:14.627200+0000","last_undegraded":"2024-08-17T18:09:14.627200+0000","last_fullsized":"2024-08-17T18:09:14.627200+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:28:50.561072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1538,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.577848+0000","last_change":"2024-08-17T18:03:51.292044+0000","last_active":"2024-08-17T18:09:14.577848+0000","last_peered":"2024-08-17T18:09:14.577848+0000","last_clean":"2024-08-17T18:09:14.577848+0000","last_became_active":"2024-08-17T18:03:51.268271+0000","last_became_peered":"2024-08-17T18:03:51.268271+0000","last_unstale":"2024-08-17T18:09:14.577848+0000","last_undegraded":"2024-08-17T18:09:14.577848+0000","last_fullsized":"2024-08-17T18:09:14.577848+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:00.892410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4756,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1510,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.578861+0000","last_change":"2024-08-17T17:23:07.455919+0000","last_active":"2024-08-17T18:09:14.578861+0000","last_peered":"2024-08-17T18:09:14.578861+0000","last_clean":"2024-08-17T18:09:14.578861+0000","last_became_active":"2024-08-17T17:23:07.455284+0000","last_became_peered":"2024-08-17T17:23:07.455284+0000","last_unstale":"2024-08-17T18:09:14.578861+0000","last_undegraded":"2024-08-17T18:09:14.578861+0000","last_fullsized":"2024-08-17T18:09:14.578861+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:37:46.260775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"347'149","reported_seq":807,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.594456+0000","last_change":"2024-08-17T18:09:15.592511+0000","last_active":"2024-08-17T18:09:15.594456+0000","last_peered":"2024-08-17T18:09:15.594456+0000","last_clean":"2024-08-17T18:09:15.594456+0000","last_became_active":"2024-08-17T18:09:15.592099+0000","last_became_peered":"2024-08-17T18:09:15.592099+0000","last_unstale":"2024-08-17T18:09:15.594456+0000","last_undegraded":"2024-08-17T18:09:15.594456+0000","last_fullsized":"2024-08-17T18:09:15.594456+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:57.271286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:57.271286+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:54.841504+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1534,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.577485+0000","last_change":"2024-08-17T18:04:43.766724+0000","last_active":"2024-08-17T18:09:14.577485+0000","last_peered":"2024-08-17T18:09:14.577485+0000","last_clean":"2024-08-17T18:09:14.577485+0000","last_became_active":"2024-08-17T18:04:43.766428+0000","last_became_peered":"2024-08-17T18:04:43.766428+0000","last_unstale":"2024-08-17T18:09:14.577485+0000","last_undegraded":"2024-08-17T18:09:14.577485+0000","last_fullsized":"2024-08-17T18:09:14.577485+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:09:49.102940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1546,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.579789+0000","last_change":"2024-08-17T18:03:51.279963+0000","last_active":"2024-08-17T18:09:14.579789+0000","last_peered":"2024-08-17T18:09:14.579789+0000","last_clean":"2024-08-17T18:09:14.579789+0000","last_became_active":"2024-08-17T18:03:51.279506+0000","last_became_peered":"2024-08-17T18:03:51.279506+0000","last_unstale":"2024-08-17T18:09:14.579789+0000","last_undegraded":"2024-08-17T18:09:14.579789+0000","last_fullsized":"2024-08-17T18:09:14.579789+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:20:07.304042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":168,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.594558+0000","last_change":"2024-08-17T18:04:14.542589+0000","last_active":"2024-08-17T18:09:15.594558+0000","last_peered":"2024-08-17T18:09:15.594558+0000","last_clean":"2024-08-17T18:09:15.594558+0000","last_became_active":"2024-08-17T18:04:14.531540+0000","last_became_peered":"2024-08-17T18:04:14.531540+0000","last_unstale":"2024-08-17T18:09:15.594558+0000","last_undegraded":"2024-08-17T18:09:15.594558+0000","last_fullsized":"2024-08-17T18:09:15.594558+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:15:33.014782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"347'149","reported_seq":804,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.592395+0000","last_change":"2024-08-17T18:09:15.592395+0000","last_active":"2024-08-17T18:09:15.592395+0000","last_peered":"2024-08-17T18:09:15.592395+0000","last_clean":"2024-08-17T18:09:15.592395+0000","last_became_active":"2024-08-17T18:09:15.591773+0000","last_became_peered":"2024-08-17T18:09:15.591773+0000","last_unstale":"2024-08-17T18:09:15.592395+0000","last_undegraded":"2024-08-17T18:09:15.592395+0000","last_fullsized":"2024-08-17T18:09:15.592395+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:53.309517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:53.309517+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:50:57.110936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":60,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.586593+0000","last_change":"2024-08-17T18:08:11.349400+0000","last_active":"2024-08-17T18:09:15.586593+0000","last_peered":"2024-08-17T18:09:15.586593+0000","last_clean":"2024-08-17T18:09:15.586593+0000","last_became_active":"2024-08-17T18:08:11.348892+0000","last_became_peered":"2024-08-17T18:08:11.348892+0000","last_unstale":"2024-08-17T18:09:15.586593+0000","last_undegraded":"2024-08-17T18:09:15.586593+0000","last_fullsized":"2024-08-17T18:09:15.586593+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:52:54.819838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2355,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.626804+0000","last_change":"2024-08-17T17:23:07.428296+0000","last_active":"2024-08-17T18:09:14.626804+0000","last_peered":"2024-08-17T18:09:14.626804+0000","last_clean":"2024-08-17T18:09:14.626804+0000","last_became_active":"2024-08-17T17:23:07.428010+0000","last_became_peered":"2024-08-17T17:23:07.428010+0000","last_unstale":"2024-08-17T18:09:14.626804+0000","last_undegraded":"2024-08-17T18:09:14.626804+0000","last_fullsized":"2024-08-17T18:09:14.626804+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:45.459212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":2398,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.587379+0000","last_change":"2024-08-17T18:04:43.800633+0000","last_active":"2024-08-17T18:09:15.587379+0000","last_peered":"2024-08-17T18:09:15.587379+0000","last_clean":"2024-08-17T18:09:15.587379+0000","last_became_active":"2024-08-17T18:04:43.784612+0000","last_became_peered":"2024-08-17T18:04:43.784612+0000","last_unstale":"2024-08-17T18:09:15.587379+0000","last_undegraded":"2024-08-17T18:09:15.587379+0000","last_fullsized":"2024-08-17T18:09:15.587379+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:11.405039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"347'149","reported_seq":775,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.595649+0000","last_change":"2024-08-17T18:09:15.595301+0000","last_active":"2024-08-17T18:09:15.595649+0000","last_peered":"2024-08-17T18:09:15.595649+0000","last_clean":"2024-08-17T18:09:15.595649+0000","last_became_active":"2024-08-17T18:09:15.595011+0000","last_became_peered":"2024-08-17T18:09:15.595011+0000","last_unstale":"2024-08-17T18:09:15.595649+0000","last_undegraded":"2024-08-17T18:09:15.595649+0000","last_fullsized":"2024-08-17T18:09:15.595649+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:58.286040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:58.286040+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":171,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.592415+0000","last_change":"2024-08-17T18:04:43.805055+0000","last_active":"2024-08-17T18:09:15.592415+0000","last_peered":"2024-08-17T18:09:15.592415+0000","last_clean":"2024-08-17T18:09:15.592415+0000","last_became_active":"2024-08-17T18:04:43.804708+0000","last_became_peered":"2024-08-17T18:04:43.804708+0000","last_unstale":"2024-08-17T18:09:15.592415+0000","last_undegraded":"2024-08-17T18:09:15.592415+0000","last_fullsized":"2024-08-17T18:09:15.592415+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:47:22.253073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":164,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.594899+0000","last_change":"2024-08-17T18:04:14.529605+0000","last_active":"2024-08-17T18:09:15.594899+0000","last_peered":"2024-08-17T18:09:15.594899+0000","last_clean":"2024-08-17T18:09:15.594899+0000","last_became_active":"2024-08-17T18:04:14.529024+0000","last_became_peered":"2024-08-17T18:04:14.529024+0000","last_unstale":"2024-08-17T18:09:15.594899+0000","last_undegraded":"2024-08-17T18:09:15.594899+0000","last_fullsized":"2024-08-17T18:09:15.594899+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:38:00.485543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2409,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.581112+0000","last_change":"2024-08-17T18:04:43.773667+0000","last_active":"2024-08-17T18:09:14.581112+0000","last_peered":"2024-08-17T18:09:14.581112+0000","last_clean":"2024-08-17T18:09:14.581112+0000","last_became_active":"2024-08-17T18:04:43.773247+0000","last_became_peered":"2024-08-17T18:04:43.773247+0000","last_unstale":"2024-08-17T18:09:14.581112+0000","last_undegraded":"2024-08-17T18:09:14.581112+0000","last_fullsized":"2024-08-17T18:09:14.581112+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:35:48.964631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"347'149","reported_seq":775,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.593931+0000","last_change":"2024-08-17T18:09:15.593038+0000","last_active":"2024-08-17T18:09:15.593931+0000","last_peered":"2024-08-17T18:09:15.593931+0000","last_clean":"2024-08-17T18:09:15.593931+0000","last_became_active":"2024-08-17T18:09:15.592534+0000","last_became_peered":"2024-08-17T18:09:15.592534+0000","last_unstale":"2024-08-17T18:09:15.593931+0000","last_undegraded":"2024-08-17T18:09:15.593931+0000","last_fullsized":"2024-08-17T18:09:15.593931+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:51.289001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:51.289001+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":19,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":235,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.590630+0000","last_change":"2024-08-17T18:04:43.771589+0000","last_active":"2024-08-17T18:09:15.590630+0000","last_peered":"2024-08-17T18:09:15.590630+0000","last_clean":"2024-08-17T18:09:15.590630+0000","last_became_active":"2024-08-17T18:04:43.769601+0000","last_became_peered":"2024-08-17T18:04:43.769601+0000","last_unstale":"2024-08-17T18:09:15.590630+0000","last_undegraded":"2024-08-17T18:09:15.590630+0000","last_fullsized":"2024-08-17T18:09:15.590630+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:42:49.228921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2425,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.580848+0000","last_change":"2024-08-17T18:04:14.537136+0000","last_active":"2024-08-17T18:09:14.580848+0000","last_peered":"2024-08-17T18:09:14.580848+0000","last_clean":"2024-08-17T18:09:14.580848+0000","last_became_active":"2024-08-17T18:04:14.536815+0000","last_became_peered":"2024-08-17T18:04:14.536815+0000","last_unstale":"2024-08-17T18:09:14.580848+0000","last_undegraded":"2024-08-17T18:09:14.580848+0000","last_fullsized":"2024-08-17T18:09:14.580848+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:02.706133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":78,"reported_epoch":1529,"state":"active+clean","last_fresh":"2024-08-17T18:09:12.366343+0000","last_change":"2024-08-17T18:04:43.773480+0000","last_active":"2024-08-17T18:09:12.366343+0000","last_peered":"2024-08-17T18:09:12.366343+0000","last_clean":"2024-08-17T18:09:12.366343+0000","last_became_active":"2024-08-17T18:04:43.773160+0000","last_became_peered":"2024-08-17T18:04:43.773160+0000","last_unstale":"2024-08-17T18:09:12.366343+0000","last_undegraded":"2024-08-17T18:09:12.366343+0000","last_fullsized":"2024-08-17T18:09:12.366343+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:07:55.468132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"347'149","reported_seq":791,"reported_epoch":1528,"state":"active+clean","last_fresh":"2024-08-17T18:09:10.546499+0000","last_change":"2024-08-17T18:08:00.317000+0000","last_active":"2024-08-17T18:09:10.546499+0000","last_peered":"2024-08-17T18:09:10.546499+0000","last_clean":"2024-08-17T18:09:10.546499+0000","last_became_active":"2024-08-17T18:06:57.361016+0000","last_became_peered":"2024-08-17T18:06:57.361016+0000","last_unstale":"2024-08-17T18:09:10.546499+0000","last_undegraded":"2024-08-17T18:09:10.546499+0000","last_fullsized":"2024-08-17T18:09:10.546499+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:00.316884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:00.316884+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:29:06.444140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":235,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.580749+0000","last_change":"2024-08-17T18:02:08.224928+0000","last_active":"2024-08-17T18:09:14.580749+0000","last_peered":"2024-08-17T18:09:14.580749+0000","last_clean":"2024-08-17T18:09:14.580749+0000","last_became_active":"2024-08-17T18:02:08.224604+0000","last_became_peered":"2024-08-17T18:02:08.224604+0000","last_unstale":"2024-08-17T18:09:14.580749+0000","last_undegraded":"2024-08-17T18:09:14.580749+0000","last_fullsized":"2024-08-17T18:09:14.580749+0000","mapping_epoch":1421,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1422,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:55.716971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1532,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.579007+0000","last_change":"2024-08-17T18:04:14.526890+0000","last_active":"2024-08-17T18:09:14.579007+0000","last_peered":"2024-08-17T18:09:14.579007+0000","last_clean":"2024-08-17T18:09:14.579007+0000","last_became_active":"2024-08-17T18:04:14.526564+0000","last_became_peered":"2024-08-17T18:04:14.526564+0000","last_unstale":"2024-08-17T18:09:14.579007+0000","last_undegraded":"2024-08-17T18:09:14.579007+0000","last_fullsized":"2024-08-17T18:09:14.579007+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:10:13.229827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":82,"reported_epoch":1529,"state":"active+clean","last_fresh":"2024-08-17T18:09:12.366232+0000","last_change":"2024-08-17T18:04:43.791749+0000","last_active":"2024-08-17T18:09:12.366232+0000","last_peered":"2024-08-17T18:09:12.366232+0000","last_clean":"2024-08-17T18:09:12.366232+0000","last_became_active":"2024-08-17T18:04:43.774474+0000","last_became_peered":"2024-08-17T18:04:43.774474+0000","last_unstale":"2024-08-17T18:09:12.366232+0000","last_undegraded":"2024-08-17T18:09:12.366232+0000","last_fullsized":"2024-08-17T18:09:12.366232+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:19:08.097560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"347'149","reported_seq":812,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.596566+0000","last_change":"2024-08-17T18:09:15.596502+0000","last_active":"2024-08-17T18:09:15.596566+0000","last_peered":"2024-08-17T18:09:15.596566+0000","last_clean":"2024-08-17T18:09:15.596566+0000","last_became_active":"2024-08-17T18:09:15.595460+0000","last_became_peered":"2024-08-17T18:09:15.595460+0000","last_unstale":"2024-08-17T18:09:15.596566+0000","last_undegraded":"2024-08-17T18:09:15.596566+0000","last_fullsized":"2024-08-17T18:09:15.596566+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:02.363531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:02.363531+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:22:42.587876+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2411,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.626446+0000","last_change":"2024-08-17T18:04:43.781133+0000","last_active":"2024-08-17T18:09:14.626446+0000","last_peered":"2024-08-17T18:09:14.626446+0000","last_clean":"2024-08-17T18:09:14.626446+0000","last_became_active":"2024-08-17T18:04:43.780787+0000","last_became_peered":"2024-08-17T18:04:43.780787+0000","last_unstale":"2024-08-17T18:09:14.626446+0000","last_undegraded":"2024-08-17T18:09:14.626446+0000","last_fullsized":"2024-08-17T18:09:14.626446+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:38:34.901162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2367,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.625076+0000","last_change":"2024-08-17T18:01:23.493896+0000","last_active":"2024-08-17T18:09:14.625076+0000","last_peered":"2024-08-17T18:09:14.625076+0000","last_clean":"2024-08-17T18:09:14.625076+0000","last_became_active":"2024-08-17T18:01:23.493551+0000","last_became_peered":"2024-08-17T18:01:23.493551+0000","last_unstale":"2024-08-17T18:09:14.625076+0000","last_undegraded":"2024-08-17T18:09:14.625076+0000","last_fullsized":"2024-08-17T18:09:14.625076+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:18:47.648483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2347,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.589216+0000","last_change":"2024-08-17T18:04:14.543532+0000","last_active":"2024-08-17T18:09:15.589216+0000","last_peered":"2024-08-17T18:09:15.589216+0000","last_clean":"2024-08-17T18:09:15.589216+0000","last_became_active":"2024-08-17T18:04:14.543131+0000","last_became_peered":"2024-08-17T18:04:14.543131+0000","last_unstale":"2024-08-17T18:09:15.589216+0000","last_undegraded":"2024-08-17T18:09:15.589216+0000","last_fullsized":"2024-08-17T18:09:15.589216+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:51:44.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"347'149","reported_seq":644,"reported_epoch":1532,"state":"peering","last_fresh":"2024-08-17T18:09:14.581757+0000","last_change":"2024-08-17T18:09:14.581757+0000","last_active":"2024-08-17T18:09:14.581757+0000","last_peered":"2024-08-17T18:09:14.496598+0000","last_clean":"2024-08-17T17:16:08.158050+0000","last_became_active":"2024-08-17T18:09:14.496207+0000","last_became_peered":"2024-08-17T18:09:14.496207+0000","last_unstale":"2024-08-17T18:09:14.581757+0000","last_undegraded":"2024-08-17T18:09:14.581757+0000","last_fullsized":"2024-08-17T18:09:14.581757+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:01.352351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:01.352351+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":214680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":178,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.592486+0000","last_change":"2024-08-17T18:07:48.539239+0000","last_active":"2024-08-17T18:09:15.592486+0000","last_peered":"2024-08-17T18:09:15.592486+0000","last_clean":"2024-08-17T18:09:15.592486+0000","last_became_active":"2024-08-17T18:07:48.538888+0000","last_became_peered":"2024-08-17T18:07:48.538888+0000","last_unstale":"2024-08-17T18:09:15.592486+0000","last_undegraded":"2024-08-17T18:09:15.592486+0000","last_fullsized":"2024-08-17T18:09:15.592486+0000","mapping_epoch":1501,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:35:23.855790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":178,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.593902+0000","last_change":"2024-08-17T18:08:11.440676+0000","last_active":"2024-08-17T18:09:15.593902+0000","last_peered":"2024-08-17T18:09:15.593902+0000","last_clean":"2024-08-17T18:09:15.593902+0000","last_became_active":"2024-08-17T18:08:11.440353+0000","last_became_peered":"2024-08-17T18:08:11.440353+0000","last_unstale":"2024-08-17T18:09:15.593902+0000","last_undegraded":"2024-08-17T18:09:15.593902+0000","last_fullsized":"2024-08-17T18:09:15.593902+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:19:22.125277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1111,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.625227+0000","last_change":"2024-08-17T18:04:14.537308+0000","last_active":"2024-08-17T18:09:14.625227+0000","last_peered":"2024-08-17T18:09:14.625227+0000","last_clean":"2024-08-17T18:09:14.625227+0000","last_became_active":"2024-08-17T18:04:14.535521+0000","last_became_peered":"2024-08-17T18:04:14.535521+0000","last_unstale":"2024-08-17T18:09:14.625227+0000","last_undegraded":"2024-08-17T18:09:14.625227+0000","last_fullsized":"2024-08-17T18:09:14.625227+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:40:00.384936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":28,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.578242+0000","last_change":"2024-08-17T18:08:37.071912+0000","last_active":"2024-08-17T18:09:14.578242+0000","last_peered":"2024-08-17T18:09:14.578242+0000","last_clean":"2024-08-17T18:09:14.578242+0000","last_became_active":"2024-08-17T18:08:37.071648+0000","last_became_peered":"2024-08-17T18:08:37.071648+0000","last_unstale":"2024-08-17T18:09:14.578242+0000","last_undegraded":"2024-08-17T18:09:14.578242+0000","last_fullsized":"2024-08-17T18:09:14.578242+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:58:57.279808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":175,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.589971+0000","last_change":"2024-08-17T18:04:43.771309+0000","last_active":"2024-08-17T18:09:15.589971+0000","last_peered":"2024-08-17T18:09:15.589971+0000","last_clean":"2024-08-17T18:09:15.589971+0000","last_became_active":"2024-08-17T18:04:43.770922+0000","last_became_peered":"2024-08-17T18:04:43.770922+0000","last_unstale":"2024-08-17T18:09:15.589971+0000","last_undegraded":"2024-08-17T18:09:15.589971+0000","last_fullsized":"2024-08-17T18:09:15.589971+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:10.173251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":82,"reported_epoch":1529,"state":"active+clean","last_fresh":"2024-08-17T18:09:12.366148+0000","last_change":"2024-08-17T18:04:43.786228+0000","last_active":"2024-08-17T18:09:12.366148+0000","last_peered":"2024-08-17T18:09:12.366148+0000","last_clean":"2024-08-17T18:09:12.366148+0000","last_became_active":"2024-08-17T18:04:43.770976+0000","last_became_peered":"2024-08-17T18:04:43.770976+0000","last_unstale":"2024-08-17T18:09:12.366148+0000","last_undegraded":"2024-08-17T18:09:12.366148+0000","last_fullsized":"2024-08-17T18:09:12.366148+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:20:10.131137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":803,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.594317+0000","last_change":"2024-08-17T18:07:54.303783+0000","last_active":"2024-08-17T18:09:15.594317+0000","last_peered":"2024-08-17T18:09:15.594317+0000","last_clean":"2024-08-17T18:09:15.594317+0000","last_became_active":"2024-08-17T18:06:57.358131+0000","last_became_peered":"2024-08-17T18:06:57.358131+0000","last_unstale":"2024-08-17T18:09:15.594317+0000","last_undegraded":"2024-08-17T18:09:15.594317+0000","last_fullsized":"2024-08-17T18:09:15.594317+0000","mapping_epoch":1460,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:54.303708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:54.303708+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:43:08.118481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"347'149","reported_seq":792,"reported_epoch":1528,"state":"active+clean","last_fresh":"2024-08-17T18:09:10.546596+0000","last_change":"2024-08-17T18:07:55.256115+0000","last_active":"2024-08-17T18:09:10.546596+0000","last_peered":"2024-08-17T18:09:10.546596+0000","last_clean":"2024-08-17T18:09:10.546596+0000","last_became_active":"2024-08-17T18:06:57.361375+0000","last_became_peered":"2024-08-17T18:06:57.361375+0000","last_unstale":"2024-08-17T18:09:10.546596+0000","last_undegraded":"2024-08-17T18:09:10.546596+0000","last_fullsized":"2024-08-17T18:09:10.546596+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:55.255983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:55.255983+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:58:45.663728+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2354,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.626375+0000","last_change":"2024-08-17T17:24:17.455710+0000","last_active":"2024-08-17T18:09:14.626375+0000","last_peered":"2024-08-17T18:09:14.626375+0000","last_clean":"2024-08-17T18:09:14.626375+0000","last_became_active":"2024-08-17T17:24:17.452989+0000","last_became_peered":"2024-08-17T17:24:17.452989+0000","last_unstale":"2024-08-17T18:09:14.626375+0000","last_undegraded":"2024-08-17T18:09:14.626375+0000","last_fullsized":"2024-08-17T18:09:14.626375+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:58:07.894968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":60,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.586316+0000","last_change":"2024-08-17T18:08:11.347750+0000","last_active":"2024-08-17T18:09:15.586316+0000","last_peered":"2024-08-17T18:09:15.586316+0000","last_clean":"2024-08-17T18:09:15.586316+0000","last_became_active":"2024-08-17T18:08:11.346796+0000","last_became_peered":"2024-08-17T18:08:11.346796+0000","last_unstale":"2024-08-17T18:09:15.586316+0000","last_undegraded":"2024-08-17T18:09:15.586316+0000","last_fullsized":"2024-08-17T18:09:15.586316+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:24.470900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1529,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.577681+0000","last_change":"2024-08-17T18:08:37.068370+0000","last_active":"2024-08-17T18:09:14.577681+0000","last_peered":"2024-08-17T18:09:14.577681+0000","last_clean":"2024-08-17T18:09:14.577681+0000","last_became_active":"2024-08-17T18:08:37.068027+0000","last_became_peered":"2024-08-17T18:08:37.068027+0000","last_unstale":"2024-08-17T18:09:14.577681+0000","last_undegraded":"2024-08-17T18:09:14.577681+0000","last_fullsized":"2024-08-17T18:09:14.577681+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:02:12.661660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1537,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.577576+0000","last_change":"2024-08-17T18:03:51.270293+0000","last_active":"2024-08-17T18:09:14.577576+0000","last_peered":"2024-08-17T18:09:14.577576+0000","last_clean":"2024-08-17T18:09:14.577576+0000","last_became_active":"2024-08-17T18:03:51.269970+0000","last_became_peered":"2024-08-17T18:03:51.269970+0000","last_unstale":"2024-08-17T18:09:14.577576+0000","last_undegraded":"2024-08-17T18:09:14.577576+0000","last_fullsized":"2024-08-17T18:09:14.577576+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:02:30.016517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"347'149","reported_seq":792,"reported_epoch":1528,"state":"active+clean","last_fresh":"2024-08-17T18:09:10.546420+0000","last_change":"2024-08-17T18:07:59.307478+0000","last_active":"2024-08-17T18:09:10.546420+0000","last_peered":"2024-08-17T18:09:10.546420+0000","last_clean":"2024-08-17T18:09:10.546420+0000","last_became_active":"2024-08-17T18:06:57.361299+0000","last_became_peered":"2024-08-17T18:06:57.361299+0000","last_unstale":"2024-08-17T18:09:10.546420+0000","last_undegraded":"2024-08-17T18:09:10.546420+0000","last_fullsized":"2024-08-17T18:09:10.546420+0000","mapping_epoch":1491,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:59.307356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:59.307356+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:21:06.088723+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1541,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.578567+0000","last_change":"2024-08-17T18:01:23.497110+0000","last_active":"2024-08-17T18:09:14.578567+0000","last_peered":"2024-08-17T18:09:14.578567+0000","last_clean":"2024-08-17T18:09:14.578567+0000","last_became_active":"2024-08-17T18:01:23.496800+0000","last_became_peered":"2024-08-17T18:01:23.496800+0000","last_unstale":"2024-08-17T18:09:14.578567+0000","last_undegraded":"2024-08-17T18:09:14.578567+0000","last_fullsized":"2024-08-17T18:09:14.578567+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:32:29.892739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1510,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.580467+0000","last_change":"2024-08-17T17:23:07.401624+0000","last_active":"2024-08-17T18:09:14.580467+0000","last_peered":"2024-08-17T18:09:14.580467+0000","last_clean":"2024-08-17T18:09:14.580467+0000","last_became_active":"2024-08-17T17:23:07.400222+0000","last_became_peered":"2024-08-17T17:23:07.400222+0000","last_unstale":"2024-08-17T18:09:14.580467+0000","last_undegraded":"2024-08-17T18:09:14.580467+0000","last_fullsized":"2024-08-17T18:09:14.580467+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:07.069989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"347'149","reported_seq":806,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.593966+0000","last_change":"2024-08-17T18:09:15.592665+0000","last_active":"2024-08-17T18:09:15.593966+0000","last_peered":"2024-08-17T18:09:15.593966+0000","last_clean":"2024-08-17T18:09:15.593966+0000","last_became_active":"2024-08-17T18:09:15.590435+0000","last_became_peered":"2024-08-17T18:09:15.590435+0000","last_unstale":"2024-08-17T18:09:15.593966+0000","last_undegraded":"2024-08-17T18:09:15.593966+0000","last_fullsized":"2024-08-17T18:09:15.593966+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:52.271487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:52.271487+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:55:12.849191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1511,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.577598+0000","last_change":"2024-08-17T17:24:17.470809+0000","last_active":"2024-08-17T18:09:14.577598+0000","last_peered":"2024-08-17T18:09:14.577598+0000","last_clean":"2024-08-17T18:09:14.577598+0000","last_became_active":"2024-08-17T17:24:17.469753+0000","last_became_peered":"2024-08-17T17:24:17.469753+0000","last_unstale":"2024-08-17T18:09:14.577598+0000","last_undegraded":"2024-08-17T18:09:14.577598+0000","last_fullsized":"2024-08-17T18:09:14.577598+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:00.049425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":2364,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.586543+0000","last_change":"2024-08-17T18:04:14.552806+0000","last_active":"2024-08-17T18:09:15.586543+0000","last_peered":"2024-08-17T18:09:15.586543+0000","last_clean":"2024-08-17T18:09:15.586543+0000","last_became_active":"2024-08-17T18:04:14.535546+0000","last_became_peered":"2024-08-17T18:04:14.535546+0000","last_unstale":"2024-08-17T18:09:15.586543+0000","last_undegraded":"2024-08-17T18:09:15.586543+0000","last_fullsized":"2024-08-17T18:09:15.586543+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:44:45.702272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":164,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.592314+0000","last_change":"2024-08-17T18:04:14.539500+0000","last_active":"2024-08-17T18:09:15.592314+0000","last_peered":"2024-08-17T18:09:15.592314+0000","last_clean":"2024-08-17T18:09:15.592314+0000","last_became_active":"2024-08-17T18:04:14.538738+0000","last_became_peered":"2024-08-17T18:04:14.538738+0000","last_unstale":"2024-08-17T18:09:15.592314+0000","last_undegraded":"2024-08-17T18:09:15.592314+0000","last_fullsized":"2024-08-17T18:09:15.592314+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:31.439947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":78,"reported_epoch":1530,"state":"active+clean","last_fresh":"2024-08-17T18:09:13.392580+0000","last_change":"2024-08-17T18:04:43.771107+0000","last_active":"2024-08-17T18:09:13.392580+0000","last_peered":"2024-08-17T18:09:13.392580+0000","last_clean":"2024-08-17T18:09:13.392580+0000","last_became_active":"2024-08-17T18:04:43.770799+0000","last_became_peered":"2024-08-17T18:04:43.770799+0000","last_unstale":"2024-08-17T18:09:13.392580+0000","last_undegraded":"2024-08-17T18:09:13.392580+0000","last_fullsized":"2024-08-17T18:09:13.392580+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:09:04.350798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.579753+0000","last_change":"2024-08-17T18:08:37.080554+0000","last_active":"2024-08-17T18:09:14.579753+0000","last_peered":"2024-08-17T18:09:14.579753+0000","last_clean":"2024-08-17T18:09:14.579753+0000","last_became_active":"2024-08-17T18:08:37.079920+0000","last_became_peered":"2024-08-17T18:08:37.079920+0000","last_unstale":"2024-08-17T18:09:14.579753+0000","last_undegraded":"2024-08-17T18:09:14.579753+0000","last_fullsized":"2024-08-17T18:09:14.579753+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:08:25.214786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":82,"reported_epoch":1530,"state":"active+clean","last_fresh":"2024-08-17T18:09:13.393443+0000","last_change":"2024-08-17T18:04:43.781111+0000","last_active":"2024-08-17T18:09:13.393443+0000","last_peered":"2024-08-17T18:09:13.393443+0000","last_clean":"2024-08-17T18:09:13.393443+0000","last_became_active":"2024-08-17T18:04:43.771839+0000","last_became_peered":"2024-08-17T18:04:43.771839+0000","last_unstale":"2024-08-17T18:09:13.393443+0000","last_undegraded":"2024-08-17T18:09:13.393443+0000","last_fullsized":"2024-08-17T18:09:13.393443+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:52:45.973049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1545,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.577355+0000","last_change":"2024-08-17T18:08:11.449492+0000","last_active":"2024-08-17T18:09:14.577355+0000","last_peered":"2024-08-17T18:09:14.577355+0000","last_clean":"2024-08-17T18:09:14.577355+0000","last_became_active":"2024-08-17T18:08:11.424764+0000","last_became_peered":"2024-08-17T18:08:11.424764+0000","last_unstale":"2024-08-17T18:09:14.577355+0000","last_undegraded":"2024-08-17T18:09:14.577355+0000","last_fullsized":"2024-08-17T18:09:14.577355+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:53:37.552880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":183,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.588811+0000","last_change":"2024-08-17T18:04:14.551061+0000","last_active":"2024-08-17T18:09:15.588811+0000","last_peered":"2024-08-17T18:09:15.588811+0000","last_clean":"2024-08-17T18:09:15.588811+0000","last_became_active":"2024-08-17T18:04:14.543296+0000","last_became_peered":"2024-08-17T18:04:14.543296+0000","last_unstale":"2024-08-17T18:09:15.588811+0000","last_undegraded":"2024-08-17T18:09:15.588811+0000","last_fullsized":"2024-08-17T18:09:15.588811+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:39:45.294265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":96,"reported_epoch":1529,"state":"active+clean","last_fresh":"2024-08-17T18:09:12.365986+0000","last_change":"2024-08-17T18:08:37.068503+0000","last_active":"2024-08-17T18:09:12.365986+0000","last_peered":"2024-08-17T18:09:12.365986+0000","last_clean":"2024-08-17T18:09:12.365986+0000","last_became_active":"2024-08-17T18:08:37.068110+0000","last_became_peered":"2024-08-17T18:08:37.068110+0000","last_unstale":"2024-08-17T18:09:12.365986+0000","last_undegraded":"2024-08-17T18:09:12.365986+0000","last_fullsized":"2024-08-17T18:09:12.365986+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:12:30.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":126,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.577698+0000","last_change":"2024-08-17T18:04:14.547798+0000","last_active":"2024-08-17T18:09:14.577698+0000","last_peered":"2024-08-17T18:09:14.577698+0000","last_clean":"2024-08-17T18:09:14.577698+0000","last_became_active":"2024-08-17T18:04:14.527496+0000","last_became_peered":"2024-08-17T18:04:14.527496+0000","last_unstale":"2024-08-17T18:09:14.577698+0000","last_undegraded":"2024-08-17T18:09:14.577698+0000","last_fullsized":"2024-08-17T18:09:14.577698+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:25:54.338846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1592,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.578243+0000","last_change":"2024-08-17T18:04:43.813536+0000","last_active":"2024-08-17T18:09:14.578243+0000","last_peered":"2024-08-17T18:09:14.578243+0000","last_clean":"2024-08-17T18:09:14.578243+0000","last_became_active":"2024-08-17T18:04:43.799823+0000","last_became_peered":"2024-08-17T18:04:43.799823+0000","last_unstale":"2024-08-17T18:09:14.578243+0000","last_undegraded":"2024-08-17T18:09:14.578243+0000","last_fullsized":"2024-08-17T18:09:14.578243+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:04:10.837113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":60,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.586926+0000","last_change":"2024-08-17T18:08:11.345061+0000","last_active":"2024-08-17T18:09:15.586926+0000","last_peered":"2024-08-17T18:09:15.586926+0000","last_clean":"2024-08-17T18:09:15.586926+0000","last_became_active":"2024-08-17T18:08:11.344755+0000","last_became_peered":"2024-08-17T18:08:11.344755+0000","last_unstale":"2024-08-17T18:09:15.586926+0000","last_undegraded":"2024-08-17T18:09:15.586926+0000","last_fullsized":"2024-08-17T18:09:15.586926+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:23:32.161453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":250,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.577486+0000","last_change":"2024-08-17T18:04:14.527380+0000","last_active":"2024-08-17T18:09:14.577486+0000","last_peered":"2024-08-17T18:09:14.577486+0000","last_clean":"2024-08-17T18:09:14.577486+0000","last_became_active":"2024-08-17T18:04:14.527076+0000","last_became_peered":"2024-08-17T18:04:14.527076+0000","last_unstale":"2024-08-17T18:09:14.577486+0000","last_undegraded":"2024-08-17T18:09:14.577486+0000","last_fullsized":"2024-08-17T18:09:14.577486+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:36:35.583709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":164,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.593230+0000","last_change":"2024-08-17T18:04:14.541814+0000","last_active":"2024-08-17T18:09:15.593230+0000","last_peered":"2024-08-17T18:09:15.593230+0000","last_clean":"2024-08-17T18:09:15.593230+0000","last_became_active":"2024-08-17T18:04:14.541403+0000","last_became_peered":"2024-08-17T18:04:14.541403+0000","last_unstale":"2024-08-17T18:09:15.593230+0000","last_undegraded":"2024-08-17T18:09:15.593230+0000","last_fullsized":"2024-08-17T18:09:15.593230+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:07:45.294924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":178,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.592561+0000","last_change":"2024-08-17T18:08:11.351307+0000","last_active":"2024-08-17T18:09:15.592561+0000","last_peered":"2024-08-17T18:09:15.592561+0000","last_clean":"2024-08-17T18:09:15.592561+0000","last_became_active":"2024-08-17T18:08:11.350926+0000","last_became_peered":"2024-08-17T18:08:11.350926+0000","last_unstale":"2024-08-17T18:09:15.592561+0000","last_undegraded":"2024-08-17T18:09:15.592561+0000","last_fullsized":"2024-08-17T18:09:15.592561+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:59:57.477492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":77,"reported_epoch":1529,"state":"active+clean","last_fresh":"2024-08-17T18:09:12.366487+0000","last_change":"2024-08-17T18:04:43.771720+0000","last_active":"2024-08-17T18:09:12.366487+0000","last_peered":"2024-08-17T18:09:12.366487+0000","last_clean":"2024-08-17T18:09:12.366487+0000","last_became_active":"2024-08-17T18:04:43.771137+0000","last_became_peered":"2024-08-17T18:04:43.771137+0000","last_unstale":"2024-08-17T18:09:12.366487+0000","last_undegraded":"2024-08-17T18:09:12.366487+0000","last_fullsized":"2024-08-17T18:09:12.366487+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:57.647909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":115,"reported_epoch":1532,"state":"active+clean","last_fresh":"2024-08-17T18:09:14.577822+0000","last_change":"2024-08-17T18:03:51.283872+0000","last_active":"2024-08-17T18:09:14.577822+0000","last_peered":"2024-08-17T18:09:14.577822+0000","last_clean":"2024-08-17T18:09:14.577822+0000","last_became_active":"2024-08-17T18:03:51.267197+0000","last_became_peered":"2024-08-17T18:03:51.267197+0000","last_unstale":"2024-08-17T18:09:14.577822+0000","last_undegraded":"2024-08-17T18:09:14.577822+0000","last_fullsized":"2024-08-17T18:09:14.577822+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:18:59.157786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2354,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.588174+0000","last_change":"2024-08-17T18:03:51.277491+0000","last_active":"2024-08-17T18:09:15.588174+0000","last_peered":"2024-08-17T18:09:15.588174+0000","last_clean":"2024-08-17T18:09:15.588174+0000","last_became_active":"2024-08-17T18:03:51.277178+0000","last_became_peered":"2024-08-17T18:03:51.277178+0000","last_unstale":"2024-08-17T18:09:15.588174+0000","last_undegraded":"2024-08-17T18:09:15.588174+0000","last_fullsized":"2024-08-17T18:09:15.588174+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:21:35.319085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":751378,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":103,"num_read_kb":225,"num_write":59,"num_write_kb":660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1079058,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":7994,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234719,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496357,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":769828,"kb_used_data":12760,"kb_used_omap":1,"kb_used_meta":757054,"kb_avail":92975324,"statfs":{"total":95995035648,"available":95206731776,"internally_reserved":0,"allocated":13066240,"data_stored":7619449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":775223751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565598,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":944608,"kb_used_data":158604,"kb_used_omap":72,"kb_used_meta":785911,"kb_avail":92800544,"statfs":{"total":95995035648,"available":95027757056,"internally_reserved":0,"allocated":162410496,"data_stored":156971830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":804773358},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":1399,"seq":6008659247200,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":279468,"kb_used_data":12696,"kb_used_omap":131,"kb_used_meta":266620,"kb_avail":93465684,"statfs":{"total":95995035648,"available":95708860416,"internally_reserved":0,"allocated":13000704,"data_stored":7554962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134291,"internal_metadata":273019757},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743186988,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":738832,"kb_used_data":84864,"kb_used_omap":66,"kb_used_meta":653885,"kb_avail":93006320,"statfs":{"total":95995035648,"available":95238471680,"internally_reserved":0,"allocated":86900736,"data_stored":81465756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":669578923},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448627,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":456532,"kb_used_data":23428,"kb_used_omap":75,"kb_used_meta":433012,"kb_avail":93288620,"statfs":{"total":95995035648,"available":95527546880,"internally_reserved":0,"allocated":23990272,"data_stored":18551849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":443404479},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757991,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":832948,"kb_used_data":31524,"kb_used_omap":1,"kb_used_meta":801406,"kb_avail":92912204,"statfs":{"total":95995035648,"available":95142096896,"internally_reserved":0,"allocated":32280576,"data_stored":26856924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":820640198},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019630,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":671084,"kb_used_data":12440,"kb_used_omap":1,"kb_used_meta":658622,"kb_avail":93074068,"statfs":{"total":95995035648,"available":95307845632,"internally_reserved":0,"allocated":12738560,"data_stored":7291792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":674429382},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T18:09:21.400 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T18:09:21.466 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:21.500 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:21.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:21.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:21.602 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:21.623 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:21.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:21.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:21.664 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:09:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:21 smithi136 ceph-mon[109861]: from='client.45980 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:09:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:21 smithi136 ceph-mon[109861]: pgmap v2789: 83 pgs: 1 peering, 82 active+clean; 739 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 37 KiB/s rd, 110 KiB/s wr, 26 op/s; 218 KiB/s, 0 objects/s recovering 2024-08-17T18:09:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:21 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:09:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:21 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/30235966' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:09:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:21 smithi136 ceph-mon[109861]: osdmap e1534: 8 total, 8 up, 7 in 2024-08-17T18:09:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:09:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:09:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:21 smithi136 ceph-mon[115078]: from='client.45980 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:09:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:21 smithi136 ceph-mon[115078]: pgmap v2789: 83 pgs: 1 peering, 82 active+clean; 739 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 37 KiB/s rd, 110 KiB/s wr, 26 op/s; 218 KiB/s, 0 objects/s recovering 2024-08-17T18:09:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:21 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:09:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:21 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/30235966' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:09:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:21 smithi136 ceph-mon[115078]: osdmap e1534: 8 total, 8 up, 7 in 2024-08-17T18:09:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:09:21.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:09:21.725 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:21.746 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:21.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:21.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:21.848 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:21.870 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:21.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:21.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:21.972 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:21.995 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:21.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:21.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:21 smithi165 ceph-mon[105517]: from='client.45980 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:09:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:21 smithi165 ceph-mon[105517]: pgmap v2789: 83 pgs: 1 peering, 82 active+clean; 739 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 37 KiB/s rd, 110 KiB/s wr, 26 op/s; 218 KiB/s, 0 objects/s recovering 2024-08-17T18:09:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:21 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:09:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:21 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/30235966' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:09:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:21 smithi165 ceph-mon[105517]: osdmap e1534: 8 total, 8 up, 7 in 2024-08-17T18:09:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:09:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:09:22.097 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:22.119 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:22.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:22.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:22.159 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:09:22.221 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:22.242 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:22.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:22.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:22.343 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:22.365 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:22.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:22.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:22.466 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:22.492 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:22.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:22.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:22.594 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:22.619 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:22.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:22.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:22.721 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:22.746 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:22.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:22.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:22.848 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:22.871 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:22.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:22.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:22.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:22.999 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:22.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:23.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:23.101 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:23.126 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:23.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:23.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:23.228 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:23.254 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:23.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:23.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:23.356 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:23.381 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:23.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:23.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:23.482 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:23.507 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:23.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:23.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:23.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:23.632 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:23.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:23.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:23.724 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:23 smithi136 ceph-mon[109861]: pgmap v2791: 83 pgs: 83 active+clean; 687 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 19 KiB/s rd, 57 KiB/s wr, 13 op/s; 143 KiB/s, 0 objects/s recovering 2024-08-17T18:09:23.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:23 smithi136 ceph-mon[115078]: pgmap v2791: 83 pgs: 83 active+clean; 687 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 19 KiB/s rd, 57 KiB/s wr, 13 op/s; 143 KiB/s, 0 objects/s recovering 2024-08-17T18:09:23.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:23.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:23.757 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:23.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:23.858 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:23.881 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:23.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:23.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:23.983 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:24.005 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:24.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:24.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:23 smithi165 ceph-mon[105517]: pgmap v2791: 83 pgs: 83 active+clean; 687 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 19 KiB/s rd, 57 KiB/s wr, 13 op/s; 143 KiB/s, 0 objects/s recovering 2024-08-17T18:09:24.108 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:24.129 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:24.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:24.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:24.161 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:09:24.231 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:24.253 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:24.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:24.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:24.354 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:24.379 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:24.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:24.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:24.411 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:09:24.481 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:24.503 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:24.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:24.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:24.604 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:24.608 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:09:24.608 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:09:24.627 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:09:24.627 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:09:24.628 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:24.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:24.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:24.646 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:09:24.646 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:09:24.663 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:09:24.663 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:09:24.679 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:09:24.679 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:09:24.695 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:09:24.696 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:09:24.712 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:09:24.713 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:09:24.729 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:09:24.729 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:09:24.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:24.754 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:24.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:24.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:24.856 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:24.879 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:24.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:24.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:24.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:24 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4279731306' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:09:24.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:24 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4279731306' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:09:24.981 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:25.004 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:25.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:25.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:25.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:24 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4279731306' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:09:25.106 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:25.128 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:25.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:25.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:25.230 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:25.251 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:25.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:25.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:25.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:25.379 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:25.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:25.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:25.481 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:25.506 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:25.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:25.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:25.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:25.633 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:25.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:25.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:25.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:25 smithi136 ceph-mon[109861]: pgmap v2792: 83 pgs: 83 active+clean; 687 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 8.6 KiB/s rd, 25 KiB/s wr, 5 op/s; 102 KiB/s, 0 objects/s recovering 2024-08-17T18:09:25.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:25 smithi136 ceph-mon[115078]: pgmap v2792: 83 pgs: 83 active+clean; 687 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 8.6 KiB/s rd, 25 KiB/s wr, 5 op/s; 102 KiB/s, 0 objects/s recovering 2024-08-17T18:09:25.736 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:25.761 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:25.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:25.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:25.863 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:25.885 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:25.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:25.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:25.987 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:25 smithi165 ceph-mon[105517]: pgmap v2792: 83 pgs: 83 active+clean; 687 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 8.6 KiB/s rd, 25 KiB/s wr, 5 op/s; 102 KiB/s, 0 objects/s recovering 2024-08-17T18:09:26.013 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:26.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:26.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:26.114 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:26.140 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:26.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:26.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:26.242 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:26.267 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:26.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:26.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:26.369 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:26.392 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:26.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:26.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:26.494 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:26.515 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:26.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:26.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:26.617 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:26.638 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:26.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:26.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:26.740 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:26.762 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:26.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:26.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:26.863 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:26.885 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:26.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:26.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:26.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:27.007 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:27.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:27.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:27.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:27.135 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:27.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:27.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:27.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:27.262 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:27.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:27.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:27.364 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:27.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:09:27.385 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:09:27.385 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:09:27.385 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:27.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:27.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:27.488 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:27.514 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:27.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:27.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:27.615 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:27.616 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'unique_pool_3' created 2024-08-17T18:09:27.642 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:27.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:27.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:27.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:09:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:09:27] "GET /metrics HTTP/1.1" 200 34214 "" "Prometheus/2.51.0" 2024-08-17T18:09:27.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:27 smithi136 ceph-mon[109861]: pgmap v2793: 83 pgs: 83 active+clean; 687 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 7.5 KiB/s rd, 22 KiB/s wr, 5 op/s; 89 KiB/s, 0 objects/s recovering 2024-08-17T18:09:27.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:27 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1466253289' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-08-17T18:09:27.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:27 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-08-17T18:09:27.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:27 smithi136 ceph-mon[115078]: pgmap v2793: 83 pgs: 83 active+clean; 687 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 7.5 KiB/s rd, 22 KiB/s wr, 5 op/s; 89 KiB/s, 0 objects/s recovering 2024-08-17T18:09:27.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:27 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1466253289' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-08-17T18:09:27.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:27 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-08-17T18:09:27.744 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:27.765 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:27.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:27.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:27.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:27.887 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:27.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:27.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:27.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:28.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:27 smithi165 ceph-mon[105517]: pgmap v2793: 83 pgs: 83 active+clean; 687 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 7.5 KiB/s rd, 22 KiB/s wr, 5 op/s; 89 KiB/s, 0 objects/s recovering 2024-08-17T18:09:28.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:27 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1466253289' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-08-17T18:09:28.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:27 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-08-17T18:09:28.015 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:28.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:28.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:28.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:28.142 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:28.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:28.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:28.244 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:28.266 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:28.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:28.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:28.368 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:28.389 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:28.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:28.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:28.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:28.513 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:28.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:28.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:28.614 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:28.641 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:28.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:28.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:28.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:28 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-08-17T18:09:28.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:28 smithi136 ceph-mon[109861]: osdmap e1535: 8 total, 8 up, 7 in 2024-08-17T18:09:28.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:28 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-08-17T18:09:28.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:28 smithi136 ceph-mon[115078]: osdmap e1535: 8 total, 8 up, 7 in 2024-08-17T18:09:28.743 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:28.765 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:28.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:28.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:28.823 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-08-17T18:09:28.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:28.888 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:28.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:28.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:28.990 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:28 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-08-17T18:09:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:28 smithi165 ceph-mon[105517]: osdmap e1535: 8 total, 8 up, 7 in 2024-08-17T18:09:29.011 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:29.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:29.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:29.113 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:29.135 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:29.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:29.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:29.237 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:29.258 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:29.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:29.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:29.360 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:29.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:29.382 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:29.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:29.483 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:29.509 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:29.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:29.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:29.611 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:29.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:29.636 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:29.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:29.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:29 smithi136 ceph-mon[109861]: pgmap v2795: 99 pgs: 1 creating+peering, 15 unknown, 83 active+clean; 687 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:29.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:29 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T18:09:29.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:29 smithi136 ceph-mon[109861]: osdmap e1536: 8 total, 8 up, 7 in 2024-08-17T18:09:29.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:29 smithi136 ceph-mon[115078]: pgmap v2795: 99 pgs: 1 creating+peering, 15 unknown, 83 active+clean; 687 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:29.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:29 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T18:09:29.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:29 smithi136 ceph-mon[115078]: osdmap e1536: 8 total, 8 up, 7 in 2024-08-17T18:09:29.737 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:29.758 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:29.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:29.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:29.860 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:29.881 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:09:29.886 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:29.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:29.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:29.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:30.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:29 smithi165 ceph-mon[105517]: pgmap v2795: 99 pgs: 1 creating+peering, 15 unknown, 83 active+clean; 687 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:29 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T18:09:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:29 smithi165 ceph-mon[105517]: osdmap e1536: 8 total, 8 up, 7 in 2024-08-17T18:09:30.015 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:30.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:30.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:30.020 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:09:30.021 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T18:09:30.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:30.143 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:30.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:30.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:30.245 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:30.270 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:30.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:30.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:30.372 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:30.393 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:09:30.397 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:30.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:30.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:30.421 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-17T18:09:30.499 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:30.525 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:30.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:30.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:30.627 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:30.649 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:30 smithi136 ceph-mon[109861]: osdmap e1537: 8 total, 8 up, 7 in 2024-08-17T18:09:30.650 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:30 smithi136 ceph-mon[115078]: osdmap e1537: 8 total, 8 up, 7 in 2024-08-17T18:09:30.652 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:30.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:30.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:30.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:30.777 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:30.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:30.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:30.879 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:30.903 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:30.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:30.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:31.005 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:30 smithi165 ceph-mon[105517]: osdmap e1537: 8 total, 8 up, 7 in 2024-08-17T18:09:31.027 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:31.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:31.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:31.057 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:09:31.129 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:31.155 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:31.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:31.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:31.257 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:31.282 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:31.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:31.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:31.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:31.406 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:31.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:31.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:31.507 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:31.530 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:31.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:31.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:31.631 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:31.653 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:31.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:31.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:31.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:31.781 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:31.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:31.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:31.882 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:31.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:31.905 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:31.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:31.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:31 smithi136 ceph-mon[109861]: from='client.45998 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:09:31.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:31 smithi136 ceph-mon[109861]: pgmap v2798: 99 pgs: 1 creating+peering, 10 unknown, 88 active+clean; 687 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:31.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:31 smithi136 ceph-mon[115078]: from='client.45998 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:09:31.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:31 smithi136 ceph-mon[115078]: pgmap v2798: 99 pgs: 1 creating+peering, 10 unknown, 88 active+clean; 687 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:32.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:32.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:31 smithi165 ceph-mon[105517]: from='client.45998 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:09:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:31 smithi165 ceph-mon[105517]: pgmap v2798: 99 pgs: 1 creating+peering, 10 unknown, 88 active+clean; 687 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:32.032 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:32.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:32.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:32.134 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:32.159 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:32.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:32.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:32.261 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:32.283 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:32.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:32.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:32.385 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:32.412 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:32.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:32.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:32.432 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":2797,"stamp":"2024-08-17T18:09:29.621099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":703281,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":68,"num_read_kb":150,"num_write":103,"num_write_kb":473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":1135098,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1698,"ondisk_log_size":1698,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":261,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":4594044,"kb_used_data":237100,"kb_used_omap":350,"kb_used_meta":4356513,"kb_avail":651622020,"statfs":{"total":671965249536,"available":667260948480,"internally_reserved":0,"allocated":242790400,"data_stored":204654508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":358888,"internal_metadata":4461069848},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.322977"},"pg_stats":[{"pgid":"56.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-17T18:09:27.605039+0000","last_change":"2024-08-17T18:09:27.605039+0000","last_active":"2024-08-17T18:09:27.605039+0000","last_peered":"2024-08-17T18:09:27.605039+0000","last_clean":"2024-08-17T18:09:27.605039+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:09:27.605039+0000","last_undegraded":"2024-08-17T18:09:27.605039+0000","last_fullsized":"2024-08-17T18:09:27.605039+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-17T18:09:27.605039+0000","last_change":"2024-08-17T18:09:27.605039+0000","last_active":"2024-08-17T18:09:27.605039+0000","last_peered":"2024-08-17T18:09:27.605039+0000","last_clean":"2024-08-17T18:09:27.605039+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:09:27.605039+0000","last_undegraded":"2024-08-17T18:09:27.605039+0000","last_fullsized":"2024-08-17T18:09:27.605039+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-17T18:09:27.605039+0000","last_change":"2024-08-17T18:09:27.605039+0000","last_active":"2024-08-17T18:09:27.605039+0000","last_peered":"2024-08-17T18:09:27.605039+0000","last_clean":"2024-08-17T18:09:27.605039+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:09:27.605039+0000","last_undegraded":"2024-08-17T18:09:27.605039+0000","last_fullsized":"2024-08-17T18:09:27.605039+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-17T18:09:27.605039+0000","last_change":"2024-08-17T18:09:27.605039+0000","last_active":"2024-08-17T18:09:27.605039+0000","last_peered":"2024-08-17T18:09:27.605039+0000","last_clean":"2024-08-17T18:09:27.605039+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:09:27.605039+0000","last_undegraded":"2024-08-17T18:09:27.605039+0000","last_fullsized":"2024-08-17T18:09:27.605039+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.b","version":"0'0","reported_seq":1,"reported_epoch":1535,"state":"creating+peering","last_fresh":"2024-08-17T18:09:27.615997+0000","last_change":"2024-08-17T18:09:27.615997+0000","last_active":"2024-08-17T18:09:27.605039+0000","last_peered":"2024-08-17T18:09:27.605039+0000","last_clean":"2024-08-17T18:09:27.605039+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:09:27.615997+0000","last_undegraded":"2024-08-17T18:09:27.615997+0000","last_fullsized":"2024-08-17T18:09:27.615997+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:47:24.480441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"56.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-17T18:09:27.605039+0000","last_change":"2024-08-17T18:09:27.605039+0000","last_active":"2024-08-17T18:09:27.605039+0000","last_peered":"2024-08-17T18:09:27.605039+0000","last_clean":"2024-08-17T18:09:27.605039+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:09:27.605039+0000","last_undegraded":"2024-08-17T18:09:27.605039+0000","last_fullsized":"2024-08-17T18:09:27.605039+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.9","version":"0'0","reported_seq":10,"reported_epoch":1536,"state":"active+clean","last_fresh":"2024-08-17T18:09:28.627487+0000","last_change":"2024-08-17T18:09:28.626545+0000","last_active":"2024-08-17T18:09:28.627487+0000","last_peered":"2024-08-17T18:09:28.627487+0000","last_clean":"2024-08-17T18:09:28.627487+0000","last_became_active":"2024-08-17T18:09:28.626300+0000","last_became_peered":"2024-08-17T18:09:28.626300+0000","last_unstale":"2024-08-17T18:09:28.627487+0000","last_undegraded":"2024-08-17T18:09:28.627487+0000","last_fullsized":"2024-08-17T18:09:28.627487+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:34:50.839206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-17T18:09:27.605039+0000","last_change":"2024-08-17T18:09:27.605039+0000","last_active":"2024-08-17T18:09:27.605039+0000","last_peered":"2024-08-17T18:09:27.605039+0000","last_clean":"2024-08-17T18:09:27.605039+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:09:27.605039+0000","last_undegraded":"2024-08-17T18:09:27.605039+0000","last_fullsized":"2024-08-17T18:09:27.605039+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-17T18:09:27.605039+0000","last_change":"2024-08-17T18:09:27.605039+0000","last_active":"2024-08-17T18:09:27.605039+0000","last_peered":"2024-08-17T18:09:27.605039+0000","last_clean":"2024-08-17T18:09:27.605039+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:09:27.605039+0000","last_undegraded":"2024-08-17T18:09:27.605039+0000","last_fullsized":"2024-08-17T18:09:27.605039+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.6","version":"0'0","reported_seq":10,"reported_epoch":1536,"state":"active+clean","last_fresh":"2024-08-17T18:09:28.630284+0000","last_change":"2024-08-17T18:09:28.629945+0000","last_active":"2024-08-17T18:09:28.630284+0000","last_peered":"2024-08-17T18:09:28.630284+0000","last_clean":"2024-08-17T18:09:28.630284+0000","last_became_active":"2024-08-17T18:09:28.629553+0000","last_became_peered":"2024-08-17T18:09:28.629553+0000","last_unstale":"2024-08-17T18:09:28.630284+0000","last_undegraded":"2024-08-17T18:09:28.630284+0000","last_fullsized":"2024-08-17T18:09:28.630284+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:00:36.875044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-17T18:09:27.605039+0000","last_change":"2024-08-17T18:09:27.605039+0000","last_active":"2024-08-17T18:09:27.605039+0000","last_peered":"2024-08-17T18:09:27.605039+0000","last_clean":"2024-08-17T18:09:27.605039+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:09:27.605039+0000","last_undegraded":"2024-08-17T18:09:27.605039+0000","last_fullsized":"2024-08-17T18:09:27.605039+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-17T18:09:27.605039+0000","last_change":"2024-08-17T18:09:27.605039+0000","last_active":"2024-08-17T18:09:27.605039+0000","last_peered":"2024-08-17T18:09:27.605039+0000","last_clean":"2024-08-17T18:09:27.605039+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:09:27.605039+0000","last_undegraded":"2024-08-17T18:09:27.605039+0000","last_fullsized":"2024-08-17T18:09:27.605039+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-17T18:09:27.605039+0000","last_change":"2024-08-17T18:09:27.605039+0000","last_active":"2024-08-17T18:09:27.605039+0000","last_peered":"2024-08-17T18:09:27.605039+0000","last_clean":"2024-08-17T18:09:27.605039+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:09:27.605039+0000","last_undegraded":"2024-08-17T18:09:27.605039+0000","last_fullsized":"2024-08-17T18:09:27.605039+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"56.2","version":"0'0","reported_seq":10,"reported_epoch":1536,"state":"active+clean","last_fresh":"2024-08-17T18:09:28.629028+0000","last_change":"2024-08-17T18:09:28.628711+0000","last_active":"2024-08-17T18:09:28.629028+0000","last_peered":"2024-08-17T18:09:28.629028+0000","last_clean":"2024-08-17T18:09:28.629028+0000","last_became_active":"2024-08-17T18:09:28.628387+0000","last_became_peered":"2024-08-17T18:09:28.628387+0000","last_unstale":"2024-08-17T18:09:28.629028+0000","last_undegraded":"2024-08-17T18:09:28.629028+0000","last_fullsized":"2024-08-17T18:09:28.629028+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:26:06.412021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.1","version":"0'0","reported_seq":10,"reported_epoch":1536,"state":"active+clean","last_fresh":"2024-08-17T18:09:28.629966+0000","last_change":"2024-08-17T18:09:28.629660+0000","last_active":"2024-08-17T18:09:28.629966+0000","last_peered":"2024-08-17T18:09:28.629966+0000","last_clean":"2024-08-17T18:09:28.629966+0000","last_became_active":"2024-08-17T18:09:28.629359+0000","last_became_peered":"2024-08-17T18:09:28.629359+0000","last_unstale":"2024-08-17T18:09:28.629966+0000","last_undegraded":"2024-08-17T18:09:28.629966+0000","last_fullsized":"2024-08-17T18:09:28.629966+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:05:58.848611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.0","version":"0'0","reported_seq":10,"reported_epoch":1536,"state":"active+clean","last_fresh":"2024-08-17T18:09:28.697498+0000","last_change":"2024-08-17T18:09:28.697077+0000","last_active":"2024-08-17T18:09:28.697498+0000","last_peered":"2024-08-17T18:09:28.697498+0000","last_clean":"2024-08-17T18:09:28.697498+0000","last_became_active":"2024-08-17T18:09:28.696676+0000","last_became_peered":"2024-08-17T18:09:28.696676+0000","last_unstale":"2024-08-17T18:09:28.697498+0000","last_undegraded":"2024-08-17T18:09:28.697498+0000","last_fullsized":"2024-08-17T18:09:28.697498+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:09.350570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":211,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.964341+0000","last_change":"2024-08-17T18:07:48.546202+0000","last_active":"2024-08-17T18:09:21.964341+0000","last_peered":"2024-08-17T18:09:21.964341+0000","last_clean":"2024-08-17T18:09:21.964341+0000","last_became_active":"2024-08-17T18:07:48.531246+0000","last_became_peered":"2024-08-17T18:07:48.531246+0000","last_unstale":"2024-08-17T18:09:21.964341+0000","last_undegraded":"2024-08-17T18:09:21.964341+0000","last_fullsized":"2024-08-17T18:09:21.964341+0000","mapping_epoch":1501,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:30.944362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1538,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.192770+0000","last_change":"2024-08-17T18:08:37.068490+0000","last_active":"2024-08-17T18:09:21.192770+0000","last_peered":"2024-08-17T18:09:21.192770+0000","last_clean":"2024-08-17T18:09:21.192770+0000","last_became_active":"2024-08-17T18:08:37.068026+0000","last_became_peered":"2024-08-17T18:08:37.068026+0000","last_unstale":"2024-08-17T18:09:21.192770+0000","last_undegraded":"2024-08-17T18:09:21.192770+0000","last_fullsized":"2024-08-17T18:09:21.192770+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:34:27.467963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":2387,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.916178+0000","last_change":"2024-08-17T18:08:37.082356+0000","last_active":"2024-08-17T18:09:21.916178+0000","last_peered":"2024-08-17T18:09:21.916178+0000","last_clean":"2024-08-17T18:09:21.916178+0000","last_became_active":"2024-08-17T18:08:37.082046+0000","last_became_peered":"2024-08-17T18:08:37.082046+0000","last_unstale":"2024-08-17T18:09:21.916178+0000","last_undegraded":"2024-08-17T18:09:21.916178+0000","last_fullsized":"2024-08-17T18:09:21.916178+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:46:13.932917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2409,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.915872+0000","last_change":"2024-08-17T18:04:14.540774+0000","last_active":"2024-08-17T18:09:21.915872+0000","last_peered":"2024-08-17T18:09:21.915872+0000","last_clean":"2024-08-17T18:09:21.915872+0000","last_became_active":"2024-08-17T18:04:14.539778+0000","last_became_peered":"2024-08-17T18:04:14.539778+0000","last_unstale":"2024-08-17T18:09:21.915872+0000","last_undegraded":"2024-08-17T18:09:21.915872+0000","last_fullsized":"2024-08-17T18:09:21.915872+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:55:02.863360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":87,"reported_epoch":1535,"state":"active+clean","last_fresh":"2024-08-17T18:09:27.618896+0000","last_change":"2024-08-17T18:04:43.777705+0000","last_active":"2024-08-17T18:09:27.618896+0000","last_peered":"2024-08-17T18:09:27.618896+0000","last_clean":"2024-08-17T18:09:27.618896+0000","last_became_active":"2024-08-17T18:04:43.771283+0000","last_became_peered":"2024-08-17T18:04:43.771283+0000","last_unstale":"2024-08-17T18:09:27.618896+0000","last_undegraded":"2024-08-17T18:09:27.618896+0000","last_fullsized":"2024-08-17T18:09:27.618896+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:13.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1654,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.915917+0000","last_change":"2024-08-17T18:04:14.534963+0000","last_active":"2024-08-17T18:09:21.915917+0000","last_peered":"2024-08-17T18:09:21.915917+0000","last_clean":"2024-08-17T18:09:21.915917+0000","last_became_active":"2024-08-17T18:04:14.534634+0000","last_became_peered":"2024-08-17T18:04:14.534634+0000","last_unstale":"2024-08-17T18:09:21.915917+0000","last_undegraded":"2024-08-17T18:09:21.915917+0000","last_fullsized":"2024-08-17T18:09:21.915917+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:33:27.287860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":87,"reported_epoch":1535,"state":"active+clean","last_fresh":"2024-08-17T18:09:27.618999+0000","last_change":"2024-08-17T18:04:43.773534+0000","last_active":"2024-08-17T18:09:27.618999+0000","last_peered":"2024-08-17T18:09:27.618999+0000","last_clean":"2024-08-17T18:09:27.618999+0000","last_became_active":"2024-08-17T18:04:43.768507+0000","last_became_peered":"2024-08-17T18:04:43.768507+0000","last_unstale":"2024-08-17T18:09:27.618999+0000","last_undegraded":"2024-08-17T18:09:27.618999+0000","last_fullsized":"2024-08-17T18:09:27.618999+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:50:57.647351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1525,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.193040+0000","last_change":"2024-08-17T17:24:17.458168+0000","last_active":"2024-08-17T18:09:21.193040+0000","last_peered":"2024-08-17T18:09:21.193040+0000","last_clean":"2024-08-17T18:09:21.193040+0000","last_became_active":"2024-08-17T17:24:17.457847+0000","last_became_peered":"2024-08-17T17:24:17.457847+0000","last_unstale":"2024-08-17T18:09:21.193040+0000","last_undegraded":"2024-08-17T18:09:21.193040+0000","last_fullsized":"2024-08-17T18:09:21.193040+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:28.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":166,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.964024+0000","last_change":"2024-08-17T18:04:14.531445+0000","last_active":"2024-08-17T18:09:21.964024+0000","last_peered":"2024-08-17T18:09:21.964024+0000","last_clean":"2024-08-17T18:09:21.964024+0000","last_became_active":"2024-08-17T18:04:14.531099+0000","last_became_peered":"2024-08-17T18:04:14.531099+0000","last_unstale":"2024-08-17T18:09:21.964024+0000","last_undegraded":"2024-08-17T18:09:21.964024+0000","last_fullsized":"2024-08-17T18:09:21.964024+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:43.335123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":76,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:20.607018+0000","last_change":"2024-08-17T18:07:48.548262+0000","last_active":"2024-08-17T18:09:20.607018+0000","last_peered":"2024-08-17T18:09:20.607018+0000","last_clean":"2024-08-17T18:09:20.607018+0000","last_became_active":"2024-08-17T18:07:48.547846+0000","last_became_peered":"2024-08-17T18:07:48.547846+0000","last_unstale":"2024-08-17T18:09:20.607018+0000","last_undegraded":"2024-08-17T18:09:20.607018+0000","last_fullsized":"2024-08-17T18:09:20.607018+0000","mapping_epoch":1501,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:24:37.513219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1542,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.192937+0000","last_change":"2024-08-17T18:01:23.519890+0000","last_active":"2024-08-17T18:09:21.192937+0000","last_peered":"2024-08-17T18:09:21.192937+0000","last_clean":"2024-08-17T18:09:21.192937+0000","last_became_active":"2024-08-17T18:01:23.519526+0000","last_became_peered":"2024-08-17T18:01:23.519526+0000","last_unstale":"2024-08-17T18:09:21.192937+0000","last_undegraded":"2024-08-17T18:09:21.192937+0000","last_fullsized":"2024-08-17T18:09:21.192937+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:04:52.662528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2375,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.916147+0000","last_change":"2024-08-17T18:01:23.491836+0000","last_active":"2024-08-17T18:09:21.916147+0000","last_peered":"2024-08-17T18:09:21.916147+0000","last_clean":"2024-08-17T18:09:21.916147+0000","last_became_active":"2024-08-17T18:01:23.491209+0000","last_became_peered":"2024-08-17T18:01:23.491209+0000","last_unstale":"2024-08-17T18:09:21.916147+0000","last_undegraded":"2024-08-17T18:09:21.916147+0000","last_fullsized":"2024-08-17T18:09:21.916147+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:35:30.767005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2377,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:22.093195+0000","last_change":"2024-08-17T17:24:17.452025+0000","last_active":"2024-08-17T18:09:22.093195+0000","last_peered":"2024-08-17T18:09:22.093195+0000","last_clean":"2024-08-17T18:09:22.093195+0000","last_became_active":"2024-08-17T17:24:17.448029+0000","last_became_peered":"2024-08-17T17:24:17.448029+0000","last_unstale":"2024-08-17T18:09:22.093195+0000","last_undegraded":"2024-08-17T18:09:22.093195+0000","last_fullsized":"2024-08-17T18:09:22.093195+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:45.089346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":180,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.964252+0000","last_change":"2024-08-17T18:08:11.358013+0000","last_active":"2024-08-17T18:09:21.964252+0000","last_peered":"2024-08-17T18:09:21.964252+0000","last_clean":"2024-08-17T18:09:21.964252+0000","last_became_active":"2024-08-17T18:08:11.357666+0000","last_became_peered":"2024-08-17T18:08:11.357666+0000","last_unstale":"2024-08-17T18:09:21.964252+0000","last_undegraded":"2024-08-17T18:09:21.964252+0000","last_fullsized":"2024-08-17T18:09:21.964252+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:05:25.290465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2389,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.915793+0000","last_change":"2024-08-17T18:04:14.539660+0000","last_active":"2024-08-17T18:09:21.915793+0000","last_peered":"2024-08-17T18:09:21.915793+0000","last_clean":"2024-08-17T18:09:21.915793+0000","last_became_active":"2024-08-17T18:04:14.537852+0000","last_became_peered":"2024-08-17T18:04:14.537852+0000","last_unstale":"2024-08-17T18:09:21.915793+0000","last_undegraded":"2024-08-17T18:09:21.915793+0000","last_fullsized":"2024-08-17T18:09:21.915793+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:47:45.026997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1538,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.589229+0000","last_change":"2024-08-17T18:04:43.769892+0000","last_active":"2024-08-17T18:09:15.589229+0000","last_peered":"2024-08-17T18:09:15.589229+0000","last_clean":"2024-08-17T18:09:15.589229+0000","last_became_active":"2024-08-17T18:04:43.769702+0000","last_became_peered":"2024-08-17T18:04:43.769702+0000","last_unstale":"2024-08-17T18:09:15.589229+0000","last_undegraded":"2024-08-17T18:09:15.589229+0000","last_fullsized":"2024-08-17T18:09:15.589229+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:39.373904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2349,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.915606+0000","last_change":"2024-08-17T17:24:17.442338+0000","last_active":"2024-08-17T18:09:21.915606+0000","last_peered":"2024-08-17T18:09:21.915606+0000","last_clean":"2024-08-17T18:09:21.915606+0000","last_became_active":"2024-08-17T17:24:17.441948+0000","last_became_peered":"2024-08-17T17:24:17.441948+0000","last_unstale":"2024-08-17T18:09:21.915606+0000","last_undegraded":"2024-08-17T18:09:21.915606+0000","last_fullsized":"2024-08-17T18:09:21.915606+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:17.891492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":173,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.964369+0000","last_change":"2024-08-17T18:04:43.777736+0000","last_active":"2024-08-17T18:09:21.964369+0000","last_peered":"2024-08-17T18:09:21.964369+0000","last_clean":"2024-08-17T18:09:21.964369+0000","last_became_active":"2024-08-17T18:04:43.777438+0000","last_became_peered":"2024-08-17T18:04:43.777438+0000","last_unstale":"2024-08-17T18:09:21.964369+0000","last_undegraded":"2024-08-17T18:09:21.964369+0000","last_fullsized":"2024-08-17T18:09:21.964369+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:24:01.255927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2329,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.915956+0000","last_change":"2024-08-17T18:04:43.915398+0000","last_active":"2024-08-17T18:09:21.915956+0000","last_peered":"2024-08-17T18:09:21.915956+0000","last_clean":"2024-08-17T18:09:21.915956+0000","last_became_active":"2024-08-17T18:04:43.915212+0000","last_became_peered":"2024-08-17T18:04:43.915212+0000","last_unstale":"2024-08-17T18:09:21.915956+0000","last_undegraded":"2024-08-17T18:09:21.915956+0000","last_fullsized":"2024-08-17T18:09:21.915956+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:43:26.901830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":100,"reported_epoch":1535,"state":"active+clean","last_fresh":"2024-08-17T18:09:27.619049+0000","last_change":"2024-08-17T18:08:11.356876+0000","last_active":"2024-08-17T18:09:27.619049+0000","last_peered":"2024-08-17T18:09:27.619049+0000","last_clean":"2024-08-17T18:09:27.619049+0000","last_became_active":"2024-08-17T18:08:11.355662+0000","last_became_peered":"2024-08-17T18:08:11.355662+0000","last_unstale":"2024-08-17T18:09:27.619049+0000","last_undegraded":"2024-08-17T18:09:27.619049+0000","last_fullsized":"2024-08-17T18:09:27.619049+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:55:00.801251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":486,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.916177+0000","last_change":"2024-08-17T18:04:14.555729+0000","last_active":"2024-08-17T18:09:21.916177+0000","last_peered":"2024-08-17T18:09:21.916177+0000","last_clean":"2024-08-17T18:09:21.916177+0000","last_became_active":"2024-08-17T18:04:14.536678+0000","last_became_peered":"2024-08-17T18:04:14.536678+0000","last_unstale":"2024-08-17T18:09:21.916177+0000","last_undegraded":"2024-08-17T18:09:21.916177+0000","last_fullsized":"2024-08-17T18:09:21.916177+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:47:29.424778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2401,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:22.093537+0000","last_change":"2024-08-17T18:08:37.079571+0000","last_active":"2024-08-17T18:09:22.093537+0000","last_peered":"2024-08-17T18:09:22.093537+0000","last_clean":"2024-08-17T18:09:22.093537+0000","last_became_active":"2024-08-17T18:08:37.079224+0000","last_became_peered":"2024-08-17T18:08:37.079224+0000","last_unstale":"2024-08-17T18:09:22.093537+0000","last_undegraded":"2024-08-17T18:09:22.093537+0000","last_fullsized":"2024-08-17T18:09:22.093537+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:28:50.561072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1540,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.192975+0000","last_change":"2024-08-17T18:03:51.292044+0000","last_active":"2024-08-17T18:09:21.192975+0000","last_peered":"2024-08-17T18:09:21.192975+0000","last_clean":"2024-08-17T18:09:21.192975+0000","last_became_active":"2024-08-17T18:03:51.268271+0000","last_became_peered":"2024-08-17T18:03:51.268271+0000","last_unstale":"2024-08-17T18:09:21.192975+0000","last_undegraded":"2024-08-17T18:09:21.192975+0000","last_fullsized":"2024-08-17T18:09:21.192975+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:00.892410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4756,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1512,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.193040+0000","last_change":"2024-08-17T17:23:07.455919+0000","last_active":"2024-08-17T18:09:21.193040+0000","last_peered":"2024-08-17T18:09:21.193040+0000","last_clean":"2024-08-17T18:09:21.193040+0000","last_became_active":"2024-08-17T17:23:07.455284+0000","last_became_peered":"2024-08-17T17:23:07.455284+0000","last_unstale":"2024-08-17T18:09:21.193040+0000","last_undegraded":"2024-08-17T18:09:21.193040+0000","last_fullsized":"2024-08-17T18:09:21.193040+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:37:46.260775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"347'149","reported_seq":809,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.915980+0000","last_change":"2024-08-17T18:09:15.592511+0000","last_active":"2024-08-17T18:09:21.915980+0000","last_peered":"2024-08-17T18:09:21.915980+0000","last_clean":"2024-08-17T18:09:21.915980+0000","last_became_active":"2024-08-17T18:09:15.592099+0000","last_became_peered":"2024-08-17T18:09:15.592099+0000","last_unstale":"2024-08-17T18:09:21.915980+0000","last_undegraded":"2024-08-17T18:09:21.915980+0000","last_fullsized":"2024-08-17T18:09:21.915980+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:57.271286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:57.271286+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:54.841504+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1536,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.193010+0000","last_change":"2024-08-17T18:04:43.766724+0000","last_active":"2024-08-17T18:09:21.193010+0000","last_peered":"2024-08-17T18:09:21.193010+0000","last_clean":"2024-08-17T18:09:21.193010+0000","last_became_active":"2024-08-17T18:04:43.766428+0000","last_became_peered":"2024-08-17T18:04:43.766428+0000","last_unstale":"2024-08-17T18:09:21.193010+0000","last_undegraded":"2024-08-17T18:09:21.193010+0000","last_fullsized":"2024-08-17T18:09:21.193010+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:09:49.102940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1548,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:20.935949+0000","last_change":"2024-08-17T18:03:51.279963+0000","last_active":"2024-08-17T18:09:20.935949+0000","last_peered":"2024-08-17T18:09:20.935949+0000","last_clean":"2024-08-17T18:09:20.935949+0000","last_became_active":"2024-08-17T18:03:51.279506+0000","last_became_peered":"2024-08-17T18:03:51.279506+0000","last_unstale":"2024-08-17T18:09:20.935949+0000","last_undegraded":"2024-08-17T18:09:20.935949+0000","last_fullsized":"2024-08-17T18:09:20.935949+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:20:07.304042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":170,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.964348+0000","last_change":"2024-08-17T18:04:14.542589+0000","last_active":"2024-08-17T18:09:21.964348+0000","last_peered":"2024-08-17T18:09:21.964348+0000","last_clean":"2024-08-17T18:09:21.964348+0000","last_became_active":"2024-08-17T18:04:14.531540+0000","last_became_peered":"2024-08-17T18:04:14.531540+0000","last_unstale":"2024-08-17T18:09:21.964348+0000","last_undegraded":"2024-08-17T18:09:21.964348+0000","last_fullsized":"2024-08-17T18:09:21.964348+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:15:33.014782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"347'149","reported_seq":806,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.193003+0000","last_change":"2024-08-17T18:09:15.592395+0000","last_active":"2024-08-17T18:09:21.193003+0000","last_peered":"2024-08-17T18:09:21.193003+0000","last_clean":"2024-08-17T18:09:21.193003+0000","last_became_active":"2024-08-17T18:09:15.591773+0000","last_became_peered":"2024-08-17T18:09:15.591773+0000","last_unstale":"2024-08-17T18:09:21.193003+0000","last_undegraded":"2024-08-17T18:09:21.193003+0000","last_fullsized":"2024-08-17T18:09:21.193003+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:53.309517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:53.309517+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:50:57.110936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":62,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.915822+0000","last_change":"2024-08-17T18:08:11.349400+0000","last_active":"2024-08-17T18:09:21.915822+0000","last_peered":"2024-08-17T18:09:21.915822+0000","last_clean":"2024-08-17T18:09:21.915822+0000","last_became_active":"2024-08-17T18:08:11.348892+0000","last_became_peered":"2024-08-17T18:08:11.348892+0000","last_unstale":"2024-08-17T18:09:21.915822+0000","last_undegraded":"2024-08-17T18:09:21.915822+0000","last_fullsized":"2024-08-17T18:09:21.915822+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:52:54.819838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2359,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:22.093777+0000","last_change":"2024-08-17T17:23:07.428296+0000","last_active":"2024-08-17T18:09:22.093777+0000","last_peered":"2024-08-17T18:09:22.093777+0000","last_clean":"2024-08-17T18:09:22.093777+0000","last_became_active":"2024-08-17T17:23:07.428010+0000","last_became_peered":"2024-08-17T17:23:07.428010+0000","last_unstale":"2024-08-17T18:09:22.093777+0000","last_undegraded":"2024-08-17T18:09:22.093777+0000","last_fullsized":"2024-08-17T18:09:22.093777+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:45.459212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":2400,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.916342+0000","last_change":"2024-08-17T18:04:43.800633+0000","last_active":"2024-08-17T18:09:21.916342+0000","last_peered":"2024-08-17T18:09:21.916342+0000","last_clean":"2024-08-17T18:09:21.916342+0000","last_became_active":"2024-08-17T18:04:43.784612+0000","last_became_peered":"2024-08-17T18:04:43.784612+0000","last_unstale":"2024-08-17T18:09:21.916342+0000","last_undegraded":"2024-08-17T18:09:21.916342+0000","last_fullsized":"2024-08-17T18:09:21.916342+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:11.405039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"347'149","reported_seq":776,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:17.223467+0000","last_change":"2024-08-17T18:09:15.595301+0000","last_active":"2024-08-17T18:09:17.223467+0000","last_peered":"2024-08-17T18:09:17.223467+0000","last_clean":"2024-08-17T18:09:17.223467+0000","last_became_active":"2024-08-17T18:09:15.595011+0000","last_became_peered":"2024-08-17T18:09:15.595011+0000","last_unstale":"2024-08-17T18:09:17.223467+0000","last_undegraded":"2024-08-17T18:09:17.223467+0000","last_fullsized":"2024-08-17T18:09:17.223467+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:58.286040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:58.286040+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":173,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.964160+0000","last_change":"2024-08-17T18:04:43.805055+0000","last_active":"2024-08-17T18:09:21.964160+0000","last_peered":"2024-08-17T18:09:21.964160+0000","last_clean":"2024-08-17T18:09:21.964160+0000","last_became_active":"2024-08-17T18:04:43.804708+0000","last_became_peered":"2024-08-17T18:04:43.804708+0000","last_unstale":"2024-08-17T18:09:21.964160+0000","last_undegraded":"2024-08-17T18:09:21.964160+0000","last_fullsized":"2024-08-17T18:09:21.964160+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:47:22.253073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":166,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.964187+0000","last_change":"2024-08-17T18:04:14.529605+0000","last_active":"2024-08-17T18:09:21.964187+0000","last_peered":"2024-08-17T18:09:21.964187+0000","last_clean":"2024-08-17T18:09:21.964187+0000","last_became_active":"2024-08-17T18:04:14.529024+0000","last_became_peered":"2024-08-17T18:04:14.529024+0000","last_unstale":"2024-08-17T18:09:21.964187+0000","last_undegraded":"2024-08-17T18:09:21.964187+0000","last_fullsized":"2024-08-17T18:09:21.964187+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:38:00.485543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2413,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:22.093715+0000","last_change":"2024-08-17T18:04:43.773667+0000","last_active":"2024-08-17T18:09:22.093715+0000","last_peered":"2024-08-17T18:09:22.093715+0000","last_clean":"2024-08-17T18:09:22.093715+0000","last_became_active":"2024-08-17T18:04:43.773247+0000","last_became_peered":"2024-08-17T18:04:43.773247+0000","last_unstale":"2024-08-17T18:09:22.093715+0000","last_undegraded":"2024-08-17T18:09:22.093715+0000","last_fullsized":"2024-08-17T18:09:22.093715+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:35:48.964631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"347'149","reported_seq":776,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:17.223484+0000","last_change":"2024-08-17T18:09:15.593038+0000","last_active":"2024-08-17T18:09:17.223484+0000","last_peered":"2024-08-17T18:09:17.223484+0000","last_clean":"2024-08-17T18:09:17.223484+0000","last_became_active":"2024-08-17T18:09:15.592534+0000","last_became_peered":"2024-08-17T18:09:15.592534+0000","last_unstale":"2024-08-17T18:09:17.223484+0000","last_undegraded":"2024-08-17T18:09:17.223484+0000","last_fullsized":"2024-08-17T18:09:17.223484+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:51.289001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:51.289001+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":19,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":237,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:20.606885+0000","last_change":"2024-08-17T18:04:43.771589+0000","last_active":"2024-08-17T18:09:20.606885+0000","last_peered":"2024-08-17T18:09:20.606885+0000","last_clean":"2024-08-17T18:09:20.606885+0000","last_became_active":"2024-08-17T18:04:43.769601+0000","last_became_peered":"2024-08-17T18:04:43.769601+0000","last_unstale":"2024-08-17T18:09:20.606885+0000","last_undegraded":"2024-08-17T18:09:20.606885+0000","last_fullsized":"2024-08-17T18:09:20.606885+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:42:49.228921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2429,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:22.093488+0000","last_change":"2024-08-17T18:04:14.537136+0000","last_active":"2024-08-17T18:09:22.093488+0000","last_peered":"2024-08-17T18:09:22.093488+0000","last_clean":"2024-08-17T18:09:22.093488+0000","last_became_active":"2024-08-17T18:04:14.536815+0000","last_became_peered":"2024-08-17T18:04:14.536815+0000","last_unstale":"2024-08-17T18:09:22.093488+0000","last_undegraded":"2024-08-17T18:09:22.093488+0000","last_fullsized":"2024-08-17T18:09:22.093488+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:02.706133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":84,"reported_epoch":1535,"state":"active+clean","last_fresh":"2024-08-17T18:09:27.619308+0000","last_change":"2024-08-17T18:04:43.773480+0000","last_active":"2024-08-17T18:09:27.619308+0000","last_peered":"2024-08-17T18:09:27.619308+0000","last_clean":"2024-08-17T18:09:27.619308+0000","last_became_active":"2024-08-17T18:04:43.773160+0000","last_became_peered":"2024-08-17T18:04:43.773160+0000","last_unstale":"2024-08-17T18:09:27.619308+0000","last_undegraded":"2024-08-17T18:09:27.619308+0000","last_fullsized":"2024-08-17T18:09:27.619308+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:07:55.468132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"347'149","reported_seq":808,"reported_epoch":1535,"state":"active+clean","last_fresh":"2024-08-17T18:09:27.618675+0000","last_change":"2024-08-17T18:09:15.596647+0000","last_active":"2024-08-17T18:09:27.618675+0000","last_peered":"2024-08-17T18:09:27.618675+0000","last_clean":"2024-08-17T18:09:27.618675+0000","last_became_active":"2024-08-17T18:09:15.596339+0000","last_became_peered":"2024-08-17T18:09:15.596339+0000","last_unstale":"2024-08-17T18:09:27.618675+0000","last_undegraded":"2024-08-17T18:09:27.618675+0000","last_fullsized":"2024-08-17T18:09:27.618675+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:00.316884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:00.316884+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:30.779927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":239,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:22.093583+0000","last_change":"2024-08-17T18:02:08.224928+0000","last_active":"2024-08-17T18:09:22.093583+0000","last_peered":"2024-08-17T18:09:22.093583+0000","last_clean":"2024-08-17T18:09:22.093583+0000","last_became_active":"2024-08-17T18:02:08.224604+0000","last_became_peered":"2024-08-17T18:02:08.224604+0000","last_unstale":"2024-08-17T18:09:22.093583+0000","last_undegraded":"2024-08-17T18:09:22.093583+0000","last_fullsized":"2024-08-17T18:09:22.093583+0000","mapping_epoch":1421,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1422,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:55.716971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1534,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.193151+0000","last_change":"2024-08-17T18:04:14.526890+0000","last_active":"2024-08-17T18:09:21.193151+0000","last_peered":"2024-08-17T18:09:21.193151+0000","last_clean":"2024-08-17T18:09:21.193151+0000","last_became_active":"2024-08-17T18:04:14.526564+0000","last_became_peered":"2024-08-17T18:04:14.526564+0000","last_unstale":"2024-08-17T18:09:21.193151+0000","last_undegraded":"2024-08-17T18:09:21.193151+0000","last_fullsized":"2024-08-17T18:09:21.193151+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:10:13.229827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":88,"reported_epoch":1535,"state":"active+clean","last_fresh":"2024-08-17T18:09:27.619087+0000","last_change":"2024-08-17T18:04:43.791749+0000","last_active":"2024-08-17T18:09:27.619087+0000","last_peered":"2024-08-17T18:09:27.619087+0000","last_clean":"2024-08-17T18:09:27.619087+0000","last_became_active":"2024-08-17T18:04:43.774474+0000","last_became_peered":"2024-08-17T18:04:43.774474+0000","last_unstale":"2024-08-17T18:09:27.619087+0000","last_undegraded":"2024-08-17T18:09:27.619087+0000","last_fullsized":"2024-08-17T18:09:27.619087+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:19:08.097560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"347'149","reported_seq":814,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.963992+0000","last_change":"2024-08-17T18:09:15.596502+0000","last_active":"2024-08-17T18:09:21.963992+0000","last_peered":"2024-08-17T18:09:21.963992+0000","last_clean":"2024-08-17T18:09:21.963992+0000","last_became_active":"2024-08-17T18:09:15.595460+0000","last_became_peered":"2024-08-17T18:09:15.595460+0000","last_unstale":"2024-08-17T18:09:21.963992+0000","last_undegraded":"2024-08-17T18:09:21.963992+0000","last_fullsized":"2024-08-17T18:09:21.963992+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:02.363531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:02.363531+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:22:42.587876+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2415,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:22.093676+0000","last_change":"2024-08-17T18:04:43.781133+0000","last_active":"2024-08-17T18:09:22.093676+0000","last_peered":"2024-08-17T18:09:22.093676+0000","last_clean":"2024-08-17T18:09:22.093676+0000","last_became_active":"2024-08-17T18:04:43.780787+0000","last_became_peered":"2024-08-17T18:04:43.780787+0000","last_unstale":"2024-08-17T18:09:22.093676+0000","last_undegraded":"2024-08-17T18:09:22.093676+0000","last_fullsized":"2024-08-17T18:09:22.093676+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:38:34.901162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2372,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:22.093315+0000","last_change":"2024-08-17T18:01:23.493896+0000","last_active":"2024-08-17T18:09:22.093315+0000","last_peered":"2024-08-17T18:09:22.093315+0000","last_clean":"2024-08-17T18:09:22.093315+0000","last_became_active":"2024-08-17T18:01:23.493551+0000","last_became_peered":"2024-08-17T18:01:23.493551+0000","last_unstale":"2024-08-17T18:09:22.093315+0000","last_undegraded":"2024-08-17T18:09:22.093315+0000","last_fullsized":"2024-08-17T18:09:22.093315+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:18:47.648483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2349,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:20.606679+0000","last_change":"2024-08-17T18:04:14.543532+0000","last_active":"2024-08-17T18:09:20.606679+0000","last_peered":"2024-08-17T18:09:20.606679+0000","last_clean":"2024-08-17T18:09:20.606679+0000","last_became_active":"2024-08-17T18:04:14.543131+0000","last_became_peered":"2024-08-17T18:04:14.543131+0000","last_unstale":"2024-08-17T18:09:20.606679+0000","last_undegraded":"2024-08-17T18:09:20.606679+0000","last_fullsized":"2024-08-17T18:09:20.606679+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:51:44.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"347'149","reported_seq":664,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:22.093355+0000","last_change":"2024-08-17T18:09:15.740862+0000","last_active":"2024-08-17T18:09:22.093355+0000","last_peered":"2024-08-17T18:09:22.093355+0000","last_clean":"2024-08-17T18:09:22.093355+0000","last_became_active":"2024-08-17T18:09:15.591452+0000","last_became_peered":"2024-08-17T18:09:15.591452+0000","last_unstale":"2024-08-17T18:09:22.093355+0000","last_undegraded":"2024-08-17T18:09:22.093355+0000","last_fullsized":"2024-08-17T18:09:22.093355+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:15.740775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:15.740775+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:29:12.741753+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":180,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.964224+0000","last_change":"2024-08-17T18:07:48.539239+0000","last_active":"2024-08-17T18:09:21.964224+0000","last_peered":"2024-08-17T18:09:21.964224+0000","last_clean":"2024-08-17T18:09:21.964224+0000","last_became_active":"2024-08-17T18:07:48.538888+0000","last_became_peered":"2024-08-17T18:07:48.538888+0000","last_unstale":"2024-08-17T18:09:21.964224+0000","last_undegraded":"2024-08-17T18:09:21.964224+0000","last_fullsized":"2024-08-17T18:09:21.964224+0000","mapping_epoch":1501,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:35:23.855790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":180,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.964139+0000","last_change":"2024-08-17T18:08:11.440676+0000","last_active":"2024-08-17T18:09:21.964139+0000","last_peered":"2024-08-17T18:09:21.964139+0000","last_clean":"2024-08-17T18:09:21.964139+0000","last_became_active":"2024-08-17T18:08:11.440353+0000","last_became_peered":"2024-08-17T18:08:11.440353+0000","last_unstale":"2024-08-17T18:09:21.964139+0000","last_undegraded":"2024-08-17T18:09:21.964139+0000","last_fullsized":"2024-08-17T18:09:21.964139+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:19:22.125277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1115,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:22.093415+0000","last_change":"2024-08-17T18:04:14.537308+0000","last_active":"2024-08-17T18:09:22.093415+0000","last_peered":"2024-08-17T18:09:22.093415+0000","last_clean":"2024-08-17T18:09:22.093415+0000","last_became_active":"2024-08-17T18:04:14.535521+0000","last_became_peered":"2024-08-17T18:04:14.535521+0000","last_unstale":"2024-08-17T18:09:22.093415+0000","last_undegraded":"2024-08-17T18:09:22.093415+0000","last_fullsized":"2024-08-17T18:09:22.093415+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:40:00.384936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":29,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.589084+0000","last_change":"2024-08-17T18:08:37.071912+0000","last_active":"2024-08-17T18:09:15.589084+0000","last_peered":"2024-08-17T18:09:15.589084+0000","last_clean":"2024-08-17T18:09:15.589084+0000","last_became_active":"2024-08-17T18:08:37.071648+0000","last_became_peered":"2024-08-17T18:08:37.071648+0000","last_unstale":"2024-08-17T18:09:15.589084+0000","last_undegraded":"2024-08-17T18:09:15.589084+0000","last_fullsized":"2024-08-17T18:09:15.589084+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:58:57.279808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":177,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:20.606792+0000","last_change":"2024-08-17T18:04:43.771309+0000","last_active":"2024-08-17T18:09:20.606792+0000","last_peered":"2024-08-17T18:09:20.606792+0000","last_clean":"2024-08-17T18:09:20.606792+0000","last_became_active":"2024-08-17T18:04:43.770922+0000","last_became_peered":"2024-08-17T18:04:43.770922+0000","last_unstale":"2024-08-17T18:09:20.606792+0000","last_undegraded":"2024-08-17T18:09:20.606792+0000","last_fullsized":"2024-08-17T18:09:20.606792+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:10.173251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":88,"reported_epoch":1535,"state":"active+clean","last_fresh":"2024-08-17T18:09:27.618885+0000","last_change":"2024-08-17T18:04:43.786228+0000","last_active":"2024-08-17T18:09:27.618885+0000","last_peered":"2024-08-17T18:09:27.618885+0000","last_clean":"2024-08-17T18:09:27.618885+0000","last_became_active":"2024-08-17T18:04:43.770976+0000","last_became_peered":"2024-08-17T18:04:43.770976+0000","last_unstale":"2024-08-17T18:09:27.618885+0000","last_undegraded":"2024-08-17T18:09:27.618885+0000","last_fullsized":"2024-08-17T18:09:27.618885+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:20:10.131137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":805,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.963838+0000","last_change":"2024-08-17T18:07:54.303783+0000","last_active":"2024-08-17T18:09:21.963838+0000","last_peered":"2024-08-17T18:09:21.963838+0000","last_clean":"2024-08-17T18:09:21.963838+0000","last_became_active":"2024-08-17T18:06:57.358131+0000","last_became_peered":"2024-08-17T18:06:57.358131+0000","last_unstale":"2024-08-17T18:09:21.963838+0000","last_undegraded":"2024-08-17T18:09:21.963838+0000","last_fullsized":"2024-08-17T18:09:21.963838+0000","mapping_epoch":1460,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:54.303708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:54.303708+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:43:08.118481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"347'149","reported_seq":809,"reported_epoch":1535,"state":"active+clean","last_fresh":"2024-08-17T18:09:27.618745+0000","last_change":"2024-08-17T18:09:15.592722+0000","last_active":"2024-08-17T18:09:27.618745+0000","last_peered":"2024-08-17T18:09:27.618745+0000","last_clean":"2024-08-17T18:09:27.618745+0000","last_became_active":"2024-08-17T18:09:15.592242+0000","last_became_peered":"2024-08-17T18:09:15.592242+0000","last_unstale":"2024-08-17T18:09:27.618745+0000","last_undegraded":"2024-08-17T18:09:27.618745+0000","last_fullsized":"2024-08-17T18:09:27.618745+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:55.255983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:55.255983+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:02:53.596275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2358,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:22.093204+0000","last_change":"2024-08-17T17:24:17.455710+0000","last_active":"2024-08-17T18:09:22.093204+0000","last_peered":"2024-08-17T18:09:22.093204+0000","last_clean":"2024-08-17T18:09:22.093204+0000","last_became_active":"2024-08-17T17:24:17.452989+0000","last_became_peered":"2024-08-17T17:24:17.452989+0000","last_unstale":"2024-08-17T18:09:22.093204+0000","last_undegraded":"2024-08-17T18:09:22.093204+0000","last_fullsized":"2024-08-17T18:09:22.093204+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:58:07.894968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":62,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.915728+0000","last_change":"2024-08-17T18:08:11.347750+0000","last_active":"2024-08-17T18:09:21.915728+0000","last_peered":"2024-08-17T18:09:21.915728+0000","last_clean":"2024-08-17T18:09:21.915728+0000","last_became_active":"2024-08-17T18:08:11.346796+0000","last_became_peered":"2024-08-17T18:08:11.346796+0000","last_unstale":"2024-08-17T18:09:21.915728+0000","last_undegraded":"2024-08-17T18:09:21.915728+0000","last_fullsized":"2024-08-17T18:09:21.915728+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:24.470900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1531,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.193077+0000","last_change":"2024-08-17T18:08:37.068370+0000","last_active":"2024-08-17T18:09:21.193077+0000","last_peered":"2024-08-17T18:09:21.193077+0000","last_clean":"2024-08-17T18:09:21.193077+0000","last_became_active":"2024-08-17T18:08:37.068027+0000","last_became_peered":"2024-08-17T18:08:37.068027+0000","last_unstale":"2024-08-17T18:09:21.193077+0000","last_undegraded":"2024-08-17T18:09:21.193077+0000","last_fullsized":"2024-08-17T18:09:21.193077+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:02:12.661660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1539,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.192908+0000","last_change":"2024-08-17T18:03:51.270293+0000","last_active":"2024-08-17T18:09:21.192908+0000","last_peered":"2024-08-17T18:09:21.192908+0000","last_clean":"2024-08-17T18:09:21.192908+0000","last_became_active":"2024-08-17T18:03:51.269970+0000","last_became_peered":"2024-08-17T18:03:51.269970+0000","last_unstale":"2024-08-17T18:09:21.192908+0000","last_undegraded":"2024-08-17T18:09:21.192908+0000","last_fullsized":"2024-08-17T18:09:21.192908+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:02:30.016517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"347'149","reported_seq":809,"reported_epoch":1535,"state":"active+clean","last_fresh":"2024-08-17T18:09:27.618615+0000","last_change":"2024-08-17T18:09:15.590888+0000","last_active":"2024-08-17T18:09:27.618615+0000","last_peered":"2024-08-17T18:09:27.618615+0000","last_clean":"2024-08-17T18:09:27.618615+0000","last_became_active":"2024-08-17T18:09:15.590345+0000","last_became_peered":"2024-08-17T18:09:15.590345+0000","last_unstale":"2024-08-17T18:09:27.618615+0000","last_undegraded":"2024-08-17T18:09:27.618615+0000","last_fullsized":"2024-08-17T18:09:27.618615+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:59.307356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:59.307356+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:40:40.796012+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1543,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.193188+0000","last_change":"2024-08-17T18:01:23.497110+0000","last_active":"2024-08-17T18:09:21.193188+0000","last_peered":"2024-08-17T18:09:21.193188+0000","last_clean":"2024-08-17T18:09:21.193188+0000","last_became_active":"2024-08-17T18:01:23.496800+0000","last_became_peered":"2024-08-17T18:01:23.496800+0000","last_unstale":"2024-08-17T18:09:21.193188+0000","last_undegraded":"2024-08-17T18:09:21.193188+0000","last_fullsized":"2024-08-17T18:09:21.193188+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:32:29.892739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1511,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.591127+0000","last_change":"2024-08-17T17:23:07.401624+0000","last_active":"2024-08-17T18:09:15.591127+0000","last_peered":"2024-08-17T18:09:15.591127+0000","last_clean":"2024-08-17T18:09:15.591127+0000","last_became_active":"2024-08-17T17:23:07.400222+0000","last_became_peered":"2024-08-17T17:23:07.400222+0000","last_unstale":"2024-08-17T18:09:15.591127+0000","last_undegraded":"2024-08-17T18:09:15.591127+0000","last_fullsized":"2024-08-17T18:09:15.591127+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:07.069989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"347'149","reported_seq":808,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.915719+0000","last_change":"2024-08-17T18:09:15.592665+0000","last_active":"2024-08-17T18:09:21.915719+0000","last_peered":"2024-08-17T18:09:21.915719+0000","last_clean":"2024-08-17T18:09:21.915719+0000","last_became_active":"2024-08-17T18:09:15.590435+0000","last_became_peered":"2024-08-17T18:09:15.590435+0000","last_unstale":"2024-08-17T18:09:21.915719+0000","last_undegraded":"2024-08-17T18:09:21.915719+0000","last_fullsized":"2024-08-17T18:09:21.915719+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:52.271487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:52.271487+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:55:12.849191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1513,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.192883+0000","last_change":"2024-08-17T17:24:17.470809+0000","last_active":"2024-08-17T18:09:21.192883+0000","last_peered":"2024-08-17T18:09:21.192883+0000","last_clean":"2024-08-17T18:09:21.192883+0000","last_became_active":"2024-08-17T17:24:17.469753+0000","last_became_peered":"2024-08-17T17:24:17.469753+0000","last_unstale":"2024-08-17T18:09:21.192883+0000","last_undegraded":"2024-08-17T18:09:21.192883+0000","last_fullsized":"2024-08-17T18:09:21.192883+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:00.049425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":2367,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.915972+0000","last_change":"2024-08-17T18:04:14.552806+0000","last_active":"2024-08-17T18:09:21.915972+0000","last_peered":"2024-08-17T18:09:21.915972+0000","last_clean":"2024-08-17T18:09:21.915972+0000","last_became_active":"2024-08-17T18:04:14.535546+0000","last_became_peered":"2024-08-17T18:04:14.535546+0000","last_unstale":"2024-08-17T18:09:21.915972+0000","last_undegraded":"2024-08-17T18:09:21.915972+0000","last_fullsized":"2024-08-17T18:09:21.915972+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:44:45.702272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":166,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.964126+0000","last_change":"2024-08-17T18:04:14.539500+0000","last_active":"2024-08-17T18:09:21.964126+0000","last_peered":"2024-08-17T18:09:21.964126+0000","last_clean":"2024-08-17T18:09:21.964126+0000","last_became_active":"2024-08-17T18:04:14.538738+0000","last_became_peered":"2024-08-17T18:04:14.538738+0000","last_unstale":"2024-08-17T18:09:21.964126+0000","last_undegraded":"2024-08-17T18:09:21.964126+0000","last_fullsized":"2024-08-17T18:09:21.964126+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:31.439947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":84,"reported_epoch":1535,"state":"active+clean","last_fresh":"2024-08-17T18:09:27.619047+0000","last_change":"2024-08-17T18:04:43.771107+0000","last_active":"2024-08-17T18:09:27.619047+0000","last_peered":"2024-08-17T18:09:27.619047+0000","last_clean":"2024-08-17T18:09:27.619047+0000","last_became_active":"2024-08-17T18:04:43.770799+0000","last_became_peered":"2024-08-17T18:04:43.770799+0000","last_unstale":"2024-08-17T18:09:27.619047+0000","last_undegraded":"2024-08-17T18:09:27.619047+0000","last_fullsized":"2024-08-17T18:09:27.619047+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:09:04.350798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.590604+0000","last_change":"2024-08-17T18:08:37.080554+0000","last_active":"2024-08-17T18:09:15.590604+0000","last_peered":"2024-08-17T18:09:15.590604+0000","last_clean":"2024-08-17T18:09:15.590604+0000","last_became_active":"2024-08-17T18:08:37.079920+0000","last_became_peered":"2024-08-17T18:08:37.079920+0000","last_unstale":"2024-08-17T18:09:15.590604+0000","last_undegraded":"2024-08-17T18:09:15.590604+0000","last_fullsized":"2024-08-17T18:09:15.590604+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:08:25.214786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":88,"reported_epoch":1535,"state":"active+clean","last_fresh":"2024-08-17T18:09:27.619124+0000","last_change":"2024-08-17T18:04:43.781111+0000","last_active":"2024-08-17T18:09:27.619124+0000","last_peered":"2024-08-17T18:09:27.619124+0000","last_clean":"2024-08-17T18:09:27.619124+0000","last_became_active":"2024-08-17T18:04:43.771839+0000","last_became_peered":"2024-08-17T18:04:43.771839+0000","last_unstale":"2024-08-17T18:09:27.619124+0000","last_undegraded":"2024-08-17T18:09:27.619124+0000","last_fullsized":"2024-08-17T18:09:27.619124+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:52:45.973049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1547,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.192699+0000","last_change":"2024-08-17T18:08:11.449492+0000","last_active":"2024-08-17T18:09:21.192699+0000","last_peered":"2024-08-17T18:09:21.192699+0000","last_clean":"2024-08-17T18:09:21.192699+0000","last_became_active":"2024-08-17T18:08:11.424764+0000","last_became_peered":"2024-08-17T18:08:11.424764+0000","last_unstale":"2024-08-17T18:09:21.192699+0000","last_undegraded":"2024-08-17T18:09:21.192699+0000","last_fullsized":"2024-08-17T18:09:21.192699+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:53:37.552880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":185,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:20.607160+0000","last_change":"2024-08-17T18:04:14.551061+0000","last_active":"2024-08-17T18:09:20.607160+0000","last_peered":"2024-08-17T18:09:20.607160+0000","last_clean":"2024-08-17T18:09:20.607160+0000","last_became_active":"2024-08-17T18:04:14.543296+0000","last_became_peered":"2024-08-17T18:04:14.543296+0000","last_unstale":"2024-08-17T18:09:20.607160+0000","last_undegraded":"2024-08-17T18:09:20.607160+0000","last_fullsized":"2024-08-17T18:09:20.607160+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:39:45.294265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":102,"reported_epoch":1535,"state":"active+clean","last_fresh":"2024-08-17T18:09:27.618694+0000","last_change":"2024-08-17T18:08:37.068503+0000","last_active":"2024-08-17T18:09:27.618694+0000","last_peered":"2024-08-17T18:09:27.618694+0000","last_clean":"2024-08-17T18:09:27.618694+0000","last_became_active":"2024-08-17T18:08:37.068110+0000","last_became_peered":"2024-08-17T18:08:37.068110+0000","last_unstale":"2024-08-17T18:09:27.618694+0000","last_undegraded":"2024-08-17T18:09:27.618694+0000","last_fullsized":"2024-08-17T18:09:27.618694+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:12:30.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":128,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.193039+0000","last_change":"2024-08-17T18:04:14.547798+0000","last_active":"2024-08-17T18:09:21.193039+0000","last_peered":"2024-08-17T18:09:21.193039+0000","last_clean":"2024-08-17T18:09:21.193039+0000","last_became_active":"2024-08-17T18:04:14.527496+0000","last_became_peered":"2024-08-17T18:04:14.527496+0000","last_unstale":"2024-08-17T18:09:21.193039+0000","last_undegraded":"2024-08-17T18:09:21.193039+0000","last_fullsized":"2024-08-17T18:09:21.193039+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:25:54.338846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1593,"reported_epoch":1533,"state":"active+clean","last_fresh":"2024-08-17T18:09:15.589508+0000","last_change":"2024-08-17T18:04:43.813536+0000","last_active":"2024-08-17T18:09:15.589508+0000","last_peered":"2024-08-17T18:09:15.589508+0000","last_clean":"2024-08-17T18:09:15.589508+0000","last_became_active":"2024-08-17T18:04:43.799823+0000","last_became_peered":"2024-08-17T18:04:43.799823+0000","last_unstale":"2024-08-17T18:09:15.589508+0000","last_undegraded":"2024-08-17T18:09:15.589508+0000","last_fullsized":"2024-08-17T18:09:15.589508+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:04:10.837113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":62,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.916236+0000","last_change":"2024-08-17T18:08:11.345061+0000","last_active":"2024-08-17T18:09:21.916236+0000","last_peered":"2024-08-17T18:09:21.916236+0000","last_clean":"2024-08-17T18:09:21.916236+0000","last_became_active":"2024-08-17T18:08:11.344755+0000","last_became_peered":"2024-08-17T18:08:11.344755+0000","last_unstale":"2024-08-17T18:09:21.916236+0000","last_undegraded":"2024-08-17T18:09:21.916236+0000","last_fullsized":"2024-08-17T18:09:21.916236+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:23:32.161453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":252,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.192800+0000","last_change":"2024-08-17T18:04:14.527380+0000","last_active":"2024-08-17T18:09:21.192800+0000","last_peered":"2024-08-17T18:09:21.192800+0000","last_clean":"2024-08-17T18:09:21.192800+0000","last_became_active":"2024-08-17T18:04:14.527076+0000","last_became_peered":"2024-08-17T18:04:14.527076+0000","last_unstale":"2024-08-17T18:09:21.192800+0000","last_undegraded":"2024-08-17T18:09:21.192800+0000","last_fullsized":"2024-08-17T18:09:21.192800+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:36:35.583709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":166,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.964337+0000","last_change":"2024-08-17T18:04:14.541814+0000","last_active":"2024-08-17T18:09:21.964337+0000","last_peered":"2024-08-17T18:09:21.964337+0000","last_clean":"2024-08-17T18:09:21.964337+0000","last_became_active":"2024-08-17T18:04:14.541403+0000","last_became_peered":"2024-08-17T18:04:14.541403+0000","last_unstale":"2024-08-17T18:09:21.964337+0000","last_undegraded":"2024-08-17T18:09:21.964337+0000","last_fullsized":"2024-08-17T18:09:21.964337+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:07:45.294924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":180,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.964053+0000","last_change":"2024-08-17T18:08:11.351307+0000","last_active":"2024-08-17T18:09:21.964053+0000","last_peered":"2024-08-17T18:09:21.964053+0000","last_clean":"2024-08-17T18:09:21.964053+0000","last_became_active":"2024-08-17T18:08:11.350926+0000","last_became_peered":"2024-08-17T18:08:11.350926+0000","last_unstale":"2024-08-17T18:09:21.964053+0000","last_undegraded":"2024-08-17T18:09:21.964053+0000","last_fullsized":"2024-08-17T18:09:21.964053+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:59:57.477492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":84,"reported_epoch":1536,"state":"active+clean","last_fresh":"2024-08-17T18:09:28.621657+0000","last_change":"2024-08-17T18:04:43.771720+0000","last_active":"2024-08-17T18:09:28.621657+0000","last_peered":"2024-08-17T18:09:28.621657+0000","last_clean":"2024-08-17T18:09:28.621657+0000","last_became_active":"2024-08-17T18:04:43.771137+0000","last_became_peered":"2024-08-17T18:04:43.771137+0000","last_unstale":"2024-08-17T18:09:28.621657+0000","last_undegraded":"2024-08-17T18:09:28.621657+0000","last_fullsized":"2024-08-17T18:09:28.621657+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:57.647909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":117,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.192876+0000","last_change":"2024-08-17T18:03:51.283872+0000","last_active":"2024-08-17T18:09:21.192876+0000","last_peered":"2024-08-17T18:09:21.192876+0000","last_clean":"2024-08-17T18:09:21.192876+0000","last_became_active":"2024-08-17T18:03:51.267197+0000","last_became_peered":"2024-08-17T18:03:51.267197+0000","last_unstale":"2024-08-17T18:09:21.192876+0000","last_undegraded":"2024-08-17T18:09:21.192876+0000","last_fullsized":"2024-08-17T18:09:21.192876+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:18:59.157786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2356,"reported_epoch":1534,"state":"active+clean","last_fresh":"2024-08-17T18:09:21.916097+0000","last_change":"2024-08-17T18:03:51.277491+0000","last_active":"2024-08-17T18:09:21.916097+0000","last_peered":"2024-08-17T18:09:21.916097+0000","last_clean":"2024-08-17T18:09:21.916097+0000","last_became_active":"2024-08-17T18:03:51.277178+0000","last_became_peered":"2024-08-17T18:03:51.277178+0000","last_unstale":"2024-08-17T18:09:21.916097+0000","last_undegraded":"2024-08-17T18:09:21.916097+0000","last_fullsized":"2024-08-17T18:09:21.916097+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:21:35.319085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":56,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":2},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":697708,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":150,"num_write":39,"num_write_kb":440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1127066,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":7994,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234721,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496359,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":769840,"kb_used_data":12772,"kb_used_omap":1,"kb_used_meta":757054,"kb_avail":92975312,"statfs":{"total":95995035648,"available":95206719488,"internally_reserved":0,"allocated":13078528,"data_stored":7627884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":775223751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565600,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":925408,"kb_used_data":139416,"kb_used_omap":72,"kb_used_meta":785911,"kb_avail":92819744,"statfs":{"total":95995035648,"available":95047417856,"internally_reserved":0,"allocated":142761984,"data_stored":137319465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":804773358},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":1399,"seq":6008659247202,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":279472,"kb_used_data":12708,"kb_used_omap":131,"kb_used_meta":266620,"kb_avail":93465680,"statfs":{"total":95995035648,"available":95708856320,"internally_reserved":0,"allocated":13012992,"data_stored":7563397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134291,"internal_metadata":273019757},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743186990,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":688220,"kb_used_data":34260,"kb_used_omap":66,"kb_used_meta":653885,"kb_avail":93056932,"statfs":{"total":95995035648,"available":95290298368,"internally_reserved":0,"allocated":35082240,"data_stored":29643154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":669578923},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448629,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":445540,"kb_used_data":12440,"kb_used_omap":75,"kb_used_meta":433012,"kb_avail":93299612,"statfs":{"total":95995035648,"available":95538802688,"internally_reserved":0,"allocated":12738560,"data_stored":7296031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":443404479},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757994,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":814476,"kb_used_data":13052,"kb_used_omap":1,"kb_used_meta":801406,"kb_avail":92930676,"statfs":{"total":95995035648,"available":95161012224,"internally_reserved":0,"allocated":13365248,"data_stored":7904350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":820640198},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019632,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":671088,"kb_used_data":12452,"kb_used_omap":1,"kb_used_meta":658622,"kb_avail":93074064,"statfs":{"total":95995035648,"available":95307841536,"internally_reserved":0,"allocated":12750848,"data_stored":7300227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":674429382},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T18:09:32.437 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph status --format=json 2024-08-17T18:09:32.514 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:32.535 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:32.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:32.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:32.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:32.662 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:32.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:32.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:32.764 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:32.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:32.787 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:32.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:32.888 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:32.913 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:32.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:32.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:33.015 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:33.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:33.037 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:33.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:33.138 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:33.160 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:33.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:33.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:33.285 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:33.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:33.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:33.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:33.410 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:33.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:33.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:33.512 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:33.534 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:33.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:33.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:33.636 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:33.659 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:33.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:33.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:33.761 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:33.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:33.783 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:33.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:33.884 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:33.908 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:33.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:33.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:33.944 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:33 smithi136 ceph-mon[109861]: pgmap v2799: 99 pgs: 1 creating+peering, 98 active+clean; 687 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:33.944 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:33 smithi136 ceph-mon[115078]: pgmap v2799: 99 pgs: 1 creating+peering, 98 active+clean; 687 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:34.009 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:33 smithi165 ceph-mon[105517]: pgmap v2799: 99 pgs: 1 creating+peering, 98 active+clean; 687 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:34.031 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:34.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:34.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:34.133 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:34.155 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:34.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:34.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:34.256 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:34.279 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:34.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:34.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:34.381 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:34.402 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:34.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:34.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:34.504 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:34.530 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:34.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:34.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:34.632 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:34.658 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:34.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:34.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:34.759 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:34.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:34.782 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:34.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:34.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:34.906 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:34.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:34.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:35.008 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:35.029 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:35.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:35.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:35.131 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:35.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:35.153 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:35.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:35.254 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:35.278 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:35.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:35.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:35.380 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:35.402 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:35.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:35.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:35.504 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:35.525 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:09:35.525 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:09:35.527 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:35.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:35.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:35.536 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:09:35.537 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:09:35.554 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:09:35.554 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:09:35.577 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:09:35.577 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:09:35.597 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:09:35.597 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:09:35.613 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:09:35.613 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:09:35.629 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:35.630 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:09:35.630 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:09:35.631 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:09:35.650 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:09:35.651 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:09:35.653 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:35.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:35.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:35.755 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:35.777 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:35.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:35.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:35.833 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:35 smithi136 ceph-mon[115078]: pgmap v2800: 99 pgs: 99 active+clean; 687 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:35.833 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:35 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1688155126' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:09:35.833 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:35 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:09:35.833 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:35 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4001667761' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:09:35.833 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:35 smithi136 ceph-mon[109861]: pgmap v2800: 99 pgs: 99 active+clean; 687 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:35.833 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:35 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1688155126' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:09:35.833 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:35 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:09:35.833 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:35 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4001667761' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:09:35.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:35.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:35.902 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:35.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:35.918 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:35 smithi165 ceph-mon[105517]: pgmap v2800: 99 pgs: 99 active+clean; 687 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:35.918 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:35 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1688155126' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:09:35.918 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:35 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:09:35.918 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:35 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4001667761' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:09:36.004 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:36.026 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:36.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:36.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:36.127 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:36.149 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:36.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:36.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:36.251 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:36.273 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:36.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:36.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:36.375 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:36.396 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:36.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:36.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:36.498 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:36.519 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:36.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:36.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:36.621 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:36.643 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:36.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:36.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:36.744 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:36.767 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:36.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:36.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:36.791 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:36 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:09:36.791 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:36 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1688155126' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:09:36.791 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:36 smithi136 ceph-mon[109861]: osdmap e1538: 8 total, 8 up, 7 in 2024-08-17T18:09:36.791 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:09:36.791 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:09:36.791 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:36 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:09:36.792 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:36 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1688155126' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:09:36.792 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:36 smithi136 ceph-mon[115078]: osdmap e1538: 8 total, 8 up, 7 in 2024-08-17T18:09:36.792 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:09:36.792 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:09:36.868 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:36.891 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:36.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:36.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:36.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:37.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:36 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:09:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:36 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1688155126' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:09:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:36 smithi165 ceph-mon[105517]: osdmap e1538: 8 total, 8 up, 7 in 2024-08-17T18:09:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:09:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:09:37.014 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:37.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:37.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:37.027 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:09:37.027 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:09:37.116 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:37.138 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:37.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:37.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:37.240 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:37.262 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:37.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:37.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:37.364 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:37.385 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:37.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:37.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:37.487 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:37.509 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:37.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:37.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:37.611 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:37.634 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:37.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:37.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:37.649 INFO:teuthology.orchestra.run.smithi136.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-08-17T18:09:37.736 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:37.757 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:37.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:37.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:37.859 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:37.881 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:37.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:37.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:37.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:37 smithi136 ceph-mon[109861]: pgmap v2802: 99 pgs: 99 active+clean; 687 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:37.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:37 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/12893934' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T18:09:37.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:37 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T18:09:37.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:09:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:09:37] "GET /metrics HTTP/1.1" 200 36548 "" "Prometheus/2.51.0" 2024-08-17T18:09:37.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:37 smithi136 ceph-mon[115078]: pgmap v2802: 99 pgs: 99 active+clean; 687 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:37.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:37 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/12893934' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T18:09:37.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:37 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T18:09:37.982 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:38.005 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:38.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:38.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:38.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:37 smithi165 ceph-mon[105517]: pgmap v2802: 99 pgs: 99 active+clean; 687 KiB data, 4.4 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:37 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/12893934' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T18:09:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:37 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T18:09:38.106 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:38.129 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:38.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:38.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:38.231 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:38.253 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:38.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:38.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:38.354 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:38.376 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:38.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:38.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:38.478 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:38.489 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:09:38.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:38.500 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:38.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:38.602 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:38.622 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:38.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:38.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:38.724 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:38.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:38.747 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:38.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:38.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:38 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T18:09:38.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:38 smithi165 ceph-mon[105517]: osdmap e1539: 8 total, 8 up, 7 in 2024-08-17T18:09:38.848 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:38.870 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:38.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:38.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:38.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:38 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T18:09:38.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:38 smithi136 ceph-mon[109861]: osdmap e1539: 8 total, 8 up, 7 in 2024-08-17T18:09:38.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:38 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T18:09:38.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:38 smithi136 ceph-mon[115078]: osdmap e1539: 8 total, 8 up, 7 in 2024-08-17T18:09:38.972 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:38.994 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:38.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:38.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:39.098 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:39.119 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:39.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:39.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:39.221 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:39.242 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:39.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:39.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:39.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:39.366 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:39.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:39.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:39.388 DEBUG:teuthology.orchestra.run.smithi136:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-08-17T18:09:39.390 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-17T18:09:39.390 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-17T18:09:39.439 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:hints = 1 2024-08-17T18:09:39.439 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-17T18:09:39.439 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Object prefix: benchmark_data_smithi136_356456 2024-08-17T18:09:39.439 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T18:09:39.439 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 0 0 0 0 0 0 - 0 2024-08-17T18:09:39.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:39.505 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:39.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:39.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:39.609 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:39.638 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:09:39.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:39.651 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:39.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:39.752 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:39.789 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:39.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:39.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:39.891 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:39.930 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:39.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:39.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:39.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:39 smithi136 ceph-mon[109861]: pgmap v2804: 99 pgs: 99 active+clean; 687 KiB data, 4.3 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:39.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:39 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T18:09:39.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:39 smithi136 ceph-mon[115078]: pgmap v2804: 99 pgs: 99 active+clean; 687 KiB data, 4.3 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:39.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:39 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T18:09:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:39 smithi165 ceph-mon[105517]: pgmap v2804: 99 pgs: 99 active+clean; 687 KiB data, 4.3 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:39 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T18:09:40.032 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:40.070 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:40.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:40.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:40.173 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:40.209 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:40.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:40.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:40.311 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:40.349 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:40.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:40.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:40.439 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 1 16 3496 3480 217.491 217.5 0.00273999 0.00458033 2024-08-17T18:09:40.451 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:40.486 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:40.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:40.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:40.588 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:40.626 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:40.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:40.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:40.713 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:09:40.731 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:40.763 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:40.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:40.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:40.864 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:40.903 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:40.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:40.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:40.931 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:40 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:09:40.931 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:40 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:09:40.931 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:40 smithi136 ceph-mon[109861]: pgmap v2805: 99 pgs: 99 active+clean; 687 KiB data, 4.3 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:40.931 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:40 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:09:40.931 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:40 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:09:40.932 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:40 smithi136 ceph-mon[115078]: pgmap v2805: 99 pgs: 99 active+clean; 687 KiB data, 4.3 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:41.005 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:41.037 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:41.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:41.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:41.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:41.170 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:41.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:41.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:40 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:09:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:40 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:09:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:40 smithi165 ceph-mon[105517]: pgmap v2805: 99 pgs: 99 active+clean; 687 KiB data, 4.3 GiB used, 621 GiB / 626 GiB avail 2024-08-17T18:09:41.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:41.305 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:41.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:41.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:41.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:41.439 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 2 16 6910 6894 215.422 213.375 0.00324696 0.00463157 2024-08-17T18:09:41.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:41.446 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:41.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:41.547 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:41.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:41.586 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:41.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:41.687 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:41.719 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:41.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:41.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:41.821 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:41.853 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:41.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:41.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:41.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:41.984 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:41.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:41.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:42.086 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:42.117 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:42.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:42.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:42.219 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:42.251 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:42.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:42.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:42.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:42.386 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:42.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:42.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:42.439 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 3 16 10126 10110 210.609 201 0.00337757 0.00474094 2024-08-17T18:09:42.488 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:42.520 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:42.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:42.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:42.622 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:42.654 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:42.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:42.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:42.756 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:42.794 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:42.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:42.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:42.896 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:42.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:42.922 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:42.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:43.023 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:43.054 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:43.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:43.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:43.156 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:43.185 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:43.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:43.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:43.232 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:09:43.232 INFO:teuthology.orchestra.run.smithi136.stdout:{"fsid":"8270436c-5cba-11ef-bccf-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false}},"mutes":[]},"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":3079,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":1539,"num_osds":8,"num_up_osds":8,"osd_up_since":1723917682,"num_in_osds":7,"osd_in_since":1723917882,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":99}],"num_pgs":99,"num_pools":5,"num_objects":2712,"data_bytes":176732977,"bytes_used":5366210560,"bytes_avail":666599038976,"bytes_total":671965249536,"write_bytes_sec":21996418,"read_op_per_sec":0,"write_op_per_sec":335},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi136.ltsuqd","status":"up:active","gid":24557}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.136:8443/","prometheus":"http://172.21.15.136:9283/"}},"servicemap":{"epoch":38,"modified":"2024-08-17T18:09:06.294281+0000","services":{}},"progress_events":{}} 2024-08-17T18:09:43.287 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:43.318 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:43.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:43.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:43.356 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:09:43.356 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T18:09:43.356 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-17T18:09:43.356 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-17T18:09:43.356 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:09:43.419 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:43.439 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 4 16 12976 12960 202.484 178.125 0.00500783 0.00493327 2024-08-17T18:09:43.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:43 smithi136 ceph-mon[109861]: pgmap v2806: 99 pgs: 99 active+clean; 169 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 21 MiB/s wr, 335 op/s 2024-08-17T18:09:43.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:43 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/19728953' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T18:09:43.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:43 smithi136 ceph-mon[115078]: pgmap v2806: 99 pgs: 99 active+clean; 169 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 21 MiB/s wr, 335 op/s 2024-08-17T18:09:43.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:43 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/19728953' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T18:09:43.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:43.458 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:43.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:43.559 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:43.598 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:43.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:43.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:43.700 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:43.737 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:43.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:43.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:43 smithi165 ceph-mon[105517]: pgmap v2806: 99 pgs: 99 active+clean; 169 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 21 MiB/s wr, 335 op/s 2024-08-17T18:09:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:43 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/19728953' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T18:09:43.839 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:43.876 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:43.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:43.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:43.977 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:44.015 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:44.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:44.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:44.117 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:44.157 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:44.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:44.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:44.259 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:44.305 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:44.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:44.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:44.406 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:44.440 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 5 16 16000 15984 199.784 189 0.00524063 0.00499903 2024-08-17T18:09:44.446 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:44.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:44.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:44.548 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:44.588 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:44.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:44.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:44.689 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:44.726 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:44.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:44.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:44.828 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:44.866 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:44.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:44.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:44.968 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:45.006 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:45.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:45.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:45.107 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:45.138 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:45.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:45.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:45.240 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:45.276 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:45.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:45.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:45.377 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:45.415 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:45.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:45.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:45.439 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 6 16 18882 18866 196.504 180.125 0.00523718 0.0050831 2024-08-17T18:09:45.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:45 smithi136 ceph-mon[109861]: pgmap v2807: 99 pgs: 99 active+clean; 511 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 59 MiB/s wr, 941 op/s 2024-08-17T18:09:45.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:45 smithi136 ceph-mon[115078]: pgmap v2807: 99 pgs: 99 active+clean; 511 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 59 MiB/s wr, 941 op/s 2024-08-17T18:09:45.517 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:45.552 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:45.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:45.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:45.653 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:45.684 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:45.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:45.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:45.727 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:09:45.752 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-08-17T18:09:45.753 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T18:09:45.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:45 smithi165 ceph-mon[105517]: pgmap v2807: 99 pgs: 99 active+clean; 511 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 59 MiB/s wr, 941 op/s 2024-08-17T18:09:45.786 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:45.815 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:45.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:45.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:45.917 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:45.948 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:45.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:45.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:46.050 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:46.091 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:46.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:46.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:46.193 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:46.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:46.244 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:46.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:46.345 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:46.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:46.385 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:46.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:46.440 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 7 16 21459 21443 191.439 161.062 0.00411098 0.0052182 2024-08-17T18:09:46.486 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:46.525 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:46.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:46.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:46.627 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:46.665 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:46.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:46.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:46.767 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:46.792 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-17T18:09:46.808 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:46.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:46.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:46.910 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:46.940 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:46.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:46.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:47.042 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:47.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:47.108 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:47.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:47.209 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:47.253 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:47.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:47.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:47.355 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:47.384 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:47.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:47.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:47.440 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 8 16 23758 23742 185.468 143.688 0.00592447 0.00538616 2024-08-17T18:09:47.486 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:47.516 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:47.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:47.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:47.618 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:47.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:47.657 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:47.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:47 smithi136 ceph-mon[109861]: pgmap v2808: 99 pgs: 99 active+clean; 1.0 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 105 MiB/s wr, 1.67k op/s 2024-08-17T18:09:47.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:09:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:09:47] "GET /metrics HTTP/1.1" 200 36548 "" "Prometheus/2.51.0" 2024-08-17T18:09:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:47 smithi136 ceph-mon[115078]: pgmap v2808: 99 pgs: 99 active+clean; 1.0 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 105 MiB/s wr, 1.67k op/s 2024-08-17T18:09:47.757 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:47 smithi165 ceph-mon[105517]: pgmap v2808: 99 pgs: 99 active+clean; 1.0 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 105 MiB/s wr, 1.67k op/s 2024-08-17T18:09:47.790 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:47.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:47.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:47.892 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:47.924 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:47.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:47.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:48.026 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:48.069 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:48.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:48.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:48.170 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:48.214 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:48.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:48.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:48.316 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:48.342 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:48.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:48.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:48.440 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 9 16 25680 25664 178.207 120.125 0.00374136 0.0056071 2024-08-17T18:09:48.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:48.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:48.479 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:48.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:48.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:48.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:48.610 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:48.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:48.711 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:48.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:48.750 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:48.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:48.851 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:48.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:48.893 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:48.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:48.994 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:49.028 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:49.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:49.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:49.130 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:49.162 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:49.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:49.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:49.264 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:49.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:49.289 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:49.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:49.387 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:09:49.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:49.417 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:49.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:49.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:49.440 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 10 16 27968 27952 174.685 143 0.0052455 0.00571886 2024-08-17T18:09:49.518 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:49.540 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:49.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:49.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:49.642 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:49.679 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:49.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:49.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:49 smithi136 ceph-mon[109861]: pgmap v2809: 99 pgs: 99 active+clean; 1.1 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-08-17T18:09:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:49 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1185906948' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:09:49.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:49 smithi136 ceph-mon[115078]: pgmap v2809: 99 pgs: 99 active+clean; 1.1 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-08-17T18:09:49.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:49 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1185906948' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:09:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:49 smithi165 ceph-mon[105517]: pgmap v2809: 99 pgs: 99 active+clean; 1.1 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-08-17T18:09:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:49 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1185906948' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:09:49.781 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:49.816 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:49.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:49.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:49.918 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:49.953 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:49.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:49.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:50.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:50.093 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:50.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:50.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:50.194 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:50.220 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:50.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:50.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:50.322 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:50.343 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:50.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:50.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:50.440 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 11 16 29353 29337 166.673 86.5625 0.00333167 0.0059938 2024-08-17T18:09:50.445 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:50.484 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:50.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:50.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:50.585 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:50.610 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:50.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:50.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:50.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:50 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:09:50.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:50 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1185906948' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:09:50.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:50 smithi136 ceph-mon[109861]: osdmap e1540: 8 total, 8 up, 7 in 2024-08-17T18:09:50.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:50 smithi136 ceph-mon[109861]: 1.7 scrub starts 2024-08-17T18:09:50.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:50 smithi136 ceph-mon[109861]: 1.7 scrub ok 2024-08-17T18:09:50.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:50 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:09:50.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:50 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1185906948' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:09:50.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:50 smithi136 ceph-mon[115078]: osdmap e1540: 8 total, 8 up, 7 in 2024-08-17T18:09:50.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:50 smithi136 ceph-mon[115078]: 1.7 scrub starts 2024-08-17T18:09:50.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:50 smithi136 ceph-mon[115078]: 1.7 scrub ok 2024-08-17T18:09:50.712 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:50.752 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:50.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:50.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:50.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:50 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:09:50.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:50 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1185906948' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:09:50.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:50 smithi165 ceph-mon[105517]: osdmap e1540: 8 total, 8 up, 7 in 2024-08-17T18:09:50.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:50 smithi165 ceph-mon[105517]: 1.7 scrub starts 2024-08-17T18:09:50.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:50 smithi165 ceph-mon[105517]: 1.7 scrub ok 2024-08-17T18:09:50.854 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:50.890 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:50.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:50.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:50.992 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:51.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:51.024 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:51.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:51.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:51.157 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:51.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:51.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:51.259 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:51.298 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:51.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:51.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:51.401 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:51.439 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:51.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:51.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:51.440 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 12 16 31404 31388 163.465 128.188 0.00751848 0.00611147 2024-08-17T18:09:51.541 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:51.574 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:51.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:51.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:51.677 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:51 smithi136 ceph-mon[109861]: pgmap v2811: 99 pgs: 99 active+clean; 1.4 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 139 MiB/s wr, 2.22k op/s 2024-08-17T18:09:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:09:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:09:51.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:51 smithi136 ceph-mon[115078]: pgmap v2811: 99 pgs: 99 active+clean; 1.4 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 139 MiB/s wr, 2.22k op/s 2024-08-17T18:09:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:09:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:09:51.709 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:51.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:51.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:51 smithi165 ceph-mon[105517]: pgmap v2811: 99 pgs: 99 active+clean; 1.4 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 139 MiB/s wr, 2.22k op/s 2024-08-17T18:09:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:09:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:09:51.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:51.843 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:51.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:51.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:51.945 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:51.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:51.981 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:51.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:52.082 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:52.114 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:52.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:52.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:52.216 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:52.248 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:52.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:52.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:52.350 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:52.376 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:52.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:52.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:52.440 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 13 16 33157 33141 159.318 109.562 0.00735125 0.00627151 2024-08-17T18:09:52.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:52.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:52.514 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:52.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:52.615 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:52.636 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:52.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:52.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:52.738 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:52.775 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:52.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:52.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:52.877 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:52.898 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:52.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:52.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:52.999 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:53.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:53.033 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:53.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:53.134 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:53.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:53.171 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:53.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:53.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:53.309 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:53.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:53.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:53.410 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:53.440 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 14 16 34897 34881 155.705 108.75 0.00867487 0.00641668 2024-08-17T18:09:53.441 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:53.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:53.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:53.543 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:53.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:53.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:53.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:53.675 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:53.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:53 smithi136 ceph-mon[115078]: pgmap v2812: 99 pgs: 99 active+clean; 1.8 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 162 MiB/s wr, 2.59k op/s 2024-08-17T18:09:53.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:53 smithi136 ceph-mon[115078]: 1.6 scrub starts 2024-08-17T18:09:53.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:53 smithi136 ceph-mon[115078]: 1.6 scrub ok 2024-08-17T18:09:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:53 smithi136 ceph-mon[109861]: pgmap v2812: 99 pgs: 99 active+clean; 1.8 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 162 MiB/s wr, 2.59k op/s 2024-08-17T18:09:53.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:53 smithi136 ceph-mon[109861]: 1.6 scrub starts 2024-08-17T18:09:53.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:53 smithi136 ceph-mon[109861]: 1.6 scrub ok 2024-08-17T18:09:53.710 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:53.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:53.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:53 smithi165 ceph-mon[105517]: pgmap v2812: 99 pgs: 99 active+clean; 1.8 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 162 MiB/s wr, 2.59k op/s 2024-08-17T18:09:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:53 smithi165 ceph-mon[105517]: 1.6 scrub starts 2024-08-17T18:09:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:53 smithi165 ceph-mon[105517]: 1.6 scrub ok 2024-08-17T18:09:53.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:53.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:53.846 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:53.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:53.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:53.972 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:53.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:53.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:54.074 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:54.108 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:54.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:54.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:54.210 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:54.236 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:54.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:54.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:54.338 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:54.375 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:54.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:54.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:54.441 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 15 16 36838 36822 153.412 121.312 0.00831608 0.00651293 2024-08-17T18:09:54.477 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:54.509 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:54.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:54.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:54.612 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:54.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:54.668 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:54.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:54.769 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:54.802 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:54.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:54.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:54.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:54.925 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:54.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:54.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:55.027 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:55.068 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:55.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:55.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:55.170 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:55.210 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:55.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:55.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:55.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:55.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:55.350 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:55.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:55.440 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 16 16 38652 38636 150.909 113.375 0.00996988 0.00662091 2024-08-17T18:09:55.451 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:55.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:55.492 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:55.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:55.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:55.621 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:55.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:55.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:55 smithi136 ceph-mon[109861]: pgmap v2813: 99 pgs: 99 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-08-17T18:09:55.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:55 smithi136 ceph-mon[115078]: pgmap v2813: 99 pgs: 99 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-08-17T18:09:55.723 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:55.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:55 smithi165 ceph-mon[105517]: pgmap v2813: 99 pgs: 99 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-08-17T18:09:55.763 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:55.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:55.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:55.865 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:55.889 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:55.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:55.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:55.991 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:56.028 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:56.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:56.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:56.129 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:56.164 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:56.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:56.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:56.266 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:56.306 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:56.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:56.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:56.409 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:56.440 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:56.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:56.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:56.441 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 17 16 40607 40591 149.219 122.188 0.00656986 0.00669617 2024-08-17T18:09:56.542 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:56.574 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:56.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:56.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:56.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:56.709 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:56.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:56.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:56.810 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:56.841 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:56.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:56.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:56.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:09:56.980 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:56.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:56.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:09:57.082 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:57.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:57.116 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:57.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:57.217 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:09:57.256 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:57.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:57.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:09:57.357 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:57.394 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:57.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:57.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:57.441 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 18 16 42647 42631 148.011 127.5 0.0111266 0.00675111 2024-08-17T18:09:57.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:57.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:57.530 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:57.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:57.631 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:57.665 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:57.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:57.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:57.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:57 smithi136 ceph-mon[109861]: pgmap v2814: 99 pgs: 99 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 128 MiB/s wr, 2.05k op/s 2024-08-17T18:09:57.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:57 smithi136 ceph-mon[115078]: pgmap v2814: 99 pgs: 99 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 128 MiB/s wr, 2.05k op/s 2024-08-17T18:09:57.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:09:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:09:57] "GET /metrics HTTP/1.1" 200 36612 "" "Prometheus/2.51.0" 2024-08-17T18:09:57.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:57 smithi165 ceph-mon[105517]: pgmap v2814: 99 pgs: 99 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 128 MiB/s wr, 2.05k op/s 2024-08-17T18:09:57.767 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:57.797 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:57.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:57.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:57.899 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:57.930 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:57.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:57.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:58.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:58.059 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:09:58.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:58.061 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:58.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:58.063 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:09:58.143 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:09:58.143 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:09:58.161 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:58.186 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:09:58.187 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:09:58.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:58.221 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:58.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:58.222 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:09:58.222 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:09:58.256 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:09:58.256 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:09:58.288 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:09:58.289 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:09:58.322 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:58.323 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:09:58.323 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:09:58.351 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:58.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:58.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:58.359 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:09:58.359 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:09:58.393 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:09:58.393 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:09:58.441 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 19 16 44331 44315 145.76 105.25 0.00772021 0.0068552 2024-08-17T18:09:58.453 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:58.485 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:58.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:58.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:58.587 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:58.625 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:58.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:58.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:58.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:58 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2279483091' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:09:58.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:58 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2279483091' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:09:58.727 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:58.760 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:58 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2279483091' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:09:58.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:58.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:58.862 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:09:58.900 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:58.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:58.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:09:59.002 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:59.034 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:59.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:59.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:59.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:59.165 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:59.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:59.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:59.267 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:09:59.300 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:59.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:59.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:09:59.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:09:59.435 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:59.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:59.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:09:59.441 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:2024-08-17T18:09:59.441424+0000 min lat: 0.00107817 max lat: 0.220609 avg lat: 0.00692044 2024-08-17T18:09:59.441 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T18:09:59.441 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 20 16 46232 46216 144.412 118.812 0.00553711 0.00692044 2024-08-17T18:09:59.510 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:09:59.536 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:59.555 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:09:59 smithi136 ceph-mon[109861]: pgmap v2815: 99 pgs: 99 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 126 MiB/s wr, 2.00k op/s 2024-08-17T18:09:59.556 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:09:59 smithi136 ceph-mon[115078]: pgmap v2815: 99 pgs: 99 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 126 MiB/s wr, 2.00k op/s 2024-08-17T18:09:59.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:59.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:59.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:09:59.674 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:09:59.725 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:09:59.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:59.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:09:59.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:09:59 smithi165 ceph-mon[105517]: pgmap v2815: 99 pgs: 99 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 126 MiB/s wr, 2.00k op/s 2024-08-17T18:09:59.827 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:09:59.855 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:59.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:59.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:09:59.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:09:59.987 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:09:59.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:09:59.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:00.089 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:00.119 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:00.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:00.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:00.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:00.258 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:00.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:00.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:00.360 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:00.388 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:00.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:00.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:00.441 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 21 16 47923 47907 142.568 105.688 0.00678637 0.00700875 2024-08-17T18:10:00.490 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:00.527 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:00.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:00.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:00.629 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:00.662 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:00.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:00.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:00.764 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:00.802 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:00.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:00.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:00.894 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:00 smithi136 ceph-mon[109861]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-08-17T18:10:00.894 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:00 smithi136 ceph-mon[109861]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-08-17T18:10:00.894 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:00 smithi136 ceph-mon[115078]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-08-17T18:10:00.894 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:00 smithi136 ceph-mon[115078]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-08-17T18:10:00.904 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:00.942 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:00.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:00.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:00 smithi165 ceph-mon[105517]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-08-17T18:10:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:00 smithi165 ceph-mon[105517]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-08-17T18:10:01.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:01.075 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:01.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:01.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:01.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:01.210 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:01.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:01.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:01.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:01.364 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:01.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:01.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:01.441 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 22 16 49746 49730 141.266 113.938 0.00806089 0.0070744 2024-08-17T18:10:01.467 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:01.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:01.502 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:01.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:01.603 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:01.643 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:01.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:01.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:01.745 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:01.775 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:01.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:01.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:01.876 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:01 smithi136 ceph-mon[109861]: pgmap v2816: 99 pgs: 99 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-08-17T18:10:01.877 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:01 smithi136 ceph-mon[115078]: pgmap v2816: 99 pgs: 99 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-08-17T18:10:01.877 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:01.930 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:01.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:01.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:02.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:01 smithi165 ceph-mon[105517]: pgmap v2816: 99 pgs: 99 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-08-17T18:10:02.032 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:02.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:02.082 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:02.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:02.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:02.214 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:02.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:02.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:02.317 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:02.359 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:02.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:02.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:02.441 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 23 16 51418 51402 139.667 104.5 0.00892862 0.00715324 2024-08-17T18:10:02.461 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:02.475 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:10:02.476 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:10:02.490 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:02.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:02.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:02.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:02.624 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:02.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:02.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:02.726 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:02.768 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:02.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:02.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:02.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:02.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:02.899 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:02.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:03.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:03.035 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:03.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:03.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:03.137 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:03.167 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:03.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:03.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:03.269 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:03.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:03.305 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:03.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:03.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:03.438 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:03.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:03.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:03.441 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 24 16 53129 53113 138.303 106.938 0.00874247 0.00722567 2024-08-17T18:10:03.540 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:03.579 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:03.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:03.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:03.599 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-17T18:10:03.681 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:03.712 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:03.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:03.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:03.814 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:03.835 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:03 smithi136 ceph-mon[109861]: pgmap v2817: 99 pgs: 99 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-08-17T18:10:03.835 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:03 smithi136 ceph-mon[115078]: pgmap v2817: 99 pgs: 99 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-08-17T18:10:03.840 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:03.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:03.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:03.919 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:10:03.941 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:03.974 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:03.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:03.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:03 smithi165 ceph-mon[105517]: pgmap v2817: 99 pgs: 99 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-08-17T18:10:04.076 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:04.105 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:04.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:04.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:04.207 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:04.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:04.274 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:04.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:04.375 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:04.408 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:04.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:04.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:04.441 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 25 16 54666 54650 136.613 96.0625 0.00791641 0.00731532 2024-08-17T18:10:04.510 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:04.543 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:04.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:04.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:04.645 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:04.673 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:04.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:04.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:04.775 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:04.811 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:04.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:04.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:04.913 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:04.952 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:04.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:04.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:05.036 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:10:05.054 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:05.132 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:10:05.133 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:10:05.144 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:05.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:05.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:05.248 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:05.277 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:05.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:05.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:05.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:05.419 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:10:05.421 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T18:10:05.435 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:05.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:05.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:05.441 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 26 16 56098 56082 134.8 89.5 0.0170366 0.00741219 2024-08-17T18:10:05.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:05.561 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:05.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:05.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:05.663 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:05.703 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:05.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:05.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:05.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:05.836 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:05.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:05.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:05.901 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:05 smithi136 ceph-mon[109861]: pgmap v2818: 99 pgs: 99 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-08-17T18:10:05.901 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:05 smithi136 ceph-mon[115078]: pgmap v2818: 99 pgs: 99 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-08-17T18:10:05.938 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:05.959 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:05.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:05.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:06.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:05 smithi165 ceph-mon[105517]: pgmap v2818: 99 pgs: 99 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-08-17T18:10:06.060 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:06.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:06.090 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:06.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:06.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:06.223 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:06.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:06.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:06.324 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:06.360 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:06.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:06.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:06.442 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 27 15 57485 57470 133.021 86.75 0.0105541 0.00751287 2024-08-17T18:10:06.462 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:06.495 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:06.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:06.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:06.597 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:06.627 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:06.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:06.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:06.728 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:06.753 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:06.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:06.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:06.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:06.885 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:06.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:06.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:06 smithi136 ceph-mon[109861]: from='client.55480 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:10:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:10:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:06 smithi136 ceph-mon[115078]: from='client.55480 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:10:06.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:10:06.987 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:06 smithi165 ceph-mon[105517]: from='client.55480 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:10:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:10:07.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:07.022 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:07.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:07.123 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:07.164 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:07.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:07.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:07.266 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:07.300 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:07.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:07.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:07.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:07.430 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:07.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:07.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:07.442 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 28 16 58934 58918 131.502 90.5 0.0063247 0.00759936 2024-08-17T18:10:07.532 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:07.579 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:07.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:07.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:07.637 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:10:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:10:07] "GET /metrics HTTP/1.1" 200 36604 "" "Prometheus/2.51.0" 2024-08-17T18:10:07.681 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:07.714 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:07.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:07.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:07.816 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:07.853 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:07.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:07.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:07.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:07 smithi136 ceph-mon[109861]: pgmap v2819: 99 pgs: 99 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-08-17T18:10:07.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:07 smithi136 ceph-mon[115078]: pgmap v2819: 99 pgs: 99 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-08-17T18:10:07.955 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:07.995 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:07.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:07.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:08.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:07 smithi165 ceph-mon[105517]: pgmap v2819: 99 pgs: 99 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-08-17T18:10:08.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:08.131 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:08.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:08.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:08.232 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:08.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:08.265 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:08.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:08.366 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:08.401 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:08.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:08.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:08.442 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 29 16 60545 60529 130.439 100.688 0.0145426 0.00766189 2024-08-17T18:10:08.502 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:08.537 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:08.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:08.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:08.638 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:08.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:08.675 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:08.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:08.776 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:08.805 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:08.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:08.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:08.907 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:08.941 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:08.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:08.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:09.043 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:09.079 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:09.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:09.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:09.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:09.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:09.218 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:09.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:09.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:09.338 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":2818,"stamp":"2024-08-17T18:10:04.311714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3302496229,"num_objects":50409,"num_object_clones":0,"num_object_copies":151227,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50447,"num_write_kb":3224545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":598400,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52081,"ondisk_log_size":52081,"up":297,"acting":297,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":298,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":14366884,"kb_used_data":9771052,"kb_used_omap":350,"kb_used_meta":4595425,"kb_avail":641849180,"statfs":{"total":671965249536,"available":657253560320,"internally_reserved":0,"allocated":10005557248,"data_stored":9967124799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":358888,"internal_metadata":4705715736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1423322714,"num_objects":21719,"num_object_clones":0,"num_object_copies":65157,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21719,"num_whiteouts":0,"num_read":-34,"num_read_kb":-75,"num_write":21699,"num_write_kb":1389796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003903"},"pg_stats":[{"pgid":"56.f","version":"1540'2985","reported_seq":3006,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:10:00.405863+0000","last_change":"2024-08-17T18:09:28.696402+0000","last_active":"2024-08-17T18:10:00.405863+0000","last_peered":"2024-08-17T18:10:00.405863+0000","last_clean":"2024-08-17T18:10:00.405863+0000","last_became_active":"2024-08-17T18:09:28.696135+0000","last_became_peered":"2024-08-17T18:09:28.696135+0000","last_unstale":"2024-08-17T18:10:00.405863+0000","last_undegraded":"2024-08-17T18:10:00.405863+0000","last_fullsized":"2024-08-17T18:10:00.405863+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":2985,"log_dups_size":0,"ondisk_log_size":2985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:44:29.924728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195624960,"num_objects":2985,"num_object_clones":0,"num_object_copies":8955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2985,"num_write_kb":191040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.e","version":"1540'2944","reported_seq":2965,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:10:00.389854+0000","last_change":"2024-08-17T18:09:28.697584+0000","last_active":"2024-08-17T18:10:00.389854+0000","last_peered":"2024-08-17T18:10:00.389854+0000","last_clean":"2024-08-17T18:10:00.389854+0000","last_became_active":"2024-08-17T18:09:28.697211+0000","last_became_peered":"2024-08-17T18:09:28.697211+0000","last_unstale":"2024-08-17T18:10:00.389854+0000","last_undegraded":"2024-08-17T18:10:00.389854+0000","last_fullsized":"2024-08-17T18:10:00.389854+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":2944,"log_dups_size":0,"ondisk_log_size":2944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:16:31.941416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192937984,"num_objects":2944,"num_object_clones":0,"num_object_copies":8832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2944,"num_write_kb":188416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.d","version":"1540'3032","reported_seq":3053,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:10:00.856785+0000","last_change":"2024-08-17T18:09:28.696225+0000","last_active":"2024-08-17T18:10:00.856785+0000","last_peered":"2024-08-17T18:10:00.856785+0000","last_clean":"2024-08-17T18:10:00.856785+0000","last_became_active":"2024-08-17T18:09:28.695916+0000","last_became_peered":"2024-08-17T18:09:28.695916+0000","last_unstale":"2024-08-17T18:10:00.856785+0000","last_undegraded":"2024-08-17T18:10:00.856785+0000","last_fullsized":"2024-08-17T18:10:00.856785+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":3032,"log_dups_size":0,"ondisk_log_size":3032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:00:20.159452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198705152,"num_objects":3032,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3032,"num_write_kb":194048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"56.c","version":"1540'2994","reported_seq":3015,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:10:00.407363+0000","last_change":"2024-08-17T18:09:28.627634+0000","last_active":"2024-08-17T18:10:00.407363+0000","last_peered":"2024-08-17T18:10:00.407363+0000","last_clean":"2024-08-17T18:10:00.407363+0000","last_became_active":"2024-08-17T18:09:28.627138+0000","last_became_peered":"2024-08-17T18:09:28.627138+0000","last_unstale":"2024-08-17T18:10:00.407363+0000","last_undegraded":"2024-08-17T18:10:00.407363+0000","last_fullsized":"2024-08-17T18:10:00.407363+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":2994,"log_dups_size":0,"ondisk_log_size":2994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:47:56.574797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196214784,"num_objects":2994,"num_object_clones":0,"num_object_copies":8982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2994,"num_write_kb":191616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.b","version":"1540'3236","reported_seq":3248,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:10:02.862130+0000","last_change":"2024-08-17T18:09:28.627309+0000","last_active":"2024-08-17T18:10:02.862130+0000","last_peered":"2024-08-17T18:10:02.862130+0000","last_clean":"2024-08-17T18:10:02.862130+0000","last_became_active":"2024-08-17T18:09:28.626970+0000","last_became_peered":"2024-08-17T18:09:28.626970+0000","last_unstale":"2024-08-17T18:10:02.862130+0000","last_undegraded":"2024-08-17T18:10:02.862130+0000","last_fullsized":"2024-08-17T18:10:02.862130+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":3236,"log_dups_size":0,"ondisk_log_size":3236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:47:24.480441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212074496,"num_objects":3236,"num_object_clones":0,"num_object_copies":9708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3236,"num_write_kb":207104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"56.a","version":"1540'3027","reported_seq":3048,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:10:00.406521+0000","last_change":"2024-08-17T18:09:28.697301+0000","last_active":"2024-08-17T18:10:00.406521+0000","last_peered":"2024-08-17T18:10:00.406521+0000","last_clean":"2024-08-17T18:10:00.406521+0000","last_became_active":"2024-08-17T18:09:28.697019+0000","last_became_peered":"2024-08-17T18:09:28.697019+0000","last_unstale":"2024-08-17T18:10:00.406521+0000","last_undegraded":"2024-08-17T18:10:00.406521+0000","last_fullsized":"2024-08-17T18:10:00.406521+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":3027,"log_dups_size":0,"ondisk_log_size":3027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:44:37.273534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198377472,"num_objects":3027,"num_object_clones":0,"num_object_copies":9081,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3027,"num_write_kb":193728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.9","version":"1540'3231","reported_seq":3245,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:10:04.062098+0000","last_change":"2024-08-17T18:09:28.626545+0000","last_active":"2024-08-17T18:10:04.062098+0000","last_peered":"2024-08-17T18:10:04.062098+0000","last_clean":"2024-08-17T18:10:04.062098+0000","last_became_active":"2024-08-17T18:09:28.626300+0000","last_became_peered":"2024-08-17T18:09:28.626300+0000","last_unstale":"2024-08-17T18:10:04.062098+0000","last_undegraded":"2024-08-17T18:10:04.062098+0000","last_fullsized":"2024-08-17T18:10:04.062098+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":3231,"log_dups_size":0,"ondisk_log_size":3231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:34:50.839206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211746816,"num_objects":3231,"num_object_clones":0,"num_object_copies":9693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3231,"num_write_kb":206784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.8","version":"1540'3088","reported_seq":3109,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:10:00.853669+0000","last_change":"2024-08-17T18:09:28.697184+0000","last_active":"2024-08-17T18:10:00.853669+0000","last_peered":"2024-08-17T18:10:00.853669+0000","last_clean":"2024-08-17T18:10:00.853669+0000","last_became_active":"2024-08-17T18:09:28.696911+0000","last_became_peered":"2024-08-17T18:09:28.696911+0000","last_unstale":"2024-08-17T18:10:00.853669+0000","last_undegraded":"2024-08-17T18:10:00.853669+0000","last_fullsized":"2024-08-17T18:10:00.853669+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":3088,"log_dups_size":0,"ondisk_log_size":3088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:39:59.778610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202375168,"num_objects":3088,"num_object_clones":0,"num_object_copies":9264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3088,"num_write_kb":197632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"56.7","version":"1540'2984","reported_seq":3005,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:10:00.874560+0000","last_change":"2024-08-17T18:09:28.628236+0000","last_active":"2024-08-17T18:10:00.874560+0000","last_peered":"2024-08-17T18:10:00.874560+0000","last_clean":"2024-08-17T18:10:00.874560+0000","last_became_active":"2024-08-17T18:09:28.627952+0000","last_became_peered":"2024-08-17T18:09:28.627952+0000","last_unstale":"2024-08-17T18:10:00.874560+0000","last_undegraded":"2024-08-17T18:10:00.874560+0000","last_fullsized":"2024-08-17T18:10:00.874560+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":2984,"log_dups_size":0,"ondisk_log_size":2984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:04:20.822971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195559424,"num_objects":2984,"num_object_clones":0,"num_object_copies":8952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2984,"num_write_kb":190976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"56.6","version":"1540'3417","reported_seq":3431,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:10:04.076240+0000","last_change":"2024-08-17T18:09:28.629945+0000","last_active":"2024-08-17T18:10:04.076240+0000","last_peered":"2024-08-17T18:10:04.076240+0000","last_clean":"2024-08-17T18:10:04.076240+0000","last_became_active":"2024-08-17T18:09:28.629553+0000","last_became_peered":"2024-08-17T18:09:28.629553+0000","last_unstale":"2024-08-17T18:10:04.076240+0000","last_undegraded":"2024-08-17T18:10:04.076240+0000","last_fullsized":"2024-08-17T18:10:04.076240+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":3417,"log_dups_size":0,"ondisk_log_size":3417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:00:36.875044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223936512,"num_objects":3417,"num_object_clones":0,"num_object_copies":10251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3417,"num_write_kb":218688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.5","version":"1540'3121","reported_seq":3142,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:10:01.777080+0000","last_change":"2024-08-17T18:09:28.697212+0000","last_active":"2024-08-17T18:10:01.777080+0000","last_peered":"2024-08-17T18:10:01.777080+0000","last_clean":"2024-08-17T18:10:01.777080+0000","last_became_active":"2024-08-17T18:09:28.696769+0000","last_became_peered":"2024-08-17T18:09:28.696769+0000","last_unstale":"2024-08-17T18:10:01.777080+0000","last_undegraded":"2024-08-17T18:10:01.777080+0000","last_fullsized":"2024-08-17T18:10:01.777080+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":3121,"log_dups_size":0,"ondisk_log_size":3121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:30:22.827483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204537856,"num_objects":3121,"num_object_clones":0,"num_object_copies":9363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3121,"num_write_kb":199744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.4","version":"1540'2998","reported_seq":3018,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:10:00.424424+0000","last_change":"2024-08-17T18:09:28.627340+0000","last_active":"2024-08-17T18:10:00.424424+0000","last_peered":"2024-08-17T18:10:00.424424+0000","last_clean":"2024-08-17T18:10:00.424424+0000","last_became_active":"2024-08-17T18:09:28.626966+0000","last_became_peered":"2024-08-17T18:09:28.626966+0000","last_unstale":"2024-08-17T18:10:00.424424+0000","last_undegraded":"2024-08-17T18:10:00.424424+0000","last_fullsized":"2024-08-17T18:10:00.424424+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":2998,"log_dups_size":0,"ondisk_log_size":2998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:18:20.264361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196476928,"num_objects":2998,"num_object_clones":0,"num_object_copies":8994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2998,"num_write_kb":191872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.3","version":"1540'2996","reported_seq":3017,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:10:00.413040+0000","last_change":"2024-08-17T18:09:28.697359+0000","last_active":"2024-08-17T18:10:00.413040+0000","last_peered":"2024-08-17T18:10:00.413040+0000","last_clean":"2024-08-17T18:10:00.413040+0000","last_became_active":"2024-08-17T18:09:28.696943+0000","last_became_peered":"2024-08-17T18:09:28.696943+0000","last_unstale":"2024-08-17T18:10:00.413040+0000","last_undegraded":"2024-08-17T18:10:00.413040+0000","last_fullsized":"2024-08-17T18:10:00.413040+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":2996,"log_dups_size":0,"ondisk_log_size":2996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:08:17.415390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196345856,"num_objects":2996,"num_object_clones":0,"num_object_copies":8988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":191744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.2","version":"1540'3460","reported_seq":3474,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:10:04.073104+0000","last_change":"2024-08-17T18:09:28.628711+0000","last_active":"2024-08-17T18:10:04.073104+0000","last_peered":"2024-08-17T18:10:04.073104+0000","last_clean":"2024-08-17T18:10:04.073104+0000","last_became_active":"2024-08-17T18:09:28.628387+0000","last_became_peered":"2024-08-17T18:09:28.628387+0000","last_unstale":"2024-08-17T18:10:04.073104+0000","last_undegraded":"2024-08-17T18:10:04.073104+0000","last_fullsized":"2024-08-17T18:10:04.073104+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":3460,"log_dups_size":0,"ondisk_log_size":3460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:26:06.412021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226754560,"num_objects":3460,"num_object_clones":0,"num_object_copies":10380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3460,"num_write_kb":221440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.1","version":"1540'3539","reported_seq":3553,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:10:04.055022+0000","last_change":"2024-08-17T18:09:28.629660+0000","last_active":"2024-08-17T18:10:04.055022+0000","last_peered":"2024-08-17T18:10:04.055022+0000","last_clean":"2024-08-17T18:10:04.055022+0000","last_became_active":"2024-08-17T18:09:28.629359+0000","last_became_peered":"2024-08-17T18:09:28.629359+0000","last_unstale":"2024-08-17T18:10:04.055022+0000","last_undegraded":"2024-08-17T18:10:04.055022+0000","last_fullsized":"2024-08-17T18:10:04.055022+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":3539,"log_dups_size":0,"ondisk_log_size":3539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:05:58.848611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231931904,"num_objects":3539,"num_object_clones":0,"num_object_copies":10617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3539,"num_write_kb":226496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.0","version":"1540'3331","reported_seq":3345,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:10:04.068488+0000","last_change":"2024-08-17T18:09:28.697077+0000","last_active":"2024-08-17T18:10:04.068488+0000","last_peered":"2024-08-17T18:10:04.068488+0000","last_clean":"2024-08-17T18:10:04.068488+0000","last_became_active":"2024-08-17T18:09:28.696676+0000","last_became_peered":"2024-08-17T18:09:28.696676+0000","last_unstale":"2024-08-17T18:10:04.068488+0000","last_undegraded":"2024-08-17T18:10:04.068488+0000","last_fullsized":"2024-08-17T18:10:04.068488+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":3331,"log_dups_size":0,"ondisk_log_size":3331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:09.350570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218300416,"num_objects":3331,"num_object_clones":0,"num_object_copies":9993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3331,"num_write_kb":213184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":223,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.392080+0000","last_change":"2024-08-17T18:07:48.546202+0000","last_active":"2024-08-17T18:09:49.392080+0000","last_peered":"2024-08-17T18:09:49.392080+0000","last_clean":"2024-08-17T18:09:49.392080+0000","last_became_active":"2024-08-17T18:07:48.531246+0000","last_became_peered":"2024-08-17T18:07:48.531246+0000","last_unstale":"2024-08-17T18:09:49.392080+0000","last_undegraded":"2024-08-17T18:09:49.392080+0000","last_fullsized":"2024-08-17T18:09:49.392080+0000","mapping_epoch":1501,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:30.944362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1543,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.537362+0000","last_change":"2024-08-17T18:08:37.068490+0000","last_active":"2024-08-17T18:09:38.537362+0000","last_peered":"2024-08-17T18:09:38.537362+0000","last_clean":"2024-08-17T18:09:38.537362+0000","last_became_active":"2024-08-17T18:08:37.068026+0000","last_became_peered":"2024-08-17T18:08:37.068026+0000","last_unstale":"2024-08-17T18:09:38.537362+0000","last_undegraded":"2024-08-17T18:09:38.537362+0000","last_fullsized":"2024-08-17T18:09:38.537362+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:34:27.467963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":2399,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.401866+0000","last_change":"2024-08-17T18:08:37.082356+0000","last_active":"2024-08-17T18:09:49.401866+0000","last_peered":"2024-08-17T18:09:49.401866+0000","last_clean":"2024-08-17T18:09:49.401866+0000","last_became_active":"2024-08-17T18:08:37.082046+0000","last_became_peered":"2024-08-17T18:08:37.082046+0000","last_unstale":"2024-08-17T18:09:49.401866+0000","last_undegraded":"2024-08-17T18:09:49.401866+0000","last_fullsized":"2024-08-17T18:09:49.401866+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:46:13.932917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2421,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.399342+0000","last_change":"2024-08-17T18:04:14.540774+0000","last_active":"2024-08-17T18:09:49.399342+0000","last_peered":"2024-08-17T18:09:49.399342+0000","last_clean":"2024-08-17T18:09:49.399342+0000","last_became_active":"2024-08-17T18:04:14.539778+0000","last_became_peered":"2024-08-17T18:04:14.539778+0000","last_unstale":"2024-08-17T18:09:49.399342+0000","last_undegraded":"2024-08-17T18:09:49.399342+0000","last_fullsized":"2024-08-17T18:09:49.399342+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:55:02.863360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":92,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.080394+0000","last_change":"2024-08-17T18:04:43.777705+0000","last_active":"2024-08-17T18:09:49.080394+0000","last_peered":"2024-08-17T18:09:49.080394+0000","last_clean":"2024-08-17T18:09:49.080394+0000","last_became_active":"2024-08-17T18:04:43.771283+0000","last_became_peered":"2024-08-17T18:04:43.771283+0000","last_unstale":"2024-08-17T18:09:49.080394+0000","last_undegraded":"2024-08-17T18:09:49.080394+0000","last_fullsized":"2024-08-17T18:09:49.080394+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:13.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1666,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.399825+0000","last_change":"2024-08-17T18:04:14.534963+0000","last_active":"2024-08-17T18:09:49.399825+0000","last_peered":"2024-08-17T18:09:49.399825+0000","last_clean":"2024-08-17T18:09:49.399825+0000","last_became_active":"2024-08-17T18:04:14.534634+0000","last_became_peered":"2024-08-17T18:04:14.534634+0000","last_unstale":"2024-08-17T18:09:49.399825+0000","last_undegraded":"2024-08-17T18:09:49.399825+0000","last_fullsized":"2024-08-17T18:09:49.399825+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:33:27.287860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":92,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:44.077757+0000","last_change":"2024-08-17T18:04:43.773534+0000","last_active":"2024-08-17T18:09:44.077757+0000","last_peered":"2024-08-17T18:09:44.077757+0000","last_clean":"2024-08-17T18:09:44.077757+0000","last_became_active":"2024-08-17T18:04:43.768507+0000","last_became_peered":"2024-08-17T18:04:43.768507+0000","last_unstale":"2024-08-17T18:09:44.077757+0000","last_undegraded":"2024-08-17T18:09:44.077757+0000","last_fullsized":"2024-08-17T18:09:44.077757+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:50:57.647351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1530,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.537757+0000","last_change":"2024-08-17T17:24:17.458168+0000","last_active":"2024-08-17T18:09:38.537757+0000","last_peered":"2024-08-17T18:09:38.537757+0000","last_clean":"2024-08-17T18:09:38.537757+0000","last_became_active":"2024-08-17T17:24:17.457847+0000","last_became_peered":"2024-08-17T17:24:17.457847+0000","last_unstale":"2024-08-17T18:09:38.537757+0000","last_undegraded":"2024-08-17T18:09:38.537757+0000","last_fullsized":"2024-08-17T18:09:38.537757+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:28.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":178,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.391739+0000","last_change":"2024-08-17T18:04:14.531445+0000","last_active":"2024-08-17T18:09:49.391739+0000","last_peered":"2024-08-17T18:09:49.391739+0000","last_clean":"2024-08-17T18:09:49.391739+0000","last_became_active":"2024-08-17T18:04:14.531099+0000","last_became_peered":"2024-08-17T18:04:14.531099+0000","last_unstale":"2024-08-17T18:09:49.391739+0000","last_undegraded":"2024-08-17T18:09:49.391739+0000","last_fullsized":"2024-08-17T18:09:49.391739+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:43.335123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":88,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.395652+0000","last_change":"2024-08-17T18:07:48.548262+0000","last_active":"2024-08-17T18:09:49.395652+0000","last_peered":"2024-08-17T18:09:49.395652+0000","last_clean":"2024-08-17T18:09:49.395652+0000","last_became_active":"2024-08-17T18:07:48.547846+0000","last_became_peered":"2024-08-17T18:07:48.547846+0000","last_unstale":"2024-08-17T18:09:49.395652+0000","last_undegraded":"2024-08-17T18:09:49.395652+0000","last_fullsized":"2024-08-17T18:09:49.395652+0000","mapping_epoch":1501,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:24:37.513219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1548,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.391664+0000","last_change":"2024-08-17T18:01:23.519890+0000","last_active":"2024-08-17T18:09:49.391664+0000","last_peered":"2024-08-17T18:09:49.391664+0000","last_clean":"2024-08-17T18:09:49.391664+0000","last_became_active":"2024-08-17T18:01:23.519526+0000","last_became_peered":"2024-08-17T18:01:23.519526+0000","last_unstale":"2024-08-17T18:09:49.391664+0000","last_undegraded":"2024-08-17T18:09:49.391664+0000","last_fullsized":"2024-08-17T18:09:49.391664+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:04:52.662528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2387,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.401252+0000","last_change":"2024-08-17T18:01:23.491836+0000","last_active":"2024-08-17T18:09:49.401252+0000","last_peered":"2024-08-17T18:09:49.401252+0000","last_clean":"2024-08-17T18:09:49.401252+0000","last_became_active":"2024-08-17T18:01:23.491209+0000","last_became_peered":"2024-08-17T18:01:23.491209+0000","last_unstale":"2024-08-17T18:09:49.401252+0000","last_undegraded":"2024-08-17T18:09:49.401252+0000","last_fullsized":"2024-08-17T18:09:49.401252+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:35:30.767005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2389,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.403581+0000","last_change":"2024-08-17T17:24:17.452025+0000","last_active":"2024-08-17T18:09:49.403581+0000","last_peered":"2024-08-17T18:09:49.403581+0000","last_clean":"2024-08-17T18:09:49.403581+0000","last_became_active":"2024-08-17T17:24:17.448029+0000","last_became_peered":"2024-08-17T17:24:17.448029+0000","last_unstale":"2024-08-17T18:09:49.403581+0000","last_undegraded":"2024-08-17T18:09:49.403581+0000","last_fullsized":"2024-08-17T18:09:49.403581+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:45.089346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":192,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.391943+0000","last_change":"2024-08-17T18:08:11.358013+0000","last_active":"2024-08-17T18:09:49.391943+0000","last_peered":"2024-08-17T18:09:49.391943+0000","last_clean":"2024-08-17T18:09:49.391943+0000","last_became_active":"2024-08-17T18:08:11.357666+0000","last_became_peered":"2024-08-17T18:08:11.357666+0000","last_unstale":"2024-08-17T18:09:49.391943+0000","last_undegraded":"2024-08-17T18:09:49.391943+0000","last_fullsized":"2024-08-17T18:09:49.391943+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:05:25.290465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2401,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.399050+0000","last_change":"2024-08-17T18:04:14.539660+0000","last_active":"2024-08-17T18:09:49.399050+0000","last_peered":"2024-08-17T18:09:49.399050+0000","last_clean":"2024-08-17T18:09:49.399050+0000","last_became_active":"2024-08-17T18:04:14.537852+0000","last_became_peered":"2024-08-17T18:04:14.537852+0000","last_unstale":"2024-08-17T18:09:49.399050+0000","last_undegraded":"2024-08-17T18:09:49.399050+0000","last_fullsized":"2024-08-17T18:09:49.399050+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:47:45.026997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1544,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:37.647966+0000","last_change":"2024-08-17T18:04:43.769892+0000","last_active":"2024-08-17T18:09:37.647966+0000","last_peered":"2024-08-17T18:09:37.647966+0000","last_clean":"2024-08-17T18:09:37.647966+0000","last_became_active":"2024-08-17T18:04:43.769702+0000","last_became_peered":"2024-08-17T18:04:43.769702+0000","last_unstale":"2024-08-17T18:09:37.647966+0000","last_undegraded":"2024-08-17T18:09:37.647966+0000","last_fullsized":"2024-08-17T18:09:37.647966+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:39.373904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2361,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.399300+0000","last_change":"2024-08-17T17:24:17.442338+0000","last_active":"2024-08-17T18:09:49.399300+0000","last_peered":"2024-08-17T18:09:49.399300+0000","last_clean":"2024-08-17T18:09:49.399300+0000","last_became_active":"2024-08-17T17:24:17.441948+0000","last_became_peered":"2024-08-17T17:24:17.441948+0000","last_unstale":"2024-08-17T18:09:49.399300+0000","last_undegraded":"2024-08-17T18:09:49.399300+0000","last_fullsized":"2024-08-17T18:09:49.399300+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:17.891492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":185,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.392461+0000","last_change":"2024-08-17T18:04:43.777736+0000","last_active":"2024-08-17T18:09:49.392461+0000","last_peered":"2024-08-17T18:09:49.392461+0000","last_clean":"2024-08-17T18:09:49.392461+0000","last_became_active":"2024-08-17T18:04:43.777438+0000","last_became_peered":"2024-08-17T18:04:43.777438+0000","last_unstale":"2024-08-17T18:09:49.392461+0000","last_undegraded":"2024-08-17T18:09:49.392461+0000","last_fullsized":"2024-08-17T18:09:49.392461+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:24:01.255927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2341,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.399681+0000","last_change":"2024-08-17T18:04:43.915398+0000","last_active":"2024-08-17T18:09:49.399681+0000","last_peered":"2024-08-17T18:09:49.399681+0000","last_clean":"2024-08-17T18:09:49.399681+0000","last_became_active":"2024-08-17T18:04:43.915212+0000","last_became_peered":"2024-08-17T18:04:43.915212+0000","last_unstale":"2024-08-17T18:09:49.399681+0000","last_undegraded":"2024-08-17T18:09:49.399681+0000","last_fullsized":"2024-08-17T18:09:49.399681+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:43:26.901830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":105,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.538017+0000","last_change":"2024-08-17T18:08:11.356876+0000","last_active":"2024-08-17T18:09:38.538017+0000","last_peered":"2024-08-17T18:09:38.538017+0000","last_clean":"2024-08-17T18:09:38.538017+0000","last_became_active":"2024-08-17T18:08:11.355662+0000","last_became_peered":"2024-08-17T18:08:11.355662+0000","last_unstale":"2024-08-17T18:09:38.538017+0000","last_undegraded":"2024-08-17T18:09:38.538017+0000","last_fullsized":"2024-08-17T18:09:38.538017+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:55:00.801251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":498,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.401334+0000","last_change":"2024-08-17T18:04:14.555729+0000","last_active":"2024-08-17T18:09:49.401334+0000","last_peered":"2024-08-17T18:09:49.401334+0000","last_clean":"2024-08-17T18:09:49.401334+0000","last_became_active":"2024-08-17T18:04:14.536678+0000","last_became_peered":"2024-08-17T18:04:14.536678+0000","last_unstale":"2024-08-17T18:09:49.401334+0000","last_undegraded":"2024-08-17T18:09:49.401334+0000","last_fullsized":"2024-08-17T18:09:49.401334+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:47:29.424778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2414,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.401225+0000","last_change":"2024-08-17T18:08:37.079571+0000","last_active":"2024-08-17T18:09:49.401225+0000","last_peered":"2024-08-17T18:09:49.401225+0000","last_clean":"2024-08-17T18:09:49.401225+0000","last_became_active":"2024-08-17T18:08:37.079224+0000","last_became_peered":"2024-08-17T18:08:37.079224+0000","last_unstale":"2024-08-17T18:09:49.401225+0000","last_undegraded":"2024-08-17T18:09:49.401225+0000","last_fullsized":"2024-08-17T18:09:49.401225+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:28:50.561072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1545,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.537805+0000","last_change":"2024-08-17T18:03:51.292044+0000","last_active":"2024-08-17T18:09:38.537805+0000","last_peered":"2024-08-17T18:09:38.537805+0000","last_clean":"2024-08-17T18:09:38.537805+0000","last_became_active":"2024-08-17T18:03:51.268271+0000","last_became_peered":"2024-08-17T18:03:51.268271+0000","last_unstale":"2024-08-17T18:09:38.537805+0000","last_undegraded":"2024-08-17T18:09:38.537805+0000","last_fullsized":"2024-08-17T18:09:38.537805+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:00.892410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4756,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1517,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.537762+0000","last_change":"2024-08-17T17:23:07.455919+0000","last_active":"2024-08-17T18:09:38.537762+0000","last_peered":"2024-08-17T18:09:38.537762+0000","last_clean":"2024-08-17T18:09:38.537762+0000","last_became_active":"2024-08-17T17:23:07.455284+0000","last_became_peered":"2024-08-17T17:23:07.455284+0000","last_unstale":"2024-08-17T18:09:38.537762+0000","last_undegraded":"2024-08-17T18:09:38.537762+0000","last_fullsized":"2024-08-17T18:09:38.537762+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:37:46.260775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"347'149","reported_seq":821,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.399510+0000","last_change":"2024-08-17T18:09:15.592511+0000","last_active":"2024-08-17T18:09:49.399510+0000","last_peered":"2024-08-17T18:09:49.399510+0000","last_clean":"2024-08-17T18:09:49.399510+0000","last_became_active":"2024-08-17T18:09:15.592099+0000","last_became_peered":"2024-08-17T18:09:15.592099+0000","last_unstale":"2024-08-17T18:09:49.399510+0000","last_undegraded":"2024-08-17T18:09:49.399510+0000","last_fullsized":"2024-08-17T18:09:49.399510+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:57.271286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:57.271286+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:54.841504+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1541,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.538061+0000","last_change":"2024-08-17T18:04:43.766724+0000","last_active":"2024-08-17T18:09:38.538061+0000","last_peered":"2024-08-17T18:09:38.538061+0000","last_clean":"2024-08-17T18:09:38.538061+0000","last_became_active":"2024-08-17T18:04:43.766428+0000","last_became_peered":"2024-08-17T18:04:43.766428+0000","last_unstale":"2024-08-17T18:09:38.538061+0000","last_undegraded":"2024-08-17T18:09:38.538061+0000","last_fullsized":"2024-08-17T18:09:38.538061+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:09:49.102940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1554,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:37.648011+0000","last_change":"2024-08-17T18:03:51.279963+0000","last_active":"2024-08-17T18:09:37.648011+0000","last_peered":"2024-08-17T18:09:37.648011+0000","last_clean":"2024-08-17T18:09:37.648011+0000","last_became_active":"2024-08-17T18:03:51.279506+0000","last_became_peered":"2024-08-17T18:03:51.279506+0000","last_unstale":"2024-08-17T18:09:37.648011+0000","last_undegraded":"2024-08-17T18:09:37.648011+0000","last_fullsized":"2024-08-17T18:09:37.648011+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:20:07.304042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":182,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.392465+0000","last_change":"2024-08-17T18:04:14.542589+0000","last_active":"2024-08-17T18:09:49.392465+0000","last_peered":"2024-08-17T18:09:49.392465+0000","last_clean":"2024-08-17T18:09:49.392465+0000","last_became_active":"2024-08-17T18:04:14.531540+0000","last_became_peered":"2024-08-17T18:04:14.531540+0000","last_unstale":"2024-08-17T18:09:49.392465+0000","last_undegraded":"2024-08-17T18:09:49.392465+0000","last_fullsized":"2024-08-17T18:09:49.392465+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:15:33.014782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"347'149","reported_seq":811,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.537955+0000","last_change":"2024-08-17T18:09:15.592395+0000","last_active":"2024-08-17T18:09:38.537955+0000","last_peered":"2024-08-17T18:09:38.537955+0000","last_clean":"2024-08-17T18:09:38.537955+0000","last_became_active":"2024-08-17T18:09:15.591773+0000","last_became_peered":"2024-08-17T18:09:15.591773+0000","last_unstale":"2024-08-17T18:09:38.537955+0000","last_undegraded":"2024-08-17T18:09:38.537955+0000","last_fullsized":"2024-08-17T18:09:38.537955+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:53.309517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:53.309517+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:50:57.110936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":74,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.399169+0000","last_change":"2024-08-17T18:08:11.349400+0000","last_active":"2024-08-17T18:09:49.399169+0000","last_peered":"2024-08-17T18:09:49.399169+0000","last_clean":"2024-08-17T18:09:49.399169+0000","last_became_active":"2024-08-17T18:08:11.348892+0000","last_became_peered":"2024-08-17T18:08:11.348892+0000","last_unstale":"2024-08-17T18:09:49.399169+0000","last_undegraded":"2024-08-17T18:09:49.399169+0000","last_fullsized":"2024-08-17T18:09:49.399169+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:52:54.819838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2371,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.400068+0000","last_change":"2024-08-17T17:23:07.428296+0000","last_active":"2024-08-17T18:09:49.400068+0000","last_peered":"2024-08-17T18:09:49.400068+0000","last_clean":"2024-08-17T18:09:49.400068+0000","last_became_active":"2024-08-17T17:23:07.428010+0000","last_became_peered":"2024-08-17T17:23:07.428010+0000","last_unstale":"2024-08-17T18:09:49.400068+0000","last_undegraded":"2024-08-17T18:09:49.400068+0000","last_fullsized":"2024-08-17T18:09:49.400068+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:45.459212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":2413,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.400277+0000","last_change":"2024-08-17T18:04:43.800633+0000","last_active":"2024-08-17T18:09:49.400277+0000","last_peered":"2024-08-17T18:09:49.400277+0000","last_clean":"2024-08-17T18:09:49.400277+0000","last_became_active":"2024-08-17T18:04:43.784612+0000","last_became_peered":"2024-08-17T18:04:43.784612+0000","last_unstale":"2024-08-17T18:09:49.400277+0000","last_undegraded":"2024-08-17T18:09:49.400277+0000","last_fullsized":"2024-08-17T18:09:49.400277+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:11.405039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"347'149","reported_seq":794,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:52.474570+0000","last_change":"2024-08-17T18:09:52.474670+0000","last_active":"2024-08-17T18:09:52.474570+0000","last_peered":"2024-08-17T18:09:52.474570+0000","last_clean":"2024-08-17T18:09:52.474570+0000","last_became_active":"2024-08-17T18:09:15.595011+0000","last_became_peered":"2024-08-17T18:09:15.595011+0000","last_unstale":"2024-08-17T18:09:52.474570+0000","last_undegraded":"2024-08-17T18:09:52.474570+0000","last_fullsized":"2024-08-17T18:09:52.474570+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:52.474544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:52.474544+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:38:21.347345+0000","scrub_duration":0.001886326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":185,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.392015+0000","last_change":"2024-08-17T18:04:43.805055+0000","last_active":"2024-08-17T18:09:49.392015+0000","last_peered":"2024-08-17T18:09:49.392015+0000","last_clean":"2024-08-17T18:09:49.392015+0000","last_became_active":"2024-08-17T18:04:43.804708+0000","last_became_peered":"2024-08-17T18:04:43.804708+0000","last_unstale":"2024-08-17T18:09:49.392015+0000","last_undegraded":"2024-08-17T18:09:49.392015+0000","last_fullsized":"2024-08-17T18:09:49.392015+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:47:22.253073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":178,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.395477+0000","last_change":"2024-08-17T18:04:14.529605+0000","last_active":"2024-08-17T18:09:49.395477+0000","last_peered":"2024-08-17T18:09:49.395477+0000","last_clean":"2024-08-17T18:09:49.395477+0000","last_became_active":"2024-08-17T18:04:14.529024+0000","last_became_peered":"2024-08-17T18:04:14.529024+0000","last_unstale":"2024-08-17T18:09:49.395477+0000","last_undegraded":"2024-08-17T18:09:49.395477+0000","last_fullsized":"2024-08-17T18:09:49.395477+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:38:00.485543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2425,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.401834+0000","last_change":"2024-08-17T18:04:43.773667+0000","last_active":"2024-08-17T18:09:49.401834+0000","last_peered":"2024-08-17T18:09:49.401834+0000","last_clean":"2024-08-17T18:09:49.401834+0000","last_became_active":"2024-08-17T18:04:43.773247+0000","last_became_peered":"2024-08-17T18:04:43.773247+0000","last_unstale":"2024-08-17T18:09:49.401834+0000","last_undegraded":"2024-08-17T18:09:49.401834+0000","last_fullsized":"2024-08-17T18:09:49.401834+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:35:48.964631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"347'149","reported_seq":794,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.461712+0000","last_change":"2024-08-17T18:09:49.461777+0000","last_active":"2024-08-17T18:09:49.461712+0000","last_peered":"2024-08-17T18:09:49.461712+0000","last_clean":"2024-08-17T18:09:49.461712+0000","last_became_active":"2024-08-17T18:09:15.592534+0000","last_became_peered":"2024-08-17T18:09:15.592534+0000","last_unstale":"2024-08-17T18:09:49.461712+0000","last_undegraded":"2024-08-17T18:09:49.461712+0000","last_fullsized":"2024-08-17T18:09:49.461712+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:49.461698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:49.461698+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:28:31.102314+0000","scrub_duration":0.0013792520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":249,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.395785+0000","last_change":"2024-08-17T18:04:43.771589+0000","last_active":"2024-08-17T18:09:49.395785+0000","last_peered":"2024-08-17T18:09:49.395785+0000","last_clean":"2024-08-17T18:09:49.395785+0000","last_became_active":"2024-08-17T18:04:43.769601+0000","last_became_peered":"2024-08-17T18:04:43.769601+0000","last_unstale":"2024-08-17T18:09:49.395785+0000","last_undegraded":"2024-08-17T18:09:49.395785+0000","last_fullsized":"2024-08-17T18:09:49.395785+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:42:49.228921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2441,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.401600+0000","last_change":"2024-08-17T18:04:14.537136+0000","last_active":"2024-08-17T18:09:49.401600+0000","last_peered":"2024-08-17T18:09:49.401600+0000","last_clean":"2024-08-17T18:09:49.401600+0000","last_became_active":"2024-08-17T18:04:14.536815+0000","last_became_peered":"2024-08-17T18:04:14.536815+0000","last_unstale":"2024-08-17T18:09:49.401600+0000","last_undegraded":"2024-08-17T18:09:49.401600+0000","last_fullsized":"2024-08-17T18:09:49.401600+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:02.706133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":88,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.537987+0000","last_change":"2024-08-17T18:04:43.773480+0000","last_active":"2024-08-17T18:09:38.537987+0000","last_peered":"2024-08-17T18:09:38.537987+0000","last_clean":"2024-08-17T18:09:38.537987+0000","last_became_active":"2024-08-17T18:04:43.773160+0000","last_became_peered":"2024-08-17T18:04:43.773160+0000","last_unstale":"2024-08-17T18:09:38.537987+0000","last_undegraded":"2024-08-17T18:09:38.537987+0000","last_fullsized":"2024-08-17T18:09:38.537987+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:07:55.468132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"347'149","reported_seq":812,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.537386+0000","last_change":"2024-08-17T18:09:15.596647+0000","last_active":"2024-08-17T18:09:38.537386+0000","last_peered":"2024-08-17T18:09:38.537386+0000","last_clean":"2024-08-17T18:09:38.537386+0000","last_became_active":"2024-08-17T18:09:15.596339+0000","last_became_peered":"2024-08-17T18:09:15.596339+0000","last_unstale":"2024-08-17T18:09:38.537386+0000","last_undegraded":"2024-08-17T18:09:38.537386+0000","last_fullsized":"2024-08-17T18:09:38.537386+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:00.316884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:00.316884+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:30.779927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":251,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.401525+0000","last_change":"2024-08-17T18:02:08.224928+0000","last_active":"2024-08-17T18:09:49.401525+0000","last_peered":"2024-08-17T18:09:49.401525+0000","last_clean":"2024-08-17T18:09:49.401525+0000","last_became_active":"2024-08-17T18:02:08.224604+0000","last_became_peered":"2024-08-17T18:02:08.224604+0000","last_unstale":"2024-08-17T18:09:49.401525+0000","last_undegraded":"2024-08-17T18:09:49.401525+0000","last_fullsized":"2024-08-17T18:09:49.401525+0000","mapping_epoch":1421,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1422,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:55.716971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1539,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.538045+0000","last_change":"2024-08-17T18:04:14.526890+0000","last_active":"2024-08-17T18:09:38.538045+0000","last_peered":"2024-08-17T18:09:38.538045+0000","last_clean":"2024-08-17T18:09:38.538045+0000","last_became_active":"2024-08-17T18:04:14.526564+0000","last_became_peered":"2024-08-17T18:04:14.526564+0000","last_unstale":"2024-08-17T18:09:38.538045+0000","last_undegraded":"2024-08-17T18:09:38.538045+0000","last_fullsized":"2024-08-17T18:09:38.538045+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:10:13.229827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":92,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.537813+0000","last_change":"2024-08-17T18:04:43.791749+0000","last_active":"2024-08-17T18:09:38.537813+0000","last_peered":"2024-08-17T18:09:38.537813+0000","last_clean":"2024-08-17T18:09:38.537813+0000","last_became_active":"2024-08-17T18:04:43.774474+0000","last_became_peered":"2024-08-17T18:04:43.774474+0000","last_unstale":"2024-08-17T18:09:38.537813+0000","last_undegraded":"2024-08-17T18:09:38.537813+0000","last_fullsized":"2024-08-17T18:09:38.537813+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:19:08.097560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"347'149","reported_seq":826,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.391570+0000","last_change":"2024-08-17T18:09:15.596502+0000","last_active":"2024-08-17T18:09:49.391570+0000","last_peered":"2024-08-17T18:09:49.391570+0000","last_clean":"2024-08-17T18:09:49.391570+0000","last_became_active":"2024-08-17T18:09:15.595460+0000","last_became_peered":"2024-08-17T18:09:15.595460+0000","last_unstale":"2024-08-17T18:09:49.391570+0000","last_undegraded":"2024-08-17T18:09:49.391570+0000","last_fullsized":"2024-08-17T18:09:49.391570+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:02.363531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:02.363531+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:22:42.587876+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2427,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.402561+0000","last_change":"2024-08-17T18:04:43.781133+0000","last_active":"2024-08-17T18:09:49.402561+0000","last_peered":"2024-08-17T18:09:49.402561+0000","last_clean":"2024-08-17T18:09:49.402561+0000","last_became_active":"2024-08-17T18:04:43.780787+0000","last_became_peered":"2024-08-17T18:04:43.780787+0000","last_unstale":"2024-08-17T18:09:49.402561+0000","last_undegraded":"2024-08-17T18:09:49.402561+0000","last_fullsized":"2024-08-17T18:09:49.402561+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:38:34.901162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2384,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.400375+0000","last_change":"2024-08-17T18:01:23.493896+0000","last_active":"2024-08-17T18:09:49.400375+0000","last_peered":"2024-08-17T18:09:49.400375+0000","last_clean":"2024-08-17T18:09:49.400375+0000","last_became_active":"2024-08-17T18:01:23.493551+0000","last_became_peered":"2024-08-17T18:01:23.493551+0000","last_unstale":"2024-08-17T18:09:49.400375+0000","last_undegraded":"2024-08-17T18:09:49.400375+0000","last_fullsized":"2024-08-17T18:09:49.400375+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:18:47.648483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2362,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.396045+0000","last_change":"2024-08-17T18:04:14.543532+0000","last_active":"2024-08-17T18:09:49.396045+0000","last_peered":"2024-08-17T18:09:49.396045+0000","last_clean":"2024-08-17T18:09:49.396045+0000","last_became_active":"2024-08-17T18:04:14.543131+0000","last_became_peered":"2024-08-17T18:04:14.543131+0000","last_unstale":"2024-08-17T18:09:49.396045+0000","last_undegraded":"2024-08-17T18:09:49.396045+0000","last_fullsized":"2024-08-17T18:09:49.396045+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:51:44.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"347'149","reported_seq":676,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.399669+0000","last_change":"2024-08-17T18:09:15.740862+0000","last_active":"2024-08-17T18:09:49.399669+0000","last_peered":"2024-08-17T18:09:49.399669+0000","last_clean":"2024-08-17T18:09:49.399669+0000","last_became_active":"2024-08-17T18:09:15.591452+0000","last_became_peered":"2024-08-17T18:09:15.591452+0000","last_unstale":"2024-08-17T18:09:49.399669+0000","last_undegraded":"2024-08-17T18:09:49.399669+0000","last_fullsized":"2024-08-17T18:09:49.399669+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:15.740775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:15.740775+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:29:12.741753+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":192,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.391895+0000","last_change":"2024-08-17T18:07:48.539239+0000","last_active":"2024-08-17T18:09:49.391895+0000","last_peered":"2024-08-17T18:09:49.391895+0000","last_clean":"2024-08-17T18:09:49.391895+0000","last_became_active":"2024-08-17T18:07:48.538888+0000","last_became_peered":"2024-08-17T18:07:48.538888+0000","last_unstale":"2024-08-17T18:09:49.391895+0000","last_undegraded":"2024-08-17T18:09:49.391895+0000","last_fullsized":"2024-08-17T18:09:49.391895+0000","mapping_epoch":1501,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:35:23.855790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":192,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.394468+0000","last_change":"2024-08-17T18:08:11.440676+0000","last_active":"2024-08-17T18:09:49.394468+0000","last_peered":"2024-08-17T18:09:49.394468+0000","last_clean":"2024-08-17T18:09:49.394468+0000","last_became_active":"2024-08-17T18:08:11.440353+0000","last_became_peered":"2024-08-17T18:08:11.440353+0000","last_unstale":"2024-08-17T18:09:49.394468+0000","last_undegraded":"2024-08-17T18:09:49.394468+0000","last_fullsized":"2024-08-17T18:09:49.394468+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:19:22.125277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1127,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.400520+0000","last_change":"2024-08-17T18:04:14.537308+0000","last_active":"2024-08-17T18:09:49.400520+0000","last_peered":"2024-08-17T18:09:49.400520+0000","last_clean":"2024-08-17T18:09:49.400520+0000","last_became_active":"2024-08-17T18:04:14.535521+0000","last_became_peered":"2024-08-17T18:04:14.535521+0000","last_unstale":"2024-08-17T18:09:49.400520+0000","last_undegraded":"2024-08-17T18:09:49.400520+0000","last_fullsized":"2024-08-17T18:09:49.400520+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:40:00.384936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":35,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:37.647810+0000","last_change":"2024-08-17T18:08:37.071912+0000","last_active":"2024-08-17T18:09:37.647810+0000","last_peered":"2024-08-17T18:09:37.647810+0000","last_clean":"2024-08-17T18:09:37.647810+0000","last_became_active":"2024-08-17T18:08:37.071648+0000","last_became_peered":"2024-08-17T18:08:37.071648+0000","last_unstale":"2024-08-17T18:09:37.647810+0000","last_undegraded":"2024-08-17T18:09:37.647810+0000","last_fullsized":"2024-08-17T18:09:37.647810+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:58:57.279808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":189,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.395713+0000","last_change":"2024-08-17T18:04:43.771309+0000","last_active":"2024-08-17T18:09:49.395713+0000","last_peered":"2024-08-17T18:09:49.395713+0000","last_clean":"2024-08-17T18:09:49.395713+0000","last_became_active":"2024-08-17T18:04:43.770922+0000","last_became_peered":"2024-08-17T18:04:43.770922+0000","last_unstale":"2024-08-17T18:09:49.395713+0000","last_undegraded":"2024-08-17T18:09:49.395713+0000","last_fullsized":"2024-08-17T18:09:49.395713+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:10.173251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":92,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.537722+0000","last_change":"2024-08-17T18:04:43.786228+0000","last_active":"2024-08-17T18:09:38.537722+0000","last_peered":"2024-08-17T18:09:38.537722+0000","last_clean":"2024-08-17T18:09:38.537722+0000","last_became_active":"2024-08-17T18:04:43.770976+0000","last_became_peered":"2024-08-17T18:04:43.770976+0000","last_unstale":"2024-08-17T18:09:38.537722+0000","last_undegraded":"2024-08-17T18:09:38.537722+0000","last_fullsized":"2024-08-17T18:09:38.537722+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:20:10.131137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":817,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.395347+0000","last_change":"2024-08-17T18:07:54.303783+0000","last_active":"2024-08-17T18:09:49.395347+0000","last_peered":"2024-08-17T18:09:49.395347+0000","last_clean":"2024-08-17T18:09:49.395347+0000","last_became_active":"2024-08-17T18:06:57.358131+0000","last_became_peered":"2024-08-17T18:06:57.358131+0000","last_unstale":"2024-08-17T18:09:49.395347+0000","last_undegraded":"2024-08-17T18:09:49.395347+0000","last_fullsized":"2024-08-17T18:09:49.395347+0000","mapping_epoch":1460,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:54.303708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:54.303708+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:43:08.118481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"347'149","reported_seq":813,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.537626+0000","last_change":"2024-08-17T18:09:15.592722+0000","last_active":"2024-08-17T18:09:38.537626+0000","last_peered":"2024-08-17T18:09:38.537626+0000","last_clean":"2024-08-17T18:09:38.537626+0000","last_became_active":"2024-08-17T18:09:15.592242+0000","last_became_peered":"2024-08-17T18:09:15.592242+0000","last_unstale":"2024-08-17T18:09:38.537626+0000","last_undegraded":"2024-08-17T18:09:38.537626+0000","last_fullsized":"2024-08-17T18:09:38.537626+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:55.255983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:55.255983+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:02:53.596275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2370,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.402200+0000","last_change":"2024-08-17T17:24:17.455710+0000","last_active":"2024-08-17T18:09:49.402200+0000","last_peered":"2024-08-17T18:09:49.402200+0000","last_clean":"2024-08-17T18:09:49.402200+0000","last_became_active":"2024-08-17T17:24:17.452989+0000","last_became_peered":"2024-08-17T17:24:17.452989+0000","last_unstale":"2024-08-17T18:09:49.402200+0000","last_undegraded":"2024-08-17T18:09:49.402200+0000","last_fullsized":"2024-08-17T18:09:49.402200+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:58:07.894968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":74,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.398851+0000","last_change":"2024-08-17T18:08:11.347750+0000","last_active":"2024-08-17T18:09:49.398851+0000","last_peered":"2024-08-17T18:09:49.398851+0000","last_clean":"2024-08-17T18:09:49.398851+0000","last_became_active":"2024-08-17T18:08:11.346796+0000","last_became_peered":"2024-08-17T18:08:11.346796+0000","last_unstale":"2024-08-17T18:09:49.398851+0000","last_undegraded":"2024-08-17T18:09:49.398851+0000","last_fullsized":"2024-08-17T18:09:49.398851+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:24.470900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1536,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.538109+0000","last_change":"2024-08-17T18:08:37.068370+0000","last_active":"2024-08-17T18:09:38.538109+0000","last_peered":"2024-08-17T18:09:38.538109+0000","last_clean":"2024-08-17T18:09:38.538109+0000","last_became_active":"2024-08-17T18:08:37.068027+0000","last_became_peered":"2024-08-17T18:08:37.068027+0000","last_unstale":"2024-08-17T18:09:38.538109+0000","last_undegraded":"2024-08-17T18:09:38.538109+0000","last_fullsized":"2024-08-17T18:09:38.538109+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:02:12.661660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1544,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.538141+0000","last_change":"2024-08-17T18:03:51.270293+0000","last_active":"2024-08-17T18:09:38.538141+0000","last_peered":"2024-08-17T18:09:38.538141+0000","last_clean":"2024-08-17T18:09:38.538141+0000","last_became_active":"2024-08-17T18:03:51.269970+0000","last_became_peered":"2024-08-17T18:03:51.269970+0000","last_unstale":"2024-08-17T18:09:38.538141+0000","last_undegraded":"2024-08-17T18:09:38.538141+0000","last_fullsized":"2024-08-17T18:09:38.538141+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:02:30.016517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"347'149","reported_seq":813,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.537205+0000","last_change":"2024-08-17T18:09:15.590888+0000","last_active":"2024-08-17T18:09:38.537205+0000","last_peered":"2024-08-17T18:09:38.537205+0000","last_clean":"2024-08-17T18:09:38.537205+0000","last_became_active":"2024-08-17T18:09:15.590345+0000","last_became_peered":"2024-08-17T18:09:15.590345+0000","last_unstale":"2024-08-17T18:09:38.537205+0000","last_undegraded":"2024-08-17T18:09:38.537205+0000","last_fullsized":"2024-08-17T18:09:38.537205+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:59.307356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:59.307356+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:40:40.796012+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1548,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.538160+0000","last_change":"2024-08-17T18:01:23.497110+0000","last_active":"2024-08-17T18:09:38.538160+0000","last_peered":"2024-08-17T18:09:38.538160+0000","last_clean":"2024-08-17T18:09:38.538160+0000","last_became_active":"2024-08-17T18:01:23.496800+0000","last_became_peered":"2024-08-17T18:01:23.496800+0000","last_unstale":"2024-08-17T18:09:38.538160+0000","last_undegraded":"2024-08-17T18:09:38.538160+0000","last_fullsized":"2024-08-17T18:09:38.538160+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:32:29.892739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1517,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:37.647648+0000","last_change":"2024-08-17T17:23:07.401624+0000","last_active":"2024-08-17T18:09:37.647648+0000","last_peered":"2024-08-17T18:09:37.647648+0000","last_clean":"2024-08-17T18:09:37.647648+0000","last_became_active":"2024-08-17T17:23:07.400222+0000","last_became_peered":"2024-08-17T17:23:07.400222+0000","last_unstale":"2024-08-17T18:09:37.647648+0000","last_undegraded":"2024-08-17T18:09:37.647648+0000","last_fullsized":"2024-08-17T18:09:37.647648+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:07.069989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"347'149","reported_seq":820,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.399046+0000","last_change":"2024-08-17T18:09:15.592665+0000","last_active":"2024-08-17T18:09:49.399046+0000","last_peered":"2024-08-17T18:09:49.399046+0000","last_clean":"2024-08-17T18:09:49.399046+0000","last_became_active":"2024-08-17T18:09:15.590435+0000","last_became_peered":"2024-08-17T18:09:15.590435+0000","last_unstale":"2024-08-17T18:09:49.399046+0000","last_undegraded":"2024-08-17T18:09:49.399046+0000","last_fullsized":"2024-08-17T18:09:49.399046+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:52.271487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:52.271487+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:55:12.849191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1518,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.537720+0000","last_change":"2024-08-17T17:24:17.470809+0000","last_active":"2024-08-17T18:09:38.537720+0000","last_peered":"2024-08-17T18:09:38.537720+0000","last_clean":"2024-08-17T18:09:38.537720+0000","last_became_active":"2024-08-17T17:24:17.469753+0000","last_became_peered":"2024-08-17T17:24:17.469753+0000","last_unstale":"2024-08-17T18:09:38.537720+0000","last_undegraded":"2024-08-17T18:09:38.537720+0000","last_fullsized":"2024-08-17T18:09:38.537720+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:00.049425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":2379,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.399623+0000","last_change":"2024-08-17T18:04:14.552806+0000","last_active":"2024-08-17T18:09:49.399623+0000","last_peered":"2024-08-17T18:09:49.399623+0000","last_clean":"2024-08-17T18:09:49.399623+0000","last_became_active":"2024-08-17T18:04:14.535546+0000","last_became_peered":"2024-08-17T18:04:14.535546+0000","last_unstale":"2024-08-17T18:09:49.399623+0000","last_undegraded":"2024-08-17T18:09:49.399623+0000","last_fullsized":"2024-08-17T18:09:49.399623+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:44:45.702272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":178,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.391894+0000","last_change":"2024-08-17T18:04:14.539500+0000","last_active":"2024-08-17T18:09:49.391894+0000","last_peered":"2024-08-17T18:09:49.391894+0000","last_clean":"2024-08-17T18:09:49.391894+0000","last_became_active":"2024-08-17T18:04:14.538738+0000","last_became_peered":"2024-08-17T18:04:14.538738+0000","last_unstale":"2024-08-17T18:09:49.391894+0000","last_undegraded":"2024-08-17T18:09:49.391894+0000","last_fullsized":"2024-08-17T18:09:49.391894+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:31.439947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":88,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.537938+0000","last_change":"2024-08-17T18:04:43.771107+0000","last_active":"2024-08-17T18:09:38.537938+0000","last_peered":"2024-08-17T18:09:38.537938+0000","last_clean":"2024-08-17T18:09:38.537938+0000","last_became_active":"2024-08-17T18:04:43.770799+0000","last_became_peered":"2024-08-17T18:04:43.770799+0000","last_unstale":"2024-08-17T18:09:38.537938+0000","last_undegraded":"2024-08-17T18:09:38.537938+0000","last_fullsized":"2024-08-17T18:09:38.537938+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:09:04.350798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":35,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:37.647428+0000","last_change":"2024-08-17T18:08:37.080554+0000","last_active":"2024-08-17T18:09:37.647428+0000","last_peered":"2024-08-17T18:09:37.647428+0000","last_clean":"2024-08-17T18:09:37.647428+0000","last_became_active":"2024-08-17T18:08:37.079920+0000","last_became_peered":"2024-08-17T18:08:37.079920+0000","last_unstale":"2024-08-17T18:09:37.647428+0000","last_undegraded":"2024-08-17T18:09:37.647428+0000","last_fullsized":"2024-08-17T18:09:37.647428+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:08:25.214786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":92,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.538085+0000","last_change":"2024-08-17T18:04:43.781111+0000","last_active":"2024-08-17T18:09:38.538085+0000","last_peered":"2024-08-17T18:09:38.538085+0000","last_clean":"2024-08-17T18:09:38.538085+0000","last_became_active":"2024-08-17T18:04:43.771839+0000","last_became_peered":"2024-08-17T18:04:43.771839+0000","last_unstale":"2024-08-17T18:09:38.538085+0000","last_undegraded":"2024-08-17T18:09:38.538085+0000","last_fullsized":"2024-08-17T18:09:38.538085+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:52:45.973049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1552,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.537425+0000","last_change":"2024-08-17T18:08:11.449492+0000","last_active":"2024-08-17T18:09:38.537425+0000","last_peered":"2024-08-17T18:09:38.537425+0000","last_clean":"2024-08-17T18:09:38.537425+0000","last_became_active":"2024-08-17T18:08:11.424764+0000","last_became_peered":"2024-08-17T18:08:11.424764+0000","last_unstale":"2024-08-17T18:09:38.537425+0000","last_undegraded":"2024-08-17T18:09:38.537425+0000","last_fullsized":"2024-08-17T18:09:38.537425+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:53:37.552880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":197,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.396280+0000","last_change":"2024-08-17T18:04:14.551061+0000","last_active":"2024-08-17T18:09:49.396280+0000","last_peered":"2024-08-17T18:09:49.396280+0000","last_clean":"2024-08-17T18:09:49.396280+0000","last_became_active":"2024-08-17T18:04:14.543296+0000","last_became_peered":"2024-08-17T18:04:14.543296+0000","last_unstale":"2024-08-17T18:09:49.396280+0000","last_undegraded":"2024-08-17T18:09:49.396280+0000","last_fullsized":"2024-08-17T18:09:49.396280+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:39:45.294265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":106,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.537306+0000","last_change":"2024-08-17T18:08:37.068503+0000","last_active":"2024-08-17T18:09:38.537306+0000","last_peered":"2024-08-17T18:09:38.537306+0000","last_clean":"2024-08-17T18:09:38.537306+0000","last_became_active":"2024-08-17T18:08:37.068110+0000","last_became_peered":"2024-08-17T18:08:37.068110+0000","last_unstale":"2024-08-17T18:09:38.537306+0000","last_undegraded":"2024-08-17T18:09:38.537306+0000","last_fullsized":"2024-08-17T18:09:38.537306+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:12:30.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":133,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.537845+0000","last_change":"2024-08-17T18:04:14.547798+0000","last_active":"2024-08-17T18:09:38.537845+0000","last_peered":"2024-08-17T18:09:38.537845+0000","last_clean":"2024-08-17T18:09:38.537845+0000","last_became_active":"2024-08-17T18:04:14.527496+0000","last_became_peered":"2024-08-17T18:04:14.527496+0000","last_unstale":"2024-08-17T18:09:38.537845+0000","last_undegraded":"2024-08-17T18:09:38.537845+0000","last_fullsized":"2024-08-17T18:09:38.537845+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:25:54.338846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1599,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:37.647503+0000","last_change":"2024-08-17T18:04:43.813536+0000","last_active":"2024-08-17T18:09:37.647503+0000","last_peered":"2024-08-17T18:09:37.647503+0000","last_clean":"2024-08-17T18:09:37.647503+0000","last_became_active":"2024-08-17T18:04:43.799823+0000","last_became_peered":"2024-08-17T18:04:43.799823+0000","last_unstale":"2024-08-17T18:09:37.647503+0000","last_undegraded":"2024-08-17T18:09:37.647503+0000","last_fullsized":"2024-08-17T18:09:37.647503+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:04:10.837113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":74,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.399993+0000","last_change":"2024-08-17T18:08:11.345061+0000","last_active":"2024-08-17T18:09:49.399993+0000","last_peered":"2024-08-17T18:09:49.399993+0000","last_clean":"2024-08-17T18:09:49.399993+0000","last_became_active":"2024-08-17T18:08:11.344755+0000","last_became_peered":"2024-08-17T18:08:11.344755+0000","last_unstale":"2024-08-17T18:09:49.399993+0000","last_undegraded":"2024-08-17T18:09:49.399993+0000","last_fullsized":"2024-08-17T18:09:49.399993+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:23:32.161453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":257,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.537988+0000","last_change":"2024-08-17T18:04:14.527380+0000","last_active":"2024-08-17T18:09:38.537988+0000","last_peered":"2024-08-17T18:09:38.537988+0000","last_clean":"2024-08-17T18:09:38.537988+0000","last_became_active":"2024-08-17T18:04:14.527076+0000","last_became_peered":"2024-08-17T18:04:14.527076+0000","last_unstale":"2024-08-17T18:09:38.537988+0000","last_undegraded":"2024-08-17T18:09:38.537988+0000","last_fullsized":"2024-08-17T18:09:38.537988+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:36:35.583709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":178,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.392052+0000","last_change":"2024-08-17T18:04:14.541814+0000","last_active":"2024-08-17T18:09:49.392052+0000","last_peered":"2024-08-17T18:09:49.392052+0000","last_clean":"2024-08-17T18:09:49.392052+0000","last_became_active":"2024-08-17T18:04:14.541403+0000","last_became_peered":"2024-08-17T18:04:14.541403+0000","last_unstale":"2024-08-17T18:09:49.392052+0000","last_undegraded":"2024-08-17T18:09:49.392052+0000","last_fullsized":"2024-08-17T18:09:49.392052+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:07:45.294924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":192,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.391823+0000","last_change":"2024-08-17T18:08:11.351307+0000","last_active":"2024-08-17T18:09:49.391823+0000","last_peered":"2024-08-17T18:09:49.391823+0000","last_clean":"2024-08-17T18:09:49.391823+0000","last_became_active":"2024-08-17T18:08:11.350926+0000","last_became_peered":"2024-08-17T18:08:11.350926+0000","last_unstale":"2024-08-17T18:09:49.391823+0000","last_undegraded":"2024-08-17T18:09:49.391823+0000","last_fullsized":"2024-08-17T18:09:49.391823+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:59:57.477492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":88,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.538028+0000","last_change":"2024-08-17T18:04:43.771720+0000","last_active":"2024-08-17T18:09:38.538028+0000","last_peered":"2024-08-17T18:09:38.538028+0000","last_clean":"2024-08-17T18:09:38.538028+0000","last_became_active":"2024-08-17T18:04:43.771137+0000","last_became_peered":"2024-08-17T18:04:43.771137+0000","last_unstale":"2024-08-17T18:09:38.538028+0000","last_undegraded":"2024-08-17T18:09:38.538028+0000","last_fullsized":"2024-08-17T18:09:38.538028+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:57.647909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":122,"reported_epoch":1539,"state":"active+clean","last_fresh":"2024-08-17T18:09:38.537585+0000","last_change":"2024-08-17T18:03:51.283872+0000","last_active":"2024-08-17T18:09:38.537585+0000","last_peered":"2024-08-17T18:09:38.537585+0000","last_clean":"2024-08-17T18:09:38.537585+0000","last_became_active":"2024-08-17T18:03:51.267197+0000","last_became_peered":"2024-08-17T18:03:51.267197+0000","last_unstale":"2024-08-17T18:09:38.537585+0000","last_undegraded":"2024-08-17T18:09:38.537585+0000","last_fullsized":"2024-08-17T18:09:38.537585+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:18:59.157786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2368,"reported_epoch":1540,"state":"active+clean","last_fresh":"2024-08-17T18:09:49.399188+0000","last_change":"2024-08-17T18:03:51.277491+0000","last_active":"2024-08-17T18:09:49.399188+0000","last_peered":"2024-08-17T18:09:49.399188+0000","last_clean":"2024-08-17T18:09:49.399188+0000","last_became_active":"2024-08-17T18:03:51.277178+0000","last_became_peered":"2024-08-17T18:03:51.277178+0000","last_unstale":"2024-08-17T18:09:49.399188+0000","last_undegraded":"2024-08-17T18:09:49.399188+0000","last_fullsized":"2024-08-17T18:09:49.399188+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:21:35.319085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":56,"num_pg":16,"stat_sum":{"num_bytes":3301900288,"num_objects":50383,"num_object_clones":0,"num_object_copies":151149,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50383,"num_write_kb":3224512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10005708800,"data_stored":10005708800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50383,"ondisk_log_size":50383,"up":48,"acting":48,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":7994,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234728,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496366,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2385100,"kb_used_data":1600132,"kb_used_omap":1,"kb_used_meta":784958,"kb_avail":91360052,"statfs":{"total":95995035648,"available":93552693248,"internally_reserved":0,"allocated":1638535168,"data_stored":1633040047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":803797447},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565607,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2074752,"kb_used_data":1254072,"kb_used_omap":72,"kb_used_meta":820599,"kb_avail":91670400,"statfs":{"total":95995035648,"available":93870489600,"internally_reserved":0,"allocated":1284169728,"data_stored":1278682732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":840293870},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":1399,"seq":6008659247209,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1751372,"kb_used_data":1422148,"kb_used_omap":131,"kb_used_meta":329084,"kb_avail":91993780,"statfs":{"total":95995035648,"available":94201630720,"internally_reserved":0,"allocated":1456279552,"data_stored":1450785480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134291,"internal_metadata":336982893},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743186997,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1799676,"kb_used_data":1135988,"kb_used_omap":66,"kb_used_meta":663613,"kb_avail":91945476,"statfs":{"total":95995035648,"available":94152167424,"internally_reserved":0,"allocated":1163251712,"data_stored":1157768149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":679540395},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448636,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1630784,"kb_used_data":1165752,"kb_used_omap":75,"kb_used_meta":464948,"kb_avail":92114368,"statfs":{"total":95995035648,"available":94325112832,"internally_reserved":0,"allocated":1193730048,"data_stored":1188243042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":476106943},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208758001,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2563076,"kb_used_data":1729660,"kb_used_omap":1,"kb_used_meta":833406,"kb_avail":91182076,"statfs":{"total":95995035648,"available":93370445824,"internally_reserved":0,"allocated":1771171840,"data_stored":1765681247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":853408198},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019639,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2162124,"kb_used_data":1463300,"kb_used_omap":1,"kb_used_meta":698814,"kb_avail":91583028,"statfs":{"total":95995035648,"available":93781020672,"internally_reserved":0,"allocated":1498419200,"data_stored":1492924102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":715585990},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1193213952,"data_stored":1193213952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1176305664,"data_stored":1176305664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1446838272,"data_stored":1446838272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1248788480,"data_stored":1248788480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1493893120,"data_stored":1493893120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1777401856,"data_stored":1777401856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1669267456,"data_stored":1669267456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T18:10:09.354 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:09.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:09.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:09.442 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 30 16 62122 62106 129.375 98.5625 0.016606 0.00772492 2024-08-17T18:10:09.455 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:09.491 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:09.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:09.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:09.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:09.633 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:09.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:09.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:09.642 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:09 smithi136 ceph-mon[109861]: pgmap v2820: 99 pgs: 99 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-08-17T18:10:09.735 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:09.773 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:09.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:09.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:09.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:09 smithi165 ceph-mon[105517]: pgmap v2820: 99 pgs: 99 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-08-17T18:10:09.875 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:09.913 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:09.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:09.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:09.934 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:09 smithi136 ceph-mon[115078]: pgmap v2820: 99 pgs: 99 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-08-17T18:10:10.016 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:10.059 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:10.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:10.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:10.161 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:10.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:10.199 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:10.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:10.300 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:10.334 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:10.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:10.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:10.436 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:10.442 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 31 16 63501 63485 127.982 86.1875 0.00992716 0.00780822 2024-08-17T18:10:10.472 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:10.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:10.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:10.574 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:10.612 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:10.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:10.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:10.714 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:10.754 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:10.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:10.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:10.855 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:10.888 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:10.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:10.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:10.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:10:10.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:10:10.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]} : dispatch 2024-08-17T18:10:10.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:10 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]} : dispatch 2024-08-17T18:10:10.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:10:10.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:10:10.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]} : dispatch 2024-08-17T18:10:10.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:10 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]} : dispatch 2024-08-17T18:10:10.989 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:11.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:10:11.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:10:11.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]} : dispatch 2024-08-17T18:10:11.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:10 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]} : dispatch 2024-08-17T18:10:11.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:11.035 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:11.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:11.136 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:11.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:11.186 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:11.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:11.287 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:11.327 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:11.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:11.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:11.428 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:11.442 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 32 15 64880 64865 126.677 86.25 0.00864513 0.00788857 2024-08-17T18:10:11.463 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:11.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:11.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:11.565 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:11.601 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:11.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:11.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:11.703 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:11.745 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:11.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:11.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:11.847 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:11.887 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:11.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:11.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:11.989 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:11 smithi165 ceph-mon[105517]: pgmap v2821: 99 pgs: 99 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-08-17T18:10:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:11 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-08-17T18:10:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:11 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]}]': finished 2024-08-17T18:10:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:11 smithi165 ceph-mon[105517]: osdmap e1541: 8 total, 8 up, 7 in 2024-08-17T18:10:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:11 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:10:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:11 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:10:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:11 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:10:12.030 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:12.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:12.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:12.132 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:12.172 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:12.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:12.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:12.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:11 smithi136 ceph-mon[109861]: pgmap v2821: 99 pgs: 99 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-08-17T18:10:12.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-08-17T18:10:12.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]}]': finished 2024-08-17T18:10:12.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:11 smithi136 ceph-mon[109861]: osdmap e1541: 8 total, 8 up, 7 in 2024-08-17T18:10:12.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:10:12.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:10:12.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:10:12.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:11 smithi136 ceph-mon[115078]: pgmap v2821: 99 pgs: 99 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-08-17T18:10:12.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:11 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-08-17T18:10:12.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:11 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 0]}]': finished 2024-08-17T18:10:12.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:11 smithi136 ceph-mon[115078]: osdmap e1541: 8 total, 8 up, 7 in 2024-08-17T18:10:12.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:11 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:10:12.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:11 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:10:12.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:11 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:10:12.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:12.303 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:12.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:12.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:12.346 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T18:10:12.405 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:12.429 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:12.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:12.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:12.442 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 33 16 66042 66026 125.037 72.5625 0.00433737 0.0079867 2024-08-17T18:10:12.531 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:12.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:12.564 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:12.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:12.665 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:12.698 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:12.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:12.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:12.800 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:12.838 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:12.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:12.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:12.940 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:12.975 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:12.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:12.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:13.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:12 smithi165 ceph-mon[105517]: osdmap e1542: 8 total, 8 up, 7 in 2024-08-17T18:10:13.014 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:12 smithi136 ceph-mon[115078]: osdmap e1542: 8 total, 8 up, 7 in 2024-08-17T18:10:13.015 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:12 smithi136 ceph-mon[109861]: osdmap e1542: 8 total, 8 up, 7 in 2024-08-17T18:10:13.077 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:13.112 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:13.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:13.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:13.214 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:13.250 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:13.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:13.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:13.352 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:13.372 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:13.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:13.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:13.442 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 34 16 67265 67249 123.608 76.4375 0.0141667 0.00808334 2024-08-17T18:10:13.474 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:13.505 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:13.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:13.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:13.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:13.639 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:13.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:13.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:13.741 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:13.777 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:13.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:13.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:13.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:13.918 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:13.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:13.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:13 smithi165 ceph-mon[105517]: pgmap v2824: 99 pgs: 1 peering, 98 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 97 MiB/s wr, 1.56k op/s 2024-08-17T18:10:14.020 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:14.052 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:14.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:14.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:14.154 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:14.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:13 smithi136 ceph-mon[115078]: pgmap v2824: 99 pgs: 1 peering, 98 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 97 MiB/s wr, 1.56k op/s 2024-08-17T18:10:14.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:14.197 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:14.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:14.198 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:13 smithi136 ceph-mon[109861]: pgmap v2824: 99 pgs: 1 peering, 98 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 97 MiB/s wr, 1.56k op/s 2024-08-17T18:10:14.298 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:14.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:14.320 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:14.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:14.421 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:14.442 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 35 16 68555 68539 122.38 80.625 0.00290689 0.00816121 2024-08-17T18:10:14.444 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:14.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:14.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:14.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:14.580 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:14.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:14.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:14.682 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:14.717 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:14.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:14.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:14.819 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:14.846 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:14.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:14.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:14.948 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:14.984 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:14.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:14.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:15.087 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:15.109 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:15.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:15.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:15.211 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:15.245 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:15.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:15.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:15.346 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:15.380 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:15.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:15.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:15.443 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 36 16 69699 69683 120.966 71.5 0.00955443 0.00826178 2024-08-17T18:10:15.482 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:15.504 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:15.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:15.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:15.606 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:15.636 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:15.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:15.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:15.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:15.759 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:15.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:15.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:15.861 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:15.897 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:15.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:15.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:15.999 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:16.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:15 smithi165 ceph-mon[105517]: pgmap v2825: 99 pgs: 1 peering, 98 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 82 MiB/s wr, 1.30k op/s 2024-08-17T18:10:16.030 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:16.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:16.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:16.133 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:16.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:16.166 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:16.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:16.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:15 smithi136 ceph-mon[109861]: pgmap v2825: 99 pgs: 1 peering, 98 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 82 MiB/s wr, 1.30k op/s 2024-08-17T18:10:16.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:15 smithi136 ceph-mon[115078]: pgmap v2825: 99 pgs: 1 peering, 98 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 82 MiB/s wr, 1.30k op/s 2024-08-17T18:10:16.268 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:16.305 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:16.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:16.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:16.407 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:16.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:16.438 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:16.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:16.442 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 37 16 71060 71044 119.995 85.0625 0.00956782 0.00832775 2024-08-17T18:10:16.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:16.570 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:16.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:16.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:16.672 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:16.710 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:16.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:16.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:16.812 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:16.841 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:16.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:16.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:16.943 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:16.972 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:16.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:16.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:17.074 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:17.109 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:17.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:17.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:17.210 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:17.242 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:17.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:17.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:17.343 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:17.380 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:17.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:17.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:17.443 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 38 16 72198 72182 118.709 71.125 0.019001 0.00841946 2024-08-17T18:10:17.482 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:17.520 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:17.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:17.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:17.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:17.645 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:17.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:17.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:17.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:10:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:10:17] "GET /metrics HTTP/1.1" 200 36604 "" "Prometheus/2.51.0" 2024-08-17T18:10:17.747 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:17.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:17.787 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:17.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:17.888 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:17.921 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:17.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:17.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:17 smithi165 ceph-mon[105517]: pgmap v2826: 99 pgs: 99 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-08-17T18:10:18.023 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:18.061 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:18.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:18.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:18.162 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:18.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:17 smithi136 ceph-mon[109861]: pgmap v2826: 99 pgs: 99 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-08-17T18:10:18.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:17 smithi136 ceph-mon[115078]: pgmap v2826: 99 pgs: 99 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-08-17T18:10:18.198 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:18.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:18.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:18.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:18.322 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:18.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:18.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:18.424 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:18.443 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 39 16 73143 73127 117.18 59.0625 0.0100179 0.0085273 2024-08-17T18:10:18.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:18.461 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:18.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:18.561 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:18.600 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:18.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:18.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:18.702 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:18.753 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:18.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:18.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:18.768 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:10:18.855 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:18.890 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:18.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:18.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:18.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:18 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2161276600' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:10:18.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:18 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:10:18.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:18 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2161276600' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:10:18.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:18 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:10:18.992 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:19.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:18 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2161276600' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:10:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:18 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:10:19.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:19.033 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:19.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:19.134 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:19.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:19.179 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:19.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:19.280 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:19.315 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:19.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:19.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:19.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:19.443 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:2024-08-17T18:10:19.443418+0000 min lat: 0.00107817 max lat: 0.220609 avg lat: 0.00860524 2024-08-17T18:10:19.443 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T18:10:19.443 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 40 16 74354 74338 116.142 75.6875 0.0116781 0.00860524 2024-08-17T18:10:19.447 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:19.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:19.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:19.549 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:19.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:19.582 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:19.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:19.683 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:19.712 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:19.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:19.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:19.814 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:19 smithi136 ceph-mon[109861]: pgmap v2827: 99 pgs: 99 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-08-17T18:10:19.814 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:19 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:10:19.814 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:19 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T18:10:19.814 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:19 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:10:19.814 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:19 smithi136 ceph-mon[109861]: osdmap e1543: 8 total, 8 up, 7 in 2024-08-17T18:10:19.815 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:19 smithi136 ceph-mon[115078]: pgmap v2827: 99 pgs: 99 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-08-17T18:10:19.815 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:19 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:10:19.815 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:19 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T18:10:19.815 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:19 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:10:19.815 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:19 smithi136 ceph-mon[115078]: osdmap e1543: 8 total, 8 up, 7 in 2024-08-17T18:10:19.815 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:19.859 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:19.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:19.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:19.961 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:19.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:19.996 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:19.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:20.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:19 smithi165 ceph-mon[105517]: pgmap v2827: 99 pgs: 99 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-08-17T18:10:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:19 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:10:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:19 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T18:10:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:19 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:10:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:19 smithi165 ceph-mon[105517]: osdmap e1543: 8 total, 8 up, 7 in 2024-08-17T18:10:20.097 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:20.123 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:20.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:20.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:20.174 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:10:20.174 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:10:20.225 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:20.261 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:20.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:20.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:20.363 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:20.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:20.406 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:20.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:20.443 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 41 15 75563 75548 115.154 75.625 0.00636173 0.00867925 2024-08-17T18:10:20.507 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:20.541 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:20.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:20.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:20.643 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:20.677 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:20.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:20.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:20.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:20.808 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:20.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:20.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:20.910 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:20.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:20.948 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:20.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:21.049 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:21.075 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:21.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:21.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:21.082 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-17T18:10:21.177 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:21.219 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:21.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:21.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:21.320 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:21.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:21.351 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:21.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:21.356 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:10:21.399 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:10:21.400 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:10:21.441 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:10:21.441 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:10:21.443 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 42 16 76746 76730 114.17 73.875 0.0282289 0.00875363 2024-08-17T18:10:21.452 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:21.477 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:21.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:21.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:21.481 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:10:21.481 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:10:21.513 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:10:21.513 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:10:21.545 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:10:21.545 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:10:21.579 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:10:21.579 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:10:21.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:21.605 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:10:21.611 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:10:21.611 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:10:21.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:21.626 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:21.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:21.650 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:10:21.651 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:10:21.726 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:21.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:21.785 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:21.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:21.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:21.913 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:21.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:21.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:21 smithi136 ceph-mon[109861]: pgmap v2829: 99 pgs: 99 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-08-17T18:10:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:10:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:10:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:21 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2614136520' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:10:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:21 smithi136 ceph-mon[115078]: pgmap v2829: 99 pgs: 99 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-08-17T18:10:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:10:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:10:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:21 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2614136520' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:10:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:21 smithi165 ceph-mon[105517]: pgmap v2829: 99 pgs: 99 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-08-17T18:10:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:10:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:10:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:21 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2614136520' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:10:22.015 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:22.055 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:22.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:22.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:22.157 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:22.187 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:22.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:22.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:22.289 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:22.328 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:22.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:22.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:22.419 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:10:22.430 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:22.444 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 43 16 77902 77886 113.193 72.25 0.0152512 0.00882924 2024-08-17T18:10:22.458 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:22.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:22.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:22.560 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:22.596 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:22.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:22.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:22.698 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:22.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:22.738 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:22.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:22.838 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:22.875 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:22.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:22.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:22.977 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:23.008 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:23.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:23.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:23.109 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:23.152 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:23.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:23.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:23.253 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:23.286 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:23.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:23.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:23.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:23.416 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:23.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:23.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:23.444 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 44 16 79221 79205 112.494 82.4375 0.0084413 0.00888414 2024-08-17T18:10:23.518 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:23.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:23.557 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:23.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:23.658 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:23.689 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:23.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:23.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:23.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:23.821 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:23.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:23.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:23.922 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:23.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:23.963 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:23.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:23 smithi165 ceph-mon[105517]: pgmap v2830: 99 pgs: 99 active+clean; 4.6 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-08-17T18:10:24.064 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:24.101 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:24.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:24.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:24.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:23 smithi136 ceph-mon[109861]: pgmap v2830: 99 pgs: 99 active+clean; 4.6 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-08-17T18:10:24.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:23 smithi136 ceph-mon[115078]: pgmap v2830: 99 pgs: 99 active+clean; 4.6 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-08-17T18:10:24.203 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:24.242 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:24.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:24.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:24.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:24.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:24.378 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:24.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:24.445 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 45 16 80525 80509 111.805 81.5 0.0216787 0.0089377 2024-08-17T18:10:24.480 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:24.512 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:24.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:24.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:24.614 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:24.657 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:24.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:24.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:24.759 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:24.787 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:24.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:24.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:24.889 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:24.928 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:24.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:24.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:25.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:25.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:25.060 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:25.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:25.161 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:25.200 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:25.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:25.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:25.302 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:25.341 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:25.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:25.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:25.443 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:25.444 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 46 16 81742 81726 111.028 76.0625 0.00953279 0.00900155 2024-08-17T18:10:25.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:25.478 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:25.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:25.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:25.618 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:25.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:25.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:25.720 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:25.761 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:25.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:25.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:25.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:25.895 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:25.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:25.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:25.997 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:26.026 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:10:26.027 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:25 smithi136 ceph-mon[115078]: pgmap v2831: 99 pgs: 99 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-08-17T18:10:26.028 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:25 smithi136 ceph-mon[109861]: pgmap v2831: 99 pgs: 99 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-08-17T18:10:26.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:26.029 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:26.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:26.031 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T18:10:26.041 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:10:26.130 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:26.161 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:26.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:26.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:26.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:25 smithi165 ceph-mon[105517]: pgmap v2831: 99 pgs: 99 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-08-17T18:10:26.262 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:26.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:26.303 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:26.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:26.404 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:26.438 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:26.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:26.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:26.445 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 47 16 82945 82929 110.265 75.1875 0.00874586 0.00906454 2024-08-17T18:10:26.447 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:10:26.540 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:26.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:26.590 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:26.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:26.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:26.722 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:26.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:26.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:26.824 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:26.866 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:26.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:26.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:26.968 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:26.999 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:27.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:27.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:27.101 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:27.105 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:26 smithi136 ceph-mon[109861]: from='client.46055 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:10:27.105 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:26 smithi136 ceph-mon[115078]: from='client.46055 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:10:27.148 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:27.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:27.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:27.250 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:27.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:26 smithi165 ceph-mon[105517]: from='client.46055 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:10:27.283 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:27.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:27.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:27.385 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:27.421 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:27.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:27.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:27.445 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 48 16 84132 84116 109.513 74.1875 0.0080674 0.00912613 2024-08-17T18:10:27.522 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:27.551 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:27.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:27.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:27.578 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":2831,"stamp":"2024-08-17T18:10:24.316768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5072754661,"num_objects":77421,"num_object_clones":0,"num_object_copies":232263,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77459,"num_write_kb":4953313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":598400,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79093,"ondisk_log_size":79093,"up":297,"acting":297,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":298,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":19678840,"kb_used_data":14897148,"kb_used_omap":350,"kb_used_meta":4781281,"kb_avail":636537224,"statfs":{"total":671965249536,"available":651814117376,"internally_reserved":0,"allocated":15254679552,"data_stored":15216089575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":358888,"internal_metadata":4896032280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":784531456,"num_objects":11971,"num_object_clones":0,"num_object_copies":35913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11971,"num_write_kb":766144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002417"},"pg_stats":[{"pgid":"56.f","version":"1543'4739","reported_seq":4766,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:20.406333+0000","last_change":"2024-08-17T18:09:28.696402+0000","last_active":"2024-08-17T18:10:20.406333+0000","last_peered":"2024-08-17T18:10:20.406333+0000","last_clean":"2024-08-17T18:10:20.406333+0000","last_became_active":"2024-08-17T18:09:28.696135+0000","last_became_peered":"2024-08-17T18:09:28.696135+0000","last_unstale":"2024-08-17T18:10:20.406333+0000","last_undegraded":"2024-08-17T18:10:20.406333+0000","last_fullsized":"2024-08-17T18:10:20.406333+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:44:29.924728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310575104,"num_objects":4739,"num_object_clones":0,"num_object_copies":14217,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4739,"num_write_kb":303296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.e","version":"1543'4704","reported_seq":4731,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:20.423172+0000","last_change":"2024-08-17T18:09:28.697584+0000","last_active":"2024-08-17T18:10:20.423172+0000","last_peered":"2024-08-17T18:10:20.423172+0000","last_clean":"2024-08-17T18:10:20.423172+0000","last_became_active":"2024-08-17T18:09:28.697211+0000","last_became_peered":"2024-08-17T18:09:28.697211+0000","last_unstale":"2024-08-17T18:10:20.423172+0000","last_undegraded":"2024-08-17T18:10:20.423172+0000","last_fullsized":"2024-08-17T18:10:20.423172+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":4704,"log_dups_size":0,"ondisk_log_size":4704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:16:31.941416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308281344,"num_objects":4704,"num_object_clones":0,"num_object_copies":14112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4704,"num_write_kb":301056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.d","version":"1543'4758","reported_seq":4785,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:20.882064+0000","last_change":"2024-08-17T18:09:28.696225+0000","last_active":"2024-08-17T18:10:20.882064+0000","last_peered":"2024-08-17T18:10:20.882064+0000","last_clean":"2024-08-17T18:10:20.882064+0000","last_became_active":"2024-08-17T18:09:28.695916+0000","last_became_peered":"2024-08-17T18:09:28.695916+0000","last_unstale":"2024-08-17T18:10:20.882064+0000","last_undegraded":"2024-08-17T18:10:20.882064+0000","last_fullsized":"2024-08-17T18:10:20.882064+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":4758,"log_dups_size":0,"ondisk_log_size":4758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:00:20.159452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311820288,"num_objects":4758,"num_object_clones":0,"num_object_copies":14274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4758,"num_write_kb":304512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"56.c","version":"1543'4766","reported_seq":4793,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:20.440811+0000","last_change":"2024-08-17T18:09:28.627634+0000","last_active":"2024-08-17T18:10:20.440811+0000","last_peered":"2024-08-17T18:10:20.440811+0000","last_clean":"2024-08-17T18:10:20.440811+0000","last_became_active":"2024-08-17T18:09:28.627138+0000","last_became_peered":"2024-08-17T18:09:28.627138+0000","last_unstale":"2024-08-17T18:10:20.440811+0000","last_undegraded":"2024-08-17T18:10:20.440811+0000","last_fullsized":"2024-08-17T18:10:20.440811+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":4766,"log_dups_size":0,"ondisk_log_size":4766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:47:56.574797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312344576,"num_objects":4766,"num_object_clones":0,"num_object_copies":14298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4766,"num_write_kb":305024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.b","version":"1543'4892","reported_seq":4909,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:22.828386+0000","last_change":"2024-08-17T18:09:28.627309+0000","last_active":"2024-08-17T18:10:22.828386+0000","last_peered":"2024-08-17T18:10:22.828386+0000","last_clean":"2024-08-17T18:10:22.828386+0000","last_became_active":"2024-08-17T18:09:28.626970+0000","last_became_peered":"2024-08-17T18:09:28.626970+0000","last_unstale":"2024-08-17T18:10:22.828386+0000","last_undegraded":"2024-08-17T18:10:22.828386+0000","last_fullsized":"2024-08-17T18:10:22.828386+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":4892,"log_dups_size":0,"ondisk_log_size":4892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:47:24.480441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320602112,"num_objects":4892,"num_object_clones":0,"num_object_copies":14676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4892,"num_write_kb":313088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"56.a","version":"1543'4784","reported_seq":4811,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:20.411828+0000","last_change":"2024-08-17T18:09:28.697301+0000","last_active":"2024-08-17T18:10:20.411828+0000","last_peered":"2024-08-17T18:10:20.411828+0000","last_clean":"2024-08-17T18:10:20.411828+0000","last_became_active":"2024-08-17T18:09:28.697019+0000","last_became_peered":"2024-08-17T18:09:28.697019+0000","last_unstale":"2024-08-17T18:10:20.411828+0000","last_undegraded":"2024-08-17T18:10:20.411828+0000","last_fullsized":"2024-08-17T18:10:20.411828+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":4784,"log_dups_size":0,"ondisk_log_size":4784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:44:37.273534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313524224,"num_objects":4784,"num_object_clones":0,"num_object_copies":14352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4784,"num_write_kb":306176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.9","version":"1543'4822","reported_seq":4839,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:24.079419+0000","last_change":"2024-08-17T18:09:28.626545+0000","last_active":"2024-08-17T18:10:24.079419+0000","last_peered":"2024-08-17T18:10:24.079419+0000","last_clean":"2024-08-17T18:10:24.079419+0000","last_became_active":"2024-08-17T18:09:28.626300+0000","last_became_peered":"2024-08-17T18:09:28.626300+0000","last_unstale":"2024-08-17T18:10:24.079419+0000","last_undegraded":"2024-08-17T18:10:24.079419+0000","last_fullsized":"2024-08-17T18:10:24.079419+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":4822,"log_dups_size":0,"ondisk_log_size":4822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:34:50.839206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316014592,"num_objects":4822,"num_object_clones":0,"num_object_copies":14466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4822,"num_write_kb":308608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.8","version":"1543'4811","reported_seq":4836,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:20.910790+0000","last_change":"2024-08-17T18:09:28.697184+0000","last_active":"2024-08-17T18:10:20.910790+0000","last_peered":"2024-08-17T18:10:20.910790+0000","last_clean":"2024-08-17T18:10:20.910790+0000","last_became_active":"2024-08-17T18:09:28.696911+0000","last_became_peered":"2024-08-17T18:09:28.696911+0000","last_unstale":"2024-08-17T18:10:20.910790+0000","last_undegraded":"2024-08-17T18:10:20.910790+0000","last_fullsized":"2024-08-17T18:10:20.910790+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":4811,"log_dups_size":0,"ondisk_log_size":4811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:39:59.778610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315293696,"num_objects":4811,"num_object_clones":0,"num_object_copies":14433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4811,"num_write_kb":307904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"56.7","version":"1543'4715","reported_seq":4741,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:20.907275+0000","last_change":"2024-08-17T18:09:28.628236+0000","last_active":"2024-08-17T18:10:20.907275+0000","last_peered":"2024-08-17T18:10:20.907275+0000","last_clean":"2024-08-17T18:10:20.907275+0000","last_became_active":"2024-08-17T18:09:28.627952+0000","last_became_peered":"2024-08-17T18:09:28.627952+0000","last_unstale":"2024-08-17T18:10:20.907275+0000","last_undegraded":"2024-08-17T18:10:20.907275+0000","last_fullsized":"2024-08-17T18:10:20.907275+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":4715,"log_dups_size":0,"ondisk_log_size":4715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:04:20.822971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309002240,"num_objects":4715,"num_object_clones":0,"num_object_copies":14145,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4715,"num_write_kb":301760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"56.6","version":"1543'5030","reported_seq":5047,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:24.089703+0000","last_change":"2024-08-17T18:09:28.629945+0000","last_active":"2024-08-17T18:10:24.089703+0000","last_peered":"2024-08-17T18:10:24.089703+0000","last_clean":"2024-08-17T18:10:24.089703+0000","last_became_active":"2024-08-17T18:09:28.629553+0000","last_became_peered":"2024-08-17T18:09:28.629553+0000","last_unstale":"2024-08-17T18:10:24.089703+0000","last_undegraded":"2024-08-17T18:10:24.089703+0000","last_fullsized":"2024-08-17T18:10:24.089703+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":5030,"log_dups_size":0,"ondisk_log_size":5030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:00:36.875044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329646080,"num_objects":5030,"num_object_clones":0,"num_object_copies":15090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5030,"num_write_kb":321920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.5","version":"1543'4814","reported_seq":4841,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:21.823818+0000","last_change":"2024-08-17T18:09:28.697212+0000","last_active":"2024-08-17T18:10:21.823818+0000","last_peered":"2024-08-17T18:10:21.823818+0000","last_clean":"2024-08-17T18:10:21.823818+0000","last_became_active":"2024-08-17T18:09:28.696769+0000","last_became_peered":"2024-08-17T18:09:28.696769+0000","last_unstale":"2024-08-17T18:10:21.823818+0000","last_undegraded":"2024-08-17T18:10:21.823818+0000","last_fullsized":"2024-08-17T18:10:21.823818+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":4814,"log_dups_size":0,"ondisk_log_size":4814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:30:22.827483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315490304,"num_objects":4814,"num_object_clones":0,"num_object_copies":14442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4814,"num_write_kb":308096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.4","version":"1543'4701","reported_seq":4728,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:20.406294+0000","last_change":"2024-08-17T18:09:28.627340+0000","last_active":"2024-08-17T18:10:20.406294+0000","last_peered":"2024-08-17T18:10:20.406294+0000","last_clean":"2024-08-17T18:10:20.406294+0000","last_became_active":"2024-08-17T18:09:28.626966+0000","last_became_peered":"2024-08-17T18:09:28.626966+0000","last_unstale":"2024-08-17T18:10:20.406294+0000","last_undegraded":"2024-08-17T18:10:20.406294+0000","last_fullsized":"2024-08-17T18:10:20.406294+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":4701,"log_dups_size":0,"ondisk_log_size":4701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:18:20.264361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308084736,"num_objects":4701,"num_object_clones":0,"num_object_copies":14103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4701,"num_write_kb":300864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.3","version":"1543'4658","reported_seq":4685,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:20.436510+0000","last_change":"2024-08-17T18:09:28.697359+0000","last_active":"2024-08-17T18:10:20.436510+0000","last_peered":"2024-08-17T18:10:20.436510+0000","last_clean":"2024-08-17T18:10:20.436510+0000","last_became_active":"2024-08-17T18:09:28.696943+0000","last_became_peered":"2024-08-17T18:09:28.696943+0000","last_unstale":"2024-08-17T18:10:20.436510+0000","last_undegraded":"2024-08-17T18:10:20.436510+0000","last_fullsized":"2024-08-17T18:10:20.436510+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":4658,"log_dups_size":0,"ondisk_log_size":4658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:08:17.415390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305266688,"num_objects":4658,"num_object_clones":0,"num_object_copies":13974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4658,"num_write_kb":298112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.2","version":"1543'5117","reported_seq":5131,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:24.109649+0000","last_change":"2024-08-17T18:09:28.628711+0000","last_active":"2024-08-17T18:10:24.109649+0000","last_peered":"2024-08-17T18:10:24.109649+0000","last_clean":"2024-08-17T18:10:24.109649+0000","last_became_active":"2024-08-17T18:09:28.628387+0000","last_became_peered":"2024-08-17T18:09:28.628387+0000","last_unstale":"2024-08-17T18:10:24.109649+0000","last_undegraded":"2024-08-17T18:10:24.109649+0000","last_fullsized":"2024-08-17T18:10:24.109649+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":5117,"log_dups_size":0,"ondisk_log_size":5117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:26:06.412021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335347712,"num_objects":5117,"num_object_clones":0,"num_object_copies":15351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5117,"num_write_kb":327488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.1","version":"1543'5121","reported_seq":5138,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:24.108501+0000","last_change":"2024-08-17T18:09:28.629660+0000","last_active":"2024-08-17T18:10:24.108501+0000","last_peered":"2024-08-17T18:10:24.108501+0000","last_clean":"2024-08-17T18:10:24.108501+0000","last_became_active":"2024-08-17T18:09:28.629359+0000","last_became_peered":"2024-08-17T18:09:28.629359+0000","last_unstale":"2024-08-17T18:10:24.108501+0000","last_undegraded":"2024-08-17T18:10:24.108501+0000","last_fullsized":"2024-08-17T18:10:24.108501+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":5121,"log_dups_size":0,"ondisk_log_size":5121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:05:58.848611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335609856,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5121,"num_write_kb":327744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.0","version":"1543'4963","reported_seq":4980,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:24.102794+0000","last_change":"2024-08-17T18:09:28.697077+0000","last_active":"2024-08-17T18:10:24.102794+0000","last_peered":"2024-08-17T18:10:24.102794+0000","last_clean":"2024-08-17T18:10:24.102794+0000","last_became_active":"2024-08-17T18:09:28.696676+0000","last_became_peered":"2024-08-17T18:09:28.696676+0000","last_unstale":"2024-08-17T18:10:24.102794+0000","last_undegraded":"2024-08-17T18:10:24.102794+0000","last_fullsized":"2024-08-17T18:10:24.102794+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":4963,"log_dups_size":0,"ondisk_log_size":4963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:09.350570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325255168,"num_objects":4963,"num_object_clones":0,"num_object_copies":14889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4963,"num_write_kb":317632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":229,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.774596+0000","last_change":"2024-08-17T18:07:48.546202+0000","last_active":"2024-08-17T18:10:18.774596+0000","last_peered":"2024-08-17T18:10:18.774596+0000","last_clean":"2024-08-17T18:10:18.774596+0000","last_became_active":"2024-08-17T18:07:48.531246+0000","last_became_peered":"2024-08-17T18:07:48.531246+0000","last_unstale":"2024-08-17T18:10:18.774596+0000","last_undegraded":"2024-08-17T18:10:18.774596+0000","last_fullsized":"2024-08-17T18:10:18.774596+0000","mapping_epoch":1501,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:30.944362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1546,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.720282+0000","last_change":"2024-08-17T18:08:37.068490+0000","last_active":"2024-08-17T18:10:11.720282+0000","last_peered":"2024-08-17T18:10:11.720282+0000","last_clean":"2024-08-17T18:10:11.720282+0000","last_became_active":"2024-08-17T18:08:37.068026+0000","last_became_peered":"2024-08-17T18:08:37.068026+0000","last_unstale":"2024-08-17T18:10:11.720282+0000","last_undegraded":"2024-08-17T18:10:11.720282+0000","last_fullsized":"2024-08-17T18:10:11.720282+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:34:27.467963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":2405,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.778482+0000","last_change":"2024-08-17T18:08:37.082356+0000","last_active":"2024-08-17T18:10:18.778482+0000","last_peered":"2024-08-17T18:10:18.778482+0000","last_clean":"2024-08-17T18:10:18.778482+0000","last_became_active":"2024-08-17T18:08:37.082046+0000","last_became_peered":"2024-08-17T18:08:37.082046+0000","last_unstale":"2024-08-17T18:10:18.778482+0000","last_undegraded":"2024-08-17T18:10:18.778482+0000","last_fullsized":"2024-08-17T18:10:18.778482+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:46:13.932917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2428,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.779413+0000","last_change":"2024-08-17T18:04:14.540774+0000","last_active":"2024-08-17T18:10:18.779413+0000","last_peered":"2024-08-17T18:10:18.779413+0000","last_clean":"2024-08-17T18:10:18.779413+0000","last_became_active":"2024-08-17T18:04:14.539778+0000","last_became_peered":"2024-08-17T18:04:14.539778+0000","last_unstale":"2024-08-17T18:10:18.779413+0000","last_undegraded":"2024-08-17T18:10:18.779413+0000","last_fullsized":"2024-08-17T18:10:18.779413+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:55:02.863360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":95,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.722923+0000","last_change":"2024-08-17T18:04:43.777705+0000","last_active":"2024-08-17T18:10:11.722923+0000","last_peered":"2024-08-17T18:10:11.722923+0000","last_clean":"2024-08-17T18:10:11.722923+0000","last_became_active":"2024-08-17T18:04:43.771283+0000","last_became_peered":"2024-08-17T18:04:43.771283+0000","last_unstale":"2024-08-17T18:10:11.722923+0000","last_undegraded":"2024-08-17T18:10:11.722923+0000","last_fullsized":"2024-08-17T18:10:11.722923+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:13.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1672,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.778284+0000","last_change":"2024-08-17T18:04:14.534963+0000","last_active":"2024-08-17T18:10:18.778284+0000","last_peered":"2024-08-17T18:10:18.778284+0000","last_clean":"2024-08-17T18:10:18.778284+0000","last_became_active":"2024-08-17T18:04:14.534634+0000","last_became_peered":"2024-08-17T18:04:14.534634+0000","last_unstale":"2024-08-17T18:10:18.778284+0000","last_undegraded":"2024-08-17T18:10:18.778284+0000","last_fullsized":"2024-08-17T18:10:18.778284+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:33:27.287860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":95,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.722953+0000","last_change":"2024-08-17T18:04:43.773534+0000","last_active":"2024-08-17T18:10:11.722953+0000","last_peered":"2024-08-17T18:10:11.722953+0000","last_clean":"2024-08-17T18:10:11.722953+0000","last_became_active":"2024-08-17T18:04:43.768507+0000","last_became_peered":"2024-08-17T18:04:43.768507+0000","last_unstale":"2024-08-17T18:10:11.722953+0000","last_undegraded":"2024-08-17T18:10:11.722953+0000","last_fullsized":"2024-08-17T18:10:11.722953+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:50:57.647351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1534,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:12.866350+0000","last_change":"2024-08-17T17:24:17.458168+0000","last_active":"2024-08-17T18:10:12.866350+0000","last_peered":"2024-08-17T18:10:12.866350+0000","last_clean":"2024-08-17T18:10:12.866350+0000","last_became_active":"2024-08-17T17:24:17.457847+0000","last_became_peered":"2024-08-17T17:24:17.457847+0000","last_unstale":"2024-08-17T18:10:12.866350+0000","last_undegraded":"2024-08-17T18:10:12.866350+0000","last_fullsized":"2024-08-17T18:10:12.866350+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:28.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":184,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.774163+0000","last_change":"2024-08-17T18:04:14.531445+0000","last_active":"2024-08-17T18:10:18.774163+0000","last_peered":"2024-08-17T18:10:18.774163+0000","last_clean":"2024-08-17T18:10:18.774163+0000","last_became_active":"2024-08-17T18:04:14.531099+0000","last_became_peered":"2024-08-17T18:04:14.531099+0000","last_unstale":"2024-08-17T18:10:18.774163+0000","last_undegraded":"2024-08-17T18:10:18.774163+0000","last_fullsized":"2024-08-17T18:10:18.774163+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:43.335123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":95,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.777075+0000","last_change":"2024-08-17T18:07:48.548262+0000","last_active":"2024-08-17T18:10:18.777075+0000","last_peered":"2024-08-17T18:10:18.777075+0000","last_clean":"2024-08-17T18:10:18.777075+0000","last_became_active":"2024-08-17T18:07:48.547846+0000","last_became_peered":"2024-08-17T18:07:48.547846+0000","last_unstale":"2024-08-17T18:10:18.777075+0000","last_undegraded":"2024-08-17T18:10:18.777075+0000","last_fullsized":"2024-08-17T18:10:18.777075+0000","mapping_epoch":1501,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:24:37.513219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1551,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.720740+0000","last_change":"2024-08-17T18:01:23.519890+0000","last_active":"2024-08-17T18:10:11.720740+0000","last_peered":"2024-08-17T18:10:11.720740+0000","last_clean":"2024-08-17T18:10:11.720740+0000","last_became_active":"2024-08-17T18:01:23.519526+0000","last_became_peered":"2024-08-17T18:01:23.519526+0000","last_unstale":"2024-08-17T18:10:11.720740+0000","last_undegraded":"2024-08-17T18:10:11.720740+0000","last_fullsized":"2024-08-17T18:10:11.720740+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:04:52.662528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2393,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.782475+0000","last_change":"2024-08-17T18:01:23.491836+0000","last_active":"2024-08-17T18:10:18.782475+0000","last_peered":"2024-08-17T18:10:18.782475+0000","last_clean":"2024-08-17T18:10:18.782475+0000","last_became_active":"2024-08-17T18:01:23.491209+0000","last_became_peered":"2024-08-17T18:01:23.491209+0000","last_unstale":"2024-08-17T18:10:18.782475+0000","last_undegraded":"2024-08-17T18:10:18.782475+0000","last_fullsized":"2024-08-17T18:10:18.782475+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:35:30.767005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2395,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.760085+0000","last_change":"2024-08-17T17:24:17.452025+0000","last_active":"2024-08-17T18:10:18.760085+0000","last_peered":"2024-08-17T18:10:18.760085+0000","last_clean":"2024-08-17T18:10:18.760085+0000","last_became_active":"2024-08-17T17:24:17.448029+0000","last_became_peered":"2024-08-17T17:24:17.448029+0000","last_unstale":"2024-08-17T18:10:18.760085+0000","last_undegraded":"2024-08-17T18:10:18.760085+0000","last_fullsized":"2024-08-17T18:10:18.760085+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:45.089346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":199,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.774393+0000","last_change":"2024-08-17T18:08:11.358013+0000","last_active":"2024-08-17T18:10:18.774393+0000","last_peered":"2024-08-17T18:10:18.774393+0000","last_clean":"2024-08-17T18:10:18.774393+0000","last_became_active":"2024-08-17T18:08:11.357666+0000","last_became_peered":"2024-08-17T18:08:11.357666+0000","last_unstale":"2024-08-17T18:10:18.774393+0000","last_undegraded":"2024-08-17T18:10:18.774393+0000","last_fullsized":"2024-08-17T18:10:18.774393+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:05:25.290465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2408,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.778871+0000","last_change":"2024-08-17T18:04:14.539660+0000","last_active":"2024-08-17T18:10:18.778871+0000","last_peered":"2024-08-17T18:10:18.778871+0000","last_clean":"2024-08-17T18:10:18.778871+0000","last_became_active":"2024-08-17T18:04:14.537852+0000","last_became_peered":"2024-08-17T18:04:14.537852+0000","last_unstale":"2024-08-17T18:10:18.778871+0000","last_undegraded":"2024-08-17T18:10:18.778871+0000","last_fullsized":"2024-08-17T18:10:18.778871+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:47:45.026997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1547,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.718649+0000","last_change":"2024-08-17T18:04:43.769892+0000","last_active":"2024-08-17T18:10:11.718649+0000","last_peered":"2024-08-17T18:10:11.718649+0000","last_clean":"2024-08-17T18:10:11.718649+0000","last_became_active":"2024-08-17T18:04:43.769702+0000","last_became_peered":"2024-08-17T18:04:43.769702+0000","last_unstale":"2024-08-17T18:10:11.718649+0000","last_undegraded":"2024-08-17T18:10:11.718649+0000","last_fullsized":"2024-08-17T18:10:11.718649+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:39.373904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2367,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.778958+0000","last_change":"2024-08-17T17:24:17.442338+0000","last_active":"2024-08-17T18:10:18.778958+0000","last_peered":"2024-08-17T18:10:18.778958+0000","last_clean":"2024-08-17T18:10:18.778958+0000","last_became_active":"2024-08-17T17:24:17.441948+0000","last_became_peered":"2024-08-17T17:24:17.441948+0000","last_unstale":"2024-08-17T18:10:18.778958+0000","last_undegraded":"2024-08-17T18:10:18.778958+0000","last_fullsized":"2024-08-17T18:10:18.778958+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:17.891492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":191,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.774742+0000","last_change":"2024-08-17T18:04:43.777736+0000","last_active":"2024-08-17T18:10:18.774742+0000","last_peered":"2024-08-17T18:10:18.774742+0000","last_clean":"2024-08-17T18:10:18.774742+0000","last_became_active":"2024-08-17T18:04:43.777438+0000","last_became_peered":"2024-08-17T18:04:43.777438+0000","last_unstale":"2024-08-17T18:10:18.774742+0000","last_undegraded":"2024-08-17T18:10:18.774742+0000","last_fullsized":"2024-08-17T18:10:18.774742+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:24:01.255927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2347,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.779502+0000","last_change":"2024-08-17T18:04:43.915398+0000","last_active":"2024-08-17T18:10:18.779502+0000","last_peered":"2024-08-17T18:10:18.779502+0000","last_clean":"2024-08-17T18:10:18.779502+0000","last_became_active":"2024-08-17T18:04:43.915212+0000","last_became_peered":"2024-08-17T18:04:43.915212+0000","last_unstale":"2024-08-17T18:10:18.779502+0000","last_undegraded":"2024-08-17T18:10:18.779502+0000","last_fullsized":"2024-08-17T18:10:18.779502+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:43:26.901830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":108,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.723888+0000","last_change":"2024-08-17T18:08:11.356876+0000","last_active":"2024-08-17T18:10:11.723888+0000","last_peered":"2024-08-17T18:10:11.723888+0000","last_clean":"2024-08-17T18:10:11.723888+0000","last_became_active":"2024-08-17T18:08:11.355662+0000","last_became_peered":"2024-08-17T18:08:11.355662+0000","last_unstale":"2024-08-17T18:10:11.723888+0000","last_undegraded":"2024-08-17T18:10:11.723888+0000","last_fullsized":"2024-08-17T18:10:11.723888+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:55:00.801251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":504,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.783336+0000","last_change":"2024-08-17T18:04:14.555729+0000","last_active":"2024-08-17T18:10:18.783336+0000","last_peered":"2024-08-17T18:10:18.783336+0000","last_clean":"2024-08-17T18:10:18.783336+0000","last_became_active":"2024-08-17T18:04:14.536678+0000","last_became_peered":"2024-08-17T18:04:14.536678+0000","last_unstale":"2024-08-17T18:10:18.783336+0000","last_undegraded":"2024-08-17T18:10:18.783336+0000","last_fullsized":"2024-08-17T18:10:18.783336+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:47:29.424778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2420,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.760683+0000","last_change":"2024-08-17T18:08:37.079571+0000","last_active":"2024-08-17T18:10:18.760683+0000","last_peered":"2024-08-17T18:10:18.760683+0000","last_clean":"2024-08-17T18:10:18.760683+0000","last_became_active":"2024-08-17T18:08:37.079224+0000","last_became_peered":"2024-08-17T18:08:37.079224+0000","last_unstale":"2024-08-17T18:10:18.760683+0000","last_undegraded":"2024-08-17T18:10:18.760683+0000","last_fullsized":"2024-08-17T18:10:18.760683+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:28:50.561072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1548,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.721462+0000","last_change":"2024-08-17T18:03:51.292044+0000","last_active":"2024-08-17T18:10:11.721462+0000","last_peered":"2024-08-17T18:10:11.721462+0000","last_clean":"2024-08-17T18:10:11.721462+0000","last_became_active":"2024-08-17T18:03:51.268271+0000","last_became_peered":"2024-08-17T18:03:51.268271+0000","last_unstale":"2024-08-17T18:10:11.721462+0000","last_undegraded":"2024-08-17T18:10:11.721462+0000","last_fullsized":"2024-08-17T18:10:11.721462+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:00.892410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4756,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1520,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.720771+0000","last_change":"2024-08-17T17:23:07.455919+0000","last_active":"2024-08-17T18:10:11.720771+0000","last_peered":"2024-08-17T18:10:11.720771+0000","last_clean":"2024-08-17T18:10:11.720771+0000","last_became_active":"2024-08-17T17:23:07.455284+0000","last_became_peered":"2024-08-17T17:23:07.455284+0000","last_unstale":"2024-08-17T18:10:11.720771+0000","last_undegraded":"2024-08-17T18:10:11.720771+0000","last_fullsized":"2024-08-17T18:10:11.720771+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:37:46.260775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"347'149","reported_seq":827,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.778760+0000","last_change":"2024-08-17T18:09:15.592511+0000","last_active":"2024-08-17T18:10:18.778760+0000","last_peered":"2024-08-17T18:10:18.778760+0000","last_clean":"2024-08-17T18:10:18.778760+0000","last_became_active":"2024-08-17T18:09:15.592099+0000","last_became_peered":"2024-08-17T18:09:15.592099+0000","last_unstale":"2024-08-17T18:10:18.778760+0000","last_undegraded":"2024-08-17T18:10:18.778760+0000","last_fullsized":"2024-08-17T18:10:18.778760+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:57.271286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:57.271286+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:54.841504+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1544,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.720317+0000","last_change":"2024-08-17T18:04:43.766724+0000","last_active":"2024-08-17T18:10:11.720317+0000","last_peered":"2024-08-17T18:10:11.720317+0000","last_clean":"2024-08-17T18:10:11.720317+0000","last_became_active":"2024-08-17T18:04:43.766428+0000","last_became_peered":"2024-08-17T18:04:43.766428+0000","last_unstale":"2024-08-17T18:10:11.720317+0000","last_undegraded":"2024-08-17T18:10:11.720317+0000","last_fullsized":"2024-08-17T18:10:11.720317+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:09:49.102940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1557,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.718712+0000","last_change":"2024-08-17T18:03:51.279963+0000","last_active":"2024-08-17T18:10:11.718712+0000","last_peered":"2024-08-17T18:10:11.718712+0000","last_clean":"2024-08-17T18:10:11.718712+0000","last_became_active":"2024-08-17T18:03:51.279506+0000","last_became_peered":"2024-08-17T18:03:51.279506+0000","last_unstale":"2024-08-17T18:10:11.718712+0000","last_undegraded":"2024-08-17T18:10:11.718712+0000","last_fullsized":"2024-08-17T18:10:11.718712+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:20:07.304042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":188,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.774443+0000","last_change":"2024-08-17T18:04:14.542589+0000","last_active":"2024-08-17T18:10:18.774443+0000","last_peered":"2024-08-17T18:10:18.774443+0000","last_clean":"2024-08-17T18:10:18.774443+0000","last_became_active":"2024-08-17T18:04:14.531540+0000","last_became_peered":"2024-08-17T18:04:14.531540+0000","last_unstale":"2024-08-17T18:10:18.774443+0000","last_undegraded":"2024-08-17T18:10:18.774443+0000","last_fullsized":"2024-08-17T18:10:18.774443+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:15:33.014782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"347'149","reported_seq":814,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.720559+0000","last_change":"2024-08-17T18:09:15.592395+0000","last_active":"2024-08-17T18:10:11.720559+0000","last_peered":"2024-08-17T18:10:11.720559+0000","last_clean":"2024-08-17T18:10:11.720559+0000","last_became_active":"2024-08-17T18:09:15.591773+0000","last_became_peered":"2024-08-17T18:09:15.591773+0000","last_unstale":"2024-08-17T18:10:11.720559+0000","last_undegraded":"2024-08-17T18:10:11.720559+0000","last_fullsized":"2024-08-17T18:10:11.720559+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:53.309517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:53.309517+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:50:57.110936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":81,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.778922+0000","last_change":"2024-08-17T18:08:11.349400+0000","last_active":"2024-08-17T18:10:18.778922+0000","last_peered":"2024-08-17T18:10:18.778922+0000","last_clean":"2024-08-17T18:10:18.778922+0000","last_became_active":"2024-08-17T18:08:11.348892+0000","last_became_peered":"2024-08-17T18:08:11.348892+0000","last_unstale":"2024-08-17T18:10:18.778922+0000","last_undegraded":"2024-08-17T18:10:18.778922+0000","last_fullsized":"2024-08-17T18:10:18.778922+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:52:54.819838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2377,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.766269+0000","last_change":"2024-08-17T17:23:07.428296+0000","last_active":"2024-08-17T18:10:18.766269+0000","last_peered":"2024-08-17T18:10:18.766269+0000","last_clean":"2024-08-17T18:10:18.766269+0000","last_became_active":"2024-08-17T17:23:07.428010+0000","last_became_peered":"2024-08-17T17:23:07.428010+0000","last_unstale":"2024-08-17T18:10:18.766269+0000","last_undegraded":"2024-08-17T18:10:18.766269+0000","last_fullsized":"2024-08-17T18:10:18.766269+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:45.459212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":2419,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.780177+0000","last_change":"2024-08-17T18:04:43.800633+0000","last_active":"2024-08-17T18:10:18.780177+0000","last_peered":"2024-08-17T18:10:18.780177+0000","last_clean":"2024-08-17T18:10:18.780177+0000","last_became_active":"2024-08-17T18:04:43.784612+0000","last_became_peered":"2024-08-17T18:04:43.784612+0000","last_unstale":"2024-08-17T18:10:18.780177+0000","last_undegraded":"2024-08-17T18:10:18.780177+0000","last_fullsized":"2024-08-17T18:10:18.780177+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:11.405039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"347'149","reported_seq":797,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.719188+0000","last_change":"2024-08-17T18:09:52.474670+0000","last_active":"2024-08-17T18:10:11.719188+0000","last_peered":"2024-08-17T18:10:11.719188+0000","last_clean":"2024-08-17T18:10:11.719188+0000","last_became_active":"2024-08-17T18:09:15.595011+0000","last_became_peered":"2024-08-17T18:09:15.595011+0000","last_unstale":"2024-08-17T18:10:11.719188+0000","last_undegraded":"2024-08-17T18:10:11.719188+0000","last_fullsized":"2024-08-17T18:10:11.719188+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:52.474544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:52.474544+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:38:21.347345+0000","scrub_duration":0.001886326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":191,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.774919+0000","last_change":"2024-08-17T18:04:43.805055+0000","last_active":"2024-08-17T18:10:18.774919+0000","last_peered":"2024-08-17T18:10:18.774919+0000","last_clean":"2024-08-17T18:10:18.774919+0000","last_became_active":"2024-08-17T18:04:43.804708+0000","last_became_peered":"2024-08-17T18:04:43.804708+0000","last_unstale":"2024-08-17T18:10:18.774919+0000","last_undegraded":"2024-08-17T18:10:18.774919+0000","last_fullsized":"2024-08-17T18:10:18.774919+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:47:22.253073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":184,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.774259+0000","last_change":"2024-08-17T18:04:14.529605+0000","last_active":"2024-08-17T18:10:18.774259+0000","last_peered":"2024-08-17T18:10:18.774259+0000","last_clean":"2024-08-17T18:10:18.774259+0000","last_became_active":"2024-08-17T18:04:14.529024+0000","last_became_peered":"2024-08-17T18:04:14.529024+0000","last_unstale":"2024-08-17T18:10:18.774259+0000","last_undegraded":"2024-08-17T18:10:18.774259+0000","last_fullsized":"2024-08-17T18:10:18.774259+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:38:00.485543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2431,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.760643+0000","last_change":"2024-08-17T18:04:43.773667+0000","last_active":"2024-08-17T18:10:18.760643+0000","last_peered":"2024-08-17T18:10:18.760643+0000","last_clean":"2024-08-17T18:10:18.760643+0000","last_became_active":"2024-08-17T18:04:43.773247+0000","last_became_peered":"2024-08-17T18:04:43.773247+0000","last_unstale":"2024-08-17T18:10:18.760643+0000","last_undegraded":"2024-08-17T18:10:18.760643+0000","last_fullsized":"2024-08-17T18:10:18.760643+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:35:48.964631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"347'149","reported_seq":797,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.718640+0000","last_change":"2024-08-17T18:09:49.461777+0000","last_active":"2024-08-17T18:10:11.718640+0000","last_peered":"2024-08-17T18:10:11.718640+0000","last_clean":"2024-08-17T18:10:11.718640+0000","last_became_active":"2024-08-17T18:09:15.592534+0000","last_became_peered":"2024-08-17T18:09:15.592534+0000","last_unstale":"2024-08-17T18:10:11.718640+0000","last_undegraded":"2024-08-17T18:10:11.718640+0000","last_fullsized":"2024-08-17T18:10:11.718640+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:49.461698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:49.461698+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:28:31.102314+0000","scrub_duration":0.0013792520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":255,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.776475+0000","last_change":"2024-08-17T18:04:43.771589+0000","last_active":"2024-08-17T18:10:18.776475+0000","last_peered":"2024-08-17T18:10:18.776475+0000","last_clean":"2024-08-17T18:10:18.776475+0000","last_became_active":"2024-08-17T18:04:43.769601+0000","last_became_peered":"2024-08-17T18:04:43.769601+0000","last_unstale":"2024-08-17T18:10:18.776475+0000","last_undegraded":"2024-08-17T18:10:18.776475+0000","last_fullsized":"2024-08-17T18:10:18.776475+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:42:49.228921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2447,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.760447+0000","last_change":"2024-08-17T18:04:14.537136+0000","last_active":"2024-08-17T18:10:18.760447+0000","last_peered":"2024-08-17T18:10:18.760447+0000","last_clean":"2024-08-17T18:10:18.760447+0000","last_became_active":"2024-08-17T18:04:14.536815+0000","last_became_peered":"2024-08-17T18:04:14.536815+0000","last_unstale":"2024-08-17T18:10:18.760447+0000","last_undegraded":"2024-08-17T18:10:18.760447+0000","last_fullsized":"2024-08-17T18:10:18.760447+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:02.706133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":91,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.722999+0000","last_change":"2024-08-17T18:04:43.773480+0000","last_active":"2024-08-17T18:10:11.722999+0000","last_peered":"2024-08-17T18:10:11.722999+0000","last_clean":"2024-08-17T18:10:11.722999+0000","last_became_active":"2024-08-17T18:04:43.773160+0000","last_became_peered":"2024-08-17T18:04:43.773160+0000","last_unstale":"2024-08-17T18:10:11.722999+0000","last_undegraded":"2024-08-17T18:10:11.722999+0000","last_fullsized":"2024-08-17T18:10:11.722999+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:07:55.468132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"347'149","reported_seq":815,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.724084+0000","last_change":"2024-08-17T18:09:15.596647+0000","last_active":"2024-08-17T18:10:11.724084+0000","last_peered":"2024-08-17T18:10:11.724084+0000","last_clean":"2024-08-17T18:10:11.724084+0000","last_became_active":"2024-08-17T18:09:15.596339+0000","last_became_peered":"2024-08-17T18:09:15.596339+0000","last_unstale":"2024-08-17T18:10:11.724084+0000","last_undegraded":"2024-08-17T18:10:11.724084+0000","last_fullsized":"2024-08-17T18:10:11.724084+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:00.316884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:00.316884+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:30.779927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":257,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.760368+0000","last_change":"2024-08-17T18:02:08.224928+0000","last_active":"2024-08-17T18:10:18.760368+0000","last_peered":"2024-08-17T18:10:18.760368+0000","last_clean":"2024-08-17T18:10:18.760368+0000","last_became_active":"2024-08-17T18:02:08.224604+0000","last_became_peered":"2024-08-17T18:02:08.224604+0000","last_unstale":"2024-08-17T18:10:18.760368+0000","last_undegraded":"2024-08-17T18:10:18.760368+0000","last_fullsized":"2024-08-17T18:10:18.760368+0000","mapping_epoch":1421,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1422,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:55.716971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1542,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.720834+0000","last_change":"2024-08-17T18:04:14.526890+0000","last_active":"2024-08-17T18:10:11.720834+0000","last_peered":"2024-08-17T18:10:11.720834+0000","last_clean":"2024-08-17T18:10:11.720834+0000","last_became_active":"2024-08-17T18:04:14.526564+0000","last_became_peered":"2024-08-17T18:04:14.526564+0000","last_unstale":"2024-08-17T18:10:11.720834+0000","last_undegraded":"2024-08-17T18:10:11.720834+0000","last_fullsized":"2024-08-17T18:10:11.720834+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:10:13.229827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":95,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.722812+0000","last_change":"2024-08-17T18:04:43.791749+0000","last_active":"2024-08-17T18:10:11.722812+0000","last_peered":"2024-08-17T18:10:11.722812+0000","last_clean":"2024-08-17T18:10:11.722812+0000","last_became_active":"2024-08-17T18:04:43.774474+0000","last_became_peered":"2024-08-17T18:04:43.774474+0000","last_unstale":"2024-08-17T18:10:11.722812+0000","last_undegraded":"2024-08-17T18:10:11.722812+0000","last_fullsized":"2024-08-17T18:10:11.722812+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:19:08.097560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"347'149","reported_seq":832,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.776250+0000","last_change":"2024-08-17T18:09:15.596502+0000","last_active":"2024-08-17T18:10:18.776250+0000","last_peered":"2024-08-17T18:10:18.776250+0000","last_clean":"2024-08-17T18:10:18.776250+0000","last_became_active":"2024-08-17T18:09:15.595460+0000","last_became_peered":"2024-08-17T18:09:15.595460+0000","last_unstale":"2024-08-17T18:10:18.776250+0000","last_undegraded":"2024-08-17T18:10:18.776250+0000","last_fullsized":"2024-08-17T18:10:18.776250+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:02.363531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:02.363531+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:22:42.587876+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2433,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.761861+0000","last_change":"2024-08-17T18:04:43.781133+0000","last_active":"2024-08-17T18:10:18.761861+0000","last_peered":"2024-08-17T18:10:18.761861+0000","last_clean":"2024-08-17T18:10:18.761861+0000","last_became_active":"2024-08-17T18:04:43.780787+0000","last_became_peered":"2024-08-17T18:04:43.780787+0000","last_unstale":"2024-08-17T18:10:18.761861+0000","last_undegraded":"2024-08-17T18:10:18.761861+0000","last_fullsized":"2024-08-17T18:10:18.761861+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:38:34.901162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2390,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.759921+0000","last_change":"2024-08-17T18:01:23.493896+0000","last_active":"2024-08-17T18:10:18.759921+0000","last_peered":"2024-08-17T18:10:18.759921+0000","last_clean":"2024-08-17T18:10:18.759921+0000","last_became_active":"2024-08-17T18:01:23.493551+0000","last_became_peered":"2024-08-17T18:01:23.493551+0000","last_unstale":"2024-08-17T18:10:18.759921+0000","last_undegraded":"2024-08-17T18:10:18.759921+0000","last_fullsized":"2024-08-17T18:10:18.759921+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:18:47.648483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2368,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.776681+0000","last_change":"2024-08-17T18:04:14.543532+0000","last_active":"2024-08-17T18:10:18.776681+0000","last_peered":"2024-08-17T18:10:18.776681+0000","last_clean":"2024-08-17T18:10:18.776681+0000","last_became_active":"2024-08-17T18:04:14.543131+0000","last_became_peered":"2024-08-17T18:04:14.543131+0000","last_unstale":"2024-08-17T18:10:18.776681+0000","last_undegraded":"2024-08-17T18:10:18.776681+0000","last_fullsized":"2024-08-17T18:10:18.776681+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:51:44.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"347'149","reported_seq":682,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.765991+0000","last_change":"2024-08-17T18:09:15.740862+0000","last_active":"2024-08-17T18:10:18.765991+0000","last_peered":"2024-08-17T18:10:18.765991+0000","last_clean":"2024-08-17T18:10:18.765991+0000","last_became_active":"2024-08-17T18:09:15.591452+0000","last_became_peered":"2024-08-17T18:09:15.591452+0000","last_unstale":"2024-08-17T18:10:18.765991+0000","last_undegraded":"2024-08-17T18:10:18.765991+0000","last_fullsized":"2024-08-17T18:10:18.765991+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:15.740775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:15.740775+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:29:12.741753+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":205,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.774257+0000","last_change":"2024-08-17T18:10:11.727045+0000","last_active":"2024-08-17T18:10:18.774257+0000","last_peered":"2024-08-17T18:10:18.774257+0000","last_clean":"2024-08-17T18:10:18.774257+0000","last_became_active":"2024-08-17T18:10:11.726725+0000","last_became_peered":"2024-08-17T18:10:11.726725+0000","last_unstale":"2024-08-17T18:10:18.774257+0000","last_undegraded":"2024-08-17T18:10:18.774257+0000","last_fullsized":"2024-08-17T18:10:18.774257+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:12:13.835640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":198,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.776515+0000","last_change":"2024-08-17T18:08:11.440676+0000","last_active":"2024-08-17T18:10:18.776515+0000","last_peered":"2024-08-17T18:10:18.776515+0000","last_clean":"2024-08-17T18:10:18.776515+0000","last_became_active":"2024-08-17T18:08:11.440353+0000","last_became_peered":"2024-08-17T18:08:11.440353+0000","last_unstale":"2024-08-17T18:10:18.776515+0000","last_undegraded":"2024-08-17T18:10:18.776515+0000","last_fullsized":"2024-08-17T18:10:18.776515+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:19:22.125277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1134,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.760055+0000","last_change":"2024-08-17T18:04:14.537308+0000","last_active":"2024-08-17T18:10:18.760055+0000","last_peered":"2024-08-17T18:10:18.760055+0000","last_clean":"2024-08-17T18:10:18.760055+0000","last_became_active":"2024-08-17T18:04:14.535521+0000","last_became_peered":"2024-08-17T18:04:14.535521+0000","last_unstale":"2024-08-17T18:10:18.760055+0000","last_undegraded":"2024-08-17T18:10:18.760055+0000","last_fullsized":"2024-08-17T18:10:18.760055+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:40:00.384936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":38,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.718498+0000","last_change":"2024-08-17T18:08:37.071912+0000","last_active":"2024-08-17T18:10:11.718498+0000","last_peered":"2024-08-17T18:10:11.718498+0000","last_clean":"2024-08-17T18:10:11.718498+0000","last_became_active":"2024-08-17T18:08:37.071648+0000","last_became_peered":"2024-08-17T18:08:37.071648+0000","last_unstale":"2024-08-17T18:10:11.718498+0000","last_undegraded":"2024-08-17T18:10:11.718498+0000","last_fullsized":"2024-08-17T18:10:11.718498+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:58:57.279808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":195,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.776318+0000","last_change":"2024-08-17T18:04:43.771309+0000","last_active":"2024-08-17T18:10:18.776318+0000","last_peered":"2024-08-17T18:10:18.776318+0000","last_clean":"2024-08-17T18:10:18.776318+0000","last_became_active":"2024-08-17T18:04:43.770922+0000","last_became_peered":"2024-08-17T18:04:43.770922+0000","last_unstale":"2024-08-17T18:10:18.776318+0000","last_undegraded":"2024-08-17T18:10:18.776318+0000","last_fullsized":"2024-08-17T18:10:18.776318+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:10.173251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":95,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.723078+0000","last_change":"2024-08-17T18:04:43.786228+0000","last_active":"2024-08-17T18:10:11.723078+0000","last_peered":"2024-08-17T18:10:11.723078+0000","last_clean":"2024-08-17T18:10:11.723078+0000","last_became_active":"2024-08-17T18:04:43.770976+0000","last_became_peered":"2024-08-17T18:04:43.770976+0000","last_unstale":"2024-08-17T18:10:11.723078+0000","last_undegraded":"2024-08-17T18:10:11.723078+0000","last_fullsized":"2024-08-17T18:10:11.723078+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:20:10.131137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":824,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.774591+0000","last_change":"2024-08-17T18:07:54.303783+0000","last_active":"2024-08-17T18:10:18.774591+0000","last_peered":"2024-08-17T18:10:18.774591+0000","last_clean":"2024-08-17T18:10:18.774591+0000","last_became_active":"2024-08-17T18:06:57.358131+0000","last_became_peered":"2024-08-17T18:06:57.358131+0000","last_unstale":"2024-08-17T18:10:18.774591+0000","last_undegraded":"2024-08-17T18:10:18.774591+0000","last_fullsized":"2024-08-17T18:10:18.774591+0000","mapping_epoch":1460,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:54.303708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:54.303708+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:43:08.118481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"347'149","reported_seq":816,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.722597+0000","last_change":"2024-08-17T18:09:15.592722+0000","last_active":"2024-08-17T18:10:11.722597+0000","last_peered":"2024-08-17T18:10:11.722597+0000","last_clean":"2024-08-17T18:10:11.722597+0000","last_became_active":"2024-08-17T18:09:15.592242+0000","last_became_peered":"2024-08-17T18:09:15.592242+0000","last_unstale":"2024-08-17T18:10:11.722597+0000","last_undegraded":"2024-08-17T18:10:11.722597+0000","last_fullsized":"2024-08-17T18:10:11.722597+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:55.255983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:55.255983+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:02:53.596275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2376,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.761569+0000","last_change":"2024-08-17T17:24:17.455710+0000","last_active":"2024-08-17T18:10:18.761569+0000","last_peered":"2024-08-17T18:10:18.761569+0000","last_clean":"2024-08-17T18:10:18.761569+0000","last_became_active":"2024-08-17T17:24:17.452989+0000","last_became_peered":"2024-08-17T17:24:17.452989+0000","last_unstale":"2024-08-17T18:10:18.761569+0000","last_undegraded":"2024-08-17T18:10:18.761569+0000","last_fullsized":"2024-08-17T18:10:18.761569+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:58:07.894968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":81,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.778763+0000","last_change":"2024-08-17T18:08:11.347750+0000","last_active":"2024-08-17T18:10:18.778763+0000","last_peered":"2024-08-17T18:10:18.778763+0000","last_clean":"2024-08-17T18:10:18.778763+0000","last_became_active":"2024-08-17T18:08:11.346796+0000","last_became_peered":"2024-08-17T18:08:11.346796+0000","last_unstale":"2024-08-17T18:10:18.778763+0000","last_undegraded":"2024-08-17T18:10:18.778763+0000","last_fullsized":"2024-08-17T18:10:18.778763+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:24.470900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1539,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.720706+0000","last_change":"2024-08-17T18:08:37.068370+0000","last_active":"2024-08-17T18:10:11.720706+0000","last_peered":"2024-08-17T18:10:11.720706+0000","last_clean":"2024-08-17T18:10:11.720706+0000","last_became_active":"2024-08-17T18:08:37.068027+0000","last_became_peered":"2024-08-17T18:08:37.068027+0000","last_unstale":"2024-08-17T18:10:11.720706+0000","last_undegraded":"2024-08-17T18:10:11.720706+0000","last_fullsized":"2024-08-17T18:10:11.720706+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:02:12.661660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1547,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.720370+0000","last_change":"2024-08-17T18:03:51.270293+0000","last_active":"2024-08-17T18:10:11.720370+0000","last_peered":"2024-08-17T18:10:11.720370+0000","last_clean":"2024-08-17T18:10:11.720370+0000","last_became_active":"2024-08-17T18:03:51.269970+0000","last_became_peered":"2024-08-17T18:03:51.269970+0000","last_unstale":"2024-08-17T18:10:11.720370+0000","last_undegraded":"2024-08-17T18:10:11.720370+0000","last_fullsized":"2024-08-17T18:10:11.720370+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:02:30.016517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"347'149","reported_seq":816,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.726448+0000","last_change":"2024-08-17T18:09:15.590888+0000","last_active":"2024-08-17T18:10:11.726448+0000","last_peered":"2024-08-17T18:10:11.726448+0000","last_clean":"2024-08-17T18:10:11.726448+0000","last_became_active":"2024-08-17T18:09:15.590345+0000","last_became_peered":"2024-08-17T18:09:15.590345+0000","last_unstale":"2024-08-17T18:10:11.726448+0000","last_undegraded":"2024-08-17T18:10:11.726448+0000","last_fullsized":"2024-08-17T18:10:11.726448+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:59.307356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:59.307356+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:40:40.796012+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1551,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.720648+0000","last_change":"2024-08-17T18:01:23.497110+0000","last_active":"2024-08-17T18:10:11.720648+0000","last_peered":"2024-08-17T18:10:11.720648+0000","last_clean":"2024-08-17T18:10:11.720648+0000","last_became_active":"2024-08-17T18:01:23.496800+0000","last_became_peered":"2024-08-17T18:01:23.496800+0000","last_unstale":"2024-08-17T18:10:11.720648+0000","last_undegraded":"2024-08-17T18:10:11.720648+0000","last_fullsized":"2024-08-17T18:10:11.720648+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:32:29.892739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1520,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.718441+0000","last_change":"2024-08-17T17:23:07.401624+0000","last_active":"2024-08-17T18:10:11.718441+0000","last_peered":"2024-08-17T18:10:11.718441+0000","last_clean":"2024-08-17T18:10:11.718441+0000","last_became_active":"2024-08-17T17:23:07.400222+0000","last_became_peered":"2024-08-17T17:23:07.400222+0000","last_unstale":"2024-08-17T18:10:11.718441+0000","last_undegraded":"2024-08-17T18:10:11.718441+0000","last_fullsized":"2024-08-17T18:10:11.718441+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:07.069989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"347'149","reported_seq":826,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.779138+0000","last_change":"2024-08-17T18:09:15.592665+0000","last_active":"2024-08-17T18:10:18.779138+0000","last_peered":"2024-08-17T18:10:18.779138+0000","last_clean":"2024-08-17T18:10:18.779138+0000","last_became_active":"2024-08-17T18:09:15.590435+0000","last_became_peered":"2024-08-17T18:09:15.590435+0000","last_unstale":"2024-08-17T18:10:18.779138+0000","last_undegraded":"2024-08-17T18:10:18.779138+0000","last_fullsized":"2024-08-17T18:10:18.779138+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:52.271487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:52.271487+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:55:12.849191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1521,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.721377+0000","last_change":"2024-08-17T17:24:17.470809+0000","last_active":"2024-08-17T18:10:11.721377+0000","last_peered":"2024-08-17T18:10:11.721377+0000","last_clean":"2024-08-17T18:10:11.721377+0000","last_became_active":"2024-08-17T17:24:17.469753+0000","last_became_peered":"2024-08-17T17:24:17.469753+0000","last_unstale":"2024-08-17T18:10:11.721377+0000","last_undegraded":"2024-08-17T18:10:11.721377+0000","last_fullsized":"2024-08-17T18:10:11.721377+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:00.049425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":2385,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.779670+0000","last_change":"2024-08-17T18:04:14.552806+0000","last_active":"2024-08-17T18:10:18.779670+0000","last_peered":"2024-08-17T18:10:18.779670+0000","last_clean":"2024-08-17T18:10:18.779670+0000","last_became_active":"2024-08-17T18:04:14.535546+0000","last_became_peered":"2024-08-17T18:04:14.535546+0000","last_unstale":"2024-08-17T18:10:18.779670+0000","last_undegraded":"2024-08-17T18:10:18.779670+0000","last_fullsized":"2024-08-17T18:10:18.779670+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:44:45.702272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":184,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.774835+0000","last_change":"2024-08-17T18:04:14.539500+0000","last_active":"2024-08-17T18:10:18.774835+0000","last_peered":"2024-08-17T18:10:18.774835+0000","last_clean":"2024-08-17T18:10:18.774835+0000","last_became_active":"2024-08-17T18:04:14.538738+0000","last_became_peered":"2024-08-17T18:04:14.538738+0000","last_unstale":"2024-08-17T18:10:18.774835+0000","last_undegraded":"2024-08-17T18:10:18.774835+0000","last_fullsized":"2024-08-17T18:10:18.774835+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:31.439947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":91,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.723027+0000","last_change":"2024-08-17T18:04:43.771107+0000","last_active":"2024-08-17T18:10:11.723027+0000","last_peered":"2024-08-17T18:10:11.723027+0000","last_clean":"2024-08-17T18:10:11.723027+0000","last_became_active":"2024-08-17T18:04:43.770799+0000","last_became_peered":"2024-08-17T18:04:43.770799+0000","last_unstale":"2024-08-17T18:10:11.723027+0000","last_undegraded":"2024-08-17T18:10:11.723027+0000","last_fullsized":"2024-08-17T18:10:11.723027+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:09:04.350798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":38,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.718265+0000","last_change":"2024-08-17T18:08:37.080554+0000","last_active":"2024-08-17T18:10:11.718265+0000","last_peered":"2024-08-17T18:10:11.718265+0000","last_clean":"2024-08-17T18:10:11.718265+0000","last_became_active":"2024-08-17T18:08:37.079920+0000","last_became_peered":"2024-08-17T18:08:37.079920+0000","last_unstale":"2024-08-17T18:10:11.718265+0000","last_undegraded":"2024-08-17T18:10:11.718265+0000","last_fullsized":"2024-08-17T18:10:11.718265+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:08:25.214786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":95,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.723185+0000","last_change":"2024-08-17T18:04:43.781111+0000","last_active":"2024-08-17T18:10:11.723185+0000","last_peered":"2024-08-17T18:10:11.723185+0000","last_clean":"2024-08-17T18:10:11.723185+0000","last_became_active":"2024-08-17T18:04:43.771839+0000","last_became_peered":"2024-08-17T18:04:43.771839+0000","last_unstale":"2024-08-17T18:10:11.723185+0000","last_undegraded":"2024-08-17T18:10:11.723185+0000","last_fullsized":"2024-08-17T18:10:11.723185+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:52:45.973049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1555,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.720202+0000","last_change":"2024-08-17T18:08:11.449492+0000","last_active":"2024-08-17T18:10:11.720202+0000","last_peered":"2024-08-17T18:10:11.720202+0000","last_clean":"2024-08-17T18:10:11.720202+0000","last_became_active":"2024-08-17T18:08:11.424764+0000","last_became_peered":"2024-08-17T18:08:11.424764+0000","last_unstale":"2024-08-17T18:10:11.720202+0000","last_undegraded":"2024-08-17T18:10:11.720202+0000","last_fullsized":"2024-08-17T18:10:11.720202+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:53:37.552880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":203,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.777377+0000","last_change":"2024-08-17T18:04:14.551061+0000","last_active":"2024-08-17T18:10:18.777377+0000","last_peered":"2024-08-17T18:10:18.777377+0000","last_clean":"2024-08-17T18:10:18.777377+0000","last_became_active":"2024-08-17T18:04:14.543296+0000","last_became_peered":"2024-08-17T18:04:14.543296+0000","last_unstale":"2024-08-17T18:10:18.777377+0000","last_undegraded":"2024-08-17T18:10:18.777377+0000","last_fullsized":"2024-08-17T18:10:18.777377+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:39:45.294265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":109,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.726557+0000","last_change":"2024-08-17T18:08:37.068503+0000","last_active":"2024-08-17T18:10:11.726557+0000","last_peered":"2024-08-17T18:10:11.726557+0000","last_clean":"2024-08-17T18:10:11.726557+0000","last_became_active":"2024-08-17T18:08:37.068110+0000","last_became_peered":"2024-08-17T18:08:37.068110+0000","last_unstale":"2024-08-17T18:10:11.726557+0000","last_undegraded":"2024-08-17T18:10:11.726557+0000","last_fullsized":"2024-08-17T18:10:11.726557+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:12:30.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":136,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.721964+0000","last_change":"2024-08-17T18:04:14.547798+0000","last_active":"2024-08-17T18:10:11.721964+0000","last_peered":"2024-08-17T18:10:11.721964+0000","last_clean":"2024-08-17T18:10:11.721964+0000","last_became_active":"2024-08-17T18:04:14.527496+0000","last_became_peered":"2024-08-17T18:04:14.527496+0000","last_unstale":"2024-08-17T18:10:11.721964+0000","last_undegraded":"2024-08-17T18:10:11.721964+0000","last_fullsized":"2024-08-17T18:10:11.721964+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:25:54.338846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1602,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.718294+0000","last_change":"2024-08-17T18:04:43.813536+0000","last_active":"2024-08-17T18:10:11.718294+0000","last_peered":"2024-08-17T18:10:11.718294+0000","last_clean":"2024-08-17T18:10:11.718294+0000","last_became_active":"2024-08-17T18:04:43.799823+0000","last_became_peered":"2024-08-17T18:04:43.799823+0000","last_unstale":"2024-08-17T18:10:11.718294+0000","last_undegraded":"2024-08-17T18:10:11.718294+0000","last_fullsized":"2024-08-17T18:10:11.718294+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:04:10.837113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":87,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.779906+0000","last_change":"2024-08-17T18:10:11.737395+0000","last_active":"2024-08-17T18:10:18.779906+0000","last_peered":"2024-08-17T18:10:18.779906+0000","last_clean":"2024-08-17T18:10:18.779906+0000","last_became_active":"2024-08-17T18:10:11.736935+0000","last_became_peered":"2024-08-17T18:10:11.736935+0000","last_unstale":"2024-08-17T18:10:18.779906+0000","last_undegraded":"2024-08-17T18:10:18.779906+0000","last_fullsized":"2024-08-17T18:10:18.779906+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:32:49.395319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":261,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.767302+0000","last_change":"2024-08-17T18:04:14.527380+0000","last_active":"2024-08-17T18:10:18.767302+0000","last_peered":"2024-08-17T18:10:18.767302+0000","last_clean":"2024-08-17T18:10:18.767302+0000","last_became_active":"2024-08-17T18:04:14.527076+0000","last_became_peered":"2024-08-17T18:04:14.527076+0000","last_unstale":"2024-08-17T18:10:18.767302+0000","last_undegraded":"2024-08-17T18:10:18.767302+0000","last_fullsized":"2024-08-17T18:10:18.767302+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:36:35.583709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":184,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.774540+0000","last_change":"2024-08-17T18:04:14.541814+0000","last_active":"2024-08-17T18:10:18.774540+0000","last_peered":"2024-08-17T18:10:18.774540+0000","last_clean":"2024-08-17T18:10:18.774540+0000","last_became_active":"2024-08-17T18:04:14.541403+0000","last_became_peered":"2024-08-17T18:04:14.541403+0000","last_unstale":"2024-08-17T18:10:18.774540+0000","last_undegraded":"2024-08-17T18:10:18.774540+0000","last_fullsized":"2024-08-17T18:10:18.774540+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:07:45.294924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":198,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.776419+0000","last_change":"2024-08-17T18:08:11.351307+0000","last_active":"2024-08-17T18:10:18.776419+0000","last_peered":"2024-08-17T18:10:18.776419+0000","last_clean":"2024-08-17T18:10:18.776419+0000","last_became_active":"2024-08-17T18:08:11.350926+0000","last_became_peered":"2024-08-17T18:08:11.350926+0000","last_unstale":"2024-08-17T18:10:18.776419+0000","last_undegraded":"2024-08-17T18:10:18.776419+0000","last_fullsized":"2024-08-17T18:10:18.776419+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:59:57.477492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":91,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.723076+0000","last_change":"2024-08-17T18:04:43.771720+0000","last_active":"2024-08-17T18:10:11.723076+0000","last_peered":"2024-08-17T18:10:11.723076+0000","last_clean":"2024-08-17T18:10:11.723076+0000","last_became_active":"2024-08-17T18:04:43.771137+0000","last_became_peered":"2024-08-17T18:04:43.771137+0000","last_unstale":"2024-08-17T18:10:11.723076+0000","last_undegraded":"2024-08-17T18:10:11.723076+0000","last_fullsized":"2024-08-17T18:10:11.723076+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:57.647909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":125,"reported_epoch":1542,"state":"active+clean","last_fresh":"2024-08-17T18:10:11.720777+0000","last_change":"2024-08-17T18:03:51.283872+0000","last_active":"2024-08-17T18:10:11.720777+0000","last_peered":"2024-08-17T18:10:11.720777+0000","last_clean":"2024-08-17T18:10:11.720777+0000","last_became_active":"2024-08-17T18:03:51.267197+0000","last_became_peered":"2024-08-17T18:03:51.267197+0000","last_unstale":"2024-08-17T18:10:11.720777+0000","last_undegraded":"2024-08-17T18:10:11.720777+0000","last_fullsized":"2024-08-17T18:10:11.720777+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:18:59.157786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2374,"reported_epoch":1543,"state":"active+clean","last_fresh":"2024-08-17T18:10:18.778963+0000","last_change":"2024-08-17T18:03:51.277491+0000","last_active":"2024-08-17T18:10:18.778963+0000","last_peered":"2024-08-17T18:10:18.778963+0000","last_clean":"2024-08-17T18:10:18.778963+0000","last_became_active":"2024-08-17T18:03:51.277178+0000","last_became_peered":"2024-08-17T18:03:51.277178+0000","last_unstale":"2024-08-17T18:10:18.778963+0000","last_undegraded":"2024-08-17T18:10:18.778963+0000","last_fullsized":"2024-08-17T18:10:18.778963+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:21:35.319085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":56,"num_pg":16,"stat_sum":{"num_bytes":5072158720,"num_objects":77395,"num_object_clones":0,"num_object_copies":232185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77395,"num_write_kb":4953280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15259795456,"data_stored":15259795456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77395,"ondisk_log_size":77395,"up":48,"acting":48,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":7994,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234732,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496370,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3310012,"kb_used_data":2466804,"kb_used_omap":1,"kb_used_meta":843198,"kb_avail":90435140,"statfs":{"total":95995035648,"available":92605583360,"internally_reserved":0,"allocated":2526007296,"data_stored":2520489671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":863435207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565611,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2747696,"kb_used_data":1871464,"kb_used_omap":72,"kb_used_meta":876151,"kb_avail":90997456,"statfs":{"total":95995035648,"available":93181394944,"internally_reserved":0,"allocated":1916379136,"data_stored":1910869636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":897179118},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":1399,"seq":6008659247213,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2501628,"kb_used_data":2166836,"kb_used_omap":131,"kb_used_meta":334652,"kb_avail":91243524,"statfs":{"total":95995035648,"available":93433368576,"internally_reserved":0,"allocated":2218840064,"data_stored":2213323488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134291,"internal_metadata":342684525},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743187001,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2492016,"kb_used_data":1810276,"kb_used_omap":66,"kb_used_meta":681661,"kb_avail":91253136,"statfs":{"total":95995035648,"available":93443211264,"internally_reserved":0,"allocated":1853722624,"data_stored":1848216557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":698021547},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448640,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2293168,"kb_used_data":1823976,"kb_used_omap":75,"kb_used_meta":469108,"kb_avail":91451984,"statfs":{"total":95995035648,"available":93646831616,"internally_reserved":0,"allocated":1867751424,"data_stored":1862241914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":480366783},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208758005,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3410996,"kb_used_data":2559532,"kb_used_omap":1,"kb_used_meta":851454,"kb_avail":90334156,"statfs":{"total":95995035648,"available":92502175744,"internally_reserved":0,"allocated":2620960768,"data_stored":2615447671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":871889350},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019643,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2923324,"kb_used_data":2198260,"kb_used_omap":1,"kb_used_meta":725054,"kb_avail":90821828,"statfs":{"total":95995035648,"available":93001551872,"internally_reserved":0,"allocated":2251018240,"data_stored":2245500638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":742455750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1867579392,"data_stored":1867579392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1858273280,"data_stored":1858273280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2211905536,"data_stored":2211905536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1905721344,"data_stored":1905721344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2247622656,"data_stored":2247622656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2628255744,"data_stored":2628255744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2540437504,"data_stored":2540437504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T18:10:27.582 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-08-17T18:10:27.582 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:10:27.583 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-08-17T18:10:27.583 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-17T18:10:27.583 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-17T18:10:27.583 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-08-17T18:10:27.653 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:27.703 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:27.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:27.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:27.716 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:10:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:10:27] "GET /metrics HTTP/1.1" 200 36605 "" "Prometheus/2.51.0" 2024-08-17T18:10:27.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:27.837 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:27.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:27.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:27.939 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:27.979 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:27.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:27.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:28.081 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:28.112 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:28.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:28.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:28.116 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:10:28.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:27 smithi136 ceph-mon[109861]: pgmap v2832: 99 pgs: 99 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-08-17T18:10:28.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:27 smithi136 ceph-mon[115078]: pgmap v2832: 99 pgs: 99 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-08-17T18:10:28.214 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:28.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:27 smithi165 ceph-mon[105517]: pgmap v2832: 99 pgs: 99 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-08-17T18:10:28.264 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:28.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:28.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:28.366 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:28.417 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:28.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:28.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:28.445 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 49 16 85280 85264 108.742 71.75 0.0182688 0.00919047 2024-08-17T18:10:28.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:28.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:28.552 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:28.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:28.653 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:28.684 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:28.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:28.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:28.786 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:28.813 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:28.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:28.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:28.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:28.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:28.952 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:28.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:29.053 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:29.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:29.100 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:29.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:29.201 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:29.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:29.233 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:29.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:29.334 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:29.371 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:10:29.371 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T18:10:29.371 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-17T18:10:29.371 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-17T18:10:29.371 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:10:29.386 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:29.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:29.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:29.445 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 50 16 86470 86454 108.054 74.375 0.00752562 0.00924891 2024-08-17T18:10:29.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:29.527 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:29.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:29.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:29.629 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:29.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:29.664 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:29.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:29.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:29.801 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:29.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:29.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:29.902 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:29.934 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:29.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:29.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:30.035 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:30.076 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:30.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:30.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:30.178 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:29 smithi136 ceph-mon[109861]: pgmap v2833: 99 pgs: 99 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-08-17T18:10:30.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:29 smithi136 ceph-mon[115078]: pgmap v2833: 99 pgs: 99 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-08-17T18:10:30.218 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:30.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:30.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:29 smithi165 ceph-mon[105517]: pgmap v2833: 99 pgs: 99 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-08-17T18:10:30.320 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:30.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:30.369 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:30.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:30.445 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 51 16 87629 87613 107.356 72.4375 0.0108935 0.00931011 2024-08-17T18:10:30.470 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:30.495 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:30.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:30.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:30.539 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-17T18:10:30.639 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:30.696 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:30.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:30.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:30.797 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:30.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:30.857 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:30.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:30.918 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:10:30.958 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:30.990 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:30.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:30.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:31.092 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:31.113 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:31.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:31.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:31.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:30 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1594676114' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:10:31.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:30 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1594676114' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:10:31.215 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:31.244 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:31.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:31.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:31.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:30 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1594676114' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:10:31.346 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:31.379 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:31.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:31.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:31.445 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 52 16 88694 88678 106.571 66.5625 0.00504212 0.00937799 2024-08-17T18:10:31.447 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:10:31.480 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:31.507 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:31.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:31.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:31.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:31.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:31.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:31.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:31.732 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:31.768 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:31.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:31.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:31.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:31.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:31.900 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:31.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:32.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:32.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:32.043 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:32.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:32.144 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:32.175 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:32.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:32.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:32.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:31 smithi136 ceph-mon[115078]: pgmap v2834: 99 pgs: 99 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-08-17T18:10:32.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:31 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:10:32.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:31 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1594676114' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:10:32.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:31 smithi136 ceph-mon[115078]: osdmap e1544: 8 total, 8 up, 7 in 2024-08-17T18:10:32.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:31 smithi136 ceph-mon[109861]: pgmap v2834: 99 pgs: 99 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-08-17T18:10:32.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:31 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:10:32.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:31 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1594676114' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:10:32.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:31 smithi136 ceph-mon[109861]: osdmap e1544: 8 total, 8 up, 7 in 2024-08-17T18:10:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:31 smithi165 ceph-mon[105517]: pgmap v2834: 99 pgs: 99 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-08-17T18:10:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:31 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:10:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:31 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1594676114' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:10:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:31 smithi165 ceph-mon[105517]: osdmap e1544: 8 total, 8 up, 7 in 2024-08-17T18:10:32.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:32.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:32.312 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:32.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:32.413 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:32.445 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 53 16 89606 89590 105.636 57 0.00664376 0.00946136 2024-08-17T18:10:32.447 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:32.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:32.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:32.549 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:32.571 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:32.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:32.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:32.673 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:32.704 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:32.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:32.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:32.806 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:32.842 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:32.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:32.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:32.944 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:32.971 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:32.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:32.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:33.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:33.106 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:33.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:33.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:33.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:33.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:33.242 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:33.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:33.342 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:33.381 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:33.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:33.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:33.445 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 54 16 90668 90652 104.909 66.375 0.0160012 0.00952679 2024-08-17T18:10:33.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:33.519 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:33.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:33.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:33.621 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:33.656 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:33.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:33.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:33.758 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:33.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:33.798 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:33.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:33.899 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:33.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:33.942 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:33.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:34.046 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:34.067 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:34.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:34.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:34.169 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:34.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:33 smithi136 ceph-mon[109861]: pgmap v2836: 99 pgs: 99 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 75 MiB/s wr, 1.21k op/s 2024-08-17T18:10:34.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:33 smithi136 ceph-mon[115078]: pgmap v2836: 99 pgs: 99 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 75 MiB/s wr, 1.21k op/s 2024-08-17T18:10:34.203 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:34.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:34.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:33 smithi165 ceph-mon[105517]: pgmap v2836: 99 pgs: 99 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 75 MiB/s wr, 1.21k op/s 2024-08-17T18:10:34.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:34.330 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:34.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:34.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:34.432 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:34.446 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 55 16 91654 91638 104.122 61.625 0.00804235 0.00959931 2024-08-17T18:10:34.462 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:34.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:34.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:34.564 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:34.601 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:34.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:34.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:34.702 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:34.736 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:34.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:34.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:34.839 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:34.868 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:34.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:34.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:34.970 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:35.001 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:35.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:35.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:35.103 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:35.149 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:35.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:35.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:35.251 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:35.303 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:35.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:35.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:35.405 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:35.437 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:35.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:35.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:35.446 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 56 16 92846 92830 103.593 74.5 0.0171 0.00964748 2024-08-17T18:10:35.539 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:35.585 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:35.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:35.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:35.687 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:35.714 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:35.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:35.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:35.816 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:35.847 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:35.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:35.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:35.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:35 smithi136 ceph-mon[109861]: pgmap v2837: 99 pgs: 99 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-08-17T18:10:35.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:35 smithi136 ceph-mon[115078]: pgmap v2837: 99 pgs: 99 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-08-17T18:10:35.950 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:35.979 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:35.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:35.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:36.081 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:36.112 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:36.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:36.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:36.213 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:36.243 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:36.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:36.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:36.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:35 smithi165 ceph-mon[105517]: pgmap v2837: 99 pgs: 99 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-08-17T18:10:36.345 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:36.376 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:36.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:36.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:36.446 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 57 16 93970 93954 103.007 70.25 0.00834868 0.0097036 2024-08-17T18:10:36.478 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:36.507 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:36.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:36.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:36.608 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:36.643 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:36.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:36.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:36.745 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:36.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:36.786 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:36.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:36.887 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:36.920 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:36.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:36.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:37.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:37.063 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:37.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:37.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:37.165 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:37.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:37.205 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:37.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:37.301 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:10:37.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:37.315 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:10:37.315 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:10:37.315 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:37 smithi136 ceph-mon[109861]: pgmap v2838: 99 pgs: 99 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-08-17T18:10:37.315 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:10:37.315 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:10:37.315 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:37 smithi136 ceph-mon[115078]: pgmap v2838: 99 pgs: 99 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-08-17T18:10:37.335 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:37.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:37.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:37.431 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:10:37.437 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:37.452 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 58 16 95091 95075 102.438 70.0625 0.037233 0.00975564 2024-08-17T18:10:37.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:10:37.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:10:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:37 smithi165 ceph-mon[105517]: pgmap v2838: 99 pgs: 99 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-08-17T18:10:37.528 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:37.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:37.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:37.577 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:10:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:10:37] "GET /metrics HTTP/1.1" 200 36599 "" "Prometheus/2.51.0" 2024-08-17T18:10:37.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:37.661 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:37.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:37.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:37.763 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:37.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:37.806 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:37.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:37.907 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:37.930 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:37.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:37.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:38.032 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:38.067 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:38.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:38.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:38.169 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:38.205 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:38.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:38.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:38.307 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:38.346 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:38.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:38.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:38.447 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 59 16 96154 96138 101.828 66.4375 0.00963915 0.00981612 2024-08-17T18:10:38.448 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:38.483 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:38.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:38.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:38.585 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:38.616 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:38.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:38.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:38.718 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:38.751 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:38.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:38.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:38.853 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:38.891 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:38.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:38.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:38.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:39.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:39.025 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:39.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:39.126 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:39.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:39.159 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:39.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:39.260 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:39.302 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:39.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:39.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:39.388 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:10:39.389 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:10:39.404 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:39.434 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:10:39.434 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:10:39.442 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:39.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:39.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:39.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:39 smithi136 ceph-mon[109861]: pgmap v2839: 99 pgs: 99 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-08-17T18:10:39.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:39 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4037075050' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:10:39.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:39 smithi136 ceph-mon[115078]: pgmap v2839: 99 pgs: 99 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-08-17T18:10:39.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:39 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4037075050' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:10:39.447 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:2024-08-17T18:10:39.447338+0000 min lat: 0.00107817 max lat: 0.220609 avg lat: 0.00987218 2024-08-17T18:10:39.447 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T18:10:39.447 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 60 16 97209 97193 101.23 65.9375 0.00734064 0.00987218 2024-08-17T18:10:39.468 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:10:39.468 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:10:39.501 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:10:39.501 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:10:39.533 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:10:39.534 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:10:39.544 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:39.568 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:10:39.568 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:10:39.584 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:39.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:39.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:39.609 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:10:39.609 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:10:39.644 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:10:39.644 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:10:39.686 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:39.728 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:39.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:39.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:39 smithi165 ceph-mon[105517]: pgmap v2839: 99 pgs: 99 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-08-17T18:10:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:39 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4037075050' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:10:39.829 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:39.875 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:39.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:39.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:39.979 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:40.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:40.013 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:40.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:40.114 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:40.144 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:40.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:40.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:40.246 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:40.287 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:40.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:40.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:40.389 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:40.422 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:40.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:40.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:40.447 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 61 16 98297 98281 100.685 68 0.0140604 0.00992714 2024-08-17T18:10:40.524 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:40.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:40.559 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:40.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:40.660 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:40.703 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:40.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:40.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:40.794 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:10:40.795 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-17T18:10:40.795 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:10:40.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:40.834 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:40.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:40.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:40.936 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:40.978 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:40.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:40.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:41.080 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:41.111 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:41.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:41.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:41.213 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:41.242 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:41.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:41.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:41.343 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:41.379 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:41.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:41.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:41.447 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 62 16 99384 99368 100.156 67.9375 0.013357 0.00997934 2024-08-17T18:10:41.481 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:41.526 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:41.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:41.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:41.628 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:41.659 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:41.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:41.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:41.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:41 smithi136 ceph-mon[109861]: pgmap v2840: 99 pgs: 99 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-08-17T18:10:41.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:41 smithi136 ceph-mon[115078]: pgmap v2840: 99 pgs: 99 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-08-17T18:10:41.760 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:41.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:41 smithi165 ceph-mon[105517]: pgmap v2840: 99 pgs: 99 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-08-17T18:10:41.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:41.804 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:41.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:41.905 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:41.941 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:41.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:41.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:42.043 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:42.087 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:42.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:42.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:42.189 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:42.233 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:42.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:42.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:42.335 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:42.374 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:42.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:42.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:42.447 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 63 16 100514 100498 99.6875 70.625 0.01425 0.0100266 2024-08-17T18:10:42.476 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:42.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:42.524 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:42.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:42.625 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:42.666 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:42.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:42.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:42.768 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:42.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:42.804 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:42.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:42.905 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:42.943 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:42.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:42.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:43.009 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:10:43.045 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:43.068 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:10:43.068 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:10:43.074 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:43.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:43.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:43.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:43.205 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:43.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:43.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:43.307 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:43.360 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:43.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:43.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:43.447 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 64 16 101555 101539 99.1464 65.0625 0.00737892 0.0100811 2024-08-17T18:10:43.462 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:43.492 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:43.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:43.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:43.595 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:43.634 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:43.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:43.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:43.684 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:43 smithi136 ceph-mon[109861]: pgmap v2841: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-08-17T18:10:43.684 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:43 smithi136 ceph-mon[115078]: pgmap v2841: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-08-17T18:10:43.735 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:43 smithi165 ceph-mon[105517]: pgmap v2841: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-08-17T18:10:43.766 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:10:43.769 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:43.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:43.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:43.871 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:43.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:43.923 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:43.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:44.024 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:44.072 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:44.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:44.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:44.173 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:44.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:44.198 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:44.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:44.299 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:44.337 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:44.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:44.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:44.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:44.447 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 65 16 102594 102578 98.6201 64.9375 0.00766953 0.0101352 2024-08-17T18:10:44.470 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:44.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:44.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:44.572 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:44.605 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:44.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:44.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:44.707 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:44.737 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:44.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:44.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:44.839 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:44.873 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:44.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:44.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:44.929 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-17T18:10:44.974 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:45.016 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:45.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:45.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:45.118 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:45.164 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:45.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:45.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:45.266 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:45.316 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:45.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:45.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:45.418 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:45.448 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 66 16 103653 103637 98.1286 66.1875 0.00913968 0.0101855 2024-08-17T18:10:45.449 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:45.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:45.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:45.551 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:45.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:45.595 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:45.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:45.696 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:45.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:45 smithi136 ceph-mon[109861]: pgmap v2842: 99 pgs: 99 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-08-17T18:10:45.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:45 smithi136 ceph-mon[115078]: pgmap v2842: 99 pgs: 99 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-08-17T18:10:45.726 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:45.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:45.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:45 smithi165 ceph-mon[105517]: pgmap v2842: 99 pgs: 99 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-08-17T18:10:45.829 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:45.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:45.877 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:45.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:45.952 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:10:45.978 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:45.993 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:10:46.003 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:46.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:46.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:46.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:46.135 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:46.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:46.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:46.237 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:46.287 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:46.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:46.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:46.389 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:46.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:46.417 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:46.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:46.448 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 67 16 104685 104669 97.6266 64.5 0.0249371 0.0102382 2024-08-17T18:10:46.518 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:46.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:46.549 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:46.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:46.650 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:46.687 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:46.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:46.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:46.789 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:46.822 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:46.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:46.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:46.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:46.954 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:46.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:46.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:47.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:47.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:47.089 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:47.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:47.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:47.215 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:47.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:47.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:47.317 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:47.355 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:47.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:47.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:47.402 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:10:47.448 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 68 16 105637 105621 97.0659 59.5 0.0146438 0.0102971 2024-08-17T18:10:47.456 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:47.495 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:47.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:47.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:47.520 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 0, 5] out_osds: [7] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:10:47.520 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:10:47.520 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-08-17T18:10:47.520 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T18:10:47.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:47.646 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:47.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:47.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:47 smithi136 ceph-mon[109861]: pgmap v2843: 99 pgs: 99 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-08-17T18:10:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:47 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/218233584' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:10:47.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:47 smithi136 ceph-mon[115078]: pgmap v2843: 99 pgs: 99 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-08-17T18:10:47.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:47 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/218233584' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:10:47.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:10:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:10:47] "GET /metrics HTTP/1.1" 200 36599 "" "Prometheus/2.51.0" 2024-08-17T18:10:47.747 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:47 smithi165 ceph-mon[105517]: pgmap v2843: 99 pgs: 99 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-08-17T18:10:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:47 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/218233584' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:10:47.777 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:47.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:47.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:47.878 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:47.910 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:47.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:47.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:48.011 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:48.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:48.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:48.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:48.145 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:48.179 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:48.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:48.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:48.281 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:48.323 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:48.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:48.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:48.425 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:48.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:48 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:10:48.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:48 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/218233584' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:10:48.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:48 smithi136 ceph-mon[109861]: osdmap e1545: 8 total, 8 up, 7 in 2024-08-17T18:10:48.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:48 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:10:48.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:48 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/218233584' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:10:48.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:48 smithi136 ceph-mon[115078]: osdmap e1545: 8 total, 8 up, 7 in 2024-08-17T18:10:48.450 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 69 16 106633 106617 96.5605 62.25 0.0121838 0.0103506 2024-08-17T18:10:48.465 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:48.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:48.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:48.567 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:48.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:48.610 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:48.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:48.712 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:48 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:10:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:48 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/218233584' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:10:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:48 smithi165 ceph-mon[105517]: osdmap e1545: 8 total, 8 up, 7 in 2024-08-17T18:10:48.761 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:48.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:48.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:48.863 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:48.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:48.899 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:48.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:49.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:49.028 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:49.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:49.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:49.130 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:49.160 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:49.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:49.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:49.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:49.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:49.294 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:49.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:49.395 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:49.420 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:49.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:49.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:49.449 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 70 16 107545 107529 95.9946 57 0.0109382 0.0104116 2024-08-17T18:10:49.522 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:49.548 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:49.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:49.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:49.557 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:49 smithi136 ceph-mon[109861]: pgmap v2845: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-08-17T18:10:49.558 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:49 smithi136 ceph-mon[115078]: pgmap v2845: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-08-17T18:10:49.650 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:49.678 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:49.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:49.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:49 smithi165 ceph-mon[105517]: pgmap v2845: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-08-17T18:10:49.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:49.825 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:49.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:49.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:49.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:49.959 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:49.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:49.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:50.060 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:50.097 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:10:50.098 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:10:50.106 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:50.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:50.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:50.208 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:50.249 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:50.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:50.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:50.313 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:10:50.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:50.385 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:50.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:50.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:50.450 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 71 16 108502 108486 95.4839 59.8125 0.0346308 0.0104675 2024-08-17T18:10:50.487 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:50.531 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:50.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:50.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:50.633 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:50.663 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:10:50.676 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:50.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:50.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:50.777 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:50.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:50.821 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:50.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:50.922 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:50.969 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:50.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:50.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:51.052 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-17T18:10:51.071 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:51.146 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:51.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:51.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:51.248 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:51.280 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:51.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:51.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:51.382 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:51.450 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 72 16 109394 109378 94.932 55.75 0.0457155 0.0105283 2024-08-17T18:10:51.455 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:51.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:51.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:51.556 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:51.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:51.609 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:51.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:51.693 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:10:51.694 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:10:51.694 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:51 smithi136 ceph-mon[109861]: pgmap v2846: 99 pgs: 99 active+clean; 6.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-08-17T18:10:51.694 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:10:51.695 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:51 smithi136 ceph-mon[115078]: pgmap v2846: 99 pgs: 99 active+clean; 6.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-08-17T18:10:51.695 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:10:51.709 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:51.734 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:10:51.734 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:10:51.738 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:51.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:51.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:51.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:51 smithi165 ceph-mon[105517]: pgmap v2846: 99 pgs: 99 active+clean; 6.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-08-17T18:10:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:10:51.773 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:10:51.773 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:10:51.804 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:10:51.805 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:10:51.839 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:10:51.839 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:10:51.840 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:51.876 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:10:51.876 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:10:51.878 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:51.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:51.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:51.908 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:10:51.908 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:10:51.940 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:10:51.941 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:10:51.980 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:52.022 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:52.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:52.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:52.124 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:52.156 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:52.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:52.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:52.258 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:52.286 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:52.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:52.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:52.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:52.417 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:52.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:52.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:52.450 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 73 16 110321 110305 94.4251 57.9375 0.0077239 0.0105848 2024-08-17T18:10:52.519 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:52.560 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:52.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:52.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:52.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:52.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:52.692 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:52.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:52.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:52 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3479982302' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:10:52.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:52 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3479982302' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:10:52.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:52 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3479982302' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:10:52.794 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:52.848 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:52.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:52.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:52.950 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:52.994 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:52.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:52.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:53.096 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:53.141 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:53.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:53.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:53.243 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:53.276 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:53.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:53.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:53.323 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:10:53.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:53.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:53.447 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:53.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:53.450 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 74 16 111301 111285 93.9767 61.25 0.0277163 0.0106355 2024-08-17T18:10:53.548 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:53.568 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:53 smithi136 ceph-mon[109861]: pgmap v2847: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-08-17T18:10:53.568 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:53 smithi136 ceph-mon[115078]: pgmap v2847: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-08-17T18:10:53.587 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:53.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:53.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:53.689 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:53.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:53.726 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:53.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:53.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:53 smithi165 ceph-mon[105517]: pgmap v2847: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-08-17T18:10:53.826 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:53.869 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:53.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:53.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:53.970 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:54.020 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:54.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:54.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:54.122 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:54.155 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:54.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:54.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:54.257 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:54.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:54.294 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:54.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:54.395 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:54.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:54.428 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:54.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:54.450 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 75 16 112330 112314 93.5811 64.3125 0.0124017 0.0106806 2024-08-17T18:10:54.530 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:54.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:54.566 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:54.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:54.667 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:54.699 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:54.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:54.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:54.790 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:10:54.801 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:54.824 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:54.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:54.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:54.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:54.953 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:54.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:54.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:55.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:55.094 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:55.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:55.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:55.195 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:55.227 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:10:55.232 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:55.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:55.334 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:55.389 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:55.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:55.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:55.450 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 76 16 113207 113191 93.071 54.8125 0.0203772 0.0107385 2024-08-17T18:10:55.491 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:55.520 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:55.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:55.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:55.618 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:10:55.620 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T18:10:55.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:55.643 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:55 smithi136 ceph-mon[109861]: pgmap v2848: 99 pgs: 99 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 62 MiB/s wr, 999 op/s 2024-08-17T18:10:55.643 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:55 smithi136 ceph-mon[115078]: pgmap v2848: 99 pgs: 99 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 62 MiB/s wr, 999 op/s 2024-08-17T18:10:55.651 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:55.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:55.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:55.753 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:55.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:55 smithi165 ceph-mon[105517]: pgmap v2848: 99 pgs: 99 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 62 MiB/s wr, 999 op/s 2024-08-17T18:10:55.781 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:55.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:55.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:55.883 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:55.916 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:55.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:55.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:56.017 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:56.066 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:56.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:56.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:56.168 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:56.209 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:56.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:56.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:56.310 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:56.343 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:56.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:56.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:56.445 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:56.452 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 77 16 114189 114173 92.6572 61.375 0.0231492 0.0107872 2024-08-17T18:10:56.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:56.487 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:56.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:56.587 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:56.625 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:56.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:56.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:56.682 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:56 smithi136 ceph-mon[109861]: from='client.46076 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:10:56.683 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:56 smithi136 ceph-mon[115078]: from='client.46076 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:10:56.721 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":2848,"stamp":"2024-08-17T18:10:54.327060+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7203657701,"num_objects":109936,"num_object_clones":0,"num_object_copies":329808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109974,"num_write_kb":7034273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":598400,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110908,"ondisk_log_size":110908,"up":297,"acting":297,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":297,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":26027092,"kb_used_data":21113308,"kb_used_omap":350,"kb_used_meta":4913377,"kb_avail":630188972,"statfs":{"total":671965249536,"available":645313507328,"internally_reserved":0,"allocated":21620027392,"data_stored":21581332051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":358888,"internal_metadata":5031298584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":655425536,"num_objects":10001,"num_object_clones":0,"num_object_copies":30003,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10001,"num_write_kb":640064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003803"},"pg_stats":[{"pgid":"56.f","version":"1545'6884","reported_seq":6915,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:50.463114+0000","last_change":"2024-08-17T18:09:28.696402+0000","last_active":"2024-08-17T18:10:50.463114+0000","last_peered":"2024-08-17T18:10:50.463114+0000","last_clean":"2024-08-17T18:10:50.463114+0000","last_became_active":"2024-08-17T18:09:28.696135+0000","last_became_peered":"2024-08-17T18:09:28.696135+0000","last_unstale":"2024-08-17T18:10:50.463114+0000","last_undegraded":"2024-08-17T18:10:50.463114+0000","last_fullsized":"2024-08-17T18:10:50.463114+0000","mapping_epoch":1535,"log_start":"1539'200","ondisk_log_start":"1539'200","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6684,"log_dups_size":0,"ondisk_log_size":6684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:44:29.924728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451149824,"num_objects":6884,"num_object_clones":0,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6884,"num_write_kb":440576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.e","version":"1545'6781","reported_seq":6812,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:50.476161+0000","last_change":"2024-08-17T18:09:28.697584+0000","last_active":"2024-08-17T18:10:50.476161+0000","last_peered":"2024-08-17T18:10:50.476161+0000","last_clean":"2024-08-17T18:10:50.476161+0000","last_became_active":"2024-08-17T18:09:28.697211+0000","last_became_peered":"2024-08-17T18:09:28.697211+0000","last_unstale":"2024-08-17T18:10:50.476161+0000","last_undegraded":"2024-08-17T18:10:50.476161+0000","last_fullsized":"2024-08-17T18:10:50.476161+0000","mapping_epoch":1535,"log_start":"1539'100","ondisk_log_start":"1539'100","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6681,"log_dups_size":0,"ondisk_log_size":6681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:16:31.941416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444399616,"num_objects":6781,"num_object_clones":0,"num_object_copies":20343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6781,"num_write_kb":433984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.d","version":"1545'6847","reported_seq":6878,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:50.923130+0000","last_change":"2024-08-17T18:09:28.696225+0000","last_active":"2024-08-17T18:10:50.923130+0000","last_peered":"2024-08-17T18:10:50.923130+0000","last_clean":"2024-08-17T18:10:50.923130+0000","last_became_active":"2024-08-17T18:09:28.695916+0000","last_became_peered":"2024-08-17T18:09:28.695916+0000","last_unstale":"2024-08-17T18:10:50.923130+0000","last_undegraded":"2024-08-17T18:10:50.923130+0000","last_fullsized":"2024-08-17T18:10:50.923130+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6847,"log_dups_size":0,"ondisk_log_size":6847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:00:20.159452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448724992,"num_objects":6847,"num_object_clones":0,"num_object_copies":20541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6847,"num_write_kb":438208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"56.c","version":"1545'6843","reported_seq":6874,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:50.435336+0000","last_change":"2024-08-17T18:09:28.627634+0000","last_active":"2024-08-17T18:10:50.435336+0000","last_peered":"2024-08-17T18:10:50.435336+0000","last_clean":"2024-08-17T18:10:50.435336+0000","last_became_active":"2024-08-17T18:09:28.627138+0000","last_became_peered":"2024-08-17T18:09:28.627138+0000","last_unstale":"2024-08-17T18:10:50.435336+0000","last_undegraded":"2024-08-17T18:10:50.435336+0000","last_fullsized":"2024-08-17T18:10:50.435336+0000","mapping_epoch":1535,"log_start":"1539'100","ondisk_log_start":"1539'100","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6743,"log_dups_size":0,"ondisk_log_size":6743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:47:56.574797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448462848,"num_objects":6843,"num_object_clones":0,"num_object_copies":20529,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6843,"num_write_kb":437952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.b","version":"1545'6871","reported_seq":6889,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:52.884906+0000","last_change":"2024-08-17T18:09:28.627309+0000","last_active":"2024-08-17T18:10:52.884906+0000","last_peered":"2024-08-17T18:10:52.884906+0000","last_clean":"2024-08-17T18:10:52.884906+0000","last_became_active":"2024-08-17T18:09:28.626970+0000","last_became_peered":"2024-08-17T18:09:28.626970+0000","last_unstale":"2024-08-17T18:10:52.884906+0000","last_undegraded":"2024-08-17T18:10:52.884906+0000","last_fullsized":"2024-08-17T18:10:52.884906+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6871,"log_dups_size":0,"ondisk_log_size":6871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:47:24.480441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450297856,"num_objects":6871,"num_object_clones":0,"num_object_copies":20613,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6871,"num_write_kb":439744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"56.a","version":"1545'6774","reported_seq":6805,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:50.475877+0000","last_change":"2024-08-17T18:09:28.697301+0000","last_active":"2024-08-17T18:10:50.475877+0000","last_peered":"2024-08-17T18:10:50.475877+0000","last_clean":"2024-08-17T18:10:50.475877+0000","last_became_active":"2024-08-17T18:09:28.697019+0000","last_became_peered":"2024-08-17T18:09:28.697019+0000","last_unstale":"2024-08-17T18:10:50.475877+0000","last_undegraded":"2024-08-17T18:10:50.475877+0000","last_fullsized":"2024-08-17T18:10:50.475877+0000","mapping_epoch":1535,"log_start":"1539'100","ondisk_log_start":"1539'100","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6674,"log_dups_size":0,"ondisk_log_size":6674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:44:37.273534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443940864,"num_objects":6774,"num_object_clones":0,"num_object_copies":20322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6774,"num_write_kb":433536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.9","version":"1545'6740","reported_seq":6759,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:54.102198+0000","last_change":"2024-08-17T18:09:28.626545+0000","last_active":"2024-08-17T18:10:54.102198+0000","last_peered":"2024-08-17T18:10:54.102198+0000","last_clean":"2024-08-17T18:10:54.102198+0000","last_became_active":"2024-08-17T18:09:28.626300+0000","last_became_peered":"2024-08-17T18:09:28.626300+0000","last_unstale":"2024-08-17T18:10:54.102198+0000","last_undegraded":"2024-08-17T18:10:54.102198+0000","last_fullsized":"2024-08-17T18:10:54.102198+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6740,"log_dups_size":0,"ondisk_log_size":6740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:34:50.839206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441712640,"num_objects":6740,"num_object_clones":0,"num_object_copies":20220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6740,"num_write_kb":431360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.8","version":"1545'6834","reported_seq":6865,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:50.923229+0000","last_change":"2024-08-17T18:09:28.697184+0000","last_active":"2024-08-17T18:10:50.923229+0000","last_peered":"2024-08-17T18:10:50.923229+0000","last_clean":"2024-08-17T18:10:50.923229+0000","last_became_active":"2024-08-17T18:09:28.696911+0000","last_became_peered":"2024-08-17T18:09:28.696911+0000","last_unstale":"2024-08-17T18:10:50.923229+0000","last_undegraded":"2024-08-17T18:10:50.923229+0000","last_fullsized":"2024-08-17T18:10:50.923229+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6834,"log_dups_size":0,"ondisk_log_size":6834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:39:59.778610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447873024,"num_objects":6834,"num_object_clones":0,"num_object_copies":20502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6834,"num_write_kb":437376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"56.7","version":"1545'6741","reported_seq":6772,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:50.910167+0000","last_change":"2024-08-17T18:09:28.628236+0000","last_active":"2024-08-17T18:10:50.910167+0000","last_peered":"2024-08-17T18:10:50.910167+0000","last_clean":"2024-08-17T18:10:50.910167+0000","last_became_active":"2024-08-17T18:09:28.627952+0000","last_became_peered":"2024-08-17T18:09:28.627952+0000","last_unstale":"2024-08-17T18:10:50.910167+0000","last_undegraded":"2024-08-17T18:10:50.910167+0000","last_fullsized":"2024-08-17T18:10:50.910167+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6741,"log_dups_size":0,"ondisk_log_size":6741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:04:20.822971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441778176,"num_objects":6741,"num_object_clones":0,"num_object_copies":20223,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6741,"num_write_kb":431424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"56.6","version":"1545'7110","reported_seq":7129,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:54.096440+0000","last_change":"2024-08-17T18:09:28.629945+0000","last_active":"2024-08-17T18:10:54.096440+0000","last_peered":"2024-08-17T18:10:54.096440+0000","last_clean":"2024-08-17T18:10:54.096440+0000","last_became_active":"2024-08-17T18:09:28.629553+0000","last_became_peered":"2024-08-17T18:09:28.629553+0000","last_unstale":"2024-08-17T18:10:54.096440+0000","last_undegraded":"2024-08-17T18:10:54.096440+0000","last_fullsized":"2024-08-17T18:10:54.096440+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":7110,"log_dups_size":0,"ondisk_log_size":7110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:00:36.875044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":465960960,"num_objects":7110,"num_object_clones":0,"num_object_copies":21330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7110,"num_write_kb":455040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.5","version":"1545'6819","reported_seq":6850,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:51.824142+0000","last_change":"2024-08-17T18:09:28.697212+0000","last_active":"2024-08-17T18:10:51.824142+0000","last_peered":"2024-08-17T18:10:51.824142+0000","last_clean":"2024-08-17T18:10:51.824142+0000","last_became_active":"2024-08-17T18:09:28.696769+0000","last_became_peered":"2024-08-17T18:09:28.696769+0000","last_unstale":"2024-08-17T18:10:51.824142+0000","last_undegraded":"2024-08-17T18:10:51.824142+0000","last_fullsized":"2024-08-17T18:10:51.824142+0000","mapping_epoch":1535,"log_start":"1539'200","ondisk_log_start":"1539'200","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6619,"log_dups_size":0,"ondisk_log_size":6619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:30:22.827483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446889984,"num_objects":6819,"num_object_clones":0,"num_object_copies":20457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6819,"num_write_kb":436416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.4","version":"1545'6728","reported_seq":6759,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:50.446291+0000","last_change":"2024-08-17T18:09:28.627340+0000","last_active":"2024-08-17T18:10:50.446291+0000","last_peered":"2024-08-17T18:10:50.446291+0000","last_clean":"2024-08-17T18:10:50.446291+0000","last_became_active":"2024-08-17T18:09:28.626966+0000","last_became_peered":"2024-08-17T18:09:28.626966+0000","last_unstale":"2024-08-17T18:10:50.446291+0000","last_undegraded":"2024-08-17T18:10:50.446291+0000","last_fullsized":"2024-08-17T18:10:50.446291+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6728,"log_dups_size":0,"ondisk_log_size":6728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:18:20.264361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440926208,"num_objects":6728,"num_object_clones":0,"num_object_copies":20184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6728,"num_write_kb":430592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.3","version":"1545'6734","reported_seq":6765,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:50.483014+0000","last_change":"2024-08-17T18:09:28.697359+0000","last_active":"2024-08-17T18:10:50.483014+0000","last_peered":"2024-08-17T18:10:50.483014+0000","last_clean":"2024-08-17T18:10:50.483014+0000","last_became_active":"2024-08-17T18:09:28.696943+0000","last_became_peered":"2024-08-17T18:09:28.696943+0000","last_unstale":"2024-08-17T18:10:50.483014+0000","last_undegraded":"2024-08-17T18:10:50.483014+0000","last_fullsized":"2024-08-17T18:10:50.483014+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6734,"log_dups_size":0,"ondisk_log_size":6734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:08:17.415390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441319424,"num_objects":6734,"num_object_clones":0,"num_object_copies":20202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6734,"num_write_kb":430976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.2","version":"1545'7092","reported_seq":7111,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:54.120203+0000","last_change":"2024-08-17T18:09:28.628711+0000","last_active":"2024-08-17T18:10:54.120203+0000","last_peered":"2024-08-17T18:10:54.120203+0000","last_clean":"2024-08-17T18:10:54.120203+0000","last_became_active":"2024-08-17T18:09:28.628387+0000","last_became_peered":"2024-08-17T18:09:28.628387+0000","last_unstale":"2024-08-17T18:10:54.120203+0000","last_undegraded":"2024-08-17T18:10:54.120203+0000","last_fullsized":"2024-08-17T18:10:54.120203+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":7092,"log_dups_size":0,"ondisk_log_size":7092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:26:06.412021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464781312,"num_objects":7092,"num_object_clones":0,"num_object_copies":21276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7092,"num_write_kb":453888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.1","version":"1545'7150","reported_seq":7169,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:54.046203+0000","last_change":"2024-08-17T18:09:28.629660+0000","last_active":"2024-08-17T18:10:54.046203+0000","last_peered":"2024-08-17T18:10:54.046203+0000","last_clean":"2024-08-17T18:10:54.046203+0000","last_became_active":"2024-08-17T18:09:28.629359+0000","last_became_peered":"2024-08-17T18:09:28.629359+0000","last_unstale":"2024-08-17T18:10:54.046203+0000","last_undegraded":"2024-08-17T18:10:54.046203+0000","last_fullsized":"2024-08-17T18:10:54.046203+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":7150,"log_dups_size":0,"ondisk_log_size":7150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:05:58.848611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":468582400,"num_objects":7150,"num_object_clones":0,"num_object_copies":21450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7150,"num_write_kb":457600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.0","version":"1545'6962","reported_seq":6980,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:54.126043+0000","last_change":"2024-08-17T18:09:28.697077+0000","last_active":"2024-08-17T18:10:54.126043+0000","last_peered":"2024-08-17T18:10:54.126043+0000","last_clean":"2024-08-17T18:10:54.126043+0000","last_became_active":"2024-08-17T18:09:28.696676+0000","last_became_peered":"2024-08-17T18:09:28.696676+0000","last_unstale":"2024-08-17T18:10:54.126043+0000","last_undegraded":"2024-08-17T18:10:54.126043+0000","last_fullsized":"2024-08-17T18:10:54.126043+0000","mapping_epoch":1535,"log_start":"0'0","ondisk_log_start":"0'0","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6962,"log_dups_size":0,"ondisk_log_size":6962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:09.350570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456261632,"num_objects":6962,"num_object_clones":0,"num_object_copies":20886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6962,"num_write_kb":445568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":234,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.417395+0000","last_change":"2024-08-17T18:07:48.546202+0000","last_active":"2024-08-17T18:10:47.417395+0000","last_peered":"2024-08-17T18:10:47.417395+0000","last_clean":"2024-08-17T18:10:47.417395+0000","last_became_active":"2024-08-17T18:07:48.531246+0000","last_became_peered":"2024-08-17T18:07:48.531246+0000","last_unstale":"2024-08-17T18:10:47.417395+0000","last_undegraded":"2024-08-17T18:10:47.417395+0000","last_fullsized":"2024-08-17T18:10:47.417395+0000","mapping_epoch":1501,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:30.944362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1549,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:37.875992+0000","last_change":"2024-08-17T18:08:37.068490+0000","last_active":"2024-08-17T18:10:37.875992+0000","last_peered":"2024-08-17T18:10:37.875992+0000","last_clean":"2024-08-17T18:10:37.875992+0000","last_became_active":"2024-08-17T18:08:37.068026+0000","last_became_peered":"2024-08-17T18:08:37.068026+0000","last_unstale":"2024-08-17T18:10:37.875992+0000","last_undegraded":"2024-08-17T18:10:37.875992+0000","last_fullsized":"2024-08-17T18:10:37.875992+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:34:27.467963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":2410,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.412263+0000","last_change":"2024-08-17T18:08:37.082356+0000","last_active":"2024-08-17T18:10:47.412263+0000","last_peered":"2024-08-17T18:10:47.412263+0000","last_clean":"2024-08-17T18:10:47.412263+0000","last_became_active":"2024-08-17T18:08:37.082046+0000","last_became_peered":"2024-08-17T18:08:37.082046+0000","last_unstale":"2024-08-17T18:10:47.412263+0000","last_undegraded":"2024-08-17T18:10:47.412263+0000","last_fullsized":"2024-08-17T18:10:47.412263+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:46:13.932917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2432,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.422338+0000","last_change":"2024-08-17T18:04:14.540774+0000","last_active":"2024-08-17T18:10:47.422338+0000","last_peered":"2024-08-17T18:10:47.422338+0000","last_clean":"2024-08-17T18:10:47.422338+0000","last_became_active":"2024-08-17T18:04:14.539778+0000","last_became_peered":"2024-08-17T18:04:14.539778+0000","last_unstale":"2024-08-17T18:10:47.422338+0000","last_undegraded":"2024-08-17T18:10:47.422338+0000","last_fullsized":"2024-08-17T18:10:47.422338+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:55:02.863360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":97,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.806972+0000","last_change":"2024-08-17T18:04:43.777705+0000","last_active":"2024-08-17T18:10:30.806972+0000","last_peered":"2024-08-17T18:10:30.806972+0000","last_clean":"2024-08-17T18:10:30.806972+0000","last_became_active":"2024-08-17T18:04:43.771283+0000","last_became_peered":"2024-08-17T18:04:43.771283+0000","last_unstale":"2024-08-17T18:10:30.806972+0000","last_undegraded":"2024-08-17T18:10:30.806972+0000","last_fullsized":"2024-08-17T18:10:30.806972+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:13.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1676,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.411398+0000","last_change":"2024-08-17T18:04:14.534963+0000","last_active":"2024-08-17T18:10:47.411398+0000","last_peered":"2024-08-17T18:10:47.411398+0000","last_clean":"2024-08-17T18:10:47.411398+0000","last_became_active":"2024-08-17T18:04:14.534634+0000","last_became_peered":"2024-08-17T18:04:14.534634+0000","last_unstale":"2024-08-17T18:10:47.411398+0000","last_undegraded":"2024-08-17T18:10:47.411398+0000","last_fullsized":"2024-08-17T18:10:47.411398+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:33:27.287860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":97,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.807060+0000","last_change":"2024-08-17T18:04:43.773534+0000","last_active":"2024-08-17T18:10:30.807060+0000","last_peered":"2024-08-17T18:10:30.807060+0000","last_clean":"2024-08-17T18:10:30.807060+0000","last_became_active":"2024-08-17T18:04:43.768507+0000","last_became_peered":"2024-08-17T18:04:43.768507+0000","last_unstale":"2024-08-17T18:10:30.807060+0000","last_undegraded":"2024-08-17T18:10:30.807060+0000","last_fullsized":"2024-08-17T18:10:30.807060+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:50:57.647351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1536,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.802325+0000","last_change":"2024-08-17T17:24:17.458168+0000","last_active":"2024-08-17T18:10:30.802325+0000","last_peered":"2024-08-17T18:10:30.802325+0000","last_clean":"2024-08-17T18:10:30.802325+0000","last_became_active":"2024-08-17T17:24:17.457847+0000","last_became_peered":"2024-08-17T17:24:17.457847+0000","last_unstale":"2024-08-17T18:10:30.802325+0000","last_undegraded":"2024-08-17T18:10:30.802325+0000","last_fullsized":"2024-08-17T18:10:30.802325+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:28.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":188,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.417333+0000","last_change":"2024-08-17T18:04:14.531445+0000","last_active":"2024-08-17T18:10:47.417333+0000","last_peered":"2024-08-17T18:10:47.417333+0000","last_clean":"2024-08-17T18:10:47.417333+0000","last_became_active":"2024-08-17T18:04:14.531099+0000","last_became_peered":"2024-08-17T18:04:14.531099+0000","last_unstale":"2024-08-17T18:10:47.417333+0000","last_undegraded":"2024-08-17T18:10:47.417333+0000","last_fullsized":"2024-08-17T18:10:47.417333+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:43.335123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":99,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.413130+0000","last_change":"2024-08-17T18:07:48.548262+0000","last_active":"2024-08-17T18:10:47.413130+0000","last_peered":"2024-08-17T18:10:47.413130+0000","last_clean":"2024-08-17T18:10:47.413130+0000","last_became_active":"2024-08-17T18:07:48.547846+0000","last_became_peered":"2024-08-17T18:07:48.547846+0000","last_unstale":"2024-08-17T18:10:47.413130+0000","last_undegraded":"2024-08-17T18:10:47.413130+0000","last_fullsized":"2024-08-17T18:10:47.413130+0000","mapping_epoch":1501,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:24:37.513219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1553,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.802913+0000","last_change":"2024-08-17T18:01:23.519890+0000","last_active":"2024-08-17T18:10:30.802913+0000","last_peered":"2024-08-17T18:10:30.802913+0000","last_clean":"2024-08-17T18:10:30.802913+0000","last_became_active":"2024-08-17T18:01:23.519526+0000","last_became_peered":"2024-08-17T18:01:23.519526+0000","last_unstale":"2024-08-17T18:10:30.802913+0000","last_undegraded":"2024-08-17T18:10:30.802913+0000","last_fullsized":"2024-08-17T18:10:30.802913+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:04:52.662528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2397,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.411972+0000","last_change":"2024-08-17T18:01:23.491836+0000","last_active":"2024-08-17T18:10:47.411972+0000","last_peered":"2024-08-17T18:10:47.411972+0000","last_clean":"2024-08-17T18:10:47.411972+0000","last_became_active":"2024-08-17T18:01:23.491209+0000","last_became_peered":"2024-08-17T18:01:23.491209+0000","last_unstale":"2024-08-17T18:10:47.411972+0000","last_undegraded":"2024-08-17T18:10:47.411972+0000","last_fullsized":"2024-08-17T18:10:47.411972+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:35:30.767005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2399,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.411873+0000","last_change":"2024-08-17T17:24:17.452025+0000","last_active":"2024-08-17T18:10:47.411873+0000","last_peered":"2024-08-17T18:10:47.411873+0000","last_clean":"2024-08-17T18:10:47.411873+0000","last_became_active":"2024-08-17T17:24:17.448029+0000","last_became_peered":"2024-08-17T17:24:17.448029+0000","last_unstale":"2024-08-17T18:10:47.411873+0000","last_undegraded":"2024-08-17T18:10:47.411873+0000","last_fullsized":"2024-08-17T18:10:47.411873+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:45.089346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":203,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.416765+0000","last_change":"2024-08-17T18:08:11.358013+0000","last_active":"2024-08-17T18:10:47.416765+0000","last_peered":"2024-08-17T18:10:47.416765+0000","last_clean":"2024-08-17T18:10:47.416765+0000","last_became_active":"2024-08-17T18:08:11.357666+0000","last_became_peered":"2024-08-17T18:08:11.357666+0000","last_unstale":"2024-08-17T18:10:47.416765+0000","last_undegraded":"2024-08-17T18:10:47.416765+0000","last_fullsized":"2024-08-17T18:10:47.416765+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:05:25.290465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2412,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.411870+0000","last_change":"2024-08-17T18:04:14.539660+0000","last_active":"2024-08-17T18:10:47.411870+0000","last_peered":"2024-08-17T18:10:47.411870+0000","last_clean":"2024-08-17T18:10:47.411870+0000","last_became_active":"2024-08-17T18:04:14.537852+0000","last_became_peered":"2024-08-17T18:04:14.537852+0000","last_unstale":"2024-08-17T18:10:47.411870+0000","last_undegraded":"2024-08-17T18:10:47.411870+0000","last_fullsized":"2024-08-17T18:10:47.411870+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:47:45.026997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1550,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.802239+0000","last_change":"2024-08-17T18:04:43.769892+0000","last_active":"2024-08-17T18:10:30.802239+0000","last_peered":"2024-08-17T18:10:30.802239+0000","last_clean":"2024-08-17T18:10:30.802239+0000","last_became_active":"2024-08-17T18:04:43.769702+0000","last_became_peered":"2024-08-17T18:04:43.769702+0000","last_unstale":"2024-08-17T18:10:30.802239+0000","last_undegraded":"2024-08-17T18:10:30.802239+0000","last_fullsized":"2024-08-17T18:10:30.802239+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:39.373904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2371,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.412032+0000","last_change":"2024-08-17T17:24:17.442338+0000","last_active":"2024-08-17T18:10:47.412032+0000","last_peered":"2024-08-17T18:10:47.412032+0000","last_clean":"2024-08-17T18:10:47.412032+0000","last_became_active":"2024-08-17T17:24:17.441948+0000","last_became_peered":"2024-08-17T17:24:17.441948+0000","last_unstale":"2024-08-17T18:10:47.412032+0000","last_undegraded":"2024-08-17T18:10:47.412032+0000","last_fullsized":"2024-08-17T18:10:47.412032+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:17.891492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":195,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.417489+0000","last_change":"2024-08-17T18:04:43.777736+0000","last_active":"2024-08-17T18:10:47.417489+0000","last_peered":"2024-08-17T18:10:47.417489+0000","last_clean":"2024-08-17T18:10:47.417489+0000","last_became_active":"2024-08-17T18:04:43.777438+0000","last_became_peered":"2024-08-17T18:04:43.777438+0000","last_unstale":"2024-08-17T18:10:47.417489+0000","last_undegraded":"2024-08-17T18:10:47.417489+0000","last_fullsized":"2024-08-17T18:10:47.417489+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:24:01.255927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2351,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.423122+0000","last_change":"2024-08-17T18:04:43.915398+0000","last_active":"2024-08-17T18:10:47.423122+0000","last_peered":"2024-08-17T18:10:47.423122+0000","last_clean":"2024-08-17T18:10:47.423122+0000","last_became_active":"2024-08-17T18:04:43.915212+0000","last_became_peered":"2024-08-17T18:04:43.915212+0000","last_unstale":"2024-08-17T18:10:47.423122+0000","last_undegraded":"2024-08-17T18:10:47.423122+0000","last_fullsized":"2024-08-17T18:10:47.423122+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:43:26.901830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":110,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.807353+0000","last_change":"2024-08-17T18:08:11.356876+0000","last_active":"2024-08-17T18:10:30.807353+0000","last_peered":"2024-08-17T18:10:30.807353+0000","last_clean":"2024-08-17T18:10:30.807353+0000","last_became_active":"2024-08-17T18:08:11.355662+0000","last_became_peered":"2024-08-17T18:08:11.355662+0000","last_unstale":"2024-08-17T18:10:30.807353+0000","last_undegraded":"2024-08-17T18:10:30.807353+0000","last_fullsized":"2024-08-17T18:10:30.807353+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:55:00.801251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":508,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.412055+0000","last_change":"2024-08-17T18:04:14.555729+0000","last_active":"2024-08-17T18:10:47.412055+0000","last_peered":"2024-08-17T18:10:47.412055+0000","last_clean":"2024-08-17T18:10:47.412055+0000","last_became_active":"2024-08-17T18:04:14.536678+0000","last_became_peered":"2024-08-17T18:04:14.536678+0000","last_unstale":"2024-08-17T18:10:47.412055+0000","last_undegraded":"2024-08-17T18:10:47.412055+0000","last_fullsized":"2024-08-17T18:10:47.412055+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:47:29.424778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2424,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.413058+0000","last_change":"2024-08-17T18:08:37.079571+0000","last_active":"2024-08-17T18:10:47.413058+0000","last_peered":"2024-08-17T18:10:47.413058+0000","last_clean":"2024-08-17T18:10:47.413058+0000","last_became_active":"2024-08-17T18:08:37.079224+0000","last_became_peered":"2024-08-17T18:08:37.079224+0000","last_unstale":"2024-08-17T18:10:47.413058+0000","last_undegraded":"2024-08-17T18:10:47.413058+0000","last_fullsized":"2024-08-17T18:10:47.413058+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:28:50.561072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1550,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.801373+0000","last_change":"2024-08-17T18:03:51.292044+0000","last_active":"2024-08-17T18:10:30.801373+0000","last_peered":"2024-08-17T18:10:30.801373+0000","last_clean":"2024-08-17T18:10:30.801373+0000","last_became_active":"2024-08-17T18:03:51.268271+0000","last_became_peered":"2024-08-17T18:03:51.268271+0000","last_unstale":"2024-08-17T18:10:30.801373+0000","last_undegraded":"2024-08-17T18:10:30.801373+0000","last_fullsized":"2024-08-17T18:10:30.801373+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:00.892410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4756,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1523,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.412018+0000","last_change":"2024-08-17T17:23:07.455919+0000","last_active":"2024-08-17T18:10:47.412018+0000","last_peered":"2024-08-17T18:10:47.412018+0000","last_clean":"2024-08-17T18:10:47.412018+0000","last_became_active":"2024-08-17T17:23:07.455284+0000","last_became_peered":"2024-08-17T17:23:07.455284+0000","last_unstale":"2024-08-17T18:10:47.412018+0000","last_undegraded":"2024-08-17T18:10:47.412018+0000","last_fullsized":"2024-08-17T18:10:47.412018+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:37:46.260775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"347'149","reported_seq":831,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.411530+0000","last_change":"2024-08-17T18:09:15.592511+0000","last_active":"2024-08-17T18:10:47.411530+0000","last_peered":"2024-08-17T18:10:47.411530+0000","last_clean":"2024-08-17T18:10:47.411530+0000","last_became_active":"2024-08-17T18:09:15.592099+0000","last_became_peered":"2024-08-17T18:09:15.592099+0000","last_unstale":"2024-08-17T18:10:47.411530+0000","last_undegraded":"2024-08-17T18:10:47.411530+0000","last_fullsized":"2024-08-17T18:10:47.411530+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:57.271286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:57.271286+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:54.841504+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1547,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.801576+0000","last_change":"2024-08-17T18:04:43.766724+0000","last_active":"2024-08-17T18:10:30.801576+0000","last_peered":"2024-08-17T18:10:30.801576+0000","last_clean":"2024-08-17T18:10:30.801576+0000","last_became_active":"2024-08-17T18:04:43.766428+0000","last_became_peered":"2024-08-17T18:04:43.766428+0000","last_unstale":"2024-08-17T18:10:30.801576+0000","last_undegraded":"2024-08-17T18:10:30.801576+0000","last_fullsized":"2024-08-17T18:10:30.801576+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:09:49.102940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1559,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.802310+0000","last_change":"2024-08-17T18:03:51.279963+0000","last_active":"2024-08-17T18:10:30.802310+0000","last_peered":"2024-08-17T18:10:30.802310+0000","last_clean":"2024-08-17T18:10:30.802310+0000","last_became_active":"2024-08-17T18:03:51.279506+0000","last_became_peered":"2024-08-17T18:03:51.279506+0000","last_unstale":"2024-08-17T18:10:30.802310+0000","last_undegraded":"2024-08-17T18:10:30.802310+0000","last_fullsized":"2024-08-17T18:10:30.802310+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:20:07.304042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":193,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.416826+0000","last_change":"2024-08-17T18:04:14.542589+0000","last_active":"2024-08-17T18:10:47.416826+0000","last_peered":"2024-08-17T18:10:47.416826+0000","last_clean":"2024-08-17T18:10:47.416826+0000","last_became_active":"2024-08-17T18:04:14.531540+0000","last_became_peered":"2024-08-17T18:04:14.531540+0000","last_unstale":"2024-08-17T18:10:47.416826+0000","last_undegraded":"2024-08-17T18:10:47.416826+0000","last_fullsized":"2024-08-17T18:10:47.416826+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:15:33.014782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"347'149","reported_seq":816,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.801531+0000","last_change":"2024-08-17T18:09:15.592395+0000","last_active":"2024-08-17T18:10:30.801531+0000","last_peered":"2024-08-17T18:10:30.801531+0000","last_clean":"2024-08-17T18:10:30.801531+0000","last_became_active":"2024-08-17T18:09:15.591773+0000","last_became_peered":"2024-08-17T18:09:15.591773+0000","last_unstale":"2024-08-17T18:10:30.801531+0000","last_undegraded":"2024-08-17T18:10:30.801531+0000","last_fullsized":"2024-08-17T18:10:30.801531+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:53.309517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:53.309517+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:50:57.110936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":85,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.411952+0000","last_change":"2024-08-17T18:08:11.349400+0000","last_active":"2024-08-17T18:10:47.411952+0000","last_peered":"2024-08-17T18:10:47.411952+0000","last_clean":"2024-08-17T18:10:47.411952+0000","last_became_active":"2024-08-17T18:08:11.348892+0000","last_became_peered":"2024-08-17T18:08:11.348892+0000","last_unstale":"2024-08-17T18:10:47.411952+0000","last_undegraded":"2024-08-17T18:10:47.411952+0000","last_fullsized":"2024-08-17T18:10:47.411952+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:52:54.819838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2382,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.412664+0000","last_change":"2024-08-17T17:23:07.428296+0000","last_active":"2024-08-17T18:10:47.412664+0000","last_peered":"2024-08-17T18:10:47.412664+0000","last_clean":"2024-08-17T18:10:47.412664+0000","last_became_active":"2024-08-17T17:23:07.428010+0000","last_became_peered":"2024-08-17T17:23:07.428010+0000","last_unstale":"2024-08-17T18:10:47.412664+0000","last_undegraded":"2024-08-17T18:10:47.412664+0000","last_fullsized":"2024-08-17T18:10:47.412664+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:45.459212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":2423,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.413549+0000","last_change":"2024-08-17T18:04:43.800633+0000","last_active":"2024-08-17T18:10:47.413549+0000","last_peered":"2024-08-17T18:10:47.413549+0000","last_clean":"2024-08-17T18:10:47.413549+0000","last_became_active":"2024-08-17T18:04:43.784612+0000","last_became_peered":"2024-08-17T18:04:43.784612+0000","last_unstale":"2024-08-17T18:10:47.413549+0000","last_undegraded":"2024-08-17T18:10:47.413549+0000","last_fullsized":"2024-08-17T18:10:47.413549+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:11.405039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"347'149","reported_seq":799,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.802177+0000","last_change":"2024-08-17T18:09:52.474670+0000","last_active":"2024-08-17T18:10:30.802177+0000","last_peered":"2024-08-17T18:10:30.802177+0000","last_clean":"2024-08-17T18:10:30.802177+0000","last_became_active":"2024-08-17T18:09:15.595011+0000","last_became_peered":"2024-08-17T18:09:15.595011+0000","last_unstale":"2024-08-17T18:10:30.802177+0000","last_undegraded":"2024-08-17T18:10:30.802177+0000","last_fullsized":"2024-08-17T18:10:30.802177+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:52.474544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:52.474544+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:38:21.347345+0000","scrub_duration":0.001886326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":196,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.417492+0000","last_change":"2024-08-17T18:04:43.805055+0000","last_active":"2024-08-17T18:10:47.417492+0000","last_peered":"2024-08-17T18:10:47.417492+0000","last_clean":"2024-08-17T18:10:47.417492+0000","last_became_active":"2024-08-17T18:04:43.804708+0000","last_became_peered":"2024-08-17T18:04:43.804708+0000","last_unstale":"2024-08-17T18:10:47.417492+0000","last_undegraded":"2024-08-17T18:10:47.417492+0000","last_fullsized":"2024-08-17T18:10:47.417492+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:47:22.253073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":189,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.421224+0000","last_change":"2024-08-17T18:04:14.529605+0000","last_active":"2024-08-17T18:10:47.421224+0000","last_peered":"2024-08-17T18:10:47.421224+0000","last_clean":"2024-08-17T18:10:47.421224+0000","last_became_active":"2024-08-17T18:04:14.529024+0000","last_became_peered":"2024-08-17T18:04:14.529024+0000","last_unstale":"2024-08-17T18:10:47.421224+0000","last_undegraded":"2024-08-17T18:10:47.421224+0000","last_fullsized":"2024-08-17T18:10:47.421224+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:38:00.485543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2435,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.420278+0000","last_change":"2024-08-17T18:04:43.773667+0000","last_active":"2024-08-17T18:10:47.420278+0000","last_peered":"2024-08-17T18:10:47.420278+0000","last_clean":"2024-08-17T18:10:47.420278+0000","last_became_active":"2024-08-17T18:04:43.773247+0000","last_became_peered":"2024-08-17T18:04:43.773247+0000","last_unstale":"2024-08-17T18:10:47.420278+0000","last_undegraded":"2024-08-17T18:10:47.420278+0000","last_fullsized":"2024-08-17T18:10:47.420278+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:35:48.964631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"347'149","reported_seq":799,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.802159+0000","last_change":"2024-08-17T18:09:49.461777+0000","last_active":"2024-08-17T18:10:30.802159+0000","last_peered":"2024-08-17T18:10:30.802159+0000","last_clean":"2024-08-17T18:10:30.802159+0000","last_became_active":"2024-08-17T18:09:15.592534+0000","last_became_peered":"2024-08-17T18:09:15.592534+0000","last_unstale":"2024-08-17T18:10:30.802159+0000","last_undegraded":"2024-08-17T18:10:30.802159+0000","last_fullsized":"2024-08-17T18:10:30.802159+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:49.461698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:49.461698+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:28:31.102314+0000","scrub_duration":0.0013792520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":260,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.411307+0000","last_change":"2024-08-17T18:04:43.771589+0000","last_active":"2024-08-17T18:10:47.411307+0000","last_peered":"2024-08-17T18:10:47.411307+0000","last_clean":"2024-08-17T18:10:47.411307+0000","last_became_active":"2024-08-17T18:04:43.769601+0000","last_became_peered":"2024-08-17T18:04:43.769601+0000","last_unstale":"2024-08-17T18:10:47.411307+0000","last_undegraded":"2024-08-17T18:10:47.411307+0000","last_fullsized":"2024-08-17T18:10:47.411307+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:42:49.228921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2451,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.420098+0000","last_change":"2024-08-17T18:04:14.537136+0000","last_active":"2024-08-17T18:10:47.420098+0000","last_peered":"2024-08-17T18:10:47.420098+0000","last_clean":"2024-08-17T18:10:47.420098+0000","last_became_active":"2024-08-17T18:04:14.536815+0000","last_became_peered":"2024-08-17T18:04:14.536815+0000","last_unstale":"2024-08-17T18:10:47.420098+0000","last_undegraded":"2024-08-17T18:10:47.420098+0000","last_fullsized":"2024-08-17T18:10:47.420098+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:02.706133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":93,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.807022+0000","last_change":"2024-08-17T18:04:43.773480+0000","last_active":"2024-08-17T18:10:30.807022+0000","last_peered":"2024-08-17T18:10:30.807022+0000","last_clean":"2024-08-17T18:10:30.807022+0000","last_became_active":"2024-08-17T18:04:43.773160+0000","last_became_peered":"2024-08-17T18:04:43.773160+0000","last_unstale":"2024-08-17T18:10:30.807022+0000","last_undegraded":"2024-08-17T18:10:30.807022+0000","last_fullsized":"2024-08-17T18:10:30.807022+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:07:55.468132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"347'149","reported_seq":817,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.806803+0000","last_change":"2024-08-17T18:09:15.596647+0000","last_active":"2024-08-17T18:10:30.806803+0000","last_peered":"2024-08-17T18:10:30.806803+0000","last_clean":"2024-08-17T18:10:30.806803+0000","last_became_active":"2024-08-17T18:09:15.596339+0000","last_became_peered":"2024-08-17T18:09:15.596339+0000","last_unstale":"2024-08-17T18:10:30.806803+0000","last_undegraded":"2024-08-17T18:10:30.806803+0000","last_fullsized":"2024-08-17T18:10:30.806803+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:00.316884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:00.316884+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:30.779927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":262,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.419973+0000","last_change":"2024-08-17T18:02:08.224928+0000","last_active":"2024-08-17T18:10:47.419973+0000","last_peered":"2024-08-17T18:10:47.419973+0000","last_clean":"2024-08-17T18:10:47.419973+0000","last_became_active":"2024-08-17T18:02:08.224604+0000","last_became_peered":"2024-08-17T18:02:08.224604+0000","last_unstale":"2024-08-17T18:10:47.419973+0000","last_undegraded":"2024-08-17T18:10:47.419973+0000","last_fullsized":"2024-08-17T18:10:47.419973+0000","mapping_epoch":1421,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1422,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:55.716971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1545,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:37.876051+0000","last_change":"2024-08-17T18:04:14.526890+0000","last_active":"2024-08-17T18:10:37.876051+0000","last_peered":"2024-08-17T18:10:37.876051+0000","last_clean":"2024-08-17T18:10:37.876051+0000","last_became_active":"2024-08-17T18:04:14.526564+0000","last_became_peered":"2024-08-17T18:04:14.526564+0000","last_unstale":"2024-08-17T18:10:37.876051+0000","last_undegraded":"2024-08-17T18:10:37.876051+0000","last_fullsized":"2024-08-17T18:10:37.876051+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:10:13.229827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":97,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.806892+0000","last_change":"2024-08-17T18:04:43.791749+0000","last_active":"2024-08-17T18:10:30.806892+0000","last_peered":"2024-08-17T18:10:30.806892+0000","last_clean":"2024-08-17T18:10:30.806892+0000","last_became_active":"2024-08-17T18:04:43.774474+0000","last_became_peered":"2024-08-17T18:04:43.774474+0000","last_unstale":"2024-08-17T18:10:30.806892+0000","last_undegraded":"2024-08-17T18:10:30.806892+0000","last_fullsized":"2024-08-17T18:10:30.806892+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:19:08.097560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"347'149","reported_seq":836,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.416995+0000","last_change":"2024-08-17T18:09:15.596502+0000","last_active":"2024-08-17T18:10:47.416995+0000","last_peered":"2024-08-17T18:10:47.416995+0000","last_clean":"2024-08-17T18:10:47.416995+0000","last_became_active":"2024-08-17T18:09:15.595460+0000","last_became_peered":"2024-08-17T18:09:15.595460+0000","last_unstale":"2024-08-17T18:10:47.416995+0000","last_undegraded":"2024-08-17T18:10:47.416995+0000","last_fullsized":"2024-08-17T18:10:47.416995+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:02.363531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:02.363531+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:22:42.587876+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2438,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.412605+0000","last_change":"2024-08-17T18:04:43.781133+0000","last_active":"2024-08-17T18:10:47.412605+0000","last_peered":"2024-08-17T18:10:47.412605+0000","last_clean":"2024-08-17T18:10:47.412605+0000","last_became_active":"2024-08-17T18:04:43.780787+0000","last_became_peered":"2024-08-17T18:04:43.780787+0000","last_unstale":"2024-08-17T18:10:47.412605+0000","last_undegraded":"2024-08-17T18:10:47.412605+0000","last_fullsized":"2024-08-17T18:10:47.412605+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:38:34.901162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2394,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.411480+0000","last_change":"2024-08-17T18:01:23.493896+0000","last_active":"2024-08-17T18:10:47.411480+0000","last_peered":"2024-08-17T18:10:47.411480+0000","last_clean":"2024-08-17T18:10:47.411480+0000","last_became_active":"2024-08-17T18:01:23.493551+0000","last_became_peered":"2024-08-17T18:01:23.493551+0000","last_unstale":"2024-08-17T18:10:47.411480+0000","last_undegraded":"2024-08-17T18:10:47.411480+0000","last_fullsized":"2024-08-17T18:10:47.411480+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:18:47.648483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2372,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.411529+0000","last_change":"2024-08-17T18:04:14.543532+0000","last_active":"2024-08-17T18:10:47.411529+0000","last_peered":"2024-08-17T18:10:47.411529+0000","last_clean":"2024-08-17T18:10:47.411529+0000","last_became_active":"2024-08-17T18:04:14.543131+0000","last_became_peered":"2024-08-17T18:04:14.543131+0000","last_unstale":"2024-08-17T18:10:47.411529+0000","last_undegraded":"2024-08-17T18:10:47.411529+0000","last_fullsized":"2024-08-17T18:10:47.411529+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:51:44.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"347'149","reported_seq":686,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.412135+0000","last_change":"2024-08-17T18:09:15.740862+0000","last_active":"2024-08-17T18:10:47.412135+0000","last_peered":"2024-08-17T18:10:47.412135+0000","last_clean":"2024-08-17T18:10:47.412135+0000","last_became_active":"2024-08-17T18:09:15.591452+0000","last_became_peered":"2024-08-17T18:09:15.591452+0000","last_unstale":"2024-08-17T18:10:47.412135+0000","last_undegraded":"2024-08-17T18:10:47.412135+0000","last_fullsized":"2024-08-17T18:10:47.412135+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:15.740775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:15.740775+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:29:12.741753+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":210,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.416726+0000","last_change":"2024-08-17T18:10:11.727045+0000","last_active":"2024-08-17T18:10:47.416726+0000","last_peered":"2024-08-17T18:10:47.416726+0000","last_clean":"2024-08-17T18:10:47.416726+0000","last_became_active":"2024-08-17T18:10:11.726725+0000","last_became_peered":"2024-08-17T18:10:11.726725+0000","last_unstale":"2024-08-17T18:10:47.416726+0000","last_undegraded":"2024-08-17T18:10:47.416726+0000","last_fullsized":"2024-08-17T18:10:47.416726+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:12:13.835640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":203,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.417485+0000","last_change":"2024-08-17T18:08:11.440676+0000","last_active":"2024-08-17T18:10:47.417485+0000","last_peered":"2024-08-17T18:10:47.417485+0000","last_clean":"2024-08-17T18:10:47.417485+0000","last_became_active":"2024-08-17T18:08:11.440353+0000","last_became_peered":"2024-08-17T18:08:11.440353+0000","last_unstale":"2024-08-17T18:10:47.417485+0000","last_undegraded":"2024-08-17T18:10:47.417485+0000","last_fullsized":"2024-08-17T18:10:47.417485+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:19:22.125277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1138,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.411621+0000","last_change":"2024-08-17T18:04:14.537308+0000","last_active":"2024-08-17T18:10:47.411621+0000","last_peered":"2024-08-17T18:10:47.411621+0000","last_clean":"2024-08-17T18:10:47.411621+0000","last_became_active":"2024-08-17T18:04:14.535521+0000","last_became_peered":"2024-08-17T18:04:14.535521+0000","last_unstale":"2024-08-17T18:10:47.411621+0000","last_undegraded":"2024-08-17T18:10:47.411621+0000","last_fullsized":"2024-08-17T18:10:47.411621+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:40:00.384936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":41,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:42.257834+0000","last_change":"2024-08-17T18:08:37.071912+0000","last_active":"2024-08-17T18:10:42.257834+0000","last_peered":"2024-08-17T18:10:42.257834+0000","last_clean":"2024-08-17T18:10:42.257834+0000","last_became_active":"2024-08-17T18:08:37.071648+0000","last_became_peered":"2024-08-17T18:08:37.071648+0000","last_unstale":"2024-08-17T18:10:42.257834+0000","last_undegraded":"2024-08-17T18:10:42.257834+0000","last_fullsized":"2024-08-17T18:10:42.257834+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:58:57.279808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":200,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.411173+0000","last_change":"2024-08-17T18:04:43.771309+0000","last_active":"2024-08-17T18:10:47.411173+0000","last_peered":"2024-08-17T18:10:47.411173+0000","last_clean":"2024-08-17T18:10:47.411173+0000","last_became_active":"2024-08-17T18:04:43.770922+0000","last_became_peered":"2024-08-17T18:04:43.770922+0000","last_unstale":"2024-08-17T18:10:47.411173+0000","last_undegraded":"2024-08-17T18:10:47.411173+0000","last_fullsized":"2024-08-17T18:10:47.411173+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:10.173251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":97,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.807130+0000","last_change":"2024-08-17T18:04:43.786228+0000","last_active":"2024-08-17T18:10:30.807130+0000","last_peered":"2024-08-17T18:10:30.807130+0000","last_clean":"2024-08-17T18:10:30.807130+0000","last_became_active":"2024-08-17T18:04:43.770976+0000","last_became_peered":"2024-08-17T18:04:43.770976+0000","last_unstale":"2024-08-17T18:10:30.807130+0000","last_undegraded":"2024-08-17T18:10:30.807130+0000","last_fullsized":"2024-08-17T18:10:30.807130+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:20:10.131137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":828,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.420595+0000","last_change":"2024-08-17T18:07:54.303783+0000","last_active":"2024-08-17T18:10:47.420595+0000","last_peered":"2024-08-17T18:10:47.420595+0000","last_clean":"2024-08-17T18:10:47.420595+0000","last_became_active":"2024-08-17T18:06:57.358131+0000","last_became_peered":"2024-08-17T18:06:57.358131+0000","last_unstale":"2024-08-17T18:10:47.420595+0000","last_undegraded":"2024-08-17T18:10:47.420595+0000","last_fullsized":"2024-08-17T18:10:47.420595+0000","mapping_epoch":1460,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:54.303708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:54.303708+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:43:08.118481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"347'149","reported_seq":818,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.806739+0000","last_change":"2024-08-17T18:09:15.592722+0000","last_active":"2024-08-17T18:10:30.806739+0000","last_peered":"2024-08-17T18:10:30.806739+0000","last_clean":"2024-08-17T18:10:30.806739+0000","last_became_active":"2024-08-17T18:09:15.592242+0000","last_became_peered":"2024-08-17T18:09:15.592242+0000","last_unstale":"2024-08-17T18:10:30.806739+0000","last_undegraded":"2024-08-17T18:10:30.806739+0000","last_fullsized":"2024-08-17T18:10:30.806739+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:55.255983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:55.255983+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:02:53.596275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2381,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.412484+0000","last_change":"2024-08-17T17:24:17.455710+0000","last_active":"2024-08-17T18:10:47.412484+0000","last_peered":"2024-08-17T18:10:47.412484+0000","last_clean":"2024-08-17T18:10:47.412484+0000","last_became_active":"2024-08-17T17:24:17.452989+0000","last_became_peered":"2024-08-17T17:24:17.452989+0000","last_unstale":"2024-08-17T18:10:47.412484+0000","last_undegraded":"2024-08-17T18:10:47.412484+0000","last_fullsized":"2024-08-17T18:10:47.412484+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:58:07.894968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":85,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.411729+0000","last_change":"2024-08-17T18:08:11.347750+0000","last_active":"2024-08-17T18:10:47.411729+0000","last_peered":"2024-08-17T18:10:47.411729+0000","last_clean":"2024-08-17T18:10:47.411729+0000","last_became_active":"2024-08-17T18:08:11.346796+0000","last_became_peered":"2024-08-17T18:08:11.346796+0000","last_unstale":"2024-08-17T18:10:47.411729+0000","last_undegraded":"2024-08-17T18:10:47.411729+0000","last_fullsized":"2024-08-17T18:10:47.411729+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:24.470900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1541,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.801474+0000","last_change":"2024-08-17T18:08:37.068370+0000","last_active":"2024-08-17T18:10:30.801474+0000","last_peered":"2024-08-17T18:10:30.801474+0000","last_clean":"2024-08-17T18:10:30.801474+0000","last_became_active":"2024-08-17T18:08:37.068027+0000","last_became_peered":"2024-08-17T18:08:37.068027+0000","last_unstale":"2024-08-17T18:10:30.801474+0000","last_undegraded":"2024-08-17T18:10:30.801474+0000","last_fullsized":"2024-08-17T18:10:30.801474+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:02:12.661660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1550,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.411515+0000","last_change":"2024-08-17T18:03:51.270293+0000","last_active":"2024-08-17T18:10:47.411515+0000","last_peered":"2024-08-17T18:10:47.411515+0000","last_clean":"2024-08-17T18:10:47.411515+0000","last_became_active":"2024-08-17T18:03:51.269970+0000","last_became_peered":"2024-08-17T18:03:51.269970+0000","last_unstale":"2024-08-17T18:10:47.411515+0000","last_undegraded":"2024-08-17T18:10:47.411515+0000","last_fullsized":"2024-08-17T18:10:47.411515+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:02:30.016517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"347'149","reported_seq":818,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.806676+0000","last_change":"2024-08-17T18:09:15.590888+0000","last_active":"2024-08-17T18:10:30.806676+0000","last_peered":"2024-08-17T18:10:30.806676+0000","last_clean":"2024-08-17T18:10:30.806676+0000","last_became_active":"2024-08-17T18:09:15.590345+0000","last_became_peered":"2024-08-17T18:09:15.590345+0000","last_unstale":"2024-08-17T18:10:30.806676+0000","last_undegraded":"2024-08-17T18:10:30.806676+0000","last_fullsized":"2024-08-17T18:10:30.806676+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:59.307356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:59.307356+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:40:40.796012+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1553,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.801917+0000","last_change":"2024-08-17T18:01:23.497110+0000","last_active":"2024-08-17T18:10:30.801917+0000","last_peered":"2024-08-17T18:10:30.801917+0000","last_clean":"2024-08-17T18:10:30.801917+0000","last_became_active":"2024-08-17T18:01:23.496800+0000","last_became_peered":"2024-08-17T18:01:23.496800+0000","last_unstale":"2024-08-17T18:10:30.801917+0000","last_undegraded":"2024-08-17T18:10:30.801917+0000","last_fullsized":"2024-08-17T18:10:30.801917+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:32:29.892739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1522,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.802869+0000","last_change":"2024-08-17T17:23:07.401624+0000","last_active":"2024-08-17T18:10:30.802869+0000","last_peered":"2024-08-17T18:10:30.802869+0000","last_clean":"2024-08-17T18:10:30.802869+0000","last_became_active":"2024-08-17T17:23:07.400222+0000","last_became_peered":"2024-08-17T17:23:07.400222+0000","last_unstale":"2024-08-17T18:10:30.802869+0000","last_undegraded":"2024-08-17T18:10:30.802869+0000","last_fullsized":"2024-08-17T18:10:30.802869+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:07.069989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"347'149","reported_seq":830,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.422947+0000","last_change":"2024-08-17T18:09:15.592665+0000","last_active":"2024-08-17T18:10:47.422947+0000","last_peered":"2024-08-17T18:10:47.422947+0000","last_clean":"2024-08-17T18:10:47.422947+0000","last_became_active":"2024-08-17T18:09:15.590435+0000","last_became_peered":"2024-08-17T18:09:15.590435+0000","last_unstale":"2024-08-17T18:10:47.422947+0000","last_undegraded":"2024-08-17T18:10:47.422947+0000","last_fullsized":"2024-08-17T18:10:47.422947+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:52.271487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:52.271487+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:55:12.849191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1523,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.801322+0000","last_change":"2024-08-17T17:24:17.470809+0000","last_active":"2024-08-17T18:10:30.801322+0000","last_peered":"2024-08-17T18:10:30.801322+0000","last_clean":"2024-08-17T18:10:30.801322+0000","last_became_active":"2024-08-17T17:24:17.469753+0000","last_became_peered":"2024-08-17T17:24:17.469753+0000","last_unstale":"2024-08-17T18:10:30.801322+0000","last_undegraded":"2024-08-17T18:10:30.801322+0000","last_fullsized":"2024-08-17T18:10:30.801322+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:00.049425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":2389,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.411394+0000","last_change":"2024-08-17T18:04:14.552806+0000","last_active":"2024-08-17T18:10:47.411394+0000","last_peered":"2024-08-17T18:10:47.411394+0000","last_clean":"2024-08-17T18:10:47.411394+0000","last_became_active":"2024-08-17T18:04:14.535546+0000","last_became_peered":"2024-08-17T18:04:14.535546+0000","last_unstale":"2024-08-17T18:10:47.411394+0000","last_undegraded":"2024-08-17T18:10:47.411394+0000","last_fullsized":"2024-08-17T18:10:47.411394+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:44:45.702272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":188,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.417417+0000","last_change":"2024-08-17T18:04:14.539500+0000","last_active":"2024-08-17T18:10:47.417417+0000","last_peered":"2024-08-17T18:10:47.417417+0000","last_clean":"2024-08-17T18:10:47.417417+0000","last_became_active":"2024-08-17T18:04:14.538738+0000","last_became_peered":"2024-08-17T18:04:14.538738+0000","last_unstale":"2024-08-17T18:10:47.417417+0000","last_undegraded":"2024-08-17T18:10:47.417417+0000","last_fullsized":"2024-08-17T18:10:47.417417+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:31.439947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":93,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.807165+0000","last_change":"2024-08-17T18:04:43.771107+0000","last_active":"2024-08-17T18:10:30.807165+0000","last_peered":"2024-08-17T18:10:30.807165+0000","last_clean":"2024-08-17T18:10:30.807165+0000","last_became_active":"2024-08-17T18:04:43.770799+0000","last_became_peered":"2024-08-17T18:04:43.770799+0000","last_unstale":"2024-08-17T18:10:30.807165+0000","last_undegraded":"2024-08-17T18:10:30.807165+0000","last_fullsized":"2024-08-17T18:10:30.807165+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:09:04.350798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":41,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.258251+0000","last_change":"2024-08-17T18:08:37.080554+0000","last_active":"2024-08-17T18:10:47.258251+0000","last_peered":"2024-08-17T18:10:47.258251+0000","last_clean":"2024-08-17T18:10:47.258251+0000","last_became_active":"2024-08-17T18:08:37.079920+0000","last_became_peered":"2024-08-17T18:08:37.079920+0000","last_unstale":"2024-08-17T18:10:47.258251+0000","last_undegraded":"2024-08-17T18:10:47.258251+0000","last_fullsized":"2024-08-17T18:10:47.258251+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:08:25.214786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":97,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.807190+0000","last_change":"2024-08-17T18:04:43.781111+0000","last_active":"2024-08-17T18:10:30.807190+0000","last_peered":"2024-08-17T18:10:30.807190+0000","last_clean":"2024-08-17T18:10:30.807190+0000","last_became_active":"2024-08-17T18:04:43.771839+0000","last_became_peered":"2024-08-17T18:04:43.771839+0000","last_unstale":"2024-08-17T18:10:30.807190+0000","last_undegraded":"2024-08-17T18:10:30.807190+0000","last_fullsized":"2024-08-17T18:10:30.807190+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:52:45.973049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1558,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.411348+0000","last_change":"2024-08-17T18:08:11.449492+0000","last_active":"2024-08-17T18:10:47.411348+0000","last_peered":"2024-08-17T18:10:47.411348+0000","last_clean":"2024-08-17T18:10:47.411348+0000","last_became_active":"2024-08-17T18:08:11.424764+0000","last_became_peered":"2024-08-17T18:08:11.424764+0000","last_unstale":"2024-08-17T18:10:47.411348+0000","last_undegraded":"2024-08-17T18:10:47.411348+0000","last_fullsized":"2024-08-17T18:10:47.411348+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:53:37.552880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":207,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.411141+0000","last_change":"2024-08-17T18:04:14.551061+0000","last_active":"2024-08-17T18:10:47.411141+0000","last_peered":"2024-08-17T18:10:47.411141+0000","last_clean":"2024-08-17T18:10:47.411141+0000","last_became_active":"2024-08-17T18:04:14.543296+0000","last_became_peered":"2024-08-17T18:04:14.543296+0000","last_unstale":"2024-08-17T18:10:47.411141+0000","last_undegraded":"2024-08-17T18:10:47.411141+0000","last_fullsized":"2024-08-17T18:10:47.411141+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:39:45.294265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":111,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.806820+0000","last_change":"2024-08-17T18:08:37.068503+0000","last_active":"2024-08-17T18:10:30.806820+0000","last_peered":"2024-08-17T18:10:30.806820+0000","last_clean":"2024-08-17T18:10:30.806820+0000","last_became_active":"2024-08-17T18:08:37.068110+0000","last_became_peered":"2024-08-17T18:08:37.068110+0000","last_unstale":"2024-08-17T18:10:30.806820+0000","last_undegraded":"2024-08-17T18:10:30.806820+0000","last_fullsized":"2024-08-17T18:10:30.806820+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:12:30.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":139,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:37.876123+0000","last_change":"2024-08-17T18:04:14.547798+0000","last_active":"2024-08-17T18:10:37.876123+0000","last_peered":"2024-08-17T18:10:37.876123+0000","last_clean":"2024-08-17T18:10:37.876123+0000","last_became_active":"2024-08-17T18:04:14.527496+0000","last_became_peered":"2024-08-17T18:04:14.527496+0000","last_unstale":"2024-08-17T18:10:37.876123+0000","last_undegraded":"2024-08-17T18:10:37.876123+0000","last_fullsized":"2024-08-17T18:10:37.876123+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:25:54.338846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1605,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:42.257795+0000","last_change":"2024-08-17T18:04:43.813536+0000","last_active":"2024-08-17T18:10:42.257795+0000","last_peered":"2024-08-17T18:10:42.257795+0000","last_clean":"2024-08-17T18:10:42.257795+0000","last_became_active":"2024-08-17T18:04:43.799823+0000","last_became_peered":"2024-08-17T18:04:43.799823+0000","last_unstale":"2024-08-17T18:10:42.257795+0000","last_undegraded":"2024-08-17T18:10:42.257795+0000","last_fullsized":"2024-08-17T18:10:42.257795+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:04:10.837113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":92,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.413311+0000","last_change":"2024-08-17T18:10:11.737395+0000","last_active":"2024-08-17T18:10:47.413311+0000","last_peered":"2024-08-17T18:10:47.413311+0000","last_clean":"2024-08-17T18:10:47.413311+0000","last_became_active":"2024-08-17T18:10:11.736935+0000","last_became_peered":"2024-08-17T18:10:11.736935+0000","last_unstale":"2024-08-17T18:10:47.413311+0000","last_undegraded":"2024-08-17T18:10:47.413311+0000","last_fullsized":"2024-08-17T18:10:47.413311+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:32:49.395319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":263,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.801528+0000","last_change":"2024-08-17T18:04:14.527380+0000","last_active":"2024-08-17T18:10:30.801528+0000","last_peered":"2024-08-17T18:10:30.801528+0000","last_clean":"2024-08-17T18:10:30.801528+0000","last_became_active":"2024-08-17T18:04:14.527076+0000","last_became_peered":"2024-08-17T18:04:14.527076+0000","last_unstale":"2024-08-17T18:10:30.801528+0000","last_undegraded":"2024-08-17T18:10:30.801528+0000","last_fullsized":"2024-08-17T18:10:30.801528+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:36:35.583709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":188,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.416858+0000","last_change":"2024-08-17T18:04:14.541814+0000","last_active":"2024-08-17T18:10:47.416858+0000","last_peered":"2024-08-17T18:10:47.416858+0000","last_clean":"2024-08-17T18:10:47.416858+0000","last_became_active":"2024-08-17T18:04:14.541403+0000","last_became_peered":"2024-08-17T18:04:14.541403+0000","last_unstale":"2024-08-17T18:10:47.416858+0000","last_undegraded":"2024-08-17T18:10:47.416858+0000","last_fullsized":"2024-08-17T18:10:47.416858+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:07:45.294924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":202,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.417127+0000","last_change":"2024-08-17T18:08:11.351307+0000","last_active":"2024-08-17T18:10:47.417127+0000","last_peered":"2024-08-17T18:10:47.417127+0000","last_clean":"2024-08-17T18:10:47.417127+0000","last_became_active":"2024-08-17T18:08:11.350926+0000","last_became_peered":"2024-08-17T18:08:11.350926+0000","last_unstale":"2024-08-17T18:10:47.417127+0000","last_undegraded":"2024-08-17T18:10:47.417127+0000","last_fullsized":"2024-08-17T18:10:47.417127+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:59:57.477492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":93,"reported_epoch":1544,"state":"active+clean","last_fresh":"2024-08-17T18:10:30.807155+0000","last_change":"2024-08-17T18:04:43.771720+0000","last_active":"2024-08-17T18:10:30.807155+0000","last_peered":"2024-08-17T18:10:30.807155+0000","last_clean":"2024-08-17T18:10:30.807155+0000","last_became_active":"2024-08-17T18:04:43.771137+0000","last_became_peered":"2024-08-17T18:04:43.771137+0000","last_unstale":"2024-08-17T18:10:30.807155+0000","last_undegraded":"2024-08-17T18:10:30.807155+0000","last_fullsized":"2024-08-17T18:10:30.807155+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:57.647909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":128,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.411705+0000","last_change":"2024-08-17T18:03:51.283872+0000","last_active":"2024-08-17T18:10:47.411705+0000","last_peered":"2024-08-17T18:10:47.411705+0000","last_clean":"2024-08-17T18:10:47.411705+0000","last_became_active":"2024-08-17T18:03:51.267197+0000","last_became_peered":"2024-08-17T18:03:51.267197+0000","last_unstale":"2024-08-17T18:10:47.411705+0000","last_undegraded":"2024-08-17T18:10:47.411705+0000","last_fullsized":"2024-08-17T18:10:47.411705+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:18:59.157786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2378,"reported_epoch":1545,"state":"active+clean","last_fresh":"2024-08-17T18:10:47.411753+0000","last_change":"2024-08-17T18:03:51.277491+0000","last_active":"2024-08-17T18:10:47.411753+0000","last_peered":"2024-08-17T18:10:47.411753+0000","last_clean":"2024-08-17T18:10:47.411753+0000","last_became_active":"2024-08-17T18:03:51.277178+0000","last_became_peered":"2024-08-17T18:03:51.277178+0000","last_unstale":"2024-08-17T18:10:47.411753+0000","last_undegraded":"2024-08-17T18:10:47.411753+0000","last_fullsized":"2024-08-17T18:10:47.411753+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:21:35.319085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":56,"num_pg":16,"stat_sum":{"num_bytes":7203061760,"num_objects":109910,"num_object_clones":0,"num_object_copies":329730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109910,"num_write_kb":7034240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21642674176,"data_stored":21642674176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109210,"ondisk_log_size":109210,"up":48,"acting":48,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":7994,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234738,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496376,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4379100,"kb_used_data":3509652,"kb_used_omap":1,"kb_used_meta":869438,"kb_avail":89366052,"statfs":{"total":95995035648,"available":91510837248,"internally_reserved":0,"allocated":3593883648,"data_stored":3588350971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":890304967},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565617,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3539536,"kb_used_data":2650824,"kb_used_omap":72,"kb_used_meta":888631,"kb_avail":90205616,"statfs":{"total":95995035648,"available":92370550784,"internally_reserved":0,"allocated":2714443776,"data_stored":2708919224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":909958638},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":1399,"seq":6008659247219,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3409052,"kb_used_data":3056212,"kb_used_omap":131,"kb_used_meta":352700,"kb_avail":90336100,"statfs":{"total":95995035648,"available":92504166400,"internally_reserved":0,"allocated":3129561088,"data_stored":3124029460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134291,"internal_metadata":361165677},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743187007,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3304972,"kb_used_data":2609348,"kb_used_omap":66,"kb_used_meta":695549,"kb_avail":90440180,"statfs":{"total":95995035648,"available":92610744320,"internally_reserved":0,"allocated":2671972352,"data_stored":2666451233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":712242859},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448646,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3087248,"kb_used_data":2608328,"kb_used_omap":75,"kb_used_meta":478836,"kb_avail":90657904,"statfs":{"total":95995035648,"available":92833693696,"internally_reserved":0,"allocated":2670927872,"data_stored":2665403310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":490328255},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208758011,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4468500,"kb_used_data":3571084,"kb_used_omap":1,"kb_used_meta":897406,"kb_avail":89276652,"statfs":{"total":95995035648,"available":91419291648,"internally_reserved":0,"allocated":3656790016,"data_stored":3651261867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":918944198},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019649,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3838684,"kb_used_data":3107860,"kb_used_omap":1,"kb_used_meta":730814,"kb_avail":89906468,"statfs":{"total":95995035648,"available":92064223232,"internally_reserved":0,"allocated":3182448640,"data_stored":3176915986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":748353990},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2664890368,"data_stored":2664890368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2670329856,"data_stored":2670329856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3138125824,"data_stored":3138125824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2717974528,"data_stored":2717974528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3173842944,"data_stored":3173842944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3668312064,"data_stored":3668312064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3609198592,"data_stored":3609198592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T18:10:56.726 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-08-17T18:10:56.727 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:56.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:56 smithi165 ceph-mon[105517]: from='client.46076 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:10:56.777 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:56.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:56.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:56.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:56.931 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:56.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:56.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:57.014 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:10:57.033 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:57.057 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:57.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:57.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:57.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:57.185 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:57.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:57.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:57.287 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:10:57.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:57.318 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:57.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:10:57.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:57.452 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 78 16 115059 115043 92.1663 54.375 0.0156657 0.0108451 2024-08-17T18:10:57.454 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:57.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:57.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:57.483 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:10:57.507 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:57 smithi136 ceph-mon[109861]: pgmap v2849: 99 pgs: 99 active+clean; 6.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 60 MiB/s wr, 962 op/s 2024-08-17T18:10:57.508 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:57 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1860925428' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:10:57.508 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:57 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:10:57.508 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:57 smithi136 ceph-mon[115078]: pgmap v2849: 99 pgs: 99 active+clean; 6.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 60 MiB/s wr, 962 op/s 2024-08-17T18:10:57.508 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:57 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1860925428' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:10:57.508 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:57 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:10:57.508 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:10:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:10:57] "GET /metrics HTTP/1.1" 200 36607 "" "Prometheus/2.51.0" 2024-08-17T18:10:57.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:57.585 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:57.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:57.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:57.687 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:57.714 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:57.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:57.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:57 smithi165 ceph-mon[105517]: pgmap v2849: 99 pgs: 99 active+clean; 6.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 60 MiB/s wr, 962 op/s 2024-08-17T18:10:57.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:57 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1860925428' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:10:57.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:57 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:10:57.815 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:57.856 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:57.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:57.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:57.958 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:10:57.994 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:57.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:57.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:10:58.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:58.139 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:58.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:58.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:58.241 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:58.286 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:58.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:58.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:58.388 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:58.422 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:58.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:58.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:58.453 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 79 16 116064 116048 91.7947 62.8125 0.0165651 0.010889 2024-08-17T18:10:58.524 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:58.558 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:58 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:10:58.558 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:58 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:10:58.558 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:58 smithi136 ceph-mon[109861]: osdmap e1546: 8 total, 8 up, 7 in 2024-08-17T18:10:58.559 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:58 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:10:58.559 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:58 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:10:58.559 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:58 smithi136 ceph-mon[115078]: osdmap e1546: 8 total, 8 up, 7 in 2024-08-17T18:10:58.563 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:58.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:58.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:58.665 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:58.695 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:58.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:58.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:58 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:10:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:58 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:10:58.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:58 smithi165 ceph-mon[105517]: osdmap e1546: 8 total, 8 up, 7 in 2024-08-17T18:10:58.796 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:10:58.829 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:58.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:58.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:10:58.931 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:58.980 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:58.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:58.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:59.082 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:59.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:59.122 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:59.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:59.223 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:10:59.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:59.257 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:59.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:10:59.358 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:59.388 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:59.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:59.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:10:59.453 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:2024-08-17T18:10:59.453049+0000 min lat: 0.00107817 max lat: 0.220609 avg lat: 0.01093 2024-08-17T18:10:59.453 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T18:10:59.453 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 80 16 117086 117070 91.4456 63.875 0.017566 0.01093 2024-08-17T18:10:59.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:10:59.518 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:59.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:59.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:10:59.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:10:59.669 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:59.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:59.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:10:59.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:10:59 smithi136 ceph-mon[109861]: pgmap v2851: 99 pgs: 99 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 60 MiB/s wr, 964 op/s 2024-08-17T18:10:59.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:10:59 smithi136 ceph-mon[115078]: pgmap v2851: 99 pgs: 99 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 60 MiB/s wr, 964 op/s 2024-08-17T18:10:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:10:59 smithi165 ceph-mon[105517]: pgmap v2851: 99 pgs: 99 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 60 MiB/s wr, 964 op/s 2024-08-17T18:10:59.771 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:10:59.812 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:10:59.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:59.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:10:59.914 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:10:59.947 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:10:59.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:10:59.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:00.049 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:00.079 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:00.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:00.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:00.181 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:00.224 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:00.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:00.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:00.325 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:00.361 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:00.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:00.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:00.374 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:11:00.453 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 81 16 118069 118053 91.0751 61.4375 0.0226083 0.0109743 2024-08-17T18:11:00.463 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:00.497 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:00.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:00.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:00.599 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:00.644 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:00.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:00.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:00.746 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:00.781 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:00.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:00.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:00.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:00.923 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:00.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:00.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:01.025 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:01.065 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:01.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:01.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:01.166 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:01.198 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:01.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:01.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:01.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:01.335 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:01.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:01.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:01.437 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:01.453 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 82 16 118988 118972 90.6649 57.4375 0.0245632 0.0110246 2024-08-17T18:11:01.473 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:01.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:01.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:01.575 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:01.611 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:01.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:01.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:01 smithi136 ceph-mon[109861]: pgmap v2852: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 60 MiB/s wr, 958 op/s 2024-08-17T18:11:01.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:01 smithi136 ceph-mon[115078]: pgmap v2852: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 60 MiB/s wr, 958 op/s 2024-08-17T18:11:01.712 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:01.751 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:01.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:01.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:01.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:01 smithi165 ceph-mon[105517]: pgmap v2852: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 60 MiB/s wr, 958 op/s 2024-08-17T18:11:01.853 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:01.902 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:01.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:01.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:02.003 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:02.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:02.047 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:02.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:02.148 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:02.181 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:02.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:02.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:02.283 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:02.321 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:02.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:02.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:02.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:02.453 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 83 16 119992 119976 90.3285 62.75 0.0171384 0.0110655 2024-08-17T18:11:02.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:02.459 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:02.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:02.560 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:02.598 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:02.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:02.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:02.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:02.720 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:02.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:02.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:02.821 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:02.856 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:02.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:02.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:02.958 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:02.999 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:02.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:02.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:03.100 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:03.139 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:03.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:03.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:03.241 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:03.282 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:03.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:03.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:03.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:03.423 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:03.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:03.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:03.453 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 84 16 120925 120909 89.9473 58.3125 0.00635224 0.0111119 2024-08-17T18:11:03.525 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:03.572 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:03.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:03.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:03.674 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:03.705 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:03.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:03.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:03 smithi165 ceph-mon[105517]: pgmap v2853: 99 pgs: 99 active+clean; 7.2 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 60 MiB/s wr, 954 op/s 2024-08-17T18:11:03.807 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:03.832 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:03.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:03.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:03.934 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:03 smithi136 ceph-mon[109861]: pgmap v2853: 99 pgs: 99 active+clean; 7.2 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 60 MiB/s wr, 954 op/s 2024-08-17T18:11:03.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:03 smithi136 ceph-mon[115078]: pgmap v2853: 99 pgs: 99 active+clean; 7.2 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 60 MiB/s wr, 954 op/s 2024-08-17T18:11:03.973 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:03.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:03.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:04.075 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:04.112 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:04.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:04.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:04.214 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:04.253 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:04.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:04.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:04.354 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:04.389 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:04.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:04.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:04.435 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:11:04.453 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 85 16 121901 121885 89.6067 61 0.0119183 0.0111546 2024-08-17T18:11:04.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:04.518 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:04.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:04.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:04.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:04.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:04.668 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:04.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:04.762 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:11:04.769 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:04.786 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:11:04.786 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:11:04.821 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:11:04.821 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:11:04.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:04.822 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:04.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:04.859 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:11:04.860 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:11:04.890 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:11:04.891 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:11:04.923 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:04.924 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:11:04.924 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:11:04.959 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:11:04.959 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:11:04.971 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:04.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:04.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:04.997 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:11:04.997 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:11:05.030 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:11:05.030 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:11:05.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:05.107 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:05.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:05.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:05.209 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:05.244 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:05.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:05.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:05.346 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:05.385 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:05.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:05.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:05.453 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 86 16 122813 122797 89.2275 57 0.0337035 0.0112019 2024-08-17T18:11:05.486 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:05.507 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:05.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:05.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:05.609 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:05.644 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:05.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:05.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:05 smithi136 ceph-mon[109861]: pgmap v2854: 99 pgs: 99 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 60 MiB/s wr, 961 op/s 2024-08-17T18:11:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:05 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/987732980' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:11:05.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:05 smithi136 ceph-mon[115078]: pgmap v2854: 99 pgs: 99 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 60 MiB/s wr, 961 op/s 2024-08-17T18:11:05.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:05 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/987732980' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:11:05.746 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:05 smithi165 ceph-mon[105517]: pgmap v2854: 99 pgs: 99 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 60 MiB/s wr, 961 op/s 2024-08-17T18:11:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:05 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/987732980' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:11:05.780 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:05.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:05.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:05.881 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:05.922 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:05.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:05.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:06.024 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:06.061 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:06.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:06.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:06.163 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:06.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:06.196 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:06.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:06.297 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:06.326 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:06.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:06.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:06.427 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:06.453 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 87 16 123779 123763 88.8957 60.375 0.0159612 0.0112442 2024-08-17T18:11:06.469 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:06.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:06.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:06.570 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:06.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:06.611 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:06.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:06.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:11:06.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:11:06.712 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:06.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:06.758 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:06.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:06.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:11:06.859 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:06.900 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:06.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:06.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:07.002 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:07.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:07.031 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:07.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:07.132 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:07.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:07.170 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:07.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:07.271 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:07.311 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:07.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:07.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:07.413 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:07.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:07.443 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:07.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:07.453 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 88 16 124762 124746 88.5836 61.4375 0.00767751 0.0112834 2024-08-17T18:11:07.544 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:07.582 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:07.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:07.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:07.683 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:07.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:07 smithi136 ceph-mon[109861]: pgmap v2855: 99 pgs: 99 active+clean; 7.5 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 60 MiB/s wr, 964 op/s 2024-08-17T18:11:07.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:07 smithi136 ceph-mon[115078]: pgmap v2855: 99 pgs: 99 active+clean; 7.5 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 60 MiB/s wr, 964 op/s 2024-08-17T18:11:07.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:11:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:11:07] "GET /metrics HTTP/1.1" 200 36608 "" "Prometheus/2.51.0" 2024-08-17T18:11:07.713 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:07.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:07.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:07.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:07 smithi165 ceph-mon[105517]: pgmap v2855: 99 pgs: 99 active+clean; 7.5 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 60 MiB/s wr, 964 op/s 2024-08-17T18:11:07.815 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:07.848 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:07.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:07.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:07.950 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:07.982 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:07.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:07.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:08.084 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:08.118 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:08.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:08.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:08.219 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:08.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:08.263 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:08.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:08.364 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:08.399 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:08.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:08.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:08.453 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 89 16 125759 125743 88.2884 62.3125 0.0196059 0.0113219 2024-08-17T18:11:08.501 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:08.533 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:08.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:08.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:08.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:08.667 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:08.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:08.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:08.768 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:08.809 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:08.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:08.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:08.868 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:11:08.911 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:08.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:08.961 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:08.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:09.062 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:09.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:09.114 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:09.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:09.215 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:09.245 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:09.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:09.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:09.347 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:09.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:09.394 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:09.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:09.453 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Total time run: 90.0057 2024-08-17T18:11:09.454 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Total writes made: 126701 2024-08-17T18:11:09.454 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Write size: 65536 2024-08-17T18:11:09.454 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Object size: 65536 2024-08-17T18:11:09.454 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Bandwidth (MB/sec): 87.9812 2024-08-17T18:11:09.454 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Stddev Bandwidth: 37.4992 2024-08-17T18:11:09.454 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Max bandwidth (MB/sec): 217.5 2024-08-17T18:11:09.454 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Min bandwidth (MB/sec): 54.375 2024-08-17T18:11:09.454 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Average IOPS: 1407 2024-08-17T18:11:09.454 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Stddev IOPS: 599.987 2024-08-17T18:11:09.455 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Max IOPS: 3480 2024-08-17T18:11:09.455 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Min IOPS: 870 2024-08-17T18:11:09.455 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Average Latency(s): 0.0113618 2024-08-17T18:11:09.455 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Stddev Latency(s): 0.00927934 2024-08-17T18:11:09.455 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Max latency(s): 0.220609 2024-08-17T18:11:09.455 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Min latency(s): 0.00107817 2024-08-17T18:11:09.461 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Cleaning up (deleting benchmark objects) 2024-08-17T18:11:09.495 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:09.524 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:09.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:09.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:09.625 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:09.657 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:09.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:09.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:09.758 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:09 smithi165 ceph-mon[105517]: pgmap v2856: 99 pgs: 99 active+clean; 7.5 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 57 MiB/s wr, 918 op/s 2024-08-17T18:11:09.786 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:09.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:09.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:09.887 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:09 smithi136 ceph-mon[109861]: pgmap v2856: 99 pgs: 99 active+clean; 7.5 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 57 MiB/s wr, 918 op/s 2024-08-17T18:11:09.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:09 smithi136 ceph-mon[115078]: pgmap v2856: 99 pgs: 99 active+clean; 7.5 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 57 MiB/s wr, 918 op/s 2024-08-17T18:11:09.888 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:09.919 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:09.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:09.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:10.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:10.049 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:10.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:10.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:10.151 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:10.187 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:10.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:10.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:10.289 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:10.323 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:10.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:10.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:10.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:10.458 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:10.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:10.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:10.523 INFO:teuthology.orchestra.run.smithi136.stderr:set pool 1 pgp_num to 11 2024-08-17T18:11:10.560 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:10.604 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:10.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:10.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:10.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:10.736 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:10.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:10.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:10 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2201556622' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T18:11:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:10 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T18:11:10.838 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:10.876 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:10.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:10.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:10.942 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:10 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2201556622' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T18:11:10.942 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:10 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T18:11:10.942 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:10 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2201556622' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T18:11:10.942 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:10 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T18:11:10.978 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:11.024 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:11.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:11.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:11.069 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:11:11.069 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:11:11.126 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:11.162 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:11.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:11.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:11.265 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:11.313 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:11.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:11.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:11.414 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:11.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:11.458 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:11.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:11.558 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:11.610 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:11.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:11.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:11 smithi136 ceph-mon[109861]: pgmap v2857: 99 pgs: 99 active+clean; 7.6 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 58 MiB/s wr, 932 op/s 2024-08-17T18:11:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:11 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-08-17T18:11:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:11 smithi136 ceph-mon[109861]: osdmap e1547: 8 total, 8 up, 7 in 2024-08-17T18:11:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:11:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:11 smithi136 ceph-mon[115078]: pgmap v2857: 99 pgs: 99 active+clean; 7.6 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 58 MiB/s wr, 932 op/s 2024-08-17T18:11:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:11 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-08-17T18:11:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:11 smithi136 ceph-mon[115078]: osdmap e1547: 8 total, 8 up, 7 in 2024-08-17T18:11:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:11 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:11:11.711 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:11.745 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:11.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:11.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:11.778 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-17T18:11:11.847 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:11.886 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:11.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:11.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:11.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:12.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:11 smithi165 ceph-mon[105517]: pgmap v2857: 99 pgs: 99 active+clean; 7.6 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 58 MiB/s wr, 932 op/s 2024-08-17T18:11:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:11 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-08-17T18:11:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:11 smithi165 ceph-mon[105517]: osdmap e1547: 8 total, 8 up, 7 in 2024-08-17T18:11:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:11 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:11:12.027 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:12.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:12.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:12.129 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:12.163 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:12.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:12.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:12.269 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:12.308 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:12.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:12.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:12.410 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:12.450 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:12.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:12.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:12.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:12.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:12.587 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:12.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:12.688 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:12.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:12.736 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:12.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:12.837 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:12.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:12.873 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:12.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:12.974 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:13.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:13.011 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:13.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:13.113 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:13.151 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:13.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:13.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:13.253 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:13.257 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:11:13.260 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:11:13.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:13.310 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:13.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:13.411 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:13.450 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:13.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:13.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:13.529 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:11:13.552 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:13.578 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:13.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:13.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:13.670 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:13 smithi136 ceph-mon[109861]: pgmap v2859: 99 pgs: 99 active+clean; 7.7 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 102 B/s rd, 55 MiB/s wr, 967 op/s 2024-08-17T18:11:13.670 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:13 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1331244301' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:11:13.671 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:13 smithi136 ceph-mon[115078]: pgmap v2859: 99 pgs: 99 active+clean; 7.7 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 102 B/s rd, 55 MiB/s wr, 967 op/s 2024-08-17T18:11:13.671 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:13 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1331244301' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:11:13.679 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:13.715 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:13.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:13.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:13 smithi165 ceph-mon[105517]: pgmap v2859: 99 pgs: 99 active+clean; 7.7 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 102 B/s rd, 55 MiB/s wr, 967 op/s 2024-08-17T18:11:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:13 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1331244301' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:11:13.817 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:13.850 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:13.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:13.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:13.952 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:13.994 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:13.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:13.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:14.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:14.136 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:14.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:14.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:14.238 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:14.269 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:14.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:14.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:14.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:14.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:14.399 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:14.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:14.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:14.525 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:14.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:14.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:14.627 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:14.665 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:14.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:14.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:14.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:14 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:11:14.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:14 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T18:11:14.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:14 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1331244301' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:11:14.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:14 smithi136 ceph-mon[109861]: osdmap e1548: 8 total, 8 up, 7 in 2024-08-17T18:11:14.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:14 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:11:14.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:14 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T18:11:14.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:14 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1331244301' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:11:14.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:14 smithi136 ceph-mon[115078]: osdmap e1548: 8 total, 8 up, 7 in 2024-08-17T18:11:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:14 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:11:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:14 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T18:11:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:14 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1331244301' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:11:14.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:14 smithi165 ceph-mon[105517]: osdmap e1548: 8 total, 8 up, 7 in 2024-08-17T18:11:14.767 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:14.801 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:14.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:14.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:14.902 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:14.934 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:14.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:14.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:15.036 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:15.070 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:15.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:15.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:15.172 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:15.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:15.232 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:15.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:15.333 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:15.367 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:15.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:15.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:15.469 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:15.508 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:15.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:15.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:15.610 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:15.638 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:15.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:15.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:15.740 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:15.780 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:15.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:15.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:15.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:15 smithi165 ceph-mon[105517]: pgmap v2861: 99 pgs: 99 active+clean; 7.6 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 127 B/s rd, 35 MiB/s wr, 873 op/s 2024-08-17T18:11:15.882 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:15.917 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:15.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:15.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:15.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:15 smithi136 ceph-mon[109861]: pgmap v2861: 99 pgs: 99 active+clean; 7.6 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 127 B/s rd, 35 MiB/s wr, 873 op/s 2024-08-17T18:11:15.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:15 smithi136 ceph-mon[115078]: pgmap v2861: 99 pgs: 99 active+clean; 7.6 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 127 B/s rd, 35 MiB/s wr, 873 op/s 2024-08-17T18:11:16.019 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:16.040 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:16.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:16.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:16.142 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:16.177 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:16.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:16.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:16.279 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:16.312 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:16.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:16.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:16.414 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:16.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:16.456 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:16.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:16.557 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:16.601 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:16.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:16.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:16.704 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:16.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:16.741 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:16.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:16.841 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:16.877 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:16.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:16.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:16.979 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:17.015 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:17.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:17.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:17.116 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:17.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:17.147 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:17.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:17.248 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:17.285 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:17.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:17.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:17.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:17.422 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:17.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:17.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:17.524 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:17.558 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:17.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:17.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:17.660 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:17.682 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:17.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:17.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:17.749 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:17 smithi136 ceph-mon[109861]: pgmap v2862: 99 pgs: 99 active+clean; 7.4 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 127 B/s rd, 30 MiB/s wr, 1.13k op/s 2024-08-17T18:11:17.749 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:17 smithi136 ceph-mon[115078]: pgmap v2862: 99 pgs: 99 active+clean; 7.4 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 127 B/s rd, 30 MiB/s wr, 1.13k op/s 2024-08-17T18:11:17.750 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:11:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:11:17] "GET /metrics HTTP/1.1" 200 36608 "" "Prometheus/2.51.0" 2024-08-17T18:11:17.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:17.825 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:17.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:17.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:17.927 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:17.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:17.968 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:17.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:17 smithi165 ceph-mon[105517]: pgmap v2862: 99 pgs: 99 active+clean; 7.4 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 127 B/s rd, 30 MiB/s wr, 1.13k op/s 2024-08-17T18:11:18.069 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:18.113 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:18.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:18.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:18.153 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 0, 5] out_osds: [7] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:11:18.153 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:11:18.154 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-08-17T18:11:18.154 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T18:11:18.215 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:18.248 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:18.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:18.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:18.350 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:18.398 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:18.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:18.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:18.501 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:18.533 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:18.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:18.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:18.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:18.657 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:18.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:18.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:18.758 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:18.780 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:18.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:18.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:18.881 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:18.911 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:18.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:18.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:19.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:19.045 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:19.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:19.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:19.147 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:19.181 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:19.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:19.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:19.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:11:19.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:11:19.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:18 smithi136 ceph-mon[109861]: pgmap v2863: 99 pgs: 99 active+clean; 7.4 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 127 B/s rd, 18 MiB/s wr, 1.01k op/s 2024-08-17T18:11:19.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:11:19.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:11:19.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:18 smithi136 ceph-mon[115078]: pgmap v2863: 99 pgs: 99 active+clean; 7.4 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 127 B/s rd, 18 MiB/s wr, 1.01k op/s 2024-08-17T18:11:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:11:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:11:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:18 smithi165 ceph-mon[105517]: pgmap v2863: 99 pgs: 99 active+clean; 7.4 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 127 B/s rd, 18 MiB/s wr, 1.01k op/s 2024-08-17T18:11:19.283 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:19.317 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:19.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:19.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:19.419 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:19.457 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:19.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:19.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:19.559 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:19.589 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:19.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:19.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:19.691 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:19.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:19.740 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:19.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:19.841 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:19.872 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:19.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:19.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:19.975 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:20.014 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:20.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:20.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:20.115 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:20.148 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:20.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:20.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:20.250 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:20.285 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:20.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:20.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:20.386 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:20.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:20.428 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:20.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:20.529 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:20.571 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:20.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:20.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:20.673 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:20.700 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:20.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:20.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:20.802 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:20.831 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:20.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:20.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:20.933 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:20.966 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:20.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:20.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:21.068 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:21.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:21.098 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:21.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:21.199 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:21.227 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:21.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:21.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:21.329 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:21.378 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:21.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:21.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:21.480 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:21.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:21.536 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:21.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:21.637 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:21.655 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:21 smithi136 ceph-mon[109861]: pgmap v2864: 99 pgs: 99 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 104 B/s rd, 15 MiB/s wr, 966 op/s 2024-08-17T18:11:21.655 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:11:21.655 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:11:21.656 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:21 smithi136 ceph-mon[115078]: pgmap v2864: 99 pgs: 99 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 104 B/s rd, 15 MiB/s wr, 966 op/s 2024-08-17T18:11:21.656 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:11:21.656 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:11:21.673 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:21.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:21.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:21.729 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:11:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:21 smithi165 ceph-mon[105517]: pgmap v2864: 99 pgs: 99 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 104 B/s rd, 15 MiB/s wr, 966 op/s 2024-08-17T18:11:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:11:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:11:21.775 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:21.801 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:21.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:21.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:21.860 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:11:21.860 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:11:21.896 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:11:21.896 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:11:21.903 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:21.960 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:11:21.960 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:11:21.970 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:21.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:21.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:22.019 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:11:22.019 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:11:22.047 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:11:22.047 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:11:22.065 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:11:22.066 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:11:22.072 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:22.093 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:11:22.094 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:11:22.098 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:22.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:22.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:22.121 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:11:22.121 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:11:22.200 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:22.241 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:22.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:22.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:22.343 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:22.377 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:22.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:22.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:22.479 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:22.515 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:22.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:22.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:22.617 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:22.655 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:22.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:22.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:22.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:22 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3372116492' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:11:22.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:22 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3372116492' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:11:22.757 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:22.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:22 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3372116492' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:11:22.801 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:22.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:22.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:22.902 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:22.937 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:22.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:22.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:23.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:23.071 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:23.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:23.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:23.173 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:23.253 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:23.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:23.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:23.322 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:11:23.322 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-17T18:11:23.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-17T18:11:23.322 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:11:23.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.322 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:192944 54.13 54:cbcafd86:::benchmark_data_smithi136_273969_object70059:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1416)", 2024-08-17T18:11:23.322 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:01:47.629197+0000", 2024-08-17T18:11:23.322 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 575.68697565299999, 2024-08-17T18:11:23.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.292988475, 2024-08-17T18:11:23.323 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:11:23.323 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:11:23.323 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:11:23.323 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:11:23.323 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:11:23.323 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:11:23.323 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 192944 2024-08-17T18:11:23.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.323 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:11:23.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:11:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:47.629197+0000", 2024-08-17T18:11:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:11:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:47.629197+0000", 2024-08-17T18:11:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:11:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:47.629197+0000", 2024-08-17T18:11:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0200000000000001e-07 2024-08-17T18:11:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:11:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:47.629198+0000", 2024-08-17T18:11:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.06e-06 2024-08-17T18:11:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:11:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:47.629201+0000", 2024-08-17T18:11:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2499999999999998e-06 2024-08-17T18:11:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:11:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:47.629204+0000", 2024-08-17T18:11:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8660000000000002e-06 2024-08-17T18:11:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:47.629481+0000", 2024-08-17T18:11:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027693599999999998 2024-08-17T18:11:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:11:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:47.629484+0000", 2024-08-17T18:11:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2509999999999999e-06 2024-08-17T18:11:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:48.860615+0000", 2024-08-17T18:11:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.23113067 2024-08-17T18:11:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:11:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:48.860773+0000", 2024-08-17T18:11:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000158333 2024-08-17T18:11:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5", 2024-08-17T18:11:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:48.861356+0000", 2024-08-17T18:11:23.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00058251899999999998 2024-08-17T18:11:23.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:11:23.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:48.868776+0000", 2024-08-17T18:11:23.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0074199289999999996 2024-08-17T18:11:23.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:48.870394+0000", 2024-08-17T18:11:23.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016179020000000001 2024-08-17T18:11:23.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:48.871421+0000", 2024-08-17T18:11:23.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001026999 2024-08-17T18:11:23.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:48.920764+0000", 2024-08-17T18:11:23.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.049343827 2024-08-17T18:11:23.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:11:23.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:48.920812+0000", 2024-08-17T18:11:23.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7104999999999997e-05 2024-08-17T18:11:23.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:11:23.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:48.922185+0000", 2024-08-17T18:11:23.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001373626 2024-08-17T18:11:23.330 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.330 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:11:23.330 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.331 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.35370.0:192948 54.13 54:caed5340:::benchmark_data_smithi136_273969_object70063:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1416)", 2024-08-17T18:11:23.331 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:01:47.629220+0000", 2024-08-17T18:11:23.331 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 575.68695281800001, 2024-08-17T18:11:23.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2953038399999999, 2024-08-17T18:11:23.331 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:11:23.331 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:11:23.331 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:11:23.331 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:11:23.331 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.35370", 2024-08-17T18:11:23.331 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:11:23.331 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 192948 2024-08-17T18:11:23.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.332 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:11:23.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:11:23.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:47.629220+0000", 2024-08-17T18:11:23.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:11:23.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:47.629220+0000", 2024-08-17T18:11:23.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:11:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:47.629220+0000", 2024-08-17T18:11:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5600000000000002e-07 2024-08-17T18:11:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:11:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:47.629221+0000", 2024-08-17T18:11:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.328e-06 2024-08-17T18:11:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:11:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:47.629225+0000", 2024-08-17T18:11:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5379999999999998e-06 2024-08-17T18:11:23.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:11:23.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:47.629227+0000", 2024-08-17T18:11:23.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7879999999999998e-06 2024-08-17T18:11:23.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:47.629507+0000", 2024-08-17T18:11:23.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027944899999999997 2024-08-17T18:11:23.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:11:23.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:47.629514+0000", 2024-08-17T18:11:23.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4970000000000004e-06 2024-08-17T18:11:23.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:48.864663+0000", 2024-08-17T18:11:23.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2351489410000001 2024-08-17T18:11:23.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:11:23.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:48.864830+0000", 2024-08-17T18:11:23.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016613500000000001 2024-08-17T18:11:23.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5", 2024-08-17T18:11:23.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:48.865368+0000", 2024-08-17T18:11:23.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00053813799999999996 2024-08-17T18:11:23.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:11:23.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:48.870479+0000", 2024-08-17T18:11:23.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0051111009999999998 2024-08-17T18:11:23.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:48.871465+0000", 2024-08-17T18:11:23.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00098596800000000004 2024-08-17T18:11:23.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:48.875045+0000", 2024-08-17T18:11:23.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035806259999999999 2024-08-17T18:11:23.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:48.922246+0000", 2024-08-17T18:11:23.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.047200472 2024-08-17T18:11:23.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:11:23.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:48.922456+0000", 2024-08-17T18:11:23.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021042700000000001 2024-08-17T18:11:23.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:11:23.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:01:48.924523+0000", 2024-08-17T18:11:23.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002067176 2024-08-17T18:11:23.338 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.338 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:11:23.338 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.339 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.45626.0:170335 55.2 55:4037a7bd:::benchmark_data_smithi136_313476_object78349:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1515)", 2024-08-17T18:11:23.339 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:08:36.080758+0000", 2024-08-17T18:11:23.339 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 167.23541471300001, 2024-08-17T18:11:23.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0985168270000001, 2024-08-17T18:11:23.339 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:11:23.339 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:11:23.339 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:11:23.339 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:11:23.339 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.45626", 2024-08-17T18:11:23.339 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:11:23.340 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 170335 2024-08-17T18:11:23.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.340 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:11:23.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:11:23.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.080758+0000", 2024-08-17T18:11:23.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:11:23.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.080758+0000", 2024-08-17T18:11:23.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.341 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:11:23.341 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.080759+0000", 2024-08-17T18:11:23.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.372e-06 2024-08-17T18:11:23.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.341 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:11:23.341 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.080764+0000", 2024-08-17T18:11:23.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6450000000000004e-06 2024-08-17T18:11:23.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.341 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:11:23.341 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.080773+0000", 2024-08-17T18:11:23.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5580000000000006e-06 2024-08-17T18:11:23.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:11:23.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.080780+0000", 2024-08-17T18:11:23.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3409999999999999e-06 2024-08-17T18:11:23.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.080848+0000", 2024-08-17T18:11:23.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8758999999999997e-05 2024-08-17T18:11:23.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:11:23.343 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.080860+0000", 2024-08-17T18:11:23.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1907e-05 2024-08-17T18:11:23.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.343 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.343 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.059824+0000", 2024-08-17T18:11:23.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97896355000000002 2024-08-17T18:11:23.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.343 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:11:23.343 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.059838+0000", 2024-08-17T18:11:23.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3708e-05 2024-08-17T18:11:23.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.344 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.344 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.173362+0000", 2024-08-17T18:11:23.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.113524307 2024-08-17T18:11:23.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.344 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:11:23.344 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.173469+0000", 2024-08-17T18:11:23.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106801 2024-08-17T18:11:23.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5,6", 2024-08-17T18:11:23.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.173805+0000", 2024-08-17T18:11:23.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00033645600000000001 2024-08-17T18:11:23.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:11:23.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.175758+0000", 2024-08-17T18:11:23.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0019531589999999999 2024-08-17T18:11:23.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.178437+0000", 2024-08-17T18:11:23.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0026783800000000002 2024-08-17T18:11:23.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.346 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.178483+0000", 2024-08-17T18:11:23.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6755000000000002e-05 2024-08-17T18:11:23.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.346 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.179161+0000", 2024-08-17T18:11:23.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00067756200000000004 2024-08-17T18:11:23.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:11:23.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.179200+0000", 2024-08-17T18:11:23.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8868000000000001e-05 2024-08-17T18:11:23.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:11:23.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.179275+0000", 2024-08-17T18:11:23.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4698999999999997e-05 2024-08-17T18:11:23.347 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.347 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:11:23.347 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.347 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.45626.0:170347 55.2 55:4738474e:::benchmark_data_smithi136_313476_object78361:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1515)", 2024-08-17T18:11:23.347 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:08:36.094869+0000", 2024-08-17T18:11:23.348 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 167.221303715, 2024-08-17T18:11:23.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0887959270000001, 2024-08-17T18:11:23.348 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:11:23.348 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:11:23.348 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:11:23.348 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:11:23.348 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.45626", 2024-08-17T18:11:23.348 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:11:23.348 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 170347 2024-08-17T18:11:23.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.348 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:11:23.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:11:23.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.094869+0000", 2024-08-17T18:11:23.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:11:23.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.094869+0000", 2024-08-17T18:11:23.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:11:23.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.094870+0000", 2024-08-17T18:11:23.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6309999999999999e-06 2024-08-17T18:11:23.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:11:23.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.094877+0000", 2024-08-17T18:11:23.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2330000000000003e-06 2024-08-17T18:11:23.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:11:23.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.094901+0000", 2024-08-17T18:11:23.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4813000000000001e-05 2024-08-17T18:11:23.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:11:23.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.094911+0000", 2024-08-17T18:11:23.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3249999999999997e-06 2024-08-17T18:11:23.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.095035+0000", 2024-08-17T18:11:23.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000124554 2024-08-17T18:11:23.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:11:23.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.095046+0000", 2024-08-17T18:11:23.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1032999999999999e-05 2024-08-17T18:11:23.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.059862+0000", 2024-08-17T18:11:23.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.96481608299999999 2024-08-17T18:11:23.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:11:23.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.059870+0000", 2024-08-17T18:11:23.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2080000000000003e-06 2024-08-17T18:11:23.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.175786+0000", 2024-08-17T18:11:23.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1159161850000001 2024-08-17T18:11:23.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:11:23.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.175914+0000", 2024-08-17T18:11:23.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012778600000000001 2024-08-17T18:11:23.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.354 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5,6", 2024-08-17T18:11:23.354 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.176280+0000", 2024-08-17T18:11:23.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036621700000000001 2024-08-17T18:11:23.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.354 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:11:23.354 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.178500+0000", 2024-08-17T18:11:23.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0022198169999999998 2024-08-17T18:11:23.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.178869+0000", 2024-08-17T18:11:23.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000369253 2024-08-17T18:11:23.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.179313+0000", 2024-08-17T18:11:23.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044423500000000002 2024-08-17T18:11:23.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.183487+0000", 2024-08-17T18:11:23.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0041739500000000001 2024-08-17T18:11:23.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:11:23.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.183547+0000", 2024-08-17T18:11:23.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0025000000000002e-05 2024-08-17T18:11:23.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:11:23.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.183665+0000", 2024-08-17T18:11:23.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011757899999999999 2024-08-17T18:11:23.356 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.356 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:11:23.356 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.357 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:27947 56.4 56:21cbfe32:::benchmark_data_smithi136_356456_object27946:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1540)", 2024-08-17T18:11:23.357 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:09:49.432911+0000", 2024-08-17T18:11:23.357 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 93.883261633999993, 2024-08-17T18:11:23.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.140233625, 2024-08-17T18:11:23.357 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:11:23.357 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:11:23.357 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:11:23.357 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:11:23.357 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:11:23.357 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:11:23.357 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 27947 2024-08-17T18:11:23.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.358 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:11:23.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:11:23.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.432911+0000", 2024-08-17T18:11:23.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:11:23.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.432911+0000", 2024-08-17T18:11:23.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:11:23.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.432911+0000", 2024-08-17T18:11:23.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9100000000000004e-07 2024-08-17T18:11:23.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:11:23.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.432934+0000", 2024-08-17T18:11:23.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2920000000000001e-05 2024-08-17T18:11:23.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:11:23.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.432943+0000", 2024-08-17T18:11:23.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2660000000000001e-06 2024-08-17T18:11:23.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:11:23.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.432949+0000", 2024-08-17T18:11:23.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3040000000000001e-06 2024-08-17T18:11:23.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.435502+0000", 2024-08-17T18:11:23.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025528130000000001 2024-08-17T18:11:23.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:11:23.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.435633+0000", 2024-08-17T18:11:23.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013102 2024-08-17T18:11:23.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4", 2024-08-17T18:11:23.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.435772+0000", 2024-08-17T18:11:23.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013916500000000001 2024-08-17T18:11:23.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:11:23.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.440445+0000", 2024-08-17T18:11:23.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0046734710000000002 2024-08-17T18:11:23.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.440508+0000", 2024-08-17T18:11:23.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2648000000000001e-05 2024-08-17T18:11:23.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573031+0000", 2024-08-17T18:11:23.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13252335000000001 2024-08-17T18:11:23.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:11:23.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573092+0000", 2024-08-17T18:11:23.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0747e-05 2024-08-17T18:11:23.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:11:23.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573144+0000", 2024-08-17T18:11:23.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2330000000000002e-05 2024-08-17T18:11:23.363 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.363 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:11:23.364 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.364 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:27951 56.4 56:222eaaba:::benchmark_data_smithi136_356456_object27950:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1540)", 2024-08-17T18:11:23.364 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:09:49.433998+0000", 2024-08-17T18:11:23.364 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 93.882173949000006, 2024-08-17T18:11:23.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13954208600000001, 2024-08-17T18:11:23.364 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:11:23.364 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:11:23.364 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:11:23.365 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:11:23.365 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:11:23.365 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:11:23.365 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 27951 2024-08-17T18:11:23.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.365 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:11:23.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:11:23.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.433998+0000", 2024-08-17T18:11:23.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:11:23.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.433998+0000", 2024-08-17T18:11:23.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:11:23.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.433999+0000", 2024-08-17T18:11:23.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0400000000000005e-07 2024-08-17T18:11:23.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:11:23.367 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434024+0000", 2024-08-17T18:11:23.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.491e-05 2024-08-17T18:11:23.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:11:23.367 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434035+0000", 2024-08-17T18:11:23.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1129e-05 2024-08-17T18:11:23.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:11:23.367 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434043+0000", 2024-08-17T18:11:23.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3069999999999997e-06 2024-08-17T18:11:23.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.368 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.368 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.368 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.437081+0000", 2024-08-17T18:11:23.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0030388279999999999 2024-08-17T18:11:23.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.368 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.368 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:11:23.368 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.437217+0000", 2024-08-17T18:11:23.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013517800000000001 2024-08-17T18:11:23.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4", 2024-08-17T18:11:23.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.437342+0000", 2024-08-17T18:11:23.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125091 2024-08-17T18:11:23.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:11:23.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.440518+0000", 2024-08-17T18:11:23.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003176636 2024-08-17T18:11:23.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.440831+0000", 2024-08-17T18:11:23.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00031303800000000002 2024-08-17T18:11:23.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573416+0000", 2024-08-17T18:11:23.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.132585077 2024-08-17T18:11:23.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:11:23.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573469+0000", 2024-08-17T18:11:23.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2899000000000003e-05 2024-08-17T18:11:23.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:11:23.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573541+0000", 2024-08-17T18:11:23.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1188999999999999e-05 2024-08-17T18:11:23.371 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.371 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:11:23.371 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.371 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:27954 56.4 56:216fadd1:::benchmark_data_smithi136_356456_object27953:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1540)", 2024-08-17T18:11:23.372 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:09:49.434097+0000", 2024-08-17T18:11:23.372 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 93.882075549000007, 2024-08-17T18:11:23.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13963388299999999, 2024-08-17T18:11:23.372 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:11:23.372 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:11:23.372 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:11:23.372 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:11:23.372 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:11:23.372 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:11:23.372 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 27954 2024-08-17T18:11:23.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.373 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:11:23.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:11:23.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434097+0000", 2024-08-17T18:11:23.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:11:23.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434097+0000", 2024-08-17T18:11:23.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:11:23.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434097+0000", 2024-08-17T18:11:23.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7800000000000002e-07 2024-08-17T18:11:23.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:11:23.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434118+0000", 2024-08-17T18:11:23.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1015000000000001e-05 2024-08-17T18:11:23.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:11:23.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434126+0000", 2024-08-17T18:11:23.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4490000000000002e-06 2024-08-17T18:11:23.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:11:23.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434131+0000", 2024-08-17T18:11:23.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8470000000000003e-06 2024-08-17T18:11:23.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.438432+0000", 2024-08-17T18:11:23.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0043018010000000001 2024-08-17T18:11:23.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:11:23.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439024+0000", 2024-08-17T18:11:23.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00059143900000000003 2024-08-17T18:11:23.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4", 2024-08-17T18:11:23.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439189+0000", 2024-08-17T18:11:23.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016538999999999999 2024-08-17T18:11:23.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.440867+0000", 2024-08-17T18:11:23.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016782069999999999 2024-08-17T18:11:23.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:11:23.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573554+0000", 2024-08-17T18:11:23.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13268660700000001 2024-08-17T18:11:23.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573637+0000", 2024-08-17T18:11:23.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3016999999999995e-05 2024-08-17T18:11:23.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:11:23.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573686+0000", 2024-08-17T18:11:23.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9038e-05 2024-08-17T18:11:23.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.378 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:11:23.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573731+0000", 2024-08-17T18:11:23.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4595000000000002e-05 2024-08-17T18:11:23.378 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.378 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:11:23.378 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.379 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:27955 56.4 56:2138118c:::benchmark_data_smithi136_356456_object27954:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1540)", 2024-08-17T18:11:23.379 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:09:49.434139+0000", 2024-08-17T18:11:23.379 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 93.882033747999998, 2024-08-17T18:11:23.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13974101799999999, 2024-08-17T18:11:23.379 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:11:23.379 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:11:23.379 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:11:23.379 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:11:23.379 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:11:23.379 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:11:23.379 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 27955 2024-08-17T18:11:23.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.379 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:11:23.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:11:23.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434139+0000", 2024-08-17T18:11:23.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:11:23.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434139+0000", 2024-08-17T18:11:23.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:11:23.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434139+0000", 2024-08-17T18:11:23.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7899999999999998e-07 2024-08-17T18:11:23.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:11:23.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434160+0000", 2024-08-17T18:11:23.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0937999999999998e-05 2024-08-17T18:11:23.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:11:23.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434166+0000", 2024-08-17T18:11:23.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1279999999999996e-06 2024-08-17T18:11:23.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:11:23.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434172+0000", 2024-08-17T18:11:23.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3789999999999997e-06 2024-08-17T18:11:23.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.441201+0000", 2024-08-17T18:11:23.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0070293500000000002 2024-08-17T18:11:23.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:11:23.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.441417+0000", 2024-08-17T18:11:23.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000215813 2024-08-17T18:11:23.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4", 2024-08-17T18:11:23.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.441553+0000", 2024-08-17T18:11:23.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013591300000000001 2024-08-17T18:11:23.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:11:23.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573560+0000", 2024-08-17T18:11:23.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.132007491 2024-08-17T18:11:23.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573600+0000", 2024-08-17T18:11:23.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9687999999999997e-05 2024-08-17T18:11:23.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.385 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573801+0000", 2024-08-17T18:11:23.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020095200000000001 2024-08-17T18:11:23.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:11:23.385 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573845+0000", 2024-08-17T18:11:23.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4202999999999997e-05 2024-08-17T18:11:23.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:11:23.385 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573880+0000", 2024-08-17T18:11:23.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4584e-05 2024-08-17T18:11:23.386 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.386 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:11:23.386 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.386 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:27962 56.f 56:faa63ae2:::benchmark_data_smithi136_356456_object27961:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1540)", 2024-08-17T18:11:23.386 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:09:49.438972+0000", 2024-08-17T18:11:23.386 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 93.877200055000003, 2024-08-17T18:11:23.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.138606172, 2024-08-17T18:11:23.386 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:11:23.387 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:11:23.387 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:11:23.387 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:11:23.387 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:11:23.387 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:11:23.387 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 27962 2024-08-17T18:11:23.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.387 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:11:23.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:11:23.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.438972+0000", 2024-08-17T18:11:23.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:11:23.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.438972+0000", 2024-08-17T18:11:23.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:11:23.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.438973+0000", 2024-08-17T18:11:23.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5600000000000005e-07 2024-08-17T18:11:23.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:11:23.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.438996+0000", 2024-08-17T18:11:23.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3244e-05 2024-08-17T18:11:23.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:11:23.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439004+0000", 2024-08-17T18:11:23.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7489999999999998e-06 2024-08-17T18:11:23.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:11:23.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439011+0000", 2024-08-17T18:11:23.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5080000000000002e-06 2024-08-17T18:11:23.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439253+0000", 2024-08-17T18:11:23.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000242105 2024-08-17T18:11:23.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:11:23.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439372+0000", 2024-08-17T18:11:23.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119466 2024-08-17T18:11:23.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,6", 2024-08-17T18:11:23.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439497+0000", 2024-08-17T18:11:23.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012526700000000001 2024-08-17T18:11:23.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.574682+0000", 2024-08-17T18:11:23.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13518470399999999 2024-08-17T18:11:23.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.574727+0000", 2024-08-17T18:11:23.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4703999999999998e-05 2024-08-17T18:11:23.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:11:23.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.577465+0000", 2024-08-17T18:11:23.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0027381950000000001 2024-08-17T18:11:23.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:11:23.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.577529+0000", 2024-08-17T18:11:23.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4333999999999998e-05 2024-08-17T18:11:23.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:11:23.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.577579+0000", 2024-08-17T18:11:23.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.914e-05 2024-08-17T18:11:23.393 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.393 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:11:23.393 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.393 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:27966 56.f 56:f201f13f:::benchmark_data_smithi136_356456_object27965:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1540)", 2024-08-17T18:11:23.393 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:09:49.439427+0000", 2024-08-17T18:11:23.393 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 93.876745213999996, 2024-08-17T18:11:23.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13837285699999999, 2024-08-17T18:11:23.393 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:11:23.394 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:11:23.394 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:11:23.394 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:11:23.394 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:11:23.394 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:11:23.394 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 27966 2024-08-17T18:11:23.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.394 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:11:23.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:11:23.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439427+0000", 2024-08-17T18:11:23.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:11:23.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439427+0000", 2024-08-17T18:11:23.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:11:23.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439428+0000", 2024-08-17T18:11:23.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.54e-07 2024-08-17T18:11:23.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:11:23.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439459+0000", 2024-08-17T18:11:23.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0703000000000001e-05 2024-08-17T18:11:23.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:11:23.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439469+0000", 2024-08-17T18:11:23.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8730000000000002e-06 2024-08-17T18:11:23.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:11:23.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439475+0000", 2024-08-17T18:11:23.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4779999999999996e-06 2024-08-17T18:11:23.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.441807+0000", 2024-08-17T18:11:23.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023322680000000002 2024-08-17T18:11:23.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:11:23.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.441929+0000", 2024-08-17T18:11:23.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121724 2024-08-17T18:11:23.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,6", 2024-08-17T18:11:23.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573147+0000", 2024-08-17T18:11:23.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13121764799999999 2024-08-17T18:11:23.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.577267+0000", 2024-08-17T18:11:23.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0041202089999999997 2024-08-17T18:11:23.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.577451+0000", 2024-08-17T18:11:23.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018379400000000001 2024-08-17T18:11:23.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:11:23.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.577585+0000", 2024-08-17T18:11:23.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013449499999999999 2024-08-17T18:11:23.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:11:23.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:23.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.577625+0000", 2024-08-17T18:11:23.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0074000000000003e-05 2024-08-17T18:11:23.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:11:23.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.577800+0000", 2024-08-17T18:11:23.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017483700000000001 2024-08-17T18:11:23.401 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.401 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:11:23.401 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.402 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:88733 56.f 56:fb6e25d6:::benchmark_data_smithi136_356456_object88732:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1544)", 2024-08-17T18:11:23.402 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.476787+0000", 2024-08-17T18:11:23.402 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 51.839385429000004, 2024-08-17T18:11:23.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17365398000000001, 2024-08-17T18:11:23.402 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:11:23.402 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:11:23.402 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:11:23.402 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:11:23.402 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:11:23.402 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:11:23.402 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 88733 2024-08-17T18:11:23.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.403 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:11:23.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:11:23.403 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.476787+0000", 2024-08-17T18:11:23.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:11:23.403 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.476787+0000", 2024-08-17T18:11:23.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:11:23.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.476788+0000", 2024-08-17T18:11:23.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8999999999999995e-07 2024-08-17T18:11:23.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:11:23.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.476809+0000", 2024-08-17T18:11:23.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1331999999999998e-05 2024-08-17T18:11:23.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:11:23.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.476819+0000", 2024-08-17T18:11:23.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9799999999999993e-06 2024-08-17T18:11:23.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:11:23.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.476826+0000", 2024-08-17T18:11:23.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.097e-06 2024-08-17T18:11:23.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.476868+0000", 2024-08-17T18:11:23.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2256000000000003e-05 2024-08-17T18:11:23.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:11:23.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.477009+0000", 2024-08-17T18:11:23.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014094800000000001 2024-08-17T18:11:23.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,6", 2024-08-17T18:11:23.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.477144+0000", 2024-08-17T18:11:23.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013450699999999999 2024-08-17T18:11:23.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.481751+0000", 2024-08-17T18:11:23.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0046066379999999997 2024-08-17T18:11:23.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:11:23.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.481761+0000", 2024-08-17T18:11:23.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0548e-05 2024-08-17T18:11:23.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650331+0000", 2024-08-17T18:11:23.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16856977100000001 2024-08-17T18:11:23.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:11:23.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650393+0000", 2024-08-17T18:11:23.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1925000000000001e-05 2024-08-17T18:11:23.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:11:23.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650441+0000", 2024-08-17T18:11:23.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8188e-05 2024-08-17T18:11:23.408 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.408 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:11:23.408 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.408 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:88735 56.c 56:3d1600ab:::benchmark_data_smithi136_356456_object88734:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1544)", 2024-08-17T18:11:23.408 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.480738+0000", 2024-08-17T18:11:23.408 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 51.835434417999998, 2024-08-17T18:11:23.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17329349599999999, 2024-08-17T18:11:23.408 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:11:23.409 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:11:23.409 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:11:23.409 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:11:23.409 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:11:23.409 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:11:23.409 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 88735 2024-08-17T18:11:23.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.409 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:11:23.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.409 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:11:23.409 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.480738+0000", 2024-08-17T18:11:23.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.410 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:11:23.410 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.480738+0000", 2024-08-17T18:11:23.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.410 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:11:23.410 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.480740+0000", 2024-08-17T18:11:23.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5710000000000001e-06 2024-08-17T18:11:23.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.410 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:11:23.410 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.480762+0000", 2024-08-17T18:11:23.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2629999999999998e-05 2024-08-17T18:11:23.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.411 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:11:23.411 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.480775+0000", 2024-08-17T18:11:23.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3027000000000001e-05 2024-08-17T18:11:23.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.411 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:11:23.411 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.480783+0000", 2024-08-17T18:11:23.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4989999999999997e-06 2024-08-17T18:11:23.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.411 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.480844+0000", 2024-08-17T18:11:23.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1001000000000001e-05 2024-08-17T18:11:23.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:11:23.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.480988+0000", 2024-08-17T18:11:23.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000144723 2024-08-17T18:11:23.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:11:23.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.481128+0000", 2024-08-17T18:11:23.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013966599999999999 2024-08-17T18:11:23.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484806+0000", 2024-08-17T18:11:23.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036775620000000001 2024-08-17T18:11:23.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:11:23.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.485844+0000", 2024-08-17T18:11:23.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001037888 2024-08-17T18:11:23.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653923+0000", 2024-08-17T18:11:23.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16807926300000001 2024-08-17T18:11:23.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:11:23.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653983+0000", 2024-08-17T18:11:23.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9837999999999999e-05 2024-08-17T18:11:23.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:11:23.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.654031+0000", 2024-08-17T18:11:23.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8828000000000001e-05 2024-08-17T18:11:23.414 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.414 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:11:23.414 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.415 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:88739 56.f 56:f7ad6e62:::benchmark_data_smithi136_356456_object88738:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1544)", 2024-08-17T18:11:23.415 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.484508+0000", 2024-08-17T18:11:23.415 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 51.831664345999997, 2024-08-17T18:11:23.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.166084435, 2024-08-17T18:11:23.415 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:11:23.415 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:11:23.415 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:11:23.415 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:11:23.415 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:11:23.415 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:11:23.415 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 88739 2024-08-17T18:11:23.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.416 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:11:23.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:11:23.416 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484508+0000", 2024-08-17T18:11:23.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:11:23.416 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484508+0000", 2024-08-17T18:11:23.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:11:23.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484510+0000", 2024-08-17T18:11:23.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.592e-06 2024-08-17T18:11:23.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:11:23.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484536+0000", 2024-08-17T18:11:23.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6548999999999999e-05 2024-08-17T18:11:23.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:11:23.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484550+0000", 2024-08-17T18:11:23.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3585e-05 2024-08-17T18:11:23.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:11:23.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484558+0000", 2024-08-17T18:11:23.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.405e-06 2024-08-17T18:11:23.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484709+0000", 2024-08-17T18:11:23.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015043899999999999 2024-08-17T18:11:23.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.419 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:11:23.419 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484884+0000", 2024-08-17T18:11:23.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000175613 2024-08-17T18:11:23.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.419 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.419 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,6", 2024-08-17T18:11:23.419 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.485059+0000", 2024-08-17T18:11:23.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017479999999999999 2024-08-17T18:11:23.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.419 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.419 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.419 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650489+0000", 2024-08-17T18:11:23.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.165430095 2024-08-17T18:11:23.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:11:23.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650498+0000", 2024-08-17T18:11:23.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9800000000000004e-06 2024-08-17T18:11:23.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650524+0000", 2024-08-17T18:11:23.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5998000000000001e-05 2024-08-17T18:11:23.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:11:23.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650559+0000", 2024-08-17T18:11:23.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4718999999999997e-05 2024-08-17T18:11:23.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:11:23.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650592+0000", 2024-08-17T18:11:23.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.366e-05 2024-08-17T18:11:23.421 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.421 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:11:23.421 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.422 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:88741 56.e 56:77ba4448:::benchmark_data_smithi136_356456_object88740:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1544)", 2024-08-17T18:11:23.422 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.484583+0000", 2024-08-17T18:11:23.422 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 51.831589082999997, 2024-08-17T18:11:23.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16996449199999999, 2024-08-17T18:11:23.422 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:11:23.422 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:11:23.422 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:11:23.422 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:11:23.422 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:11:23.422 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:11:23.422 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 88741 2024-08-17T18:11:23.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.422 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:11:23.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.423 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:11:23.423 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484583+0000", 2024-08-17T18:11:23.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.423 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:11:23.423 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484583+0000", 2024-08-17T18:11:23.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.423 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:11:23.423 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484584+0000", 2024-08-17T18:11:23.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5700000000000002e-07 2024-08-17T18:11:23.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.424 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:11:23.424 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484604+0000", 2024-08-17T18:11:23.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9806999999999999e-05 2024-08-17T18:11:23.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.424 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:11:23.424 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484612+0000", 2024-08-17T18:11:23.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3820000000000004e-06 2024-08-17T18:11:23.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:11:23.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484618+0000", 2024-08-17T18:11:23.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8300000000000001e-06 2024-08-17T18:11:23.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.485895+0000", 2024-08-17T18:11:23.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012773629999999999 2024-08-17T18:11:23.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:11:23.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486015+0000", 2024-08-17T18:11:23.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119335 2024-08-17T18:11:23.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,6", 2024-08-17T18:11:23.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486131+0000", 2024-08-17T18:11:23.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011650999999999999 2024-08-17T18:11:23.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653049+0000", 2024-08-17T18:11:23.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.166917811 2024-08-17T18:11:23.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653095+0000", 2024-08-17T18:11:23.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6360999999999999e-05 2024-08-17T18:11:23.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:11:23.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.654443+0000", 2024-08-17T18:11:23.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0013480129999999999 2024-08-17T18:11:23.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:11:23.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.654495+0000", 2024-08-17T18:11:23.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2076000000000001e-05 2024-08-17T18:11:23.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:11:23.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.654548+0000", 2024-08-17T18:11:23.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2447000000000001e-05 2024-08-17T18:11:23.428 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.428 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:11:23.428 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.428 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:88746 56.c 56:31d7e917:::benchmark_data_smithi136_356456_object88745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1544)", 2024-08-17T18:11:23.428 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.486034+0000", 2024-08-17T18:11:23.429 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 51.830138208999998, 2024-08-17T18:11:23.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.176589883, 2024-08-17T18:11:23.429 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:11:23.429 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:11:23.429 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:11:23.429 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:11:23.429 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:11:23.429 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:11:23.429 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 88746 2024-08-17T18:11:23.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.429 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:11:23.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:11:23.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486034+0000", 2024-08-17T18:11:23.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:11:23.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486034+0000", 2024-08-17T18:11:23.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:11:23.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486035+0000", 2024-08-17T18:11:23.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9099999999999991e-07 2024-08-17T18:11:23.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:11:23.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486057+0000", 2024-08-17T18:11:23.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.143e-05 2024-08-17T18:11:23.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:11:23.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486067+0000", 2024-08-17T18:11:23.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0572e-05 2024-08-17T18:11:23.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:11:23.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486074+0000", 2024-08-17T18:11:23.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5780000000000004e-06 2024-08-17T18:11:23.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.432 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.432 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650549+0000", 2024-08-17T18:11:23.432 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.164474957 2024-08-17T18:11:23.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.432 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:11:23.432 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650720+0000", 2024-08-17T18:11:23.432 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017162199999999999 2024-08-17T18:11:23.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.432 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:11:23.432 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650849+0000", 2024-08-17T18:11:23.432 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012822300000000001 2024-08-17T18:11:23.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:11:23.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.662108+0000", 2024-08-17T18:11:23.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011259725 2024-08-17T18:11:23.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.662194+0000", 2024-08-17T18:11:23.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6069999999999994e-05 2024-08-17T18:11:23.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.662434+0000", 2024-08-17T18:11:23.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000240103 2024-08-17T18:11:23.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:11:23.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.662489+0000", 2024-08-17T18:11:23.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4880000000000003e-05 2024-08-17T18:11:23.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:11:23.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.662624+0000", 2024-08-17T18:11:23.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013473200000000001 2024-08-17T18:11:23.434 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.434 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:11:23.434 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.435 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:88747 56.c 56:3c73eda4:::benchmark_data_smithi136_356456_object88746:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1544)", 2024-08-17T18:11:23.435 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.486091+0000", 2024-08-17T18:11:23.435 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 51.830081604999997, 2024-08-17T18:11:23.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.180009683, 2024-08-17T18:11:23.435 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:11:23.435 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:11:23.435 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:11:23.435 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:11:23.435 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:11:23.435 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:11:23.435 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 88747 2024-08-17T18:11:23.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.435 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:11:23.435 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:11:23.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486091+0000", 2024-08-17T18:11:23.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:11:23.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486091+0000", 2024-08-17T18:11:23.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:11:23.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486091+0000", 2024-08-17T18:11:23.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2e-07 2024-08-17T18:11:23.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:11:23.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486110+0000", 2024-08-17T18:11:23.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8410000000000002e-05 2024-08-17T18:11:23.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:11:23.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486117+0000", 2024-08-17T18:11:23.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1049999999999997e-06 2024-08-17T18:11:23.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:11:23.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486122+0000", 2024-08-17T18:11:23.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8119999999999998e-06 2024-08-17T18:11:23.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.654390+0000", 2024-08-17T18:11:23.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16826818199999999 2024-08-17T18:11:23.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:11:23.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.655078+0000", 2024-08-17T18:11:23.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00068772199999999999 2024-08-17T18:11:23.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:11:23.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.655223+0000", 2024-08-17T18:11:23.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000145167 2024-08-17T18:11:23.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.662237+0000", 2024-08-17T18:11:23.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0070139419999999996 2024-08-17T18:11:23.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.666004+0000", 2024-08-17T18:11:23.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037672869999999998 2024-08-17T18:11:23.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:11:23.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.666014+0000", 2024-08-17T18:11:23.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0484e-05 2024-08-17T18:11:23.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:11:23.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.666062+0000", 2024-08-17T18:11:23.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7919e-05 2024-08-17T18:11:23.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:11:23.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.666100+0000", 2024-08-17T18:11:23.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8133000000000002e-05 2024-08-17T18:11:23.441 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.441 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:11:23.441 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.441 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.46025.0:88733 56.f e1544/1535 ondisk, result = 0)", 2024-08-17T18:11:23.441 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.486213+0000", 2024-08-17T18:11:23.441 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 51.82995974, 2024-08-17T18:11:23.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16423845200000001, 2024-08-17T18:11:23.441 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:11:23.441 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:11:23.441 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T18:11:23.442 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:11:23.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:11:23.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486213+0000", 2024-08-17T18:11:23.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:11:23.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486213+0000", 2024-08-17T18:11:23.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:11:23.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486213+0000", 2024-08-17T18:11:23.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1999999999999999e-07 2024-08-17T18:11:23.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:11:23.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486223+0000", 2024-08-17T18:11:23.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9890000000000004e-06 2024-08-17T18:11:23.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:11:23.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486234+0000", 2024-08-17T18:11:23.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0883999999999999e-05 2024-08-17T18:11:23.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:11:23.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486241+0000", 2024-08-17T18:11:23.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6780000000000003e-06 2024-08-17T18:11:23.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650315+0000", 2024-08-17T18:11:23.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16407380599999999 2024-08-17T18:11:23.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:11:23.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650324+0000", 2024-08-17T18:11:23.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0799999999999995e-06 2024-08-17T18:11:23.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:11:23.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650451+0000", 2024-08-17T18:11:23.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012729500000000001 2024-08-17T18:11:23.445 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.445 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:11:23.445 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.445 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:88748 56.a 56:5de94025:::benchmark_data_smithi136_356456_object88747:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1544)", 2024-08-17T18:11:23.445 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.486267+0000", 2024-08-17T18:11:23.445 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 51.829905089999997, 2024-08-17T18:11:23.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17806833399999999, 2024-08-17T18:11:23.446 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:11:23.446 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:11:23.446 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:11:23.446 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:11:23.446 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:11:23.446 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:11:23.446 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 88748 2024-08-17T18:11:23.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.446 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:11:23.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:11:23.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486267+0000", 2024-08-17T18:11:23.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:11:23.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486267+0000", 2024-08-17T18:11:23.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:11:23.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486268+0000", 2024-08-17T18:11:23.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2799999999999995e-07 2024-08-17T18:11:23.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:11:23.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486290+0000", 2024-08-17T18:11:23.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2306999999999998e-05 2024-08-17T18:11:23.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:11:23.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486299+0000", 2024-08-17T18:11:23.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0860000000000002e-06 2024-08-17T18:11:23.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:11:23.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486305+0000", 2024-08-17T18:11:23.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5189999999999998e-06 2024-08-17T18:11:23.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486361+0000", 2024-08-17T18:11:23.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5878999999999998e-05 2024-08-17T18:11:23.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:11:23.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486474+0000", 2024-08-17T18:11:23.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112804 2024-08-17T18:11:23.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,6", 2024-08-17T18:11:23.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650225+0000", 2024-08-17T18:11:23.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16375092599999999 2024-08-17T18:11:23.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.660909+0000", 2024-08-17T18:11:23.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010684206 2024-08-17T18:11:23.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:11:23.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.660952+0000", 2024-08-17T18:11:23.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3374999999999997e-05 2024-08-17T18:11:23.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:11:23.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.664236+0000", 2024-08-17T18:11:23.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032840529999999999 2024-08-17T18:11:23.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:11:23.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.664294+0000", 2024-08-17T18:11:23.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7621999999999998e-05 2024-08-17T18:11:23.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:11:23.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.664336+0000", 2024-08-17T18:11:23.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1729e-05 2024-08-17T18:11:23.452 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.452 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:11:23.452 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.453 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.46025.0:88735 56.c e1544/1535 ondisk, result = 0)", 2024-08-17T18:11:23.453 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.486671+0000", 2024-08-17T18:11:23.453 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 51.829501571999998, 2024-08-17T18:11:23.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.167366402, 2024-08-17T18:11:23.453 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:11:23.453 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:11:23.453 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T18:11:23.453 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:11:23.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:11:23.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486671+0000", 2024-08-17T18:11:23.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:11:23.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486671+0000", 2024-08-17T18:11:23.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:11:23.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486671+0000", 2024-08-17T18:11:23.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.13e-07 2024-08-17T18:11:23.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:11:23.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486675+0000", 2024-08-17T18:11:23.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5190000000000001e-06 2024-08-17T18:11:23.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:11:23.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650207+0000", 2024-08-17T18:11:23.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.163532289 2024-08-17T18:11:23.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:11:23.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650216+0000", 2024-08-17T18:11:23.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0389999999999993e-06 2024-08-17T18:11:23.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653909+0000", 2024-08-17T18:11:23.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036931970000000001 2024-08-17T18:11:23.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:11:23.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653917+0000", 2024-08-17T18:11:23.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4580000000000004e-06 2024-08-17T18:11:23.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:11:23.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.654037+0000", 2024-08-17T18:11:23.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120287 2024-08-17T18:11:23.457 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.457 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:11:23.457 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.457 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.46025.0:88741 56.e e1544/1535 ondisk, result = 0)", 2024-08-17T18:11:23.457 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.489475+0000", 2024-08-17T18:11:23.457 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 51.826696927, 2024-08-17T18:11:23.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.163581478, 2024-08-17T18:11:23.457 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:11:23.458 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:11:23.458 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T18:11:23.458 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:11:23.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:11:23.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.489475+0000", 2024-08-17T18:11:23.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:11:23.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.489475+0000", 2024-08-17T18:11:23.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:23.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:11:23.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.489477+0000", 2024-08-17T18:11:23.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.032e-06 2024-08-17T18:11:23.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:11:23.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.489482+0000", 2024-08-17T18:11:23.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4090000000000002e-06 2024-08-17T18:11:23.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:11:23.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650118+0000", 2024-08-17T18:11:23.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.160635842 2024-08-17T18:11:23.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:11:23.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650133+0000", 2024-08-17T18:11:23.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5534000000000001e-05 2024-08-17T18:11:23.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:23.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653033+0000", 2024-08-17T18:11:23.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0028997620000000002 2024-08-17T18:11:23.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:11:23.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653042+0000", 2024-08-17T18:11:23.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3810000000000005e-06 2024-08-17T18:11:23.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:23.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:23.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:11:23.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653057+0000", 2024-08-17T18:11:23.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4518000000000001e-05 2024-08-17T18:11:23.462 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.462 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:11:23.462 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.462 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:23.462 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:11:23.462 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:11:23.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:23.464 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:23.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:23.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:23.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:23.600 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:23.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:23.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:23 smithi136 ceph-mon[109861]: pgmap v2865: 99 pgs: 99 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.1 MiB/s wr, 920 op/s 2024-08-17T18:11:23.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:23 smithi136 ceph-mon[115078]: pgmap v2865: 99 pgs: 99 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.1 MiB/s wr, 920 op/s 2024-08-17T18:11:23.701 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:23.739 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:23.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:23.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:23 smithi165 ceph-mon[105517]: pgmap v2865: 99 pgs: 99 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.1 MiB/s wr, 920 op/s 2024-08-17T18:11:23.840 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:23.871 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:23.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:23.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:23.974 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:24.001 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:24.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:24.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:24.103 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:24.137 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:24.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:24.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:24.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:24.275 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:24.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:24.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:24.377 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:24.416 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:24.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:24.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:24.518 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:24.550 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:24.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:24.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:24.651 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:24.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:24.689 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:24.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:24.790 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:24.825 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:24.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:24.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:24.855 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:11:24.927 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:24.976 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:24.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:24.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:25.078 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:25.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:25.122 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:25.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:25.223 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:25.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:25.262 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:25.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:25.363 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:25.392 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:25.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:25.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:25.494 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:25.501 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-17T18:11:25.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:25.548 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:25.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:25.649 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:25.698 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:25.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:25.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:25.724 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:25 smithi136 ceph-mon[109861]: pgmap v2866: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 521 KiB/s wr, 990 op/s 2024-08-17T18:11:25.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:25 smithi136 ceph-mon[115078]: pgmap v2866: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 521 KiB/s wr, 990 op/s 2024-08-17T18:11:25.799 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:25.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:25.842 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:25.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:25.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:25.987 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:25.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:25 smithi165 ceph-mon[105517]: pgmap v2866: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 521 KiB/s wr, 990 op/s 2024-08-17T18:11:26.089 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:26.148 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:26.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:26.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:26.249 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:26.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:26.298 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:26.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:26.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:26.432 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:26.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:26.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:26.534 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:26.547 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:11:26.563 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:26.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:26.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:26.665 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:26.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:26.705 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:26.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:26.806 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:26.851 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:26.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:26.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:26.934 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:11:26.954 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:26.980 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:26.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:26.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:27.082 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:27.109 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:27.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:27.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:27.210 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:27.254 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:27.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:27.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:27.355 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:27.383 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:27.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:27.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:27.485 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:27.520 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:27.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:27.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:27.622 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:27.651 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:27.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:27.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:27.721 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:27 smithi136 ceph-mon[109861]: pgmap v2867: 99 pgs: 99 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 986 op/s 2024-08-17T18:11:27.722 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:27 smithi136 ceph-mon[115078]: pgmap v2867: 99 pgs: 99 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 986 op/s 2024-08-17T18:11:27.722 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:11:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:11:27] "GET /metrics HTTP/1.1" 200 36610 "" "Prometheus/2.51.0" 2024-08-17T18:11:27.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:27.793 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:27.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:27.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:27.803 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:11:27.895 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:27.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:27.938 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:27.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:28.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:27 smithi165 ceph-mon[105517]: pgmap v2867: 99 pgs: 99 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 986 op/s 2024-08-17T18:11:28.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:28.074 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:28.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:28.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:28.176 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:28.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:28.214 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:28.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:28.315 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:28.345 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:28.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:28.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:28.448 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:28.478 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:28.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:28.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:28.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:28.610 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:28.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:28.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:28.712 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:28.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:28.752 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:28.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:28.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:28.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:28.884 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:28.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:28.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:29.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:29.035 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:29.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:29.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:29.169 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:29.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:29.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:29.271 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:29.318 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:29.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:29.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:29.420 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:29.451 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:29.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:29.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:29.553 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:29.585 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:29.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:29.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:29.687 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:29.714 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:29.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:29.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:29.816 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:29.856 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:29.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:29.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:29.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:29 smithi136 ceph-mon[109861]: pgmap v2868: 99 pgs: 99 active+clean; 6.8 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 811 op/s 2024-08-17T18:11:29.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:29 smithi136 ceph-mon[115078]: pgmap v2868: 99 pgs: 99 active+clean; 6.8 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 811 op/s 2024-08-17T18:11:29.959 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:29.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:29.997 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:29.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:30.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:29 smithi165 ceph-mon[105517]: pgmap v2868: 99 pgs: 99 active+clean; 6.8 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 811 op/s 2024-08-17T18:11:30.098 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:30.128 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:30.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:30.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:30.229 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:30.280 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:30.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:30.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:30.382 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:30.421 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:30.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:30.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:30.523 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:30.526 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:11:30.527 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T18:11:30.555 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:30.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:30.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:30.656 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:30.684 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:30.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:30.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:30.786 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:30.823 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:30.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:30.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:30.926 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:30.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:30.965 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:30.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:31.066 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:31.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:31.118 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:31.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:31.219 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:31.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:31.265 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:31.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:31.366 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:31.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:31.399 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:31.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:31.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:31.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:31.535 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:31.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:31.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:31.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:31.667 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:31.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:31.768 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:31.804 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:31.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:31.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:31.906 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:31.953 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:31 smithi136 ceph-mon[115078]: pgmap v2869: 99 pgs: 99 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 888 op/s 2024-08-17T18:11:31.953 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:31 smithi136 ceph-mon[115078]: from='client.55531 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:11:31.954 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:31 smithi136 ceph-mon[109861]: pgmap v2869: 99 pgs: 99 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 888 op/s 2024-08-17T18:11:31.954 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:31 smithi136 ceph-mon[109861]: from='client.55531 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:11:31.969 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:31.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:31.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:32.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:31 smithi165 ceph-mon[105517]: pgmap v2869: 99 pgs: 99 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 888 op/s 2024-08-17T18:11:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:31 smithi165 ceph-mon[105517]: from='client.55531 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:11:32.072 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:32.104 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:32.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:32.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:32.206 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:32.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:32.266 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:32.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:32.367 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:32.405 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:32.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:32.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:32.506 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:32.552 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:32.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:32.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:32.654 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:32.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:32.685 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:32.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:32.786 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:32.812 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:32.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:32.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:32.868 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:11:32.868 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:11:32.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:32.868 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.46025.0:148830 56.d e1548/1535 56:b5766b65:::benchmark_data_smithi136_356456_object22126:head v 1548'9296, mlcod=1548'9294)", 2024-08-17T18:11:32.868 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:11:32.797522+0000", 2024-08-17T18:11:32.868 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.001646801, 2024-08-17T18:11:32.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016748220000000001, 2024-08-17T18:11:32.868 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:11:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:11:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T18:11:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:11:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:11:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:32.797522+0000", 2024-08-17T18:11:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:11:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:32.797522+0000", 2024-08-17T18:11:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:11:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:11:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:32.797523+0000", 2024-08-17T18:11:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.15e-06 2024-08-17T18:11:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:11:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:32.797533+0000", 2024-08-17T18:11:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4460000000000007e-06 2024-08-17T18:11:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:11:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:32.797556+0000", 2024-08-17T18:11:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2802999999999998e-05 2024-08-17T18:11:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:11:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:32.797564+0000", 2024-08-17T18:11:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8650000000000007e-06 2024-08-17T18:11:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:11:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:32.797653+0000", 2024-08-17T18:11:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8432999999999998e-05 2024-08-17T18:11:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:11:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:11:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:11:32.872 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:32.797674+0000", 2024-08-17T18:11:32.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1092999999999999e-05 2024-08-17T18:11:32.872 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:32.872 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:11:32.872 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:32.872 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:11:32.872 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-17T18:11:32.872 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 1 2024-08-17T18:11:32.872 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:11:32.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:32.956 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:32.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:32.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:33.058 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:33.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:33.111 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:33.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:33.212 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:33.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:33.255 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:33.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:33.356 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:33.395 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:33.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:33.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:33.497 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:33.531 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:33.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:33.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:33.633 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:33.671 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:33.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:33.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:33.773 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:33.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:33 smithi136 ceph-mon[109861]: pgmap v2870: 99 pgs: 99 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s 2024-08-17T18:11:33.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:33 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4194878891' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:11:33.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:33 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:11:33.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:33 smithi136 ceph-mon[115078]: pgmap v2870: 99 pgs: 99 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s 2024-08-17T18:11:33.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:33 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4194878891' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:11:33.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:33 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:11:33.808 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:33.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:33.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:33.910 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:33.942 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:33.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:33.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:33 smithi165 ceph-mon[105517]: pgmap v2870: 99 pgs: 99 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s 2024-08-17T18:11:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:33 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4194878891' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:11:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:33 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:11:34.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:34.084 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:34.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:34.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:34.185 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:34.232 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:34.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:34.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:34.239 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":2869,"stamp":"2024-08-17T18:11:30.343763+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7227054077,"num_objects":110294,"num_object_clones":0,"num_object_copies":330882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110294,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143200,"num_write_kb":8108898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":598400,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117734,"ondisk_log_size":117734,"up":297,"acting":297,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":297,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":26385328,"kb_used_data":21328504,"kb_used_omap":350,"kb_used_meta":5056353,"kb_avail":629830736,"statfs":{"total":671965249536,"available":644946673664,"internally_reserved":0,"allocated":21840388096,"data_stored":21801525083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":358888,"internal_metadata":5177706008},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,4,7,10,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-699596800,"num_objects":-10675,"num_object_clones":0,"num_object_copies":-32025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10675,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008796"},"pg_stats":[{"pgid":"56.f","version":"1548'8959","reported_seq":8996,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:25.631851+0000","last_change":"2024-08-17T18:09:28.696402+0000","last_active":"2024-08-17T18:11:25.631851+0000","last_peered":"2024-08-17T18:11:25.631851+0000","last_clean":"2024-08-17T18:11:25.631851+0000","last_became_active":"2024-08-17T18:09:28.696135+0000","last_became_peered":"2024-08-17T18:09:28.696135+0000","last_unstale":"2024-08-17T18:11:25.631851+0000","last_undegraded":"2024-08-17T18:11:25.631851+0000","last_fullsized":"2024-08-17T18:11:25.631851+0000","mapping_epoch":1535,"log_start":"1540'2200","ondisk_log_start":"1540'2200","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6759,"log_dups_size":0,"ondisk_log_size":6759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:44:29.924728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":465764352,"num_objects":7107,"num_object_clones":0,"num_object_copies":21321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8959,"num_write_kb":514112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.e","version":"1548'8804","reported_seq":8841,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:25.651068+0000","last_change":"2024-08-17T18:09:28.697584+0000","last_active":"2024-08-17T18:11:25.651068+0000","last_peered":"2024-08-17T18:11:25.651068+0000","last_clean":"2024-08-17T18:11:25.651068+0000","last_became_active":"2024-08-17T18:09:28.697211+0000","last_became_peered":"2024-08-17T18:09:28.697211+0000","last_unstale":"2024-08-17T18:11:25.651068+0000","last_undegraded":"2024-08-17T18:11:25.651068+0000","last_fullsized":"2024-08-17T18:11:25.651068+0000","mapping_epoch":1535,"log_start":"1540'2100","ondisk_log_start":"1540'2100","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6704,"log_dups_size":0,"ondisk_log_size":6704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:16:31.941416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459931648,"num_objects":7018,"num_object_clones":0,"num_object_copies":21054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8804,"num_write_kb":506304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.d","version":"1548'8878","reported_seq":8916,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:26.000236+0000","last_change":"2024-08-17T18:09:28.696225+0000","last_active":"2024-08-17T18:11:26.000236+0000","last_peered":"2024-08-17T18:11:26.000236+0000","last_clean":"2024-08-17T18:11:26.000236+0000","last_became_active":"2024-08-17T18:09:28.695916+0000","last_became_peered":"2024-08-17T18:09:28.695916+0000","last_unstale":"2024-08-17T18:11:26.000236+0000","last_undegraded":"2024-08-17T18:11:26.000236+0000","last_fullsized":"2024-08-17T18:11:26.000236+0000","mapping_epoch":1535,"log_start":"1539'1300","ondisk_log_start":"1539'1300","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":7578,"log_dups_size":0,"ondisk_log_size":7578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:00:20.159452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456065048,"num_objects":6960,"num_object_clones":0,"num_object_copies":20880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6960,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8878,"num_write_kb":506753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"56.c","version":"1548'8901","reported_seq":8937,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:25.686084+0000","last_change":"2024-08-17T18:09:28.627634+0000","last_active":"2024-08-17T18:11:25.686084+0000","last_peered":"2024-08-17T18:11:25.686084+0000","last_clean":"2024-08-17T18:11:25.686084+0000","last_became_active":"2024-08-17T18:09:28.627138+0000","last_became_peered":"2024-08-17T18:09:28.627138+0000","last_unstale":"2024-08-17T18:11:25.686084+0000","last_undegraded":"2024-08-17T18:11:25.686084+0000","last_fullsized":"2024-08-17T18:11:25.686084+0000","mapping_epoch":1535,"log_start":"1540'2200","ondisk_log_start":"1540'2200","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6701,"log_dups_size":0,"ondisk_log_size":6701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:47:56.574797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459341824,"num_objects":7009,"num_object_clones":0,"num_object_copies":21027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8901,"num_write_kb":509120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.b","version":"1548'8962","reported_seq":8984,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:27.871605+0000","last_change":"2024-08-17T18:09:28.627309+0000","last_active":"2024-08-17T18:11:27.871605+0000","last_peered":"2024-08-17T18:11:27.871605+0000","last_clean":"2024-08-17T18:11:27.871605+0000","last_became_active":"2024-08-17T18:09:28.626970+0000","last_became_peered":"2024-08-17T18:09:28.626970+0000","last_unstale":"2024-08-17T18:11:27.871605+0000","last_undegraded":"2024-08-17T18:11:27.871605+0000","last_fullsized":"2024-08-17T18:11:27.871605+0000","mapping_epoch":1535,"log_start":"1540'2100","ondisk_log_start":"1540'2100","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6862,"log_dups_size":0,"ondisk_log_size":6862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:47:24.480441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445906944,"num_objects":6804,"num_object_clones":0,"num_object_copies":20412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8962,"num_write_kb":504512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"56.a","version":"1548'8828","reported_seq":8865,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:25.673832+0000","last_change":"2024-08-17T18:09:28.697301+0000","last_active":"2024-08-17T18:11:25.673832+0000","last_peered":"2024-08-17T18:11:25.673832+0000","last_clean":"2024-08-17T18:11:25.673832+0000","last_became_active":"2024-08-17T18:09:28.697019+0000","last_became_peered":"2024-08-17T18:09:28.697019+0000","last_unstale":"2024-08-17T18:11:25.673832+0000","last_undegraded":"2024-08-17T18:11:25.673832+0000","last_fullsized":"2024-08-17T18:11:25.673832+0000","mapping_epoch":1535,"log_start":"1540'2100","ondisk_log_start":"1540'2100","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6728,"log_dups_size":0,"ondisk_log_size":6728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:44:37.273534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453902336,"num_objects":6926,"num_object_clones":0,"num_object_copies":20778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8828,"num_write_kb":504128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.9","version":"1548'8790","reported_seq":8808,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:29.198978+0000","last_change":"2024-08-17T18:09:28.626545+0000","last_active":"2024-08-17T18:11:29.198978+0000","last_peered":"2024-08-17T18:11:29.198978+0000","last_clean":"2024-08-17T18:11:29.198978+0000","last_became_active":"2024-08-17T18:09:28.626300+0000","last_became_peered":"2024-08-17T18:09:28.626300+0000","last_unstale":"2024-08-17T18:11:29.198978+0000","last_undegraded":"2024-08-17T18:11:29.198978+0000","last_fullsized":"2024-08-17T18:11:29.198978+0000","mapping_epoch":1535,"log_start":"1539'800","ondisk_log_start":"1539'800","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":7990,"log_dups_size":0,"ondisk_log_size":7990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:34:50.839206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432275456,"num_objects":6596,"num_object_clones":0,"num_object_copies":19788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8790,"num_write_kb":492352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.8","version":"1548'8874","reported_seq":8911,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:26.006220+0000","last_change":"2024-08-17T18:09:28.697184+0000","last_active":"2024-08-17T18:11:26.006220+0000","last_peered":"2024-08-17T18:11:26.006220+0000","last_clean":"2024-08-17T18:11:26.006220+0000","last_became_active":"2024-08-17T18:09:28.696911+0000","last_became_peered":"2024-08-17T18:09:28.696911+0000","last_unstale":"2024-08-17T18:11:26.006220+0000","last_undegraded":"2024-08-17T18:11:26.006220+0000","last_fullsized":"2024-08-17T18:11:26.006220+0000","mapping_epoch":1535,"log_start":"1539'1300","ondisk_log_start":"1539'1300","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":7574,"log_dups_size":0,"ondisk_log_size":7574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:39:59.778610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453640192,"num_objects":6922,"num_object_clones":0,"num_object_copies":20766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8874,"num_write_kb":505472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"56.7","version":"1548'8813","reported_seq":8850,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:26.015311+0000","last_change":"2024-08-17T18:09:28.628236+0000","last_active":"2024-08-17T18:11:26.015311+0000","last_peered":"2024-08-17T18:11:26.015311+0000","last_clean":"2024-08-17T18:11:26.015311+0000","last_became_active":"2024-08-17T18:09:28.627952+0000","last_became_peered":"2024-08-17T18:09:28.627952+0000","last_unstale":"2024-08-17T18:11:26.015311+0000","last_undegraded":"2024-08-17T18:11:26.015311+0000","last_fullsized":"2024-08-17T18:11:26.015311+0000","mapping_epoch":1535,"log_start":"1539'1300","ondisk_log_start":"1539'1300","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":7513,"log_dups_size":0,"ondisk_log_size":7513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:04:20.822971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451608576,"num_objects":6891,"num_object_clones":0,"num_object_copies":20673,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8813,"num_write_kb":502528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"56.6","version":"1548'9177","reported_seq":9199,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:29.176030+0000","last_change":"2024-08-17T18:09:28.629945+0000","last_active":"2024-08-17T18:11:29.176030+0000","last_peered":"2024-08-17T18:11:29.176030+0000","last_clean":"2024-08-17T18:11:29.176030+0000","last_became_active":"2024-08-17T18:09:28.629553+0000","last_became_peered":"2024-08-17T18:09:28.629553+0000","last_unstale":"2024-08-17T18:11:29.176030+0000","last_undegraded":"2024-08-17T18:11:29.176030+0000","last_fullsized":"2024-08-17T18:11:29.176030+0000","mapping_epoch":1535,"log_start":"1539'1200","ondisk_log_start":"1539'1200","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":7977,"log_dups_size":0,"ondisk_log_size":7977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:00:36.875044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448724992,"num_objects":6847,"num_object_clones":0,"num_object_copies":20541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9177,"num_write_kb":512768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.5","version":"1548'8946","reported_seq":8983,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:26.912908+0000","last_change":"2024-08-17T18:09:28.697212+0000","last_active":"2024-08-17T18:11:26.912908+0000","last_peered":"2024-08-17T18:11:26.912908+0000","last_clean":"2024-08-17T18:11:26.912908+0000","last_became_active":"2024-08-17T18:09:28.696769+0000","last_became_peered":"2024-08-17T18:09:28.696769+0000","last_unstale":"2024-08-17T18:11:26.912908+0000","last_undegraded":"2024-08-17T18:11:26.912908+0000","last_fullsized":"2024-08-17T18:11:26.912908+0000","mapping_epoch":1535,"log_start":"1540'2400","ondisk_log_start":"1540'2400","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6546,"log_dups_size":0,"ondisk_log_size":6546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:30:22.827483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454033408,"num_objects":6928,"num_object_clones":0,"num_object_copies":20784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8946,"num_write_kb":507968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.4","version":"1548'8871","reported_seq":8908,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:25.637475+0000","last_change":"2024-08-17T18:09:28.627340+0000","last_active":"2024-08-17T18:11:25.637475+0000","last_peered":"2024-08-17T18:11:25.637475+0000","last_clean":"2024-08-17T18:11:25.637475+0000","last_became_active":"2024-08-17T18:09:28.626966+0000","last_became_peered":"2024-08-17T18:09:28.626966+0000","last_unstale":"2024-08-17T18:11:25.637475+0000","last_undegraded":"2024-08-17T18:11:25.637475+0000","last_fullsized":"2024-08-17T18:11:25.637475+0000","mapping_epoch":1535,"log_start":"1540'2200","ondisk_log_start":"1540'2200","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6671,"log_dups_size":0,"ondisk_log_size":6671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:18:20.264361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456982528,"num_objects":6973,"num_object_clones":0,"num_object_copies":20919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8871,"num_write_kb":507008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.3","version":"1548'8780","reported_seq":8817,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:25.615124+0000","last_change":"2024-08-17T18:09:28.697359+0000","last_active":"2024-08-17T18:11:25.615124+0000","last_peered":"2024-08-17T18:11:25.615124+0000","last_clean":"2024-08-17T18:11:25.615124+0000","last_became_active":"2024-08-17T18:09:28.696943+0000","last_became_peered":"2024-08-17T18:09:28.696943+0000","last_unstale":"2024-08-17T18:11:25.615124+0000","last_undegraded":"2024-08-17T18:11:25.615124+0000","last_fullsized":"2024-08-17T18:11:25.615124+0000","mapping_epoch":1535,"log_start":"1540'2100","ondisk_log_start":"1540'2100","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6680,"log_dups_size":0,"ondisk_log_size":6680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:08:17.415390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453115904,"num_objects":6914,"num_object_clones":0,"num_object_copies":20742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8780,"num_write_kb":502208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.2","version":"1548'9214","reported_seq":9236,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:29.192236+0000","last_change":"2024-08-17T18:09:28.628711+0000","last_active":"2024-08-17T18:11:29.192236+0000","last_peered":"2024-08-17T18:11:29.192236+0000","last_clean":"2024-08-17T18:11:29.192236+0000","last_became_active":"2024-08-17T18:09:28.628387+0000","last_became_peered":"2024-08-17T18:09:28.628387+0000","last_unstale":"2024-08-17T18:11:29.192236+0000","last_undegraded":"2024-08-17T18:11:29.192236+0000","last_fullsized":"2024-08-17T18:11:29.192236+0000","mapping_epoch":1535,"log_start":"1539'1300","ondisk_log_start":"1539'1300","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":7914,"log_dups_size":0,"ondisk_log_size":7914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:26:06.412021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445644800,"num_objects":6800,"num_object_clones":0,"num_object_copies":20400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9214,"num_write_kb":512448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.1","version":"1548'9331","reported_seq":9353,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:29.135842+0000","last_change":"2024-08-17T18:09:28.629660+0000","last_active":"2024-08-17T18:11:29.135842+0000","last_peered":"2024-08-17T18:11:29.135842+0000","last_clean":"2024-08-17T18:11:29.135842+0000","last_became_active":"2024-08-17T18:09:28.629359+0000","last_became_peered":"2024-08-17T18:09:28.629359+0000","last_unstale":"2024-08-17T18:11:29.135842+0000","last_undegraded":"2024-08-17T18:11:29.135842+0000","last_fullsized":"2024-08-17T18:11:29.135842+0000","mapping_epoch":1535,"log_start":"1539'1400","ondisk_log_start":"1539'1400","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":7931,"log_dups_size":0,"ondisk_log_size":7931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:05:58.848611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446627840,"num_objects":6815,"num_object_clones":0,"num_object_copies":20445,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9331,"num_write_kb":516672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.0","version":"1548'9008","reported_seq":9030,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:29.188741+0000","last_change":"2024-08-17T18:09:28.697077+0000","last_active":"2024-08-17T18:11:29.188741+0000","last_peered":"2024-08-17T18:11:29.188741+0000","last_clean":"2024-08-17T18:11:29.188741+0000","last_became_active":"2024-08-17T18:09:28.696676+0000","last_became_peered":"2024-08-17T18:09:28.696676+0000","last_unstale":"2024-08-17T18:11:29.188741+0000","last_undegraded":"2024-08-17T18:11:29.188741+0000","last_fullsized":"2024-08-17T18:11:29.188741+0000","mapping_epoch":1535,"log_start":"1539'1100","ondisk_log_start":"1539'1100","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":7908,"log_dups_size":0,"ondisk_log_size":7908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:09.350570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442892288,"num_objects":6758,"num_object_clones":0,"num_object_copies":20274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9008,"num_write_kb":504512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":240,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.539924+0000","last_change":"2024-08-17T18:07:48.546202+0000","last_active":"2024-08-17T18:11:13.539924+0000","last_peered":"2024-08-17T18:11:13.539924+0000","last_clean":"2024-08-17T18:11:13.539924+0000","last_became_active":"2024-08-17T18:07:48.531246+0000","last_became_peered":"2024-08-17T18:07:48.531246+0000","last_unstale":"2024-08-17T18:11:13.539924+0000","last_undegraded":"2024-08-17T18:11:13.539924+0000","last_fullsized":"2024-08-17T18:11:13.539924+0000","mapping_epoch":1501,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:30.944362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1552,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.528727+0000","last_change":"2024-08-17T18:08:37.068490+0000","last_active":"2024-08-17T18:11:10.528727+0000","last_peered":"2024-08-17T18:11:10.528727+0000","last_clean":"2024-08-17T18:11:10.528727+0000","last_became_active":"2024-08-17T18:08:37.068026+0000","last_became_peered":"2024-08-17T18:08:37.068026+0000","last_unstale":"2024-08-17T18:11:10.528727+0000","last_undegraded":"2024-08-17T18:11:10.528727+0000","last_fullsized":"2024-08-17T18:11:10.528727+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:34:27.467963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":2416,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.535817+0000","last_change":"2024-08-17T18:08:37.082356+0000","last_active":"2024-08-17T18:11:13.535817+0000","last_peered":"2024-08-17T18:11:13.535817+0000","last_clean":"2024-08-17T18:11:13.535817+0000","last_became_active":"2024-08-17T18:08:37.082046+0000","last_became_peered":"2024-08-17T18:08:37.082046+0000","last_unstale":"2024-08-17T18:11:13.535817+0000","last_undegraded":"2024-08-17T18:11:13.535817+0000","last_fullsized":"2024-08-17T18:11:13.535817+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:46:13.932917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2438,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.540635+0000","last_change":"2024-08-17T18:04:14.540774+0000","last_active":"2024-08-17T18:11:13.540635+0000","last_peered":"2024-08-17T18:11:13.540635+0000","last_clean":"2024-08-17T18:11:13.540635+0000","last_became_active":"2024-08-17T18:04:14.539778+0000","last_became_peered":"2024-08-17T18:04:14.539778+0000","last_unstale":"2024-08-17T18:11:13.540635+0000","last_undegraded":"2024-08-17T18:11:13.540635+0000","last_fullsized":"2024-08-17T18:11:13.540635+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:55:02.863360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":100,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.531326+0000","last_change":"2024-08-17T18:04:43.777705+0000","last_active":"2024-08-17T18:11:10.531326+0000","last_peered":"2024-08-17T18:11:10.531326+0000","last_clean":"2024-08-17T18:11:10.531326+0000","last_became_active":"2024-08-17T18:04:43.771283+0000","last_became_peered":"2024-08-17T18:04:43.771283+0000","last_unstale":"2024-08-17T18:11:10.531326+0000","last_undegraded":"2024-08-17T18:11:10.531326+0000","last_fullsized":"2024-08-17T18:11:10.531326+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:13.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1682,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.535563+0000","last_change":"2024-08-17T18:04:14.534963+0000","last_active":"2024-08-17T18:11:13.535563+0000","last_peered":"2024-08-17T18:11:13.535563+0000","last_clean":"2024-08-17T18:11:13.535563+0000","last_became_active":"2024-08-17T18:04:14.534634+0000","last_became_peered":"2024-08-17T18:04:14.534634+0000","last_unstale":"2024-08-17T18:11:13.535563+0000","last_undegraded":"2024-08-17T18:11:13.535563+0000","last_fullsized":"2024-08-17T18:11:13.535563+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:33:27.287860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":100,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.531468+0000","last_change":"2024-08-17T18:04:43.773534+0000","last_active":"2024-08-17T18:11:10.531468+0000","last_peered":"2024-08-17T18:11:10.531468+0000","last_clean":"2024-08-17T18:11:10.531468+0000","last_became_active":"2024-08-17T18:04:43.768507+0000","last_became_peered":"2024-08-17T18:04:43.768507+0000","last_unstale":"2024-08-17T18:11:10.531468+0000","last_undegraded":"2024-08-17T18:11:10.531468+0000","last_fullsized":"2024-08-17T18:11:10.531468+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:50:57.647351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1539,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.529123+0000","last_change":"2024-08-17T17:24:17.458168+0000","last_active":"2024-08-17T18:11:10.529123+0000","last_peered":"2024-08-17T18:11:10.529123+0000","last_clean":"2024-08-17T18:11:10.529123+0000","last_became_active":"2024-08-17T17:24:17.457847+0000","last_became_peered":"2024-08-17T17:24:17.457847+0000","last_unstale":"2024-08-17T18:11:10.529123+0000","last_undegraded":"2024-08-17T18:11:10.529123+0000","last_fullsized":"2024-08-17T18:11:10.529123+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:28.717735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":194,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.539452+0000","last_change":"2024-08-17T18:04:14.531445+0000","last_active":"2024-08-17T18:11:13.539452+0000","last_peered":"2024-08-17T18:11:13.539452+0000","last_clean":"2024-08-17T18:11:13.539452+0000","last_became_active":"2024-08-17T18:04:14.531099+0000","last_became_peered":"2024-08-17T18:04:14.531099+0000","last_unstale":"2024-08-17T18:11:13.539452+0000","last_undegraded":"2024-08-17T18:11:13.539452+0000","last_fullsized":"2024-08-17T18:11:13.539452+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:43.335123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":105,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.538906+0000","last_change":"2024-08-17T18:07:48.548262+0000","last_active":"2024-08-17T18:11:13.538906+0000","last_peered":"2024-08-17T18:11:13.538906+0000","last_clean":"2024-08-17T18:11:13.538906+0000","last_became_active":"2024-08-17T18:07:48.547846+0000","last_became_peered":"2024-08-17T18:07:48.547846+0000","last_unstale":"2024-08-17T18:11:13.538906+0000","last_undegraded":"2024-08-17T18:11:13.538906+0000","last_fullsized":"2024-08-17T18:11:13.538906+0000","mapping_epoch":1501,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:24:37.513219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1556,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.530804+0000","last_change":"2024-08-17T18:01:23.519890+0000","last_active":"2024-08-17T18:11:10.530804+0000","last_peered":"2024-08-17T18:11:10.530804+0000","last_clean":"2024-08-17T18:11:10.530804+0000","last_became_active":"2024-08-17T18:01:23.519526+0000","last_became_peered":"2024-08-17T18:01:23.519526+0000","last_unstale":"2024-08-17T18:11:10.530804+0000","last_undegraded":"2024-08-17T18:11:10.530804+0000","last_fullsized":"2024-08-17T18:11:10.530804+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:04:52.662528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2403,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.537542+0000","last_change":"2024-08-17T18:01:23.491836+0000","last_active":"2024-08-17T18:11:13.537542+0000","last_peered":"2024-08-17T18:11:13.537542+0000","last_clean":"2024-08-17T18:11:13.537542+0000","last_became_active":"2024-08-17T18:01:23.491209+0000","last_became_peered":"2024-08-17T18:01:23.491209+0000","last_unstale":"2024-08-17T18:11:13.537542+0000","last_undegraded":"2024-08-17T18:11:13.537542+0000","last_fullsized":"2024-08-17T18:11:13.537542+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:35:30.767005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2405,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.551862+0000","last_change":"2024-08-17T17:24:17.452025+0000","last_active":"2024-08-17T18:11:13.551862+0000","last_peered":"2024-08-17T18:11:13.551862+0000","last_clean":"2024-08-17T18:11:13.551862+0000","last_became_active":"2024-08-17T17:24:17.448029+0000","last_became_peered":"2024-08-17T17:24:17.448029+0000","last_unstale":"2024-08-17T18:11:13.551862+0000","last_undegraded":"2024-08-17T18:11:13.551862+0000","last_fullsized":"2024-08-17T18:11:13.551862+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:45.089346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":209,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.539741+0000","last_change":"2024-08-17T18:08:11.358013+0000","last_active":"2024-08-17T18:11:13.539741+0000","last_peered":"2024-08-17T18:11:13.539741+0000","last_clean":"2024-08-17T18:11:13.539741+0000","last_became_active":"2024-08-17T18:08:11.357666+0000","last_became_peered":"2024-08-17T18:08:11.357666+0000","last_unstale":"2024-08-17T18:11:13.539741+0000","last_undegraded":"2024-08-17T18:11:13.539741+0000","last_fullsized":"2024-08-17T18:11:13.539741+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:05:25.290465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2418,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.544241+0000","last_change":"2024-08-17T18:04:14.539660+0000","last_active":"2024-08-17T18:11:13.544241+0000","last_peered":"2024-08-17T18:11:13.544241+0000","last_clean":"2024-08-17T18:11:13.544241+0000","last_became_active":"2024-08-17T18:04:14.537852+0000","last_became_peered":"2024-08-17T18:04:14.537852+0000","last_unstale":"2024-08-17T18:11:13.544241+0000","last_undegraded":"2024-08-17T18:11:13.544241+0000","last_fullsized":"2024-08-17T18:11:13.544241+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:47:45.026997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1553,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.521395+0000","last_change":"2024-08-17T18:04:43.769892+0000","last_active":"2024-08-17T18:11:10.521395+0000","last_peered":"2024-08-17T18:11:10.521395+0000","last_clean":"2024-08-17T18:11:10.521395+0000","last_became_active":"2024-08-17T18:04:43.769702+0000","last_became_peered":"2024-08-17T18:04:43.769702+0000","last_unstale":"2024-08-17T18:11:10.521395+0000","last_undegraded":"2024-08-17T18:11:10.521395+0000","last_fullsized":"2024-08-17T18:11:10.521395+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:39.373904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2378,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.544337+0000","last_change":"2024-08-17T17:24:17.442338+0000","last_active":"2024-08-17T18:11:13.544337+0000","last_peered":"2024-08-17T18:11:13.544337+0000","last_clean":"2024-08-17T18:11:13.544337+0000","last_became_active":"2024-08-17T17:24:17.441948+0000","last_became_peered":"2024-08-17T17:24:17.441948+0000","last_unstale":"2024-08-17T18:11:13.544337+0000","last_undegraded":"2024-08-17T18:11:13.544337+0000","last_fullsized":"2024-08-17T18:11:13.544337+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:17.891492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":201,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.540017+0000","last_change":"2024-08-17T18:04:43.777736+0000","last_active":"2024-08-17T18:11:13.540017+0000","last_peered":"2024-08-17T18:11:13.540017+0000","last_clean":"2024-08-17T18:11:13.540017+0000","last_became_active":"2024-08-17T18:04:43.777438+0000","last_became_peered":"2024-08-17T18:04:43.777438+0000","last_unstale":"2024-08-17T18:11:13.540017+0000","last_undegraded":"2024-08-17T18:11:13.540017+0000","last_fullsized":"2024-08-17T18:11:13.540017+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:24:01.255927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2358,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.540769+0000","last_change":"2024-08-17T18:04:43.915398+0000","last_active":"2024-08-17T18:11:13.540769+0000","last_peered":"2024-08-17T18:11:13.540769+0000","last_clean":"2024-08-17T18:11:13.540769+0000","last_became_active":"2024-08-17T18:04:43.915212+0000","last_became_peered":"2024-08-17T18:04:43.915212+0000","last_unstale":"2024-08-17T18:11:13.540769+0000","last_undegraded":"2024-08-17T18:11:13.540769+0000","last_fullsized":"2024-08-17T18:11:13.540769+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:43:26.901830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":113,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.531414+0000","last_change":"2024-08-17T18:08:11.356876+0000","last_active":"2024-08-17T18:11:10.531414+0000","last_peered":"2024-08-17T18:11:10.531414+0000","last_clean":"2024-08-17T18:11:10.531414+0000","last_became_active":"2024-08-17T18:08:11.355662+0000","last_became_peered":"2024-08-17T18:08:11.355662+0000","last_unstale":"2024-08-17T18:11:10.531414+0000","last_undegraded":"2024-08-17T18:11:10.531414+0000","last_fullsized":"2024-08-17T18:11:10.531414+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:55:00.801251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":515,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.537624+0000","last_change":"2024-08-17T18:04:14.555729+0000","last_active":"2024-08-17T18:11:13.537624+0000","last_peered":"2024-08-17T18:11:13.537624+0000","last_clean":"2024-08-17T18:11:13.537624+0000","last_became_active":"2024-08-17T18:04:14.536678+0000","last_became_peered":"2024-08-17T18:04:14.536678+0000","last_unstale":"2024-08-17T18:11:13.537624+0000","last_undegraded":"2024-08-17T18:11:13.537624+0000","last_fullsized":"2024-08-17T18:11:13.537624+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:47:29.424778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2430,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.553154+0000","last_change":"2024-08-17T18:08:37.079571+0000","last_active":"2024-08-17T18:11:13.553154+0000","last_peered":"2024-08-17T18:11:13.553154+0000","last_clean":"2024-08-17T18:11:13.553154+0000","last_became_active":"2024-08-17T18:08:37.079224+0000","last_became_peered":"2024-08-17T18:08:37.079224+0000","last_unstale":"2024-08-17T18:11:13.553154+0000","last_undegraded":"2024-08-17T18:11:13.553154+0000","last_fullsized":"2024-08-17T18:11:13.553154+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:28:50.561072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1554,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.529628+0000","last_change":"2024-08-17T18:03:51.292044+0000","last_active":"2024-08-17T18:11:10.529628+0000","last_peered":"2024-08-17T18:11:10.529628+0000","last_clean":"2024-08-17T18:11:10.529628+0000","last_became_active":"2024-08-17T18:03:51.268271+0000","last_became_peered":"2024-08-17T18:03:51.268271+0000","last_unstale":"2024-08-17T18:11:10.529628+0000","last_undegraded":"2024-08-17T18:11:10.529628+0000","last_fullsized":"2024-08-17T18:11:10.529628+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:00.892410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4756,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1526,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.529195+0000","last_change":"2024-08-17T17:23:07.455919+0000","last_active":"2024-08-17T18:11:10.529195+0000","last_peered":"2024-08-17T18:11:10.529195+0000","last_clean":"2024-08-17T18:11:10.529195+0000","last_became_active":"2024-08-17T17:23:07.455284+0000","last_became_peered":"2024-08-17T17:23:07.455284+0000","last_unstale":"2024-08-17T18:11:10.529195+0000","last_undegraded":"2024-08-17T18:11:10.529195+0000","last_fullsized":"2024-08-17T18:11:10.529195+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:37:46.260775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"347'149","reported_seq":839,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.535704+0000","last_change":"2024-08-17T18:09:15.592511+0000","last_active":"2024-08-17T18:11:13.535704+0000","last_peered":"2024-08-17T18:11:13.535704+0000","last_clean":"2024-08-17T18:11:13.535704+0000","last_became_active":"2024-08-17T18:09:15.592099+0000","last_became_peered":"2024-08-17T18:09:15.592099+0000","last_unstale":"2024-08-17T18:11:13.535704+0000","last_undegraded":"2024-08-17T18:11:13.535704+0000","last_fullsized":"2024-08-17T18:11:13.535704+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:57.271286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:57.271286+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:35:02.167640+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1550,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.529040+0000","last_change":"2024-08-17T18:04:43.766724+0000","last_active":"2024-08-17T18:11:10.529040+0000","last_peered":"2024-08-17T18:11:10.529040+0000","last_clean":"2024-08-17T18:11:10.529040+0000","last_became_active":"2024-08-17T18:04:43.766428+0000","last_became_peered":"2024-08-17T18:04:43.766428+0000","last_unstale":"2024-08-17T18:11:10.529040+0000","last_undegraded":"2024-08-17T18:11:10.529040+0000","last_fullsized":"2024-08-17T18:11:10.529040+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:09:49.102940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1562,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.521509+0000","last_change":"2024-08-17T18:03:51.279963+0000","last_active":"2024-08-17T18:11:10.521509+0000","last_peered":"2024-08-17T18:11:10.521509+0000","last_clean":"2024-08-17T18:11:10.521509+0000","last_became_active":"2024-08-17T18:03:51.279506+0000","last_became_peered":"2024-08-17T18:03:51.279506+0000","last_unstale":"2024-08-17T18:11:10.521509+0000","last_undegraded":"2024-08-17T18:11:10.521509+0000","last_fullsized":"2024-08-17T18:11:10.521509+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:20:07.304042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":199,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.539825+0000","last_change":"2024-08-17T18:04:14.542589+0000","last_active":"2024-08-17T18:11:13.539825+0000","last_peered":"2024-08-17T18:11:13.539825+0000","last_clean":"2024-08-17T18:11:13.539825+0000","last_became_active":"2024-08-17T18:04:14.531540+0000","last_became_peered":"2024-08-17T18:04:14.531540+0000","last_unstale":"2024-08-17T18:11:13.539825+0000","last_undegraded":"2024-08-17T18:11:13.539825+0000","last_fullsized":"2024-08-17T18:11:13.539825+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:15:33.014782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"347'149","reported_seq":820,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.537128+0000","last_change":"2024-08-17T18:09:15.592395+0000","last_active":"2024-08-17T18:11:13.537128+0000","last_peered":"2024-08-17T18:11:13.537128+0000","last_clean":"2024-08-17T18:11:13.537128+0000","last_became_active":"2024-08-17T18:09:15.591773+0000","last_became_peered":"2024-08-17T18:09:15.591773+0000","last_unstale":"2024-08-17T18:11:13.537128+0000","last_undegraded":"2024-08-17T18:11:13.537128+0000","last_fullsized":"2024-08-17T18:11:13.537128+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:53.309517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:53.309517+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:50:57.110936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":91,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.544332+0000","last_change":"2024-08-17T18:08:11.349400+0000","last_active":"2024-08-17T18:11:13.544332+0000","last_peered":"2024-08-17T18:11:13.544332+0000","last_clean":"2024-08-17T18:11:13.544332+0000","last_became_active":"2024-08-17T18:08:11.348892+0000","last_became_peered":"2024-08-17T18:08:11.348892+0000","last_unstale":"2024-08-17T18:11:13.544332+0000","last_undegraded":"2024-08-17T18:11:13.544332+0000","last_fullsized":"2024-08-17T18:11:13.544332+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:52:54.819838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2388,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.552639+0000","last_change":"2024-08-17T17:23:07.428296+0000","last_active":"2024-08-17T18:11:13.552639+0000","last_peered":"2024-08-17T18:11:13.552639+0000","last_clean":"2024-08-17T18:11:13.552639+0000","last_became_active":"2024-08-17T17:23:07.428010+0000","last_became_peered":"2024-08-17T17:23:07.428010+0000","last_unstale":"2024-08-17T18:11:13.552639+0000","last_undegraded":"2024-08-17T18:11:13.552639+0000","last_fullsized":"2024-08-17T18:11:13.552639+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:45.459212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":2429,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.536285+0000","last_change":"2024-08-17T18:04:43.800633+0000","last_active":"2024-08-17T18:11:13.536285+0000","last_peered":"2024-08-17T18:11:13.536285+0000","last_clean":"2024-08-17T18:11:13.536285+0000","last_became_active":"2024-08-17T18:04:43.784612+0000","last_became_peered":"2024-08-17T18:04:43.784612+0000","last_unstale":"2024-08-17T18:11:13.536285+0000","last_undegraded":"2024-08-17T18:11:13.536285+0000","last_fullsized":"2024-08-17T18:11:13.536285+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:11.405039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"347'149","reported_seq":803,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.534423+0000","last_change":"2024-08-17T18:09:52.474670+0000","last_active":"2024-08-17T18:11:13.534423+0000","last_peered":"2024-08-17T18:11:13.534423+0000","last_clean":"2024-08-17T18:11:13.534423+0000","last_became_active":"2024-08-17T18:09:15.595011+0000","last_became_peered":"2024-08-17T18:09:15.595011+0000","last_unstale":"2024-08-17T18:11:13.534423+0000","last_undegraded":"2024-08-17T18:11:13.534423+0000","last_fullsized":"2024-08-17T18:11:13.534423+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:52.474544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:52.474544+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:38:21.347345+0000","scrub_duration":0.001886326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":202,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.539646+0000","last_change":"2024-08-17T18:04:43.805055+0000","last_active":"2024-08-17T18:11:13.539646+0000","last_peered":"2024-08-17T18:11:13.539646+0000","last_clean":"2024-08-17T18:11:13.539646+0000","last_became_active":"2024-08-17T18:04:43.804708+0000","last_became_peered":"2024-08-17T18:04:43.804708+0000","last_unstale":"2024-08-17T18:11:13.539646+0000","last_undegraded":"2024-08-17T18:11:13.539646+0000","last_fullsized":"2024-08-17T18:11:13.539646+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:47:22.253073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":195,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.553671+0000","last_change":"2024-08-17T18:04:14.529605+0000","last_active":"2024-08-17T18:11:13.553671+0000","last_peered":"2024-08-17T18:11:13.553671+0000","last_clean":"2024-08-17T18:11:13.553671+0000","last_became_active":"2024-08-17T18:04:14.529024+0000","last_became_peered":"2024-08-17T18:04:14.529024+0000","last_unstale":"2024-08-17T18:11:13.553671+0000","last_undegraded":"2024-08-17T18:11:13.553671+0000","last_fullsized":"2024-08-17T18:11:13.553671+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:38:00.485543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2442,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.566899+0000","last_change":"2024-08-17T18:04:43.773667+0000","last_active":"2024-08-17T18:11:13.566899+0000","last_peered":"2024-08-17T18:11:13.566899+0000","last_clean":"2024-08-17T18:11:13.566899+0000","last_became_active":"2024-08-17T18:04:43.773247+0000","last_became_peered":"2024-08-17T18:04:43.773247+0000","last_unstale":"2024-08-17T18:11:13.566899+0000","last_undegraded":"2024-08-17T18:11:13.566899+0000","last_fullsized":"2024-08-17T18:11:13.566899+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:35:48.964631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"347'149","reported_seq":803,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.534269+0000","last_change":"2024-08-17T18:09:49.461777+0000","last_active":"2024-08-17T18:11:13.534269+0000","last_peered":"2024-08-17T18:11:13.534269+0000","last_clean":"2024-08-17T18:11:13.534269+0000","last_became_active":"2024-08-17T18:09:15.592534+0000","last_became_peered":"2024-08-17T18:09:15.592534+0000","last_unstale":"2024-08-17T18:11:13.534269+0000","last_undegraded":"2024-08-17T18:11:13.534269+0000","last_fullsized":"2024-08-17T18:11:13.534269+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:49.461698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:49.461698+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:28:31.102314+0000","scrub_duration":0.0013792520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":266,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.539414+0000","last_change":"2024-08-17T18:04:43.771589+0000","last_active":"2024-08-17T18:11:13.539414+0000","last_peered":"2024-08-17T18:11:13.539414+0000","last_clean":"2024-08-17T18:11:13.539414+0000","last_became_active":"2024-08-17T18:04:43.769601+0000","last_became_peered":"2024-08-17T18:04:43.769601+0000","last_unstale":"2024-08-17T18:11:13.539414+0000","last_undegraded":"2024-08-17T18:11:13.539414+0000","last_fullsized":"2024-08-17T18:11:13.539414+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:42:49.228921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2458,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.566678+0000","last_change":"2024-08-17T18:04:14.537136+0000","last_active":"2024-08-17T18:11:13.566678+0000","last_peered":"2024-08-17T18:11:13.566678+0000","last_clean":"2024-08-17T18:11:13.566678+0000","last_became_active":"2024-08-17T18:04:14.536815+0000","last_became_peered":"2024-08-17T18:04:14.536815+0000","last_unstale":"2024-08-17T18:11:13.566678+0000","last_undegraded":"2024-08-17T18:11:13.566678+0000","last_fullsized":"2024-08-17T18:11:13.566678+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:02.706133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":97,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.531697+0000","last_change":"2024-08-17T18:04:43.773480+0000","last_active":"2024-08-17T18:11:10.531697+0000","last_peered":"2024-08-17T18:11:10.531697+0000","last_clean":"2024-08-17T18:11:10.531697+0000","last_became_active":"2024-08-17T18:04:43.773160+0000","last_became_peered":"2024-08-17T18:04:43.773160+0000","last_unstale":"2024-08-17T18:11:10.531697+0000","last_undegraded":"2024-08-17T18:11:10.531697+0000","last_fullsized":"2024-08-17T18:11:10.531697+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:07:55.468132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"347'149","reported_seq":821,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.531589+0000","last_change":"2024-08-17T18:09:15.596647+0000","last_active":"2024-08-17T18:11:13.531589+0000","last_peered":"2024-08-17T18:11:13.531589+0000","last_clean":"2024-08-17T18:11:13.531589+0000","last_became_active":"2024-08-17T18:09:15.596339+0000","last_became_peered":"2024-08-17T18:09:15.596339+0000","last_unstale":"2024-08-17T18:11:13.531589+0000","last_undegraded":"2024-08-17T18:11:13.531589+0000","last_fullsized":"2024-08-17T18:11:13.531589+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:00.316884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:00.316884+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:30.779927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":268,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.565300+0000","last_change":"2024-08-17T18:02:08.224928+0000","last_active":"2024-08-17T18:11:13.565300+0000","last_peered":"2024-08-17T18:11:13.565300+0000","last_clean":"2024-08-17T18:11:13.565300+0000","last_became_active":"2024-08-17T18:02:08.224604+0000","last_became_peered":"2024-08-17T18:02:08.224604+0000","last_unstale":"2024-08-17T18:11:13.565300+0000","last_undegraded":"2024-08-17T18:11:13.565300+0000","last_fullsized":"2024-08-17T18:11:13.565300+0000","mapping_epoch":1421,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1422,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:55.716971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1548,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.529283+0000","last_change":"2024-08-17T18:04:14.526890+0000","last_active":"2024-08-17T18:11:10.529283+0000","last_peered":"2024-08-17T18:11:10.529283+0000","last_clean":"2024-08-17T18:11:10.529283+0000","last_became_active":"2024-08-17T18:04:14.526564+0000","last_became_peered":"2024-08-17T18:04:14.526564+0000","last_unstale":"2024-08-17T18:11:10.529283+0000","last_undegraded":"2024-08-17T18:11:10.529283+0000","last_fullsized":"2024-08-17T18:11:10.529283+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:10:13.229827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":101,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.531552+0000","last_change":"2024-08-17T18:04:43.791749+0000","last_active":"2024-08-17T18:11:10.531552+0000","last_peered":"2024-08-17T18:11:10.531552+0000","last_clean":"2024-08-17T18:11:10.531552+0000","last_became_active":"2024-08-17T18:04:43.774474+0000","last_became_peered":"2024-08-17T18:04:43.774474+0000","last_unstale":"2024-08-17T18:11:10.531552+0000","last_undegraded":"2024-08-17T18:11:10.531552+0000","last_fullsized":"2024-08-17T18:11:10.531552+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:19:08.097560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"347'149","reported_seq":844,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.541557+0000","last_change":"2024-08-17T18:09:15.596502+0000","last_active":"2024-08-17T18:11:13.541557+0000","last_peered":"2024-08-17T18:11:13.541557+0000","last_clean":"2024-08-17T18:11:13.541557+0000","last_became_active":"2024-08-17T18:09:15.595460+0000","last_became_peered":"2024-08-17T18:09:15.595460+0000","last_unstale":"2024-08-17T18:11:13.541557+0000","last_undegraded":"2024-08-17T18:11:13.541557+0000","last_fullsized":"2024-08-17T18:11:13.541557+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:02.363531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:02.363531+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:36:27.680567+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2444,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.553336+0000","last_change":"2024-08-17T18:04:43.781133+0000","last_active":"2024-08-17T18:11:13.553336+0000","last_peered":"2024-08-17T18:11:13.553336+0000","last_clean":"2024-08-17T18:11:13.553336+0000","last_became_active":"2024-08-17T18:04:43.780787+0000","last_became_peered":"2024-08-17T18:04:43.780787+0000","last_unstale":"2024-08-17T18:11:13.553336+0000","last_undegraded":"2024-08-17T18:11:13.553336+0000","last_fullsized":"2024-08-17T18:11:13.553336+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:38:34.901162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2401,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.551306+0000","last_change":"2024-08-17T18:01:23.493896+0000","last_active":"2024-08-17T18:11:13.551306+0000","last_peered":"2024-08-17T18:11:13.551306+0000","last_clean":"2024-08-17T18:11:13.551306+0000","last_became_active":"2024-08-17T18:01:23.493551+0000","last_became_peered":"2024-08-17T18:01:23.493551+0000","last_unstale":"2024-08-17T18:11:13.551306+0000","last_undegraded":"2024-08-17T18:11:13.551306+0000","last_fullsized":"2024-08-17T18:11:13.551306+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:18:47.648483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2378,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.545940+0000","last_change":"2024-08-17T18:04:14.543532+0000","last_active":"2024-08-17T18:11:13.545940+0000","last_peered":"2024-08-17T18:11:13.545940+0000","last_clean":"2024-08-17T18:11:13.545940+0000","last_became_active":"2024-08-17T18:04:14.543131+0000","last_became_peered":"2024-08-17T18:04:14.543131+0000","last_unstale":"2024-08-17T18:11:13.545940+0000","last_undegraded":"2024-08-17T18:11:13.545940+0000","last_fullsized":"2024-08-17T18:11:13.545940+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:51:44.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"347'149","reported_seq":694,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.551836+0000","last_change":"2024-08-17T18:09:15.740862+0000","last_active":"2024-08-17T18:11:13.551836+0000","last_peered":"2024-08-17T18:11:13.551836+0000","last_clean":"2024-08-17T18:11:13.551836+0000","last_became_active":"2024-08-17T18:09:15.591452+0000","last_became_peered":"2024-08-17T18:09:15.591452+0000","last_unstale":"2024-08-17T18:11:13.551836+0000","last_undegraded":"2024-08-17T18:11:13.551836+0000","last_fullsized":"2024-08-17T18:11:13.551836+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:15.740775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:15.740775+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:51:20.687268+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":216,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.539635+0000","last_change":"2024-08-17T18:10:11.727045+0000","last_active":"2024-08-17T18:11:13.539635+0000","last_peered":"2024-08-17T18:11:13.539635+0000","last_clean":"2024-08-17T18:11:13.539635+0000","last_became_active":"2024-08-17T18:10:11.726725+0000","last_became_peered":"2024-08-17T18:10:11.726725+0000","last_unstale":"2024-08-17T18:11:13.539635+0000","last_undegraded":"2024-08-17T18:11:13.539635+0000","last_fullsized":"2024-08-17T18:11:13.539635+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:12:13.835640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":209,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.540302+0000","last_change":"2024-08-17T18:08:11.440676+0000","last_active":"2024-08-17T18:11:13.540302+0000","last_peered":"2024-08-17T18:11:13.540302+0000","last_clean":"2024-08-17T18:11:13.540302+0000","last_became_active":"2024-08-17T18:08:11.440353+0000","last_became_peered":"2024-08-17T18:08:11.440353+0000","last_unstale":"2024-08-17T18:11:13.540302+0000","last_undegraded":"2024-08-17T18:11:13.540302+0000","last_fullsized":"2024-08-17T18:11:13.540302+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:19:22.125277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1144,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.551362+0000","last_change":"2024-08-17T18:04:14.537308+0000","last_active":"2024-08-17T18:11:13.551362+0000","last_peered":"2024-08-17T18:11:13.551362+0000","last_clean":"2024-08-17T18:11:13.551362+0000","last_became_active":"2024-08-17T18:04:14.535521+0000","last_became_peered":"2024-08-17T18:04:14.535521+0000","last_unstale":"2024-08-17T18:11:13.551362+0000","last_undegraded":"2024-08-17T18:11:13.551362+0000","last_fullsized":"2024-08-17T18:11:13.551362+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:40:00.384936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":44,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.521753+0000","last_change":"2024-08-17T18:08:37.071912+0000","last_active":"2024-08-17T18:11:10.521753+0000","last_peered":"2024-08-17T18:11:10.521753+0000","last_clean":"2024-08-17T18:11:10.521753+0000","last_became_active":"2024-08-17T18:08:37.071648+0000","last_became_peered":"2024-08-17T18:08:37.071648+0000","last_unstale":"2024-08-17T18:11:10.521753+0000","last_undegraded":"2024-08-17T18:11:10.521753+0000","last_fullsized":"2024-08-17T18:11:10.521753+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:58:57.279808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":206,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.538894+0000","last_change":"2024-08-17T18:04:43.771309+0000","last_active":"2024-08-17T18:11:13.538894+0000","last_peered":"2024-08-17T18:11:13.538894+0000","last_clean":"2024-08-17T18:11:13.538894+0000","last_became_active":"2024-08-17T18:04:43.770922+0000","last_became_peered":"2024-08-17T18:04:43.770922+0000","last_unstale":"2024-08-17T18:11:13.538894+0000","last_undegraded":"2024-08-17T18:11:13.538894+0000","last_fullsized":"2024-08-17T18:11:13.538894+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:10.173251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":101,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.531630+0000","last_change":"2024-08-17T18:04:43.786228+0000","last_active":"2024-08-17T18:11:10.531630+0000","last_peered":"2024-08-17T18:11:10.531630+0000","last_clean":"2024-08-17T18:11:10.531630+0000","last_became_active":"2024-08-17T18:04:43.770976+0000","last_became_peered":"2024-08-17T18:04:43.770976+0000","last_unstale":"2024-08-17T18:11:10.531630+0000","last_undegraded":"2024-08-17T18:11:10.531630+0000","last_fullsized":"2024-08-17T18:11:10.531630+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:20:10.131137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":835,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.541253+0000","last_change":"2024-08-17T18:07:54.303783+0000","last_active":"2024-08-17T18:11:13.541253+0000","last_peered":"2024-08-17T18:11:13.541253+0000","last_clean":"2024-08-17T18:11:13.541253+0000","last_became_active":"2024-08-17T18:06:57.358131+0000","last_became_peered":"2024-08-17T18:06:57.358131+0000","last_unstale":"2024-08-17T18:11:13.541253+0000","last_undegraded":"2024-08-17T18:11:13.541253+0000","last_fullsized":"2024-08-17T18:11:13.541253+0000","mapping_epoch":1460,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1492,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:54.303708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:54.303708+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:30:45.253719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"347'149","reported_seq":822,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.531046+0000","last_change":"2024-08-17T18:09:15.592722+0000","last_active":"2024-08-17T18:11:13.531046+0000","last_peered":"2024-08-17T18:11:13.531046+0000","last_clean":"2024-08-17T18:11:13.531046+0000","last_became_active":"2024-08-17T18:09:15.592242+0000","last_became_peered":"2024-08-17T18:09:15.592242+0000","last_unstale":"2024-08-17T18:11:13.531046+0000","last_undegraded":"2024-08-17T18:11:13.531046+0000","last_fullsized":"2024-08-17T18:11:13.531046+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:55.255983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:55.255983+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:02:53.596275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2387,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.553221+0000","last_change":"2024-08-17T17:24:17.455710+0000","last_active":"2024-08-17T18:11:13.553221+0000","last_peered":"2024-08-17T18:11:13.553221+0000","last_clean":"2024-08-17T18:11:13.553221+0000","last_became_active":"2024-08-17T17:24:17.452989+0000","last_became_peered":"2024-08-17T17:24:17.452989+0000","last_unstale":"2024-08-17T18:11:13.553221+0000","last_undegraded":"2024-08-17T18:11:13.553221+0000","last_fullsized":"2024-08-17T18:11:13.553221+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:58:07.894968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":91,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.543603+0000","last_change":"2024-08-17T18:08:11.347750+0000","last_active":"2024-08-17T18:11:13.543603+0000","last_peered":"2024-08-17T18:11:13.543603+0000","last_clean":"2024-08-17T18:11:13.543603+0000","last_became_active":"2024-08-17T18:08:11.346796+0000","last_became_peered":"2024-08-17T18:08:11.346796+0000","last_unstale":"2024-08-17T18:11:13.543603+0000","last_undegraded":"2024-08-17T18:11:13.543603+0000","last_fullsized":"2024-08-17T18:11:13.543603+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:24.470900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1545,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.529631+0000","last_change":"2024-08-17T18:08:37.068370+0000","last_active":"2024-08-17T18:11:10.529631+0000","last_peered":"2024-08-17T18:11:10.529631+0000","last_clean":"2024-08-17T18:11:10.529631+0000","last_became_active":"2024-08-17T18:08:37.068027+0000","last_became_peered":"2024-08-17T18:08:37.068027+0000","last_unstale":"2024-08-17T18:11:10.529631+0000","last_undegraded":"2024-08-17T18:11:10.529631+0000","last_fullsized":"2024-08-17T18:11:10.529631+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:02:12.661660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1553,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.529160+0000","last_change":"2024-08-17T18:03:51.270293+0000","last_active":"2024-08-17T18:11:10.529160+0000","last_peered":"2024-08-17T18:11:10.529160+0000","last_clean":"2024-08-17T18:11:10.529160+0000","last_became_active":"2024-08-17T18:03:51.269970+0000","last_became_peered":"2024-08-17T18:03:51.269970+0000","last_unstale":"2024-08-17T18:11:10.529160+0000","last_undegraded":"2024-08-17T18:11:10.529160+0000","last_fullsized":"2024-08-17T18:11:10.529160+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:02:30.016517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"347'149","reported_seq":822,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.536191+0000","last_change":"2024-08-17T18:09:15.590888+0000","last_active":"2024-08-17T18:11:13.536191+0000","last_peered":"2024-08-17T18:11:13.536191+0000","last_clean":"2024-08-17T18:11:13.536191+0000","last_became_active":"2024-08-17T18:09:15.590345+0000","last_became_peered":"2024-08-17T18:09:15.590345+0000","last_unstale":"2024-08-17T18:11:13.536191+0000","last_undegraded":"2024-08-17T18:11:13.536191+0000","last_fullsized":"2024-08-17T18:11:13.536191+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:59.307356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:59.307356+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:40:40.796012+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1557,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.529101+0000","last_change":"2024-08-17T18:01:23.497110+0000","last_active":"2024-08-17T18:11:10.529101+0000","last_peered":"2024-08-17T18:11:10.529101+0000","last_clean":"2024-08-17T18:11:10.529101+0000","last_became_active":"2024-08-17T18:01:23.496800+0000","last_became_peered":"2024-08-17T18:01:23.496800+0000","last_unstale":"2024-08-17T18:11:10.529101+0000","last_undegraded":"2024-08-17T18:11:10.529101+0000","last_fullsized":"2024-08-17T18:11:10.529101+0000","mapping_epoch":1399,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:32:29.892739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1526,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.521601+0000","last_change":"2024-08-17T17:23:07.401624+0000","last_active":"2024-08-17T18:11:10.521601+0000","last_peered":"2024-08-17T18:11:10.521601+0000","last_clean":"2024-08-17T18:11:10.521601+0000","last_became_active":"2024-08-17T17:23:07.400222+0000","last_became_peered":"2024-08-17T17:23:07.400222+0000","last_unstale":"2024-08-17T18:11:10.521601+0000","last_undegraded":"2024-08-17T18:11:10.521601+0000","last_fullsized":"2024-08-17T18:11:10.521601+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":668,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:07.069989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"347'149","reported_seq":838,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.540443+0000","last_change":"2024-08-17T18:09:15.592665+0000","last_active":"2024-08-17T18:11:13.540443+0000","last_peered":"2024-08-17T18:11:13.540443+0000","last_clean":"2024-08-17T18:11:13.540443+0000","last_became_active":"2024-08-17T18:09:15.590435+0000","last_became_peered":"2024-08-17T18:09:15.590435+0000","last_unstale":"2024-08-17T18:11:13.540443+0000","last_undegraded":"2024-08-17T18:11:13.540443+0000","last_fullsized":"2024-08-17T18:11:13.540443+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:52.271487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:52.271487+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:18:07.889477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1527,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.529730+0000","last_change":"2024-08-17T17:24:17.470809+0000","last_active":"2024-08-17T18:11:10.529730+0000","last_peered":"2024-08-17T18:11:10.529730+0000","last_clean":"2024-08-17T18:11:10.529730+0000","last_became_active":"2024-08-17T17:24:17.469753+0000","last_became_peered":"2024-08-17T17:24:17.469753+0000","last_unstale":"2024-08-17T18:11:10.529730+0000","last_undegraded":"2024-08-17T18:11:10.529730+0000","last_fullsized":"2024-08-17T18:11:10.529730+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:00.049425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":2395,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.535522+0000","last_change":"2024-08-17T18:04:14.552806+0000","last_active":"2024-08-17T18:11:13.535522+0000","last_peered":"2024-08-17T18:11:13.535522+0000","last_clean":"2024-08-17T18:11:13.535522+0000","last_became_active":"2024-08-17T18:04:14.535546+0000","last_became_peered":"2024-08-17T18:04:14.535546+0000","last_unstale":"2024-08-17T18:11:13.535522+0000","last_undegraded":"2024-08-17T18:11:13.535522+0000","last_fullsized":"2024-08-17T18:11:13.535522+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:44:45.702272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":195,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.539547+0000","last_change":"2024-08-17T18:04:14.539500+0000","last_active":"2024-08-17T18:11:13.539547+0000","last_peered":"2024-08-17T18:11:13.539547+0000","last_clean":"2024-08-17T18:11:13.539547+0000","last_became_active":"2024-08-17T18:04:14.538738+0000","last_became_peered":"2024-08-17T18:04:14.538738+0000","last_unstale":"2024-08-17T18:11:13.539547+0000","last_undegraded":"2024-08-17T18:11:13.539547+0000","last_fullsized":"2024-08-17T18:11:13.539547+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:31.439947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":97,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.531295+0000","last_change":"2024-08-17T18:04:43.771107+0000","last_active":"2024-08-17T18:11:13.531295+0000","last_peered":"2024-08-17T18:11:13.531295+0000","last_clean":"2024-08-17T18:11:13.531295+0000","last_became_active":"2024-08-17T18:04:43.770799+0000","last_became_peered":"2024-08-17T18:04:43.770799+0000","last_unstale":"2024-08-17T18:11:13.531295+0000","last_undegraded":"2024-08-17T18:11:13.531295+0000","last_fullsized":"2024-08-17T18:11:13.531295+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:09:04.350798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":44,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.521428+0000","last_change":"2024-08-17T18:08:37.080554+0000","last_active":"2024-08-17T18:11:10.521428+0000","last_peered":"2024-08-17T18:11:10.521428+0000","last_clean":"2024-08-17T18:11:10.521428+0000","last_became_active":"2024-08-17T18:08:37.079920+0000","last_became_peered":"2024-08-17T18:08:37.079920+0000","last_unstale":"2024-08-17T18:11:10.521428+0000","last_undegraded":"2024-08-17T18:11:10.521428+0000","last_fullsized":"2024-08-17T18:11:10.521428+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:08:25.214786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":101,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.531458+0000","last_change":"2024-08-17T18:04:43.781111+0000","last_active":"2024-08-17T18:11:13.531458+0000","last_peered":"2024-08-17T18:11:13.531458+0000","last_clean":"2024-08-17T18:11:13.531458+0000","last_became_active":"2024-08-17T18:04:43.771839+0000","last_became_peered":"2024-08-17T18:04:43.771839+0000","last_unstale":"2024-08-17T18:11:13.531458+0000","last_undegraded":"2024-08-17T18:11:13.531458+0000","last_fullsized":"2024-08-17T18:11:13.531458+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:52:45.973049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1561,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.528847+0000","last_change":"2024-08-17T18:08:11.449492+0000","last_active":"2024-08-17T18:11:10.528847+0000","last_peered":"2024-08-17T18:11:10.528847+0000","last_clean":"2024-08-17T18:11:10.528847+0000","last_became_active":"2024-08-17T18:08:11.424764+0000","last_became_peered":"2024-08-17T18:08:11.424764+0000","last_unstale":"2024-08-17T18:11:10.528847+0000","last_undegraded":"2024-08-17T18:11:10.528847+0000","last_fullsized":"2024-08-17T18:11:10.528847+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:53:37.552880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":214,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.541007+0000","last_change":"2024-08-17T18:04:14.551061+0000","last_active":"2024-08-17T18:11:13.541007+0000","last_peered":"2024-08-17T18:11:13.541007+0000","last_clean":"2024-08-17T18:11:13.541007+0000","last_became_active":"2024-08-17T18:04:14.543296+0000","last_became_peered":"2024-08-17T18:04:14.543296+0000","last_unstale":"2024-08-17T18:11:13.541007+0000","last_undegraded":"2024-08-17T18:11:13.541007+0000","last_fullsized":"2024-08-17T18:11:13.541007+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:39:45.294265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":115,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.536309+0000","last_change":"2024-08-17T18:08:37.068503+0000","last_active":"2024-08-17T18:11:13.536309+0000","last_peered":"2024-08-17T18:11:13.536309+0000","last_clean":"2024-08-17T18:11:13.536309+0000","last_became_active":"2024-08-17T18:08:37.068110+0000","last_became_peered":"2024-08-17T18:08:37.068110+0000","last_unstale":"2024-08-17T18:11:13.536309+0000","last_undegraded":"2024-08-17T18:11:13.536309+0000","last_fullsized":"2024-08-17T18:11:13.536309+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:12:30.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":142,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.529540+0000","last_change":"2024-08-17T18:04:14.547798+0000","last_active":"2024-08-17T18:11:10.529540+0000","last_peered":"2024-08-17T18:11:10.529540+0000","last_clean":"2024-08-17T18:11:10.529540+0000","last_became_active":"2024-08-17T18:04:14.527496+0000","last_became_peered":"2024-08-17T18:04:14.527496+0000","last_unstale":"2024-08-17T18:11:10.529540+0000","last_undegraded":"2024-08-17T18:11:10.529540+0000","last_fullsized":"2024-08-17T18:11:10.529540+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:25:54.338846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1608,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.522194+0000","last_change":"2024-08-17T18:04:43.813536+0000","last_active":"2024-08-17T18:11:10.522194+0000","last_peered":"2024-08-17T18:11:10.522194+0000","last_clean":"2024-08-17T18:11:10.522194+0000","last_became_active":"2024-08-17T18:04:43.799823+0000","last_became_peered":"2024-08-17T18:04:43.799823+0000","last_unstale":"2024-08-17T18:11:10.522194+0000","last_undegraded":"2024-08-17T18:11:10.522194+0000","last_fullsized":"2024-08-17T18:11:10.522194+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:04:10.837113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":98,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.536076+0000","last_change":"2024-08-17T18:10:11.737395+0000","last_active":"2024-08-17T18:11:13.536076+0000","last_peered":"2024-08-17T18:11:13.536076+0000","last_clean":"2024-08-17T18:11:13.536076+0000","last_became_active":"2024-08-17T18:10:11.736935+0000","last_became_peered":"2024-08-17T18:10:11.736935+0000","last_unstale":"2024-08-17T18:11:13.536076+0000","last_undegraded":"2024-08-17T18:11:13.536076+0000","last_fullsized":"2024-08-17T18:11:13.536076+0000","mapping_epoch":1541,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1542,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:32:49.395319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":266,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.528970+0000","last_change":"2024-08-17T18:04:14.527380+0000","last_active":"2024-08-17T18:11:10.528970+0000","last_peered":"2024-08-17T18:11:10.528970+0000","last_clean":"2024-08-17T18:11:10.528970+0000","last_became_active":"2024-08-17T18:04:14.527076+0000","last_became_peered":"2024-08-17T18:04:14.527076+0000","last_unstale":"2024-08-17T18:11:10.528970+0000","last_undegraded":"2024-08-17T18:11:10.528970+0000","last_fullsized":"2024-08-17T18:11:10.528970+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:36:35.583709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":194,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.539919+0000","last_change":"2024-08-17T18:04:14.541814+0000","last_active":"2024-08-17T18:11:13.539919+0000","last_peered":"2024-08-17T18:11:13.539919+0000","last_clean":"2024-08-17T18:11:13.539919+0000","last_became_active":"2024-08-17T18:04:14.541403+0000","last_became_peered":"2024-08-17T18:04:14.541403+0000","last_unstale":"2024-08-17T18:11:13.539919+0000","last_undegraded":"2024-08-17T18:11:13.539919+0000","last_fullsized":"2024-08-17T18:11:13.539919+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:07:45.294924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":209,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.541748+0000","last_change":"2024-08-17T18:08:11.351307+0000","last_active":"2024-08-17T18:11:13.541748+0000","last_peered":"2024-08-17T18:11:13.541748+0000","last_clean":"2024-08-17T18:11:13.541748+0000","last_became_active":"2024-08-17T18:08:11.350926+0000","last_became_peered":"2024-08-17T18:08:11.350926+0000","last_unstale":"2024-08-17T18:11:13.541748+0000","last_undegraded":"2024-08-17T18:11:13.541748+0000","last_fullsized":"2024-08-17T18:11:13.541748+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:59:57.477492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":96,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.531760+0000","last_change":"2024-08-17T18:04:43.771720+0000","last_active":"2024-08-17T18:11:10.531760+0000","last_peered":"2024-08-17T18:11:10.531760+0000","last_clean":"2024-08-17T18:11:10.531760+0000","last_became_active":"2024-08-17T18:04:43.771137+0000","last_became_peered":"2024-08-17T18:04:43.771137+0000","last_unstale":"2024-08-17T18:11:10.531760+0000","last_undegraded":"2024-08-17T18:11:10.531760+0000","last_fullsized":"2024-08-17T18:11:10.531760+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:57.647909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":131,"reported_epoch":1547,"state":"active+clean","last_fresh":"2024-08-17T18:11:10.528929+0000","last_change":"2024-08-17T18:03:51.283872+0000","last_active":"2024-08-17T18:11:10.528929+0000","last_peered":"2024-08-17T18:11:10.528929+0000","last_clean":"2024-08-17T18:11:10.528929+0000","last_became_active":"2024-08-17T18:03:51.267197+0000","last_became_peered":"2024-08-17T18:03:51.267197+0000","last_unstale":"2024-08-17T18:11:10.528929+0000","last_undegraded":"2024-08-17T18:11:10.528929+0000","last_fullsized":"2024-08-17T18:11:10.528929+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:18:59.157786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2385,"reported_epoch":1548,"state":"active+clean","last_fresh":"2024-08-17T18:11:13.535846+0000","last_change":"2024-08-17T18:03:51.277491+0000","last_active":"2024-08-17T18:11:13.535846+0000","last_peered":"2024-08-17T18:11:13.535846+0000","last_clean":"2024-08-17T18:11:13.535846+0000","last_became_active":"2024-08-17T18:03:51.277178+0000","last_became_peered":"2024-08-17T18:03:51.277178+0000","last_unstale":"2024-08-17T18:11:13.535846+0000","last_undegraded":"2024-08-17T18:11:13.535846+0000","last_fullsized":"2024-08-17T18:11:13.535846+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:21:35.319085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":56,"num_pg":16,"stat_sum":{"num_bytes":7226458136,"num_objects":110268,"num_object_clones":0,"num_object_copies":330804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110268,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143136,"num_write_kb":8108865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21638164480,"data_stored":21638152264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116036,"ondisk_log_size":116036,"up":48,"acting":48,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":7994,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234745,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496383,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4424592,"kb_used_data":3537096,"kb_used_omap":1,"kb_used_meta":887486,"kb_avail":89320560,"statfs":{"total":95995035648,"available":91464253440,"internally_reserved":0,"allocated":3621986304,"data_stored":3616427347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":908786119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565624,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3618116,"kb_used_data":2697468,"kb_used_omap":72,"kb_used_meta":920567,"kb_avail":90127036,"statfs":{"total":95995035648,"available":92290084864,"internally_reserved":0,"allocated":2762207232,"data_stored":2756656400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":942661102},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":1399,"seq":6008659247226,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3478988,"kb_used_data":3108100,"kb_used_omap":131,"kb_used_meta":370748,"kb_avail":90266164,"statfs":{"total":95995035648,"available":92432551936,"internally_reserved":0,"allocated":3182694400,"data_stored":3177140564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134291,"internal_metadata":379646829},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743187014,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3442236,"kb_used_data":2720244,"kb_used_omap":66,"kb_used_meta":721917,"kb_avail":90302916,"statfs":{"total":95995035648,"available":92470185984,"internally_reserved":0,"allocated":2785529856,"data_stored":2779986529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":739243691},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448653,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3189316,"kb_used_data":2679804,"kb_used_omap":75,"kb_used_meta":509364,"kb_avail":90555836,"statfs":{"total":95995035648,"available":92729176064,"internally_reserved":0,"allocated":2744119296,"data_stored":2738568454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":521588927},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208758018,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4404740,"kb_used_data":3493372,"kb_used_omap":1,"kb_used_meta":911358,"kb_avail":89340412,"statfs":{"total":95995035648,"available":91484581888,"internally_reserved":0,"allocated":3577212928,"data_stored":3571662571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":933231046},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,4,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019656,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3827340,"kb_used_data":3092420,"kb_used_omap":1,"kb_used_meta":734910,"kb_avail":89917812,"statfs":{"total":95995035648,"available":92075839488,"internally_reserved":0,"allocated":3166638080,"data_stored":3161083218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":752548294},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2716209152,"data_stored":2716205080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2748907520,"data_stored":2748907520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3154706432,"data_stored":3154706432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2727546880,"data_stored":2727542808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3144417280,"data_stored":3144417280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3537895424,"data_stored":3537895424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3608481792,"data_stored":3608477720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T18:11:34.244 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-08-17T18:11:34.305 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:11:34.334 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:34.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:34.403 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:34.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:34.504 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:34.535 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:34.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:34.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:34.637 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:34.666 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:34.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:34.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:34.768 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:34.797 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:34.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:34.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:34.899 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:34.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:34.932 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:34.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:34.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:34 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:11:34.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:34 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:11:34.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:34 smithi136 ceph-mon[109861]: osdmap e1549: 8 total, 8 up, 7 in 2024-08-17T18:11:34.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:34 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:11:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:34 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:11:34.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:34 smithi136 ceph-mon[115078]: osdmap e1549: 8 total, 8 up, 7 in 2024-08-17T18:11:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:34 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:11:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:34 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:11:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:34 smithi165 ceph-mon[105517]: osdmap e1549: 8 total, 8 up, 7 in 2024-08-17T18:11:35.033 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:35.074 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:35.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:35.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:35.109 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:11:35.176 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:35.225 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:35.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:35.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:35.327 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:35.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:35.360 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:35.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:35.461 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:35.495 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:35.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:35.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:35.597 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:35.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:35.635 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:35.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:35.736 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:35.770 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:35.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:35.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:35.872 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:35.902 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:35.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:35.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:35.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:35 smithi136 ceph-mon[109861]: pgmap v2872: 99 pgs: 99 active+clean; 6.4 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 975 op/s 2024-08-17T18:11:35.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:35 smithi136 ceph-mon[115078]: pgmap v2872: 99 pgs: 99 active+clean; 6.4 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 975 op/s 2024-08-17T18:11:35.989 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-17T18:11:36.004 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:36.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:35 smithi165 ceph-mon[105517]: pgmap v2872: 99 pgs: 99 active+clean; 6.4 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 975 op/s 2024-08-17T18:11:36.040 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:36.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:36.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:36.068 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:11:36.142 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:36.173 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:36.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:36.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:36.275 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:36.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:36.325 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:36.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:36.426 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:36.473 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:36.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:36.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:36.575 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:36.602 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:36.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:36.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:36.705 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:36.751 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:36.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:36.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:36.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:36.893 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:36.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:36.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:36.995 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:37.021 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:37.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:37.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:37.123 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:37.139 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:11:37.149 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:37.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:37.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:37.251 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:37.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:37.303 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:37.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:37.404 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:37.434 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:37.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:37.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:37.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:11:37.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:11:37.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:37 smithi136 ceph-mon[109861]: pgmap v2873: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 983 op/s 2024-08-17T18:11:37.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:11:37.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:11:37.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:11:37.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:11:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:11:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:11:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:37 smithi136 ceph-mon[115078]: pgmap v2873: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 983 op/s 2024-08-17T18:11:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:11:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:11:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:11:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:11:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:11:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:11:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:37 smithi165 ceph-mon[105517]: pgmap v2873: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 983 op/s 2024-08-17T18:11:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:11:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:11:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:11:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:11:37.536 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:37.564 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:37.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:37.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:37.665 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:37.701 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:37.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:37.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:37.722 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:11:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:11:37] "GET /metrics HTTP/1.1" 200 36610 "" "Prometheus/2.51.0" 2024-08-17T18:11:37.803 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:37.836 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:37.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:37.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:37.938 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:37.951 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:11:37.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:37.992 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:37.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:38.093 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:38.131 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:38.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:38.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:38.233 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:38.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:38.264 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:38.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:38.365 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:38.396 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:38.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:38.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:38.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:11:38.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:11:38.498 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:38.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:38 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:11:38.535 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:38.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:38.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:38.636 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:38.673 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:38.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:38.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:38.775 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:38.800 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:38.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:38.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:38.902 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:38.936 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:38.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:38.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:39.038 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:39.069 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:39.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:39.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:39.171 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:39.201 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:39.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:39.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:39.303 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:39.334 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:39.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:39.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:39.394 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:39 smithi136 ceph-mon[109861]: pgmap v2874: 99 pgs: 99 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 983 op/s 2024-08-17T18:11:39.394 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:39 smithi136 ceph-mon[115078]: pgmap v2874: 99 pgs: 99 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 983 op/s 2024-08-17T18:11:39.436 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:39.472 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:39.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:39.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:39.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:39 smithi165 ceph-mon[105517]: pgmap v2874: 99 pgs: 99 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 983 op/s 2024-08-17T18:11:39.574 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:39.617 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:39.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:39.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:39.719 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:39.756 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:39.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:39.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:39.858 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:39.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:39.900 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:39.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:40.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:40.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:40.040 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:40.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:40.141 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:40.175 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:40.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:40.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:40.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:40.323 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:40.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:40.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:40.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:40.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:40.465 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:40.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:40.566 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:40.596 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:40.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:40.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:40.698 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:40.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:40.731 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:40.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:40.832 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:40.839 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:11:40.865 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:40.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:40.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:40.967 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:40.994 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:40.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:40.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:41.096 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:41.137 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:41.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:41.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:41.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:41.271 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:41.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:41.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:41.373 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:41.413 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:41.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:41.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:41.515 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:41.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:41.559 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:41.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:41.660 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:41.675 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:41 smithi165 ceph-mon[105517]: pgmap v2875: 99 pgs: 99 active+clean; 6.1 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 987 op/s 2024-08-17T18:11:41.696 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:41.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:41 smithi136 ceph-mon[109861]: pgmap v2875: 99 pgs: 99 active+clean; 6.1 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 987 op/s 2024-08-17T18:11:41.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:41 smithi136 ceph-mon[115078]: pgmap v2875: 99 pgs: 99 active+clean; 6.1 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 987 op/s 2024-08-17T18:11:41.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:41.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:41.798 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:41.835 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:41.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:41.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:41.937 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:41.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:41.974 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:41.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:42.075 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:42.113 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:42.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:42.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:42.215 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:42.252 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:42.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:42.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:42.354 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:42.386 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:42.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:42.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:42.424 INFO:teuthology.orchestra.run.smithi136.stderr:set pool 1 pgp_num to 11 2024-08-17T18:11:42.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:42.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:42.533 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:42.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:42.635 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:42.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:42.677 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:42.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:42.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:42 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3078095922' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T18:11:42.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:42 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3078095922' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T18:11:42.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:42 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3078095922' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T18:11:42.778 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:42.818 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:42.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:42.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:42.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:42.950 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:42.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:42.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:43.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:43.079 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:43.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:43.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:43.181 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:43.223 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:43.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:43.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:43.325 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:43.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:43.371 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:43.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:43.472 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:43.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:43.506 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:43.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:43.570 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:11:43.571 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T18:11:43.571 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-17T18:11:43.571 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-17T18:11:43.571 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:11:43.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:43.632 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:11:43.632 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:11:43.632 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:11:43.632 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:11:43.632 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:11:43.632 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:11:43.633 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:11:43.633 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:11:43.658 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:43.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:43.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:43.663 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:11:43.663 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:11:43.664 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:43 smithi136 ceph-mon[109861]: pgmap v2876: 99 pgs: 99 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 975 op/s 2024-08-17T18:11:43.664 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:43 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3078095922' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-08-17T18:11:43.664 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:43 smithi136 ceph-mon[109861]: osdmap e1550: 8 total, 8 up, 7 in 2024-08-17T18:11:43.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:43 smithi136 ceph-mon[115078]: pgmap v2876: 99 pgs: 99 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 975 op/s 2024-08-17T18:11:43.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:43 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3078095922' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-08-17T18:11:43.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:43 smithi136 ceph-mon[115078]: osdmap e1550: 8 total, 8 up, 7 in 2024-08-17T18:11:43.698 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:11:43.698 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:11:43.739 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:11:43.739 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:11:43.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:43 smithi165 ceph-mon[105517]: pgmap v2876: 99 pgs: 99 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 975 op/s 2024-08-17T18:11:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:43 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3078095922' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-08-17T18:11:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:43 smithi165 ceph-mon[105517]: osdmap e1550: 8 total, 8 up, 7 in 2024-08-17T18:11:43.770 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:11:43.771 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:11:43.800 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:43.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:43.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:43.902 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:43.941 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:43.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:43.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:44.043 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:44.084 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:44.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:44.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:44.186 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:44.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:44.220 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:44.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:44.322 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:44.356 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:44.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:44.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:44.458 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:44.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:44.493 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:44.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:44.593 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:44.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:44.627 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:44.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:44.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:44 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2354235575' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:11:44.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:44 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2354235575' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:11:44.728 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:44.756 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:44.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:44.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:44.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:44 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2354235575' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:11:44.858 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:44.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:44.901 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:44.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:45.002 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:45.038 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:45.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:45.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:45.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:45.180 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:45.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:45.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:45.282 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:45.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:45.315 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:45.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:45.416 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:45.439 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:45.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:45.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:45.541 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:45.572 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:45.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:45.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:45.674 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:45.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:45.712 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:45.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:45.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:45 smithi165 ceph-mon[105517]: pgmap v2878: 99 pgs: 99 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 974 op/s 2024-08-17T18:11:45.813 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:45.849 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:45.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:45.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:45 smithi136 ceph-mon[109861]: pgmap v2878: 99 pgs: 99 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 974 op/s 2024-08-17T18:11:45.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:45 smithi136 ceph-mon[115078]: pgmap v2878: 99 pgs: 99 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 974 op/s 2024-08-17T18:11:45.952 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:45.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:45.988 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:45.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:46.090 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:46.116 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:46.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:46.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:46.218 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:46.261 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:46.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:46.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:46.363 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:46.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:46.391 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:46.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:46.492 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:46.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:46.525 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:46.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:46.626 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:46.668 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:46.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:46.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:46.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:46.805 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:46.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:46.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:46.907 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:46.943 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:46.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:46.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:47.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:47.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:47.076 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:47.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:47.177 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:47.199 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:47.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:47.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:47.301 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:47.408 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:47.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:47.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:47.510 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:47.539 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:47.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:47.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:47.641 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:47.673 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:47.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:47.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:47 smithi165 ceph-mon[105517]: pgmap v2879: 99 pgs: 99 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 976 op/s 2024-08-17T18:11:47.775 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:47.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:47.828 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:47.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:47.929 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:47.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:47 smithi136 ceph-mon[109861]: pgmap v2879: 99 pgs: 99 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 976 op/s 2024-08-17T18:11:47.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:47 smithi136 ceph-mon[115078]: pgmap v2879: 99 pgs: 99 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 976 op/s 2024-08-17T18:11:47.947 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:11:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:11:47] "GET /metrics HTTP/1.1" 200 36610 "" "Prometheus/2.51.0" 2024-08-17T18:11:47.969 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:47.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:47.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:48.071 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:48.107 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:48.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:48.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:48.209 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:48.251 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:48.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:48.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:48.353 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:48.395 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:48.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:48.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:48.497 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:48.518 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:48.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:48.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:48.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:48.659 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:48.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:48.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:48.761 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:48.785 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:48.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:48.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:48.817 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-17T18:11:48.887 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:48.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:48.941 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:48.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:49.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:49.069 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:49.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:49.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:49.171 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:49.213 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:49.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:49.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:49.315 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:49.355 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:49.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:49.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:49.457 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:49.489 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:49.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:49.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:49.516 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:11:49.591 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:49.622 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:49.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:49.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:49.724 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:49.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:49.760 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:49.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:49 smithi165 ceph-mon[105517]: pgmap v2880: 99 pgs: 99 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 981 op/s 2024-08-17T18:11:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:49 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/671025029' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:11:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:49 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:11:49.861 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:49.887 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:49 smithi136 ceph-mon[109861]: pgmap v2880: 99 pgs: 99 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 981 op/s 2024-08-17T18:11:49.887 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:49 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/671025029' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:11:49.887 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:49 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:11:49.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:49 smithi136 ceph-mon[115078]: pgmap v2880: 99 pgs: 99 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 981 op/s 2024-08-17T18:11:49.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:49 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/671025029' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:11:49.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:49 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:11:49.889 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:49.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:49.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:49.990 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:50.026 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:50.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:50.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:50.128 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:50.159 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:50.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:50.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:50.261 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:50.292 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:50.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:50.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:50.394 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:50.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:50.436 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:50.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:50.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:50.572 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:50.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:50.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:50.675 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:50.715 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:50.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:50.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:50.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:50 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:11:50.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:50 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:11:50.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:50 smithi165 ceph-mon[105517]: osdmap e1551: 8 total, 8 up, 7 in 2024-08-17T18:11:50.817 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:50.845 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:50.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:50.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:50.849 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:50 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:11:50.849 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:50 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:11:50.849 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:50 smithi136 ceph-mon[109861]: osdmap e1551: 8 total, 8 up, 7 in 2024-08-17T18:11:50.849 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:50 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:11:50.849 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:50 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:11:50.849 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:50 smithi136 ceph-mon[115078]: osdmap e1551: 8 total, 8 up, 7 in 2024-08-17T18:11:50.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:50.982 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:50.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:50.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:51.084 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:51.118 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:51.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:51.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:51.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:51.248 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:51.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:51.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:51.350 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:51.389 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:51.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:51.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:51.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:51.535 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:51.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:51.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:51.637 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:51.674 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:11:51.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:51.697 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:51.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:51.776 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:51 smithi165 ceph-mon[105517]: pgmap v2882: 99 pgs: 99 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 837 op/s 2024-08-17T18:11:51.777 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:11:51.798 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:51.818 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:51 smithi136 ceph-mon[109861]: pgmap v2882: 99 pgs: 99 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 837 op/s 2024-08-17T18:11:51.819 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:11:51.819 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:51 smithi136 ceph-mon[115078]: pgmap v2882: 99 pgs: 99 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 837 op/s 2024-08-17T18:11:51.819 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:11:51.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:51.845 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:51.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:51.946 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:51.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:51.989 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:51.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:52.090 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:52.122 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:52.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:52.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:52.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:52.261 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:52.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:52.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:52.363 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:52.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:52.402 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:52.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:52.503 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:52.549 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:52.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:52.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:52.651 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:52.686 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:52.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:52.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:52.788 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:52.824 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:52.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:52.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:52.886 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:11:52.926 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:52.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:52.972 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:52.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:53.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:53.104 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:53.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:53.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:53.206 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:53.251 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:53.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:53.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:53.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:53.384 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:53.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:53.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:53.486 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:53.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:53.530 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:53.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:53.631 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:53.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:11:53.658 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:53.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:53.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:53.679 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:11:53.679 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:11:53.760 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:53.799 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:53.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:53.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:53.901 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:53.912 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:11:53.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:53 smithi136 ceph-mon[109861]: pgmap v2883: 99 pgs: 99 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 977 op/s 2024-08-17T18:11:53.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:53 smithi136 ceph-mon[115078]: pgmap v2883: 99 pgs: 99 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 977 op/s 2024-08-17T18:11:53.965 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:53.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:53.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:54.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:53 smithi165 ceph-mon[105517]: pgmap v2883: 99 pgs: 99 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 977 op/s 2024-08-17T18:11:54.067 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:54.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:54.113 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:54.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:54.214 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:54.256 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:54.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:54.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:54.358 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:54.394 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:54.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:54.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:54.496 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:54.530 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-17T18:11:54.555 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:54.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:54.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:54.657 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:54.704 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:54.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:54.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:54.806 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:54.842 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:54.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:54.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:54.854 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:11:54.920 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 6, 0, 5] out_osds: [7] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:11:54.920 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:11:54.921 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [1, 2, 3, 4, 6, 0, 5] 2024-08-17T18:11:54.921 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd out 1 2024-08-17T18:11:54.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:55.015 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:55.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:55.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:55.117 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:55.170 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:55.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:55.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:55.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:55.313 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:55.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:55.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:55.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:55.449 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:55.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:55.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:55.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:55.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:55.588 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:55.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:55.689 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:55.722 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:55.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:55.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:55.824 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:55.861 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:55.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:55.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:55.913 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:11:55.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:55 smithi136 ceph-mon[109861]: pgmap v2884: 99 pgs: 99 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 955 op/s 2024-08-17T18:11:55.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:55 smithi136 ceph-mon[115078]: pgmap v2884: 99 pgs: 99 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 955 op/s 2024-08-17T18:11:55.963 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:56.001 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:56.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:56.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:55 smithi165 ceph-mon[105517]: pgmap v2884: 99 pgs: 99 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 955 op/s 2024-08-17T18:11:56.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:56.135 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:56.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:56.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:56.237 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:56.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:56.285 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:56.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:56.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:56.426 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:56.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:56.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:56.528 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:56.555 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:56.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:56.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:56.657 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:56.704 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:56.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:56.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:56.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:56.838 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:56.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:56.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:56.924 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:11:56.924 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:11:56.940 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:56.982 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:56.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:56.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:57.084 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:57.109 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:57.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:57.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:57.211 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:57.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:57.252 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:57.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:57.353 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:57.388 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:57.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:57.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:57.475 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:11:57.476 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:11:57.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:11:57.510 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:11:57.511 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:11:57.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:57.527 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:57.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:11:57.552 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:11:57.552 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:11:57.595 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:11:57.596 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:11:57.620 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-17T18:11:57.628 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:57.630 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:11:57.631 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:11:57.650 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:11:57.650 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:11:57.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:57.653 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:57.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:57.666 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:11:57.666 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:11:57.667 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:57 smithi136 ceph-mon[109861]: pgmap v2885: 99 pgs: 99 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 955 op/s 2024-08-17T18:11:57.667 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:57 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1109314012' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:11:57.667 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:57 smithi136 ceph-mon[115078]: pgmap v2885: 99 pgs: 99 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 955 op/s 2024-08-17T18:11:57.667 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:57 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1109314012' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:11:57.667 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:11:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:11:57] "GET /metrics HTTP/1.1" 200 36610 "" "Prometheus/2.51.0" 2024-08-17T18:11:57.688 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:11:57.688 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:11:57.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:11:57.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:57.804 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:57.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:11:57.905 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:11:57.946 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:57.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:57.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:11:57.948 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:11:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:57 smithi165 ceph-mon[105517]: pgmap v2885: 99 pgs: 99 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 955 op/s 2024-08-17T18:11:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:57 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1109314012' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:11:58.048 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:58.085 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:58.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:58.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:58.186 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:58.225 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:58.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:58.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:58.328 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:58.365 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:58.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:58.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:58.467 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:11:58.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:58.508 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:58.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:11:58.609 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:58.643 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:58.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:58.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:58.745 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:58.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:58.782 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:58.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:58.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:58.919 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:58.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:58.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:59.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:59.051 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:59.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:59.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:59.154 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:59.189 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:59.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:59.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:59.291 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:11:59.330 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:59.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:59.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:11:59.432 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:59.474 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:59.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:59.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:59.576 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:11:59.611 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:11:59.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:59.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:11:59.713 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:11:59.749 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:59.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:59.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:11:59.830 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:11:59 smithi165 ceph-mon[105517]: pgmap v2886: 99 pgs: 99 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 951 op/s 2024-08-17T18:11:59.851 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:11:59.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:11:59.902 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:11:59.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:11:59.925 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:11:59 smithi136 ceph-mon[109861]: pgmap v2886: 99 pgs: 99 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 951 op/s 2024-08-17T18:11:59.926 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:11:59 smithi136 ceph-mon[115078]: pgmap v2886: 99 pgs: 99 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 951 op/s 2024-08-17T18:11:59.976 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:11:59.977 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:11:59.977 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-08-17T18:11:59.977 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-08-17T18:11:59.977 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:12:00.002 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:00.036 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:00.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:00.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:00.137 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:00.167 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:00.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:00.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:00.268 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:00.302 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:00.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:00.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:00.361 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:12:00.404 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:00.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:00.440 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:00.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:00.541 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:00.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:00.581 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:00.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:00.663 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:12:00.677 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-17T18:12:00.682 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:00.717 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:00.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:00.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:00.819 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:00.858 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:00.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:00.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:00.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:00 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/789139387' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:12:00.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:00 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:12:00.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:00 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/789139387' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:12:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:00 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:12:00.959 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:00.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:01.000 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:01.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:01.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:00 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/789139387' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:12:01.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:00 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:12:01.018 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:12:01.101 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:01.137 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:01.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:01.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:01.238 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:01.269 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:01.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:01.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:01.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:01.402 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:01.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:01.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:01.504 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:01.555 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:01.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:01.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:01.657 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:01.691 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:01.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:01.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:01.697 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:01 smithi165 ceph-mon[105517]: pgmap v2887: 99 pgs: 99 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 889 op/s 2024-08-17T18:12:01.697 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:01 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:12:01.697 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:01 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:12:01.697 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:01 smithi165 ceph-mon[105517]: osdmap e1552: 8 total, 8 up, 7 in 2024-08-17T18:12:01.792 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:01.821 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:01.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:01.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:01.923 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:01.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:01 smithi136 ceph-mon[109861]: pgmap v2887: 99 pgs: 99 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 889 op/s 2024-08-17T18:12:01.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:01 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:12:01.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:01 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:12:01.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:01 smithi136 ceph-mon[109861]: osdmap e1552: 8 total, 8 up, 7 in 2024-08-17T18:12:01.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:01 smithi136 ceph-mon[115078]: pgmap v2887: 99 pgs: 99 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 889 op/s 2024-08-17T18:12:01.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:01 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:12:01.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:01 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:12:01.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:01 smithi136 ceph-mon[115078]: osdmap e1552: 8 total, 8 up, 7 in 2024-08-17T18:12:01.953 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:01.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:01.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:02.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:02.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:02.099 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:02.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:02.200 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:02.242 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:02.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:02.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:02.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:02.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:02.376 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:02.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:02.398 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:12:02.477 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:02.506 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:02.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:02.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:02.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:02.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:02.644 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:02.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:02.745 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:02.781 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:02.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:02.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:02.882 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:02.918 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:02.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:02.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:03.020 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:03.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:03.068 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:03.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:03.169 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:03.197 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:12:03.198 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:03.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:03.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:03.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:03.345 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:03.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:03.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:03.389 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:12:03.389 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T18:12:03.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T18:12:03.389 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:12:03.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.389 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:225596 54.13 54:cd19ad8a:::benchmark_data_smithi136_273969_object102711:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1432)", 2024-08-17T18:12:03.389 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:02:44.066036+0000", 2024-08-17T18:12:03.390 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 559.32060116800005, 2024-08-17T18:12:03.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.43773147800000001, 2024-08-17T18:12:03.390 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:03.390 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:03.390 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:03.390 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:12:03.390 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:12:03.390 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 225596 2024-08-17T18:12:03.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.390 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:03.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:03.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.066036+0000", 2024-08-17T18:12:03.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:03.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.066036+0000", 2024-08-17T18:12:03.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:03.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.066035+0000", 2024-08-17T18:12:03.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:12:03.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:03.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.066044+0000", 2024-08-17T18:12:03.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.755e-06 2024-08-17T18:12:03.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:03.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.066059+0000", 2024-08-17T18:12:03.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.436e-05 2024-08-17T18:12:03.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:03.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.066074+0000", 2024-08-17T18:12:03.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4979e-05 2024-08-17T18:12:03.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.115216+0000", 2024-08-17T18:12:03.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.049142369999999998 2024-08-17T18:12:03.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:03.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.116943+0000", 2024-08-17T18:12:03.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001727541 2024-08-17T18:12:03.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.148100+0000", 2024-08-17T18:12:03.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.031156461 2024-08-17T18:12:03.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:03.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.150155+0000", 2024-08-17T18:12:03.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020547400000000002 2024-08-17T18:12:03.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.481627+0000", 2024-08-17T18:12:03.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.33147232100000001 2024-08-17T18:12:03.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:03.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.482760+0000", 2024-08-17T18:12:03.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011329160000000001 2024-08-17T18:12:03.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3,6", 2024-08-17T18:12:03.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.485322+0000", 2024-08-17T18:12:03.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002562099 2024-08-17T18:12:03.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.499491+0000", 2024-08-17T18:12:03.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014168912000000001 2024-08-17T18:12:03.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:03.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.499611+0000", 2024-08-17T18:12:03.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120238 2024-08-17T18:12:03.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.500077+0000", 2024-08-17T18:12:03.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00046617400000000003 2024-08-17T18:12:03.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.500485+0000", 2024-08-17T18:12:03.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00040769899999999997 2024-08-17T18:12:03.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:03.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.501117+0000", 2024-08-17T18:12:03.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063198699999999998 2024-08-17T18:12:03.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:03.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.503768+0000", 2024-08-17T18:12:03.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002650992 2024-08-17T18:12:03.398 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.398 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:03.398 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.398 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:225630 54.13 54:c9de4b4e:::benchmark_data_smithi136_273969_object102745:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1432)", 2024-08-17T18:12:03.398 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:02:44.156620+0000", 2024-08-17T18:12:03.398 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 559.23001779000003, 2024-08-17T18:12:03.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.37701020200000002, 2024-08-17T18:12:03.399 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:03.399 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:03.399 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:03.399 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:12:03.399 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:12:03.399 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 225630 2024-08-17T18:12:03.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.399 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:03.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:03.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.156620+0000", 2024-08-17T18:12:03.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:03.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.156620+0000", 2024-08-17T18:12:03.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:03.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.156618+0000", 2024-08-17T18:12:03.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:12:03.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:03.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.156626+0000", 2024-08-17T18:12:03.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5839999999999997e-06 2024-08-17T18:12:03.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:03.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.156638+0000", 2024-08-17T18:12:03.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2571e-05 2024-08-17T18:12:03.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:03.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.156644+0000", 2024-08-17T18:12:03.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7459999999999997e-06 2024-08-17T18:12:03.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.195120+0000", 2024-08-17T18:12:03.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038475580000000002 2024-08-17T18:12:03.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:03.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.196666+0000", 2024-08-17T18:12:03.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001545942 2024-08-17T18:12:03.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.225928+0000", 2024-08-17T18:12:03.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.029262326000000002 2024-08-17T18:12:03.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:03.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.227723+0000", 2024-08-17T18:12:03.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017948409999999999 2024-08-17T18:12:03.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.511497+0000", 2024-08-17T18:12:03.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.283773732 2024-08-17T18:12:03.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:03.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.512657+0000", 2024-08-17T18:12:03.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001160281 2024-08-17T18:12:03.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3,6", 2024-08-17T18:12:03.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.515816+0000", 2024-08-17T18:12:03.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031589090000000001 2024-08-17T18:12:03.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.521864+0000", 2024-08-17T18:12:03.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00604851 2024-08-17T18:12:03.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.522417+0000", 2024-08-17T18:12:03.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000552573 2024-08-17T18:12:03.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.522853+0000", 2024-08-17T18:12:03.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000435726 2024-08-17T18:12:03.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:03.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.530235+0000", 2024-08-17T18:12:03.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073823810000000004 2024-08-17T18:12:03.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:03.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.530870+0000", 2024-08-17T18:12:03.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063546999999999996 2024-08-17T18:12:03.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:03.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.533630+0000", 2024-08-17T18:12:03.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027595649999999998 2024-08-17T18:12:03.408 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.408 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:03.408 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.408 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:225708 54.13 54:c8de4c3f:::benchmark_data_smithi136_273969_object102823:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1432)", 2024-08-17T18:12:03.408 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:02:44.294046+0000", 2024-08-17T18:12:03.408 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 559.09259209300001, 2024-08-17T18:12:03.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.42225885600000002, 2024-08-17T18:12:03.409 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:03.409 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:03.409 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:03.409 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:12:03.409 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:12:03.409 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 225708 2024-08-17T18:12:03.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.409 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:03.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:03.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.294046+0000", 2024-08-17T18:12:03.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:03.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.294046+0000", 2024-08-17T18:12:03.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:03.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.294044+0000", 2024-08-17T18:12:03.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:12:03.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:03.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.294053+0000", 2024-08-17T18:12:03.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0149999999999992e-06 2024-08-17T18:12:03.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:03.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.294077+0000", 2024-08-17T18:12:03.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4702000000000002e-05 2024-08-17T18:12:03.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:03.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.294086+0000", 2024-08-17T18:12:03.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6580000000000008e-06 2024-08-17T18:12:03.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.338638+0000", 2024-08-17T18:12:03.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.044552302000000002 2024-08-17T18:12:03.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:03.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.340107+0000", 2024-08-17T18:12:03.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014688329999999999 2024-08-17T18:12:03.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.370415+0000", 2024-08-17T18:12:03.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.030307844 2024-08-17T18:12:03.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:03.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.372136+0000", 2024-08-17T18:12:03.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001720638 2024-08-17T18:12:03.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.693850+0000", 2024-08-17T18:12:03.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.32171407000000002 2024-08-17T18:12:03.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:03.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.695281+0000", 2024-08-17T18:12:03.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014316120000000001 2024-08-17T18:12:03.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3,6", 2024-08-17T18:12:03.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.698396+0000", 2024-08-17T18:12:03.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031149699999999999 2024-08-17T18:12:03.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.711946+0000", 2024-08-17T18:12:03.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013549863000000001 2024-08-17T18:12:03.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:03.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.712077+0000", 2024-08-17T18:12:03.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013130100000000001 2024-08-17T18:12:03.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.712686+0000", 2024-08-17T18:12:03.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00060844200000000005 2024-08-17T18:12:03.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.713101+0000", 2024-08-17T18:12:03.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00041545800000000001 2024-08-17T18:12:03.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:03.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.713746+0000", 2024-08-17T18:12:03.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064484600000000003 2024-08-17T18:12:03.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:03.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.716304+0000", 2024-08-17T18:12:03.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00255821 2024-08-17T18:12:03.417 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.417 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:03.417 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.417 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:225726 54.13 54:cffbfbfe:::benchmark_data_smithi136_273969_object102841:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1432)", 2024-08-17T18:12:03.417 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:02:44.373017+0000", 2024-08-17T18:12:03.417 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 559.01362036700004, 2024-08-17T18:12:03.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.37556931799999999, 2024-08-17T18:12:03.417 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:03.417 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:03.417 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:03.418 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:12:03.418 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:12:03.418 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 225726 2024-08-17T18:12:03.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.418 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:03.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:03.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.373017+0000", 2024-08-17T18:12:03.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:03.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.373017+0000", 2024-08-17T18:12:03.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:03.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.373016+0000", 2024-08-17T18:12:03.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:12:03.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:03.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.373023+0000", 2024-08-17T18:12:03.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7259999999999996e-06 2024-08-17T18:12:03.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:03.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.373034+0000", 2024-08-17T18:12:03.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1188e-05 2024-08-17T18:12:03.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:03.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.373041+0000", 2024-08-17T18:12:03.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4989999999999997e-06 2024-08-17T18:12:03.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.377568+0000", 2024-08-17T18:12:03.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045265230000000002 2024-08-17T18:12:03.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:03.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.379031+0000", 2024-08-17T18:12:03.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014636060000000001 2024-08-17T18:12:03.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.399243+0000", 2024-08-17T18:12:03.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020211432000000001 2024-08-17T18:12:03.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:03.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.400699+0000", 2024-08-17T18:12:03.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014560059999999999 2024-08-17T18:12:03.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.720106+0000", 2024-08-17T18:12:03.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31940689900000002 2024-08-17T18:12:03.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:03.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.721491+0000", 2024-08-17T18:12:03.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013853730000000001 2024-08-17T18:12:03.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3,6", 2024-08-17T18:12:03.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.724408+0000", 2024-08-17T18:12:03.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029167329999999999 2024-08-17T18:12:03.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:03.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.742582+0000", 2024-08-17T18:12:03.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018174625999999999 2024-08-17T18:12:03.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.743209+0000", 2024-08-17T18:12:03.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062605199999999999 2024-08-17T18:12:03.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.743694+0000", 2024-08-17T18:12:03.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048579299999999999 2024-08-17T18:12:03.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.744395+0000", 2024-08-17T18:12:03.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00070037500000000004 2024-08-17T18:12:03.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:03.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.745057+0000", 2024-08-17T18:12:03.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00066246800000000004 2024-08-17T18:12:03.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:03.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.748587+0000", 2024-08-17T18:12:03.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035294649999999999 2024-08-17T18:12:03.426 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.426 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:03.426 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.426 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:225736 54.13 54:cb56deac:::benchmark_data_smithi136_273969_object102851:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1432)", 2024-08-17T18:12:03.426 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:02:44.403924+0000", 2024-08-17T18:12:03.426 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 558.982713633, 2024-08-17T18:12:03.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.38515698500000001, 2024-08-17T18:12:03.426 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:03.426 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:03.426 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:03.427 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:12:03.427 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:12:03.427 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 225736 2024-08-17T18:12:03.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.427 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:03.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:03.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.403924+0000", 2024-08-17T18:12:03.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:03.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.403924+0000", 2024-08-17T18:12:03.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:03.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.403924+0000", 2024-08-17T18:12:03.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:12:03.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:03.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.403926+0000", 2024-08-17T18:12:03.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8220000000000001e-06 2024-08-17T18:12:03.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:03.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.403930+0000", 2024-08-17T18:12:03.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5789999999999997e-06 2024-08-17T18:12:03.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:03.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.403933+0000", 2024-08-17T18:12:03.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0249999999999998e-06 2024-08-17T18:12:03.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.426304+0000", 2024-08-17T18:12:03.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022370788999999999 2024-08-17T18:12:03.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:03.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.427874+0000", 2024-08-17T18:12:03.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015702019999999999 2024-08-17T18:12:03.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.443230+0000", 2024-08-17T18:12:03.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015356025000000001 2024-08-17T18:12:03.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:03.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.444629+0000", 2024-08-17T18:12:03.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013990039999999999 2024-08-17T18:12:03.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.759686+0000", 2024-08-17T18:12:03.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31505686700000002 2024-08-17T18:12:03.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:03.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.761111+0000", 2024-08-17T18:12:03.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001424841 2024-08-17T18:12:03.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3,6", 2024-08-17T18:12:03.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.764411+0000", 2024-08-17T18:12:03.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033003540000000001 2024-08-17T18:12:03.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.784096+0000", 2024-08-17T18:12:03.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019684994000000001 2024-08-17T18:12:03.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:03.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.784248+0000", 2024-08-17T18:12:03.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000152206 2024-08-17T18:12:03.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.784722+0000", 2024-08-17T18:12:03.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00047387699999999999 2024-08-17T18:12:03.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.785172+0000", 2024-08-17T18:12:03.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00044934499999999997 2024-08-17T18:12:03.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:03.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.785935+0000", 2024-08-17T18:12:03.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00076316899999999996 2024-08-17T18:12:03.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:03.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.789081+0000", 2024-08-17T18:12:03.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031461380000000001 2024-08-17T18:12:03.434 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.434 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:03.434 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.435 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:225741 54.13 54:cf633abf:::benchmark_data_smithi136_273969_object102856:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1432)", 2024-08-17T18:12:03.435 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:02:44.423047+0000", 2024-08-17T18:12:03.435 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 558.96359049800003, 2024-08-17T18:12:03.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.37510067600000002, 2024-08-17T18:12:03.435 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:03.435 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:03.435 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:03.435 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:12:03.435 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:12:03.435 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 225741 2024-08-17T18:12:03.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.436 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:03.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:03.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.423047+0000", 2024-08-17T18:12:03.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:03.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.423047+0000", 2024-08-17T18:12:03.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:03.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.423045+0000", 2024-08-17T18:12:03.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:12:03.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:03.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.423052+0000", 2024-08-17T18:12:03.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.99e-06 2024-08-17T18:12:03.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:03.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.423065+0000", 2024-08-17T18:12:03.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2388000000000001e-05 2024-08-17T18:12:03.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:03.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.423080+0000", 2024-08-17T18:12:03.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5319999999999999e-05 2024-08-17T18:12:03.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.445280+0000", 2024-08-17T18:12:03.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022200280999999999 2024-08-17T18:12:03.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:03.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.446651+0000", 2024-08-17T18:12:03.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013706230000000001 2024-08-17T18:12:03.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.458226+0000", 2024-08-17T18:12:03.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011575383999999999 2024-08-17T18:12:03.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:03.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.459558+0000", 2024-08-17T18:12:03.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001331392 2024-08-17T18:12:03.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.774045+0000", 2024-08-17T18:12:03.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.31448730400000002 2024-08-17T18:12:03.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:03.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.775473+0000", 2024-08-17T18:12:03.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014280289999999999 2024-08-17T18:12:03.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3,6", 2024-08-17T18:12:03.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.778691+0000", 2024-08-17T18:12:03.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032179330000000001 2024-08-17T18:12:03.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.789601+0000", 2024-08-17T18:12:03.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010909785999999999 2024-08-17T18:12:03.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.790045+0000", 2024-08-17T18:12:03.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00044432599999999999 2024-08-17T18:12:03.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.790493+0000", 2024-08-17T18:12:03.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00044791600000000003 2024-08-17T18:12:03.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:03.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.794118+0000", 2024-08-17T18:12:03.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036250330000000002 2024-08-17T18:12:03.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:03.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.794888+0000", 2024-08-17T18:12:03.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00077017100000000005 2024-08-17T18:12:03.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:03.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.798148+0000", 2024-08-17T18:12:03.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032597479999999998 2024-08-17T18:12:03.443 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.443 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:03.443 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.444 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:225746 54.13 54:cd31b559:::benchmark_data_smithi136_273969_object102861:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1432)", 2024-08-17T18:12:03.444 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:02:44.434456+0000", 2024-08-17T18:12:03.444 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 558.95218142199997, 2024-08-17T18:12:03.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39569336100000002, 2024-08-17T18:12:03.444 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:03.444 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:03.444 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:03.444 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:12:03.444 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:12:03.444 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 225746 2024-08-17T18:12:03.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.444 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:03.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:03.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.434456+0000", 2024-08-17T18:12:03.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:03.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.434456+0000", 2024-08-17T18:12:03.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:03.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.434454+0000", 2024-08-17T18:12:03.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:12:03.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:03.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.434463+0000", 2024-08-17T18:12:03.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8680000000000005e-06 2024-08-17T18:12:03.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:03.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.434480+0000", 2024-08-17T18:12:03.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6439e-05 2024-08-17T18:12:03.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:03.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.434491+0000", 2024-08-17T18:12:03.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1313e-05 2024-08-17T18:12:03.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.446806+0000", 2024-08-17T18:12:03.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012314899000000001 2024-08-17T18:12:03.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:03.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.448405+0000", 2024-08-17T18:12:03.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015990850000000001 2024-08-17T18:12:03.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.466879+0000", 2024-08-17T18:12:03.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018474411 2024-08-17T18:12:03.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:03.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.468269+0000", 2024-08-17T18:12:03.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001389248 2024-08-17T18:12:03.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.798379+0000", 2024-08-17T18:12:03.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.33011053800000001 2024-08-17T18:12:03.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:03.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.799956+0000", 2024-08-17T18:12:03.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001576556 2024-08-17T18:12:03.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3,6", 2024-08-17T18:12:03.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.803389+0000", 2024-08-17T18:12:03.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003433251 2024-08-17T18:12:03.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:03.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.820881+0000", 2024-08-17T18:12:03.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017492137000000001 2024-08-17T18:12:03.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.821430+0000", 2024-08-17T18:12:03.450 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:03.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00054860300000000005 2024-08-17T18:12:03.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.824675+0000", 2024-08-17T18:12:03.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032452449999999999 2024-08-17T18:12:03.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.825554+0000", 2024-08-17T18:12:03.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00087920299999999995 2024-08-17T18:12:03.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:03.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.826348+0000", 2024-08-17T18:12:03.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00079352000000000003 2024-08-17T18:12:03.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:03.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.830150+0000", 2024-08-17T18:12:03.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003801914 2024-08-17T18:12:03.454 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.454 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:03.454 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.454 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:225749 54.13 54:ce590af3:::benchmark_data_smithi136_273969_object102864:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1432)", 2024-08-17T18:12:03.454 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:02:44.444581+0000", 2024-08-17T18:12:03.455 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 558.94205648900004, 2024-08-17T18:12:03.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39119509699999999, 2024-08-17T18:12:03.455 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:03.455 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:03.455 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:03.455 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:12:03.455 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:12:03.455 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 225749 2024-08-17T18:12:03.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.455 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:03.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:03.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.444581+0000", 2024-08-17T18:12:03.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:03.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.444581+0000", 2024-08-17T18:12:03.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:03.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.444579+0000", 2024-08-17T18:12:03.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:12:03.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:03.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.444588+0000", 2024-08-17T18:12:03.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0609999999999996e-06 2024-08-17T18:12:03.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:03.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.444603+0000", 2024-08-17T18:12:03.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5238e-05 2024-08-17T18:12:03.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:03.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.444611+0000", 2024-08-17T18:12:03.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.473e-06 2024-08-17T18:12:03.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.449053+0000", 2024-08-17T18:12:03.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044420450000000004 2024-08-17T18:12:03.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:03.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.450581+0000", 2024-08-17T18:12:03.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015280789999999999 2024-08-17T18:12:03.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.475806+0000", 2024-08-17T18:12:03.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.025224628999999998 2024-08-17T18:12:03.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:03.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.477168+0000", 2024-08-17T18:12:03.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013618580000000001 2024-08-17T18:12:03.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.811953+0000", 2024-08-17T18:12:03.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.33478503799999998 2024-08-17T18:12:03.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:03.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.813244+0000", 2024-08-17T18:12:03.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001291267 2024-08-17T18:12:03.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3,6", 2024-08-17T18:12:03.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.816563+0000", 2024-08-17T18:12:03.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033184249999999998 2024-08-17T18:12:03.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:03.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.825055+0000", 2024-08-17T18:12:03.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0084921990000000006 2024-08-17T18:12:03.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.830683+0000", 2024-08-17T18:12:03.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056282499999999996 2024-08-17T18:12:03.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.831176+0000", 2024-08-17T18:12:03.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00049274300000000002 2024-08-17T18:12:03.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.831718+0000", 2024-08-17T18:12:03.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00054253999999999999 2024-08-17T18:12:03.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:03.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.832515+0000", 2024-08-17T18:12:03.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00079617299999999995 2024-08-17T18:12:03.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:03.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.835776+0000", 2024-08-17T18:12:03.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003261758 2024-08-17T18:12:03.466 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.466 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:03.466 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.467 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35370.0:225751 54.13 54:cee3306f:::benchmark_data_smithi136_273969_object102866:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1432)", 2024-08-17T18:12:03.467 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:02:44.505436+0000", 2024-08-17T18:12:03.467 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 558.88120171200001, 2024-08-17T18:12:03.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.36635309100000002, 2024-08-17T18:12:03.467 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:03.467 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:03.467 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:03.467 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35370", 2024-08-17T18:12:03.467 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3357040294", 2024-08-17T18:12:03.467 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 225751 2024-08-17T18:12:03.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.468 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:03.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:03.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.505436+0000", 2024-08-17T18:12:03.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:03.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.505436+0000", 2024-08-17T18:12:03.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:03.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.505435+0000", 2024-08-17T18:12:03.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:12:03.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:03.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.505443+0000", 2024-08-17T18:12:03.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7439999999999996e-06 2024-08-17T18:12:03.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:03.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.505456+0000", 2024-08-17T18:12:03.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2678e-05 2024-08-17T18:12:03.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:03.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.505462+0000", 2024-08-17T18:12:03.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4600000000000002e-06 2024-08-17T18:12:03.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.519881+0000", 2024-08-17T18:12:03.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014419566 2024-08-17T18:12:03.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:03.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.521440+0000", 2024-08-17T18:12:03.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015584170000000001 2024-08-17T18:12:03.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.533833+0000", 2024-08-17T18:12:03.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012393397 2024-08-17T18:12:03.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:03.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.535346+0000", 2024-08-17T18:12:03.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001512858 2024-08-17T18:12:03.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.841962+0000", 2024-08-17T18:12:03.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.30661596299999999 2024-08-17T18:12:03.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:03.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.843285+0000", 2024-08-17T18:12:03.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013231060000000001 2024-08-17T18:12:03.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3,6", 2024-08-17T18:12:03.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.847874+0000", 2024-08-17T18:12:03.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045894940000000004 2024-08-17T18:12:03.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.863945+0000", 2024-08-17T18:12:03.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016070404 2024-08-17T18:12:03.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.864429+0000", 2024-08-17T18:12:03.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048449799999999998 2024-08-17T18:12:03.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.864862+0000", 2024-08-17T18:12:03.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00043241500000000001 2024-08-17T18:12:03.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:03.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.867624+0000", 2024-08-17T18:12:03.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002762745 2024-08-17T18:12:03.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:03.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.868495+0000", 2024-08-17T18:12:03.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00087013399999999995 2024-08-17T18:12:03.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:03.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:02:44.871789+0000", 2024-08-17T18:12:03.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032944670000000001 2024-08-17T18:12:03.477 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.477 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:03.477 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.477 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:39666 55.d 55:b581e00a:::benchmark_data_smithi136_313476_object39665:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1456)", 2024-08-17T18:12:03.477 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:04:14.320291+0000", 2024-08-17T18:12:03.477 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 469.06634681700001, 2024-08-17T18:12:03.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6548982109999999, 2024-08-17T18:12:03.478 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:03.478 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:12:03.478 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:03.478 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:12:03.478 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:12:03.478 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 39666 2024-08-17T18:12:03.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.478 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:03.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:03.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.320291+0000", 2024-08-17T18:12:03.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:03.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.320291+0000", 2024-08-17T18:12:03.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:03.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.320290+0000", 2024-08-17T18:12:03.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:12:03.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:03.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.320344+0000", 2024-08-17T18:12:03.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3773999999999997e-05 2024-08-17T18:12:03.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:03.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.320353+0000", 2024-08-17T18:12:03.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.73e-06 2024-08-17T18:12:03.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:03.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.320359+0000", 2024-08-17T18:12:03.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9549999999999997e-06 2024-08-17T18:12:03.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.320390+0000", 2024-08-17T18:12:03.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0771000000000001e-05 2024-08-17T18:12:03.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:03.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.320492+0000", 2024-08-17T18:12:03.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010220199999999999 2024-08-17T18:12:03.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-08-17T18:12:03.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.320602+0000", 2024-08-17T18:12:03.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109413 2024-08-17T18:12:03.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:03.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.322056+0000", 2024-08-17T18:12:03.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014548129999999999 2024-08-17T18:12:03.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.322909+0000", 2024-08-17T18:12:03.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00085237900000000001 2024-08-17T18:12:03.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.336838+0000", 2024-08-17T18:12:03.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013928779000000001 2024-08-17T18:12:03.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:12:03.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.336873+0000", 2024-08-17T18:12:03.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5371999999999999e-05 2024-08-17T18:12:03.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.518312+0000", 2024-08-17T18:12:03.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.181438549 2024-08-17T18:12:03.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:12:03.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:14.518324+0000", 2024-08-17T18:12:03.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.237e-05 2024-08-17T18:12:03.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.975073+0000", 2024-08-17T18:12:03.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.456749498 2024-08-17T18:12:03.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:03.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:04:15.975189+0000", 2024-08-17T18:12:03.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011560799999999999 2024-08-17T18:12:03.486 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.486 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:03.486 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.487 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:152972 56.b 56:df4add57:::benchmark_data_smithi136_356456_object26268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1549)", 2024-08-17T18:12:03.487 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:11:37.099913+0000", 2024-08-17T18:12:03.487 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 26.286724284999998, 2024-08-17T18:12:03.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.040421569999999997, 2024-08-17T18:12:03.487 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:03.487 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:03.487 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:03.487 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:12:03.487 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:12:03.487 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 152972 2024-08-17T18:12:03.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.488 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:03.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:03.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.099913+0000", 2024-08-17T18:12:03.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:03.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.099913+0000", 2024-08-17T18:12:03.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:03.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.099913+0000", 2024-08-17T18:12:03.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:12:03.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:03.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.099916+0000", 2024-08-17T18:12:03.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0400000000000001e-06 2024-08-17T18:12:03.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:03.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.099951+0000", 2024-08-17T18:12:03.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5345000000000001e-05 2024-08-17T18:12:03.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:03.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.099959+0000", 2024-08-17T18:12:03.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8650000000000001e-06 2024-08-17T18:12:03.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.103685+0000", 2024-08-17T18:12:03.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037260739999999998 2024-08-17T18:12:03.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:03.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.103779+0000", 2024-08-17T18:12:03.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4203999999999995e-05 2024-08-17T18:12:03.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-08-17T18:12:03.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.103855+0000", 2024-08-17T18:12:03.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5801000000000002e-05 2024-08-17T18:12:03.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:03.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.113098+0000", 2024-08-17T18:12:03.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0092428709999999997 2024-08-17T18:12:03.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.113496+0000", 2024-08-17T18:12:03.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039804199999999998 2024-08-17T18:12:03.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.140270+0000", 2024-08-17T18:12:03.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.026773617 2024-08-17T18:12:03.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:03.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.140307+0000", 2024-08-17T18:12:03.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6908e-05 2024-08-17T18:12:03.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:03.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.140335+0000", 2024-08-17T18:12:03.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8303999999999999e-05 2024-08-17T18:12:03.494 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.494 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:03.494 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.494 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:152973 56.b 56:d733f9b8:::benchmark_data_smithi136_356456_object26269:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1549)", 2024-08-17T18:12:03.494 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:11:37.099974+0000", 2024-08-17T18:12:03.494 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 26.286663364999999, 2024-08-17T18:12:03.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.040554212999999999, 2024-08-17T18:12:03.495 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:03.495 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:03.495 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:03.495 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:12:03.495 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:12:03.495 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 152973 2024-08-17T18:12:03.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.495 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:03.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:03.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.099974+0000", 2024-08-17T18:12:03.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:03.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.099974+0000", 2024-08-17T18:12:03.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:03.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.099974+0000", 2024-08-17T18:12:03.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:12:03.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:03.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.099978+0000", 2024-08-17T18:12:03.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9199999999999997e-06 2024-08-17T18:12:03.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:03.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.099986+0000", 2024-08-17T18:12:03.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3270000000000001e-06 2024-08-17T18:12:03.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:03.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.099991+0000", 2024-08-17T18:12:03.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3739999999999997e-06 2024-08-17T18:12:03.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.106921+0000", 2024-08-17T18:12:03.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069300439999999998 2024-08-17T18:12:03.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:03.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.107031+0000", 2024-08-17T18:12:03.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010980300000000001 2024-08-17T18:12:03.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-08-17T18:12:03.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.107147+0000", 2024-08-17T18:12:03.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011537800000000001 2024-08-17T18:12:03.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:03.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.113105+0000", 2024-08-17T18:12:03.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059584989999999999 2024-08-17T18:12:03.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.113542+0000", 2024-08-17T18:12:03.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00043656000000000001 2024-08-17T18:12:03.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.140403+0000", 2024-08-17T18:12:03.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.026860927 2024-08-17T18:12:03.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:03.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.140451+0000", 2024-08-17T18:12:03.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8124999999999997e-05 2024-08-17T18:12:03.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:03.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:37.140529+0000", 2024-08-17T18:12:03.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7782999999999997e-05 2024-08-17T18:12:03.501 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.501 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:03.501 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.502 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:172853 56.b 56:dfbc367a:::benchmark_data_smithi136_356456_object46149:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1551)", 2024-08-17T18:12:03.502 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:11:57.635146+0000", 2024-08-17T18:12:03.502 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.7514912630000001, 2024-08-17T18:12:03.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.053694954000000003, 2024-08-17T18:12:03.502 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:03.502 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:03.502 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:03.502 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:12:03.502 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:12:03.503 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 172853 2024-08-17T18:12:03.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.503 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:03.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:03.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.635146+0000", 2024-08-17T18:12:03.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:03.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.635146+0000", 2024-08-17T18:12:03.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:03.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.635145+0000", 2024-08-17T18:12:03.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:12:03.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:03.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.635152+0000", 2024-08-17T18:12:03.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.221e-06 2024-08-17T18:12:03.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:03.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.635166+0000", 2024-08-17T18:12:03.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3424999999999999e-05 2024-08-17T18:12:03.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:03.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.635172+0000", 2024-08-17T18:12:03.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3890000000000002e-06 2024-08-17T18:12:03.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.635211+0000", 2024-08-17T18:12:03.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8856000000000002e-05 2024-08-17T18:12:03.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:03.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.635303+0000", 2024-08-17T18:12:03.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2157999999999994e-05 2024-08-17T18:12:03.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-08-17T18:12:03.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.635384+0000", 2024-08-17T18:12:03.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1279000000000006e-05 2024-08-17T18:12:03.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:03.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.638392+0000", 2024-08-17T18:12:03.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030071730000000001 2024-08-17T18:12:03.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.638689+0000", 2024-08-17T18:12:03.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029701499999999997 2024-08-17T18:12:03.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.688755+0000", 2024-08-17T18:12:03.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.050066345999999998 2024-08-17T18:12:03.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:03.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.688803+0000", 2024-08-17T18:12:03.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7537000000000002e-05 2024-08-17T18:12:03.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:03.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.688841+0000", 2024-08-17T18:12:03.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8818e-05 2024-08-17T18:12:03.509 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.509 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:03.509 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.509 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:172938 56.b 56:dc6934f6:::benchmark_data_smithi136_356456_object46234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1551)", 2024-08-17T18:12:03.509 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:11:57.768450+0000", 2024-08-17T18:12:03.509 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.6181879370000001, 2024-08-17T18:12:03.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.029051717000000001, 2024-08-17T18:12:03.510 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:03.510 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:03.510 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:03.510 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:12:03.510 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:12:03.510 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 172938 2024-08-17T18:12:03.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.510 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:03.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:03.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.768450+0000", 2024-08-17T18:12:03.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:03.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.768450+0000", 2024-08-17T18:12:03.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:03.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.768448+0000", 2024-08-17T18:12:03.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:12:03.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:03.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.768456+0000", 2024-08-17T18:12:03.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7780000000000007e-06 2024-08-17T18:12:03.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:03.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.768469+0000", 2024-08-17T18:12:03.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3346999999999999e-05 2024-08-17T18:12:03.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:03.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.768481+0000", 2024-08-17T18:12:03.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1926999999999999e-05 2024-08-17T18:12:03.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.768603+0000", 2024-08-17T18:12:03.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012255499999999999 2024-08-17T18:12:03.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.514 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:03.514 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.768725+0000", 2024-08-17T18:12:03.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0001215 2024-08-17T18:12:03.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.514 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-08-17T18:12:03.514 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.768896+0000", 2024-08-17T18:12:03.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017111200000000001 2024-08-17T18:12:03.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.785685+0000", 2024-08-17T18:12:03.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016788897000000001 2024-08-17T18:12:03.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.792339+0000", 2024-08-17T18:12:03.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066542249999999997 2024-08-17T18:12:03.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:03.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.797343+0000", 2024-08-17T18:12:03.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050038399999999999 2024-08-17T18:12:03.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:03.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.797444+0000", 2024-08-17T18:12:03.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000101017 2024-08-17T18:12:03.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:03.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.797501+0000", 2024-08-17T18:12:03.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7368999999999999e-05 2024-08-17T18:12:03.517 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.517 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:03.517 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.517 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:172942 56.b 56:d879eae0:::benchmark_data_smithi136_356456_object46238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1551)", 2024-08-17T18:12:03.517 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:11:57.768847+0000", 2024-08-17T18:12:03.517 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.6177909990000003, 2024-08-17T18:12:03.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.02876679, 2024-08-17T18:12:03.517 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:03.517 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:03.518 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:03.518 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:12:03.518 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:12:03.518 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 172942 2024-08-17T18:12:03.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.518 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:03.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:03.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.768847+0000", 2024-08-17T18:12:03.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:03.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.768847+0000", 2024-08-17T18:12:03.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:03.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.768846+0000", 2024-08-17T18:12:03.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:12:03.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:03.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.768851+0000", 2024-08-17T18:12:03.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0250000000000002e-06 2024-08-17T18:12:03.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:03.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.768859+0000", 2024-08-17T18:12:03.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9190000000000005e-06 2024-08-17T18:12:03.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:03.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.768865+0000", 2024-08-17T18:12:03.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0310000000000004e-06 2024-08-17T18:12:03.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.776392+0000", 2024-08-17T18:12:03.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0075265269999999999 2024-08-17T18:12:03.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:03.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.776515+0000", 2024-08-17T18:12:03.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012254599999999999 2024-08-17T18:12:03.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-08-17T18:12:03.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.776633+0000", 2024-08-17T18:12:03.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118658 2024-08-17T18:12:03.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.785719+0000", 2024-08-17T18:12:03.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0090862670000000003 2024-08-17T18:12:03.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.792559+0000", 2024-08-17T18:12:03.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068391820000000001 2024-08-17T18:12:03.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:03.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.797512+0000", 2024-08-17T18:12:03.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049532079999999997 2024-08-17T18:12:03.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:03.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.797579+0000", 2024-08-17T18:12:03.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7504999999999998e-05 2024-08-17T18:12:03.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:03.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.797613+0000", 2024-08-17T18:12:03.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4071e-05 2024-08-17T18:12:03.524 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.524 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:03.524 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.525 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:173118 56.b 56:dcb30507:::benchmark_data_smithi136_356456_object46414:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1551)", 2024-08-17T18:12:03.525 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:11:57.948328+0000", 2024-08-17T18:12:03.525 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.4383096220000002, 2024-08-17T18:12:03.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.030034600000000002, 2024-08-17T18:12:03.525 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:03.525 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:03.525 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:03.525 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:12:03.525 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:12:03.525 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 173118 2024-08-17T18:12:03.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.526 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:03.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:03.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.948328+0000", 2024-08-17T18:12:03.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:03.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.948328+0000", 2024-08-17T18:12:03.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:03.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.948327+0000", 2024-08-17T18:12:03.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:12:03.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:03.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.948333+0000", 2024-08-17T18:12:03.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2260000000000002e-06 2024-08-17T18:12:03.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:03.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.948345+0000", 2024-08-17T18:12:03.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2452999999999999e-05 2024-08-17T18:12:03.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:03.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.948353+0000", 2024-08-17T18:12:03.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.374e-06 2024-08-17T18:12:03.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.951032+0000", 2024-08-17T18:12:03.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026792280000000001 2024-08-17T18:12:03.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:03.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.951178+0000", 2024-08-17T18:12:03.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014574799999999999 2024-08-17T18:12:03.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-08-17T18:12:03.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.951298+0000", 2024-08-17T18:12:03.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012044099999999999 2024-08-17T18:12:03.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:03.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.962985+0000", 2024-08-17T18:12:03.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011687105 2024-08-17T18:12:03.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.963164+0000", 2024-08-17T18:12:03.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017839200000000001 2024-08-17T18:12:03.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.978236+0000", 2024-08-17T18:12:03.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015071953000000001 2024-08-17T18:12:03.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:03.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.978322+0000", 2024-08-17T18:12:03.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6617000000000001e-05 2024-08-17T18:12:03.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:03.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:57.978363+0000", 2024-08-17T18:12:03.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0509000000000001e-05 2024-08-17T18:12:03.532 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.532 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:03.532 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.532 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:173257 56.b 56:dcd4a208:::benchmark_data_smithi136_356456_object46553:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1551)", 2024-08-17T18:12:03.532 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:11:58.080532+0000", 2024-08-17T18:12:03.532 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.3061055540000002, 2024-08-17T18:12:03.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.029750685999999998, 2024-08-17T18:12:03.533 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:03.533 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:03.533 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:03.533 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:12:03.533 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:12:03.533 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 173257 2024-08-17T18:12:03.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.533 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:03.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:03.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:58.080532+0000", 2024-08-17T18:12:03.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:03.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:58.080532+0000", 2024-08-17T18:12:03.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:03.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:58.080531+0000", 2024-08-17T18:12:03.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:12:03.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:03.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:58.080537+0000", 2024-08-17T18:12:03.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8780000000000003e-06 2024-08-17T18:12:03.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:03.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:58.080553+0000", 2024-08-17T18:12:03.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.592e-05 2024-08-17T18:12:03.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:03.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:58.080560+0000", 2024-08-17T18:12:03.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3499999999999999e-06 2024-08-17T18:12:03.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:58.086582+0000", 2024-08-17T18:12:03.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060219699999999998 2024-08-17T18:12:03.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:03.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:58.086727+0000", 2024-08-17T18:12:03.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014531900000000001 2024-08-17T18:12:03.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-08-17T18:12:03.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:58.086848+0000", 2024-08-17T18:12:03.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012056 2024-08-17T18:12:03.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:03.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:58.089586+0000", 2024-08-17T18:12:03.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027382169999999998 2024-08-17T18:12:03.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:58.093907+0000", 2024-08-17T18:12:03.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004320656 2024-08-17T18:12:03.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:58.110187+0000", 2024-08-17T18:12:03.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01628077 2024-08-17T18:12:03.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:03.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:58.110243+0000", 2024-08-17T18:12:03.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5875000000000003e-05 2024-08-17T18:12:03.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:03.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:11:58.110283+0000", 2024-08-17T18:12:03.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9478999999999999e-05 2024-08-17T18:12:03.540 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.540 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:03.541 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.541 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46025.0:176310 56.7 e1552/1535 56:ed6fc98e:::benchmark_data_smithi136_356456_object49606:head v 1552'10900, mlcod=1552'10898)", 2024-08-17T18:12:03.541 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:01.169323+0000", 2024-08-17T18:12:03.541 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.2173145280000002, 2024-08-17T18:12:03.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.028459079000000002, 2024-08-17T18:12:03.541 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:03.541 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:03.541 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:03.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:03.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.169323+0000", 2024-08-17T18:12:03.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:03.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.169323+0000", 2024-08-17T18:12:03.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:03.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.169322+0000", 2024-08-17T18:12:03.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:12:03.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:03.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.169328+0000", 2024-08-17T18:12:03.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4970000000000001e-06 2024-08-17T18:12:03.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:03.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.169337+0000", 2024-08-17T18:12:03.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.166e-06 2024-08-17T18:12:03.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:03.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.169341+0000", 2024-08-17T18:12:03.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3590000000000001e-06 2024-08-17T18:12:03.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.169379+0000", 2024-08-17T18:12:03.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7410000000000003e-05 2024-08-17T18:12:03.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:03.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.169398+0000", 2024-08-17T18:12:03.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9360000000000001e-05 2024-08-17T18:12:03.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:03.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.197722+0000", 2024-08-17T18:12:03.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.028323457999999999 2024-08-17T18:12:03.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:03.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.197782+0000", 2024-08-17T18:12:03.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.054e-05 2024-08-17T18:12:03.546 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.546 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:03.546 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.546 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:176316 56.b 56:d57233da:::benchmark_data_smithi136_356456_object49612:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1552)", 2024-08-17T18:12:03.546 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:01.172048+0000", 2024-08-17T18:12:03.547 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.2145897419999998, 2024-08-17T18:12:03.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.043191457000000003, 2024-08-17T18:12:03.547 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:03.547 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:03.547 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:03.547 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:12:03.547 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:12:03.547 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 176316 2024-08-17T18:12:03.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.547 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:03.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:03.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.172048+0000", 2024-08-17T18:12:03.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:03.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.172048+0000", 2024-08-17T18:12:03.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:03.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.172046+0000", 2024-08-17T18:12:03.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:12:03.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:03.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.172052+0000", 2024-08-17T18:12:03.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.305e-06 2024-08-17T18:12:03.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:03.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.172078+0000", 2024-08-17T18:12:03.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6458999999999999e-05 2024-08-17T18:12:03.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:03.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.172084+0000", 2024-08-17T18:12:03.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0159999999999997e-06 2024-08-17T18:12:03.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.172134+0000", 2024-08-17T18:12:03.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9495e-05 2024-08-17T18:12:03.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:03.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.172262+0000", 2024-08-17T18:12:03.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012809799999999999 2024-08-17T18:12:03.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-08-17T18:12:03.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.172379+0000", 2024-08-17T18:12:03.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000117093 2024-08-17T18:12:03.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:03.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.203168+0000", 2024-08-17T18:12:03.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.030789440000000001 2024-08-17T18:12:03.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.203227+0000", 2024-08-17T18:12:03.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8984999999999999e-05 2024-08-17T18:12:03.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.215155+0000", 2024-08-17T18:12:03.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011927577 2024-08-17T18:12:03.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:03.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.215202+0000", 2024-08-17T18:12:03.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7280000000000001e-05 2024-08-17T18:12:03.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:03.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.215239+0000", 2024-08-17T18:12:03.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7159000000000001e-05 2024-08-17T18:12:03.554 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.554 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:03.554 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.554 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:176318 56.b 56:d8c20bfe:::benchmark_data_smithi136_356456_object49614:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1552)", 2024-08-17T18:12:03.554 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:01.174269+0000", 2024-08-17T18:12:03.554 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.212368723, 2024-08-17T18:12:03.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.041121447999999998, 2024-08-17T18:12:03.555 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:03.555 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:03.555 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:03.555 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:12:03.555 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:12:03.555 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 176318 2024-08-17T18:12:03.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.555 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:03.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:03.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.174269+0000", 2024-08-17T18:12:03.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:03.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.174269+0000", 2024-08-17T18:12:03.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:03.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:03.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.174268+0000", 2024-08-17T18:12:03.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:12:03.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:03.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.174273+0000", 2024-08-17T18:12:03.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9760000000000003e-06 2024-08-17T18:12:03.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:03.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.196129+0000", 2024-08-17T18:12:03.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021855731 2024-08-17T18:12:03.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:03.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.196141+0000", 2024-08-17T18:12:03.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.271e-05 2024-08-17T18:12:03.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:03.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.199579+0000", 2024-08-17T18:12:03.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003437463 2024-08-17T18:12:03.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:03.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.199734+0000", 2024-08-17T18:12:03.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015490799999999999 2024-08-17T18:12:03.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-08-17T18:12:03.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.199847+0000", 2024-08-17T18:12:03.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113399 2024-08-17T18:12:03.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:03.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.205012+0000", 2024-08-17T18:12:03.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051651730000000003 2024-08-17T18:12:03.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.212676+0000", 2024-08-17T18:12:03.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0076632260000000004 2024-08-17T18:12:03.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:03.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.215294+0000", 2024-08-17T18:12:03.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026182499999999999 2024-08-17T18:12:03.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:03.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.215353+0000", 2024-08-17T18:12:03.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8695e-05 2024-08-17T18:12:03.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:03.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:03.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:03.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:01.215390+0000", 2024-08-17T18:12:03.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7830999999999997e-05 2024-08-17T18:12:03.561 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.561 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:03.562 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.562 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:03.562 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:03.562 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:12:03.571 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:03.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:03.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:03.653 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:03 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a[109838]: 2024-08-17T18:12:03.533+0000 7f5b8e77d640 -1 mon.a@0(leader).osd e1552 definitely_dead 0 2024-08-17T18:12:03.672 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:03.677 INFO:teuthology.orchestra.run.smithi136.stderr:marked out osd.1. 2024-08-17T18:12:03.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:03.712 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:03.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:03.813 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:03.834 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:03.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:03.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:03.936 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:03 smithi136 ceph-mon[109861]: pgmap v2889: 99 pgs: 99 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 970 op/s 2024-08-17T18:12:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:03 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1000646867' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-08-17T18:12:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:03 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-08-17T18:12:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:03 smithi136 ceph-mon[109861]: Client client.admin marked osd.1 out, while it was still marked up 2024-08-17T18:12:03.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:03 smithi136 ceph-mon[115078]: pgmap v2889: 99 pgs: 99 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 970 op/s 2024-08-17T18:12:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:03 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1000646867' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-08-17T18:12:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:03 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-08-17T18:12:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:03 smithi136 ceph-mon[115078]: Client client.admin marked osd.1 out, while it was still marked up 2024-08-17T18:12:03.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:03.958 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:03.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:03 smithi165 ceph-mon[105517]: pgmap v2889: 99 pgs: 99 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 970 op/s 2024-08-17T18:12:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:03 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1000646867' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-08-17T18:12:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:03 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-08-17T18:12:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:03 smithi165 ceph-mon[105517]: Client client.admin marked osd.1 out, while it was still marked up 2024-08-17T18:12:04.060 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:04.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:04.082 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:04.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:04.099 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:12:04.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:04.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:04.205 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:04.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:04.306 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:04.328 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:04.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:04.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:04.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-17T18:12:04.429 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:04.451 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:04.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:04.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:04.552 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:04.575 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:04.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:04.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:04.618 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:12:04.676 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:04.697 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:04.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:04.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:04.799 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:04.820 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:04.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:04.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:04.882 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:04 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-08-17T18:12:04.883 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:04 smithi136 ceph-mon[109861]: osdmap e1553: 8 total, 8 up, 6 in 2024-08-17T18:12:04.883 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:04 smithi136 ceph-mon[109861]: osdmap e1554: 8 total, 8 up, 6 in 2024-08-17T18:12:04.883 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:04 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-08-17T18:12:04.883 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:04 smithi136 ceph-mon[115078]: osdmap e1553: 8 total, 8 up, 6 in 2024-08-17T18:12:04.883 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:04 smithi136 ceph-mon[115078]: osdmap e1554: 8 total, 8 up, 6 in 2024-08-17T18:12:04.922 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:04.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:04.944 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:04.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:04 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-08-17T18:12:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:04 smithi165 ceph-mon[105517]: osdmap e1553: 8 total, 8 up, 6 in 2024-08-17T18:12:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:04 smithi165 ceph-mon[105517]: osdmap e1554: 8 total, 8 up, 6 in 2024-08-17T18:12:05.045 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:05.066 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:05.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:05.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:05.168 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:05.189 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:05.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:05.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:05.291 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:05.313 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:05.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:05.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:05.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:05.437 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:05.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:05.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:05.539 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:05.560 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:05.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:05.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:05.662 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:05.688 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:05.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:05.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:05 smithi165 ceph-mon[105517]: pgmap v2891: 99 pgs: 1 unknown, 5 peering, 93 active+clean; 4.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 880 op/s 2024-08-17T18:12:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:05 smithi165 ceph-mon[105517]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-08-17T18:12:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:05 smithi165 ceph-mon[105517]: osdmap e1555: 8 total, 8 up, 6 in 2024-08-17T18:12:05.790 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:05.793 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:05 smithi136 ceph-mon[109861]: pgmap v2891: 99 pgs: 1 unknown, 5 peering, 93 active+clean; 4.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 880 op/s 2024-08-17T18:12:05.793 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:05 smithi136 ceph-mon[109861]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-08-17T18:12:05.793 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:05 smithi136 ceph-mon[109861]: osdmap e1555: 8 total, 8 up, 6 in 2024-08-17T18:12:05.793 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:05 smithi136 ceph-mon[115078]: pgmap v2891: 99 pgs: 1 unknown, 5 peering, 93 active+clean; 4.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 880 op/s 2024-08-17T18:12:05.793 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:05 smithi136 ceph-mon[115078]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-08-17T18:12:05.793 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:05 smithi136 ceph-mon[115078]: osdmap e1555: 8 total, 8 up, 6 in 2024-08-17T18:12:05.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:05.814 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:05.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:05.915 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:05.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:05.953 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:05.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:06.054 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:06.092 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:06.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:06.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:06.110 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:12:06.114 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:12:06.114 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:12:06.151 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:12:06.151 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:12:06.189 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:12:06.189 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:12:06.193 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:06.222 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:12:06.222 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:12:06.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:06.250 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:06.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:06.257 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:12:06.257 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:12:06.289 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:12:06.289 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:12:06.328 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:12:06.328 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:12:06.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:06.359 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:12:06.359 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:12:06.389 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:06.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:06.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:06.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:06.531 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:06.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:06.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:06.633 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:06.674 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:06.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:06.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:06.776 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:06.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:06.846 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:06.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:06.907 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:06 smithi136 ceph-mon[115078]: 56.a starting backfill to osd.3 from (0'0,0'0] MAX to 1552'11191 2024-08-17T18:12:06.907 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:06 smithi136 ceph-mon[115078]: 56.c starting backfill to osd.4 from (0'0,0'0] MAX to 1552'11187 2024-08-17T18:12:06.907 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:06 smithi136 ceph-mon[115078]: 56.f starting backfill to osd.0 from (0'0,0'0] MAX to 1552'11257 2024-08-17T18:12:06.907 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:06 smithi136 ceph-mon[115078]: 56.4 starting backfill to osd.5 from (0'0,0'0] MAX to 1552'11186 2024-08-17T18:12:06.907 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:06 smithi136 ceph-mon[115078]: 56.e starting backfill to osd.5 from (0'0,0'0] MAX to 1552'11125 2024-08-17T18:12:06.907 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:06 smithi136 ceph-mon[115078]: 56.3 starting backfill to osd.4 from (0'0,0'0] MAX to 1552'11109 2024-08-17T18:12:06.907 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:06 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4113406815' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:12:06.907 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:12:06.907 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:12:06.907 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:06 smithi136 ceph-mon[115078]: osdmap e1556: 8 total, 8 up, 6 in 2024-08-17T18:12:06.908 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:06 smithi136 ceph-mon[109861]: 56.a starting backfill to osd.3 from (0'0,0'0] MAX to 1552'11191 2024-08-17T18:12:06.908 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:06 smithi136 ceph-mon[109861]: 56.c starting backfill to osd.4 from (0'0,0'0] MAX to 1552'11187 2024-08-17T18:12:06.908 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:06 smithi136 ceph-mon[109861]: 56.f starting backfill to osd.0 from (0'0,0'0] MAX to 1552'11257 2024-08-17T18:12:06.908 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:06 smithi136 ceph-mon[109861]: 56.4 starting backfill to osd.5 from (0'0,0'0] MAX to 1552'11186 2024-08-17T18:12:06.908 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:06 smithi136 ceph-mon[109861]: 56.e starting backfill to osd.5 from (0'0,0'0] MAX to 1552'11125 2024-08-17T18:12:06.908 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:06 smithi136 ceph-mon[109861]: 56.3 starting backfill to osd.4 from (0'0,0'0] MAX to 1552'11109 2024-08-17T18:12:06.908 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:06 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4113406815' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:12:06.908 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:12:06.908 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:12:06.909 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:06 smithi136 ceph-mon[109861]: osdmap e1556: 8 total, 8 up, 6 in 2024-08-17T18:12:06.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:06.984 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:06 smithi165 ceph-mon[105517]: 56.a starting backfill to osd.3 from (0'0,0'0] MAX to 1552'11191 2024-08-17T18:12:06.984 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:06 smithi165 ceph-mon[105517]: 56.c starting backfill to osd.4 from (0'0,0'0] MAX to 1552'11187 2024-08-17T18:12:06.984 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:06 smithi165 ceph-mon[105517]: 56.f starting backfill to osd.0 from (0'0,0'0] MAX to 1552'11257 2024-08-17T18:12:06.984 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:06 smithi165 ceph-mon[105517]: 56.4 starting backfill to osd.5 from (0'0,0'0] MAX to 1552'11186 2024-08-17T18:12:06.984 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:06 smithi165 ceph-mon[105517]: 56.e starting backfill to osd.5 from (0'0,0'0] MAX to 1552'11125 2024-08-17T18:12:06.984 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:06 smithi165 ceph-mon[105517]: 56.3 starting backfill to osd.4 from (0'0,0'0] MAX to 1552'11109 2024-08-17T18:12:06.984 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:06 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4113406815' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:12:06.984 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:12:06.984 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:12:06.984 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:06 smithi165 ceph-mon[105517]: osdmap e1556: 8 total, 8 up, 6 in 2024-08-17T18:12:06.990 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:06.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:06.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:07.092 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:07.128 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:07.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:07.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:07.230 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:07.252 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:07.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:07.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:07.354 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:07.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:07.390 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:07.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:07.491 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:07.522 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:07.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:07.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:07.624 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:07.656 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:07.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:07.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:07.680 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:12:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:12:07] "GET /metrics HTTP/1.1" 200 36677 "" "Prometheus/2.51.0" 2024-08-17T18:12:07.758 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:07.790 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:07.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:07.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:07.891 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:07.937 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:07.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:07.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:07.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:07 smithi136 ceph-mon[109861]: pgmap v2894: 99 pgs: 1 active+undersized+remapped, 1 unknown, 5 peering, 92 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 946 op/s; 0 B/s, 0 objects/s recovering 2024-08-17T18:12:07.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:07 smithi136 ceph-mon[109861]: osdmap e1557: 8 total, 8 up, 6 in 2024-08-17T18:12:07.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:07 smithi136 ceph-mon[115078]: pgmap v2894: 99 pgs: 1 active+undersized+remapped, 1 unknown, 5 peering, 92 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 946 op/s; 0 B/s, 0 objects/s recovering 2024-08-17T18:12:07.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:07 smithi136 ceph-mon[115078]: osdmap e1557: 8 total, 8 up, 6 in 2024-08-17T18:12:08.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:07 smithi165 ceph-mon[105517]: pgmap v2894: 99 pgs: 1 active+undersized+remapped, 1 unknown, 5 peering, 92 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 0 B/s wr, 946 op/s; 0 B/s, 0 objects/s recovering 2024-08-17T18:12:08.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:07 smithi165 ceph-mon[105517]: osdmap e1557: 8 total, 8 up, 6 in 2024-08-17T18:12:08.038 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:08.078 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:08.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:08.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:08.180 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:08.226 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:08.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:08.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:08.328 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:08.377 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:08.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:08.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:08.479 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:08.488 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:12:08.488 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:12:08.517 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:08.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:08.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:08.619 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:08.663 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:08.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:08.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:08.765 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:08.805 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:08.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:08.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:08.907 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:08.945 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:08.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:08.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:09.046 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:09.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:09.092 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:09.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:09.193 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:09.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:09.229 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:09.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:09.249 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-17T18:12:09.329 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:09.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:09.372 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:09.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:09.473 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:09.516 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:09.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:09.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:09.618 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:09.625 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:12:09.652 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:09.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:09.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:09.699 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:12:09.753 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:09.794 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:09.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:09.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:09.896 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:09.906 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 6, 0, 5] out_osds: [7, 1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:12:09.907 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:12:09.907 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_3 2024-08-17T18:12:09.907 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T18:12:09.936 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:09.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:09.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:09.964 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:09 smithi165 ceph-mon[105517]: pgmap v2897: 99 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_wait, 1 active+undersized+remapped, 1 unknown, 6 peering, 86 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 19 KiB/s rd, 55 KiB/s wr, 573 op/s; 22546/221847 objects misplaced (10.163%); 4.7 MiB/s, 74 objects/s recovering 2024-08-17T18:12:09.964 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:09 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3155933812' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:12:09.964 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:09 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:12:09.972 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:09 smithi136 ceph-mon[109861]: pgmap v2897: 99 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_wait, 1 active+undersized+remapped, 1 unknown, 6 peering, 86 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 19 KiB/s rd, 55 KiB/s wr, 573 op/s; 22546/221847 objects misplaced (10.163%); 4.7 MiB/s, 74 objects/s recovering 2024-08-17T18:12:09.972 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:09 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3155933812' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:12:09.972 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:09 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:12:09.972 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:09 smithi136 ceph-mon[115078]: pgmap v2897: 99 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_wait, 1 active+undersized+remapped, 1 unknown, 6 peering, 86 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 19 KiB/s rd, 55 KiB/s wr, 573 op/s; 22546/221847 objects misplaced (10.163%); 4.7 MiB/s, 74 objects/s recovering 2024-08-17T18:12:09.972 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:09 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3155933812' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:12:09.972 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:09 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:12:10.038 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:10.072 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:10.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:10.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:10.174 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:10.208 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:10.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:10.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:10.310 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:10.353 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:10.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:10.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:10.396 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:12:10.454 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:10.498 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:10.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:10.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:10.600 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:10.635 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:10.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:10.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:10.737 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:10.776 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:10.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:10.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:10.878 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:10.918 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:10.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:10.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:10.979 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:10 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:12:10.980 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:10 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:12:10.980 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:10 smithi165 ceph-mon[105517]: osdmap e1558: 8 total, 8 up, 6 in 2024-08-17T18:12:11.020 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:11.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:11.078 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:11.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:11.179 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:11.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:10 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:12:11.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:10 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:12:11.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:10 smithi136 ceph-mon[109861]: osdmap e1558: 8 total, 8 up, 6 in 2024-08-17T18:12:11.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:10 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:12:11.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:10 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:12:11.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:10 smithi136 ceph-mon[115078]: osdmap e1558: 8 total, 8 up, 6 in 2024-08-17T18:12:11.220 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:11.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:11.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:11.322 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:11.366 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:11.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:11.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:11.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:11.497 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:11.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:11.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:11.598 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:11.640 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:11.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:11.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:11.742 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:11.778 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:11.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:11.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:11.818 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:12:11.818 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:12:11.880 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:11.917 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:11.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:11.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:11 smithi165 ceph-mon[105517]: pgmap v2899: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 1 active+undersized+remapped, 1 peering, 91 active+clean; 4.4 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 16 KiB/s rd, 47 KiB/s wr, 753 op/s; 26621/218043 objects misplaced (12.209%); 9.7 MiB/s, 155 objects/s recovering 2024-08-17T18:12:12.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:11 smithi165 ceph-mon[105517]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-08-17T18:12:12.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:11 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T18:12:12.019 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:12.052 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:12.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:12.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:12.154 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:12.189 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:12.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:12.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:12.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:11 smithi136 ceph-mon[109861]: pgmap v2899: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 1 active+undersized+remapped, 1 peering, 91 active+clean; 4.4 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 16 KiB/s rd, 47 KiB/s wr, 753 op/s; 26621/218043 objects misplaced (12.209%); 9.7 MiB/s, 155 objects/s recovering 2024-08-17T18:12:12.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:11 smithi136 ceph-mon[109861]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-08-17T18:12:12.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:11 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T18:12:12.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:11 smithi136 ceph-mon[115078]: pgmap v2899: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 1 active+undersized+remapped, 1 peering, 91 active+clean; 4.4 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 16 KiB/s rd, 47 KiB/s wr, 753 op/s; 26621/218043 objects misplaced (12.209%); 9.7 MiB/s, 155 objects/s recovering 2024-08-17T18:12:12.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:11 smithi136 ceph-mon[115078]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-08-17T18:12:12.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:11 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T18:12:12.291 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:12.331 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:12.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:12.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:12.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:12.472 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:12.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:12.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:12.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:12.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:12.614 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:12.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:12.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:12.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:12.755 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:12.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:12.856 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:12.891 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:12.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:12.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:12.993 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:13.023 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:13.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:13.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:13.125 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:13.161 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:13.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:13.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:13.242 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:12:13.263 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:13.286 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:12:13.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:13.328 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:13.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:13.429 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:13.470 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:13.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:13.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:13.572 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:13.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:13.620 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:13.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:13.721 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:13.753 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:13.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:13.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:13.839 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:13 smithi136 ceph-mon[109861]: pgmap v2900: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 4.3 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 12 KiB/s rd, 37 KiB/s wr, 847 op/s; 25304/211914 objects misplaced (11.941%); 12 MiB/s, 198 objects/s recovering 2024-08-17T18:12:13.839 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:13 smithi136 ceph-mon[115078]: pgmap v2900: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 4.3 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 12 KiB/s rd, 37 KiB/s wr, 847 op/s; 25304/211914 objects misplaced (11.941%); 12 MiB/s, 198 objects/s recovering 2024-08-17T18:12:13.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:13.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:13.891 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:13.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:13.992 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:13 smithi165 ceph-mon[105517]: pgmap v2900: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 4.3 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 12 KiB/s rd, 37 KiB/s wr, 847 op/s; 25304/211914 objects misplaced (11.941%); 12 MiB/s, 198 objects/s recovering 2024-08-17T18:12:14.025 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:14.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:14.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:14.126 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:14.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:14.162 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:14.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:14.263 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:14.303 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:14.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:14.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:14.405 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:14.442 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:14.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:14.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:14.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:14.585 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:14.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:14.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:14.687 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:14.719 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:14.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:14.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:14.821 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:14.859 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:14.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:14.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:14.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:14.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:14.995 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:14.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:15.031 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-17T18:12:15.097 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:15.123 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:15.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:15.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:15.216 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:12:15.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:15.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:15.264 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:15.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:15.365 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:15.417 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:15.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:15.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:15.457 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:12:15.519 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:15.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:15.570 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:15.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:15.671 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:15.713 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:15.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:15.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:15.815 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:15.854 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:15.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:15.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:15.857 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:15 smithi165 ceph-mon[105517]: pgmap v2901: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 4.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 9.7 KiB/s rd, 28 KiB/s wr, 1.01k op/s; 23356/203688 objects misplaced (11.467%); 18 MiB/s, 295 objects/s recovering 2024-08-17T18:12:15.923 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:15 smithi136 ceph-mon[109861]: pgmap v2901: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 4.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 9.7 KiB/s rd, 28 KiB/s wr, 1.01k op/s; 23356/203688 objects misplaced (11.467%); 18 MiB/s, 295 objects/s recovering 2024-08-17T18:12:15.923 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:15 smithi136 ceph-mon[115078]: pgmap v2901: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 4.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 9.7 KiB/s rd, 28 KiB/s wr, 1.01k op/s; 23356/203688 objects misplaced (11.467%); 18 MiB/s, 295 objects/s recovering 2024-08-17T18:12:15.956 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:15.994 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:15.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:15.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:16.067 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:12:16.096 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:16.126 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:12:16.127 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:12:16.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:16.150 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:16.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:16.163 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:12:16.163 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:12:16.198 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:12:16.198 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:12:16.232 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:12:16.233 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:12:16.251 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:16.267 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:12:16.267 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:12:16.289 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:16.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:16.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:16.301 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:12:16.302 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:12:16.334 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:12:16.334 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:12:16.368 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:12:16.369 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:12:16.391 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:16.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:16.432 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:16.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:16.534 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:16.570 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:16.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:16.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:16.672 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:16.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:16.711 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:16.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:16 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/666106154' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:12:16.812 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:16.856 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:16.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:16.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:16.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:16 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/666106154' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:12:16.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:16 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/666106154' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:12:16.958 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:17.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:17.002 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:17.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:17.103 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:17.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:17.141 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:17.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:17.242 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:17.277 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:17.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:17.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:17.379 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:17.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:17.428 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:17.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:17.529 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:17.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:17.559 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:17.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:17.568 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:12:17.568 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T18:12:17.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T18:12:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:12:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140115 55.16 55:686c39f0:::benchmark_data_smithi136_313476_object48129:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1500)", 2024-08-17T18:12:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.143129+0000", 2024-08-17T18:12:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 270.419828462, 2024-08-17T18:12:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.994428729, 2024-08-17T18:12:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:12:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:12:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140115 2024-08-17T18:12:17.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.143129+0000", 2024-08-17T18:12:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.143129+0000", 2024-08-17T18:12:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.143128+0000", 2024-08-17T18:12:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:12:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.143135+0000", 2024-08-17T18:12:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1010000000000003e-06 2024-08-17T18:12:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.143146+0000", 2024-08-17T18:12:17.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0953e-05 2024-08-17T18:12:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.143152+0000", 2024-08-17T18:12:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8119999999999996e-06 2024-08-17T18:12:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.168312+0000", 2024-08-17T18:12:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.025160491 2024-08-17T18:12:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.171272+0000", 2024-08-17T18:12:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002959555 2024-08-17T18:12:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.190464+0000", 2024-08-17T18:12:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019191881000000001 2024-08-17T18:12:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.192704+0000", 2024-08-17T18:12:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002239974 2024-08-17T18:12:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537367+0000", 2024-08-17T18:12:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.34466343999999999 2024-08-17T18:12:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:12:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537377+0000", 2024-08-17T18:12:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7100000000000002e-06 2024-08-17T18:12:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.030542+0000", 2024-08-17T18:12:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4931654029999999 2024-08-17T18:12:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.033326+0000", 2024-08-17T18:12:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027832360000000001 2024-08-17T18:12:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:12:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.037511+0000", 2024-08-17T18:12:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041850910000000002 2024-08-17T18:12:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.041644+0000", 2024-08-17T18:12:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041336869999999996 2024-08-17T18:12:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.051430+0000", 2024-08-17T18:12:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0097861519999999994 2024-08-17T18:12:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.131645+0000", 2024-08-17T18:12:17.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.080214368999999994 2024-08-17T18:12:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.132824+0000", 2024-08-17T18:12:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001179399 2024-08-17T18:12:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.137558+0000", 2024-08-17T18:12:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0047339340000000004 2024-08-17T18:12:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140118 55.6 55:63934a67:::benchmark_data_smithi136_313476_object48132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1500)", 2024-08-17T18:12:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.146936+0000", 2024-08-17T18:12:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 270.41602181100001, 2024-08-17T18:12:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8275733330000001, 2024-08-17T18:12:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:12:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:12:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140118 2024-08-17T18:12:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.578 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.146936+0000", 2024-08-17T18:12:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.146936+0000", 2024-08-17T18:12:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:17.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.146935+0000", 2024-08-17T18:12:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:12:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.146940+0000", 2024-08-17T18:12:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6169999999999998e-06 2024-08-17T18:12:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.146950+0000", 2024-08-17T18:12:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6909999999999992e-06 2024-08-17T18:12:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.146956+0000", 2024-08-17T18:12:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7699999999999998e-06 2024-08-17T18:12:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.171831+0000", 2024-08-17T18:12:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.024875853 2024-08-17T18:12:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.174174+0000", 2024-08-17T18:12:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00234293 2024-08-17T18:12:17.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.187094+0000", 2024-08-17T18:12:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012919717000000001 2024-08-17T18:12:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.189879+0000", 2024-08-17T18:12:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002785142 2024-08-17T18:12:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537107+0000", 2024-08-17T18:12:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.34722777700000002 2024-08-17T18:12:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:12:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537117+0000", 2024-08-17T18:12:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0414999999999999e-05 2024-08-17T18:12:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.946076+0000", 2024-08-17T18:12:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4089589140000003 2024-08-17T18:12:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.949008+0000", 2024-08-17T18:12:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029318690000000001 2024-08-17T18:12:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:12:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.953707+0000", 2024-08-17T18:12:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046990069999999998 2024-08-17T18:12:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.966900+0000", 2024-08-17T18:12:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013192812 2024-08-17T18:12:17.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.967853+0000", 2024-08-17T18:12:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00095252599999999998 2024-08-17T18:12:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.968105+0000", 2024-08-17T18:12:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025275999999999999 2024-08-17T18:12:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.969431+0000", 2024-08-17T18:12:17.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013257869999999999 2024-08-17T18:12:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.974509+0000", 2024-08-17T18:12:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050782379999999997 2024-08-17T18:12:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140121 55.6 55:62195bf4:::benchmark_data_smithi136_313476_object48135:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1500)", 2024-08-17T18:12:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.155517+0000", 2024-08-17T18:12:17.586 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 270.40744067700001, 2024-08-17T18:12:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9475979289999996, 2024-08-17T18:12:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:12:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:12:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140121 2024-08-17T18:12:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.155517+0000", 2024-08-17T18:12:17.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.155517+0000", 2024-08-17T18:12:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.155516+0000", 2024-08-17T18:12:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:12:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:17.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.155523+0000", 2024-08-17T18:12:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7630000000000003e-06 2024-08-17T18:12:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.155537+0000", 2024-08-17T18:12:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4782e-05 2024-08-17T18:12:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.155544+0000", 2024-08-17T18:12:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8000000000000001e-06 2024-08-17T18:12:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.175182+0000", 2024-08-17T18:12:17.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019637852000000001 2024-08-17T18:12:17.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:17.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.177528+0000", 2024-08-17T18:12:17.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002346374 2024-08-17T18:12:17.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.199421+0000", 2024-08-17T18:12:17.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.02189261 2024-08-17T18:12:17.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:17.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.201679+0000", 2024-08-17T18:12:17.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002258164 2024-08-17T18:12:17.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537146+0000", 2024-08-17T18:12:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.33546662100000002 2024-08-17T18:12:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:12:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537156+0000", 2024-08-17T18:12:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8830000000000001e-06 2024-08-17T18:12:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.974898+0000", 2024-08-17T18:12:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4377422559999999 2024-08-17T18:12:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.978346+0000", 2024-08-17T18:12:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003448441 2024-08-17T18:12:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:12:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.982964+0000", 2024-08-17T18:12:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046176430000000003 2024-08-17T18:12:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.997825+0000", 2024-08-17T18:12:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014860923 2024-08-17T18:12:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.998982+0000", 2024-08-17T18:12:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00115672 2024-08-17T18:12:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.096437+0000", 2024-08-17T18:12:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.097455183000000001 2024-08-17T18:12:17.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.097744+0000", 2024-08-17T18:12:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001306794 2024-08-17T18:12:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.103115+0000", 2024-08-17T18:12:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00537157 2024-08-17T18:12:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140134 55.6 55:6053b4ec:::benchmark_data_smithi136_313476_object48148:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1500)", 2024-08-17T18:12:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.187965+0000", 2024-08-17T18:12:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 270.37499283099999, 2024-08-17T18:12:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9270218950000002, 2024-08-17T18:12:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:12:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:12:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140134 2024-08-17T18:12:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.596 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.187965+0000", 2024-08-17T18:12:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.187965+0000", 2024-08-17T18:12:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:17.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.187964+0000", 2024-08-17T18:12:17.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:12:17.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.187969+0000", 2024-08-17T18:12:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5470000000000004e-06 2024-08-17T18:12:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.187981+0000", 2024-08-17T18:12:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.182e-05 2024-08-17T18:12:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.187988+0000", 2024-08-17T18:12:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9569999999999996e-06 2024-08-17T18:12:17.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.202404+0000", 2024-08-17T18:12:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014416258 2024-08-17T18:12:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.204940+0000", 2024-08-17T18:12:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002536061 2024-08-17T18:12:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.251179+0000", 2024-08-17T18:12:17.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.046238824999999997 2024-08-17T18:12:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.253521+0000", 2024-08-17T18:12:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00234191 2024-08-17T18:12:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537183+0000", 2024-08-17T18:12:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.28366188799999997 2024-08-17T18:12:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:12:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537192+0000", 2024-08-17T18:12:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6920000000000008e-06 2024-08-17T18:12:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.986670+0000", 2024-08-17T18:12:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4494780689999995 2024-08-17T18:12:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.989571+0000", 2024-08-17T18:12:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029015310000000002 2024-08-17T18:12:17.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:12:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.993999+0000", 2024-08-17T18:12:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044280639999999998 2024-08-17T18:12:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.998741+0000", 2024-08-17T18:12:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0047416170000000001 2024-08-17T18:12:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:17.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.999237+0000", 2024-08-17T18:12:17.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00049553300000000004 2024-08-17T18:12:17.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.604 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.107406+0000", 2024-08-17T18:12:17.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10816967399999999 2024-08-17T18:12:17.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:17.604 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.108896+0000", 2024-08-17T18:12:17.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001489654 2024-08-17T18:12:17.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:17.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.114987+0000", 2024-08-17T18:12:17.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060910670000000004 2024-08-17T18:12:17.605 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.605 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:17.605 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.605 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140142 55.16 55:6fe98626:::benchmark_data_smithi136_313476_object48156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1500)", 2024-08-17T18:12:17.605 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.245454+0000", 2024-08-17T18:12:17.605 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 270.31750420700001, 2024-08-17T18:12:17.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6765945460000005, 2024-08-17T18:12:17.605 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:17.605 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:17.606 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:17.606 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:12:17.606 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:12:17.606 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140142 2024-08-17T18:12:17.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.606 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:17.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.606 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:17.606 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.245454+0000", 2024-08-17T18:12:17.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.606 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:17.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.245454+0000", 2024-08-17T18:12:17.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:17.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.245452+0000", 2024-08-17T18:12:17.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:12:17.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:17.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.245509+0000", 2024-08-17T18:12:17.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6351999999999999e-05 2024-08-17T18:12:17.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.608 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:17.608 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.245521+0000", 2024-08-17T18:12:17.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2826000000000001e-05 2024-08-17T18:12:17.608 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.608 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.608 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:17.608 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.245529+0000", 2024-08-17T18:12:17.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4780000000000002e-06 2024-08-17T18:12:17.608 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.608 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.608 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.608 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.251081+0000", 2024-08-17T18:12:17.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055522970000000003 2024-08-17T18:12:17.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:17.609 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.253259+0000", 2024-08-17T18:12:17.609 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002178003 2024-08-17T18:12:17.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.609 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.265864+0000", 2024-08-17T18:12:17.609 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012605269000000001 2024-08-17T18:12:17.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.610 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:17.610 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.267795+0000", 2024-08-17T18:12:17.610 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001930502 2024-08-17T18:12:17.610 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.610 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.610 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.610 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537220+0000", 2024-08-17T18:12:17.610 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.26942472699999997 2024-08-17T18:12:17.610 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.610 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.610 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:12:17.610 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537229+0000", 2024-08-17T18:12:17.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7729999999999994e-06 2024-08-17T18:12:17.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.611 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.611 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:50.959941+0000", 2024-08-17T18:12:17.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4227114749999998 2024-08-17T18:12:17.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.611 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:17.611 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:50.962824+0000", 2024-08-17T18:12:17.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028828310000000002 2024-08-17T18:12:17.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.611 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:12:17.612 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:50.967194+0000", 2024-08-17T18:12:17.612 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043701160000000003 2024-08-17T18:12:17.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.612 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.612 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:17.612 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:52.733611+0000", 2024-08-17T18:12:17.612 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.766416706 2024-08-17T18:12:17.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.612 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.612 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.612 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.914906+0000", 2024-08-17T18:12:17.612 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1812954549999999 2024-08-17T18:12:17.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.612 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.613 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.613 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.915610+0000", 2024-08-17T18:12:17.613 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00070412299999999997 2024-08-17T18:12:17.613 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.613 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.613 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:17.613 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.917012+0000", 2024-08-17T18:12:17.613 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001401577 2024-08-17T18:12:17.613 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.613 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.613 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:17.613 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.922048+0000", 2024-08-17T18:12:17.614 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050365399999999999 2024-08-17T18:12:17.614 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.614 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:17.614 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.614 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.614 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.614 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140144 55.16 55:6a89b7a5:::benchmark_data_smithi136_313476_object48158:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1500)", 2024-08-17T18:12:17.614 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.245905+0000", 2024-08-17T18:12:17.614 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 270.31705298000003, 2024-08-17T18:12:17.614 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8777299039999997, 2024-08-17T18:12:17.614 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:17.614 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:17.614 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:17.615 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:12:17.615 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:12:17.615 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140144 2024-08-17T18:12:17.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.615 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:17.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.615 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:17.615 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.245905+0000", 2024-08-17T18:12:17.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.615 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:17.615 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.245905+0000", 2024-08-17T18:12:17.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.616 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.616 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:17.616 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.245904+0000", 2024-08-17T18:12:17.616 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:12:17.616 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.616 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:17.616 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.245909+0000", 2024-08-17T18:12:17.616 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2739999999999998e-06 2024-08-17T18:12:17.616 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.616 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:17.616 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.245922+0000", 2024-08-17T18:12:17.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2374999999999999e-05 2024-08-17T18:12:17.617 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:17.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.245929+0000", 2024-08-17T18:12:17.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1849999999999998e-06 2024-08-17T18:12:17.617 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.253500+0000", 2024-08-17T18:12:17.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0075709100000000001 2024-08-17T18:12:17.617 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:17.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.255502+0000", 2024-08-17T18:12:17.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020023689999999999 2024-08-17T18:12:17.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.275777+0000", 2024-08-17T18:12:17.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020275027000000001 2024-08-17T18:12:17.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:17.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.277719+0000", 2024-08-17T18:12:17.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019421569999999999 2024-08-17T18:12:17.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537294+0000", 2024-08-17T18:12:17.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25957475800000002 2024-08-17T18:12:17.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:12:17.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537303+0000", 2024-08-17T18:12:17.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7760000000000003e-06 2024-08-17T18:12:17.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.008985+0000", 2024-08-17T18:12:17.620 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4716823100000003 2024-08-17T18:12:17.620 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.620 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.620 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:17.620 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.011852+0000", 2024-08-17T18:12:17.620 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028672609999999999 2024-08-17T18:12:17.620 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.620 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.620 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:12:17.620 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.016105+0000", 2024-08-17T18:12:17.620 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004252736 2024-08-17T18:12:17.620 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.620 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.621 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.621 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.020400+0000", 2024-08-17T18:12:17.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042951789999999997 2024-08-17T18:12:17.621 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.621 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.621 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:17.621 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.050959+0000", 2024-08-17T18:12:17.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.030559118 2024-08-17T18:12:17.621 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.621 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.621 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.621 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.116967+0000", 2024-08-17T18:12:17.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.066007328000000004 2024-08-17T18:12:17.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:17.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.118335+0000", 2024-08-17T18:12:17.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013681559999999999 2024-08-17T18:12:17.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:17.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.123635+0000", 2024-08-17T18:12:17.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005299893 2024-08-17T18:12:17.622 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.622 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:17.622 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.623 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140146 55.16 55:6ac64e84:::benchmark_data_smithi136_313476_object48160:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1500)", 2024-08-17T18:12:17.623 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.251449+0000", 2024-08-17T18:12:17.623 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 270.31150848099998, 2024-08-17T18:12:17.623 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8637902650000004, 2024-08-17T18:12:17.623 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:17.623 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:17.623 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:17.623 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:12:17.623 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:12:17.623 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140146 2024-08-17T18:12:17.623 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.623 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:17.624 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.624 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:17.624 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.251449+0000", 2024-08-17T18:12:17.624 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.624 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.624 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:17.624 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.251449+0000", 2024-08-17T18:12:17.624 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.624 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.624 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:17.624 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.251448+0000", 2024-08-17T18:12:17.624 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:12:17.625 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.625 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.625 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:17.625 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.251455+0000", 2024-08-17T18:12:17.625 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6250000000000001e-06 2024-08-17T18:12:17.625 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.625 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.625 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:17.625 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.251466+0000", 2024-08-17T18:12:17.625 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1518999999999999e-05 2024-08-17T18:12:17.625 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.625 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.625 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:17.625 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.251472+0000", 2024-08-17T18:12:17.626 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6200000000000004e-06 2024-08-17T18:12:17.626 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.626 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.626 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.626 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.255771+0000", 2024-08-17T18:12:17.626 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042986459999999997 2024-08-17T18:12:17.626 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.626 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.626 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:17.626 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.257827+0000", 2024-08-17T18:12:17.626 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020561529999999998 2024-08-17T18:12:17.626 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.626 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.627 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.286005+0000", 2024-08-17T18:12:17.627 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.028178279000000001 2024-08-17T18:12:17.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.627 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.628 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:17.628 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.287934+0000", 2024-08-17T18:12:17.628 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001928527 2024-08-17T18:12:17.628 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.628 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.628 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.628 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537257+0000", 2024-08-17T18:12:17.628 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.24932309699999999 2024-08-17T18:12:17.628 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.628 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.628 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:12:17.628 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537266+0000", 2024-08-17T18:12:17.629 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5389999999999996e-06 2024-08-17T18:12:17.629 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.629 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.629 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.629 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.998491+0000", 2024-08-17T18:12:17.629 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4612248159999996 2024-08-17T18:12:17.629 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.629 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.629 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:17.629 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.001285+0000", 2024-08-17T18:12:17.629 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027934000000000001 2024-08-17T18:12:17.629 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.629 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:12:17.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.005549+0000", 2024-08-17T18:12:17.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042640719999999998 2024-08-17T18:12:17.630 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.630 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.019664+0000", 2024-08-17T18:12:17.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014115259999999999 2024-08-17T18:12:17.630 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.630 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:17.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.050737+0000", 2024-08-17T18:12:17.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.031073554999999999 2024-08-17T18:12:17.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.108051+0000", 2024-08-17T18:12:17.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.057313192999999998 2024-08-17T18:12:17.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:17.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.109526+0000", 2024-08-17T18:12:17.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014752809999999999 2024-08-17T18:12:17.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:17.632 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.115240+0000", 2024-08-17T18:12:17.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057137150000000003 2024-08-17T18:12:17.632 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.632 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:17.632 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.632 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.632 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140177 55.16 55:69108c58:::benchmark_data_smithi136_313476_object48191:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1500)", 2024-08-17T18:12:17.632 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.409890+0000", 2024-08-17T18:12:17.632 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 270.15306744700001, 2024-08-17T18:12:17.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7210113929999995, 2024-08-17T18:12:17.632 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:17.632 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:17.632 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:17.633 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:12:17.633 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:12:17.633 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140177 2024-08-17T18:12:17.633 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.633 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:17.633 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.633 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:17.633 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.409890+0000", 2024-08-17T18:12:17.633 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.633 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.633 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.633 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:17.633 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.409890+0000", 2024-08-17T18:12:17.634 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.634 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.634 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.634 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:17.634 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.409889+0000", 2024-08-17T18:12:17.634 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:12:17.634 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.634 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.634 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:17.634 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.409898+0000", 2024-08-17T18:12:17.634 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9730000000000003e-06 2024-08-17T18:12:17.634 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.634 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.635 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:17.635 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.409914+0000", 2024-08-17T18:12:17.635 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.658e-05 2024-08-17T18:12:17.635 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.635 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.635 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:17.635 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.409924+0000", 2024-08-17T18:12:17.635 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2769999999999994e-06 2024-08-17T18:12:17.635 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.635 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.635 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.635 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.410911+0000", 2024-08-17T18:12:17.635 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00098741999999999992 2024-08-17T18:12:17.636 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.636 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.636 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:17.636 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.412914+0000", 2024-08-17T18:12:17.636 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002003459 2024-08-17T18:12:17.636 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.636 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.636 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.636 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.429694+0000", 2024-08-17T18:12:17.636 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016779755 2024-08-17T18:12:17.636 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.636 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.636 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:17.636 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.432264+0000", 2024-08-17T18:12:17.637 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002569876 2024-08-17T18:12:17.637 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.637 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.637 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.637 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537330+0000", 2024-08-17T18:12:17.637 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.105066137 2024-08-17T18:12:17.637 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.637 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.637 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:12:17.637 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537340+0000", 2024-08-17T18:12:17.637 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6500000000000008e-06 2024-08-17T18:12:17.637 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.637 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.638 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.638 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.020664+0000", 2024-08-17T18:12:17.638 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4833246239999998 2024-08-17T18:12:17.638 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.638 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.638 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:17.638 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.023441+0000", 2024-08-17T18:12:17.638 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002776868 2024-08-17T18:12:17.638 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.638 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.638 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:12:17.638 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.027597+0000", 2024-08-17T18:12:17.639 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041561000000000002 2024-08-17T18:12:17.639 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.639 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.639 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.639 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.040926+0000", 2024-08-17T18:12:17.639 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013328361 2024-08-17T18:12:17.639 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.639 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.639 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:17.639 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.051205+0000", 2024-08-17T18:12:17.639 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010279501999999999 2024-08-17T18:12:17.639 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.639 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.639 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.640 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.124473+0000", 2024-08-17T18:12:17.640 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.073267551 2024-08-17T18:12:17.640 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.640 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.640 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:17.640 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.125608+0000", 2024-08-17T18:12:17.640 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011352999999999999 2024-08-17T18:12:17.640 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.640 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.640 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:17.640 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.130902+0000", 2024-08-17T18:12:17.640 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052936540000000001 2024-08-17T18:12:17.640 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.640 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:17.641 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.641 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.641 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.641 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140201 55.6 55:667a10e9:::benchmark_data_smithi136_313476_object48215:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1500)", 2024-08-17T18:12:17.641 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.438859+0000", 2024-08-17T18:12:17.641 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 270.12409934999999, 2024-08-17T18:12:17.641 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5271923440000004, 2024-08-17T18:12:17.641 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:17.641 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:17.641 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:17.641 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:12:17.641 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:12:17.641 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140201 2024-08-17T18:12:17.642 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.642 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:17.642 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.642 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:17.642 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.438859+0000", 2024-08-17T18:12:17.642 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.642 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.642 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.642 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:17.642 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.438859+0000", 2024-08-17T18:12:17.642 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.642 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.642 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.643 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:17.643 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.438857+0000", 2024-08-17T18:12:17.643 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:12:17.643 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.643 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.643 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:17.643 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.438863+0000", 2024-08-17T18:12:17.643 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5699999999999998e-06 2024-08-17T18:12:17.643 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.643 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.643 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:17.643 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.438876+0000", 2024-08-17T18:12:17.643 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2816999999999999e-05 2024-08-17T18:12:17.644 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.644 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.644 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:17.644 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.438883+0000", 2024-08-17T18:12:17.644 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6590000000000001e-06 2024-08-17T18:12:17.644 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.644 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.644 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.644 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.454188+0000", 2024-08-17T18:12:17.644 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015305309 2024-08-17T18:12:17.644 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.644 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.644 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:17.645 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.456778+0000", 2024-08-17T18:12:17.645 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025896230000000001 2024-08-17T18:12:17.645 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.645 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.645 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.645 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.501828+0000", 2024-08-17T18:12:17.645 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.045050458000000002 2024-08-17T18:12:17.645 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.645 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.645 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:17.645 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.504450+0000", 2024-08-17T18:12:17.645 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002621974 2024-08-17T18:12:17.645 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.646 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.646 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.646 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537059+0000", 2024-08-17T18:12:17.646 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.032608613000000002 2024-08-17T18:12:17.646 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.646 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.646 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:12:17.646 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537076+0000", 2024-08-17T18:12:17.646 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7144000000000001e-05 2024-08-17T18:12:17.646 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.646 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.646 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.646 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.933389+0000", 2024-08-17T18:12:17.647 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3963127950000001 2024-08-17T18:12:17.647 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.647 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.647 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:17.647 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.936377+0000", 2024-08-17T18:12:17.647 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029886380000000001 2024-08-17T18:12:17.647 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.647 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.647 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:12:17.647 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.941168+0000", 2024-08-17T18:12:17.647 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0047900670000000003 2024-08-17T18:12:17.647 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.647 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.647 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.648 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.958227+0000", 2024-08-17T18:12:17.648 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017059748 2024-08-17T18:12:17.648 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.648 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.648 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.648 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.959204+0000", 2024-08-17T18:12:17.648 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0009770899999999999 2024-08-17T18:12:17.648 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.648 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.648 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:17.648 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.959445+0000", 2024-08-17T18:12:17.648 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000240974 2024-08-17T18:12:17.648 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.649 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.649 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:17.649 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.960683+0000", 2024-08-17T18:12:17.649 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012378020000000001 2024-08-17T18:12:17.649 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.649 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.649 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:17.649 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.966051+0000", 2024-08-17T18:12:17.649 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0053677109999999998 2024-08-17T18:12:17.649 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.649 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:17.649 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.649 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.650 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.650 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140219 55.6 55:638f9cb2:::benchmark_data_smithi136_313476_object48233:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1500)", 2024-08-17T18:12:17.650 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.475188+0000", 2024-08-17T18:12:17.650 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 270.087769611, 2024-08-17T18:12:17.650 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8959626580000002, 2024-08-17T18:12:17.650 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:17.650 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:17.650 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:17.650 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:12:17.650 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:12:17.650 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140219 2024-08-17T18:12:17.650 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.650 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:17.651 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.651 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:17.651 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.475188+0000", 2024-08-17T18:12:17.651 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.651 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.651 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.651 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:17.651 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.475188+0000", 2024-08-17T18:12:17.651 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.651 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.651 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.651 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:17.651 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.475187+0000", 2024-08-17T18:12:17.651 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:12:17.652 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.652 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.652 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:17.652 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.475194+0000", 2024-08-17T18:12:17.652 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9990000000000002e-06 2024-08-17T18:12:17.652 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.652 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.652 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:17.652 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.475208+0000", 2024-08-17T18:12:17.652 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4423000000000001e-05 2024-08-17T18:12:17.652 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.652 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.652 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:17.653 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.475216+0000", 2024-08-17T18:12:17.653 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8639999999999996e-06 2024-08-17T18:12:17.653 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.653 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.653 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.653 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.512299+0000", 2024-08-17T18:12:17.653 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037082708999999998 2024-08-17T18:12:17.653 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.653 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.653 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:17.653 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.514754+0000", 2024-08-17T18:12:17.653 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024551780000000001 2024-08-17T18:12:17.653 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.653 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.654 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.654 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.535464+0000", 2024-08-17T18:12:17.654 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020710304999999998 2024-08-17T18:12:17.654 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.654 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.654 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:12:17.654 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.535476+0000", 2024-08-17T18:12:17.654 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1453e-05 2024-08-17T18:12:17.654 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.654 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.654 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.654 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:52.788288+0000", 2024-08-17T18:12:17.654 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2528118360000002 2024-08-17T18:12:17.655 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.655 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.655 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:17.655 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:52.791022+0000", 2024-08-17T18:12:17.655 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027340939999999998 2024-08-17T18:12:17.655 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.655 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.655 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.655 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.930296+0000", 2024-08-17T18:12:17.655 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1392738250000001 2024-08-17T18:12:17.655 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.655 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.655 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:17.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:53.933061+0000", 2024-08-17T18:12:17.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027656360000000001 2024-08-17T18:12:17.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.656 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.180790+0000", 2024-08-17T18:12:17.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.24772897999999999 2024-08-17T18:12:17.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.656 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:17.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.183020+0000", 2024-08-17T18:12:17.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002229483 2024-08-17T18:12:17.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.657 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:12:17.657 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.187688+0000", 2024-08-17T18:12:17.657 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046687259999999998 2024-08-17T18:12:17.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.657 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.657 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.191766+0000", 2024-08-17T18:12:17.657 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040775059999999998 2024-08-17T18:12:17.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.657 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:17.657 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.215058+0000", 2024-08-17T18:12:17.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023292335000000001 2024-08-17T18:12:17.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.658 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.658 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.361167+0000", 2024-08-17T18:12:17.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.14610825799999999 2024-08-17T18:12:17.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.658 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:17.658 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.362649+0000", 2024-08-17T18:12:17.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014826959999999999 2024-08-17T18:12:17.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.659 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:17.659 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.371151+0000", 2024-08-17T18:12:17.659 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0085017180000000001 2024-08-17T18:12:17.659 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.659 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:17.659 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.659 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.659 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.659 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140245 55.16 55:6a908b3e:::benchmark_data_smithi136_313476_object48259:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1501)", 2024-08-17T18:12:17.659 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.537627+0000", 2024-08-17T18:12:17.659 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 270.02533077099997, 2024-08-17T18:12:17.659 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8502157009999998, 2024-08-17T18:12:17.659 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:17.659 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:17.660 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:17.660 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:12:17.660 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:12:17.660 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140245 2024-08-17T18:12:17.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.660 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:17.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:17.660 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537627+0000", 2024-08-17T18:12:17.660 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:17.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537627+0000", 2024-08-17T18:12:17.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.661 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:17.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537626+0000", 2024-08-17T18:12:17.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:12:17.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.661 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:17.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537631+0000", 2024-08-17T18:12:17.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3120000000000001e-06 2024-08-17T18:12:17.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:17.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537643+0000", 2024-08-17T18:12:17.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2278e-05 2024-08-17T18:12:17.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:17.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537650+0000", 2024-08-17T18:12:17.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8199999999999999e-06 2024-08-17T18:12:17.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537714+0000", 2024-08-17T18:12:17.663 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3718000000000002e-05 2024-08-17T18:12:17.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.663 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.663 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:12:17.663 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.537723+0000", 2024-08-17T18:12:17.663 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3039999999999994e-06 2024-08-17T18:12:17.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.663 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.663 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.663 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.042001+0000", 2024-08-17T18:12:17.663 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5042780990000004 2024-08-17T18:12:17.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.663 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.663 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:17.664 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.044575+0000", 2024-08-17T18:12:17.664 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025734820000000002 2024-08-17T18:12:17.664 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.664 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.664 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.664 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.091971+0000", 2024-08-17T18:12:17.664 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.047396471000000003 2024-08-17T18:12:17.664 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.664 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.664 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:17.664 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.094508+0000", 2024-08-17T18:12:17.664 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025367599999999999 2024-08-17T18:12:17.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.232886+0000", 2024-08-17T18:12:17.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.138377737 2024-08-17T18:12:17.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:17.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.234976+0000", 2024-08-17T18:12:17.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020901000000000001 2024-08-17T18:12:17.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:12:17.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.239301+0000", 2024-08-17T18:12:17.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043252610000000004 2024-08-17T18:12:17.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.259472+0000", 2024-08-17T18:12:17.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020170738000000001 2024-08-17T18:12:17.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:17.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.260547+0000", 2024-08-17T18:12:17.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00107544 2024-08-17T18:12:17.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.667 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.380530+0000", 2024-08-17T18:12:17.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.119982598 2024-08-17T18:12:17.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.667 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:17.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.382034+0000", 2024-08-17T18:12:17.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015040279999999999 2024-08-17T18:12:17.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.667 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:17.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.387843+0000", 2024-08-17T18:12:17.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058090759999999998 2024-08-17T18:12:17.667 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.668 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:17.668 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.668 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.668 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.668 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140251 55.16 55:6fd0bdc9:::benchmark_data_smithi136_313476_object48265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1501)", 2024-08-17T18:12:17.668 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.556377+0000", 2024-08-17T18:12:17.668 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 270.00658119799999, 2024-08-17T18:12:17.668 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8137197040000004, 2024-08-17T18:12:17.668 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:17.668 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:17.668 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:17.668 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:12:17.668 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:12:17.668 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140251 2024-08-17T18:12:17.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.669 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:17.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:17.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.556377+0000", 2024-08-17T18:12:17.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:17.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.556377+0000", 2024-08-17T18:12:17.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.670 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:17.670 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.556375+0000", 2024-08-17T18:12:17.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:12:17.670 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.670 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:17.670 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.556381+0000", 2024-08-17T18:12:17.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4539999999999998e-06 2024-08-17T18:12:17.670 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.670 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:17.671 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:17.672 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.556396+0000", 2024-08-17T18:12:17.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5579000000000001e-05 2024-08-17T18:12:17.672 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.672 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.672 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:17.672 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.556403+0000", 2024-08-17T18:12:17.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2089999999999999e-06 2024-08-17T18:12:17.672 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.672 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.560709+0000", 2024-08-17T18:12:17.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043056860000000004 2024-08-17T18:12:17.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:12:17.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.560719+0000", 2024-08-17T18:12:17.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4360000000000008e-06 2024-08-17T18:12:17.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.044919+0000", 2024-08-17T18:12:17.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.484200446 2024-08-17T18:12:17.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:17.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.047552+0000", 2024-08-17T18:12:17.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026333939999999998 2024-08-17T18:12:17.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.064624+0000", 2024-08-17T18:12:17.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017071651 2024-08-17T18:12:17.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:17.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.067296+0000", 2024-08-17T18:12:17.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002672439 2024-08-17T18:12:17.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.675 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.675 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.198782+0000", 2024-08-17T18:12:17.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13148515799999999 2024-08-17T18:12:17.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.675 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:17.675 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.200937+0000", 2024-08-17T18:12:17.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021558940000000002 2024-08-17T18:12:17.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.676 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:12:17.676 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.205362+0000", 2024-08-17T18:12:17.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044241949999999997 2024-08-17T18:12:17.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.676 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.676 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.225794+0000", 2024-08-17T18:12:17.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020432344000000002 2024-08-17T18:12:17.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.676 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:17.676 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.226024+0000", 2024-08-17T18:12:17.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022981600000000001 2024-08-17T18:12:17.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.677 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.361923+0000", 2024-08-17T18:12:17.677 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13589936499999999 2024-08-17T18:12:17.677 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.677 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:17.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.363412+0000", 2024-08-17T18:12:17.677 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014891329999999999 2024-08-17T18:12:17.677 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.677 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:17.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.370096+0000", 2024-08-17T18:12:17.678 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006684026 2024-08-17T18:12:17.678 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.678 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:17.678 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.678 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.678 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140266 55.16 55:6f3984ca:::benchmark_data_smithi136_313476_object48280:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1501)", 2024-08-17T18:12:17.678 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.575512+0000", 2024-08-17T18:12:17.678 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 269.98744619399997, 2024-08-17T18:12:17.678 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.804134779, 2024-08-17T18:12:17.678 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:17.678 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:17.678 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:17.679 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:12:17.679 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:12:17.679 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140266 2024-08-17T18:12:17.679 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.679 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:17.679 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.679 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:17.679 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.575512+0000", 2024-08-17T18:12:17.679 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.679 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.679 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.679 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:17.679 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.575512+0000", 2024-08-17T18:12:17.680 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.680 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.680 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.680 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:17.680 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.575511+0000", 2024-08-17T18:12:17.680 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:12:17.680 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.680 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.680 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:17.680 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.575516+0000", 2024-08-17T18:12:17.680 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2510000000000002e-06 2024-08-17T18:12:17.680 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.680 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.681 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:17.681 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.575527+0000", 2024-08-17T18:12:17.681 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0993e-05 2024-08-17T18:12:17.681 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.681 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:17.681 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.575534+0000", 2024-08-17T18:12:17.681 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3560000000000004e-06 2024-08-17T18:12:17.681 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.681 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.681 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.575580+0000", 2024-08-17T18:12:17.681 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6164e-05 2024-08-17T18:12:17.681 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.682 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.682 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:12:17.682 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.575588+0000", 2024-08-17T18:12:17.682 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4939999999999997e-06 2024-08-17T18:12:17.682 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.682 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.682 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.682 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.047938+0000", 2024-08-17T18:12:17.682 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4723502579999996 2024-08-17T18:12:17.682 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.682 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.682 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:17.682 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.050500+0000", 2024-08-17T18:12:17.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025613760000000002 2024-08-17T18:12:17.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.683 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.683 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.077424+0000", 2024-08-17T18:12:17.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.026924766999999999 2024-08-17T18:12:17.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.683 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:17.683 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.080382+0000", 2024-08-17T18:12:17.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002957627 2024-08-17T18:12:17.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.683 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.684 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.214958+0000", 2024-08-17T18:12:17.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13457627 2024-08-17T18:12:17.684 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.684 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.684 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:17.684 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.217434+0000", 2024-08-17T18:12:17.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024758839999999998 2024-08-17T18:12:17.684 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.684 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.684 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:12:17.684 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.221763+0000", 2024-08-17T18:12:17.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043288859999999997 2024-08-17T18:12:17.684 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.685 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.685 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.685 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.226843+0000", 2024-08-17T18:12:17.685 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050802130000000001 2024-08-17T18:12:17.685 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.685 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.685 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:17.685 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.242789+0000", 2024-08-17T18:12:17.685 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015945773999999999 2024-08-17T18:12:17.685 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.685 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.685 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.685 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.372330+0000", 2024-08-17T18:12:17.685 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12954122500000001 2024-08-17T18:12:17.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:17.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.373797+0000", 2024-08-17T18:12:17.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001466619 2024-08-17T18:12:17.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:17.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.379646+0000", 2024-08-17T18:12:17.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058496149999999998 2024-08-17T18:12:17.686 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.686 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:17.686 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.687 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140287 55.16 55:6c1e48be:::benchmark_data_smithi136_313476_object48301:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1501)", 2024-08-17T18:12:17.687 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:47.673616+0000", 2024-08-17T18:12:17.687 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 269.88934178, 2024-08-17T18:12:17.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7225061789999998, 2024-08-17T18:12:17.687 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:17.687 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:17.687 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:17.687 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:12:17.687 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:12:17.687 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140287 2024-08-17T18:12:17.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.688 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:17.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:17.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.673616+0000", 2024-08-17T18:12:17.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:17.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.673616+0000", 2024-08-17T18:12:17.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:17.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.673615+0000", 2024-08-17T18:12:17.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:12:17.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.689 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:17.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.673620+0000", 2024-08-17T18:12:17.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4380000000000003e-06 2024-08-17T18:12:17.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.689 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:17.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.673633+0000", 2024-08-17T18:12:17.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2404e-05 2024-08-17T18:12:17.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:17.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.673640+0000", 2024-08-17T18:12:17.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1180000000000003e-06 2024-08-17T18:12:17.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.673683+0000", 2024-08-17T18:12:17.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3061e-05 2024-08-17T18:12:17.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:12:17.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:47.673690+0000", 2024-08-17T18:12:17.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.328e-06 2024-08-17T18:12:17.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.051769+0000", 2024-08-17T18:12:17.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3780791890000001 2024-08-17T18:12:17.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:17.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.054411+0000", 2024-08-17T18:12:17.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026415950000000001 2024-08-17T18:12:17.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.137821+0000", 2024-08-17T18:12:17.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.083410350999999994 2024-08-17T18:12:17.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:17.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.140473+0000", 2024-08-17T18:12:17.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002651863 2024-08-17T18:12:17.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.248857+0000", 2024-08-17T18:12:17.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.108384208 2024-08-17T18:12:17.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:17.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.250989+0000", 2024-08-17T18:12:17.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002131431 2024-08-17T18:12:17.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:12:17.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.255463+0000", 2024-08-17T18:12:17.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044744290000000003 2024-08-17T18:12:17.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.260314+0000", 2024-08-17T18:12:17.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048502850000000002 2024-08-17T18:12:17.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:17.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.277247+0000", 2024-08-17T18:12:17.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016933320000000002 2024-08-17T18:12:17.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.388740+0000", 2024-08-17T18:12:17.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11149316300000001 2024-08-17T18:12:17.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:17.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.390215+0000", 2024-08-17T18:12:17.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001475098 2024-08-17T18:12:17.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:17.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.396122+0000", 2024-08-17T18:12:17.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059071169999999999 2024-08-17T18:12:17.695 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.695 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:17.695 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.695 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140290 55.16 55:6b7eed24:::benchmark_data_smithi136_313476_object48304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1502)", 2024-08-17T18:12:17.696 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:48.586146+0000", 2024-08-17T18:12:17.696 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 268.97681156499999, 2024-08-17T18:12:17.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8181273789999999, 2024-08-17T18:12:17.696 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:17.696 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:17.696 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:17.696 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:12:17.696 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:12:17.696 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140290 2024-08-17T18:12:17.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.696 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:17.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:17.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:48.586146+0000", 2024-08-17T18:12:17.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:17.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:48.586146+0000", 2024-08-17T18:12:17.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:17.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:48.586145+0000", 2024-08-17T18:12:17.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:12:17.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:17.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:48.586152+0000", 2024-08-17T18:12:17.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0010000000000004e-06 2024-08-17T18:12:17.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:17.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:48.586167+0000", 2024-08-17T18:12:17.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5792999999999999e-05 2024-08-17T18:12:17.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:17.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:48.586176+0000", 2024-08-17T18:12:17.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2870000000000004e-06 2024-08-17T18:12:17.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.140715+0000", 2024-08-17T18:12:17.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5545393939999999 2024-08-17T18:12:17.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:17.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.143350+0000", 2024-08-17T18:12:17.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026350840000000002 2024-08-17T18:12:17.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.159079+0000", 2024-08-17T18:12:17.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015728829 2024-08-17T18:12:17.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:17.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.161595+0000", 2024-08-17T18:12:17.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002516346 2024-08-17T18:12:17.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.266505+0000", 2024-08-17T18:12:17.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.104909926 2024-08-17T18:12:17.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:17.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.268641+0000", 2024-08-17T18:12:17.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021357619999999998 2024-08-17T18:12:17.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:12:17.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.273043+0000", 2024-08-17T18:12:17.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044022840000000002 2024-08-17T18:12:17.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.276859+0000", 2024-08-17T18:12:17.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038157690000000001 2024-08-17T18:12:17.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:17.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.278368+0000", 2024-08-17T18:12:17.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001508793 2024-08-17T18:12:17.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.396986+0000", 2024-08-17T18:12:17.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.118618009 2024-08-17T18:12:17.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:17.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.398430+0000", 2024-08-17T18:12:17.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014436290000000001 2024-08-17T18:12:17.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:17.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.404274+0000", 2024-08-17T18:12:17.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058441459999999997 2024-08-17T18:12:17.703 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.703 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:17.703 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.704 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.45626.0:140307 55.16 55:6c589f5c:::benchmark_data_smithi136_313476_object48321:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1502)", 2024-08-17T18:12:17.704 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:07:48.690392+0000", 2024-08-17T18:12:17.704 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 268.87256613400001, 2024-08-17T18:12:17.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7685365280000003, 2024-08-17T18:12:17.704 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:17.704 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:17.704 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:17.704 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.45626", 2024-08-17T18:12:17.704 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:12:17.704 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 140307 2024-08-17T18:12:17.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.704 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:17.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:17.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:48.690392+0000", 2024-08-17T18:12:17.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:17.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:48.690392+0000", 2024-08-17T18:12:17.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:17.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:48.690390+0000", 2024-08-17T18:12:17.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:12:17.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:17.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:48.690397+0000", 2024-08-17T18:12:17.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5030000000000002e-06 2024-08-17T18:12:17.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:17.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:48.690410+0000", 2024-08-17T18:12:17.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3171e-05 2024-08-17T18:12:17.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:17.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:48.690416+0000", 2024-08-17T18:12:17.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5429999999999999e-06 2024-08-17T18:12:17.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.143646+0000", 2024-08-17T18:12:17.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.453229951 2024-08-17T18:12:17.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:12:17.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.146316+0000", 2024-08-17T18:12:17.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002669879 2024-08-17T18:12:17.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.339560+0000", 2024-08-17T18:12:17.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.193243955 2024-08-17T18:12:17.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:12:17.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.342864+0000", 2024-08-17T18:12:17.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033041469999999999 2024-08-17T18:12:17.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.413238+0000", 2024-08-17T18:12:17.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.070374183000000007 2024-08-17T18:12:17.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:17.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.415827+0000", 2024-08-17T18:12:17.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025888909999999998 2024-08-17T18:12:17.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:12:17.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.423214+0000", 2024-08-17T18:12:17.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073863410000000003 2024-08-17T18:12:17.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.434611+0000", 2024-08-17T18:12:17.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011397025 2024-08-17T18:12:17.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.435576+0000", 2024-08-17T18:12:17.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00096511299999999997 2024-08-17T18:12:17.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:17.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.451855+0000", 2024-08-17T18:12:17.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016278678000000001 2024-08-17T18:12:17.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:17.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.453176+0000", 2024-08-17T18:12:17.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001320994 2024-08-17T18:12:17.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:17.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:07:54.458928+0000", 2024-08-17T18:12:17.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057527510000000004 2024-08-17T18:12:17.711 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.711 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:17.712 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.712 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:178743 56.c 56:3082e7b9:::benchmark_data_smithi136_356456_object52039:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:12:17.712 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:03.741426+0000", 2024-08-17T18:12:17.712 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 13.821532276999999, 2024-08-17T18:12:17.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1284821909999998, 2024-08-17T18:12:17.712 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:17.712 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:17.712 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:17.712 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:12:17.712 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:12:17.712 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 178743 2024-08-17T18:12:17.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.713 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:17.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:17.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.741426+0000", 2024-08-17T18:12:17.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:17.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.741426+0000", 2024-08-17T18:12:17.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:17.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.741425+0000", 2024-08-17T18:12:17.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:12:17.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:17.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.741429+0000", 2024-08-17T18:12:17.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0180000000000003e-06 2024-08-17T18:12:17.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:17.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.741440+0000", 2024-08-17T18:12:17.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0987e-05 2024-08-17T18:12:17.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:17.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.741447+0000", 2024-08-17T18:12:17.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.764e-06 2024-08-17T18:12:17.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.741494+0000", 2024-08-17T18:12:17.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7239e-05 2024-08-17T18:12:17.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:12:17.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.741504+0000", 2024-08-17T18:12:17.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0508e-05 2024-08-17T18:12:17.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.636664+0000", 2024-08-17T18:12:17.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.89515925799999996 2024-08-17T18:12:17.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:12:17.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.636677+0000", 2024-08-17T18:12:17.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2901999999999999e-05 2024-08-17T18:12:17.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.833030+0000", 2024-08-17T18:12:17.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1963536420000001 2024-08-17T18:12:17.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:17.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.835819+0000", 2024-08-17T18:12:17.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027889910000000002 2024-08-17T18:12:17.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2,4", 2024-08-17T18:12:17.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.836104+0000", 2024-08-17T18:12:17.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028455599999999999 2024-08-17T18:12:17.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.851359+0000", 2024-08-17T18:12:17.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015255651 2024-08-17T18:12:17.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.851398+0000", 2024-08-17T18:12:17.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8257000000000003e-05 2024-08-17T18:12:17.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:17.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.856494+0000", 2024-08-17T18:12:17.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050968100000000002 2024-08-17T18:12:17.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.869682+0000", 2024-08-17T18:12:17.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013187211000000001 2024-08-17T18:12:17.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:17.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.869784+0000", 2024-08-17T18:12:17.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010231899999999999 2024-08-17T18:12:17.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:17.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.869908+0000", 2024-08-17T18:12:17.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012377699999999999 2024-08-17T18:12:17.720 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.721 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:17.721 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.721 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:178760 56.c 56:32895434:::benchmark_data_smithi136_356456_object52056:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:12:17.721 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:03.770242+0000", 2024-08-17T18:12:17.721 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 13.792715503, 2024-08-17T18:12:17.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.104038246, 2024-08-17T18:12:17.721 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:17.721 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:17.721 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:17.721 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:12:17.721 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:12:17.722 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 178760 2024-08-17T18:12:17.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.722 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:17.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:17.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.770242+0000", 2024-08-17T18:12:17.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:17.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.770242+0000", 2024-08-17T18:12:17.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:17.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.770242+0000", 2024-08-17T18:12:17.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:12:17.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:17.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.770246+0000", 2024-08-17T18:12:17.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7670000000000003e-06 2024-08-17T18:12:17.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:17.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.770257+0000", 2024-08-17T18:12:17.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0920999999999999e-05 2024-08-17T18:12:17.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:17.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.770265+0000", 2024-08-17T18:12:17.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.43e-06 2024-08-17T18:12:17.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.770301+0000", 2024-08-17T18:12:17.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6751000000000001e-05 2024-08-17T18:12:17.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:12:17.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.770309+0000", 2024-08-17T18:12:17.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4869999999999996e-06 2024-08-17T18:12:17.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.638358+0000", 2024-08-17T18:12:17.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.86804906800000003 2024-08-17T18:12:17.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:12:17.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.638368+0000", 2024-08-17T18:12:17.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9310000000000003e-06 2024-08-17T18:12:17.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.856525+0000", 2024-08-17T18:12:17.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2181566720000001 2024-08-17T18:12:17.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:17.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.856666+0000", 2024-08-17T18:12:17.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000141824 2024-08-17T18:12:17.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2,4", 2024-08-17T18:12:17.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.856813+0000", 2024-08-17T18:12:17.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000146098 2024-08-17T18:12:17.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.862289+0000", 2024-08-17T18:12:17.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054769010000000002 2024-08-17T18:12:17.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.869646+0000", 2024-08-17T18:12:17.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073564140000000004 2024-08-17T18:12:17.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.869950+0000", 2024-08-17T18:12:17.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000304006 2024-08-17T18:12:17.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:17.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.874059+0000", 2024-08-17T18:12:17.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041089150000000003 2024-08-17T18:12:17.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:17.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.874172+0000", 2024-08-17T18:12:17.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113752 2024-08-17T18:12:17.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:17.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.874281+0000", 2024-08-17T18:12:17.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108119 2024-08-17T18:12:17.729 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.729 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:17.729 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.729 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:178955 56.c 56:3789cb28:::benchmark_data_smithi136_356456_object52251:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1555)", 2024-08-17T18:12:17.729 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:06.073569+0000", 2024-08-17T18:12:17.729 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 11.489389298000001, 2024-08-17T18:12:17.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.061047085000000001, 2024-08-17T18:12:17.729 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:17.729 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:17.730 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:17.730 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:12:17.730 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:12:17.730 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 178955 2024-08-17T18:12:17.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.730 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:17.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:17.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.073569+0000", 2024-08-17T18:12:17.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:17.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.073569+0000", 2024-08-17T18:12:17.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:17.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.073567+0000", 2024-08-17T18:12:17.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:12:17.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:17.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.073572+0000", 2024-08-17T18:12:17.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2009999999999998e-06 2024-08-17T18:12:17.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:17.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.073584+0000", 2024-08-17T18:12:17.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1935e-05 2024-08-17T18:12:17.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:17.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.073591+0000", 2024-08-17T18:12:17.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9240000000000001e-06 2024-08-17T18:12:17.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.073628+0000", 2024-08-17T18:12:17.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7078e-05 2024-08-17T18:12:17.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:17.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.114908+0000", 2024-08-17T18:12:17.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.041279586 2024-08-17T18:12:17.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2,4", 2024-08-17T18:12:17.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.115057+0000", 2024-08-17T18:12:17.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014904999999999999 2024-08-17T18:12:17.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.125202+0000", 2024-08-17T18:12:17.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010144914999999999 2024-08-17T18:12:17.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:17.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.125211+0000", 2024-08-17T18:12:17.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8780000000000004e-06 2024-08-17T18:12:17.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.125245+0000", 2024-08-17T18:12:17.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3745000000000003e-05 2024-08-17T18:12:17.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.134354+0000", 2024-08-17T18:12:17.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0091098399999999993 2024-08-17T18:12:17.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:17.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.134502+0000", 2024-08-17T18:12:17.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014805999999999999 2024-08-17T18:12:17.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:17.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.134616+0000", 2024-08-17T18:12:17.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011325299999999999 2024-08-17T18:12:17.736 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.736 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:17.736 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.736 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:178958 56.2 56:40b32970:::benchmark_data_smithi136_356456_object52254:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1555)", 2024-08-17T18:12:17.736 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:06.073691+0000", 2024-08-17T18:12:17.736 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 11.489266848, 2024-08-17T18:12:17.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.061620680999999997, 2024-08-17T18:12:17.736 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:17.736 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:12:17.736 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:17.736 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:12:17.736 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:12:17.737 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 178958 2024-08-17T18:12:17.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.737 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:17.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:17.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.073691+0000", 2024-08-17T18:12:17.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:17.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.073691+0000", 2024-08-17T18:12:17.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:17.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:17.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.073690+0000", 2024-08-17T18:12:17.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:12:17.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:17.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.073695+0000", 2024-08-17T18:12:17.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2599999999999999e-06 2024-08-17T18:12:17.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:17.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.115546+0000", 2024-08-17T18:12:17.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.041851364000000002 2024-08-17T18:12:17.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:17.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.115558+0000", 2024-08-17T18:12:17.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1712000000000001e-05 2024-08-17T18:12:17.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:17.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.115841+0000", 2024-08-17T18:12:17.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028326000000000002 2024-08-17T18:12:17.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:17.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.115954+0000", 2024-08-17T18:12:17.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113335 2024-08-17T18:12:17.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-08-17T18:12:17.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.116057+0000", 2024-08-17T18:12:17.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103056 2024-08-17T18:12:17.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:17.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.122633+0000", 2024-08-17T18:12:17.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0065759340000000003 2024-08-17T18:12:17.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.122670+0000", 2024-08-17T18:12:17.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6597999999999998e-05 2024-08-17T18:12:17.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:17.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.134601+0000", 2024-08-17T18:12:17.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011931491000000001 2024-08-17T18:12:17.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:12:17.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.135272+0000", 2024-08-17T18:12:17.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00067051000000000003 2024-08-17T18:12:17.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:17.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:17.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:12:17.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:06.135312+0000", 2024-08-17T18:12:17.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9895999999999999e-05 2024-08-17T18:12:17.742 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.742 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:17.742 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.742 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:17.742 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:17.742 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:12:17.746 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:12:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:12:17] "GET /metrics HTTP/1.1" 200 36677 "" "Prometheus/2.51.0" 2024-08-17T18:12:17.757 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:17.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:17.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:17.859 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:17.902 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:17.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:17.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:18.004 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:17 smithi165 ceph-mon[105517]: pgmap v2902: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 4.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 884 op/s; 23356/200991 objects misplaced (11.620%); 15 MiB/s, 248 objects/s recovering 2024-08-17T18:12:18.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:18.046 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:18.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:18.147 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:18.163 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:17 smithi136 ceph-mon[109861]: pgmap v2902: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 4.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 884 op/s; 23356/200991 objects misplaced (11.620%); 15 MiB/s, 248 objects/s recovering 2024-08-17T18:12:18.164 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:17 smithi136 ceph-mon[115078]: pgmap v2902: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 4.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 884 op/s; 23356/200991 objects misplaced (11.620%); 15 MiB/s, 248 objects/s recovering 2024-08-17T18:12:18.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:18.189 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:18.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:18.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:18.334 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:18.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:18.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:18.436 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:18.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:18.477 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:18.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:18.578 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:18.620 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:18.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:18.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:18.722 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:18.743 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:18.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:18.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:18.845 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:18.890 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:18.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:18.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:18.960 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:12:18.965 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T18:12:18.993 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:19.042 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:19.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:19.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:19.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:19.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:19.175 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:19.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:19.276 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:19.308 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:19.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:19.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:19.410 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:19.450 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:19.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:19.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:19.552 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:19.586 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:19.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:19.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:19.688 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:19.719 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:19.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:19.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:19.821 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:19.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:19.863 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:19.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:19.964 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:19.999 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:20.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:20.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:19 smithi165 ceph-mon[105517]: pgmap v2903: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 4.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 910 op/s; 21008/194511 objects misplaced (10.800%); 18 MiB/s, 291 objects/s recovering 2024-08-17T18:12:20.053 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-17T18:12:20.100 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:20.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:20.166 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:20.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:20.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:19 smithi136 ceph-mon[109861]: pgmap v2903: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 4.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 910 op/s; 21008/194511 objects misplaced (10.800%); 18 MiB/s, 291 objects/s recovering 2024-08-17T18:12:20.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:19 smithi136 ceph-mon[115078]: pgmap v2903: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 4.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 910 op/s; 21008/194511 objects misplaced (10.800%); 18 MiB/s, 291 objects/s recovering 2024-08-17T18:12:20.268 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:20.321 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:20.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:20.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:20.382 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:12:20.422 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:20.435 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:12:20.462 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:20.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:20.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:20.564 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:20.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:20.607 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:20.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:20.708 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:20.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:20.747 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:20.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:20.848 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:20.882 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:20.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:20.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:20.984 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:21.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:20 smithi165 ceph-mon[105517]: from='client.46136 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:12:21.037 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:21.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:21.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:21.139 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:21.189 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:21.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:21.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:21.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:20 smithi136 ceph-mon[109861]: from='client.46136 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:12:21.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:20 smithi136 ceph-mon[115078]: from='client.46136 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:12:21.292 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:21.327 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:21.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:21.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:21.429 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:21.470 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:21.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:21.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:21.572 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:21.610 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:21.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:21.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:21.712 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:21.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:21.755 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:21.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:21.856 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:21.878 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":2903,"stamp":"2024-08-17T18:12:18.360744+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4248037795,"num_objects":64837,"num_object_clones":0,"num_object_copies":194511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":21008,"num_objects_unfound":0,"num_objects_dirty":64837,"num_whiteouts":0,"num_read":36,"num_read_kb":76,"num_write":188677,"num_write_kb":8109118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3274,"num_bytes_recovered":211106278,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99919,"ondisk_log_size":99919,"up":297,"acting":297,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":297,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":15651324,"kb_used_data":11154124,"kb_used_omap":284,"kb_used_meta":4496803,"kb_avail":546819588,"statfs":{"total":575970213888,"available":559943258112,"internally_reserved":0,"allocated":11421822976,"data_stored":11388056065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":290963,"internal_metadata":4604727149},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-597164032,"num_objects":-9112,"num_object_clones":0,"num_object_copies":-27336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1538,"num_objects_unfound":0,"num_objects_dirty":-9112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9112,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2915,"num_bytes_recovered":190906387,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003892"},"pg_stats":[{"pgid":"56.f","version":"1558'12018","reported_seq":14268,"reported_epoch":1558,"state":"active+remapped+backfilling","last_fresh":"2024-08-17T18:12:17.940171+0000","last_change":"2024-08-17T18:12:05.852978+0000","last_active":"2024-08-17T18:12:17.940171+0000","last_peered":"2024-08-17T18:12:17.940171+0000","last_clean":"2024-08-17T18:12:03.673812+0000","last_became_active":"2024-08-17T18:12:05.739165+0000","last_became_peered":"2024-08-17T18:12:05.739165+0000","last_unstale":"2024-08-17T18:12:17.940171+0000","last_undegraded":"2024-08-17T18:12:17.940171+0000","last_fullsized":"2024-08-17T18:12:17.940171+0000","mapping_epoch":1554,"log_start":"1544'5575","ondisk_log_start":"1544'5575","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6443,"log_dups_size":0,"ondisk_log_size":6443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:34:06.197478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265289728,"num_objects":4048,"num_object_clones":0,"num_object_copies":12144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3038,"num_objects_unfound":0,"num_objects_dirty":4048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12018,"num_write_kb":514112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1094,"num_bytes_recovered":71696384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,1],"avail_no_missing":["4","1","6"],"object_location_counts":[{"shards":"1,4,6","objects":4048}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"56.e","version":"1558'11799","reported_seq":13973,"reported_epoch":1558,"state":"active+remapped+backfilling","last_fresh":"2024-08-17T18:12:17.096534+0000","last_change":"2024-08-17T18:12:05.812481+0000","last_active":"2024-08-17T18:12:17.096534+0000","last_peered":"2024-08-17T18:12:17.096534+0000","last_clean":"2024-08-17T18:12:03.645603+0000","last_became_active":"2024-08-17T18:12:05.749772+0000","last_became_peered":"2024-08-17T18:12:05.749772+0000","last_unstale":"2024-08-17T18:12:17.096534+0000","last_undegraded":"2024-08-17T18:12:17.096534+0000","last_fullsized":"2024-08-17T18:12:17.096534+0000","mapping_epoch":1554,"log_start":"1543'4800","ondisk_log_start":"1543'4800","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6999,"log_dups_size":0,"ondisk_log_size":6999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263651328,"num_objects":4023,"num_object_clones":0,"num_object_copies":12069,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3042,"num_objects_unfound":0,"num_objects_dirty":4023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11799,"num_write_kb":506304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1056,"num_bytes_recovered":69206016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,1],"avail_no_missing":["2","1","6"],"object_location_counts":[{"shards":"1,2,6","objects":4023}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"56.d","version":"1558'11766","reported_seq":11825,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:16.134749+0000","last_change":"2024-08-17T18:09:28.696225+0000","last_active":"2024-08-17T18:12:16.134749+0000","last_peered":"2024-08-17T18:12:16.134749+0000","last_clean":"2024-08-17T18:12:16.134749+0000","last_became_active":"2024-08-17T18:09:28.695916+0000","last_became_peered":"2024-08-17T18:09:28.695916+0000","last_unstale":"2024-08-17T18:12:16.134749+0000","last_undegraded":"2024-08-17T18:12:16.134749+0000","last_fullsized":"2024-08-17T18:12:16.134749+0000","mapping_epoch":1535,"log_start":"1544'5547","ondisk_log_start":"1544'5547","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6219,"log_dups_size":0,"ondisk_log_size":6219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:00:20.159452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266797080,"num_objects":4072,"num_object_clones":0,"num_object_copies":12216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4072,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11766,"num_write_kb":506753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"56.c","version":"1558'11724","reported_seq":13900,"reported_epoch":1558,"state":"active+remapped+backfilling","last_fresh":"2024-08-17T18:12:14.285790+0000","last_change":"2024-08-17T18:12:05.862507+0000","last_active":"2024-08-17T18:12:14.285790+0000","last_peered":"2024-08-17T18:12:14.285790+0000","last_clean":"2024-08-17T18:12:03.623805+0000","last_became_active":"2024-08-17T18:12:05.766857+0000","last_became_peered":"2024-08-17T18:12:05.766857+0000","last_unstale":"2024-08-17T18:12:14.285790+0000","last_undegraded":"2024-08-17T18:12:14.285790+0000","last_fullsized":"2024-08-17T18:12:14.285790+0000","mapping_epoch":1554,"log_start":"1544'5918","ondisk_log_start":"1544'5918","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":5806,"log_dups_size":0,"ondisk_log_size":5806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:29:37.641681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274333696,"num_objects":4186,"num_object_clones":0,"num_object_copies":12558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3194,"num_objects_unfound":0,"num_objects_dirty":4186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11724,"num_write_kb":509120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1058,"num_bytes_recovered":69337088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,1],"avail_no_missing":["5","1","2"],"object_location_counts":[{"shards":"1,2,5","objects":4186}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.b","version":"1558'11898","reported_seq":11931,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:17.927816+0000","last_change":"2024-08-17T18:09:28.627309+0000","last_active":"2024-08-17T18:12:17.927816+0000","last_peered":"2024-08-17T18:12:17.927816+0000","last_clean":"2024-08-17T18:12:17.927816+0000","last_became_active":"2024-08-17T18:09:28.626970+0000","last_became_peered":"2024-08-17T18:09:28.626970+0000","last_unstale":"2024-08-17T18:12:17.927816+0000","last_undegraded":"2024-08-17T18:12:17.927816+0000","last_fullsized":"2024-08-17T18:12:17.927816+0000","mapping_epoch":1535,"log_start":"1543'5431","ondisk_log_start":"1543'5431","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6467,"log_dups_size":0,"ondisk_log_size":6467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:47:24.480441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253493248,"num_objects":3868,"num_object_clones":0,"num_object_copies":11604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11898,"num_write_kb":504512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"56.a","version":"1558'11920","reported_seq":11978,"reported_epoch":1558,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:12:17.940569+0000","last_change":"2024-08-17T18:12:05.848842+0000","last_active":"2024-08-17T18:12:17.940569+0000","last_peered":"2024-08-17T18:12:17.940569+0000","last_clean":"2024-08-17T18:12:03.669468+0000","last_became_active":"2024-08-17T18:12:05.780262+0000","last_became_peered":"2024-08-17T18:12:05.780262+0000","last_unstale":"2024-08-17T18:12:17.940569+0000","last_undegraded":"2024-08-17T18:12:17.940569+0000","last_fullsized":"2024-08-17T18:12:17.940569+0000","mapping_epoch":1554,"log_start":"1543'5509","ondisk_log_start":"1543'5509","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6411,"log_dups_size":0,"ondisk_log_size":6411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:50:58.206192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251265024,"num_objects":3834,"num_object_clones":0,"num_object_copies":11502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3834,"num_objects_unfound":0,"num_objects_dirty":3834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11920,"num_write_kb":504128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,1],"avail_no_missing":["4","1","6"],"object_location_counts":[{"shards":"1,4,6","objects":3834}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"56.9","version":"1558'11283","reported_seq":11315,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:14.284318+0000","last_change":"2024-08-17T18:09:28.626545+0000","last_active":"2024-08-17T18:12:14.284318+0000","last_peered":"2024-08-17T18:12:14.284318+0000","last_clean":"2024-08-17T18:12:14.284318+0000","last_became_active":"2024-08-17T18:09:28.626300+0000","last_became_peered":"2024-08-17T18:09:28.626300+0000","last_unstale":"2024-08-17T18:12:14.284318+0000","last_undegraded":"2024-08-17T18:12:14.284318+0000","last_fullsized":"2024-08-17T18:12:14.284318+0000","mapping_epoch":1535,"log_start":"1544'5442","ondisk_log_start":"1544'5442","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":5841,"log_dups_size":0,"ondisk_log_size":5841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:34:50.839206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268894208,"num_objects":4103,"num_object_clones":0,"num_object_copies":12309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11283,"num_write_kb":492352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.8","version":"1558'11773","reported_seq":11831,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:16.154045+0000","last_change":"2024-08-17T18:09:28.697184+0000","last_active":"2024-08-17T18:12:16.154045+0000","last_peered":"2024-08-17T18:12:16.154045+0000","last_clean":"2024-08-17T18:12:16.154045+0000","last_became_active":"2024-08-17T18:09:28.696911+0000","last_became_peered":"2024-08-17T18:09:28.696911+0000","last_unstale":"2024-08-17T18:12:16.154045+0000","last_undegraded":"2024-08-17T18:12:16.154045+0000","last_fullsized":"2024-08-17T18:12:16.154045+0000","mapping_epoch":1535,"log_start":"1544'5569","ondisk_log_start":"1544'5569","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6204,"log_dups_size":0,"ondisk_log_size":6204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:39:59.778610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263651328,"num_objects":4023,"num_object_clones":0,"num_object_copies":12069,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11773,"num_write_kb":505472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"56.7","version":"1558'11690","reported_seq":11748,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:16.158844+0000","last_change":"2024-08-17T18:09:28.628236+0000","last_active":"2024-08-17T18:12:16.158844+0000","last_peered":"2024-08-17T18:12:16.158844+0000","last_clean":"2024-08-17T18:12:16.158844+0000","last_became_active":"2024-08-17T18:09:28.627952+0000","last_became_peered":"2024-08-17T18:09:28.627952+0000","last_unstale":"2024-08-17T18:12:16.158844+0000","last_undegraded":"2024-08-17T18:12:16.158844+0000","last_fullsized":"2024-08-17T18:12:16.158844+0000","mapping_epoch":1535,"log_start":"1544'5546","ondisk_log_start":"1544'5546","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6144,"log_dups_size":0,"ondisk_log_size":6144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:04:20.822971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263061504,"num_objects":4014,"num_object_clones":0,"num_object_copies":12042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11690,"num_write_kb":502528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"56.6","version":"1558'11791","reported_seq":11824,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:14.259868+0000","last_change":"2024-08-17T18:09:28.629945+0000","last_active":"2024-08-17T18:12:14.259868+0000","last_peered":"2024-08-17T18:12:14.259868+0000","last_clean":"2024-08-17T18:12:14.259868+0000","last_became_active":"2024-08-17T18:09:28.629553+0000","last_became_peered":"2024-08-17T18:09:28.629553+0000","last_unstale":"2024-08-17T18:12:14.259868+0000","last_undegraded":"2024-08-17T18:12:14.259868+0000","last_fullsized":"2024-08-17T18:12:14.259868+0000","mapping_epoch":1535,"log_start":"1544'6016","ondisk_log_start":"1544'6016","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":5775,"log_dups_size":0,"ondisk_log_size":5775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:00:36.875044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277413888,"num_objects":4233,"num_object_clones":0,"num_object_copies":12699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11791,"num_write_kb":512768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.5","version":"1558'11854","reported_seq":11912,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:17.012123+0000","last_change":"2024-08-17T18:09:28.697212+0000","last_active":"2024-08-17T18:12:17.012123+0000","last_peered":"2024-08-17T18:12:17.012123+0000","last_clean":"2024-08-17T18:12:17.012123+0000","last_became_active":"2024-08-17T18:09:28.696769+0000","last_became_peered":"2024-08-17T18:09:28.696769+0000","last_unstale":"2024-08-17T18:12:17.012123+0000","last_undegraded":"2024-08-17T18:12:17.012123+0000","last_fullsized":"2024-08-17T18:12:17.012123+0000","mapping_epoch":1535,"log_start":"1544'6314","ondisk_log_start":"1544'6314","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":5540,"log_dups_size":0,"ondisk_log_size":5540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:30:22.827483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263454720,"num_objects":4020,"num_object_clones":0,"num_object_copies":12060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11854,"num_write_kb":507968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.4","version":"1558'11879","reported_seq":11941,"reported_epoch":1558,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:12:17.092202+0000","last_change":"2024-08-17T18:12:05.794339+0000","last_active":"2024-08-17T18:12:17.092202+0000","last_peered":"2024-08-17T18:12:17.092202+0000","last_clean":"2024-08-17T18:12:03.656945+0000","last_became_active":"2024-08-17T18:12:05.767724+0000","last_became_peered":"2024-08-17T18:12:05.767724+0000","last_unstale":"2024-08-17T18:12:17.092202+0000","last_undegraded":"2024-08-17T18:12:17.092202+0000","last_fullsized":"2024-08-17T18:12:17.092202+0000","mapping_epoch":1554,"log_start":"1543'4900","ondisk_log_start":"1543'4900","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":6979,"log_dups_size":0,"ondisk_log_size":6979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259850240,"num_objects":3965,"num_object_clones":0,"num_object_copies":11895,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3965,"num_objects_unfound":0,"num_objects_dirty":3965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":507008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,1],"avail_no_missing":["2","1","4"],"object_location_counts":[{"shards":"1,2,4","objects":3965}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"56.3","version":"1558'11759","reported_seq":11820,"reported_epoch":1558,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:12:17.352069+0000","last_change":"2024-08-17T18:12:05.846089+0000","last_active":"2024-08-17T18:12:17.352069+0000","last_peered":"2024-08-17T18:12:17.352069+0000","last_clean":"2024-08-17T18:12:03.598979+0000","last_became_active":"2024-08-17T18:12:05.780420+0000","last_became_peered":"2024-08-17T18:12:05.780420+0000","last_unstale":"2024-08-17T18:12:17.352069+0000","last_undegraded":"2024-08-17T18:12:17.352069+0000","last_fullsized":"2024-08-17T18:12:17.352069+0000","mapping_epoch":1554,"log_start":"1544'5827","ondisk_log_start":"1544'5827","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":5932,"log_dups_size":0,"ondisk_log_size":5932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:51:43.057449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257884160,"num_objects":3935,"num_object_clones":0,"num_object_copies":11805,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3935,"num_objects_unfound":0,"num_objects_dirty":3935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11759,"num_write_kb":502208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,1],"avail_no_missing":["6","1","5"],"object_location_counts":[{"shards":"1,5,6","objects":3935}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"56.2","version":"1558'11852","reported_seq":11885,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:14.269199+0000","last_change":"2024-08-17T18:09:28.628711+0000","last_active":"2024-08-17T18:12:14.269199+0000","last_peered":"2024-08-17T18:12:14.269199+0000","last_clean":"2024-08-17T18:12:14.269199+0000","last_became_active":"2024-08-17T18:09:28.628387+0000","last_became_peered":"2024-08-17T18:09:28.628387+0000","last_unstale":"2024-08-17T18:12:14.269199+0000","last_undegraded":"2024-08-17T18:12:14.269199+0000","last_fullsized":"2024-08-17T18:12:14.269199+0000","mapping_epoch":1535,"log_start":"1544'6053","ondisk_log_start":"1544'6053","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":5799,"log_dups_size":0,"ondisk_log_size":5799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:26:06.412021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272760832,"num_objects":4162,"num_object_clones":0,"num_object_copies":12486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11852,"num_write_kb":512448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.1","version":"1558'11965","reported_seq":11998,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:14.253894+0000","last_change":"2024-08-17T18:09:28.629660+0000","last_active":"2024-08-17T18:12:14.253894+0000","last_peered":"2024-08-17T18:12:14.253894+0000","last_clean":"2024-08-17T18:12:14.253894+0000","last_became_active":"2024-08-17T18:09:28.629359+0000","last_became_peered":"2024-08-17T18:09:28.629359+0000","last_unstale":"2024-08-17T18:12:14.253894+0000","last_undegraded":"2024-08-17T18:12:14.253894+0000","last_fullsized":"2024-08-17T18:12:14.253894+0000","mapping_epoch":1535,"log_start":"1544'6123","ondisk_log_start":"1544'6123","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":5842,"log_dups_size":0,"ondisk_log_size":5842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:05:58.848611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274006016,"num_objects":4181,"num_object_clones":0,"num_object_copies":12543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11965,"num_write_kb":516672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.0","version":"1558'11622","reported_seq":11655,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:14.269042+0000","last_change":"2024-08-17T18:09:28.697077+0000","last_active":"2024-08-17T18:12:14.269042+0000","last_peered":"2024-08-17T18:12:14.269042+0000","last_clean":"2024-08-17T18:12:14.269042+0000","last_became_active":"2024-08-17T18:09:28.696676+0000","last_became_peered":"2024-08-17T18:09:28.696676+0000","last_unstale":"2024-08-17T18:12:14.269042+0000","last_undegraded":"2024-08-17T18:12:14.269042+0000","last_fullsized":"2024-08-17T18:12:14.269042+0000","mapping_epoch":1535,"log_start":"1544'5802","ondisk_log_start":"1544'5802","created":1535,"last_epoch_clean":1536,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:09:27.605039+0000","last_clean_scrub_stamp":"2024-08-17T18:09:27.605039+0000","objects_scrubbed":0,"log_size":5820,"log_dups_size":0,"ondisk_log_size":5820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:09.350570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271581184,"num_objects":4144,"num_object_clones":0,"num_object_copies":12432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11622,"num_write_kb":504512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":260,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.704480+0000","last_change":"2024-08-17T18:07:48.546202+0000","last_active":"2024-08-17T18:12:09.704480+0000","last_peered":"2024-08-17T18:12:09.704480+0000","last_clean":"2024-08-17T18:12:09.704480+0000","last_became_active":"2024-08-17T18:07:48.531246+0000","last_became_peered":"2024-08-17T18:07:48.531246+0000","last_unstale":"2024-08-17T18:12:09.704480+0000","last_undegraded":"2024-08-17T18:12:09.704480+0000","last_fullsized":"2024-08-17T18:12:09.704480+0000","mapping_epoch":1501,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:30.944362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1571,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.637244+0000","last_change":"2024-08-17T18:12:04.648224+0000","last_active":"2024-08-17T18:12:07.637244+0000","last_peered":"2024-08-17T18:12:07.637244+0000","last_clean":"2024-08-17T18:12:07.637244+0000","last_became_active":"2024-08-17T18:12:04.647903+0000","last_became_peered":"2024-08-17T18:12:04.647903+0000","last_unstale":"2024-08-17T18:12:07.637244+0000","last_undegraded":"2024-08-17T18:12:07.637244+0000","last_fullsized":"2024-08-17T18:12:07.637244+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:44:48.596376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":2436,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.708809+0000","last_change":"2024-08-17T18:08:37.082356+0000","last_active":"2024-08-17T18:12:09.708809+0000","last_peered":"2024-08-17T18:12:09.708809+0000","last_clean":"2024-08-17T18:12:09.708809+0000","last_became_active":"2024-08-17T18:08:37.082046+0000","last_became_peered":"2024-08-17T18:08:37.082046+0000","last_unstale":"2024-08-17T18:12:09.708809+0000","last_undegraded":"2024-08-17T18:12:09.708809+0000","last_fullsized":"2024-08-17T18:12:09.708809+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:46:13.932917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2458,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.706914+0000","last_change":"2024-08-17T18:04:14.540774+0000","last_active":"2024-08-17T18:12:09.706914+0000","last_peered":"2024-08-17T18:12:09.706914+0000","last_clean":"2024-08-17T18:12:09.706914+0000","last_became_active":"2024-08-17T18:04:14.539778+0000","last_became_peered":"2024-08-17T18:04:14.539778+0000","last_unstale":"2024-08-17T18:12:09.706914+0000","last_undegraded":"2024-08-17T18:12:09.706914+0000","last_fullsized":"2024-08-17T18:12:09.706914+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:55:02.863360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":110,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.635209+0000","last_change":"2024-08-17T18:04:43.777705+0000","last_active":"2024-08-17T18:12:07.635209+0000","last_peered":"2024-08-17T18:12:07.635209+0000","last_clean":"2024-08-17T18:12:07.635209+0000","last_became_active":"2024-08-17T18:04:43.771283+0000","last_became_peered":"2024-08-17T18:04:43.771283+0000","last_unstale":"2024-08-17T18:12:07.635209+0000","last_undegraded":"2024-08-17T18:12:07.635209+0000","last_fullsized":"2024-08-17T18:12:07.635209+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:13.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1703,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.707464+0000","last_change":"2024-08-17T18:04:14.534963+0000","last_active":"2024-08-17T18:12:09.707464+0000","last_peered":"2024-08-17T18:12:09.707464+0000","last_clean":"2024-08-17T18:12:09.707464+0000","last_became_active":"2024-08-17T18:04:14.534634+0000","last_became_peered":"2024-08-17T18:04:14.534634+0000","last_unstale":"2024-08-17T18:12:09.707464+0000","last_undegraded":"2024-08-17T18:12:09.707464+0000","last_fullsized":"2024-08-17T18:12:09.707464+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:33:27.287860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":124,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.695105+0000","last_change":"2024-08-17T18:12:04.655551+0000","last_active":"2024-08-17T18:12:09.695105+0000","last_peered":"2024-08-17T18:12:09.695105+0000","last_clean":"2024-08-17T18:12:09.695105+0000","last_became_active":"2024-08-17T18:12:04.646075+0000","last_became_peered":"2024-08-17T18:12:04.646075+0000","last_unstale":"2024-08-17T18:12:09.695105+0000","last_undegraded":"2024-08-17T18:12:09.695105+0000","last_fullsized":"2024-08-17T18:12:09.695105+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:19:19.300135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1558,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.676896+0000","last_change":"2024-08-17T18:12:04.644282+0000","last_active":"2024-08-17T18:12:07.676896+0000","last_peered":"2024-08-17T18:12:07.676896+0000","last_clean":"2024-08-17T18:12:07.676896+0000","last_became_active":"2024-08-17T18:12:04.643972+0000","last_became_peered":"2024-08-17T18:12:04.643972+0000","last_unstale":"2024-08-17T18:12:07.676896+0000","last_undegraded":"2024-08-17T18:12:07.676896+0000","last_fullsized":"2024-08-17T18:12:07.676896+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:08:04.343895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":222,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.704868+0000","last_change":"2024-08-17T18:12:04.653631+0000","last_active":"2024-08-17T18:12:09.704868+0000","last_peered":"2024-08-17T18:12:09.704868+0000","last_clean":"2024-08-17T18:12:09.704868+0000","last_became_active":"2024-08-17T18:12:04.653089+0000","last_became_peered":"2024-08-17T18:12:04.653089+0000","last_unstale":"2024-08-17T18:12:09.704868+0000","last_undegraded":"2024-08-17T18:12:09.704868+0000","last_fullsized":"2024-08-17T18:12:09.704868+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:32:25.425133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":132,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.696379+0000","last_change":"2024-08-17T18:12:04.645675+0000","last_active":"2024-08-17T18:12:09.696379+0000","last_peered":"2024-08-17T18:12:09.696379+0000","last_clean":"2024-08-17T18:12:09.696379+0000","last_became_active":"2024-08-17T18:12:04.645312+0000","last_became_peered":"2024-08-17T18:12:04.645312+0000","last_unstale":"2024-08-17T18:12:09.696379+0000","last_undegraded":"2024-08-17T18:12:09.696379+0000","last_fullsized":"2024-08-17T18:12:09.696379+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:46:23.762194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1575,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.676114+0000","last_change":"2024-08-17T18:12:04.638043+0000","last_active":"2024-08-17T18:12:07.676114+0000","last_peered":"2024-08-17T18:12:07.676114+0000","last_clean":"2024-08-17T18:12:07.676114+0000","last_became_active":"2024-08-17T18:12:04.637575+0000","last_became_peered":"2024-08-17T18:12:04.637575+0000","last_unstale":"2024-08-17T18:12:07.676114+0000","last_undegraded":"2024-08-17T18:12:07.676114+0000","last_fullsized":"2024-08-17T18:12:07.676114+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:26:53.888302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2431,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.709853+0000","last_change":"2024-08-17T18:12:04.653780+0000","last_active":"2024-08-17T18:12:09.709853+0000","last_peered":"2024-08-17T18:12:09.709853+0000","last_clean":"2024-08-17T18:12:09.709853+0000","last_became_active":"2024-08-17T18:12:04.653358+0000","last_became_peered":"2024-08-17T18:12:04.653358+0000","last_unstale":"2024-08-17T18:12:09.709853+0000","last_undegraded":"2024-08-17T18:12:09.709853+0000","last_fullsized":"2024-08-17T18:12:09.709853+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:03:09.767610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":680,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.707006+0000","last_change":"2024-08-17T18:12:04.651674+0000","last_active":"2024-08-17T18:12:09.707006+0000","last_peered":"2024-08-17T18:12:09.707006+0000","last_clean":"2024-08-17T18:12:09.707006+0000","last_became_active":"2024-08-17T18:12:04.651335+0000","last_became_peered":"2024-08-17T18:12:04.651335+0000","last_unstale":"2024-08-17T18:12:09.707006+0000","last_undegraded":"2024-08-17T18:12:09.707006+0000","last_fullsized":"2024-08-17T18:12:09.707006+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:50:39.957541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":236,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.704448+0000","last_change":"2024-08-17T18:12:04.651181+0000","last_active":"2024-08-17T18:12:09.704448+0000","last_peered":"2024-08-17T18:12:09.704448+0000","last_clean":"2024-08-17T18:12:09.704448+0000","last_became_active":"2024-08-17T18:12:04.650792+0000","last_became_peered":"2024-08-17T18:12:04.650792+0000","last_unstale":"2024-08-17T18:12:09.704448+0000","last_undegraded":"2024-08-17T18:12:09.704448+0000","last_fullsized":"2024-08-17T18:12:09.704448+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:04:01.114344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2445,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.705576+0000","last_change":"2024-08-17T18:12:04.653558+0000","last_active":"2024-08-17T18:12:09.705576+0000","last_peered":"2024-08-17T18:12:09.705576+0000","last_clean":"2024-08-17T18:12:09.705576+0000","last_became_active":"2024-08-17T18:12:04.651871+0000","last_became_peered":"2024-08-17T18:12:04.651871+0000","last_unstale":"2024-08-17T18:12:09.705576+0000","last_undegraded":"2024-08-17T18:12:09.705576+0000","last_fullsized":"2024-08-17T18:12:09.705576+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:01:49.758940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1572,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.677231+0000","last_change":"2024-08-17T18:12:04.641507+0000","last_active":"2024-08-17T18:12:07.677231+0000","last_peered":"2024-08-17T18:12:07.677231+0000","last_clean":"2024-08-17T18:12:07.677231+0000","last_became_active":"2024-08-17T18:12:04.641206+0000","last_became_peered":"2024-08-17T18:12:04.641206+0000","last_unstale":"2024-08-17T18:12:07.677231+0000","last_undegraded":"2024-08-17T18:12:07.677231+0000","last_fullsized":"2024-08-17T18:12:07.677231+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:35:15.727854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2405,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.705761+0000","last_change":"2024-08-17T18:12:04.653294+0000","last_active":"2024-08-17T18:12:09.705761+0000","last_peered":"2024-08-17T18:12:09.705761+0000","last_clean":"2024-08-17T18:12:09.705761+0000","last_became_active":"2024-08-17T18:12:04.650327+0000","last_became_peered":"2024-08-17T18:12:04.650327+0000","last_unstale":"2024-08-17T18:12:09.705761+0000","last_undegraded":"2024-08-17T18:12:09.705761+0000","last_fullsized":"2024-08-17T18:12:09.705761+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:37:00.934195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":222,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.704583+0000","last_change":"2024-08-17T18:04:43.777736+0000","last_active":"2024-08-17T18:12:09.704583+0000","last_peered":"2024-08-17T18:12:09.704583+0000","last_clean":"2024-08-17T18:12:09.704583+0000","last_became_active":"2024-08-17T18:04:43.777438+0000","last_became_peered":"2024-08-17T18:04:43.777438+0000","last_unstale":"2024-08-17T18:12:09.704583+0000","last_undegraded":"2024-08-17T18:12:09.704583+0000","last_fullsized":"2024-08-17T18:12:09.704583+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:24:01.255927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2378,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.707076+0000","last_change":"2024-08-17T18:04:43.915398+0000","last_active":"2024-08-17T18:12:09.707076+0000","last_peered":"2024-08-17T18:12:09.707076+0000","last_clean":"2024-08-17T18:12:09.707076+0000","last_became_active":"2024-08-17T18:04:43.915212+0000","last_became_peered":"2024-08-17T18:04:43.915212+0000","last_unstale":"2024-08-17T18:12:09.707076+0000","last_undegraded":"2024-08-17T18:12:09.707076+0000","last_fullsized":"2024-08-17T18:12:09.707076+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:43:26.901830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":133,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.680305+0000","last_change":"2024-08-17T18:12:04.645565+0000","last_active":"2024-08-17T18:12:07.680305+0000","last_peered":"2024-08-17T18:12:07.680305+0000","last_clean":"2024-08-17T18:12:07.680305+0000","last_became_active":"2024-08-17T18:12:04.644807+0000","last_became_peered":"2024-08-17T18:12:04.644807+0000","last_unstale":"2024-08-17T18:12:07.680305+0000","last_undegraded":"2024-08-17T18:12:07.680305+0000","last_fullsized":"2024-08-17T18:12:07.680305+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:51:31.003257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":535,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.709937+0000","last_change":"2024-08-17T18:04:14.555729+0000","last_active":"2024-08-17T18:12:09.709937+0000","last_peered":"2024-08-17T18:12:09.709937+0000","last_clean":"2024-08-17T18:12:09.709937+0000","last_became_active":"2024-08-17T18:04:14.536678+0000","last_became_peered":"2024-08-17T18:04:14.536678+0000","last_unstale":"2024-08-17T18:12:09.709937+0000","last_undegraded":"2024-08-17T18:12:09.709937+0000","last_fullsized":"2024-08-17T18:12:09.709937+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:47:29.424778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2371,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.638697+0000","last_change":"2024-08-17T18:12:04.649882+0000","last_active":"2024-08-17T18:12:07.638697+0000","last_peered":"2024-08-17T18:12:07.638697+0000","last_clean":"2024-08-17T18:12:07.638697+0000","last_became_active":"2024-08-17T18:12:04.649185+0000","last_became_peered":"2024-08-17T18:12:04.649185+0000","last_unstale":"2024-08-17T18:12:07.638697+0000","last_undegraded":"2024-08-17T18:12:07.638697+0000","last_fullsized":"2024-08-17T18:12:07.638697+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:42.653827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1564,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.676704+0000","last_change":"2024-08-17T18:03:51.292044+0000","last_active":"2024-08-17T18:12:07.676704+0000","last_peered":"2024-08-17T18:12:07.676704+0000","last_clean":"2024-08-17T18:12:07.676704+0000","last_became_active":"2024-08-17T18:03:51.268271+0000","last_became_peered":"2024-08-17T18:03:51.268271+0000","last_unstale":"2024-08-17T18:12:07.676704+0000","last_undegraded":"2024-08-17T18:12:07.676704+0000","last_fullsized":"2024-08-17T18:12:07.676704+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:00.892410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4756,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1545,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.676963+0000","last_change":"2024-08-17T18:12:04.639269+0000","last_active":"2024-08-17T18:12:07.676963+0000","last_peered":"2024-08-17T18:12:07.676963+0000","last_clean":"2024-08-17T18:12:07.676963+0000","last_became_active":"2024-08-17T18:12:04.638752+0000","last_became_peered":"2024-08-17T18:12:04.638752+0000","last_unstale":"2024-08-17T18:12:07.676963+0000","last_undegraded":"2024-08-17T18:12:07.676963+0000","last_fullsized":"2024-08-17T18:12:07.676963+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:50:00.462972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"347'149","reported_seq":860,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.707558+0000","last_change":"2024-08-17T18:09:15.592511+0000","last_active":"2024-08-17T18:12:09.707558+0000","last_peered":"2024-08-17T18:12:09.707558+0000","last_clean":"2024-08-17T18:12:09.707558+0000","last_became_active":"2024-08-17T18:09:15.592099+0000","last_became_peered":"2024-08-17T18:09:15.592099+0000","last_unstale":"2024-08-17T18:12:09.707558+0000","last_undegraded":"2024-08-17T18:12:09.707558+0000","last_fullsized":"2024-08-17T18:12:09.707558+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:57.271286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:57.271286+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:58:58.662516+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1560,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.637377+0000","last_change":"2024-08-17T18:04:43.766724+0000","last_active":"2024-08-17T18:12:07.637377+0000","last_peered":"2024-08-17T18:12:07.637377+0000","last_clean":"2024-08-17T18:12:07.637377+0000","last_became_active":"2024-08-17T18:04:43.766428+0000","last_became_peered":"2024-08-17T18:04:43.766428+0000","last_unstale":"2024-08-17T18:12:07.637377+0000","last_undegraded":"2024-08-17T18:12:07.637377+0000","last_fullsized":"2024-08-17T18:12:07.637377+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:09:49.102940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1573,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.677300+0000","last_change":"2024-08-17T18:03:51.279963+0000","last_active":"2024-08-17T18:12:07.677300+0000","last_peered":"2024-08-17T18:12:07.677300+0000","last_clean":"2024-08-17T18:12:07.677300+0000","last_became_active":"2024-08-17T18:03:51.279506+0000","last_became_peered":"2024-08-17T18:03:51.279506+0000","last_unstale":"2024-08-17T18:12:07.677300+0000","last_undegraded":"2024-08-17T18:12:07.677300+0000","last_fullsized":"2024-08-17T18:12:07.677300+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:20:07.304042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":230,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.704526+0000","last_change":"2024-08-17T18:12:04.653301+0000","last_active":"2024-08-17T18:12:09.704526+0000","last_peered":"2024-08-17T18:12:09.704526+0000","last_clean":"2024-08-17T18:12:09.704526+0000","last_became_active":"2024-08-17T18:12:04.642708+0000","last_became_peered":"2024-08-17T18:12:04.642708+0000","last_unstale":"2024-08-17T18:12:09.704526+0000","last_undegraded":"2024-08-17T18:12:09.704526+0000","last_fullsized":"2024-08-17T18:12:09.704526+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:22:49.885441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"347'149","reported_seq":830,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.637605+0000","last_change":"2024-08-17T18:09:15.592395+0000","last_active":"2024-08-17T18:12:07.637605+0000","last_peered":"2024-08-17T18:12:07.637605+0000","last_clean":"2024-08-17T18:12:07.637605+0000","last_became_active":"2024-08-17T18:09:15.591773+0000","last_became_peered":"2024-08-17T18:09:15.591773+0000","last_unstale":"2024-08-17T18:12:07.637605+0000","last_undegraded":"2024-08-17T18:12:07.637605+0000","last_fullsized":"2024-08-17T18:12:07.637605+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:53.309517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:53.309517+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:50:57.110936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":119,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.705713+0000","last_change":"2024-08-17T18:12:04.653363+0000","last_active":"2024-08-17T18:12:09.705713+0000","last_peered":"2024-08-17T18:12:09.705713+0000","last_clean":"2024-08-17T18:12:09.705713+0000","last_became_active":"2024-08-17T18:12:04.651837+0000","last_became_peered":"2024-08-17T18:12:04.651837+0000","last_unstale":"2024-08-17T18:12:09.705713+0000","last_undegraded":"2024-08-17T18:12:09.705713+0000","last_fullsized":"2024-08-17T18:12:09.705713+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:23:50.020272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":616,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.677004+0000","last_change":"2024-08-17T18:12:04.645849+0000","last_active":"2024-08-17T18:12:07.677004+0000","last_peered":"2024-08-17T18:12:07.677004+0000","last_clean":"2024-08-17T18:12:07.677004+0000","last_became_active":"2024-08-17T18:12:04.645539+0000","last_became_peered":"2024-08-17T18:12:04.645539+0000","last_unstale":"2024-08-17T18:12:07.677004+0000","last_undegraded":"2024-08-17T18:12:07.677004+0000","last_fullsized":"2024-08-17T18:12:07.677004+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:41:11.756265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":2450,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.708418+0000","last_change":"2024-08-17T18:04:43.800633+0000","last_active":"2024-08-17T18:12:09.708418+0000","last_peered":"2024-08-17T18:12:09.708418+0000","last_clean":"2024-08-17T18:12:09.708418+0000","last_became_active":"2024-08-17T18:04:43.784612+0000","last_became_peered":"2024-08-17T18:04:43.784612+0000","last_unstale":"2024-08-17T18:12:09.708418+0000","last_undegraded":"2024-08-17T18:12:09.708418+0000","last_fullsized":"2024-08-17T18:12:09.708418+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:11.405039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"347'149","reported_seq":813,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.676924+0000","last_change":"2024-08-17T18:09:52.474670+0000","last_active":"2024-08-17T18:12:07.676924+0000","last_peered":"2024-08-17T18:12:07.676924+0000","last_clean":"2024-08-17T18:12:07.676924+0000","last_became_active":"2024-08-17T18:09:15.595011+0000","last_became_peered":"2024-08-17T18:09:15.595011+0000","last_unstale":"2024-08-17T18:12:07.676924+0000","last_undegraded":"2024-08-17T18:12:07.676924+0000","last_fullsized":"2024-08-17T18:12:07.676924+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:52.474544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:52.474544+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:38:21.347345+0000","scrub_duration":0.001886326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":229,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.705046+0000","last_change":"2024-08-17T18:12:04.643348+0000","last_active":"2024-08-17T18:12:09.705046+0000","last_peered":"2024-08-17T18:12:09.705046+0000","last_clean":"2024-08-17T18:12:09.705046+0000","last_became_active":"2024-08-17T18:12:04.641662+0000","last_became_peered":"2024-08-17T18:12:04.641662+0000","last_unstale":"2024-08-17T18:12:09.705046+0000","last_undegraded":"2024-08-17T18:12:09.705046+0000","last_fullsized":"2024-08-17T18:12:09.705046+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:59.730975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":222,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.705667+0000","last_change":"2024-08-17T18:12:04.649331+0000","last_active":"2024-08-17T18:12:09.705667+0000","last_peered":"2024-08-17T18:12:09.705667+0000","last_clean":"2024-08-17T18:12:09.705667+0000","last_became_active":"2024-08-17T18:12:04.646946+0000","last_became_peered":"2024-08-17T18:12:04.646946+0000","last_unstale":"2024-08-17T18:12:09.705667+0000","last_undegraded":"2024-08-17T18:12:09.705667+0000","last_fullsized":"2024-08-17T18:12:09.705667+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:58:41.839200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2283,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.696807+0000","last_change":"2024-08-17T18:12:04.641886+0000","last_active":"2024-08-17T18:12:09.696807+0000","last_peered":"2024-08-17T18:12:09.696807+0000","last_clean":"2024-08-17T18:12:09.696807+0000","last_became_active":"2024-08-17T18:12:04.640395+0000","last_became_peered":"2024-08-17T18:12:04.640395+0000","last_unstale":"2024-08-17T18:12:09.696807+0000","last_undegraded":"2024-08-17T18:12:09.696807+0000","last_fullsized":"2024-08-17T18:12:09.696807+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:20:13.078751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"347'149","reported_seq":813,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.677156+0000","last_change":"2024-08-17T18:09:49.461777+0000","last_active":"2024-08-17T18:12:07.677156+0000","last_peered":"2024-08-17T18:12:07.677156+0000","last_clean":"2024-08-17T18:12:07.677156+0000","last_became_active":"2024-08-17T18:09:15.592534+0000","last_became_peered":"2024-08-17T18:09:15.592534+0000","last_unstale":"2024-08-17T18:12:07.677156+0000","last_undegraded":"2024-08-17T18:12:07.677156+0000","last_fullsized":"2024-08-17T18:12:07.677156+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:49.461698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:49.461698+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:28:31.102314+0000","scrub_duration":0.0013792520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":103,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.681797+0000","last_change":"2024-08-17T18:12:04.639394+0000","last_active":"2024-08-17T18:12:07.681797+0000","last_peered":"2024-08-17T18:12:07.681797+0000","last_clean":"2024-08-17T18:12:07.681797+0000","last_became_active":"2024-08-17T18:12:04.638935+0000","last_became_peered":"2024-08-17T18:12:04.638935+0000","last_unstale":"2024-08-17T18:12:07.681797+0000","last_undegraded":"2024-08-17T18:12:07.681797+0000","last_fullsized":"2024-08-17T18:12:07.681797+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:21:16.994042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2285,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.704131+0000","last_change":"2024-08-17T18:12:04.644280+0000","last_active":"2024-08-17T18:12:09.704131+0000","last_peered":"2024-08-17T18:12:09.704131+0000","last_clean":"2024-08-17T18:12:09.704131+0000","last_became_active":"2024-08-17T18:12:04.643749+0000","last_became_peered":"2024-08-17T18:12:04.643749+0000","last_unstale":"2024-08-17T18:12:09.704131+0000","last_undegraded":"2024-08-17T18:12:09.704131+0000","last_fullsized":"2024-08-17T18:12:09.704131+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:56:35.227968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":107,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.635286+0000","last_change":"2024-08-17T18:04:43.773480+0000","last_active":"2024-08-17T18:12:07.635286+0000","last_peered":"2024-08-17T18:12:07.635286+0000","last_clean":"2024-08-17T18:12:07.635286+0000","last_became_active":"2024-08-17T18:04:43.773160+0000","last_became_peered":"2024-08-17T18:04:43.773160+0000","last_unstale":"2024-08-17T18:12:07.635286+0000","last_undegraded":"2024-08-17T18:12:07.635286+0000","last_fullsized":"2024-08-17T18:12:07.635286+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:07:55.468132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"347'149","reported_seq":831,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.680379+0000","last_change":"2024-08-17T18:09:15.596647+0000","last_active":"2024-08-17T18:12:07.680379+0000","last_peered":"2024-08-17T18:12:07.680379+0000","last_clean":"2024-08-17T18:12:07.680379+0000","last_became_active":"2024-08-17T18:09:15.596339+0000","last_became_peered":"2024-08-17T18:09:15.596339+0000","last_unstale":"2024-08-17T18:12:07.680379+0000","last_undegraded":"2024-08-17T18:12:07.680379+0000","last_fullsized":"2024-08-17T18:12:07.680379+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:00.316884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:00.316884+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:30.779927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1410,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.696662+0000","last_change":"2024-08-17T18:12:04.648484+0000","last_active":"2024-08-17T18:12:09.696662+0000","last_peered":"2024-08-17T18:12:09.696662+0000","last_clean":"2024-08-17T18:12:09.696662+0000","last_became_active":"2024-08-17T18:12:04.647502+0000","last_became_peered":"2024-08-17T18:12:04.647502+0000","last_unstale":"2024-08-17T18:12:09.696662+0000","last_undegraded":"2024-08-17T18:12:09.696662+0000","last_fullsized":"2024-08-17T18:12:09.696662+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:08:44.064487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1558,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.677179+0000","last_change":"2024-08-17T18:04:14.526890+0000","last_active":"2024-08-17T18:12:07.677179+0000","last_peered":"2024-08-17T18:12:07.677179+0000","last_clean":"2024-08-17T18:12:07.677179+0000","last_became_active":"2024-08-17T18:04:14.526564+0000","last_became_peered":"2024-08-17T18:04:14.526564+0000","last_unstale":"2024-08-17T18:12:07.677179+0000","last_undegraded":"2024-08-17T18:12:07.677179+0000","last_fullsized":"2024-08-17T18:12:07.677179+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:10:13.229827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":124,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.681867+0000","last_change":"2024-08-17T18:12:04.658437+0000","last_active":"2024-08-17T18:12:07.681867+0000","last_peered":"2024-08-17T18:12:07.681867+0000","last_clean":"2024-08-17T18:12:07.681867+0000","last_became_active":"2024-08-17T18:12:04.647294+0000","last_became_peered":"2024-08-17T18:12:04.647294+0000","last_unstale":"2024-08-17T18:12:07.681867+0000","last_undegraded":"2024-08-17T18:12:07.681867+0000","last_fullsized":"2024-08-17T18:12:07.681867+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:05:39.701204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"347'149","reported_seq":865,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.704765+0000","last_change":"2024-08-17T18:09:15.596502+0000","last_active":"2024-08-17T18:12:09.704765+0000","last_peered":"2024-08-17T18:12:09.704765+0000","last_clean":"2024-08-17T18:12:09.704765+0000","last_became_active":"2024-08-17T18:09:15.595460+0000","last_became_peered":"2024-08-17T18:09:15.595460+0000","last_unstale":"2024-08-17T18:12:09.704765+0000","last_undegraded":"2024-08-17T18:12:09.704765+0000","last_fullsized":"2024-08-17T18:12:09.704765+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:02.363531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:02.363531+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:40.918706+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2281,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.679785+0000","last_change":"2024-08-17T18:12:04.647110+0000","last_active":"2024-08-17T18:12:07.679785+0000","last_peered":"2024-08-17T18:12:07.679785+0000","last_clean":"2024-08-17T18:12:07.679785+0000","last_became_active":"2024-08-17T18:12:04.646668+0000","last_became_peered":"2024-08-17T18:12:04.646668+0000","last_unstale":"2024-08-17T18:12:07.679785+0000","last_undegraded":"2024-08-17T18:12:07.679785+0000","last_fullsized":"2024-08-17T18:12:07.679785+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:03.402811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2111,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.695273+0000","last_change":"2024-08-17T18:12:04.648190+0000","last_active":"2024-08-17T18:12:09.695273+0000","last_peered":"2024-08-17T18:12:09.695273+0000","last_clean":"2024-08-17T18:12:09.695273+0000","last_became_active":"2024-08-17T18:12:04.647877+0000","last_became_peered":"2024-08-17T18:12:04.647877+0000","last_unstale":"2024-08-17T18:12:09.695273+0000","last_undegraded":"2024-08-17T18:12:09.695273+0000","last_fullsized":"2024-08-17T18:12:09.695273+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:18.129337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2399,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.695396+0000","last_change":"2024-08-17T18:04:14.543532+0000","last_active":"2024-08-17T18:12:09.695396+0000","last_peered":"2024-08-17T18:12:09.695396+0000","last_clean":"2024-08-17T18:12:09.695396+0000","last_became_active":"2024-08-17T18:04:14.543131+0000","last_became_peered":"2024-08-17T18:04:14.543131+0000","last_unstale":"2024-08-17T18:12:09.695396+0000","last_undegraded":"2024-08-17T18:12:09.695396+0000","last_fullsized":"2024-08-17T18:12:09.695396+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:51:44.327066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"347'149","reported_seq":776,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.699080+0000","last_change":"2024-08-17T18:12:07.698213+0000","last_active":"2024-08-17T18:12:07.699080+0000","last_peered":"2024-08-17T18:12:07.699080+0000","last_clean":"2024-08-17T18:12:07.699080+0000","last_became_active":"2024-08-17T18:12:07.693969+0000","last_became_peered":"2024-08-17T18:12:07.693969+0000","last_unstale":"2024-08-17T18:12:07.699080+0000","last_undegraded":"2024-08-17T18:12:07.699080+0000","last_fullsized":"2024-08-17T18:12:07.699080+0000","mapping_epoch":1556,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1557,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:15.740775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:15.740775+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":243,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.704365+0000","last_change":"2024-08-17T18:12:04.642603+0000","last_active":"2024-08-17T18:12:09.704365+0000","last_peered":"2024-08-17T18:12:09.704365+0000","last_clean":"2024-08-17T18:12:09.704365+0000","last_became_active":"2024-08-17T18:12:04.642025+0000","last_became_peered":"2024-08-17T18:12:04.642025+0000","last_unstale":"2024-08-17T18:12:09.704365+0000","last_undegraded":"2024-08-17T18:12:09.704365+0000","last_fullsized":"2024-08-17T18:12:09.704365+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:06:50.276904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":229,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.703926+0000","last_change":"2024-08-17T18:08:11.440676+0000","last_active":"2024-08-17T18:12:09.703926+0000","last_peered":"2024-08-17T18:12:09.703926+0000","last_clean":"2024-08-17T18:12:09.703926+0000","last_became_active":"2024-08-17T18:08:11.440353+0000","last_became_peered":"2024-08-17T18:08:11.440353+0000","last_unstale":"2024-08-17T18:12:09.703926+0000","last_undegraded":"2024-08-17T18:12:09.703926+0000","last_fullsized":"2024-08-17T18:12:09.703926+0000","mapping_epoch":1508,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1509,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:19:22.125277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":972,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.705200+0000","last_change":"2024-08-17T18:12:04.645715+0000","last_active":"2024-08-17T18:12:09.705200+0000","last_peered":"2024-08-17T18:12:09.705200+0000","last_clean":"2024-08-17T18:12:09.705200+0000","last_became_active":"2024-08-17T18:12:04.643655+0000","last_became_peered":"2024-08-17T18:12:04.643655+0000","last_unstale":"2024-08-17T18:12:09.705200+0000","last_undegraded":"2024-08-17T18:12:09.705200+0000","last_fullsized":"2024-08-17T18:12:09.705200+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:37:40.204982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":54,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.676433+0000","last_change":"2024-08-17T18:08:37.071912+0000","last_active":"2024-08-17T18:12:07.676433+0000","last_peered":"2024-08-17T18:12:07.676433+0000","last_clean":"2024-08-17T18:12:07.676433+0000","last_became_active":"2024-08-17T18:08:37.071648+0000","last_became_peered":"2024-08-17T18:08:37.071648+0000","last_unstale":"2024-08-17T18:12:07.676433+0000","last_undegraded":"2024-08-17T18:12:07.676433+0000","last_fullsized":"2024-08-17T18:12:07.676433+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:58:57.279808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":226,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.697622+0000","last_change":"2024-08-17T18:04:43.771309+0000","last_active":"2024-08-17T18:12:09.697622+0000","last_peered":"2024-08-17T18:12:09.697622+0000","last_clean":"2024-08-17T18:12:09.697622+0000","last_became_active":"2024-08-17T18:04:43.770922+0000","last_became_peered":"2024-08-17T18:04:43.770922+0000","last_unstale":"2024-08-17T18:12:09.697622+0000","last_undegraded":"2024-08-17T18:12:09.697622+0000","last_fullsized":"2024-08-17T18:12:09.697622+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:10.173251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":111,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.679943+0000","last_change":"2024-08-17T18:04:43.786228+0000","last_active":"2024-08-17T18:12:07.679943+0000","last_peered":"2024-08-17T18:12:07.679943+0000","last_clean":"2024-08-17T18:12:07.679943+0000","last_became_active":"2024-08-17T18:04:43.770976+0000","last_became_peered":"2024-08-17T18:04:43.770976+0000","last_unstale":"2024-08-17T18:12:07.679943+0000","last_undegraded":"2024-08-17T18:12:07.679943+0000","last_fullsized":"2024-08-17T18:12:07.679943+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:20:10.131137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":866,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.705335+0000","last_change":"2024-08-17T18:12:07.677305+0000","last_active":"2024-08-17T18:12:09.705335+0000","last_peered":"2024-08-17T18:12:09.705335+0000","last_clean":"2024-08-17T18:12:09.705335+0000","last_became_active":"2024-08-17T18:12:07.677002+0000","last_became_peered":"2024-08-17T18:12:07.677002+0000","last_unstale":"2024-08-17T18:12:09.705335+0000","last_undegraded":"2024-08-17T18:12:09.705335+0000","last_fullsized":"2024-08-17T18:12:09.705335+0000","mapping_epoch":1556,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1557,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:54.303708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:54.303708+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:11:29.280261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"347'149","reported_seq":832,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.681623+0000","last_change":"2024-08-17T18:09:15.592722+0000","last_active":"2024-08-17T18:12:07.681623+0000","last_peered":"2024-08-17T18:12:07.681623+0000","last_clean":"2024-08-17T18:12:07.681623+0000","last_became_active":"2024-08-17T18:09:15.592242+0000","last_became_peered":"2024-08-17T18:09:15.592242+0000","last_unstale":"2024-08-17T18:12:07.681623+0000","last_undegraded":"2024-08-17T18:12:07.681623+0000","last_fullsized":"2024-08-17T18:12:07.681623+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:55.255983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:55.255983+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:02:53.596275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":660,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.708284+0000","last_change":"2024-08-17T18:12:04.648636+0000","last_active":"2024-08-17T18:12:09.708284+0000","last_peered":"2024-08-17T18:12:09.708284+0000","last_clean":"2024-08-17T18:12:09.708284+0000","last_became_active":"2024-08-17T18:12:04.648256+0000","last_became_peered":"2024-08-17T18:12:04.648256+0000","last_unstale":"2024-08-17T18:12:09.708284+0000","last_undegraded":"2024-08-17T18:12:09.708284+0000","last_fullsized":"2024-08-17T18:12:09.708284+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:22:20.544510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":118,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.705417+0000","last_change":"2024-08-17T18:12:04.646177+0000","last_active":"2024-08-17T18:12:09.705417+0000","last_peered":"2024-08-17T18:12:09.705417+0000","last_clean":"2024-08-17T18:12:09.705417+0000","last_became_active":"2024-08-17T18:12:04.645729+0000","last_became_peered":"2024-08-17T18:12:04.645729+0000","last_unstale":"2024-08-17T18:12:09.705417+0000","last_undegraded":"2024-08-17T18:12:09.705417+0000","last_fullsized":"2024-08-17T18:12:09.705417+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:40:24.212640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1564,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.676238+0000","last_change":"2024-08-17T18:12:04.646652+0000","last_active":"2024-08-17T18:12:07.676238+0000","last_peered":"2024-08-17T18:12:07.676238+0000","last_clean":"2024-08-17T18:12:07.676238+0000","last_became_active":"2024-08-17T18:12:04.645626+0000","last_became_peered":"2024-08-17T18:12:04.645626+0000","last_unstale":"2024-08-17T18:12:07.676238+0000","last_undegraded":"2024-08-17T18:12:07.676238+0000","last_fullsized":"2024-08-17T18:12:07.676238+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:14:12.941368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1572,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.637454+0000","last_change":"2024-08-17T18:12:04.639709+0000","last_active":"2024-08-17T18:12:07.637454+0000","last_peered":"2024-08-17T18:12:07.637454+0000","last_clean":"2024-08-17T18:12:07.637454+0000","last_became_active":"2024-08-17T18:12:04.639383+0000","last_became_peered":"2024-08-17T18:12:04.639383+0000","last_unstale":"2024-08-17T18:12:07.637454+0000","last_undegraded":"2024-08-17T18:12:07.637454+0000","last_fullsized":"2024-08-17T18:12:07.637454+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:16:49.177518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"347'149","reported_seq":832,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.635127+0000","last_change":"2024-08-17T18:09:15.590888+0000","last_active":"2024-08-17T18:12:07.635127+0000","last_peered":"2024-08-17T18:12:07.635127+0000","last_clean":"2024-08-17T18:12:07.635127+0000","last_became_active":"2024-08-17T18:09:15.590345+0000","last_became_peered":"2024-08-17T18:09:15.590345+0000","last_unstale":"2024-08-17T18:12:07.635127+0000","last_undegraded":"2024-08-17T18:12:07.635127+0000","last_fullsized":"2024-08-17T18:12:07.635127+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:59.307356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:59.307356+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:40:40.796012+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1576,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.637789+0000","last_change":"2024-08-17T18:12:04.648228+0000","last_active":"2024-08-17T18:12:07.637789+0000","last_peered":"2024-08-17T18:12:07.637789+0000","last_clean":"2024-08-17T18:12:07.637789+0000","last_became_active":"2024-08-17T18:12:04.647906+0000","last_became_peered":"2024-08-17T18:12:04.647906+0000","last_unstale":"2024-08-17T18:12:07.637789+0000","last_undegraded":"2024-08-17T18:12:07.637789+0000","last_fullsized":"2024-08-17T18:12:07.637789+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:51:30.081540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1545,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.638865+0000","last_change":"2024-08-17T18:12:04.645661+0000","last_active":"2024-08-17T18:12:07.638865+0000","last_peered":"2024-08-17T18:12:07.638865+0000","last_clean":"2024-08-17T18:12:07.638865+0000","last_became_active":"2024-08-17T18:12:04.645319+0000","last_became_peered":"2024-08-17T18:12:04.645319+0000","last_unstale":"2024-08-17T18:12:07.638865+0000","last_undegraded":"2024-08-17T18:12:07.638865+0000","last_fullsized":"2024-08-17T18:12:07.638865+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:15:51.806712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"347'149","reported_seq":859,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.706605+0000","last_change":"2024-08-17T18:09:15.592665+0000","last_active":"2024-08-17T18:12:09.706605+0000","last_peered":"2024-08-17T18:12:09.706605+0000","last_clean":"2024-08-17T18:12:09.706605+0000","last_became_active":"2024-08-17T18:09:15.590435+0000","last_became_peered":"2024-08-17T18:09:15.590435+0000","last_unstale":"2024-08-17T18:12:09.706605+0000","last_undegraded":"2024-08-17T18:12:09.706605+0000","last_fullsized":"2024-08-17T18:12:09.706605+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:52.271487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:52.271487+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:32:45.789611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1537,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.676613+0000","last_change":"2024-08-17T17:24:17.470809+0000","last_active":"2024-08-17T18:12:07.676613+0000","last_peered":"2024-08-17T18:12:07.676613+0000","last_clean":"2024-08-17T18:12:07.676613+0000","last_became_active":"2024-08-17T17:24:17.469753+0000","last_became_peered":"2024-08-17T17:24:17.469753+0000","last_unstale":"2024-08-17T18:12:07.676613+0000","last_undegraded":"2024-08-17T18:12:07.676613+0000","last_fullsized":"2024-08-17T18:12:07.676613+0000","mapping_epoch":686,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":687,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:00.049425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":2416,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.707384+0000","last_change":"2024-08-17T18:04:14.552806+0000","last_active":"2024-08-17T18:12:09.707384+0000","last_peered":"2024-08-17T18:12:09.707384+0000","last_clean":"2024-08-17T18:12:09.707384+0000","last_became_active":"2024-08-17T18:04:14.535546+0000","last_became_peered":"2024-08-17T18:04:14.535546+0000","last_unstale":"2024-08-17T18:12:09.707384+0000","last_undegraded":"2024-08-17T18:12:09.707384+0000","last_fullsized":"2024-08-17T18:12:09.707384+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:44:45.702272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":222,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.704964+0000","last_change":"2024-08-17T18:12:04.652288+0000","last_active":"2024-08-17T18:12:09.704964+0000","last_peered":"2024-08-17T18:12:09.704964+0000","last_clean":"2024-08-17T18:12:09.704964+0000","last_became_active":"2024-08-17T18:12:04.651653+0000","last_became_peered":"2024-08-17T18:12:04.651653+0000","last_unstale":"2024-08-17T18:12:09.704964+0000","last_undegraded":"2024-08-17T18:12:09.704964+0000","last_fullsized":"2024-08-17T18:12:09.704964+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:44:43.443728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":107,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.684640+0000","last_change":"2024-08-17T18:04:43.771107+0000","last_active":"2024-08-17T18:12:07.684640+0000","last_peered":"2024-08-17T18:12:07.684640+0000","last_clean":"2024-08-17T18:12:07.684640+0000","last_became_active":"2024-08-17T18:04:43.770799+0000","last_became_peered":"2024-08-17T18:04:43.770799+0000","last_unstale":"2024-08-17T18:12:07.684640+0000","last_undegraded":"2024-08-17T18:12:07.684640+0000","last_fullsized":"2024-08-17T18:12:07.684640+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:09:04.350798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":63,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.638755+0000","last_change":"2024-08-17T18:12:04.649216+0000","last_active":"2024-08-17T18:12:07.638755+0000","last_peered":"2024-08-17T18:12:07.638755+0000","last_clean":"2024-08-17T18:12:07.638755+0000","last_became_active":"2024-08-17T18:12:04.648753+0000","last_became_peered":"2024-08-17T18:12:04.648753+0000","last_unstale":"2024-08-17T18:12:07.638755+0000","last_undegraded":"2024-08-17T18:12:07.638755+0000","last_fullsized":"2024-08-17T18:12:07.638755+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:42:43.981181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":111,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.635441+0000","last_change":"2024-08-17T18:04:43.781111+0000","last_active":"2024-08-17T18:12:07.635441+0000","last_peered":"2024-08-17T18:12:07.635441+0000","last_clean":"2024-08-17T18:12:07.635441+0000","last_became_active":"2024-08-17T18:04:43.771839+0000","last_became_peered":"2024-08-17T18:04:43.771839+0000","last_unstale":"2024-08-17T18:12:07.635441+0000","last_undegraded":"2024-08-17T18:12:07.635441+0000","last_fullsized":"2024-08-17T18:12:07.635441+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:52:45.973049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1580,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.637174+0000","last_change":"2024-08-17T18:12:04.644174+0000","last_active":"2024-08-17T18:12:07.637174+0000","last_peered":"2024-08-17T18:12:07.637174+0000","last_clean":"2024-08-17T18:12:07.637174+0000","last_became_active":"2024-08-17T18:12:04.643808+0000","last_became_peered":"2024-08-17T18:12:04.643808+0000","last_unstale":"2024-08-17T18:12:07.637174+0000","last_undegraded":"2024-08-17T18:12:07.637174+0000","last_fullsized":"2024-08-17T18:12:07.637174+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:48:59.015535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":245,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.696315+0000","last_change":"2024-08-17T18:12:04.655292+0000","last_active":"2024-08-17T18:12:09.696315+0000","last_peered":"2024-08-17T18:12:09.696315+0000","last_clean":"2024-08-17T18:12:09.696315+0000","last_became_active":"2024-08-17T18:12:04.641282+0000","last_became_peered":"2024-08-17T18:12:04.641282+0000","last_unstale":"2024-08-17T18:12:09.696315+0000","last_undegraded":"2024-08-17T18:12:09.696315+0000","last_fullsized":"2024-08-17T18:12:09.696315+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:36:21.436823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":125,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.635260+0000","last_change":"2024-08-17T18:08:37.068503+0000","last_active":"2024-08-17T18:12:07.635260+0000","last_peered":"2024-08-17T18:12:07.635260+0000","last_clean":"2024-08-17T18:12:07.635260+0000","last_became_active":"2024-08-17T18:08:37.068110+0000","last_became_peered":"2024-08-17T18:08:37.068110+0000","last_unstale":"2024-08-17T18:12:07.635260+0000","last_undegraded":"2024-08-17T18:12:07.635260+0000","last_fullsized":"2024-08-17T18:12:07.635260+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:12:30.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":165,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.676275+0000","last_change":"2024-08-17T18:12:04.644320+0000","last_active":"2024-08-17T18:12:07.676275+0000","last_peered":"2024-08-17T18:12:07.676275+0000","last_clean":"2024-08-17T18:12:07.676275+0000","last_became_active":"2024-08-17T18:12:04.638742+0000","last_became_peered":"2024-08-17T18:12:04.638742+0000","last_unstale":"2024-08-17T18:12:07.676275+0000","last_undegraded":"2024-08-17T18:12:07.676275+0000","last_fullsized":"2024-08-17T18:12:07.676275+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:37:54.461942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1618,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.638494+0000","last_change":"2024-08-17T18:04:43.813536+0000","last_active":"2024-08-17T18:12:07.638494+0000","last_peered":"2024-08-17T18:12:07.638494+0000","last_clean":"2024-08-17T18:12:07.638494+0000","last_became_active":"2024-08-17T18:04:43.799823+0000","last_became_peered":"2024-08-17T18:04:43.799823+0000","last_unstale":"2024-08-17T18:12:07.638494+0000","last_undegraded":"2024-08-17T18:12:07.638494+0000","last_fullsized":"2024-08-17T18:12:07.638494+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:04:10.837113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":125,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.708156+0000","last_change":"2024-08-17T18:12:04.651071+0000","last_active":"2024-08-17T18:12:09.708156+0000","last_peered":"2024-08-17T18:12:09.708156+0000","last_clean":"2024-08-17T18:12:09.708156+0000","last_became_active":"2024-08-17T18:12:04.650715+0000","last_became_peered":"2024-08-17T18:12:04.650715+0000","last_unstale":"2024-08-17T18:12:09.708156+0000","last_undegraded":"2024-08-17T18:12:09.708156+0000","last_fullsized":"2024-08-17T18:12:09.708156+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:47:52.494862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":276,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.637306+0000","last_change":"2024-08-17T18:04:14.527380+0000","last_active":"2024-08-17T18:12:07.637306+0000","last_peered":"2024-08-17T18:12:07.637306+0000","last_clean":"2024-08-17T18:12:07.637306+0000","last_became_active":"2024-08-17T18:04:14.527076+0000","last_became_peered":"2024-08-17T18:04:14.527076+0000","last_unstale":"2024-08-17T18:12:07.637306+0000","last_undegraded":"2024-08-17T18:12:07.637306+0000","last_fullsized":"2024-08-17T18:12:07.637306+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:36:35.583709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":215,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.704624+0000","last_change":"2024-08-17T18:04:14.541814+0000","last_active":"2024-08-17T18:12:09.704624+0000","last_peered":"2024-08-17T18:12:09.704624+0000","last_clean":"2024-08-17T18:12:09.704624+0000","last_became_active":"2024-08-17T18:04:14.541403+0000","last_became_peered":"2024-08-17T18:04:14.541403+0000","last_unstale":"2024-08-17T18:12:09.704624+0000","last_undegraded":"2024-08-17T18:12:09.704624+0000","last_fullsized":"2024-08-17T18:12:09.704624+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:07:45.294924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":236,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.704914+0000","last_change":"2024-08-17T18:12:04.647076+0000","last_active":"2024-08-17T18:12:09.704914+0000","last_peered":"2024-08-17T18:12:09.704914+0000","last_clean":"2024-08-17T18:12:09.704914+0000","last_became_active":"2024-08-17T18:12:04.646745+0000","last_became_peered":"2024-08-17T18:12:04.646745+0000","last_unstale":"2024-08-17T18:12:09.704914+0000","last_undegraded":"2024-08-17T18:12:09.704914+0000","last_fullsized":"2024-08-17T18:12:09.704914+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:56:53.262951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":107,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.635356+0000","last_change":"2024-08-17T18:04:43.771720+0000","last_active":"2024-08-17T18:12:07.635356+0000","last_peered":"2024-08-17T18:12:07.635356+0000","last_clean":"2024-08-17T18:12:07.635356+0000","last_became_active":"2024-08-17T18:04:43.771137+0000","last_became_peered":"2024-08-17T18:04:43.771137+0000","last_unstale":"2024-08-17T18:12:07.635356+0000","last_undegraded":"2024-08-17T18:12:07.635356+0000","last_fullsized":"2024-08-17T18:12:07.635356+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:57.647909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":141,"reported_epoch":1557,"state":"active+clean","last_fresh":"2024-08-17T18:12:07.677126+0000","last_change":"2024-08-17T18:03:51.283872+0000","last_active":"2024-08-17T18:12:07.677126+0000","last_peered":"2024-08-17T18:12:07.677126+0000","last_clean":"2024-08-17T18:12:07.677126+0000","last_became_active":"2024-08-17T18:03:51.267197+0000","last_became_peered":"2024-08-17T18:03:51.267197+0000","last_unstale":"2024-08-17T18:12:07.677126+0000","last_undegraded":"2024-08-17T18:12:07.677126+0000","last_fullsized":"2024-08-17T18:12:07.677126+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:18:59.157786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2412,"reported_epoch":1558,"state":"active+clean","last_fresh":"2024-08-17T18:12:09.707755+0000","last_change":"2024-08-17T18:12:04.643883+0000","last_active":"2024-08-17T18:12:09.707755+0000","last_peered":"2024-08-17T18:12:09.707755+0000","last_clean":"2024-08-17T18:12:09.707755+0000","last_became_active":"2024-08-17T18:12:04.643561+0000","last_became_peered":"2024-08-17T18:12:04.643561+0000","last_unstale":"2024-08-17T18:12:09.707755+0000","last_undegraded":"2024-08-17T18:12:09.707755+0000","last_fullsized":"2024-08-17T18:12:09.707755+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:14:16.873872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":56,"num_pg":16,"stat_sum":{"num_bytes":4247388184,"num_objects":64811,"num_object_clones":0,"num_object_copies":194433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":21008,"num_objects_unfound":0,"num_objects_dirty":64811,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188593,"num_write_kb":8108865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3208,"num_bytes_recovered":210239488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12855226368,"data_stored":12855214152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98221,"ondisk_log_size":98221,"up":48,"acting":48,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644038,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":858717,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":8016,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234755,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496393,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2945648,"kb_used_data":2052328,"kb_used_omap":1,"kb_used_meta":893246,"kb_avail":90799504,"statfs":{"total":95995035648,"available":92978692096,"internally_reserved":0,"allocated":2101583872,"data_stored":2095951736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":914684359},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565634,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2498788,"kb_used_data":1578140,"kb_used_omap":72,"kb_used_meta":920567,"kb_avail":91246364,"statfs":{"total":95995035648,"available":93436276736,"internally_reserved":0,"allocated":1616015360,"data_stored":1610391349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":942661102},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":1399,"seq":6008659247236,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2209388,"kb_used_data":1812132,"kb_used_omap":131,"kb_used_meta":397116,"kb_avail":91535764,"statfs":{"total":95995035648,"available":93732622336,"internally_reserved":0,"allocated":1855623168,"data_stored":1849996153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134291,"internal_metadata":406647661},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":2864743187024,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448663,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2160744,"kb_used_data":1627680,"kb_used_omap":75,"kb_used_meta":532980,"kb_avail":91584408,"statfs":{"total":95995035648,"available":93782433792,"internally_reserved":0,"allocated":1666744320,"data_stored":1661116222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":545771711},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208758027,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3162664,"kb_used_data":2211104,"kb_used_omap":1,"kb_used_meta":951550,"kb_avail":90582488,"statfs":{"total":95995035648,"available":92756467712,"internally_reserved":0,"allocated":2264170496,"data_stored":2258542886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":974387654},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019666,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2674092,"kb_used_data":1872740,"kb_used_omap":1,"kb_used_meta":801342,"kb_avail":91071060,"statfs":{"total":95995035648,"available":93256765440,"internally_reserved":0,"allocated":1917685760,"data_stored":1912057719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":820574662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1638928384,"data_stored":1638924312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1599995904,"data_stored":1599995904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1827602432,"data_stored":1827602432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1579028480,"data_stored":1579024408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1898053632,"data_stored":1898053632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2225864704,"data_stored":2225864704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2085752832,"data_stored":2085748760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T18:12:21.883 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-17T18:12:21.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool set unique_pool_3 pg_num 26 2024-08-17T18:12:21.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:21.898 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:21.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:22.000 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:21 smithi165 ceph-mon[105517]: pgmap v2904: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 3.8 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 902 op/s; 20055/189162 objects misplaced (10.602%); 19 MiB/s, 303 objects/s recovering 2024-08-17T18:12:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:12:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:12:22.047 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:22.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:22.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:22.149 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:22.193 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:22.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:22.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:22.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:21 smithi136 ceph-mon[115078]: pgmap v2904: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 3.8 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 902 op/s; 20055/189162 objects misplaced (10.602%); 19 MiB/s, 303 objects/s recovering 2024-08-17T18:12:22.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:12:22.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:12:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:21 smithi136 ceph-mon[109861]: pgmap v2904: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 3.8 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 902 op/s; 20055/189162 objects misplaced (10.602%); 19 MiB/s, 303 objects/s recovering 2024-08-17T18:12:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:12:22.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:12:22.295 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:22.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:22.331 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:22.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:22.432 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:22.464 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:22.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:22.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:22.566 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:22.598 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:22.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:22.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:22.700 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:22.735 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:22.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:22.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:22.785 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:12:22.837 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:22.871 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:22.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:22.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:22.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:23.004 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:23.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:23.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:23.106 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:23.147 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:23.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:23.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:23.249 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:23.291 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:23.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:23.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:23.393 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:23.430 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:23.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:23.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:23.532 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:23.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:23.566 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:23.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:23.668 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:23.689 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:23.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:23.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:23.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:23.830 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:23.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:23.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:23.858 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:23 smithi165 ceph-mon[105517]: pgmap v2905: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 3.7 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 947 op/s; 19142/183933 objects misplaced (10.407%); 19 MiB/s, 306 objects/s recovering 2024-08-17T18:12:23.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:23.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:23.981 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:23.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:24.082 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:24.118 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:24.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:24.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:24.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:23 smithi136 ceph-mon[109861]: pgmap v2905: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 3.7 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 947 op/s; 19142/183933 objects misplaced (10.407%); 19 MiB/s, 306 objects/s recovering 2024-08-17T18:12:24.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:23 smithi136 ceph-mon[115078]: pgmap v2905: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 3.7 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 947 op/s; 19142/183933 objects misplaced (10.407%); 19 MiB/s, 306 objects/s recovering 2024-08-17T18:12:24.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:24.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:24.270 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:24.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:24.371 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:24.428 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:24.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:24.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:24.530 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:24.561 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:12:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:12:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:196502 56.3 56:ce8cfee2:::benchmark_data_smithi136_356456_object69798:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1558)", 2024-08-17T18:12:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:24.548011+0000", 2024-08-17T18:12:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.010841959999999999, 2024-08-17T18:12:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010874185999999999, 2024-08-17T18:12:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-08-17T18:12:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:12:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:12:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 196502 2024-08-17T18:12:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.548011+0000", 2024-08-17T18:12:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.548011+0000", 2024-08-17T18:12:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.548009+0000", 2024-08-17T18:12:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:12:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.548015+0000", 2024-08-17T18:12:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6149999999999996e-06 2024-08-17T18:12:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.548026+0000", 2024-08-17T18:12:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1593000000000001e-05 2024-08-17T18:12:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.548033+0000", 2024-08-17T18:12:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5509999999999996e-06 2024-08-17T18:12:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.548333+0000", 2024-08-17T18:12:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000300198 2024-08-17T18:12:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.548466+0000", 2024-08-17T18:12:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013304899999999999 2024-08-17T18:12:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4,5", 2024-08-17T18:12:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.548601+0000", 2024-08-17T18:12:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000134413 2024-08-17T18:12:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.556098+0000", 2024-08-17T18:12:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.007497302 2024-08-17T18:12:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.556612+0000", 2024-08-17T18:12:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00051424299999999995 2024-08-17T18:12:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:24.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.556757+0000", 2024-08-17T18:12:24.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000144828 2024-08-17T18:12:24.567 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:24.567 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:24.567 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:24.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.567 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:196505 56.3 56:cbc8444d:::benchmark_data_smithi136_356456_object69801:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1558)", 2024-08-17T18:12:24.567 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:24.548985+0000", 2024-08-17T18:12:24.567 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0098681250000000002, 2024-08-17T18:12:24.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0099909249999999995, 2024-08-17T18:12:24.567 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-08-17T18:12:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:12:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:12:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:12:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 196505 2024-08-17T18:12:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.548985+0000", 2024-08-17T18:12:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.548985+0000", 2024-08-17T18:12:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.548984+0000", 2024-08-17T18:12:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:12:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.548991+0000", 2024-08-17T18:12:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2470000000000002e-06 2024-08-17T18:12:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.549002+0000", 2024-08-17T18:12:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1457000000000001e-05 2024-08-17T18:12:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.549009+0000", 2024-08-17T18:12:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4030000000000004e-06 2024-08-17T18:12:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.552051+0000", 2024-08-17T18:12:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030424269999999999 2024-08-17T18:12:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.552413+0000", 2024-08-17T18:12:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036217900000000002 2024-08-17T18:12:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4,5", 2024-08-17T18:12:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.552556+0000", 2024-08-17T18:12:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000142449 2024-08-17T18:12:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:12:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.556619+0000", 2024-08-17T18:12:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040632710000000002 2024-08-17T18:12:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:12:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.558847+0000", 2024-08-17T18:12:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022277709999999999 2024-08-17T18:12:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46025.0:196500 56.f e1558/1554 56:fa81855b:::benchmark_data_smithi136_356456_object69796:head v 1558'12397, mlcod=1558'12394)", 2024-08-17T18:12:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:24.556696+0000", 2024-08-17T18:12:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0021562859999999999, 2024-08-17T18:12:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023325440000000002, 2024-08-17T18:12:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:12:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.556696+0000", 2024-08-17T18:12:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.556696+0000", 2024-08-17T18:12:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.556696+0000", 2024-08-17T18:12:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:12:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.556700+0000", 2024-08-17T18:12:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9620000000000004e-06 2024-08-17T18:12:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.556706+0000", 2024-08-17T18:12:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6790000000000002e-06 2024-08-17T18:12:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.556710+0000", 2024-08-17T18:12:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9689999999999996e-06 2024-08-17T18:12:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.556783+0000", 2024-08-17T18:12:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2989000000000002e-05 2024-08-17T18:12:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.556805+0000", 2024-08-17T18:12:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2687000000000001e-05 2024-08-17T18:12:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46025.0:196507 56.e e1558/1554 56:743a7c4b:::benchmark_data_smithi136_356456_object69803:head v 1558'12228, mlcod=1558'12225)", 2024-08-17T18:12:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:24.557294+0000", 2024-08-17T18:12:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.001559054, 2024-08-17T18:12:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017963219999999999, 2024-08-17T18:12:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:12:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.557294+0000", 2024-08-17T18:12:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.557294+0000", 2024-08-17T18:12:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.557293+0000", 2024-08-17T18:12:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:12:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.557302+0000", 2024-08-17T18:12:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2450000000000004e-06 2024-08-17T18:12:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.557313+0000", 2024-08-17T18:12:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1285e-05 2024-08-17T18:12:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.557320+0000", 2024-08-17T18:12:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3629999999999999e-06 2024-08-17T18:12:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:12:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.557362+0000", 2024-08-17T18:12:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2641000000000001e-05 2024-08-17T18:12:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:12:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.557386+0000", 2024-08-17T18:12:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3666e-05 2024-08-17T18:12:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46025.0:196503 56.f e1558/1554)", 2024-08-17T18:12:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:24.558048+0000", 2024-08-17T18:12:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00080437100000000001, 2024-08-17T18:12:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001094504, 2024-08-17T18:12:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-08-17T18:12:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.558048+0000", 2024-08-17T18:12:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.558048+0000", 2024-08-17T18:12:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.558048+0000", 2024-08-17T18:12:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:12:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.558054+0000", 2024-08-17T18:12:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1079999999999998e-06 2024-08-17T18:12:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.558073+0000", 2024-08-17T18:12:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9443e-05 2024-08-17T18:12:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.558080+0000", 2024-08-17T18:12:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8589999999999999e-06 2024-08-17T18:12:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.46025.0:196505 56.3 e1558/1554)", 2024-08-17T18:12:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:24.559044+0000", 2024-08-17T18:12:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4294967295.9998083, 2024-08-17T18:12:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000144246, 2024-08-17T18:12:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:12:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-08-17T18:12:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:12:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:12:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.559044+0000", 2024-08-17T18:12:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:12:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.559044+0000", 2024-08-17T18:12:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:12:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:12:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.559044+0000", 2024-08-17T18:12:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:12:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:12:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.559049+0000", 2024-08-17T18:12:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8169999999999998e-06 2024-08-17T18:12:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:12:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.559060+0000", 2024-08-17T18:12:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1761e-05 2024-08-17T18:12:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:12:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:12:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:12:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:24.559074+0000", 2024-08-17T18:12:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3577000000000001e-05 2024-08-17T18:12:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:12:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:12:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-08-17T18:12:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 6 2024-08-17T18:12:24.585 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:12:24.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:24.586 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:24.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:24.591 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:12:24.688 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:24.728 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:24.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:24.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:24.790 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:12:24.829 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:24.869 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:24.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:24.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:24.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:24 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2622778693' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:12:24.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:24 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2622778693' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:12:24.969 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:24 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2622778693' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:12:24.970 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:25.003 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:25.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:25.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:25.106 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:25.143 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:25.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:25.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:25.244 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:25.281 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:25.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:25.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:25.383 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:25.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:25.434 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:25.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:25.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-17T18:12:25.535 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:25.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:25.591 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:25.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:25.692 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:25.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:25.754 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:25.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:25.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:25.866 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:12:25.887 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:25.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:25.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:25.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:25 smithi136 ceph-mon[109861]: pgmap v2906: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 3.6 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 1.01k op/s; 17045/175365 objects misplaced (9.720%); 23 MiB/s, 369 objects/s recovering 2024-08-17T18:12:25.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:25 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:12:25.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:25 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2622778693' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:12:25.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:25 smithi136 ceph-mon[109861]: osdmap e1559: 8 total, 8 up, 6 in 2024-08-17T18:12:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:25 smithi136 ceph-mon[115078]: pgmap v2906: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 3.6 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 1.01k op/s; 17045/175365 objects misplaced (9.720%); 23 MiB/s, 369 objects/s recovering 2024-08-17T18:12:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:25 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:12:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:25 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2622778693' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:12:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:25 smithi136 ceph-mon[115078]: osdmap e1559: 8 total, 8 up, 6 in 2024-08-17T18:12:25.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:26.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:26.030 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:26.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:26.061 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:25 smithi165 ceph-mon[105517]: pgmap v2906: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 3.6 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 1.01k op/s; 17045/175365 objects misplaced (9.720%); 23 MiB/s, 369 objects/s recovering 2024-08-17T18:12:26.061 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:25 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:12:26.061 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:25 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2622778693' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:12:26.061 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:25 smithi165 ceph-mon[105517]: osdmap e1559: 8 total, 8 up, 6 in 2024-08-17T18:12:26.131 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:26.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:26.167 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:26.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:26.268 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:26.314 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:26.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:26.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:26.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:26.453 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:26.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:26.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:26.555 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:26.588 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:26.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:26.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:26.690 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:26.733 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:26.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:26.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:26.835 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:26.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:26.874 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:26.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:26.975 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:27.014 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:27.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:27.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:27.116 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:27.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:27.163 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:27.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:27.264 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:27.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:27.319 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:27.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:27.420 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:27.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:27.458 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:27.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:27.560 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:27.602 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:27.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:27.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:27.703 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:12:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:12:27] "GET /metrics HTTP/1.1" 200 36715 "" "Prometheus/2.51.0" 2024-08-17T18:12:27.704 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:27.745 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:27.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:27.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:27.847 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:27.871 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:12:27.871 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:12:27.871 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-08-17T18:12:27.871 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-08-17T18:12:27.871 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:12:27.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:27.882 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:27.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:27.983 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:28.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:27 smithi165 ceph-mon[105517]: pgmap v2908: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 3.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 945 op/s; 17045/172623 objects misplaced (9.874%); 21 MiB/s, 334 objects/s recovering 2024-08-17T18:12:28.025 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:28.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:28.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:28.037 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:27 smithi136 ceph-mon[109861]: pgmap v2908: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 3.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 945 op/s; 17045/172623 objects misplaced (9.874%); 21 MiB/s, 334 objects/s recovering 2024-08-17T18:12:28.037 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:27 smithi136 ceph-mon[115078]: pgmap v2908: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 3.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 945 op/s; 17045/172623 objects misplaced (9.874%); 21 MiB/s, 334 objects/s recovering 2024-08-17T18:12:28.126 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:28.165 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:28.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:28.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:28.178 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:12:28.266 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:28.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:28.314 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:28.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:28.415 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:28.458 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:28.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:28.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:28.560 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:28.618 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:28.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:28.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:28.719 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:12:28.719 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:12:28.720 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:28.733 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:12:28.733 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:12:28.753 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:28.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:28.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:28.785 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:12:28.785 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:12:28.818 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:12:28.819 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:12:28.838 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:28 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3531183080' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:12:28.838 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:28 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3531183080' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:12:28.855 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:28.861 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:12:28.862 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:12:28.884 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:28.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:28.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:28.914 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:12:28.915 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:12:28.915 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:12:28.958 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:12:28.958 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:12:28.985 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:28.996 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:12:28.996 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:12:29.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:29.022 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:29.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:29.123 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:29.162 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:29.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:29.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:29.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:28 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3531183080' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:12:29.264 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:29.303 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:29.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:29.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:29.405 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:29.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:29.426 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:29.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:29.527 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:29.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:29.562 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:29.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:29.663 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:29.699 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:29.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:29.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:29.801 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:29.818 INFO:teuthology.orchestra.run.smithi136.stderr:set pool 56 pg_num to 26 2024-08-17T18:12:29.833 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:29.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:29.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:29.935 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:29.969 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-17T18:12:29.970 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:29.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:29.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:30.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:30.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:30.123 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:30.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:29 smithi136 ceph-mon[109861]: pgmap v2909: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 3.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 940 op/s; 14814/166299 objects misplaced (8.908%); 21 MiB/s, 336 objects/s recovering 2024-08-17T18:12:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:29 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/477131982' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T18:12:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:29 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T18:12:30.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:29 smithi136 ceph-mon[115078]: pgmap v2909: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 3.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 940 op/s; 14814/166299 objects misplaced (8.908%); 21 MiB/s, 336 objects/s recovering 2024-08-17T18:12:30.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:29 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/477131982' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T18:12:30.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:29 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T18:12:30.223 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:29 smithi165 ceph-mon[105517]: pgmap v2909: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 3.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 940 op/s; 14814/166299 objects misplaced (8.908%); 21 MiB/s, 336 objects/s recovering 2024-08-17T18:12:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:29 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/477131982' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T18:12:30.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:29 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T18:12:30.273 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:30.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:30.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:30.375 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:30.385 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:12:30.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:30.412 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:30.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:30.513 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:30.551 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:30.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:30.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:30.653 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:30.687 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:30.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:30.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:30.788 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:30.830 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:30.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:30.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:30.932 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:31.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:31.008 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:31.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:31.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:31.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:31.141 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:31.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:31.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:30 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-08-17T18:12:31.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:30 smithi136 ceph-mon[109861]: osdmap e1560: 8 total, 8 up, 6 in 2024-08-17T18:12:31.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"} : dispatch 2024-08-17T18:12:31.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:30 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-08-17T18:12:31.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:30 smithi136 ceph-mon[115078]: osdmap e1560: 8 total, 8 up, 6 in 2024-08-17T18:12:31.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"} : dispatch 2024-08-17T18:12:31.242 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:31.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:30 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-08-17T18:12:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:30 smithi165 ceph-mon[105517]: osdmap e1560: 8 total, 8 up, 6 in 2024-08-17T18:12:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:30 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"} : dispatch 2024-08-17T18:12:31.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:31.270 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:31.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:31.371 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:31.396 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:31.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:31.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:31.498 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:31.519 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:31.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:31.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:31.621 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:31.642 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:31.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:31.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:31.744 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:31.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:31.768 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:31.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:31.869 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:31.900 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:31.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:31.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:32.002 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:32.023 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:32.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:32.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:32.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:32.150 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:32.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:32.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:32.162 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:31 smithi165 ceph-mon[105517]: pgmap v2911: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 3.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 951 op/s; 14029/161088 objects misplaced (8.709%); 22 MiB/s, 355 objects/s recovering 2024-08-17T18:12:32.162 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:31 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-17T18:12:32.162 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:31 smithi165 ceph-mon[105517]: osdmap e1561: 8 total, 8 up, 6 in 2024-08-17T18:12:32.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:31 smithi136 ceph-mon[109861]: pgmap v2911: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 3.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 951 op/s; 14029/161088 objects misplaced (8.709%); 22 MiB/s, 355 objects/s recovering 2024-08-17T18:12:32.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:31 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-17T18:12:32.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:31 smithi136 ceph-mon[109861]: osdmap e1561: 8 total, 8 up, 6 in 2024-08-17T18:12:32.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:31 smithi136 ceph-mon[115078]: pgmap v2911: 99 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 93 active+clean; 3.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 951 op/s; 14029/161088 objects misplaced (8.709%); 22 MiB/s, 355 objects/s recovering 2024-08-17T18:12:32.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:31 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-17T18:12:32.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:31 smithi136 ceph-mon[115078]: osdmap e1561: 8 total, 8 up, 6 in 2024-08-17T18:12:32.252 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:32.278 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:32.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:32.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:32.380 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:32.403 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:32.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:32.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:32.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:32.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:32.528 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:32.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:32.628 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:32.654 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:32.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:32.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:32.756 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:32.779 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:32.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:32.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:32.881 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:32.912 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:32.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:32.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:33.014 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:33.037 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:33.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:33.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:33.085 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:32 smithi165 ceph-mon[105517]: osdmap e1562: 8 total, 8 up, 6 in 2024-08-17T18:12:33.085 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:32 smithi165 ceph-mon[105517]: 56.3 continuing backfill to osd.4 from (1545'6627,1560'12531] MIN to 1560'12531 2024-08-17T18:12:33.085 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:32 smithi165 ceph-mon[105517]: 56.4 starting backfill to osd.5 from (1544'5600,1560'12634] MIN to 1560'12635 2024-08-17T18:12:33.085 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:12:33.085 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:12:33.139 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:33.166 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:33.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:33.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:33.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:32 smithi136 ceph-mon[109861]: osdmap e1562: 8 total, 8 up, 6 in 2024-08-17T18:12:33.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:32 smithi136 ceph-mon[109861]: 56.3 continuing backfill to osd.4 from (1545'6627,1560'12531] MIN to 1560'12531 2024-08-17T18:12:33.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:32 smithi136 ceph-mon[109861]: 56.4 starting backfill to osd.5 from (1544'5600,1560'12634] MIN to 1560'12635 2024-08-17T18:12:33.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:32 smithi136 ceph-mon[115078]: osdmap e1562: 8 total, 8 up, 6 in 2024-08-17T18:12:33.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:32 smithi136 ceph-mon[115078]: 56.3 continuing backfill to osd.4 from (1545'6627,1560'12531] MIN to 1560'12531 2024-08-17T18:12:33.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:32 smithi136 ceph-mon[115078]: 56.4 starting backfill to osd.5 from (1544'5600,1560'12634] MIN to 1560'12635 2024-08-17T18:12:33.268 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:33.293 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:33.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:33.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:33.398 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:33.710 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:33.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:33.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:33.749 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-17T18:12:33.812 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:33.835 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:33.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:33.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:33.938 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:33.973 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:33.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:33.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:33 smithi165 ceph-mon[105517]: pgmap v2914: 109 pgs: 2 peering, 9 unknown, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 92 active+clean; 3.2 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 891 op/s; 11499/156561 objects misplaced (7.345%); 23 MiB/s, 374 objects/s recovering 2024-08-17T18:12:34.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:33 smithi165 ceph-mon[105517]: osdmap e1563: 8 total, 8 up, 6 in 2024-08-17T18:12:34.022 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:33 smithi136 ceph-mon[109861]: pgmap v2914: 109 pgs: 2 peering, 9 unknown, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 92 active+clean; 3.2 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 891 op/s; 11499/156561 objects misplaced (7.345%); 23 MiB/s, 374 objects/s recovering 2024-08-17T18:12:34.022 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:33 smithi136 ceph-mon[109861]: osdmap e1563: 8 total, 8 up, 6 in 2024-08-17T18:12:34.022 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:33 smithi136 ceph-mon[115078]: pgmap v2914: 109 pgs: 2 peering, 9 unknown, 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 92 active+clean; 3.2 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 891 op/s; 11499/156561 objects misplaced (7.345%); 23 MiB/s, 374 objects/s recovering 2024-08-17T18:12:34.022 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:33 smithi136 ceph-mon[115078]: osdmap e1563: 8 total, 8 up, 6 in 2024-08-17T18:12:34.076 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:34.076 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:12:34.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:34.129 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:34.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:34.230 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:34.274 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:34.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:34.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:34.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:34.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:34.415 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:34.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:34.516 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:34.562 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:34.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:34.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:34.664 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:34.697 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:34.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:34.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:34.799 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:34.823 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:34.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:34.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:34.925 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:34.948 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:34.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:34.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:34.956 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:12:35.013 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:34 smithi136 ceph-mon[109861]: 56.13 continuing backfill to osd.4 from (1545'6627,1560'12531] MIN to 1560'12531 2024-08-17T18:12:35.013 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:34 smithi136 ceph-mon[109861]: osdmap e1564: 8 total, 8 up, 6 in 2024-08-17T18:12:35.013 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:34 smithi136 ceph-mon[109861]: 56.14 starting backfill to osd.5 from (1544'5600,1560'12634] MIN to 1560'12635 2024-08-17T18:12:35.013 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:34 smithi136 ceph-mon[109861]: osdmap e1565: 8 total, 8 up, 6 in 2024-08-17T18:12:35.013 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:34 smithi136 ceph-mon[115078]: 56.13 continuing backfill to osd.4 from (1545'6627,1560'12531] MIN to 1560'12531 2024-08-17T18:12:35.014 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:34 smithi136 ceph-mon[115078]: osdmap e1564: 8 total, 8 up, 6 in 2024-08-17T18:12:35.014 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:34 smithi136 ceph-mon[115078]: 56.14 starting backfill to osd.5 from (1544'5600,1560'12634] MIN to 1560'12635 2024-08-17T18:12:35.014 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:34 smithi136 ceph-mon[115078]: osdmap e1565: 8 total, 8 up, 6 in 2024-08-17T18:12:35.050 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:35.077 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:35.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:35.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:35.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:35.204 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:35.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:35.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:35.224 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:12:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:34 smithi165 ceph-mon[105517]: 56.13 continuing backfill to osd.4 from (1545'6627,1560'12531] MIN to 1560'12531 2024-08-17T18:12:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:34 smithi165 ceph-mon[105517]: osdmap e1564: 8 total, 8 up, 6 in 2024-08-17T18:12:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:34 smithi165 ceph-mon[105517]: 56.14 starting backfill to osd.5 from (1544'5600,1560'12634] MIN to 1560'12635 2024-08-17T18:12:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:34 smithi165 ceph-mon[105517]: osdmap e1565: 8 total, 8 up, 6 in 2024-08-17T18:12:35.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:35.334 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:35.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:35.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:35.435 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:35.459 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:35.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:35.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:35.561 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:35.584 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:35.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:35.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:35.637 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:12:35.686 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:35.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:35.719 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:35.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:35.820 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:35.856 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:35.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:35.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:35.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:35 smithi136 ceph-mon[109861]: pgmap v2917: 109 pgs: 2 peering, 4 unknown, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 98 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 767 op/s; 7565/151869 objects misplaced (4.981%); 38 MiB/s, 614 objects/s recovering 2024-08-17T18:12:35.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:35 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2727348805' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:12:35.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:35 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:12:35.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:35 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2727348805' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:12:35.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:35 smithi136 ceph-mon[109861]: osdmap e1566: 8 total, 8 up, 6 in 2024-08-17T18:12:35.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:35 smithi136 ceph-mon[115078]: pgmap v2917: 109 pgs: 2 peering, 4 unknown, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 98 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 767 op/s; 7565/151869 objects misplaced (4.981%); 38 MiB/s, 614 objects/s recovering 2024-08-17T18:12:35.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:35 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2727348805' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:12:35.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:35 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:12:35.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:35 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2727348805' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:12:35.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:35 smithi136 ceph-mon[115078]: osdmap e1566: 8 total, 8 up, 6 in 2024-08-17T18:12:35.957 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:35.984 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:35.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:35.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:35.986 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:35 smithi165 ceph-mon[105517]: pgmap v2917: 109 pgs: 2 peering, 4 unknown, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 98 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 767 op/s; 7565/151869 objects misplaced (4.981%); 38 MiB/s, 614 objects/s recovering 2024-08-17T18:12:35.986 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:35 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2727348805' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:12:35.986 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:35 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:12:35.986 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:35 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2727348805' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:12:35.986 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:35 smithi165 ceph-mon[105517]: osdmap e1566: 8 total, 8 up, 6 in 2024-08-17T18:12:36.031 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:12:36.031 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:12:36.087 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:36.111 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:36.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:36.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:36.212 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:36.237 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:36.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:36.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:36.339 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:36.362 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:36.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:36.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:36.464 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:36.487 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:36.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:36.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:36.589 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:36.616 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:36.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:36.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:36.719 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:36.754 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:36.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:36.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:36.856 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:36.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:36.888 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:36.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:36.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-17T18:12:36.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:37.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:37.031 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:37.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:37.132 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:37.163 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:37.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:37.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:37.228 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:12:37.228 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:12:37.228 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:37 smithi165 ceph-mon[105517]: pgmap v2920: 109 pgs: 2 unknown, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 102 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 476 op/s; 7565/150840 objects misplaced (5.015%); 29 MiB/s, 463 objects/s recovering 2024-08-17T18:12:37.228 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:37 smithi165 ceph-mon[105517]: osdmap e1567: 8 total, 8 up, 6 in 2024-08-17T18:12:37.265 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:37.299 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:37.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:37.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:37.307 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:12:37.401 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:37.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:37.439 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:37.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:37.540 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:37.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:37.575 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:37.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:37.677 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:12:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:12:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:37 smithi136 ceph-mon[109861]: pgmap v2920: 109 pgs: 2 unknown, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 102 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 476 op/s; 7565/150840 objects misplaced (5.015%); 29 MiB/s, 463 objects/s recovering 2024-08-17T18:12:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:37 smithi136 ceph-mon[109861]: osdmap e1567: 8 total, 8 up, 6 in 2024-08-17T18:12:37.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:12:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:12:37] "GET /metrics HTTP/1.1" 200 36715 "" "Prometheus/2.51.0" 2024-08-17T18:12:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:12:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:12:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:37 smithi136 ceph-mon[115078]: pgmap v2920: 109 pgs: 2 unknown, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 102 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 476 op/s; 7565/150840 objects misplaced (5.015%); 29 MiB/s, 463 objects/s recovering 2024-08-17T18:12:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:37 smithi136 ceph-mon[115078]: osdmap e1567: 8 total, 8 up, 6 in 2024-08-17T18:12:37.706 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:37.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:37.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:37.808 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:37.855 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:37.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:37.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:37.957 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:37.988 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:37.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:37.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:38.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:38.118 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:38.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:38.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:38.220 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:38.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:38.287 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:38.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:38.307 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:12:38.388 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:38.425 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:38.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:38.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:38.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:38 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:12:38.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:38 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:12:38.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:38 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:12:38.527 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:38.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:38.562 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:38.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:38.648 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 6, 0, 5] out_osds: [7, 1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:12:38.648 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:12:38.649 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-08-17T18:12:38.649 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-08-17T18:12:38.663 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:38.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:38.745 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:38.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:38.846 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:38.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:38.903 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:38.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:38.915 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:12:38.915 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:12:38.934 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:12:38.947 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:12:38.948 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:12:38.987 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:12:38.987 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:12:39.005 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:39.022 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:12:39.022 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:12:39.042 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:39.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:39.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:39.053 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:12:39.053 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:12:39.108 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:12:39.108 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:12:39.132 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:12:39.133 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:12:39.144 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:39.167 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:12:39.167 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:12:39.180 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:39.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:39.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:39.282 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:39.323 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:39.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:39.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:39.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:39.431 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:39 smithi165 ceph-mon[105517]: pgmap v2922: 109 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 105 active+clean; 3.0 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 520 op/s; 4693/147255 objects misplaced (3.187%); 86 MiB/s, 1.37k objects/s recovering 2024-08-17T18:12:39.431 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:39 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-08-17T18:12:39.431 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:39 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2927866771' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:12:39.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:39 smithi136 ceph-mon[109861]: pgmap v2922: 109 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 105 active+clean; 3.0 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 520 op/s; 4693/147255 objects misplaced (3.187%); 86 MiB/s, 1.37k objects/s recovering 2024-08-17T18:12:39.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:39 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-08-17T18:12:39.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:39 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2927866771' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:12:39.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:39 smithi136 ceph-mon[115078]: pgmap v2922: 109 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 105 active+clean; 3.0 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 520 op/s; 4693/147255 objects misplaced (3.187%); 86 MiB/s, 1.37k objects/s recovering 2024-08-17T18:12:39.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:39 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-08-17T18:12:39.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:39 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2927866771' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:12:39.465 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:39.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:39.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:39.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:39.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:39.602 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:39.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:39.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:39.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:39.735 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:39.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:39.836 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:39.862 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:39.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:39.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:39.963 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:39.991 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:39.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:39.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:40.092 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:40.115 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:40.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:40.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:40.217 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:40.243 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:40.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:40.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:40.345 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:40.369 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:40.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:40.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:40.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:40 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "17"}]': finished 2024-08-17T18:12:40.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:40 smithi136 ceph-mon[115078]: osdmap e1568: 8 total, 8 up, 6 in 2024-08-17T18:12:40.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:40 smithi136 ceph-mon[115078]: osdmap e1569: 8 total, 8 up, 6 in 2024-08-17T18:12:40.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:40 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "17"}]': finished 2024-08-17T18:12:40.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:40 smithi136 ceph-mon[109861]: osdmap e1568: 8 total, 8 up, 6 in 2024-08-17T18:12:40.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:40 smithi136 ceph-mon[109861]: osdmap e1569: 8 total, 8 up, 6 in 2024-08-17T18:12:40.471 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:40.497 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:40.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:40.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:40.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:40 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "17"}]': finished 2024-08-17T18:12:40.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:40 smithi165 ceph-mon[105517]: osdmap e1568: 8 total, 8 up, 6 in 2024-08-17T18:12:40.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:40 smithi165 ceph-mon[105517]: osdmap e1569: 8 total, 8 up, 6 in 2024-08-17T18:12:40.600 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:40.623 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:40.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:40.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:40.724 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:40.747 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:40.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:40.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:40.848 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:40.871 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:40.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:40.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:40.973 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:40.998 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:40.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:41.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:41.100 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:41.137 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:41.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:41.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:41.239 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:41.261 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:41.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:41.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:41.363 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:41.391 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:41.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:41.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:41.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:41 smithi136 ceph-mon[109861]: pgmap v2925: 109 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 105 active+clean; 2.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 650 op/s; 4693/142632 objects misplaced (3.290%); 75 MiB/s, 1.20k objects/s recovering 2024-08-17T18:12:41.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:41 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-08-17T18:12:41.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:41 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "18"}]': finished 2024-08-17T18:12:41.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:41 smithi136 ceph-mon[109861]: 56.10 starting backfill to osd.0 from (0'0,0'0] MAX to 1567'12732 2024-08-17T18:12:41.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:41 smithi136 ceph-mon[109861]: osdmap e1570: 8 total, 8 up, 6 in 2024-08-17T18:12:41.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:41 smithi136 ceph-mon[109861]: 56.10 starting backfill to osd.4 from (0'0,0'0] MAX to 1567'12732 2024-08-17T18:12:41.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:41 smithi136 ceph-mon[115078]: pgmap v2925: 109 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 105 active+clean; 2.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 650 op/s; 4693/142632 objects misplaced (3.290%); 75 MiB/s, 1.20k objects/s recovering 2024-08-17T18:12:41.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:41 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-08-17T18:12:41.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:41 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "18"}]': finished 2024-08-17T18:12:41.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:41 smithi136 ceph-mon[115078]: 56.10 starting backfill to osd.0 from (0'0,0'0] MAX to 1567'12732 2024-08-17T18:12:41.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:41 smithi136 ceph-mon[115078]: osdmap e1570: 8 total, 8 up, 6 in 2024-08-17T18:12:41.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:41 smithi136 ceph-mon[115078]: 56.10 starting backfill to osd.4 from (0'0,0'0] MAX to 1567'12732 2024-08-17T18:12:41.494 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:41.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:41 smithi165 ceph-mon[105517]: pgmap v2925: 109 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 105 active+clean; 2.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 650 op/s; 4693/142632 objects misplaced (3.290%); 75 MiB/s, 1.20k objects/s recovering 2024-08-17T18:12:41.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:41 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-08-17T18:12:41.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:41 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "18"}]': finished 2024-08-17T18:12:41.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:41 smithi165 ceph-mon[105517]: 56.10 starting backfill to osd.0 from (0'0,0'0] MAX to 1567'12732 2024-08-17T18:12:41.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:41 smithi165 ceph-mon[105517]: osdmap e1570: 8 total, 8 up, 6 in 2024-08-17T18:12:41.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:41 smithi165 ceph-mon[105517]: 56.10 starting backfill to osd.4 from (0'0,0'0] MAX to 1567'12732 2024-08-17T18:12:41.518 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:41.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:41.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:41.620 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:41.636 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:12:41.644 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:41.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:41.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:41.746 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:41.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:41.779 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:41.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:41.880 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:41.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:41.916 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:41.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:42.017 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:42.043 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:42.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:42.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:42.145 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:42.170 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:42.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:42.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:42.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:42.296 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:42.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:42.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:42.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:42.421 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:42.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:42.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:42.524 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:42.545 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:42.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:42.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:42.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:42.675 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:42.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:42.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:42.695 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:42 smithi136 ceph-mon[109861]: osdmap e1571: 8 total, 8 up, 6 in 2024-08-17T18:12:42.695 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:42 smithi136 ceph-mon[115078]: osdmap e1571: 8 total, 8 up, 6 in 2024-08-17T18:12:42.777 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:42.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:42.812 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:42.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:42.913 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:42.943 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:42.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:42.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:43.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:42 smithi165 ceph-mon[105517]: osdmap e1571: 8 total, 8 up, 6 in 2024-08-17T18:12:43.045 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:43.081 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:43.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:43.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:43.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:43.223 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:43.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:43.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:43.325 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:43.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:43.363 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:43.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:43.464 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:43.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:43.503 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:43.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:43.604 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:43.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:43.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:43.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:43.670 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:12:43.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:43.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:43.783 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:43.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:43.846 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:43 smithi136 ceph-mon[115078]: pgmap v2928: 109 pgs: 1 remapped+peering, 1 active+clean+scrubbing, 4 active+remapped+backfilling, 103 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 773 op/s; 5402/137967 objects misplaced (3.915%); 27 MiB/s, 426 objects/s recovering 2024-08-17T18:12:43.847 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:43 smithi136 ceph-mon[115078]: 56.11 starting backfill to osd.4 from (0'0,0'0] MAX to 1569'13102 2024-08-17T18:12:43.847 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:43 smithi136 ceph-mon[115078]: osdmap e1572: 8 total, 8 up, 6 in 2024-08-17T18:12:43.847 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:43 smithi136 ceph-mon[115078]: 56.11 starting backfill to osd.6 from (0'0,0'0] MAX to 1569'13102 2024-08-17T18:12:43.848 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:43 smithi136 ceph-mon[109861]: pgmap v2928: 109 pgs: 1 remapped+peering, 1 active+clean+scrubbing, 4 active+remapped+backfilling, 103 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 773 op/s; 5402/137967 objects misplaced (3.915%); 27 MiB/s, 426 objects/s recovering 2024-08-17T18:12:43.848 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:43 smithi136 ceph-mon[109861]: 56.11 starting backfill to osd.4 from (0'0,0'0] MAX to 1569'13102 2024-08-17T18:12:43.848 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:43 smithi136 ceph-mon[109861]: osdmap e1572: 8 total, 8 up, 6 in 2024-08-17T18:12:43.848 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:43 smithi136 ceph-mon[109861]: 56.11 starting backfill to osd.6 from (0'0,0'0] MAX to 1569'13102 2024-08-17T18:12:43.884 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:43.905 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:43.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:43.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:44.007 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:43 smithi165 ceph-mon[105517]: pgmap v2928: 109 pgs: 1 remapped+peering, 1 active+clean+scrubbing, 4 active+remapped+backfilling, 103 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 773 op/s; 5402/137967 objects misplaced (3.915%); 27 MiB/s, 426 objects/s recovering 2024-08-17T18:12:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:43 smithi165 ceph-mon[105517]: 56.11 starting backfill to osd.4 from (0'0,0'0] MAX to 1569'13102 2024-08-17T18:12:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:43 smithi165 ceph-mon[105517]: osdmap e1572: 8 total, 8 up, 6 in 2024-08-17T18:12:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:43 smithi165 ceph-mon[105517]: 56.11 starting backfill to osd.6 from (0'0,0'0] MAX to 1569'13102 2024-08-17T18:12:44.034 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:44.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:44.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:44.135 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:44.162 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:44.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:44.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:44.264 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:44.292 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:44.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:44.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:44.394 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:44.417 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:44.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:44.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:44.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:44.523 INFO:teuthology.orchestra.run.smithi136.stdout:moved 0 / 327 (0%) 2024-08-17T18:12:44.523 INFO:teuthology.orchestra.run.smithi136.stdout:avg 54.5 2024-08-17T18:12:44.523 INFO:teuthology.orchestra.run.smithi136.stdout:stddev 3.3541 -> 3.3541 (expected baseline 6.73919) 2024-08-17T18:12:44.523 INFO:teuthology.orchestra.run.smithi136.stdout:min osd.2 with 48 -> 48 pgs (0.880734 -> 0.880734 * mean) 2024-08-17T18:12:44.523 INFO:teuthology.orchestra.run.smithi136.stdout:max osd.3 with 58 -> 58 pgs (1.06422 -> 1.06422 * mean) 2024-08-17T18:12:44.523 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:12:44.523 INFO:teuthology.orchestra.run.smithi136.stdout:oload 1000 2024-08-17T18:12:44.523 INFO:teuthology.orchestra.run.smithi136.stdout:max_change 1 2024-08-17T18:12:44.524 INFO:teuthology.orchestra.run.smithi136.stdout:max_change_osds 4 2024-08-17T18:12:44.524 INFO:teuthology.orchestra.run.smithi136.stdout:average_utilization 0.0238 2024-08-17T18:12:44.524 INFO:teuthology.orchestra.run.smithi136.stdout:overload_utilization 0.2378 2024-08-17T18:12:44.524 INFO:teuthology.orchestra.run.smithi136.stderr:no change 2024-08-17T18:12:44.540 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:44.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:44.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:44.641 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:12:44.642 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:44.666 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:44.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:44.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:44.767 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:44.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:44.800 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:44.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:44.889 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:44 smithi136 ceph-mon[109861]: 56.18 scrub starts 2024-08-17T18:12:44.890 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:44 smithi136 ceph-mon[109861]: osdmap e1573: 8 total, 8 up, 6 in 2024-08-17T18:12:44.890 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:44 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/982882675' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:12:44.890 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:44 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:12:44.890 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:44 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:12:44.890 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:44 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:12:44.890 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:44 smithi136 ceph-mon[109861]: osdmap e1574: 8 total, 8 up, 6 in 2024-08-17T18:12:44.890 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:44 smithi136 ceph-mon[115078]: 56.18 scrub starts 2024-08-17T18:12:44.890 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:44 smithi136 ceph-mon[115078]: osdmap e1573: 8 total, 8 up, 6 in 2024-08-17T18:12:44.891 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:44 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/982882675' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:12:44.891 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:44 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:12:44.891 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:44 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:12:44.891 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:44 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:12:44.891 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:44 smithi136 ceph-mon[115078]: osdmap e1574: 8 total, 8 up, 6 in 2024-08-17T18:12:44.901 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:44.930 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:44.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:44.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:44 smithi165 ceph-mon[105517]: 56.18 scrub starts 2024-08-17T18:12:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:44 smithi165 ceph-mon[105517]: osdmap e1573: 8 total, 8 up, 6 in 2024-08-17T18:12:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:44 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/982882675' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:12:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:44 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:12:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:44 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:12:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:44 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:12:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:44 smithi165 ceph-mon[105517]: osdmap e1574: 8 total, 8 up, 6 in 2024-08-17T18:12:45.033 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:45.054 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:45.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:45.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:45.156 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:45.180 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:45.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:45.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:45.282 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:45.305 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:45.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:45.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:45.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:45.431 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:45.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:45.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:45.534 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:45.557 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:45.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:45.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:45.659 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:45.681 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:45.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:45.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:45.783 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:45.815 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:45.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:45.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:45.917 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:45 smithi136 ceph-mon[109861]: pgmap v2931: 109 pgs: 1 peering, 1 remapped+peering, 1 active+clean+scrubbing, 3 active+remapped+backfilling, 103 active+clean; 2.7 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 762 op/s; 2970/133482 objects misplaced (2.225%); 62 MiB/s, 999 objects/s recovering 2024-08-17T18:12:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:45 smithi136 ceph-mon[109861]: 56.8 scrub starts 2024-08-17T18:12:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:45 smithi136 ceph-mon[109861]: from='client.55588 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:12:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:45 smithi136 ceph-mon[109861]: osdmap e1575: 8 total, 8 up, 6 in 2024-08-17T18:12:45.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:45 smithi136 ceph-mon[115078]: pgmap v2931: 109 pgs: 1 peering, 1 remapped+peering, 1 active+clean+scrubbing, 3 active+remapped+backfilling, 103 active+clean; 2.7 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 762 op/s; 2970/133482 objects misplaced (2.225%); 62 MiB/s, 999 objects/s recovering 2024-08-17T18:12:45.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:45 smithi136 ceph-mon[115078]: 56.8 scrub starts 2024-08-17T18:12:45.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:45 smithi136 ceph-mon[115078]: from='client.55588 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:12:45.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:45 smithi136 ceph-mon[115078]: osdmap e1575: 8 total, 8 up, 6 in 2024-08-17T18:12:45.951 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:45.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:45.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:45.997 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:45 smithi165 ceph-mon[105517]: pgmap v2931: 109 pgs: 1 peering, 1 remapped+peering, 1 active+clean+scrubbing, 3 active+remapped+backfilling, 103 active+clean; 2.7 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 762 op/s; 2970/133482 objects misplaced (2.225%); 62 MiB/s, 999 objects/s recovering 2024-08-17T18:12:45.997 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:45 smithi165 ceph-mon[105517]: 56.8 scrub starts 2024-08-17T18:12:45.998 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:45 smithi165 ceph-mon[105517]: from='client.55588 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:12:45.998 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:45 smithi165 ceph-mon[105517]: osdmap e1575: 8 total, 8 up, 6 in 2024-08-17T18:12:46.053 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:46.080 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:46.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:46.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:46.182 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:46.206 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:46.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:46.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:46.308 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:46.330 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:46.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:46.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:46.390 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-08-17T18:12:46.390 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd test-reweight-by-utilization 110 3.0 2024-08-17T18:12:46.418 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:12:46.432 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:46.462 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:46.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:46.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:46.564 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:46.590 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:46.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:46.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:46.692 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:46.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:46.727 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:46.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:46.762 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:12:46.762 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:12:46.762 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-08-17T18:12:46.762 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-08-17T18:12:46.762 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:12:46.829 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:46.867 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:46.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:46.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:46.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:46 smithi136 ceph-mon[109861]: 56.2 scrub starts 2024-08-17T18:12:46.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:46 smithi136 ceph-mon[109861]: osdmap e1576: 8 total, 8 up, 6 in 2024-08-17T18:12:46.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:46 smithi136 ceph-mon[115078]: 56.2 scrub starts 2024-08-17T18:12:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:46 smithi136 ceph-mon[115078]: osdmap e1576: 8 total, 8 up, 6 in 2024-08-17T18:12:46.969 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:46.980 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:46 smithi165 ceph-mon[105517]: 56.2 scrub starts 2024-08-17T18:12:46.980 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:46 smithi165 ceph-mon[105517]: osdmap e1576: 8 total, 8 up, 6 in 2024-08-17T18:12:47.003 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:47.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:47.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:47.105 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:47.143 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:47.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:47.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:47.245 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:47.282 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:47.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:47.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:47.418 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:47.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:47.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:47.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:47.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:47.572 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:47.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:47.633 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:12:47.663 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:12:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:12:47] "GET /metrics HTTP/1.1" 200 36715 "" "Prometheus/2.51.0" 2024-08-17T18:12:47.673 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:47.708 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:47.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:47.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:47.810 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:47.852 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:47.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:47.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:47.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:47 smithi136 ceph-mon[109861]: pgmap v2934: 109 pgs: 2 peering, 1 remapped+peering, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 103 active+clean; 2.7 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 526 op/s; 766/131649 objects misplaced (0.582%); 53 MiB/s, 843 objects/s recovering 2024-08-17T18:12:47.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:47 smithi136 ceph-mon[109861]: 56.2 scrub ok 2024-08-17T18:12:47.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:47 smithi136 ceph-mon[109861]: 56.6 scrub starts 2024-08-17T18:12:47.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:47 smithi136 ceph-mon[115078]: pgmap v2934: 109 pgs: 2 peering, 1 remapped+peering, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 103 active+clean; 2.7 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 526 op/s; 766/131649 objects misplaced (0.582%); 53 MiB/s, 843 objects/s recovering 2024-08-17T18:12:47.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:47 smithi136 ceph-mon[115078]: 56.2 scrub ok 2024-08-17T18:12:47.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:47 smithi136 ceph-mon[115078]: 56.6 scrub starts 2024-08-17T18:12:47.954 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:47.970 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:47 smithi165 ceph-mon[105517]: pgmap v2934: 109 pgs: 2 peering, 1 remapped+peering, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 103 active+clean; 2.7 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 526 op/s; 766/131649 objects misplaced (0.582%); 53 MiB/s, 843 objects/s recovering 2024-08-17T18:12:47.971 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:47 smithi165 ceph-mon[105517]: 56.2 scrub ok 2024-08-17T18:12:47.971 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:47 smithi165 ceph-mon[105517]: 56.6 scrub starts 2024-08-17T18:12:47.986 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:47.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:47.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:48.008 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-17T18:12:48.088 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:48.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:48.116 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:48.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:48.217 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:48.275 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:48.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:48.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:48.377 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:48.402 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:48.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:48.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:48.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:48.536 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:48.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:48.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:48.638 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:48.670 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:48.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:48.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:48.772 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:48.808 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:48.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:48.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:48.910 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:48.931 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:12:48.938 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:48.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:48.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:49.040 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:49.084 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:49.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:49.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:49.186 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:49.220 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:49.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:49.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:49.321 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:49.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:49.365 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:49.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:49.466 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:49.497 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:49.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:49.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:49.599 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:49.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:49.633 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:49.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:49.733 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:49.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:49.789 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:49.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:49.889 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:49.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:49.924 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:49.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:49.925 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:49 smithi165 ceph-mon[105517]: pgmap v2936: 109 pgs: 1 peering, 1 active+remapped+backfilling, 107 active+clean; 2.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 657 op/s; 1566/126438 objects misplaced (1.239%); 31 MiB/s, 497 objects/s recovering 2024-08-17T18:12:49.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:49 smithi136 ceph-mon[109861]: pgmap v2936: 109 pgs: 1 peering, 1 active+remapped+backfilling, 107 active+clean; 2.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 657 op/s; 1566/126438 objects misplaced (1.239%); 31 MiB/s, 497 objects/s recovering 2024-08-17T18:12:49.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:49 smithi136 ceph-mon[115078]: pgmap v2936: 109 pgs: 1 peering, 1 active+remapped+backfilling, 107 active+clean; 2.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 657 op/s; 1566/126438 objects misplaced (1.239%); 31 MiB/s, 497 objects/s recovering 2024-08-17T18:12:50.025 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:50.058 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:50.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:50.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:50.159 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:50.195 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:50.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:50.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:50.297 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:50.331 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:50.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:50.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:50.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:50.476 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:50.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:50.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:50.578 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:50.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:50.625 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:50.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:50.726 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:50.786 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:50.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:50.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:50.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:50 smithi165 ceph-mon[105517]: 56.0 scrub starts 2024-08-17T18:12:50.889 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:50.928 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:50.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:50.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:50.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:50 smithi136 ceph-mon[109861]: 56.0 scrub starts 2024-08-17T18:12:50.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:50 smithi136 ceph-mon[115078]: 56.0 scrub starts 2024-08-17T18:12:51.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:51.070 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:51.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:51.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:51.172 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:51.200 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:51.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:51.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:51.302 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:51.335 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:51.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:51.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:51.437 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:51.465 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:51.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:51.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:51.509 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:12:51.509 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T18:12:51.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T18:12:51.509 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [] 2024-08-17T18:12:51.509 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:12:51.568 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:51.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:51.605 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:51.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:51.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:51.747 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:51.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:51.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:51.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:51 smithi165 ceph-mon[105517]: pgmap v2937: 109 pgs: 1 peering, 1 active+remapped+backfilling, 107 active+clean; 2.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1566/122289 objects misplaced (1.281%); 24 MiB/s, 391 objects/s recovering 2024-08-17T18:12:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:12:51.849 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:51.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:51.882 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:51.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:51.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:51 smithi136 ceph-mon[109861]: pgmap v2937: 109 pgs: 1 peering, 1 active+remapped+backfilling, 107 active+clean; 2.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1566/122289 objects misplaced (1.281%); 24 MiB/s, 391 objects/s recovering 2024-08-17T18:12:51.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:12:51.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:51 smithi136 ceph-mon[115078]: pgmap v2937: 109 pgs: 1 peering, 1 active+remapped+backfilling, 107 active+clean; 2.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1566/122289 objects misplaced (1.281%); 24 MiB/s, 391 objects/s recovering 2024-08-17T18:12:51.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:12:51.983 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:52.017 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:52.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:52.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:52.119 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:52.153 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:52.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:52.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:52.201 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-17T18:12:52.255 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:52.291 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:52.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:52.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:52.393 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:52.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:52.430 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:52.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:52.531 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:52.561 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:52.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:52.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:52.663 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:52.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:52.696 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:52.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:52.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:52.832 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:52.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:52.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:52.934 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:52.967 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:52.968 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-08-17T18:12:52.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:52.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:53.069 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:53.117 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:53.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:53.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:53.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-08-17T18:12:53.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-08-17T18:12:53.220 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:53.241 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:53.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:53.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:53.343 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:53.365 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:53.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:53.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:53.466 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:53.487 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:53.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:53.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:53.589 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:53.610 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:53.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:53.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:53.712 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:53.734 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:53.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:53.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:53.836 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:53.857 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:53.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:53.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:53.959 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:53.981 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:53.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:53.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:54.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:53 smithi165 ceph-mon[105517]: pgmap v2938: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.3 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 130/113541 objects misplaced (0.114%); 24 MiB/s, 387 objects/s recovering 2024-08-17T18:12:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "19"}]': finished 2024-08-17T18:12:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:53 smithi165 ceph-mon[105517]: osdmap e1577: 8 total, 8 up, 6 in 2024-08-17T18:12:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:53 smithi165 ceph-mon[105517]: 56.15 scrub starts 2024-08-17T18:12:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:53 smithi165 ceph-mon[105517]: 56.11 scrub starts 2024-08-17T18:12:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:53 smithi165 ceph-mon[105517]: osdmap e1578: 8 total, 8 up, 6 in 2024-08-17T18:12:54.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:53 smithi136 ceph-mon[109861]: pgmap v2938: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.3 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 130/113541 objects misplaced (0.114%); 24 MiB/s, 387 objects/s recovering 2024-08-17T18:12:54.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "19"}]': finished 2024-08-17T18:12:54.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:53 smithi136 ceph-mon[109861]: osdmap e1577: 8 total, 8 up, 6 in 2024-08-17T18:12:54.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:53 smithi136 ceph-mon[109861]: 56.15 scrub starts 2024-08-17T18:12:54.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:53 smithi136 ceph-mon[109861]: 56.11 scrub starts 2024-08-17T18:12:54.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:53 smithi136 ceph-mon[109861]: osdmap e1578: 8 total, 8 up, 6 in 2024-08-17T18:12:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:53 smithi136 ceph-mon[115078]: pgmap v2938: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.3 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 130/113541 objects misplaced (0.114%); 24 MiB/s, 387 objects/s recovering 2024-08-17T18:12:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "19"}]': finished 2024-08-17T18:12:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:53 smithi136 ceph-mon[115078]: osdmap e1577: 8 total, 8 up, 6 in 2024-08-17T18:12:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:53 smithi136 ceph-mon[115078]: 56.15 scrub starts 2024-08-17T18:12:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:53 smithi136 ceph-mon[115078]: 56.11 scrub starts 2024-08-17T18:12:54.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:53 smithi136 ceph-mon[115078]: osdmap e1578: 8 total, 8 up, 6 in 2024-08-17T18:12:54.083 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:54.105 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:54.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:54.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:54.207 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:54.228 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:54.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:54.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:54.330 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:54.352 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:54.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:54.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:54.453 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:54.475 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:54.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:54.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:54.577 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:54.599 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:54.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:54.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:54.702 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:54.731 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:54.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:54.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:54.833 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:54.863 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:54.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:54.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:54.965 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:54.995 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:54.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:54.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:55.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:54 smithi165 ceph-mon[105517]: 56.15 scrub ok 2024-08-17T18:12:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:54 smithi165 ceph-mon[105517]: 56.11 scrub ok 2024-08-17T18:12:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:54 smithi165 ceph-mon[105517]: 56.10 scrub starts 2024-08-17T18:12:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:54 smithi165 ceph-mon[105517]: 56.10 scrub ok 2024-08-17T18:12:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:54 smithi165 ceph-mon[105517]: 1.3 scrub starts 2024-08-17T18:12:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:54 smithi165 ceph-mon[105517]: 1.3 scrub ok 2024-08-17T18:12:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:54 smithi165 ceph-mon[105517]: 56.5 scrub starts 2024-08-17T18:12:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:54 smithi165 ceph-mon[105517]: 56.14 scrub starts 2024-08-17T18:12:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:54 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:12:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:54 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:12:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:54 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-08-17T18:12:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:54 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "20"}]': finished 2024-08-17T18:12:55.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:54 smithi165 ceph-mon[105517]: osdmap e1579: 8 total, 8 up, 6 in 2024-08-17T18:12:55.097 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:55.127 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:55.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:55.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:55.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[109861]: 56.15 scrub ok 2024-08-17T18:12:55.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[109861]: 56.11 scrub ok 2024-08-17T18:12:55.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[109861]: 56.10 scrub starts 2024-08-17T18:12:55.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[109861]: 56.10 scrub ok 2024-08-17T18:12:55.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[109861]: 1.3 scrub starts 2024-08-17T18:12:55.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[109861]: 1.3 scrub ok 2024-08-17T18:12:55.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[109861]: 56.5 scrub starts 2024-08-17T18:12:55.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[109861]: 56.14 scrub starts 2024-08-17T18:12:55.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:12:55.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:12:55.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-08-17T18:12:55.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "20"}]': finished 2024-08-17T18:12:55.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[109861]: osdmap e1579: 8 total, 8 up, 6 in 2024-08-17T18:12:55.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[115078]: 56.15 scrub ok 2024-08-17T18:12:55.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[115078]: 56.11 scrub ok 2024-08-17T18:12:55.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[115078]: 56.10 scrub starts 2024-08-17T18:12:55.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[115078]: 56.10 scrub ok 2024-08-17T18:12:55.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[115078]: 1.3 scrub starts 2024-08-17T18:12:55.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[115078]: 1.3 scrub ok 2024-08-17T18:12:55.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[115078]: 56.5 scrub starts 2024-08-17T18:12:55.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[115078]: 56.14 scrub starts 2024-08-17T18:12:55.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:12:55.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:12:55.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-08-17T18:12:55.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "20"}]': finished 2024-08-17T18:12:55.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:54 smithi136 ceph-mon[115078]: osdmap e1579: 8 total, 8 up, 6 in 2024-08-17T18:12:55.229 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:55.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:55.253 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:55.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:55.354 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:55.378 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:55.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:55.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:55.480 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:55.500 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:55.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:55.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:55.602 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:55.624 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:55.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:55.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:55.726 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:55.748 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:55.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:55.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:55.850 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:55.871 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:55.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:55.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:55.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:55 smithi136 ceph-mon[109861]: 56.5 scrub ok 2024-08-17T18:12:55.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:55 smithi136 ceph-mon[109861]: 56.14 scrub ok 2024-08-17T18:12:55.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:55 smithi136 ceph-mon[109861]: pgmap v2941: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 324 op/s; 130/104145 objects misplaced (0.125%); 15 MiB/s, 246 objects/s recovering 2024-08-17T18:12:55.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:55 smithi136 ceph-mon[109861]: 56.18 scrub starts 2024-08-17T18:12:55.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:55 smithi136 ceph-mon[109861]: 56.18 scrub ok 2024-08-17T18:12:55.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:55 smithi136 ceph-mon[109861]: 56.3 scrub starts 2024-08-17T18:12:55.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:55 smithi136 ceph-mon[109861]: 56.3 scrub ok 2024-08-17T18:12:55.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:55 smithi136 ceph-mon[109861]: 56.12 starting backfill to osd.4 from (0'0,0'0] MAX to 1576'13367 2024-08-17T18:12:55.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:55 smithi136 ceph-mon[109861]: osdmap e1580: 8 total, 8 up, 6 in 2024-08-17T18:12:55.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:55 smithi136 ceph-mon[115078]: 56.5 scrub ok 2024-08-17T18:12:55.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:55 smithi136 ceph-mon[115078]: 56.14 scrub ok 2024-08-17T18:12:55.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:55 smithi136 ceph-mon[115078]: pgmap v2941: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 324 op/s; 130/104145 objects misplaced (0.125%); 15 MiB/s, 246 objects/s recovering 2024-08-17T18:12:55.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:55 smithi136 ceph-mon[115078]: 56.18 scrub starts 2024-08-17T18:12:55.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:55 smithi136 ceph-mon[115078]: 56.18 scrub ok 2024-08-17T18:12:55.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:55 smithi136 ceph-mon[115078]: 56.3 scrub starts 2024-08-17T18:12:55.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:55 smithi136 ceph-mon[115078]: 56.3 scrub ok 2024-08-17T18:12:55.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:55 smithi136 ceph-mon[115078]: 56.12 starting backfill to osd.4 from (0'0,0'0] MAX to 1576'13367 2024-08-17T18:12:55.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:55 smithi136 ceph-mon[115078]: osdmap e1580: 8 total, 8 up, 6 in 2024-08-17T18:12:55.972 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:55.993 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:55.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:55.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:56.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:55 smithi165 ceph-mon[105517]: 56.5 scrub ok 2024-08-17T18:12:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:55 smithi165 ceph-mon[105517]: 56.14 scrub ok 2024-08-17T18:12:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:55 smithi165 ceph-mon[105517]: pgmap v2941: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 324 op/s; 130/104145 objects misplaced (0.125%); 15 MiB/s, 246 objects/s recovering 2024-08-17T18:12:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:55 smithi165 ceph-mon[105517]: 56.18 scrub starts 2024-08-17T18:12:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:55 smithi165 ceph-mon[105517]: 56.18 scrub ok 2024-08-17T18:12:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:55 smithi165 ceph-mon[105517]: 56.3 scrub starts 2024-08-17T18:12:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:55 smithi165 ceph-mon[105517]: 56.3 scrub ok 2024-08-17T18:12:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:55 smithi165 ceph-mon[105517]: 56.12 starting backfill to osd.4 from (0'0,0'0] MAX to 1576'13367 2024-08-17T18:12:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:55 smithi165 ceph-mon[105517]: osdmap e1580: 8 total, 8 up, 6 in 2024-08-17T18:12:56.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:56.117 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:56.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:56.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:56.218 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:56.242 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:56.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:56.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:56.343 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:56.366 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:56.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:56.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:56.468 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:12:56.491 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:56.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:56.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:12:56.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:56.614 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:56.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:56.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:56.716 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:56.739 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:56.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:56.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:56.841 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:56.868 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:56.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:56.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:56.970 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:56.995 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:56.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:56.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:57.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:56 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-08-17T18:12:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:56 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "21"}]': finished 2024-08-17T18:12:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:56 smithi165 ceph-mon[105517]: osdmap e1581: 8 total, 8 up, 6 in 2024-08-17T18:12:57.097 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:57.119 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:57.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:57.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:57.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:56 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-08-17T18:12:57.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:56 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "21"}]': finished 2024-08-17T18:12:57.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:56 smithi136 ceph-mon[115078]: osdmap e1581: 8 total, 8 up, 6 in 2024-08-17T18:12:57.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:56 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-08-17T18:12:57.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:56 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "21"}]': finished 2024-08-17T18:12:57.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:56 smithi136 ceph-mon[109861]: osdmap e1581: 8 total, 8 up, 6 in 2024-08-17T18:12:57.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:57.243 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:57.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:57.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:57.344 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:57.367 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:57.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:57.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:57.468 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:57.490 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:57.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:57.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:57.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:57.614 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:57.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:57.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:57.716 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:57.720 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:12:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:12:57] "GET /metrics HTTP/1.1" 200 36711 "" "Prometheus/2.51.0" 2024-08-17T18:12:57.743 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:57.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:57.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:57.845 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:57.870 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:57.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:57.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:57.972 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:12:57.979 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:57 smithi136 ceph-mon[109861]: pgmap v2944: 109 pgs: 2 active+remapped+backfilling, 107 active+clean; 2.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 480/101655 objects misplaced (0.472%) 2024-08-17T18:12:57.979 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:57 smithi136 ceph-mon[109861]: 56.1 scrub starts 2024-08-17T18:12:57.979 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:57 smithi136 ceph-mon[109861]: 56.1 scrub ok 2024-08-17T18:12:57.979 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:57 smithi136 ceph-mon[109861]: 56.13 starting backfill to osd.3 from (0'0,0'0] MAX to 1577'13069 2024-08-17T18:12:57.979 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:57 smithi136 ceph-mon[109861]: osdmap e1582: 8 total, 8 up, 6 in 2024-08-17T18:12:57.979 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:57 smithi136 ceph-mon[115078]: pgmap v2944: 109 pgs: 2 active+remapped+backfilling, 107 active+clean; 2.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 480/101655 objects misplaced (0.472%) 2024-08-17T18:12:57.979 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:57 smithi136 ceph-mon[115078]: 56.1 scrub starts 2024-08-17T18:12:57.979 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:57 smithi136 ceph-mon[115078]: 56.1 scrub ok 2024-08-17T18:12:57.979 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:57 smithi136 ceph-mon[115078]: 56.13 starting backfill to osd.3 from (0'0,0'0] MAX to 1577'13069 2024-08-17T18:12:57.980 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:57 smithi136 ceph-mon[115078]: osdmap e1582: 8 total, 8 up, 6 in 2024-08-17T18:12:57.996 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:57.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:57.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:12:58.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:57 smithi165 ceph-mon[105517]: pgmap v2944: 109 pgs: 2 active+remapped+backfilling, 107 active+clean; 2.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 480/101655 objects misplaced (0.472%) 2024-08-17T18:12:58.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:57 smithi165 ceph-mon[105517]: 56.1 scrub starts 2024-08-17T18:12:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:57 smithi165 ceph-mon[105517]: 56.1 scrub ok 2024-08-17T18:12:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:57 smithi165 ceph-mon[105517]: 56.13 starting backfill to osd.3 from (0'0,0'0] MAX to 1577'13069 2024-08-17T18:12:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:57 smithi165 ceph-mon[105517]: osdmap e1582: 8 total, 8 up, 6 in 2024-08-17T18:12:58.098 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:58.120 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:58.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:58.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:58.222 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:58.244 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:58.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:58.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:58.345 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:58.367 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:58.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:58.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:58.464 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:12:58.464 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:12:58.464 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:12:58.464 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:12:58.464 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:12:58.464 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:12:58.464 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:12:58.464 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:12:58.465 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:12:58.465 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:12:58.465 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:12:58.465 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:12:58.465 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:12:58.465 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:12:58.465 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:12:58.465 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:12:58.468 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:58.490 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:58.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:58.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:58.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:58.613 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:58.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:58.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:58.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:58.736 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:58.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:58.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:58.838 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:58.859 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:58.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:58.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:58.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:12:58.983 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:58.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:58.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:12:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:58 smithi165 ceph-mon[105517]: 56.8 scrub starts 2024-08-17T18:12:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:58 smithi165 ceph-mon[105517]: 56.8 scrub ok 2024-08-17T18:12:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:58 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1308150985' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:12:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:58 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-08-17T18:12:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:58 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "22"}]': finished 2024-08-17T18:12:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:58 smithi165 ceph-mon[105517]: osdmap e1583: 8 total, 8 up, 6 in 2024-08-17T18:12:59.084 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:59.105 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:59.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:59.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:59.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:58 smithi136 ceph-mon[109861]: 56.8 scrub starts 2024-08-17T18:12:59.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:58 smithi136 ceph-mon[109861]: 56.8 scrub ok 2024-08-17T18:12:59.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:58 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1308150985' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:12:59.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:58 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-08-17T18:12:59.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:58 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "22"}]': finished 2024-08-17T18:12:59.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:58 smithi136 ceph-mon[109861]: osdmap e1583: 8 total, 8 up, 6 in 2024-08-17T18:12:59.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:58 smithi136 ceph-mon[115078]: 56.8 scrub starts 2024-08-17T18:12:59.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:58 smithi136 ceph-mon[115078]: 56.8 scrub ok 2024-08-17T18:12:59.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:58 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1308150985' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:12:59.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:58 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-08-17T18:12:59.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:58 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "22"}]': finished 2024-08-17T18:12:59.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:58 smithi136 ceph-mon[115078]: osdmap e1583: 8 total, 8 up, 6 in 2024-08-17T18:12:59.207 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:59.234 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:59.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:59.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:59.336 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:12:59.373 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:59.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:59.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:12:59.475 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:12:59.504 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:59.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:59.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:12:59.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:12:59.631 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:59.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:59.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:12:59.733 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:12:59.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:59.765 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:12:59.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:12:59.866 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:12:59.897 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:12:59.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:12:59.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:12:59.928 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:12:59.928 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:12:59.932 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:12:59.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:59 smithi136 ceph-mon[109861]: pgmap v2947: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 350/99549 objects misplaced (0.352%) 2024-08-17T18:12:59.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:59 smithi136 ceph-mon[109861]: 56.17 scrub starts 2024-08-17T18:12:59.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:59 smithi136 ceph-mon[109861]: 56.17 scrub ok 2024-08-17T18:12:59.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:59 smithi136 ceph-mon[109861]: 56.14 starting backfill to osd.0 from (0'0,0'0] MAX to 1579'13183 2024-08-17T18:12:59.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:59 smithi136 ceph-mon[109861]: 56.4 scrub starts 2024-08-17T18:12:59.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:59 smithi136 ceph-mon[109861]: 56.4 scrub ok 2024-08-17T18:12:59.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:59 smithi136 ceph-mon[109861]: 56.9 scrub starts 2024-08-17T18:12:59.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:12:59 smithi136 ceph-mon[109861]: osdmap e1584: 8 total, 8 up, 6 in 2024-08-17T18:12:59.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:59 smithi136 ceph-mon[115078]: pgmap v2947: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 350/99549 objects misplaced (0.352%) 2024-08-17T18:12:59.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:59 smithi136 ceph-mon[115078]: 56.17 scrub starts 2024-08-17T18:12:59.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:59 smithi136 ceph-mon[115078]: 56.17 scrub ok 2024-08-17T18:12:59.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:59 smithi136 ceph-mon[115078]: 56.14 starting backfill to osd.0 from (0'0,0'0] MAX to 1579'13183 2024-08-17T18:12:59.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:59 smithi136 ceph-mon[115078]: 56.4 scrub starts 2024-08-17T18:12:59.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:59 smithi136 ceph-mon[115078]: 56.4 scrub ok 2024-08-17T18:12:59.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:59 smithi136 ceph-mon[115078]: 56.9 scrub starts 2024-08-17T18:12:59.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:12:59 smithi136 ceph-mon[115078]: osdmap e1584: 8 total, 8 up, 6 in 2024-08-17T18:12:59.999 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:00.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:59 smithi165 ceph-mon[105517]: pgmap v2947: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 350/99549 objects misplaced (0.352%) 2024-08-17T18:13:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:59 smithi165 ceph-mon[105517]: 56.17 scrub starts 2024-08-17T18:13:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:59 smithi165 ceph-mon[105517]: 56.17 scrub ok 2024-08-17T18:13:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:59 smithi165 ceph-mon[105517]: 56.14 starting backfill to osd.0 from (0'0,0'0] MAX to 1579'13183 2024-08-17T18:13:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:59 smithi165 ceph-mon[105517]: 56.4 scrub starts 2024-08-17T18:13:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:59 smithi165 ceph-mon[105517]: 56.4 scrub ok 2024-08-17T18:13:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:59 smithi165 ceph-mon[105517]: 56.9 scrub starts 2024-08-17T18:13:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:12:59 smithi165 ceph-mon[105517]: osdmap e1584: 8 total, 8 up, 6 in 2024-08-17T18:13:00.023 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:00.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:00.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:00.125 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:00.149 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:00.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:00.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:00.251 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:00.274 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:00.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:00.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:00.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:00.400 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:00.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:00.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:00.502 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:00.527 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:00.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:00.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:00.629 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:00.652 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:00.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:00.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:00.754 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:00.787 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:00.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:00.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:00.889 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:00.924 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:00.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:00.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:01.026 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:01.061 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:01.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:01.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:01.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:00 smithi136 ceph-mon[109861]: 56.12 scrub starts 2024-08-17T18:13:01.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:00 smithi136 ceph-mon[109861]: 56.12 scrub ok 2024-08-17T18:13:01.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:00 smithi136 ceph-mon[109861]: 56.9 scrub ok 2024-08-17T18:13:01.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:00 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:13:01.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:00 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:13:01.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:00 smithi136 ceph-mon[109861]: pgmap v2950: 109 pgs: 1 peering, 1 active+remapped+backfilling, 107 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 350/98235 objects misplaced (0.356%) 2024-08-17T18:13:01.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:00 smithi136 ceph-mon[109861]: osdmap e1585: 8 total, 8 up, 6 in 2024-08-17T18:13:01.080 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:00 smithi136 ceph-mon[115078]: 56.12 scrub starts 2024-08-17T18:13:01.081 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:00 smithi136 ceph-mon[115078]: 56.12 scrub ok 2024-08-17T18:13:01.081 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:00 smithi136 ceph-mon[115078]: 56.9 scrub ok 2024-08-17T18:13:01.081 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:00 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:13:01.081 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:00 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:13:01.081 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:00 smithi136 ceph-mon[115078]: pgmap v2950: 109 pgs: 1 peering, 1 active+remapped+backfilling, 107 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 350/98235 objects misplaced (0.356%) 2024-08-17T18:13:01.081 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:00 smithi136 ceph-mon[115078]: osdmap e1585: 8 total, 8 up, 6 in 2024-08-17T18:13:01.162 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:01.195 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:01.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:01.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:00 smithi165 ceph-mon[105517]: 56.12 scrub starts 2024-08-17T18:13:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:00 smithi165 ceph-mon[105517]: 56.12 scrub ok 2024-08-17T18:13:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:00 smithi165 ceph-mon[105517]: 56.9 scrub ok 2024-08-17T18:13:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:00 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:13:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:00 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:13:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:00 smithi165 ceph-mon[105517]: pgmap v2950: 109 pgs: 1 peering, 1 active+remapped+backfilling, 107 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 350/98235 objects misplaced (0.356%) 2024-08-17T18:13:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:00 smithi165 ceph-mon[105517]: osdmap e1585: 8 total, 8 up, 6 in 2024-08-17T18:13:01.297 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:01.334 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:01.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:01.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:01.436 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:01.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:01.476 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:01.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:01.577 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:01.611 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:01.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:01.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:01.712 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:01.744 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:01.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:01.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:01.846 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:01.884 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:01.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:01.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:01.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:02.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:02.024 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:02.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:02.125 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:02.151 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:02.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:02.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:02.252 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:02.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:01 smithi165 ceph-mon[105517]: 56.7 scrub starts 2024-08-17T18:13:02.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:01 smithi165 ceph-mon[105517]: 56.7 scrub ok 2024-08-17T18:13:02.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:01 smithi165 ceph-mon[105517]: 56.15 starting backfill to osd.2 from (0'0,0'0] MAX to 1582'13244 2024-08-17T18:13:02.286 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:02.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:02.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:02.388 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:02.421 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:02.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:02.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:02.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:01 smithi136 ceph-mon[109861]: 56.7 scrub starts 2024-08-17T18:13:02.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:01 smithi136 ceph-mon[109861]: 56.7 scrub ok 2024-08-17T18:13:02.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:01 smithi136 ceph-mon[109861]: 56.15 starting backfill to osd.2 from (0'0,0'0] MAX to 1582'13244 2024-08-17T18:13:02.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:01 smithi136 ceph-mon[115078]: 56.7 scrub starts 2024-08-17T18:13:02.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:01 smithi136 ceph-mon[115078]: 56.7 scrub ok 2024-08-17T18:13:02.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:01 smithi136 ceph-mon[115078]: 56.15 starting backfill to osd.2 from (0'0,0'0] MAX to 1582'13244 2024-08-17T18:13:02.512 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:13:02.523 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:02.543 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:02.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:02.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:02.645 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:02.687 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:02.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:02.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:02.790 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:02.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:02.825 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:02.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:02.926 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:02.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:02.963 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:02.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:03.064 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:03.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:03.098 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:03.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:03.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:02 smithi136 ceph-mon[109861]: pgmap v2952: 109 pgs: 2 active+remapped+backfilling, 1 peering, 106 active+clean; 1.9 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 730/94689 objects misplaced (0.771%) 2024-08-17T18:13:03.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:02 smithi136 ceph-mon[115078]: pgmap v2952: 109 pgs: 2 active+remapped+backfilling, 1 peering, 106 active+clean; 1.9 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 730/94689 objects misplaced (0.771%) 2024-08-17T18:13:03.200 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:03.232 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:03.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:03.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:03.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:02 smithi165 ceph-mon[105517]: pgmap v2952: 109 pgs: 2 active+remapped+backfilling, 1 peering, 106 active+clean; 1.9 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 730/94689 objects misplaced (0.771%) 2024-08-17T18:13:03.334 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:03.364 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:03.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:03.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:03.466 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:03.491 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:03.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:03.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:03.593 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:03.632 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:03.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:03.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:03.733 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:03.760 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:03.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:03.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:03.862 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:03.887 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:03.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:03.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:03.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:04.011 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:04.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:04.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:04.113 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:04.150 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:04.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:04.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:04.252 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:03 smithi165 ceph-mon[105517]: osdmap e1586: 8 total, 8 up, 6 in 2024-08-17T18:13:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:03 smithi165 ceph-mon[105517]: 56.16 scrub starts 2024-08-17T18:13:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:13:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:13:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:13:04.286 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:04.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:04.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:04.388 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:04.410 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:04.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:04.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:04.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:03 smithi136 ceph-mon[109861]: osdmap e1586: 8 total, 8 up, 6 in 2024-08-17T18:13:04.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:03 smithi136 ceph-mon[109861]: 56.16 scrub starts 2024-08-17T18:13:04.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:13:04.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:13:04.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:13:04.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:03 smithi136 ceph-mon[115078]: osdmap e1586: 8 total, 8 up, 6 in 2024-08-17T18:13:04.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:03 smithi136 ceph-mon[115078]: 56.16 scrub starts 2024-08-17T18:13:04.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:13:04.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:13:04.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:13:04.512 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:04.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:04.546 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:04.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:04.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:04.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:04.680 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:04.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:04.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:04.802 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:04.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:04.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:04.904 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:04.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:04.927 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:04.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:05.028 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:05.049 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:05.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:05.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:05.151 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:05.173 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:05.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:05.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:04 smithi165 ceph-mon[105517]: 56.16 scrub ok 2024-08-17T18:13:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:04 smithi165 ceph-mon[105517]: osdmap e1587: 8 total, 8 up, 6 in 2024-08-17T18:13:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:04 smithi165 ceph-mon[105517]: pgmap v2955: 109 pgs: 2 active+remapped+backfilling, 107 active+clean; 1.8 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 730/87051 objects misplaced (0.839%) 2024-08-17T18:13:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:04 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-08-17T18:13:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:04 smithi165 ceph-mon[105517]: 56.19 scrub starts 2024-08-17T18:13:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:04 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "23"}]': finished 2024-08-17T18:13:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:04 smithi165 ceph-mon[105517]: osdmap e1588: 8 total, 8 up, 6 in 2024-08-17T18:13:05.275 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:05.297 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:05.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:05.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:05.399 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:05.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:05.422 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:05.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:05.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:04 smithi136 ceph-mon[109861]: 56.16 scrub ok 2024-08-17T18:13:05.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:04 smithi136 ceph-mon[109861]: osdmap e1587: 8 total, 8 up, 6 in 2024-08-17T18:13:05.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:04 smithi136 ceph-mon[109861]: pgmap v2955: 109 pgs: 2 active+remapped+backfilling, 107 active+clean; 1.8 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 730/87051 objects misplaced (0.839%) 2024-08-17T18:13:05.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:04 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-08-17T18:13:05.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:04 smithi136 ceph-mon[109861]: 56.19 scrub starts 2024-08-17T18:13:05.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:04 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "23"}]': finished 2024-08-17T18:13:05.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:04 smithi136 ceph-mon[109861]: osdmap e1588: 8 total, 8 up, 6 in 2024-08-17T18:13:05.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:04 smithi136 ceph-mon[115078]: 56.16 scrub ok 2024-08-17T18:13:05.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:04 smithi136 ceph-mon[115078]: osdmap e1587: 8 total, 8 up, 6 in 2024-08-17T18:13:05.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:04 smithi136 ceph-mon[115078]: pgmap v2955: 109 pgs: 2 active+remapped+backfilling, 107 active+clean; 1.8 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 730/87051 objects misplaced (0.839%) 2024-08-17T18:13:05.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:04 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-08-17T18:13:05.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:04 smithi136 ceph-mon[115078]: 56.19 scrub starts 2024-08-17T18:13:05.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:04 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "23"}]': finished 2024-08-17T18:13:05.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:04 smithi136 ceph-mon[115078]: osdmap e1588: 8 total, 8 up, 6 in 2024-08-17T18:13:05.524 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:05.550 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:05.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:05.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:05.652 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:05.678 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:05.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:05.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:05.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:05.807 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:05.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:05.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:05.909 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:05.911 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:13:05.932 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:05.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:05.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:06.034 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:06.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:06.057 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:06.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:06.157 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:06.179 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:06.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:06.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:06.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:05 smithi136 ceph-mon[109861]: 56.19 scrub ok 2024-08-17T18:13:06.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:05 smithi136 ceph-mon[109861]: osdmap e1589: 8 total, 8 up, 6 in 2024-08-17T18:13:06.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:05 smithi136 ceph-mon[115078]: 56.19 scrub ok 2024-08-17T18:13:06.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:05 smithi136 ceph-mon[115078]: osdmap e1589: 8 total, 8 up, 6 in 2024-08-17T18:13:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:05 smithi165 ceph-mon[105517]: 56.19 scrub ok 2024-08-17T18:13:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:05 smithi165 ceph-mon[105517]: osdmap e1589: 8 total, 8 up, 6 in 2024-08-17T18:13:06.281 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:06.302 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:06.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:06.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:06.404 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:06.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:06.427 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:06.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:06.528 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:06.550 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:06.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:06.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:06.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:06.675 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:06.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:06.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:06.776 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:06.807 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:06.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:06.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:06.909 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:06.940 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:06.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:06.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:07.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:07.066 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:07.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:07.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:07.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:07.189 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:07.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:07.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:07.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:13:07.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:06 smithi136 ceph-mon[109861]: pgmap v2958: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.7 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 719/83409 objects misplaced (0.862%); 176 KiB/s, 2 objects/s recovering 2024-08-17T18:13:07.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "24"} : dispatch 2024-08-17T18:13:07.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "24"}]': finished 2024-08-17T18:13:07.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:06 smithi136 ceph-mon[109861]: 56.16 starting backfill to osd.2 from (0'0,0'0] MAX to 1587'13562 2024-08-17T18:13:07.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:06 smithi136 ceph-mon[109861]: osdmap e1590: 8 total, 8 up, 6 in 2024-08-17T18:13:07.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:06 smithi136 ceph-mon[109861]: 56.16 starting backfill to osd.6 from (0'0,0'0] MAX to 1587'13562 2024-08-17T18:13:07.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:13:07.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:06 smithi136 ceph-mon[115078]: pgmap v2958: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.7 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 719/83409 objects misplaced (0.862%); 176 KiB/s, 2 objects/s recovering 2024-08-17T18:13:07.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "24"} : dispatch 2024-08-17T18:13:07.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "24"}]': finished 2024-08-17T18:13:07.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:06 smithi136 ceph-mon[115078]: 56.16 starting backfill to osd.2 from (0'0,0'0] MAX to 1587'13562 2024-08-17T18:13:07.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:06 smithi136 ceph-mon[115078]: osdmap e1590: 8 total, 8 up, 6 in 2024-08-17T18:13:07.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:06 smithi136 ceph-mon[115078]: 56.16 starting backfill to osd.6 from (0'0,0'0] MAX to 1587'13562 2024-08-17T18:13:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:13:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:06 smithi165 ceph-mon[105517]: pgmap v2958: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.7 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 719/83409 objects misplaced (0.862%); 176 KiB/s, 2 objects/s recovering 2024-08-17T18:13:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "24"} : dispatch 2024-08-17T18:13:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "24"}]': finished 2024-08-17T18:13:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:06 smithi165 ceph-mon[105517]: 56.16 starting backfill to osd.2 from (0'0,0'0] MAX to 1587'13562 2024-08-17T18:13:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:06 smithi165 ceph-mon[105517]: osdmap e1590: 8 total, 8 up, 6 in 2024-08-17T18:13:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:06 smithi165 ceph-mon[105517]: 56.16 starting backfill to osd.6 from (0'0,0'0] MAX to 1587'13562 2024-08-17T18:13:07.292 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:07.315 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:07.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:07.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:07.416 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:07.439 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:07.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:07.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:07.541 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:07.562 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:07.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:07.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:07.664 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:07.687 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:07.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:07.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:07.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:13:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:13:07] "GET /metrics HTTP/1.1" 200 36679 "" "Prometheus/2.51.0" 2024-08-17T18:13:07.789 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:07.823 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:07.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:07.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:07.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:07.957 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:13:07.957 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:13:07.961 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:07.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:07.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:08.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:08.098 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:08.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:08.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:08.200 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:08.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:08.236 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:08.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:08.337 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:08.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:08.375 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:08.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:08.476 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:08.510 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:08.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:08.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:08.611 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:08.650 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:08.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:08.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:08.752 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:08.788 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:08.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:08.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:08.890 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:08.925 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:08.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:08.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:08.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:08 smithi136 ceph-mon[109861]: osdmap e1591: 8 total, 8 up, 6 in 2024-08-17T18:13:08.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:08 smithi136 ceph-mon[109861]: 56.13 scrub starts 2024-08-17T18:13:08.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:08 smithi136 ceph-mon[115078]: osdmap e1591: 8 total, 8 up, 6 in 2024-08-17T18:13:08.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:08 smithi136 ceph-mon[115078]: 56.13 scrub starts 2024-08-17T18:13:09.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:08 smithi165 ceph-mon[105517]: osdmap e1591: 8 total, 8 up, 6 in 2024-08-17T18:13:09.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:08 smithi165 ceph-mon[105517]: 56.13 scrub starts 2024-08-17T18:13:09.027 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:09.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:09.074 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:09.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:09.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:09.215 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:09.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:09.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:09.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:09.352 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:09.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:09.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:09.454 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:09.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:09.496 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:09.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:09.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:09.636 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:09.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:09.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:09.751 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:09.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:09.785 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:09.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:09.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:09.907 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:09.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:09.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:09.912 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:09 smithi136 ceph-mon[109861]: pgmap v2961: 109 pgs: 1 peering, 1 active+remapped+backfilling, 107 active+clean; 1.6 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 606 op/s; 1468/79767 objects misplaced (1.840%); 12 MiB/s, 193 objects/s recovering 2024-08-17T18:13:09.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:09 smithi136 ceph-mon[115078]: pgmap v2961: 109 pgs: 1 peering, 1 active+remapped+backfilling, 107 active+clean; 1.6 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 606 op/s; 1468/79767 objects misplaced (1.840%); 12 MiB/s, 193 objects/s recovering 2024-08-17T18:13:10.009 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:10.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:09 smithi165 ceph-mon[105517]: pgmap v2961: 109 pgs: 1 peering, 1 active+remapped+backfilling, 107 active+clean; 1.6 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 606 op/s; 1468/79767 objects misplaced (1.840%); 12 MiB/s, 193 objects/s recovering 2024-08-17T18:13:10.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:10.041 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:10.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:10.142 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:10.166 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:10.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:10.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:10.196 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:13:09 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T18:13:09.879+0000 7f0cc2ea7640 -1 mgr.server reply reply (2) No such file or directory no change 2024-08-17T18:13:10.269 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:10.306 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:10.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:10.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:10.384 INFO:teuthology.orchestra.run.smithi136.stdout:moved 17 / 327 (5.19878%) 2024-08-17T18:13:10.384 INFO:teuthology.orchestra.run.smithi136.stdout:avg 54.5 2024-08-17T18:13:10.384 INFO:teuthology.orchestra.run.smithi136.stdout:stddev 2.36291 -> 4.46281 (expected baseline 6.73919) 2024-08-17T18:13:10.384 INFO:teuthology.orchestra.run.smithi136.stdout:min osd.2 with 50 -> 55 pgs (0.917431 -> 1.00917 * mean) 2024-08-17T18:13:10.384 INFO:teuthology.orchestra.run.smithi136.stdout:max osd.3 with 57 -> 59 pgs (1.04587 -> 1.08257 * mean) 2024-08-17T18:13:10.384 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:13:10.384 INFO:teuthology.orchestra.run.smithi136.stdout:oload 110 2024-08-17T18:13:10.384 INFO:teuthology.orchestra.run.smithi136.stdout:max_change 3 2024-08-17T18:13:10.384 INFO:teuthology.orchestra.run.smithi136.stdout:max_change_osds 4 2024-08-17T18:13:10.384 INFO:teuthology.orchestra.run.smithi136.stdout:average_utilization 0.0184 2024-08-17T18:13:10.385 INFO:teuthology.orchestra.run.smithi136.stdout:overload_utilization 0.0202 2024-08-17T18:13:10.385 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5 weight 1.0000 -> 0.7914 2024-08-17T18:13:10.385 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4 weight 1.0000 -> 0.8844 2024-08-17T18:13:10.385 INFO:teuthology.orchestra.run.smithi136.stderr:no change 2024-08-17T18:13:10.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:10.433 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:10.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:10.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:10.535 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:10.571 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:10.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:10.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:10.673 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:10.706 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:10.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:10.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:10.809 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:10.840 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:10.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:10.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:10.942 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:10.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:10.973 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:10.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:11.075 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:11.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:11.113 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:11.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:11.214 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:11.246 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:11.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:11.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:11.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:11.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:11.385 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:11.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:11.486 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:11.516 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:11.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:11.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:11.617 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:11.646 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:11.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:11.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:11.748 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:11.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:11.780 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:11.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:11.838 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-17T18:13:11.881 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:11.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:11.916 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:11.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:11.933 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:11 smithi136 ceph-mon[115078]: from='client.46181 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:13:11.933 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:11 smithi136 ceph-mon[115078]: pgmap v2962: 109 pgs: 1 peering, 1 active+remapped+backfilling, 107 active+clean; 1.6 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1468/76794 objects misplaced (1.912%); 8.4 MiB/s, 135 objects/s recovering 2024-08-17T18:13:11.933 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:11 smithi136 ceph-mon[109861]: from='client.46181 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:13:11.933 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:11 smithi136 ceph-mon[109861]: pgmap v2962: 109 pgs: 1 peering, 1 active+remapped+backfilling, 107 active+clean; 1.6 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1468/76794 objects misplaced (1.912%); 8.4 MiB/s, 135 objects/s recovering 2024-08-17T18:13:11.979 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:11 smithi165 ceph-mon[105517]: from='client.46181 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:13:11.979 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:11 smithi165 ceph-mon[105517]: pgmap v2962: 109 pgs: 1 peering, 1 active+remapped+backfilling, 107 active+clean; 1.6 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1468/76794 objects misplaced (1.912%); 8.4 MiB/s, 135 objects/s recovering 2024-08-17T18:13:12.018 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:12.051 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:12.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:12.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:12.153 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:12.183 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:12.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:12.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:12.286 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:12.309 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:12.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:12.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:12.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:12.422 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-08-17T18:13:12.422 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight-by-utilization 1000 1.0 2024-08-17T18:13:12.434 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:12.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:12.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:12.536 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:12.557 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:12.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:12.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:12.659 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:12.685 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:12.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:12.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:12.786 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:12.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:12.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:12.919 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:12.945 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:12.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:12 smithi136 ceph-mon[115078]: osdmap e1592: 8 total, 8 up, 6 in 2024-08-17T18:13:12.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "25"} : dispatch 2024-08-17T18:13:12.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:12 smithi136 ceph-mon[109861]: osdmap e1592: 8 total, 8 up, 6 in 2024-08-17T18:13:12.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "25"} : dispatch 2024-08-17T18:13:12.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:12.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:12 smithi165 ceph-mon[105517]: osdmap e1592: 8 total, 8 up, 6 in 2024-08-17T18:13:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:12 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "25"} : dispatch 2024-08-17T18:13:13.049 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:13.070 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:13.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:13.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:13.172 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:13.197 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:13.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:13.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:13.216 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:13:13.216 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:13:13.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:13.321 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:13.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:13.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:13.423 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:13.445 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:13.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:13.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:13.550 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:13.573 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:13.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:13.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:13.673 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:13:13.675 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:13.696 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:13.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:13.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:13.798 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:13.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:13.821 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:13.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:13.923 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:13.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:13.945 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:13.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:13 smithi136 ceph-mon[109861]: pgmap v2964: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.4 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 1468/69453 objects misplaced (2.114%); 7.9 MiB/s, 127 objects/s recovering 2024-08-17T18:13:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:13 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "25"}]': finished 2024-08-17T18:13:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:13 smithi136 ceph-mon[109861]: osdmap e1593: 8 total, 8 up, 6 in 2024-08-17T18:13:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:13 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3074685479' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:13:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:13 smithi136 ceph-mon[115078]: pgmap v2964: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.4 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 1468/69453 objects misplaced (2.114%); 7.9 MiB/s, 127 objects/s recovering 2024-08-17T18:13:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:13 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "25"}]': finished 2024-08-17T18:13:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:13 smithi136 ceph-mon[115078]: osdmap e1593: 8 total, 8 up, 6 in 2024-08-17T18:13:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:13 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3074685479' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:13:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:13 smithi165 ceph-mon[105517]: pgmap v2964: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.4 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 1468/69453 objects misplaced (2.114%); 7.9 MiB/s, 127 objects/s recovering 2024-08-17T18:13:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:13 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "25"}]': finished 2024-08-17T18:13:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:13 smithi165 ceph-mon[105517]: osdmap e1593: 8 total, 8 up, 6 in 2024-08-17T18:13:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:13 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3074685479' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:13:14.046 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:14.068 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:14.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:14.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:14.170 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:14.192 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:14.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:14.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:14.294 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:14.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:14.316 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:14.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:14.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:14.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:14.439 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:14.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:14.540 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:14.564 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:14.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:14.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:14.666 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:14.690 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:14.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:14.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:14.792 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:14.813 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:14.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:14.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:14.846 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:14 smithi136 ceph-mon[115078]: 56.6 scrub starts 2024-08-17T18:13:14.846 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:14 smithi136 ceph-mon[115078]: 56.6 scrub ok 2024-08-17T18:13:14.846 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:14 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:13:14.846 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:14 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T18:13:14.846 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:14 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3074685479' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:13:14.846 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:14 smithi136 ceph-mon[115078]: osdmap e1594: 8 total, 8 up, 6 in 2024-08-17T18:13:14.846 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:14 smithi136 ceph-mon[109861]: 56.6 scrub starts 2024-08-17T18:13:14.847 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:14 smithi136 ceph-mon[109861]: 56.6 scrub ok 2024-08-17T18:13:14.847 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:14 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:13:14.847 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:14 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T18:13:14.847 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:14 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3074685479' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:13:14.847 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:14 smithi136 ceph-mon[109861]: osdmap e1594: 8 total, 8 up, 6 in 2024-08-17T18:13:14.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:14.943 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:13:14.944 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:13:14.944 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:13:14.944 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:13:14.944 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:13:14.944 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:13:14.944 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:13:14.944 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:13:14.945 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:14.946 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:13:14.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:14.947 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:13:14.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:14.976 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:13:14.976 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:13:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:14 smithi165 ceph-mon[105517]: 56.6 scrub starts 2024-08-17T18:13:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:14 smithi165 ceph-mon[105517]: 56.6 scrub ok 2024-08-17T18:13:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:14 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:13:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:14 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T18:13:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:14 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3074685479' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:13:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:14 smithi165 ceph-mon[105517]: osdmap e1594: 8 total, 8 up, 6 in 2024-08-17T18:13:15.013 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:13:15.014 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:13:15.044 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:13:15.045 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:13:15.048 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:15.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:15.086 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:15.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:15.187 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:15.221 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:15.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:15.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:15.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:15.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:15.363 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:15.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:15.464 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:15.499 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:15.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:15.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:15.601 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:15.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:15.637 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:15.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:15.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:15.771 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:15.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:15.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:15.873 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:15.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:15.909 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:15.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:15.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:15 smithi136 ceph-mon[109861]: pgmap v2967: 109 pgs: 1 peering, 108 active+clean; 1.2 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 7.1 MiB/s, 114 objects/s recovering 2024-08-17T18:13:15.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:15 smithi136 ceph-mon[109861]: osdmap e1595: 8 total, 8 up, 6 in 2024-08-17T18:13:15.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:15 smithi136 ceph-mon[109861]: 56.18 starting backfill to osd.3 from (0'0,0'0] MAX to 1592'13678 2024-08-17T18:13:15.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:15 smithi136 ceph-mon[109861]: 56.18 starting backfill to osd.5 from (0'0,0'0] MAX to 1592'13678 2024-08-17T18:13:15.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:15 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/758007493' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:13:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:15 smithi136 ceph-mon[115078]: pgmap v2967: 109 pgs: 1 peering, 108 active+clean; 1.2 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 7.1 MiB/s, 114 objects/s recovering 2024-08-17T18:13:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:15 smithi136 ceph-mon[115078]: osdmap e1595: 8 total, 8 up, 6 in 2024-08-17T18:13:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:15 smithi136 ceph-mon[115078]: 56.18 starting backfill to osd.3 from (0'0,0'0] MAX to 1592'13678 2024-08-17T18:13:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:15 smithi136 ceph-mon[115078]: 56.18 starting backfill to osd.5 from (0'0,0'0] MAX to 1592'13678 2024-08-17T18:13:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:15 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/758007493' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:13:16.010 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:15 smithi165 ceph-mon[105517]: pgmap v2967: 109 pgs: 1 peering, 108 active+clean; 1.2 GiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 7.1 MiB/s, 114 objects/s recovering 2024-08-17T18:13:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:15 smithi165 ceph-mon[105517]: osdmap e1595: 8 total, 8 up, 6 in 2024-08-17T18:13:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:15 smithi165 ceph-mon[105517]: 56.18 starting backfill to osd.3 from (0'0,0'0] MAX to 1592'13678 2024-08-17T18:13:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:15 smithi165 ceph-mon[105517]: 56.18 starting backfill to osd.5 from (0'0,0'0] MAX to 1592'13678 2024-08-17T18:13:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:15 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/758007493' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:13:16.043 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:16.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:16.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:16.145 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:16.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:16.177 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:16.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:16.278 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:16.310 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:16.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:16.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:16.412 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:16.448 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:16.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:16.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:16.550 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:16.586 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:16.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:16.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:16.688 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:16.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:16.725 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:16.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:16.827 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:16.859 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:16.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:16.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:16.962 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:16.995 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:16.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:16.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:17.098 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:17.136 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:17.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:17.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:17.238 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:17.275 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:17.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:17.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:17.377 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:17.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:17.422 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:17.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:17.523 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:17.567 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:17.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:17.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:17.638 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:13:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:13:17] "GET /metrics HTTP/1.1" 200 36679 "" "Prometheus/2.51.0" 2024-08-17T18:13:17.669 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:17.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:17.707 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:17.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:17.808 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:17.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:17.850 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:17.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:17.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:17 smithi136 ceph-mon[109861]: pgmap v2969: 109 pgs: 1 active+remapped+backfilling, 1 peering, 107 active+clean; 1.2 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 744/57780 objects misplaced (1.288%); 11 MiB/s, 177 objects/s recovering 2024-08-17T18:13:17.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:17 smithi136 ceph-mon[115078]: pgmap v2969: 109 pgs: 1 active+remapped+backfilling, 1 peering, 107 active+clean; 1.2 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 744/57780 objects misplaced (1.288%); 11 MiB/s, 177 objects/s recovering 2024-08-17T18:13:17.952 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:17.989 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:17.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:17.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:17 smithi165 ceph-mon[105517]: pgmap v2969: 109 pgs: 1 active+remapped+backfilling, 1 peering, 107 active+clean; 1.2 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 744/57780 objects misplaced (1.288%); 11 MiB/s, 177 objects/s recovering 2024-08-17T18:13:18.091 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:18.122 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:18.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:18.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:18.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:18.262 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:18.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:18.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:18.366 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:18.400 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:18.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:18.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:18.502 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:18.545 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:18.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:18.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:18.606 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:13:18.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:18.709 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:18.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:18.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:18.811 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:18.855 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:18.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:18.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:18.871 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:18 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-08-17T18:13:18.871 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:18 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-08-17T18:13:18.933 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:13:18.933 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T18:13:18.933 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-17T18:13:18.933 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-17T18:13:18.934 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:13:18.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:18.988 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:18.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:18.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:18 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-08-17T18:13:19.052 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:13:19.091 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:19.115 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:19.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:19.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:19.217 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:19.239 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:19.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:19.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:19.340 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:19.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:19.363 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:19.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:19.465 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:19.486 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:19.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:19.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:19.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:19.609 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:19.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:19.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:19.711 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:19.732 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:19.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:19.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:19.788 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:19 smithi136 ceph-mon[115078]: pgmap v2970: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.1 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 744/51873 objects misplaced (1.434%); 8.7 MiB/s, 139 objects/s recovering 2024-08-17T18:13:19.788 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:19 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-08-17T18:13:19.788 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:19 smithi136 ceph-mon[115078]: osdmap e1596: 8 total, 8 up, 6 in 2024-08-17T18:13:19.788 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:19 smithi136 ceph-mon[115078]: 56.0 scrub starts 2024-08-17T18:13:19.788 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:19 smithi136 ceph-mon[115078]: 56.0 scrub ok 2024-08-17T18:13:19.788 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:19 smithi136 ceph-mon[115078]: osdmap e1597: 8 total, 8 up, 6 in 2024-08-17T18:13:19.788 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:19 smithi136 ceph-mon[109861]: pgmap v2970: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.1 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 744/51873 objects misplaced (1.434%); 8.7 MiB/s, 139 objects/s recovering 2024-08-17T18:13:19.788 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:19 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-08-17T18:13:19.789 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:19 smithi136 ceph-mon[109861]: osdmap e1596: 8 total, 8 up, 6 in 2024-08-17T18:13:19.789 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:19 smithi136 ceph-mon[109861]: 56.0 scrub starts 2024-08-17T18:13:19.789 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:19 smithi136 ceph-mon[109861]: 56.0 scrub ok 2024-08-17T18:13:19.789 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:19 smithi136 ceph-mon[109861]: osdmap e1597: 8 total, 8 up, 6 in 2024-08-17T18:13:19.810 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-17T18:13:19.834 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:19.855 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:19.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:19.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:19.957 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:19.979 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:19.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:19.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:19 smithi165 ceph-mon[105517]: pgmap v2970: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.1 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 744/51873 objects misplaced (1.434%); 8.7 MiB/s, 139 objects/s recovering 2024-08-17T18:13:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:19 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-08-17T18:13:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:19 smithi165 ceph-mon[105517]: osdmap e1596: 8 total, 8 up, 6 in 2024-08-17T18:13:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:19 smithi165 ceph-mon[105517]: 56.0 scrub starts 2024-08-17T18:13:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:19 smithi165 ceph-mon[105517]: 56.0 scrub ok 2024-08-17T18:13:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:19 smithi165 ceph-mon[105517]: osdmap e1597: 8 total, 8 up, 6 in 2024-08-17T18:13:20.082 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:20.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:20.103 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:20.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:20.204 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:20.226 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:20.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:20.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:20.265 INFO:teuthology.orchestra.run.smithi136.stdout:moved 0 / 327 (0%) 2024-08-17T18:13:20.265 INFO:teuthology.orchestra.run.smithi136.stdout:avg 54.5 2024-08-17T18:13:20.265 INFO:teuthology.orchestra.run.smithi136.stdout:stddev 2.81366 -> 2.81366 (expected baseline 6.73919) 2024-08-17T18:13:20.265 INFO:teuthology.orchestra.run.smithi136.stdout:min osd.2 with 49 -> 49 pgs (0.899083 -> 0.899083 * mean) 2024-08-17T18:13:20.266 INFO:teuthology.orchestra.run.smithi136.stdout:max osd.3 with 58 -> 58 pgs (1.06422 -> 1.06422 * mean) 2024-08-17T18:13:20.266 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:13:20.266 INFO:teuthology.orchestra.run.smithi136.stdout:oload 1000 2024-08-17T18:13:20.266 INFO:teuthology.orchestra.run.smithi136.stdout:max_change 1 2024-08-17T18:13:20.266 INFO:teuthology.orchestra.run.smithi136.stdout:max_change_osds 4 2024-08-17T18:13:20.266 INFO:teuthology.orchestra.run.smithi136.stdout:average_utilization 0.0149 2024-08-17T18:13:20.266 INFO:teuthology.orchestra.run.smithi136.stdout:overload_utilization 0.1485 2024-08-17T18:13:20.266 INFO:teuthology.orchestra.run.smithi136.stderr:no change 2024-08-17T18:13:20.328 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:20.349 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:20.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:20.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:20.451 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:20.473 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:20.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:20.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:20.487 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:13:20.574 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:20.595 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:20.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:20.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:20.697 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:20.720 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:20.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:20.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:20.821 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:20.843 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:20.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:20.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:20.945 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:20.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:20 smithi136 ceph-mon[109861]: 56.13 scrub starts 2024-08-17T18:13:20.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:20 smithi136 ceph-mon[109861]: osdmap e1598: 8 total, 8 up, 6 in 2024-08-17T18:13:20.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:20 smithi136 ceph-mon[115078]: 56.13 scrub starts 2024-08-17T18:13:20.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:20 smithi136 ceph-mon[115078]: osdmap e1598: 8 total, 8 up, 6 in 2024-08-17T18:13:20.967 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:20.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:20.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:21.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:20 smithi165 ceph-mon[105517]: 56.13 scrub starts 2024-08-17T18:13:21.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:20 smithi165 ceph-mon[105517]: osdmap e1598: 8 total, 8 up, 6 in 2024-08-17T18:13:21.068 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:21.074 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-08-17T18:13:21.075 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd test-reweight-by-utilization 110 3.0 2024-08-17T18:13:21.090 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:21.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:21.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:21.192 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:21.214 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:21.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:21.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:21.316 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:21.337 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:21.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:21.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:21.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:21.463 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:21.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:21.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:21.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:21.603 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:21.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:21.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:21.705 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:21.736 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:21.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:21.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:21.838 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:21.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:21.877 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:21.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:21 smithi136 ceph-mon[109861]: from='client.36318 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:13:21.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:21 smithi136 ceph-mon[109861]: pgmap v2973: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 990 MiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 722 op/s; 744/47586 objects misplaced (1.563%); 1.6 MiB/s, 25 objects/s recovering 2024-08-17T18:13:21.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:21 smithi136 ceph-mon[109861]: 56.13 scrub ok 2024-08-17T18:13:21.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:21 smithi136 ceph-mon[109861]: 56.19 starting backfill to osd.4 from (0'0,0'0] MAX to 1595'13380 2024-08-17T18:13:21.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:21 smithi136 ceph-mon[109861]: 56.19 starting backfill to osd.6 from (0'0,0'0] MAX to 1595'13380 2024-08-17T18:13:21.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:13:21.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:13:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:21 smithi136 ceph-mon[115078]: from='client.36318 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:13:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:21 smithi136 ceph-mon[115078]: pgmap v2973: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 990 MiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 722 op/s; 744/47586 objects misplaced (1.563%); 1.6 MiB/s, 25 objects/s recovering 2024-08-17T18:13:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:21 smithi136 ceph-mon[115078]: 56.13 scrub ok 2024-08-17T18:13:21.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:21 smithi136 ceph-mon[115078]: 56.19 starting backfill to osd.4 from (0'0,0'0] MAX to 1595'13380 2024-08-17T18:13:21.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:21 smithi136 ceph-mon[115078]: 56.19 starting backfill to osd.6 from (0'0,0'0] MAX to 1595'13380 2024-08-17T18:13:21.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:13:21.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:13:21.978 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:21 smithi165 ceph-mon[105517]: from='client.36318 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:13:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:21 smithi165 ceph-mon[105517]: pgmap v2973: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 990 MiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 722 op/s; 744/47586 objects misplaced (1.563%); 1.6 MiB/s, 25 objects/s recovering 2024-08-17T18:13:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:21 smithi165 ceph-mon[105517]: 56.13 scrub ok 2024-08-17T18:13:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:21 smithi165 ceph-mon[105517]: 56.19 starting backfill to osd.4 from (0'0,0'0] MAX to 1595'13380 2024-08-17T18:13:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:21 smithi165 ceph-mon[105517]: 56.19 starting backfill to osd.6 from (0'0,0'0] MAX to 1595'13380 2024-08-17T18:13:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:13:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:13:22.015 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:22.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:22.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:22.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:22.155 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:22.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:22.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:22.257 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:22.294 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:22.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:22.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:22.396 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:22.431 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:22.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:22.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:22.452 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:13:22.533 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:22.541 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:13:22.586 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:22.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:22.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:22.688 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:22.723 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:22.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:22.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:22.825 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:22.861 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:22.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:22.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:22.963 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:22.997 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:22.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:22.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:23.099 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:23.132 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:23.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:23.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:23.234 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:23.268 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:23.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:23.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:23.370 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:23.402 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:23.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:23.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:23.504 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:23.545 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:23.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:23.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:23.647 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:23.684 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:23.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:23.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:23.786 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:23.816 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:23.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:23.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:23.918 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:23.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:23.951 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:23.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:23 smithi165 ceph-mon[105517]: pgmap v2975: 109 pgs: 1 activating+remapped, 108 active+clean; 874 MiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 875 op/s; 864/42015 objects misplaced (2.056%); 3.3 MiB/s, 52 objects/s recovering 2024-08-17T18:13:24.052 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:24.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:23 smithi136 ceph-mon[109861]: pgmap v2975: 109 pgs: 1 activating+remapped, 108 active+clean; 874 MiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 875 op/s; 864/42015 objects misplaced (2.056%); 3.3 MiB/s, 52 objects/s recovering 2024-08-17T18:13:24.074 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:23 smithi136 ceph-mon[115078]: pgmap v2975: 109 pgs: 1 activating+remapped, 108 active+clean; 874 MiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 875 op/s; 864/42015 objects misplaced (2.056%); 3.3 MiB/s, 52 objects/s recovering 2024-08-17T18:13:24.087 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:24.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:24.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:24.189 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:24.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:24.222 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:24.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:24.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:24.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:24.363 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:24.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:24.464 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:24.507 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:24.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:24.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:24.574 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:13:24.609 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:24.641 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:24.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:24.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:24.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:24.777 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:24.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:24.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:24.879 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:24.911 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:24.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:24.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:25.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:25.043 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:25.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:25.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:25.145 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:25.176 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:25.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:25.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:25.278 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:25.301 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:25.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:25.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:25.403 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:25.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:25.425 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:25.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:25.527 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:25.552 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:25.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:25.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:25.655 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:25.676 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:25.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:25.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:25.778 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:25.815 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:25.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:25.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:25.906 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:25 smithi136 ceph-mon[109861]: pgmap v2976: 109 pgs: 1 activating+remapped, 108 active+clean; 813 MiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 710 op/s; 864/39084 objects misplaced (2.211%); 3.3 MiB/s, 52 objects/s recovering 2024-08-17T18:13:25.907 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:25 smithi136 ceph-mon[109861]: osdmap e1599: 8 total, 8 up, 6 in 2024-08-17T18:13:25.907 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:25 smithi136 ceph-mon[115078]: pgmap v2976: 109 pgs: 1 activating+remapped, 108 active+clean; 813 MiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 710 op/s; 864/39084 objects misplaced (2.211%); 3.3 MiB/s, 52 objects/s recovering 2024-08-17T18:13:25.907 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:25 smithi136 ceph-mon[115078]: osdmap e1599: 8 total, 8 up, 6 in 2024-08-17T18:13:25.917 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:25.955 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:25.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:25.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:25 smithi165 ceph-mon[105517]: pgmap v2976: 109 pgs: 1 activating+remapped, 108 active+clean; 813 MiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 0 B/s wr, 710 op/s; 864/39084 objects misplaced (2.211%); 3.3 MiB/s, 52 objects/s recovering 2024-08-17T18:13:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:25 smithi165 ceph-mon[105517]: osdmap e1599: 8 total, 8 up, 6 in 2024-08-17T18:13:26.057 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:26.084 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:26.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:26.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:26.185 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:26.196 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:13:26.197 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:13:26.206 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:26.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:26.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:26.214 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:13:26.215 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:13:26.230 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:13:26.231 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:13:26.246 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:13:26.246 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:13:26.277 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:13:26.278 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:13:26.305 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:13:26.306 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:13:26.308 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:26.338 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:13:26.339 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:13:26.344 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:26.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:26.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:26.367 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:13:26.367 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:13:26.446 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:26.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:26.484 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:26.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:26.585 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:26.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:26.618 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:26.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:26.719 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:26.753 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:26.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:26.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:26.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:26 smithi136 ceph-mon[109861]: osdmap e1600: 8 total, 8 up, 6 in 2024-08-17T18:13:26.797 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:26 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2521485548' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:13:26.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:26 smithi136 ceph-mon[115078]: osdmap e1600: 8 total, 8 up, 6 in 2024-08-17T18:13:26.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:26 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2521485548' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:13:26.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:26.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:26.889 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:26.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:26.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:26 smithi165 ceph-mon[105517]: osdmap e1600: 8 total, 8 up, 6 in 2024-08-17T18:13:27.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:26 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2521485548' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:13:27.028 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:27.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:27.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:27.130 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:27.163 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:27.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:27.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:27.265 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:27.301 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:27.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:27.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:27.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:27.440 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:27.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:27.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:27.542 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:27.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:27.581 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:27.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:27.682 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:27.694 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:27 smithi136 ceph-mon[109861]: pgmap v2979: 109 pgs: 1 activating+remapped, 108 active+clean; 661 MiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 864/31758 objects misplaced (2.721%) 2024-08-17T18:13:27.694 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:27 smithi136 ceph-mon[115078]: pgmap v2979: 109 pgs: 1 activating+remapped, 108 active+clean; 661 MiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 864/31758 objects misplaced (2.721%) 2024-08-17T18:13:27.695 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:13:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:13:27] "GET /metrics HTTP/1.1" 200 36687 "" "Prometheus/2.51.0" 2024-08-17T18:13:27.695 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:13:27.695 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:13:27.719 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:27.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:27.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:27.820 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:27.844 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:27.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:27.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:27.945 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:27.966 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:27.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:27.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:28.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:27 smithi165 ceph-mon[105517]: pgmap v2979: 109 pgs: 1 activating+remapped, 108 active+clean; 661 MiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 864/31758 objects misplaced (2.721%) 2024-08-17T18:13:28.068 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:28.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:28.103 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:28.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:28.203 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:28.242 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:28.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:28.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:28.343 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:28.381 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:28.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:28.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:28.466 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-17T18:13:28.483 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:28.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:28.524 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:28.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:28.625 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:28.659 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:28.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:28.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:28.761 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:28.806 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:28.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:28.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:28.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:28.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:28.942 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:28.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:29.043 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:29.079 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:29.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:29.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:29.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:29.191 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:13:29.219 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:29.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:29.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:29.322 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:29.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:29.362 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:29.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:29.446 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:13:29 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T18:13:29.098+0000 7f0cc2ea7640 -1 mgr.server reply reply (2) No such file or directory no change 2024-08-17T18:13:29.463 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:29.488 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:29.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:29.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:29.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:29.624 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:29.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:29.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:29.727 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:29.764 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:29.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:29.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:29.866 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:29.904 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:29.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:29.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:29.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:29 smithi136 ceph-mon[109861]: pgmap v2980: 109 pgs: 109 active+clean; 469 MiB data, 6.3 GiB used, 530 GiB / 536 GiB avail 2024-08-17T18:13:29.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:29 smithi136 ceph-mon[115078]: pgmap v2980: 109 pgs: 109 active+clean; 469 MiB data, 6.3 GiB used, 530 GiB / 536 GiB avail 2024-08-17T18:13:30.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:30.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:29 smithi165 ceph-mon[105517]: pgmap v2980: 109 pgs: 109 active+clean; 469 MiB data, 6.3 GiB used, 530 GiB / 536 GiB avail 2024-08-17T18:13:30.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:30.048 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:30.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:30.150 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:30.173 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:30.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:30.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:30.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:30.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:30.314 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:30.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:30.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:30.452 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:30.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:30.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:30.554 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:30.575 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:30.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:30.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:30.677 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:30.714 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:30.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:30.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:30.816 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:30.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:30.846 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:30.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:30.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:30 smithi136 ceph-mon[115078]: from='client.36330 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:13:30.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:30 smithi136 ceph-mon[109861]: from='client.36330 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:13:30.948 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:30.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:30.971 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:30.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:30 smithi165 ceph-mon[105517]: from='client.36330 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:13:31.072 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:31.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:31.110 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:31.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:31.211 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:31.244 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:31.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:31.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:31.346 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:31.384 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:31.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:31.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:31.486 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:31.517 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:31.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:31.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:31.619 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:31.650 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:31.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:31.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:31.752 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:31.752 INFO:teuthology.orchestra.run.smithi136.stdout:moved 17 / 327 (5.19878%) 2024-08-17T18:13:31.753 INFO:teuthology.orchestra.run.smithi136.stdout:avg 54.5 2024-08-17T18:13:31.753 INFO:teuthology.orchestra.run.smithi136.stdout:stddev 2.81366 -> 6.07591 (expected baseline 6.73919) 2024-08-17T18:13:31.753 INFO:teuthology.orchestra.run.smithi136.stdout:min osd.2 with 49 -> 50 pgs (0.899083 -> 0.917431 * mean) 2024-08-17T18:13:31.753 INFO:teuthology.orchestra.run.smithi136.stdout:max osd.3 with 58 -> 63 pgs (1.06422 -> 1.15596 * mean) 2024-08-17T18:13:31.753 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:13:31.753 INFO:teuthology.orchestra.run.smithi136.stdout:oload 110 2024-08-17T18:13:31.753 INFO:teuthology.orchestra.run.smithi136.stdout:max_change 3 2024-08-17T18:13:31.753 INFO:teuthology.orchestra.run.smithi136.stdout:max_change_osds 4 2024-08-17T18:13:31.753 INFO:teuthology.orchestra.run.smithi136.stdout:average_utilization 0.0117 2024-08-17T18:13:31.753 INFO:teuthology.orchestra.run.smithi136.stdout:overload_utilization 0.0129 2024-08-17T18:13:31.753 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6 weight 1.0000 -> 0.8848 2024-08-17T18:13:31.753 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5 weight 1.0000 -> 0.7389 2024-08-17T18:13:31.754 INFO:teuthology.orchestra.run.smithi136.stderr:no change 2024-08-17T18:13:31.786 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:31.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:31.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:31.888 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:31.919 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:31.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:31.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:31.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:31 smithi136 ceph-mon[109861]: pgmap v2981: 109 pgs: 109 active+clean; 379 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail 2024-08-17T18:13:31.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:31 smithi136 ceph-mon[115078]: pgmap v2981: 109 pgs: 109 active+clean; 379 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail 2024-08-17T18:13:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:31 smithi165 ceph-mon[105517]: pgmap v2981: 109 pgs: 109 active+clean; 379 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail 2024-08-17T18:13:32.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:32.056 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:32.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:32.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:32.158 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:32.197 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:32.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:32.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:32.299 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:32.341 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:32.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:32.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:32.443 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:32.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:32.475 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:32.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:32.493 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Removed 126701 objects 2024-08-17T18:13:32.494 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Clean up completed and total clean up time :143.033 2024-08-17T18:13:32.576 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:32.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:32.599 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:32.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:32.700 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:32.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:32.722 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:32.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:32.823 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:32.844 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:32.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:32.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:32.946 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:32.967 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:32.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:32.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:33.069 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:33.093 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:33.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:33.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:33.195 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:33.219 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:33.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:33.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:33.320 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:33.342 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:33.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:33.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:33.444 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:33.470 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:33.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:33.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:33.571 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:33.596 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:33.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:33.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:33.698 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:33.720 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:33.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:33.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:33.822 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:33.844 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:33.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:33.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:33.941 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:33 smithi136 ceph-mon[109861]: pgmap v2982: 109 pgs: 109 active+clean; 205 MiB data, 5.6 GiB used, 531 GiB / 536 GiB avail 2024-08-17T18:13:33.942 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:33 smithi136 ceph-mon[115078]: pgmap v2982: 109 pgs: 109 active+clean; 205 MiB data, 5.6 GiB used, 531 GiB / 536 GiB avail 2024-08-17T18:13:33.946 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:33.971 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-08-17T18:13:33.971 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd test-reweight-by-utilization 110 0.05 2024-08-17T18:13:33.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:33.973 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:33.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:34.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:33 smithi165 ceph-mon[105517]: pgmap v2982: 109 pgs: 109 active+clean; 205 MiB data, 5.6 GiB used, 531 GiB / 536 GiB avail 2024-08-17T18:13:34.074 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:34.096 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:34.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:34.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:34.197 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:34.219 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:34.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:34.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:34.321 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:34.343 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:34.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:34.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:34.445 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:34.466 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:34.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:34.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:34.568 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:34.589 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:34.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:34.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:34.691 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:34.714 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:34.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:34.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:34.815 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:34.837 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:34.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:34.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:34.939 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:34.960 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:34.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:34.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:35.062 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:35.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:35.087 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:35.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:35.188 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:35.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:35.211 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:35.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:35.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:35.335 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:35.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:35.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:35.436 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:35.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:35.460 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:35.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:35.510 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:13:35.510 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:13:35.544 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:13:35.561 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:35.583 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:35.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:35.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:35.708 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:35.730 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:35.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:35.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:35.833 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:35.854 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:35.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:35.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:35.906 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:35 smithi136 ceph-mon[109861]: pgmap v2983: 109 pgs: 109 active+clean; 96 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 2.2 MiB/s, 35 objects/s recovering 2024-08-17T18:13:35.906 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:35 smithi136 ceph-mon[115078]: pgmap v2983: 109 pgs: 109 active+clean; 96 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 2.2 MiB/s, 35 objects/s recovering 2024-08-17T18:13:35.956 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:35.979 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:35.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:35.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:35 smithi165 ceph-mon[105517]: pgmap v2983: 109 pgs: 109 active+clean; 96 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 1.07k op/s; 2.2 MiB/s, 35 objects/s recovering 2024-08-17T18:13:36.081 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:36.103 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:36.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:36.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:36.205 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:36.228 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:36.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:36.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:36.329 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:36.350 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:36.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:36.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:36.452 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:36.476 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:36.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:36.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:36.578 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:36.601 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:36.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:36.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:36.696 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:13:36.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:36.725 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:36.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:36.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:36.827 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:36.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:36.850 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:36.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:36 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/599481035' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:13:36.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:36 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:13:36.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:13:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:36 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/599481035' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:13:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:36 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:13:36.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:13:36.951 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:36.973 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:36.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:36.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:36 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/599481035' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:13:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:36 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:13:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:13:37.075 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:37.099 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:37.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:37.200 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:37.225 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:37.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:37.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:37.327 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:37.343 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-08-17T18:13:37.343 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-08-17T18:13:37.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:37.350 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:37.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:37.452 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:37.473 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:37.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:37.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:37.575 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:37.601 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:37.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:37.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:37.609 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:13:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:13:37] "GET /metrics HTTP/1.1" 200 36686 "" "Prometheus/2.51.0" 2024-08-17T18:13:37.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:37.729 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:37.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:37.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:37.831 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:37.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:37.853 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:37.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:37.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:37 smithi136 ceph-mon[109861]: pgmap v2984: 109 pgs: 109 active+clean; 6.5 MiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 1.10k op/s; 2.0 MiB/s, 32 objects/s recovering 2024-08-17T18:13:37.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:37 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:13:37.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:37 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:13:37.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:37 smithi136 ceph-mon[109861]: osdmap e1601: 8 total, 8 up, 6 in 2024-08-17T18:13:37.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:37 smithi136 ceph-mon[115078]: pgmap v2984: 109 pgs: 109 active+clean; 6.5 MiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 1.10k op/s; 2.0 MiB/s, 32 objects/s recovering 2024-08-17T18:13:37.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:37 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:13:37.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:37 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:13:37.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:37 smithi136 ceph-mon[115078]: osdmap e1601: 8 total, 8 up, 6 in 2024-08-17T18:13:37.954 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:37.975 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:37.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:37.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:37 smithi165 ceph-mon[105517]: pgmap v2984: 109 pgs: 109 active+clean; 6.5 MiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 1.10k op/s; 2.0 MiB/s, 32 objects/s recovering 2024-08-17T18:13:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:37 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:13:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:37 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:13:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:37 smithi165 ceph-mon[105517]: osdmap e1601: 8 total, 8 up, 6 in 2024-08-17T18:13:38.077 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:38.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:38.103 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:38.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:38.204 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:38.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:38.230 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:38.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:38.331 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:38.356 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:38.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:38.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:38.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:38.484 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:38.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:38.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:38.585 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:38.610 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:38.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:38.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:38.712 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:38.735 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:38.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:38.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:38.837 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:38.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:38.863 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:38.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:38.964 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:38.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:38.990 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:38.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:39.091 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:39.116 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:39.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:39.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:39.219 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:39.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:39.250 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:39.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:39.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:39.375 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:39.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:39.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:39.477 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:39.502 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:39.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:39.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:39.604 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:39.626 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:39.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:39.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:39.632 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:13:39.728 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:39.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:39.750 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:39.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:39.758 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:39 smithi136 ceph-mon[109861]: pgmap v2986: 109 pgs: 109 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 749 op/s 2024-08-17T18:13:39.758 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:39 smithi136 ceph-mon[115078]: pgmap v2986: 109 pgs: 109 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 749 op/s 2024-08-17T18:13:39.785 INFO:teuthology.orchestra.run.smithi136.stdout:moved 7 / 327 (2.14067%) 2024-08-17T18:13:39.785 INFO:teuthology.orchestra.run.smithi136.stdout:avg 54.5 2024-08-17T18:13:39.785 INFO:teuthology.orchestra.run.smithi136.stdout:stddev 2.81366 -> 2.06155 (expected baseline 6.73919) 2024-08-17T18:13:39.785 INFO:teuthology.orchestra.run.smithi136.stdout:min osd.2 with 49 -> 52 pgs (0.899083 -> 0.954128 * mean) 2024-08-17T18:13:39.785 INFO:teuthology.orchestra.run.smithi136.stdout:max osd.3 with 58 -> 57 pgs (1.06422 -> 1.04587 * mean) 2024-08-17T18:13:39.785 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:13:39.785 INFO:teuthology.orchestra.run.smithi136.stdout:oload 110 2024-08-17T18:13:39.785 INFO:teuthology.orchestra.run.smithi136.stdout:max_change 0.05 2024-08-17T18:13:39.785 INFO:teuthology.orchestra.run.smithi136.stdout:max_change_osds 4 2024-08-17T18:13:39.785 INFO:teuthology.orchestra.run.smithi136.stdout:average_utilization 0.0089 2024-08-17T18:13:39.785 INFO:teuthology.orchestra.run.smithi136.stdout:overload_utilization 0.0098 2024-08-17T18:13:39.786 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3 weight 1.0000 -> 0.9500 2024-08-17T18:13:39.786 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6 weight 1.0000 -> 0.9500 2024-08-17T18:13:39.786 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5 weight 1.0000 -> 0.9500 2024-08-17T18:13:39.786 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4 weight 1.0000 -> 0.9500 2024-08-17T18:13:39.786 INFO:teuthology.orchestra.run.smithi136.stderr:no change 2024-08-17T18:13:39.851 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:39.874 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:39.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:39.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:39.976 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:40.002 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:40.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:40.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:39 smithi165 ceph-mon[105517]: pgmap v2986: 109 pgs: 109 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 749 op/s 2024-08-17T18:13:40.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:40.130 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:40.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:40.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:40.188 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:13:39 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: 2024-08-17T18:13:39.783+0000 7f0cc2ea7640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-08-17T18:13:40.232 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:40.254 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:40.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:40.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:40.304 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:13:40.305 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:13:40.355 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:40.381 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:40.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:40.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:40.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:40.505 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:40.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:40.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:40.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:40.633 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:40.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:40.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:40.735 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:40.759 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:40.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:40.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:40.861 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:40.885 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:40.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:40.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:40.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:40 smithi136 ceph-mon[115078]: from='client.36342 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:13:40.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:40 smithi136 ceph-mon[109861]: from='client.36342 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:13:40.987 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:41.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:41.010 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:41.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:41.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:40 smithi165 ceph-mon[105517]: from='client.36342 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:13:41.072 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-17T18:13:41.111 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:41.137 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:41.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:41.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:41.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:41.265 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:41.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:41.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:41.367 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:41.389 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:41.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:41.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:41.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:41.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:41.513 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:41.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:41.614 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:41.639 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:41.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:41.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:41.741 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:41.763 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:41.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:41.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:41.815 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:41 smithi136 ceph-mon[109861]: pgmap v2987: 109 pgs: 109 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 605 op/s 2024-08-17T18:13:41.815 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:41 smithi136 ceph-mon[115078]: pgmap v2987: 109 pgs: 109 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 605 op/s 2024-08-17T18:13:41.865 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:41.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:41.892 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:41.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:41.993 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:42.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:41 smithi165 ceph-mon[105517]: pgmap v2987: 109 pgs: 109 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 605 op/s 2024-08-17T18:13:42.015 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:42.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:42.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:42.117 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:42.138 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:42.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:42.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:42.240 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:42.266 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:42.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:42.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:42.364 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:13:42.367 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:42.393 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:42.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:42.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:42.495 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:42.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:42.517 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:42.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:42.576 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:13:42.576 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:13:42.600 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:13:42.600 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:13:42.617 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:13:42.617 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:13:42.618 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:13:42.618 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:42.627 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:13:42.644 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:13:42.644 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:13:42.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:42.645 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:42.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:42.660 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:13:42.661 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:13:42.677 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:13:42.677 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:13:42.692 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:13:42.693 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:13:42.710 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:13:42.710 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:13:42.746 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:42.771 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:42.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:42.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:42.849 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:42 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/47030251' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:13:42.850 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:42 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/47030251' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:13:42.872 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:42.894 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:42.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:42.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:42.996 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:43.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:42 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/47030251' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:13:43.022 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:43.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:43.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:43.123 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:43.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:43.146 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:43.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:43.247 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:43.274 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:43.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:43.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:43.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:43.398 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:43.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:43.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:43.500 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:43.521 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:43.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:43.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:43.623 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:43.649 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:43.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:43.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:43.751 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:43.773 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:43.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:43.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:43.829 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:43 smithi136 ceph-mon[109861]: pgmap v2988: 109 pgs: 109 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 327 op/s 2024-08-17T18:13:43.829 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:43 smithi136 ceph-mon[115078]: pgmap v2988: 109 pgs: 109 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 327 op/s 2024-08-17T18:13:43.875 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:43.901 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:43.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:43.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:44.003 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:44.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:43 smithi165 ceph-mon[105517]: pgmap v2988: 109 pgs: 109 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 327 op/s 2024-08-17T18:13:44.025 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:44.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:44.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:44.127 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:44.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:44.153 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:44.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:44.254 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:44.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:44.283 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:44.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:44.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:44.407 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:44.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:44.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:44.509 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:44.530 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:44.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:44.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:44.633 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:44.654 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:44.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:44.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:44.756 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:44.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:44.778 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:44.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:44.880 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:44.906 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:44.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:44.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:45.008 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:45.035 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:45.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:45.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:45.137 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:45.162 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:45.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:45.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:45.264 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:45.285 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:45.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:45.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:45.387 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:45.409 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:45.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:45.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:45.511 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:45.537 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:45.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:45.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:45.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:13:45.639 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:45.665 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:45.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:45.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:45.767 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:45.789 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:45.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:45.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:45.891 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:45.913 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:45.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:45.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:45 smithi136 ceph-mon[109861]: pgmap v2989: 109 pgs: 109 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 153 op/s 2024-08-17T18:13:45.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:45 smithi136 ceph-mon[115078]: pgmap v2989: 109 pgs: 109 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 153 op/s 2024-08-17T18:13:46.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:45 smithi165 ceph-mon[105517]: pgmap v2989: 109 pgs: 109 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 153 op/s 2024-08-17T18:13:46.016 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:46.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:46.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:46.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:46.145 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:46.169 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:46.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:46.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:46.271 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:46.294 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:46.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:46.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:46.396 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:46.421 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:46.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:46.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:46.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:46.548 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:46.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:46.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:46.650 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:46.671 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:46.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:46.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:46.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:46.799 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:46.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:46.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:46.901 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:46.922 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:46.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:46.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:47.024 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:47.046 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:47.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:47.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:47.148 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:47.169 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:47.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:47.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:47.271 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:47.296 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:47.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:47.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:47.334 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 6, 0, 5] out_osds: [7, 1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:13:47.334 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:13:47.334 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-08-17T18:13:47.335 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-17T18:13:47.335 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-17T18:13:47.335 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-08-17T18:13:47.398 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:47.420 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:47.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:47.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:47.451 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:13:47.451 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T18:13:47.451 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-17T18:13:47.452 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-17T18:13:47.452 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:13:47.522 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:47.544 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:47.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:47.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:47.556 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:13:47.644 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:13:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:13:47] "GET /metrics HTTP/1.1" 200 36686 "" "Prometheus/2.51.0" 2024-08-17T18:13:47.646 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:47.668 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:47.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:47.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:47.725 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'unique_pool_3' removed 2024-08-17T18:13:47.770 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:47.796 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:47.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:47.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:47.897 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:47.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:47.921 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:47.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:47.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:47 smithi136 ceph-mon[109861]: pgmap v2990: 109 pgs: 109 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 9 op/s 2024-08-17T18:13:47.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:47 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3127480690' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T18:13:47.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:47 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T18:13:47.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:47 smithi136 ceph-mon[115078]: pgmap v2990: 109 pgs: 109 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 9 op/s 2024-08-17T18:13:47.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:47 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3127480690' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T18:13:47.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:47 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T18:13:47.983 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:47 smithi165 ceph-mon[105517]: pgmap v2990: 109 pgs: 109 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 9 op/s 2024-08-17T18:13:47.983 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:47 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3127480690' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T18:13:47.983 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:47 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T18:13:48.022 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:48.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:48.045 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:48.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:48.146 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:48.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:48.168 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:48.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:48.269 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:48.290 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:48.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:48.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:48.392 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:48.414 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:48.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:48.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:48.516 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:48.537 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:48.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:48.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:48.639 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:48.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:48.663 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:48.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:48.740 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-17T18:13:48.764 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:48.804 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:48.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:48.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:48.906 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:48.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:48.936 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:48.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:48.957 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:13:48.982 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:48 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T18:13:48.982 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:48 smithi165 ceph-mon[105517]: osdmap e1602: 8 total, 8 up, 6 in 2024-08-17T18:13:49.038 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:49.043 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:48 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T18:13:49.043 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:48 smithi136 ceph-mon[109861]: osdmap e1602: 8 total, 8 up, 6 in 2024-08-17T18:13:49.043 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:48 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T18:13:49.043 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:48 smithi136 ceph-mon[115078]: osdmap e1602: 8 total, 8 up, 6 in 2024-08-17T18:13:49.060 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:49.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:49.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:49.162 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:49.183 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:49.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:49.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:49.287 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:49.308 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:49.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:49.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:49.380 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:13:49.380 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-17T18:13:49.380 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:13:49.409 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:49.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:49.432 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:49.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:49.474 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-17T18:13:49.475 INFO:tasks.radosbench:Beginning radosbench... 2024-08-17T18:13:49.475 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-08-17T18:13:49.475 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-08-17T18:13:49.533 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:49.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:49.555 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:49.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:49.656 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:49.677 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:49.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:49.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:49.778 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:49.799 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:49.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:49.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:49.901 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:49.922 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:49.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:49.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:49.991 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:49 smithi136 ceph-mon[115078]: pgmap v2992: 83 pgs: 83 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:13:49.991 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:49 smithi136 ceph-mon[109861]: pgmap v2992: 83 pgs: 83 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:13:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:49 smithi165 ceph-mon[105517]: pgmap v2992: 83 pgs: 83 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:13:50.024 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:50.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:50.047 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:50.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:50.112 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:13:50.148 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:50.169 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:50.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:50.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:50.271 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:50.293 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:50.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:50.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:50.395 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:50.416 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:50.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:50.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:50.518 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:50.539 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:50.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:50.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:50.641 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:50.662 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:50.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:50.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:50.738 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:13:50.764 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:50.786 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:50.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:50.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:50.888 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:50.910 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:50.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:50.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:50 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3461841576' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:13:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:50 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:13:51.012 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:51.033 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:51.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:51.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:51.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:50 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3461841576' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:13:51.073 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:50 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:13:51.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:50 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3461841576' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:13:51.073 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:50 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:13:51.136 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:51.157 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:51.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:51.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:51.259 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:51.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:51.281 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:51.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:51.382 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:51.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:51.404 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:51.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:51.505 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:51.526 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:51.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:51.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:51.628 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:51.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:51.651 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:51.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:51.752 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:51.774 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:51.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:51.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:51.876 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:51.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:51.898 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:51.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:51.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:51 smithi136 ceph-mon[115078]: pgmap v2993: 83 pgs: 83 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:13:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:51 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:13:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:51 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:13:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:51 smithi136 ceph-mon[115078]: osdmap e1603: 8 total, 8 up, 6 in 2024-08-17T18:13:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:13:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:13:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:51 smithi136 ceph-mon[109861]: pgmap v2993: 83 pgs: 83 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:13:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:51 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:13:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:51 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:13:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:51 smithi136 ceph-mon[109861]: osdmap e1603: 8 total, 8 up, 6 in 2024-08-17T18:13:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:13:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:13:51.999 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:51 smithi165 ceph-mon[105517]: pgmap v2993: 83 pgs: 83 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:13:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:51 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:13:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:51 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:13:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:51 smithi165 ceph-mon[105517]: osdmap e1603: 8 total, 8 up, 6 in 2024-08-17T18:13:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:13:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:13:52.020 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:52.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:52.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:52.122 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:52.147 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:52.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:52.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:52.248 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:52.270 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:52.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:52.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:52.372 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:52.393 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:52.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:52.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:52.495 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:52.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:52.521 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:52.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:52.622 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:52.646 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:52.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:52.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:52.748 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:52.772 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:52.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:52.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:52.874 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:52.897 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:52.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:52.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:52.999 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:53.023 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:53.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:53.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:53.111 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:13:53.125 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:53.149 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:53.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:53.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:53.251 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:53.272 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:53.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:53.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:53.374 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:53.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:53.401 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:53.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:53.502 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:53.528 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:53.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:53.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:53.630 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:53.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:53.652 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:53.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:53.740 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'unique_pool_4' created 2024-08-17T18:13:53.754 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:53.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:53.777 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:53.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:53.878 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:53.905 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:53.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:53.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:53.985 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:53 smithi136 ceph-mon[109861]: pgmap v2995: 83 pgs: 83 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:13:53.985 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:53 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1005258514' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-08-17T18:13:53.985 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:53 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-08-17T18:13:53.985 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:53 smithi136 ceph-mon[115078]: pgmap v2995: 83 pgs: 83 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:13:53.985 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:53 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1005258514' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-08-17T18:13:53.985 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:53 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-08-17T18:13:54.007 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:54.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:53 smithi165 ceph-mon[105517]: pgmap v2995: 83 pgs: 83 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:13:54.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:53 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1005258514' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-08-17T18:13:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:53 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-08-17T18:13:54.032 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:54.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:54.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:54.134 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:54.156 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:54.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:54.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:54.258 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:54.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:54.280 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:54.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:54.381 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:54.404 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:54.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:54.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:54.505 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:54.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:54.528 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:54.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:54.629 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:54.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:54.651 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:54.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:54.722 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:13:54.753 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:54.787 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:54.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:54.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:54.889 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:54.911 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:54 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-08-17T18:13:54.912 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:54 smithi136 ceph-mon[109861]: osdmap e1604: 8 total, 8 up, 6 in 2024-08-17T18:13:54.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:54 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-08-17T18:13:54.912 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:54 smithi136 ceph-mon[115078]: osdmap e1604: 8 total, 8 up, 6 in 2024-08-17T18:13:54.913 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:54.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:54.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:54.974 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 6, 0, 5] out_osds: [7, 1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:13:54.974 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:13:54.974 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [2, 3, 4, 6, 0, 5] 2024-08-17T18:13:54.974 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd out 3 2024-08-17T18:13:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:54 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-08-17T18:13:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:54 smithi165 ceph-mon[105517]: osdmap e1604: 8 total, 8 up, 6 in 2024-08-17T18:13:55.015 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:55.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:55.041 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:55.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:55.142 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:13:55.163 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:55.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:55.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:13:55.210 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:13:55.211 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:13:55.222 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:13:55.222 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:13:55.241 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:13:55.241 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:13:55.262 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:13:55.263 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:13:55.265 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:55.279 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:13:55.280 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:13:55.292 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:55.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:55.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:55.296 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:13:55.297 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:13:55.312 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:13:55.313 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:13:55.329 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:13:55.329 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:13:55.393 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:55.419 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:55.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:55.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:55.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:55.545 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:55.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:55.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:55.646 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:55.651 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-08-17T18:13:55.671 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:55.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:55.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:55.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:55.796 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:55.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:55.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:55.898 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:55.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:55.921 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:55.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:55.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:55 smithi136 ceph-mon[109861]: pgmap v2997: 99 pgs: 16 unknown, 83 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:13:55.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:55 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T18:13:55.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:55 smithi136 ceph-mon[109861]: osdmap e1605: 8 total, 8 up, 6 in 2024-08-17T18:13:55.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:55 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4147716396' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:13:55.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:55 smithi136 ceph-mon[115078]: pgmap v2997: 99 pgs: 16 unknown, 83 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:13:55.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:55 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T18:13:55.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:55 smithi136 ceph-mon[115078]: osdmap e1605: 8 total, 8 up, 6 in 2024-08-17T18:13:55.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:55 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4147716396' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:13:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:55 smithi165 ceph-mon[105517]: pgmap v2997: 99 pgs: 16 unknown, 83 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:13:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:55 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T18:13:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:55 smithi165 ceph-mon[105517]: osdmap e1605: 8 total, 8 up, 6 in 2024-08-17T18:13:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:55 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4147716396' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:13:56.022 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:56.046 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:56.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:56.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:56.148 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:56.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:56.171 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:56.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:56.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:56.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:56.297 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:56.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:56.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:56.421 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:56.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:56.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:56.523 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:56.546 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:56.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:56.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:56.648 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:56.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:56.672 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:56.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:56.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:56.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:56.797 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:56.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:56.898 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:56.921 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:56.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:56.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:56.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:56 smithi136 ceph-mon[109861]: osdmap e1606: 8 total, 8 up, 6 in 2024-08-17T18:13:56.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:56 smithi136 ceph-mon[115078]: osdmap e1606: 8 total, 8 up, 6 in 2024-08-17T18:13:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:56 smithi165 ceph-mon[105517]: osdmap e1606: 8 total, 8 up, 6 in 2024-08-17T18:13:57.021 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:13:57.023 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:57.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:57.047 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:57.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:57.148 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:57.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:57.171 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:57.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:57.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:57.296 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:57.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:57.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:57.297 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:13:57.398 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:57.420 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:57.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:57.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:57.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:57.546 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:57.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:57.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:57.638 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:13:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:13:57] "GET /metrics HTTP/1.1" 200 34265 "" "Prometheus/2.51.0" 2024-08-17T18:13:57.648 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:57.672 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:57.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:57.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:57.763 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:13:57.763 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:13:57.774 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:57.795 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:57.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:57.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:57.897 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:57.919 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:57.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:57.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:57.942 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:57 smithi136 ceph-mon[109861]: pgmap v3000: 99 pgs: 4 creating+peering, 9 unknown, 86 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:13:57.942 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:57 smithi136 ceph-mon[115078]: pgmap v3000: 99 pgs: 4 creating+peering, 9 unknown, 86 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:13:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:57 smithi165 ceph-mon[105517]: pgmap v3000: 99 pgs: 4 creating+peering, 9 unknown, 86 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:13:58.020 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:58.042 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:58.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:58.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:58.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:58.170 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:58.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:58.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:58.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:58.298 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:58.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:58.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:58.400 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:58.421 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:58.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:58.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:58.523 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:13:58.544 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:58.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:58.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:13:58.646 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:58.668 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:58.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:58.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:58.720 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-17T18:13:58.729 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:13:58.769 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:58.790 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:58.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:58.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:58.892 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:13:58.918 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:58.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:58.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:13:58.979 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:13:59.020 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:59.046 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:59.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:59.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:59.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:13:59.173 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:59.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:59.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:13:59.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:59.301 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:59.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:59.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:13:59.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:13:59.428 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:59.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:59.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:13:59.530 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:59.552 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:59.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:59.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:59.654 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:13:59.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:59.676 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:59.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:13:59.777 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:13:59.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:59.799 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:13:59.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:13:59.854 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:13:59 smithi136 ceph-mon[109861]: pgmap v3001: 99 pgs: 4 creating+peering, 95 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:13:59.854 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:13:59 smithi136 ceph-mon[115078]: pgmap v3001: 99 pgs: 4 creating+peering, 95 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:13:59.900 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:13:59.927 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:13:59.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:13:59.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:13:59 smithi165 ceph-mon[105517]: pgmap v3001: 99 pgs: 4 creating+peering, 95 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:14:00.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:00.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:00.055 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:00.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:00.156 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:00.182 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:00.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:00.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:00.284 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:00.309 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:00.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:00.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:00.411 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:00.434 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:00.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:00.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:00.536 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:00.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:00.558 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:00.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:00.659 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:00.682 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:00.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:00.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:00.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:00.809 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:00.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:00.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:00.911 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:00.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:00.934 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:00.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:00.995 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:14:00.995 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:14:01.036 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:01.061 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:01.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:01.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:01.163 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:01.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:01.185 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:01.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:01.286 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:01.310 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:01.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:01.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:01.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:01.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:01.439 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:01.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:01.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:01.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:01.567 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:01.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:01.668 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:01.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:01.691 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:01.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:01.793 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:01.819 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:01.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:01.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:01.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:01.947 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:01.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:01.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:02.049 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:02.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:02.074 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:02.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:02.150 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:01 smithi136 ceph-mon[109861]: pgmap v3002: 99 pgs: 99 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:14:02.150 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:01 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/8163561' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:14:02.151 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:01 smithi136 ceph-mon[115078]: pgmap v3002: 99 pgs: 99 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:14:02.151 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:01 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/8163561' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:14:02.176 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:02.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:02.199 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:02.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:02.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:01 smithi165 ceph-mon[105517]: pgmap v3002: 99 pgs: 99 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:14:02.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:01 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/8163561' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:14:02.300 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:02.326 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:02.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:02.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:02.429 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:02.450 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:02.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:02.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:02.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:02.574 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:02.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:02.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:02.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:02.702 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:02.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:02.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:02.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:02.831 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:02.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:02.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:02.933 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:02.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:02 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:14:02.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:02 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/8163561' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:14:02.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:02 smithi136 ceph-mon[109861]: osdmap e1607: 8 total, 8 up, 6 in 2024-08-17T18:14:02.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:02 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:14:02.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:02 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/8163561' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:14:02.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:02 smithi136 ceph-mon[115078]: osdmap e1607: 8 total, 8 up, 6 in 2024-08-17T18:14:02.955 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:02.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:02.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:03.057 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:03.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:03.080 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:03.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:03.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:03.206 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:03.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:03.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:03.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:02 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:14:03.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:02 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/8163561' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:14:03.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:02 smithi165 ceph-mon[105517]: osdmap e1607: 8 total, 8 up, 6 in 2024-08-17T18:14:03.308 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:03.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:03.336 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:03.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:03.437 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:03.460 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:03.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:03.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:03.562 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:03.588 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:03.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:03.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:03.690 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:03.714 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:03.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:03.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:03.816 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:03.838 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:03.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:03.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:03.939 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:03.960 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:03.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:03.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:04.062 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:04.085 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:04.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:04.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:04.096 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:03 smithi136 ceph-mon[115078]: pgmap v3004: 99 pgs: 99 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:14:04.096 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:03 smithi136 ceph-mon[109861]: pgmap v3004: 99 pgs: 99 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:14:04.096 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:03 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a[109838]: 2024-08-17T18:14:03.862+0000 7f5b8e77d640 -1 mon.a@0(leader).osd e1607 definitely_dead 0 2024-08-17T18:14:04.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:04.208 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:04.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:04.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:03 smithi165 ceph-mon[105517]: pgmap v3004: 99 pgs: 99 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:14:04.310 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:04.332 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:04.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:04.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:04.434 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:04.456 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:04.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:04.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:04.558 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:04.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:04.581 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:04.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:04.682 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:04.703 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:04.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:04.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:04.782 INFO:teuthology.orchestra.run.smithi136.stderr:marked out osd.3. 2024-08-17T18:14:04.783 INFO:teuthology.orchestra.run.smithi136.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-08-17T18:14:04.805 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:04.833 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:04.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:04.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:04.934 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:04.956 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:04.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:04.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:04.966 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:14:05.058 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:05.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:05.080 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:05.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:05.137 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:04 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:14:05.137 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:04 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3273364757' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-08-17T18:14:05.137 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:04 smithi136 ceph-mon[109861]: Client client.admin marked osd.3 out, while it was still marked up 2024-08-17T18:14:05.138 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:04 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2575032381' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T18:14:05.138 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:04 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:14:05.138 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:04 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3273364757' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-08-17T18:14:05.138 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:04 smithi136 ceph-mon[115078]: Client client.admin marked osd.3 out, while it was still marked up 2024-08-17T18:14:05.138 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:04 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2575032381' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T18:14:05.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:05.202 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:05.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:05.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:04 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:14:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:04 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3273364757' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-08-17T18:14:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:04 smithi165 ceph-mon[105517]: Client client.admin marked osd.3 out, while it was still marked up 2024-08-17T18:14:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:04 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2575032381' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T18:14:05.304 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:05.328 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:05.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:05.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:05.430 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:05.454 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:05.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:05.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:05.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:05.578 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:05.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:05.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:05.680 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:05.702 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:05.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:05.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:05.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:05.833 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:05.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:05.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:05.935 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:05.958 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:05.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:05.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:06.060 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:06.082 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:06.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:06.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:06.169 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:05 smithi165 ceph-mon[105517]: pgmap v3005: 99 pgs: 99 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:14:06.169 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:05 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3273364757' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-08-17T18:14:06.169 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:05 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2575032381' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T18:14:06.169 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:05 smithi165 ceph-mon[105517]: osdmap e1608: 8 total, 8 up, 5 in 2024-08-17T18:14:06.184 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:06.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:05 smithi136 ceph-mon[115078]: pgmap v3005: 99 pgs: 99 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:14:06.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:05 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3273364757' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-08-17T18:14:06.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:05 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2575032381' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T18:14:06.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:05 smithi136 ceph-mon[115078]: osdmap e1608: 8 total, 8 up, 5 in 2024-08-17T18:14:06.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:05 smithi136 ceph-mon[109861]: pgmap v3005: 99 pgs: 99 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-08-17T18:14:06.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:05 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3273364757' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-08-17T18:14:06.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:05 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2575032381' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T18:14:06.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:05 smithi136 ceph-mon[109861]: osdmap e1608: 8 total, 8 up, 5 in 2024-08-17T18:14:06.205 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:06.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:06.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:06.307 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:06.328 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:06.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:06.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:06.430 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:06.453 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:06.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:06.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:06.555 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:06.577 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:06.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:06.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:06.679 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:06.703 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:06.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:06.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:06.806 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:06.829 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:06.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:06.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:06.931 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:06.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:06 smithi136 ceph-mon[115078]: osdmap e1609: 8 total, 8 up, 5 in 2024-08-17T18:14:06.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:14:06.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:14:06.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:06 smithi136 ceph-mon[115078]: osdmap e1610: 8 total, 8 up, 5 in 2024-08-17T18:14:06.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:06 smithi136 ceph-mon[109861]: osdmap e1609: 8 total, 8 up, 5 in 2024-08-17T18:14:06.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:14:06.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:14:06.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:06 smithi136 ceph-mon[109861]: osdmap e1610: 8 total, 8 up, 5 in 2024-08-17T18:14:06.952 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:06.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:06.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:07.054 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:07.076 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:07.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:07.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:07.163 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:06 smithi165 ceph-mon[105517]: osdmap e1609: 8 total, 8 up, 5 in 2024-08-17T18:14:07.163 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:14:07.163 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:14:07.163 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:06 smithi165 ceph-mon[105517]: osdmap e1610: 8 total, 8 up, 5 in 2024-08-17T18:14:07.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:07.200 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:07.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:07.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:07.302 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:07.323 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:07.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:07.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:07.425 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:07.447 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:07.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:07.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:07.534 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:14:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:14:07] "GET /metrics HTTP/1.1" 200 36605 "" "Prometheus/2.51.0" 2024-08-17T18:14:07.549 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:07.572 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:07.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:07.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:07.674 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:07.696 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:07.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:07.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:07.721 DEBUG:teuthology.orchestra.run.smithi136:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-08-17T18:14:07.723 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-17T18:14:07.724 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-17T18:14:07.774 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:hints = 1 2024-08-17T18:14:07.774 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-17T18:14:07.774 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Object prefix: benchmark_data_smithi136_389235 2024-08-17T18:14:07.774 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T18:14:07.774 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 0 0 0 0 0 0 - 0 2024-08-17T18:14:07.799 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:07.835 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:07.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:07.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:07.937 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:07.946 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:14:07.971 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:07.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:07.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:08.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:07 smithi165 ceph-mon[105517]: pgmap v3008: 99 pgs: 99 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail 2024-08-17T18:14:08.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:07 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T18:14:08.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:07 smithi165 ceph-mon[105517]: osdmap e1611: 8 total, 8 up, 5 in 2024-08-17T18:14:08.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:08.104 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:08.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:08.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:08.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:07 smithi136 ceph-mon[109861]: pgmap v3008: 99 pgs: 99 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail 2024-08-17T18:14:08.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:07 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T18:14:08.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:07 smithi136 ceph-mon[109861]: osdmap e1611: 8 total, 8 up, 5 in 2024-08-17T18:14:08.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:07 smithi136 ceph-mon[115078]: pgmap v3008: 99 pgs: 99 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail 2024-08-17T18:14:08.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:07 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T18:14:08.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:07 smithi136 ceph-mon[115078]: osdmap e1611: 8 total, 8 up, 5 in 2024-08-17T18:14:08.206 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:08.240 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:08.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:08.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:08.342 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:08.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:08.395 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:08.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:08.496 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:08.532 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:08.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:08.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:08.634 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:08.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:08.670 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:08.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:08.771 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:08.775 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 1 16 3816 3800 237.487 237.5 0.00821165 0.00419989 2024-08-17T18:14:08.807 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:08.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:08.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:08.840 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:14:08.841 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:14:08.908 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:08.945 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:08.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:08.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:09.047 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:09.081 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:09.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:09.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:09.183 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:09.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:09.217 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:09.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:09.318 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:09.352 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:09.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:09.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:09.454 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:09.490 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:09.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:09.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:09.591 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:09.625 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:09.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:09.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:09 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:14:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:09 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:14:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:09 smithi136 ceph-mon[109861]: pgmap v3011: 99 pgs: 3 active+undersized+remapped, 96 active+clean; 4.9 MiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.1 MiB/s wr, 17 op/s 2024-08-17T18:14:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:09 smithi136 ceph-mon[109861]: osdmap e1612: 8 total, 8 up, 5 in 2024-08-17T18:14:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:09 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:14:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:09 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:14:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:09 smithi136 ceph-mon[115078]: pgmap v3011: 99 pgs: 3 active+undersized+remapped, 96 active+clean; 4.9 MiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.1 MiB/s wr, 17 op/s 2024-08-17T18:14:09.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:09 smithi136 ceph-mon[115078]: osdmap e1612: 8 total, 8 up, 5 in 2024-08-17T18:14:09.727 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:09 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:14:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:09 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:14:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:09 smithi165 ceph-mon[105517]: pgmap v3011: 99 pgs: 3 active+undersized+remapped, 96 active+clean; 4.9 MiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.1 MiB/s wr, 17 op/s 2024-08-17T18:14:09.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:09 smithi165 ceph-mon[105517]: osdmap e1612: 8 total, 8 up, 5 in 2024-08-17T18:14:09.764 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:09.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:09.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:09.774 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 2 15 7643 7628 238.358 239.25 0.00333073 0.00418588 2024-08-17T18:14:09.791 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-17T18:14:09.866 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:09.921 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:09.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:09.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:10.026 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:10.069 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:10.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:10.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:10.092 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:14:10.092 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:14:10.122 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:14:10.123 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:14:10.159 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:14:10.159 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:14:10.171 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:10.193 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:14:10.193 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:14:10.211 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:10.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:10.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:10.227 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:14:10.227 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:14:10.259 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:14:10.259 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:14:10.293 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:14:10.293 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:14:10.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:10.325 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:14:10.325 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:14:10.349 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:10.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:10.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:10.451 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:10.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:10.491 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:10.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:10.591 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:10.629 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:10.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:10.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:10.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:10 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1489057115' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:14:10.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:10 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1489057115' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:14:10.731 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:10 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1489057115' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:14:10.771 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:10.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:10.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:10.774 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 3 16 11003 10987 228.881 209.938 0.00915046 0.00436139 2024-08-17T18:14:10.874 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:10.908 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:10.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:10.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:11.009 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:11.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:11.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:11.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:11.146 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:11.182 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:11.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:11.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:11.204 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:14:11.248 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:14:11.284 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:11.317 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:11.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:11.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:11.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:11.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:11 smithi136 ceph-mon[109861]: pgmap v3013: 99 pgs: 3 active+undersized+remapped, 96 active+clean; 106 MiB data, 3.9 GiB used, 443 GiB / 447 GiB avail; 23 MiB/s wr, 366 op/s; 56 KiB/s, 0 objects/s recovering 2024-08-17T18:14:11.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:11 smithi136 ceph-mon[115078]: pgmap v3013: 99 pgs: 3 active+undersized+remapped, 96 active+clean; 106 MiB data, 3.9 GiB used, 443 GiB / 447 GiB avail; 23 MiB/s wr, 366 op/s; 56 KiB/s, 0 objects/s recovering 2024-08-17T18:14:11.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:11.458 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:11.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:11.559 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:11.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:11.606 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:11.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:11.707 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:11.743 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:11.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:11.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:11.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:11 smithi165 ceph-mon[105517]: pgmap v3013: 99 pgs: 3 active+undersized+remapped, 96 active+clean; 106 MiB data, 3.9 GiB used, 443 GiB / 447 GiB avail; 23 MiB/s wr, 366 op/s; 56 KiB/s, 0 objects/s recovering 2024-08-17T18:14:11.774 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 4 16 13992 13976 218.36 186.812 0.00868127 0.0045728 2024-08-17T18:14:11.845 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:11.882 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:11.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:11.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:11.984 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:12.023 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:12.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:12.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:12.125 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:12.159 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:12.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:12.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:12.261 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:12.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:12.294 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:12.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:12.395 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:12.426 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:12.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:12.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:12.528 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:12.561 INFO:tasks.thrashosds.thrasher:in_osds: [2, 4, 6, 0, 5] out_osds: [7, 1, 3] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:14:12.561 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:14:12.562 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_4 2024-08-17T18:14:12.562 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T18:14:12.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:12.564 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:12.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:12.665 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:12.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:12.704 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:12.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:12.775 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 5 16 16949 16933 211.629 184.812 0.00494634 0.00471963 2024-08-17T18:14:12.804 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:12.836 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:12.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:12.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:12.939 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:12.978 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:12.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:12.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:13.080 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:13.116 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:13.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:13.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:13.200 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:14:13.218 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:13.256 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:13.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:13.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:13.357 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:13.396 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:13.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:13.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:13.498 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:13.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:13.534 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:13.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:13.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:13.667 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:13.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:13.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:13.705 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:13 smithi136 ceph-mon[109861]: pgmap v3014: 99 pgs: 2 active+undersized+remapped, 97 active+clean; 545 MiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 91 MiB/s wr, 1.45k op/s; 43 KiB/s, 0 objects/s recovering 2024-08-17T18:14:13.706 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:13 smithi136 ceph-mon[115078]: pgmap v3014: 99 pgs: 2 active+undersized+remapped, 97 active+clean; 545 MiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 91 MiB/s wr, 1.45k op/s; 43 KiB/s, 0 objects/s recovering 2024-08-17T18:14:13.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:13 smithi165 ceph-mon[105517]: pgmap v3014: 99 pgs: 2 active+undersized+remapped, 97 active+clean; 545 MiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 91 MiB/s wr, 1.45k op/s; 43 KiB/s, 0 objects/s recovering 2024-08-17T18:14:13.769 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:13.775 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 6 16 19803 19787 206.085 178.375 0.00699902 0.00484405 2024-08-17T18:14:13.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:13.813 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:13.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:13.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:13.950 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:13.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:13.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:14.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:14.087 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:14.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:14.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:14.140 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:14:14.188 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:14.207 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:14:14.213 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:14.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:14.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:14.315 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:14.356 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:14.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:14.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:14.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:14.493 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:14.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:14.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:14.595 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:14.626 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:14.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:14.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:14.728 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:14.762 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:14.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:14.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:14.775 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 7 16 22314 22298 199.062 156.938 0.00169494 0.00500695 2024-08-17T18:14:14.863 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:14.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:14.900 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:14.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:15.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:15.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:15.036 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:15.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:15.136 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:15.161 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:15.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:15.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:15.263 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:15.307 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:15.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:15.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:15.410 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:15.431 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:15.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:15.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:15.473 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:14:15.533 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:15.566 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:15.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:15.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:15.668 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:15.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:15.701 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:15.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:15 smithi165 ceph-mon[105517]: pgmap v3015: 99 pgs: 99 active+clean; 797 MiB data, 6.1 GiB used, 441 GiB / 447 GiB avail; 105 MiB/s wr, 1.67k op/s; 34 KiB/s, 0 objects/s recovering 2024-08-17T18:14:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:15 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1619301453' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:14:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:15 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:14:15.775 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 8 16 24411 24395 190.561 131.062 0.00408423 0.00524178 2024-08-17T18:14:15.802 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:15.839 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:15.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:15.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:15.941 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:15.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:15 smithi136 ceph-mon[109861]: pgmap v3015: 99 pgs: 99 active+clean; 797 MiB data, 6.1 GiB used, 441 GiB / 447 GiB avail; 105 MiB/s wr, 1.67k op/s; 34 KiB/s, 0 objects/s recovering 2024-08-17T18:14:15.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:15 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1619301453' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:14:15.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:15 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:14:15.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:15 smithi136 ceph-mon[115078]: pgmap v3015: 99 pgs: 99 active+clean; 797 MiB data, 6.1 GiB used, 441 GiB / 447 GiB avail; 105 MiB/s wr, 1.67k op/s; 34 KiB/s, 0 objects/s recovering 2024-08-17T18:14:15.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:15 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1619301453' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:14:15.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:15 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:14:15.976 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:15.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:15.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:16.078 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:16.113 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:16.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:16.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:16.215 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:16.250 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:16.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:16.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:16.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:16.384 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:16.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:16.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:16.486 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:16.519 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:16.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:16.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:16.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:16.653 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:16.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:16.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:16.755 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:16 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:14:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:16 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T18:14:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:16 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:14:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:16 smithi165 ceph-mon[105517]: osdmap e1613: 8 total, 8 up, 5 in 2024-08-17T18:14:16.775 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 9 16 26848 26832 186.31 152.312 0.00692529 0.0053623 2024-08-17T18:14:16.789 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:16.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:16.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:16.891 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:16.914 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:16.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:16.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:16.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:16 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:14:16.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:16 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T18:14:16.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:16 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:14:16.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:16 smithi136 ceph-mon[109861]: osdmap e1613: 8 total, 8 up, 5 in 2024-08-17T18:14:16.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:16 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:14:16.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:16 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T18:14:16.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:16 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:14:16.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:16 smithi136 ceph-mon[115078]: osdmap e1613: 8 total, 8 up, 5 in 2024-08-17T18:14:17.015 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:17.050 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:17.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:17.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:17.151 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:17.187 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:17.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:17.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:17.289 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:17.332 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:17.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:17.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:17.434 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:17.473 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:17.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:17.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:17.575 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:17.610 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:17.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:17.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:17.661 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:17 smithi136 ceph-mon[109861]: pgmap v3017: 99 pgs: 99 active+clean; 1.2 GiB data, 7.1 GiB used, 440 GiB / 447 GiB avail; 148 MiB/s wr, 2.37k op/s; 32 KiB/s, 0 objects/s recovering 2024-08-17T18:14:17.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:17 smithi136 ceph-mon[115078]: pgmap v3017: 99 pgs: 99 active+clean; 1.2 GiB data, 7.1 GiB used, 440 GiB / 447 GiB avail; 148 MiB/s wr, 2.37k op/s; 32 KiB/s, 0 objects/s recovering 2024-08-17T18:14:17.661 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:14:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:14:17] "GET /metrics HTTP/1.1" 200 36605 "" "Prometheus/2.51.0" 2024-08-17T18:14:17.712 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:17.747 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:17.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:17.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:17 smithi165 ceph-mon[105517]: pgmap v3017: 99 pgs: 99 active+clean; 1.2 GiB data, 7.1 GiB used, 440 GiB / 447 GiB avail; 148 MiB/s wr, 2.37k op/s; 32 KiB/s, 0 objects/s recovering 2024-08-17T18:14:17.775 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 10 16 29013 28997 181.21 135.312 0.00740737 0.00551355 2024-08-17T18:14:17.849 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:17.887 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:17.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:17.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:17.988 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:18.023 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:18.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:18.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:18.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:18.160 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:18.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:18.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:18.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:18.295 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:18.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:18.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:18.397 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:18.435 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:18.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:18.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:18.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:18.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:18.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:18.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:18.674 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:18.724 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:18.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:18.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:18.775 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 11 16 31390 31374 178.241 148.562 0.00359238 0.00560504 2024-08-17T18:14:18.826 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:18.866 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:18.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:18.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:18.968 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:18.995 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:18.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:18.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:19.097 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:19.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:19.136 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:19.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:19.237 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:19.271 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:19.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:19.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:19.373 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:19.406 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:19.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:19.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:19.508 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:19.547 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:19.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:19.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:19.649 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:19.680 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:19.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:19.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:19 smithi165 ceph-mon[105517]: pgmap v3018: 99 pgs: 99 active+clean; 1.6 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 168 MiB/s wr, 2.69k op/s; 27 KiB/s, 0 objects/s recovering 2024-08-17T18:14:19.769 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:19 smithi136 ceph-mon[109861]: pgmap v3018: 99 pgs: 99 active+clean; 1.6 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 168 MiB/s wr, 2.69k op/s; 27 KiB/s, 0 objects/s recovering 2024-08-17T18:14:19.770 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:19 smithi136 ceph-mon[115078]: pgmap v3018: 99 pgs: 99 active+clean; 1.6 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 168 MiB/s wr, 2.69k op/s; 27 KiB/s, 0 objects/s recovering 2024-08-17T18:14:19.775 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 12 16 33528 33512 174.521 133.625 0.00619003 0.00572428 2024-08-17T18:14:19.782 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:19.824 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:19.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:19.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:19.926 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:19.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:19.964 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:19.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:20.065 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:20.098 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:20.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:20.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:20.200 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:20.234 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:20.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:20.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:20.336 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:20.371 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:20.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:20.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:20.473 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:20.506 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:20.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:20.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:20.608 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:20.645 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:20.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:20.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:20.746 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:20.775 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 13 16 35670 35654 171.394 133.875 0.00980855 0.00582921 2024-08-17T18:14:20.783 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:20.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:20.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:20.845 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:14:20.884 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:20.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:20.930 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:20.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:21.030 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:21.061 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:21.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:21.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:21.164 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:21.199 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:21.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:21.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:21.301 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:21.334 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:21.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:21.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:21.437 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:21.472 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:21.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:21.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:21.574 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:21.595 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:21.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:21.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:21.697 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:21.732 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:21.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:21.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:21.738 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:21 smithi165 ceph-mon[105517]: pgmap v3019: 99 pgs: 99 active+clean; 1.8 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 169 MiB/s wr, 2.71k op/s 2024-08-17T18:14:21.738 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:14:21.738 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:14:21.775 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 14 16 37457 37441 167.129 111.688 0.0327973 0.00597387 2024-08-17T18:14:21.834 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:21.867 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:21.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:21.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:21.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:21 smithi136 ceph-mon[109861]: pgmap v3019: 99 pgs: 99 active+clean; 1.8 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 169 MiB/s wr, 2.71k op/s 2024-08-17T18:14:21.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:14:21.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:14:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:21 smithi136 ceph-mon[115078]: pgmap v3019: 99 pgs: 99 active+clean; 1.8 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 169 MiB/s wr, 2.71k op/s 2024-08-17T18:14:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:14:21.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:14:21.969 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:22.019 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:22.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:22.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:22.121 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:22.174 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:22.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:22.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:22.276 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:22.317 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:22.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:22.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:22.418 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:22.440 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:22.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:22.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:22.542 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:22.577 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:22.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:22.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:22.680 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:22.718 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:22.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:22.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:22.776 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 15 16 39512 39496 164.548 128.438 0.0107753 0.00607278 2024-08-17T18:14:22.820 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:22.855 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:22.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:22.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:22.958 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:22.979 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:22.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:22.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:23.081 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:23.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:23.115 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:23.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:23.216 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:23.237 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:23.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:23.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:23.339 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:23.371 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:23.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:23.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:23.473 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:23.506 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:23.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:23.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:23.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:23.644 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:23.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:23.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:23.745 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:23 smithi165 ceph-mon[105517]: pgmap v3020: 99 pgs: 99 active+clean; 2.1 GiB data, 9.9 GiB used, 437 GiB / 447 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-08-17T18:14:23.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:23.774 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:23.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:23.776 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 16 16 41235 41219 160.994 107.688 0.00627224 0.00620618 2024-08-17T18:14:23.875 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:23.909 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:23.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:23.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:23.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:23 smithi136 ceph-mon[109861]: pgmap v3020: 99 pgs: 99 active+clean; 2.1 GiB data, 9.9 GiB used, 437 GiB / 447 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-08-17T18:14:23.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:23 smithi136 ceph-mon[115078]: pgmap v3020: 99 pgs: 99 active+clean; 2.1 GiB data, 9.9 GiB used, 437 GiB / 447 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-08-17T18:14:24.012 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:24.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:24.050 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:24.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:24.151 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:24.180 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:24.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:24.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:24.282 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:24.314 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:24.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:24.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:24.416 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:24.448 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:24.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:24.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:24.550 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:24.571 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:24.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:24.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:24.673 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:24.694 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:24.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:24.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:24.776 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 17 16 43074 43058 158.284 114.938 0.00780725 0.00631277 2024-08-17T18:14:24.796 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:24.834 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:24.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:24.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:24.936 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:24.972 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:24.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:24.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:25.074 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:25.110 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:25.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:25.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:25.212 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:25.248 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:25.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:25.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:25.350 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:25.387 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:25.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:25.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:25.489 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:25.512 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:25.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:25.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:25.614 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:25.643 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:25.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:25.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:25.745 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:25.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:25 smithi165 ceph-mon[105517]: pgmap v3021: 99 pgs: 99 active+clean; 2.2 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 150 MiB/s wr, 2.40k op/s 2024-08-17T18:14:25.776 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 18 16 44910 44894 155.865 114.75 0.00631283 0.0064109 2024-08-17T18:14:25.779 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:25.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:25.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:25.880 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:25.917 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:25.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:25.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:25.923 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:25 smithi136 ceph-mon[109861]: pgmap v3021: 99 pgs: 99 active+clean; 2.2 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 150 MiB/s wr, 2.40k op/s 2024-08-17T18:14:25.923 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:25 smithi136 ceph-mon[115078]: pgmap v3021: 99 pgs: 99 active+clean; 2.2 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 150 MiB/s wr, 2.40k op/s 2024-08-17T18:14:26.019 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:26.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:26.061 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:26.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:26.162 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:26.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:26.201 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:26.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:26.303 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:26.339 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:26.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:26.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:26.441 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:26.478 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:26.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:26.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:26.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:26.590 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:14:26.590 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T18:14:26.590 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-17T18:14:26.590 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-17T18:14:26.590 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:14:26.610 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:26.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:26.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:26.712 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:26.745 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:26.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:26.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:26.776 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 19 16 46655 46639 153.402 109.062 0.013544 0.00650697 2024-08-17T18:14:26.848 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:26.882 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:26.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:26.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:26.984 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:27.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:27.019 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:27.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:27.121 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:27.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:27.158 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:27.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:27.259 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:27.291 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:27.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:27.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:27.393 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:27.427 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:27.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:27.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:27.529 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:27.564 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:27.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:27.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:27.666 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:27.692 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:27.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:27.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:27.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:27 smithi136 ceph-mon[109861]: pgmap v3022: 99 pgs: 99 active+clean; 2.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-08-17T18:14:27.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:27 smithi136 ceph-mon[115078]: pgmap v3022: 99 pgs: 99 active+clean; 2.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-08-17T18:14:27.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:14:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:14:27] "GET /metrics HTTP/1.1" 200 36666 "" "Prometheus/2.51.0" 2024-08-17T18:14:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:27 smithi165 ceph-mon[105517]: pgmap v3022: 99 pgs: 99 active+clean; 2.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-08-17T18:14:27.776 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:2024-08-17T18:14:27.776589+0000 min lat: 0.00125927 max lat: 0.115528 avg lat: 0.00659706 2024-08-17T18:14:27.776 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T18:14:27.776 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 20 16 48495 48479 151.481 115 0.00372801 0.00659706 2024-08-17T18:14:27.793 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:27.817 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:27.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:27.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:27.919 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:27.951 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:27.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:27.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:28.053 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:28.085 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:28.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:28.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:28.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:28.222 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:28.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:28.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:28.324 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:28.368 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:28.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:28.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:28.469 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:28.501 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:28.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:28.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:28.603 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:28.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:28.644 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:28.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:28.745 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:28.776 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 21 16 50138 50122 149.157 102.688 0.00630689 0.00669898 2024-08-17T18:14:28.783 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:28.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:28.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:28.885 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:28.919 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:28.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:28.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:29.021 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:29.063 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:29.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:29.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:29.092 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:14:29.092 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:14:29.092 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:14:29.092 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:14:29.092 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:14:29.093 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:14:29.093 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:14:29.093 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:14:29.093 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:14:29.093 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:14:29.093 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:14:29.093 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:14:29.094 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:14:29.094 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:14:29.094 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:14:29.094 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:14:29.165 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:29.201 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:29.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:29.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:29.302 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:29.339 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:29.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:29.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:29.441 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:29.487 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:29.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:29.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:29.589 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:29.621 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:29.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:29.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:29.723 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:29.757 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:29.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:29.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:29 smithi165 ceph-mon[105517]: pgmap v3023: 99 pgs: 99 active+clean; 2.8 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 141 MiB/s wr, 2.25k op/s 2024-08-17T18:14:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:29 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4188708249' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:14:29.776 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 22 16 51939 51923 147.493 112.562 0.00596448 0.00677505 2024-08-17T18:14:29.832 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:29 smithi136 ceph-mon[109861]: pgmap v3023: 99 pgs: 99 active+clean; 2.8 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 141 MiB/s wr, 2.25k op/s 2024-08-17T18:14:29.832 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:29 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4188708249' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:14:29.832 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:29 smithi136 ceph-mon[115078]: pgmap v3023: 99 pgs: 99 active+clean; 2.8 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 141 MiB/s wr, 2.25k op/s 2024-08-17T18:14:29.832 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:29 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4188708249' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:14:29.859 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:29.871 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-17T18:14:29.901 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:29.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:29.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:30.003 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:30.038 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:30.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:30.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:30.140 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:30.176 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:30.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:30.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:30.278 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:30.312 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:30.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:30.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:30.414 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:30.452 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:30.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:30.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:30.554 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:30.585 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:30.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:30.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:30.687 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:30.724 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:30.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:30.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:30.777 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 23 16 53726 53710 145.932 111.688 0.0125865 0.00684801 2024-08-17T18:14:30.826 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:30.859 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:30.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:30.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:30.961 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:30.996 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:30.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:30.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:31.098 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:31.136 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:31.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:31.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:31.238 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:31.274 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:31.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:31.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:31.376 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:31.411 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:31.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:31.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:31.513 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:31.547 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:31.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:31.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:31.649 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:31.686 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:31.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:31.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:31.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:31 smithi165 ceph-mon[105517]: pgmap v3024: 99 pgs: 99 active+clean; 2.9 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-08-17T18:14:31.777 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 24 16 55405 55389 144.223 104.938 0.00970159 0.00692877 2024-08-17T18:14:31.788 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:31.823 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:31.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:31.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:31.925 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:31.932 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:14:31.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:31 smithi136 ceph-mon[115078]: pgmap v3024: 99 pgs: 99 active+clean; 2.9 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-08-17T18:14:31.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:31 smithi136 ceph-mon[109861]: pgmap v3024: 99 pgs: 99 active+clean; 2.9 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-08-17T18:14:31.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:31.979 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:31.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:32.080 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:32.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:32.120 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:32.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:32.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:32.254 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:32.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:32.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:32.356 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:32.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:32.390 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:32.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:32.486 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:14:32.491 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:32.525 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:32.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:32.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:32.627 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:32.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:32.661 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:32.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:32.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:32.777 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 25 16 57001 56985 142.443 99.75 0.00808476 0.00701469 2024-08-17T18:14:32.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:32.808 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:32.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:32.879 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:14:32.881 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T18:14:32.909 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:32.945 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:32.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:32.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:33.046 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:33.080 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:33.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:33.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:33.182 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:33.212 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:33.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:33.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:33.314 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:33.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:33.351 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:33.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:33.453 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:33.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:33.484 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:33.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:33.585 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:33.620 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:33.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:33.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:33.714 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":3025,"stamp":"2024-08-17T18:14:32.401401+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3447527397,"num_objects":52622,"num_object_clones":0,"num_object_copies":157866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52660,"num_write_kb":3366177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":861109,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54294,"ondisk_log_size":54294,"up":297,"acting":297,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":297,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":14073172,"kb_used_data":9848364,"kb_used_omap":211,"kb_used_meta":4224556,"kb_avail":454652588,"statfs":{"total":479975178240,"available":465564250112,"internally_reserved":0,"allocated":10084724736,"data_stored":10063027711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":216705,"internal_metadata":4325945727},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1562378240,"num_objects":23840,"num_object_clones":0,"num_object_copies":71520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23840,"num_write_kb":1525760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003807"},"pg_stats":[{"pgid":"57.f","version":"1613'3590","reported_seq":3625,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:32.175755+0000","last_change":"2024-08-17T18:14:05.806463+0000","last_active":"2024-08-17T18:14:32.175755+0000","last_peered":"2024-08-17T18:14:32.175755+0000","last_clean":"2024-08-17T18:14:32.175755+0000","last_became_active":"2024-08-17T18:14:05.806138+0000","last_became_peered":"2024-08-17T18:14:05.806138+0000","last_unstale":"2024-08-17T18:14:32.175755+0000","last_undegraded":"2024-08-17T18:14:32.175755+0000","last_fullsized":"2024-08-17T18:14:32.175755+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":1604,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3590,"log_dups_size":0,"ondisk_log_size":3590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:53:37.123915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235274240,"num_objects":3590,"num_object_clones":0,"num_object_copies":10770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3590,"num_write_kb":229760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"57.e","version":"1613'3009","reported_seq":3027,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:27.412868+0000","last_change":"2024-08-17T18:13:54.755799+0000","last_active":"2024-08-17T18:14:27.412868+0000","last_peered":"2024-08-17T18:14:27.412868+0000","last_clean":"2024-08-17T18:14:27.412868+0000","last_became_active":"2024-08-17T18:13:54.755500+0000","last_became_peered":"2024-08-17T18:13:54.755500+0000","last_unstale":"2024-08-17T18:14:27.412868+0000","last_undegraded":"2024-08-17T18:14:27.412868+0000","last_fullsized":"2024-08-17T18:14:27.412868+0000","mapping_epoch":1604,"log_start":"0'0","ondisk_log_start":"0'0","created":1604,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3009,"log_dups_size":0,"ondisk_log_size":3009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:54:46.896946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197197824,"num_objects":3009,"num_object_clones":0,"num_object_copies":9027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3009,"num_write_kb":192576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"57.d","version":"1613'2965","reported_seq":2992,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:27.421612+0000","last_change":"2024-08-17T18:14:05.809436+0000","last_active":"2024-08-17T18:14:27.421612+0000","last_peered":"2024-08-17T18:14:27.421612+0000","last_clean":"2024-08-17T18:14:27.421612+0000","last_became_active":"2024-08-17T18:14:05.809141+0000","last_became_peered":"2024-08-17T18:14:05.809141+0000","last_unstale":"2024-08-17T18:14:27.421612+0000","last_undegraded":"2024-08-17T18:14:27.421612+0000","last_fullsized":"2024-08-17T18:14:27.421612+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":1604,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":2965,"log_dups_size":0,"ondisk_log_size":2965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:19:12.945456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194314240,"num_objects":2965,"num_object_clones":0,"num_object_copies":8895,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2965,"num_write_kb":189760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"57.c","version":"1613'2996","reported_seq":3014,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:27.425883+0000","last_change":"2024-08-17T18:13:54.756490+0000","last_active":"2024-08-17T18:14:27.425883+0000","last_peered":"2024-08-17T18:14:27.425883+0000","last_clean":"2024-08-17T18:14:27.425883+0000","last_became_active":"2024-08-17T18:13:54.756109+0000","last_became_peered":"2024-08-17T18:13:54.756109+0000","last_unstale":"2024-08-17T18:14:27.425883+0000","last_undegraded":"2024-08-17T18:14:27.425883+0000","last_fullsized":"2024-08-17T18:14:27.425883+0000","mapping_epoch":1604,"log_start":"0'0","ondisk_log_start":"0'0","created":1604,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":2996,"log_dups_size":0,"ondisk_log_size":2996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:09:29.691722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196345856,"num_objects":2996,"num_object_clones":0,"num_object_copies":8988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2996,"num_write_kb":191744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"57.b","version":"1613'3476","reported_seq":3505,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:32.195617+0000","last_change":"2024-08-17T18:13:54.755437+0000","last_active":"2024-08-17T18:14:32.195617+0000","last_peered":"2024-08-17T18:14:32.195617+0000","last_clean":"2024-08-17T18:14:32.195617+0000","last_became_active":"2024-08-17T18:13:54.755175+0000","last_became_peered":"2024-08-17T18:13:54.755175+0000","last_unstale":"2024-08-17T18:14:32.195617+0000","last_undegraded":"2024-08-17T18:14:32.195617+0000","last_fullsized":"2024-08-17T18:14:32.195617+0000","mapping_epoch":1604,"log_start":"0'0","ondisk_log_start":"0'0","created":1604,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3476,"log_dups_size":0,"ondisk_log_size":3476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:46:19.119919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227803136,"num_objects":3476,"num_object_clones":0,"num_object_copies":10428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3476,"num_write_kb":222464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"57.a","version":"1613'3319","reported_seq":3345,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:29.405924+0000","last_change":"2024-08-17T18:14:05.808353+0000","last_active":"2024-08-17T18:14:29.405924+0000","last_peered":"2024-08-17T18:14:29.405924+0000","last_clean":"2024-08-17T18:14:29.405924+0000","last_became_active":"2024-08-17T18:14:05.808036+0000","last_became_peered":"2024-08-17T18:14:05.808036+0000","last_unstale":"2024-08-17T18:14:29.405924+0000","last_undegraded":"2024-08-17T18:14:29.405924+0000","last_fullsized":"2024-08-17T18:14:29.405924+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":1604,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3319,"log_dups_size":0,"ondisk_log_size":3319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:27:03.149189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217513984,"num_objects":3319,"num_object_clones":0,"num_object_copies":9957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3319,"num_write_kb":212416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.9","version":"1613'3052","reported_seq":3068,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:28.038790+0000","last_change":"2024-08-17T18:14:05.805948+0000","last_active":"2024-08-17T18:14:28.038790+0000","last_peered":"2024-08-17T18:14:28.038790+0000","last_clean":"2024-08-17T18:14:28.038790+0000","last_became_active":"2024-08-17T18:14:05.805619+0000","last_became_peered":"2024-08-17T18:14:05.805619+0000","last_unstale":"2024-08-17T18:14:28.038790+0000","last_undegraded":"2024-08-17T18:14:28.038790+0000","last_fullsized":"2024-08-17T18:14:28.038790+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":1604,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3052,"log_dups_size":0,"ondisk_log_size":3052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:11:12.989197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200015872,"num_objects":3052,"num_object_clones":0,"num_object_copies":9156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3052,"num_write_kb":195328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"57.8","version":"1613'3268","reported_seq":3286,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:29.396760+0000","last_change":"2024-08-17T18:13:54.752953+0000","last_active":"2024-08-17T18:14:29.396760+0000","last_peered":"2024-08-17T18:14:29.396760+0000","last_clean":"2024-08-17T18:14:29.396760+0000","last_became_active":"2024-08-17T18:13:54.752629+0000","last_became_peered":"2024-08-17T18:13:54.752629+0000","last_unstale":"2024-08-17T18:14:29.396760+0000","last_undegraded":"2024-08-17T18:14:29.396760+0000","last_fullsized":"2024-08-17T18:14:29.396760+0000","mapping_epoch":1604,"log_start":"0'0","ondisk_log_start":"0'0","created":1604,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3268,"log_dups_size":0,"ondisk_log_size":3268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:37:36.662889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214171648,"num_objects":3268,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3268,"num_write_kb":209152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.7","version":"1613'3455","reported_seq":3484,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:32.183914+0000","last_change":"2024-08-17T18:13:54.757619+0000","last_active":"2024-08-17T18:14:32.183914+0000","last_peered":"2024-08-17T18:14:32.183914+0000","last_clean":"2024-08-17T18:14:32.183914+0000","last_became_active":"2024-08-17T18:13:54.757350+0000","last_became_peered":"2024-08-17T18:13:54.757350+0000","last_unstale":"2024-08-17T18:14:32.183914+0000","last_undegraded":"2024-08-17T18:14:32.183914+0000","last_fullsized":"2024-08-17T18:14:32.183914+0000","mapping_epoch":1604,"log_start":"0'0","ondisk_log_start":"0'0","created":1604,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3455,"log_dups_size":0,"ondisk_log_size":3455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:10:42.230930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226426880,"num_objects":3455,"num_object_clones":0,"num_object_copies":10365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3455,"num_write_kb":221120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"57.6","version":"1613'3169","reported_seq":3187,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:29.402566+0000","last_change":"2024-08-17T18:13:54.753836+0000","last_active":"2024-08-17T18:14:29.402566+0000","last_peered":"2024-08-17T18:14:29.402566+0000","last_clean":"2024-08-17T18:14:29.402566+0000","last_became_active":"2024-08-17T18:13:54.753491+0000","last_became_peered":"2024-08-17T18:13:54.753491+0000","last_unstale":"2024-08-17T18:14:29.402566+0000","last_undegraded":"2024-08-17T18:14:29.402566+0000","last_fullsized":"2024-08-17T18:14:29.402566+0000","mapping_epoch":1604,"log_start":"0'0","ondisk_log_start":"0'0","created":1604,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3169,"log_dups_size":0,"ondisk_log_size":3169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:25:39.947345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207683584,"num_objects":3169,"num_object_clones":0,"num_object_copies":9507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3169,"num_write_kb":202816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.5","version":"1613'3229","reported_seq":3254,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:29.409968+0000","last_change":"2024-08-17T18:14:05.816918+0000","last_active":"2024-08-17T18:14:29.409968+0000","last_peered":"2024-08-17T18:14:29.409968+0000","last_clean":"2024-08-17T18:14:29.409968+0000","last_became_active":"2024-08-17T18:14:05.816435+0000","last_became_peered":"2024-08-17T18:14:05.816435+0000","last_unstale":"2024-08-17T18:14:29.409968+0000","last_undegraded":"2024-08-17T18:14:29.409968+0000","last_fullsized":"2024-08-17T18:14:29.409968+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":1604,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3229,"log_dups_size":0,"ondisk_log_size":3229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:00:37.731402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211615744,"num_objects":3229,"num_object_clones":0,"num_object_copies":9687,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3229,"num_write_kb":206656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.4","version":"1613'3339","reported_seq":3372,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:31.245927+0000","last_change":"2024-08-17T18:14:05.816218+0000","last_active":"2024-08-17T18:14:31.245927+0000","last_peered":"2024-08-17T18:14:31.245927+0000","last_clean":"2024-08-17T18:14:31.245927+0000","last_became_active":"2024-08-17T18:14:05.815891+0000","last_became_peered":"2024-08-17T18:14:05.815891+0000","last_unstale":"2024-08-17T18:14:31.245927+0000","last_undegraded":"2024-08-17T18:14:31.245927+0000","last_fullsized":"2024-08-17T18:14:31.245927+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":1604,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3339,"log_dups_size":0,"ondisk_log_size":3339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:41:25.876290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218824704,"num_objects":3339,"num_object_clones":0,"num_object_copies":10017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3339,"num_write_kb":213696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"57.3","version":"1613'3407","reported_seq":3428,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:32.216785+0000","last_change":"2024-08-17T18:14:05.805740+0000","last_active":"2024-08-17T18:14:32.216785+0000","last_peered":"2024-08-17T18:14:32.216785+0000","last_clean":"2024-08-17T18:14:32.216785+0000","last_became_active":"2024-08-17T18:14:05.805277+0000","last_became_peered":"2024-08-17T18:14:05.805277+0000","last_unstale":"2024-08-17T18:14:32.216785+0000","last_undegraded":"2024-08-17T18:14:32.216785+0000","last_fullsized":"2024-08-17T18:14:32.216785+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":1604,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3407,"log_dups_size":0,"ondisk_log_size":3407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:43:34.590878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223281152,"num_objects":3407,"num_object_clones":0,"num_object_copies":10221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3407,"num_write_kb":218048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"57.2","version":"1613'3356","reported_seq":3391,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:31.228987+0000","last_change":"2024-08-17T18:14:05.803012+0000","last_active":"2024-08-17T18:14:31.228987+0000","last_peered":"2024-08-17T18:14:31.228987+0000","last_clean":"2024-08-17T18:14:31.228987+0000","last_became_active":"2024-08-17T18:14:05.802670+0000","last_became_peered":"2024-08-17T18:14:05.802670+0000","last_unstale":"2024-08-17T18:14:31.228987+0000","last_undegraded":"2024-08-17T18:14:31.228987+0000","last_fullsized":"2024-08-17T18:14:31.228987+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":1604,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3356,"log_dups_size":0,"ondisk_log_size":3356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:20:05.233325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219938816,"num_objects":3356,"num_object_clones":0,"num_object_copies":10068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3356,"num_write_kb":214784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"57.1","version":"1613'3413","reported_seq":3441,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:31.237607+0000","last_change":"2024-08-17T18:13:54.754488+0000","last_active":"2024-08-17T18:14:31.237607+0000","last_peered":"2024-08-17T18:14:31.237607+0000","last_clean":"2024-08-17T18:14:31.237607+0000","last_became_active":"2024-08-17T18:13:54.754192+0000","last_became_peered":"2024-08-17T18:13:54.754192+0000","last_unstale":"2024-08-17T18:14:31.237607+0000","last_undegraded":"2024-08-17T18:14:31.237607+0000","last_fullsized":"2024-08-17T18:14:31.237607+0000","mapping_epoch":1604,"log_start":"0'0","ondisk_log_start":"0'0","created":1604,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3413,"log_dups_size":0,"ondisk_log_size":3413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:56:15.191787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223674368,"num_objects":3413,"num_object_clones":0,"num_object_copies":10239,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3413,"num_write_kb":218432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"57.0","version":"1613'3553","reported_seq":3588,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:32.209415+0000","last_change":"2024-08-17T18:14:05.810838+0000","last_active":"2024-08-17T18:14:32.209415+0000","last_peered":"2024-08-17T18:14:32.209415+0000","last_clean":"2024-08-17T18:14:32.209415+0000","last_became_active":"2024-08-17T18:14:05.810313+0000","last_became_peered":"2024-08-17T18:14:05.810313+0000","last_unstale":"2024-08-17T18:14:32.209415+0000","last_undegraded":"2024-08-17T18:14:32.209415+0000","last_fullsized":"2024-08-17T18:14:32.209415+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":1604,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3553,"log_dups_size":0,"ondisk_log_size":3553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:38:16.563940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232849408,"num_objects":3553,"num_object_clones":0,"num_object_copies":10659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3553,"num_write_kb":227392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":371,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.475311+0000","last_change":"2024-08-17T18:07:48.546202+0000","last_active":"2024-08-17T18:14:15.475311+0000","last_peered":"2024-08-17T18:14:15.475311+0000","last_clean":"2024-08-17T18:14:15.475311+0000","last_became_active":"2024-08-17T18:07:48.531246+0000","last_became_peered":"2024-08-17T18:07:48.531246+0000","last_unstale":"2024-08-17T18:14:15.475311+0000","last_undegraded":"2024-08-17T18:14:15.475311+0000","last_fullsized":"2024-08-17T18:14:15.475311+0000","mapping_epoch":1501,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1502,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:30.944362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1636,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.804798+0000","last_change":"2024-08-17T18:14:05.808703+0000","last_active":"2024-08-17T18:14:08.804798+0000","last_peered":"2024-08-17T18:14:08.804798+0000","last_clean":"2024-08-17T18:14:08.804798+0000","last_became_active":"2024-08-17T18:14:05.808320+0000","last_became_peered":"2024-08-17T18:14:05.808320+0000","last_unstale":"2024-08-17T18:14:08.804798+0000","last_undegraded":"2024-08-17T18:14:08.804798+0000","last_fullsized":"2024-08-17T18:14:08.804798+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:04:14.260536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":2357,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.798489+0000","last_change":"2024-08-17T18:14:05.803706+0000","last_active":"2024-08-17T18:14:08.798489+0000","last_peered":"2024-08-17T18:14:08.798489+0000","last_clean":"2024-08-17T18:14:08.798489+0000","last_became_active":"2024-08-17T18:14:05.803370+0000","last_became_peered":"2024-08-17T18:14:05.803370+0000","last_unstale":"2024-08-17T18:14:08.798489+0000","last_undegraded":"2024-08-17T18:14:08.798489+0000","last_fullsized":"2024-08-17T18:14:08.798489+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:09:17.148157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2266,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.476474+0000","last_change":"2024-08-17T18:14:05.812342+0000","last_active":"2024-08-17T18:14:15.476474+0000","last_peered":"2024-08-17T18:14:15.476474+0000","last_clean":"2024-08-17T18:14:15.476474+0000","last_became_active":"2024-08-17T18:14:05.810560+0000","last_became_peered":"2024-08-17T18:14:05.810560+0000","last_unstale":"2024-08-17T18:14:15.476474+0000","last_undegraded":"2024-08-17T18:14:15.476474+0000","last_fullsized":"2024-08-17T18:14:15.476474+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:11:26.357218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":166,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.803978+0000","last_change":"2024-08-17T18:04:43.777705+0000","last_active":"2024-08-17T18:14:08.803978+0000","last_peered":"2024-08-17T18:14:08.803978+0000","last_clean":"2024-08-17T18:14:08.803978+0000","last_became_active":"2024-08-17T18:04:43.771283+0000","last_became_peered":"2024-08-17T18:04:43.771283+0000","last_unstale":"2024-08-17T18:14:08.803978+0000","last_undegraded":"2024-08-17T18:14:08.803978+0000","last_fullsized":"2024-08-17T18:14:08.803978+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:13.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1510,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.474652+0000","last_change":"2024-08-17T18:14:05.807579+0000","last_active":"2024-08-17T18:14:15.474652+0000","last_peered":"2024-08-17T18:14:15.474652+0000","last_clean":"2024-08-17T18:14:15.474652+0000","last_became_active":"2024-08-17T18:14:05.807243+0000","last_became_peered":"2024-08-17T18:14:05.807243+0000","last_unstale":"2024-08-17T18:14:15.474652+0000","last_undegraded":"2024-08-17T18:14:15.474652+0000","last_fullsized":"2024-08-17T18:14:15.474652+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:03.398668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":179,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.803364+0000","last_change":"2024-08-17T18:12:04.655551+0000","last_active":"2024-08-17T18:14:08.803364+0000","last_peered":"2024-08-17T18:14:08.803364+0000","last_clean":"2024-08-17T18:14:08.803364+0000","last_became_active":"2024-08-17T18:12:04.646075+0000","last_became_peered":"2024-08-17T18:12:04.646075+0000","last_unstale":"2024-08-17T18:14:08.803364+0000","last_undegraded":"2024-08-17T18:14:08.803364+0000","last_fullsized":"2024-08-17T18:14:08.803364+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:19:19.300135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1623,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.805653+0000","last_change":"2024-08-17T18:14:05.808640+0000","last_active":"2024-08-17T18:14:08.805653+0000","last_peered":"2024-08-17T18:14:08.805653+0000","last_clean":"2024-08-17T18:14:08.805653+0000","last_became_active":"2024-08-17T18:14:05.808331+0000","last_became_peered":"2024-08-17T18:14:05.808331+0000","last_unstale":"2024-08-17T18:14:08.805653+0000","last_undegraded":"2024-08-17T18:14:08.805653+0000","last_fullsized":"2024-08-17T18:14:08.805653+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:03:20.051524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":340,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.473623+0000","last_change":"2024-08-17T18:14:05.809017+0000","last_active":"2024-08-17T18:14:15.473623+0000","last_peered":"2024-08-17T18:14:15.473623+0000","last_clean":"2024-08-17T18:14:15.473623+0000","last_became_active":"2024-08-17T18:14:05.808522+0000","last_became_peered":"2024-08-17T18:14:05.808522+0000","last_unstale":"2024-08-17T18:14:15.473623+0000","last_undegraded":"2024-08-17T18:14:15.473623+0000","last_fullsized":"2024-08-17T18:14:15.473623+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:36:44.341354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":250,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.477566+0000","last_change":"2024-08-17T18:14:05.811244+0000","last_active":"2024-08-17T18:14:15.477566+0000","last_peered":"2024-08-17T18:14:15.477566+0000","last_clean":"2024-08-17T18:14:15.477566+0000","last_became_active":"2024-08-17T18:14:05.810645+0000","last_became_peered":"2024-08-17T18:14:05.810645+0000","last_unstale":"2024-08-17T18:14:15.477566+0000","last_undegraded":"2024-08-17T18:14:15.477566+0000","last_fullsized":"2024-08-17T18:14:15.477566+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:35.278959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1631,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.805375+0000","last_change":"2024-08-17T18:12:04.638043+0000","last_active":"2024-08-17T18:14:08.805375+0000","last_peered":"2024-08-17T18:14:08.805375+0000","last_clean":"2024-08-17T18:14:08.805375+0000","last_became_active":"2024-08-17T18:12:04.637575+0000","last_became_peered":"2024-08-17T18:12:04.637575+0000","last_unstale":"2024-08-17T18:14:08.805375+0000","last_undegraded":"2024-08-17T18:14:08.805375+0000","last_fullsized":"2024-08-17T18:14:08.805375+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:26:53.888302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2429,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.805515+0000","last_change":"2024-08-17T18:14:05.805805+0000","last_active":"2024-08-17T18:14:08.805515+0000","last_peered":"2024-08-17T18:14:08.805515+0000","last_clean":"2024-08-17T18:14:08.805515+0000","last_became_active":"2024-08-17T18:14:05.805466+0000","last_became_peered":"2024-08-17T18:14:05.805466+0000","last_unstale":"2024-08-17T18:14:08.805515+0000","last_undegraded":"2024-08-17T18:14:08.805515+0000","last_fullsized":"2024-08-17T18:14:08.805515+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:47:38.239357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":677,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.805307+0000","last_change":"2024-08-17T18:14:05.810781+0000","last_active":"2024-08-17T18:14:08.805307+0000","last_peered":"2024-08-17T18:14:08.805307+0000","last_clean":"2024-08-17T18:14:08.805307+0000","last_became_active":"2024-08-17T18:14:05.810006+0000","last_became_peered":"2024-08-17T18:14:05.810006+0000","last_unstale":"2024-08-17T18:14:08.805307+0000","last_undegraded":"2024-08-17T18:14:08.805307+0000","last_fullsized":"2024-08-17T18:14:08.805307+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:02:07.579483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":355,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.474013+0000","last_change":"2024-08-17T18:14:05.817698+0000","last_active":"2024-08-17T18:14:15.474013+0000","last_peered":"2024-08-17T18:14:15.474013+0000","last_clean":"2024-08-17T18:14:15.474013+0000","last_became_active":"2024-08-17T18:14:05.817367+0000","last_became_peered":"2024-08-17T18:14:05.817367+0000","last_unstale":"2024-08-17T18:14:15.474013+0000","last_undegraded":"2024-08-17T18:14:15.474013+0000","last_fullsized":"2024-08-17T18:14:15.474013+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:57:14.549161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2448,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.473248+0000","last_change":"2024-08-17T18:14:05.817804+0000","last_active":"2024-08-17T18:14:15.473248+0000","last_peered":"2024-08-17T18:14:15.473248+0000","last_clean":"2024-08-17T18:14:15.473248+0000","last_became_active":"2024-08-17T18:14:05.817459+0000","last_became_peered":"2024-08-17T18:14:05.817459+0000","last_unstale":"2024-08-17T18:14:15.473248+0000","last_undegraded":"2024-08-17T18:14:15.473248+0000","last_fullsized":"2024-08-17T18:14:15.473248+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:50:59.956413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1628,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.799086+0000","last_change":"2024-08-17T18:12:04.641507+0000","last_active":"2024-08-17T18:14:08.799086+0000","last_peered":"2024-08-17T18:14:08.799086+0000","last_clean":"2024-08-17T18:14:08.799086+0000","last_became_active":"2024-08-17T18:12:04.641206+0000","last_became_peered":"2024-08-17T18:12:04.641206+0000","last_unstale":"2024-08-17T18:14:08.799086+0000","last_undegraded":"2024-08-17T18:14:08.799086+0000","last_fullsized":"2024-08-17T18:14:08.799086+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:35:15.727854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2403,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.799151+0000","last_change":"2024-08-17T18:14:05.802920+0000","last_active":"2024-08-17T18:14:08.799151+0000","last_peered":"2024-08-17T18:14:08.799151+0000","last_clean":"2024-08-17T18:14:08.799151+0000","last_became_active":"2024-08-17T18:14:05.802633+0000","last_became_peered":"2024-08-17T18:14:05.802633+0000","last_unstale":"2024-08-17T18:14:08.799151+0000","last_undegraded":"2024-08-17T18:14:08.799151+0000","last_fullsized":"2024-08-17T18:14:08.799151+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:18:19.553313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":340,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.475413+0000","last_change":"2024-08-17T18:14:05.817748+0000","last_active":"2024-08-17T18:14:15.475413+0000","last_peered":"2024-08-17T18:14:15.475413+0000","last_clean":"2024-08-17T18:14:15.475413+0000","last_became_active":"2024-08-17T18:14:05.817189+0000","last_became_peered":"2024-08-17T18:14:05.817189+0000","last_unstale":"2024-08-17T18:14:15.475413+0000","last_undegraded":"2024-08-17T18:14:15.475413+0000","last_fullsized":"2024-08-17T18:14:15.475413+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:56:50.081279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2195,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.799363+0000","last_change":"2024-08-17T18:14:05.803945+0000","last_active":"2024-08-17T18:14:08.799363+0000","last_peered":"2024-08-17T18:14:08.799363+0000","last_clean":"2024-08-17T18:14:08.799363+0000","last_became_active":"2024-08-17T18:14:05.803644+0000","last_became_peered":"2024-08-17T18:14:05.803644+0000","last_unstale":"2024-08-17T18:14:08.799363+0000","last_undegraded":"2024-08-17T18:14:08.799363+0000","last_fullsized":"2024-08-17T18:14:08.799363+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:01:16.051303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":198,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.475628+0000","last_change":"2024-08-17T18:14:05.810598+0000","last_active":"2024-08-17T18:14:15.475628+0000","last_peered":"2024-08-17T18:14:15.475628+0000","last_clean":"2024-08-17T18:14:15.475628+0000","last_became_active":"2024-08-17T18:14:05.810199+0000","last_became_peered":"2024-08-17T18:14:05.810199+0000","last_unstale":"2024-08-17T18:14:15.475628+0000","last_undegraded":"2024-08-17T18:14:15.475628+0000","last_fullsized":"2024-08-17T18:14:15.475628+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:00.046370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":34,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.805692+0000","last_change":"2024-08-17T18:14:05.819912+0000","last_active":"2024-08-17T18:14:08.805692+0000","last_peered":"2024-08-17T18:14:08.805692+0000","last_clean":"2024-08-17T18:14:08.805692+0000","last_became_active":"2024-08-17T18:14:05.809301+0000","last_became_peered":"2024-08-17T18:14:05.809301+0000","last_unstale":"2024-08-17T18:14:08.805692+0000","last_undegraded":"2024-08-17T18:14:08.805692+0000","last_fullsized":"2024-08-17T18:14:08.805692+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:04:11.329721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2427,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.477413+0000","last_change":"2024-08-17T18:12:04.649882+0000","last_active":"2024-08-17T18:14:15.477413+0000","last_peered":"2024-08-17T18:14:15.477413+0000","last_clean":"2024-08-17T18:14:15.477413+0000","last_became_active":"2024-08-17T18:12:04.649185+0000","last_became_peered":"2024-08-17T18:12:04.649185+0000","last_unstale":"2024-08-17T18:14:15.477413+0000","last_undegraded":"2024-08-17T18:14:15.477413+0000","last_fullsized":"2024-08-17T18:14:15.477413+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:42.653827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1620,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.805964+0000","last_change":"2024-08-17T18:03:51.292044+0000","last_active":"2024-08-17T18:14:08.805964+0000","last_peered":"2024-08-17T18:14:08.805964+0000","last_clean":"2024-08-17T18:14:08.805964+0000","last_became_active":"2024-08-17T18:03:51.268271+0000","last_became_peered":"2024-08-17T18:03:51.268271+0000","last_unstale":"2024-08-17T18:14:08.805964+0000","last_undegraded":"2024-08-17T18:14:08.805964+0000","last_fullsized":"2024-08-17T18:14:08.805964+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:00.892410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4756,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1601,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.805737+0000","last_change":"2024-08-17T18:12:04.639269+0000","last_active":"2024-08-17T18:14:08.805737+0000","last_peered":"2024-08-17T18:14:08.805737+0000","last_clean":"2024-08-17T18:14:08.805737+0000","last_became_active":"2024-08-17T18:12:04.638752+0000","last_became_peered":"2024-08-17T18:12:04.638752+0000","last_unstale":"2024-08-17T18:14:08.805737+0000","last_undegraded":"2024-08-17T18:14:08.805737+0000","last_fullsized":"2024-08-17T18:14:08.805737+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:50:00.462972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"347'149","reported_seq":821,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:09.393074+0000","last_change":"2024-08-17T18:14:08.809660+0000","last_active":"2024-08-17T18:14:09.393074+0000","last_peered":"2024-08-17T18:14:09.393074+0000","last_clean":"2024-08-17T18:14:09.393074+0000","last_became_active":"2024-08-17T18:14:08.809289+0000","last_became_peered":"2024-08-17T18:14:08.809289+0000","last_unstale":"2024-08-17T18:14:09.393074+0000","last_undegraded":"2024-08-17T18:14:09.393074+0000","last_fullsized":"2024-08-17T18:14:09.393074+0000","mapping_epoch":1611,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1612,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:57.271286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:57.271286+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:58:26.052040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1616,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.805485+0000","last_change":"2024-08-17T18:04:43.766724+0000","last_active":"2024-08-17T18:14:08.805485+0000","last_peered":"2024-08-17T18:14:08.805485+0000","last_clean":"2024-08-17T18:14:08.805485+0000","last_became_active":"2024-08-17T18:04:43.766428+0000","last_became_peered":"2024-08-17T18:04:43.766428+0000","last_unstale":"2024-08-17T18:14:08.805485+0000","last_undegraded":"2024-08-17T18:14:08.805485+0000","last_fullsized":"2024-08-17T18:14:08.805485+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:09:49.102940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1638,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.799185+0000","last_change":"2024-08-17T18:14:05.810263+0000","last_active":"2024-08-17T18:14:08.799185+0000","last_peered":"2024-08-17T18:14:08.799185+0000","last_clean":"2024-08-17T18:14:08.799185+0000","last_became_active":"2024-08-17T18:14:05.809966+0000","last_became_peered":"2024-08-17T18:14:05.809966+0000","last_unstale":"2024-08-17T18:14:08.799185+0000","last_undegraded":"2024-08-17T18:14:08.799185+0000","last_fullsized":"2024-08-17T18:14:08.799185+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:10:52.282935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":341,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.474066+0000","last_change":"2024-08-17T18:12:04.653301+0000","last_active":"2024-08-17T18:14:15.474066+0000","last_peered":"2024-08-17T18:14:15.474066+0000","last_clean":"2024-08-17T18:14:15.474066+0000","last_became_active":"2024-08-17T18:12:04.642708+0000","last_became_peered":"2024-08-17T18:12:04.642708+0000","last_unstale":"2024-08-17T18:14:15.474066+0000","last_undegraded":"2024-08-17T18:14:15.474066+0000","last_fullsized":"2024-08-17T18:14:15.474066+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:22:49.885441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"347'149","reported_seq":886,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.805355+0000","last_change":"2024-08-17T18:09:15.592395+0000","last_active":"2024-08-17T18:14:08.805355+0000","last_peered":"2024-08-17T18:14:08.805355+0000","last_clean":"2024-08-17T18:14:08.805355+0000","last_became_active":"2024-08-17T18:09:15.591773+0000","last_became_peered":"2024-08-17T18:09:15.591773+0000","last_unstale":"2024-08-17T18:14:08.805355+0000","last_undegraded":"2024-08-17T18:14:08.805355+0000","last_fullsized":"2024-08-17T18:14:08.805355+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:53.309517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:53.309517+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:50:57.110936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2299,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.473440+0000","last_change":"2024-08-17T18:14:05.818280+0000","last_active":"2024-08-17T18:14:15.473440+0000","last_peered":"2024-08-17T18:14:15.473440+0000","last_clean":"2024-08-17T18:14:15.473440+0000","last_became_active":"2024-08-17T18:14:05.817964+0000","last_became_peered":"2024-08-17T18:14:05.817964+0000","last_unstale":"2024-08-17T18:14:15.473440+0000","last_undegraded":"2024-08-17T18:14:15.473440+0000","last_fullsized":"2024-08-17T18:14:15.473440+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:40.557508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":681,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.478276+0000","last_change":"2024-08-17T18:14:05.813132+0000","last_active":"2024-08-17T18:14:15.478276+0000","last_peered":"2024-08-17T18:14:15.478276+0000","last_clean":"2024-08-17T18:14:15.478276+0000","last_became_active":"2024-08-17T18:14:05.812684+0000","last_became_peered":"2024-08-17T18:14:05.812684+0000","last_unstale":"2024-08-17T18:14:15.478276+0000","last_undegraded":"2024-08-17T18:14:15.478276+0000","last_fullsized":"2024-08-17T18:14:15.478276+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:30:56.396969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":2246,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.475136+0000","last_change":"2024-08-17T18:14:05.825767+0000","last_active":"2024-08-17T18:14:15.475136+0000","last_peered":"2024-08-17T18:14:15.475136+0000","last_clean":"2024-08-17T18:14:15.475136+0000","last_became_active":"2024-08-17T18:14:05.818449+0000","last_became_peered":"2024-08-17T18:14:05.818449+0000","last_unstale":"2024-08-17T18:14:15.475136+0000","last_undegraded":"2024-08-17T18:14:15.475136+0000","last_fullsized":"2024-08-17T18:14:15.475136+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:58:57.889315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"347'149","reported_seq":886,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.809868+0000","last_change":"2024-08-17T18:14:08.809420+0000","last_active":"2024-08-17T18:14:08.809868+0000","last_peered":"2024-08-17T18:14:08.809868+0000","last_clean":"2024-08-17T18:14:08.809868+0000","last_became_active":"2024-08-17T18:14:08.809030+0000","last_became_peered":"2024-08-17T18:14:08.809030+0000","last_unstale":"2024-08-17T18:14:08.809868+0000","last_undegraded":"2024-08-17T18:14:08.809868+0000","last_fullsized":"2024-08-17T18:14:08.809868+0000","mapping_epoch":1611,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1612,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:52.474544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:52.474544+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:01:23.383908+0000","scrub_duration":0.001886326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":340,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.473706+0000","last_change":"2024-08-17T18:12:04.643348+0000","last_active":"2024-08-17T18:14:15.473706+0000","last_peered":"2024-08-17T18:14:15.473706+0000","last_clean":"2024-08-17T18:14:15.473706+0000","last_became_active":"2024-08-17T18:12:04.641662+0000","last_became_peered":"2024-08-17T18:12:04.641662+0000","last_unstale":"2024-08-17T18:14:15.473706+0000","last_undegraded":"2024-08-17T18:14:15.473706+0000","last_fullsized":"2024-08-17T18:14:15.473706+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:59.730975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":333,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.474068+0000","last_change":"2024-08-17T18:12:04.649331+0000","last_active":"2024-08-17T18:14:15.474068+0000","last_peered":"2024-08-17T18:14:15.474068+0000","last_clean":"2024-08-17T18:14:15.474068+0000","last_became_active":"2024-08-17T18:12:04.646946+0000","last_became_peered":"2024-08-17T18:12:04.646946+0000","last_unstale":"2024-08-17T18:14:15.474068+0000","last_undegraded":"2024-08-17T18:14:15.474068+0000","last_fullsized":"2024-08-17T18:14:15.474068+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:58:41.839200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2394,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.477914+0000","last_change":"2024-08-17T18:12:04.641886+0000","last_active":"2024-08-17T18:14:15.477914+0000","last_peered":"2024-08-17T18:14:15.477914+0000","last_clean":"2024-08-17T18:14:15.477914+0000","last_became_active":"2024-08-17T18:12:04.640395+0000","last_became_peered":"2024-08-17T18:12:04.640395+0000","last_unstale":"2024-08-17T18:14:15.477914+0000","last_undegraded":"2024-08-17T18:14:15.477914+0000","last_fullsized":"2024-08-17T18:14:15.477914+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:20:13.078751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"347'149","reported_seq":886,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.808879+0000","last_change":"2024-08-17T18:14:08.808000+0000","last_active":"2024-08-17T18:14:08.808879+0000","last_peered":"2024-08-17T18:14:08.808879+0000","last_clean":"2024-08-17T18:14:08.808879+0000","last_became_active":"2024-08-17T18:14:08.807605+0000","last_became_peered":"2024-08-17T18:14:08.807605+0000","last_unstale":"2024-08-17T18:14:08.808879+0000","last_undegraded":"2024-08-17T18:14:08.808879+0000","last_fullsized":"2024-08-17T18:14:08.808879+0000","mapping_epoch":1611,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1612,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:49.461698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:49.461698+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:43:24.566663+0000","scrub_duration":0.0013792520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":159,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:09.393032+0000","last_change":"2024-08-17T18:12:04.639394+0000","last_active":"2024-08-17T18:14:09.393032+0000","last_peered":"2024-08-17T18:14:09.393032+0000","last_clean":"2024-08-17T18:14:09.393032+0000","last_became_active":"2024-08-17T18:12:04.638935+0000","last_became_peered":"2024-08-17T18:12:04.638935+0000","last_unstale":"2024-08-17T18:14:09.393032+0000","last_undegraded":"2024-08-17T18:14:09.393032+0000","last_fullsized":"2024-08-17T18:14:09.393032+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:21:16.994042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2403,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.474036+0000","last_change":"2024-08-17T18:14:05.808588+0000","last_active":"2024-08-17T18:14:15.474036+0000","last_peered":"2024-08-17T18:14:15.474036+0000","last_clean":"2024-08-17T18:14:15.474036+0000","last_became_active":"2024-08-17T18:14:05.808105+0000","last_became_peered":"2024-08-17T18:14:05.808105+0000","last_unstale":"2024-08-17T18:14:15.474036+0000","last_undegraded":"2024-08-17T18:14:15.474036+0000","last_fullsized":"2024-08-17T18:14:15.474036+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:01:57.158744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":172,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.804062+0000","last_change":"2024-08-17T18:14:05.817143+0000","last_active":"2024-08-17T18:14:08.804062+0000","last_peered":"2024-08-17T18:14:08.804062+0000","last_clean":"2024-08-17T18:14:08.804062+0000","last_became_active":"2024-08-17T18:14:05.816518+0000","last_became_peered":"2024-08-17T18:14:05.816518+0000","last_unstale":"2024-08-17T18:14:08.804062+0000","last_undegraded":"2024-08-17T18:14:08.804062+0000","last_fullsized":"2024-08-17T18:14:08.804062+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:29:14.833759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"347'149","reported_seq":887,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.801259+0000","last_change":"2024-08-17T18:09:15.596647+0000","last_active":"2024-08-17T18:14:08.801259+0000","last_peered":"2024-08-17T18:14:08.801259+0000","last_clean":"2024-08-17T18:14:08.801259+0000","last_became_active":"2024-08-17T18:09:15.596339+0000","last_became_peered":"2024-08-17T18:09:15.596339+0000","last_unstale":"2024-08-17T18:14:08.801259+0000","last_undegraded":"2024-08-17T18:14:08.801259+0000","last_fullsized":"2024-08-17T18:14:08.801259+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:00.316884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:00.316884+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:30.779927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1528,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.477837+0000","last_change":"2024-08-17T18:14:05.806747+0000","last_active":"2024-08-17T18:14:15.477837+0000","last_peered":"2024-08-17T18:14:15.477837+0000","last_clean":"2024-08-17T18:14:15.477837+0000","last_became_active":"2024-08-17T18:14:05.805943+0000","last_became_peered":"2024-08-17T18:14:05.805943+0000","last_unstale":"2024-08-17T18:14:15.477837+0000","last_undegraded":"2024-08-17T18:14:15.477837+0000","last_fullsized":"2024-08-17T18:14:15.477837+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:33:51.579715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1614,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.805891+0000","last_change":"2024-08-17T18:04:14.526890+0000","last_active":"2024-08-17T18:14:08.805891+0000","last_peered":"2024-08-17T18:14:08.805891+0000","last_clean":"2024-08-17T18:14:08.805891+0000","last_became_active":"2024-08-17T18:04:14.526564+0000","last_became_peered":"2024-08-17T18:04:14.526564+0000","last_unstale":"2024-08-17T18:14:08.805891+0000","last_undegraded":"2024-08-17T18:14:08.805891+0000","last_fullsized":"2024-08-17T18:14:08.805891+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:10:13.229827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":180,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.803739+0000","last_change":"2024-08-17T18:12:04.658437+0000","last_active":"2024-08-17T18:14:08.803739+0000","last_peered":"2024-08-17T18:14:08.803739+0000","last_clean":"2024-08-17T18:14:08.803739+0000","last_became_active":"2024-08-17T18:12:04.647294+0000","last_became_peered":"2024-08-17T18:12:04.647294+0000","last_unstale":"2024-08-17T18:14:08.803739+0000","last_undegraded":"2024-08-17T18:14:08.803739+0000","last_fullsized":"2024-08-17T18:14:08.803739+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:05:39.701204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"347'149","reported_seq":976,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.474563+0000","last_change":"2024-08-17T18:09:15.596502+0000","last_active":"2024-08-17T18:14:15.474563+0000","last_peered":"2024-08-17T18:14:15.474563+0000","last_clean":"2024-08-17T18:14:15.474563+0000","last_became_active":"2024-08-17T18:09:15.595460+0000","last_became_peered":"2024-08-17T18:09:15.595460+0000","last_unstale":"2024-08-17T18:14:15.474563+0000","last_undegraded":"2024-08-17T18:14:15.474563+0000","last_fullsized":"2024-08-17T18:14:15.474563+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:02.363531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:02.363531+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:40.918706+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2346,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:14.395233+0000","last_change":"2024-08-17T18:14:05.812981+0000","last_active":"2024-08-17T18:14:14.395233+0000","last_peered":"2024-08-17T18:14:14.395233+0000","last_clean":"2024-08-17T18:14:14.395233+0000","last_became_active":"2024-08-17T18:14:05.812684+0000","last_became_peered":"2024-08-17T18:14:05.812684+0000","last_unstale":"2024-08-17T18:14:14.395233+0000","last_undegraded":"2024-08-17T18:14:14.395233+0000","last_fullsized":"2024-08-17T18:14:14.395233+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:03:28.664775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2222,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.476242+0000","last_change":"2024-08-17T18:12:04.648190+0000","last_active":"2024-08-17T18:14:15.476242+0000","last_peered":"2024-08-17T18:14:15.476242+0000","last_clean":"2024-08-17T18:14:15.476242+0000","last_became_active":"2024-08-17T18:12:04.647877+0000","last_became_peered":"2024-08-17T18:12:04.647877+0000","last_unstale":"2024-08-17T18:14:15.476242+0000","last_undegraded":"2024-08-17T18:14:15.476242+0000","last_fullsized":"2024-08-17T18:14:15.476242+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:18.129337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2517,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.476442+0000","last_change":"2024-08-17T18:14:05.811655+0000","last_active":"2024-08-17T18:14:15.476442+0000","last_peered":"2024-08-17T18:14:15.476442+0000","last_clean":"2024-08-17T18:14:15.476442+0000","last_became_active":"2024-08-17T18:14:05.808974+0000","last_became_peered":"2024-08-17T18:14:05.808974+0000","last_unstale":"2024-08-17T18:14:15.476442+0000","last_undegraded":"2024-08-17T18:14:15.476442+0000","last_fullsized":"2024-08-17T18:14:15.476442+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:37:31.935317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"347'149","reported_seq":844,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:12.429265+0000","last_change":"2024-08-17T18:12:53.586781+0000","last_active":"2024-08-17T18:14:12.429265+0000","last_peered":"2024-08-17T18:14:12.429265+0000","last_clean":"2024-08-17T18:14:12.429265+0000","last_became_active":"2024-08-17T18:12:07.693969+0000","last_became_peered":"2024-08-17T18:12:07.693969+0000","last_unstale":"2024-08-17T18:14:12.429265+0000","last_undegraded":"2024-08-17T18:14:12.429265+0000","last_fullsized":"2024-08-17T18:14:12.429265+0000","mapping_epoch":1556,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1557,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:12:53.586672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:12:53.586672+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:05:00.904360+0000","scrub_duration":0.001460025,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":354,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.473877+0000","last_change":"2024-08-17T18:12:04.642603+0000","last_active":"2024-08-17T18:14:15.473877+0000","last_peered":"2024-08-17T18:14:15.473877+0000","last_clean":"2024-08-17T18:14:15.473877+0000","last_became_active":"2024-08-17T18:12:04.642025+0000","last_became_peered":"2024-08-17T18:12:04.642025+0000","last_unstale":"2024-08-17T18:14:15.473877+0000","last_undegraded":"2024-08-17T18:14:15.473877+0000","last_fullsized":"2024-08-17T18:14:15.473877+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:06:50.276904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":347,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.473722+0000","last_change":"2024-08-17T18:14:05.819016+0000","last_active":"2024-08-17T18:14:15.473722+0000","last_peered":"2024-08-17T18:14:15.473722+0000","last_clean":"2024-08-17T18:14:15.473722+0000","last_became_active":"2024-08-17T18:14:05.816900+0000","last_became_peered":"2024-08-17T18:14:05.816900+0000","last_unstale":"2024-08-17T18:14:15.473722+0000","last_undegraded":"2024-08-17T18:14:15.473722+0000","last_fullsized":"2024-08-17T18:14:15.473722+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:15:36.862952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1090,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.473328+0000","last_change":"2024-08-17T18:14:05.804231+0000","last_active":"2024-08-17T18:14:15.473328+0000","last_peered":"2024-08-17T18:14:15.473328+0000","last_clean":"2024-08-17T18:14:15.473328+0000","last_became_active":"2024-08-17T18:14:05.803625+0000","last_became_peered":"2024-08-17T18:14:05.803625+0000","last_unstale":"2024-08-17T18:14:15.473328+0000","last_undegraded":"2024-08-17T18:14:15.473328+0000","last_fullsized":"2024-08-17T18:14:15.473328+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:18:33.017235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":110,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.799914+0000","last_change":"2024-08-17T18:08:37.071912+0000","last_active":"2024-08-17T18:14:08.799914+0000","last_peered":"2024-08-17T18:14:08.799914+0000","last_clean":"2024-08-17T18:14:08.799914+0000","last_became_active":"2024-08-17T18:08:37.071648+0000","last_became_peered":"2024-08-17T18:08:37.071648+0000","last_unstale":"2024-08-17T18:14:08.799914+0000","last_undegraded":"2024-08-17T18:14:08.799914+0000","last_fullsized":"2024-08-17T18:14:08.799914+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:58:57.279808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":43,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:09.393014+0000","last_change":"2024-08-17T18:14:05.815709+0000","last_active":"2024-08-17T18:14:09.393014+0000","last_peered":"2024-08-17T18:14:09.393014+0000","last_clean":"2024-08-17T18:14:09.393014+0000","last_became_active":"2024-08-17T18:14:05.815411+0000","last_became_peered":"2024-08-17T18:14:05.815411+0000","last_unstale":"2024-08-17T18:14:09.393014+0000","last_undegraded":"2024-08-17T18:14:09.393014+0000","last_fullsized":"2024-08-17T18:14:09.393014+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:56:03.062386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":180,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.803049+0000","last_change":"2024-08-17T18:14:05.824962+0000","last_active":"2024-08-17T18:14:08.803049+0000","last_peered":"2024-08-17T18:14:08.803049+0000","last_clean":"2024-08-17T18:14:08.803049+0000","last_became_active":"2024-08-17T18:14:05.815667+0000","last_became_peered":"2024-08-17T18:14:05.815667+0000","last_unstale":"2024-08-17T18:14:08.803049+0000","last_undegraded":"2024-08-17T18:14:08.803049+0000","last_fullsized":"2024-08-17T18:14:08.803049+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:44:49.527853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":978,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.474223+0000","last_change":"2024-08-17T18:12:07.677305+0000","last_active":"2024-08-17T18:14:15.474223+0000","last_peered":"2024-08-17T18:14:15.474223+0000","last_clean":"2024-08-17T18:14:15.474223+0000","last_became_active":"2024-08-17T18:12:07.677002+0000","last_became_peered":"2024-08-17T18:12:07.677002+0000","last_unstale":"2024-08-17T18:14:15.474223+0000","last_undegraded":"2024-08-17T18:14:15.474223+0000","last_fullsized":"2024-08-17T18:14:15.474223+0000","mapping_epoch":1556,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1557,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:54.303708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:54.303708+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:11:29.280261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"347'149","reported_seq":888,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.803418+0000","last_change":"2024-08-17T18:09:15.592722+0000","last_active":"2024-08-17T18:14:08.803418+0000","last_peered":"2024-08-17T18:14:08.803418+0000","last_clean":"2024-08-17T18:14:08.803418+0000","last_became_active":"2024-08-17T18:09:15.592242+0000","last_became_peered":"2024-08-17T18:09:15.592242+0000","last_unstale":"2024-08-17T18:14:08.803418+0000","last_undegraded":"2024-08-17T18:14:08.803418+0000","last_fullsized":"2024-08-17T18:14:08.803418+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:55.255983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:55.255983+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:02:53.596275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":657,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.807164+0000","last_change":"2024-08-17T18:14:05.810827+0000","last_active":"2024-08-17T18:14:08.807164+0000","last_peered":"2024-08-17T18:14:08.807164+0000","last_clean":"2024-08-17T18:14:08.807164+0000","last_became_active":"2024-08-17T18:14:05.810468+0000","last_became_peered":"2024-08-17T18:14:05.810468+0000","last_unstale":"2024-08-17T18:14:08.807164+0000","last_undegraded":"2024-08-17T18:14:08.807164+0000","last_fullsized":"2024-08-17T18:14:08.807164+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:09:27.684771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2216,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.804722+0000","last_change":"2024-08-17T18:14:05.804960+0000","last_active":"2024-08-17T18:14:08.804722+0000","last_peered":"2024-08-17T18:14:08.804722+0000","last_clean":"2024-08-17T18:14:08.804722+0000","last_became_active":"2024-08-17T18:14:05.804656+0000","last_became_peered":"2024-08-17T18:14:05.804656+0000","last_unstale":"2024-08-17T18:14:08.804722+0000","last_undegraded":"2024-08-17T18:14:08.804722+0000","last_fullsized":"2024-08-17T18:14:08.804722+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:17:32.568323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1629,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.807256+0000","last_change":"2024-08-17T18:14:05.805956+0000","last_active":"2024-08-17T18:14:08.807256+0000","last_peered":"2024-08-17T18:14:08.807256+0000","last_clean":"2024-08-17T18:14:08.807256+0000","last_became_active":"2024-08-17T18:14:05.805628+0000","last_became_peered":"2024-08-17T18:14:05.805628+0000","last_unstale":"2024-08-17T18:14:08.807256+0000","last_undegraded":"2024-08-17T18:14:08.807256+0000","last_fullsized":"2024-08-17T18:14:08.807256+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:53:29.617620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1628,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.805528+0000","last_change":"2024-08-17T18:12:04.639709+0000","last_active":"2024-08-17T18:14:08.805528+0000","last_peered":"2024-08-17T18:14:08.805528+0000","last_clean":"2024-08-17T18:14:08.805528+0000","last_became_active":"2024-08-17T18:12:04.639383+0000","last_became_peered":"2024-08-17T18:12:04.639383+0000","last_unstale":"2024-08-17T18:14:08.805528+0000","last_undegraded":"2024-08-17T18:14:08.805528+0000","last_fullsized":"2024-08-17T18:14:08.805528+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:16:49.177518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"347'149","reported_seq":909,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.807092+0000","last_change":"2024-08-17T18:14:08.806779+0000","last_active":"2024-08-17T18:14:08.807092+0000","last_peered":"2024-08-17T18:14:08.807092+0000","last_clean":"2024-08-17T18:14:08.807092+0000","last_became_active":"2024-08-17T18:14:08.806410+0000","last_became_peered":"2024-08-17T18:14:08.806410+0000","last_unstale":"2024-08-17T18:14:08.807092+0000","last_undegraded":"2024-08-17T18:14:08.807092+0000","last_fullsized":"2024-08-17T18:14:08.807092+0000","mapping_epoch":1611,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1612,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:59.307356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:59.307356+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:52:45.658090+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1632,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.805597+0000","last_change":"2024-08-17T18:12:04.648228+0000","last_active":"2024-08-17T18:14:08.805597+0000","last_peered":"2024-08-17T18:14:08.805597+0000","last_clean":"2024-08-17T18:14:08.805597+0000","last_became_active":"2024-08-17T18:12:04.647906+0000","last_became_peered":"2024-08-17T18:12:04.647906+0000","last_unstale":"2024-08-17T18:14:08.805597+0000","last_undegraded":"2024-08-17T18:14:08.805597+0000","last_fullsized":"2024-08-17T18:14:08.805597+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:51:30.081540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1610,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.797709+0000","last_change":"2024-08-17T18:14:05.802624+0000","last_active":"2024-08-17T18:14:08.797709+0000","last_peered":"2024-08-17T18:14:08.797709+0000","last_clean":"2024-08-17T18:14:08.797709+0000","last_became_active":"2024-08-17T18:14:05.802177+0000","last_became_peered":"2024-08-17T18:14:05.802177+0000","last_unstale":"2024-08-17T18:14:08.797709+0000","last_undegraded":"2024-08-17T18:14:08.797709+0000","last_fullsized":"2024-08-17T18:14:08.797709+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:49:07.674636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"347'149","reported_seq":816,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.476337+0000","last_change":"2024-08-17T18:14:08.807477+0000","last_active":"2024-08-17T18:14:15.476337+0000","last_peered":"2024-08-17T18:14:15.476337+0000","last_clean":"2024-08-17T18:14:15.476337+0000","last_became_active":"2024-08-17T18:14:08.807110+0000","last_became_peered":"2024-08-17T18:14:08.807110+0000","last_unstale":"2024-08-17T18:14:15.476337+0000","last_undegraded":"2024-08-17T18:14:15.476337+0000","last_fullsized":"2024-08-17T18:14:15.476337+0000","mapping_epoch":1611,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1612,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:52.271487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:52.271487+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:46:28.440514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1602,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.805819+0000","last_change":"2024-08-17T18:14:05.810790+0000","last_active":"2024-08-17T18:14:08.805819+0000","last_peered":"2024-08-17T18:14:08.805819+0000","last_clean":"2024-08-17T18:14:08.805819+0000","last_became_active":"2024-08-17T18:14:05.810506+0000","last_became_peered":"2024-08-17T18:14:05.810506+0000","last_unstale":"2024-08-17T18:14:08.805819+0000","last_undegraded":"2024-08-17T18:14:08.805819+0000","last_fullsized":"2024-08-17T18:14:08.805819+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:15:11.045397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":2223,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.473477+0000","last_change":"2024-08-17T18:14:05.815944+0000","last_active":"2024-08-17T18:14:15.473477+0000","last_peered":"2024-08-17T18:14:15.473477+0000","last_clean":"2024-08-17T18:14:15.473477+0000","last_became_active":"2024-08-17T18:14:05.807238+0000","last_became_peered":"2024-08-17T18:14:05.807238+0000","last_unstale":"2024-08-17T18:14:15.473477+0000","last_undegraded":"2024-08-17T18:14:15.473477+0000","last_fullsized":"2024-08-17T18:14:15.473477+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:16.485620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":340,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.473698+0000","last_change":"2024-08-17T18:14:05.805855+0000","last_active":"2024-08-17T18:14:15.473698+0000","last_peered":"2024-08-17T18:14:15.473698+0000","last_clean":"2024-08-17T18:14:15.473698+0000","last_became_active":"2024-08-17T18:14:05.805485+0000","last_became_peered":"2024-08-17T18:14:05.805485+0000","last_unstale":"2024-08-17T18:14:15.473698+0000","last_undegraded":"2024-08-17T18:14:15.473698+0000","last_fullsized":"2024-08-17T18:14:15.473698+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:49:20.533746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":172,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.803461+0000","last_change":"2024-08-17T18:14:05.815112+0000","last_active":"2024-08-17T18:14:08.803461+0000","last_peered":"2024-08-17T18:14:08.803461+0000","last_clean":"2024-08-17T18:14:08.803461+0000","last_became_active":"2024-08-17T18:14:05.814806+0000","last_became_peered":"2024-08-17T18:14:05.814806+0000","last_unstale":"2024-08-17T18:14:08.803461+0000","last_undegraded":"2024-08-17T18:14:08.803461+0000","last_fullsized":"2024-08-17T18:14:08.803461+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:16:12.812066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":119,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.797479+0000","last_change":"2024-08-17T18:12:04.649216+0000","last_active":"2024-08-17T18:14:08.797479+0000","last_peered":"2024-08-17T18:14:08.797479+0000","last_clean":"2024-08-17T18:14:08.797479+0000","last_became_active":"2024-08-17T18:12:04.648753+0000","last_became_peered":"2024-08-17T18:12:04.648753+0000","last_unstale":"2024-08-17T18:14:08.797479+0000","last_undegraded":"2024-08-17T18:14:08.797479+0000","last_fullsized":"2024-08-17T18:14:08.797479+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:42:43.981181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":180,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.804343+0000","last_change":"2024-08-17T18:14:05.823272+0000","last_active":"2024-08-17T18:14:08.804343+0000","last_peered":"2024-08-17T18:14:08.804343+0000","last_clean":"2024-08-17T18:14:08.804343+0000","last_became_active":"2024-08-17T18:14:05.816054+0000","last_became_peered":"2024-08-17T18:14:05.816054+0000","last_unstale":"2024-08-17T18:14:08.804343+0000","last_undegraded":"2024-08-17T18:14:08.804343+0000","last_fullsized":"2024-08-17T18:14:08.804343+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:22:48.996624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1645,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.805120+0000","last_change":"2024-08-17T18:14:05.809877+0000","last_active":"2024-08-17T18:14:08.805120+0000","last_peered":"2024-08-17T18:14:08.805120+0000","last_clean":"2024-08-17T18:14:08.805120+0000","last_became_active":"2024-08-17T18:14:05.809582+0000","last_became_peered":"2024-08-17T18:14:05.809582+0000","last_unstale":"2024-08-17T18:14:08.805120+0000","last_undegraded":"2024-08-17T18:14:08.805120+0000","last_fullsized":"2024-08-17T18:14:08.805120+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:20.740796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":356,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.477028+0000","last_change":"2024-08-17T18:12:04.655292+0000","last_active":"2024-08-17T18:14:15.477028+0000","last_peered":"2024-08-17T18:14:15.477028+0000","last_clean":"2024-08-17T18:14:15.477028+0000","last_became_active":"2024-08-17T18:12:04.641282+0000","last_became_peered":"2024-08-17T18:12:04.641282+0000","last_unstale":"2024-08-17T18:14:15.477028+0000","last_undegraded":"2024-08-17T18:14:15.477028+0000","last_fullsized":"2024-08-17T18:14:15.477028+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:36:21.436823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":181,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.800406+0000","last_change":"2024-08-17T18:08:37.068503+0000","last_active":"2024-08-17T18:14:08.800406+0000","last_peered":"2024-08-17T18:14:08.800406+0000","last_clean":"2024-08-17T18:14:08.800406+0000","last_became_active":"2024-08-17T18:08:37.068110+0000","last_became_peered":"2024-08-17T18:08:37.068110+0000","last_unstale":"2024-08-17T18:14:08.800406+0000","last_undegraded":"2024-08-17T18:14:08.800406+0000","last_fullsized":"2024-08-17T18:14:08.800406+0000","mapping_epoch":1515,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:12:30.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":221,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.807147+0000","last_change":"2024-08-17T18:12:04.644320+0000","last_active":"2024-08-17T18:14:08.807147+0000","last_peered":"2024-08-17T18:14:08.807147+0000","last_clean":"2024-08-17T18:14:08.807147+0000","last_became_active":"2024-08-17T18:12:04.638742+0000","last_became_peered":"2024-08-17T18:12:04.638742+0000","last_unstale":"2024-08-17T18:14:08.807147+0000","last_undegraded":"2024-08-17T18:14:08.807147+0000","last_fullsized":"2024-08-17T18:14:08.807147+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:37:54.461942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1674,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.799305+0000","last_change":"2024-08-17T18:04:43.813536+0000","last_active":"2024-08-17T18:14:08.799305+0000","last_peered":"2024-08-17T18:14:08.799305+0000","last_clean":"2024-08-17T18:14:08.799305+0000","last_became_active":"2024-08-17T18:04:43.799823+0000","last_became_peered":"2024-08-17T18:04:43.799823+0000","last_unstale":"2024-08-17T18:14:08.799305+0000","last_undegraded":"2024-08-17T18:14:08.799305+0000","last_fullsized":"2024-08-17T18:14:08.799305+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:04:10.837113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2199,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.481842+0000","last_change":"2024-08-17T18:14:05.809890+0000","last_active":"2024-08-17T18:14:15.481842+0000","last_peered":"2024-08-17T18:14:15.481842+0000","last_clean":"2024-08-17T18:14:15.481842+0000","last_became_active":"2024-08-17T18:14:05.809456+0000","last_became_peered":"2024-08-17T18:14:05.809456+0000","last_unstale":"2024-08-17T18:14:15.481842+0000","last_undegraded":"2024-08-17T18:14:15.481842+0000","last_fullsized":"2024-08-17T18:14:15.481842+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:37:38.463851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":332,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.805390+0000","last_change":"2024-08-17T18:04:14.527380+0000","last_active":"2024-08-17T18:14:08.805390+0000","last_peered":"2024-08-17T18:14:08.805390+0000","last_clean":"2024-08-17T18:14:08.805390+0000","last_became_active":"2024-08-17T18:04:14.527076+0000","last_became_peered":"2024-08-17T18:04:14.527076+0000","last_unstale":"2024-08-17T18:14:08.805390+0000","last_undegraded":"2024-08-17T18:14:08.805390+0000","last_fullsized":"2024-08-17T18:14:08.805390+0000","mapping_epoch":1457,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1458,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:36:35.583709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":333,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.474126+0000","last_change":"2024-08-17T18:14:05.810250+0000","last_active":"2024-08-17T18:14:15.474126+0000","last_peered":"2024-08-17T18:14:15.474126+0000","last_clean":"2024-08-17T18:14:15.474126+0000","last_became_active":"2024-08-17T18:14:05.809933+0000","last_became_peered":"2024-08-17T18:14:05.809933+0000","last_unstale":"2024-08-17T18:14:15.474126+0000","last_undegraded":"2024-08-17T18:14:15.474126+0000","last_fullsized":"2024-08-17T18:14:15.474126+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:37:19.154055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":354,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.474750+0000","last_change":"2024-08-17T18:14:05.817957+0000","last_active":"2024-08-17T18:14:15.474750+0000","last_peered":"2024-08-17T18:14:15.474750+0000","last_clean":"2024-08-17T18:14:15.474750+0000","last_became_active":"2024-08-17T18:14:05.817152+0000","last_became_peered":"2024-08-17T18:14:05.817152+0000","last_unstale":"2024-08-17T18:14:15.474750+0000","last_undegraded":"2024-08-17T18:14:15.474750+0000","last_fullsized":"2024-08-17T18:14:15.474750+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:30:50.724758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":172,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.474871+0000","last_change":"2024-08-17T18:14:05.816753+0000","last_active":"2024-08-17T18:14:15.474871+0000","last_peered":"2024-08-17T18:14:15.474871+0000","last_clean":"2024-08-17T18:14:15.474871+0000","last_became_active":"2024-08-17T18:14:05.816044+0000","last_became_peered":"2024-08-17T18:14:05.816044+0000","last_unstale":"2024-08-17T18:14:15.474871+0000","last_undegraded":"2024-08-17T18:14:15.474871+0000","last_fullsized":"2024-08-17T18:14:15.474871+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:15:26.066215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":197,"reported_epoch":1612,"state":"active+clean","last_fresh":"2024-08-17T18:14:08.805401+0000","last_change":"2024-08-17T18:03:51.283872+0000","last_active":"2024-08-17T18:14:08.805401+0000","last_peered":"2024-08-17T18:14:08.805401+0000","last_clean":"2024-08-17T18:14:08.805401+0000","last_became_active":"2024-08-17T18:03:51.267197+0000","last_became_peered":"2024-08-17T18:03:51.267197+0000","last_unstale":"2024-08-17T18:14:08.805401+0000","last_undegraded":"2024-08-17T18:14:08.805401+0000","last_fullsized":"2024-08-17T18:14:08.805401+0000","mapping_epoch":1449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:18:59.157786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2415,"reported_epoch":1613,"state":"active+clean","last_fresh":"2024-08-17T18:14:15.477663+0000","last_change":"2024-08-17T18:14:05.806800+0000","last_active":"2024-08-17T18:14:15.477663+0000","last_peered":"2024-08-17T18:14:15.477663+0000","last_clean":"2024-08-17T18:14:15.477663+0000","last_became_active":"2024-08-17T18:14:05.806389+0000","last_became_peered":"2024-08-17T18:14:05.806389+0000","last_unstale":"2024-08-17T18:14:15.477663+0000","last_undegraded":"2024-08-17T18:14:15.477663+0000","last_fullsized":"2024-08-17T18:14:15.477663+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:02:05.755754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":57,"num_pg":16,"stat_sum":{"num_bytes":3446931456,"num_objects":52596,"num_object_clones":0,"num_object_copies":157788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52596,"num_write_kb":3366144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10195697664,"data_stored":10195697664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52596,"ondisk_log_size":52596,"up":48,"acting":48,"num_store_stats":6},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":853056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":7996,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":16844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234782,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496419,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3110340,"kb_used_data":2086652,"kb_used_omap":1,"kb_used_meta":1023678,"kb_avail":90634812,"statfs":{"total":95995035648,"available":92810047488,"internally_reserved":0,"allocated":2136731648,"data_stored":2132781353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1048246727},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":2946347565661,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":1399,"seq":6008659247263,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2946864,"kb_used_data":2459368,"kb_used_omap":131,"kb_used_meta":487356,"kb_avail":90798288,"statfs":{"total":95995035648,"available":92977446912,"internally_reserved":0,"allocated":2518392832,"data_stored":2513462484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134291,"internal_metadata":499053421},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":2864743187051,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448690,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2853224,"kb_used_data":2159136,"kb_used_omap":75,"kb_used_meta":694004,"kb_avail":90891928,"statfs":{"total":95995035648,"available":93073334272,"internally_reserved":0,"allocated":2210955264,"data_stored":2206027905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":710660287},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208758054,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2675384,"kb_used_data":1597488,"kb_used_omap":1,"kb_used_meta":1077886,"kb_avail":91069768,"statfs":{"total":95995035648,"available":93255442432,"internally_reserved":0,"allocated":1635827712,"data_stored":1631885429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1103755718},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019692,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2487360,"kb_used_data":1545720,"kb_used_omap":1,"kb_used_meta":941630,"kb_avail":91257792,"statfs":{"total":95995035648,"available":93447979008,"internally_reserved":0,"allocated":1582817280,"data_stored":1578870540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":964229574},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2225930240,"data_stored":2225930240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2525298688,"data_stored":2525298688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1603600384,"data_stored":1603600384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1683816448,"data_stored":1683816448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2157051904,"data_stored":2157051904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T18:14:33.719 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-17T18:14:33.720 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool set unique_pool_4 pg_num 26 2024-08-17T18:14:33.722 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:33.753 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:33.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:33.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:33.755 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:33 smithi136 ceph-mon[109861]: pgmap v3025: 99 pgs: 99 active+clean; 3.2 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 124 MiB/s wr, 1.99k op/s 2024-08-17T18:14:33.755 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:33 smithi136 ceph-mon[109861]: from='client.46298 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:14:33.755 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:33 smithi136 ceph-mon[115078]: pgmap v3025: 99 pgs: 99 active+clean; 3.2 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 124 MiB/s wr, 1.99k op/s 2024-08-17T18:14:33.755 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:33 smithi136 ceph-mon[115078]: from='client.46298 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:14:33.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:33 smithi165 ceph-mon[105517]: pgmap v3025: 99 pgs: 99 active+clean; 3.2 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 124 MiB/s wr, 1.99k op/s 2024-08-17T18:14:33.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:33 smithi165 ceph-mon[105517]: from='client.46298 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:14:33.777 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 26 16 58658 58642 140.947 103.562 0.00934079 0.00708985 2024-08-17T18:14:33.856 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:33.893 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:33.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:33.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:33.995 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:34.029 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:34.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:34.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:34.131 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:34.166 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:34.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:34.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:34.268 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:34.312 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:34.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:34.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:34.414 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:34.452 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:34.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:34.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:34.554 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:34.591 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:34.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:34.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:34.693 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:34.731 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:34.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:34.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:34.778 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 27 16 60262 60246 139.439 100.25 0.0117224 0.00716719 2024-08-17T18:14:34.833 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:34.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:34.871 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:34.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:34.972 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:35.007 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:35.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:35.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:35.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:35.143 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:35.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:35.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:35.244 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:35.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:35.280 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:35.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:35.381 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:35.427 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:35.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:35.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:35.529 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:35.559 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:35.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:35.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:35.620 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:35 smithi136 ceph-mon[109861]: pgmap v3026: 99 pgs: 99 active+clean; 3.3 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 107 MiB/s wr, 1.72k op/s 2024-08-17T18:14:35.620 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:35 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:14:35.620 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:35 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:14:35.620 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:35 smithi136 ceph-mon[115078]: pgmap v3026: 99 pgs: 99 active+clean; 3.3 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 107 MiB/s wr, 1.72k op/s 2024-08-17T18:14:35.620 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:35 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:14:35.620 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:35 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:14:35.632 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:14:35.632 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:14:35.633 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:14:35.661 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:35.689 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:35.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:35.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:35 smithi165 ceph-mon[105517]: pgmap v3026: 99 pgs: 99 active+clean; 3.3 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 107 MiB/s wr, 1.72k op/s 2024-08-17T18:14:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:35 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:14:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:35 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:14:35.778 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 28 16 61765 61749 137.814 93.9375 0.0096257 0.00725102 2024-08-17T18:14:35.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:35.823 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:35.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:35.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:35.925 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:35.966 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:35.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:35.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:36.068 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:36.107 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:36.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:36.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:36.209 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:36.248 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:36.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:36.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:36.349 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:36.386 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:36.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:36.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:36.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:36.523 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:36.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:36.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:36.624 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:36.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:36.666 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:36.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:14:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:14:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:14:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:36 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2520646359' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:14:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:14:36.768 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:36.778 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 29 16 63157 63141 136.061 87 0.0057961 0.00734488 2024-08-17T18:14:36.802 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:36.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:36.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:36.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:14:36.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:14:36.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:14:36.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:36 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2520646359' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:14:36.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:14:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:14:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:14:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:14:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:36 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2520646359' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:14:36.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:14:36.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:36.925 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:14:36.938 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:36.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:36.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:37.040 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:37.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:37.074 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:37.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:37.175 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:37.210 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:37.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:37.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:37.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:37.343 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:37.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:37.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:37.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:37.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:37.480 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:37.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:37.505 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:14:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:14:37] "GET /metrics HTTP/1.1" 200 36673 "" "Prometheus/2.51.0" 2024-08-17T18:14:37.581 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:37.602 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:37.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:37.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:37.704 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:37.741 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:37.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:37.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:37 smithi165 ceph-mon[105517]: pgmap v3027: 99 pgs: 99 active+clean; 3.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-08-17T18:14:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:37 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:14:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:37 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2520646359' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:14:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:37 smithi165 ceph-mon[105517]: osdmap e1614: 8 total, 8 up, 5 in 2024-08-17T18:14:37.778 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 30 16 64567 64551 134.463 88.125 0.00875387 0.00743194 2024-08-17T18:14:37.842 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:37.873 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:37.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:37.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:37.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:37 smithi136 ceph-mon[109861]: pgmap v3027: 99 pgs: 99 active+clean; 3.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-08-17T18:14:37.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:37 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:14:37.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:37 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2520646359' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:14:37.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:37 smithi136 ceph-mon[109861]: osdmap e1614: 8 total, 8 up, 5 in 2024-08-17T18:14:37.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:37 smithi136 ceph-mon[115078]: pgmap v3027: 99 pgs: 99 active+clean; 3.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-08-17T18:14:37.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:37 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:14:37.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:37 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2520646359' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:14:37.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:37 smithi136 ceph-mon[115078]: osdmap e1614: 8 total, 8 up, 5 in 2024-08-17T18:14:37.976 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:38.007 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:38.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:38.109 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:38.143 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:38.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:38.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:38.245 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:38.280 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:38.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:38.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:38.382 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:38.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:38.420 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:38.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:38.521 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:38.559 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:38.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:38.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:38.661 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:38.696 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:38.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:38.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:38.778 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 31 16 65981 65965 132.976 88.375 0.00765435 0.00751559 2024-08-17T18:14:38.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:38.828 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:38.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:38.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:38.930 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:38.962 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:38.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:38.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:39.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:39.101 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:39.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:39.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:39.203 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:39.241 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:39.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:39.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:39.343 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:39.379 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:39.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:39.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:39.481 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:39.517 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:39.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:39.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:39.618 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:39.648 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:39.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:39.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:39 smithi136 ceph-mon[109861]: pgmap v3029: 99 pgs: 99 active+clean; 3.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-08-17T18:14:39.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:39 smithi136 ceph-mon[115078]: pgmap v3029: 99 pgs: 99 active+clean; 3.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-08-17T18:14:39.751 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:39.778 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 32 16 67423 67407 131.637 90.125 0.00655075 0.00759128 2024-08-17T18:14:39.783 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:39.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:39.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:39.885 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:39.926 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:39.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:39.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:40.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:39 smithi165 ceph-mon[105517]: pgmap v3029: 99 pgs: 99 active+clean; 3.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-08-17T18:14:40.027 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:40.066 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:40.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:40.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:40.168 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:40.201 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:40.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:40.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:40.302 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:40.339 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:40.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:40.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:40.441 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:40.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:40.477 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:40.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:40.578 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:40.615 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:40.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:40.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:40.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:40.750 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:40.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:40.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:40.778 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 33 16 68851 68835 130.352 89.25 0.00504015 0.00766637 2024-08-17T18:14:40.852 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:40.891 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:40.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:40.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:40.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:41.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:41.031 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:41.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:41.132 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:41.173 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:41.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:41.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:41.275 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:41.307 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:41.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:41.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:41.409 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:41.449 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:41.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:41.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:41.551 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:41.587 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:41.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:41.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:41.690 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:41.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:41 smithi136 ceph-mon[109861]: pgmap v3030: 99 pgs: 99 active+clean; 3.9 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-08-17T18:14:41.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:41 smithi136 ceph-mon[115078]: pgmap v3030: 99 pgs: 99 active+clean; 3.9 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-08-17T18:14:41.721 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:41.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:41.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:41.779 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 34 16 70207 70191 129.011 84.75 0.0084546 0.00774625 2024-08-17T18:14:41.822 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:41.855 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:41.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:41.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:41.957 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:41.993 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:41.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:41.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:42.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:41 smithi165 ceph-mon[105517]: pgmap v3030: 99 pgs: 99 active+clean; 3.9 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-08-17T18:14:42.095 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:42.127 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:42.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:42.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:42.229 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:42.267 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:42.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:42.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:42.368 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:42.398 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:42.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:42.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:42.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:42.536 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:42.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:42.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:42.637 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:42.672 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:42.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:42.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:42.774 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:42.778 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 35 16 71637 71621 127.878 89.375 0.0131001 0.00781547 2024-08-17T18:14:42.814 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:42.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:42.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:42.916 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:42.947 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:42.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:42.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:43.049 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:43.085 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:43.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:43.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:43.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:43.225 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:43.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:43.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:43.327 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:43.363 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:43.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:43.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:43.465 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:43.495 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:43.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:43.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:43.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:43.632 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:43.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:43.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:43.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:43.765 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:43.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:43.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:43.777 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:43 smithi136 ceph-mon[109861]: pgmap v3031: 99 pgs: 99 active+clean; 4.1 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-08-17T18:14:43.777 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:43 smithi136 ceph-mon[115078]: pgmap v3031: 99 pgs: 99 active+clean; 4.1 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-08-17T18:14:43.778 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 36 16 72951 72935 126.607 82.125 0.0112608 0.0078935 2024-08-17T18:14:43.868 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:43.902 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:43.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:43.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:44.004 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:43 smithi165 ceph-mon[105517]: pgmap v3031: 99 pgs: 99 active+clean; 4.1 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-08-17T18:14:44.055 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:44.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:44.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:44.156 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:44.196 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:44.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:44.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:44.298 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:44.331 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:44.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:44.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:44.433 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:44.472 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:44.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:44.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:44.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:44.612 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:44.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:44.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:44.650 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:14:44.714 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:44.771 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:44.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:44.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:44.779 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 37 16 74340 74324 125.531 86.8125 0.00585664 0.00796131 2024-08-17T18:14:44.873 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:44.918 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:44.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:44.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:45.020 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:45.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:45.059 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:45.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:45.160 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:45.195 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:45.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:45.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:45.297 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:45.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:45.337 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:45.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:45.438 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:45.483 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:14:45.483 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:14:45.494 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:45.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:45.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:45.517 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:14:45.517 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:14:45.547 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:14:45.547 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:14:45.584 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:45 smithi136 ceph-mon[109861]: pgmap v3032: 99 pgs: 99 active+clean; 4.3 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-08-17T18:14:45.584 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:45 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1783373588' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:14:45.584 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:45 smithi136 ceph-mon[115078]: pgmap v3032: 99 pgs: 99 active+clean; 4.3 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-08-17T18:14:45.585 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:45 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1783373588' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:14:45.585 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:14:45.585 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:14:45.597 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:45.621 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:14:45.622 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:14:45.632 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:45.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:45.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:45.664 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:14:45.664 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:14:45.705 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:14:45.705 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:14:45.717 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:14:45.717 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-17T18:14:45.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-17T18:14:45.717 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:14:45.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.717 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.45626.0:170335 55.2 55:4037a7bd:::benchmark_data_smithi136_313476_object78349:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1515)", 2024-08-17T18:14:45.717 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:08:36.080758+0000", 2024-08-17T18:14:45.717 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 369.63342807800001, 2024-08-17T18:14:45.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0985168270000001, 2024-08-17T18:14:45.718 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:14:45.718 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:14:45.718 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:14:45.718 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:14:45.718 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.45626", 2024-08-17T18:14:45.718 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:14:45.718 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 170335 2024-08-17T18:14:45.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.718 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:14:45.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:14:45.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.080758+0000", 2024-08-17T18:14:45.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:14:45.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.080758+0000", 2024-08-17T18:14:45.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:14:45.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.080759+0000", 2024-08-17T18:14:45.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.372e-06 2024-08-17T18:14:45.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:14:45.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.080764+0000", 2024-08-17T18:14:45.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6450000000000004e-06 2024-08-17T18:14:45.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:14:45.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.080773+0000", 2024-08-17T18:14:45.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5580000000000006e-06 2024-08-17T18:14:45.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:14:45.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.080780+0000", 2024-08-17T18:14:45.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3409999999999999e-06 2024-08-17T18:14:45.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:14:45.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.080848+0000", 2024-08-17T18:14:45.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8758999999999997e-05 2024-08-17T18:14:45.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:14:45.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.080860+0000", 2024-08-17T18:14:45.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1907e-05 2024-08-17T18:14:45.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:14:45.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.059824+0000", 2024-08-17T18:14:45.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97896355000000002 2024-08-17T18:14:45.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:14:45.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.059838+0000", 2024-08-17T18:14:45.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3708e-05 2024-08-17T18:14:45.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:14:45.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.173362+0000", 2024-08-17T18:14:45.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.113524307 2024-08-17T18:14:45.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:14:45.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.173469+0000", 2024-08-17T18:14:45.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106801 2024-08-17T18:14:45.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5,6", 2024-08-17T18:14:45.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.173805+0000", 2024-08-17T18:14:45.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00033645600000000001 2024-08-17T18:14:45.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:14:45.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.175758+0000", 2024-08-17T18:14:45.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0019531589999999999 2024-08-17T18:14:45.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.178437+0000", 2024-08-17T18:14:45.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0026783800000000002 2024-08-17T18:14:45.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.178483+0000", 2024-08-17T18:14:45.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6755000000000002e-05 2024-08-17T18:14:45.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.179161+0000", 2024-08-17T18:14:45.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00067756200000000004 2024-08-17T18:14:45.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:14:45.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.179200+0000", 2024-08-17T18:14:45.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8868000000000001e-05 2024-08-17T18:14:45.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:14:45.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.179275+0000", 2024-08-17T18:14:45.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4698999999999997e-05 2024-08-17T18:14:45.725 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.725 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:14:45.725 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.726 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.45626.0:170347 55.2 55:4738474e:::benchmark_data_smithi136_313476_object78361:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1515)", 2024-08-17T18:14:45.726 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:08:36.094869+0000", 2024-08-17T18:14:45.726 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 369.61931707999997, 2024-08-17T18:14:45.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0887959270000001, 2024-08-17T18:14:45.726 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:14:45.726 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:14:45.726 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:14:45.726 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:14:45.726 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.45626", 2024-08-17T18:14:45.726 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:14:45.727 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 170347 2024-08-17T18:14:45.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.727 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:14:45.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:14:45.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.094869+0000", 2024-08-17T18:14:45.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:14:45.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.094869+0000", 2024-08-17T18:14:45.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:14:45.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.094870+0000", 2024-08-17T18:14:45.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6309999999999999e-06 2024-08-17T18:14:45.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:14:45.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.094877+0000", 2024-08-17T18:14:45.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2330000000000003e-06 2024-08-17T18:14:45.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:14:45.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.094901+0000", 2024-08-17T18:14:45.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4813000000000001e-05 2024-08-17T18:14:45.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:14:45.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.094911+0000", 2024-08-17T18:14:45.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3249999999999997e-06 2024-08-17T18:14:45.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:14:45.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.095035+0000", 2024-08-17T18:14:45.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000124554 2024-08-17T18:14:45.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:14:45.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.095046+0000", 2024-08-17T18:14:45.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1032999999999999e-05 2024-08-17T18:14:45.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:14:45.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.059862+0000", 2024-08-17T18:14:45.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.96481608299999999 2024-08-17T18:14:45.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:14:45.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.059870+0000", 2024-08-17T18:14:45.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2080000000000003e-06 2024-08-17T18:14:45.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:14:45.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.175786+0000", 2024-08-17T18:14:45.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1159161850000001 2024-08-17T18:14:45.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:14:45.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.175914+0000", 2024-08-17T18:14:45.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012778600000000001 2024-08-17T18:14:45.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5,6", 2024-08-17T18:14:45.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.176280+0000", 2024-08-17T18:14:45.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036621700000000001 2024-08-17T18:14:45.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:14:45.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.178500+0000", 2024-08-17T18:14:45.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0022198169999999998 2024-08-17T18:14:45.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.178869+0000", 2024-08-17T18:14:45.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000369253 2024-08-17T18:14:45.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.179313+0000", 2024-08-17T18:14:45.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044423500000000002 2024-08-17T18:14:45.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.183487+0000", 2024-08-17T18:14:45.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0041739500000000001 2024-08-17T18:14:45.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:14:45.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.183547+0000", 2024-08-17T18:14:45.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0025000000000002e-05 2024-08-17T18:14:45.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:14:45.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.183665+0000", 2024-08-17T18:14:45.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011757899999999999 2024-08-17T18:14:45.735 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.735 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:14:45.735 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.735 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:27947 56.4 56:21cbfe32:::benchmark_data_smithi136_356456_object27946:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1540)", 2024-08-17T18:14:45.735 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:09:49.432911+0000", 2024-08-17T18:14:45.735 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 296.281274999, 2024-08-17T18:14:45.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.140233625, 2024-08-17T18:14:45.736 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:14:45.736 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:14:45.736 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:14:45.736 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:14:45.736 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:14:45.736 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:14:45.736 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 27947 2024-08-17T18:14:45.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.736 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:14:45.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:14:45.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.432911+0000", 2024-08-17T18:14:45.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:14:45.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.432911+0000", 2024-08-17T18:14:45.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:14:45.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.432911+0000", 2024-08-17T18:14:45.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9100000000000004e-07 2024-08-17T18:14:45.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:14:45.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.432934+0000", 2024-08-17T18:14:45.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2920000000000001e-05 2024-08-17T18:14:45.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:14:45.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.432943+0000", 2024-08-17T18:14:45.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2660000000000001e-06 2024-08-17T18:14:45.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:14:45.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.432949+0000", 2024-08-17T18:14:45.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3040000000000001e-06 2024-08-17T18:14:45.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:14:45.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.435502+0000", 2024-08-17T18:14:45.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025528130000000001 2024-08-17T18:14:45.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:14:45.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.435633+0000", 2024-08-17T18:14:45.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013102 2024-08-17T18:14:45.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4", 2024-08-17T18:14:45.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.435772+0000", 2024-08-17T18:14:45.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013916500000000001 2024-08-17T18:14:45.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:14:45.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.440445+0000", 2024-08-17T18:14:45.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0046734710000000002 2024-08-17T18:14:45.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.440508+0000", 2024-08-17T18:14:45.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2648000000000001e-05 2024-08-17T18:14:45.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573031+0000", 2024-08-17T18:14:45.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13252335000000001 2024-08-17T18:14:45.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:14:45.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573092+0000", 2024-08-17T18:14:45.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0747e-05 2024-08-17T18:14:45.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:14:45.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573144+0000", 2024-08-17T18:14:45.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2330000000000002e-05 2024-08-17T18:14:45.741 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.741 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:14:45.741 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.742 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:27951 56.4 56:222eaaba:::benchmark_data_smithi136_356456_object27950:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1540)", 2024-08-17T18:14:45.742 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:09:49.433998+0000", 2024-08-17T18:14:45.742 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 296.28018731399999, 2024-08-17T18:14:45.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13954208600000001, 2024-08-17T18:14:45.742 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:14:45.742 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:14:45.742 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:14:45.742 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:14:45.742 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:14:45.742 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:14:45.742 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 27951 2024-08-17T18:14:45.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.743 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:14:45.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:14:45.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.433998+0000", 2024-08-17T18:14:45.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:14:45.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.433998+0000", 2024-08-17T18:14:45.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:14:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.433999+0000", 2024-08-17T18:14:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0400000000000005e-07 2024-08-17T18:14:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:14:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434024+0000", 2024-08-17T18:14:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.491e-05 2024-08-17T18:14:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:14:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434035+0000", 2024-08-17T18:14:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1129e-05 2024-08-17T18:14:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:14:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434043+0000", 2024-08-17T18:14:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3069999999999997e-06 2024-08-17T18:14:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:14:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.437081+0000", 2024-08-17T18:14:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0030388279999999999 2024-08-17T18:14:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:14:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.437217+0000", 2024-08-17T18:14:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013517800000000001 2024-08-17T18:14:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4", 2024-08-17T18:14:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.437342+0000", 2024-08-17T18:14:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125091 2024-08-17T18:14:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:14:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.440518+0000", 2024-08-17T18:14:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003176636 2024-08-17T18:14:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.440831+0000", 2024-08-17T18:14:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00031303800000000002 2024-08-17T18:14:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573416+0000", 2024-08-17T18:14:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.132585077 2024-08-17T18:14:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:14:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573469+0000", 2024-08-17T18:14:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2899000000000003e-05 2024-08-17T18:14:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:14:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573541+0000", 2024-08-17T18:14:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1188999999999999e-05 2024-08-17T18:14:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:14:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:27954 56.4 56:216fadd1:::benchmark_data_smithi136_356456_object27953:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1540)", 2024-08-17T18:14:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:09:49.434097+0000", 2024-08-17T18:14:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 296.28008891399998, 2024-08-17T18:14:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13963388299999999, 2024-08-17T18:14:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:14:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:14:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:14:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:14:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:14:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:14:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 27954 2024-08-17T18:14:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:14:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:14:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434097+0000", 2024-08-17T18:14:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:14:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434097+0000", 2024-08-17T18:14:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:14:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434097+0000", 2024-08-17T18:14:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7800000000000002e-07 2024-08-17T18:14:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:14:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434118+0000", 2024-08-17T18:14:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1015000000000001e-05 2024-08-17T18:14:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:14:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434126+0000", 2024-08-17T18:14:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4490000000000002e-06 2024-08-17T18:14:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:14:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434131+0000", 2024-08-17T18:14:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8470000000000003e-06 2024-08-17T18:14:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:14:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.438432+0000", 2024-08-17T18:14:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0043018010000000001 2024-08-17T18:14:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:14:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439024+0000", 2024-08-17T18:14:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00059143900000000003 2024-08-17T18:14:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4", 2024-08-17T18:14:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439189+0000", 2024-08-17T18:14:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016538999999999999 2024-08-17T18:14:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.440867+0000", 2024-08-17T18:14:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016782069999999999 2024-08-17T18:14:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:14:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573554+0000", 2024-08-17T18:14:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13268660700000001 2024-08-17T18:14:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573637+0000", 2024-08-17T18:14:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3016999999999995e-05 2024-08-17T18:14:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:14:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573686+0000", 2024-08-17T18:14:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9038e-05 2024-08-17T18:14:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:14:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573731+0000", 2024-08-17T18:14:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4595000000000002e-05 2024-08-17T18:14:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:14:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:27955 56.4 56:2138118c:::benchmark_data_smithi136_356456_object27954:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1540)", 2024-08-17T18:14:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:09:49.434139+0000", 2024-08-17T18:14:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 296.28004711300002, 2024-08-17T18:14:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13974101799999999, 2024-08-17T18:14:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:14:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:14:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:14:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:14:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:14:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:14:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 27955 2024-08-17T18:14:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:14:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:14:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434139+0000", 2024-08-17T18:14:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:14:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434139+0000", 2024-08-17T18:14:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:14:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434139+0000", 2024-08-17T18:14:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7899999999999998e-07 2024-08-17T18:14:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:14:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434160+0000", 2024-08-17T18:14:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0937999999999998e-05 2024-08-17T18:14:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:14:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434166+0000", 2024-08-17T18:14:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1279999999999996e-06 2024-08-17T18:14:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:14:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434172+0000", 2024-08-17T18:14:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3789999999999997e-06 2024-08-17T18:14:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:14:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.441201+0000", 2024-08-17T18:14:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0070293500000000002 2024-08-17T18:14:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:14:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.441417+0000", 2024-08-17T18:14:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000215813 2024-08-17T18:14:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4", 2024-08-17T18:14:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.441553+0000", 2024-08-17T18:14:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013591300000000001 2024-08-17T18:14:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:14:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573560+0000", 2024-08-17T18:14:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.132007491 2024-08-17T18:14:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573600+0000", 2024-08-17T18:14:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9687999999999997e-05 2024-08-17T18:14:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573801+0000", 2024-08-17T18:14:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020095200000000001 2024-08-17T18:14:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:14:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573845+0000", 2024-08-17T18:14:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4202999999999997e-05 2024-08-17T18:14:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:14:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573880+0000", 2024-08-17T18:14:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4584e-05 2024-08-17T18:14:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:14:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:27962 56.f 56:faa63ae2:::benchmark_data_smithi136_356456_object27961:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1540)", 2024-08-17T18:14:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:09:49.438972+0000", 2024-08-17T18:14:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 296.27521342, 2024-08-17T18:14:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.138606172, 2024-08-17T18:14:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:14:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:14:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:14:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:14:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:14:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:14:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 27962 2024-08-17T18:14:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:14:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:14:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.438972+0000", 2024-08-17T18:14:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:14:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.438972+0000", 2024-08-17T18:14:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:14:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.438973+0000", 2024-08-17T18:14:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5600000000000005e-07 2024-08-17T18:14:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:14:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.438996+0000", 2024-08-17T18:14:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3244e-05 2024-08-17T18:14:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:14:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439004+0000", 2024-08-17T18:14:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7489999999999998e-06 2024-08-17T18:14:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:14:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439011+0000", 2024-08-17T18:14:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5080000000000002e-06 2024-08-17T18:14:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:14:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439253+0000", 2024-08-17T18:14:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000242105 2024-08-17T18:14:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:14:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439372+0000", 2024-08-17T18:14:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119466 2024-08-17T18:14:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,6", 2024-08-17T18:14:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439497+0000", 2024-08-17T18:14:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012526700000000001 2024-08-17T18:14:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.574682+0000", 2024-08-17T18:14:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13518470399999999 2024-08-17T18:14:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.574727+0000", 2024-08-17T18:14:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4703999999999998e-05 2024-08-17T18:14:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:14:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.577465+0000", 2024-08-17T18:14:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0027381950000000001 2024-08-17T18:14:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:14:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.577529+0000", 2024-08-17T18:14:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4333999999999998e-05 2024-08-17T18:14:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:14:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.577579+0000", 2024-08-17T18:14:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.914e-05 2024-08-17T18:14:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:14:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:27966 56.f 56:f201f13f:::benchmark_data_smithi136_356456_object27965:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1540)", 2024-08-17T18:14:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:09:49.439427+0000", 2024-08-17T18:14:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 296.27475857899998, 2024-08-17T18:14:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13837285699999999, 2024-08-17T18:14:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:14:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:14:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:14:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:14:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:14:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:14:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 27966 2024-08-17T18:14:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:14:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:14:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439427+0000", 2024-08-17T18:14:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:14:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439427+0000", 2024-08-17T18:14:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:14:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439428+0000", 2024-08-17T18:14:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.54e-07 2024-08-17T18:14:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:14:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439459+0000", 2024-08-17T18:14:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0703000000000001e-05 2024-08-17T18:14:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:14:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439469+0000", 2024-08-17T18:14:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8730000000000002e-06 2024-08-17T18:14:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:14:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439475+0000", 2024-08-17T18:14:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4779999999999996e-06 2024-08-17T18:14:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:14:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.441807+0000", 2024-08-17T18:14:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023322680000000002 2024-08-17T18:14:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:14:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.441929+0000", 2024-08-17T18:14:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121724 2024-08-17T18:14:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,6", 2024-08-17T18:14:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573147+0000", 2024-08-17T18:14:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13121764799999999 2024-08-17T18:14:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.577267+0000", 2024-08-17T18:14:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0041202089999999997 2024-08-17T18:14:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.577451+0000", 2024-08-17T18:14:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018379400000000001 2024-08-17T18:14:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:14:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.577585+0000", 2024-08-17T18:14:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013449499999999999 2024-08-17T18:14:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:14:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.577625+0000", 2024-08-17T18:14:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0074000000000003e-05 2024-08-17T18:14:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:14:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.577800+0000", 2024-08-17T18:14:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017483700000000001 2024-08-17T18:14:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:14:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.774 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:88733 56.f 56:fb6e25d6:::benchmark_data_smithi136_356456_object88732:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1544)", 2024-08-17T18:14:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.476787+0000", 2024-08-17T18:14:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 254.237398794, 2024-08-17T18:14:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17365398000000001, 2024-08-17T18:14:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:14:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:14:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:14:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:14:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:14:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:14:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 88733 2024-08-17T18:14:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:14:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:14:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.476787+0000", 2024-08-17T18:14:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:14:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.476787+0000", 2024-08-17T18:14:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:14:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.476788+0000", 2024-08-17T18:14:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8999999999999995e-07 2024-08-17T18:14:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:14:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.476809+0000", 2024-08-17T18:14:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1331999999999998e-05 2024-08-17T18:14:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:14:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.476819+0000", 2024-08-17T18:14:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9799999999999993e-06 2024-08-17T18:14:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:14:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.476826+0000", 2024-08-17T18:14:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.097e-06 2024-08-17T18:14:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:14:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.476868+0000", 2024-08-17T18:14:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2256000000000003e-05 2024-08-17T18:14:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:14:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.477009+0000", 2024-08-17T18:14:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014094800000000001 2024-08-17T18:14:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,6", 2024-08-17T18:14:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.477144+0000", 2024-08-17T18:14:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013450699999999999 2024-08-17T18:14:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.481751+0000", 2024-08-17T18:14:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0046066379999999997 2024-08-17T18:14:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:14:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.481761+0000", 2024-08-17T18:14:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0548e-05 2024-08-17T18:14:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650331+0000", 2024-08-17T18:14:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16856977100000001 2024-08-17T18:14:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:14:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650393+0000", 2024-08-17T18:14:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1925000000000001e-05 2024-08-17T18:14:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:14:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650441+0000", 2024-08-17T18:14:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8188e-05 2024-08-17T18:14:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:14:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:88735 56.c 56:3d1600ab:::benchmark_data_smithi136_356456_object88734:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1544)", 2024-08-17T18:14:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.480738+0000", 2024-08-17T18:14:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 254.233447783, 2024-08-17T18:14:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17329349599999999, 2024-08-17T18:14:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:14:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:14:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:14:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:14:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:14:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:14:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 88735 2024-08-17T18:14:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:14:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:14:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.480738+0000", 2024-08-17T18:14:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:14:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.480738+0000", 2024-08-17T18:14:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:14:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.480740+0000", 2024-08-17T18:14:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5710000000000001e-06 2024-08-17T18:14:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:14:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.480762+0000", 2024-08-17T18:14:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2629999999999998e-05 2024-08-17T18:14:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:14:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.480775+0000", 2024-08-17T18:14:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3027000000000001e-05 2024-08-17T18:14:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:14:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.480783+0000", 2024-08-17T18:14:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4989999999999997e-06 2024-08-17T18:14:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:14:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.480844+0000", 2024-08-17T18:14:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1001000000000001e-05 2024-08-17T18:14:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:14:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.480988+0000", 2024-08-17T18:14:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000144723 2024-08-17T18:14:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:14:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.481128+0000", 2024-08-17T18:14:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013966599999999999 2024-08-17T18:14:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484806+0000", 2024-08-17T18:14:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036775620000000001 2024-08-17T18:14:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:14:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.485844+0000", 2024-08-17T18:14:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001037888 2024-08-17T18:14:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653923+0000", 2024-08-17T18:14:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16807926300000001 2024-08-17T18:14:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:14:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653983+0000", 2024-08-17T18:14:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9837999999999999e-05 2024-08-17T18:14:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:14:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.654031+0000", 2024-08-17T18:14:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8828000000000001e-05 2024-08-17T18:14:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:14:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:88739 56.f 56:f7ad6e62:::benchmark_data_smithi136_356456_object88738:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1544)", 2024-08-17T18:14:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.484508+0000", 2024-08-17T18:14:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 254.22967771099999, 2024-08-17T18:14:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.166084435, 2024-08-17T18:14:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:14:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:14:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:14:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:14:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:14:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:14:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 88739 2024-08-17T18:14:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:14:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.790 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:14:45.790 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:14:45.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:14:45.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484508+0000", 2024-08-17T18:14:45.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:14:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484508+0000", 2024-08-17T18:14:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:14:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484510+0000", 2024-08-17T18:14:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.592e-06 2024-08-17T18:14:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:14:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484536+0000", 2024-08-17T18:14:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6548999999999999e-05 2024-08-17T18:14:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:14:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484550+0000", 2024-08-17T18:14:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3585e-05 2024-08-17T18:14:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:14:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484558+0000", 2024-08-17T18:14:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.405e-06 2024-08-17T18:14:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:14:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484709+0000", 2024-08-17T18:14:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015043899999999999 2024-08-17T18:14:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:14:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484884+0000", 2024-08-17T18:14:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000175613 2024-08-17T18:14:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,6", 2024-08-17T18:14:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.485059+0000", 2024-08-17T18:14:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017479999999999999 2024-08-17T18:14:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650489+0000", 2024-08-17T18:14:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.165430095 2024-08-17T18:14:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:14:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650498+0000", 2024-08-17T18:14:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9800000000000004e-06 2024-08-17T18:14:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650524+0000", 2024-08-17T18:14:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5998000000000001e-05 2024-08-17T18:14:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:14:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650559+0000", 2024-08-17T18:14:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4718999999999997e-05 2024-08-17T18:14:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:14:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650592+0000", 2024-08-17T18:14:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.366e-05 2024-08-17T18:14:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:14:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:88741 56.e 56:77ba4448:::benchmark_data_smithi136_356456_object88740:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1544)", 2024-08-17T18:14:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.484583+0000", 2024-08-17T18:14:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 254.22960244800001, 2024-08-17T18:14:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16996449199999999, 2024-08-17T18:14:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:14:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:14:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:14:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:14:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:14:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:14:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 88741 2024-08-17T18:14:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:14:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:14:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484583+0000", 2024-08-17T18:14:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:14:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484583+0000", 2024-08-17T18:14:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:14:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484584+0000", 2024-08-17T18:14:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5700000000000002e-07 2024-08-17T18:14:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:14:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484604+0000", 2024-08-17T18:14:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9806999999999999e-05 2024-08-17T18:14:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:14:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484612+0000", 2024-08-17T18:14:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3820000000000004e-06 2024-08-17T18:14:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:14:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484618+0000", 2024-08-17T18:14:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8300000000000001e-06 2024-08-17T18:14:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:14:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.485895+0000", 2024-08-17T18:14:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012773629999999999 2024-08-17T18:14:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:14:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486015+0000", 2024-08-17T18:14:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119335 2024-08-17T18:14:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,6", 2024-08-17T18:14:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486131+0000", 2024-08-17T18:14:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011650999999999999 2024-08-17T18:14:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653049+0000", 2024-08-17T18:14:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.166917811 2024-08-17T18:14:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653095+0000", 2024-08-17T18:14:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6360999999999999e-05 2024-08-17T18:14:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:14:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.654443+0000", 2024-08-17T18:14:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0013480129999999999 2024-08-17T18:14:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:14:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.654495+0000", 2024-08-17T18:14:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2076000000000001e-05 2024-08-17T18:14:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:14:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.654548+0000", 2024-08-17T18:14:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2447000000000001e-05 2024-08-17T18:14:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:14:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:88746 56.c 56:31d7e917:::benchmark_data_smithi136_356456_object88745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1544)", 2024-08-17T18:14:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.486034+0000", 2024-08-17T18:14:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 254.22815157400001, 2024-08-17T18:14:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.176589883, 2024-08-17T18:14:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:14:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:14:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:14:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:14:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:14:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:14:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 88746 2024-08-17T18:14:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:14:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:14:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486034+0000", 2024-08-17T18:14:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:14:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486034+0000", 2024-08-17T18:14:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:14:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486035+0000", 2024-08-17T18:14:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9099999999999991e-07 2024-08-17T18:14:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:14:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486057+0000", 2024-08-17T18:14:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.143e-05 2024-08-17T18:14:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:14:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486067+0000", 2024-08-17T18:14:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0572e-05 2024-08-17T18:14:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:14:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486074+0000", 2024-08-17T18:14:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5780000000000004e-06 2024-08-17T18:14:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:14:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650549+0000", 2024-08-17T18:14:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.164474957 2024-08-17T18:14:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:14:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650720+0000", 2024-08-17T18:14:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017162199999999999 2024-08-17T18:14:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:14:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650849+0000", 2024-08-17T18:14:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012822300000000001 2024-08-17T18:14:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:14:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.662108+0000", 2024-08-17T18:14:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011259725 2024-08-17T18:14:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.662194+0000", 2024-08-17T18:14:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6069999999999994e-05 2024-08-17T18:14:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.662434+0000", 2024-08-17T18:14:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000240103 2024-08-17T18:14:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:14:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.662489+0000", 2024-08-17T18:14:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4880000000000003e-05 2024-08-17T18:14:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:14:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.662624+0000", 2024-08-17T18:14:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013473200000000001 2024-08-17T18:14:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:14:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:88747 56.c 56:3c73eda4:::benchmark_data_smithi136_356456_object88746:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1544)", 2024-08-17T18:14:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.486091+0000", 2024-08-17T18:14:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 254.22809497, 2024-08-17T18:14:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.180009683, 2024-08-17T18:14:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:14:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:14:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:14:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:14:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:14:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:14:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 88747 2024-08-17T18:14:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:14:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:14:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486091+0000", 2024-08-17T18:14:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:14:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486091+0000", 2024-08-17T18:14:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:14:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486091+0000", 2024-08-17T18:14:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2e-07 2024-08-17T18:14:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:14:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486110+0000", 2024-08-17T18:14:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8410000000000002e-05 2024-08-17T18:14:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:14:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486117+0000", 2024-08-17T18:14:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1049999999999997e-06 2024-08-17T18:14:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:14:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486122+0000", 2024-08-17T18:14:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8119999999999998e-06 2024-08-17T18:14:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:14:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.654390+0000", 2024-08-17T18:14:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16826818199999999 2024-08-17T18:14:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:14:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.655078+0000", 2024-08-17T18:14:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00068772199999999999 2024-08-17T18:14:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:14:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.655223+0000", 2024-08-17T18:14:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000145167 2024-08-17T18:14:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.662237+0000", 2024-08-17T18:14:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0070139419999999996 2024-08-17T18:14:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.666004+0000", 2024-08-17T18:14:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037672869999999998 2024-08-17T18:14:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:14:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.666014+0000", 2024-08-17T18:14:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0484e-05 2024-08-17T18:14:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:14:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.666062+0000", 2024-08-17T18:14:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7919e-05 2024-08-17T18:14:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:14:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.666100+0000", 2024-08-17T18:14:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8133000000000002e-05 2024-08-17T18:14:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:14:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.46025.0:88733 56.f e1544/1535 ondisk, result = 0)", 2024-08-17T18:14:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.486213+0000", 2024-08-17T18:14:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 254.22797310499999, 2024-08-17T18:14:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16423845200000001, 2024-08-17T18:14:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:14:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:14:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T18:14:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:14:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:14:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486213+0000", 2024-08-17T18:14:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:14:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486213+0000", 2024-08-17T18:14:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:14:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486213+0000", 2024-08-17T18:14:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1999999999999999e-07 2024-08-17T18:14:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:14:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486223+0000", 2024-08-17T18:14:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9890000000000004e-06 2024-08-17T18:14:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:14:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486234+0000", 2024-08-17T18:14:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0883999999999999e-05 2024-08-17T18:14:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:14:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486241+0000", 2024-08-17T18:14:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6780000000000003e-06 2024-08-17T18:14:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:14:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650315+0000", 2024-08-17T18:14:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16407380599999999 2024-08-17T18:14:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:14:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650324+0000", 2024-08-17T18:14:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0799999999999995e-06 2024-08-17T18:14:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:14:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650451+0000", 2024-08-17T18:14:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012729500000000001 2024-08-17T18:14:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:14:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:88748 56.a 56:5de94025:::benchmark_data_smithi136_356456_object88747:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1544)", 2024-08-17T18:14:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.486267+0000", 2024-08-17T18:14:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 254.22791845500001, 2024-08-17T18:14:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17806833399999999, 2024-08-17T18:14:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:14:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:14:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:14:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:14:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:14:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:14:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 88748 2024-08-17T18:14:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:14:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:14:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486267+0000", 2024-08-17T18:14:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:14:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486267+0000", 2024-08-17T18:14:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:14:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486268+0000", 2024-08-17T18:14:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2799999999999995e-07 2024-08-17T18:14:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:14:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486290+0000", 2024-08-17T18:14:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2306999999999998e-05 2024-08-17T18:14:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:14:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486299+0000", 2024-08-17T18:14:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0860000000000002e-06 2024-08-17T18:14:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:14:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486305+0000", 2024-08-17T18:14:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5189999999999998e-06 2024-08-17T18:14:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:14:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486361+0000", 2024-08-17T18:14:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5878999999999998e-05 2024-08-17T18:14:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:14:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486474+0000", 2024-08-17T18:14:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112804 2024-08-17T18:14:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,6", 2024-08-17T18:14:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650225+0000", 2024-08-17T18:14:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16375092599999999 2024-08-17T18:14:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.660909+0000", 2024-08-17T18:14:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010684206 2024-08-17T18:14:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.660952+0000", 2024-08-17T18:14:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3374999999999997e-05 2024-08-17T18:14:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:14:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.664236+0000", 2024-08-17T18:14:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032840529999999999 2024-08-17T18:14:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:14:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.664294+0000", 2024-08-17T18:14:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7621999999999998e-05 2024-08-17T18:14:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:14:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.664336+0000", 2024-08-17T18:14:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1729e-05 2024-08-17T18:14:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:14:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.46025.0:88735 56.c e1544/1535 ondisk, result = 0)", 2024-08-17T18:14:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.486671+0000", 2024-08-17T18:14:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 254.227514937, 2024-08-17T18:14:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.167366402, 2024-08-17T18:14:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:14:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:14:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T18:14:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:14:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:14:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486671+0000", 2024-08-17T18:14:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:14:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486671+0000", 2024-08-17T18:14:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:14:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486671+0000", 2024-08-17T18:14:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.13e-07 2024-08-17T18:14:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:14:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486675+0000", 2024-08-17T18:14:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5190000000000001e-06 2024-08-17T18:14:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:14:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650207+0000", 2024-08-17T18:14:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.163532289 2024-08-17T18:14:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:14:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650216+0000", 2024-08-17T18:14:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0389999999999993e-06 2024-08-17T18:14:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:14:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653909+0000", 2024-08-17T18:14:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036931970000000001 2024-08-17T18:14:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:14:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653917+0000", 2024-08-17T18:14:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4580000000000004e-06 2024-08-17T18:14:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:14:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.654037+0000", 2024-08-17T18:14:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120287 2024-08-17T18:14:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:14:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.46025.0:88741 56.e e1544/1535 ondisk, result = 0)", 2024-08-17T18:14:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.489475+0000", 2024-08-17T18:14:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 254.224710292, 2024-08-17T18:14:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.163581478, 2024-08-17T18:14:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:14:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:14:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T18:14:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:14:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:14:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.489475+0000", 2024-08-17T18:14:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:14:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.489475+0000", 2024-08-17T18:14:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:14:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.489477+0000", 2024-08-17T18:14:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.032e-06 2024-08-17T18:14:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:14:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.489482+0000", 2024-08-17T18:14:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4090000000000002e-06 2024-08-17T18:14:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:14:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650118+0000", 2024-08-17T18:14:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.160635842 2024-08-17T18:14:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:14:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650133+0000", 2024-08-17T18:14:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5534000000000001e-05 2024-08-17T18:14:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:14:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653033+0000", 2024-08-17T18:14:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0028997620000000002 2024-08-17T18:14:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.834 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 38 16 75711 75695 124.482 85.6875 0.0108456 0.00802852 2024-08-17T18:14:45.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:14:45.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653042+0000", 2024-08-17T18:14:45.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3810000000000005e-06 2024-08-17T18:14:45.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:14:45.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653057+0000", 2024-08-17T18:14:45.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4518000000000001e-05 2024-08-17T18:14:45.835 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.835 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:14:45.835 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.835 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:172837 56.e 56:7f6a0292:::benchmark_data_smithi136_356456_object46133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1551)", 2024-08-17T18:14:45.835 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:11:57.615790+0000", 2024-08-17T18:14:45.835 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 168.09839605600001, 2024-08-17T18:14:45.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078513984999999994, 2024-08-17T18:14:45.836 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:14:45.836 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:14:45.836 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:14:45.836 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:14:45.836 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:14:45.836 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:14:45.836 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 172837 2024-08-17T18:14:45.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.836 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:14:45.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:14:45.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.615790+0000", 2024-08-17T18:14:45.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:14:45.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.615790+0000", 2024-08-17T18:14:45.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:14:45.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.615791+0000", 2024-08-17T18:14:45.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4130000000000001e-06 2024-08-17T18:14:45.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:14:45.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.615796+0000", 2024-08-17T18:14:45.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4959999999999998e-06 2024-08-17T18:14:45.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:14:45.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.615808+0000", 2024-08-17T18:14:45.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1917e-05 2024-08-17T18:14:45.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:14:45.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.615815+0000", 2024-08-17T18:14:45.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9070000000000007e-06 2024-08-17T18:14:45.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:14:45.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.625307+0000", 2024-08-17T18:14:45.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0094920149999999995 2024-08-17T18:14:45.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:14:45.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.677360+0000", 2024-08-17T18:14:45.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.052052534999999997 2024-08-17T18:14:45.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,6", 2024-08-17T18:14:45.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.677480+0000", 2024-08-17T18:14:45.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119876 2024-08-17T18:14:45.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.690420+0000", 2024-08-17T18:14:45.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012940579000000001 2024-08-17T18:14:45.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:14:45.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.694117+0000", 2024-08-17T18:14:45.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003696595 2024-08-17T18:14:45.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.694203+0000", 2024-08-17T18:14:45.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6070999999999996e-05 2024-08-17T18:14:45.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:14:45.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.694258+0000", 2024-08-17T18:14:45.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4712000000000001e-05 2024-08-17T18:14:45.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:14:45.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.694304+0000", 2024-08-17T18:14:45.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5868999999999997e-05 2024-08-17T18:14:45.841 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.842 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:14:45.842 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.842 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:172847 56.e 56:7d8be824:::benchmark_data_smithi136_356456_object46143:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1551)", 2024-08-17T18:14:45.842 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:11:57.626997+0000", 2024-08-17T18:14:45.842 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 168.08718899600001, 2024-08-17T18:14:45.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.074151992999999999, 2024-08-17T18:14:45.842 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:14:45.842 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:14:45.842 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:14:45.842 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:14:45.842 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:14:45.842 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:14:45.843 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 172847 2024-08-17T18:14:45.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.843 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:14:45.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:14:45.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.626997+0000", 2024-08-17T18:14:45.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:14:45.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.626997+0000", 2024-08-17T18:14:45.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:14:45.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:14:45.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.626998+0000", 2024-08-17T18:14:45.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2929999999999999e-06 2024-08-17T18:14:45.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:14:45.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.627003+0000", 2024-08-17T18:14:45.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0139999999999998e-06 2024-08-17T18:14:45.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:14:45.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.676155+0000", 2024-08-17T18:14:45.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.049152108999999999 2024-08-17T18:14:45.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:14:45.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.676180+0000", 2024-08-17T18:14:45.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4691000000000001e-05 2024-08-17T18:14:45.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:14:45.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.682649+0000", 2024-08-17T18:14:45.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0064692400000000002 2024-08-17T18:14:45.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:14:45.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.682815+0000", 2024-08-17T18:14:45.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000165778 2024-08-17T18:14:45.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,6", 2024-08-17T18:14:45.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.683064+0000", 2024-08-17T18:14:45.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024944300000000002 2024-08-17T18:14:45.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:14:45.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.694122+0000", 2024-08-17T18:14:45.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011058047 2024-08-17T18:14:45.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.694160+0000", 2024-08-17T18:14:45.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8061e-05 2024-08-17T18:14:45.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:14:45.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.701028+0000", 2024-08-17T18:14:45.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0068671380000000001 2024-08-17T18:14:45.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:14:45.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.701098+0000", 2024-08-17T18:14:45.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0215000000000005e-05 2024-08-17T18:14:45.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:14:45.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:14:45.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:14:45.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.701149+0000", 2024-08-17T18:14:45.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0964000000000002e-05 2024-08-17T18:14:45.848 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.848 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:14:45.848 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.848 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:14:45.848 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:14:45.848 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:14:45.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:45.850 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:45.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:45.951 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:45.989 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:45.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:45.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:46.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:45 smithi165 ceph-mon[105517]: pgmap v3032: 99 pgs: 99 active+clean; 4.3 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-08-17T18:14:46.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:45 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1783373588' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:14:46.091 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:46.133 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:46.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:46.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:46.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:46.267 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:46.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:46.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:46.369 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:46.397 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:46.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:46.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:46.499 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:46.534 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:46.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:46.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:46.544 INFO:teuthology.orchestra.run.smithi136.stderr:set pool 57 pg_num to 26 2024-08-17T18:14:46.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:46.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:46.678 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:46.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:46.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:46.780 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 39 16 77047 77031 123.431 83.5 0.00345045 0.00809659 2024-08-17T18:14:46.806 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:46.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:46.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:46.907 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:46.945 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:46.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:46 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/723483563' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T18:14:46.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:46 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T18:14:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:46 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/723483563' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T18:14:46.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:46 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T18:14:46.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:46.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:47.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:46 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/723483563' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T18:14:47.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:46 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-08-17T18:14:47.040 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-17T18:14:47.048 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:47.128 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:47.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:47.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:47.231 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:47.270 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:47.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:47.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:47.372 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:47.408 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:47.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:47.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:47.510 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:47.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:47.552 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:47.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:47.653 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:47.684 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:47.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:47.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:47 smithi136 ceph-mon[109861]: pgmap v3033: 99 pgs: 99 active+clean; 4.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-08-17T18:14:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:47 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-08-17T18:14:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:47 smithi136 ceph-mon[109861]: osdmap e1615: 8 total, 8 up, 5 in 2024-08-17T18:14:47.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:47 smithi136 ceph-mon[115078]: pgmap v3033: 99 pgs: 99 active+clean; 4.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-08-17T18:14:47.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:47 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-08-17T18:14:47.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:47 smithi136 ceph-mon[115078]: osdmap e1615: 8 total, 8 up, 5 in 2024-08-17T18:14:47.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:14:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:14:47] "GET /metrics HTTP/1.1" 200 36673 "" "Prometheus/2.51.0" 2024-08-17T18:14:47.779 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:2024-08-17T18:14:47.779597+0000 min lat: 0.00125927 max lat: 0.125939 avg lat: 0.0081599 2024-08-17T18:14:47.779 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T18:14:47.779 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 40 16 78413 78397 122.48 85.375 0.00846013 0.0081599 2024-08-17T18:14:47.787 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:47.817 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:47.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:47.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:47.919 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:47.955 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:47.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:47.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:47 smithi165 ceph-mon[105517]: pgmap v3033: 99 pgs: 99 active+clean; 4.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-08-17T18:14:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:47 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-08-17T18:14:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:47 smithi165 ceph-mon[105517]: osdmap e1615: 8 total, 8 up, 5 in 2024-08-17T18:14:48.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:48.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:48.089 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:48.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:48.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:48.225 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:48.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:48.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:48.326 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:48.363 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:48.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:48.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:48.465 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:48.499 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:48.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:48.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:48.585 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:14:48.599 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:14:48.601 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:48.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:48.680 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:48.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:48.763 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:48 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-08-17T18:14:48.763 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:48 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-08-17T18:14:48.781 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:48.782 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 41 16 79445 79429 121.065 64.5 0.187441 0.00821457 2024-08-17T18:14:48.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:48.829 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:48.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:48.930 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:48.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:48.952 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:48.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:49.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:48 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-08-17T18:14:49.053 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:49.076 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:49.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:49.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:49.178 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:49.199 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:49.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:49.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:49.300 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:49.324 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:49.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:49.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:49.425 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:49.447 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:49.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:49.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:49.548 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:49.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:49.577 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:49.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:49.678 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:49.709 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:49.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:49.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:49.779 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 42 16 79698 79682 118.559 15.8125 0.00944823 0.00842958 2024-08-17T18:14:49.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:49.852 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:49.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:49.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:49.886 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:49 smithi136 ceph-mon[109861]: pgmap v3035: 99 pgs: 99 active+clean; 4.7 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-08-17T18:14:49.886 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:49 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-17T18:14:49.886 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:49 smithi136 ceph-mon[109861]: osdmap e1616: 8 total, 8 up, 5 in 2024-08-17T18:14:49.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:49 smithi136 ceph-mon[115078]: pgmap v3035: 99 pgs: 99 active+clean; 4.7 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-08-17T18:14:49.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:49 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-17T18:14:49.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:49 smithi136 ceph-mon[115078]: osdmap e1616: 8 total, 8 up, 5 in 2024-08-17T18:14:49.887 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:14:49.954 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:49.994 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:49.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:49.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:50.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:49 smithi165 ceph-mon[105517]: pgmap v3035: 99 pgs: 99 active+clean; 4.7 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-08-17T18:14:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:49 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-17T18:14:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:49 smithi165 ceph-mon[105517]: osdmap e1616: 8 total, 8 up, 5 in 2024-08-17T18:14:50.095 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:50.131 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:50.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:50.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:50.233 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:50.269 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:50.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:50.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:50.371 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:50.420 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:50.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:50.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:50.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:50.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:50.553 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:50.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:50.654 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:50.691 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:50.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:50.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:50.779 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 43 16 81297 81281 118.126 99.9375 0.00454897 0.00846081 2024-08-17T18:14:50.792 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:50.819 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:50.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:50.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:50.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:50.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:50 smithi136 ceph-mon[115078]: osdmap e1617: 8 total, 8 up, 5 in 2024-08-17T18:14:50.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:50 smithi136 ceph-mon[115078]: 57.4 scrub starts 2024-08-17T18:14:50.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:50 smithi136 ceph-mon[115078]: 57.7 scrub starts 2024-08-17T18:14:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:50 smithi136 ceph-mon[109861]: osdmap e1617: 8 total, 8 up, 5 in 2024-08-17T18:14:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:50 smithi136 ceph-mon[109861]: 57.4 scrub starts 2024-08-17T18:14:50.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:50 smithi136 ceph-mon[109861]: 57.7 scrub starts 2024-08-17T18:14:50.956 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:50.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:50.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:51.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:50 smithi165 ceph-mon[105517]: osdmap e1617: 8 total, 8 up, 5 in 2024-08-17T18:14:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:50 smithi165 ceph-mon[105517]: 57.4 scrub starts 2024-08-17T18:14:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:50 smithi165 ceph-mon[105517]: 57.7 scrub starts 2024-08-17T18:14:51.057 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:51.087 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:51.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:51.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:51.189 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:51.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:51.221 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:51.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:51.322 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:51.355 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:51.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:51.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:51.457 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:51.491 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:51.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:51.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:51.592 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:51.631 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:51.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:51.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:51.733 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:51.770 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:51.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:51.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:51.779 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 44 16 82921 82905 117.748 101.5 0.00414563 0.0084885 2024-08-17T18:14:51.871 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:51.903 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:51.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:51.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:51.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:51 smithi136 ceph-mon[115078]: pgmap v3038: 109 pgs: 3 peering, 10 unknown, 96 active+clean; 4.8 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-08-17T18:14:51.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:51 smithi136 ceph-mon[115078]: 57.1 scrub starts 2024-08-17T18:14:51.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:51 smithi136 ceph-mon[115078]: 57.3 scrub starts 2024-08-17T18:14:51.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:14:51.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:14:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:51 smithi136 ceph-mon[109861]: pgmap v3038: 109 pgs: 3 peering, 10 unknown, 96 active+clean; 4.8 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-08-17T18:14:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:51 smithi136 ceph-mon[109861]: 57.1 scrub starts 2024-08-17T18:14:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:51 smithi136 ceph-mon[109861]: 57.3 scrub starts 2024-08-17T18:14:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:14:51.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:14:52.007 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:51 smithi165 ceph-mon[105517]: pgmap v3038: 109 pgs: 3 peering, 10 unknown, 96 active+clean; 4.8 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-08-17T18:14:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:51 smithi165 ceph-mon[105517]: 57.1 scrub starts 2024-08-17T18:14:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:51 smithi165 ceph-mon[105517]: 57.3 scrub starts 2024-08-17T18:14:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:14:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:14:52.048 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:52.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:52.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:52.150 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:52.193 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:52.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:52.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:52.295 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:52.327 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:52.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:52.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:52.429 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:52.466 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:52.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:52.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:52.568 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:52.602 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:52.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:52.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:52.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:52 smithi136 ceph-mon[109861]: 57.11 scrub starts 2024-08-17T18:14:52.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:52 smithi136 ceph-mon[109861]: 57.0 scrub starts 2024-08-17T18:14:52.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:52 smithi136 ceph-mon[115078]: 57.11 scrub starts 2024-08-17T18:14:52.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:52 smithi136 ceph-mon[115078]: 57.0 scrub starts 2024-08-17T18:14:52.704 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:52.737 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:52.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:52.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:52.780 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 45 16 84488 84472 117.308 97.9375 0.00977783 0.00852029 2024-08-17T18:14:52.838 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:52.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:52.887 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:52.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:52.988 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:52 smithi165 ceph-mon[105517]: 57.11 scrub starts 2024-08-17T18:14:53.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:52 smithi165 ceph-mon[105517]: 57.0 scrub starts 2024-08-17T18:14:53.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:53.028 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:53.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:53.129 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:53.137 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:14:53.138 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T18:14:53.138 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-17T18:14:53.138 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:14:53.163 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:53.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:53.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:53.264 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:53.287 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:53.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:53.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:53.388 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:53.421 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:53.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:53.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:53.523 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:53.549 INFO:tasks.thrashosds.thrasher:in_osds: [2, 4, 6, 0, 5] out_osds: [7, 1, 3] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:14:53.549 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:14:53.549 INFO:tasks.thrashosds.thrasher:Reweighting osd 0 to 0.16172691944412626 2024-08-17T18:14:53.549 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 0 0.16172691944412626 2024-08-17T18:14:53.557 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:53.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:53.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:53.607 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:14:53.659 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:53.694 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:53.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:53.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:53.780 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 46 16 86008 85992 116.822 95 0.00712589 0.00855559 2024-08-17T18:14:53.796 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:53.805 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:53 smithi136 ceph-mon[115078]: pgmap v3039: 109 pgs: 3 peering, 4 unknown, 102 active+clean; 4.9 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 86 MiB/s wr, 1.37k op/s; 21 KiB/s, 0 objects/s recovering 2024-08-17T18:14:53.805 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:53 smithi136 ceph-mon[115078]: 57.2 scrub starts 2024-08-17T18:14:53.805 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:53 smithi136 ceph-mon[115078]: 57.17 scrub starts 2024-08-17T18:14:53.806 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:53 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1070908563' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:14:53.806 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:53 smithi136 ceph-mon[109861]: pgmap v3039: 109 pgs: 3 peering, 4 unknown, 102 active+clean; 4.9 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 86 MiB/s wr, 1.37k op/s; 21 KiB/s, 0 objects/s recovering 2024-08-17T18:14:53.806 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:53 smithi136 ceph-mon[109861]: 57.2 scrub starts 2024-08-17T18:14:53.806 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:53 smithi136 ceph-mon[109861]: 57.17 scrub starts 2024-08-17T18:14:53.806 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:53 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1070908563' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:14:53.850 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-17T18:14:53.864 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:53.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:53.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:53.965 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:54.005 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:54.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:54.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:54.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:53 smithi165 ceph-mon[105517]: pgmap v3039: 109 pgs: 3 peering, 4 unknown, 102 active+clean; 4.9 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 86 MiB/s wr, 1.37k op/s; 21 KiB/s, 0 objects/s recovering 2024-08-17T18:14:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:53 smithi165 ceph-mon[105517]: 57.2 scrub starts 2024-08-17T18:14:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:53 smithi165 ceph-mon[105517]: 57.17 scrub starts 2024-08-17T18:14:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:53 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1070908563' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:14:54.107 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:54.153 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:54.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:54.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:54.255 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:54.288 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:54.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:54.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:54.390 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:54.423 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:54.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:54.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:54.524 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:54.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:54.560 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:54.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:54.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:54.684 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:14:54.694 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:54.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:54.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:54.752 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:14:54.760 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:14:54.781 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 47 16 87698 87682 116.58 105.625 0.0132669 0.0085731 2024-08-17T18:14:54.796 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:54.828 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:54.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:54.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:54.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:54 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:14:54.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:54 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1070908563' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:14:54.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:54 smithi136 ceph-mon[115078]: osdmap e1618: 8 total, 8 up, 5 in 2024-08-17T18:14:54.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:54 smithi136 ceph-mon[115078]: 57.12 scrub starts 2024-08-17T18:14:54.848 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:54 smithi136 ceph-mon[115078]: 57.10 scrub starts 2024-08-17T18:14:54.849 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:54 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:14:54.849 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:54 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1070908563' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:14:54.849 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:54 smithi136 ceph-mon[109861]: osdmap e1618: 8 total, 8 up, 5 in 2024-08-17T18:14:54.849 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:54 smithi136 ceph-mon[109861]: 57.12 scrub starts 2024-08-17T18:14:54.849 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:54 smithi136 ceph-mon[109861]: 57.10 scrub starts 2024-08-17T18:14:54.929 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:54.968 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:54.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:54.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:55.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:54 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:14:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:54 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1070908563' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:14:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:54 smithi165 ceph-mon[105517]: osdmap e1618: 8 total, 8 up, 5 in 2024-08-17T18:14:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:54 smithi165 ceph-mon[105517]: 57.12 scrub starts 2024-08-17T18:14:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:54 smithi165 ceph-mon[105517]: 57.10 scrub starts 2024-08-17T18:14:55.070 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:55.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:55.108 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:55.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:55.209 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:55.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:55.247 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:55.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:55.348 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:55.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:55.383 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:55.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:55.484 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:55.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:55.510 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:55.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:55.611 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:55.646 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:55.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:55.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:55.748 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:55.775 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:55.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:55.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:55.781 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 48 16 89292 89276 116.227 99.625 0.00574163 0.00859949 2024-08-17T18:14:55.877 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:55.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:55.911 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:55.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:55.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:55 smithi136 ceph-mon[109861]: pgmap v3041: 109 pgs: 3 peering, 3 unknown, 103 active+clean; 5.0 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 61 MiB/s wr, 982 op/s; 32 KiB/s, 0 objects/s recovering 2024-08-17T18:14:55.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:55 smithi136 ceph-mon[109861]: 57.14 scrub starts 2024-08-17T18:14:55.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:55 smithi136 ceph-mon[109861]: 57.13 scrub starts 2024-08-17T18:14:55.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:55 smithi136 ceph-mon[115078]: pgmap v3041: 109 pgs: 3 peering, 3 unknown, 103 active+clean; 5.0 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 61 MiB/s wr, 982 op/s; 32 KiB/s, 0 objects/s recovering 2024-08-17T18:14:55.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:55 smithi136 ceph-mon[115078]: 57.14 scrub starts 2024-08-17T18:14:55.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:55 smithi136 ceph-mon[115078]: 57.13 scrub starts 2024-08-17T18:14:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:55 smithi165 ceph-mon[105517]: pgmap v3041: 109 pgs: 3 peering, 3 unknown, 103 active+clean; 5.0 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 61 MiB/s wr, 982 op/s; 32 KiB/s, 0 objects/s recovering 2024-08-17T18:14:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:55 smithi165 ceph-mon[105517]: 57.14 scrub starts 2024-08-17T18:14:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:55 smithi165 ceph-mon[105517]: 57.13 scrub starts 2024-08-17T18:14:56.011 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:56.045 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:56.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:56.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:56.147 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:56.185 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:56.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:56.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:56.287 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:56.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:56.318 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:56.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:56.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:56.453 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:56.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:56.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:56.555 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:56.580 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:56.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:56.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:56.645 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:14:56.682 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:56.724 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:56.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:56.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:56.781 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 49 16 90899 90883 115.905 100.438 0.00792843 0.00861425 2024-08-17T18:14:56.826 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:56.856 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:56.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:56.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:56.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:56 smithi136 ceph-mon[109861]: 57.4 scrub starts 2024-08-17T18:14:56.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:56 smithi136 ceph-mon[109861]: 57.7 scrub starts 2024-08-17T18:14:56.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:56 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-08-17T18:14:56.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:56 smithi136 ceph-mon[115078]: 57.4 scrub starts 2024-08-17T18:14:56.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:56 smithi136 ceph-mon[115078]: 57.7 scrub starts 2024-08-17T18:14:56.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:56 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-08-17T18:14:56.958 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:56.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:56.984 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:56.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:57.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:56 smithi165 ceph-mon[105517]: 57.4 scrub starts 2024-08-17T18:14:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:56 smithi165 ceph-mon[105517]: 57.7 scrub starts 2024-08-17T18:14:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:56 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-08-17T18:14:57.085 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:57.108 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:57.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:57.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:57.210 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:14:57.232 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:57.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:57.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:14:57.333 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:57.355 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:57.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:57.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:57.457 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:57.478 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:57.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:57.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:57.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:57.601 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:57.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:57.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:57.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:57.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:57.725 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:57.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:57.782 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 50 16 91217 91201 113.984 19.875 0.00286916 0.0085962 2024-08-17T18:14:57.826 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:57.848 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:57.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:57.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:57.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:14:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:14:57] "GET /metrics HTTP/1.1" 200 36675 "" "Prometheus/2.51.0" 2024-08-17T18:14:57.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:57 smithi136 ceph-mon[109861]: pgmap v3042: 109 pgs: 109 active+clean; 5.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 74 MiB/s wr, 1.19k op/s; 24 KiB/s, 0 objects/s recovering 2024-08-17T18:14:57.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:57 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"}]': finished 2024-08-17T18:14:57.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:57 smithi136 ceph-mon[109861]: osdmap e1619: 8 total, 8 up, 5 in 2024-08-17T18:14:57.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:57 smithi136 ceph-mon[109861]: 57.4 scrub starts 2024-08-17T18:14:57.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:57 smithi136 ceph-mon[109861]: 57.17 scrub starts 2024-08-17T18:14:57.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:57 smithi136 ceph-mon[115078]: pgmap v3042: 109 pgs: 109 active+clean; 5.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 74 MiB/s wr, 1.19k op/s; 24 KiB/s, 0 objects/s recovering 2024-08-17T18:14:57.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:57 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"}]': finished 2024-08-17T18:14:57.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:57 smithi136 ceph-mon[115078]: osdmap e1619: 8 total, 8 up, 5 in 2024-08-17T18:14:57.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:57 smithi136 ceph-mon[115078]: 57.4 scrub starts 2024-08-17T18:14:57.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:57 smithi136 ceph-mon[115078]: 57.17 scrub starts 2024-08-17T18:14:57.950 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:57.972 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:57.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:57.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:58.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:57 smithi165 ceph-mon[105517]: pgmap v3042: 109 pgs: 109 active+clean; 5.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 74 MiB/s wr, 1.19k op/s; 24 KiB/s, 0 objects/s recovering 2024-08-17T18:14:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:57 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"}]': finished 2024-08-17T18:14:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:57 smithi165 ceph-mon[105517]: osdmap e1619: 8 total, 8 up, 5 in 2024-08-17T18:14:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:57 smithi165 ceph-mon[105517]: 57.4 scrub starts 2024-08-17T18:14:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:57 smithi165 ceph-mon[105517]: 57.17 scrub starts 2024-08-17T18:14:58.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:58.095 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:58.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:58.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:58.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:14:58.220 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:58.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:58.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:14:58.322 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:58.344 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:58.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:58.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:58.446 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:58.469 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:58.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:58.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:58.571 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:58.597 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:58.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:58.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:58.699 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:14:58.721 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:58.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:58.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:14:58.782 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 51 15 91240 91225 111.779 1.5 0.00758506 0.00875569 2024-08-17T18:14:58.823 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:58.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:58.861 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:58.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:58 smithi136 ceph-mon[109861]: osdmap e1620: 8 total, 8 up, 5 in 2024-08-17T18:14:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:58 smithi136 ceph-mon[109861]: 57.14 scrub starts 2024-08-17T18:14:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:58 smithi136 ceph-mon[109861]: 57.3 scrub starts 2024-08-17T18:14:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:58 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-08-17T18:14:58.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:58 smithi136 ceph-mon[115078]: osdmap e1620: 8 total, 8 up, 5 in 2024-08-17T18:14:58.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:58 smithi136 ceph-mon[115078]: 57.14 scrub starts 2024-08-17T18:14:58.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:58 smithi136 ceph-mon[115078]: 57.3 scrub starts 2024-08-17T18:14:58.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:58 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-08-17T18:14:58.962 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:58.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:58.984 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:58.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:58 smithi165 ceph-mon[105517]: osdmap e1620: 8 total, 8 up, 5 in 2024-08-17T18:14:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:58 smithi165 ceph-mon[105517]: 57.14 scrub starts 2024-08-17T18:14:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:58 smithi165 ceph-mon[105517]: 57.3 scrub starts 2024-08-17T18:14:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:58 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-08-17T18:14:59.084 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:59.110 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:59.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:59.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:59.212 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:59.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:59.241 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:59.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:59.342 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:59.365 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:59.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:59.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:14:59.467 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:14:59.489 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:59.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:59.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:14:59.591 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:14:59.613 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:59.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:59.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:14:59.714 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:14:59.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:59.736 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:59.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:14:59.782 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 52 16 91768 91752 110.262 32.9375 0.00219532 0.00892665 2024-08-17T18:14:59.837 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:14:59.858 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:14:59.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:59.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:14:59.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:59 smithi136 ceph-mon[115078]: pgmap v3045: 109 pgs: 109 active+clean; 5.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 92 MiB/s wr, 1.46k op/s; 11 KiB/s, 0 objects/s recovering 2024-08-17T18:14:59.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:59 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"}]': finished 2024-08-17T18:14:59.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:59 smithi136 ceph-mon[115078]: osdmap e1621: 8 total, 8 up, 5 in 2024-08-17T18:14:59.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:59 smithi136 ceph-mon[115078]: 57.0 scrub starts 2024-08-17T18:14:59.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:14:59 smithi136 ceph-mon[115078]: osdmap e1622: 8 total, 8 up, 5 in 2024-08-17T18:14:59.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:59 smithi136 ceph-mon[109861]: pgmap v3045: 109 pgs: 109 active+clean; 5.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 92 MiB/s wr, 1.46k op/s; 11 KiB/s, 0 objects/s recovering 2024-08-17T18:14:59.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:59 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"}]': finished 2024-08-17T18:14:59.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:59 smithi136 ceph-mon[109861]: osdmap e1621: 8 total, 8 up, 5 in 2024-08-17T18:14:59.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:59 smithi136 ceph-mon[109861]: 57.0 scrub starts 2024-08-17T18:14:59.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:14:59 smithi136 ceph-mon[109861]: osdmap e1622: 8 total, 8 up, 5 in 2024-08-17T18:14:59.960 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:14:59.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:14:59.982 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:14:59.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:00.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:59 smithi165 ceph-mon[105517]: pgmap v3045: 109 pgs: 109 active+clean; 5.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 92 MiB/s wr, 1.46k op/s; 11 KiB/s, 0 objects/s recovering 2024-08-17T18:15:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:59 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"}]': finished 2024-08-17T18:15:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:59 smithi165 ceph-mon[105517]: osdmap e1621: 8 total, 8 up, 5 in 2024-08-17T18:15:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:59 smithi165 ceph-mon[105517]: 57.0 scrub starts 2024-08-17T18:15:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:14:59 smithi165 ceph-mon[105517]: osdmap e1622: 8 total, 8 up, 5 in 2024-08-17T18:15:00.083 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:00.105 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:00.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:00.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:00.207 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:00.228 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:00.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:00.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:00.330 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:00.351 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:00.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:00.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:00.452 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:00.474 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:00.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:00.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:00.576 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:00.597 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:00.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:00.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:00.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:00.731 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:00.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:00.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:00.782 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 53 16 91911 91895 108.351 8.9375 0.0089423 0.0092222 2024-08-17T18:15:00.833 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:00.864 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:00.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:00.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:00.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:00 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-08-17T18:15:00.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:00 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"}]': finished 2024-08-17T18:15:00.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:00 smithi136 ceph-mon[109861]: osdmap e1623: 8 total, 8 up, 5 in 2024-08-17T18:15:00.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:00 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-08-17T18:15:00.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:00 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"}]': finished 2024-08-17T18:15:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:00 smithi136 ceph-mon[115078]: osdmap e1623: 8 total, 8 up, 5 in 2024-08-17T18:15:00.966 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:00.993 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:00.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:00.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:01.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:00 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-08-17T18:15:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:00 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"}]': finished 2024-08-17T18:15:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:00 smithi165 ceph-mon[105517]: osdmap e1623: 8 total, 8 up, 5 in 2024-08-17T18:15:01.094 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:01.117 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:01.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:01.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:01.219 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:01.241 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:01.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:01.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:01.342 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:01.364 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:01.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:01.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:01.466 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:01.486 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:01.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:01.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:01.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:01.609 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:01.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:01.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:01 smithi136 ceph-mon[109861]: pgmap v3048: 109 pgs: 109 active+clean; 5.6 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-08-17T18:15:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:01 smithi136 ceph-mon[109861]: 57.11 starting backfill to osd.4 from (0'0,0'0] MAX to 1619'5423 2024-08-17T18:15:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:01 smithi136 ceph-mon[109861]: 57.0 scrub starts 2024-08-17T18:15:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:01 smithi136 ceph-mon[109861]: osdmap e1624: 8 total, 8 up, 5 in 2024-08-17T18:15:01.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:01 smithi136 ceph-mon[115078]: pgmap v3048: 109 pgs: 109 active+clean; 5.6 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-08-17T18:15:01.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:01 smithi136 ceph-mon[115078]: 57.11 starting backfill to osd.4 from (0'0,0'0] MAX to 1619'5423 2024-08-17T18:15:01.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:01 smithi136 ceph-mon[115078]: 57.0 scrub starts 2024-08-17T18:15:01.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:01 smithi136 ceph-mon[115078]: osdmap e1624: 8 total, 8 up, 5 in 2024-08-17T18:15:01.711 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:01.733 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:01.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:01.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:01.782 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 54 16 92244 92228 106.73 20.8125 0.00272141 0.00920512 2024-08-17T18:15:01.835 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:01.856 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:01.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:01.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:01.958 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:01.980 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:01.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:01.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:02.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:01 smithi165 ceph-mon[105517]: pgmap v3048: 109 pgs: 109 active+clean; 5.6 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-08-17T18:15:02.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:01 smithi165 ceph-mon[105517]: 57.11 starting backfill to osd.4 from (0'0,0'0] MAX to 1619'5423 2024-08-17T18:15:02.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:01 smithi165 ceph-mon[105517]: 57.0 scrub starts 2024-08-17T18:15:02.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:01 smithi165 ceph-mon[105517]: osdmap e1624: 8 total, 8 up, 5 in 2024-08-17T18:15:02.082 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:02.104 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:02.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:02.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:02.206 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:02.228 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:02.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:02.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:02.330 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:02.353 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:02.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:02.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:02.454 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:02.475 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:02.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:02.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:02.577 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:02.598 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:02.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:02.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:02.700 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:02.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:02.723 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:02.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:02.782 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 55 16 92244 92228 104.789 0 - 0.00920512 2024-08-17T18:15:02.825 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:02.846 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:02.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:02.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:02.948 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:02.986 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:02.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:02.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:03.088 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:03.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:03.124 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:03.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:03.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:03.265 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:03.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:03.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:03.367 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:03.401 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:03.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:03.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:03.503 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:03.541 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:03.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:03.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:03.643 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:03.678 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:03.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:03.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:03.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:03.783 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 56 15 93411 93396 104.221 36.5 0.00692419 0.00958913 2024-08-17T18:15:03.814 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:03.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:03.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:03.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:03 smithi136 ceph-mon[109861]: pgmap v3051: 109 pgs: 1 active+clean+scrubbing, 1 peering, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 105 active+clean; 5.6 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 32 MiB/s wr, 517 op/s; 8294/275604 objects misplaced (3.009%); 3.2 MiB/s, 50 objects/s recovering 2024-08-17T18:15:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:03 smithi136 ceph-mon[109861]: osdmap e1625: 8 total, 8 up, 5 in 2024-08-17T18:15:03.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:03 smithi136 ceph-mon[115078]: pgmap v3051: 109 pgs: 1 active+clean+scrubbing, 1 peering, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 105 active+clean; 5.6 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 32 MiB/s wr, 517 op/s; 8294/275604 objects misplaced (3.009%); 3.2 MiB/s, 50 objects/s recovering 2024-08-17T18:15:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:03 smithi136 ceph-mon[115078]: osdmap e1625: 8 total, 8 up, 5 in 2024-08-17T18:15:03.956 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:03.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:03.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:04.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:03 smithi165 ceph-mon[105517]: pgmap v3051: 109 pgs: 1 active+clean+scrubbing, 1 peering, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 105 active+clean; 5.6 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 32 MiB/s wr, 517 op/s; 8294/275604 objects misplaced (3.009%); 3.2 MiB/s, 50 objects/s recovering 2024-08-17T18:15:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:03 smithi165 ceph-mon[105517]: osdmap e1625: 8 total, 8 up, 5 in 2024-08-17T18:15:04.057 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:04.073 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:15:04.109 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:04.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:04.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:04.211 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:04.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:04.246 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:04.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:04.347 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:04.379 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:04.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:04.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:04.481 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:04.517 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:04.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:04.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:04.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:04.653 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:04.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:04.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:04.755 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:04.782 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 57 16 94814 94798 103.93 87.625 0.0188667 0.00961731 2024-08-17T18:15:04.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:04.793 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:04.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:04.894 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:04.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:04.927 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:04.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:05.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:05.065 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:05.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:05.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:05.167 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:05.188 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:05.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:05.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:05.290 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:05.320 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:05.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:05.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:05.422 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:05.453 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:05.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:05.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:05.555 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:05.581 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:05.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:05.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:05.682 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:05.714 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:05.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:05.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:05.782 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 58 15 96181 96166 103.612 85.5 0.011919 0.00964676 2024-08-17T18:15:05.816 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:05.848 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:05.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:05.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:05 smithi136 ceph-mon[109861]: pgmap v3053: 109 pgs: 1 active+clean+scrubbing, 1 peering, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 105 active+clean; 5.6 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 16 MiB/s wr, 250 op/s; 8294/276444 objects misplaced (3.000%); 2.6 MiB/s, 42 objects/s recovering 2024-08-17T18:15:05.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:05 smithi136 ceph-mon[115078]: pgmap v3053: 109 pgs: 1 active+clean+scrubbing, 1 peering, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 105 active+clean; 5.6 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 16 MiB/s wr, 250 op/s; 8294/276444 objects misplaced (3.000%); 2.6 MiB/s, 42 objects/s recovering 2024-08-17T18:15:05.951 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:05.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:05.990 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:05.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:06.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:05 smithi165 ceph-mon[105517]: pgmap v3053: 109 pgs: 1 active+clean+scrubbing, 1 peering, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 105 active+clean; 5.6 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 16 MiB/s wr, 250 op/s; 8294/276444 objects misplaced (3.000%); 2.6 MiB/s, 42 objects/s recovering 2024-08-17T18:15:06.091 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:06.121 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:06.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:06.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:06.223 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:06.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:06.259 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:06.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:06.360 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:06.391 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:06.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:06.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:06.493 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:06.526 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:06.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:06.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:06.629 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:06.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:06.667 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:06.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:06.768 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:06.782 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 59 16 97615 97599 103.374 89.5625 0.00948375 0.00966902 2024-08-17T18:15:06.807 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:06.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:06.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:06.856 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:15:06.857 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:15:06.909 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:06.929 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:15:06.929 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:15:06.942 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:06.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:06.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:06.975 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:15:06.975 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:15:07.009 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:15:07.010 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:15:07.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:15:07.041 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:15:07.041 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:15:07.044 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:07.076 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:15:07.076 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:15:07.080 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:07.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:07.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:07.108 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:15:07.108 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:15:07.145 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:15:07.146 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:15:07.176 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:15:07.176 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:15:07.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:07.222 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:07.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:07.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:07.324 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:07.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:07.357 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:07.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:07.458 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:07.490 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:07.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:07.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:07.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:07.627 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:07.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:07.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:07.729 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:07.756 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:07 smithi136 ceph-mon[109861]: pgmap v3054: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 105 active+clean; 5.7 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 30 MiB/s wr, 477 op/s; 2830/281451 objects degraded (1.006%); 7803/281451 objects misplaced (2.772%); 5.3 MiB/s, 85 objects/s recovering 2024-08-17T18:15:07.756 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:07 smithi136 ceph-mon[109861]: Health check failed: Degraded data redundancy: 2830/281451 objects degraded (1.006%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:15:07.756 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:07 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2074863903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:15:07.756 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:07 smithi136 ceph-mon[109861]: 57.0 scrub starts 2024-08-17T18:15:07.757 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:15:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:15:07] "GET /metrics HTTP/1.1" 200 36695 "" "Prometheus/2.51.0" 2024-08-17T18:15:07.757 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:07 smithi136 ceph-mon[115078]: pgmap v3054: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 105 active+clean; 5.7 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 30 MiB/s wr, 477 op/s; 2830/281451 objects degraded (1.006%); 7803/281451 objects misplaced (2.772%); 5.3 MiB/s, 85 objects/s recovering 2024-08-17T18:15:07.757 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:07 smithi136 ceph-mon[115078]: Health check failed: Degraded data redundancy: 2830/281451 objects degraded (1.006%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:15:07.758 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:07 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2074863903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:15:07.758 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:07 smithi136 ceph-mon[115078]: 57.0 scrub starts 2024-08-17T18:15:07.772 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:07.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:07.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:07.782 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:2024-08-17T18:15:07.782957+0000 min lat: 0.00125927 max lat: 2.19838 avg lat: 0.00968544 2024-08-17T18:15:07.783 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T18:15:07.783 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 60 16 99101 99085 103.199 92.875 0.0126123 0.00968544 2024-08-17T18:15:07.817 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:15:07.817 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:15:07.876 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:07.915 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:07.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:07.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:08.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:07 smithi165 ceph-mon[105517]: pgmap v3054: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 105 active+clean; 5.7 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 30 MiB/s wr, 477 op/s; 2830/281451 objects degraded (1.006%); 7803/281451 objects misplaced (2.772%); 5.3 MiB/s, 85 objects/s recovering 2024-08-17T18:15:08.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:07 smithi165 ceph-mon[105517]: Health check failed: Degraded data redundancy: 2830/281451 objects degraded (1.006%), 1 pg degraded (PG_DEGRADED) 2024-08-17T18:15:08.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:07 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2074863903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:15:08.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:07 smithi165 ceph-mon[105517]: 57.0 scrub starts 2024-08-17T18:15:08.016 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:08.058 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:08.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:08.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:08.160 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:08.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:08.201 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:08.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:08.302 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:08.427 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:08.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:08.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:08.529 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:08.563 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:08.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:08.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:08.665 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:08.695 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:08.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:08.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:08.783 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 61 15 100608 100593 103.052 94.25 0.00841937 0.0096994 2024-08-17T18:15:08.797 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:08.829 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:08.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:08.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:08.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:08.969 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:08.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:08.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:09.071 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:09.110 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:09.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:09.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:09.212 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:09.249 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:09.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:09.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:09.350 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:09.385 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:09.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:09.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:09.487 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:09.520 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:09.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:09.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:09.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:09.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:09.661 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:09.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:09.665 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.0 to 0.161727 (2966) 2024-08-17T18:15:09.761 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:09.782 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 62 16 101949 101933 102.741 83.75 0.0179528 0.00971503 2024-08-17T18:15:09.785 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:09.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:09.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:09.887 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:09.909 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:09.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:09.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:09 smithi136 ceph-mon[109861]: pgmap v3055: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.9 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 50 MiB/s wr, 804 op/s; 2830/291615 objects degraded (0.970%); 7803/291615 objects misplaced (2.676%); 4.1 MiB/s, 65 objects/s recovering 2024-08-17T18:15:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:09 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1511909745' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 0.16172691944412626} : dispatch 2024-08-17T18:15:09.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:09 smithi136 ceph-mon[115078]: pgmap v3055: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.9 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 50 MiB/s wr, 804 op/s; 2830/291615 objects degraded (0.970%); 7803/291615 objects misplaced (2.676%); 4.1 MiB/s, 65 objects/s recovering 2024-08-17T18:15:09.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:09 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1511909745' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 0.16172691944412626} : dispatch 2024-08-17T18:15:10.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:09 smithi165 ceph-mon[105517]: pgmap v3055: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.9 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 50 MiB/s wr, 804 op/s; 2830/291615 objects degraded (0.970%); 7803/291615 objects misplaced (2.676%); 4.1 MiB/s, 65 objects/s recovering 2024-08-17T18:15:10.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:09 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1511909745' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 0.16172691944412626} : dispatch 2024-08-17T18:15:10.011 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:10.032 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:10.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:10.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:10.134 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:10.155 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:10.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:10.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:10.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:10.280 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:10.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:10.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:10.356 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:15:10.381 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:10.402 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:10.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:10.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:10.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:10.531 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:10.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:10.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:10.633 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:10.655 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:10.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:10.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:10.757 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:10.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:10.779 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:10.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:10.782 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 63 16 101960 101944 101.121 0.6875 0.00644334 0.00977561 2024-08-17T18:15:10.880 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:10.901 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:10.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:10.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:10.942 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:10 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1511909745' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 0.16172691944412626}]': finished 2024-08-17T18:15:10.942 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:10 smithi136 ceph-mon[109861]: osdmap e1626: 8 total, 8 up, 5 in 2024-08-17T18:15:10.942 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:10 smithi136 ceph-mon[109861]: osdmap e1627: 8 total, 8 up, 5 in 2024-08-17T18:15:10.942 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:10 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1511909745' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 0.16172691944412626}]': finished 2024-08-17T18:15:10.942 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:10 smithi136 ceph-mon[115078]: osdmap e1626: 8 total, 8 up, 5 in 2024-08-17T18:15:10.942 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:10 smithi136 ceph-mon[115078]: osdmap e1627: 8 total, 8 up, 5 in 2024-08-17T18:15:11.003 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:11.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:10 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1511909745' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 0.16172691944412626}]': finished 2024-08-17T18:15:11.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:10 smithi165 ceph-mon[105517]: osdmap e1626: 8 total, 8 up, 5 in 2024-08-17T18:15:11.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:10 smithi165 ceph-mon[105517]: osdmap e1627: 8 total, 8 up, 5 in 2024-08-17T18:15:11.025 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:11.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:11.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:11.127 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:11.148 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:11.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:11.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:11.250 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:11.271 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:11.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:11.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:11.373 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:11.386 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-17T18:15:11.395 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:11.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:11.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:11.497 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:11.519 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:11.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:11.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:11.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:11.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:11.643 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:11.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:11.672 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:15:11.744 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:11.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:11.770 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:11.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:11.783 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 64 16 101963 101947 99.5439 0.1875 2.05729 0.00982609 2024-08-17T18:15:11.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:11.904 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:11.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:11.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:11.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:11 smithi136 ceph-mon[115078]: pgmap v3057: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.0 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 55 MiB/s wr, 884 op/s; 2830/296844 objects degraded (0.953%); 7803/296844 objects misplaced (2.629%); 2.4 MiB/s, 38 objects/s recovering 2024-08-17T18:15:11.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:11 smithi136 ceph-mon[115078]: 57.11 continuing backfill to osd.4 from (1611'100,1625'5752] MIN to 1625'5752 2024-08-17T18:15:11.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:11 smithi136 ceph-mon[115078]: 57.11 starting backfill to osd.5 from (0'0,0'0] MAX to 1625'5752 2024-08-17T18:15:11.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:11 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2930624159' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:15:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:11 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:15:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:11 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:15:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:11 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:15:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:11 smithi136 ceph-mon[115078]: osdmap e1628: 8 total, 8 up, 5 in 2024-08-17T18:15:11.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:11 smithi136 ceph-mon[109861]: pgmap v3057: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.0 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 55 MiB/s wr, 884 op/s; 2830/296844 objects degraded (0.953%); 7803/296844 objects misplaced (2.629%); 2.4 MiB/s, 38 objects/s recovering 2024-08-17T18:15:11.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:11 smithi136 ceph-mon[109861]: 57.11 continuing backfill to osd.4 from (1611'100,1625'5752] MIN to 1625'5752 2024-08-17T18:15:11.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:11 smithi136 ceph-mon[109861]: 57.11 starting backfill to osd.5 from (0'0,0'0] MAX to 1625'5752 2024-08-17T18:15:11.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:11 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2930624159' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:15:11.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:11 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:15:11.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:11 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:15:11.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:11 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:15:11.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:11 smithi136 ceph-mon[109861]: osdmap e1628: 8 total, 8 up, 5 in 2024-08-17T18:15:12.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:11 smithi165 ceph-mon[105517]: pgmap v3057: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.0 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 55 MiB/s wr, 884 op/s; 2830/296844 objects degraded (0.953%); 7803/296844 objects misplaced (2.629%); 2.4 MiB/s, 38 objects/s recovering 2024-08-17T18:15:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:11 smithi165 ceph-mon[105517]: 57.11 continuing backfill to osd.4 from (1611'100,1625'5752] MIN to 1625'5752 2024-08-17T18:15:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:11 smithi165 ceph-mon[105517]: 57.11 starting backfill to osd.5 from (0'0,0'0] MAX to 1625'5752 2024-08-17T18:15:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:11 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2930624159' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:15:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:11 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:15:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:11 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:15:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:11 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:15:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:11 smithi165 ceph-mon[105517]: osdmap e1628: 8 total, 8 up, 5 in 2024-08-17T18:15:12.042 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:12.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:12.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:12.046 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:15:12.143 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:12.165 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:12.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:12.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:12.267 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:12.288 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:12.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:12.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:12.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:12.413 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:12.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:12.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:12.514 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:12.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:12.539 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:12.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:12.640 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:12.662 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:12.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:12.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:12.764 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:12.769 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:12 smithi136 ceph-mon[109861]: 57.1 starting backfill to osd.4 from (0'0,0'0] MAX to 1625'5782 2024-08-17T18:15:12.769 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:12 smithi136 ceph-mon[109861]: 57.f starting backfill to osd.4 from (0'0,0'0] MAX to 1625'6466 2024-08-17T18:15:12.769 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:12 smithi136 ceph-mon[109861]: osdmap e1629: 8 total, 8 up, 5 in 2024-08-17T18:15:12.769 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:12 smithi136 ceph-mon[115078]: 57.1 starting backfill to osd.4 from (0'0,0'0] MAX to 1625'5782 2024-08-17T18:15:12.769 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:12 smithi136 ceph-mon[115078]: 57.f starting backfill to osd.4 from (0'0,0'0] MAX to 1625'6466 2024-08-17T18:15:12.769 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:12 smithi136 ceph-mon[115078]: osdmap e1629: 8 total, 8 up, 5 in 2024-08-17T18:15:12.783 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 65 16 102080 102064 98.1251 7.3125 0.00295056 0.0100637 2024-08-17T18:15:12.788 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:12.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:12.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:12.890 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:12.912 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:12.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:12.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:12 smithi165 ceph-mon[105517]: 57.1 starting backfill to osd.4 from (0'0,0'0] MAX to 1625'5782 2024-08-17T18:15:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:12 smithi165 ceph-mon[105517]: 57.f starting backfill to osd.4 from (0'0,0'0] MAX to 1625'6466 2024-08-17T18:15:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:12 smithi165 ceph-mon[105517]: osdmap e1629: 8 total, 8 up, 5 in 2024-08-17T18:15:13.013 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:13.034 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:13.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:13.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:13.136 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:13.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:13.159 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:13.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:13.260 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:13.282 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:13.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:13.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:13.368 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:15:13.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:13.407 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:13.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:13.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:13.509 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:13.531 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:13.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:13.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:13.633 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:13.656 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:13.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:13.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:13.759 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:13.781 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:13.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:13.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:13.783 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 66 16 102080 102064 96.6384 0 - 0.0100637 2024-08-17T18:15:13.811 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:13 smithi136 ceph-mon[109861]: pgmap v3060: 109 pgs: 1 active+recovery_wait+remapped, 1 activating+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 6.1 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 69 MiB/s wr, 1.10k op/s; 9187/301329 objects degraded (3.049%); 17033/301329 objects misplaced (5.653%); 3.4 MiB/s, 54 objects/s recovering 2024-08-17T18:15:13.811 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:13 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 9187/301329 objects degraded (3.049%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:13.811 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:13 smithi136 ceph-mon[109861]: osdmap e1630: 8 total, 8 up, 5 in 2024-08-17T18:15:13.812 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:13 smithi136 ceph-mon[115078]: pgmap v3060: 109 pgs: 1 active+recovery_wait+remapped, 1 activating+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 6.1 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 69 MiB/s wr, 1.10k op/s; 9187/301329 objects degraded (3.049%); 17033/301329 objects misplaced (5.653%); 3.4 MiB/s, 54 objects/s recovering 2024-08-17T18:15:13.812 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:13 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 9187/301329 objects degraded (3.049%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:13.812 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:13 smithi136 ceph-mon[115078]: osdmap e1630: 8 total, 8 up, 5 in 2024-08-17T18:15:13.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:13.905 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:13.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:13.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:14.007 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:14.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:13 smithi165 ceph-mon[105517]: pgmap v3060: 109 pgs: 1 active+recovery_wait+remapped, 1 activating+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 103 active+clean; 6.1 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 69 MiB/s wr, 1.10k op/s; 9187/301329 objects degraded (3.049%); 17033/301329 objects misplaced (5.653%); 3.4 MiB/s, 54 objects/s recovering 2024-08-17T18:15:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:13 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 9187/301329 objects degraded (3.049%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:13 smithi165 ceph-mon[105517]: osdmap e1630: 8 total, 8 up, 5 in 2024-08-17T18:15:14.028 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:14.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:14.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:14.130 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:14.153 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:14.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:14.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:14.255 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:14.276 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:14.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:14.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:14.378 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:14.400 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:14.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:14.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:14.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:15:14.502 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:14.557 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:14.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:14.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:14.659 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:14.682 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:14.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:14.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:14.783 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 67 16 102080 102064 95.1961 0 - 0.0100637 2024-08-17T18:15:14.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:14.806 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:14.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:14.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:14.908 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:14.929 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:14.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:14.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:15.031 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:15.055 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:15.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:15.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:15.157 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:15.178 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:15.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:15.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:15.280 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:15.301 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:15.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:15.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:15.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:15.426 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:15.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:15.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:15.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 18:15:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=grafana.update.checker t=2024-08-17T18:15:15.191343673Z level=info msg="Update check succeeded" duration=60.732348ms 2024-08-17T18:15:15.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 18:15:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=plugins.update.checker t=2024-08-17T18:15:15.192999047Z level=info msg="Update check succeeded" duration=70.525451ms 2024-08-17T18:15:15.528 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:15.551 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:15.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:15.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:15.653 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:15.675 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:15.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:15.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:15.777 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:15.783 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 68 16 102119 102103 93.8321 0.8125 0.00734951 0.0103555 2024-08-17T18:15:15.807 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:15.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:15.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:15.874 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:15 smithi136 ceph-mon[109861]: pgmap v3063: 109 pgs: 2 active+undersized+remapped, 1 active+recovery_wait+remapped, 1 activating+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 6.2 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 18 KiB/s rd, 37 MiB/s wr, 603 op/s; 9187/303930 objects degraded (3.023%); 20299/303930 objects misplaced (6.679%); 5.3 MiB/s, 83 objects/s recovering 2024-08-17T18:15:15.874 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:15 smithi136 ceph-mon[115078]: pgmap v3063: 109 pgs: 2 active+undersized+remapped, 1 active+recovery_wait+remapped, 1 activating+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 6.2 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 18 KiB/s rd, 37 MiB/s wr, 603 op/s; 9187/303930 objects degraded (3.023%); 20299/303930 objects misplaced (6.679%); 5.3 MiB/s, 83 objects/s recovering 2024-08-17T18:15:15.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:15.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:15.943 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:15.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:15 smithi165 ceph-mon[105517]: pgmap v3063: 109 pgs: 2 active+undersized+remapped, 1 active+recovery_wait+remapped, 1 activating+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 6.2 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 18 KiB/s rd, 37 MiB/s wr, 603 op/s; 9187/303930 objects degraded (3.023%); 20299/303930 objects misplaced (6.679%); 5.3 MiB/s, 83 objects/s recovering 2024-08-17T18:15:16.011 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 18:15:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=cleanup t=2024-08-17T18:15:15.876998155Z level=info msg="Completed cleanup jobs" duration=839.176227ms 2024-08-17T18:15:16.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:16.072 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:16.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:16.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:16.121 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:15:16.121 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:15:16.142 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:15:16.142 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:15:16.161 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:15:16.161 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:15:16.174 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:16.179 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:15:16.180 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:15:16.200 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:15:16.200 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:15:16.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:16.201 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:16.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:16.220 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:15:16.220 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:15:16.243 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:15:16.243 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:15:16.267 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:15:16.267 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:15:16.303 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:16.338 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:16.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:16.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:16.440 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:16.467 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:16.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:16.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:16.569 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:16.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:16.602 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:16.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:16.703 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:16.727 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:16.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:16.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:16.783 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 69 16 102780 102764 93.0709 41.3125 0.00310711 0.0107215 2024-08-17T18:15:16.830 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:16.832 INFO:tasks.thrashosds.thrasher:in_osds: [2, 4, 6, 0, 5] out_osds: [7, 1, 3] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:15:16.832 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:15:16.832 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-08-17T18:15:16.832 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-17T18:15:16.832 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-17T18:15:16.833 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-08-17T18:15:16.857 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:16.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:16.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:16.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:16 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2146063542' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:15:16.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:16 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2146063542' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:15:16.960 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:16.996 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:16.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:16.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:16 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2146063542' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:15:17.098 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:17.126 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:17.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:17.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:17.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:17.251 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:17.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:17.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:17.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:17.379 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:17.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:17.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:17.462 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:15:17.481 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:17.484 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:15:17.484 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-17T18:15:17.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-17T18:15:17.484 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:15:17.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.484 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.45626.0:170332 55.e 55:7b002905:::benchmark_data_smithi136_313476_object78346:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1515)", 2024-08-17T18:15:17.484 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:08:36.079968+0000", 2024-08-17T18:15:17.485 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 401.40013960700003, 2024-08-17T18:15:17.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1469576539999999, 2024-08-17T18:15:17.485 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:15:17.485 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:15:17.485 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:17.485 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:15:17.485 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.45626", 2024-08-17T18:15:17.485 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:15:17.485 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 170332 2024-08-17T18:15:17.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.485 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:15:17.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:15:17.486 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.079968+0000", 2024-08-17T18:15:17.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.486 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.486 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:15:17.486 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.079968+0000", 2024-08-17T18:15:17.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.486 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.486 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:15:17.486 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.079970+0000", 2024-08-17T18:15:17.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.513e-06 2024-08-17T18:15:17.487 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.487 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:15:17.487 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.079975+0000", 2024-08-17T18:15:17.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2079999999999999e-06 2024-08-17T18:15:17.487 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.487 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:15:17.487 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.081423+0000", 2024-08-17T18:15:17.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014476510000000001 2024-08-17T18:15:17.487 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.487 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:15:17.488 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.081436+0000", 2024-08-17T18:15:17.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3257e-05 2024-08-17T18:15:17.488 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.488 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.488 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.488 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.081512+0000", 2024-08-17T18:15:17.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5860999999999998e-05 2024-08-17T18:15:17.488 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.488 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.488 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.488 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.081523+0000", 2024-08-17T18:15:17.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0781999999999999e-05 2024-08-17T18:15:17.488 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.489 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.489 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.074548+0000", 2024-08-17T18:15:17.489 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.99302576899999995 2024-08-17T18:15:17.489 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.489 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.489 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.074558+0000", 2024-08-17T18:15:17.489 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9000000000000001e-06 2024-08-17T18:15:17.489 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.489 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.489 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.217960+0000", 2024-08-17T18:15:17.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.143401941 2024-08-17T18:15:17.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.490 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.490 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:15:17.490 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.218051+0000", 2024-08-17T18:15:17.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0923999999999999e-05 2024-08-17T18:15:17.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.490 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.490 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,4,6", 2024-08-17T18:15:17.490 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.218134+0000", 2024-08-17T18:15:17.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3158999999999994e-05 2024-08-17T18:15:17.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.491 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.491 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.221266+0000", 2024-08-17T18:15:17.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031318940000000001 2024-08-17T18:15:17.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.491 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:15:17.491 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.226531+0000", 2024-08-17T18:15:17.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0052650860000000004 2024-08-17T18:15:17.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.226711+0000", 2024-08-17T18:15:17.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017958500000000001 2024-08-17T18:15:17.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.226793+0000", 2024-08-17T18:15:17.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1648000000000002e-05 2024-08-17T18:15:17.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:15:17.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.226837+0000", 2024-08-17T18:15:17.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4931000000000001e-05 2024-08-17T18:15:17.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:15:17.493 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.226926+0000", 2024-08-17T18:15:17.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8545e-05 2024-08-17T18:15:17.493 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.493 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:15:17.493 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.493 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.45626.0:170333 55.e 55:71b1d6be:::benchmark_data_smithi136_313476_object78347:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1515)", 2024-08-17T18:15:17.493 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:08:36.081477+0000", 2024-08-17T18:15:17.493 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 401.39863066800001, 2024-08-17T18:15:17.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1481386210000002, 2024-08-17T18:15:17.494 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:15:17.494 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:15:17.494 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:17.494 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:15:17.494 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.45626", 2024-08-17T18:15:17.494 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:15:17.494 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 170333 2024-08-17T18:15:17.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.494 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:15:17.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.494 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:15:17.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.081477+0000", 2024-08-17T18:15:17.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.495 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:15:17.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.081477+0000", 2024-08-17T18:15:17.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.495 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:15:17.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.081479+0000", 2024-08-17T18:15:17.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.719e-06 2024-08-17T18:15:17.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.495 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:15:17.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.081483+0000", 2024-08-17T18:15:17.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1799999999999998e-06 2024-08-17T18:15:17.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:15:17.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.081493+0000", 2024-08-17T18:15:17.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7629999999999995e-06 2024-08-17T18:15:17.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:15:17.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.081498+0000", 2024-08-17T18:15:17.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1309999999999997e-06 2024-08-17T18:15:17.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.081547+0000", 2024-08-17T18:15:17.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9088000000000002e-05 2024-08-17T18:15:17.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.081555+0000", 2024-08-17T18:15:17.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5569999999999998e-06 2024-08-17T18:15:17.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.074580+0000", 2024-08-17T18:15:17.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.99302500599999999 2024-08-17T18:15:17.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.498 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.074586+0000", 2024-08-17T18:15:17.498 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6479999999999997e-06 2024-08-17T18:15:17.498 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.498 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.221352+0000", 2024-08-17T18:15:17.498 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1467654860000001 2024-08-17T18:15:17.498 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.498 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:15:17.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.221475+0000", 2024-08-17T18:15:17.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000122927 2024-08-17T18:15:17.499 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.499 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.499 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,4,6", 2024-08-17T18:15:17.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.221583+0000", 2024-08-17T18:15:17.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000108149 2024-08-17T18:15:17.500 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.500 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.500 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:15:17.500 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.226719+0000", 2024-08-17T18:15:17.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0051364540000000004 2024-08-17T18:15:17.500 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.500 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.500 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.500 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.226744+0000", 2024-08-17T18:15:17.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4250999999999999e-05 2024-08-17T18:15:17.500 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.500 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.501 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.226973+0000", 2024-08-17T18:15:17.501 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022908299999999999 2024-08-17T18:15:17.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.501 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.229455+0000", 2024-08-17T18:15:17.501 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002482635 2024-08-17T18:15:17.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:15:17.501 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.229493+0000", 2024-08-17T18:15:17.501 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7187999999999997e-05 2024-08-17T18:15:17.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.502 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.502 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:15:17.502 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.229616+0000", 2024-08-17T18:15:17.502 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123356 2024-08-17T18:15:17.502 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.502 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:15:17.502 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.502 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.502 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.502 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.45626.0:170353 55.e 55:7ce4130f:::benchmark_data_smithi136_313476_object78367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1515)", 2024-08-17T18:15:17.502 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:08:36.103771+0000", 2024-08-17T18:15:17.503 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 401.37633702699998, 2024-08-17T18:15:17.503 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1323814909999999, 2024-08-17T18:15:17.503 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:15:17.503 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:15:17.503 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:17.503 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:15:17.503 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.45626", 2024-08-17T18:15:17.503 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:15:17.503 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 170353 2024-08-17T18:15:17.503 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.503 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:15:17.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:15:17.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.103771+0000", 2024-08-17T18:15:17.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.504 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:15:17.504 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.103771+0000", 2024-08-17T18:15:17.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.504 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:15:17.504 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.103773+0000", 2024-08-17T18:15:17.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8199999999999999e-06 2024-08-17T18:15:17.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.505 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:15:17.505 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.103778+0000", 2024-08-17T18:15:17.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5670000000000002e-06 2024-08-17T18:15:17.505 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.505 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.505 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:15:17.505 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.103798+0000", 2024-08-17T18:15:17.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9369999999999999e-05 2024-08-17T18:15:17.505 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.505 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.505 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:15:17.505 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.103810+0000", 2024-08-17T18:15:17.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2334000000000001e-05 2024-08-17T18:15:17.505 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.104552+0000", 2024-08-17T18:15:17.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00074228 2024-08-17T18:15:17.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.104564+0000", 2024-08-17T18:15:17.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1323e-05 2024-08-17T18:15:17.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.074606+0000", 2024-08-17T18:15:17.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97004254199999995 2024-08-17T18:15:17.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.507 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.074613+0000", 2024-08-17T18:15:17.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4230000000000002e-06 2024-08-17T18:15:17.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.507 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.223689+0000", 2024-08-17T18:15:17.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.149076553 2024-08-17T18:15:17.508 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.508 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.508 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:15:17.508 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.223790+0000", 2024-08-17T18:15:17.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000100746 2024-08-17T18:15:17.508 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.508 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.508 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,4,6", 2024-08-17T18:15:17.508 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.223898+0000", 2024-08-17T18:15:17.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000108069 2024-08-17T18:15:17.508 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.508 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.508 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:15:17.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.226753+0000", 2024-08-17T18:15:17.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0028554589999999999 2024-08-17T18:15:17.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.227019+0000", 2024-08-17T18:15:17.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00026544300000000003 2024-08-17T18:15:17.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.229676+0000", 2024-08-17T18:15:17.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0026567959999999999 2024-08-17T18:15:17.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.236049+0000", 2024-08-17T18:15:17.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063734129999999996 2024-08-17T18:15:17.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:15:17.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.236087+0000", 2024-08-17T18:15:17.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8138e-05 2024-08-17T18:15:17.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:15:17.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.236152+0000", 2024-08-17T18:15:17.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5215000000000005e-05 2024-08-17T18:15:17.511 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.511 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:15:17.511 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.511 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.45626.0:170388 55.e 55:73d38651:::benchmark_data_smithi136_313476_object78402:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1515)", 2024-08-17T18:15:17.511 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:08:36.147358+0000", 2024-08-17T18:15:17.511 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 401.33274957499998, 2024-08-17T18:15:17.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.089062492, 2024-08-17T18:15:17.511 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:15:17.512 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:15:17.512 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:17.512 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:15:17.512 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.45626", 2024-08-17T18:15:17.512 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:15:17.512 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 170388 2024-08-17T18:15:17.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.512 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:15:17.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:15:17.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.147358+0000", 2024-08-17T18:15:17.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:15:17.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.147358+0000", 2024-08-17T18:15:17.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:15:17.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.147360+0000", 2024-08-17T18:15:17.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9589999999999998e-06 2024-08-17T18:15:17.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:15:17.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.147365+0000", 2024-08-17T18:15:17.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4769999999999997e-06 2024-08-17T18:15:17.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:15:17.514 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.147385+0000", 2024-08-17T18:15:17.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0463999999999999e-05 2024-08-17T18:15:17.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:15:17.514 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.147394+0000", 2024-08-17T18:15:17.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4449999999999996e-06 2024-08-17T18:15:17.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.147458+0000", 2024-08-17T18:15:17.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3849000000000005e-05 2024-08-17T18:15:17.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.147469+0000", 2024-08-17T18:15:17.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0992000000000001e-05 2024-08-17T18:15:17.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.074632+0000", 2024-08-17T18:15:17.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92716353600000001 2024-08-17T18:15:17.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.074639+0000", 2024-08-17T18:15:17.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7089999999999996e-06 2024-08-17T18:15:17.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.227058+0000", 2024-08-17T18:15:17.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1524190249999999 2024-08-17T18:15:17.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:15:17.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.227142+0000", 2024-08-17T18:15:17.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3789999999999996e-05 2024-08-17T18:15:17.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,4,6", 2024-08-17T18:15:17.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.227219+0000", 2024-08-17T18:15:17.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7169000000000006e-05 2024-08-17T18:15:17.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:15:17.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.232595+0000", 2024-08-17T18:15:17.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0053759649999999999 2024-08-17T18:15:17.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.236202+0000", 2024-08-17T18:15:17.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036070070000000002 2024-08-17T18:15:17.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.236250+0000", 2024-08-17T18:15:17.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8637999999999997e-05 2024-08-17T18:15:17.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.236294+0000", 2024-08-17T18:15:17.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3461000000000003e-05 2024-08-17T18:15:17.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:15:17.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.236338+0000", 2024-08-17T18:15:17.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4332999999999997e-05 2024-08-17T18:15:17.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:15:17.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.236421+0000", 2024-08-17T18:15:17.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2672999999999999e-05 2024-08-17T18:15:17.519 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.519 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:15:17.519 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.520 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.45626.0:170391 55.e 55:7e47ab9c:::benchmark_data_smithi136_313476_object78405:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1515)", 2024-08-17T18:15:17.520 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:08:36.148404+0000", 2024-08-17T18:15:17.520 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 401.33170422400002, 2024-08-17T18:15:17.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0882907230000001, 2024-08-17T18:15:17.520 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:15:17.520 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:15:17.520 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:17.520 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:15:17.520 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.45626", 2024-08-17T18:15:17.520 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:15:17.520 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 170391 2024-08-17T18:15:17.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.521 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:15:17.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:15:17.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.148404+0000", 2024-08-17T18:15:17.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:15:17.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.148404+0000", 2024-08-17T18:15:17.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:15:17.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.148405+0000", 2024-08-17T18:15:17.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.124e-06 2024-08-17T18:15:17.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:15:17.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.148410+0000", 2024-08-17T18:15:17.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7090000000000001e-06 2024-08-17T18:15:17.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:15:17.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.148426+0000", 2024-08-17T18:15:17.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5971e-05 2024-08-17T18:15:17.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:15:17.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.148433+0000", 2024-08-17T18:15:17.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0949999999999998e-06 2024-08-17T18:15:17.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.148534+0000", 2024-08-17T18:15:17.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000101685 2024-08-17T18:15:17.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.148543+0000", 2024-08-17T18:15:17.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0829999999999993e-06 2024-08-17T18:15:17.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.074658+0000", 2024-08-17T18:15:17.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92611462899999997 2024-08-17T18:15:17.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.074665+0000", 2024-08-17T18:15:17.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6549999999999998e-06 2024-08-17T18:15:17.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.229716+0000", 2024-08-17T18:15:17.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.155051235 2024-08-17T18:15:17.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:15:17.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.229800+0000", 2024-08-17T18:15:17.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3887000000000005e-05 2024-08-17T18:15:17.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,4,6", 2024-08-17T18:15:17.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.229873+0000", 2024-08-17T18:15:17.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2952000000000002e-05 2024-08-17T18:15:17.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:15:17.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.236214+0000", 2024-08-17T18:15:17.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063415099999999999 2024-08-17T18:15:17.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.236473+0000", 2024-08-17T18:15:17.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025867000000000002 2024-08-17T18:15:17.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.236510+0000", 2024-08-17T18:15:17.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6582999999999999e-05 2024-08-17T18:15:17.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.236572+0000", 2024-08-17T18:15:17.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2181000000000007e-05 2024-08-17T18:15:17.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:15:17.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.236606+0000", 2024-08-17T18:15:17.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4455999999999997e-05 2024-08-17T18:15:17.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:15:17.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.236694+0000", 2024-08-17T18:15:17.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8298000000000001e-05 2024-08-17T18:15:17.528 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.528 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:15:17.528 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.528 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.45626.0:170393 55.e 55:7f8b5e67:::benchmark_data_smithi136_313476_object78407:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1515)", 2024-08-17T18:15:17.528 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:08:36.152549+0000", 2024-08-17T18:15:17.528 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 401.32755858100001, 2024-08-17T18:15:17.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0843999520000001, 2024-08-17T18:15:17.528 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:15:17.529 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:15:17.529 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:17.529 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:15:17.529 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.45626", 2024-08-17T18:15:17.529 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:15:17.529 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 170393 2024-08-17T18:15:17.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.529 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:15:17.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:15:17.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.152549+0000", 2024-08-17T18:15:17.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:15:17.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.152549+0000", 2024-08-17T18:15:17.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:15:17.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.152550+0000", 2024-08-17T18:15:17.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.04e-06 2024-08-17T18:15:17.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:15:17.531 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.152554+0000", 2024-08-17T18:15:17.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2329999999999998e-06 2024-08-17T18:15:17.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.531 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:15:17.531 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.152563+0000", 2024-08-17T18:15:17.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0799999999999995e-06 2024-08-17T18:15:17.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.531 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:15:17.531 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.152569+0000", 2024-08-17T18:15:17.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1929999999999998e-06 2024-08-17T18:15:17.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.152627+0000", 2024-08-17T18:15:17.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7930999999999998e-05 2024-08-17T18:15:17.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.152634+0000", 2024-08-17T18:15:17.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4499999999999998e-06 2024-08-17T18:15:17.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.074684+0000", 2024-08-17T18:15:17.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92204976999999999 2024-08-17T18:15:17.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.074697+0000", 2024-08-17T18:15:17.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2652e-05 2024-08-17T18:15:17.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.232638+0000", 2024-08-17T18:15:17.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1579416840000001 2024-08-17T18:15:17.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:15:17.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.232765+0000", 2024-08-17T18:15:17.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012684999999999999 2024-08-17T18:15:17.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,4,6", 2024-08-17T18:15:17.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.232892+0000", 2024-08-17T18:15:17.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012690800000000001 2024-08-17T18:15:17.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:15:17.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.236432+0000", 2024-08-17T18:15:17.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003539496 2024-08-17T18:15:17.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.236542+0000", 2024-08-17T18:15:17.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110244 2024-08-17T18:15:17.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.236756+0000", 2024-08-17T18:15:17.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000214515 2024-08-17T18:15:17.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.236811+0000", 2024-08-17T18:15:17.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4821999999999998e-05 2024-08-17T18:15:17.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:15:17.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.236859+0000", 2024-08-17T18:15:17.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7403e-05 2024-08-17T18:15:17.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:15:17.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.236949+0000", 2024-08-17T18:15:17.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0680999999999995e-05 2024-08-17T18:15:17.536 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.536 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:15:17.537 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.537 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.45626.0:170394 55.e 55:7a360c76:::benchmark_data_smithi136_313476_object78408:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1515)", 2024-08-17T18:15:17.537 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:08:36.154255+0000", 2024-08-17T18:15:17.537 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 401.32585257599999, 2024-08-17T18:15:17.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0940399479999998, 2024-08-17T18:15:17.537 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:15:17.537 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:15:17.537 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:17.537 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:15:17.537 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.45626", 2024-08-17T18:15:17.537 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:15:17.537 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 170394 2024-08-17T18:15:17.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.538 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:15:17.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:15:17.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.154255+0000", 2024-08-17T18:15:17.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:15:17.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.154255+0000", 2024-08-17T18:15:17.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:15:17.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.154256+0000", 2024-08-17T18:15:17.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9299999999999998e-07 2024-08-17T18:15:17.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:15:17.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.154258+0000", 2024-08-17T18:15:17.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4880000000000001e-06 2024-08-17T18:15:17.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:15:17.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.154266+0000", 2024-08-17T18:15:17.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2089999999999999e-06 2024-08-17T18:15:17.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:15:17.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.154271+0000", 2024-08-17T18:15:17.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5470000000000004e-06 2024-08-17T18:15:17.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.154320+0000", 2024-08-17T18:15:17.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8992000000000001e-05 2024-08-17T18:15:17.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.154327+0000", 2024-08-17T18:15:17.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9330000000000004e-06 2024-08-17T18:15:17.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.074719+0000", 2024-08-17T18:15:17.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92039238599999995 2024-08-17T18:15:17.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.074727+0000", 2024-08-17T18:15:17.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.055e-06 2024-08-17T18:15:17.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.236978+0000", 2024-08-17T18:15:17.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1622500549999999 2024-08-17T18:15:17.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:15:17.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.237069+0000", 2024-08-17T18:15:17.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1658999999999998e-05 2024-08-17T18:15:17.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,4,6", 2024-08-17T18:15:17.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.237171+0000", 2024-08-17T18:15:17.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102243 2024-08-17T18:15:17.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.240233+0000", 2024-08-17T18:15:17.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0030618429999999999 2024-08-17T18:15:17.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.240272+0000", 2024-08-17T18:15:17.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8822999999999997e-05 2024-08-17T18:15:17.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.248103+0000", 2024-08-17T18:15:17.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0078304210000000006 2024-08-17T18:15:17.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:15:17.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.248119+0000", 2024-08-17T18:15:17.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6855999999999999e-05 2024-08-17T18:15:17.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:15:17.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.248175+0000", 2024-08-17T18:15:17.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5661000000000001e-05 2024-08-17T18:15:17.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.546 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:15:17.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.248295+0000", 2024-08-17T18:15:17.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120284 2024-08-17T18:15:17.546 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.546 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:15:17.546 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.546 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207580 56.10 56:0b80fbc4:::benchmark_data_smithi136_356456_object80876:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:15:17.546 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.637715+0000", 2024-08-17T18:15:17.546 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 157.84239326100001, 2024-08-17T18:15:17.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4471485369999999, 2024-08-17T18:15:17.547 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:15:17.547 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:15:17.547 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:17.547 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:15:17.547 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:15:17.547 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:17.547 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207580 2024-08-17T18:15:17.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.547 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:15:17.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.547 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:15:17.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637715+0000", 2024-08-17T18:15:17.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:15:17.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637715+0000", 2024-08-17T18:15:17.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:15:17.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637717+0000", 2024-08-17T18:15:17.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9860000000000001e-06 2024-08-17T18:15:17.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:15:17.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637722+0000", 2024-08-17T18:15:17.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5890000000000002e-06 2024-08-17T18:15:17.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:15:17.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637739+0000", 2024-08-17T18:15:17.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6606e-05 2024-08-17T18:15:17.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:15:17.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637749+0000", 2024-08-17T18:15:17.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6530000000000006e-06 2024-08-17T18:15:17.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637835+0000", 2024-08-17T18:15:17.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6347000000000005e-05 2024-08-17T18:15:17.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-08-17T18:15:17.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637842+0000", 2024-08-17T18:15:17.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8759999999999999e-06 2024-08-17T18:15:17.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.639138+0000", 2024-08-17T18:15:17.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012963180000000001 2024-08-17T18:15:17.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.639147+0000", 2024-08-17T18:15:17.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0610000000000002e-06 2024-08-17T18:15:17.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.019011+0000", 2024-08-17T18:15:17.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3798637710000001 2024-08-17T18:15:17.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:15:17.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.019132+0000", 2024-08-17T18:15:17.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012087300000000001 2024-08-17T18:15:17.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:15:17.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.019367+0000", 2024-08-17T18:15:17.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023531499999999999 2024-08-17T18:15:17.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.553 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.553 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.021924+0000", 2024-08-17T18:15:17.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025573229999999998 2024-08-17T18:15:17.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.553 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.553 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.553 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.021962+0000", 2024-08-17T18:15:17.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7457999999999999e-05 2024-08-17T18:15:17.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.553 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.553 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:15:17.553 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.021974+0000", 2024-08-17T18:15:17.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1963999999999999e-05 2024-08-17T18:15:17.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.022118+0000", 2024-08-17T18:15:17.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014464799999999999 2024-08-17T18:15:17.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.084688+0000", 2024-08-17T18:15:17.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.062569939000000005 2024-08-17T18:15:17.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:15:17.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.084755+0000", 2024-08-17T18:15:17.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6804999999999995e-05 2024-08-17T18:15:17.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:15:17.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.084863+0000", 2024-08-17T18:15:17.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010800499999999999 2024-08-17T18:15:17.555 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.555 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:15:17.555 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.555 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207645 56.10 56:0f00368f:::benchmark_data_smithi136_356456_object80941:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:15:17.556 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.637768+0000", 2024-08-17T18:15:17.556 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 157.84234036999999, 2024-08-17T18:15:17.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4472689160000001, 2024-08-17T18:15:17.556 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:15:17.556 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:15:17.556 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:17.556 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:15:17.556 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:15:17.556 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:17.556 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207645 2024-08-17T18:15:17.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.556 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:15:17.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:15:17.557 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637768+0000", 2024-08-17T18:15:17.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.557 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:15:17.557 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637768+0000", 2024-08-17T18:15:17.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.557 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:15:17.557 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637768+0000", 2024-08-17T18:15:17.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5000000000000003e-07 2024-08-17T18:15:17.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.558 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:15:17.558 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637771+0000", 2024-08-17T18:15:17.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8090000000000001e-06 2024-08-17T18:15:17.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.558 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:15:17.558 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637778+0000", 2024-08-17T18:15:17.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3130000000000001e-06 2024-08-17T18:15:17.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.558 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:15:17.559 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:15:17.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:17.560 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:17.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:17.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637784+0000", 2024-08-17T18:15:17.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5809999999999996e-06 2024-08-17T18:15:17.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637912+0000", 2024-08-17T18:15:17.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012843300000000001 2024-08-17T18:15:17.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.562 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:15:17.562 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637918+0000", 2024-08-17T18:15:17.562 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.772e-06 2024-08-17T18:15:17.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.562 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.562 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.639992+0000", 2024-08-17T18:15:17.562 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002074237 2024-08-17T18:15:17.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640001+0000", 2024-08-17T18:15:17.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3799999999999994e-06 2024-08-17T18:15:17.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.020861+0000", 2024-08-17T18:15:17.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.380859995 2024-08-17T18:15:17.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:15:17.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.020934+0000", 2024-08-17T18:15:17.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3367000000000004e-05 2024-08-17T18:15:17.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:15:17.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.021081+0000", 2024-08-17T18:15:17.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000146961 2024-08-17T18:15:17.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:15:17.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.024244+0000", 2024-08-17T18:15:17.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003163014 2024-08-17T18:15:17.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.024323+0000", 2024-08-17T18:15:17.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8669000000000002e-05 2024-08-17T18:15:17.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.024366+0000", 2024-08-17T18:15:17.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3390999999999998e-05 2024-08-17T18:15:17.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.071609+0000", 2024-08-17T18:15:17.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.047243185 2024-08-17T18:15:17.566 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.566 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.566 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.084906+0000", 2024-08-17T18:15:17.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013296894 2024-08-17T18:15:17.566 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.566 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:15:17.566 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.084940+0000", 2024-08-17T18:15:17.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4344000000000002e-05 2024-08-17T18:15:17.566 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:15:17.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.085037+0000", 2024-08-17T18:15:17.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6020999999999994e-05 2024-08-17T18:15:17.567 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.567 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:15:17.567 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.567 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207658 56.10 56:0afb0108:::benchmark_data_smithi136_356456_object80954:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:15:17.567 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.637878+0000", 2024-08-17T18:15:17.567 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 157.842230261, 2024-08-17T18:15:17.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.449536841, 2024-08-17T18:15:17.567 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:15:17.567 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:15:17.568 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:17.568 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:15:17.568 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:15:17.568 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:17.568 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207658 2024-08-17T18:15:17.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.568 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:15:17.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:15:17.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637878+0000", 2024-08-17T18:15:17.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:15:17.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637878+0000", 2024-08-17T18:15:17.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:15:17.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637879+0000", 2024-08-17T18:15:17.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4099999999999997e-07 2024-08-17T18:15:17.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:15:17.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637883+0000", 2024-08-17T18:15:17.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9639999999999997e-06 2024-08-17T18:15:17.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.570 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:15:17.570 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637890+0000", 2024-08-17T18:15:17.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7549999999999994e-06 2024-08-17T18:15:17.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.570 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:15:17.570 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637898+0000", 2024-08-17T18:15:17.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.746e-06 2024-08-17T18:15:17.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.571 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.571 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637943+0000", 2024-08-17T18:15:17.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4591e-05 2024-08-17T18:15:17.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.571 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:15:17.571 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637945+0000", 2024-08-17T18:15:17.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8830000000000002e-06 2024-08-17T18:15:17.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.571 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.571 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640036+0000", 2024-08-17T18:15:17.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002090742 2024-08-17T18:15:17.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640044+0000", 2024-08-17T18:15:17.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1620000000000008e-06 2024-08-17T18:15:17.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.022173+0000", 2024-08-17T18:15:17.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3821288540000001 2024-08-17T18:15:17.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:15:17.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.022306+0000", 2024-08-17T18:15:17.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013228400000000001 2024-08-17T18:15:17.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:15:17.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.022549+0000", 2024-08-17T18:15:17.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000243916 2024-08-17T18:15:17.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:15:17.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.024334+0000", 2024-08-17T18:15:17.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017846019999999999 2024-08-17T18:15:17.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.574 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.024402+0000", 2024-08-17T18:15:17.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8189999999999996e-05 2024-08-17T18:15:17.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.574 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.024435+0000", 2024-08-17T18:15:17.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2679999999999999e-05 2024-08-17T18:15:17.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.574 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.072897+0000", 2024-08-17T18:15:17.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048462489999999997 2024-08-17T18:15:17.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.575 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.575 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.087216+0000", 2024-08-17T18:15:17.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.01431905 2024-08-17T18:15:17.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.575 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:15:17.575 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.087285+0000", 2024-08-17T18:15:17.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8727999999999997e-05 2024-08-17T18:15:17.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.575 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:15:17.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.087415+0000", 2024-08-17T18:15:17.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012936299999999999 2024-08-17T18:15:17.576 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.576 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:15:17.576 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.576 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207672 56.10 56:0dff339c:::benchmark_data_smithi136_356456_object80968:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:15:17.576 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.637910+0000", 2024-08-17T18:15:17.576 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 157.84219748999999, 2024-08-17T18:15:17.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4496818419999999, 2024-08-17T18:15:17.576 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:15:17.576 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:15:17.577 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:17.577 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:15:17.577 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:15:17.577 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:17.577 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207672 2024-08-17T18:15:17.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.577 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:15:17.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:15:17.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637910+0000", 2024-08-17T18:15:17.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:15:17.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637910+0000", 2024-08-17T18:15:17.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:15:17.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637911+0000", 2024-08-17T18:15:17.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1899999999999998e-07 2024-08-17T18:15:17.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:15:17.579 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637913+0000", 2024-08-17T18:15:17.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2560000000000001e-06 2024-08-17T18:15:17.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.579 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:15:17.579 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637920+0000", 2024-08-17T18:15:17.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9210000000000003e-06 2024-08-17T18:15:17.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.579 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:15:17.579 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637925+0000", 2024-08-17T18:15:17.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3639999999999998e-06 2024-08-17T18:15:17.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637968+0000", 2024-08-17T18:15:17.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2484999999999997e-05 2024-08-17T18:15:17.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:15:17.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637972+0000", 2024-08-17T18:15:17.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4800000000000003e-06 2024-08-17T18:15:17.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640079+0000", 2024-08-17T18:15:17.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021068150000000002 2024-08-17T18:15:17.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640087+0000", 2024-08-17T18:15:17.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9820000000000008e-06 2024-08-17T18:15:17.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.023406+0000", 2024-08-17T18:15:17.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3833189859999999 2024-08-17T18:15:17.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:15:17.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.023479+0000", 2024-08-17T18:15:17.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2476000000000002e-05 2024-08-17T18:15:17.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:15:17.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.023617+0000", 2024-08-17T18:15:17.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000138169 2024-08-17T18:15:17.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:15:17.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.025302+0000", 2024-08-17T18:15:17.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001685305 2024-08-17T18:15:17.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.026534+0000", 2024-08-17T18:15:17.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001232325 2024-08-17T18:15:17.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.026587+0000", 2024-08-17T18:15:17.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2573e-05 2024-08-17T18:15:17.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.072933+0000", 2024-08-17T18:15:17.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.046346119999999998 2024-08-17T18:15:17.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.087463+0000", 2024-08-17T18:15:17.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014529801 2024-08-17T18:15:17.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:15:17.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.087498+0000", 2024-08-17T18:15:17.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5318000000000003e-05 2024-08-17T18:15:17.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:15:17.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.087592+0000", 2024-08-17T18:15:17.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4047000000000003e-05 2024-08-17T18:15:17.585 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.585 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:15:17.585 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.585 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207721 56.10 56:0dd4c85f:::benchmark_data_smithi136_356456_object81017:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:15:17.585 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.637971+0000", 2024-08-17T18:15:17.585 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 157.84213668500001, 2024-08-17T18:15:17.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.454932664, 2024-08-17T18:15:17.586 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:15:17.586 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:15:17.586 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:17.586 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:15:17.586 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:15:17.586 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:17.586 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207721 2024-08-17T18:15:17.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.586 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:15:17.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:15:17.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637971+0000", 2024-08-17T18:15:17.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:15:17.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637971+0000", 2024-08-17T18:15:17.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:15:17.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637972+0000", 2024-08-17T18:15:17.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5099999999999999e-07 2024-08-17T18:15:17.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:15:17.588 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637976+0000", 2024-08-17T18:15:17.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5920000000000001e-06 2024-08-17T18:15:17.588 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.588 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:15:17.588 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637990+0000", 2024-08-17T18:15:17.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4015e-05 2024-08-17T18:15:17.588 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.588 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:15:17.588 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637997+0000", 2024-08-17T18:15:17.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8839999999999994e-06 2024-08-17T18:15:17.588 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.588 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638034+0000", 2024-08-17T18:15:17.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6928999999999998e-05 2024-08-17T18:15:17.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:15:17.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638039+0000", 2024-08-17T18:15:17.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3449999999999999e-06 2024-08-17T18:15:17.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640141+0000", 2024-08-17T18:15:17.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021019279999999999 2024-08-17T18:15:17.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640147+0000", 2024-08-17T18:15:17.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5459999999999997e-06 2024-08-17T18:15:17.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.024461+0000", 2024-08-17T18:15:17.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3843137730000001 2024-08-17T18:15:17.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:15:17.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.024529+0000", 2024-08-17T18:15:17.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8180000000000001e-05 2024-08-17T18:15:17.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:15:17.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.024668+0000", 2024-08-17T18:15:17.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013920600000000001 2024-08-17T18:15:17.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.026651+0000", 2024-08-17T18:15:17.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0019829600000000002 2024-08-17T18:15:17.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.592 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.592 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.026684+0000", 2024-08-17T18:15:17.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2381e-05 2024-08-17T18:15:17.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.592 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:15:17.592 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.026703+0000", 2024-08-17T18:15:17.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9213999999999999e-05 2024-08-17T18:15:17.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.592 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.592 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.074256+0000", 2024-08-17T18:15:17.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.047553554999999997 2024-08-17T18:15:17.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.092689+0000", 2024-08-17T18:15:17.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018432568999999999 2024-08-17T18:15:17.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:15:17.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.092761+0000", 2024-08-17T18:15:17.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2466999999999996e-05 2024-08-17T18:15:17.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:15:17.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.092904+0000", 2024-08-17T18:15:17.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014246899999999999 2024-08-17T18:15:17.594 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.594 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:15:17.594 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.594 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207760 56.10 56:094a1205:::benchmark_data_smithi136_356456_object81056:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:15:17.594 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.638010+0000", 2024-08-17T18:15:17.594 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 157.84209762500001, 2024-08-17T18:15:17.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.45637813, 2024-08-17T18:15:17.594 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:15:17.595 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:15:17.595 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:17.595 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:15:17.595 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:15:17.595 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:17.595 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207760 2024-08-17T18:15:17.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.595 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:15:17.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:15:17.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638010+0000", 2024-08-17T18:15:17.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:15:17.596 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638010+0000", 2024-08-17T18:15:17.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.596 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:15:17.596 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638011+0000", 2024-08-17T18:15:17.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9900000000000001e-07 2024-08-17T18:15:17.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.596 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:15:17.596 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638013+0000", 2024-08-17T18:15:17.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6280000000000001e-06 2024-08-17T18:15:17.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:15:17.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638021+0000", 2024-08-17T18:15:17.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1350000000000003e-06 2024-08-17T18:15:17.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:15:17.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638026+0000", 2024-08-17T18:15:17.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3179999999999997e-06 2024-08-17T18:15:17.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638071+0000", 2024-08-17T18:15:17.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4708e-05 2024-08-17T18:15:17.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:15:17.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638075+0000", 2024-08-17T18:15:17.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.741e-06 2024-08-17T18:15:17.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640205+0000", 2024-08-17T18:15:17.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00212978 2024-08-17T18:15:17.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640212+0000", 2024-08-17T18:15:17.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0060000000000003e-06 2024-08-17T18:15:17.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.025333+0000", 2024-08-17T18:15:17.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3851207940000001 2024-08-17T18:15:17.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:15:17.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.025405+0000", 2024-08-17T18:15:17.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1772000000000003e-05 2024-08-17T18:15:17.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:15:17.600 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.025543+0000", 2024-08-17T18:15:17.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013786199999999999 2024-08-17T18:15:17.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:15:17.600 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.026707+0000", 2024-08-17T18:15:17.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001164371 2024-08-17T18:15:17.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.026758+0000", 2024-08-17T18:15:17.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1034e-05 2024-08-17T18:15:17.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.027663+0000", 2024-08-17T18:15:17.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00090472399999999998 2024-08-17T18:15:17.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.074299+0000", 2024-08-17T18:15:17.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.046636098000000001 2024-08-17T18:15:17.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.602 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.094154+0000", 2024-08-17T18:15:17.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019855695999999999 2024-08-17T18:15:17.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:15:17.602 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.094262+0000", 2024-08-17T18:15:17.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010785099999999999 2024-08-17T18:15:17.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:15:17.603 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.094388+0000", 2024-08-17T18:15:17.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012611299999999999 2024-08-17T18:15:17.603 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.603 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:15:17.603 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.603 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.603 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207790 56.10 56:0d05e484:::benchmark_data_smithi136_356456_object81086:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:15:17.603 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.638033+0000", 2024-08-17T18:15:17.603 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 157.84207511899999, 2024-08-17T18:15:17.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4570861230000001, 2024-08-17T18:15:17.603 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:15:17.603 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:15:17.603 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:17.604 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:15:17.604 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:15:17.604 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:17.604 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207790 2024-08-17T18:15:17.604 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.605 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:15:17.605 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.605 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:15:17.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638033+0000", 2024-08-17T18:15:17.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.606 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:15:17.606 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638033+0000", 2024-08-17T18:15:17.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.606 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:15:17.606 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638033+0000", 2024-08-17T18:15:17.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9099999999999999e-07 2024-08-17T18:15:17.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.607 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:15:17.607 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638035+0000", 2024-08-17T18:15:17.607 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8759999999999999e-06 2024-08-17T18:15:17.607 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.607 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:15:17.607 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638041+0000", 2024-08-17T18:15:17.607 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1099999999999999e-06 2024-08-17T18:15:17.607 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.607 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:15:17.607 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638046+0000", 2024-08-17T18:15:17.608 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8890000000000001e-06 2024-08-17T18:15:17.608 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.608 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.608 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.608 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638143+0000", 2024-08-17T18:15:17.608 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7078999999999997e-05 2024-08-17T18:15:17.608 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.608 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.608 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:15:17.608 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638148+0000", 2024-08-17T18:15:17.608 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6890000000000003e-06 2024-08-17T18:15:17.608 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.608 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640249+0000", 2024-08-17T18:15:17.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021008480000000002 2024-08-17T18:15:17.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640256+0000", 2024-08-17T18:15:17.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1690000000000003e-06 2024-08-17T18:15:17.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.026793+0000", 2024-08-17T18:15:17.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3865375549999999 2024-08-17T18:15:17.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.610 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:15:17.610 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.026868+0000", 2024-08-17T18:15:17.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4140999999999994e-05 2024-08-17T18:15:17.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.610 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:15:17.610 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.027003+0000", 2024-08-17T18:15:17.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000135418 2024-08-17T18:15:17.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.611 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.029543+0000", 2024-08-17T18:15:17.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025404120000000001 2024-08-17T18:15:17.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:15:17.611 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.029552+0000", 2024-08-17T18:15:17.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2099999999999993e-06 2024-08-17T18:15:17.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.611 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.029576+0000", 2024-08-17T18:15:17.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4085999999999999e-05 2024-08-17T18:15:17.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.075573+0000", 2024-08-17T18:15:17.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.045997008999999998 2024-08-17T18:15:17.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.094908+0000", 2024-08-17T18:15:17.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019335082 2024-08-17T18:15:17.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:15:17.613 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.094980+0000", 2024-08-17T18:15:17.613 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1841000000000006e-05 2024-08-17T18:15:17.613 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.613 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:15:17.613 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.095119+0000", 2024-08-17T18:15:17.613 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013931800000000001 2024-08-17T18:15:17.613 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.613 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:15:17.613 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.613 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.613 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207793 56.10 56:0f777e78:::benchmark_data_smithi136_356456_object81089:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:15:17.614 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.638105+0000", 2024-08-17T18:15:17.614 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 157.84200319300001, 2024-08-17T18:15:17.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4573933100000001, 2024-08-17T18:15:17.614 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:15:17.614 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:15:17.614 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:17.614 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:15:17.614 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:15:17.614 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:17.614 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207793 2024-08-17T18:15:17.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.614 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:15:17.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:15:17.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638105+0000", 2024-08-17T18:15:17.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:15:17.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638105+0000", 2024-08-17T18:15:17.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:15:17.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638105+0000", 2024-08-17T18:15:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6400000000000002e-07 2024-08-17T18:15:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:15:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638109+0000", 2024-08-17T18:15:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.653e-06 2024-08-17T18:15:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:15:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638130+0000", 2024-08-17T18:15:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0471000000000001e-05 2024-08-17T18:15:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:15:17.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638136+0000", 2024-08-17T18:15:17.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7880000000000001e-06 2024-08-17T18:15:17.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638183+0000", 2024-08-17T18:15:17.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6270000000000003e-05 2024-08-17T18:15:17.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:15:17.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638188+0000", 2024-08-17T18:15:17.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7019999999999998e-06 2024-08-17T18:15:17.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640287+0000", 2024-08-17T18:15:17.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020988669999999999 2024-08-17T18:15:17.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640291+0000", 2024-08-17T18:15:17.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1949999999999996e-06 2024-08-17T18:15:17.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.027702+0000", 2024-08-17T18:15:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.387410799 2024-08-17T18:15:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:15:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.027786+0000", 2024-08-17T18:15:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3397000000000001e-05 2024-08-17T18:15:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:15:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.027950+0000", 2024-08-17T18:15:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016487300000000001 2024-08-17T18:15:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.029610+0000", 2024-08-17T18:15:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016590610000000001 2024-08-17T18:15:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:15:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.029626+0000", 2024-08-17T18:15:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6116e-05 2024-08-17T18:15:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.029668+0000", 2024-08-17T18:15:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2135999999999998e-05 2024-08-17T18:15:17.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.621 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.075602+0000", 2024-08-17T18:15:17.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.045934540000000003 2024-08-17T18:15:17.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.621 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.095322+0000", 2024-08-17T18:15:17.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019719235000000002 2024-08-17T18:15:17.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:15:17.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.095384+0000", 2024-08-17T18:15:17.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2131999999999994e-05 2024-08-17T18:15:17.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:15:17.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.095498+0000", 2024-08-17T18:15:17.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114411 2024-08-17T18:15:17.622 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.622 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:15:17.622 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.623 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.623 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207810 56.10 56:0ea6e34e:::benchmark_data_smithi136_356456_object81106:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:15:17.623 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.638147+0000", 2024-08-17T18:15:17.623 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 157.841960508, 2024-08-17T18:15:17.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4576238779999999, 2024-08-17T18:15:17.623 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:15:17.623 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:15:17.623 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:17.623 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:15:17.623 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:15:17.623 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:17.623 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207810 2024-08-17T18:15:17.623 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:15:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:15:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638147+0000", 2024-08-17T18:15:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:15:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638147+0000", 2024-08-17T18:15:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:15:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638148+0000", 2024-08-17T18:15:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2e-07 2024-08-17T18:15:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:15:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638150+0000", 2024-08-17T18:15:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2290000000000002e-06 2024-08-17T18:15:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:15:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638156+0000", 2024-08-17T18:15:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9410000000000004e-06 2024-08-17T18:15:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:15:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638160+0000", 2024-08-17T18:15:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1930000000000003e-06 2024-08-17T18:15:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638213+0000", 2024-08-17T18:15:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2367999999999997e-05 2024-08-17T18:15:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:15:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638218+0000", 2024-08-17T18:15:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9899999999999997e-06 2024-08-17T18:15:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640323+0000", 2024-08-17T18:15:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002105539 2024-08-17T18:15:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640330+0000", 2024-08-17T18:15:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0539999999999997e-06 2024-08-17T18:15:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.028618+0000", 2024-08-17T18:15:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.38828763 2024-08-17T18:15:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:15:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.028740+0000", 2024-08-17T18:15:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000122014 2024-08-17T18:15:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:15:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.028870+0000", 2024-08-17T18:15:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000130562 2024-08-17T18:15:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:15:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.030547+0000", 2024-08-17T18:15:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001676564 2024-08-17T18:15:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.031449+0000", 2024-08-17T18:15:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00090164300000000004 2024-08-17T18:15:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.031488+0000", 2024-08-17T18:15:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9415999999999997e-05 2024-08-17T18:15:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.076959+0000", 2024-08-17T18:15:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.045470857000000003 2024-08-17T18:15:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.095546+0000", 2024-08-17T18:15:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018586928999999999 2024-08-17T18:15:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:15:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.095612+0000", 2024-08-17T18:15:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6029000000000001e-05 2024-08-17T18:15:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:15:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.095771+0000", 2024-08-17T18:15:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0001594 2024-08-17T18:15:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:15:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207868 56.10 56:0f591836:::benchmark_data_smithi136_356456_object81164:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:15:17.632 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.638177+0000", 2024-08-17T18:15:17.632 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 157.84193100799999, 2024-08-17T18:15:17.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4578162130000001, 2024-08-17T18:15:17.632 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:15:17.632 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:15:17.632 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:17.632 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:15:17.632 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:15:17.632 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:17.632 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207868 2024-08-17T18:15:17.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.632 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:15:17.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:15:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638177+0000", 2024-08-17T18:15:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:15:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638177+0000", 2024-08-17T18:15:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:15:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638177+0000", 2024-08-17T18:15:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1600000000000002e-07 2024-08-17T18:15:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:15:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638179+0000", 2024-08-17T18:15:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9870000000000002e-06 2024-08-17T18:15:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:15:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638186+0000", 2024-08-17T18:15:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0679999999999999e-06 2024-08-17T18:15:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:15:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638192+0000", 2024-08-17T18:15:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2159999999999997e-06 2024-08-17T18:15:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638244+0000", 2024-08-17T18:15:17.635 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:15:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:15:17] "GET /metrics HTTP/1.1" 200 36695 "" "Prometheus/2.51.0" 2024-08-17T18:15:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2305000000000002e-05 2024-08-17T18:15:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:15:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638249+0000", 2024-08-17T18:15:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3630000000000002e-06 2024-08-17T18:15:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640355+0000", 2024-08-17T18:15:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002105928 2024-08-17T18:15:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640359+0000", 2024-08-17T18:15:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.495e-06 2024-08-17T18:15:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.029712+0000", 2024-08-17T18:15:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3893531050000001 2024-08-17T18:15:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:15:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.029797+0000", 2024-08-17T18:15:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5116000000000003e-05 2024-08-17T18:15:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:15:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.029932+0000", 2024-08-17T18:15:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013527600000000001 2024-08-17T18:15:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.031527+0000", 2024-08-17T18:15:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015951819999999999 2024-08-17T18:15:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:15:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.031537+0000", 2024-08-17T18:15:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1560000000000001e-06 2024-08-17T18:15:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.031564+0000", 2024-08-17T18:15:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7455999999999999e-05 2024-08-17T18:15:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.076994+0000", 2024-08-17T18:15:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.045429888000000002 2024-08-17T18:15:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.095815+0000", 2024-08-17T18:15:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018821457999999999 2024-08-17T18:15:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:15:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.095878+0000", 2024-08-17T18:15:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2148000000000002e-05 2024-08-17T18:15:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:15:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.095993+0000", 2024-08-17T18:15:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011565000000000001 2024-08-17T18:15:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:15:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207897 56.10 56:0cb54b06:::benchmark_data_smithi136_356456_object81193:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:15:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.638208+0000", 2024-08-17T18:15:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 157.84189964800001, 2024-08-17T18:15:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.458027999, 2024-08-17T18:15:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:15:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:15:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:15:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:15:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207897 2024-08-17T18:15:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:15:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:15:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638208+0000", 2024-08-17T18:15:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:15:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638208+0000", 2024-08-17T18:15:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:15:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638209+0000", 2024-08-17T18:15:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3000000000000001e-07 2024-08-17T18:15:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:15:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638211+0000", 2024-08-17T18:15:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.902e-06 2024-08-17T18:15:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:15:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638218+0000", 2024-08-17T18:15:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1539999999999996e-06 2024-08-17T18:15:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:15:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638223+0000", 2024-08-17T18:15:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4009999999999996e-06 2024-08-17T18:15:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638274+0000", 2024-08-17T18:15:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0943000000000003e-05 2024-08-17T18:15:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:15:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638279+0000", 2024-08-17T18:15:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8579999999999999e-06 2024-08-17T18:15:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640382+0000", 2024-08-17T18:15:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021030879999999999 2024-08-17T18:15:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640389+0000", 2024-08-17T18:15:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8199999999999999e-06 2024-08-17T18:15:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.030583+0000", 2024-08-17T18:15:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.390193805 2024-08-17T18:15:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:15:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.030654+0000", 2024-08-17T18:15:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1778000000000003e-05 2024-08-17T18:15:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:15:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.030800+0000", 2024-08-17T18:15:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000145296 2024-08-17T18:15:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:15:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.032454+0000", 2024-08-17T18:15:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001653821 2024-08-17T18:15:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.033404+0000", 2024-08-17T18:15:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00095007900000000005 2024-08-17T18:15:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.033438+0000", 2024-08-17T18:15:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4153000000000003e-05 2024-08-17T18:15:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.079229+0000", 2024-08-17T18:15:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.045791559000000003 2024-08-17T18:15:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.096073+0000", 2024-08-17T18:15:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016844066000000001 2024-08-17T18:15:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:15:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.096126+0000", 2024-08-17T18:15:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2318000000000003e-05 2024-08-17T18:15:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:15:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.096236+0000", 2024-08-17T18:15:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110528 2024-08-17T18:15:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:15:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207920 56.10 56:0b09234c:::benchmark_data_smithi136_356456_object81216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:15:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.658933+0000", 2024-08-17T18:15:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 157.821175055, 2024-08-17T18:15:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4375237670000001, 2024-08-17T18:15:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:15:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:15:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:15:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:15:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207920 2024-08-17T18:15:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:15:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:15:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.658933+0000", 2024-08-17T18:15:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:15:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.658933+0000", 2024-08-17T18:15:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:15:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.658934+0000", 2024-08-17T18:15:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6099999999999999e-07 2024-08-17T18:15:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:15:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.658938+0000", 2024-08-17T18:15:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2780000000000004e-06 2024-08-17T18:15:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:15:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.658950+0000", 2024-08-17T18:15:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2018000000000001e-05 2024-08-17T18:15:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:15:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.658957+0000", 2024-08-17T18:15:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1779999999999997e-06 2024-08-17T18:15:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.659004+0000", 2024-08-17T18:15:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6539999999999998e-05 2024-08-17T18:15:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.659011+0000", 2024-08-17T18:15:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8419999999999999e-06 2024-08-17T18:15:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.031602+0000", 2024-08-17T18:15:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.372591723 2024-08-17T18:15:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:15:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.031674+0000", 2024-08-17T18:15:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1555000000000002e-05 2024-08-17T18:15:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:15:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.031810+0000", 2024-08-17T18:15:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013635000000000001 2024-08-17T18:15:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.033469+0000", 2024-08-17T18:15:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001658318 2024-08-17T18:15:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:15:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.033476+0000", 2024-08-17T18:15:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4460000000000004e-06 2024-08-17T18:15:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.033508+0000", 2024-08-17T18:15:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1977999999999998e-05 2024-08-17T18:15:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.079279+0000", 2024-08-17T18:15:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.045770862000000002 2024-08-17T18:15:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.096279+0000", 2024-08-17T18:15:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017000576999999999 2024-08-17T18:15:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:15:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.096334+0000", 2024-08-17T18:15:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4858999999999998e-05 2024-08-17T18:15:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:15:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.096457+0000", 2024-08-17T18:15:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012238200000000001 2024-08-17T18:15:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:15:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207937 56.10 56:0c91c958:::benchmark_data_smithi136_356456_object81233:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:15:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.681079+0000", 2024-08-17T18:15:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 157.79902863500001, 2024-08-17T18:15:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4157404689999999, 2024-08-17T18:15:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:15:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:15:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:15:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:15:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207937 2024-08-17T18:15:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:15:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:15:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.681079+0000", 2024-08-17T18:15:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:15:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.681079+0000", 2024-08-17T18:15:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:15:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:15:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.681081+0000", 2024-08-17T18:15:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.268e-06 2024-08-17T18:15:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:15:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.681085+0000", 2024-08-17T18:15:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1400000000000002e-06 2024-08-17T18:15:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:15:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.681096+0000", 2024-08-17T18:15:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0818999999999999e-05 2024-08-17T18:15:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:15:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.681101+0000", 2024-08-17T18:15:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2619999999999998e-06 2024-08-17T18:15:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.681161+0000", 2024-08-17T18:15:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9818000000000002e-05 2024-08-17T18:15:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:15:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.681171+0000", 2024-08-17T18:15:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0093e-05 2024-08-17T18:15:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:15:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.032475+0000", 2024-08-17T18:15:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3513044779999999 2024-08-17T18:15:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:15:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.032553+0000", 2024-08-17T18:15:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7330999999999995e-05 2024-08-17T18:15:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:15:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.032721+0000", 2024-08-17T18:15:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016832500000000001 2024-08-17T18:15:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:15:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.034404+0000", 2024-08-17T18:15:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001683466 2024-08-17T18:15:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.035385+0000", 2024-08-17T18:15:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00098110499999999991 2024-08-17T18:15:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.035421+0000", 2024-08-17T18:15:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5408999999999999e-05 2024-08-17T18:15:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.079323+0000", 2024-08-17T18:15:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.043902651000000001 2024-08-17T18:15:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.096646+0000", 2024-08-17T18:15:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017322758000000001 2024-08-17T18:15:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:15:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.096707+0000", 2024-08-17T18:15:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0699999999999998e-05 2024-08-17T18:15:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:15:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:15:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:15:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.096820+0000", 2024-08-17T18:15:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112846 2024-08-17T18:15:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:15:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:15:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:15:17.666 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:15:17.666 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:17.697 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:17.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:17.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:17.783 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 70 16 103418 103402 92.311 39.875 0.00459585 0.0108094 2024-08-17T18:15:17.799 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:17.826 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:17.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:17.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:17.888 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:17 smithi136 ceph-mon[109861]: pgmap v3064: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+undersized+remapped, 1 active+recovery_wait+remapped, 2 activating+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 9 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 6.2 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 13 KiB/s rd, 34 MiB/s wr, 560 op/s; 34768/306348 objects degraded (11.349%); 19019/306348 objects misplaced (6.208%); 11 MiB/s, 172 objects/s recovering 2024-08-17T18:15:17.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:17 smithi136 ceph-mon[115078]: pgmap v3064: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+undersized+remapped, 1 active+recovery_wait+remapped, 2 activating+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 9 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 6.2 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 13 KiB/s rd, 34 MiB/s wr, 560 op/s; 34768/306348 objects degraded (11.349%); 19019/306348 objects misplaced (6.208%); 11 MiB/s, 172 objects/s recovering 2024-08-17T18:15:17.928 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:17.957 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:17.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:17.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:17 smithi165 ceph-mon[105517]: pgmap v3064: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+undersized+remapped, 1 active+recovery_wait+remapped, 2 activating+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 9 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 6.2 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 13 KiB/s rd, 34 MiB/s wr, 560 op/s; 34768/306348 objects degraded (11.349%); 19019/306348 objects misplaced (6.208%); 11 MiB/s, 172 objects/s recovering 2024-08-17T18:15:18.059 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:18.085 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:18.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:18.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:18.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:18.214 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:18.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:18.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:18.316 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:18.340 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:18.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:18.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:18.442 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:18.475 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:18.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:18.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:18.576 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:18.606 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:18.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:18.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:18.708 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:18.740 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:18.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:18.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:18.783 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 71 16 103965 103949 91.4924 34.1875 0.00398005 0.0109053 2024-08-17T18:15:18.842 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:18.867 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:18.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:18.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:18.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:18 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 34768/308067 objects degraded (11.286%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:18.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:18 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 34768/308067 objects degraded (11.286%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:18.969 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:18.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:18.998 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:18.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:18 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 34768/308067 objects degraded (11.286%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:19.065 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:15:19.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:19.131 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:19.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:19.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:19.233 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:19.263 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:19.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:19.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:19.365 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:19.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:19.395 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:19.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:19.496 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:19.522 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:19.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:19.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:19.624 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:19.650 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:19.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:19.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:19.752 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:19.782 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:19.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:19.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:19.783 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 72 16 104656 104640 90.8214 43.1875 0.00301514 0.0109865 2024-08-17T18:15:19.884 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:19.919 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:19.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:19.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:20.016 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:19 smithi136 ceph-mon[109861]: pgmap v3065: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 6.3 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 11 KiB/s rd, 32 MiB/s wr, 513 op/s; 34768/308067 objects degraded (11.286%); 19123/308067 objects misplaced (6.207%); 6.2 MiB/s, 98 objects/s recovering 2024-08-17T18:15:20.017 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:19 smithi165 ceph-mon[105517]: pgmap v3065: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 6.3 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 11 KiB/s rd, 32 MiB/s wr, 513 op/s; 34768/308067 objects degraded (11.286%); 19123/308067 objects misplaced (6.207%); 6.2 MiB/s, 98 objects/s recovering 2024-08-17T18:15:20.017 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:19 smithi136 ceph-mon[115078]: pgmap v3065: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 6.3 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 11 KiB/s rd, 32 MiB/s wr, 513 op/s; 34768/308067 objects degraded (11.286%); 19123/308067 objects misplaced (6.207%); 6.2 MiB/s, 98 objects/s recovering 2024-08-17T18:15:20.020 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:20.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:20.045 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:20.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:20.146 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:20.185 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:20.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:20.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:20.287 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:20.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:20.318 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:20.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:20.418 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:20.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:20.453 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:20.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:20.554 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:20.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:20.582 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:20.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:20.683 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:20.722 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:20.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:20.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:20.784 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 73 16 105361 105345 90.1809 44.0625 0.00455642 0.0110676 2024-08-17T18:15:20.824 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:20.849 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:20.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:20.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:20.952 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:20.981 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:20.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:20.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:21.083 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:21.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:21.114 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:21.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:21.215 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:21.247 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:21.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:21.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:21.349 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:21.372 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:21.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:21.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:21.474 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:21.506 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:21.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:21.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:21.609 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:21.639 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:21.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:21.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:21.741 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:21.768 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:21.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:21.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:21.784 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 74 16 106043 106027 89.5382 42.625 0.00932623 0.0111478 2024-08-17T18:15:21.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:21.901 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:21.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:21.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:22.003 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:22.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:21 smithi165 ceph-mon[105517]: pgmap v3066: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 6.3 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 9.2 KiB/s rd, 23 MiB/s wr, 380 op/s; 34768/310296 objects degraded (11.205%); 19123/310296 objects misplaced (6.163%); 5.2 MiB/s, 82 objects/s recovering 2024-08-17T18:15:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:15:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:15:22.026 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:22.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:22.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:22.128 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:22.156 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:22.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:22.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:22.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:21 smithi136 ceph-mon[115078]: pgmap v3066: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 6.3 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 9.2 KiB/s rd, 23 MiB/s wr, 380 op/s; 34768/310296 objects degraded (11.205%); 19123/310296 objects misplaced (6.163%); 5.2 MiB/s, 82 objects/s recovering 2024-08-17T18:15:22.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:15:22.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:15:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:21 smithi136 ceph-mon[109861]: pgmap v3066: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 6.3 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 9.2 KiB/s rd, 23 MiB/s wr, 380 op/s; 34768/310296 objects degraded (11.205%); 19123/310296 objects misplaced (6.163%); 5.2 MiB/s, 82 objects/s recovering 2024-08-17T18:15:22.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:15:22.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:15:22.258 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:22.285 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:22.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:22.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:22.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:22.416 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:22.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:22.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:22.518 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:22.548 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:22.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:22.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:22.650 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:22.674 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:22.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:22.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:22.776 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:22.783 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 75 16 106685 106669 88.8794 40.125 0.00502664 0.0112322 2024-08-17T18:15:22.810 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:22.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:22.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:22.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:22.941 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:22.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:22.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:23.043 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:23.073 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:23.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:23.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:23.175 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:23.202 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:23.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:23.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:23.304 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:23.335 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:23.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:23.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:23.437 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:23.463 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:23.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:23.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:23.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:23.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:23.602 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:23.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:23.703 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:23.731 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:23.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:23.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:23.784 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 76 16 107327 107311 88.2379 40.125 0.00502066 0.0113157 2024-08-17T18:15:23.833 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:23.867 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:23.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:23.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:23.969 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:23.992 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:23.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:23.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:24.094 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:24.118 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:24.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:24.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:24.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:23 smithi136 ceph-mon[115078]: pgmap v3067: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 6.4 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 23 MiB/s wr, 373 op/s; 34768/314853 objects degraded (11.043%); 18380/314853 objects misplaced (5.838%); 7.9 MiB/s, 126 objects/s recovering 2024-08-17T18:15:24.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:23 smithi136 ceph-mon[109861]: pgmap v3067: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 6.4 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 23 MiB/s wr, 373 op/s; 34768/314853 objects degraded (11.043%); 18380/314853 objects misplaced (5.838%); 7.9 MiB/s, 126 objects/s recovering 2024-08-17T18:15:24.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:24.257 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:24.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:24.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:24.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:23 smithi165 ceph-mon[105517]: pgmap v3067: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 6.4 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 23 MiB/s wr, 373 op/s; 34768/314853 objects degraded (11.043%); 18380/314853 objects misplaced (5.838%); 7.9 MiB/s, 126 objects/s recovering 2024-08-17T18:15:24.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:24.397 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:24.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:24.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:24.499 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:24.534 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:24.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:24.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:24.636 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:24.660 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:24.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:24.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:24.762 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:24.784 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 77 16 107907 107891 87.5627 36.25 0.0504632 0.0113943 2024-08-17T18:15:24.790 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:24.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:24.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:24.892 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:24.926 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:24.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:24.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:25.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:25.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:25.061 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:25.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:25.162 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:25.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:24 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 34768/314853 objects degraded (11.043%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:25.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:24 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 34768/314853 objects degraded (11.043%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:25.198 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:25.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:25.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:25.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:24 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 34768/314853 objects degraded (11.043%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:25.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:25.324 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:25.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:25.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:25.426 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:25.450 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:25.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:25.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:25.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:25.575 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:25.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:25.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:25.677 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:25.697 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-17T18:15:25.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:25.704 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:25.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:25.784 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 78 16 108488 108472 86.9056 36.3125 0.00663869 0.0114742 2024-08-17T18:15:25.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:25.838 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:25.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:25.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:25.940 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:25.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:25 smithi136 ceph-mon[109861]: pgmap v3068: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 6.4 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 25 MiB/s wr, 404 op/s; 34768/316965 objects degraded (10.969%); 18483/316965 objects misplaced (5.831%); 7.2 MiB/s, 114 objects/s recovering 2024-08-17T18:15:25.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:25 smithi136 ceph-mon[115078]: pgmap v3068: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 6.4 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 25 MiB/s wr, 404 op/s; 34768/316965 objects degraded (10.969%); 18483/316965 objects misplaced (5.831%); 7.2 MiB/s, 114 objects/s recovering 2024-08-17T18:15:25.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:25.965 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:25.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:25 smithi165 ceph-mon[105517]: pgmap v3068: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 6.4 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 25 MiB/s wr, 404 op/s; 34768/316965 objects degraded (10.969%); 18483/316965 objects misplaced (5.831%); 7.2 MiB/s, 114 objects/s recovering 2024-08-17T18:15:26.066 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:26.102 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:26.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:26.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:26.205 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:26.229 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:26.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:26.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:26.331 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:26.347 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:15:26.356 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:26.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:26.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:26.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:26.495 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:26.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:26.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:26.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:26.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:26.621 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:26.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:26.722 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:26.743 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:26.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:26.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:26.784 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 79 16 109047 109031 86.2478 34.9375 0.00699211 0.0115703 2024-08-17T18:15:26.846 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:26.873 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:26.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:26.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:26.975 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:27.000 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:27.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:27.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:27.102 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:27.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:27.133 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:27.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:27.234 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:27.257 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:27.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:27.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:27.359 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:27.383 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:27.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:27.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:27.485 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:27.510 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:27.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:27.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:27.612 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:27.639 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:27.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:27.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:27.741 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:27.765 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:27.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:27.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:27.784 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:2024-08-17T18:15:27.784580+0000 min lat: 0.00125927 max lat: 4.02884 avg lat: 0.0116267 2024-08-17T18:15:27.784 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T18:15:27.784 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 80 16 109578 109562 85.5845 33.1875 0.00269255 0.0116267 2024-08-17T18:15:27.796 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:15:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:15:27] "GET /metrics HTTP/1.1" 200 36739 "" "Prometheus/2.51.0" 2024-08-17T18:15:27.867 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:27.908 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:27.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:27.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:28.009 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:28.034 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:28.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:28.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:28.079 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:27 smithi165 ceph-mon[105517]: pgmap v3069: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 6.5 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 31 MiB/s wr, 489 op/s; 34768/321546 objects degraded (10.813%); 17085/321546 objects misplaced (5.313%); 11 MiB/s, 168 objects/s recovering 2024-08-17T18:15:28.079 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:27 smithi165 ceph-mon[105517]: osdmap e1631: 8 total, 8 up, 5 in 2024-08-17T18:15:28.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:28.176 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:28.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:28.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:28.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:27 smithi136 ceph-mon[109861]: pgmap v3069: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 6.5 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 31 MiB/s wr, 489 op/s; 34768/321546 objects degraded (10.813%); 17085/321546 objects misplaced (5.313%); 11 MiB/s, 168 objects/s recovering 2024-08-17T18:15:28.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:27 smithi136 ceph-mon[109861]: osdmap e1631: 8 total, 8 up, 5 in 2024-08-17T18:15:28.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:27 smithi136 ceph-mon[115078]: pgmap v3069: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 11 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 6.5 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 31 MiB/s wr, 489 op/s; 34768/321546 objects degraded (10.813%); 17085/321546 objects misplaced (5.313%); 11 MiB/s, 168 objects/s recovering 2024-08-17T18:15:28.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:27 smithi136 ceph-mon[115078]: osdmap e1631: 8 total, 8 up, 5 in 2024-08-17T18:15:28.278 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:28.303 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:28.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:28.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:28.405 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:28.443 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:28.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:28.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:28.546 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:28.569 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:15:28.570 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:15:28.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.570 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:110095 57.12 57.cd78cd72 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1632)", 2024-08-17T18:15:28.570 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:28.450116+0000", 2024-08-17T18:15:28.570 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.11611826, 2024-08-17T18:15:28.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.11614478, 2024-08-17T18:15:28.571 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:28.571 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-08-17T18:15:28.571 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:28.571 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:15:28.572 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:15:28.572 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 110095 2024-08-17T18:15:28.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.572 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:28.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:28.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.450116+0000", 2024-08-17T18:15:28.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:28.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.450116+0000", 2024-08-17T18:15:28.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:28.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.450114+0000", 2024-08-17T18:15:28.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:15:28.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:28.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.450182+0000", 2024-08-17T18:15:28.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7867000000000006e-05 2024-08-17T18:15:28.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:28.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.450195+0000", 2024-08-17T18:15:28.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3386e-05 2024-08-17T18:15:28.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.450204+0000", 2024-08-17T18:15:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4160000000000004e-06 2024-08-17T18:15:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:110132 57.a e1632/1608 57:55ca6255:::benchmark_data_smithi136_389235_object110131:head v 1632'6944, mlcod=1632'6937)", 2024-08-17T18:15:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:28.507814+0000", 2024-08-17T18:15:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.058420413999999997, 2024-08-17T18:15:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.058507528000000003, 2024-08-17T18:15:28.578 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:28.579 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:15:28.579 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:28.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:28.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.507814+0000", 2024-08-17T18:15:28.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:28.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.507814+0000", 2024-08-17T18:15:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.507814+0000", 2024-08-17T18:15:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:15:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.507835+0000", 2024-08-17T18:15:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1814000000000002e-05 2024-08-17T18:15:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.507843+0000", 2024-08-17T18:15:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2930000000000003e-06 2024-08-17T18:15:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.509054+0000", 2024-08-17T18:15:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001211307 2024-08-17T18:15:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.525295+0000", 2024-08-17T18:15:28.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016240772000000001 2024-08-17T18:15:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:15:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.525312+0000", 2024-08-17T18:15:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6759999999999999e-05 2024-08-17T18:15:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.582 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:110166 57.6 e1632/1627 57:64e3253a:::benchmark_data_smithi136_389235_object110165:head v 1632'5842, mlcod=0'0)", 2024-08-17T18:15:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:28.559799+0000", 2024-08-17T18:15:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0064353680000000003, 2024-08-17T18:15:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006569961, 2024-08-17T18:15:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:15:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.559799+0000", 2024-08-17T18:15:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:28.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.559799+0000", 2024-08-17T18:15:28.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:28.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.559798+0000", 2024-08-17T18:15:28.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:15:28.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.559826+0000", 2024-08-17T18:15:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8225000000000002e-05 2024-08-17T18:15:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.559838+0000", 2024-08-17T18:15:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1915e-05 2024-08-17T18:15:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:28.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.559852+0000", 2024-08-17T18:15:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4293e-05 2024-08-17T18:15:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.559905+0000", 2024-08-17T18:15:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2457999999999998e-05 2024-08-17T18:15:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:15:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.559930+0000", 2024-08-17T18:15:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4901000000000001e-05 2024-08-17T18:15:28.586 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:110168 57.6 e1632/1627 57:6709dce7:::benchmark_data_smithi136_389235_object110167:head v 1632'5843, mlcod=0'0)", 2024-08-17T18:15:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:28.562873+0000", 2024-08-17T18:15:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0033615799999999999, 2024-08-17T18:15:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035405279999999998, 2024-08-17T18:15:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:15:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:28.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.562873+0000", 2024-08-17T18:15:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.562873+0000", 2024-08-17T18:15:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.562872+0000", 2024-08-17T18:15:28.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:15:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.562903+0000", 2024-08-17T18:15:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0825999999999999e-05 2024-08-17T18:15:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.562918+0000", 2024-08-17T18:15:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5773999999999998e-05 2024-08-17T18:15:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.562925+0000", 2024-08-17T18:15:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.058e-06 2024-08-17T18:15:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.565048+0000", 2024-08-17T18:15:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021220840000000002 2024-08-17T18:15:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:15:28.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.565090+0000", 2024-08-17T18:15:28.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2488999999999999e-05 2024-08-17T18:15:28.591 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.591 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:28.591 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.591 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:110139 57.a e1632/1608 57:5b7abfbd:::benchmark_data_smithi136_389235_object110138:head v 1632'6945, mlcod=1632'6940)", 2024-08-17T18:15:28.591 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:28.518273+0000", 2024-08-17T18:15:28.591 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.047960905999999998, 2024-08-17T18:15:28.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.048188662, 2024-08-17T18:15:28.592 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:28.592 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:15:28.592 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:28.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:28.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.518273+0000", 2024-08-17T18:15:28.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.518273+0000", 2024-08-17T18:15:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.518272+0000", 2024-08-17T18:15:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:15:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.518303+0000", 2024-08-17T18:15:28.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1044000000000003e-05 2024-08-17T18:15:28.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:28.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.518314+0000", 2024-08-17T18:15:28.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1037e-05 2024-08-17T18:15:28.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:28.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.518320+0000", 2024-08-17T18:15:28.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7989999999999999e-06 2024-08-17T18:15:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.527637+0000", 2024-08-17T18:15:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0093170790000000007 2024-08-17T18:15:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:15:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.527658+0000", 2024-08-17T18:15:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1019e-05 2024-08-17T18:15:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:28.595 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:109966 57.12 57:4b3f7e69:::benchmark_data_smithi136_389235_object109965:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1632)", 2024-08-17T18:15:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:28.273495+0000", 2024-08-17T18:15:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.29273903299999998, 2024-08-17T18:15:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.29301940399999998, 2024-08-17T18:15:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "waiting for sub ops", 2024-08-17T18:15:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:15:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:15:28.596 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 109966 2024-08-17T18:15:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.273495+0000", 2024-08-17T18:15:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.273495+0000", 2024-08-17T18:15:28.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.273494+0000", 2024-08-17T18:15:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:15:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.273565+0000", 2024-08-17T18:15:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0711999999999997e-05 2024-08-17T18:15:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:28.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.273578+0000", 2024-08-17T18:15:28.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3356999999999999e-05 2024-08-17T18:15:28.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:28.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.273585+0000", 2024-08-17T18:15:28.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8020000000000003e-06 2024-08-17T18:15:28.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:28.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.511228+0000", 2024-08-17T18:15:28.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.23764331499999999 2024-08-17T18:15:28.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:15:28.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.529048+0000", 2024-08-17T18:15:28.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017819156999999999 2024-08-17T18:15:28.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:15:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.547945+0000", 2024-08-17T18:15:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018897481000000001 2024-08-17T18:15:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:110149 57.12 57.40facf72 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1632)", 2024-08-17T18:15:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:28.527772+0000", 2024-08-17T18:15:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.038462598000000001, 2024-08-17T18:15:28.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.038808329000000003, 2024-08-17T18:15:28.602 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:28.602 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-08-17T18:15:28.602 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:28.603 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:15:28.603 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:15:28.603 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 110149 2024-08-17T18:15:28.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.603 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:28.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:28.604 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.527772+0000", 2024-08-17T18:15:28.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:28.604 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.527772+0000", 2024-08-17T18:15:28.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.527771+0000", 2024-08-17T18:15:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:15:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.527801+0000", 2024-08-17T18:15:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0267e-05 2024-08-17T18:15:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.527813+0000", 2024-08-17T18:15:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1708e-05 2024-08-17T18:15:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.527819+0000", 2024-08-17T18:15:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5849999999999996e-06 2024-08-17T18:15:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.606 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:28.607 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.607 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:110078 57.12 57.6f83be92 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1632)", 2024-08-17T18:15:28.607 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:28.413503+0000", 2024-08-17T18:15:28.607 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.152731067, 2024-08-17T18:15:28.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15312588699999999, 2024-08-17T18:15:28.607 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:28.608 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-08-17T18:15:28.608 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:28.608 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:15:28.608 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:15:28.608 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 110078 2024-08-17T18:15:28.608 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.608 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:28.608 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:28.609 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.413503+0000", 2024-08-17T18:15:28.609 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:28.609 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.413503+0000", 2024-08-17T18:15:28.609 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.610 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.610 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:28.610 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.413502+0000", 2024-08-17T18:15:28.610 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:15:28.610 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.610 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.610 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:28.610 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.413547+0000", 2024-08-17T18:15:28.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5204000000000003e-05 2024-08-17T18:15:28.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.612 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:28.612 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.413561+0000", 2024-08-17T18:15:28.612 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4694e-05 2024-08-17T18:15:28.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.612 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.612 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:28.612 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.413571+0000", 2024-08-17T18:15:28.612 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8770000000000005e-06 2024-08-17T18:15:28.612 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.612 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:28.612 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.613 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.613 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:110048 57.12 57.9f7e3152 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1632)", 2024-08-17T18:15:28.613 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:28.375040+0000", 2024-08-17T18:15:28.613 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.19119386799999999, 2024-08-17T18:15:28.613 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.191646183, 2024-08-17T18:15:28.613 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:28.613 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-08-17T18:15:28.613 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:28.614 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:15:28.614 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:15:28.614 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 110048 2024-08-17T18:15:28.614 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.614 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:28.614 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.614 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:28.614 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.375040+0000", 2024-08-17T18:15:28.614 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.615 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:28.615 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.375040+0000", 2024-08-17T18:15:28.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.615 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:28.615 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.375039+0000", 2024-08-17T18:15:28.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:15:28.616 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.616 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:28.616 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.375167+0000", 2024-08-17T18:15:28.616 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000128384 2024-08-17T18:15:28.616 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.616 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:28.616 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.375181+0000", 2024-08-17T18:15:28.616 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3889e-05 2024-08-17T18:15:28.616 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:28.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.375189+0000", 2024-08-17T18:15:28.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5000000000000002e-06 2024-08-17T18:15:28.617 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.617 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:28.617 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.617 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.617 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:110152 57.2 e1632/1627 57:40d230c7:::benchmark_data_smithi136_389235_object110151:head v 1632'5894, mlcod=0'0)", 2024-08-17T18:15:28.617 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:28.533021+0000", 2024-08-17T18:15:28.617 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.033213075000000002, 2024-08-17T18:15:28.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.033719486, 2024-08-17T18:15:28.618 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:28.618 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:15:28.618 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:28.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:28.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.533021+0000", 2024-08-17T18:15:28.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:28.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.533021+0000", 2024-08-17T18:15:28.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:28.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.533020+0000", 2024-08-17T18:15:28.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:15:28.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:28.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.533364+0000", 2024-08-17T18:15:28.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034367499999999999 2024-08-17T18:15:28.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.620 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:28.620 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.533383+0000", 2024-08-17T18:15:28.620 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8632999999999999e-05 2024-08-17T18:15:28.620 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.620 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.620 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:28.620 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.533391+0000", 2024-08-17T18:15:28.620 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0169999999999996e-06 2024-08-17T18:15:28.620 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.620 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.620 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:28.620 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.560993+0000", 2024-08-17T18:15:28.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.027602044999999999 2024-08-17T18:15:28.621 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.621 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.621 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:15:28.621 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.561043+0000", 2024-08-17T18:15:28.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0735999999999997e-05 2024-08-17T18:15:28.621 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.621 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:28.621 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.621 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.621 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.621 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:110117 57.12 57.8e6199f2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1632)", 2024-08-17T18:15:28.622 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:28.486933+0000", 2024-08-17T18:15:28.622 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.079301369999999996, 2024-08-17T18:15:28.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.079864347000000002, 2024-08-17T18:15:28.622 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:28.622 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-08-17T18:15:28.622 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:28.622 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:15:28.622 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:15:28.622 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 110117 2024-08-17T18:15:28.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.622 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:28.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.623 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:28.623 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.486933+0000", 2024-08-17T18:15:28.623 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.623 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.623 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:28.623 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.486933+0000", 2024-08-17T18:15:28.623 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.623 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.623 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:28.623 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.486932+0000", 2024-08-17T18:15:28.624 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:15:28.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.624 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.624 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:28.624 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.486994+0000", 2024-08-17T18:15:28.624 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1950000000000001e-05 2024-08-17T18:15:28.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.624 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.624 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:28.624 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.487005+0000", 2024-08-17T18:15:28.624 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1072e-05 2024-08-17T18:15:28.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.624 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.625 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:28.625 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.487011+0000", 2024-08-17T18:15:28.625 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1569999999999997e-06 2024-08-17T18:15:28.625 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.625 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:28.625 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.625 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.625 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.625 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:110060 57.12 57.3333d112 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1632)", 2024-08-17T18:15:28.625 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:28.388267+0000", 2024-08-17T18:15:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.17796773199999999, 2024-08-17T18:15:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17858684499999999, 2024-08-17T18:15:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-08-17T18:15:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:15:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:15:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 110060 2024-08-17T18:15:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:28.626 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.388267+0000", 2024-08-17T18:15:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.388267+0000", 2024-08-17T18:15:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.388265+0000", 2024-08-17T18:15:28.627 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:15:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.388329+0000", 2024-08-17T18:15:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4449000000000006e-05 2024-08-17T18:15:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.388342+0000", 2024-08-17T18:15:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.293e-05 2024-08-17T18:15:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.628 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.388349+0000", 2024-08-17T18:15:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1470000000000004e-06 2024-08-17T18:15:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.46277.0:109966 57.12 e1632/1627)", 2024-08-17T18:15:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:28.552082+0000", 2024-08-17T18:15:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.014152694, 2024-08-17T18:15:28.629 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014824333, 2024-08-17T18:15:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-08-17T18:15:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.552082+0000", 2024-08-17T18:15:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.552082+0000", 2024-08-17T18:15:28.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.552081+0000", 2024-08-17T18:15:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:15:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.552088+0000", 2024-08-17T18:15:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1269999999999997e-06 2024-08-17T18:15:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.552097+0000", 2024-08-17T18:15:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6020000000000006e-06 2024-08-17T18:15:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.552104+0000", 2024-08-17T18:15:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.46e-06 2024-08-17T18:15:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.632 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:109993 57.12 57:4ca9e046:::benchmark_data_smithi136_389235_object109992:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1632)", 2024-08-17T18:15:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:28.307913+0000", 2024-08-17T18:15:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.25832133200000001, 2024-08-17T18:15:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.259047056, 2024-08-17T18:15:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:15:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:15:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:15:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 109993 2024-08-17T18:15:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.633 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:28.634 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.634 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:28.634 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.307913+0000", 2024-08-17T18:15:28.634 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.634 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.634 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.634 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:28.634 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.307913+0000", 2024-08-17T18:15:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.307902+0000", 2024-08-17T18:15:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999886 2024-08-17T18:15:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.307947+0000", 2024-08-17T18:15:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5198999999999999e-05 2024-08-17T18:15:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.635 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.636 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:28.636 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.307961+0000", 2024-08-17T18:15:28.636 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4377e-05 2024-08-17T18:15:28.636 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.636 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.636 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:28.636 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.307968+0000", 2024-08-17T18:15:28.636 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.55e-06 2024-08-17T18:15:28.636 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.636 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.636 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.563561+0000", 2024-08-17T18:15:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25559362299999999 2024-08-17T18:15:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.46277.0:109966 57.12 e1632/1627)", 2024-08-17T18:15:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:28.557386+0000", 2024-08-17T18:15:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0088480529999999998, 2024-08-17T18:15:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.009626842, 2024-08-17T18:15:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:28.637 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-08-17T18:15:28.638 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:28.638 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.638 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:28.638 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.557386+0000", 2024-08-17T18:15:28.638 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.638 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.638 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.638 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:28.638 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.557386+0000", 2024-08-17T18:15:28.639 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:28.639 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.639 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.639 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:28.639 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.557385+0000", 2024-08-17T18:15:28.639 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:15:28.640 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.640 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.640 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:28.640 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.557394+0000", 2024-08-17T18:15:28.640 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0319999999999992e-06 2024-08-17T18:15:28.640 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.640 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.557410+0000", 2024-08-17T18:15:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6317e-05 2024-08-17T18:15:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:28.557418+0000", 2024-08-17T18:15:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1550000000000007e-06 2024-08-17T18:15:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.641 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:28.642 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.642 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:28.642 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-08-17T18:15:28.642 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 15 2024-08-17T18:15:28.642 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:15:28.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:28.643 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:28.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:28.744 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:28.775 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:28.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:28.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:28.784 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 81 16 110302 110286 85.0866 45.25 0.00368762 0.011728 2024-08-17T18:15:28.878 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:28.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:28.903 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:28.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:29.004 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:29.034 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:29.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:29.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:29.055 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:28 smithi165 ceph-mon[105517]: osdmap e1632: 8 total, 8 up, 5 in 2024-08-17T18:15:29.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:29.167 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:29.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:29.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:29.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:28 smithi136 ceph-mon[109861]: osdmap e1632: 8 total, 8 up, 5 in 2024-08-17T18:15:29.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:28 smithi136 ceph-mon[115078]: osdmap e1632: 8 total, 8 up, 5 in 2024-08-17T18:15:29.269 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:29.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:29.302 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:29.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:29.397 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-17T18:15:29.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:29.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:29.471 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:29.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:29.572 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:29.597 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:29.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:29.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:29.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:29.708 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:15:29.723 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:29.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:29.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:29.784 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 82 16 110906 110890 84.5093 37.75 0.00519423 0.0118074 2024-08-17T18:15:29.825 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:29.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:29.855 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:29.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:29.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:29.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:29.984 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:29.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:30.085 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:30.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:30.114 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:30.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:30.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:29 smithi165 ceph-mon[105517]: pgmap v3072: 109 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 6.6 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 41 MiB/s wr, 662 op/s; 31595/326208 objects degraded (9.686%); 20446/326208 objects misplaced (6.268%); 11 MiB/s, 182 objects/s recovering 2024-08-17T18:15:30.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:29 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 31595/326208 objects degraded (9.686%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:29 smithi136 ceph-mon[109861]: pgmap v3072: 109 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 6.6 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 41 MiB/s wr, 662 op/s; 31595/326208 objects degraded (9.686%); 20446/326208 objects misplaced (6.268%); 11 MiB/s, 182 objects/s recovering 2024-08-17T18:15:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:29 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 31595/326208 objects degraded (9.686%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:30.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:29 smithi136 ceph-mon[115078]: pgmap v3072: 109 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 6.6 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 41 MiB/s wr, 662 op/s; 31595/326208 objects degraded (9.686%); 20446/326208 objects misplaced (6.268%); 11 MiB/s, 182 objects/s recovering 2024-08-17T18:15:30.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:29 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 31595/326208 objects degraded (9.686%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:30.215 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:30.242 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:30.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:30.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:30.344 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:30.371 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:30.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:30.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:30.472 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:30.495 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:30.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:30.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:30.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:30.622 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:30.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:30.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:30.724 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:30.749 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:30.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:30.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:30.784 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 83 16 111535 111519 83.9647 39.3125 0.00828749 0.0118856 2024-08-17T18:15:30.851 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:30.875 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:30.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:30.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:30.977 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:31.008 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:31.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:31.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:31.109 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:31.134 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:31.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:31.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:31.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:31.269 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:31.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:31.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:31.371 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:31.401 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:31.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:31.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:31.503 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:31.529 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:31.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:31.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:31.631 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:31.653 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:31.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:31.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:31.755 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:31.784 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 84 16 112176 112160 83.442 40.0625 0.00936927 0.0119647 2024-08-17T18:15:31.785 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:31.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:31.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:31.887 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:31.910 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:31.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:31.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:31 smithi165 ceph-mon[105517]: pgmap v3073: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.7 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 37 MiB/s wr, 594 op/s; 31594/329127 objects degraded (9.599%); 20416/329127 objects misplaced (6.203%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T18:15:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:31 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2459519318' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:15:32.012 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:32.035 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:32.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:32.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:32.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:31 smithi136 ceph-mon[109861]: pgmap v3073: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.7 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 37 MiB/s wr, 594 op/s; 31594/329127 objects degraded (9.599%); 20416/329127 objects misplaced (6.203%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T18:15:32.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:31 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2459519318' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:15:32.080 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:31 smithi136 ceph-mon[115078]: pgmap v3073: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.7 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 37 MiB/s wr, 594 op/s; 31594/329127 objects degraded (9.599%); 20416/329127 objects misplaced (6.203%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T18:15:32.081 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:31 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2459519318' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:15:32.081 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:15:32.137 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:32.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:32.163 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:32.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:32.263 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:32.298 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:32.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:32.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:32.399 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:32.421 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:32.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:32.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:32.523 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:32.558 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:32.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:32.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:32.664 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:32.691 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:32.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:32.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:32.751 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:15:32.751 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:15:32.751 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-08-17T18:15:32.751 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-08-17T18:15:32.752 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:15:32.784 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 85 16 112778 112762 82.903 37.625 0.00968459 0.0120529 2024-08-17T18:15:32.793 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:32.818 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:32.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:32.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:32.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:32.957 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:32.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:32.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:32 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:15:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:32 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2459519318' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:15:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:32 smithi165 ceph-mon[105517]: osdmap e1633: 8 total, 8 up, 5 in 2024-08-17T18:15:33.059 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:33.077 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:15:33.093 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:33.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:33.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:33.195 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:33.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:32 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:15:33.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:32 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2459519318' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:15:33.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:32 smithi136 ceph-mon[109861]: osdmap e1633: 8 total, 8 up, 5 in 2024-08-17T18:15:33.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:32 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:15:33.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:32 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2459519318' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:15:33.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:32 smithi136 ceph-mon[115078]: osdmap e1633: 8 total, 8 up, 5 in 2024-08-17T18:15:33.224 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:33.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:33.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:33.326 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:33.350 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:33.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:33.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:33.452 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:33.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:33.485 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:33.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:33.586 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:33.611 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:33.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:33.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:33.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-17T18:15:33.713 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:33.741 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:33.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:33.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:33.785 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 86 16 113468 113452 82.4404 43.125 0.00444919 0.0121029 2024-08-17T18:15:33.843 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:33.871 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:33.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:33.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:33.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:33 smithi136 ceph-mon[109861]: pgmap v3075: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.8 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 41 MiB/s wr, 654 op/s; 31594/333336 objects degraded (9.478%); 20407/333336 objects misplaced (6.122%); 5.6 MiB/s, 89 objects/s recovering 2024-08-17T18:15:33.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:33 smithi136 ceph-mon[115078]: pgmap v3075: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.8 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 41 MiB/s wr, 654 op/s; 31594/333336 objects degraded (9.478%); 20407/333336 objects misplaced (6.122%); 5.6 MiB/s, 89 objects/s recovering 2024-08-17T18:15:33.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:33.979 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:15:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:33 smithi165 ceph-mon[105517]: pgmap v3075: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.8 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 41 MiB/s wr, 654 op/s; 31594/333336 objects degraded (9.478%); 20407/333336 objects misplaced (6.122%); 5.6 MiB/s, 89 objects/s recovering 2024-08-17T18:15:34.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:34.024 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:34.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:34.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:34.152 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:34.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:34.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:34.254 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:34.282 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:34.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:34.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:34.302 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:15:34.302 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-17T18:15:34.302 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:15:34.385 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:34.421 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:34.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:34.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:34.523 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:34.552 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:34.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:34.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:34.654 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:34.681 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:34.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:34.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:34.782 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:34.785 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 87 16 114107 114091 81.9519 39.9375 0.314285 0.0121808 2024-08-17T18:15:34.818 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:34.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:34.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:34.896 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:34 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 31594/333336 objects degraded (9.478%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:34.896 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:34 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 31594/333336 objects degraded (9.478%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:34.920 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:34.944 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:34.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:34.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:35.046 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:35.070 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:35.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:35.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:35.172 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:35.200 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:35.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:35.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:35.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:34 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 31594/333336 objects degraded (9.478%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:35.302 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:35.324 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:35.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:35.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:35.426 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:35.450 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:35.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:35.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:35.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:35.577 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:35.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:35.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:35.670 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:15:35.679 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:35.742 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:35.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:35.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:35.785 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 88 16 114835 114819 81.5376 45.5 0.006156 0.0122518 2024-08-17T18:15:35.844 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:35.872 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:35.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:35.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:35.974 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:35.997 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:15:35.997 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T18:15:35.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T18:15:35.998 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:15:35.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:35.998 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:178706 56.a 56:5b57367e:::benchmark_data_smithi136_356456_object52002:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:15:35.998 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:03.691091+0000", 2024-08-17T18:15:35.998 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 212.30334381899999, 2024-08-17T18:15:35.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.157627722, 2024-08-17T18:15:35.998 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:35.998 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:15:35.998 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:35.998 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:15:35.998 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:35.999 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 178706 2024-08-17T18:15:35.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:35.999 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:35.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:35.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:35.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691091+0000", 2024-08-17T18:15:35.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:35.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:35.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:35.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:35.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691091+0000", 2024-08-17T18:15:35.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:35.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:35.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:36.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691089+0000", 2024-08-17T18:15:36.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:15:36.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:36.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691096+0000", 2024-08-17T18:15:36.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.503e-06 2024-08-17T18:15:36.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:36.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691108+0000", 2024-08-17T18:15:36.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2129e-05 2024-08-17T18:15:36.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:36.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691117+0000", 2024-08-17T18:15:36.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4279999999999996e-06 2024-08-17T18:15:36.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.700684+0000", 2024-08-17T18:15:36.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0095666109999999992 2024-08-17T18:15:36.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.700694+0000", 2024-08-17T18:15:36.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0535e-05 2024-08-17T18:15:36.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.630895+0000", 2024-08-17T18:15:36.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93020063799999997 2024-08-17T18:15:36.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.630909+0000", 2024-08-17T18:15:36.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4681000000000001e-05 2024-08-17T18:15:36.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.846356+0000", 2024-08-17T18:15:36.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2154467900000001 2024-08-17T18:15:36.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:15:36.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.848718+0000", 2024-08-17T18:15:36.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002362276 2024-08-17T18:15:36.003 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.003 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:36.003 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.004 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:178708 56.f 56:fa635417:::benchmark_data_smithi136_356456_object52004:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:15:36.004 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:03.691130+0000", 2024-08-17T18:15:36.004 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 212.30330425700001, 2024-08-17T18:15:36.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.140675372, 2024-08-17T18:15:36.004 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:36.004 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:15:36.004 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:36.004 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:15:36.004 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:36.004 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 178708 2024-08-17T18:15:36.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.004 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:36.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:36.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691130+0000", 2024-08-17T18:15:36.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:36.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691130+0000", 2024-08-17T18:15:36.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:36.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691130+0000", 2024-08-17T18:15:36.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:15:36.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:36.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691133+0000", 2024-08-17T18:15:36.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0479999999999999e-06 2024-08-17T18:15:36.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:36.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691139+0000", 2024-08-17T18:15:36.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4860000000000002e-06 2024-08-17T18:15:36.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:36.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691144+0000", 2024-08-17T18:15:36.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0640000000000001e-06 2024-08-17T18:15:36.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.701188+0000", 2024-08-17T18:15:36.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010043593999999999 2024-08-17T18:15:36.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.701197+0000", 2024-08-17T18:15:36.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8060000000000001e-06 2024-08-17T18:15:36.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.630897+0000", 2024-08-17T18:15:36.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.92970020200000003 2024-08-17T18:15:36.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.630907+0000", 2024-08-17T18:15:36.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9119999999999993e-06 2024-08-17T18:15:36.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.829354+0000", 2024-08-17T18:15:36.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.198446897 2024-08-17T18:15:36.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:15:36.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.831806+0000", 2024-08-17T18:15:36.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002451815 2024-08-17T18:15:36.009 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.009 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:36.009 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.009 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:178716 56.a 56:5e090e51:::benchmark_data_smithi136_356456_object52012:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:15:36.009 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:03.691150+0000", 2024-08-17T18:15:36.009 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 212.303284096, 2024-08-17T18:15:36.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1748315850000002, 2024-08-17T18:15:36.010 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:36.010 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:36.010 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:36.010 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:15:36.010 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:36.010 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 178716 2024-08-17T18:15:36.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.010 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:36.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:36.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691150+0000", 2024-08-17T18:15:36.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:36.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691150+0000", 2024-08-17T18:15:36.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:36.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691150+0000", 2024-08-17T18:15:36.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:15:36.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:36.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691152+0000", 2024-08-17T18:15:36.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2120000000000002e-06 2024-08-17T18:15:36.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:36.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691158+0000", 2024-08-17T18:15:36.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:36.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8479999999999997e-06 2024-08-17T18:15:36.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:36.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691163+0000", 2024-08-17T18:15:36.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6840000000000004e-06 2024-08-17T18:15:36.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.700716+0000", 2024-08-17T18:15:36.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0095534420000000005 2024-08-17T18:15:36.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.700724+0000", 2024-08-17T18:15:36.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7659999999999998e-06 2024-08-17T18:15:36.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.630937+0000", 2024-08-17T18:15:36.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93021298399999996 2024-08-17T18:15:36.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.630947+0000", 2024-08-17T18:15:36.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9170000000000001e-06 2024-08-17T18:15:36.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.848942+0000", 2024-08-17T18:15:36.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.217995406 2024-08-17T18:15:36.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:15:36.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.849605+0000", 2024-08-17T18:15:36.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00066261399999999995 2024-08-17T18:15:36.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3,6", 2024-08-17T18:15:36.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.849886+0000", 2024-08-17T18:15:36.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000281437 2024-08-17T18:15:36.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.865742+0000", 2024-08-17T18:15:36.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015855622999999999 2024-08-17T18:15:36.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:15:36.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.865751+0000", 2024-08-17T18:15:36.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9230000000000008e-06 2024-08-17T18:15:36.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.865785+0000", 2024-08-17T18:15:36.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4532000000000001e-05 2024-08-17T18:15:36.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.865821+0000", 2024-08-17T18:15:36.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5692000000000003e-05 2024-08-17T18:15:36.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:15:36.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.865869+0000", 2024-08-17T18:15:36.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.816e-05 2024-08-17T18:15:36.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:15:36.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.865982+0000", 2024-08-17T18:15:36.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112749 2024-08-17T18:15:36.019 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.019 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:36.019 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.019 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:178717 56.f 56:f25e328a:::benchmark_data_smithi136_356456_object52013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:15:36.019 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:03.698538+0000", 2024-08-17T18:15:36.019 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 212.29589661399999, 2024-08-17T18:15:36.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.154307932, 2024-08-17T18:15:36.019 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:36.019 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:36.019 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:36.019 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:15:36.020 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:36.020 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 178717 2024-08-17T18:15:36.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.020 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:36.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:36.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.698538+0000", 2024-08-17T18:15:36.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:36.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.698538+0000", 2024-08-17T18:15:36.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:36.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.698537+0000", 2024-08-17T18:15:36.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:15:36.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:36.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.698548+0000", 2024-08-17T18:15:36.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0995000000000001e-05 2024-08-17T18:15:36.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:36.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.698558+0000", 2024-08-17T18:15:36.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.005e-05 2024-08-17T18:15:36.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:36.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.698564+0000", 2024-08-17T18:15:36.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.494e-06 2024-08-17T18:15:36.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.701220+0000", 2024-08-17T18:15:36.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026557120000000002 2024-08-17T18:15:36.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.701227+0000", 2024-08-17T18:15:36.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2200000000000003e-06 2024-08-17T18:15:36.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.630930+0000", 2024-08-17T18:15:36.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.92970317000000002 2024-08-17T18:15:36.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.630938+0000", 2024-08-17T18:15:36.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1289999999999996e-06 2024-08-17T18:15:36.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.832162+0000", 2024-08-17T18:15:36.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2012241939999999 2024-08-17T18:15:36.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:15:36.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.832939+0000", 2024-08-17T18:15:36.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00077643899999999997 2024-08-17T18:15:36.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1,6", 2024-08-17T18:15:36.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.833419+0000", 2024-08-17T18:15:36.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048046500000000002 2024-08-17T18:15:36.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:15:36.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.844527+0000", 2024-08-17T18:15:36.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011107637 2024-08-17T18:15:36.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.844562+0000", 2024-08-17T18:15:36.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5005000000000001e-05 2024-08-17T18:15:36.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.852662+0000", 2024-08-17T18:15:36.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080996279999999993 2024-08-17T18:15:36.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.852693+0000", 2024-08-17T18:15:36.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1347000000000003e-05 2024-08-17T18:15:36.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:15:36.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.852740+0000", 2024-08-17T18:15:36.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6626999999999999e-05 2024-08-17T18:15:36.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:15:36.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.852846+0000", 2024-08-17T18:15:36.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010621 2024-08-17T18:15:36.027 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.027 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:36.027 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.027 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:178728 56.f 56:faf5142a:::benchmark_data_smithi136_356456_object52024:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:15:36.027 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:03.722294+0000", 2024-08-17T18:15:36.027 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 212.27214058800001, 2024-08-17T18:15:36.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1390697379999999, 2024-08-17T18:15:36.028 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:36.028 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:36.028 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:36.028 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:15:36.028 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:36.028 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 178728 2024-08-17T18:15:36.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.028 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:36.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:36.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.722294+0000", 2024-08-17T18:15:36.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:36.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.722294+0000", 2024-08-17T18:15:36.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:36.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.722293+0000", 2024-08-17T18:15:36.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:15:36.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:36.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.722298+0000", 2024-08-17T18:15:36.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0760000000000002e-06 2024-08-17T18:15:36.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:36.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.722308+0000", 2024-08-17T18:15:36.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0288999999999999e-05 2024-08-17T18:15:36.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:36.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.722313+0000", 2024-08-17T18:15:36.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7650000000000001e-06 2024-08-17T18:15:36.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.722348+0000", 2024-08-17T18:15:36.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5262000000000002e-05 2024-08-17T18:15:36.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.722356+0000", 2024-08-17T18:15:36.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.396e-06 2024-08-17T18:15:36.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.630986+0000", 2024-08-17T18:15:36.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90863069100000005 2024-08-17T18:15:36.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.630995+0000", 2024-08-17T18:15:36.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1009999999999992e-06 2024-08-17T18:15:36.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.840435+0000", 2024-08-17T18:15:36.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.209439967 2024-08-17T18:15:36.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:15:36.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.840634+0000", 2024-08-17T18:15:36.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019990000000000001 2024-08-17T18:15:36.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1,6", 2024-08-17T18:15:36.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.840774+0000", 2024-08-17T18:15:36.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000139846 2024-08-17T18:15:36.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:15:36.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.844569+0000", 2024-08-17T18:15:36.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037951809999999999 2024-08-17T18:15:36.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.852881+0000", 2024-08-17T18:15:36.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.008311578 2024-08-17T18:15:36.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.852911+0000", 2024-08-17T18:15:36.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9935000000000001e-05 2024-08-17T18:15:36.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.859827+0000", 2024-08-17T18:15:36.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069164550000000002 2024-08-17T18:15:36.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:15:36.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.861216+0000", 2024-08-17T18:15:36.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013883140000000001 2024-08-17T18:15:36.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:15:36.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.861364+0000", 2024-08-17T18:15:36.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000147971 2024-08-17T18:15:36.036 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.036 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:36.036 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.036 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:178747 56.a 56:537dd351:::benchmark_data_smithi136_356456_object52043:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:15:36.036 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:03.749973+0000", 2024-08-17T18:15:36.036 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 212.24446188300001, 2024-08-17T18:15:36.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1268650779999998, 2024-08-17T18:15:36.036 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:36.036 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:36.036 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:36.037 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:15:36.037 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:36.037 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 178747 2024-08-17T18:15:36.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.037 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:36.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:36.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.749973+0000", 2024-08-17T18:15:36.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:36.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.749973+0000", 2024-08-17T18:15:36.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:36.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.749971+0000", 2024-08-17T18:15:36.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:15:36.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:36.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.749977+0000", 2024-08-17T18:15:36.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2750000000000001e-06 2024-08-17T18:15:36.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:36.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.749991+0000", 2024-08-17T18:15:36.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4019000000000001e-05 2024-08-17T18:15:36.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:36.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.749998+0000", 2024-08-17T18:15:36.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9999999999999999e-06 2024-08-17T18:15:36.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.750060+0000", 2024-08-17T18:15:36.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1975000000000002e-05 2024-08-17T18:15:36.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.750080+0000", 2024-08-17T18:15:36.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9573999999999999e-05 2024-08-17T18:15:36.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.630970+0000", 2024-08-17T18:15:36.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.88089031200000001 2024-08-17T18:15:36.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.630980+0000", 2024-08-17T18:15:36.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3239999999999992e-06 2024-08-17T18:15:36.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.862147+0000", 2024-08-17T18:15:36.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2311678589999999 2024-08-17T18:15:36.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:15:36.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.862289+0000", 2024-08-17T18:15:36.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000141089 2024-08-17T18:15:36.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3,6", 2024-08-17T18:15:36.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.862440+0000", 2024-08-17T18:15:36.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000151137 2024-08-17T18:15:36.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.873035+0000", 2024-08-17T18:15:36.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010595345000000001 2024-08-17T18:15:36.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.873076+0000", 2024-08-17T18:15:36.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0983999999999999e-05 2024-08-17T18:15:36.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.873111+0000", 2024-08-17T18:15:36.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4768999999999999e-05 2024-08-17T18:15:36.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:15:36.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.876641+0000", 2024-08-17T18:15:36.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035304680000000001 2024-08-17T18:15:36.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:15:36.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.876727+0000", 2024-08-17T18:15:36.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5708e-05 2024-08-17T18:15:36.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:15:36.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.876838+0000", 2024-08-17T18:15:36.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011078000000000001 2024-08-17T18:15:36.044 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.044 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:36.044 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.044 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:178752 56.a 56:57c711e2:::benchmark_data_smithi136_356456_object52048:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:15:36.044 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:03.755857+0000", 2024-08-17T18:15:36.044 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 212.23857758, 2024-08-17T18:15:36.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1242790989999998, 2024-08-17T18:15:36.045 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:36.045 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:36.045 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:36.045 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:15:36.045 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:36.045 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 178752 2024-08-17T18:15:36.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.045 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:36.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:36.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.755857+0000", 2024-08-17T18:15:36.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:36.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.755857+0000", 2024-08-17T18:15:36.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:36.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.755856+0000", 2024-08-17T18:15:36.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:15:36.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:36.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.755863+0000", 2024-08-17T18:15:36.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.396e-06 2024-08-17T18:15:36.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:36.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.755879+0000", 2024-08-17T18:15:36.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5313999999999999e-05 2024-08-17T18:15:36.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:36.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.755884+0000", 2024-08-17T18:15:36.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.682e-06 2024-08-17T18:15:36.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.755928+0000", 2024-08-17T18:15:36.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3137999999999999e-05 2024-08-17T18:15:36.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.755936+0000", 2024-08-17T18:15:36.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7790000000000001e-06 2024-08-17T18:15:36.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.631003+0000", 2024-08-17T18:15:36.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.87506654100000003 2024-08-17T18:15:36.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.631013+0000", 2024-08-17T18:15:36.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6539999999999994e-06 2024-08-17T18:15:36.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.866014+0000", 2024-08-17T18:15:36.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2350009879999999 2024-08-17T18:15:36.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:15:36.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.866130+0000", 2024-08-17T18:15:36.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011678100000000001 2024-08-17T18:15:36.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3,6", 2024-08-17T18:15:36.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.866256+0000", 2024-08-17T18:15:36.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012540000000000001 2024-08-17T18:15:36.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:15:36.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.876848+0000", 2024-08-17T18:15:36.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010592257000000001 2024-08-17T18:15:36.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.879118+0000", 2024-08-17T18:15:36.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022697989999999999 2024-08-17T18:15:36.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.879489+0000", 2024-08-17T18:15:36.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037104599999999998 2024-08-17T18:15:36.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.879974+0000", 2024-08-17T18:15:36.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048566899999999999 2024-08-17T18:15:36.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:15:36.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.880023+0000", 2024-08-17T18:15:36.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8986000000000002e-05 2024-08-17T18:15:36.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:15:36.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.880136+0000", 2024-08-17T18:15:36.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112607 2024-08-17T18:15:36.052 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.052 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:36.052 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.053 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:178758 56.f 56:f2f04132:::benchmark_data_smithi136_356456_object52054:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:15:36.053 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:03.765193+0000", 2024-08-17T18:15:36.053 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 212.22924162300001, 2024-08-17T18:15:36.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0967104569999999, 2024-08-17T18:15:36.053 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:36.053 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:36.053 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:36.053 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:15:36.053 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:36.053 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 178758 2024-08-17T18:15:36.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.053 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:36.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:36.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.765193+0000", 2024-08-17T18:15:36.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:36.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.765193+0000", 2024-08-17T18:15:36.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:36.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.765192+0000", 2024-08-17T18:15:36.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:15:36.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:36.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.765196+0000", 2024-08-17T18:15:36.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.454e-06 2024-08-17T18:15:36.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:36.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.765208+0000", 2024-08-17T18:15:36.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.173e-05 2024-08-17T18:15:36.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:36.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.765214+0000", 2024-08-17T18:15:36.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9970000000000004e-06 2024-08-17T18:15:36.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.765259+0000", 2024-08-17T18:15:36.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4753999999999999e-05 2024-08-17T18:15:36.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.057 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:36.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:36.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.765267+0000", 2024-08-17T18:15:36.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.729e-06 2024-08-17T18:15:36.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.631015+0000", 2024-08-17T18:15:36.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.86574868900000002 2024-08-17T18:15:36.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.631023+0000", 2024-08-17T18:15:36.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3039999999999999e-06 2024-08-17T18:15:36.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.844590+0000", 2024-08-17T18:15:36.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2135671560000001 2024-08-17T18:15:36.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:15:36.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.844710+0000", 2024-08-17T18:15:36.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120407 2024-08-17T18:15:36.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1,6", 2024-08-17T18:15:36.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.844840+0000", 2024-08-17T18:15:36.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012999800000000001 2024-08-17T18:15:36.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.853150+0000", 2024-08-17T18:15:36.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0083100050000000005 2024-08-17T18:15:36.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.853179+0000", 2024-08-17T18:15:36.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8564e-05 2024-08-17T18:15:36.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:15:36.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.861376+0000", 2024-08-17T18:15:36.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0081970010000000006 2024-08-17T18:15:36.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.861415+0000", 2024-08-17T18:15:36.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8921999999999998e-05 2024-08-17T18:15:36.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:15:36.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.861728+0000", 2024-08-17T18:15:36.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00031290799999999998 2024-08-17T18:15:36.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:15:36.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.861903+0000", 2024-08-17T18:15:36.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017579999999999999 2024-08-17T18:15:36.063 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.063 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:36.063 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.064 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:210135 56.a 56:568f5a1f:::benchmark_data_smithi136_356456_object83431:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1574)", 2024-08-17T18:15:36.064 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:44.732222+0000", 2024-08-17T18:15:36.064 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 171.26221236800001, 2024-08-17T18:15:36.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.95183570399999995, 2024-08-17T18:15:36.064 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:36.064 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:36.064 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:36.064 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:15:36.064 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:36.064 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 210135 2024-08-17T18:15:36.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.064 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:36.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:36.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.732222+0000", 2024-08-17T18:15:36.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:36.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.732222+0000", 2024-08-17T18:15:36.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:36.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.732221+0000", 2024-08-17T18:15:36.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:15:36.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:36.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.732226+0000", 2024-08-17T18:15:36.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1000000000000003e-06 2024-08-17T18:15:36.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:36.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.732236+0000", 2024-08-17T18:15:36.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0001e-05 2024-08-17T18:15:36.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:36.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.732243+0000", 2024-08-17T18:15:36.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6290000000000004e-06 2024-08-17T18:15:36.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.732289+0000", 2024-08-17T18:15:36.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5989999999999998e-05 2024-08-17T18:15:36.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.732297+0000", 2024-08-17T18:15:36.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4109999999999996e-06 2024-08-17T18:15:36.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.676550+0000", 2024-08-17T18:15:36.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94425278800000001 2024-08-17T18:15:36.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:15:36.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.676649+0000", 2024-08-17T18:15:36.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8399000000000004e-05 2024-08-17T18:15:36.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-08-17T18:15:36.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.676748+0000", 2024-08-17T18:15:36.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9660000000000005e-05 2024-08-17T18:15:36.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:15:36.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.681921+0000", 2024-08-17T18:15:36.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051730450000000002 2024-08-17T18:15:36.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.683899+0000", 2024-08-17T18:15:36.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019780750000000001 2024-08-17T18:15:36.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.683990+0000", 2024-08-17T18:15:36.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0943e-05 2024-08-17T18:15:36.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:15:36.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.684021+0000", 2024-08-17T18:15:36.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0533000000000001e-05 2024-08-17T18:15:36.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:15:36.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.684058+0000", 2024-08-17T18:15:36.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6937000000000002e-05 2024-08-17T18:15:36.070 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.070 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:36.071 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.071 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:210147 56.a 56:575af9e8:::benchmark_data_smithi136_356456_object83443:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1574)", 2024-08-17T18:15:36.071 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:44.739176+0000", 2024-08-17T18:15:36.071 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 171.25525898000001, 2024-08-17T18:15:36.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94500086599999999, 2024-08-17T18:15:36.071 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:36.071 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:36.071 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:36.071 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:15:36.071 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:36.071 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 210147 2024-08-17T18:15:36.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.072 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:36.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:36.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.739176+0000", 2024-08-17T18:15:36.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:36.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.739176+0000", 2024-08-17T18:15:36.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:36.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.739174+0000", 2024-08-17T18:15:36.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:15:36.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:36.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.739179+0000", 2024-08-17T18:15:36.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6840000000000004e-06 2024-08-17T18:15:36.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:36.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.739191+0000", 2024-08-17T18:15:36.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2330999999999999e-05 2024-08-17T18:15:36.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:36.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.739198+0000", 2024-08-17T18:15:36.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8090000000000004e-06 2024-08-17T18:15:36.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.739249+0000", 2024-08-17T18:15:36.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0825000000000002e-05 2024-08-17T18:15:36.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.739257+0000", 2024-08-17T18:15:36.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9929999999999995e-06 2024-08-17T18:15:36.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.679597+0000", 2024-08-17T18:15:36.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94033990499999998 2024-08-17T18:15:36.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:15:36.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.679709+0000", 2024-08-17T18:15:36.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011232500000000001 2024-08-17T18:15:36.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-08-17T18:15:36.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.679779+0000", 2024-08-17T18:15:36.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0005999999999993e-05 2024-08-17T18:15:36.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.684097+0000", 2024-08-17T18:15:36.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043174789999999999 2024-08-17T18:15:36.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:15:36.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.684105+0000", 2024-08-17T18:15:36.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0730000000000005e-06 2024-08-17T18:15:36.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.684133+0000", 2024-08-17T18:15:36.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8529000000000001e-05 2024-08-17T18:15:36.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:15:36.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.684158+0000", 2024-08-17T18:15:36.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4700999999999999e-05 2024-08-17T18:15:36.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:15:36.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.684176+0000", 2024-08-17T18:15:36.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8445e-05 2024-08-17T18:15:36.077 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.078 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:36.078 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.078 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:210148 56.a 56:59e80994:::benchmark_data_smithi136_356456_object83444:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1574)", 2024-08-17T18:15:36.078 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:44.739887+0000", 2024-08-17T18:15:36.078 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 171.254547898, 2024-08-17T18:15:36.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94443850399999996, 2024-08-17T18:15:36.078 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:36.078 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:36.078 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:36.078 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:15:36.078 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:36.078 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 210148 2024-08-17T18:15:36.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.079 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:36.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:36.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.739887+0000", 2024-08-17T18:15:36.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:36.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.739887+0000", 2024-08-17T18:15:36.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:36.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.739886+0000", 2024-08-17T18:15:36.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:15:36.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:36.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.739893+0000", 2024-08-17T18:15:36.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3320000000000003e-06 2024-08-17T18:15:36.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:36.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.739906+0000", 2024-08-17T18:15:36.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2771e-05 2024-08-17T18:15:36.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:36.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.739912+0000", 2024-08-17T18:15:36.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5239999999999997e-06 2024-08-17T18:15:36.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.739970+0000", 2024-08-17T18:15:36.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7323e-05 2024-08-17T18:15:36.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.739977+0000", 2024-08-17T18:15:36.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4889999999999998e-06 2024-08-17T18:15:36.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.681954+0000", 2024-08-17T18:15:36.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94197663600000003 2024-08-17T18:15:36.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:15:36.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.682019+0000", 2024-08-17T18:15:36.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5313000000000003e-05 2024-08-17T18:15:36.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-08-17T18:15:36.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.682095+0000", 2024-08-17T18:15:36.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6062999999999993e-05 2024-08-17T18:15:36.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.684222+0000", 2024-08-17T18:15:36.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021266280000000002 2024-08-17T18:15:36.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:15:36.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.684230+0000", 2024-08-17T18:15:36.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2500000000000006e-06 2024-08-17T18:15:36.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.684256+0000", 2024-08-17T18:15:36.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6395000000000001e-05 2024-08-17T18:15:36.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:15:36.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.684292+0000", 2024-08-17T18:15:36.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5206000000000001e-05 2024-08-17T18:15:36.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:15:36.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.684325+0000", 2024-08-17T18:15:36.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3472000000000001e-05 2024-08-17T18:15:36.085 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.085 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:36.085 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.085 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:210189 56.a 56:5c9823c1:::benchmark_data_smithi136_356456_object83485:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1574)", 2024-08-17T18:15:36.085 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:44.768638+0000", 2024-08-17T18:15:36.085 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 171.22579642400001, 2024-08-17T18:15:36.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.92899130399999996, 2024-08-17T18:15:36.085 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:36.086 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:36.086 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:36.086 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:15:36.086 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:36.086 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 210189 2024-08-17T18:15:36.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.086 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:36.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:36.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.768638+0000", 2024-08-17T18:15:36.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:36.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.768638+0000", 2024-08-17T18:15:36.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:36.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.768637+0000", 2024-08-17T18:15:36.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:15:36.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:36.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.768641+0000", 2024-08-17T18:15:36.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4100000000000001e-06 2024-08-17T18:15:36.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:36.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.768650+0000", 2024-08-17T18:15:36.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6850000000000007e-06 2024-08-17T18:15:36.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:36.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.768656+0000", 2024-08-17T18:15:36.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0009999999999998e-06 2024-08-17T18:15:36.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.768703+0000", 2024-08-17T18:15:36.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7101000000000002e-05 2024-08-17T18:15:36.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.768711+0000", 2024-08-17T18:15:36.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9279999999999993e-06 2024-08-17T18:15:36.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.684372+0000", 2024-08-17T18:15:36.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91566089799999995 2024-08-17T18:15:36.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:15:36.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.684482+0000", 2024-08-17T18:15:36.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000110021 2024-08-17T18:15:36.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-08-17T18:15:36.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.684903+0000", 2024-08-17T18:15:36.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042054499999999998 2024-08-17T18:15:36.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.691514+0000", 2024-08-17T18:15:36.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066113450000000002 2024-08-17T18:15:36.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:15:36.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.694302+0000", 2024-08-17T18:15:36.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027884659999999999 2024-08-17T18:15:36.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.697567+0000", 2024-08-17T18:15:36.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032646730000000001 2024-08-17T18:15:36.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:15:36.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.697596+0000", 2024-08-17T18:15:36.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8472000000000001e-05 2024-08-17T18:15:36.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:15:36.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.697629+0000", 2024-08-17T18:15:36.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3887000000000003e-05 2024-08-17T18:15:36.092 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.092 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:36.092 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.092 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:210197 56.a 56:53bbec3a:::benchmark_data_smithi136_356456_object83493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1574)", 2024-08-17T18:15:36.092 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:44.771363+0000", 2024-08-17T18:15:36.092 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 171.22307104999999, 2024-08-17T18:15:36.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.92638406900000003, 2024-08-17T18:15:36.093 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:36.093 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:36.093 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:36.093 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:15:36.093 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:36.093 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 210197 2024-08-17T18:15:36.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.093 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:36.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:36.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.771363+0000", 2024-08-17T18:15:36.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:36.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.771363+0000", 2024-08-17T18:15:36.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:36.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.771362+0000", 2024-08-17T18:15:36.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:15:36.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:36.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.771368+0000", 2024-08-17T18:15:36.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6099999999999997e-06 2024-08-17T18:15:36.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:36.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.771382+0000", 2024-08-17T18:15:36.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4076999999999999e-05 2024-08-17T18:15:36.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:36.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.771389+0000", 2024-08-17T18:15:36.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3329999999999999e-06 2024-08-17T18:15:36.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.771440+0000", 2024-08-17T18:15:36.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0997e-05 2024-08-17T18:15:36.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:44.771448+0000", 2024-08-17T18:15:36.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7789999999999996e-06 2024-08-17T18:15:36.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.688814+0000", 2024-08-17T18:15:36.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91736601600000001 2024-08-17T18:15:36.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:15:36.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.688910+0000", 2024-08-17T18:15:36.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5903000000000007e-05 2024-08-17T18:15:36.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,6", 2024-08-17T18:15:36.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.689027+0000", 2024-08-17T18:15:36.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011683700000000001 2024-08-17T18:15:36.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.697527+0000", 2024-08-17T18:15:36.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0084999329999999994 2024-08-17T18:15:36.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:15:36.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.697538+0000", 2024-08-17T18:15:36.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1321999999999999e-05 2024-08-17T18:15:36.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.697673+0000", 2024-08-17T18:15:36.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013547599999999999 2024-08-17T18:15:36.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:15:36.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.697717+0000", 2024-08-17T18:15:36.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3072999999999999e-05 2024-08-17T18:15:36.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:15:36.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:45.697748+0000", 2024-08-17T18:15:36.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0998999999999999e-05 2024-08-17T18:15:36.099 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.099 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:36.099 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.099 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:239325 56.19 56.f205c1f9 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e1596)", 2024-08-17T18:15:36.100 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:13:18.721318+0000", 2024-08-17T18:15:36.100 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 137.27311647400001, 2024-08-17T18:15:36.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.92796505799999995, 2024-08-17T18:15:36.100 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:36.100 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:15:36.100 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:36.100 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:15:36.100 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:36.100 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 239325 2024-08-17T18:15:36.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.100 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:36.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:36.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:18.721318+0000", 2024-08-17T18:15:36.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:36.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:18.721318+0000", 2024-08-17T18:15:36.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:36.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:18.721317+0000", 2024-08-17T18:15:36.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:15:36.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:36.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:18.721322+0000", 2024-08-17T18:15:36.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4020000000000004e-06 2024-08-17T18:15:36.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:36.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:18.721333+0000", 2024-08-17T18:15:36.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1416999999999999e-05 2024-08-17T18:15:36.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:36.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:18.721341+0000", 2024-08-17T18:15:36.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5920000000000003e-06 2024-08-17T18:15:36.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:18.721371+0000", 2024-08-17T18:15:36.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0548e-05 2024-08-17T18:15:36.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:18.721379+0000", 2024-08-17T18:15:36.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0099999999999995e-06 2024-08-17T18:15:36.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:19.649269+0000", 2024-08-17T18:15:36.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.92788948800000004 2024-08-17T18:15:36.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:15:36.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:19.649283+0000", 2024-08-17T18:15:36.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4403000000000001e-05 2024-08-17T18:15:36.104 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.104 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:36.104 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.104 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:239353 56.19 56.41c84b39 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1596)", 2024-08-17T18:15:36.104 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:13:18.735739+0000", 2024-08-17T18:15:36.104 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 137.25869592399999, 2024-08-17T18:15:36.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91358022999999999, 2024-08-17T18:15:36.105 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:36.105 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:15:36.105 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:36.105 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:15:36.105 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:36.105 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 239353 2024-08-17T18:15:36.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.105 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:36.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:36.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:18.735739+0000", 2024-08-17T18:15:36.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:36.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:18.735739+0000", 2024-08-17T18:15:36.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:36.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:18.735738+0000", 2024-08-17T18:15:36.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:15:36.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:36.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:18.735749+0000", 2024-08-17T18:15:36.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1817000000000001e-05 2024-08-17T18:15:36.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:36.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:18.735763+0000", 2024-08-17T18:15:36.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3407999999999999e-05 2024-08-17T18:15:36.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:36.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:18.735770+0000", 2024-08-17T18:15:36.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4359999999999996e-06 2024-08-17T18:15:36.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:18.735808+0000", 2024-08-17T18:15:36.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7669999999999997e-05 2024-08-17T18:15:36.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:18.735815+0000", 2024-08-17T18:15:36.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2200000000000003e-06 2024-08-17T18:15:36.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:19.649310+0000", 2024-08-17T18:15:36.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91349502900000001 2024-08-17T18:15:36.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:15:36.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:19.649319+0000", 2024-08-17T18:15:36.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6219999999999998e-06 2024-08-17T18:15:36.109 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.109 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:36.109 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.109 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:239356 56.19 56.34064559 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1596)", 2024-08-17T18:15:36.109 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:13:18.735790+0000", 2024-08-17T18:15:36.109 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 137.25864450099999, 2024-08-17T18:15:36.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91356045299999999, 2024-08-17T18:15:36.109 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:36.110 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:15:36.110 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:36.110 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:15:36.110 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:36.110 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 239356 2024-08-17T18:15:36.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.110 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:36.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:36.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:18.735790+0000", 2024-08-17T18:15:36.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:36.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:18.735790+0000", 2024-08-17T18:15:36.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:36.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:18.735790+0000", 2024-08-17T18:15:36.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:15:36.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:36.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:18.735793+0000", 2024-08-17T18:15:36.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.631e-06 2024-08-17T18:15:36.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:36.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:18.735804+0000", 2024-08-17T18:15:36.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1127999999999999e-05 2024-08-17T18:15:36.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:36.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:18.735810+0000", 2024-08-17T18:15:36.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8109999999999999e-06 2024-08-17T18:15:36.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:18.735837+0000", 2024-08-17T18:15:36.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6531000000000001e-05 2024-08-17T18:15:36.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:18.735841+0000", 2024-08-17T18:15:36.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7230000000000004e-06 2024-08-17T18:15:36.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:19.649342+0000", 2024-08-17T18:15:36.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91350098899999999 2024-08-17T18:15:36.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:15:36.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:19.649350+0000", 2024-08-17T18:15:36.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1680000000000004e-06 2024-08-17T18:15:36.114 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.114 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:36.114 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.114 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:244136 56.19 56:9db8a74a:::benchmark_data_smithi136_356456_object117432:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1599)", 2024-08-17T18:15:36.114 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:13:24.660689+0000", 2024-08-17T18:15:36.114 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 131.33374510300001, 2024-08-17T18:15:36.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.026706788, 2024-08-17T18:15:36.114 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:36.114 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:36.114 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:36.115 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:15:36.115 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:36.115 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 244136 2024-08-17T18:15:36.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.115 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:36.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:36.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:24.660689+0000", 2024-08-17T18:15:36.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:36.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:24.660689+0000", 2024-08-17T18:15:36.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:36.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:24.660688+0000", 2024-08-17T18:15:36.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:15:36.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:36.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:24.660692+0000", 2024-08-17T18:15:36.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9779999999999999e-06 2024-08-17T18:15:36.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:36.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:24.660701+0000", 2024-08-17T18:15:36.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6090000000000001e-06 2024-08-17T18:15:36.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:36.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:24.660708+0000", 2024-08-17T18:15:36.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.596e-06 2024-08-17T18:15:36.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:24.660813+0000", 2024-08-17T18:15:36.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104889 2024-08-17T18:15:36.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:24.660821+0000", 2024-08-17T18:15:36.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6209999999999993e-06 2024-08-17T18:15:36.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:25.682712+0000", 2024-08-17T18:15:36.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.021890475 2024-08-17T18:15:36.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:15:36.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:25.682855+0000", 2024-08-17T18:15:36.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014324699999999999 2024-08-17T18:15:36.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,6", 2024-08-17T18:15:36.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:25.683153+0000", 2024-08-17T18:15:36.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029861999999999998 2024-08-17T18:15:36.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:15:36.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:25.687244+0000", 2024-08-17T18:15:36.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040904369999999997 2024-08-17T18:15:36.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:25.687299+0000", 2024-08-17T18:15:36.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5130000000000002e-05 2024-08-17T18:15:36.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:25.687339+0000", 2024-08-17T18:15:36.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0064000000000001e-05 2024-08-17T18:15:36.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:15:36.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:25.687372+0000", 2024-08-17T18:15:36.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3154000000000001e-05 2024-08-17T18:15:36.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:15:36.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:25.687396+0000", 2024-08-17T18:15:36.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3941999999999999e-05 2024-08-17T18:15:36.121 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.122 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:36.122 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.122 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:244171 56.19 56:9bd8cb48:::benchmark_data_smithi136_356456_object117467:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1599)", 2024-08-17T18:15:36.122 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:13:24.684867+0000", 2024-08-17T18:15:36.122 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 131.30956762700001, 2024-08-17T18:15:36.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.002801592, 2024-08-17T18:15:36.122 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:36.122 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:36.122 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:36.122 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:15:36.122 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:36.123 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 244171 2024-08-17T18:15:36.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.123 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:36.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:36.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:24.684867+0000", 2024-08-17T18:15:36.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:36.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:24.684867+0000", 2024-08-17T18:15:36.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:36.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:24.684866+0000", 2024-08-17T18:15:36.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:15:36.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:36.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:24.684872+0000", 2024-08-17T18:15:36.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6609999999999997e-06 2024-08-17T18:15:36.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:36.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:24.684883+0000", 2024-08-17T18:15:36.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1279e-05 2024-08-17T18:15:36.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:36.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:24.684890+0000", 2024-08-17T18:15:36.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.956e-06 2024-08-17T18:15:36.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:24.685092+0000", 2024-08-17T18:15:36.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000202377 2024-08-17T18:15:36.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:24.685100+0000", 2024-08-17T18:15:36.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0839999999999992e-06 2024-08-17T18:15:36.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:25.684973+0000", 2024-08-17T18:15:36.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99987230900000001 2024-08-17T18:15:36.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:15:36.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:25.685096+0000", 2024-08-17T18:15:36.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000123397 2024-08-17T18:15:36.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,6", 2024-08-17T18:15:36.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:25.685320+0000", 2024-08-17T18:15:36.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022425 2024-08-17T18:15:36.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:15:36.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:25.687478+0000", 2024-08-17T18:15:36.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002157778 2024-08-17T18:15:36.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:25.687530+0000", 2024-08-17T18:15:36.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1609e-05 2024-08-17T18:15:36.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:25.687588+0000", 2024-08-17T18:15:36.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8118000000000002e-05 2024-08-17T18:15:36.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:15:36.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:25.687631+0000", 2024-08-17T18:15:36.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3272000000000002e-05 2024-08-17T18:15:36.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:15:36.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:25.687669+0000", 2024-08-17T18:15:36.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7517e-05 2024-08-17T18:15:36.129 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.129 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:36.129 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.129 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:244185 56.19 56:99e4c72a:::benchmark_data_smithi136_356456_object117481:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1599)", 2024-08-17T18:15:36.129 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:13:24.693138+0000", 2024-08-17T18:15:36.129 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 131.30129685899999, 2024-08-17T18:15:36.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99732947299999997, 2024-08-17T18:15:36.129 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:36.129 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:15:36.129 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:36.129 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:15:36.129 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:15:36.130 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 244185 2024-08-17T18:15:36.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.130 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:36.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:36.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:24.693138+0000", 2024-08-17T18:15:36.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:36.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:24.693138+0000", 2024-08-17T18:15:36.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:36.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:24.693137+0000", 2024-08-17T18:15:36.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:15:36.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:36.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:24.693143+0000", 2024-08-17T18:15:36.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.28e-06 2024-08-17T18:15:36.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:36.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:24.693153+0000", 2024-08-17T18:15:36.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8009999999999998e-06 2024-08-17T18:15:36.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:36.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:24.693159+0000", 2024-08-17T18:15:36.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0249999999999999e-06 2024-08-17T18:15:36.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:24.693244+0000", 2024-08-17T18:15:36.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4573000000000005e-05 2024-08-17T18:15:36.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:24.693252+0000", 2024-08-17T18:15:36.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5250000000000005e-06 2024-08-17T18:15:36.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:25.686211+0000", 2024-08-17T18:15:36.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99295906099999998 2024-08-17T18:15:36.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:15:36.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:25.686299+0000", 2024-08-17T18:15:36.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7732000000000006e-05 2024-08-17T18:15:36.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,6", 2024-08-17T18:15:36.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:25.686447+0000", 2024-08-17T18:15:36.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014777099999999999 2024-08-17T18:15:36.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:15:36.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:25.687681+0000", 2024-08-17T18:15:36.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012343510000000001 2024-08-17T18:15:36.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:25.687714+0000", 2024-08-17T18:15:36.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2468999999999997e-05 2024-08-17T18:15:36.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:15:36.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:25.690383+0000", 2024-08-17T18:15:36.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026696689999999999 2024-08-17T18:15:36.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:15:36.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:25.690430+0000", 2024-08-17T18:15:36.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6981999999999998e-05 2024-08-17T18:15:36.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:15:36.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:13:25.690467+0000", 2024-08-17T18:15:36.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6956000000000003e-05 2024-08-17T18:15:36.136 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.136 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:36.136 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.136 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:79442 57.9 57:96fc1529:::benchmark_data_smithi136_389235_object79441:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1616)", 2024-08-17T18:15:36.136 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:14:48.570046+0000", 2024-08-17T18:15:36.136 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 47.424388643999997, 2024-08-17T18:15:36.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.036364769, 2024-08-17T18:15:36.137 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:15:36.137 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:15:36.137 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:15:36.137 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:15:36.137 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:15:36.137 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 79442 2024-08-17T18:15:36.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.137 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:15:36.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:15:36.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:14:48.570046+0000", 2024-08-17T18:15:36.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:15:36.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:14:48.570046+0000", 2024-08-17T18:15:36.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:15:36.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:15:36.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:14:48.570045+0000", 2024-08-17T18:15:36.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:15:36.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:15:36.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:14:48.570167+0000", 2024-08-17T18:15:36.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121876 2024-08-17T18:15:36.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:15:36.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:14:48.570178+0000", 2024-08-17T18:15:36.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0861e-05 2024-08-17T18:15:36.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:15:36.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:14:48.570185+0000", 2024-08-17T18:15:36.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7910000000000005e-06 2024-08-17T18:15:36.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:14:48.678715+0000", 2024-08-17T18:15:36.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10852977799999999 2024-08-17T18:15:36.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:15:36.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:14:48.678740+0000", 2024-08-17T18:15:36.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5150000000000001e-05 2024-08-17T18:15:36.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:14:49.603336+0000", 2024-08-17T18:15:36.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.924595206 2024-08-17T18:15:36.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:15:36.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:14:49.604003+0000", 2024-08-17T18:15:36.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00066727099999999999 2024-08-17T18:15:36.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:15:36.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:14:49.606328+0000", 2024-08-17T18:15:36.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023247490000000001 2024-08-17T18:15:36.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:15:36.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:15:36.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:15:36.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:14:49.606411+0000", 2024-08-17T18:15:36.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3094000000000001e-05 2024-08-17T18:15:36.141 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.142 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:36.142 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.142 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:15:36.142 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:15:36.142 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:15:36.142 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:35 smithi165 ceph-mon[105517]: pgmap v3076: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.8 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 38 MiB/s wr, 601 op/s; 31594/335325 objects degraded (9.422%); 20500/335325 objects misplaced (6.113%); 4.4 MiB/s, 70 objects/s recovering 2024-08-17T18:15:36.158 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:36.185 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:36.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:36.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:36.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:35 smithi136 ceph-mon[109861]: pgmap v3076: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.8 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 38 MiB/s wr, 601 op/s; 31594/335325 objects degraded (9.422%); 20500/335325 objects misplaced (6.113%); 4.4 MiB/s, 70 objects/s recovering 2024-08-17T18:15:36.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:35 smithi136 ceph-mon[115078]: pgmap v3076: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.8 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 38 MiB/s wr, 601 op/s; 31594/335325 objects degraded (9.422%); 20500/335325 objects misplaced (6.113%); 4.4 MiB/s, 70 objects/s recovering 2024-08-17T18:15:36.288 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:36.326 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:36.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:36.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:36.428 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:36.462 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:36.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:36.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:36.567 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:36.592 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:36.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:36.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:36.694 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:36.721 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:36.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:36.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:36.785 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 89 16 115438 115422 81.0449 37.6875 0.00486833 0.0123124 2024-08-17T18:15:36.823 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:36.854 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:36.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:36.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:36.956 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:36.978 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:36.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:36.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:36.994 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:15:36.994 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:15:36.995 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:15:37.038 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-17T18:15:37.080 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:37.103 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:37.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:37.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:37.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:15:37.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:15:37.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:15:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:15:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:15:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:15:37.204 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:37.247 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:37.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:37.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:37.349 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:37.371 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:37.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:37.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:37.473 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:37.498 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:37.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:37.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:37.599 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:37.628 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:37.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:37.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:37.730 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:37.753 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:37.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:37.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:37.785 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 90 15 115919 115904 80.4791 30.125 0.00329717 0.012393 2024-08-17T18:15:37.831 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:15:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:15:37] "GET /metrics HTTP/1.1" 200 36729 "" "Prometheus/2.51.0" 2024-08-17T18:15:37.856 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:37.878 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:37.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:37.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:37.980 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:38.001 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:38.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:38.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:38.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:37 smithi165 ceph-mon[105517]: pgmap v3077: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.9 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 33 MiB/s wr, 524 op/s; 31594/339801 objects degraded (9.298%); 20708/339801 objects misplaced (6.094%); 3.3 MiB/s, 52 objects/s recovering 2024-08-17T18:15:38.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:38.125 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:38.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:38.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:38.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:37 smithi136 ceph-mon[109861]: pgmap v3077: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.9 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 33 MiB/s wr, 524 op/s; 31594/339801 objects degraded (9.298%); 20708/339801 objects misplaced (6.094%); 3.3 MiB/s, 52 objects/s recovering 2024-08-17T18:15:38.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:37 smithi136 ceph-mon[115078]: pgmap v3077: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.9 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 33 MiB/s wr, 524 op/s; 31594/339801 objects degraded (9.298%); 20708/339801 objects misplaced (6.094%); 3.3 MiB/s, 52 objects/s recovering 2024-08-17T18:15:38.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:38.248 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:38.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:38.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:38.350 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:38.372 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:38.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:38.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:38.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:38.496 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:38.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:38.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:38.598 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:38.619 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:38.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:38.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:38.721 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:38.742 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:38.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:38.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:38.785 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Total time run: 90.2143 2024-08-17T18:15:38.785 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Total writes made: 115919 2024-08-17T18:15:38.785 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Write size: 65536 2024-08-17T18:15:38.785 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Object size: 65536 2024-08-17T18:15:38.785 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Bandwidth (MB/sec): 80.3081 2024-08-17T18:15:38.785 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Stddev Bandwidth: 53.1033 2024-08-17T18:15:38.785 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Max bandwidth (MB/sec): 239.25 2024-08-17T18:15:38.786 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Min bandwidth (MB/sec): 0 2024-08-17T18:15:38.786 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Average IOPS: 1284 2024-08-17T18:15:38.786 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Stddev IOPS: 849.653 2024-08-17T18:15:38.786 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Max IOPS: 3828 2024-08-17T18:15:38.786 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Min IOPS: 0 2024-08-17T18:15:38.786 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Average Latency(s): 0.0124288 2024-08-17T18:15:38.786 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Stddev Latency(s): 0.0709408 2024-08-17T18:15:38.786 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Max latency(s): 4.02884 2024-08-17T18:15:38.786 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Min latency(s): 0.00125927 2024-08-17T18:15:38.791 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Cleaning up (deleting benchmark objects) 2024-08-17T18:15:38.808 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:15:38.845 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:38.893 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:38.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:38.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:38.995 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:39.382 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:39.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:39.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:39.484 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:39.512 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:39.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:39.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:39.613 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:39.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:39.637 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:39.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:39.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:39.767 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:39.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:39.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:39.869 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:39.892 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:39.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:39.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:39.994 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:40.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:40.028 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:40.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:40.129 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:40.155 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:40.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:40.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:40.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:39 smithi136 ceph-mon[109861]: pgmap v3078: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.0 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 39 MiB/s wr, 625 op/s; 31594/344991 objects degraded (9.158%); 20521/344991 objects misplaced (5.948%); 5.8 MiB/s, 93 objects/s recovering 2024-08-17T18:15:40.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:39 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 31594/344991 objects degraded (9.158%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:40.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:39 smithi136 ceph-mon[115078]: pgmap v3078: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.0 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 39 MiB/s wr, 625 op/s; 31594/344991 objects degraded (9.158%); 20521/344991 objects misplaced (5.948%); 5.8 MiB/s, 93 objects/s recovering 2024-08-17T18:15:40.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:39 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 31594/344991 objects degraded (9.158%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:40.230 INFO:tasks.thrashosds.thrasher:in_osds: [2, 4, 6, 0, 5] out_osds: [7, 1, 3] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:15:40.230 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:15:40.230 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-08-17T18:15:40.231 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd in 7 2024-08-17T18:15:40.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:40.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:39 smithi165 ceph-mon[105517]: pgmap v3078: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.0 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 39 MiB/s wr, 625 op/s; 31594/344991 objects degraded (9.158%); 20521/344991 objects misplaced (5.948%); 5.8 MiB/s, 93 objects/s recovering 2024-08-17T18:15:40.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:39 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 31594/344991 objects degraded (9.158%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:40.296 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:40.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:40.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:40.398 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:40.423 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:40.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:40.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:40.525 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:40.548 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:40.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:40.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:40.650 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:40.673 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:40.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:40.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:40.775 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:40.799 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:40.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:40.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:40.901 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:40.922 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:40.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:40.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:41.024 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:41.047 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:41.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:41.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:41.149 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:41.172 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:41.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:41.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:41.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:41.297 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:41.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:41.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:41.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:41.435 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:41.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:41.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:41.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:41.568 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:41.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:41.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:41.670 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:41.694 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:41.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:41.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:41.796 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:41.821 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:41.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:41.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:41.923 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:41.945 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:41.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:41.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:42.037 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:15:42.046 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:42.057 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:41 smithi165 ceph-mon[105517]: pgmap v3079: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.1 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 37 MiB/s wr, 611 op/s; 31594/346701 objects degraded (9.113%); 20521/346701 objects misplaced (5.919%); 5.7 MiB/s, 91 objects/s recovering 2024-08-17T18:15:42.075 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:42.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:42.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:42.177 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:42.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:41 smithi136 ceph-mon[109861]: pgmap v3079: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.1 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 37 MiB/s wr, 611 op/s; 31594/346701 objects degraded (9.113%); 20521/346701 objects misplaced (5.919%); 5.7 MiB/s, 91 objects/s recovering 2024-08-17T18:15:42.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:41 smithi136 ceph-mon[115078]: pgmap v3079: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.1 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 37 MiB/s wr, 611 op/s; 31594/346701 objects degraded (9.113%); 20521/346701 objects misplaced (5.919%); 5.7 MiB/s, 91 objects/s recovering 2024-08-17T18:15:42.213 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:42.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:42.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:42.315 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:42.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:42.340 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:42.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:42.346 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:15:42.347 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:15:42.364 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:15:42.365 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:15:42.383 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:15:42.383 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:15:42.407 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:15:42.407 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:15:42.427 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:15:42.427 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:15:42.441 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:42.444 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:15:42.445 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:15:42.464 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:42.465 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:15:42.465 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:15:42.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:42.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:42.475 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:15:42.475 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:15:42.487 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:15:42.487 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:15:42.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:42.592 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:42.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:42.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:42.694 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:42.721 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:42.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:42.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:42.823 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:42.848 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:42.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:42.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:42.950 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:42.973 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:42.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:42.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:43.075 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:43.105 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:42 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/78031888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:15:43.112 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:42 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/78031888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:15:43.112 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:42 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/78031888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:15:43.114 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:43.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:43.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:43.215 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:43.240 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:43.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:43.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:43.342 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:43.366 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:43.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:43.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:43.459 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-17T18:15:43.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:43.492 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:43.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:43.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:43.594 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:43.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:43.632 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:43.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:43.733 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:43.754 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:43.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:43.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:43.856 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:43.857 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:15:43.887 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:43.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:43.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:43.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:44.028 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:44.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:44.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:44.096 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:15:44.131 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:44.154 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:44.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:44.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:44.158 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:43 smithi136 ceph-mon[109861]: pgmap v3080: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.0 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 34 MiB/s wr, 618 op/s; 31561/345453 objects degraded (9.136%); 19659/345453 objects misplaced (5.691%); 10 MiB/s, 159 objects/s recovering 2024-08-17T18:15:44.159 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:43 smithi136 ceph-mon[115078]: pgmap v3080: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.0 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 34 MiB/s wr, 618 op/s; 31561/345453 objects degraded (9.136%); 19659/345453 objects misplaced (5.691%); 10 MiB/s, 159 objects/s recovering 2024-08-17T18:15:44.256 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:44.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:43 smithi165 ceph-mon[105517]: pgmap v3080: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.0 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 34 MiB/s wr, 618 op/s; 31561/345453 objects degraded (9.136%); 19659/345453 objects misplaced (5.691%); 10 MiB/s, 159 objects/s recovering 2024-08-17T18:15:44.280 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:44.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:44.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:44.383 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:44.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:44.413 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:44.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:44.514 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:44.546 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:44.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:44.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:44.650 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:44.672 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:44.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:44.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:44.774 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:44.803 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:44.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:44.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:44.905 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:44.926 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:44 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 31561/345453 objects degraded (9.136%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:44.927 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:44.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:44.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:45.029 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:45.055 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:45.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:45.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:45.094 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:44 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 31561/345453 objects degraded (9.136%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:45.094 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:44 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 31561/345453 objects degraded (9.136%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:45.160 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:45.194 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:45.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:45.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:45.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:45.323 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:45.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:45.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:45.425 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:45.451 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:45.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:45.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:45.553 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:45.580 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:45.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:45.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:45.682 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:45.708 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:45.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:45.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:45.810 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:45.834 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:45.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:45.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:45.886 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:15:45.936 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:45 smithi136 ceph-mon[109861]: pgmap v3081: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.0 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 85 B/s rd, 25 MiB/s wr, 492 op/s; 31561/344625 objects degraded (9.158%); 19615/344625 objects misplaced (5.692%); 6.5 MiB/s, 104 objects/s recovering 2024-08-17T18:15:45.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:45 smithi136 ceph-mon[115078]: pgmap v3081: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.0 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 85 B/s rd, 25 MiB/s wr, 492 op/s; 31561/344625 objects degraded (9.158%); 19615/344625 objects misplaced (5.692%); 6.5 MiB/s, 104 objects/s recovering 2024-08-17T18:15:45.966 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:45.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:45.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:46.068 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:46.092 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:46.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:46.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:46.194 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:46.217 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:46.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:46.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:46.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:45 smithi165 ceph-mon[105517]: pgmap v3081: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.0 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 85 B/s rd, 25 MiB/s wr, 492 op/s; 31561/344625 objects degraded (9.158%); 19615/344625 objects misplaced (5.692%); 6.5 MiB/s, 104 objects/s recovering 2024-08-17T18:15:46.320 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:46.344 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:46.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:46.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:46.446 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:46.468 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:46.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:46.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:46.570 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:46.601 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:46.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:46.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:46.703 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:46.727 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:46.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:46.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:46.828 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:46.856 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:46.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:46.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:46.863 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:15:46.959 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:46.966 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:46 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/576464637' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:15:46.999 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:46.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:47.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:47.100 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:47.107 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:46 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/576464637' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:15:47.107 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:46 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/576464637' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:15:47.125 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:47.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:47.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:47.227 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:47.251 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:47.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:47.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:47.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:47.385 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:47.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:47.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:47.487 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:47.510 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:47.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:47.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:47.568 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:15:47.568 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:15:47.613 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:47.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:47.637 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:47.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:47.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:47.767 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:47.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:47.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:47.857 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:15:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:15:47] "GET /metrics HTTP/1.1" 200 36729 "" "Prometheus/2.51.0" 2024-08-17T18:15:47.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:47.893 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:47.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:47.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:47.995 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:48.016 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:48.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:48.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:48.118 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:48.158 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:48.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:48.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:48.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:47 smithi136 ceph-mon[109861]: pgmap v3082: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.0 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 85 B/s rd, 22 MiB/s wr, 515 op/s; 31497/341793 objects degraded (9.215%); 19501/341793 objects misplaced (5.706%); 6.9 MiB/s, 110 objects/s recovering 2024-08-17T18:15:48.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:47 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:15:48.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:47 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/576464637' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:15:48.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:47 smithi136 ceph-mon[109861]: osdmap e1634: 8 total, 8 up, 5 in 2024-08-17T18:15:48.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:47 smithi136 ceph-mon[115078]: pgmap v3082: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.0 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 85 B/s rd, 22 MiB/s wr, 515 op/s; 31497/341793 objects degraded (9.215%); 19501/341793 objects misplaced (5.706%); 6.9 MiB/s, 110 objects/s recovering 2024-08-17T18:15:48.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:47 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:15:48.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:47 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/576464637' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:15:48.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:47 smithi136 ceph-mon[115078]: osdmap e1634: 8 total, 8 up, 5 in 2024-08-17T18:15:48.216 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:47 smithi165 ceph-mon[105517]: pgmap v3082: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 7.0 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 85 B/s rd, 22 MiB/s wr, 515 op/s; 31497/341793 objects degraded (9.215%); 19501/341793 objects misplaced (5.706%); 6.9 MiB/s, 110 objects/s recovering 2024-08-17T18:15:48.216 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:47 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:15:48.216 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:47 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/576464637' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:15:48.216 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:47 smithi165 ceph-mon[105517]: osdmap e1634: 8 total, 8 up, 5 in 2024-08-17T18:15:48.260 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:48.288 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:48.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:48.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:48.390 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:48.413 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:48.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:48.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:48.515 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:48.540 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:48.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:48.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:48.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-17T18:15:48.642 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:48.664 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:48.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:48.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:48.766 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:48.789 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:48.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:48.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:48.891 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:48.898 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:15:48.914 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:48.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:48.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:49.016 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:49.041 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:49.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:49.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:49.143 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:49.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:49.178 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:49.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:49.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:49 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:15:49.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:49 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:15:49.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:49 smithi136 ceph-mon[109861]: pgmap v3084: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.9 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 102 B/s rd, 5.9 MiB/s wr, 414 op/s; 31497/338235 objects degraded (9.312%); 18286/338235 objects misplaced (5.406%); 12 MiB/s, 184 objects/s recovering 2024-08-17T18:15:49.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:49 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:15:49.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:49 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:15:49.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:49 smithi136 ceph-mon[115078]: pgmap v3084: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.9 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 102 B/s rd, 5.9 MiB/s wr, 414 op/s; 31497/338235 objects degraded (9.312%); 18286/338235 objects misplaced (5.406%); 12 MiB/s, 184 objects/s recovering 2024-08-17T18:15:49.279 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:49.301 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:49.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:49.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:49.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:49 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:15:49.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:49 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:15:49.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:49 smithi165 ceph-mon[105517]: pgmap v3084: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.9 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 102 B/s rd, 5.9 MiB/s wr, 414 op/s; 31497/338235 objects degraded (9.312%); 18286/338235 objects misplaced (5.406%); 12 MiB/s, 184 objects/s recovering 2024-08-17T18:15:49.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:49.426 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:49.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:49.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:49.528 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:49.552 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:49.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:49.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:49.654 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:49.678 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:49.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:49.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:49.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:49.804 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:49.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:49.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:49.906 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:49.927 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:49.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:49.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:50.029 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:50.051 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:50.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:50.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:50.152 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:50.174 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:50.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:50.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:50.204 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:15:50.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:50.300 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:50.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:50.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:50.402 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:50.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:50.429 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:50.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:50.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:50 smithi136 ceph-mon[109861]: osdmap e1635: 8 total, 8 up, 5 in 2024-08-17T18:15:50.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:50 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 31497/338235 objects degraded (9.312%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:50.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:50 smithi136 ceph-mon[115078]: osdmap e1635: 8 total, 8 up, 5 in 2024-08-17T18:15:50.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:50 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 31497/338235 objects degraded (9.312%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:50.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:50 smithi165 ceph-mon[105517]: osdmap e1635: 8 total, 8 up, 5 in 2024-08-17T18:15:50.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:50 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 31497/338235 objects degraded (9.312%), 10 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:50.530 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:50.558 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:50.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:50.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:50.660 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:50.695 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:50.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:50.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:50.797 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:50.807 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:15:50.807 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:15:50.819 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:50.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:50.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:50.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:50.945 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:50.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:50.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:51.047 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:51.069 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:51.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:51.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:51.171 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:51.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:51.196 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:51.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:51.260 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:51 smithi136 ceph-mon[115078]: osdmap e1636: 8 total, 8 up, 5 in 2024-08-17T18:15:51.261 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:51 smithi136 ceph-mon[115078]: pgmap v3087: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.8 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 0 B/s wr, 496 op/s; 31497/335694 objects degraded (9.383%); 18286/335694 objects misplaced (5.447%); 11 MiB/s, 177 objects/s recovering 2024-08-17T18:15:51.261 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:51 smithi136 ceph-mon[109861]: osdmap e1636: 8 total, 8 up, 5 in 2024-08-17T18:15:51.261 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:51 smithi136 ceph-mon[109861]: pgmap v3087: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.8 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 0 B/s wr, 496 op/s; 31497/335694 objects degraded (9.383%); 18286/335694 objects misplaced (5.447%); 11 MiB/s, 177 objects/s recovering 2024-08-17T18:15:51.261 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:51 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a[109838]: 2024-08-17T18:15:51.203+0000 7f5b8e77d640 -1 mon.a@0(leader).osd e1636 definitely_dead 0 2024-08-17T18:15:51.297 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:51.319 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:51.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:51.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:51.421 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:51.442 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:51.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:51.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:51.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:51 smithi165 ceph-mon[105517]: osdmap e1636: 8 total, 8 up, 5 in 2024-08-17T18:15:51.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:51 smithi165 ceph-mon[105517]: pgmap v3087: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 10 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.8 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 0 B/s wr, 496 op/s; 31497/335694 objects degraded (9.383%); 18286/335694 objects misplaced (5.447%); 11 MiB/s, 177 objects/s recovering 2024-08-17T18:15:51.544 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:51.569 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:51.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:51.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:51.671 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:51.693 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:51.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:51.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:51.795 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:51.819 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:51.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:51.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:51.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:51.950 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:51.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:51.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:52.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:52.080 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:52.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:52.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:52.123 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-17T18:15:52.149 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:15:52.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:52.208 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:52.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:52.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:52.211 INFO:teuthology.orchestra.run.smithi136.stderr:marked in osd.7. 2024-08-17T18:15:52.310 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:52.335 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:52.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:52.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:52.343 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:52 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3209698083' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-08-17T18:15:52.343 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:15:52.343 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:52 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:15:52.343 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:52 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3209698083' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-08-17T18:15:52.344 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:15:52.344 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:52 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:15:52.372 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:15:52.437 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:52.460 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:52.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:52.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:52.494 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:52 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3209698083' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-08-17T18:15:52.495 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:15:52.495 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:52 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:15:52.562 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:52.584 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:52.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:52.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:52.685 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:52.708 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:52.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:52.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:52.782 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:15:52.783 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:15:52.785 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:15:52.786 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:15:52.803 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:15:52.803 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:15:52.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:52.820 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:15:52.820 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:15:52.832 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:52.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:52.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:52.839 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:15:52.839 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:15:52.852 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:15:52.853 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:15:52.869 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:15:52.869 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:15:52.885 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:15:52.886 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:15:52.934 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:52.955 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:52.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:52.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:53.057 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:53.079 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:53.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:53.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:53.181 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:53.203 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:53.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:53.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:53.305 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:53.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:53 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3209698083' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-08-17T18:15:53.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:53 smithi136 ceph-mon[109861]: osdmap e1637: 8 total, 8 up, 6 in 2024-08-17T18:15:53.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:53 smithi136 ceph-mon[109861]: pgmap v3089: 109 pgs: 1 remapped+peering, 12 peering, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.8 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 0 B/s wr, 550 op/s; 25478/332610 objects degraded (7.660%); 16538/332610 objects misplaced (4.972%); 17 MiB/s, 266 objects/s recovering 2024-08-17T18:15:53.330 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:53 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1173005604' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:15:53.331 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:53 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3209698083' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-08-17T18:15:53.331 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:53 smithi136 ceph-mon[115078]: osdmap e1637: 8 total, 8 up, 6 in 2024-08-17T18:15:53.331 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:53 smithi136 ceph-mon[115078]: pgmap v3089: 109 pgs: 1 remapped+peering, 12 peering, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.8 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 0 B/s wr, 550 op/s; 25478/332610 objects degraded (7.660%); 16538/332610 objects misplaced (4.972%); 17 MiB/s, 266 objects/s recovering 2024-08-17T18:15:53.331 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:53 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1173005604' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:15:53.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:53.351 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:53.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:53.452 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:53.474 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:53.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:53.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:53.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:53 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3209698083' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-08-17T18:15:53.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:53 smithi165 ceph-mon[105517]: osdmap e1637: 8 total, 8 up, 6 in 2024-08-17T18:15:53.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:53 smithi165 ceph-mon[105517]: pgmap v3089: 109 pgs: 1 remapped+peering, 12 peering, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.8 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 0 B/s wr, 550 op/s; 25478/332610 objects degraded (7.660%); 16538/332610 objects misplaced (4.972%); 17 MiB/s, 266 objects/s recovering 2024-08-17T18:15:53.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:53 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1173005604' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:15:53.576 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:53.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:53.620 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:53.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:53.722 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:53.743 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:53.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:53.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:53.845 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:53.868 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:53.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:53.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:53.970 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:53.991 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:53.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:53.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:54.093 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:54.115 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:54.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:54.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:54.216 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:54.248 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:54.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:54.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:54.350 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:54.382 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:54.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:54.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:54.484 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:54.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:54 smithi165 ceph-mon[105517]: 57.1 starting backfill to osd.7 from (0'0,0'0] MAX to 1636'6390 2024-08-17T18:15:54.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:54 smithi165 ceph-mon[105517]: osdmap e1638: 8 total, 8 up, 6 in 2024-08-17T18:15:54.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:54 smithi165 ceph-mon[105517]: 57.f starting backfill to osd.7 from (0'0,0'0] MAX to 1636'7740 2024-08-17T18:15:54.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:54.535 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:54.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:54.546 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:54 smithi136 ceph-mon[109861]: 57.1 starting backfill to osd.7 from (0'0,0'0] MAX to 1636'6390 2024-08-17T18:15:54.546 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:54 smithi136 ceph-mon[109861]: osdmap e1638: 8 total, 8 up, 6 in 2024-08-17T18:15:54.546 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:54 smithi136 ceph-mon[109861]: 57.f starting backfill to osd.7 from (0'0,0'0] MAX to 1636'7740 2024-08-17T18:15:54.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:54 smithi136 ceph-mon[115078]: 57.1 starting backfill to osd.7 from (0'0,0'0] MAX to 1636'6390 2024-08-17T18:15:54.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:54 smithi136 ceph-mon[115078]: osdmap e1638: 8 total, 8 up, 6 in 2024-08-17T18:15:54.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:54 smithi136 ceph-mon[115078]: 57.f starting backfill to osd.7 from (0'0,0'0] MAX to 1636'7740 2024-08-17T18:15:54.637 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:54.658 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:54.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:54.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:54.760 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:54.781 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:54.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:54.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:54.886 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:54.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:54.908 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:54.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:54.944 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-08-17T18:15:55.009 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:55.034 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:55.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:55.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:55.059 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:15:55.060 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:15:55.060 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-08-17T18:15:55.060 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:15:55.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:55.162 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:55.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:55.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:55.264 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:55.285 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:55.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:55.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:55.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:55.409 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:55.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:55.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:55 smithi165 ceph-mon[105517]: 57.18 starting backfill to osd.7 from (0'0,0'0] MAX to 1636'6374 2024-08-17T18:15:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:55 smithi165 ceph-mon[105517]: osdmap e1639: 8 total, 8 up, 6 in 2024-08-17T18:15:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:55 smithi165 ceph-mon[105517]: 57.e starting backfill to osd.7 from (0'0,0'0] MAX to 1636'7635 2024-08-17T18:15:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:55 smithi165 ceph-mon[105517]: 57.5 starting backfill to osd.7 from (0'0,0'0] MAX to 1636'6346 2024-08-17T18:15:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:55 smithi165 ceph-mon[105517]: pgmap v3092: 109 pgs: 2 remapped+peering, 21 peering, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 6.7 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 0 B/s wr, 322 op/s; 25478/331590 objects degraded (7.684%); 12966/331590 objects misplaced (3.910%); 7.1 MiB/s, 113 objects/s recovering 2024-08-17T18:15:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:55 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 25478/332610 objects degraded (7.660%), 9 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:55 smithi165 ceph-mon[105517]: osdmap e1640: 8 total, 8 up, 6 in 2024-08-17T18:15:55.512 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:55.536 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:55.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:55.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:55.637 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:55.661 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:55.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:55.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:55 smithi136 ceph-mon[109861]: 57.18 starting backfill to osd.7 from (0'0,0'0] MAX to 1636'6374 2024-08-17T18:15:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:55 smithi136 ceph-mon[109861]: osdmap e1639: 8 total, 8 up, 6 in 2024-08-17T18:15:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:55 smithi136 ceph-mon[109861]: 57.e starting backfill to osd.7 from (0'0,0'0] MAX to 1636'7635 2024-08-17T18:15:55.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:55 smithi136 ceph-mon[109861]: 57.5 starting backfill to osd.7 from (0'0,0'0] MAX to 1636'6346 2024-08-17T18:15:55.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:55 smithi136 ceph-mon[109861]: pgmap v3092: 109 pgs: 2 remapped+peering, 21 peering, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 6.7 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 0 B/s wr, 322 op/s; 25478/331590 objects degraded (7.684%); 12966/331590 objects misplaced (3.910%); 7.1 MiB/s, 113 objects/s recovering 2024-08-17T18:15:55.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:55 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 25478/332610 objects degraded (7.660%), 9 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:55.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:55 smithi136 ceph-mon[109861]: osdmap e1640: 8 total, 8 up, 6 in 2024-08-17T18:15:55.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:55 smithi136 ceph-mon[115078]: 57.18 starting backfill to osd.7 from (0'0,0'0] MAX to 1636'6374 2024-08-17T18:15:55.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:55 smithi136 ceph-mon[115078]: osdmap e1639: 8 total, 8 up, 6 in 2024-08-17T18:15:55.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:55 smithi136 ceph-mon[115078]: 57.e starting backfill to osd.7 from (0'0,0'0] MAX to 1636'7635 2024-08-17T18:15:55.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:55 smithi136 ceph-mon[115078]: 57.5 starting backfill to osd.7 from (0'0,0'0] MAX to 1636'6346 2024-08-17T18:15:55.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:55 smithi136 ceph-mon[115078]: pgmap v3092: 109 pgs: 2 remapped+peering, 21 peering, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 6.7 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 0 B/s wr, 322 op/s; 25478/331590 objects degraded (7.684%); 12966/331590 objects misplaced (3.910%); 7.1 MiB/s, 113 objects/s recovering 2024-08-17T18:15:55.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:55 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 25478/332610 objects degraded (7.660%), 9 pgs degraded (PG_DEGRADED) 2024-08-17T18:15:55.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:55 smithi136 ceph-mon[115078]: osdmap e1640: 8 total, 8 up, 6 in 2024-08-17T18:15:55.755 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-17T18:15:55.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:55.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:55.786 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:55.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:55.795 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:15:55.887 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:55.908 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:55.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:55.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:55.992 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:15:56.009 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:56.033 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:56.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:56.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:56.048 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:15:56.135 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:56.158 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:56.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:56.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:56.260 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:56.284 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:56.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:56.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:56.386 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:56.408 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:56.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:56.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:56.510 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:56.532 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:56.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:56.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:56.634 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:56.656 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:56.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:56.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:56.757 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:56.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:56 smithi165 ceph-mon[105517]: osdmap e1641: 8 total, 8 up, 6 in 2024-08-17T18:15:56.779 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:56.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:56.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:56.881 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:15:56.903 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:56.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:56.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:15:56.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:56 smithi136 ceph-mon[109861]: osdmap e1641: 8 total, 8 up, 6 in 2024-08-17T18:15:56.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:56 smithi136 ceph-mon[115078]: osdmap e1641: 8 total, 8 up, 6 in 2024-08-17T18:15:57.006 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:57.028 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:57.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:57.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:57.130 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:57.151 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:57.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:57.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:57.253 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:57.275 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:57.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:57.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:57.378 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:57.399 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:57.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:57.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:57.501 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:15:57.522 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:57.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:57.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:15:57.608 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:15:57.624 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:57.647 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:57.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:57.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:57.691 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:15:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:15:57] "GET /metrics HTTP/1.1" 200 36735 "" "Prometheus/2.51.0" 2024-08-17T18:15:57.749 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:57 smithi165 ceph-mon[105517]: pgmap v3095: 109 pgs: 1 activating+undersized+remapped, 1 activating+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 3 remapped+peering, 1 active+remapped, 21 peering, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 70 active+clean; 6.7 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 0 B/s wr, 216 op/s; 28135/330012 objects degraded (8.525%); 16651/330012 objects misplaced (5.046%); 1024 KiB/s, 16 objects/s recovering 2024-08-17T18:15:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:57 smithi165 ceph-mon[105517]: 57.a starting backfill to osd.7 from (0'0,0'0] MAX to 1636'7749 2024-08-17T18:15:57.771 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:57.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:57.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:57.872 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:57.896 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:57.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:57.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:57.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:57 smithi136 ceph-mon[115078]: pgmap v3095: 109 pgs: 1 activating+undersized+remapped, 1 activating+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 3 remapped+peering, 1 active+remapped, 21 peering, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 70 active+clean; 6.7 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 0 B/s wr, 216 op/s; 28135/330012 objects degraded (8.525%); 16651/330012 objects misplaced (5.046%); 1024 KiB/s, 16 objects/s recovering 2024-08-17T18:15:57.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:57 smithi136 ceph-mon[115078]: 57.a starting backfill to osd.7 from (0'0,0'0] MAX to 1636'7749 2024-08-17T18:15:57.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:57 smithi136 ceph-mon[109861]: pgmap v3095: 109 pgs: 1 activating+undersized+remapped, 1 activating+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 3 remapped+peering, 1 active+remapped, 21 peering, 1 active+recovery_wait, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 70 active+clean; 6.7 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 0 B/s wr, 216 op/s; 28135/330012 objects degraded (8.525%); 16651/330012 objects misplaced (5.046%); 1024 KiB/s, 16 objects/s recovering 2024-08-17T18:15:57.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:57 smithi136 ceph-mon[109861]: 57.a starting backfill to osd.7 from (0'0,0'0] MAX to 1636'7749 2024-08-17T18:15:57.998 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:15:58.021 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:58.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:58.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:15:58.123 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:58.145 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:58.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:58.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:58.247 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:58.268 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:58.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:58.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:58.370 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:58.393 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:58.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:58.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:58.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:15:58.518 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:58.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:58.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:15:58.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:58.645 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:58.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:58.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:58.747 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:15:58.769 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:58.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:58.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:15:58.871 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:58.893 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:58.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:58.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:58.994 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:59.016 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:59.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:59.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:59.118 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:15:59.141 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:59.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:59.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:15:59.243 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:59.264 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:59.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:59.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:59.366 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:59.389 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:59.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:59.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:59.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:15:59.513 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:15:59.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:59.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:15:59.615 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:59.636 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:59.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:59.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:59.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:59 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:15:59.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:59 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:15:59.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:59 smithi136 ceph-mon[115078]: pgmap v3096: 109 pgs: 84 active+clean, 12 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+remapped, 1 activating, 1 remapped+peering, 1 active+recovery_wait+degraded+remapped, 2 activating+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 activating+degraded+remapped, 1 activating+undersized+remapped; 6.7 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 142 op/s; 55116/329367 objects degraded (16.734%); 28076/329367 objects misplaced (8.524%); 2.6 MiB/s, 44 objects/s recovering 2024-08-17T18:15:59.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:59 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:15:59.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:59 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:15:59.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:15:59 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:15:59.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:59 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:15:59.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:59 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:15:59.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:59 smithi136 ceph-mon[109861]: pgmap v3096: 109 pgs: 84 active+clean, 12 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+remapped, 1 activating, 1 remapped+peering, 1 active+recovery_wait+degraded+remapped, 2 activating+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 activating+degraded+remapped, 1 activating+undersized+remapped; 6.7 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 142 op/s; 55116/329367 objects degraded (16.734%); 28076/329367 objects misplaced (8.524%); 2.6 MiB/s, 44 objects/s recovering 2024-08-17T18:15:59.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:59 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:15:59.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:59 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:15:59.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:15:59 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:15:59.738 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:59.759 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:59.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:59.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:59.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:59 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:15:59.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:59 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:15:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:59 smithi165 ceph-mon[105517]: pgmap v3096: 109 pgs: 84 active+clean, 12 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+remapped, 1 activating, 1 remapped+peering, 1 active+recovery_wait+degraded+remapped, 2 activating+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 activating+degraded+remapped, 1 activating+undersized+remapped; 6.7 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 142 op/s; 55116/329367 objects degraded (16.734%); 28076/329367 objects misplaced (8.524%); 2.6 MiB/s, 44 objects/s recovering 2024-08-17T18:15:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:59 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:15:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:59 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:15:59.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:15:59 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:15:59.860 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:15:59.882 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:15:59.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:15:59.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:15:59.945 INFO:tasks.thrashosds.thrasher:in_osds: [2, 4, 6, 0, 5, 7] out_osds: [1, 3] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:15:59.945 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:15:59.946 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-08-17T18:15:59.946 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-08-17T18:15:59.984 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:00.006 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:00.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:00.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:00.107 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:00.134 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:00.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:00.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:00.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:00.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:00.259 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:00.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:00.360 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:00.382 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:00.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:00.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:00.419 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:16:00.484 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:00.507 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:00.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:00.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:00.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:00 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 55116/329367 objects degraded (16.734%), 16 pgs degraded (PG_DEGRADED) 2024-08-17T18:16:00.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:00 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3615287829' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:16:00.567 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:00 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 55116/329367 objects degraded (16.734%), 16 pgs degraded (PG_DEGRADED) 2024-08-17T18:16:00.567 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:00 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3615287829' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:16:00.567 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:00 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 55116/329367 objects degraded (16.734%), 16 pgs degraded (PG_DEGRADED) 2024-08-17T18:16:00.567 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:00 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3615287829' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:16:00.610 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:00.643 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:00.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:00.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:00.745 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:00.768 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:00.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:00.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:00.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:00.906 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:00.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:00.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:01.008 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:01.011 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:16:01.011 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:16:01.012 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:16:01.012 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:16:01.012 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:16:01.012 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:16:01.012 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:16:01.012 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:16:01.012 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:16:01.012 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:16:01.013 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:16:01.013 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:16:01.013 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:16:01.013 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:16:01.013 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:16:01.013 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:16:01.031 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:01.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:01.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:01.134 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:01.164 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:01.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:01.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:01.266 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:01.288 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:01.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:01.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:01.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:01.416 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:01.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:01.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:01.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:01.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:01.545 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:01.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:01.582 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:16:01.646 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:01.675 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:01.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:01.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:01 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:16:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:01 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3615287829' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:16:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:01 smithi136 ceph-mon[109861]: osdmap e1642: 8 total, 8 up, 6 in 2024-08-17T18:16:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:01 smithi136 ceph-mon[109861]: pgmap v3098: 109 pgs: 1 activating+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 84 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped, 1 activating, 1 activating+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 activating+undersized+remapped; 6.7 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 124 op/s; 55115/329340 objects degraded (16.735%); 34426/329340 objects misplaced (10.453%); 7.8 MiB/s, 126 objects/s recovering 2024-08-17T18:16:01.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:01 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1969667406' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:16:01.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:01 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:16:01.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:01 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3615287829' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:16:01.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:01 smithi136 ceph-mon[115078]: osdmap e1642: 8 total, 8 up, 6 in 2024-08-17T18:16:01.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:01 smithi136 ceph-mon[115078]: pgmap v3098: 109 pgs: 1 activating+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 84 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped, 1 activating, 1 activating+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 activating+undersized+remapped; 6.7 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 124 op/s; 55115/329340 objects degraded (16.735%); 34426/329340 objects misplaced (10.453%); 7.8 MiB/s, 126 objects/s recovering 2024-08-17T18:16:01.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:01 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1969667406' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:16:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:01 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:16:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:01 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3615287829' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:16:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:01 smithi165 ceph-mon[105517]: osdmap e1642: 8 total, 8 up, 6 in 2024-08-17T18:16:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:01 smithi165 ceph-mon[105517]: pgmap v3098: 109 pgs: 1 activating+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 84 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped, 1 activating, 1 activating+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 activating+undersized+remapped; 6.7 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 124 op/s; 55115/329340 objects degraded (16.735%); 34426/329340 objects misplaced (10.453%); 7.8 MiB/s, 126 objects/s recovering 2024-08-17T18:16:01.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:01 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1969667406' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:16:01.777 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:01.801 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:01.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:01.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:01.903 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:01.912 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:16:01.912 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:16:01.928 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:01.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:01.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:02.030 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:02.052 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:02.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:02.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:02.154 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:02.182 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:02.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:02.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:02.284 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:02.317 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:02.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:02.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:02.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:02.448 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:02.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:02.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:02.550 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:02.576 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:02.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:02.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:02.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:02 smithi165 ceph-mon[105517]: osdmap e1643: 8 total, 8 up, 6 in 2024-08-17T18:16:02.632 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-17T18:16:02.658 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:02 smithi136 ceph-mon[109861]: osdmap e1643: 8 total, 8 up, 6 in 2024-08-17T18:16:02.658 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:02 smithi136 ceph-mon[115078]: osdmap e1643: 8 total, 8 up, 6 in 2024-08-17T18:16:02.679 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:02.741 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:02.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:02.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:02.843 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:02.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:02.866 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:02.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:02.967 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:02.978 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:16:02.998 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:02.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:02.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:03.100 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:03.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:03.124 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:03.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:03.226 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:03.256 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:03.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:03.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:03.358 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:03.380 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:03.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:03.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:03.482 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:03.504 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:03.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:03.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:03.600 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:16:03.606 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:03.622 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:03 smithi136 ceph-mon[109861]: pgmap v3100: 109 pgs: 1 active+remapped, 1 activating+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 1 active+recovery_wait, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait; 6.7 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 71 op/s; 55087/328584 objects degraded (16.765%); 34329/328584 objects misplaced (10.448%); 6.4 MiB/s, 104 objects/s recovering 2024-08-17T18:16:03.622 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:03 smithi136 ceph-mon[115078]: pgmap v3100: 109 pgs: 1 active+remapped, 1 activating+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 1 active+recovery_wait, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait; 6.7 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 71 op/s; 55087/328584 objects degraded (16.765%); 34329/328584 objects misplaced (10.448%); 6.4 MiB/s, 104 objects/s recovering 2024-08-17T18:16:03.633 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:03.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:03.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:03.735 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:03.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:03 smithi165 ceph-mon[105517]: pgmap v3100: 109 pgs: 1 active+remapped, 1 activating+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 1 active+recovery_wait, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait; 6.7 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 71 op/s; 55087/328584 objects degraded (16.765%); 34329/328584 objects misplaced (10.448%); 6.4 MiB/s, 104 objects/s recovering 2024-08-17T18:16:03.761 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:03.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:03.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:03.863 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:03.888 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:03.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:03.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:03.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:04.017 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:04.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:04.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:04.121 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:04.150 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:04.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:04.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:04.252 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:04.278 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:04.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:04.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:04.380 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:04.394 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:16:04.403 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:04.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:04.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:04.505 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:04.534 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:04.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:04.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:04.566 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-08-17T18:16:04.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:04.637 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:16:04.669 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:04.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:04.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:04.675 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:16:04.771 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:04.798 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:04.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:04.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:04.900 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:04.922 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:04.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:04.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:05.024 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:05.056 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:05.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:05.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:05.158 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:05.192 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:05.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:05.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:05.294 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:05.329 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:05.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:05.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:05.431 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:05.453 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:05.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:05.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:05.555 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:05.576 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:05.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:05.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:05.678 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:05.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:05 smithi136 ceph-mon[115078]: pgmap v3101: 109 pgs: 1 active+remapped, 1 activating+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 1 active+recovery_wait, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait; 6.7 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 69 op/s; 55087/328335 objects degraded (16.778%); 34311/328335 objects misplaced (10.450%); 5.4 MiB/s, 87 objects/s recovering 2024-08-17T18:16:05.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:05 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 55087/328335 objects degraded (16.778%), 16 pgs degraded (PG_DEGRADED) 2024-08-17T18:16:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:05 smithi136 ceph-mon[109861]: pgmap v3101: 109 pgs: 1 active+remapped, 1 activating+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 1 active+recovery_wait, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait; 6.7 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 69 op/s; 55087/328335 objects degraded (16.778%); 34311/328335 objects misplaced (10.450%); 5.4 MiB/s, 87 objects/s recovering 2024-08-17T18:16:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:05 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 55087/328335 objects degraded (16.778%), 16 pgs degraded (PG_DEGRADED) 2024-08-17T18:16:05.698 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:05.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:05.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:05 smithi165 ceph-mon[105517]: pgmap v3101: 109 pgs: 1 active+remapped, 1 activating+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 1 active+recovery_wait, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait; 6.7 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 69 op/s; 55087/328335 objects degraded (16.778%); 34311/328335 objects misplaced (10.450%); 5.4 MiB/s, 87 objects/s recovering 2024-08-17T18:16:05.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:05 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 55087/328335 objects degraded (16.778%), 16 pgs degraded (PG_DEGRADED) 2024-08-17T18:16:05.800 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:05.822 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:05.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:05.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:05.924 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:05.947 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:05.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:05.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:06.049 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:06.070 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:06.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:06.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:06.089 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:16:06.173 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:06.196 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:06.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:06.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:06.298 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:06.320 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:06.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:06.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:06.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:06.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:16:06.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:16:06.452 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:06.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:06.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:06.458 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:16:06.459 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:16:06.553 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:06.574 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:06.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:06.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:06.676 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:06.698 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:06.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:06.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:16:06.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:06.832 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:06.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:06.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:06.934 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:06.964 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:06.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:06.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:07.066 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:07.087 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:07.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:07.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:07.179 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-17T18:16:07.189 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:07.211 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:07.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:07.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:07.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:07.339 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:07.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:07.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:07.441 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:07.462 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:07.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:07.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:07.464 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:16:07.546 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:07 smithi136 ceph-mon[109861]: pgmap v3102: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.7 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 95 op/s; 55001/327075 objects degraded (16.816%); 33714/327075 objects misplaced (10.308%); 8.4 MiB/s, 134 objects/s recovering 2024-08-17T18:16:07.546 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:07 smithi136 ceph-mon[115078]: pgmap v3102: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.7 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 95 op/s; 55001/327075 objects degraded (16.816%); 33714/327075 objects misplaced (10.308%); 8.4 MiB/s, 134 objects/s recovering 2024-08-17T18:16:07.547 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:16:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:16:07] "GET /metrics HTTP/1.1" 200 36734 "" "Prometheus/2.51.0" 2024-08-17T18:16:07.564 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:07.587 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:07.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:07.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:07.688 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:07.718 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:07.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:07.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:07.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:07 smithi165 ceph-mon[105517]: pgmap v3102: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.7 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 95 op/s; 55001/327075 objects degraded (16.816%); 33714/327075 objects misplaced (10.308%); 8.4 MiB/s, 134 objects/s recovering 2024-08-17T18:16:07.820 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:07.841 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:07.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:07.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:07.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:07.965 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:07.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:07.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:08.067 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:08.095 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:08.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:08.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:08.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:08.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:08.220 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:08.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:08.321 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:08.352 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:08.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:08.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:08.453 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:08.477 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:08.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:08.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:08.579 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:08.602 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:08.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:08.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:08.704 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:08.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:08.747 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:08.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:08.848 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:08.870 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:08.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:08.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:08.972 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:08.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:08.995 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:08.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:09.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:09.123 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:09.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:09.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:09.183 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:16:09.183 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:16:09.183 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:16:09.183 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:16:09.184 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:16:09.184 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:16:09.184 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:16:09.184 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:16:09.184 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:16:09.184 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:16:09.184 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:16:09.185 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:16:09.185 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:16:09.185 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:16:09.185 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:16:09.185 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:16:09.225 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:09.246 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:09.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:09.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:09.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:09.379 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:09.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:09.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:09.441 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:16:09.441 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:16:09.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:09.441 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(57.4 1643/1637 [PushOp(57:259bb1ee:::benchmark_data_smithi136_389235_object55006:head, version: 1613'3373, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(57:259bb1ee:::benchmark_data_smithi136_389235_object55006:head@1613'3373, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-17T18:16:09.442 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:09.439949+0000", 2024-08-17T18:16:09.442 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00059975499999999999, 2024-08-17T18:16:09.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064586799999999996, 2024-08-17T18:16:09.442 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:16:09.442 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:16:09.442 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:16:09.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:09.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:16:09.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:09.439949+0000", 2024-08-17T18:16:09.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:09.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:09.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:09.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:16:09.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:09.439949+0000", 2024-08-17T18:16:09.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:09.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:09.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:09.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:16:09.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:09.439948+0000", 2024-08-17T18:16:09.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:16:09.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:09.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:09.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:16:09.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:09.439983+0000", 2024-08-17T18:16:09.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4938999999999999e-05 2024-08-17T18:16:09.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:09.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:09.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:16:09.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:09.440007+0000", 2024-08-17T18:16:09.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4159999999999999e-05 2024-08-17T18:16:09.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:09.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:09.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:16:09.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:09.440023+0000", 2024-08-17T18:16:09.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6010000000000001e-05 2024-08-17T18:16:09.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:09.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:09.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:16:09.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:09.440111+0000", 2024-08-17T18:16:09.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7787999999999994e-05 2024-08-17T18:16:09.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:09.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:09.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:16:09.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:09.440127+0000", 2024-08-17T18:16:09.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5985999999999999e-05 2024-08-17T18:16:09.445 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:09.445 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:16:09.445 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:09.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:09.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:09.445 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:124617 57.16 e1643/1637 57:6f88408b:::benchmark_data_smithi136_389235_object8695:head v 1643'6222, mlcod=1612'388)", 2024-08-17T18:16:09.445 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:09.440231+0000", 2024-08-17T18:16:09.446 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.00031794000000000001, 2024-08-17T18:16:09.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042541500000000001, 2024-08-17T18:16:09.446 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:16:09.446 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:16:09.446 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:16:09.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:09.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:16:09.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:09.440231+0000", 2024-08-17T18:16:09.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:09.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:09.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:09.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:16:09.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:09.440231+0000", 2024-08-17T18:16:09.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:09.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:09.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:09.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:16:09.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:09.440231+0000", 2024-08-17T18:16:09.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:16:09.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:09.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:09.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:16:09.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:09.440237+0000", 2024-08-17T18:16:09.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4590000000000003e-06 2024-08-17T18:16:09.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:09.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:09.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:16:09.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:09.440247+0000", 2024-08-17T18:16:09.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5349999999999993e-06 2024-08-17T18:16:09.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:09.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:09.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:16:09.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:09.440253+0000", 2024-08-17T18:16:09.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2099999999999998e-06 2024-08-17T18:16:09.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:09.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:09.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:16:09.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:09.440300+0000", 2024-08-17T18:16:09.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7685000000000002e-05 2024-08-17T18:16:09.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:09.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:09.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:16:09.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:09.440326+0000", 2024-08-17T18:16:09.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6009000000000001e-05 2024-08-17T18:16:09.452 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:09.452 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:16:09.452 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:09.452 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:09.452 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-08-17T18:16:09.452 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 2 2024-08-17T18:16:09.452 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:16:09.481 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:09.503 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:09.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:09.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:09.587 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:09 smithi165 ceph-mon[105517]: pgmap v3103: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+remapped, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.6 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 213 op/s; 54879/324219 objects degraded (16.927%); 33280/324219 objects misplaced (10.265%); 6.5 MiB/s, 103 objects/s recovering 2024-08-17T18:16:09.587 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:09 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2470863569' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:16:09.606 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:09.628 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:09.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:09.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:09 smithi136 ceph-mon[109861]: pgmap v3103: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+remapped, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.6 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 213 op/s; 54879/324219 objects degraded (16.927%); 33280/324219 objects misplaced (10.265%); 6.5 MiB/s, 103 objects/s recovering 2024-08-17T18:16:09.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:09 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2470863569' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:16:09.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:09 smithi136 ceph-mon[115078]: pgmap v3103: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+remapped, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.6 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 213 op/s; 54879/324219 objects degraded (16.927%); 33280/324219 objects misplaced (10.265%); 6.5 MiB/s, 103 objects/s recovering 2024-08-17T18:16:09.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:09 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2470863569' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:16:09.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:09.760 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:09.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:09.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:09.862 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:09.893 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:09.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:09.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:09.995 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:10.028 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:10.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:10.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:10.130 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:10.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:10.167 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:10.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:10.268 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:10.297 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:10.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:10.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:10.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:10.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:10.428 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:10.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:10.530 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:10.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:10.556 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:10.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:10.595 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-17T18:16:10.657 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:10.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:10.683 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:10.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:10.701 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:10 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 54879/324219 objects degraded (16.927%), 16 pgs degraded (PG_DEGRADED) 2024-08-17T18:16:10.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:10 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 54879/324219 objects degraded (16.927%), 16 pgs degraded (PG_DEGRADED) 2024-08-17T18:16:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:10 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 54879/324219 objects degraded (16.927%), 16 pgs degraded (PG_DEGRADED) 2024-08-17T18:16:10.784 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:10.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:10.807 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:10.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:10.908 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:10.924 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:16:10.931 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:10.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:10.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:11.033 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:11.055 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:11.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:11.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:11.157 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:11.178 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:11.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:11.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:11.280 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:11.314 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:11.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:11.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:11.416 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:11.438 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:11.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:11.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:11.457 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:16:11.540 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:11.561 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:11.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:11.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:11.663 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:11.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:11.693 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:11.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:11.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:11 smithi136 ceph-mon[115078]: pgmap v3104: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+remapped, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.6 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 170 op/s; 54879/324219 objects degraded (16.927%); 33280/324219 objects misplaced (10.265%); 5.2 MiB/s, 83 objects/s recovering 2024-08-17T18:16:11.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:11 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3736894778' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:16:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:11 smithi136 ceph-mon[109861]: pgmap v3104: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+remapped, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.6 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 170 op/s; 54879/324219 objects degraded (16.927%); 33280/324219 objects misplaced (10.265%); 5.2 MiB/s, 83 objects/s recovering 2024-08-17T18:16:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:11 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3736894778' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:16:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:11 smithi165 ceph-mon[105517]: pgmap v3104: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+remapped, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.6 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 170 op/s; 54879/324219 objects degraded (16.927%); 33280/324219 objects misplaced (10.265%); 5.2 MiB/s, 83 objects/s recovering 2024-08-17T18:16:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:11 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3736894778' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:16:11.794 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:11.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:11.828 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:11.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:11.897 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:16:11.929 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:11.952 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:11.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:11.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:12.054 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:12.076 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:12.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:12.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:12.178 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:12.189 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-08-17T18:16:12.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:12.200 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:12.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:12.300 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:12.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:12.341 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:12.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:12.442 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:12.466 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:12.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:12.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:12.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:12 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:16:12.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:12 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3736894778' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:16:12.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:12 smithi165 ceph-mon[105517]: osdmap e1644: 8 total, 8 up, 6 in 2024-08-17T18:16:12.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:12 smithi165 ceph-mon[105517]: 1.0 scrub starts 2024-08-17T18:16:12.567 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:12.590 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:12.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:12.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:12.692 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:12.727 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:12.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:12.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:12.777 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:12 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:16:12.777 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:12 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3736894778' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:16:12.777 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:12 smithi136 ceph-mon[115078]: osdmap e1644: 8 total, 8 up, 6 in 2024-08-17T18:16:12.777 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:12 smithi136 ceph-mon[115078]: 1.0 scrub starts 2024-08-17T18:16:12.778 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:12 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:16:12.778 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:12 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3736894778' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:16:12.778 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:12 smithi136 ceph-mon[109861]: osdmap e1644: 8 total, 8 up, 6 in 2024-08-17T18:16:12.778 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:12 smithi136 ceph-mon[109861]: 1.0 scrub starts 2024-08-17T18:16:12.829 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:12.831 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:16:12.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:12.879 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:12.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:12.944 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:16:12.945 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:16:12.945 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-08-17T18:16:12.945 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-08-17T18:16:12.945 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:16:12.980 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:13.002 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:13.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:13.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:13.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:13.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:13.135 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:13.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:13.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:13.265 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:13.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:13.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:13.367 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:13.388 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:13.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:13.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:13.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:13.522 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:13.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:13.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:13.624 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:13.657 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:13.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:13.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:13.735 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:13 smithi136 ceph-mon[109861]: pgmap v3106: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 1 active+remapped, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.5 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 294 op/s; 54701/319752 objects degraded (17.107%); 32386/319752 objects misplaced (10.128%); 9.1 MiB/s, 145 objects/s recovering 2024-08-17T18:16:13.736 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:13 smithi136 ceph-mon[115078]: pgmap v3106: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 1 active+remapped, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.5 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 294 op/s; 54701/319752 objects degraded (17.107%); 32386/319752 objects misplaced (10.128%); 9.1 MiB/s, 145 objects/s recovering 2024-08-17T18:16:13.759 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:13.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:13 smithi165 ceph-mon[105517]: pgmap v3106: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 1 active+remapped, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.5 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 294 op/s; 54701/319752 objects degraded (17.107%); 32386/319752 objects misplaced (10.128%); 9.1 MiB/s, 145 objects/s recovering 2024-08-17T18:16:13.791 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:13.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:13.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:13.893 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:13.927 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:13.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:13.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:14.029 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:14.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:14.052 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:14.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:14.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:14.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:14.177 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:14.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:14.278 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:14.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-17T18:16:14.306 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:14.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:14.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:14.408 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:14.430 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:14.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:14.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:14.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:14 smithi165 ceph-mon[105517]: 1.0 scrub starts 2024-08-17T18:16:14.532 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:14.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:14.554 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:14.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:14.557 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:16:14.656 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:14.683 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:14.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:14.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:14.705 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:16:14.721 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:14 smithi136 ceph-mon[109861]: 1.0 scrub starts 2024-08-17T18:16:14.721 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:14 smithi136 ceph-mon[115078]: 1.0 scrub starts 2024-08-17T18:16:14.785 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:14.804 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:16:14.817 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:14.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:14.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:14.919 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:14.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:14.955 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:14.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:15.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:15.079 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:15.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:15.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:15.180 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:15.202 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:15.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:15.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:15.305 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:15.336 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:15.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:15.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:15.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:15.465 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:15.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:15.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:15.567 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:15.590 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:15.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:15.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:15.624 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:15 smithi136 ceph-mon[109861]: pgmap v3107: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 1 active+remapped, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.5 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 286 op/s; 54701/319752 objects degraded (17.107%); 32386/319752 objects misplaced (10.128%); 9.1 MiB/s, 145 objects/s recovering 2024-08-17T18:16:15.625 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:15 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 54701/319752 objects degraded (17.107%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:16:15.625 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:15 smithi136 ceph-mon[109861]: 1.0 scrub starts 2024-08-17T18:16:15.625 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:15 smithi136 ceph-mon[115078]: pgmap v3107: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 1 active+remapped, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.5 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 286 op/s; 54701/319752 objects degraded (17.107%); 32386/319752 objects misplaced (10.128%); 9.1 MiB/s, 145 objects/s recovering 2024-08-17T18:16:15.625 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:15 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 54701/319752 objects degraded (17.107%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:16:15.625 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:15 smithi136 ceph-mon[115078]: 1.0 scrub starts 2024-08-17T18:16:15.646 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-08-17T18:16:15.692 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:15.752 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:15.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:15.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:15.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:15 smithi165 ceph-mon[105517]: pgmap v3107: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 1 active+remapped, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.5 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 286 op/s; 54701/319752 objects degraded (17.107%); 32386/319752 objects misplaced (10.128%); 9.1 MiB/s, 145 objects/s recovering 2024-08-17T18:16:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:15 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 54701/319752 objects degraded (17.107%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:16:15.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:15 smithi165 ceph-mon[105517]: 1.0 scrub starts 2024-08-17T18:16:15.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:15.881 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:15.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:15.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:15.983 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:16.008 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:16.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:16.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:16.110 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:16.116 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:16:16.142 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:16.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:16.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:16.244 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:16.278 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:16.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:16.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:16.380 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:16.401 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:16.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:16.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:16.471 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:16:16.472 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T18:16:16.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T18:16:16.472 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:16:16.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.472 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(57.4 1638/1637 [PushOp(57:255d77a7:::benchmark_data_smithi136_389235_object6558:head, version: 1612'398, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(57:255d77a7:::benchmark_data_smithi136_389235_object6558:head@1612'398, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-17T18:16:16.472 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:53.838992+0000", 2024-08-17T18:16:16.472 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.628228601, 2024-08-17T18:16:16.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018792053999999999, 2024-08-17T18:16:16.473 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:16:16.473 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:16:16.473 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:16:16.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:16:16.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:53.838992+0000", 2024-08-17T18:16:16.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:16:16.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:53.838992+0000", 2024-08-17T18:16:16.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:16:16.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:53.838990+0000", 2024-08-17T18:16:16.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:16:16.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:16:16.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:53.856257+0000", 2024-08-17T18:16:16.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017267141 2024-08-17T18:16:16.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:16:16.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:53.856279+0000", 2024-08-17T18:16:16.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2127e-05 2024-08-17T18:16:16.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:16:16.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:53.856292+0000", 2024-08-17T18:16:16.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2988e-05 2024-08-17T18:16:16.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:16:16.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:53.856354+0000", 2024-08-17T18:16:16.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1080999999999993e-05 2024-08-17T18:16:16.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:16:16.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:53.856365+0000", 2024-08-17T18:16:16.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1172000000000001e-05 2024-08-17T18:16:16.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:16:16.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:53.857784+0000", 2024-08-17T18:16:16.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001418859 2024-08-17T18:16:16.477 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.477 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:16:16.477 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.477 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(57.4 1639/1637 [PushOp(57:234d17e1:::benchmark_data_smithi136_389235_object8004:head, version: 1612'481, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(57:234d17e1:::benchmark_data_smithi136_389235_object8004:head@1612'481, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-17T18:16:16.477 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:54.274764+0000", 2024-08-17T18:16:16.477 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.192455816999999, 2024-08-17T18:16:16.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014243666, 2024-08-17T18:16:16.478 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:16:16.478 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:16:16.478 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:16:16.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:16:16.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.274764+0000", 2024-08-17T18:16:16.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:16:16.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.274764+0000", 2024-08-17T18:16:16.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:16:16.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.274764+0000", 2024-08-17T18:16:16.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:16:16.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:16:16.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.274809+0000", 2024-08-17T18:16:16.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5568999999999997e-05 2024-08-17T18:16:16.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:16:16.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.286916+0000", 2024-08-17T18:16:16.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012107087000000001 2024-08-17T18:16:16.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:16:16.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.286942+0000", 2024-08-17T18:16:16.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5791000000000001e-05 2024-08-17T18:16:16.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:16:16.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.287016+0000", 2024-08-17T18:16:16.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3466000000000004e-05 2024-08-17T18:16:16.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:16:16.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.287031+0000", 2024-08-17T18:16:16.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4904e-05 2024-08-17T18:16:16.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:16:16.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.289008+0000", 2024-08-17T18:16:16.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019774760000000001 2024-08-17T18:16:16.482 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.482 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:16:16.482 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.482 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:122062 57.7 e1639/1638 57:e1d83814:::benchmark_data_smithi136_389235_object6140:head v 1639'6214, mlcod=0'0)", 2024-08-17T18:16:16.482 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:54.473613+0000", 2024-08-17T18:16:16.482 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.993606786000001, 2024-08-17T18:16:16.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020281128999999998, 2024-08-17T18:16:16.483 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:16:16.483 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:16.483 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:16:16.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:16:16.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.473613+0000", 2024-08-17T18:16:16.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:16:16.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.473613+0000", 2024-08-17T18:16:16.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:16:16.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.473613+0000", 2024-08-17T18:16:16.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:16:16.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:16:16.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.473618+0000", 2024-08-17T18:16:16.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4389999999999999e-06 2024-08-17T18:16:16.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:16:16.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.473632+0000", 2024-08-17T18:16:16.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3271999999999999e-05 2024-08-17T18:16:16.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:16:16.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.473639+0000", 2024-08-17T18:16:16.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3010000000000001e-06 2024-08-17T18:16:16.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:16:16.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.473711+0000", 2024-08-17T18:16:16.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1938000000000001e-05 2024-08-17T18:16:16.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:16:16.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.481952+0000", 2024-08-17T18:16:16.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082410980000000005 2024-08-17T18:16:16.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:16:16.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.493569+0000", 2024-08-17T18:16:16.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011616631000000001 2024-08-17T18:16:16.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:16:16.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.493894+0000", 2024-08-17T18:16:16.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00032575399999999998 2024-08-17T18:16:16.487 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.487 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:16:16.487 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.488 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:122074 57.7 e1639/1638 57:e209ec89:::benchmark_data_smithi136_389235_object6152:head v 1639'6215, mlcod=0'0)", 2024-08-17T18:16:16.488 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:54.476897+0000", 2024-08-17T18:16:16.488 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.990323513, 2024-08-17T18:16:16.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017065542, 2024-08-17T18:16:16.488 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:16:16.488 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:16.488 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:16:16.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:16:16.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.476897+0000", 2024-08-17T18:16:16.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:16:16.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.476897+0000", 2024-08-17T18:16:16.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:16:16.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.476895+0000", 2024-08-17T18:16:16.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:16:16.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:16:16.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.476904+0000", 2024-08-17T18:16:16.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0920000000000004e-06 2024-08-17T18:16:16.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:16:16.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.476919+0000", 2024-08-17T18:16:16.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5095e-05 2024-08-17T18:16:16.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:16:16.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.476928+0000", 2024-08-17T18:16:16.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7800000000000006e-06 2024-08-17T18:16:16.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:16:16.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.487526+0000", 2024-08-17T18:16:16.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010597641 2024-08-17T18:16:16.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:16:16.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.487557+0000", 2024-08-17T18:16:16.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0753999999999997e-05 2024-08-17T18:16:16.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:16:16.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.493900+0000", 2024-08-17T18:16:16.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006343413 2024-08-17T18:16:16.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:16:16.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.493962+0000", 2024-08-17T18:16:16.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2125999999999994e-05 2024-08-17T18:16:16.493 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.493 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:16:16.493 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.493 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(57.4 1639/1637 [PushOp(57:2104cd40:::benchmark_data_smithi136_389235_object8254:head, version: 1612'499, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(57:2104cd40:::benchmark_data_smithi136_389235_object8254:head@1612'499, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-17T18:16:16.493 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:54.508098+0000", 2024-08-17T18:16:16.493 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.959121767999999, 2024-08-17T18:16:16.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012346317000000001, 2024-08-17T18:16:16.494 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:16:16.494 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:16:16.494 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:16:16.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:16:16.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.508098+0000", 2024-08-17T18:16:16.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:16:16.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.508098+0000", 2024-08-17T18:16:16.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:16:16.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.508096+0000", 2024-08-17T18:16:16.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:16:16.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:16:16.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.508143+0000", 2024-08-17T18:16:16.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6672000000000003e-05 2024-08-17T18:16:16.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:16:16.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.508166+0000", 2024-08-17T18:16:16.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2806000000000002e-05 2024-08-17T18:16:16.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:16:16.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.508183+0000", 2024-08-17T18:16:16.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7479e-05 2024-08-17T18:16:16.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:16:16.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.508831+0000", 2024-08-17T18:16:16.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00064794199999999998 2024-08-17T18:16:16.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:16:16.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.508853+0000", 2024-08-17T18:16:16.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1441e-05 2024-08-17T18:16:16.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:16:16.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.520445+0000", 2024-08-17T18:16:16.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011591884 2024-08-17T18:16:16.498 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.498 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:16:16.498 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.498 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:122080 57.1 e1639/1637 57:81b3e6cb:::benchmark_data_smithi136_389235_object6158:head v 1639'6393, mlcod=1638'6391)", 2024-08-17T18:16:16.498 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:54.519625+0000", 2024-08-17T18:16:16.498 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.947594898999998, 2024-08-17T18:16:16.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011604830999999999, 2024-08-17T18:16:16.499 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:16:16.499 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:16.499 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:16:16.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:16:16.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.519625+0000", 2024-08-17T18:16:16.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:16:16.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.519625+0000", 2024-08-17T18:16:16.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:16:16.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.519624+0000", 2024-08-17T18:16:16.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:16:16.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:16:16.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.519634+0000", 2024-08-17T18:16:16.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0297e-05 2024-08-17T18:16:16.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:16:16.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.519653+0000", 2024-08-17T18:16:16.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8736000000000002e-05 2024-08-17T18:16:16.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:16:16.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.519662+0000", 2024-08-17T18:16:16.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9199999999999993e-06 2024-08-17T18:16:16.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:16:16.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.521142+0000", 2024-08-17T18:16:16.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001480355 2024-08-17T18:16:16.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:16:16.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.521166+0000", 2024-08-17T18:16:16.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3997000000000001e-05 2024-08-17T18:16:16.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:16:16.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.531174+0000", 2024-08-17T18:16:16.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010008019999999999 2024-08-17T18:16:16.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:16:16.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.531230+0000", 2024-08-17T18:16:16.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5779999999999998e-05 2024-08-17T18:16:16.503 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.503 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:16:16.503 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.504 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:125112 57.14 e1643/1637 57:295571b6:::benchmark_data_smithi136_389235_object9190:head v 1643'6309, mlcod=1612'374)", 2024-08-17T18:16:16.504 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:11.250885+0000", 2024-08-17T18:16:16.504 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.2163354379999998, 2024-08-17T18:16:16.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012966872000000001, 2024-08-17T18:16:16.504 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:16:16.504 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:16.504 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:16:16.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:16:16.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.250885+0000", 2024-08-17T18:16:16.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:16:16.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.250885+0000", 2024-08-17T18:16:16.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:16:16.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.250883+0000", 2024-08-17T18:16:16.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:16:16.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:16:16.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.250892+0000", 2024-08-17T18:16:16.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3129999999999999e-06 2024-08-17T18:16:16.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:16:16.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.250907+0000", 2024-08-17T18:16:16.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5415999999999999e-05 2024-08-17T18:16:16.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:16:16.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.250914+0000", 2024-08-17T18:16:16.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7850000000000003e-06 2024-08-17T18:16:16.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:16:16.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.250968+0000", 2024-08-17T18:16:16.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4113999999999997e-05 2024-08-17T18:16:16.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:16:16.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.250993+0000", 2024-08-17T18:16:16.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4443999999999998e-05 2024-08-17T18:16:16.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:16:16.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.263619+0000", 2024-08-17T18:16:16.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012626661000000001 2024-08-17T18:16:16.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:16:16.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.263852+0000", 2024-08-17T18:16:16.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023235000000000001 2024-08-17T18:16:16.508 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.508 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:16:16.509 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.509 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:125256 57.8 e1644/1637 57:15755085:::benchmark_data_smithi136_389235_object9334:head v 1644'6385, mlcod=1612'390)", 2024-08-17T18:16:16.509 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:11.800262+0000", 2024-08-17T18:16:16.509 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.6669578959999996, 2024-08-17T18:16:16.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018951777999999999, 2024-08-17T18:16:16.509 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:16:16.509 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:16.509 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:16:16.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:16:16.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.800262+0000", 2024-08-17T18:16:16.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:16:16.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.800262+0000", 2024-08-17T18:16:16.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:16:16.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.800261+0000", 2024-08-17T18:16:16.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:16:16.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:16:16.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.800271+0000", 2024-08-17T18:16:16.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4739999999999994e-06 2024-08-17T18:16:16.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:16:16.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.800285+0000", 2024-08-17T18:16:16.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4861000000000001e-05 2024-08-17T18:16:16.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:16:16.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.800294+0000", 2024-08-17T18:16:16.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1629999999999996e-06 2024-08-17T18:16:16.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:16:16.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.801016+0000", 2024-08-17T18:16:16.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00072281000000000005 2024-08-17T18:16:16.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:16:16.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.801040+0000", 2024-08-17T18:16:16.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3887999999999999e-05 2024-08-17T18:16:16.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:16:16.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.819184+0000", 2024-08-17T18:16:16.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018144090000000002 2024-08-17T18:16:16.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:16:16.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.819214+0000", 2024-08-17T18:16:16.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9671000000000001e-05 2024-08-17T18:16:16.514 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.514 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:16:16.514 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.514 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:125257 57.8 e1644/1637 57:152df420:::benchmark_data_smithi136_389235_object9335:head v 1644'6386, mlcod=1612'390)", 2024-08-17T18:16:16.514 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:11.801937+0000", 2024-08-17T18:16:16.514 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.66528308, 2024-08-17T18:16:16.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01730307, 2024-08-17T18:16:16.514 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:16:16.515 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:16.515 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:16:16.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:16:16.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.801937+0000", 2024-08-17T18:16:16.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:16:16.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.801937+0000", 2024-08-17T18:16:16.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:16:16.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.801936+0000", 2024-08-17T18:16:16.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:16:16.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:16:16.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.801943+0000", 2024-08-17T18:16:16.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.72e-06 2024-08-17T18:16:16.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:16:16.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.806845+0000", 2024-08-17T18:16:16.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049022409999999999 2024-08-17T18:16:16.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:16:16.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.806855+0000", 2024-08-17T18:16:16.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0536999999999999e-05 2024-08-17T18:16:16.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:16:16.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.809944+0000", 2024-08-17T18:16:16.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030886379999999999 2024-08-17T18:16:16.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:16:16.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.809972+0000", 2024-08-17T18:16:16.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7858e-05 2024-08-17T18:16:16.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:16:16.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.819220+0000", 2024-08-17T18:16:16.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0092476999999999993 2024-08-17T18:16:16.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:16:16.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:11.819240+0000", 2024-08-17T18:16:16.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0446e-05 2024-08-17T18:16:16.519 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.519 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:16:16.519 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.519 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:125404 57.14 e1644/1637 57:2c65d4e5:::benchmark_data_smithi136_389235_object9482:head v 1644'6321, mlcod=1612'374)", 2024-08-17T18:16:16.519 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:12.470946+0000", 2024-08-17T18:16:16.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:16.521 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.9962737109999997, 2024-08-17T18:16:16.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011021067000000001, 2024-08-17T18:16:16.521 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:16:16.521 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:16.522 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:16:16.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:16:16.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:12.470946+0000", 2024-08-17T18:16:16.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:16:16.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:12.470946+0000", 2024-08-17T18:16:16.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:16:16.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:12.470945+0000", 2024-08-17T18:16:16.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:16:16.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:16:16.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:12.470953+0000", 2024-08-17T18:16:16.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2250000000000003e-06 2024-08-17T18:16:16.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:16:16.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:12.470964+0000", 2024-08-17T18:16:16.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1959e-05 2024-08-17T18:16:16.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:16:16.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:12.470971+0000", 2024-08-17T18:16:16.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9360000000000002e-06 2024-08-17T18:16:16.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:16:16.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:12.471036+0000", 2024-08-17T18:16:16.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4558000000000001e-05 2024-08-17T18:16:16.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:16:16.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:12.471061+0000", 2024-08-17T18:16:16.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4881000000000001e-05 2024-08-17T18:16:16.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:16:16.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:12.481915+0000", 2024-08-17T18:16:16.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010854610000000001 2024-08-17T18:16:16.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:16:16.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:12.481967+0000", 2024-08-17T18:16:16.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2015999999999998e-05 2024-08-17T18:16:16.526 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.526 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:16:16.526 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.526 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:125550 57.7 e1644/1638 57:e378a285:::benchmark_data_smithi136_389235_object9628:head v 1644'6323, mlcod=1612'382)", 2024-08-17T18:16:16.526 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:12.875738+0000", 2024-08-17T18:16:16.527 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.5914826039999999, 2024-08-17T18:16:16.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011693602000000001, 2024-08-17T18:16:16.527 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:16:16.527 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:16.527 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:16:16.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:16:16.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:12.875738+0000", 2024-08-17T18:16:16.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:16:16.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:12.875738+0000", 2024-08-17T18:16:16.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:16:16.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:12.875737+0000", 2024-08-17T18:16:16.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:16:16.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:16:16.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:12.875742+0000", 2024-08-17T18:16:16.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9690000000000002e-06 2024-08-17T18:16:16.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:16:16.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:12.875749+0000", 2024-08-17T18:16:16.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8480000000000003e-06 2024-08-17T18:16:16.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:16:16.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:12.875753+0000", 2024-08-17T18:16:16.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.425e-06 2024-08-17T18:16:16.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:16:16.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:12.876387+0000", 2024-08-17T18:16:16.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063342800000000001 2024-08-17T18:16:16.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:16:16.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:12.876410+0000", 2024-08-17T18:16:16.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3626999999999999e-05 2024-08-17T18:16:16.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:16:16.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:12.887392+0000", 2024-08-17T18:16:16.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010981225000000001 2024-08-17T18:16:16.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:16:16.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:12.887431+0000", 2024-08-17T18:16:16.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9495e-05 2024-08-17T18:16:16.531 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.531 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:16:16.531 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.532 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:125835 57.10 e1644/1638 57:0e944827:::benchmark_data_smithi136_389235_object9913:head v 1644'6429, mlcod=1612'385)", 2024-08-17T18:16:16.532 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:13.689871+0000", 2024-08-17T18:16:16.532 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.777349187, 2024-08-17T18:16:16.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012346322, 2024-08-17T18:16:16.532 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:16:16.532 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:16.532 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:16:16.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:16:16.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:13.689871+0000", 2024-08-17T18:16:16.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:16:16.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:13.689871+0000", 2024-08-17T18:16:16.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:16:16.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:13.689870+0000", 2024-08-17T18:16:16.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:16:16.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:16:16.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:13.689876+0000", 2024-08-17T18:16:16.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9850000000000003e-06 2024-08-17T18:16:16.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:16:16.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:13.689889+0000", 2024-08-17T18:16:16.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2366e-05 2024-08-17T18:16:16.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:16:16.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:13.689894+0000", 2024-08-17T18:16:16.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7289999999999997e-06 2024-08-17T18:16:16.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:16:16.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:13.689940+0000", 2024-08-17T18:16:16.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5997e-05 2024-08-17T18:16:16.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:16:16.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:13.689970+0000", 2024-08-17T18:16:16.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9632e-05 2024-08-17T18:16:16.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:16:16.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:13.702186+0000", 2024-08-17T18:16:16.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012215808 2024-08-17T18:16:16.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:16:16.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:13.702217+0000", 2024-08-17T18:16:16.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1405999999999997e-05 2024-08-17T18:16:16.537 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.537 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:16:16.537 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.537 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:125911 57.0 e1644/1638 57:00d15df0:::benchmark_data_smithi136_389235_object9989:head v 1644'6431, mlcod=1612'386)", 2024-08-17T18:16:16.537 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:13.859866+0000", 2024-08-17T18:16:16.537 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.6073544879999999, 2024-08-17T18:16:16.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010867642, 2024-08-17T18:16:16.537 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:16:16.537 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:16.537 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:16:16.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:16:16.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:13.859866+0000", 2024-08-17T18:16:16.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:16:16.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:13.859866+0000", 2024-08-17T18:16:16.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:16:16.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:13.859865+0000", 2024-08-17T18:16:16.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:16:16.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:16:16.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:13.859871+0000", 2024-08-17T18:16:16.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8300000000000001e-06 2024-08-17T18:16:16.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:16:16.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:13.859884+0000", 2024-08-17T18:16:16.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3424000000000001e-05 2024-08-17T18:16:16.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:16:16.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:13.859890+0000", 2024-08-17T18:16:16.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0769999999999996e-06 2024-08-17T18:16:16.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:16:16.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:13.859934+0000", 2024-08-17T18:16:16.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3946000000000002e-05 2024-08-17T18:16:16.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:16:16.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:13.859957+0000", 2024-08-17T18:16:16.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2742999999999999e-05 2024-08-17T18:16:16.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:16:16.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:13.870709+0000", 2024-08-17T18:16:16.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01075186 2024-08-17T18:16:16.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:16:16.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:13.870733+0000", 2024-08-17T18:16:16.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4559999999999999e-05 2024-08-17T18:16:16.542 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.542 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:16:16.542 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.542 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:126613 57.e e1644/1638 57:73ce39f1:::benchmark_data_smithi136_389235_object10691:head v 1644'7954, mlcod=1644'7952)", 2024-08-17T18:16:16.543 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:16.256248+0000", 2024-08-17T18:16:16.543 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.210971664, 2024-08-17T18:16:16.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01390487, 2024-08-17T18:16:16.543 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:16:16.543 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:16.543 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:16:16.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:16:16.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.256248+0000", 2024-08-17T18:16:16.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:16:16.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.256248+0000", 2024-08-17T18:16:16.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:16:16.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.256247+0000", 2024-08-17T18:16:16.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:16:16.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:16:16.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.256256+0000", 2024-08-17T18:16:16.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1400000000000006e-06 2024-08-17T18:16:16.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:16:16.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.256274+0000", 2024-08-17T18:16:16.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7388e-05 2024-08-17T18:16:16.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:16:16.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.256282+0000", 2024-08-17T18:16:16.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0860000000000002e-06 2024-08-17T18:16:16.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:16:16.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.257553+0000", 2024-08-17T18:16:16.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012709990000000001 2024-08-17T18:16:16.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:16:16.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.257575+0000", 2024-08-17T18:16:16.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2634e-05 2024-08-17T18:16:16.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:16:16.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.270116+0000", 2024-08-17T18:16:16.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012540427 2024-08-17T18:16:16.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:16:16.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.270153+0000", 2024-08-17T18:16:16.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7533000000000001e-05 2024-08-17T18:16:16.547 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.548 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:16:16.548 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.548 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:126614 57.6 e1644/1637 57:65962d0e:::benchmark_data_smithi136_389235_object10692:head v 1644'6349, mlcod=1612'390)", 2024-08-17T18:16:16.548 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:16.259032+0000", 2024-08-17T18:16:16.548 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.208188387, 2024-08-17T18:16:16.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011147539999999999, 2024-08-17T18:16:16.548 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:16:16.548 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:16.548 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:16:16.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:16:16.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.259032+0000", 2024-08-17T18:16:16.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:16:16.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.259032+0000", 2024-08-17T18:16:16.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:16:16.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.259031+0000", 2024-08-17T18:16:16.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:16:16.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:16:16.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.259037+0000", 2024-08-17T18:16:16.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2260000000000002e-06 2024-08-17T18:16:16.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:16:16.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.259047+0000", 2024-08-17T18:16:16.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3689999999999996e-06 2024-08-17T18:16:16.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:16:16.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.259052+0000", 2024-08-17T18:16:16.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6149999999999996e-06 2024-08-17T18:16:16.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:16:16.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.259211+0000", 2024-08-17T18:16:16.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000158288 2024-08-17T18:16:16.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:16:16.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.259235+0000", 2024-08-17T18:16:16.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4559e-05 2024-08-17T18:16:16.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:16:16.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.270156+0000", 2024-08-17T18:16:16.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010920994 2024-08-17T18:16:16.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:16:16.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.270179+0000", 2024-08-17T18:16:16.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3079999999999999e-05 2024-08-17T18:16:16.553 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.553 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:16:16.553 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.553 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:126621 57.15 e1644/1637 57:abbfeb81:::benchmark_data_smithi136_389235_object10699:head v 1644'6466, mlcod=1612'394)", 2024-08-17T18:16:16.553 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:16.275507+0000", 2024-08-17T18:16:16.553 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.19171307500000001, 2024-08-17T18:16:16.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011826844, 2024-08-17T18:16:16.554 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:16:16.554 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:16.554 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:16:16.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:16:16.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.275507+0000", 2024-08-17T18:16:16.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:16:16.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.275507+0000", 2024-08-17T18:16:16.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:16:16.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.275507+0000", 2024-08-17T18:16:16.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:16:16.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:16:16.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.275511+0000", 2024-08-17T18:16:16.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5850000000000001e-06 2024-08-17T18:16:16.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:16:16.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.275518+0000", 2024-08-17T18:16:16.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4540000000000004e-06 2024-08-17T18:16:16.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:16:16.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.275522+0000", 2024-08-17T18:16:16.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3810000000000001e-06 2024-08-17T18:16:16.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:16:16.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.278126+0000", 2024-08-17T18:16:16.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002604084 2024-08-17T18:16:16.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:16:16.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.278151+0000", 2024-08-17T18:16:16.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5086e-05 2024-08-17T18:16:16.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:16:16.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.287299+0000", 2024-08-17T18:16:16.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0091477030000000001 2024-08-17T18:16:16.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:16:16.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.287334+0000", 2024-08-17T18:16:16.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4952e-05 2024-08-17T18:16:16.558 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.558 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:16:16.558 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.558 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:126629 57.7 e1644/1638 57:e5e5a53e:::benchmark_data_smithi136_389235_object10707:head v 1644'6347, mlcod=1612'382)", 2024-08-17T18:16:16.559 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:16.290053+0000", 2024-08-17T18:16:16.559 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.17716669099999999, 2024-08-17T18:16:16.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011842333999999999, 2024-08-17T18:16:16.559 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:16:16.559 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:16.559 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:16:16.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:16:16.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.290053+0000", 2024-08-17T18:16:16.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:16:16.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.290053+0000", 2024-08-17T18:16:16.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:16:16.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.290052+0000", 2024-08-17T18:16:16.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:16:16.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:16:16.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.290063+0000", 2024-08-17T18:16:16.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0747999999999999e-05 2024-08-17T18:16:16.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:16:16.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.290091+0000", 2024-08-17T18:16:16.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7915999999999998e-05 2024-08-17T18:16:16.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:16:16.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.290098+0000", 2024-08-17T18:16:16.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2520000000000002e-06 2024-08-17T18:16:16.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:16:16.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.290160+0000", 2024-08-17T18:16:16.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1966999999999998e-05 2024-08-17T18:16:16.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:16:16.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.290187+0000", 2024-08-17T18:16:16.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6789000000000001e-05 2024-08-17T18:16:16.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:16:16.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.301856+0000", 2024-08-17T18:16:16.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011669252 2024-08-17T18:16:16.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:16:16.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.301896+0000", 2024-08-17T18:16:16.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9826000000000001e-05 2024-08-17T18:16:16.563 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.563 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:16:16.563 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.564 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:126632 57.6 e1644/1637 57:636ceac9:::benchmark_data_smithi136_389235_object10710:head v 1644'6352, mlcod=1612'390)", 2024-08-17T18:16:16.564 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:16.297793+0000", 2024-08-17T18:16:16.564 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.16942712600000001, 2024-08-17T18:16:16.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018956291, 2024-08-17T18:16:16.564 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:16:16.564 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:16.564 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:16:16.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:16:16.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.297793+0000", 2024-08-17T18:16:16.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:16:16.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.297793+0000", 2024-08-17T18:16:16.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:16:16.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.297792+0000", 2024-08-17T18:16:16.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:16:16.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:16:16.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.297801+0000", 2024-08-17T18:16:16.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.037e-06 2024-08-17T18:16:16.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:16:16.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.297812+0000", 2024-08-17T18:16:16.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1554e-05 2024-08-17T18:16:16.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:16:16.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.297819+0000", 2024-08-17T18:16:16.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8090000000000004e-06 2024-08-17T18:16:16.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:16:16.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.297865+0000", 2024-08-17T18:16:16.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6315e-05 2024-08-17T18:16:16.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:16:16.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.297896+0000", 2024-08-17T18:16:16.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0031000000000001e-05 2024-08-17T18:16:16.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:16:16.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.316729+0000", 2024-08-17T18:16:16.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018833783 2024-08-17T18:16:16.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:16:16.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.316749+0000", 2024-08-17T18:16:16.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9996e-05 2024-08-17T18:16:16.568 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.568 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:16:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:126638 57.6 e1644/1637 57:66cd2012:::benchmark_data_smithi136_389235_object10716:head v 1644'6353, mlcod=1612'390)", 2024-08-17T18:16:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:16.303440+0000", 2024-08-17T18:16:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.16378002699999999, 2024-08-17T18:16:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013330775, 2024-08-17T18:16:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:16:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:16:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:16:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.303440+0000", 2024-08-17T18:16:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:16:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.303440+0000", 2024-08-17T18:16:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:16:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.303439+0000", 2024-08-17T18:16:16.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:16:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:16:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.303447+0000", 2024-08-17T18:16:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9920000000000007e-06 2024-08-17T18:16:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:16:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.303461+0000", 2024-08-17T18:16:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4133e-05 2024-08-17T18:16:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:16:16.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.303468+0000", 2024-08-17T18:16:16.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5400000000000001e-06 2024-08-17T18:16:16.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:16:16.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.303529+0000", 2024-08-17T18:16:16.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1189e-05 2024-08-17T18:16:16.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:16:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.303554+0000", 2024-08-17T18:16:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4618999999999999e-05 2024-08-17T18:16:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:16:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.316753+0000", 2024-08-17T18:16:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013199287000000001 2024-08-17T18:16:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:16:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.316771+0000", 2024-08-17T18:16:16.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7997999999999999e-05 2024-08-17T18:16:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:16:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:126641 57.17 e1644/1638 57:ee890758:::benchmark_data_smithi136_389235_object10719:head v 1644'6276, mlcod=1612'381)", 2024-08-17T18:16:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:16.305082+0000", 2024-08-17T18:16:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.16213780999999999, 2024-08-17T18:16:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011484738, 2024-08-17T18:16:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:16:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:16.574 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:16:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:16:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.305082+0000", 2024-08-17T18:16:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:16:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.305082+0000", 2024-08-17T18:16:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:16:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:16:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.305081+0000", 2024-08-17T18:16:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:16:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:16:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.305088+0000", 2024-08-17T18:16:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7569999999999998e-06 2024-08-17T18:16:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:16:16.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.305101+0000", 2024-08-17T18:16:16.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2866e-05 2024-08-17T18:16:16.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:16:16.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.305107+0000", 2024-08-17T18:16:16.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5809999999999996e-06 2024-08-17T18:16:16.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:16:16.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.305158+0000", 2024-08-17T18:16:16.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1762999999999999e-05 2024-08-17T18:16:16.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:16:16.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.305182+0000", 2024-08-17T18:16:16.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3612e-05 2024-08-17T18:16:16.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:16:16.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.316490+0000", 2024-08-17T18:16:16.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011308227000000001 2024-08-17T18:16:16.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:16:16.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:16:16.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:16:16.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:16.316567+0000", 2024-08-17T18:16:16.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6761000000000006e-05 2024-08-17T18:16:16.579 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.579 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:16:16.579 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.579 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:16:16.579 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:16:16.579 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:16:16.608 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:16.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:16.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:16.710 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:16.738 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:16.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:16.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:16.840 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:16.864 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:16.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:16.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:16.965 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:16.989 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:16.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:16.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:17.091 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:17.122 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:17.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:17.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:17.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:17.255 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:17.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:17.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:17.357 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:17.377 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-17T18:16:17.387 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:17.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:17.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:17.489 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:17.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:17.520 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:17.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:17.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:17.655 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:17.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:17.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:17.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:17 smithi136 ceph-mon[109861]: pgmap v3108: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.5 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 249 op/s; 54701/319593 objects degraded (17.116%); 32280/319593 objects misplaced (10.100%); 5.7 MiB/s, 90 objects/s recovering 2024-08-17T18:16:17.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:17 smithi136 ceph-mon[109861]: 1.0 scrub starts 2024-08-17T18:16:17.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:16:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:16:17] "GET /metrics HTTP/1.1" 200 36734 "" "Prometheus/2.51.0" 2024-08-17T18:16:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:17 smithi136 ceph-mon[115078]: pgmap v3108: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.5 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 249 op/s; 54701/319593 objects degraded (17.116%); 32280/319593 objects misplaced (10.100%); 5.7 MiB/s, 90 objects/s recovering 2024-08-17T18:16:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:17 smithi136 ceph-mon[115078]: 1.0 scrub starts 2024-08-17T18:16:17.758 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:17 smithi165 ceph-mon[105517]: pgmap v3108: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.5 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 249 op/s; 54701/319593 objects degraded (17.116%); 32280/319593 objects misplaced (10.100%); 5.7 MiB/s, 90 objects/s recovering 2024-08-17T18:16:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:17 smithi165 ceph-mon[105517]: 1.0 scrub starts 2024-08-17T18:16:17.785 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:17.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:17.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:17.886 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:17.909 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:17.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:17.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:17.959 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:16:17.959 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:16:17.959 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:16:17.959 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:16:17.959 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:16:17.960 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:16:17.960 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:16:17.960 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:16:17.960 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:16:17.960 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:16:17.960 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:16:17.960 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:16:17.960 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:16:17.960 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:16:17.961 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:16:17.961 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:16:18.011 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:18.038 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:18.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:18.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:18.139 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:18.162 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:18.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:18.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:18.264 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:18.288 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:18.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:18.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:18.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:18.419 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:18.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:18.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:18.521 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:18.545 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:18.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:18.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:18.647 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:18.681 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:18.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:18.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:18.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:18 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1272048109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:16:18.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:18 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1272048109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:16:18.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:18 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1272048109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:16:18.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:18.812 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:18.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:18.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:18.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:18.938 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:18.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:18.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:19.040 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:19.063 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:19.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:19.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:19.165 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:19.187 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:19.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:19.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:19.289 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:19.327 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:19.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:19.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:19.429 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:19.451 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:19.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:19.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:19.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:19.579 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:19.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:19.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:19.681 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:19.707 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:19.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:19.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:19 smithi165 ceph-mon[105517]: pgmap v3109: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 1 active+remapped, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.4 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 310 op/s; 54520/314901 objects degraded (17.313%); 31351/314901 objects misplaced (9.956%); 8.5 MiB/s, 135 objects/s recovering 2024-08-17T18:16:19.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:19 smithi165 ceph-mon[105517]: 1.0 scrub starts 2024-08-17T18:16:19.781 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:19 smithi136 ceph-mon[109861]: pgmap v3109: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 1 active+remapped, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.4 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 310 op/s; 54520/314901 objects degraded (17.313%); 31351/314901 objects misplaced (9.956%); 8.5 MiB/s, 135 objects/s recovering 2024-08-17T18:16:19.781 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:19 smithi136 ceph-mon[109861]: 1.0 scrub starts 2024-08-17T18:16:19.781 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:19 smithi136 ceph-mon[115078]: pgmap v3109: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 1 active+remapped, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.4 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 310 op/s; 54520/314901 objects degraded (17.313%); 31351/314901 objects misplaced (9.956%); 8.5 MiB/s, 135 objects/s recovering 2024-08-17T18:16:19.781 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:19 smithi136 ceph-mon[115078]: 1.0 scrub starts 2024-08-17T18:16:19.798 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:16:19.808 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:19.839 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:19.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:19.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:19.941 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:19.965 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:19.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:19.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:20.067 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:20.092 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:20.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:20.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:20.194 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:20.216 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:20.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:20.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:20.318 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:20.339 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:20.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:20.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:20.441 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:20.471 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:20.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:20.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:20.481 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:16:20.572 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:20.600 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:20.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:20.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:20.661 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:16:20.685 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:20 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 54520/314901 objects degraded (17.313%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:16:20.685 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:20 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3504861088' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:16:20.685 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:20 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 54520/314901 objects degraded (17.313%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:16:20.685 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:20 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3504861088' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:16:20.702 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:20.727 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:20.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:20.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:20 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 54520/314901 objects degraded (17.313%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:16:20.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:20 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3504861088' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:16:20.775 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:16:20.829 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:20.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:20.853 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:20.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:20.954 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:20.984 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:20.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:20.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:21.086 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:21.111 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:21.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:21.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:21.213 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:21.238 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:21.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:21.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:21.340 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:21.367 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:21.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:21.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:21.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:21.491 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:21.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:21.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:21.572 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:21 smithi136 ceph-mon[109861]: pgmap v3110: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 1 active+remapped, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.4 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 310 op/s; 54520/314901 objects degraded (17.313%); 31351/314901 objects misplaced (9.956%); 8.5 MiB/s, 135 objects/s recovering 2024-08-17T18:16:21.572 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:21 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:16:21.572 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:21 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3504861088' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:16:21.572 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:21 smithi136 ceph-mon[109861]: osdmap e1645: 8 total, 8 up, 6 in 2024-08-17T18:16:21.572 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:21 smithi136 ceph-mon[109861]: 1.0 scrub starts 2024-08-17T18:16:21.572 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:16:21.572 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:16:21.572 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:21 smithi136 ceph-mon[115078]: pgmap v3110: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 1 active+remapped, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.4 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 310 op/s; 54520/314901 objects degraded (17.313%); 31351/314901 objects misplaced (9.956%); 8.5 MiB/s, 135 objects/s recovering 2024-08-17T18:16:21.572 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:21 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:16:21.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:21 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3504861088' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:16:21.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:21 smithi136 ceph-mon[115078]: osdmap e1645: 8 total, 8 up, 6 in 2024-08-17T18:16:21.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:21 smithi136 ceph-mon[115078]: 1.0 scrub starts 2024-08-17T18:16:21.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:16:21.573 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:16:21.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:21.623 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:21.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:21.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:21.725 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:21.747 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:21.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:21.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:21 smithi165 ceph-mon[105517]: pgmap v3110: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 1 active+remapped, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.4 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 310 op/s; 54520/314901 objects degraded (17.313%); 31351/314901 objects misplaced (9.956%); 8.5 MiB/s, 135 objects/s recovering 2024-08-17T18:16:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:21 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:16:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:21 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3504861088' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:16:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:21 smithi165 ceph-mon[105517]: osdmap e1645: 8 total, 8 up, 6 in 2024-08-17T18:16:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:21 smithi165 ceph-mon[105517]: 1.0 scrub starts 2024-08-17T18:16:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:16:21.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:16:21.851 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:21.873 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:21.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:21.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:21.975 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:22.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:22.001 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:22.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:22.103 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:22.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:22.125 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:22.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:22.227 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:22.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:22.250 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:22.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:22.351 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:22.372 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:22.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:22.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:22.474 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:22.496 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:22.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:22.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:22.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:22.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:22.621 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:22.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:22.722 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:22.740 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:22 smithi136 ceph-mon[109861]: 1.0 scrub starts 2024-08-17T18:16:22.740 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:22 smithi136 ceph-mon[115078]: 1.0 scrub starts 2024-08-17T18:16:22.746 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:22.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:22.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:22.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:22 smithi165 ceph-mon[105517]: 1.0 scrub starts 2024-08-17T18:16:22.848 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:22.876 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:22.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:22.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:22.978 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:23.001 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:23.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:23.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:23.107 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:23.129 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:23.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:23.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:23.230 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:23.254 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:23.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:23.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:23.332 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:16:23.355 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:23.398 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:23.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:23.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:23.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:23.528 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:23.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:23.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:23.629 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:23.633 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:23 smithi136 ceph-mon[109861]: pgmap v3112: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.3 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 346 op/s; 54297/309345 objects degraded (17.552%); 30009/309345 objects misplaced (9.701%); 11 MiB/s, 170 objects/s recovering 2024-08-17T18:16:23.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:23 smithi136 ceph-mon[115078]: pgmap v3112: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.3 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 346 op/s; 54297/309345 objects degraded (17.552%); 30009/309345 objects misplaced (9.701%); 11 MiB/s, 170 objects/s recovering 2024-08-17T18:16:23.651 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:23.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:23.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:23.753 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:23.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:23 smithi165 ceph-mon[105517]: pgmap v3112: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.3 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 346 op/s; 54297/309345 objects degraded (17.552%); 30009/309345 objects misplaced (9.701%); 11 MiB/s, 170 objects/s recovering 2024-08-17T18:16:23.777 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:23.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:23.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:23.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:23.901 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:23.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:23.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:24.003 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:24.027 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:24.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:24.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:24.129 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:24.150 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:24.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:24.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:24.252 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:24.274 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:24.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:24.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:24.376 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:24.403 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:24.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:24.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:24.505 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:24.531 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:24.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:24.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:24.633 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:24.664 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:24.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:24.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:24.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:24 smithi165 ceph-mon[105517]: osdmap e1646: 8 total, 8 up, 6 in 2024-08-17T18:16:24.766 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:24.792 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:24.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:24.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:24.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:24.925 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:24.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:24.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:24.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:24 smithi136 ceph-mon[109861]: osdmap e1646: 8 total, 8 up, 6 in 2024-08-17T18:16:24.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:24 smithi136 ceph-mon[115078]: osdmap e1646: 8 total, 8 up, 6 in 2024-08-17T18:16:25.026 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:25.048 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:25.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:25.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:25.072 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-08-17T18:16:25.150 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:25.180 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:25.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:25.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:25.283 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:25.297 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:16:25.310 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:25.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:25.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:25.313 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:16:25.314 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:16:25.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:25.435 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:25.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:25.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:25.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:25.565 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:25.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:25.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:25.574 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:16:25.666 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:25.679 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:25 smithi136 ceph-mon[109861]: pgmap v3114: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.3 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 426 op/s; 54297/309345 objects degraded (17.552%); 30009/309345 objects misplaced (9.701%); 13 MiB/s, 213 objects/s recovering 2024-08-17T18:16:25.679 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:25 smithi136 ceph-mon[109861]: osdmap e1647: 8 total, 8 up, 6 in 2024-08-17T18:16:25.679 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:25 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 54297/309345 objects degraded (17.552%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:16:25.680 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:25 smithi136 ceph-mon[115078]: pgmap v3114: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.3 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 426 op/s; 54297/309345 objects degraded (17.552%); 30009/309345 objects misplaced (9.701%); 13 MiB/s, 213 objects/s recovering 2024-08-17T18:16:25.680 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:25 smithi136 ceph-mon[115078]: osdmap e1647: 8 total, 8 up, 6 in 2024-08-17T18:16:25.680 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:25 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 54297/309345 objects degraded (17.552%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:16:25.690 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:25.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:25.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:25 smithi165 ceph-mon[105517]: pgmap v3114: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 6.3 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 426 op/s; 54297/309345 objects degraded (17.552%); 30009/309345 objects misplaced (9.701%); 13 MiB/s, 213 objects/s recovering 2024-08-17T18:16:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:25 smithi165 ceph-mon[105517]: osdmap e1647: 8 total, 8 up, 6 in 2024-08-17T18:16:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:25 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 54297/309345 objects degraded (17.552%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:16:25.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:25.814 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:25.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:25.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:25.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:25.945 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:25.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:25.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:26.047 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:26.073 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:26.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:26.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:26.175 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:26.196 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:26.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:26.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:26.206 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:16:26.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-17T18:16:26.298 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:26.327 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:26.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:26.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:26.429 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:26.455 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:26.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:26.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:26.556 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:26.579 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:26.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:26.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:26.681 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:26.705 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:26.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:26.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:26.807 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:26.829 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:26.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:26.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:26.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:26.957 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:26.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:26.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:27.059 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:27.081 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:27.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:27.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:27.183 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:27.204 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:27.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:27.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:27.306 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:27.327 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:27.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:27.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:27.428 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:27.451 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:27.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:27.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:27.552 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:27.576 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:27.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:27.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:27.678 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:27.700 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:27.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:27.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:27.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:27 smithi165 ceph-mon[105517]: pgmap v3116: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 6.3 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 315 op/s; 54297/309222 objects degraded (17.559%); 29927/309222 objects misplaced (9.678%); 10 MiB/s, 162 objects/s recovering 2024-08-17T18:16:27.801 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:27.832 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:27.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:27.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:27.933 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:27.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:27 smithi136 ceph-mon[109861]: pgmap v3116: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 6.3 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 315 op/s; 54297/309222 objects degraded (17.559%); 29927/309222 objects misplaced (9.678%); 10 MiB/s, 162 objects/s recovering 2024-08-17T18:16:27.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:27 smithi136 ceph-mon[115078]: pgmap v3116: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 6.3 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 315 op/s; 54297/309222 objects degraded (17.559%); 29927/309222 objects misplaced (9.678%); 10 MiB/s, 162 objects/s recovering 2024-08-17T18:16:27.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:16:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:16:27] "GET /metrics HTTP/1.1" 200 36730 "" "Prometheus/2.51.0" 2024-08-17T18:16:27.955 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:27.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:27.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:28.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:28.082 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:28.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:28.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:28.184 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:28.215 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:28.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:28.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:28.317 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:28.340 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:28.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:28.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:28.442 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:28.464 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:28.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:28.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:28.566 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:28.588 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:28.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:28.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:28.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:28.715 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:28.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:28.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:28.817 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:28.821 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:28 smithi136 ceph-mon[109861]: 1.0 scrub starts 2024-08-17T18:16:28.821 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:28 smithi136 ceph-mon[115078]: 1.0 scrub starts 2024-08-17T18:16:28.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:28.840 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:28.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:28.941 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:28.962 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:28.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:28.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:29.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:28 smithi165 ceph-mon[105517]: 1.0 scrub starts 2024-08-17T18:16:29.064 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:29.089 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:29.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:29.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:29.191 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:29.214 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:29.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:29.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:29.316 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:29.348 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:29.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:29.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:29.449 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:29.471 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:29.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:29.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:29.573 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:29.594 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:29.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:29.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:29.696 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:29.728 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:29.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:29.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:29.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:29 smithi165 ceph-mon[105517]: pgmap v3117: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 6.2 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 431 op/s; 48337/304599 objects degraded (15.869%); 34844/304599 objects misplaced (11.439%); 11 MiB/s, 200 objects/s recovering 2024-08-17T18:16:29.785 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:29 smithi136 ceph-mon[109861]: pgmap v3117: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 6.2 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 431 op/s; 48337/304599 objects degraded (15.869%); 34844/304599 objects misplaced (11.439%); 11 MiB/s, 200 objects/s recovering 2024-08-17T18:16:29.786 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:29 smithi136 ceph-mon[115078]: pgmap v3117: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 6.2 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 431 op/s; 48337/304599 objects degraded (15.869%); 34844/304599 objects misplaced (11.439%); 11 MiB/s, 200 objects/s recovering 2024-08-17T18:16:29.830 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:29.863 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:29.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:29.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:29.965 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:29.989 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:29.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:29.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:30.091 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:30.108 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:16:30.124 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:30.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:30.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:30.226 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:30.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:30.248 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:30.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:30.349 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:30.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:30.374 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:30.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:30.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:30.498 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:30.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:30.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:30.600 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:30.627 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:30.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:30.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:30.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:30.762 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:30.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:30.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:30.863 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:30.887 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:30.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:30.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:30.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:30 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 48337/304599 objects degraded (15.869%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:16:30.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:30 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 48337/304599 objects degraded (15.869%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:16:30.989 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:31.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:30 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 48337/304599 objects degraded (15.869%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:16:31.011 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:31.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:31.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:31.113 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:31.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:31.143 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:31.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:31.244 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:31.266 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:31.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:31.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:31.368 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:31.390 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:31.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:31.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:31.492 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:31.518 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:31.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:31.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:31.620 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:31.641 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:31.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:31.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:31.743 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:31.766 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:31.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:31.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:31.807 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:31 smithi136 ceph-mon[109861]: pgmap v3118: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 6.2 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 197 op/s; 48337/304599 objects degraded (15.869%); 34844/304599 objects misplaced (11.439%); 3.5 MiB/s, 77 objects/s recovering 2024-08-17T18:16:31.807 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:31 smithi136 ceph-mon[115078]: pgmap v3118: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 6.2 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 197 op/s; 48337/304599 objects degraded (15.869%); 34844/304599 objects misplaced (11.439%); 3.5 MiB/s, 77 objects/s recovering 2024-08-17T18:16:31.868 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:31.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:31.890 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:31.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:31.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:32.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:31 smithi165 ceph-mon[105517]: pgmap v3118: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 6.2 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 197 op/s; 48337/304599 objects degraded (15.869%); 34844/304599 objects misplaced (11.439%); 3.5 MiB/s, 77 objects/s recovering 2024-08-17T18:16:32.018 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:32.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:32.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:32.120 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:32.142 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:32.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:32.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:32.244 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:32.266 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:32.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:32.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:32.368 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:32.392 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:32.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:32.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:32.494 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:32.518 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:32.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:32.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:32.620 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:32.646 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:32.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:32.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:32.748 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:32.760 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:32 smithi136 ceph-mon[109861]: 1.0 scrub starts 2024-08-17T18:16:32.760 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:32 smithi136 ceph-mon[115078]: 1.0 scrub starts 2024-08-17T18:16:32.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:32.775 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:32.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:32.876 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:32.901 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:32.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:32.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:33.003 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:33.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:32 smithi165 ceph-mon[105517]: 1.0 scrub starts 2024-08-17T18:16:33.025 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:33.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:33.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:33.126 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:33.148 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:33.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:33.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:33.250 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:33.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:33.282 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:33.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:33.383 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:33.414 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:33.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:33.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:33.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:33.521 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:16:33.545 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:33.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:33.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:33.647 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:33.671 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:33.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:33.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:33 smithi136 ceph-mon[109861]: pgmap v3119: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 6.1 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 362 op/s; 48239/299631 objects degraded (16.099%); 34254/299631 objects misplaced (11.432%); 4.2 MiB/s, 86 objects/s recovering 2024-08-17T18:16:33.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:33 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/978828837' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:16:33.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:33 smithi136 ceph-mon[115078]: pgmap v3119: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 6.1 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 362 op/s; 48239/299631 objects degraded (16.099%); 34254/299631 objects misplaced (11.432%); 4.2 MiB/s, 86 objects/s recovering 2024-08-17T18:16:33.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:33 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/978828837' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:16:33.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:33.798 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:33.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:33.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:33.901 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:33.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:33.923 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:33.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:33 smithi165 ceph-mon[105517]: pgmap v3119: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 6.1 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 362 op/s; 48239/299631 objects degraded (16.099%); 34254/299631 objects misplaced (11.432%); 4.2 MiB/s, 86 objects/s recovering 2024-08-17T18:16:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:33 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/978828837' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:16:34.025 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:34.047 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:34.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:34.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:34.149 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:34.180 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:34.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:34.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:34.282 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:34.305 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:34.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:34.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:34.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:34.429 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:34.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:34.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:34.531 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:34.556 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:16:34.557 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:16:34.562 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:34.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:34.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:34.576 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:16:34.576 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:16:34.593 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:16:34.594 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:16:34.594 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:34 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:16:34.594 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:34 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/978828837' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:16:34.594 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:34 smithi136 ceph-mon[109861]: osdmap e1648: 8 total, 8 up, 6 in 2024-08-17T18:16:34.594 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:34 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:16:34.594 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:34 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/978828837' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:16:34.595 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:34 smithi136 ceph-mon[115078]: osdmap e1648: 8 total, 8 up, 6 in 2024-08-17T18:16:34.610 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:16:34.611 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:16:34.628 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:16:34.629 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:16:34.649 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:16:34.649 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:16:34.664 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:34.667 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:16:34.667 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:16:34.687 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:34.687 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:16:34.688 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:16:34.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:34.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:34.789 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:34.818 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:34.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:34.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:34.920 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:34.941 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:34.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:34.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:35.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:34 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:16:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:34 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/978828837' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:16:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:34 smithi165 ceph-mon[105517]: osdmap e1648: 8 total, 8 up, 6 in 2024-08-17T18:16:35.043 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:35.068 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:35.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:35.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:35.166 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-08-17T18:16:35.170 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:35.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:35.231 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:35.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:35.332 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:35.354 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:35.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:35.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:35.456 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:35.482 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:35.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:35.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:35.584 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:35.605 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:35.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:35.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:35.707 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:35.730 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:35.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:35.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:35.820 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:35 smithi136 ceph-mon[109861]: pgmap v3121: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 6.1 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 325 op/s; 48239/299631 objects degraded (16.099%); 34254/299631 objects misplaced (11.432%); 3.7 MiB/s, 77 objects/s recovering 2024-08-17T18:16:35.820 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:35 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2584076908' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:16:35.821 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:35 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 48239/299631 objects degraded (16.099%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:16:35.821 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:35 smithi136 ceph-mon[115078]: pgmap v3121: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 6.1 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 325 op/s; 48239/299631 objects degraded (16.099%); 34254/299631 objects misplaced (11.432%); 3.7 MiB/s, 77 objects/s recovering 2024-08-17T18:16:35.821 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:35 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2584076908' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:16:35.821 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:35 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 48239/299631 objects degraded (16.099%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:16:35.834 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:35.856 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:35.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:35.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:35.958 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:35.981 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:35.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:35.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:36.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:35 smithi165 ceph-mon[105517]: pgmap v3121: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 6.1 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 325 op/s; 48239/299631 objects degraded (16.099%); 34254/299631 objects misplaced (11.432%); 3.7 MiB/s, 77 objects/s recovering 2024-08-17T18:16:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:35 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2584076908' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:16:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:35 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 48239/299631 objects degraded (16.099%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:16:36.083 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:36.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:36.106 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:36.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:36.207 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:36.235 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:36.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:36.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:36.337 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:36.364 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:36.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:36.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:36.466 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:36.493 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:36.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:36.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:36.595 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:36.616 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:36.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:36.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:36.718 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:36.740 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:36.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:36.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:36.842 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:36.867 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:36.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:36.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:36.969 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:36.995 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:36.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:36.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:37.054 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:16:37.054 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T18:16:37.054 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-17T18:16:37.054 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-17T18:16:37.054 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:16:37.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:37.127 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:37.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:37.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:37.229 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:37.260 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:37.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:37.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:37.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:37.383 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:37.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:37.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:37.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:16:37.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:16:37.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:37 smithi136 ceph-mon[115078]: pgmap v3122: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 6.1 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 323 op/s; 48239/299511 objects degraded (16.106%); 34174/299511 objects misplaced (11.410%); 3.7 MiB/s, 76 objects/s recovering 2024-08-17T18:16:37.385 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:16:37.385 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:16:37.385 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:37 smithi136 ceph-mon[109861]: pgmap v3122: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 6.1 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 323 op/s; 48239/299511 objects degraded (16.106%); 34174/299511 objects misplaced (11.410%); 3.7 MiB/s, 76 objects/s recovering 2024-08-17T18:16:37.484 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:37.508 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:37.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:37.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:37.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:16:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:16:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:37 smithi165 ceph-mon[105517]: pgmap v3122: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 6.1 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 323 op/s; 48239/299511 objects degraded (16.106%); 34174/299511 objects misplaced (11.410%); 3.7 MiB/s, 76 objects/s recovering 2024-08-17T18:16:37.610 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:37.633 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:37.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:37.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:37.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:16:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:16:37] "GET /metrics HTTP/1.1" 200 36727 "" "Prometheus/2.51.0" 2024-08-17T18:16:37.736 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:37.762 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:37.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:37.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:37.841 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:16:37.844 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:16:37.864 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-17T18:16:37.864 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:37.892 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:37.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:37.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:37.994 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:38.019 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:38.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:38.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:38.121 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:38.144 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:38.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:38.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:38.246 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:38.267 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:38.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:38.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:38.369 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:38.392 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:38.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:38.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:38.493 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:38.513 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:16:38.515 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:38.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:38.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:38.606 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:16:38.610 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:16:38.617 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:38.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:38.644 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:38.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:38.745 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:38.774 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:38.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:38.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:38.876 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:38.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:38.906 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:38.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:39.007 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:39.035 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:39.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:39.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:39.137 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:39.158 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:39.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:39.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:39.260 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:39.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:39.282 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:39.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:39.383 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:39.404 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:39.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:39.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:39.506 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:39.528 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:39.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:39.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:39.630 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:39.651 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:39.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:39.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:39.683 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:39 smithi136 ceph-mon[109861]: pgmap v3123: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 6.0 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 313 op/s; 48130/295203 objects degraded (16.304%); 33754/295203 objects misplaced (11.434%); 1.9 MiB/s, 31 objects/s recovering 2024-08-17T18:16:39.683 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:39 smithi136 ceph-mon[115078]: pgmap v3123: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 6.0 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 313 op/s; 48130/295203 objects degraded (16.304%); 33754/295203 objects misplaced (11.434%); 1.9 MiB/s, 31 objects/s recovering 2024-08-17T18:16:39.753 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:39.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:39 smithi165 ceph-mon[105517]: pgmap v3123: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 6.0 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 313 op/s; 48130/295203 objects degraded (16.304%); 33754/295203 objects misplaced (11.434%); 1.9 MiB/s, 31 objects/s recovering 2024-08-17T18:16:39.774 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:39.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:39.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:39.876 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:39.898 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:39.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:39.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:40.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:40.026 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:40.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:40.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:40.128 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:40.158 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:40.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:40.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:40.260 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:40.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:40.282 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:40.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:40.383 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:40.404 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:40.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:40.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:40.506 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:40.537 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:40.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:40.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:40.639 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:40.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:40 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 48130/295203 objects degraded (16.304%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:16:40.666 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:40.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:40.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:40.768 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:40.792 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:40.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:40.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:40.831 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:40 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 48130/295203 objects degraded (16.304%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:16:40.831 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:40 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 48130/295203 objects degraded (16.304%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:16:40.894 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:40.916 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:40.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:40.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:41.018 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:41.040 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:41.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:41.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:41.142 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:41.165 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:41.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:41.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:41.267 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:41.298 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:41.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:41.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:41.400 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:41.422 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:41.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:41.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:41.524 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:41.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:41.547 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:41.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:41.649 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:41.671 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:41.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:41.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:41.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:41 smithi136 ceph-mon[109861]: pgmap v3124: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 6.0 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 313 op/s; 48130/295203 objects degraded (16.304%); 33754/295203 objects misplaced (11.434%); 1.9 MiB/s, 31 objects/s recovering 2024-08-17T18:16:41.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:41 smithi136 ceph-mon[115078]: pgmap v3124: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 6.0 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 313 op/s; 48130/295203 objects degraded (16.304%); 33754/295203 objects misplaced (11.434%); 1.9 MiB/s, 31 objects/s recovering 2024-08-17T18:16:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:41 smithi165 ceph-mon[105517]: pgmap v3124: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 6.0 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 313 op/s; 48130/295203 objects degraded (16.304%); 33754/295203 objects misplaced (11.434%); 1.9 MiB/s, 31 objects/s recovering 2024-08-17T18:16:41.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:41.795 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:41.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:41.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:41.897 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:41.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:41.924 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:41.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:42.025 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:42.047 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:42.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:42.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:42.149 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:42.176 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:42.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:42.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:42.278 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:42.300 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:42.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:42.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:42.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:42.425 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:42.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:42.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:42.527 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:42.550 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:42.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:42.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:42.652 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:42.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:42.675 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:42.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:42.776 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:42.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:42.799 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:42.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:42.900 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:42.923 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:42.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:42.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:43.025 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:43.058 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:43.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:43.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:43.160 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:43.190 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:43.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:43.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:43.292 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:43.315 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:43.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:43.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:43.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:43.443 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:43.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:43.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:43.499 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:16:43.545 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:43.566 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:43.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:43.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:43.668 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:43.695 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:43.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:43.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:43 smithi165 ceph-mon[105517]: pgmap v3125: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.9 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 292 op/s; 48047/290856 objects degraded (16.519%); 33209/290856 objects misplaced (11.418%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T18:16:43.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:43 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3721445790' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:16:43.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:43.823 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:43.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:43.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:43.842 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:43 smithi136 ceph-mon[109861]: pgmap v3125: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.9 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 292 op/s; 48047/290856 objects degraded (16.519%); 33209/290856 objects misplaced (11.418%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T18:16:43.842 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:43 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3721445790' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:16:43.843 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:43 smithi136 ceph-mon[115078]: pgmap v3125: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.9 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 292 op/s; 48047/290856 objects degraded (16.519%); 33209/290856 objects misplaced (11.418%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T18:16:43.843 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:43 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3721445790' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:16:43.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:43.952 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:43.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:43.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:44.054 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:44.080 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:44.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:44.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:44.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:44.202 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:44.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:44.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:44.304 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:44.325 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:44.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:44.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:44.427 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:44.450 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:44.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:44.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:44.552 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:44.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:44.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:44.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:44.675 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:44.698 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:44.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:44.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:44.800 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:44.822 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:44.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:44.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:44.924 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:44.945 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:44.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:44.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:44.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:44 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:16:44.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:44 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3721445790' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:16:44.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:44 smithi136 ceph-mon[109861]: osdmap e1649: 8 total, 8 up, 6 in 2024-08-17T18:16:44.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:44 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:16:44.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:44 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3721445790' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:16:44.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:44 smithi136 ceph-mon[115078]: osdmap e1649: 8 total, 8 up, 6 in 2024-08-17T18:16:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:44 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:16:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:44 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3721445790' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:16:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:44 smithi165 ceph-mon[105517]: osdmap e1649: 8 total, 8 up, 6 in 2024-08-17T18:16:45.048 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:45.069 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:45.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:45.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:45.171 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:45.193 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:45.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:45.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:45.247 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:16:45.247 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-17T18:16:45.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-17T18:16:45.247 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:16:45.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.248 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:178744 56.e 56:7d5ba0cd:::benchmark_data_smithi136_356456_object52040:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:16:45.248 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:03.744439+0000", 2024-08-17T18:16:45.248 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 281.50060918000003, 2024-08-17T18:16:45.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0679633769999999, 2024-08-17T18:16:45.248 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:16:45.248 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:16:45.248 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:45.248 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:16:45.249 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:16:45.249 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:16:45.249 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 178744 2024-08-17T18:16:45.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.249 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:16:45.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:16:45.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.744439+0000", 2024-08-17T18:16:45.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:16:45.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.744439+0000", 2024-08-17T18:16:45.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:16:45.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.744441+0000", 2024-08-17T18:16:45.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6139999999999999e-06 2024-08-17T18:16:45.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:16:45.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.744445+0000", 2024-08-17T18:16:45.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5399999999999997e-06 2024-08-17T18:16:45.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:16:45.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.744457+0000", 2024-08-17T18:16:45.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1501999999999999e-05 2024-08-17T18:16:45.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:16:45.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.744464+0000", 2024-08-17T18:16:45.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1029999999999996e-06 2024-08-17T18:16:45.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.744532+0000", 2024-08-17T18:16:45.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8479000000000006e-05 2024-08-17T18:16:45.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:16:45.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.744543+0000", 2024-08-17T18:16:45.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0903999999999999e-05 2024-08-17T18:16:45.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:04.638101+0000", 2024-08-17T18:16:45.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.89355757099999999 2024-08-17T18:16:45.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:16:45.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:04.638171+0000", 2024-08-17T18:16:45.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0232000000000001e-05 2024-08-17T18:16:45.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.767792+0000", 2024-08-17T18:16:45.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1296210959999999 2024-08-17T18:16:45.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:16:45.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.770002+0000", 2024-08-17T18:16:45.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0022101109999999998 2024-08-17T18:16:45.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,5,6", 2024-08-17T18:16:45.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.770135+0000", 2024-08-17T18:16:45.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013292800000000001 2024-08-17T18:16:45.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:16:45.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.773517+0000", 2024-08-17T18:16:45.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033824060000000001 2024-08-17T18:16:45.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.773550+0000", 2024-08-17T18:16:45.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2356e-05 2024-08-17T18:16:45.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.774189+0000", 2024-08-17T18:16:45.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00063878000000000003 2024-08-17T18:16:45.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.812282+0000", 2024-08-17T18:16:45.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038093285999999997 2024-08-17T18:16:45.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:16:45.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.812325+0000", 2024-08-17T18:16:45.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3479999999999997e-05 2024-08-17T18:16:45.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:16:45.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.812402+0000", 2024-08-17T18:16:45.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6989000000000004e-05 2024-08-17T18:16:45.257 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.257 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:16:45.257 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.257 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:178748 56.4 56:2df471d4:::benchmark_data_smithi136_356456_object52044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:16:45.257 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:03.750998+0000", 2024-08-17T18:16:45.258 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 281.494050184, 2024-08-17T18:16:45.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1112687879999998, 2024-08-17T18:16:45.258 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:16:45.258 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:16:45.258 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:45.258 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:16:45.258 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:16:45.258 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:16:45.258 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 178748 2024-08-17T18:16:45.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.258 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:16:45.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:16:45.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.750998+0000", 2024-08-17T18:16:45.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:16:45.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.750998+0000", 2024-08-17T18:16:45.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:16:45.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.751000+0000", 2024-08-17T18:16:45.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9850000000000001e-06 2024-08-17T18:16:45.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:16:45.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.751007+0000", 2024-08-17T18:16:45.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5240000000000003e-06 2024-08-17T18:16:45.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:16:45.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.751029+0000", 2024-08-17T18:16:45.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1356999999999999e-05 2024-08-17T18:16:45.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:16:45.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.751038+0000", 2024-08-17T18:16:45.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1589999999999999e-06 2024-08-17T18:16:45.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.751199+0000", 2024-08-17T18:16:45.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000160951 2024-08-17T18:16:45.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:16:45.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.751212+0000", 2024-08-17T18:16:45.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2661e-05 2024-08-17T18:16:45.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:04.631969+0000", 2024-08-17T18:16:45.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.88075765500000003 2024-08-17T18:16:45.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:16:45.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:04.631981+0000", 2024-08-17T18:16:45.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1421e-05 2024-08-17T18:16:45.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.789391+0000", 2024-08-17T18:16:45.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1574099740000001 2024-08-17T18:16:45.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:16:45.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.791331+0000", 2024-08-17T18:16:45.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001940329 2024-08-17T18:16:45.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,4,5", 2024-08-17T18:16:45.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.791416+0000", 2024-08-17T18:16:45.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5017000000000003e-05 2024-08-17T18:16:45.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:16:45.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.797561+0000", 2024-08-17T18:16:45.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00614534 2024-08-17T18:16:45.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.797955+0000", 2024-08-17T18:16:45.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00039357900000000002 2024-08-17T18:16:45.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.818425+0000", 2024-08-17T18:16:45.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020469696999999999 2024-08-17T18:16:45.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.862070+0000", 2024-08-17T18:16:45.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.043645617999999997 2024-08-17T18:16:45.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:16:45.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.862136+0000", 2024-08-17T18:16:45.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6055000000000004e-05 2024-08-17T18:16:45.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:16:45.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.862267+0000", 2024-08-17T18:16:45.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013046600000000001 2024-08-17T18:16:45.266 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.266 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:16:45.266 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.266 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:178750 56.4 56:2d326a25:::benchmark_data_smithi136_356456_object52046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:16:45.266 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:03.752916+0000", 2024-08-17T18:16:45.266 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 281.49213167599999, 2024-08-17T18:16:45.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.10957587, 2024-08-17T18:16:45.266 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:16:45.266 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:16:45.266 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:45.267 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:16:45.267 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:16:45.267 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:16:45.267 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 178750 2024-08-17T18:16:45.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.267 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:16:45.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:16:45.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.752916+0000", 2024-08-17T18:16:45.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:16:45.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.752916+0000", 2024-08-17T18:16:45.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:16:45.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.752918+0000", 2024-08-17T18:16:45.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.945e-06 2024-08-17T18:16:45.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:16:45.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.752923+0000", 2024-08-17T18:16:45.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8570000000000003e-06 2024-08-17T18:16:45.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:16:45.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.752942+0000", 2024-08-17T18:16:45.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8831e-05 2024-08-17T18:16:45.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:16:45.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.752950+0000", 2024-08-17T18:16:45.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2600000000000005e-06 2024-08-17T18:16:45.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.753014+0000", 2024-08-17T18:16:45.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4207000000000004e-05 2024-08-17T18:16:45.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:16:45.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.753023+0000", 2024-08-17T18:16:45.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1920000000000005e-06 2024-08-17T18:16:45.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:04.632023+0000", 2024-08-17T18:16:45.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.87900044399999999 2024-08-17T18:16:45.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:16:45.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:04.632031+0000", 2024-08-17T18:16:45.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0800000000000006e-06 2024-08-17T18:16:45.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.794441+0000", 2024-08-17T18:16:45.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1624093689999999 2024-08-17T18:16:45.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:16:45.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.794534+0000", 2024-08-17T18:16:45.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3392000000000003e-05 2024-08-17T18:16:45.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,4,5", 2024-08-17T18:16:45.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.794615+0000", 2024-08-17T18:16:45.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0704000000000005e-05 2024-08-17T18:16:45.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:16:45.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.797750+0000", 2024-08-17T18:16:45.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031352559999999999 2024-08-17T18:16:45.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.798004+0000", 2024-08-17T18:16:45.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025451500000000003 2024-08-17T18:16:45.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.820269+0000", 2024-08-17T18:16:45.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.022264610000000001 2024-08-17T18:16:45.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.862310+0000", 2024-08-17T18:16:45.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042040688 2024-08-17T18:16:45.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:16:45.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.862367+0000", 2024-08-17T18:16:45.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7466999999999997e-05 2024-08-17T18:16:45.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:16:45.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.862492+0000", 2024-08-17T18:16:45.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125053 2024-08-17T18:16:45.274 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.274 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:16:45.274 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.275 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207973 56.11 56:8f4c55ea:::benchmark_data_smithi136_356456_object81269:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1571)", 2024-08-17T18:16:45.275 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:41.639460+0000", 2024-08-17T18:16:45.275 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 243.60558835500001, 2024-08-17T18:16:45.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1260926149999999, 2024-08-17T18:16:45.275 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:16:45.275 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:16:45.275 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:45.275 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:16:45.275 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:16:45.275 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:16:45.275 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207973 2024-08-17T18:16:45.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.275 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:16:45.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:16:45.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639460+0000", 2024-08-17T18:16:45.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:16:45.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639460+0000", 2024-08-17T18:16:45.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:16:45.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639462+0000", 2024-08-17T18:16:45.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8029999999999999e-06 2024-08-17T18:16:45.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:16:45.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639468+0000", 2024-08-17T18:16:45.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4840000000000001e-06 2024-08-17T18:16:45.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:16:45.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639484+0000", 2024-08-17T18:16:45.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5696e-05 2024-08-17T18:16:45.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:16:45.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639493+0000", 2024-08-17T18:16:45.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6809999999999993e-06 2024-08-17T18:16:45.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639869+0000", 2024-08-17T18:16:45.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00037529600000000001 2024-08-17T18:16:45.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-08-17T18:16:45.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639875+0000", 2024-08-17T18:16:45.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7150000000000001e-06 2024-08-17T18:16:45.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641557+0000", 2024-08-17T18:16:45.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001682092 2024-08-17T18:16:45.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:16:45.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641567+0000", 2024-08-17T18:16:45.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9909999999999997e-06 2024-08-17T18:16:45.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.727904+0000", 2024-08-17T18:16:45.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.086336765 2024-08-17T18:16:45.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:16:45.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.728036+0000", 2024-08-17T18:16:45.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013191400000000001 2024-08-17T18:16:45.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5,6", 2024-08-17T18:16:45.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.728178+0000", 2024-08-17T18:16:45.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014163399999999999 2024-08-17T18:16:45.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:16:45.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.731499+0000", 2024-08-17T18:16:45.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033215580000000001 2024-08-17T18:16:45.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.731551+0000", 2024-08-17T18:16:45.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1950000000000002e-05 2024-08-17T18:16:45.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.731613+0000", 2024-08-17T18:16:45.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1699999999999995e-05 2024-08-17T18:16:45.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.755578+0000", 2024-08-17T18:16:45.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023964798999999998 2024-08-17T18:16:45.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.765381+0000", 2024-08-17T18:16:45.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0098029959999999996 2024-08-17T18:16:45.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:16:45.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.765435+0000", 2024-08-17T18:16:45.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4178000000000002e-05 2024-08-17T18:16:45.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:16:45.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.765552+0000", 2024-08-17T18:16:45.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011736299999999999 2024-08-17T18:16:45.287 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.287 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:16:45.288 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.288 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207989 56.11 56:8c8a752e:::benchmark_data_smithi136_356456_object81285:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1571)", 2024-08-17T18:16:45.288 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:41.639834+0000", 2024-08-17T18:16:45.288 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 243.605214225, 2024-08-17T18:16:45.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1259544850000001, 2024-08-17T18:16:45.288 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:16:45.288 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:16:45.288 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:45.288 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:16:45.288 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:16:45.289 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:16:45.289 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207989 2024-08-17T18:16:45.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.289 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:16:45.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:16:45.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639834+0000", 2024-08-17T18:16:45.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:16:45.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639834+0000", 2024-08-17T18:16:45.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:16:45.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639834+0000", 2024-08-17T18:16:45.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5099999999999999e-07 2024-08-17T18:16:45.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:16:45.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639839+0000", 2024-08-17T18:16:45.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1200000000000004e-06 2024-08-17T18:16:45.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:16:45.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639850+0000", 2024-08-17T18:16:45.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1474000000000001e-05 2024-08-17T18:16:45.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:16:45.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639858+0000", 2024-08-17T18:16:45.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4370000000000001e-06 2024-08-17T18:16:45.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639932+0000", 2024-08-17T18:16:45.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4584000000000003e-05 2024-08-17T18:16:45.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:16:45.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639937+0000", 2024-08-17T18:16:45.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2800000000000003e-06 2024-08-17T18:16:45.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641657+0000", 2024-08-17T18:16:45.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001719918 2024-08-17T18:16:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:16:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641665+0000", 2024-08-17T18:16:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.267e-06 2024-08-17T18:16:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.729531+0000", 2024-08-17T18:16:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0878667689999999 2024-08-17T18:16:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:16:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.729652+0000", 2024-08-17T18:16:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012072499999999999 2024-08-17T18:16:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5,6", 2024-08-17T18:16:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.729797+0000", 2024-08-17T18:16:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000145091 2024-08-17T18:16:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:16:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.731628+0000", 2024-08-17T18:16:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001831292 2024-08-17T18:16:45.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.731674+0000", 2024-08-17T18:16:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5991000000000001e-05 2024-08-17T18:16:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.731710+0000", 2024-08-17T18:16:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5945999999999998e-05 2024-08-17T18:16:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.755616+0000", 2024-08-17T18:16:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023905951000000002 2024-08-17T18:16:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.765593+0000", 2024-08-17T18:16:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0099765789999999993 2024-08-17T18:16:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:16:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.765646+0000", 2024-08-17T18:16:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3556e-05 2024-08-17T18:16:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:16:45.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.765788+0000", 2024-08-17T18:16:45.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014185399999999999 2024-08-17T18:16:45.297 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.297 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:16:45.297 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.297 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:208008 56.11 56:8e595f6d:::benchmark_data_smithi136_356456_object81304:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1571)", 2024-08-17T18:16:45.297 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:41.639870+0000", 2024-08-17T18:16:45.297 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 243.60517799499999, 2024-08-17T18:16:45.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.126137792, 2024-08-17T18:16:45.298 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:16:45.298 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:16:45.298 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:45.298 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:16:45.298 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:16:45.298 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:16:45.298 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 208008 2024-08-17T18:16:45.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.298 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:16:45.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:16:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639870+0000", 2024-08-17T18:16:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:16:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639870+0000", 2024-08-17T18:16:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:16:45.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639871+0000", 2024-08-17T18:16:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4799999999999998e-07 2024-08-17T18:16:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:16:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639873+0000", 2024-08-17T18:16:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7269999999999999e-06 2024-08-17T18:16:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:16:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639881+0000", 2024-08-17T18:16:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3000000000000004e-06 2024-08-17T18:16:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:16:45.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639887+0000", 2024-08-17T18:16:45.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9329999999999998e-06 2024-08-17T18:16:45.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639961+0000", 2024-08-17T18:16:45.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4760999999999998e-05 2024-08-17T18:16:45.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:16:45.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639965+0000", 2024-08-17T18:16:45.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1409999999999998e-06 2024-08-17T18:16:45.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641697+0000", 2024-08-17T18:16:45.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017317299999999999 2024-08-17T18:16:45.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:16:45.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641704+0000", 2024-08-17T18:16:45.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5089999999999999e-06 2024-08-17T18:16:45.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.730558+0000", 2024-08-17T18:16:45.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0888542299999999 2024-08-17T18:16:45.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:16:45.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.730657+0000", 2024-08-17T18:16:45.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8605999999999997e-05 2024-08-17T18:16:45.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5,6", 2024-08-17T18:16:45.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.730746+0000", 2024-08-17T18:16:45.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9424000000000003e-05 2024-08-17T18:16:45.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:16:45.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.731720+0000", 2024-08-17T18:16:45.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00097365500000000003 2024-08-17T18:16:45.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.731751+0000", 2024-08-17T18:16:45.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1131999999999999e-05 2024-08-17T18:16:45.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.731785+0000", 2024-08-17T18:16:45.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4709000000000003e-05 2024-08-17T18:16:45.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.757960+0000", 2024-08-17T18:16:45.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.026174922999999999 2024-08-17T18:16:45.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.765828+0000", 2024-08-17T18:16:45.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0078676200000000005 2024-08-17T18:16:45.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:16:45.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.765885+0000", 2024-08-17T18:16:45.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6780000000000002e-05 2024-08-17T18:16:45.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:16:45.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.766008+0000", 2024-08-17T18:16:45.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012306400000000001 2024-08-17T18:16:45.306 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.306 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:16:45.306 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.306 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:208039 56.11 56:8c1c5a39:::benchmark_data_smithi136_356456_object81335:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1571)", 2024-08-17T18:16:45.307 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:41.639895+0000", 2024-08-17T18:16:45.307 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 243.6051531, 2024-08-17T18:16:45.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.136357155, 2024-08-17T18:16:45.307 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:16:45.307 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:16:45.307 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:45.307 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:16:45.307 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:16:45.307 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:16:45.307 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 208039 2024-08-17T18:16:45.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.307 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:16:45.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:16:45.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639895+0000", 2024-08-17T18:16:45.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:16:45.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639895+0000", 2024-08-17T18:16:45.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:16:45.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639895+0000", 2024-08-17T18:16:45.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6400000000000003e-07 2024-08-17T18:16:45.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:16:45.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639898+0000", 2024-08-17T18:16:45.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.137e-06 2024-08-17T18:16:45.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:16:45.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639904+0000", 2024-08-17T18:16:45.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5930000000000002e-06 2024-08-17T18:16:45.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:16:45.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639910+0000", 2024-08-17T18:16:45.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8150000000000002e-06 2024-08-17T18:16:45.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639989+0000", 2024-08-17T18:16:45.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8971e-05 2024-08-17T18:16:45.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:16:45.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639993+0000", 2024-08-17T18:16:45.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4440000000000001e-06 2024-08-17T18:16:45.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641726+0000", 2024-08-17T18:16:45.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017329439999999999 2024-08-17T18:16:45.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:16:45.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641733+0000", 2024-08-17T18:16:45.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7059999999999998e-06 2024-08-17T18:16:45.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.731811+0000", 2024-08-17T18:16:45.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0900782069999999 2024-08-17T18:16:45.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:16:45.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.731884+0000", 2024-08-17T18:16:45.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2576000000000004e-05 2024-08-17T18:16:45.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5,6", 2024-08-17T18:16:45.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.731955+0000", 2024-08-17T18:16:45.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0858000000000005e-05 2024-08-17T18:16:45.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.734312+0000", 2024-08-17T18:16:45.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023571429999999999 2024-08-17T18:16:45.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:16:45.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.734326+0000", 2024-08-17T18:16:45.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4623999999999999e-05 2024-08-17T18:16:45.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.734372+0000", 2024-08-17T18:16:45.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5782000000000003e-05 2024-08-17T18:16:45.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.757997+0000", 2024-08-17T18:16:45.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023624303999999999 2024-08-17T18:16:45.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.776038+0000", 2024-08-17T18:16:45.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018041855999999998 2024-08-17T18:16:45.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:16:45.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.776093+0000", 2024-08-17T18:16:45.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4972999999999997e-05 2024-08-17T18:16:45.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:16:45.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.776252+0000", 2024-08-17T18:16:45.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000158758 2024-08-17T18:16:45.315 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.316 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:16:45.316 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.316 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:208114 56.11 56:89afd164:::benchmark_data_smithi136_356456_object81410:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1571)", 2024-08-17T18:16:45.316 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:41.639931+0000", 2024-08-17T18:16:45.316 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 243.605116638, 2024-08-17T18:16:45.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.136538536, 2024-08-17T18:16:45.316 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:16:45.316 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:16:45.316 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:45.316 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:16:45.316 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:16:45.317 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:16:45.317 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 208114 2024-08-17T18:16:45.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.317 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:16:45.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:16:45.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639931+0000", 2024-08-17T18:16:45.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:16:45.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639931+0000", 2024-08-17T18:16:45.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:16:45.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639932+0000", 2024-08-17T18:16:45.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2799999999999995e-07 2024-08-17T18:16:45.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:16:45.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639937+0000", 2024-08-17T18:16:45.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6129999999999997e-06 2024-08-17T18:16:45.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:16:45.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639949+0000", 2024-08-17T18:16:45.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2335e-05 2024-08-17T18:16:45.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:16:45.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639957+0000", 2024-08-17T18:16:45.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0460000000000006e-06 2024-08-17T18:16:45.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.640018+0000", 2024-08-17T18:16:45.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0485000000000001e-05 2024-08-17T18:16:45.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:16:45.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.640023+0000", 2024-08-17T18:16:45.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9609999999999996e-06 2024-08-17T18:16:45.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641788+0000", 2024-08-17T18:16:45.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017650669999999999 2024-08-17T18:16:45.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:16:45.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641794+0000", 2024-08-17T18:16:45.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5050000000000004e-06 2024-08-17T18:16:45.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.732597+0000", 2024-08-17T18:16:45.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0908024000000001 2024-08-17T18:16:45.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:16:45.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.732667+0000", 2024-08-17T18:16:45.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0400000000000004e-05 2024-08-17T18:16:45.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5,6", 2024-08-17T18:16:45.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.732753+0000", 2024-08-17T18:16:45.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6296000000000002e-05 2024-08-17T18:16:45.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.734433+0000", 2024-08-17T18:16:45.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016800809999999999 2024-08-17T18:16:45.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:16:45.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.734442+0000", 2024-08-17T18:16:45.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9039999999999998e-06 2024-08-17T18:16:45.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.734481+0000", 2024-08-17T18:16:45.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8492999999999999e-05 2024-08-17T18:16:45.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.763231+0000", 2024-08-17T18:16:45.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.028750047000000001 2024-08-17T18:16:45.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.776300+0000", 2024-08-17T18:16:45.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.01306944 2024-08-17T18:16:45.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:16:45.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.776353+0000", 2024-08-17T18:16:45.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2197000000000002e-05 2024-08-17T18:16:45.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:16:45.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.776470+0000", 2024-08-17T18:16:45.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117438 2024-08-17T18:16:45.324 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.325 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:16:45.325 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.325 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:208125 56.11 56:892bf98a:::benchmark_data_smithi136_356456_object81421:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1571)", 2024-08-17T18:16:45.325 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:41.639970+0000", 2024-08-17T18:16:45.325 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 243.605078209, 2024-08-17T18:16:45.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1367746009999999, 2024-08-17T18:16:45.325 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:16:45.325 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:16:45.325 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:45.325 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:16:45.325 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:16:45.325 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:16:45.326 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 208125 2024-08-17T18:16:45.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.326 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:16:45.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:16:45.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639970+0000", 2024-08-17T18:16:45.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:16:45.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639970+0000", 2024-08-17T18:16:45.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:16:45.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639970+0000", 2024-08-17T18:16:45.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7800000000000001e-07 2024-08-17T18:16:45.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:16:45.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639973+0000", 2024-08-17T18:16:45.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.531e-06 2024-08-17T18:16:45.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:16:45.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639980+0000", 2024-08-17T18:16:45.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5910000000000001e-06 2024-08-17T18:16:45.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:16:45.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639985+0000", 2024-08-17T18:16:45.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3090000000000003e-06 2024-08-17T18:16:45.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.640047+0000", 2024-08-17T18:16:45.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1711000000000006e-05 2024-08-17T18:16:45.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:16:45.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.640051+0000", 2024-08-17T18:16:45.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5159999999999996e-06 2024-08-17T18:16:45.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641818+0000", 2024-08-17T18:16:45.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00176687 2024-08-17T18:16:45.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:16:45.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641824+0000", 2024-08-17T18:16:45.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2450000000000003e-06 2024-08-17T18:16:45.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.733420+0000", 2024-08-17T18:16:45.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.091596206 2024-08-17T18:16:45.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:16:45.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.733507+0000", 2024-08-17T18:16:45.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6594000000000005e-05 2024-08-17T18:16:45.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5,6", 2024-08-17T18:16:45.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.733604+0000", 2024-08-17T18:16:45.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6539999999999994e-05 2024-08-17T18:16:45.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:16:45.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.735282+0000", 2024-08-17T18:16:45.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001678087 2024-08-17T18:16:45.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.735313+0000", 2024-08-17T18:16:45.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1677000000000002e-05 2024-08-17T18:16:45.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.735366+0000", 2024-08-17T18:16:45.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2442999999999999e-05 2024-08-17T18:16:45.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.763561+0000", 2024-08-17T18:16:45.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.028195726000000001 2024-08-17T18:16:45.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.776535+0000", 2024-08-17T18:16:45.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012973088000000001 2024-08-17T18:16:45.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:16:45.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.776597+0000", 2024-08-17T18:16:45.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2722e-05 2024-08-17T18:16:45.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:16:45.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.776744+0000", 2024-08-17T18:16:45.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000147167 2024-08-17T18:16:45.334 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.334 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:16:45.334 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.334 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:208199 56.11 56:89c8702b:::benchmark_data_smithi136_356456_object81495:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1571)", 2024-08-17T18:16:45.334 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:41.639992+0000", 2024-08-17T18:16:45.334 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 243.60505629799999, 2024-08-17T18:16:45.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1370467849999999, 2024-08-17T18:16:45.334 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:16:45.334 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:16:45.335 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:45.335 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:16:45.335 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:16:45.335 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:16:45.335 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 208199 2024-08-17T18:16:45.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.335 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:16:45.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:16:45.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639992+0000", 2024-08-17T18:16:45.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:16:45.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639992+0000", 2024-08-17T18:16:45.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:16:45.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639992+0000", 2024-08-17T18:16:45.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.72e-07 2024-08-17T18:16:45.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:16:45.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639994+0000", 2024-08-17T18:16:45.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6470000000000001e-06 2024-08-17T18:16:45.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:16:45.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.640000+0000", 2024-08-17T18:16:45.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2219999999999999e-06 2024-08-17T18:16:45.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:16:45.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.640005+0000", 2024-08-17T18:16:45.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9749999999999998e-06 2024-08-17T18:16:45.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.640075+0000", 2024-08-17T18:16:45.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0371999999999996e-05 2024-08-17T18:16:45.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:16:45.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.640080+0000", 2024-08-17T18:16:45.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6419999999999998e-06 2024-08-17T18:16:45.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641864+0000", 2024-08-17T18:16:45.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001784171 2024-08-17T18:16:45.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.339 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:16:45.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641871+0000", 2024-08-17T18:16:45.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7530000000000004e-06 2024-08-17T18:16:45.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.339 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.734508+0000", 2024-08-17T18:16:45.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0926368179999999 2024-08-17T18:16:45.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.339 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:16:45.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.734579+0000", 2024-08-17T18:16:45.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0908000000000006e-05 2024-08-17T18:16:45.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5,6", 2024-08-17T18:16:45.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.734651+0000", 2024-08-17T18:16:45.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2687000000000004e-05 2024-08-17T18:16:45.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.737567+0000", 2024-08-17T18:16:45.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0029158840000000001 2024-08-17T18:16:45.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.341 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.341 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.737606+0000", 2024-08-17T18:16:45.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8507000000000002e-05 2024-08-17T18:16:45.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.341 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:16:45.341 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.737614+0000", 2024-08-17T18:16:45.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0260000000000007e-06 2024-08-17T18:16:45.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.341 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.766052+0000", 2024-08-17T18:16:45.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.028438341999999998 2024-08-17T18:16:45.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.776806+0000", 2024-08-17T18:16:45.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010753953 2024-08-17T18:16:45.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:16:45.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.776867+0000", 2024-08-17T18:16:45.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0511999999999999e-05 2024-08-17T18:16:45.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.343 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:16:45.343 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.777039+0000", 2024-08-17T18:16:45.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017199399999999999 2024-08-17T18:16:45.343 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.343 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:16:45.343 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.343 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:208276 56.11 56:8fd203dd:::benchmark_data_smithi136_356456_object81572:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1571)", 2024-08-17T18:16:45.343 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:41.741074+0000", 2024-08-17T18:16:45.344 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 243.503973817, 2024-08-17T18:16:45.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0362123560000001, 2024-08-17T18:16:45.344 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:16:45.344 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:16:45.344 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:45.344 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:16:45.344 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:16:45.344 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:16:45.344 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 208276 2024-08-17T18:16:45.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.344 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:16:45.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.344 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:16:45.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.741074+0000", 2024-08-17T18:16:45.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:16:45.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.741074+0000", 2024-08-17T18:16:45.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:16:45.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.741075+0000", 2024-08-17T18:16:45.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3699999999999999e-07 2024-08-17T18:16:45.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.346 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:16:45.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.741080+0000", 2024-08-17T18:16:45.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5109999999999997e-06 2024-08-17T18:16:45.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.346 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:16:45.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.741090+0000", 2024-08-17T18:16:45.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.046e-05 2024-08-17T18:16:45.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.346 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:16:45.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.741095+0000", 2024-08-17T18:16:45.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1030000000000001e-06 2024-08-17T18:16:45.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.741147+0000", 2024-08-17T18:16:45.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1709000000000002e-05 2024-08-17T18:16:45.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:16:45.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.741153+0000", 2024-08-17T18:16:45.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5679999999999999e-06 2024-08-17T18:16:45.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.348 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.348 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.735424+0000", 2024-08-17T18:16:45.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.99427149299999995 2024-08-17T18:16:45.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.348 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:16:45.348 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.735530+0000", 2024-08-17T18:16:45.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106152 2024-08-17T18:16:45.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5,6", 2024-08-17T18:16:45.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.735614+0000", 2024-08-17T18:16:45.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4209999999999995e-05 2024-08-17T18:16:45.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:16:45.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.737618+0000", 2024-08-17T18:16:45.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020032930000000002 2024-08-17T18:16:45.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.737674+0000", 2024-08-17T18:16:45.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6208999999999997e-05 2024-08-17T18:16:45.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.737716+0000", 2024-08-17T18:16:45.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2484999999999997e-05 2024-08-17T18:16:45.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.766091+0000", 2024-08-17T18:16:45.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.028374183000000001 2024-08-17T18:16:45.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.777099+0000", 2024-08-17T18:16:45.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011008688000000001 2024-08-17T18:16:45.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:16:45.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.777153+0000", 2024-08-17T18:16:45.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4054000000000001e-05 2024-08-17T18:16:45.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:16:45.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.777287+0000", 2024-08-17T18:16:45.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000133301 2024-08-17T18:16:45.351 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.352 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:16:45.352 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.352 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:208316 56.11 56:8a69a9a8:::benchmark_data_smithi136_356456_object81612:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1571)", 2024-08-17T18:16:45.352 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:41.796638+0000", 2024-08-17T18:16:45.352 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 243.44840999799999, 2024-08-17T18:16:45.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.98089237500000004, 2024-08-17T18:16:45.352 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:16:45.352 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:16:45.352 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:45.352 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:16:45.353 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:16:45.353 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:16:45.353 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 208316 2024-08-17T18:16:45.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.353 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:16:45.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:16:45.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.796638+0000", 2024-08-17T18:16:45.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:16:45.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.796638+0000", 2024-08-17T18:16:45.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.354 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:16:45.354 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.796639+0000", 2024-08-17T18:16:45.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1969999999999999e-06 2024-08-17T18:16:45.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.354 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:16:45.354 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.796644+0000", 2024-08-17T18:16:45.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8210000000000001e-06 2024-08-17T18:16:45.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:16:45.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.796657+0000", 2024-08-17T18:16:45.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2788e-05 2024-08-17T18:16:45.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:16:45.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.796664+0000", 2024-08-17T18:16:45.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4030000000000001e-06 2024-08-17T18:16:45.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.796740+0000", 2024-08-17T18:16:45.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5889999999999993e-05 2024-08-17T18:16:45.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:16:45.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.796750+0000", 2024-08-17T18:16:45.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9480000000000003e-06 2024-08-17T18:16:45.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.736314+0000", 2024-08-17T18:16:45.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93956432700000003 2024-08-17T18:16:45.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:16:45.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.736392+0000", 2024-08-17T18:16:45.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7879000000000004e-05 2024-08-17T18:16:45.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5,6", 2024-08-17T18:16:45.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.736511+0000", 2024-08-17T18:16:45.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118306 2024-08-17T18:16:45.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.737774+0000", 2024-08-17T18:16:45.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012633189999999999 2024-08-17T18:16:45.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:16:45.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.738882+0000", 2024-08-17T18:16:45.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011084280000000001 2024-08-17T18:16:45.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.738938+0000", 2024-08-17T18:16:45.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5615999999999997e-05 2024-08-17T18:16:45.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.766343+0000", 2024-08-17T18:16:45.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.027404896000000002 2024-08-17T18:16:45.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.777348+0000", 2024-08-17T18:16:45.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011005251000000001 2024-08-17T18:16:45.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:16:45.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.777407+0000", 2024-08-17T18:16:45.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8659999999999997e-05 2024-08-17T18:16:45.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:16:45.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.777530+0000", 2024-08-17T18:16:45.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012364599999999999 2024-08-17T18:16:45.360 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.360 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:16:45.360 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.361 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:209638 56.14 56:291d58d8:::benchmark_data_smithi136_356456_object82934:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1572)", 2024-08-17T18:16:45.361 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:43.664215+0000", 2024-08-17T18:16:45.361 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 241.580833383, 2024-08-17T18:16:45.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0141130700000001, 2024-08-17T18:16:45.361 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:16:45.361 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:16:45.361 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:45.361 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:16:45.361 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:16:45.361 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:16:45.361 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 209638 2024-08-17T18:16:45.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.361 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:16:45.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:16:45.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.664215+0000", 2024-08-17T18:16:45.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:16:45.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.664215+0000", 2024-08-17T18:16:45.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:16:45.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.664216+0000", 2024-08-17T18:16:45.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.668e-06 2024-08-17T18:16:45.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:16:45.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.664222+0000", 2024-08-17T18:16:45.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.037e-06 2024-08-17T18:16:45.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:16:45.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.664235+0000", 2024-08-17T18:16:45.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.306e-05 2024-08-17T18:16:45.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:16:45.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.664244+0000", 2024-08-17T18:16:45.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3909999999999998e-06 2024-08-17T18:16:45.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.666941+0000", 2024-08-17T18:16:45.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002697199 2024-08-17T18:16:45.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:16:45.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.666948+0000", 2024-08-17T18:16:45.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9580000000000001e-06 2024-08-17T18:16:45.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.674159+0000", 2024-08-17T18:16:45.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0072107130000001 2024-08-17T18:16:45.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:16:45.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.674288+0000", 2024-08-17T18:16:45.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012949000000000001 2024-08-17T18:16:45.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:16:45.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.674631+0000", 2024-08-17T18:16:45.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034243000000000001 2024-08-17T18:16:45.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:16:45.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.677264+0000", 2024-08-17T18:16:45.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0026335199999999999 2024-08-17T18:16:45.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.678003+0000", 2024-08-17T18:16:45.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00073914399999999996 2024-08-17T18:16:45.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.367 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.678204+0000", 2024-08-17T18:16:45.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020022700000000001 2024-08-17T18:16:45.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:16:45.367 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.678258+0000", 2024-08-17T18:16:45.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4459000000000002e-05 2024-08-17T18:16:45.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:16:45.368 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.678328+0000", 2024-08-17T18:16:45.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9774e-05 2024-08-17T18:16:45.368 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.368 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:16:45.368 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.368 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.368 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:209670 56.14 56:2f03d7bc:::benchmark_data_smithi136_356456_object82966:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1573)", 2024-08-17T18:16:45.368 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:43.689044+0000", 2024-08-17T18:16:45.368 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 241.556004465, 2024-08-17T18:16:45.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.99400135300000003, 2024-08-17T18:16:45.368 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:16:45.368 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:16:45.369 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:45.369 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:16:45.369 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:16:45.369 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:16:45.369 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 209670 2024-08-17T18:16:45.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.369 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:16:45.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:16:45.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.689044+0000", 2024-08-17T18:16:45.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:16:45.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.689044+0000", 2024-08-17T18:16:45.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:16:45.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.689044+0000", 2024-08-17T18:16:45.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5600000000000005e-07 2024-08-17T18:16:45.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:16:45.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.689048+0000", 2024-08-17T18:16:45.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0600000000000001e-06 2024-08-17T18:16:45.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:16:45.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.689058+0000", 2024-08-17T18:16:45.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0009e-05 2024-08-17T18:16:45.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:16:45.371 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:45.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.689064+0000", 2024-08-17T18:16:45.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6200000000000004e-06 2024-08-17T18:16:45.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.689179+0000", 2024-08-17T18:16:45.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011511200000000001 2024-08-17T18:16:45.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:16:45.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.689188+0000", 2024-08-17T18:16:45.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9809999999999992e-06 2024-08-17T18:16:45.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.678366+0000", 2024-08-17T18:16:45.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.989177627 2024-08-17T18:16:45.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:16:45.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.678475+0000", 2024-08-17T18:16:45.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109596 2024-08-17T18:16:45.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:16:45.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.678849+0000", 2024-08-17T18:16:45.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000373605 2024-08-17T18:16:45.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.682806+0000", 2024-08-17T18:16:45.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039571370000000003 2024-08-17T18:16:45.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.682860+0000", 2024-08-17T18:16:45.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4181000000000002e-05 2024-08-17T18:16:45.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:16:45.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.682976+0000", 2024-08-17T18:16:45.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115846 2024-08-17T18:16:45.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:16:45.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.683005+0000", 2024-08-17T18:16:45.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8751e-05 2024-08-17T18:16:45.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:16:45.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.683045+0000", 2024-08-17T18:16:45.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0172000000000001e-05 2024-08-17T18:16:45.377 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.377 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:16:45.377 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.377 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:209734 56.14 56:28776341:::benchmark_data_smithi136_356456_object83030:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1573)", 2024-08-17T18:16:45.377 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:43.751846+0000", 2024-08-17T18:16:45.377 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 241.49320230699999, 2024-08-17T18:16:45.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93361844599999999, 2024-08-17T18:16:45.377 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:16:45.377 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:16:45.377 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:45.378 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:16:45.378 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:16:45.378 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:16:45.378 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 209734 2024-08-17T18:16:45.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.378 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:16:45.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.378 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:16:45.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.751846+0000", 2024-08-17T18:16:45.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.378 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:16:45.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.751846+0000", 2024-08-17T18:16:45.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:16:45.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.751847+0000", 2024-08-17T18:16:45.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4139999999999999e-06 2024-08-17T18:16:45.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:16:45.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.751853+0000", 2024-08-17T18:16:45.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4620000000000004e-06 2024-08-17T18:16:45.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:16:45.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.751865+0000", 2024-08-17T18:16:45.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2364999999999999e-05 2024-08-17T18:16:45.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:16:45.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.751873+0000", 2024-08-17T18:16:45.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.661e-06 2024-08-17T18:16:45.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.751931+0000", 2024-08-17T18:16:45.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7908000000000002e-05 2024-08-17T18:16:45.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:16:45.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.751940+0000", 2024-08-17T18:16:45.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9479999999999997e-06 2024-08-17T18:16:45.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.680863+0000", 2024-08-17T18:16:45.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92892299099999998 2024-08-17T18:16:45.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:16:45.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.680965+0000", 2024-08-17T18:16:45.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102586 2024-08-17T18:16:45.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:16:45.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.681257+0000", 2024-08-17T18:16:45.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029193399999999999 2024-08-17T18:16:45.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.682889+0000", 2024-08-17T18:16:45.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016320869999999999 2024-08-17T18:16:45.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.683068+0000", 2024-08-17T18:16:45.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017931300000000001 2024-08-17T18:16:45.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:16:45.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.685215+0000", 2024-08-17T18:16:45.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021464380000000001 2024-08-17T18:16:45.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:16:45.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.685329+0000", 2024-08-17T18:16:45.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011388 2024-08-17T18:16:45.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:16:45.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.685464+0000", 2024-08-17T18:16:45.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013545899999999999 2024-08-17T18:16:45.384 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.384 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:16:45.384 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.385 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:79433 57.3 57:c0b091a2:::benchmark_data_smithi136_389235_object79432:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1616)", 2024-08-17T18:16:45.385 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:14:48.570059+0000", 2024-08-17T18:16:45.385 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 116.674989369, 2024-08-17T18:16:45.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0321689679999999, 2024-08-17T18:16:45.385 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:16:45.385 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:16:45.385 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:16:45.385 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:16:45.385 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:16:45.385 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:16:45.385 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 79433 2024-08-17T18:16:45.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.386 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:16:45.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:16:45.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.570059+0000", 2024-08-17T18:16:45.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:16:45.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.570059+0000", 2024-08-17T18:16:45.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:16:45.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.570060+0000", 2024-08-17T18:16:45.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5340000000000001e-06 2024-08-17T18:16:45.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:16:45.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.570132+0000", 2024-08-17T18:16:45.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1439000000000005e-05 2024-08-17T18:16:45.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:16:45.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.570149+0000", 2024-08-17T18:16:45.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7110999999999999e-05 2024-08-17T18:16:45.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:16:45.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.570156+0000", 2024-08-17T18:16:45.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1940000000000001e-06 2024-08-17T18:16:45.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.571294+0000", 2024-08-17T18:16:45.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001138132 2024-08-17T18:16:45.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-08-17T18:16:45.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.571299+0000", 2024-08-17T18:16:45.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7829999999999998e-06 2024-08-17T18:16:45.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.729716+0000", 2024-08-17T18:16:45.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15841707899999999 2024-08-17T18:16:45.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:16:45.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.729729+0000", 2024-08-17T18:16:45.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3424000000000001e-05 2024-08-17T18:16:45.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:49.602143+0000", 2024-08-17T18:16:45.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.87241396100000002 2024-08-17T18:16:45.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:16:45.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:49.602228+0000", 2024-08-17T18:16:45.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4311e-05 2024-08-17T18:16:45.390 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.390 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:16:45.391 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.391 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:79437 57.3 57:c2190512:::benchmark_data_smithi136_389235_object79436:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1616)", 2024-08-17T18:16:45.391 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:14:48.570171+0000", 2024-08-17T18:16:45.391 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 116.674877111, 2024-08-17T18:16:45.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.032151153, 2024-08-17T18:16:45.391 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:16:45.391 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:16:45.391 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:16:45.391 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:16:45.391 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:16:45.391 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:16:45.392 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 79437 2024-08-17T18:16:45.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.392 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:16:45.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:16:45.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.570171+0000", 2024-08-17T18:16:45.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:16:45.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.570171+0000", 2024-08-17T18:16:45.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:16:45.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.570172+0000", 2024-08-17T18:16:45.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4799999999999998e-07 2024-08-17T18:16:45.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:16:45.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.570190+0000", 2024-08-17T18:16:45.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8467000000000001e-05 2024-08-17T18:16:45.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:16:45.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.570197+0000", 2024-08-17T18:16:45.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.905e-06 2024-08-17T18:16:45.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:16:45.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.570202+0000", 2024-08-17T18:16:45.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4639999999999997e-06 2024-08-17T18:16:45.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.571336+0000", 2024-08-17T18:16:45.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001133818 2024-08-17T18:16:45.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:16:45.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.571340+0000", 2024-08-17T18:16:45.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1849999999999997e-06 2024-08-17T18:16:45.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.729754+0000", 2024-08-17T18:16:45.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15841395699999999 2024-08-17T18:16:45.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:16:45.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.729762+0000", 2024-08-17T18:16:45.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5190000000000003e-06 2024-08-17T18:16:45.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:49.602262+0000", 2024-08-17T18:16:45.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.87249982000000004 2024-08-17T18:16:45.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:16:45.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:49.602322+0000", 2024-08-17T18:16:45.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0470000000000002e-05 2024-08-17T18:16:45.396 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.396 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:16:45.396 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.397 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:122085 57.2 57:42ce4696:::benchmark_data_smithi136_389235_object6163:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1639)", 2024-08-17T18:16:45.397 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:54.574198+0000", 2024-08-17T18:16:45.397 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.670850360000003, 2024-08-17T18:16:45.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5265542280000002, 2024-08-17T18:16:45.397 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:16:45.397 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:16:45.397 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:45.397 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:16:45.397 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:16:45.397 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:16:45.397 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122085 2024-08-17T18:16:45.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.398 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:16:45.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:16:45.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.574198+0000", 2024-08-17T18:16:45.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:16:45.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.574198+0000", 2024-08-17T18:16:45.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:16:45.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.574199+0000", 2024-08-17T18:16:45.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4300000000000001e-07 2024-08-17T18:16:45.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:16:45.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.574203+0000", 2024-08-17T18:16:45.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4170000000000002e-06 2024-08-17T18:16:45.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:16:45.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.574214+0000", 2024-08-17T18:16:45.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0672999999999999e-05 2024-08-17T18:16:45.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:16:45.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.574219+0000", 2024-08-17T18:16:45.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0529999999999997e-06 2024-08-17T18:16:45.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.574268+0000", 2024-08-17T18:16:45.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8732e-05 2024-08-17T18:16:45.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:16:45.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.574368+0000", 2024-08-17T18:16:45.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010063199999999999 2024-08-17T18:16:45.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:16:45.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.574725+0000", 2024-08-17T18:16:45.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00035646700000000001 2024-08-17T18:16:45.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:16:45.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.576207+0000", 2024-08-17T18:16:45.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014825979999999999 2024-08-17T18:16:45.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.402 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.402 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.577429+0000", 2024-08-17T18:16:45.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001221576 2024-08-17T18:16:45.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.402 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.402 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.099083+0000", 2024-08-17T18:16:45.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5216544700000001 2024-08-17T18:16:45.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.402 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:16:45.403 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.099169+0000", 2024-08-17T18:16:45.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5350000000000001e-05 2024-08-17T18:16:45.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:16:45.403 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.100752+0000", 2024-08-17T18:16:45.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015833170000000001 2024-08-17T18:16:45.403 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.403 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:16:45.403 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.403 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:122093 57.2 57:407dd288:::benchmark_data_smithi136_389235_object6171:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1639)", 2024-08-17T18:16:45.404 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:54.621943+0000", 2024-08-17T18:16:45.404 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.623105027999998, 2024-08-17T18:16:45.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4794903640000001, 2024-08-17T18:16:45.404 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:16:45.404 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:16:45.404 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:45.404 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:16:45.404 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:16:45.404 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:16:45.404 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122093 2024-08-17T18:16:45.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.404 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:16:45.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:16:45.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.621943+0000", 2024-08-17T18:16:45.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:16:45.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.621943+0000", 2024-08-17T18:16:45.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:16:45.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.621944+0000", 2024-08-17T18:16:45.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4300000000000002e-07 2024-08-17T18:16:45.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:16:45.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.621947+0000", 2024-08-17T18:16:45.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9660000000000001e-06 2024-08-17T18:16:45.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:16:45.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.621956+0000", 2024-08-17T18:16:45.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2329999999999995e-06 2024-08-17T18:16:45.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:16:45.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.621963+0000", 2024-08-17T18:16:45.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.968e-06 2024-08-17T18:16:45.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.622012+0000", 2024-08-17T18:16:45.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8982e-05 2024-08-17T18:16:45.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:16:45.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.622107+0000", 2024-08-17T18:16:45.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4905e-05 2024-08-17T18:16:45.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:16:45.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.622409+0000", 2024-08-17T18:16:45.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00030236500000000002 2024-08-17T18:16:45.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:16:45.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.623875+0000", 2024-08-17T18:16:45.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014658869999999999 2024-08-17T18:16:45.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.409 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.409 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.624591+0000", 2024-08-17T18:16:45.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00071544800000000004 2024-08-17T18:16:45.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.409 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.409 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.100795+0000", 2024-08-17T18:16:45.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.476204418 2024-08-17T18:16:45.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.409 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:16:45.409 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.100830+0000", 2024-08-17T18:16:45.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4641999999999998e-05 2024-08-17T18:16:45.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.410 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:16:45.410 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.101433+0000", 2024-08-17T18:16:45.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00060380700000000004 2024-08-17T18:16:45.410 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.410 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:16:45.410 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.410 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:122102 57.2 57:436a03b4:::benchmark_data_smithi136_389235_object6180:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1641)", 2024-08-17T18:16:45.410 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:57.128536+0000", 2024-08-17T18:16:45.410 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 48.116511987999999, 2024-08-17T18:16:45.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3281144779999998, 2024-08-17T18:16:45.411 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:16:45.411 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:16:45.411 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:16:45.411 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:16:45.411 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:16:45.411 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:16:45.411 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122102 2024-08-17T18:16:45.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.411 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:16:45.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.411 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:16:45.411 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.128536+0000", 2024-08-17T18:16:45.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:16:45.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.128536+0000", 2024-08-17T18:16:45.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:16:45.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:16:45.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.128537+0000", 2024-08-17T18:16:45.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.079e-06 2024-08-17T18:16:45.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:16:45.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.128542+0000", 2024-08-17T18:16:45.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5460000000000002e-06 2024-08-17T18:16:45.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:16:45.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.128552+0000", 2024-08-17T18:16:45.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0550999999999999e-05 2024-08-17T18:16:45.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:16:45.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.128557+0000", 2024-08-17T18:16:45.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0000000000000004e-06 2024-08-17T18:16:45.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:16:45.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.128614+0000", 2024-08-17T18:16:45.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6749999999999997e-05 2024-08-17T18:16:45.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:16:45.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.128723+0000", 2024-08-17T18:16:45.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109079 2024-08-17T18:16:45.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:16:45.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.129047+0000", 2024-08-17T18:16:45.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00032373900000000003 2024-08-17T18:16:45.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.415 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:16:45.415 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.130562+0000", 2024-08-17T18:16:45.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001515504 2024-08-17T18:16:45.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.415 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.415 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.131282+0000", 2024-08-17T18:16:45.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000719756 2024-08-17T18:16:45.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.415 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:16:45.416 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:16:00.455193+0000", 2024-08-17T18:16:45.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3239109089999999 2024-08-17T18:16:45.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:16:45.416 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:16:00.455255+0000", 2024-08-17T18:16:45.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2345e-05 2024-08-17T18:16:45.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:16:45.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:16:45.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:16:45.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:16:00.456651+0000", 2024-08-17T18:16:45.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00139522 2024-08-17T18:16:45.417 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.417 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:16:45.417 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.417 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:16:45.417 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:16:45.417 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:16:45.438 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:45.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:45.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:45.540 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:45.562 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:45.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:45.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:45.664 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:45.686 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:45.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:45.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:45.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:45 smithi136 ceph-mon[109861]: pgmap v3127: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.9 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 292 op/s; 48047/290856 objects degraded (16.519%); 33209/290856 objects misplaced (11.418%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T18:16:45.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:45 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 48047/290856 objects degraded (16.519%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:16:45.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:45 smithi136 ceph-mon[115078]: pgmap v3127: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.9 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 292 op/s; 48047/290856 objects degraded (16.519%); 33209/290856 objects misplaced (11.418%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T18:16:45.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:45 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 48047/290856 objects degraded (16.519%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:16:45.716 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:16:45.788 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:45.815 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:45.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:45.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:45.917 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:45.938 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:45.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:45.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:46.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:45 smithi165 ceph-mon[105517]: pgmap v3127: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.9 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 292 op/s; 48047/290856 objects degraded (16.519%); 33209/290856 objects misplaced (11.418%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T18:16:46.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:45 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 48047/290856 objects degraded (16.519%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:16:46.040 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:46.067 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:46.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:46.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:46.169 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:46.197 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:46.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:46.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:46.298 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:46.323 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:46.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:46.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:46.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:46.452 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:46.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:46.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:46.499 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-17T18:16:46.554 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:46.576 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:46.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:46.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:46.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:46.704 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:46.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:46.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:46.806 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:46.832 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:46.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:46.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:46.934 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:46.957 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:46.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:46.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:47.040 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:16:47.059 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:47.085 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:47.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:47.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:47.187 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:47.210 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:47.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:47.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:47.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:47.336 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:47.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:47.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:47.437 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:47.463 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:47.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:47.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:47.493 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:16:47.508 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:16:47.566 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:47.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:47.588 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:47.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:47.689 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:47.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:47 smithi136 ceph-mon[109861]: pgmap v3128: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.9 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 292 op/s; 48047/290724 objects degraded (16.527%); 33121/290724 objects misplaced (11.393%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T18:16:47.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:47 smithi136 ceph-mon[115078]: pgmap v3128: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.9 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 292 op/s; 48047/290724 objects degraded (16.527%); 33121/290724 objects misplaced (11.393%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T18:16:47.697 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:16:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:16:47] "GET /metrics HTTP/1.1" 200 36727 "" "Prometheus/2.51.0" 2024-08-17T18:16:47.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:47.713 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:47.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:47.814 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:47.838 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:47.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:47.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:47.940 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:47.963 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:47.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:47.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:47 smithi165 ceph-mon[105517]: pgmap v3128: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.9 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 292 op/s; 48047/290724 objects degraded (16.527%); 33121/290724 objects misplaced (11.393%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T18:16:48.065 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:48.087 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:48.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:48.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:48.189 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:48.212 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:48.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:48.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:48.230 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:16:48.231 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:16:48.248 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:16:48.248 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:16:48.265 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:16:48.265 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:16:48.284 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:16:48.285 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:16:48.303 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:16:48.303 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:16:48.314 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:48.321 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:16:48.321 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:16:48.340 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:48.341 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:16:48.342 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:16:48.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:48.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:48.357 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:16:48.358 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:16:48.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:48.465 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:48.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:48.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:48.566 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:48.591 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:48.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:48.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:48.694 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:48.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:48 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1953104885' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:16:48.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:48 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1953104885' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:16:48.717 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:48.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:48.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:48.819 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:48.842 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:48.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:48.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:48.864 INFO:tasks.thrashosds.thrasher:in_osds: [2, 4, 6, 0, 5, 7] out_osds: [1, 3] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:16:48.864 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:16:48.865 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-08-17T18:16:48.865 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-08-17T18:16:48.943 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:48.968 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:48.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:48.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:49.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:48 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1953104885' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:16:49.071 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:49.093 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:49.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:49.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:49.195 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:49.221 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:49.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:49.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:49.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:49.358 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:49.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:49.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:49.460 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:49.487 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:49.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:49.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:49.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:49.618 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:49.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:49.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:49 smithi136 ceph-mon[109861]: pgmap v3129: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.8 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 312 op/s; 47947/285834 objects degraded (16.774%); 32610/285834 objects misplaced (11.409%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T18:16:49.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:49 smithi136 ceph-mon[115078]: pgmap v3129: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.8 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 312 op/s; 47947/285834 objects degraded (16.774%); 32610/285834 objects misplaced (11.409%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T18:16:49.720 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:49.743 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:49.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:49.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:49.845 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:49.869 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:49.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:49.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:49.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:50.000 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:50.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:50.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:50.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:49 smithi165 ceph-mon[105517]: pgmap v3129: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.8 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 312 op/s; 47947/285834 objects degraded (16.774%); 32610/285834 objects misplaced (11.409%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T18:16:50.102 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:50.125 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:50.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:50.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:50.226 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:50.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:50.256 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:50.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:50.357 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:50.388 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:50.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:50.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:50.492 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:50.513 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:50.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:50.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:50.615 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:50.634 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:50 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 47947/285834 objects degraded (16.774%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:16:50.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:50 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 47947/285834 objects degraded (16.774%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:16:50.641 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:50.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:50.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:50.743 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:50.765 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:50.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:50.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:50.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:50.889 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:50.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:50.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:50.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:51.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:50 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 47947/285834 objects degraded (16.774%), 16 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:16:51.019 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:51.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:51.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:51.120 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:51.146 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:51.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:51.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:51.157 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:16:51.248 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:51.270 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:51.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:51.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:51.372 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:51.394 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:51.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:51.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:51.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:51.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:51.531 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:51.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:51.632 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:51.657 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:51.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:51.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:51 smithi136 ceph-mon[109861]: pgmap v3130: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.8 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 312 op/s; 47947/285834 objects degraded (16.774%); 32610/285834 objects misplaced (11.409%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T18:16:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:16:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:51 smithi136 ceph-mon[115078]: pgmap v3130: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.8 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 312 op/s; 47947/285834 objects degraded (16.774%); 32610/285834 objects misplaced (11.409%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T18:16:51.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:16:51.762 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:51.784 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:51.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:51.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:51.886 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:51.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:51.910 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:51.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:51.923 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:51 smithi165 ceph-mon[105517]: pgmap v3130: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.8 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 312 op/s; 47947/285834 objects degraded (16.774%); 32610/285834 objects misplaced (11.409%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T18:16:51.923 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:16:52.010 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:52.028 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:16:52.032 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:52.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:52.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:52.134 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:52.156 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:52.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:52.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:52.258 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:52.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:52.283 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:52.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:52.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:52.406 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:52.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:52.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:52.508 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:52.531 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:52.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:52.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:52.634 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:52.656 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:52.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:52.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:52.758 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:52.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:52.785 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:52.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:52.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:52.911 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:52.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:52.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:53.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:53.035 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:53.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:53.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:53.137 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:53.158 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:53.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:53.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:53.260 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:53.282 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:53.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:53.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:53.385 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:53.412 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:53.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:53.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:53.514 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:53.542 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:53.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:53.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:53.613 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:53 smithi136 ceph-mon[109861]: pgmap v3131: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.7 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 333 op/s; 47843/280845 objects degraded (17.035%); 32027/280845 objects misplaced (11.404%); 1.9 MiB/s, 31 objects/s recovering 2024-08-17T18:16:53.613 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:53 smithi136 ceph-mon[115078]: pgmap v3131: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.7 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 333 op/s; 47843/280845 objects degraded (17.035%); 32027/280845 objects misplaced (11.404%); 1.9 MiB/s, 31 objects/s recovering 2024-08-17T18:16:53.643 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:53.664 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:53.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:53.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:53.759 INFO:teuthology.orchestra.run.smithi136.stdout:moved 0 / 327 (0%) 2024-08-17T18:16:53.760 INFO:teuthology.orchestra.run.smithi136.stdout:avg 54.5 2024-08-17T18:16:53.760 INFO:teuthology.orchestra.run.smithi136.stdout:stddev 22.6918 -> 22.6918 (expected baseline 6.73919) 2024-08-17T18:16:53.760 INFO:teuthology.orchestra.run.smithi136.stdout:min osd.0 with 6 -> 6 pgs (0.110092 -> 0.110092 * mean) 2024-08-17T18:16:53.760 INFO:teuthology.orchestra.run.smithi136.stdout:max osd.6 with 73 -> 73 pgs (1.33945 -> 1.33945 * mean) 2024-08-17T18:16:53.760 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:16:53.760 INFO:teuthology.orchestra.run.smithi136.stdout:oload 1000 2024-08-17T18:16:53.760 INFO:teuthology.orchestra.run.smithi136.stdout:max_change 1 2024-08-17T18:16:53.760 INFO:teuthology.orchestra.run.smithi136.stdout:max_change_osds 4 2024-08-17T18:16:53.760 INFO:teuthology.orchestra.run.smithi136.stdout:average_utilization 0.0441 2024-08-17T18:16:53.760 INFO:teuthology.orchestra.run.smithi136.stdout:overload_utilization 0.4410 2024-08-17T18:16:53.761 INFO:teuthology.orchestra.run.smithi136.stderr:no change 2024-08-17T18:16:53.766 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:53.787 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:53.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:53.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:53.889 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:53.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:53.913 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:53.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:53 smithi165 ceph-mon[105517]: pgmap v3131: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.7 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 333 op/s; 47843/280845 objects degraded (17.035%); 32027/280845 objects misplaced (11.404%); 1.9 MiB/s, 31 objects/s recovering 2024-08-17T18:16:54.014 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:54.035 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:54.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:54.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:54.138 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:54.159 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:54.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:54.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:54.261 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:54.283 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:54.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:54.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:54.385 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:54.417 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:54.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:54.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:54.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:54.549 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:54.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:54.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:54.651 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:54.673 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:54.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:54.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:54.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:54 smithi136 ceph-mon[109861]: from='client.36603 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:16:54.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:54 smithi136 ceph-mon[115078]: from='client.36603 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:16:54.775 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:54.798 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:54.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:54.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:54.900 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:54.935 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:54.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:54.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:55.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:54 smithi165 ceph-mon[105517]: from='client.36603 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:16:55.037 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:55.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:55.059 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:55.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:55.094 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-08-17T18:16:55.095 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight-by-utilization 1000 0.05 2024-08-17T18:16:55.160 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:55.181 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:55.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:55.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:55.283 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:55.308 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:55.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:55.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:55.410 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:55.413 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:16:55.413 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:16:55.433 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:55.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:55.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:55.535 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:55.566 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:55.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:55.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:55.668 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:55.691 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:55.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:55.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:55 smithi136 ceph-mon[109861]: pgmap v3132: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.7 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 304 op/s; 47843/280845 objects degraded (17.035%); 32027/280845 objects misplaced (11.404%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T18:16:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:55 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3096637359' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:16:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:55 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 47843/280845 objects degraded (17.035%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-17T18:16:55.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:55 smithi136 ceph-mon[115078]: pgmap v3132: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.7 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 304 op/s; 47843/280845 objects degraded (17.035%); 32027/280845 objects misplaced (11.404%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T18:16:55.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:55 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3096637359' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:16:55.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:55 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 47843/280845 objects degraded (17.035%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-17T18:16:55.793 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:55.818 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:55.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:55.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:55.920 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:55.940 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:55.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:55.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:55 smithi165 ceph-mon[105517]: pgmap v3132: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.7 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 304 op/s; 47843/280845 objects degraded (17.035%); 32027/280845 objects misplaced (11.404%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T18:16:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:55 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3096637359' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:16:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:55 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 47843/280845 objects degraded (17.035%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-17T18:16:56.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:56.076 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:56.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:56.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:56.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:56.201 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:56.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:56.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:56.218 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-17T18:16:56.303 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:56.328 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:56.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:56.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:56.380 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:16:56.429 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:56.450 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:56.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:56.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:56.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:56.576 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:56.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:56.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:56.678 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:56.700 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:56.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:56.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:56.802 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:56.829 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:56.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:56.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:56.931 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:56.943 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:56 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:16:56.943 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:56 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3096637359' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:16:56.943 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:56 smithi136 ceph-mon[109861]: osdmap e1650: 8 total, 8 up, 6 in 2024-08-17T18:16:56.943 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:56 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:16:56.943 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:56 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3096637359' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:16:56.944 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:56 smithi136 ceph-mon[115078]: osdmap e1650: 8 total, 8 up, 6 in 2024-08-17T18:16:56.954 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:56.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:56.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:56 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:16:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:56 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3096637359' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:16:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:56 smithi165 ceph-mon[105517]: osdmap e1650: 8 total, 8 up, 6 in 2024-08-17T18:16:57.036 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:16:57.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:57.059 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:16:57.081 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:57.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:57.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:57.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:57.207 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:57.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:57.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:57.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:57.339 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:57.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:57.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:57.441 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:16:57.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:57.465 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:57.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:16:57.566 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:57.587 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:57.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:57.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:57.689 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:16:57.718 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:57.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:57.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:16:57.749 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:57 smithi136 ceph-mon[109861]: pgmap v3134: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.7 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 333 op/s; 47843/280719 objects degraded (17.043%); 31943/280719 objects misplaced (11.379%); 1.9 MiB/s, 31 objects/s recovering 2024-08-17T18:16:57.750 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:57 smithi136 ceph-mon[115078]: pgmap v3134: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.7 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 333 op/s; 47843/280719 objects degraded (17.043%); 31943/280719 objects misplaced (11.379%); 1.9 MiB/s, 31 objects/s recovering 2024-08-17T18:16:57.750 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:16:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:16:57] "GET /metrics HTTP/1.1" 200 36728 "" "Prometheus/2.51.0" 2024-08-17T18:16:57.819 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:57.842 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:57.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:57.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:57.944 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:57.967 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:16:57.967 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:57.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:57.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:57 smithi165 ceph-mon[105517]: pgmap v3134: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.7 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 333 op/s; 47843/280719 objects degraded (17.043%); 31943/280719 objects misplaced (11.379%); 1.9 MiB/s, 31 objects/s recovering 2024-08-17T18:16:58.069 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:58.091 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:58.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:58.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:58.193 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:58.215 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:58.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:58.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:58.318 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:58.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:58.340 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:58.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:58.441 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:58.463 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:58.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:58.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:58.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:58.588 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:58.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:58.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:58.690 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:58.719 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:58.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:58.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:58.822 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:16:58.845 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:58.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:58.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:16:58.947 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:58.966 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:16:58.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:58.970 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:58.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:59.071 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:59.094 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:59.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:59.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:59.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:59.218 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:59.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:59.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:16:59.320 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:16:59.343 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:59.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:59.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:16:59.444 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:16:59.465 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:59.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:59.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:16:59.568 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:59.597 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:59.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:59.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:59.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:16:59.720 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:59.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:59.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:16:59.817 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:59 smithi136 ceph-mon[109861]: pgmap v3135: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.6 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 306 op/s; 47775/276651 objects degraded (17.269%); 31508/276651 objects misplaced (11.389%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T18:16:59.817 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:16:59 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:16:59.817 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:59 smithi136 ceph-mon[115078]: pgmap v3135: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.6 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 306 op/s; 47775/276651 objects degraded (17.269%); 31508/276651 objects misplaced (11.389%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T18:16:59.817 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:16:59 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:16:59.823 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:16:59.854 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:16:59.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:59.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:16:59.898 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:59 smithi165 ceph-mon[105517]: pgmap v3135: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.6 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 306 op/s; 47775/276651 objects degraded (17.269%); 31508/276651 objects misplaced (11.389%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T18:16:59.898 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:16:59 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:16:59.956 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:16:59.977 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:16:59.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:16:59.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:00.079 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:00.106 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:00.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:00.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:00.208 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:00.231 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:00.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:00.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:00.332 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:00.360 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:00.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:00.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:00.462 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:00.503 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:00.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:00.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:00.605 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:00.636 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:00.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:00.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:00.737 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:00.773 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:00.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:00.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:00.822 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:00 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 47775/276651 objects degraded (17.269%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:00.875 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:00.898 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:00.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:00.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:00.913 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:00 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 47775/276651 objects degraded (17.269%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:00.913 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:00 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 47775/276651 objects degraded (17.269%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:01.000 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:01.022 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:01.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:01.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:01.124 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:01.152 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:01.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:01.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:01.254 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:01.276 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:01.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:01.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:01.378 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:01.399 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:01.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:01.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:01.501 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:01.542 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:17:01.542 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:17:01.542 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:01.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:01.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:01.644 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:01.667 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:01.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:01.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:01.769 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:01.802 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:01.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:01.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:01.846 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:01 smithi136 ceph-mon[109861]: pgmap v3136: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.6 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 306 op/s; 47775/276651 objects degraded (17.269%); 31508/276651 objects misplaced (11.389%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T18:17:01.846 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:01 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1526217861' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:17:01.847 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:01 smithi136 ceph-mon[115078]: pgmap v3136: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.6 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 306 op/s; 47775/276651 objects degraded (17.269%); 31508/276651 objects misplaced (11.389%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T18:17:01.847 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:01 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1526217861' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:17:01.904 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:01 smithi165 ceph-mon[105517]: pgmap v3136: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.6 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 306 op/s; 47775/276651 objects degraded (17.269%); 31508/276651 objects misplaced (11.389%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T18:17:01.904 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:01 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1526217861' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:17:01.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:01.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:01.937 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:01.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:02.038 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:02.068 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:02.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:02.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:02.170 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:02.199 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:02.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:02.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:02.301 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:02.332 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:02.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:02.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:02.434 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:02.439 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:17:02.439 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:17:02.439 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:17:02.439 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:17:02.440 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:17:02.440 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:17:02.440 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:17:02.440 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:17:02.440 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:17:02.440 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:17:02.440 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:17:02.441 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:17:02.441 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:17:02.441 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:17:02.441 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:17:02.441 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:17:02.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:02.457 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:02.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:02.558 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:02.581 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:02.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:02.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:02.683 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:02.706 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:02.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:02.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:02.807 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:02.836 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:02.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:02.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:02.938 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:02.960 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:02.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:02.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:03.062 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:03.083 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:17:03.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:03.086 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:03.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:03.187 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:03.209 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:03.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:03.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:03.311 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:03.333 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:03.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:03.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:03.435 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:03.456 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:03.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:03.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:03.558 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:03.579 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:03.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:03.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:03.681 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:03.683 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-17T18:17:03.706 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:03.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:03.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:03.808 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:03.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:03 smithi136 ceph-mon[109861]: pgmap v3137: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 326 op/s; 47650/271062 objects degraded (17.579%); 30878/271062 objects misplaced (11.391%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T18:17:03.827 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:03 smithi136 ceph-mon[109861]: 1.0 scrub starts 2024-08-17T18:17:03.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:03 smithi136 ceph-mon[115078]: pgmap v3137: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 326 op/s; 47650/271062 objects degraded (17.579%); 30878/271062 objects misplaced (11.391%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T18:17:03.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:03 smithi136 ceph-mon[115078]: 1.0 scrub starts 2024-08-17T18:17:03.831 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:03.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:03.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:03.933 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:03.956 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:03.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:03.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:04.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:03 smithi165 ceph-mon[105517]: pgmap v3137: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 326 op/s; 47650/271062 objects degraded (17.579%); 30878/271062 objects misplaced (11.391%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T18:17:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:03 smithi165 ceph-mon[105517]: 1.0 scrub starts 2024-08-17T18:17:04.058 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:04.082 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:04.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:04.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:04.184 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:04.214 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:04.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:04.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:04.242 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:17:04.316 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:04.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:04.348 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:04.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:04.411 INFO:teuthology.orchestra.run.smithi136.stdout:moved 0 / 327 (0%) 2024-08-17T18:17:04.411 INFO:teuthology.orchestra.run.smithi136.stdout:avg 54.5 2024-08-17T18:17:04.412 INFO:teuthology.orchestra.run.smithi136.stdout:stddev 22.6918 -> 22.6918 (expected baseline 6.73919) 2024-08-17T18:17:04.412 INFO:teuthology.orchestra.run.smithi136.stdout:min osd.0 with 6 -> 6 pgs (0.110092 -> 0.110092 * mean) 2024-08-17T18:17:04.412 INFO:teuthology.orchestra.run.smithi136.stdout:max osd.6 with 73 -> 73 pgs (1.33945 -> 1.33945 * mean) 2024-08-17T18:17:04.412 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:17:04.412 INFO:teuthology.orchestra.run.smithi136.stdout:oload 1000 2024-08-17T18:17:04.412 INFO:teuthology.orchestra.run.smithi136.stdout:max_change 0.05 2024-08-17T18:17:04.412 INFO:teuthology.orchestra.run.smithi136.stdout:max_change_osds 4 2024-08-17T18:17:04.412 INFO:teuthology.orchestra.run.smithi136.stdout:average_utilization 0.0432 2024-08-17T18:17:04.412 INFO:teuthology.orchestra.run.smithi136.stdout:overload_utilization 0.4320 2024-08-17T18:17:04.412 INFO:teuthology.orchestra.run.smithi136.stderr:no change 2024-08-17T18:17:04.449 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:04.471 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:04.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:04.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:04.573 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:04.595 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:04.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:04.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:04.600 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:04 smithi136 ceph-mon[109861]: 1.0 scrub starts 2024-08-17T18:17:04.600 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:04 smithi136 ceph-mon[115078]: 1.0 scrub starts 2024-08-17T18:17:04.697 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:04.723 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:04.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:04.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:04.825 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:04.853 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:04.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:04.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:04.935 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:04 smithi165 ceph-mon[105517]: 1.0 scrub starts 2024-08-17T18:17:04.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:04.983 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:04.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:04.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:05.036 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:17:05.085 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:05.106 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:05.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:05.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:05.160 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-08-17T18:17:05.160 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight-by-utilization 110 3.0 2024-08-17T18:17:05.208 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:05.231 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:05.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:05.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:05.333 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:05.366 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:05.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:05.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:05.468 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:05.493 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:05.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:05.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:05.595 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:05.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:05.627 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:05.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:05.652 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:05 smithi136 ceph-mon[109861]: from='client.36621 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:17:05.653 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:05 smithi136 ceph-mon[109861]: pgmap v3138: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 326 op/s; 47650/271062 objects degraded (17.579%); 30878/271062 objects misplaced (11.391%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T18:17:05.653 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:05 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 47650/271062 objects degraded (17.579%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:05.653 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:17:05.653 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:05 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:17:05.653 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:05 smithi136 ceph-mon[115078]: from='client.36621 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:17:05.653 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:05 smithi136 ceph-mon[115078]: pgmap v3138: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 326 op/s; 47650/271062 objects degraded (17.579%); 30878/271062 objects misplaced (11.391%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T18:17:05.653 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:05 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 47650/271062 objects degraded (17.579%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:05.653 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:17:05.653 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:05 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:17:05.728 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:05.750 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:05.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:05.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:05.852 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:05.873 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:05.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:05.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:05.975 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:06.002 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:06.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:06.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:06.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:05 smithi165 ceph-mon[105517]: from='client.36621 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:17:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:05 smithi165 ceph-mon[105517]: pgmap v3138: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 326 op/s; 47650/271062 objects degraded (17.579%); 30878/271062 objects misplaced (11.391%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T18:17:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:05 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 47650/271062 objects degraded (17.579%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:17:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:05 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:17:06.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:06.126 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:06.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:06.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:06.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:06.251 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:06.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:06.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:06.353 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:06.375 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:06.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:06.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:06.477 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:06.498 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:06.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:06.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:06.600 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:06.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:06.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:06.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:06.731 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:06.755 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:06.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:06.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:06.857 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:06.878 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:06.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:06.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:06.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:17:06.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:17:06.979 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:07.000 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:07.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:07.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:07.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:17:07.101 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:07.122 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:07.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:07.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:07.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:07.258 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:07.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:07.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:07.360 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:07.388 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:07.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:07.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:07.490 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:07.513 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:07.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:07.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:07.615 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:07.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:07.646 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:07.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:07.739 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:07 smithi136 ceph-mon[109861]: pgmap v3139: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 303 op/s; 47650/270933 objects degraded (17.587%); 30792/270933 objects misplaced (11.365%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T18:17:07.740 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:07 smithi136 ceph-mon[115078]: pgmap v3139: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 303 op/s; 47650/270933 objects degraded (17.587%); 30792/270933 objects misplaced (11.365%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T18:17:07.740 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:17:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:17:07] "GET /metrics HTTP/1.1" 200 36732 "" "Prometheus/2.51.0" 2024-08-17T18:17:07.747 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:07.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:07.782 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:07.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:07.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:07.905 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:07.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:07.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:08.007 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:08.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:07 smithi165 ceph-mon[105517]: pgmap v3139: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 303 op/s; 47650/270933 objects degraded (17.587%); 30792/270933 objects misplaced (11.365%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T18:17:08.028 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:08.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:08.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:08.130 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:08.156 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:08.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:08.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:08.258 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:08.279 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:08.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:08.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:08.381 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:08.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:08.416 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:08.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:08.517 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:08.545 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:08.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:08.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:08.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:08.671 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:08.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:08.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:08.773 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:08.796 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:08.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:08.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:08.898 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:08.927 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:08.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:08.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:09.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:09.052 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:09.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:09.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:09.154 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:09.176 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:09.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:09.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:09.278 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:09.304 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:09.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:09.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:09.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:09.427 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:09.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:09.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:09.529 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:09.553 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:09.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:09.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:09.655 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:09.676 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:09.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:09.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:09.778 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:09.802 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:09.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:09.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:09.896 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:09 smithi136 ceph-mon[109861]: pgmap v3140: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.4 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 391 op/s; 47564/266604 objects degraded (17.841%); 30363/266604 objects misplaced (11.389%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T18:17:09.896 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:09 smithi136 ceph-mon[115078]: pgmap v3140: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.4 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 391 op/s; 47564/266604 objects degraded (17.841%); 30363/266604 objects misplaced (11.389%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T18:17:09.905 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:09.917 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:09 smithi165 ceph-mon[105517]: pgmap v3140: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.4 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 391 op/s; 47564/266604 objects degraded (17.841%); 30363/266604 objects misplaced (11.389%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T18:17:09.927 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:09.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:09.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:10.029 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:10.051 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:10.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:10.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:10.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:10.175 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:10.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:10.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:10.278 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:10.303 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:10.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:10.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:10.404 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:10.428 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:10.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:10.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:10.530 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:10.553 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:10.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:10.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:10.655 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:10.677 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:10.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:10.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:10.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:10 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 47564/266604 objects degraded (17.841%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:10.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:10 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 47564/266604 objects degraded (17.841%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:10.712 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:17:10.712 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-17T18:17:10.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-17T18:17:10.713 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:17:10.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.713 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:219427 56.12 56:48959621:::benchmark_data_smithi136_356456_object92723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1577)", 2024-08-17T18:17:10.713 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:52.752468+0000", 2024-08-17T18:17:10.713 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 256.64226272899998, 2024-08-17T18:17:10.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0230039949999998, 2024-08-17T18:17:10.713 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:10.713 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:10.713 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:10.713 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:10.713 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:17:10.713 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:17:10.714 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 219427 2024-08-17T18:17:10.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.714 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:10.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:10.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.752468+0000", 2024-08-17T18:17:10.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:10.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.752468+0000", 2024-08-17T18:17:10.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:10.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.752469+0000", 2024-08-17T18:17:10.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5000000000000001e-07 2024-08-17T18:17:10.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:10.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.752473+0000", 2024-08-17T18:17:10.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4859999999999999e-06 2024-08-17T18:17:10.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:10.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.752488+0000", 2024-08-17T18:17:10.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4704e-05 2024-08-17T18:17:10.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:10.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.752495+0000", 2024-08-17T18:17:10.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.605e-06 2024-08-17T18:17:10.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.752559+0000", 2024-08-17T18:17:10.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3542999999999998e-05 2024-08-17T18:17:10.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.752569+0000", 2024-08-17T18:17:10.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9059999999999997e-06 2024-08-17T18:17:10.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:53.699132+0000", 2024-08-17T18:17:10.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.94656289999999998 2024-08-17T18:17:10.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:53.699144+0000", 2024-08-17T18:17:10.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2337999999999999e-05 2024-08-17T18:17:10.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.771308+0000", 2024-08-17T18:17:10.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.072163816 2024-08-17T18:17:10.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:10.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.771433+0000", 2024-08-17T18:17:10.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012502600000000001 2024-08-17T18:17:10.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5", 2024-08-17T18:17:10.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.771708+0000", 2024-08-17T18:17:10.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000274679 2024-08-17T18:17:10.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:10.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775059+0000", 2024-08-17T18:17:10.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033514619999999999 2024-08-17T18:17:10.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775097+0000", 2024-08-17T18:17:10.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7625e-05 2024-08-17T18:17:10.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775256+0000", 2024-08-17T18:17:10.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015889699999999999 2024-08-17T18:17:10.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775310+0000", 2024-08-17T18:17:10.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4381e-05 2024-08-17T18:17:10.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:10.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775350+0000", 2024-08-17T18:17:10.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9864999999999999e-05 2024-08-17T18:17:10.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:10.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775472+0000", 2024-08-17T18:17:10.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012181200000000001 2024-08-17T18:17:10.721 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.721 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:10.721 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.721 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:219447 56.12 56:4a3a3ee4:::benchmark_data_smithi136_356456_object92743:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1577)", 2024-08-17T18:17:10.721 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:52.767811+0000", 2024-08-17T18:17:10.722 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 256.62691894099999, 2024-08-17T18:17:10.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.007993371, 2024-08-17T18:17:10.722 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:10.722 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:10.722 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:10.722 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:10.722 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:17:10.722 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:17:10.722 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 219447 2024-08-17T18:17:10.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.722 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:10.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:10.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.767811+0000", 2024-08-17T18:17:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.767811+0000", 2024-08-17T18:17:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.767812+0000", 2024-08-17T18:17:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7100000000000001e-07 2024-08-17T18:17:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.767816+0000", 2024-08-17T18:17:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.563e-06 2024-08-17T18:17:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.767824+0000", 2024-08-17T18:17:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5539999999999998e-06 2024-08-17T18:17:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.767829+0000", 2024-08-17T18:17:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0610000000000003e-06 2024-08-17T18:17:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.767880+0000", 2024-08-17T18:17:10.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0963e-05 2024-08-17T18:17:10.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.767887+0000", 2024-08-17T18:17:10.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3589999999999996e-06 2024-08-17T18:17:10.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:53.699170+0000", 2024-08-17T18:17:10.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93128367700000003 2024-08-17T18:17:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:53.699179+0000", 2024-08-17T18:17:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4589999999999998e-06 2024-08-17T18:17:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.773078+0000", 2024-08-17T18:17:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0738989130000001 2024-08-17T18:17:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.773156+0000", 2024-08-17T18:17:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8424000000000007e-05 2024-08-17T18:17:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5", 2024-08-17T18:17:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.773294+0000", 2024-08-17T18:17:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000138367 2024-08-17T18:17:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775205+0000", 2024-08-17T18:17:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0019106800000000001 2024-08-17T18:17:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775516+0000", 2024-08-17T18:17:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00031132399999999998 2024-08-17T18:17:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775654+0000", 2024-08-17T18:17:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013750399999999999 2024-08-17T18:17:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775688+0000", 2024-08-17T18:17:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4180000000000001e-05 2024-08-17T18:17:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775731+0000", 2024-08-17T18:17:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2864000000000002e-05 2024-08-17T18:17:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775805+0000", 2024-08-17T18:17:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3807999999999995e-05 2024-08-17T18:17:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.730 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:219454 56.12 56:496432b9:::benchmark_data_smithi136_356456_object92750:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1577)", 2024-08-17T18:17:10.730 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:52.771621+0000", 2024-08-17T18:17:10.730 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 256.62310911899999, 2024-08-17T18:17:10.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.004395073, 2024-08-17T18:17:10.730 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:10.730 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:10.730 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:10.730 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:10.730 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:17:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:17:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 219454 2024-08-17T18:17:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.771621+0000", 2024-08-17T18:17:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.771621+0000", 2024-08-17T18:17:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:10.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.771622+0000", 2024-08-17T18:17:10.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4999999999999998e-07 2024-08-17T18:17:10.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:10.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.771624+0000", 2024-08-17T18:17:10.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6359999999999998e-06 2024-08-17T18:17:10.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.771630+0000", 2024-08-17T18:17:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4790000000000004e-06 2024-08-17T18:17:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.771634+0000", 2024-08-17T18:17:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8e-06 2024-08-17T18:17:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.771687+0000", 2024-08-17T18:17:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3449000000000003e-05 2024-08-17T18:17:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.771707+0000", 2024-08-17T18:17:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9898000000000002e-05 2024-08-17T18:17:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:53.699204+0000", 2024-08-17T18:17:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92749685800000004 2024-08-17T18:17:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:53.699212+0000", 2024-08-17T18:17:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8760000000000005e-06 2024-08-17T18:17:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.774136+0000", 2024-08-17T18:17:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0749248140000001 2024-08-17T18:17:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.774205+0000", 2024-08-17T18:17:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8476e-05 2024-08-17T18:17:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5", 2024-08-17T18:17:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.774378+0000", 2024-08-17T18:17:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017333999999999999 2024-08-17T18:17:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775483+0000", 2024-08-17T18:17:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001105036 2024-08-17T18:17:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775857+0000", 2024-08-17T18:17:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00037334500000000002 2024-08-17T18:17:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775895+0000", 2024-08-17T18:17:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8600999999999999e-05 2024-08-17T18:17:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775930+0000", 2024-08-17T18:17:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5145000000000003e-05 2024-08-17T18:17:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:10.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775962+0000", 2024-08-17T18:17:10.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1380000000000001e-05 2024-08-17T18:17:10.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:10.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.776016+0000", 2024-08-17T18:17:10.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4490000000000002e-05 2024-08-17T18:17:10.738 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.738 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:10.738 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.738 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:219488 56.12 56:4d9fe1f5:::benchmark_data_smithi136_356456_object92784:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1577)", 2024-08-17T18:17:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:52.814105+0000", 2024-08-17T18:17:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 256.58062531299998, 2024-08-17T18:17:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.974696437, 2024-08-17T18:17:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:17:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:17:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 219488 2024-08-17T18:17:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.814105+0000", 2024-08-17T18:17:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.814105+0000", 2024-08-17T18:17:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.814106+0000", 2024-08-17T18:17:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.308e-06 2024-08-17T18:17:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.814111+0000", 2024-08-17T18:17:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7430000000000002e-06 2024-08-17T18:17:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.814128+0000", 2024-08-17T18:17:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6473999999999998e-05 2024-08-17T18:17:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.814136+0000", 2024-08-17T18:17:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6319999999999997e-06 2024-08-17T18:17:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.814199+0000", 2024-08-17T18:17:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2967999999999997e-05 2024-08-17T18:17:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.814206+0000", 2024-08-17T18:17:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2339999999999997e-06 2024-08-17T18:17:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:53.699945+0000", 2024-08-17T18:17:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.88573835199999995 2024-08-17T18:17:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:53.699955+0000", 2024-08-17T18:17:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0208e-05 2024-08-17T18:17:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.782619+0000", 2024-08-17T18:17:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.082663836 2024-08-17T18:17:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.782742+0000", 2024-08-17T18:17:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000122915 2024-08-17T18:17:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5", 2024-08-17T18:17:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.783001+0000", 2024-08-17T18:17:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025919199999999999 2024-08-17T18:17:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.788503+0000", 2024-08-17T18:17:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0055020629999999997 2024-08-17T18:17:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.788553+0000", 2024-08-17T18:17:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9591e-05 2024-08-17T18:17:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.788616+0000", 2024-08-17T18:17:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3733999999999997e-05 2024-08-17T18:17:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.788665+0000", 2024-08-17T18:17:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8930000000000001e-05 2024-08-17T18:17:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.788710+0000", 2024-08-17T18:17:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4722000000000003e-05 2024-08-17T18:17:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.788801+0000", 2024-08-17T18:17:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1534999999999997e-05 2024-08-17T18:17:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:219495 56.12 56:4a7b6ab5:::benchmark_data_smithi136_356456_object92791:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1577)", 2024-08-17T18:17:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:52.822737+0000", 2024-08-17T18:17:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 256.57199334000001, 2024-08-17T18:17:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9663730479999999, 2024-08-17T18:17:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:17:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:17:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 219495 2024-08-17T18:17:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.748 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:10.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:10.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.822737+0000", 2024-08-17T18:17:10.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:10.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.822737+0000", 2024-08-17T18:17:10.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.822738+0000", 2024-08-17T18:17:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.35e-06 2024-08-17T18:17:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.822743+0000", 2024-08-17T18:17:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4410000000000003e-06 2024-08-17T18:17:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.822755+0000", 2024-08-17T18:17:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2031000000000001e-05 2024-08-17T18:17:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.822762+0000", 2024-08-17T18:17:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5630000000000003e-06 2024-08-17T18:17:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.822820+0000", 2024-08-17T18:17:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7855000000000001e-05 2024-08-17T18:17:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.822830+0000", 2024-08-17T18:17:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5119999999999997e-06 2024-08-17T18:17:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:53.700051+0000", 2024-08-17T18:17:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.87722086300000002 2024-08-17T18:17:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:53.700057+0000", 2024-08-17T18:17:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1600000000000003e-06 2024-08-17T18:17:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.784269+0000", 2024-08-17T18:17:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0842123809999999 2024-08-17T18:17:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.784385+0000", 2024-08-17T18:17:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116039 2024-08-17T18:17:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5", 2024-08-17T18:17:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.784611+0000", 2024-08-17T18:17:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022579300000000001 2024-08-17T18:17:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.788636+0000", 2024-08-17T18:17:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040248389999999997 2024-08-17T18:17:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.788855+0000", 2024-08-17T18:17:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021960799999999999 2024-08-17T18:17:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.788893+0000", 2024-08-17T18:17:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7834999999999999e-05 2024-08-17T18:17:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.788935+0000", 2024-08-17T18:17:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1292999999999999e-05 2024-08-17T18:17:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.788991+0000", 2024-08-17T18:17:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6440999999999997e-05 2024-08-17T18:17:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.789110+0000", 2024-08-17T18:17:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011904399999999999 2024-08-17T18:17:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.755 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:10.755 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.755 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:219594 56.12 56:4c578227:::benchmark_data_smithi136_356456_object92890:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1577)", 2024-08-17T18:17:10.755 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:52.898347+0000", 2024-08-17T18:17:10.755 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 256.496382966, 2024-08-17T18:17:10.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.89111677, 2024-08-17T18:17:10.755 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:10.755 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:10.755 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:17:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:17:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 219594 2024-08-17T18:17:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.898347+0000", 2024-08-17T18:17:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.898347+0000", 2024-08-17T18:17:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.898349+0000", 2024-08-17T18:17:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.621e-06 2024-08-17T18:17:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.898354+0000", 2024-08-17T18:17:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2079999999999999e-06 2024-08-17T18:17:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.898365+0000", 2024-08-17T18:17:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0403e-05 2024-08-17T18:17:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.898372+0000", 2024-08-17T18:17:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0419999999999997e-06 2024-08-17T18:17:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.898432+0000", 2024-08-17T18:17:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0794000000000001e-05 2024-08-17T18:17:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.898444+0000", 2024-08-17T18:17:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1504e-05 2024-08-17T18:17:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:53.700084+0000", 2024-08-17T18:17:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.80163971000000001 2024-08-17T18:17:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:53.700089+0000", 2024-08-17T18:17:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.852e-06 2024-08-17T18:17:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.785913+0000", 2024-08-17T18:17:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.085824007 2024-08-17T18:17:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.786032+0000", 2024-08-17T18:17:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011839499999999999 2024-08-17T18:17:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5", 2024-08-17T18:17:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.786224+0000", 2024-08-17T18:17:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019196600000000001 2024-08-17T18:17:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.788644+0000", 2024-08-17T18:17:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0024205310000000001 2024-08-17T18:17:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.789164+0000", 2024-08-17T18:17:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00051947500000000004 2024-08-17T18:17:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.789209+0000", 2024-08-17T18:17:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4965000000000001e-05 2024-08-17T18:17:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.789277+0000", 2024-08-17T18:17:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8555999999999999e-05 2024-08-17T18:17:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.789335+0000", 2024-08-17T18:17:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7323e-05 2024-08-17T18:17:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.789464+0000", 2024-08-17T18:17:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012941800000000001 2024-08-17T18:17:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:91775 57.12 57:4c5b5116:::benchmark_data_smithi136_389235_object91774:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:17:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.694206+0000", 2024-08-17T18:17:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 128.70052479200001, 2024-08-17T18:17:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1991839390000001, 2024-08-17T18:17:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:17:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:17:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91775 2024-08-17T18:17:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.694206+0000", 2024-08-17T18:17:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.694206+0000", 2024-08-17T18:17:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.694206+0000", 2024-08-17T18:17:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6400000000000001e-07 2024-08-17T18:17:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.694222+0000", 2024-08-17T18:17:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5503999999999999e-05 2024-08-17T18:17:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.694230+0000", 2024-08-17T18:17:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2660000000000001e-06 2024-08-17T18:17:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.694235+0000", 2024-08-17T18:17:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5940000000000004e-06 2024-08-17T18:17:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.694293+0000", 2024-08-17T18:17:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8186999999999997e-05 2024-08-17T18:17:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.694303+0000", 2024-08-17T18:17:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0059e-05 2024-08-17T18:17:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.646421+0000", 2024-08-17T18:17:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.95211796000000004 2024-08-17T18:17:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.646435+0000", 2024-08-17T18:17:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3908e-05 2024-08-17T18:17:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.887386+0000", 2024-08-17T18:17:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.240951269 2024-08-17T18:17:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.887571+0000", 2024-08-17T18:17:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018437799999999999 2024-08-17T18:17:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:17:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.887823+0000", 2024-08-17T18:17:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025225300000000003 2024-08-17T18:17:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.892158+0000", 2024-08-17T18:17:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.004335401 2024-08-17T18:17:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.892215+0000", 2024-08-17T18:17:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6533000000000002e-05 2024-08-17T18:17:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.892345+0000", 2024-08-17T18:17:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012994600000000001 2024-08-17T18:17:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.892406+0000", 2024-08-17T18:17:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1282e-05 2024-08-17T18:17:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.893389+0000", 2024-08-17T18:17:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00098353500000000001 2024-08-17T18:17:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:91798 57.12 57:4d0d602d:::benchmark_data_smithi136_389235_object91797:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:17:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.714863+0000", 2024-08-17T18:17:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 128.67986719000001, 2024-08-17T18:17:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.186937039, 2024-08-17T18:17:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:17:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:17:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91798 2024-08-17T18:17:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.714863+0000", 2024-08-17T18:17:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.714863+0000", 2024-08-17T18:17:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.714864+0000", 2024-08-17T18:17:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5099999999999998e-07 2024-08-17T18:17:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.714889+0000", 2024-08-17T18:17:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5029e-05 2024-08-17T18:17:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.714897+0000", 2024-08-17T18:17:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2570000000000007e-06 2024-08-17T18:17:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.714903+0000", 2024-08-17T18:17:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9390000000000002e-06 2024-08-17T18:17:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.714971+0000", 2024-08-17T18:17:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7811999999999994e-05 2024-08-17T18:17:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.714982+0000", 2024-08-17T18:17:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0814e-05 2024-08-17T18:17:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.646899+0000", 2024-08-17T18:17:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93191730399999995 2024-08-17T18:17:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.646905+0000", 2024-08-17T18:17:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6779999999999997e-06 2024-08-17T18:17:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.889432+0000", 2024-08-17T18:17:10.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.242527468 2024-08-17T18:17:10.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:10.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.889612+0000", 2024-08-17T18:17:10.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000179722 2024-08-17T18:17:10.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:17:10.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.889890+0000", 2024-08-17T18:17:10.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027764899999999998 2024-08-17T18:17:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.892278+0000", 2024-08-17T18:17:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023882690000000002 2024-08-17T18:17:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.892292+0000", 2024-08-17T18:17:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4258e-05 2024-08-17T18:17:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.900664+0000", 2024-08-17T18:17:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0083714710000000001 2024-08-17T18:17:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.900720+0000", 2024-08-17T18:17:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6434000000000002e-05 2024-08-17T18:17:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.901800+0000", 2024-08-17T18:17:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001079984 2024-08-17T18:17:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:91831 57.12 57:49f84744:::benchmark_data_smithi136_389235_object91830:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:17:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.733890+0000", 2024-08-17T18:17:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 128.66084029300001, 2024-08-17T18:17:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1688886919999999, 2024-08-17T18:17:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:17:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:17:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91831 2024-08-17T18:17:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.733890+0000", 2024-08-17T18:17:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.733890+0000", 2024-08-17T18:17:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.733891+0000", 2024-08-17T18:17:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7400000000000002e-07 2024-08-17T18:17:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.733910+0000", 2024-08-17T18:17:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9232000000000002e-05 2024-08-17T18:17:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.733921+0000", 2024-08-17T18:17:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0787e-05 2024-08-17T18:17:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:10.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.733928+0000", 2024-08-17T18:17:10.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4320000000000002e-06 2024-08-17T18:17:10.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.733984+0000", 2024-08-17T18:17:10.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6233000000000002e-05 2024-08-17T18:17:10.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734004+0000", 2024-08-17T18:17:10.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9307e-05 2024-08-17T18:17:10.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.646939+0000", 2024-08-17T18:17:10.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.91293501099999996 2024-08-17T18:17:10.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.646948+0000", 2024-08-17T18:17:10.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9709999999999993e-06 2024-08-17T18:17:10.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.890760+0000", 2024-08-17T18:17:10.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2438118359999999 2024-08-17T18:17:10.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:10.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.890914+0000", 2024-08-17T18:17:10.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015469099999999999 2024-08-17T18:17:10.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:17:10.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.891161+0000", 2024-08-17T18:17:10.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024684400000000002 2024-08-17T18:17:10.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.894974+0000", 2024-08-17T18:17:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038126700000000002 2024-08-17T18:17:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.895037+0000", 2024-08-17T18:17:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3083000000000006e-05 2024-08-17T18:17:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.901890+0000", 2024-08-17T18:17:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0068527170000000004 2024-08-17T18:17:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.901927+0000", 2024-08-17T18:17:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7088999999999997e-05 2024-08-17T18:17:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.902779+0000", 2024-08-17T18:17:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00085201500000000002 2024-08-17T18:17:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:91833 57.12 57:4fab0a31:::benchmark_data_smithi136_389235_object91832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:17:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.734763+0000", 2024-08-17T18:17:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 128.659967091, 2024-08-17T18:17:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1724328439999998, 2024-08-17T18:17:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:17:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:17:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91833 2024-08-17T18:17:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734763+0000", 2024-08-17T18:17:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734763+0000", 2024-08-17T18:17:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734764+0000", 2024-08-17T18:17:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7699999999999992e-07 2024-08-17T18:17:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734787+0000", 2024-08-17T18:17:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3243000000000001e-05 2024-08-17T18:17:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734800+0000", 2024-08-17T18:17:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2903e-05 2024-08-17T18:17:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734807+0000", 2024-08-17T18:17:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1459999999999999e-06 2024-08-17T18:17:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734860+0000", 2024-08-17T18:17:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2874999999999998e-05 2024-08-17T18:17:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734869+0000", 2024-08-17T18:17:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9039999999999998e-06 2024-08-17T18:17:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.646980+0000", 2024-08-17T18:17:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.91211099200000001 2024-08-17T18:17:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.646987+0000", 2024-08-17T18:17:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1149999999999996e-06 2024-08-17T18:17:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.893439+0000", 2024-08-17T18:17:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2464515359999999 2024-08-17T18:17:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.893736+0000", 2024-08-17T18:17:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029668499999999998 2024-08-17T18:17:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:17:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.893960+0000", 2024-08-17T18:17:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000224696 2024-08-17T18:17:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.897423+0000", 2024-08-17T18:17:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034624370000000001 2024-08-17T18:17:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.897433+0000", 2024-08-17T18:17:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0197000000000001e-05 2024-08-17T18:17:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.906207+0000", 2024-08-17T18:17:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0087737289999999992 2024-08-17T18:17:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.906242+0000", 2024-08-17T18:17:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5824000000000001e-05 2024-08-17T18:17:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907196+0000", 2024-08-17T18:17:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00095358499999999996 2024-08-17T18:17:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:91851 57.12 57:4935f3b2:::benchmark_data_smithi136_389235_object91850:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:17:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.746420+0000", 2024-08-17T18:17:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 128.64831003099999, 2024-08-17T18:17:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1658063599999999, 2024-08-17T18:17:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:17:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:17:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91851 2024-08-17T18:17:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.746420+0000", 2024-08-17T18:17:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.746420+0000", 2024-08-17T18:17:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.746421+0000", 2024-08-17T18:17:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3900000000000004e-07 2024-08-17T18:17:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.746444+0000", 2024-08-17T18:17:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2609999999999999e-05 2024-08-17T18:17:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.746452+0000", 2024-08-17T18:17:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0499999999999992e-06 2024-08-17T18:17:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.746457+0000", 2024-08-17T18:17:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3310000000000003e-06 2024-08-17T18:17:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.747175+0000", 2024-08-17T18:17:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00071805899999999995 2024-08-17T18:17:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.747184+0000", 2024-08-17T18:17:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2960000000000004e-06 2024-08-17T18:17:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647023+0000", 2024-08-17T18:17:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.89983893699999995 2024-08-17T18:17:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647032+0000", 2024-08-17T18:17:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7339999999999997e-06 2024-08-17T18:17:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.895088+0000", 2024-08-17T18:17:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2480557990000001 2024-08-17T18:17:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.895247+0000", 2024-08-17T18:17:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000159512 2024-08-17T18:17:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:17:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.895471+0000", 2024-08-17T18:17:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022369299999999999 2024-08-17T18:17:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.899385+0000", 2024-08-17T18:17:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003913583 2024-08-17T18:17:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.900594+0000", 2024-08-17T18:17:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012091599999999999 2024-08-17T18:17:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.910847+0000", 2024-08-17T18:17:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010253536000000001 2024-08-17T18:17:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.910887+0000", 2024-08-17T18:17:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0167999999999999e-05 2024-08-17T18:17:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.912227+0000", 2024-08-17T18:17:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0013392530000000001 2024-08-17T18:17:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:91877 57.12 57:4e2ca095:::benchmark_data_smithi136_389235_object91876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:17:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.764679+0000", 2024-08-17T18:17:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 128.63005124599999, 2024-08-17T18:17:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.148880057, 2024-08-17T18:17:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:17:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:17:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91877 2024-08-17T18:17:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.764679+0000", 2024-08-17T18:17:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.764679+0000", 2024-08-17T18:17:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.764681+0000", 2024-08-17T18:17:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5239999999999999e-06 2024-08-17T18:17:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.764715+0000", 2024-08-17T18:17:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3964000000000002e-05 2024-08-17T18:17:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.764727+0000", 2024-08-17T18:17:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2762000000000001e-05 2024-08-17T18:17:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.764735+0000", 2024-08-17T18:17:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3769999999999998e-06 2024-08-17T18:17:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.764809+0000", 2024-08-17T18:17:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3935000000000003e-05 2024-08-17T18:17:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.764821+0000", 2024-08-17T18:17:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2024e-05 2024-08-17T18:17:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647058+0000", 2024-08-17T18:17:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.88223715999999996 2024-08-17T18:17:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647062+0000", 2024-08-17T18:17:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1230000000000002e-06 2024-08-17T18:17:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.896097+0000", 2024-08-17T18:17:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.249034932 2024-08-17T18:17:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.896247+0000", 2024-08-17T18:17:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000150309 2024-08-17T18:17:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:17:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.896475+0000", 2024-08-17T18:17:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000227815 2024-08-17T18:17:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.900604+0000", 2024-08-17T18:17:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0041288979999999998 2024-08-17T18:17:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.901852+0000", 2024-08-17T18:17:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012485809999999999 2024-08-17T18:17:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.912289+0000", 2024-08-17T18:17:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010436453 2024-08-17T18:17:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.912355+0000", 2024-08-17T18:17:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5947999999999999e-05 2024-08-17T18:17:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.913559+0000", 2024-08-17T18:17:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012042520000000001 2024-08-17T18:17:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:91955 57.12 57:4acb1630:::benchmark_data_smithi136_389235_object91954:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:17:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.812317+0000", 2024-08-17T18:17:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 128.58241294999999, 2024-08-17T18:17:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1059336489999998, 2024-08-17T18:17:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:17:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:17:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91955 2024-08-17T18:17:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.812317+0000", 2024-08-17T18:17:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.812317+0000", 2024-08-17T18:17:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.812319+0000", 2024-08-17T18:17:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.31e-06 2024-08-17T18:17:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.812338+0000", 2024-08-17T18:17:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8881000000000001e-05 2024-08-17T18:17:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.812345+0000", 2024-08-17T18:17:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9470000000000003e-06 2024-08-17T18:17:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.812350+0000", 2024-08-17T18:17:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3919999999999996e-06 2024-08-17T18:17:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.812427+0000", 2024-08-17T18:17:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6713999999999998e-05 2024-08-17T18:17:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.812437+0000", 2024-08-17T18:17:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0720000000000001e-05 2024-08-17T18:17:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647088+0000", 2024-08-17T18:17:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.83465034999999999 2024-08-17T18:17:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647095+0000", 2024-08-17T18:17:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6599999999999995e-06 2024-08-17T18:17:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.897471+0000", 2024-08-17T18:17:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2503756109999999 2024-08-17T18:17:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.897618+0000", 2024-08-17T18:17:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000146699 2024-08-17T18:17:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:17:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.897843+0000", 2024-08-17T18:17:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022513899999999999 2024-08-17T18:17:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.900609+0000", 2024-08-17T18:17:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0027664709999999999 2024-08-17T18:17:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.903881+0000", 2024-08-17T18:17:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032721510000000001 2024-08-17T18:17:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.916916+0000", 2024-08-17T18:17:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013034108000000001 2024-08-17T18:17:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.916997+0000", 2024-08-17T18:17:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1508000000000006e-05 2024-08-17T18:17:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.918251+0000", 2024-08-17T18:17:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012539879999999999 2024-08-17T18:17:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:91997 57.12 57:480a6a35:::benchmark_data_smithi136_389235_object91996:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:17:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.839160+0000", 2024-08-17T18:17:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 128.55557015100001, 2024-08-17T18:17:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0802216869999999, 2024-08-17T18:17:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:17:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:17:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91997 2024-08-17T18:17:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.839160+0000", 2024-08-17T18:17:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.839160+0000", 2024-08-17T18:17:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.839161+0000", 2024-08-17T18:17:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1510000000000001e-06 2024-08-17T18:17:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.839185+0000", 2024-08-17T18:17:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3805e-05 2024-08-17T18:17:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.839194+0000", 2024-08-17T18:17:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1500000000000005e-06 2024-08-17T18:17:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.839201+0000", 2024-08-17T18:17:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2720000000000003e-06 2024-08-17T18:17:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.839374+0000", 2024-08-17T18:17:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017318800000000001 2024-08-17T18:17:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.839420+0000", 2024-08-17T18:17:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6050999999999997e-05 2024-08-17T18:17:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647119+0000", 2024-08-17T18:17:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.80769961099999998 2024-08-17T18:17:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647123+0000", 2024-08-17T18:17:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9400000000000004e-06 2024-08-17T18:17:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.898480+0000", 2024-08-17T18:17:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2513567079999999 2024-08-17T18:17:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.898570+0000", 2024-08-17T18:17:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0339000000000004e-05 2024-08-17T18:17:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:17:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.898732+0000", 2024-08-17T18:17:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000161356 2024-08-17T18:17:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.901862+0000", 2024-08-17T18:17:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031297849999999999 2024-08-17T18:17:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.903932+0000", 2024-08-17T18:17:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020700010000000001 2024-08-17T18:17:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.918308+0000", 2024-08-17T18:17:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014376938000000001 2024-08-17T18:17:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.918369+0000", 2024-08-17T18:17:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0520999999999999e-05 2024-08-17T18:17:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.919382+0000", 2024-08-17T18:17:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001012871 2024-08-17T18:17:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:92000 57.12 57:4c400e2c:::benchmark_data_smithi136_389235_object91999:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:17:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.840632+0000", 2024-08-17T18:17:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 128.55409875800001, 2024-08-17T18:17:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0960051650000002, 2024-08-17T18:17:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:17:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:17:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 92000 2024-08-17T18:17:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.840632+0000", 2024-08-17T18:17:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.840632+0000", 2024-08-17T18:17:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.840633+0000", 2024-08-17T18:17:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.404e-06 2024-08-17T18:17:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.840653+0000", 2024-08-17T18:17:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0111000000000001e-05 2024-08-17T18:17:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.840664+0000", 2024-08-17T18:17:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0988999999999999e-05 2024-08-17T18:17:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.840671+0000", 2024-08-17T18:17:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4810000000000003e-06 2024-08-17T18:17:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.840748+0000", 2024-08-17T18:17:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7634999999999998e-05 2024-08-17T18:17:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.840758+0000", 2024-08-17T18:17:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6309999999999998e-06 2024-08-17T18:17:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647137+0000", 2024-08-17T18:17:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.80637933399999995 2024-08-17T18:17:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647141+0000", 2024-08-17T18:17:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8260000000000003e-06 2024-08-17T18:17:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.899425+0000", 2024-08-17T18:17:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2522841220000001 2024-08-17T18:17:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.899560+0000", 2024-08-17T18:17:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013490299999999999 2024-08-17T18:17:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:17:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.899790+0000", 2024-08-17T18:17:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0002299 2024-08-17T18:17:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.903824+0000", 2024-08-17T18:17:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040337039999999999 2024-08-17T18:17:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.903988+0000", 2024-08-17T18:17:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000164752 2024-08-17T18:17:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.935038+0000", 2024-08-17T18:17:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.031049243000000001 2024-08-17T18:17:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.935099+0000", 2024-08-17T18:17:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1532000000000007e-05 2024-08-17T18:17:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.936637+0000", 2024-08-17T18:17:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001537598 2024-08-17T18:17:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:92004 57.12 57:4b569127:::benchmark_data_smithi136_389235_object92003:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:17:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.841912+0000", 2024-08-17T18:17:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 128.55281795600001, 2024-08-17T18:17:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.095822149, 2024-08-17T18:17:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:17:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:17:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 92004 2024-08-17T18:17:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.841912+0000", 2024-08-17T18:17:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.841912+0000", 2024-08-17T18:17:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.841914+0000", 2024-08-17T18:17:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2890000000000001e-06 2024-08-17T18:17:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.841937+0000", 2024-08-17T18:17:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3725000000000001e-05 2024-08-17T18:17:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.841950+0000", 2024-08-17T18:17:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3012e-05 2024-08-17T18:17:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.841957+0000", 2024-08-17T18:17:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8140000000000003e-06 2024-08-17T18:17:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.842028+0000", 2024-08-17T18:17:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0962000000000003e-05 2024-08-17T18:17:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.842036+0000", 2024-08-17T18:17:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9049999999999997e-06 2024-08-17T18:17:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647172+0000", 2024-08-17T18:17:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.80513599999999996 2024-08-17T18:17:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647180+0000", 2024-08-17T18:17:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0749999999999998e-06 2024-08-17T18:17:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.902828+0000", 2024-08-17T18:17:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.255647797 2024-08-17T18:17:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.902942+0000", 2024-08-17T18:17:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113795 2024-08-17T18:17:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:17:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.903135+0000", 2024-08-17T18:17:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000193043 2024-08-17T18:17:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907209+0000", 2024-08-17T18:17:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040746159999999997 2024-08-17T18:17:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907255+0000", 2024-08-17T18:17:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5674999999999999e-05 2024-08-17T18:17:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.936682+0000", 2024-08-17T18:17:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.029426833999999999 2024-08-17T18:17:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.936745+0000", 2024-08-17T18:17:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2617e-05 2024-08-17T18:17:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.937735+0000", 2024-08-17T18:17:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00098999000000000005 2024-08-17T18:17:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:92015 57.12 57:4fcb2167:::benchmark_data_smithi136_389235_object92014:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:17:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.850088+0000", 2024-08-17T18:17:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 128.544642764, 2024-08-17T18:17:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0964700700000001, 2024-08-17T18:17:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:17:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:17:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 92015 2024-08-17T18:17:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.850088+0000", 2024-08-17T18:17:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.850088+0000", 2024-08-17T18:17:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.850089+0000", 2024-08-17T18:17:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.063e-06 2024-08-17T18:17:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.850109+0000", 2024-08-17T18:17:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0438999999999999e-05 2024-08-17T18:17:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.850121+0000", 2024-08-17T18:17:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1982e-05 2024-08-17T18:17:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.850128+0000", 2024-08-17T18:17:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8990000000000004e-06 2024-08-17T18:17:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.850188+0000", 2024-08-17T18:17:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0013999999999998e-05 2024-08-17T18:17:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.850197+0000", 2024-08-17T18:17:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9679999999999995e-06 2024-08-17T18:17:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647204+0000", 2024-08-17T18:17:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.79700673499999997 2024-08-17T18:17:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647208+0000", 2024-08-17T18:17:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8940000000000003e-06 2024-08-17T18:17:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.904025+0000", 2024-08-17T18:17:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.256817004 2024-08-17T18:17:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.904124+0000", 2024-08-17T18:17:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9591000000000002e-05 2024-08-17T18:17:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:17:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.904271+0000", 2024-08-17T18:17:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014643300000000001 2024-08-17T18:17:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907318+0000", 2024-08-17T18:17:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0030469889999999999 2024-08-17T18:17:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907332+0000", 2024-08-17T18:17:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4800000000000001e-05 2024-08-17T18:17:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.944953+0000", 2024-08-17T18:17:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0376207 2024-08-17T18:17:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.945020+0000", 2024-08-17T18:17:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6569999999999995e-05 2024-08-17T18:17:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.946558+0000", 2024-08-17T18:17:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001537989 2024-08-17T18:17:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:92043 57.12 57:4c633a12:::benchmark_data_smithi136_389235_object92042:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:17:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.878840+0000", 2024-08-17T18:17:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 128.515890622, 2024-08-17T18:17:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0690841519999998, 2024-08-17T18:17:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:17:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:17:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 92043 2024-08-17T18:17:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.878840+0000", 2024-08-17T18:17:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.878840+0000", 2024-08-17T18:17:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.878842+0000", 2024-08-17T18:17:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.835e-06 2024-08-17T18:17:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.878872+0000", 2024-08-17T18:17:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0314000000000001e-05 2024-08-17T18:17:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.878887+0000", 2024-08-17T18:17:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5167999999999999e-05 2024-08-17T18:17:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.878894+0000", 2024-08-17T18:17:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3470000000000001e-06 2024-08-17T18:17:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.878959+0000", 2024-08-17T18:17:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4885e-05 2024-08-17T18:17:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.878973+0000", 2024-08-17T18:17:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3757000000000001e-05 2024-08-17T18:17:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647236+0000", 2024-08-17T18:17:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.76826348300000002 2024-08-17T18:17:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647244+0000", 2024-08-17T18:17:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.452e-06 2024-08-17T18:17:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.904870+0000", 2024-08-17T18:17:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2576263270000001 2024-08-17T18:17:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.905024+0000", 2024-08-17T18:17:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000153669 2024-08-17T18:17:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:17:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.905261+0000", 2024-08-17T18:17:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023719299999999999 2024-08-17T18:17:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907339+0000", 2024-08-17T18:17:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020783189999999999 2024-08-17T18:17:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907391+0000", 2024-08-17T18:17:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2040999999999999e-05 2024-08-17T18:17:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.946619+0000", 2024-08-17T18:17:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039227646999999997 2024-08-17T18:17:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.946682+0000", 2024-08-17T18:17:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3275999999999995e-05 2024-08-17T18:17:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.947924+0000", 2024-08-17T18:17:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001241439 2024-08-17T18:17:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:92046 57.12 57:49923df3:::benchmark_data_smithi136_389235_object92045:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:17:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.881358+0000", 2024-08-17T18:17:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 128.51337202400001, 2024-08-17T18:17:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0681408019999998, 2024-08-17T18:17:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:17:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:17:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 92046 2024-08-17T18:17:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.881358+0000", 2024-08-17T18:17:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.881358+0000", 2024-08-17T18:17:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.881360+0000", 2024-08-17T18:17:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7710000000000001e-06 2024-08-17T18:17:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.881387+0000", 2024-08-17T18:17:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6526e-05 2024-08-17T18:17:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.881401+0000", 2024-08-17T18:17:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4718e-05 2024-08-17T18:17:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.881410+0000", 2024-08-17T18:17:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8240000000000006e-06 2024-08-17T18:17:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.881519+0000", 2024-08-17T18:17:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109304 2024-08-17T18:17:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.881531+0000", 2024-08-17T18:17:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1226000000000001e-05 2024-08-17T18:17:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647335+0000", 2024-08-17T18:17:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.76580386099999997 2024-08-17T18:17:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647344+0000", 2024-08-17T18:17:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8700000000000004e-06 2024-08-17T18:17:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907434+0000", 2024-08-17T18:17:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.260090033 2024-08-17T18:17:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907581+0000", 2024-08-17T18:17:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014699299999999999 2024-08-17T18:17:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:17:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907826+0000", 2024-08-17T18:17:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024497800000000003 2024-08-17T18:17:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:10.862 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.913576+0000", 2024-08-17T18:17:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0057498929999999998 2024-08-17T18:17:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.913640+0000", 2024-08-17T18:17:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3325999999999996e-05 2024-08-17T18:17:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.948005+0000", 2024-08-17T18:17:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034365035000000002 2024-08-17T18:17:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.948074+0000", 2024-08-17T18:17:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8984000000000002e-05 2024-08-17T18:17:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.949499+0000", 2024-08-17T18:17:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014254599999999999 2024-08-17T18:17:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:92175 57.12 57:494e5fb2:::benchmark_data_smithi136_389235_object92174:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:17:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:01.049817+0000", 2024-08-17T18:17:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 128.34491342600001, 2024-08-17T18:17:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.900916759, 2024-08-17T18:17:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:17:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:17:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 92175 2024-08-17T18:17:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.049817+0000", 2024-08-17T18:17:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.049817+0000", 2024-08-17T18:17:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.049818+0000", 2024-08-17T18:17:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.562e-06 2024-08-17T18:17:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.049836+0000", 2024-08-17T18:17:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8045000000000001e-05 2024-08-17T18:17:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.049849+0000", 2024-08-17T18:17:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2554999999999999e-05 2024-08-17T18:17:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.049855+0000", 2024-08-17T18:17:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3169999999999998e-06 2024-08-17T18:17:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.049917+0000", 2024-08-17T18:17:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2015999999999997e-05 2024-08-17T18:17:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.049929+0000", 2024-08-17T18:17:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1861e-05 2024-08-17T18:17:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647378+0000", 2024-08-17T18:17:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59744883800000004 2024-08-17T18:17:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647385+0000", 2024-08-17T18:17:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0929999999999997e-06 2024-08-17T18:17:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.908719+0000", 2024-08-17T18:17:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2613335079999999 2024-08-17T18:17:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.908865+0000", 2024-08-17T18:17:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014667200000000001 2024-08-17T18:17:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:17:10.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.909091+0000", 2024-08-17T18:17:10.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022598799999999999 2024-08-17T18:17:10.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:10.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.913584+0000", 2024-08-17T18:17:10.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0044922920000000002 2024-08-17T18:17:10.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.913700+0000", 2024-08-17T18:17:10.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011666 2024-08-17T18:17:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.949569+0000", 2024-08-17T18:17:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035869036999999999 2024-08-17T18:17:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.949631+0000", 2024-08-17T18:17:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1266000000000006e-05 2024-08-17T18:17:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.950734+0000", 2024-08-17T18:17:10.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001103049 2024-08-17T18:17:10.872 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.872 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:10.872 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.872 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:10.872 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:10.872 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:17:10.872 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:10.873 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:10 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 47564/266604 objects degraded (17.841%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:10.903 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:10.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:10.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:11.005 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:11.036 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:11.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:11.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:11.138 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:11.159 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:11.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:11.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:11.261 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:11.287 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:11.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:11.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:11.389 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:11.420 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:11.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:11.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:11.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 18:17:11 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=infra.usagestats t=2024-08-17T18:17:11.060257118Z level=info msg="Usage stats are ready to report" 2024-08-17T18:17:11.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:11.544 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:11.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:11.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:11.585 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:17:11.646 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:11.681 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:11.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:11.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:11.685 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-17T18:17:11.784 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:11.814 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:11.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:11.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:11.915 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:11.937 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:11.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:11.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:11.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:11 smithi136 ceph-mon[115078]: pgmap v3141: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.4 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 279 op/s; 47564/266604 objects degraded (17.841%); 30363/266604 objects misplaced (11.389%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T18:17:11.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:11 smithi136 ceph-mon[115078]: 1.0 scrub starts 2024-08-17T18:17:11.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:11 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2536416089' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:17:11.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:11 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:17:11.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:11 smithi136 ceph-mon[109861]: pgmap v3141: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.4 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 279 op/s; 47564/266604 objects degraded (17.841%); 30363/266604 objects misplaced (11.389%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T18:17:11.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:11 smithi136 ceph-mon[109861]: 1.0 scrub starts 2024-08-17T18:17:11.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:11 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2536416089' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:17:11.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:11 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:17:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:11 smithi165 ceph-mon[105517]: pgmap v3141: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.4 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 0 B/s wr, 279 op/s; 47564/266604 objects degraded (17.841%); 30363/266604 objects misplaced (11.389%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T18:17:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:11 smithi165 ceph-mon[105517]: 1.0 scrub starts 2024-08-17T18:17:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:11 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2536416089' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:17:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:11 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:17:12.040 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:12.061 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:12.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:12.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:12.162 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:12.759 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:12 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:17:12.759 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:12 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:17:12.759 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:12 smithi136 ceph-mon[109861]: osdmap e1651: 8 total, 8 up, 6 in 2024-08-17T18:17:12.759 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:12 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:17:12.759 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:12 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:17:12.760 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:12 smithi136 ceph-mon[115078]: osdmap e1651: 8 total, 8 up, 6 in 2024-08-17T18:17:12.785 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:12.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:12.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:12.815 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:17:12.865 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:17:12.872 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:17:12.887 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:12.911 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:12.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:12.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:12 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:17:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:12 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:17:13.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:12 smithi165 ceph-mon[105517]: osdmap e1651: 8 total, 8 up, 6 in 2024-08-17T18:17:13.013 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:13.037 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:13.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:13.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:13.140 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:13.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:13.164 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:13.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:13.266 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:13.290 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:13.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:13.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:13.392 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:13.422 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:13.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:13.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:13.524 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:13.549 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:13.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:13.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:13.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:13.674 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:13.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:13.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:13.776 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:13.797 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:13.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:13.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:13.852 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:13 smithi136 ceph-mon[109861]: pgmap v3143: 109 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.3 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 296 op/s; 47453/262176 objects degraded (18.100%); 29999/262176 objects misplaced (11.442%); 1.6 MiB/s, 25 objects/s recovering 2024-08-17T18:17:13.852 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:13 smithi136 ceph-mon[109861]: 1.0 deep-scrub starts 2024-08-17T18:17:13.852 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:13 smithi136 ceph-mon[115078]: pgmap v3143: 109 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.3 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 296 op/s; 47453/262176 objects degraded (18.100%); 29999/262176 objects misplaced (11.442%); 1.6 MiB/s, 25 objects/s recovering 2024-08-17T18:17:13.852 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:13 smithi136 ceph-mon[115078]: 1.0 deep-scrub starts 2024-08-17T18:17:13.900 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:13.926 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:13.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:13.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:13 smithi165 ceph-mon[105517]: pgmap v3143: 109 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.3 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 296 op/s; 47453/262176 objects degraded (18.100%); 29999/262176 objects misplaced (11.442%); 1.6 MiB/s, 25 objects/s recovering 2024-08-17T18:17:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:13 smithi165 ceph-mon[105517]: 1.0 deep-scrub starts 2024-08-17T18:17:14.027 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:14.050 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:14.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:14.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:14.151 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:14.180 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:14.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:14.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:14.282 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:14.304 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:14.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:14.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:14.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:14.431 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:14.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:14.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:14.533 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:14.566 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:14.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:14.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:14.668 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:14.690 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:14.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:14.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:14.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:17:14.792 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:14.830 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:14.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:14.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:14.932 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:14.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:14 smithi136 ceph-mon[115078]: 1.0 scrub starts 2024-08-17T18:17:14.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:14 smithi136 ceph-mon[109861]: 1.0 scrub starts 2024-08-17T18:17:14.952 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:14.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:14.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:15.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:14 smithi165 ceph-mon[105517]: 1.0 scrub starts 2024-08-17T18:17:15.054 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:15.077 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:15.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:15.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:15.179 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:15.209 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:15.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:15.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:15.311 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:15.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:15.342 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:15.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:15.443 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:15.466 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:15.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:15.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:15.567 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:15.588 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:15.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:15.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:15.690 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:15.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:15.722 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:15.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:15.823 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:15.844 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:15 smithi136 ceph-mon[109861]: pgmap v3144: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.3 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 326 op/s; 47453/261258 objects degraded (18.163%); 29769/261258 objects misplaced (11.394%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T18:17:15.844 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:15 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 47453/261258 objects degraded (18.163%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:15.845 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:15 smithi136 ceph-mon[109861]: 1.0 scrub starts 2024-08-17T18:17:15.845 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:15 smithi136 ceph-mon[115078]: pgmap v3144: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.3 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 326 op/s; 47453/261258 objects degraded (18.163%); 29769/261258 objects misplaced (11.394%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T18:17:15.845 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:15 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 47453/261258 objects degraded (18.163%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:15.845 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:15 smithi136 ceph-mon[115078]: 1.0 scrub starts 2024-08-17T18:17:15.845 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:15.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:15.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:15.948 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:15.975 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:15.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:15.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:15 smithi165 ceph-mon[105517]: pgmap v3144: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.3 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 326 op/s; 47453/261258 objects degraded (18.163%); 29769/261258 objects misplaced (11.394%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T18:17:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:15 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 47453/261258 objects degraded (18.163%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:15 smithi165 ceph-mon[105517]: 1.0 scrub starts 2024-08-17T18:17:16.078 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:16.107 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:16.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:16.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:16.209 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:16.236 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:16.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:16.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:16.338 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:16.369 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:16.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:16.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:16.471 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:16.497 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:16.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:16.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:16.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:16.621 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:16.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:16.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:16.723 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:16.751 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:16.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:16.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:16.853 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:16.876 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:16.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:16.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:16.978 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:17.000 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:17.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:17.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:17.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:17.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:17.130 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:17.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:17.231 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:17.255 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:17.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:17.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:17.342 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:17:17.342 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:17:17.356 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:17.381 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:17.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:17.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:17.484 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:17.505 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:17.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:17.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:17.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:17.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:17.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:17.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:17.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:17:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:17:17] "GET /metrics HTTP/1.1" 200 36732 "" "Prometheus/2.51.0" 2024-08-17T18:17:17.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:17 smithi136 ceph-mon[109861]: pgmap v3145: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.3 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 328 op/s; 47453/261075 objects degraded (18.176%); 29647/261075 objects misplaced (11.356%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T18:17:17.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:17 smithi136 ceph-mon[109861]: 1.0 scrub starts 2024-08-17T18:17:17.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:17 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2358879316' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:17:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:17 smithi136 ceph-mon[115078]: pgmap v3145: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.3 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 328 op/s; 47453/261075 objects degraded (18.176%); 29647/261075 objects misplaced (11.356%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T18:17:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:17 smithi136 ceph-mon[115078]: 1.0 scrub starts 2024-08-17T18:17:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:17 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2358879316' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:17:17.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:17.754 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:17.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:17.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:17.856 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:17.880 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:17.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:17.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:17.982 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:18.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:17 smithi165 ceph-mon[105517]: pgmap v3145: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.3 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 328 op/s; 47453/261075 objects degraded (18.176%); 29647/261075 objects misplaced (11.356%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T18:17:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:17 smithi165 ceph-mon[105517]: 1.0 scrub starts 2024-08-17T18:17:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:17 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2358879316' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:17:18.014 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:18.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:18.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:18.050 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:17:18.051 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:17:18.051 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:17:18.051 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:17:18.051 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:17:18.051 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:17:18.051 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:17:18.051 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:17:18.051 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:17:18.051 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:17:18.052 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:17:18.052 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:17:18.052 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:17:18.052 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:17:18.052 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:17:18.052 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:17:18.115 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:18.143 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:18.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:18.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:18.245 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:18.276 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:18.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:18.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:18.378 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:18.401 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:18.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:18.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:18.503 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:18.529 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:18.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:18.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:18.631 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:18.655 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:18.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:18.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:18.758 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:18.789 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:18.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:18.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:18.895 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:18.924 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:18.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:18.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:19.026 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:19.049 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:19.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:19.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:19.151 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:19.172 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:19.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:19.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:19.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:19.300 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:19.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:19.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:19.401 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:19.423 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:19.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:19.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:19.525 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:19.547 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:19.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:19.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:19.649 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:19.674 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:19.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:19.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:19.690 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:19 smithi136 ceph-mon[109861]: pgmap v3146: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.2 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 383 op/s; 47314/255093 objects degraded (18.548%); 29054/255093 objects misplaced (11.390%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T18:17:19.691 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:19 smithi136 ceph-mon[115078]: pgmap v3146: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.2 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 383 op/s; 47314/255093 objects degraded (18.548%); 29054/255093 objects misplaced (11.390%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T18:17:19.776 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:19.798 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:19.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:19.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:19.900 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:19.923 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:19.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:19.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:19 smithi165 ceph-mon[105517]: pgmap v3146: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.2 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 383 op/s; 47314/255093 objects degraded (18.548%); 29054/255093 objects misplaced (11.390%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T18:17:20.026 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:20.047 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:20.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:20.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:20.136 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:17:20.140 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-17T18:17:20.149 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:20.175 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:20.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:20.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:20.279 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:20.304 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:20.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:20.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:20.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:20.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:20.431 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:20.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:20.533 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:20.554 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:20.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:20.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:20.656 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:20.679 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:20.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:20.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:20.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:20 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 47314/255093 objects degraded (18.548%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:20.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:20 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 47314/255093 objects degraded (18.548%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:20.781 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:20.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:20.805 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:20.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:20.888 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:20 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 47314/255093 objects degraded (18.548%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:20.906 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:20.930 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:20.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:20.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:21.032 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:21.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:21.055 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:21.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:21.108 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:17:21.157 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:21.182 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:21.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:21.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:21.284 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:21.305 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:21.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:21.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:21.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:21.431 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:21.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:21.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:21.532 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:21.554 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:21.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:21.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:21.606 INFO:teuthology.orchestra.run.smithi136.stdout:moved 33 / 327 (10.0917%) 2024-08-17T18:17:21.606 INFO:teuthology.orchestra.run.smithi136.stdout:avg 54.5 2024-08-17T18:17:21.606 INFO:teuthology.orchestra.run.smithi136.stdout:stddev 22.6918 -> 21.7543 (expected baseline 6.73919) 2024-08-17T18:17:21.607 INFO:teuthology.orchestra.run.smithi136.stdout:min osd.0 with 6 -> 8 pgs (0.110092 -> 0.146789 * mean) 2024-08-17T18:17:21.607 INFO:teuthology.orchestra.run.smithi136.stdout:max osd.6 with 73 -> 56 pgs (1.33945 -> 1.02752 * mean) 2024-08-17T18:17:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:17:21.607 INFO:teuthology.orchestra.run.smithi136.stdout:oload 110 2024-08-17T18:17:21.607 INFO:teuthology.orchestra.run.smithi136.stdout:max_change 3 2024-08-17T18:17:21.607 INFO:teuthology.orchestra.run.smithi136.stdout:max_change_osds 4 2024-08-17T18:17:21.607 INFO:teuthology.orchestra.run.smithi136.stdout:average_utilization 0.0417 2024-08-17T18:17:21.607 INFO:teuthology.orchestra.run.smithi136.stdout:overload_utilization 0.0459 2024-08-17T18:17:21.607 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6 weight 1.0000 -> 0.7538 2024-08-17T18:17:21.607 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2 weight 1.0000 -> 0.8569 2024-08-17T18:17:21.607 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0 weight 0.1617 -> 0.1391 2024-08-17T18:17:21.607 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5 weight 1.0000 -> 0.8915 2024-08-17T18:17:21.656 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:21.678 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:21.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:21.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:21.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:21.799 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:21 smithi136 ceph-mon[109861]: pgmap v3147: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.2 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 383 op/s; 47314/255093 objects degraded (18.548%); 29054/255093 objects misplaced (11.390%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T18:17:21.799 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'9114','2':'56161','5':'58428','6':'49402'}"} : dispatch 2024-08-17T18:17:21.799 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:17:21.799 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:17:21.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:21 smithi136 ceph-mon[115078]: pgmap v3147: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.2 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 383 op/s; 47314/255093 objects degraded (18.548%); 29054/255093 objects misplaced (11.390%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T18:17:21.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'9114','2':'56161','5':'58428','6':'49402'}"} : dispatch 2024-08-17T18:17:21.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:17:21.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:17:21.803 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:21.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:21.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:21.905 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:21.926 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:21.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:21.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:22.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:21 smithi165 ceph-mon[105517]: pgmap v3147: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.2 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 383 op/s; 47314/255093 objects degraded (18.548%); 29054/255093 objects misplaced (11.390%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T18:17:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'9114','2':'56161','5':'58428','6':'49402'}"} : dispatch 2024-08-17T18:17:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:17:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:17:22.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:22.050 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:22.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:22.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:22.152 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:22.173 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:22.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:22.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:22.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:22.296 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:22.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:22.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:22.398 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:22.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:22.421 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:22.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:22.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:22.545 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:22.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:22.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:22.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:22.672 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:22.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:22.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:22.774 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:22.799 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:22.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:22.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:22.816 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:22 smithi136 ceph-mon[109861]: from='client.46508 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:17:22.816 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'9114','2':'56161','5':'58428','6':'49402'}"}]': finished 2024-08-17T18:17:22.816 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:22 smithi136 ceph-mon[109861]: osdmap e1652: 8 total, 8 up, 6 in 2024-08-17T18:17:22.817 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:22 smithi136 ceph-mon[115078]: from='client.46508 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:17:22.817 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'9114','2':'56161','5':'58428','6':'49402'}"}]': finished 2024-08-17T18:17:22.817 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:22 smithi136 ceph-mon[115078]: osdmap e1652: 8 total, 8 up, 6 in 2024-08-17T18:17:22.901 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:22.924 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:22.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:22.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:22 smithi165 ceph-mon[105517]: from='client.46508 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:17:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'9114','2':'56161','5':'58428','6':'49402'}"}]': finished 2024-08-17T18:17:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:22 smithi165 ceph-mon[105517]: osdmap e1652: 8 total, 8 up, 6 in 2024-08-17T18:17:23.026 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:23.048 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:23.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:23.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:23.150 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:23.173 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:23.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:23.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:23.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:23.297 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:23.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:23.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:23.398 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:23.422 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:23.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:23.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:23.488 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:17:23.488 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:17:23.524 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:23.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:23.547 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:23.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:23.625 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:17:23.649 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:23.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:23.703 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:23.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:23.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:23.845 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:23.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:23.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:23.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:23 smithi136 ceph-mon[109861]: pgmap v3149: 109 pgs: 7 remapped+peering, 11 peering, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 unknown, 75 active+clean, 7 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 351 op/s; 23254/251631 objects degraded (9.241%); 28577/251631 objects misplaced (11.357%); 2.1 MiB/s, 34 objects/s recovering 2024-08-17T18:17:23.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:23 smithi136 ceph-mon[109861]: Health check failed: Reduced data availability: 4 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-08-17T18:17:23.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:23 smithi136 ceph-mon[109861]: osdmap e1653: 8 total, 8 up, 6 in 2024-08-17T18:17:23.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:23 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2046733872' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:17:23.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:23 smithi136 ceph-mon[115078]: pgmap v3149: 109 pgs: 7 remapped+peering, 11 peering, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 unknown, 75 active+clean, 7 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 351 op/s; 23254/251631 objects degraded (9.241%); 28577/251631 objects misplaced (11.357%); 2.1 MiB/s, 34 objects/s recovering 2024-08-17T18:17:23.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:23 smithi136 ceph-mon[115078]: Health check failed: Reduced data availability: 4 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-08-17T18:17:23.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:23 smithi136 ceph-mon[115078]: osdmap e1653: 8 total, 8 up, 6 in 2024-08-17T18:17:23.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:23 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2046733872' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:17:23.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:23.985 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:23.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:23.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:23 smithi165 ceph-mon[105517]: pgmap v3149: 109 pgs: 7 remapped+peering, 11 peering, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 unknown, 75 active+clean, 7 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 351 op/s; 23254/251631 objects degraded (9.241%); 28577/251631 objects misplaced (11.357%); 2.1 MiB/s, 34 objects/s recovering 2024-08-17T18:17:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:23 smithi165 ceph-mon[105517]: Health check failed: Reduced data availability: 4 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-08-17T18:17:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:23 smithi165 ceph-mon[105517]: osdmap e1653: 8 total, 8 up, 6 in 2024-08-17T18:17:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:23 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2046733872' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:17:24.088 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:24.110 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:24.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:24.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:24.212 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:24.250 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:24.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:24.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:24.268 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-08-17T18:17:24.269 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-08-17T18:17:24.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:24.388 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:24.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:24.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:24.490 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:24.512 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:24.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:24.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:24.613 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:24.657 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:24.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:24.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:24.759 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:24.777 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:24 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:17:24.777 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:24 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2046733872' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:17:24.778 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:24 smithi136 ceph-mon[109861]: osdmap e1654: 8 total, 8 up, 6 in 2024-08-17T18:17:24.778 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:24 smithi136 ceph-mon[109861]: 57.11 continuing backfill to osd.4 from (1611'100,1651'7245] MIN to 1651'7245 2024-08-17T18:17:24.778 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:24 smithi136 ceph-mon[109861]: 57.11 continuing backfill to osd.5 from (1611'100,1651'7245] MIN to 1651'7245 2024-08-17T18:17:24.778 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:24 smithi136 ceph-mon[109861]: 57.d starting backfill to osd.4 from (0'0,0'0] MAX to 1651'9233 2024-08-17T18:17:24.778 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:24 smithi136 ceph-mon[109861]: 57.e continuing backfill to osd.7 from (1614'4769,1651'9327] MIN to 1651'9327 2024-08-17T18:17:24.778 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:24 smithi136 ceph-mon[109861]: 57.5 continuing backfill to osd.7 from (1613'1578,1651'7183] MIN to 1651'7183 2024-08-17T18:17:24.778 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:24 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:17:24.779 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:24 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2046733872' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:17:24.779 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:24 smithi136 ceph-mon[115078]: osdmap e1654: 8 total, 8 up, 6 in 2024-08-17T18:17:24.779 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:24 smithi136 ceph-mon[115078]: 57.11 continuing backfill to osd.4 from (1611'100,1651'7245] MIN to 1651'7245 2024-08-17T18:17:24.779 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:24 smithi136 ceph-mon[115078]: 57.11 continuing backfill to osd.5 from (1611'100,1651'7245] MIN to 1651'7245 2024-08-17T18:17:24.779 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:24 smithi136 ceph-mon[115078]: 57.d starting backfill to osd.4 from (0'0,0'0] MAX to 1651'9233 2024-08-17T18:17:24.779 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:24 smithi136 ceph-mon[115078]: 57.e continuing backfill to osd.7 from (1614'4769,1651'9327] MIN to 1651'9327 2024-08-17T18:17:24.780 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:24 smithi136 ceph-mon[115078]: 57.5 continuing backfill to osd.7 from (1613'1578,1651'7183] MIN to 1651'7183 2024-08-17T18:17:24.786 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:24.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:24.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:24.888 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:24.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:24.924 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:24.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:25.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:24 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:17:25.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:24 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2046733872' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:17:25.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:24 smithi165 ceph-mon[105517]: osdmap e1654: 8 total, 8 up, 6 in 2024-08-17T18:17:25.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:24 smithi165 ceph-mon[105517]: 57.11 continuing backfill to osd.4 from (1611'100,1651'7245] MIN to 1651'7245 2024-08-17T18:17:25.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:24 smithi165 ceph-mon[105517]: 57.11 continuing backfill to osd.5 from (1611'100,1651'7245] MIN to 1651'7245 2024-08-17T18:17:25.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:24 smithi165 ceph-mon[105517]: 57.d starting backfill to osd.4 from (0'0,0'0] MAX to 1651'9233 2024-08-17T18:17:25.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:24 smithi165 ceph-mon[105517]: 57.e continuing backfill to osd.7 from (1614'4769,1651'9327] MIN to 1651'9327 2024-08-17T18:17:25.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:24 smithi165 ceph-mon[105517]: 57.5 continuing backfill to osd.7 from (1613'1578,1651'7183] MIN to 1651'7183 2024-08-17T18:17:25.025 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:25.047 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:25.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:25.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:25.149 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:25.184 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:25.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:25.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:25.285 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:25.308 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:25.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:25.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:25.409 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:25.442 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:25.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:25.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:25.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:25.579 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:25.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:25.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:25.680 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:25.712 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:25.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:25.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:25.814 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:25.820 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:25 smithi136 ceph-mon[109861]: pgmap v3152: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 8 remapped+peering, 20 peering, 1 active+recovery_wait, 1 active+recovery_wait+degraded+remapped, 2 unknown, 65 active+clean, 7 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 5.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 12 KiB/s rd, 37 KiB/s wr, 243 op/s; 23254/250875 objects degraded (9.269%); 23128/250875 objects misplaced (9.219%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T18:17:25.821 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:25 smithi136 ceph-mon[109861]: osdmap e1655: 8 total, 8 up, 6 in 2024-08-17T18:17:25.821 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:25 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 23254/250875 objects degraded (9.269%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:25.821 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:25 smithi136 ceph-mon[115078]: pgmap v3152: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 8 remapped+peering, 20 peering, 1 active+recovery_wait, 1 active+recovery_wait+degraded+remapped, 2 unknown, 65 active+clean, 7 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 5.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 12 KiB/s rd, 37 KiB/s wr, 243 op/s; 23254/250875 objects degraded (9.269%); 23128/250875 objects misplaced (9.219%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T18:17:25.821 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:25 smithi136 ceph-mon[115078]: osdmap e1655: 8 total, 8 up, 6 in 2024-08-17T18:17:25.821 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:25 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 23254/250875 objects degraded (9.269%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:25.839 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:25.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:25.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:25.941 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:25.962 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:25.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:25.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:25 smithi165 ceph-mon[105517]: pgmap v3152: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 8 remapped+peering, 20 peering, 1 active+recovery_wait, 1 active+recovery_wait+degraded+remapped, 2 unknown, 65 active+clean, 7 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait; 5.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 12 KiB/s rd, 37 KiB/s wr, 243 op/s; 23254/250875 objects degraded (9.269%); 23128/250875 objects misplaced (9.219%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T18:17:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:25 smithi165 ceph-mon[105517]: osdmap e1655: 8 total, 8 up, 6 in 2024-08-17T18:17:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:25 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 23254/250875 objects degraded (9.269%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:26.064 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:26.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:26.098 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:26.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:26.199 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:26.221 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:26.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:26.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:26.323 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:26.344 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:26.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:26.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:26.446 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:26.467 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:26.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:26.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:26.569 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:26.592 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:26.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:26.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:26.694 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:26.728 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:26.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:26.830 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:26.860 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:26.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:26.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:26.930 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:26 smithi136 ceph-mon[115078]: osdmap e1656: 8 total, 8 up, 6 in 2024-08-17T18:17:26.930 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:26 smithi136 ceph-mon[109861]: osdmap e1656: 8 total, 8 up, 6 in 2024-08-17T18:17:26.962 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:26.987 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:26.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:26.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:27.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:26 smithi165 ceph-mon[105517]: osdmap e1656: 8 total, 8 up, 6 in 2024-08-17T18:17:27.089 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:27.122 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:27.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:27.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:27.224 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:27.246 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:27.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:27.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:27.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:27.376 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:27.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:27.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:27.478 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:27.505 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:27.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:27.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:27.561 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:17:27.586 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:17:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:17:27] "GET /metrics HTTP/1.1" 200 36733 "" "Prometheus/2.51.0" 2024-08-17T18:17:27.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:27.636 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:27.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:27.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:27.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:27.764 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:27.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:27.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:27.866 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:27.888 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:27.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:27.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:27.935 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:27 smithi136 ceph-mon[109861]: pgmap v3155: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 8 remapped+peering, 20 peering, 1 active+recovery_wait, 1 active+recovery_wait+degraded+remapped, 2 unknown, 64 active+clean, 7 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 19 KiB/s rd, 55 KiB/s wr, 132 op/s; 23254/250203 objects degraded (9.294%); 28217/250203 objects misplaced (11.278%); 1.1 MiB/s, 17 objects/s recovering 2024-08-17T18:17:27.935 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:27 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:17:27.935 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:27 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:17:27.936 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:27 smithi136 ceph-mon[115078]: pgmap v3155: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 8 remapped+peering, 20 peering, 1 active+recovery_wait, 1 active+recovery_wait+degraded+remapped, 2 unknown, 64 active+clean, 7 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 19 KiB/s rd, 55 KiB/s wr, 132 op/s; 23254/250203 objects degraded (9.294%); 28217/250203 objects misplaced (11.278%); 1.1 MiB/s, 17 objects/s recovering 2024-08-17T18:17:27.936 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:27 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:17:27.936 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:27 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:17:27.990 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:28.013 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:28.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:28.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:28.115 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:28.140 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:28.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:28.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:28.242 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:28.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:27 smithi165 ceph-mon[105517]: pgmap v3155: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 8 remapped+peering, 20 peering, 1 active+recovery_wait, 1 active+recovery_wait+degraded+remapped, 2 unknown, 64 active+clean, 7 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 5.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 19 KiB/s rd, 55 KiB/s wr, 132 op/s; 23254/250203 objects degraded (9.294%); 28217/250203 objects misplaced (11.278%); 1.1 MiB/s, 17 objects/s recovering 2024-08-17T18:17:28.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:27 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:17:28.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:27 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:17:28.265 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:28.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:28.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:28.368 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:28.401 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:28.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:28.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:28.503 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:28.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:28.530 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:28.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:28.631 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:28.653 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:28.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:28.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:28.755 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:28.777 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:28.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:28.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:28.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:28.901 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:28.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:28.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:29.003 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:29.026 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:29.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:29.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:29.116 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:28 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:17:29.116 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:28 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:17:29.116 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:28 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:17:29.116 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:28 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:17:29.116 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:28 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:17:29.117 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:28 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:17:29.128 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:29.149 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:29.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:29.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:29.226 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:17:29.226 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:17:29.252 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:29.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:28 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:17:29.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:28 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:17:29.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:28 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:17:29.273 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:29.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:29.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:29.375 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:29.401 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:29.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:29.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:29.503 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:29.532 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:29.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:29.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:29.634 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:29.663 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:29.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:29.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:29.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:29.788 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:29.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:29.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:29.890 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:29.919 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:29.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:29.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:30.021 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:30.044 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:30.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:30.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:30.146 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:30.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:30.171 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:30.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:29 smithi136 ceph-mon[109861]: pgmap v3156: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 5.0 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 251 op/s; 55800/246471 objects degraded (22.640%); 33530/246471 objects misplaced (13.604%); 7.0 MiB/s, 114 objects/s recovering 2024-08-17T18:17:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:29 smithi136 ceph-mon[109861]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 6 pgs peering) 2024-08-17T18:17:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:29 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 55800/246471 objects degraded (22.640%), 17 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:30.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:29 smithi136 ceph-mon[115078]: pgmap v3156: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 5.0 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 251 op/s; 55800/246471 objects degraded (22.640%); 33530/246471 objects misplaced (13.604%); 7.0 MiB/s, 114 objects/s recovering 2024-08-17T18:17:30.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:29 smithi136 ceph-mon[115078]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 6 pgs peering) 2024-08-17T18:17:30.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:29 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 55800/246471 objects degraded (22.640%), 17 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:29 smithi165 ceph-mon[105517]: pgmap v3156: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 5.0 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 251 op/s; 55800/246471 objects degraded (22.640%); 33530/246471 objects misplaced (13.604%); 7.0 MiB/s, 114 objects/s recovering 2024-08-17T18:17:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:29 smithi165 ceph-mon[105517]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 6 pgs peering) 2024-08-17T18:17:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:29 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 55800/246471 objects degraded (22.640%), 17 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:30.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:30.300 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:30.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:30.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:30.402 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:30.421 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:17:30.422 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:17:30.425 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:30.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:30.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:30.441 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:17:30.441 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:17:30.445 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:17:30.460 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:17:30.460 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:17:30.478 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:17:30.479 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:17:30.483 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:17:30.528 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:30.549 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:30.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:30.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:30.617 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:17:30.617 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:17:30.640 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:17:30.641 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:17:30.655 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:30.658 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:17:30.659 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:17:30.679 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:17:30.679 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:17:30.687 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:30.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:30.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:30.789 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:30.820 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:30.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:30.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:30.922 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:30.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:30 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1730428704' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:17:30.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:30 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1730428704' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:17:30.954 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:30.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:30.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:31.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:31.080 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:31.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:31.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:31.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:31.211 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:31.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:31.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:30 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1730428704' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:17:31.314 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:31.336 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:31.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:31.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:31.438 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:31.462 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:31.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:31.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:31.564 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:31.586 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:31.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:31.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:31.688 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:31.709 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:31.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:31.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:31.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:31.836 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:31.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:31.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:31.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:31.970 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:31.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:31.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:32.042 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:31 smithi136 ceph-mon[109861]: pgmap v3157: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 5.0 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 214 op/s; 55800/246471 objects degraded (22.640%); 33530/246471 objects misplaced (13.604%); 6.0 MiB/s, 97 objects/s recovering 2024-08-17T18:17:32.043 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:31 smithi136 ceph-mon[115078]: pgmap v3157: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 5.0 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 214 op/s; 55800/246471 objects degraded (22.640%); 33530/246471 objects misplaced (13.604%); 6.0 MiB/s, 97 objects/s recovering 2024-08-17T18:17:32.072 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:32.104 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:32.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:32.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:32.206 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:32.229 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:32.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:32.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:31 smithi165 ceph-mon[105517]: pgmap v3157: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 5.0 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 0 B/s wr, 214 op/s; 55800/246471 objects degraded (22.640%); 33530/246471 objects misplaced (13.604%); 6.0 MiB/s, 97 objects/s recovering 2024-08-17T18:17:32.331 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:32.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:32.367 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:32.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:32.469 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:32.494 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:32.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:32.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:32.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:32.618 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:32.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:32.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:32.720 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:32.749 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:32.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:32.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:32.851 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:32.876 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:32.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:32.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:32.978 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:33.007 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:33.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:33.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:33.108 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:33.140 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:33.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:33.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:33.159 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-17T18:17:33.242 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:33.270 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:33.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:33.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:33.372 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:33.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:33.394 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:33.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:33.495 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:33.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:17:33.517 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:33.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:33.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:33.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:33.642 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:33.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:33.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:33.744 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:33.767 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:33.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:33.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:33.869 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:33.891 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:33.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:33.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:33.993 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:34.019 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:34.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:34.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:34.117 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:33 smithi136 ceph-mon[109861]: pgmap v3158: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.9 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 349 op/s; 55363/242484 objects degraded (22.832%); 32623/242484 objects misplaced (13.454%); 11 MiB/s, 180 objects/s recovering 2024-08-17T18:17:34.117 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:33 smithi136 ceph-mon[115078]: pgmap v3158: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.9 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 349 op/s; 55363/242484 objects degraded (22.832%); 32623/242484 objects misplaced (13.454%); 11 MiB/s, 180 objects/s recovering 2024-08-17T18:17:34.121 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:34.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:34.147 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:34.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:34.248 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:34.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:33 smithi165 ceph-mon[105517]: pgmap v3158: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.9 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 349 op/s; 55363/242484 objects degraded (22.832%); 32623/242484 objects misplaced (13.454%); 11 MiB/s, 180 objects/s recovering 2024-08-17T18:17:34.270 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:34.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:34.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:34.372 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:34.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:34.403 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:34.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:34.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:34.527 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:34.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:34.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:34.629 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:34.660 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:34.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:34.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:34.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:34.789 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:34.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:34.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:34.818 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:17:34.890 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:34.926 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:34.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:34.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:35.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:35.063 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:35.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:35.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:35.164 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:35.191 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:35.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:35.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:35.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:34 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/821989367' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:17:35.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:34 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 55363/242484 objects degraded (22.832%), 17 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:35.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:34 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/821989367' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:17:35.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:34 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 55363/242484 objects degraded (22.832%), 17 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:34 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/821989367' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:17:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:34 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 55363/242484 objects degraded (22.832%), 17 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:35.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:35.327 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:35.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:35.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:35.429 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:35.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:35.472 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:35.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:35.573 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:35.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:35.610 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:35.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:35.711 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:35.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:35.768 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:35.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:35.869 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:35.901 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:35.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:35.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:35.997 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:35 smithi136 ceph-mon[109861]: pgmap v3159: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.9 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 308 op/s; 55246/241797 objects degraded (22.848%); 32526/241797 objects misplaced (13.452%); 9.5 MiB/s, 152 objects/s recovering 2024-08-17T18:17:35.997 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:35 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:17:35.997 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:35 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/821989367' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:17:35.997 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:35 smithi136 ceph-mon[109861]: osdmap e1657: 8 total, 8 up, 6 in 2024-08-17T18:17:35.998 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:35 smithi136 ceph-mon[115078]: pgmap v3159: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.9 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 308 op/s; 55246/241797 objects degraded (22.848%); 32526/241797 objects misplaced (13.452%); 9.5 MiB/s, 152 objects/s recovering 2024-08-17T18:17:35.998 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:35 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:17:35.998 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:35 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/821989367' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:17:35.998 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:35 smithi136 ceph-mon[115078]: osdmap e1657: 8 total, 8 up, 6 in 2024-08-17T18:17:36.003 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:36.028 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:36.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:36.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:36.130 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:36.158 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:36.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:36.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:36.241 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:17:36.242 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:17:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:35 smithi165 ceph-mon[105517]: pgmap v3159: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.9 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 308 op/s; 55246/241797 objects degraded (22.848%); 32526/241797 objects misplaced (13.452%); 9.5 MiB/s, 152 objects/s recovering 2024-08-17T18:17:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:35 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:17:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:35 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/821989367' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:17:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:35 smithi165 ceph-mon[105517]: osdmap e1657: 8 total, 8 up, 6 in 2024-08-17T18:17:36.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:36.309 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:36.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:36.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:36.411 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:36.440 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:36.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:36.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:36.461 INFO:teuthology.orchestra.run.smithi136.stdout:moved 0 / 327 (0%) 2024-08-17T18:17:36.462 INFO:teuthology.orchestra.run.smithi136.stdout:avg 54.5 2024-08-17T18:17:36.462 INFO:teuthology.orchestra.run.smithi136.stdout:stddev 21.7543 -> 21.7543 (expected baseline 6.73919) 2024-08-17T18:17:36.462 INFO:teuthology.orchestra.run.smithi136.stdout:min osd.0 with 8 -> 8 pgs (0.146789 -> 0.146789 * mean) 2024-08-17T18:17:36.462 INFO:teuthology.orchestra.run.smithi136.stdout:max osd.7 with 74 -> 74 pgs (1.3578 -> 1.3578 * mean) 2024-08-17T18:17:36.462 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:17:36.462 INFO:teuthology.orchestra.run.smithi136.stdout:oload 1000 2024-08-17T18:17:36.462 INFO:teuthology.orchestra.run.smithi136.stdout:max_change 0.05 2024-08-17T18:17:36.462 INFO:teuthology.orchestra.run.smithi136.stdout:max_change_osds 4 2024-08-17T18:17:36.462 INFO:teuthology.orchestra.run.smithi136.stdout:average_utilization 0.0404 2024-08-17T18:17:36.462 INFO:teuthology.orchestra.run.smithi136.stdout:overload_utilization 0.4044 2024-08-17T18:17:36.463 INFO:teuthology.orchestra.run.smithi136.stderr:no change 2024-08-17T18:17:36.542 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:36.584 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:36.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:36.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:36.686 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:36.712 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:36.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:36.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:36.813 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:36.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:36.836 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:36.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:36.937 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:36.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:36.963 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:36.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:37.064 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:37.096 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:37.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:37.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:37.198 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:37.220 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:37.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:37.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:37.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:37.380 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:37.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:37.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:37.481 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:37.487 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:17:37.487 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:17:37.487 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:37 smithi136 ceph-mon[109861]: pgmap v3161: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.9 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 296 op/s; 55246/241299 objects degraded (22.895%); 32304/241299 objects misplaced (13.388%); 9.3 MiB/s, 149 objects/s recovering 2024-08-17T18:17:37.487 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:37 smithi136 ceph-mon[109861]: from='client.36663 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:17:37.487 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:17:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:17:37] "GET /metrics HTTP/1.1" 200 36777 "" "Prometheus/2.51.0" 2024-08-17T18:17:37.488 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:17:37.488 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:17:37.488 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:37 smithi136 ceph-mon[115078]: pgmap v3161: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.9 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 296 op/s; 55246/241299 objects degraded (22.895%); 32304/241299 objects misplaced (13.388%); 9.3 MiB/s, 149 objects/s recovering 2024-08-17T18:17:37.488 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:37 smithi136 ceph-mon[115078]: from='client.36663 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:17:37.505 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:37.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:37.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:37.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:17:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:17:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:37 smithi165 ceph-mon[105517]: pgmap v3161: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.9 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 296 op/s; 55246/241299 objects degraded (22.895%); 32304/241299 objects misplaced (13.388%); 9.3 MiB/s, 149 objects/s recovering 2024-08-17T18:17:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:37 smithi165 ceph-mon[105517]: from='client.36663 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:17:37.514 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-08-17T18:17:37.514 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-08-17T18:17:37.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:37.634 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:37.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:37.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:37.736 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:37.759 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:37.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:37.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:37.772 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:17:37.861 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:37.885 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:38.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:38.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:38.122 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:38.145 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:38.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:38.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:38.247 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:38.272 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:38.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:38.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:38.374 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:38.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:38.397 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:38.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:38.498 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:38.521 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:38.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:38.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:38.623 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:38.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:38.666 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:38.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:38.767 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:38.802 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:38.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:38.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:38.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:38.934 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:38.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:38.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:39.036 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:39.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:39.063 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:39.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:39.164 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:39.196 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:39.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:39.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:39.298 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:39.320 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:39.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:39.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:39.385 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:17:39.385 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-17T18:17:39.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-17T18:17:39.385 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:17:39.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.385 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.45626.0:170335 55.2 55:4037a7bd:::benchmark_data_smithi136_313476_object78349:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1515)", 2024-08-17T18:17:39.386 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:08:36.080758+0000", 2024-08-17T18:17:39.386 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 543.30222575200003, 2024-08-17T18:17:39.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0985168270000001, 2024-08-17T18:17:39.386 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:39.386 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:39.386 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:39.386 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:39.386 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.45626", 2024-08-17T18:17:39.387 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:17:39.387 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 170335 2024-08-17T18:17:39.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.387 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:39.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:39.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.080758+0000", 2024-08-17T18:17:39.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:39.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.080758+0000", 2024-08-17T18:17:39.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:39.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.080759+0000", 2024-08-17T18:17:39.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.372e-06 2024-08-17T18:17:39.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:39.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.080764+0000", 2024-08-17T18:17:39.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6450000000000004e-06 2024-08-17T18:17:39.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:39.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.080773+0000", 2024-08-17T18:17:39.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5580000000000006e-06 2024-08-17T18:17:39.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:39.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.080780+0000", 2024-08-17T18:17:39.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3409999999999999e-06 2024-08-17T18:17:39.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:39.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.080848+0000", 2024-08-17T18:17:39.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8758999999999997e-05 2024-08-17T18:17:39.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:39.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.080860+0000", 2024-08-17T18:17:39.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1907e-05 2024-08-17T18:17:39.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:39.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.059824+0000", 2024-08-17T18:17:39.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97896355000000002 2024-08-17T18:17:39.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:39.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.059838+0000", 2024-08-17T18:17:39.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3708e-05 2024-08-17T18:17:39.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:39.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.173362+0000", 2024-08-17T18:17:39.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.113524307 2024-08-17T18:17:39.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:39.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.173469+0000", 2024-08-17T18:17:39.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106801 2024-08-17T18:17:39.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5,6", 2024-08-17T18:17:39.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.173805+0000", 2024-08-17T18:17:39.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00033645600000000001 2024-08-17T18:17:39.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:39.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.175758+0000", 2024-08-17T18:17:39.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0019531589999999999 2024-08-17T18:17:39.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.178437+0000", 2024-08-17T18:17:39.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0026783800000000002 2024-08-17T18:17:39.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.178483+0000", 2024-08-17T18:17:39.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6755000000000002e-05 2024-08-17T18:17:39.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.179161+0000", 2024-08-17T18:17:39.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00067756200000000004 2024-08-17T18:17:39.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:39.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.179200+0000", 2024-08-17T18:17:39.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8868000000000001e-05 2024-08-17T18:17:39.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:39.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.179275+0000", 2024-08-17T18:17:39.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4698999999999997e-05 2024-08-17T18:17:39.394 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.45626.0:170347 55.2 55:4738474e:::benchmark_data_smithi136_313476_object78361:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1515)", 2024-08-17T18:17:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:08:36.094869+0000", 2024-08-17T18:17:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 543.28811475400005, 2024-08-17T18:17:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0887959270000001, 2024-08-17T18:17:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.45626", 2024-08-17T18:17:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2772635857", 2024-08-17T18:17:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 170347 2024-08-17T18:17:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.094869+0000", 2024-08-17T18:17:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.094869+0000", 2024-08-17T18:17:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.094870+0000", 2024-08-17T18:17:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6309999999999999e-06 2024-08-17T18:17:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.094877+0000", 2024-08-17T18:17:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2330000000000003e-06 2024-08-17T18:17:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.094901+0000", 2024-08-17T18:17:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4813000000000001e-05 2024-08-17T18:17:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.094911+0000", 2024-08-17T18:17:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3249999999999997e-06 2024-08-17T18:17:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.095035+0000", 2024-08-17T18:17:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000124554 2024-08-17T18:17:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:36.095046+0000", 2024-08-17T18:17:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1032999999999999e-05 2024-08-17T18:17:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.059862+0000", 2024-08-17T18:17:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.96481608299999999 2024-08-17T18:17:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:17:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:37.059870+0000", 2024-08-17T18:17:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2080000000000003e-06 2024-08-17T18:17:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.175786+0000", 2024-08-17T18:17:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1159161850000001 2024-08-17T18:17:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.175914+0000", 2024-08-17T18:17:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012778600000000001 2024-08-17T18:17:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5,6", 2024-08-17T18:17:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.176280+0000", 2024-08-17T18:17:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036621700000000001 2024-08-17T18:17:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.178500+0000", 2024-08-17T18:17:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0022198169999999998 2024-08-17T18:17:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.178869+0000", 2024-08-17T18:17:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000369253 2024-08-17T18:17:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.179313+0000", 2024-08-17T18:17:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044423500000000002 2024-08-17T18:17:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.183487+0000", 2024-08-17T18:17:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0041739500000000001 2024-08-17T18:17:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.183547+0000", 2024-08-17T18:17:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0025000000000002e-05 2024-08-17T18:17:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:08:38.183665+0000", 2024-08-17T18:17:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011757899999999999 2024-08-17T18:17:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:27947 56.4 56:21cbfe32:::benchmark_data_smithi136_356456_object27946:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1540)", 2024-08-17T18:17:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:09:49.432911+0000", 2024-08-17T18:17:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 469.95007267300002, 2024-08-17T18:17:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.140233625, 2024-08-17T18:17:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:17:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:17:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 27947 2024-08-17T18:17:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.432911+0000", 2024-08-17T18:17:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.432911+0000", 2024-08-17T18:17:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.432911+0000", 2024-08-17T18:17:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9100000000000004e-07 2024-08-17T18:17:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.432934+0000", 2024-08-17T18:17:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2920000000000001e-05 2024-08-17T18:17:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.432943+0000", 2024-08-17T18:17:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2660000000000001e-06 2024-08-17T18:17:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.432949+0000", 2024-08-17T18:17:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3040000000000001e-06 2024-08-17T18:17:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:39.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.435502+0000", 2024-08-17T18:17:39.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025528130000000001 2024-08-17T18:17:39.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:39.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.435633+0000", 2024-08-17T18:17:39.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013102 2024-08-17T18:17:39.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4", 2024-08-17T18:17:39.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.435772+0000", 2024-08-17T18:17:39.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013916500000000001 2024-08-17T18:17:39.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:39.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.440445+0000", 2024-08-17T18:17:39.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0046734710000000002 2024-08-17T18:17:39.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.440508+0000", 2024-08-17T18:17:39.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2648000000000001e-05 2024-08-17T18:17:39.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573031+0000", 2024-08-17T18:17:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13252335000000001 2024-08-17T18:17:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573092+0000", 2024-08-17T18:17:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0747e-05 2024-08-17T18:17:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573144+0000", 2024-08-17T18:17:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2330000000000002e-05 2024-08-17T18:17:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:27951 56.4 56:222eaaba:::benchmark_data_smithi136_356456_object27950:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1540)", 2024-08-17T18:17:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:09:49.433998+0000", 2024-08-17T18:17:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 469.94898498800001, 2024-08-17T18:17:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13954208600000001, 2024-08-17T18:17:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:39.411 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:17:39.411 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:17:39.411 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 27951 2024-08-17T18:17:39.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.411 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:39.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.411 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:39.411 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.433998+0000", 2024-08-17T18:17:39.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.411 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:39.411 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.433998+0000", 2024-08-17T18:17:39.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.433999+0000", 2024-08-17T18:17:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0400000000000005e-07 2024-08-17T18:17:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434024+0000", 2024-08-17T18:17:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.491e-05 2024-08-17T18:17:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434035+0000", 2024-08-17T18:17:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1129e-05 2024-08-17T18:17:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434043+0000", 2024-08-17T18:17:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3069999999999997e-06 2024-08-17T18:17:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.437081+0000", 2024-08-17T18:17:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0030388279999999999 2024-08-17T18:17:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:39.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.437217+0000", 2024-08-17T18:17:39.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013517800000000001 2024-08-17T18:17:39.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4", 2024-08-17T18:17:39.415 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.437342+0000", 2024-08-17T18:17:39.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125091 2024-08-17T18:17:39.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.415 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:39.415 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.440518+0000", 2024-08-17T18:17:39.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003176636 2024-08-17T18:17:39.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.440831+0000", 2024-08-17T18:17:39.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00031303800000000002 2024-08-17T18:17:39.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573416+0000", 2024-08-17T18:17:39.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.132585077 2024-08-17T18:17:39.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573469+0000", 2024-08-17T18:17:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2899000000000003e-05 2024-08-17T18:17:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573541+0000", 2024-08-17T18:17:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1188999999999999e-05 2024-08-17T18:17:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.419 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:27954 56.4 56:216fadd1:::benchmark_data_smithi136_356456_object27953:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1540)", 2024-08-17T18:17:39.419 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:09:49.434097+0000", 2024-08-17T18:17:39.419 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 469.94888658799999, 2024-08-17T18:17:39.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13963388299999999, 2024-08-17T18:17:39.419 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:39.419 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:39.419 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:39.419 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:39.419 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:17:39.419 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:17:39.419 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 27954 2024-08-17T18:17:39.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.419 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434097+0000", 2024-08-17T18:17:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434097+0000", 2024-08-17T18:17:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434097+0000", 2024-08-17T18:17:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7800000000000002e-07 2024-08-17T18:17:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434118+0000", 2024-08-17T18:17:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1015000000000001e-05 2024-08-17T18:17:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434126+0000", 2024-08-17T18:17:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4490000000000002e-06 2024-08-17T18:17:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:39.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434131+0000", 2024-08-17T18:17:39.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8470000000000003e-06 2024-08-17T18:17:39.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:39.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.438432+0000", 2024-08-17T18:17:39.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0043018010000000001 2024-08-17T18:17:39.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:39.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439024+0000", 2024-08-17T18:17:39.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00059143900000000003 2024-08-17T18:17:39.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.423 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4", 2024-08-17T18:17:39.423 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439189+0000", 2024-08-17T18:17:39.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016538999999999999 2024-08-17T18:17:39.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.423 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.423 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.440867+0000", 2024-08-17T18:17:39.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016782069999999999 2024-08-17T18:17:39.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.423 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573554+0000", 2024-08-17T18:17:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13268660700000001 2024-08-17T18:17:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573637+0000", 2024-08-17T18:17:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3016999999999995e-05 2024-08-17T18:17:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573686+0000", 2024-08-17T18:17:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9038e-05 2024-08-17T18:17:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573731+0000", 2024-08-17T18:17:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4595000000000002e-05 2024-08-17T18:17:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:27955 56.4 56:2138118c:::benchmark_data_smithi136_356456_object27954:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1540)", 2024-08-17T18:17:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:09:49.434139+0000", 2024-08-17T18:17:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 469.94884478699998, 2024-08-17T18:17:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13974101799999999, 2024-08-17T18:17:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:17:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:17:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 27955 2024-08-17T18:17:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434139+0000", 2024-08-17T18:17:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434139+0000", 2024-08-17T18:17:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434139+0000", 2024-08-17T18:17:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7899999999999998e-07 2024-08-17T18:17:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434160+0000", 2024-08-17T18:17:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0937999999999998e-05 2024-08-17T18:17:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:39.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434166+0000", 2024-08-17T18:17:39.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1279999999999996e-06 2024-08-17T18:17:39.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:39.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.434172+0000", 2024-08-17T18:17:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3789999999999997e-06 2024-08-17T18:17:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.441201+0000", 2024-08-17T18:17:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0070293500000000002 2024-08-17T18:17:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.441417+0000", 2024-08-17T18:17:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000215813 2024-08-17T18:17:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4", 2024-08-17T18:17:39.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.441553+0000", 2024-08-17T18:17:39.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013591300000000001 2024-08-17T18:17:39.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:39.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573560+0000", 2024-08-17T18:17:39.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.132007491 2024-08-17T18:17:39.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573600+0000", 2024-08-17T18:17:39.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9687999999999997e-05 2024-08-17T18:17:39.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573801+0000", 2024-08-17T18:17:39.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020095200000000001 2024-08-17T18:17:39.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:39.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573845+0000", 2024-08-17T18:17:39.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4202999999999997e-05 2024-08-17T18:17:39.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:39.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573880+0000", 2024-08-17T18:17:39.432 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4584e-05 2024-08-17T18:17:39.432 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.432 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:39.432 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.432 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:27962 56.f 56:faa63ae2:::benchmark_data_smithi136_356456_object27961:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1540)", 2024-08-17T18:17:39.432 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:09:49.438972+0000", 2024-08-17T18:17:39.432 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 469.94401109400002, 2024-08-17T18:17:39.432 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.138606172, 2024-08-17T18:17:39.432 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:39.432 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:39.432 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:39.433 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:39.433 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:17:39.433 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:17:39.433 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 27962 2024-08-17T18:17:39.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.433 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:39.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:39.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.438972+0000", 2024-08-17T18:17:39.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:39.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.438972+0000", 2024-08-17T18:17:39.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:39.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.438973+0000", 2024-08-17T18:17:39.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5600000000000005e-07 2024-08-17T18:17:39.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:39.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.438996+0000", 2024-08-17T18:17:39.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3244e-05 2024-08-17T18:17:39.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.435 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:39.435 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439004+0000", 2024-08-17T18:17:39.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7489999999999998e-06 2024-08-17T18:17:39.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.435 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.435 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:39.435 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439011+0000", 2024-08-17T18:17:39.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5080000000000002e-06 2024-08-17T18:17:39.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.435 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.435 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:39.435 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439253+0000", 2024-08-17T18:17:39.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000242105 2024-08-17T18:17:39.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:39.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439372+0000", 2024-08-17T18:17:39.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119466 2024-08-17T18:17:39.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,6", 2024-08-17T18:17:39.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439497+0000", 2024-08-17T18:17:39.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012526700000000001 2024-08-17T18:17:39.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.574682+0000", 2024-08-17T18:17:39.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13518470399999999 2024-08-17T18:17:39.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.574727+0000", 2024-08-17T18:17:39.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4703999999999998e-05 2024-08-17T18:17:39.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:39.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.577465+0000", 2024-08-17T18:17:39.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0027381950000000001 2024-08-17T18:17:39.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:39.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.577529+0000", 2024-08-17T18:17:39.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4333999999999998e-05 2024-08-17T18:17:39.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:39.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.577579+0000", 2024-08-17T18:17:39.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.914e-05 2024-08-17T18:17:39.438 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.438 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:39.438 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.439 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:27966 56.f 56:f201f13f:::benchmark_data_smithi136_356456_object27965:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1540)", 2024-08-17T18:17:39.439 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:09:49.439427+0000", 2024-08-17T18:17:39.439 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 469.943556253, 2024-08-17T18:17:39.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13837285699999999, 2024-08-17T18:17:39.439 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:39.439 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:39.439 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:39.439 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:39.439 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:17:39.439 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:17:39.439 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 27966 2024-08-17T18:17:39.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.439 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:39.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:39.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439427+0000", 2024-08-17T18:17:39.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:39.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439427+0000", 2024-08-17T18:17:39.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:39.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439428+0000", 2024-08-17T18:17:39.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.54e-07 2024-08-17T18:17:39.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:39.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439459+0000", 2024-08-17T18:17:39.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0703000000000001e-05 2024-08-17T18:17:39.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:39.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439469+0000", 2024-08-17T18:17:39.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8730000000000002e-06 2024-08-17T18:17:39.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:39.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.439475+0000", 2024-08-17T18:17:39.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4779999999999996e-06 2024-08-17T18:17:39.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:39.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.441807+0000", 2024-08-17T18:17:39.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023322680000000002 2024-08-17T18:17:39.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:39.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.441929+0000", 2024-08-17T18:17:39.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121724 2024-08-17T18:17:39.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,6", 2024-08-17T18:17:39.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.573147+0000", 2024-08-17T18:17:39.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13121764799999999 2024-08-17T18:17:39.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.577267+0000", 2024-08-17T18:17:39.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0041202089999999997 2024-08-17T18:17:39.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.577451+0000", 2024-08-17T18:17:39.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018379400000000001 2024-08-17T18:17:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.577585+0000", 2024-08-17T18:17:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013449499999999999 2024-08-17T18:17:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.577625+0000", 2024-08-17T18:17:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0074000000000003e-05 2024-08-17T18:17:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:09:49.577800+0000", 2024-08-17T18:17:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017483700000000001 2024-08-17T18:17:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:88733 56.f 56:fb6e25d6:::benchmark_data_smithi136_356456_object88732:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1544)", 2024-08-17T18:17:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.476787+0000", 2024-08-17T18:17:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 427.90619646800002, 2024-08-17T18:17:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17365398000000001, 2024-08-17T18:17:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:17:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:17:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 88733 2024-08-17T18:17:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.476787+0000", 2024-08-17T18:17:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.476787+0000", 2024-08-17T18:17:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.476788+0000", 2024-08-17T18:17:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8999999999999995e-07 2024-08-17T18:17:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.476809+0000", 2024-08-17T18:17:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1331999999999998e-05 2024-08-17T18:17:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.476819+0000", 2024-08-17T18:17:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9799999999999993e-06 2024-08-17T18:17:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.476826+0000", 2024-08-17T18:17:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.097e-06 2024-08-17T18:17:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.476868+0000", 2024-08-17T18:17:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2256000000000003e-05 2024-08-17T18:17:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.477009+0000", 2024-08-17T18:17:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014094800000000001 2024-08-17T18:17:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,6", 2024-08-17T18:17:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.477144+0000", 2024-08-17T18:17:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013450699999999999 2024-08-17T18:17:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.481751+0000", 2024-08-17T18:17:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0046066379999999997 2024-08-17T18:17:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.481761+0000", 2024-08-17T18:17:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0548e-05 2024-08-17T18:17:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650331+0000", 2024-08-17T18:17:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16856977100000001 2024-08-17T18:17:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650393+0000", 2024-08-17T18:17:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1925000000000001e-05 2024-08-17T18:17:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650441+0000", 2024-08-17T18:17:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8188e-05 2024-08-17T18:17:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:88735 56.c 56:3d1600ab:::benchmark_data_smithi136_356456_object88734:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1544)", 2024-08-17T18:17:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.480738+0000", 2024-08-17T18:17:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 427.90224545699999, 2024-08-17T18:17:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17329349599999999, 2024-08-17T18:17:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:17:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:17:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 88735 2024-08-17T18:17:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.480738+0000", 2024-08-17T18:17:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.480738+0000", 2024-08-17T18:17:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.480740+0000", 2024-08-17T18:17:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5710000000000001e-06 2024-08-17T18:17:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.480762+0000", 2024-08-17T18:17:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2629999999999998e-05 2024-08-17T18:17:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.480775+0000", 2024-08-17T18:17:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3027000000000001e-05 2024-08-17T18:17:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.480783+0000", 2024-08-17T18:17:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4989999999999997e-06 2024-08-17T18:17:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.480844+0000", 2024-08-17T18:17:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1001000000000001e-05 2024-08-17T18:17:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.480988+0000", 2024-08-17T18:17:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000144723 2024-08-17T18:17:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:17:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.481128+0000", 2024-08-17T18:17:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013966599999999999 2024-08-17T18:17:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484806+0000", 2024-08-17T18:17:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036775620000000001 2024-08-17T18:17:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.485844+0000", 2024-08-17T18:17:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001037888 2024-08-17T18:17:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653923+0000", 2024-08-17T18:17:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16807926300000001 2024-08-17T18:17:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653983+0000", 2024-08-17T18:17:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9837999999999999e-05 2024-08-17T18:17:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.654031+0000", 2024-08-17T18:17:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8828000000000001e-05 2024-08-17T18:17:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:88739 56.f 56:f7ad6e62:::benchmark_data_smithi136_356456_object88738:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1544)", 2024-08-17T18:17:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.484508+0000", 2024-08-17T18:17:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 427.89847538499998, 2024-08-17T18:17:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.166084435, 2024-08-17T18:17:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:17:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:17:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 88739 2024-08-17T18:17:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484508+0000", 2024-08-17T18:17:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484508+0000", 2024-08-17T18:17:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484510+0000", 2024-08-17T18:17:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.592e-06 2024-08-17T18:17:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484536+0000", 2024-08-17T18:17:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6548999999999999e-05 2024-08-17T18:17:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484550+0000", 2024-08-17T18:17:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3585e-05 2024-08-17T18:17:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484558+0000", 2024-08-17T18:17:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.405e-06 2024-08-17T18:17:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484709+0000", 2024-08-17T18:17:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015043899999999999 2024-08-17T18:17:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484884+0000", 2024-08-17T18:17:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000175613 2024-08-17T18:17:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,6", 2024-08-17T18:17:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.485059+0000", 2024-08-17T18:17:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017479999999999999 2024-08-17T18:17:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650489+0000", 2024-08-17T18:17:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.165430095 2024-08-17T18:17:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650498+0000", 2024-08-17T18:17:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9800000000000004e-06 2024-08-17T18:17:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650524+0000", 2024-08-17T18:17:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5998000000000001e-05 2024-08-17T18:17:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650559+0000", 2024-08-17T18:17:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4718999999999997e-05 2024-08-17T18:17:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650592+0000", 2024-08-17T18:17:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.366e-05 2024-08-17T18:17:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:88741 56.e 56:77ba4448:::benchmark_data_smithi136_356456_object88740:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1544)", 2024-08-17T18:17:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.484583+0000", 2024-08-17T18:17:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 427.898400122, 2024-08-17T18:17:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16996449199999999, 2024-08-17T18:17:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:17:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:17:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 88741 2024-08-17T18:17:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484583+0000", 2024-08-17T18:17:39.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:39.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484583+0000", 2024-08-17T18:17:39.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:39.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484584+0000", 2024-08-17T18:17:39.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5700000000000002e-07 2024-08-17T18:17:39.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:39.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484604+0000", 2024-08-17T18:17:39.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9806999999999999e-05 2024-08-17T18:17:39.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484612+0000", 2024-08-17T18:17:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3820000000000004e-06 2024-08-17T18:17:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.484618+0000", 2024-08-17T18:17:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8300000000000001e-06 2024-08-17T18:17:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.485895+0000", 2024-08-17T18:17:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012773629999999999 2024-08-17T18:17:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486015+0000", 2024-08-17T18:17:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119335 2024-08-17T18:17:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,6", 2024-08-17T18:17:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486131+0000", 2024-08-17T18:17:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011650999999999999 2024-08-17T18:17:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653049+0000", 2024-08-17T18:17:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.166917811 2024-08-17T18:17:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653095+0000", 2024-08-17T18:17:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6360999999999999e-05 2024-08-17T18:17:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.654443+0000", 2024-08-17T18:17:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0013480129999999999 2024-08-17T18:17:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.654495+0000", 2024-08-17T18:17:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2076000000000001e-05 2024-08-17T18:17:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.654548+0000", 2024-08-17T18:17:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2447000000000001e-05 2024-08-17T18:17:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:88746 56.c 56:31d7e917:::benchmark_data_smithi136_356456_object88745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1544)", 2024-08-17T18:17:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.486034+0000", 2024-08-17T18:17:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 427.896949248, 2024-08-17T18:17:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.176589883, 2024-08-17T18:17:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:17:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:17:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 88746 2024-08-17T18:17:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486034+0000", 2024-08-17T18:17:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486034+0000", 2024-08-17T18:17:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486035+0000", 2024-08-17T18:17:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9099999999999991e-07 2024-08-17T18:17:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486057+0000", 2024-08-17T18:17:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.143e-05 2024-08-17T18:17:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486067+0000", 2024-08-17T18:17:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0572e-05 2024-08-17T18:17:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486074+0000", 2024-08-17T18:17:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5780000000000004e-06 2024-08-17T18:17:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650549+0000", 2024-08-17T18:17:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.164474957 2024-08-17T18:17:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650720+0000", 2024-08-17T18:17:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017162199999999999 2024-08-17T18:17:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:17:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650849+0000", 2024-08-17T18:17:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012822300000000001 2024-08-17T18:17:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.662108+0000", 2024-08-17T18:17:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011259725 2024-08-17T18:17:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.662194+0000", 2024-08-17T18:17:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6069999999999994e-05 2024-08-17T18:17:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.662434+0000", 2024-08-17T18:17:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000240103 2024-08-17T18:17:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.662489+0000", 2024-08-17T18:17:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4880000000000003e-05 2024-08-17T18:17:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.662624+0000", 2024-08-17T18:17:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013473200000000001 2024-08-17T18:17:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:88747 56.c 56:3c73eda4:::benchmark_data_smithi136_356456_object88746:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1544)", 2024-08-17T18:17:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.486091+0000", 2024-08-17T18:17:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 427.89689264399999, 2024-08-17T18:17:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.180009683, 2024-08-17T18:17:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:17:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:17:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 88747 2024-08-17T18:17:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486091+0000", 2024-08-17T18:17:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486091+0000", 2024-08-17T18:17:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486091+0000", 2024-08-17T18:17:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2e-07 2024-08-17T18:17:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486110+0000", 2024-08-17T18:17:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8410000000000002e-05 2024-08-17T18:17:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486117+0000", 2024-08-17T18:17:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1049999999999997e-06 2024-08-17T18:17:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486122+0000", 2024-08-17T18:17:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8119999999999998e-06 2024-08-17T18:17:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.654390+0000", 2024-08-17T18:17:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16826818199999999 2024-08-17T18:17:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.655078+0000", 2024-08-17T18:17:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00068772199999999999 2024-08-17T18:17:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,5", 2024-08-17T18:17:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.655223+0000", 2024-08-17T18:17:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000145167 2024-08-17T18:17:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.662237+0000", 2024-08-17T18:17:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0070139419999999996 2024-08-17T18:17:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.666004+0000", 2024-08-17T18:17:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037672869999999998 2024-08-17T18:17:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.666014+0000", 2024-08-17T18:17:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0484e-05 2024-08-17T18:17:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.666062+0000", 2024-08-17T18:17:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7919e-05 2024-08-17T18:17:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.666100+0000", 2024-08-17T18:17:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8133000000000002e-05 2024-08-17T18:17:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.46025.0:88733 56.f e1544/1535 ondisk, result = 0)", 2024-08-17T18:17:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.486213+0000", 2024-08-17T18:17:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 427.89677077900001, 2024-08-17T18:17:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16423845200000001, 2024-08-17T18:17:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T18:17:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486213+0000", 2024-08-17T18:17:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486213+0000", 2024-08-17T18:17:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486213+0000", 2024-08-17T18:17:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1999999999999999e-07 2024-08-17T18:17:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486223+0000", 2024-08-17T18:17:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9890000000000004e-06 2024-08-17T18:17:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486234+0000", 2024-08-17T18:17:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0883999999999999e-05 2024-08-17T18:17:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486241+0000", 2024-08-17T18:17:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6780000000000003e-06 2024-08-17T18:17:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650315+0000", 2024-08-17T18:17:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16407380599999999 2024-08-17T18:17:39.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.491 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:39.491 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650324+0000", 2024-08-17T18:17:39.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0799999999999995e-06 2024-08-17T18:17:39.491 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:39.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:39.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650451+0000", 2024-08-17T18:17:39.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012729500000000001 2024-08-17T18:17:39.492 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:88748 56.a 56:5de94025:::benchmark_data_smithi136_356456_object88747:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1544)", 2024-08-17T18:17:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.486267+0000", 2024-08-17T18:17:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 427.89671612900003, 2024-08-17T18:17:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17806833399999999, 2024-08-17T18:17:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:17:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:17:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 88748 2024-08-17T18:17:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486267+0000", 2024-08-17T18:17:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486267+0000", 2024-08-17T18:17:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486268+0000", 2024-08-17T18:17:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2799999999999995e-07 2024-08-17T18:17:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486290+0000", 2024-08-17T18:17:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2306999999999998e-05 2024-08-17T18:17:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486299+0000", 2024-08-17T18:17:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0860000000000002e-06 2024-08-17T18:17:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486305+0000", 2024-08-17T18:17:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5189999999999998e-06 2024-08-17T18:17:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486361+0000", 2024-08-17T18:17:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5878999999999998e-05 2024-08-17T18:17:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486474+0000", 2024-08-17T18:17:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112804 2024-08-17T18:17:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,6", 2024-08-17T18:17:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650225+0000", 2024-08-17T18:17:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16375092599999999 2024-08-17T18:17:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.660909+0000", 2024-08-17T18:17:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010684206 2024-08-17T18:17:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.660952+0000", 2024-08-17T18:17:39.498 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3374999999999997e-05 2024-08-17T18:17:39.498 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.498 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:39.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.664236+0000", 2024-08-17T18:17:39.498 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032840529999999999 2024-08-17T18:17:39.498 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.498 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.664294+0000", 2024-08-17T18:17:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7621999999999998e-05 2024-08-17T18:17:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.664336+0000", 2024-08-17T18:17:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1729e-05 2024-08-17T18:17:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.46025.0:88735 56.c e1544/1535 ondisk, result = 0)", 2024-08-17T18:17:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.486671+0000", 2024-08-17T18:17:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 427.89631261099998, 2024-08-17T18:17:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.167366402, 2024-08-17T18:17:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T18:17:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486671+0000", 2024-08-17T18:17:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:39.501 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486671+0000", 2024-08-17T18:17:39.501 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:39.501 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486671+0000", 2024-08-17T18:17:39.501 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.13e-07 2024-08-17T18:17:39.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:39.501 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.486675+0000", 2024-08-17T18:17:39.501 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5190000000000001e-06 2024-08-17T18:17:39.502 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.502 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.502 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:39.502 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650207+0000", 2024-08-17T18:17:39.502 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.163532289 2024-08-17T18:17:39.502 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.502 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.502 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:39.502 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650216+0000", 2024-08-17T18:17:39.502 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0389999999999993e-06 2024-08-17T18:17:39.502 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:39.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653909+0000", 2024-08-17T18:17:39.503 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036931970000000001 2024-08-17T18:17:39.503 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:39.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653917+0000", 2024-08-17T18:17:39.503 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4580000000000004e-06 2024-08-17T18:17:39.503 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:39.504 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.654037+0000", 2024-08-17T18:17:39.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120287 2024-08-17T18:17:39.504 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.504 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:39.504 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.504 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.46025.0:88741 56.e e1544/1535 ondisk, result = 0)", 2024-08-17T18:17:39.504 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:10:31.489475+0000", 2024-08-17T18:17:39.504 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 427.89350796600002, 2024-08-17T18:17:39.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.163581478, 2024-08-17T18:17:39.504 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:39.504 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:39.504 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T18:17:39.505 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:39.505 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.505 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:39.505 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.489475+0000", 2024-08-17T18:17:39.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.505 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.505 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.505 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:39.505 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.489475+0000", 2024-08-17T18:17:39.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.505 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.505 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.505 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:39.505 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.489477+0000", 2024-08-17T18:17:39.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.032e-06 2024-08-17T18:17:39.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:39.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.489482+0000", 2024-08-17T18:17:39.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4090000000000002e-06 2024-08-17T18:17:39.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:39.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650118+0000", 2024-08-17T18:17:39.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.160635842 2024-08-17T18:17:39.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.507 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:39.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.650133+0000", 2024-08-17T18:17:39.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5534000000000001e-05 2024-08-17T18:17:39.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.507 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:39.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653033+0000", 2024-08-17T18:17:39.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0028997620000000002 2024-08-17T18:17:39.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.508 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.508 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:39.508 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653042+0000", 2024-08-17T18:17:39.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3810000000000005e-06 2024-08-17T18:17:39.508 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.508 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.508 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:39.508 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:10:31.653057+0000", 2024-08-17T18:17:39.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4518000000000001e-05 2024-08-17T18:17:39.508 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.508 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:39.508 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.508 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.509 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:172837 56.e 56:7f6a0292:::benchmark_data_smithi136_356456_object46133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1551)", 2024-08-17T18:17:39.509 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:11:57.615790+0000", 2024-08-17T18:17:39.509 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 341.76719372999997, 2024-08-17T18:17:39.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078513984999999994, 2024-08-17T18:17:39.509 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:39.509 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:39.509 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:39.509 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:39.509 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:17:39.509 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:17:39.509 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 172837 2024-08-17T18:17:39.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.509 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:39.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:39.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.615790+0000", 2024-08-17T18:17:39.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:39.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.615790+0000", 2024-08-17T18:17:39.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:39.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.615791+0000", 2024-08-17T18:17:39.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4130000000000001e-06 2024-08-17T18:17:39.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:39.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.615796+0000", 2024-08-17T18:17:39.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4959999999999998e-06 2024-08-17T18:17:39.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:39.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.615808+0000", 2024-08-17T18:17:39.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1917e-05 2024-08-17T18:17:39.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:39.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.615815+0000", 2024-08-17T18:17:39.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9070000000000007e-06 2024-08-17T18:17:39.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:39.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.625307+0000", 2024-08-17T18:17:39.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0094920149999999995 2024-08-17T18:17:39.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:39.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.677360+0000", 2024-08-17T18:17:39.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.052052534999999997 2024-08-17T18:17:39.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,6", 2024-08-17T18:17:39.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.677480+0000", 2024-08-17T18:17:39.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119876 2024-08-17T18:17:39.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.690420+0000", 2024-08-17T18:17:39.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012940579000000001 2024-08-17T18:17:39.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:39.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.694117+0000", 2024-08-17T18:17:39.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003696595 2024-08-17T18:17:39.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.514 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.694203+0000", 2024-08-17T18:17:39.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6070999999999996e-05 2024-08-17T18:17:39.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:39.514 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.694258+0000", 2024-08-17T18:17:39.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4712000000000001e-05 2024-08-17T18:17:39.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:39.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.694304+0000", 2024-08-17T18:17:39.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5868999999999997e-05 2024-08-17T18:17:39.515 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.515 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:39.515 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.515 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:172847 56.e 56:7d8be824:::benchmark_data_smithi136_356456_object46143:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1551)", 2024-08-17T18:17:39.515 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:11:57.626997+0000", 2024-08-17T18:17:39.515 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 341.75598667000003, 2024-08-17T18:17:39.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.074151992999999999, 2024-08-17T18:17:39.515 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:17:39.515 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:17:39.515 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:17:39.516 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:17:39.516 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:17:39.516 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:17:39.516 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 172847 2024-08-17T18:17:39.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.516 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:17:39.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:17:39.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.626997+0000", 2024-08-17T18:17:39.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:17:39.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.626997+0000", 2024-08-17T18:17:39.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:17:39.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:17:39.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.626998+0000", 2024-08-17T18:17:39.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2929999999999999e-06 2024-08-17T18:17:39.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:17:39.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.627003+0000", 2024-08-17T18:17:39.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0139999999999998e-06 2024-08-17T18:17:39.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:17:39.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.676155+0000", 2024-08-17T18:17:39.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.049152108999999999 2024-08-17T18:17:39.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:17:39.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.676180+0000", 2024-08-17T18:17:39.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4691000000000001e-05 2024-08-17T18:17:39.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:17:39.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.682649+0000", 2024-08-17T18:17:39.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0064692400000000002 2024-08-17T18:17:39.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:17:39.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.682815+0000", 2024-08-17T18:17:39.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000165778 2024-08-17T18:17:39.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,6", 2024-08-17T18:17:39.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.683064+0000", 2024-08-17T18:17:39.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024944300000000002 2024-08-17T18:17:39.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:17:39.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.694122+0000", 2024-08-17T18:17:39.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011058047 2024-08-17T18:17:39.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.694160+0000", 2024-08-17T18:17:39.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8061e-05 2024-08-17T18:17:39.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:17:39.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.701028+0000", 2024-08-17T18:17:39.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0068671380000000001 2024-08-17T18:17:39.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:17:39.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.701098+0000", 2024-08-17T18:17:39.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0215000000000005e-05 2024-08-17T18:17:39.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:17:39.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:17:39.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:17:39.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:11:57.701149+0000", 2024-08-17T18:17:39.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0964000000000002e-05 2024-08-17T18:17:39.521 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.521 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:39.521 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.521 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:17:39.521 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:17:39.521 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:17:39.548 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:39.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:39.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:39.650 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:39.678 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:39.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:39.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:39.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:39 smithi136 ceph-mon[109861]: pgmap v3162: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 348 op/s; 54598/236007 objects degraded (23.134%); 31427/236007 objects misplaced (13.316%); 11 MiB/s, 175 objects/s recovering 2024-08-17T18:17:39.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:39 smithi136 ceph-mon[115078]: pgmap v3162: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 348 op/s; 54598/236007 objects degraded (23.134%); 31427/236007 objects misplaced (13.316%); 11 MiB/s, 175 objects/s recovering 2024-08-17T18:17:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:39 smithi165 ceph-mon[105517]: pgmap v3162: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 348 op/s; 54598/236007 objects degraded (23.134%); 31427/236007 objects misplaced (13.316%); 11 MiB/s, 175 objects/s recovering 2024-08-17T18:17:39.780 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:39.811 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:39.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:39.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:39.913 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:39.945 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:39.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:39.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:40.046 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:40.072 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:40.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:40.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:40.174 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:40.199 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:40.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:40.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:40.301 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:40.337 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:40.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:40.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:40.440 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:40.477 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:40.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:40.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:40.579 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:40.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:40.617 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:40.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:40.718 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:40.747 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:40.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:40.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:40.849 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:40.865 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:40 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 54598/236007 objects degraded (23.134%), 17 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:40.865 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:40 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 54598/236007 objects degraded (23.134%), 17 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:40.883 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:40.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:40.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:40.985 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:41.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:40 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 54598/236007 objects degraded (23.134%), 17 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:41.014 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:41.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:41.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:41.116 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:41.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:41.141 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:41.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:41.242 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:41.281 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:41.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:41.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:41.383 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:41.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:41.426 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:41.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:41.435 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-17T18:17:41.527 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:41.557 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:41.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:41.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:41.590 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:17:41.590 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:17:41.590 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:17:41.590 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:17:41.590 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:17:41.590 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:17:41.591 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:17:41.591 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:17:41.591 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:17:41.591 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:17:41.591 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:17:41.591 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:17:41.592 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:17:41.592 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:17:41.592 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:17:41.592 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:17:41.660 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:41.696 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:41.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:41 smithi136 ceph-mon[109861]: pgmap v3163: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 348 op/s; 54598/236007 objects degraded (23.134%); 31427/236007 objects misplaced (13.316%); 11 MiB/s, 175 objects/s recovering 2024-08-17T18:17:41.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:41 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2720674380' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:17:41.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:41 smithi136 ceph-mon[115078]: pgmap v3163: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 348 op/s; 54598/236007 objects degraded (23.134%); 31427/236007 objects misplaced (13.316%); 11 MiB/s, 175 objects/s recovering 2024-08-17T18:17:41.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:41 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2720674380' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:17:41.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:41.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:41 smithi165 ceph-mon[105517]: pgmap v3163: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 348 op/s; 54598/236007 objects degraded (23.134%); 31427/236007 objects misplaced (13.316%); 11 MiB/s, 175 objects/s recovering 2024-08-17T18:17:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:41 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2720674380' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:17:41.798 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:17:41.799 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:17:41.799 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:41.833 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:41.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:41.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:41.935 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:41.958 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:41.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:41.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:42.060 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:42.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:42.097 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:42.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:42.198 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:42.221 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:42.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:42.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:42.323 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:42.346 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:42.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:42.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:42.448 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:42.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:42.487 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:42.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:42.588 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:42.612 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:42.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:42.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:42.715 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:42.739 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:42.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:42.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:42.841 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:42.864 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:42.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:42.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:42.966 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:43.000 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:43.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:43.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:43.102 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:43.126 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:43.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:43.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:43.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:43.250 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:43.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:43.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:43.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:43.388 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:43.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:43.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:43.449 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:17:43.490 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:43.511 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:43.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:43.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:43.613 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:43.636 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:43.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:43.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:43.663 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:43 smithi136 ceph-mon[109861]: pgmap v3164: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 369 op/s; 54103/231381 objects degraded (23.383%); 30405/231381 objects misplaced (13.141%); 12 MiB/s, 186 objects/s recovering 2024-08-17T18:17:43.663 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:43 smithi136 ceph-mon[115078]: pgmap v3164: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 369 op/s; 54103/231381 objects degraded (23.383%); 30405/231381 objects misplaced (13.141%); 12 MiB/s, 186 objects/s recovering 2024-08-17T18:17:43.738 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:43.760 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:43.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:43.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:43.861 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:43.885 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:43.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:43.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:43.987 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:44.009 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:44.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:44.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:43 smithi165 ceph-mon[105517]: pgmap v3164: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 369 op/s; 54103/231381 objects degraded (23.383%); 30405/231381 objects misplaced (13.141%); 12 MiB/s, 186 objects/s recovering 2024-08-17T18:17:44.111 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:44.135 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:44.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:44.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:44.237 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:44.273 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:44.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:44.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:44.375 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:44.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:17:44.401 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:44.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:44.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:44.503 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:44.535 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:44.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:44.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:44.637 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:44.665 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:44.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:44.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:44.767 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:44.790 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:44.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:44.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:44.892 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:44.915 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:44.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:44.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:45.017 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:45.046 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:45.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:45.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:45.148 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:45.171 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:45.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:45.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:45.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:45.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:45.305 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:45.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:45.406 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:45.429 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:45.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:45.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:45.531 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:45.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:45.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:45.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:45.674 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:45.696 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:45.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:45.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:45.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:45.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:45.827 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:45.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:45.838 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:45 smithi136 ceph-mon[109861]: pgmap v3165: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 376 op/s; 53996/230496 objects degraded (23.426%); 30283/230496 objects misplaced (13.138%); 12 MiB/s, 186 objects/s recovering 2024-08-17T18:17:45.838 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:45 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 53996/230496 objects degraded (23.426%), 17 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:45.838 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:45 smithi136 ceph-mon[115078]: pgmap v3165: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 376 op/s; 53996/230496 objects degraded (23.426%); 30283/230496 objects misplaced (13.138%); 12 MiB/s, 186 objects/s recovering 2024-08-17T18:17:45.838 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:45 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 53996/230496 objects degraded (23.426%), 17 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:45.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:45.963 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:45.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:45.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:46.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:45 smithi165 ceph-mon[105517]: pgmap v3165: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 376 op/s; 53996/230496 objects degraded (23.426%); 30283/230496 objects misplaced (13.138%); 12 MiB/s, 186 objects/s recovering 2024-08-17T18:17:46.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:45 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 53996/230496 objects degraded (23.426%), 17 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:46.065 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:46.066 INFO:teuthology.orchestra.run.smithi136.stdout:moved 0 / 327 (0%) 2024-08-17T18:17:46.066 INFO:teuthology.orchestra.run.smithi136.stdout:avg 54.5 2024-08-17T18:17:46.066 INFO:teuthology.orchestra.run.smithi136.stdout:stddev 21.7543 -> 21.7543 (expected baseline 6.73919) 2024-08-17T18:17:46.066 INFO:teuthology.orchestra.run.smithi136.stdout:min osd.0 with 8 -> 8 pgs (0.146789 -> 0.146789 * mean) 2024-08-17T18:17:46.066 INFO:teuthology.orchestra.run.smithi136.stdout:max osd.7 with 74 -> 74 pgs (1.3578 -> 1.3578 * mean) 2024-08-17T18:17:46.067 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:17:46.067 INFO:teuthology.orchestra.run.smithi136.stdout:oload 1000 2024-08-17T18:17:46.067 INFO:teuthology.orchestra.run.smithi136.stdout:max_change 1 2024-08-17T18:17:46.067 INFO:teuthology.orchestra.run.smithi136.stdout:max_change_osds 4 2024-08-17T18:17:46.067 INFO:teuthology.orchestra.run.smithi136.stdout:average_utilization 0.0395 2024-08-17T18:17:46.067 INFO:teuthology.orchestra.run.smithi136.stdout:overload_utilization 0.3952 2024-08-17T18:17:46.067 INFO:teuthology.orchestra.run.smithi136.stderr:no change 2024-08-17T18:17:46.105 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:46.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:46.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:46.207 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:46.238 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:46.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:46.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:46.340 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:46.374 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:46.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:46.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:46.476 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:46.499 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:46.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:46.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:46.532 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:17:46.601 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:46.625 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:46.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:46.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:46.726 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:46.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:46.764 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:46.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:46.865 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:46.888 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:46.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:46.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:46.939 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:46 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/572287970' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:17:46.940 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:46 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:17:46.940 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:46 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/572287970' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:17:46.940 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:46 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:17:46.990 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:47.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:46 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/572287970' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:17:47.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:46 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:17:47.013 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:47.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:47.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:47.114 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:47.137 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:47.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:47.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:47.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:47.275 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:47.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:47.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:47.377 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:47.402 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:47.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:47.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:47.504 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:47.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:47.537 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:47.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:47.638 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:47.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:47.661 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:47.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:47.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:17:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:17:47] "GET /metrics HTTP/1.1" 200 36777 "" "Prometheus/2.51.0" 2024-08-17T18:17:47.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:47 smithi136 ceph-mon[109861]: from='client.46541 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:17:47.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:47 smithi136 ceph-mon[109861]: pgmap v3166: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 341 op/s; 53996/229875 objects degraded (23.489%); 30012/229875 objects misplaced (13.056%); 10 MiB/s, 159 objects/s recovering 2024-08-17T18:17:47.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:47 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:17:47.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:47 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:17:47.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:47 smithi136 ceph-mon[109861]: osdmap e1658: 8 total, 8 up, 6 in 2024-08-17T18:17:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:47 smithi136 ceph-mon[115078]: from='client.46541 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:17:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:47 smithi136 ceph-mon[115078]: pgmap v3166: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 341 op/s; 53996/229875 objects degraded (23.489%); 30012/229875 objects misplaced (13.056%); 10 MiB/s, 159 objects/s recovering 2024-08-17T18:17:47.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:47 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:17:47.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:47 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:17:47.698 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:47 smithi136 ceph-mon[115078]: osdmap e1658: 8 total, 8 up, 6 in 2024-08-17T18:17:47.762 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:47.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:47.795 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:47.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:47.896 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:47.923 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:47.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:47.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:47 smithi165 ceph-mon[105517]: from='client.46541 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T18:17:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:47 smithi165 ceph-mon[105517]: pgmap v3166: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 0 B/s wr, 341 op/s; 53996/229875 objects degraded (23.489%); 30012/229875 objects misplaced (13.056%); 10 MiB/s, 159 objects/s recovering 2024-08-17T18:17:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:47 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:17:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:47 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:17:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:47 smithi165 ceph-mon[105517]: osdmap e1658: 8 total, 8 up, 6 in 2024-08-17T18:17:48.025 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:48.047 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:48.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:48.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:48.149 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:48.174 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:48.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:48.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:48.276 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:48.302 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:48.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:48.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:48.404 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:48.427 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:48.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:48.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:48.530 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:48.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:48.563 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:48.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:48.609 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:17:48.664 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:48.691 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:48.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:48.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:48.793 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:48.820 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:48.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:48.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:48.922 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:48.948 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:48.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:48.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:49.050 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:49.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:49.076 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:49.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:49.177 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:49.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:49.206 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:49.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:49.307 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:49.334 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:49.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:49.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:49.364 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:17:49.437 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:49.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:49.470 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:49.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:49.571 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:49.604 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:49.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:49.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:49.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:49 smithi136 ceph-mon[109861]: pgmap v3168: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.6 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 406 op/s; 53278/223809 objects degraded (23.805%); 28980/223809 objects misplaced (12.949%); 12 MiB/s, 195 objects/s recovering 2024-08-17T18:17:49.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:49 smithi136 ceph-mon[115078]: pgmap v3168: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.6 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 406 op/s; 53278/223809 objects degraded (23.805%); 28980/223809 objects misplaced (12.949%); 12 MiB/s, 195 objects/s recovering 2024-08-17T18:17:49.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:49.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:49.738 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:49.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:49.839 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:49.871 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:49.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:49.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:49.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:50.000 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:50.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:50.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:50.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:49 smithi165 ceph-mon[105517]: pgmap v3168: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.6 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 406 op/s; 53278/223809 objects degraded (23.805%); 28980/223809 objects misplaced (12.949%); 12 MiB/s, 195 objects/s recovering 2024-08-17T18:17:50.102 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:50.125 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:50.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:50.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:50.227 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:50.249 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:50.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:50.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:50.263 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:17:50.263 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:17:50.351 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:50.374 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:50.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:50.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:50.475 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:50.497 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:50.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:50.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:50.599 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:50.623 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:50.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:50.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:50.724 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:50.746 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:50.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:50.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:50.848 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:50.872 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:50.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:50.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:50.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:50 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 53278/223809 objects degraded (23.805%), 17 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:50.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:50 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 53278/223809 objects degraded (23.805%), 17 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:50.975 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:51.007 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:51.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:51.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:51.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:50 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 53278/223809 objects degraded (23.805%), 17 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:51.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:51.138 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:51.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:51.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:51.240 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:51.262 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:51.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:51.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:51.366 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:51.388 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:51.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:51.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:51.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:51.517 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:51.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:51.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:51.619 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:51.645 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:51.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:51.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:51.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:51 smithi136 ceph-mon[115078]: pgmap v3169: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.6 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 406 op/s; 53278/223809 objects degraded (23.805%); 28980/223809 objects misplaced (12.949%); 12 MiB/s, 195 objects/s recovering 2024-08-17T18:17:51.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:17:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:51 smithi136 ceph-mon[109861]: pgmap v3169: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.6 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 406 op/s; 53278/223809 objects degraded (23.805%); 28980/223809 objects misplaced (12.949%); 12 MiB/s, 195 objects/s recovering 2024-08-17T18:17:51.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:17:51.747 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:51.770 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:51.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:51.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:51.795 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-17T18:17:51.848 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:17:51.872 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:51.895 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:51.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:51.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:51.997 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:51 smithi165 ceph-mon[105517]: pgmap v3169: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.6 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 406 op/s; 53278/223809 objects degraded (23.805%); 28980/223809 objects misplaced (12.949%); 12 MiB/s, 195 objects/s recovering 2024-08-17T18:17:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:17:52.018 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:52.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:52.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:52.120 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:52.152 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:52.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:52.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:52.254 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:52.277 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:52.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:52.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:52.379 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:52.404 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:52.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:52.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:52.506 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:52.533 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:52.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:52.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:52.576 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:17:52.576 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:17:52.596 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:17:52.596 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:17:52.614 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:17:52.615 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:17:52.631 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:17:52.632 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:17:52.634 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:52.649 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:17:52.650 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:17:52.657 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:52.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:52.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:52.666 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:17:52.666 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:17:52.684 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:17:52.685 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:17:52.701 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:17:52.701 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:17:52.759 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:52.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:52.781 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:52.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:52.882 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:52.915 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:52.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:52.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:53.017 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:53.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:53.051 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:53.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:53.152 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:53.176 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:53.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:53.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:53.278 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:53.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:53.311 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:53.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:53.412 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:53.434 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:53.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:53.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:53.535 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:53.571 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:53.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:53.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:53.585 INFO:tasks.thrashosds.thrasher:in_osds: [2, 4, 6, 0, 5, 7] out_osds: [1, 3] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:17:53.585 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:17:53.586 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-08-17T18:17:53.586 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T18:17:53.673 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:53.704 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:53.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:53.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:53.806 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:53.829 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:53.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:53.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:53.871 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:53 smithi136 ceph-mon[109861]: pgmap v3170: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.5 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 401 op/s; 52781/219321 objects degraded (24.066%); 27782/219321 objects misplaced (12.667%); 13 MiB/s, 224 objects/s recovering 2024-08-17T18:17:53.871 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:53 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2209628836' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:17:53.871 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:53 smithi136 ceph-mon[115078]: pgmap v3170: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.5 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 401 op/s; 52781/219321 objects degraded (24.066%); 27782/219321 objects misplaced (12.667%); 13 MiB/s, 224 objects/s recovering 2024-08-17T18:17:53.871 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:53 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2209628836' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:17:53.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:53.967 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:53.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:53.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:53 smithi165 ceph-mon[105517]: pgmap v3170: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.5 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 401 op/s; 52781/219321 objects degraded (24.066%); 27782/219321 objects misplaced (12.667%); 13 MiB/s, 224 objects/s recovering 2024-08-17T18:17:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:53 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2209628836' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:17:54.068 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:54.091 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:54.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:54.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:54.193 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:54.213 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:17:54.229 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:54.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:54.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:54.330 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:54.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:54.367 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:54.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:54.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:54.489 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:54.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:54.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:54.591 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:54.625 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:54.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:54.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:54.727 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:54.761 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:54.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:54.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:54.785 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:17:54.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:54.899 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:54.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:54.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:55.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:55.031 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:55.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:55.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:55.133 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:55.154 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:55.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:55.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:55.256 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:55.285 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:55.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:55.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:55.387 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:55.408 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:55.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:55.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:55.510 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:55.541 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:55.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:55.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:55.547 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:17:55.643 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:55.677 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:55.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:55.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:55 smithi136 ceph-mon[109861]: pgmap v3171: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 400 op/s; 52652/218475 objects degraded (24.100%); 27675/218475 objects misplaced (12.667%); 13 MiB/s, 225 objects/s recovering 2024-08-17T18:17:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:55 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 52652/218475 objects degraded (24.100%), 17 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:55.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:55 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1077556395' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:17:55.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:55 smithi136 ceph-mon[115078]: pgmap v3171: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 400 op/s; 52652/218475 objects degraded (24.100%); 27675/218475 objects misplaced (12.667%); 13 MiB/s, 225 objects/s recovering 2024-08-17T18:17:55.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:55 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 52652/218475 objects degraded (24.100%), 17 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:55.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:55 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1077556395' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:17:55.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:55.812 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:55.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:55.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:55.913 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:55.945 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:55.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:55.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:56.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:55 smithi165 ceph-mon[105517]: pgmap v3171: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 400 op/s; 52652/218475 objects degraded (24.100%); 27675/218475 objects misplaced (12.667%); 13 MiB/s, 225 objects/s recovering 2024-08-17T18:17:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:55 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 52652/218475 objects degraded (24.100%), 17 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T18:17:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:55 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1077556395' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:17:56.047 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:56.080 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:56.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:56.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:56.182 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:56.206 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:56.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:56.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:56.211 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:17:56.309 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:56.332 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:56.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:56.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:56.434 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:17:56.461 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:56.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:56.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:17:56.563 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:56.596 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:56.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:56.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:56.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:56.721 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:56.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:56.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:56.823 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:56.853 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:56.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:56.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:56.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:56 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:17:56.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:56 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1077556395' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:17:56.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:56 smithi136 ceph-mon[115078]: osdmap e1659: 8 total, 8 up, 6 in 2024-08-17T18:17:56.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:56 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:17:56.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:56 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1077556395' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:17:56.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:56 smithi136 ceph-mon[109861]: osdmap e1659: 8 total, 8 up, 6 in 2024-08-17T18:17:56.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:56.976 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:56.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:56.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:57.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:56 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:17:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:56 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1077556395' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:17:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:56 smithi165 ceph-mon[105517]: osdmap e1659: 8 total, 8 up, 6 in 2024-08-17T18:17:57.078 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:57.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:57.101 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:57.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:57.202 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:57.224 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:57.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:57.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:57.326 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:57.348 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:57.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:57.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:57.450 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:57.474 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:57.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:57.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:57.576 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:57.604 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:57.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:57.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:57.706 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:57.730 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:57.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:57.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:57.832 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:57.855 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:57.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:57.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:57.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:57 smithi136 ceph-mon[109861]: pgmap v3173: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 403 op/s; 52652/217866 objects degraded (24.167%); 27402/217866 objects misplaced (12.577%); 13 MiB/s, 226 objects/s recovering 2024-08-17T18:17:57.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:57 smithi136 ceph-mon[109861]: osdmap e1660: 8 total, 8 up, 6 in 2024-08-17T18:17:57.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:17:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:17:57] "GET /metrics HTTP/1.1" 200 36776 "" "Prometheus/2.51.0" 2024-08-17T18:17:57.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:57 smithi136 ceph-mon[115078]: pgmap v3173: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 403 op/s; 52652/217866 objects degraded (24.167%); 27402/217866 objects misplaced (12.577%); 13 MiB/s, 226 objects/s recovering 2024-08-17T18:17:57.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:57 smithi136 ceph-mon[115078]: osdmap e1660: 8 total, 8 up, 6 in 2024-08-17T18:17:57.956 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:57.980 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:57.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:57.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:58.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:57 smithi165 ceph-mon[105517]: pgmap v3173: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 14 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 403 op/s; 52652/217866 objects degraded (24.167%); 27402/217866 objects misplaced (12.577%); 13 MiB/s, 226 objects/s recovering 2024-08-17T18:17:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:57 smithi165 ceph-mon[105517]: osdmap e1660: 8 total, 8 up, 6 in 2024-08-17T18:17:58.082 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:58.105 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:58.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:58.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:58.207 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:58.247 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:58.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:58.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:58.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:58.379 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:58.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:58.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:58.481 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:58.503 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:58.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:58.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:58.605 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:17:58.633 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:58.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:58.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:17:58.735 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:58.757 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:58.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:58.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:17:58.821 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:58 smithi136 ceph-mon[109861]: osdmap e1661: 8 total, 8 up, 6 in 2024-08-17T18:17:58.821 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:58 smithi136 ceph-mon[115078]: osdmap e1661: 8 total, 8 up, 6 in 2024-08-17T18:17:58.859 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:17:58.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:58.891 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:58.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:17:58.992 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:59.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:58 smithi165 ceph-mon[105517]: osdmap e1661: 8 total, 8 up, 6 in 2024-08-17T18:17:59.014 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:59.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:59.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:59.116 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:59.138 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:59.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:59.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:59.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:59.269 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:59.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:59.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:59.371 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:17:59.395 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:59.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:59.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:17:59.497 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:17:59.521 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:59.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:59.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:17:59.623 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:17:59.656 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:59.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:59.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:17:59.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:17:59 smithi136 ceph-mon[109861]: pgmap v3176: 109 pgs: 1 peering, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 383 op/s; 48756/212412 objects degraded (22.954%); 28990/212412 objects misplaced (13.648%); 7.6 MiB/s, 271 objects/s recovering 2024-08-17T18:17:59.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:17:59 smithi136 ceph-mon[115078]: pgmap v3176: 109 pgs: 1 peering, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 383 op/s; 48756/212412 objects degraded (22.954%); 28990/212412 objects misplaced (13.648%); 7.6 MiB/s, 271 objects/s recovering 2024-08-17T18:17:59.758 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:17:59.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:59.791 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:17:59.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:17:59.892 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:17:59.915 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:17:59.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:17:59.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:17:59 smithi165 ceph-mon[105517]: pgmap v3176: 109 pgs: 1 peering, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 383 op/s; 48756/212412 objects degraded (22.954%); 28990/212412 objects misplaced (13.648%); 7.6 MiB/s, 271 objects/s recovering 2024-08-17T18:18:00.017 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:00.051 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:00.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:00.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:00.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:00.176 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:00.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:00.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:00.278 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:00.303 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:00.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:00.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:00.405 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:00.429 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:00.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:00.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:00.531 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:00.553 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:00.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:00.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:00.655 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:00.686 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:00.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:00.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:00.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:00 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 48756/212412 objects degraded (22.954%), 16 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:00.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:00 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 48756/212412 objects degraded (22.954%), 16 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:00.787 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:00.813 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:00.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:00.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:00.914 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:00.953 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:00.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:00.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:01.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:00 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 48756/212412 objects degraded (22.954%), 16 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:01.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:01.078 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:01.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:01.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:01.180 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:01.212 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:18:01.215 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:01.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:01.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:01.223 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:18:01.223 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T18:18:01.223 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-17T18:18:01.224 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-17T18:18:01.224 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:18:01.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:01.345 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:01.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:01.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:01.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:01.478 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:01.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:01.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:01.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:01.605 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:01.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:01.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:01.707 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:01.730 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:18:01.732 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:01.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:01.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:01.834 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:01.856 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:01.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:01.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:01.894 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:01 smithi136 ceph-mon[109861]: pgmap v3177: 109 pgs: 1 peering, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 336 op/s; 48756/212412 objects degraded (22.954%); 28990/212412 objects misplaced (13.648%); 7.0 MiB/s, 260 objects/s recovering 2024-08-17T18:18:01.894 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:01 smithi136 ceph-mon[115078]: pgmap v3177: 109 pgs: 1 peering, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 336 op/s; 48756/212412 objects degraded (22.954%); 28990/212412 objects misplaced (13.648%); 7.0 MiB/s, 260 objects/s recovering 2024-08-17T18:18:01.958 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:01.990 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:01.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:01.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:02.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:01 smithi165 ceph-mon[105517]: pgmap v3177: 109 pgs: 1 peering, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 336 op/s; 48756/212412 objects degraded (22.954%); 28990/212412 objects misplaced (13.648%); 7.0 MiB/s, 260 objects/s recovering 2024-08-17T18:18:02.092 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:02.116 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:02.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:02.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:02.217 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:02.242 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:02.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:02.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:02.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:02.371 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:02.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:02.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:02.473 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:02.505 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:02.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:02.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:02.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:02.642 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:02.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:02.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:02.744 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:02.769 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:02.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:02.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:02.871 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:02.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:02.907 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:02.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:03.008 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:03.032 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:03.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:03.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:03.134 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:03.157 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:03.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:03.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:03.259 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:03.289 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:03.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:03.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:03.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:03.425 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:03.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:03.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:03.527 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:03.558 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:03.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:03.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:03.660 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:03.685 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:03.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:03.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:03.787 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:03.810 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:03.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:03.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:03.911 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:03.935 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:03.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:03.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:03 smithi136 ceph-mon[109861]: pgmap v3178: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 546 op/s; 48176/207129 objects degraded (23.259%); 27947/207129 objects misplaced (13.493%); 15 MiB/s, 361 objects/s recovering 2024-08-17T18:18:03.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:03 smithi136 ceph-mon[115078]: pgmap v3178: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 546 op/s; 48176/207129 objects degraded (23.259%); 27947/207129 objects misplaced (13.493%); 15 MiB/s, 361 objects/s recovering 2024-08-17T18:18:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:03 smithi165 ceph-mon[105517]: pgmap v3178: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 546 op/s; 48176/207129 objects degraded (23.259%); 27947/207129 objects misplaced (13.493%); 15 MiB/s, 361 objects/s recovering 2024-08-17T18:18:04.037 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:04.068 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:04.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:04.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:04.170 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:04.203 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:04.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:04.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:04.305 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:04.336 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:04.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:04.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:04.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:04.462 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:04.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:04.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:04.564 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:04.597 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:04.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:04.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:04.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:04.721 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:04.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:04.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:04.823 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:04.848 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:04.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:04.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:04.949 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:04.980 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:04.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:04.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:05.083 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:05.105 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:05.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:05.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:05.207 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:05.229 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:05.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:05.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:05.331 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:05.356 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:05.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:05.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:05.458 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:05.476 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-17T18:18:05.485 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:05.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:05.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:05.588 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:05.612 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:05.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:05.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:05 smithi136 ceph-mon[109861]: pgmap v3179: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 491 op/s; 48046/206076 objects degraded (23.315%); 27806/206076 objects misplaced (13.493%); 13 MiB/s, 320 objects/s recovering 2024-08-17T18:18:05.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:05 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 48046/206076 objects degraded (23.315%), 16 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:05.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:05 smithi136 ceph-mon[115078]: pgmap v3179: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 491 op/s; 48046/206076 objects degraded (23.315%); 27806/206076 objects misplaced (13.493%); 13 MiB/s, 320 objects/s recovering 2024-08-17T18:18:05.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:05 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 48046/206076 objects degraded (23.315%), 16 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:05.714 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:05.745 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:05.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:05.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:05.847 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:05.885 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:05.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:05.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:05.987 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:05 smithi165 ceph-mon[105517]: pgmap v3179: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 491 op/s; 48046/206076 objects degraded (23.315%); 27806/206076 objects misplaced (13.493%); 13 MiB/s, 320 objects/s recovering 2024-08-17T18:18:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:05 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 48046/206076 objects degraded (23.315%), 16 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:06.019 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:06.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:06.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:06.121 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:06.155 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:06.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:06.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:06.257 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:06.277 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:06.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:06.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:06.347 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:18:06.349 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T18:18:06.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:06.402 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:06.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:06.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:06.503 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:06.539 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:06.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:06.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:06.641 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:06.666 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:06.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:06.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:06.768 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:06.804 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:06.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:06.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:06.906 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:06.930 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:06.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:06.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:07.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:07.066 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:07.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:07.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:07.151 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":3179,"stamp":"2024-08-17T18:18:04.463482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4500732744,"num_objects":68692,"num_object_clones":0,"num_object_copies":206076,"num_objects_missing_on_primary":1372,"num_objects_missing":1372,"num_objects_degraded":48046,"num_objects_misplaced":27806,"num_objects_unfound":0,"num_objects_dirty":68692,"num_whiteouts":0,"num_read":70,"num_read_kb":150,"num_write":163277,"num_write_kb":7419289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20735,"num_bytes_recovered":1276416040,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140962,"ondisk_log_size":140962,"up":327,"acting":313,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":21041652,"kb_used_data":15438788,"kb_used_omap":213,"kb_used_meta":5602602,"kb_avail":541429260,"statfs":{"total":575970213888,"available":554423562240,"internally_reserved":0,"allocated":15809318912,"data_stored":15783590852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":218294,"internal_metadata":5737065290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,4,4,0,3,4,1,3,3,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-257556480,"num_objects":-3930,"num_object_clones":0,"num_object_copies":-11790,"num_objects_missing_on_primary":-123,"num_objects_missing":-123,"num_objects_degraded":-4606,"num_objects_misplaced":404,"num_objects_unfound":0,"num_objects_dirty":-3930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3930,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2565,"num_bytes_recovered":109576192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002143"},"pg_stats":[{"pgid":"57.19","version":"1661'7541","reported_seq":7615,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:18:02.905635+0000","last_change":"2024-08-17T18:14:49.611603+0000","last_active":"2024-08-17T18:18:02.905635+0000","last_peered":"2024-08-17T18:18:02.905635+0000","last_clean":"2024-08-17T18:18:02.905635+0000","last_became_active":"2024-08-17T18:14:49.606497+0000","last_became_peered":"2024-08-17T18:14:49.606497+0000","last_unstale":"2024-08-17T18:18:02.905635+0000","last_undegraded":"2024-08-17T18:18:02.905635+0000","last_fullsized":"2024-08-17T18:18:02.905635+0000","mapping_epoch":1616,"log_start":"1613'1964","ondisk_log_start":"1613'1964","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5577,"log_dups_size":0,"ondisk_log_size":5577,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140017664,"num_objects":2137,"num_object_clones":0,"num_object_copies":6411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5063,"num_write_kb":230368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"57.18","version":"1661'7692","reported_seq":10421,"reported_epoch":1661,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:18:01.887699+0000","last_change":"2024-08-17T18:15:54.579791+0000","last_active":"2024-08-17T18:18:01.887699+0000","last_peered":"2024-08-17T18:18:01.887699+0000","last_clean":"2024-08-17T18:15:09.641861+0000","last_became_active":"2024-08-17T18:15:54.569392+0000","last_became_peered":"2024-08-17T18:15:54.569392+0000","last_unstale":"2024-08-17T18:18:01.887699+0000","last_undegraded":"2024-08-17T18:15:54.395452+0000","last_fullsized":"2024-08-17T18:18:01.887699+0000","mapping_epoch":1638,"log_start":"1613'2737","ondisk_log_start":"1613'2737","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4955,"log_dups_size":0,"ondisk_log_size":4955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138838016,"num_objects":2119,"num_object_clones":0,"num_object_copies":6357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":210,"num_objects_unfound":0,"num_objects_dirty":2119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5185,"num_write_kb":233696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1318,"num_bytes_recovered":86376448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[5,6,4],"avail_no_missing":["5","0","6"],"object_location_counts":[{"shards":"0,5,6","objects":1694},{"shards":"4,5,6","objects":425}],"blocked_by":[],"up_primary":7,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.17","version":"1661'7384","reported_seq":8436,"reported_epoch":1661,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:18:02.195610+0000","last_change":"2024-08-17T18:17:23.728286+0000","last_active":"2024-08-17T18:18:02.195610+0000","last_peered":"2024-08-17T18:18:02.195610+0000","last_clean":"2024-08-17T18:15:52.210567+0000","last_became_active":"2024-08-17T18:17:23.718959+0000","last_became_peered":"2024-08-17T18:17:23.718959+0000","last_unstale":"2024-08-17T18:18:02.195610+0000","last_undegraded":"2024-08-17T18:17:23.713377+0000","last_fullsized":"2024-08-17T18:17:23.630677+0000","mapping_epoch":1653,"log_start":"1613'1585","ondisk_log_start":"1613'1585","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5799,"log_dups_size":0,"ondisk_log_size":5799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137265152,"num_objects":2095,"num_object_clones":0,"num_object_copies":6285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4953,"num_write_kb":225504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":446,"num_bytes_recovered":29229056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,4,6","objects":2077},{"shards":"2,7","objects":18}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"57.16","version":"1661'7410","reported_seq":14620,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:18:01.548778+0000","last_change":"2024-08-17T18:17:57.587085+0000","last_active":"2024-08-17T18:18:01.548778+0000","last_peered":"2024-08-17T18:18:01.548778+0000","last_clean":"2024-08-17T18:18:01.548778+0000","last_became_active":"2024-08-17T18:17:57.576108+0000","last_became_peered":"2024-08-17T18:17:57.576108+0000","last_unstale":"2024-08-17T18:18:01.548778+0000","last_undegraded":"2024-08-17T18:18:01.548778+0000","last_fullsized":"2024-08-17T18:18:01.548778+0000","mapping_epoch":1660,"log_start":"1613'2342","ondisk_log_start":"1613'2342","created":1616,"last_epoch_clean":1661,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5068,"log_dups_size":0,"ondisk_log_size":5068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134905856,"num_objects":2059,"num_object_clones":0,"num_object_copies":6177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4975,"num_write_kb":225056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":166985728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.15","version":"1661'7569","reported_seq":8995,"reported_epoch":1661,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:18:01.919162+0000","last_change":"2024-08-17T18:17:23.977015+0000","last_active":"2024-08-17T18:18:01.919162+0000","last_peered":"2024-08-17T18:18:01.919162+0000","last_clean":"2024-08-17T18:15:09.587418+0000","last_became_active":"2024-08-17T18:17:23.958208+0000","last_became_peered":"2024-08-17T18:17:23.958208+0000","last_unstale":"2024-08-17T18:18:01.919162+0000","last_undegraded":"2024-08-17T18:17:23.810234+0000","last_fullsized":"2024-08-17T18:17:23.637867+0000","mapping_epoch":1653,"log_start":"1613'2331","ondisk_log_start":"1613'2331","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5238,"log_dups_size":0,"ondisk_log_size":5238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147095552,"num_objects":2245,"num_object_clones":0,"num_object_copies":6735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5809,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5067,"num_write_kb":233952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":440,"num_bytes_recovered":28770304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7],"avail_no_missing":["5","0","6"],"object_location_counts":[{"shards":"0,5,6","objects":1711},{"shards":"2,5,6","objects":443},{"shards":"5,7","objects":91}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.14","version":"1661'7466","reported_seq":7602,"reported_epoch":1661,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:18:01.808853+0000","last_change":"2024-08-17T18:17:22.720441+0000","last_active":"2024-08-17T18:18:01.808853+0000","last_peered":"2024-08-17T18:18:01.808853+0000","last_clean":"2024-08-17T18:15:09.651475+0000","last_became_active":"2024-08-17T18:15:53.663522+0000","last_became_peered":"2024-08-17T18:15:53.663522+0000","last_unstale":"2024-08-17T18:18:01.808853+0000","last_undegraded":"2024-08-17T18:15:53.309793+0000","last_fullsized":"2024-08-17T18:15:53.227799+0000","mapping_epoch":1637,"log_start":"1613'2311","ondisk_log_start":"1613'2311","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5155,"log_dups_size":0,"ondisk_log_size":5155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137560064,"num_objects":2099,"num_object_clones":0,"num_object_copies":6297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3558,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5013,"num_write_kb":227584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,4],"avail_no_missing":["5","0","4"],"object_location_counts":[{"shards":"0,4,5","objects":1660},{"shards":"4,5","objects":-1261},{"shards":"4,5,6","objects":441}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.13","version":"1661'7476","reported_seq":8523,"reported_epoch":1661,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:18:01.802056+0000","last_change":"2024-08-17T18:17:23.894092+0000","last_active":"2024-08-17T18:18:01.802056+0000","last_peered":"2024-08-17T18:18:01.802056+0000","last_clean":"2024-08-17T18:15:52.116504+0000","last_became_active":"2024-08-17T18:17:23.845614+0000","last_became_peered":"2024-08-17T18:17:23.845614+0000","last_unstale":"2024-08-17T18:18:01.802056+0000","last_undegraded":"2024-08-17T18:17:23.781590+0000","last_fullsized":"2024-08-17T18:17:23.627784+0000","mapping_epoch":1653,"log_start":"1612'1034","ondisk_log_start":"1612'1034","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":6442,"log_dups_size":0,"ondisk_log_size":6442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140017664,"num_objects":2137,"num_object_clones":0,"num_object_copies":6411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5027,"num_write_kb":229216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":453,"num_bytes_recovered":29687808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"0,2,6","objects":2114},{"shards":"6,7","objects":23}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"57.12","version":"1661'7495","reported_seq":11805,"reported_epoch":1661,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:18:02.943752+0000","last_change":"2024-08-17T18:17:22.682686+0000","last_active":"2024-08-17T18:18:02.943752+0000","last_peered":"2024-08-17T18:18:02.943752+0000","last_clean":"2024-08-17T18:14:59.641932+0000","last_became_active":"2024-08-17T18:16:00.332694+0000","last_became_peered":"2024-08-17T18:16:00.332694+0000","last_unstale":"2024-08-17T18:18:02.943752+0000","last_undegraded":"2024-08-17T18:15:57.039315+0000","last_fullsized":"2024-08-17T18:15:53.226474+0000","mapping_epoch":1637,"log_start":"1613'1834","ondisk_log_start":"1613'1834","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5661,"log_dups_size":0,"ondisk_log_size":5661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140148736,"num_objects":2139,"num_object_clones":0,"num_object_copies":6417,"num_objects_missing_on_primary":1372,"num_objects_missing":1372,"num_objects_degraded":3666,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5039,"num_write_kb":229664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1445,"num_bytes_recovered":94633984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,2,4,5","objects":7},{"shards":"0,2,5","objects":1372},{"shards":"0,4,5","objects":302},{"shards":"2,4,5","objects":441},{"shards":"4,5","objects":10}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"57.11","version":"1661'7719","reported_seq":7876,"reported_epoch":1661,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:18:01.202646+0000","last_change":"2024-08-17T18:17:23.654437+0000","last_active":"2024-08-17T18:18:01.202646+0000","last_peered":"2024-08-17T18:18:01.202646+0000","last_clean":"2024-08-17T18:14:57.659306+0000","last_became_active":"2024-08-17T18:17:23.639891+0000","last_became_peered":"2024-08-17T18:17:23.639891+0000","last_unstale":"2024-08-17T18:18:01.202646+0000","last_undegraded":"2024-08-17T18:18:01.202646+0000","last_fullsized":"2024-08-17T18:18:01.202646+0000","mapping_epoch":1653,"log_start":"1611'100","ondisk_log_start":"1611'100","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":7619,"log_dups_size":0,"ondisk_log_size":7619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142639104,"num_objects":2177,"num_object_clones":0,"num_object_copies":6531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4354,"num_objects_unfound":0,"num_objects_dirty":2177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5199,"num_write_kb":236000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[0,6,2],"avail_no_missing":["0","2","6"],"object_location_counts":[{"shards":"0,2,6","objects":2177}],"blocked_by":[],"up_primary":4,"acting_primary":0,"purged_snaps":[]},{"pgid":"57.10","version":"1661'7627","reported_seq":17359,"reported_epoch":1661,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:18:01.730989+0000","last_change":"2024-08-17T18:17:23.909491+0000","last_active":"2024-08-17T18:18:01.730989+0000","last_peered":"2024-08-17T18:18:01.730989+0000","last_clean":"2024-08-17T18:15:52.133173+0000","last_became_active":"2024-08-17T18:17:23.807810+0000","last_became_peered":"2024-08-17T18:17:23.807810+0000","last_unstale":"2024-08-17T18:18:01.730989+0000","last_undegraded":"2024-08-17T18:17:23.796965+0000","last_fullsized":"2024-08-17T18:17:23.626843+0000","mapping_epoch":1653,"log_start":"1612'1449","ondisk_log_start":"1612'1449","created":1616,"last_epoch_clean":1632,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":6178,"log_dups_size":0,"ondisk_log_size":6178,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136347648,"num_objects":2081,"num_object_clones":0,"num_object_copies":6243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5135,"num_write_kb":230880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3354,"num_bytes_recovered":219807744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7],"avail_no_missing":["6","0","4","5"],"object_location_counts":[{"shards":"2,4,5,6","objects":1355},{"shards":"2,5,6","objects":760},{"shards":"6,7","objects":-34}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"57.f","version":"1661'10378","reported_seq":10545,"reported_epoch":1661,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:18:02.224541+0000","last_change":"2024-08-17T18:15:53.709162+0000","last_active":"2024-08-17T18:18:02.224541+0000","last_peered":"2024-08-17T18:18:02.224541+0000","last_clean":"2024-08-17T18:15:09.658885+0000","last_became_active":"2024-08-17T18:15:53.364563+0000","last_became_peered":"2024-08-17T18:15:53.364563+0000","last_unstale":"2024-08-17T18:18:02.224541+0000","last_undegraded":"2024-08-17T18:18:02.224541+0000","last_fullsized":"2024-08-17T18:18:02.224541+0000","mapping_epoch":1637,"log_start":"1625'6158","ondisk_log_start":"1625'6158","created":1604,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4220,"log_dups_size":0,"ondisk_log_size":4220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283770880,"num_objects":4330,"num_object_clones":0,"num_object_copies":12990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4325,"num_objects_unfound":0,"num_objects_dirty":4330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10378,"num_write_kb":470656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":458752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,6,0],"avail_no_missing":["2","0","6"],"object_location_counts":[{"shards":"0,2,6","objects":4330}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"57.e","version":"1661'10273","reported_seq":10355,"reported_epoch":1661,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:18:02.896971+0000","last_change":"2024-08-17T18:17:23.713508+0000","last_active":"2024-08-17T18:18:02.896971+0000","last_peered":"2024-08-17T18:18:02.896971+0000","last_clean":"2024-08-17T18:15:52.176790+0000","last_became_active":"2024-08-17T18:17:23.688213+0000","last_became_peered":"2024-08-17T18:17:23.688213+0000","last_unstale":"2024-08-17T18:18:02.896971+0000","last_undegraded":"2024-08-17T18:18:02.896971+0000","last_fullsized":"2024-08-17T18:18:02.896971+0000","mapping_epoch":1653,"log_start":"1625'6118","ondisk_log_start":"1625'6118","created":1604,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4155,"log_dups_size":0,"ondisk_log_size":4155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:12:51.202869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280166400,"num_objects":4275,"num_object_clones":0,"num_object_copies":12825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4275,"num_objects_unfound":0,"num_objects_dirty":4275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10273,"num_write_kb":465536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[4,5,6],"avail_no_missing":["4","5","6"],"object_location_counts":[{"shards":"4,5,6","objects":4275}],"blocked_by":[],"up_primary":7,"acting_primary":4,"purged_snaps":[]},{"pgid":"57.d","version":"1661'10149","reported_seq":10241,"reported_epoch":1661,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:18:01.245880+0000","last_change":"2024-08-17T18:17:23.746860+0000","last_active":"2024-08-17T18:18:01.245880+0000","last_peered":"2024-08-17T18:18:01.245880+0000","last_clean":"2024-08-17T18:17:21.525603+0000","last_became_active":"2024-08-17T18:17:23.725004+0000","last_became_peered":"2024-08-17T18:17:23.725004+0000","last_unstale":"2024-08-17T18:18:01.245880+0000","last_undegraded":"2024-08-17T18:18:01.245880+0000","last_fullsized":"2024-08-17T18:18:01.245880+0000","mapping_epoch":1653,"log_start":"1617'5227","ondisk_log_start":"1617'5227","created":1604,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4922,"log_dups_size":0,"ondisk_log_size":4922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283377688,"num_objects":4325,"num_object_clones":0,"num_object_copies":12975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4325,"num_objects_unfound":0,"num_objects_dirty":4325,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10149,"num_write_kb":463105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,6],"avail_no_missing":["0","2","6"],"object_location_counts":[{"shards":"0,2,6","objects":4325}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"57.c","version":"1661'10328","reported_seq":10395,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:18:01.742131+0000","last_change":"2024-08-17T18:13:54.756490+0000","last_active":"2024-08-17T18:18:01.742131+0000","last_peered":"2024-08-17T18:18:01.742131+0000","last_clean":"2024-08-17T18:18:01.742131+0000","last_became_active":"2024-08-17T18:13:54.756109+0000","last_became_peered":"2024-08-17T18:13:54.756109+0000","last_unstale":"2024-08-17T18:18:01.742131+0000","last_undegraded":"2024-08-17T18:18:01.742131+0000","last_fullsized":"2024-08-17T18:18:01.742131+0000","mapping_epoch":1604,"log_start":"1621'5833","ondisk_log_start":"1621'5833","created":1604,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4495,"log_dups_size":0,"ondisk_log_size":4495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:09:29.691722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290717696,"num_objects":4436,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10328,"num_write_kb":472448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"57.b","version":"1661'10158","reported_seq":10289,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:18:02.289831+0000","last_change":"2024-08-17T18:13:54.755437+0000","last_active":"2024-08-17T18:18:02.289831+0000","last_peered":"2024-08-17T18:18:02.289831+0000","last_clean":"2024-08-17T18:18:02.289831+0000","last_became_active":"2024-08-17T18:13:54.755175+0000","last_became_peered":"2024-08-17T18:13:54.755175+0000","last_unstale":"2024-08-17T18:18:02.289831+0000","last_undegraded":"2024-08-17T18:18:02.289831+0000","last_fullsized":"2024-08-17T18:18:02.289831+0000","mapping_epoch":1604,"log_start":"1625'5956","ondisk_log_start":"1625'5956","created":1604,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4202,"log_dups_size":0,"ondisk_log_size":4202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T06:08:26.806583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278528000,"num_objects":4250,"num_object_clones":0,"num_object_copies":12750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10158,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"57.a","version":"1661'10383","reported_seq":10467,"reported_epoch":1661,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:18:01.797963+0000","last_change":"2024-08-17T18:16:00.484984+0000","last_active":"2024-08-17T18:18:01.797963+0000","last_peered":"2024-08-17T18:18:01.797963+0000","last_clean":"2024-08-17T18:15:52.195344+0000","last_became_active":"2024-08-17T18:16:00.479150+0000","last_became_peered":"2024-08-17T18:16:00.479150+0000","last_unstale":"2024-08-17T18:18:01.797963+0000","last_undegraded":"2024-08-17T18:18:01.797963+0000","last_fullsized":"2024-08-17T18:18:01.797963+0000","mapping_epoch":1638,"log_start":"1630'6507","ondisk_log_start":"1630'6507","created":1604,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3876,"log_dups_size":0,"ondisk_log_size":3876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:43:11.948383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278200320,"num_objects":4245,"num_object_clones":0,"num_object_copies":12735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4245,"num_objects_unfound":0,"num_objects_dirty":4245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10383,"num_write_kb":468096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,4,2],"avail_no_missing":["5","2","4"],"object_location_counts":[{"shards":"2,4,5","objects":4245}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.9","version":"1661'7637","reported_seq":7715,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:18:02.804443+0000","last_change":"2024-08-17T18:14:49.612139+0000","last_active":"2024-08-17T18:18:02.804443+0000","last_peered":"2024-08-17T18:18:02.804443+0000","last_clean":"2024-08-17T18:18:02.804443+0000","last_became_active":"2024-08-17T18:14:49.600673+0000","last_became_peered":"2024-08-17T18:14:49.600673+0000","last_unstale":"2024-08-17T18:18:02.804443+0000","last_undegraded":"2024-08-17T18:18:02.804443+0000","last_fullsized":"2024-08-17T18:18:02.804443+0000","mapping_epoch":1608,"log_start":"1613'2030","ondisk_log_start":"1613'2030","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5607,"log_dups_size":0,"ondisk_log_size":5607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137658368,"num_objects":2100,"num_object_clones":0,"num_object_copies":6300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5158,"num_write_kb":232288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"57.8","version":"1661'7587","reported_seq":7677,"reported_epoch":1661,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:18:01.790287+0000","last_change":"2024-08-17T18:15:53.709131+0000","last_active":"2024-08-17T18:18:01.790287+0000","last_peered":"2024-08-17T18:18:01.790287+0000","last_clean":"2024-08-17T18:15:09.643183+0000","last_became_active":"2024-08-17T18:15:53.694715+0000","last_became_peered":"2024-08-17T18:15:53.694715+0000","last_unstale":"2024-08-17T18:18:01.790287+0000","last_undegraded":"2024-08-17T18:15:53.332758+0000","last_fullsized":"2024-08-17T18:15:53.222025+0000","mapping_epoch":1637,"log_start":"1613'2406","ondisk_log_start":"1613'2406","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5181,"log_dups_size":0,"ondisk_log_size":5181,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137854976,"num_objects":2103,"num_object_clones":0,"num_object_copies":6309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3551,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5079,"num_write_kb":229856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[5,6],"avail_no_missing":["5","0","6"],"object_location_counts":[{"shards":"0,5,6","objects":1640},{"shards":"4,5,6","objects":423},{"shards":"5,6","objects":-1271}],"blocked_by":[],"up_primary":7,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.7","version":"1661'7493","reported_seq":8563,"reported_epoch":1661,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:18:02.294716+0000","last_change":"2024-08-17T18:17:23.729203+0000","last_active":"2024-08-17T18:18:02.294716+0000","last_peered":"2024-08-17T18:18:02.294716+0000","last_clean":"2024-08-17T18:15:52.213196+0000","last_became_active":"2024-08-17T18:17:23.719428+0000","last_became_peered":"2024-08-17T18:17:23.719428+0000","last_unstale":"2024-08-17T18:18:02.294716+0000","last_undegraded":"2024-08-17T18:17:23.715305+0000","last_fullsized":"2024-08-17T18:17:23.631126+0000","mapping_epoch":1653,"log_start":"1613'1764","ondisk_log_start":"1613'1764","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5729,"log_dups_size":0,"ondisk_log_size":5729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137199616,"num_objects":2093,"num_object_clones":0,"num_object_copies":6279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3965,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5061,"num_write_kb":228960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":455,"num_bytes_recovered":29818880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,4,6","objects":2111},{"shards":"2,7","objects":-18}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"57.6","version":"1661'7469","reported_seq":7559,"reported_epoch":1661,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:18:01.621306+0000","last_change":"2024-08-17T18:15:53.705715+0000","last_active":"2024-08-17T18:18:01.621306+0000","last_peered":"2024-08-17T18:18:01.621306+0000","last_clean":"2024-08-17T18:15:09.669695+0000","last_became_active":"2024-08-17T18:15:53.694052+0000","last_became_peered":"2024-08-17T18:15:53.694052+0000","last_unstale":"2024-08-17T18:18:01.621306+0000","last_undegraded":"2024-08-17T18:15:53.322865+0000","last_fullsized":"2024-08-17T18:15:53.252808+0000","mapping_epoch":1637,"log_start":"1613'2289","ondisk_log_start":"1613'2289","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5180,"log_dups_size":0,"ondisk_log_size":5180,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139558912,"num_objects":2129,"num_object_clones":0,"num_object_copies":6387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3544,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5033,"num_write_kb":229216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2],"avail_no_missing":["5","0","2"],"object_location_counts":[{"shards":"0,2,5","objects":1664},{"shards":"2,4,5","objects":427},{"shards":"2,5","objects":-1216}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.5","version":"1661'7660","reported_seq":10400,"reported_epoch":1661,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:18:01.878121+0000","last_change":"2024-08-17T18:17:23.915346+0000","last_active":"2024-08-17T18:18:01.878121+0000","last_peered":"2024-08-17T18:18:01.878121+0000","last_clean":"2024-08-17T18:15:09.654427+0000","last_became_active":"2024-08-17T18:17:23.864041+0000","last_became_peered":"2024-08-17T18:17:23.864041+0000","last_unstale":"2024-08-17T18:18:01.878121+0000","last_undegraded":"2024-08-17T18:17:23.768072+0000","last_fullsized":"2024-08-17T18:18:01.878121+0000","mapping_epoch":1653,"log_start":"1613'2717","ondisk_log_start":"1613'2717","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4943,"log_dups_size":0,"ondisk_log_size":4943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138641408,"num_objects":2115,"num_object_clones":0,"num_object_copies":6345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1978,"num_objects_misplaced":2115,"num_objects_unfound":0,"num_objects_dirty":2115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5157,"num_write_kb":232736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1314,"num_bytes_recovered":86114304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,2,6],"avail_no_missing":["5","0","6"],"object_location_counts":[{"shards":"0,5,6","objects":1771},{"shards":"2,5,6","objects":344}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.4","version":"1661'7469","reported_seq":14794,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:18:01.539487+0000","last_change":"2024-08-17T18:16:24.522950+0000","last_active":"2024-08-17T18:18:01.539487+0000","last_peered":"2024-08-17T18:18:01.539487+0000","last_clean":"2024-08-17T18:18:01.539487+0000","last_became_active":"2024-08-17T18:16:24.516544+0000","last_became_peered":"2024-08-17T18:16:24.516544+0000","last_unstale":"2024-08-17T18:18:01.539487+0000","last_undegraded":"2024-08-17T18:18:01.539487+0000","last_fullsized":"2024-08-17T18:18:01.539487+0000","mapping_epoch":1646,"log_start":"1613'2369","ondisk_log_start":"1613'2369","created":1604,"last_epoch_clean":1647,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5100,"log_dups_size":0,"ondisk_log_size":5100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142213120,"num_objects":2170,"num_object_clones":0,"num_object_copies":6510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5016,"num_write_kb":229952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3591,"num_bytes_recovered":223805440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.3","version":"1661'7476","reported_seq":8481,"reported_epoch":1661,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:18:01.707603+0000","last_change":"2024-08-17T18:17:23.892338+0000","last_active":"2024-08-17T18:18:01.707603+0000","last_peered":"2024-08-17T18:18:01.707603+0000","last_clean":"2024-08-17T18:15:52.162248+0000","last_became_active":"2024-08-17T18:17:23.847915+0000","last_became_peered":"2024-08-17T18:17:23.847915+0000","last_unstale":"2024-08-17T18:18:01.707603+0000","last_undegraded":"2024-08-17T18:17:23.785246+0000","last_fullsized":"2024-08-17T18:17:23.628148+0000","mapping_epoch":1653,"log_start":"1612'1136","ondisk_log_start":"1612'1136","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":6340,"log_dups_size":0,"ondisk_log_size":6340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144343040,"num_objects":2202,"num_object_clones":0,"num_object_copies":6606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5026,"num_write_kb":231328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":430,"num_bytes_recovered":28180480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"0,2,6","objects":2225},{"shards":"6,7","objects":-23}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"57.2","version":"1661'7548","reported_seq":7975,"reported_epoch":1661,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:18:02.167194+0000","last_change":"2024-08-17T18:17:22.698625+0000","last_active":"2024-08-17T18:18:02.167194+0000","last_peered":"2024-08-17T18:18:02.167194+0000","last_clean":"2024-08-17T18:15:09.574793+0000","last_became_active":"2024-08-17T18:15:15.766379+0000","last_became_peered":"2024-08-17T18:15:15.766379+0000","last_unstale":"2024-08-17T18:18:02.167194+0000","last_undegraded":"2024-08-17T18:15:11.702342+0000","last_fullsized":"2024-08-17T18:15:11.673779+0000","mapping_epoch":1627,"log_start":"1613'1875","ondisk_log_start":"1613'1875","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5673,"log_dups_size":0,"ondisk_log_size":5673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143097856,"num_objects":2183,"num_object_clones":0,"num_object_copies":6549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5091,"num_write_kb":232800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":8781824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5],"avail_no_missing":["2","0","5"],"object_location_counts":[{"shards":"0,2,5","objects":1730},{"shards":"2,5","objects":-868}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"57.1","version":"1661'7642","reported_seq":7795,"reported_epoch":1661,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:18:01.649542+0000","last_change":"2024-08-17T18:15:53.720586+0000","last_active":"2024-08-17T18:18:01.649542+0000","last_peered":"2024-08-17T18:18:01.649542+0000","last_clean":"2024-08-17T18:15:09.579791+0000","last_became_active":"2024-08-17T18:15:53.381797+0000","last_became_peered":"2024-08-17T18:15:53.381797+0000","last_unstale":"2024-08-17T18:18:01.649542+0000","last_undegraded":"2024-08-17T18:18:01.649542+0000","last_fullsized":"2024-08-17T18:18:01.649542+0000","mapping_epoch":1637,"log_start":"1612'1421","ondisk_log_start":"1612'1421","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":6221,"log_dups_size":0,"ondisk_log_size":6221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148340736,"num_objects":2263,"num_object_clones":0,"num_object_copies":6789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2263,"num_objects_unfound":0,"num_objects_dirty":2263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5121,"num_write_kb":236320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":589824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,0],"avail_no_missing":["6","0","2"],"object_location_counts":[{"shards":"0,2,6","objects":2263}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"57.0","version":"1661'7600","reported_seq":15166,"reported_epoch":1661,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T18:18:02.307307+0000","last_change":"2024-08-17T18:17:56.095456+0000","last_active":"2024-08-17T18:18:02.307307+0000","last_peered":"2024-08-17T18:18:02.307307+0000","last_clean":"2024-08-17T18:15:52.206449+0000","last_became_active":"2024-08-17T18:15:54.490296+0000","last_became_peered":"2024-08-17T18:15:54.490296+0000","last_unstale":"2024-08-17T18:18:02.307307+0000","last_undegraded":"2024-08-17T18:15:54.249047+0000","last_fullsized":"2024-08-17T18:15:54.222754+0000","mapping_epoch":1638,"log_start":"1613'1872","ondisk_log_start":"1613'1872","created":1604,"last_epoch_clean":1636,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5728,"log_dups_size":0,"ondisk_log_size":5728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141524992,"num_objects":2159,"num_object_clones":0,"num_object_copies":6477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1194,"num_objects_misplaced":1694,"num_objects_unfound":0,"num_objects_dirty":2159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5107,"num_write_kb":232544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3696,"num_bytes_recovered":242221056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4],"avail_no_missing":["2","0","4","5"],"object_location_counts":[{"shards":"2,4","objects":-729},{"shards":"2,4,5","objects":1694}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":97,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.563380+0000","last_change":"2024-08-17T18:17:22.697769+0000","last_active":"2024-08-17T18:17:56.563380+0000","last_peered":"2024-08-17T18:17:56.563380+0000","last_clean":"2024-08-17T18:17:56.563380+0000","last_became_active":"2024-08-17T18:17:22.688989+0000","last_became_peered":"2024-08-17T18:17:22.688989+0000","last_unstale":"2024-08-17T18:17:56.563380+0000","last_undegraded":"2024-08-17T18:17:56.563380+0000","last_fullsized":"2024-08-17T18:17:56.563380+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:10:04.677811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2328,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1695,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:17:57.594734+0000","last_change":"2024-08-17T18:17:22.657973+0000","last_active":"2024-08-17T18:17:57.594734+0000","last_peered":"2024-08-17T18:17:57.594734+0000","last_clean":"2024-08-17T18:17:57.594734+0000","last_became_active":"2024-08-17T18:17:22.657722+0000","last_became_peered":"2024-08-17T18:17:22.657722+0000","last_unstale":"2024-08-17T18:17:57.594734+0000","last_undegraded":"2024-08-17T18:17:57.594734+0000","last_fullsized":"2024-08-17T18:17:57.594734+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:10:02.669115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":29,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:17:57.575849+0000","last_change":"2024-08-17T18:17:22.664262+0000","last_active":"2024-08-17T18:17:57.575849+0000","last_peered":"2024-08-17T18:17:57.575849+0000","last_clean":"2024-08-17T18:17:57.575849+0000","last_became_active":"2024-08-17T18:17:22.663863+0000","last_became_peered":"2024-08-17T18:17:22.663863+0000","last_unstale":"2024-08-17T18:17:57.575849+0000","last_undegraded":"2024-08-17T18:17:57.575849+0000","last_fullsized":"2024-08-17T18:17:57.575849+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:19:58.867528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2370,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:17:57.574890+0000","last_change":"2024-08-17T18:15:10.692079+0000","last_active":"2024-08-17T18:17:57.574890+0000","last_peered":"2024-08-17T18:17:57.574890+0000","last_clean":"2024-08-17T18:17:57.574890+0000","last_became_active":"2024-08-17T18:15:10.691698+0000","last_became_peered":"2024-08-17T18:15:10.691698+0000","last_unstale":"2024-08-17T18:17:57.574890+0000","last_undegraded":"2024-08-17T18:17:57.574890+0000","last_fullsized":"2024-08-17T18:17:57.574890+0000","mapping_epoch":1626,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:40:54.209209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":216,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.568674+0000","last_change":"2024-08-17T18:04:43.777705+0000","last_active":"2024-08-17T18:17:56.568674+0000","last_peered":"2024-08-17T18:17:56.568674+0000","last_clean":"2024-08-17T18:17:56.568674+0000","last_became_active":"2024-08-17T18:04:43.771283+0000","last_became_peered":"2024-08-17T18:04:43.771283+0000","last_unstale":"2024-08-17T18:17:56.568674+0000","last_undegraded":"2024-08-17T18:17:56.568674+0000","last_fullsized":"2024-08-17T18:17:56.568674+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:13.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1573,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:17:57.574999+0000","last_change":"2024-08-17T18:15:10.688557+0000","last_active":"2024-08-17T18:17:57.574999+0000","last_peered":"2024-08-17T18:17:57.574999+0000","last_clean":"2024-08-17T18:17:57.574999+0000","last_became_active":"2024-08-17T18:15:10.688203+0000","last_became_peered":"2024-08-17T18:15:10.688203+0000","last_unstale":"2024-08-17T18:17:57.574999+0000","last_undegraded":"2024-08-17T18:17:57.574999+0000","last_fullsized":"2024-08-17T18:17:57.574999+0000","mapping_epoch":1626,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:18:18.389795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":229,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.564212+0000","last_change":"2024-08-17T18:12:04.655551+0000","last_active":"2024-08-17T18:17:56.564212+0000","last_peered":"2024-08-17T18:17:56.564212+0000","last_clean":"2024-08-17T18:17:56.564212+0000","last_became_active":"2024-08-17T18:12:04.646075+0000","last_became_peered":"2024-08-17T18:12:04.646075+0000","last_unstale":"2024-08-17T18:17:56.564212+0000","last_undegraded":"2024-08-17T18:17:56.564212+0000","last_fullsized":"2024-08-17T18:17:56.564212+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:19:19.300135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1691,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.580595+0000","last_change":"2024-08-17T18:15:53.245777+0000","last_active":"2024-08-17T18:17:56.580595+0000","last_peered":"2024-08-17T18:17:56.580595+0000","last_clean":"2024-08-17T18:17:56.580595+0000","last_became_active":"2024-08-17T18:15:53.244780+0000","last_became_peered":"2024-08-17T18:15:53.244780+0000","last_unstale":"2024-08-17T18:17:56.580595+0000","last_undegraded":"2024-08-17T18:17:56.580595+0000","last_fullsized":"2024-08-17T18:17:56.580595+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:58:59.267941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":178,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.580561+0000","last_change":"2024-08-17T18:15:53.245927+0000","last_active":"2024-08-17T18:17:56.580561+0000","last_peered":"2024-08-17T18:17:56.580561+0000","last_clean":"2024-08-17T18:17:56.580561+0000","last_became_active":"2024-08-17T18:15:53.245507+0000","last_became_peered":"2024-08-17T18:15:53.245507+0000","last_unstale":"2024-08-17T18:17:56.580561+0000","last_undegraded":"2024-08-17T18:17:56.580561+0000","last_fullsized":"2024-08-17T18:17:56.580561+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:19:53.881065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":295,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.578963+0000","last_change":"2024-08-17T18:17:22.686658+0000","last_active":"2024-08-17T18:17:56.578963+0000","last_peered":"2024-08-17T18:17:56.578963+0000","last_clean":"2024-08-17T18:17:56.578963+0000","last_became_active":"2024-08-17T18:17:22.686168+0000","last_became_peered":"2024-08-17T18:17:22.686168+0000","last_unstale":"2024-08-17T18:17:56.578963+0000","last_undegraded":"2024-08-17T18:17:56.578963+0000","last_fullsized":"2024-08-17T18:17:56.578963+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:45:15.719551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1696,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.605379+0000","last_change":"2024-08-17T18:16:00.324143+0000","last_active":"2024-08-17T18:17:56.605379+0000","last_peered":"2024-08-17T18:17:56.605379+0000","last_clean":"2024-08-17T18:17:56.605379+0000","last_became_active":"2024-08-17T18:15:57.106000+0000","last_became_peered":"2024-08-17T18:15:57.106000+0000","last_unstale":"2024-08-17T18:17:56.605379+0000","last_undegraded":"2024-08-17T18:17:56.605379+0000","last_fullsized":"2024-08-17T18:17:56.605379+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1641,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:43.117961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2487,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.579367+0000","last_change":"2024-08-17T18:17:22.650246+0000","last_active":"2024-08-17T18:17:56.579367+0000","last_peered":"2024-08-17T18:17:56.579367+0000","last_clean":"2024-08-17T18:17:56.579367+0000","last_became_active":"2024-08-17T18:17:22.648321+0000","last_became_peered":"2024-08-17T18:17:22.648321+0000","last_unstale":"2024-08-17T18:17:56.579367+0000","last_undegraded":"2024-08-17T18:17:56.579367+0000","last_fullsized":"2024-08-17T18:17:56.579367+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:20:58.736213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":735,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.580893+0000","last_change":"2024-08-17T18:15:53.245174+0000","last_active":"2024-08-17T18:17:56.580893+0000","last_peered":"2024-08-17T18:17:56.580893+0000","last_clean":"2024-08-17T18:17:56.580893+0000","last_became_active":"2024-08-17T18:15:53.244246+0000","last_became_peered":"2024-08-17T18:15:53.244246+0000","last_unstale":"2024-08-17T18:17:56.580893+0000","last_undegraded":"2024-08-17T18:17:56.580893+0000","last_fullsized":"2024-08-17T18:17:56.580893+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:08:54.490329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":392,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.569697+0000","last_change":"2024-08-17T18:15:53.361758+0000","last_active":"2024-08-17T18:17:56.569697+0000","last_peered":"2024-08-17T18:17:56.569697+0000","last_clean":"2024-08-17T18:17:56.569697+0000","last_became_active":"2024-08-17T18:15:53.361404+0000","last_became_peered":"2024-08-17T18:15:53.361404+0000","last_unstale":"2024-08-17T18:17:56.569697+0000","last_undegraded":"2024-08-17T18:17:56.569697+0000","last_fullsized":"2024-08-17T18:17:56.569697+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:10:35.871503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2518,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:17:57.578166+0000","last_change":"2024-08-17T18:15:53.369983+0000","last_active":"2024-08-17T18:17:57.578166+0000","last_peered":"2024-08-17T18:17:57.578166+0000","last_clean":"2024-08-17T18:17:57.578166+0000","last_became_active":"2024-08-17T18:15:53.369523+0000","last_became_peered":"2024-08-17T18:15:53.369523+0000","last_unstale":"2024-08-17T18:17:57.578166+0000","last_undegraded":"2024-08-17T18:17:57.578166+0000","last_fullsized":"2024-08-17T18:17:57.578166+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:53:41.351103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1696,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.560885+0000","last_change":"2024-08-17T18:15:53.366179+0000","last_active":"2024-08-17T18:17:56.560885+0000","last_peered":"2024-08-17T18:17:56.560885+0000","last_clean":"2024-08-17T18:17:56.560885+0000","last_became_active":"2024-08-17T18:15:53.365436+0000","last_became_peered":"2024-08-17T18:15:53.365436+0000","last_unstale":"2024-08-17T18:17:56.560885+0000","last_undegraded":"2024-08-17T18:17:56.560885+0000","last_fullsized":"2024-08-17T18:17:56.560885+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:52:15.261406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2452,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.557883+0000","last_change":"2024-08-17T18:14:05.802920+0000","last_active":"2024-08-17T18:17:56.557883+0000","last_peered":"2024-08-17T18:17:56.557883+0000","last_clean":"2024-08-17T18:17:56.557883+0000","last_became_active":"2024-08-17T18:14:05.802633+0000","last_became_peered":"2024-08-17T18:14:05.802633+0000","last_unstale":"2024-08-17T18:17:56.557883+0000","last_undegraded":"2024-08-17T18:17:56.557883+0000","last_fullsized":"2024-08-17T18:17:56.557883+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:18:19.553313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":378,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.563456+0000","last_change":"2024-08-17T18:17:22.690404+0000","last_active":"2024-08-17T18:17:56.563456+0000","last_peered":"2024-08-17T18:17:56.563456+0000","last_clean":"2024-08-17T18:17:56.563456+0000","last_became_active":"2024-08-17T18:17:22.689603+0000","last_became_peered":"2024-08-17T18:17:22.689603+0000","last_unstale":"2024-08-17T18:17:56.563456+0000","last_undegraded":"2024-08-17T18:17:56.563456+0000","last_fullsized":"2024-08-17T18:17:56.563456+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:01:47.000783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2262,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.562551+0000","last_change":"2024-08-17T18:15:53.236938+0000","last_active":"2024-08-17T18:17:56.562551+0000","last_peered":"2024-08-17T18:17:56.562551+0000","last_clean":"2024-08-17T18:17:56.562551+0000","last_became_active":"2024-08-17T18:15:53.236099+0000","last_became_peered":"2024-08-17T18:15:53.236099+0000","last_unstale":"2024-08-17T18:17:56.562551+0000","last_undegraded":"2024-08-17T18:17:56.562551+0000","last_fullsized":"2024-08-17T18:17:56.562551+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:29:07.234751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":265,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.569823+0000","last_change":"2024-08-17T18:15:53.708149+0000","last_active":"2024-08-17T18:17:56.569823+0000","last_peered":"2024-08-17T18:17:56.569823+0000","last_clean":"2024-08-17T18:17:56.569823+0000","last_became_active":"2024-08-17T18:15:53.705159+0000","last_became_peered":"2024-08-17T18:15:53.705159+0000","last_unstale":"2024-08-17T18:17:56.569823+0000","last_undegraded":"2024-08-17T18:17:56.569823+0000","last_fullsized":"2024-08-17T18:17:56.569823+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:34:45.976723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":128,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.579525+0000","last_change":"2024-08-17T18:17:22.715883+0000","last_active":"2024-08-17T18:17:56.579525+0000","last_peered":"2024-08-17T18:17:56.579525+0000","last_clean":"2024-08-17T18:17:56.579525+0000","last_became_active":"2024-08-17T18:17:22.661959+0000","last_became_peered":"2024-08-17T18:17:22.661959+0000","last_unstale":"2024-08-17T18:17:56.579525+0000","last_undegraded":"2024-08-17T18:17:56.579525+0000","last_fullsized":"2024-08-17T18:17:56.579525+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:27.891227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2470,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:17:57.575497+0000","last_change":"2024-08-17T18:17:22.685988+0000","last_active":"2024-08-17T18:17:57.575497+0000","last_peered":"2024-08-17T18:17:57.575497+0000","last_clean":"2024-08-17T18:17:57.575497+0000","last_became_active":"2024-08-17T18:17:22.685791+0000","last_became_peered":"2024-08-17T18:17:22.685791+0000","last_unstale":"2024-08-17T18:17:57.575497+0000","last_undegraded":"2024-08-17T18:17:57.575497+0000","last_fullsized":"2024-08-17T18:17:57.575497+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:10:28.276954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1689,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:17:57.596447+0000","last_change":"2024-08-17T18:15:53.687128+0000","last_active":"2024-08-17T18:17:57.596447+0000","last_peered":"2024-08-17T18:17:57.596447+0000","last_clean":"2024-08-17T18:17:57.596447+0000","last_became_active":"2024-08-17T18:15:53.246153+0000","last_became_peered":"2024-08-17T18:15:53.246153+0000","last_unstale":"2024-08-17T18:17:57.596447+0000","last_undegraded":"2024-08-17T18:17:57.596447+0000","last_fullsized":"2024-08-17T18:17:57.596447+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:45:14.801100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1651,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:17:57.595174+0000","last_change":"2024-08-17T18:12:04.639269+0000","last_active":"2024-08-17T18:17:57.595174+0000","last_peered":"2024-08-17T18:17:57.595174+0000","last_clean":"2024-08-17T18:17:57.595174+0000","last_became_active":"2024-08-17T18:12:04.638752+0000","last_became_peered":"2024-08-17T18:12:04.638752+0000","last_unstale":"2024-08-17T18:17:57.595174+0000","last_undegraded":"2024-08-17T18:17:57.595174+0000","last_fullsized":"2024-08-17T18:17:57.595174+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:50:00.462972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"347'149","reported_seq":878,"reported_epoch":1660,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:17:56.564002+0000","last_change":"2024-08-17T18:15:54.275353+0000","last_active":"2024-08-17T18:17:56.564002+0000","last_peered":"2024-08-17T18:17:56.564002+0000","last_clean":"2024-08-17T18:15:50.198296+0000","last_became_active":"2024-08-17T18:15:54.274784+0000","last_became_peered":"2024-08-17T18:15:54.274784+0000","last_unstale":"2024-08-17T18:17:56.564002+0000","last_undegraded":"2024-08-17T18:15:54.237201+0000","last_fullsized":"2024-08-17T18:15:54.231871+0000","mapping_epoch":1638,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1612,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:57.271286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:57.271286+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:17:55.297120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,6],"avail_no_missing":["5","4","6"],"object_location_counts":[{"shards":"4,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1666,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.581014+0000","last_change":"2024-08-17T18:04:43.766724+0000","last_active":"2024-08-17T18:17:56.581014+0000","last_peered":"2024-08-17T18:17:56.581014+0000","last_clean":"2024-08-17T18:17:56.581014+0000","last_became_active":"2024-08-17T18:04:43.766428+0000","last_became_peered":"2024-08-17T18:04:43.766428+0000","last_unstale":"2024-08-17T18:17:56.581014+0000","last_undegraded":"2024-08-17T18:17:56.581014+0000","last_fullsized":"2024-08-17T18:17:56.581014+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:09:49.102940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1696,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.560953+0000","last_change":"2024-08-17T18:15:53.366349+0000","last_active":"2024-08-17T18:17:56.560953+0000","last_peered":"2024-08-17T18:17:56.560953+0000","last_clean":"2024-08-17T18:17:56.560953+0000","last_became_active":"2024-08-17T18:15:53.365863+0000","last_became_peered":"2024-08-17T18:15:53.365863+0000","last_unstale":"2024-08-17T18:17:56.560953+0000","last_undegraded":"2024-08-17T18:17:56.560953+0000","last_fullsized":"2024-08-17T18:17:56.560953+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:52.700374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":89,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.579530+0000","last_change":"2024-08-17T18:17:22.720026+0000","last_active":"2024-08-17T18:17:56.579530+0000","last_peered":"2024-08-17T18:17:56.579530+0000","last_clean":"2024-08-17T18:17:56.579530+0000","last_became_active":"2024-08-17T18:17:22.662636+0000","last_became_peered":"2024-08-17T18:17:22.662636+0000","last_unstale":"2024-08-17T18:17:56.579530+0000","last_undegraded":"2024-08-17T18:17:56.579530+0000","last_fullsized":"2024-08-17T18:17:56.579530+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:37:15.928482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"347'149","reported_seq":953,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.579226+0000","last_change":"2024-08-17T18:17:25.672410+0000","last_active":"2024-08-17T18:17:56.579226+0000","last_peered":"2024-08-17T18:17:56.579226+0000","last_clean":"2024-08-17T18:17:56.579226+0000","last_became_active":"2024-08-17T18:17:25.671993+0000","last_became_peered":"2024-08-17T18:17:25.671993+0000","last_unstale":"2024-08-17T18:17:56.579226+0000","last_undegraded":"2024-08-17T18:17:56.579226+0000","last_fullsized":"2024-08-17T18:17:56.579226+0000","mapping_epoch":1655,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1656,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:53.309517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:53.309517+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:21:08.929478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":35,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:17:57.574935+0000","last_change":"2024-08-17T18:15:53.392627+0000","last_active":"2024-08-17T18:17:57.574935+0000","last_peered":"2024-08-17T18:17:57.574935+0000","last_clean":"2024-08-17T18:17:57.574935+0000","last_became_active":"2024-08-17T18:15:53.391708+0000","last_became_peered":"2024-08-17T18:15:53.391708+0000","last_unstale":"2024-08-17T18:17:57.574935+0000","last_undegraded":"2024-08-17T18:17:57.574935+0000","last_fullsized":"2024-08-17T18:17:57.574935+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:59:37.120803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":739,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.560961+0000","last_change":"2024-08-17T18:15:10.698861+0000","last_active":"2024-08-17T18:17:56.560961+0000","last_peered":"2024-08-17T18:17:56.560961+0000","last_clean":"2024-08-17T18:17:56.560961+0000","last_became_active":"2024-08-17T18:15:10.698519+0000","last_became_peered":"2024-08-17T18:15:10.698519+0000","last_unstale":"2024-08-17T18:17:56.560961+0000","last_undegraded":"2024-08-17T18:17:56.560961+0000","last_fullsized":"2024-08-17T18:17:56.560961+0000","mapping_epoch":1626,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:35.743390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":2310,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.563652+0000","last_change":"2024-08-17T18:15:53.691634+0000","last_active":"2024-08-17T18:17:56.563652+0000","last_peered":"2024-08-17T18:17:56.563652+0000","last_clean":"2024-08-17T18:17:56.563652+0000","last_became_active":"2024-08-17T18:15:53.390476+0000","last_became_peered":"2024-08-17T18:15:53.390476+0000","last_unstale":"2024-08-17T18:17:56.563652+0000","last_undegraded":"2024-08-17T18:17:56.563652+0000","last_fullsized":"2024-08-17T18:17:56.563652+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:22.788856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"347'149","reported_seq":970,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.866004+0000","last_change":"2024-08-17T18:15:55.696099+0000","last_active":"2024-08-17T18:17:56.866004+0000","last_peered":"2024-08-17T18:17:56.866004+0000","last_clean":"2024-08-17T18:17:56.866004+0000","last_became_active":"2024-08-17T18:15:55.695667+0000","last_became_peered":"2024-08-17T18:15:55.695667+0000","last_unstale":"2024-08-17T18:17:56.866004+0000","last_undegraded":"2024-08-17T18:17:56.866004+0000","last_fullsized":"2024-08-17T18:17:56.866004+0000","mapping_epoch":1640,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1641,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:52.474544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:52.474544+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:16:49.496860+0000","scrub_duration":0.001886326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":276,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.562643+0000","last_change":"2024-08-17T18:17:22.690072+0000","last_active":"2024-08-17T18:17:56.562643+0000","last_peered":"2024-08-17T18:17:56.562643+0000","last_clean":"2024-08-17T18:17:56.562643+0000","last_became_active":"2024-08-17T18:17:22.688530+0000","last_became_peered":"2024-08-17T18:17:22.688530+0000","last_unstale":"2024-08-17T18:17:56.562643+0000","last_undegraded":"2024-08-17T18:17:56.562643+0000","last_fullsized":"2024-08-17T18:17:56.562643+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:40:26.170186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":34,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.573779+0000","last_change":"2024-08-17T18:15:53.248168+0000","last_active":"2024-08-17T18:17:56.573779+0000","last_peered":"2024-08-17T18:17:56.573779+0000","last_clean":"2024-08-17T18:17:56.573779+0000","last_became_active":"2024-08-17T18:15:53.247888+0000","last_became_peered":"2024-08-17T18:15:53.247888+0000","last_unstale":"2024-08-17T18:17:56.573779+0000","last_undegraded":"2024-08-17T18:17:56.573779+0000","last_fullsized":"2024-08-17T18:17:56.573779+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:30:42.725233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2498,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:17:57.575965+0000","last_change":"2024-08-17T18:15:10.692541+0000","last_active":"2024-08-17T18:17:57.575965+0000","last_peered":"2024-08-17T18:17:57.575965+0000","last_clean":"2024-08-17T18:17:57.575965+0000","last_became_active":"2024-08-17T18:15:10.691876+0000","last_became_peered":"2024-08-17T18:15:10.691876+0000","last_unstale":"2024-08-17T18:17:57.575965+0000","last_undegraded":"2024-08-17T18:17:57.575965+0000","last_fullsized":"2024-08-17T18:17:57.575965+0000","mapping_epoch":1626,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:39:22.253615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"347'149","reported_seq":953,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.866023+0000","last_change":"2024-08-17T18:15:55.699890+0000","last_active":"2024-08-17T18:17:56.866023+0000","last_peered":"2024-08-17T18:17:56.866023+0000","last_clean":"2024-08-17T18:17:56.866023+0000","last_became_active":"2024-08-17T18:15:55.699294+0000","last_became_peered":"2024-08-17T18:15:55.699294+0000","last_unstale":"2024-08-17T18:17:56.866023+0000","last_undegraded":"2024-08-17T18:17:56.866023+0000","last_fullsized":"2024-08-17T18:17:56.866023+0000","mapping_epoch":1640,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1641,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:49.461698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:49.461698+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:20:25.524119+0000","scrub_duration":0.0013792520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":226,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.563450+0000","last_change":"2024-08-17T18:17:22.687498+0000","last_active":"2024-08-17T18:17:56.563450+0000","last_peered":"2024-08-17T18:17:56.563450+0000","last_clean":"2024-08-17T18:17:56.563450+0000","last_became_active":"2024-08-17T18:17:22.687176+0000","last_became_peered":"2024-08-17T18:17:22.687176+0000","last_unstale":"2024-08-17T18:17:56.563450+0000","last_undegraded":"2024-08-17T18:17:56.563450+0000","last_fullsized":"2024-08-17T18:17:56.563450+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:32:42.738739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2507,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:17:57.617135+0000","last_change":"2024-08-17T18:15:53.230320+0000","last_active":"2024-08-17T18:17:57.617135+0000","last_peered":"2024-08-17T18:17:57.617135+0000","last_clean":"2024-08-17T18:17:57.617135+0000","last_became_active":"2024-08-17T18:15:53.230016+0000","last_became_peered":"2024-08-17T18:15:53.230016+0000","last_unstale":"2024-08-17T18:17:57.617135+0000","last_undegraded":"2024-08-17T18:17:57.617135+0000","last_fullsized":"2024-08-17T18:17:57.617135+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:15:28.925103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":222,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.568756+0000","last_change":"2024-08-17T18:14:05.817143+0000","last_active":"2024-08-17T18:17:56.568756+0000","last_peered":"2024-08-17T18:17:56.568756+0000","last_clean":"2024-08-17T18:17:56.568756+0000","last_became_active":"2024-08-17T18:14:05.816518+0000","last_became_peered":"2024-08-17T18:14:05.816518+0000","last_unstale":"2024-08-17T18:17:56.568756+0000","last_undegraded":"2024-08-17T18:17:56.568756+0000","last_fullsized":"2024-08-17T18:17:56.568756+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:29:14.833759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"347'149","reported_seq":954,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.564715+0000","last_change":"2024-08-17T18:15:13.681264+0000","last_active":"2024-08-17T18:17:56.564715+0000","last_peered":"2024-08-17T18:17:56.564715+0000","last_clean":"2024-08-17T18:17:56.564715+0000","last_became_active":"2024-08-17T18:15:13.680918+0000","last_became_peered":"2024-08-17T18:15:13.680918+0000","last_unstale":"2024-08-17T18:17:56.564715+0000","last_undegraded":"2024-08-17T18:17:56.564715+0000","last_fullsized":"2024-08-17T18:17:56.564715+0000","mapping_epoch":1629,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1630,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:00.316884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:00.316884+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:37:32.924833+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1639,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:17:57.575832+0000","last_change":"2024-08-17T18:17:22.689614+0000","last_active":"2024-08-17T18:17:57.575832+0000","last_peered":"2024-08-17T18:17:57.575832+0000","last_clean":"2024-08-17T18:17:57.575832+0000","last_became_active":"2024-08-17T18:17:22.689385+0000","last_became_peered":"2024-08-17T18:17:22.689385+0000","last_unstale":"2024-08-17T18:17:57.575832+0000","last_undegraded":"2024-08-17T18:17:57.575832+0000","last_fullsized":"2024-08-17T18:17:57.575832+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:27:20.931230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1673,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.581240+0000","last_change":"2024-08-17T18:15:10.695565+0000","last_active":"2024-08-17T18:17:56.581240+0000","last_peered":"2024-08-17T18:17:56.581240+0000","last_clean":"2024-08-17T18:17:56.581240+0000","last_became_active":"2024-08-17T18:15:10.695225+0000","last_became_peered":"2024-08-17T18:15:10.695225+0000","last_unstale":"2024-08-17T18:17:56.581240+0000","last_undegraded":"2024-08-17T18:17:56.581240+0000","last_fullsized":"2024-08-17T18:17:56.581240+0000","mapping_epoch":1626,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:28:55.675702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":259,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.563627+0000","last_change":"2024-08-17T18:15:53.712282+0000","last_active":"2024-08-17T18:17:56.563627+0000","last_peered":"2024-08-17T18:17:56.563627+0000","last_clean":"2024-08-17T18:17:56.563627+0000","last_became_active":"2024-08-17T18:15:53.302235+0000","last_became_peered":"2024-08-17T18:15:53.302235+0000","last_unstale":"2024-08-17T18:17:56.563627+0000","last_undegraded":"2024-08-17T18:17:56.563627+0000","last_fullsized":"2024-08-17T18:17:56.563627+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:25:36.767529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":95,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"347'149","reported_seq":814,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.605225+0000","last_change":"2024-08-17T18:17:22.681616+0000","last_active":"2024-08-17T18:17:56.605225+0000","last_peered":"2024-08-17T18:17:56.605225+0000","last_clean":"2024-08-17T18:17:56.605225+0000","last_became_active":"2024-08-17T18:17:22.681184+0000","last_became_peered":"2024-08-17T18:17:22.681184+0000","last_unstale":"2024-08-17T18:17:56.605225+0000","last_undegraded":"2024-08-17T18:17:56.605225+0000","last_fullsized":"2024-08-17T18:17:56.605225+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:02.363531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:02.363531+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:50:39.567707+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2413,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.563525+0000","last_change":"2024-08-17T18:15:53.393779+0000","last_active":"2024-08-17T18:17:56.563525+0000","last_peered":"2024-08-17T18:17:56.563525+0000","last_clean":"2024-08-17T18:17:56.563525+0000","last_became_active":"2024-08-17T18:15:53.390928+0000","last_became_peered":"2024-08-17T18:15:53.390928+0000","last_unstale":"2024-08-17T18:17:56.563525+0000","last_undegraded":"2024-08-17T18:17:56.563525+0000","last_fullsized":"2024-08-17T18:17:56.563525+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:46:51.327206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2319,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:17:57.578369+0000","last_change":"2024-08-17T18:12:04.648190+0000","last_active":"2024-08-17T18:17:57.578369+0000","last_peered":"2024-08-17T18:17:57.578369+0000","last_clean":"2024-08-17T18:17:57.578369+0000","last_became_active":"2024-08-17T18:12:04.647877+0000","last_became_peered":"2024-08-17T18:12:04.647877+0000","last_unstale":"2024-08-17T18:17:57.578369+0000","last_undegraded":"2024-08-17T18:17:57.578369+0000","last_fullsized":"2024-08-17T18:17:57.578369+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:18.129337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2615,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:17:57.574895+0000","last_change":"2024-08-17T18:14:05.811655+0000","last_active":"2024-08-17T18:17:57.574895+0000","last_peered":"2024-08-17T18:17:57.574895+0000","last_clean":"2024-08-17T18:17:57.574895+0000","last_became_active":"2024-08-17T18:14:05.808974+0000","last_became_peered":"2024-08-17T18:14:05.808974+0000","last_unstale":"2024-08-17T18:17:57.574895+0000","last_undegraded":"2024-08-17T18:17:57.574895+0000","last_fullsized":"2024-08-17T18:17:57.574895+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:37:31.935317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"347'149","reported_seq":663,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.580404+0000","last_change":"2024-08-17T18:17:25.671061+0000","last_active":"2024-08-17T18:17:56.580404+0000","last_peered":"2024-08-17T18:17:56.580404+0000","last_clean":"2024-08-17T18:17:56.580404+0000","last_became_active":"2024-08-17T18:17:25.670747+0000","last_became_peered":"2024-08-17T18:17:25.670747+0000","last_unstale":"2024-08-17T18:17:56.580404+0000","last_undegraded":"2024-08-17T18:17:56.580404+0000","last_fullsized":"2024-08-17T18:17:56.580404+0000","mapping_epoch":1655,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1656,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:12:53.586672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:12:53.586672+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":214680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":314,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:17:57.575710+0000","last_change":"2024-08-17T18:15:53.370263+0000","last_active":"2024-08-17T18:17:57.575710+0000","last_peered":"2024-08-17T18:17:57.575710+0000","last_clean":"2024-08-17T18:17:57.575710+0000","last_became_active":"2024-08-17T18:15:53.369971+0000","last_became_peered":"2024-08-17T18:15:53.369971+0000","last_unstale":"2024-08-17T18:17:57.575710+0000","last_undegraded":"2024-08-17T18:17:57.575710+0000","last_fullsized":"2024-08-17T18:17:57.575710+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:51:08.633886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":828,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.580742+0000","last_change":"2024-08-17T18:15:53.287221+0000","last_active":"2024-08-17T18:17:56.580742+0000","last_peered":"2024-08-17T18:17:56.580742+0000","last_clean":"2024-08-17T18:17:56.580742+0000","last_became_active":"2024-08-17T18:15:53.286808+0000","last_became_peered":"2024-08-17T18:15:53.286808+0000","last_unstale":"2024-08-17T18:17:56.580742+0000","last_undegraded":"2024-08-17T18:17:56.580742+0000","last_fullsized":"2024-08-17T18:17:56.580742+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:09:11.420394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1127,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.565749+0000","last_change":"2024-08-17T18:15:53.338306+0000","last_active":"2024-08-17T18:17:56.565749+0000","last_peered":"2024-08-17T18:17:56.565749+0000","last_clean":"2024-08-17T18:17:56.565749+0000","last_became_active":"2024-08-17T18:15:53.337578+0000","last_became_peered":"2024-08-17T18:15:53.337578+0000","last_unstale":"2024-08-17T18:17:56.565749+0000","last_undegraded":"2024-08-17T18:17:56.565749+0000","last_fullsized":"2024-08-17T18:17:56.565749+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:33.533415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1499,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.579650+0000","last_change":"2024-08-17T18:17:22.661436+0000","last_active":"2024-08-17T18:17:56.579650+0000","last_peered":"2024-08-17T18:17:56.579650+0000","last_clean":"2024-08-17T18:17:56.579650+0000","last_became_active":"2024-08-17T18:17:22.661156+0000","last_became_peered":"2024-08-17T18:17:22.661156+0000","last_unstale":"2024-08-17T18:17:56.579650+0000","last_undegraded":"2024-08-17T18:17:56.579650+0000","last_fullsized":"2024-08-17T18:17:56.579650+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:59:55.521842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":92,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.563377+0000","last_change":"2024-08-17T18:14:05.815709+0000","last_active":"2024-08-17T18:17:56.563377+0000","last_peered":"2024-08-17T18:17:56.563377+0000","last_clean":"2024-08-17T18:17:56.563377+0000","last_became_active":"2024-08-17T18:14:05.815411+0000","last_became_peered":"2024-08-17T18:14:05.815411+0000","last_unstale":"2024-08-17T18:17:56.563377+0000","last_undegraded":"2024-08-17T18:17:56.563377+0000","last_fullsized":"2024-08-17T18:17:56.563377+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:56:03.062386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":242,"reported_epoch":1660,"state":"active+recovery_wait","last_fresh":"2024-08-17T18:17:56.562743+0000","last_change":"2024-08-17T18:15:54.221085+0000","last_active":"2024-08-17T18:17:56.562743+0000","last_peered":"2024-08-17T18:17:56.562743+0000","last_clean":"2024-08-17T18:15:50.189470+0000","last_became_active":"2024-08-17T18:15:53.304106+0000","last_became_peered":"2024-08-17T18:15:53.304106+0000","last_unstale":"2024-08-17T18:17:56.562743+0000","last_undegraded":"2024-08-17T18:17:56.562743+0000","last_fullsized":"2024-08-17T18:17:56.562743+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:21.409368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":["5","2","6","7"],"object_location_counts":[{"shards":"2,5,7","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":840,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.561400+0000","last_change":"2024-08-17T18:17:16.835493+0000","last_active":"2024-08-17T18:17:56.561400+0000","last_peered":"2024-08-17T18:17:56.561400+0000","last_clean":"2024-08-17T18:17:56.561400+0000","last_became_active":"2024-08-17T18:15:55.698769+0000","last_became_peered":"2024-08-17T18:15:55.698769+0000","last_unstale":"2024-08-17T18:17:56.561400+0000","last_undegraded":"2024-08-17T18:17:56.561400+0000","last_fullsized":"2024-08-17T18:17:56.561400+0000","mapping_epoch":1640,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1641,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:54.303708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:54.303708+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":74,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"347'149","reported_seq":938,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:17:57.566374+0000","last_change":"2024-08-17T18:09:15.592722+0000","last_active":"2024-08-17T18:17:57.566374+0000","last_peered":"2024-08-17T18:17:57.566374+0000","last_clean":"2024-08-17T18:17:57.566374+0000","last_became_active":"2024-08-17T18:09:15.592242+0000","last_became_peered":"2024-08-17T18:09:15.592242+0000","last_unstale":"2024-08-17T18:17:57.566374+0000","last_undegraded":"2024-08-17T18:17:57.566374+0000","last_fullsized":"2024-08-17T18:17:57.566374+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:55.255983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:55.255983+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:02:53.596275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":715,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.580573+0000","last_change":"2024-08-17T18:15:53.287283+0000","last_active":"2024-08-17T18:17:56.580573+0000","last_peered":"2024-08-17T18:17:56.580573+0000","last_clean":"2024-08-17T18:17:56.580573+0000","last_became_active":"2024-08-17T18:15:53.286962+0000","last_became_peered":"2024-08-17T18:15:53.286962+0000","last_unstale":"2024-08-17T18:17:56.580573+0000","last_undegraded":"2024-08-17T18:17:56.580573+0000","last_fullsized":"2024-08-17T18:17:56.580573+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:45:24.433251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":34,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.574360+0000","last_change":"2024-08-17T18:15:53.392789+0000","last_active":"2024-08-17T18:17:56.574360+0000","last_peered":"2024-08-17T18:17:56.574360+0000","last_clean":"2024-08-17T18:17:56.574360+0000","last_became_active":"2024-08-17T18:15:53.392099+0000","last_became_peered":"2024-08-17T18:15:53.392099+0000","last_unstale":"2024-08-17T18:17:56.574360+0000","last_undegraded":"2024-08-17T18:17:56.574360+0000","last_fullsized":"2024-08-17T18:17:56.574360+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:31:53.051105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1688,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.579607+0000","last_change":"2024-08-17T18:17:22.663907+0000","last_active":"2024-08-17T18:17:56.579607+0000","last_peered":"2024-08-17T18:17:56.579607+0000","last_clean":"2024-08-17T18:17:56.579607+0000","last_became_active":"2024-08-17T18:17:22.663591+0000","last_became_peered":"2024-08-17T18:17:22.663591+0000","last_unstale":"2024-08-17T18:17:56.579607+0000","last_undegraded":"2024-08-17T18:17:56.579607+0000","last_fullsized":"2024-08-17T18:17:56.579607+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:18:05.897999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1687,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.581086+0000","last_change":"2024-08-17T18:15:53.244851+0000","last_active":"2024-08-17T18:17:56.581086+0000","last_peered":"2024-08-17T18:17:56.581086+0000","last_clean":"2024-08-17T18:17:56.581086+0000","last_became_active":"2024-08-17T18:15:53.240074+0000","last_became_peered":"2024-08-17T18:15:53.240074+0000","last_unstale":"2024-08-17T18:17:56.581086+0000","last_undegraded":"2024-08-17T18:17:56.581086+0000","last_fullsized":"2024-08-17T18:17:56.581086+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:09:00.168413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"347'149","reported_seq":818,"reported_epoch":1661,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:17:57.578231+0000","last_change":"2024-08-17T18:17:23.643164+0000","last_active":"2024-08-17T18:17:57.578231+0000","last_peered":"2024-08-17T18:17:57.578231+0000","last_clean":"2024-08-17T18:09:10.546471+0000","last_became_active":"2024-08-17T18:17:23.642817+0000","last_became_peered":"2024-08-17T18:17:23.642817+0000","last_unstale":"2024-08-17T18:17:57.578231+0000","last_undegraded":"2024-08-17T18:17:23.630159+0000","last_fullsized":"2024-08-17T18:17:23.628965+0000","mapping_epoch":1653,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1612,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:59.307356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:59.307356+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,0],"avail_no_missing":["2","5","6"],"object_location_counts":[{"shards":"2,5,6","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1682,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.579437+0000","last_change":"2024-08-17T18:12:04.648228+0000","last_active":"2024-08-17T18:17:56.579437+0000","last_peered":"2024-08-17T18:17:56.579437+0000","last_clean":"2024-08-17T18:17:56.579437+0000","last_became_active":"2024-08-17T18:12:04.647906+0000","last_became_peered":"2024-08-17T18:12:04.647906+0000","last_unstale":"2024-08-17T18:17:56.579437+0000","last_undegraded":"2024-08-17T18:17:56.579437+0000","last_fullsized":"2024-08-17T18:17:56.579437+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:51:30.081540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1660,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.558870+0000","last_change":"2024-08-17T18:14:05.802624+0000","last_active":"2024-08-17T18:17:56.558870+0000","last_peered":"2024-08-17T18:17:56.558870+0000","last_clean":"2024-08-17T18:17:56.558870+0000","last_became_active":"2024-08-17T18:14:05.802177+0000","last_became_peered":"2024-08-17T18:14:05.802177+0000","last_unstale":"2024-08-17T18:17:56.558870+0000","last_undegraded":"2024-08-17T18:17:56.558870+0000","last_fullsized":"2024-08-17T18:17:56.558870+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:49:07.674636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"347'149","reported_seq":896,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.573454+0000","last_change":"2024-08-17T18:15:55.705146+0000","last_active":"2024-08-17T18:17:56.573454+0000","last_peered":"2024-08-17T18:17:56.573454+0000","last_clean":"2024-08-17T18:17:56.573454+0000","last_became_active":"2024-08-17T18:15:55.704757+0000","last_became_peered":"2024-08-17T18:15:55.704757+0000","last_unstale":"2024-08-17T18:17:56.573454+0000","last_undegraded":"2024-08-17T18:17:56.573454+0000","last_fullsized":"2024-08-17T18:17:56.573454+0000","mapping_epoch":1640,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1641,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:52.271487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:52.271487+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:45:10.264823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1661,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.579733+0000","last_change":"2024-08-17T18:15:53.245213+0000","last_active":"2024-08-17T18:17:56.579733+0000","last_peered":"2024-08-17T18:17:56.579733+0000","last_clean":"2024-08-17T18:17:56.579733+0000","last_became_active":"2024-08-17T18:15:53.244536+0000","last_became_peered":"2024-08-17T18:15:53.244536+0000","last_unstale":"2024-08-17T18:17:56.579733+0000","last_undegraded":"2024-08-17T18:17:56.579733+0000","last_fullsized":"2024-08-17T18:17:56.579733+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:42:59.928191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":76,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.580740+0000","last_change":"2024-08-17T18:15:53.327899+0000","last_active":"2024-08-17T18:17:56.580740+0000","last_peered":"2024-08-17T18:17:56.580740+0000","last_clean":"2024-08-17T18:17:56.580740+0000","last_became_active":"2024-08-17T18:15:53.246161+0000","last_became_peered":"2024-08-17T18:15:53.246161+0000","last_unstale":"2024-08-17T18:17:56.580740+0000","last_undegraded":"2024-08-17T18:17:56.580740+0000","last_fullsized":"2024-08-17T18:17:56.580740+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:17:31.392739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":706,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.580484+0000","last_change":"2024-08-17T18:15:10.689631+0000","last_active":"2024-08-17T18:17:56.580484+0000","last_peered":"2024-08-17T18:17:56.580484+0000","last_clean":"2024-08-17T18:17:56.580484+0000","last_became_active":"2024-08-17T18:15:10.689080+0000","last_became_peered":"2024-08-17T18:15:10.689080+0000","last_unstale":"2024-08-17T18:17:56.580484+0000","last_undegraded":"2024-08-17T18:17:56.580484+0000","last_fullsized":"2024-08-17T18:17:56.580484+0000","mapping_epoch":1626,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:51:34.103410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":230,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.564283+0000","last_change":"2024-08-17T18:15:10.702942+0000","last_active":"2024-08-17T18:17:56.564283+0000","last_peered":"2024-08-17T18:17:56.564283+0000","last_clean":"2024-08-17T18:17:56.564283+0000","last_became_active":"2024-08-17T18:15:10.702621+0000","last_became_peered":"2024-08-17T18:15:10.702621+0000","last_unstale":"2024-08-17T18:17:56.564283+0000","last_undegraded":"2024-08-17T18:17:56.564283+0000","last_fullsized":"2024-08-17T18:17:56.564283+0000","mapping_epoch":1626,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:02:03.856185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1480,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.563296+0000","last_change":"2024-08-17T18:17:22.676810+0000","last_active":"2024-08-17T18:17:56.563296+0000","last_peered":"2024-08-17T18:17:56.563296+0000","last_clean":"2024-08-17T18:17:56.563296+0000","last_became_active":"2024-08-17T18:17:22.676441+0000","last_became_peered":"2024-08-17T18:17:22.676441+0000","last_unstale":"2024-08-17T18:17:56.563296+0000","last_undegraded":"2024-08-17T18:17:56.563296+0000","last_fullsized":"2024-08-17T18:17:56.563296+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:27.799139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":242,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.568914+0000","last_change":"2024-08-17T18:15:53.661488+0000","last_active":"2024-08-17T18:17:56.568914+0000","last_peered":"2024-08-17T18:17:56.568914+0000","last_clean":"2024-08-17T18:17:56.568914+0000","last_became_active":"2024-08-17T18:15:53.308611+0000","last_became_peered":"2024-08-17T18:15:53.308611+0000","last_unstale":"2024-08-17T18:17:56.568914+0000","last_undegraded":"2024-08-17T18:17:56.568914+0000","last_fullsized":"2024-08-17T18:17:56.568914+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:46:37.056632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1695,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.580821+0000","last_change":"2024-08-17T18:14:05.809877+0000","last_active":"2024-08-17T18:17:56.580821+0000","last_peered":"2024-08-17T18:17:56.580821+0000","last_clean":"2024-08-17T18:17:56.580821+0000","last_became_active":"2024-08-17T18:14:05.809582+0000","last_became_peered":"2024-08-17T18:14:05.809582+0000","last_unstale":"2024-08-17T18:17:56.580821+0000","last_undegraded":"2024-08-17T18:17:56.580821+0000","last_fullsized":"2024-08-17T18:17:56.580821+0000","mapping_epoch":1608,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:20.740796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":38,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.573934+0000","last_change":"2024-08-17T18:15:53.708176+0000","last_active":"2024-08-17T18:17:56.573934+0000","last_peered":"2024-08-17T18:17:56.573934+0000","last_clean":"2024-08-17T18:17:56.573934+0000","last_became_active":"2024-08-17T18:15:53.669563+0000","last_became_peered":"2024-08-17T18:15:53.669563+0000","last_unstale":"2024-08-17T18:17:56.573934+0000","last_undegraded":"2024-08-17T18:17:56.573934+0000","last_fullsized":"2024-08-17T18:17:56.573934+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:52:17.018687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":249,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.565958+0000","last_change":"2024-08-17T18:17:22.687310+0000","last_active":"2024-08-17T18:17:56.565958+0000","last_peered":"2024-08-17T18:17:56.565958+0000","last_clean":"2024-08-17T18:17:56.565958+0000","last_became_active":"2024-08-17T18:17:22.686966+0000","last_became_peered":"2024-08-17T18:17:22.686966+0000","last_unstale":"2024-08-17T18:17:56.565958+0000","last_undegraded":"2024-08-17T18:17:56.565958+0000","last_fullsized":"2024-08-17T18:17:56.565958+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:46:11.568465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":38,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.574400+0000","last_change":"2024-08-17T18:15:53.461774+0000","last_active":"2024-08-17T18:17:56.574400+0000","last_peered":"2024-08-17T18:17:56.574400+0000","last_clean":"2024-08-17T18:17:56.574400+0000","last_became_active":"2024-08-17T18:15:53.287744+0000","last_became_peered":"2024-08-17T18:15:53.287744+0000","last_unstale":"2024-08-17T18:17:56.574400+0000","last_undegraded":"2024-08-17T18:17:56.574400+0000","last_fullsized":"2024-08-17T18:17:56.574400+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:25.168664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1762,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.561314+0000","last_change":"2024-08-17T18:17:22.712061+0000","last_active":"2024-08-17T18:17:56.561314+0000","last_peered":"2024-08-17T18:17:56.561314+0000","last_clean":"2024-08-17T18:17:56.561314+0000","last_became_active":"2024-08-17T18:17:22.650005+0000","last_became_peered":"2024-08-17T18:17:22.650005+0000","last_unstale":"2024-08-17T18:17:56.561314+0000","last_undegraded":"2024-08-17T18:17:56.561314+0000","last_fullsized":"2024-08-17T18:17:56.561314+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:08:00.180964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2310,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:17:57.579765+0000","last_change":"2024-08-17T18:15:53.364411+0000","last_active":"2024-08-17T18:17:57.579765+0000","last_peered":"2024-08-17T18:17:57.579765+0000","last_clean":"2024-08-17T18:17:57.579765+0000","last_became_active":"2024-08-17T18:15:53.364157+0000","last_became_peered":"2024-08-17T18:15:53.364157+0000","last_unstale":"2024-08-17T18:17:57.579765+0000","last_undegraded":"2024-08-17T18:17:57.579765+0000","last_fullsized":"2024-08-17T18:17:57.579765+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:53:25.874429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":400,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.580956+0000","last_change":"2024-08-17T18:15:53.245247+0000","last_active":"2024-08-17T18:17:56.580956+0000","last_peered":"2024-08-17T18:17:56.580956+0000","last_clean":"2024-08-17T18:17:56.580956+0000","last_became_active":"2024-08-17T18:15:53.244627+0000","last_became_peered":"2024-08-17T18:15:53.244627+0000","last_unstale":"2024-08-17T18:17:56.580956+0000","last_undegraded":"2024-08-17T18:17:56.580956+0000","last_fullsized":"2024-08-17T18:17:56.580956+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:42:03.556013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":927,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:17:57.575932+0000","last_change":"2024-08-17T18:15:53.237223+0000","last_active":"2024-08-17T18:17:57.575932+0000","last_peered":"2024-08-17T18:17:57.575932+0000","last_clean":"2024-08-17T18:17:57.575932+0000","last_became_active":"2024-08-17T18:15:53.236868+0000","last_became_peered":"2024-08-17T18:15:53.236868+0000","last_unstale":"2024-08-17T18:17:57.575932+0000","last_undegraded":"2024-08-17T18:17:57.575932+0000","last_fullsized":"2024-08-17T18:17:57.575932+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:02:32.603791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":432,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:17:57.575110+0000","last_change":"2024-08-17T18:17:22.651676+0000","last_active":"2024-08-17T18:17:57.575110+0000","last_peered":"2024-08-17T18:17:57.575110+0000","last_clean":"2024-08-17T18:17:57.575110+0000","last_became_active":"2024-08-17T18:17:22.651327+0000","last_became_peered":"2024-08-17T18:17:22.651327+0000","last_unstale":"2024-08-17T18:17:57.575110+0000","last_undegraded":"2024-08-17T18:17:57.575110+0000","last_fullsized":"2024-08-17T18:17:57.575110+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:29:41.898158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":239,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.568833+0000","last_change":"2024-08-17T18:15:53.308968+0000","last_active":"2024-08-17T18:17:56.568833+0000","last_peered":"2024-08-17T18:17:56.568833+0000","last_clean":"2024-08-17T18:17:56.568833+0000","last_became_active":"2024-08-17T18:15:53.308603+0000","last_became_peered":"2024-08-17T18:15:53.308603+0000","last_unstale":"2024-08-17T18:17:56.568833+0000","last_undegraded":"2024-08-17T18:17:56.568833+0000","last_fullsized":"2024-08-17T18:17:56.568833+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:27:09.638894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":40,"reported_epoch":1660,"state":"active+clean","last_fresh":"2024-08-17T18:17:56.575012+0000","last_change":"2024-08-17T18:15:53.271113+0000","last_active":"2024-08-17T18:17:56.575012+0000","last_peered":"2024-08-17T18:17:56.575012+0000","last_clean":"2024-08-17T18:17:56.575012+0000","last_became_active":"2024-08-17T18:15:53.230983+0000","last_became_peered":"2024-08-17T18:15:53.230983+0000","last_unstale":"2024-08-17T18:17:56.575012+0000","last_undegraded":"2024-08-17T18:17:56.575012+0000","last_fullsized":"2024-08-17T18:17:56.575012+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:00:35.315798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2519,"reported_epoch":1661,"state":"active+clean","last_fresh":"2024-08-17T18:17:57.575670+0000","last_change":"2024-08-17T18:15:53.369784+0000","last_active":"2024-08-17T18:17:57.575670+0000","last_peered":"2024-08-17T18:17:57.575670+0000","last_clean":"2024-08-17T18:17:57.575670+0000","last_became_active":"2024-08-17T18:15:53.369444+0000","last_became_peered":"2024-08-17T18:15:53.369444+0000","last_unstale":"2024-08-17T18:17:57.575670+0000","last_undegraded":"2024-08-17T18:17:57.575670+0000","last_fullsized":"2024-08-17T18:17:57.575670+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:50.591133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":57,"num_pg":26,"stat_sum":{"num_bytes":4500029464,"num_objects":68666,"num_object_clones":0,"num_object_copies":205998,"num_objects_missing_on_primary":1372,"num_objects_missing":1372,"num_objects_degraded":48043,"num_objects_misplaced":27806,"num_objects_unfound":0,"num_objects_dirty":68666,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163174,"num_write_kb":7418817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20648,"num_bytes_recovered":1275592704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15720722432,"data_stored":15720710216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139264,"ondisk_log_size":139264,"up":78,"acting":66,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":697707,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":149,"num_write":39,"num_write_kb":439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":810709,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":31,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":95,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":95,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":12532,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":22204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234824,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852292,"kb_used_data":307708,"kb_used_omap":1,"kb_used_meta":544574,"kb_avail":92892860,"statfs":{"total":95995035648,"available":95122288640,"internally_reserved":0,"allocated":315092992,"data_stored":310799858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":557644235},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496462,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4527236,"kb_used_data":3298940,"kb_used_omap":1,"kb_used_meta":1228286,"kb_avail":89217916,"statfs":{"total":95995035648,"available":91359145984,"internally_reserved":0,"allocated":3378114560,"data_stored":3373822430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1257765319},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":2946347565703,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":1399,"seq":6008659247305,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3947212,"kb_used_data":3269508,"kb_used_omap":131,"kb_used_meta":677564,"kb_avail":89797940,"statfs":{"total":95995035648,"available":91953090560,"internally_reserved":0,"allocated":3347976192,"data_stored":3343675958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134291,"internal_metadata":693826413},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":2864743187093,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448732,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4221352,"kb_used_data":3441184,"kb_used_omap":75,"kb_used_meta":780084,"kb_avail":89523800,"statfs":{"total":95995035648,"available":91672371200,"internally_reserved":0,"allocated":3523772416,"data_stored":3519502999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":798806207},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208758097,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3854904,"kb_used_data":2582768,"kb_used_omap":1,"kb_used_meta":1272126,"kb_avail":89890248,"statfs":{"total":95995035648,"available":92047613952,"internally_reserved":0,"allocated":2644754432,"data_stored":2640469832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1302657478},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019734,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3638656,"kb_used_data":2538680,"kb_used_omap":1,"kb_used_meta":1099966,"kb_avail":90106496,"statfs":{"total":95995035648,"available":92269051904,"internally_reserved":0,"allocated":2599608320,"data_stored":2595319775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1126365638},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,0,1,1,3,3,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3502051328,"data_stored":3502047256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3331133440,"data_stored":3331129368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2580545536,"data_stored":2580545536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2634022912,"data_stored":2634022912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3367112704,"data_stored":3367108632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":305856512,"data_stored":305856512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T18:18:07.157 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool set unique_pool_4 pgp_num 26 2024-08-17T18:18:07.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:07.198 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:07.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:07.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:07.300 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:07.335 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:07.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:07.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:07.351 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:18:07.357 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:18:07.437 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:07.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:07.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:18:07.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:07 smithi136 ceph-mon[115078]: from='client.36699 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:18:07.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:07 smithi136 ceph-mon[115078]: pgmap v3180: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 415 op/s; 48046/205509 objects degraded (23.379%); 27549/205509 objects misplaced (13.405%); 11 MiB/s, 258 objects/s recovering 2024-08-17T18:18:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:18:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:07 smithi136 ceph-mon[109861]: from='client.36699 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:18:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:07 smithi136 ceph-mon[109861]: pgmap v3180: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 415 op/s; 48046/205509 objects degraded (23.379%); 27549/205509 objects misplaced (13.405%); 11 MiB/s, 258 objects/s recovering 2024-08-17T18:18:07.462 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:07.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:07.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:07.518 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:18:07.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:07 smithi165 ceph-mon[105517]: from='client.36699 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:18:07.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:07 smithi165 ceph-mon[105517]: pgmap v3180: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 415 op/s; 48046/205509 objects degraded (23.379%); 27549/205509 objects misplaced (13.405%); 11 MiB/s, 258 objects/s recovering 2024-08-17T18:18:07.563 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:07.588 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:07.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:07.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:07.689 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:07.710 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:18:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:18:07] "GET /metrics HTTP/1.1" 200 36776 "" "Prometheus/2.51.0" 2024-08-17T18:18:07.717 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:07.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:07.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:07.819 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:07.847 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:07.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:07.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:07.949 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:07.980 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:07.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:07.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:08.081 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:08.106 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:08.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:08.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:08.279 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:08.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:08.310 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:08.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:08.411 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:08.435 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:08.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:08.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:08.537 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:08.565 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:08.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:08.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:08.667 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:08.690 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:08.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:08.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:08.792 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:08.814 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:08.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:08.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:08.916 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:08.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:08.955 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:08.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:09.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:09.080 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:09.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:09.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:09.212 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:09.251 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:09.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:09.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:09.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:09.388 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:09.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:09.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:09.491 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:09.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:09.520 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:09.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:09.622 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:09.647 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:09.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:09.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:09.749 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:09.783 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:09.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:09.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:09.885 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:09.920 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:09.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:09.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:09.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:09 smithi136 ceph-mon[109861]: pgmap v3181: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.1 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 419 op/s; 47286/199665 objects degraded (23.683%); 26535/199665 objects misplaced (13.290%); 12 MiB/s, 199 objects/s recovering 2024-08-17T18:18:09.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:09 smithi136 ceph-mon[115078]: pgmap v3181: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.1 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 419 op/s; 47286/199665 objects degraded (23.683%); 26535/199665 objects misplaced (13.290%); 12 MiB/s, 199 objects/s recovering 2024-08-17T18:18:10.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:09 smithi165 ceph-mon[105517]: pgmap v3181: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.1 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 419 op/s; 47286/199665 objects degraded (23.683%); 26535/199665 objects misplaced (13.290%); 12 MiB/s, 199 objects/s recovering 2024-08-17T18:18:10.022 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:10.058 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:10.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:10.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:10.160 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:10.187 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:10.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:10.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:10.289 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:10.313 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:10.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:10.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:10.415 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:10.439 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:10.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:10.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:10.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:10.562 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:10.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:10.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:10.664 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:10.690 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:10.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:10.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:10.716 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:10 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 47286/199665 objects degraded (23.683%), 16 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:10.717 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:10 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 47286/199665 objects degraded (23.683%), 16 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:10.793 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:10.826 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:10.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:10.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:10.929 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:10.956 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:10.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:10.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:11.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:10 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 47286/199665 objects degraded (23.683%), 16 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:11.059 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:11.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:11.097 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:11.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:11.198 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:11.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:11.230 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:11.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:11.331 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:11.362 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:11.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:11.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:11.464 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:11.494 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:11.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:11.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:11.596 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:11.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:11.620 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:11.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:11.629 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:11 smithi136 ceph-mon[109861]: pgmap v3182: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.1 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 353 op/s; 47286/199665 objects degraded (23.683%); 26535/199665 objects misplaced (13.290%); 11 MiB/s, 175 objects/s recovering 2024-08-17T18:18:11.630 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:11 smithi136 ceph-mon[115078]: pgmap v3182: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.1 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 353 op/s; 47286/199665 objects degraded (23.683%); 26535/199665 objects misplaced (13.290%); 11 MiB/s, 175 objects/s recovering 2024-08-17T18:18:11.722 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:11.757 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:11.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:11.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:11.858 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:11.881 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:11.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:11.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:11.983 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:11 smithi165 ceph-mon[105517]: pgmap v3182: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 4.1 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 353 op/s; 47286/199665 objects degraded (23.683%); 26535/199665 objects misplaced (13.290%); 11 MiB/s, 175 objects/s recovering 2024-08-17T18:18:12.025 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:12.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:12.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:12.048 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:18:12.060 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:18:12.060 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:18:12.085 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:18:12.085 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:18:12.105 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:18:12.105 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:18:12.125 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:18:12.125 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:18:12.127 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:12.154 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:18:12.154 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:18:12.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:12.155 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:12.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:12.173 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:18:12.173 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:18:12.192 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:18:12.192 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:18:12.215 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:18:12.215 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:18:12.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:12.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:12.283 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:12.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:12.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:12.418 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:12.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:12.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:12.519 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:12.545 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:12.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:12.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:12.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:12.671 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:12.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:12.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:12.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:12.803 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:12.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:12.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:12.806 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:12 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3712387319' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:18:12.807 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:12 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3712387319' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:18:12.836 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:18:12.836 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:18:12.905 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:12.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:12.927 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:12.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:13.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:12 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3712387319' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:18:13.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:13.050 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:13.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:13.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:13.152 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:13.175 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:13.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:13.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:13.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:13.309 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:13.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:13.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:13.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:13.440 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:13.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:13.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:13.542 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:13.565 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:13.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:13.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:13.667 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:13.700 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:13.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:13.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:13.802 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:13.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:13.830 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:13.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:13.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:13 smithi136 ceph-mon[109861]: pgmap v3183: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 3.9 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 513 op/s; 46644/193911 objects degraded (24.054%); 25312/193911 objects misplaced (13.053%); 17 MiB/s, 266 objects/s recovering 2024-08-17T18:18:13.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:13 smithi136 ceph-mon[115078]: pgmap v3183: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 3.9 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 513 op/s; 46644/193911 objects degraded (24.054%); 25312/193911 objects misplaced (13.053%); 17 MiB/s, 266 objects/s recovering 2024-08-17T18:18:13.958 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:13.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:13.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:13 smithi165 ceph-mon[105517]: pgmap v3183: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 3.9 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 513 op/s; 46644/193911 objects degraded (24.054%); 25312/193911 objects misplaced (13.053%); 17 MiB/s, 266 objects/s recovering 2024-08-17T18:18:14.060 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:14.091 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:14.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:14.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:14.193 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:14.220 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:14.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:14.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:14.322 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:14.345 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:14.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:14.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:14.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:14.476 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:14.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:14.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:14.578 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:14.601 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:14.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:14.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:14.704 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:14.735 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:14.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:14.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:14.837 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:14.865 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:14.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:14.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:14.967 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:14.993 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:14.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:14.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:15.095 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:15.126 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-17T18:18:15.129 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:15.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:15.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:15.231 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:15.256 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:15.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:15.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:15.358 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:15.381 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:15.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:15.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:15.480 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:18:15.482 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:15.528 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:15.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:15.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:15.630 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:15.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:15.653 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:15.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:15.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:15.791 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:15.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:15.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:15.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:15 smithi165 ceph-mon[105517]: pgmap v3184: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 3.9 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 394 op/s; 46507/192915 objects degraded (24.108%); 25178/192915 objects misplaced (13.051%); 12 MiB/s, 193 objects/s recovering 2024-08-17T18:18:15.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:15 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 46507/192915 objects degraded (24.108%), 16 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:15.852 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:15 smithi136 ceph-mon[109861]: pgmap v3184: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 3.9 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 394 op/s; 46507/192915 objects degraded (24.108%); 25178/192915 objects misplaced (13.051%); 12 MiB/s, 193 objects/s recovering 2024-08-17T18:18:15.852 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:15 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 46507/192915 objects degraded (24.108%), 16 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:15.853 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:15 smithi136 ceph-mon[115078]: pgmap v3184: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 3.9 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 394 op/s; 46507/192915 objects degraded (24.108%); 25178/192915 objects misplaced (13.051%); 12 MiB/s, 193 objects/s recovering 2024-08-17T18:18:15.853 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:15 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 46507/192915 objects degraded (24.108%), 16 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:15.894 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:15.918 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:15.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:15.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:16.019 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:16.047 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:16.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:16.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:16.149 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:16.172 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:16.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:16.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:16.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:16.306 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:16.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:16.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:16.408 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:16.442 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:16.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:16.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:16.544 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:16.557 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:18:16.570 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:16.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:16.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:16.672 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:16.695 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:16.695 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:18:16.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:16.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:16 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2860742271' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:18:16.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:16 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:18:16.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:16.831 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:16.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:16.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:16.932 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:16.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:16 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2860742271' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:18:16.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:16 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:18:16.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:16 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2860742271' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:18:16.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:16 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:18:16.971 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:16.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:16.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:17.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:17.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:17.108 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:17.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:17.209 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:17.242 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:17.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:17.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:17.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:17.370 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:17.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:17.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:17.472 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:17.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:17.507 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:17.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:17.586 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:18:17.587 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:18:17.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:17.644 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:17.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:17.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:17.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:18:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:18:17] "GET /metrics HTTP/1.1" 200 36776 "" "Prometheus/2.51.0" 2024-08-17T18:18:17.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:17 smithi136 ceph-mon[109861]: pgmap v3185: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 3.9 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 381 op/s; 46507/192348 objects degraded (24.179%); 24932/192348 objects misplaced (12.962%); 12 MiB/s, 188 objects/s recovering 2024-08-17T18:18:17.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:17 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:18:17.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:17 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:18:17.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:17 smithi136 ceph-mon[109861]: osdmap e1662: 8 total, 8 up, 6 in 2024-08-17T18:18:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:17 smithi136 ceph-mon[115078]: pgmap v3185: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 3.9 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 381 op/s; 46507/192348 objects degraded (24.179%); 24932/192348 objects misplaced (12.962%); 12 MiB/s, 188 objects/s recovering 2024-08-17T18:18:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:17 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:18:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:17 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:18:17.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:17 smithi136 ceph-mon[115078]: osdmap e1662: 8 total, 8 up, 6 in 2024-08-17T18:18:17.746 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:17 smithi165 ceph-mon[105517]: pgmap v3185: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 3.9 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 381 op/s; 46507/192348 objects degraded (24.179%); 24932/192348 objects misplaced (12.962%); 12 MiB/s, 188 objects/s recovering 2024-08-17T18:18:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:17 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:18:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:17 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:18:17.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:17 smithi165 ceph-mon[105517]: osdmap e1662: 8 total, 8 up, 6 in 2024-08-17T18:18:17.771 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:17.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:17.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:17.873 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:17.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:17.917 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:17.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:18.018 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:18.032 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:18:18.054 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:18.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:18.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:18.156 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:18.191 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:18.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:18.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:18.293 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:18.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-17T18:18:18.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:18.336 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:18.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:18.437 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:18.461 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:18.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:18.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:18.563 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:18.576 INFO:teuthology.orchestra.run.smithi136.stderr:set pool 57 pgp_num to 26 2024-08-17T18:18:18.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:18.595 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:18.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:18.696 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:18.724 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:18:18.729 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:18.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:18.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:18.831 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:18.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:18.860 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:18.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:18.869 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:18 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4065715933' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"} : dispatch 2024-08-17T18:18:18.869 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:18 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"} : dispatch 2024-08-17T18:18:18.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:18 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4065715933' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"} : dispatch 2024-08-17T18:18:18.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:18 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"} : dispatch 2024-08-17T18:18:18.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:18 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4065715933' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"} : dispatch 2024-08-17T18:18:18.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:18 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"} : dispatch 2024-08-17T18:18:18.961 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:18.985 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:18.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:18.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:19.087 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:19.111 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:19.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:19.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:19.213 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:19.253 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:19.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:19.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:19.355 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:19.379 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:19.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:19.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:19.444 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:18:19.482 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:19.511 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:19.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:19.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:19.613 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:19.636 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:19.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:19.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:19.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:19.771 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:19.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:19.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:19.873 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:19.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:19.899 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:19.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:19.944 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:19 smithi136 ceph-mon[109861]: pgmap v3187: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 3.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 437 op/s; 45788/186531 objects degraded (24.547%); 23692/186531 objects misplaced (12.701%); 15 MiB/s, 242 objects/s recovering 2024-08-17T18:18:19.944 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:19 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]': finished 2024-08-17T18:18:19.944 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:19 smithi136 ceph-mon[109861]: osdmap e1663: 8 total, 8 up, 6 in 2024-08-17T18:18:19.944 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:19 smithi136 ceph-mon[115078]: pgmap v3187: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 3.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 437 op/s; 45788/186531 objects degraded (24.547%); 23692/186531 objects misplaced (12.701%); 15 MiB/s, 242 objects/s recovering 2024-08-17T18:18:19.944 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:19 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]': finished 2024-08-17T18:18:19.944 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:19 smithi136 ceph-mon[115078]: osdmap e1663: 8 total, 8 up, 6 in 2024-08-17T18:18:20.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:19 smithi165 ceph-mon[105517]: pgmap v3187: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 3.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 437 op/s; 45788/186531 objects degraded (24.547%); 23692/186531 objects misplaced (12.701%); 15 MiB/s, 242 objects/s recovering 2024-08-17T18:18:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:19 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]': finished 2024-08-17T18:18:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:19 smithi165 ceph-mon[105517]: osdmap e1663: 8 total, 8 up, 6 in 2024-08-17T18:18:20.026 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:20.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:20.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:20.128 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:20.157 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:20.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:20.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:20.259 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:20.285 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:20.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:20.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:20.387 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:20.411 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:20.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:20.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:20.462 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:18:20.463 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:18:20.483 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:18:20.484 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:18:20.502 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:18:20.503 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:18:20.513 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:20.521 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:18:20.521 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:18:20.540 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:18:20.540 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:18:20.548 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:20.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:20.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:20.563 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:18:20.563 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:18:20.584 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:18:20.585 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:18:20.585 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:20 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 45788/186531 objects degraded (24.547%), 16 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:20.585 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:20 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4051048990' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:18:20.585 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:20 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 45788/186531 objects degraded (24.547%), 16 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:20.585 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:20 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4051048990' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:18:20.606 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:18:20.606 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:18:20.650 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:20.682 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:20.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:20.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:20.729 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:18:20.729 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:18:20.729 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-08-17T18:18:20.729 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-08-17T18:18:20.729 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:18:20.787 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:20.810 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:20.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:20.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:20.912 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:20.934 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:20.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:20.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:21.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:20 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 45788/186531 objects degraded (24.547%), 16 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:21.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:20 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4051048990' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:18:21.036 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:21.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:21.063 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:21.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:21.164 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:21.186 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:21.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:21.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:21.288 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:21.310 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:21.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:21.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:21.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:21.434 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:21.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:21.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:21.536 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:21.560 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:21.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:21.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:21.662 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:21.684 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:21.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:21.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:21.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:21 smithi136 ceph-mon[115078]: pgmap v3189: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 3.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 307 op/s; 45788/186531 objects degraded (24.547%); 23692/186531 objects misplaced (12.701%); 10 MiB/s, 166 objects/s recovering 2024-08-17T18:18:21.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:21.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:18:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:21 smithi136 ceph-mon[109861]: pgmap v3189: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 3.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 307 op/s; 45788/186531 objects degraded (24.547%); 23692/186531 objects misplaced (12.701%); 10 MiB/s, 166 objects/s recovering 2024-08-17T18:18:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:18:21.786 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:21.808 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:21.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:21.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:21.836 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:18:21.910 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:21.943 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:21.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:21.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:21 smithi165 ceph-mon[105517]: pgmap v3189: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 3.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 307 op/s; 45788/186531 objects degraded (24.547%); 23692/186531 objects misplaced (12.701%); 10 MiB/s, 166 objects/s recovering 2024-08-17T18:18:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:18:22.046 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:22.080 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:22.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:22.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:22.182 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:22.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-17T18:18:22.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:22.206 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:22.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:22.307 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:22.331 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:22.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:22.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:22.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:22.463 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:22.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:22.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:22.477 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:18:22.565 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:22.588 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:22.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:22.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:22.690 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:22.714 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:22.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:22.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:22.816 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:22.838 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:22.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:22.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:22.940 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:22.979 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:22.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:22.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:23.081 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:23.103 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:23.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:23.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:23.206 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:23.227 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:23.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:23.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:23.328 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:23.368 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:23.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:23.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:23.470 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:23.494 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:23.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:23.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:23.596 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:23.631 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:23.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:23.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:23.664 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:23 smithi136 ceph-mon[109861]: pgmap v3190: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 3.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 493 op/s; 44530/181068 objects degraded (24.593%); 22991/181068 objects misplaced (12.697%); 14 MiB/s, 333 objects/s recovering 2024-08-17T18:18:23.664 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:23 smithi136 ceph-mon[115078]: pgmap v3190: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 3.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 493 op/s; 44530/181068 objects degraded (24.593%); 22991/181068 objects misplaced (12.697%); 14 MiB/s, 333 objects/s recovering 2024-08-17T18:18:23.733 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:23.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:23 smithi165 ceph-mon[105517]: pgmap v3190: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 3.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 493 op/s; 44530/181068 objects degraded (24.593%); 22991/181068 objects misplaced (12.697%); 14 MiB/s, 333 objects/s recovering 2024-08-17T18:18:23.764 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:23.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:23.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:23.838 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:18:23.866 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:23.901 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:23.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:23.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:24.004 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:24.027 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:24.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:24.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:24.069 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:18:24.129 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:24.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:24.171 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:24.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:24.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:24.310 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:24.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:24.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:24.412 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:24.434 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:24.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:24.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:24.438 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:18:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T18:18:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T18:18:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:18:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:178706 56.a 56:5b57367e:::benchmark_data_smithi136_356456_object52002:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:18:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:03.691091+0000", 2024-08-17T18:18:24.438 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 380.74280657200001, 2024-08-17T18:18:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.157627722, 2024-08-17T18:18:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:18:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:18:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:18:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 178706 2024-08-17T18:18:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:24.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691091+0000", 2024-08-17T18:18:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691091+0000", 2024-08-17T18:18:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:24.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691089+0000", 2024-08-17T18:18:24.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:18:24.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:24.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691096+0000", 2024-08-17T18:18:24.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.503e-06 2024-08-17T18:18:24.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:24.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691108+0000", 2024-08-17T18:18:24.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2129e-05 2024-08-17T18:18:24.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:24.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691117+0000", 2024-08-17T18:18:24.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4279999999999996e-06 2024-08-17T18:18:24.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.700684+0000", 2024-08-17T18:18:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0095666109999999992 2024-08-17T18:18:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:18:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.700694+0000", 2024-08-17T18:18:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0535e-05 2024-08-17T18:18:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.630895+0000", 2024-08-17T18:18:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93020063799999997 2024-08-17T18:18:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:18:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.630909+0000", 2024-08-17T18:18:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4681000000000001e-05 2024-08-17T18:18:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.846356+0000", 2024-08-17T18:18:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2154467900000001 2024-08-17T18:18:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:24.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.848718+0000", 2024-08-17T18:18:24.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002362276 2024-08-17T18:18:24.445 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.445 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:24.445 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.445 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:178708 56.f 56:fa635417:::benchmark_data_smithi136_356456_object52004:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:18:24.445 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:03.691130+0000", 2024-08-17T18:18:24.445 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 380.74276701000002, 2024-08-17T18:18:24.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.140675372, 2024-08-17T18:18:24.445 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:24.445 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:18:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:18:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:18:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 178708 2024-08-17T18:18:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691130+0000", 2024-08-17T18:18:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691130+0000", 2024-08-17T18:18:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691130+0000", 2024-08-17T18:18:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:18:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691133+0000", 2024-08-17T18:18:24.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0479999999999999e-06 2024-08-17T18:18:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691139+0000", 2024-08-17T18:18:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4860000000000002e-06 2024-08-17T18:18:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691144+0000", 2024-08-17T18:18:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0640000000000001e-06 2024-08-17T18:18:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.701188+0000", 2024-08-17T18:18:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010043593999999999 2024-08-17T18:18:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:18:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.701197+0000", 2024-08-17T18:18:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8060000000000001e-06 2024-08-17T18:18:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.630897+0000", 2024-08-17T18:18:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.92970020200000003 2024-08-17T18:18:24.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:18:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.630907+0000", 2024-08-17T18:18:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9119999999999993e-06 2024-08-17T18:18:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.829354+0000", 2024-08-17T18:18:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.198446897 2024-08-17T18:18:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:24.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.831806+0000", 2024-08-17T18:18:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002451815 2024-08-17T18:18:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:178716 56.a 56:5e090e51:::benchmark_data_smithi136_356456_object52012:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:18:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:03.691150+0000", 2024-08-17T18:18:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 380.74274684900001, 2024-08-17T18:18:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1748315850000002, 2024-08-17T18:18:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:24.451 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:18:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:18:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 178716 2024-08-17T18:18:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691150+0000", 2024-08-17T18:18:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:24.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691150+0000", 2024-08-17T18:18:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691150+0000", 2024-08-17T18:18:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:18:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691152+0000", 2024-08-17T18:18:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2120000000000002e-06 2024-08-17T18:18:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:24.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691158+0000", 2024-08-17T18:18:24.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8479999999999997e-06 2024-08-17T18:18:24.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.691163+0000", 2024-08-17T18:18:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6840000000000004e-06 2024-08-17T18:18:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.700716+0000", 2024-08-17T18:18:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0095534420000000005 2024-08-17T18:18:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:18:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.700724+0000", 2024-08-17T18:18:24.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7659999999999998e-06 2024-08-17T18:18:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.630937+0000", 2024-08-17T18:18:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.93021298399999996 2024-08-17T18:18:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:18:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.630947+0000", 2024-08-17T18:18:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9170000000000001e-06 2024-08-17T18:18:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.848942+0000", 2024-08-17T18:18:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.217995406 2024-08-17T18:18:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.849605+0000", 2024-08-17T18:18:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00066261399999999995 2024-08-17T18:18:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3,6", 2024-08-17T18:18:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.849886+0000", 2024-08-17T18:18:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000281437 2024-08-17T18:18:24.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.865742+0000", 2024-08-17T18:18:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015855622999999999 2024-08-17T18:18:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.865751+0000", 2024-08-17T18:18:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9230000000000008e-06 2024-08-17T18:18:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.865785+0000", 2024-08-17T18:18:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4532000000000001e-05 2024-08-17T18:18:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.865821+0000", 2024-08-17T18:18:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5692000000000003e-05 2024-08-17T18:18:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.865869+0000", 2024-08-17T18:18:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.816e-05 2024-08-17T18:18:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:24.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.865982+0000", 2024-08-17T18:18:24.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112749 2024-08-17T18:18:24.460 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.460 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:24.460 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.460 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:178717 56.f 56:f25e328a:::benchmark_data_smithi136_356456_object52013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:18:24.460 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:03.698538+0000", 2024-08-17T18:18:24.460 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 380.735359367, 2024-08-17T18:18:24.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.154307932, 2024-08-17T18:18:24.460 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:18:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:18:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 178717 2024-08-17T18:18:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.698538+0000", 2024-08-17T18:18:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.698538+0000", 2024-08-17T18:18:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.698537+0000", 2024-08-17T18:18:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:18:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:24.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.698548+0000", 2024-08-17T18:18:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0995000000000001e-05 2024-08-17T18:18:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.698558+0000", 2024-08-17T18:18:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.005e-05 2024-08-17T18:18:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.698564+0000", 2024-08-17T18:18:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.494e-06 2024-08-17T18:18:24.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.701220+0000", 2024-08-17T18:18:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026557120000000002 2024-08-17T18:18:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:18:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.701227+0000", 2024-08-17T18:18:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2200000000000003e-06 2024-08-17T18:18:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.630930+0000", 2024-08-17T18:18:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.92970317000000002 2024-08-17T18:18:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:18:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.630938+0000", 2024-08-17T18:18:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1289999999999996e-06 2024-08-17T18:18:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.832162+0000", 2024-08-17T18:18:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2012241939999999 2024-08-17T18:18:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.832939+0000", 2024-08-17T18:18:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00077643899999999997 2024-08-17T18:18:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1,6", 2024-08-17T18:18:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.833419+0000", 2024-08-17T18:18:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048046500000000002 2024-08-17T18:18:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:24.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.844527+0000", 2024-08-17T18:18:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011107637 2024-08-17T18:18:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.844562+0000", 2024-08-17T18:18:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5005000000000001e-05 2024-08-17T18:18:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.852662+0000", 2024-08-17T18:18:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080996279999999993 2024-08-17T18:18:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.852693+0000", 2024-08-17T18:18:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1347000000000003e-05 2024-08-17T18:18:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.852740+0000", 2024-08-17T18:18:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6626999999999999e-05 2024-08-17T18:18:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.852846+0000", 2024-08-17T18:18:24.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010621 2024-08-17T18:18:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:122085 57.2 e1639/1627 57:42ce4696:::benchmark_data_smithi136_389235_object6163:head v 1639'6295, mlcod=0'0)", 2024-08-17T18:18:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:54.574819+0000", 2024-08-17T18:18:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 149.85907794299999, 2024-08-17T18:18:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5239920300000001, 2024-08-17T18:18:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:24.469 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.574819+0000", 2024-08-17T18:18:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.574819+0000", 2024-08-17T18:18:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:24.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.574818+0000", 2024-08-17T18:18:24.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:18:24.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:24.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.574826+0000", 2024-08-17T18:18:24.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7100000000000007e-06 2024-08-17T18:18:24.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:24.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.574840+0000", 2024-08-17T18:18:24.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3709999999999999e-05 2024-08-17T18:18:24.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:24.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.574847+0000", 2024-08-17T18:18:24.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.764e-06 2024-08-17T18:18:24.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.091103+0000", 2024-08-17T18:18:24.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5162560969999999 2024-08-17T18:18:24.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:24.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.091123+0000", 2024-08-17T18:18:24.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0537e-05 2024-08-17T18:18:24.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:24.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.098763+0000", 2024-08-17T18:18:24.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0076399299999999996 2024-08-17T18:18:24.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.098811+0000", 2024-08-17T18:18:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8256e-05 2024-08-17T18:18:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:122093 57.2 e1639/1627 57:407dd288:::benchmark_data_smithi136_389235_object6171:head v 1639'6296, mlcod=0'0)", 2024-08-17T18:18:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:54.622473+0000", 2024-08-17T18:18:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 149.811424042, 2024-08-17T18:18:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.476358522, 2024-08-17T18:18:24.474 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.622473+0000", 2024-08-17T18:18:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.622473+0000", 2024-08-17T18:18:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.622472+0000", 2024-08-17T18:18:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:18:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.622481+0000", 2024-08-17T18:18:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1880000000000008e-06 2024-08-17T18:18:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.622494+0000", 2024-08-17T18:18:24.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2513e-05 2024-08-17T18:18:24.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:24.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.622500+0000", 2024-08-17T18:18:24.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7769999999999999e-06 2024-08-17T18:18:24.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.094420+0000", 2024-08-17T18:18:24.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4719201590000002 2024-08-17T18:18:24.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:24.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.094441+0000", 2024-08-17T18:18:24.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1554000000000001e-05 2024-08-17T18:18:24.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.098815+0000", 2024-08-17T18:18:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004373633 2024-08-17T18:18:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.098832+0000", 2024-08-17T18:18:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6744000000000001e-05 2024-08-17T18:18:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:122102 57.2 e1641/1627 57:436a03b4:::benchmark_data_smithi136_389235_object6180:head v 1641'6297, mlcod=1612'631)", 2024-08-17T18:18:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:57.129135+0000", 2024-08-17T18:18:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 147.30476221000001, 2024-08-17T18:18:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3257561870000001, 2024-08-17T18:18:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.129135+0000", 2024-08-17T18:18:24.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.129135+0000", 2024-08-17T18:18:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.129135+0000", 2024-08-17T18:18:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:18:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.129143+0000", 2024-08-17T18:18:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1459999999999996e-06 2024-08-17T18:18:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.129152+0000", 2024-08-17T18:18:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0440000000000001e-06 2024-08-17T18:18:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.129157+0000", 2024-08-17T18:18:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7740000000000001e-06 2024-08-17T18:18:24.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.324262+0000", 2024-08-17T18:18:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1951041249999999 2024-08-17T18:18:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.324289+0000", 2024-08-17T18:18:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7246e-05 2024-08-17T18:18:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:24.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.454857+0000", 2024-08-17T18:18:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13056785200000001 2024-08-17T18:18:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.454891+0000", 2024-08-17T18:18:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4575e-05 2024-08-17T18:18:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:133120 57.12 57:49924cf9:::benchmark_data_smithi136_389235_object17198:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1648)", 2024-08-17T18:18:24.487 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:35.543899+0000", 2024-08-17T18:18:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 108.889998378, 2024-08-17T18:18:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1856874199999998, 2024-08-17T18:18:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133120 2024-08-17T18:18:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:35.543899+0000", 2024-08-17T18:18:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:35.543899+0000", 2024-08-17T18:18:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:35.543897+0000", 2024-08-17T18:18:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:18:24.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:35.543905+0000", 2024-08-17T18:18:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7819999999999994e-06 2024-08-17T18:18:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:35.543919+0000", 2024-08-17T18:18:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.432e-05 2024-08-17T18:18:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:24.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:35.543926+0000", 2024-08-17T18:18:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8279999999999997e-06 2024-08-17T18:18:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:35.643933+0000", 2024-08-17T18:18:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100006952 2024-08-17T18:18:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:18:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:35.646011+0000", 2024-08-17T18:18:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002078009 2024-08-17T18:18:24.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:36.551693+0000", 2024-08-17T18:18:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90568204399999996 2024-08-17T18:18:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:18:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:36.557914+0000", 2024-08-17T18:18:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062204959999999998 2024-08-17T18:18:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.616506+0000", 2024-08-17T18:18:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.058592816 2024-08-17T18:18:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.619828+0000", 2024-08-17T18:18:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033217440000000002 2024-08-17T18:18:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:18:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.626665+0000", 2024-08-17T18:18:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068367649999999999 2024-08-17T18:18:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.632385+0000", 2024-08-17T18:18:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057198839999999997 2024-08-17T18:18:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.633470+0000", 2024-08-17T18:18:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010857499999999999 2024-08-17T18:18:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.716484+0000", 2024-08-17T18:18:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.083013179000000006 2024-08-17T18:18:24.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.718934+0000", 2024-08-17T18:18:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024506319999999999 2024-08-17T18:18:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.729586+0000", 2024-08-17T18:18:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010652053 2024-08-17T18:18:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:133992 57.12 57:48ed4d24:::benchmark_data_smithi136_389235_object18070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1648)", 2024-08-17T18:18:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:38.781600+0000", 2024-08-17T18:18:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 105.65229709800001, 2024-08-17T18:18:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1134712009999999, 2024-08-17T18:18:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133992 2024-08-17T18:18:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:24.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.781600+0000", 2024-08-17T18:18:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.781600+0000", 2024-08-17T18:18:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.781599+0000", 2024-08-17T18:18:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:18:24.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.781604+0000", 2024-08-17T18:18:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8099999999999997e-06 2024-08-17T18:18:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.781615+0000", 2024-08-17T18:18:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0771000000000001e-05 2024-08-17T18:18:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:24.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.781620+0000", 2024-08-17T18:18:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9740000000000001e-06 2024-08-17T18:18:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.847439+0000", 2024-08-17T18:18:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.065819201999999993 2024-08-17T18:18:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:18:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.851794+0000", 2024-08-17T18:18:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043556259999999996 2024-08-17T18:18:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:39.873915+0000", 2024-08-17T18:18:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0221207800000001 2024-08-17T18:18:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:18:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:39.878239+0000", 2024-08-17T18:18:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004323716 2024-08-17T18:18:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.852201+0000", 2024-08-17T18:18:24.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.973961909 2024-08-17T18:18:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.855640+0000", 2024-08-17T18:18:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034392709999999998 2024-08-17T18:18:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:18:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.862697+0000", 2024-08-17T18:18:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070570750000000003 2024-08-17T18:18:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.882488+0000", 2024-08-17T18:18:24.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019790395999999998 2024-08-17T18:18:24.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.883769+0000", 2024-08-17T18:18:24.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012813869999999999 2024-08-17T18:18:24.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:24.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.884136+0000", 2024-08-17T18:18:24.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036747899999999998 2024-08-17T18:18:24.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:24.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.885963+0000", 2024-08-17T18:18:24.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00182672 2024-08-17T18:18:24.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:24.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.895071+0000", 2024-08-17T18:18:24.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0091082130000000004 2024-08-17T18:18:24.504 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:134465 57.12 57:49a43719:::benchmark_data_smithi136_389235_object18543:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1648)", 2024-08-17T18:18:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:40.473026+0000", 2024-08-17T18:18:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 103.96087124899999, 2024-08-17T18:18:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5757722840000001, 2024-08-17T18:18:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:24.505 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 134465 2024-08-17T18:18:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:40.473026+0000", 2024-08-17T18:18:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:24.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:40.473026+0000", 2024-08-17T18:18:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:40.473025+0000", 2024-08-17T18:18:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:18:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:40.473032+0000", 2024-08-17T18:18:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1280000000000002e-06 2024-08-17T18:18:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:40.473049+0000", 2024-08-17T18:18:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7317999999999999e-05 2024-08-17T18:18:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:40.473057+0000", 2024-08-17T18:18:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4309999999999994e-06 2024-08-17T18:18:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:40.572818+0000", 2024-08-17T18:18:24.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099760653000000005 2024-08-17T18:18:24.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:18:24.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:40.574727+0000", 2024-08-17T18:18:24.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019091240000000001 2024-08-17T18:18:24.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.377268+0000", 2024-08-17T18:18:24.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.80254037899999997 2024-08-17T18:18:24.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:18:24.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.382386+0000", 2024-08-17T18:18:24.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051183130000000002 2024-08-17T18:18:24.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:42.982237+0000", 2024-08-17T18:18:24.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.599851355 2024-08-17T18:18:24.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:24.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:42.987818+0000", 2024-08-17T18:18:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055806980000000003 2024-08-17T18:18:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:18:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:42.998928+0000", 2024-08-17T18:18:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011109734 2024-08-17T18:18:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.024007+0000", 2024-08-17T18:18:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.025079534000000001 2024-08-17T18:18:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.025175+0000", 2024-08-17T18:18:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011681350000000001 2024-08-17T18:18:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.026529+0000", 2024-08-17T18:18:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013539509999999999 2024-08-17T18:18:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.029124+0000", 2024-08-17T18:18:24.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025947209999999999 2024-08-17T18:18:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.048798+0000", 2024-08-17T18:18:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019674259999999999 2024-08-17T18:18:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:134821 57.12 57:49381b19:::benchmark_data_smithi136_389235_object18899:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1648)", 2024-08-17T18:18:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:41.544450+0000", 2024-08-17T18:18:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 102.889447625, 2024-08-17T18:18:24.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5392011000000001, 2024-08-17T18:18:24.514 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:24.514 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:24.514 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:24.514 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:24.514 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:24.514 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 134821 2024-08-17T18:18:24.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.514 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:24.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:24.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.544450+0000", 2024-08-17T18:18:24.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:24.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.544450+0000", 2024-08-17T18:18:24.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:24.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.544448+0000", 2024-08-17T18:18:24.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:18:24.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:24.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.544454+0000", 2024-08-17T18:18:24.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0630000000000002e-06 2024-08-17T18:18:24.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:24.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.544466+0000", 2024-08-17T18:18:24.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2170999999999999e-05 2024-08-17T18:18:24.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.516 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:24.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.544474+0000", 2024-08-17T18:18:24.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3069999999999997e-06 2024-08-17T18:18:24.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.566862+0000", 2024-08-17T18:18:24.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022388354999999999 2024-08-17T18:18:24.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:18:24.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.568805+0000", 2024-08-17T18:18:24.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019431330000000001 2024-08-17T18:18:24.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:42.238732+0000", 2024-08-17T18:18:24.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.66992708300000003 2024-08-17T18:18:24.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:18:24.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:42.243750+0000", 2024-08-17T18:18:24.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050172599999999999 2024-08-17T18:18:24.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.029453+0000", 2024-08-17T18:18:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.785703324 2024-08-17T18:18:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.032708+0000", 2024-08-17T18:18:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032548080000000001 2024-08-17T18:18:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:18:24.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.039182+0000", 2024-08-17T18:18:24.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064746630000000003 2024-08-17T18:18:24.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.050280+0000", 2024-08-17T18:18:24.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011097205000000001 2024-08-17T18:18:24.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.051571+0000", 2024-08-17T18:18:24.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001291169 2024-08-17T18:18:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.070117+0000", 2024-08-17T18:18:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018546475 2024-08-17T18:18:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.072184+0000", 2024-08-17T18:18:24.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020664170000000001 2024-08-17T18:18:24.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:24.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.083651+0000", 2024-08-17T18:18:24.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011467073 2024-08-17T18:18:24.522 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.522 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:24.522 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.523 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:135268 57.12 57:4ffd0969:::benchmark_data_smithi136_389235_object19346:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1648)", 2024-08-17T18:18:24.523 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:43.180362+0000", 2024-08-17T18:18:24.523 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 101.25353577600001, 2024-08-17T18:18:24.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.63819413, 2024-08-17T18:18:24.523 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:24.523 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:24.523 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:24.523 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 135268 2024-08-17T18:18:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.180362+0000", 2024-08-17T18:18:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:24.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.180362+0000", 2024-08-17T18:18:24.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:24.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.180360+0000", 2024-08-17T18:18:24.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:18:24.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:24.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.180367+0000", 2024-08-17T18:18:24.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5989999999999998e-06 2024-08-17T18:18:24.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:24.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.180379+0000", 2024-08-17T18:18:24.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1735e-05 2024-08-17T18:18:24.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:24.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.180384+0000", 2024-08-17T18:18:24.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8069999999999996e-06 2024-08-17T18:18:24.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.286732+0000", 2024-08-17T18:18:24.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.106347284 2024-08-17T18:18:24.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:18:24.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.289232+0000", 2024-08-17T18:18:24.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002500178 2024-08-17T18:18:24.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.963018+0000", 2024-08-17T18:18:24.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.67378577200000001 2024-08-17T18:18:24.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:18:24.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.967249+0000", 2024-08-17T18:18:24.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042313009999999998 2024-08-17T18:18:24.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.762994+0000", 2024-08-17T18:18:24.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.795744819 2024-08-17T18:18:24.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:24.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.766441+0000", 2024-08-17T18:18:24.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034468820000000001 2024-08-17T18:18:24.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:18:24.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.773636+0000", 2024-08-17T18:18:24.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071958530000000003 2024-08-17T18:18:24.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.800098+0000", 2024-08-17T18:18:24.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.026461710999999999 2024-08-17T18:18:24.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:24.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.800485+0000", 2024-08-17T18:18:24.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038679399999999998 2024-08-17T18:18:24.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.801755+0000", 2024-08-17T18:18:24.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001270427 2024-08-17T18:18:24.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:24.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.805500+0000", 2024-08-17T18:18:24.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037451469999999999 2024-08-17T18:18:24.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:24.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.818556+0000", 2024-08-17T18:18:24.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013055152 2024-08-17T18:18:24.532 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.532 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:24.532 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.532 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:135559 57.12 57:481716dc:::benchmark_data_smithi136_389235_object19637:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1649)", 2024-08-17T18:18:24.532 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:44.137515+0000", 2024-08-17T18:18:24.532 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 100.296382765, 2024-08-17T18:18:24.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0680285010000001, 2024-08-17T18:18:24.533 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:24.533 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:24.533 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:24.533 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:24.533 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:24.533 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 135559 2024-08-17T18:18:24.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.533 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:24.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:24.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.137515+0000", 2024-08-17T18:18:24.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:24.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.137515+0000", 2024-08-17T18:18:24.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:24.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.137513+0000", 2024-08-17T18:18:24.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:18:24.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:24.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.137521+0000", 2024-08-17T18:18:24.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0010000000000001e-06 2024-08-17T18:18:24.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:24.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.137538+0000", 2024-08-17T18:18:24.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6816e-05 2024-08-17T18:18:24.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:24.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.137546+0000", 2024-08-17T18:18:24.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2199999999999992e-06 2024-08-17T18:18:24.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.259176+0000", 2024-08-17T18:18:24.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.121630083 2024-08-17T18:18:24.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:18:24.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.261072+0000", 2024-08-17T18:18:24.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001895863 2024-08-17T18:18:24.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.204887+0000", 2024-08-17T18:18:24.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94381471800000005 2024-08-17T18:18:24.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:18:24.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.209973+0000", 2024-08-17T18:18:24.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005086775 2024-08-17T18:18:24.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:47.165222+0000", 2024-08-17T18:18:24.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9552484919999999 2024-08-17T18:18:24.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:24.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:47.169551+0000", 2024-08-17T18:18:24.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043290500000000001 2024-08-17T18:18:24.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:18:24.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:47.180180+0000", 2024-08-17T18:18:24.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010628629000000001 2024-08-17T18:18:24.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:47.188542+0000", 2024-08-17T18:18:24.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0083620650000000001 2024-08-17T18:18:24.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:47.191455+0000", 2024-08-17T18:18:24.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029128700000000001 2024-08-17T18:18:24.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:24.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:47.191846+0000", 2024-08-17T18:18:24.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000391911 2024-08-17T18:18:24.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:24.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:47.194664+0000", 2024-08-17T18:18:24.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028179059999999998 2024-08-17T18:18:24.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:24.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:47.205543+0000", 2024-08-17T18:18:24.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010878649000000001 2024-08-17T18:18:24.541 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.541 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:24.541 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.541 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:141699 57.12 57:4fa5408d:::benchmark_data_smithi136_389235_object25777:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1650)", 2024-08-17T18:18:24.541 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:02.480434+0000", 2024-08-17T18:18:24.541 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 81.953463065999998, 2024-08-17T18:18:24.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.198213231, 2024-08-17T18:18:24.542 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:24.542 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:24.542 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:24.542 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:24.542 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:24.542 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 141699 2024-08-17T18:18:24.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.542 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:24.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:24.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:02.480434+0000", 2024-08-17T18:18:24.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:24.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:02.480434+0000", 2024-08-17T18:18:24.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:24.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:02.480433+0000", 2024-08-17T18:18:24.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:18:24.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:24.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:02.480440+0000", 2024-08-17T18:18:24.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2400000000000001e-06 2024-08-17T18:18:24.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:24.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:02.480454+0000", 2024-08-17T18:18:24.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3973e-05 2024-08-17T18:18:24.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:24.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:02.480462+0000", 2024-08-17T18:18:24.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2300000000000008e-06 2024-08-17T18:18:24.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:02.510495+0000", 2024-08-17T18:18:24.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.030032631000000001 2024-08-17T18:18:24.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:18:24.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:02.512703+0000", 2024-08-17T18:18:24.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022079579999999999 2024-08-17T18:18:24.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.293890+0000", 2024-08-17T18:18:24.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.78118731900000005 2024-08-17T18:18:24.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:18:24.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.297795+0000", 2024-08-17T18:18:24.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039044959999999999 2024-08-17T18:18:24.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.625210+0000", 2024-08-17T18:18:24.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3274149319999999 2024-08-17T18:18:24.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:24.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.629619+0000", 2024-08-17T18:18:24.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044094820000000002 2024-08-17T18:18:24.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:18:24.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.636707+0000", 2024-08-17T18:18:24.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070873990000000003 2024-08-17T18:18:24.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.665989+0000", 2024-08-17T18:18:24.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.029282077 2024-08-17T18:18:24.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:24.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.666332+0000", 2024-08-17T18:18:24.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034313799999999999 2024-08-17T18:18:24.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.668196+0000", 2024-08-17T18:18:24.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001863969 2024-08-17T18:18:24.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:24.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.669949+0000", 2024-08-17T18:18:24.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017531300000000001 2024-08-17T18:18:24.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:24.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.678647+0000", 2024-08-17T18:18:24.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0086985150000000004 2024-08-17T18:18:24.549 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.549 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:24.549 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.549 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:142338 57.12 57:4cde03a7:::benchmark_data_smithi136_389235_object26416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1650)", 2024-08-17T18:18:24.549 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:04.803270+0000", 2024-08-17T18:18:24.549 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 79.630627125999993, 2024-08-17T18:18:24.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5418123230000003, 2024-08-17T18:18:24.549 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:24.549 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:24.550 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:24.550 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:24.550 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:24.550 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 142338 2024-08-17T18:18:24.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.550 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:24.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:24.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.803270+0000", 2024-08-17T18:18:24.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:24.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.803270+0000", 2024-08-17T18:18:24.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:24.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.803269+0000", 2024-08-17T18:18:24.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:18:24.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:24.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.803277+0000", 2024-08-17T18:18:24.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9999999999999996e-06 2024-08-17T18:18:24.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.803292+0000", 2024-08-17T18:18:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5608e-05 2024-08-17T18:18:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.803300+0000", 2024-08-17T18:18:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5839999999999997e-06 2024-08-17T18:18:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.830271+0000", 2024-08-17T18:18:24.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.026971517 2024-08-17T18:18:24.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:18:24.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.832589+0000", 2024-08-17T18:18:24.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023174799999999998 2024-08-17T18:18:24.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:05.753306+0000", 2024-08-17T18:18:24.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.92071752200000001 2024-08-17T18:18:24.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:18:24.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:05.757990+0000", 2024-08-17T18:18:24.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004683235 2024-08-17T18:18:24.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.296145+0000", 2024-08-17T18:18:24.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5381556270000001 2024-08-17T18:18:24.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:24.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.299493+0000", 2024-08-17T18:18:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033477310000000001 2024-08-17T18:18:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:18:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.306116+0000", 2024-08-17T18:18:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006623253 2024-08-17T18:18:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.325542+0000", 2024-08-17T18:18:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019425575 2024-08-17T18:18:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.330732+0000", 2024-08-17T18:18:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051902720000000001 2024-08-17T18:18:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.332141+0000", 2024-08-17T18:18:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014088799999999999 2024-08-17T18:18:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.334377+0000", 2024-08-17T18:18:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022364239999999999 2024-08-17T18:18:24.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:24.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.345082+0000", 2024-08-17T18:18:24.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010705077 2024-08-17T18:18:24.557 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.557 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:24.557 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.557 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:143066 57.12 57:4e20cb33:::benchmark_data_smithi136_389235_object27144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1650)", 2024-08-17T18:18:24.557 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:07.229551+0000", 2024-08-17T18:18:24.557 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 77.204346619999995, 2024-08-17T18:18:24.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.970721556, 2024-08-17T18:18:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 143066 2024-08-17T18:18:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.229551+0000", 2024-08-17T18:18:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.229551+0000", 2024-08-17T18:18:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.229549+0000", 2024-08-17T18:18:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:18:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:24.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.229562+0000", 2024-08-17T18:18:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3594000000000001e-05 2024-08-17T18:18:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.229576+0000", 2024-08-17T18:18:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3481e-05 2024-08-17T18:18:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.229584+0000", 2024-08-17T18:18:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5539999999999998e-06 2024-08-17T18:18:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.345598+0000", 2024-08-17T18:18:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.116014002 2024-08-17T18:18:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:18:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.347905+0000", 2024-08-17T18:18:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023068730000000001 2024-08-17T18:18:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.239359+0000", 2024-08-17T18:18:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.89145372499999997 2024-08-17T18:18:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:18:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.243463+0000", 2024-08-17T18:18:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041039589999999999 2024-08-17T18:18:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.112003+0000", 2024-08-17T18:18:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.868540396 2024-08-17T18:18:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.116413+0000", 2024-08-17T18:18:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044100650000000003 2024-08-17T18:18:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:18:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.125278+0000", 2024-08-17T18:18:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0088651159999999993 2024-08-17T18:18:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.137166+0000", 2024-08-17T18:18:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011887246000000001 2024-08-17T18:18:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.138635+0000", 2024-08-17T18:18:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001469595 2024-08-17T18:18:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.186048+0000", 2024-08-17T18:18:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.047412965000000001 2024-08-17T18:18:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.188412+0000", 2024-08-17T18:18:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002364054 2024-08-17T18:18:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.200272+0000", 2024-08-17T18:18:24.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01186006 2024-08-17T18:18:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:143493 57.12 57:4dd25d03:::benchmark_data_smithi136_389235_object27571:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1650)", 2024-08-17T18:18:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:08.252789+0000", 2024-08-17T18:18:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 76.181108433999995, 2024-08-17T18:18:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0232178630000002, 2024-08-17T18:18:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:24.566 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:24.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 143493 2024-08-17T18:18:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:24.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.252789+0000", 2024-08-17T18:18:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.252789+0000", 2024-08-17T18:18:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.252787+0000", 2024-08-17T18:18:24.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:18:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.252795+0000", 2024-08-17T18:18:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7840000000000004e-06 2024-08-17T18:18:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.252808+0000", 2024-08-17T18:18:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.291e-05 2024-08-17T18:18:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.252814+0000", 2024-08-17T18:18:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2600000000000002e-06 2024-08-17T18:18:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.299118+0000", 2024-08-17T18:18:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.046304317999999997 2024-08-17T18:18:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:18:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.301303+0000", 2024-08-17T18:18:24.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002185144 2024-08-17T18:18:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.358653+0000", 2024-08-17T18:18:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0573500520000001 2024-08-17T18:18:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:18:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.362431+0000", 2024-08-17T18:18:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037779129999999999 2024-08-17T18:18:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:11.243257+0000", 2024-08-17T18:18:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8808254760000001 2024-08-17T18:18:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:11.246415+0000", 2024-08-17T18:18:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003157726 2024-08-17T18:18:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:18:24.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:11.253380+0000", 2024-08-17T18:18:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069659270000000002 2024-08-17T18:18:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:11.263911+0000", 2024-08-17T18:18:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010530891000000001 2024-08-17T18:18:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:11.264967+0000", 2024-08-17T18:18:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001055769 2024-08-17T18:18:24.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:11.265311+0000", 2024-08-17T18:18:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034373200000000002 2024-08-17T18:18:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:11.267008+0000", 2024-08-17T18:18:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001696779 2024-08-17T18:18:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:11.276007+0000", 2024-08-17T18:18:24.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0089990529999999999 2024-08-17T18:18:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:144015 57.12 57:4d3166e5:::benchmark_data_smithi136_389235_object28093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1650)", 2024-08-17T18:18:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:09.497595+0000", 2024-08-17T18:18:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 74.936301951999994, 2024-08-17T18:18:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0067860620000002, 2024-08-17T18:18:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:24.576 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 144015 2024-08-17T18:18:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.497595+0000", 2024-08-17T18:18:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.497595+0000", 2024-08-17T18:18:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.497594+0000", 2024-08-17T18:18:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:18:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.497599+0000", 2024-08-17T18:18:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.096e-06 2024-08-17T18:18:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:24.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.497610+0000", 2024-08-17T18:18:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1107000000000001e-05 2024-08-17T18:18:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.497616+0000", 2024-08-17T18:18:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5459999999999999e-06 2024-08-17T18:18:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.599005+0000", 2024-08-17T18:18:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101389497 2024-08-17T18:18:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:18:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.600888+0000", 2024-08-17T18:18:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001883006 2024-08-17T18:18:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.353923+0000", 2024-08-17T18:18:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.75303402200000003 2024-08-17T18:18:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:18:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.359101+0000", 2024-08-17T18:18:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005178958 2024-08-17T18:18:24.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.408708+0000", 2024-08-17T18:18:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0496060800000002 2024-08-17T18:18:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.413557+0000", 2024-08-17T18:18:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048495170000000002 2024-08-17T18:18:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:18:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.423699+0000", 2024-08-17T18:18:24.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010141554000000001 2024-08-17T18:18:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.453884+0000", 2024-08-17T18:18:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.030185025000000001 2024-08-17T18:18:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.489415+0000", 2024-08-17T18:18:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035530897999999998 2024-08-17T18:18:24.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.491456+0000", 2024-08-17T18:18:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020412389999999998 2024-08-17T18:18:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.493673+0000", 2024-08-17T18:18:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022170179999999999 2024-08-17T18:18:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.504381+0000", 2024-08-17T18:18:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010708611999999999 2024-08-17T18:18:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:145041 57.12 57:4cef9719:::benchmark_data_smithi136_389235_object29119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1651)", 2024-08-17T18:18:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:12.526739+0000", 2024-08-17T18:18:24.584 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 71.907158676999998, 2024-08-17T18:18:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3206357639999999, 2024-08-17T18:18:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 145041 2024-08-17T18:18:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.526739+0000", 2024-08-17T18:18:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.526739+0000", 2024-08-17T18:18:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.586 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.526737+0000", 2024-08-17T18:18:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:18:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.526745+0000", 2024-08-17T18:18:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0960000000000001e-06 2024-08-17T18:18:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:24.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.526762+0000", 2024-08-17T18:18:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6514000000000001e-05 2024-08-17T18:18:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.526770+0000", 2024-08-17T18:18:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7849999999999997e-06 2024-08-17T18:18:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.575365+0000", 2024-08-17T18:18:24.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.048594641000000001 2024-08-17T18:18:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:18:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.577274+0000", 2024-08-17T18:18:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001909371 2024-08-17T18:18:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.206156+0000", 2024-08-17T18:18:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.62888149900000001 2024-08-17T18:18:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:18:24.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.216312+0000", 2024-08-17T18:18:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010156263 2024-08-17T18:18:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.814900+0000", 2024-08-17T18:18:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5985881389999999 2024-08-17T18:18:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.818177+0000", 2024-08-17T18:18:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032767489999999998 2024-08-17T18:18:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:18:24.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.825676+0000", 2024-08-17T18:18:24.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0074987159999999999 2024-08-17T18:18:24.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.833778+0000", 2024-08-17T18:18:24.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0081025070000000001 2024-08-17T18:18:24.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.834861+0000", 2024-08-17T18:18:24.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010832140000000001 2024-08-17T18:18:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.835223+0000", 2024-08-17T18:18:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036148499999999999 2024-08-17T18:18:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.837322+0000", 2024-08-17T18:18:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002099212 2024-08-17T18:18:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:24.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.847374+0000", 2024-08-17T18:18:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010052195999999999 2024-08-17T18:18:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:145558 57.12 57:4e414c7c:::benchmark_data_smithi136_389235_object29636:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1651)", 2024-08-17T18:18:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:13.886158+0000", 2024-08-17T18:18:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 70.547739386000003, 2024-08-17T18:18:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5611586079999999, 2024-08-17T18:18:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:24.593 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 145558 2024-08-17T18:18:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.886158+0000", 2024-08-17T18:18:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.886158+0000", 2024-08-17T18:18:24.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.886157+0000", 2024-08-17T18:18:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:18:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.886162+0000", 2024-08-17T18:18:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7880000000000003e-06 2024-08-17T18:18:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:24.596 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.886174+0000", 2024-08-17T18:18:24.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1508999999999999e-05 2024-08-17T18:18:24.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.596 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:24.596 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.886180+0000", 2024-08-17T18:18:24.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8089999999999998e-06 2024-08-17T18:18:24.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.596 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.596 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.902475+0000", 2024-08-17T18:18:24.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016294869999999999 2024-08-17T18:18:24.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:18:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.904366+0000", 2024-08-17T18:18:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001891083 2024-08-17T18:18:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.790935+0000", 2024-08-17T18:18:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.88656959800000001 2024-08-17T18:18:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:18:24.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.795601+0000", 2024-08-17T18:18:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046654030000000003 2024-08-17T18:18:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:16.393619+0000", 2024-08-17T18:18:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.598018199 2024-08-17T18:18:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:16.397485+0000", 2024-08-17T18:18:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003866455 2024-08-17T18:18:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:18:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:16.404509+0000", 2024-08-17T18:18:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070232319999999999 2024-08-17T18:18:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:16.431091+0000", 2024-08-17T18:18:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.026582179000000001 2024-08-17T18:18:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:16.432698+0000", 2024-08-17T18:18:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001607107 2024-08-17T18:18:24.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:16.434119+0000", 2024-08-17T18:18:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001420775 2024-08-17T18:18:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:16.436158+0000", 2024-08-17T18:18:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002039482 2024-08-17T18:18:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:16.447317+0000", 2024-08-17T18:18:24.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011158404 2024-08-17T18:18:24.601 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.601 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:24.601 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.601 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:24.601 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:24.601 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:18:24.602 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:18:24.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:24.635 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:24.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:24.736 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:24.744 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:24 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2522629718' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:18:24.744 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:24 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:18:24.744 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:24 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2522629718' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:18:24.744 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:24 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:18:24.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:24 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2522629718' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:18:24.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:24 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:18:24.769 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:24.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:24.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:24.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:24.897 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:24.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:24.999 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:25.021 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:25.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:25.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:25.123 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:25.144 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:25.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:25.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:25.246 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:25.269 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:25.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:25.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:25.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:25.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:25.398 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:25.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:25.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-17T18:18:25.499 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:25.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:25.531 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:25.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:25.632 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:25.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:25.677 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:25.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:25 smithi165 ceph-mon[105517]: pgmap v3191: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 3.7 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 512 op/s; 44388/180051 objects degraded (24.653%); 22868/180051 objects misplaced (12.701%); 15 MiB/s, 341 objects/s recovering 2024-08-17T18:18:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:25 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:18:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:25 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:18:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:25 smithi165 ceph-mon[105517]: osdmap e1664: 8 total, 8 up, 6 in 2024-08-17T18:18:25.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:25 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 44388/180051 objects degraded (24.653%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:25.779 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:25.795 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:18:25.803 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:25.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:25.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:25.896 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:25 smithi136 ceph-mon[109861]: pgmap v3191: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 3.7 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 512 op/s; 44388/180051 objects degraded (24.653%); 22868/180051 objects misplaced (12.701%); 15 MiB/s, 341 objects/s recovering 2024-08-17T18:18:25.896 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:25 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:18:25.896 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:25 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:18:25.896 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:25 smithi136 ceph-mon[109861]: osdmap e1664: 8 total, 8 up, 6 in 2024-08-17T18:18:25.896 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:25 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 44388/180051 objects degraded (24.653%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:25.897 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:25 smithi136 ceph-mon[115078]: pgmap v3191: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 6 active+remapped+backfill_wait; 3.7 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 512 op/s; 44388/180051 objects degraded (24.653%); 22868/180051 objects misplaced (12.701%); 15 MiB/s, 341 objects/s recovering 2024-08-17T18:18:25.897 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:25 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:18:25.897 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:25 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:18:25.897 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:25 smithi136 ceph-mon[115078]: osdmap e1664: 8 total, 8 up, 6 in 2024-08-17T18:18:25.897 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:25 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 44388/180051 objects degraded (24.653%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:25.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:25.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:25.941 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:25.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:26.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:26.043 INFO:tasks.thrashosds.thrasher:in_osds: [2, 4, 6, 0, 5, 7] out_osds: [1, 3] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:18:26.043 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:18:26.043 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-08-17T18:18:26.043 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd in 3 2024-08-17T18:18:26.075 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:26.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:26.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:26.177 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:26.209 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:26.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:26.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:26.311 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:26.333 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:26.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:26.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:26.416 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:18:26.434 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:26.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:26.461 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:26.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:26.562 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:26.585 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:26.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:26.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:26.687 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:26.709 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:26.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:26.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:26.811 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:26.833 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:26.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:26.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:26.911 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:26 smithi136 ceph-mon[109861]: osdmap e1665: 8 total, 8 up, 6 in 2024-08-17T18:18:26.911 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:26 smithi136 ceph-mon[115078]: osdmap e1665: 8 total, 8 up, 6 in 2024-08-17T18:18:26.935 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:26.967 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:26.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:26.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:26 smithi165 ceph-mon[105517]: osdmap e1665: 8 total, 8 up, 6 in 2024-08-17T18:18:27.069 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:27.097 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:27.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:27.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:27.199 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:27.221 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:27.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:27.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:27.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:27.350 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:27.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:27.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:27.452 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:27.480 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:27.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:27.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:27.576 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:18:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:18:27] "GET /metrics HTTP/1.1" 200 36776 "" "Prometheus/2.51.0" 2024-08-17T18:18:27.582 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:27.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:27.619 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:27.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:27.720 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:27.755 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:27.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:27.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:27.857 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:27.881 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:27.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:27.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:27.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:27 smithi136 ceph-mon[109861]: pgmap v3194: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 3.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 301 op/s; 44388/179397 objects degraded (24.743%); 22571/179397 objects misplaced (12.582%); 4.7 MiB/s, 185 objects/s recovering 2024-08-17T18:18:27.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:27 smithi136 ceph-mon[115078]: pgmap v3194: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 3.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 301 op/s; 44388/179397 objects degraded (24.743%); 22571/179397 objects misplaced (12.582%); 4.7 MiB/s, 185 objects/s recovering 2024-08-17T18:18:27.983 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:28.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:27 smithi165 ceph-mon[105517]: pgmap v3194: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+degraded+remapped, 86 active+clean, 13 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 3.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 301 op/s; 44388/179397 objects degraded (24.743%); 22571/179397 objects misplaced (12.582%); 4.7 MiB/s, 185 objects/s recovering 2024-08-17T18:18:28.017 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:28.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:28.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:28.119 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:28.144 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:28.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:28.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:28.218 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:18:28.219 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:18:28.222 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:18:28.239 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:18:28.239 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:18:28.246 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:28.267 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:18:28.268 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:18:28.278 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:28.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:28.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:28.292 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:18:28.292 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:18:28.313 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:18:28.314 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:18:28.335 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:18:28.335 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:18:28.366 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:18:28.367 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:18:28.380 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:28.391 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:18:28.391 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:18:28.408 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:28.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:28.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:28.510 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:28.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:28.535 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:28.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:28.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:28.665 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:28.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:28.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:28.767 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:28.788 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:28.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:28.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:28.820 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:28 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:18:28.820 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:28 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/315954764' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:18:28.820 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:28 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:28.820 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:28 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:28.820 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:28 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:18:28.820 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:28 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/315954764' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:18:28.820 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:28 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:28.820 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:28 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:28.890 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:28.919 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:28.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:28.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:28 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:18:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:28 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/315954764' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:18:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:28 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:28 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:29.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:29.048 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:29.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:29.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:29.150 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:29.181 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:29.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:29.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:29.283 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:29.305 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:29.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:29.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:29.407 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:29.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:29 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a[109838]: 2024-08-17T18:18:29.141+0000 7f5b8e77d640 -1 mon.a@0(leader).osd e1665 definitely_dead 0 2024-08-17T18:18:29.451 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:29.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:29.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:29.553 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:29.577 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:29.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:29.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:29.609 INFO:teuthology.orchestra.run.smithi136.stderr:marked in osd.3. 2024-08-17T18:18:29.679 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:29.729 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:29.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:29.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:29.831 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:29.852 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:29.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:29.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:29.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:29 smithi136 ceph-mon[109861]: pgmap v3195: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+remapped, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 12 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 3.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 543 op/s; 40150/173484 objects degraded (23.143%); 24821/173484 objects misplaced (14.307%); 9.4 MiB/s, 323 objects/s recovering 2024-08-17T18:18:29.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:29 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1350467826' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-08-17T18:18:29.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:29 smithi136 ceph-mon[115078]: pgmap v3195: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+remapped, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 12 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 3.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 543 op/s; 40150/173484 objects degraded (23.143%); 24821/173484 objects misplaced (14.307%); 9.4 MiB/s, 323 objects/s recovering 2024-08-17T18:18:29.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:29 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1350467826' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-08-17T18:18:29.954 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:29.978 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:29.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:29.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:29 smithi165 ceph-mon[105517]: pgmap v3195: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+remapped, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 12 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 3.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 543 op/s; 40150/173484 objects degraded (23.143%); 24821/173484 objects misplaced (14.307%); 9.4 MiB/s, 323 objects/s recovering 2024-08-17T18:18:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:29 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1350467826' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-08-17T18:18:30.081 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:30.106 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:30.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:30.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:30.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:30.234 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:30.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:30.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:30.336 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:30.359 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:30.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:30.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:30.461 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:30.483 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:30.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:30.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:30.503 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-08-17T18:18:30.585 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:30.610 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:30.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:30.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:30.620 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:18:30.712 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:30.751 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:30.752 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:30 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1350467826' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-08-17T18:18:30.752 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:30 smithi136 ceph-mon[115078]: osdmap e1666: 8 total, 8 up, 7 in 2024-08-17T18:18:30.752 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:30 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 40150/173484 objects degraded (23.143%), 15 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:30.752 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:30 smithi136 ceph-mon[115078]: osdmap e1667: 8 total, 8 up, 7 in 2024-08-17T18:18:30.752 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:30 smithi136 ceph-mon[115078]: 57.11 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'8117 2024-08-17T18:18:30.752 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:30 smithi136 ceph-mon[115078]: 57.d starting backfill to osd.3 from (0'0,0'0] MAX to 1665'10880 2024-08-17T18:18:30.752 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:30 smithi136 ceph-mon[115078]: 57.11 continuing backfill to osd.4 from (1611'100,1665'8117] MIN to 1665'8117 2024-08-17T18:18:30.752 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:30 smithi136 ceph-mon[115078]: 57.11 continuing backfill to osd.5 from (1611'100,1665'8117] MIN to 1665'8117 2024-08-17T18:18:30.752 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:30.752 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:30 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:30.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:30.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:30.754 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:30 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1350467826' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-08-17T18:18:30.754 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:30 smithi136 ceph-mon[109861]: osdmap e1666: 8 total, 8 up, 7 in 2024-08-17T18:18:30.754 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:30 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 40150/173484 objects degraded (23.143%), 15 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:30.755 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:30 smithi136 ceph-mon[109861]: osdmap e1667: 8 total, 8 up, 7 in 2024-08-17T18:18:30.755 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:30 smithi136 ceph-mon[109861]: 57.11 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'8117 2024-08-17T18:18:30.755 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:30 smithi136 ceph-mon[109861]: 57.d starting backfill to osd.3 from (0'0,0'0] MAX to 1665'10880 2024-08-17T18:18:30.755 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:30 smithi136 ceph-mon[109861]: 57.11 continuing backfill to osd.4 from (1611'100,1665'8117] MIN to 1665'8117 2024-08-17T18:18:30.755 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:30 smithi136 ceph-mon[109861]: 57.11 continuing backfill to osd.5 from (1611'100,1665'8117] MIN to 1665'8117 2024-08-17T18:18:30.755 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:30.755 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:30 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:30.855 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:30.881 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:30.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:30.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:30.917 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:30 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1350467826' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-08-17T18:18:30.918 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:30 smithi165 ceph-mon[105517]: osdmap e1666: 8 total, 8 up, 7 in 2024-08-17T18:18:30.918 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:30 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 40150/173484 objects degraded (23.143%), 15 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:30.918 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:30 smithi165 ceph-mon[105517]: osdmap e1667: 8 total, 8 up, 7 in 2024-08-17T18:18:30.918 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:30 smithi165 ceph-mon[105517]: 57.11 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'8117 2024-08-17T18:18:30.918 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:30 smithi165 ceph-mon[105517]: 57.d starting backfill to osd.3 from (0'0,0'0] MAX to 1665'10880 2024-08-17T18:18:30.918 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:30 smithi165 ceph-mon[105517]: 57.11 continuing backfill to osd.4 from (1611'100,1665'8117] MIN to 1665'8117 2024-08-17T18:18:30.918 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:30 smithi165 ceph-mon[105517]: 57.11 continuing backfill to osd.5 from (1611'100,1665'8117] MIN to 1665'8117 2024-08-17T18:18:30.918 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:30 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:30.919 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:30 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:30.983 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:31.006 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:31.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:31.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:31.108 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:31.129 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:31.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:31.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:31.231 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:31.259 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:31.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:31.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:31.361 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:31.383 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:31.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:31.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:31.485 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:31.508 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:31.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:31.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:31.610 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:31.632 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:31.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:31.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:31.663 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:18:31.735 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:31.776 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:31.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:31.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:31.877 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:31.898 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:31.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:31.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:31.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:31 smithi136 ceph-mon[115078]: 57.a starting backfill to osd.3 from (0'0,0'0] MAX to 1665'11124 2024-08-17T18:18:31.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:31 smithi136 ceph-mon[115078]: 57.a continuing backfill to osd.7 from (1633'7207,1665'11124] MIN to 1665'11124 2024-08-17T18:18:31.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:31 smithi136 ceph-mon[115078]: 57.5 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'8061 2024-08-17T18:18:31.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:31 smithi136 ceph-mon[115078]: 57.5 starting backfill to osd.7 from (1613'3461,1665'8060] MIN to 1665'8061 2024-08-17T18:18:31.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:31 smithi136 ceph-mon[115078]: pgmap v3198: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+remapped, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 12 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 3.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 372 op/s; 40150/173484 objects degraded (23.143%); 24821/173484 objects misplaced (14.307%); 6.5 MiB/s, 190 objects/s recovering 2024-08-17T18:18:31.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:31 smithi136 ceph-mon[115078]: osdmap e1668: 8 total, 8 up, 7 in 2024-08-17T18:18:31.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:31 smithi136 ceph-mon[115078]: 57.9 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'8002 2024-08-17T18:18:31.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:31 smithi136 ceph-mon[115078]: 57.12 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7855 2024-08-17T18:18:31.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:31 smithi136 ceph-mon[115078]: 57.2 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7927 2024-08-17T18:18:31.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:31 smithi136 ceph-mon[115078]: 57.19 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7902 2024-08-17T18:18:31.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:31 smithi136 ceph-mon[109861]: 57.a starting backfill to osd.3 from (0'0,0'0] MAX to 1665'11124 2024-08-17T18:18:31.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:31 smithi136 ceph-mon[109861]: 57.a continuing backfill to osd.7 from (1633'7207,1665'11124] MIN to 1665'11124 2024-08-17T18:18:31.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:31 smithi136 ceph-mon[109861]: 57.5 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'8061 2024-08-17T18:18:31.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:31 smithi136 ceph-mon[109861]: 57.5 starting backfill to osd.7 from (1613'3461,1665'8060] MIN to 1665'8061 2024-08-17T18:18:31.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:31 smithi136 ceph-mon[109861]: pgmap v3198: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+remapped, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 12 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 3.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 372 op/s; 40150/173484 objects degraded (23.143%); 24821/173484 objects misplaced (14.307%); 6.5 MiB/s, 190 objects/s recovering 2024-08-17T18:18:31.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:31 smithi136 ceph-mon[109861]: osdmap e1668: 8 total, 8 up, 7 in 2024-08-17T18:18:31.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:31 smithi136 ceph-mon[109861]: 57.9 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'8002 2024-08-17T18:18:31.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:31 smithi136 ceph-mon[109861]: 57.12 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7855 2024-08-17T18:18:31.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:31 smithi136 ceph-mon[109861]: 57.2 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7927 2024-08-17T18:18:31.949 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:31 smithi136 ceph-mon[109861]: 57.19 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7902 2024-08-17T18:18:32.000 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:32.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:31 smithi165 ceph-mon[105517]: 57.a starting backfill to osd.3 from (0'0,0'0] MAX to 1665'11124 2024-08-17T18:18:32.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:31 smithi165 ceph-mon[105517]: 57.a continuing backfill to osd.7 from (1633'7207,1665'11124] MIN to 1665'11124 2024-08-17T18:18:32.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:31 smithi165 ceph-mon[105517]: 57.5 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'8061 2024-08-17T18:18:32.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:31 smithi165 ceph-mon[105517]: 57.5 starting backfill to osd.7 from (1613'3461,1665'8060] MIN to 1665'8061 2024-08-17T18:18:32.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:31 smithi165 ceph-mon[105517]: pgmap v3198: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+remapped, 2 active+recovery_wait+degraded+remapped, 87 active+clean, 12 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait; 3.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 372 op/s; 40150/173484 objects degraded (23.143%); 24821/173484 objects misplaced (14.307%); 6.5 MiB/s, 190 objects/s recovering 2024-08-17T18:18:32.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:31 smithi165 ceph-mon[105517]: osdmap e1668: 8 total, 8 up, 7 in 2024-08-17T18:18:32.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:31 smithi165 ceph-mon[105517]: 57.9 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'8002 2024-08-17T18:18:32.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:31 smithi165 ceph-mon[105517]: 57.12 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7855 2024-08-17T18:18:32.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:31 smithi165 ceph-mon[105517]: 57.2 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7927 2024-08-17T18:18:32.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:31 smithi165 ceph-mon[105517]: 57.19 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7902 2024-08-17T18:18:32.021 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:32.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:32.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:32.123 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:32.146 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:32.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:32.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:32.248 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:32.271 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:32.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:32.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:32.373 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:32.400 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:32.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:32.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:32.501 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:32.524 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:32.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:32.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:32.625 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:32.648 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:32.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:32.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:32.750 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:32.771 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:32.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:32.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:32.874 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:32.897 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:32.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:32.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:32.999 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:32 smithi165 ceph-mon[105517]: osdmap e1669: 8 total, 8 up, 7 in 2024-08-17T18:18:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:32 smithi165 ceph-mon[105517]: 57.10 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7982 2024-08-17T18:18:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:32 smithi165 ceph-mon[105517]: 57.13 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7846 2024-08-17T18:18:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:32 smithi165 ceph-mon[105517]: 57.0 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7972 2024-08-17T18:18:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:32 smithi165 ceph-mon[105517]: 57.4 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7848 2024-08-17T18:18:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:32 smithi165 ceph-mon[105517]: 57.3 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7877 2024-08-17T18:18:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:32 smithi165 ceph-mon[105517]: 57.15 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7949 2024-08-17T18:18:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:32 smithi165 ceph-mon[105517]: 57.14 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7824 2024-08-17T18:18:33.021 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:33.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:33.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:33.067 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:32 smithi136 ceph-mon[109861]: osdmap e1669: 8 total, 8 up, 7 in 2024-08-17T18:18:33.067 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:32 smithi136 ceph-mon[109861]: 57.10 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7982 2024-08-17T18:18:33.067 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:32 smithi136 ceph-mon[109861]: 57.13 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7846 2024-08-17T18:18:33.067 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:32 smithi136 ceph-mon[109861]: 57.0 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7972 2024-08-17T18:18:33.067 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:32 smithi136 ceph-mon[109861]: 57.4 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7848 2024-08-17T18:18:33.067 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:32 smithi136 ceph-mon[109861]: 57.3 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7877 2024-08-17T18:18:33.067 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:32 smithi136 ceph-mon[109861]: 57.15 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7949 2024-08-17T18:18:33.067 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:32 smithi136 ceph-mon[109861]: 57.14 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7824 2024-08-17T18:18:33.068 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:32 smithi136 ceph-mon[115078]: osdmap e1669: 8 total, 8 up, 7 in 2024-08-17T18:18:33.068 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:32 smithi136 ceph-mon[115078]: 57.10 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7982 2024-08-17T18:18:33.068 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:32 smithi136 ceph-mon[115078]: 57.13 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7846 2024-08-17T18:18:33.068 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:32 smithi136 ceph-mon[115078]: 57.0 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7972 2024-08-17T18:18:33.068 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:32 smithi136 ceph-mon[115078]: 57.4 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7848 2024-08-17T18:18:33.068 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:32 smithi136 ceph-mon[115078]: 57.3 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7877 2024-08-17T18:18:33.068 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:32 smithi136 ceph-mon[115078]: 57.15 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7949 2024-08-17T18:18:33.068 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:32 smithi136 ceph-mon[115078]: 57.14 starting backfill to osd.3 from (0'0,0'0] MAX to 1665'7824 2024-08-17T18:18:33.124 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:33.145 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:33.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:33.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:33.194 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:18:33.194 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:18:33.248 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:33.274 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:33.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:33.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:33.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:33.400 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:33.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:33.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:33.501 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:33.523 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:33.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:33.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:33.625 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:33.647 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:33.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:33.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:33.728 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:33 smithi136 ceph-mon[109861]: pgmap v3201: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 3 active+undersized+remapped, 1 active+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 82 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 1 activating+degraded+remapped, 6 active+remapped+backfill_wait; 3.5 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 262 op/s; 30961/170334 objects degraded (18.177%); 42401/170334 objects misplaced (24.893%); 16 MiB/s, 256 objects/s recovering 2024-08-17T18:18:33.729 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:33 smithi136 ceph-mon[109861]: osdmap e1670: 8 total, 8 up, 7 in 2024-08-17T18:18:33.749 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:33.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:33.771 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:33.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:33.872 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:33.900 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:33.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:33.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:33.989 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:33 smithi136 ceph-mon[115078]: pgmap v3201: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 3 active+undersized+remapped, 1 active+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 82 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 1 activating+degraded+remapped, 6 active+remapped+backfill_wait; 3.5 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 262 op/s; 30961/170334 objects degraded (18.177%); 42401/170334 objects misplaced (24.893%); 16 MiB/s, 256 objects/s recovering 2024-08-17T18:18:33.989 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:33 smithi136 ceph-mon[115078]: osdmap e1670: 8 total, 8 up, 7 in 2024-08-17T18:18:34.003 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:34.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:33 smithi165 ceph-mon[105517]: pgmap v3201: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+degraded, 3 active+undersized+remapped, 1 active+remapped, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 82 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 1 activating+degraded+remapped, 6 active+remapped+backfill_wait; 3.5 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 262 op/s; 30961/170334 objects degraded (18.177%); 42401/170334 objects misplaced (24.893%); 16 MiB/s, 256 objects/s recovering 2024-08-17T18:18:34.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:33 smithi165 ceph-mon[105517]: osdmap e1670: 8 total, 8 up, 7 in 2024-08-17T18:18:34.024 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:34.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:34.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:34.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:34.148 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:34.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:34.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:34.250 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:34.272 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:34.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:34.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:34.374 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:34.399 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:34.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:34.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:34.408 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:18:34.500 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:34.527 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:34.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:34.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:34.629 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:34.654 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:34.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:34.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:34.741 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:18:34.755 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:34.792 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:34.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:34.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:34.821 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:34 smithi136 ceph-mon[109861]: osdmap e1671: 8 total, 8 up, 7 in 2024-08-17T18:18:34.821 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:34 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2300771401' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:18:34.821 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:34 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:18:34.821 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:34 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 30961/170334 objects degraded (18.177%), 16 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:34.822 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:34 smithi136 ceph-mon[115078]: osdmap e1671: 8 total, 8 up, 7 in 2024-08-17T18:18:34.822 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:34 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2300771401' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:18:34.822 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:34 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:18:34.822 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:34 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 30961/170334 objects degraded (18.177%), 16 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:34.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:34.927 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:34.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:34.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:34 smithi165 ceph-mon[105517]: osdmap e1671: 8 total, 8 up, 7 in 2024-08-17T18:18:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:34 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2300771401' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:18:35.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:34 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:18:35.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:34 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 30961/170334 objects degraded (18.177%), 16 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:35.029 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:35.050 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:35.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:35.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:35.152 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:35.185 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:35.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:35.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:35.287 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:35.312 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:35.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:35.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:35.414 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:35.439 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:35.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:35.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:35.503 INFO:tasks.thrashosds.thrasher:in_osds: [2, 4, 6, 0, 5, 7, 3] out_osds: [1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:18:35.504 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:18:35.504 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-08-17T18:18:35.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T18:18:35.541 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:35.570 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-17T18:18:35.571 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:35.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:35.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:35.673 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:35.696 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:35.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:35.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:35.798 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:35.820 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:35.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:35.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:35.905 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:35 smithi136 ceph-mon[109861]: pgmap v3204: 109 pgs: 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 3 active+undersized+remapped, 1 active+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 79 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 1 activating+degraded+remapped, 8 active+remapped+backfill_wait; 3.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 300 op/s; 33748/169872 objects degraded (19.867%); 44828/169872 objects misplaced (26.389%); 16 MiB/s, 261 objects/s recovering 2024-08-17T18:18:35.905 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:35 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:18:35.905 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:35 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:18:35.905 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:35 smithi136 ceph-mon[109861]: osdmap e1672: 8 total, 8 up, 7 in 2024-08-17T18:18:35.905 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:35 smithi136 ceph-mon[115078]: pgmap v3204: 109 pgs: 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 3 active+undersized+remapped, 1 active+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 79 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 1 activating+degraded+remapped, 8 active+remapped+backfill_wait; 3.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 300 op/s; 33748/169872 objects degraded (19.867%); 44828/169872 objects misplaced (26.389%); 16 MiB/s, 261 objects/s recovering 2024-08-17T18:18:35.906 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:35 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:18:35.906 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:35 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:18:35.906 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:35 smithi136 ceph-mon[115078]: osdmap e1672: 8 total, 8 up, 7 in 2024-08-17T18:18:35.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:35.959 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:35.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:35.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:36.008 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:35 smithi165 ceph-mon[105517]: pgmap v3204: 109 pgs: 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 3 active+undersized+remapped, 1 active+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 79 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 1 activating+degraded+remapped, 8 active+remapped+backfill_wait; 3.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 300 op/s; 33748/169872 objects degraded (19.867%); 44828/169872 objects misplaced (26.389%); 16 MiB/s, 261 objects/s recovering 2024-08-17T18:18:36.008 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:35 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:18:36.008 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:35 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:18:36.008 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:35 smithi165 ceph-mon[105517]: osdmap e1672: 8 total, 8 up, 7 in 2024-08-17T18:18:36.061 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:36.084 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:36.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:36.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:36.186 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:36.219 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:36.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:36.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:36.322 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:36.345 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:36.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:36.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:36.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:36.478 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:36.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:36.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:36.582 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:36.594 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:18:36.606 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:36.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:36.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:36.707 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:36.741 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:36.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:36.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:36.843 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:36.867 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:36.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:36.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:36.970 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:36.997 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:36.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:36.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:37.002 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:18:37.026 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:18:37.027 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:18:37.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:37.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:37.126 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:37.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:37.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:37.255 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:37.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:37.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:37.335 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:18:37.335 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:18:37.356 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:18:37.357 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:18:37.357 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:37.376 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:18:37.376 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:18:37.382 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:37.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:37.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:37.394 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:18:37.395 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:18:37.413 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:18:37.413 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:18:37.429 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:18:37.429 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:18:37.485 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:37.507 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:37.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:37.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:37.569 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:18:37.570 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:18:37.589 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:18:37.589 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:18:37.590 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:18:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:18:37] "GET /metrics HTTP/1.1" 200 36766 "" "Prometheus/2.51.0" 2024-08-17T18:18:37.609 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:37.634 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:37.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:37.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:37.736 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:37.767 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:37.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:37.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:37.869 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:37.900 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:37.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:37.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:37.933 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:18:37.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:37 smithi136 ceph-mon[109861]: pgmap v3206: 109 pgs: 1 active+remapped, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 3 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 79 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 1 activating+degraded+remapped, 8 active+remapped+backfill_wait; 3.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 240 op/s; 33748/169629 objects degraded (19.895%); 44704/169629 objects misplaced (26.354%); 7.0 MiB/s, 113 objects/s recovering 2024-08-17T18:18:37.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:37 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1878181969' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:18:37.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:37 smithi136 ceph-mon[115078]: pgmap v3206: 109 pgs: 1 active+remapped, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 3 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 79 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 1 activating+degraded+remapped, 8 active+remapped+backfill_wait; 3.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 240 op/s; 33748/169629 objects degraded (19.895%); 44704/169629 objects misplaced (26.354%); 7.0 MiB/s, 113 objects/s recovering 2024-08-17T18:18:37.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:37 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1878181969' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:18:38.003 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:38.007 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:37 smithi165 ceph-mon[105517]: pgmap v3206: 109 pgs: 1 active+remapped, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 3 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 79 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 1 activating+degraded+remapped, 8 active+remapped+backfill_wait; 3.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 240 op/s; 33748/169629 objects degraded (19.895%); 44704/169629 objects misplaced (26.354%); 7.0 MiB/s, 113 objects/s recovering 2024-08-17T18:18:38.007 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:37 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1878181969' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:18:38.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:38.027 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:38.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:38.128 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:38.152 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:38.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:38.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:38.254 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:38.277 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:38.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:38.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:38.380 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:38.411 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:38.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:38.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:38.513 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:38.534 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:38.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:38.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:38.661 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:38.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:38.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:38.763 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:38.802 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:38.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:38.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:38.908 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:38.931 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:38.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:38.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:39.034 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:39.056 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:39.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:39.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:39.158 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:39.180 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:39.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:39.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:39.281 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:39.306 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:39.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:39.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:39.408 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:39.504 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:39.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:39.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:39.606 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:39.636 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:39.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:39.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:39.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:39.774 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:39.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:39.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:39.876 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:39.908 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:39.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:39.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:40.010 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:39 smithi165 ceph-mon[105517]: pgmap v3207: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+remapped, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 169 op/s; 33364/167286 objects degraded (19.944%); 43841/167286 objects misplaced (26.207%); 8.4 MiB/s, 134 objects/s recovering 2024-08-17T18:18:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:39 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 33364/167286 objects degraded (19.944%), 16 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:40.033 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:40.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:40.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:40.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:40.186 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:40.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:40.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:40.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:39 smithi136 ceph-mon[109861]: pgmap v3207: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+remapped, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 169 op/s; 33364/167286 objects degraded (19.944%); 43841/167286 objects misplaced (26.207%); 8.4 MiB/s, 134 objects/s recovering 2024-08-17T18:18:40.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:39 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 33364/167286 objects degraded (19.944%), 16 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:40.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:39 smithi136 ceph-mon[115078]: pgmap v3207: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+remapped, 1 active+recovery_wait+remapped, 1 active+undersized+degraded+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 169 op/s; 33364/167286 objects degraded (19.944%); 43841/167286 objects misplaced (26.207%); 8.4 MiB/s, 134 objects/s recovering 2024-08-17T18:18:40.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:39 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 33364/167286 objects degraded (19.944%), 16 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:40.288 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:40.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:40.314 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:40.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:40.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:40.468 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:40.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:40.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:40.569 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:40.627 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:40.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:40.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:40.729 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:40.753 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:40.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:40.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:40.858 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:40.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:40.888 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:40.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:40.989 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:41.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:41.012 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:41.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:41.113 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:41.137 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:41.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:41.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:41.239 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:41.261 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:41.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:41.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:41.363 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:41.386 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:41.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:41.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:41.488 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:41.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:41.527 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:41.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:41.571 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:18:41.571 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:18:41.571 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-08-17T18:18:41.571 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-08-17T18:18:41.572 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:18:41.628 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:41.655 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:41.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:41.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:41.757 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:41.791 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:41.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:41.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:41.892 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:41.924 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:41.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:41.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:42.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:41 smithi165 ceph-mon[105517]: pgmap v3208: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 173 op/s; 33238/166293 objects degraded (19.988%); 43575/166293 objects misplaced (26.204%); 7.0 MiB/s, 112 objects/s recovering 2024-08-17T18:18:42.026 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:42.050 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:42.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:42.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:42.152 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:42.175 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:42.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:42.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:42.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:41 smithi136 ceph-mon[109861]: pgmap v3208: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 173 op/s; 33238/166293 objects degraded (19.988%); 43575/166293 objects misplaced (26.204%); 7.0 MiB/s, 112 objects/s recovering 2024-08-17T18:18:42.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:41 smithi136 ceph-mon[115078]: pgmap v3208: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 173 op/s; 33238/166293 objects degraded (19.988%); 43575/166293 objects misplaced (26.204%); 7.0 MiB/s, 112 objects/s recovering 2024-08-17T18:18:42.277 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:42.303 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:42.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:42.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:42.404 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:42.429 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:42.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:42.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:42.531 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:42.553 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:42.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:42.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:42.655 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:42.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:42.693 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:42.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:42.794 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:42.817 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:42.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:42.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:42.918 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:42.948 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-17T18:18:42.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:42.987 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:42.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:43.089 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:43.113 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:43.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:43.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:43.215 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:43.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:43.268 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:43.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:43.369 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:43.393 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:43.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:43.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:43.495 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:43.530 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:43.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:43.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:43.632 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:43.655 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:43.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:43.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:43.756 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:43.780 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:43.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:43.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:43.843 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:18:43.882 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:43.905 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:43.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:43.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:43.982 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:43 smithi165 ceph-mon[105517]: pgmap v3209: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 319 op/s; 32618/161943 objects degraded (20.142%); 42218/161943 objects misplaced (26.070%); 13 MiB/s, 207 objects/s recovering 2024-08-17T18:18:44.007 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:44.033 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:43 smithi136 ceph-mon[109861]: pgmap v3209: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 319 op/s; 32618/161943 objects degraded (20.142%); 42218/161943 objects misplaced (26.070%); 13 MiB/s, 207 objects/s recovering 2024-08-17T18:18:44.033 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:43 smithi136 ceph-mon[115078]: pgmap v3209: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 319 op/s; 32618/161943 objects degraded (20.142%); 42218/161943 objects misplaced (26.070%); 13 MiB/s, 207 objects/s recovering 2024-08-17T18:18:44.036 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:44.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:44.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:44.139 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:44.182 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:44.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:44.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:44.264 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:18:44.284 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:44.322 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:44.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:44.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:44.425 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:44.447 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:44.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:44.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:44.549 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:44.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:44.573 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:44.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:44.674 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:44.701 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:44.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:44.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:44.759 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:18:44.803 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:44.825 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:44.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:44.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:44.927 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:44.951 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:44.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:44.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:44 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 32618/161943 objects degraded (20.142%), 16 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:45.052 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:45.080 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:45.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:45.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:45.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:44 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 32618/161943 objects degraded (20.142%), 16 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:45.082 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:44 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 32618/161943 objects degraded (20.142%), 16 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:45.182 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:45.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:45.206 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:45.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:45.307 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:45.329 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:45.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:45.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:45.431 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:45.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:45.458 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:45.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:45.562 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:45.585 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:45.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:45.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:45.686 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:45.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:45.726 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:45.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:45.827 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:45.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:45.850 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:45.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:45.951 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:45.972 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:45.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:45.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:46.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:45 smithi165 ceph-mon[105517]: pgmap v3210: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 264 op/s; 32618/161943 objects degraded (20.142%); 42218/161943 objects misplaced (26.070%); 11 MiB/s, 179 objects/s recovering 2024-08-17T18:18:46.054 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:18:46.059 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:45 smithi136 ceph-mon[109861]: pgmap v3210: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 264 op/s; 32618/161943 objects degraded (20.142%); 42218/161943 objects misplaced (26.070%); 11 MiB/s, 179 objects/s recovering 2024-08-17T18:18:46.060 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:45 smithi136 ceph-mon[115078]: pgmap v3210: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 264 op/s; 32618/161943 objects degraded (20.142%); 42218/161943 objects misplaced (26.070%); 11 MiB/s, 179 objects/s recovering 2024-08-17T18:18:46.074 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:46.103 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:46.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:46.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:46.204 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:46.230 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:46.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:46.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:46.332 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:46.361 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:46.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:46.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:46.463 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:46.494 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:46.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:46.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:46.596 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:46.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:46.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:46.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:46.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:46.757 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:46.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:46.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:46.859 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:46.895 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:46.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:46.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:46.997 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:47.021 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:47.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:47.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:47.123 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:47.149 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:47.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:47.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:47.251 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:47.281 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:47.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:47.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:47.382 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:47.413 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:47.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:47.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:47.515 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:47.545 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:47.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:47.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:47.648 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:47.672 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:47.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:47.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:47.764 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:18:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:18:47] "GET /metrics HTTP/1.1" 200 36766 "" "Prometheus/2.51.0" 2024-08-17T18:18:47.774 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:47.823 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:47.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:47.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:47.925 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:47.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:47.948 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:47.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:48.050 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:48.085 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:47 smithi136 ceph-mon[109861]: pgmap v3211: 109 pgs: 1 active+recovery_wait+remapped, 1 active+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 268 op/s; 32495/160416 objects degraded (20.257%); 41694/160416 objects misplaced (25.991%); 9.9 MiB/s, 158 objects/s recovering 2024-08-17T18:18:48.086 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:47 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:48.086 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:47 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:48.086 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:47 smithi136 ceph-mon[115078]: pgmap v3211: 109 pgs: 1 active+recovery_wait+remapped, 1 active+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 268 op/s; 32495/160416 objects degraded (20.257%); 41694/160416 objects misplaced (25.991%); 9.9 MiB/s, 158 objects/s recovering 2024-08-17T18:18:48.086 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:47 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:48.086 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:47 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:48.100 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:48.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:48.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:48.123 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:47 smithi165 ceph-mon[105517]: pgmap v3211: 109 pgs: 1 active+recovery_wait+remapped, 1 active+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 268 op/s; 32495/160416 objects degraded (20.257%); 41694/160416 objects misplaced (25.991%); 9.9 MiB/s, 158 objects/s recovering 2024-08-17T18:18:48.123 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:47 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:48.123 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:47 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:48.202 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:48.236 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:48.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:48.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:48.338 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:48.363 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:48.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:48.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:48.465 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:48.505 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:48.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:48.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:48.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:48.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:48.640 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:48.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:48.741 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:48.779 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:48.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:48.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:48.881 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:48.906 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:48.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:48.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:49.008 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:49.050 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:49.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:49.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:49.152 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:49.192 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:49.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:49.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:49.295 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:49.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:49.325 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:49.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:49.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:49.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:49.469 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:49.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:49.570 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:49.601 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:49.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:49.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:49.705 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:49.750 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:49.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:49.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:49.759 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:18:49.772 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:18:49.789 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:18:49.789 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T18:18:49.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T18:18:49.789 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:18:49.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.789 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:178743 56.c 56:3082e7b9:::benchmark_data_smithi136_356456_object52039:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:18:49.789 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:03.741426+0000", 2024-08-17T18:18:49.789 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 406.042149247, 2024-08-17T18:18:49.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1284821909999998, 2024-08-17T18:18:49.789 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:49.789 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:49.790 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:49.790 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:18:49.790 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:18:49.790 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 178743 2024-08-17T18:18:49.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.790 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:49.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:49.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.741426+0000", 2024-08-17T18:18:49.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:49.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.741426+0000", 2024-08-17T18:18:49.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:49.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.741425+0000", 2024-08-17T18:18:49.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:18:49.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:49.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.741429+0000", 2024-08-17T18:18:49.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0180000000000003e-06 2024-08-17T18:18:49.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:49.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.741440+0000", 2024-08-17T18:18:49.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0987e-05 2024-08-17T18:18:49.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.793 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:49.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.741447+0000", 2024-08-17T18:18:49.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.764e-06 2024-08-17T18:18:49.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.793 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.741494+0000", 2024-08-17T18:18:49.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7239e-05 2024-08-17T18:18:49.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:18:49.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.741504+0000", 2024-08-17T18:18:49.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0508e-05 2024-08-17T18:18:49.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.636664+0000", 2024-08-17T18:18:49.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.89515925799999996 2024-08-17T18:18:49.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:18:49.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.636677+0000", 2024-08-17T18:18:49.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2901999999999999e-05 2024-08-17T18:18:49.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.833030+0000", 2024-08-17T18:18:49.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1963536420000001 2024-08-17T18:18:49.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:49.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.835819+0000", 2024-08-17T18:18:49.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027889910000000002 2024-08-17T18:18:49.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2,4", 2024-08-17T18:18:49.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.836104+0000", 2024-08-17T18:18:49.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028455599999999999 2024-08-17T18:18:49.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.851359+0000", 2024-08-17T18:18:49.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015255651 2024-08-17T18:18:49.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.851398+0000", 2024-08-17T18:18:49.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8257000000000003e-05 2024-08-17T18:18:49.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:49.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.856494+0000", 2024-08-17T18:18:49.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050968100000000002 2024-08-17T18:18:49.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.869682+0000", 2024-08-17T18:18:49.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013187211000000001 2024-08-17T18:18:49.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:49.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.869784+0000", 2024-08-17T18:18:49.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010231899999999999 2024-08-17T18:18:49.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:49.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.869908+0000", 2024-08-17T18:18:49.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012377699999999999 2024-08-17T18:18:49.800 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.800 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:49.800 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.800 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:178760 56.c 56:32895434:::benchmark_data_smithi136_356456_object52056:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:18:49.800 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:03.770242+0000", 2024-08-17T18:18:49.800 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 406.01333247299999, 2024-08-17T18:18:49.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.104038246, 2024-08-17T18:18:49.801 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:49.801 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:49.801 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:49.801 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:18:49.801 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:18:49.801 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 178760 2024-08-17T18:18:49.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.801 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:49.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:49.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.770242+0000", 2024-08-17T18:18:49.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:49.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.770242+0000", 2024-08-17T18:18:49.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:49.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.770242+0000", 2024-08-17T18:18:49.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:18:49.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:49.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.770246+0000", 2024-08-17T18:18:49.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7670000000000003e-06 2024-08-17T18:18:49.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:49.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.770257+0000", 2024-08-17T18:18:49.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0920999999999999e-05 2024-08-17T18:18:49.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:49.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.770265+0000", 2024-08-17T18:18:49.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.43e-06 2024-08-17T18:18:49.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.770301+0000", 2024-08-17T18:18:49.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6751000000000001e-05 2024-08-17T18:18:49.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:18:49.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.770309+0000", 2024-08-17T18:18:49.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4869999999999996e-06 2024-08-17T18:18:49.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.638358+0000", 2024-08-17T18:18:49.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.86804906800000003 2024-08-17T18:18:49.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:18:49.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.638368+0000", 2024-08-17T18:18:49.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9310000000000003e-06 2024-08-17T18:18:49.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.856525+0000", 2024-08-17T18:18:49.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2181566720000001 2024-08-17T18:18:49.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:49.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.856666+0000", 2024-08-17T18:18:49.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000141824 2024-08-17T18:18:49.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2,4", 2024-08-17T18:18:49.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.856813+0000", 2024-08-17T18:18:49.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000146098 2024-08-17T18:18:49.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.862289+0000", 2024-08-17T18:18:49.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054769010000000002 2024-08-17T18:18:49.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.869646+0000", 2024-08-17T18:18:49.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073564140000000004 2024-08-17T18:18:49.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.869950+0000", 2024-08-17T18:18:49.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000304006 2024-08-17T18:18:49.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:49.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.874059+0000", 2024-08-17T18:18:49.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041089150000000003 2024-08-17T18:18:49.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:49.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.874172+0000", 2024-08-17T18:18:49.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113752 2024-08-17T18:18:49.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:49.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.874281+0000", 2024-08-17T18:18:49.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108119 2024-08-17T18:18:49.810 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.810 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:49.810 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.810 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:122081 57.a 57:53ce68b1:::benchmark_data_smithi136_389235_object6159:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1639)", 2024-08-17T18:18:49.810 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:54.535181+0000", 2024-08-17T18:18:49.810 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 175.248393445, 2024-08-17T18:18:49.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9832697580000005, 2024-08-17T18:18:49.811 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:49.811 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:49.811 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:49.811 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:49.811 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:49.811 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 122081 2024-08-17T18:18:49.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.811 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:49.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:49.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.535181+0000", 2024-08-17T18:18:49.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:49.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.535181+0000", 2024-08-17T18:18:49.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:49.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.535179+0000", 2024-08-17T18:18:49.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999976 2024-08-17T18:18:49.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:49.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.535190+0000", 2024-08-17T18:18:49.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0815000000000001e-05 2024-08-17T18:18:49.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:49.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.535205+0000", 2024-08-17T18:18:49.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4994000000000001e-05 2024-08-17T18:18:49.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:49.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.535213+0000", 2024-08-17T18:18:49.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0700000000000007e-06 2024-08-17T18:18:49.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.535305+0000", 2024-08-17T18:18:49.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1905999999999997e-05 2024-08-17T18:18:49.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:18:49.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.535317+0000", 2024-08-17T18:18:49.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2707e-05 2024-08-17T18:18:49.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.490000+0000", 2024-08-17T18:18:49.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9546827809999998 2024-08-17T18:18:49.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:49.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.490110+0000", 2024-08-17T18:18:49.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109694 2024-08-17T18:18:49.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4,7", 2024-08-17T18:18:49.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.490197+0000", 2024-08-17T18:18:49.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6959999999999994e-05 2024-08-17T18:18:49.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.492688+0000", 2024-08-17T18:18:49.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024907750000000002 2024-08-17T18:18:49.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:49.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.492711+0000", 2024-08-17T18:18:49.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3742000000000001e-05 2024-08-17T18:18:49.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.492744+0000", 2024-08-17T18:18:49.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2737999999999997e-05 2024-08-17T18:18:49.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.518347+0000", 2024-08-17T18:18:49.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.025602666999999999 2024-08-17T18:18:49.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:49.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.518375+0000", 2024-08-17T18:18:49.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8277e-05 2024-08-17T18:18:49.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:49.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.518451+0000", 2024-08-17T18:18:49.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6106000000000006e-05 2024-08-17T18:18:49.817 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.817 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:49.818 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.818 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:122098 57.a 57:51bdebe7:::benchmark_data_smithi136_389235_object6176:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1641)", 2024-08-17T18:18:49.818 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:57.114235+0000", 2024-08-17T18:18:49.818 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 172.66933982800001, 2024-08-17T18:18:49.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4821008249999998, 2024-08-17T18:18:49.818 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:49.818 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:49.818 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:49.818 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:49.818 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:49.818 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 122098 2024-08-17T18:18:49.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.819 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:49.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:49.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.114235+0000", 2024-08-17T18:18:49.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:49.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.114235+0000", 2024-08-17T18:18:49.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:49.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.114234+0000", 2024-08-17T18:18:49.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:18:49.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:49.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.114240+0000", 2024-08-17T18:18:49.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6050000000000003e-06 2024-08-17T18:18:49.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:49.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.114311+0000", 2024-08-17T18:18:49.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0778999999999994e-05 2024-08-17T18:18:49.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:49.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.114318+0000", 2024-08-17T18:18:49.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2019999999999998e-06 2024-08-17T18:18:49.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.114378+0000", 2024-08-17T18:18:49.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9960999999999997e-05 2024-08-17T18:18:49.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:18:49.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.114390+0000", 2024-08-17T18:18:49.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2199e-05 2024-08-17T18:18:49.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.492765+0000", 2024-08-17T18:18:49.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.378374617 2024-08-17T18:18:49.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:49.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.492902+0000", 2024-08-17T18:18:49.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000137298 2024-08-17T18:18:49.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4,7", 2024-08-17T18:18:49.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.493002+0000", 2024-08-17T18:18:49.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9696000000000002e-05 2024-08-17T18:18:49.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.495054+0000", 2024-08-17T18:18:49.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020514890000000001 2024-08-17T18:18:49.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.495096+0000", 2024-08-17T18:18:49.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2559999999999999e-05 2024-08-17T18:18:49.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:49.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.495240+0000", 2024-08-17T18:18:49.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000144375 2024-08-17T18:18:49.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.596152+0000", 2024-08-17T18:18:49.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100911139 2024-08-17T18:18:49.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:49.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.596213+0000", 2024-08-17T18:18:49.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1654000000000003e-05 2024-08-17T18:18:49.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:49.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.596336+0000", 2024-08-17T18:18:49.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122605 2024-08-17T18:18:49.826 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.827 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:49.827 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.827 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:160898 57.16 57:6bcdff34:::benchmark_data_smithi136_389235_object44976:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1659)", 2024-08-17T18:18:49.827 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:56.563419+0000", 2024-08-17T18:18:49.827 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 53.220155878, 2024-08-17T18:18:49.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0259137519999999, 2024-08-17T18:18:49.827 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:49.827 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:49.827 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:49.827 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:49.828 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:49.828 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 160898 2024-08-17T18:18:49.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.828 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:49.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:49.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.563419+0000", 2024-08-17T18:18:49.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:49.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.563419+0000", 2024-08-17T18:18:49.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:49.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.563417+0000", 2024-08-17T18:18:49.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:18:49.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:49.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.563427+0000", 2024-08-17T18:18:49.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0105000000000001e-05 2024-08-17T18:18:49.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:49.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.563454+0000", 2024-08-17T18:18:49.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.639e-05 2024-08-17T18:18:49.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:49.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.563464+0000", 2024-08-17T18:18:49.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0795000000000001e-05 2024-08-17T18:18:49.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.575209+0000", 2024-08-17T18:18:49.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011744675 2024-08-17T18:18:49.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:18:49.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.575220+0000", 2024-08-17T18:18:49.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1182999999999999e-05 2024-08-17T18:18:49.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.584432+0000", 2024-08-17T18:18:49.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.009211565 2024-08-17T18:18:49.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:49.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.584575+0000", 2024-08-17T18:18:49.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014335199999999999 2024-08-17T18:18:49.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T18:18:49.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.584924+0000", 2024-08-17T18:18:49.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034901399999999998 2024-08-17T18:18:49.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.589159+0000", 2024-08-17T18:18:49.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00423516 2024-08-17T18:18:49.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.589205+0000", 2024-08-17T18:18:49.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5757999999999998e-05 2024-08-17T18:18:49.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:49.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.589215+0000", 2024-08-17T18:18:49.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0037000000000001e-05 2024-08-17T18:18:49.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:49.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.589287+0000", 2024-08-17T18:18:49.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2329000000000005e-05 2024-08-17T18:18:49.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:49.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.589333+0000", 2024-08-17T18:18:49.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5303000000000003e-05 2024-08-17T18:18:49.835 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.836 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:49.836 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.836 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:160913 57.16 57:6c3e772a:::benchmark_data_smithi136_389235_object44991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1660)", 2024-08-17T18:18:49.836 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:56.598041+0000", 2024-08-17T18:18:49.836 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 53.18553429, 2024-08-17T18:18:49.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99375612099999999, 2024-08-17T18:18:49.836 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:49.836 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:49.837 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:49.837 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:49.837 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:49.837 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 160913 2024-08-17T18:18:49.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.837 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:49.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:49.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598041+0000", 2024-08-17T18:18:49.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:49.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598041+0000", 2024-08-17T18:18:49.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:49.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598040+0000", 2024-08-17T18:18:49.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:18:49.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:49.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598044+0000", 2024-08-17T18:18:49.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.685e-06 2024-08-17T18:18:49.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:49.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598055+0000", 2024-08-17T18:18:49.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.043e-05 2024-08-17T18:18:49.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:49.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598061+0000", 2024-08-17T18:18:49.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4269999999999996e-06 2024-08-17T18:18:49.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598122+0000", 2024-08-17T18:18:49.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.101e-05 2024-08-17T18:18:49.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:18:49.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598131+0000", 2024-08-17T18:18:49.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9679999999999995e-06 2024-08-17T18:18:49.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.587199+0000", 2024-08-17T18:18:49.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98906787600000001 2024-08-17T18:18:49.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:49.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.587308+0000", 2024-08-17T18:18:49.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108712 2024-08-17T18:18:49.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T18:18:49.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.587567+0000", 2024-08-17T18:18:49.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025900400000000002 2024-08-17T18:18:49.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.589363+0000", 2024-08-17T18:18:49.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017963650000000001 2024-08-17T18:18:49.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.591651+0000", 2024-08-17T18:18:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022876120000000001 2024-08-17T18:18:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.591719+0000", 2024-08-17T18:18:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8367000000000005e-05 2024-08-17T18:18:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.591759+0000", 2024-08-17T18:18:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9480999999999997e-05 2024-08-17T18:18:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:49.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.591797+0000", 2024-08-17T18:18:49.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7957999999999998e-05 2024-08-17T18:18:49.844 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.844 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:49.844 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.844 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:160914 57.16 57:681070b7:::benchmark_data_smithi136_389235_object44992:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1660)", 2024-08-17T18:18:49.844 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:56.598883+0000", 2024-08-17T18:18:49.844 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 53.184691571999998, 2024-08-17T18:18:49.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0028806669999999, 2024-08-17T18:18:49.844 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:49.845 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:49.845 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:49.845 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:49.845 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:49.845 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 160914 2024-08-17T18:18:49.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.845 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:49.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:49.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598883+0000", 2024-08-17T18:18:49.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:49.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598883+0000", 2024-08-17T18:18:49.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:49.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598882+0000", 2024-08-17T18:18:49.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:18:49.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:49.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598888+0000", 2024-08-17T18:18:49.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9750000000000004e-06 2024-08-17T18:18:49.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:49.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598903+0000", 2024-08-17T18:18:49.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4534e-05 2024-08-17T18:18:49.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:49.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598910+0000", 2024-08-17T18:18:49.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0979999999999996e-06 2024-08-17T18:18:49.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598952+0000", 2024-08-17T18:18:49.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2006999999999999e-05 2024-08-17T18:18:49.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:18:49.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598960+0000", 2024-08-17T18:18:49.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3899999999999993e-06 2024-08-17T18:18:49.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.589545+0000", 2024-08-17T18:18:49.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.990585037 2024-08-17T18:18:49.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:49.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.589681+0000", 2024-08-17T18:18:49.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000136379 2024-08-17T18:18:49.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T18:18:49.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.590020+0000", 2024-08-17T18:18:49.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033900400000000001 2024-08-17T18:18:49.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.591710+0000", 2024-08-17T18:18:49.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016892420000000001 2024-08-17T18:18:49.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.597521+0000", 2024-08-17T18:18:49.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058114159999999998 2024-08-17T18:18:49.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:49.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.601595+0000", 2024-08-17T18:18:49.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040736549999999998 2024-08-17T18:18:49.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:49.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.601719+0000", 2024-08-17T18:18:49.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012428 2024-08-17T18:18:49.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:49.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.601764+0000", 2024-08-17T18:18:49.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4897000000000001e-05 2024-08-17T18:18:49.851 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.851 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:49.851 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.851 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:160956 57.16 57:69ce9ed4:::benchmark_data_smithi136_389235_object45034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1660)", 2024-08-17T18:18:49.851 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:56.725647+0000", 2024-08-17T18:18:49.851 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 53.057927562000003, 2024-08-17T18:18:49.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.88076719599999997, 2024-08-17T18:18:49.852 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:49.852 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:49.852 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:49.852 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:49.852 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:49.852 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 160956 2024-08-17T18:18:49.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.852 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:49.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:49.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.725647+0000", 2024-08-17T18:18:49.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:49.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.725647+0000", 2024-08-17T18:18:49.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:49.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.725646+0000", 2024-08-17T18:18:49.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:18:49.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:49.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.725652+0000", 2024-08-17T18:18:49.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2759999999999997e-06 2024-08-17T18:18:49.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:49.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.725664+0000", 2024-08-17T18:18:49.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1616e-05 2024-08-17T18:18:49.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:49.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.725670+0000", 2024-08-17T18:18:49.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4359999999999999e-06 2024-08-17T18:18:49.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.725720+0000", 2024-08-17T18:18:49.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9599999999999999e-05 2024-08-17T18:18:49.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:18:49.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.725727+0000", 2024-08-17T18:18:49.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1799999999999999e-06 2024-08-17T18:18:49.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.591831+0000", 2024-08-17T18:18:49.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.86610343700000003 2024-08-17T18:18:49.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:49.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.591902+0000", 2024-08-17T18:18:49.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1347999999999995e-05 2024-08-17T18:18:49.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T18:18:49.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.592142+0000", 2024-08-17T18:18:49.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024018299999999999 2024-08-17T18:18:49.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.597454+0000", 2024-08-17T18:18:49.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0053123390000000001 2024-08-17T18:18:49.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.597555+0000", 2024-08-17T18:18:49.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000100389 2024-08-17T18:18:49.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:49.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606321+0000", 2024-08-17T18:18:49.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0087664409999999998 2024-08-17T18:18:49.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:49.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606372+0000", 2024-08-17T18:18:49.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0843000000000001e-05 2024-08-17T18:18:49.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:49.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606414+0000", 2024-08-17T18:18:49.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2333999999999998e-05 2024-08-17T18:18:49.859 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.859 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:49.862 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.862 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:161012 57.16 57:6e84d2aa:::benchmark_data_smithi136_389235_object45090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1660)", 2024-08-17T18:18:49.862 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:56.934535+0000", 2024-08-17T18:18:49.862 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.849039937000001, 2024-08-17T18:18:49.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.68200467600000003, 2024-08-17T18:18:49.862 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:49.862 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:49.863 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:49.863 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:49.863 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:49.863 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161012 2024-08-17T18:18:49.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.863 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:49.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:49.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.934535+0000", 2024-08-17T18:18:49.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:49.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.934535+0000", 2024-08-17T18:18:49.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:49.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.934533+0000", 2024-08-17T18:18:49.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:18:49.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:49.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.934540+0000", 2024-08-17T18:18:49.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2890000000000003e-06 2024-08-17T18:18:49.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:49.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.934551+0000", 2024-08-17T18:18:49.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1762e-05 2024-08-17T18:18:49.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:49.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.934557+0000", 2024-08-17T18:18:49.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4399999999999996e-06 2024-08-17T18:18:49.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.934601+0000", 2024-08-17T18:18:49.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4533000000000002e-05 2024-08-17T18:18:49.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:18:49.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.934610+0000", 2024-08-17T18:18:49.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2199999999999992e-06 2024-08-17T18:18:49.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.594900+0000", 2024-08-17T18:18:49.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.66029078600000002 2024-08-17T18:18:49.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:49.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.595020+0000", 2024-08-17T18:18:49.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119622 2024-08-17T18:18:49.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T18:18:49.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.595434+0000", 2024-08-17T18:18:49.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000413805 2024-08-17T18:18:49.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:49.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606426+0000", 2024-08-17T18:18:49.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010991772 2024-08-17T18:18:49.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606470+0000", 2024-08-17T18:18:49.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4162999999999997e-05 2024-08-17T18:18:49.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616445+0000", 2024-08-17T18:18:49.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0099751089999999994 2024-08-17T18:18:49.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:49.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616496+0000", 2024-08-17T18:18:49.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1069000000000002e-05 2024-08-17T18:18:49.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:49.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616540+0000", 2024-08-17T18:18:49.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3649000000000002e-05 2024-08-17T18:18:49.871 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.871 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:49.871 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.871 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:161032 57.16 57:6919a724:::benchmark_data_smithi136_389235_object45110:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1660)", 2024-08-17T18:18:49.872 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:57.066208+0000", 2024-08-17T18:18:49.872 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.717366667999997, 2024-08-17T18:18:49.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.55043567800000004, 2024-08-17T18:18:49.872 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:49.872 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:49.872 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:49.872 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:49.872 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:49.872 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161032 2024-08-17T18:18:49.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.872 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:49.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:49.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.066208+0000", 2024-08-17T18:18:49.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:49.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.066208+0000", 2024-08-17T18:18:49.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:49.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.066206+0000", 2024-08-17T18:18:49.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:18:49.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:49.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.066216+0000", 2024-08-17T18:18:49.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9129999999999998e-06 2024-08-17T18:18:49.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:49.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.066235+0000", 2024-08-17T18:18:49.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8524000000000001e-05 2024-08-17T18:18:49.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:49.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.066244+0000", 2024-08-17T18:18:49.875 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:49.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9039999999999998e-06 2024-08-17T18:18:49.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.066297+0000", 2024-08-17T18:18:49.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3087999999999997e-05 2024-08-17T18:18:49.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:18:49.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.066306+0000", 2024-08-17T18:18:49.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2140000000000002e-06 2024-08-17T18:18:49.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.597579+0000", 2024-08-17T18:18:49.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.53127298999999994 2024-08-17T18:18:49.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:49.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.597711+0000", 2024-08-17T18:18:49.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013171400000000001 2024-08-17T18:18:49.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T18:18:49.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.598077+0000", 2024-08-17T18:18:49.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036651799999999998 2024-08-17T18:18:49.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:49.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606431+0000", 2024-08-17T18:18:49.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0083538680000000004 2024-08-17T18:18:49.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606528+0000", 2024-08-17T18:18:49.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7275999999999995e-05 2024-08-17T18:18:49.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616574+0000", 2024-08-17T18:18:49.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010045554 2024-08-17T18:18:49.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:49.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616610+0000", 2024-08-17T18:18:49.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6565e-05 2024-08-17T18:18:49.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:49.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616644+0000", 2024-08-17T18:18:49.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3522999999999997e-05 2024-08-17T18:18:49.880 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.880 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:49.880 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.881 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:161078 57.16 57:6e3b9c80:::benchmark_data_smithi136_389235_object45156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1660)", 2024-08-17T18:18:49.881 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:57.238646+0000", 2024-08-17T18:18:49.881 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.544928441000003, 2024-08-17T18:18:49.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.37810427000000002, 2024-08-17T18:18:49.881 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:49.881 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:49.881 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:49.881 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:49.881 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:49.881 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161078 2024-08-17T18:18:49.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.881 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:49.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:49.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.238646+0000", 2024-08-17T18:18:49.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:49.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.238646+0000", 2024-08-17T18:18:49.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:49.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.238645+0000", 2024-08-17T18:18:49.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:18:49.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:49.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.238652+0000", 2024-08-17T18:18:49.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8349999999999998e-06 2024-08-17T18:18:49.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:49.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.238666+0000", 2024-08-17T18:18:49.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4346e-05 2024-08-17T18:18:49.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:49.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.238674+0000", 2024-08-17T18:18:49.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9889999999999992e-06 2024-08-17T18:18:49.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.239725+0000", 2024-08-17T18:18:49.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001050743 2024-08-17T18:18:49.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:18:49.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.239737+0000", 2024-08-17T18:18:49.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2510000000000001e-05 2024-08-17T18:18:49.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.599446+0000", 2024-08-17T18:18:49.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.359708378 2024-08-17T18:18:49.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:49.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.599580+0000", 2024-08-17T18:18:49.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000134485 2024-08-17T18:18:49.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T18:18:49.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.599828+0000", 2024-08-17T18:18:49.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024764999999999998 2024-08-17T18:18:49.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:49.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606479+0000", 2024-08-17T18:18:49.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066513500000000003 2024-08-17T18:18:49.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606563+0000", 2024-08-17T18:18:49.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3973000000000004e-05 2024-08-17T18:18:49.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616677+0000", 2024-08-17T18:18:49.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010113924999999999 2024-08-17T18:18:49.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:49.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616718+0000", 2024-08-17T18:18:49.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0836e-05 2024-08-17T18:18:49.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:49.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616751+0000", 2024-08-17T18:18:49.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2685999999999998e-05 2024-08-17T18:18:49.888 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.888 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:49.888 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.888 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:161118 57.16 57:69cf4939:::benchmark_data_smithi136_389235_object45196:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1660)", 2024-08-17T18:18:49.888 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:57.400404+0000", 2024-08-17T18:18:49.888 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.383170419999999, 2024-08-17T18:18:49.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.216441996, 2024-08-17T18:18:49.888 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:49.889 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:49.889 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:49.889 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:49.889 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:49.889 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161118 2024-08-17T18:18:49.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.889 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:49.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:49.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.400404+0000", 2024-08-17T18:18:49.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:49.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.400404+0000", 2024-08-17T18:18:49.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:49.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.400403+0000", 2024-08-17T18:18:49.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:18:49.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:49.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.400410+0000", 2024-08-17T18:18:49.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8349999999999998e-06 2024-08-17T18:18:49.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:49.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.400423+0000", 2024-08-17T18:18:49.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3344e-05 2024-08-17T18:18:49.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:49.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.400430+0000", 2024-08-17T18:18:49.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6279999999999999e-06 2024-08-17T18:18:49.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.400473+0000", 2024-08-17T18:18:49.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3040000000000001e-05 2024-08-17T18:18:49.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:18:49.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.400481+0000", 2024-08-17T18:18:49.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3899999999999993e-06 2024-08-17T18:18:49.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.601810+0000", 2024-08-17T18:18:49.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.201329281 2024-08-17T18:18:49.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:49.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.601943+0000", 2024-08-17T18:18:49.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013258000000000001 2024-08-17T18:18:49.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T18:18:49.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.602306+0000", 2024-08-17T18:18:49.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000363469 2024-08-17T18:18:49.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:49.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606484+0000", 2024-08-17T18:18:49.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041776690000000002 2024-08-17T18:18:49.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606599+0000", 2024-08-17T18:18:49.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115176 2024-08-17T18:18:49.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616777+0000", 2024-08-17T18:18:49.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010177598 2024-08-17T18:18:49.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:49.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616816+0000", 2024-08-17T18:18:49.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9555999999999999e-05 2024-08-17T18:18:49.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:49.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616846+0000", 2024-08-17T18:18:49.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9924e-05 2024-08-17T18:18:49.896 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.896 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:49.896 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.896 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:164936 57.a 57:5d3ff99b:::benchmark_data_smithi136_389235_object49014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1661)", 2024-08-17T18:18:49.897 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:06.210033+0000", 2024-08-17T18:18:49.897 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.573542250000003, 2024-08-17T18:18:49.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27177604100000002, 2024-08-17T18:18:49.897 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:49.897 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:49.897 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:49.897 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:49.897 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:49.897 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164936 2024-08-17T18:18:49.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.898 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:49.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:49.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210033+0000", 2024-08-17T18:18:49.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:49.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210033+0000", 2024-08-17T18:18:49.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:49.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210031+0000", 2024-08-17T18:18:49.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:18:49.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:49.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210039+0000", 2024-08-17T18:18:49.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9470000000000003e-06 2024-08-17T18:18:49.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:49.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210051+0000", 2024-08-17T18:18:49.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2771e-05 2024-08-17T18:18:49.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:49.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210058+0000", 2024-08-17T18:18:49.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8109999999999997e-06 2024-08-17T18:18:49.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210117+0000", 2024-08-17T18:18:49.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9086000000000003e-05 2024-08-17T18:18:49.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:49.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210260+0000", 2024-08-17T18:18:49.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014312299999999999 2024-08-17T18:18:49.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4,7", 2024-08-17T18:18:49.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210404+0000", 2024-08-17T18:18:49.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014415700000000001 2024-08-17T18:18:49.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.213570+0000", 2024-08-17T18:18:49.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031656869999999999 2024-08-17T18:18:49.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.213627+0000", 2024-08-17T18:18:49.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6935999999999998e-05 2024-08-17T18:18:49.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:49.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.213639+0000", 2024-08-17T18:18:49.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2143e-05 2024-08-17T18:18:49.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.481648+0000", 2024-08-17T18:18:49.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.26800850399999998 2024-08-17T18:18:49.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:49.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.481700+0000", 2024-08-17T18:18:49.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2073000000000001e-05 2024-08-17T18:18:49.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:49.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.481809+0000", 2024-08-17T18:18:49.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108818 2024-08-17T18:18:49.905 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.905 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:49.905 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.905 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:164940 57.18 57:1d27d23b:::benchmark_data_smithi136_389235_object49018:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1661)", 2024-08-17T18:18:49.905 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:06.217411+0000", 2024-08-17T18:18:49.905 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.566164112000003, 2024-08-17T18:18:49.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.19810450199999999, 2024-08-17T18:18:49.905 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:49.905 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:49.906 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:49.906 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:49.906 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:49.906 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164940 2024-08-17T18:18:49.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.906 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:49.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:49.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.217411+0000", 2024-08-17T18:18:49.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:49.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.217411+0000", 2024-08-17T18:18:49.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:49.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.217409+0000", 2024-08-17T18:18:49.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:18:49.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:49.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.217416+0000", 2024-08-17T18:18:49.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7569999999999998e-06 2024-08-17T18:18:49.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:49.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.217426+0000", 2024-08-17T18:18:49.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0101999999999999e-05 2024-08-17T18:18:49.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:49.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.217432+0000", 2024-08-17T18:18:49.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5560000000000004e-06 2024-08-17T18:18:49.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.228404+0000", 2024-08-17T18:18:49.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01097158 2024-08-17T18:18:49.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:18:49.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.229358+0000", 2024-08-17T18:18:49.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00095366600000000004 2024-08-17T18:18:49.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.399727+0000", 2024-08-17T18:18:49.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.170368836 2024-08-17T18:18:49.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:49.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.399823+0000", 2024-08-17T18:18:49.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6718999999999993e-05 2024-08-17T18:18:49.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,6,7", 2024-08-17T18:18:49.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.400264+0000", 2024-08-17T18:18:49.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00044098899999999998 2024-08-17T18:18:49.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.412476+0000", 2024-08-17T18:18:49.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012212097999999999 2024-08-17T18:18:49.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.412511+0000", 2024-08-17T18:18:49.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4360999999999998e-05 2024-08-17T18:18:49.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.412579+0000", 2024-08-17T18:18:49.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8625000000000001e-05 2024-08-17T18:18:49.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:49.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.414150+0000", 2024-08-17T18:18:49.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001570388 2024-08-17T18:18:49.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:49.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.414233+0000", 2024-08-17T18:18:49.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3357999999999997e-05 2024-08-17T18:18:49.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:49.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.415515+0000", 2024-08-17T18:18:49.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012821760000000001 2024-08-17T18:18:49.912 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.912 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:49.912 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.913 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:164941 57.5 57:a13b1393:::benchmark_data_smithi136_389235_object49019:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1661)", 2024-08-17T18:18:49.913 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:06.220898+0000", 2024-08-17T18:18:49.913 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.56267725, 2024-08-17T18:18:49.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18599918600000001, 2024-08-17T18:18:49.913 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:49.913 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:49.913 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:49.913 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:49.913 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:49.913 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164941 2024-08-17T18:18:49.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.913 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:49.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:49.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.220898+0000", 2024-08-17T18:18:49.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:49.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.220898+0000", 2024-08-17T18:18:49.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:49.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.220896+0000", 2024-08-17T18:18:49.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:18:49.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:49.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.220907+0000", 2024-08-17T18:18:49.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1054e-05 2024-08-17T18:18:49.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:49.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.220921+0000", 2024-08-17T18:18:49.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4337e-05 2024-08-17T18:18:49.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:49.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.220929+0000", 2024-08-17T18:18:49.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8329999999999994e-06 2024-08-17T18:18:49.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.916 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.224750+0000", 2024-08-17T18:18:49.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038211059999999999 2024-08-17T18:18:49.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:18:49.916 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.228033+0000", 2024-08-17T18:18:49.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032827220000000001 2024-08-17T18:18:49.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.398166+0000", 2024-08-17T18:18:49.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.170132799 2024-08-17T18:18:49.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:49.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.398276+0000", 2024-08-17T18:18:49.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000110061 2024-08-17T18:18:49.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,6,7", 2024-08-17T18:18:49.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.398712+0000", 2024-08-17T18:18:49.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000436301 2024-08-17T18:18:49.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:49.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.402790+0000", 2024-08-17T18:18:49.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040774060000000004 2024-08-17T18:18:49.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.405041+0000", 2024-08-17T18:18:49.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022516279999999999 2024-08-17T18:18:49.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.405083+0000", 2024-08-17T18:18:49.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1831e-05 2024-08-17T18:18:49.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.405121+0000", 2024-08-17T18:18:49.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8309000000000002e-05 2024-08-17T18:18:49.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:49.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.405230+0000", 2024-08-17T18:18:49.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108514 2024-08-17T18:18:49.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:49.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.406897+0000", 2024-08-17T18:18:49.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001666837 2024-08-17T18:18:49.920 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.920 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:49.920 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.920 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:164944 57.18 57:1be526b4:::benchmark_data_smithi136_389235_object49022:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1661)", 2024-08-17T18:18:49.921 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:06.229268+0000", 2024-08-17T18:18:49.921 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.554306900999997, 2024-08-17T18:18:49.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.19321551200000001, 2024-08-17T18:18:49.921 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:49.921 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:49.921 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:49.921 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:49.921 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:49.921 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164944 2024-08-17T18:18:49.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.921 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:49.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:49.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.229268+0000", 2024-08-17T18:18:49.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:49.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.229268+0000", 2024-08-17T18:18:49.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:49.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.229266+0000", 2024-08-17T18:18:49.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:18:49.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:49.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.229273+0000", 2024-08-17T18:18:49.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5220000000000004e-06 2024-08-17T18:18:49.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:49.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.229284+0000", 2024-08-17T18:18:49.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1751e-05 2024-08-17T18:18:49.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:49.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.229292+0000", 2024-08-17T18:18:49.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2910000000000002e-06 2024-08-17T18:18:49.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.230181+0000", 2024-08-17T18:18:49.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00088951799999999995 2024-08-17T18:18:49.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:18:49.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.230817+0000", 2024-08-17T18:18:49.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00063602099999999996 2024-08-17T18:18:49.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.408788+0000", 2024-08-17T18:18:49.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.17797079499999999 2024-08-17T18:18:49.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:49.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.408942+0000", 2024-08-17T18:18:49.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000153614 2024-08-17T18:18:49.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,6,7", 2024-08-17T18:18:49.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.409412+0000", 2024-08-17T18:18:49.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00047059700000000001 2024-08-17T18:18:49.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.412544+0000", 2024-08-17T18:18:49.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031313040000000001 2024-08-17T18:18:49.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.414141+0000", 2024-08-17T18:18:49.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015969179999999999 2024-08-17T18:18:49.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:49.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.415527+0000", 2024-08-17T18:18:49.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00138671 2024-08-17T18:18:49.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.420528+0000", 2024-08-17T18:18:49.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050005960000000004 2024-08-17T18:18:49.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:49.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.420586+0000", 2024-08-17T18:18:49.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7920000000000001e-05 2024-08-17T18:18:49.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:49.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.422483+0000", 2024-08-17T18:18:49.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001897686 2024-08-17T18:18:49.929 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.929 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:49.929 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.930 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:164945 57.a 57:55ee62ab:::benchmark_data_smithi136_389235_object49023:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1661)", 2024-08-17T18:18:49.930 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:06.230920+0000", 2024-08-17T18:18:49.930 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.552654341, 2024-08-17T18:18:49.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25469593200000001, 2024-08-17T18:18:49.930 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:49.930 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:49.930 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:49.930 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:49.930 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:49.931 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164945 2024-08-17T18:18:49.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.931 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:49.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:49.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.230920+0000", 2024-08-17T18:18:49.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:49.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.230920+0000", 2024-08-17T18:18:49.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:49.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.230916+0000", 2024-08-17T18:18:49.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999957 2024-08-17T18:18:49.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:49.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.230926+0000", 2024-08-17T18:18:49.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4390000000000006e-06 2024-08-17T18:18:49.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:49.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.230938+0000", 2024-08-17T18:18:49.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2094e-05 2024-08-17T18:18:49.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:49.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.396017+0000", 2024-08-17T18:18:49.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16507881099999999 2024-08-17T18:18:49.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.396075+0000", 2024-08-17T18:18:49.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8207e-05 2024-08-17T18:18:49.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:49.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.396236+0000", 2024-08-17T18:18:49.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016120800000000001 2024-08-17T18:18:49.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4,7", 2024-08-17T18:18:49.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.396397+0000", 2024-08-17T18:18:49.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000160571 2024-08-17T18:18:49.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.398644+0000", 2024-08-17T18:18:49.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022471930000000002 2024-08-17T18:18:49.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:49.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.400494+0000", 2024-08-17T18:18:49.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018502049999999999 2024-08-17T18:18:49.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.404981+0000", 2024-08-17T18:18:49.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044866910000000001 2024-08-17T18:18:49.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.485449+0000", 2024-08-17T18:18:49.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.080468711999999998 2024-08-17T18:18:49.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:49.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.485506+0000", 2024-08-17T18:18:49.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6317999999999998e-05 2024-08-17T18:18:49.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:49.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.485616+0000", 2024-08-17T18:18:49.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000110649 2024-08-17T18:18:49.937 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.937 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:49.938 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.938 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:166033 57.a 57:54afa2af:::benchmark_data_smithi136_389235_object50111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1661)", 2024-08-17T18:18:49.938 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:08.728444+0000", 2024-08-17T18:18:49.938 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 41.055130761999997, 2024-08-17T18:18:49.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.189001066, 2024-08-17T18:18:49.938 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:49.938 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:49.938 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:49.938 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:49.938 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:49.938 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 166033 2024-08-17T18:18:49.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.939 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:49.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:49.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:08.728444+0000", 2024-08-17T18:18:49.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:49.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:08.728444+0000", 2024-08-17T18:18:49.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:49.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:08.728442+0000", 2024-08-17T18:18:49.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999976 2024-08-17T18:18:49.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.940 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:49 smithi165 ceph-mon[105517]: pgmap v3212: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 1 active+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 366 op/s; 31861/156414 objects degraded (20.370%); 40533/156414 objects misplaced (25.914%); 15 MiB/s, 239 objects/s recovering 2024-08-17T18:18:49.940 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:49 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3615719511' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:18:49.940 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:49 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:18:49.940 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:49 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 31861/156414 objects degraded (20.370%), 16 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:49.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:49.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:08.728451+0000", 2024-08-17T18:18:49.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5559999999999996e-06 2024-08-17T18:18:49.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:49.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:08.767850+0000", 2024-08-17T18:18:49.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.039398786999999998 2024-08-17T18:18:49.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:49.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:08.767867+0000", 2024-08-17T18:18:49.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6443000000000002e-05 2024-08-17T18:18:49.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:08.769629+0000", 2024-08-17T18:18:49.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001762596 2024-08-17T18:18:49.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:49.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:08.769728+0000", 2024-08-17T18:18:49.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9167999999999996e-05 2024-08-17T18:18:49.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4,7", 2024-08-17T18:18:49.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:08.769828+0000", 2024-08-17T18:18:49.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9129000000000006e-05 2024-08-17T18:18:49.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:08.773046+0000", 2024-08-17T18:18:49.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032188970000000001 2024-08-17T18:18:49.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:08.778690+0000", 2024-08-17T18:18:49.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056440830000000003 2024-08-17T18:18:49.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:49.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:08.778739+0000", 2024-08-17T18:18:49.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8145000000000001e-05 2024-08-17T18:18:49.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:08.917335+0000", 2024-08-17T18:18:49.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13859656300000001 2024-08-17T18:18:49.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:49.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:08.917373+0000", 2024-08-17T18:18:49.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7984000000000001e-05 2024-08-17T18:18:49.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:49.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:08.917445+0000", 2024-08-17T18:18:49.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1957000000000002e-05 2024-08-17T18:18:49.946 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.946 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:49.946 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.946 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:176928 57.5 57:a4d794c4:::benchmark_data_smithi136_389235_object61006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1672)", 2024-08-17T18:18:49.946 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:39.097614+0000", 2024-08-17T18:18:49.946 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 10.685961089999999, 2024-08-17T18:18:49.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.183983539, 2024-08-17T18:18:49.946 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:49.946 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:49.946 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:49.946 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:49.947 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:49.947 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 176928 2024-08-17T18:18:49.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.947 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:49.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:49.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.097614+0000", 2024-08-17T18:18:49.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:49.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.097614+0000", 2024-08-17T18:18:49.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:49.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.097612+0000", 2024-08-17T18:18:49.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:18:49.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:49.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.097620+0000", 2024-08-17T18:18:49.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1340000000000004e-06 2024-08-17T18:18:49.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:49.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.249868+0000", 2024-08-17T18:18:49.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.152248511 2024-08-17T18:18:49.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:49.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.249883+0000", 2024-08-17T18:18:49.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4392e-05 2024-08-17T18:18:49.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.252923+0000", 2024-08-17T18:18:49.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030403299999999999 2024-08-17T18:18:49.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:18:49.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.253547+0000", 2024-08-17T18:18:49.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062345800000000002 2024-08-17T18:18:49.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.259258+0000", 2024-08-17T18:18:49.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057116679999999996 2024-08-17T18:18:49.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:49.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.259363+0000", 2024-08-17T18:18:49.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104559 2024-08-17T18:18:49.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3,6,7", 2024-08-17T18:18:49.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.259782+0000", 2024-08-17T18:18:49.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00041873399999999998 2024-08-17T18:18:49.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.269220+0000", 2024-08-17T18:18:49.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0094387189999999999 2024-08-17T18:18:49.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.276142+0000", 2024-08-17T18:18:49.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069217489999999996 2024-08-17T18:18:49.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.276179+0000", 2024-08-17T18:18:49.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7426999999999999e-05 2024-08-17T18:18:49.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.276245+0000", 2024-08-17T18:18:49.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5784999999999994e-05 2024-08-17T18:18:49.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:49.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.277368+0000", 2024-08-17T18:18:49.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001122669 2024-08-17T18:18:49.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:49.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.277515+0000", 2024-08-17T18:18:49.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000146651 2024-08-17T18:18:49.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:49.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.281597+0000", 2024-08-17T18:18:49.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040827729999999996 2024-08-17T18:18:49.955 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.955 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:49.955 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.956 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:179471 57.a 57:5cdb5d10:::benchmark_data_smithi136_389235_object63549:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1672)", 2024-08-17T18:18:49.956 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:45.818546+0000", 2024-08-17T18:18:49.956 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 3.9650291580000001, 2024-08-17T18:18:49.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.204180628, 2024-08-17T18:18:49.956 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:18:49.956 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:18:49.956 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:18:49.956 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:18:49.956 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:18:49.956 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 179471 2024-08-17T18:18:49.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.956 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:18:49.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:18:49.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.818546+0000", 2024-08-17T18:18:49.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:18:49.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.818546+0000", 2024-08-17T18:18:49.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:18:49.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:18:49.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.818544+0000", 2024-08-17T18:18:49.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:18:49.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:18:49.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.818551+0000", 2024-08-17T18:18:49.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5699999999999998e-06 2024-08-17T18:18:49.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:18:49.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.818565+0000", 2024-08-17T18:18:49.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4055e-05 2024-08-17T18:18:49.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:18:49.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.818573+0000", 2024-08-17T18:18:49.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7000000000000008e-06 2024-08-17T18:18:49.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:18:49.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.818619+0000", 2024-08-17T18:18:49.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6653999999999998e-05 2024-08-17T18:18:49.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:18:49.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.818760+0000", 2024-08-17T18:18:49.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014030299999999999 2024-08-17T18:18:49.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,3,4,7", 2024-08-17T18:18:49.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.818912+0000", 2024-08-17T18:18:49.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015206599999999999 2024-08-17T18:18:49.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.821171+0000", 2024-08-17T18:18:49.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022595610000000002 2024-08-17T18:18:49.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.821301+0000", 2024-08-17T18:18:49.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012987099999999999 2024-08-17T18:18:49.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.821336+0000", 2024-08-17T18:18:49.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4663000000000003e-05 2024-08-17T18:18:49.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:18:49.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.823126+0000", 2024-08-17T18:18:49.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017902630000000001 2024-08-17T18:18:49.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:18:49.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:46.022580+0000", 2024-08-17T18:18:49.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.19945381100000001 2024-08-17T18:18:49.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:18:49.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:46.022640+0000", 2024-08-17T18:18:49.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0534e-05 2024-08-17T18:18:49.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:18:49.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:18:49.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:18:49.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:46.022726+0000", 2024-08-17T18:18:49.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5964999999999994e-05 2024-08-17T18:18:49.963 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.963 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:49.964 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.964 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:18:49.964 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:18:49.964 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:18:49.966 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:18:49.967 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:49 smithi136 ceph-mon[109861]: pgmap v3212: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 1 active+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 366 op/s; 31861/156414 objects degraded (20.370%); 40533/156414 objects misplaced (25.914%); 15 MiB/s, 239 objects/s recovering 2024-08-17T18:18:49.967 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:49 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3615719511' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:18:49.967 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:49 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:18:49.967 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:49 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 31861/156414 objects degraded (20.370%), 16 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:49.968 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:49 smithi136 ceph-mon[115078]: pgmap v3212: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 1 active+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 366 op/s; 31861/156414 objects degraded (20.370%); 40533/156414 objects misplaced (25.914%); 15 MiB/s, 239 objects/s recovering 2024-08-17T18:18:49.968 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:49 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3615719511' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:18:49.968 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:49 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:18:49.968 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:49 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 31861/156414 objects degraded (20.370%), 16 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:49.972 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T18:18:49.975 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:49.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:49.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:50.077 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:50.124 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:50.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:50.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:50.226 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:50.259 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:50.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:50.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:50.361 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:50.387 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:50.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:50.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:50.489 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:50.524 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:50.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:50.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:50.625 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:50.665 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:50.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:50.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:50.767 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:50.802 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:50.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:50.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:50.822 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":3213,"stamp":"2024-08-17T18:18:49.767154+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3388980040,"num_objects":51728,"num_object_clones":0,"num_object_copies":155184,"num_objects_missing_on_primary":831,"num_objects_missing":831,"num_objects_degraded":31739,"num_objects_misplaced":40187,"num_objects_unfound":0,"num_objects_dirty":51728,"num_whiteouts":0,"num_read":70,"num_read_kb":150,"num_write":180241,"num_write_kb":7419289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30837,"num_bytes_recovered":1847627816,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136091,"ondisk_log_size":136091,"up":327,"acting":320,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":422,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":19542316,"kb_used_data":12788468,"kb_used_omap":285,"kb_used_meta":6753506,"kb_avail":636673748,"statfs":{"total":671965249536,"available":651953917952,"internally_reserved":0,"allocated":13095391232,"data_stored":13064872468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":292552,"internal_metadata":6915590456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,2,5,1,2,0,2,0,3,2,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-264372224,"num_objects":-4034,"num_object_clones":0,"num_object_copies":-12102,"num_objects_missing_on_primary":-184,"num_objects_missing":-184,"num_objects_degraded":-1625,"num_objects_misplaced":-3654,"num_objects_unfound":0,"num_objects_dirty":-4034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4034,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2153,"num_bytes_recovered":141099008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.293233"},"pg_stats":[{"pgid":"57.19","version":"1672'8108","reported_seq":8201,"reported_epoch":1672,"state":"active+recovery_wait+remapped","last_fresh":"2024-08-17T18:18:49.506282+0000","last_change":"2024-08-17T18:18:30.832671+0000","last_active":"2024-08-17T18:18:49.506282+0000","last_peered":"2024-08-17T18:18:49.506282+0000","last_clean":"2024-08-17T18:18:29.597061+0000","last_became_active":"2024-08-17T18:18:30.810367+0000","last_became_peered":"2024-08-17T18:18:30.810367+0000","last_unstale":"2024-08-17T18:18:49.506282+0000","last_undegraded":"2024-08-17T18:18:49.506282+0000","last_fullsized":"2024-08-17T18:18:49.506282+0000","mapping_epoch":1667,"log_start":"1613'2676","ondisk_log_start":"1613'2676","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5432,"log_dups_size":0,"ondisk_log_size":5432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102858752,"num_objects":1570,"num_object_clones":0,"num_object_copies":4710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1570,"num_objects_unfound":0,"num_objects_dirty":1570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5630,"num_write_kb":230368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[4,5,6],"avail_no_missing":["4","5","6"],"object_location_counts":[{"shards":"4,5,6","objects":1570}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"57.18","version":"1672'8204","reported_seq":11968,"reported_epoch":1672,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:18:47.070147+0000","last_change":"2024-08-17T18:15:54.579791+0000","last_active":"2024-08-17T18:18:47.070147+0000","last_peered":"2024-08-17T18:18:47.070147+0000","last_clean":"2024-08-17T18:15:09.641861+0000","last_became_active":"2024-08-17T18:15:54.569392+0000","last_became_peered":"2024-08-17T18:15:54.569392+0000","last_unstale":"2024-08-17T18:18:47.070147+0000","last_undegraded":"2024-08-17T18:15:54.395452+0000","last_fullsized":"2024-08-17T18:18:47.070147+0000","mapping_epoch":1638,"log_start":"1613'3775","ondisk_log_start":"1613'3775","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4429,"log_dups_size":0,"ondisk_log_size":4429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105283584,"num_objects":1607,"num_object_clones":0,"num_object_copies":4821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1397,"num_objects_misplaced":210,"num_objects_unfound":0,"num_objects_dirty":1607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5697,"num_write_kb":233696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1830,"num_bytes_recovered":119930880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[5,6,4],"avail_no_missing":["5","0","6"],"object_location_counts":[{"shards":"0,5,6","objects":1182},{"shards":"4,5,6","objects":425}],"blocked_by":[],"up_primary":7,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.17","version":"1672'7889","reported_seq":9976,"reported_epoch":1672,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:18:47.333929+0000","last_change":"2024-08-17T18:18:29.904207+0000","last_active":"2024-08-17T18:18:47.333929+0000","last_peered":"2024-08-17T18:18:47.333929+0000","last_clean":"2024-08-17T18:15:52.210567+0000","last_became_active":"2024-08-17T18:17:23.718959+0000","last_became_peered":"2024-08-17T18:17:23.718959+0000","last_unstale":"2024-08-17T18:18:47.333929+0000","last_undegraded":"2024-08-17T18:18:29.902629+0000","last_fullsized":"2024-08-17T18:17:23.630677+0000","mapping_epoch":1653,"log_start":"1613'2599","ondisk_log_start":"1613'2599","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5290,"log_dups_size":0,"ondisk_log_size":5290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104169472,"num_objects":1590,"num_object_clones":0,"num_object_copies":4770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3360,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5458,"num_write_kb":225504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":951,"num_bytes_recovered":62324736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,4,6","objects":1572},{"shards":"2,7","objects":18}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"57.16","version":"1672'7952","reported_seq":15173,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:47.001385+0000","last_change":"2024-08-17T18:17:57.587085+0000","last_active":"2024-08-17T18:18:47.001385+0000","last_peered":"2024-08-17T18:18:47.001385+0000","last_clean":"2024-08-17T18:18:47.001385+0000","last_became_active":"2024-08-17T18:17:57.576108+0000","last_became_peered":"2024-08-17T18:17:57.576108+0000","last_unstale":"2024-08-17T18:18:47.001385+0000","last_undegraded":"2024-08-17T18:18:47.001385+0000","last_fullsized":"2024-08-17T18:18:47.001385+0000","mapping_epoch":1660,"log_start":"1613'3387","ondisk_log_start":"1613'3387","created":1616,"last_epoch_clean":1661,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4565,"log_dups_size":0,"ondisk_log_size":4565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99385344,"num_objects":1517,"num_object_clones":0,"num_object_copies":4551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5517,"num_write_kb":225056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":166985728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.15","version":"1672'8090","reported_seq":11096,"reported_epoch":1672,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:18:46.999748+0000","last_change":"2024-08-17T18:18:32.060713+0000","last_active":"2024-08-17T18:18:46.999748+0000","last_peered":"2024-08-17T18:18:46.999748+0000","last_clean":"2024-08-17T18:15:09.587418+0000","last_became_active":"2024-08-17T18:18:32.037569+0000","last_became_peered":"2024-08-17T18:18:32.037569+0000","last_unstale":"2024-08-17T18:18:46.999748+0000","last_undegraded":"2024-08-17T18:18:31.866840+0000","last_fullsized":"2024-08-17T18:18:46.999748+0000","mapping_epoch":1668,"log_start":"1613'3436","ondisk_log_start":"1613'3436","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4654,"log_dups_size":0,"ondisk_log_size":4654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112951296,"num_objects":1724,"num_object_clones":0,"num_object_copies":5172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3887,"num_objects_misplaced":443,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5588,"num_write_kb":233952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":960,"num_bytes_recovered":62849024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,7,2],"avail_no_missing":["5","0","6"],"object_location_counts":[{"shards":"0,5,6","objects":1191},{"shards":"2,5,6","objects":443},{"shards":"2,5,7","objects":90}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.14","version":"1672'7963","reported_seq":8398,"reported_epoch":1672,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:18:47.097195+0000","last_change":"2024-08-17T18:18:32.046673+0000","last_active":"2024-08-17T18:18:47.097195+0000","last_peered":"2024-08-17T18:18:47.097195+0000","last_clean":"2024-08-17T18:15:09.651475+0000","last_became_active":"2024-08-17T18:18:31.915649+0000","last_became_peered":"2024-08-17T18:18:31.915649+0000","last_unstale":"2024-08-17T18:18:47.097195+0000","last_undegraded":"2024-08-17T18:18:31.906636+0000","last_fullsized":"2024-08-17T18:18:47.097195+0000","mapping_epoch":1668,"log_start":"1613'3295","ondisk_log_start":"1613'3295","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4668,"log_dups_size":0,"ondisk_log_size":4668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104988672,"num_objects":1602,"num_object_clones":0,"num_object_copies":4806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1602,"num_objects_misplaced":3419,"num_objects_unfound":0,"num_objects_dirty":1602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5510,"num_write_kb":227584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":9175040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[5,7,4],"avail_no_missing":["5","0","4"],"object_location_counts":[{"shards":"0,4,5","objects":1163},{"shards":"4,5,6","objects":441},{"shards":"4,5,7","objects":-2}],"blocked_by":[],"up_primary":3,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.13","version":"1672'7996","reported_seq":10104,"reported_epoch":1672,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:18:46.928105+0000","last_change":"2024-08-17T18:18:31.912458+0000","last_active":"2024-08-17T18:18:46.928105+0000","last_peered":"2024-08-17T18:18:46.928105+0000","last_clean":"2024-08-17T18:15:52.116504+0000","last_became_active":"2024-08-17T18:18:31.877372+0000","last_became_peered":"2024-08-17T18:18:31.877372+0000","last_unstale":"2024-08-17T18:18:46.928105+0000","last_undegraded":"2024-08-17T18:18:31.859470+0000","last_fullsized":"2024-08-17T18:18:46.928105+0000","mapping_epoch":1668,"log_start":"1613'1792","ondisk_log_start":"1613'1792","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":6204,"log_dups_size":0,"ondisk_log_size":6204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105938944,"num_objects":1617,"num_object_clones":0,"num_object_copies":4851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2429,"num_objects_misplaced":1774,"num_objects_unfound":0,"num_objects_dirty":1617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5547,"num_write_kb":229216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":973,"num_bytes_recovered":63766528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[6,7,0],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"0,2,6","objects":1594},{"shards":"0,6,7","objects":23}],"blocked_by":[],"up_primary":3,"acting_primary":6,"purged_snaps":[]},{"pgid":"57.12","version":"1672'8042","reported_seq":14006,"reported_epoch":1672,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:18:49.480371+0000","last_change":"2024-08-17T18:18:34.413365+0000","last_active":"2024-08-17T18:18:49.480371+0000","last_peered":"2024-08-17T18:18:49.480371+0000","last_clean":"2024-08-17T18:14:59.641932+0000","last_became_active":"2024-08-17T18:18:33.841392+0000","last_became_peered":"2024-08-17T18:18:33.841392+0000","last_unstale":"2024-08-17T18:18:49.480371+0000","last_undegraded":"2024-08-17T18:18:32.544363+0000","last_fullsized":"2024-08-17T18:18:30.719339+0000","mapping_epoch":1667,"log_start":"1613'2563","ondisk_log_start":"1613'2563","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5479,"log_dups_size":0,"ondisk_log_size":5479,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104300544,"num_objects":1592,"num_object_clones":0,"num_object_copies":4776,"num_objects_missing_on_primary":831,"num_objects_missing":831,"num_objects_degraded":2423,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5586,"num_write_kb":229664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1990,"num_bytes_recovered":130351104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[4,7],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,2,4,5","objects":2},{"shards":"0,2,5","objects":831},{"shards":"0,4,5","objects":302},{"shards":"2,4,5","objects":441},{"shards":"4,7","objects":16}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"57.11","version":"1672'8249","reported_seq":8440,"reported_epoch":1672,"state":"active+remapped","last_fresh":"2024-08-17T18:18:45.765292+0000","last_change":"2024-08-17T18:18:29.889219+0000","last_active":"2024-08-17T18:18:45.765292+0000","last_peered":"2024-08-17T18:18:45.765292+0000","last_clean":"2024-08-17T18:14:57.659306+0000","last_became_active":"2024-08-17T18:18:29.827929+0000","last_became_peered":"2024-08-17T18:18:29.827929+0000","last_unstale":"2024-08-17T18:18:45.765292+0000","last_undegraded":"2024-08-17T18:18:45.765292+0000","last_fullsized":"2024-08-17T18:18:45.765292+0000","mapping_epoch":1666,"log_start":"1611'100","ondisk_log_start":"1611'100","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":8149,"log_dups_size":0,"ondisk_log_size":8149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107905024,"num_objects":1647,"num_object_clones":0,"num_object_copies":4941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4941,"num_objects_unfound":0,"num_objects_dirty":1647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5729,"num_write_kb":236000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[0,6,2],"avail_no_missing":["0","2","6"],"object_location_counts":[{"shards":"0,2,6","objects":1647}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"57.10","version":"1672'8137","reported_seq":18910,"reported_epoch":1672,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:18:46.992648+0000","last_change":"2024-08-17T18:18:31.925795+0000","last_active":"2024-08-17T18:18:46.992648+0000","last_peered":"2024-08-17T18:18:46.992648+0000","last_clean":"2024-08-17T18:15:52.133173+0000","last_became_active":"2024-08-17T18:18:31.867521+0000","last_became_peered":"2024-08-17T18:18:31.867521+0000","last_unstale":"2024-08-17T18:18:46.992648+0000","last_undegraded":"2024-08-17T18:18:31.852455+0000","last_fullsized":"2024-08-17T18:18:46.992648+0000","mapping_epoch":1668,"log_start":"1613'2044","ondisk_log_start":"1613'2044","created":1616,"last_epoch_clean":1632,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":6093,"log_dups_size":0,"ondisk_log_size":6093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102924288,"num_objects":1571,"num_object_clones":0,"num_object_copies":4713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2428,"num_objects_misplaced":1803,"num_objects_unfound":0,"num_objects_dirty":1571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5645,"num_write_kb":230880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3864,"num_bytes_recovered":253231104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,2],"avail_no_missing":["6","0","4","5"],"object_location_counts":[{"shards":"2,4,5,6","objects":845},{"shards":"2,5,6","objects":760},{"shards":"2,6,7","objects":-34}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"57.f","version":"1672'11459","reported_seq":11649,"reported_epoch":1672,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:18:47.286594+0000","last_change":"2024-08-17T18:15:53.709162+0000","last_active":"2024-08-17T18:18:47.286594+0000","last_peered":"2024-08-17T18:18:47.286594+0000","last_clean":"2024-08-17T18:15:09.658885+0000","last_became_active":"2024-08-17T18:15:53.364563+0000","last_became_peered":"2024-08-17T18:15:53.364563+0000","last_unstale":"2024-08-17T18:18:47.286594+0000","last_undegraded":"2024-08-17T18:18:47.286594+0000","last_fullsized":"2024-08-17T18:18:47.286594+0000","mapping_epoch":1637,"log_start":"1633'7258","ondisk_log_start":"1633'7258","created":1604,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4201,"log_dups_size":0,"ondisk_log_size":4201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212926464,"num_objects":3249,"num_object_clones":0,"num_object_copies":9747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3245,"num_objects_unfound":0,"num_objects_dirty":3249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11459,"num_write_kb":470656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":458752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,6,0],"avail_no_missing":["2","0","6"],"object_location_counts":[{"shards":"0,2,6","objects":3249}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"57.e","version":"1672'11360","reported_seq":11455,"reported_epoch":1672,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:18:49.448168+0000","last_change":"2024-08-17T18:17:23.713508+0000","last_active":"2024-08-17T18:18:49.448168+0000","last_peered":"2024-08-17T18:18:49.448168+0000","last_clean":"2024-08-17T18:15:52.176790+0000","last_became_active":"2024-08-17T18:17:23.688213+0000","last_became_peered":"2024-08-17T18:17:23.688213+0000","last_unstale":"2024-08-17T18:18:49.448168+0000","last_undegraded":"2024-08-17T18:18:49.448168+0000","last_fullsized":"2024-08-17T18:18:49.448168+0000","mapping_epoch":1653,"log_start":"1630'6818","ondisk_log_start":"1630'6818","created":1604,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4542,"log_dups_size":0,"ondisk_log_size":4542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:12:51.202869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208928768,"num_objects":3188,"num_object_clones":0,"num_object_copies":9564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3188,"num_objects_unfound":0,"num_objects_dirty":3188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11360,"num_write_kb":465536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[4,5,6],"avail_no_missing":["4","5","6"],"object_location_counts":[{"shards":"4,5,6","objects":3188}],"blocked_by":[],"up_primary":7,"acting_primary":4,"purged_snaps":[]},{"pgid":"57.d","version":"1672'11176","reported_seq":12315,"reported_epoch":1672,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:18:46.262757+0000","last_change":"2024-08-17T18:18:30.832971+0000","last_active":"2024-08-17T18:18:46.262757+0000","last_peered":"2024-08-17T18:18:46.262757+0000","last_clean":"2024-08-17T18:17:21.525603+0000","last_became_active":"2024-08-17T18:18:29.819768+0000","last_became_peered":"2024-08-17T18:18:29.819768+0000","last_unstale":"2024-08-17T18:18:46.262757+0000","last_undegraded":"2024-08-17T18:18:46.262757+0000","last_fullsized":"2024-08-17T18:18:46.262757+0000","mapping_epoch":1666,"log_start":"1617'5227","ondisk_log_start":"1617'5227","created":1604,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5949,"log_dups_size":0,"ondisk_log_size":5949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216072216,"num_objects":3298,"num_object_clones":0,"num_object_copies":9894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2833,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11176,"num_write_kb":463105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":508,"num_bytes_recovered":33292288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,2,6],"avail_no_missing":["0","2","6"],"object_location_counts":[{"shards":"0,2,6","objects":3298}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"57.c","version":"1672'11453","reported_seq":11532,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:47.019702+0000","last_change":"2024-08-17T18:13:54.756490+0000","last_active":"2024-08-17T18:18:47.019702+0000","last_peered":"2024-08-17T18:18:47.019702+0000","last_clean":"2024-08-17T18:18:47.019702+0000","last_became_active":"2024-08-17T18:13:54.756109+0000","last_became_peered":"2024-08-17T18:13:54.756109+0000","last_unstale":"2024-08-17T18:18:47.019702+0000","last_undegraded":"2024-08-17T18:18:47.019702+0000","last_fullsized":"2024-08-17T18:18:47.019702+0000","mapping_epoch":1604,"log_start":"1630'6633","ondisk_log_start":"1630'6633","created":1604,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:09:29.691722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216989696,"num_objects":3311,"num_object_clones":0,"num_object_copies":9933,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11453,"num_write_kb":472448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"57.b","version":"1672'11186","reported_seq":11341,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:47.251584+0000","last_change":"2024-08-17T18:13:54.755437+0000","last_active":"2024-08-17T18:18:47.251584+0000","last_peered":"2024-08-17T18:18:47.251584+0000","last_clean":"2024-08-17T18:18:47.251584+0000","last_became_active":"2024-08-17T18:13:54.755175+0000","last_became_peered":"2024-08-17T18:13:54.755175+0000","last_unstale":"2024-08-17T18:18:47.251584+0000","last_undegraded":"2024-08-17T18:18:47.251584+0000","last_fullsized":"2024-08-17T18:18:47.251584+0000","mapping_epoch":1604,"log_start":"1633'6956","ondisk_log_start":"1633'6956","created":1604,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4230,"log_dups_size":0,"ondisk_log_size":4230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:27.516981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211156992,"num_objects":3222,"num_object_clones":0,"num_object_copies":9666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11186,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"57.a","version":"1672'11386","reported_seq":11489,"reported_epoch":1672,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:18:46.992496+0000","last_change":"2024-08-17T18:18:29.943391+0000","last_active":"2024-08-17T18:18:46.992496+0000","last_peered":"2024-08-17T18:18:46.992496+0000","last_clean":"2024-08-17T18:15:52.195344+0000","last_became_active":"2024-08-17T18:18:29.889053+0000","last_became_peered":"2024-08-17T18:18:29.889053+0000","last_unstale":"2024-08-17T18:18:46.992496+0000","last_undegraded":"2024-08-17T18:18:46.992496+0000","last_fullsized":"2024-08-17T18:18:46.992496+0000","mapping_epoch":1666,"log_start":"1633'7507","ondisk_log_start":"1633'7507","created":1604,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3879,"log_dups_size":0,"ondisk_log_size":3879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:46:14.340957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212467712,"num_objects":3242,"num_object_clones":0,"num_object_copies":9726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6484,"num_objects_unfound":0,"num_objects_dirty":3242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11386,"num_write_kb":468096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,4,2],"avail_no_missing":["5","2","4"],"object_location_counts":[{"shards":"2,4,5","objects":3242}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.9","version":"1672'8153","reported_seq":8250,"reported_epoch":1672,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:18:49.275402+0000","last_change":"2024-08-17T18:18:30.827730+0000","last_active":"2024-08-17T18:18:49.275402+0000","last_peered":"2024-08-17T18:18:49.275402+0000","last_clean":"2024-08-17T18:18:29.565716+0000","last_became_active":"2024-08-17T18:18:30.810210+0000","last_became_peered":"2024-08-17T18:18:30.810210+0000","last_unstale":"2024-08-17T18:18:49.275402+0000","last_undegraded":"2024-08-17T18:18:49.275402+0000","last_fullsized":"2024-08-17T18:18:49.275402+0000","mapping_epoch":1667,"log_start":"1613'2777","ondisk_log_start":"1613'2777","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5376,"log_dups_size":0,"ondisk_log_size":5376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103841792,"num_objects":1584,"num_object_clones":0,"num_object_copies":4752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1584,"num_objects_unfound":0,"num_objects_dirty":1584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5674,"num_write_kb":232288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[4,5,6],"avail_no_missing":["4","5","6"],"object_location_counts":[{"shards":"4,5,6","objects":1584}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"57.8","version":"1672'8137","reported_seq":11176,"reported_epoch":1672,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T18:18:47.132804+0000","last_change":"2024-08-17T18:18:29.930289+0000","last_active":"2024-08-17T18:18:47.132804+0000","last_peered":"2024-08-17T18:18:47.132804+0000","last_clean":"2024-08-17T18:15:09.643183+0000","last_became_active":"2024-08-17T18:15:53.694715+0000","last_became_peered":"2024-08-17T18:15:53.694715+0000","last_unstale":"2024-08-17T18:18:47.132804+0000","last_undegraded":"2024-08-17T18:15:53.332758+0000","last_fullsized":"2024-08-17T18:15:53.222025+0000","mapping_epoch":1637,"log_start":"1613'3489","ondisk_log_start":"1613'3489","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4648,"log_dups_size":0,"ondisk_log_size":4648,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101810176,"num_objects":1553,"num_object_clones":0,"num_object_copies":4659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1709,"num_objects_misplaced":197,"num_objects_unfound":0,"num_objects_dirty":1553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5629,"num_write_kb":229856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1468,"num_bytes_recovered":96206848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[5,6],"avail_no_missing":["5","0","6"],"object_location_counts":[{"shards":"4,5,6","objects":197},{"shards":"5,6","objects":-353}],"blocked_by":[],"up_primary":7,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.7","version":"1672'8021","reported_seq":10170,"reported_epoch":1672,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:18:47.325171+0000","last_change":"2024-08-17T18:17:23.729203+0000","last_active":"2024-08-17T18:18:47.325171+0000","last_peered":"2024-08-17T18:18:47.325171+0000","last_clean":"2024-08-17T18:15:52.213196+0000","last_became_active":"2024-08-17T18:17:23.719428+0000","last_became_peered":"2024-08-17T18:17:23.719428+0000","last_unstale":"2024-08-17T18:18:47.325171+0000","last_undegraded":"2024-08-17T18:17:23.715305+0000","last_fullsized":"2024-08-17T18:17:23.631126+0000","mapping_epoch":1653,"log_start":"1613'2828","ondisk_log_start":"1613'2828","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5193,"log_dups_size":0,"ondisk_log_size":5193,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102596608,"num_objects":1565,"num_object_clones":0,"num_object_copies":4695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5589,"num_write_kb":228960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":983,"num_bytes_recovered":64421888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,4,6","objects":1583},{"shards":"2,7","objects":-18}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"57.6","version":"1672'7960","reported_seq":8065,"reported_epoch":1672,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:18:47.008118+0000","last_change":"2024-08-17T18:18:29.888707+0000","last_active":"2024-08-17T18:18:47.008118+0000","last_peered":"2024-08-17T18:18:47.008118+0000","last_clean":"2024-08-17T18:15:09.669695+0000","last_became_active":"2024-08-17T18:15:53.694052+0000","last_became_peered":"2024-08-17T18:15:53.694052+0000","last_unstale":"2024-08-17T18:18:47.008118+0000","last_undegraded":"2024-08-17T18:15:53.322865+0000","last_fullsized":"2024-08-17T18:15:53.252808+0000","mapping_epoch":1637,"log_start":"1613'3306","ondisk_log_start":"1613'3306","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4654,"log_dups_size":0,"ondisk_log_size":4654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107380736,"num_objects":1638,"num_object_clones":0,"num_object_copies":4914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3544,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5524,"num_write_kb":229216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2],"avail_no_missing":["5","0","2"],"object_location_counts":[{"shards":"0,2,5","objects":1173},{"shards":"2,4,5","objects":427},{"shards":"2,5","objects":-1707}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.5","version":"1672'8214","reported_seq":12086,"reported_epoch":1672,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:18:46.902482+0000","last_change":"2024-08-17T18:18:30.123049+0000","last_active":"2024-08-17T18:18:46.902482+0000","last_peered":"2024-08-17T18:18:46.902482+0000","last_clean":"2024-08-17T18:15:09.654427+0000","last_became_active":"2024-08-17T18:18:30.099398+0000","last_became_peered":"2024-08-17T18:18:30.099398+0000","last_unstale":"2024-08-17T18:18:46.902482+0000","last_undegraded":"2024-08-17T18:18:30.063992+0000","last_fullsized":"2024-08-17T18:18:46.902482+0000","mapping_epoch":1666,"log_start":"1613'3785","ondisk_log_start":"1613'3785","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4429,"log_dups_size":0,"ondisk_log_size":4429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102334464,"num_objects":1561,"num_object_clones":0,"num_object_copies":4683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1424,"num_objects_misplaced":1698,"num_objects_unfound":0,"num_objects_dirty":1561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5711,"num_write_kb":232736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1869,"num_bytes_recovered":122421248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,2,6],"avail_no_missing":["5","0","6"],"object_location_counts":[{"shards":"0,5,6","objects":1217},{"shards":"2,5,6","objects":344}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.4","version":"1672'8053","reported_seq":15394,"reported_epoch":1672,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:18:49.405196+0000","last_change":"2024-08-17T18:18:32.134646+0000","last_active":"2024-08-17T18:18:49.405196+0000","last_peered":"2024-08-17T18:18:49.405196+0000","last_clean":"2024-08-17T18:18:29.526320+0000","last_became_active":"2024-08-17T18:18:32.122736+0000","last_became_peered":"2024-08-17T18:18:32.122736+0000","last_unstale":"2024-08-17T18:18:49.405196+0000","last_undegraded":"2024-08-17T18:18:49.405196+0000","last_fullsized":"2024-08-17T18:18:49.405196+0000","mapping_epoch":1668,"log_start":"1613'3121","ondisk_log_start":"1613'3121","created":1604,"last_epoch_clean":1647,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4932,"log_dups_size":0,"ondisk_log_size":4932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103940096,"num_objects":1586,"num_object_clones":0,"num_object_copies":4758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1586,"num_objects_unfound":0,"num_objects_dirty":1586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5600,"num_write_kb":229952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3591,"num_bytes_recovered":223805440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[4,5,7],"avail_no_missing":["4","5","7"],"object_location_counts":[{"shards":"4,5,7","objects":1586}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"57.3","version":"1672'8045","reported_seq":10209,"reported_epoch":1672,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:18:47.036095+0000","last_change":"2024-08-17T18:18:31.936546+0000","last_active":"2024-08-17T18:18:47.036095+0000","last_peered":"2024-08-17T18:18:47.036095+0000","last_clean":"2024-08-17T18:15:52.162248+0000","last_became_active":"2024-08-17T18:18:31.914458+0000","last_became_peered":"2024-08-17T18:18:31.914458+0000","last_unstale":"2024-08-17T18:18:47.036095+0000","last_undegraded":"2024-08-17T18:18:31.855752+0000","last_fullsized":"2024-08-17T18:18:47.036095+0000","mapping_epoch":1668,"log_start":"1613'1949","ondisk_log_start":"1613'1949","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":6096,"log_dups_size":0,"ondisk_log_size":6096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107053056,"num_objects":1633,"num_object_clones":0,"num_object_copies":4899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2434,"num_objects_misplaced":1837,"num_objects_unfound":0,"num_objects_dirty":1633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5595,"num_write_kb":231328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":999,"num_bytes_recovered":65470464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[6,7,0],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"0,2,6","objects":1656},{"shards":"0,6,7","objects":-23}],"blocked_by":[],"up_primary":3,"acting_primary":6,"purged_snaps":[]},{"pgid":"57.2","version":"1672'8071","reported_seq":8813,"reported_epoch":1672,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:18:47.173259+0000","last_change":"2024-08-17T18:18:33.840312+0000","last_active":"2024-08-17T18:18:47.173259+0000","last_peered":"2024-08-17T18:18:47.173259+0000","last_clean":"2024-08-17T18:15:09.574793+0000","last_became_active":"2024-08-17T18:18:33.824434+0000","last_became_peered":"2024-08-17T18:18:33.824434+0000","last_unstale":"2024-08-17T18:18:47.173259+0000","last_undegraded":"2024-08-17T18:18:30.818589+0000","last_fullsized":"2024-08-17T18:18:47.173259+0000","mapping_epoch":1667,"log_start":"1613'2919","ondisk_log_start":"1613'2919","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5152,"log_dups_size":0,"ondisk_log_size":5152,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108822528,"num_objects":1660,"num_object_clones":0,"num_object_copies":4980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5614,"num_write_kb":232800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":278,"num_bytes_recovered":18219008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[2,5,4],"avail_no_missing":["2","0","5"],"object_location_counts":[{"shards":"0,2,5","objects":1207},{"shards":"2,4,5","objects":453}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"57.1","version":"1672'8178","reported_seq":8342,"reported_epoch":1672,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:18:46.814075+0000","last_change":"2024-08-17T18:15:53.720586+0000","last_active":"2024-08-17T18:18:46.814075+0000","last_peered":"2024-08-17T18:18:46.814075+0000","last_clean":"2024-08-17T18:15:09.579791+0000","last_became_active":"2024-08-17T18:15:53.381797+0000","last_became_peered":"2024-08-17T18:15:53.381797+0000","last_unstale":"2024-08-17T18:18:46.814075+0000","last_undegraded":"2024-08-17T18:18:46.814075+0000","last_fullsized":"2024-08-17T18:18:46.814075+0000","mapping_epoch":1637,"log_start":"1613'2059","ondisk_log_start":"1613'2059","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":6119,"log_dups_size":0,"ondisk_log_size":6119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113213440,"num_objects":1727,"num_object_clones":0,"num_object_copies":5181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1727,"num_objects_unfound":0,"num_objects_dirty":1727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5657,"num_write_kb":236320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":589824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,0],"avail_no_missing":["6","0","2"],"object_location_counts":[{"shards":"0,2,6","objects":1727}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"57.0","version":"1672'8111","reported_seq":21863,"reported_epoch":1672,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:18:47.295024+0000","last_change":"2024-08-17T18:18:31.919624+0000","last_active":"2024-08-17T18:18:47.295024+0000","last_peered":"2024-08-17T18:18:47.295024+0000","last_clean":"2024-08-17T18:18:29.609086+0000","last_became_active":"2024-08-17T18:18:31.860120+0000","last_became_peered":"2024-08-17T18:18:31.860120+0000","last_unstale":"2024-08-17T18:18:47.295024+0000","last_undegraded":"2024-08-17T18:18:47.295024+0000","last_fullsized":"2024-08-17T18:18:47.295024+0000","mapping_epoch":1668,"log_start":"1613'2901","ondisk_log_start":"1613'2901","created":1604,"last_epoch_clean":1665,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5210,"log_dups_size":0,"ondisk_log_size":5210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108036096,"num_objects":1648,"num_object_clones":0,"num_object_copies":4944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1648,"num_objects_unfound":0,"num_objects_dirty":1648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5618,"num_write_kb":232544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6772,"num_bytes_recovered":353239040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,7],"avail_no_missing":["2","0","4","5","7"],"object_location_counts":[{"shards":"2,4,7","objects":1648}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":108,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.761740+0000","last_change":"2024-08-17T18:17:22.697769+0000","last_active":"2024-08-17T18:18:33.761740+0000","last_peered":"2024-08-17T18:18:33.761740+0000","last_clean":"2024-08-17T18:18:33.761740+0000","last_became_active":"2024-08-17T18:17:22.688989+0000","last_became_peered":"2024-08-17T18:17:22.688989+0000","last_unstale":"2024-08-17T18:18:33.761740+0000","last_undegraded":"2024-08-17T18:18:33.761740+0000","last_fullsized":"2024-08-17T18:18:33.761740+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:10:04.677811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2328,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1715,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.730353+0000","last_change":"2024-08-17T18:18:29.727855+0000","last_active":"2024-08-17T18:18:33.730353+0000","last_peered":"2024-08-17T18:18:33.730353+0000","last_clean":"2024-08-17T18:18:33.730353+0000","last_became_active":"2024-08-17T18:18:29.727091+0000","last_became_peered":"2024-08-17T18:18:29.727091+0000","last_unstale":"2024-08-17T18:18:33.730353+0000","last_undegraded":"2024-08-17T18:18:33.730353+0000","last_fullsized":"2024-08-17T18:18:33.730353+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:04.660094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":22,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.752603+0000","last_change":"2024-08-17T18:18:29.895953+0000","last_active":"2024-08-17T18:18:34.752603+0000","last_peered":"2024-08-17T18:18:34.752603+0000","last_clean":"2024-08-17T18:18:34.752603+0000","last_became_active":"2024-08-17T18:18:29.895460+0000","last_became_peered":"2024-08-17T18:18:29.895460+0000","last_unstale":"2024-08-17T18:18:34.752603+0000","last_undegraded":"2024-08-17T18:18:34.752603+0000","last_fullsized":"2024-08-17T18:18:34.752603+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:04:41.818991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.752561+0000","last_change":"2024-08-17T18:18:29.732425+0000","last_active":"2024-08-17T18:18:34.752561+0000","last_peered":"2024-08-17T18:18:34.752561+0000","last_clean":"2024-08-17T18:18:34.752561+0000","last_became_active":"2024-08-17T18:18:29.731726+0000","last_became_peered":"2024-08-17T18:18:29.731726+0000","last_unstale":"2024-08-17T18:18:34.752561+0000","last_undegraded":"2024-08-17T18:18:34.752561+0000","last_fullsized":"2024-08-17T18:18:34.752561+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:01:33.556710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":227,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.766228+0000","last_change":"2024-08-17T18:04:43.777705+0000","last_active":"2024-08-17T18:18:33.766228+0000","last_peered":"2024-08-17T18:18:33.766228+0000","last_clean":"2024-08-17T18:18:33.766228+0000","last_became_active":"2024-08-17T18:04:43.771283+0000","last_became_peered":"2024-08-17T18:04:43.771283+0000","last_unstale":"2024-08-17T18:18:33.766228+0000","last_undegraded":"2024-08-17T18:18:33.766228+0000","last_fullsized":"2024-08-17T18:18:33.766228+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:13.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.751856+0000","last_change":"2024-08-17T18:18:29.734730+0000","last_active":"2024-08-17T18:18:34.751856+0000","last_peered":"2024-08-17T18:18:34.751856+0000","last_clean":"2024-08-17T18:18:34.751856+0000","last_became_active":"2024-08-17T18:18:29.734362+0000","last_became_peered":"2024-08-17T18:18:29.734362+0000","last_unstale":"2024-08-17T18:18:34.751856+0000","last_undegraded":"2024-08-17T18:18:34.751856+0000","last_fullsized":"2024-08-17T18:18:34.751856+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:58:48.630762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":240,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.759500+0000","last_change":"2024-08-17T18:12:04.655551+0000","last_active":"2024-08-17T18:18:33.759500+0000","last_peered":"2024-08-17T18:18:33.759500+0000","last_clean":"2024-08-17T18:18:33.759500+0000","last_became_active":"2024-08-17T18:12:04.646075+0000","last_became_peered":"2024-08-17T18:12:04.646075+0000","last_unstale":"2024-08-17T18:18:33.759500+0000","last_undegraded":"2024-08-17T18:18:33.759500+0000","last_fullsized":"2024-08-17T18:18:33.759500+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:19:19.300135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1711,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.758822+0000","last_change":"2024-08-17T18:18:29.727683+0000","last_active":"2024-08-17T18:18:33.758822+0000","last_peered":"2024-08-17T18:18:33.758822+0000","last_clean":"2024-08-17T18:18:33.758822+0000","last_became_active":"2024-08-17T18:18:29.727378+0000","last_became_peered":"2024-08-17T18:18:29.727378+0000","last_unstale":"2024-08-17T18:18:33.758822+0000","last_undegraded":"2024-08-17T18:18:33.758822+0000","last_fullsized":"2024-08-17T18:18:33.758822+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:42.989331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":198,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.758727+0000","last_change":"2024-08-17T18:18:29.882947+0000","last_active":"2024-08-17T18:18:33.758727+0000","last_peered":"2024-08-17T18:18:33.758727+0000","last_clean":"2024-08-17T18:18:33.758727+0000","last_became_active":"2024-08-17T18:18:29.882533+0000","last_became_peered":"2024-08-17T18:18:29.882533+0000","last_unstale":"2024-08-17T18:18:33.758727+0000","last_undegraded":"2024-08-17T18:18:33.758727+0000","last_fullsized":"2024-08-17T18:18:33.758727+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:34:25.726288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":23,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.752260+0000","last_change":"2024-08-17T18:18:29.734046+0000","last_active":"2024-08-17T18:18:34.752260+0000","last_peered":"2024-08-17T18:18:34.752260+0000","last_clean":"2024-08-17T18:18:34.752260+0000","last_became_active":"2024-08-17T18:18:29.733663+0000","last_became_peered":"2024-08-17T18:18:29.733663+0000","last_unstale":"2024-08-17T18:18:34.752260+0000","last_undegraded":"2024-08-17T18:18:34.752260+0000","last_fullsized":"2024-08-17T18:18:34.752260+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:08:09.146146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1706,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.399601+0000","last_change":"2024-08-17T18:16:00.324143+0000","last_active":"2024-08-17T18:18:34.399601+0000","last_peered":"2024-08-17T18:18:34.399601+0000","last_clean":"2024-08-17T18:18:34.399601+0000","last_became_active":"2024-08-17T18:15:57.106000+0000","last_became_peered":"2024-08-17T18:15:57.106000+0000","last_unstale":"2024-08-17T18:18:34.399601+0000","last_undegraded":"2024-08-17T18:18:34.399601+0000","last_fullsized":"2024-08-17T18:18:34.399601+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1641,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:43.117961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.753457+0000","last_change":"2024-08-17T18:18:29.733983+0000","last_active":"2024-08-17T18:18:34.753457+0000","last_peered":"2024-08-17T18:18:34.753457+0000","last_clean":"2024-08-17T18:18:34.753457+0000","last_became_active":"2024-08-17T18:18:29.733635+0000","last_became_peered":"2024-08-17T18:18:29.733635+0000","last_unstale":"2024-08-17T18:18:34.753457+0000","last_undegraded":"2024-08-17T18:18:34.753457+0000","last_fullsized":"2024-08-17T18:18:34.753457+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:47:15.631401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.752408+0000","last_change":"2024-08-17T18:18:29.888141+0000","last_active":"2024-08-17T18:18:34.752408+0000","last_peered":"2024-08-17T18:18:34.752408+0000","last_clean":"2024-08-17T18:18:34.752408+0000","last_became_active":"2024-08-17T18:18:29.887829+0000","last_became_peered":"2024-08-17T18:18:29.887829+0000","last_unstale":"2024-08-17T18:18:34.752408+0000","last_undegraded":"2024-08-17T18:18:34.752408+0000","last_fullsized":"2024-08-17T18:18:34.752408+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:23:13.375118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":22,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.752707+0000","last_change":"2024-08-17T18:18:29.896201+0000","last_active":"2024-08-17T18:18:34.752707+0000","last_peered":"2024-08-17T18:18:34.752707+0000","last_clean":"2024-08-17T18:18:34.752707+0000","last_became_active":"2024-08-17T18:18:29.895687+0000","last_became_peered":"2024-08-17T18:18:29.895687+0000","last_unstale":"2024-08-17T18:18:34.752707+0000","last_undegraded":"2024-08-17T18:18:34.752707+0000","last_fullsized":"2024-08-17T18:18:34.752707+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:24:52.525561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.751608+0000","last_change":"2024-08-17T18:18:29.888209+0000","last_active":"2024-08-17T18:18:34.751608+0000","last_peered":"2024-08-17T18:18:34.751608+0000","last_clean":"2024-08-17T18:18:34.751608+0000","last_became_active":"2024-08-17T18:18:29.887651+0000","last_became_peered":"2024-08-17T18:18:29.887651+0000","last_unstale":"2024-08-17T18:18:34.751608+0000","last_undegraded":"2024-08-17T18:18:34.751608+0000","last_fullsized":"2024-08-17T18:18:34.751608+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:55:59.802775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1717,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.738866+0000","last_change":"2024-08-17T18:18:29.887228+0000","last_active":"2024-08-17T18:18:34.738866+0000","last_peered":"2024-08-17T18:18:34.738866+0000","last_clean":"2024-08-17T18:18:34.738866+0000","last_became_active":"2024-08-17T18:18:29.884394+0000","last_became_peered":"2024-08-17T18:18:29.884394+0000","last_unstale":"2024-08-17T18:18:34.738866+0000","last_undegraded":"2024-08-17T18:18:34.738866+0000","last_fullsized":"2024-08-17T18:18:34.738866+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:41:42.256082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.751700+0000","last_change":"2024-08-17T18:18:29.727770+0000","last_active":"2024-08-17T18:18:34.751700+0000","last_peered":"2024-08-17T18:18:34.751700+0000","last_clean":"2024-08-17T18:18:34.751700+0000","last_became_active":"2024-08-17T18:18:29.727324+0000","last_became_peered":"2024-08-17T18:18:29.727324+0000","last_unstale":"2024-08-17T18:18:34.751700+0000","last_undegraded":"2024-08-17T18:18:34.751700+0000","last_fullsized":"2024-08-17T18:18:34.751700+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:20:29.342237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":398,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.761824+0000","last_change":"2024-08-17T18:18:29.734707+0000","last_active":"2024-08-17T18:18:33.761824+0000","last_peered":"2024-08-17T18:18:33.761824+0000","last_clean":"2024-08-17T18:18:33.761824+0000","last_became_active":"2024-08-17T18:18:29.734420+0000","last_became_peered":"2024-08-17T18:18:29.734420+0000","last_unstale":"2024-08-17T18:18:33.761824+0000","last_undegraded":"2024-08-17T18:18:33.761824+0000","last_fullsized":"2024-08-17T18:18:33.761824+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:07:29.048061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.752324+0000","last_change":"2024-08-17T18:18:29.880752+0000","last_active":"2024-08-17T18:18:34.752324+0000","last_peered":"2024-08-17T18:18:34.752324+0000","last_clean":"2024-08-17T18:18:34.752324+0000","last_became_active":"2024-08-17T18:18:29.880508+0000","last_became_peered":"2024-08-17T18:18:29.880508+0000","last_unstale":"2024-08-17T18:18:34.752324+0000","last_undegraded":"2024-08-17T18:18:34.752324+0000","last_fullsized":"2024-08-17T18:18:34.752324+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:35:25.743073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":285,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.765197+0000","last_change":"2024-08-17T18:18:29.900929+0000","last_active":"2024-08-17T18:18:33.765197+0000","last_peered":"2024-08-17T18:18:33.765197+0000","last_clean":"2024-08-17T18:18:33.765197+0000","last_became_active":"2024-08-17T18:18:29.900417+0000","last_became_peered":"2024-08-17T18:18:29.900417+0000","last_unstale":"2024-08-17T18:18:33.765197+0000","last_undegraded":"2024-08-17T18:18:33.765197+0000","last_fullsized":"2024-08-17T18:18:33.765197+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:07:45.752696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":28,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.753357+0000","last_change":"2024-08-17T18:18:29.928751+0000","last_active":"2024-08-17T18:18:34.753357+0000","last_peered":"2024-08-17T18:18:34.753357+0000","last_clean":"2024-08-17T18:18:34.753357+0000","last_became_active":"2024-08-17T18:18:29.898102+0000","last_became_peered":"2024-08-17T18:18:29.898102+0000","last_unstale":"2024-08-17T18:18:34.753357+0000","last_undegraded":"2024-08-17T18:18:34.753357+0000","last_fullsized":"2024-08-17T18:18:34.753357+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:21.098266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2492,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.752857+0000","last_change":"2024-08-17T18:17:22.685988+0000","last_active":"2024-08-17T18:18:34.752857+0000","last_peered":"2024-08-17T18:18:34.752857+0000","last_clean":"2024-08-17T18:18:34.752857+0000","last_became_active":"2024-08-17T18:17:22.685791+0000","last_became_peered":"2024-08-17T18:17:22.685791+0000","last_unstale":"2024-08-17T18:18:34.752857+0000","last_undegraded":"2024-08-17T18:18:34.752857+0000","last_fullsized":"2024-08-17T18:18:34.752857+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:10:28.276954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1700,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.759786+0000","last_change":"2024-08-17T18:15:53.687128+0000","last_active":"2024-08-17T18:18:33.759786+0000","last_peered":"2024-08-17T18:18:33.759786+0000","last_clean":"2024-08-17T18:18:33.759786+0000","last_became_active":"2024-08-17T18:15:53.246153+0000","last_became_peered":"2024-08-17T18:15:53.246153+0000","last_unstale":"2024-08-17T18:18:33.759786+0000","last_undegraded":"2024-08-17T18:18:33.759786+0000","last_fullsized":"2024-08-17T18:18:33.759786+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:45:14.801100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1662,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.758907+0000","last_change":"2024-08-17T18:12:04.639269+0000","last_active":"2024-08-17T18:18:33.758907+0000","last_peered":"2024-08-17T18:18:33.758907+0000","last_clean":"2024-08-17T18:18:33.758907+0000","last_became_active":"2024-08-17T18:12:04.638752+0000","last_became_peered":"2024-08-17T18:12:04.638752+0000","last_unstale":"2024-08-17T18:18:33.758907+0000","last_undegraded":"2024-08-17T18:18:33.758907+0000","last_fullsized":"2024-08-17T18:18:33.758907+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:50:00.462972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"347'149","reported_seq":899,"reported_epoch":1671,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:18:33.761446+0000","last_change":"2024-08-17T18:18:30.750089+0000","last_active":"2024-08-17T18:18:33.761446+0000","last_peered":"2024-08-17T18:18:33.761446+0000","last_clean":"2024-08-17T18:15:50.198296+0000","last_became_active":"2024-08-17T18:18:30.749746+0000","last_became_peered":"2024-08-17T18:18:30.749746+0000","last_unstale":"2024-08-17T18:18:33.761446+0000","last_undegraded":"2024-08-17T18:18:30.723633+0000","last_fullsized":"2024-08-17T18:18:30.723013+0000","mapping_epoch":1667,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1612,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:57.271286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:57.271286+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:22.670286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[5,7],"avail_no_missing":["5","4","6"],"object_location_counts":[{"shards":"4,5,6","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1678,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.749024+0000","last_change":"2024-08-17T18:04:43.766724+0000","last_active":"2024-08-17T18:18:34.749024+0000","last_peered":"2024-08-17T18:18:34.749024+0000","last_clean":"2024-08-17T18:18:34.749024+0000","last_became_active":"2024-08-17T18:04:43.766428+0000","last_became_peered":"2024-08-17T18:04:43.766428+0000","last_unstale":"2024-08-17T18:18:34.749024+0000","last_undegraded":"2024-08-17T18:18:34.749024+0000","last_fullsized":"2024-08-17T18:18:34.749024+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:09:49.102940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1708,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.759751+0000","last_change":"2024-08-17T18:15:53.366349+0000","last_active":"2024-08-17T18:18:33.759751+0000","last_peered":"2024-08-17T18:18:33.759751+0000","last_clean":"2024-08-17T18:18:33.759751+0000","last_became_active":"2024-08-17T18:15:53.365863+0000","last_became_peered":"2024-08-17T18:15:53.365863+0000","last_unstale":"2024-08-17T18:18:33.759751+0000","last_undegraded":"2024-08-17T18:18:33.759751+0000","last_fullsized":"2024-08-17T18:18:33.759751+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:52.700374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":100,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.759186+0000","last_change":"2024-08-17T18:17:22.720026+0000","last_active":"2024-08-17T18:18:33.759186+0000","last_peered":"2024-08-17T18:18:33.759186+0000","last_clean":"2024-08-17T18:18:33.759186+0000","last_became_active":"2024-08-17T18:17:22.662636+0000","last_became_peered":"2024-08-17T18:17:22.662636+0000","last_unstale":"2024-08-17T18:18:33.759186+0000","last_undegraded":"2024-08-17T18:18:33.759186+0000","last_fullsized":"2024-08-17T18:18:33.759186+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:37:15.928482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"347'149","reported_seq":964,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.758270+0000","last_change":"2024-08-17T18:17:25.672410+0000","last_active":"2024-08-17T18:18:33.758270+0000","last_peered":"2024-08-17T18:18:33.758270+0000","last_clean":"2024-08-17T18:18:33.758270+0000","last_became_active":"2024-08-17T18:17:25.671993+0000","last_became_peered":"2024-08-17T18:17:25.671993+0000","last_unstale":"2024-08-17T18:18:33.758270+0000","last_undegraded":"2024-08-17T18:18:33.758270+0000","last_fullsized":"2024-08-17T18:18:33.758270+0000","mapping_epoch":1655,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1656,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:53.309517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:53.309517+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:21:08.929478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":22,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.751885+0000","last_change":"2024-08-17T18:18:29.888368+0000","last_active":"2024-08-17T18:18:34.751885+0000","last_peered":"2024-08-17T18:18:34.751885+0000","last_clean":"2024-08-17T18:18:34.751885+0000","last_became_active":"2024-08-17T18:18:29.887731+0000","last_became_peered":"2024-08-17T18:18:29.887731+0000","last_unstale":"2024-08-17T18:18:34.751885+0000","last_undegraded":"2024-08-17T18:18:34.751885+0000","last_fullsized":"2024-08-17T18:18:34.751885+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:47:07.960969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":760,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.739212+0000","last_change":"2024-08-17T18:18:29.728878+0000","last_active":"2024-08-17T18:18:34.739212+0000","last_peered":"2024-08-17T18:18:34.739212+0000","last_clean":"2024-08-17T18:18:34.739212+0000","last_became_active":"2024-08-17T18:18:29.728295+0000","last_became_peered":"2024-08-17T18:18:29.728295+0000","last_unstale":"2024-08-17T18:18:34.739212+0000","last_undegraded":"2024-08-17T18:18:34.739212+0000","last_fullsized":"2024-08-17T18:18:34.739212+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:37:24.603653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":27,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.753192+0000","last_change":"2024-08-17T18:18:29.904137+0000","last_active":"2024-08-17T18:18:34.753192+0000","last_peered":"2024-08-17T18:18:34.753192+0000","last_clean":"2024-08-17T18:18:34.753192+0000","last_became_active":"2024-08-17T18:18:29.894843+0000","last_became_peered":"2024-08-17T18:18:29.894843+0000","last_unstale":"2024-08-17T18:18:34.753192+0000","last_undegraded":"2024-08-17T18:18:34.753192+0000","last_fullsized":"2024-08-17T18:18:34.753192+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:23:04.104922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"347'149","reported_seq":998,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.762785+0000","last_change":"2024-08-17T18:18:33.761955+0000","last_active":"2024-08-17T18:18:33.762785+0000","last_peered":"2024-08-17T18:18:33.762785+0000","last_clean":"2024-08-17T18:18:33.762785+0000","last_became_active":"2024-08-17T18:18:33.761579+0000","last_became_peered":"2024-08-17T18:18:33.761579+0000","last_unstale":"2024-08-17T18:18:33.762785+0000","last_undegraded":"2024-08-17T18:18:33.762785+0000","last_fullsized":"2024-08-17T18:18:33.762785+0000","mapping_epoch":1670,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1671,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:52.474544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:52.474544+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:35:23.712308+0000","scrub_duration":0.001886326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":287,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.724991+0000","last_change":"2024-08-17T18:17:22.690072+0000","last_active":"2024-08-17T18:18:33.724991+0000","last_peered":"2024-08-17T18:18:33.724991+0000","last_clean":"2024-08-17T18:18:33.724991+0000","last_became_active":"2024-08-17T18:17:22.688530+0000","last_became_peered":"2024-08-17T18:17:22.688530+0000","last_unstale":"2024-08-17T18:18:33.724991+0000","last_undegraded":"2024-08-17T18:18:33.724991+0000","last_fullsized":"2024-08-17T18:18:33.724991+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:40:26.170186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":45,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.755989+0000","last_change":"2024-08-17T18:15:53.248168+0000","last_active":"2024-08-17T18:18:33.755989+0000","last_peered":"2024-08-17T18:18:33.755989+0000","last_clean":"2024-08-17T18:18:33.755989+0000","last_became_active":"2024-08-17T18:15:53.247888+0000","last_became_peered":"2024-08-17T18:15:53.247888+0000","last_unstale":"2024-08-17T18:18:33.755989+0000","last_undegraded":"2024-08-17T18:18:33.755989+0000","last_fullsized":"2024-08-17T18:18:33.755989+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:30:42.725233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2521,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.750232+0000","last_change":"2024-08-17T18:15:10.692541+0000","last_active":"2024-08-17T18:18:34.750232+0000","last_peered":"2024-08-17T18:18:34.750232+0000","last_clean":"2024-08-17T18:18:34.750232+0000","last_became_active":"2024-08-17T18:15:10.691876+0000","last_became_peered":"2024-08-17T18:15:10.691876+0000","last_unstale":"2024-08-17T18:18:34.750232+0000","last_undegraded":"2024-08-17T18:18:34.750232+0000","last_fullsized":"2024-08-17T18:18:34.750232+0000","mapping_epoch":1626,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:39:22.253615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"347'149","reported_seq":981,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.762773+0000","last_change":"2024-08-17T18:18:33.761996+0000","last_active":"2024-08-17T18:18:33.762773+0000","last_peered":"2024-08-17T18:18:33.762773+0000","last_clean":"2024-08-17T18:18:33.762773+0000","last_became_active":"2024-08-17T18:18:33.761606+0000","last_became_peered":"2024-08-17T18:18:33.761606+0000","last_unstale":"2024-08-17T18:18:33.762773+0000","last_undegraded":"2024-08-17T18:18:33.762773+0000","last_fullsized":"2024-08-17T18:18:33.762773+0000","mapping_epoch":1670,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1671,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:49.461698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:49.461698+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:29:01.698411+0000","scrub_duration":0.0013792520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":238,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.761993+0000","last_change":"2024-08-17T18:17:22.687498+0000","last_active":"2024-08-17T18:18:33.761993+0000","last_peered":"2024-08-17T18:18:33.761993+0000","last_clean":"2024-08-17T18:18:33.761993+0000","last_became_active":"2024-08-17T18:17:22.687176+0000","last_became_peered":"2024-08-17T18:17:22.687176+0000","last_unstale":"2024-08-17T18:18:33.761993+0000","last_undegraded":"2024-08-17T18:18:33.761993+0000","last_fullsized":"2024-08-17T18:18:33.761993+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:32:42.738739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2530,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.748591+0000","last_change":"2024-08-17T18:15:53.230320+0000","last_active":"2024-08-17T18:18:34.748591+0000","last_peered":"2024-08-17T18:18:34.748591+0000","last_clean":"2024-08-17T18:18:34.748591+0000","last_became_active":"2024-08-17T18:15:53.230016+0000","last_became_peered":"2024-08-17T18:15:53.230016+0000","last_unstale":"2024-08-17T18:18:34.748591+0000","last_undegraded":"2024-08-17T18:18:34.748591+0000","last_fullsized":"2024-08-17T18:18:34.748591+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:15:28.925103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":242,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.766313+0000","last_change":"2024-08-17T18:18:29.735362+0000","last_active":"2024-08-17T18:18:33.766313+0000","last_peered":"2024-08-17T18:18:33.766313+0000","last_clean":"2024-08-17T18:18:33.766313+0000","last_became_active":"2024-08-17T18:18:29.734752+0000","last_became_peered":"2024-08-17T18:18:29.734752+0000","last_unstale":"2024-08-17T18:18:33.766313+0000","last_undegraded":"2024-08-17T18:18:33.766313+0000","last_fullsized":"2024-08-17T18:18:33.766313+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:25.709698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"347'149","reported_seq":965,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.762492+0000","last_change":"2024-08-17T18:15:13.681264+0000","last_active":"2024-08-17T18:18:33.762492+0000","last_peered":"2024-08-17T18:18:33.762492+0000","last_clean":"2024-08-17T18:18:33.762492+0000","last_became_active":"2024-08-17T18:15:13.680918+0000","last_became_peered":"2024-08-17T18:15:13.680918+0000","last_unstale":"2024-08-17T18:18:33.762492+0000","last_undegraded":"2024-08-17T18:18:33.762492+0000","last_fullsized":"2024-08-17T18:18:33.762492+0000","mapping_epoch":1629,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1630,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:00.316884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:00.316884+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:37:32.924833+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1669,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.750095+0000","last_change":"2024-08-17T18:18:29.889561+0000","last_active":"2024-08-17T18:18:34.750095+0000","last_peered":"2024-08-17T18:18:34.750095+0000","last_clean":"2024-08-17T18:18:34.750095+0000","last_became_active":"2024-08-17T18:18:29.889192+0000","last_became_peered":"2024-08-17T18:18:29.889192+0000","last_unstale":"2024-08-17T18:18:34.750095+0000","last_undegraded":"2024-08-17T18:18:34.750095+0000","last_fullsized":"2024-08-17T18:18:34.750095+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:06:06.579862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1684,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.759055+0000","last_change":"2024-08-17T18:15:10.695565+0000","last_active":"2024-08-17T18:18:33.759055+0000","last_peered":"2024-08-17T18:18:33.759055+0000","last_clean":"2024-08-17T18:18:33.759055+0000","last_became_active":"2024-08-17T18:15:10.695225+0000","last_became_peered":"2024-08-17T18:15:10.695225+0000","last_unstale":"2024-08-17T18:18:33.759055+0000","last_undegraded":"2024-08-17T18:18:33.759055+0000","last_fullsized":"2024-08-17T18:18:33.759055+0000","mapping_epoch":1626,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:28:55.675702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":277,"reported_epoch":1671,"state":"active+recovery_wait+degraded","last_fresh":"2024-08-17T18:18:33.762207+0000","last_change":"2024-08-17T18:18:29.800821+0000","last_active":"2024-08-17T18:18:33.762207+0000","last_peered":"2024-08-17T18:18:33.762207+0000","last_clean":"2024-08-17T18:18:25.612782+0000","last_became_active":"2024-08-17T18:18:29.800502+0000","last_became_peered":"2024-08-17T18:18:29.800502+0000","last_unstale":"2024-08-17T18:18:33.762207+0000","last_undegraded":"2024-08-17T18:18:29.738218+0000","last_fullsized":"2024-08-17T18:18:33.762207+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:08:12.507444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":95,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":["5","2","6","7"],"object_location_counts":[{"shards":"2,5,6,7","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"347'149","reported_seq":824,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.399329+0000","last_change":"2024-08-17T18:17:22.681616+0000","last_active":"2024-08-17T18:18:34.399329+0000","last_peered":"2024-08-17T18:18:34.399329+0000","last_clean":"2024-08-17T18:18:34.399329+0000","last_became_active":"2024-08-17T18:17:22.681184+0000","last_became_peered":"2024-08-17T18:17:22.681184+0000","last_unstale":"2024-08-17T18:18:34.399329+0000","last_undegraded":"2024-08-17T18:18:34.399329+0000","last_fullsized":"2024-08-17T18:18:34.399329+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:02.363531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:02.363531+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:50:39.567707+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2434,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.761909+0000","last_change":"2024-08-17T18:18:29.898545+0000","last_active":"2024-08-17T18:18:33.761909+0000","last_peered":"2024-08-17T18:18:33.761909+0000","last_clean":"2024-08-17T18:18:33.761909+0000","last_became_active":"2024-08-17T18:18:29.896509+0000","last_became_peered":"2024-08-17T18:18:29.896509+0000","last_unstale":"2024-08-17T18:18:33.761909+0000","last_undegraded":"2024-08-17T18:18:33.761909+0000","last_fullsized":"2024-08-17T18:18:33.761909+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:37:09.107372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2342,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.749736+0000","last_change":"2024-08-17T18:12:04.648190+0000","last_active":"2024-08-17T18:18:34.749736+0000","last_peered":"2024-08-17T18:18:34.749736+0000","last_clean":"2024-08-17T18:18:34.749736+0000","last_became_active":"2024-08-17T18:12:04.647877+0000","last_became_peered":"2024-08-17T18:12:04.647877+0000","last_unstale":"2024-08-17T18:18:34.749736+0000","last_undegraded":"2024-08-17T18:18:34.749736+0000","last_fullsized":"2024-08-17T18:18:34.749736+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:18.129337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2644,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.750024+0000","last_change":"2024-08-17T18:18:29.742742+0000","last_active":"2024-08-17T18:18:34.750024+0000","last_peered":"2024-08-17T18:18:34.750024+0000","last_clean":"2024-08-17T18:18:34.750024+0000","last_became_active":"2024-08-17T18:18:29.742393+0000","last_became_peered":"2024-08-17T18:18:29.742393+0000","last_unstale":"2024-08-17T18:18:34.750024+0000","last_undegraded":"2024-08-17T18:18:34.750024+0000","last_fullsized":"2024-08-17T18:18:34.750024+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:57:32.858608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"347'149","reported_seq":691,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.730383+0000","last_change":"2024-08-17T18:18:32.735002+0000","last_active":"2024-08-17T18:18:33.730383+0000","last_peered":"2024-08-17T18:18:33.730383+0000","last_clean":"2024-08-17T18:18:33.730383+0000","last_became_active":"2024-08-17T18:18:32.734291+0000","last_became_peered":"2024-08-17T18:18:32.734291+0000","last_unstale":"2024-08-17T18:18:33.730383+0000","last_undegraded":"2024-08-17T18:18:33.730383+0000","last_fullsized":"2024-08-17T18:18:33.730383+0000","mapping_epoch":1669,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1670,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:12:53.586672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:12:53.586672+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":214680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":336,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.749619+0000","last_change":"2024-08-17T18:15:53.370263+0000","last_active":"2024-08-17T18:18:34.749619+0000","last_peered":"2024-08-17T18:18:34.749619+0000","last_clean":"2024-08-17T18:18:34.749619+0000","last_became_active":"2024-08-17T18:15:53.369971+0000","last_became_peered":"2024-08-17T18:15:53.369971+0000","last_unstale":"2024-08-17T18:18:34.749619+0000","last_undegraded":"2024-08-17T18:18:34.749619+0000","last_fullsized":"2024-08-17T18:18:34.749619+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:51:08.633886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":848,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.730501+0000","last_change":"2024-08-17T18:18:29.727602+0000","last_active":"2024-08-17T18:18:33.730501+0000","last_peered":"2024-08-17T18:18:33.730501+0000","last_clean":"2024-08-17T18:18:33.730501+0000","last_became_active":"2024-08-17T18:18:29.727190+0000","last_became_peered":"2024-08-17T18:18:29.727190+0000","last_unstale":"2024-08-17T18:18:33.730501+0000","last_undegraded":"2024-08-17T18:18:33.730501+0000","last_fullsized":"2024-08-17T18:18:33.730501+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:13:09.502199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.751794+0000","last_change":"2024-08-17T18:18:29.888513+0000","last_active":"2024-08-17T18:18:34.751794+0000","last_peered":"2024-08-17T18:18:34.751794+0000","last_clean":"2024-08-17T18:18:34.751794+0000","last_became_active":"2024-08-17T18:18:29.888065+0000","last_became_peered":"2024-08-17T18:18:29.888065+0000","last_unstale":"2024-08-17T18:18:34.751794+0000","last_undegraded":"2024-08-17T18:18:34.751794+0000","last_fullsized":"2024-08-17T18:18:34.751794+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:18.952476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1519,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.756013+0000","last_change":"2024-08-17T18:18:29.895726+0000","last_active":"2024-08-17T18:18:33.756013+0000","last_peered":"2024-08-17T18:18:33.756013+0000","last_clean":"2024-08-17T18:18:33.756013+0000","last_became_active":"2024-08-17T18:18:29.895383+0000","last_became_peered":"2024-08-17T18:18:29.895383+0000","last_unstale":"2024-08-17T18:18:33.756013+0000","last_undegraded":"2024-08-17T18:18:33.756013+0000","last_fullsized":"2024-08-17T18:18:33.756013+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:18:30.787712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.751763+0000","last_change":"2024-08-17T18:18:29.800143+0000","last_active":"2024-08-17T18:18:34.751763+0000","last_peered":"2024-08-17T18:18:34.751763+0000","last_clean":"2024-08-17T18:18:34.751763+0000","last_became_active":"2024-08-17T18:18:29.799896+0000","last_became_peered":"2024-08-17T18:18:29.799896+0000","last_unstale":"2024-08-17T18:18:34.751763+0000","last_undegraded":"2024-08-17T18:18:34.751763+0000","last_fullsized":"2024-08-17T18:18:34.751763+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:27:18.144391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":253,"reported_epoch":1671,"state":"active+recovery_wait","last_fresh":"2024-08-17T18:18:33.725085+0000","last_change":"2024-08-17T18:15:54.221085+0000","last_active":"2024-08-17T18:18:33.725085+0000","last_peered":"2024-08-17T18:18:33.725085+0000","last_clean":"2024-08-17T18:15:50.189470+0000","last_became_active":"2024-08-17T18:15:53.304106+0000","last_became_peered":"2024-08-17T18:15:53.304106+0000","last_unstale":"2024-08-17T18:18:33.725085+0000","last_undegraded":"2024-08-17T18:18:33.725085+0000","last_fullsized":"2024-08-17T18:18:33.725085+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:21.409368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":["5","2","6","7"],"object_location_counts":[{"shards":"2,5,7","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":851,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.756864+0000","last_change":"2024-08-17T18:17:16.835493+0000","last_active":"2024-08-17T18:18:33.756864+0000","last_peered":"2024-08-17T18:18:33.756864+0000","last_clean":"2024-08-17T18:18:33.756864+0000","last_became_active":"2024-08-17T18:15:55.698769+0000","last_became_peered":"2024-08-17T18:15:55.698769+0000","last_unstale":"2024-08-17T18:18:33.756864+0000","last_undegraded":"2024-08-17T18:18:33.756864+0000","last_fullsized":"2024-08-17T18:18:33.756864+0000","mapping_epoch":1640,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1641,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:54.303708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:54.303708+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":74,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"347'149","reported_seq":949,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.725143+0000","last_change":"2024-08-17T18:09:15.592722+0000","last_active":"2024-08-17T18:18:33.725143+0000","last_peered":"2024-08-17T18:18:33.725143+0000","last_clean":"2024-08-17T18:18:33.725143+0000","last_became_active":"2024-08-17T18:09:15.592242+0000","last_became_peered":"2024-08-17T18:09:15.592242+0000","last_unstale":"2024-08-17T18:18:33.725143+0000","last_undegraded":"2024-08-17T18:18:33.725143+0000","last_fullsized":"2024-08-17T18:18:33.725143+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:55.255983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:55.255983+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:02:53.596275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.753103+0000","last_change":"2024-08-17T18:18:29.895120+0000","last_active":"2024-08-17T18:18:34.753103+0000","last_peered":"2024-08-17T18:18:34.753103+0000","last_clean":"2024-08-17T18:18:34.753103+0000","last_became_active":"2024-08-17T18:18:29.894811+0000","last_became_peered":"2024-08-17T18:18:29.894811+0000","last_unstale":"2024-08-17T18:18:34.753103+0000","last_undegraded":"2024-08-17T18:18:34.753103+0000","last_fullsized":"2024-08-17T18:18:34.753103+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:44:28.402299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.751511+0000","last_change":"2024-08-17T18:18:29.900296+0000","last_active":"2024-08-17T18:18:34.751511+0000","last_peered":"2024-08-17T18:18:34.751511+0000","last_clean":"2024-08-17T18:18:34.751511+0000","last_became_active":"2024-08-17T18:18:29.899848+0000","last_became_peered":"2024-08-17T18:18:29.899848+0000","last_unstale":"2024-08-17T18:18:34.751511+0000","last_undegraded":"2024-08-17T18:18:34.751511+0000","last_fullsized":"2024-08-17T18:18:34.751511+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:09:15.545423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1708,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.759288+0000","last_change":"2024-08-17T18:18:29.882860+0000","last_active":"2024-08-17T18:18:33.759288+0000","last_peered":"2024-08-17T18:18:33.759288+0000","last_clean":"2024-08-17T18:18:33.759288+0000","last_became_active":"2024-08-17T18:18:29.882539+0000","last_became_peered":"2024-08-17T18:18:29.882539+0000","last_unstale":"2024-08-17T18:18:33.759288+0000","last_undegraded":"2024-08-17T18:18:33.759288+0000","last_fullsized":"2024-08-17T18:18:33.759288+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:12:52.056802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1698,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.757170+0000","last_change":"2024-08-17T18:15:53.244851+0000","last_active":"2024-08-17T18:18:33.757170+0000","last_peered":"2024-08-17T18:18:33.757170+0000","last_clean":"2024-08-17T18:18:33.757170+0000","last_became_active":"2024-08-17T18:15:53.240074+0000","last_became_peered":"2024-08-17T18:15:53.240074+0000","last_unstale":"2024-08-17T18:18:33.757170+0000","last_undegraded":"2024-08-17T18:18:33.757170+0000","last_fullsized":"2024-08-17T18:18:33.757170+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:09:00.168413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"347'149","reported_seq":844,"reported_epoch":1672,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:18:34.748794+0000","last_change":"2024-08-17T18:18:30.729274+0000","last_active":"2024-08-17T18:18:34.748794+0000","last_peered":"2024-08-17T18:18:34.748794+0000","last_clean":"2024-08-17T18:09:10.546471+0000","last_became_active":"2024-08-17T18:18:30.728913+0000","last_became_peered":"2024-08-17T18:18:30.728913+0000","last_unstale":"2024-08-17T18:18:34.748794+0000","last_undegraded":"2024-08-17T18:18:30.719341+0000","last_fullsized":"2024-08-17T18:18:30.718846+0000","mapping_epoch":1667,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1612,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:59.307356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:59.307356+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,4],"avail_no_missing":["2","5","6"],"object_location_counts":[{"shards":"2,5,6","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1693,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.755836+0000","last_change":"2024-08-17T18:12:04.648228+0000","last_active":"2024-08-17T18:18:33.755836+0000","last_peered":"2024-08-17T18:18:33.755836+0000","last_clean":"2024-08-17T18:18:33.755836+0000","last_became_active":"2024-08-17T18:12:04.647906+0000","last_became_peered":"2024-08-17T18:12:04.647906+0000","last_unstale":"2024-08-17T18:18:33.755836+0000","last_undegraded":"2024-08-17T18:18:33.755836+0000","last_fullsized":"2024-08-17T18:18:33.755836+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:51:30.081540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1680,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.726176+0000","last_change":"2024-08-17T18:18:29.728681+0000","last_active":"2024-08-17T18:18:33.726176+0000","last_peered":"2024-08-17T18:18:33.726176+0000","last_clean":"2024-08-17T18:18:33.726176+0000","last_became_active":"2024-08-17T18:18:29.728215+0000","last_became_peered":"2024-08-17T18:18:29.728215+0000","last_unstale":"2024-08-17T18:18:33.726176+0000","last_undegraded":"2024-08-17T18:18:33.726176+0000","last_fullsized":"2024-08-17T18:18:33.726176+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:48:41.152839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"347'149","reported_seq":925,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.758629+0000","last_change":"2024-08-17T18:18:33.757347+0000","last_active":"2024-08-17T18:18:33.758629+0000","last_peered":"2024-08-17T18:18:33.758629+0000","last_clean":"2024-08-17T18:18:33.758629+0000","last_became_active":"2024-08-17T18:18:33.756960+0000","last_became_peered":"2024-08-17T18:18:33.756960+0000","last_unstale":"2024-08-17T18:18:33.758629+0000","last_undegraded":"2024-08-17T18:18:33.758629+0000","last_fullsized":"2024-08-17T18:18:33.758629+0000","mapping_epoch":1670,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1671,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:52.271487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:52.271487+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:37:36.389244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1681,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.759852+0000","last_change":"2024-08-17T18:18:29.883479+0000","last_active":"2024-08-17T18:18:33.759852+0000","last_peered":"2024-08-17T18:18:33.759852+0000","last_clean":"2024-08-17T18:18:33.759852+0000","last_became_active":"2024-08-17T18:18:29.883185+0000","last_became_peered":"2024-08-17T18:18:29.883185+0000","last_unstale":"2024-08-17T18:18:33.759852+0000","last_undegraded":"2024-08-17T18:18:33.759852+0000","last_fullsized":"2024-08-17T18:18:33.759852+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:11:04.148316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":26,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.753604+0000","last_change":"2024-08-17T18:18:29.891624+0000","last_active":"2024-08-17T18:18:34.753604+0000","last_peered":"2024-08-17T18:18:34.753604+0000","last_clean":"2024-08-17T18:18:34.753604+0000","last_became_active":"2024-08-17T18:18:29.880707+0000","last_became_peered":"2024-08-17T18:18:29.880707+0000","last_unstale":"2024-08-17T18:18:34.753604+0000","last_undegraded":"2024-08-17T18:18:34.753604+0000","last_fullsized":"2024-08-17T18:18:34.753604+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:04.490121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":726,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.758626+0000","last_change":"2024-08-17T18:18:29.727626+0000","last_active":"2024-08-17T18:18:33.758626+0000","last_peered":"2024-08-17T18:18:33.758626+0000","last_clean":"2024-08-17T18:18:33.758626+0000","last_became_active":"2024-08-17T18:18:29.727331+0000","last_became_peered":"2024-08-17T18:18:29.727331+0000","last_unstale":"2024-08-17T18:18:33.758626+0000","last_undegraded":"2024-08-17T18:18:33.758626+0000","last_fullsized":"2024-08-17T18:18:33.758626+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:46:16.169400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":251,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.759583+0000","last_change":"2024-08-17T18:18:29.735189+0000","last_active":"2024-08-17T18:18:33.759583+0000","last_peered":"2024-08-17T18:18:33.759583+0000","last_clean":"2024-08-17T18:18:33.759583+0000","last_became_active":"2024-08-17T18:18:29.734681+0000","last_became_peered":"2024-08-17T18:18:29.734681+0000","last_unstale":"2024-08-17T18:18:33.759583+0000","last_undegraded":"2024-08-17T18:18:33.759583+0000","last_fullsized":"2024-08-17T18:18:33.759583+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:03:20.353696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1500,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.725232+0000","last_change":"2024-08-17T18:18:29.943292+0000","last_active":"2024-08-17T18:18:33.725232+0000","last_peered":"2024-08-17T18:18:33.725232+0000","last_clean":"2024-08-17T18:18:33.725232+0000","last_became_active":"2024-08-17T18:18:29.942947+0000","last_became_peered":"2024-08-17T18:18:29.942947+0000","last_unstale":"2024-08-17T18:18:33.725232+0000","last_undegraded":"2024-08-17T18:18:33.725232+0000","last_fullsized":"2024-08-17T18:18:33.725232+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:19:04.210562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":254,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.766472+0000","last_change":"2024-08-17T18:15:53.661488+0000","last_active":"2024-08-17T18:18:33.766472+0000","last_peered":"2024-08-17T18:18:33.766472+0000","last_clean":"2024-08-17T18:18:33.766472+0000","last_became_active":"2024-08-17T18:15:53.308611+0000","last_became_peered":"2024-08-17T18:15:53.308611+0000","last_unstale":"2024-08-17T18:18:33.766472+0000","last_undegraded":"2024-08-17T18:18:33.766472+0000","last_fullsized":"2024-08-17T18:18:33.766472+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:46:37.056632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1715,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.757248+0000","last_change":"2024-08-17T18:18:29.728301+0000","last_active":"2024-08-17T18:18:33.757248+0000","last_peered":"2024-08-17T18:18:33.757248+0000","last_clean":"2024-08-17T18:18:33.757248+0000","last_became_active":"2024-08-17T18:18:29.728019+0000","last_became_peered":"2024-08-17T18:18:29.728019+0000","last_unstale":"2024-08-17T18:18:33.757248+0000","last_undegraded":"2024-08-17T18:18:33.757248+0000","last_fullsized":"2024-08-17T18:18:33.757248+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:57:56.453165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":49,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.757160+0000","last_change":"2024-08-17T18:15:53.708176+0000","last_active":"2024-08-17T18:18:33.757160+0000","last_peered":"2024-08-17T18:18:33.757160+0000","last_clean":"2024-08-17T18:18:33.757160+0000","last_became_active":"2024-08-17T18:15:53.669563+0000","last_became_peered":"2024-08-17T18:15:53.669563+0000","last_unstale":"2024-08-17T18:18:33.757160+0000","last_undegraded":"2024-08-17T18:18:33.757160+0000","last_fullsized":"2024-08-17T18:18:33.757160+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:52:17.018687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":260,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.761381+0000","last_change":"2024-08-17T18:17:22.687310+0000","last_active":"2024-08-17T18:18:33.761381+0000","last_peered":"2024-08-17T18:18:33.761381+0000","last_clean":"2024-08-17T18:18:33.761381+0000","last_became_active":"2024-08-17T18:17:22.686966+0000","last_became_peered":"2024-08-17T18:17:22.686966+0000","last_unstale":"2024-08-17T18:18:33.761381+0000","last_undegraded":"2024-08-17T18:18:33.761381+0000","last_fullsized":"2024-08-17T18:18:33.761381+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:46:11.568465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":49,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.757905+0000","last_change":"2024-08-17T18:15:53.461774+0000","last_active":"2024-08-17T18:18:33.757905+0000","last_peered":"2024-08-17T18:18:33.757905+0000","last_clean":"2024-08-17T18:18:33.757905+0000","last_became_active":"2024-08-17T18:15:53.287744+0000","last_became_peered":"2024-08-17T18:15:53.287744+0000","last_unstale":"2024-08-17T18:18:33.757905+0000","last_undegraded":"2024-08-17T18:18:33.757905+0000","last_fullsized":"2024-08-17T18:18:33.757905+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:25.168664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1792,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.756786+0000","last_change":"2024-08-17T18:18:29.897922+0000","last_active":"2024-08-17T18:18:33.756786+0000","last_peered":"2024-08-17T18:18:33.756786+0000","last_clean":"2024-08-17T18:18:33.756786+0000","last_became_active":"2024-08-17T18:18:29.883879+0000","last_became_peered":"2024-08-17T18:18:29.883879+0000","last_unstale":"2024-08-17T18:18:33.756786+0000","last_undegraded":"2024-08-17T18:18:33.756786+0000","last_fullsized":"2024-08-17T18:18:33.756786+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:54:31.992298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.753291+0000","last_change":"2024-08-17T18:18:29.890043+0000","last_active":"2024-08-17T18:18:34.753291+0000","last_peered":"2024-08-17T18:18:34.753291+0000","last_clean":"2024-08-17T18:18:34.753291+0000","last_became_active":"2024-08-17T18:18:29.889687+0000","last_became_peered":"2024-08-17T18:18:29.889687+0000","last_unstale":"2024-08-17T18:18:34.753291+0000","last_undegraded":"2024-08-17T18:18:34.753291+0000","last_fullsized":"2024-08-17T18:18:34.753291+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:55:34.595507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":411,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.757027+0000","last_change":"2024-08-17T18:15:53.245247+0000","last_active":"2024-08-17T18:18:33.757027+0000","last_peered":"2024-08-17T18:18:33.757027+0000","last_clean":"2024-08-17T18:18:33.757027+0000","last_became_active":"2024-08-17T18:15:53.244627+0000","last_became_peered":"2024-08-17T18:15:53.244627+0000","last_unstale":"2024-08-17T18:18:33.757027+0000","last_undegraded":"2024-08-17T18:18:33.757027+0000","last_fullsized":"2024-08-17T18:18:33.757027+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:42:03.556013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.752790+0000","last_change":"2024-08-17T18:18:29.895804+0000","last_active":"2024-08-17T18:18:34.752790+0000","last_peered":"2024-08-17T18:18:34.752790+0000","last_clean":"2024-08-17T18:18:34.752790+0000","last_became_active":"2024-08-17T18:18:29.895352+0000","last_became_peered":"2024-08-17T18:18:29.895352+0000","last_unstale":"2024-08-17T18:18:34.752790+0000","last_undegraded":"2024-08-17T18:18:34.752790+0000","last_fullsized":"2024-08-17T18:18:34.752790+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:32:29.222569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.751945+0000","last_change":"2024-08-17T18:18:29.888110+0000","last_active":"2024-08-17T18:18:34.751945+0000","last_peered":"2024-08-17T18:18:34.751945+0000","last_clean":"2024-08-17T18:18:34.751945+0000","last_became_active":"2024-08-17T18:18:29.887787+0000","last_became_peered":"2024-08-17T18:18:29.887787+0000","last_unstale":"2024-08-17T18:18:34.751945+0000","last_undegraded":"2024-08-17T18:18:34.751945+0000","last_fullsized":"2024-08-17T18:18:34.751945+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:46:10.231616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":259,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.766386+0000","last_change":"2024-08-17T18:18:29.900128+0000","last_active":"2024-08-17T18:18:33.766386+0000","last_peered":"2024-08-17T18:18:33.766386+0000","last_clean":"2024-08-17T18:18:33.766386+0000","last_became_active":"2024-08-17T18:18:29.899279+0000","last_became_peered":"2024-08-17T18:18:29.899279+0000","last_unstale":"2024-08-17T18:18:33.766386+0000","last_undegraded":"2024-08-17T18:18:33.766386+0000","last_fullsized":"2024-08-17T18:18:33.766386+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:01:29.045498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":52,"reported_epoch":1671,"state":"active+clean","last_fresh":"2024-08-17T18:18:33.724428+0000","last_change":"2024-08-17T18:15:53.271113+0000","last_active":"2024-08-17T18:18:33.724428+0000","last_peered":"2024-08-17T18:18:33.724428+0000","last_clean":"2024-08-17T18:18:33.724428+0000","last_became_active":"2024-08-17T18:15:53.230983+0000","last_became_peered":"2024-08-17T18:15:53.230983+0000","last_unstale":"2024-08-17T18:18:33.724428+0000","last_undegraded":"2024-08-17T18:18:33.724428+0000","last_fullsized":"2024-08-17T18:18:33.724428+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:00:35.315798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":1672,"state":"active+clean","last_fresh":"2024-08-17T18:18:34.752160+0000","last_change":"2024-08-17T18:18:29.888785+0000","last_active":"2024-08-17T18:18:34.752160+0000","last_peered":"2024-08-17T18:18:34.752160+0000","last_clean":"2024-08-17T18:18:34.752160+0000","last_became_active":"2024-08-17T18:18:29.888475+0000","last_became_peered":"2024-08-17T18:18:29.888475+0000","last_unstale":"2024-08-17T18:18:34.752160+0000","last_undegraded":"2024-08-17T18:18:34.752160+0000","last_fullsized":"2024-08-17T18:18:34.752160+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:44:28.968751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":57,"num_pg":26,"stat_sum":{"num_bytes":3388276760,"num_objects":51702,"num_object_clones":0,"num_object_copies":155106,"num_objects_missing_on_primary":831,"num_objects_missing":831,"num_objects_degraded":31734,"num_objects_misplaced":40187,"num_objects_unfound":0,"num_objects_dirty":51702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180138,"num_write_kb":7418817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30748,"num_bytes_recovered":1846804480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12993507328,"data_stored":12993495112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134393,"ondisk_log_size":134393,"up":78,"acting":73,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":697707,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":149,"num_write":39,"num_write_kb":439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":810709,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":31,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":95,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":95,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":12532,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":22134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234833,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":998580,"kb_used_data":399852,"kb_used_omap":1,"kb_used_meta":598718,"kb_avail":92746572,"statfs":{"total":95995035648,"available":94972489728,"internally_reserved":0,"allocated":409448448,"data_stored":405080835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":613087691},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496471,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3867188,"kb_used_data":2630700,"kb_used_omap":1,"kb_used_meta":1236478,"kb_avail":89877964,"statfs":{"total":95995035648,"available":92035035136,"internally_reserved":0,"allocated":2693836800,"data_stored":2689470191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1266153927},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565712,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1068056,"kb_used_data":39120,"kb_used_omap":72,"kb_used_meta":1028855,"kb_avail":92677096,"statfs":{"total":95995035648,"available":94901346304,"internally_reserved":0,"allocated":40058880,"data_stored":35715050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":1053548014},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":1399,"seq":6008659247314,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3332604,"kb_used_data":2636852,"kb_used_omap":131,"kb_used_meta":695612,"kb_avail":90412548,"statfs":{"total":95995035648,"available":92582449152,"internally_reserved":0,"allocated":2700136448,"data_stored":2695761735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134291,"internal_metadata":712307565},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":2864743187102,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448741,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3837912,"kb_used_data":3052176,"kb_used_omap":75,"kb_used_meta":785652,"kb_avail":89907240,"statfs":{"total":95995035648,"available":92065013760,"internally_reserved":0,"allocated":3125428224,"data_stored":3121084328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":804507839},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208758106,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3340584,"kb_used_data":2050400,"kb_used_omap":1,"kb_used_meta":1290174,"kb_avail":90404568,"statfs":{"total":95995035648,"available":92574277632,"internally_reserved":0,"allocated":2099609600,"data_stored":2095250521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1321138630},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019743,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3097392,"kb_used_data":1979368,"kb_used_omap":1,"kb_used_meta":1118014,"kb_avail":90647760,"statfs":{"total":95995035648,"available":92823306240,"internally_reserved":0,"allocated":2026872832,"data_stored":2022509808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1144846790},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,1,0,0,0,3,2,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3110342656,"data_stored":3110338584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2679640064,"data_stored":2679635992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":30343168,"data_stored":30343168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2015363072,"data_stored":2015363072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2087649280,"data_stored":2087649280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2669875200,"data_stored":2669871128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":400293888,"data_stored":400293888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T18:18:50.828 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-08-17T18:18:50.905 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:50.937 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:50.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:50.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:50 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:18:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:50 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:18:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:50 smithi165 ceph-mon[105517]: osdmap e1673: 8 total, 8 up, 7 in 2024-08-17T18:18:51.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:50 smithi165 ceph-mon[105517]: from='client.46631 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:18:51.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:51.069 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:51.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:51.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:51.107 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:50 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:18:51.108 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:50 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:18:51.108 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:50 smithi136 ceph-mon[109861]: osdmap e1673: 8 total, 8 up, 7 in 2024-08-17T18:18:51.108 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:50 smithi136 ceph-mon[109861]: from='client.46631 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:18:51.108 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:50 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:18:51.108 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:50 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:18:51.108 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:50 smithi136 ceph-mon[115078]: osdmap e1673: 8 total, 8 up, 7 in 2024-08-17T18:18:51.109 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:50 smithi136 ceph-mon[115078]: from='client.46631 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:18:51.171 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:51.201 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:51.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:51.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:51.302 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:51.329 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:51.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:51.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:51.424 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-17T18:18:51.431 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:51.454 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:51.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:51.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:51.556 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:51.579 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:51.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:51.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:51.680 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:51.706 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:51.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:51.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:51.808 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:51.831 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:51.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:51.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:51.933 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:51.957 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:51.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:51.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:51 smithi165 ceph-mon[105517]: pgmap v3214: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 370 op/s; 31739/155184 objects degraded (20.452%); 40187/155184 objects misplaced (25.896%); 13 MiB/s, 208 objects/s recovering 2024-08-17T18:18:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:18:52.059 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:52.077 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:18:52.085 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:52.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:52.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:52.156 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:51 smithi136 ceph-mon[109861]: pgmap v3214: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 370 op/s; 31739/155184 objects degraded (20.452%); 40187/155184 objects misplaced (25.896%); 13 MiB/s, 208 objects/s recovering 2024-08-17T18:18:52.157 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:52.157 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:18:52.157 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:51 smithi136 ceph-mon[115078]: pgmap v3214: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 370 op/s; 31739/155184 objects degraded (20.452%); 40187/155184 objects misplaced (25.896%); 13 MiB/s, 208 objects/s recovering 2024-08-17T18:18:52.157 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:52.157 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:18:52.187 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:52.210 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:52.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:52.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:52.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:52.353 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:52.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:52.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:52.455 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:52.478 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:52.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:52.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:52.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:52.604 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:52.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:52.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:52.706 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:52.729 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:52.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:52.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:52.831 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:52.853 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:52.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:52.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:52.955 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:52.976 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:52.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:52.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:53.078 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:53.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:53.103 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:53.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:53.204 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:53.242 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:53.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:53.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:53 smithi165 ceph-mon[105517]: pgmap v3215: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 410 op/s; 30906/149637 objects degraded (20.654%); 38696/149637 objects misplaced (25.860%); 13 MiB/s, 227 objects/s recovering 2024-08-17T18:18:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:18:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:18:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:53.320 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:53.320 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:53.320 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:53 smithi136 ceph-mon[109861]: pgmap v3215: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 410 op/s; 30906/149637 objects degraded (20.654%); 38696/149637 objects misplaced (25.860%); 13 MiB/s, 227 objects/s recovering 2024-08-17T18:18:53.320 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:18:53.320 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:18:53.320 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:53.320 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:53.320 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:53.321 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:53 smithi136 ceph-mon[115078]: pgmap v3215: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 410 op/s; 30906/149637 objects degraded (20.654%); 38696/149637 objects misplaced (25.860%); 13 MiB/s, 227 objects/s recovering 2024-08-17T18:18:53.321 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:18:53.321 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:18:53.321 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:18:53.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:53.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:53.378 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:53.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:53.379 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:18:53.379 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:18:53.380 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:18:53.380 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:18:53.380 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:18:53.380 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:18:53.380 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:18:53.380 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:18:53.399 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:18:53.400 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:18:53.423 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:18:53.423 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:18:53.451 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:18:53.452 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:18:53.452 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:18:53.472 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:18:53.473 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:18:53.479 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:53.504 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:53.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:53.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:53.606 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:53.630 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:53.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:53.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:53.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:53.757 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:53.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:53.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:53.859 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:53.883 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:53.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:53.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:53.985 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:54.007 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:54.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:54.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:54.085 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:18:54.085 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:18:54.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:54.146 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:54.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:54.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:54.182 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:18:54.248 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:54.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:54 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/614366681' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:18:54.280 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:54.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:54.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:54.308 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:54 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/614366681' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:18:54.308 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:54 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/614366681' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:18:54.381 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:54.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:54.414 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:54.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:54.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:54.538 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:54.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:54.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:54.640 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:54.661 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:54.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:54.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:54.763 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:54.787 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:54.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:54.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:54.843 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-17T18:18:54.889 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:54.916 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:54.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:54.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:55.018 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:55.042 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:55.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:55.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:55.131 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:18:55.143 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:55.167 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:55.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:55.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:55 smithi165 ceph-mon[105517]: pgmap v3216: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 410 op/s; 30906/149637 objects degraded (20.654%); 38696/149637 objects misplaced (25.860%); 13 MiB/s, 227 objects/s recovering 2024-08-17T18:18:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:55 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 30906/149637 objects degraded (20.654%), 16 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:55.269 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:55.291 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:55.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:55.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:55.393 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:55.418 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:55.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:55.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:18:55.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:55 smithi136 ceph-mon[109861]: pgmap v3216: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 410 op/s; 30906/149637 objects degraded (20.654%); 38696/149637 objects misplaced (25.860%); 13 MiB/s, 227 objects/s recovering 2024-08-17T18:18:55.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:55 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 30906/149637 objects degraded (20.654%), 16 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:55.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:55 smithi136 ceph-mon[115078]: pgmap v3216: 109 pgs: 1 active+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 410 op/s; 30906/149637 objects degraded (20.654%); 38696/149637 objects misplaced (25.860%); 13 MiB/s, 227 objects/s recovering 2024-08-17T18:18:55.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:55 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 30906/149637 objects degraded (20.654%), 16 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:18:55.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:55.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:55.556 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:55.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:55.657 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:55.694 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:55.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:55.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:55.796 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:55.822 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:55.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:55.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:55.871 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:18:55.923 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:55.956 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:55.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:55.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:56.058 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:56.091 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:56.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:56.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:56.193 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:56.228 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:56.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:56.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:56.329 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:56.374 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:56.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:56.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:56.477 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:56.505 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:56.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:56.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:56.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:56.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:56.646 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:56.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:56.747 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:56.772 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:56.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:56.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:56.874 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:56.896 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:56.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:56.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:56.998 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:57.030 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:57.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:57.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:57.132 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:57.153 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:57.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:57.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:57.255 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:57.279 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:57.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:57.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:57.381 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:57.404 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:57.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:57.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:57.505 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:57.538 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:57.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:57.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:57.640 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:57.642 INFO:teuthology.orchestra.run.smithi136.stderr:set pool 1 pgp_num to 11 2024-08-17T18:18:57.680 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:57.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:57.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:57.782 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:57.807 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:57.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:57.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:57.838 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:57 smithi165 ceph-mon[105517]: pgmap v3217: 109 pgs: 1 active+recovery_wait+remapped, 1 active+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 416 op/s; 30767/147903 objects degraded (20.802%); 38070/147903 objects misplaced (25.740%); 13 MiB/s, 228 objects/s recovering 2024-08-17T18:18:57.838 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:57 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/148413495' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T18:18:57.838 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:57 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T18:18:57.862 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:18:57.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:57 smithi136 ceph-mon[115078]: pgmap v3217: 109 pgs: 1 active+recovery_wait+remapped, 1 active+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 416 op/s; 30767/147903 objects degraded (20.802%); 38070/147903 objects misplaced (25.740%); 13 MiB/s, 228 objects/s recovering 2024-08-17T18:18:57.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:57 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/148413495' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T18:18:57.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:57 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T18:18:57.888 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:57 smithi136 ceph-mon[109861]: pgmap v3217: 109 pgs: 1 active+recovery_wait+remapped, 1 active+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 3.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 416 op/s; 30767/147903 objects degraded (20.802%); 38070/147903 objects misplaced (25.740%); 13 MiB/s, 228 objects/s recovering 2024-08-17T18:18:57.888 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:57 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/148413495' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T18:18:57.888 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:57 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T18:18:57.889 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:18:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:18:57] "GET /metrics HTTP/1.1" 200 36777 "" "Prometheus/2.51.0" 2024-08-17T18:18:57.909 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:57.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:57.960 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:57.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:58.061 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:58.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:58.105 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:58.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:58.206 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:58.240 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:58.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:58.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:58.342 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:58.370 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:58.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:58.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:58.472 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:58.505 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:58.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:58.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:58.567 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:18:58.567 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:18:58.608 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:58.631 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:58.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:58.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:58.732 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:58.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:58 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-08-17T18:18:58.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:58 smithi165 ceph-mon[105517]: osdmap e1674: 8 total, 8 up, 7 in 2024-08-17T18:18:58.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:58.771 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:58.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:58.874 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:58.896 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:58.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:58.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:58 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-08-17T18:18:58.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:58 smithi136 ceph-mon[109861]: osdmap e1674: 8 total, 8 up, 7 in 2024-08-17T18:18:58.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:58 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-08-17T18:18:58.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:58 smithi136 ceph-mon[115078]: osdmap e1674: 8 total, 8 up, 7 in 2024-08-17T18:18:58.997 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:18:59.020 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:59.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:59.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:18:59.121 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:18:59.144 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:59.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:59.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:18:59.246 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:18:59.271 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-17T18:18:59.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:59.285 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:59.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:18:59.387 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:18:59.409 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:59.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:59.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:18:59.470 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:18:59.511 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:18:59.545 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:59.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:59.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:18:59.628 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:18:59.647 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:18:59.686 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:59.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:59.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:18:59.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:18:59 smithi136 ceph-mon[109861]: pgmap v3219: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 1 active+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 2.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 454 op/s; 29409/143298 objects degraded (20.523%); 37199/143298 objects misplaced (25.959%); 12 MiB/s, 285 objects/s recovering 2024-08-17T18:18:59.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:18:59 smithi136 ceph-mon[115078]: pgmap v3219: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 1 active+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 2.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 454 op/s; 29409/143298 objects degraded (20.523%); 37199/143298 objects misplaced (25.959%); 12 MiB/s, 285 objects/s recovering 2024-08-17T18:18:59.788 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:18:59.823 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:18:59.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:59.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:18:59.925 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:18:59.932 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:18:59 smithi165 ceph-mon[105517]: pgmap v3219: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 1 active+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 2.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 454 op/s; 29409/143298 objects degraded (20.523%); 37199/143298 objects misplaced (25.959%); 12 MiB/s, 285 objects/s recovering 2024-08-17T18:18:59.950 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:18:59.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:18:59.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:00.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:00.087 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:00.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:00.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:00.189 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:00.226 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:00.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:00.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:00.328 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:00.357 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:00.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:00.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:00.459 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:00.483 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:00.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:00.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:00.559 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:19:00.585 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:00.609 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:00.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:00.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:00.712 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:00.740 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:00.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:00.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:00.842 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:00.857 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:00 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3307011797' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:19:00.857 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:00 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 29409/143298 objects degraded (20.523%), 16 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:00.857 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:00 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3307011797' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:19:00.857 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:00 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 29409/143298 objects degraded (20.523%), 16 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:00.867 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:00.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:00.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:00.960 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:00 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3307011797' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:19:00.961 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:00 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 29409/143298 objects degraded (20.523%), 16 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:00.969 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:01.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:01.009 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:01.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:01.110 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:01.141 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:01.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:01.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:01.243 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:01.265 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:01.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:01.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:01.367 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:01.391 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:01.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:01.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:01.493 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:01.518 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:01.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:01.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:01.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:01.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:01.643 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:01.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:01.744 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:01.777 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:01.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:01.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:01.812 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:01 smithi136 ceph-mon[109861]: pgmap v3220: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 2.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 434 op/s; 29269/142149 objects degraded (20.590%); 36887/142149 objects misplaced (25.950%); 11 MiB/s, 255 objects/s recovering 2024-08-17T18:19:01.812 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:01 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:19:01.812 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:01 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3307011797' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:19:01.812 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:01 smithi136 ceph-mon[109861]: osdmap e1675: 8 total, 8 up, 7 in 2024-08-17T18:19:01.812 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:01 smithi136 ceph-mon[115078]: pgmap v3220: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 2.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 434 op/s; 29269/142149 objects degraded (20.590%); 36887/142149 objects misplaced (25.950%); 11 MiB/s, 255 objects/s recovering 2024-08-17T18:19:01.812 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:01 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:19:01.813 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:01 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3307011797' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:19:01.813 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:01 smithi136 ceph-mon[115078]: osdmap e1675: 8 total, 8 up, 7 in 2024-08-17T18:19:01.879 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:01.905 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:01.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:01.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:02.007 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:02.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:01 smithi165 ceph-mon[105517]: pgmap v3220: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait; 2.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 434 op/s; 29269/142149 objects degraded (20.590%); 36887/142149 objects misplaced (25.950%); 11 MiB/s, 255 objects/s recovering 2024-08-17T18:19:02.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:01 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:19:02.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:01 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3307011797' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:19:02.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:01 smithi165 ceph-mon[105517]: osdmap e1675: 8 total, 8 up, 7 in 2024-08-17T18:19:02.035 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:02.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:02.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:02.075 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:19:02.076 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:19:02.093 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:19:02.093 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:19:02.110 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:19:02.110 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:19:02.129 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:19:02.129 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:19:02.137 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:02.148 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:19:02.148 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:19:02.163 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:02.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:02.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:02.167 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:19:02.167 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:19:02.194 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:19:02.194 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:19:02.218 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:19:02.218 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:19:02.265 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:02.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:02.291 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:02.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:02.392 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:02.434 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:02.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:02.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:02.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:02.574 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:02.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:02.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:02.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:02 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3444691727' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:19:02.676 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:02.700 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:02.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:02.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:02.802 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:02.842 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:02.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:02.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:02.944 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:02.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:02 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3444691727' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:19:02.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:02 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3444691727' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:19:02.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:02.983 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:02.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:03.084 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:03.108 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:03.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:03.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:03.210 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:03.235 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:19:03.235 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T18:19:03.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T18:19:03.235 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:19:03.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.236 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:178724 56.3 56:ca256064:::benchmark_data_smithi136_356456_object52020:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:19:03.236 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:03.715635+0000", 2024-08-17T18:19:03.236 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 418.80388855000001, 2024-08-17T18:19:03.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.159103827, 2024-08-17T18:19:03.236 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:19:03.236 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:03.236 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:19:03.236 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:19:03.236 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:19:03.236 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 178724 2024-08-17T18:19:03.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.237 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:19:03.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:19:03.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.715635+0000", 2024-08-17T18:19:03.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:19:03.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.715635+0000", 2024-08-17T18:19:03.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:19:03.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.715634+0000", 2024-08-17T18:19:03.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:19:03.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:19:03.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.715640+0000", 2024-08-17T18:19:03.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8660000000000002e-06 2024-08-17T18:19:03.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:19:03.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.715651+0000", 2024-08-17T18:19:03.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1202e-05 2024-08-17T18:19:03.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:19:03.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.715656+0000", 2024-08-17T18:19:03.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3210000000000004e-06 2024-08-17T18:19:03.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.715720+0000", 2024-08-17T18:19:03.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3246999999999999e-05 2024-08-17T18:19:03.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:19:03.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.715729+0000", 2024-08-17T18:19:03.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1889999999999996e-06 2024-08-17T18:19:03.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.631688+0000", 2024-08-17T18:19:03.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91595881700000004 2024-08-17T18:19:03.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:19:03.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.631698+0000", 2024-08-17T18:19:03.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0329000000000001e-05 2024-08-17T18:19:03.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.833971+0000", 2024-08-17T18:19:03.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2022729590000001 2024-08-17T18:19:03.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:19:03.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.838163+0000", 2024-08-17T18:19:03.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004192622 2024-08-17T18:19:03.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4,5", 2024-08-17T18:19:03.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.838431+0000", 2024-08-17T18:19:03.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026806399999999999 2024-08-17T18:19:03.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:19:03.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.852710+0000", 2024-08-17T18:19:03.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014278661999999999 2024-08-17T18:19:03.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.852753+0000", 2024-08-17T18:19:03.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2372e-05 2024-08-17T18:19:03.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.863145+0000", 2024-08-17T18:19:03.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01039202 2024-08-17T18:19:03.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.874572+0000", 2024-08-17T18:19:03.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011427165 2024-08-17T18:19:03.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:19:03.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.874618+0000", 2024-08-17T18:19:03.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6165000000000003e-05 2024-08-17T18:19:03.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:19:03.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.874739+0000", 2024-08-17T18:19:03.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121204 2024-08-17T18:19:03.244 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.244 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:19:03.244 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.244 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:178726 56.3 56:c3df7639:::benchmark_data_smithi136_356456_object52022:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:19:03.244 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:03.719811+0000", 2024-08-17T18:19:03.244 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 418.79971269200001, 2024-08-17T18:19:03.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1598328310000001, 2024-08-17T18:19:03.244 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:19:03.245 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:03.245 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:19:03.245 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:19:03.245 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:19:03.245 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 178726 2024-08-17T18:19:03.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.245 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:19:03.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:19:03.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.719811+0000", 2024-08-17T18:19:03.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:19:03.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.719811+0000", 2024-08-17T18:19:03.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:19:03.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.719810+0000", 2024-08-17T18:19:03.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:19:03.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:19:03.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.719814+0000", 2024-08-17T18:19:03.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7859999999999998e-06 2024-08-17T18:19:03.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:19:03.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.719822+0000", 2024-08-17T18:19:03.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4120000000000001e-06 2024-08-17T18:19:03.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:19:03.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.719827+0000", 2024-08-17T18:19:03.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.301e-06 2024-08-17T18:19:03.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.719865+0000", 2024-08-17T18:19:03.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8177999999999999e-05 2024-08-17T18:19:03.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:19:03.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.719873+0000", 2024-08-17T18:19:03.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7999999999999999e-06 2024-08-17T18:19:03.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.631721+0000", 2024-08-17T18:19:03.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91184803199999998 2024-08-17T18:19:03.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:19:03.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.631729+0000", 2024-08-17T18:19:03.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0609999999999996e-06 2024-08-17T18:19:03.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.846202+0000", 2024-08-17T18:19:03.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2144735609999999 2024-08-17T18:19:03.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:19:03.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.846318+0000", 2024-08-17T18:19:03.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116112 2024-08-17T18:19:03.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4,5", 2024-08-17T18:19:03.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.846442+0000", 2024-08-17T18:19:03.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012396400000000001 2024-08-17T18:19:03.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:19:03.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.852716+0000", 2024-08-17T18:19:03.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062739409999999999 2024-08-17T18:19:03.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.852790+0000", 2024-08-17T18:19:03.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4035000000000005e-05 2024-08-17T18:19:03.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.863179+0000", 2024-08-17T18:19:03.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010388313999999999 2024-08-17T18:19:03.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.879424+0000", 2024-08-17T18:19:03.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016245349999999999 2024-08-17T18:19:03.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:19:03.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.879524+0000", 2024-08-17T18:19:03.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9796999999999993e-05 2024-08-17T18:19:03.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:19:03.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.879644+0000", 2024-08-17T18:19:03.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000120177 2024-08-17T18:19:03.252 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.252 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:19:03.252 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.252 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:178755 56.3 56:c40830a4:::benchmark_data_smithi136_356456_object52051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:19:03.253 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:03.760914+0000", 2024-08-17T18:19:03.253 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 418.758609589, 2024-08-17T18:19:03.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.119016674, 2024-08-17T18:19:03.253 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:19:03.253 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:03.253 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:19:03.253 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:19:03.253 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:19:03.253 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 178755 2024-08-17T18:19:03.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.253 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:19:03.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:19:03.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.760914+0000", 2024-08-17T18:19:03.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:19:03.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.760914+0000", 2024-08-17T18:19:03.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:19:03.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.760912+0000", 2024-08-17T18:19:03.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:19:03.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:19:03.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.760921+0000", 2024-08-17T18:19:03.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8109999999999992e-06 2024-08-17T18:19:03.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:19:03.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.760940+0000", 2024-08-17T18:19:03.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9066999999999999e-05 2024-08-17T18:19:03.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:19:03.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.760950+0000", 2024-08-17T18:19:03.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3309999999999993e-06 2024-08-17T18:19:03.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.761001+0000", 2024-08-17T18:19:03.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1393000000000001e-05 2024-08-17T18:19:03.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:19:03.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:03.761009+0000", 2024-08-17T18:19:03.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9480000000000008e-06 2024-08-17T18:19:03.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.631751+0000", 2024-08-17T18:19:03.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.87074218199999998 2024-08-17T18:19:03.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:19:03.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:04.631759+0000", 2024-08-17T18:19:03.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6590000000000007e-06 2024-08-17T18:19:03.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.849473+0000", 2024-08-17T18:19:03.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2177139050000001 2024-08-17T18:19:03.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:19:03.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.849597+0000", 2024-08-17T18:19:03.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000124301 2024-08-17T18:19:03.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4,5", 2024-08-17T18:19:03.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.849734+0000", 2024-08-17T18:19:03.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013656000000000001 2024-08-17T18:19:03.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:19:03.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.854014+0000", 2024-08-17T18:19:03.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042802450000000002 2024-08-17T18:19:03.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.861075+0000", 2024-08-17T18:19:03.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070614409999999999 2024-08-17T18:19:03.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.863207+0000", 2024-08-17T18:19:03.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002131558 2024-08-17T18:19:03.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.879687+0000", 2024-08-17T18:19:03.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01648057 2024-08-17T18:19:03.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:19:03.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.879823+0000", 2024-08-17T18:19:03.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013595 2024-08-17T18:19:03.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:19:03.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:05.879931+0000", 2024-08-17T18:19:03.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000107506 2024-08-17T18:19:03.260 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.260 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:208217 56.13 56:cf7b1a31:::benchmark_data_smithi136_356456_object81513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1571)", 2024-08-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:41.681475+0000", 2024-08-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 380.83804871900003, 2024-08-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99705307600000004, 2024-08-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 208217 2024-08-17T18:19:03.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.262 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:19:03.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:19:03.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.681475+0000", 2024-08-17T18:19:03.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:19:03.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.681475+0000", 2024-08-17T18:19:03.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:19:03.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.681474+0000", 2024-08-17T18:19:03.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:19:03.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:19:03.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.681480+0000", 2024-08-17T18:19:03.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5980000000000002e-06 2024-08-17T18:19:03.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.263 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:19:03.263 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.681497+0000", 2024-08-17T18:19:03.263 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7252e-05 2024-08-17T18:19:03.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:19:03.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.681505+0000", 2024-08-17T18:19:03.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6000000000000001e-06 2024-08-17T18:19:03.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.681562+0000", 2024-08-17T18:19:03.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7082e-05 2024-08-17T18:19:03.264 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.264 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.264 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:19:03.264 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.681575+0000", 2024-08-17T18:19:03.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2408e-05 2024-08-17T18:19:03.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.670403+0000", 2024-08-17T18:19:03.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.988828296 2024-08-17T18:19:03.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:19:03.265 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.670541+0000", 2024-08-17T18:19:03.265 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013815999999999999 2024-08-17T18:19:03.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.265 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:19:03.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.670876+0000", 2024-08-17T18:19:03.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000334968 2024-08-17T18:19:03.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.674948+0000", 2024-08-17T18:19:03.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040718150000000003 2024-08-17T18:19:03.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.674980+0000", 2024-08-17T18:19:03.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2147999999999998e-05 2024-08-17T18:19:03.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:19:03.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.678455+0000", 2024-08-17T18:19:03.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034748750000000001 2024-08-17T18:19:03.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:19:03.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.678488+0000", 2024-08-17T18:19:03.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3096999999999998e-05 2024-08-17T18:19:03.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.678528+0000", 2024-08-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0237000000000001e-05 2024-08-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:208226 56.13 56:ccae28b7:::benchmark_data_smithi136_356456_object81522:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1571)", 2024-08-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:41.690277+0000", 2024-08-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 380.829246732, 2024-08-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98844818999999995, 2024-08-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:03.268 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:19:03.269 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:19:03.269 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:19:03.269 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 208226 2024-08-17T18:19:03.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.269 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:19:03.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:19:03.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.690277+0000", 2024-08-17T18:19:03.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:19:03.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.690277+0000", 2024-08-17T18:19:03.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:19:03.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.690275+0000", 2024-08-17T18:19:03.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:19:03.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:19:03.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.690282+0000", 2024-08-17T18:19:03.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7499999999999997e-06 2024-08-17T18:19:03.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:19:03.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.690300+0000", 2024-08-17T18:19:03.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7578999999999999e-05 2024-08-17T18:19:03.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:19:03.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.690307+0000", 2024-08-17T18:19:03.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6429999999999995e-06 2024-08-17T18:19:03.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.690371+0000", 2024-08-17T18:19:03.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4102000000000004e-05 2024-08-17T18:19:03.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:19:03.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.690385+0000", 2024-08-17T18:19:03.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3538e-05 2024-08-17T18:19:03.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.673059+0000", 2024-08-17T18:19:03.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98267378900000002 2024-08-17T18:19:03.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:19:03.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.673184+0000", 2024-08-17T18:19:03.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000125425 2024-08-17T18:19:03.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:19:03.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.673492+0000", 2024-08-17T18:19:03.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000308086 2024-08-17T18:19:03.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.678447+0000", 2024-08-17T18:19:03.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049545279999999997 2024-08-17T18:19:03.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:19:03.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.678610+0000", 2024-08-17T18:19:03.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016375 2024-08-17T18:19:03.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.678645+0000", 2024-08-17T18:19:03.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4907000000000003e-05 2024-08-17T18:19:03.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:19:03.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.678685+0000", 2024-08-17T18:19:03.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0030999999999997e-05 2024-08-17T18:19:03.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:19:03.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.678725+0000", 2024-08-17T18:19:03.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9926999999999999e-05 2024-08-17T18:19:03.275 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.275 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:19:03.275 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.276 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:208233 56.13 56:ca8b6cab:::benchmark_data_smithi136_356456_object81529:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1571)", 2024-08-17T18:19:03.276 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:41.696234+0000", 2024-08-17T18:19:03.276 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 380.82328999399999, 2024-08-17T18:19:03.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98632644800000002, 2024-08-17T18:19:03.276 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:19:03.276 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:03.276 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:19:03.276 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:19:03.276 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:19:03.276 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 208233 2024-08-17T18:19:03.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.276 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:19:03.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:19:03.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.696234+0000", 2024-08-17T18:19:03.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:19:03.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.696234+0000", 2024-08-17T18:19:03.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:19:03.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.696232+0000", 2024-08-17T18:19:03.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:19:03.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:19:03.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.696239+0000", 2024-08-17T18:19:03.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0449999999999997e-06 2024-08-17T18:19:03.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:19:03.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.696250+0000", 2024-08-17T18:19:03.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.146e-05 2024-08-17T18:19:03.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:19:03.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.696256+0000", 2024-08-17T18:19:03.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.224e-06 2024-08-17T18:19:03.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.696317+0000", 2024-08-17T18:19:03.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0906999999999998e-05 2024-08-17T18:19:03.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:19:03.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.696331+0000", 2024-08-17T18:19:03.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3709999999999999e-05 2024-08-17T18:19:03.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.675000+0000", 2024-08-17T18:19:03.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97866883000000005 2024-08-17T18:19:03.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:19:03.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.675109+0000", 2024-08-17T18:19:03.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010919 2024-08-17T18:19:03.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:19:03.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.675362+0000", 2024-08-17T18:19:03.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025293599999999999 2024-08-17T18:19:03.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:19:03.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.678739+0000", 2024-08-17T18:19:03.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003376815 2024-08-17T18:19:03.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.678779+0000", 2024-08-17T18:19:03.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0108999999999998e-05 2024-08-17T18:19:03.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.682477+0000", 2024-08-17T18:19:03.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036984460000000002 2024-08-17T18:19:03.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:19:03.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.682522+0000", 2024-08-17T18:19:03.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4604999999999997e-05 2024-08-17T18:19:03.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:19:03.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.682560+0000", 2024-08-17T18:19:03.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8519000000000002e-05 2024-08-17T18:19:03.283 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.283 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:19:03.283 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.283 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:208236 56.13 56:c8f0ef61:::benchmark_data_smithi136_356456_object81532:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1571)", 2024-08-17T18:19:03.283 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:41.699376+0000", 2024-08-17T18:19:03.283 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 380.82014756400002, 2024-08-17T18:19:03.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98577144999999999, 2024-08-17T18:19:03.283 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:19:03.283 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:03.283 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:19:03.283 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:19:03.284 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:19:03.284 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 208236 2024-08-17T18:19:03.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.284 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:19:03.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:19:03.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.699376+0000", 2024-08-17T18:19:03.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:19:03.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.699376+0000", 2024-08-17T18:19:03.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:19:03.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.699375+0000", 2024-08-17T18:19:03.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:19:03.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:19:03.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.699382+0000", 2024-08-17T18:19:03.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4989999999999997e-06 2024-08-17T18:19:03.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:19:03.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.699397+0000", 2024-08-17T18:19:03.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5160999999999999e-05 2024-08-17T18:19:03.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:19:03.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.699405+0000", 2024-08-17T18:19:03.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6620000000000005e-06 2024-08-17T18:19:03.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.699475+0000", 2024-08-17T18:19:03.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0176e-05 2024-08-17T18:19:03.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:19:03.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.699488+0000", 2024-08-17T18:19:03.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2850999999999999e-05 2024-08-17T18:19:03.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.676643+0000", 2024-08-17T18:19:03.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97715538000000002 2024-08-17T18:19:03.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:19:03.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.676752+0000", 2024-08-17T18:19:03.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108952 2024-08-17T18:19:03.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.288 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:19:03.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.676969+0000", 2024-08-17T18:19:03.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000216768 2024-08-17T18:19:03.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:19:03.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.678743+0000", 2024-08-17T18:19:03.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017739730000000001 2024-08-17T18:19:03.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.682909+0000", 2024-08-17T18:19:03.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041658679999999997 2024-08-17T18:19:03.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.685059+0000", 2024-08-17T18:19:03.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021496129999999999 2024-08-17T18:19:03.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:19:03.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.685109+0000", 2024-08-17T18:19:03.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0464000000000003e-05 2024-08-17T18:19:03.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:19:03.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.685148+0000", 2024-08-17T18:19:03.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8634999999999999e-05 2024-08-17T18:19:03.290 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.290 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:19:03.291 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.291 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:208257 56.13 56:cca897c2:::benchmark_data_smithi136_356456_object81553:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1571)", 2024-08-17T18:19:03.291 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:41.722159+0000", 2024-08-17T18:19:03.291 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 380.79736523600002, 2024-08-17T18:19:03.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.96511230400000003, 2024-08-17T18:19:03.291 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:19:03.291 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:03.291 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:19:03.291 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:19:03.291 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:19:03.291 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 208257 2024-08-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.722159+0000", 2024-08-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.722159+0000", 2024-08-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:19:03.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.722157+0000", 2024-08-17T18:19:03.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:19:03.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:19:03.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.722164+0000", 2024-08-17T18:19:03.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0790000000000003e-06 2024-08-17T18:19:03.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:19:03.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.722178+0000", 2024-08-17T18:19:03.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4148000000000001e-05 2024-08-17T18:19:03.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:19:03.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.722186+0000", 2024-08-17T18:19:03.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7360000000000001e-06 2024-08-17T18:19:03.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.722241+0000", 2024-08-17T18:19:03.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5405000000000002e-05 2024-08-17T18:19:03.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:19:03.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:41.722254+0000", 2024-08-17T18:19:03.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2765999999999999e-05 2024-08-17T18:19:03.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.678804+0000", 2024-08-17T18:19:03.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.95654943100000001 2024-08-17T18:19:03.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:19:03.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.678928+0000", 2024-08-17T18:19:03.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000124499 2024-08-17T18:19:03.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:19:03.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.679213+0000", 2024-08-17T18:19:03.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000285376 2024-08-17T18:19:03.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:19:03.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.682574+0000", 2024-08-17T18:19:03.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033602749999999998 2024-08-17T18:19:03.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.685185+0000", 2024-08-17T18:19:03.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00261123 2024-08-17T18:19:03.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.687196+0000", 2024-08-17T18:19:03.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020108750000000001 2024-08-17T18:19:03.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:19:03.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.687235+0000", 2024-08-17T18:19:03.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9446999999999997e-05 2024-08-17T18:19:03.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:19:03.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.687271+0000", 2024-08-17T18:19:03.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5672e-05 2024-08-17T18:19:03.298 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.298 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:19:03.298 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.298 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46025.0:208404 56.13 56:cb6b8fb9:::benchmark_data_smithi136_356456_object81700:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1571)", 2024-08-17T18:19:03.298 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:12:42.003513+0000", 2024-08-17T18:19:03.298 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 380.51601070700002, 2024-08-17T18:19:03.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.68646327699999998, 2024-08-17T18:19:03.298 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:19:03.299 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:03.299 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:19:03.299 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46025", 2024-08-17T18:19:03.299 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:19:03.299 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 208404 2024-08-17T18:19:03.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.299 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:19:03.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:19:03.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.003513+0000", 2024-08-17T18:19:03.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:19:03.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.003513+0000", 2024-08-17T18:19:03.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:19:03.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.003512+0000", 2024-08-17T18:19:03.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:19:03.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:19:03.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.003519+0000", 2024-08-17T18:19:03.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2420000000000003e-06 2024-08-17T18:19:03.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:19:03.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.003530+0000", 2024-08-17T18:19:03.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.13e-05 2024-08-17T18:19:03.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:19:03.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.003536+0000", 2024-08-17T18:19:03.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4729999999999999e-06 2024-08-17T18:19:03.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.003594+0000", 2024-08-17T18:19:03.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8607000000000003e-05 2024-08-17T18:19:03.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:19:03.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.003606+0000", 2024-08-17T18:19:03.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1409e-05 2024-08-17T18:19:03.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.680583+0000", 2024-08-17T18:19:03.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.67697759999999996 2024-08-17T18:19:03.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:19:03.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.680687+0000", 2024-08-17T18:19:03.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103795 2024-08-17T18:19:03.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:19:03.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.680915+0000", 2024-08-17T18:19:03.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022817000000000001 2024-08-17T18:19:03.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:19:03.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.685157+0000", 2024-08-17T18:19:03.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042420820000000003 2024-08-17T18:19:03.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.685211+0000", 2024-08-17T18:19:03.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3180999999999998e-05 2024-08-17T18:19:03.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.689892+0000", 2024-08-17T18:19:03.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046817760000000003 2024-08-17T18:19:03.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:19:03.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.689937+0000", 2024-08-17T18:19:03.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4771999999999998e-05 2024-08-17T18:19:03.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:19:03.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:12:42.689976+0000", 2024-08-17T18:19:03.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9227000000000002e-05 2024-08-17T18:19:03.306 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.306 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:19:03.306 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.306 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:164948 57.13 57:c98a2be7:::benchmark_data_smithi136_389235_object49026:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1661)", 2024-08-17T18:19:03.307 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:06.232191+0000", 2024-08-17T18:19:03.307 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 56.287332968999998, 2024-08-17T18:19:03.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.186331312, 2024-08-17T18:19:03.307 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:19:03.307 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:03.307 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:19:03.307 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:19:03.307 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:19:03.307 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164948 2024-08-17T18:19:03.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.307 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:19:03.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:19:03.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.232191+0000", 2024-08-17T18:19:03.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:19:03.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.232191+0000", 2024-08-17T18:19:03.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:19:03.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.232189+0000", 2024-08-17T18:19:03.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:19:03.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:19:03.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.232196+0000", 2024-08-17T18:19:03.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8889999999999996e-06 2024-08-17T18:19:03.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:19:03.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.232207+0000", 2024-08-17T18:19:03.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.026e-05 2024-08-17T18:19:03.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:19:03.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.232213+0000", 2024-08-17T18:19:03.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8850000000000001e-06 2024-08-17T18:19:03.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.233791+0000", 2024-08-17T18:19:03.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001578025 2024-08-17T18:19:03.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:19:03.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.234460+0000", 2024-08-17T18:19:03.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00066837799999999996 2024-08-17T18:19:03.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.400592+0000", 2024-08-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16613243799999999 2024-08-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.400682+0000", 2024-08-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9889999999999995e-05 2024-08-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,7", 2024-08-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.401176+0000", 2024-08-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00049354300000000004 2024-08-17T18:19:03.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:19:03.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.405894+0000", 2024-08-17T18:19:03.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0047179539999999999 2024-08-17T18:19:03.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.406071+0000", 2024-08-17T18:19:03.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017692200000000001 2024-08-17T18:19:03.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.416708+0000", 2024-08-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010637562 2024-08-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.416773+0000", 2024-08-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4752999999999995e-05 2024-08-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.418522+0000", 2024-08-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017492810000000001 2024-08-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:19:03.313 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:164949 57.13 57:cef80ec3:::benchmark_data_smithi136_389235_object49027:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1661)", 2024-08-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:06.235864+0000", 2024-08-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 56.283660183000002, 2024-08-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.18931387399999999, 2024-08-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164949 2024-08-17T18:19:03.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.315 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:19:03.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:19:03.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.235864+0000", 2024-08-17T18:19:03.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:19:03.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.235864+0000", 2024-08-17T18:19:03.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:19:03.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.235862+0000", 2024-08-17T18:19:03.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:19:03.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:19:03.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.235872+0000", 2024-08-17T18:19:03.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9839999999999996e-06 2024-08-17T18:19:03.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:19:03.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.235885+0000", 2024-08-17T18:19:03.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2536e-05 2024-08-17T18:19:03.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:19:03.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.235892+0000", 2024-08-17T18:19:03.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2409999999999998e-06 2024-08-17T18:19:03.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.235944+0000", 2024-08-17T18:19:03.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2027999999999997e-05 2024-08-17T18:19:03.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:19:03.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.397491+0000", 2024-08-17T18:19:03.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16154682000000001 2024-08-17T18:19:03.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.412949+0000", 2024-08-17T18:19:03.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015458641 2024-08-17T18:19:03.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.318 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:19:03.318 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.413063+0000", 2024-08-17T18:19:03.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113634 2024-08-17T18:19:03.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,7", 2024-08-17T18:19:03.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.413692+0000", 2024-08-17T18:19:03.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062856800000000003 2024-08-17T18:19:03.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.418567+0000", 2024-08-17T18:19:03.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048755589999999998 2024-08-17T18:19:03.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.419699+0000", 2024-08-17T18:19:03.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001132271 2024-08-17T18:19:03.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:19:03.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.421487+0000", 2024-08-17T18:19:03.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017875409999999999 2024-08-17T18:19:03.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:19:03.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.421546+0000", 2024-08-17T18:19:03.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8576000000000003e-05 2024-08-17T18:19:03.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:19:03.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.425177+0000", 2024-08-17T18:19:03.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036319289999999999 2024-08-17T18:19:03.322 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.322 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:19:03.322 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.322 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPushReply(57.13 1661/1653 [PushReplyOp(57:c98a2be7:::benchmark_data_smithi136_389235_object49026:head)])", 2024-08-17T18:19:03.322 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:06.236250+0000", 2024-08-17T18:19:03.322 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 56.283273567000002, 2024-08-17T18:19:03.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.164314829, 2024-08-17T18:19:03.323 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:19:03.323 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:19:03.323 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:19:03.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:19:03.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.236250+0000", 2024-08-17T18:19:03.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:19:03.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.236250+0000", 2024-08-17T18:19:03.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:19:03.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.236249+0000", 2024-08-17T18:19:03.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:19:03.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:19:03.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.236255+0000", 2024-08-17T18:19:03.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0909999999999998e-06 2024-08-17T18:19:03.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:19:03.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.396307+0000", 2024-08-17T18:19:03.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16005135200000001 2024-08-17T18:19:03.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:19:03.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.396318+0000", 2024-08-17T18:19:03.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1420000000000001e-05 2024-08-17T18:19:03.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.397526+0000", 2024-08-17T18:19:03.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012080579999999999 2024-08-17T18:19:03.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:19:03.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.400565+0000", 2024-08-17T18:19:03.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003038752 2024-08-17T18:19:03.326 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.326 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:19:03.326 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.326 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:164950 57.d e1661/1653 57:b5298db8:::benchmark_data_smithi136_389235_object49028:head v 1661'10271, mlcod=1661'10269)", 2024-08-17T18:19:03.326 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:06.244575+0000", 2024-08-17T18:19:03.326 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 56.27494926, 2024-08-17T18:19:03.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15488284199999999, 2024-08-17T18:19:03.326 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:19:03.326 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:03.326 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.244575+0000", 2024-08-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.244575+0000", 2024-08-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.244573+0000", 2024-08-17T18:19:03.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.244585+0000", 2024-08-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2265e-05 2024-08-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.396183+0000", 2024-08-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15159708899999999 2024-08-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:19:03.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.396194+0000", 2024-08-17T18:19:03.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1416999999999999e-05 2024-08-17T18:19:03.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.396246+0000", 2024-08-17T18:19:03.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2395000000000002e-05 2024-08-17T18:19:03.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:19:03.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.396272+0000", 2024-08-17T18:19:03.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5432999999999999e-05 2024-08-17T18:19:03.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:19:03.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.399383+0000", 2024-08-17T18:19:03.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031107399999999999 2024-08-17T18:19:03.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:19:03.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.399457+0000", 2024-08-17T18:19:03.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4870999999999995e-05 2024-08-17T18:19:03.330 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.330 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:19:03.330 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.330 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:176917 57.3 57:c1b88941:::benchmark_data_smithi136_389235_object60995:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1672)", 2024-08-17T18:19:03.331 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:39.072490+0000", 2024-08-17T18:19:03.331 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.447033811000001, 2024-08-17T18:19:03.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.180348183, 2024-08-17T18:19:03.331 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:19:03.331 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:03.331 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:19:03.331 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:19:03.331 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:19:03.331 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 176917 2024-08-17T18:19:03.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.331 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:19:03.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.331 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:19:03.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.072490+0000", 2024-08-17T18:19:03.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:19:03.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.072490+0000", 2024-08-17T18:19:03.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:19:03.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.072488+0000", 2024-08-17T18:19:03.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:19:03.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:19:03.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.072495+0000", 2024-08-17T18:19:03.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7780000000000002e-06 2024-08-17T18:19:03.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:19:03.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.072507+0000", 2024-08-17T18:19:03.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2625e-05 2024-08-17T18:19:03.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:19:03.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.072514+0000", 2024-08-17T18:19:03.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9789999999999996e-06 2024-08-17T18:19:03.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.072562+0000", 2024-08-17T18:19:03.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7763999999999999e-05 2024-08-17T18:19:03.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-08-17T18:19:03.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.075501+0000", 2024-08-17T18:19:03.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029391349999999998 2024-08-17T18:19:03.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.082964+0000", 2024-08-17T18:19:03.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0074624030000000003 2024-08-17T18:19:03.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:19:03.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.083052+0000", 2024-08-17T18:19:03.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7948000000000006e-05 2024-08-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3,7", 2024-08-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.083505+0000", 2024-08-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00045366599999999998 2024-08-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.086828+0000", 2024-08-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033222299999999998 2024-08-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.086870+0000", 2024-08-17T18:19:03.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1916000000000003e-05 2024-08-17T18:19:03.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.086903+0000", 2024-08-17T18:19:03.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3192999999999999e-05 2024-08-17T18:19:03.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.094360+0000", 2024-08-17T18:19:03.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0074574819999999997 2024-08-17T18:19:03.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:19:03.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.094392+0000", 2024-08-17T18:19:03.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1390999999999998e-05 2024-08-17T18:19:03.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:19:03.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.252838+0000", 2024-08-17T18:19:03.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15844660399999999 2024-08-17T18:19:03.338 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.338 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:19:03.338 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.338 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:176923 57.b e1672/1604 57:ddbc0df9:::benchmark_data_smithi136_389235_object61001:head v 1672'10986, mlcod=1672'10984)", 2024-08-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:39.090690+0000", 2024-08-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.428834220999999, 2024-08-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15969454499999999, 2024-08-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.090690+0000", 2024-08-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.090690+0000", 2024-08-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.090689+0000", 2024-08-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.090697+0000", 2024-08-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0770000000000008e-06 2024-08-17T18:19:03.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:19:03.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.090708+0000", 2024-08-17T18:19:03.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1311999999999999e-05 2024-08-17T18:19:03.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:19:03.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.090715+0000", 2024-08-17T18:19:03.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.686e-06 2024-08-17T18:19:03.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.090792+0000", 2024-08-17T18:19:03.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6889000000000002e-05 2024-08-17T18:19:03.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:19:03.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.090815+0000", 2024-08-17T18:19:03.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2989999999999998e-05 2024-08-17T18:19:03.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:19:03.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.250339+0000", 2024-08-17T18:19:03.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15952429800000001 2024-08-17T18:19:03.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:19:03.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.250384+0000", 2024-08-17T18:19:03.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5232000000000003e-05 2024-08-17T18:19:03.343 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.343 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:19:03.343 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.343 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:176924 57.b e1672/1604 57:ddc16149:::benchmark_data_smithi136_389235_object61002:head v 1672'10987, mlcod=1672'10984)", 2024-08-17T18:19:03.343 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:39.090730+0000", 2024-08-17T18:19:03.343 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.428793614, 2024-08-17T18:19:03.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15968903400000001, 2024-08-17T18:19:03.343 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:19:03.343 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:03.344 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:19:03.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:19:03.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.090730+0000", 2024-08-17T18:19:03.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:19:03.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.090730+0000", 2024-08-17T18:19:03.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.090730+0000", 2024-08-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.090733+0000", 2024-08-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5030000000000002e-06 2024-08-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.090739+0000", 2024-08-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5389999999999998e-06 2024-08-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.090743+0000", 2024-08-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7780000000000001e-06 2024-08-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.092903+0000", 2024-08-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021597959999999999 2024-08-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.092923+0000", 2024-08-17T18:19:03.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0288e-05 2024-08-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.250387+0000", 2024-08-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15746444100000001 2024-08-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.250419+0000", 2024-08-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1959999999999999e-05 2024-08-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.348 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.46277.0:176917 57.3 e1672/1668 ondisk, result = 0)", 2024-08-17T18:19:03.348 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:39.092106+0000", 2024-08-17T18:19:03.348 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.427417521999999, 2024-08-17T18:19:03.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.160741939, 2024-08-17T18:19:03.348 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:19:03.348 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:19:03.348 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:19:03.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:19:03.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.092106+0000", 2024-08-17T18:19:03.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:19:03.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.092106+0000", 2024-08-17T18:19:03.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:19:03.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.092106+0000", 2024-08-17T18:19:03.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:19:03.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:19:03.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.092108+0000", 2024-08-17T18:19:03.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2900000000000001e-06 2024-08-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.092114+0000", 2024-08-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6200000000000004e-06 2024-08-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.092118+0000", 2024-08-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.788e-06 2024-08-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.094351+0000", 2024-08-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022335359999999999 2024-08-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.094356+0000", 2024-08-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0950000000000003e-06 2024-08-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.252848+0000", 2024-08-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15849197900000001 2024-08-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.351 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:176926 57.1 57:8118b0e1:::benchmark_data_smithi136_389235_object61004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1672)", 2024-08-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:39.093271+0000", 2024-08-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.426253172999999, 2024-08-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15747156100000001, 2024-08-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:19:03.352 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 176926 2024-08-17T18:19:03.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.353 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:19:03.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:19:03.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.093271+0000", 2024-08-17T18:19:03.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:19:03.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.093271+0000", 2024-08-17T18:19:03.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:19:03.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.093269+0000", 2024-08-17T18:19:03.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:19:03.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:19:03.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.093275+0000", 2024-08-17T18:19:03.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1460000000000001e-06 2024-08-17T18:19:03.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:19:03.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.093286+0000", 2024-08-17T18:19:03.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0682999999999999e-05 2024-08-17T18:19:03.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:19:03.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.093291+0000", 2024-08-17T18:19:03.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0050000000000004e-06 2024-08-17T18:19:03.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.093338+0000", 2024-08-17T18:19:03.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6581e-05 2024-08-17T18:19:03.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:19:03.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.093468+0000", 2024-08-17T18:19:03.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013067299999999999 2024-08-17T18:19:03.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2,7", 2024-08-17T18:19:03.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.093920+0000", 2024-08-17T18:19:03.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00045204499999999999 2024-08-17T18:19:03.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.250443+0000", 2024-08-17T18:19:03.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.156522777 2024-08-17T18:19:03.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:19:03.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.250452+0000", 2024-08-17T18:19:03.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7469999999999994e-06 2024-08-17T18:19:03.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.250545+0000", 2024-08-17T18:19:03.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2919999999999998e-05 2024-08-17T18:19:03.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:03.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.250603+0000", 2024-08-17T18:19:03.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8594000000000002e-05 2024-08-17T18:19:03.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:19:03.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.250652+0000", 2024-08-17T18:19:03.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8350999999999998e-05 2024-08-17T18:19:03.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:19:03.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.250742+0000", 2024-08-17T18:19:03.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0372999999999997e-05 2024-08-17T18:19:03.358 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.358 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:19:03.358 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.359 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.46277.0:176926 57.1 e1672/1637 ondisk, result = 0)", 2024-08-17T18:19:03.359 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:39.096533+0000", 2024-08-17T18:19:03.359 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.422990676000001, 2024-08-17T18:19:03.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15401411400000001, 2024-08-17T18:19:03.359 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:19:03.359 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:19:03.359 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:19:03.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:19:03.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.096533+0000", 2024-08-17T18:19:03.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.096533+0000", 2024-08-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.096532+0000", 2024-08-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:19:03.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.096541+0000", 2024-08-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3769999999999996e-06 2024-08-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.250498+0000", 2024-08-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15395747700000001 2024-08-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.250505+0000", 2024-08-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3579999999999999e-06 2024-08-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.250533+0000", 2024-08-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8609999999999999e-05 2024-08-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.250539+0000", 2024-08-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0750000000000003e-06 2024-08-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.250547+0000", 2024-08-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9969999999999998e-06 2024-08-17T18:19:03.362 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.46277.0:176926 57.1 e1672/1637 ondisk, result = 0)", 2024-08-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:39.096555+0000", 2024-08-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.422968874999999, 2024-08-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15419086800000001, 2024-08-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.363 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:19:03.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.096555+0000", 2024-08-17T18:19:03.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:19:03.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.096555+0000", 2024-08-17T18:19:03.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:19:03.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:19:03.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.096554+0000", 2024-08-17T18:19:03.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:19:03.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:19:03.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.096559+0000", 2024-08-17T18:19:03.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9250000000000003e-06 2024-08-17T18:19:03.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:19:03.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.250561+0000", 2024-08-17T18:19:03.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.15400230200000001 2024-08-17T18:19:03.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:19:03.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.250568+0000", 2024-08-17T18:19:03.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7320000000000001e-06 2024-08-17T18:19:03.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:19:03.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.250594+0000", 2024-08-17T18:19:03.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.561e-05 2024-08-17T18:19:03.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:19:03.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.250599+0000", 2024-08-17T18:19:03.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1569999999999999e-06 2024-08-17T18:19:03.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:19:03.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:19:03.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:19:03.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:39.250746+0000", 2024-08-17T18:19:03.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000146887 2024-08-17T18:19:03.367 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.367 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:19:03.367 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.369 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:19:03.370 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:19:03.370 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:19:03.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:03.371 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:03.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:03.472 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:03.504 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:03.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:03.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:03.557 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:19:03.606 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:03.644 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:03.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:03.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:03.746 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:03.775 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:03.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:03.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:03.826 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:19:03.877 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:03.906 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:03.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:03.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:03 smithi136 ceph-mon[109861]: pgmap v3222: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+remapped+backfill_wait; 2.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 521 op/s; 27604/137124 objects degraded (20.131%); 35745/137124 objects misplaced (26.068%); 12 MiB/s, 377 objects/s recovering 2024-08-17T18:19:03.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:03 smithi136 ceph-mon[115078]: pgmap v3222: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+remapped+backfill_wait; 2.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 521 op/s; 27604/137124 objects degraded (20.131%); 35745/137124 objects misplaced (26.068%); 12 MiB/s, 377 objects/s recovering 2024-08-17T18:19:04.008 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:03 smithi165 ceph-mon[105517]: pgmap v3222: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+remapped+backfill_wait; 2.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 521 op/s; 27604/137124 objects degraded (20.131%); 35745/137124 objects misplaced (26.068%); 12 MiB/s, 377 objects/s recovering 2024-08-17T18:19:04.038 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:04.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:04.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:04.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:04.164 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:04.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:04.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:04.181 INFO:tasks.thrashosds.thrasher:in_osds: [2, 4, 6, 0, 5, 7, 3] out_osds: [1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:19:04.181 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:19:04.182 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-08-17T18:19:04.182 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-17T18:19:04.182 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-17T18:19:04.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-08-17T18:19:04.266 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:04.297 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:04.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:04.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:04.400 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:04.432 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:04.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:04.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:04.534 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:04.560 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:04.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:04.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:04.661 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:04.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:04.696 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:04.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:04.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:04.831 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:04.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:04.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:04.933 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:04.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:04 smithi136 ceph-mon[109861]: osdmap e1676: 8 total, 8 up, 7 in 2024-08-17T18:19:04.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:04 smithi136 ceph-mon[115078]: osdmap e1676: 8 total, 8 up, 7 in 2024-08-17T18:19:04.963 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:04.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:04.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:04.985 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:19:05.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:04 smithi165 ceph-mon[105517]: osdmap e1676: 8 total, 8 up, 7 in 2024-08-17T18:19:05.066 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:05.091 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:05.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:05.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:05.193 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:05.217 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:05.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:05.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:05.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:05.358 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:05.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:05.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:05.459 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:05.490 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:05.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:05.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:05.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:05.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:05.626 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:05.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:05.727 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:05.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:05.768 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:05.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:05.869 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:05.907 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:05.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:05.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:05 smithi136 ceph-mon[109861]: pgmap v3224: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+remapped+backfill_wait; 2.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 296 op/s; 27604/137124 objects degraded (20.131%); 35745/137124 objects misplaced (26.068%); 6.6 MiB/s, 240 objects/s recovering 2024-08-17T18:19:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:05 smithi136 ceph-mon[109861]: osdmap e1677: 8 total, 8 up, 7 in 2024-08-17T18:19:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:05 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 27604/137124 objects degraded (20.131%), 16 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:05.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:05 smithi136 ceph-mon[115078]: pgmap v3224: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+remapped+backfill_wait; 2.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 296 op/s; 27604/137124 objects degraded (20.131%); 35745/137124 objects misplaced (26.068%); 6.6 MiB/s, 240 objects/s recovering 2024-08-17T18:19:05.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:05 smithi136 ceph-mon[115078]: osdmap e1677: 8 total, 8 up, 7 in 2024-08-17T18:19:05.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:05 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 27604/137124 objects degraded (20.131%), 16 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:06.009 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:06.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:05 smithi165 ceph-mon[105517]: pgmap v3224: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 9 active+remapped+backfill_wait; 2.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 296 op/s; 27604/137124 objects degraded (20.131%); 35745/137124 objects misplaced (26.068%); 6.6 MiB/s, 240 objects/s recovering 2024-08-17T18:19:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:05 smithi165 ceph-mon[105517]: osdmap e1677: 8 total, 8 up, 7 in 2024-08-17T18:19:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:05 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 27604/137124 objects degraded (20.131%), 16 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:06.048 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:06.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:06.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:06.150 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:06.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:06.208 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:06.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:06.310 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:06.320 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:19:06.347 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:06.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:06.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:06.449 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:06.472 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:06.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:06.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:06.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:06.607 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:06.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:06.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:06.709 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:06.737 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:06.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:06.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:06.839 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:06.879 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:06.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:06.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:06.981 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:07.021 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:07.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:07.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:07.103 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:19:07.123 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:07.134 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-17T18:19:07.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:07.169 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:07.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:07.236 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:19:07.270 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:07.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:07.311 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:07.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:07.412 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:07.431 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:19:07.432 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:19:07.432 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:07 smithi165 ceph-mon[105517]: pgmap v3226: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 387 op/s; 27450/135171 objects degraded (20.308%); 35034/135171 objects misplaced (25.918%); 7.8 MiB/s, 279 objects/s recovering 2024-08-17T18:19:07.432 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:07 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/433322406' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:19:07.437 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:07.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:07.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:07.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:07.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:07.575 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:07.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:07.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:07.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:19:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:19:07] "GET /metrics HTTP/1.1" 200 36775 "" "Prometheus/2.51.0" 2024-08-17T18:19:07.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:19:07.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:19:07.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:07 smithi136 ceph-mon[115078]: pgmap v3226: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 387 op/s; 27450/135171 objects degraded (20.308%); 35034/135171 objects misplaced (25.918%); 7.8 MiB/s, 279 objects/s recovering 2024-08-17T18:19:07.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:07 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/433322406' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:19:07.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:19:07.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:19:07.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:07 smithi136 ceph-mon[109861]: pgmap v3226: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 82 active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 387 op/s; 27450/135171 objects degraded (20.308%); 35034/135171 objects misplaced (25.918%); 7.8 MiB/s, 279 objects/s recovering 2024-08-17T18:19:07.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:07 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/433322406' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:19:07.704 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:07.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:07.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:07.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:07.837 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:07.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:07.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:07.939 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:07.964 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:07.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:07.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:08.065 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:08.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:08.090 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:08.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:08.192 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:08.226 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:08.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:08.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:08.328 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:08.359 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:08.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:08.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:08.461 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:08.483 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:08.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:08.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:08.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:08 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:19:08.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:08 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/433322406' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:19:08.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:08 smithi165 ceph-mon[105517]: osdmap e1678: 8 total, 8 up, 7 in 2024-08-17T18:19:08.585 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:08.615 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:08.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:08.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:08.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:08 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:19:08.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:08 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/433322406' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:19:08.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:08 smithi136 ceph-mon[109861]: osdmap e1678: 8 total, 8 up, 7 in 2024-08-17T18:19:08.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:08 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:19:08.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:08 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/433322406' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:19:08.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:08 smithi136 ceph-mon[115078]: osdmap e1678: 8 total, 8 up, 7 in 2024-08-17T18:19:08.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:08.744 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:08.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:08.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:08.846 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:08.881 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:08.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:08.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:08.943 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:19:08.983 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:09.043 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:09.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:09.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:09.145 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:09.171 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:09.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:09.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:09.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:09.315 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:09.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:09.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:09.417 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:09.431 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:09 smithi165 ceph-mon[105517]: pgmap v3228: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 391 op/s; 23341/130071 objects degraded (17.945%); 35033/130071 objects misplaced (26.934%); 12 MiB/s, 212 objects/s recovering 2024-08-17T18:19:09.444 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:09.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:09.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:09.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:09.562 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:09 smithi136 ceph-mon[109861]: pgmap v3228: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 391 op/s; 23341/130071 objects degraded (17.945%); 35033/130071 objects misplaced (26.934%); 12 MiB/s, 212 objects/s recovering 2024-08-17T18:19:09.563 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:09 smithi136 ceph-mon[115078]: pgmap v3228: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 391 op/s; 23341/130071 objects degraded (17.945%); 35033/130071 objects misplaced (26.934%); 12 MiB/s, 212 objects/s recovering 2024-08-17T18:19:09.573 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:09.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:09.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:09.595 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:19:09.596 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "" 2024-08-17T18:19:09.596 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:19:09.675 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:09.717 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:09.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:09.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:09.819 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:09.844 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:09.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:09.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:09.857 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:19:09.857 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:19:09.879 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:19:09.880 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:19:09.904 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:19:09.905 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:19:09.930 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:19:09.930 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:19:09.946 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:09.950 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:19:09.950 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:19:09.970 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:09.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:09.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:09.977 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:19:09.977 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:19:09.999 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:19:09.999 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:19:10.021 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:19:10.021 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:19:10.072 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:10.103 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:10.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:10.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:10.205 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:10.237 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:10.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:10.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:10.340 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:10.373 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:10.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:10.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:10.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:10.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:10 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 23341/130071 objects degraded (17.945%), 14 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:10.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:10 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3774211592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:19:10.515 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:10.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:10.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:10.617 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:10.648 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:10.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:10.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:10.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:10 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 23341/130071 objects degraded (17.945%), 14 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:10.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:10 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3774211592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:19:10.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:10 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 23341/130071 objects degraded (17.945%), 14 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:10.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:10 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3774211592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:19:10.750 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:10.758 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:19:10.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:10.775 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:10.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:10.876 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:10.911 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:10.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:10.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:11.013 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:11.017 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:19:11.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:11.044 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:11.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:11.145 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:11.170 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:11.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:11.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:11.271 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:11.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:11.310 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:11.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:11.393 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:11 smithi136 ceph-mon[109861]: pgmap v3229: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 412 op/s; 23203/128565 objects degraded (18.048%); 34600/128565 objects misplaced (26.912%); 11 MiB/s, 194 objects/s recovering 2024-08-17T18:19:11.393 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:11 smithi136 ceph-mon[115078]: pgmap v3229: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 412 op/s; 23203/128565 objects degraded (18.048%); 34600/128565 objects misplaced (26.912%); 11 MiB/s, 194 objects/s recovering 2024-08-17T18:19:11.411 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:11.454 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:11.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:11.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:11.496 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:11 smithi165 ceph-mon[105517]: pgmap v3229: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 412 op/s; 23203/128565 objects degraded (18.048%); 34600/128565 objects misplaced (26.912%); 11 MiB/s, 194 objects/s recovering 2024-08-17T18:19:11.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:11.601 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:11.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:11.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:11.703 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:11.729 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:11.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:11.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:11.831 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:11.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:11.857 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:11.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:11.958 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:11.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:12.304 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:12.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:12.305 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:19:12.305 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:19:12.406 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:12.434 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:12.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:12.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:12.536 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:12.565 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:12.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:12.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:12.667 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:12.693 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:12.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:12.795 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:12.818 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:12.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:12.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:12.826 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:19:12.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:12.958 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:12.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:12.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:13.060 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:13.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:13.087 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:13.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:13.188 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:13.217 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:13.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:13.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:13.319 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:13.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:13.348 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:13.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:13.449 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:13.471 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:13.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:13.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:13.573 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:13.600 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:13.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:13.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:13.702 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:13.734 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:13.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:13.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:13.836 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:13.876 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:13.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:13.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:13 smithi136 ceph-mon[109861]: pgmap v3230: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 610 op/s; 22441/122466 objects degraded (18.324%); 32737/122466 objects misplaced (26.732%); 19 MiB/s, 311 objects/s recovering 2024-08-17T18:19:13.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:13 smithi136 ceph-mon[115078]: pgmap v3230: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 610 op/s; 22441/122466 objects degraded (18.324%); 32737/122466 objects misplaced (26.732%); 19 MiB/s, 311 objects/s recovering 2024-08-17T18:19:13.978 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:13 smithi165 ceph-mon[105517]: pgmap v3230: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 610 op/s; 22441/122466 objects degraded (18.324%); 32737/122466 objects misplaced (26.732%); 19 MiB/s, 311 objects/s recovering 2024-08-17T18:19:14.014 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:14.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:14.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:14.116 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:14.155 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:14.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:14.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:14.257 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:14.300 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:14.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:14.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:14.401 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:14.437 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:14.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:14.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:14.539 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:14.569 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:14.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:14.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:14.670 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:14.707 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:14.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:14.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:14.808 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-17T18:19:14.810 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:14.864 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:14.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:14.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:14.966 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:14.991 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:14.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:14.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:15.092 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:15.127 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:15.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:15.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:15.153 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:19:15.216 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:19:15.229 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:15.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:15.271 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:15.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:15.287 INFO:tasks.thrashosds.thrasher:in_osds: [2, 4, 6, 0, 5, 7, 3] out_osds: [1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:19:15.287 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:19:15.288 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-08-17T18:19:15.288 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T18:19:15.372 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:15.399 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:15.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:15.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:15.501 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:15.549 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:15.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:15.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:15.560 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:19:15.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:15.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:15.682 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:15.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:15.752 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:15 smithi165 ceph-mon[105517]: pgmap v3231: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 493 op/s; 22441/122466 objects degraded (18.324%); 32737/122466 objects misplaced (26.732%); 15 MiB/s, 251 objects/s recovering 2024-08-17T18:19:15.752 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:15 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3991698358' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:19:15.752 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:15 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:19:15.752 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:15 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 22441/122466 objects degraded (18.324%), 14 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:15.783 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:15.810 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:15.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:15.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:15.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:15.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:15 smithi136 ceph-mon[109861]: pgmap v3231: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 493 op/s; 22441/122466 objects degraded (18.324%); 32737/122466 objects misplaced (26.732%); 15 MiB/s, 251 objects/s recovering 2024-08-17T18:19:15.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:15 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3991698358' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:19:15.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:15 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:19:15.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:15 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 22441/122466 objects degraded (18.324%), 14 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:15 smithi136 ceph-mon[115078]: pgmap v3231: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 493 op/s; 22441/122466 objects degraded (18.324%); 32737/122466 objects misplaced (26.732%); 15 MiB/s, 251 objects/s recovering 2024-08-17T18:19:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:15 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3991698358' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:19:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:15 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:19:15.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:15 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 22441/122466 objects degraded (18.324%), 14 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:15.950 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:15.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:15.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:16.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:16.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:16.087 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:16.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:16.188 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:16.214 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:16.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:16.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:16.316 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:16.356 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:16.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:16.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:16.458 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:16.482 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:16.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:16.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:16.587 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:16.613 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:16.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:16.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:16.715 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:16.741 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:16.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:16.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:16.843 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:16.869 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:16.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:16.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:16.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:16 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:19:16.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:16 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:19:16.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:16 smithi136 ceph-mon[115078]: osdmap e1679: 8 total, 8 up, 7 in 2024-08-17T18:19:16.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:16 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:19:16.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:16 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:19:16.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:16 smithi136 ceph-mon[109861]: osdmap e1679: 8 total, 8 up, 7 in 2024-08-17T18:19:16.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:17.008 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:17.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:17.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:16 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:19:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:16 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:19:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:16 smithi165 ceph-mon[105517]: osdmap e1679: 8 total, 8 up, 7 in 2024-08-17T18:19:17.111 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:17.134 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:17.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:17.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:17.235 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:17.260 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:17.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:17.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:17.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:17.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:17.400 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:17.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:17.501 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:17.526 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:17.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:17.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:17.628 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:17.653 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:17.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:17.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:17.755 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:17 smithi165 ceph-mon[105517]: pgmap v3233: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 418 op/s; 22295/120291 objects degraded (18.534%); 31933/120291 objects misplaced (26.546%); 13 MiB/s, 202 objects/s recovering 2024-08-17T18:19:17.755 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:17.791 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:17.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:17.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:17.861 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:19:17.861 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:19:17.893 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:17.925 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:17.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:17.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:17.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:17 smithi136 ceph-mon[109861]: pgmap v3233: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 418 op/s; 22295/120291 objects degraded (18.534%); 31933/120291 objects misplaced (26.546%); 13 MiB/s, 202 objects/s recovering 2024-08-17T18:19:17.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:17 smithi136 ceph-mon[115078]: pgmap v3233: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 418 op/s; 22295/120291 objects degraded (18.534%); 31933/120291 objects misplaced (26.546%); 13 MiB/s, 202 objects/s recovering 2024-08-17T18:19:17.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:19:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:19:17] "GET /metrics HTTP/1.1" 200 36775 "" "Prometheus/2.51.0" 2024-08-17T18:19:18.027 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:18.053 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:18.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:18.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:18.155 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:18.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:18.192 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:18.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:18.293 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:18.320 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:18.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:18.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:18.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:18.461 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:18.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:18.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:18.563 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:18.589 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:18.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:18.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:18.602 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-17T18:19:18.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:18.750 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:18.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:18.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:18.852 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:18.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:18.878 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:18.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:18.946 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:19:18.979 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:19.001 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:19.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:19.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:19.103 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:19.129 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:19.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:19.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:19.231 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:19.268 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:19.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:19.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:19.370 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:19.394 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:19.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:19.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:19.496 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:19.519 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:19.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:19.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:19.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:19.656 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:19.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:19.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:19.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:19 smithi136 ceph-mon[109861]: pgmap v3234: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 518 op/s; 21445/114510 objects degraded (18.728%); 30420/114510 objects misplaced (26.565%); 16 MiB/s, 248 objects/s recovering 2024-08-17T18:19:19.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:19 smithi136 ceph-mon[115078]: pgmap v3234: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 518 op/s; 21445/114510 objects degraded (18.728%); 30420/114510 objects misplaced (26.565%); 16 MiB/s, 248 objects/s recovering 2024-08-17T18:19:19.759 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:19.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:19 smithi165 ceph-mon[105517]: pgmap v3234: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 518 op/s; 21445/114510 objects degraded (18.728%); 30420/114510 objects misplaced (26.565%); 16 MiB/s, 248 objects/s recovering 2024-08-17T18:19:19.786 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:19.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:19.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:19.888 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:19.920 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:19.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:19.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:20.022 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:20.047 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:20.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:20.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:20.149 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:20.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:20.174 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:20.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:20.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:20.316 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:20.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:20.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:20.418 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:20.441 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:20.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:20.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:20.543 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:20.570 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:20.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:20.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:20.672 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:20.698 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:20.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:20.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:20.800 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:20.825 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:20.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:20.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:20.881 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:20 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 21445/114510 objects degraded (18.728%), 14 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:20.927 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:20.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:20 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 21445/114510 objects degraded (18.728%), 14 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:20.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:20 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 21445/114510 objects degraded (18.728%), 14 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:20.948 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:19:20.948 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:19:20.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:20.974 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:20.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:21.076 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:21.112 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:21.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:21.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:21.214 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:21.260 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:21.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:21.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:21.362 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:21.387 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:21.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:21.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:21.489 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:21.516 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:21.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:21.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:21.618 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:21.654 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:21.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:21.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:21 smithi136 ceph-mon[109861]: pgmap v3235: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 515 op/s; 21290/113091 objects degraded (18.826%); 30024/113091 objects misplaced (26.549%); 16 MiB/s, 249 objects/s recovering 2024-08-17T18:19:21.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:19:21.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:21 smithi136 ceph-mon[115078]: pgmap v3235: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 515 op/s; 21290/113091 objects degraded (18.826%); 30024/113091 objects misplaced (26.549%); 16 MiB/s, 249 objects/s recovering 2024-08-17T18:19:21.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:19:21.757 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:21.785 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:21.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:21.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:21.887 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:21.912 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:21.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:21.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:21 smithi165 ceph-mon[105517]: pgmap v3235: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 515 op/s; 21290/113091 objects degraded (18.826%); 30024/113091 objects misplaced (26.549%); 16 MiB/s, 249 objects/s recovering 2024-08-17T18:19:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:19:22.014 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:22.039 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:22.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:22.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:22.140 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:22.169 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:22.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:22.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:22.271 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:22.311 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:22.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:22.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:22.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:22.442 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:22.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:22.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:22.545 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:22.574 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:22.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:22.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:22.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:22.704 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:22.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:22.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:22.806 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:22.833 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:22.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:22.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:22.914 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:19:22.935 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:22.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:22.960 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:22.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:23.061 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:23.091 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:23.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:23.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:23.192 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:23.224 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:23.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:23.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:23.326 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:23.360 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:23.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:23.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:23.462 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:23.484 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:23.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:23.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:23.586 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:23.611 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:23.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:23.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:23.712 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:23.743 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:23.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:23.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:23.845 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:23.871 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:23 smithi136 ceph-mon[109861]: pgmap v3236: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 519 op/s; 20431/106881 objects degraded (19.116%); 28506/106881 objects misplaced (26.671%); 14 MiB/s, 264 objects/s recovering 2024-08-17T18:19:23.872 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:23 smithi136 ceph-mon[115078]: pgmap v3236: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 519 op/s; 20431/106881 objects degraded (19.116%); 28506/106881 objects misplaced (26.671%); 14 MiB/s, 264 objects/s recovering 2024-08-17T18:19:23.876 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:23.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:23.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:23.978 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:24.001 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:24.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:24.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:24.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:23 smithi165 ceph-mon[105517]: pgmap v3236: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 519 op/s; 20431/106881 objects degraded (19.116%); 28506/106881 objects misplaced (26.671%); 14 MiB/s, 264 objects/s recovering 2024-08-17T18:19:24.102 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:24.135 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:24.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:24.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:24.237 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:24.272 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:24.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:24.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:24.374 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:24.400 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:24.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:24.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:24.502 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:24.526 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:24.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:24.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:24.627 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:24.658 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:24.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:24.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:24.760 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:24.784 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:24.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:24.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:24.849 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:19:24.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:24.921 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:24.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:24.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:25.022 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:25.048 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:25.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:25.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:25.082 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-17T18:19:25.151 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:25.179 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:25.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:25.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:25.281 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:25.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:25.320 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:25.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:25.395 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:19:25.395 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:19:25.420 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:25.428 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:19:25.428 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:19:25.450 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:19:25.450 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:19:25.458 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:25.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:25.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:25.469 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:19:25.469 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:19:25.498 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:19:25.498 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:19:25.524 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:19:25.525 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:19:25.549 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:19:25.550 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:19:25.560 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:25.585 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:19:25.585 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:19:25.600 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:25.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:25.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:25.702 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:25.727 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:25.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:25.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:25.828 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:25.829 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:25 smithi136 ceph-mon[109861]: pgmap v3237: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 519 op/s; 20431/106881 objects degraded (19.116%); 28506/106881 objects misplaced (26.671%); 14 MiB/s, 264 objects/s recovering 2024-08-17T18:19:25.829 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:25 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 20431/106881 objects degraded (19.116%), 15 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:25.829 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:25 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/667871119' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:19:25.830 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:25 smithi136 ceph-mon[115078]: pgmap v3237: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 519 op/s; 20431/106881 objects degraded (19.116%); 28506/106881 objects misplaced (26.671%); 14 MiB/s, 264 objects/s recovering 2024-08-17T18:19:25.830 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:25 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 20431/106881 objects degraded (19.116%), 15 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:25.830 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:25 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/667871119' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:19:25.855 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:25.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:25.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:25.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:25.996 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:25.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:25.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:26.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:25 smithi165 ceph-mon[105517]: pgmap v3237: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 519 op/s; 20431/106881 objects degraded (19.116%); 28506/106881 objects misplaced (26.671%); 14 MiB/s, 264 objects/s recovering 2024-08-17T18:19:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:25 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 20431/106881 objects degraded (19.116%), 15 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:25 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/667871119' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:19:26.098 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:26.122 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:26.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:26.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:26.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:26.260 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:26.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:26.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:26.350 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:19:26.362 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:26.393 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:26.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:26.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:26.495 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:26.519 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:26.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:26.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:26.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:26.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:26.662 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:26.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:26.763 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:26.800 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:26.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:26.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:26.901 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:26.925 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:26.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:26.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:27.027 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:27.049 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:27.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:27.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:27.152 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:27.179 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:27.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:27.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:27.227 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:19:27.281 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:27.307 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:27.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:27.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:27.372 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:19:27.373 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T18:19:27.409 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:27.434 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:27.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:27.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:27.536 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:27.569 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:27.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:27.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:27.583 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:19:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:19:27] "GET /metrics HTTP/1.1" 200 36778 "" "Prometheus/2.51.0" 2024-08-17T18:19:27.671 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:27.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:27.722 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:27.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:27.823 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:27.837 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:27 smithi136 ceph-mon[115078]: pgmap v3238: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 495 op/s; 20258/104757 objects degraded (19.338%); 27733/104757 objects misplaced (26.474%); 13 MiB/s, 243 objects/s recovering 2024-08-17T18:19:27.837 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:27 smithi136 ceph-mon[115078]: osdmap e1680: 8 total, 8 up, 7 in 2024-08-17T18:19:27.837 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:27 smithi136 ceph-mon[109861]: pgmap v3238: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 495 op/s; 20258/104757 objects degraded (19.338%); 27733/104757 objects misplaced (26.474%); 13 MiB/s, 243 objects/s recovering 2024-08-17T18:19:27.837 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:27 smithi136 ceph-mon[109861]: osdmap e1680: 8 total, 8 up, 7 in 2024-08-17T18:19:27.863 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:27.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:27.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:27.965 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:27.999 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:27.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:28.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:28.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:27 smithi165 ceph-mon[105517]: pgmap v3238: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovery_wait+degraded, 8 active+recovery_wait+degraded+remapped, 83 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 495 op/s; 20258/104757 objects degraded (19.338%); 27733/104757 objects misplaced (26.474%); 13 MiB/s, 243 objects/s recovering 2024-08-17T18:19:28.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:27 smithi165 ceph-mon[105517]: osdmap e1680: 8 total, 8 up, 7 in 2024-08-17T18:19:28.100 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:28.125 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:28.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:28.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:28.226 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:28.272 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:28.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:28.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:28.374 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:28.397 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:28.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:28.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:28.499 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:28.523 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:28.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:28.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:28.625 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:28.657 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:28.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:28.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:28.699 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":3239,"stamp":"2024-08-17T18:19:26.581748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2287385416,"num_objects":34919,"num_object_clones":0,"num_object_copies":104757,"num_objects_missing_on_primary":308,"num_objects_missing":308,"num_objects_degraded":20258,"num_objects_misplaced":27733,"num_objects_unfound":0,"num_objects_dirty":34919,"num_whiteouts":0,"num_read":70,"num_read_kb":150,"num_write":197050,"num_write_kb":7419289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40223,"num_bytes_recovered":2310901800,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136317,"ondisk_log_size":136317,"up":327,"acting":321,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":422,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":16794236,"kb_used_data":9955524,"kb_used_omap":285,"kb_used_meta":6838370,"kb_avail":639421828,"statfs":{"total":671965249536,"available":654767951872,"internally_reserved":0,"allocated":10194456576,"data_stored":10163600181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":292552,"internal_metadata":7002491192},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-339345408,"num_objects":-5178,"num_object_clones":0,"num_object_copies":-15534,"num_objects_missing_on_primary":-164,"num_objects_missing":-164,"num_objects_degraded":-2037,"num_objects_misplaced":-4200,"num_objects_unfound":0,"num_objects_dirty":-5178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5178,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2664,"num_bytes_recovered":146341888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.096780"},"pg_stats":[{"pgid":"57.19","version":"1679'8627","reported_seq":8730,"reported_epoch":1679,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:19:24.728656+0000","last_change":"2024-08-17T18:19:02.633485+0000","last_active":"2024-08-17T18:19:24.728656+0000","last_peered":"2024-08-17T18:19:24.728656+0000","last_clean":"2024-08-17T18:18:29.597061+0000","last_became_active":"2024-08-17T18:18:30.810367+0000","last_became_peered":"2024-08-17T18:18:30.810367+0000","last_unstale":"2024-08-17T18:19:24.728656+0000","last_undegraded":"2024-08-17T18:19:24.728656+0000","last_fullsized":"2024-08-17T18:19:24.728656+0000","mapping_epoch":1667,"log_start":"1613'3296","ondisk_log_start":"1613'3296","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5331,"log_dups_size":0,"ondisk_log_size":5331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68845568,"num_objects":1051,"num_object_clones":0,"num_object_copies":3153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1051,"num_objects_unfound":0,"num_objects_dirty":1051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6149,"num_write_kb":230368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[4,5,6],"avail_no_missing":["4","5","6"],"object_location_counts":[{"shards":"4,5,6","objects":1051}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"57.18","version":"1679'8741","reported_seq":13586,"reported_epoch":1679,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:19:22.132173+0000","last_change":"2024-08-17T18:15:54.579791+0000","last_active":"2024-08-17T18:19:22.132173+0000","last_peered":"2024-08-17T18:19:22.132173+0000","last_clean":"2024-08-17T18:15:09.641861+0000","last_became_active":"2024-08-17T18:15:54.569392+0000","last_became_peered":"2024-08-17T18:15:54.569392+0000","last_unstale":"2024-08-17T18:19:22.132173+0000","last_undegraded":"2024-08-17T18:15:54.395452+0000","last_fullsized":"2024-08-17T18:19:22.132173+0000","mapping_epoch":1638,"log_start":"1614'4778","ondisk_log_start":"1614'4778","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3963,"log_dups_size":0,"ondisk_log_size":3963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70090752,"num_objects":1070,"num_object_clones":0,"num_object_copies":3210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":860,"num_objects_misplaced":210,"num_objects_unfound":0,"num_objects_dirty":1070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6234,"num_write_kb":233696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2367,"num_bytes_recovered":155123712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[5,6,4],"avail_no_missing":["5","0","6"],"object_location_counts":[{"shards":"0,5,6","objects":645},{"shards":"4,5,6","objects":425}],"blocked_by":[],"up_primary":7,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.17","version":"1679'8419","reported_seq":11582,"reported_epoch":1679,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:19:22.370914+0000","last_change":"2024-08-17T18:18:29.904207+0000","last_active":"2024-08-17T18:19:22.370914+0000","last_peered":"2024-08-17T18:19:22.370914+0000","last_clean":"2024-08-17T18:15:52.210567+0000","last_became_active":"2024-08-17T18:17:23.718959+0000","last_became_peered":"2024-08-17T18:17:23.718959+0000","last_unstale":"2024-08-17T18:19:22.370914+0000","last_undegraded":"2024-08-17T18:18:29.902629+0000","last_fullsized":"2024-08-17T18:17:23.630677+0000","mapping_epoch":1653,"log_start":"1613'3674","ondisk_log_start":"1613'3674","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4745,"log_dups_size":0,"ondisk_log_size":4745,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69435392,"num_objects":1060,"num_object_clones":0,"num_object_copies":3180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2829,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5988,"num_write_kb":225504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1482,"num_bytes_recovered":97124352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,4,6","objects":1041},{"shards":"2,7","objects":19}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"57.16","version":"1679'8452","reported_seq":15680,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:21.985137+0000","last_change":"2024-08-17T18:17:57.587085+0000","last_active":"2024-08-17T18:19:21.985137+0000","last_peered":"2024-08-17T18:19:21.985137+0000","last_clean":"2024-08-17T18:19:21.985137+0000","last_became_active":"2024-08-17T18:17:57.576108+0000","last_became_peered":"2024-08-17T18:17:57.576108+0000","last_unstale":"2024-08-17T18:19:21.985137+0000","last_undegraded":"2024-08-17T18:19:21.985137+0000","last_fullsized":"2024-08-17T18:19:21.985137+0000","mapping_epoch":1660,"log_start":"1614'4341","ondisk_log_start":"1614'4341","created":1616,"last_epoch_clean":1661,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4111,"log_dups_size":0,"ondisk_log_size":4111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66617344,"num_objects":1017,"num_object_clones":0,"num_object_copies":3051,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6017,"num_write_kb":225056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":166985728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.15","version":"1679'8615","reported_seq":13202,"reported_epoch":1679,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:19:22.160148+0000","last_change":"2024-08-17T18:18:32.060713+0000","last_active":"2024-08-17T18:19:22.160148+0000","last_peered":"2024-08-17T18:19:22.160148+0000","last_clean":"2024-08-17T18:15:09.587418+0000","last_became_active":"2024-08-17T18:18:32.037569+0000","last_became_peered":"2024-08-17T18:18:32.037569+0000","last_unstale":"2024-08-17T18:19:22.160148+0000","last_undegraded":"2024-08-17T18:18:31.866840+0000","last_fullsized":"2024-08-17T18:19:22.160148+0000","mapping_epoch":1668,"log_start":"1614'4512","ondisk_log_start":"1614'4512","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4103,"log_dups_size":0,"ondisk_log_size":4103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78544896,"num_objects":1199,"num_object_clones":0,"num_object_copies":3597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2837,"num_objects_misplaced":443,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6113,"num_write_kb":233952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1485,"num_bytes_recovered":97255424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,7,2],"avail_no_missing":["5","0","6"],"object_location_counts":[{"shards":"0,5,6","objects":666},{"shards":"2,5,6","objects":443},{"shards":"2,5,7","objects":90}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.14","version":"1679'8505","reported_seq":10031,"reported_epoch":1679,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:19:22.121862+0000","last_change":"2024-08-17T18:18:32.046673+0000","last_active":"2024-08-17T18:19:22.121862+0000","last_peered":"2024-08-17T18:19:22.121862+0000","last_clean":"2024-08-17T18:15:09.651475+0000","last_became_active":"2024-08-17T18:18:31.915649+0000","last_became_peered":"2024-08-17T18:18:31.915649+0000","last_unstale":"2024-08-17T18:19:22.121862+0000","last_undegraded":"2024-08-17T18:18:31.906636+0000","last_fullsized":"2024-08-17T18:19:22.121862+0000","mapping_epoch":1668,"log_start":"1614'4409","ondisk_log_start":"1614'4409","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4096,"log_dups_size":0,"ondisk_log_size":4096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69468160,"num_objects":1060,"num_object_clones":0,"num_object_copies":3180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1060,"num_objects_misplaced":2877,"num_objects_unfound":0,"num_objects_dirty":1060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6052,"num_write_kb":227584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":682,"num_bytes_recovered":44695552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[5,7,4],"avail_no_missing":["5","0","4"],"object_location_counts":[{"shards":"0,4,5","objects":621},{"shards":"4,5,6","objects":441},{"shards":"4,5,7","objects":-2}],"blocked_by":[],"up_primary":3,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.13","version":"1679'8492","reported_seq":11599,"reported_epoch":1679,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:19:22.086729+0000","last_change":"2024-08-17T18:18:31.912458+0000","last_active":"2024-08-17T18:19:22.086729+0000","last_peered":"2024-08-17T18:19:22.086729+0000","last_clean":"2024-08-17T18:15:52.116504+0000","last_became_active":"2024-08-17T18:18:31.877372+0000","last_became_peered":"2024-08-17T18:18:31.877372+0000","last_unstale":"2024-08-17T18:19:22.086729+0000","last_undegraded":"2024-08-17T18:18:31.859470+0000","last_fullsized":"2024-08-17T18:19:22.086729+0000","mapping_epoch":1668,"log_start":"1613'1792","ondisk_log_start":"1613'1792","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":6700,"log_dups_size":0,"ondisk_log_size":6700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73433088,"num_objects":1121,"num_object_clones":0,"num_object_copies":3363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1933,"num_objects_misplaced":1278,"num_objects_unfound":0,"num_objects_dirty":1121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6043,"num_write_kb":229216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1469,"num_bytes_recovered":96272384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[6,7,0],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"0,2,6","objects":1098},{"shards":"0,6,7","objects":23}],"blocked_by":[],"up_primary":3,"acting_primary":6,"purged_snaps":[]},{"pgid":"57.12","version":"1679'8564","reported_seq":16102,"reported_epoch":1679,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:19:24.856403+0000","last_change":"2024-08-17T18:18:34.413365+0000","last_active":"2024-08-17T18:19:24.856403+0000","last_peered":"2024-08-17T18:19:24.856403+0000","last_clean":"2024-08-17T18:14:59.641932+0000","last_became_active":"2024-08-17T18:18:33.841392+0000","last_became_peered":"2024-08-17T18:18:33.841392+0000","last_unstale":"2024-08-17T18:19:24.856403+0000","last_undegraded":"2024-08-17T18:18:32.544363+0000","last_fullsized":"2024-08-17T18:18:30.719339+0000","mapping_epoch":1667,"log_start":"1613'3104","ondisk_log_start":"1613'3104","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5460,"log_dups_size":0,"ondisk_log_size":5460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70090752,"num_objects":1070,"num_object_clones":0,"num_object_copies":3210,"num_objects_missing_on_primary":308,"num_objects_missing":308,"num_objects_degraded":1378,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6108,"num_write_kb":229664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2512,"num_bytes_recovered":164560896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[4,7],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,2,4,5","objects":3},{"shards":"0,2,5","objects":308},{"shards":"0,4,5","objects":302},{"shards":"2,4,5","objects":441},{"shards":"4,7","objects":16}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"57.11","version":"1679'8870","reported_seq":9077,"reported_epoch":1679,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:19:26.067050+0000","last_change":"2024-08-17T18:18:59.889490+0000","last_active":"2024-08-17T18:19:26.067050+0000","last_peered":"2024-08-17T18:19:26.067050+0000","last_clean":"2024-08-17T18:14:57.659306+0000","last_became_active":"2024-08-17T18:18:29.827929+0000","last_became_peered":"2024-08-17T18:18:29.827929+0000","last_unstale":"2024-08-17T18:19:26.067050+0000","last_undegraded":"2024-08-17T18:19:26.067050+0000","last_fullsized":"2024-08-17T18:19:26.067050+0000","mapping_epoch":1666,"log_start":"1611'100","ondisk_log_start":"1611'100","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":8770,"log_dups_size":0,"ondisk_log_size":8770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67207168,"num_objects":1026,"num_object_clones":0,"num_object_copies":3078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3078,"num_objects_unfound":0,"num_objects_dirty":1026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6350,"num_write_kb":236000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[0,6,2],"avail_no_missing":["0","2","6"],"object_location_counts":[{"shards":"0,2,6","objects":1026}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"57.10","version":"1679'8669","reported_seq":20513,"reported_epoch":1679,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:19:22.029185+0000","last_change":"2024-08-17T18:18:31.925795+0000","last_active":"2024-08-17T18:19:22.029185+0000","last_peered":"2024-08-17T18:19:22.029185+0000","last_clean":"2024-08-17T18:15:52.133173+0000","last_became_active":"2024-08-17T18:18:31.867521+0000","last_became_peered":"2024-08-17T18:18:31.867521+0000","last_unstale":"2024-08-17T18:19:22.029185+0000","last_undegraded":"2024-08-17T18:18:31.852455+0000","last_fullsized":"2024-08-17T18:19:22.029185+0000","mapping_epoch":1668,"log_start":"1613'2044","ondisk_log_start":"1613'2044","created":1616,"last_epoch_clean":1632,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":6625,"log_dups_size":0,"ondisk_log_size":6625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68059136,"num_objects":1039,"num_object_clones":0,"num_object_copies":3117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1896,"num_objects_misplaced":1271,"num_objects_unfound":0,"num_objects_dirty":1039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6177,"num_write_kb":230880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4396,"num_bytes_recovered":288096256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,2],"avail_no_missing":["6","0","4","5"],"object_location_counts":[{"shards":"2,4,5,6","objects":313},{"shards":"2,5,6","objects":760},{"shards":"2,6,7","objects":-34}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"57.f","version":"1679'12497","reported_seq":12701,"reported_epoch":1679,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:19:22.350668+0000","last_change":"2024-08-17T18:15:53.709162+0000","last_active":"2024-08-17T18:19:22.350668+0000","last_peered":"2024-08-17T18:19:22.350668+0000","last_clean":"2024-08-17T18:15:09.658885+0000","last_became_active":"2024-08-17T18:15:53.364563+0000","last_became_peered":"2024-08-17T18:15:53.364563+0000","last_unstale":"2024-08-17T18:19:22.350668+0000","last_undegraded":"2024-08-17T18:19:22.350668+0000","last_fullsized":"2024-08-17T18:19:22.350668+0000","mapping_epoch":1637,"log_start":"1647'8258","ondisk_log_start":"1647'8258","created":1604,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4239,"log_dups_size":0,"ondisk_log_size":4239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144900096,"num_objects":2211,"num_object_clones":0,"num_object_copies":6633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2209,"num_objects_unfound":0,"num_objects_dirty":2211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12497,"num_write_kb":470656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":458752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,6,0],"avail_no_missing":["2","0","6"],"object_location_counts":[{"shards":"0,2,6","objects":2211}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"57.e","version":"1679'12404","reported_seq":12506,"reported_epoch":1679,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:19:24.831819+0000","last_change":"2024-08-17T18:17:23.713508+0000","last_active":"2024-08-17T18:19:24.831819+0000","last_peered":"2024-08-17T18:19:24.831819+0000","last_clean":"2024-08-17T18:15:52.176790+0000","last_became_active":"2024-08-17T18:17:23.688213+0000","last_became_peered":"2024-08-17T18:17:23.688213+0000","last_unstale":"2024-08-17T18:19:24.831819+0000","last_undegraded":"2024-08-17T18:19:24.831819+0000","last_fullsized":"2024-08-17T18:19:24.831819+0000","mapping_epoch":1653,"log_start":"1643'7818","ondisk_log_start":"1643'7818","created":1604,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4586,"log_dups_size":0,"ondisk_log_size":4586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:12:51.202869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140509184,"num_objects":2144,"num_object_clones":0,"num_object_copies":6432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2144,"num_objects_unfound":0,"num_objects_dirty":2144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12404,"num_write_kb":465536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[4,5,6],"avail_no_missing":["4","5","6"],"object_location_counts":[{"shards":"4,5,6","objects":2144}],"blocked_by":[],"up_primary":7,"acting_primary":4,"purged_snaps":[]},{"pgid":"57.d","version":"1679'12352","reported_seq":13505,"reported_epoch":1679,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:19:26.277090+0000","last_change":"2024-08-17T18:18:30.832971+0000","last_active":"2024-08-17T18:19:26.277090+0000","last_peered":"2024-08-17T18:19:26.277090+0000","last_clean":"2024-08-17T18:17:21.525603+0000","last_became_active":"2024-08-17T18:18:29.819768+0000","last_became_peered":"2024-08-17T18:18:29.819768+0000","last_unstale":"2024-08-17T18:19:26.277090+0000","last_undegraded":"2024-08-17T18:19:26.277090+0000","last_fullsized":"2024-08-17T18:19:26.277090+0000","mapping_epoch":1666,"log_start":"1617'5227","ondisk_log_start":"1617'5227","created":1604,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":7125,"log_dups_size":0,"ondisk_log_size":7125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139001880,"num_objects":2122,"num_object_clones":0,"num_object_copies":6366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1821,"num_objects_unfound":0,"num_objects_dirty":2122,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12352,"num_write_kb":463105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":508,"num_bytes_recovered":33292288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,2,6],"avail_no_missing":["0","2","6"],"object_location_counts":[{"shards":"0,2,6","objects":2122}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"57.c","version":"1679'12514","reported_seq":12600,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:22.059635+0000","last_change":"2024-08-17T18:13:54.756490+0000","last_active":"2024-08-17T18:19:22.059635+0000","last_peered":"2024-08-17T18:19:22.059635+0000","last_clean":"2024-08-17T18:19:22.059635+0000","last_became_active":"2024-08-17T18:13:54.756109+0000","last_became_peered":"2024-08-17T18:13:54.756109+0000","last_unstale":"2024-08-17T18:19:22.059635+0000","last_undegraded":"2024-08-17T18:19:22.059635+0000","last_fullsized":"2024-08-17T18:19:22.059635+0000","mapping_epoch":1604,"log_start":"1631'6933","ondisk_log_start":"1631'6933","created":1604,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5581,"log_dups_size":0,"ondisk_log_size":5581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:09:29.691722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147456000,"num_objects":2250,"num_object_clones":0,"num_object_copies":6750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12514,"num_write_kb":472448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"57.b","version":"1679'12218","reported_seq":12387,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:22.360017+0000","last_change":"2024-08-17T18:13:54.755437+0000","last_active":"2024-08-17T18:19:22.360017+0000","last_peered":"2024-08-17T18:19:22.360017+0000","last_clean":"2024-08-17T18:19:22.360017+0000","last_became_active":"2024-08-17T18:13:54.755175+0000","last_became_peered":"2024-08-17T18:13:54.755175+0000","last_unstale":"2024-08-17T18:19:22.360017+0000","last_undegraded":"2024-08-17T18:19:22.360017+0000","last_fullsized":"2024-08-17T18:19:22.360017+0000","mapping_epoch":1604,"log_start":"1644'7956","ondisk_log_start":"1644'7956","created":1604,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4262,"log_dups_size":0,"ondisk_log_size":4262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:27.516981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143523840,"num_objects":2190,"num_object_clones":0,"num_object_copies":6570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12218,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"57.a","version":"1679'12409","reported_seq":12519,"reported_epoch":1679,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:19:22.076267+0000","last_change":"2024-08-17T18:18:29.943391+0000","last_active":"2024-08-17T18:19:22.076267+0000","last_peered":"2024-08-17T18:19:22.076267+0000","last_clean":"2024-08-17T18:15:52.195344+0000","last_became_active":"2024-08-17T18:18:29.889053+0000","last_became_peered":"2024-08-17T18:18:29.889053+0000","last_unstale":"2024-08-17T18:19:22.076267+0000","last_undegraded":"2024-08-17T18:19:22.076267+0000","last_fullsized":"2024-08-17T18:19:22.076267+0000","mapping_epoch":1666,"log_start":"1648'8507","ondisk_log_start":"1648'8507","created":1604,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3902,"log_dups_size":0,"ondisk_log_size":3902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:46:14.340957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145424384,"num_objects":2219,"num_object_clones":0,"num_object_copies":6657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4438,"num_objects_unfound":0,"num_objects_dirty":2219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12409,"num_write_kb":468096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,4,2],"avail_no_missing":["5","2","4"],"object_location_counts":[{"shards":"2,4,5","objects":2219}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.9","version":"1679'8647","reported_seq":8751,"reported_epoch":1679,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:19:24.852858+0000","last_change":"2024-08-17T18:18:30.827730+0000","last_active":"2024-08-17T18:19:24.852858+0000","last_peered":"2024-08-17T18:19:24.852858+0000","last_clean":"2024-08-17T18:18:29.565716+0000","last_became_active":"2024-08-17T18:18:30.810210+0000","last_became_peered":"2024-08-17T18:18:30.810210+0000","last_unstale":"2024-08-17T18:19:24.852858+0000","last_undegraded":"2024-08-17T18:19:24.852858+0000","last_fullsized":"2024-08-17T18:19:24.852858+0000","mapping_epoch":1667,"log_start":"1613'3166","ondisk_log_start":"1613'3166","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5481,"log_dups_size":0,"ondisk_log_size":5481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71467008,"num_objects":1090,"num_object_clones":0,"num_object_copies":3270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1090,"num_objects_unfound":0,"num_objects_dirty":1090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6168,"num_write_kb":232288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[4,5,6],"avail_no_missing":["4","5","6"],"object_location_counts":[{"shards":"4,5,6","objects":1090}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"57.8","version":"1679'8597","reported_seq":15821,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:21.764627+0000","last_change":"2024-08-17T18:19:04.670956+0000","last_active":"2024-08-17T18:19:21.764627+0000","last_peered":"2024-08-17T18:19:21.764627+0000","last_clean":"2024-08-17T18:19:21.764627+0000","last_became_active":"2024-08-17T18:19:04.648021+0000","last_became_peered":"2024-08-17T18:19:04.648021+0000","last_unstale":"2024-08-17T18:19:21.764627+0000","last_undegraded":"2024-08-17T18:19:21.764627+0000","last_fullsized":"2024-08-17T18:19:21.764627+0000","mapping_epoch":1676,"log_start":"1613'3854","ondisk_log_start":"1613'3854","created":1604,"last_epoch_clean":1677,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4743,"log_dups_size":0,"ondisk_log_size":4743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71663616,"num_objects":1093,"num_object_clones":0,"num_object_copies":3279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6089,"num_write_kb":229856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3551,"num_bytes_recovered":109117440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"57.7","version":"1679'8496","reported_seq":14915,"reported_epoch":1679,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T18:19:22.370941+0000","last_change":"2024-08-17T18:19:22.371097+0000","last_active":"2024-08-17T18:19:22.370941+0000","last_peered":"2024-08-17T18:19:22.370941+0000","last_clean":"2024-08-17T18:15:52.213196+0000","last_became_active":"2024-08-17T18:17:23.719428+0000","last_became_peered":"2024-08-17T18:17:23.719428+0000","last_unstale":"2024-08-17T18:19:22.370941+0000","last_undegraded":"2024-08-17T18:19:22.370941+0000","last_fullsized":"2024-08-17T18:17:23.631126+0000","mapping_epoch":1653,"log_start":"1613'3759","ondisk_log_start":"1613'3759","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4737,"log_dups_size":0,"ondisk_log_size":4737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71467008,"num_objects":1090,"num_object_clones":0,"num_object_copies":3270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6064,"num_write_kb":228960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2997,"num_bytes_recovered":168165376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,7","objects":1090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"57.6","version":"1679'8446","reported_seq":8558,"reported_epoch":1679,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:19:22.068196+0000","last_change":"2024-08-17T18:18:29.888707+0000","last_active":"2024-08-17T18:19:22.068196+0000","last_peered":"2024-08-17T18:19:22.068196+0000","last_clean":"2024-08-17T18:15:09.669695+0000","last_became_active":"2024-08-17T18:15:53.694052+0000","last_became_peered":"2024-08-17T18:15:53.694052+0000","last_unstale":"2024-08-17T18:19:22.068196+0000","last_undegraded":"2024-08-17T18:15:53.322865+0000","last_fullsized":"2024-08-17T18:15:53.252808+0000","mapping_epoch":1637,"log_start":"1614'4326","ondisk_log_start":"1614'4326","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4120,"log_dups_size":0,"ondisk_log_size":4120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75530240,"num_objects":1152,"num_object_clones":0,"num_object_copies":3456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3544,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6010,"num_write_kb":229216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2],"avail_no_missing":["5","0","2"],"object_location_counts":[{"shards":"0,2,5","objects":687},{"shards":"2,4,5","objects":427},{"shards":"2,5","objects":-2193}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.5","version":"1679'8769","reported_seq":13758,"reported_epoch":1679,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:19:22.130701+0000","last_change":"2024-08-17T18:18:30.123049+0000","last_active":"2024-08-17T18:19:22.130701+0000","last_peered":"2024-08-17T18:19:22.130701+0000","last_clean":"2024-08-17T18:15:09.654427+0000","last_became_active":"2024-08-17T18:18:30.099398+0000","last_became_peered":"2024-08-17T18:18:30.099398+0000","last_unstale":"2024-08-17T18:19:22.130701+0000","last_undegraded":"2024-08-17T18:18:30.063992+0000","last_fullsized":"2024-08-17T18:19:22.130701+0000","mapping_epoch":1666,"log_start":"1615'4832","ondisk_log_start":"1615'4832","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3937,"log_dups_size":0,"ondisk_log_size":3937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65961984,"num_objects":1006,"num_object_clones":0,"num_object_copies":3018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":869,"num_objects_misplaced":1143,"num_objects_unfound":0,"num_objects_dirty":1006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6266,"num_write_kb":232736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2424,"num_bytes_recovered":158793728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,2,6],"avail_no_missing":["5","0","6"],"object_location_counts":[{"shards":"0,5,6","objects":662},{"shards":"2,5,6","objects":344}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.4","version":"1679'8577","reported_seq":15925,"reported_epoch":1679,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:19:24.827755+0000","last_change":"2024-08-17T18:18:32.134646+0000","last_active":"2024-08-17T18:19:24.827755+0000","last_peered":"2024-08-17T18:19:24.827755+0000","last_clean":"2024-08-17T18:18:29.526320+0000","last_became_active":"2024-08-17T18:18:32.122736+0000","last_became_peered":"2024-08-17T18:18:32.122736+0000","last_unstale":"2024-08-17T18:19:24.827755+0000","last_undegraded":"2024-08-17T18:19:24.827755+0000","last_fullsized":"2024-08-17T18:19:24.827755+0000","mapping_epoch":1668,"log_start":"1613'3186","ondisk_log_start":"1613'3186","created":1604,"last_epoch_clean":1647,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5391,"log_dups_size":0,"ondisk_log_size":5391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69599232,"num_objects":1062,"num_object_clones":0,"num_object_copies":3186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1062,"num_objects_unfound":0,"num_objects_dirty":1062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6124,"num_write_kb":229952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3591,"num_bytes_recovered":223805440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[4,5,7],"avail_no_missing":["4","5","7"],"object_location_counts":[{"shards":"4,5,7","objects":1062}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"57.3","version":"1679'8567","reported_seq":11786,"reported_epoch":1679,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:19:22.088501+0000","last_change":"2024-08-17T18:18:31.936546+0000","last_active":"2024-08-17T18:19:22.088501+0000","last_peered":"2024-08-17T18:19:22.088501+0000","last_clean":"2024-08-17T18:15:52.162248+0000","last_became_active":"2024-08-17T18:18:31.914458+0000","last_became_peered":"2024-08-17T18:18:31.914458+0000","last_unstale":"2024-08-17T18:19:22.088501+0000","last_undegraded":"2024-08-17T18:18:31.855752+0000","last_fullsized":"2024-08-17T18:19:22.088501+0000","mapping_epoch":1668,"log_start":"1613'1949","ondisk_log_start":"1613'1949","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":6618,"log_dups_size":0,"ondisk_log_size":6618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72843264,"num_objects":1111,"num_object_clones":0,"num_object_copies":3333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1912,"num_objects_misplaced":1313,"num_objects_unfound":0,"num_objects_dirty":1111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6117,"num_write_kb":231328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1523,"num_bytes_recovered":99811328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[6,7,0],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"0,2,6","objects":1132},{"shards":"0,6,7","objects":-21}],"blocked_by":[],"up_primary":3,"acting_primary":6,"purged_snaps":[]},{"pgid":"57.2","version":"1679'8596","reported_seq":10403,"reported_epoch":1679,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:19:22.273951+0000","last_change":"2024-08-17T18:18:33.840312+0000","last_active":"2024-08-17T18:19:22.273951+0000","last_peered":"2024-08-17T18:19:22.273951+0000","last_clean":"2024-08-17T18:15:09.574793+0000","last_became_active":"2024-08-17T18:18:33.824434+0000","last_became_peered":"2024-08-17T18:18:33.824434+0000","last_unstale":"2024-08-17T18:19:22.273951+0000","last_undegraded":"2024-08-17T18:18:30.818589+0000","last_fullsized":"2024-08-17T18:19:22.273951+0000","mapping_epoch":1667,"log_start":"1614'3978","ondisk_log_start":"1614'3978","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74416128,"num_objects":1135,"num_object_clones":0,"num_object_copies":3405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6139,"num_write_kb":232800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":803,"num_bytes_recovered":52625408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[2,5,4],"avail_no_missing":["2","0","5"],"object_location_counts":[{"shards":"0,2,5","objects":682},{"shards":"2,4,5","objects":453}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"57.1","version":"1679'8726","reported_seq":8898,"reported_epoch":1679,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:19:22.104299+0000","last_change":"2024-08-17T18:15:53.720586+0000","last_active":"2024-08-17T18:19:22.104299+0000","last_peered":"2024-08-17T18:19:22.104299+0000","last_clean":"2024-08-17T18:15:09.579791+0000","last_became_active":"2024-08-17T18:15:53.381797+0000","last_became_peered":"2024-08-17T18:15:53.381797+0000","last_unstale":"2024-08-17T18:19:22.104299+0000","last_undegraded":"2024-08-17T18:19:22.104299+0000","last_fullsized":"2024-08-17T18:19:22.104299+0000","mapping_epoch":1637,"log_start":"1613'2059","ondisk_log_start":"1613'2059","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":6667,"log_dups_size":0,"ondisk_log_size":6667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77299712,"num_objects":1179,"num_object_clones":0,"num_object_copies":3537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1179,"num_objects_unfound":0,"num_objects_dirty":1179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6205,"num_write_kb":236320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":589824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,0],"avail_no_missing":["6","0","2"],"object_location_counts":[{"shards":"0,2,6","objects":1179}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"57.0","version":"1679'8633","reported_seq":22399,"reported_epoch":1679,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:19:22.354222+0000","last_change":"2024-08-17T18:18:31.919624+0000","last_active":"2024-08-17T18:19:22.354222+0000","last_peered":"2024-08-17T18:19:22.354222+0000","last_clean":"2024-08-17T18:18:29.609086+0000","last_became_active":"2024-08-17T18:18:31.860120+0000","last_became_peered":"2024-08-17T18:18:31.860120+0000","last_unstale":"2024-08-17T18:19:22.354222+0000","last_undegraded":"2024-08-17T18:19:22.354222+0000","last_fullsized":"2024-08-17T18:19:22.354222+0000","mapping_epoch":1668,"log_start":"1613'3925","ondisk_log_start":"1613'3925","created":1604,"last_epoch_clean":1665,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4708,"log_dups_size":0,"ondisk_log_size":4708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73826304,"num_objects":1126,"num_object_clones":0,"num_object_copies":3378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1126,"num_objects_unfound":0,"num_objects_dirty":1126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6140,"num_write_kb":232544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6772,"num_bytes_recovered":353239040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,7],"avail_no_missing":["2","0","4","5","7"],"object_location_counts":[{"shards":"2,4,7","objects":1126}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":116,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.594580+0000","last_change":"2024-08-17T18:17:22.697769+0000","last_active":"2024-08-17T18:19:15.594580+0000","last_peered":"2024-08-17T18:19:15.594580+0000","last_clean":"2024-08-17T18:19:15.594580+0000","last_became_active":"2024-08-17T18:17:22.688989+0000","last_became_peered":"2024-08-17T18:17:22.688989+0000","last_unstale":"2024-08-17T18:19:15.594580+0000","last_undegraded":"2024-08-17T18:19:15.594580+0000","last_fullsized":"2024-08-17T18:19:15.594580+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:10:04.677811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2328,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1722,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.255610+0000","last_change":"2024-08-17T18:18:29.727855+0000","last_active":"2024-08-17T18:19:07.255610+0000","last_peered":"2024-08-17T18:19:07.255610+0000","last_clean":"2024-08-17T18:19:07.255610+0000","last_became_active":"2024-08-17T18:18:29.727091+0000","last_became_peered":"2024-08-17T18:18:29.727091+0000","last_unstale":"2024-08-17T18:19:07.255610+0000","last_undegraded":"2024-08-17T18:19:07.255610+0000","last_fullsized":"2024-08-17T18:19:07.255610+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:04.660094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":36,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.601913+0000","last_change":"2024-08-17T18:18:29.895953+0000","last_active":"2024-08-17T18:19:15.601913+0000","last_peered":"2024-08-17T18:19:15.601913+0000","last_clean":"2024-08-17T18:19:15.601913+0000","last_became_active":"2024-08-17T18:18:29.895460+0000","last_became_peered":"2024-08-17T18:18:29.895460+0000","last_unstale":"2024-08-17T18:19:15.601913+0000","last_undegraded":"2024-08-17T18:19:15.601913+0000","last_fullsized":"2024-08-17T18:19:15.601913+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:04:41.818991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":36,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.602786+0000","last_change":"2024-08-17T18:18:29.732425+0000","last_active":"2024-08-17T18:19:15.602786+0000","last_peered":"2024-08-17T18:19:15.602786+0000","last_clean":"2024-08-17T18:19:15.602786+0000","last_became_active":"2024-08-17T18:18:29.731726+0000","last_became_peered":"2024-08-17T18:18:29.731726+0000","last_unstale":"2024-08-17T18:19:15.602786+0000","last_undegraded":"2024-08-17T18:19:15.602786+0000","last_fullsized":"2024-08-17T18:19:15.602786+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:01:33.556710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":234,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.264649+0000","last_change":"2024-08-17T18:04:43.777705+0000","last_active":"2024-08-17T18:19:07.264649+0000","last_peered":"2024-08-17T18:19:07.264649+0000","last_clean":"2024-08-17T18:19:07.264649+0000","last_became_active":"2024-08-17T18:04:43.771283+0000","last_became_peered":"2024-08-17T18:04:43.771283+0000","last_unstale":"2024-08-17T18:19:07.264649+0000","last_undegraded":"2024-08-17T18:19:07.264649+0000","last_fullsized":"2024-08-17T18:19:07.264649+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:13.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":36,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.603063+0000","last_change":"2024-08-17T18:18:29.734730+0000","last_active":"2024-08-17T18:19:15.603063+0000","last_peered":"2024-08-17T18:19:15.603063+0000","last_clean":"2024-08-17T18:19:15.603063+0000","last_became_active":"2024-08-17T18:18:29.734362+0000","last_became_peered":"2024-08-17T18:18:29.734362+0000","last_unstale":"2024-08-17T18:19:15.603063+0000","last_undegraded":"2024-08-17T18:19:15.603063+0000","last_fullsized":"2024-08-17T18:19:15.603063+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:58:48.630762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":247,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.259720+0000","last_change":"2024-08-17T18:12:04.655551+0000","last_active":"2024-08-17T18:19:07.259720+0000","last_peered":"2024-08-17T18:19:07.259720+0000","last_clean":"2024-08-17T18:19:07.259720+0000","last_became_active":"2024-08-17T18:12:04.646075+0000","last_became_peered":"2024-08-17T18:12:04.646075+0000","last_unstale":"2024-08-17T18:19:07.259720+0000","last_undegraded":"2024-08-17T18:19:07.259720+0000","last_fullsized":"2024-08-17T18:19:07.259720+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:19:19.300135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1718,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.255192+0000","last_change":"2024-08-17T18:18:29.727683+0000","last_active":"2024-08-17T18:19:07.255192+0000","last_peered":"2024-08-17T18:19:07.255192+0000","last_clean":"2024-08-17T18:19:07.255192+0000","last_became_active":"2024-08-17T18:18:29.727378+0000","last_became_peered":"2024-08-17T18:18:29.727378+0000","last_unstale":"2024-08-17T18:19:07.255192+0000","last_undegraded":"2024-08-17T18:19:07.255192+0000","last_fullsized":"2024-08-17T18:19:07.255192+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:42.989331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":205,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.255110+0000","last_change":"2024-08-17T18:18:29.882947+0000","last_active":"2024-08-17T18:19:07.255110+0000","last_peered":"2024-08-17T18:19:07.255110+0000","last_clean":"2024-08-17T18:19:07.255110+0000","last_became_active":"2024-08-17T18:18:29.882533+0000","last_became_peered":"2024-08-17T18:18:29.882533+0000","last_unstale":"2024-08-17T18:19:07.255110+0000","last_undegraded":"2024-08-17T18:19:07.255110+0000","last_fullsized":"2024-08-17T18:19:07.255110+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:34:25.726288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":37,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.602224+0000","last_change":"2024-08-17T18:18:29.734046+0000","last_active":"2024-08-17T18:19:15.602224+0000","last_peered":"2024-08-17T18:19:15.602224+0000","last_clean":"2024-08-17T18:19:15.602224+0000","last_became_active":"2024-08-17T18:18:29.733663+0000","last_became_peered":"2024-08-17T18:18:29.733663+0000","last_unstale":"2024-08-17T18:19:15.602224+0000","last_undegraded":"2024-08-17T18:19:15.602224+0000","last_fullsized":"2024-08-17T18:19:15.602224+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:08:09.146146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1713,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.274256+0000","last_change":"2024-08-17T18:16:00.324143+0000","last_active":"2024-08-17T18:19:07.274256+0000","last_peered":"2024-08-17T18:19:07.274256+0000","last_clean":"2024-08-17T18:19:07.274256+0000","last_became_active":"2024-08-17T18:15:57.106000+0000","last_became_peered":"2024-08-17T18:15:57.106000+0000","last_unstale":"2024-08-17T18:19:07.274256+0000","last_undegraded":"2024-08-17T18:19:07.274256+0000","last_fullsized":"2024-08-17T18:19:07.274256+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1641,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:43.117961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":36,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.603394+0000","last_change":"2024-08-17T18:18:29.733983+0000","last_active":"2024-08-17T18:19:15.603394+0000","last_peered":"2024-08-17T18:19:15.603394+0000","last_clean":"2024-08-17T18:19:15.603394+0000","last_became_active":"2024-08-17T18:18:29.733635+0000","last_became_peered":"2024-08-17T18:18:29.733635+0000","last_unstale":"2024-08-17T18:19:15.603394+0000","last_undegraded":"2024-08-17T18:19:15.603394+0000","last_fullsized":"2024-08-17T18:19:15.603394+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:47:15.631401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":36,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.602638+0000","last_change":"2024-08-17T18:18:29.888141+0000","last_active":"2024-08-17T18:19:15.602638+0000","last_peered":"2024-08-17T18:19:15.602638+0000","last_clean":"2024-08-17T18:19:15.602638+0000","last_became_active":"2024-08-17T18:18:29.887829+0000","last_became_peered":"2024-08-17T18:18:29.887829+0000","last_unstale":"2024-08-17T18:19:15.602638+0000","last_undegraded":"2024-08-17T18:19:15.602638+0000","last_fullsized":"2024-08-17T18:19:15.602638+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:23:13.375118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":36,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.602017+0000","last_change":"2024-08-17T18:18:29.896201+0000","last_active":"2024-08-17T18:19:15.602017+0000","last_peered":"2024-08-17T18:19:15.602017+0000","last_clean":"2024-08-17T18:19:15.602017+0000","last_became_active":"2024-08-17T18:18:29.895687+0000","last_became_peered":"2024-08-17T18:18:29.895687+0000","last_unstale":"2024-08-17T18:19:15.602017+0000","last_undegraded":"2024-08-17T18:19:15.602017+0000","last_fullsized":"2024-08-17T18:19:15.602017+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:24:52.525561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":36,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.601791+0000","last_change":"2024-08-17T18:18:29.888209+0000","last_active":"2024-08-17T18:19:15.601791+0000","last_peered":"2024-08-17T18:19:15.601791+0000","last_clean":"2024-08-17T18:19:15.601791+0000","last_became_active":"2024-08-17T18:18:29.887651+0000","last_became_peered":"2024-08-17T18:18:29.887651+0000","last_unstale":"2024-08-17T18:19:15.601791+0000","last_undegraded":"2024-08-17T18:19:15.601791+0000","last_fullsized":"2024-08-17T18:19:15.601791+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:55:59.802775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1724,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.256245+0000","last_change":"2024-08-17T18:18:29.887228+0000","last_active":"2024-08-17T18:19:07.256245+0000","last_peered":"2024-08-17T18:19:07.256245+0000","last_clean":"2024-08-17T18:19:07.256245+0000","last_became_active":"2024-08-17T18:18:29.884394+0000","last_became_peered":"2024-08-17T18:18:29.884394+0000","last_unstale":"2024-08-17T18:19:07.256245+0000","last_undegraded":"2024-08-17T18:19:07.256245+0000","last_fullsized":"2024-08-17T18:19:07.256245+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:41:42.256082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":36,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.601874+0000","last_change":"2024-08-17T18:18:29.727770+0000","last_active":"2024-08-17T18:19:15.601874+0000","last_peered":"2024-08-17T18:19:15.601874+0000","last_clean":"2024-08-17T18:19:15.601874+0000","last_became_active":"2024-08-17T18:18:29.727324+0000","last_became_peered":"2024-08-17T18:18:29.727324+0000","last_unstale":"2024-08-17T18:19:15.601874+0000","last_undegraded":"2024-08-17T18:19:15.601874+0000","last_fullsized":"2024-08-17T18:19:15.601874+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:20:29.342237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":405,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.262130+0000","last_change":"2024-08-17T18:18:29.734707+0000","last_active":"2024-08-17T18:19:07.262130+0000","last_peered":"2024-08-17T18:19:07.262130+0000","last_clean":"2024-08-17T18:19:07.262130+0000","last_became_active":"2024-08-17T18:18:29.734420+0000","last_became_peered":"2024-08-17T18:18:29.734420+0000","last_unstale":"2024-08-17T18:19:07.262130+0000","last_undegraded":"2024-08-17T18:19:07.262130+0000","last_fullsized":"2024-08-17T18:19:07.262130+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:07:29.048061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":36,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.602607+0000","last_change":"2024-08-17T18:18:29.880752+0000","last_active":"2024-08-17T18:19:15.602607+0000","last_peered":"2024-08-17T18:19:15.602607+0000","last_clean":"2024-08-17T18:19:15.602607+0000","last_became_active":"2024-08-17T18:18:29.880508+0000","last_became_peered":"2024-08-17T18:18:29.880508+0000","last_unstale":"2024-08-17T18:19:15.602607+0000","last_undegraded":"2024-08-17T18:19:15.602607+0000","last_fullsized":"2024-08-17T18:19:15.602607+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:35:25.743073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":293,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:12.157229+0000","last_change":"2024-08-17T18:18:29.900929+0000","last_active":"2024-08-17T18:19:12.157229+0000","last_peered":"2024-08-17T18:19:12.157229+0000","last_clean":"2024-08-17T18:19:12.157229+0000","last_became_active":"2024-08-17T18:18:29.900417+0000","last_became_peered":"2024-08-17T18:18:29.900417+0000","last_unstale":"2024-08-17T18:19:12.157229+0000","last_undegraded":"2024-08-17T18:19:12.157229+0000","last_fullsized":"2024-08-17T18:19:12.157229+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:07:45.752696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":42,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.603392+0000","last_change":"2024-08-17T18:18:29.928751+0000","last_active":"2024-08-17T18:19:15.603392+0000","last_peered":"2024-08-17T18:19:15.603392+0000","last_clean":"2024-08-17T18:19:15.603392+0000","last_became_active":"2024-08-17T18:18:29.898102+0000","last_became_peered":"2024-08-17T18:18:29.898102+0000","last_unstale":"2024-08-17T18:19:15.603392+0000","last_undegraded":"2024-08-17T18:19:15.603392+0000","last_fullsized":"2024-08-17T18:19:15.603392+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:21.098266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2506,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.574234+0000","last_change":"2024-08-17T18:17:22.685988+0000","last_active":"2024-08-17T18:19:15.574234+0000","last_peered":"2024-08-17T18:19:15.574234+0000","last_clean":"2024-08-17T18:19:15.574234+0000","last_became_active":"2024-08-17T18:17:22.685791+0000","last_became_peered":"2024-08-17T18:17:22.685791+0000","last_unstale":"2024-08-17T18:19:15.574234+0000","last_undegraded":"2024-08-17T18:19:15.574234+0000","last_fullsized":"2024-08-17T18:19:15.574234+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:10:28.276954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1707,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.256781+0000","last_change":"2024-08-17T18:15:53.687128+0000","last_active":"2024-08-17T18:19:07.256781+0000","last_peered":"2024-08-17T18:19:07.256781+0000","last_clean":"2024-08-17T18:19:07.256781+0000","last_became_active":"2024-08-17T18:15:53.246153+0000","last_became_peered":"2024-08-17T18:15:53.246153+0000","last_unstale":"2024-08-17T18:19:07.256781+0000","last_undegraded":"2024-08-17T18:19:07.256781+0000","last_fullsized":"2024-08-17T18:19:07.256781+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:45:14.801100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1669,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.255266+0000","last_change":"2024-08-17T18:12:04.639269+0000","last_active":"2024-08-17T18:19:07.255266+0000","last_peered":"2024-08-17T18:19:07.255266+0000","last_clean":"2024-08-17T18:19:07.255266+0000","last_became_active":"2024-08-17T18:12:04.638752+0000","last_became_peered":"2024-08-17T18:12:04.638752+0000","last_unstale":"2024-08-17T18:19:07.255266+0000","last_undegraded":"2024-08-17T18:19:07.255266+0000","last_fullsized":"2024-08-17T18:19:07.255266+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:50:00.462972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"347'149","reported_seq":906,"reported_epoch":1678,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:19:07.259467+0000","last_change":"2024-08-17T18:18:30.750089+0000","last_active":"2024-08-17T18:19:07.259467+0000","last_peered":"2024-08-17T18:19:07.259467+0000","last_clean":"2024-08-17T18:15:50.198296+0000","last_became_active":"2024-08-17T18:18:30.749746+0000","last_became_peered":"2024-08-17T18:18:30.749746+0000","last_unstale":"2024-08-17T18:19:07.259467+0000","last_undegraded":"2024-08-17T18:18:30.723633+0000","last_fullsized":"2024-08-17T18:18:30.723013+0000","mapping_epoch":1667,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1612,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:57.271286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:57.271286+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:22.670286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[5,7],"avail_no_missing":["5","4","6"],"object_location_counts":[{"shards":"4,5,6","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1685,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.255838+0000","last_change":"2024-08-17T18:04:43.766724+0000","last_active":"2024-08-17T18:19:07.255838+0000","last_peered":"2024-08-17T18:19:07.255838+0000","last_clean":"2024-08-17T18:19:07.255838+0000","last_became_active":"2024-08-17T18:04:43.766428+0000","last_became_peered":"2024-08-17T18:04:43.766428+0000","last_unstale":"2024-08-17T18:19:07.255838+0000","last_undegraded":"2024-08-17T18:19:07.255838+0000","last_fullsized":"2024-08-17T18:19:07.255838+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:09:49.102940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1715,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.256338+0000","last_change":"2024-08-17T18:15:53.366349+0000","last_active":"2024-08-17T18:19:07.256338+0000","last_peered":"2024-08-17T18:19:07.256338+0000","last_clean":"2024-08-17T18:19:07.256338+0000","last_became_active":"2024-08-17T18:15:53.365863+0000","last_became_peered":"2024-08-17T18:15:53.365863+0000","last_unstale":"2024-08-17T18:19:07.256338+0000","last_undegraded":"2024-08-17T18:19:07.256338+0000","last_fullsized":"2024-08-17T18:19:07.256338+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:52.700374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":107,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.257171+0000","last_change":"2024-08-17T18:17:22.720026+0000","last_active":"2024-08-17T18:19:07.257171+0000","last_peered":"2024-08-17T18:19:07.257171+0000","last_clean":"2024-08-17T18:19:07.257171+0000","last_became_active":"2024-08-17T18:17:22.662636+0000","last_became_peered":"2024-08-17T18:17:22.662636+0000","last_unstale":"2024-08-17T18:19:07.257171+0000","last_undegraded":"2024-08-17T18:19:07.257171+0000","last_fullsized":"2024-08-17T18:19:07.257171+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:37:15.928482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"347'149","reported_seq":971,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.255672+0000","last_change":"2024-08-17T18:17:25.672410+0000","last_active":"2024-08-17T18:19:07.255672+0000","last_peered":"2024-08-17T18:19:07.255672+0000","last_clean":"2024-08-17T18:19:07.255672+0000","last_became_active":"2024-08-17T18:17:25.671993+0000","last_became_peered":"2024-08-17T18:17:25.671993+0000","last_unstale":"2024-08-17T18:19:07.255672+0000","last_undegraded":"2024-08-17T18:19:07.255672+0000","last_fullsized":"2024-08-17T18:19:07.255672+0000","mapping_epoch":1655,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1656,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:53.309517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:53.309517+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:21:08.929478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":36,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.602864+0000","last_change":"2024-08-17T18:18:29.888368+0000","last_active":"2024-08-17T18:19:15.602864+0000","last_peered":"2024-08-17T18:19:15.602864+0000","last_clean":"2024-08-17T18:19:15.602864+0000","last_became_active":"2024-08-17T18:18:29.887731+0000","last_became_peered":"2024-08-17T18:18:29.887731+0000","last_unstale":"2024-08-17T18:19:15.602864+0000","last_undegraded":"2024-08-17T18:19:15.602864+0000","last_fullsized":"2024-08-17T18:19:15.602864+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:47:07.960969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":767,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.256669+0000","last_change":"2024-08-17T18:18:29.728878+0000","last_active":"2024-08-17T18:19:07.256669+0000","last_peered":"2024-08-17T18:19:07.256669+0000","last_clean":"2024-08-17T18:19:07.256669+0000","last_became_active":"2024-08-17T18:18:29.728295+0000","last_became_peered":"2024-08-17T18:18:29.728295+0000","last_unstale":"2024-08-17T18:19:07.256669+0000","last_undegraded":"2024-08-17T18:19:07.256669+0000","last_fullsized":"2024-08-17T18:19:07.256669+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:37:24.603653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":41,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.602217+0000","last_change":"2024-08-17T18:18:29.904137+0000","last_active":"2024-08-17T18:19:15.602217+0000","last_peered":"2024-08-17T18:19:15.602217+0000","last_clean":"2024-08-17T18:19:15.602217+0000","last_became_active":"2024-08-17T18:18:29.894843+0000","last_became_peered":"2024-08-17T18:18:29.894843+0000","last_unstale":"2024-08-17T18:19:15.602217+0000","last_undegraded":"2024-08-17T18:19:15.602217+0000","last_fullsized":"2024-08-17T18:19:15.602217+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:23:04.104922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"347'149","reported_seq":1005,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.256593+0000","last_change":"2024-08-17T18:18:33.761955+0000","last_active":"2024-08-17T18:19:07.256593+0000","last_peered":"2024-08-17T18:19:07.256593+0000","last_clean":"2024-08-17T18:19:07.256593+0000","last_became_active":"2024-08-17T18:18:33.761579+0000","last_became_peered":"2024-08-17T18:18:33.761579+0000","last_unstale":"2024-08-17T18:19:07.256593+0000","last_undegraded":"2024-08-17T18:19:07.256593+0000","last_fullsized":"2024-08-17T18:19:07.256593+0000","mapping_epoch":1670,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1671,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:52.474544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:52.474544+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:35:23.712308+0000","scrub_duration":0.001886326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":294,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.259005+0000","last_change":"2024-08-17T18:17:22.690072+0000","last_active":"2024-08-17T18:19:07.259005+0000","last_peered":"2024-08-17T18:19:07.259005+0000","last_clean":"2024-08-17T18:19:07.259005+0000","last_became_active":"2024-08-17T18:17:22.688530+0000","last_became_peered":"2024-08-17T18:17:22.688530+0000","last_unstale":"2024-08-17T18:19:07.259005+0000","last_undegraded":"2024-08-17T18:19:07.259005+0000","last_fullsized":"2024-08-17T18:19:07.259005+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:40:26.170186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":52,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.246844+0000","last_change":"2024-08-17T18:15:53.248168+0000","last_active":"2024-08-17T18:19:07.246844+0000","last_peered":"2024-08-17T18:19:07.246844+0000","last_clean":"2024-08-17T18:19:07.246844+0000","last_became_active":"2024-08-17T18:15:53.247888+0000","last_became_peered":"2024-08-17T18:15:53.247888+0000","last_unstale":"2024-08-17T18:19:07.246844+0000","last_undegraded":"2024-08-17T18:19:07.246844+0000","last_fullsized":"2024-08-17T18:19:07.246844+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:30:42.725233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2535,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.573680+0000","last_change":"2024-08-17T18:15:10.692541+0000","last_active":"2024-08-17T18:19:15.573680+0000","last_peered":"2024-08-17T18:19:15.573680+0000","last_clean":"2024-08-17T18:19:15.573680+0000","last_became_active":"2024-08-17T18:15:10.691876+0000","last_became_peered":"2024-08-17T18:15:10.691876+0000","last_unstale":"2024-08-17T18:19:15.573680+0000","last_undegraded":"2024-08-17T18:19:15.573680+0000","last_fullsized":"2024-08-17T18:19:15.573680+0000","mapping_epoch":1626,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:39:22.253615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"347'149","reported_seq":988,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.256103+0000","last_change":"2024-08-17T18:18:33.761996+0000","last_active":"2024-08-17T18:19:07.256103+0000","last_peered":"2024-08-17T18:19:07.256103+0000","last_clean":"2024-08-17T18:19:07.256103+0000","last_became_active":"2024-08-17T18:18:33.761606+0000","last_became_peered":"2024-08-17T18:18:33.761606+0000","last_unstale":"2024-08-17T18:19:07.256103+0000","last_undegraded":"2024-08-17T18:19:07.256103+0000","last_fullsized":"2024-08-17T18:19:07.256103+0000","mapping_epoch":1670,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1671,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:49.461698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:49.461698+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:29:01.698411+0000","scrub_duration":0.0013792520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":245,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.260402+0000","last_change":"2024-08-17T18:17:22.687498+0000","last_active":"2024-08-17T18:19:07.260402+0000","last_peered":"2024-08-17T18:19:07.260402+0000","last_clean":"2024-08-17T18:19:07.260402+0000","last_became_active":"2024-08-17T18:17:22.687176+0000","last_became_peered":"2024-08-17T18:17:22.687176+0000","last_unstale":"2024-08-17T18:19:07.260402+0000","last_undegraded":"2024-08-17T18:19:07.260402+0000","last_fullsized":"2024-08-17T18:19:07.260402+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:32:42.738739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2544,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.583849+0000","last_change":"2024-08-17T18:15:53.230320+0000","last_active":"2024-08-17T18:19:15.583849+0000","last_peered":"2024-08-17T18:19:15.583849+0000","last_clean":"2024-08-17T18:19:15.583849+0000","last_became_active":"2024-08-17T18:15:53.230016+0000","last_became_peered":"2024-08-17T18:15:53.230016+0000","last_unstale":"2024-08-17T18:19:15.583849+0000","last_undegraded":"2024-08-17T18:19:15.583849+0000","last_fullsized":"2024-08-17T18:19:15.583849+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:15:28.925103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":249,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.264721+0000","last_change":"2024-08-17T18:18:29.735362+0000","last_active":"2024-08-17T18:19:07.264721+0000","last_peered":"2024-08-17T18:19:07.264721+0000","last_clean":"2024-08-17T18:19:07.264721+0000","last_became_active":"2024-08-17T18:18:29.734752+0000","last_became_peered":"2024-08-17T18:18:29.734752+0000","last_unstale":"2024-08-17T18:19:07.264721+0000","last_undegraded":"2024-08-17T18:19:07.264721+0000","last_fullsized":"2024-08-17T18:19:07.264721+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:25.709698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"347'149","reported_seq":972,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.260039+0000","last_change":"2024-08-17T18:15:13.681264+0000","last_active":"2024-08-17T18:19:07.260039+0000","last_peered":"2024-08-17T18:19:07.260039+0000","last_clean":"2024-08-17T18:19:07.260039+0000","last_became_active":"2024-08-17T18:15:13.680918+0000","last_became_peered":"2024-08-17T18:15:13.680918+0000","last_unstale":"2024-08-17T18:19:07.260039+0000","last_undegraded":"2024-08-17T18:19:07.260039+0000","last_fullsized":"2024-08-17T18:19:07.260039+0000","mapping_epoch":1629,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1630,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:00.316884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:00.316884+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:37:32.924833+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1683,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.573499+0000","last_change":"2024-08-17T18:18:29.889561+0000","last_active":"2024-08-17T18:19:15.573499+0000","last_peered":"2024-08-17T18:19:15.573499+0000","last_clean":"2024-08-17T18:19:15.573499+0000","last_became_active":"2024-08-17T18:18:29.889192+0000","last_became_peered":"2024-08-17T18:18:29.889192+0000","last_unstale":"2024-08-17T18:19:15.573499+0000","last_undegraded":"2024-08-17T18:19:15.573499+0000","last_fullsized":"2024-08-17T18:19:15.573499+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:06:06.579862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1692,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.255404+0000","last_change":"2024-08-17T18:15:10.695565+0000","last_active":"2024-08-17T18:19:07.255404+0000","last_peered":"2024-08-17T18:19:07.255404+0000","last_clean":"2024-08-17T18:19:07.255404+0000","last_became_active":"2024-08-17T18:15:10.695225+0000","last_became_peered":"2024-08-17T18:15:10.695225+0000","last_unstale":"2024-08-17T18:19:07.255404+0000","last_undegraded":"2024-08-17T18:19:07.255404+0000","last_fullsized":"2024-08-17T18:19:07.255404+0000","mapping_epoch":1626,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:28:55.675702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":285,"reported_epoch":1679,"state":"active+recovery_wait+degraded","last_fresh":"2024-08-17T18:19:15.591693+0000","last_change":"2024-08-17T18:18:29.800821+0000","last_active":"2024-08-17T18:19:15.591693+0000","last_peered":"2024-08-17T18:19:15.591693+0000","last_clean":"2024-08-17T18:18:25.612782+0000","last_became_active":"2024-08-17T18:18:29.800502+0000","last_became_peered":"2024-08-17T18:18:29.800502+0000","last_unstale":"2024-08-17T18:19:15.591693+0000","last_undegraded":"2024-08-17T18:18:29.738218+0000","last_fullsized":"2024-08-17T18:19:15.591693+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:08:12.507444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":95,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":["5","2","6","7"],"object_location_counts":[{"shards":"2,5,6,7","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"347'149","reported_seq":832,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.671259+0000","last_change":"2024-08-17T18:17:22.681616+0000","last_active":"2024-08-17T18:19:15.671259+0000","last_peered":"2024-08-17T18:19:15.671259+0000","last_clean":"2024-08-17T18:19:15.671259+0000","last_became_active":"2024-08-17T18:17:22.681184+0000","last_became_peered":"2024-08-17T18:17:22.681184+0000","last_unstale":"2024-08-17T18:19:15.671259+0000","last_undegraded":"2024-08-17T18:19:15.671259+0000","last_fullsized":"2024-08-17T18:19:15.671259+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:02.363531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:02.363531+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:50:39.567707+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2441,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.262199+0000","last_change":"2024-08-17T18:18:29.898545+0000","last_active":"2024-08-17T18:19:07.262199+0000","last_peered":"2024-08-17T18:19:07.262199+0000","last_clean":"2024-08-17T18:19:07.262199+0000","last_became_active":"2024-08-17T18:18:29.896509+0000","last_became_peered":"2024-08-17T18:18:29.896509+0000","last_unstale":"2024-08-17T18:19:07.262199+0000","last_undegraded":"2024-08-17T18:19:07.262199+0000","last_fullsized":"2024-08-17T18:19:07.262199+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:37:09.107372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2356,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.573848+0000","last_change":"2024-08-17T18:12:04.648190+0000","last_active":"2024-08-17T18:19:15.573848+0000","last_peered":"2024-08-17T18:19:15.573848+0000","last_clean":"2024-08-17T18:19:15.573848+0000","last_became_active":"2024-08-17T18:12:04.647877+0000","last_became_peered":"2024-08-17T18:12:04.647877+0000","last_unstale":"2024-08-17T18:19:15.573848+0000","last_undegraded":"2024-08-17T18:19:15.573848+0000","last_fullsized":"2024-08-17T18:19:15.573848+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:18.129337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2658,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.557878+0000","last_change":"2024-08-17T18:18:29.742742+0000","last_active":"2024-08-17T18:19:15.557878+0000","last_peered":"2024-08-17T18:19:15.557878+0000","last_clean":"2024-08-17T18:19:15.557878+0000","last_became_active":"2024-08-17T18:18:29.742393+0000","last_became_peered":"2024-08-17T18:18:29.742393+0000","last_unstale":"2024-08-17T18:19:15.557878+0000","last_undegraded":"2024-08-17T18:19:15.557878+0000","last_fullsized":"2024-08-17T18:19:15.557878+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:57:32.858608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"347'149","reported_seq":698,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.254871+0000","last_change":"2024-08-17T18:18:32.735002+0000","last_active":"2024-08-17T18:19:07.254871+0000","last_peered":"2024-08-17T18:19:07.254871+0000","last_clean":"2024-08-17T18:19:07.254871+0000","last_became_active":"2024-08-17T18:18:32.734291+0000","last_became_peered":"2024-08-17T18:18:32.734291+0000","last_unstale":"2024-08-17T18:19:07.254871+0000","last_undegraded":"2024-08-17T18:19:07.254871+0000","last_fullsized":"2024-08-17T18:19:07.254871+0000","mapping_epoch":1669,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1670,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:12:53.586672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:12:53.586672+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":214680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":350,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.573737+0000","last_change":"2024-08-17T18:15:53.370263+0000","last_active":"2024-08-17T18:19:15.573737+0000","last_peered":"2024-08-17T18:19:15.573737+0000","last_clean":"2024-08-17T18:19:15.573737+0000","last_became_active":"2024-08-17T18:15:53.369971+0000","last_became_peered":"2024-08-17T18:15:53.369971+0000","last_unstale":"2024-08-17T18:19:15.573737+0000","last_undegraded":"2024-08-17T18:19:15.573737+0000","last_fullsized":"2024-08-17T18:19:15.573737+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:51:08.633886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":856,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.583576+0000","last_change":"2024-08-17T18:18:29.727602+0000","last_active":"2024-08-17T18:19:15.583576+0000","last_peered":"2024-08-17T18:19:15.583576+0000","last_clean":"2024-08-17T18:19:15.583576+0000","last_became_active":"2024-08-17T18:18:29.727190+0000","last_became_peered":"2024-08-17T18:18:29.727190+0000","last_unstale":"2024-08-17T18:19:15.583576+0000","last_undegraded":"2024-08-17T18:19:15.583576+0000","last_fullsized":"2024-08-17T18:19:15.583576+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:13:09.502199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":36,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.602929+0000","last_change":"2024-08-17T18:18:29.888513+0000","last_active":"2024-08-17T18:19:15.602929+0000","last_peered":"2024-08-17T18:19:15.602929+0000","last_clean":"2024-08-17T18:19:15.602929+0000","last_became_active":"2024-08-17T18:18:29.888065+0000","last_became_peered":"2024-08-17T18:18:29.888065+0000","last_unstale":"2024-08-17T18:19:15.602929+0000","last_undegraded":"2024-08-17T18:19:15.602929+0000","last_fullsized":"2024-08-17T18:19:15.602929+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:18.952476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1526,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.256051+0000","last_change":"2024-08-17T18:18:29.895726+0000","last_active":"2024-08-17T18:19:07.256051+0000","last_peered":"2024-08-17T18:19:07.256051+0000","last_clean":"2024-08-17T18:19:07.256051+0000","last_became_active":"2024-08-17T18:18:29.895383+0000","last_became_peered":"2024-08-17T18:18:29.895383+0000","last_unstale":"2024-08-17T18:19:07.256051+0000","last_undegraded":"2024-08-17T18:19:07.256051+0000","last_fullsized":"2024-08-17T18:19:07.256051+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:18:30.787712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":36,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.603029+0000","last_change":"2024-08-17T18:18:29.800143+0000","last_active":"2024-08-17T18:19:15.603029+0000","last_peered":"2024-08-17T18:19:15.603029+0000","last_clean":"2024-08-17T18:19:15.603029+0000","last_became_active":"2024-08-17T18:18:29.799896+0000","last_became_peered":"2024-08-17T18:18:29.799896+0000","last_unstale":"2024-08-17T18:19:15.603029+0000","last_undegraded":"2024-08-17T18:19:15.603029+0000","last_fullsized":"2024-08-17T18:19:15.603029+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:27:18.144391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":261,"reported_epoch":1679,"state":"active+recovery_wait","last_fresh":"2024-08-17T18:19:15.587816+0000","last_change":"2024-08-17T18:15:54.221085+0000","last_active":"2024-08-17T18:19:15.587816+0000","last_peered":"2024-08-17T18:19:15.587816+0000","last_clean":"2024-08-17T18:15:50.189470+0000","last_became_active":"2024-08-17T18:15:53.304106+0000","last_became_peered":"2024-08-17T18:15:53.304106+0000","last_unstale":"2024-08-17T18:19:15.587816+0000","last_undegraded":"2024-08-17T18:19:15.587816+0000","last_fullsized":"2024-08-17T18:19:15.587816+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:21.409368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":["5","2","6","7"],"object_location_counts":[{"shards":"2,5,7","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":859,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.579863+0000","last_change":"2024-08-17T18:17:16.835493+0000","last_active":"2024-08-17T18:19:15.579863+0000","last_peered":"2024-08-17T18:19:15.579863+0000","last_clean":"2024-08-17T18:19:15.579863+0000","last_became_active":"2024-08-17T18:15:55.698769+0000","last_became_peered":"2024-08-17T18:15:55.698769+0000","last_unstale":"2024-08-17T18:19:15.579863+0000","last_undegraded":"2024-08-17T18:19:15.579863+0000","last_fullsized":"2024-08-17T18:19:15.579863+0000","mapping_epoch":1640,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1641,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:54.303708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:54.303708+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":74,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"347'149","reported_seq":956,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.259646+0000","last_change":"2024-08-17T18:09:15.592722+0000","last_active":"2024-08-17T18:19:07.259646+0000","last_peered":"2024-08-17T18:19:07.259646+0000","last_clean":"2024-08-17T18:19:07.259646+0000","last_became_active":"2024-08-17T18:09:15.592242+0000","last_became_peered":"2024-08-17T18:09:15.592242+0000","last_unstale":"2024-08-17T18:19:07.259646+0000","last_undegraded":"2024-08-17T18:19:07.259646+0000","last_fullsized":"2024-08-17T18:19:07.259646+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:55.255983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:55.255983+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:02:53.596275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":36,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.602027+0000","last_change":"2024-08-17T18:18:29.895120+0000","last_active":"2024-08-17T18:19:15.602027+0000","last_peered":"2024-08-17T18:19:15.602027+0000","last_clean":"2024-08-17T18:19:15.602027+0000","last_became_active":"2024-08-17T18:18:29.894811+0000","last_became_peered":"2024-08-17T18:18:29.894811+0000","last_unstale":"2024-08-17T18:19:15.602027+0000","last_undegraded":"2024-08-17T18:19:15.602027+0000","last_fullsized":"2024-08-17T18:19:15.602027+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:44:28.402299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":36,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.601681+0000","last_change":"2024-08-17T18:18:29.900296+0000","last_active":"2024-08-17T18:19:15.601681+0000","last_peered":"2024-08-17T18:19:15.601681+0000","last_clean":"2024-08-17T18:19:15.601681+0000","last_became_active":"2024-08-17T18:18:29.899848+0000","last_became_peered":"2024-08-17T18:18:29.899848+0000","last_unstale":"2024-08-17T18:19:15.601681+0000","last_undegraded":"2024-08-17T18:19:15.601681+0000","last_fullsized":"2024-08-17T18:19:15.601681+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:09:15.545423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1715,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.257249+0000","last_change":"2024-08-17T18:18:29.882860+0000","last_active":"2024-08-17T18:19:07.257249+0000","last_peered":"2024-08-17T18:19:07.257249+0000","last_clean":"2024-08-17T18:19:07.257249+0000","last_became_active":"2024-08-17T18:18:29.882539+0000","last_became_peered":"2024-08-17T18:18:29.882539+0000","last_unstale":"2024-08-17T18:19:07.257249+0000","last_undegraded":"2024-08-17T18:19:07.257249+0000","last_fullsized":"2024-08-17T18:19:07.257249+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:12:52.056802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1705,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.255904+0000","last_change":"2024-08-17T18:15:53.244851+0000","last_active":"2024-08-17T18:19:07.255904+0000","last_peered":"2024-08-17T18:19:07.255904+0000","last_clean":"2024-08-17T18:19:07.255904+0000","last_became_active":"2024-08-17T18:15:53.240074+0000","last_became_peered":"2024-08-17T18:15:53.240074+0000","last_unstale":"2024-08-17T18:19:07.255904+0000","last_undegraded":"2024-08-17T18:19:07.255904+0000","last_fullsized":"2024-08-17T18:19:07.255904+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:09:00.168413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"347'149","reported_seq":858,"reported_epoch":1679,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:19:15.573407+0000","last_change":"2024-08-17T18:18:30.729274+0000","last_active":"2024-08-17T18:19:15.573407+0000","last_peered":"2024-08-17T18:19:15.573407+0000","last_clean":"2024-08-17T18:09:10.546471+0000","last_became_active":"2024-08-17T18:18:30.728913+0000","last_became_peered":"2024-08-17T18:18:30.728913+0000","last_unstale":"2024-08-17T18:19:15.573407+0000","last_undegraded":"2024-08-17T18:18:30.719341+0000","last_fullsized":"2024-08-17T18:18:30.718846+0000","mapping_epoch":1667,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1612,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:59.307356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:59.307356+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,4],"avail_no_missing":["2","5","6"],"object_location_counts":[{"shards":"2,5,6","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1700,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.255860+0000","last_change":"2024-08-17T18:12:04.648228+0000","last_active":"2024-08-17T18:19:07.255860+0000","last_peered":"2024-08-17T18:19:07.255860+0000","last_clean":"2024-08-17T18:19:07.255860+0000","last_became_active":"2024-08-17T18:12:04.647906+0000","last_became_peered":"2024-08-17T18:12:04.647906+0000","last_unstale":"2024-08-17T18:19:07.255860+0000","last_undegraded":"2024-08-17T18:19:07.255860+0000","last_fullsized":"2024-08-17T18:19:07.255860+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:51:30.081540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1687,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.254687+0000","last_change":"2024-08-17T18:18:29.728681+0000","last_active":"2024-08-17T18:19:07.254687+0000","last_peered":"2024-08-17T18:19:07.254687+0000","last_clean":"2024-08-17T18:19:07.254687+0000","last_became_active":"2024-08-17T18:18:29.728215+0000","last_became_peered":"2024-08-17T18:18:29.728215+0000","last_unstale":"2024-08-17T18:19:07.254687+0000","last_undegraded":"2024-08-17T18:19:07.254687+0000","last_fullsized":"2024-08-17T18:19:07.254687+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:48:41.152839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"347'149","reported_seq":932,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.245872+0000","last_change":"2024-08-17T18:18:33.757347+0000","last_active":"2024-08-17T18:19:07.245872+0000","last_peered":"2024-08-17T18:19:07.245872+0000","last_clean":"2024-08-17T18:19:07.245872+0000","last_became_active":"2024-08-17T18:18:33.756960+0000","last_became_peered":"2024-08-17T18:18:33.756960+0000","last_unstale":"2024-08-17T18:19:07.245872+0000","last_undegraded":"2024-08-17T18:19:07.245872+0000","last_fullsized":"2024-08-17T18:19:07.245872+0000","mapping_epoch":1670,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1671,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:52.271487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:52.271487+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:37:36.389244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1688,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.256701+0000","last_change":"2024-08-17T18:18:29.883479+0000","last_active":"2024-08-17T18:19:07.256701+0000","last_peered":"2024-08-17T18:19:07.256701+0000","last_clean":"2024-08-17T18:19:07.256701+0000","last_became_active":"2024-08-17T18:18:29.883185+0000","last_became_peered":"2024-08-17T18:18:29.883185+0000","last_unstale":"2024-08-17T18:19:07.256701+0000","last_undegraded":"2024-08-17T18:19:07.256701+0000","last_fullsized":"2024-08-17T18:19:07.256701+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:11:04.148316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":40,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.602390+0000","last_change":"2024-08-17T18:18:29.891624+0000","last_active":"2024-08-17T18:19:15.602390+0000","last_peered":"2024-08-17T18:19:15.602390+0000","last_clean":"2024-08-17T18:19:15.602390+0000","last_became_active":"2024-08-17T18:18:29.880707+0000","last_became_peered":"2024-08-17T18:18:29.880707+0000","last_unstale":"2024-08-17T18:19:15.602390+0000","last_undegraded":"2024-08-17T18:19:15.602390+0000","last_fullsized":"2024-08-17T18:19:15.602390+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:04.490121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":733,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.255023+0000","last_change":"2024-08-17T18:18:29.727626+0000","last_active":"2024-08-17T18:19:07.255023+0000","last_peered":"2024-08-17T18:19:07.255023+0000","last_clean":"2024-08-17T18:19:07.255023+0000","last_became_active":"2024-08-17T18:18:29.727331+0000","last_became_peered":"2024-08-17T18:18:29.727331+0000","last_unstale":"2024-08-17T18:19:07.255023+0000","last_undegraded":"2024-08-17T18:19:07.255023+0000","last_fullsized":"2024-08-17T18:19:07.255023+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:46:16.169400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":258,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.261653+0000","last_change":"2024-08-17T18:18:29.735189+0000","last_active":"2024-08-17T18:19:07.261653+0000","last_peered":"2024-08-17T18:19:07.261653+0000","last_clean":"2024-08-17T18:19:07.261653+0000","last_became_active":"2024-08-17T18:18:29.734681+0000","last_became_peered":"2024-08-17T18:18:29.734681+0000","last_unstale":"2024-08-17T18:19:07.261653+0000","last_undegraded":"2024-08-17T18:19:07.261653+0000","last_fullsized":"2024-08-17T18:19:07.261653+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:03:20.353696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1507,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.260290+0000","last_change":"2024-08-17T18:18:29.943292+0000","last_active":"2024-08-17T18:19:07.260290+0000","last_peered":"2024-08-17T18:19:07.260290+0000","last_clean":"2024-08-17T18:19:07.260290+0000","last_became_active":"2024-08-17T18:18:29.942947+0000","last_became_peered":"2024-08-17T18:18:29.942947+0000","last_unstale":"2024-08-17T18:19:07.260290+0000","last_undegraded":"2024-08-17T18:19:07.260290+0000","last_fullsized":"2024-08-17T18:19:07.260290+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:19:04.210562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":261,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.264859+0000","last_change":"2024-08-17T18:15:53.661488+0000","last_active":"2024-08-17T18:19:07.264859+0000","last_peered":"2024-08-17T18:19:07.264859+0000","last_clean":"2024-08-17T18:19:07.264859+0000","last_became_active":"2024-08-17T18:15:53.308611+0000","last_became_peered":"2024-08-17T18:15:53.308611+0000","last_unstale":"2024-08-17T18:19:07.264859+0000","last_undegraded":"2024-08-17T18:19:07.264859+0000","last_fullsized":"2024-08-17T18:19:07.264859+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:46:37.056632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1722,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.255661+0000","last_change":"2024-08-17T18:18:29.728301+0000","last_active":"2024-08-17T18:19:07.255661+0000","last_peered":"2024-08-17T18:19:07.255661+0000","last_clean":"2024-08-17T18:19:07.255661+0000","last_became_active":"2024-08-17T18:18:29.728019+0000","last_became_peered":"2024-08-17T18:18:29.728019+0000","last_unstale":"2024-08-17T18:19:07.255661+0000","last_undegraded":"2024-08-17T18:19:07.255661+0000","last_fullsized":"2024-08-17T18:19:07.255661+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:57:56.453165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":56,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.246426+0000","last_change":"2024-08-17T18:15:53.708176+0000","last_active":"2024-08-17T18:19:07.246426+0000","last_peered":"2024-08-17T18:19:07.246426+0000","last_clean":"2024-08-17T18:19:07.246426+0000","last_became_active":"2024-08-17T18:15:53.669563+0000","last_became_peered":"2024-08-17T18:15:53.669563+0000","last_unstale":"2024-08-17T18:19:07.246426+0000","last_undegraded":"2024-08-17T18:19:07.246426+0000","last_fullsized":"2024-08-17T18:19:07.246426+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:52:17.018687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":267,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.269509+0000","last_change":"2024-08-17T18:17:22.687310+0000","last_active":"2024-08-17T18:19:07.269509+0000","last_peered":"2024-08-17T18:19:07.269509+0000","last_clean":"2024-08-17T18:19:07.269509+0000","last_became_active":"2024-08-17T18:17:22.686966+0000","last_became_peered":"2024-08-17T18:17:22.686966+0000","last_unstale":"2024-08-17T18:19:07.269509+0000","last_undegraded":"2024-08-17T18:19:07.269509+0000","last_fullsized":"2024-08-17T18:19:07.269509+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:46:11.568465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":56,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.247973+0000","last_change":"2024-08-17T18:15:53.461774+0000","last_active":"2024-08-17T18:19:07.247973+0000","last_peered":"2024-08-17T18:19:07.247973+0000","last_clean":"2024-08-17T18:19:07.247973+0000","last_became_active":"2024-08-17T18:15:53.287744+0000","last_became_peered":"2024-08-17T18:15:53.287744+0000","last_unstale":"2024-08-17T18:19:07.247973+0000","last_undegraded":"2024-08-17T18:19:07.247973+0000","last_fullsized":"2024-08-17T18:19:07.247973+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:25.168664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1799,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.262409+0000","last_change":"2024-08-17T18:18:29.897922+0000","last_active":"2024-08-17T18:19:07.262409+0000","last_peered":"2024-08-17T18:19:07.262409+0000","last_clean":"2024-08-17T18:19:07.262409+0000","last_became_active":"2024-08-17T18:18:29.883879+0000","last_became_peered":"2024-08-17T18:18:29.883879+0000","last_unstale":"2024-08-17T18:19:07.262409+0000","last_undegraded":"2024-08-17T18:19:07.262409+0000","last_fullsized":"2024-08-17T18:19:07.262409+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:54:31.992298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":36,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.602141+0000","last_change":"2024-08-17T18:18:29.890043+0000","last_active":"2024-08-17T18:19:15.602141+0000","last_peered":"2024-08-17T18:19:15.602141+0000","last_clean":"2024-08-17T18:19:15.602141+0000","last_became_active":"2024-08-17T18:18:29.889687+0000","last_became_peered":"2024-08-17T18:18:29.889687+0000","last_unstale":"2024-08-17T18:19:15.602141+0000","last_undegraded":"2024-08-17T18:19:15.602141+0000","last_fullsized":"2024-08-17T18:19:15.602141+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:55:34.595507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":418,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.255776+0000","last_change":"2024-08-17T18:15:53.245247+0000","last_active":"2024-08-17T18:19:07.255776+0000","last_peered":"2024-08-17T18:19:07.255776+0000","last_clean":"2024-08-17T18:19:07.255776+0000","last_became_active":"2024-08-17T18:15:53.244627+0000","last_became_peered":"2024-08-17T18:15:53.244627+0000","last_unstale":"2024-08-17T18:19:07.255776+0000","last_undegraded":"2024-08-17T18:19:07.255776+0000","last_fullsized":"2024-08-17T18:19:07.255776+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:42:03.556013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":36,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.602098+0000","last_change":"2024-08-17T18:18:29.895804+0000","last_active":"2024-08-17T18:19:15.602098+0000","last_peered":"2024-08-17T18:19:15.602098+0000","last_clean":"2024-08-17T18:19:15.602098+0000","last_became_active":"2024-08-17T18:18:29.895352+0000","last_became_peered":"2024-08-17T18:18:29.895352+0000","last_unstale":"2024-08-17T18:19:15.602098+0000","last_undegraded":"2024-08-17T18:19:15.602098+0000","last_fullsized":"2024-08-17T18:19:15.602098+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:32:29.222569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":36,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.603141+0000","last_change":"2024-08-17T18:18:29.888110+0000","last_active":"2024-08-17T18:19:15.603141+0000","last_peered":"2024-08-17T18:19:15.603141+0000","last_clean":"2024-08-17T18:19:15.603141+0000","last_became_active":"2024-08-17T18:18:29.887787+0000","last_became_peered":"2024-08-17T18:18:29.887787+0000","last_unstale":"2024-08-17T18:19:15.603141+0000","last_undegraded":"2024-08-17T18:19:15.603141+0000","last_fullsized":"2024-08-17T18:19:15.603141+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:46:10.231616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":267,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.264786+0000","last_change":"2024-08-17T18:18:29.900128+0000","last_active":"2024-08-17T18:19:07.264786+0000","last_peered":"2024-08-17T18:19:07.264786+0000","last_clean":"2024-08-17T18:19:07.264786+0000","last_became_active":"2024-08-17T18:18:29.899279+0000","last_became_peered":"2024-08-17T18:18:29.899279+0000","last_unstale":"2024-08-17T18:19:07.264786+0000","last_undegraded":"2024-08-17T18:19:07.264786+0000","last_fullsized":"2024-08-17T18:19:07.264786+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:01:29.045498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":59,"reported_epoch":1678,"state":"active+clean","last_fresh":"2024-08-17T18:19:07.255915+0000","last_change":"2024-08-17T18:15:53.271113+0000","last_active":"2024-08-17T18:19:07.255915+0000","last_peered":"2024-08-17T18:19:07.255915+0000","last_clean":"2024-08-17T18:19:07.255915+0000","last_became_active":"2024-08-17T18:15:53.230983+0000","last_became_peered":"2024-08-17T18:15:53.230983+0000","last_unstale":"2024-08-17T18:19:07.255915+0000","last_undegraded":"2024-08-17T18:19:07.255915+0000","last_fullsized":"2024-08-17T18:19:07.255915+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:00:35.315798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":36,"reported_epoch":1679,"state":"active+clean","last_fresh":"2024-08-17T18:19:15.602134+0000","last_change":"2024-08-17T18:18:29.888785+0000","last_active":"2024-08-17T18:19:15.602134+0000","last_peered":"2024-08-17T18:19:15.602134+0000","last_clean":"2024-08-17T18:19:15.602134+0000","last_became_active":"2024-08-17T18:18:29.888475+0000","last_became_peered":"2024-08-17T18:18:29.888475+0000","last_unstale":"2024-08-17T18:19:15.602134+0000","last_undegraded":"2024-08-17T18:19:15.602134+0000","last_fullsized":"2024-08-17T18:19:15.602134+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:44:28.968751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":57,"num_pg":26,"stat_sum":{"num_bytes":2286682136,"num_objects":34893,"num_object_clones":0,"num_object_copies":104679,"num_objects_missing_on_primary":308,"num_objects_missing":308,"num_objects_degraded":20253,"num_objects_misplaced":27733,"num_objects_unfound":0,"num_objects_dirty":34893,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196947,"num_write_kb":7418817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40134,"num_bytes_recovered":2310078464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10087706624,"data_stored":10087694408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134619,"ondisk_log_size":134619,"up":78,"acting":74,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":697707,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":149,"num_write":39,"num_write_kb":439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":810709,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":31,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":95,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":95,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":12532,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":22134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234840,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":973348,"kb_used_data":360668,"kb_used_omap":1,"kb_used_meta":612670,"kb_avail":92771804,"statfs":{"total":95995035648,"available":94998327296,"internally_reserved":0,"allocated":369324032,"data_stored":364908186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":627374539},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496478,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3196772,"kb_used_data":1950428,"kb_used_omap":1,"kb_used_meta":1246334,"kb_avail":90548380,"statfs":{"total":95995035648,"available":92721541120,"internally_reserved":0,"allocated":1997238272,"data_stored":1992823430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1276246471},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565719,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1063176,"kb_used_data":30080,"kb_used_omap":72,"kb_used_meta":1033015,"kb_avail":92681976,"statfs":{"total":95995035648,"available":94906343424,"internally_reserved":0,"allocated":30801920,"data_stored":26409857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":1057807854},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":1399,"seq":6008659247321,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2736108,"kb_used_data":2005668,"kb_used_omap":131,"kb_used_meta":730300,"kb_avail":91009044,"statfs":{"total":95995035648,"available":93193261056,"internally_reserved":0,"allocated":2053804032,"data_stored":2049381086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134291,"internal_metadata":747828077},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":2864743187110,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448749,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3423624,"kb_used_data":2633728,"kb_used_omap":75,"kb_used_meta":789812,"kb_avail":90321528,"statfs":{"total":95995035648,"available":92489244672,"internally_reserved":0,"allocated":2696937472,"data_stored":2692545343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":808767679},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208758113,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2833688,"kb_used_data":1521360,"kb_used_omap":1,"kb_used_meta":1312318,"kb_avail":90911464,"statfs":{"total":95995035648,"available":93093339136,"internally_reserved":0,"allocated":1557872640,"data_stored":1553465328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1343814086},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019750,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2567520,"kb_used_data":1453592,"kb_used_omap":1,"kb_used_meta":1113918,"kb_avail":91177632,"statfs":{"total":95995035648,"available":93365895168,"internally_reserved":0,"allocated":1488478208,"data_stored":1484066951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1140652486},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2676625408,"data_stored":2676621336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2034896896,"data_stored":2034892824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20971520,"data_stored":20971520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1466105856,"data_stored":1466105856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1546125312,"data_stored":1546125312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1986859008,"data_stored":1986854936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":356122624,"data_stored":356122624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T18:19:28.704 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool set unique_pool_4 pgp_num 26 2024-08-17T18:19:28.759 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:28.787 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:28.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:28.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:28.889 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:28.925 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:28 smithi136 ceph-mon[109861]: from='client.46679 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:19:28.925 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:28 smithi136 ceph-mon[109861]: osdmap e1681: 8 total, 8 up, 7 in 2024-08-17T18:19:28.925 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:28 smithi136 ceph-mon[115078]: from='client.46679 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:19:28.925 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:28 smithi136 ceph-mon[115078]: osdmap e1681: 8 total, 8 up, 7 in 2024-08-17T18:19:28.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:28.928 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:28.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:28.946 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:19:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:28 smithi165 ceph-mon[105517]: from='client.46679 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:19:29.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:28 smithi165 ceph-mon[105517]: osdmap e1681: 8 total, 8 up, 7 in 2024-08-17T18:19:29.029 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:29.059 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:29.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:29.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:29.161 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:29.195 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:29.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:29.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:29.298 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:29.335 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:29.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:29.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:29.437 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:29.477 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:29.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:29.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:29.578 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:29.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:29.606 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:29.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:29.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:29 smithi136 ceph-mon[109861]: pgmap v3241: 109 pgs: 1 active+recovering+degraded+remapped, 1 peering, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 84 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 549 op/s; 17732/99909 objects degraded (17.748%); 27644/99909 objects misplaced (27.669%); 12 MiB/s, 331 objects/s recovering 2024-08-17T18:19:29.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:29 smithi136 ceph-mon[115078]: pgmap v3241: 109 pgs: 1 active+recovering+degraded+remapped, 1 peering, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 84 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 549 op/s; 17732/99909 objects degraded (17.748%); 27644/99909 objects misplaced (27.669%); 12 MiB/s, 331 objects/s recovering 2024-08-17T18:19:29.707 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:29.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:29.747 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:29.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:29.848 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:29.872 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:29.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:29.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:29.974 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:30.002 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:30.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:30.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:30.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:29 smithi165 ceph-mon[105517]: pgmap v3241: 109 pgs: 1 active+recovering+degraded+remapped, 1 peering, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 84 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 549 op/s; 17732/99909 objects degraded (17.748%); 27644/99909 objects misplaced (27.669%); 12 MiB/s, 331 objects/s recovering 2024-08-17T18:19:30.105 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:30.131 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:30.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:30.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:30.233 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:30.256 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:30.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:30.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:30.358 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:30.382 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:30.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:30.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:30.484 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:30.522 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:30.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:30.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:30.624 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:30.649 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:30.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:30.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:30.751 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:30.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:30.792 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:30.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:30.893 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:30.918 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:30.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:30.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:30.935 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:30 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 17732/99909 objects degraded (17.748%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:30.937 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:30 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 17732/99909 objects degraded (17.748%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:30.943 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:19:31.000 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:19:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:30 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 17732/99909 objects degraded (17.748%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:31.020 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:31.049 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:31.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:31.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:31.150 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:31.175 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:31.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:31.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:31.277 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:31.302 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:31.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:31.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:31.404 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:31.430 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:31.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:31.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:31.532 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:31.562 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:31.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:31.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:31.664 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:31.690 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:31.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:31.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:31.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:31.829 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:31.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:31.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:31.931 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:31.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:31 smithi136 ceph-mon[109861]: pgmap v3242: 109 pgs: 1 active+recovering+degraded+remapped, 1 peering, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 84 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 354 op/s; 17569/98361 objects degraded (17.862%); 27208/98361 objects misplaced (27.661%); 6.1 MiB/s, 178 objects/s recovering 2024-08-17T18:19:31.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:31 smithi136 ceph-mon[115078]: pgmap v3242: 109 pgs: 1 active+recovering+degraded+remapped, 1 peering, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 84 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 354 op/s; 17569/98361 objects degraded (17.862%); 27208/98361 objects misplaced (27.661%); 6.1 MiB/s, 178 objects/s recovering 2024-08-17T18:19:31.956 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:31.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:31.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:31 smithi165 ceph-mon[105517]: pgmap v3242: 109 pgs: 1 active+recovering+degraded+remapped, 1 peering, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 84 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 2.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 354 op/s; 17569/98361 objects degraded (17.862%); 27208/98361 objects misplaced (27.661%); 6.1 MiB/s, 178 objects/s recovering 2024-08-17T18:19:32.057 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:32.106 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:32.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:32.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:32.208 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:32.237 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:32.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:32.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:32.340 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:32.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:32.381 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:32.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:32.483 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:32.523 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:32.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:32.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:32.623 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:19:32.626 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:32.657 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:32.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:32.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:32.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:32.792 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:32.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:32.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:32.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:32.923 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:32.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:32.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:32.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:32 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1894500842' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:19:32.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:32 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:19:32.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:32 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1894500842' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:19:32.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:32 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:19:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:32 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1894500842' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:19:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:32 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:19:33.025 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:33.060 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:33.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:33.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:33.162 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:33.185 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:33.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:33.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:33.287 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:33.310 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:33.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:33.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:33.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:33.444 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:33.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:33.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:33.546 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:33.578 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:33.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:33.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:33.680 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:33.689 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:33 smithi136 ceph-mon[109861]: pgmap v3243: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.9 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 633 op/s; 16774/91665 objects degraded (18.299%); 25663/91665 objects misplaced (27.997%); 12 MiB/s, 321 objects/s recovering 2024-08-17T18:19:33.689 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:33 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:19:33.689 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:33 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:19:33.689 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:33 smithi136 ceph-mon[109861]: osdmap e1682: 8 total, 8 up, 7 in 2024-08-17T18:19:33.689 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:33 smithi136 ceph-mon[115078]: pgmap v3243: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.9 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 633 op/s; 16774/91665 objects degraded (18.299%); 25663/91665 objects misplaced (27.997%); 12 MiB/s, 321 objects/s recovering 2024-08-17T18:19:33.690 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:33 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:19:33.690 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:33 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:19:33.690 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:33 smithi136 ceph-mon[115078]: osdmap e1682: 8 total, 8 up, 7 in 2024-08-17T18:19:33.714 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:33.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:33.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:33.816 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:33.850 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:33.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:33.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:33.952 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:33.984 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:33.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:33.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:33 smithi165 ceph-mon[105517]: pgmap v3243: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.9 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 633 op/s; 16774/91665 objects degraded (18.299%); 25663/91665 objects misplaced (27.997%); 12 MiB/s, 321 objects/s recovering 2024-08-17T18:19:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:33 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:19:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:33 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:19:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:33 smithi165 ceph-mon[105517]: osdmap e1682: 8 total, 8 up, 7 in 2024-08-17T18:19:34.086 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:34.124 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:34.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:34.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:34.226 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:34.263 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:34.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:34.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:34.365 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:34.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:34.409 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:34.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:34.510 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:34.536 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:34.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:34.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:34.625 INFO:teuthology.orchestra.run.smithi136.stderr:set pool 57 pgp_num to 26 2024-08-17T18:19:34.639 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:34.667 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:34.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:34.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:34.769 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:34.801 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:34.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:34.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:34.903 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:34.940 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:34.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:34.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:34.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:34 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2727016762' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"} : dispatch 2024-08-17T18:19:34.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:34 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"} : dispatch 2024-08-17T18:19:34.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:34 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2727016762' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"} : dispatch 2024-08-17T18:19:34.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:34 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"} : dispatch 2024-08-17T18:19:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:34 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2727016762' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"} : dispatch 2024-08-17T18:19:35.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:34 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"} : dispatch 2024-08-17T18:19:35.042 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:35.067 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:35.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:35.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:35.169 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:35.205 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:35.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:35.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:35.307 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:35.334 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:35.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:35.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:35.435 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:35.457 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:19:35.474 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:35.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:35.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:35.576 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:35.611 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:35.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:35.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:35.713 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:35.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:35.749 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:35.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:35.850 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:35.877 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:35.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:35.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:35.927 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:35 smithi136 ceph-mon[109861]: pgmap v3245: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.9 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 551 op/s; 16774/91665 objects degraded (18.299%); 25663/91665 objects misplaced (27.997%); 12 MiB/s, 314 objects/s recovering 2024-08-17T18:19:35.927 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:35 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]': finished 2024-08-17T18:19:35.927 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:35 smithi136 ceph-mon[109861]: osdmap e1683: 8 total, 8 up, 7 in 2024-08-17T18:19:35.927 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:35 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 16774/91665 objects degraded (18.299%), 13 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:35.927 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:35 smithi136 ceph-mon[109861]: osdmap e1684: 8 total, 8 up, 7 in 2024-08-17T18:19:35.927 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:35 smithi136 ceph-mon[115078]: pgmap v3245: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.9 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 551 op/s; 16774/91665 objects degraded (18.299%); 25663/91665 objects misplaced (27.997%); 12 MiB/s, 314 objects/s recovering 2024-08-17T18:19:35.927 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:35 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]': finished 2024-08-17T18:19:35.927 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:35 smithi136 ceph-mon[115078]: osdmap e1683: 8 total, 8 up, 7 in 2024-08-17T18:19:35.928 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:35 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 16774/91665 objects degraded (18.299%), 13 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:35.928 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:35 smithi136 ceph-mon[115078]: osdmap e1684: 8 total, 8 up, 7 in 2024-08-17T18:19:35.979 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:36.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:36.008 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:36.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:36.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:35 smithi165 ceph-mon[105517]: pgmap v3245: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.9 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 551 op/s; 16774/91665 objects degraded (18.299%); 25663/91665 objects misplaced (27.997%); 12 MiB/s, 314 objects/s recovering 2024-08-17T18:19:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:35 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]': finished 2024-08-17T18:19:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:35 smithi165 ceph-mon[105517]: osdmap e1683: 8 total, 8 up, 7 in 2024-08-17T18:19:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:35 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 16774/91665 objects degraded (18.299%), 13 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:35 smithi165 ceph-mon[105517]: osdmap e1684: 8 total, 8 up, 7 in 2024-08-17T18:19:36.109 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:36.137 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:36.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:36.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:36.239 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:36.267 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:36.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:36.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:36.369 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:36.406 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:36.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:36.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:36.468 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:19:36.468 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:19:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:203984 57.2 57:40281079:::benchmark_data_smithi136_389235_object88062:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1684)", 2024-08-17T18:19:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:19:36.448050+0000", 2024-08-17T18:19:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.018731581000000001, 2024-08-17T18:19:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018758997999999999, 2024-08-17T18:19:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:19:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:19:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for degraded object", 2024-08-17T18:19:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:19:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:19:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:19:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 203984 2024-08-17T18:19:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:19:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:19:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.448050+0000", 2024-08-17T18:19:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:19:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.448050+0000", 2024-08-17T18:19:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:19:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.448051+0000", 2024-08-17T18:19:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.308e-06 2024-08-17T18:19:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:19:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.448056+0000", 2024-08-17T18:19:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0830000000000003e-06 2024-08-17T18:19:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:19:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.448070+0000", 2024-08-17T18:19:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3169999999999999e-05 2024-08-17T18:19:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:19:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.448077+0000", 2024-08-17T18:19:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2130000000000002e-06 2024-08-17T18:19:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.448139+0000", 2024-08-17T18:19:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1781999999999999e-05 2024-08-17T18:19:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:19:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.448805+0000", 2024-08-17T18:19:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00066635000000000002 2024-08-17T18:19:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:203966 57.2 57:41b60c37:::benchmark_data_smithi136_389235_object88044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1684)", 2024-08-17T18:19:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:19:36.416501+0000", 2024-08-17T18:19:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.050281102000000001, 2024-08-17T18:19:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050370402000000002, 2024-08-17T18:19:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:19:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:19:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-17T18:19:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:19:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:19:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:19:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 203966 2024-08-17T18:19:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:19:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:19:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.416501+0000", 2024-08-17T18:19:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:19:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.416501+0000", 2024-08-17T18:19:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:19:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.416501+0000", 2024-08-17T18:19:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1699999999999997e-07 2024-08-17T18:19:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:19:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.416505+0000", 2024-08-17T18:19:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.636e-06 2024-08-17T18:19:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:19:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.416516+0000", 2024-08-17T18:19:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0679999999999999e-05 2024-08-17T18:19:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:19:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.416522+0000", 2024-08-17T18:19:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3879999999999997e-06 2024-08-17T18:19:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.418803+0000", 2024-08-17T18:19:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0022807840000000001 2024-08-17T18:19:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:19:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.419495+0000", 2024-08-17T18:19:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00069178399999999998 2024-08-17T18:19:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.465789+0000", 2024-08-17T18:19:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.046294180999999997 2024-08-17T18:19:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:19:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.465889+0000", 2024-08-17T18:19:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9784999999999994e-05 2024-08-17T18:19:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5", 2024-08-17T18:19:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.466324+0000", 2024-08-17T18:19:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00043491899999999997 2024-08-17T18:19:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.46277.0:203995 57.d e1684/1666 57:be4a89b6:::benchmark_data_smithi136_389235_object88073:head v 1684'12717, mlcod=1684'12715)", 2024-08-17T18:19:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:19:36.465205+0000", 2024-08-17T18:19:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.001576558, 2024-08-17T18:19:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001731773, 2024-08-17T18:19:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:19:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:19:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T18:19:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:19:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:19:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.465205+0000", 2024-08-17T18:19:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:19:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.465205+0000", 2024-08-17T18:19:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:19:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.465206+0000", 2024-08-17T18:19:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2399999999999996e-07 2024-08-17T18:19:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:19:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.465213+0000", 2024-08-17T18:19:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1439999999999997e-06 2024-08-17T18:19:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:19:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.465228+0000", 2024-08-17T18:19:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5081e-05 2024-08-17T18:19:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:19:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.465235+0000", 2024-08-17T18:19:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8689999999999998e-06 2024-08-17T18:19:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.465289+0000", 2024-08-17T18:19:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3854000000000003e-05 2024-08-17T18:19:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:19:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.465313+0000", 2024-08-17T18:19:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3767000000000001e-05 2024-08-17T18:19:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.483 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:203996 57.b 57:d2abd20e:::benchmark_data_smithi136_389235_object88074:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1684)", 2024-08-17T18:19:36.483 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:19:36.466268+0000", 2024-08-17T18:19:36.483 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.00051394899999999998, 2024-08-17T18:19:36.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000715039, 2024-08-17T18:19:36.483 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:19:36.483 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:19:36.483 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:19:36.484 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:19:36.484 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:19:36.484 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:19:36.484 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 203996 2024-08-17T18:19:36.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.484 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:19:36.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:19:36.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.466268+0000", 2024-08-17T18:19:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:19:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.466268+0000", 2024-08-17T18:19:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:19:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.466269+0000", 2024-08-17T18:19:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3370000000000001e-06 2024-08-17T18:19:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:19:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.466274+0000", 2024-08-17T18:19:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4149999999999998e-06 2024-08-17T18:19:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:19:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.466284+0000", 2024-08-17T18:19:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8789999999999998e-06 2024-08-17T18:19:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:19:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.466292+0000", 2024-08-17T18:19:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2069999999999998e-06 2024-08-17T18:19:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:36.466421+0000", 2024-08-17T18:19:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012943199999999999 2024-08-17T18:19:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-17T18:19:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 4 2024-08-17T18:19:36.488 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:19:36.508 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:36.549 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:36.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:36.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:36.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:36.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:36.686 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:36.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:36.787 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:36.816 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:36.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:36.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:36.918 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:36.940 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:36.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:36.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:37.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:37.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:37.085 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:37.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:37.186 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:37.212 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:37.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:37.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:37.240 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:19:37.240 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:19:37.269 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:19:37.269 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:19:37.292 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:19:37.292 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:19:37.311 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:19:37.311 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:19:37.314 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:37.343 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:19:37.343 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:19:37.357 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:37.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:37.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:37.368 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:19:37.368 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:19:37.368 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:37 smithi136 ceph-mon[109861]: pgmap v3248: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 506 op/s; 16582/89241 objects degraded (18.581%); 24798/89241 objects misplaced (27.788%); 9.2 MiB/s, 206 objects/s recovering 2024-08-17T18:19:37.368 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:37 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1376295280' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:19:37.369 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:19:37.369 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:19:37.369 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:37 smithi136 ceph-mon[115078]: pgmap v3248: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 506 op/s; 16582/89241 objects degraded (18.581%); 24798/89241 objects misplaced (27.788%); 9.2 MiB/s, 206 objects/s recovering 2024-08-17T18:19:37.369 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:37 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1376295280' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:19:37.372 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:19:37.372 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:19:37.399 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:19:37.399 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:19:37.426 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:19:37.427 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:19:37.459 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:37.494 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:37.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:37.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:37.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:19:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:19:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:37 smithi165 ceph-mon[105517]: pgmap v3248: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 506 op/s; 16582/89241 objects degraded (18.581%); 24798/89241 objects misplaced (27.788%); 9.2 MiB/s, 206 objects/s recovering 2024-08-17T18:19:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:37 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1376295280' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:19:37.596 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:37.628 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:37.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:37.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:37.693 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:19:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:19:37] "GET /metrics HTTP/1.1" 200 36773 "" "Prometheus/2.51.0" 2024-08-17T18:19:37.721 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-17T18:19:37.730 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:37.789 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:37.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:37.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:37.891 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:37.924 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:37.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:37.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:38.009 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:19:38.025 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:38.058 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:38.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:38.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:38.160 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:38.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:38.198 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:38.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:38.299 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:38.333 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:38.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:38.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:38.435 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:38.472 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:38.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:38.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:38.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:38.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:38.611 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:38.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:38.712 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:38.752 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:38.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:38.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:38.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:38.897 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:38.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:38.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:38.999 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:39.027 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:39.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:39.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:39.129 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:39.171 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:39.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:39.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:39.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:39.313 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:39.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:39.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:39.415 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:39.446 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:39.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:39.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:39.455 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:19:39.548 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:39.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:39.574 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:39.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:39.675 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:39.701 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:39.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:39.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:39.803 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:39.830 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:39.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:39.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:39.932 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:39.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:39 smithi136 ceph-mon[109861]: pgmap v3249: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 469 op/s; 15787/83205 objects degraded (18.974%); 23566/83205 objects misplaced (28.323%); 9.3 MiB/s, 214 objects/s recovering 2024-08-17T18:19:39.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:39 smithi136 ceph-mon[115078]: pgmap v3249: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 469 op/s; 15787/83205 objects degraded (18.974%); 23566/83205 objects misplaced (28.323%); 9.3 MiB/s, 214 objects/s recovering 2024-08-17T18:19:39.958 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:39.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:39.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:40.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:39 smithi165 ceph-mon[105517]: pgmap v3249: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 469 op/s; 15787/83205 objects degraded (18.974%); 23566/83205 objects misplaced (28.323%); 9.3 MiB/s, 214 objects/s recovering 2024-08-17T18:19:40.059 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:40.084 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:40.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:40.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:40.186 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:40.211 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:40.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:40.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:40.313 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:40.340 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:40.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:40.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:40.442 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:40.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:40.489 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:40.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:40.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:40.634 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:40.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:40.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:40.736 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:40.759 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:40.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:40.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:40 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 15787/83205 objects degraded (18.974%), 13 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:40.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:19:40.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:40 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 15787/83205 objects degraded (18.974%), 13 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:40.826 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:40 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 15787/83205 objects degraded (18.974%), 13 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:40.861 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:40.898 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:40.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:40.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:41.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:41.038 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:41.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:41.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:41.140 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:41.167 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:41.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:41.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:41.268 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:41.307 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:41.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:41.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:41.409 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:41.445 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:41.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:41.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:41.547 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:41.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:41.581 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:41.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:41.682 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:41.718 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:41.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:41.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:41.805 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:41 smithi136 ceph-mon[109861]: pgmap v3250: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 433 op/s; 15604/81438 objects degraded (19.161%); 23071/81438 objects misplaced (28.330%); 7.8 MiB/s, 175 objects/s recovering 2024-08-17T18:19:41.805 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:41 smithi136 ceph-mon[115078]: pgmap v3250: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 433 op/s; 15604/81438 objects degraded (19.161%); 23071/81438 objects misplaced (28.330%); 7.8 MiB/s, 175 objects/s recovering 2024-08-17T18:19:41.811 INFO:tasks.thrashosds.thrasher:in_osds: [2, 4, 6, 0, 5, 7, 3] out_osds: [1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:19:41.811 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:19:41.811 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-08-17T18:19:41.811 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T18:19:41.819 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:41.885 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:41.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:41.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:41.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:42.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:41 smithi165 ceph-mon[105517]: pgmap v3250: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 433 op/s; 15604/81438 objects degraded (19.161%); 23071/81438 objects misplaced (28.330%); 7.8 MiB/s, 175 objects/s recovering 2024-08-17T18:19:42.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:42.019 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:42.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:42.120 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:42.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:42.149 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:42.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:42.250 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:42.276 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:42.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:42.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:42.378 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:42.403 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:42.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:42.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:42.505 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:42.536 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:42.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:42.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:42.637 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:42.666 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:19:42.681 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:42.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:42.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:42.700 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:19:42.700 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:19:42.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:42.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:42.822 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:42.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:42.923 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:42.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:43.038 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:43.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:43.139 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:43.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:43.174 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:43.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:43.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:43.316 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:43.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:43.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:43.418 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:43.460 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:43.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:43.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:43.562 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:43.574 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:19:43.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:43.602 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:43.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:43.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:43 smithi136 ceph-mon[109861]: pgmap v3251: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.5 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 745 op/s; 14724/73764 objects degraded (19.961%); 20734/73764 objects misplaced (28.109%); 16 MiB/s, 310 objects/s recovering 2024-08-17T18:19:43.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:43 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2795368303' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:19:43.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:43 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:19:43.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:43 smithi136 ceph-mon[115078]: pgmap v3251: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.5 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 745 op/s; 14724/73764 objects degraded (19.961%); 20734/73764 objects misplaced (28.109%); 16 MiB/s, 310 objects/s recovering 2024-08-17T18:19:43.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:43 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2795368303' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:19:43.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:43 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:19:43.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:43.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:43.738 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:43.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:43.839 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:43.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:43.863 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:43.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:43.868 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-17T18:19:43.964 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:44.001 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:44.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:44.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:43 smithi165 ceph-mon[105517]: pgmap v3251: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.5 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 745 op/s; 14724/73764 objects degraded (19.961%); 20734/73764 objects misplaced (28.109%); 16 MiB/s, 310 objects/s recovering 2024-08-17T18:19:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:43 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2795368303' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:19:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:43 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:19:44.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:44.143 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:44.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:44.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:44.245 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:44.285 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:44.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:44.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:44.303 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:19:44.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:44.424 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:44.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:44.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:44.526 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:44.551 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:44.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:44.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:44.653 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:44.679 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:44.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:44.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:44.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:44 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:19:44.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:44 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:19:44.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:44 smithi136 ceph-mon[109861]: osdmap e1685: 8 total, 8 up, 7 in 2024-08-17T18:19:44.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:44 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:19:44.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:44 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:19:44.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:44 smithi136 ceph-mon[115078]: osdmap e1685: 8 total, 8 up, 7 in 2024-08-17T18:19:44.781 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:44.807 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:44.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:44.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:44.909 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:44.935 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:44.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:44.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:44 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:19:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:44 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:19:45.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:44 smithi165 ceph-mon[105517]: osdmap e1685: 8 total, 8 up, 7 in 2024-08-17T18:19:45.036 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:45.076 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:45.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:45.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:45.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:45.203 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:45.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:45.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:45.305 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:45.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:45.332 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:45.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:45.433 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:45.455 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:45.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:45.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:45.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:45.592 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:45.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:45.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:45.694 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:45.725 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:45.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:45.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:45.803 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:45 smithi136 ceph-mon[109861]: pgmap v3253: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.5 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 610 op/s; 14724/73764 objects degraded (19.961%); 20734/73764 objects misplaced (28.109%); 13 MiB/s, 254 objects/s recovering 2024-08-17T18:19:45.804 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:45 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 14724/73764 objects degraded (19.961%), 13 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:45.804 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:45 smithi136 ceph-mon[115078]: pgmap v3253: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.5 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 610 op/s; 14724/73764 objects degraded (19.961%); 20734/73764 objects misplaced (28.109%); 13 MiB/s, 254 objects/s recovering 2024-08-17T18:19:45.804 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:45 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 14724/73764 objects degraded (19.961%), 13 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:45.827 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:45.855 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:45.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:45.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:45.957 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:45.981 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:45.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:45.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:46.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:45 smithi165 ceph-mon[105517]: pgmap v3253: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.5 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 610 op/s; 14724/73764 objects degraded (19.961%); 20734/73764 objects misplaced (28.109%); 13 MiB/s, 254 objects/s recovering 2024-08-17T18:19:46.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:45 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 14724/73764 objects degraded (19.961%), 13 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:46.083 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:46.108 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:46.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:46.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:46.209 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:46.251 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:46.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:46.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:46.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:46.402 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:46.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:46.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:46.504 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:46.538 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:46.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:46.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:46.640 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:46.675 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:46.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:46.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:46.776 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:46.803 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:46.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:46.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:46.905 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:46.946 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:46.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:46.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:47.048 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:47.075 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:47.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:47.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:47.177 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:47.206 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:47.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:47.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:47.308 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:47.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:47.350 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:47.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:47.451 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:47.491 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:47.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:47.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:47.567 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:19:47.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:47.633 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:47.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:47.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:47.718 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:47 smithi136 ceph-mon[109861]: pgmap v3254: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 607 op/s; 14586/71004 objects degraded (20.543%); 19772/71004 objects misplaced (27.846%); 13 MiB/s, 248 objects/s recovering 2024-08-17T18:19:47.718 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:47 smithi136 ceph-mon[115078]: pgmap v3254: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 607 op/s; 14586/71004 objects degraded (20.543%); 19772/71004 objects misplaced (27.846%); 13 MiB/s, 248 objects/s recovering 2024-08-17T18:19:47.718 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:19:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:19:47] "GET /metrics HTTP/1.1" 200 36773 "" "Prometheus/2.51.0" 2024-08-17T18:19:47.735 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:47.765 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:47.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:47.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:47.867 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:47.911 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:47.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:47.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:47 smithi165 ceph-mon[105517]: pgmap v3254: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 607 op/s; 14586/71004 objects degraded (20.543%); 19772/71004 objects misplaced (27.846%); 13 MiB/s, 248 objects/s recovering 2024-08-17T18:19:48.012 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:48.041 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:48.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:48.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:48.142 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:48.171 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:48.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:48.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:48.242 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:19:48.273 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:48.318 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:48.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:48.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:48.420 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:48.453 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:48.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:48.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:48.544 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:19:48.544 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:19:48.554 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:48.577 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:19:48.578 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:19:48.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:48.594 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:48.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:48.612 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:19:48.612 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:19:48.643 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:19:48.643 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:19:48.644 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:48 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1686685262' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:19:48.644 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:48 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1686685262' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:19:48.674 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:19:48.674 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:19:48.696 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:48.715 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:19:48.715 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:19:48.739 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:48.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:48.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:48.757 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:19:48.757 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:19:48.798 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:19:48.798 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:19:48.841 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:48.873 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:48.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:48.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:48.975 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:49.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:48 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1686685262' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:19:49.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:49.013 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:49.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:49.114 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:49.155 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:49.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:49.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:49.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:49.284 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:49.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:49.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:49.386 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:49.410 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:49.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:49.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:49.512 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:49.539 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:49.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:49.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:49.641 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:49.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:49.678 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:49.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:49.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:49.829 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:49.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:49.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:49.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:49.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:49 smithi136 ceph-mon[109861]: pgmap v3255: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 653 op/s; 13621/63594 objects degraded (21.419%); 17969/63594 objects misplaced (28.256%); 15 MiB/s, 240 objects/s recovering 2024-08-17T18:19:49.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:49 smithi136 ceph-mon[115078]: pgmap v3255: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 653 op/s; 13621/63594 objects degraded (21.419%); 17969/63594 objects misplaced (28.256%); 15 MiB/s, 240 objects/s recovering 2024-08-17T18:19:49.971 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:49.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:49.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:49 smithi165 ceph-mon[105517]: pgmap v3255: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 653 op/s; 13621/63594 objects degraded (21.419%); 17969/63594 objects misplaced (28.256%); 15 MiB/s, 240 objects/s recovering 2024-08-17T18:19:50.074 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:50.100 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:50.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:50.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:50.202 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:50.239 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:50.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:50.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:50.341 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:50.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:50.381 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:50.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:50.482 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:50.507 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:50.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:50.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:50.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:50.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:50.645 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:50.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:50.746 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:50.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:50.772 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:50.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:50.873 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:50.904 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:50.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:50.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:50.940 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:50 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 13621/63594 objects degraded (21.419%), 13 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:50.940 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:50 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 13621/63594 objects degraded (21.419%), 13 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:51.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:51.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:50 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 13621/63594 objects degraded (21.419%), 13 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:51.041 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:51.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:51.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:51.143 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:51.185 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:51.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:51.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:51.288 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:51.330 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:51.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:51.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:51.431 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:51.464 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:51.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:51.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:51.566 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:51.593 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:51.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:51.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:51.695 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:51.736 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:51.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:51.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:51.838 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:51.864 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:51.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:51.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:51.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:51 smithi136 ceph-mon[109861]: pgmap v3256: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 660 op/s; 13527/61614 objects degraded (21.954%); 17403/61614 objects misplaced (28.245%); 15 MiB/s, 240 objects/s recovering 2024-08-17T18:19:51.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:19:51.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:19:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:51 smithi136 ceph-mon[115078]: pgmap v3256: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 660 op/s; 13527/61614 objects degraded (21.954%); 17403/61614 objects misplaced (28.245%); 15 MiB/s, 240 objects/s recovering 2024-08-17T18:19:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:19:51.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:19:51.966 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:51.988 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:51.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:51.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:51 smithi165 ceph-mon[105517]: pgmap v3256: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 660 op/s; 13527/61614 objects degraded (21.954%); 17403/61614 objects misplaced (28.245%); 15 MiB/s, 240 objects/s recovering 2024-08-17T18:19:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:19:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:19:52.090 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:52.119 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:52.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:52.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:52.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:52.262 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:52.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:52.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:52.364 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:52.390 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:52.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:52.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:52.491 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:52.528 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:52.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:52.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:52.577 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:19:52.579 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T18:19:52.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:52.673 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:52.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:52.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:52.774 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:52.799 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:52.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:52.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:52.901 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:52.961 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:52.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:52.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:53.062 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:53.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:53.113 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:53.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:53.214 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:53.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:53.290 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:53.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:53.391 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:53.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:53.447 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:53.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:53.549 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:53.611 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:53.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:53.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:53.713 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:53.757 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:53.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:53.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:53.859 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:53.899 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:53.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:53.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:53.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:53 smithi136 ceph-mon[109861]: from='client.46709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:19:53.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:53 smithi136 ceph-mon[109861]: pgmap v3257: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 701 op/s; 12505/52722 objects degraded (23.719%); 14914/52722 objects misplaced (28.288%); 16 MiB/s, 257 objects/s recovering 2024-08-17T18:19:53.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:53 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:19:53.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:53 smithi136 ceph-mon[115078]: from='client.46709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:19:53.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:53 smithi136 ceph-mon[115078]: pgmap v3257: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 701 op/s; 12505/52722 objects degraded (23.719%); 14914/52722 objects misplaced (28.288%); 16 MiB/s, 257 objects/s recovering 2024-08-17T18:19:53.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:53 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:19:53.994 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:53 smithi165 ceph-mon[105517]: from='client.46709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:19:53.994 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:53 smithi165 ceph-mon[105517]: pgmap v3257: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 701 op/s; 12505/52722 objects degraded (23.719%); 14914/52722 objects misplaced (28.288%); 16 MiB/s, 257 objects/s recovering 2024-08-17T18:19:53.994 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:53 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:19:54.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:54.049 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:54.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:54.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:54.151 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:54.177 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:54.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:54.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:54.278 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:54.320 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:54.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:54.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:54.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:54.471 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:54.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:54.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:54.509 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:19:54.535 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":3256,"stamp":"2024-08-17T18:19:50.495927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1344912200,"num_objects":20538,"num_object_clones":0,"num_object_copies":61614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13527,"num_objects_misplaced":17403,"num_objects_unfound":0,"num_objects_dirty":20538,"num_whiteouts":0,"num_read":70,"num_read_kb":150,"num_write":211431,"num_write_kb":7419289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46500,"num_bytes_recovered":2629865512,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134572,"ondisk_log_size":134572,"up":327,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":421,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":14408992,"kb_used_data":7455780,"kb_used_omap":285,"kb_used_meta":6952674,"kb_avail":641807072,"statfs":{"total":671965249536,"available":657210441728,"internally_reserved":0,"allocated":7634718720,"data_stored":7603570670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":292552,"internal_metadata":7119538488},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-433061888,"num_objects":-6608,"num_object_clones":0,"num_object_copies":-19824,"num_objects_missing_on_primary":-39,"num_objects_missing":-39,"num_objects_degraded":-2077,"num_objects_misplaced":-5668,"num_objects_unfound":0,"num_objects_dirty":-6608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6608,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2410,"num_bytes_recovered":157941760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004185"},"pg_stats":[{"pgid":"57.19","version":"1685'9113","reported_seq":9223,"reported_epoch":1685,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:19:49.942192+0000","last_change":"2024-08-17T18:19:02.633485+0000","last_active":"2024-08-17T18:19:49.942192+0000","last_peered":"2024-08-17T18:19:49.942192+0000","last_clean":"2024-08-17T18:18:29.597061+0000","last_became_active":"2024-08-17T18:18:30.810367+0000","last_became_peered":"2024-08-17T18:18:30.810367+0000","last_unstale":"2024-08-17T18:19:49.942192+0000","last_undegraded":"2024-08-17T18:19:49.942192+0000","last_fullsized":"2024-08-17T18:19:49.942192+0000","mapping_epoch":1667,"log_start":"1614'4201","ondisk_log_start":"1614'4201","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4912,"log_dups_size":0,"ondisk_log_size":4912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36995072,"num_objects":565,"num_object_clones":0,"num_object_copies":1695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":565,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6635,"num_write_kb":230368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[4,5,6],"avail_no_missing":["4","5","6"],"object_location_counts":[{"shards":"4,5,6","objects":565}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"57.18","version":"1685'9178","reported_seq":14904,"reported_epoch":1685,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:19:47.172372+0000","last_change":"2024-08-17T18:15:54.579791+0000","last_active":"2024-08-17T18:19:47.172372+0000","last_peered":"2024-08-17T18:19:47.172372+0000","last_clean":"2024-08-17T18:15:09.641861+0000","last_became_active":"2024-08-17T18:15:54.569392+0000","last_became_peered":"2024-08-17T18:15:54.569392+0000","last_unstale":"2024-08-17T18:19:47.172372+0000","last_undegraded":"2024-08-17T18:15:54.395452+0000","last_fullsized":"2024-08-17T18:19:47.172372+0000","mapping_epoch":1638,"log_start":"1618'5327","ondisk_log_start":"1618'5327","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3851,"log_dups_size":0,"ondisk_log_size":3851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41451520,"num_objects":633,"num_object_clones":0,"num_object_copies":1899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":423,"num_objects_misplaced":210,"num_objects_unfound":0,"num_objects_dirty":633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6671,"num_write_kb":233696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2804,"num_bytes_recovered":183762944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[5,6,4],"avail_no_missing":["5","0","6"],"object_location_counts":[{"shards":"0,5,6","objects":208},{"shards":"4,5,6","objects":425}],"blocked_by":[],"up_primary":7,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.17","version":"1685'8863","reported_seq":12925,"reported_epoch":1685,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:19:47.371228+0000","last_change":"2024-08-17T18:18:29.904207+0000","last_active":"2024-08-17T18:19:47.371228+0000","last_peered":"2024-08-17T18:19:47.371228+0000","last_clean":"2024-08-17T18:15:52.210567+0000","last_became_active":"2024-08-17T18:17:23.718959+0000","last_became_peered":"2024-08-17T18:17:23.718959+0000","last_unstale":"2024-08-17T18:19:47.371228+0000","last_undegraded":"2024-08-17T18:18:29.902629+0000","last_fullsized":"2024-08-17T18:17:23.630677+0000","mapping_epoch":1653,"log_start":"1614'4549","ondisk_log_start":"1614'4549","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4314,"log_dups_size":0,"ondisk_log_size":4314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40337408,"num_objects":616,"num_object_clones":0,"num_object_copies":1848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6432,"num_write_kb":225504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1925,"num_bytes_recovered":126156800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,4,6","objects":598},{"shards":"2,7","objects":18}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"57.16","version":"1685'8855","reported_seq":16090,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:47.139485+0000","last_change":"2024-08-17T18:17:57.587085+0000","last_active":"2024-08-17T18:19:47.139485+0000","last_peered":"2024-08-17T18:19:47.139485+0000","last_clean":"2024-08-17T18:19:47.139485+0000","last_became_active":"2024-08-17T18:17:57.576108+0000","last_became_peered":"2024-08-17T18:17:57.576108+0000","last_unstale":"2024-08-17T18:19:47.139485+0000","last_undegraded":"2024-08-17T18:19:47.139485+0000","last_fullsized":"2024-08-17T18:19:47.139485+0000","mapping_epoch":1660,"log_start":"1617'5005","ondisk_log_start":"1617'5005","created":1616,"last_epoch_clean":1661,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3850,"log_dups_size":0,"ondisk_log_size":3850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40206336,"num_objects":614,"num_object_clones":0,"num_object_copies":1842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6420,"num_write_kb":225056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":166985728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.15","version":"1685'9064","reported_seq":15006,"reported_epoch":1685,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:19:47.094774+0000","last_change":"2024-08-17T18:18:32.060713+0000","last_active":"2024-08-17T18:19:47.094774+0000","last_peered":"2024-08-17T18:19:47.094774+0000","last_clean":"2024-08-17T18:15:09.587418+0000","last_became_active":"2024-08-17T18:18:32.037569+0000","last_became_peered":"2024-08-17T18:18:32.037569+0000","last_unstale":"2024-08-17T18:19:47.094774+0000","last_undegraded":"2024-08-17T18:18:31.866840+0000","last_fullsized":"2024-08-17T18:19:47.094774+0000","mapping_epoch":1668,"log_start":"1617'5169","ondisk_log_start":"1617'5169","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3895,"log_dups_size":0,"ondisk_log_size":3895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49119232,"num_objects":750,"num_object_clones":0,"num_object_copies":2250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":443,"num_objects_unfound":0,"num_objects_dirty":750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6562,"num_write_kb":233952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1934,"num_bytes_recovered":126681088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,7,2],"avail_no_missing":["5","0","6"],"object_location_counts":[{"shards":"0,5,6","objects":217},{"shards":"2,5,6","objects":443},{"shards":"2,5,7","objects":90}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.14","version":"1685'8909","reported_seq":11249,"reported_epoch":1685,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:19:47.141057+0000","last_change":"2024-08-17T18:18:32.046673+0000","last_active":"2024-08-17T18:19:47.141057+0000","last_peered":"2024-08-17T18:19:47.141057+0000","last_clean":"2024-08-17T18:15:09.651475+0000","last_became_active":"2024-08-17T18:18:31.915649+0000","last_became_peered":"2024-08-17T18:18:31.915649+0000","last_unstale":"2024-08-17T18:19:47.141057+0000","last_undegraded":"2024-08-17T18:18:31.906636+0000","last_fullsized":"2024-08-17T18:19:47.141057+0000","mapping_epoch":1668,"log_start":"1617'5026","ondisk_log_start":"1617'5026","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3883,"log_dups_size":0,"ondisk_log_size":3883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42991616,"num_objects":656,"num_object_clones":0,"num_object_copies":1968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":656,"num_objects_misplaced":2473,"num_objects_unfound":0,"num_objects_dirty":656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6456,"num_write_kb":227584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1086,"num_bytes_recovered":71172096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[5,7,4],"avail_no_missing":["5","0","4"],"object_location_counts":[{"shards":"0,4,5","objects":217},{"shards":"4,5,6","objects":441},{"shards":"4,5,7","objects":-2}],"blocked_by":[],"up_primary":3,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.13","version":"1685'8946","reported_seq":16150,"reported_epoch":1685,"state":"active+recovering+remapped","last_fresh":"2024-08-17T18:19:47.165005+0000","last_change":"2024-08-17T18:19:36.947579+0000","last_active":"2024-08-17T18:19:47.165005+0000","last_peered":"2024-08-17T18:19:47.165005+0000","last_clean":"2024-08-17T18:15:52.116504+0000","last_became_active":"2024-08-17T18:18:31.877372+0000","last_became_peered":"2024-08-17T18:18:31.877372+0000","last_unstale":"2024-08-17T18:19:47.165005+0000","last_undegraded":"2024-08-17T18:19:47.165005+0000","last_fullsized":"2024-08-17T18:19:47.165005+0000","mapping_epoch":1668,"log_start":"1613'1840","ondisk_log_start":"1613'1840","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":7106,"log_dups_size":0,"ondisk_log_size":7106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43679744,"num_objects":667,"num_object_clones":0,"num_object_copies":2001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":542,"num_objects_unfound":0,"num_objects_dirty":667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6497,"num_write_kb":229216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3108,"num_bytes_recovered":150470656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[6,7,0],"avail_no_missing":["6","0","2"],"object_location_counts":[{"shards":"0,2,6","objects":271},{"shards":"0,6,7","objects":396}],"blocked_by":[],"up_primary":3,"acting_primary":6,"purged_snaps":[]},{"pgid":"57.12","version":"1685'9026","reported_seq":17802,"reported_epoch":1685,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:19:49.897176+0000","last_change":"2024-08-17T18:18:34.413365+0000","last_active":"2024-08-17T18:19:49.897176+0000","last_peered":"2024-08-17T18:19:49.897176+0000","last_clean":"2024-08-17T18:14:59.641932+0000","last_became_active":"2024-08-17T18:18:33.841392+0000","last_became_peered":"2024-08-17T18:18:33.841392+0000","last_unstale":"2024-08-17T18:19:49.897176+0000","last_undegraded":"2024-08-17T18:18:32.544363+0000","last_fullsized":"2024-08-17T18:18:30.719339+0000","mapping_epoch":1667,"log_start":"1614'4018","ondisk_log_start":"1614'4018","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39813120,"num_objects":608,"num_object_clones":0,"num_object_copies":1824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6570,"num_write_kb":229664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2974,"num_bytes_recovered":194838528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[4,7],"avail_no_missing":["4","0","5"],"object_location_counts":[{"shards":"0,4,5","objects":151},{"shards":"2,4,5","objects":441},{"shards":"4,7","objects":16}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"57.11","version":"1685'9228","reported_seq":9448,"reported_epoch":1685,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:19:46.298766+0000","last_change":"2024-08-17T18:18:59.889490+0000","last_active":"2024-08-17T18:19:46.298766+0000","last_peered":"2024-08-17T18:19:46.298766+0000","last_clean":"2024-08-17T18:14:57.659306+0000","last_became_active":"2024-08-17T18:18:29.827929+0000","last_became_peered":"2024-08-17T18:18:29.827929+0000","last_unstale":"2024-08-17T18:19:46.298766+0000","last_undegraded":"2024-08-17T18:19:46.298766+0000","last_fullsized":"2024-08-17T18:19:46.298766+0000","mapping_epoch":1666,"log_start":"1611'100","ondisk_log_start":"1611'100","created":1616,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":9128,"log_dups_size":0,"ondisk_log_size":9128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43745280,"num_objects":668,"num_object_clones":0,"num_object_copies":2004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2004,"num_objects_unfound":0,"num_objects_dirty":668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6708,"num_write_kb":236000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[0,6,2],"avail_no_missing":["0","2","6"],"object_location_counts":[{"shards":"0,2,6","objects":668}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"57.10","version":"1685'9117","reported_seq":21864,"reported_epoch":1685,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:19:47.145522+0000","last_change":"2024-08-17T18:18:31.925795+0000","last_active":"2024-08-17T18:19:47.145522+0000","last_peered":"2024-08-17T18:19:47.145522+0000","last_clean":"2024-08-17T18:15:52.133173+0000","last_became_active":"2024-08-17T18:18:31.867521+0000","last_became_peered":"2024-08-17T18:18:31.867521+0000","last_unstale":"2024-08-17T18:19:47.145522+0000","last_undegraded":"2024-08-17T18:18:31.852455+0000","last_fullsized":"2024-08-17T18:19:47.145522+0000","mapping_epoch":1668,"log_start":"1613'2161","ondisk_log_start":"1613'2161","created":1616,"last_epoch_clean":1632,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":6956,"log_dups_size":0,"ondisk_log_size":6956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38699008,"num_objects":591,"num_object_clones":0,"num_object_copies":1773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1448,"num_objects_misplaced":823,"num_objects_unfound":0,"num_objects_dirty":591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6625,"num_write_kb":230880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4844,"num_bytes_recovered":317456384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,2],"avail_no_missing":["6","0","4","5"],"object_location_counts":[{"shards":"2,5,6","objects":625},{"shards":"2,6,7","objects":-34}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"57.f","version":"1685'13412","reported_seq":13628,"reported_epoch":1685,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:19:47.342332+0000","last_change":"2024-08-17T18:15:53.709162+0000","last_active":"2024-08-17T18:19:47.342332+0000","last_peered":"2024-08-17T18:19:47.342332+0000","last_clean":"2024-08-17T18:15:09.658885+0000","last_became_active":"2024-08-17T18:15:53.364563+0000","last_became_peered":"2024-08-17T18:15:53.364563+0000","last_unstale":"2024-08-17T18:19:47.342332+0000","last_undegraded":"2024-08-17T18:19:47.342332+0000","last_fullsized":"2024-08-17T18:19:47.342332+0000","mapping_epoch":1637,"log_start":"1650'9158","ondisk_log_start":"1650'9158","created":1604,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4254,"log_dups_size":0,"ondisk_log_size":4254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84934656,"num_objects":1296,"num_object_clones":0,"num_object_copies":3888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1295,"num_objects_unfound":0,"num_objects_dirty":1296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13412,"num_write_kb":470656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":458752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,6,0],"avail_no_missing":["2","0","6"],"object_location_counts":[{"shards":"0,2,6","objects":1296}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"57.e","version":"1685'13388","reported_seq":13496,"reported_epoch":1685,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:19:49.893845+0000","last_change":"2024-08-17T18:17:23.713508+0000","last_active":"2024-08-17T18:19:49.893845+0000","last_peered":"2024-08-17T18:19:49.893845+0000","last_clean":"2024-08-17T18:15:52.176790+0000","last_became_active":"2024-08-17T18:17:23.688213+0000","last_became_peered":"2024-08-17T18:17:23.688213+0000","last_unstale":"2024-08-17T18:19:49.893845+0000","last_undegraded":"2024-08-17T18:19:49.893845+0000","last_fullsized":"2024-08-17T18:19:49.893845+0000","mapping_epoch":1653,"log_start":"1650'8818","ondisk_log_start":"1650'8818","created":1604,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4570,"log_dups_size":0,"ondisk_log_size":4570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:12:51.202869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76021760,"num_objects":1160,"num_object_clones":0,"num_object_copies":3480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1160,"num_objects_unfound":0,"num_objects_dirty":1160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13388,"num_write_kb":465536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[4,5,6],"avail_no_missing":["4","5","6"],"object_location_counts":[{"shards":"4,5,6","objects":1160}],"blocked_by":[],"up_primary":7,"acting_primary":4,"purged_snaps":[]},{"pgid":"57.d","version":"1685'13114","reported_seq":14280,"reported_epoch":1685,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:19:46.201893+0000","last_change":"2024-08-17T18:18:30.832971+0000","last_active":"2024-08-17T18:19:46.201893+0000","last_peered":"2024-08-17T18:19:46.201893+0000","last_clean":"2024-08-17T18:17:21.525603+0000","last_became_active":"2024-08-17T18:18:29.819768+0000","last_became_peered":"2024-08-17T18:18:29.819768+0000","last_unstale":"2024-08-17T18:19:46.201893+0000","last_undegraded":"2024-08-17T18:19:46.201893+0000","last_fullsized":"2024-08-17T18:19:46.201893+0000","mapping_epoch":1666,"log_start":"1617'5227","ondisk_log_start":"1617'5227","created":1604,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":7887,"log_dups_size":0,"ondisk_log_size":7887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89063448,"num_objects":1360,"num_object_clones":0,"num_object_copies":4080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1166,"num_objects_unfound":0,"num_objects_dirty":1360,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13114,"num_write_kb":463105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":508,"num_bytes_recovered":33292288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,2,6],"avail_no_missing":["0","2","6"],"object_location_counts":[{"shards":"0,2,6","objects":1360}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"57.c","version":"1685'13432","reported_seq":13524,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:47.158422+0000","last_change":"2024-08-17T18:13:54.756490+0000","last_active":"2024-08-17T18:19:47.158422+0000","last_peered":"2024-08-17T18:19:47.158422+0000","last_clean":"2024-08-17T18:19:47.158422+0000","last_became_active":"2024-08-17T18:13:54.756109+0000","last_became_peered":"2024-08-17T18:13:54.756109+0000","last_unstale":"2024-08-17T18:19:47.158422+0000","last_undegraded":"2024-08-17T18:19:47.158422+0000","last_fullsized":"2024-08-17T18:19:47.158422+0000","mapping_epoch":1604,"log_start":"1643'7833","ondisk_log_start":"1643'7833","created":1604,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5599,"log_dups_size":0,"ondisk_log_size":5599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:09:29.691722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87293952,"num_objects":1332,"num_object_clones":0,"num_object_copies":3996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13432,"num_write_kb":472448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"57.b","version":"1685'13092","reported_seq":13274,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:47.354536+0000","last_change":"2024-08-17T18:13:54.755437+0000","last_active":"2024-08-17T18:19:47.354536+0000","last_peered":"2024-08-17T18:19:47.354536+0000","last_clean":"2024-08-17T18:19:47.354536+0000","last_became_active":"2024-08-17T18:13:54.755175+0000","last_became_peered":"2024-08-17T18:13:54.755175+0000","last_unstale":"2024-08-17T18:19:47.354536+0000","last_undegraded":"2024-08-17T18:19:47.354536+0000","last_fullsized":"2024-08-17T18:19:47.354536+0000","mapping_epoch":1604,"log_start":"1650'8856","ondisk_log_start":"1650'8856","created":1604,"last_epoch_clean":1605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4236,"log_dups_size":0,"ondisk_log_size":4236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:31.707115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86245376,"num_objects":1316,"num_object_clones":0,"num_object_copies":3948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13092,"num_write_kb":461056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"57.a","version":"1685'13320","reported_seq":13436,"reported_epoch":1685,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:19:47.156942+0000","last_change":"2024-08-17T18:18:29.943391+0000","last_active":"2024-08-17T18:19:47.156942+0000","last_peered":"2024-08-17T18:19:47.156942+0000","last_clean":"2024-08-17T18:15:52.195344+0000","last_became_active":"2024-08-17T18:18:29.889053+0000","last_became_peered":"2024-08-17T18:18:29.889053+0000","last_unstale":"2024-08-17T18:19:47.156942+0000","last_undegraded":"2024-08-17T18:19:47.156942+0000","last_fullsized":"2024-08-17T18:19:47.156942+0000","mapping_epoch":1666,"log_start":"1654'9407","ondisk_log_start":"1654'9407","created":1604,"last_epoch_clean":1609,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3913,"log_dups_size":0,"ondisk_log_size":3913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:46:14.340957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85721088,"num_objects":1308,"num_object_clones":0,"num_object_copies":3924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2616,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13320,"num_write_kb":468096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,4,2],"avail_no_missing":["5","2","4"],"object_location_counts":[{"shards":"2,4,5","objects":1308}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.9","version":"1685'9135","reported_seq":9245,"reported_epoch":1685,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:19:49.919725+0000","last_change":"2024-08-17T18:18:30.827730+0000","last_active":"2024-08-17T18:19:49.919725+0000","last_peered":"2024-08-17T18:19:49.919725+0000","last_clean":"2024-08-17T18:18:29.565716+0000","last_became_active":"2024-08-17T18:18:30.810210+0000","last_became_peered":"2024-08-17T18:18:30.810210+0000","last_unstale":"2024-08-17T18:19:49.919725+0000","last_undegraded":"2024-08-17T18:19:49.919725+0000","last_fullsized":"2024-08-17T18:19:49.919725+0000","mapping_epoch":1667,"log_start":"1614'4195","ondisk_log_start":"1614'4195","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4940,"log_dups_size":0,"ondisk_log_size":4940,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39485440,"num_objects":602,"num_object_clones":0,"num_object_copies":1806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":602,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6656,"num_write_kb":232288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[4,5,6],"avail_no_missing":["4","5","6"],"object_location_counts":[{"shards":"4,5,6","objects":602}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"57.8","version":"1685'9045","reported_seq":16275,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:46.875330+0000","last_change":"2024-08-17T18:19:04.670956+0000","last_active":"2024-08-17T18:19:46.875330+0000","last_peered":"2024-08-17T18:19:46.875330+0000","last_clean":"2024-08-17T18:19:46.875330+0000","last_became_active":"2024-08-17T18:19:04.648021+0000","last_became_peered":"2024-08-17T18:19:04.648021+0000","last_unstale":"2024-08-17T18:19:46.875330+0000","last_undegraded":"2024-08-17T18:19:46.875330+0000","last_fullsized":"2024-08-17T18:19:46.875330+0000","mapping_epoch":1676,"log_start":"1613'3854","ondisk_log_start":"1613'3854","created":1604,"last_epoch_clean":1677,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":5191,"log_dups_size":0,"ondisk_log_size":5191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42303488,"num_objects":645,"num_object_clones":0,"num_object_copies":1935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6537,"num_write_kb":229856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3551,"num_bytes_recovered":109117440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"57.7","version":"1685'8970","reported_seq":17209,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:47.342259+0000","last_change":"2024-08-17T18:19:46.066740+0000","last_active":"2024-08-17T18:19:47.342259+0000","last_peered":"2024-08-17T18:19:47.342259+0000","last_clean":"2024-08-17T18:19:47.342259+0000","last_became_active":"2024-08-17T18:19:27.606570+0000","last_became_peered":"2024-08-17T18:19:27.606570+0000","last_unstale":"2024-08-17T18:19:47.342259+0000","last_undegraded":"2024-08-17T18:19:47.342259+0000","last_fullsized":"2024-08-17T18:19:47.342259+0000","mapping_epoch":1680,"log_start":"1615'4714","ondisk_log_start":"1615'4714","created":1604,"last_epoch_clean":1681,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4256,"log_dups_size":0,"ondisk_log_size":4256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T18:19:51.032290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40402944,"num_objects":616,"num_object_clones":0,"num_object_copies":1848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6538,"num_write_kb":228960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3595,"num_bytes_recovered":168165376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"57.6","version":"1685'8902","reported_seq":9020,"reported_epoch":1685,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:19:47.068670+0000","last_change":"2024-08-17T18:18:29.888707+0000","last_active":"2024-08-17T18:19:47.068670+0000","last_peered":"2024-08-17T18:19:47.068670+0000","last_clean":"2024-08-17T18:15:09.669695+0000","last_became_active":"2024-08-17T18:15:53.694052+0000","last_became_peered":"2024-08-17T18:15:53.694052+0000","last_unstale":"2024-08-17T18:19:47.068670+0000","last_undegraded":"2024-08-17T18:15:53.322865+0000","last_fullsized":"2024-08-17T18:15:53.252808+0000","mapping_epoch":1637,"log_start":"1617'4981","ondisk_log_start":"1617'4981","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3921,"log_dups_size":0,"ondisk_log_size":3921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45645824,"num_objects":696,"num_object_clones":0,"num_object_copies":2088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3544,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6466,"num_write_kb":229216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2],"avail_no_missing":["5","0","2"],"object_location_counts":[{"shards":"0,2,5","objects":231},{"shards":"2,4,5","objects":427},{"shards":"2,5","objects":-2649}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.5","version":"1685'9218","reported_seq":15113,"reported_epoch":1685,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:19:47.196655+0000","last_change":"2024-08-17T18:18:30.123049+0000","last_active":"2024-08-17T18:19:47.196655+0000","last_peered":"2024-08-17T18:19:47.196655+0000","last_clean":"2024-08-17T18:15:09.654427+0000","last_became_active":"2024-08-17T18:18:30.099398+0000","last_became_peered":"2024-08-17T18:18:30.099398+0000","last_unstale":"2024-08-17T18:19:47.196655+0000","last_undegraded":"2024-08-17T18:18:30.063992+0000","last_fullsized":"2024-08-17T18:19:47.196655+0000","mapping_epoch":1666,"log_start":"1618'5323","ondisk_log_start":"1618'5323","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":3895,"log_dups_size":0,"ondisk_log_size":3895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36536320,"num_objects":557,"num_object_clones":0,"num_object_copies":1671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":419,"num_objects_misplaced":695,"num_objects_unfound":0,"num_objects_dirty":557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6715,"num_write_kb":232736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2874,"num_bytes_recovered":188284928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,2,6],"avail_no_missing":["5","0","6"],"object_location_counts":[{"shards":"0,5,6","objects":212},{"shards":"2,5,6","objects":345}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"57.4","version":"1685'9063","reported_seq":16417,"reported_epoch":1685,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:19:49.906688+0000","last_change":"2024-08-17T18:18:32.134646+0000","last_active":"2024-08-17T18:19:49.906688+0000","last_peered":"2024-08-17T18:19:49.906688+0000","last_clean":"2024-08-17T18:18:29.526320+0000","last_became_active":"2024-08-17T18:18:32.122736+0000","last_became_peered":"2024-08-17T18:18:32.122736+0000","last_unstale":"2024-08-17T18:19:49.906688+0000","last_undegraded":"2024-08-17T18:19:49.906688+0000","last_fullsized":"2024-08-17T18:19:49.906688+0000","mapping_epoch":1668,"log_start":"1614'4113","ondisk_log_start":"1614'4113","created":1604,"last_epoch_clean":1647,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4950,"log_dups_size":0,"ondisk_log_size":4950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":576,"num_object_clones":0,"num_object_copies":1728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":576,"num_objects_unfound":0,"num_objects_dirty":576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6610,"num_write_kb":229952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3591,"num_bytes_recovered":223805440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[4,5,7],"avail_no_missing":["4","5","7"],"object_location_counts":[{"shards":"4,5,7","objects":576}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"57.3","version":"1685'9031","reported_seq":13181,"reported_epoch":1685,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:19:47.122997+0000","last_change":"2024-08-17T18:18:31.936546+0000","last_active":"2024-08-17T18:19:47.122997+0000","last_peered":"2024-08-17T18:19:47.122997+0000","last_clean":"2024-08-17T18:15:52.162248+0000","last_became_active":"2024-08-17T18:18:31.914458+0000","last_became_peered":"2024-08-17T18:18:31.914458+0000","last_unstale":"2024-08-17T18:19:47.122997+0000","last_undegraded":"2024-08-17T18:18:31.855752+0000","last_fullsized":"2024-08-17T18:19:47.122997+0000","mapping_epoch":1668,"log_start":"1613'2117","ondisk_log_start":"1613'2117","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":6914,"log_dups_size":0,"ondisk_log_size":6914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42434560,"num_objects":647,"num_object_clones":0,"num_object_copies":1941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1448,"num_objects_misplaced":851,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6581,"num_write_kb":231328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":130088960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[6,7,0],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"0,2,6","objects":670},{"shards":"0,6,7","objects":-23}],"blocked_by":[],"up_primary":3,"acting_primary":6,"purged_snaps":[]},{"pgid":"57.2","version":"1685'9080","reported_seq":11869,"reported_epoch":1685,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:19:47.376987+0000","last_change":"2024-08-17T18:18:33.840312+0000","last_active":"2024-08-17T18:19:47.376987+0000","last_peered":"2024-08-17T18:19:47.376987+0000","last_clean":"2024-08-17T18:15:09.574793+0000","last_became_active":"2024-08-17T18:18:33.824434+0000","last_became_peered":"2024-08-17T18:18:33.824434+0000","last_unstale":"2024-08-17T18:19:47.376987+0000","last_undegraded":"2024-08-17T18:18:30.818589+0000","last_fullsized":"2024-08-17T18:19:47.376987+0000","mapping_epoch":1667,"log_start":"1615'4822","ondisk_log_start":"1615'4822","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4258,"log_dups_size":0,"ondisk_log_size":4258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42696704,"num_objects":651,"num_object_clones":0,"num_object_copies":1953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":651,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6623,"num_write_kb":232800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1288,"num_bytes_recovered":84410368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[2,5,4],"avail_no_missing":["2","0","5"],"object_location_counts":[{"shards":"0,2,5","objects":197},{"shards":"2,4,5","objects":454}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"57.1","version":"1685'9202","reported_seq":9380,"reported_epoch":1685,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:19:47.114448+0000","last_change":"2024-08-17T18:15:53.720586+0000","last_active":"2024-08-17T18:19:47.114448+0000","last_peered":"2024-08-17T18:19:47.114448+0000","last_clean":"2024-08-17T18:15:09.579791+0000","last_became_active":"2024-08-17T18:15:53.381797+0000","last_became_peered":"2024-08-17T18:15:53.381797+0000","last_unstale":"2024-08-17T18:19:47.114448+0000","last_undegraded":"2024-08-17T18:19:47.114448+0000","last_fullsized":"2024-08-17T18:19:47.114448+0000","mapping_epoch":1637,"log_start":"1613'2259","ondisk_log_start":"1613'2259","created":1604,"last_epoch_clean":1617,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":6943,"log_dups_size":0,"ondisk_log_size":6943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46104576,"num_objects":703,"num_object_clones":0,"num_object_copies":2109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":703,"num_objects_unfound":0,"num_objects_dirty":703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6681,"num_write_kb":236320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":589824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,0],"avail_no_missing":["6","0","2"],"object_location_counts":[{"shards":"0,2,6","objects":703}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"57.0","version":"1685'9080","reported_seq":22858,"reported_epoch":1685,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:19:47.346886+0000","last_change":"2024-08-17T18:18:31.919624+0000","last_active":"2024-08-17T18:19:47.346886+0000","last_peered":"2024-08-17T18:19:47.346886+0000","last_clean":"2024-08-17T18:18:29.609086+0000","last_became_active":"2024-08-17T18:18:31.860120+0000","last_became_peered":"2024-08-17T18:18:31.860120+0000","last_unstale":"2024-08-17T18:19:47.346886+0000","last_undegraded":"2024-08-17T18:19:47.346886+0000","last_fullsized":"2024-08-17T18:19:47.346886+0000","mapping_epoch":1668,"log_start":"1615'4836","ondisk_log_start":"1615'4836","created":1604,"last_epoch_clean":1665,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:13:53.730820+0000","last_clean_scrub_stamp":"2024-08-17T18:13:53.730820+0000","objects_scrubbed":0,"log_size":4244,"log_dups_size":0,"ondisk_log_size":4244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44531712,"num_objects":679,"num_object_clones":0,"num_object_copies":2037,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":679,"num_objects_unfound":0,"num_objects_dirty":679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6587,"num_write_kb":232544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6772,"num_bytes_recovered":353239040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,7],"avail_no_missing":["2","0","4","5","7"],"object_location_counts":[{"shards":"2,4,7","objects":679}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":122,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.730592+0000","last_change":"2024-08-17T18:17:22.697769+0000","last_active":"2024-08-17T18:19:34.730592+0000","last_peered":"2024-08-17T18:19:34.730592+0000","last_clean":"2024-08-17T18:19:34.730592+0000","last_became_active":"2024-08-17T18:17:22.688989+0000","last_became_peered":"2024-08-17T18:17:22.688989+0000","last_unstale":"2024-08-17T18:19:34.730592+0000","last_undegraded":"2024-08-17T18:19:34.730592+0000","last_fullsized":"2024-08-17T18:19:34.730592+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:10:04.677811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2328,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1728,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.754642+0000","last_change":"2024-08-17T18:18:29.727855+0000","last_active":"2024-08-17T18:19:34.754642+0000","last_peered":"2024-08-17T18:19:34.754642+0000","last_clean":"2024-08-17T18:19:34.754642+0000","last_became_active":"2024-08-17T18:18:29.727091+0000","last_became_peered":"2024-08-17T18:18:29.727091+0000","last_unstale":"2024-08-17T18:19:34.754642+0000","last_undegraded":"2024-08-17T18:19:34.754642+0000","last_fullsized":"2024-08-17T18:19:34.754642+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:04.660094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":48,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.581644+0000","last_change":"2024-08-17T18:18:29.895953+0000","last_active":"2024-08-17T18:19:43.581644+0000","last_peered":"2024-08-17T18:19:43.581644+0000","last_clean":"2024-08-17T18:19:43.581644+0000","last_became_active":"2024-08-17T18:18:29.895460+0000","last_became_peered":"2024-08-17T18:18:29.895460+0000","last_unstale":"2024-08-17T18:19:43.581644+0000","last_undegraded":"2024-08-17T18:19:43.581644+0000","last_fullsized":"2024-08-17T18:19:43.581644+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:04:41.818991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":48,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.582112+0000","last_change":"2024-08-17T18:18:29.732425+0000","last_active":"2024-08-17T18:19:43.582112+0000","last_peered":"2024-08-17T18:19:43.582112+0000","last_clean":"2024-08-17T18:19:43.582112+0000","last_became_active":"2024-08-17T18:18:29.731726+0000","last_became_peered":"2024-08-17T18:18:29.731726+0000","last_unstale":"2024-08-17T18:19:43.582112+0000","last_undegraded":"2024-08-17T18:19:43.582112+0000","last_fullsized":"2024-08-17T18:19:43.582112+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:01:33.556710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":240,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.737005+0000","last_change":"2024-08-17T18:04:43.777705+0000","last_active":"2024-08-17T18:19:34.737005+0000","last_peered":"2024-08-17T18:19:34.737005+0000","last_clean":"2024-08-17T18:19:34.737005+0000","last_became_active":"2024-08-17T18:04:43.771283+0000","last_became_peered":"2024-08-17T18:04:43.771283+0000","last_unstale":"2024-08-17T18:19:34.737005+0000","last_undegraded":"2024-08-17T18:19:34.737005+0000","last_fullsized":"2024-08-17T18:19:34.737005+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:13.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":48,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.580723+0000","last_change":"2024-08-17T18:18:29.734730+0000","last_active":"2024-08-17T18:19:43.580723+0000","last_peered":"2024-08-17T18:19:43.580723+0000","last_clean":"2024-08-17T18:19:43.580723+0000","last_became_active":"2024-08-17T18:18:29.734362+0000","last_became_peered":"2024-08-17T18:18:29.734362+0000","last_unstale":"2024-08-17T18:19:43.580723+0000","last_undegraded":"2024-08-17T18:19:43.580723+0000","last_fullsized":"2024-08-17T18:19:43.580723+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:58:48.630762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":254,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.596323+0000","last_change":"2024-08-17T18:12:04.655551+0000","last_active":"2024-08-17T18:19:43.596323+0000","last_peered":"2024-08-17T18:19:43.596323+0000","last_clean":"2024-08-17T18:19:43.596323+0000","last_became_active":"2024-08-17T18:12:04.646075+0000","last_became_peered":"2024-08-17T18:12:04.646075+0000","last_unstale":"2024-08-17T18:19:43.596323+0000","last_undegraded":"2024-08-17T18:19:43.596323+0000","last_fullsized":"2024-08-17T18:19:43.596323+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:19:19.300135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1724,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.746318+0000","last_change":"2024-08-17T18:18:29.727683+0000","last_active":"2024-08-17T18:19:34.746318+0000","last_peered":"2024-08-17T18:19:34.746318+0000","last_clean":"2024-08-17T18:19:34.746318+0000","last_became_active":"2024-08-17T18:18:29.727378+0000","last_became_peered":"2024-08-17T18:18:29.727378+0000","last_unstale":"2024-08-17T18:19:34.746318+0000","last_undegraded":"2024-08-17T18:19:34.746318+0000","last_fullsized":"2024-08-17T18:19:34.746318+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:42.989331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":211,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.746239+0000","last_change":"2024-08-17T18:18:29.882947+0000","last_active":"2024-08-17T18:19:34.746239+0000","last_peered":"2024-08-17T18:19:34.746239+0000","last_clean":"2024-08-17T18:19:34.746239+0000","last_became_active":"2024-08-17T18:18:29.882533+0000","last_became_peered":"2024-08-17T18:18:29.882533+0000","last_unstale":"2024-08-17T18:19:34.746239+0000","last_undegraded":"2024-08-17T18:19:34.746239+0000","last_fullsized":"2024-08-17T18:19:34.746239+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:34:25.726288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":49,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.579846+0000","last_change":"2024-08-17T18:18:29.734046+0000","last_active":"2024-08-17T18:19:43.579846+0000","last_peered":"2024-08-17T18:19:43.579846+0000","last_clean":"2024-08-17T18:19:43.579846+0000","last_became_active":"2024-08-17T18:18:29.733663+0000","last_became_peered":"2024-08-17T18:18:29.733663+0000","last_unstale":"2024-08-17T18:19:43.579846+0000","last_undegraded":"2024-08-17T18:19:43.579846+0000","last_fullsized":"2024-08-17T18:19:43.579846+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:08:09.146146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1719,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.768501+0000","last_change":"2024-08-17T18:16:00.324143+0000","last_active":"2024-08-17T18:19:34.768501+0000","last_peered":"2024-08-17T18:19:34.768501+0000","last_clean":"2024-08-17T18:19:34.768501+0000","last_became_active":"2024-08-17T18:15:57.106000+0000","last_became_peered":"2024-08-17T18:15:57.106000+0000","last_unstale":"2024-08-17T18:19:34.768501+0000","last_undegraded":"2024-08-17T18:19:34.768501+0000","last_fullsized":"2024-08-17T18:19:34.768501+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1641,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:43.117961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":48,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.580487+0000","last_change":"2024-08-17T18:18:29.733983+0000","last_active":"2024-08-17T18:19:43.580487+0000","last_peered":"2024-08-17T18:19:43.580487+0000","last_clean":"2024-08-17T18:19:43.580487+0000","last_became_active":"2024-08-17T18:18:29.733635+0000","last_became_peered":"2024-08-17T18:18:29.733635+0000","last_unstale":"2024-08-17T18:19:43.580487+0000","last_undegraded":"2024-08-17T18:19:43.580487+0000","last_fullsized":"2024-08-17T18:19:43.580487+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:47:15.631401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":48,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.579373+0000","last_change":"2024-08-17T18:18:29.888141+0000","last_active":"2024-08-17T18:19:43.579373+0000","last_peered":"2024-08-17T18:19:43.579373+0000","last_clean":"2024-08-17T18:19:43.579373+0000","last_became_active":"2024-08-17T18:18:29.887829+0000","last_became_peered":"2024-08-17T18:18:29.887829+0000","last_unstale":"2024-08-17T18:19:43.579373+0000","last_undegraded":"2024-08-17T18:19:43.579373+0000","last_fullsized":"2024-08-17T18:19:43.579373+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:23:13.375118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":48,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.581666+0000","last_change":"2024-08-17T18:18:29.896201+0000","last_active":"2024-08-17T18:19:43.581666+0000","last_peered":"2024-08-17T18:19:43.581666+0000","last_clean":"2024-08-17T18:19:43.581666+0000","last_became_active":"2024-08-17T18:18:29.895687+0000","last_became_peered":"2024-08-17T18:18:29.895687+0000","last_unstale":"2024-08-17T18:19:43.581666+0000","last_undegraded":"2024-08-17T18:19:43.581666+0000","last_fullsized":"2024-08-17T18:19:43.581666+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:24:52.525561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":48,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.579214+0000","last_change":"2024-08-17T18:18:29.888209+0000","last_active":"2024-08-17T18:19:43.579214+0000","last_peered":"2024-08-17T18:19:43.579214+0000","last_clean":"2024-08-17T18:19:43.579214+0000","last_became_active":"2024-08-17T18:18:29.887651+0000","last_became_peered":"2024-08-17T18:18:29.887651+0000","last_unstale":"2024-08-17T18:19:43.579214+0000","last_undegraded":"2024-08-17T18:19:43.579214+0000","last_fullsized":"2024-08-17T18:19:43.579214+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:55:59.802775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1730,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.736995+0000","last_change":"2024-08-17T18:18:29.887228+0000","last_active":"2024-08-17T18:19:34.736995+0000","last_peered":"2024-08-17T18:19:34.736995+0000","last_clean":"2024-08-17T18:19:34.736995+0000","last_became_active":"2024-08-17T18:18:29.884394+0000","last_became_peered":"2024-08-17T18:18:29.884394+0000","last_unstale":"2024-08-17T18:19:34.736995+0000","last_undegraded":"2024-08-17T18:19:34.736995+0000","last_fullsized":"2024-08-17T18:19:34.736995+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:41:42.256082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":48,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.579269+0000","last_change":"2024-08-17T18:18:29.727770+0000","last_active":"2024-08-17T18:19:43.579269+0000","last_peered":"2024-08-17T18:19:43.579269+0000","last_clean":"2024-08-17T18:19:43.579269+0000","last_became_active":"2024-08-17T18:18:29.727324+0000","last_became_peered":"2024-08-17T18:18:29.727324+0000","last_unstale":"2024-08-17T18:19:43.579269+0000","last_undegraded":"2024-08-17T18:19:43.579269+0000","last_fullsized":"2024-08-17T18:19:43.579269+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:20:29.342237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":411,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.730666+0000","last_change":"2024-08-17T18:18:29.734707+0000","last_active":"2024-08-17T18:19:34.730666+0000","last_peered":"2024-08-17T18:19:34.730666+0000","last_clean":"2024-08-17T18:19:34.730666+0000","last_became_active":"2024-08-17T18:18:29.734420+0000","last_became_peered":"2024-08-17T18:18:29.734420+0000","last_unstale":"2024-08-17T18:19:34.730666+0000","last_undegraded":"2024-08-17T18:19:34.730666+0000","last_fullsized":"2024-08-17T18:19:34.730666+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:07:29.048061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":48,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.579281+0000","last_change":"2024-08-17T18:18:29.880752+0000","last_active":"2024-08-17T18:19:43.579281+0000","last_peered":"2024-08-17T18:19:43.579281+0000","last_clean":"2024-08-17T18:19:43.579281+0000","last_became_active":"2024-08-17T18:18:29.880508+0000","last_became_peered":"2024-08-17T18:18:29.880508+0000","last_unstale":"2024-08-17T18:19:43.579281+0000","last_undegraded":"2024-08-17T18:19:43.579281+0000","last_fullsized":"2024-08-17T18:19:43.579281+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:35:25.743073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":299,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.738873+0000","last_change":"2024-08-17T18:18:29.900929+0000","last_active":"2024-08-17T18:19:34.738873+0000","last_peered":"2024-08-17T18:19:34.738873+0000","last_clean":"2024-08-17T18:19:34.738873+0000","last_became_active":"2024-08-17T18:18:29.900417+0000","last_became_peered":"2024-08-17T18:18:29.900417+0000","last_unstale":"2024-08-17T18:19:34.738873+0000","last_undegraded":"2024-08-17T18:19:34.738873+0000","last_fullsized":"2024-08-17T18:19:34.738873+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:07:45.752696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":54,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.580434+0000","last_change":"2024-08-17T18:18:29.928751+0000","last_active":"2024-08-17T18:19:43.580434+0000","last_peered":"2024-08-17T18:19:43.580434+0000","last_clean":"2024-08-17T18:19:43.580434+0000","last_became_active":"2024-08-17T18:18:29.898102+0000","last_became_peered":"2024-08-17T18:18:29.898102+0000","last_unstale":"2024-08-17T18:19:43.580434+0000","last_undegraded":"2024-08-17T18:19:43.580434+0000","last_fullsized":"2024-08-17T18:19:43.580434+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:21.098266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2519,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.582346+0000","last_change":"2024-08-17T18:17:22.685988+0000","last_active":"2024-08-17T18:19:43.582346+0000","last_peered":"2024-08-17T18:19:43.582346+0000","last_clean":"2024-08-17T18:19:43.582346+0000","last_became_active":"2024-08-17T18:17:22.685791+0000","last_became_peered":"2024-08-17T18:17:22.685791+0000","last_unstale":"2024-08-17T18:19:43.582346+0000","last_undegraded":"2024-08-17T18:19:43.582346+0000","last_fullsized":"2024-08-17T18:19:43.582346+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:10:28.276954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1713,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.748907+0000","last_change":"2024-08-17T18:15:53.687128+0000","last_active":"2024-08-17T18:19:34.748907+0000","last_peered":"2024-08-17T18:19:34.748907+0000","last_clean":"2024-08-17T18:19:34.748907+0000","last_became_active":"2024-08-17T18:15:53.246153+0000","last_became_peered":"2024-08-17T18:15:53.246153+0000","last_unstale":"2024-08-17T18:19:34.748907+0000","last_undegraded":"2024-08-17T18:19:34.748907+0000","last_fullsized":"2024-08-17T18:19:34.748907+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:45:14.801100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1675,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.746390+0000","last_change":"2024-08-17T18:12:04.639269+0000","last_active":"2024-08-17T18:19:34.746390+0000","last_peered":"2024-08-17T18:19:34.746390+0000","last_clean":"2024-08-17T18:19:34.746390+0000","last_became_active":"2024-08-17T18:12:04.638752+0000","last_became_peered":"2024-08-17T18:12:04.638752+0000","last_unstale":"2024-08-17T18:19:34.746390+0000","last_undegraded":"2024-08-17T18:19:34.746390+0000","last_fullsized":"2024-08-17T18:19:34.746390+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:50:00.462972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"347'149","reported_seq":912,"reported_epoch":1684,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:19:34.732687+0000","last_change":"2024-08-17T18:18:30.750089+0000","last_active":"2024-08-17T18:19:34.732687+0000","last_peered":"2024-08-17T18:19:34.732687+0000","last_clean":"2024-08-17T18:15:50.198296+0000","last_became_active":"2024-08-17T18:18:30.749746+0000","last_became_peered":"2024-08-17T18:18:30.749746+0000","last_unstale":"2024-08-17T18:19:34.732687+0000","last_undegraded":"2024-08-17T18:18:30.723633+0000","last_fullsized":"2024-08-17T18:18:30.723013+0000","mapping_epoch":1667,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1612,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:57.271286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:57.271286+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:22.670286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[5,7],"avail_no_missing":["5","4","6"],"object_location_counts":[{"shards":"4,5,6","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1691,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.748216+0000","last_change":"2024-08-17T18:04:43.766724+0000","last_active":"2024-08-17T18:19:34.748216+0000","last_peered":"2024-08-17T18:19:34.748216+0000","last_clean":"2024-08-17T18:19:34.748216+0000","last_became_active":"2024-08-17T18:04:43.766428+0000","last_became_peered":"2024-08-17T18:04:43.766428+0000","last_unstale":"2024-08-17T18:19:34.748216+0000","last_undegraded":"2024-08-17T18:19:34.748216+0000","last_fullsized":"2024-08-17T18:19:34.748216+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:09:49.102940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1721,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.737070+0000","last_change":"2024-08-17T18:15:53.366349+0000","last_active":"2024-08-17T18:19:34.737070+0000","last_peered":"2024-08-17T18:19:34.737070+0000","last_clean":"2024-08-17T18:19:34.737070+0000","last_became_active":"2024-08-17T18:15:53.365863+0000","last_became_peered":"2024-08-17T18:15:53.365863+0000","last_unstale":"2024-08-17T18:19:34.737070+0000","last_undegraded":"2024-08-17T18:19:34.737070+0000","last_fullsized":"2024-08-17T18:19:34.737070+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:52.700374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":114,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.749298+0000","last_change":"2024-08-17T18:17:22.720026+0000","last_active":"2024-08-17T18:19:34.749298+0000","last_peered":"2024-08-17T18:19:34.749298+0000","last_clean":"2024-08-17T18:19:34.749298+0000","last_became_active":"2024-08-17T18:17:22.662636+0000","last_became_peered":"2024-08-17T18:17:22.662636+0000","last_unstale":"2024-08-17T18:19:34.749298+0000","last_undegraded":"2024-08-17T18:19:34.749298+0000","last_fullsized":"2024-08-17T18:19:34.749298+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:37:15.928482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"347'149","reported_seq":978,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.579579+0000","last_change":"2024-08-17T18:17:25.672410+0000","last_active":"2024-08-17T18:19:43.579579+0000","last_peered":"2024-08-17T18:19:43.579579+0000","last_clean":"2024-08-17T18:19:43.579579+0000","last_became_active":"2024-08-17T18:17:25.671993+0000","last_became_peered":"2024-08-17T18:17:25.671993+0000","last_unstale":"2024-08-17T18:19:43.579579+0000","last_undegraded":"2024-08-17T18:19:43.579579+0000","last_fullsized":"2024-08-17T18:19:43.579579+0000","mapping_epoch":1655,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1656,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:53.309517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:53.309517+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:21:08.929478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":48,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.579353+0000","last_change":"2024-08-17T18:18:29.888368+0000","last_active":"2024-08-17T18:19:43.579353+0000","last_peered":"2024-08-17T18:19:43.579353+0000","last_clean":"2024-08-17T18:19:43.579353+0000","last_became_active":"2024-08-17T18:18:29.887731+0000","last_became_peered":"2024-08-17T18:18:29.887731+0000","last_unstale":"2024-08-17T18:19:43.579353+0000","last_undegraded":"2024-08-17T18:19:43.579353+0000","last_fullsized":"2024-08-17T18:19:43.579353+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:47:07.960969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":773,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.737311+0000","last_change":"2024-08-17T18:18:29.728878+0000","last_active":"2024-08-17T18:19:34.737311+0000","last_peered":"2024-08-17T18:19:34.737311+0000","last_clean":"2024-08-17T18:19:34.737311+0000","last_became_active":"2024-08-17T18:18:29.728295+0000","last_became_peered":"2024-08-17T18:18:29.728295+0000","last_unstale":"2024-08-17T18:19:34.737311+0000","last_undegraded":"2024-08-17T18:19:34.737311+0000","last_fullsized":"2024-08-17T18:19:34.737311+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:37:24.603653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":53,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.581456+0000","last_change":"2024-08-17T18:18:29.904137+0000","last_active":"2024-08-17T18:19:43.581456+0000","last_peered":"2024-08-17T18:19:43.581456+0000","last_clean":"2024-08-17T18:19:43.581456+0000","last_became_active":"2024-08-17T18:18:29.894843+0000","last_became_peered":"2024-08-17T18:18:29.894843+0000","last_unstale":"2024-08-17T18:19:43.581456+0000","last_undegraded":"2024-08-17T18:19:43.581456+0000","last_fullsized":"2024-08-17T18:19:43.581456+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:23:04.104922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"347'149","reported_seq":1011,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.737268+0000","last_change":"2024-08-17T18:18:33.761955+0000","last_active":"2024-08-17T18:19:34.737268+0000","last_peered":"2024-08-17T18:19:34.737268+0000","last_clean":"2024-08-17T18:19:34.737268+0000","last_became_active":"2024-08-17T18:18:33.761579+0000","last_became_peered":"2024-08-17T18:18:33.761579+0000","last_unstale":"2024-08-17T18:19:34.737268+0000","last_undegraded":"2024-08-17T18:19:34.737268+0000","last_fullsized":"2024-08-17T18:19:34.737268+0000","mapping_epoch":1670,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1671,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:52.474544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:52.474544+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:35:23.712308+0000","scrub_duration":0.001886326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":301,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.729906+0000","last_change":"2024-08-17T18:17:22.690072+0000","last_active":"2024-08-17T18:19:34.729906+0000","last_peered":"2024-08-17T18:19:34.729906+0000","last_clean":"2024-08-17T18:19:34.729906+0000","last_became_active":"2024-08-17T18:17:22.688530+0000","last_became_peered":"2024-08-17T18:17:22.688530+0000","last_unstale":"2024-08-17T18:19:34.729906+0000","last_undegraded":"2024-08-17T18:19:34.729906+0000","last_fullsized":"2024-08-17T18:19:34.729906+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:40:26.170186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":58,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.747807+0000","last_change":"2024-08-17T18:15:53.248168+0000","last_active":"2024-08-17T18:19:34.747807+0000","last_peered":"2024-08-17T18:19:34.747807+0000","last_clean":"2024-08-17T18:19:34.747807+0000","last_became_active":"2024-08-17T18:15:53.247888+0000","last_became_peered":"2024-08-17T18:15:53.247888+0000","last_unstale":"2024-08-17T18:19:34.747807+0000","last_undegraded":"2024-08-17T18:19:34.747807+0000","last_fullsized":"2024-08-17T18:19:34.747807+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:30:42.725233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2547,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.583765+0000","last_change":"2024-08-17T18:15:10.692541+0000","last_active":"2024-08-17T18:19:43.583765+0000","last_peered":"2024-08-17T18:19:43.583765+0000","last_clean":"2024-08-17T18:19:43.583765+0000","last_became_active":"2024-08-17T18:15:10.691876+0000","last_became_peered":"2024-08-17T18:15:10.691876+0000","last_unstale":"2024-08-17T18:19:43.583765+0000","last_undegraded":"2024-08-17T18:19:43.583765+0000","last_fullsized":"2024-08-17T18:19:43.583765+0000","mapping_epoch":1626,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:39:22.253615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"347'149","reported_seq":994,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.736919+0000","last_change":"2024-08-17T18:18:33.761996+0000","last_active":"2024-08-17T18:19:34.736919+0000","last_peered":"2024-08-17T18:19:34.736919+0000","last_clean":"2024-08-17T18:19:34.736919+0000","last_became_active":"2024-08-17T18:18:33.761606+0000","last_became_peered":"2024-08-17T18:18:33.761606+0000","last_unstale":"2024-08-17T18:19:34.736919+0000","last_undegraded":"2024-08-17T18:19:34.736919+0000","last_fullsized":"2024-08-17T18:19:34.736919+0000","mapping_epoch":1670,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1671,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:49.461698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:49.461698+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:29:01.698411+0000","scrub_duration":0.0013792520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":251,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.739264+0000","last_change":"2024-08-17T18:17:22.687498+0000","last_active":"2024-08-17T18:19:34.739264+0000","last_peered":"2024-08-17T18:19:34.739264+0000","last_clean":"2024-08-17T18:19:34.739264+0000","last_became_active":"2024-08-17T18:17:22.687176+0000","last_became_peered":"2024-08-17T18:17:22.687176+0000","last_unstale":"2024-08-17T18:19:34.739264+0000","last_undegraded":"2024-08-17T18:19:34.739264+0000","last_fullsized":"2024-08-17T18:19:34.739264+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:32:42.738739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2556,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.583655+0000","last_change":"2024-08-17T18:15:53.230320+0000","last_active":"2024-08-17T18:19:43.583655+0000","last_peered":"2024-08-17T18:19:43.583655+0000","last_clean":"2024-08-17T18:19:43.583655+0000","last_became_active":"2024-08-17T18:15:53.230016+0000","last_became_peered":"2024-08-17T18:15:53.230016+0000","last_unstale":"2024-08-17T18:19:43.583655+0000","last_undegraded":"2024-08-17T18:19:43.583655+0000","last_fullsized":"2024-08-17T18:19:43.583655+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:15:28.925103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":256,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.737092+0000","last_change":"2024-08-17T18:18:29.735362+0000","last_active":"2024-08-17T18:19:34.737092+0000","last_peered":"2024-08-17T18:19:34.737092+0000","last_clean":"2024-08-17T18:19:34.737092+0000","last_became_active":"2024-08-17T18:18:29.734752+0000","last_became_peered":"2024-08-17T18:18:29.734752+0000","last_unstale":"2024-08-17T18:19:34.737092+0000","last_undegraded":"2024-08-17T18:19:34.737092+0000","last_fullsized":"2024-08-17T18:19:34.737092+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:25.709698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"347'149","reported_seq":979,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:42.195846+0000","last_change":"2024-08-17T18:15:13.681264+0000","last_active":"2024-08-17T18:19:42.195846+0000","last_peered":"2024-08-17T18:19:42.195846+0000","last_clean":"2024-08-17T18:19:42.195846+0000","last_became_active":"2024-08-17T18:15:13.680918+0000","last_became_peered":"2024-08-17T18:15:13.680918+0000","last_unstale":"2024-08-17T18:19:42.195846+0000","last_undegraded":"2024-08-17T18:19:42.195846+0000","last_fullsized":"2024-08-17T18:19:42.195846+0000","mapping_epoch":1629,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1630,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:00.316884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:00.316884+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:37:32.924833+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1695,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.583624+0000","last_change":"2024-08-17T18:18:29.889561+0000","last_active":"2024-08-17T18:19:43.583624+0000","last_peered":"2024-08-17T18:19:43.583624+0000","last_clean":"2024-08-17T18:19:43.583624+0000","last_became_active":"2024-08-17T18:18:29.889192+0000","last_became_peered":"2024-08-17T18:18:29.889192+0000","last_unstale":"2024-08-17T18:19:43.583624+0000","last_undegraded":"2024-08-17T18:19:43.583624+0000","last_fullsized":"2024-08-17T18:19:43.583624+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:06:06.579862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1698,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.749621+0000","last_change":"2024-08-17T18:15:10.695565+0000","last_active":"2024-08-17T18:19:34.749621+0000","last_peered":"2024-08-17T18:19:34.749621+0000","last_clean":"2024-08-17T18:19:34.749621+0000","last_became_active":"2024-08-17T18:15:10.695225+0000","last_became_peered":"2024-08-17T18:15:10.695225+0000","last_unstale":"2024-08-17T18:19:34.749621+0000","last_undegraded":"2024-08-17T18:19:34.749621+0000","last_fullsized":"2024-08-17T18:19:34.749621+0000","mapping_epoch":1626,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:28:55.675702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":291,"reported_epoch":1684,"state":"active+recovery_wait+degraded","last_fresh":"2024-08-17T18:19:34.739425+0000","last_change":"2024-08-17T18:18:29.800821+0000","last_active":"2024-08-17T18:19:34.739425+0000","last_peered":"2024-08-17T18:19:34.739425+0000","last_clean":"2024-08-17T18:18:25.612782+0000","last_became_active":"2024-08-17T18:18:29.800502+0000","last_became_peered":"2024-08-17T18:18:29.800502+0000","last_unstale":"2024-08-17T18:19:34.739425+0000","last_undegraded":"2024-08-17T18:18:29.738218+0000","last_fullsized":"2024-08-17T18:19:34.739425+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:08:12.507444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":95,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":["5","2","6","7"],"object_location_counts":[{"shards":"2,5,6,7","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"347'149","reported_seq":838,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.768239+0000","last_change":"2024-08-17T18:17:22.681616+0000","last_active":"2024-08-17T18:19:34.768239+0000","last_peered":"2024-08-17T18:19:34.768239+0000","last_clean":"2024-08-17T18:19:34.768239+0000","last_became_active":"2024-08-17T18:17:22.681184+0000","last_became_peered":"2024-08-17T18:17:22.681184+0000","last_unstale":"2024-08-17T18:19:34.768239+0000","last_undegraded":"2024-08-17T18:19:34.768239+0000","last_fullsized":"2024-08-17T18:19:34.768239+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:02.363531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:02.363531+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:50:39.567707+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2447,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.730733+0000","last_change":"2024-08-17T18:18:29.898545+0000","last_active":"2024-08-17T18:19:34.730733+0000","last_peered":"2024-08-17T18:19:34.730733+0000","last_clean":"2024-08-17T18:19:34.730733+0000","last_became_active":"2024-08-17T18:18:29.896509+0000","last_became_peered":"2024-08-17T18:18:29.896509+0000","last_unstale":"2024-08-17T18:19:34.730733+0000","last_undegraded":"2024-08-17T18:19:34.730733+0000","last_fullsized":"2024-08-17T18:19:34.730733+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:37:09.107372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2368,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.582241+0000","last_change":"2024-08-17T18:12:04.648190+0000","last_active":"2024-08-17T18:19:43.582241+0000","last_peered":"2024-08-17T18:19:43.582241+0000","last_clean":"2024-08-17T18:19:43.582241+0000","last_became_active":"2024-08-17T18:12:04.647877+0000","last_became_peered":"2024-08-17T18:12:04.647877+0000","last_unstale":"2024-08-17T18:19:43.582241+0000","last_undegraded":"2024-08-17T18:19:43.582241+0000","last_fullsized":"2024-08-17T18:19:43.582241+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:18.129337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2670,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.581861+0000","last_change":"2024-08-17T18:18:29.742742+0000","last_active":"2024-08-17T18:19:43.581861+0000","last_peered":"2024-08-17T18:19:43.581861+0000","last_clean":"2024-08-17T18:19:43.581861+0000","last_became_active":"2024-08-17T18:18:29.742393+0000","last_became_peered":"2024-08-17T18:18:29.742393+0000","last_unstale":"2024-08-17T18:19:43.581861+0000","last_undegraded":"2024-08-17T18:19:43.581861+0000","last_fullsized":"2024-08-17T18:19:43.581861+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:57:32.858608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"347'149","reported_seq":705,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.745990+0000","last_change":"2024-08-17T18:18:32.735002+0000","last_active":"2024-08-17T18:19:34.745990+0000","last_peered":"2024-08-17T18:19:34.745990+0000","last_clean":"2024-08-17T18:19:34.745990+0000","last_became_active":"2024-08-17T18:18:32.734291+0000","last_became_peered":"2024-08-17T18:18:32.734291+0000","last_unstale":"2024-08-17T18:19:34.745990+0000","last_undegraded":"2024-08-17T18:19:34.745990+0000","last_fullsized":"2024-08-17T18:19:34.745990+0000","mapping_epoch":1669,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1670,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:12:53.586672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:12:53.586672+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":214680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":363,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.582885+0000","last_change":"2024-08-17T18:15:53.370263+0000","last_active":"2024-08-17T18:19:43.582885+0000","last_peered":"2024-08-17T18:19:43.582885+0000","last_clean":"2024-08-17T18:19:43.582885+0000","last_became_active":"2024-08-17T18:15:53.369971+0000","last_became_peered":"2024-08-17T18:15:53.369971+0000","last_unstale":"2024-08-17T18:19:43.582885+0000","last_undegraded":"2024-08-17T18:19:43.582885+0000","last_fullsized":"2024-08-17T18:19:43.582885+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:51:08.633886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":862,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.747954+0000","last_change":"2024-08-17T18:18:29.727602+0000","last_active":"2024-08-17T18:19:34.747954+0000","last_peered":"2024-08-17T18:19:34.747954+0000","last_clean":"2024-08-17T18:19:34.747954+0000","last_became_active":"2024-08-17T18:18:29.727190+0000","last_became_peered":"2024-08-17T18:18:29.727190+0000","last_unstale":"2024-08-17T18:19:34.747954+0000","last_undegraded":"2024-08-17T18:19:34.747954+0000","last_fullsized":"2024-08-17T18:19:34.747954+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:13:09.502199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":48,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.579330+0000","last_change":"2024-08-17T18:18:29.888513+0000","last_active":"2024-08-17T18:19:43.579330+0000","last_peered":"2024-08-17T18:19:43.579330+0000","last_clean":"2024-08-17T18:19:43.579330+0000","last_became_active":"2024-08-17T18:18:29.888065+0000","last_became_peered":"2024-08-17T18:18:29.888065+0000","last_unstale":"2024-08-17T18:19:43.579330+0000","last_undegraded":"2024-08-17T18:19:43.579330+0000","last_fullsized":"2024-08-17T18:19:43.579330+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:18.952476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1533,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.748579+0000","last_change":"2024-08-17T18:18:29.895726+0000","last_active":"2024-08-17T18:19:34.748579+0000","last_peered":"2024-08-17T18:19:34.748579+0000","last_clean":"2024-08-17T18:19:34.748579+0000","last_became_active":"2024-08-17T18:18:29.895383+0000","last_became_peered":"2024-08-17T18:18:29.895383+0000","last_unstale":"2024-08-17T18:19:34.748579+0000","last_undegraded":"2024-08-17T18:19:34.748579+0000","last_fullsized":"2024-08-17T18:19:34.748579+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:18:30.787712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":48,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.580622+0000","last_change":"2024-08-17T18:18:29.800143+0000","last_active":"2024-08-17T18:19:43.580622+0000","last_peered":"2024-08-17T18:19:43.580622+0000","last_clean":"2024-08-17T18:19:43.580622+0000","last_became_active":"2024-08-17T18:18:29.799896+0000","last_became_peered":"2024-08-17T18:18:29.799896+0000","last_unstale":"2024-08-17T18:19:43.580622+0000","last_undegraded":"2024-08-17T18:19:43.580622+0000","last_fullsized":"2024-08-17T18:19:43.580622+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:27:18.144391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":271,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.730005+0000","last_change":"2024-08-17T18:19:26.374702+0000","last_active":"2024-08-17T18:19:34.730005+0000","last_peered":"2024-08-17T18:19:34.730005+0000","last_clean":"2024-08-17T18:19:34.730005+0000","last_became_active":"2024-08-17T18:15:53.304106+0000","last_became_peered":"2024-08-17T18:15:53.304106+0000","last_unstale":"2024-08-17T18:19:34.730005+0000","last_undegraded":"2024-08-17T18:19:34.730005+0000","last_fullsized":"2024-08-17T18:19:34.730005+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:21.409368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":865,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.743851+0000","last_change":"2024-08-17T18:17:16.835493+0000","last_active":"2024-08-17T18:19:34.743851+0000","last_peered":"2024-08-17T18:19:34.743851+0000","last_clean":"2024-08-17T18:19:34.743851+0000","last_became_active":"2024-08-17T18:15:55.698769+0000","last_became_peered":"2024-08-17T18:15:55.698769+0000","last_unstale":"2024-08-17T18:19:34.743851+0000","last_undegraded":"2024-08-17T18:19:34.743851+0000","last_fullsized":"2024-08-17T18:19:34.743851+0000","mapping_epoch":1640,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1641,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:54.303708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:54.303708+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":74,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"347'149","reported_seq":962,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.739044+0000","last_change":"2024-08-17T18:09:15.592722+0000","last_active":"2024-08-17T18:19:34.739044+0000","last_peered":"2024-08-17T18:19:34.739044+0000","last_clean":"2024-08-17T18:19:34.739044+0000","last_became_active":"2024-08-17T18:09:15.592242+0000","last_became_peered":"2024-08-17T18:09:15.592242+0000","last_unstale":"2024-08-17T18:19:34.739044+0000","last_undegraded":"2024-08-17T18:19:34.739044+0000","last_fullsized":"2024-08-17T18:19:34.739044+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:55.255983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:55.255983+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:02:53.596275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":48,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.581377+0000","last_change":"2024-08-17T18:18:29.895120+0000","last_active":"2024-08-17T18:19:43.581377+0000","last_peered":"2024-08-17T18:19:43.581377+0000","last_clean":"2024-08-17T18:19:43.581377+0000","last_became_active":"2024-08-17T18:18:29.894811+0000","last_became_peered":"2024-08-17T18:18:29.894811+0000","last_unstale":"2024-08-17T18:19:43.581377+0000","last_undegraded":"2024-08-17T18:19:43.581377+0000","last_fullsized":"2024-08-17T18:19:43.581377+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:44:28.402299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":48,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.579170+0000","last_change":"2024-08-17T18:18:29.900296+0000","last_active":"2024-08-17T18:19:43.579170+0000","last_peered":"2024-08-17T18:19:43.579170+0000","last_clean":"2024-08-17T18:19:43.579170+0000","last_became_active":"2024-08-17T18:18:29.899848+0000","last_became_peered":"2024-08-17T18:18:29.899848+0000","last_unstale":"2024-08-17T18:19:43.579170+0000","last_undegraded":"2024-08-17T18:19:43.579170+0000","last_fullsized":"2024-08-17T18:19:43.579170+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:09:15.545423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1721,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.749379+0000","last_change":"2024-08-17T18:18:29.882860+0000","last_active":"2024-08-17T18:19:34.749379+0000","last_peered":"2024-08-17T18:19:34.749379+0000","last_clean":"2024-08-17T18:19:34.749379+0000","last_became_active":"2024-08-17T18:18:29.882539+0000","last_became_peered":"2024-08-17T18:18:29.882539+0000","last_unstale":"2024-08-17T18:19:34.749379+0000","last_undegraded":"2024-08-17T18:19:34.749379+0000","last_fullsized":"2024-08-17T18:19:34.749379+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:12:52.056802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1712,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:39.924089+0000","last_change":"2024-08-17T18:15:53.244851+0000","last_active":"2024-08-17T18:19:39.924089+0000","last_peered":"2024-08-17T18:19:39.924089+0000","last_clean":"2024-08-17T18:19:39.924089+0000","last_became_active":"2024-08-17T18:15:53.240074+0000","last_became_peered":"2024-08-17T18:15:53.240074+0000","last_unstale":"2024-08-17T18:19:39.924089+0000","last_undegraded":"2024-08-17T18:19:39.924089+0000","last_fullsized":"2024-08-17T18:19:39.924089+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:09:00.168413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"347'149","reported_seq":871,"reported_epoch":1685,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:19:43.581985+0000","last_change":"2024-08-17T18:18:30.729274+0000","last_active":"2024-08-17T18:19:43.581985+0000","last_peered":"2024-08-17T18:19:43.581985+0000","last_clean":"2024-08-17T18:09:10.546471+0000","last_became_active":"2024-08-17T18:18:30.728913+0000","last_became_peered":"2024-08-17T18:18:30.728913+0000","last_unstale":"2024-08-17T18:19:43.581985+0000","last_undegraded":"2024-08-17T18:18:30.719341+0000","last_fullsized":"2024-08-17T18:18:30.718846+0000","mapping_epoch":1667,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1612,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:59.307356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:59.307356+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,4],"avail_no_missing":["2","5","6"],"object_location_counts":[{"shards":"2,5,6","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1706,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.748394+0000","last_change":"2024-08-17T18:12:04.648228+0000","last_active":"2024-08-17T18:19:34.748394+0000","last_peered":"2024-08-17T18:19:34.748394+0000","last_clean":"2024-08-17T18:19:34.748394+0000","last_became_active":"2024-08-17T18:12:04.647906+0000","last_became_peered":"2024-08-17T18:12:04.647906+0000","last_unstale":"2024-08-17T18:19:34.748394+0000","last_undegraded":"2024-08-17T18:19:34.748394+0000","last_fullsized":"2024-08-17T18:19:34.748394+0000","mapping_epoch":1553,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:51:30.081540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1694,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.574784+0000","last_change":"2024-08-17T18:18:29.728681+0000","last_active":"2024-08-17T18:19:43.574784+0000","last_peered":"2024-08-17T18:19:43.574784+0000","last_clean":"2024-08-17T18:19:43.574784+0000","last_became_active":"2024-08-17T18:18:29.728215+0000","last_became_peered":"2024-08-17T18:18:29.728215+0000","last_unstale":"2024-08-17T18:19:43.574784+0000","last_undegraded":"2024-08-17T18:19:43.574784+0000","last_fullsized":"2024-08-17T18:19:43.574784+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:48:41.152839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"347'149","reported_seq":938,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.747366+0000","last_change":"2024-08-17T18:18:33.757347+0000","last_active":"2024-08-17T18:19:34.747366+0000","last_peered":"2024-08-17T18:19:34.747366+0000","last_clean":"2024-08-17T18:19:34.747366+0000","last_became_active":"2024-08-17T18:18:33.756960+0000","last_became_peered":"2024-08-17T18:18:33.756960+0000","last_unstale":"2024-08-17T18:19:34.747366+0000","last_undegraded":"2024-08-17T18:19:34.747366+0000","last_fullsized":"2024-08-17T18:19:34.747366+0000","mapping_epoch":1670,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1671,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:52.271487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:52.271487+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:37:36.389244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1694,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.748818+0000","last_change":"2024-08-17T18:18:29.883479+0000","last_active":"2024-08-17T18:19:34.748818+0000","last_peered":"2024-08-17T18:19:34.748818+0000","last_clean":"2024-08-17T18:19:34.748818+0000","last_became_active":"2024-08-17T18:18:29.883185+0000","last_became_peered":"2024-08-17T18:18:29.883185+0000","last_unstale":"2024-08-17T18:19:34.748818+0000","last_undegraded":"2024-08-17T18:19:34.748818+0000","last_fullsized":"2024-08-17T18:19:34.748818+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:11:04.148316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":52,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.580177+0000","last_change":"2024-08-17T18:18:29.891624+0000","last_active":"2024-08-17T18:19:43.580177+0000","last_peered":"2024-08-17T18:19:43.580177+0000","last_clean":"2024-08-17T18:19:43.580177+0000","last_became_active":"2024-08-17T18:18:29.880707+0000","last_became_peered":"2024-08-17T18:18:29.880707+0000","last_unstale":"2024-08-17T18:19:43.580177+0000","last_undegraded":"2024-08-17T18:19:43.580177+0000","last_fullsized":"2024-08-17T18:19:43.580177+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:04.490121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":740,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:39.934109+0000","last_change":"2024-08-17T18:18:29.727626+0000","last_active":"2024-08-17T18:19:39.934109+0000","last_peered":"2024-08-17T18:19:39.934109+0000","last_clean":"2024-08-17T18:19:39.934109+0000","last_became_active":"2024-08-17T18:18:29.727331+0000","last_became_peered":"2024-08-17T18:18:29.727331+0000","last_unstale":"2024-08-17T18:19:39.934109+0000","last_undegraded":"2024-08-17T18:19:39.934109+0000","last_fullsized":"2024-08-17T18:19:39.934109+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:46:16.169400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":264,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.731220+0000","last_change":"2024-08-17T18:18:29.735189+0000","last_active":"2024-08-17T18:19:34.731220+0000","last_peered":"2024-08-17T18:19:34.731220+0000","last_clean":"2024-08-17T18:19:34.731220+0000","last_became_active":"2024-08-17T18:18:29.734681+0000","last_became_peered":"2024-08-17T18:18:29.734681+0000","last_unstale":"2024-08-17T18:19:34.731220+0000","last_undegraded":"2024-08-17T18:19:34.731220+0000","last_fullsized":"2024-08-17T18:19:34.731220+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:03:20.353696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1514,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.739138+0000","last_change":"2024-08-17T18:18:29.943292+0000","last_active":"2024-08-17T18:19:34.739138+0000","last_peered":"2024-08-17T18:19:34.739138+0000","last_clean":"2024-08-17T18:19:34.739138+0000","last_became_active":"2024-08-17T18:18:29.942947+0000","last_became_peered":"2024-08-17T18:18:29.942947+0000","last_unstale":"2024-08-17T18:19:34.739138+0000","last_undegraded":"2024-08-17T18:19:34.739138+0000","last_fullsized":"2024-08-17T18:19:34.739138+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:19:04.210562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":267,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.737247+0000","last_change":"2024-08-17T18:15:53.661488+0000","last_active":"2024-08-17T18:19:34.737247+0000","last_peered":"2024-08-17T18:19:34.737247+0000","last_clean":"2024-08-17T18:19:34.737247+0000","last_became_active":"2024-08-17T18:15:53.308611+0000","last_became_peered":"2024-08-17T18:15:53.308611+0000","last_unstale":"2024-08-17T18:19:34.737247+0000","last_undegraded":"2024-08-17T18:19:34.737247+0000","last_fullsized":"2024-08-17T18:19:34.737247+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:46:37.056632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1729,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:39.924038+0000","last_change":"2024-08-17T18:18:29.728301+0000","last_active":"2024-08-17T18:19:39.924038+0000","last_peered":"2024-08-17T18:19:39.924038+0000","last_clean":"2024-08-17T18:19:39.924038+0000","last_became_active":"2024-08-17T18:18:29.728019+0000","last_became_peered":"2024-08-17T18:18:29.728019+0000","last_unstale":"2024-08-17T18:19:39.924038+0000","last_undegraded":"2024-08-17T18:19:39.924038+0000","last_fullsized":"2024-08-17T18:19:39.924038+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:57:56.453165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":62,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.749278+0000","last_change":"2024-08-17T18:15:53.708176+0000","last_active":"2024-08-17T18:19:34.749278+0000","last_peered":"2024-08-17T18:19:34.749278+0000","last_clean":"2024-08-17T18:19:34.749278+0000","last_became_active":"2024-08-17T18:15:53.669563+0000","last_became_peered":"2024-08-17T18:15:53.669563+0000","last_unstale":"2024-08-17T18:19:34.749278+0000","last_undegraded":"2024-08-17T18:19:34.749278+0000","last_fullsized":"2024-08-17T18:19:34.749278+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:52:17.018687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":273,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.731559+0000","last_change":"2024-08-17T18:17:22.687310+0000","last_active":"2024-08-17T18:19:34.731559+0000","last_peered":"2024-08-17T18:19:34.731559+0000","last_clean":"2024-08-17T18:19:34.731559+0000","last_became_active":"2024-08-17T18:17:22.686966+0000","last_became_peered":"2024-08-17T18:17:22.686966+0000","last_unstale":"2024-08-17T18:19:34.731559+0000","last_undegraded":"2024-08-17T18:19:34.731559+0000","last_fullsized":"2024-08-17T18:19:34.731559+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:46:11.568465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":62,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.749950+0000","last_change":"2024-08-17T18:15:53.461774+0000","last_active":"2024-08-17T18:19:34.749950+0000","last_peered":"2024-08-17T18:19:34.749950+0000","last_clean":"2024-08-17T18:19:34.749950+0000","last_became_active":"2024-08-17T18:15:53.287744+0000","last_became_peered":"2024-08-17T18:15:53.287744+0000","last_unstale":"2024-08-17T18:19:34.749950+0000","last_undegraded":"2024-08-17T18:19:34.749950+0000","last_fullsized":"2024-08-17T18:19:34.749950+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:25.168664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1805,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.743766+0000","last_change":"2024-08-17T18:18:29.897922+0000","last_active":"2024-08-17T18:19:34.743766+0000","last_peered":"2024-08-17T18:19:34.743766+0000","last_clean":"2024-08-17T18:19:34.743766+0000","last_became_active":"2024-08-17T18:18:29.883879+0000","last_became_peered":"2024-08-17T18:18:29.883879+0000","last_unstale":"2024-08-17T18:19:34.743766+0000","last_undegraded":"2024-08-17T18:19:34.743766+0000","last_fullsized":"2024-08-17T18:19:34.743766+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:54:31.992298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":48,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.581557+0000","last_change":"2024-08-17T18:18:29.890043+0000","last_active":"2024-08-17T18:19:43.581557+0000","last_peered":"2024-08-17T18:19:43.581557+0000","last_clean":"2024-08-17T18:19:43.581557+0000","last_became_active":"2024-08-17T18:18:29.889687+0000","last_became_peered":"2024-08-17T18:18:29.889687+0000","last_unstale":"2024-08-17T18:19:43.581557+0000","last_undegraded":"2024-08-17T18:19:43.581557+0000","last_fullsized":"2024-08-17T18:19:43.581557+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:55:34.595507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":425,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:39.924060+0000","last_change":"2024-08-17T18:15:53.245247+0000","last_active":"2024-08-17T18:19:39.924060+0000","last_peered":"2024-08-17T18:19:39.924060+0000","last_clean":"2024-08-17T18:19:39.924060+0000","last_became_active":"2024-08-17T18:15:53.244627+0000","last_became_peered":"2024-08-17T18:15:53.244627+0000","last_unstale":"2024-08-17T18:19:39.924060+0000","last_undegraded":"2024-08-17T18:19:39.924060+0000","last_fullsized":"2024-08-17T18:19:39.924060+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:42:03.556013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":48,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.581774+0000","last_change":"2024-08-17T18:18:29.895804+0000","last_active":"2024-08-17T18:19:43.581774+0000","last_peered":"2024-08-17T18:19:43.581774+0000","last_clean":"2024-08-17T18:19:43.581774+0000","last_became_active":"2024-08-17T18:18:29.895352+0000","last_became_peered":"2024-08-17T18:18:29.895352+0000","last_unstale":"2024-08-17T18:19:43.581774+0000","last_undegraded":"2024-08-17T18:19:43.581774+0000","last_fullsized":"2024-08-17T18:19:43.581774+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:32:29.222569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":48,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.580799+0000","last_change":"2024-08-17T18:18:29.888110+0000","last_active":"2024-08-17T18:19:43.580799+0000","last_peered":"2024-08-17T18:19:43.580799+0000","last_clean":"2024-08-17T18:19:43.580799+0000","last_became_active":"2024-08-17T18:18:29.887787+0000","last_became_peered":"2024-08-17T18:18:29.887787+0000","last_unstale":"2024-08-17T18:19:43.580799+0000","last_undegraded":"2024-08-17T18:19:43.580799+0000","last_fullsized":"2024-08-17T18:19:43.580799+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:46:10.231616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":273,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.737166+0000","last_change":"2024-08-17T18:18:29.900128+0000","last_active":"2024-08-17T18:19:34.737166+0000","last_peered":"2024-08-17T18:19:34.737166+0000","last_clean":"2024-08-17T18:19:34.737166+0000","last_became_active":"2024-08-17T18:18:29.899279+0000","last_became_peered":"2024-08-17T18:18:29.899279+0000","last_unstale":"2024-08-17T18:19:34.737166+0000","last_undegraded":"2024-08-17T18:19:34.737166+0000","last_fullsized":"2024-08-17T18:19:34.737166+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:01:29.045498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":65,"reported_epoch":1684,"state":"active+clean","last_fresh":"2024-08-17T18:19:34.748053+0000","last_change":"2024-08-17T18:15:53.271113+0000","last_active":"2024-08-17T18:19:34.748053+0000","last_peered":"2024-08-17T18:19:34.748053+0000","last_clean":"2024-08-17T18:19:34.748053+0000","last_became_active":"2024-08-17T18:15:53.230983+0000","last_became_peered":"2024-08-17T18:15:53.230983+0000","last_unstale":"2024-08-17T18:19:34.748053+0000","last_undegraded":"2024-08-17T18:19:34.748053+0000","last_fullsized":"2024-08-17T18:19:34.748053+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:00:35.315798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":48,"reported_epoch":1685,"state":"active+clean","last_fresh":"2024-08-17T18:19:43.579756+0000","last_change":"2024-08-17T18:18:29.888785+0000","last_active":"2024-08-17T18:19:43.579756+0000","last_peered":"2024-08-17T18:19:43.579756+0000","last_clean":"2024-08-17T18:19:43.579756+0000","last_became_active":"2024-08-17T18:18:29.888475+0000","last_became_peered":"2024-08-17T18:18:29.888475+0000","last_unstale":"2024-08-17T18:19:43.579756+0000","last_undegraded":"2024-08-17T18:19:43.579756+0000","last_fullsized":"2024-08-17T18:19:43.579756+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:44:28.968751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":57,"num_pg":26,"stat_sum":{"num_bytes":1344208920,"num_objects":20512,"num_object_clones":0,"num_object_copies":61536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13522,"num_objects_misplaced":17403,"num_objects_unfound":0,"num_objects_dirty":20512,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":211328,"num_write_kb":7418817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46411,"num_bytes_recovered":2629042176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7512076288,"data_stored":7512064072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132874,"ondisk_log_size":132874,"up":78,"acting":75,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":697707,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":149,"num_write":39,"num_write_kb":439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":810709,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":31,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":95,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":95,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":12532,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":22134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234845,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":865544,"kb_used_data":243004,"kb_used_omap":1,"kb_used_meta":622526,"kb_avail":92879608,"statfs":{"total":95995035648,"available":95108718592,"internally_reserved":0,"allocated":248836096,"data_stored":244378585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":637467083},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496483,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2653892,"kb_used_data":1389436,"kb_used_omap":1,"kb_used_meta":1264382,"kb_avail":91091260,"statfs":{"total":95995035648,"available":93277450240,"internally_reserved":0,"allocated":1422782464,"data_stored":1418325957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1294727623},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565724,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1068904,"kb_used_data":21920,"kb_used_omap":72,"kb_used_meta":1046903,"kb_avail":92676248,"statfs":{"total":95995035648,"available":94900477952,"internally_reserved":0,"allocated":22446080,"data_stored":18012352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":1072029166},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":1399,"seq":6008659247326,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2193356,"kb_used_data":1453188,"kb_used_omap":131,"kb_used_meta":740028,"kb_avail":91551796,"statfs":{"total":95995035648,"available":93749039104,"internally_reserved":0,"allocated":1488064512,"data_stored":1483599901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134291,"internal_metadata":757789549},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":2864743187114,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448753,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3125608,"kb_used_data":2317664,"kb_used_omap":75,"kb_used_meta":807860,"kb_avail":90619544,"statfs":{"total":95995035648,"available":92794413056,"internally_reserved":0,"allocated":2373287936,"data_stored":2368854142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":827248831},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208758118,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2384184,"kb_used_data":1061872,"kb_used_omap":1,"kb_used_meta":1322174,"kb_avail":91360968,"statfs":{"total":95995035648,"available":93553631232,"internally_reserved":0,"allocated":1087356928,"data_stored":1082907951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1353906630},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019755,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2117504,"kb_used_data":968696,"kb_used_omap":1,"kb_used_meta":1148798,"kb_avail":91627648,"statfs":{"total":95995035648,"available":93826711552,"internally_reserved":0,"allocated":991944704,"data_stored":987491782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1176369606},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2358251520,"data_stored":2358247448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1468010496,"data_stored":1468006424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12255232,"data_stored":12255232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":974258176,"data_stored":974258176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1074528256,"data_stored":1074528256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1389563904,"data_stored":1389559832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":57,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":235208704,"data_stored":235208704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T18:19:54.540 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-08-17T18:19:54.573 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:54.613 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:54.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:54.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:54.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:54.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:54.769 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:54.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:54.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:54.897 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:54.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:54.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:54.999 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:55.028 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:55.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:55.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:55.130 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:55.162 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:55.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:55.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:55.264 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:55.289 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:55.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:55.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:55.391 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:55.415 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:55.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:55.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:55.517 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:55.543 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:55.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:55.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:55.645 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:55.681 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:55.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:55.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:55.783 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:55.808 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:55.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:55.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:55.910 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:55.936 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:55.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:55.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:55.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:55 smithi136 ceph-mon[109861]: pgmap v3258: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 641 op/s; 12505/52722 objects degraded (23.719%); 14914/52722 objects misplaced (28.288%); 14 MiB/s, 235 objects/s recovering 2024-08-17T18:19:55.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:55 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 12505/52722 objects degraded (23.719%), 13 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:55.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:55 smithi136 ceph-mon[115078]: pgmap v3258: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 641 op/s; 12505/52722 objects degraded (23.719%); 14914/52722 objects misplaced (28.288%); 14 MiB/s, 235 objects/s recovering 2024-08-17T18:19:55.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:55 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 12505/52722 objects degraded (23.719%), 13 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:56.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:55 smithi165 ceph-mon[105517]: pgmap v3258: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 641 op/s; 12505/52722 objects degraded (23.719%); 14914/52722 objects misplaced (28.288%); 14 MiB/s, 235 objects/s recovering 2024-08-17T18:19:56.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:55 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 12505/52722 objects degraded (23.719%), 13 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:19:56.037 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:56.038 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:19:56.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:56.097 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:56.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:56.163 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:19:56.198 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:56.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:56.246 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:56.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:56.347 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:56.394 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:56.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:56.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:56.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:56.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:56.532 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:56.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:56.633 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:56.663 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:56.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:56.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:56.765 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:19:56.804 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:56.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:56.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:19:56.906 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:56.946 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:56.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:56.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:57.048 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:19:57.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:57.126 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:57.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:19:57.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:57.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:57.272 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:57.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:57.373 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:57.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:57.442 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:57.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:57.543 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:57.570 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:57.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:57.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:57.672 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:57.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:57 smithi136 ceph-mon[109861]: pgmap v3259: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 672 op/s; 12398/49551 objects degraded (25.021%); 13749/49551 objects misplaced (27.747%); 14 MiB/s, 223 objects/s recovering 2024-08-17T18:19:57.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:57 smithi136 ceph-mon[115078]: pgmap v3259: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 672 op/s; 12398/49551 objects degraded (25.021%); 13749/49551 objects misplaced (27.747%); 14 MiB/s, 223 objects/s recovering 2024-08-17T18:19:57.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:19:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:19:57] "GET /metrics HTTP/1.1" 200 36774 "" "Prometheus/2.51.0" 2024-08-17T18:19:57.726 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:57.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:57.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:57.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:57 smithi165 ceph-mon[105517]: pgmap v3259: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 5 active+recovery_wait+undersized+degraded+remapped, 10 active+remapped+backfill_wait; 1.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 672 op/s; 12398/49551 objects degraded (25.021%); 13749/49551 objects misplaced (27.747%); 14 MiB/s, 223 objects/s recovering 2024-08-17T18:19:57.831 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:57.857 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:57.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:57.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:57.959 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:57.987 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:57.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:57.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:58.088 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:58.115 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:58.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:58.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:58.217 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:58.258 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:58.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:58.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:58.360 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:58.389 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:58.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:58.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:58.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:58.532 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:58.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:58.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:58.635 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:58.663 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:58.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:58.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:58.765 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:19:58.813 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:58.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:58.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:19:58.915 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:58.943 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:58.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:58.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:59.045 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:59.078 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:59.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:59.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:59.180 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:19:59.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:59.219 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:59.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:19:59.314 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:19:59.315 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-17T18:19:59.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-17T18:19:59.315 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:19:59.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.315 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:178744 56.e 56:7d5ba0cd:::benchmark_data_smithi136_356456_object52040:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:19:59.315 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:03.744439+0000", 2024-08-17T18:19:59.315 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 475.56635619899998, 2024-08-17T18:19:59.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0679633769999999, 2024-08-17T18:19:59.315 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:19:59.315 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:19:59.315 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:59.315 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:19:59.315 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:19:59.316 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:19:59.316 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 178744 2024-08-17T18:19:59.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.316 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:19:59.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:19:59.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.744439+0000", 2024-08-17T18:19:59.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:19:59.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.744439+0000", 2024-08-17T18:19:59.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:19:59.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.744441+0000", 2024-08-17T18:19:59.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6139999999999999e-06 2024-08-17T18:19:59.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:19:59.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.744445+0000", 2024-08-17T18:19:59.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5399999999999997e-06 2024-08-17T18:19:59.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:19:59.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.744457+0000", 2024-08-17T18:19:59.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1501999999999999e-05 2024-08-17T18:19:59.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:19:59.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.744464+0000", 2024-08-17T18:19:59.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1029999999999996e-06 2024-08-17T18:19:59.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.744532+0000", 2024-08-17T18:19:59.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8479000000000006e-05 2024-08-17T18:19:59.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:19:59.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.744543+0000", 2024-08-17T18:19:59.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0903999999999999e-05 2024-08-17T18:19:59.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:04.638101+0000", 2024-08-17T18:19:59.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.89355757099999999 2024-08-17T18:19:59.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:19:59.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:04.638171+0000", 2024-08-17T18:19:59.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0232000000000001e-05 2024-08-17T18:19:59.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.767792+0000", 2024-08-17T18:19:59.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1296210959999999 2024-08-17T18:19:59.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:19:59.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.770002+0000", 2024-08-17T18:19:59.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0022101109999999998 2024-08-17T18:19:59.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,5,6", 2024-08-17T18:19:59.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.770135+0000", 2024-08-17T18:19:59.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013292800000000001 2024-08-17T18:19:59.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:19:59.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.773517+0000", 2024-08-17T18:19:59.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033824060000000001 2024-08-17T18:19:59.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.773550+0000", 2024-08-17T18:19:59.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2356e-05 2024-08-17T18:19:59.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.774189+0000", 2024-08-17T18:19:59.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00063878000000000003 2024-08-17T18:19:59.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.812282+0000", 2024-08-17T18:19:59.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038093285999999997 2024-08-17T18:19:59.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:19:59.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.812325+0000", 2024-08-17T18:19:59.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3479999999999997e-05 2024-08-17T18:19:59.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:19:59.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.812402+0000", 2024-08-17T18:19:59.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6989000000000004e-05 2024-08-17T18:19:59.323 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.323 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:59.323 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.323 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:178748 56.4 56:2df471d4:::benchmark_data_smithi136_356456_object52044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:19:59.323 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:03.750998+0000", 2024-08-17T18:19:59.323 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 475.55979720300002, 2024-08-17T18:19:59.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1112687879999998, 2024-08-17T18:19:59.324 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:19:59.324 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:19:59.324 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:59.324 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:19:59.324 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:19:59.324 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:19:59.324 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 178748 2024-08-17T18:19:59.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.324 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:19:59.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:19:59.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.750998+0000", 2024-08-17T18:19:59.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:19:59.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.750998+0000", 2024-08-17T18:19:59.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:19:59.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.751000+0000", 2024-08-17T18:19:59.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9850000000000001e-06 2024-08-17T18:19:59.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:19:59.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.751007+0000", 2024-08-17T18:19:59.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5240000000000003e-06 2024-08-17T18:19:59.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:19:59.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.751029+0000", 2024-08-17T18:19:59.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1356999999999999e-05 2024-08-17T18:19:59.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:19:59.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.751038+0000", 2024-08-17T18:19:59.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1589999999999999e-06 2024-08-17T18:19:59.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.751199+0000", 2024-08-17T18:19:59.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000160951 2024-08-17T18:19:59.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:19:59.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.751212+0000", 2024-08-17T18:19:59.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2661e-05 2024-08-17T18:19:59.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:04.631969+0000", 2024-08-17T18:19:59.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.88075765500000003 2024-08-17T18:19:59.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:19:59.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:04.631981+0000", 2024-08-17T18:19:59.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1421e-05 2024-08-17T18:19:59.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.789391+0000", 2024-08-17T18:19:59.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1574099740000001 2024-08-17T18:19:59.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:19:59.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.791331+0000", 2024-08-17T18:19:59.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001940329 2024-08-17T18:19:59.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,4,5", 2024-08-17T18:19:59.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.791416+0000", 2024-08-17T18:19:59.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5017000000000003e-05 2024-08-17T18:19:59.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:19:59.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.797561+0000", 2024-08-17T18:19:59.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00614534 2024-08-17T18:19:59.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.797955+0000", 2024-08-17T18:19:59.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00039357900000000002 2024-08-17T18:19:59.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.818425+0000", 2024-08-17T18:19:59.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020469696999999999 2024-08-17T18:19:59.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.862070+0000", 2024-08-17T18:19:59.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.043645617999999997 2024-08-17T18:19:59.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:19:59.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.862136+0000", 2024-08-17T18:19:59.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6055000000000004e-05 2024-08-17T18:19:59.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:19:59.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.862267+0000", 2024-08-17T18:19:59.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013046600000000001 2024-08-17T18:19:59.331 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.331 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:59.331 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.331 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:178750 56.4 56:2d326a25:::benchmark_data_smithi136_356456_object52046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1553)", 2024-08-17T18:19:59.331 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:03.752916+0000", 2024-08-17T18:19:59.331 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 475.557878695, 2024-08-17T18:19:59.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.10957587, 2024-08-17T18:19:59.331 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:19:59.332 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:19:59.332 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:59.332 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:19:59.332 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:19:59.332 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:19:59.332 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 178750 2024-08-17T18:19:59.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.332 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:19:59.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:19:59.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.752916+0000", 2024-08-17T18:19:59.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:19:59.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.752916+0000", 2024-08-17T18:19:59.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:19:59.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.752918+0000", 2024-08-17T18:19:59.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.945e-06 2024-08-17T18:19:59.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:19:59.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.752923+0000", 2024-08-17T18:19:59.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8570000000000003e-06 2024-08-17T18:19:59.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:19:59.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.752942+0000", 2024-08-17T18:19:59.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8831e-05 2024-08-17T18:19:59.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:19:59.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.752950+0000", 2024-08-17T18:19:59.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2600000000000005e-06 2024-08-17T18:19:59.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.753014+0000", 2024-08-17T18:19:59.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4207000000000004e-05 2024-08-17T18:19:59.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:19:59.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:03.753023+0000", 2024-08-17T18:19:59.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1920000000000005e-06 2024-08-17T18:19:59.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:04.632023+0000", 2024-08-17T18:19:59.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.87900044399999999 2024-08-17T18:19:59.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:19:59.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:04.632031+0000", 2024-08-17T18:19:59.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0800000000000006e-06 2024-08-17T18:19:59.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.794441+0000", 2024-08-17T18:19:59.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1624093689999999 2024-08-17T18:19:59.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:19:59.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.794534+0000", 2024-08-17T18:19:59.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3392000000000003e-05 2024-08-17T18:19:59.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,4,5", 2024-08-17T18:19:59.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.794615+0000", 2024-08-17T18:19:59.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0704000000000005e-05 2024-08-17T18:19:59.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:19:59.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.797750+0000", 2024-08-17T18:19:59.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031352559999999999 2024-08-17T18:19:59.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.798004+0000", 2024-08-17T18:19:59.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025451500000000003 2024-08-17T18:19:59.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.820269+0000", 2024-08-17T18:19:59.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.022264610000000001 2024-08-17T18:19:59.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.862310+0000", 2024-08-17T18:19:59.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042040688 2024-08-17T18:19:59.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:19:59.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.862367+0000", 2024-08-17T18:19:59.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7466999999999997e-05 2024-08-17T18:19:59.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.339 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:19:59.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:05.862492+0000", 2024-08-17T18:19:59.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125053 2024-08-17T18:19:59.339 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.339 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:59.339 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.339 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207973 56.11 56:8f4c55ea:::benchmark_data_smithi136_356456_object81269:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1571)", 2024-08-17T18:19:59.339 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:41.639460+0000", 2024-08-17T18:19:59.339 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 437.67133537400002, 2024-08-17T18:19:59.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1260926149999999, 2024-08-17T18:19:59.339 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:19:59.339 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:19:59.340 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:59.340 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:19:59.340 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:19:59.340 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:19:59.340 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207973 2024-08-17T18:19:59.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.340 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:19:59.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:19:59.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639460+0000", 2024-08-17T18:19:59.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.341 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:19:59.341 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639460+0000", 2024-08-17T18:19:59.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.341 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:19:59.341 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639462+0000", 2024-08-17T18:19:59.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8029999999999999e-06 2024-08-17T18:19:59.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.341 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:19:59.341 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639468+0000", 2024-08-17T18:19:59.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4840000000000001e-06 2024-08-17T18:19:59.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:19:59.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639484+0000", 2024-08-17T18:19:59.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5696e-05 2024-08-17T18:19:59.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.342 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:19:59.344 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:19:59.344 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639493+0000", 2024-08-17T18:19:59.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6809999999999993e-06 2024-08-17T18:19:59.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.344 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.344 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639869+0000", 2024-08-17T18:19:59.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00037529600000000001 2024-08-17T18:19:59.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.344 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-08-17T18:19:59.344 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639875+0000", 2024-08-17T18:19:59.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7150000000000001e-06 2024-08-17T18:19:59.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641557+0000", 2024-08-17T18:19:59.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001682092 2024-08-17T18:19:59.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:19:59.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641567+0000", 2024-08-17T18:19:59.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9909999999999997e-06 2024-08-17T18:19:59.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.346 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.727904+0000", 2024-08-17T18:19:59.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.086336765 2024-08-17T18:19:59.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.346 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:19:59.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.728036+0000", 2024-08-17T18:19:59.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013191400000000001 2024-08-17T18:19:59.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.346 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5,6", 2024-08-17T18:19:59.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.728178+0000", 2024-08-17T18:19:59.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014163399999999999 2024-08-17T18:19:59.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:19:59.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.731499+0000", 2024-08-17T18:19:59.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033215580000000001 2024-08-17T18:19:59.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.731551+0000", 2024-08-17T18:19:59.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1950000000000002e-05 2024-08-17T18:19:59.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.348 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.348 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.731613+0000", 2024-08-17T18:19:59.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1699999999999995e-05 2024-08-17T18:19:59.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.348 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.348 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.755578+0000", 2024-08-17T18:19:59.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023964798999999998 2024-08-17T18:19:59.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.348 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.348 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.765381+0000", 2024-08-17T18:19:59.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0098029959999999996 2024-08-17T18:19:59.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:19:59.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.765435+0000", 2024-08-17T18:19:59.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4178000000000002e-05 2024-08-17T18:19:59.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:19:59.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.765552+0000", 2024-08-17T18:19:59.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011736299999999999 2024-08-17T18:19:59.349 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.349 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:59.349 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.350 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207989 56.11 56:8c8a752e:::benchmark_data_smithi136_356456_object81285:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1571)", 2024-08-17T18:19:59.350 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:41.639834+0000", 2024-08-17T18:19:59.350 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 437.67096124400001, 2024-08-17T18:19:59.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1259544850000001, 2024-08-17T18:19:59.350 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:19:59.350 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:19:59.350 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:59.350 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:19:59.350 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:19:59.350 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:19:59.351 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207989 2024-08-17T18:19:59.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.351 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:19:59.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:19:59.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639834+0000", 2024-08-17T18:19:59.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:19:59.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639834+0000", 2024-08-17T18:19:59.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:19:59.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639834+0000", 2024-08-17T18:19:59.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5099999999999999e-07 2024-08-17T18:19:59.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:19:59.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639839+0000", 2024-08-17T18:19:59.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1200000000000004e-06 2024-08-17T18:19:59.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:19:59.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639850+0000", 2024-08-17T18:19:59.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1474000000000001e-05 2024-08-17T18:19:59.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:19:59.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639858+0000", 2024-08-17T18:19:59.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4370000000000001e-06 2024-08-17T18:19:59.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639932+0000", 2024-08-17T18:19:59.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4584000000000003e-05 2024-08-17T18:19:59.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.354 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:19:59.354 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639937+0000", 2024-08-17T18:19:59.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2800000000000003e-06 2024-08-17T18:19:59.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.354 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.354 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641657+0000", 2024-08-17T18:19:59.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001719918 2024-08-17T18:19:59.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.354 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:19:59.354 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641665+0000", 2024-08-17T18:19:59.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.267e-06 2024-08-17T18:19:59.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.729531+0000", 2024-08-17T18:19:59.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0878667689999999 2024-08-17T18:19:59.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:19:59.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.729652+0000", 2024-08-17T18:19:59.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012072499999999999 2024-08-17T18:19:59.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5,6", 2024-08-17T18:19:59.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.729797+0000", 2024-08-17T18:19:59.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000145091 2024-08-17T18:19:59.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:19:59.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.731628+0000", 2024-08-17T18:19:59.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001831292 2024-08-17T18:19:59.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.731674+0000", 2024-08-17T18:19:59.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5991000000000001e-05 2024-08-17T18:19:59.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.731710+0000", 2024-08-17T18:19:59.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5945999999999998e-05 2024-08-17T18:19:59.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.755616+0000", 2024-08-17T18:19:59.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023905951000000002 2024-08-17T18:19:59.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.765593+0000", 2024-08-17T18:19:59.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0099765789999999993 2024-08-17T18:19:59.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:19:59.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.765646+0000", 2024-08-17T18:19:59.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3556e-05 2024-08-17T18:19:59.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:19:59.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.765788+0000", 2024-08-17T18:19:59.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014185399999999999 2024-08-17T18:19:59.359 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.359 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:59.359 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.359 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:208008 56.11 56:8e595f6d:::benchmark_data_smithi136_356456_object81304:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1571)", 2024-08-17T18:19:59.359 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:41.639870+0000", 2024-08-17T18:19:59.359 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 437.67092501399998, 2024-08-17T18:19:59.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.126137792, 2024-08-17T18:19:59.359 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:19:59.359 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:19:59.359 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:59.359 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:19:59.360 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:19:59.360 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:19:59.360 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 208008 2024-08-17T18:19:59.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.360 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:19:59.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:19:59.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639870+0000", 2024-08-17T18:19:59.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:19:59.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639870+0000", 2024-08-17T18:19:59.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:19:59.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639871+0000", 2024-08-17T18:19:59.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4799999999999998e-07 2024-08-17T18:19:59.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:19:59.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639873+0000", 2024-08-17T18:19:59.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7269999999999999e-06 2024-08-17T18:19:59.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:19:59.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639881+0000", 2024-08-17T18:19:59.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3000000000000004e-06 2024-08-17T18:19:59.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:19:59.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639887+0000", 2024-08-17T18:19:59.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9329999999999998e-06 2024-08-17T18:19:59.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639961+0000", 2024-08-17T18:19:59.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4760999999999998e-05 2024-08-17T18:19:59.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:19:59.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639965+0000", 2024-08-17T18:19:59.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1409999999999998e-06 2024-08-17T18:19:59.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641697+0000", 2024-08-17T18:19:59.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017317299999999999 2024-08-17T18:19:59.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:19:59.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641704+0000", 2024-08-17T18:19:59.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5089999999999999e-06 2024-08-17T18:19:59.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.730558+0000", 2024-08-17T18:19:59.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0888542299999999 2024-08-17T18:19:59.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:19:59.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.730657+0000", 2024-08-17T18:19:59.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8605999999999997e-05 2024-08-17T18:19:59.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5,6", 2024-08-17T18:19:59.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.730746+0000", 2024-08-17T18:19:59.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9424000000000003e-05 2024-08-17T18:19:59.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:19:59.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.731720+0000", 2024-08-17T18:19:59.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00097365500000000003 2024-08-17T18:19:59.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.731751+0000", 2024-08-17T18:19:59.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1131999999999999e-05 2024-08-17T18:19:59.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.731785+0000", 2024-08-17T18:19:59.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4709000000000003e-05 2024-08-17T18:19:59.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.757960+0000", 2024-08-17T18:19:59.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.026174922999999999 2024-08-17T18:19:59.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.765828+0000", 2024-08-17T18:19:59.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0078676200000000005 2024-08-17T18:19:59.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:19:59.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.765885+0000", 2024-08-17T18:19:59.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6780000000000002e-05 2024-08-17T18:19:59.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:19:59.367 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.766008+0000", 2024-08-17T18:19:59.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012306400000000001 2024-08-17T18:19:59.367 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.367 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:59.367 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.368 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:208039 56.11 56:8c1c5a39:::benchmark_data_smithi136_356456_object81335:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1571)", 2024-08-17T18:19:59.368 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:41.639895+0000", 2024-08-17T18:19:59.368 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 437.67090011900001, 2024-08-17T18:19:59.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.136357155, 2024-08-17T18:19:59.368 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:19:59.368 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:19:59.368 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:59.368 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:19:59.368 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:19:59.368 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:19:59.368 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 208039 2024-08-17T18:19:59.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.368 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:19:59.368 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.368 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:19:59.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639895+0000", 2024-08-17T18:19:59.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:19:59.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639895+0000", 2024-08-17T18:19:59.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:19:59.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639895+0000", 2024-08-17T18:19:59.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6400000000000003e-07 2024-08-17T18:19:59.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:19:59.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639898+0000", 2024-08-17T18:19:59.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.137e-06 2024-08-17T18:19:59.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:19:59.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639904+0000", 2024-08-17T18:19:59.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5930000000000002e-06 2024-08-17T18:19:59.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:19:59.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639910+0000", 2024-08-17T18:19:59.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8150000000000002e-06 2024-08-17T18:19:59.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639989+0000", 2024-08-17T18:19:59.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8971e-05 2024-08-17T18:19:59.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:19:59.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639993+0000", 2024-08-17T18:19:59.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4440000000000001e-06 2024-08-17T18:19:59.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641726+0000", 2024-08-17T18:19:59.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017329439999999999 2024-08-17T18:19:59.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.372 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:19:59.372 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641733+0000", 2024-08-17T18:19:59.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7059999999999998e-06 2024-08-17T18:19:59.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.372 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.372 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.731811+0000", 2024-08-17T18:19:59.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0900782069999999 2024-08-17T18:19:59.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.372 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:19:59.372 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.731884+0000", 2024-08-17T18:19:59.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2576000000000004e-05 2024-08-17T18:19:59.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5,6", 2024-08-17T18:19:59.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.731955+0000", 2024-08-17T18:19:59.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0858000000000005e-05 2024-08-17T18:19:59.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.734312+0000", 2024-08-17T18:19:59.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023571429999999999 2024-08-17T18:19:59.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:19:59.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.734326+0000", 2024-08-17T18:19:59.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4623999999999999e-05 2024-08-17T18:19:59.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.734372+0000", 2024-08-17T18:19:59.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5782000000000003e-05 2024-08-17T18:19:59.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.757997+0000", 2024-08-17T18:19:59.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023624303999999999 2024-08-17T18:19:59.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.776038+0000", 2024-08-17T18:19:59.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018041855999999998 2024-08-17T18:19:59.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:19:59.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.776093+0000", 2024-08-17T18:19:59.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4972999999999997e-05 2024-08-17T18:19:59.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:19:59.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.776252+0000", 2024-08-17T18:19:59.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000158758 2024-08-17T18:19:59.375 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.376 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:59.376 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.376 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:208114 56.11 56:89afd164:::benchmark_data_smithi136_356456_object81410:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1571)", 2024-08-17T18:19:59.376 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:41.639931+0000", 2024-08-17T18:19:59.376 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 437.67086365699998, 2024-08-17T18:19:59.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.136538536, 2024-08-17T18:19:59.376 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:19:59.376 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:19:59.376 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:59.376 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:19:59.376 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:19:59.376 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:19:59.376 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 208114 2024-08-17T18:19:59.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.377 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:19:59.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:19:59.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639931+0000", 2024-08-17T18:19:59.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:19:59.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639931+0000", 2024-08-17T18:19:59.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:19:59.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639932+0000", 2024-08-17T18:19:59.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2799999999999995e-07 2024-08-17T18:19:59.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.378 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:19:59.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639937+0000", 2024-08-17T18:19:59.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6129999999999997e-06 2024-08-17T18:19:59.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.378 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:19:59.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639949+0000", 2024-08-17T18:19:59.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2335e-05 2024-08-17T18:19:59.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:19:59.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639957+0000", 2024-08-17T18:19:59.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0460000000000006e-06 2024-08-17T18:19:59.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.640018+0000", 2024-08-17T18:19:59.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0485000000000001e-05 2024-08-17T18:19:59.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:19:59.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.640023+0000", 2024-08-17T18:19:59.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9609999999999996e-06 2024-08-17T18:19:59.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641788+0000", 2024-08-17T18:19:59.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017650669999999999 2024-08-17T18:19:59.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:19:59.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641794+0000", 2024-08-17T18:19:59.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5050000000000004e-06 2024-08-17T18:19:59.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.732597+0000", 2024-08-17T18:19:59.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0908024000000001 2024-08-17T18:19:59.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:19:59.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.732667+0000", 2024-08-17T18:19:59.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0400000000000004e-05 2024-08-17T18:19:59.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5,6", 2024-08-17T18:19:59.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.732753+0000", 2024-08-17T18:19:59.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6296000000000002e-05 2024-08-17T18:19:59.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.734433+0000", 2024-08-17T18:19:59.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016800809999999999 2024-08-17T18:19:59.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:19:59.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.734442+0000", 2024-08-17T18:19:59.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9039999999999998e-06 2024-08-17T18:19:59.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.734481+0000", 2024-08-17T18:19:59.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8492999999999999e-05 2024-08-17T18:19:59.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.763231+0000", 2024-08-17T18:19:59.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.028750047000000001 2024-08-17T18:19:59.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.776300+0000", 2024-08-17T18:19:59.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.01306944 2024-08-17T18:19:59.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:19:59.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.776353+0000", 2024-08-17T18:19:59.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2197000000000002e-05 2024-08-17T18:19:59.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:19:59.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.776470+0000", 2024-08-17T18:19:59.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117438 2024-08-17T18:19:59.384 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.384 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:59.384 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.384 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:208125 56.11 56:892bf98a:::benchmark_data_smithi136_356456_object81421:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1571)", 2024-08-17T18:19:59.384 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:41.639970+0000", 2024-08-17T18:19:59.384 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 437.67082522800001, 2024-08-17T18:19:59.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1367746009999999, 2024-08-17T18:19:59.384 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:19:59.385 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:19:59.385 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:59.385 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:19:59.385 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:19:59.385 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:19:59.385 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 208125 2024-08-17T18:19:59.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.385 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:19:59.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:19:59.385 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639970+0000", 2024-08-17T18:19:59.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:19:59.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639970+0000", 2024-08-17T18:19:59.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:19:59.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639970+0000", 2024-08-17T18:19:59.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7800000000000001e-07 2024-08-17T18:19:59.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:19:59.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639973+0000", 2024-08-17T18:19:59.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.531e-06 2024-08-17T18:19:59.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:19:59.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639980+0000", 2024-08-17T18:19:59.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5910000000000001e-06 2024-08-17T18:19:59.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:19:59.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639985+0000", 2024-08-17T18:19:59.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3090000000000003e-06 2024-08-17T18:19:59.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.640047+0000", 2024-08-17T18:19:59.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1711000000000006e-05 2024-08-17T18:19:59.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:19:59.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.640051+0000", 2024-08-17T18:19:59.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5159999999999996e-06 2024-08-17T18:19:59.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641818+0000", 2024-08-17T18:19:59.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00176687 2024-08-17T18:19:59.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:19:59.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641824+0000", 2024-08-17T18:19:59.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2450000000000003e-06 2024-08-17T18:19:59.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.733420+0000", 2024-08-17T18:19:59.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.091596206 2024-08-17T18:19:59.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:19:59.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.733507+0000", 2024-08-17T18:19:59.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6594000000000005e-05 2024-08-17T18:19:59.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5,6", 2024-08-17T18:19:59.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.733604+0000", 2024-08-17T18:19:59.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6539999999999994e-05 2024-08-17T18:19:59.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:19:59.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.735282+0000", 2024-08-17T18:19:59.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001678087 2024-08-17T18:19:59.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.735313+0000", 2024-08-17T18:19:59.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1677000000000002e-05 2024-08-17T18:19:59.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.735366+0000", 2024-08-17T18:19:59.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2442999999999999e-05 2024-08-17T18:19:59.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.763561+0000", 2024-08-17T18:19:59.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.028195726000000001 2024-08-17T18:19:59.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.776535+0000", 2024-08-17T18:19:59.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012973088000000001 2024-08-17T18:19:59.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:19:59.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.776597+0000", 2024-08-17T18:19:59.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2722e-05 2024-08-17T18:19:59.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:19:59.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.776744+0000", 2024-08-17T18:19:59.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000147167 2024-08-17T18:19:59.392 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.392 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:59.392 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.392 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:208199 56.11 56:89c8702b:::benchmark_data_smithi136_356456_object81495:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1571)", 2024-08-17T18:19:59.393 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:41.639992+0000", 2024-08-17T18:19:59.393 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 437.67080331699998, 2024-08-17T18:19:59.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1370467849999999, 2024-08-17T18:19:59.393 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:19:59.393 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:19:59.393 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:59.393 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:19:59.393 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:19:59.393 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:19:59.393 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 208199 2024-08-17T18:19:59.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.393 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:19:59.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:19:59.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639992+0000", 2024-08-17T18:19:59.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:19:59.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639992+0000", 2024-08-17T18:19:59.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:19:59.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639992+0000", 2024-08-17T18:19:59.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.72e-07 2024-08-17T18:19:59.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:19:59.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.639994+0000", 2024-08-17T18:19:59.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6470000000000001e-06 2024-08-17T18:19:59.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:19:59.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.640000+0000", 2024-08-17T18:19:59.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2219999999999999e-06 2024-08-17T18:19:59.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:19:59.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.640005+0000", 2024-08-17T18:19:59.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9749999999999998e-06 2024-08-17T18:19:59.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.640075+0000", 2024-08-17T18:19:59.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0371999999999996e-05 2024-08-17T18:19:59.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:19:59.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.640080+0000", 2024-08-17T18:19:59.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6419999999999998e-06 2024-08-17T18:19:59.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641864+0000", 2024-08-17T18:19:59.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001784171 2024-08-17T18:19:59.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:19:59.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.641871+0000", 2024-08-17T18:19:59.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7530000000000004e-06 2024-08-17T18:19:59.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.734508+0000", 2024-08-17T18:19:59.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0926368179999999 2024-08-17T18:19:59.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:19:59.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.734579+0000", 2024-08-17T18:19:59.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0908000000000006e-05 2024-08-17T18:19:59.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5,6", 2024-08-17T18:19:59.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.734651+0000", 2024-08-17T18:19:59.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2687000000000004e-05 2024-08-17T18:19:59.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.737567+0000", 2024-08-17T18:19:59.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0029158840000000001 2024-08-17T18:19:59.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.737606+0000", 2024-08-17T18:19:59.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8507000000000002e-05 2024-08-17T18:19:59.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:19:59.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.737614+0000", 2024-08-17T18:19:59.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0260000000000007e-06 2024-08-17T18:19:59.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.766052+0000", 2024-08-17T18:19:59.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.028438341999999998 2024-08-17T18:19:59.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.776806+0000", 2024-08-17T18:19:59.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010753953 2024-08-17T18:19:59.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:19:59.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.776867+0000", 2024-08-17T18:19:59.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0511999999999999e-05 2024-08-17T18:19:59.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:19:59.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.777039+0000", 2024-08-17T18:19:59.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017199399999999999 2024-08-17T18:19:59.402 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.402 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:59.402 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.402 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:208276 56.11 56:8fd203dd:::benchmark_data_smithi136_356456_object81572:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1571)", 2024-08-17T18:19:59.402 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:41.741074+0000", 2024-08-17T18:19:59.402 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 437.56972083599999, 2024-08-17T18:19:59.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0362123560000001, 2024-08-17T18:19:59.402 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:19:59.402 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:19:59.402 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:59.402 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:19:59.403 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:19:59.403 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:19:59.403 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 208276 2024-08-17T18:19:59.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.403 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:19:59.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:19:59.403 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.741074+0000", 2024-08-17T18:19:59.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:19:59.403 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.741074+0000", 2024-08-17T18:19:59.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:19:59.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.741075+0000", 2024-08-17T18:19:59.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3699999999999999e-07 2024-08-17T18:19:59.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:19:59.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.741080+0000", 2024-08-17T18:19:59.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5109999999999997e-06 2024-08-17T18:19:59.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:19:59.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.741090+0000", 2024-08-17T18:19:59.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.046e-05 2024-08-17T18:19:59.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:19:59.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.741095+0000", 2024-08-17T18:19:59.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1030000000000001e-06 2024-08-17T18:19:59.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.741147+0000", 2024-08-17T18:19:59.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1709000000000002e-05 2024-08-17T18:19:59.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:19:59.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.741153+0000", 2024-08-17T18:19:59.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5679999999999999e-06 2024-08-17T18:19:59.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.735424+0000", 2024-08-17T18:19:59.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.99427149299999995 2024-08-17T18:19:59.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:19:59.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.735530+0000", 2024-08-17T18:19:59.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106152 2024-08-17T18:19:59.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5,6", 2024-08-17T18:19:59.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.735614+0000", 2024-08-17T18:19:59.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4209999999999995e-05 2024-08-17T18:19:59.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:19:59.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.737618+0000", 2024-08-17T18:19:59.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020032930000000002 2024-08-17T18:19:59.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.737674+0000", 2024-08-17T18:19:59.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6208999999999997e-05 2024-08-17T18:19:59.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.737716+0000", 2024-08-17T18:19:59.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2484999999999997e-05 2024-08-17T18:19:59.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.766091+0000", 2024-08-17T18:19:59.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.028374183000000001 2024-08-17T18:19:59.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.777099+0000", 2024-08-17T18:19:59.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011008688000000001 2024-08-17T18:19:59.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.409 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:19:59.409 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.777153+0000", 2024-08-17T18:19:59.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4054000000000001e-05 2024-08-17T18:19:59.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.409 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:19:59.409 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.777287+0000", 2024-08-17T18:19:59.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000133301 2024-08-17T18:19:59.409 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.409 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:59.409 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.410 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:208316 56.11 56:8a69a9a8:::benchmark_data_smithi136_356456_object81612:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1571)", 2024-08-17T18:19:59.410 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:41.796638+0000", 2024-08-17T18:19:59.410 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 437.514157017, 2024-08-17T18:19:59.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.98089237500000004, 2024-08-17T18:19:59.410 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:19:59.410 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:19:59.410 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:59.410 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:19:59.410 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:19:59.410 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:19:59.410 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 208316 2024-08-17T18:19:59.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.411 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:19:59.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.411 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:19:59.411 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.796638+0000", 2024-08-17T18:19:59.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.411 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:19:59.411 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.796638+0000", 2024-08-17T18:19:59.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.411 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:19:59.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.796639+0000", 2024-08-17T18:19:59.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1969999999999999e-06 2024-08-17T18:19:59.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:19:59.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.796644+0000", 2024-08-17T18:19:59.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8210000000000001e-06 2024-08-17T18:19:59.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:19:59.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.796657+0000", 2024-08-17T18:19:59.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2788e-05 2024-08-17T18:19:59.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:19:59.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.796664+0000", 2024-08-17T18:19:59.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4030000000000001e-06 2024-08-17T18:19:59.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.796740+0000", 2024-08-17T18:19:59.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5889999999999993e-05 2024-08-17T18:19:59.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:19:59.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.796750+0000", 2024-08-17T18:19:59.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9480000000000003e-06 2024-08-17T18:19:59.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.736314+0000", 2024-08-17T18:19:59.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93956432700000003 2024-08-17T18:19:59.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:19:59.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.736392+0000", 2024-08-17T18:19:59.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7879000000000004e-05 2024-08-17T18:19:59.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5,6", 2024-08-17T18:19:59.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.736511+0000", 2024-08-17T18:19:59.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118306 2024-08-17T18:19:59.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.415 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.415 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.737774+0000", 2024-08-17T18:19:59.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012633189999999999 2024-08-17T18:19:59.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.415 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:19:59.415 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.738882+0000", 2024-08-17T18:19:59.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011084280000000001 2024-08-17T18:19:59.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.415 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.415 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.738938+0000", 2024-08-17T18:19:59.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5615999999999997e-05 2024-08-17T18:19:59.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.416 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.766343+0000", 2024-08-17T18:19:59.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.027404896000000002 2024-08-17T18:19:59.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.416 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.777348+0000", 2024-08-17T18:19:59.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011005251000000001 2024-08-17T18:19:59.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:19:59.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.777407+0000", 2024-08-17T18:19:59.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8659999999999997e-05 2024-08-17T18:19:59.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:19:59.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:42.777530+0000", 2024-08-17T18:19:59.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012364599999999999 2024-08-17T18:19:59.417 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.417 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:59.417 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.417 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:209638 56.14 56:291d58d8:::benchmark_data_smithi136_356456_object82934:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1572)", 2024-08-17T18:19:59.417 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:43.664215+0000", 2024-08-17T18:19:59.418 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 435.64658040199998, 2024-08-17T18:19:59.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0141130700000001, 2024-08-17T18:19:59.418 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:19:59.418 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:19:59.418 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:59.418 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:19:59.418 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:19:59.418 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:19:59.418 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 209638 2024-08-17T18:19:59.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.418 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:19:59.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:19:59.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.664215+0000", 2024-08-17T18:19:59.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.419 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.419 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:19:59.419 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.664215+0000", 2024-08-17T18:19:59.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.419 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.419 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:19:59.419 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.664216+0000", 2024-08-17T18:19:59.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.668e-06 2024-08-17T18:19:59.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.419 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.419 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:19:59.419 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.664222+0000", 2024-08-17T18:19:59.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.037e-06 2024-08-17T18:19:59.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:19:59.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.664235+0000", 2024-08-17T18:19:59.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.306e-05 2024-08-17T18:19:59.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:19:59.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.664244+0000", 2024-08-17T18:19:59.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3909999999999998e-06 2024-08-17T18:19:59.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.666941+0000", 2024-08-17T18:19:59.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002697199 2024-08-17T18:19:59.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:19:59.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.666948+0000", 2024-08-17T18:19:59.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9580000000000001e-06 2024-08-17T18:19:59.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.674159+0000", 2024-08-17T18:19:59.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0072107130000001 2024-08-17T18:19:59.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:19:59.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.674288+0000", 2024-08-17T18:19:59.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012949000000000001 2024-08-17T18:19:59.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:19:59.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.674631+0000", 2024-08-17T18:19:59.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034243000000000001 2024-08-17T18:19:59.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:19:59.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.677264+0000", 2024-08-17T18:19:59.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0026335199999999999 2024-08-17T18:19:59.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.423 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.423 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.678003+0000", 2024-08-17T18:19:59.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00073914399999999996 2024-08-17T18:19:59.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.423 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.423 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.678204+0000", 2024-08-17T18:19:59.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020022700000000001 2024-08-17T18:19:59.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.423 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:19:59.423 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.678258+0000", 2024-08-17T18:19:59.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4459000000000002e-05 2024-08-17T18:19:59.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.424 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:19:59.424 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.678328+0000", 2024-08-17T18:19:59.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9774e-05 2024-08-17T18:19:59.424 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.424 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:59.424 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.424 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:209670 56.14 56:2f03d7bc:::benchmark_data_smithi136_356456_object82966:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1573)", 2024-08-17T18:19:59.424 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:43.689044+0000", 2024-08-17T18:19:59.424 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 435.62175148400001, 2024-08-17T18:19:59.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.99400135300000003, 2024-08-17T18:19:59.424 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:19:59.425 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:19:59.425 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:59.425 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:19:59.425 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:19:59.425 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:19:59.425 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 209670 2024-08-17T18:19:59.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.425 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:19:59.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:19:59.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.689044+0000", 2024-08-17T18:19:59.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:19:59.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.689044+0000", 2024-08-17T18:19:59.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:19:59.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.689044+0000", 2024-08-17T18:19:59.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5600000000000005e-07 2024-08-17T18:19:59.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:19:59.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.689048+0000", 2024-08-17T18:19:59.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0600000000000001e-06 2024-08-17T18:19:59.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:19:59.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.689058+0000", 2024-08-17T18:19:59.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0009e-05 2024-08-17T18:19:59.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:19:59.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.689064+0000", 2024-08-17T18:19:59.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6200000000000004e-06 2024-08-17T18:19:59.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.689179+0000", 2024-08-17T18:19:59.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011511200000000001 2024-08-17T18:19:59.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:19:59.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:43.689188+0000", 2024-08-17T18:19:59.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9809999999999992e-06 2024-08-17T18:19:59.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.678366+0000", 2024-08-17T18:19:59.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.989177627 2024-08-17T18:19:59.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:19:59.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.678475+0000", 2024-08-17T18:19:59.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109596 2024-08-17T18:19:59.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:19:59.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.678849+0000", 2024-08-17T18:19:59.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000373605 2024-08-17T18:19:59.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.682806+0000", 2024-08-17T18:19:59.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039571370000000003 2024-08-17T18:19:59.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.682860+0000", 2024-08-17T18:19:59.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4181000000000002e-05 2024-08-17T18:19:59.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:19:59.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.682976+0000", 2024-08-17T18:19:59.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115846 2024-08-17T18:19:59.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:19:59.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.683005+0000", 2024-08-17T18:19:59.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8751e-05 2024-08-17T18:19:59.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:19:59.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:44.683045+0000", 2024-08-17T18:19:59.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0172000000000001e-05 2024-08-17T18:19:59.431 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.431 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:59.431 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.432 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:79433 57.3 57:c0b091a2:::benchmark_data_smithi136_389235_object79432:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1616)", 2024-08-17T18:19:59.432 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:14:48.570059+0000", 2024-08-17T18:19:59.432 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 310.74073638800002, 2024-08-17T18:19:59.432 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0321689679999999, 2024-08-17T18:19:59.432 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:19:59.432 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:19:59.432 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:19:59.432 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:19:59.432 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:19:59.432 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:19:59.432 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 79433 2024-08-17T18:19:59.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.432 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:19:59.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:19:59.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.570059+0000", 2024-08-17T18:19:59.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:19:59.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.570059+0000", 2024-08-17T18:19:59.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:19:59.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.570060+0000", 2024-08-17T18:19:59.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5340000000000001e-06 2024-08-17T18:19:59.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:19:59.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.570132+0000", 2024-08-17T18:19:59.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1439000000000005e-05 2024-08-17T18:19:59.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:19:59.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.570149+0000", 2024-08-17T18:19:59.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7110999999999999e-05 2024-08-17T18:19:59.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:19:59.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.570156+0000", 2024-08-17T18:19:59.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1940000000000001e-06 2024-08-17T18:19:59.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.435 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.435 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.435 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.571294+0000", 2024-08-17T18:19:59.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001138132 2024-08-17T18:19:59.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.435 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.435 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-08-17T18:19:59.435 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.571299+0000", 2024-08-17T18:19:59.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7829999999999998e-06 2024-08-17T18:19:59.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.435 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.729716+0000", 2024-08-17T18:19:59.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15841707899999999 2024-08-17T18:19:59.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:19:59.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.729729+0000", 2024-08-17T18:19:59.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3424000000000001e-05 2024-08-17T18:19:59.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:49.602143+0000", 2024-08-17T18:19:59.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.87241396100000002 2024-08-17T18:19:59.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:19:59.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:49.602228+0000", 2024-08-17T18:19:59.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4311e-05 2024-08-17T18:19:59.437 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.437 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:59.437 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.437 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:79437 57.3 57:c2190512:::benchmark_data_smithi136_389235_object79436:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1616)", 2024-08-17T18:19:59.437 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:14:48.570171+0000", 2024-08-17T18:19:59.437 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 310.74062413000001, 2024-08-17T18:19:59.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.032151153, 2024-08-17T18:19:59.438 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:19:59.438 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:19:59.438 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:19:59.438 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:19:59.438 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:19:59.438 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:19:59.438 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 79437 2024-08-17T18:19:59.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.438 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:19:59.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:19:59.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.570171+0000", 2024-08-17T18:19:59.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:19:59.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.570171+0000", 2024-08-17T18:19:59.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:19:59.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.570172+0000", 2024-08-17T18:19:59.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4799999999999998e-07 2024-08-17T18:19:59.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:19:59.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.570190+0000", 2024-08-17T18:19:59.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8467000000000001e-05 2024-08-17T18:19:59.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:19:59.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.570197+0000", 2024-08-17T18:19:59.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.905e-06 2024-08-17T18:19:59.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:19:59.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.570202+0000", 2024-08-17T18:19:59.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4639999999999997e-06 2024-08-17T18:19:59.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.571336+0000", 2024-08-17T18:19:59.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001133818 2024-08-17T18:19:59.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:19:59.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.571340+0000", 2024-08-17T18:19:59.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1849999999999997e-06 2024-08-17T18:19:59.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.729754+0000", 2024-08-17T18:19:59.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15841395699999999 2024-08-17T18:19:59.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:19:59.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:48.729762+0000", 2024-08-17T18:19:59.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5190000000000003e-06 2024-08-17T18:19:59.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:49.602262+0000", 2024-08-17T18:19:59.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.87249982000000004 2024-08-17T18:19:59.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:19:59.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:14:49.602322+0000", 2024-08-17T18:19:59.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0470000000000002e-05 2024-08-17T18:19:59.443 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.443 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:59.443 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.443 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:122085 57.2 57:42ce4696:::benchmark_data_smithi136_389235_object6163:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1639)", 2024-08-17T18:19:59.443 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:54.574198+0000", 2024-08-17T18:19:59.443 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 244.73659737899999, 2024-08-17T18:19:59.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5265542280000002, 2024-08-17T18:19:59.444 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:19:59.444 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:19:59.444 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:59.444 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:19:59.444 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:19:59.444 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:19:59.444 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122085 2024-08-17T18:19:59.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.444 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:19:59.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:19:59.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.574198+0000", 2024-08-17T18:19:59.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:19:59.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.574198+0000", 2024-08-17T18:19:59.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:19:59.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.574199+0000", 2024-08-17T18:19:59.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4300000000000001e-07 2024-08-17T18:19:59.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:19:59.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.574203+0000", 2024-08-17T18:19:59.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4170000000000002e-06 2024-08-17T18:19:59.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:19:59.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.574214+0000", 2024-08-17T18:19:59.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0672999999999999e-05 2024-08-17T18:19:59.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:19:59.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.574219+0000", 2024-08-17T18:19:59.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0529999999999997e-06 2024-08-17T18:19:59.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.574268+0000", 2024-08-17T18:19:59.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8732e-05 2024-08-17T18:19:59.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:19:59.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.574368+0000", 2024-08-17T18:19:59.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010063199999999999 2024-08-17T18:19:59.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:19:59.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.574725+0000", 2024-08-17T18:19:59.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00035646700000000001 2024-08-17T18:19:59.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:19:59.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.576207+0000", 2024-08-17T18:19:59.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014825979999999999 2024-08-17T18:19:59.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.577429+0000", 2024-08-17T18:19:59.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001221576 2024-08-17T18:19:59.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.099083+0000", 2024-08-17T18:19:59.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5216544700000001 2024-08-17T18:19:59.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:19:59.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.099169+0000", 2024-08-17T18:19:59.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5350000000000001e-05 2024-08-17T18:19:59.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:19:59.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.100752+0000", 2024-08-17T18:19:59.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015833170000000001 2024-08-17T18:19:59.450 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.450 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:59.450 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.450 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:122093 57.2 57:407dd288:::benchmark_data_smithi136_389235_object6171:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1639)", 2024-08-17T18:19:59.450 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:54.621943+0000", 2024-08-17T18:19:59.450 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 244.68885204700001, 2024-08-17T18:19:59.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4794903640000001, 2024-08-17T18:19:59.450 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:19:59.450 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:19:59.451 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:59.451 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:19:59.451 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:19:59.451 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:19:59.451 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122093 2024-08-17T18:19:59.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.451 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:19:59.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:19:59.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.621943+0000", 2024-08-17T18:19:59.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:19:59.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.621943+0000", 2024-08-17T18:19:59.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:19:59.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.621944+0000", 2024-08-17T18:19:59.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4300000000000002e-07 2024-08-17T18:19:59.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:19:59.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.621947+0000", 2024-08-17T18:19:59.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9660000000000001e-06 2024-08-17T18:19:59.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:19:59.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.621956+0000", 2024-08-17T18:19:59.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2329999999999995e-06 2024-08-17T18:19:59.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:19:59.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.621963+0000", 2024-08-17T18:19:59.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.968e-06 2024-08-17T18:19:59.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.622012+0000", 2024-08-17T18:19:59.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8982e-05 2024-08-17T18:19:59.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:19:59.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.622107+0000", 2024-08-17T18:19:59.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4905e-05 2024-08-17T18:19:59.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:19:59.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.622409+0000", 2024-08-17T18:19:59.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00030236500000000002 2024-08-17T18:19:59.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:19:59.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.623875+0000", 2024-08-17T18:19:59.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014658869999999999 2024-08-17T18:19:59.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:54.624591+0000", 2024-08-17T18:19:59.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00071544800000000004 2024-08-17T18:19:59.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.100795+0000", 2024-08-17T18:19:59.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.476204418 2024-08-17T18:19:59.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:19:59.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.100830+0000", 2024-08-17T18:19:59.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4641999999999998e-05 2024-08-17T18:19:59.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:19:59.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.101433+0000", 2024-08-17T18:19:59.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00060380700000000004 2024-08-17T18:19:59.456 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.456 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:59.456 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.457 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:122102 57.2 57:436a03b4:::benchmark_data_smithi136_389235_object6180:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1641)", 2024-08-17T18:19:59.457 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:57.128536+0000", 2024-08-17T18:19:59.457 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 242.182259007, 2024-08-17T18:19:59.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3281144779999998, 2024-08-17T18:19:59.457 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:19:59.457 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:19:59.457 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:59.457 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:19:59.457 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:19:59.457 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:19:59.458 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122102 2024-08-17T18:19:59.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.458 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:19:59.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:19:59.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.128536+0000", 2024-08-17T18:19:59.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:19:59.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.128536+0000", 2024-08-17T18:19:59.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:19:59.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.128537+0000", 2024-08-17T18:19:59.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.079e-06 2024-08-17T18:19:59.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:19:59.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.128542+0000", 2024-08-17T18:19:59.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5460000000000002e-06 2024-08-17T18:19:59.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:19:59.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.128552+0000", 2024-08-17T18:19:59.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0550999999999999e-05 2024-08-17T18:19:59.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:19:59.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.128557+0000", 2024-08-17T18:19:59.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0000000000000004e-06 2024-08-17T18:19:59.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.128614+0000", 2024-08-17T18:19:59.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6749999999999997e-05 2024-08-17T18:19:59.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:19:59.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.128723+0000", 2024-08-17T18:19:59.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109079 2024-08-17T18:19:59.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:19:59.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.129047+0000", 2024-08-17T18:19:59.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00032373900000000003 2024-08-17T18:19:59.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:19:59.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.130562+0000", 2024-08-17T18:19:59.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001515504 2024-08-17T18:19:59.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:57.131282+0000", 2024-08-17T18:19:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000719756 2024-08-17T18:19:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:16:00.455193+0000", 2024-08-17T18:19:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3239109089999999 2024-08-17T18:19:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:19:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:16:00.455255+0000", 2024-08-17T18:19:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2345e-05 2024-08-17T18:19:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:19:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:16:00.456651+0000", 2024-08-17T18:19:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00139522 2024-08-17T18:19:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:198656 57.7 57:e7bb643e:::benchmark_data_smithi136_389235_object82734:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1680)", 2024-08-17T18:19:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:19:26.687038+0000", 2024-08-17T18:19:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.623756614999998, 2024-08-17T18:19:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.95698935600000001, 2024-08-17T18:19:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:19:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:19:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:19:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:19:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:19:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:19:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 198656 2024-08-17T18:19:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:19:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:19:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:26.687038+0000", 2024-08-17T18:19:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:19:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:26.687038+0000", 2024-08-17T18:19:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:19:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:19:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:26.687040+0000", 2024-08-17T18:19:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2589999999999999e-06 2024-08-17T18:19:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:19:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:26.687044+0000", 2024-08-17T18:19:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2719999999999999e-06 2024-08-17T18:19:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:19:59.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:26.687054+0000", 2024-08-17T18:19:59.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0189e-05 2024-08-17T18:19:59.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:19:59.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:26.687059+0000", 2024-08-17T18:19:59.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7879999999999997e-06 2024-08-17T18:19:59.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:26.687104+0000", 2024-08-17T18:19:59.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5484e-05 2024-08-17T18:19:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:19:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:26.687112+0000", 2024-08-17T18:19:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9140000000000008e-06 2024-08-17T18:19:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:19:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:27.621279+0000", 2024-08-17T18:19:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93416628300000004 2024-08-17T18:19:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:19:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:27.621382+0000", 2024-08-17T18:19:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103251 2024-08-17T18:19:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,7", 2024-08-17T18:19:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:27.621763+0000", 2024-08-17T18:19:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038148300000000001 2024-08-17T18:19:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:19:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:27.623922+0000", 2024-08-17T18:19:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021585990000000002 2024-08-17T18:19:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:27.628190+0000", 2024-08-17T18:19:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.004268015 2024-08-17T18:19:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:19:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:27.643966+0000", 2024-08-17T18:19:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015776202 2024-08-17T18:19:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:19:59.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:27.643999+0000", 2024-08-17T18:19:59.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2598999999999997e-05 2024-08-17T18:19:59.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:19:59.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:19:59.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:19:59.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:27.644028+0000", 2024-08-17T18:19:59.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9017999999999999e-05 2024-08-17T18:19:59.470 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.470 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:59.470 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.470 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:19:59.471 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:19:59.471 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:19:59.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:59.472 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:59.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:19:59.573 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:19:59.599 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:59.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:59.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:19:59.624 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:19:59.640 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:59 smithi136 ceph-mon[109861]: pgmap v3260: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 872 MiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 808 op/s; 9286/41892 objects degraded (22.167%); 13066/41892 objects misplaced (31.190%); 18 MiB/s, 302 objects/s recovering 2024-08-17T18:19:59.640 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:59 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1710992459' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:19:59.640 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:59 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:19:59.641 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:59 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:19:59.641 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:19:59 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:19:59.641 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:59 smithi136 ceph-mon[115078]: pgmap v3260: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 872 MiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 808 op/s; 9286/41892 objects degraded (22.167%); 13066/41892 objects misplaced (31.190%); 18 MiB/s, 302 objects/s recovering 2024-08-17T18:19:59.641 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:59 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1710992459' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:19:59.641 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:59 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:19:59.641 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:59 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:19:59.641 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:19:59 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:19:59.701 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:19:59.744 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:19:59.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:59.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:19:59.847 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:19:59.875 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:19:59.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:59.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:19:59.977 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:00.008 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:00.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:00.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:59 smithi165 ceph-mon[105517]: pgmap v3260: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 872 MiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 808 op/s; 9286/41892 objects degraded (22.167%); 13066/41892 objects misplaced (31.190%); 18 MiB/s, 302 objects/s recovering 2024-08-17T18:20:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:59 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1710992459' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:20:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:59 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:20:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:59 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:19:59 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:00.110 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:00.139 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:00.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:00.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:00.241 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:00.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:00.297 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:00.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:00.398 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:00.422 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:00.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:00.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:00.524 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:00.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:00.575 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:00.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:00.676 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:00.703 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:00.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:00.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:00.805 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:00.833 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:00.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:00.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:00.935 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:00.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:20:00.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:20:00.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[109861]: osdmap e1686: 8 total, 8 up, 7 in 2024-08-17T18:20:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 9286/41892 objects degraded (22.167%), 12 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:20:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[109861]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Degraded data redundancy: 9286/41892 objects degraded (22.167%), 12 pgs degraded, 5 pgs undersized 2024-08-17T18:20:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[109861]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-08-17T18:20:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[109861]: [WRN] PG_DEGRADED: Degraded data redundancy: 9286/41892 objects degraded (22.167%), 12 pgs degraded, 5 pgs undersized 2024-08-17T18:20:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[109861]: pg 1.4 is stuck undersized for 87s, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,7] 2024-08-17T18:20:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[109861]: pg 1.8 is stuck undersized for 87s, current state active+recovery_wait+undersized+degraded+remapped, last acting [2,4] 2024-08-17T18:20:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[109861]: pg 2.2 is active+recovery_wait+degraded, acting [5,6,3] 2024-08-17T18:20:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[109861]: pg 57.2 is active+recovery_wait+degraded+remapped, acting [2,5,4] 2024-08-17T18:20:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[109861]: pg 57.3 is active+recovery_wait+degraded+remapped, acting [6,7,0] 2024-08-17T18:20:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[109861]: pg 57.5 is active+recovery_wait+degraded+remapped, acting [5,2,6] 2024-08-17T18:20:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[109861]: pg 57.6 is stuck undersized for 4m, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,2] 2024-08-17T18:20:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[109861]: pg 57.10 is active+recovery_wait+degraded+remapped, acting [6,7,2] 2024-08-17T18:20:00.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[109861]: pg 57.12 is stuck undersized for 87s, current state active+recovery_wait+undersized+degraded+remapped, last acting [4,7] 2024-08-17T18:20:00.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[109861]: pg 57.14 is active+recovery_wait+degraded+remapped, acting [5,7,4] 2024-08-17T18:20:00.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[109861]: pg 57.15 is active+recovery_wait+degraded+remapped, acting [5,7,2] 2024-08-17T18:20:00.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[109861]: pg 57.17 is stuck undersized for 2m, current state active+recovering+undersized+remapped, last acting [2,7] 2024-08-17T18:20:00.948 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[109861]: pg 57.18 is active+recovery_wait+degraded+remapped, acting [5,6,4] 2024-08-17T18:20:00.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:20:00.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:20:00.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[115078]: osdmap e1686: 8 total, 8 up, 7 in 2024-08-17T18:20:00.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 9286/41892 objects degraded (22.167%), 12 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:20:00.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[115078]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Degraded data redundancy: 9286/41892 objects degraded (22.167%), 12 pgs degraded, 5 pgs undersized 2024-08-17T18:20:00.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[115078]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-08-17T18:20:00.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[115078]: [WRN] PG_DEGRADED: Degraded data redundancy: 9286/41892 objects degraded (22.167%), 12 pgs degraded, 5 pgs undersized 2024-08-17T18:20:00.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[115078]: pg 1.4 is stuck undersized for 87s, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,7] 2024-08-17T18:20:00.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[115078]: pg 1.8 is stuck undersized for 87s, current state active+recovery_wait+undersized+degraded+remapped, last acting [2,4] 2024-08-17T18:20:00.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[115078]: pg 2.2 is active+recovery_wait+degraded, acting [5,6,3] 2024-08-17T18:20:00.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[115078]: pg 57.2 is active+recovery_wait+degraded+remapped, acting [2,5,4] 2024-08-17T18:20:00.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[115078]: pg 57.3 is active+recovery_wait+degraded+remapped, acting [6,7,0] 2024-08-17T18:20:00.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[115078]: pg 57.5 is active+recovery_wait+degraded+remapped, acting [5,2,6] 2024-08-17T18:20:00.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[115078]: pg 57.6 is stuck undersized for 4m, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,2] 2024-08-17T18:20:00.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[115078]: pg 57.10 is active+recovery_wait+degraded+remapped, acting [6,7,2] 2024-08-17T18:20:00.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[115078]: pg 57.12 is stuck undersized for 87s, current state active+recovery_wait+undersized+degraded+remapped, last acting [4,7] 2024-08-17T18:20:00.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[115078]: pg 57.14 is active+recovery_wait+degraded+remapped, acting [5,7,4] 2024-08-17T18:20:00.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[115078]: pg 57.15 is active+recovery_wait+degraded+remapped, acting [5,7,2] 2024-08-17T18:20:00.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[115078]: pg 57.17 is stuck undersized for 2m, current state active+recovering+undersized+remapped, last acting [2,7] 2024-08-17T18:20:00.949 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:00 smithi136 ceph-mon[115078]: pg 57.18 is active+recovery_wait+degraded+remapped, acting [5,6,4] 2024-08-17T18:20:00.959 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:00.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:00.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:00 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:20:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:00 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:20:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:00 smithi165 ceph-mon[105517]: osdmap e1686: 8 total, 8 up, 7 in 2024-08-17T18:20:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:00 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 9286/41892 objects degraded (22.167%), 12 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:20:01.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:00 smithi165 ceph-mon[105517]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Degraded data redundancy: 9286/41892 objects degraded (22.167%), 12 pgs degraded, 5 pgs undersized 2024-08-17T18:20:01.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:00 smithi165 ceph-mon[105517]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-08-17T18:20:01.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:00 smithi165 ceph-mon[105517]: [WRN] PG_DEGRADED: Degraded data redundancy: 9286/41892 objects degraded (22.167%), 12 pgs degraded, 5 pgs undersized 2024-08-17T18:20:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:00 smithi165 ceph-mon[105517]: pg 1.4 is stuck undersized for 87s, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,7] 2024-08-17T18:20:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:00 smithi165 ceph-mon[105517]: pg 1.8 is stuck undersized for 87s, current state active+recovery_wait+undersized+degraded+remapped, last acting [2,4] 2024-08-17T18:20:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:00 smithi165 ceph-mon[105517]: pg 2.2 is active+recovery_wait+degraded, acting [5,6,3] 2024-08-17T18:20:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:00 smithi165 ceph-mon[105517]: pg 57.2 is active+recovery_wait+degraded+remapped, acting [2,5,4] 2024-08-17T18:20:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:00 smithi165 ceph-mon[105517]: pg 57.3 is active+recovery_wait+degraded+remapped, acting [6,7,0] 2024-08-17T18:20:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:00 smithi165 ceph-mon[105517]: pg 57.5 is active+recovery_wait+degraded+remapped, acting [5,2,6] 2024-08-17T18:20:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:00 smithi165 ceph-mon[105517]: pg 57.6 is stuck undersized for 4m, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,2] 2024-08-17T18:20:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:00 smithi165 ceph-mon[105517]: pg 57.10 is active+recovery_wait+degraded+remapped, acting [6,7,2] 2024-08-17T18:20:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:00 smithi165 ceph-mon[105517]: pg 57.12 is stuck undersized for 87s, current state active+recovery_wait+undersized+degraded+remapped, last acting [4,7] 2024-08-17T18:20:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:00 smithi165 ceph-mon[105517]: pg 57.14 is active+recovery_wait+degraded+remapped, acting [5,7,4] 2024-08-17T18:20:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:00 smithi165 ceph-mon[105517]: pg 57.15 is active+recovery_wait+degraded+remapped, acting [5,7,2] 2024-08-17T18:20:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:00 smithi165 ceph-mon[105517]: pg 57.17 is stuck undersized for 2m, current state active+recovering+undersized+remapped, last acting [2,7] 2024-08-17T18:20:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:00 smithi165 ceph-mon[105517]: pg 57.18 is active+recovery_wait+degraded+remapped, acting [5,6,4] 2024-08-17T18:20:01.061 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:01.086 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:01.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:01.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:01.183 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-17T18:20:01.187 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:01.254 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:01.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:01.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:01.355 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:01.390 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:01.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:01.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:01.491 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:01.532 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:01.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:01.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:01.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:01.681 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:01.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:01.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:01.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:01.821 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:01.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:01.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:01.847 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:20:01.923 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:01.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:01 smithi136 ceph-mon[109861]: pgmap v3262: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 828 MiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 727 op/s; 9156/39792 objects degraded (23.010%); 12496/39792 objects misplaced (31.403%); 15 MiB/s, 255 objects/s recovering 2024-08-17T18:20:01.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:01 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:01.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:01 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:01.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:01 smithi136 ceph-mon[115078]: pgmap v3262: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 828 MiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 727 op/s; 9156/39792 objects degraded (23.010%); 12496/39792 objects misplaced (31.403%); 15 MiB/s, 255 objects/s recovering 2024-08-17T18:20:01.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:01 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:01.950 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:01 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:01.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:01.957 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:01.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:02.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:01 smithi165 ceph-mon[105517]: pgmap v3262: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 828 MiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 727 op/s; 9156/39792 objects degraded (23.010%); 12496/39792 objects misplaced (31.403%); 15 MiB/s, 255 objects/s recovering 2024-08-17T18:20:02.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:01 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:02.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:01 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:02.059 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:02.100 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:02.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:02.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:02.202 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:02.230 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:02.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:02.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:02.333 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:02.374 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:02.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:02.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:02.476 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:02.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:02.503 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:02.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:02.604 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:02.630 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:02.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:02.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:02.732 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:02.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:02.773 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:02.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:02.874 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:02.903 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:02.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:02.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:03.005 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:03.040 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:03.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:03.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:03.141 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:03.184 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:03.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:03.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:03.286 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:03.309 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:03.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:03.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:03.411 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:03.437 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:03.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:03.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:03.467 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:20:03.539 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:03.576 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:03.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:03.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:03.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:03.715 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:03.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:03.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:03 smithi165 ceph-mon[105517]: pgmap v3263: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 635 MiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 8554/30546 objects degraded (28.004%); 9776/30546 objects misplaced (32.004%); 12 MiB/s, 206 objects/s recovering 2024-08-17T18:20:03.817 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:03.859 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:03.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:03.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:03.932 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:03 smithi136 ceph-mon[109861]: pgmap v3263: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 635 MiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 8554/30546 objects degraded (28.004%); 9776/30546 objects misplaced (32.004%); 12 MiB/s, 206 objects/s recovering 2024-08-17T18:20:03.932 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:03 smithi136 ceph-mon[115078]: pgmap v3263: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 635 MiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 8554/30546 objects degraded (28.004%); 9776/30546 objects misplaced (32.004%); 12 MiB/s, 206 objects/s recovering 2024-08-17T18:20:03.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:04.004 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:04.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:04.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:04.106 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:04.144 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:04.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:04.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:04.246 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:04.275 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:04.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:04.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:04.346 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:20:04.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:04.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:04.440 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:04.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:04.541 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:04.582 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:04.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:04.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:04.684 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:04.731 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:04.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:04.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:04.832 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:04.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:04.859 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:04.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:04.960 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:04.996 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:04.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:04.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:05.098 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:05.125 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:05.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:05.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:05.227 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:05.252 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:05.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:05.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:05.354 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:05.389 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:05.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:05.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:05.491 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:05.536 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:05.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:05.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:05.637 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:05.678 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:20:05.678 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:20:05.684 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:05.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:05.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:05.710 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:20:05.710 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:20:05.742 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:20:05.743 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:20:05.769 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:20:05.769 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:20:05.786 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:05.794 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:20:05.794 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:20:05.795 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:05 smithi136 ceph-mon[109861]: pgmap v3264: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 635 MiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 8554/30546 objects degraded (28.004%); 9776/30546 objects misplaced (32.004%); 12 MiB/s, 206 objects/s recovering 2024-08-17T18:20:05.795 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:05 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 8554/30546 objects degraded (28.004%), 12 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:20:05.795 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:05 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2478266304' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:20:05.795 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:05 smithi136 ceph-mon[115078]: pgmap v3264: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 635 MiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 8554/30546 objects degraded (28.004%); 9776/30546 objects misplaced (32.004%); 12 MiB/s, 206 objects/s recovering 2024-08-17T18:20:05.795 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:05 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 8554/30546 objects degraded (28.004%), 12 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:20:05.795 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:05 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2478266304' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:20:05.821 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:20:05.822 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:20:05.827 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:05.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:05.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:05.847 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:20:05.847 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:20:05.879 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:20:05.879 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:20:05.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:05.967 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:05.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:05.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:05 smithi165 ceph-mon[105517]: pgmap v3264: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 635 MiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 8554/30546 objects degraded (28.004%); 9776/30546 objects misplaced (32.004%); 12 MiB/s, 206 objects/s recovering 2024-08-17T18:20:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:05 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 8554/30546 objects degraded (28.004%), 12 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:20:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:05 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2478266304' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:20:06.068 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:06.112 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:06.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:06.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:06.213 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:06.260 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:06.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:06.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:06.362 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:06.388 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:06.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:06.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:06.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:06.531 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:06.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:06.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:06.633 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:06.678 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:06.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:06.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:20:06.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:06.814 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:06.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:06.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:06.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:06.935 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:20:06.935 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:20:06.959 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:06.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:06.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:07.061 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:07.093 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:07.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:07.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:07.195 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:07.228 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:07.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:07.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:07.330 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:07.365 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:07.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:07.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:07.467 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:07.495 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:07.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:07.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:07.566 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:20:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:20:07] "GET /metrics HTTP/1.1" 200 36772 "" "Prometheus/2.51.0" 2024-08-17T18:20:07.597 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:07.637 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:07.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:07.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:07.738 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:07.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:07 smithi165 ceph-mon[105517]: pgmap v3265: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 566 MiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 743 op/s; 8439/27225 objects degraded (30.997%); 8593/27225 objects misplaced (31.563%); 12 MiB/s, 206 objects/s recovering 2024-08-17T18:20:07.773 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:07.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:07.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:07.876 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:07.905 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:07.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:07.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:07.939 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:07 smithi136 ceph-mon[109861]: pgmap v3265: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 566 MiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 743 op/s; 8439/27225 objects degraded (30.997%); 8593/27225 objects misplaced (31.563%); 12 MiB/s, 206 objects/s recovering 2024-08-17T18:20:07.939 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:07 smithi136 ceph-mon[115078]: pgmap v3265: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 566 MiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 743 op/s; 8439/27225 objects degraded (30.997%); 8593/27225 objects misplaced (31.563%); 12 MiB/s, 206 objects/s recovering 2024-08-17T18:20:08.007 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:08.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:08.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:08.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:08.145 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:08.172 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:08.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:08.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:08.274 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:08.304 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:08.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:08.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:08.406 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:08.434 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:08.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:08.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:08.536 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:08.562 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:08.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:08.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:08.794 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:08.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:08.828 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:08.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:08.836 INFO:teuthology.orchestra.run.smithi136.stderr:set pool 1 pgp_num to 11 2024-08-17T18:20:08.929 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:08.957 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:08.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:08.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:09.058 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:09.083 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:09.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:09.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:09.184 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:09.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:09.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:08 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:09.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:08 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/453096272' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T18:20:09.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:08 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T18:20:09.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:09.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:08 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:09.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:08 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/453096272' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T18:20:09.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:08 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T18:20:09.214 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:09.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:09.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:08 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:08 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/453096272' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T18:20:09.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:08 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T18:20:09.315 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:09.357 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:09.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:09.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:09.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:09.498 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:09.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:09.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:09.600 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:09.629 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:09.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:09.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:09.731 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:09.759 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:09.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:09.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:09.860 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:09.884 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:09.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:09.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:09 smithi136 ceph-mon[109861]: pgmap v3266: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 388 MiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 773 op/s; 7627/18684 objects degraded (40.821%); 6736/18684 objects misplaced (36.052%); 9.2 MiB/s, 180 objects/s recovering 2024-08-17T18:20:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:09 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-08-17T18:20:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:09 smithi136 ceph-mon[109861]: osdmap e1687: 8 total, 8 up, 7 in 2024-08-17T18:20:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:09 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 7627/18684 objects degraded (40.821%), 12 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:20:09.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:09 smithi136 ceph-mon[115078]: pgmap v3266: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 388 MiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 773 op/s; 7627/18684 objects degraded (40.821%); 6736/18684 objects misplaced (36.052%); 9.2 MiB/s, 180 objects/s recovering 2024-08-17T18:20:09.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:09 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-08-17T18:20:09.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:09 smithi136 ceph-mon[115078]: osdmap e1687: 8 total, 8 up, 7 in 2024-08-17T18:20:09.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:09 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 7627/18684 objects degraded (40.821%), 12 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:20:09.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:10.012 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:10.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:10.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:10.114 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:10.138 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:10.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:10.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:10.161 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:09 smithi165 ceph-mon[105517]: pgmap v3266: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 388 MiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 773 op/s; 7627/18684 objects degraded (40.821%); 6736/18684 objects misplaced (36.052%); 9.2 MiB/s, 180 objects/s recovering 2024-08-17T18:20:10.161 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:09 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-08-17T18:20:10.161 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:09 smithi165 ceph-mon[105517]: osdmap e1687: 8 total, 8 up, 7 in 2024-08-17T18:20:10.162 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:09 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 7627/18684 objects degraded (40.821%), 12 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:20:10.240 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:10.290 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:10.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:10.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:10.392 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:10.417 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:10.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:10.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:10.519 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:10.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:10.547 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:10.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:10.648 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:10.672 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:10.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:10.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:10.774 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:10.812 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:10.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:10.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:10.913 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:10.950 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:10.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:10.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:11.052 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:11.080 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:11.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:11.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:11.088 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:20:11.088 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:20:11.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:11.229 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:11.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:11.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:11.331 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:11.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:11.361 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:11.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:11.463 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:11.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:11.510 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:11.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:11.612 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:11.641 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:11.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:11.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:11.704 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:20:11.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:11.790 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:11.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:11.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:11.892 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:11.920 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:11.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:11.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:11 smithi165 ceph-mon[105517]: pgmap v3268: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 343 MiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 776 op/s; 7495/16497 objects degraded (45.433%); 6139/16497 objects misplaced (37.213%); 9.2 MiB/s, 180 objects/s recovering 2024-08-17T18:20:12.022 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:12.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:12.058 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:12.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:12.159 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:12.192 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:12.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:12.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:12.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:11 smithi136 ceph-mon[109861]: pgmap v3268: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 343 MiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 776 op/s; 7495/16497 objects degraded (45.433%); 6139/16497 objects misplaced (37.213%); 9.2 MiB/s, 180 objects/s recovering 2024-08-17T18:20:12.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:11 smithi136 ceph-mon[115078]: pgmap v3268: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 343 MiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 776 op/s; 7495/16497 objects degraded (45.433%); 6139/16497 objects misplaced (37.213%); 9.2 MiB/s, 180 objects/s recovering 2024-08-17T18:20:12.291 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:20:12.294 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:12.319 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-17T18:20:12.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:12.329 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:12.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:12.430 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:12.471 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:12.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:12.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:12.572 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:12.640 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:12.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:12.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:12.741 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:12.768 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:12.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:12.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:12.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:12.897 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:12.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:12.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:12.999 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:13.040 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:13.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:13.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:13.142 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:13.166 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:13.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:13.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:13.268 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:13.294 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:13.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:13.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:13.396 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:13.420 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:13.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:13.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:13.522 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:13.553 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:13.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:13.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:13.655 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:13.684 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:13.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:13.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:13.786 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:13.815 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:13.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:13.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:13.917 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:13 smithi136 ceph-mon[109861]: pgmap v3269: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 6 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 136 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 802 op/s; 6637/6585 objects degraded (100.790%); 3807/6585 objects misplaced (57.813%); 7.8 MiB/s, 183 objects/s recovering 2024-08-17T18:20:13.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:13 smithi136 ceph-mon[115078]: pgmap v3269: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 6 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 136 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 802 op/s; 6637/6585 objects degraded (100.790%); 3807/6585 objects misplaced (57.813%); 7.8 MiB/s, 183 objects/s recovering 2024-08-17T18:20:13.960 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:13.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:13.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:14.062 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:14.088 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:14.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:14.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:14.189 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:14.234 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:14.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:14.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:13 smithi165 ceph-mon[105517]: pgmap v3269: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 6 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 136 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 802 op/s; 6637/6585 objects degraded (100.790%); 3807/6585 objects misplaced (57.813%); 7.8 MiB/s, 183 objects/s recovering 2024-08-17T18:20:14.337 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:14.380 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:14.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:14.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:14.482 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:14.511 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:14.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:14.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:14.582 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Removed 115919 objects 2024-08-17T18:20:14.582 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Clean up completed and total clean up time :275.79 2024-08-17T18:20:14.613 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:14.636 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:14.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:14.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:14.739 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:14.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:14.763 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:14.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:14.864 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:14.886 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:14.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:14.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:14.988 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:15.010 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:15.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:15.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:15.112 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:15.133 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:15.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:15.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:15.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:14 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 6637/6585 objects degraded (100.790%), 11 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:20:15.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:14 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 6637/6585 objects degraded (100.790%), 11 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:20:15.235 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:15.256 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:15.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:15.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:15.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:14 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 6637/6585 objects degraded (100.790%), 11 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:20:15.358 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:15.379 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:15.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:15.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:15.480 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:15.501 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:15.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:15.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:15.525 INFO:tasks.thrashosds.thrasher:in_osds: [2, 4, 6, 0, 5, 7, 3] out_osds: [1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:20:15.525 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:20:15.525 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-08-17T18:20:15.525 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd in 1 2024-08-17T18:20:15.604 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:15.626 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:15.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:15.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:15.728 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:15.750 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:15.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:15.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:15.851 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:15.877 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:15.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:15.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:15.978 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:16.001 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:16.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:16.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:16.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:16.125 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:16.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:16.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:16.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:15 smithi136 ceph-mon[109861]: pgmap v3270: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 6 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 136 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 802 op/s; 6637/6585 objects degraded (100.790%); 3807/6585 objects misplaced (57.813%); 7.8 MiB/s, 183 objects/s recovering 2024-08-17T18:20:16.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:15 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/194379440' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:20:16.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:15 smithi136 ceph-mon[115078]: pgmap v3270: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 6 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 136 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 802 op/s; 6637/6585 objects degraded (100.790%); 3807/6585 objects misplaced (57.813%); 7.8 MiB/s, 183 objects/s recovering 2024-08-17T18:20:16.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:15 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/194379440' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:20:16.227 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:16.248 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:16.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:16.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:16.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:15 smithi165 ceph-mon[105517]: pgmap v3270: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 6 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 136 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 802 op/s; 6637/6585 objects degraded (100.790%); 3807/6585 objects misplaced (57.813%); 7.8 MiB/s, 183 objects/s recovering 2024-08-17T18:20:16.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:15 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/194379440' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:20:16.349 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:16.370 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:16.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:16.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:16.472 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:16.493 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:16.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:16.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:16.595 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:16.618 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:16.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:16.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:16.720 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:16.742 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:16.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:16.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:16.844 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:16.868 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:16.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:16.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:16.970 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:16.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:16.992 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:16.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:17.093 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:17.115 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:17.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:17.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:17.121 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:16 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:20:17.122 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:16 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/194379440' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:20:17.122 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:16 smithi136 ceph-mon[109861]: osdmap e1688: 8 total, 8 up, 7 in 2024-08-17T18:20:17.122 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:16 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:20:17.122 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:16 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/194379440' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:20:17.122 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:16 smithi136 ceph-mon[115078]: osdmap e1688: 8 total, 8 up, 7 in 2024-08-17T18:20:17.217 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:17.239 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:17.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:17.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:17.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:16 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:20:17.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:16 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/194379440' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:20:17.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:16 smithi165 ceph-mon[105517]: osdmap e1688: 8 total, 8 up, 7 in 2024-08-17T18:20:17.340 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:17.363 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:17.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:17.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:17.371 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:20:17.465 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:17.486 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:17.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:17.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:17.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:17.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:17.612 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:17.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:17.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:20:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:20:17] "GET /metrics HTTP/1.1" 200 36772 "" "Prometheus/2.51.0" 2024-08-17T18:20:17.713 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:17.734 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:17.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:17.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:17.835 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:17.857 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:17.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:17.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:17.958 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:17.982 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:17.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:17.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:18.083 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:18.104 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:18.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:18.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:18.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:17 smithi136 ceph-mon[109861]: pgmap v3272: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 6 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 85 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 616 op/s; 6529/4116 objects degraded (158.625%); 3012/4116 objects misplaced (73.178%); 5.8 MiB/s, 129 objects/s recovering 2024-08-17T18:20:18.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:17 smithi136 ceph-mon[109861]: osdmap e1689: 8 total, 8 up, 7 in 2024-08-17T18:20:18.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:17 smithi136 ceph-mon[115078]: pgmap v3272: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 6 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 85 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 616 op/s; 6529/4116 objects degraded (158.625%); 3012/4116 objects misplaced (73.178%); 5.8 MiB/s, 129 objects/s recovering 2024-08-17T18:20:18.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:17 smithi136 ceph-mon[115078]: osdmap e1689: 8 total, 8 up, 7 in 2024-08-17T18:20:18.206 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:18.227 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:18.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:18.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:18.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:17 smithi165 ceph-mon[105517]: pgmap v3272: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 6 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 11 active+remapped+backfill_wait; 85 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 616 op/s; 6529/4116 objects degraded (158.625%); 3012/4116 objects misplaced (73.178%); 5.8 MiB/s, 129 objects/s recovering 2024-08-17T18:20:18.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:17 smithi165 ceph-mon[105517]: osdmap e1689: 8 total, 8 up, 7 in 2024-08-17T18:20:18.329 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:18.351 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:18.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:18.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:18.452 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:18.475 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:18.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:18.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:18.576 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:18.598 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:18.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:18.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:18.699 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:18.722 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:18.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:18.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:18.824 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:18.845 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:18.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:18.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:18.947 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:18.968 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:18.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:18.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:19.070 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:19.094 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:19.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:19.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:19.196 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:19.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:18 smithi136 ceph-mon[109861]: osdmap e1690: 8 total, 8 up, 7 in 2024-08-17T18:20:19.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:18 smithi136 ceph-mon[109861]: 57.12 continuing backfill to osd.3 from (1617'5062,1687'9618] MIN to 1687'9618 2024-08-17T18:20:19.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:18 smithi136 ceph-mon[115078]: osdmap e1690: 8 total, 8 up, 7 in 2024-08-17T18:20:19.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:18 smithi136 ceph-mon[115078]: 57.12 continuing backfill to osd.3 from (1617'5062,1687'9618] MIN to 1687'9618 2024-08-17T18:20:19.218 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:19.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:19.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:18 smithi165 ceph-mon[105517]: osdmap e1690: 8 total, 8 up, 7 in 2024-08-17T18:20:19.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:18 smithi165 ceph-mon[105517]: 57.12 continuing backfill to osd.3 from (1617'5062,1687'9618] MIN to 1687'9618 2024-08-17T18:20:19.320 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:19.342 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:19.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:19.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:19.444 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:19.468 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:19.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:19.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:19.570 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:19.594 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:19.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:19.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:19.696 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:19.719 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:19.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:19.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:19.821 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:19.843 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:19.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:19.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:19.945 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:19.966 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:19.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:19.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:20.024 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-08-17T18:20:20.024 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-08-17T18:20:20.068 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:20.091 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:20.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:20.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:20.193 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:20.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:19 smithi136 ceph-mon[115078]: pgmap v3275: 109 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 12 active+remapped+backfill_wait; 19 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 354 op/s; 5428/978 objects degraded (555.010%); 2095/978 objects misplaced (214.213%); 3.6 MiB/s, 259 objects/s recovering 2024-08-17T18:20:20.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:19 smithi136 ceph-mon[115078]: osdmap e1691: 8 total, 8 up, 7 in 2024-08-17T18:20:20.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:19 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5428/978 objects degraded (555.010%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:20:20.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:19 smithi136 ceph-mon[109861]: pgmap v3275: 109 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 12 active+remapped+backfill_wait; 19 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 354 op/s; 5428/978 objects degraded (555.010%); 2095/978 objects misplaced (214.213%); 3.6 MiB/s, 259 objects/s recovering 2024-08-17T18:20:20.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:19 smithi136 ceph-mon[109861]: osdmap e1691: 8 total, 8 up, 7 in 2024-08-17T18:20:20.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:19 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5428/978 objects degraded (555.010%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:20:20.215 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:20.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:20.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:19 smithi165 ceph-mon[105517]: pgmap v3275: 109 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 12 active+remapped+backfill_wait; 19 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 354 op/s; 5428/978 objects degraded (555.010%); 2095/978 objects misplaced (214.213%); 3.6 MiB/s, 259 objects/s recovering 2024-08-17T18:20:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:19 smithi165 ceph-mon[105517]: osdmap e1691: 8 total, 8 up, 7 in 2024-08-17T18:20:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:19 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5428/978 objects degraded (555.010%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:20:20.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:20.339 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:20.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:20.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:20.441 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:20.463 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:20.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:20.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:20.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:20.587 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:20.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:20.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:20.667 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:20:20.668 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:20:20.670 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:20:20.689 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:20.712 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:20.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:20.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:20.814 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:20.836 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:20.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:20.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:20.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:20.960 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:20.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:20.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:21.062 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:21.084 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:21.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:21.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:21.155 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:20 smithi136 ceph-mon[109861]: osdmap e1692: 8 total, 8 up, 7 in 2024-08-17T18:20:21.155 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:20 smithi136 ceph-mon[115078]: osdmap e1692: 8 total, 8 up, 7 in 2024-08-17T18:20:21.185 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:21.207 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:21.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:21.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:21.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:20 smithi165 ceph-mon[105517]: osdmap e1692: 8 total, 8 up, 7 in 2024-08-17T18:20:21.309 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:21.332 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:21.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:21.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:21.434 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:21.457 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:21.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:21.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:21.559 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:21.581 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:21.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:21.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:21.683 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:21.696 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:20:21.706 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:21.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:21.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:21.809 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:21.831 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:21.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:21.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:21.933 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:21.955 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:21.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:21.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:22.057 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:22.079 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:22.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:22.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:22.125 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:21 smithi136 ceph-mon[115078]: pgmap v3278: 109 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 13 active+remapped+backfill_wait; 19 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 316 op/s; 5412/978 objects degraded (553.374%); 2111/978 objects misplaced (215.849%); 3.6 MiB/s, 405 objects/s recovering 2024-08-17T18:20:22.125 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:20:22.125 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:22.125 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:22.125 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:21 smithi136 ceph-mon[109861]: pgmap v3278: 109 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 13 active+remapped+backfill_wait; 19 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 316 op/s; 5412/978 objects degraded (553.374%); 2111/978 objects misplaced (215.849%); 3.6 MiB/s, 405 objects/s recovering 2024-08-17T18:20:22.126 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:20:22.126 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:22.126 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:22.150 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:21 smithi165 ceph-mon[105517]: pgmap v3278: 109 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 85 active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 13 active+remapped+backfill_wait; 19 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 316 op/s; 5412/978 objects degraded (553.374%); 2111/978 objects misplaced (215.849%); 3.6 MiB/s, 405 objects/s recovering 2024-08-17T18:20:22.150 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:20:22.151 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:22.151 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:22.181 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:22.202 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:22.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:22.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:22.304 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:22.326 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:22.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:22.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:22.428 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:22.451 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:22.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:22.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:22.553 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:22.574 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:22.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:22.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:22.595 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:20:22.676 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:22.698 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:22.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:22.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:22.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:22.821 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:22.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:22.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:22.924 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:22.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:22.946 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:22.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:23.047 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:23.071 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:23.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:23.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:23.127 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:20:23.127 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:20:23.127 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:23.127 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:20:23.127 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:20:23.127 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:23.173 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:23.196 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:23.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:23.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:20:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:20:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:23.297 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:23.319 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:23.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:23.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:23.421 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:23.443 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:23.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:23.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:23.547 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:23.568 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:23.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:23.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:23.670 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:23.692 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:23.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:23.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:23.794 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:23.816 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:23.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:23.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:23.918 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:23.940 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:23.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:23.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:24.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:24.064 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:24.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:24.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:24.166 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:24.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:24.189 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:24.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:24.192 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:23 smithi136 ceph-mon[109861]: pgmap v3279: 109 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+remapped, 89 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 14 active+remapped+backfill_wait; 19 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 224 op/s; 4266/978 objects degraded (436.196%); 1894/978 objects misplaced (193.661%); 2.6 MiB/s, 716 objects/s recovering 2024-08-17T18:20:24.192 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:23 smithi136 ceph-mon[115078]: pgmap v3279: 109 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+remapped, 89 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 14 active+remapped+backfill_wait; 19 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 224 op/s; 4266/978 objects degraded (436.196%); 1894/978 objects misplaced (193.661%); 2.6 MiB/s, 716 objects/s recovering 2024-08-17T18:20:24.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:23 smithi165 ceph-mon[105517]: pgmap v3279: 109 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+remapped, 89 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 14 active+remapped+backfill_wait; 19 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 224 op/s; 4266/978 objects degraded (436.196%); 1894/978 objects misplaced (193.661%); 2.6 MiB/s, 716 objects/s recovering 2024-08-17T18:20:24.290 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:24.315 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:24.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:24.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:24.417 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:24.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:24.440 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:24.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:24.541 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:24.562 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:24.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:24.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:24.663 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:24.686 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:24.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:24.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:24.788 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:24.811 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:24.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:24.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:24.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:24.935 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:24.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:24.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:25.037 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:25.058 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:25.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:25.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:25.160 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:25.182 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:25.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:25.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:25.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:24 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 4266/978 objects degraded (436.196%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:20:25.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:24 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 4266/978 objects degraded (436.196%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:20:25.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:24 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 4266/978 objects degraded (436.196%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:20:25.284 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:25.306 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:25.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:25.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:25.408 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:25.432 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:25.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:25.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:25.533 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:25.556 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:25.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:25.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:25.658 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:25.682 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:25.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:25.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:25.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:25.806 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:25.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:25.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:25.908 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:25.950 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:25.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:25.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:26.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:26.076 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:26.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:26.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:26.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:26.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:25 smithi136 ceph-mon[109861]: pgmap v3280: 109 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+remapped, 89 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 14 active+remapped+backfill_wait; 19 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 4266/978 objects degraded (436.196%); 1894/978 objects misplaced (193.661%); 39 KiB/s, 390 objects/s recovering 2024-08-17T18:20:26.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:25 smithi136 ceph-mon[109861]: osdmap e1693: 8 total, 8 up, 7 in 2024-08-17T18:20:26.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:25 smithi136 ceph-mon[115078]: pgmap v3280: 109 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+remapped, 89 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 14 active+remapped+backfill_wait; 19 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 4266/978 objects degraded (436.196%); 1894/978 objects misplaced (193.661%); 39 KiB/s, 390 objects/s recovering 2024-08-17T18:20:26.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:25 smithi136 ceph-mon[115078]: osdmap e1693: 8 total, 8 up, 7 in 2024-08-17T18:20:26.199 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:26.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:26.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:26.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:25 smithi165 ceph-mon[105517]: pgmap v3280: 109 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+remapped, 89 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 14 active+remapped+backfill_wait; 19 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 4266/978 objects degraded (436.196%); 1894/978 objects misplaced (193.661%); 39 KiB/s, 390 objects/s recovering 2024-08-17T18:20:26.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:25 smithi165 ceph-mon[105517]: osdmap e1693: 8 total, 8 up, 7 in 2024-08-17T18:20:26.301 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:26.324 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:26.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:26.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:26.426 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:26.449 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:26.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:26.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:26.551 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:26.573 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:26.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:26.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:26.675 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:26.697 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:26.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:26.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:26.799 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:26.820 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:26.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:26.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:26.922 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:26.945 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:26.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:26.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:27.047 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:27.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:27.073 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:27.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:27.174 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:27.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:26 smithi136 ceph-mon[109861]: osdmap e1694: 8 total, 8 up, 7 in 2024-08-17T18:20:27.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:26 smithi136 ceph-mon[109861]: 57.0 scrub starts 2024-08-17T18:20:27.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:26 smithi136 ceph-mon[109861]: 57.0 scrub ok 2024-08-17T18:20:27.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:26 smithi136 ceph-mon[115078]: osdmap e1694: 8 total, 8 up, 7 in 2024-08-17T18:20:27.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:26 smithi136 ceph-mon[115078]: 57.0 scrub starts 2024-08-17T18:20:27.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:26 smithi136 ceph-mon[115078]: 57.0 scrub ok 2024-08-17T18:20:27.200 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:27.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:27.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:27.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:26 smithi165 ceph-mon[105517]: osdmap e1694: 8 total, 8 up, 7 in 2024-08-17T18:20:27.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:26 smithi165 ceph-mon[105517]: 57.0 scrub starts 2024-08-17T18:20:27.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:26 smithi165 ceph-mon[105517]: 57.0 scrub ok 2024-08-17T18:20:27.301 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:27.324 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:27.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:27.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:27.426 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:27.450 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:27.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:27.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:27.552 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:27.574 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:27.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:27.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:27.676 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:27.702 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:27.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:27.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:27.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:27.826 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:27.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:27.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:27.874 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:20:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:20:27] "GET /metrics HTTP/1.1" 200 36765 "" "Prometheus/2.51.0" 2024-08-17T18:20:27.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:27.949 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:27.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:27.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:28.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:28.072 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:28.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:28.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:28.127 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:27 smithi136 ceph-mon[109861]: pgmap v3283: 109 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+remapped, 90 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 13 active+remapped+backfill_wait; 19 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 4266/978 objects degraded (436.196%); 1894/978 objects misplaced (193.661%); 39 KiB/s, 390 objects/s recovering 2024-08-17T18:20:28.127 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:27 smithi136 ceph-mon[109861]: 1.3 scrub starts 2024-08-17T18:20:28.127 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:27 smithi136 ceph-mon[115078]: pgmap v3283: 109 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+remapped, 90 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 13 active+remapped+backfill_wait; 19 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 4266/978 objects degraded (436.196%); 1894/978 objects misplaced (193.661%); 39 KiB/s, 390 objects/s recovering 2024-08-17T18:20:28.128 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:27 smithi136 ceph-mon[115078]: 1.3 scrub starts 2024-08-17T18:20:28.174 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:28.196 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:28.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:28.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:28.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:27 smithi165 ceph-mon[105517]: pgmap v3283: 109 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+remapped, 90 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 13 active+remapped+backfill_wait; 19 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 4266/978 objects degraded (436.196%); 1894/978 objects misplaced (193.661%); 39 KiB/s, 390 objects/s recovering 2024-08-17T18:20:28.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:27 smithi165 ceph-mon[105517]: 1.3 scrub starts 2024-08-17T18:20:28.298 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:28.321 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:28.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:28.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:28.422 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:28.445 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:28.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:28.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:28.481 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:20:28.481 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:20:28.503 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:20:28.503 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:20:28.521 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:20:28.521 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:20:28.537 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:20:28.537 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:20:28.547 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:28.553 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:20:28.553 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:20:28.568 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:20:28.569 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:20:28.569 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:28.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:28.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:28.585 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:20:28.586 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:20:28.601 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:20:28.602 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:20:28.671 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:28.693 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:28.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:28.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:28.795 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:28.817 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:28.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:28.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:28.919 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:28.941 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:28.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:28.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:29.025 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:20:29.025 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T18:20:29.025 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-17T18:20:29.025 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-17T18:20:29.025 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:20:29.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:29.066 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:29.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:29.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:29.167 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:29.189 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:29.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:29.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:29.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:28 smithi136 ceph-mon[109861]: 57.1 scrub starts 2024-08-17T18:20:29.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:28 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4222356095' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:20:29.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:28 smithi136 ceph-mon[115078]: 57.1 scrub starts 2024-08-17T18:20:29.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:28 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4222356095' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:20:29.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:28 smithi165 ceph-mon[105517]: 57.1 scrub starts 2024-08-17T18:20:29.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:28 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4222356095' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:20:29.291 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:29.313 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:29.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:29.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:29.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:29.436 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:29.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:29.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:29.538 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:29.560 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:29.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:29.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:29.662 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:29.683 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:29.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:29.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:29.730 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-17T18:20:29.785 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:29.806 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:29.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:29.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:29.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:29.930 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:29.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:29.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:30.032 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:30.054 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:30.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:30.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:30.155 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:30.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:30.178 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:30.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:29 smithi136 ceph-mon[109861]: pgmap v3284: 109 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 100 active+clean, 4 active+remapped+backfill_wait; 17 MiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 3184/876 objects degraded (363.470%); 1815/876 objects misplaced (207.192%) 2024-08-17T18:20:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:29 smithi136 ceph-mon[109861]: 57.10 scrub starts 2024-08-17T18:20:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:29 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 3184/876 objects degraded (363.470%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:20:30.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:29 smithi136 ceph-mon[115078]: pgmap v3284: 109 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 100 active+clean, 4 active+remapped+backfill_wait; 17 MiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 3184/876 objects degraded (363.470%); 1815/876 objects misplaced (207.192%) 2024-08-17T18:20:30.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:29 smithi136 ceph-mon[115078]: 57.10 scrub starts 2024-08-17T18:20:30.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:29 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 3184/876 objects degraded (363.470%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:20:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:29 smithi165 ceph-mon[105517]: pgmap v3284: 109 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 100 active+clean, 4 active+remapped+backfill_wait; 17 MiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 3184/876 objects degraded (363.470%); 1815/876 objects misplaced (207.192%) 2024-08-17T18:20:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:29 smithi165 ceph-mon[105517]: 57.10 scrub starts 2024-08-17T18:20:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:29 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 3184/876 objects degraded (363.470%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:20:30.278 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:30.300 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:30.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:30.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:30.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:30.425 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:30.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:30.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:30.493 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:20:30.526 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:30.548 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:30.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:30.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:30.650 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:30.672 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:30.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:30.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:30.774 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:30.795 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:30.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:30.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:30.897 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:30.915 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a[109838]: 2024-08-17T18:20:30.695+0000 7f5b8e77d640 -1 mon.a@0(leader).osd e1694 definitely_dead 0 2024-08-17T18:20:30.915 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:30 smithi136 ceph-mon[109861]: 1.3 scrub starts 2024-08-17T18:20:30.923 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:30.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:30.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:30.925 INFO:teuthology.orchestra.run.smithi136.stderr:marked in osd.1. 2024-08-17T18:20:31.025 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:31.046 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:31.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:31.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:31.148 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:31.170 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:31.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:31.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:31.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:30 smithi136 ceph-mon[109861]: 1.0 scrub starts 2024-08-17T18:20:31.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:30 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1736556527' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-08-17T18:20:31.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:30 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-08-17T18:20:31.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:30 smithi136 ceph-mon[115078]: 1.3 scrub starts 2024-08-17T18:20:31.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:30 smithi136 ceph-mon[115078]: 1.0 scrub starts 2024-08-17T18:20:31.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:30 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1736556527' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-08-17T18:20:31.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:30 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-08-17T18:20:31.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:30 smithi165 ceph-mon[105517]: 1.3 scrub starts 2024-08-17T18:20:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:30 smithi165 ceph-mon[105517]: 1.0 scrub starts 2024-08-17T18:20:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:30 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1736556527' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-08-17T18:20:31.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:30 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-08-17T18:20:31.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:31.294 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:31.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:31.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:31.396 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:31.418 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:31.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:31.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:31.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:31.542 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:31.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:31.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:31.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:31.666 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:31.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:31.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:31.768 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:31.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:31.790 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:31.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:31.891 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:31.914 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:31.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:31.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:32.016 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:32.047 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:32.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:32.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:32.149 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:32.171 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:32.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:32.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:32.176 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:31 smithi136 ceph-mon[115078]: pgmap v3285: 109 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 99 active+clean, 4 active+remapped+backfill_wait; 17 MiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 3184/876 objects degraded (363.470%); 1815/876 objects misplaced (207.192%) 2024-08-17T18:20:32.177 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:31 smithi136 ceph-mon[115078]: 1.3 scrub starts 2024-08-17T18:20:32.177 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:31 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-08-17T18:20:32.177 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:31 smithi136 ceph-mon[115078]: osdmap e1695: 8 total, 8 up, 8 in 2024-08-17T18:20:32.177 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:31 smithi136 ceph-mon[109861]: pgmap v3285: 109 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 99 active+clean, 4 active+remapped+backfill_wait; 17 MiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 3184/876 objects degraded (363.470%); 1815/876 objects misplaced (207.192%) 2024-08-17T18:20:32.177 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:31 smithi136 ceph-mon[109861]: 1.3 scrub starts 2024-08-17T18:20:32.177 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:31 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-08-17T18:20:32.177 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:31 smithi136 ceph-mon[109861]: osdmap e1695: 8 total, 8 up, 8 in 2024-08-17T18:20:32.198 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-08-17T18:20:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:31 smithi165 ceph-mon[105517]: pgmap v3285: 109 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 99 active+clean, 4 active+remapped+backfill_wait; 17 MiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 3184/876 objects degraded (363.470%); 1815/876 objects misplaced (207.192%) 2024-08-17T18:20:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:31 smithi165 ceph-mon[105517]: 1.3 scrub starts 2024-08-17T18:20:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:31 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-08-17T18:20:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:31 smithi165 ceph-mon[105517]: osdmap e1695: 8 total, 8 up, 8 in 2024-08-17T18:20:32.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:32.294 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:32.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:32.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:32.396 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:32.419 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:32.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:32.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:32.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:32.543 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:32.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:32.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:32.645 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:32.668 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:32.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:32.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:32.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:32.792 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:32.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:32.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:32.893 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:32.902 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:20:32.917 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:32.918 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'unique_pool_4' removed 2024-08-17T18:20:32.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:32.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:33.020 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:33.059 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:33.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:33.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:33.158 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:20:33.161 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:33.181 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:32 smithi136 ceph-mon[109861]: osdmap e1696: 8 total, 8 up, 8 in 2024-08-17T18:20:33.181 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:32 smithi136 ceph-mon[109861]: 57.14 starting backfill to osd.1 from (0'0,0'0] MAX to 1687'9567 2024-08-17T18:20:33.181 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:32 smithi136 ceph-mon[109861]: 57.e starting backfill to osd.1 from (0'0,0'0] MAX to 1687'14548 2024-08-17T18:20:33.182 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:32 smithi136 ceph-mon[109861]: 57.14 continuing backfill to osd.3 from (1630'5626,1687'9567] MIN to 1687'9567 2024-08-17T18:20:33.182 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:32 smithi136 ceph-mon[109861]: 57.e continuing backfill to osd.7 from (1658'9918,1687'14548] MIN to 1687'14548 2024-08-17T18:20:33.182 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:32 smithi136 ceph-mon[109861]: pgmap v3288: 109 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+undersized+remapped, 16 peering, 7 remapped+peering, 2 active+recovery_wait+remapped, 76 active+clean, 3 active+remapped+backfill_wait; 17 MiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 503/876 objects degraded (57.420%) 2024-08-17T18:20:33.182 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:32 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1125106658' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T18:20:33.182 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:32 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1125106658' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T18:20:33.182 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:32 smithi136 ceph-mon[109861]: osdmap e1697: 8 total, 8 up, 8 in 2024-08-17T18:20:33.182 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:32 smithi136 ceph-mon[115078]: osdmap e1696: 8 total, 8 up, 8 in 2024-08-17T18:20:33.183 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:32 smithi136 ceph-mon[115078]: 57.14 starting backfill to osd.1 from (0'0,0'0] MAX to 1687'9567 2024-08-17T18:20:33.183 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:32 smithi136 ceph-mon[115078]: 57.e starting backfill to osd.1 from (0'0,0'0] MAX to 1687'14548 2024-08-17T18:20:33.183 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:32 smithi136 ceph-mon[115078]: 57.14 continuing backfill to osd.3 from (1630'5626,1687'9567] MIN to 1687'9567 2024-08-17T18:20:33.183 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:32 smithi136 ceph-mon[115078]: 57.e continuing backfill to osd.7 from (1658'9918,1687'14548] MIN to 1687'14548 2024-08-17T18:20:33.183 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:32 smithi136 ceph-mon[115078]: pgmap v3288: 109 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+undersized+remapped, 16 peering, 7 remapped+peering, 2 active+recovery_wait+remapped, 76 active+clean, 3 active+remapped+backfill_wait; 17 MiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 503/876 objects degraded (57.420%) 2024-08-17T18:20:33.183 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:32 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1125106658' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T18:20:33.183 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:32 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1125106658' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T18:20:33.183 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:32 smithi136 ceph-mon[115078]: osdmap e1697: 8 total, 8 up, 8 in 2024-08-17T18:20:33.203 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:33.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:33.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:32 smithi165 ceph-mon[105517]: osdmap e1696: 8 total, 8 up, 8 in 2024-08-17T18:20:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:32 smithi165 ceph-mon[105517]: 57.14 starting backfill to osd.1 from (0'0,0'0] MAX to 1687'9567 2024-08-17T18:20:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:32 smithi165 ceph-mon[105517]: 57.e starting backfill to osd.1 from (0'0,0'0] MAX to 1687'14548 2024-08-17T18:20:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:32 smithi165 ceph-mon[105517]: 57.14 continuing backfill to osd.3 from (1630'5626,1687'9567] MIN to 1687'9567 2024-08-17T18:20:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:32 smithi165 ceph-mon[105517]: 57.e continuing backfill to osd.7 from (1658'9918,1687'14548] MIN to 1687'14548 2024-08-17T18:20:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:32 smithi165 ceph-mon[105517]: pgmap v3288: 109 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+undersized+remapped, 16 peering, 7 remapped+peering, 2 active+recovery_wait+remapped, 76 active+clean, 3 active+remapped+backfill_wait; 17 MiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 503/876 objects degraded (57.420%) 2024-08-17T18:20:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:32 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1125106658' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T18:20:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:32 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1125106658' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T18:20:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:32 smithi165 ceph-mon[105517]: osdmap e1697: 8 total, 8 up, 8 in 2024-08-17T18:20:33.305 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:33.326 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:33.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:33.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:33.428 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:33.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:33.450 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:33.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:33.552 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:33.572 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:33.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:33.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:33.674 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-17T18:20:33.675 INFO:tasks.radosbench:Beginning radosbench... 2024-08-17T18:20:33.675 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-08-17T18:20:33.676 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool create unique_pool_5 16 2024-08-17T18:20:33.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:33.697 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:33.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:33.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:33.799 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:33.822 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:33.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:33.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:33.924 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:33.946 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:33.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:33.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:34.048 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:34.071 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:34.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:34.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:34.173 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:34.194 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:34.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:34.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:34.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:33 smithi136 ceph-mon[109861]: Health check failed: Reduced data availability: 7 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-08-17T18:20:34.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:33 smithi136 ceph-mon[109861]: osdmap e1698: 8 total, 8 up, 8 in 2024-08-17T18:20:34.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:33 smithi136 ceph-mon[115078]: Health check failed: Reduced data availability: 7 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-08-17T18:20:34.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:33 smithi136 ceph-mon[115078]: osdmap e1698: 8 total, 8 up, 8 in 2024-08-17T18:20:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:33 smithi165 ceph-mon[105517]: Health check failed: Reduced data availability: 7 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-08-17T18:20:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:33 smithi165 ceph-mon[105517]: osdmap e1698: 8 total, 8 up, 8 in 2024-08-17T18:20:34.297 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:34.324 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:34.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:34.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:34.426 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:34.451 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:34.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:34.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:34.553 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:34.580 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:34.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:34.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:34.682 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:34.708 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:34.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:34.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:34.810 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:34.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:34.833 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:34.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:34.934 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:34.957 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:34.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:34.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:35.059 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:35.080 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:35.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:35.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:35.182 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:35.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:34 smithi136 ceph-mon[115078]: pgmap v3291: 83 pgs: 1 active+recovery_wait+degraded, 1 active+undersized+remapped, 16 peering, 2 remapped+peering, 63 active+clean; 687 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1/78 objects degraded (1.282%) 2024-08-17T18:20:35.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:34 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 503/876 objects degraded (57.420%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:20:35.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:34 smithi136 ceph-mon[115078]: osdmap e1699: 8 total, 8 up, 8 in 2024-08-17T18:20:35.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:34 smithi136 ceph-mon[109861]: pgmap v3291: 83 pgs: 1 active+recovery_wait+degraded, 1 active+undersized+remapped, 16 peering, 2 remapped+peering, 63 active+clean; 687 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1/78 objects degraded (1.282%) 2024-08-17T18:20:35.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:34 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 503/876 objects degraded (57.420%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:20:35.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:34 smithi136 ceph-mon[109861]: osdmap e1699: 8 total, 8 up, 8 in 2024-08-17T18:20:35.204 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:35.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:35.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:34 smithi165 ceph-mon[105517]: pgmap v3291: 83 pgs: 1 active+recovery_wait+degraded, 1 active+undersized+remapped, 16 peering, 2 remapped+peering, 63 active+clean; 687 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1/78 objects degraded (1.282%) 2024-08-17T18:20:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:34 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 503/876 objects degraded (57.420%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:20:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:34 smithi165 ceph-mon[105517]: osdmap e1699: 8 total, 8 up, 8 in 2024-08-17T18:20:35.306 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:35.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:35.328 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:35.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:35.429 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:35.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:35.452 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:35.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:35.554 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:35.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:35.576 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:35.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:35.677 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:35.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:35.704 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:35.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:35.805 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:35.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:35.829 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:35.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:35.891 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:20:35.929 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:35.951 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:35.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:35.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:35.961 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:20:36.053 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:36.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:36.077 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:36.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:36.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:36.185 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:35 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1090551899' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:20:36.185 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:35 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:20:36.186 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:35 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1090551899' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:20:36.186 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:35 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:20:36.204 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:36.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:36.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:35 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1090551899' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:20:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:35 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:20:36.307 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:36.331 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:36.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:36.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:36.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:36.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:36.461 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:36.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:36.562 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:36.584 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:36.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:36.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:36.686 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:36.708 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:36.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:36.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:36.810 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:36.831 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:36.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:36.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:36.933 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:36.960 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:36.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:36.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:37.062 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:37.085 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:37.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:37.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:37.187 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:37.198 INFO:tasks.thrashosds.thrasher:in_osds: [2, 4, 6, 0, 5, 7, 3, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:20:37.199 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:20:37.199 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-17T18:20:37.199 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 1.000000 2024-08-17T18:20:37.199 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 1 1 2024-08-17T18:20:37.202 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:36 smithi136 ceph-mon[109861]: 1.3 scrub starts 2024-08-17T18:20:37.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:36 smithi136 ceph-mon[109861]: 1.3 scrub ok 2024-08-17T18:20:37.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:36 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:20:37.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:36 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:20:37.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:36 smithi136 ceph-mon[109861]: osdmap e1700: 8 total, 8 up, 8 in 2024-08-17T18:20:37.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:37.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:20:37.203 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:36 smithi136 ceph-mon[109861]: pgmap v3294: 83 pgs: 1 active+recovery_wait+degraded, 1 active+undersized+remapped, 1 remapped+peering, 80 active+clean; 634 KiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 1/78 objects degraded (1.282%) 2024-08-17T18:20:37.203 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:36 smithi136 ceph-mon[115078]: 1.3 scrub starts 2024-08-17T18:20:37.204 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:36 smithi136 ceph-mon[115078]: 1.3 scrub ok 2024-08-17T18:20:37.204 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:36 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:20:37.204 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:36 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:20:37.204 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:36 smithi136 ceph-mon[115078]: osdmap e1700: 8 total, 8 up, 8 in 2024-08-17T18:20:37.204 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:37.204 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:20:37.204 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:36 smithi136 ceph-mon[115078]: pgmap v3294: 83 pgs: 1 active+recovery_wait+degraded, 1 active+undersized+remapped, 1 remapped+peering, 80 active+clean; 634 KiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 1/78 objects degraded (1.282%) 2024-08-17T18:20:37.209 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:20:37.209 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:37.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:37.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:36 smithi165 ceph-mon[105517]: 1.3 scrub starts 2024-08-17T18:20:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:36 smithi165 ceph-mon[105517]: 1.3 scrub ok 2024-08-17T18:20:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:36 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:20:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:36 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:20:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:36 smithi165 ceph-mon[105517]: osdmap e1700: 8 total, 8 up, 8 in 2024-08-17T18:20:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:20:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:36 smithi165 ceph-mon[105517]: pgmap v3294: 83 pgs: 1 active+recovery_wait+degraded, 1 active+undersized+remapped, 1 remapped+peering, 80 active+clean; 634 KiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 1/78 objects degraded (1.282%) 2024-08-17T18:20:37.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:37.338 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:37.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:37.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:37.441 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:37.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:37.465 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:37.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:37.566 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:37.593 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:37.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:37.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:37.641 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:20:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:20:37] "GET /metrics HTTP/1.1" 200 34343 "" "Prometheus/2.51.0" 2024-08-17T18:20:37.694 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:37.716 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:37.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:37.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:37.818 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:37.840 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:37.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:37.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:37.942 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:37.964 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:37.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:37.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:38.066 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:38.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:38.093 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:38.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:38.194 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:38.216 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:38.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:38.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:38.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:37 smithi165 ceph-mon[105517]: 1.0 scrub starts 2024-08-17T18:20:38.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:37 smithi165 ceph-mon[105517]: 1.0 scrub ok 2024-08-17T18:20:38.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:37 smithi165 ceph-mon[105517]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 7 pgs peering) 2024-08-17T18:20:38.265 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:37 smithi136 ceph-mon[109861]: 1.0 scrub starts 2024-08-17T18:20:38.265 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:37 smithi136 ceph-mon[109861]: 1.0 scrub ok 2024-08-17T18:20:38.266 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:37 smithi136 ceph-mon[109861]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 7 pgs peering) 2024-08-17T18:20:38.266 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:37 smithi136 ceph-mon[115078]: 1.0 scrub starts 2024-08-17T18:20:38.266 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:37 smithi136 ceph-mon[115078]: 1.0 scrub ok 2024-08-17T18:20:38.266 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:37 smithi136 ceph-mon[115078]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 7 pgs peering) 2024-08-17T18:20:38.270 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:20:38.270 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-17T18:20:38.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-17T18:20:38.270 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:20:38.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.270 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:219427 56.12 56:48959621:::benchmark_data_smithi136_356456_object92723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1577)", 2024-08-17T18:20:38.271 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:52.752468+0000", 2024-08-17T18:20:38.271 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 465.51164633399998, 2024-08-17T18:20:38.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0230039949999998, 2024-08-17T18:20:38.271 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:20:38.271 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:20:38.271 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:20:38.271 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:20:38.271 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:20:38.271 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:20:38.272 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 219427 2024-08-17T18:20:38.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.272 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:20:38.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:20:38.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.752468+0000", 2024-08-17T18:20:38.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:20:38.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.752468+0000", 2024-08-17T18:20:38.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:20:38.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.752469+0000", 2024-08-17T18:20:38.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5000000000000001e-07 2024-08-17T18:20:38.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:20:38.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.752473+0000", 2024-08-17T18:20:38.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4859999999999999e-06 2024-08-17T18:20:38.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:20:38.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.752488+0000", 2024-08-17T18:20:38.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4704e-05 2024-08-17T18:20:38.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:20:38.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.752495+0000", 2024-08-17T18:20:38.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.605e-06 2024-08-17T18:20:38.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.752559+0000", 2024-08-17T18:20:38.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3542999999999998e-05 2024-08-17T18:20:38.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.752569+0000", 2024-08-17T18:20:38.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9059999999999997e-06 2024-08-17T18:20:38.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:53.699132+0000", 2024-08-17T18:20:38.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.94656289999999998 2024-08-17T18:20:38.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:53.699144+0000", 2024-08-17T18:20:38.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2337999999999999e-05 2024-08-17T18:20:38.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.771308+0000", 2024-08-17T18:20:38.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.072163816 2024-08-17T18:20:38.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:20:38.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.771433+0000", 2024-08-17T18:20:38.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012502600000000001 2024-08-17T18:20:38.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5", 2024-08-17T18:20:38.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.771708+0000", 2024-08-17T18:20:38.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000274679 2024-08-17T18:20:38.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:20:38.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775059+0000", 2024-08-17T18:20:38.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033514619999999999 2024-08-17T18:20:38.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775097+0000", 2024-08-17T18:20:38.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7625e-05 2024-08-17T18:20:38.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775256+0000", 2024-08-17T18:20:38.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015889699999999999 2024-08-17T18:20:38.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775310+0000", 2024-08-17T18:20:38.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4381e-05 2024-08-17T18:20:38.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:20:38.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775350+0000", 2024-08-17T18:20:38.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9864999999999999e-05 2024-08-17T18:20:38.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:20:38.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775472+0000", 2024-08-17T18:20:38.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012181200000000001 2024-08-17T18:20:38.280 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.280 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:20:38.280 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.280 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:219447 56.12 56:4a3a3ee4:::benchmark_data_smithi136_356456_object92743:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1577)", 2024-08-17T18:20:38.280 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:52.767811+0000", 2024-08-17T18:20:38.280 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 465.49630254599998, 2024-08-17T18:20:38.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.007993371, 2024-08-17T18:20:38.280 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:20:38.280 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:20:38.280 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:20:38.280 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:20:38.281 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:20:38.281 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:20:38.281 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 219447 2024-08-17T18:20:38.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.281 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:20:38.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:20:38.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.767811+0000", 2024-08-17T18:20:38.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:20:38.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.767811+0000", 2024-08-17T18:20:38.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:20:38.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.767812+0000", 2024-08-17T18:20:38.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7100000000000001e-07 2024-08-17T18:20:38.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:20:38.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.767816+0000", 2024-08-17T18:20:38.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.563e-06 2024-08-17T18:20:38.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:20:38.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.767824+0000", 2024-08-17T18:20:38.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5539999999999998e-06 2024-08-17T18:20:38.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:20:38.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.767829+0000", 2024-08-17T18:20:38.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0610000000000003e-06 2024-08-17T18:20:38.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.767880+0000", 2024-08-17T18:20:38.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0963e-05 2024-08-17T18:20:38.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.767887+0000", 2024-08-17T18:20:38.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3589999999999996e-06 2024-08-17T18:20:38.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:53.699170+0000", 2024-08-17T18:20:38.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93128367700000003 2024-08-17T18:20:38.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:53.699179+0000", 2024-08-17T18:20:38.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4589999999999998e-06 2024-08-17T18:20:38.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.773078+0000", 2024-08-17T18:20:38.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0738989130000001 2024-08-17T18:20:38.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:20:38.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.773156+0000", 2024-08-17T18:20:38.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8424000000000007e-05 2024-08-17T18:20:38.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5", 2024-08-17T18:20:38.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.773294+0000", 2024-08-17T18:20:38.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000138367 2024-08-17T18:20:38.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:20:38.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775205+0000", 2024-08-17T18:20:38.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0019106800000000001 2024-08-17T18:20:38.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775516+0000", 2024-08-17T18:20:38.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00031132399999999998 2024-08-17T18:20:38.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775654+0000", 2024-08-17T18:20:38.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013750399999999999 2024-08-17T18:20:38.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775688+0000", 2024-08-17T18:20:38.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4180000000000001e-05 2024-08-17T18:20:38.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:20:38.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775731+0000", 2024-08-17T18:20:38.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2864000000000002e-05 2024-08-17T18:20:38.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:20:38.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775805+0000", 2024-08-17T18:20:38.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3807999999999995e-05 2024-08-17T18:20:38.288 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.288 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:20:38.288 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.288 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:219454 56.12 56:496432b9:::benchmark_data_smithi136_356456_object92750:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1577)", 2024-08-17T18:20:38.288 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:52.771621+0000", 2024-08-17T18:20:38.288 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 465.49249272399999, 2024-08-17T18:20:38.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.004395073, 2024-08-17T18:20:38.289 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:20:38.289 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:20:38.289 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:20:38.289 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:20:38.289 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:20:38.289 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:20:38.289 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 219454 2024-08-17T18:20:38.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.289 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:20:38.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:20:38.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.771621+0000", 2024-08-17T18:20:38.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:20:38.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.771621+0000", 2024-08-17T18:20:38.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:20:38.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.771622+0000", 2024-08-17T18:20:38.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4999999999999998e-07 2024-08-17T18:20:38.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:20:38.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.771624+0000", 2024-08-17T18:20:38.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6359999999999998e-06 2024-08-17T18:20:38.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:20:38.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.771630+0000", 2024-08-17T18:20:38.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4790000000000004e-06 2024-08-17T18:20:38.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:20:38.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.771634+0000", 2024-08-17T18:20:38.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8e-06 2024-08-17T18:20:38.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.771687+0000", 2024-08-17T18:20:38.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3449000000000003e-05 2024-08-17T18:20:38.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.771707+0000", 2024-08-17T18:20:38.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9898000000000002e-05 2024-08-17T18:20:38.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:53.699204+0000", 2024-08-17T18:20:38.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92749685800000004 2024-08-17T18:20:38.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:53.699212+0000", 2024-08-17T18:20:38.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8760000000000005e-06 2024-08-17T18:20:38.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.774136+0000", 2024-08-17T18:20:38.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0749248140000001 2024-08-17T18:20:38.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:20:38.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.774205+0000", 2024-08-17T18:20:38.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8476e-05 2024-08-17T18:20:38.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5", 2024-08-17T18:20:38.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.774378+0000", 2024-08-17T18:20:38.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017333999999999999 2024-08-17T18:20:38.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:20:38.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775483+0000", 2024-08-17T18:20:38.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001105036 2024-08-17T18:20:38.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775857+0000", 2024-08-17T18:20:38.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00037334500000000002 2024-08-17T18:20:38.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775895+0000", 2024-08-17T18:20:38.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8600999999999999e-05 2024-08-17T18:20:38.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775930+0000", 2024-08-17T18:20:38.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5145000000000003e-05 2024-08-17T18:20:38.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:20:38.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.775962+0000", 2024-08-17T18:20:38.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1380000000000001e-05 2024-08-17T18:20:38.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:20:38.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.776016+0000", 2024-08-17T18:20:38.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4490000000000002e-05 2024-08-17T18:20:38.296 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.296 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:20:38.296 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.297 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:219488 56.12 56:4d9fe1f5:::benchmark_data_smithi136_356456_object92784:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1577)", 2024-08-17T18:20:38.297 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:52.814105+0000", 2024-08-17T18:20:38.297 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 465.45000891799998, 2024-08-17T18:20:38.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.974696437, 2024-08-17T18:20:38.297 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:20:38.297 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:20:38.297 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:20:38.297 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:20:38.297 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:20:38.297 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:20:38.297 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 219488 2024-08-17T18:20:38.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.297 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:20:38.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:20:38.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.814105+0000", 2024-08-17T18:20:38.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:20:38.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.814105+0000", 2024-08-17T18:20:38.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:20:38.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.814106+0000", 2024-08-17T18:20:38.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.308e-06 2024-08-17T18:20:38.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:20:38.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.814111+0000", 2024-08-17T18:20:38.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7430000000000002e-06 2024-08-17T18:20:38.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:20:38.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.814128+0000", 2024-08-17T18:20:38.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6473999999999998e-05 2024-08-17T18:20:38.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:20:38.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.814136+0000", 2024-08-17T18:20:38.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6319999999999997e-06 2024-08-17T18:20:38.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.814199+0000", 2024-08-17T18:20:38.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2967999999999997e-05 2024-08-17T18:20:38.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.814206+0000", 2024-08-17T18:20:38.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2339999999999997e-06 2024-08-17T18:20:38.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:53.699945+0000", 2024-08-17T18:20:38.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.88573835199999995 2024-08-17T18:20:38.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:53.699955+0000", 2024-08-17T18:20:38.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0208e-05 2024-08-17T18:20:38.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.782619+0000", 2024-08-17T18:20:38.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.082663836 2024-08-17T18:20:38.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:20:38.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.782742+0000", 2024-08-17T18:20:38.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000122915 2024-08-17T18:20:38.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5", 2024-08-17T18:20:38.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.783001+0000", 2024-08-17T18:20:38.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025919199999999999 2024-08-17T18:20:38.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:20:38.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.788503+0000", 2024-08-17T18:20:38.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0055020629999999997 2024-08-17T18:20:38.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.788553+0000", 2024-08-17T18:20:38.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9591e-05 2024-08-17T18:20:38.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.788616+0000", 2024-08-17T18:20:38.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3733999999999997e-05 2024-08-17T18:20:38.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.788665+0000", 2024-08-17T18:20:38.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8930000000000001e-05 2024-08-17T18:20:38.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:20:38.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.788710+0000", 2024-08-17T18:20:38.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4722000000000003e-05 2024-08-17T18:20:38.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:20:38.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.788801+0000", 2024-08-17T18:20:38.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1534999999999997e-05 2024-08-17T18:20:38.304 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.304 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:20:38.304 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.305 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:219495 56.12 56:4a7b6ab5:::benchmark_data_smithi136_356456_object92791:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1577)", 2024-08-17T18:20:38.305 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:52.822737+0000", 2024-08-17T18:20:38.305 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 465.441376945, 2024-08-17T18:20:38.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9663730479999999, 2024-08-17T18:20:38.305 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:20:38.305 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:20:38.305 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:20:38.305 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:20:38.305 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:20:38.305 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:20:38.305 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 219495 2024-08-17T18:20:38.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.306 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:20:38.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:20:38.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.822737+0000", 2024-08-17T18:20:38.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:20:38.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.822737+0000", 2024-08-17T18:20:38.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:20:38.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.822738+0000", 2024-08-17T18:20:38.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.35e-06 2024-08-17T18:20:38.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:20:38.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.822743+0000", 2024-08-17T18:20:38.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4410000000000003e-06 2024-08-17T18:20:38.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:20:38.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.822755+0000", 2024-08-17T18:20:38.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2031000000000001e-05 2024-08-17T18:20:38.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:20:38.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.822762+0000", 2024-08-17T18:20:38.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5630000000000003e-06 2024-08-17T18:20:38.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.822820+0000", 2024-08-17T18:20:38.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7855000000000001e-05 2024-08-17T18:20:38.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.822830+0000", 2024-08-17T18:20:38.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5119999999999997e-06 2024-08-17T18:20:38.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:53.700051+0000", 2024-08-17T18:20:38.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.87722086300000002 2024-08-17T18:20:38.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:53.700057+0000", 2024-08-17T18:20:38.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1600000000000003e-06 2024-08-17T18:20:38.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.784269+0000", 2024-08-17T18:20:38.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0842123809999999 2024-08-17T18:20:38.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:20:38.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.784385+0000", 2024-08-17T18:20:38.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116039 2024-08-17T18:20:38.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5", 2024-08-17T18:20:38.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.784611+0000", 2024-08-17T18:20:38.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022579300000000001 2024-08-17T18:20:38.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:20:38.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.788636+0000", 2024-08-17T18:20:38.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040248389999999997 2024-08-17T18:20:38.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.788855+0000", 2024-08-17T18:20:38.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021960799999999999 2024-08-17T18:20:38.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.788893+0000", 2024-08-17T18:20:38.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7834999999999999e-05 2024-08-17T18:20:38.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.788935+0000", 2024-08-17T18:20:38.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1292999999999999e-05 2024-08-17T18:20:38.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:20:38.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.788991+0000", 2024-08-17T18:20:38.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6440999999999997e-05 2024-08-17T18:20:38.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:20:38.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.789110+0000", 2024-08-17T18:20:38.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011904399999999999 2024-08-17T18:20:38.312 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.312 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:20:38.313 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.313 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:219594 56.12 56:4c578227:::benchmark_data_smithi136_356456_object92890:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1577)", 2024-08-17T18:20:38.313 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:52.898347+0000", 2024-08-17T18:20:38.314 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 465.36576657099999, 2024-08-17T18:20:38.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.89111677, 2024-08-17T18:20:38.314 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:20:38.314 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:20:38.314 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:20:38.315 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:20:38.315 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:20:38.315 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:20:38.315 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 219594 2024-08-17T18:20:38.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.315 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:20:38.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:20:38.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.898347+0000", 2024-08-17T18:20:38.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:20:38.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.898347+0000", 2024-08-17T18:20:38.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:20:38.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.898349+0000", 2024-08-17T18:20:38.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.621e-06 2024-08-17T18:20:38.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:20:38.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.898354+0000", 2024-08-17T18:20:38.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2079999999999999e-06 2024-08-17T18:20:38.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:20:38.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.898365+0000", 2024-08-17T18:20:38.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0403e-05 2024-08-17T18:20:38.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:20:38.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.898372+0000", 2024-08-17T18:20:38.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0419999999999997e-06 2024-08-17T18:20:38.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.898432+0000", 2024-08-17T18:20:38.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0794000000000001e-05 2024-08-17T18:20:38.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:52.898444+0000", 2024-08-17T18:20:38.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1504e-05 2024-08-17T18:20:38.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:53.700084+0000", 2024-08-17T18:20:38.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.80163971000000001 2024-08-17T18:20:38.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:53.700089+0000", 2024-08-17T18:20:38.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.852e-06 2024-08-17T18:20:38.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.785913+0000", 2024-08-17T18:20:38.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.085824007 2024-08-17T18:20:38.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:20:38.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.786032+0000", 2024-08-17T18:20:38.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011839499999999999 2024-08-17T18:20:38.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,4,5", 2024-08-17T18:20:38.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.786224+0000", 2024-08-17T18:20:38.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019196600000000001 2024-08-17T18:20:38.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:20:38.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.788644+0000", 2024-08-17T18:20:38.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0024205310000000001 2024-08-17T18:20:38.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.789164+0000", 2024-08-17T18:20:38.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00051947500000000004 2024-08-17T18:20:38.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.789209+0000", 2024-08-17T18:20:38.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4965000000000001e-05 2024-08-17T18:20:38.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.789277+0000", 2024-08-17T18:20:38.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8555999999999999e-05 2024-08-17T18:20:38.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:20:38.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.789335+0000", 2024-08-17T18:20:38.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7323e-05 2024-08-17T18:20:38.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:20:38.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:54.789464+0000", 2024-08-17T18:20:38.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012941800000000001 2024-08-17T18:20:38.322 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.322 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:20:38.322 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.323 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:91775 57.12 57:4c5b5116:::benchmark_data_smithi136_389235_object91774:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:20:38.323 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.694206+0000", 2024-08-17T18:20:38.323 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 337.56990839700001, 2024-08-17T18:20:38.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1991839390000001, 2024-08-17T18:20:38.323 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:20:38.323 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:20:38.323 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:20:38.323 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:20:38.323 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:20:38.323 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:20:38.323 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91775 2024-08-17T18:20:38.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.323 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:20:38.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:20:38.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.694206+0000", 2024-08-17T18:20:38.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:20:38.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.694206+0000", 2024-08-17T18:20:38.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:20:38.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.694206+0000", 2024-08-17T18:20:38.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6400000000000001e-07 2024-08-17T18:20:38.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:20:38.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.694222+0000", 2024-08-17T18:20:38.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5503999999999999e-05 2024-08-17T18:20:38.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:20:38.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.694230+0000", 2024-08-17T18:20:38.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2660000000000001e-06 2024-08-17T18:20:38.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:20:38.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.694235+0000", 2024-08-17T18:20:38.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5940000000000004e-06 2024-08-17T18:20:38.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.694293+0000", 2024-08-17T18:20:38.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8186999999999997e-05 2024-08-17T18:20:38.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.694303+0000", 2024-08-17T18:20:38.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0059e-05 2024-08-17T18:20:38.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.646421+0000", 2024-08-17T18:20:38.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.95211796000000004 2024-08-17T18:20:38.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.646435+0000", 2024-08-17T18:20:38.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3908e-05 2024-08-17T18:20:38.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.887386+0000", 2024-08-17T18:20:38.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.240951269 2024-08-17T18:20:38.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:20:38.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.887571+0000", 2024-08-17T18:20:38.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018437799999999999 2024-08-17T18:20:38.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:20:38.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.887823+0000", 2024-08-17T18:20:38.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025225300000000003 2024-08-17T18:20:38.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:20:38.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.892158+0000", 2024-08-17T18:20:38.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.004335401 2024-08-17T18:20:38.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.892215+0000", 2024-08-17T18:20:38.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6533000000000002e-05 2024-08-17T18:20:38.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.892345+0000", 2024-08-17T18:20:38.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012994600000000001 2024-08-17T18:20:38.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:20:38.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.892406+0000", 2024-08-17T18:20:38.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1282e-05 2024-08-17T18:20:38.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:20:38.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.893389+0000", 2024-08-17T18:20:38.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00098353500000000001 2024-08-17T18:20:38.330 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.330 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:20:38.330 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.330 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:91798 57.12 57:4d0d602d:::benchmark_data_smithi136_389235_object91797:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:20:38.331 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.714863+0000", 2024-08-17T18:20:38.331 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 337.54925079499998, 2024-08-17T18:20:38.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.186937039, 2024-08-17T18:20:38.331 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:20:38.331 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:20:38.331 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:20:38.331 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:20:38.331 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:20:38.331 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:20:38.331 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91798 2024-08-17T18:20:38.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.331 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:20:38.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:20:38.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.714863+0000", 2024-08-17T18:20:38.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:20:38.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.714863+0000", 2024-08-17T18:20:38.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:20:38.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.714864+0000", 2024-08-17T18:20:38.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5099999999999998e-07 2024-08-17T18:20:38.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:20:38.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.714889+0000", 2024-08-17T18:20:38.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5029e-05 2024-08-17T18:20:38.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:20:38.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.714897+0000", 2024-08-17T18:20:38.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2570000000000007e-06 2024-08-17T18:20:38.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:20:38.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.714903+0000", 2024-08-17T18:20:38.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9390000000000002e-06 2024-08-17T18:20:38.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.714971+0000", 2024-08-17T18:20:38.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7811999999999994e-05 2024-08-17T18:20:38.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.714982+0000", 2024-08-17T18:20:38.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0814e-05 2024-08-17T18:20:38.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.646899+0000", 2024-08-17T18:20:38.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93191730399999995 2024-08-17T18:20:38.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.646905+0000", 2024-08-17T18:20:38.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6779999999999997e-06 2024-08-17T18:20:38.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.889432+0000", 2024-08-17T18:20:38.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.242527468 2024-08-17T18:20:38.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:20:38.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.889612+0000", 2024-08-17T18:20:38.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000179722 2024-08-17T18:20:38.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:20:38.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.889890+0000", 2024-08-17T18:20:38.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027764899999999998 2024-08-17T18:20:38.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.892278+0000", 2024-08-17T18:20:38.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023882690000000002 2024-08-17T18:20:38.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:20:38.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.892292+0000", 2024-08-17T18:20:38.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4258e-05 2024-08-17T18:20:38.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.900664+0000", 2024-08-17T18:20:38.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0083714710000000001 2024-08-17T18:20:38.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:20:38.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.900720+0000", 2024-08-17T18:20:38.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6434000000000002e-05 2024-08-17T18:20:38.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:20:38.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.901800+0000", 2024-08-17T18:20:38.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001079984 2024-08-17T18:20:38.338 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.338 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:20:38.338 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.338 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:91831 57.12 57:49f84744:::benchmark_data_smithi136_389235_object91830:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:20:38.338 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.733890+0000", 2024-08-17T18:20:38.338 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 337.53022389799997, 2024-08-17T18:20:38.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1688886919999999, 2024-08-17T18:20:38.339 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:20:38.339 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:20:38.339 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:20:38.339 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:20:38.339 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:20:38.339 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:20:38.339 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91831 2024-08-17T18:20:38.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.339 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:20:38.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.339 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:20:38.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.733890+0000", 2024-08-17T18:20:38.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:20:38.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.733890+0000", 2024-08-17T18:20:38.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:20:38.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.733891+0000", 2024-08-17T18:20:38.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7400000000000002e-07 2024-08-17T18:20:38.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:20:38.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.733910+0000", 2024-08-17T18:20:38.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9232000000000002e-05 2024-08-17T18:20:38.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.341 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:20:38.341 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.733921+0000", 2024-08-17T18:20:38.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0787e-05 2024-08-17T18:20:38.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.341 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:20:38.341 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.733928+0000", 2024-08-17T18:20:38.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4320000000000002e-06 2024-08-17T18:20:38.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.341 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.341 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.733984+0000", 2024-08-17T18:20:38.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6233000000000002e-05 2024-08-17T18:20:38.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734004+0000", 2024-08-17T18:20:38.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9307e-05 2024-08-17T18:20:38.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.646939+0000", 2024-08-17T18:20:38.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.91293501099999996 2024-08-17T18:20:38.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.343 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.646948+0000", 2024-08-17T18:20:38.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9709999999999993e-06 2024-08-17T18:20:38.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.343 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.343 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.890760+0000", 2024-08-17T18:20:38.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2438118359999999 2024-08-17T18:20:38.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.343 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:20:38.343 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.890914+0000", 2024-08-17T18:20:38.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015469099999999999 2024-08-17T18:20:38.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.344 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:20:38.344 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.891161+0000", 2024-08-17T18:20:38.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024684400000000002 2024-08-17T18:20:38.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.344 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:20:38.344 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.894974+0000", 2024-08-17T18:20:38.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038126700000000002 2024-08-17T18:20:38.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.344 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.344 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.895037+0000", 2024-08-17T18:20:38.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3083000000000006e-05 2024-08-17T18:20:38.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.901890+0000", 2024-08-17T18:20:38.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0068527170000000004 2024-08-17T18:20:38.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:20:38.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.901927+0000", 2024-08-17T18:20:38.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7088999999999997e-05 2024-08-17T18:20:38.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:20:38.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.902779+0000", 2024-08-17T18:20:38.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00085201500000000002 2024-08-17T18:20:38.346 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.346 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:20:38.346 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.346 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:91833 57.12 57:4fab0a31:::benchmark_data_smithi136_389235_object91832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:20:38.346 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.734763+0000", 2024-08-17T18:20:38.346 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 337.52935069599999, 2024-08-17T18:20:38.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1724328439999998, 2024-08-17T18:20:38.346 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:20:38.346 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:20:38.347 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:20:38.347 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:20:38.347 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:20:38.347 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:20:38.347 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91833 2024-08-17T18:20:38.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.347 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:20:38.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:20:38.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734763+0000", 2024-08-17T18:20:38.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.348 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:20:38.348 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734763+0000", 2024-08-17T18:20:38.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.348 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:20:38.348 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734764+0000", 2024-08-17T18:20:38.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7699999999999992e-07 2024-08-17T18:20:38.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.348 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:20:38.348 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734787+0000", 2024-08-17T18:20:38.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3243000000000001e-05 2024-08-17T18:20:38.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:20:38.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734800+0000", 2024-08-17T18:20:38.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2903e-05 2024-08-17T18:20:38.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:20:38.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734807+0000", 2024-08-17T18:20:38.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1459999999999999e-06 2024-08-17T18:20:38.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734860+0000", 2024-08-17T18:20:38.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2874999999999998e-05 2024-08-17T18:20:38.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734869+0000", 2024-08-17T18:20:38.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9039999999999998e-06 2024-08-17T18:20:38.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.646980+0000", 2024-08-17T18:20:38.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.91211099200000001 2024-08-17T18:20:38.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.646987+0000", 2024-08-17T18:20:38.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1149999999999996e-06 2024-08-17T18:20:38.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.893439+0000", 2024-08-17T18:20:38.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2464515359999999 2024-08-17T18:20:38.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:20:38.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.893736+0000", 2024-08-17T18:20:38.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029668499999999998 2024-08-17T18:20:38.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:20:38.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.893960+0000", 2024-08-17T18:20:38.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000224696 2024-08-17T18:20:38.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.897423+0000", 2024-08-17T18:20:38.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034624370000000001 2024-08-17T18:20:38.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:20:38.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.897433+0000", 2024-08-17T18:20:38.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0197000000000001e-05 2024-08-17T18:20:38.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.906207+0000", 2024-08-17T18:20:38.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0087737289999999992 2024-08-17T18:20:38.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:20:38.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.906242+0000", 2024-08-17T18:20:38.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5824000000000001e-05 2024-08-17T18:20:38.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:20:38.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907196+0000", 2024-08-17T18:20:38.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00095358499999999996 2024-08-17T18:20:38.354 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.354 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:20:38.354 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.354 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:91851 57.12 57:4935f3b2:::benchmark_data_smithi136_389235_object91850:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:20:38.354 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.746420+0000", 2024-08-17T18:20:38.354 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 337.51769363599999, 2024-08-17T18:20:38.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1658063599999999, 2024-08-17T18:20:38.354 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:20:38.354 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:20:38.354 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:20:38.354 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:20:38.355 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:20:38.355 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:20:38.355 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91851 2024-08-17T18:20:38.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.355 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:20:38.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:20:38.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.746420+0000", 2024-08-17T18:20:38.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:20:38.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.746420+0000", 2024-08-17T18:20:38.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:20:38.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.746421+0000", 2024-08-17T18:20:38.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3900000000000004e-07 2024-08-17T18:20:38.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:20:38.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.746444+0000", 2024-08-17T18:20:38.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2609999999999999e-05 2024-08-17T18:20:38.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:20:38.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.746452+0000", 2024-08-17T18:20:38.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0499999999999992e-06 2024-08-17T18:20:38.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:20:38.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.746457+0000", 2024-08-17T18:20:38.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3310000000000003e-06 2024-08-17T18:20:38.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.747175+0000", 2024-08-17T18:20:38.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00071805899999999995 2024-08-17T18:20:38.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.747184+0000", 2024-08-17T18:20:38.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2960000000000004e-06 2024-08-17T18:20:38.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647023+0000", 2024-08-17T18:20:38.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.89983893699999995 2024-08-17T18:20:38.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647032+0000", 2024-08-17T18:20:38.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7339999999999997e-06 2024-08-17T18:20:38.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.895088+0000", 2024-08-17T18:20:38.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2480557990000001 2024-08-17T18:20:38.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:20:38.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.895247+0000", 2024-08-17T18:20:38.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000159512 2024-08-17T18:20:38.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:20:38.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.895471+0000", 2024-08-17T18:20:38.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022369299999999999 2024-08-17T18:20:38.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:20:38.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.899385+0000", 2024-08-17T18:20:38.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003913583 2024-08-17T18:20:38.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.900594+0000", 2024-08-17T18:20:38.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012091599999999999 2024-08-17T18:20:38.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.910847+0000", 2024-08-17T18:20:38.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010253536000000001 2024-08-17T18:20:38.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:20:38.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.910887+0000", 2024-08-17T18:20:38.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0167999999999999e-05 2024-08-17T18:20:38.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:20:38.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.912227+0000", 2024-08-17T18:20:38.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0013392530000000001 2024-08-17T18:20:38.361 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.361 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:20:38.361 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.362 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:91877 57.12 57:4e2ca095:::benchmark_data_smithi136_389235_object91876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:20:38.362 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.764679+0000", 2024-08-17T18:20:38.362 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 337.49943485099999, 2024-08-17T18:20:38.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.148880057, 2024-08-17T18:20:38.362 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:20:38.365 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:20:38.365 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:20:38.365 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:20:38.365 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:20:38.365 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:20:38.365 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91877 2024-08-17T18:20:38.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.366 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:20:38.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:20:38.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.764679+0000", 2024-08-17T18:20:38.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:20:38.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.764679+0000", 2024-08-17T18:20:38.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:20:38.367 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.764681+0000", 2024-08-17T18:20:38.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5239999999999999e-06 2024-08-17T18:20:38.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:20:38.367 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.764715+0000", 2024-08-17T18:20:38.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3964000000000002e-05 2024-08-17T18:20:38.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:20:38.367 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.764727+0000", 2024-08-17T18:20:38.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2762000000000001e-05 2024-08-17T18:20:38.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.368 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.368 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:20:38.368 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.764735+0000", 2024-08-17T18:20:38.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3769999999999998e-06 2024-08-17T18:20:38.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.368 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.368 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.368 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.764809+0000", 2024-08-17T18:20:38.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3935000000000003e-05 2024-08-17T18:20:38.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.764821+0000", 2024-08-17T18:20:38.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2024e-05 2024-08-17T18:20:38.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647058+0000", 2024-08-17T18:20:38.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.88223715999999996 2024-08-17T18:20:38.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647062+0000", 2024-08-17T18:20:38.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1230000000000002e-06 2024-08-17T18:20:38.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.896097+0000", 2024-08-17T18:20:38.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.249034932 2024-08-17T18:20:38.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:20:38.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.896247+0000", 2024-08-17T18:20:38.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000150309 2024-08-17T18:20:38.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:20:38.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.896475+0000", 2024-08-17T18:20:38.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000227815 2024-08-17T18:20:38.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:20:38.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.900604+0000", 2024-08-17T18:20:38.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0041288979999999998 2024-08-17T18:20:38.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.901852+0000", 2024-08-17T18:20:38.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012485809999999999 2024-08-17T18:20:38.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.372 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.372 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.912289+0000", 2024-08-17T18:20:38.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010436453 2024-08-17T18:20:38.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.372 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:20:38.372 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.912355+0000", 2024-08-17T18:20:38.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5947999999999999e-05 2024-08-17T18:20:38.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:20:38.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.913559+0000", 2024-08-17T18:20:38.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012042520000000001 2024-08-17T18:20:38.373 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.373 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:20:38.373 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.373 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:91955 57.12 57:4acb1630:::benchmark_data_smithi136_389235_object91954:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:20:38.373 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.812317+0000", 2024-08-17T18:20:38.373 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 337.45179655499999, 2024-08-17T18:20:38.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1059336489999998, 2024-08-17T18:20:38.373 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:20:38.374 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:20:38.374 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:20:38.374 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:20:38.374 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:20:38.374 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:20:38.374 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91955 2024-08-17T18:20:38.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.374 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:20:38.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:20:38.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.812317+0000", 2024-08-17T18:20:38.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:20:38.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.812317+0000", 2024-08-17T18:20:38.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:20:38.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.812319+0000", 2024-08-17T18:20:38.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.31e-06 2024-08-17T18:20:38.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:20:38.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.812338+0000", 2024-08-17T18:20:38.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8881000000000001e-05 2024-08-17T18:20:38.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:20:38.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.812345+0000", 2024-08-17T18:20:38.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9470000000000003e-06 2024-08-17T18:20:38.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:20:38.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.812350+0000", 2024-08-17T18:20:38.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3919999999999996e-06 2024-08-17T18:20:38.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.812427+0000", 2024-08-17T18:20:38.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6713999999999998e-05 2024-08-17T18:20:38.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.812437+0000", 2024-08-17T18:20:38.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0720000000000001e-05 2024-08-17T18:20:38.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647088+0000", 2024-08-17T18:20:38.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.83465034999999999 2024-08-17T18:20:38.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.378 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647095+0000", 2024-08-17T18:20:38.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6599999999999995e-06 2024-08-17T18:20:38.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.378 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.897471+0000", 2024-08-17T18:20:38.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2503756109999999 2024-08-17T18:20:38.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.378 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:20:38.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.897618+0000", 2024-08-17T18:20:38.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000146699 2024-08-17T18:20:38.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:20:38.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.897843+0000", 2024-08-17T18:20:38.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022513899999999999 2024-08-17T18:20:38.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:20:38.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.900609+0000", 2024-08-17T18:20:38.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0027664709999999999 2024-08-17T18:20:38.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.903881+0000", 2024-08-17T18:20:38.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032721510000000001 2024-08-17T18:20:38.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.916916+0000", 2024-08-17T18:20:38.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013034108000000001 2024-08-17T18:20:38.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:20:38.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.916997+0000", 2024-08-17T18:20:38.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1508000000000006e-05 2024-08-17T18:20:38.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:20:38.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.918251+0000", 2024-08-17T18:20:38.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012539879999999999 2024-08-17T18:20:38.381 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.381 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:20:38.381 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.381 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:91997 57.12 57:480a6a35:::benchmark_data_smithi136_389235_object91996:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:20:38.382 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.839160+0000", 2024-08-17T18:20:38.382 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 337.42495375599998, 2024-08-17T18:20:38.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0802216869999999, 2024-08-17T18:20:38.382 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:20:38.382 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:20:38.382 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:20:38.382 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:20:38.382 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:20:38.382 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:20:38.383 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91997 2024-08-17T18:20:38.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.383 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:20:38.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:20:38.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.839160+0000", 2024-08-17T18:20:38.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:20:38.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.839160+0000", 2024-08-17T18:20:38.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:20:38.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.839161+0000", 2024-08-17T18:20:38.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1510000000000001e-06 2024-08-17T18:20:38.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:20:38.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.839185+0000", 2024-08-17T18:20:38.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3805e-05 2024-08-17T18:20:38.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:20:38.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.839194+0000", 2024-08-17T18:20:38.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1500000000000005e-06 2024-08-17T18:20:38.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:20:38.385 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.839201+0000", 2024-08-17T18:20:38.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2720000000000003e-06 2024-08-17T18:20:38.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.385 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.839374+0000", 2024-08-17T18:20:38.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017318800000000001 2024-08-17T18:20:38.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.839420+0000", 2024-08-17T18:20:38.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6050999999999997e-05 2024-08-17T18:20:38.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647119+0000", 2024-08-17T18:20:38.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.80769961099999998 2024-08-17T18:20:38.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647123+0000", 2024-08-17T18:20:38.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9400000000000004e-06 2024-08-17T18:20:38.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.898480+0000", 2024-08-17T18:20:38.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2513567079999999 2024-08-17T18:20:38.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:20:38.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.898570+0000", 2024-08-17T18:20:38.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0339000000000004e-05 2024-08-17T18:20:38.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:20:38.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.898732+0000", 2024-08-17T18:20:38.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000161356 2024-08-17T18:20:38.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:20:38.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.901862+0000", 2024-08-17T18:20:38.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031297849999999999 2024-08-17T18:20:38.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.903932+0000", 2024-08-17T18:20:38.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020700010000000001 2024-08-17T18:20:38.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.918308+0000", 2024-08-17T18:20:38.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014376938000000001 2024-08-17T18:20:38.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:20:38.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.918369+0000", 2024-08-17T18:20:38.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0520999999999999e-05 2024-08-17T18:20:38.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:20:38.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.919382+0000", 2024-08-17T18:20:38.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001012871 2024-08-17T18:20:38.389 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.389 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:20:38.390 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.390 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:92000 57.12 57:4c400e2c:::benchmark_data_smithi136_389235_object91999:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:20:38.390 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.840632+0000", 2024-08-17T18:20:38.390 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 337.42348236300001, 2024-08-17T18:20:38.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0960051650000002, 2024-08-17T18:20:38.390 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:20:38.390 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:20:38.390 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:20:38.390 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:20:38.390 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:20:38.391 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:20:38.391 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 92000 2024-08-17T18:20:38.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.391 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:20:38.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:20:38.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.840632+0000", 2024-08-17T18:20:38.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:20:38.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.840632+0000", 2024-08-17T18:20:38.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:20:38.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.840633+0000", 2024-08-17T18:20:38.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.404e-06 2024-08-17T18:20:38.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:20:38.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.840653+0000", 2024-08-17T18:20:38.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0111000000000001e-05 2024-08-17T18:20:38.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:20:38.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.840664+0000", 2024-08-17T18:20:38.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0988999999999999e-05 2024-08-17T18:20:38.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:20:38.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.840671+0000", 2024-08-17T18:20:38.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4810000000000003e-06 2024-08-17T18:20:38.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.840748+0000", 2024-08-17T18:20:38.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7634999999999998e-05 2024-08-17T18:20:38.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.840758+0000", 2024-08-17T18:20:38.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6309999999999998e-06 2024-08-17T18:20:38.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647137+0000", 2024-08-17T18:20:38.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.80637933399999995 2024-08-17T18:20:38.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647141+0000", 2024-08-17T18:20:38.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8260000000000003e-06 2024-08-17T18:20:38.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.899425+0000", 2024-08-17T18:20:38.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2522841220000001 2024-08-17T18:20:38.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:20:38.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.899560+0000", 2024-08-17T18:20:38.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013490299999999999 2024-08-17T18:20:38.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:20:38.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.899790+0000", 2024-08-17T18:20:38.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0002299 2024-08-17T18:20:38.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:20:38.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.903824+0000", 2024-08-17T18:20:38.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040337039999999999 2024-08-17T18:20:38.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.903988+0000", 2024-08-17T18:20:38.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000164752 2024-08-17T18:20:38.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.935038+0000", 2024-08-17T18:20:38.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.031049243000000001 2024-08-17T18:20:38.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:20:38.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.935099+0000", 2024-08-17T18:20:38.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1532000000000007e-05 2024-08-17T18:20:38.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:20:38.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.936637+0000", 2024-08-17T18:20:38.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001537598 2024-08-17T18:20:38.398 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.398 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:20:38.398 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.398 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:92004 57.12 57:4b569127:::benchmark_data_smithi136_389235_object92003:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:20:38.398 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.841912+0000", 2024-08-17T18:20:38.398 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 337.42220156100001, 2024-08-17T18:20:38.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.095822149, 2024-08-17T18:20:38.398 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:20:38.398 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:20:38.398 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:20:38.399 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:20:38.399 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:20:38.399 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:20:38.399 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 92004 2024-08-17T18:20:38.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.399 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:20:38.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:20:38.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.841912+0000", 2024-08-17T18:20:38.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:20:38.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.841912+0000", 2024-08-17T18:20:38.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:20:38.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.841914+0000", 2024-08-17T18:20:38.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2890000000000001e-06 2024-08-17T18:20:38.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:20:38.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.841937+0000", 2024-08-17T18:20:38.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3725000000000001e-05 2024-08-17T18:20:38.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:20:38.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.841950+0000", 2024-08-17T18:20:38.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3012e-05 2024-08-17T18:20:38.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:20:38.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.841957+0000", 2024-08-17T18:20:38.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8140000000000003e-06 2024-08-17T18:20:38.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.842028+0000", 2024-08-17T18:20:38.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0962000000000003e-05 2024-08-17T18:20:38.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.402 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.402 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.842036+0000", 2024-08-17T18:20:38.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9049999999999997e-06 2024-08-17T18:20:38.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.402 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:38.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647172+0000", 2024-08-17T18:20:38.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.80513599999999996 2024-08-17T18:20:38.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647180+0000", 2024-08-17T18:20:38.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0749999999999998e-06 2024-08-17T18:20:38.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.902828+0000", 2024-08-17T18:20:38.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.255647797 2024-08-17T18:20:38.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:20:38.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.902942+0000", 2024-08-17T18:20:38.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113795 2024-08-17T18:20:38.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:20:38.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.903135+0000", 2024-08-17T18:20:38.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000193043 2024-08-17T18:20:38.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:20:38.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907209+0000", 2024-08-17T18:20:38.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040746159999999997 2024-08-17T18:20:38.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907255+0000", 2024-08-17T18:20:38.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5674999999999999e-05 2024-08-17T18:20:38.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.936682+0000", 2024-08-17T18:20:38.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.029426833999999999 2024-08-17T18:20:38.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:20:38.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.936745+0000", 2024-08-17T18:20:38.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2617e-05 2024-08-17T18:20:38.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:20:38.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.937735+0000", 2024-08-17T18:20:38.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00098999000000000005 2024-08-17T18:20:38.407 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.407 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:20:38.408 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.408 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:92015 57.12 57:4fcb2167:::benchmark_data_smithi136_389235_object92014:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:20:38.408 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.850088+0000", 2024-08-17T18:20:38.408 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 337.414026369, 2024-08-17T18:20:38.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0964700700000001, 2024-08-17T18:20:38.408 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:20:38.408 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:20:38.408 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:20:38.408 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:20:38.408 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:20:38.408 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:20:38.409 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 92015 2024-08-17T18:20:38.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.409 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:20:38.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.409 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:20:38.409 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.850088+0000", 2024-08-17T18:20:38.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.409 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:20:38.409 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.850088+0000", 2024-08-17T18:20:38.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.410 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:20:38.410 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.850089+0000", 2024-08-17T18:20:38.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.063e-06 2024-08-17T18:20:38.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.410 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:20:38.410 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.850109+0000", 2024-08-17T18:20:38.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0438999999999999e-05 2024-08-17T18:20:38.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.410 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:20:38.410 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.850121+0000", 2024-08-17T18:20:38.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1982e-05 2024-08-17T18:20:38.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.411 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:20:38.411 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.850128+0000", 2024-08-17T18:20:38.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8990000000000004e-06 2024-08-17T18:20:38.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.411 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.411 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.850188+0000", 2024-08-17T18:20:38.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0013999999999998e-05 2024-08-17T18:20:38.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.850197+0000", 2024-08-17T18:20:38.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9679999999999995e-06 2024-08-17T18:20:38.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647204+0000", 2024-08-17T18:20:38.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.79700673499999997 2024-08-17T18:20:38.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647208+0000", 2024-08-17T18:20:38.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8940000000000003e-06 2024-08-17T18:20:38.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.904025+0000", 2024-08-17T18:20:38.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.256817004 2024-08-17T18:20:38.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:20:38.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.904124+0000", 2024-08-17T18:20:38.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9591000000000002e-05 2024-08-17T18:20:38.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:20:38.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.904271+0000", 2024-08-17T18:20:38.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014643300000000001 2024-08-17T18:20:38.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907318+0000", 2024-08-17T18:20:38.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0030469889999999999 2024-08-17T18:20:38.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:20:38.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907332+0000", 2024-08-17T18:20:38.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4800000000000001e-05 2024-08-17T18:20:38.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.415 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.415 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.944953+0000", 2024-08-17T18:20:38.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0376207 2024-08-17T18:20:38.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.415 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:20:38.415 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.945020+0000", 2024-08-17T18:20:38.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6569999999999995e-05 2024-08-17T18:20:38.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.415 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:20:38.415 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.946558+0000", 2024-08-17T18:20:38.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001537989 2024-08-17T18:20:38.416 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.416 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:20:38.416 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.416 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:92043 57.12 57:4c633a12:::benchmark_data_smithi136_389235_object92042:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:20:38.416 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.878840+0000", 2024-08-17T18:20:38.416 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 337.38527422700002, 2024-08-17T18:20:38.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0690841519999998, 2024-08-17T18:20:38.416 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:20:38.416 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:20:38.416 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:20:38.416 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:20:38.417 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:20:38.417 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:20:38.417 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 92043 2024-08-17T18:20:38.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.417 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:20:38.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:20:38.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.878840+0000", 2024-08-17T18:20:38.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:20:38.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.878840+0000", 2024-08-17T18:20:38.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:20:38.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.878842+0000", 2024-08-17T18:20:38.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.835e-06 2024-08-17T18:20:38.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:20:38.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.878872+0000", 2024-08-17T18:20:38.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0314000000000001e-05 2024-08-17T18:20:38.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.419 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:20:38.419 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.878887+0000", 2024-08-17T18:20:38.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5167999999999999e-05 2024-08-17T18:20:38.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.419 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.419 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:20:38.419 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.878894+0000", 2024-08-17T18:20:38.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3470000000000001e-06 2024-08-17T18:20:38.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.878959+0000", 2024-08-17T18:20:38.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4885e-05 2024-08-17T18:20:38.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.878973+0000", 2024-08-17T18:20:38.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3757000000000001e-05 2024-08-17T18:20:38.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647236+0000", 2024-08-17T18:20:38.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.76826348300000002 2024-08-17T18:20:38.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.423 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647244+0000", 2024-08-17T18:20:38.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.452e-06 2024-08-17T18:20:38.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.423 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.423 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.904870+0000", 2024-08-17T18:20:38.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2576263270000001 2024-08-17T18:20:38.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.424 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:20:38.424 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.905024+0000", 2024-08-17T18:20:38.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000153669 2024-08-17T18:20:38.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.424 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:20:38.424 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.905261+0000", 2024-08-17T18:20:38.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023719299999999999 2024-08-17T18:20:38.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.424 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:20:38.424 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907339+0000", 2024-08-17T18:20:38.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020783189999999999 2024-08-17T18:20:38.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907391+0000", 2024-08-17T18:20:38.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2040999999999999e-05 2024-08-17T18:20:38.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.946619+0000", 2024-08-17T18:20:38.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039227646999999997 2024-08-17T18:20:38.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:20:38.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.946682+0000", 2024-08-17T18:20:38.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3275999999999995e-05 2024-08-17T18:20:38.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:20:38.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.947924+0000", 2024-08-17T18:20:38.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001241439 2024-08-17T18:20:38.426 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.426 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:20:38.426 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.426 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:92046 57.12 57:49923df3:::benchmark_data_smithi136_389235_object92045:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:20:38.427 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.881358+0000", 2024-08-17T18:20:38.427 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 337.38275562899997, 2024-08-17T18:20:38.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0681408019999998, 2024-08-17T18:20:38.427 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:20:38.427 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:20:38.427 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:20:38.427 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:20:38.427 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:20:38.427 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:20:38.427 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 92046 2024-08-17T18:20:38.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.427 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:20:38.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:20:38.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.881358+0000", 2024-08-17T18:20:38.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:20:38.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.881358+0000", 2024-08-17T18:20:38.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:20:38.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.881360+0000", 2024-08-17T18:20:38.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7710000000000001e-06 2024-08-17T18:20:38.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:20:38.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.881387+0000", 2024-08-17T18:20:38.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6526e-05 2024-08-17T18:20:38.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:20:38.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.881401+0000", 2024-08-17T18:20:38.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4718e-05 2024-08-17T18:20:38.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:20:38.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.881410+0000", 2024-08-17T18:20:38.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8240000000000006e-06 2024-08-17T18:20:38.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.881519+0000", 2024-08-17T18:20:38.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109304 2024-08-17T18:20:38.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.881531+0000", 2024-08-17T18:20:38.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1226000000000001e-05 2024-08-17T18:20:38.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647335+0000", 2024-08-17T18:20:38.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.76580386099999997 2024-08-17T18:20:38.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647344+0000", 2024-08-17T18:20:38.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8700000000000004e-06 2024-08-17T18:20:38.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907434+0000", 2024-08-17T18:20:38.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.260090033 2024-08-17T18:20:38.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.432 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:20:38.432 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907581+0000", 2024-08-17T18:20:38.432 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014699299999999999 2024-08-17T18:20:38.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.432 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:20:38.432 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907826+0000", 2024-08-17T18:20:38.432 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024497800000000003 2024-08-17T18:20:38.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:20:38.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.913576+0000", 2024-08-17T18:20:38.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0057498929999999998 2024-08-17T18:20:38.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.913640+0000", 2024-08-17T18:20:38.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3325999999999996e-05 2024-08-17T18:20:38.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.948005+0000", 2024-08-17T18:20:38.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034365035000000002 2024-08-17T18:20:38.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:20:38.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.948074+0000", 2024-08-17T18:20:38.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8984000000000002e-05 2024-08-17T18:20:38.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:20:38.435 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.949499+0000", 2024-08-17T18:20:38.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014254599999999999 2024-08-17T18:20:38.435 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.435 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:20:38.435 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.435 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.435 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:92175 57.12 57:494e5fb2:::benchmark_data_smithi136_389235_object92174:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:20:38.435 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:01.049817+0000", 2024-08-17T18:20:38.435 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 337.214297031, 2024-08-17T18:20:38.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.900916759, 2024-08-17T18:20:38.435 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:20:38.435 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:20:38.436 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:20:38.436 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:20:38.436 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:20:38.436 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:20:38.436 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 92175 2024-08-17T18:20:38.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.436 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:20:38.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:20:38.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.049817+0000", 2024-08-17T18:20:38.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:20:38.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.049817+0000", 2024-08-17T18:20:38.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:20:38.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:20:38.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.049818+0000", 2024-08-17T18:20:38.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.562e-06 2024-08-17T18:20:38.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:20:38.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.049836+0000", 2024-08-17T18:20:38.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8045000000000001e-05 2024-08-17T18:20:38.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:20:38.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.049849+0000", 2024-08-17T18:20:38.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2554999999999999e-05 2024-08-17T18:20:38.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:20:38.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.049855+0000", 2024-08-17T18:20:38.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3169999999999998e-06 2024-08-17T18:20:38.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.049917+0000", 2024-08-17T18:20:38.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2015999999999997e-05 2024-08-17T18:20:38.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.049929+0000", 2024-08-17T18:20:38.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1861e-05 2024-08-17T18:20:38.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647378+0000", 2024-08-17T18:20:38.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59744883800000004 2024-08-17T18:20:38.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:20:38.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647385+0000", 2024-08-17T18:20:38.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0929999999999997e-06 2024-08-17T18:20:38.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:20:38.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.908719+0000", 2024-08-17T18:20:38.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2613335079999999 2024-08-17T18:20:38.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:20:38.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.908865+0000", 2024-08-17T18:20:38.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014667200000000001 2024-08-17T18:20:38.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:20:38.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.909091+0000", 2024-08-17T18:20:38.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022598799999999999 2024-08-17T18:20:38.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:20:38.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.913584+0000", 2024-08-17T18:20:38.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0044922920000000002 2024-08-17T18:20:38.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.913700+0000", 2024-08-17T18:20:38.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011666 2024-08-17T18:20:38.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:20:38.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.949569+0000", 2024-08-17T18:20:38.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035869036999999999 2024-08-17T18:20:38.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:20:38.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.949631+0000", 2024-08-17T18:20:38.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1266000000000006e-05 2024-08-17T18:20:38.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:20:38.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:20:38.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:20:38.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.950734+0000", 2024-08-17T18:20:38.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001103049 2024-08-17T18:20:38.443 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.443 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:20:38.443 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.443 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:20:38.443 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:20:38.443 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:20:38.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:38.445 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:38.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:38.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:38.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:38.569 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:38.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:38.670 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:38.695 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:38.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:38.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:38.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:38.822 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:38.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:38.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:38.925 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:38.949 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:38.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:38.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:39.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:39.077 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:39.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:39.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:39.180 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:39.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:38 smithi136 ceph-mon[109861]: pgmap v3295: 83 pgs: 83 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-08-17T18:20:39.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:38 smithi136 ceph-mon[115078]: pgmap v3295: 83 pgs: 83 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-08-17T18:20:39.205 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:39.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:39.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:39.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:38 smithi165 ceph-mon[105517]: pgmap v3295: 83 pgs: 83 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-08-17T18:20:39.307 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:39.331 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:39.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:39.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:39.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:39.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:39.459 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:39.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:39.560 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:39.583 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:39.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:39.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:39.685 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:39.709 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:39.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:39.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:39.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:39.838 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:39.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:39.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:39.940 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:39.964 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:39.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:39.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:40.039 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-17T18:20:40.065 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:40.074 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:20:40.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:40.090 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:40.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:40.192 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:40.215 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:40.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:40.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:40.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:39 smithi165 ceph-mon[105517]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/78 objects degraded (1.282%), 1 pg degraded, 1 pg undersized) 2024-08-17T18:20:40.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:39 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T18:20:40.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:40.338 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:40.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:40.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:40.440 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:40.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:39 smithi136 ceph-mon[109861]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/78 objects degraded (1.282%), 1 pg degraded, 1 pg undersized) 2024-08-17T18:20:40.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:39 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T18:20:40.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:39 smithi136 ceph-mon[115078]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/78 objects degraded (1.282%), 1 pg degraded, 1 pg undersized) 2024-08-17T18:20:40.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:39 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T18:20:40.462 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:40.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:40.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:40.567 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:40.589 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:40.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:40.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:40.692 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:40.714 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:40.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:40.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:40.815 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:40.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:40.842 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:40.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:40.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:40.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:40.965 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:40.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:41.059 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:40 smithi136 ceph-mon[109861]: pgmap v3296: 83 pgs: 83 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-08-17T18:20:41.059 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:40 smithi136 ceph-mon[115078]: pgmap v3296: 83 pgs: 83 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-08-17T18:20:41.066 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:41.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:41.089 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:41.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:41.190 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:41.212 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:41.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:41.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:41.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:40 smithi165 ceph-mon[105517]: pgmap v3296: 83 pgs: 83 active+clean; 582 KiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-08-17T18:20:41.313 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:41.335 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:41.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:41.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:41.437 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:41.458 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:41.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:41.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:41.560 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:41.570 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:20:41.571 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:20:41.578 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:20:41.578 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:20:41.581 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:20:41.581 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:20:41.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:41.582 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:41.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:41.601 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:20:41.601 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:20:41.618 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:20:41.621 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:20:41.621 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:20:41.650 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:20:41.650 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:20:41.666 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:20:41.667 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:20:41.683 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:20:41.683 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:20:41.683 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:41.698 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:20:41.699 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:20:41.704 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:41.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:41.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:41.806 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:41.829 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:41.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:41.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:41.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:41.952 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:41.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:41.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:41.980 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'unique_pool_5' created 2024-08-17T18:20:42.054 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:42.076 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:42.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:42.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:42.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:42.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:42.200 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:42.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:41 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/260462736' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-08-17T18:20:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:41 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3256405537' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:20:42.301 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:42.314 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:41 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/260462736' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-08-17T18:20:42.314 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:41 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3256405537' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:20:42.314 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:41 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/260462736' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-08-17T18:20:42.314 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:41 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3256405537' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:20:42.324 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:42.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:42.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:42.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:42.447 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:42.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:42.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:42.549 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:42.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:42.570 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:42.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:42.671 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:42.695 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:42.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:42.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:42.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:42.819 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:42.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:42.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:42.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:42.942 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:42.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:42.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:43.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:43.066 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:43.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:43.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:43.168 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:43.190 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:43.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:43.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:43.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:42 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/260462736' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-08-17T18:20:43.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:42 smithi165 ceph-mon[105517]: osdmap e1701: 8 total, 8 up, 8 in 2024-08-17T18:20:43.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:42 smithi165 ceph-mon[105517]: pgmap v3298: 99 pgs: 9 creating+peering, 7 unknown, 83 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-08-17T18:20:43.291 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:43.313 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:43.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:43.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:43.415 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:43.433 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:42 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/260462736' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-08-17T18:20:43.433 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:42 smithi136 ceph-mon[109861]: osdmap e1701: 8 total, 8 up, 8 in 2024-08-17T18:20:43.434 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:42 smithi136 ceph-mon[109861]: pgmap v3298: 99 pgs: 9 creating+peering, 7 unknown, 83 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-08-17T18:20:43.434 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:42 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/260462736' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-08-17T18:20:43.434 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:42 smithi136 ceph-mon[115078]: osdmap e1701: 8 total, 8 up, 8 in 2024-08-17T18:20:43.434 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:42 smithi136 ceph-mon[115078]: pgmap v3298: 99 pgs: 9 creating+peering, 7 unknown, 83 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-08-17T18:20:43.440 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:43.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:43.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:43.542 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:43.564 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:43.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:43.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:43.666 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:43.687 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:43.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:43.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:43.789 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:43.810 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:43.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:43.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:43.912 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:43.935 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:43.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:43.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:44.037 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:44.059 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:44.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:44.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:44.161 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:44.182 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:44.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:44.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:44.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:43 smithi165 ceph-mon[105517]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T18:20:44.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:43 smithi165 ceph-mon[105517]: osdmap e1702: 8 total, 8 up, 8 in 2024-08-17T18:20:44.284 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:44.307 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:44.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:44.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:44.335 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:43 smithi136 ceph-mon[109861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T18:20:44.335 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:43 smithi136 ceph-mon[109861]: osdmap e1702: 8 total, 8 up, 8 in 2024-08-17T18:20:44.335 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:43 smithi136 ceph-mon[115078]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T18:20:44.336 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:43 smithi136 ceph-mon[115078]: osdmap e1702: 8 total, 8 up, 8 in 2024-08-17T18:20:44.409 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:44.718 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:20:44.738 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:44.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:44.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:44.840 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:44.867 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:44.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:44.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:44.969 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:44.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:44.992 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:44.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:45.093 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:45.119 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:45.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:45.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:45.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:44 smithi136 ceph-mon[109861]: osdmap e1703: 8 total, 8 up, 8 in 2024-08-17T18:20:45.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:44 smithi136 ceph-mon[109861]: pgmap v3301: 99 pgs: 9 creating+peering, 7 unknown, 83 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:20:45.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:44 smithi136 ceph-mon[115078]: osdmap e1703: 8 total, 8 up, 8 in 2024-08-17T18:20:45.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:44 smithi136 ceph-mon[115078]: pgmap v3301: 99 pgs: 9 creating+peering, 7 unknown, 83 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:20:45.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:45.246 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:45.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:45.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:45.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:44 smithi165 ceph-mon[105517]: osdmap e1703: 8 total, 8 up, 8 in 2024-08-17T18:20:45.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:44 smithi165 ceph-mon[105517]: pgmap v3301: 99 pgs: 9 creating+peering, 7 unknown, 83 active+clean; 582 KiB data, 9.5 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:20:45.348 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:45.370 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:45.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:45.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:45.472 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:45.494 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:45.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:45.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:45.596 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:45.622 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:45.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:45.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:45.724 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:45.750 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:45.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:45.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:45.851 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:45.877 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:45.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:45.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:45.926 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-08-17T18:20:45.979 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:46.003 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:46.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:46.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:46.105 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:46.127 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:46.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:46.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:46.229 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:46.255 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:46.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:46.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:46.357 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:46.378 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:46.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:46.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:46.475 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:20:46.480 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:46.506 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:46.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:46.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:46.533 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:46 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2702461578' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:20:46.533 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:46 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:20:46.534 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:46 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2702461578' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:20:46.534 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:46 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:20:46.608 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:46.630 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:46.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:46.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:46.732 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:46.759 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:46.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:46.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:46.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:46 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2702461578' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:20:46.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:46 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:20:46.861 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:46.883 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:46.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:46.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:46.985 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:47.006 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:47.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:47.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:47.108 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:47.134 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:47.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:47.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:47.236 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:47.258 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:47.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:47.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:47.359 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:47.381 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:47.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:47.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:47.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:47.505 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:47.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:47.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:47.597 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:20:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:20:47] "GET /metrics HTTP/1.1" 200 34343 "" "Prometheus/2.51.0" 2024-08-17T18:20:47.597 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:47 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:20:47.597 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:47 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:20:47.597 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:47 smithi136 ceph-mon[115078]: osdmap e1704: 8 total, 8 up, 8 in 2024-08-17T18:20:47.597 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:47 smithi136 ceph-mon[115078]: pgmap v3303: 99 pgs: 9 creating+peering, 90 active+clean; 582 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:20:47.598 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:47 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:20:47.598 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:47 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:20:47.598 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:47 smithi136 ceph-mon[109861]: osdmap e1704: 8 total, 8 up, 8 in 2024-08-17T18:20:47.598 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:47 smithi136 ceph-mon[109861]: pgmap v3303: 99 pgs: 9 creating+peering, 90 active+clean; 582 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:20:47.606 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:47.628 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:47.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:47.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:47.730 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:47.751 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:47.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:47.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:47 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:20:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:47 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:20:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:47 smithi165 ceph-mon[105517]: osdmap e1704: 8 total, 8 up, 8 in 2024-08-17T18:20:47.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:47 smithi165 ceph-mon[105517]: pgmap v3303: 99 pgs: 9 creating+peering, 90 active+clean; 582 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:20:47.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:47.878 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:47.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:47.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:47.980 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:48.002 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:48.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:48.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:48.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:48.128 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:20:48.129 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:48.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:48.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:48.231 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:48.252 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:48.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:48.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:48.354 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:48.378 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:48.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:48.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:48.480 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:48.496 INFO:teuthology.orchestra.run.smithi136.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-08-17T18:20:48.505 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:48.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:48.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:48.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:48.629 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:48.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:48.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:48.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:48 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3938579467' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-08-17T18:20:48.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:48 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-08-17T18:20:48.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:48 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3938579467' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-08-17T18:20:48.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:48 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-08-17T18:20:48.731 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:48.752 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:48.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:48.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:48.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:48 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3938579467' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-08-17T18:20:48.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:48 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-08-17T18:20:48.854 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:48.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:48.877 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:48.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:48.978 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:48.999 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:49.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:49.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:49.101 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:49.126 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:49.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:49.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:49.228 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:49.254 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:49.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:49.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:49.356 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:49.380 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:49.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:49.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:49.482 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:49.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:49.508 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:49.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:49.588 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:20:49.588 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:20:49.610 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:49.631 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:49.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:49.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:49.733 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:49.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:49.760 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:49.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:49 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-08-17T18:20:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:49 smithi165 ceph-mon[105517]: osdmap e1705: 8 total, 8 up, 8 in 2024-08-17T18:20:49.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:49 smithi165 ceph-mon[105517]: pgmap v3305: 99 pgs: 99 active+clean; 582 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:20:49.862 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:49.886 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:49.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:49.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:49.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:49 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-08-17T18:20:49.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:49 smithi136 ceph-mon[109861]: osdmap e1705: 8 total, 8 up, 8 in 2024-08-17T18:20:49.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:49 smithi136 ceph-mon[109861]: pgmap v3305: 99 pgs: 99 active+clean; 582 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:20:49.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:49 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-08-17T18:20:49.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:49 smithi136 ceph-mon[115078]: osdmap e1705: 8 total, 8 up, 8 in 2024-08-17T18:20:49.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:49 smithi136 ceph-mon[115078]: pgmap v3305: 99 pgs: 99 active+clean; 582 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:20:49.987 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:50.004 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:20:50.010 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:50.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:50.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:50.112 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:50.135 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:50.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:50.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:50.237 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:50.261 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:50.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:50.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:50.363 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:50.387 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:50.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:50.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:50.489 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:50.511 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:50.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:50.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:50.613 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:50.636 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:50.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:50.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:50.737 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:50.760 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:50.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:50.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:50.862 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:50.884 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:50.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:50.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:50.910 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-17T18:20:50.956 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:20:50.987 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:51.015 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:51.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:51.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:51.118 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:51.140 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:51.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:51.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:51.241 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:51.263 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:51.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:51.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:51.365 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:51.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:51.390 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:51.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:51.491 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:51.514 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:51.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:51.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:51.616 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:51.639 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:51.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:51.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:51.741 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:51.764 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:51.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:51.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:51.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:51 smithi136 ceph-mon[109861]: pgmap v3306: 99 pgs: 99 active+clean; 582 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:20:51.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:51.824 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:20:51.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:51 smithi136 ceph-mon[115078]: pgmap v3306: 99 pgs: 99 active+clean; 582 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:20:51.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:51.825 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:20:51.865 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:51.888 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:51.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:51.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:51.989 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:51 smithi165 ceph-mon[105517]: pgmap v3306: 99 pgs: 99 active+clean; 582 KiB data, 9.4 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:20:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:20:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:20:52.012 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:52.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:52.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:52.114 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:52.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:52.138 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:52.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:52.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:52.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:52.262 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:52.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:52.363 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:52.386 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:52.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:52.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:52.486 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:20:52.486 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:20:52.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:52.502 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:20:52.503 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:20:52.512 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:52.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:52.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:52.523 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:20:52.523 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:20:52.539 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:20:52.539 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:20:52.556 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:20:52.557 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:20:52.572 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:20:52.572 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:20:52.588 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:20:52.588 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:20:52.604 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:20:52.604 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:20:52.615 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:52.638 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:52.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:52.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:52.740 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:52.762 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:52.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:52.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:52.843 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:52 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/480970976' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:20:52.843 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:52 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/480970976' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:20:52.864 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:52.885 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:52.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:52.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:52.987 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:53.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:52 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/480970976' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:20:53.011 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:53.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:53.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:53.113 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:53.136 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:53.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:53.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:53.237 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:53.261 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:53.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:53.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:53.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:53.386 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:53.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:53.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:53.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:53.511 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:53.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:53.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:53.613 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:53.635 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:53.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:53.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:53.737 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:53.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:53.763 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:53.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:53.864 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:53.886 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:53.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:53.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:53.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:53 smithi136 ceph-mon[109861]: pgmap v3307: 99 pgs: 99 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:20:53.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:53 smithi136 ceph-mon[115078]: pgmap v3307: 99 pgs: 99 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:20:53.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:54.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:53 smithi165 ceph-mon[105517]: pgmap v3307: 99 pgs: 99 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:20:54.014 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:54.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:54.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:54.088 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:20:54.116 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:54.141 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:54.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:54.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:54.243 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:54.265 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:54.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:54.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:54.367 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:54.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:54.394 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:54.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:54.495 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:54.521 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:54.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:54.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:54.623 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:54.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:54.649 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:54.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:54.751 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:54.777 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:54.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:54.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:54.879 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:54.905 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:54.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:54.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:55.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:55.033 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:55.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:55.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:55.135 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:55.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:55.159 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:55.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:55.260 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:55.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:55.282 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:55.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:55.383 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:55.409 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:55.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:55.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:55.425 INFO:tasks.thrashosds.thrasher:in_osds: [2, 4, 6, 0, 5, 7, 3, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:20:55.425 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:20:55.511 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:55.536 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:55.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:55.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:55.609 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:20:55.637 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:55.674 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:55.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:55.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:55.777 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:55.802 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:55.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:55.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:55.895 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:55 smithi136 ceph-mon[115078]: pgmap v3308: 99 pgs: 99 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:20:55.895 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:55 smithi136 ceph-mon[109861]: pgmap v3308: 99 pgs: 99 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:20:55.900 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:20:55.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:55.929 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:55.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:55.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:55 smithi165 ceph-mon[105517]: pgmap v3308: 99 pgs: 99 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:20:56.031 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:56.054 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:56.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:56.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:56.156 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:56.180 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:56.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:56.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:56.282 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:56.307 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:56.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:56.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:56.408 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:56.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:56.433 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:56.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:20:56.534 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:56.555 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:56.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:56.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:56.657 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:56.678 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:56.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:56.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:56.762 INFO:teuthology.orchestra.run.smithi136.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-08-17T18:20:56.767 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:20:56.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:20:56.803 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:56.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:56.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:20:56.905 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:56.932 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:56.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:56.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:56 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2379577251' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T18:20:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:56 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/619395164' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:20:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:56 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:20:57.034 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:57.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:57.062 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:57.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:57.132 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:56 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2379577251' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T18:20:57.132 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:56 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/619395164' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:20:57.132 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:56 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:20:57.132 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:56 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2379577251' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T18:20:57.132 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:56 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/619395164' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:20:57.132 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:56 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:20:57.163 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:57.185 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:57.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:57.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:57.287 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:57.308 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:57.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:57.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:57.410 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:57.432 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:57.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:57.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:57.533 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:57.559 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:57.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:57.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:57.661 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:57.683 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:57.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:57.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:57.755 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:20:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:20:57] "GET /metrics HTTP/1.1" 200 36662 "" "Prometheus/2.51.0" 2024-08-17T18:20:57.755 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:57 smithi136 ceph-mon[109861]: pgmap v3309: 99 pgs: 99 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:20:57.785 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:20:57.807 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:57.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:57.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:20:57.909 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:20:57.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:57.931 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:57.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:20:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:57 smithi165 ceph-mon[105517]: pgmap v3309: 99 pgs: 99 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:20:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:57 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:20:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:57 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2379577251' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T18:20:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:57 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:20:58.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:57 smithi165 ceph-mon[105517]: osdmap e1706: 8 total, 8 up, 8 in 2024-08-17T18:20:58.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:58.052 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:57 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:20:58.052 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:57 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2379577251' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T18:20:58.052 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:57 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:20:58.052 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:57 smithi136 ceph-mon[109861]: osdmap e1706: 8 total, 8 up, 8 in 2024-08-17T18:20:58.053 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:57 smithi136 ceph-mon[115078]: pgmap v3309: 99 pgs: 99 active+clean; 582 KiB data, 9.2 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:20:58.053 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:57 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:20:58.053 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:57 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2379577251' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T18:20:58.053 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:57 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:20:58.053 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:57 smithi136 ceph-mon[115078]: osdmap e1706: 8 total, 8 up, 8 in 2024-08-17T18:20:58.059 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:58.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:58.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:58.161 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:20:58.186 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:58.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:58.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:20:58.289 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:58.314 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:58.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:58.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:58.416 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:58.442 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:58.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:58.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:58.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:58.569 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:58.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:58.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:58.671 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:58.697 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:58.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:58.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:58.799 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:58.825 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:58.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:58.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:58.927 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:58.953 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:58.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:58.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:59.055 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:59.080 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:59.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:59.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:59.126 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:58 smithi136 ceph-mon[109861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T18:20:59.126 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:58 smithi136 ceph-mon[115078]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T18:20:59.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:59.202 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:20:59.208 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:59.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:59.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:58 smithi165 ceph-mon[105517]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T18:20:59.311 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:59.337 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:59.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:59.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:59.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:20:59.465 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:59.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:59.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:20:59.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:59.594 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:59.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:59.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:59.696 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:20:59.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:59.723 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:59.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:20:59.824 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:20:59.855 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:20:59.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:59.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:20:59.957 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:20:59.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:20:59.985 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:20:59.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:00.008 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:20:59 smithi136 ceph-mon[109861]: pgmap v3311: 99 pgs: 99 active+clean; 582 KiB data, 9.1 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:21:00.008 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:20:59 smithi136 ceph-mon[115078]: pgmap v3311: 99 pgs: 99 active+clean; 582 KiB data, 9.1 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:21:00.073 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:21:00.087 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:00.111 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:00.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:00.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:00.150 DEBUG:teuthology.orchestra.run.smithi136:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-08-17T18:21:00.151 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-08-17T18:21:00.152 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T18:21:00.152 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:21:00.152 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T18:21:00.152 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-17T18:21:00.152 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-17T18:21:00.153 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:21:00.158 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-17T18:21:00.158 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-17T18:21:00.208 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:hints = 1 2024-08-17T18:21:00.209 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-17T18:21:00.209 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Object prefix: benchmark_data_smithi136_442124 2024-08-17T18:21:00.210 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T18:21:00.210 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 0 0 0 0 0 0 - 0 2024-08-17T18:21:00.213 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:00.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:20:59 smithi165 ceph-mon[105517]: pgmap v3311: 99 pgs: 99 active+clean; 582 KiB data, 9.1 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:21:00.261 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:00.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:00.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:00.364 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:00.390 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:00.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:00.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:00.493 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:00.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:00.523 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:00.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:00.624 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:00.664 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:00.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:00.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:00.766 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:00.795 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:00.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:00.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:00.897 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:00.933 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:00.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:00.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:00.988 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-17T18:21:01.035 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:01.074 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:01.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:01.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:01.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:01.203 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:01.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:01.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:01.208 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 1 16 3334 3318 207.364 207.375 0.00438794 0.00480582 2024-08-17T18:21:01.305 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:01.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:01.346 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:01.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:01.447 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:01.487 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:01.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:01.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:01.589 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:01.627 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:01.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:01.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:01.729 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:01.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:01.771 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:01.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:01.863 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:21:01.866 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:21:01.872 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:01.921 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:01.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:01.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:02.023 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:02.028 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:01 smithi136 ceph-mon[109861]: pgmap v3312: 99 pgs: 99 active+clean; 582 KiB data, 9.1 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:21:02.029 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:01 smithi136 ceph-mon[115078]: pgmap v3312: 99 pgs: 99 active+clean; 582 KiB data, 9.1 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:21:02.053 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:02.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:02.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:02.154 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:02.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:02.194 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:02.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:02.209 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 2 16 6466 6450 201.547 195.75 0.00559936 0.00495066 2024-08-17T18:21:02.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:01 smithi165 ceph-mon[105517]: pgmap v3312: 99 pgs: 99 active+clean; 582 KiB data, 9.1 GiB used, 706 GiB / 715 GiB avail 2024-08-17T18:21:02.295 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:02.333 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:02.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:02.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:02.435 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:02.463 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:02.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:02.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:02.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:02.591 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:02.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:02.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:02.693 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:02.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:02.733 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:02.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:02.834 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:02.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:02.873 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:02.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:02.974 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:03.002 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:03.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:03.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:03.104 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:03.145 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:03.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:03.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:03.209 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 3 16 9431 9415 196.128 185.312 0.00614694 0.00509073 2024-08-17T18:21:03.247 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:03.285 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:03.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:03.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:03.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:03.416 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:03.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:03.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:03.518 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:03.547 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:03.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:03.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:03.649 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:03.687 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:03.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:03.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:03.788 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:03.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:03.828 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:03.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:03.929 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:03.956 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:03.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:03.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:03.997 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:21:03.997 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:21:03.997 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:21:03.997 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:21:03.998 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:21:03.998 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:21:03.998 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:21:03.998 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:21:03.998 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:21:03.998 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:21:03.998 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:21:03.998 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:21:03.998 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:21:03.998 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:21:03.998 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:21:03.999 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:21:04.058 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:04.096 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:04.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:04.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:04.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:03 smithi136 ceph-mon[109861]: pgmap v3313: 99 pgs: 99 active+clean; 293 MiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 29 MiB/s wr, 468 op/s 2024-08-17T18:21:04.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:03 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2969833064' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:21:04.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:03 smithi136 ceph-mon[115078]: pgmap v3313: 99 pgs: 99 active+clean; 293 MiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 29 MiB/s wr, 468 op/s 2024-08-17T18:21:04.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:03 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2969833064' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:21:04.198 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:04.209 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 4 16 12121 12105 189.123 168.125 0.0112439 0.00527573 2024-08-17T18:21:04.241 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:04.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:04.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:04.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:03 smithi165 ceph-mon[105517]: pgmap v3313: 99 pgs: 99 active+clean; 293 MiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 29 MiB/s wr, 468 op/s 2024-08-17T18:21:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:03 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2969833064' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:21:04.343 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:04.384 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:04.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:04.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:04.486 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:04.514 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:04.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:04.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:04.616 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:04.645 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:04.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:04.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:04.747 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:04.787 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:04.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:04.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:04.889 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:04.924 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:04.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:04.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:05.026 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:05.064 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:05.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:05.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:05.166 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:05.203 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:05.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:05.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:05.209 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 5 16 14896 14880 185.983 173.438 0.00641879 0.00537104 2024-08-17T18:21:05.305 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:05.345 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:05.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:05.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:05.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:05.475 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:05.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:05.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:05.577 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:05.603 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:05.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:05.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:05.705 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:05.734 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:05.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:05.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:05.836 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:05.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:05.874 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:05.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:05.975 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:06.002 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:06.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:06.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:06.007 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:21:06.104 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:06.130 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:05 smithi136 ceph-mon[109861]: pgmap v3314: 99 pgs: 99 active+clean; 293 MiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 29 MiB/s wr, 468 op/s 2024-08-17T18:21:06.131 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:05 smithi136 ceph-mon[115078]: pgmap v3314: 99 pgs: 99 active+clean; 293 MiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 29 MiB/s wr, 468 op/s 2024-08-17T18:21:06.133 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:06.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:06.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:06.209 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 6 16 17266 17250 179.67 148.125 0.00331138 0.00547821 2024-08-17T18:21:06.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:06.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:05 smithi165 ceph-mon[105517]: pgmap v3314: 99 pgs: 99 active+clean; 293 MiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 29 MiB/s wr, 468 op/s 2024-08-17T18:21:06.266 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:06.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:06.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:06.372 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:06.401 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:06.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:06.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:06.503 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:06.540 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:06.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:06.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:06.643 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:06.670 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:06.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:06.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:06.758 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:21:06.772 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:06.843 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:06.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:06.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:06.945 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:06.992 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:06.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:06.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:07.094 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:07.138 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:07.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:07.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:07.209 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 7 16 19671 19655 175.474 150.312 0.0041729 0.00569107 2024-08-17T18:21:07.240 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:07.277 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:07.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:07.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:07.379 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:07.407 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:07.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:07.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:07.509 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:21:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:21:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:07 smithi165 ceph-mon[105517]: pgmap v3315: 99 pgs: 99 active+clean; 677 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-08-17T18:21:07.564 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:07.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:07.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:07.666 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:07.673 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:21:07.674 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:21:07.674 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:07 smithi136 ceph-mon[109861]: pgmap v3315: 99 pgs: 99 active+clean; 677 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-08-17T18:21:07.674 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:21:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:21:07] "GET /metrics HTTP/1.1" 200 36725 "" "Prometheus/2.51.0" 2024-08-17T18:21:07.674 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:21:07.674 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:21:07.674 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:07 smithi136 ceph-mon[115078]: pgmap v3315: 99 pgs: 99 active+clean; 677 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-08-17T18:21:07.692 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:07.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:07.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:07.794 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:07.843 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:07.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:07.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:07.945 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:07.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:07.983 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:07.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:08.084 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:08.112 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:08.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:08.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:08.209 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 8 16 21975 21959 171.538 144 0.00993305 0.0058244 2024-08-17T18:21:08.214 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:08.242 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:08.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:08.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:08.343 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:08.371 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:08.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:08.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:08.473 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:08.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:08.516 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:08.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:08.617 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:08.658 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:08.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:08.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:08.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:08.788 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:08.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:08.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:08.890 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:08.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:08.932 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:08.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:09.033 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:09.058 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:09.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:09.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:09.160 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:09.189 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:09.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:09.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:09.209 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 9 16 23861 23845 165.574 117.875 0.0113725 0.00603379 2024-08-17T18:21:09.291 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:09.318 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:09.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:09.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:09.420 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:09.457 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:09.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:09.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:09.559 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:09.580 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:21:09.594 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:09.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:09.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:09.696 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:21:09.697 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:09.702 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T18:21:09.708 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:21:09.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:09.723 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:09.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:09.824 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:09.849 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:09.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:09.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:09 smithi136 ceph-mon[109861]: pgmap v3316: 99 pgs: 99 active+clean; 1.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-08-17T18:21:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:09 smithi136 ceph-mon[109861]: from='client.36945 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:21:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:09 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3277393497' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:21:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:09 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:21:09.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:09 smithi136 ceph-mon[115078]: pgmap v3316: 99 pgs: 99 active+clean; 1.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-08-17T18:21:09.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:09 smithi136 ceph-mon[115078]: from='client.36945 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:21:09.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:09 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3277393497' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:21:09.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:09 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:21:09.951 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:09.972 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:09.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:09.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:10.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:09 smithi165 ceph-mon[105517]: pgmap v3316: 99 pgs: 99 active+clean; 1.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-08-17T18:21:10.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:09 smithi165 ceph-mon[105517]: from='client.36945 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:21:10.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:09 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3277393497' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:21:10.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:09 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:21:10.075 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:10.096 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:10.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:10.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:10.198 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:10.209 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 10 16 25608 25592 159.935 109.188 0.00218043 0.00623266 2024-08-17T18:21:10.221 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:10.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:10.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:10.323 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:10.367 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:10.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:10.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:10.469 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:10.509 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:10.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:10.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:10.611 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:10.647 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:10.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:10.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:10.749 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:10.776 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:10.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:10.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:10.878 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:10.898 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:10.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:10.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:10.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:10 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:21:10.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:10 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:21:10.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:10 smithi136 ceph-mon[109861]: osdmap e1707: 8 total, 8 up, 8 in 2024-08-17T18:21:10.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:10 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:21:10.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:10 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:21:10.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:10 smithi136 ceph-mon[115078]: osdmap e1707: 8 total, 8 up, 8 in 2024-08-17T18:21:11.000 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:11.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:10 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:21:11.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:10 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:21:11.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:10 smithi165 ceph-mon[105517]: osdmap e1707: 8 total, 8 up, 8 in 2024-08-17T18:21:11.039 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:11.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:11.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:11.141 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:11.176 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:11.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:11.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:11.209 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 11 16 27353 27337 155.309 109.062 0.00242212 0.00643158 2024-08-17T18:21:11.279 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:11.308 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:11.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:11.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:11.410 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:11.437 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:11.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:11.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:11.539 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:11.570 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:11.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:11.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:11.672 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:11.693 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:11.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:11.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:11.795 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:11.833 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:11.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:11.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:11.936 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:11.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:11 smithi136 ceph-mon[109861]: pgmap v3318: 99 pgs: 99 active+clean; 1.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-08-17T18:21:11.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [1, 2]} : dispatch 2024-08-17T18:21:11.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]} : dispatch 2024-08-17T18:21:11.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 6]} : dispatch 2024-08-17T18:21:11.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 6]} : dispatch 2024-08-17T18:21:11.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]} : dispatch 2024-08-17T18:21:11.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-08-17T18:21:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:11 smithi136 ceph-mon[115078]: pgmap v3318: 99 pgs: 99 active+clean; 1.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-08-17T18:21:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:11 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [1, 2]} : dispatch 2024-08-17T18:21:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:11 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]} : dispatch 2024-08-17T18:21:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:11 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 6]} : dispatch 2024-08-17T18:21:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:11 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 6]} : dispatch 2024-08-17T18:21:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:11 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]} : dispatch 2024-08-17T18:21:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:11 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-08-17T18:21:11.974 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:11.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:11.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:11 smithi165 ceph-mon[105517]: pgmap v3318: 99 pgs: 99 active+clean; 1.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-08-17T18:21:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:11 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [1, 2]} : dispatch 2024-08-17T18:21:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:11 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]} : dispatch 2024-08-17T18:21:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:11 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 6]} : dispatch 2024-08-17T18:21:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:11 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 6]} : dispatch 2024-08-17T18:21:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:11 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]} : dispatch 2024-08-17T18:21:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:11 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-08-17T18:21:12.076 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:12.097 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:12.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:12.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:12.199 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:12.210 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 12 16 28646 28630 149.1 80.8125 0.00393556 0.00670104 2024-08-17T18:21:12.235 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:12.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:12.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:12.337 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:12.366 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:12.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:12.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:12.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:12.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:12.510 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:12.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:12.611 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:12.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:12.653 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:12.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:12.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:12.779 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:12.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:12.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:12.881 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:12.905 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:12.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:12.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:12.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [1, 2]}]': finished 2024-08-17T18:21:12.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]': finished 2024-08-17T18:21:12.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 6]}]': finished 2024-08-17T18:21:12.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 6]}]': finished 2024-08-17T18:21:12.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-08-17T18:21:12.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-08-17T18:21:12.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:12 smithi136 ceph-mon[109861]: osdmap e1708: 8 total, 8 up, 8 in 2024-08-17T18:21:12.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [1, 2]}]': finished 2024-08-17T18:21:12.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]': finished 2024-08-17T18:21:12.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 6]}]': finished 2024-08-17T18:21:12.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 6]}]': finished 2024-08-17T18:21:12.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-08-17T18:21:12.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-08-17T18:21:12.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:12 smithi136 ceph-mon[115078]: osdmap e1708: 8 total, 8 up, 8 in 2024-08-17T18:21:13.007 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:13.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:12 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [1, 2]}]': finished 2024-08-17T18:21:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:12 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]': finished 2024-08-17T18:21:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:12 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 6]}]': finished 2024-08-17T18:21:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:12 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 6]}]': finished 2024-08-17T18:21:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:12 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-08-17T18:21:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:12 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-08-17T18:21:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:12 smithi165 ceph-mon[105517]: osdmap e1708: 8 total, 8 up, 8 in 2024-08-17T18:21:13.042 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:13.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:13.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:13.144 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:13.183 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:13.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:13.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:13.210 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 13 16 30444 30428 146.275 112.375 0.00419458 0.00682913 2024-08-17T18:21:13.285 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:13.317 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:13.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:13.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:13.419 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:13.447 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:13.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:13.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:13.549 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:13.578 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:13.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:13.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:13.680 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:13.716 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:13.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:13.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:13.818 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:13.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:13.857 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:13.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:13 smithi136 ceph-mon[109861]: pgmap v3320: 99 pgs: 6 peering, 93 active+clean; 1.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 181 MiB/s wr, 2.90k op/s 2024-08-17T18:21:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:13 smithi136 ceph-mon[109861]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-08-17T18:21:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:13 smithi136 ceph-mon[109861]: osdmap e1709: 8 total, 8 up, 8 in 2024-08-17T18:21:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:13 smithi136 ceph-mon[115078]: pgmap v3320: 99 pgs: 6 peering, 93 active+clean; 1.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 181 MiB/s wr, 2.90k op/s 2024-08-17T18:21:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:13 smithi136 ceph-mon[115078]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-08-17T18:21:13.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:13 smithi136 ceph-mon[115078]: osdmap e1709: 8 total, 8 up, 8 in 2024-08-17T18:21:13.958 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:13.985 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:13.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:13.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:13 smithi165 ceph-mon[105517]: pgmap v3320: 99 pgs: 6 peering, 93 active+clean; 1.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 181 MiB/s wr, 2.90k op/s 2024-08-17T18:21:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:13 smithi165 ceph-mon[105517]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-08-17T18:21:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:13 smithi165 ceph-mon[105517]: osdmap e1709: 8 total, 8 up, 8 in 2024-08-17T18:21:14.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:14.115 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:14.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:14.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:14.210 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 14 16 32321 32305 144.205 117.312 0.00376197 0.00692919 2024-08-17T18:21:14.216 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:14.244 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:14.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:14.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:14.346 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:14.373 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:14.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:14.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:14.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:14.503 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:14.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:14.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:14.605 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:14.628 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:14.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:14.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:14.730 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:14.768 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:14.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:14.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:14.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:14.906 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:14.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:14.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:15.008 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:15.035 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:15.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:15.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:15.137 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:15.278 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 15 16 34134 34118 142.145 113.312 0.00783317 0.00702981 2024-08-17T18:21:15.313 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:15.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:15.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:15.415 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:15.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:15.455 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:15.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:15.556 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:15.596 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:15.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:15.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:15.698 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:15.720 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:15.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:15.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:15.822 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:15.857 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:15.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:15.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:15.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:15 smithi136 ceph-mon[109861]: pgmap v3322: 99 pgs: 6 peering, 93 active+clean; 1.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-08-17T18:21:15.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:15 smithi136 ceph-mon[115078]: pgmap v3322: 99 pgs: 6 peering, 93 active+clean; 1.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-08-17T18:21:15.960 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:15.986 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:15.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:15.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:16.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:15 smithi165 ceph-mon[105517]: pgmap v3322: 99 pgs: 6 peering, 93 active+clean; 1.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-08-17T18:21:16.088 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:16.125 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:16.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:16.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:16.210 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 16 16 35774 35758 139.667 102.5 0.00360482 0.00714799 2024-08-17T18:21:16.227 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:16.263 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:16.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:16.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:16.365 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:16.401 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:16.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:16.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:16.503 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:16.531 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:16.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:16.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:16.633 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:16.659 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:16.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:16.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:16.761 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:16.797 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:16.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:16.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:16.899 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:16.928 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:16.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:16.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:17.031 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:17.056 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:17.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:17.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:17.158 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:17.184 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:17.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:17.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:17.210 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 17 16 37435 37419 137.557 103.812 0.0052034 0.00726423 2024-08-17T18:21:17.287 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:17.315 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:17.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:17.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:17.417 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:17.456 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:17.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:17.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:17.558 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:17.598 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:17.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:17.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:17.700 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:17.729 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:17.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:17.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:17.748 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:17 smithi136 ceph-mon[109861]: pgmap v3323: 99 pgs: 6 peering, 93 active+clean; 1.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-08-17T18:21:17.748 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:17 smithi136 ceph-mon[115078]: pgmap v3323: 99 pgs: 6 peering, 93 active+clean; 1.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-08-17T18:21:17.748 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:21:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:21:17] "GET /metrics HTTP/1.1" 200 36725 "" "Prometheus/2.51.0" 2024-08-17T18:21:17.831 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:17.858 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:17.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:17.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:17.960 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:17.986 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:17.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:17.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:18.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:17 smithi165 ceph-mon[105517]: pgmap v3323: 99 pgs: 6 peering, 93 active+clean; 1.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-08-17T18:21:18.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:18.115 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:18.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:18.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:18.210 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 18 16 39232 39216 136.154 112.312 0.0057183 0.00733939 2024-08-17T18:21:18.217 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:18.257 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:18.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:18.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:18.359 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:18.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:18.404 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:18.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:18.506 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:18.529 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:18.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:18.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:18.631 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:18.668 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:18.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:18.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:18.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:18.793 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:18.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:18.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:18.895 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:18.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:18.934 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:18.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:18.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:18 smithi136 ceph-mon[109861]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-08-17T18:21:18.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:18 smithi136 ceph-mon[115078]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-08-17T18:21:19.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:18 smithi165 ceph-mon[105517]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-08-17T18:21:19.034 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:19.085 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:19.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:19.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:19.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:19.210 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 19 16 40667 40651 133.708 89.6875 0.00618208 0.00747338 2024-08-17T18:21:19.214 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:19.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:19.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:19.316 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:19.347 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:19.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:19.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:19.449 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:19.484 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:19.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:19.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:19.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:19.610 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:19.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:19.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:19.712 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:19.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:19.761 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:19.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:19.862 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:19.899 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:19.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:19.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:19.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:19 smithi136 ceph-mon[109861]: pgmap v3324: 99 pgs: 99 active+clean; 2.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 132 MiB/s wr, 2.11k op/s 2024-08-17T18:21:19.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:19 smithi136 ceph-mon[115078]: pgmap v3324: 99 pgs: 99 active+clean; 2.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 132 MiB/s wr, 2.11k op/s 2024-08-17T18:21:20.000 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:20.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:19 smithi165 ceph-mon[105517]: pgmap v3324: 99 pgs: 99 active+clean; 2.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 132 MiB/s wr, 2.11k op/s 2024-08-17T18:21:20.041 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:20.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:20.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:20.143 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:20.177 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:20.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:20.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:20.210 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:2024-08-17T18:21:20.211045+0000 min lat: 0.00120749 max lat: 0.207961 avg lat: 0.00758129 2024-08-17T18:21:20.211 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T18:21:20.211 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 20 16 42191 42175 131.785 95.25 0.0100228 0.00758129 2024-08-17T18:21:20.279 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:20.306 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:20.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:20.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:20.408 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:20.446 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:20.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:20.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:20.548 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:20.578 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:20.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:20.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:20.680 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:20.722 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:20.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:20.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:20.824 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:20.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:20.865 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:20.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:20.966 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:20.990 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:20.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:20.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:21.025 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:21:21.092 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:21.117 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:21.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:21.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:21.211 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 21 16 43734 43718 130.101 96.4375 0.0084515 0.00768072 2024-08-17T18:21:21.219 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:21.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:21.287 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:21.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:21.298 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":3316,"stamp":"2024-08-17T18:21:08.519211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1195382757,"num_objects":18257,"num_object_clones":0,"num_object_copies":54771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18295,"num_write_kb":1166817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":930735,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19929,"ondisk_log_size":19929,"up":297,"acting":297,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":308,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":12764472,"kb_used_data":4852408,"kb_used_omap":352,"kb_used_meta":7911583,"kb_avail":737196744,"statfs":{"total":767960285184,"available":754889465856,"internally_reserved":0,"allocated":4968865792,"data_stored":4932181143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":360477,"internal_metadata":8101461987},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1194786816,"num_objects":18231,"num_object_clones":0,"num_object_copies":54693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18231,"num_write_kb":1166784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.754736"},"pg_stats":[{"pgid":"58.f","version":"1706'1111","reported_seq":1132,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:21:06.122065+0000","last_change":"2024-08-17T18:20:43.426604+0000","last_active":"2024-08-17T18:21:06.122065+0000","last_peered":"2024-08-17T18:21:06.122065+0000","last_clean":"2024-08-17T18:21:06.122065+0000","last_became_active":"2024-08-17T18:20:43.426286+0000","last_became_peered":"2024-08-17T18:20:43.426286+0000","last_unstale":"2024-08-17T18:21:06.122065+0000","last_undegraded":"2024-08-17T18:21:06.122065+0000","last_fullsized":"2024-08-17T18:21:06.122065+0000","mapping_epoch":1701,"log_start":"0'0","ondisk_log_start":"0'0","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":1111,"log_dups_size":0,"ondisk_log_size":1111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:31:25.455392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72810496,"num_objects":1111,"num_object_clones":0,"num_object_copies":3333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1111,"num_write_kb":71104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.e","version":"1706'1102","reported_seq":1122,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:21:06.120600+0000","last_change":"2024-08-17T18:20:42.998689+0000","last_active":"2024-08-17T18:21:06.120600+0000","last_peered":"2024-08-17T18:21:06.120600+0000","last_clean":"2024-08-17T18:21:06.120600+0000","last_became_active":"2024-08-17T18:20:42.998159+0000","last_became_peered":"2024-08-17T18:20:42.998159+0000","last_unstale":"2024-08-17T18:21:06.120600+0000","last_undegraded":"2024-08-17T18:21:06.120600+0000","last_fullsized":"2024-08-17T18:21:06.120600+0000","mapping_epoch":1701,"log_start":"0'0","ondisk_log_start":"0'0","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":1102,"log_dups_size":0,"ondisk_log_size":1102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:09:38.837478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72220672,"num_objects":1102,"num_object_clones":0,"num_object_copies":3306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1102,"num_write_kb":70528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.d","version":"1706'1060","reported_seq":1080,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:21:06.111324+0000","last_change":"2024-08-17T18:20:42.999590+0000","last_active":"2024-08-17T18:21:06.111324+0000","last_peered":"2024-08-17T18:21:06.111324+0000","last_clean":"2024-08-17T18:21:06.111324+0000","last_became_active":"2024-08-17T18:20:42.999303+0000","last_became_peered":"2024-08-17T18:20:42.999303+0000","last_unstale":"2024-08-17T18:21:06.111324+0000","last_undegraded":"2024-08-17T18:21:06.111324+0000","last_fullsized":"2024-08-17T18:21:06.111324+0000","mapping_epoch":1701,"log_start":"0'0","ondisk_log_start":"0'0","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":1060,"log_dups_size":0,"ondisk_log_size":1060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:59:22.783318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69468160,"num_objects":1060,"num_object_clones":0,"num_object_copies":3180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1060,"num_write_kb":67840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.c","version":"1706'1258","reported_seq":1279,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:21:07.395163+0000","last_change":"2024-08-17T18:20:43.000893+0000","last_active":"2024-08-17T18:21:07.395163+0000","last_peered":"2024-08-17T18:21:07.395163+0000","last_clean":"2024-08-17T18:21:07.395163+0000","last_became_active":"2024-08-17T18:20:43.000579+0000","last_became_peered":"2024-08-17T18:20:43.000579+0000","last_unstale":"2024-08-17T18:21:07.395163+0000","last_undegraded":"2024-08-17T18:21:07.395163+0000","last_fullsized":"2024-08-17T18:21:07.395163+0000","mapping_epoch":1701,"log_start":"0'0","ondisk_log_start":"0'0","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":1258,"log_dups_size":0,"ondisk_log_size":1258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:00:36.582958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82444288,"num_objects":1258,"num_object_clones":0,"num_object_copies":3774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1258,"num_write_kb":80512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"58.b","version":"1706'1078","reported_seq":1099,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:21:06.112255+0000","last_change":"2024-08-17T18:20:43.737504+0000","last_active":"2024-08-17T18:21:06.112255+0000","last_peered":"2024-08-17T18:21:06.112255+0000","last_clean":"2024-08-17T18:21:06.112255+0000","last_became_active":"2024-08-17T18:20:43.737162+0000","last_became_peered":"2024-08-17T18:20:43.737162+0000","last_unstale":"2024-08-17T18:21:06.112255+0000","last_undegraded":"2024-08-17T18:21:06.112255+0000","last_fullsized":"2024-08-17T18:21:06.112255+0000","mapping_epoch":1701,"log_start":"0'0","ondisk_log_start":"0'0","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":1078,"log_dups_size":0,"ondisk_log_size":1078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:17:40.060918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70647808,"num_objects":1078,"num_object_clones":0,"num_object_copies":3234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1078,"num_write_kb":68992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.a","version":"1706'1241","reported_seq":1262,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:21:07.136594+0000","last_change":"2024-08-17T18:20:43.737650+0000","last_active":"2024-08-17T18:21:07.136594+0000","last_peered":"2024-08-17T18:21:07.136594+0000","last_clean":"2024-08-17T18:21:07.136594+0000","last_became_active":"2024-08-17T18:20:43.737290+0000","last_became_peered":"2024-08-17T18:20:43.737290+0000","last_unstale":"2024-08-17T18:21:07.136594+0000","last_undegraded":"2024-08-17T18:21:07.136594+0000","last_fullsized":"2024-08-17T18:21:07.136594+0000","mapping_epoch":1701,"log_start":"0'0","ondisk_log_start":"0'0","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":1241,"log_dups_size":0,"ondisk_log_size":1241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:44:53.761192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81330176,"num_objects":1241,"num_object_clones":0,"num_object_copies":3723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1241,"num_write_kb":79424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"58.9","version":"1706'1222","reported_seq":1243,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:21:07.138840+0000","last_change":"2024-08-17T18:20:42.999362+0000","last_active":"2024-08-17T18:21:07.138840+0000","last_peered":"2024-08-17T18:21:07.138840+0000","last_clean":"2024-08-17T18:21:07.138840+0000","last_became_active":"2024-08-17T18:20:42.999060+0000","last_became_peered":"2024-08-17T18:20:42.999060+0000","last_unstale":"2024-08-17T18:21:07.138840+0000","last_undegraded":"2024-08-17T18:21:07.138840+0000","last_fullsized":"2024-08-17T18:21:07.138840+0000","mapping_epoch":1701,"log_start":"0'0","ondisk_log_start":"0'0","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":1222,"log_dups_size":0,"ondisk_log_size":1222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:59:42.840375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80084992,"num_objects":1222,"num_object_clones":0,"num_object_copies":3666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1222,"num_write_kb":78208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"58.8","version":"1706'1056","reported_seq":1077,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:21:06.118012+0000","last_change":"2024-08-17T18:20:42.999325+0000","last_active":"2024-08-17T18:21:06.118012+0000","last_peered":"2024-08-17T18:21:06.118012+0000","last_clean":"2024-08-17T18:21:06.118012+0000","last_became_active":"2024-08-17T18:20:42.998979+0000","last_became_peered":"2024-08-17T18:20:42.998979+0000","last_unstale":"2024-08-17T18:21:06.118012+0000","last_undegraded":"2024-08-17T18:21:06.118012+0000","last_fullsized":"2024-08-17T18:21:06.118012+0000","mapping_epoch":1701,"log_start":"0'0","ondisk_log_start":"0'0","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":1056,"log_dups_size":0,"ondisk_log_size":1056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:31:41.072870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69206016,"num_objects":1056,"num_object_clones":0,"num_object_copies":3168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1056,"num_write_kb":67584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.7","version":"1706'1288","reported_seq":1309,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:21:07.379610+0000","last_change":"2024-08-17T18:20:42.999549+0000","last_active":"2024-08-17T18:21:07.379610+0000","last_peered":"2024-08-17T18:21:07.379610+0000","last_clean":"2024-08-17T18:21:07.379610+0000","last_became_active":"2024-08-17T18:20:42.999264+0000","last_became_peered":"2024-08-17T18:20:42.999264+0000","last_unstale":"2024-08-17T18:21:07.379610+0000","last_undegraded":"2024-08-17T18:21:07.379610+0000","last_fullsized":"2024-08-17T18:21:07.379610+0000","mapping_epoch":1701,"log_start":"0'0","ondisk_log_start":"0'0","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":1288,"log_dups_size":0,"ondisk_log_size":1288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:50:23.816496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84410368,"num_objects":1288,"num_object_clones":0,"num_object_copies":3864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1288,"num_write_kb":82432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"58.6","version":"1706'1200","reported_seq":1221,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:21:07.146437+0000","last_change":"2024-08-17T18:20:43.000233+0000","last_active":"2024-08-17T18:21:07.146437+0000","last_peered":"2024-08-17T18:21:07.146437+0000","last_clean":"2024-08-17T18:21:07.146437+0000","last_became_active":"2024-08-17T18:20:42.999927+0000","last_became_peered":"2024-08-17T18:20:42.999927+0000","last_unstale":"2024-08-17T18:21:07.146437+0000","last_undegraded":"2024-08-17T18:21:07.146437+0000","last_fullsized":"2024-08-17T18:21:07.146437+0000","mapping_epoch":1701,"log_start":"0'0","ondisk_log_start":"0'0","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":1200,"log_dups_size":0,"ondisk_log_size":1200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:45:46.168515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78643200,"num_objects":1200,"num_object_clones":0,"num_object_copies":3600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1200,"num_write_kb":76800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"58.5","version":"1706'876","reported_seq":890,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:21:04.980621+0000","last_change":"2024-08-17T18:20:42.999364+0000","last_active":"2024-08-17T18:21:04.980621+0000","last_peered":"2024-08-17T18:21:04.980621+0000","last_clean":"2024-08-17T18:21:04.980621+0000","last_became_active":"2024-08-17T18:20:42.999101+0000","last_became_peered":"2024-08-17T18:20:42.999101+0000","last_unstale":"2024-08-17T18:21:04.980621+0000","last_undegraded":"2024-08-17T18:21:04.980621+0000","last_fullsized":"2024-08-17T18:21:04.980621+0000","mapping_epoch":1701,"log_start":"0'0","ondisk_log_start":"0'0","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":876,"log_dups_size":0,"ondisk_log_size":876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:46:49.192109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57409536,"num_objects":876,"num_object_clones":0,"num_object_copies":2628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":876,"num_write_kb":56064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"58.4","version":"1706'1235","reported_seq":1256,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:21:07.139593+0000","last_change":"2024-08-17T18:20:43.001327+0000","last_active":"2024-08-17T18:21:07.139593+0000","last_peered":"2024-08-17T18:21:07.139593+0000","last_clean":"2024-08-17T18:21:07.139593+0000","last_became_active":"2024-08-17T18:20:43.000942+0000","last_became_peered":"2024-08-17T18:20:43.000942+0000","last_unstale":"2024-08-17T18:21:07.139593+0000","last_undegraded":"2024-08-17T18:21:07.139593+0000","last_fullsized":"2024-08-17T18:21:07.139593+0000","mapping_epoch":1701,"log_start":"0'0","ondisk_log_start":"0'0","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":1235,"log_dups_size":0,"ondisk_log_size":1235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:47:27.757138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80936960,"num_objects":1235,"num_object_clones":0,"num_object_copies":3705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1235,"num_write_kb":79040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"58.3","version":"1706'1196","reported_seq":1217,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:21:07.142275+0000","last_change":"2024-08-17T18:20:43.426768+0000","last_active":"2024-08-17T18:21:07.142275+0000","last_peered":"2024-08-17T18:21:07.142275+0000","last_clean":"2024-08-17T18:21:07.142275+0000","last_became_active":"2024-08-17T18:20:43.426272+0000","last_became_peered":"2024-08-17T18:20:43.426272+0000","last_unstale":"2024-08-17T18:21:07.142275+0000","last_undegraded":"2024-08-17T18:21:07.142275+0000","last_fullsized":"2024-08-17T18:21:07.142275+0000","mapping_epoch":1701,"log_start":"0'0","ondisk_log_start":"0'0","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":1196,"log_dups_size":0,"ondisk_log_size":1196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:29:57.872272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78381056,"num_objects":1196,"num_object_clones":0,"num_object_copies":3588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1196,"num_write_kb":76544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"58.2","version":"1706'1252","reported_seq":1266,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:21:07.282656+0000","last_change":"2024-08-17T18:20:43.001236+0000","last_active":"2024-08-17T18:21:07.282656+0000","last_peered":"2024-08-17T18:21:07.282656+0000","last_clean":"2024-08-17T18:21:07.282656+0000","last_became_active":"2024-08-17T18:20:43.000940+0000","last_became_peered":"2024-08-17T18:20:43.000940+0000","last_unstale":"2024-08-17T18:21:07.282656+0000","last_undegraded":"2024-08-17T18:21:07.282656+0000","last_fullsized":"2024-08-17T18:21:07.282656+0000","mapping_epoch":1701,"log_start":"0'0","ondisk_log_start":"0'0","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":1252,"log_dups_size":0,"ondisk_log_size":1252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:58.629399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82051072,"num_objects":1252,"num_object_clones":0,"num_object_copies":3756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1252,"num_write_kb":80128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"58.1","version":"1706'1182","reported_seq":1196,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:21:07.195734+0000","last_change":"2024-08-17T18:20:43.737526+0000","last_active":"2024-08-17T18:21:07.195734+0000","last_peered":"2024-08-17T18:21:07.195734+0000","last_clean":"2024-08-17T18:21:07.195734+0000","last_became_active":"2024-08-17T18:20:43.737209+0000","last_became_peered":"2024-08-17T18:20:43.737209+0000","last_unstale":"2024-08-17T18:21:07.195734+0000","last_undegraded":"2024-08-17T18:21:07.195734+0000","last_fullsized":"2024-08-17T18:21:07.195734+0000","mapping_epoch":1701,"log_start":"0'0","ondisk_log_start":"0'0","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":1182,"log_dups_size":0,"ondisk_log_size":1182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:22:15.045846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77463552,"num_objects":1182,"num_object_clones":0,"num_object_copies":3546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1182,"num_write_kb":75648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"58.0","version":"1706'874","reported_seq":887,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:21:04.980860+0000","last_change":"2024-08-17T18:20:43.737940+0000","last_active":"2024-08-17T18:21:04.980860+0000","last_peered":"2024-08-17T18:21:04.980860+0000","last_clean":"2024-08-17T18:21:04.980860+0000","last_became_active":"2024-08-17T18:20:43.737219+0000","last_became_peered":"2024-08-17T18:20:43.737219+0000","last_unstale":"2024-08-17T18:21:04.980860+0000","last_undegraded":"2024-08-17T18:21:04.980860+0000","last_fullsized":"2024-08-17T18:21:04.980860+0000","mapping_epoch":1701,"log_start":"0'0","ondisk_log_start":"0'0","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":874,"log_dups_size":0,"ondisk_log_size":874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:28:39.197553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57278464,"num_objects":874,"num_object_clones":0,"num_object_copies":2622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":874,"num_write_kb":55936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":143,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.367176+0000","last_change":"2024-08-17T18:17:22.697769+0000","last_active":"2024-08-17T18:20:49.367176+0000","last_peered":"2024-08-17T18:20:49.367176+0000","last_clean":"2024-08-17T18:20:49.367176+0000","last_became_active":"2024-08-17T18:17:22.688989+0000","last_became_peered":"2024-08-17T18:17:22.688989+0000","last_unstale":"2024-08-17T18:20:49.367176+0000","last_undegraded":"2024-08-17T18:20:49.367176+0000","last_fullsized":"2024-08-17T18:20:49.367176+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:10:04.677811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2328,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1759,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.337723+0000","last_change":"2024-08-17T18:20:31.941175+0000","last_active":"2024-08-17T18:20:49.337723+0000","last_peered":"2024-08-17T18:20:49.337723+0000","last_clean":"2024-08-17T18:20:49.337723+0000","last_became_active":"2024-08-17T18:20:31.940188+0000","last_became_peered":"2024-08-17T18:20:31.940188+0000","last_unstale":"2024-08-17T18:20:49.337723+0000","last_undegraded":"2024-08-17T18:20:49.337723+0000","last_fullsized":"2024-08-17T18:20:49.337723+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:59:47.833655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":98,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:56.766143+0000","last_change":"2024-08-17T18:20:31.934958+0000","last_active":"2024-08-17T18:20:56.766143+0000","last_peered":"2024-08-17T18:20:56.766143+0000","last_clean":"2024-08-17T18:20:56.766143+0000","last_became_active":"2024-08-17T18:20:31.934541+0000","last_became_peered":"2024-08-17T18:20:31.934541+0000","last_unstale":"2024-08-17T18:20:56.766143+0000","last_undegraded":"2024-08-17T18:20:56.766143+0000","last_fullsized":"2024-08-17T18:20:56.766143+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:45:45.634351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":91,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:56.765853+0000","last_change":"2024-08-17T18:18:29.732425+0000","last_active":"2024-08-17T18:20:56.765853+0000","last_peered":"2024-08-17T18:20:56.765853+0000","last_clean":"2024-08-17T18:20:56.765853+0000","last_became_active":"2024-08-17T18:18:29.731726+0000","last_became_peered":"2024-08-17T18:18:29.731726+0000","last_unstale":"2024-08-17T18:20:56.765853+0000","last_undegraded":"2024-08-17T18:20:56.765853+0000","last_fullsized":"2024-08-17T18:20:56.765853+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:01:33.556710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":262,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.367138+0000","last_change":"2024-08-17T18:04:43.777705+0000","last_active":"2024-08-17T18:20:49.367138+0000","last_peered":"2024-08-17T18:20:49.367138+0000","last_clean":"2024-08-17T18:20:49.367138+0000","last_became_active":"2024-08-17T18:04:43.771283+0000","last_became_peered":"2024-08-17T18:04:43.771283+0000","last_unstale":"2024-08-17T18:20:49.367138+0000","last_undegraded":"2024-08-17T18:20:49.367138+0000","last_fullsized":"2024-08-17T18:20:49.367138+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:13.772506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":91,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:56.765877+0000","last_change":"2024-08-17T18:18:29.734730+0000","last_active":"2024-08-17T18:20:56.765877+0000","last_peered":"2024-08-17T18:20:56.765877+0000","last_clean":"2024-08-17T18:20:56.765877+0000","last_became_active":"2024-08-17T18:18:29.734362+0000","last_became_peered":"2024-08-17T18:18:29.734362+0000","last_unstale":"2024-08-17T18:20:56.765877+0000","last_undegraded":"2024-08-17T18:20:56.765877+0000","last_fullsized":"2024-08-17T18:20:56.765877+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:58:48.630762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":288,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.367374+0000","last_change":"2024-08-17T18:20:32.094881+0000","last_active":"2024-08-17T18:20:49.367374+0000","last_peered":"2024-08-17T18:20:49.367374+0000","last_clean":"2024-08-17T18:20:49.367374+0000","last_became_active":"2024-08-17T18:20:32.077429+0000","last_became_peered":"2024-08-17T18:20:32.077429+0000","last_unstale":"2024-08-17T18:20:49.367374+0000","last_undegraded":"2024-08-17T18:20:49.367374+0000","last_fullsized":"2024-08-17T18:20:49.367374+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:08:26.290851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1755,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.337939+0000","last_change":"2024-08-17T18:20:31.943437+0000","last_active":"2024-08-17T18:20:49.337939+0000","last_peered":"2024-08-17T18:20:49.337939+0000","last_clean":"2024-08-17T18:20:49.337939+0000","last_became_active":"2024-08-17T18:20:31.942637+0000","last_became_peered":"2024-08-17T18:20:31.942637+0000","last_unstale":"2024-08-17T18:20:49.337939+0000","last_undegraded":"2024-08-17T18:20:49.337939+0000","last_fullsized":"2024-08-17T18:20:49.337939+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:16:38.845505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":242,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.337982+0000","last_change":"2024-08-17T18:20:31.944329+0000","last_active":"2024-08-17T18:20:49.337982+0000","last_peered":"2024-08-17T18:20:49.337982+0000","last_clean":"2024-08-17T18:20:49.337982+0000","last_became_active":"2024-08-17T18:20:31.943817+0000","last_became_peered":"2024-08-17T18:20:31.943817+0000","last_unstale":"2024-08-17T18:20:49.337982+0000","last_undegraded":"2024-08-17T18:20:49.337982+0000","last_fullsized":"2024-08-17T18:20:49.337982+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:07:23.659833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":32,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:57.827267+0000","last_change":"2024-08-17T18:20:31.938850+0000","last_active":"2024-08-17T18:20:57.827267+0000","last_peered":"2024-08-17T18:20:57.827267+0000","last_clean":"2024-08-17T18:20:57.827267+0000","last_became_active":"2024-08-17T18:20:31.938526+0000","last_became_peered":"2024-08-17T18:20:31.938526+0000","last_unstale":"2024-08-17T18:20:57.827267+0000","last_undegraded":"2024-08-17T18:20:57.827267+0000","last_fullsized":"2024-08-17T18:20:57.827267+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:51:37.118079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1750,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.337944+0000","last_change":"2024-08-17T18:20:31.942863+0000","last_active":"2024-08-17T18:20:49.337944+0000","last_peered":"2024-08-17T18:20:49.337944+0000","last_clean":"2024-08-17T18:20:49.337944+0000","last_became_active":"2024-08-17T18:20:31.941281+0000","last_became_peered":"2024-08-17T18:20:31.941281+0000","last_unstale":"2024-08-17T18:20:49.337944+0000","last_undegraded":"2024-08-17T18:20:49.337944+0000","last_fullsized":"2024-08-17T18:20:49.337944+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:48:05.556975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":98,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:56.766268+0000","last_change":"2024-08-17T18:20:31.933142+0000","last_active":"2024-08-17T18:20:56.766268+0000","last_peered":"2024-08-17T18:20:56.766268+0000","last_clean":"2024-08-17T18:20:56.766268+0000","last_became_active":"2024-08-17T18:20:31.932798+0000","last_became_peered":"2024-08-17T18:20:31.932798+0000","last_unstale":"2024-08-17T18:20:56.766268+0000","last_undegraded":"2024-08-17T18:20:56.766268+0000","last_fullsized":"2024-08-17T18:20:56.766268+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:32:51.059352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":32,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:57.826522+0000","last_change":"2024-08-17T18:20:31.937627+0000","last_active":"2024-08-17T18:20:57.826522+0000","last_peered":"2024-08-17T18:20:57.826522+0000","last_clean":"2024-08-17T18:20:57.826522+0000","last_became_active":"2024-08-17T18:20:31.937361+0000","last_became_peered":"2024-08-17T18:20:31.937361+0000","last_unstale":"2024-08-17T18:20:57.826522+0000","last_undegraded":"2024-08-17T18:20:57.826522+0000","last_fullsized":"2024-08-17T18:20:57.826522+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:01:26.309656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":98,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:56.766540+0000","last_change":"2024-08-17T18:20:31.935139+0000","last_active":"2024-08-17T18:20:56.766540+0000","last_peered":"2024-08-17T18:20:56.766540+0000","last_clean":"2024-08-17T18:20:56.766540+0000","last_became_active":"2024-08-17T18:20:31.934773+0000","last_became_peered":"2024-08-17T18:20:31.934773+0000","last_unstale":"2024-08-17T18:20:56.766540+0000","last_undegraded":"2024-08-17T18:20:56.766540+0000","last_fullsized":"2024-08-17T18:20:56.766540+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:36:58.469978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":98,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:56.765986+0000","last_change":"2024-08-17T18:20:31.934190+0000","last_active":"2024-08-17T18:20:56.765986+0000","last_peered":"2024-08-17T18:20:56.765986+0000","last_clean":"2024-08-17T18:20:56.765986+0000","last_became_active":"2024-08-17T18:20:31.933832+0000","last_became_peered":"2024-08-17T18:20:31.933832+0000","last_unstale":"2024-08-17T18:20:56.765986+0000","last_undegraded":"2024-08-17T18:20:56.765986+0000","last_fullsized":"2024-08-17T18:20:56.765986+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:04:17.792039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1761,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:52.217035+0000","last_change":"2024-08-17T18:20:31.935367+0000","last_active":"2024-08-17T18:20:52.217035+0000","last_peered":"2024-08-17T18:20:52.217035+0000","last_clean":"2024-08-17T18:20:52.217035+0000","last_became_active":"2024-08-17T18:20:31.935040+0000","last_became_peered":"2024-08-17T18:20:31.935040+0000","last_unstale":"2024-08-17T18:20:52.217035+0000","last_undegraded":"2024-08-17T18:20:52.217035+0000","last_fullsized":"2024-08-17T18:20:52.217035+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:51:36.444839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":98,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:56.766035+0000","last_change":"2024-08-17T18:20:31.926368+0000","last_active":"2024-08-17T18:20:56.766035+0000","last_peered":"2024-08-17T18:20:56.766035+0000","last_clean":"2024-08-17T18:20:56.766035+0000","last_became_active":"2024-08-17T18:20:31.925869+0000","last_became_peered":"2024-08-17T18:20:31.925869+0000","last_unstale":"2024-08-17T18:20:56.766035+0000","last_undegraded":"2024-08-17T18:20:56.766035+0000","last_fullsized":"2024-08-17T18:20:56.766035+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:56:09.271660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":442,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.367268+0000","last_change":"2024-08-17T18:20:31.938425+0000","last_active":"2024-08-17T18:20:49.367268+0000","last_peered":"2024-08-17T18:20:49.367268+0000","last_clean":"2024-08-17T18:20:49.367268+0000","last_became_active":"2024-08-17T18:20:31.938119+0000","last_became_peered":"2024-08-17T18:20:31.938119+0000","last_unstale":"2024-08-17T18:20:49.367268+0000","last_undegraded":"2024-08-17T18:20:49.367268+0000","last_fullsized":"2024-08-17T18:20:49.367268+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:00:05.040959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":91,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:56.765670+0000","last_change":"2024-08-17T18:18:29.880752+0000","last_active":"2024-08-17T18:20:56.765670+0000","last_peered":"2024-08-17T18:20:56.765670+0000","last_clean":"2024-08-17T18:20:56.765670+0000","last_became_active":"2024-08-17T18:18:29.880508+0000","last_became_peered":"2024-08-17T18:18:29.880508+0000","last_unstale":"2024-08-17T18:20:56.765670+0000","last_undegraded":"2024-08-17T18:20:56.765670+0000","last_fullsized":"2024-08-17T18:20:56.765670+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:35:25.743073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":329,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.367547+0000","last_change":"2024-08-17T18:20:31.939447+0000","last_active":"2024-08-17T18:20:49.367547+0000","last_peered":"2024-08-17T18:20:49.367547+0000","last_clean":"2024-08-17T18:20:49.367547+0000","last_became_active":"2024-08-17T18:20:31.939138+0000","last_became_peered":"2024-08-17T18:20:31.939138+0000","last_unstale":"2024-08-17T18:20:49.367547+0000","last_undegraded":"2024-08-17T18:20:49.367547+0000","last_fullsized":"2024-08-17T18:20:49.367547+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:02:08.378299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":97,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:56.766183+0000","last_change":"2024-08-17T18:18:29.928751+0000","last_active":"2024-08-17T18:20:56.766183+0000","last_peered":"2024-08-17T18:20:56.766183+0000","last_clean":"2024-08-17T18:20:56.766183+0000","last_became_active":"2024-08-17T18:18:29.898102+0000","last_became_peered":"2024-08-17T18:18:29.898102+0000","last_unstale":"2024-08-17T18:20:56.766183+0000","last_undegraded":"2024-08-17T18:20:56.766183+0000","last_fullsized":"2024-08-17T18:20:56.766183+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:21.098266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":32,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:57.826757+0000","last_change":"2024-08-17T18:20:31.931746+0000","last_active":"2024-08-17T18:20:57.826757+0000","last_peered":"2024-08-17T18:20:57.826757+0000","last_clean":"2024-08-17T18:20:57.826757+0000","last_became_active":"2024-08-17T18:20:31.931417+0000","last_became_peered":"2024-08-17T18:20:31.931417+0000","last_unstale":"2024-08-17T18:20:57.826757+0000","last_undegraded":"2024-08-17T18:20:57.826757+0000","last_fullsized":"2024-08-17T18:20:57.826757+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:19:12.330754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1735,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.337529+0000","last_change":"2024-08-17T18:15:53.687128+0000","last_active":"2024-08-17T18:20:49.337529+0000","last_peered":"2024-08-17T18:20:49.337529+0000","last_clean":"2024-08-17T18:20:49.337529+0000","last_became_active":"2024-08-17T18:15:53.246153+0000","last_became_peered":"2024-08-17T18:15:53.246153+0000","last_unstale":"2024-08-17T18:20:49.337529+0000","last_undegraded":"2024-08-17T18:20:49.337529+0000","last_fullsized":"2024-08-17T18:20:49.337529+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:45:14.801100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1706,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.338019+0000","last_change":"2024-08-17T18:20:31.943017+0000","last_active":"2024-08-17T18:20:49.338019+0000","last_peered":"2024-08-17T18:20:49.338019+0000","last_clean":"2024-08-17T18:20:49.338019+0000","last_became_active":"2024-08-17T18:20:31.940315+0000","last_became_peered":"2024-08-17T18:20:31.940315+0000","last_unstale":"2024-08-17T18:20:49.338019+0000","last_undegraded":"2024-08-17T18:20:49.338019+0000","last_fullsized":"2024-08-17T18:20:49.338019+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:05:21.574452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"347'149","reported_seq":918,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:56.765788+0000","last_change":"2024-08-17T18:20:33.927014+0000","last_active":"2024-08-17T18:20:56.765788+0000","last_peered":"2024-08-17T18:20:56.765788+0000","last_clean":"2024-08-17T18:20:56.765788+0000","last_became_active":"2024-08-17T18:20:33.926619+0000","last_became_peered":"2024-08-17T18:20:33.926619+0000","last_unstale":"2024-08-17T18:20:56.765788+0000","last_undegraded":"2024-08-17T18:20:56.765788+0000","last_fullsized":"2024-08-17T18:20:56.765788+0000","mapping_epoch":1697,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1698,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:57.271286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:57.271286+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:32:03.547828+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1713,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.984446+0000","last_change":"2024-08-17T18:04:43.766724+0000","last_active":"2024-08-17T18:20:49.984446+0000","last_peered":"2024-08-17T18:20:49.984446+0000","last_clean":"2024-08-17T18:20:49.984446+0000","last_became_active":"2024-08-17T18:04:43.766428+0000","last_became_peered":"2024-08-17T18:04:43.766428+0000","last_unstale":"2024-08-17T18:20:49.984446+0000","last_undegraded":"2024-08-17T18:20:49.984446+0000","last_fullsized":"2024-08-17T18:20:49.984446+0000","mapping_epoch":1464,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:09:49.102940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1743,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.367361+0000","last_change":"2024-08-17T18:15:53.366349+0000","last_active":"2024-08-17T18:20:49.367361+0000","last_peered":"2024-08-17T18:20:49.367361+0000","last_clean":"2024-08-17T18:20:49.367361+0000","last_became_active":"2024-08-17T18:15:53.365863+0000","last_became_peered":"2024-08-17T18:15:53.365863+0000","last_unstale":"2024-08-17T18:20:49.367361+0000","last_undegraded":"2024-08-17T18:20:49.367361+0000","last_fullsized":"2024-08-17T18:20:49.367361+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:52.700374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":36,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:57.827347+0000","last_change":"2024-08-17T18:20:32.074166+0000","last_active":"2024-08-17T18:20:57.827347+0000","last_peered":"2024-08-17T18:20:57.827347+0000","last_clean":"2024-08-17T18:20:57.827347+0000","last_became_active":"2024-08-17T18:20:32.003273+0000","last_became_peered":"2024-08-17T18:20:32.003273+0000","last_unstale":"2024-08-17T18:20:57.827347+0000","last_undegraded":"2024-08-17T18:20:57.827347+0000","last_fullsized":"2024-08-17T18:20:57.827347+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:16:07.701801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"347'149","reported_seq":999,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.338192+0000","last_change":"2024-08-17T18:17:25.672410+0000","last_active":"2024-08-17T18:20:49.338192+0000","last_peered":"2024-08-17T18:20:49.338192+0000","last_clean":"2024-08-17T18:20:49.338192+0000","last_became_active":"2024-08-17T18:17:25.671993+0000","last_became_peered":"2024-08-17T18:17:25.671993+0000","last_unstale":"2024-08-17T18:20:49.338192+0000","last_undegraded":"2024-08-17T18:20:49.338192+0000","last_fullsized":"2024-08-17T18:20:49.338192+0000","mapping_epoch":1655,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1656,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:53.309517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:53.309517+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:21:08.929478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":98,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:56.766297+0000","last_change":"2024-08-17T18:20:31.936022+0000","last_active":"2024-08-17T18:20:56.766297+0000","last_peered":"2024-08-17T18:20:56.766297+0000","last_clean":"2024-08-17T18:20:56.766297+0000","last_became_active":"2024-08-17T18:20:31.934784+0000","last_became_peered":"2024-08-17T18:20:31.934784+0000","last_unstale":"2024-08-17T18:20:56.766297+0000","last_undegraded":"2024-08-17T18:20:56.766297+0000","last_fullsized":"2024-08-17T18:20:56.766297+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:10.079366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":32,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:57.827373+0000","last_change":"2024-08-17T18:20:32.003736+0000","last_active":"2024-08-17T18:20:57.827373+0000","last_peered":"2024-08-17T18:20:57.827373+0000","last_clean":"2024-08-17T18:20:57.827373+0000","last_became_active":"2024-08-17T18:20:31.952789+0000","last_became_peered":"2024-08-17T18:20:31.952789+0000","last_unstale":"2024-08-17T18:20:57.827373+0000","last_undegraded":"2024-08-17T18:20:57.827373+0000","last_fullsized":"2024-08-17T18:20:57.827373+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:32.636726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":96,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:56.766301+0000","last_change":"2024-08-17T18:18:29.904137+0000","last_active":"2024-08-17T18:20:56.766301+0000","last_peered":"2024-08-17T18:20:56.766301+0000","last_clean":"2024-08-17T18:20:56.766301+0000","last_became_active":"2024-08-17T18:18:29.894843+0000","last_became_peered":"2024-08-17T18:18:29.894843+0000","last_unstale":"2024-08-17T18:20:56.766301+0000","last_undegraded":"2024-08-17T18:20:56.766301+0000","last_fullsized":"2024-08-17T18:20:56.766301+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:23:04.104922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"347'149","reported_seq":1033,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.366819+0000","last_change":"2024-08-17T18:18:33.761955+0000","last_active":"2024-08-17T18:20:49.366819+0000","last_peered":"2024-08-17T18:20:49.366819+0000","last_clean":"2024-08-17T18:20:49.366819+0000","last_became_active":"2024-08-17T18:18:33.761579+0000","last_became_peered":"2024-08-17T18:18:33.761579+0000","last_unstale":"2024-08-17T18:20:49.366819+0000","last_undegraded":"2024-08-17T18:20:49.366819+0000","last_fullsized":"2024-08-17T18:20:49.366819+0000","mapping_epoch":1670,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1671,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:52.474544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:52.474544+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:35:23.712308+0000","scrub_duration":0.001886326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":331,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.367554+0000","last_change":"2024-08-17T18:20:31.939608+0000","last_active":"2024-08-17T18:20:49.367554+0000","last_peered":"2024-08-17T18:20:49.367554+0000","last_clean":"2024-08-17T18:20:49.367554+0000","last_became_active":"2024-08-17T18:20:31.939304+0000","last_became_peered":"2024-08-17T18:20:31.939304+0000","last_unstale":"2024-08-17T18:20:49.367554+0000","last_undegraded":"2024-08-17T18:20:49.367554+0000","last_fullsized":"2024-08-17T18:20:49.367554+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:26.714728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":89,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:48.866436+0000","last_change":"2024-08-17T18:20:31.940586+0000","last_active":"2024-08-17T18:20:48.866436+0000","last_peered":"2024-08-17T18:20:48.866436+0000","last_clean":"2024-08-17T18:20:48.866436+0000","last_became_active":"2024-08-17T18:20:31.940318+0000","last_became_peered":"2024-08-17T18:20:31.940318+0000","last_unstale":"2024-08-17T18:20:48.866436+0000","last_undegraded":"2024-08-17T18:20:48.866436+0000","last_fullsized":"2024-08-17T18:20:48.866436+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:51:59.436474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":32,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:57.827080+0000","last_change":"2024-08-17T18:20:32.012874+0000","last_active":"2024-08-17T18:20:57.827080+0000","last_peered":"2024-08-17T18:20:57.827080+0000","last_clean":"2024-08-17T18:20:57.827080+0000","last_became_active":"2024-08-17T18:20:32.012553+0000","last_became_peered":"2024-08-17T18:20:32.012553+0000","last_unstale":"2024-08-17T18:20:57.827080+0000","last_undegraded":"2024-08-17T18:20:57.827080+0000","last_fullsized":"2024-08-17T18:20:57.827080+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:14:17.188745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"347'149","reported_seq":1016,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.367163+0000","last_change":"2024-08-17T18:18:33.761996+0000","last_active":"2024-08-17T18:20:49.367163+0000","last_peered":"2024-08-17T18:20:49.367163+0000","last_clean":"2024-08-17T18:20:49.367163+0000","last_became_active":"2024-08-17T18:18:33.761606+0000","last_became_peered":"2024-08-17T18:18:33.761606+0000","last_unstale":"2024-08-17T18:20:49.367163+0000","last_undegraded":"2024-08-17T18:20:49.367163+0000","last_fullsized":"2024-08-17T18:20:49.367163+0000","mapping_epoch":1670,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1671,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:49.461698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:49.461698+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:29:01.698411+0000","scrub_duration":0.0013792520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":32,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:57.826821+0000","last_change":"2024-08-17T18:20:31.938437+0000","last_active":"2024-08-17T18:20:57.826821+0000","last_peered":"2024-08-17T18:20:57.826821+0000","last_clean":"2024-08-17T18:20:57.826821+0000","last_became_active":"2024-08-17T18:20:31.938056+0000","last_became_peered":"2024-08-17T18:20:31.938056+0000","last_unstale":"2024-08-17T18:20:57.826821+0000","last_undegraded":"2024-08-17T18:20:57.826821+0000","last_fullsized":"2024-08-17T18:20:57.826821+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:56:04.931588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:57.827366+0000","last_change":"2024-08-17T18:20:32.014875+0000","last_active":"2024-08-17T18:20:57.827366+0000","last_peered":"2024-08-17T18:20:57.827366+0000","last_clean":"2024-08-17T18:20:57.827366+0000","last_became_active":"2024-08-17T18:20:32.014631+0000","last_became_peered":"2024-08-17T18:20:32.014631+0000","last_unstale":"2024-08-17T18:20:57.827366+0000","last_undegraded":"2024-08-17T18:20:57.827366+0000","last_fullsized":"2024-08-17T18:20:57.827366+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:00:50.978635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":277,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.367231+0000","last_change":"2024-08-17T18:18:29.735362+0000","last_active":"2024-08-17T18:20:49.367231+0000","last_peered":"2024-08-17T18:20:49.367231+0000","last_clean":"2024-08-17T18:20:49.367231+0000","last_became_active":"2024-08-17T18:18:29.734752+0000","last_became_peered":"2024-08-17T18:18:29.734752+0000","last_unstale":"2024-08-17T18:20:49.367231+0000","last_undegraded":"2024-08-17T18:20:49.367231+0000","last_fullsized":"2024-08-17T18:20:49.367231+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:25.709698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"347'149","reported_seq":1000,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.366906+0000","last_change":"2024-08-17T18:15:13.681264+0000","last_active":"2024-08-17T18:20:49.366906+0000","last_peered":"2024-08-17T18:20:49.366906+0000","last_clean":"2024-08-17T18:20:49.366906+0000","last_became_active":"2024-08-17T18:15:13.680918+0000","last_became_peered":"2024-08-17T18:15:13.680918+0000","last_unstale":"2024-08-17T18:20:49.366906+0000","last_undegraded":"2024-08-17T18:20:49.366906+0000","last_fullsized":"2024-08-17T18:20:49.366906+0000","mapping_epoch":1629,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1630,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:00.316884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:00.316884+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:37:32.924833+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":32,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:57.827273+0000","last_change":"2024-08-17T18:20:31.929653+0000","last_active":"2024-08-17T18:20:57.827273+0000","last_peered":"2024-08-17T18:20:57.827273+0000","last_clean":"2024-08-17T18:20:57.827273+0000","last_became_active":"2024-08-17T18:20:31.929318+0000","last_became_peered":"2024-08-17T18:20:31.929318+0000","last_unstale":"2024-08-17T18:20:57.827273+0000","last_undegraded":"2024-08-17T18:20:57.827273+0000","last_fullsized":"2024-08-17T18:20:57.827273+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:46.590164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1729,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:57.351759+0000","last_change":"2024-08-17T18:20:31.942070+0000","last_active":"2024-08-17T18:20:57.351759+0000","last_peered":"2024-08-17T18:20:57.351759+0000","last_clean":"2024-08-17T18:20:57.351759+0000","last_became_active":"2024-08-17T18:20:31.939901+0000","last_became_peered":"2024-08-17T18:20:31.939901+0000","last_unstale":"2024-08-17T18:20:57.351759+0000","last_undegraded":"2024-08-17T18:20:57.351759+0000","last_fullsized":"2024-08-17T18:20:57.351759+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:34:09.048600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":331,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.367242+0000","last_change":"2024-08-17T18:20:33.151618+0000","last_active":"2024-08-17T18:20:49.367242+0000","last_peered":"2024-08-17T18:20:49.367242+0000","last_clean":"2024-08-17T18:20:49.367242+0000","last_became_active":"2024-08-17T18:20:31.939252+0000","last_became_peered":"2024-08-17T18:20:31.939252+0000","last_unstale":"2024-08-17T18:20:49.367242+0000","last_undegraded":"2024-08-17T18:20:49.367242+0000","last_fullsized":"2024-08-17T18:20:49.367242+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1697,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:18:17.752443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"347'149","reported_seq":859,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.337801+0000","last_change":"2024-08-17T18:17:22.681616+0000","last_active":"2024-08-17T18:20:49.337801+0000","last_peered":"2024-08-17T18:20:49.337801+0000","last_clean":"2024-08-17T18:20:49.337801+0000","last_became_active":"2024-08-17T18:17:22.681184+0000","last_became_peered":"2024-08-17T18:17:22.681184+0000","last_unstale":"2024-08-17T18:20:49.337801+0000","last_undegraded":"2024-08-17T18:20:49.337801+0000","last_fullsized":"2024-08-17T18:20:49.337801+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:02.363531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:02.363531+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:50:39.567707+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":32,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:57.827176+0000","last_change":"2024-08-17T18:20:31.941405+0000","last_active":"2024-08-17T18:20:57.827176+0000","last_peered":"2024-08-17T18:20:57.827176+0000","last_clean":"2024-08-17T18:20:57.827176+0000","last_became_active":"2024-08-17T18:20:31.940786+0000","last_became_peered":"2024-08-17T18:20:31.940786+0000","last_unstale":"2024-08-17T18:20:57.827176+0000","last_undegraded":"2024-08-17T18:20:57.827176+0000","last_fullsized":"2024-08-17T18:20:57.827176+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:26:47.386304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":32,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:57.826501+0000","last_change":"2024-08-17T18:20:31.927970+0000","last_active":"2024-08-17T18:20:57.826501+0000","last_peered":"2024-08-17T18:20:57.826501+0000","last_clean":"2024-08-17T18:20:57.826501+0000","last_became_active":"2024-08-17T18:20:31.927687+0000","last_became_peered":"2024-08-17T18:20:31.927687+0000","last_unstale":"2024-08-17T18:20:57.826501+0000","last_undegraded":"2024-08-17T18:20:57.826501+0000","last_fullsized":"2024-08-17T18:20:57.826501+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:20:33.279076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2713,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:57.826400+0000","last_change":"2024-08-17T18:18:29.742742+0000","last_active":"2024-08-17T18:20:57.826400+0000","last_peered":"2024-08-17T18:20:57.826400+0000","last_clean":"2024-08-17T18:20:57.826400+0000","last_became_active":"2024-08-17T18:18:29.742393+0000","last_became_peered":"2024-08-17T18:18:29.742393+0000","last_unstale":"2024-08-17T18:20:57.826400+0000","last_undegraded":"2024-08-17T18:20:57.826400+0000","last_fullsized":"2024-08-17T18:20:57.826400+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:57:32.858608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"347'149","reported_seq":709,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:57.827245+0000","last_change":"2024-08-17T18:20:35.854267+0000","last_active":"2024-08-17T18:20:57.827245+0000","last_peered":"2024-08-17T18:20:57.827245+0000","last_clean":"2024-08-17T18:20:57.827245+0000","last_became_active":"2024-08-17T18:20:34.926649+0000","last_became_peered":"2024-08-17T18:20:34.926649+0000","last_unstale":"2024-08-17T18:20:57.827245+0000","last_undegraded":"2024-08-17T18:20:57.827245+0000","last_fullsized":"2024-08-17T18:20:57.827245+0000","mapping_epoch":1698,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1699,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:20:35.854154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:20:35.854154+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:23:46.928857+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":32,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:57.827422+0000","last_change":"2024-08-17T18:20:31.941131+0000","last_active":"2024-08-17T18:20:57.827422+0000","last_peered":"2024-08-17T18:20:57.827422+0000","last_clean":"2024-08-17T18:20:57.827422+0000","last_became_active":"2024-08-17T18:20:31.940667+0000","last_became_peered":"2024-08-17T18:20:31.940667+0000","last_unstale":"2024-08-17T18:20:57.827422+0000","last_undegraded":"2024-08-17T18:20:57.827422+0000","last_fullsized":"2024-08-17T18:20:57.827422+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:04:07.512353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":883,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.337790+0000","last_change":"2024-08-17T18:18:29.727602+0000","last_active":"2024-08-17T18:20:49.337790+0000","last_peered":"2024-08-17T18:20:49.337790+0000","last_clean":"2024-08-17T18:20:49.337790+0000","last_became_active":"2024-08-17T18:18:29.727190+0000","last_became_peered":"2024-08-17T18:18:29.727190+0000","last_unstale":"2024-08-17T18:20:49.337790+0000","last_undegraded":"2024-08-17T18:20:49.337790+0000","last_fullsized":"2024-08-17T18:20:49.337790+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:13:09.502199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":32,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:57.826406+0000","last_change":"2024-08-17T18:20:31.940564+0000","last_active":"2024-08-17T18:20:57.826406+0000","last_peered":"2024-08-17T18:20:57.826406+0000","last_clean":"2024-08-17T18:20:57.826406+0000","last_became_active":"2024-08-17T18:20:31.939897+0000","last_became_peered":"2024-08-17T18:20:31.939897+0000","last_unstale":"2024-08-17T18:20:57.826406+0000","last_undegraded":"2024-08-17T18:20:57.826406+0000","last_fullsized":"2024-08-17T18:20:57.826406+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:43:25.307705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1554,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.338425+0000","last_change":"2024-08-17T18:18:29.895726+0000","last_active":"2024-08-17T18:20:49.338425+0000","last_peered":"2024-08-17T18:20:49.338425+0000","last_clean":"2024-08-17T18:20:49.338425+0000","last_became_active":"2024-08-17T18:18:29.895383+0000","last_became_peered":"2024-08-17T18:18:29.895383+0000","last_unstale":"2024-08-17T18:20:49.338425+0000","last_undegraded":"2024-08-17T18:20:49.338425+0000","last_fullsized":"2024-08-17T18:20:49.338425+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:18:30.787712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":91,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:56.765663+0000","last_change":"2024-08-17T18:18:29.800143+0000","last_active":"2024-08-17T18:20:56.765663+0000","last_peered":"2024-08-17T18:20:56.765663+0000","last_clean":"2024-08-17T18:20:56.765663+0000","last_became_active":"2024-08-17T18:18:29.799896+0000","last_became_peered":"2024-08-17T18:18:29.799896+0000","last_unstale":"2024-08-17T18:20:56.765663+0000","last_undegraded":"2024-08-17T18:20:56.765663+0000","last_fullsized":"2024-08-17T18:20:56.765663+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:27:18.144391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":292,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.367582+0000","last_change":"2024-08-17T18:19:26.374702+0000","last_active":"2024-08-17T18:20:49.367582+0000","last_peered":"2024-08-17T18:20:49.367582+0000","last_clean":"2024-08-17T18:20:49.367582+0000","last_became_active":"2024-08-17T18:15:53.304106+0000","last_became_peered":"2024-08-17T18:15:53.304106+0000","last_unstale":"2024-08-17T18:20:49.367582+0000","last_undegraded":"2024-08-17T18:20:49.367582+0000","last_fullsized":"2024-08-17T18:20:49.367582+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:21.409368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":916,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.366643+0000","last_change":"2024-08-17T18:20:36.599898+0000","last_active":"2024-08-17T18:20:49.366643+0000","last_peered":"2024-08-17T18:20:49.366643+0000","last_clean":"2024-08-17T18:20:49.366643+0000","last_became_active":"2024-08-17T18:20:34.925442+0000","last_became_peered":"2024-08-17T18:20:34.925442+0000","last_unstale":"2024-08-17T18:20:49.366643+0000","last_undegraded":"2024-08-17T18:20:49.366643+0000","last_fullsized":"2024-08-17T18:20:49.366643+0000","mapping_epoch":1698,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1699,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:20:36.599786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:20:36.599786+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:19:44.919196+0000","scrub_duration":0.0032508300000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"347'149","reported_seq":984,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.366988+0000","last_change":"2024-08-17T18:09:15.592722+0000","last_active":"2024-08-17T18:20:49.366988+0000","last_peered":"2024-08-17T18:20:49.366988+0000","last_clean":"2024-08-17T18:20:49.366988+0000","last_became_active":"2024-08-17T18:09:15.592242+0000","last_became_peered":"2024-08-17T18:09:15.592242+0000","last_unstale":"2024-08-17T18:20:49.366988+0000","last_undegraded":"2024-08-17T18:20:49.366988+0000","last_fullsized":"2024-08-17T18:20:49.366988+0000","mapping_epoch":1532,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1533,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:55.255983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:55.255983+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:02:53.596275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":32,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:57.827125+0000","last_change":"2024-08-17T18:20:31.939871+0000","last_active":"2024-08-17T18:20:57.827125+0000","last_peered":"2024-08-17T18:20:57.827125+0000","last_clean":"2024-08-17T18:20:57.827125+0000","last_became_active":"2024-08-17T18:20:31.939535+0000","last_became_peered":"2024-08-17T18:20:31.939535+0000","last_unstale":"2024-08-17T18:20:57.827125+0000","last_undegraded":"2024-08-17T18:20:57.827125+0000","last_fullsized":"2024-08-17T18:20:57.827125+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:43:31.630668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":98,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:56.765593+0000","last_change":"2024-08-17T18:20:31.932222+0000","last_active":"2024-08-17T18:20:56.765593+0000","last_peered":"2024-08-17T18:20:56.765593+0000","last_clean":"2024-08-17T18:20:56.765593+0000","last_became_active":"2024-08-17T18:20:31.931840+0000","last_became_peered":"2024-08-17T18:20:31.931840+0000","last_unstale":"2024-08-17T18:20:56.765593+0000","last_undegraded":"2024-08-17T18:20:56.765593+0000","last_fullsized":"2024-08-17T18:20:56.765593+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:55:09.048395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1752,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.338324+0000","last_change":"2024-08-17T18:20:32.072538+0000","last_active":"2024-08-17T18:20:49.338324+0000","last_peered":"2024-08-17T18:20:49.338324+0000","last_clean":"2024-08-17T18:20:49.338324+0000","last_became_active":"2024-08-17T18:20:32.040195+0000","last_became_peered":"2024-08-17T18:20:32.040195+0000","last_unstale":"2024-08-17T18:20:49.338324+0000","last_undegraded":"2024-08-17T18:20:49.338324+0000","last_fullsized":"2024-08-17T18:20:49.338324+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:48:39.154753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1733,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.337502+0000","last_change":"2024-08-17T18:15:53.244851+0000","last_active":"2024-08-17T18:20:49.337502+0000","last_peered":"2024-08-17T18:20:49.337502+0000","last_clean":"2024-08-17T18:20:49.337502+0000","last_became_active":"2024-08-17T18:15:53.240074+0000","last_became_peered":"2024-08-17T18:15:53.240074+0000","last_unstale":"2024-08-17T18:20:49.337502+0000","last_undegraded":"2024-08-17T18:20:49.337502+0000","last_fullsized":"2024-08-17T18:20:49.337502+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:09:00.168413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"347'149","reported_seq":925,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:57.826307+0000","last_change":"2024-08-17T18:20:19.879500+0000","last_active":"2024-08-17T18:20:57.826307+0000","last_peered":"2024-08-17T18:20:57.826307+0000","last_clean":"2024-08-17T18:20:57.826307+0000","last_became_active":"2024-08-17T18:20:19.878987+0000","last_became_peered":"2024-08-17T18:20:19.878987+0000","last_unstale":"2024-08-17T18:20:57.826307+0000","last_undegraded":"2024-08-17T18:20:57.826307+0000","last_fullsized":"2024-08-17T18:20:57.826307+0000","mapping_epoch":1691,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1692,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:59.307356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:59.307356+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:42:09.573258+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1737,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.338357+0000","last_change":"2024-08-17T18:20:31.942019+0000","last_active":"2024-08-17T18:20:49.338357+0000","last_peered":"2024-08-17T18:20:49.338357+0000","last_clean":"2024-08-17T18:20:49.338357+0000","last_became_active":"2024-08-17T18:20:31.941701+0000","last_became_peered":"2024-08-17T18:20:31.941701+0000","last_unstale":"2024-08-17T18:20:49.338357+0000","last_undegraded":"2024-08-17T18:20:49.338357+0000","last_fullsized":"2024-08-17T18:20:49.338357+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:58.908049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1724,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.366583+0000","last_change":"2024-08-17T18:20:31.929938+0000","last_active":"2024-08-17T18:20:49.366583+0000","last_peered":"2024-08-17T18:20:49.366583+0000","last_clean":"2024-08-17T18:20:49.366583+0000","last_became_active":"2024-08-17T18:20:31.929616+0000","last_became_peered":"2024-08-17T18:20:31.929616+0000","last_unstale":"2024-08-17T18:20:49.366583+0000","last_undegraded":"2024-08-17T18:20:49.366583+0000","last_fullsized":"2024-08-17T18:20:49.366583+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:12.324951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"347'149","reported_seq":960,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:48.866098+0000","last_change":"2024-08-17T18:18:33.757347+0000","last_active":"2024-08-17T18:20:48.866098+0000","last_peered":"2024-08-17T18:20:48.866098+0000","last_clean":"2024-08-17T18:20:48.866098+0000","last_became_active":"2024-08-17T18:18:33.756960+0000","last_became_peered":"2024-08-17T18:18:33.756960+0000","last_unstale":"2024-08-17T18:20:48.866098+0000","last_undegraded":"2024-08-17T18:20:48.866098+0000","last_fullsized":"2024-08-17T18:20:48.866098+0000","mapping_epoch":1670,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1671,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:52.271487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:52.271487+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:37:36.389244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1716,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.338147+0000","last_change":"2024-08-17T18:18:29.883479+0000","last_active":"2024-08-17T18:20:49.338147+0000","last_peered":"2024-08-17T18:20:49.338147+0000","last_clean":"2024-08-17T18:20:49.338147+0000","last_became_active":"2024-08-17T18:18:29.883185+0000","last_became_peered":"2024-08-17T18:18:29.883185+0000","last_unstale":"2024-08-17T18:20:49.338147+0000","last_undegraded":"2024-08-17T18:20:49.338147+0000","last_fullsized":"2024-08-17T18:20:49.338147+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:11:04.148316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":106,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:56.766146+0000","last_change":"2024-08-17T18:20:31.944000+0000","last_active":"2024-08-17T18:20:56.766146+0000","last_peered":"2024-08-17T18:20:56.766146+0000","last_clean":"2024-08-17T18:20:56.766146+0000","last_became_active":"2024-08-17T18:20:31.934895+0000","last_became_peered":"2024-08-17T18:20:31.934895+0000","last_unstale":"2024-08-17T18:20:56.766146+0000","last_undegraded":"2024-08-17T18:20:56.766146+0000","last_fullsized":"2024-08-17T18:20:56.766146+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:44:48.744274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":32,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:57.827183+0000","last_change":"2024-08-17T18:20:31.938511+0000","last_active":"2024-08-17T18:20:57.827183+0000","last_peered":"2024-08-17T18:20:57.827183+0000","last_clean":"2024-08-17T18:20:57.827183+0000","last_became_active":"2024-08-17T18:20:31.938196+0000","last_became_peered":"2024-08-17T18:20:31.938196+0000","last_unstale":"2024-08-17T18:20:57.827183+0000","last_undegraded":"2024-08-17T18:20:57.827183+0000","last_fullsized":"2024-08-17T18:20:57.827183+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:48:19.325607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":286,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.367467+0000","last_change":"2024-08-17T18:18:29.735189+0000","last_active":"2024-08-17T18:20:49.367467+0000","last_peered":"2024-08-17T18:20:49.367467+0000","last_clean":"2024-08-17T18:20:49.367467+0000","last_became_active":"2024-08-17T18:18:29.734681+0000","last_became_peered":"2024-08-17T18:18:29.734681+0000","last_unstale":"2024-08-17T18:20:49.367467+0000","last_undegraded":"2024-08-17T18:20:49.367467+0000","last_fullsized":"2024-08-17T18:20:49.367467+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:03:20.353696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":32,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:57.826851+0000","last_change":"2024-08-17T18:20:31.940149+0000","last_active":"2024-08-17T18:20:57.826851+0000","last_peered":"2024-08-17T18:20:57.826851+0000","last_clean":"2024-08-17T18:20:57.826851+0000","last_became_active":"2024-08-17T18:20:31.939839+0000","last_became_peered":"2024-08-17T18:20:31.939839+0000","last_unstale":"2024-08-17T18:20:57.826851+0000","last_undegraded":"2024-08-17T18:20:57.826851+0000","last_fullsized":"2024-08-17T18:20:57.826851+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:19:42.313911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":289,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.366714+0000","last_change":"2024-08-17T18:15:53.661488+0000","last_active":"2024-08-17T18:20:49.366714+0000","last_peered":"2024-08-17T18:20:49.366714+0000","last_clean":"2024-08-17T18:20:49.366714+0000","last_became_active":"2024-08-17T18:15:53.308611+0000","last_became_peered":"2024-08-17T18:15:53.308611+0000","last_unstale":"2024-08-17T18:20:49.366714+0000","last_undegraded":"2024-08-17T18:20:49.366714+0000","last_fullsized":"2024-08-17T18:20:49.366714+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:46:37.056632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1759,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.337769+0000","last_change":"2024-08-17T18:20:31.942625+0000","last_active":"2024-08-17T18:20:49.337769+0000","last_peered":"2024-08-17T18:20:49.337769+0000","last_clean":"2024-08-17T18:20:49.337769+0000","last_became_active":"2024-08-17T18:20:31.942099+0000","last_became_peered":"2024-08-17T18:20:31.942099+0000","last_unstale":"2024-08-17T18:20:49.337769+0000","last_undegraded":"2024-08-17T18:20:49.337769+0000","last_fullsized":"2024-08-17T18:20:49.337769+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:22.877846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":97,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:48.866567+0000","last_change":"2024-08-17T18:20:32.006404+0000","last_active":"2024-08-17T18:20:48.866567+0000","last_peered":"2024-08-17T18:20:48.866567+0000","last_clean":"2024-08-17T18:20:48.866567+0000","last_became_active":"2024-08-17T18:20:31.947053+0000","last_became_peered":"2024-08-17T18:20:31.947053+0000","last_unstale":"2024-08-17T18:20:48.866567+0000","last_undegraded":"2024-08-17T18:20:48.866567+0000","last_fullsized":"2024-08-17T18:20:48.866567+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:21:06.537725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":295,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.366876+0000","last_change":"2024-08-17T18:17:22.687310+0000","last_active":"2024-08-17T18:20:49.366876+0000","last_peered":"2024-08-17T18:20:49.366876+0000","last_clean":"2024-08-17T18:20:49.366876+0000","last_became_active":"2024-08-17T18:17:22.686966+0000","last_became_peered":"2024-08-17T18:17:22.686966+0000","last_unstale":"2024-08-17T18:20:49.366876+0000","last_undegraded":"2024-08-17T18:20:49.366876+0000","last_fullsized":"2024-08-17T18:20:49.366876+0000","mapping_epoch":1652,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1653,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:46:11.568465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":97,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:48.866653+0000","last_change":"2024-08-17T18:20:31.958228+0000","last_active":"2024-08-17T18:20:48.866653+0000","last_peered":"2024-08-17T18:20:48.866653+0000","last_clean":"2024-08-17T18:20:48.866653+0000","last_became_active":"2024-08-17T18:20:31.946334+0000","last_became_peered":"2024-08-17T18:20:31.946334+0000","last_unstale":"2024-08-17T18:20:48.866653+0000","last_undegraded":"2024-08-17T18:20:48.866653+0000","last_fullsized":"2024-08-17T18:20:48.866653+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:48:26.723103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1827,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.366739+0000","last_change":"2024-08-17T18:18:29.897922+0000","last_active":"2024-08-17T18:20:49.366739+0000","last_peered":"2024-08-17T18:20:49.366739+0000","last_clean":"2024-08-17T18:20:49.366739+0000","last_became_active":"2024-08-17T18:18:29.883879+0000","last_became_peered":"2024-08-17T18:18:29.883879+0000","last_unstale":"2024-08-17T18:20:49.366739+0000","last_undegraded":"2024-08-17T18:20:49.366739+0000","last_fullsized":"2024-08-17T18:20:49.366739+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:54:31.992298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":98,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:56.766537+0000","last_change":"2024-08-17T18:20:31.935279+0000","last_active":"2024-08-17T18:20:56.766537+0000","last_peered":"2024-08-17T18:20:56.766537+0000","last_clean":"2024-08-17T18:20:56.766537+0000","last_became_active":"2024-08-17T18:20:31.934475+0000","last_became_peered":"2024-08-17T18:20:31.934475+0000","last_unstale":"2024-08-17T18:20:56.766537+0000","last_undegraded":"2024-08-17T18:20:56.766537+0000","last_fullsized":"2024-08-17T18:20:56.766537+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:24:44.485841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":446,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.337617+0000","last_change":"2024-08-17T18:15:53.245247+0000","last_active":"2024-08-17T18:20:49.337617+0000","last_peered":"2024-08-17T18:20:49.337617+0000","last_clean":"2024-08-17T18:20:49.337617+0000","last_became_active":"2024-08-17T18:15:53.244627+0000","last_became_peered":"2024-08-17T18:15:53.244627+0000","last_unstale":"2024-08-17T18:20:49.337617+0000","last_undegraded":"2024-08-17T18:20:49.337617+0000","last_fullsized":"2024-08-17T18:20:49.337617+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:42:03.556013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":91,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:56.766612+0000","last_change":"2024-08-17T18:18:29.895804+0000","last_active":"2024-08-17T18:20:56.766612+0000","last_peered":"2024-08-17T18:20:56.766612+0000","last_clean":"2024-08-17T18:20:56.766612+0000","last_became_active":"2024-08-17T18:18:29.895352+0000","last_became_peered":"2024-08-17T18:18:29.895352+0000","last_unstale":"2024-08-17T18:20:56.766612+0000","last_undegraded":"2024-08-17T18:20:56.766612+0000","last_fullsized":"2024-08-17T18:20:56.766612+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:32:29.222569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":98,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:56.765843+0000","last_change":"2024-08-17T18:20:31.936952+0000","last_active":"2024-08-17T18:20:56.765843+0000","last_peered":"2024-08-17T18:20:56.765843+0000","last_clean":"2024-08-17T18:20:56.765843+0000","last_became_active":"2024-08-17T18:20:31.936602+0000","last_became_peered":"2024-08-17T18:20:31.936602+0000","last_unstale":"2024-08-17T18:20:56.765843+0000","last_undegraded":"2024-08-17T18:20:56.765843+0000","last_fullsized":"2024-08-17T18:20:56.765843+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:09:46.856871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":294,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:49.367305+0000","last_change":"2024-08-17T18:18:29.900128+0000","last_active":"2024-08-17T18:20:49.367305+0000","last_peered":"2024-08-17T18:20:49.367305+0000","last_clean":"2024-08-17T18:20:49.367305+0000","last_became_active":"2024-08-17T18:18:29.899279+0000","last_became_peered":"2024-08-17T18:18:29.899279+0000","last_unstale":"2024-08-17T18:20:49.367305+0000","last_undegraded":"2024-08-17T18:20:49.367305+0000","last_fullsized":"2024-08-17T18:20:49.367305+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:01:29.045498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":87,"reported_epoch":1705,"state":"active+clean","last_fresh":"2024-08-17T18:20:48.866300+0000","last_change":"2024-08-17T18:15:53.271113+0000","last_active":"2024-08-17T18:20:48.866300+0000","last_peered":"2024-08-17T18:20:48.866300+0000","last_clean":"2024-08-17T18:20:48.866300+0000","last_became_active":"2024-08-17T18:15:53.230983+0000","last_became_peered":"2024-08-17T18:15:53.230983+0000","last_unstale":"2024-08-17T18:20:48.866300+0000","last_undegraded":"2024-08-17T18:20:48.866300+0000","last_fullsized":"2024-08-17T18:20:48.866300+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:00:35.315798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":98,"reported_epoch":1706,"state":"active+clean","last_fresh":"2024-08-17T18:20:56.766153+0000","last_change":"2024-08-17T18:20:31.935545+0000","last_active":"2024-08-17T18:20:56.766153+0000","last_peered":"2024-08-17T18:20:56.766153+0000","last_clean":"2024-08-17T18:20:56.766153+0000","last_became_active":"2024-08-17T18:20:31.934479+0000","last_became_peered":"2024-08-17T18:20:31.934479+0000","last_unstale":"2024-08-17T18:20:56.766153+0000","last_undegraded":"2024-08-17T18:20:56.766153+0000","last_fullsized":"2024-08-17T18:20:56.766153+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:28:08.491066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":58,"num_pg":16,"stat_sum":{"num_bytes":1194786816,"num_objects":18231,"num_object_clones":0,"num_object_copies":54693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18231,"num_write_kb":1166784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3634167808,"data_stored":3634167808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18231,"ondisk_log_size":18231,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":918048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":12554,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234861,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":989380,"kb_used_data":300412,"kb_used_omap":1,"kb_used_meta":688958,"kb_avail":92755772,"statfs":{"total":95995035648,"available":94981910528,"internally_reserved":0,"allocated":307621888,"data_stored":303025667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":705493451},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496499,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1744704,"kb_used_data":454072,"kb_used_omap":1,"kb_used_meta":1290622,"kb_avail":92000448,"statfs":{"total":95995035648,"available":94208458752,"internally_reserved":0,"allocated":464969728,"data_stored":460378058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1321597383},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565740,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1811056,"kb_used_data":754344,"kb_used_omap":72,"kb_used_meta":1056631,"kb_avail":91934096,"statfs":{"total":95995035648,"available":94140514304,"internally_reserved":0,"allocated":772448256,"data_stored":767868189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":1081990638},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":1399,"seq":6008659247342,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1415792,"kb_used_data":643688,"kb_used_omap":131,"kb_used_meta":771964,"kb_avail":92329360,"statfs":{"total":95995035648,"available":94545264640,"internally_reserved":0,"allocated":659136512,"data_stored":654556389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134291,"internal_metadata":790492013},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743187130,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1369520,"kb_used_data":607208,"kb_used_omap":66,"kb_used_meta":762173,"kb_avail":92375632,"statfs":{"total":95995035648,"available":94592647168,"internally_reserved":0,"allocated":621780992,"data_stored":617200332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":780465835},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448769,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2107112,"kb_used_data":1295008,"kb_used_omap":75,"kb_used_meta":812020,"kb_avail":91638040,"statfs":{"total":95995035648,"available":93837352960,"internally_reserved":0,"allocated":1326088192,"data_stored":1321515683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":831508671},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208758134,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1845300,"kb_used_data":496876,"kb_used_omap":1,"kb_used_meta":1348414,"kb_avail":91899852,"statfs":{"total":95995035648,"available":94105448448,"internally_reserved":0,"allocated":508801024,"data_stored":504216886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1380776390},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019770,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1481608,"kb_used_data":300800,"kb_used_omap":1,"kb_used_meta":1180798,"kb_avail":92263544,"statfs":{"total":95995035648,"available":94477869056,"internally_reserved":0,"allocated":308019200,"data_stored":303419939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1209137606},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":144572416,"data_stored":144572416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":632684544,"data_stored":632684544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":661389312,"data_stored":661389312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":802947072,"data_stored":802947072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":345440256,"data_stored":345440256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":244908032,"data_stored":244908032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":487063552,"data_stored":487063552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":315162624,"data_stored":315162624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T18:21:21.303 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-08-17T18:21:21.388 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:21.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:21.434 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:21.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:21.535 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:21.561 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:21.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:21.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:21.663 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:21.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:21.709 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:21.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:21.810 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:21.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:21.842 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:21.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:21.943 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:21.967 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:21.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:21.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:21 smithi165 ceph-mon[105517]: pgmap v3325: 99 pgs: 99 active+clean; 2.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-08-17T18:21:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:21:22.069 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:22.095 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:21 smithi136 ceph-mon[115078]: pgmap v3325: 99 pgs: 99 active+clean; 2.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-08-17T18:21:22.096 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:21:22.096 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:21 smithi136 ceph-mon[109861]: pgmap v3325: 99 pgs: 99 active+clean; 2.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-08-17T18:21:22.096 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:21:22.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:22.119 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:22.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:22.211 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 22 16 45231 45215 128.44 93.5625 0.00312039 0.00778087 2024-08-17T18:21:22.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:22.248 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:22.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:22.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:22.349 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:22.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:22.390 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:22.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:22.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:22.532 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:22.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:22.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:22.634 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:22.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:22.674 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:22.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:22.775 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:22.833 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:22.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:22.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:22.935 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:22.975 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:22.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:22.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:23.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:22 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:21:23.041 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:22 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:21:23.042 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:22 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:21:23.077 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:23.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:23.112 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:23.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:23.211 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 23 16 46597 46581 126.567 85.375 0.00737138 0.00789166 2024-08-17T18:21:23.213 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:23.245 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:23.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:23.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:23.347 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:23.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:23.387 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:23.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:23.488 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:23.518 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:23.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:23.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:23.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:23.656 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:23.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:23.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:23.758 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:23.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:23.801 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:23.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:23.901 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:23.934 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:23.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:23.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:24.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:23 smithi165 ceph-mon[105517]: pgmap v3326: 99 pgs: 99 active+clean; 2.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-08-17T18:21:24.036 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:24.072 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:24.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:24.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:24.174 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:24.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:23 smithi136 ceph-mon[109861]: pgmap v3326: 99 pgs: 99 active+clean; 2.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-08-17T18:21:24.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:23 smithi136 ceph-mon[115078]: pgmap v3326: 99 pgs: 99 active+clean; 2.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-08-17T18:21:24.210 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:24.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:24.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:24.211 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 24 16 48205 48189 125.481 100.5 0.0193612 0.00796373 2024-08-17T18:21:24.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:24.333 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:24.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:24.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:24.435 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:24.474 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:24.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:24.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:24.576 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:24.615 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:24.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:24.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:24.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:24.754 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:24.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:24.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:24.856 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:24.891 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:24.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:24.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:24.993 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:25.030 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:25.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:25.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:25.132 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:25.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:25.165 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:25.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:25.211 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 25 16 49637 49621 124.041 89.5 0.0101498 0.00805632 2024-08-17T18:21:25.267 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:25.297 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:25.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:25.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:25.398 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:21:25.398 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:21:25.400 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:25.428 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:25.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:25.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:25.530 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:25.558 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:25.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:25.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:25.661 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:25.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:25.692 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:25.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:25.793 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:25.825 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:25.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:25.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:25.927 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:25.962 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:25.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:25.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:25 smithi165 ceph-mon[105517]: pgmap v3327: 99 pgs: 99 active+clean; 2.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-08-17T18:21:26.066 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:26.110 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:26.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:26.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:26.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:25 smithi136 ceph-mon[109861]: pgmap v3327: 99 pgs: 99 active+clean; 2.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-08-17T18:21:26.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:25 smithi136 ceph-mon[115078]: pgmap v3327: 99 pgs: 99 active+clean; 2.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-08-17T18:21:26.211 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 26 16 51153 51137 122.914 94.75 0.00927012 0.00813121 2024-08-17T18:21:26.212 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:26.246 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:26.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:26.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:26.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:26.377 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:26.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:26.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:26.479 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:26.506 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:26.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:26.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:26.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:26.644 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:26.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:26.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:26.681 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:21:26.689 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:21:26.746 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:26.772 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:26.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:26.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:26.874 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:26.905 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:26.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:26.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:27.006 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:27.043 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:27.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:27.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:27.145 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:27.191 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:27.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:27.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:27.211 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 27 16 52662 52646 121.854 94.3125 0.0132399 0.008202 2024-08-17T18:21:27.286 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-17T18:21:27.293 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:27.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:27.376 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:27.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:27.477 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:27.503 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:27.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:27.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:27.532 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:21:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:21:27] "GET /metrics HTTP/1.1" 200 36724 "" "Prometheus/2.51.0" 2024-08-17T18:21:27.604 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:27.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:27.641 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:27.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:27.742 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:27.780 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:27.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:27.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:27.812 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:27 smithi136 ceph-mon[109861]: pgmap v3328: 99 pgs: 99 active+clean; 2.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-08-17T18:21:27.813 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:27 smithi136 ceph-mon[115078]: pgmap v3328: 99 pgs: 99 active+clean; 2.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-08-17T18:21:27.882 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:27.926 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:27.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:27.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:28.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:27 smithi165 ceph-mon[105517]: pgmap v3328: 99 pgs: 99 active+clean; 2.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-08-17T18:21:28.028 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:28.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:28.073 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:28.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:28.174 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:28.197 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:28.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:28.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:28.211 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 28 16 53980 53964 120.444 82.375 0.00360341 0.00829718 2024-08-17T18:21:28.299 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:28.339 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:28.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:28.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:28.441 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:28.488 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:28.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:28.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:28.594 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:28.622 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:28.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:28.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:28.724 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:28.760 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:28.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:28.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:28.862 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:28.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:28.901 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:28.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:29.003 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:29.035 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:29.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:29.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:29.137 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:29.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:29.183 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:29.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:29.212 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 29 16 55354 55338 119.249 85.875 0.00931569 0.00838039 2024-08-17T18:21:29.284 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:29.319 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:29.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:29.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:29.421 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:29.469 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:29.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:29.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:29.571 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:29.596 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:29.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:29.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:29.698 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:29.725 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:29.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:29.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:29.826 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:29.854 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:29.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:29.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:29.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:29 smithi136 ceph-mon[109861]: pgmap v3329: 99 pgs: 99 active+clean; 3.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-08-17T18:21:29.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:29 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:21:29.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:29 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:21:29.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:29 smithi136 ceph-mon[115078]: pgmap v3329: 99 pgs: 99 active+clean; 3.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-08-17T18:21:29.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:29 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:21:29.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:29 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:21:29.955 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:29.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:29.994 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:29.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:30.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:29 smithi165 ceph-mon[105517]: pgmap v3329: 99 pgs: 99 active+clean; 3.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-08-17T18:21:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:29 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:21:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:29 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:21:30.095 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:30.118 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:30.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:30.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:30.212 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 30 16 56758 56742 118.199 87.75 0.0136335 0.00845472 2024-08-17T18:21:30.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:30.244 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:30.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:30.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:30.283 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:21:30.283 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:21:30.324 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:21:30.325 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:21:30.346 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:30.367 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:21:30.367 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:21:30.372 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:30.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:30.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:30.406 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:21:30.406 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:21:30.436 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:21:30.436 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:21:30.472 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:21:30.473 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:21:30.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:30.502 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:30.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:30.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:30.504 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:21:30.504 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:21:30.550 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:21:30.550 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:21:30.604 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:30.645 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:30.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:30.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:30.747 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:30.779 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:30.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:30.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:30.808 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:30 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1954876599' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:21:30.808 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:30 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1954876599' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:21:30.881 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:30.925 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:30.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:30.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:30 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1954876599' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:21:31.027 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:31.067 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:31.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:31.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:31.169 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:31.196 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:31.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:31.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:31.212 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 31 16 58137 58121 117.166 86.1875 0.0131242 0.00852916 2024-08-17T18:21:31.298 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:31.338 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:31.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:31.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:31.440 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:31.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:31.468 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:31.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:31.569 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:31.603 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:31.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:31.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:31.705 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:31.726 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:31.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:31.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:31.828 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:31.868 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:31.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:31.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:31.901 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:31 smithi136 ceph-mon[109861]: pgmap v3330: 99 pgs: 99 active+clean; 3.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-08-17T18:21:31.902 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:31 smithi136 ceph-mon[115078]: pgmap v3330: 99 pgs: 99 active+clean; 3.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-08-17T18:21:31.970 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:31.996 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:31.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:31.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:32.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:31 smithi165 ceph-mon[105517]: pgmap v3330: 99 pgs: 99 active+clean; 3.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-08-17T18:21:32.099 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:32.136 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:32.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:32.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:32.212 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 32 16 59275 59259 115.727 71.125 0.00762576 0.00863095 2024-08-17T18:21:32.238 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:32.275 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:32.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:32.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:32.377 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:32.414 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:32.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:32.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:32.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:32.552 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:32.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:32.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:32.653 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:32.694 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:32.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:32.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:32.796 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:32.830 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:32.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:32.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:32.932 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:32.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:32 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/129264232' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T18:21:32.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:32 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/129264232' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T18:21:32.957 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:32.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:32.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:33.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:32 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/129264232' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T18:21:33.062 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:33.099 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:33.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:33.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:33.201 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:33.212 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 33 16 60492 60476 114.525 76.0625 0.0170168 0.00872668 2024-08-17T18:21:33.227 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:33.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:33.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:33.329 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:33.356 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:33.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:33.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:33.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:33.486 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:33.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:33.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:33.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:33.615 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:33.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:33.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:33.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:33.742 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:33.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:33.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:33.844 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:33.886 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:33.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:33.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:33.988 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:34.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:33 smithi165 ceph-mon[105517]: pgmap v3331: 99 pgs: 99 active+clean; 3.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 110 MiB/s wr, 1.75k op/s 2024-08-17T18:21:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:33 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/129264232' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-08-17T18:21:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:33 smithi165 ceph-mon[105517]: osdmap e1710: 8 total, 8 up, 8 in 2024-08-17T18:21:34.014 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:34.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:34.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:34.022 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:33 smithi136 ceph-mon[115078]: pgmap v3331: 99 pgs: 99 active+clean; 3.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 110 MiB/s wr, 1.75k op/s 2024-08-17T18:21:34.022 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:33 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/129264232' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-08-17T18:21:34.022 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:33 smithi136 ceph-mon[115078]: osdmap e1710: 8 total, 8 up, 8 in 2024-08-17T18:21:34.024 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:33 smithi136 ceph-mon[109861]: pgmap v3331: 99 pgs: 99 active+clean; 3.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 110 MiB/s wr, 1.75k op/s 2024-08-17T18:21:34.024 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:33 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/129264232' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-08-17T18:21:34.024 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:33 smithi136 ceph-mon[109861]: osdmap e1710: 8 total, 8 up, 8 in 2024-08-17T18:21:34.116 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:34.166 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:34.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:34.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:34.213 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 34 16 61820 61804 113.595 83 0.010389 0.008798 2024-08-17T18:21:34.268 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:34.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:34.318 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:34.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:34.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:34.438 INFO:teuthology.orchestra.run.smithi136.stderr:set pool 1 pgp_num to 11 2024-08-17T18:21:34.446 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:34.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:34.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:34.548 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:34.585 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:34.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:34.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:34.687 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:34.731 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:34.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:34.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:34.833 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:34.862 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:34.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:34.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:34.964 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:34.991 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:34.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:34.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:35.093 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:35.123 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:35.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:35.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:35.213 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 35 16 63201 63185 112.815 86.3125 0.0138224 0.0088581 2024-08-17T18:21:35.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:35.251 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:35.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:35.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:35.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:35.381 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:35.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:35.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:35.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:35.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:35.529 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:35.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:35.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:35.656 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:35.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:35.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:35.757 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:35.791 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:35.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:35.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:35.893 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:35.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:35.932 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:35.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:36.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:35 smithi165 ceph-mon[105517]: pgmap v3333: 99 pgs: 99 active+clean; 3.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-08-17T18:21:36.033 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:36.072 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:36.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:36.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:36.173 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:36.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:35 smithi136 ceph-mon[109861]: pgmap v3333: 99 pgs: 99 active+clean; 3.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-08-17T18:21:36.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:35 smithi136 ceph-mon[115078]: pgmap v3333: 99 pgs: 99 active+clean; 3.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-08-17T18:21:36.201 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:36.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:36.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:36.213 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 36 16 64544 64528 112.013 83.9375 0.0109471 0.00892267 2024-08-17T18:21:36.303 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:36.330 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:36.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:36.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:36.432 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:36.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:36.469 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:36.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:36.570 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:36.625 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:21:36.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:36.636 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:36.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:36.737 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:36.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:36.797 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:36.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:36.898 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:36.929 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:36.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:36.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:36.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:21:36.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:21:37.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:21:37.031 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:37.056 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:37.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:37.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:37.158 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:37.202 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:37.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:37.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:37.213 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 37 16 65694 65678 110.928 71.875 0.0233869 0.009009 2024-08-17T18:21:37.304 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:37.338 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:37.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:37.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:37.440 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:37.480 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:37.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:37.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:37.582 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:37.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:37.609 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:37.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:37.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:21:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:21:37] "GET /metrics HTTP/1.1" 200 36727 "" "Prometheus/2.51.0" 2024-08-17T18:21:37.711 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:37.742 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:21:37.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:37.758 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:37.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:37.859 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:37.900 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:37.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:37.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:38.001 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:38.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:37 smithi165 ceph-mon[105517]: pgmap v3334: 99 pgs: 99 active+clean; 3.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-08-17T18:21:38.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:37 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1345372469' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:21:38.030 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:38.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:38.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:38.133 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:38.160 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:38.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:38.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:38.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:37 smithi136 ceph-mon[109861]: pgmap v3334: 99 pgs: 99 active+clean; 3.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-08-17T18:21:38.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:37 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1345372469' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:21:38.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:37 smithi136 ceph-mon[115078]: pgmap v3334: 99 pgs: 99 active+clean; 3.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-08-17T18:21:38.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:37 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1345372469' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:21:38.213 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 38 16 66951 66935 110.076 78.5625 0.00448171 0.00907877 2024-08-17T18:21:38.261 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:38.301 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:38.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:38.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:38.403 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:38.447 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:38.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:38.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:38.548 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:38.578 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:38.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:38.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:38.616 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:21:38.684 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:21:38.709 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:38.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:38.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:21:38.811 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:38.832 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:38.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:38.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:38.935 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:38.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:38 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:21:38.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:38 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T18:21:38.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:38 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1345372469' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:21:38.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:38 smithi136 ceph-mon[109861]: osdmap e1711: 8 total, 8 up, 8 in 2024-08-17T18:21:38.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:38 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:21:38.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:38 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T18:21:38.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:38 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1345372469' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:21:38.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:38 smithi136 ceph-mon[115078]: osdmap e1711: 8 total, 8 up, 8 in 2024-08-17T18:21:38.971 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:38.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:38.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:39.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:38 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:21:39.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:38 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T18:21:39.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:38 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1345372469' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:21:39.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:38 smithi165 ceph-mon[105517]: osdmap e1711: 8 total, 8 up, 8 in 2024-08-17T18:21:39.073 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:39.107 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:39.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:39.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:39.209 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:39.213 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 39 16 68112 68096 109.114 72.5625 0.0111322 0.00915854 2024-08-17T18:21:39.257 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:39.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:39.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:39.358 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:39.384 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:39.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:39.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:39.486 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:39.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:39.515 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:39.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:39.616 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:39.653 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:39.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:39.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:39.755 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:39.786 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:39.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:39.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:39.888 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:39.913 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:39.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:39.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:40.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:39 smithi165 ceph-mon[105517]: pgmap v3336: 99 pgs: 99 active+clean; 4.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-08-17T18:21:40.015 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:40.038 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:39 smithi136 ceph-mon[109861]: pgmap v3336: 99 pgs: 99 active+clean; 4.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-08-17T18:21:40.038 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:39 smithi136 ceph-mon[115078]: pgmap v3336: 99 pgs: 99 active+clean; 4.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-08-17T18:21:40.059 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:40.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:40.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:40.161 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:40.186 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:40.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:40.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:40.211 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:21:40.211 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:21:40.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.211 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:69302 58.a 58:57607be5:::benchmark_data_smithi136_442124_object69301:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1711)", 2024-08-17T18:21:40.211 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:21:40.197928+0000", 2024-08-17T18:21:40.211 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0095893369999999999, 2024-08-17T18:21:40.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0096236719999999998, 2024-08-17T18:21:40.211 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:21:40.211 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:21:40.211 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-17T18:21:40.211 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:21:40.212 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:21:40.212 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:21:40.212 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69302 2024-08-17T18:21:40.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.212 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:21:40.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:21:40.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.197928+0000", 2024-08-17T18:21:40.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:21:40.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:21:40.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.197928+0000", 2024-08-17T18:21:40.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:21:40.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:21:40.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.197930+0000", 2024-08-17T18:21:40.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7269999999999999e-06 2024-08-17T18:21:40.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:21:40.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.197961+0000", 2024-08-17T18:21:40.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1241000000000001e-05 2024-08-17T18:21:40.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:21:40.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.197974+0000", 2024-08-17T18:21:40.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3569e-05 2024-08-17T18:21:40.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:21:40.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.197982+0000", 2024-08-17T18:21:40.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3719999999999999e-06 2024-08-17T18:21:40.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:21:40.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.198045+0000", 2024-08-17T18:21:40.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3397000000000003e-05 2024-08-17T18:21:40.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:21:40.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.198210+0000", 2024-08-17T18:21:40.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000164807 2024-08-17T18:21:40.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6,7", 2024-08-17T18:21:40.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.198363+0000", 2024-08-17T18:21:40.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000152674 2024-08-17T18:21:40.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:21:40.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.206731+0000", 2024-08-17T18:21:40.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0083680219999999993 2024-08-17T18:21:40.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:21:40.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.206771+0000", 2024-08-17T18:21:40.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0391e-05 2024-08-17T18:21:40.217 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:21:40.217 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:21:40.217 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:21:40.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.217 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:69304 58.6 58:6d56aa5e:::benchmark_data_smithi136_442124_object69303:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1711)", 2024-08-17T18:21:40.217 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:21:40.198285+0000", 2024-08-17T18:21:40.217 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0092323470000000001, 2024-08-17T18:21:40.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0093459619999999993, 2024-08-17T18:21:40.217 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:21:40.217 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:21:40.217 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-17T18:21:40.217 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:21:40.218 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:21:40.218 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:21:40.218 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69304 2024-08-17T18:21:40.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.218 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:21:40.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:21:40.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.198285+0000", 2024-08-17T18:21:40.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:21:40.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:21:40.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.198285+0000", 2024-08-17T18:21:40.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:21:40.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:21:40.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.198286+0000", 2024-08-17T18:21:40.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7899999999999997e-07 2024-08-17T18:21:40.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:21:40.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.198308+0000", 2024-08-17T18:21:40.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2262000000000001e-05 2024-08-17T18:21:40.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:21:40.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.198319+0000", 2024-08-17T18:21:40.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0747999999999999e-05 2024-08-17T18:21:40.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:21:40.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.198325+0000", 2024-08-17T18:21:40.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1809999999999998e-06 2024-08-17T18:21:40.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:21:40.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.198502+0000", 2024-08-17T18:21:40.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000177345 2024-08-17T18:21:40.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:21:40.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.198642+0000", 2024-08-17T18:21:40.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000139331 2024-08-17T18:21:40.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,4", 2024-08-17T18:21:40.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.198940+0000", 2024-08-17T18:21:40.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029846400000000002 2024-08-17T18:21:40.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:21:40.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.201730+0000", 2024-08-17T18:21:40.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0027899610000000001 2024-08-17T18:21:40.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:21:40.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205041+0000", 2024-08-17T18:21:40.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033111059999999999 2024-08-17T18:21:40.222 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:21:40.222 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:21:40.222 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:21:40.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.222 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:69305 58.3 58:c1e8a1aa:::benchmark_data_smithi136_442124_object69304:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1711)", 2024-08-17T18:21:40.222 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:21:40.198345+0000", 2024-08-17T18:21:40.222 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.009172286, 2024-08-17T18:21:40.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0093625089999999998, 2024-08-17T18:21:40.223 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:21:40.223 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:21:40.223 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-17T18:21:40.223 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:21:40.223 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:21:40.223 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:21:40.223 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69305 2024-08-17T18:21:40.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.223 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:21:40.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:21:40.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.198345+0000", 2024-08-17T18:21:40.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:21:40.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:21:40.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.198345+0000", 2024-08-17T18:21:40.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:21:40.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:21:40.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.198345+0000", 2024-08-17T18:21:40.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8800000000000003e-07 2024-08-17T18:21:40.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:21:40.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.198365+0000", 2024-08-17T18:21:40.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9490000000000001e-05 2024-08-17T18:21:40.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:21:40.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.198379+0000", 2024-08-17T18:21:40.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4263999999999999e-05 2024-08-17T18:21:40.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:21:40.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.198385+0000", 2024-08-17T18:21:40.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6740000000000002e-06 2024-08-17T18:21:40.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:21:40.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.198709+0000", 2024-08-17T18:21:40.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00032383899999999997 2024-08-17T18:21:40.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:21:40.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.198884+0000", 2024-08-17T18:21:40.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017491300000000001 2024-08-17T18:21:40.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,5", 2024-08-17T18:21:40.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.199016+0000", 2024-08-17T18:21:40.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013271800000000001 2024-08-17T18:21:40.229 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:21:40.229 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:21:40.229 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:21:40.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.229 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.36936.0:69305 58.3 e1711/1701)", 2024-08-17T18:21:40.230 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:21:40.201949+0000", 2024-08-17T18:21:40.230 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0055687669999999996, 2024-08-17T18:21:40.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0058169830000000004, 2024-08-17T18:21:40.230 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:21:40.230 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:21:40.230 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-08-17T18:21:40.230 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:21:40.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:21:40.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.201949+0000", 2024-08-17T18:21:40.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:21:40.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:21:40.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.201949+0000", 2024-08-17T18:21:40.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:21:40.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:21:40.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.201949+0000", 2024-08-17T18:21:40.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.65e-07 2024-08-17T18:21:40.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:21:40.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.201951+0000", 2024-08-17T18:21:40.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3120000000000001e-06 2024-08-17T18:21:40.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.232 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:21:40.232 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.201958+0000", 2024-08-17T18:21:40.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6209999999999998e-06 2024-08-17T18:21:40.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.232 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:21:40.232 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.201962+0000", 2024-08-17T18:21:40.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4159999999999997e-06 2024-08-17T18:21:40.232 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:21:40.232 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:21:40.232 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:21:40.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.233 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.36936.0:69310 58.0 e1711/1701 58:08bb3d60:::benchmark_data_smithi136_442124_object69309:head v 1711'4349, mlcod=1711'4347)", 2024-08-17T18:21:40.233 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:21:40.205720+0000", 2024-08-17T18:21:40.233 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0017976330000000001, 2024-08-17T18:21:40.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002086559, 2024-08-17T18:21:40.233 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:21:40.233 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:21:40.233 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T18:21:40.233 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:21:40.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:21:40.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205720+0000", 2024-08-17T18:21:40.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:21:40.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:21:40.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205720+0000", 2024-08-17T18:21:40.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:21:40.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:21:40.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205720+0000", 2024-08-17T18:21:40.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7999999999999996e-07 2024-08-17T18:21:40.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:21:40.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205757+0000", 2024-08-17T18:21:40.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7088000000000001e-05 2024-08-17T18:21:40.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:21:40.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205766+0000", 2024-08-17T18:21:40.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3799999999999994e-06 2024-08-17T18:21:40.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:21:40.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205771+0000", 2024-08-17T18:21:40.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1749999999999996e-06 2024-08-17T18:21:40.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:21:40.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.206042+0000", 2024-08-17T18:21:40.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027167599999999999 2024-08-17T18:21:40.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:21:40.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.206061+0000", 2024-08-17T18:21:40.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8331000000000001e-05 2024-08-17T18:21:40.237 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:21:40.237 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:21:40.237 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:21:40.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.237 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.36936.0:69307 58.7 e1711/1701 58:e3ec3bef:::benchmark_data_smithi136_442124_object69306:head v 1711'4399, mlcod=1711'4395)", 2024-08-17T18:21:40.237 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:21:40.205775+0000", 2024-08-17T18:21:40.237 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0017424199999999999, 2024-08-17T18:21:40.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002079029, 2024-08-17T18:21:40.238 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:21:40.238 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:21:40.238 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T18:21:40.238 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:21:40.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:21:40.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205775+0000", 2024-08-17T18:21:40.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:21:40.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:21:40.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205775+0000", 2024-08-17T18:21:40.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:21:40.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:21:40.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205776+0000", 2024-08-17T18:21:40.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6199999999999997e-07 2024-08-17T18:21:40.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:21:40.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205799+0000", 2024-08-17T18:21:40.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3674000000000001e-05 2024-08-17T18:21:40.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:21:40.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205811+0000", 2024-08-17T18:21:40.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1249e-05 2024-08-17T18:21:40.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:21:40.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205817+0000", 2024-08-17T18:21:40.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2890000000000003e-06 2024-08-17T18:21:40.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:21:40.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205906+0000", 2024-08-17T18:21:40.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9213000000000001e-05 2024-08-17T18:21:40.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.241 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:21:40.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205926+0000", 2024-08-17T18:21:40.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9551999999999998e-05 2024-08-17T18:21:40.241 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:21:40.241 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:21:40.241 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:21:40.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.241 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.36936.0:69311 58.0 e1711/1701)", 2024-08-17T18:21:40.241 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:21:40.205786+0000", 2024-08-17T18:21:40.241 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.001731497, 2024-08-17T18:21:40.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020998290000000001, 2024-08-17T18:21:40.242 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:21:40.242 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:21:40.242 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-08-17T18:21:40.242 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:21:40.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:21:40.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205786+0000", 2024-08-17T18:21:40.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:21:40.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:21:40.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205786+0000", 2024-08-17T18:21:40.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:21:40.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:21:40.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205786+0000", 2024-08-17T18:21:40.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8799999999999998e-07 2024-08-17T18:21:40.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:21:40.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205823+0000", 2024-08-17T18:21:40.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7110999999999998e-05 2024-08-17T18:21:40.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:21:40.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205831+0000", 2024-08-17T18:21:40.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6580000000000002e-06 2024-08-17T18:21:40.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:21:40.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205836+0000", 2024-08-17T18:21:40.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4099999999999999e-06 2024-08-17T18:21:40.244 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:21:40.244 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:21:40.244 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:21:40.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.245 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.36936.0:69314 58.0 e1711/1701)", 2024-08-17T18:21:40.245 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:21:40.205845+0000", 2024-08-17T18:21:40.245 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0016726169999999999, 2024-08-17T18:21:40.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020621289999999998, 2024-08-17T18:21:40.245 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:21:40.245 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:21:40.245 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-08-17T18:21:40.245 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:21:40.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:21:40.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205845+0000", 2024-08-17T18:21:40.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:21:40.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:21:40.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205845+0000", 2024-08-17T18:21:40.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:21:40.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:21:40.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205845+0000", 2024-08-17T18:21:40.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1600000000000002e-07 2024-08-17T18:21:40.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:21:40.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205873+0000", 2024-08-17T18:21:40.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8109000000000001e-05 2024-08-17T18:21:40.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:21:40.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205879+0000", 2024-08-17T18:21:40.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0850000000000002e-06 2024-08-17T18:21:40.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:21:40.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.205885+0000", 2024-08-17T18:21:40.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9190000000000004e-06 2024-08-17T18:21:40.248 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:21:40.248 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:21:40.248 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:21:40.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.248 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.36936.0:69315 58.d e1711/1701 58:b01729d6:::benchmark_data_smithi136_442124_object69314:head v 1711'4303, mlcod=1711'4301)", 2024-08-17T18:21:40.248 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:21:40.206552+0000", 2024-08-17T18:21:40.248 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.000965163, 2024-08-17T18:21:40.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001376314, 2024-08-17T18:21:40.248 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:21:40.249 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:21:40.249 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T18:21:40.249 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:21:40.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:21:40.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.206552+0000", 2024-08-17T18:21:40.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:21:40.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:21:40.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.206552+0000", 2024-08-17T18:21:40.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:21:40.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:21:40.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.206553+0000", 2024-08-17T18:21:40.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.99e-07 2024-08-17T18:21:40.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:21:40.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.206575+0000", 2024-08-17T18:21:40.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2118000000000001e-05 2024-08-17T18:21:40.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:21:40.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.206586+0000", 2024-08-17T18:21:40.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0715e-05 2024-08-17T18:21:40.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:21:40.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.206592+0000", 2024-08-17T18:21:40.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0850000000000002e-06 2024-08-17T18:21:40.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:21:40.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.206960+0000", 2024-08-17T18:21:40.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036803399999999999 2024-08-17T18:21:40.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:21:40.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:21:40.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:21:40.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.206977+0000", 2024-08-17T18:21:40.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7180000000000002e-05 2024-08-17T18:21:40.252 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:21:40.252 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:21:40.252 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:21:40.252 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:21:40.252 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-17T18:21:40.252 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 9 2024-08-17T18:21:40.252 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:21:40.252 INFO:tasks.thrashosds.thrasher:in_osds: [2, 4, 6, 0, 5, 7, 3, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:21:40.253 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:21:40.253 INFO:tasks.thrashosds.thrasher:Killing osd 6, live_osds are [0, 1, 3, 4, 5, 6, 7, 2] 2024-08-17T18:21:40.253 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-08-17T18:21:40.253 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.6 2024-08-17T18:21:40.254 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:2024-08-17T18:21:40.216899+0000 min lat: 0.00120749 max lat: 0.207961 avg lat: 0.00922967 2024-08-17T18:21:40.254 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T18:21:40.254 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 40 16 69323 69307 108.271 75.6875 0.0271859 0.00922967 2024-08-17T18:21:40.288 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:40.325 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:40.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:40.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:40.430 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:40.467 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:40.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:40.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:40.569 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:40.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:40.607 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:40.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:40.670 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:21:40 smithi165 systemd[1]: Stopping Ceph osd.6 for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T18:21:40.709 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:40.745 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:40.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:40.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:40.847 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:40.868 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:40.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:40.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:40.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:40 smithi136 ceph-mon[109861]: osd.6 marked itself down and dead 2024-08-17T18:21:40.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:40 smithi136 ceph-mon[115078]: osd.6 marked itself down and dead 2024-08-17T18:21:40.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:40.995 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:40.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:40.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:40.999 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:40 smithi165 ceph-mon[105517]: osd.6 marked itself down and dead 2024-08-17T18:21:40.999 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:21:40 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[52331]: 2024-08-17T18:21:40.668+0000 7f914cd12640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T18:21:40.999 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:21:40 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[52331]: 2024-08-17T18:21:40.668+0000 7f914cd12640 -1 osd.6 1711 *** Got signal Terminated *** 2024-08-17T18:21:40.999 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:21:40 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[52331]: 2024-08-17T18:21:40.668+0000 7f914cd12640 -1 osd.6 1711 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T18:21:41.097 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:41.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:41.121 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:41.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:41.207 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-17T18:21:41.216 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 41 16 69982 69966 106.635 41.1875 0.00373444 0.00927028 2024-08-17T18:21:41.222 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:41.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:41.244 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:41.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:41.285 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:21:41.345 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:41.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:41.383 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:41.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:41.484 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:41.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:41.513 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:41.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:41.614 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:41.639 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:41.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:41.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:41.741 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:41.742 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:21:41 smithi165 podman[280933]: 2024-08-17 18:21:41.43547799 +0000 UTC m=+0.896413081 container died 7bc5fd925293f228b006ad617b0784075e0bdc286546605647daec6a2b152f14 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=reef-ced649f, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-08-17T18:21:41.768 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:41.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:41.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:41.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:41.897 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:21:41.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:41.935 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:41.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:42.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:41 smithi165 ceph-mon[105517]: pgmap v3337: 99 pgs: 99 active+clean; 4.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 57 MiB/s wr, 904 op/s 2024-08-17T18:21:42.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:41 smithi165 ceph-mon[105517]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T18:21:42.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:41 smithi165 ceph-mon[105517]: osdmap e1712: 8 total, 7 up, 8 in 2024-08-17T18:21:42.011 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:21:41 smithi165 podman[280933]: 2024-08-17 18:21:41.941906747 +0000 UTC m=+1.402841827 container cleanup 7bc5fd925293f228b006ad617b0784075e0bdc286546605647daec6a2b152f14 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=reef-ced649f, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0) 2024-08-17T18:21:42.011 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:21:41 smithi165 bash[280933]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6 2024-08-17T18:21:42.035 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:41 smithi136 ceph-mon[109861]: pgmap v3337: 99 pgs: 99 active+clean; 4.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 57 MiB/s wr, 904 op/s 2024-08-17T18:21:42.035 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:41 smithi136 ceph-mon[109861]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T18:21:42.035 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:41 smithi136 ceph-mon[109861]: osdmap e1712: 8 total, 7 up, 8 in 2024-08-17T18:21:42.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:41 smithi136 ceph-mon[115078]: pgmap v3337: 99 pgs: 99 active+clean; 4.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 57 MiB/s wr, 904 op/s 2024-08-17T18:21:42.035 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:41 smithi136 ceph-mon[115078]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T18:21:42.036 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:41 smithi136 ceph-mon[115078]: osdmap e1712: 8 total, 7 up, 8 in 2024-08-17T18:21:42.036 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:42.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:42.085 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:42.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:42.186 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:42.211 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:42.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:42.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:42.217 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 42 16 70478 70462 104.833 31 0.0150608 0.00953363 2024-08-17T18:21:42.313 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:42.342 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:42.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:42.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:42.444 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:42.487 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:42.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:42.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:42.590 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:42.624 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:42.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:42.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:42.691 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:21:42 smithi165 podman[281011]: 2024-08-17 18:21:42.268725769 +0000 UTC m=+0.826735999 container remove 7bc5fd925293f228b006ad617b0784075e0bdc286546605647daec6a2b152f14 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-ced649f, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-08-17T18:21:42.726 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:42.764 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:42.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:42.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:42.866 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:42.903 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:42.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:42.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:42.990 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:42 smithi165 ceph-mon[105517]: osdmap e1713: 8 total, 7 up, 8 in 2024-08-17T18:21:42.990 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:21:42 smithi165 podman[281170]: 2024-08-17 18:21:42.596996714 +0000 UTC m=+0.027551366 image pull c870753d67c2504e5ac044c83696772f8b7527a928cc764ed24eeea7fea57d6e quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408 2024-08-17T18:21:42.990 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:21:42 smithi165 podman[281170]: 2024-08-17 18:21:42.749642034 +0000 UTC m=+0.180196675 container create 4c8a0cbbeb20c10c568f04e2703e2056e5ef9832ffca0eeddd6d288bca2796d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=reef-ced649f, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T18:21:43.004 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:43.030 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:42 smithi136 ceph-mon[109861]: osdmap e1713: 8 total, 7 up, 8 in 2024-08-17T18:21:43.031 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:42 smithi136 ceph-mon[115078]: osdmap e1713: 8 total, 7 up, 8 in 2024-08-17T18:21:43.032 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:43.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:43.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:43.134 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:43.187 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:43.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:43.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:43.217 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 43 16 71689 71673 104.155 75.6875 0.0130798 0.00959557 2024-08-17T18:21:43.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:21:42 smithi165 podman[281170]: 2024-08-17 18:21:42.989689785 +0000 UTC m=+0.420244433 container init 4c8a0cbbeb20c10c568f04e2703e2056e5ef9832ffca0eeddd6d288bca2796d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=reef-ced649f, io.buildah.version=1.36.0) 2024-08-17T18:21:43.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:21:42 smithi165 podman[281170]: 2024-08-17 18:21:42.996318859 +0000 UTC m=+0.426873516 container start 4c8a0cbbeb20c10c568f04e2703e2056e5ef9832ffca0eeddd6d288bca2796d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=reef-ced649f, io.buildah.version=1.36.0) 2024-08-17T18:21:43.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:21:43 smithi165 podman[281170]: 2024-08-17 18:21:43.098025272 +0000 UTC m=+0.528579921 container attach 4c8a0cbbeb20c10c568f04e2703e2056e5ef9832ffca0eeddd6d288bca2796d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=reef-ced649f, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-08-17T18:21:43.290 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:43.320 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:43.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:43.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:43.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:43.447 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:43.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:43.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:43.550 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:43.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:43.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:43.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:43.675 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:43.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:43.708 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:43.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:43.761 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:21:43 smithi165 podman[281170]: 2024-08-17 18:21:43.499468305 +0000 UTC m=+0.930022966 container died 4c8a0cbbeb20c10c568f04e2703e2056e5ef9832ffca0eeddd6d288bca2796d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=reef-ced649f, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-08-17T18:21:43.808 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:43.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:43.832 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:43.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:43.933 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:43.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:43.959 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:43.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:44.061 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:44.087 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:44.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:44.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:44.189 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:44.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:43 smithi136 ceph-mon[109861]: pgmap v3340: 99 pgs: 8 active+undersized+degraded, 11 active+undersized, 6 peering, 1 stale+active+clean, 73 active+clean; 4.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 12 KiB/s rd, 90 MiB/s wr, 1.45k op/s; 13292/210546 objects degraded (6.313%); 54 KiB/s, 0 objects/s recovering 2024-08-17T18:21:44.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:43 smithi136 ceph-mon[109861]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-17T18:21:44.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:43 smithi136 ceph-mon[109861]: Health check failed: Degraded data redundancy: 13292/210546 objects degraded (6.313%), 8 pgs degraded (PG_DEGRADED) 2024-08-17T18:21:44.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:43 smithi136 ceph-mon[115078]: pgmap v3340: 99 pgs: 8 active+undersized+degraded, 11 active+undersized, 6 peering, 1 stale+active+clean, 73 active+clean; 4.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 12 KiB/s rd, 90 MiB/s wr, 1.45k op/s; 13292/210546 objects degraded (6.313%); 54 KiB/s, 0 objects/s recovering 2024-08-17T18:21:44.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:43 smithi136 ceph-mon[115078]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-17T18:21:44.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:43 smithi136 ceph-mon[115078]: Health check failed: Degraded data redundancy: 13292/210546 objects degraded (6.313%), 8 pgs degraded (PG_DEGRADED) 2024-08-17T18:21:44.212 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:44.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:44.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:44.217 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 44 16 72810 72794 103.38 70.0625 0.0195452 0.00966753 2024-08-17T18:21:44.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:43 smithi165 ceph-mon[105517]: pgmap v3340: 99 pgs: 8 active+undersized+degraded, 11 active+undersized, 6 peering, 1 stale+active+clean, 73 active+clean; 4.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 12 KiB/s rd, 90 MiB/s wr, 1.45k op/s; 13292/210546 objects degraded (6.313%); 54 KiB/s, 0 objects/s recovering 2024-08-17T18:21:44.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:43 smithi165 ceph-mon[105517]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-17T18:21:44.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:43 smithi165 ceph-mon[105517]: Health check failed: Degraded data redundancy: 13292/210546 objects degraded (6.313%), 8 pgs degraded (PG_DEGRADED) 2024-08-17T18:21:44.314 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:44.338 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:44.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:44.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:44.440 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:44.476 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:44.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:44.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:44.578 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:44.600 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.6.service' 2024-08-17T18:21:44.605 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:44.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:44.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:44.664 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T18:21:44.664 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-08-17T18:21:44.707 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:44.747 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:44.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:44.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:44.849 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:44.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:44.874 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:44.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:44.975 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:45.008 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:45.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:45.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:45.110 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:45.134 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:45.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:45.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:45.217 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 45 16 74037 74021 102.787 76.6875 0.0105547 0.00972353 2024-08-17T18:21:45.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:45.262 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:45.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:45.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:45.364 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:45.389 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:45.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:45.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:45.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:45.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:45.532 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:45.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:45.633 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:45.657 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:45.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:45.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:45.759 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:45.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:45.809 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:45.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:45.828 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:21:45.830 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:21:45.910 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:45.922 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:21:45.923 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:21:45.962 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:21:45.963 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:21:45.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:45.970 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:45.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:45.983 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:45 smithi136 ceph-mon[109861]: pgmap v3341: 99 pgs: 8 active+undersized+degraded, 11 active+undersized, 6 peering, 1 stale+active+clean, 73 active+clean; 4.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 11 KiB/s rd, 47 MiB/s wr, 758 op/s; 13292/210546 objects degraded (6.313%); 48 KiB/s, 0 objects/s recovering 2024-08-17T18:21:45.984 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:45 smithi136 ceph-mon[115078]: pgmap v3341: 99 pgs: 8 active+undersized+degraded, 11 active+undersized, 6 peering, 1 stale+active+clean, 73 active+clean; 4.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 11 KiB/s rd, 47 MiB/s wr, 758 op/s; 13292/210546 objects degraded (6.313%); 48 KiB/s, 0 objects/s recovering 2024-08-17T18:21:45.997 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:21:45.998 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:21:46.037 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:21:46.037 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:21:46.065 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:21:46.066 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:21:46.071 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:46.098 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:21:46.098 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:21:46.099 INFO:teuthology.orchestra.run.smithi136.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-17T18:21:46.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:46.112 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:46.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:46.128 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:21:46.128 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:21:46.213 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:46.218 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 46 16 75085 75069 101.976 65.5 0.012575 0.00980023 2024-08-17T18:21:46.247 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:46.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:46.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:46.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:45 smithi165 ceph-mon[105517]: pgmap v3341: 99 pgs: 8 active+undersized+degraded, 11 active+undersized, 6 peering, 1 stale+active+clean, 73 active+clean; 4.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 11 KiB/s rd, 47 MiB/s wr, 758 op/s; 13292/210546 objects degraded (6.313%); 48 KiB/s, 0 objects/s recovering 2024-08-17T18:21:46.349 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:46.373 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:46.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:46.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:46.475 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:46.508 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:46.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:46.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:46.610 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:46.687 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:46.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:46.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:46.789 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:46.828 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:46.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:46.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:46.930 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:46.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:46 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2792643880' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:21:46.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:46 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2792643880' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:21:46.955 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:46.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:46.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:47.057 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:47.079 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:47.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:47.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:47.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:47.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:47.206 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:47.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:47.218 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 47 16 76318 76302 101.445 77.0625 0.018416 0.00985167 2024-08-17T18:21:47.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:46 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2792643880' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:21:47.308 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:47.356 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:47.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:47.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:47.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:47.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:47.483 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:47.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:47.584 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:47.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:47.626 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:47.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:47.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:21:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:21:47] "GET /metrics HTTP/1.1" 200 36727 "" "Prometheus/2.51.0" 2024-08-17T18:21:47.727 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:47.751 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:47.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:47.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:47.853 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:47.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:47.921 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:47.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:48.022 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:48.060 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:48.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:48.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:48.164 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:48.189 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:48.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:48.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:48.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:47 smithi136 ceph-mon[109861]: pgmap v3342: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 4.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 9.2 KiB/s rd, 59 MiB/s wr, 957 op/s; 27266/218085 objects degraded (12.502%); 49 KiB/s, 0 objects/s recovering 2024-08-17T18:21:48.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:47 smithi136 ceph-mon[109861]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-17T18:21:48.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:47 smithi136 ceph-mon[115078]: pgmap v3342: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 4.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 9.2 KiB/s rd, 59 MiB/s wr, 957 op/s; 27266/218085 objects degraded (12.502%); 49 KiB/s, 0 objects/s recovering 2024-08-17T18:21:48.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:47 smithi136 ceph-mon[115078]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-17T18:21:48.218 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 48 16 77478 77462 100.842 72.5 0.00538206 0.00991071 2024-08-17T18:21:48.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:47 smithi165 ceph-mon[105517]: pgmap v3342: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 4.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 9.2 KiB/s rd, 59 MiB/s wr, 957 op/s; 27266/218085 objects degraded (12.502%); 49 KiB/s, 0 objects/s recovering 2024-08-17T18:21:48.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:47 smithi165 ceph-mon[105517]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-17T18:21:48.291 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:48.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:48.338 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:48.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:48.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:48.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:48.476 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:48.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:48.577 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:48.613 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:48.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:48.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:48.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:48.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:48.755 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:48.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:48.763 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-17T18:21:48.764 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-17T18:21:48.856 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:48.901 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:48.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:48.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:48.985 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:48 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 28340/226974 objects degraded (12.486%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:21:48.985 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:48 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 28340/226974 objects degraded (12.486%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:21:49.003 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:49.042 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:49.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:49.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:49.147 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:49.190 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:49.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:49.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:49.218 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 49 16 78604 78588 100.22 70.375 0.02676 0.00997207 2024-08-17T18:21:49.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:48 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 28340/226974 objects degraded (12.486%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:21:49.292 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:49.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:49.332 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:49.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:49.433 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:49.474 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:49.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:49.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:49.576 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:49.618 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:49.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:49.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:49.664 INFO:tasks.thrashosds.thrasher:in_osds: [2, 4, 6, 0, 5, 7, 3, 1] out_osds: [] dead_osds: [6] live_osds: [0, 1, 3, 4, 5, 7, 2] 2024-08-17T18:21:49.665 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:21:49.665 INFO:tasks.thrashosds.thrasher:Reweighting osd 1 to 0.5171000204088023 2024-08-17T18:21:49.665 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 1 0.5171000204088023 2024-08-17T18:21:49.720 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:49.743 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:49.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:49.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:49.845 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:49.870 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:49.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:49.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:49.972 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:50.013 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:50.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:50.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:50.115 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:50.148 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:50.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:50.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:50.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:49 smithi136 ceph-mon[109861]: pgmap v3343: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 4.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 9.2 KiB/s rd, 76 MiB/s wr, 1.23k op/s; 28340/226974 objects degraded (12.486%); 49 KiB/s, 0 objects/s recovering 2024-08-17T18:21:50.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:49 smithi136 ceph-mon[115078]: pgmap v3343: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 4.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 9.2 KiB/s rd, 76 MiB/s wr, 1.23k op/s; 28340/226974 objects degraded (12.486%); 49 KiB/s, 0 objects/s recovering 2024-08-17T18:21:50.218 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 50 16 79652 79636 99.526 65.5 0.00916141 0.0100418 2024-08-17T18:21:50.249 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:50.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:49 smithi165 ceph-mon[105517]: pgmap v3343: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 4.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 9.2 KiB/s rd, 76 MiB/s wr, 1.23k op/s; 28340/226974 objects degraded (12.486%); 49 KiB/s, 0 objects/s recovering 2024-08-17T18:21:50.291 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:50.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:50.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:50.393 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:50.427 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:50.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:50.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:50.531 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:50.555 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:50.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:50.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:50.657 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:50.694 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:50.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:50.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:50.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:21:50.796 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:50.848 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:50.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:50.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:50.950 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:51.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:51.013 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:51.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:51.114 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:51.158 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:51.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:51.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:51.218 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 51 16 80651 80635 98.7988 62.4375 0.00910609 0.0101157 2024-08-17T18:21:51.260 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:51.281 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:51.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:51.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:51.383 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:51.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:51.422 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:51.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:51.523 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:51.547 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:51.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:51.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:51.648 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:51.691 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:51.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:51.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:51.793 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:51.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:51.846 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:51.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:51.947 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:51.988 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:51.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:51.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:52.090 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:52.134 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:51 smithi136 ceph-mon[109861]: pgmap v3344: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 4.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 7.6 KiB/s rd, 67 MiB/s wr, 1.08k op/s; 28340/229245 objects degraded (12.362%); 40 KiB/s, 0 objects/s recovering 2024-08-17T18:21:52.135 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:21:52.135 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:21:52.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:51 smithi136 ceph-mon[115078]: pgmap v3344: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 4.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 7.6 KiB/s rd, 67 MiB/s wr, 1.08k op/s; 28340/229245 objects degraded (12.362%); 40 KiB/s, 0 objects/s recovering 2024-08-17T18:21:52.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:21:52.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:21:52.136 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:52.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:52.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:52.218 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 52 16 81695 81679 98.1535 65.25 0.0158237 0.0101818 2024-08-17T18:21:52.238 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:52.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:51 smithi165 ceph-mon[105517]: pgmap v3344: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 4.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 7.6 KiB/s rd, 67 MiB/s wr, 1.08k op/s; 28340/229245 objects degraded (12.362%); 40 KiB/s, 0 objects/s recovering 2024-08-17T18:21:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:21:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:21:52.274 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:52.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:52.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:52.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:52.425 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:52.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:52.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:52.527 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:52.548 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:52.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:52.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:52.650 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:52.689 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:52.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:52.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:52.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:52.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:52.816 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:52.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:52.917 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:52.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:52.950 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:52.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:53.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:53.074 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:53.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:53.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:53.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:53.201 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:53.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:53.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:53.218 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 53 16 82685 82669 97.4689 61.875 0.0279092 0.0102535 2024-08-17T18:21:53.303 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:53.327 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:53.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:53.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:53.429 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:53.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:53.473 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:53.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:53.574 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:53.622 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:53.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:53.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:53.724 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:53.762 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:53.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:53.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:53.864 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:53.886 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:53.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:53.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:53.988 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:54.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:53 smithi165 ceph-mon[105517]: pgmap v3345: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 5.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 78 MiB/s wr, 1.24k op/s; 30392/243528 objects degraded (12.480%); 12 KiB/s, 0 objects/s recovering 2024-08-17T18:21:54.024 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:54.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:54.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:54.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:54.151 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:54.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:54.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:54.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:53 smithi136 ceph-mon[109861]: pgmap v3345: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 5.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 78 MiB/s wr, 1.24k op/s; 30392/243528 objects degraded (12.480%); 12 KiB/s, 0 objects/s recovering 2024-08-17T18:21:54.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:53 smithi136 ceph-mon[115078]: pgmap v3345: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 5.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 78 MiB/s wr, 1.24k op/s; 30392/243528 objects degraded (12.480%); 12 KiB/s, 0 objects/s recovering 2024-08-17T18:21:54.219 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 54 16 83672 83656 96.8059 61.6875 0.0259447 0.0103235 2024-08-17T18:21:54.253 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:54.277 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:54.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:54.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:54.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:54.393 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:21:54.393 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:21:54.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:54.460 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:54.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:54.561 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:54.616 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:54.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:54.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:54.718 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:54.741 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:54.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:54.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:54.843 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:54.866 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:54.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:54.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:54.968 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:55.013 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:55.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:55.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:55.115 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:55.158 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:55.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:55.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:55.219 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 55 16 84634 84618 96.139 60.125 0.0103733 0.0103966 2024-08-17T18:21:55.260 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:55.300 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:55.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:55.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:55.402 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:55.434 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:21:55.435 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:55 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:21:55.435 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:55 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:21:55.435 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:55 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2722941902' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:21:55.435 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:55 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:21:55.435 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:55 smithi136 ceph-mon[109861]: pgmap v3346: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 5.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 57 MiB/s wr, 915 op/s; 30392/243528 objects degraded (12.480%); 5.3 KiB/s, 0 objects/s recovering 2024-08-17T18:21:55.435 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:55 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 30392/243528 objects degraded (12.480%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:21:55.436 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:55 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:21:55.436 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:55 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:21:55.436 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:55 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2722941902' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:21:55.436 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:55 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:21:55.436 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:55 smithi136 ceph-mon[115078]: pgmap v3346: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 5.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 57 MiB/s wr, 915 op/s; 30392/243528 objects degraded (12.480%); 5.3 KiB/s, 0 objects/s recovering 2024-08-17T18:21:55.436 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:55 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 30392/243528 objects degraded (12.480%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:21:55.441 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:55.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:55.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:55 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:21:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:55 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:21:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:55 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2722941902' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:21:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:55 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:21:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:55 smithi165 ceph-mon[105517]: pgmap v3346: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 5.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 57 MiB/s wr, 915 op/s; 30392/243528 objects degraded (12.480%); 5.3 KiB/s, 0 objects/s recovering 2024-08-17T18:21:55.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:55 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 30392/243528 objects degraded (12.480%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:21:55.543 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:55.567 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:55.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:55.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:55.663 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:21:55.669 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:55.690 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:55.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:55.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:55.792 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:55.816 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:55.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:55.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:55.919 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:55.964 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:55.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:55.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:56.066 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:56.074 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:21:56.106 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-17T18:21:56.131 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:56.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:56.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:56.222 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 56 16 85636 85620 95.5392 62.625 0.0212964 0.0104604 2024-08-17T18:21:56.233 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:56.254 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:56.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:56.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:56.311 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:56 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:21:56.312 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:56 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:21:56.312 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:56 smithi136 ceph-mon[109861]: osdmap e1714: 8 total, 7 up, 8 in 2024-08-17T18:21:56.312 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:56 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:21:56.312 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:56 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:21:56.312 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:56 smithi136 ceph-mon[115078]: osdmap e1714: 8 total, 7 up, 8 in 2024-08-17T18:21:56.356 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:56.405 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:56.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:56.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:56.507 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:56.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:56 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:21:56.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:56 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:21:56.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:56 smithi165 ceph-mon[105517]: osdmap e1714: 8 total, 7 up, 8 in 2024-08-17T18:21:56.544 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:56.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:56.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:56.646 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:56.685 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:56.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:56.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:56.787 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:56.829 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:56.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:56.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:56.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:21:56.964 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:56.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:56.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:21:57.066 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:57.096 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:57.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:57.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:57.198 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:57.220 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 57 16 86682 86666 95.0099 65.375 0.0133337 0.0105193 2024-08-17T18:21:57.222 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:57.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:57.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:57.324 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:57.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:57.380 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:57.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:57.392 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:57 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:21:57.392 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:57 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:21:57.392 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:57 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:21:57.392 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:57 smithi136 ceph-mon[109861]: pgmap v3348: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 5.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 31332/250854 objects degraded (12.490%) 2024-08-17T18:21:57.393 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:57 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:21:57.393 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:57 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:21:57.393 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:57 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:21:57.393 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:57 smithi136 ceph-mon[115078]: pgmap v3348: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 5.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 31332/250854 objects degraded (12.490%) 2024-08-17T18:21:57.482 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:57.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:57 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:21:57.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:57 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:21:57.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:57 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:21:57.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:57 smithi165 ceph-mon[105517]: pgmap v3348: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 5.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 31332/250854 objects degraded (12.490%) 2024-08-17T18:21:57.522 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:57.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:57.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:57.624 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:57.647 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:57.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:57.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:57.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:21:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:21:57] "GET /metrics HTTP/1.1" 200 36762 "" "Prometheus/2.51.0" 2024-08-17T18:21:57.749 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:57.771 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:57.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:57.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:57.873 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:57.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:57.916 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:57.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:58.017 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:58.076 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:58.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:58.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:58.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:58.202 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:58.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:58.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:58.220 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 58 16 87735 87719 94.5065 65.8125 0.0101657 0.010576 2024-08-17T18:21:58.304 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:21:58.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:58.345 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:58.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:21:58.445 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:58.470 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:58.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:58.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:58.545 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:21:58.572 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:58.617 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:58.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:58.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:58.719 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:58.758 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:58.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:58.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:58.860 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:21:58.909 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:58.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:58.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:21:59.012 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:59.036 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:59.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:59.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:59.138 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:59.160 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:59.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:59.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:59.220 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 59 16 88779 88763 94.0105 65.25 0.0171649 0.0106317 2024-08-17T18:21:59.262 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:21:59.308 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:59.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:59.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:21:59.410 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:59.435 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:59.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:59.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:59.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:21:59.559 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:59.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:59.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:21:59.661 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:21:59.685 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:21:59.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:59.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:21:59.787 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:59.824 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:59.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:59.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:21:59.926 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:21:59.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:59 smithi136 ceph-mon[109861]: pgmap v3349: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 5.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 65 MiB/s wr, 1.05k op/s; 32281/258414 objects degraded (12.492%) 2024-08-17T18:21:59.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:21:59 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3017305323' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:21:59.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:59 smithi136 ceph-mon[115078]: pgmap v3349: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 5.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 65 MiB/s wr, 1.05k op/s; 32281/258414 objects degraded (12.492%) 2024-08-17T18:21:59.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:21:59 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3017305323' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:21:59.950 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:21:59.964 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:21:59.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:21:59.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:00.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:59 smithi165 ceph-mon[105517]: pgmap v3349: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 5.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 65 MiB/s wr, 1.05k op/s; 32281/258414 objects degraded (12.492%) 2024-08-17T18:22:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:21:59 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3017305323' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:22:00.066 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:00.118 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:00.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:00.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:00.219 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:00.221 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:2024-08-17T18:22:00.220615+0000 min lat: 0.00120749 max lat: 1.05177 avg lat: 0.0106928 2024-08-17T18:22:00.222 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T18:22:00.222 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 60 16 89770 89754 93.476 61.9375 0.0256014 0.0106928 2024-08-17T18:22:00.265 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:00.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:00.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:00.367 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:00.389 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:00.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:00.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:00.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:00.539 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:00.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:00.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:00.640 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:00.678 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:00.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:00.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:00.780 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:00.803 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:00.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:00.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:00.905 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:00.928 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:00.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:00.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:00.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:00 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 32281/258414 objects degraded (12.492%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:00.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:00 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 32281/258414 objects degraded (12.492%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:01.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:00 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 32281/258414 objects degraded (12.492%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:01.030 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:01.072 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:01.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:01.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:01.123 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:22:01.123 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:22:01.123 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:22:01.123 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:22:01.124 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:22:01.124 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:22:01.124 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:22:01.124 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:22:01.124 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:22:01.124 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:22:01.124 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:22:01.124 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:22:01.124 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:22:01.124 INFO:teuthology.orchestra.run.smithi136.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-17T18:22:01.124 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:22:01.174 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:01.201 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:01.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:01.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:01.220 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 61 16 90796 90780 92.9947 64.125 0.0170483 0.0107477 2024-08-17T18:22:01.303 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:01.325 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:01.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:01.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:01.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:01.451 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:01.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:01.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:01.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:01.594 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:01.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:01.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:01.696 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:01.729 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:01 smithi136 ceph-mon[109861]: pgmap v3350: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 5.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s; 32281/260286 objects degraded (12.402%) 2024-08-17T18:22:01.730 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:01 smithi136 ceph-mon[115078]: pgmap v3350: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 5.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s; 32281/260286 objects degraded (12.402%) 2024-08-17T18:22:01.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:01.731 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:01.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:01.832 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:01.858 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:01.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:01.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:01.960 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:02.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:02.003 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:02.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:02.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:01 smithi165 ceph-mon[105517]: pgmap v3350: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 5.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s; 32281/260286 objects degraded (12.402%) 2024-08-17T18:22:02.104 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:02.142 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:02.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:02.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:02.220 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 62 16 91822 91806 92.5291 64.125 0.0161707 0.0108012 2024-08-17T18:22:02.244 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:02.284 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:02.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:02.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:02.305 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-17T18:22:02.305 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-17T18:22:02.387 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:02.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:02.434 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:02.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:02.538 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:02.581 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:02.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:02.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:02.683 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:02.722 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:02.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:02.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:02.824 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:02.849 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:02.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:02.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:02.951 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:02.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:02.995 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:02.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:03.096 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:03.138 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:03.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:03.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:03.220 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 63 16 92788 92772 92.0186 60.375 0.0070742 0.0108617 2024-08-17T18:22:03.240 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:03.284 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:03.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:03.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:03.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:03.411 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:03.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:03.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:03.513 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:03.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:03.563 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:03.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:03.564 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:22:03.656 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.1 to 0.5171 (8460) 2024-08-17T18:22:03.664 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:03.688 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:03.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:03.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:03.790 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:03.812 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:03.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:03.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:03.913 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:03.934 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:03.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:03.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:03 smithi136 ceph-mon[109861]: pgmap v3351: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 34243/273849 objects degraded (12.504%) 2024-08-17T18:22:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:03 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2839602265' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 0.5171000204088023} : dispatch 2024-08-17T18:22:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:03 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 0.5171000204088023} : dispatch 2024-08-17T18:22:03.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:03 smithi136 ceph-mon[115078]: pgmap v3351: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 34243/273849 objects degraded (12.504%) 2024-08-17T18:22:03.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:03 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2839602265' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 0.5171000204088023} : dispatch 2024-08-17T18:22:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:03 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 0.5171000204088023} : dispatch 2024-08-17T18:22:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:03 smithi165 ceph-mon[105517]: pgmap v3351: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 34243/273849 objects degraded (12.504%) 2024-08-17T18:22:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:03 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2839602265' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 0.5171000204088023} : dispatch 2024-08-17T18:22:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:03 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 0.5171000204088023} : dispatch 2024-08-17T18:22:04.036 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:04.058 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:04.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:04.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:04.159 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:04.181 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:04.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:04.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:04.220 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 64 16 93264 93248 91.0457 29.75 0.00322903 0.0108901 2024-08-17T18:22:04.283 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:04.305 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:04.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:04.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:04.306 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:22:04.407 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:04.433 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:04.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:04.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:04.535 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:04.561 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:04.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:04.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:04.663 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:04.690 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:04.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:04.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:04.791 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:04.813 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:04.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:04.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:04.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:04.920 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:04 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 0.5171000204088023}]': finished 2024-08-17T18:22:04.921 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:04 smithi136 ceph-mon[115078]: osdmap e1715: 8 total, 7 up, 8 in 2024-08-17T18:22:04.921 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:04 smithi136 ceph-mon[115078]: osdmap e1716: 8 total, 7 up, 8 in 2024-08-17T18:22:04.921 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:04 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 0.5171000204088023}]': finished 2024-08-17T18:22:04.921 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:04 smithi136 ceph-mon[109861]: osdmap e1715: 8 total, 7 up, 8 in 2024-08-17T18:22:04.921 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:04 smithi136 ceph-mon[109861]: osdmap e1716: 8 total, 7 up, 8 in 2024-08-17T18:22:04.941 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:04.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:04.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:04 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 0.5171000204088023}]': finished 2024-08-17T18:22:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:04 smithi165 ceph-mon[105517]: osdmap e1715: 8 total, 7 up, 8 in 2024-08-17T18:22:05.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:04 smithi165 ceph-mon[105517]: osdmap e1716: 8 total, 7 up, 8 in 2024-08-17T18:22:05.043 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:05.069 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:05.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:05.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:05.171 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:05.196 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:05.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:05.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:05.220 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 65 16 93264 93248 89.6451 0 - 0.0108901 2024-08-17T18:22:05.298 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:05.320 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:05.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:05.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:05.421 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:05.447 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:05.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:05.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:05.549 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:05.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:05.584 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:05.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:05.606 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:22:05.627 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:22:05.627 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-17T18:22:05.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-17T18:22:05.628 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:22:05.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.628 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207580 56.10 56:0b80fbc4:::benchmark_data_smithi136_356456_object80876:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:22:05.628 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.637715+0000", 2024-08-17T18:22:05.628 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 565.98757747299999, 2024-08-17T18:22:05.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4471485369999999, 2024-08-17T18:22:05.628 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:22:05.628 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:22:05.628 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:05.628 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:22:05.628 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:22:05.628 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:22:05.628 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207580 2024-08-17T18:22:05.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.629 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:22:05.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:22:05.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637715+0000", 2024-08-17T18:22:05.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:22:05.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637715+0000", 2024-08-17T18:22:05.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:22:05.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637717+0000", 2024-08-17T18:22:05.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9860000000000001e-06 2024-08-17T18:22:05.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:22:05.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637722+0000", 2024-08-17T18:22:05.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5890000000000002e-06 2024-08-17T18:22:05.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:22:05.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637739+0000", 2024-08-17T18:22:05.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6606e-05 2024-08-17T18:22:05.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:22:05.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637749+0000", 2024-08-17T18:22:05.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6530000000000006e-06 2024-08-17T18:22:05.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.632 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637835+0000", 2024-08-17T18:22:05.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6347000000000005e-05 2024-08-17T18:22:05.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.632 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-08-17T18:22:05.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637842+0000", 2024-08-17T18:22:05.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8759999999999999e-06 2024-08-17T18:22:05.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.639138+0000", 2024-08-17T18:22:05.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012963180000000001 2024-08-17T18:22:05.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:22:05.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.639147+0000", 2024-08-17T18:22:05.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0610000000000002e-06 2024-08-17T18:22:05.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.019011+0000", 2024-08-17T18:22:05.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3798637710000001 2024-08-17T18:22:05.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:22:05.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.019132+0000", 2024-08-17T18:22:05.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012087300000000001 2024-08-17T18:22:05.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:22:05.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.019367+0000", 2024-08-17T18:22:05.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023531499999999999 2024-08-17T18:22:05.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.021924+0000", 2024-08-17T18:22:05.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025573229999999998 2024-08-17T18:22:05.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.021962+0000", 2024-08-17T18:22:05.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7457999999999999e-05 2024-08-17T18:22:05.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:22:05.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.021974+0000", 2024-08-17T18:22:05.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1963999999999999e-05 2024-08-17T18:22:05.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.022118+0000", 2024-08-17T18:22:05.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014464799999999999 2024-08-17T18:22:05.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.084688+0000", 2024-08-17T18:22:05.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.062569939000000005 2024-08-17T18:22:05.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:22:05.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.084755+0000", 2024-08-17T18:22:05.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6804999999999995e-05 2024-08-17T18:22:05.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:22:05.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.084863+0000", 2024-08-17T18:22:05.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010800499999999999 2024-08-17T18:22:05.639 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.639 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:22:05.640 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.640 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207645 56.10 56:0f00368f:::benchmark_data_smithi136_356456_object80941:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:22:05.640 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.637768+0000", 2024-08-17T18:22:05.640 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 565.98752458199999, 2024-08-17T18:22:05.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4472689160000001, 2024-08-17T18:22:05.640 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:22:05.640 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:22:05.640 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:05.640 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:22:05.640 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:22:05.640 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:22:05.641 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207645 2024-08-17T18:22:05.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.641 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:22:05.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:22:05.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637768+0000", 2024-08-17T18:22:05.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:22:05.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637768+0000", 2024-08-17T18:22:05.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:22:05.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637768+0000", 2024-08-17T18:22:05.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5000000000000003e-07 2024-08-17T18:22:05.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:22:05.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637771+0000", 2024-08-17T18:22:05.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8090000000000001e-06 2024-08-17T18:22:05.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:22:05.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637778+0000", 2024-08-17T18:22:05.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3130000000000001e-06 2024-08-17T18:22:05.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:22:05.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637784+0000", 2024-08-17T18:22:05.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5809999999999996e-06 2024-08-17T18:22:05.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637912+0000", 2024-08-17T18:22:05.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012843300000000001 2024-08-17T18:22:05.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:22:05.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637918+0000", 2024-08-17T18:22:05.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.772e-06 2024-08-17T18:22:05.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.639992+0000", 2024-08-17T18:22:05.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002074237 2024-08-17T18:22:05.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:22:05.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640001+0000", 2024-08-17T18:22:05.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3799999999999994e-06 2024-08-17T18:22:05.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.020861+0000", 2024-08-17T18:22:05.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.380859995 2024-08-17T18:22:05.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:22:05.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.020934+0000", 2024-08-17T18:22:05.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3367000000000004e-05 2024-08-17T18:22:05.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:22:05.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.021081+0000", 2024-08-17T18:22:05.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000146961 2024-08-17T18:22:05.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:22:05.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.024244+0000", 2024-08-17T18:22:05.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003163014 2024-08-17T18:22:05.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.024323+0000", 2024-08-17T18:22:05.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8669000000000002e-05 2024-08-17T18:22:05.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.024366+0000", 2024-08-17T18:22:05.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3390999999999998e-05 2024-08-17T18:22:05.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.071609+0000", 2024-08-17T18:22:05.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.047243185 2024-08-17T18:22:05.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.084906+0000", 2024-08-17T18:22:05.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013296894 2024-08-17T18:22:05.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:22:05.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.084940+0000", 2024-08-17T18:22:05.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4344000000000002e-05 2024-08-17T18:22:05.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:22:05.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.085037+0000", 2024-08-17T18:22:05.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6020999999999994e-05 2024-08-17T18:22:05.650 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.650 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:22:05.650 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.650 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207658 56.10 56:0afb0108:::benchmark_data_smithi136_356456_object80954:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:22:05.650 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.637878+0000", 2024-08-17T18:22:05.650 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 565.98741447299994, 2024-08-17T18:22:05.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.449536841, 2024-08-17T18:22:05.650 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:22:05.650 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:22:05.650 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:05.651 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:22:05.651 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:22:05.651 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:22:05.651 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207658 2024-08-17T18:22:05.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.651 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:22:05.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:22:05.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637878+0000", 2024-08-17T18:22:05.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:22:05.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637878+0000", 2024-08-17T18:22:05.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:22:05.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637879+0000", 2024-08-17T18:22:05.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4099999999999997e-07 2024-08-17T18:22:05.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:22:05.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637883+0000", 2024-08-17T18:22:05.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9639999999999997e-06 2024-08-17T18:22:05.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:22:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637890+0000", 2024-08-17T18:22:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7549999999999994e-06 2024-08-17T18:22:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:22:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637898+0000", 2024-08-17T18:22:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.746e-06 2024-08-17T18:22:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637943+0000", 2024-08-17T18:22:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4591e-05 2024-08-17T18:22:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:22:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637945+0000", 2024-08-17T18:22:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8830000000000002e-06 2024-08-17T18:22:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640036+0000", 2024-08-17T18:22:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002090742 2024-08-17T18:22:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:22:05.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640044+0000", 2024-08-17T18:22:05.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1620000000000008e-06 2024-08-17T18:22:05.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.022173+0000", 2024-08-17T18:22:05.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3821288540000001 2024-08-17T18:22:05.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:22:05.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.022306+0000", 2024-08-17T18:22:05.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013228400000000001 2024-08-17T18:22:05.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:22:05.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.022549+0000", 2024-08-17T18:22:05.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000243916 2024-08-17T18:22:05.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:22:05.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.024334+0000", 2024-08-17T18:22:05.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017846019999999999 2024-08-17T18:22:05.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.024402+0000", 2024-08-17T18:22:05.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8189999999999996e-05 2024-08-17T18:22:05.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.024435+0000", 2024-08-17T18:22:05.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2679999999999999e-05 2024-08-17T18:22:05.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.072897+0000", 2024-08-17T18:22:05.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048462489999999997 2024-08-17T18:22:05.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.087216+0000", 2024-08-17T18:22:05.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.01431905 2024-08-17T18:22:05.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:22:05.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.087285+0000", 2024-08-17T18:22:05.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8727999999999997e-05 2024-08-17T18:22:05.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:22:05.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.087415+0000", 2024-08-17T18:22:05.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012936299999999999 2024-08-17T18:22:05.660 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.660 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:22:05.660 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.660 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207672 56.10 56:0dff339c:::benchmark_data_smithi136_356456_object80968:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:22:05.661 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.637910+0000", 2024-08-17T18:22:05.661 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 565.98738170199999, 2024-08-17T18:22:05.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4496818419999999, 2024-08-17T18:22:05.661 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:22:05.661 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:22:05.661 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:05.661 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:22:05.661 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:22:05.661 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:22:05.661 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207672 2024-08-17T18:22:05.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.661 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:22:05.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:22:05.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637910+0000", 2024-08-17T18:22:05.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:22:05.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637910+0000", 2024-08-17T18:22:05.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:22:05.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637911+0000", 2024-08-17T18:22:05.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1899999999999998e-07 2024-08-17T18:22:05.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:22:05.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637913+0000", 2024-08-17T18:22:05.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2560000000000001e-06 2024-08-17T18:22:05.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:22:05.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637920+0000", 2024-08-17T18:22:05.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9210000000000003e-06 2024-08-17T18:22:05.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:22:05.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637925+0000", 2024-08-17T18:22:05.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3639999999999998e-06 2024-08-17T18:22:05.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637968+0000", 2024-08-17T18:22:05.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2484999999999997e-05 2024-08-17T18:22:05.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:22:05.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637972+0000", 2024-08-17T18:22:05.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4800000000000003e-06 2024-08-17T18:22:05.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640079+0000", 2024-08-17T18:22:05.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021068150000000002 2024-08-17T18:22:05.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:22:05.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640087+0000", 2024-08-17T18:22:05.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9820000000000008e-06 2024-08-17T18:22:05.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.023406+0000", 2024-08-17T18:22:05.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3833189859999999 2024-08-17T18:22:05.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:22:05.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.023479+0000", 2024-08-17T18:22:05.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2476000000000002e-05 2024-08-17T18:22:05.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:22:05.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.023617+0000", 2024-08-17T18:22:05.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000138169 2024-08-17T18:22:05.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:22:05.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.025302+0000", 2024-08-17T18:22:05.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001685305 2024-08-17T18:22:05.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.026534+0000", 2024-08-17T18:22:05.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001232325 2024-08-17T18:22:05.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.026587+0000", 2024-08-17T18:22:05.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2573e-05 2024-08-17T18:22:05.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.072933+0000", 2024-08-17T18:22:05.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.046346119999999998 2024-08-17T18:22:05.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.087463+0000", 2024-08-17T18:22:05.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014529801 2024-08-17T18:22:05.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:22:05.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.087498+0000", 2024-08-17T18:22:05.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5318000000000003e-05 2024-08-17T18:22:05.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:22:05.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.087592+0000", 2024-08-17T18:22:05.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4047000000000003e-05 2024-08-17T18:22:05.670 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.670 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:22:05.670 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.670 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207721 56.10 56:0dd4c85f:::benchmark_data_smithi136_356456_object81017:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:22:05.670 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.637971+0000", 2024-08-17T18:22:05.670 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 565.98732089700002, 2024-08-17T18:22:05.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.454932664, 2024-08-17T18:22:05.671 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:22:05.671 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:22:05.671 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:05.671 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:22:05.671 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:22:05.671 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:22:05.671 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207721 2024-08-17T18:22:05.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.671 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:22:05.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:22:05.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637971+0000", 2024-08-17T18:22:05.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:22:05.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637971+0000", 2024-08-17T18:22:05.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:22:05.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637972+0000", 2024-08-17T18:22:05.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5099999999999999e-07 2024-08-17T18:22:05.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:22:05.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637976+0000", 2024-08-17T18:22:05.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5920000000000001e-06 2024-08-17T18:22:05.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:22:05.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637990+0000", 2024-08-17T18:22:05.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4015e-05 2024-08-17T18:22:05.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:22:05.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.637997+0000", 2024-08-17T18:22:05.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8839999999999994e-06 2024-08-17T18:22:05.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638034+0000", 2024-08-17T18:22:05.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6928999999999998e-05 2024-08-17T18:22:05.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:22:05.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638039+0000", 2024-08-17T18:22:05.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3449999999999999e-06 2024-08-17T18:22:05.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640141+0000", 2024-08-17T18:22:05.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021019279999999999 2024-08-17T18:22:05.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:22:05.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640147+0000", 2024-08-17T18:22:05.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5459999999999997e-06 2024-08-17T18:22:05.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.024461+0000", 2024-08-17T18:22:05.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3843137730000001 2024-08-17T18:22:05.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:22:05.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.024529+0000", 2024-08-17T18:22:05.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8180000000000001e-05 2024-08-17T18:22:05.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:22:05.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.024668+0000", 2024-08-17T18:22:05.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013920600000000001 2024-08-17T18:22:05.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.026651+0000", 2024-08-17T18:22:05.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0019829600000000002 2024-08-17T18:22:05.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.026684+0000", 2024-08-17T18:22:05.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2381e-05 2024-08-17T18:22:05.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:22:05.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.026703+0000", 2024-08-17T18:22:05.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9213999999999999e-05 2024-08-17T18:22:05.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.074256+0000", 2024-08-17T18:22:05.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.047553554999999997 2024-08-17T18:22:05.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.092689+0000", 2024-08-17T18:22:05.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018432568999999999 2024-08-17T18:22:05.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:22:05.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.092761+0000", 2024-08-17T18:22:05.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2466999999999996e-05 2024-08-17T18:22:05.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:22:05.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.092904+0000", 2024-08-17T18:22:05.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014246899999999999 2024-08-17T18:22:05.681 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.681 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:22:05.681 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.682 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207760 56.10 56:094a1205:::benchmark_data_smithi136_356456_object81056:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:22:05.682 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.638010+0000", 2024-08-17T18:22:05.682 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 565.98728183699995, 2024-08-17T18:22:05.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.45637813, 2024-08-17T18:22:05.682 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:22:05.682 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:22:05.682 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:05.683 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:22:05.683 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:22:05.683 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:22:05.683 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207760 2024-08-17T18:22:05.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.683 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:22:05.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:22:05.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638010+0000", 2024-08-17T18:22:05.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:22:05.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638010+0000", 2024-08-17T18:22:05.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:22:05.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638011+0000", 2024-08-17T18:22:05.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9900000000000001e-07 2024-08-17T18:22:05.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:22:05.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638013+0000", 2024-08-17T18:22:05.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6280000000000001e-06 2024-08-17T18:22:05.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:22:05.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638021+0000", 2024-08-17T18:22:05.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1350000000000003e-06 2024-08-17T18:22:05.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:22:05.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638026+0000", 2024-08-17T18:22:05.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3179999999999997e-06 2024-08-17T18:22:05.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638071+0000", 2024-08-17T18:22:05.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4708e-05 2024-08-17T18:22:05.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:22:05.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638075+0000", 2024-08-17T18:22:05.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.741e-06 2024-08-17T18:22:05.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640205+0000", 2024-08-17T18:22:05.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00212978 2024-08-17T18:22:05.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:22:05.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640212+0000", 2024-08-17T18:22:05.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0060000000000003e-06 2024-08-17T18:22:05.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.025333+0000", 2024-08-17T18:22:05.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3851207940000001 2024-08-17T18:22:05.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:22:05.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.025405+0000", 2024-08-17T18:22:05.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1772000000000003e-05 2024-08-17T18:22:05.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:22:05.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.025543+0000", 2024-08-17T18:22:05.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013786199999999999 2024-08-17T18:22:05.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:22:05.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.026707+0000", 2024-08-17T18:22:05.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001164371 2024-08-17T18:22:05.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.026758+0000", 2024-08-17T18:22:05.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1034e-05 2024-08-17T18:22:05.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.027663+0000", 2024-08-17T18:22:05.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00090472399999999998 2024-08-17T18:22:05.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.074299+0000", 2024-08-17T18:22:05.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.046636098000000001 2024-08-17T18:22:05.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.094154+0000", 2024-08-17T18:22:05.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019855695999999999 2024-08-17T18:22:05.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:22:05.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.094262+0000", 2024-08-17T18:22:05.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010785099999999999 2024-08-17T18:22:05.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:22:05.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.094388+0000", 2024-08-17T18:22:05.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012611299999999999 2024-08-17T18:22:05.692 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.692 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:22:05.693 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.693 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207790 56.10 56:0d05e484:::benchmark_data_smithi136_356456_object81086:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:22:05.693 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.638033+0000", 2024-08-17T18:22:05.693 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 565.98725933100002, 2024-08-17T18:22:05.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4570861230000001, 2024-08-17T18:22:05.693 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:22:05.693 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:22:05.693 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:05.693 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:22:05.693 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:22:05.694 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:22:05.694 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207790 2024-08-17T18:22:05.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.694 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:22:05.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:22:05.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638033+0000", 2024-08-17T18:22:05.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:22:05.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638033+0000", 2024-08-17T18:22:05.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:22:05.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638033+0000", 2024-08-17T18:22:05.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9099999999999999e-07 2024-08-17T18:22:05.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:22:05.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638035+0000", 2024-08-17T18:22:05.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8759999999999999e-06 2024-08-17T18:22:05.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:22:05.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638041+0000", 2024-08-17T18:22:05.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1099999999999999e-06 2024-08-17T18:22:05.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:22:05.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638046+0000", 2024-08-17T18:22:05.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8890000000000001e-06 2024-08-17T18:22:05.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638143+0000", 2024-08-17T18:22:05.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7078999999999997e-05 2024-08-17T18:22:05.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:22:05.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638148+0000", 2024-08-17T18:22:05.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6890000000000003e-06 2024-08-17T18:22:05.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640249+0000", 2024-08-17T18:22:05.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021008480000000002 2024-08-17T18:22:05.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:22:05.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640256+0000", 2024-08-17T18:22:05.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1690000000000003e-06 2024-08-17T18:22:05.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.026793+0000", 2024-08-17T18:22:05.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3865375549999999 2024-08-17T18:22:05.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:22:05.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.026868+0000", 2024-08-17T18:22:05.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4140999999999994e-05 2024-08-17T18:22:05.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:22:05.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.027003+0000", 2024-08-17T18:22:05.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000135418 2024-08-17T18:22:05.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.029543+0000", 2024-08-17T18:22:05.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025404120000000001 2024-08-17T18:22:05.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:22:05.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.029552+0000", 2024-08-17T18:22:05.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2099999999999993e-06 2024-08-17T18:22:05.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.029576+0000", 2024-08-17T18:22:05.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4085999999999999e-05 2024-08-17T18:22:05.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.075573+0000", 2024-08-17T18:22:05.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.045997008999999998 2024-08-17T18:22:05.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.094908+0000", 2024-08-17T18:22:05.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019335082 2024-08-17T18:22:05.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:22:05.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.094980+0000", 2024-08-17T18:22:05.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1841000000000006e-05 2024-08-17T18:22:05.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:22:05.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.095119+0000", 2024-08-17T18:22:05.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013931800000000001 2024-08-17T18:22:05.701 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.702 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:22:05.702 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.702 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207793 56.10 56:0f777e78:::benchmark_data_smithi136_356456_object81089:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:22:05.702 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.638105+0000", 2024-08-17T18:22:05.702 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 565.98718740499999, 2024-08-17T18:22:05.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4573933100000001, 2024-08-17T18:22:05.702 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:22:05.702 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:22:05.702 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:05.702 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:22:05.702 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:22:05.703 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:22:05.703 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207793 2024-08-17T18:22:05.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.703 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:22:05.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:22:05.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638105+0000", 2024-08-17T18:22:05.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:22:05.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638105+0000", 2024-08-17T18:22:05.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:22:05.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638105+0000", 2024-08-17T18:22:05.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6400000000000002e-07 2024-08-17T18:22:05.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:22:05.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638109+0000", 2024-08-17T18:22:05.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.653e-06 2024-08-17T18:22:05.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:22:05.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638130+0000", 2024-08-17T18:22:05.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0471000000000001e-05 2024-08-17T18:22:05.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:22:05.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638136+0000", 2024-08-17T18:22:05.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7880000000000001e-06 2024-08-17T18:22:05.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638183+0000", 2024-08-17T18:22:05.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6270000000000003e-05 2024-08-17T18:22:05.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:22:05.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638188+0000", 2024-08-17T18:22:05.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7019999999999998e-06 2024-08-17T18:22:05.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640287+0000", 2024-08-17T18:22:05.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020988669999999999 2024-08-17T18:22:05.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:22:05.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640291+0000", 2024-08-17T18:22:05.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1949999999999996e-06 2024-08-17T18:22:05.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.027702+0000", 2024-08-17T18:22:05.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.387410799 2024-08-17T18:22:05.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:22:05.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.027786+0000", 2024-08-17T18:22:05.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3397000000000001e-05 2024-08-17T18:22:05.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:22:05.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.027950+0000", 2024-08-17T18:22:05.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016487300000000001 2024-08-17T18:22:05.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.029610+0000", 2024-08-17T18:22:05.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016590610000000001 2024-08-17T18:22:05.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:22:05.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.029626+0000", 2024-08-17T18:22:05.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6116e-05 2024-08-17T18:22:05.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.029668+0000", 2024-08-17T18:22:05.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2135999999999998e-05 2024-08-17T18:22:05.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.075602+0000", 2024-08-17T18:22:05.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.045934540000000003 2024-08-17T18:22:05.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.095322+0000", 2024-08-17T18:22:05.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019719235000000002 2024-08-17T18:22:05.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:22:05.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.095384+0000", 2024-08-17T18:22:05.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2131999999999994e-05 2024-08-17T18:22:05.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:22:05.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.095498+0000", 2024-08-17T18:22:05.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114411 2024-08-17T18:22:05.710 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.710 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:22:05.710 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.710 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:05.712 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207810 56.10 56:0ea6e34e:::benchmark_data_smithi136_356456_object81106:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:22:05.712 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.638147+0000", 2024-08-17T18:22:05.712 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 565.98714471999995, 2024-08-17T18:22:05.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4576238779999999, 2024-08-17T18:22:05.713 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:22:05.713 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:22:05.713 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:05.713 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:22:05.713 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:22:05.713 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:22:05.713 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207810 2024-08-17T18:22:05.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.713 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:22:05.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:22:05.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638147+0000", 2024-08-17T18:22:05.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:22:05.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638147+0000", 2024-08-17T18:22:05.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:22:05.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638148+0000", 2024-08-17T18:22:05.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2e-07 2024-08-17T18:22:05.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:22:05.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638150+0000", 2024-08-17T18:22:05.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2290000000000002e-06 2024-08-17T18:22:05.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:22:05.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638156+0000", 2024-08-17T18:22:05.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9410000000000004e-06 2024-08-17T18:22:05.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:22:05.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638160+0000", 2024-08-17T18:22:05.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1930000000000003e-06 2024-08-17T18:22:05.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638213+0000", 2024-08-17T18:22:05.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2367999999999997e-05 2024-08-17T18:22:05.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:22:05.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638218+0000", 2024-08-17T18:22:05.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9899999999999997e-06 2024-08-17T18:22:05.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640323+0000", 2024-08-17T18:22:05.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002105539 2024-08-17T18:22:05.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:22:05.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640330+0000", 2024-08-17T18:22:05.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0539999999999997e-06 2024-08-17T18:22:05.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.028618+0000", 2024-08-17T18:22:05.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.38828763 2024-08-17T18:22:05.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:22:05.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.028740+0000", 2024-08-17T18:22:05.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000122014 2024-08-17T18:22:05.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:22:05.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.028870+0000", 2024-08-17T18:22:05.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000130562 2024-08-17T18:22:05.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:22:05.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.030547+0000", 2024-08-17T18:22:05.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001676564 2024-08-17T18:22:05.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.031449+0000", 2024-08-17T18:22:05.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00090164300000000004 2024-08-17T18:22:05.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.031488+0000", 2024-08-17T18:22:05.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9415999999999997e-05 2024-08-17T18:22:05.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.076959+0000", 2024-08-17T18:22:05.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.045470857000000003 2024-08-17T18:22:05.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.095546+0000", 2024-08-17T18:22:05.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018586928999999999 2024-08-17T18:22:05.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:22:05.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.095612+0000", 2024-08-17T18:22:05.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6029000000000001e-05 2024-08-17T18:22:05.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:22:05.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.095771+0000", 2024-08-17T18:22:05.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0001594 2024-08-17T18:22:05.722 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.722 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:22:05.722 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.722 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207868 56.10 56:0f591836:::benchmark_data_smithi136_356456_object81164:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:22:05.722 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.638177+0000", 2024-08-17T18:22:05.722 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 565.98711521999996, 2024-08-17T18:22:05.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4578162130000001, 2024-08-17T18:22:05.723 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:22:05.723 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:22:05.723 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:05.723 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:22:05.723 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:22:05.723 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:22:05.723 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207868 2024-08-17T18:22:05.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.723 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:22:05.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:22:05.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638177+0000", 2024-08-17T18:22:05.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:22:05.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638177+0000", 2024-08-17T18:22:05.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:22:05.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638177+0000", 2024-08-17T18:22:05.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1600000000000002e-07 2024-08-17T18:22:05.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:22:05.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638179+0000", 2024-08-17T18:22:05.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9870000000000002e-06 2024-08-17T18:22:05.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:22:05.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638186+0000", 2024-08-17T18:22:05.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0679999999999999e-06 2024-08-17T18:22:05.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:22:05.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638192+0000", 2024-08-17T18:22:05.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2159999999999997e-06 2024-08-17T18:22:05.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638244+0000", 2024-08-17T18:22:05.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2305000000000002e-05 2024-08-17T18:22:05.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:22:05.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638249+0000", 2024-08-17T18:22:05.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3630000000000002e-06 2024-08-17T18:22:05.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640355+0000", 2024-08-17T18:22:05.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002105928 2024-08-17T18:22:05.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:22:05.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640359+0000", 2024-08-17T18:22:05.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.495e-06 2024-08-17T18:22:05.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.029712+0000", 2024-08-17T18:22:05.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3893531050000001 2024-08-17T18:22:05.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:22:05.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.029797+0000", 2024-08-17T18:22:05.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5116000000000003e-05 2024-08-17T18:22:05.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:22:05.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.029932+0000", 2024-08-17T18:22:05.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013527600000000001 2024-08-17T18:22:05.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.031527+0000", 2024-08-17T18:22:05.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015951819999999999 2024-08-17T18:22:05.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:22:05.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.031537+0000", 2024-08-17T18:22:05.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1560000000000001e-06 2024-08-17T18:22:05.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.031564+0000", 2024-08-17T18:22:05.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7455999999999999e-05 2024-08-17T18:22:05.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.076994+0000", 2024-08-17T18:22:05.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.045429888000000002 2024-08-17T18:22:05.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.095815+0000", 2024-08-17T18:22:05.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018821457999999999 2024-08-17T18:22:05.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:22:05.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.095878+0000", 2024-08-17T18:22:05.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2148000000000002e-05 2024-08-17T18:22:05.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:22:05.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.095993+0000", 2024-08-17T18:22:05.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011565000000000001 2024-08-17T18:22:05.732 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.732 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:22:05.732 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.732 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207897 56.10 56:0cb54b06:::benchmark_data_smithi136_356456_object81193:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:22:05.732 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.638208+0000", 2024-08-17T18:22:05.732 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 565.98708385999998, 2024-08-17T18:22:05.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.458027999, 2024-08-17T18:22:05.733 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:22:05.733 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:22:05.733 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:05.733 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:22:05.733 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:22:05.733 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:22:05.733 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207897 2024-08-17T18:22:05.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.733 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:22:05.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:22:05.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638208+0000", 2024-08-17T18:22:05.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:22:05.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638208+0000", 2024-08-17T18:22:05.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:22:05.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638209+0000", 2024-08-17T18:22:05.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3000000000000001e-07 2024-08-17T18:22:05.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:22:05.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638211+0000", 2024-08-17T18:22:05.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.902e-06 2024-08-17T18:22:05.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:22:05.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638218+0000", 2024-08-17T18:22:05.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1539999999999996e-06 2024-08-17T18:22:05.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:22:05.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638223+0000", 2024-08-17T18:22:05.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4009999999999996e-06 2024-08-17T18:22:05.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638274+0000", 2024-08-17T18:22:05.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0943000000000003e-05 2024-08-17T18:22:05.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:22:05.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.638279+0000", 2024-08-17T18:22:05.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8579999999999999e-06 2024-08-17T18:22:05.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640382+0000", 2024-08-17T18:22:05.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021030879999999999 2024-08-17T18:22:05.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:22:05.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.640389+0000", 2024-08-17T18:22:05.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8199999999999999e-06 2024-08-17T18:22:05.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.030583+0000", 2024-08-17T18:22:05.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.390193805 2024-08-17T18:22:05.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:22:05.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.030654+0000", 2024-08-17T18:22:05.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1778000000000003e-05 2024-08-17T18:22:05.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:22:05.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.030800+0000", 2024-08-17T18:22:05.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000145296 2024-08-17T18:22:05.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:22:05.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.032454+0000", 2024-08-17T18:22:05.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001653821 2024-08-17T18:22:05.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.033404+0000", 2024-08-17T18:22:05.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00095007900000000005 2024-08-17T18:22:05.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.033438+0000", 2024-08-17T18:22:05.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4153000000000003e-05 2024-08-17T18:22:05.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.079229+0000", 2024-08-17T18:22:05.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.045791559000000003 2024-08-17T18:22:05.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.096073+0000", 2024-08-17T18:22:05.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016844066000000001 2024-08-17T18:22:05.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:22:05.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.096126+0000", 2024-08-17T18:22:05.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2318000000000003e-05 2024-08-17T18:22:05.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:22:05.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.096236+0000", 2024-08-17T18:22:05.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110528 2024-08-17T18:22:05.741 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.741 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:22:05.741 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.741 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207920 56.10 56:0b09234c:::benchmark_data_smithi136_356456_object81216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:22:05.742 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.658933+0000", 2024-08-17T18:22:05.742 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 565.96635926700003, 2024-08-17T18:22:05.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4375237670000001, 2024-08-17T18:22:05.742 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:22:05.742 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:22:05.742 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:05.742 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:22:05.742 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:22:05.742 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:22:05.742 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207920 2024-08-17T18:22:05.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.742 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:22:05.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:22:05.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.658933+0000", 2024-08-17T18:22:05.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:22:05.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.658933+0000", 2024-08-17T18:22:05.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:22:05.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.658934+0000", 2024-08-17T18:22:05.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6099999999999999e-07 2024-08-17T18:22:05.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:22:05.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.658938+0000", 2024-08-17T18:22:05.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2780000000000004e-06 2024-08-17T18:22:05.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:22:05.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.658950+0000", 2024-08-17T18:22:05.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2018000000000001e-05 2024-08-17T18:22:05.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:22:05.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.658957+0000", 2024-08-17T18:22:05.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1779999999999997e-06 2024-08-17T18:22:05.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.659004+0000", 2024-08-17T18:22:05.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6539999999999998e-05 2024-08-17T18:22:05.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:22:05.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.659011+0000", 2024-08-17T18:22:05.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8419999999999999e-06 2024-08-17T18:22:05.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.031602+0000", 2024-08-17T18:22:05.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.372591723 2024-08-17T18:22:05.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:22:05.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.031674+0000", 2024-08-17T18:22:05.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1555000000000002e-05 2024-08-17T18:22:05.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:22:05.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.031810+0000", 2024-08-17T18:22:05.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013635000000000001 2024-08-17T18:22:05.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.033469+0000", 2024-08-17T18:22:05.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001658318 2024-08-17T18:22:05.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:22:05.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.033476+0000", 2024-08-17T18:22:05.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4460000000000004e-06 2024-08-17T18:22:05.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.033508+0000", 2024-08-17T18:22:05.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1977999999999998e-05 2024-08-17T18:22:05.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.079279+0000", 2024-08-17T18:22:05.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.045770862000000002 2024-08-17T18:22:05.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.096279+0000", 2024-08-17T18:22:05.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017000576999999999 2024-08-17T18:22:05.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:22:05.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.096334+0000", 2024-08-17T18:22:05.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4858999999999998e-05 2024-08-17T18:22:05.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:22:05.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.096457+0000", 2024-08-17T18:22:05.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012238200000000001 2024-08-17T18:22:05.750 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.750 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:22:05.750 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.750 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46025.0:207937 56.10 56:0c91c958:::benchmark_data_smithi136_356456_object81233:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1569)", 2024-08-17T18:22:05.750 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:12:39.681079+0000", 2024-08-17T18:22:05.750 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 565.94421284700002, 2024-08-17T18:22:05.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4157404689999999, 2024-08-17T18:22:05.750 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:22:05.750 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:22:05.750 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:05.750 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:22:05.751 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46025", 2024-08-17T18:22:05.751 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2257115034", 2024-08-17T18:22:05.751 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 207937 2024-08-17T18:22:05.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.751 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:22:05.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:22:05.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.681079+0000", 2024-08-17T18:22:05.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:22:05.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.681079+0000", 2024-08-17T18:22:05.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:22:05.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.681081+0000", 2024-08-17T18:22:05.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.268e-06 2024-08-17T18:22:05.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:22:05.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.681085+0000", 2024-08-17T18:22:05.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1400000000000002e-06 2024-08-17T18:22:05.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:22:05.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.681096+0000", 2024-08-17T18:22:05.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0818999999999999e-05 2024-08-17T18:22:05.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:22:05.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.681101+0000", 2024-08-17T18:22:05.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2619999999999998e-06 2024-08-17T18:22:05.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.681161+0000", 2024-08-17T18:22:05.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9818000000000002e-05 2024-08-17T18:22:05.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:22:05.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:39.681171+0000", 2024-08-17T18:22:05.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0093e-05 2024-08-17T18:22:05.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.032475+0000", 2024-08-17T18:22:05.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3513044779999999 2024-08-17T18:22:05.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:22:05.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.032553+0000", 2024-08-17T18:22:05.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7330999999999995e-05 2024-08-17T18:22:05.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4,5,6", 2024-08-17T18:22:05.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.032721+0000", 2024-08-17T18:22:05.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016832500000000001 2024-08-17T18:22:05.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:22:05.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.034404+0000", 2024-08-17T18:22:05.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001683466 2024-08-17T18:22:05.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.035385+0000", 2024-08-17T18:22:05.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00098110499999999991 2024-08-17T18:22:05.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.035421+0000", 2024-08-17T18:22:05.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5408999999999999e-05 2024-08-17T18:22:05.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.079323+0000", 2024-08-17T18:22:05.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.043902651000000001 2024-08-17T18:22:05.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.096646+0000", 2024-08-17T18:22:05.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017322758000000001 2024-08-17T18:22:05.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:22:05.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.096707+0000", 2024-08-17T18:22:05.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0699999999999998e-05 2024-08-17T18:22:05.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:22:05.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:12:41.096820+0000", 2024-08-17T18:22:05.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112846 2024-08-17T18:22:05.758 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.759 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:22:05.759 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.759 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.46277.0:214474 57.d e1685/1666 57:b233cfa3:::benchmark_data_smithi136_389235_object98552:head v 1685'13390, mlcod=1685'13388)", 2024-08-17T18:22:05.759 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:19:51.844917+0000", 2024-08-17T18:22:05.759 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 133.78037546600001, 2024-08-17T18:22:05.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15288263699999999, 2024-08-17T18:22:05.759 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:22:05.759 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:22:05.759 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:05.759 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:22:05.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:22:05.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.844917+0000", 2024-08-17T18:22:05.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:22:05.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.844917+0000", 2024-08-17T18:22:05.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:22:05.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.844918+0000", 2024-08-17T18:22:05.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1179999999999999e-06 2024-08-17T18:22:05.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:22:05.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.844926+0000", 2024-08-17T18:22:05.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9699999999999999e-06 2024-08-17T18:22:05.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:22:05.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.844938+0000", 2024-08-17T18:22:05.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2435e-05 2024-08-17T18:22:05.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:22:05.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.844946+0000", 2024-08-17T18:22:05.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1499999999999999e-06 2024-08-17T18:22:05.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.845006+0000", 2024-08-17T18:22:05.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9478000000000002e-05 2024-08-17T18:22:05.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:22:05.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.845029+0000", 2024-08-17T18:22:05.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3237000000000001e-05 2024-08-17T18:22:05.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:22:05.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.997760+0000", 2024-08-17T18:22:05.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15273125100000001 2024-08-17T18:22:05.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:22:05.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.997799+0000", 2024-08-17T18:22:05.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8998000000000002e-05 2024-08-17T18:22:05.763 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.764 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:22:05.764 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.764 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.46277.0:214462 57.15 e1685/1668 57:a902ba35:::benchmark_data_smithi136_389235_object98540:head v 1685'9172, mlcod=1625'5605)", 2024-08-17T18:22:05.764 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:19:51.845033+0000", 2024-08-17T18:22:05.764 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 133.780258777, 2024-08-17T18:22:05.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15945930799999999, 2024-08-17T18:22:05.764 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:22:05.764 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:22:05.764 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:05.764 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:22:05.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:22:05.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.845033+0000", 2024-08-17T18:22:05.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:22:05.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.845033+0000", 2024-08-17T18:22:05.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:22:05.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.845034+0000", 2024-08-17T18:22:05.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.37e-07 2024-08-17T18:22:05.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:22:05.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.845040+0000", 2024-08-17T18:22:05.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9159999999999998e-06 2024-08-17T18:22:05.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:22:05.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.845050+0000", 2024-08-17T18:22:05.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0387999999999999e-05 2024-08-17T18:22:05.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:22:05.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.845057+0000", 2024-08-17T18:22:05.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8589999999999999e-06 2024-08-17T18:22:05.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.990354+0000", 2024-08-17T18:22:05.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14529714699999999 2024-08-17T18:22:05.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:22:05.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.990395+0000", 2024-08-17T18:22:05.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0453e-05 2024-08-17T18:22:05.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:22:05.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:52.002335+0000", 2024-08-17T18:22:05.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011940638 2024-08-17T18:22:05.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:22:05.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:52.004493+0000", 2024-08-17T18:22:05.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021572700000000002 2024-08-17T18:22:05.768 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.768 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:22:05.768 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.769 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.46277.0:214464 57.3 e1685/1668 57:c624003f:::benchmark_data_smithi136_389235_object98542:head v 1685'9140, mlcod=1625'5502)", 2024-08-17T18:22:05.769 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:19:51.846697+0000", 2024-08-17T18:22:05.769 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 133.778595134, 2024-08-17T18:22:05.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.149338639, 2024-08-17T18:22:05.769 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:22:05.769 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:22:05.769 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:05.769 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:22:05.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:22:05.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.846697+0000", 2024-08-17T18:22:05.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:22:05.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.846697+0000", 2024-08-17T18:22:05.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:22:05.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.846698+0000", 2024-08-17T18:22:05.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6500000000000008e-07 2024-08-17T18:22:05.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:22:05.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.846708+0000", 2024-08-17T18:22:05.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0006e-05 2024-08-17T18:22:05.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:22:05.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.990341+0000", 2024-08-17T18:22:05.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14363261899999999 2024-08-17T18:22:05.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:22:05.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.990358+0000", 2024-08-17T18:22:05.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7748e-05 2024-08-17T18:22:05.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.990461+0000", 2024-08-17T18:22:05.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102358 2024-08-17T18:22:05.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:22:05.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.992194+0000", 2024-08-17T18:22:05.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017329380000000001 2024-08-17T18:22:05.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:22:05.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.995957+0000", 2024-08-17T18:22:05.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003763569 2024-08-17T18:22:05.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:22:05.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.996036+0000", 2024-08-17T18:22:05.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8436000000000006e-05 2024-08-17T18:22:05.773 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.773 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:22:05.773 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.773 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.46277.0:214477 57.0 e1685/1668 57:048f9909:::benchmark_data_smithi136_389235_object98555:head v 1685'9177, mlcod=1685'9175)", 2024-08-17T18:22:05.773 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:19:51.848532+0000", 2024-08-17T18:22:05.773 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 133.77676006900001, 2024-08-17T18:22:05.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14435190000000001, 2024-08-17T18:22:05.773 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:22:05.774 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:22:05.774 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:05.774 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:22:05.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:22:05.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.848532+0000", 2024-08-17T18:22:05.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:22:05.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.848532+0000", 2024-08-17T18:22:05.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:22:05.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.848533+0000", 2024-08-17T18:22:05.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0720000000000001e-06 2024-08-17T18:22:05.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:22:05.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.848543+0000", 2024-08-17T18:22:05.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8320000000000001e-06 2024-08-17T18:22:05.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:22:05.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.990343+0000", 2024-08-17T18:22:05.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14179973000000001 2024-08-17T18:22:05.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:22:05.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.990358+0000", 2024-08-17T18:22:05.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5636000000000001e-05 2024-08-17T18:22:05.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.990438+0000", 2024-08-17T18:22:05.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9912000000000003e-05 2024-08-17T18:22:05.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:22:05.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.990461+0000", 2024-08-17T18:22:05.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2405999999999999e-05 2024-08-17T18:22:05.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:22:05.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.992841+0000", 2024-08-17T18:22:05.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023802039999999999 2024-08-17T18:22:05.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:22:05.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:19:51.992884+0000", 2024-08-17T18:22:05.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3108000000000001e-05 2024-08-17T18:22:05.778 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.778 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:22:05.778 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.778 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:69947 58.a 58:5888783f:::benchmark_data_smithi136_442124_object69946:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1711)", 2024-08-17T18:22:05.778 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:21:40.739143+0000", 2024-08-17T18:22:05.778 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.886149654, 2024-08-17T18:22:05.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0483350549999999, 2024-08-17T18:22:05.779 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:22:05.779 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:22:05.779 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:22:05.779 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:22:05.779 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:22:05.779 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:22:05.779 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69947 2024-08-17T18:22:05.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.779 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:22:05.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:22:05.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.739143+0000", 2024-08-17T18:22:05.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:22:05.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.739143+0000", 2024-08-17T18:22:05.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:22:05.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.739143+0000", 2024-08-17T18:22:05.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5200000000000002e-07 2024-08-17T18:22:05.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:22:05.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.739163+0000", 2024-08-17T18:22:05.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9667999999999999e-05 2024-08-17T18:22:05.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:22:05.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.739170+0000", 2024-08-17T18:22:05.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9430000000000003e-06 2024-08-17T18:22:05.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:22:05.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.739174+0000", 2024-08-17T18:22:05.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5569999999999997e-06 2024-08-17T18:22:05.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.746090+0000", 2024-08-17T18:22:05.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0069157979999999999 2024-08-17T18:22:05.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:22:05.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.746226+0000", 2024-08-17T18:22:05.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013563099999999999 2024-08-17T18:22:05.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6,7", 2024-08-17T18:22:05.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.746355+0000", 2024-08-17T18:22:05.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012918700000000001 2024-08-17T18:22:05.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.750541+0000", 2024-08-17T18:22:05.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.004186431 2024-08-17T18:22:05.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.753731+0000", 2024-08-17T18:22:05.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031901590000000001 2024-08-17T18:22:05.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766078+0000", 2024-08-17T18:22:05.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012346677 2024-08-17T18:22:05.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:22:05.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766089+0000", 2024-08-17T18:22:05.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0541e-05 2024-08-17T18:22:05.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.787406+0000", 2024-08-17T18:22:05.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0213175569999999 2024-08-17T18:22:05.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:22:05.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.787478+0000", 2024-08-17T18:22:05.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1453999999999997e-05 2024-08-17T18:22:05.786 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.786 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:22:05.787 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.787 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:69950 58.a 58:58412d0b:::benchmark_data_smithi136_442124_object69949:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1711)", 2024-08-17T18:22:05.787 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:21:40.741265+0000", 2024-08-17T18:22:05.787 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.884027498000002, 2024-08-17T18:22:05.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.046524625, 2024-08-17T18:22:05.787 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:22:05.787 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:22:05.787 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:22:05.787 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:22:05.787 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:22:05.788 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:22:05.788 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69950 2024-08-17T18:22:05.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.788 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:22:05.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:22:05.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.741265+0000", 2024-08-17T18:22:05.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:22:05.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.741265+0000", 2024-08-17T18:22:05.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:22:05.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.741266+0000", 2024-08-17T18:22:05.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0500000000000002e-07 2024-08-17T18:22:05.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:22:05.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.741286+0000", 2024-08-17T18:22:05.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0758e-05 2024-08-17T18:22:05.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:22:05.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.741298+0000", 2024-08-17T18:22:05.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1469e-05 2024-08-17T18:22:05.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:22:05.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.741305+0000", 2024-08-17T18:22:05.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6070000000000002e-06 2024-08-17T18:22:05.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.750568+0000", 2024-08-17T18:22:05.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0092621260000000007 2024-08-17T18:22:05.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:22:05.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.750716+0000", 2024-08-17T18:22:05.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014864300000000001 2024-08-17T18:22:05.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6,7", 2024-08-17T18:22:05.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.750845+0000", 2024-08-17T18:22:05.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000128393 2024-08-17T18:22:05.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:05.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.754840+0000", 2024-08-17T18:22:05.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003995452 2024-08-17T18:22:05.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.769621+0000", 2024-08-17T18:22:05.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014781424 2024-08-17T18:22:05.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:22:05.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.769631+0000", 2024-08-17T18:22:05.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9280000000000005e-06 2024-08-17T18:22:05.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.787712+0000", 2024-08-17T18:22:05.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.018080248 2024-08-17T18:22:05.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:22:05.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.787789+0000", 2024-08-17T18:22:05.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7671999999999998e-05 2024-08-17T18:22:05.797 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.797 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:22:05.797 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.797 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:69953 58.a 58:57453b76:::benchmark_data_smithi136_442124_object69952:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1711)", 2024-08-17T18:22:05.797 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:21:40.744934+0000", 2024-08-17T18:22:05.797 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.880358650000002, 2024-08-17T18:22:05.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0462948329999999, 2024-08-17T18:22:05.797 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:22:05.797 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:22:05.798 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:22:05.798 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:22:05.798 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:22:05.798 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:22:05.798 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69953 2024-08-17T18:22:05.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.798 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:22:05.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:22:05.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.744934+0000", 2024-08-17T18:22:05.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:22:05.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.744934+0000", 2024-08-17T18:22:05.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:22:05.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:22:05.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.744934+0000", 2024-08-17T18:22:05.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2799999999999996e-07 2024-08-17T18:22:05.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:22:05.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.744956+0000", 2024-08-17T18:22:05.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1590000000000002e-05 2024-08-17T18:22:05.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:22:05.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.744963+0000", 2024-08-17T18:22:05.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.888e-06 2024-08-17T18:22:05.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:22:05.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.744968+0000", 2024-08-17T18:22:05.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4729999999999999e-06 2024-08-17T18:22:05.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.757200+0000", 2024-08-17T18:22:05.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012231801 2024-08-17T18:22:05.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:22:05.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.757652+0000", 2024-08-17T18:22:05.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00045206400000000001 2024-08-17T18:22:05.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6,7", 2024-08-17T18:22:05.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.757813+0000", 2024-08-17T18:22:05.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016159400000000001 2024-08-17T18:22:05.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.769660+0000", 2024-08-17T18:22:05.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011846262 2024-08-17T18:22:05.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:22:05.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.769667+0000", 2024-08-17T18:22:05.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9280000000000004e-06 2024-08-17T18:22:05.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.787828+0000", 2024-08-17T18:22:05.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.01816095 2024-08-17T18:22:05.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:22:05.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.788566+0000", 2024-08-17T18:22:05.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00073831500000000002 2024-08-17T18:22:05.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:22:05.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.791125+0000", 2024-08-17T18:22:05.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025590360000000002 2024-08-17T18:22:05.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:22:05.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:22:05.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:22:05.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.791228+0000", 2024-08-17T18:22:05.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103404 2024-08-17T18:22:05.803 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.803 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:22:05.804 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.804 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:22:05.804 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:22:05.804 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:22:05.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:05.805 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:05.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:05.884 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:05 smithi136 ceph-mon[109861]: pgmap v3353: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 34243/273849 objects degraded (12.504%) 2024-08-17T18:22:05.884 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:05 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 34243/273849 objects degraded (12.504%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:05.884 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:05 smithi136 ceph-mon[109861]: osdmap e1717: 8 total, 7 up, 8 in 2024-08-17T18:22:05.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:05 smithi136 ceph-mon[115078]: pgmap v3353: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 34243/273849 objects degraded (12.504%) 2024-08-17T18:22:05.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:05 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 34243/273849 objects degraded (12.504%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:05.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:05 smithi136 ceph-mon[115078]: osdmap e1717: 8 total, 7 up, 8 in 2024-08-17T18:22:05.906 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:05.911 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:05 smithi165 ceph-mon[105517]: pgmap v3353: 99 pgs: 12 active+undersized+degraded, 20 active+undersized, 67 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 34243/273849 objects degraded (12.504%) 2024-08-17T18:22:05.911 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:05 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 34243/273849 objects degraded (12.504%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:05.911 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:05 smithi165 ceph-mon[105517]: osdmap e1717: 8 total, 7 up, 8 in 2024-08-17T18:22:05.944 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:05.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:06.046 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:06.086 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:06.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:06.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:06.188 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:06.215 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:06.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:06.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:06.220 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 66 16 93654 93638 88.6562 12.1875 0.0190305 0.0112742 2024-08-17T18:22:06.316 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:06.337 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:06.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:06.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:06.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:06.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:06.469 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:06.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:06.571 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:06.600 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:06.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:06.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:06.702 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:06.728 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:06.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:06.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:06.830 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:06.859 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:06.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:06.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:06.961 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:07.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:07.008 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:07.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:07.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:07.136 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:07.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:07.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:07.221 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 67 16 94417 94401 88.0447 47.6875 0.00803613 0.0113524 2024-08-17T18:22:07.238 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:07.266 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:07.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:07.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:07.368 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:07.392 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:07.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:07.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:07.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:07 smithi136 ceph-mon[109861]: 58.e starting backfill to osd.7 from (0'0,0'0] MAX to 1714'5788 2024-08-17T18:22:07.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:07 smithi136 ceph-mon[109861]: 58.d starting backfill to osd.4 from (0'0,0'0] MAX to 1714'5761 2024-08-17T18:22:07.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:22:07.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:22:07.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:07 smithi136 ceph-mon[109861]: pgmap v3356: 99 pgs: 1 active+undersized+remapped, 5 remapped+peering, 12 active+undersized+degraded, 15 active+undersized, 66 active+clean; 5.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 59 MiB/s wr, 945 op/s; 34770/277305 objects degraded (12.539%) 2024-08-17T18:22:07.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:07 smithi136 ceph-mon[109861]: osdmap e1718: 8 total, 7 up, 8 in 2024-08-17T18:22:07.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:07 smithi136 ceph-mon[115078]: 58.e starting backfill to osd.7 from (0'0,0'0] MAX to 1714'5788 2024-08-17T18:22:07.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:07 smithi136 ceph-mon[115078]: 58.d starting backfill to osd.4 from (0'0,0'0] MAX to 1714'5761 2024-08-17T18:22:07.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:22:07.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:22:07.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:07 smithi136 ceph-mon[115078]: pgmap v3356: 99 pgs: 1 active+undersized+remapped, 5 remapped+peering, 12 active+undersized+degraded, 15 active+undersized, 66 active+clean; 5.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 59 MiB/s wr, 945 op/s; 34770/277305 objects degraded (12.539%) 2024-08-17T18:22:07.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:07 smithi136 ceph-mon[115078]: osdmap e1718: 8 total, 7 up, 8 in 2024-08-17T18:22:07.494 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:07.510 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:07 smithi165 ceph-mon[105517]: 58.e starting backfill to osd.7 from (0'0,0'0] MAX to 1714'5788 2024-08-17T18:22:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:07 smithi165 ceph-mon[105517]: 58.d starting backfill to osd.4 from (0'0,0'0] MAX to 1714'5761 2024-08-17T18:22:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:22:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:22:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:07 smithi165 ceph-mon[105517]: pgmap v3356: 99 pgs: 1 active+undersized+remapped, 5 remapped+peering, 12 active+undersized+degraded, 15 active+undersized, 66 active+clean; 5.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 59 MiB/s wr, 945 op/s; 34770/277305 objects degraded (12.539%) 2024-08-17T18:22:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:07 smithi165 ceph-mon[105517]: osdmap e1718: 8 total, 7 up, 8 in 2024-08-17T18:22:07.535 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:07.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:07.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:07.637 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:07.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:07.673 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:07.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:07.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:07.801 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:07.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:07.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:07.903 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:07.929 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:07.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:07.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:07.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:22:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:22:07] "GET /metrics HTTP/1.1" 200 36752 "" "Prometheus/2.51.0" 2024-08-17T18:22:08.031 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:08.057 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:08.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:08.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:08.160 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:08.185 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:08.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:08.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:08.221 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 68 16 95322 95306 87.5817 56.5625 0.00798304 0.0114128 2024-08-17T18:22:08.287 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:08.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:08.320 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:08.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:08.421 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:08.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:08.446 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:08.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:08.547 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:08.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:08.583 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:08.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:08.684 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:08.709 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:08.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:08.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:08.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:08.837 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:08.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:08.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:08.940 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:08.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:08 smithi136 ceph-mon[109861]: osdmap e1719: 8 total, 7 up, 8 in 2024-08-17T18:22:08.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:08 smithi136 ceph-mon[115078]: osdmap e1719: 8 total, 7 up, 8 in 2024-08-17T18:22:08.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:08.977 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:08.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:09.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:08 smithi165 ceph-mon[105517]: osdmap e1719: 8 total, 7 up, 8 in 2024-08-17T18:22:09.078 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:09.101 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:09.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:09.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:09.203 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:09.222 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 69 16 96212 96196 87.1175 55.625 0.0177326 0.011472 2024-08-17T18:22:09.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:09.250 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:09.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:09.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:09.377 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:09.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:09.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:09.479 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:09.516 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:09.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:09.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:09.619 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:09.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:09.646 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:09.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:09.690 INFO:tasks.thrashosds.thrasher:in_osds: [2, 4, 6, 0, 5, 7, 3, 1] out_osds: [] dead_osds: [6] live_osds: [0, 1, 3, 4, 5, 7, 2] 2024-08-17T18:22:09.690 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:22:09.690 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-08-17T18:22:09.690 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-17T18:22:09.690 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-17T18:22:09.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-08-17T18:22:09.747 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:09.772 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:09.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:09.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:09.874 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:09.906 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:09.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:09.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:09 smithi136 ceph-mon[109861]: pgmap v3359: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 6 remapped+peering, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 5.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 46 MiB/s wr, 738 op/s; 35264/282711 objects degraded (12.474%); 17372/282711 objects misplaced (6.145%); 5.4 MiB/s, 86 objects/s recovering 2024-08-17T18:22:09.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:09 smithi136 ceph-mon[115078]: pgmap v3359: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 6 remapped+peering, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 5.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 46 MiB/s wr, 738 op/s; 35264/282711 objects degraded (12.474%); 17372/282711 objects misplaced (6.145%); 5.4 MiB/s, 86 objects/s recovering 2024-08-17T18:22:09.971 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:22:10.008 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:10.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:09 smithi165 ceph-mon[105517]: pgmap v3359: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 6 remapped+peering, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 5.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 46 MiB/s wr, 738 op/s; 35264/282711 objects degraded (12.474%); 17372/282711 objects misplaced (6.145%); 5.4 MiB/s, 86 objects/s recovering 2024-08-17T18:22:10.035 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:10.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:10.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:10.137 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:10.159 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:10.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:10.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:10.222 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 70 16 97028 97012 86.6015 51 0.00980661 0.0115412 2024-08-17T18:22:10.261 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:10.289 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:10.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:10.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:10.390 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:10.424 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:10.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:10.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:10.526 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:10.552 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:10.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:10.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:10.655 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:10.679 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:10.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:10.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:10 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 35264/282711 objects degraded (12.474%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:10.781 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:10.812 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:10.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:10.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:10.914 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:10.940 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:10.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:10.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:10.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:10 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 35264/282711 objects degraded (12.474%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:10.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:10 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 35264/282711 objects degraded (12.474%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:11.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:11.065 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:11.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:11.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:11.167 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:11.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:11.190 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:11.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:11.222 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 71 16 97898 97882 86.1476 54.375 0.00563768 0.0115984 2024-08-17T18:22:11.291 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:11.329 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:11.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:11.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:11.431 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:11.453 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:11.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:11.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:11.555 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:11.593 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:11.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:11.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:11.694 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:11.722 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:11.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:11.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:11.823 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:11.836 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:22:11.836 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-17T18:22:11.836 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:22:11.836 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:11 smithi165 ceph-mon[105517]: pgmap v3360: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 7 active+clean+remapped, 1 remapped+peering, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 5.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 36 MiB/s wr, 574 op/s; 35264/284028 objects degraded (12.416%); 17372/284028 objects misplaced (6.116%); 3.7 MiB/s, 58 objects/s recovering 2024-08-17T18:22:11.847 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:11.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:11.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:11.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:11 smithi136 ceph-mon[109861]: pgmap v3360: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 7 active+clean+remapped, 1 remapped+peering, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 5.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 36 MiB/s wr, 574 op/s; 35264/284028 objects degraded (12.416%); 17372/284028 objects misplaced (6.116%); 3.7 MiB/s, 58 objects/s recovering 2024-08-17T18:22:11.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:11 smithi136 ceph-mon[115078]: pgmap v3360: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 7 active+clean+remapped, 1 remapped+peering, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 5.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 36 MiB/s wr, 574 op/s; 35264/284028 objects degraded (12.416%); 17372/284028 objects misplaced (6.116%); 3.7 MiB/s, 58 objects/s recovering 2024-08-17T18:22:11.949 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:11.972 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:11.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:11.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:12.074 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:12.119 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:12.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:12.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:12.221 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:12.222 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 72 16 98557 98541 85.5232 41.1875 0.00901836 0.0116865 2024-08-17T18:22:12.265 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:12.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:12.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:12.367 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:12.395 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:12.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:12.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:12.497 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:12.537 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:12.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:12.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:12.639 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:12.666 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:12.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:12.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:12.768 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:12.792 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:12.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:12.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:12.894 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:12.930 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:12.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:12.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:13.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:13.058 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:13.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:13.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:13.160 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:13.185 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:13.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:13.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:13.222 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 73 16 99473 99457 85.1358 57.25 0.0130766 0.01174 2024-08-17T18:22:13.287 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:13.315 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:13.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:13.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:13.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:13.444 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:13.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:13.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:13.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:13.583 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:13.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:13.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:13.685 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:13.722 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:13.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:13.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:13.824 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:13.848 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:13.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:13.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:13.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:13 smithi136 ceph-mon[109861]: pgmap v3361: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.0 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 60 MiB/s wr, 961 op/s; 36894/295125 objects degraded (12.501%); 16859/295125 objects misplaced (5.712%); 12 MiB/s, 189 objects/s recovering 2024-08-17T18:22:13.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:13 smithi136 ceph-mon[115078]: pgmap v3361: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.0 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 60 MiB/s wr, 961 op/s; 36894/295125 objects degraded (12.501%); 16859/295125 objects misplaced (5.712%); 12 MiB/s, 189 objects/s recovering 2024-08-17T18:22:13.950 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:13.981 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:13.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:13.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:14.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:13 smithi165 ceph-mon[105517]: pgmap v3361: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.0 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 60 MiB/s wr, 961 op/s; 36894/295125 objects degraded (12.501%); 16859/295125 objects misplaced (5.712%); 12 MiB/s, 189 objects/s recovering 2024-08-17T18:22:14.083 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:14.122 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:14.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:14.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:14.222 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 74 16 100400 100384 84.7682 57.9375 0.0266293 0.0117915 2024-08-17T18:22:14.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:14.248 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:14.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:14.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:14.350 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:14.375 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:14.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:14.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:14.477 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:14.513 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:14.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:14.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:14.614 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:14.649 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:14.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:14.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:14.751 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:14.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:14.791 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:14.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:14.892 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:14.925 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:14.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:14.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:15.027 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:15.065 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:15.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:15.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:15.167 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:15.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:15.207 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:15.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:15.222 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 75 16 101277 101261 84.3688 54.8125 0.0413333 0.0118457 2024-08-17T18:22:15.308 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:15.333 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:15.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:15.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:15.435 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:15.461 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:15.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:15.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:15.563 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:15.606 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:15.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:15.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:15.708 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:15.733 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:15.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:15.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:15.835 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:15.861 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:15.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:15.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:15.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:15 smithi136 ceph-mon[115078]: pgmap v3362: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.0 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 46 MiB/s wr, 742 op/s; 36894/295125 objects degraded (12.501%); 16859/295125 objects misplaced (5.712%); 10 MiB/s, 163 objects/s recovering 2024-08-17T18:22:15.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:15 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 36894/295125 objects degraded (12.501%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:15.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:15 smithi136 ceph-mon[109861]: pgmap v3362: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.0 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 46 MiB/s wr, 742 op/s; 36894/295125 objects degraded (12.501%); 16859/295125 objects misplaced (5.712%); 10 MiB/s, 163 objects/s recovering 2024-08-17T18:22:15.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:15 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 36894/295125 objects degraded (12.501%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:15.963 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:16.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:16.000 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:16.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:16.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:15 smithi165 ceph-mon[105517]: pgmap v3362: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.0 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 46 MiB/s wr, 742 op/s; 36894/295125 objects degraded (12.501%); 16859/295125 objects misplaced (5.712%); 10 MiB/s, 163 objects/s recovering 2024-08-17T18:22:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:15 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 36894/295125 objects degraded (12.501%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:16.101 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:16.136 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:16.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:16.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:16.222 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 76 16 102134 102118 83.9634 53.5625 0.0272307 0.0119038 2024-08-17T18:22:16.241 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:16.265 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:16.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:16.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:16.367 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:16.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:16.408 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:16.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:16.509 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:16.552 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:16.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:16.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:16.654 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:16.680 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:16.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:16.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:16.781 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:16.832 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:16.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:16.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:16.934 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:16.978 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:16.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:16.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:17.080 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:17.104 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:17.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:17.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:17.205 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:17.222 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 77 16 103008 102992 83.5824 54.625 0.0113328 0.0119589 2024-08-17T18:22:17.231 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:17.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:17.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:17.333 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:17.361 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:17.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:17.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:17.462 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:17.486 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:17.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:17.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:17.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:17.624 INFO:tasks.thrashosds.thrasher:in_osds: [2, 4, 6, 0, 5, 7, 3, 1] out_osds: [] dead_osds: [6] live_osds: [0, 1, 3, 4, 5, 7, 2] 2024-08-17T18:22:17.624 INFO:tasks.thrashosds.thrasher:Reviving osd 6 2024-08-17T18:22:17.624 INFO:tasks.cephadm.osd.6:Restarting osd.6 (starting--it wasn't running)... 2024-08-17T18:22:17.624 DEBUG:teuthology.orchestra.run.smithi165:osd.6> sudo journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.6.service 2024-08-17T18:22:17.627 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl start ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.6 2024-08-17T18:22:17.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:17.653 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:17.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:17.730 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:17 smithi136 ceph-mon[109861]: pgmap v3363: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 47 MiB/s wr, 759 op/s; 37670/299856 objects degraded (12.563%); 16859/299856 objects misplaced (5.622%); 8.2 MiB/s, 132 objects/s recovering 2024-08-17T18:22:17.730 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:22:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:22:17] "GET /metrics HTTP/1.1" 200 36752 "" "Prometheus/2.51.0" 2024-08-17T18:22:17.730 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:17 smithi136 ceph-mon[115078]: pgmap v3363: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 47 MiB/s wr, 759 op/s; 37670/299856 objects degraded (12.563%); 16859/299856 objects misplaced (5.622%); 8.2 MiB/s, 132 objects/s recovering 2024-08-17T18:22:17.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:17.766 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-17T18:22:17.781 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:17.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:17.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:17.884 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:17.916 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:17.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:17.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:17.983 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:17 smithi165 ceph-mon[105517]: pgmap v3363: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 47 MiB/s wr, 759 op/s; 37670/299856 objects degraded (12.563%); 16859/299856 objects misplaced (5.622%); 8.2 MiB/s, 132 objects/s recovering 2024-08-17T18:22:17.983 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:17 smithi165 systemd[1]: Starting Ceph osd.6 for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T18:22:18.019 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:18.042 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:18.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:18.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:18.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:18.174 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:18.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:18.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:18.222 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 78 16 103751 103735 83.1062 46.4375 0.0926252 0.0120245 2024-08-17T18:22:18.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:17 smithi165 podman[283552]: 2024-08-17 18:22:17.885666932 +0000 UTC m=+0.023631309 image pull c870753d67c2504e5ac044c83696772f8b7527a928cc764ed24eeea7fea57d6e quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408 2024-08-17T18:22:18.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:18 smithi165 podman[283552]: 2024-08-17 18:22:18.013105369 +0000 UTC m=+0.151069724 container create 4b000bccc80cbac8b0483265fa56757b364ae76362e5fcf3394e0c920fe847db (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, RELEASE=reef-ced649f, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T18:22:18.276 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:18.302 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:18.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:18.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:18.404 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:18.452 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:18.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:18.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:18.465 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:22:18.554 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:18.581 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:18.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:18.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:18.604 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:18 smithi165 podman[283552]: 2024-08-17 18:22:18.274765623 +0000 UTC m=+0.412729989 container init 4b000bccc80cbac8b0483265fa56757b364ae76362e5fcf3394e0c920fe847db (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6-activate, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=reef-ced649f, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-17T18:22:18.604 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:18 smithi165 podman[283552]: 2024-08-17 18:22:18.281399047 +0000 UTC m=+0.419363413 container start 4b000bccc80cbac8b0483265fa56757b364ae76362e5fcf3394e0c920fe847db (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6-activate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=reef-ced649f, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T18:22:18.604 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:18 smithi165 podman[283552]: 2024-08-17 18:22:18.382990863 +0000 UTC m=+0.520955224 container attach 4b000bccc80cbac8b0483265fa56757b364ae76362e5fcf3394e0c920fe847db (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6-activate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-ced649f, org.label-schema.license=GPLv2) 2024-08-17T18:22:18.636 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:22:18.636 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:22:18.683 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:18.683 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:22:18.684 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:22:18.717 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:22:18.720 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:18.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:18.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:18.730 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:22:18.731 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:22:18.774 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:22:18.775 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:22:18.814 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:22:18.814 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:22:18.822 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:18.852 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:22:18.852 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:22:18.853 INFO:teuthology.orchestra.run.smithi136.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-17T18:22:18.869 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:18.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:18.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:18.884 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:22:18.884 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:22:18.971 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:18 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3874278271' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:22:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:18 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1413875445' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:22:19.011 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:18 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6-activate[283612]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-08-17T18:22:19.011 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:18 smithi165 bash[283552]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-08-17T18:22:19.011 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:18 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6-activate[283612]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-08-17T18:22:19.011 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:18 smithi165 bash[283552]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-08-17T18:22:19.012 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:18 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6-activate[283612]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-08-17T18:22:19.012 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:18 smithi165 bash[283552]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-08-17T18:22:19.012 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:18 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6-activate[283612]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-08-17T18:22:19.012 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:18 smithi165 bash[283552]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-08-17T18:22:19.012 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:18 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6-activate[283612]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-08-17T18:22:19.012 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:18 smithi165 bash[283552]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-08-17T18:22:19.012 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:18 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6-activate[283612]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-08-17T18:22:19.012 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:18 smithi165 bash[283552]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-08-17T18:22:19.012 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:18 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6-activate[283612]: --> ceph-volume raw activate successful for osd ID: 6 2024-08-17T18:22:19.012 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:18 smithi165 bash[283552]: --> ceph-volume raw activate successful for osd ID: 6 2024-08-17T18:22:19.012 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:18 smithi165 podman[283552]: 2024-08-17 18:22:18.781136177 +0000 UTC m=+0.919100541 container died 4b000bccc80cbac8b0483265fa56757b364ae76362e5fcf3394e0c920fe847db (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6-activate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=reef-ced649f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_BRANCH=HEAD) 2024-08-17T18:22:19.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:19.016 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:19.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:19.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:19.143 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:19.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:19.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:19.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:18 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3874278271' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:22:19.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:18 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1413875445' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:22:19.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:18 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3874278271' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:22:19.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:18 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1413875445' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:22:19.223 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 79 16 104477 104461 82.6285 45.375 0.0111402 0.0120958 2024-08-17T18:22:19.245 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:19.270 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:19.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:19.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:19.372 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:19.418 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:19.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:19.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:19.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:19.545 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:19.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:19.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:19.647 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:19.675 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:19 smithi165 podman[283845]: 2024-08-17 18:22:19.424022165 +0000 UTC m=+0.666444609 container remove 4b000bccc80cbac8b0483265fa56757b364ae76362e5fcf3394e0c920fe847db (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6-activate, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=reef-ced649f, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-08-17T18:22:19.676 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:19.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:19.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:19.778 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:19.805 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:19.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:19.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:19.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:19.940 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:19.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:19.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:19 smithi165 ceph-mon[105517]: pgmap v3364: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 49 MiB/s wr, 784 op/s; 38500/308358 objects degraded (12.485%); 16415/308358 objects misplaced (5.323%); 11 MiB/s, 172 objects/s recovering 2024-08-17T18:22:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:19 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:22:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:19 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3874278271' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:22:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:19 smithi165 ceph-mon[105517]: osdmap e1720: 8 total, 7 up, 8 in 2024-08-17T18:22:20.019 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:19 smithi136 ceph-mon[109861]: pgmap v3364: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 49 MiB/s wr, 784 op/s; 38500/308358 objects degraded (12.485%); 16415/308358 objects misplaced (5.323%); 11 MiB/s, 172 objects/s recovering 2024-08-17T18:22:20.019 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:19 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:22:20.019 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:19 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3874278271' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:22:20.019 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:19 smithi136 ceph-mon[109861]: osdmap e1720: 8 total, 7 up, 8 in 2024-08-17T18:22:20.019 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:19 smithi136 ceph-mon[115078]: pgmap v3364: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 49 MiB/s wr, 784 op/s; 38500/308358 objects degraded (12.485%); 16415/308358 objects misplaced (5.323%); 11 MiB/s, 172 objects/s recovering 2024-08-17T18:22:20.019 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:19 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:22:20.019 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:19 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3874278271' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:22:20.020 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:19 smithi136 ceph-mon[115078]: osdmap e1720: 8 total, 7 up, 8 in 2024-08-17T18:22:20.042 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:20.079 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:20.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:20.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:20.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:20.210 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:20.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:20.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:20.222 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:2024-08-17T18:22:20.223285+0000 min lat: 0.00120749 max lat: 2.22146 avg lat: 0.0121449 2024-08-17T18:22:20.223 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T18:22:20.223 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 80 16 105374 105358 82.2964 56.0625 0.0102226 0.0121449 2024-08-17T18:22:20.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:20.337 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:20.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:20.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:20.351 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-17T18:22:20.351 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-17T18:22:20.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:20.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:20.476 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:20.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:20.578 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:20.617 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:20.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:20.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:20.719 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:20.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:20.746 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:20.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:20.847 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:20.872 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:20.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:20.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:20.941 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:20 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 38500/308358 objects degraded (12.485%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:20.941 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:20 smithi165 podman[284091]: 2024-08-17 18:22:20.657009485 +0000 UTC m=+0.027008182 image pull c870753d67c2504e5ac044c83696772f8b7527a928cc764ed24eeea7fea57d6e quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408 2024-08-17T18:22:20.941 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:20 smithi165 podman[284091]: 2024-08-17 18:22:20.789720704 +0000 UTC m=+0.159719393 container create f0789491d7c7fae594262071cd5b815b9765d728379aa50813edd28bcba1cb5d (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=reef-ced649f, org.label-schema.build-date=20240731) 2024-08-17T18:22:20.974 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:21.017 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:21.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:21.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:21.119 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:21.153 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:21.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:21.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:21.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:20 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 38500/308358 objects degraded (12.485%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:21.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:20 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 38500/308358 objects degraded (12.485%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:21.223 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 81 16 106286 106270 81.9841 57 0.0109316 0.0121918 2024-08-17T18:22:21.255 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:21.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:21.296 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:21.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:21.397 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:21.425 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:21.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:21.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:21.511 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:21 smithi165 podman[284091]: 2024-08-17 18:22:21.279182825 +0000 UTC m=+0.649181518 container init f0789491d7c7fae594262071cd5b815b9765d728379aa50813edd28bcba1cb5d (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, RELEASE=reef-ced649f) 2024-08-17T18:22:21.511 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:21 smithi165 podman[284091]: 2024-08-17 18:22:21.286186461 +0000 UTC m=+0.656185144 container start f0789491d7c7fae594262071cd5b815b9765d728379aa50813edd28bcba1cb5d (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=reef-ced649f, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-08-17T18:22:21.511 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:21 smithi165 ceph-osd[284200]: -- 172.21.15.165:0/1245716765 <== mon.1 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a0ad2bb1e0 con 0x55a0ad42e000 2024-08-17T18:22:21.527 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:21.555 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:21.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:21.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:21.557 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:22:21.557 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:22:21.657 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:21.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:21.689 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:21.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:21.790 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:21.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:21.826 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:21.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:21.848 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:22:21.880 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:21 smithi165 ceph-mon[105517]: pgmap v3366: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 54 MiB/s wr, 860 op/s; 38500/309849 objects degraded (12.425%); 16415/309849 objects misplaced (5.298%); 12 MiB/s, 188 objects/s recovering 2024-08-17T18:22:21.880 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:22:21.927 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:21.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:21.954 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:21.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:22.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:22.123 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:22.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:22.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:21 smithi136 ceph-mon[109861]: pgmap v3366: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 54 MiB/s wr, 860 op/s; 38500/309849 objects degraded (12.425%); 16415/309849 objects misplaced (5.298%); 12 MiB/s, 188 objects/s recovering 2024-08-17T18:22:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:22:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:21 smithi136 ceph-mon[115078]: pgmap v3366: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 54 MiB/s wr, 860 op/s; 38500/309849 objects degraded (12.425%); 16415/309849 objects misplaced (5.298%); 12 MiB/s, 188 objects/s recovering 2024-08-17T18:22:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:22:22.223 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 82 16 107148 107132 81.6409 53.875 0.0223807 0.0122428 2024-08-17T18:22:22.226 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:22.249 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:22.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:22.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:22.260 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:21 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:22:21.880+0000 7fb1a31ba640 -1 Falling back to public interface 2024-08-17T18:22:22.267 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:22:22.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:22.352 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:22:22.377 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:22.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:22.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:22.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:22.508 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-17T18:22:22.548 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:22.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:22.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:22.650 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:22.718 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:22.719 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:22 smithi165 bash[284091]: f0789491d7c7fae594262071cd5b815b9765d728379aa50813edd28bcba1cb5d 2024-08-17T18:22:22.719 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:22 smithi165 systemd[1]: Started Ceph osd.6 for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T18:22:22.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:22.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:22.721 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-17T18:22:22.781 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:22:22.821 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:22.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:22.876 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:22.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:22.977 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:23.023 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:23.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:23.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:23.125 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:23.162 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:23.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:23.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:23.223 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 83 16 107966 107950 81.2732 51.125 0.0190628 0.0122961 2024-08-17T18:22:23.265 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:23.312 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:23.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:23.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:23.414 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:23.450 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:23.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:23.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:23.551 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:23.587 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:23.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:23.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:23.622 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:22:23.689 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:23.733 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:23.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:23.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:23.835 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:23.865 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:23.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:23.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:23.967 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:23.974 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:22:24.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:23 smithi165 ceph-mon[105517]: pgmap v3367: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+clean+remapped+scrubbing, 1 active+recovering+undersized+remapped, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 45 MiB/s wr, 725 op/s; 39801/316914 objects degraded (12.559%); 16100/316914 objects misplaced (5.080%); 10 MiB/s, 165 objects/s recovering 2024-08-17T18:22:24.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:24.015 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:24.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:24.116 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:24.137 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:23 smithi136 ceph-mon[109861]: pgmap v3367: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+clean+remapped+scrubbing, 1 active+recovering+undersized+remapped, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 45 MiB/s wr, 725 op/s; 39801/316914 objects degraded (12.559%); 16100/316914 objects misplaced (5.080%); 10 MiB/s, 165 objects/s recovering 2024-08-17T18:22:24.137 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:23 smithi136 ceph-mon[115078]: pgmap v3367: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+clean+remapped+scrubbing, 1 active+recovering+undersized+remapped, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 45 MiB/s wr, 725 op/s; 39801/316914 objects degraded (12.559%); 16100/316914 objects misplaced (5.080%); 10 MiB/s, 165 objects/s recovering 2024-08-17T18:22:24.142 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:24.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:24.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:24.224 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 84 16 108837 108821 80.9537 54.4375 0.00978335 0.0123458 2024-08-17T18:22:24.244 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:24.275 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:24.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:24.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:24.378 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:24.409 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:24.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:24.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:24.511 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:24.551 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:24.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:24.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:24.652 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:24.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:24.694 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:24.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:24.795 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:24.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:24.830 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:24.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:24.931 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:24.979 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:24.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:24.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:25.081 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:25.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:25.117 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:25.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:25.218 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:25.223 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 85 16 109636 109620 80.5887 49.9375 0.0220085 0.0124024 2024-08-17T18:22:25.247 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:25.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:25.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:25.350 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:25.386 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:25.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:25.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:25.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:25.527 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:25.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:25.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:25.629 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:25.658 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:25.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:25.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:25.727 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:22:25.760 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:25.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:25.784 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:25.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:25.886 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:25.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:25.916 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:25.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:25.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:25 smithi136 ceph-mon[109861]: pgmap v3368: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 53 MiB/s wr, 855 op/s; 40066/320790 objects degraded (12.490%); 16031/320790 objects misplaced (4.997%); 11 MiB/s, 172 objects/s recovering 2024-08-17T18:22:25.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:25 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 40066/320790 objects degraded (12.490%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:25.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:25 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1945626308' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:22:25.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:25 smithi136 ceph-mon[115078]: pgmap v3368: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 53 MiB/s wr, 855 op/s; 40066/320790 objects degraded (12.490%); 16031/320790 objects misplaced (4.997%); 11 MiB/s, 172 objects/s recovering 2024-08-17T18:22:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:25 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 40066/320790 objects degraded (12.490%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:25.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:25 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1945626308' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:22:26.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:25 smithi165 ceph-mon[105517]: pgmap v3368: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 53 MiB/s wr, 855 op/s; 40066/320790 objects degraded (12.490%); 16031/320790 objects misplaced (4.997%); 11 MiB/s, 172 objects/s recovering 2024-08-17T18:22:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:25 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 40066/320790 objects degraded (12.490%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:26.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:25 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1945626308' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:22:26.017 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:26.057 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:26.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:26.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:26.151 INFO:teuthology.orchestra.run.smithi165.stderr:no valid command found; 10 closest matches: 2024-08-17T18:22:26.151 INFO:teuthology.orchestra.run.smithi165.stderr:0 2024-08-17T18:22:26.151 INFO:teuthology.orchestra.run.smithi165.stderr:1 2024-08-17T18:22:26.152 INFO:teuthology.orchestra.run.smithi165.stderr:2 2024-08-17T18:22:26.152 INFO:teuthology.orchestra.run.smithi165.stderr:abort 2024-08-17T18:22:26.152 INFO:teuthology.orchestra.run.smithi165.stderr:assert 2024-08-17T18:22:26.152 INFO:teuthology.orchestra.run.smithi165.stderr:bluefs debug_inject_read_zeros 2024-08-17T18:22:26.152 INFO:teuthology.orchestra.run.smithi165.stderr:bluefs files list 2024-08-17T18:22:26.152 INFO:teuthology.orchestra.run.smithi165.stderr:bluefs stats 2024-08-17T18:22:26.152 INFO:teuthology.orchestra.run.smithi165.stderr:bluestore bluefs device info [] 2024-08-17T18:22:26.152 INFO:teuthology.orchestra.run.smithi165.stderr:config diff 2024-08-17T18:22:26.152 INFO:teuthology.orchestra.run.smithi165.stderr:admin_socket: invalid command 2024-08-17T18:22:26.158 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:26.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:26.209 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:26.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:26.225 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 86 16 110418 110402 80.2189 48.875 0.0206124 0.0124596 2024-08-17T18:22:26.310 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:26.364 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:26.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:26.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:26.465 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:26.494 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:26.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:26.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:26.595 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:26.631 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:26.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:26.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:26.733 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:26.759 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:26.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:26.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:26.861 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:26.888 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:26 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:22:26.888 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:26 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1945626308' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:22:26.888 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:26 smithi165 ceph-mon[105517]: osdmap e1721: 8 total, 7 up, 8 in 2024-08-17T18:22:26.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:26.889 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:26.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:26.924 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-17T18:22:26.925 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-08-17T18:22:26.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:26 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:22:26.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:26 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1945626308' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:22:26.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:26 smithi136 ceph-mon[109861]: osdmap e1721: 8 total, 7 up, 8 in 2024-08-17T18:22:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:26 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:22:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:26 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1945626308' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:22:26.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:26 smithi136 ceph-mon[115078]: osdmap e1721: 8 total, 7 up, 8 in 2024-08-17T18:22:26.989 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:27.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:27.029 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:27.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:27.130 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:27.175 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:27.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:27.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:27.225 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 87 16 111216 111200 79.8702 49.875 0.00386457 0.0125104 2024-08-17T18:22:27.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:27.304 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:27.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:27.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:27.406 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:27.446 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:27.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:27.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:27.548 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:27.576 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:27.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:27.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:27.677 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:27.718 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:27.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:27.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:27.741 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:22:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:22:27] "GET /metrics HTTP/1.1" 200 36775 "" "Prometheus/2.51.0" 2024-08-17T18:22:27.821 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:27.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:27.861 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:27.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:27.961 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:28.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:27 smithi165 ceph-mon[105517]: pgmap v3370: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 45 MiB/s wr, 713 op/s; 40830/325491 objects degraded (12.544%); 16031/325491 objects misplaced (4.925%); 6.2 MiB/s, 99 objects/s recovering 2024-08-17T18:22:28.017 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:27 smithi136 ceph-mon[115078]: pgmap v3370: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 45 MiB/s wr, 713 op/s; 40830/325491 objects degraded (12.544%); 16031/325491 objects misplaced (4.925%); 6.2 MiB/s, 99 objects/s recovering 2024-08-17T18:22:28.017 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:28.017 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:27 smithi136 ceph-mon[109861]: pgmap v3370: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 45 MiB/s wr, 713 op/s; 40830/325491 objects degraded (12.544%); 16031/325491 objects misplaced (4.925%); 6.2 MiB/s, 99 objects/s recovering 2024-08-17T18:22:28.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:28.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:28.121 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:28.149 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:28.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:28.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:28.225 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 88 16 112101 112085 79.5911 55.3125 0.0233616 0.0125588 2024-08-17T18:22:28.251 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:28.277 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:28.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:28.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:28.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:28.420 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:28.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:28.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:28.522 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:28.547 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:28.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:28.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:28.600 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:22:28.600 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:22:28.631 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:22:28.632 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:22:28.648 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:28.665 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:22:28.665 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:22:28.676 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:28.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:28.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:28.699 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:22:28.699 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:22:28.731 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:22:28.731 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:22:28.777 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:28.811 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:22:28.811 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:22:28.812 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:28.813 INFO:teuthology.orchestra.run.smithi136.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-08-17T18:22:28.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:28.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:28.851 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:22:28.851 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:22:28.851 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:28 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1701290236' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:22:28.852 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:28 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1701290236' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:22:28.914 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:28.943 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:28.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:28.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:29.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:28 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1701290236' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:22:29.045 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:29.083 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:29.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:29.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:29.185 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:29.209 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:29.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:29.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:29.225 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 89 16 112807 112791 79.1926 44.125 0.013356 0.0126075 2024-08-17T18:22:29.311 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:29.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:29.354 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:29.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:29.412 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:22:29.412 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:22:29.412 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-08-17T18:22:29.412 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-08-17T18:22:29.412 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:22:29.455 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:29.501 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:29.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:29.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:29.604 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:29.627 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:29.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:29.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:29.655 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-17T18:22:29.656 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-17T18:22:29.729 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:29.774 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:29.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:29.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:29.875 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:29.902 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:29.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:29.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:29.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:29 smithi136 ceph-mon[109861]: pgmap v3371: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 8 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.8 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 53 MiB/s wr, 844 op/s; 41629/333303 objects degraded (12.490%); 15592/333303 objects misplaced (4.678%); 10 MiB/s, 163 objects/s recovering 2024-08-17T18:22:29.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:29 smithi136 ceph-mon[115078]: pgmap v3371: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 8 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.8 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 53 MiB/s wr, 844 op/s; 41629/333303 objects degraded (12.490%); 15592/333303 objects misplaced (4.678%); 10 MiB/s, 163 objects/s recovering 2024-08-17T18:22:30.003 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:30.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:29 smithi165 ceph-mon[105517]: pgmap v3371: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 8 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.8 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 53 MiB/s wr, 844 op/s; 41629/333303 objects degraded (12.490%); 15592/333303 objects misplaced (4.678%); 10 MiB/s, 163 objects/s recovering 2024-08-17T18:22:30.021 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:22:30.035 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:30.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:30.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:30.137 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:30.182 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:30.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:30.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:30.186 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-17T18:22:30.225 INFO:tasks.radosbench.radosbench.0.smithi136.stdout: 90 7 113613 113606 78.8786 50.9375 0.0959782 0.0126679 2024-08-17T18:22:30.284 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:30.307 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:30.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:30.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:30.327 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:22:30.408 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:30.426 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:22:30.430 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:30.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:30.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:30.532 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:30.555 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:30.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:30.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:30.657 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:30.681 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:30.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:30.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:30.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:30.808 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:30.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:30.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:30.896 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:30 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 41629/333303 objects degraded (12.490%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:30.909 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:30.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:30.932 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:30.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:31.033 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:31.057 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:31.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:31.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:31.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:31.183 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:31.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:31.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:31.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:30 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 41629/333303 objects degraded (12.490%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:31.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:30 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 41629/333303 objects degraded (12.490%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:31.225 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Total time run: 90.0381 2024-08-17T18:22:31.225 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Total writes made: 113613 2024-08-17T18:22:31.225 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Write size: 65536 2024-08-17T18:22:31.226 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Object size: 65536 2024-08-17T18:22:31.226 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Bandwidth (MB/sec): 78.8645 2024-08-17T18:22:31.226 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Stddev Bandwidth: 37.1598 2024-08-17T18:22:31.226 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Max bandwidth (MB/sec): 207.375 2024-08-17T18:22:31.226 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Min bandwidth (MB/sec): 0 2024-08-17T18:22:31.226 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Average IOPS: 1261 2024-08-17T18:22:31.226 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Stddev IOPS: 594.558 2024-08-17T18:22:31.226 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Max IOPS: 3318 2024-08-17T18:22:31.226 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Min IOPS: 0 2024-08-17T18:22:31.227 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Average Latency(s): 0.0126727 2024-08-17T18:22:31.227 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Stddev Latency(s): 0.0301779 2024-08-17T18:22:31.227 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Max latency(s): 2.22146 2024-08-17T18:22:31.227 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Min latency(s): 0.00120749 2024-08-17T18:22:31.232 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Cleaning up (deleting benchmark objects) 2024-08-17T18:22:31.285 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:31.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:31.330 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:31.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:31.431 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:31.462 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:31.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:31.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:31.564 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:31.589 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:31.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:31.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:31.657 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:22:31.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:31.717 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:31.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:31.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:31.819 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:31.846 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:31.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:31.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:31.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-17T18:22:31.948 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:31.952 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:31 smithi136 ceph-mon[109861]: pgmap v3372: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 8 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.8 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 52 MiB/s wr, 828 op/s; 41629/334716 objects degraded (12.437%); 15592/334716 objects misplaced (4.658%); 10 MiB/s, 161 objects/s recovering 2024-08-17T18:22:31.952 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:31 smithi136 ceph-mon[115078]: pgmap v3372: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 8 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.8 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 52 MiB/s wr, 828 op/s; 41629/334716 objects degraded (12.437%); 15592/334716 objects misplaced (4.658%); 10 MiB/s, 161 objects/s recovering 2024-08-17T18:22:31.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:31.991 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:31.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:32.090 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:31 smithi165 ceph-mon[105517]: pgmap v3372: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 8 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.8 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 52 MiB/s wr, 828 op/s; 41629/334716 objects degraded (12.437%); 15592/334716 objects misplaced (4.658%); 10 MiB/s, 161 objects/s recovering 2024-08-17T18:22:32.093 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:32.132 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:32.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:32.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:32.234 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:22:32.235 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:32.284 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:32.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:32.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:32.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:32.413 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:32.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:32.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:32.514 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:32.544 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:32.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:32.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:32.646 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:32.698 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:32.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:32.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:32.800 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:32.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:32.845 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:32.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:32.946 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:32.979 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:32.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:32.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:33.081 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:33.111 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:33.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:33.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:33.213 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:33.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:33.263 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:33.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:33.364 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:33.403 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:33.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:33.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:33.505 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:33.531 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:22:33.556 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:33.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:33.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:33.658 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:33.669 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:22:33.669 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T18:22:33.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T18:22:33.669 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:22:33.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.670 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:122085 57.2 e1639/1627 57:42ce4696:::benchmark_data_smithi136_389235_object6163:head v 1639'6295, mlcod=0'0)", 2024-08-17T18:22:33.670 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:54.574819+0000", 2024-08-17T18:22:33.670 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 398.63681496499999, 2024-08-17T18:22:33.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5239920300000001, 2024-08-17T18:22:33.671 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:33.671 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:33.671 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:33.671 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.671 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:33.671 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.574819+0000", 2024-08-17T18:22:33.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.672 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.672 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.672 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:33.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.574819+0000", 2024-08-17T18:22:33.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:33.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.574818+0000", 2024-08-17T18:22:33.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:22:33.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:33.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.574826+0000", 2024-08-17T18:22:33.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7100000000000007e-06 2024-08-17T18:22:33.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.675 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:33.675 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.574840+0000", 2024-08-17T18:22:33.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3709999999999999e-05 2024-08-17T18:22:33.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.676 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:33.676 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.574847+0000", 2024-08-17T18:22:33.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.764e-06 2024-08-17T18:22:33.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.677 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.091103+0000", 2024-08-17T18:22:33.677 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5162560969999999 2024-08-17T18:22:33.677 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.677 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:33.678 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.091123+0000", 2024-08-17T18:22:33.678 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0537e-05 2024-08-17T18:22:33.678 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.678 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:33.678 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.098763+0000", 2024-08-17T18:22:33.679 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0076399299999999996 2024-08-17T18:22:33.679 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.679 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.679 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:33.679 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.098811+0000", 2024-08-17T18:22:33.679 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8256e-05 2024-08-17T18:22:33.680 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.680 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:33.680 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.680 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.680 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.680 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:122093 57.2 e1639/1627 57:407dd288:::benchmark_data_smithi136_389235_object6171:head v 1639'6296, mlcod=0'0)", 2024-08-17T18:22:33.680 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:54.622473+0000", 2024-08-17T18:22:33.681 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 398.589161064, 2024-08-17T18:22:33.681 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.476358522, 2024-08-17T18:22:33.681 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:33.681 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:33.681 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:33.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.681 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:33.682 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.622473+0000", 2024-08-17T18:22:33.682 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.682 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.682 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.682 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:33.683 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.622473+0000", 2024-08-17T18:22:33.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.684 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:33.684 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.622472+0000", 2024-08-17T18:22:33.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:22:33.684 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.684 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.684 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:33.685 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.622481+0000", 2024-08-17T18:22:33.685 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1880000000000008e-06 2024-08-17T18:22:33.685 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.685 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.685 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:33.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.622494+0000", 2024-08-17T18:22:33.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2513e-05 2024-08-17T18:22:33.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:33.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.622500+0000", 2024-08-17T18:22:33.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7769999999999999e-06 2024-08-17T18:22:33.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.687 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.094420+0000", 2024-08-17T18:22:33.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4719201590000002 2024-08-17T18:22:33.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:33.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.094441+0000", 2024-08-17T18:22:33.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1554000000000001e-05 2024-08-17T18:22:33.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:33.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.098815+0000", 2024-08-17T18:22:33.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004373633 2024-08-17T18:22:33.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.689 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:33.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.098832+0000", 2024-08-17T18:22:33.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6744000000000001e-05 2024-08-17T18:22:33.689 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.690 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:33.690 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.690 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:122102 57.2 e1641/1627 57:436a03b4:::benchmark_data_smithi136_389235_object6180:head v 1641'6297, mlcod=1612'631)", 2024-08-17T18:22:33.690 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:57.129135+0000", 2024-08-17T18:22:33.691 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 396.08249923199998, 2024-08-17T18:22:33.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3257561870000001, 2024-08-17T18:22:33.691 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:33.691 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:33.691 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:33.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:33.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.129135+0000", 2024-08-17T18:22:33.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:33.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.129135+0000", 2024-08-17T18:22:33.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:33.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.129135+0000", 2024-08-17T18:22:33.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:22:33.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:33.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.129143+0000", 2024-08-17T18:22:33.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1459999999999996e-06 2024-08-17T18:22:33.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:33.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.129152+0000", 2024-08-17T18:22:33.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0440000000000001e-06 2024-08-17T18:22:33.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:33.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.129157+0000", 2024-08-17T18:22:33.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7740000000000001e-06 2024-08-17T18:22:33.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.324262+0000", 2024-08-17T18:22:33.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1951041249999999 2024-08-17T18:22:33.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:33.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.324289+0000", 2024-08-17T18:22:33.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7246e-05 2024-08-17T18:22:33.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:33.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.454857+0000", 2024-08-17T18:22:33.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13056785200000001 2024-08-17T18:22:33.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:33.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.454891+0000", 2024-08-17T18:22:33.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4575e-05 2024-08-17T18:22:33.696 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.696 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:33.696 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.696 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:133120 57.12 57:49924cf9:::benchmark_data_smithi136_389235_object17198:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1648)", 2024-08-17T18:22:33.696 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:35.543899+0000", 2024-08-17T18:22:33.696 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 357.66773540000003, 2024-08-17T18:22:33.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1856874199999998, 2024-08-17T18:22:33.696 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:33.697 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:33.697 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:33.697 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:22:33.697 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:22:33.697 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133120 2024-08-17T18:22:33.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.697 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:33.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:33.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:35.543899+0000", 2024-08-17T18:22:33.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:33.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:35.543899+0000", 2024-08-17T18:22:33.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:33.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:35.543897+0000", 2024-08-17T18:22:33.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:22:33.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:33.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:35.543905+0000", 2024-08-17T18:22:33.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7819999999999994e-06 2024-08-17T18:22:33.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:33.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:35.543919+0000", 2024-08-17T18:22:33.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.432e-05 2024-08-17T18:22:33.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:33.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:35.543926+0000", 2024-08-17T18:22:33.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8279999999999997e-06 2024-08-17T18:22:33.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:35.643933+0000", 2024-08-17T18:22:33.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100006952 2024-08-17T18:22:33.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:22:33.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:35.646011+0000", 2024-08-17T18:22:33.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002078009 2024-08-17T18:22:33.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:36.551693+0000", 2024-08-17T18:22:33.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90568204399999996 2024-08-17T18:22:33.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:22:33.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:36.557914+0000", 2024-08-17T18:22:33.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062204959999999998 2024-08-17T18:22:33.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.616506+0000", 2024-08-17T18:22:33.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.058592816 2024-08-17T18:22:33.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:33.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.619828+0000", 2024-08-17T18:22:33.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033217440000000002 2024-08-17T18:22:33.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:22:33.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.626665+0000", 2024-08-17T18:22:33.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068367649999999999 2024-08-17T18:22:33.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.632385+0000", 2024-08-17T18:22:33.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057198839999999997 2024-08-17T18:22:33.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.633470+0000", 2024-08-17T18:22:33.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010857499999999999 2024-08-17T18:22:33.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:33.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.716484+0000", 2024-08-17T18:22:33.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.083013179000000006 2024-08-17T18:22:33.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:33.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.718934+0000", 2024-08-17T18:22:33.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024506319999999999 2024-08-17T18:22:33.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:33.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.729586+0000", 2024-08-17T18:22:33.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010652053 2024-08-17T18:22:33.705 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.705 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:33.705 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.705 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:133992 57.12 57:48ed4d24:::benchmark_data_smithi136_389235_object18070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1648)", 2024-08-17T18:22:33.706 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:38.781600+0000", 2024-08-17T18:22:33.706 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 354.43003412000002, 2024-08-17T18:22:33.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1134712009999999, 2024-08-17T18:22:33.706 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:33.706 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:33.706 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:33.706 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:22:33.706 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:22:33.706 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133992 2024-08-17T18:22:33.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.707 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:33.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:33.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.781600+0000", 2024-08-17T18:22:33.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:33.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.781600+0000", 2024-08-17T18:22:33.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:33.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.781599+0000", 2024-08-17T18:22:33.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:22:33.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:33.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.781604+0000", 2024-08-17T18:22:33.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8099999999999997e-06 2024-08-17T18:22:33.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:33.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.781615+0000", 2024-08-17T18:22:33.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0771000000000001e-05 2024-08-17T18:22:33.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:33.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.781620+0000", 2024-08-17T18:22:33.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9740000000000001e-06 2024-08-17T18:22:33.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.847439+0000", 2024-08-17T18:22:33.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.065819201999999993 2024-08-17T18:22:33.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:22:33.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.851794+0000", 2024-08-17T18:22:33.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043556259999999996 2024-08-17T18:22:33.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:39.873915+0000", 2024-08-17T18:22:33.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0221207800000001 2024-08-17T18:22:33.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:22:33.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:39.878239+0000", 2024-08-17T18:22:33.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004323716 2024-08-17T18:22:33.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.852201+0000", 2024-08-17T18:22:33.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.973961909 2024-08-17T18:22:33.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:33.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.855640+0000", 2024-08-17T18:22:33.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034392709999999998 2024-08-17T18:22:33.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:22:33.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.862697+0000", 2024-08-17T18:22:33.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070570750000000003 2024-08-17T18:22:33.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.882488+0000", 2024-08-17T18:22:33.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019790395999999998 2024-08-17T18:22:33.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.883769+0000", 2024-08-17T18:22:33.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012813869999999999 2024-08-17T18:22:33.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:33.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.884136+0000", 2024-08-17T18:22:33.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036747899999999998 2024-08-17T18:22:33.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:33.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.885963+0000", 2024-08-17T18:22:33.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00182672 2024-08-17T18:22:33.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:33.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.895071+0000", 2024-08-17T18:22:33.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0091082130000000004 2024-08-17T18:22:33.715 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.715 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:33.715 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.715 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:134465 57.12 57:49a43719:::benchmark_data_smithi136_389235_object18543:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1648)", 2024-08-17T18:22:33.715 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:40.473026+0000", 2024-08-17T18:22:33.716 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 352.73860827099998, 2024-08-17T18:22:33.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5757722840000001, 2024-08-17T18:22:33.716 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:33.716 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:33.716 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:33.716 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:22:33.716 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:22:33.716 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 134465 2024-08-17T18:22:33.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.716 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:33.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:33.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:40.473026+0000", 2024-08-17T18:22:33.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:33.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:40.473026+0000", 2024-08-17T18:22:33.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:33.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:40.473025+0000", 2024-08-17T18:22:33.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:22:33.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:33.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:40.473032+0000", 2024-08-17T18:22:33.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1280000000000002e-06 2024-08-17T18:22:33.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:33.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:40.473049+0000", 2024-08-17T18:22:33.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7317999999999999e-05 2024-08-17T18:22:33.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:33.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:40.473057+0000", 2024-08-17T18:22:33.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4309999999999994e-06 2024-08-17T18:22:33.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:40.572818+0000", 2024-08-17T18:22:33.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099760653000000005 2024-08-17T18:22:33.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:22:33.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:40.574727+0000", 2024-08-17T18:22:33.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019091240000000001 2024-08-17T18:22:33.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.377268+0000", 2024-08-17T18:22:33.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.80254037899999997 2024-08-17T18:22:33.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:22:33.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.382386+0000", 2024-08-17T18:22:33.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051183130000000002 2024-08-17T18:22:33.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:42.982237+0000", 2024-08-17T18:22:33.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.599851355 2024-08-17T18:22:33.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:33.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:42.987818+0000", 2024-08-17T18:22:33.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055806980000000003 2024-08-17T18:22:33.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:22:33.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:42.998928+0000", 2024-08-17T18:22:33.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011109734 2024-08-17T18:22:33.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:33.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.024007+0000", 2024-08-17T18:22:33.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.025079534000000001 2024-08-17T18:22:33.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.025175+0000", 2024-08-17T18:22:33.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011681350000000001 2024-08-17T18:22:33.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.026529+0000", 2024-08-17T18:22:33.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013539509999999999 2024-08-17T18:22:33.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:33.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.029124+0000", 2024-08-17T18:22:33.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025947209999999999 2024-08-17T18:22:33.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:33.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.048798+0000", 2024-08-17T18:22:33.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019674259999999999 2024-08-17T18:22:33.725 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.725 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:33.725 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.725 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:134821 57.12 57:49381b19:::benchmark_data_smithi136_389235_object18899:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1648)", 2024-08-17T18:22:33.725 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:41.544450+0000", 2024-08-17T18:22:33.725 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 351.667184647, 2024-08-17T18:22:33.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5392011000000001, 2024-08-17T18:22:33.725 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:33.725 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:33.726 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:33.726 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:22:33.726 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:22:33.726 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 134821 2024-08-17T18:22:33.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.726 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:33.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:33.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.544450+0000", 2024-08-17T18:22:33.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:33.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.544450+0000", 2024-08-17T18:22:33.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:33.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.544448+0000", 2024-08-17T18:22:33.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:22:33.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:33.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.544454+0000", 2024-08-17T18:22:33.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0630000000000002e-06 2024-08-17T18:22:33.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:33.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.544466+0000", 2024-08-17T18:22:33.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2170999999999999e-05 2024-08-17T18:22:33.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:33.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.544474+0000", 2024-08-17T18:22:33.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3069999999999997e-06 2024-08-17T18:22:33.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.566862+0000", 2024-08-17T18:22:33.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022388354999999999 2024-08-17T18:22:33.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:22:33.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.568805+0000", 2024-08-17T18:22:33.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019431330000000001 2024-08-17T18:22:33.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:42.238732+0000", 2024-08-17T18:22:33.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.66992708300000003 2024-08-17T18:22:33.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:22:33.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:42.243750+0000", 2024-08-17T18:22:33.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050172599999999999 2024-08-17T18:22:33.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.029453+0000", 2024-08-17T18:22:33.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.785703324 2024-08-17T18:22:33.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:33.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.032708+0000", 2024-08-17T18:22:33.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032548080000000001 2024-08-17T18:22:33.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:22:33.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.039182+0000", 2024-08-17T18:22:33.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064746630000000003 2024-08-17T18:22:33.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.050280+0000", 2024-08-17T18:22:33.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011097205000000001 2024-08-17T18:22:33.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.051571+0000", 2024-08-17T18:22:33.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001291169 2024-08-17T18:22:33.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:33.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.070117+0000", 2024-08-17T18:22:33.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018546475 2024-08-17T18:22:33.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:33.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.072184+0000", 2024-08-17T18:22:33.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020664170000000001 2024-08-17T18:22:33.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:33.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.083651+0000", 2024-08-17T18:22:33.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011467073 2024-08-17T18:22:33.734 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.734 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:33.734 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.735 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:135268 57.12 57:4ffd0969:::benchmark_data_smithi136_389235_object19346:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1648)", 2024-08-17T18:22:33.735 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:43.180362+0000", 2024-08-17T18:22:33.735 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 350.03127279799997, 2024-08-17T18:22:33.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.63819413, 2024-08-17T18:22:33.735 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:33.735 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:33.735 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:33.735 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:22:33.735 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:22:33.735 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 135268 2024-08-17T18:22:33.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.736 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:33.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:33.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.180362+0000", 2024-08-17T18:22:33.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:33.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.180362+0000", 2024-08-17T18:22:33.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:33.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.180360+0000", 2024-08-17T18:22:33.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:22:33.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:33.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.180367+0000", 2024-08-17T18:22:33.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5989999999999998e-06 2024-08-17T18:22:33.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:33.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.180379+0000", 2024-08-17T18:22:33.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1735e-05 2024-08-17T18:22:33.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:33.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.180384+0000", 2024-08-17T18:22:33.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8069999999999996e-06 2024-08-17T18:22:33.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.286732+0000", 2024-08-17T18:22:33.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.106347284 2024-08-17T18:22:33.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:22:33.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.289232+0000", 2024-08-17T18:22:33.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002500178 2024-08-17T18:22:33.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.963018+0000", 2024-08-17T18:22:33.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.67378577200000001 2024-08-17T18:22:33.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:22:33.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.967249+0000", 2024-08-17T18:22:33.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042313009999999998 2024-08-17T18:22:33.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.762994+0000", 2024-08-17T18:22:33.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.795744819 2024-08-17T18:22:33.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:33.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.766441+0000", 2024-08-17T18:22:33.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034468820000000001 2024-08-17T18:22:33.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:22:33.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.773636+0000", 2024-08-17T18:22:33.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071958530000000003 2024-08-17T18:22:33.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.800098+0000", 2024-08-17T18:22:33.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.026461710999999999 2024-08-17T18:22:33.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:33.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.800485+0000", 2024-08-17T18:22:33.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038679399999999998 2024-08-17T18:22:33.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.801755+0000", 2024-08-17T18:22:33.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001270427 2024-08-17T18:22:33.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:33.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.805500+0000", 2024-08-17T18:22:33.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037451469999999999 2024-08-17T18:22:33.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:33.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.818556+0000", 2024-08-17T18:22:33.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013055152 2024-08-17T18:22:33.744 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.744 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:33.744 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.744 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:135559 57.12 57:481716dc:::benchmark_data_smithi136_389235_object19637:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1649)", 2024-08-17T18:22:33.744 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:44.137515+0000", 2024-08-17T18:22:33.744 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 349.07411978699997, 2024-08-17T18:22:33.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0680285010000001, 2024-08-17T18:22:33.744 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:33.744 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:33.744 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:33.744 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:22:33.745 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:22:33.745 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 135559 2024-08-17T18:22:33.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.745 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:33.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:33.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.137515+0000", 2024-08-17T18:22:33.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:33.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.137515+0000", 2024-08-17T18:22:33.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:33.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.137513+0000", 2024-08-17T18:22:33.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:22:33.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:33.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.137521+0000", 2024-08-17T18:22:33.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0010000000000001e-06 2024-08-17T18:22:33.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:33.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.137538+0000", 2024-08-17T18:22:33.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6816e-05 2024-08-17T18:22:33.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:33.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.137546+0000", 2024-08-17T18:22:33.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2199999999999992e-06 2024-08-17T18:22:33.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.259176+0000", 2024-08-17T18:22:33.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.121630083 2024-08-17T18:22:33.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:22:33.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.261072+0000", 2024-08-17T18:22:33.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001895863 2024-08-17T18:22:33.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.204887+0000", 2024-08-17T18:22:33.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94381471800000005 2024-08-17T18:22:33.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:22:33.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.209973+0000", 2024-08-17T18:22:33.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005086775 2024-08-17T18:22:33.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:47.165222+0000", 2024-08-17T18:22:33.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9552484919999999 2024-08-17T18:22:33.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:33.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:47.169551+0000", 2024-08-17T18:22:33.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043290500000000001 2024-08-17T18:22:33.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:22:33.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:47.180180+0000", 2024-08-17T18:22:33.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010628629000000001 2024-08-17T18:22:33.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:47.188542+0000", 2024-08-17T18:22:33.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0083620650000000001 2024-08-17T18:22:33.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:47.191455+0000", 2024-08-17T18:22:33.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029128700000000001 2024-08-17T18:22:33.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:33.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:47.191846+0000", 2024-08-17T18:22:33.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000391911 2024-08-17T18:22:33.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:33.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:47.194664+0000", 2024-08-17T18:22:33.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028179059999999998 2024-08-17T18:22:33.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:33.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:47.205543+0000", 2024-08-17T18:22:33.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010878649000000001 2024-08-17T18:22:33.753 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.753 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:33.753 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.753 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:141699 57.12 57:4fa5408d:::benchmark_data_smithi136_389235_object25777:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1650)", 2024-08-17T18:22:33.753 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:02.480434+0000", 2024-08-17T18:22:33.753 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 330.73120008799998, 2024-08-17T18:22:33.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.198213231, 2024-08-17T18:22:33.753 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:33.754 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:33.754 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:33.754 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:22:33.754 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:22:33.754 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 141699 2024-08-17T18:22:33.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.754 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:33.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:33.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:02.480434+0000", 2024-08-17T18:22:33.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:33.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:02.480434+0000", 2024-08-17T18:22:33.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:33.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:02.480433+0000", 2024-08-17T18:22:33.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:22:33.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:33.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:02.480440+0000", 2024-08-17T18:22:33.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2400000000000001e-06 2024-08-17T18:22:33.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:33.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:02.480454+0000", 2024-08-17T18:22:33.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3973e-05 2024-08-17T18:22:33.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:33.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:02.480462+0000", 2024-08-17T18:22:33.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2300000000000008e-06 2024-08-17T18:22:33.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:02.510495+0000", 2024-08-17T18:22:33.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.030032631000000001 2024-08-17T18:22:33.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:22:33.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:02.512703+0000", 2024-08-17T18:22:33.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022079579999999999 2024-08-17T18:22:33.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.293890+0000", 2024-08-17T18:22:33.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.78118731900000005 2024-08-17T18:22:33.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:22:33.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.297795+0000", 2024-08-17T18:22:33.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039044959999999999 2024-08-17T18:22:33.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.625210+0000", 2024-08-17T18:22:33.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3274149319999999 2024-08-17T18:22:33.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:33.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.629619+0000", 2024-08-17T18:22:33.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044094820000000002 2024-08-17T18:22:33.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:22:33.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.636707+0000", 2024-08-17T18:22:33.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070873990000000003 2024-08-17T18:22:33.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.665989+0000", 2024-08-17T18:22:33.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.029282077 2024-08-17T18:22:33.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:33.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.666332+0000", 2024-08-17T18:22:33.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034313799999999999 2024-08-17T18:22:33.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.668196+0000", 2024-08-17T18:22:33.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001863969 2024-08-17T18:22:33.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:33.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.669949+0000", 2024-08-17T18:22:33.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017531300000000001 2024-08-17T18:22:33.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:33.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.678647+0000", 2024-08-17T18:22:33.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0086985150000000004 2024-08-17T18:22:33.763 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.763 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:33.763 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.763 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:142338 57.12 57:4cde03a7:::benchmark_data_smithi136_389235_object26416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1650)", 2024-08-17T18:22:33.764 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:04.803270+0000", 2024-08-17T18:22:33.764 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 328.40836414799998, 2024-08-17T18:22:33.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5418123230000003, 2024-08-17T18:22:33.764 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:33.764 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:33.764 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:33.764 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:22:33.764 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:22:33.764 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 142338 2024-08-17T18:22:33.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.764 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:33.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:33.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.803270+0000", 2024-08-17T18:22:33.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:33.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.803270+0000", 2024-08-17T18:22:33.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:33.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.803269+0000", 2024-08-17T18:22:33.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:22:33.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:33.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.803277+0000", 2024-08-17T18:22:33.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9999999999999996e-06 2024-08-17T18:22:33.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:33.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.803292+0000", 2024-08-17T18:22:33.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5608e-05 2024-08-17T18:22:33.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:33.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.803300+0000", 2024-08-17T18:22:33.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5839999999999997e-06 2024-08-17T18:22:33.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.830271+0000", 2024-08-17T18:22:33.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.026971517 2024-08-17T18:22:33.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:22:33.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.832589+0000", 2024-08-17T18:22:33.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023174799999999998 2024-08-17T18:22:33.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:05.753306+0000", 2024-08-17T18:22:33.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.92071752200000001 2024-08-17T18:22:33.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:22:33.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:05.757990+0000", 2024-08-17T18:22:33.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004683235 2024-08-17T18:22:33.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.296145+0000", 2024-08-17T18:22:33.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5381556270000001 2024-08-17T18:22:33.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:33.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.299493+0000", 2024-08-17T18:22:33.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033477310000000001 2024-08-17T18:22:33.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:22:33.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.306116+0000", 2024-08-17T18:22:33.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006623253 2024-08-17T18:22:33.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:33.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.325542+0000", 2024-08-17T18:22:33.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019425575 2024-08-17T18:22:33.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.330732+0000", 2024-08-17T18:22:33.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051902720000000001 2024-08-17T18:22:33.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.332141+0000", 2024-08-17T18:22:33.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014088799999999999 2024-08-17T18:22:33.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:33.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.334377+0000", 2024-08-17T18:22:33.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022364239999999999 2024-08-17T18:22:33.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:33.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.345082+0000", 2024-08-17T18:22:33.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010705077 2024-08-17T18:22:33.775 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.775 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:33.775 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.776 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:143066 57.12 57:4e20cb33:::benchmark_data_smithi136_389235_object27144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1650)", 2024-08-17T18:22:33.776 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:07.229551+0000", 2024-08-17T18:22:33.776 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 325.98208364200002, 2024-08-17T18:22:33.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.970721556, 2024-08-17T18:22:33.776 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:33.776 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:33.777 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:33.777 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:22:33.777 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:22:33.777 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 143066 2024-08-17T18:22:33.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.777 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:33.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:33.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.229551+0000", 2024-08-17T18:22:33.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:33.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.229551+0000", 2024-08-17T18:22:33.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:33.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.229549+0000", 2024-08-17T18:22:33.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:22:33.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:33.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.229562+0000", 2024-08-17T18:22:33.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3594000000000001e-05 2024-08-17T18:22:33.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:33.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.229576+0000", 2024-08-17T18:22:33.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3481e-05 2024-08-17T18:22:33.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:33.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.229584+0000", 2024-08-17T18:22:33.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5539999999999998e-06 2024-08-17T18:22:33.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.345598+0000", 2024-08-17T18:22:33.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.116014002 2024-08-17T18:22:33.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:22:33.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.347905+0000", 2024-08-17T18:22:33.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023068730000000001 2024-08-17T18:22:33.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.239359+0000", 2024-08-17T18:22:33.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.89145372499999997 2024-08-17T18:22:33.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:22:33.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.243463+0000", 2024-08-17T18:22:33.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041039589999999999 2024-08-17T18:22:33.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.112003+0000", 2024-08-17T18:22:33.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.868540396 2024-08-17T18:22:33.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:33.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.116413+0000", 2024-08-17T18:22:33.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044100650000000003 2024-08-17T18:22:33.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:22:33.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.125278+0000", 2024-08-17T18:22:33.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0088651159999999993 2024-08-17T18:22:33.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.137166+0000", 2024-08-17T18:22:33.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011887246000000001 2024-08-17T18:22:33.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.138635+0000", 2024-08-17T18:22:33.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001469595 2024-08-17T18:22:33.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:33.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.186048+0000", 2024-08-17T18:22:33.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.047412965000000001 2024-08-17T18:22:33.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:33.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.188412+0000", 2024-08-17T18:22:33.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002364054 2024-08-17T18:22:33.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:33.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.200272+0000", 2024-08-17T18:22:33.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01186006 2024-08-17T18:22:33.793 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.793 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:33.793 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.794 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:143493 57.12 57:4dd25d03:::benchmark_data_smithi136_389235_object27571:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1650)", 2024-08-17T18:22:33.794 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:08.252789+0000", 2024-08-17T18:22:33.794 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 324.95884545600001, 2024-08-17T18:22:33.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0232178630000002, 2024-08-17T18:22:33.794 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:33.794 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:33.794 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:33.795 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:22:33.795 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:22:33.795 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 143493 2024-08-17T18:22:33.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.795 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:33.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:33.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.252789+0000", 2024-08-17T18:22:33.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:33.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.252789+0000", 2024-08-17T18:22:33.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:33.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.252787+0000", 2024-08-17T18:22:33.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:22:33.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:33.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.252795+0000", 2024-08-17T18:22:33.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7840000000000004e-06 2024-08-17T18:22:33.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:33.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.252808+0000", 2024-08-17T18:22:33.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.291e-05 2024-08-17T18:22:33.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:33.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.252814+0000", 2024-08-17T18:22:33.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2600000000000002e-06 2024-08-17T18:22:33.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.299118+0000", 2024-08-17T18:22:33.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.046304317999999997 2024-08-17T18:22:33.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:22:33.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.301303+0000", 2024-08-17T18:22:33.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002185144 2024-08-17T18:22:33.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.358653+0000", 2024-08-17T18:22:33.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0573500520000001 2024-08-17T18:22:33.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:22:33.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.362431+0000", 2024-08-17T18:22:33.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037779129999999999 2024-08-17T18:22:33.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:11.243257+0000", 2024-08-17T18:22:33.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8808254760000001 2024-08-17T18:22:33.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:33.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:11.246415+0000", 2024-08-17T18:22:33.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003157726 2024-08-17T18:22:33.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:22:33.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:11.253380+0000", 2024-08-17T18:22:33.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069659270000000002 2024-08-17T18:22:33.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:11.263911+0000", 2024-08-17T18:22:33.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010530891000000001 2024-08-17T18:22:33.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:11.264967+0000", 2024-08-17T18:22:33.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001055769 2024-08-17T18:22:33.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:33.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:11.265311+0000", 2024-08-17T18:22:33.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034373200000000002 2024-08-17T18:22:33.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:33.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:11.267008+0000", 2024-08-17T18:22:33.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001696779 2024-08-17T18:22:33.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:33.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:11.276007+0000", 2024-08-17T18:22:33.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0089990529999999999 2024-08-17T18:22:33.806 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.806 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:33.806 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.806 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:144015 57.12 57:4d3166e5:::benchmark_data_smithi136_389235_object28093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1650)", 2024-08-17T18:22:33.807 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:09.497595+0000", 2024-08-17T18:22:33.807 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 323.714038974, 2024-08-17T18:22:33.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0067860620000002, 2024-08-17T18:22:33.807 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:33.807 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:33.807 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:33.807 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:22:33.807 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:22:33.807 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 144015 2024-08-17T18:22:33.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.807 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:33.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:33.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.497595+0000", 2024-08-17T18:22:33.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:33.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.497595+0000", 2024-08-17T18:22:33.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:33.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.497594+0000", 2024-08-17T18:22:33.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:22:33.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:33.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.497599+0000", 2024-08-17T18:22:33.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.096e-06 2024-08-17T18:22:33.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:33.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.497610+0000", 2024-08-17T18:22:33.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1107000000000001e-05 2024-08-17T18:22:33.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:33.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.497616+0000", 2024-08-17T18:22:33.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5459999999999999e-06 2024-08-17T18:22:33.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.599005+0000", 2024-08-17T18:22:33.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101389497 2024-08-17T18:22:33.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:22:33.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.600888+0000", 2024-08-17T18:22:33.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001883006 2024-08-17T18:22:33.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.353923+0000", 2024-08-17T18:22:33.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.75303402200000003 2024-08-17T18:22:33.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:22:33.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.359101+0000", 2024-08-17T18:22:33.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005178958 2024-08-17T18:22:33.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.408708+0000", 2024-08-17T18:22:33.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0496060800000002 2024-08-17T18:22:33.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:33.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.413557+0000", 2024-08-17T18:22:33.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048495170000000002 2024-08-17T18:22:33.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:22:33.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.423699+0000", 2024-08-17T18:22:33.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010141554000000001 2024-08-17T18:22:33.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:33.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.453884+0000", 2024-08-17T18:22:33.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.030185025000000001 2024-08-17T18:22:33.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.489415+0000", 2024-08-17T18:22:33.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035530897999999998 2024-08-17T18:22:33.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.491456+0000", 2024-08-17T18:22:33.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020412389999999998 2024-08-17T18:22:33.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:33.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.493673+0000", 2024-08-17T18:22:33.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022170179999999999 2024-08-17T18:22:33.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:33.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.504381+0000", 2024-08-17T18:22:33.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010708611999999999 2024-08-17T18:22:33.815 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.815 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:33.815 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.816 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:145041 57.12 57:4cef9719:::benchmark_data_smithi136_389235_object29119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1651)", 2024-08-17T18:22:33.816 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:12.526739+0000", 2024-08-17T18:22:33.816 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 320.68489569899998, 2024-08-17T18:22:33.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3206357639999999, 2024-08-17T18:22:33.816 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:33.816 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:33.816 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:33.816 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:22:33.816 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:22:33.816 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 145041 2024-08-17T18:22:33.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.816 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:33.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:33.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.526739+0000", 2024-08-17T18:22:33.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:33.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.526739+0000", 2024-08-17T18:22:33.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:33.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.526737+0000", 2024-08-17T18:22:33.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:22:33.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:33.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.526745+0000", 2024-08-17T18:22:33.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0960000000000001e-06 2024-08-17T18:22:33.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:33.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.526762+0000", 2024-08-17T18:22:33.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6514000000000001e-05 2024-08-17T18:22:33.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:33.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.526770+0000", 2024-08-17T18:22:33.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7849999999999997e-06 2024-08-17T18:22:33.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.575365+0000", 2024-08-17T18:22:33.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.048594641000000001 2024-08-17T18:22:33.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:22:33.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.577274+0000", 2024-08-17T18:22:33.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001909371 2024-08-17T18:22:33.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.206156+0000", 2024-08-17T18:22:33.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.62888149900000001 2024-08-17T18:22:33.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:22:33.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.216312+0000", 2024-08-17T18:22:33.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010156263 2024-08-17T18:22:33.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.814900+0000", 2024-08-17T18:22:33.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5985881389999999 2024-08-17T18:22:33.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:33.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.818177+0000", 2024-08-17T18:22:33.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032767489999999998 2024-08-17T18:22:33.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:22:33.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.825676+0000", 2024-08-17T18:22:33.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0074987159999999999 2024-08-17T18:22:33.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.833778+0000", 2024-08-17T18:22:33.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0081025070000000001 2024-08-17T18:22:33.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.834861+0000", 2024-08-17T18:22:33.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010832140000000001 2024-08-17T18:22:33.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:33.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.835223+0000", 2024-08-17T18:22:33.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036148499999999999 2024-08-17T18:22:33.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:33.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.837322+0000", 2024-08-17T18:22:33.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002099212 2024-08-17T18:22:33.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:33.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.847374+0000", 2024-08-17T18:22:33.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010052195999999999 2024-08-17T18:22:33.825 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.825 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:33.825 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.825 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:145558 57.12 57:4e414c7c:::benchmark_data_smithi136_389235_object29636:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1651)", 2024-08-17T18:22:33.825 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:13.886158+0000", 2024-08-17T18:22:33.825 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 319.32547640799999, 2024-08-17T18:22:33.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5611586079999999, 2024-08-17T18:22:33.825 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:33.826 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:33.826 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:33.826 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:22:33.826 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:22:33.826 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 145558 2024-08-17T18:22:33.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.826 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:33.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:33.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.886158+0000", 2024-08-17T18:22:33.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:33.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.886158+0000", 2024-08-17T18:22:33.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:33.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.886157+0000", 2024-08-17T18:22:33.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:22:33.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:33.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.886162+0000", 2024-08-17T18:22:33.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7880000000000003e-06 2024-08-17T18:22:33.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:33.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.886174+0000", 2024-08-17T18:22:33.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1508999999999999e-05 2024-08-17T18:22:33.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:33.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.886180+0000", 2024-08-17T18:22:33.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8089999999999998e-06 2024-08-17T18:22:33.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.902475+0000", 2024-08-17T18:22:33.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016294869999999999 2024-08-17T18:22:33.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:22:33.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.904366+0000", 2024-08-17T18:22:33.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001891083 2024-08-17T18:22:33.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.790935+0000", 2024-08-17T18:22:33.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.88656959800000001 2024-08-17T18:22:33.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:22:33.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.795601+0000", 2024-08-17T18:22:33.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046654030000000003 2024-08-17T18:22:33.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:16.393619+0000", 2024-08-17T18:22:33.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.598018199 2024-08-17T18:22:33.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:33.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:16.397485+0000", 2024-08-17T18:22:33.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003866455 2024-08-17T18:22:33.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:22:33.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:16.404509+0000", 2024-08-17T18:22:33.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070232319999999999 2024-08-17T18:22:33.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:33.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:16.431091+0000", 2024-08-17T18:22:33.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.026582179000000001 2024-08-17T18:22:33.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:16.432698+0000", 2024-08-17T18:22:33.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001607107 2024-08-17T18:22:33.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:16.434119+0000", 2024-08-17T18:22:33.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001420775 2024-08-17T18:22:33.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:33.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:16.436158+0000", 2024-08-17T18:22:33.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002039482 2024-08-17T18:22:33.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:33.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:16.447317+0000", 2024-08-17T18:22:33.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011158404 2024-08-17T18:22:33.833 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.833 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:33.833 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.833 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:98334 58.5 58:ad0a3cf8:::benchmark_data_smithi136_442124_object98333:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1719)", 2024-08-17T18:22:33.833 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:11.776143+0000", 2024-08-17T18:22:33.833 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.435491385999999, 2024-08-17T18:22:33.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.229128743, 2024-08-17T18:22:33.834 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:33.834 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:33.834 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:33.834 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:22:33.834 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:22:33.834 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 98334 2024-08-17T18:22:33.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.834 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:33.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:33.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.776143+0000", 2024-08-17T18:22:33.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:33.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.776143+0000", 2024-08-17T18:22:33.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:33.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.776142+0000", 2024-08-17T18:22:33.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:22:33.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:33.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.776191+0000", 2024-08-17T18:22:33.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9236000000000001e-05 2024-08-17T18:22:33.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:33.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.776202+0000", 2024-08-17T18:22:33.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0672e-05 2024-08-17T18:22:33.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:33.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.776207+0000", 2024-08-17T18:22:33.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.967e-06 2024-08-17T18:22:33.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.776254+0000", 2024-08-17T18:22:33.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7333999999999998e-05 2024-08-17T18:22:33.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:33.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.776412+0000", 2024-08-17T18:22:33.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015820400000000001 2024-08-17T18:22:33.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:22:33.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.776585+0000", 2024-08-17T18:22:33.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017275999999999999 2024-08-17T18:22:33.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:33.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.785732+0000", 2024-08-17T18:22:33.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0091473490000000008 2024-08-17T18:22:33.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.792529+0000", 2024-08-17T18:22:33.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0067964030000000003 2024-08-17T18:22:33.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:12.005174+0000", 2024-08-17T18:22:33.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.21264483200000001 2024-08-17T18:22:33.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:33.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:12.005245+0000", 2024-08-17T18:22:33.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1373999999999998e-05 2024-08-17T18:22:33.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:33.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:12.005272+0000", 2024-08-17T18:22:33.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6591999999999999e-05 2024-08-17T18:22:33.842 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.842 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:33.842 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.842 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:98335 58.5 58:a0187004:::benchmark_data_smithi136_442124_object98334:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1719)", 2024-08-17T18:22:33.842 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:11.776528+0000", 2024-08-17T18:22:33.842 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.435105845999999, 2024-08-17T18:22:33.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.22945246999999999, 2024-08-17T18:22:33.843 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:33.843 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:33.843 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:33.843 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:22:33.843 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:22:33.843 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 98335 2024-08-17T18:22:33.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.843 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:33.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:33.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.776528+0000", 2024-08-17T18:22:33.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:33.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.776528+0000", 2024-08-17T18:22:33.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:33.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.776528+0000", 2024-08-17T18:22:33.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:22:33.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:33.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.776595+0000", 2024-08-17T18:22:33.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7657999999999995e-05 2024-08-17T18:22:33.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:33.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.776603+0000", 2024-08-17T18:22:33.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1439999999999997e-06 2024-08-17T18:22:33.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:33.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.776606+0000", 2024-08-17T18:22:33.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8060000000000001e-06 2024-08-17T18:22:33.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.779370+0000", 2024-08-17T18:22:33.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027631029999999998 2024-08-17T18:22:33.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:33.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.779500+0000", 2024-08-17T18:22:33.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013019 2024-08-17T18:22:33.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:22:33.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.779657+0000", 2024-08-17T18:22:33.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000157532 2024-08-17T18:22:33.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:33.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.785741+0000", 2024-08-17T18:22:33.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060833069999999996 2024-08-17T18:22:33.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.792553+0000", 2024-08-17T18:22:33.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068127259999999999 2024-08-17T18:22:33.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:12.005884+0000", 2024-08-17T18:22:33.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.21333020499999999 2024-08-17T18:22:33.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:33.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:12.005944+0000", 2024-08-17T18:22:33.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0735e-05 2024-08-17T18:22:33.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:33.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:12.005981+0000", 2024-08-17T18:22:33.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6701e-05 2024-08-17T18:22:33.848 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.849 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:33.849 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.849 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:98339 58.5 58:a66d1591:::benchmark_data_smithi136_442124_object98338:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1719)", 2024-08-17T18:22:33.849 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:11.782674+0000", 2024-08-17T18:22:33.849 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.428960012000001, 2024-08-17T18:22:33.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.236340036, 2024-08-17T18:22:33.849 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:33.849 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:33.849 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:33.849 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:22:33.849 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:22:33.849 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 98339 2024-08-17T18:22:33.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.850 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:33.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:33.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.782674+0000", 2024-08-17T18:22:33.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:33.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.782674+0000", 2024-08-17T18:22:33.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:33.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.782672+0000", 2024-08-17T18:22:33.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:22:33.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:33.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.782718+0000", 2024-08-17T18:22:33.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5500999999999997e-05 2024-08-17T18:22:33.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:33.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.782732+0000", 2024-08-17T18:22:33.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4619e-05 2024-08-17T18:22:33.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:33.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.782739+0000", 2024-08-17T18:22:33.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0979999999999996e-06 2024-08-17T18:22:33.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.782828+0000", 2024-08-17T18:22:33.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8536999999999996e-05 2024-08-17T18:22:33.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:33.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.782947+0000", 2024-08-17T18:22:33.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011874700000000001 2024-08-17T18:22:33.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:22:33.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.783054+0000", 2024-08-17T18:22:33.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000107753 2024-08-17T18:22:33.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:33.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.785828+0000", 2024-08-17T18:22:33.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027738400000000001 2024-08-17T18:22:33.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.792580+0000", 2024-08-17T18:22:33.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0067515400000000003 2024-08-17T18:22:33.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:12.018930+0000", 2024-08-17T18:22:33.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.226349773 2024-08-17T18:22:33.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:33.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:12.018973+0000", 2024-08-17T18:22:33.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3662999999999999e-05 2024-08-17T18:22:33.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:33.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:12.019014+0000", 2024-08-17T18:22:33.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1034e-05 2024-08-17T18:22:33.856 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.856 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:33.856 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.856 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:98352 58.5 58:a3e31360:::benchmark_data_smithi136_442124_object98351:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1719)", 2024-08-17T18:22:33.856 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:11.798025+0000", 2024-08-17T18:22:33.856 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.413608868000001, 2024-08-17T18:22:33.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.222673916, 2024-08-17T18:22:33.857 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:33.857 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:33.857 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:33.857 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:22:33.857 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:22:33.857 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 98352 2024-08-17T18:22:33.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.857 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:33.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:33.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.798025+0000", 2024-08-17T18:22:33.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:33.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.798025+0000", 2024-08-17T18:22:33.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:33.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:33.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.798025+0000", 2024-08-17T18:22:33.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:22:33.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:33.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.798097+0000", 2024-08-17T18:22:33.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2412999999999999e-05 2024-08-17T18:22:33.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:33.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.798106+0000", 2024-08-17T18:22:33.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6349999999999995e-06 2024-08-17T18:22:33.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:33.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.798112+0000", 2024-08-17T18:22:33.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1109999999999996e-06 2024-08-17T18:22:33.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:33.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.798169+0000", 2024-08-17T18:22:33.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7197999999999997e-05 2024-08-17T18:22:33.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:33.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.798291+0000", 2024-08-17T18:22:33.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122302 2024-08-17T18:22:33.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:22:33.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.798389+0000", 2024-08-17T18:22:33.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7437999999999998e-05 2024-08-17T18:22:33.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:33.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.800955+0000", 2024-08-17T18:22:33.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025665190000000002 2024-08-17T18:22:33.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:11.801319+0000", 2024-08-17T18:22:33.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036412 2024-08-17T18:22:33.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:33.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:12.020645+0000", 2024-08-17T18:22:33.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.21932553799999999 2024-08-17T18:22:33.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:33.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:12.020677+0000", 2024-08-17T18:22:33.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1832999999999998e-05 2024-08-17T18:22:33.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:33.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:33.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:33.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:12.020699+0000", 2024-08-17T18:22:33.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2617999999999999e-05 2024-08-17T18:22:33.863 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.863 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:33.863 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.863 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:33.863 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:33.863 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:22:33.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:33.865 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:33.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:33.865 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:22:33.965 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:34.009 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:34.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:34.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:33 smithi165 ceph-mon[105517]: pgmap v3373: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+clean+remapped+scrubbing, 1 active+recovering+undersized+remapped, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 46 MiB/s wr, 773 op/s; 42238/337815 objects degraded (12.503%); 14499/337815 objects misplaced (4.292%); 13 MiB/s, 210 objects/s recovering 2024-08-17T18:22:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:33 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/199887185' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:22:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:33 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:22:34.111 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:34.138 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:34.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:34.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:34.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:33 smithi136 ceph-mon[109861]: pgmap v3373: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+clean+remapped+scrubbing, 1 active+recovering+undersized+remapped, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 46 MiB/s wr, 773 op/s; 42238/337815 objects degraded (12.503%); 14499/337815 objects misplaced (4.292%); 13 MiB/s, 210 objects/s recovering 2024-08-17T18:22:34.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:33 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/199887185' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:22:34.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:33 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:22:34.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:33 smithi136 ceph-mon[115078]: pgmap v3373: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+clean+remapped+scrubbing, 1 active+recovering+undersized+remapped, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 46 MiB/s wr, 773 op/s; 42238/337815 objects degraded (12.503%); 14499/337815 objects misplaced (4.292%); 13 MiB/s, 210 objects/s recovering 2024-08-17T18:22:34.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:33 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/199887185' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:22:34.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:33 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:22:34.240 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:34.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:34.271 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:34.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:34.372 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:34.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:34.411 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:34.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:34.511 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:22:34.098+0000 7fb1a31ba640 -1 osd.6 1711 log_to_monitors true 2024-08-17T18:22:34.512 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:34.537 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:34.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:34.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:34.639 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:34.660 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:34.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:34.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:34.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:34.799 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:34.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:34.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:34.837 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:34 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:22:34.837 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:34 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:22:34.837 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:34 smithi165 ceph-mon[105517]: osdmap e1722: 8 total, 7 up, 8 in 2024-08-17T18:22:34.837 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:34 smithi165 ceph-mon[105517]: from='osd.6 [v2:172.21.15.165:6816/2775283828,v1:172.21.15.165:6817/2775283828]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-08-17T18:22:34.837 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:34 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 42238/337815 objects degraded (12.503%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:34.900 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:34.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:34.928 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:34.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:35.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:35.058 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:35.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:35.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:35.077 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-17T18:22:35.145 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:22:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:22:34.835+0000 7fb196f5c640 -1 osd.6 1723 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T18:22:35.161 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:35.172 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:34 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:22:35.172 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:34 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:22:35.172 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:34 smithi136 ceph-mon[109861]: osdmap e1722: 8 total, 7 up, 8 in 2024-08-17T18:22:35.172 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:34 smithi136 ceph-mon[109861]: from='osd.6 [v2:172.21.15.165:6816/2775283828,v1:172.21.15.165:6817/2775283828]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-08-17T18:22:35.172 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:34 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 42238/337815 objects degraded (12.503%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:35.172 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:34 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:22:35.173 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:34 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:22:35.173 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:34 smithi136 ceph-mon[115078]: osdmap e1722: 8 total, 7 up, 8 in 2024-08-17T18:22:35.173 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:34 smithi136 ceph-mon[115078]: from='osd.6 [v2:172.21.15.165:6816/2775283828,v1:172.21.15.165:6817/2775283828]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-08-17T18:22:35.173 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:34 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 42238/337815 objects degraded (12.503%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:35.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:35.203 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:35.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:35.304 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:35.346 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:35.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:35.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:35.448 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:35.476 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:35.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:35.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:35.578 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:35.613 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:22:35.613 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:22:35.613 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-08-17T18:22:35.613 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:22:35.614 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:35.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:35.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:35.716 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:35.755 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:35.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:35.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:35.857 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:35.866 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:22:35.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:35.879 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:35.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:35.980 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:36.003 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:36.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:36.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:36.105 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:36.128 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:36.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:36.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:36.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:35 smithi136 ceph-mon[115078]: pgmap v3375: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 115 B/s rd, 43 MiB/s wr, 786 op/s; 42277/338448 objects degraded (12.491%); 14376/338448 objects misplaced (4.248%); 15 MiB/s, 243 objects/s recovering 2024-08-17T18:22:36.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:35 smithi136 ceph-mon[115078]: from='osd.6 [v2:172.21.15.165:6816/2775283828,v1:172.21.15.165:6817/2775283828]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-08-17T18:22:36.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:35 smithi136 ceph-mon[115078]: osdmap e1723: 8 total, 7 up, 8 in 2024-08-17T18:22:36.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:35 smithi136 ceph-mon[115078]: from='osd.6 [v2:172.21.15.165:6816/2775283828,v1:172.21.15.165:6817/2775283828]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-08-17T18:22:36.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:35 smithi136 ceph-mon[109861]: pgmap v3375: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 115 B/s rd, 43 MiB/s wr, 786 op/s; 42277/338448 objects degraded (12.491%); 14376/338448 objects misplaced (4.248%); 15 MiB/s, 243 objects/s recovering 2024-08-17T18:22:36.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:35 smithi136 ceph-mon[109861]: from='osd.6 [v2:172.21.15.165:6816/2775283828,v1:172.21.15.165:6817/2775283828]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-08-17T18:22:36.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:35 smithi136 ceph-mon[109861]: osdmap e1723: 8 total, 7 up, 8 in 2024-08-17T18:22:36.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:35 smithi136 ceph-mon[109861]: from='osd.6 [v2:172.21.15.165:6816/2775283828,v1:172.21.15.165:6817/2775283828]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-08-17T18:22:36.230 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:36.256 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:36.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:36.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:35 smithi165 ceph-mon[105517]: pgmap v3375: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 12 active+undersized+degraded, 13 active+undersized, 63 active+clean; 6.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 115 B/s rd, 43 MiB/s wr, 786 op/s; 42277/338448 objects degraded (12.491%); 14376/338448 objects misplaced (4.248%); 15 MiB/s, 243 objects/s recovering 2024-08-17T18:22:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:35 smithi165 ceph-mon[105517]: from='osd.6 [v2:172.21.15.165:6816/2775283828,v1:172.21.15.165:6817/2775283828]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-08-17T18:22:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:35 smithi165 ceph-mon[105517]: osdmap e1723: 8 total, 7 up, 8 in 2024-08-17T18:22:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:35 smithi165 ceph-mon[105517]: from='osd.6 [v2:172.21.15.165:6816/2775283828,v1:172.21.15.165:6817/2775283828]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]} : dispatch 2024-08-17T18:22:36.346 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 2 1 2024-08-17T18:22:36.359 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:22:36.381 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:36.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:36.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:22:36.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:36.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:36.513 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:36.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:36.614 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:22:36.637 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:36.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:36.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:22:36.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:22:36.761 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:36.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:36.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:22:36.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:36.885 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:36.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:36.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:36.987 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:37.010 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:37.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:37.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:36 smithi165 ceph-mon[105517]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T18:22:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:36 smithi165 ceph-mon[105517]: osd.6 [v2:172.21.15.165:6816/2775283828,v1:172.21.15.165:6817/2775283828] boot 2024-08-17T18:22:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:36 smithi165 ceph-mon[105517]: osdmap e1724: 8 total, 8 up, 8 in 2024-08-17T18:22:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-17T18:22:37.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:22:37.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:22:37.111 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:37.133 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:37.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:37.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:37.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:36 smithi136 ceph-mon[109861]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T18:22:37.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:36 smithi136 ceph-mon[109861]: osd.6 [v2:172.21.15.165:6816/2775283828,v1:172.21.15.165:6817/2775283828] boot 2024-08-17T18:22:37.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:36 smithi136 ceph-mon[109861]: osdmap e1724: 8 total, 8 up, 8 in 2024-08-17T18:22:37.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-17T18:22:37.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:22:37.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:22:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:36 smithi136 ceph-mon[115078]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T18:22:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:36 smithi136 ceph-mon[115078]: osd.6 [v2:172.21.15.165:6816/2775283828,v1:172.21.15.165:6817/2775283828] boot 2024-08-17T18:22:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:36 smithi136 ceph-mon[115078]: osdmap e1724: 8 total, 8 up, 8 in 2024-08-17T18:22:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-17T18:22:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:22:37.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:22:37.235 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:37.257 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:37.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:37.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:37.359 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:37.381 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:37.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:37.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:37.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:37.505 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:37.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:37.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:37.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:22:37.628 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:37.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:37.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:22:37.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:37.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:37.753 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:37.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:37.831 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:22:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:22:37] "GET /metrics HTTP/1.1" 200 36631 "" "Prometheus/2.51.0" 2024-08-17T18:22:37.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:37.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:37.892 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:37.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:37.993 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:38.029 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:38.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:38.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:38.130 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:38.159 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:38.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:38.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:38.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:37 smithi136 ceph-mon[109861]: pgmap v3378: 99 pgs: 3 peering, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 10 active+undersized+degraded, 12 active+undersized, 63 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 170 B/s rd, 22 MiB/s wr, 659 op/s; 28071/335259 objects degraded (8.373%); 14376/335259 objects misplaced (4.288%); 14 MiB/s, 222 objects/s recovering 2024-08-17T18:22:38.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:37 smithi136 ceph-mon[109861]: osdmap e1725: 8 total, 8 up, 8 in 2024-08-17T18:22:38.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:37 smithi136 ceph-mon[115078]: pgmap v3378: 99 pgs: 3 peering, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 10 active+undersized+degraded, 12 active+undersized, 63 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 170 B/s rd, 22 MiB/s wr, 659 op/s; 28071/335259 objects degraded (8.373%); 14376/335259 objects misplaced (4.288%); 14 MiB/s, 222 objects/s recovering 2024-08-17T18:22:38.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:37 smithi136 ceph-mon[115078]: osdmap e1725: 8 total, 8 up, 8 in 2024-08-17T18:22:38.260 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:38.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:37 smithi165 ceph-mon[105517]: pgmap v3378: 99 pgs: 3 peering, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+clean+remapped+scrubbing, 7 active+clean+remapped, 10 active+undersized+degraded, 12 active+undersized, 63 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 170 B/s rd, 22 MiB/s wr, 659 op/s; 28071/335259 objects degraded (8.373%); 14376/335259 objects misplaced (4.288%); 14 MiB/s, 222 objects/s recovering 2024-08-17T18:22:38.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:37 smithi165 ceph-mon[105517]: osdmap e1725: 8 total, 8 up, 8 in 2024-08-17T18:22:38.290 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:38.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:38.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:38.392 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:38.417 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:38.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:38.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:38.518 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:22:38.540 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:38.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:38.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:22:38.642 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:38.667 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:38.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:38.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:38.769 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:38.799 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:38.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:38.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:38.901 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:38.929 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:38.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:38.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:39.031 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:39.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:39.064 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:39.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:39.166 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:39.182 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:38 smithi165 ceph-mon[105517]: osdmap e1726: 8 total, 8 up, 8 in 2024-08-17T18:22:39.201 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:38 smithi136 ceph-mon[109861]: osdmap e1726: 8 total, 8 up, 8 in 2024-08-17T18:22:39.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:38 smithi136 ceph-mon[115078]: osdmap e1726: 8 total, 8 up, 8 in 2024-08-17T18:22:39.205 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:39.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:39.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:39.217 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:22:39.217 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:22:39.217 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-08-17T18:22:39.217 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:22:39.307 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:39.349 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:39.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:39.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:39.451 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:39.478 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:39.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:39.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:39.581 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:39.601 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:39.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:39.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:39.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:39.732 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:39.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:39.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:39.834 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:39.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:39 smithi165 ceph-mon[105517]: pgmap v3381: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 3 remapped+peering, 5 peering, 1 active+clean+remapped+scrubbing, 5 active+clean+remapped, 3 active+undersized+degraded, 8 active+undersized, 71 active+clean; 6.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 368 KiB/s wr, 686 op/s; 6952/330345 objects degraded (2.104%); 12586/330345 objects misplaced (3.810%); 25 MiB/s, 394 objects/s recovering 2024-08-17T18:22:39.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:39 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 6952/330345 objects degraded (2.104%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:39.863 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:39.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:39.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:39.946 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-17T18:22:39.965 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:40.011 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:40.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:40.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:40.113 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:40.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:40.152 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:40.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:40.171 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:39 smithi136 ceph-mon[109861]: pgmap v3381: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 3 remapped+peering, 5 peering, 1 active+clean+remapped+scrubbing, 5 active+clean+remapped, 3 active+undersized+degraded, 8 active+undersized, 71 active+clean; 6.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 368 KiB/s wr, 686 op/s; 6952/330345 objects degraded (2.104%); 12586/330345 objects misplaced (3.810%); 25 MiB/s, 394 objects/s recovering 2024-08-17T18:22:40.171 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:39 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 6952/330345 objects degraded (2.104%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:40.172 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:39 smithi136 ceph-mon[115078]: pgmap v3381: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 3 remapped+peering, 5 peering, 1 active+clean+remapped+scrubbing, 5 active+clean+remapped, 3 active+undersized+degraded, 8 active+undersized, 71 active+clean; 6.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 368 KiB/s wr, 686 op/s; 6952/330345 objects degraded (2.104%); 12586/330345 objects misplaced (3.810%); 25 MiB/s, 394 objects/s recovering 2024-08-17T18:22:40.172 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:39 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 6952/330345 objects degraded (2.104%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:40.232 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:22:40.253 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:40.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:40.277 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:40.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:40.379 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:40.411 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:40.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:40.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:40.513 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:40.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:40.539 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:40.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:40.640 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:40.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:40.681 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:40.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:40.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:40.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:40.813 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:40.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:40.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:40.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:40.946 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:40.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:41.047 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:22:41.085 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:41.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:41.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:22:41.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:41.223 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:41.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:41.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:41.325 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:41.360 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:41.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:41.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:41.463 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:41.510 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:41.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:41.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:41.612 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:22:41.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:41.660 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:41.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:22:41.761 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:41.808 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:41.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:41.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:41.910 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:22:41.954 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:41.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:41.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:22:42.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:41 smithi165 ceph-mon[105517]: pgmap v3382: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 3 remapped+peering, 5 peering, 1 active+undersized+degraded, 87 active+clean; 6.7 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 255 KiB/s wr, 521 op/s; 6944/329556 objects degraded (2.107%); 12586/329556 objects misplaced (3.819%); 17 MiB/s, 271 objects/s recovering 2024-08-17T18:22:42.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:22:42.086 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:42.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:42.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:22:42.188 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:42.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:41 smithi136 ceph-mon[109861]: pgmap v3382: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 3 remapped+peering, 5 peering, 1 active+undersized+degraded, 87 active+clean; 6.7 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 255 KiB/s wr, 521 op/s; 6944/329556 objects degraded (2.107%); 12586/329556 objects misplaced (3.819%); 17 MiB/s, 271 objects/s recovering 2024-08-17T18:22:42.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:41 smithi136 ceph-mon[115078]: pgmap v3382: 99 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 3 remapped+peering, 5 peering, 1 active+undersized+degraded, 87 active+clean; 6.7 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 255 KiB/s wr, 521 op/s; 6944/329556 objects degraded (2.107%); 12586/329556 objects misplaced (3.819%); 17 MiB/s, 271 objects/s recovering 2024-08-17T18:22:42.225 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:42.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:42.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:42.236 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:22:42.236 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:22:42.236 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-08-17T18:22:42.236 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-08-17T18:22:42.236 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:22:42.327 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:22:42.362 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:42.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:42.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:22:42.464 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:22:42.503 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:42.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:42.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:22:42.605 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:42.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:42.656 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:42.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:42.757 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:42.784 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:42.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:42.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:42.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:42.916 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:42.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:42.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:42.998 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-17T18:22:43.019 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:43.065 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:43.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:43.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:43.170 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:43.215 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:43.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:43.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:43.299 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:22:43.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:43.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:43.365 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:43.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:43.466 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:43.499 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:43.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:43.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:43.601 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:43.623 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:43.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:43.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:43.724 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:43.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:43.763 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:43.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:43.864 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:43.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:43.904 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:43.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:43.977 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:43 smithi165 ceph-mon[105517]: pgmap v3383: 99 pgs: 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 remapped+peering, 1 peering, 89 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 649 op/s; 14481/324237 objects degraded (4.466%); 12124/324237 objects misplaced (3.739%); 18 MiB/s, 293 objects/s recovering 2024-08-17T18:22:44.005 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:44.039 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:44.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:44.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:44.141 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:44.187 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:44.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:44.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:44.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:43 smithi136 ceph-mon[109861]: pgmap v3383: 99 pgs: 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 remapped+peering, 1 peering, 89 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 649 op/s; 14481/324237 objects degraded (4.466%); 12124/324237 objects misplaced (3.739%); 18 MiB/s, 293 objects/s recovering 2024-08-17T18:22:44.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:43 smithi136 ceph-mon[115078]: pgmap v3383: 99 pgs: 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 remapped+peering, 1 peering, 89 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 649 op/s; 14481/324237 objects degraded (4.466%); 12124/324237 objects misplaced (3.739%); 18 MiB/s, 293 objects/s recovering 2024-08-17T18:22:44.290 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:44.327 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:44.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:44.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:44.428 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:44.465 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:44.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:44.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:44.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:44.595 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:44.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:44.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:44.697 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:44.719 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:44.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:44.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:44.821 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:44.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:44.854 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:44.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:44.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:44.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:44.989 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:44.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:45.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:45.121 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:45.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:45.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:45.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:44 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 14481/324237 objects degraded (4.466%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:45.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:44 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 14481/324237 objects degraded (4.466%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:45.222 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:45.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:44 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 14481/324237 objects degraded (4.466%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:45.261 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:45.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:45.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:45.363 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:45.392 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:45.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:45.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:45.494 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:22:45.518 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:45.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:45.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:22:45.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:45.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:45.656 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:45.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:45.758 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:45.796 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:45.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:45.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:45.898 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:45.933 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:45.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:45.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:46.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:45 smithi165 ceph-mon[105517]: pgmap v3384: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 90 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 585 op/s; 17477/321201 objects degraded (5.441%); 12015/321201 objects misplaced (3.741%); 16 MiB/s, 261 objects/s recovering 2024-08-17T18:22:46.035 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:46.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:46.071 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:46.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:46.074 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:22:46.074 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T18:22:46.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T18:22:46.074 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:22:46.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.075 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:122081 57.a 57:53ce68b1:::benchmark_data_smithi136_389235_object6159:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1639)", 2024-08-17T18:22:46.075 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:54.535181+0000", 2024-08-17T18:22:46.075 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 411.53622239600003, 2024-08-17T18:22:46.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9832697580000005, 2024-08-17T18:22:46.075 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:46.075 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:46.075 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:46.075 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:22:46.075 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:22:46.075 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 122081 2024-08-17T18:22:46.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.075 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:46.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:46.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.535181+0000", 2024-08-17T18:22:46.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:46.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.535181+0000", 2024-08-17T18:22:46.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:46.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.535179+0000", 2024-08-17T18:22:46.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999976 2024-08-17T18:22:46.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:46.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.535190+0000", 2024-08-17T18:22:46.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0815000000000001e-05 2024-08-17T18:22:46.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:46.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.535205+0000", 2024-08-17T18:22:46.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4994000000000001e-05 2024-08-17T18:22:46.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:46.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.535213+0000", 2024-08-17T18:22:46.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0700000000000007e-06 2024-08-17T18:22:46.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.535305+0000", 2024-08-17T18:22:46.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1905999999999997e-05 2024-08-17T18:22:46.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:22:46.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:54.535317+0000", 2024-08-17T18:22:46.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2707e-05 2024-08-17T18:22:46.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.490000+0000", 2024-08-17T18:22:46.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9546827809999998 2024-08-17T18:22:46.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:46.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.490110+0000", 2024-08-17T18:22:46.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109694 2024-08-17T18:22:46.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4,7", 2024-08-17T18:22:46.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.490197+0000", 2024-08-17T18:22:46.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6959999999999994e-05 2024-08-17T18:22:46.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.492688+0000", 2024-08-17T18:22:46.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024907750000000002 2024-08-17T18:22:46.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:46.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.492711+0000", 2024-08-17T18:22:46.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3742000000000001e-05 2024-08-17T18:22:46.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.492744+0000", 2024-08-17T18:22:46.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2737999999999997e-05 2024-08-17T18:22:46.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.518347+0000", 2024-08-17T18:22:46.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.025602666999999999 2024-08-17T18:22:46.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:46.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.518375+0000", 2024-08-17T18:22:46.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8277e-05 2024-08-17T18:22:46.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:46.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.518451+0000", 2024-08-17T18:22:46.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6106000000000006e-05 2024-08-17T18:22:46.081 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.081 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:46.081 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.082 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:122098 57.a 57:51bdebe7:::benchmark_data_smithi136_389235_object6176:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1641)", 2024-08-17T18:22:46.082 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:15:57.114235+0000", 2024-08-17T18:22:46.082 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 408.95716877900003, 2024-08-17T18:22:46.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4821008249999998, 2024-08-17T18:22:46.082 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:46.082 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:46.082 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:46.082 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:22:46.082 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:22:46.082 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 122098 2024-08-17T18:22:46.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.082 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:46.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:46.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.114235+0000", 2024-08-17T18:22:46.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:46.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.114235+0000", 2024-08-17T18:22:46.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:46.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.114234+0000", 2024-08-17T18:22:46.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:22:46.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:46.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.114240+0000", 2024-08-17T18:22:46.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6050000000000003e-06 2024-08-17T18:22:46.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:46.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.114311+0000", 2024-08-17T18:22:46.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0778999999999994e-05 2024-08-17T18:22:46.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:46.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.114318+0000", 2024-08-17T18:22:46.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2019999999999998e-06 2024-08-17T18:22:46.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.114378+0000", 2024-08-17T18:22:46.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9960999999999997e-05 2024-08-17T18:22:46.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:22:46.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:15:57.114390+0000", 2024-08-17T18:22:46.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2199e-05 2024-08-17T18:22:46.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.492765+0000", 2024-08-17T18:22:46.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.378374617 2024-08-17T18:22:46.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:46.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.492902+0000", 2024-08-17T18:22:46.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000137298 2024-08-17T18:22:46.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4,7", 2024-08-17T18:22:46.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.493002+0000", 2024-08-17T18:22:46.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9696000000000002e-05 2024-08-17T18:22:46.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.495054+0000", 2024-08-17T18:22:46.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020514890000000001 2024-08-17T18:22:46.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.495096+0000", 2024-08-17T18:22:46.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2559999999999999e-05 2024-08-17T18:22:46.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:46.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.495240+0000", 2024-08-17T18:22:46.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000144375 2024-08-17T18:22:46.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.596152+0000", 2024-08-17T18:22:46.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100911139 2024-08-17T18:22:46.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:46.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.596213+0000", 2024-08-17T18:22:46.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1654000000000003e-05 2024-08-17T18:22:46.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:46.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:00.596336+0000", 2024-08-17T18:22:46.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122605 2024-08-17T18:22:46.089 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.089 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:46.089 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.090 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:160898 57.16 57:6bcdff34:::benchmark_data_smithi136_389235_object44976:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1659)", 2024-08-17T18:22:46.090 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:56.563419+0000", 2024-08-17T18:22:46.090 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 289.50798482900001, 2024-08-17T18:22:46.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0259137519999999, 2024-08-17T18:22:46.090 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:46.090 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:46.090 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:46.090 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:22:46.090 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:22:46.090 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 160898 2024-08-17T18:22:46.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.090 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:46.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:46.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.563419+0000", 2024-08-17T18:22:46.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:46.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.563419+0000", 2024-08-17T18:22:46.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:46.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.563417+0000", 2024-08-17T18:22:46.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:22:46.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:46.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.563427+0000", 2024-08-17T18:22:46.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0105000000000001e-05 2024-08-17T18:22:46.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:46.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.563454+0000", 2024-08-17T18:22:46.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.639e-05 2024-08-17T18:22:46.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:46.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.563464+0000", 2024-08-17T18:22:46.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0795000000000001e-05 2024-08-17T18:22:46.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.575209+0000", 2024-08-17T18:22:46.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011744675 2024-08-17T18:22:46.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:22:46.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.575220+0000", 2024-08-17T18:22:46.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1182999999999999e-05 2024-08-17T18:22:46.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.584432+0000", 2024-08-17T18:22:46.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.009211565 2024-08-17T18:22:46.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:46.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.584575+0000", 2024-08-17T18:22:46.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014335199999999999 2024-08-17T18:22:46.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T18:22:46.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.584924+0000", 2024-08-17T18:22:46.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034901399999999998 2024-08-17T18:22:46.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.589159+0000", 2024-08-17T18:22:46.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00423516 2024-08-17T18:22:46.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.589205+0000", 2024-08-17T18:22:46.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5757999999999998e-05 2024-08-17T18:22:46.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:46.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.589215+0000", 2024-08-17T18:22:46.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0037000000000001e-05 2024-08-17T18:22:46.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:46.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.589287+0000", 2024-08-17T18:22:46.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2329000000000005e-05 2024-08-17T18:22:46.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:46.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.589333+0000", 2024-08-17T18:22:46.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5303000000000003e-05 2024-08-17T18:22:46.096 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.096 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:46.096 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.097 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:160913 57.16 57:6c3e772a:::benchmark_data_smithi136_389235_object44991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1660)", 2024-08-17T18:22:46.097 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:56.598041+0000", 2024-08-17T18:22:46.097 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 289.47336324100002, 2024-08-17T18:22:46.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99375612099999999, 2024-08-17T18:22:46.097 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:46.097 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:46.097 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:46.097 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:22:46.097 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:22:46.097 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 160913 2024-08-17T18:22:46.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.097 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:46.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:46.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598041+0000", 2024-08-17T18:22:46.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:46.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598041+0000", 2024-08-17T18:22:46.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:46.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598040+0000", 2024-08-17T18:22:46.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:22:46.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:46.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598044+0000", 2024-08-17T18:22:46.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.685e-06 2024-08-17T18:22:46.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:46.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598055+0000", 2024-08-17T18:22:46.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.043e-05 2024-08-17T18:22:46.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:46.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598061+0000", 2024-08-17T18:22:46.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4269999999999996e-06 2024-08-17T18:22:46.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598122+0000", 2024-08-17T18:22:46.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.101e-05 2024-08-17T18:22:46.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:22:46.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598131+0000", 2024-08-17T18:22:46.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9679999999999995e-06 2024-08-17T18:22:46.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.587199+0000", 2024-08-17T18:22:46.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98906787600000001 2024-08-17T18:22:46.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:46.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.587308+0000", 2024-08-17T18:22:46.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108712 2024-08-17T18:22:46.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T18:22:46.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.587567+0000", 2024-08-17T18:22:46.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025900400000000002 2024-08-17T18:22:46.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.589363+0000", 2024-08-17T18:22:46.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017963650000000001 2024-08-17T18:22:46.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.591651+0000", 2024-08-17T18:22:46.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022876120000000001 2024-08-17T18:22:46.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:46.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.591719+0000", 2024-08-17T18:22:46.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8367000000000005e-05 2024-08-17T18:22:46.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:46.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.591759+0000", 2024-08-17T18:22:46.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9480999999999997e-05 2024-08-17T18:22:46.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:46.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.591797+0000", 2024-08-17T18:22:46.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7957999999999998e-05 2024-08-17T18:22:46.104 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.104 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:46.104 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.104 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:160914 57.16 57:681070b7:::benchmark_data_smithi136_389235_object44992:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1660)", 2024-08-17T18:22:46.104 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:56.598883+0000", 2024-08-17T18:22:46.104 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 289.47252052300001, 2024-08-17T18:22:46.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0028806669999999, 2024-08-17T18:22:46.104 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:46.104 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:46.104 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:46.104 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:22:46.105 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:22:46.105 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 160914 2024-08-17T18:22:46.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.105 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:46.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:46.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598883+0000", 2024-08-17T18:22:46.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:46.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598883+0000", 2024-08-17T18:22:46.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:46.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598882+0000", 2024-08-17T18:22:46.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:22:46.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:46.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598888+0000", 2024-08-17T18:22:46.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9750000000000004e-06 2024-08-17T18:22:46.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:46.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598903+0000", 2024-08-17T18:22:46.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4534e-05 2024-08-17T18:22:46.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:46.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598910+0000", 2024-08-17T18:22:46.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0979999999999996e-06 2024-08-17T18:22:46.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598952+0000", 2024-08-17T18:22:46.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2006999999999999e-05 2024-08-17T18:22:46.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:22:46.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598960+0000", 2024-08-17T18:22:46.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3899999999999993e-06 2024-08-17T18:22:46.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.589545+0000", 2024-08-17T18:22:46.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.990585037 2024-08-17T18:22:46.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:46.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.589681+0000", 2024-08-17T18:22:46.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000136379 2024-08-17T18:22:46.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T18:22:46.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.590020+0000", 2024-08-17T18:22:46.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033900400000000001 2024-08-17T18:22:46.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.591710+0000", 2024-08-17T18:22:46.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016892420000000001 2024-08-17T18:22:46.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.597521+0000", 2024-08-17T18:22:46.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058114159999999998 2024-08-17T18:22:46.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:46.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.601595+0000", 2024-08-17T18:22:46.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040736549999999998 2024-08-17T18:22:46.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:46.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.601719+0000", 2024-08-17T18:22:46.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012428 2024-08-17T18:22:46.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:46.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.601764+0000", 2024-08-17T18:22:46.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4897000000000001e-05 2024-08-17T18:22:46.110 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.110 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:46.111 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.111 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:160956 57.16 57:69ce9ed4:::benchmark_data_smithi136_389235_object45034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1660)", 2024-08-17T18:22:46.111 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:56.725647+0000", 2024-08-17T18:22:46.111 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 289.34575651300003, 2024-08-17T18:22:46.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.88076719599999997, 2024-08-17T18:22:46.111 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:46.111 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:46.111 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:46.111 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:22:46.111 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:22:46.111 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 160956 2024-08-17T18:22:46.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.112 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:46.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:46.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.725647+0000", 2024-08-17T18:22:46.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:46.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.725647+0000", 2024-08-17T18:22:46.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:46.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.725646+0000", 2024-08-17T18:22:46.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:22:46.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:46.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.725652+0000", 2024-08-17T18:22:46.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2759999999999997e-06 2024-08-17T18:22:46.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:46.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.725664+0000", 2024-08-17T18:22:46.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1616e-05 2024-08-17T18:22:46.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:46.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.725670+0000", 2024-08-17T18:22:46.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4359999999999999e-06 2024-08-17T18:22:46.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.725720+0000", 2024-08-17T18:22:46.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9599999999999999e-05 2024-08-17T18:22:46.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:22:46.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.725727+0000", 2024-08-17T18:22:46.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1799999999999999e-06 2024-08-17T18:22:46.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.591831+0000", 2024-08-17T18:22:46.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.86610343700000003 2024-08-17T18:22:46.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:46.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.591902+0000", 2024-08-17T18:22:46.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1347999999999995e-05 2024-08-17T18:22:46.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T18:22:46.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.592142+0000", 2024-08-17T18:22:46.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024018299999999999 2024-08-17T18:22:46.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.597454+0000", 2024-08-17T18:22:46.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0053123390000000001 2024-08-17T18:22:46.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.597555+0000", 2024-08-17T18:22:46.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000100389 2024-08-17T18:22:46.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:46.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606321+0000", 2024-08-17T18:22:46.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0087664409999999998 2024-08-17T18:22:46.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:46.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606372+0000", 2024-08-17T18:22:46.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0843000000000001e-05 2024-08-17T18:22:46.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:46.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606414+0000", 2024-08-17T18:22:46.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2333999999999998e-05 2024-08-17T18:22:46.117 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.117 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:46.117 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.117 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:161012 57.16 57:6e84d2aa:::benchmark_data_smithi136_389235_object45090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1660)", 2024-08-17T18:22:46.118 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:56.934535+0000", 2024-08-17T18:22:46.118 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 289.13686888799998, 2024-08-17T18:22:46.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.68200467600000003, 2024-08-17T18:22:46.118 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:46.118 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:46.118 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:46.118 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:22:46.118 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:22:46.118 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161012 2024-08-17T18:22:46.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.118 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:46.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:46.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.934535+0000", 2024-08-17T18:22:46.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:46.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.934535+0000", 2024-08-17T18:22:46.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:46.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.934533+0000", 2024-08-17T18:22:46.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:22:46.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:46.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.934540+0000", 2024-08-17T18:22:46.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2890000000000003e-06 2024-08-17T18:22:46.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:46.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.934551+0000", 2024-08-17T18:22:46.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1762e-05 2024-08-17T18:22:46.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:46.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.934557+0000", 2024-08-17T18:22:46.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4399999999999996e-06 2024-08-17T18:22:46.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.934601+0000", 2024-08-17T18:22:46.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4533000000000002e-05 2024-08-17T18:22:46.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:22:46.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.934610+0000", 2024-08-17T18:22:46.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2199999999999992e-06 2024-08-17T18:22:46.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.594900+0000", 2024-08-17T18:22:46.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.66029078600000002 2024-08-17T18:22:46.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:46.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.595020+0000", 2024-08-17T18:22:46.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119622 2024-08-17T18:22:46.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T18:22:46.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.595434+0000", 2024-08-17T18:22:46.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000413805 2024-08-17T18:22:46.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:46.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606426+0000", 2024-08-17T18:22:46.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010991772 2024-08-17T18:22:46.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606470+0000", 2024-08-17T18:22:46.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4162999999999997e-05 2024-08-17T18:22:46.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616445+0000", 2024-08-17T18:22:46.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0099751089999999994 2024-08-17T18:22:46.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:46.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616496+0000", 2024-08-17T18:22:46.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1069000000000002e-05 2024-08-17T18:22:46.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:46.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616540+0000", 2024-08-17T18:22:46.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3649000000000002e-05 2024-08-17T18:22:46.124 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.124 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:46.124 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.125 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:161032 57.16 57:6919a724:::benchmark_data_smithi136_389235_object45110:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1660)", 2024-08-17T18:22:46.125 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:57.066208+0000", 2024-08-17T18:22:46.125 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 289.00519561900001, 2024-08-17T18:22:46.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.55043567800000004, 2024-08-17T18:22:46.125 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:46.125 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:46.125 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:46.125 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:22:46.125 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:22:46.125 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161032 2024-08-17T18:22:46.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.125 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:46.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:46.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.066208+0000", 2024-08-17T18:22:46.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:46.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.066208+0000", 2024-08-17T18:22:46.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:46.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.066206+0000", 2024-08-17T18:22:46.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:22:46.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:46.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.066216+0000", 2024-08-17T18:22:46.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9129999999999998e-06 2024-08-17T18:22:46.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:46.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.066235+0000", 2024-08-17T18:22:46.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8524000000000001e-05 2024-08-17T18:22:46.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:46.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.066244+0000", 2024-08-17T18:22:46.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9039999999999998e-06 2024-08-17T18:22:46.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.066297+0000", 2024-08-17T18:22:46.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3087999999999997e-05 2024-08-17T18:22:46.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:22:46.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.066306+0000", 2024-08-17T18:22:46.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2140000000000002e-06 2024-08-17T18:22:46.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.597579+0000", 2024-08-17T18:22:46.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.53127298999999994 2024-08-17T18:22:46.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:46.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.597711+0000", 2024-08-17T18:22:46.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013171400000000001 2024-08-17T18:22:46.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T18:22:46.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.598077+0000", 2024-08-17T18:22:46.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036651799999999998 2024-08-17T18:22:46.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:46.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606431+0000", 2024-08-17T18:22:46.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0083538680000000004 2024-08-17T18:22:46.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606528+0000", 2024-08-17T18:22:46.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7275999999999995e-05 2024-08-17T18:22:46.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616574+0000", 2024-08-17T18:22:46.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010045554 2024-08-17T18:22:46.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:46.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616610+0000", 2024-08-17T18:22:46.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6565e-05 2024-08-17T18:22:46.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:46.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616644+0000", 2024-08-17T18:22:46.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3522999999999997e-05 2024-08-17T18:22:46.132 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.132 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:46.133 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.133 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:161078 57.16 57:6e3b9c80:::benchmark_data_smithi136_389235_object45156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1660)", 2024-08-17T18:22:46.133 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:57.238646+0000", 2024-08-17T18:22:46.133 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 288.83275739200002, 2024-08-17T18:22:46.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.37810427000000002, 2024-08-17T18:22:46.133 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:46.133 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:46.133 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:46.133 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:22:46.133 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:22:46.133 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161078 2024-08-17T18:22:46.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.134 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:46.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:46.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.238646+0000", 2024-08-17T18:22:46.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:46.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.238646+0000", 2024-08-17T18:22:46.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:46.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.238645+0000", 2024-08-17T18:22:46.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:22:46.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:46.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.238652+0000", 2024-08-17T18:22:46.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8349999999999998e-06 2024-08-17T18:22:46.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:46.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.238666+0000", 2024-08-17T18:22:46.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4346e-05 2024-08-17T18:22:46.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:46.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.238674+0000", 2024-08-17T18:22:46.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9889999999999992e-06 2024-08-17T18:22:46.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.239725+0000", 2024-08-17T18:22:46.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001050743 2024-08-17T18:22:46.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:22:46.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.239737+0000", 2024-08-17T18:22:46.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2510000000000001e-05 2024-08-17T18:22:46.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.599446+0000", 2024-08-17T18:22:46.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.359708378 2024-08-17T18:22:46.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:46.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.599580+0000", 2024-08-17T18:22:46.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000134485 2024-08-17T18:22:46.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T18:22:46.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.599828+0000", 2024-08-17T18:22:46.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024764999999999998 2024-08-17T18:22:46.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:46.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606479+0000", 2024-08-17T18:22:46.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066513500000000003 2024-08-17T18:22:46.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606563+0000", 2024-08-17T18:22:46.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3973000000000004e-05 2024-08-17T18:22:46.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616677+0000", 2024-08-17T18:22:46.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010113924999999999 2024-08-17T18:22:46.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:46.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616718+0000", 2024-08-17T18:22:46.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0836e-05 2024-08-17T18:22:46.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:46.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616751+0000", 2024-08-17T18:22:46.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2685999999999998e-05 2024-08-17T18:22:46.142 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.142 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:46.142 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.142 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:164936 57.a 57:5d3ff99b:::benchmark_data_smithi136_389235_object49014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1661)", 2024-08-17T18:22:46.142 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:06.210033+0000", 2024-08-17T18:22:46.142 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 279.861371201, 2024-08-17T18:22:46.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27177604100000002, 2024-08-17T18:22:46.142 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:46.142 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:46.142 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:46.143 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:22:46.143 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:22:46.143 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164936 2024-08-17T18:22:46.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.143 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:46.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:46.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210033+0000", 2024-08-17T18:22:46.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:46.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210033+0000", 2024-08-17T18:22:46.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:46.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210031+0000", 2024-08-17T18:22:46.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:22:46.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:46.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210039+0000", 2024-08-17T18:22:46.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9470000000000003e-06 2024-08-17T18:22:46.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:46.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210051+0000", 2024-08-17T18:22:46.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2771e-05 2024-08-17T18:22:46.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:46.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210058+0000", 2024-08-17T18:22:46.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8109999999999997e-06 2024-08-17T18:22:46.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210117+0000", 2024-08-17T18:22:46.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9086000000000003e-05 2024-08-17T18:22:46.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:46.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210260+0000", 2024-08-17T18:22:46.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014312299999999999 2024-08-17T18:22:46.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4,7", 2024-08-17T18:22:46.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210404+0000", 2024-08-17T18:22:46.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014415700000000001 2024-08-17T18:22:46.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.213570+0000", 2024-08-17T18:22:46.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031656869999999999 2024-08-17T18:22:46.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.213627+0000", 2024-08-17T18:22:46.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6935999999999998e-05 2024-08-17T18:22:46.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:46.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.213639+0000", 2024-08-17T18:22:46.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2143e-05 2024-08-17T18:22:46.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.481648+0000", 2024-08-17T18:22:46.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.26800850399999998 2024-08-17T18:22:46.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:46.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.481700+0000", 2024-08-17T18:22:46.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2073000000000001e-05 2024-08-17T18:22:46.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:46.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.481809+0000", 2024-08-17T18:22:46.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108818 2024-08-17T18:22:46.148 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.148 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:46.148 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.149 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:164945 57.a 57:55ee62ab:::benchmark_data_smithi136_389235_object49023:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1661)", 2024-08-17T18:22:46.149 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:06.230920+0000", 2024-08-17T18:22:46.149 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 279.84048329199999, 2024-08-17T18:22:46.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25469593200000001, 2024-08-17T18:22:46.149 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:46.149 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:46.149 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:46.149 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:22:46.149 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:22:46.149 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164945 2024-08-17T18:22:46.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.149 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:46.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:46.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.230920+0000", 2024-08-17T18:22:46.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:46.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.230920+0000", 2024-08-17T18:22:46.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:46.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.230916+0000", 2024-08-17T18:22:46.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999957 2024-08-17T18:22:46.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:46.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.230926+0000", 2024-08-17T18:22:46.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4390000000000006e-06 2024-08-17T18:22:46.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:46.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.230938+0000", 2024-08-17T18:22:46.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2094e-05 2024-08-17T18:22:46.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:46.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.396017+0000", 2024-08-17T18:22:46.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16507881099999999 2024-08-17T18:22:46.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.396075+0000", 2024-08-17T18:22:46.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8207e-05 2024-08-17T18:22:46.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:46.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.396236+0000", 2024-08-17T18:22:46.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016120800000000001 2024-08-17T18:22:46.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4,7", 2024-08-17T18:22:46.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.396397+0000", 2024-08-17T18:22:46.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000160571 2024-08-17T18:22:46.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.398644+0000", 2024-08-17T18:22:46.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022471930000000002 2024-08-17T18:22:46.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:46.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.400494+0000", 2024-08-17T18:22:46.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018502049999999999 2024-08-17T18:22:46.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.404981+0000", 2024-08-17T18:22:46.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044866910000000001 2024-08-17T18:22:46.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.485449+0000", 2024-08-17T18:22:46.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.080468711999999998 2024-08-17T18:22:46.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:46.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.485506+0000", 2024-08-17T18:22:46.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6317999999999998e-05 2024-08-17T18:22:46.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:46.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.485616+0000", 2024-08-17T18:22:46.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000110649 2024-08-17T18:22:46.155 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.155 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:46.155 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.155 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:117666 58.2 58:47c92b81:::benchmark_data_smithi136_442124_object4050:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1726)", 2024-08-17T18:22:46.155 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:38.291041+0000", 2024-08-17T18:22:46.155 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.7803632330000001, 2024-08-17T18:22:46.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27569550999999998, 2024-08-17T18:22:46.155 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:46.156 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:46.156 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:46.156 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:22:46.156 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:22:46.156 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 117666 2024-08-17T18:22:46.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.156 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:46.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:46.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.291041+0000", 2024-08-17T18:22:46.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:46.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.291041+0000", 2024-08-17T18:22:46.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:46.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.291038+0000", 2024-08-17T18:22:46.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999976 2024-08-17T18:22:46.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:46.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.291049+0000", 2024-08-17T18:22:46.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0883999999999999e-05 2024-08-17T18:22:46.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:46.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.291074+0000", 2024-08-17T18:22:46.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4553999999999999e-05 2024-08-17T18:22:46.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:46.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.291083+0000", 2024-08-17T18:22:46.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6590000000000002e-06 2024-08-17T18:22:46.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.291146+0000", 2024-08-17T18:22:46.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2277e-05 2024-08-17T18:22:46.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:46.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.291278+0000", 2024-08-17T18:22:46.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000132356 2024-08-17T18:22:46.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,4", 2024-08-17T18:22:46.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.291391+0000", 2024-08-17T18:22:46.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011304899999999999 2024-08-17T18:22:46.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:46.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.295625+0000", 2024-08-17T18:22:46.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042343459999999999 2024-08-17T18:22:46.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.308090+0000", 2024-08-17T18:22:46.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012464648 2024-08-17T18:22:46.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566641+0000", 2024-08-17T18:22:46.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25855076999999999 2024-08-17T18:22:46.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:46.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566689+0000", 2024-08-17T18:22:46.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8615000000000001e-05 2024-08-17T18:22:46.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:46.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566736+0000", 2024-08-17T18:22:46.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6635999999999999e-05 2024-08-17T18:22:46.161 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.161 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:46.161 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.162 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:117670 58.2 58:47193288:::benchmark_data_smithi136_442124_object4054:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1726)", 2024-08-17T18:22:46.162 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:38.295548+0000", 2024-08-17T18:22:46.162 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.7758552979999997, 2024-08-17T18:22:46.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27135621100000001, 2024-08-17T18:22:46.162 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:46.162 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:46.162 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:46.162 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:22:46.162 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:22:46.162 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 117670 2024-08-17T18:22:46.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.162 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:46.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:46.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.295548+0000", 2024-08-17T18:22:46.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:46.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.295548+0000", 2024-08-17T18:22:46.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:46.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.295546+0000", 2024-08-17T18:22:46.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999976 2024-08-17T18:22:46.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:46.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.295557+0000", 2024-08-17T18:22:46.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.076e-05 2024-08-17T18:22:46.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:46.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.295572+0000", 2024-08-17T18:22:46.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5009e-05 2024-08-17T18:22:46.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:46.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.295581+0000", 2024-08-17T18:22:46.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2410000000000001e-06 2024-08-17T18:22:46.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.295648+0000", 2024-08-17T18:22:46.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6795e-05 2024-08-17T18:22:46.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:46.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.295776+0000", 2024-08-17T18:22:46.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000127811 2024-08-17T18:22:46.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,4", 2024-08-17T18:22:46.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.295890+0000", 2024-08-17T18:22:46.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000114164 2024-08-17T18:22:46.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:46.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.303987+0000", 2024-08-17T18:22:46.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080971359999999996 2024-08-17T18:22:46.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566789+0000", 2024-08-17T18:22:46.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.26280209500000001 2024-08-17T18:22:46.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566836+0000", 2024-08-17T18:22:46.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6341000000000002e-05 2024-08-17T18:22:46.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:46.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566872+0000", 2024-08-17T18:22:46.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6581000000000001e-05 2024-08-17T18:22:46.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:46.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566905+0000", 2024-08-17T18:22:46.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2428000000000002e-05 2024-08-17T18:22:46.168 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.168 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:46.168 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.168 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:117672 58.2 58:45143eb0:::benchmark_data_smithi136_442124_object4056:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1726)", 2024-08-17T18:22:46.168 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:38.296417+0000", 2024-08-17T18:22:46.168 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.7749869550000001, 2024-08-17T18:22:46.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27061583500000003, 2024-08-17T18:22:46.168 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:46.168 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:46.168 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:46.168 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:22:46.169 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:22:46.169 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 117672 2024-08-17T18:22:46.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.169 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:46.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:46.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.296417+0000", 2024-08-17T18:22:46.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:46.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.296417+0000", 2024-08-17T18:22:46.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:46.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.296416+0000", 2024-08-17T18:22:46.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:22:46.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:46.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.296421+0000", 2024-08-17T18:22:46.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9779999999999996e-06 2024-08-17T18:22:46.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:46.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.296431+0000", 2024-08-17T18:22:46.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8379999999999997e-06 2024-08-17T18:22:46.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:46.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.296439+0000", 2024-08-17T18:22:46.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6920000000000008e-06 2024-08-17T18:22:46.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.299968+0000", 2024-08-17T18:22:46.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035287209999999999 2024-08-17T18:22:46.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:46.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.300084+0000", 2024-08-17T18:22:46.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011565699999999999 2024-08-17T18:22:46.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,4", 2024-08-17T18:22:46.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.300188+0000", 2024-08-17T18:22:46.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104132 2024-08-17T18:22:46.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:46.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.308107+0000", 2024-08-17T18:22:46.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0079187189999999994 2024-08-17T18:22:46.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566948+0000", 2024-08-17T18:22:46.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25884148299999998 2024-08-17T18:22:46.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566978+0000", 2024-08-17T18:22:46.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9751999999999999e-05 2024-08-17T18:22:46.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:46.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.567004+0000", 2024-08-17T18:22:46.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5573000000000001e-05 2024-08-17T18:22:46.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:46.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.567033+0000", 2024-08-17T18:22:46.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9153999999999999e-05 2024-08-17T18:22:46.174 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.174 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:46.174 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.174 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:117675 58.2 58:4a294109:::benchmark_data_smithi136_442124_object4059:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1726)", 2024-08-17T18:22:46.174 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:38.300334+0000", 2024-08-17T18:22:46.174 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.7710698120000004, 2024-08-17T18:22:46.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.26687857300000001, 2024-08-17T18:22:46.174 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:46.174 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:46.174 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:46.175 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:22:46.175 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:22:46.175 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 117675 2024-08-17T18:22:46.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.175 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:46.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:46.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.300334+0000", 2024-08-17T18:22:46.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:46.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.300334+0000", 2024-08-17T18:22:46.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:46.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.300332+0000", 2024-08-17T18:22:46.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:22:46.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:46.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.300340+0000", 2024-08-17T18:22:46.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.79e-06 2024-08-17T18:22:46.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:46.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.300354+0000", 2024-08-17T18:22:46.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4029000000000001e-05 2024-08-17T18:22:46.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:46.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.300362+0000", 2024-08-17T18:22:46.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6140000000000003e-06 2024-08-17T18:22:46.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.304024+0000", 2024-08-17T18:22:46.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036621219999999999 2024-08-17T18:22:46.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:46.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.304165+0000", 2024-08-17T18:22:46.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014138599999999999 2024-08-17T18:22:46.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,4", 2024-08-17T18:22:46.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.304277+0000", 2024-08-17T18:22:46.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011175699999999999 2024-08-17T18:22:46.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:46.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.312292+0000", 2024-08-17T18:22:46.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080151070000000005 2024-08-17T18:22:46.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.567081+0000", 2024-08-17T18:22:46.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25478909399999999 2024-08-17T18:22:46.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.567128+0000", 2024-08-17T18:22:46.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6828e-05 2024-08-17T18:22:46.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:46.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.567173+0000", 2024-08-17T18:22:46.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5215e-05 2024-08-17T18:22:46.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:46.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.567213+0000", 2024-08-17T18:22:46.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9303e-05 2024-08-17T18:22:46.180 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.180 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:46.180 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.180 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:117682 58.2 58:489a1685:::benchmark_data_smithi136_442124_object4066:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1726)", 2024-08-17T18:22:46.180 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:38.304875+0000", 2024-08-17T18:22:46.180 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.7665288720000003, 2024-08-17T18:22:46.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.26249927099999998, 2024-08-17T18:22:46.180 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:46.181 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:22:46.181 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:22:46.181 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:22:46.181 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:22:46.181 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 117682 2024-08-17T18:22:46.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.181 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:46.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:46.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.304875+0000", 2024-08-17T18:22:46.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:46.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.304875+0000", 2024-08-17T18:22:46.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:46.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.304873+0000", 2024-08-17T18:22:46.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:22:46.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:46.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.304880+0000", 2024-08-17T18:22:46.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5579999999999997e-06 2024-08-17T18:22:46.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:46.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.304893+0000", 2024-08-17T18:22:46.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2835e-05 2024-08-17T18:22:46.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:46.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.304910+0000", 2024-08-17T18:22:46.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7674000000000001e-05 2024-08-17T18:22:46.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.308127+0000", 2024-08-17T18:22:46.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032166270000000001 2024-08-17T18:22:46.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:46.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.308250+0000", 2024-08-17T18:22:46.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012278300000000001 2024-08-17T18:22:46.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,4", 2024-08-17T18:22:46.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.308361+0000", 2024-08-17T18:22:46.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000111047 2024-08-17T18:22:46.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:22:46.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.312516+0000", 2024-08-17T18:22:46.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041553759999999997 2024-08-17T18:22:46.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.567262+0000", 2024-08-17T18:22:46.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25474614499999998 2024-08-17T18:22:46.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:22:46.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.567300+0000", 2024-08-17T18:22:46.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7829e-05 2024-08-17T18:22:46.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:22:46.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.567342+0000", 2024-08-17T18:22:46.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1551000000000002e-05 2024-08-17T18:22:46.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:46.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.567374+0000", 2024-08-17T18:22:46.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2487000000000003e-05 2024-08-17T18:22:46.186 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.186 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:46.186 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.186 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.36936.0:117666 58.2 e1726/1701 ondisk, result = 0)", 2024-08-17T18:22:46.186 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:38.316274+0000", 2024-08-17T18:22:46.186 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.755130276, 2024-08-17T18:22:46.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25047019199999998, 2024-08-17T18:22:46.187 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:46.187 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:22:46.187 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:46.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:46.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316274+0000", 2024-08-17T18:22:46.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:46.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316274+0000", 2024-08-17T18:22:46.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:46.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316273+0000", 2024-08-17T18:22:46.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:22:46.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:46.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316279+0000", 2024-08-17T18:22:46.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9320000000000001e-06 2024-08-17T18:22:46.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:46.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316288+0000", 2024-08-17T18:22:46.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0250000000000008e-06 2024-08-17T18:22:46.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:46.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316294+0000", 2024-08-17T18:22:46.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3380000000000001e-06 2024-08-17T18:22:46.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316338+0000", 2024-08-17T18:22:46.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3930999999999997e-05 2024-08-17T18:22:46.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:46.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566627+0000", 2024-08-17T18:22:46.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25028892600000002 2024-08-17T18:22:46.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:46.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566744+0000", 2024-08-17T18:22:46.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011685500000000001 2024-08-17T18:22:46.190 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.190 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:46.190 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.190 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.36936.0:117670 58.2 e1726/1701 ondisk, result = 0)", 2024-08-17T18:22:46.190 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:38.316304+0000", 2024-08-17T18:22:46.191 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.7550997200000005, 2024-08-17T18:22:46.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25048983800000002, 2024-08-17T18:22:46.191 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:46.191 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:22:46.191 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:46.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:46.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316304+0000", 2024-08-17T18:22:46.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:46.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316304+0000", 2024-08-17T18:22:46.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:46.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316304+0000", 2024-08-17T18:22:46.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:22:46.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:46.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316306+0000", 2024-08-17T18:22:46.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3250000000000002e-06 2024-08-17T18:22:46.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:46.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316311+0000", 2024-08-17T18:22:46.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2259999999999996e-06 2024-08-17T18:22:46.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:46.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316315+0000", 2024-08-17T18:22:46.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2310000000000001e-06 2024-08-17T18:22:46.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566774+0000", 2024-08-17T18:22:46.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25045969899999998 2024-08-17T18:22:46.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:46.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566783+0000", 2024-08-17T18:22:46.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5339999999999999e-06 2024-08-17T18:22:46.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:46.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566794+0000", 2024-08-17T18:22:46.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1085e-05 2024-08-17T18:22:46.194 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.194 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:46.194 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.194 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.36936.0:117672 58.2 e1726/1701 ondisk, result = 0)", 2024-08-17T18:22:46.195 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:38.316343+0000", 2024-08-17T18:22:46.195 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.7550611849999997, 2024-08-17T18:22:46.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25060864999999999, 2024-08-17T18:22:46.195 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:46.195 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:22:46.195 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:46.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:46.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316343+0000", 2024-08-17T18:22:46.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:46.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316343+0000", 2024-08-17T18:22:46.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:46.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316342+0000", 2024-08-17T18:22:46.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:22:46.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:46.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316345+0000", 2024-08-17T18:22:46.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.988e-06 2024-08-17T18:22:46.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:46.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566609+0000", 2024-08-17T18:22:46.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25026386699999997 2024-08-17T18:22:46.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:46.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566622+0000", 2024-08-17T18:22:46.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2916e-05 2024-08-17T18:22:46.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566937+0000", 2024-08-17T18:22:46.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000315013 2024-08-17T18:22:46.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:46.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566943+0000", 2024-08-17T18:22:46.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8660000000000002e-06 2024-08-17T18:22:46.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:46.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566951+0000", 2024-08-17T18:22:46.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2989999999999997e-06 2024-08-17T18:22:46.198 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.198 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:46.198 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.199 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.36936.0:117670 58.2 e1726/1701 ondisk, result = 0)", 2024-08-17T18:22:46.199 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:38.316955+0000", 2024-08-17T18:22:46.199 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.7544490230000003, 2024-08-17T18:22:46.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.249955276, 2024-08-17T18:22:46.199 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:22:46.199 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:22:46.199 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:22:46.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:22:46.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316955+0000", 2024-08-17T18:22:46.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:22:46.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316955+0000", 2024-08-17T18:22:46.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:22:46.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:22:46.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316954+0000", 2024-08-17T18:22:46.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:22:46.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:22:46.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316959+0000", 2024-08-17T18:22:46.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.921e-06 2024-08-17T18:22:46.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:22:46.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566597+0000", 2024-08-17T18:22:46.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.249638255 2024-08-17T18:22:46.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:22:46.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566612+0000", 2024-08-17T18:22:46.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.454e-05 2024-08-17T18:22:46.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:22:46.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566823+0000", 2024-08-17T18:22:46.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021107400000000001 2024-08-17T18:22:46.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:22:46.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566830+0000", 2024-08-17T18:22:46.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3320000000000003e-06 2024-08-17T18:22:46.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:22:46.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:22:46.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:22:46.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566910+0000", 2024-08-17T18:22:46.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9765999999999995e-05 2024-08-17T18:22:46.202 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.202 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:46.202 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.203 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:22:46.203 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:22:46.203 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:22:46.203 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:46.204 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:45 smithi136 ceph-mon[109861]: pgmap v3384: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 90 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 585 op/s; 17477/321201 objects degraded (5.441%); 12015/321201 objects misplaced (3.741%); 16 MiB/s, 261 objects/s recovering 2024-08-17T18:22:46.204 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:45 smithi136 ceph-mon[115078]: pgmap v3384: 99 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 90 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 585 op/s; 17477/321201 objects degraded (5.441%); 12015/321201 objects misplaced (3.741%); 16 MiB/s, 261 objects/s recovering 2024-08-17T18:22:46.238 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:46.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:46.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:46.340 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:46.377 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:46.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:46.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:46.479 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:46.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:46.533 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:46.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:46.634 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:46.665 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:46.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:46.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:46.768 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:46.800 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:46.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:46.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:46.902 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:46.941 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:46.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:46.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:46.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-17T18:22:47.043 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:22:47.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:47.078 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:47.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:22:47.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:47.216 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:47.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:47.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:47.307 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:22:47.318 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:47.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:47.360 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:47.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:47.461 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:47.490 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:47.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:47.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:47.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:47.627 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:47.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:47.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:47.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:22:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:22:47] "GET /metrics HTTP/1.1" 200 36631 "" "Prometheus/2.51.0" 2024-08-17T18:22:47.730 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:47.778 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:47.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:47.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:47.880 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:47.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:47.924 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:47.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:48.025 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:48.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:48.073 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:48.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:48.105 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:47 smithi136 ceph-mon[109861]: pgmap v3385: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 614 op/s; 17246/317295 objects degraded (5.435%); 12015/317295 objects misplaced (3.787%); 15 MiB/s, 238 objects/s recovering 2024-08-17T18:22:48.105 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:47 smithi136 ceph-mon[115078]: pgmap v3385: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 614 op/s; 17246/317295 objects degraded (5.435%); 12015/317295 objects misplaced (3.787%); 15 MiB/s, 238 objects/s recovering 2024-08-17T18:22:48.174 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:48.182 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:22:48.182 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:22:48.183 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:22:48.222 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:22:48.223 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:22:48.257 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:48.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:48.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:48.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:47 smithi165 ceph-mon[105517]: pgmap v3385: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 614 op/s; 17246/317295 objects degraded (5.435%); 12015/317295 objects misplaced (3.787%); 15 MiB/s, 238 objects/s recovering 2024-08-17T18:22:48.263 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:22:48.263 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:22:48.291 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:22:48.291 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:22:48.323 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:22:48.323 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:22:48.354 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:22:48.354 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:22:48.358 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:48.394 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:22:48.394 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-08-17T18:22:48.400 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:48.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:48.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:48.427 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:22:48.427 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:22:48.502 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:48.534 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:48.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:48.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:48.636 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:48.669 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:48.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:48.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:48.771 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:48.803 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:48.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:48.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:48.905 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:48.928 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:48.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:48.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:48.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:48 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3105480553' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:22:48.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:48 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3105480553' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:22:49.030 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:49.071 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:49.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:49.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:49.173 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:49.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:49.210 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:49.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:49.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:48 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3105480553' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:22:49.296 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:22:49.296 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:22:49.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:49.340 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:49.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:49.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:49.443 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:49.459 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:22:49.479 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:49.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:49.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:49.581 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:49.605 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:49.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:49.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:49.707 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:49.756 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:49.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:49.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:49.858 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:49.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:49.895 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:49.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:49.996 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:50.023 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:50.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:50.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:50.125 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:50.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:50.172 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:50.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:50.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:49 smithi136 ceph-mon[109861]: pgmap v3386: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 6.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 619 op/s; 17246/310377 objects degraded (5.556%); 11672/310377 objects misplaced (3.761%); 4.9 MiB/s, 77 objects/s recovering 2024-08-17T18:22:50.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:49 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 17246/310377 objects degraded (5.556%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:50.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:49 smithi136 ceph-mon[115078]: pgmap v3386: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 6.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 619 op/s; 17246/310377 objects degraded (5.556%); 11672/310377 objects misplaced (3.761%); 4.9 MiB/s, 77 objects/s recovering 2024-08-17T18:22:50.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:49 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 17246/310377 objects degraded (5.556%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:50.197 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:49 smithi165 ceph-mon[105517]: pgmap v3386: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 6.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 619 op/s; 17246/310377 objects degraded (5.556%); 11672/310377 objects misplaced (3.761%); 4.9 MiB/s, 77 objects/s recovering 2024-08-17T18:22:50.197 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:49 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 17246/310377 objects degraded (5.556%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:50.238 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-17T18:22:50.273 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:50.313 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:50.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:50.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:50.415 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:50.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:50.461 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:50.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:50.538 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:22:50.562 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:50.602 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:50.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:50.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:50.704 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:50.736 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:50.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:50.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:50.838 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:50.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:50.876 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:50.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:50.977 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:51.012 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:51.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:51.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:51.109 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:22:51.115 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:51.174 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:51.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:51.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:51.276 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:51.304 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:51.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:51.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:51.406 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:51.460 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:51.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:51.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:51.564 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:51.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:51.609 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:51.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:51.711 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:51.735 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:51.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:51.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:51.837 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:51.886 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:51.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:51.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:51.988 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:52.032 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:52.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:52.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:52.042 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:51 smithi136 ceph-mon[109861]: pgmap v3387: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 6.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 590 op/s; 17246/309084 objects degraded (5.580%); 11672/309084 objects misplaced (3.776%); 4.4 MiB/s, 69 objects/s recovering 2024-08-17T18:22:52.042 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:22:52.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:51 smithi136 ceph-mon[115078]: pgmap v3387: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 6.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 590 op/s; 17246/309084 objects degraded (5.580%); 11672/309084 objects misplaced (3.776%); 4.4 MiB/s, 69 objects/s recovering 2024-08-17T18:22:52.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:22:52.134 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:52.164 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:52.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:52.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:52.167 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:51 smithi165 ceph-mon[105517]: pgmap v3387: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 6.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 590 op/s; 17246/309084 objects degraded (5.580%); 11672/309084 objects misplaced (3.776%); 4.4 MiB/s, 69 objects/s recovering 2024-08-17T18:22:52.167 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:22:52.266 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:52.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:52.306 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:52.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:52.408 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:52.441 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:52.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:52.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:52.494 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:22:52.494 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:22:52.543 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:52.567 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:22:52.568 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:52.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:52.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:52.671 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:52.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:52.699 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:52.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:52.800 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:52.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:52.860 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:52.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:52.915 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.2 to 1 (10000) 2024-08-17T18:22:52.962 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:52.987 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:52.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:52.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:53.089 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:53.118 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:53.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:53.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:53.191 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:52 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1802030961' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T18:22:53.191 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:52 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T18:22:53.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:52 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1802030961' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T18:22:53.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:52 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T18:22:53.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:52 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1802030961' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T18:22:53.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:52 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T18:22:53.219 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-17T18:22:53.220 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:53.247 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:53.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:53.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:53.349 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:53.371 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:53.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:53.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:53.442 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:22:53.473 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:53.495 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:53.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:53.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:53.597 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:22:53.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:53.620 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:53.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:22:53.721 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:53.744 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:53.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:53.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:53.846 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:53.871 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:53.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:53.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:53.972 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:53.995 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:53.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:53.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:54.097 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:54.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:54.119 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:54.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:54.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:53 smithi136 ceph-mon[115078]: pgmap v3388: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 657 op/s; 17023/305877 objects degraded (5.565%); 11672/305877 objects misplaced (3.816%); 5.5 MiB/s, 88 objects/s recovering 2024-08-17T18:22:54.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:53 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T18:22:54.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:53 smithi136 ceph-mon[115078]: osdmap e1727: 8 total, 8 up, 8 in 2024-08-17T18:22:54.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:53 smithi136 ceph-mon[109861]: pgmap v3388: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 657 op/s; 17023/305877 objects degraded (5.565%); 11672/305877 objects misplaced (3.816%); 5.5 MiB/s, 88 objects/s recovering 2024-08-17T18:22:54.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:53 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T18:22:54.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:53 smithi136 ceph-mon[109861]: osdmap e1727: 8 total, 8 up, 8 in 2024-08-17T18:22:54.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:54.244 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:54.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:54.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:54.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:53 smithi165 ceph-mon[105517]: pgmap v3388: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 657 op/s; 17023/305877 objects degraded (5.565%); 11672/305877 objects misplaced (3.816%); 5.5 MiB/s, 88 objects/s recovering 2024-08-17T18:22:54.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:53 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T18:22:54.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:53 smithi165 ceph-mon[105517]: osdmap e1727: 8 total, 8 up, 8 in 2024-08-17T18:22:54.345 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:22:54.367 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:54.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:54.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:22:54.469 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:54.490 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:54.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:54.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:54.592 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:54.613 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:54.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:54.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:54.715 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:54.738 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:54.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:54.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:54.840 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:54.861 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:54.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:54.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:54.963 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:54.986 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:54.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:54.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:55.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:54 smithi165 ceph-mon[105517]: osdmap e1728: 8 total, 8 up, 8 in 2024-08-17T18:22:55.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:54 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 17023/305877 objects degraded (5.565%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:55.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:55.110 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:55.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:55.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:55.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:54 smithi136 ceph-mon[109861]: osdmap e1728: 8 total, 8 up, 8 in 2024-08-17T18:22:55.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:54 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 17023/305877 objects degraded (5.565%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:55.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:54 smithi136 ceph-mon[115078]: osdmap e1728: 8 total, 8 up, 8 in 2024-08-17T18:22:55.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:54 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 17023/305877 objects degraded (5.565%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T18:22:55.211 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:55.232 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:55.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:55.270 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:22:55.270 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:22:55.334 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:55.356 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:55.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:55.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:55.458 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:55.481 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:55.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:55.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:55.583 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:55.616 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:55.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:55.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:55.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:55.761 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:55.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:55.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:55.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:22:55.893 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:55.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:55.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:22:55.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:55 smithi136 ceph-mon[109861]: pgmap v3391: 99 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 6.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 17041/300318 objects degraded (5.674%); 11350/300318 objects misplaced (3.779%); 3.2 MiB/s, 50 objects/s recovering 2024-08-17T18:22:55.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:55 smithi136 ceph-mon[115078]: pgmap v3391: 99 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 6.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 17041/300318 objects degraded (5.674%); 11350/300318 objects misplaced (3.779%); 3.2 MiB/s, 50 objects/s recovering 2024-08-17T18:22:55.978 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-17T18:22:55.994 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:56.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:56.077 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:56.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:56.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:56.207 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:56.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:56.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:56.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:55 smithi165 ceph-mon[105517]: pgmap v3391: 99 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 6.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 17041/300318 objects degraded (5.674%); 11350/300318 objects misplaced (3.779%); 3.2 MiB/s, 50 objects/s recovering 2024-08-17T18:22:56.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:55 smithi165 ceph-mon[105517]: osdmap e1729: 8 total, 8 up, 8 in 2024-08-17T18:22:56.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:55 smithi165 ceph-mon[105517]: 58.f starting backfill to osd.2 from (0'0,0'0] MAX to 1726'8100 2024-08-17T18:22:56.309 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:56.359 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:56.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:56.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:56.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:55 smithi136 ceph-mon[109861]: osdmap e1729: 8 total, 8 up, 8 in 2024-08-17T18:22:56.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:55 smithi136 ceph-mon[109861]: 58.f starting backfill to osd.2 from (0'0,0'0] MAX to 1726'8100 2024-08-17T18:22:56.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:55 smithi136 ceph-mon[115078]: osdmap e1729: 8 total, 8 up, 8 in 2024-08-17T18:22:56.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:55 smithi136 ceph-mon[115078]: 58.f starting backfill to osd.2 from (0'0,0'0] MAX to 1726'8100 2024-08-17T18:22:56.461 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:56.498 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:56.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:56.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:56.600 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:22:56.632 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:56.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:56.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:22:56.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:56.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:56.790 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:56.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:56.892 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:56.921 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:56.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:56.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:57.023 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:57.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:57.081 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:57.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:57.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:57.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:56 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:22:57.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:56 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:22:57.210 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:57.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:57.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:57.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:56 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:22:57.311 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:57.356 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:57.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:57.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:57.458 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:57.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:57.495 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:57.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:57.596 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:57.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:57.631 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:57.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:57.732 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:57.771 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:57.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:57.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:57.873 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:22:57.911 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:57.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:57.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:22:57.938 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:22:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:22:57] "GET /metrics HTTP/1.1" 200 36779 "" "Prometheus/2.51.0" 2024-08-17T18:22:58.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:22:58.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:58.040 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:58.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:22:58.141 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:22:58.169 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:58.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:58.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:22:58.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:57 smithi136 ceph-mon[109861]: pgmap v3393: 99 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 6.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 532 op/s; 17041/299490 objects degraded (5.690%); 11350/299490 objects misplaced (3.790%); 3.3 MiB/s, 52 objects/s recovering 2024-08-17T18:22:58.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:57 smithi136 ceph-mon[115078]: pgmap v3393: 99 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 6.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 532 op/s; 17041/299490 objects degraded (5.690%); 11350/299490 objects misplaced (3.790%); 3.3 MiB/s, 52 objects/s recovering 2024-08-17T18:22:58.250 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:57 smithi165 ceph-mon[105517]: pgmap v3393: 99 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 6.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 532 op/s; 17041/299490 objects degraded (5.690%); 11350/299490 objects misplaced (3.790%); 3.3 MiB/s, 52 objects/s recovering 2024-08-17T18:22:58.271 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:58.316 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:58.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:58.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:58.418 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:58.455 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:58.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:58.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:58.557 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:22:58.583 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:58.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:58.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:22:58.685 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:22:58.742 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:58.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:58.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:22:58.844 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:58.886 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:58.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:58.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:58.989 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:22:59.037 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:59.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:59.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:22:59.139 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:22:59.167 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:59.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:59.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:22:59.190 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 4 1 2024-08-17T18:22:59.244 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:22:59.269 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:22:59.271 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:22:59.292 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:59.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:59.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:22:59.394 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:22:59.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:59.422 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:22:59.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:22:59.523 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:22:59.611 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:59.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:59.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:22:59.713 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:59.761 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:59.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:59.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:22:59.863 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:22:59.899 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:22:59.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:22:59.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:00.001 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:00.028 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:00.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:00.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:00.130 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:00.172 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:00.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:00.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:00.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:59 smithi136 ceph-mon[109861]: pgmap v3394: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 6.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 652 op/s; 21551/294120 objects degraded (7.327%); 12362/294120 objects misplaced (4.203%); 6.6 MiB/s, 106 objects/s recovering 2024-08-17T18:23:00.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:22:59 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 21551/294120 objects degraded (7.327%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T18:23:00.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:59 smithi136 ceph-mon[115078]: pgmap v3394: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 6.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 652 op/s; 21551/294120 objects degraded (7.327%); 12362/294120 objects misplaced (4.203%); 6.6 MiB/s, 106 objects/s recovering 2024-08-17T18:23:00.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:22:59 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 21551/294120 objects degraded (7.327%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T18:23:00.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:59 smithi165 ceph-mon[105517]: pgmap v3394: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 6.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 652 op/s; 21551/294120 objects degraded (7.327%); 12362/294120 objects misplaced (4.203%); 6.6 MiB/s, 106 objects/s recovering 2024-08-17T18:23:00.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:22:59 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 21551/294120 objects degraded (7.327%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T18:23:00.274 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:00.317 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:00.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:00.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:00.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:00.443 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:00.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:00.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:00.545 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:00.602 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:00.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:00.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:00.704 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:00.758 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:00.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:00.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:00.860 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:00.899 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:00.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:00.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:00.966 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:23:01.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:01.037 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:01.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:01.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:01.139 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:01.177 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:01.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:01.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:01.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:00 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/314995729' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:23:01.279 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:01.319 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:01.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:01.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:01.401 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:00 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/314995729' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:23:01.402 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:00 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/314995729' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:23:01.421 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:01.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:01.456 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:01.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:01.557 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:01.593 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:01.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:01.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:01.695 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:01.721 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:01.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:01.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:01.823 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:01.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:01.855 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:01.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:01.956 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:01.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:01.991 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:01.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:02.092 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:02.136 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:02.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:02.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:02.154 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:23:02.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:01 smithi136 ceph-mon[109861]: pgmap v3395: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 6.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 327 op/s; 21551/292773 objects degraded (7.361%); 12362/292773 objects misplaced (4.222%); 4.4 MiB/s, 70 objects/s recovering 2024-08-17T18:23:02.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:01 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:23:02.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:01 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/314995729' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:23:02.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:01 smithi136 ceph-mon[109861]: osdmap e1730: 8 total, 8 up, 8 in 2024-08-17T18:23:02.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:01 smithi136 ceph-mon[115078]: pgmap v3395: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 6.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 327 op/s; 21551/292773 objects degraded (7.361%); 12362/292773 objects misplaced (4.222%); 4.4 MiB/s, 70 objects/s recovering 2024-08-17T18:23:02.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:01 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:23:02.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:01 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/314995729' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:23:02.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:01 smithi136 ceph-mon[115078]: osdmap e1730: 8 total, 8 up, 8 in 2024-08-17T18:23:02.239 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:02.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:01 smithi165 ceph-mon[105517]: pgmap v3395: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 6.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 327 op/s; 21551/292773 objects degraded (7.361%); 12362/292773 objects misplaced (4.222%); 4.4 MiB/s, 70 objects/s recovering 2024-08-17T18:23:02.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:01 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:23:02.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:01 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/314995729' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:23:02.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:01 smithi165 ceph-mon[105517]: osdmap e1730: 8 total, 8 up, 8 in 2024-08-17T18:23:02.285 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:02.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:02.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:02.387 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:02.427 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:02.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:02.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:02.529 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:02.565 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:02.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:02.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:02.667 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:02.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:02.689 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:02.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:02.790 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:02.830 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:02.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:02.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:02.931 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:02.967 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:02.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:02.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:03.069 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:03.118 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:03.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:03.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:03.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:03.249 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:03.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:03.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:03.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:03.378 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:03.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:03.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:03.480 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:03.505 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:03.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:03.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:03.606 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:03.644 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:03.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:03.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:03.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:23:03.707 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:23:03.707 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:03 smithi136 ceph-mon[109861]: pgmap v3397: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 457 op/s; 21352/289332 objects degraded (7.380%); 12121/289332 objects misplaced (4.189%); 5.8 MiB/s, 92 objects/s recovering 2024-08-17T18:23:03.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:23:03.708 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:23:03.708 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:03 smithi136 ceph-mon[115078]: pgmap v3397: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 457 op/s; 21352/289332 objects degraded (7.380%); 12121/289332 objects misplaced (4.189%); 5.8 MiB/s, 92 objects/s recovering 2024-08-17T18:23:03.746 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:23:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:23:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:03 smithi165 ceph-mon[105517]: pgmap v3397: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 457 op/s; 21352/289332 objects degraded (7.380%); 12121/289332 objects misplaced (4.189%); 5.8 MiB/s, 92 objects/s recovering 2024-08-17T18:23:03.781 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:03.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:03.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:03.834 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:23:03.882 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:03.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:03.944 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:03.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:04.045 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:04.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:04.097 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:04.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:04.198 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:04.239 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:04.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:04.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:04.341 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:04.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:04.386 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:04.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:04.487 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:04.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:04.529 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:04.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:04.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:04.662 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:04.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:04.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:04.764 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:04.800 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:04.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:04.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:04.902 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:04.939 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:04.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:04.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:05.041 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:05.085 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:05.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:05.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:05.187 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:05.220 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:05.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:05.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:05.321 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:05.349 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:05.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:05.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:05.451 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:05.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:05.491 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:05.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:05.592 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:05.628 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:05.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:05.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:05.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:05.758 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:05.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:05.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:05.860 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:05.890 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:05.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:05.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:05 smithi136 ceph-mon[109861]: pgmap v3398: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 603 op/s; 21352/282843 objects degraded (7.549%); 11893/282843 objects misplaced (4.205%); 4.8 MiB/s, 76 objects/s recovering 2024-08-17T18:23:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:05 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 21352/282843 objects degraded (7.549%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T18:23:05.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:05 smithi136 ceph-mon[115078]: pgmap v3398: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 603 op/s; 21352/282843 objects degraded (7.549%); 11893/282843 objects misplaced (4.205%); 4.8 MiB/s, 76 objects/s recovering 2024-08-17T18:23:05.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:05 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 21352/282843 objects degraded (7.549%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T18:23:05.992 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:05 smithi165 ceph-mon[105517]: pgmap v3398: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 603 op/s; 21352/282843 objects degraded (7.549%); 11893/282843 objects misplaced (4.205%); 4.8 MiB/s, 76 objects/s recovering 2024-08-17T18:23:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:05 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 21352/282843 objects degraded (7.549%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T18:23:06.017 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:06.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:06.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:06.119 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:06.151 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:06.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:06.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:06.252 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:06.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:06.341 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:06.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:06.442 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:06.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:06.470 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:06.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:06.571 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:06.613 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:06.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:06.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:06.715 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:06.762 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:06.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:06.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:06.864 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:06.899 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:06.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:06.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:07.001 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:07.030 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:07.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:07.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:07.132 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:07.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:07.172 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:07.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:07.273 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:07.320 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:07.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:07.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:07.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:07.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:07.452 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:07.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:23:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:23:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:07 smithi165 ceph-mon[105517]: pgmap v3399: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 675 op/s; 21192/279207 objects degraded (7.590%); 11717/279207 objects misplaced (4.197%); 5.6 MiB/s, 90 objects/s recovering 2024-08-17T18:23:07.540 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:23:07.540 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:23:07.540 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:07 smithi136 ceph-mon[109861]: pgmap v3399: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 675 op/s; 21192/279207 objects degraded (7.590%); 11717/279207 objects misplaced (4.197%); 5.6 MiB/s, 90 objects/s recovering 2024-08-17T18:23:07.540 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:23:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:23:07] "GET /metrics HTTP/1.1" 200 36765 "" "Prometheus/2.51.0" 2024-08-17T18:23:07.541 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:23:07.541 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:23:07.541 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:07 smithi136 ceph-mon[115078]: pgmap v3399: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 675 op/s; 21192/279207 objects degraded (7.590%); 11717/279207 objects misplaced (4.197%); 5.6 MiB/s, 90 objects/s recovering 2024-08-17T18:23:07.553 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:07.589 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:07.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:07.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:07.690 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:07.731 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:07.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:07.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:07.833 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:07.859 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:07.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:07.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:07.961 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:08.023 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:08.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:08.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:08.125 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:08.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:08.176 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:08.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:08.278 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:08.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:08.330 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:08.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:08.421 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:23:08.422 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:23:08.432 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:08.463 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:23:08.464 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:23:08.465 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:08 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/942860599' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:23:08.465 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:08 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/942860599' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:23:08.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:08.480 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:08.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:08.499 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:23:08.499 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:23:08.541 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:23:08.542 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:23:08.570 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:23:08.571 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:23:08.581 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:08.606 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:23:08.606 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:23:08.623 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:08.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:08.639 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:23:08.639 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:23:08.669 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:23:08.669 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:23:08.725 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:08.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:08 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/942860599' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:23:08.766 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:08.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:08.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:08.868 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:08.901 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:08.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:08.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:09.003 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:09.029 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:09.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:09.131 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:09.178 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:09.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:09.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:09.280 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:09.307 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:09.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:09.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:09.409 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:09.450 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:09.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:09.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:09.525 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:09 smithi136 ceph-mon[109861]: pgmap v3400: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 21192/272061 objects degraded (7.789%); 11461/272061 objects misplaced (4.213%); 2.2 MiB/s, 35 objects/s recovering 2024-08-17T18:23:09.525 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:09 smithi136 ceph-mon[115078]: pgmap v3400: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 21192/272061 objects degraded (7.789%); 11461/272061 objects misplaced (4.213%); 2.2 MiB/s, 35 objects/s recovering 2024-08-17T18:23:09.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:09.589 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:09.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:09.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:09.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:09.717 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:09.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:09.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:09.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:09 smithi165 ceph-mon[105517]: pgmap v3400: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 21192/272061 objects degraded (7.789%); 11461/272061 objects misplaced (4.213%); 2.2 MiB/s, 35 objects/s recovering 2024-08-17T18:23:09.818 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:09.845 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:09.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:09.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:09.946 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:09.974 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:09.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:09.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:10.075 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:10.115 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:10.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:10.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:10.216 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:10.243 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:10.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:10.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:10.345 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:10.386 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:10.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:10.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:10.487 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:10.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:10.527 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:10.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:10.628 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:10.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:10.668 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:10.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:10.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:10 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 21192/272061 objects degraded (7.789%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:23:10.769 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:10.808 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:10.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:10.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:10.910 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:10.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:10 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 21192/272061 objects degraded (7.789%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:23:10.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:10 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 21192/272061 objects degraded (7.789%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:23:10.951 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:10.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:10.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:11.053 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:11.076 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:11.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:11.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:11.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:11.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:11.215 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:11.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:11.317 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:11.362 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:11.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:11.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:11.464 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:11.503 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:11.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:11.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:11.605 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:11.652 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:11.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:11.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:11.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:11 smithi136 ceph-mon[109861]: pgmap v3401: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 831 op/s; 20972/267810 objects degraded (7.831%); 11208/267810 objects misplaced (4.185%); 3.6 MiB/s, 57 objects/s recovering 2024-08-17T18:23:11.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:11 smithi136 ceph-mon[115078]: pgmap v3401: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 831 op/s; 20972/267810 objects degraded (7.831%); 11208/267810 objects misplaced (4.185%); 3.6 MiB/s, 57 objects/s recovering 2024-08-17T18:23:11.753 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:11.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:11 smithi165 ceph-mon[105517]: pgmap v3401: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 831 op/s; 20972/267810 objects degraded (7.831%); 11208/267810 objects misplaced (4.185%); 3.6 MiB/s, 57 objects/s recovering 2024-08-17T18:23:11.807 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:11.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:11.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:11.909 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:11.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:11.944 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:11.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:12.045 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:12.088 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:12.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:12.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:12.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:12.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:12.217 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:12.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:12.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:12.347 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:12.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:12.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:12.449 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:12.455 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:23:12.455 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:23:12.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.455 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:138374 58.7 58:eacd6b2c:::benchmark_data_smithi136_442124_object24758:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1730)", 2024-08-17T18:23:12.455 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:09.261455+0000", 2024-08-17T18:23:12.455 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013639788999999999, 2024-08-17T18:23:12.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013671139000000001, 2024-08-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 138374 2024-08-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:23:12.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.261455+0000", 2024-08-17T18:23:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:23:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:23:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.261455+0000", 2024-08-17T18:23:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:23:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:23:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.261457+0000", 2024-08-17T18:23:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.708e-06 2024-08-17T18:23:12.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.261462+0000", 2024-08-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0089999999999998e-06 2024-08-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.261477+0000", 2024-08-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4713e-05 2024-08-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:23:12.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.261484+0000", 2024-08-17T18:23:12.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2729999999999997e-06 2024-08-17T18:23:12.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:23:12.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.261566+0000", 2024-08-17T18:23:12.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1718999999999995e-05 2024-08-17T18:23:12.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:23:12.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.261775+0000", 2024-08-17T18:23:12.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020918900000000001 2024-08-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,6", 2024-08-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.262193+0000", 2024-08-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00041868199999999997 2024-08-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:138377 58.e 58:77863ac8:::benchmark_data_smithi136_442124_object24761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1730)", 2024-08-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:09.263319+0000", 2024-08-17T18:23:12.460 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.011775879, 2024-08-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011875976999999999, 2024-08-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 138377 2024-08-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:23:12.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.263319+0000", 2024-08-17T18:23:12.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:23:12.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:23:12.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.263319+0000", 2024-08-17T18:23:12.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:23:12.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:23:12.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.263320+0000", 2024-08-17T18:23:12.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0720000000000001e-06 2024-08-17T18:23:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:23:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.263325+0000", 2024-08-17T18:23:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5839999999999996e-06 2024-08-17T18:23:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:23:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.263335+0000", 2024-08-17T18:23:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0726e-05 2024-08-17T18:23:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:23:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.263342+0000", 2024-08-17T18:23:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7680000000000003e-06 2024-08-17T18:23:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:23:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.263416+0000", 2024-08-17T18:23:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4158000000000004e-05 2024-08-17T18:23:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:23:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.263531+0000", 2024-08-17T18:23:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114968 2024-08-17T18:23:12.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.263815+0000", 2024-08-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00028409099999999999 2024-08-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.267912+0000", 2024-08-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040963099999999997 2024-08-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:23:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.269014+0000", 2024-08-17T18:23:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011027299999999999 2024-08-17T18:23:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:23:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:23:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:23:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.36936.0:138380 58.5 e1730/1701 58:ae8d49e7:::benchmark_data_smithi136_442124_object24764:head v 1730'8611, mlcod=1730'8609)", 2024-08-17T18:23:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:09.265697+0000", 2024-08-17T18:23:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0093979419999999994, 2024-08-17T18:23:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0095654639999999992, 2024-08-17T18:23:12.466 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:23:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:23:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T18:23:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:23:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:23:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.265697+0000", 2024-08-17T18:23:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:23:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:23:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.265697+0000", 2024-08-17T18:23:12.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:23:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:23:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.265698+0000", 2024-08-17T18:23:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1699999999999997e-07 2024-08-17T18:23:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:23:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.265703+0000", 2024-08-17T18:23:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5130000000000004e-06 2024-08-17T18:23:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:23:12.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.265717+0000", 2024-08-17T18:23:12.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3292000000000001e-05 2024-08-17T18:23:12.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:23:12.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.265723+0000", 2024-08-17T18:23:12.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7780000000000002e-06 2024-08-17T18:23:12.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:23:12.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.268084+0000", 2024-08-17T18:23:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023606059999999999 2024-08-17T18:23:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:23:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.268129+0000", 2024-08-17T18:23:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4880999999999999e-05 2024-08-17T18:23:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:23:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:23:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:23:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.36936.0:138374 58.7 e1730/1725)", 2024-08-17T18:23:12.470 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:09.269184+0000", 2024-08-17T18:23:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0059113300000000002, 2024-08-17T18:23:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0061226850000000001, 2024-08-17T18:23:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:23:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:23:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-08-17T18:23:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:23:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:23:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.269184+0000", 2024-08-17T18:23:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:23:12.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:23:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.269184+0000", 2024-08-17T18:23:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:23:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:23:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.269184+0000", 2024-08-17T18:23:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4e-07 2024-08-17T18:23:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:23:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.269192+0000", 2024-08-17T18:23:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1280000000000002e-06 2024-08-17T18:23:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:23:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.269203+0000", 2024-08-17T18:23:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1306e-05 2024-08-17T18:23:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:23:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.269209+0000", 2024-08-17T18:23:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4489999999999996e-06 2024-08-17T18:23:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:23:12.473 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:23:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:23:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:138383 58.e 58:7ba0dd2b:::benchmark_data_smithi136_442124_object24767:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1730)", 2024-08-17T18:23:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:09.269654+0000", 2024-08-17T18:23:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0054411850000000003, 2024-08-17T18:23:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0056932850000000002, 2024-08-17T18:23:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:23:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:23:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-17T18:23:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:23:12.474 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:23:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:23:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 138383 2024-08-17T18:23:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:23:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:23:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.269654+0000", 2024-08-17T18:23:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:23:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:23:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.269654+0000", 2024-08-17T18:23:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:23:12.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:23:12.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.269655+0000", 2024-08-17T18:23:12.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6479999999999999e-06 2024-08-17T18:23:12.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:23:12.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.269660+0000", 2024-08-17T18:23:12.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.775e-06 2024-08-17T18:23:12.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:23:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.269674+0000", 2024-08-17T18:23:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3596e-05 2024-08-17T18:23:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:23:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.269680+0000", 2024-08-17T18:23:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.618e-06 2024-08-17T18:23:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:23:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.271232+0000", 2024-08-17T18:23:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001551385 2024-08-17T18:23:12.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.478 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.478 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:23:12.478 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.271343+0000", 2024-08-17T18:23:12.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111469 2024-08-17T18:23:12.478 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.478 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.478 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:23:12.478 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.271451+0000", 2024-08-17T18:23:12.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010789 2024-08-17T18:23:12.478 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:23:12.478 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:23:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:23:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.36936.0:138382 58.4 e1730/1701 58:2876b7c5:::benchmark_data_smithi136_442124_object24766:head v 1730'8861, mlcod=1730'8859)", 2024-08-17T18:23:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:09.271182+0000", 2024-08-17T18:23:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0039126270000000001, 2024-08-17T18:23:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0042208050000000002, 2024-08-17T18:23:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:23:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:23:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T18:23:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:23:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:23:12.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.271182+0000", 2024-08-17T18:23:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:23:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:23:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.271182+0000", 2024-08-17T18:23:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:23:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:23:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.271183+0000", 2024-08-17T18:23:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.031e-06 2024-08-17T18:23:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:23:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.271191+0000", 2024-08-17T18:23:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5329999999999997e-06 2024-08-17T18:23:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:23:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.273114+0000", 2024-08-17T18:23:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0019232940000000001 2024-08-17T18:23:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:23:12.481 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.273122+0000", 2024-08-17T18:23:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2109999999999998e-06 2024-08-17T18:23:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:23:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.273354+0000", 2024-08-17T18:23:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023135199999999999 2024-08-17T18:23:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:23:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.273399+0000", 2024-08-17T18:23:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5763999999999998e-05 2024-08-17T18:23:12.482 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:23:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:23:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:23:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.36936.0:138377 58.e e1730/1716)", 2024-08-17T18:23:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:09.275076+0000", 2024-08-17T18:23:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 1.8924e-05, 2024-08-17T18:23:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036851899999999999, 2024-08-17T18:23:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:23:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:23:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-08-17T18:23:12.483 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:23:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:23:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275076+0000", 2024-08-17T18:23:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:23:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:23:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275076+0000", 2024-08-17T18:23:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:23:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:23:12.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275077+0000", 2024-08-17T18:23:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0139999999999999e-06 2024-08-17T18:23:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:23:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275083+0000", 2024-08-17T18:23:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9649999999999996e-06 2024-08-17T18:23:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:23:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275094+0000", 2024-08-17T18:23:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1374e-05 2024-08-17T18:23:12.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:23:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275102+0000", 2024-08-17T18:23:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3540000000000002e-06 2024-08-17T18:23:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:23:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:23:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:23:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.36936.0:138384 58.8 e1730/1701 58:1e7c616e:::benchmark_data_smithi136_442124_object24768:head v 1730'8563, mlcod=1730'8561)", 2024-08-17T18:23:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:09.275126+0000", 2024-08-17T18:23:12.486 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4294967295.9999685, 2024-08-17T18:23:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00035600699999999998, 2024-08-17T18:23:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:23:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:23:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T18:23:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:23:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:23:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275126+0000", 2024-08-17T18:23:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:23:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.487 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:23:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275126+0000", 2024-08-17T18:23:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:23:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:23:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275127+0000", 2024-08-17T18:23:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4799999999999999e-07 2024-08-17T18:23:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:23:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275130+0000", 2024-08-17T18:23:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7419999999999999e-06 2024-08-17T18:23:12.488 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:23:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275137+0000", 2024-08-17T18:23:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7170000000000002e-06 2024-08-17T18:23:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:23:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275142+0000", 2024-08-17T18:23:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9420000000000003e-06 2024-08-17T18:23:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.489 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:23:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275186+0000", 2024-08-17T18:23:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4206999999999999e-05 2024-08-17T18:23:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:23:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275204+0000", 2024-08-17T18:23:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8046999999999998e-05 2024-08-17T18:23:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:23:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:23:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:23:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.490 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:138386 58.c 58:3cd8237c:::benchmark_data_smithi136_442124_object24770:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1730)", 2024-08-17T18:23:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:09.275162+0000", 2024-08-17T18:23:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4294967295.9999328, 2024-08-17T18:23:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000368682, 2024-08-17T18:23:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:23:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:23:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-17T18:23:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:23:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:23:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:23:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 138386 2024-08-17T18:23:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:23:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:23:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275162+0000", 2024-08-17T18:23:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:23:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:23:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275162+0000", 2024-08-17T18:23:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:23:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:23:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275163+0000", 2024-08-17T18:23:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1349999999999999e-06 2024-08-17T18:23:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:23:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275166+0000", 2024-08-17T18:23:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5050000000000002e-06 2024-08-17T18:23:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:23:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275175+0000", 2024-08-17T18:23:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6580000000000008e-06 2024-08-17T18:23:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:23:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275180+0000", 2024-08-17T18:23:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.767e-06 2024-08-17T18:23:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:23:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275278+0000", 2024-08-17T18:23:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7759999999999999e-05 2024-08-17T18:23:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:23:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275405+0000", 2024-08-17T18:23:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000126933 2024-08-17T18:23:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,6", 2024-08-17T18:23:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275527+0000", 2024-08-17T18:23:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012176900000000001 2024-08-17T18:23:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:23:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:23:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:23:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.36936.0:138383 58.e e1730/1716)", 2024-08-17T18:23:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:09.275233+0000", 2024-08-17T18:23:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4294967295.9998622, 2024-08-17T18:23:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00035013600000000002, 2024-08-17T18:23:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:23:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:23:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-08-17T18:23:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:23:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:23:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275233+0000", 2024-08-17T18:23:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:23:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:23:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275233+0000", 2024-08-17T18:23:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:23:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:23:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275233+0000", 2024-08-17T18:23:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7700000000000005e-07 2024-08-17T18:23:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:23:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275241+0000", 2024-08-17T18:23:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5089999999999996e-06 2024-08-17T18:23:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:23:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275249+0000", 2024-08-17T18:23:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8200000000000003e-06 2024-08-17T18:23:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:23:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:23:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:23:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:09.275254+0000", 2024-08-17T18:23:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8930000000000004e-06 2024-08-17T18:23:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:23:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:23:12.499 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:23:12.499 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:23:12.499 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-17T18:23:12.499 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 10 2024-08-17T18:23:12.499 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:23:12.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:12.501 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:12.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:12.602 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:12.623 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:12.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:12.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:12.725 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:12.746 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:12.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:12.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:12.848 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:12.890 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:12.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:12.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:12.992 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:13.023 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:13.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:13.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:13.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:13.153 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:13.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:13.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:13.255 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:13.289 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:13.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:13.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:13.347 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:23:13.391 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:13.417 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:13.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:13.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:13.519 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:13.569 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:13.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:13.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:13.671 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:13.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:13 smithi136 ceph-mon[109861]: pgmap v3402: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 736 op/s; 20972/267147 objects degraded (7.850%); 11208/267147 objects misplaced (4.195%); 3.1 MiB/s, 49 objects/s recovering 2024-08-17T18:23:13.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:13 smithi136 ceph-mon[115078]: pgmap v3402: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 736 op/s; 20972/267147 objects degraded (7.850%); 11208/267147 objects misplaced (4.195%); 3.1 MiB/s, 49 objects/s recovering 2024-08-17T18:23:13.702 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:13.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:13.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:13.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:13.841 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:13.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:13.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:13.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:13.986 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:13.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:13.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:13 smithi165 ceph-mon[105517]: pgmap v3402: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 736 op/s; 20972/267147 objects degraded (7.850%); 11208/267147 objects misplaced (4.195%); 3.1 MiB/s, 49 objects/s recovering 2024-08-17T18:23:14.088 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:14.127 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:14.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:14.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:14.229 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:14.272 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:14.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:14.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:14.374 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:14.403 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:14.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:14.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:14.505 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:14.533 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:14.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:14.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:14.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:14.663 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:14.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:14.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:14.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:14.815 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:14.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:14.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:14.918 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:14.972 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:14.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:14.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:15.074 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:15.119 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:15.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:15.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:15.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:15.249 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:15.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:15.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:15.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:15.378 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:15.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:15.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:15.480 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:15.513 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:15.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:15.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:15.580 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:23:15.608 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-17T18:23:15.615 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:15.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:15.704 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:15.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:15.805 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:15.846 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:15 smithi136 ceph-mon[109861]: pgmap v3403: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 785 op/s; 20972/261051 objects degraded (8.034%); 10977/261051 objects misplaced (4.205%); 2.0 MiB/s, 31 objects/s recovering 2024-08-17T18:23:15.846 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:15 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 20972/261051 objects degraded (8.034%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:23:15.846 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:15 smithi136 ceph-mon[115078]: pgmap v3403: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 785 op/s; 20972/261051 objects degraded (8.034%); 10977/261051 objects misplaced (4.205%); 2.0 MiB/s, 31 objects/s recovering 2024-08-17T18:23:15.846 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:15 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 20972/261051 objects degraded (8.034%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:23:15.887 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:15.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:15.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:15.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:16.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:15 smithi165 ceph-mon[105517]: pgmap v3403: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 785 op/s; 20972/261051 objects degraded (8.034%); 10977/261051 objects misplaced (4.205%); 2.0 MiB/s, 31 objects/s recovering 2024-08-17T18:23:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:15 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 20972/261051 objects degraded (8.034%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:23:16.014 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:16.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:16.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:16.116 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:16.163 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:16.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:16.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:16.264 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:16.292 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:16.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:16.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:16.394 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:16.442 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:16.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:16.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:16.544 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:16.596 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:16.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:16.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:16.698 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:16.739 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:16.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:16.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:16.841 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:16.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:16.873 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:16.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:16.974 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:17.003 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:17.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:17.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:17.105 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:17.140 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:17.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:17.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:17.242 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:17.279 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:17.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:17.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:17.381 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:17.422 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:17.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:17.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:17.524 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:17.549 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:17.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:17.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:17.578 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:23:17.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:17.665 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.4 to 1 (10000) 2024-08-17T18:23:17.677 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:17.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:17.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:17.685 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:17 smithi136 ceph-mon[109861]: pgmap v3404: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 719 op/s; 20741/256944 objects degraded (8.072%); 10737/256944 objects misplaced (4.179%); 3.2 MiB/s, 50 objects/s recovering 2024-08-17T18:23:17.686 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:17 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2882120595' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T18:23:17.686 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:17 smithi136 ceph-mon[115078]: pgmap v3404: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 719 op/s; 20741/256944 objects degraded (8.072%); 10737/256944 objects misplaced (4.179%); 3.2 MiB/s, 50 objects/s recovering 2024-08-17T18:23:17.686 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:17 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2882120595' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T18:23:17.687 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:23:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:23:17] "GET /metrics HTTP/1.1" 200 36765 "" "Prometheus/2.51.0" 2024-08-17T18:23:17.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:17.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:17.815 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:17.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:17.916 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:17.962 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:17.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:17.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:17 smithi165 ceph-mon[105517]: pgmap v3404: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 719 op/s; 20741/256944 objects degraded (8.072%); 10737/256944 objects misplaced (4.179%); 3.2 MiB/s, 50 objects/s recovering 2024-08-17T18:23:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:17 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2882120595' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T18:23:18.064 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:18.090 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:18.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:18.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:18.192 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:18.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:18.233 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:18.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:18.334 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:18.358 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:18.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:18.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:18.460 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:18.488 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:18.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:18.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:18.589 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:18.614 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:18.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:18.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:18.633 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 6 1 2024-08-17T18:23:18.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:18.744 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:18.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:18.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:18.845 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:18.864 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:18 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2882120595' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-17T18:23:18.864 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:18 smithi136 ceph-mon[109861]: osdmap e1731: 8 total, 8 up, 8 in 2024-08-17T18:23:18.865 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:18 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2882120595' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-17T18:23:18.865 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:18 smithi136 ceph-mon[115078]: osdmap e1731: 8 total, 8 up, 8 in 2024-08-17T18:23:18.870 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:18.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:18.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:18.971 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:19.009 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:19.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:19.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:18 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2882120595' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-17T18:23:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:18 smithi165 ceph-mon[105517]: osdmap e1731: 8 total, 8 up, 8 in 2024-08-17T18:23:19.112 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:19.139 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:19.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:19.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:19.241 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:19.284 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:19.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:19.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:19.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:19.414 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:19.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:19.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:19.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:19.560 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:19.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:19.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:19.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:19.690 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:19.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:19.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:19.792 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:19.829 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:19.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:19.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:19.931 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:19.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:19 smithi136 ceph-mon[109861]: pgmap v3406: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 744 op/s; 20741/249723 objects degraded (8.306%); 10485/249723 objects misplaced (4.199%); 2.8 MiB/s, 45 objects/s recovering 2024-08-17T18:23:19.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:19 smithi136 ceph-mon[115078]: pgmap v3406: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 744 op/s; 20741/249723 objects degraded (8.306%); 10485/249723 objects misplaced (4.199%); 2.8 MiB/s, 45 objects/s recovering 2024-08-17T18:23:19.967 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:19.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:19.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:19 smithi165 ceph-mon[105517]: pgmap v3406: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 744 op/s; 20741/249723 objects degraded (8.306%); 10485/249723 objects misplaced (4.199%); 2.8 MiB/s, 45 objects/s recovering 2024-08-17T18:23:20.068 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:20.103 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:20.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:20.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:20.205 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:20.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:20.242 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:20.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:20.343 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:20.392 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:20.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:20.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:20.494 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:20.521 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:20.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:20.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:20.623 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:20.653 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:20.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:20.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:20.703 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:23:20.755 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:20.782 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:20.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:20.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:20.883 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:20.910 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:20.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:20.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:21.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:20 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 20741/249723 objects degraded (8.306%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:23:21.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:20 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3289376361' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:23:21.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:20 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:23:21.012 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:21.046 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:21.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:21.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:21.148 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:21.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:20 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 20741/249723 objects degraded (8.306%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:23:21.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:20 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3289376361' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:23:21.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:20 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:23:21.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:20 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 20741/249723 objects degraded (8.306%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:23:21.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:20 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3289376361' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:23:21.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:20 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:23:21.199 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:21.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:21.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:21.300 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:21.329 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:21.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:21.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:21.431 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:21.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:21.474 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:21.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:21.575 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:21.610 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:21.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:21.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:21.712 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:21.752 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:21.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:21.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:21.854 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:21.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:21.887 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:21.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:21.988 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:21 smithi165 ceph-mon[105517]: pgmap v3407: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 730 op/s; 20506/245880 objects degraded (8.340%); 10272/245880 objects misplaced (4.178%); 2.9 MiB/s, 46 objects/s recovering 2024-08-17T18:23:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:21 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:23:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:21 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:23:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:21 smithi165 ceph-mon[105517]: osdmap e1732: 8 total, 8 up, 8 in 2024-08-17T18:23:22.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:23:22.017 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:22.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:22.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:22.119 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:22.159 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:22.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:22.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:21 smithi136 ceph-mon[109861]: pgmap v3407: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 730 op/s; 20506/245880 objects degraded (8.340%); 10272/245880 objects misplaced (4.178%); 2.9 MiB/s, 46 objects/s recovering 2024-08-17T18:23:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:21 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:23:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:21 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:23:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:21 smithi136 ceph-mon[109861]: osdmap e1732: 8 total, 8 up, 8 in 2024-08-17T18:23:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:23:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:21 smithi136 ceph-mon[115078]: pgmap v3407: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 730 op/s; 20506/245880 objects degraded (8.340%); 10272/245880 objects misplaced (4.178%); 2.9 MiB/s, 46 objects/s recovering 2024-08-17T18:23:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:21 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:23:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:21 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:23:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:21 smithi136 ceph-mon[115078]: osdmap e1732: 8 total, 8 up, 8 in 2024-08-17T18:23:22.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:23:22.261 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:22.300 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:22.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:22.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:22.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:22.427 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:22.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:22.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:22.529 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:22.560 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:22.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:22.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:22.567 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:23:22.662 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:22.687 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:22.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:22.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:22.789 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:22.812 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:22.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:22.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:22.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:22.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:22.937 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:22.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:23.038 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:23.059 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:23.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:23.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:23.161 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:23.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:23.183 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:23.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:23.284 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:23.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:23.315 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:23.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:23.416 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:23.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:23.448 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:23.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:23.549 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:23.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:23.587 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:23.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:23.689 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:23.712 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:23.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:23.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:23.814 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:23.855 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:23.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:23.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:23.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:23.986 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:23.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:23.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:24.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:23 smithi165 ceph-mon[105517]: pgmap v3409: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 660 op/s; 20506/245205 objects degraded (8.363%); 10272/245205 objects misplaced (4.189%); 3.6 MiB/s, 58 objects/s recovering 2024-08-17T18:23:24.087 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:24.117 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:24.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:24.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:24.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:23 smithi136 ceph-mon[109861]: pgmap v3409: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 660 op/s; 20506/245205 objects degraded (8.363%); 10272/245205 objects misplaced (4.189%); 3.6 MiB/s, 58 objects/s recovering 2024-08-17T18:23:24.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:23 smithi136 ceph-mon[115078]: pgmap v3409: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 660 op/s; 20506/245205 objects degraded (8.363%); 10272/245205 objects misplaced (4.189%); 3.6 MiB/s, 58 objects/s recovering 2024-08-17T18:23:24.219 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:24.257 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:24.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:24.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:24.359 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:24.389 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:24.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:24.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:24.490 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:24.513 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:24.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:24.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:24.615 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:24.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:24.657 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:24.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:24.758 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:24.797 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:24.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:24.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:24.899 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:24.931 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:24.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:24.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:25.033 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:25.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:25.061 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:25.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:25.162 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:25.198 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:25.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:25.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:25.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:25.337 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:25.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:25.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:25.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:25.473 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:25.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:25.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:25.575 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:25.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:25.612 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:25.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:25.713 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:25.755 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:25.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:25.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:25.857 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:25.900 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:25.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:25.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:26.002 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:26.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:25 smithi165 ceph-mon[105517]: pgmap v3410: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 4.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 743 op/s; 20506/239100 objects degraded (8.576%); 10051/239100 objects misplaced (4.204%); 1.8 MiB/s, 29 objects/s recovering 2024-08-17T18:23:26.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:25 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 20506/239100 objects degraded (8.576%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:23:26.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:26.030 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:26.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:26.063 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:25 smithi136 ceph-mon[115078]: pgmap v3410: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 4.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 743 op/s; 20506/239100 objects degraded (8.576%); 10051/239100 objects misplaced (4.204%); 1.8 MiB/s, 29 objects/s recovering 2024-08-17T18:23:26.063 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:25 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 20506/239100 objects degraded (8.576%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:23:26.064 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:25 smithi136 ceph-mon[109861]: pgmap v3410: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 4.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 743 op/s; 20506/239100 objects degraded (8.576%); 10051/239100 objects misplaced (4.204%); 1.8 MiB/s, 29 objects/s recovering 2024-08-17T18:23:26.065 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:25 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 20506/239100 objects degraded (8.576%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:23:26.131 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:26.167 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:26.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:26.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:26.268 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:26.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:26.326 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:26.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:26.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:26.451 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:26.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:26.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:26.491 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:23:26.491 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:23:26.535 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:23:26.536 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:23:26.553 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:26.575 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:23:26.576 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:23:26.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:26.603 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:26.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:26.624 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:23:26.625 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:23:26.663 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:23:26.663 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:23:26.700 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:23:26.701 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:23:26.704 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:26.734 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:26.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:26.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:26.746 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:23:26.747 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:23:26.784 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:23:26.784 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:23:26.836 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:26.866 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:26.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:26.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:26.915 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:26 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2638108717' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:23:26.917 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:26 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2638108717' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:23:26.968 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:27.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:26 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2638108717' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:23:27.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:27.022 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:27.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:27.123 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:27.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:27.151 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:27.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:27.252 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:27.290 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:27.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:27.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:27.392 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:27.430 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:27.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:27.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:27.532 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:27.568 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:27.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:27.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:27.670 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:27.701 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:27.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:27.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:27.719 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:23:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:23:27] "GET /metrics HTTP/1.1" 200 36759 "" "Prometheus/2.51.0" 2024-08-17T18:23:27.803 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:27.844 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:27.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:27.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:27.866 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:23:27.946 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:27.974 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:27.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:27.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:28.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:27 smithi165 ceph-mon[105517]: pgmap v3411: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 777 op/s; 20256/235437 objects degraded (8.604%); 9832/235437 objects misplaced (4.176%); 3.4 MiB/s, 54 objects/s recovering 2024-08-17T18:23:28.077 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:28.144 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:28.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:28.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:28.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:27 smithi136 ceph-mon[109861]: pgmap v3411: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 777 op/s; 20256/235437 objects degraded (8.604%); 9832/235437 objects misplaced (4.176%); 3.4 MiB/s, 54 objects/s recovering 2024-08-17T18:23:28.198 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:27 smithi136 ceph-mon[115078]: pgmap v3411: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 777 op/s; 20256/235437 objects degraded (8.604%); 9832/235437 objects misplaced (4.176%); 3.4 MiB/s, 54 objects/s recovering 2024-08-17T18:23:28.246 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:28.272 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:28.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:28.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:28.374 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:28.405 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:28.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:28.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:28.507 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:28.536 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:28.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:28.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:28.638 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:28.663 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:28.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:28.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:28.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:28.818 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:28.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:28.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:28.919 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:28.968 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:28.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:28.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:29.070 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:29.111 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:29.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:29.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:29.212 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:29.253 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:29.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:29.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:29.355 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:29.391 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:29.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:29.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:29.494 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:29.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:29.527 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:29.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:29.628 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:29.667 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:29.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:29.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:29.724 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:29 smithi136 ceph-mon[109861]: pgmap v3412: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 700 op/s; 20256/228693 objects degraded (8.857%); 9591/228693 objects misplaced (4.194%); 3.0 MiB/s, 48 objects/s recovering 2024-08-17T18:23:29.769 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:29.793 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:29.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:29.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:29.895 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:29.922 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:29.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:29.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:29 smithi165 ceph-mon[105517]: pgmap v3412: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 700 op/s; 20256/228693 objects degraded (8.857%); 9591/228693 objects misplaced (4.194%); 3.0 MiB/s, 48 objects/s recovering 2024-08-17T18:23:30.024 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:30.059 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:30.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:30.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:30.064 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:29 smithi136 ceph-mon[115078]: pgmap v3412: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 700 op/s; 20256/228693 objects degraded (8.857%); 9591/228693 objects misplaced (4.194%); 3.0 MiB/s, 48 objects/s recovering 2024-08-17T18:23:30.134 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:23:30.134 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:23:30.160 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:30.188 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:30.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:30.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:30.290 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:30.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:30.327 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:30.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:30.428 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:30.465 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:30.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:30.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:30.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:30.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:30.598 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:30.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:30.666 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:23:30.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:30.765 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:30.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:30.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:30.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:30.922 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:30.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:30.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:31.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:30 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 20256/228693 objects degraded (8.857%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:23:31.023 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:31.054 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:31.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:31.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:31.143 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:30 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 20256/228693 objects degraded (8.857%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:23:31.144 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:30 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 20256/228693 objects degraded (8.857%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:23:31.155 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:31.189 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-17T18:23:31.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:31.213 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:31.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:31.314 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:31.373 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:31.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:31.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:31.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:31.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:23:31.504 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:31.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:31.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:31.555 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:23:31.606 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:31.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:31.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:31.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:31.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:31.793 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:31.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:31.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:31.895 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:31.930 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:31.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:31.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:32.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:31 smithi165 ceph-mon[105517]: pgmap v3413: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 717 op/s; 19981/224328 objects degraded (8.907%); 9333/224328 objects misplaced (4.160%); 3.3 MiB/s, 52 objects/s recovering 2024-08-17T18:23:32.032 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:32.066 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:32.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:32.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:32.168 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:32.187 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:31 smithi136 ceph-mon[115078]: pgmap v3413: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 717 op/s; 19981/224328 objects degraded (8.907%); 9333/224328 objects misplaced (4.160%); 3.3 MiB/s, 52 objects/s recovering 2024-08-17T18:23:32.187 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:31 smithi136 ceph-mon[109861]: pgmap v3413: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 717 op/s; 19981/224328 objects degraded (8.907%); 9333/224328 objects misplaced (4.160%); 3.3 MiB/s, 52 objects/s recovering 2024-08-17T18:23:32.208 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:32.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:32.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:32.311 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:32.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:32.336 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:32.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:32.437 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:32.485 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:32.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:32.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:32.587 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:32.627 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:32.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:32.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:32.729 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:32.755 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:32.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:32.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:32.857 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:32.884 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:32.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:32.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:32.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:33.011 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:33.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:33.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:33.112 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:33.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:33.155 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:33.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:33.256 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:33.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:33.309 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:33.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:33.410 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:33.453 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:33.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:33.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:33.556 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:33.603 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:33.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:33.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:33.705 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:33.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:33.739 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:33.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:33.840 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:33.888 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:33.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:33.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:33.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:33 smithi136 ceph-mon[109861]: pgmap v3414: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 4.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 623 op/s; 19981/223653 objects degraded (8.934%); 9333/223653 objects misplaced (4.173%); 2.8 MiB/s, 44 objects/s recovering 2024-08-17T18:23:33.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:33 smithi136 ceph-mon[115078]: pgmap v3414: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 4.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 623 op/s; 19981/223653 objects degraded (8.934%); 9333/223653 objects misplaced (4.173%); 2.8 MiB/s, 44 objects/s recovering 2024-08-17T18:23:33.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:34.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:33 smithi165 ceph-mon[105517]: pgmap v3414: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 4.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 623 op/s; 19981/223653 objects degraded (8.934%); 9333/223653 objects misplaced (4.173%); 2.8 MiB/s, 44 objects/s recovering 2024-08-17T18:23:34.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:34.022 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:34.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:34.123 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:34.156 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:34.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:34.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:34.258 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:34.286 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:34.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:34.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:34.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:34.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:34.418 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:34.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:34.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:34.547 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:34.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:34.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:34.649 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:34.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:34.718 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:34.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:34.819 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:34.845 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:34.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:34.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:34.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:34.976 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:34.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:34.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:35.078 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:35.137 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:35.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:35.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:35.239 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:35.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:35.286 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:35.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:35.310 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.6 to 1 (10000) 2024-08-17T18:23:35.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:35.411 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:35.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:35.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:35.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:35 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:23:35.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:35 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:23:35.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:35 smithi136 ceph-mon[109861]: pgmap v3415: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 4.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 775 op/s; 19981/217257 objects degraded (9.197%); 9108/217257 objects misplaced (4.192%); 2.7 MiB/s, 43 objects/s recovering 2024-08-17T18:23:35.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:35 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 19981/223653 objects degraded (8.934%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:23:35.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:35 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3015579094' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T18:23:35.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:35 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T18:23:35.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:35 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:23:35.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:35 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:23:35.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:35 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:23:35.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:35 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:23:35.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:35 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:23:35.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:35 smithi136 ceph-mon[115078]: pgmap v3415: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 4.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 775 op/s; 19981/217257 objects degraded (9.197%); 9108/217257 objects misplaced (4.192%); 2.7 MiB/s, 43 objects/s recovering 2024-08-17T18:23:35.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:35 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 19981/223653 objects degraded (8.934%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:23:35.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:35 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3015579094' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T18:23:35.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:35 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T18:23:35.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:35 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:23:35.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:35 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:23:35.448 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:35 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:23:35.513 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:35.536 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:35.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:35.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:35.638 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:35.659 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:35.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:35.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:35.760 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:35 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:23:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:35 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:23:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:35 smithi165 ceph-mon[105517]: pgmap v3415: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 4.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 775 op/s; 19981/217257 objects degraded (9.197%); 9108/217257 objects misplaced (4.192%); 2.7 MiB/s, 43 objects/s recovering 2024-08-17T18:23:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:35 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 19981/223653 objects degraded (8.934%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T18:23:35.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:35 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3015579094' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T18:23:35.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:35 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T18:23:35.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:35 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:23:35.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:35 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:23:35.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:35 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:23:35.783 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:35.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:35.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:35.885 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:35.907 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:35.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:35.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:36.008 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:36.031 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:36.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:36.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:36.132 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:36.154 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:36.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:36.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:36.255 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:36.277 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:36.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:36.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:36.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:36.400 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:36.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:36.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:36.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:36 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T18:23:36.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:36 smithi136 ceph-mon[109861]: osdmap e1733: 8 total, 8 up, 8 in 2024-08-17T18:23:36.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:23:36.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:36 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T18:23:36.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:36 smithi136 ceph-mon[115078]: osdmap e1733: 8 total, 8 up, 8 in 2024-08-17T18:23:36.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:23:36.482 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 0 1 2024-08-17T18:23:36.501 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:36.502 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:23:36.522 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:36.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:36.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:36.624 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:36.646 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:36.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:36.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:36.728 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:23:36.748 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:36.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:36 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T18:23:36.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:36 smithi165 ceph-mon[105517]: osdmap e1733: 8 total, 8 up, 8 in 2024-08-17T18:23:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:23:36.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:36.775 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:36.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:36.876 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:36.898 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:36.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:36.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:37.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:37.021 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:37.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:37.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:37.123 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:37.145 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:37.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:37.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:37.247 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:37.269 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:37.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:37.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:37.318 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:23:37.371 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:37.394 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:37.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:37.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:37.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:37.518 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:37.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:37.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:37.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:37.651 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:37.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:37.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:37.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:23:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:23:37] "GET /metrics HTTP/1.1" 200 36748 "" "Prometheus/2.51.0" 2024-08-17T18:23:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:37 smithi136 ceph-mon[109861]: osdmap e1734: 8 total, 8 up, 8 in 2024-08-17T18:23:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:37 smithi136 ceph-mon[109861]: pgmap v3418: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 642 op/s; 19703/213267 objects degraded (9.239%); 8870/213267 objects misplaced (4.159%); 4.3 MiB/s, 69 objects/s recovering 2024-08-17T18:23:37.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:37 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3575088173' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:23:37.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:37 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:23:37.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:37 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:23:37.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:37 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:23:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:37 smithi136 ceph-mon[115078]: osdmap e1734: 8 total, 8 up, 8 in 2024-08-17T18:23:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:37 smithi136 ceph-mon[115078]: pgmap v3418: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 642 op/s; 19703/213267 objects degraded (9.239%); 8870/213267 objects misplaced (4.159%); 4.3 MiB/s, 69 objects/s recovering 2024-08-17T18:23:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:37 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3575088173' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:23:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:37 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:23:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:37 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:23:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:37 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:23:37.753 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:37.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:37 smithi165 ceph-mon[105517]: osdmap e1734: 8 total, 8 up, 8 in 2024-08-17T18:23:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:37 smithi165 ceph-mon[105517]: pgmap v3418: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 642 op/s; 19703/213267 objects degraded (9.239%); 8870/213267 objects misplaced (4.159%); 4.3 MiB/s, 69 objects/s recovering 2024-08-17T18:23:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:37 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3575088173' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:23:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:37 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:23:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:37 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:23:37.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:37 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:23:37.791 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:37.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:37.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:37.893 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:37.928 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:37.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:37.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:38.029 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:38.077 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:38.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:38.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:38.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:38.218 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:38.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:38.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:38.321 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:38.363 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:38.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:38.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:38.465 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:38.492 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:38.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:38.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:38.594 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:38.622 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:38.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:38.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:38.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:38 smithi136 ceph-mon[109861]: 58.9 starting backfill to osd.6 from (0'0,0'0] MAX to 1732'10125 2024-08-17T18:23:38.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:38 smithi136 ceph-mon[109861]: osdmap e1735: 8 total, 8 up, 8 in 2024-08-17T18:23:38.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:38 smithi136 ceph-mon[109861]: 58.4 starting backfill to osd.6 from (0'0,0'0] MAX to 1732'10075 2024-08-17T18:23:38.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:38 smithi136 ceph-mon[115078]: 58.9 starting backfill to osd.6 from (0'0,0'0] MAX to 1732'10125 2024-08-17T18:23:38.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:38 smithi136 ceph-mon[115078]: osdmap e1735: 8 total, 8 up, 8 in 2024-08-17T18:23:38.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:38 smithi136 ceph-mon[115078]: 58.4 starting backfill to osd.6 from (0'0,0'0] MAX to 1732'10075 2024-08-17T18:23:38.724 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:38.759 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:38.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:38.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:38.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:38 smithi165 ceph-mon[105517]: 58.9 starting backfill to osd.6 from (0'0,0'0] MAX to 1732'10125 2024-08-17T18:23:38.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:38 smithi165 ceph-mon[105517]: osdmap e1735: 8 total, 8 up, 8 in 2024-08-17T18:23:38.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:38 smithi165 ceph-mon[105517]: 58.4 starting backfill to osd.6 from (0'0,0'0] MAX to 1732'10075 2024-08-17T18:23:38.860 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:38.897 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:38.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:38.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:38.999 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:39.035 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:39.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:39.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:39.137 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:39.171 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:39.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:39.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:39.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:39.302 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:39.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:39.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:39.404 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:39.436 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:39.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:39.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:39.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:39.561 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:39 smithi136 ceph-mon[109861]: osdmap e1736: 8 total, 8 up, 8 in 2024-08-17T18:23:39.561 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:39 smithi136 ceph-mon[109861]: pgmap v3421: 99 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 676 op/s; 19156/209142 objects degraded (9.159%); 13205/209142 objects misplaced (6.314%); 4.6 MiB/s, 74 objects/s recovering 2024-08-17T18:23:39.561 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:39 smithi136 ceph-mon[109861]: osdmap e1737: 8 total, 8 up, 8 in 2024-08-17T18:23:39.561 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:39 smithi136 ceph-mon[115078]: osdmap e1736: 8 total, 8 up, 8 in 2024-08-17T18:23:39.561 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:39 smithi136 ceph-mon[115078]: pgmap v3421: 99 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 676 op/s; 19156/209142 objects degraded (9.159%); 13205/209142 objects misplaced (6.314%); 4.6 MiB/s, 74 objects/s recovering 2024-08-17T18:23:39.561 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:39 smithi136 ceph-mon[115078]: osdmap e1737: 8 total, 8 up, 8 in 2024-08-17T18:23:39.564 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:39.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:39.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:39.666 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:39.707 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:39.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:39.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:39.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:39 smithi165 ceph-mon[105517]: osdmap e1736: 8 total, 8 up, 8 in 2024-08-17T18:23:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:39 smithi165 ceph-mon[105517]: pgmap v3421: 99 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 676 op/s; 19156/209142 objects degraded (9.159%); 13205/209142 objects misplaced (6.314%); 4.6 MiB/s, 74 objects/s recovering 2024-08-17T18:23:39.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:39 smithi165 ceph-mon[105517]: osdmap e1737: 8 total, 8 up, 8 in 2024-08-17T18:23:39.809 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:39.839 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:39.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:39.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:39.941 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:39.974 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:39.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:39.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:40.076 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:40.123 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:40.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:40.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:40.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:40.254 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:40.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:40.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:40.356 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:40.386 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:40.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:40.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:40.487 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:40.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:40.527 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:40.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:40.603 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:40 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 19156/209142 objects degraded (9.159%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:23:40.603 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:40 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 19156/209142 objects degraded (9.159%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:23:40.628 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:40.653 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:40.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:40.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:40.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:40 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 19156/209142 objects degraded (9.159%), 8 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T18:23:40.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:40.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:40.779 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:40.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:40.880 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:40.901 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:40.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:40.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:41.003 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:41.025 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:41.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:41.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:41.127 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:41.149 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:41.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:41.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:41.251 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:41.272 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:41.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:41.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:41.374 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:41.396 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:41.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:41.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:41.498 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:41.530 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:41.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:41.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:41.555 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:23:41.555 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:23:41.611 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:23:41.632 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:41.645 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:41 smithi136 ceph-mon[109861]: osdmap e1738: 8 total, 8 up, 8 in 2024-08-17T18:23:41.645 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:41 smithi136 ceph-mon[109861]: pgmap v3424: 99 pgs: 1 peering, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 538 op/s; 18600/206802 objects degraded (8.994%); 13086/206802 objects misplaced (6.328%); 9.0 MiB/s, 205 objects/s recovering 2024-08-17T18:23:41.645 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:41 smithi136 ceph-mon[115078]: osdmap e1738: 8 total, 8 up, 8 in 2024-08-17T18:23:41.645 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:41 smithi136 ceph-mon[115078]: pgmap v3424: 99 pgs: 1 peering, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 538 op/s; 18600/206802 objects degraded (8.994%); 13086/206802 objects misplaced (6.328%); 9.0 MiB/s, 205 objects/s recovering 2024-08-17T18:23:41.702 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:41.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:41.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:41.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:41 smithi165 ceph-mon[105517]: osdmap e1738: 8 total, 8 up, 8 in 2024-08-17T18:23:41.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:41 smithi165 ceph-mon[105517]: pgmap v3424: 99 pgs: 1 peering, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 538 op/s; 18600/206802 objects degraded (8.994%); 13086/206802 objects misplaced (6.328%); 9.0 MiB/s, 205 objects/s recovering 2024-08-17T18:23:41.804 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:41.859 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:41.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:41.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:41.960 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:41.992 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:41.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:41.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:42.094 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:42.131 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:42.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:42.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:42.233 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:42.271 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:42.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:42.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:42.319 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:23:42.335 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:23:42.335 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:23:42.374 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:42.382 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:23:42.382 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:23:42.403 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:42.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:42.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:42.414 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:23:42.415 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:23:42.415 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:42 smithi136 ceph-mon[109861]: osdmap e1739: 8 total, 8 up, 8 in 2024-08-17T18:23:42.415 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:42 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/320490015' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:23:42.415 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:42 smithi136 ceph-mon[115078]: osdmap e1739: 8 total, 8 up, 8 in 2024-08-17T18:23:42.416 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:42 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/320490015' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:23:42.470 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:23:42.471 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:23:42.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:42.505 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:23:42.505 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:23:42.543 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:23:42.543 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:23:42.556 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:42.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:42.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:42.576 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:23:42.576 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:23:42.609 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:23:42.609 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:23:42.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:42 smithi165 ceph-mon[105517]: osdmap e1739: 8 total, 8 up, 8 in 2024-08-17T18:23:42.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:42 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/320490015' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:23:42.659 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:42.686 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:42.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:42.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:42.788 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:42.823 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-17T18:23:42.835 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:42.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:42.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:42.937 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:42.971 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:42.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:42.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:43.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:43.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:43.112 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:43.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:43.126 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:23:43.213 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:43.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:43.247 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:43.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:43.348 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:43.381 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:43.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:43.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:43.395 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:43 smithi136 ceph-mon[109861]: pgmap v3426: 99 pgs: 1 peering, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 230 op/s; 18600/206193 objects degraded (9.021%); 13086/206193 objects misplaced (6.346%); 8.2 MiB/s, 188 objects/s recovering 2024-08-17T18:23:43.482 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:43.504 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:23:43.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:43.540 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:43.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:43.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:43 smithi165 ceph-mon[105517]: pgmap v3426: 99 pgs: 1 peering, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 230 op/s; 18600/206193 objects degraded (9.021%); 13086/206193 objects misplaced (6.346%); 8.2 MiB/s, 188 objects/s recovering 2024-08-17T18:23:43.641 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:43.654 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:43 smithi136 ceph-mon[115078]: pgmap v3426: 99 pgs: 1 peering, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 230 op/s; 18600/206193 objects degraded (9.021%); 13086/206193 objects misplaced (6.346%); 8.2 MiB/s, 188 objects/s recovering 2024-08-17T18:23:43.683 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:43.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:43.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:43.785 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:43.813 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:43.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:43.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:43.915 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:43.966 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:43.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:43.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:44.068 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:44.113 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:44.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:44.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:44.215 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:44.245 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:44.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:44.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:44.346 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:44.379 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:44.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:44.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:44.481 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:44.506 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:44.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:44.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:44.609 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:44.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:44.639 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:44.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:44.739 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:44.769 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:44.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:44.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:44.871 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:44.899 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:44.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:44.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:45.002 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:45.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:45.024 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:45.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:45.125 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:45.164 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:45.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:45.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:45.266 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:45.268 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:23:45.268 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:23:45.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.268 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:162346 58.f e1739/1728 58:f1e781ff:::benchmark_data_smithi136_442124_object48730:head v 1739'10293, mlcod=1711'4451)", 2024-08-17T18:23:45.268 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:23:45.263061+0000", 2024-08-17T18:23:45.268 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.003024307, 2024-08-17T18:23:45.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030528479999999999, 2024-08-17T18:23:45.268 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:23:45.268 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:23:45.268 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:23:45.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:23:45.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.263061+0000", 2024-08-17T18:23:45.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:45.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:23:45.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.263061+0000", 2024-08-17T18:23:45.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:45.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:23:45.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.263060+0000", 2024-08-17T18:23:45.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:23:45.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:23:45.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.263075+0000", 2024-08-17T18:23:45.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5807e-05 2024-08-17T18:23:45.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:23:45.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.263088+0000", 2024-08-17T18:23:45.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.241e-05 2024-08-17T18:23:45.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:23:45.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.263097+0000", 2024-08-17T18:23:45.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8410000000000006e-06 2024-08-17T18:23:45.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:45.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.263136+0000", 2024-08-17T18:23:45.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9053e-05 2024-08-17T18:23:45.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:23:45.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.263165+0000", 2024-08-17T18:23:45.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9045000000000001e-05 2024-08-17T18:23:45.272 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:45.272 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:45.272 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:45.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.272 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:162347 58.7 e1739/1725 58:e5772111:::benchmark_data_smithi136_442124_object48731:head v 1739'10282, mlcod=1711'4442)", 2024-08-17T18:23:45.272 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:23:45.264110+0000", 2024-08-17T18:23:45.272 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.001975671, 2024-08-17T18:23:45.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002067869, 2024-08-17T18:23:45.273 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:23:45.273 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:23:45.273 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:23:45.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:23:45.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.264110+0000", 2024-08-17T18:23:45.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:45.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.273 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:23:45.273 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.264110+0000", 2024-08-17T18:23:45.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:45.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:23:45.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.264109+0000", 2024-08-17T18:23:45.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:23:45.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:23:45.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.264115+0000", 2024-08-17T18:23:45.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4320000000000004e-06 2024-08-17T18:23:45.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:23:45.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.264129+0000", 2024-08-17T18:23:45.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3448999999999999e-05 2024-08-17T18:23:45.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:23:45.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.264134+0000", 2024-08-17T18:23:45.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3589999999999999e-06 2024-08-17T18:23:45.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:45.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.264172+0000", 2024-08-17T18:23:45.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8417000000000002e-05 2024-08-17T18:23:45.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:23:45.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.264196+0000", 2024-08-17T18:23:45.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3958999999999999e-05 2024-08-17T18:23:45.276 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:45.276 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:45.276 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:45.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.277 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(58.a 1739/1725 [PushOp(58:53a5f08d:::benchmark_data_smithi136_442124_object74728:head, version: 1713'4679, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(58:53a5f08d:::benchmark_data_smithi136_442124_object74728:head@1713'4679, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-17T18:23:45.277 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:23:45.264249+0000", 2024-08-17T18:23:45.277 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0018365359999999999, 2024-08-17T18:23:45.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002004657, 2024-08-17T18:23:45.277 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:23:45.277 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:23:45.277 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:23:45.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:23:45.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.264249+0000", 2024-08-17T18:23:45.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:45.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:23:45.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.264249+0000", 2024-08-17T18:23:45.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:45.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:23:45.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.264248+0000", 2024-08-17T18:23:45.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:23:45.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:23:45.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.264292+0000", 2024-08-17T18:23:45.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4434999999999997e-05 2024-08-17T18:23:45.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:23:45.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.264312+0000", 2024-08-17T18:23:45.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0281000000000001e-05 2024-08-17T18:23:45.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:23:45.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.264326+0000", 2024-08-17T18:23:45.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3434e-05 2024-08-17T18:23:45.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:45.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.264416+0000", 2024-08-17T18:23:45.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0056999999999995e-05 2024-08-17T18:23:45.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:23:45.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.264429+0000", 2024-08-17T18:23:45.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3574000000000001e-05 2024-08-17T18:23:45.280 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:45.280 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:45.280 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:45.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.280 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:162344 58.a e1739/1725)", 2024-08-17T18:23:45.280 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:23:45.264692+0000", 2024-08-17T18:23:45.281 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.0013928670000000001, 2024-08-17T18:23:45.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001610525, 2024-08-17T18:23:45.281 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:23:45.281 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "queued for pg", 2024-08-17T18:23:45.281 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:23:45.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:23:45.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.264692+0000", 2024-08-17T18:23:45.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:45.281 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.281 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.281 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:23:45.281 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.264692+0000", 2024-08-17T18:23:45.281 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:45.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:23:45.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.264692+0000", 2024-08-17T18:23:45.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:23:45.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:23:45.282 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.264697+0000", 2024-08-17T18:23:45.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5180000000000004e-06 2024-08-17T18:23:45.282 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.282 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.282 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:23:45.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.264706+0000", 2024-08-17T18:23:45.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6549999999999993e-06 2024-08-17T18:23:45.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:45.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:45.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:23:45.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:45.264711+0000", 2024-08-17T18:23:45.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8080000000000004e-06 2024-08-17T18:23:45.283 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:45.283 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:45.283 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:45.283 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:45.283 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-08-17T18:23:45.283 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 4 2024-08-17T18:23:45.283 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:23:45.308 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:45.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:45.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:45.410 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:45.439 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:45.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:45.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:45.513 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:23:45.541 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:45.576 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:45.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:45.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:45.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:45.703 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:45.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:45.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:45.805 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:45.852 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:45.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:45.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:45 smithi136 ceph-mon[109861]: pgmap v3427: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 peering, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 449 op/s; 17953/201042 objects degraded (8.930%); 12722/201042 objects misplaced (6.328%); 10 MiB/s, 200 objects/s recovering 2024-08-17T18:23:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:45 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 17953/201042 objects degraded (8.930%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:23:45.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:45 smithi136 ceph-mon[115078]: pgmap v3427: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 peering, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 449 op/s; 17953/201042 objects degraded (8.930%); 12722/201042 objects misplaced (6.328%); 10 MiB/s, 200 objects/s recovering 2024-08-17T18:23:45.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:45 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 17953/201042 objects degraded (8.930%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:23:45.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:45.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:45.987 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:45.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:46.009 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:45 smithi165 ceph-mon[105517]: pgmap v3427: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 peering, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 449 op/s; 17953/201042 objects degraded (8.930%); 12722/201042 objects misplaced (6.328%); 10 MiB/s, 200 objects/s recovering 2024-08-17T18:23:46.009 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:45 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 17953/201042 objects degraded (8.930%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:23:46.045 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-17T18:23:46.088 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:46.122 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:46.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:46.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:46.224 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:46.257 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:46.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:46.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:46.359 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:46.404 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:46.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:46.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:46.406 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:23:46.507 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:46.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:46.538 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:46.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:46.639 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:46.680 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:46.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:46.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:46.781 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:46.806 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:46.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:46.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:46.911 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:46.948 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:46.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:46.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:47.050 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:47.090 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:47.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:47.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:47.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:47.223 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:47.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:47.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:47.325 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:47.356 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:47.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:47.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:47.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:47.486 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:47.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:47.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:47.587 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:47.622 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:47.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:47.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:47.724 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:47.752 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:47.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:47.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:47.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:47.889 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:47.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:47.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:47.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:47 smithi136 ceph-mon[109861]: pgmap v3428: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 523 op/s; 17953/197751 objects degraded (9.079%); 12534/197751 objects misplaced (6.338%); 8.2 MiB/s, 165 objects/s recovering 2024-08-17T18:23:47.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:23:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:23:47] "GET /metrics HTTP/1.1" 200 36748 "" "Prometheus/2.51.0" 2024-08-17T18:23:47.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:47 smithi136 ceph-mon[115078]: pgmap v3428: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 523 op/s; 17953/197751 objects degraded (9.079%); 12534/197751 objects misplaced (6.338%); 8.2 MiB/s, 165 objects/s recovering 2024-08-17T18:23:47.980 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:47 smithi165 ceph-mon[105517]: pgmap v3428: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 523 op/s; 17953/197751 objects degraded (9.079%); 12534/197751 objects misplaced (6.338%); 8.2 MiB/s, 165 objects/s recovering 2024-08-17T18:23:47.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:48.030 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:48.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:48.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:48.132 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:48.160 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:48.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:48.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:48.261 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:48.293 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:48.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:48.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:48.395 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:48.435 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:48.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:48.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:48.536 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:48.569 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:48.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:48.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:48.671 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:48.707 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:48.707 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:23:48.708 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:23:48.708 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-08-17T18:23:48.708 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-08-17T18:23:48.708 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:23:48.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:48.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:48.809 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:48.832 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:48.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:48.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:48.934 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:48.977 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:48.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:48.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:49.078 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:49.110 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:49.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:49.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:49.212 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:49.238 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:49.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:49.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:49.339 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:49.384 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:49.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:49.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:49.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-17T18:23:49.485 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:49.538 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:49.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:49.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:49.641 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:49.672 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:49.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:49.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:49.737 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:23:49.774 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:49.816 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:49.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:49.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:49.918 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:49.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:49 smithi136 ceph-mon[109861]: pgmap v3429: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 700 op/s; 17512/191112 objects degraded (9.163%); 12081/191112 objects misplaced (6.321%); 11 MiB/s, 199 objects/s recovering 2024-08-17T18:23:49.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:49 smithi136 ceph-mon[115078]: pgmap v3429: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 700 op/s; 17512/191112 objects degraded (9.163%); 12081/191112 objects misplaced (6.321%); 11 MiB/s, 199 objects/s recovering 2024-08-17T18:23:49.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:49.947 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:49.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:49 smithi165 ceph-mon[105517]: pgmap v3429: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 700 op/s; 17512/191112 objects degraded (9.163%); 12081/191112 objects misplaced (6.321%); 11 MiB/s, 199 objects/s recovering 2024-08-17T18:23:50.049 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:50.082 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:50.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:50.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:50.184 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:50.227 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:50.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:50.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:50.329 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:50.370 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:50.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:50.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:50.472 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:50.495 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:50.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:50.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:50.597 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:50.633 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:50.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:50.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:50.735 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:50.765 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:50.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:50.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:50.867 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:50.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:50.904 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:50.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:50.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:50 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 17512/191112 objects degraded (9.163%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:23:50.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:50 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 17512/191112 objects degraded (9.163%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:23:51.006 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:51.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:50 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 17512/191112 objects degraded (9.163%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:23:51.038 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:51.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:51.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:51.140 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:51.178 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:51.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:51.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:51.280 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:51.321 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:51.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:51.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:51.422 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:51.462 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:51.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:51.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:51.564 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:51.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:51.610 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:51.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:51.709 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:51 smithi165 ceph-mon[105517]: pgmap v3430: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 650 op/s; 17512/187278 objects degraded (9.351%); 11890/187278 objects misplaced (6.349%); 5.2 MiB/s, 83 objects/s recovering 2024-08-17T18:23:51.709 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:23:51.711 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:51.744 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:51.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:51.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:51.777 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:23:51.777 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T18:23:51.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T18:23:51.778 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:23:51.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.778 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:117397 58.1 58.be219b71 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e1724)", 2024-08-17T18:23:51.778 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:35.829777+0000", 2024-08-17T18:23:51.778 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.946001434999999, 2024-08-17T18:23:51.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.000250192, 2024-08-17T18:23:51.778 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:23:51.778 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:23:51.778 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:23:51.778 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:23:51.778 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:23:51.778 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 117397 2024-08-17T18:23:51.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.779 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:23:51.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:23:51.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829777+0000", 2024-08-17T18:23:51.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:23:51.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829777+0000", 2024-08-17T18:23:51.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:23:51.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829776+0000", 2024-08-17T18:23:51.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:23:51.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:23:51.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829782+0000", 2024-08-17T18:23:51.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9379999999999997e-06 2024-08-17T18:23:51.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:23:51.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829793+0000", 2024-08-17T18:23:51.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1321999999999999e-05 2024-08-17T18:23:51.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:23:51.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829802+0000", 2024-08-17T18:23:51.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4770000000000003e-06 2024-08-17T18:23:51.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:51.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829919+0000", 2024-08-17T18:23:51.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000117695 2024-08-17T18:23:51.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:23:51.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829942+0000", 2024-08-17T18:23:51.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2864e-05 2024-08-17T18:23:51.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:51.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:36.830017+0000", 2024-08-17T18:23:51.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.000074567 2024-08-17T18:23:51.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:23:51.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:36.830028+0000", 2024-08-17T18:23:51.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0794e-05 2024-08-17T18:23:51.782 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.782 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:51.782 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.783 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:117402 58.1 58.b36cf831 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e1724)", 2024-08-17T18:23:51.783 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:35.829819+0000", 2024-08-17T18:23:51.783 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.945959404000007, 2024-08-17T18:23:51.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0002586389999999, 2024-08-17T18:23:51.783 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:23:51.783 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:23:51.783 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:23:51.783 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:23:51.783 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:23:51.783 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 117402 2024-08-17T18:23:51.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.784 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:23:51.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:23:51.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829819+0000", 2024-08-17T18:23:51.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:23:51.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829819+0000", 2024-08-17T18:23:51.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:23:51.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829819+0000", 2024-08-17T18:23:51.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:23:51.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:23:51.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829822+0000", 2024-08-17T18:23:51.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0450000000000001e-06 2024-08-17T18:23:51.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:23:51.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829828+0000", 2024-08-17T18:23:51.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1650000000000003e-06 2024-08-17T18:23:51.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:23:51.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829834+0000", 2024-08-17T18:23:51.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1140000000000002e-06 2024-08-17T18:23:51.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:51.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829966+0000", 2024-08-17T18:23:51.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013191200000000001 2024-08-17T18:23:51.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:23:51.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829974+0000", 2024-08-17T18:23:51.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6369999999999999e-06 2024-08-17T18:23:51.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:51.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:36.830062+0000", 2024-08-17T18:23:51.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0000880130000001 2024-08-17T18:23:51.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:23:51.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:36.830078+0000", 2024-08-17T18:23:51.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6158999999999999e-05 2024-08-17T18:23:51.787 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.787 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:51.788 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.788 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:117414 58.1 58.711aa421 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e1724)", 2024-08-17T18:23:51.788 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:35.829839+0000", 2024-08-17T18:23:51.788 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.945939906000007, 2024-08-17T18:23:51.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0002668699999999, 2024-08-17T18:23:51.788 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:23:51.788 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:23:51.788 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:23:51.788 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:23:51.788 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:23:51.788 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 117414 2024-08-17T18:23:51.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.789 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:23:51.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:23:51.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829839+0000", 2024-08-17T18:23:51.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:23:51.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829839+0000", 2024-08-17T18:23:51.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:23:51.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829839+0000", 2024-08-17T18:23:51.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:23:51.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:23:51.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829841+0000", 2024-08-17T18:23:51.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0200000000000001e-06 2024-08-17T18:23:51.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:23:51.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829846+0000", 2024-08-17T18:23:51.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7459999999999997e-06 2024-08-17T18:23:51.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:23:51.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829851+0000", 2024-08-17T18:23:51.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5739999999999997e-06 2024-08-17T18:23:51.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:51.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829995+0000", 2024-08-17T18:23:51.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000144444 2024-08-17T18:23:51.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:23:51.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.830003+0000", 2024-08-17T18:23:51.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4390000000000003e-06 2024-08-17T18:23:51.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:51.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:36.830098+0000", 2024-08-17T18:23:51.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.000094966 2024-08-17T18:23:51.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:23:51.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:36.830106+0000", 2024-08-17T18:23:51.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0520000000000002e-06 2024-08-17T18:23:51.794 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.794 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:51.794 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.794 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:117429 58.1 58.76990531 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1724)", 2024-08-17T18:23:51.794 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:35.951905+0000", 2024-08-17T18:23:51.794 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.823874114999995, 2024-08-17T18:23:51.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.87823835900000002, 2024-08-17T18:23:51.794 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:23:51.794 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:23:51.795 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:23:51.795 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:23:51.795 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:23:51.795 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 117429 2024-08-17T18:23:51.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.795 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:23:51.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:23:51.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.951905+0000", 2024-08-17T18:23:51.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:23:51.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.951905+0000", 2024-08-17T18:23:51.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:23:51.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.951903+0000", 2024-08-17T18:23:51.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:23:51.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:23:51.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.951912+0000", 2024-08-17T18:23:51.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1810000000000007e-06 2024-08-17T18:23:51.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:23:51.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.951929+0000", 2024-08-17T18:23:51.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7011999999999999e-05 2024-08-17T18:23:51.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:23:51.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.951937+0000", 2024-08-17T18:23:51.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0169999999999996e-06 2024-08-17T18:23:51.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:51.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.951999+0000", 2024-08-17T18:23:51.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1558999999999998e-05 2024-08-17T18:23:51.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:23:51.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.952010+0000", 2024-08-17T18:23:51.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1681999999999999e-05 2024-08-17T18:23:51.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:51.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:36.830135+0000", 2024-08-17T18:23:51.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.87812500699999996 2024-08-17T18:23:51.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:23:51.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:36.830143+0000", 2024-08-17T18:23:51.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6879999999999999e-06 2024-08-17T18:23:51.799 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.799 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:51.799 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.799 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(58.1 1726/1725 [PushOp(58:819a1470:::benchmark_data_smithi136_442124_object70687:head, version: 1713'4214, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(58:819a1470:::benchmark_data_smithi136_442124_object70687:head@1713'4214, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-17T18:23:51.799 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:38.654181+0000", 2024-08-17T18:23:51.799 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 73.121597584, 2024-08-17T18:23:51.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10120472799999999, 2024-08-17T18:23:51.799 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:23:51.799 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:23:51.799 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:23:51.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:23:51.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.654181+0000", 2024-08-17T18:23:51.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:23:51.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.654181+0000", 2024-08-17T18:23:51.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:23:51.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.654180+0000", 2024-08-17T18:23:51.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:23:51.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:23:51.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.654235+0000", 2024-08-17T18:23:51.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5226000000000003e-05 2024-08-17T18:23:51.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:23:51.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.654254+0000", 2024-08-17T18:23:51.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8909000000000001e-05 2024-08-17T18:23:51.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:23:51.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.654265+0000", 2024-08-17T18:23:51.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1060000000000001e-05 2024-08-17T18:23:51.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:51.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.654322+0000", 2024-08-17T18:23:51.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6719999999999999e-05 2024-08-17T18:23:51.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:23:51.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.654331+0000", 2024-08-17T18:23:51.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7909999999999993e-06 2024-08-17T18:23:51.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:23:51.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.755386+0000", 2024-08-17T18:23:51.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10105546 2024-08-17T18:23:51.803 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.803 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:51.803 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.803 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:117809 58.1 e1726/1725 58:8f45627e:::benchmark_data_smithi136_442124_object4193:head v 1726'7198, mlcod=1713'4211)", 2024-08-17T18:23:51.803 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:38.655864+0000", 2024-08-17T18:23:51.803 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 73.119914386999994, 2024-08-17T18:23:51.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.112263075, 2024-08-17T18:23:51.803 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:23:51.804 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:23:51.804 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:23:51.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:23:51.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.655864+0000", 2024-08-17T18:23:51.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:23:51.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.655864+0000", 2024-08-17T18:23:51.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:23:51.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.655863+0000", 2024-08-17T18:23:51.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:23:51.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:23:51.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.655873+0000", 2024-08-17T18:23:51.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8719999999999997e-06 2024-08-17T18:23:51.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:23:51.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.655888+0000", 2024-08-17T18:23:51.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5167000000000001e-05 2024-08-17T18:23:51.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:23:51.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.755384+0000", 2024-08-17T18:23:51.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099495919000000002 2024-08-17T18:23:51.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:51.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.755425+0000", 2024-08-17T18:23:51.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1090000000000001e-05 2024-08-17T18:23:51.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:23:51.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.755462+0000", 2024-08-17T18:23:51.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7357000000000002e-05 2024-08-17T18:23:51.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:23:51.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.768089+0000", 2024-08-17T18:23:51.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012626597 2024-08-17T18:23:51.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:23:51.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.768128+0000", 2024-08-17T18:23:51.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8470999999999998e-05 2024-08-17T18:23:51.807 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.807 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:51.808 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.808 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:117815 58.7 e1726/1725 58:e7c1fcb2:::benchmark_data_smithi136_442124_object4199:head v 1726'7461, mlcod=1711'4442)", 2024-08-17T18:23:51.808 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:38.660622+0000", 2024-08-17T18:23:51.808 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 73.115156565999996, 2024-08-17T18:23:51.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10757431100000001, 2024-08-17T18:23:51.808 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:23:51.808 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:23:51.808 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:23:51.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:23:51.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.660622+0000", 2024-08-17T18:23:51.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:23:51.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.660622+0000", 2024-08-17T18:23:51.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:23:51.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.660621+0000", 2024-08-17T18:23:51.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:23:51.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:23:51.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.660629+0000", 2024-08-17T18:23:51.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2819999999999996e-06 2024-08-17T18:23:51.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:23:51.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.755380+0000", 2024-08-17T18:23:51.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.094751276999999995 2024-08-17T18:23:51.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:23:51.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.755396+0000", 2024-08-17T18:23:51.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5979000000000001e-05 2024-08-17T18:23:51.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:51.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.755474+0000", 2024-08-17T18:23:51.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7813999999999997e-05 2024-08-17T18:23:51.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:23:51.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.755498+0000", 2024-08-17T18:23:51.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3697e-05 2024-08-17T18:23:51.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:23:51.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.768155+0000", 2024-08-17T18:23:51.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012656769 2024-08-17T18:23:51.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:23:51.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.768197+0000", 2024-08-17T18:23:51.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1844000000000001e-05 2024-08-17T18:23:51.812 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.812 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:51.812 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.812 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:117825 58.a e1726/1725 58:5bedac79:::benchmark_data_smithi136_442124_object4209:head v 1726'7417, mlcod=1711'4393)", 2024-08-17T18:23:51.812 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:38.676100+0000", 2024-08-17T18:23:51.812 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 73.099678830000002, 2024-08-17T18:23:51.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.089370592999999998, 2024-08-17T18:23:51.812 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:23:51.813 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:23:51.813 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:23:51.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:23:51.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.676100+0000", 2024-08-17T18:23:51.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:23:51.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.676100+0000", 2024-08-17T18:23:51.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:23:51.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.676098+0000", 2024-08-17T18:23:51.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:23:51.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:23:51.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.676111+0000", 2024-08-17T18:23:51.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2140000000000001e-05 2024-08-17T18:23:51.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:23:51.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.755381+0000", 2024-08-17T18:23:51.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.079270102999999995 2024-08-17T18:23:51.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:23:51.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.755395+0000", 2024-08-17T18:23:51.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4713e-05 2024-08-17T18:23:51.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:51.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.755438+0000", 2024-08-17T18:23:51.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2305000000000002e-05 2024-08-17T18:23:51.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:23:51.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.755462+0000", 2024-08-17T18:23:51.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4760999999999999e-05 2024-08-17T18:23:51.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:23:51.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.765436+0000", 2024-08-17T18:23:51.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0099739259999999993 2024-08-17T18:23:51.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:23:51.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.765471+0000", 2024-08-17T18:23:51.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4214999999999997e-05 2024-08-17T18:23:51.816 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.816 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:51.817 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.817 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:117823 58.1 e1726/1725 58:897b25ec:::benchmark_data_smithi136_442124_object4207:head v 1726'7201, mlcod=1713'4211)", 2024-08-17T18:23:51.817 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:38.755511+0000", 2024-08-17T18:23:51.817 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 73.020268297000001, 2024-08-17T18:23:51.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.039277597999999997, 2024-08-17T18:23:51.817 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:23:51.817 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:23:51.817 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:23:51.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:23:51.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.755511+0000", 2024-08-17T18:23:51.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:23:51.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.755511+0000", 2024-08-17T18:23:51.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:23:51.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.755510+0000", 2024-08-17T18:23:51.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967296 2024-08-17T18:23:51.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:23:51.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.755513+0000", 2024-08-17T18:23:51.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0060000000000001e-06 2024-08-17T18:23:51.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:23:51.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.755518+0000", 2024-08-17T18:23:51.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3340000000000003e-06 2024-08-17T18:23:51.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:23:51.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.755521+0000", 2024-08-17T18:23:51.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2200000000000001e-06 2024-08-17T18:23:51.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:51.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.768157+0000", 2024-08-17T18:23:51.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012636176000000001 2024-08-17T18:23:51.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:23:51.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.768186+0000", 2024-08-17T18:23:51.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8524999999999999e-05 2024-08-17T18:23:51.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:23:51.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.794763+0000", 2024-08-17T18:23:51.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.026577495 2024-08-17T18:23:51.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:23:51.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.794788+0000", 2024-08-17T18:23:51.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5074000000000001e-05 2024-08-17T18:23:51.821 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.821 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:51.821 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.821 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:117831 58.1 e1726/1725 58:8d98cc5a:::benchmark_data_smithi136_442124_object4215:head v 1726'7202, mlcod=1713'4211)", 2024-08-17T18:23:51.821 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:38.755524+0000", 2024-08-17T18:23:51.822 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 73.020254649999998, 2024-08-17T18:23:51.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.039278445000000002, 2024-08-17T18:23:51.822 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:23:51.822 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:23:51.822 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:23:51.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:23:51.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.755524+0000", 2024-08-17T18:23:51.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:23:51.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.755524+0000", 2024-08-17T18:23:51.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:23:51.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.755524+0000", 2024-08-17T18:23:51.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:23:51.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:23:51.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.755531+0000", 2024-08-17T18:23:51.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5510000000000002e-06 2024-08-17T18:23:51.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:23:51.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.755536+0000", 2024-08-17T18:23:51.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7360000000000001e-06 2024-08-17T18:23:51.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:23:51.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.755539+0000", 2024-08-17T18:23:51.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1820000000000002e-06 2024-08-17T18:23:51.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:51.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.773017+0000", 2024-08-17T18:23:51.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017477565 2024-08-17T18:23:51.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:23:51.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.773037+0000", 2024-08-17T18:23:51.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0491000000000001e-05 2024-08-17T18:23:51.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:23:51.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.794790+0000", 2024-08-17T18:23:51.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021752641999999999 2024-08-17T18:23:51.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:23:51.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.794803+0000", 2024-08-17T18:23:51.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2525e-05 2024-08-17T18:23:51.826 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.826 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:51.826 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.826 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:120220 58.7 e1726/1725 58:e2b3dca2:::benchmark_data_smithi136_442124_object6604:head v 1726'7623, mlcod=1711'4442)", 2024-08-17T18:23:51.826 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:42.086313+0000", 2024-08-17T18:23:51.826 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 69.689465705000003, 2024-08-17T18:23:51.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036731606, 2024-08-17T18:23:51.826 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:23:51.827 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:23:51.827 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:23:51.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:23:51.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.086313+0000", 2024-08-17T18:23:51.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:23:51.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.086313+0000", 2024-08-17T18:23:51.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:23:51.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.086312+0000", 2024-08-17T18:23:51.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:23:51.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:23:51.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.086320+0000", 2024-08-17T18:23:51.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0829999999999998e-06 2024-08-17T18:23:51.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:23:51.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.086329+0000", 2024-08-17T18:23:51.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2879999999999998e-06 2024-08-17T18:23:51.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:23:51.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.086334+0000", 2024-08-17T18:23:51.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5799999999999999e-06 2024-08-17T18:23:51.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:51.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.087938+0000", 2024-08-17T18:23:51.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016037180000000001 2024-08-17T18:23:51.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:23:51.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.087966+0000", 2024-08-17T18:23:51.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8059e-05 2024-08-17T18:23:51.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:23:51.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.122917+0000", 2024-08-17T18:23:51.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034950855000000003 2024-08-17T18:23:51.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:23:51.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.123045+0000", 2024-08-17T18:23:51.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012769499999999999 2024-08-17T18:23:51.831 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.831 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:51.831 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.831 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:120439 58.1 e1726/1725 58:8918813b:::benchmark_data_smithi136_442124_object6823:head v 1726'7360, mlcod=1713'4341)", 2024-08-17T18:23:51.831 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:42.408790+0000", 2024-08-17T18:23:51.831 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 69.366988397, 2024-08-17T18:23:51.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036140013999999998, 2024-08-17T18:23:51.832 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:23:51.832 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:23:51.832 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:23:51.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:23:51.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.408790+0000", 2024-08-17T18:23:51.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:23:51.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.408790+0000", 2024-08-17T18:23:51.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:23:51.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.408789+0000", 2024-08-17T18:23:51.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:23:51.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:23:51.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.408797+0000", 2024-08-17T18:23:51.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7849999999999992e-06 2024-08-17T18:23:51.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:23:51.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.408815+0000", 2024-08-17T18:23:51.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7975999999999999e-05 2024-08-17T18:23:51.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:23:51.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.408822+0000", 2024-08-17T18:23:51.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2219999999999996e-06 2024-08-17T18:23:51.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:51.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.413057+0000", 2024-08-17T18:23:51.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042350260000000002 2024-08-17T18:23:51.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:23:51.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.413094+0000", 2024-08-17T18:23:51.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6242999999999998e-05 2024-08-17T18:23:51.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:23:51.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.444889+0000", 2024-08-17T18:23:51.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.031795111000000001 2024-08-17T18:23:51.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:23:51.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.444930+0000", 2024-08-17T18:23:51.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1709000000000003e-05 2024-08-17T18:23:51.836 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.836 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:51.836 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.836 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:120444 58.1 e1726/1725 58:847b86ed:::benchmark_data_smithi136_442124_object6828:head v 1726'7362, mlcod=1713'4341)", 2024-08-17T18:23:51.836 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:42.416476+0000", 2024-08-17T18:23:51.836 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 69.359303056000002, 2024-08-17T18:23:51.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.033636010000000001, 2024-08-17T18:23:51.836 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:23:51.836 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:23:51.836 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:23:51.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:23:51.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.416476+0000", 2024-08-17T18:23:51.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:23:51.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.416476+0000", 2024-08-17T18:23:51.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:23:51.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.416475+0000", 2024-08-17T18:23:51.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:23:51.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:23:51.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.416482+0000", 2024-08-17T18:23:51.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5179999999999998e-06 2024-08-17T18:23:51.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:23:51.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.416495+0000", 2024-08-17T18:23:51.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2967e-05 2024-08-17T18:23:51.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:23:51.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.416502+0000", 2024-08-17T18:23:51.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9759999999999998e-06 2024-08-17T18:23:51.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:51.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.426966+0000", 2024-08-17T18:23:51.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010463742999999999 2024-08-17T18:23:51.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:23:51.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.426992+0000", 2024-08-17T18:23:51.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5854999999999999e-05 2024-08-17T18:23:51.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:23:51.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.450020+0000", 2024-08-17T18:23:51.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023028198999999999 2024-08-17T18:23:51.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:23:51.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.450112+0000", 2024-08-17T18:23:51.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1909000000000004e-05 2024-08-17T18:23:51.840 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.840 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:51.840 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.840 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:120454 58.1 e1726/1725 58:8f74881d:::benchmark_data_smithi136_442124_object6838:head v 1726'7366, mlcod=1713'4341)", 2024-08-17T18:23:51.841 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:42.436976+0000", 2024-08-17T18:23:51.841 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 69.338802741999999, 2024-08-17T18:23:51.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.050402693999999998, 2024-08-17T18:23:51.841 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:23:51.841 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:23:51.841 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:23:51.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:23:51.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.436976+0000", 2024-08-17T18:23:51.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:23:51.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.436976+0000", 2024-08-17T18:23:51.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:23:51.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.436975+0000", 2024-08-17T18:23:51.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:23:51.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:23:51.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.436985+0000", 2024-08-17T18:23:51.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0188e-05 2024-08-17T18:23:51.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:23:51.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.437000+0000", 2024-08-17T18:23:51.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4598e-05 2024-08-17T18:23:51.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:23:51.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.437008+0000", 2024-08-17T18:23:51.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6589999999999996e-06 2024-08-17T18:23:51.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:51.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.450192+0000", 2024-08-17T18:23:51.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013183746999999999 2024-08-17T18:23:51.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:23:51.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.450233+0000", 2024-08-17T18:23:51.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0524e-05 2024-08-17T18:23:51.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:23:51.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.487238+0000", 2024-08-17T18:23:51.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037004890999999998 2024-08-17T18:23:51.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:23:51.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.487379+0000", 2024-08-17T18:23:51.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014117900000000001 2024-08-17T18:23:51.845 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.845 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:51.845 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.845 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:120460 58.1 e1726/1725 58:8ae05a61:::benchmark_data_smithi136_442124_object6844:head v 1726'7367, mlcod=1713'4341)", 2024-08-17T18:23:51.845 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:42.440381+0000", 2024-08-17T18:23:51.845 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 69.335398206999997, 2024-08-17T18:23:51.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.047097597999999997, 2024-08-17T18:23:51.845 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:23:51.845 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:23:51.845 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:23:51.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:23:51.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.440381+0000", 2024-08-17T18:23:51.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:23:51.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.440381+0000", 2024-08-17T18:23:51.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:23:51.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.440380+0000", 2024-08-17T18:23:51.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:23:51.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:23:51.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.440385+0000", 2024-08-17T18:23:51.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2179999999999998e-06 2024-08-17T18:23:51.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:23:51.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.440396+0000", 2024-08-17T18:23:51.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1282e-05 2024-08-17T18:23:51.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:23:51.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.440401+0000", 2024-08-17T18:23:51.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7060000000000003e-06 2024-08-17T18:23:51.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:51.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.457332+0000", 2024-08-17T18:23:51.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016930562 2024-08-17T18:23:51.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:23:51.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.457355+0000", 2024-08-17T18:23:51.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3680999999999999e-05 2024-08-17T18:23:51.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:23:51.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.487383+0000", 2024-08-17T18:23:51.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.030027740000000001 2024-08-17T18:23:51.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:23:51.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:42.487478+0000", 2024-08-17T18:23:51.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5141000000000003e-05 2024-08-17T18:23:51.849 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.849 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:51.849 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.850 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:165377 58.9 e1739/1734 58:9333e0e0:::benchmark_data_smithi136_442124_object51761:head v 1739'10662, mlcod=0'0)", 2024-08-17T18:23:51.850 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:23:49.697270+0000", 2024-08-17T18:23:51.850 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.078508748, 2024-08-17T18:23:51.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035794581999999998, 2024-08-17T18:23:51.850 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:23:51.850 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:23:51.850 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:23:51.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:23:51.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.697270+0000", 2024-08-17T18:23:51.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:23:51.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.697270+0000", 2024-08-17T18:23:51.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:23:51.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.697269+0000", 2024-08-17T18:23:51.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:23:51.851 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:51.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:23:51.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.697276+0000", 2024-08-17T18:23:51.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3010000000000001e-06 2024-08-17T18:23:51.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:23:51.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.697288+0000", 2024-08-17T18:23:51.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1464999999999999e-05 2024-08-17T18:23:51.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:23:51.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.697295+0000", 2024-08-17T18:23:51.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7340000000000002e-06 2024-08-17T18:23:51.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:51.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.705237+0000", 2024-08-17T18:23:51.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0079425910000000006 2024-08-17T18:23:51.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:23:51.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.705299+0000", 2024-08-17T18:23:51.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1393e-05 2024-08-17T18:23:51.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:23:51.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.731237+0000", 2024-08-17T18:23:51.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.025938391000000002 2024-08-17T18:23:51.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:23:51.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.733065+0000", 2024-08-17T18:23:51.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018276119999999999 2024-08-17T18:23:51.855 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.855 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:51.855 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.855 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:165408 58.1 e1739/1725 58:8e4eab9b:::benchmark_data_smithi136_442124_object51792:head v 1739'10017, mlcod=1713'4950)", 2024-08-17T18:23:51.856 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:23:49.702738+0000", 2024-08-17T18:23:51.856 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.0730412989999998, 2024-08-17T18:23:51.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034096782999999999, 2024-08-17T18:23:51.856 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:23:51.856 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:23:51.856 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:23:51.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:23:51.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.702738+0000", 2024-08-17T18:23:51.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:23:51.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.702738+0000", 2024-08-17T18:23:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:23:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.702736+0000", 2024-08-17T18:23:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:23:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:23:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.702744+0000", 2024-08-17T18:23:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8730000000000007e-06 2024-08-17T18:23:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:23:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.702760+0000", 2024-08-17T18:23:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5410999999999999e-05 2024-08-17T18:23:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:23:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.702767+0000", 2024-08-17T18:23:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0879999999999997e-06 2024-08-17T18:23:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.705953+0000", 2024-08-17T18:23:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031865180000000002 2024-08-17T18:23:51.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:23:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.705979+0000", 2024-08-17T18:23:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6177999999999999e-05 2024-08-17T18:23:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:23:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.736797+0000", 2024-08-17T18:23:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.030817174999999999 2024-08-17T18:23:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:23:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.736834+0000", 2024-08-17T18:23:51.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.765e-05 2024-08-17T18:23:51.860 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.860 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:51.860 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.860 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:165416 58.1 e1739/1725 58:8b0539da:::benchmark_data_smithi136_442124_object51800:head v 1739'10020, mlcod=1713'4950)", 2024-08-17T18:23:51.860 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:23:49.719928+0000", 2024-08-17T18:23:51.860 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.0558509950000001, 2024-08-17T18:23:51.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037836821999999999, 2024-08-17T18:23:51.860 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:23:51.860 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:23:51.860 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:23:51.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:23:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.719928+0000", 2024-08-17T18:23:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:23:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.719928+0000", 2024-08-17T18:23:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:23:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.719927+0000", 2024-08-17T18:23:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:23:51.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:23:51.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.719936+0000", 2024-08-17T18:23:51.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5869999999999993e-06 2024-08-17T18:23:51.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:23:51.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.719949+0000", 2024-08-17T18:23:51.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2982999999999999e-05 2024-08-17T18:23:51.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:23:51.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.719954+0000", 2024-08-17T18:23:51.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.276e-06 2024-08-17T18:23:51.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:51.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.724978+0000", 2024-08-17T18:23:51.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050235770000000004 2024-08-17T18:23:51.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:23:51.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.725001+0000", 2024-08-17T18:23:51.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3309e-05 2024-08-17T18:23:51.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:23:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.757718+0000", 2024-08-17T18:23:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.032717165999999999 2024-08-17T18:23:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:23:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.757765+0000", 2024-08-17T18:23:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6650999999999998e-05 2024-08-17T18:23:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:165387 58.9 e1739/1734 58:9f766e78:::benchmark_data_smithi136_442124_object51771:head v 1739'10665, mlcod=0'0)", 2024-08-17T18:23:51.864 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:23:49.723745+0000", 2024-08-17T18:23:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.0520339349999999, 2024-08-17T18:23:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035469696000000002, 2024-08-17T18:23:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:23:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:23:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:23:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:23:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.723745+0000", 2024-08-17T18:23:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:23:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.723745+0000", 2024-08-17T18:23:51.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:23:51.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.723744+0000", 2024-08-17T18:23:51.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:23:51.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:23:51.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.723752+0000", 2024-08-17T18:23:51.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8679999999999999e-06 2024-08-17T18:23:51.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:23:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.723764+0000", 2024-08-17T18:23:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2439999999999999e-05 2024-08-17T18:23:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:23:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.723772+0000", 2024-08-17T18:23:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4130000000000006e-06 2024-08-17T18:23:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.726935+0000", 2024-08-17T18:23:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031621980000000002 2024-08-17T18:23:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:23:51.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.728589+0000", 2024-08-17T18:23:51.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016546589999999999 2024-08-17T18:23:51.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:23:51.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.759126+0000", 2024-08-17T18:23:51.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.030536396 2024-08-17T18:23:51.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:23:51.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.759215+0000", 2024-08-17T18:23:51.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8879000000000001e-05 2024-08-17T18:23:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:165419 58.1 e1739/1725 58:870c4463:::benchmark_data_smithi136_442124_object51803:head v 1739'10021, mlcod=1713'4950)", 2024-08-17T18:23:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:23:49.724424+0000", 2024-08-17T18:23:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 2.0513544100000001, 2024-08-17T18:23:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.034827186000000003, 2024-08-17T18:23:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:23:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:23:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:23:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:23:51.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.724424+0000", 2024-08-17T18:23:51.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:23:51.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.724424+0000", 2024-08-17T18:23:51.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:23:51.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:23:51.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.724424+0000", 2024-08-17T18:23:51.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:23:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:23:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.724429+0000", 2024-08-17T18:23:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8099999999999997e-06 2024-08-17T18:23:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:23:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.724437+0000", 2024-08-17T18:23:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8109999999999992e-06 2024-08-17T18:23:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:23:51.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.724443+0000", 2024-08-17T18:23:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9630000000000003e-06 2024-08-17T18:23:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:23:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.731293+0000", 2024-08-17T18:23:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068495479999999996 2024-08-17T18:23:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:23:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.731315+0000", 2024-08-17T18:23:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2520000000000001e-05 2024-08-17T18:23:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:23:51.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.759218+0000", 2024-08-17T18:23:51.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.027902396999999999 2024-08-17T18:23:51.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:23:51.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:23:51.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:23:51.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:49.759252+0000", 2024-08-17T18:23:51.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3753e-05 2024-08-17T18:23:51.873 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.873 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:51.873 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.873 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:23:51.873 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:23:51.873 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:23:51.905 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:51.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:51.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:51.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:51 smithi136 ceph-mon[109861]: pgmap v3430: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 650 op/s; 17512/187278 objects degraded (9.351%); 11890/187278 objects misplaced (6.349%); 5.2 MiB/s, 83 objects/s recovering 2024-08-17T18:23:51.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:23:51.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:51 smithi136 ceph-mon[115078]: pgmap v3430: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 650 op/s; 17512/187278 objects degraded (9.351%); 11890/187278 objects misplaced (6.349%); 5.2 MiB/s, 83 objects/s recovering 2024-08-17T18:23:51.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:23:52.007 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:52.047 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:52.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:52.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:52.149 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:52.178 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:52.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:52.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:52.280 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:52.317 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:52.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:52.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:52.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:52.446 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:52.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:52.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:52.548 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:52.577 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:52.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:52.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:52.679 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:52.721 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:52.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:52.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:52.823 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:52.851 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:52.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:52.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:52.953 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:52.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:52.987 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:52.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:53.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:53.121 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:53.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:53.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:53.170 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-17T18:23:53.223 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:53.251 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:53.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:53.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:53.352 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:53.405 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:53.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:53.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:53.507 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:53.548 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:53.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:53.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:53.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:53.678 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:53.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:53.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:53.780 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:53.805 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:53.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:53.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:53.907 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:53.938 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:53.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:53.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:54.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:53 smithi165 ceph-mon[105517]: pgmap v3431: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 597 op/s; 17512/186678 objects degraded (9.381%); 11890/186678 objects misplaced (6.369%); 4.7 MiB/s, 74 objects/s recovering 2024-08-17T18:23:54.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:54.069 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:54.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:54.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:54.171 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:54.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:53 smithi136 ceph-mon[109861]: pgmap v3431: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 597 op/s; 17512/186678 objects degraded (9.381%); 11890/186678 objects misplaced (6.369%); 4.7 MiB/s, 74 objects/s recovering 2024-08-17T18:23:54.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:53 smithi136 ceph-mon[115078]: pgmap v3431: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 597 op/s; 17512/186678 objects degraded (9.381%); 11890/186678 objects misplaced (6.369%); 4.7 MiB/s, 74 objects/s recovering 2024-08-17T18:23:54.202 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:54.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:54.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:54.304 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:54.342 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:54.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:54.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:54.444 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:54.475 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:54.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:54.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:54.577 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:54.606 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:54.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:54.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:54.708 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:54.741 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:54.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:54.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:54.843 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:54.869 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:54.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:54.970 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:54.993 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:54.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:54.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:55.095 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:55.132 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:55.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:55.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:55.234 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:55.262 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:55.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:55.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:55.363 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:55.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:55.409 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:55.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:55.510 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:55.536 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:55.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:55.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:55.638 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:55.679 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:55.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:55.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:55.781 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:55.803 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:55.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:55.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:55.905 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:55.933 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:55.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:55.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:56.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:55 smithi165 ceph-mon[105517]: pgmap v3432: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 701 op/s; 17090/180927 objects degraded (9.446%); 11463/180927 objects misplaced (6.336%); 6.6 MiB/s, 105 objects/s recovering 2024-08-17T18:23:56.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:55 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 17090/180927 objects degraded (9.446%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:23:56.035 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:56.064 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:56.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:56.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:56.166 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:56.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:55 smithi136 ceph-mon[115078]: pgmap v3432: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 701 op/s; 17090/180927 objects degraded (9.446%); 11463/180927 objects misplaced (6.336%); 6.6 MiB/s, 105 objects/s recovering 2024-08-17T18:23:56.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:55 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 17090/180927 objects degraded (9.446%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:23:56.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:55 smithi136 ceph-mon[109861]: pgmap v3432: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 701 op/s; 17090/180927 objects degraded (9.446%); 11463/180927 objects misplaced (6.336%); 6.6 MiB/s, 105 objects/s recovering 2024-08-17T18:23:56.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:55 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 17090/180927 objects degraded (9.446%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:23:56.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:56.207 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:56.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:56.308 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:56.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:56.349 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:56.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:56.450 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:56.489 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:56.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:56.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:56.591 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:56.625 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:56.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:56.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:56.727 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:56.757 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:56.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:56.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:56.859 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:56.889 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:56.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:56.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:56.991 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:23:57.032 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:57.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:57.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:23:57.134 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:57.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:57.165 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:57.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:57.266 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:57.299 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:57.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:57.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:57.401 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:57.443 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:57.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:57.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:57.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:57.580 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:57.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:57.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:57.683 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:57.708 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:23:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:23:57] "GET /metrics HTTP/1.1" 200 36749 "" "Prometheus/2.51.0" 2024-08-17T18:23:57.712 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:57.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:57.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:57.814 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:57.845 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:57.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:57.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:57.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:23:57.971 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:57.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:57.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:23:58.000 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:57 smithi136 ceph-mon[109861]: pgmap v3433: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 668 op/s; 17090/176973 objects degraded (9.657%); 11226/176973 objects misplaced (6.343%); 4.5 MiB/s, 71 objects/s recovering 2024-08-17T18:23:58.000 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:57 smithi136 ceph-mon[115078]: pgmap v3433: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 668 op/s; 17090/176973 objects degraded (9.657%); 11226/176973 objects misplaced (6.343%); 4.5 MiB/s, 71 objects/s recovering 2024-08-17T18:23:58.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:57 smithi165 ceph-mon[105517]: pgmap v3433: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 668 op/s; 17090/176973 objects degraded (9.657%); 11226/176973 objects misplaced (6.343%); 4.5 MiB/s, 71 objects/s recovering 2024-08-17T18:23:58.073 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:58.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:58.114 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:58.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:58.215 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:58.241 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:58.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:58.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:58.342 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:23:58.343 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:23:58.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:58.411 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:58.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:23:58.512 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:58.556 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:58.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:58.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:58.658 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:58.684 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:23:58.697 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:58.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:58.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:58.732 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.0 to 1 (10000) 2024-08-17T18:23:58.799 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:58.835 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:58 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/942872511' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T18:23:58.835 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:58 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T18:23:58.835 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:58 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/942872511' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T18:23:58.835 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:58 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T18:23:58.836 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:58.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:58.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:58.938 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:23:58.962 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:58.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:58.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:23:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:58 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/942872511' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T18:23:59.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:58 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T18:23:59.064 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:59.086 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:59.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:59.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:59.189 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:59.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:59.212 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:59.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:59.314 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:23:59.336 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:23:59.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:59.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:23:59.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:59.461 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:59.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:59.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:23:59.563 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:59.584 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:59.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:59.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:59.686 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:23:59.707 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:59.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:59.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:23:59.725 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:23:59.809 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:23:59.830 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:59.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:59.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:23:59.853 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:59 smithi136 ceph-mon[115078]: pgmap v3434: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 759 op/s; 16647/170382 objects degraded (9.770%); 10756/170382 objects misplaced (6.313%); 6.8 MiB/s, 108 objects/s recovering 2024-08-17T18:23:59.853 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:59 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T18:23:59.854 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:59 smithi136 ceph-mon[115078]: osdmap e1740: 8 total, 8 up, 8 in 2024-08-17T18:23:59.854 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:59 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/979115286' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:23:59.854 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:59 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:23:59.854 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:59 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:23:59.854 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:59 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:23:59.854 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:23:59 smithi136 ceph-mon[115078]: osdmap e1741: 8 total, 8 up, 8 in 2024-08-17T18:23:59.854 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:59 smithi136 ceph-mon[109861]: pgmap v3434: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 759 op/s; 16647/170382 objects degraded (9.770%); 10756/170382 objects misplaced (6.313%); 6.8 MiB/s, 108 objects/s recovering 2024-08-17T18:23:59.854 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:59 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T18:23:59.854 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:59 smithi136 ceph-mon[109861]: osdmap e1740: 8 total, 8 up, 8 in 2024-08-17T18:23:59.855 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:59 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/979115286' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:23:59.855 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:59 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:23:59.855 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:59 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:23:59.855 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:59 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:23:59.855 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:23:59 smithi136 ceph-mon[109861]: osdmap e1741: 8 total, 8 up, 8 in 2024-08-17T18:23:59.932 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:23:59.954 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:23:59.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:23:59.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:59 smithi165 ceph-mon[105517]: pgmap v3434: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 3.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 759 op/s; 16647/170382 objects degraded (9.770%); 10756/170382 objects misplaced (6.313%); 6.8 MiB/s, 108 objects/s recovering 2024-08-17T18:24:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:59 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T18:24:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:59 smithi165 ceph-mon[105517]: osdmap e1740: 8 total, 8 up, 8 in 2024-08-17T18:24:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:59 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/979115286' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:24:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:59 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:24:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:59 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:24:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:59 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:24:00.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:23:59 smithi165 ceph-mon[105517]: osdmap e1741: 8 total, 8 up, 8 in 2024-08-17T18:24:00.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:00.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:00.081 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:00.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:00.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:00.203 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:00.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:00.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:00.305 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:00.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:00.328 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:00.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:00.429 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:00.450 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:00.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:00.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:00.552 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:00.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:00.575 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:00.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:00.676 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:00.698 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:00.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:00.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:00.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:00.821 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:00.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:00.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:00.923 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:00.958 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:00.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:00.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:00 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 16647/170382 objects degraded (9.770%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:01.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:00 smithi165 ceph-mon[105517]: osdmap e1742: 8 total, 8 up, 8 in 2024-08-17T18:24:01.060 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:01.084 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:01.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:01.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:01.186 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:01.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:00 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 16647/170382 objects degraded (9.770%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:01.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:00 smithi136 ceph-mon[109861]: osdmap e1742: 8 total, 8 up, 8 in 2024-08-17T18:24:01.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:00 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 16647/170382 objects degraded (9.770%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:01.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:00 smithi136 ceph-mon[115078]: osdmap e1742: 8 total, 8 up, 8 in 2024-08-17T18:24:01.207 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:01.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:01.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:01.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:01.331 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:01.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:01.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:01.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:01.455 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:01.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:01.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:01.557 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:01.580 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:01.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:01.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:01.682 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:01.703 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:01.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:01.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:01.804 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:01.827 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:01.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:01.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:01.929 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:01.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:01.952 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:01.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:02.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:01 smithi165 ceph-mon[105517]: pgmap v3437: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 remapped+peering, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 788 op/s; 14078/167754 objects degraded (8.392%); 9751/167754 objects misplaced (5.813%); 7.6 MiB/s, 121 objects/s recovering 2024-08-17T18:24:02.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:01 smithi165 ceph-mon[105517]: 58.a starting backfill to osd.0 from (0'0,0'0] MAX to 1739'10782 2024-08-17T18:24:02.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:01 smithi165 ceph-mon[105517]: 58.1 starting backfill to osd.0 from (0'0,0'0] MAX to 1739'10421 2024-08-17T18:24:02.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:01 smithi165 ceph-mon[105517]: 58.5 starting backfill to osd.0 from (0'0,0'0] MAX to 1739'10672 2024-08-17T18:24:02.053 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:02.076 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:02.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:02.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:02.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:02.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:01 smithi136 ceph-mon[109861]: pgmap v3437: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 remapped+peering, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 788 op/s; 14078/167754 objects degraded (8.392%); 9751/167754 objects misplaced (5.813%); 7.6 MiB/s, 121 objects/s recovering 2024-08-17T18:24:02.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:01 smithi136 ceph-mon[109861]: 58.a starting backfill to osd.0 from (0'0,0'0] MAX to 1739'10782 2024-08-17T18:24:02.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:01 smithi136 ceph-mon[109861]: 58.1 starting backfill to osd.0 from (0'0,0'0] MAX to 1739'10421 2024-08-17T18:24:02.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:01 smithi136 ceph-mon[109861]: 58.5 starting backfill to osd.0 from (0'0,0'0] MAX to 1739'10672 2024-08-17T18:24:02.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:01 smithi136 ceph-mon[115078]: pgmap v3437: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 remapped+peering, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 788 op/s; 14078/167754 objects degraded (8.392%); 9751/167754 objects misplaced (5.813%); 7.6 MiB/s, 121 objects/s recovering 2024-08-17T18:24:02.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:01 smithi136 ceph-mon[115078]: 58.a starting backfill to osd.0 from (0'0,0'0] MAX to 1739'10782 2024-08-17T18:24:02.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:01 smithi136 ceph-mon[115078]: 58.1 starting backfill to osd.0 from (0'0,0'0] MAX to 1739'10421 2024-08-17T18:24:02.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:01 smithi136 ceph-mon[115078]: 58.5 starting backfill to osd.0 from (0'0,0'0] MAX to 1739'10672 2024-08-17T18:24:02.200 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:02.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:02.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:02.302 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:02.325 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:02.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:02.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:02.427 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:02.448 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:02.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:02.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:02.549 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:02.571 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:02.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:02.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:02.673 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:02.696 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:02.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:02.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:02.798 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:02.820 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:02.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:02.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:02.922 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:02.951 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:02.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:02.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:03.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:02 smithi165 ceph-mon[105517]: osdmap e1743: 8 total, 8 up, 8 in 2024-08-17T18:24:03.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:02 smithi165 ceph-mon[105517]: osdmap e1744: 8 total, 8 up, 8 in 2024-08-17T18:24:03.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:03.079 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:03.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:03.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:03.180 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:03.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:02 smithi136 ceph-mon[109861]: osdmap e1743: 8 total, 8 up, 8 in 2024-08-17T18:24:03.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:02 smithi136 ceph-mon[109861]: osdmap e1744: 8 total, 8 up, 8 in 2024-08-17T18:24:03.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:02 smithi136 ceph-mon[115078]: osdmap e1743: 8 total, 8 up, 8 in 2024-08-17T18:24:03.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:02 smithi136 ceph-mon[115078]: osdmap e1744: 8 total, 8 up, 8 in 2024-08-17T18:24:03.210 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:03.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:03.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:03.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:03.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:03.342 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:03.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:03.443 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:03.486 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:03.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:03.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:03.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:03.616 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:03.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:03.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:03.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:03.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:03.749 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:03.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:03.850 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:03.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:03.886 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:03.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:03.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:04.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:03 smithi165 ceph-mon[105517]: pgmap v3440: 99 pgs: 1 peering, 1 active+recovering+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 remapped+peering, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 239 op/s; 14078/167502 objects degraded (8.405%); 9751/167502 objects misplaced (5.821%); 1.6 MiB/s, 27 objects/s recovering 2024-08-17T18:24:04.014 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:03 smithi136 ceph-mon[115078]: pgmap v3440: 99 pgs: 1 peering, 1 active+recovering+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 remapped+peering, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 239 op/s; 14078/167502 objects degraded (8.405%); 9751/167502 objects misplaced (5.821%); 1.6 MiB/s, 27 objects/s recovering 2024-08-17T18:24:04.014 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:03 smithi136 ceph-mon[109861]: pgmap v3440: 99 pgs: 1 peering, 1 active+recovering+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 remapped+peering, 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 239 op/s; 14078/167502 objects degraded (8.405%); 9751/167502 objects misplaced (5.821%); 1.6 MiB/s, 27 objects/s recovering 2024-08-17T18:24:04.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:04.025 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:04.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:04.126 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:04.153 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:04.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:04.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:04.255 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:04.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:04.291 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:04.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:04.392 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:04.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:04.431 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:04.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:04.533 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:04.568 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:04.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:04.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:04.670 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:04.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:04.704 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:04.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:04.805 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:04.852 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:04.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:04.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:04.954 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:04.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:04.994 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:04.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:05.078 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:24:05.078 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:24:05.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.079 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.36936.0:173248 58.5 e1744/1741 58:ac590516:::benchmark_data_smithi136_442124_object59632:head v 1744'10770, mlcod=1744'10768)", 2024-08-17T18:24:05.079 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:24:05.071112+0000", 2024-08-17T18:24:05.079 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0062318720000000003, 2024-08-17T18:24:05.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0062600690000000001, 2024-08-17T18:24:05.079 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:24:05.079 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:24:05.079 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T18:24:05.079 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:24:05.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:24:05.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:05.071112+0000", 2024-08-17T18:24:05.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:05.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:05.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:24:05.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:05.071112+0000", 2024-08-17T18:24:05.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:05.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:05.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:24:05.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:05.071115+0000", 2024-08-17T18:24:05.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.749e-06 2024-08-17T18:24:05.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:05.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:24:05.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:05.071125+0000", 2024-08-17T18:24:05.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0019e-05 2024-08-17T18:24:05.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:05.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:24:05.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:05.071147+0000", 2024-08-17T18:24:05.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1722999999999999e-05 2024-08-17T18:24:05.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:05.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:24:05.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:05.071157+0000", 2024-08-17T18:24:05.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9960000000000005e-06 2024-08-17T18:24:05.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:05.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:05.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:05.071252+0000", 2024-08-17T18:24:05.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4498000000000002e-05 2024-08-17T18:24:05.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:05.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:24:05.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:05.071276+0000", 2024-08-17T18:24:05.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4409999999999998e-05 2024-08-17T18:24:05.083 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:05.083 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:24:05.083 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:05.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:05.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.083 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.36936.0:173253 58.5 e1744/1741 58:a861b7ef:::benchmark_data_smithi136_442124_object59637:head v 1744'10771, mlcod=1744'10768)", 2024-08-17T18:24:05.083 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:24:05.074336+0000", 2024-08-17T18:24:05.083 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0030077400000000001, 2024-08-17T18:24:05.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0030899450000000002, 2024-08-17T18:24:05.083 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:24:05.083 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:24:05.084 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-17T18:24:05.084 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:24:05.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:24:05.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:05.074336+0000", 2024-08-17T18:24:05.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:05.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:05.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:24:05.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:05.074336+0000", 2024-08-17T18:24:05.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:05.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:05.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:24:05.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:05.074337+0000", 2024-08-17T18:24:05.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7899999999999998e-07 2024-08-17T18:24:05.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:05.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:24:05.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:05.074343+0000", 2024-08-17T18:24:05.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4320000000000004e-06 2024-08-17T18:24:05.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:05.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:24:05.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:05.074361+0000", 2024-08-17T18:24:05.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7876e-05 2024-08-17T18:24:05.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:05.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:24:05.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:05.074368+0000", 2024-08-17T18:24:05.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2300000000000002e-06 2024-08-17T18:24:05.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:05.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:05.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:05.074468+0000", 2024-08-17T18:24:05.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010006299999999999 2024-08-17T18:24:05.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:05.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:24:05.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:05.074492+0000", 2024-08-17T18:24:05.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4195000000000001e-05 2024-08-17T18:24:05.087 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:05.087 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:24:05.087 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:05.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:05.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.151 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.36936.0:173254 58.f e1744/1728)", 2024-08-17T18:24:05.151 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:24:05.077453+0000", 2024-08-17T18:24:05.151 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4294967295.9998908, 2024-08-17T18:24:05.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2017000000000001e-05, 2024-08-17T18:24:05.152 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:24:05.152 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:24:05.152 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-08-17T18:24:05.152 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:24:05.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:24:05.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:05.077453+0000", 2024-08-17T18:24:05.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:05.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:05.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:24:05.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:05.077453+0000", 2024-08-17T18:24:05.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:05.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:05.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:24:05.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:05.077454+0000", 2024-08-17T18:24:05.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0130000000000001e-06 2024-08-17T18:24:05.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:05.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:24:05.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:05.077461+0000", 2024-08-17T18:24:05.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8349999999999998e-06 2024-08-17T18:24:05.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:05.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:24:05.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:05.077473+0000", 2024-08-17T18:24:05.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.238e-05 2024-08-17T18:24:05.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:05.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:05.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:24:05.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:05.077480+0000", 2024-08-17T18:24:05.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5919999999999997e-06 2024-08-17T18:24:05.154 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:05.154 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:24:05.154 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:05.154 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:05.155 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-17T18:24:05.155 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 3 2024-08-17T18:24:05.155 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:24:05.155 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:05.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:05.186 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:05.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:05.287 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:05.324 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:05.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:05.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:05.426 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:05.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:05.462 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:05.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:05.528 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 5 1 2024-08-17T18:24:05.563 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:05.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:05.611 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:05.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:05.712 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:05.742 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:05.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:05.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:05.844 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:05.898 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:05.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:05.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:05.923 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:24:06.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:06.015 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:05 smithi136 ceph-mon[109861]: pgmap v3442: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 peering, 2 active+recovering+degraded+remapped, 3 remapped+peering, 3 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 287 op/s; 13528/166176 objects degraded (8.141%); 9653/166176 objects misplaced (5.809%); 8.4 MiB/s, 135 objects/s recovering 2024-08-17T18:24:06.015 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:05 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 13528/166176 objects degraded (8.141%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:06.015 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:05 smithi136 ceph-mon[115078]: pgmap v3442: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 peering, 2 active+recovering+degraded+remapped, 3 remapped+peering, 3 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 287 op/s; 13528/166176 objects degraded (8.141%); 9653/166176 objects misplaced (5.809%); 8.4 MiB/s, 135 objects/s recovering 2024-08-17T18:24:06.015 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:05 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 13528/166176 objects degraded (8.141%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:06.029 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:06.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:06.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:06.131 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:06.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:06.183 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:06.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:06.208 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:24:06.208 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:24:06.246 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:24:06.246 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:24:06.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:05 smithi165 ceph-mon[105517]: pgmap v3442: 99 pgs: 1 active+recovery_wait+degraded+remapped, 1 peering, 2 active+recovering+degraded+remapped, 3 remapped+peering, 3 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 287 op/s; 13528/166176 objects degraded (8.141%); 9653/166176 objects misplaced (5.809%); 8.4 MiB/s, 135 objects/s recovering 2024-08-17T18:24:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:05 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 13528/166176 objects degraded (8.141%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:06.274 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:24:06.274 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:24:06.284 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:06.305 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:24:06.305 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:24:06.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:06.324 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:06.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:06.336 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:24:06.336 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:24:06.365 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:24:06.366 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:24:06.402 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:24:06.402 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:24:06.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:06.434 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:24:06.434 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:24:06.456 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:06.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:06.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:06.557 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:06.594 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:06.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:06.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:06.696 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:06.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:06.735 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:06.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:06.836 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:06.873 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:06.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:06.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:06.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:06 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1543225947' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:24:06.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:24:06.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:24:06.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:06 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1543225947' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:24:06.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:24:06.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:24:06.975 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:07.007 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:07.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:07.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:07.109 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:07.128 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-17T18:24:07.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:07.147 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:07.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:07.248 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:06 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1543225947' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:24:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:24:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:24:07.288 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:07.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:07.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:07.390 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:07.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:07.431 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:07.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:07.532 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:07.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:07.567 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:07.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:07.668 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:07.701 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:07.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:07.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:07.767 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:24:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:24:07] "GET /metrics HTTP/1.1" 200 36736 "" "Prometheus/2.51.0" 2024-08-17T18:24:07.803 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:07.831 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:07.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:07.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:07.933 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:07.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:07.960 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:07.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:08.037 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:07 smithi136 ceph-mon[109861]: pgmap v3443: 99 pgs: 3 active+recovery_wait+degraded+remapped, 1 peering, 1 active+recovering+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 191 op/s; 15953/164307 objects degraded (9.709%); 17182/164307 objects misplaced (10.457%); 6.1 MiB/s, 98 objects/s recovering 2024-08-17T18:24:08.037 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:07 smithi136 ceph-mon[115078]: pgmap v3443: 99 pgs: 3 active+recovery_wait+degraded+remapped, 1 peering, 1 active+recovering+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 191 op/s; 15953/164307 objects degraded (9.709%); 17182/164307 objects misplaced (10.457%); 6.1 MiB/s, 98 objects/s recovering 2024-08-17T18:24:08.057 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:24:08.061 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:08.086 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:08.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:08.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:08.105 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:24:08.188 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:08.216 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:08.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:08.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:08.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:07 smithi165 ceph-mon[105517]: pgmap v3443: 99 pgs: 3 active+recovery_wait+degraded+remapped, 1 peering, 1 active+recovering+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 191 op/s; 15953/164307 objects degraded (9.709%); 17182/164307 objects misplaced (10.457%); 6.1 MiB/s, 98 objects/s recovering 2024-08-17T18:24:08.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:08.344 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:08.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:08.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:08.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:08.472 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:08.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:08.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:08.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:08.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:08.604 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:08.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:08.705 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:08.734 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:08.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:08.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:08.769 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:24:08.836 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:08.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:08.869 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:08.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:08.970 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:09.007 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:09.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:09.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:09.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:09.145 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:09.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:09.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:09.246 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:09.287 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:09.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:09.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:09.389 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:09.430 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:09.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:09.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:09.532 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:09.561 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:09.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:09.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:09.663 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:09.695 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:09.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:09.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:09.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:09.830 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:09.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:09.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:09.932 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:09 smithi136 ceph-mon[109861]: pgmap v3444: 99 pgs: 1 active+recovering+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 393 op/s; 15441/158454 objects degraded (9.745%); 16770/158454 objects misplaced (10.584%); 8.7 MiB/s, 140 objects/s recovering 2024-08-17T18:24:09.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:09 smithi136 ceph-mon[115078]: pgmap v3444: 99 pgs: 1 active+recovering+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 393 op/s; 15441/158454 objects degraded (9.745%); 16770/158454 objects misplaced (10.584%); 8.7 MiB/s, 140 objects/s recovering 2024-08-17T18:24:09.965 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:09.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:09.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:10.039 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:24:10.062 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:09 smithi165 ceph-mon[105517]: pgmap v3444: 99 pgs: 1 active+recovering+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 393 op/s; 15441/158454 objects degraded (9.745%); 16770/158454 objects misplaced (10.584%); 8.7 MiB/s, 140 objects/s recovering 2024-08-17T18:24:10.067 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:10.127 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:10.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:10.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:10.229 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:10.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:10.262 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:10.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:10.363 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:10.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:10.395 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:10.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:10.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:10.528 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:10.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:10.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:10.629 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:10.659 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:10.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:10.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:10.761 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:10.790 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:10.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:10.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:10.892 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:10.924 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:10.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:10.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:10.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:10 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 15441/158454 objects degraded (9.745%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:10.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:10 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 15441/158454 objects degraded (9.745%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:10.986 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:10 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 15441/158454 objects degraded (9.745%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:11.026 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:11.057 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:11.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:11.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:11.159 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:11.190 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:11.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:11.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:11.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:11.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:11.327 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:11.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:11.428 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:11.461 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:11.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:11.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:11.563 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:11.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:11.610 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:11.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:11.711 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:11.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:11.752 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:11.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:11.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:11.882 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:11.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:11.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:11.984 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:12.013 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:12.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:12.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:12.057 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:11 smithi136 ceph-mon[109861]: pgmap v3445: 99 pgs: 1 active+recovering+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 416 op/s; 15441/156672 objects degraded (9.856%); 16406/156672 objects misplaced (10.472%); 7.8 MiB/s, 124 objects/s recovering 2024-08-17T18:24:12.057 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:11 smithi136 ceph-mon[115078]: pgmap v3445: 99 pgs: 1 active+recovering+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 416 op/s; 15441/156672 objects degraded (9.856%); 16406/156672 objects misplaced (10.472%); 7.8 MiB/s, 124 objects/s recovering 2024-08-17T18:24:12.115 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:12.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:12.149 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:12.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:12.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:11 smithi165 ceph-mon[105517]: pgmap v3445: 99 pgs: 1 active+recovering+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 416 op/s; 15441/156672 objects degraded (9.856%); 16406/156672 objects misplaced (10.472%); 7.8 MiB/s, 124 objects/s recovering 2024-08-17T18:24:12.250 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:12.282 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:12.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:12.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:12.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:12.424 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:12.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:12.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:12.528 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:12.568 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:12.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:12.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:12.670 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:12.710 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:12.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:12.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:12.810 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:24:12.811 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:12.863 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:12.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:12.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:12.966 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:13.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:13.012 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:13.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:13.113 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:13.155 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:13.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:13.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:13.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:12 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3242174555' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:24:13.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:12 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:24:13.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:12 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3242174555' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:24:13.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:12 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:24:13.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:13.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:12 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3242174555' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:24:13.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:12 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:24:13.303 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:13.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:13.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:13.406 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:13.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:13.439 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:13.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:13.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:13.569 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:13.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:13.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:13.670 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:13.703 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:13.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:13.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:13.805 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:13.811 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.5 to 1 (10000) 2024-08-17T18:24:13.852 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:13.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:13.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:13.915 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:13 smithi136 ceph-mon[109861]: pgmap v3446: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 3.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 440 op/s; 15441/154296 objects degraded (10.007%); 16205/154296 objects misplaced (10.503%); 6.9 MiB/s, 110 objects/s recovering 2024-08-17T18:24:13.916 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:13 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:24:13.916 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:13 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:24:13.916 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:13 smithi136 ceph-mon[109861]: osdmap e1745: 8 total, 8 up, 8 in 2024-08-17T18:24:13.916 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:13 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1092420' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T18:24:13.916 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:13 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T18:24:13.916 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:13 smithi136 ceph-mon[115078]: pgmap v3446: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 3.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 440 op/s; 15441/154296 objects degraded (10.007%); 16205/154296 objects misplaced (10.503%); 6.9 MiB/s, 110 objects/s recovering 2024-08-17T18:24:13.916 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:13 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:24:13.916 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:13 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:24:13.916 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:13 smithi136 ceph-mon[115078]: osdmap e1745: 8 total, 8 up, 8 in 2024-08-17T18:24:13.916 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:13 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1092420' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T18:24:13.916 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:13 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T18:24:13.954 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:13.954 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:24:13.955 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:24:13.984 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:13.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:13.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:14.086 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:14.114 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:14.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:14.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:14.215 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:14.248 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:14.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:14.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:13 smithi165 ceph-mon[105517]: pgmap v3446: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 3.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 440 op/s; 15441/154296 objects degraded (10.007%); 16205/154296 objects misplaced (10.503%); 6.9 MiB/s, 110 objects/s recovering 2024-08-17T18:24:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:13 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:24:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:13 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:24:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:13 smithi165 ceph-mon[105517]: osdmap e1745: 8 total, 8 up, 8 in 2024-08-17T18:24:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:13 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1092420' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T18:24:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:13 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T18:24:14.350 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:14.391 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:14.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:14.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:14.492 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:14.522 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:14.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:14.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:14.624 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:14.626 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 7 1 2024-08-17T18:24:14.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:14.657 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:14.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:14.758 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:14.795 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:14.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:14.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:14.806 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:24:14.897 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:14.898 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:14 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T18:24:14.899 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:14 smithi136 ceph-mon[115078]: osdmap e1746: 8 total, 8 up, 8 in 2024-08-17T18:24:14.899 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:14 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 15441/154296 objects degraded (10.007%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:14.899 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:14 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T18:24:14.899 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:14 smithi136 ceph-mon[109861]: osdmap e1746: 8 total, 8 up, 8 in 2024-08-17T18:24:14.899 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:14 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 15441/154296 objects degraded (10.007%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:14.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:14.927 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:14.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:15.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:15.063 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:15.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:15.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:15.165 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:15.203 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:15.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:15.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:15.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:14 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T18:24:15.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:14 smithi165 ceph-mon[105517]: osdmap e1746: 8 total, 8 up, 8 in 2024-08-17T18:24:15.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:14 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 15441/154296 objects degraded (10.007%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:15.305 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:15.337 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:15.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:15.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:15.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:15.475 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:15.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:15.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:15.577 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:15.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:15.611 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:15.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:15.712 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:15.761 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-17T18:24:15.767 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:15.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:15.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:15.799 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:24:15.869 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:15.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:15.933 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:15.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:16.034 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:16.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:15 smithi136 ceph-mon[109861]: pgmap v3449: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 14893/149193 objects degraded (9.982%); 15816/149193 objects misplaced (10.601%); 8.3 MiB/s, 132 objects/s recovering 2024-08-17T18:24:16.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:15 smithi136 ceph-mon[109861]: osdmap e1747: 8 total, 8 up, 8 in 2024-08-17T18:24:16.083 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:15 smithi136 ceph-mon[115078]: pgmap v3449: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 14893/149193 objects degraded (9.982%); 15816/149193 objects misplaced (10.601%); 8.3 MiB/s, 132 objects/s recovering 2024-08-17T18:24:16.083 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:15 smithi136 ceph-mon[115078]: osdmap e1747: 8 total, 8 up, 8 in 2024-08-17T18:24:16.110 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:16.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:16.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:16.212 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:16.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:16.251 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:16.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:16.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:15 smithi165 ceph-mon[105517]: pgmap v3449: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 14893/149193 objects degraded (9.982%); 15816/149193 objects misplaced (10.601%); 8.3 MiB/s, 132 objects/s recovering 2024-08-17T18:24:16.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:15 smithi165 ceph-mon[105517]: osdmap e1747: 8 total, 8 up, 8 in 2024-08-17T18:24:16.352 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:16.389 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:16.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:16.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:16.491 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:16.525 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:16.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:16.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:16.627 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:16.662 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:16.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:16.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:16.763 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:16.795 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:16.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:16.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:16.897 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:16.936 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:16.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:16.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:17.038 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:17.074 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:17.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:17.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:17.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:17.204 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:17.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:17.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:17.245 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:24:17.305 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:17.333 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:17.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:17.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:17.435 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:17.490 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:17.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:17.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:17.591 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:17.620 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:17.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:17.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:17.722 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:17.751 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:17.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:17.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:17.828 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:24:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:24:17] "GET /metrics HTTP/1.1" 200 36736 "" "Prometheus/2.51.0" 2024-08-17T18:24:17.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:17.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:17.883 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:17.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:17.984 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:18.014 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:18.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:18.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:18.116 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:18.153 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:18.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:18.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:18.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:17 smithi136 ceph-mon[109861]: pgmap v3451: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 14599/144222 objects degraded (10.123%); 15109/144222 objects misplaced (10.476%); 8.8 MiB/s, 140 objects/s recovering 2024-08-17T18:24:18.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:17 smithi136 ceph-mon[115078]: pgmap v3451: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 14599/144222 objects degraded (10.123%); 15109/144222 objects misplaced (10.476%); 8.8 MiB/s, 140 objects/s recovering 2024-08-17T18:24:18.255 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:18.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:17 smithi165 ceph-mon[105517]: pgmap v3451: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 14599/144222 objects degraded (10.123%); 15109/144222 objects misplaced (10.476%); 8.8 MiB/s, 140 objects/s recovering 2024-08-17T18:24:18.285 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:18.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:18.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:18.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:18.416 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:18.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:18.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:18.518 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:18.541 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:18.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:18.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:18.643 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:18.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:18.671 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:18.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:18.772 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:18.807 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:18.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:18.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:18.909 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:18.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:18.949 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:18.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:19.050 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:19.080 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:19.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:19.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:19.183 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:19.225 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:19.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:19.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:19.327 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:19.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:19.357 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:19.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:19.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:19.484 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:19.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:19.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:19.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:19.614 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:19.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:19.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:19.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:19.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:19.751 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:19.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:19.852 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:19.879 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:19.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:19.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:19.981 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:20.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:20.011 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:20.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:20.112 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:20.142 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:20.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:20.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:20.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:19 smithi136 ceph-mon[109861]: pgmap v3452: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 723 op/s; 14599/141261 objects degraded (10.335%); 14904/141261 objects misplaced (10.551%); 8.8 MiB/s, 140 objects/s recovering 2024-08-17T18:24:20.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:19 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 14599/141261 objects degraded (10.335%), 8 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:20.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:19 smithi136 ceph-mon[115078]: pgmap v3452: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 723 op/s; 14599/141261 objects degraded (10.335%); 14904/141261 objects misplaced (10.551%); 8.8 MiB/s, 140 objects/s recovering 2024-08-17T18:24:20.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:19 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 14599/141261 objects degraded (10.335%), 8 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:20.244 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:20.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:19 smithi165 ceph-mon[105517]: pgmap v3452: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 723 op/s; 14599/141261 objects degraded (10.335%); 14904/141261 objects misplaced (10.551%); 8.8 MiB/s, 140 objects/s recovering 2024-08-17T18:24:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:19 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 14599/141261 objects degraded (10.335%), 8 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:20.268 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:20.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:20.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:20.369 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:20.400 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:20.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:20.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:20.498 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:24:20.501 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:20.554 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:20.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:20.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:20.566 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:24:20.656 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:20.689 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:20.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:20.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:20.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:20.821 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:20.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:20.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:20.923 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:20.959 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:20.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:20.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:21.060 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:21.091 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:21.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:21.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:21.193 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:21.222 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:21.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:21.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:21.324 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:21.349 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:21.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:21.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:21.452 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:21.478 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:21.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:21.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:21.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:21.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:21.611 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:21.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:21.713 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:21.747 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:21.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:21.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:21.849 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:21.877 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:21.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:21.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:21.979 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:22.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:22.014 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:22.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:22.022 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:24:22.022 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:24:22.051 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:24:22.051 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:24:22.081 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:24:22.081 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:24:22.110 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:24:22.111 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:24:22.111 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:21 smithi136 ceph-mon[109861]: pgmap v3453: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 756 op/s; 13829/136593 objects degraded (10.124%); 14324/136593 objects misplaced (10.487%); 13 MiB/s, 207 objects/s recovering 2024-08-17T18:24:22.111 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:24:22.111 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:24:22.112 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:21 smithi136 ceph-mon[115078]: pgmap v3453: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 756 op/s; 13829/136593 objects degraded (10.124%); 14324/136593 objects misplaced (10.487%); 13 MiB/s, 207 objects/s recovering 2024-08-17T18:24:22.112 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:24:22.112 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:24:22.116 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:22.140 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:24:22.141 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:24:22.144 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:22.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:22.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:22.175 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:24:22.175 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:24:22.207 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:24:22.207 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:24:22.243 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:24:22.243 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:24:22.246 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:21 smithi165 ceph-mon[105517]: pgmap v3453: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 756 op/s; 13829/136593 objects degraded (10.124%); 14324/136593 objects misplaced (10.487%); 13 MiB/s, 207 objects/s recovering 2024-08-17T18:24:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:24:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:24:22.270 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:22.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:22.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:22.372 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:22.411 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:22.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:22.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:22.513 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:22.538 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:22.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:22.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:22.640 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:22.669 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:22.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:22.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:22.771 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:22.800 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:22.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:22.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:22.846 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.7 to 1 (10000) 2024-08-17T18:24:22.902 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:22.932 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:22.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:22.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:23.034 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:23.066 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:23.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:23.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:23.167 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:23.195 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:23.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:23.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:23.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:22 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/299434361' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:24:23.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:22 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/720798570' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T18:24:23.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:22 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T18:24:23.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:22 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/299434361' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:24:23.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:22 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/720798570' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T18:24:23.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:22 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T18:24:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:22 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/299434361' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:24:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:22 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/720798570' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T18:24:23.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:22 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T18:24:23.297 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:23.329 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:23.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:23.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:23.431 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:23.460 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:23.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:23.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:23.562 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:23.586 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:23.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:23.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:23.688 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:23.713 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:23.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:23.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:23.814 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:23.839 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:23.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:23.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:23.941 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:23.969 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:23.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:23.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:24.071 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:24.096 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:24.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:24.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:24.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:23 smithi136 ceph-mon[109861]: pgmap v3454: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 564 op/s; 13772/134295 objects degraded (10.255%); 14190/134295 objects misplaced (10.566%); 7.6 MiB/s, 121 objects/s recovering 2024-08-17T18:24:24.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:23 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-17T18:24:24.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:23 smithi136 ceph-mon[109861]: osdmap e1748: 8 total, 8 up, 8 in 2024-08-17T18:24:24.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:23 smithi136 ceph-mon[115078]: pgmap v3454: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 564 op/s; 13772/134295 objects degraded (10.255%); 14190/134295 objects misplaced (10.566%); 7.6 MiB/s, 121 objects/s recovering 2024-08-17T18:24:24.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:23 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-17T18:24:24.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:23 smithi136 ceph-mon[115078]: osdmap e1748: 8 total, 8 up, 8 in 2024-08-17T18:24:24.197 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:24.223 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:24.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:24.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:24.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:23 smithi165 ceph-mon[105517]: pgmap v3454: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 564 op/s; 13772/134295 objects degraded (10.255%); 14190/134295 objects misplaced (10.566%); 7.6 MiB/s, 121 objects/s recovering 2024-08-17T18:24:24.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:23 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-17T18:24:24.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:23 smithi165 ceph-mon[105517]: osdmap e1748: 8 total, 8 up, 8 in 2024-08-17T18:24:24.325 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:24.350 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:24.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:24.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:24.451 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:24.479 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:24.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:24.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:24.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:24.604 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:24.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:24.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:24.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:24.730 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:24.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:24.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:24.833 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:24.857 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:24.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:24.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:24.959 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:24.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:24.985 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:24.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:25.086 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:25.121 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:25.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:25.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:25.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:24 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 13772/134295 objects degraded (10.255%), 8 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:25.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:24 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 13772/134295 objects degraded (10.255%), 8 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:25.223 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:25.253 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:25.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:25.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:25.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:24 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 13772/134295 objects degraded (10.255%), 8 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:25.355 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:25.383 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:25.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:25.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:25.485 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:25.528 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:25.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:25.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:25.630 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:25.657 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:25.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:25.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:25.759 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:25.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:25.795 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:25.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:25.896 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:25.925 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:25.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:25.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:26.027 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:26.056 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:26.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:26.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:26.158 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:26.188 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:26.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:26.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:26.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:25 smithi136 ceph-mon[109861]: pgmap v3456: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 579 op/s; 13772/132189 objects degraded (10.418%); 14024/132189 objects misplaced (10.609%); 6.8 MiB/s, 109 objects/s recovering 2024-08-17T18:24:26.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:25 smithi136 ceph-mon[115078]: pgmap v3456: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 579 op/s; 13772/132189 objects degraded (10.418%); 14024/132189 objects misplaced (10.609%); 6.8 MiB/s, 109 objects/s recovering 2024-08-17T18:24:26.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:25 smithi165 ceph-mon[105517]: pgmap v3456: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 579 op/s; 13772/132189 objects degraded (10.418%); 14024/132189 objects misplaced (10.609%); 6.8 MiB/s, 109 objects/s recovering 2024-08-17T18:24:26.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:26.316 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:26.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:26.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:26.418 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:26.447 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:26.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:26.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:26.549 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:26.577 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:26.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:26.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:26.679 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:26.707 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:26.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:26.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:26.810 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:26.840 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:26.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:26.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:26.942 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:26.968 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:26.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:26.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:27.070 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:27.103 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:27.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:27.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:27.205 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:27.237 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:27.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:27.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:27.339 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:27.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:27.371 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:27.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:27.472 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:27.496 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:27.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:27.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:27.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:27.631 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:27.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:27.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:27.733 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:27.754 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:27.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:27.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:27.856 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:27.864 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:24:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:24:27] "GET /metrics HTTP/1.1" 200 36726 "" "Prometheus/2.51.0" 2024-08-17T18:24:27.884 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:27.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:27.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:27.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:28.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:28.017 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:28.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:28.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:28.147 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:28.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:28.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:28.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:27 smithi136 ceph-mon[109861]: pgmap v3457: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 582 op/s; 10630/126735 objects degraded (8.388%); 17415/126735 objects misplaced (13.741%); 7.1 MiB/s, 139 objects/s recovering 2024-08-17T18:24:28.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:27 smithi136 ceph-mon[115078]: pgmap v3457: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 582 op/s; 10630/126735 objects degraded (8.388%); 17415/126735 objects misplaced (13.741%); 7.1 MiB/s, 139 objects/s recovering 2024-08-17T18:24:28.249 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:28.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:27 smithi165 ceph-mon[105517]: pgmap v3457: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 582 op/s; 10630/126735 objects degraded (8.388%); 17415/126735 objects misplaced (13.741%); 7.1 MiB/s, 139 objects/s recovering 2024-08-17T18:24:28.275 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:28.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:28.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:28.377 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:28.410 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:28.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:28.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:28.512 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:28.543 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:28.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:28.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:28.645 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:28.676 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:28.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:28.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:28.777 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:28.813 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:28.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:28.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:28.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:28.943 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:28.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:28.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:29.045 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:29.076 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:29.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:29.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:29.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:29.204 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:29.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:29.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:29.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:29.336 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:29.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:29.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:29.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:29.469 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:29.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:29.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:29.571 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:29.600 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:29.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:29.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:29.702 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:29.731 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:29.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:29.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:29.833 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:29.868 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:29.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:29.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:29.970 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:29.997 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:29.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:29.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:30.098 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:30.126 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:30.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:30.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:29 smithi136 ceph-mon[109861]: pgmap v3458: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 573 op/s; 10630/124050 objects degraded (8.569%); 17236/124050 objects misplaced (13.894%); 7.1 MiB/s, 139 objects/s recovering 2024-08-17T18:24:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:29 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 10630/124050 objects degraded (8.569%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:30.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:29 smithi136 ceph-mon[115078]: pgmap v3458: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 573 op/s; 10630/124050 objects degraded (8.569%); 17236/124050 objects misplaced (13.894%); 7.1 MiB/s, 139 objects/s recovering 2024-08-17T18:24:30.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:29 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 10630/124050 objects degraded (8.569%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:30.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:29 smithi165 ceph-mon[105517]: pgmap v3458: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 573 op/s; 10630/124050 objects degraded (8.569%); 17236/124050 objects misplaced (13.894%); 7.1 MiB/s, 139 objects/s recovering 2024-08-17T18:24:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:29 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 10630/124050 objects degraded (8.569%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:30.268 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:30.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:30.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:30.370 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:30.404 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:30.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:30.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:30.506 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:30.539 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:30.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:30.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:30.641 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:30.670 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:30.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:30.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:30.772 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:30.799 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:30.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:30.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:30.901 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:30.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:30.935 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:30.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:31.036 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:31.065 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:31.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:31.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:31.167 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:31.207 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:31.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:31.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:31.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:31.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:31.340 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:31.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:31.441 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:31.475 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:31.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:31.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:31.577 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:31.607 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:31.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:31.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:31.709 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:31.735 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:31.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:31.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:31.836 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:31.864 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:31.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:31.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:31.966 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:31.995 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:31.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:31.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:32.096 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:32.131 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:32.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:32.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:32.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:31 smithi136 ceph-mon[109861]: pgmap v3459: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 566 op/s; 10630/119598 objects degraded (8.888%); 16213/119598 objects misplaced (13.556%); 4.2 MiB/s, 93 objects/s recovering 2024-08-17T18:24:32.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:31 smithi136 ceph-mon[115078]: pgmap v3459: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 566 op/s; 10630/119598 objects degraded (8.888%); 16213/119598 objects misplaced (13.556%); 4.2 MiB/s, 93 objects/s recovering 2024-08-17T18:24:32.233 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:32.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:31 smithi165 ceph-mon[105517]: pgmap v3459: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 566 op/s; 10630/119598 objects degraded (8.888%); 16213/119598 objects misplaced (13.556%); 4.2 MiB/s, 93 objects/s recovering 2024-08-17T18:24:32.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:32.267 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:32.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:32.368 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:32.399 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:32.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:32.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:32.501 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:32.528 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:32.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:32.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:32.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:32.664 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:32.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:32.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:32.766 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:32.802 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:32.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:32.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:32.904 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:32.935 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:32.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:32.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:33.037 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:33.066 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:33.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:33.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:33.168 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:33.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:33.207 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:33.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:33.308 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:33.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:33.336 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:33.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:33.437 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:33.466 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:33.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:33.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:33.568 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:33.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:33.602 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:33.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:33.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:33.730 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:33.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:33.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:33.832 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:33.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:33.864 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:33.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:33.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:33 smithi136 ceph-mon[109861]: pgmap v3460: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 564 op/s; 10436/117363 objects degraded (8.892%); 16046/117363 objects misplaced (13.672%); 5.3 MiB/s, 109 objects/s recovering 2024-08-17T18:24:33.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:33 smithi136 ceph-mon[115078]: pgmap v3460: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 564 op/s; 10436/117363 objects degraded (8.892%); 16046/117363 objects misplaced (13.672%); 5.3 MiB/s, 109 objects/s recovering 2024-08-17T18:24:33.965 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:33.991 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:33.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:33.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:34.093 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:34.125 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:34.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:34.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:34.227 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:34.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:33 smithi165 ceph-mon[105517]: pgmap v3460: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 564 op/s; 10436/117363 objects degraded (8.892%); 16046/117363 objects misplaced (13.672%); 5.3 MiB/s, 109 objects/s recovering 2024-08-17T18:24:34.261 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:34.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:34.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:34.363 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:34.390 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:34.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:34.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:34.492 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:34.521 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:34.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:34.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:34.623 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:34.653 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:34.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:34.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:34.755 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:34.786 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:34.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:34.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:34.889 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:34.920 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:34.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:34.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:34.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 3 1 2024-08-17T18:24:35.022 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:35.052 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:35.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:35.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:35.154 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:35.180 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:35.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:35.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:35.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:34 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 10436/117363 objects degraded (8.892%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:35.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:34 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 10436/117363 objects degraded (8.892%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:34 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 10436/117363 objects degraded (8.892%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:35.282 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:35.308 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:35.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:35.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:35.410 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:35.443 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:35.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:35.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:35.545 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:35.585 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:35.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:35.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:35.686 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:35.703 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:24:35.703 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-17T18:24:35.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-17T18:24:35.704 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:24:35.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.704 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:91775 57.12 57:4c5b5116:::benchmark_data_smithi136_389235_object91774:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:24:35.704 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.694206+0000", 2024-08-17T18:24:35.704 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 575.00526135400003, 2024-08-17T18:24:35.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1991839390000001, 2024-08-17T18:24:35.704 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:24:35.704 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:24:35.704 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:24:35.704 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:24:35.705 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:24:35.705 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:24:35.705 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91775 2024-08-17T18:24:35.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.705 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:24:35.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:24:35.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.694206+0000", 2024-08-17T18:24:35.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:24:35.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.694206+0000", 2024-08-17T18:24:35.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:24:35.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.694206+0000", 2024-08-17T18:24:35.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6400000000000001e-07 2024-08-17T18:24:35.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:24:35.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.694222+0000", 2024-08-17T18:24:35.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5503999999999999e-05 2024-08-17T18:24:35.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:24:35.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.694230+0000", 2024-08-17T18:24:35.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2660000000000001e-06 2024-08-17T18:24:35.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:24:35.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.694235+0000", 2024-08-17T18:24:35.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5940000000000004e-06 2024-08-17T18:24:35.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.694293+0000", 2024-08-17T18:24:35.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8186999999999997e-05 2024-08-17T18:24:35.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.694303+0000", 2024-08-17T18:24:35.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0059e-05 2024-08-17T18:24:35.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.646421+0000", 2024-08-17T18:24:35.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.95211796000000004 2024-08-17T18:24:35.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.646435+0000", 2024-08-17T18:24:35.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3908e-05 2024-08-17T18:24:35.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.887386+0000", 2024-08-17T18:24:35.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.240951269 2024-08-17T18:24:35.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:24:35.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.887571+0000", 2024-08-17T18:24:35.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018437799999999999 2024-08-17T18:24:35.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:24:35.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.887823+0000", 2024-08-17T18:24:35.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025225300000000003 2024-08-17T18:24:35.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:24:35.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.892158+0000", 2024-08-17T18:24:35.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.004335401 2024-08-17T18:24:35.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.892215+0000", 2024-08-17T18:24:35.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6533000000000002e-05 2024-08-17T18:24:35.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.892345+0000", 2024-08-17T18:24:35.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012994600000000001 2024-08-17T18:24:35.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:24:35.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.892406+0000", 2024-08-17T18:24:35.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1282e-05 2024-08-17T18:24:35.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:24:35.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.893389+0000", 2024-08-17T18:24:35.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00098353500000000001 2024-08-17T18:24:35.715 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.715 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:24:35.715 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.715 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:91798 57.12 57:4d0d602d:::benchmark_data_smithi136_389235_object91797:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:24:35.715 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.714863+0000", 2024-08-17T18:24:35.715 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 574.984603752, 2024-08-17T18:24:35.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.186937039, 2024-08-17T18:24:35.715 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:24:35.715 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:24:35.715 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:24:35.715 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:24:35.716 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:24:35.716 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:24:35.716 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91798 2024-08-17T18:24:35.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.716 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:24:35.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:24:35.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.714863+0000", 2024-08-17T18:24:35.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:24:35.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.714863+0000", 2024-08-17T18:24:35.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:24:35.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.714864+0000", 2024-08-17T18:24:35.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5099999999999998e-07 2024-08-17T18:24:35.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:24:35.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.714889+0000", 2024-08-17T18:24:35.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5029e-05 2024-08-17T18:24:35.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:24:35.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.714897+0000", 2024-08-17T18:24:35.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2570000000000007e-06 2024-08-17T18:24:35.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:24:35.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.714903+0000", 2024-08-17T18:24:35.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9390000000000002e-06 2024-08-17T18:24:35.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.714971+0000", 2024-08-17T18:24:35.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7811999999999994e-05 2024-08-17T18:24:35.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.714982+0000", 2024-08-17T18:24:35.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0814e-05 2024-08-17T18:24:35.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.646899+0000", 2024-08-17T18:24:35.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93191730399999995 2024-08-17T18:24:35.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.646905+0000", 2024-08-17T18:24:35.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6779999999999997e-06 2024-08-17T18:24:35.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.889432+0000", 2024-08-17T18:24:35.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.242527468 2024-08-17T18:24:35.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:24:35.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.889612+0000", 2024-08-17T18:24:35.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000179722 2024-08-17T18:24:35.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:24:35.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.889890+0000", 2024-08-17T18:24:35.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027764899999999998 2024-08-17T18:24:35.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.892278+0000", 2024-08-17T18:24:35.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023882690000000002 2024-08-17T18:24:35.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:24:35.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.892292+0000", 2024-08-17T18:24:35.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4258e-05 2024-08-17T18:24:35.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.900664+0000", 2024-08-17T18:24:35.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0083714710000000001 2024-08-17T18:24:35.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:24:35.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.900720+0000", 2024-08-17T18:24:35.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6434000000000002e-05 2024-08-17T18:24:35.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:24:35.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.901800+0000", 2024-08-17T18:24:35.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001079984 2024-08-17T18:24:35.724 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.724 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:24:35.724 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.724 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:91831 57.12 57:49f84744:::benchmark_data_smithi136_389235_object91830:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:24:35.724 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.733890+0000", 2024-08-17T18:24:35.725 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 574.96557685499999, 2024-08-17T18:24:35.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1688886919999999, 2024-08-17T18:24:35.725 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:24:35.725 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:24:35.725 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:24:35.725 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:24:35.725 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:24:35.725 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:24:35.725 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91831 2024-08-17T18:24:35.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.725 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:24:35.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:24:35.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.733890+0000", 2024-08-17T18:24:35.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:24:35.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.733890+0000", 2024-08-17T18:24:35.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:24:35.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.733891+0000", 2024-08-17T18:24:35.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7400000000000002e-07 2024-08-17T18:24:35.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:24:35.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.733910+0000", 2024-08-17T18:24:35.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9232000000000002e-05 2024-08-17T18:24:35.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:24:35.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.733921+0000", 2024-08-17T18:24:35.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0787e-05 2024-08-17T18:24:35.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:24:35.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.733928+0000", 2024-08-17T18:24:35.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4320000000000002e-06 2024-08-17T18:24:35.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.733984+0000", 2024-08-17T18:24:35.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6233000000000002e-05 2024-08-17T18:24:35.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734004+0000", 2024-08-17T18:24:35.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9307e-05 2024-08-17T18:24:35.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.646939+0000", 2024-08-17T18:24:35.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.91293501099999996 2024-08-17T18:24:35.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.646948+0000", 2024-08-17T18:24:35.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9709999999999993e-06 2024-08-17T18:24:35.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.890760+0000", 2024-08-17T18:24:35.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2438118359999999 2024-08-17T18:24:35.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:24:35.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.890914+0000", 2024-08-17T18:24:35.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015469099999999999 2024-08-17T18:24:35.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:24:35.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.891161+0000", 2024-08-17T18:24:35.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024684400000000002 2024-08-17T18:24:35.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:24:35.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.894974+0000", 2024-08-17T18:24:35.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038126700000000002 2024-08-17T18:24:35.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.895037+0000", 2024-08-17T18:24:35.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3083000000000006e-05 2024-08-17T18:24:35.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.901890+0000", 2024-08-17T18:24:35.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0068527170000000004 2024-08-17T18:24:35.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:24:35.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.901927+0000", 2024-08-17T18:24:35.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7088999999999997e-05 2024-08-17T18:24:35.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:24:35.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.902779+0000", 2024-08-17T18:24:35.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00085201500000000002 2024-08-17T18:24:35.732 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.732 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:24:35.733 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.733 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:91833 57.12 57:4fab0a31:::benchmark_data_smithi136_389235_object91832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:24:35.733 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.734763+0000", 2024-08-17T18:24:35.733 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 574.96470365300002, 2024-08-17T18:24:35.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1724328439999998, 2024-08-17T18:24:35.733 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:24:35.733 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:24:35.733 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:24:35.733 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:24:35.733 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:24:35.733 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:24:35.734 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91833 2024-08-17T18:24:35.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.734 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:24:35.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:24:35.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734763+0000", 2024-08-17T18:24:35.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:24:35.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734763+0000", 2024-08-17T18:24:35.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:24:35.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734764+0000", 2024-08-17T18:24:35.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7699999999999992e-07 2024-08-17T18:24:35.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:24:35.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734787+0000", 2024-08-17T18:24:35.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3243000000000001e-05 2024-08-17T18:24:35.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:24:35.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734800+0000", 2024-08-17T18:24:35.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2903e-05 2024-08-17T18:24:35.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:24:35.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734807+0000", 2024-08-17T18:24:35.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1459999999999999e-06 2024-08-17T18:24:35.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734860+0000", 2024-08-17T18:24:35.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2874999999999998e-05 2024-08-17T18:24:35.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.734869+0000", 2024-08-17T18:24:35.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9039999999999998e-06 2024-08-17T18:24:35.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.646980+0000", 2024-08-17T18:24:35.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.91211099200000001 2024-08-17T18:24:35.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.646987+0000", 2024-08-17T18:24:35.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1149999999999996e-06 2024-08-17T18:24:35.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.893439+0000", 2024-08-17T18:24:35.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2464515359999999 2024-08-17T18:24:35.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:24:35.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.893736+0000", 2024-08-17T18:24:35.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029668499999999998 2024-08-17T18:24:35.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:24:35.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.893960+0000", 2024-08-17T18:24:35.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000224696 2024-08-17T18:24:35.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.897423+0000", 2024-08-17T18:24:35.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034624370000000001 2024-08-17T18:24:35.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:24:35.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.897433+0000", 2024-08-17T18:24:35.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0197000000000001e-05 2024-08-17T18:24:35.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.906207+0000", 2024-08-17T18:24:35.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0087737289999999992 2024-08-17T18:24:35.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:24:35.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.906242+0000", 2024-08-17T18:24:35.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5824000000000001e-05 2024-08-17T18:24:35.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:24:35.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907196+0000", 2024-08-17T18:24:35.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00095358499999999996 2024-08-17T18:24:35.741 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.741 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:24:35.741 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.741 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:91851 57.12 57:4935f3b2:::benchmark_data_smithi136_389235_object91850:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:24:35.741 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.746420+0000", 2024-08-17T18:24:35.741 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 574.95304659299995, 2024-08-17T18:24:35.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1658063599999999, 2024-08-17T18:24:35.741 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:24:35.741 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:24:35.741 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:24:35.742 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:24:35.742 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:24:35.742 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:24:35.742 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91851 2024-08-17T18:24:35.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.742 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:24:35.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:24:35.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.746420+0000", 2024-08-17T18:24:35.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:24:35.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.746420+0000", 2024-08-17T18:24:35.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:24:35.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.746421+0000", 2024-08-17T18:24:35.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3900000000000004e-07 2024-08-17T18:24:35.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:24:35.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.746444+0000", 2024-08-17T18:24:35.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2609999999999999e-05 2024-08-17T18:24:35.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:24:35.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.746452+0000", 2024-08-17T18:24:35.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0499999999999992e-06 2024-08-17T18:24:35.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:24:35.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.746457+0000", 2024-08-17T18:24:35.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3310000000000003e-06 2024-08-17T18:24:35.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.747175+0000", 2024-08-17T18:24:35.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00071805899999999995 2024-08-17T18:24:35.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.747184+0000", 2024-08-17T18:24:35.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2960000000000004e-06 2024-08-17T18:24:35.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647023+0000", 2024-08-17T18:24:35.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.89983893699999995 2024-08-17T18:24:35.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647032+0000", 2024-08-17T18:24:35.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7339999999999997e-06 2024-08-17T18:24:35.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.895088+0000", 2024-08-17T18:24:35.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2480557990000001 2024-08-17T18:24:35.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:24:35.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.895247+0000", 2024-08-17T18:24:35.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000159512 2024-08-17T18:24:35.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:24:35.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.895471+0000", 2024-08-17T18:24:35.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022369299999999999 2024-08-17T18:24:35.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:24:35.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.899385+0000", 2024-08-17T18:24:35.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003913583 2024-08-17T18:24:35.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.900594+0000", 2024-08-17T18:24:35.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012091599999999999 2024-08-17T18:24:35.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.910847+0000", 2024-08-17T18:24:35.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010253536000000001 2024-08-17T18:24:35.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:24:35.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.910887+0000", 2024-08-17T18:24:35.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0167999999999999e-05 2024-08-17T18:24:35.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:24:35.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.912227+0000", 2024-08-17T18:24:35.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0013392530000000001 2024-08-17T18:24:35.749 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.749 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:24:35.749 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.749 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:91877 57.12 57:4e2ca095:::benchmark_data_smithi136_389235_object91876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:24:35.749 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.764679+0000", 2024-08-17T18:24:35.749 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 574.93478780800001, 2024-08-17T18:24:35.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.148880057, 2024-08-17T18:24:35.749 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:24:35.749 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:24:35.749 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:24:35.749 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:24:35.750 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:24:35.750 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:24:35.750 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91877 2024-08-17T18:24:35.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.750 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:24:35.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:24:35.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.764679+0000", 2024-08-17T18:24:35.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:24:35.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.764679+0000", 2024-08-17T18:24:35.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:24:35.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.764681+0000", 2024-08-17T18:24:35.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5239999999999999e-06 2024-08-17T18:24:35.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:24:35.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.764715+0000", 2024-08-17T18:24:35.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3964000000000002e-05 2024-08-17T18:24:35.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:24:35.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.764727+0000", 2024-08-17T18:24:35.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2762000000000001e-05 2024-08-17T18:24:35.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:24:35.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.764735+0000", 2024-08-17T18:24:35.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3769999999999998e-06 2024-08-17T18:24:35.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.764809+0000", 2024-08-17T18:24:35.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3935000000000003e-05 2024-08-17T18:24:35.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.764821+0000", 2024-08-17T18:24:35.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2024e-05 2024-08-17T18:24:35.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647058+0000", 2024-08-17T18:24:35.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.88223715999999996 2024-08-17T18:24:35.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647062+0000", 2024-08-17T18:24:35.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1230000000000002e-06 2024-08-17T18:24:35.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.896097+0000", 2024-08-17T18:24:35.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.249034932 2024-08-17T18:24:35.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:24:35.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.896247+0000", 2024-08-17T18:24:35.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000150309 2024-08-17T18:24:35.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:24:35.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.896475+0000", 2024-08-17T18:24:35.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000227815 2024-08-17T18:24:35.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:24:35.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.900604+0000", 2024-08-17T18:24:35.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0041288979999999998 2024-08-17T18:24:35.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.901852+0000", 2024-08-17T18:24:35.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012485809999999999 2024-08-17T18:24:35.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.912289+0000", 2024-08-17T18:24:35.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010436453 2024-08-17T18:24:35.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:24:35.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.912355+0000", 2024-08-17T18:24:35.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5947999999999999e-05 2024-08-17T18:24:35.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:24:35.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.913559+0000", 2024-08-17T18:24:35.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012042520000000001 2024-08-17T18:24:35.757 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.757 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:24:35.757 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.757 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:91955 57.12 57:4acb1630:::benchmark_data_smithi136_389235_object91954:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:24:35.757 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.812317+0000", 2024-08-17T18:24:35.757 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 574.88714951199995, 2024-08-17T18:24:35.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1059336489999998, 2024-08-17T18:24:35.757 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:24:35.757 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:24:35.757 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:24:35.758 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:24:35.758 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:24:35.758 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:24:35.758 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91955 2024-08-17T18:24:35.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.758 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:24:35.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:24:35.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.812317+0000", 2024-08-17T18:24:35.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:24:35.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.812317+0000", 2024-08-17T18:24:35.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:24:35.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.812319+0000", 2024-08-17T18:24:35.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.31e-06 2024-08-17T18:24:35.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:24:35.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.812338+0000", 2024-08-17T18:24:35.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8881000000000001e-05 2024-08-17T18:24:35.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:24:35.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.812345+0000", 2024-08-17T18:24:35.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9470000000000003e-06 2024-08-17T18:24:35.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:24:35.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.812350+0000", 2024-08-17T18:24:35.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3919999999999996e-06 2024-08-17T18:24:35.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.812427+0000", 2024-08-17T18:24:35.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6713999999999998e-05 2024-08-17T18:24:35.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.812437+0000", 2024-08-17T18:24:35.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0720000000000001e-05 2024-08-17T18:24:35.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647088+0000", 2024-08-17T18:24:35.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.83465034999999999 2024-08-17T18:24:35.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647095+0000", 2024-08-17T18:24:35.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6599999999999995e-06 2024-08-17T18:24:35.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.897471+0000", 2024-08-17T18:24:35.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2503756109999999 2024-08-17T18:24:35.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:24:35.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.897618+0000", 2024-08-17T18:24:35.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000146699 2024-08-17T18:24:35.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:24:35.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.897843+0000", 2024-08-17T18:24:35.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022513899999999999 2024-08-17T18:24:35.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:24:35.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.900609+0000", 2024-08-17T18:24:35.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0027664709999999999 2024-08-17T18:24:35.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.903881+0000", 2024-08-17T18:24:35.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032721510000000001 2024-08-17T18:24:35.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.916916+0000", 2024-08-17T18:24:35.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013034108000000001 2024-08-17T18:24:35.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:24:35.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.916997+0000", 2024-08-17T18:24:35.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1508000000000006e-05 2024-08-17T18:24:35.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:24:35.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.918251+0000", 2024-08-17T18:24:35.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012539879999999999 2024-08-17T18:24:35.765 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.765 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:24:35.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:35.765 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:35.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:35.767 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.767 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:91997 57.12 57:480a6a35:::benchmark_data_smithi136_389235_object91996:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:24:35.767 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.839160+0000", 2024-08-17T18:24:35.767 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 574.860306713, 2024-08-17T18:24:35.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0802216869999999, 2024-08-17T18:24:35.767 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:24:35.767 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:24:35.767 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:24:35.767 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:24:35.767 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:24:35.768 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:24:35.768 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91997 2024-08-17T18:24:35.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.768 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:24:35.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:24:35.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.839160+0000", 2024-08-17T18:24:35.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:24:35.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.839160+0000", 2024-08-17T18:24:35.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:24:35.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.839161+0000", 2024-08-17T18:24:35.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1510000000000001e-06 2024-08-17T18:24:35.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:24:35.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.839185+0000", 2024-08-17T18:24:35.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3805e-05 2024-08-17T18:24:35.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:24:35.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.839194+0000", 2024-08-17T18:24:35.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1500000000000005e-06 2024-08-17T18:24:35.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:24:35.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.839201+0000", 2024-08-17T18:24:35.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2720000000000003e-06 2024-08-17T18:24:35.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.839374+0000", 2024-08-17T18:24:35.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017318800000000001 2024-08-17T18:24:35.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.839420+0000", 2024-08-17T18:24:35.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6050999999999997e-05 2024-08-17T18:24:35.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647119+0000", 2024-08-17T18:24:35.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.80769961099999998 2024-08-17T18:24:35.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647123+0000", 2024-08-17T18:24:35.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9400000000000004e-06 2024-08-17T18:24:35.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.898480+0000", 2024-08-17T18:24:35.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2513567079999999 2024-08-17T18:24:35.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:24:35.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.898570+0000", 2024-08-17T18:24:35.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0339000000000004e-05 2024-08-17T18:24:35.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:24:35.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.898732+0000", 2024-08-17T18:24:35.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000161356 2024-08-17T18:24:35.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:24:35.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.901862+0000", 2024-08-17T18:24:35.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031297849999999999 2024-08-17T18:24:35.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.903932+0000", 2024-08-17T18:24:35.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020700010000000001 2024-08-17T18:24:35.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.918308+0000", 2024-08-17T18:24:35.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014376938000000001 2024-08-17T18:24:35.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:24:35.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.918369+0000", 2024-08-17T18:24:35.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0520999999999999e-05 2024-08-17T18:24:35.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:24:35.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.919382+0000", 2024-08-17T18:24:35.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001012871 2024-08-17T18:24:35.775 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.775 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:24:35.775 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.775 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:92000 57.12 57:4c400e2c:::benchmark_data_smithi136_389235_object91999:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:24:35.775 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.840632+0000", 2024-08-17T18:24:35.775 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 574.85883532000003, 2024-08-17T18:24:35.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0960051650000002, 2024-08-17T18:24:35.776 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:24:35.776 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:24:35.776 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:24:35.776 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:24:35.776 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:24:35.776 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:24:35.776 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 92000 2024-08-17T18:24:35.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.776 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:24:35.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:24:35.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.840632+0000", 2024-08-17T18:24:35.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:24:35.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.840632+0000", 2024-08-17T18:24:35.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:24:35.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.840633+0000", 2024-08-17T18:24:35.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.404e-06 2024-08-17T18:24:35.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:24:35.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.840653+0000", 2024-08-17T18:24:35.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0111000000000001e-05 2024-08-17T18:24:35.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:24:35.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.840664+0000", 2024-08-17T18:24:35.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0988999999999999e-05 2024-08-17T18:24:35.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:24:35.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.840671+0000", 2024-08-17T18:24:35.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4810000000000003e-06 2024-08-17T18:24:35.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.840748+0000", 2024-08-17T18:24:35.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7634999999999998e-05 2024-08-17T18:24:35.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.840758+0000", 2024-08-17T18:24:35.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6309999999999998e-06 2024-08-17T18:24:35.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647137+0000", 2024-08-17T18:24:35.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.80637933399999995 2024-08-17T18:24:35.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647141+0000", 2024-08-17T18:24:35.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8260000000000003e-06 2024-08-17T18:24:35.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.899425+0000", 2024-08-17T18:24:35.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2522841220000001 2024-08-17T18:24:35.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:24:35.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.899560+0000", 2024-08-17T18:24:35.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013490299999999999 2024-08-17T18:24:35.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:24:35.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.899790+0000", 2024-08-17T18:24:35.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0002299 2024-08-17T18:24:35.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:24:35.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.903824+0000", 2024-08-17T18:24:35.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040337039999999999 2024-08-17T18:24:35.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.903988+0000", 2024-08-17T18:24:35.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000164752 2024-08-17T18:24:35.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.935038+0000", 2024-08-17T18:24:35.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.031049243000000001 2024-08-17T18:24:35.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:24:35.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.935099+0000", 2024-08-17T18:24:35.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1532000000000007e-05 2024-08-17T18:24:35.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:24:35.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.936637+0000", 2024-08-17T18:24:35.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001537598 2024-08-17T18:24:35.783 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.783 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:24:35.783 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.783 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:92004 57.12 57:4b569127:::benchmark_data_smithi136_389235_object92003:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:24:35.783 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.841912+0000", 2024-08-17T18:24:35.783 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 574.85755451800003, 2024-08-17T18:24:35.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.095822149, 2024-08-17T18:24:35.784 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:24:35.784 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:24:35.784 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:24:35.784 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:24:35.784 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:24:35.784 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:24:35.784 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 92004 2024-08-17T18:24:35.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.784 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:24:35.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:24:35.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.841912+0000", 2024-08-17T18:24:35.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:24:35.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.841912+0000", 2024-08-17T18:24:35.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:24:35.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.841914+0000", 2024-08-17T18:24:35.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2890000000000001e-06 2024-08-17T18:24:35.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:24:35.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.841937+0000", 2024-08-17T18:24:35.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3725000000000001e-05 2024-08-17T18:24:35.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:24:35.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.841950+0000", 2024-08-17T18:24:35.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3012e-05 2024-08-17T18:24:35.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:24:35.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.841957+0000", 2024-08-17T18:24:35.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8140000000000003e-06 2024-08-17T18:24:35.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.842028+0000", 2024-08-17T18:24:35.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0962000000000003e-05 2024-08-17T18:24:35.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.842036+0000", 2024-08-17T18:24:35.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9049999999999997e-06 2024-08-17T18:24:35.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647172+0000", 2024-08-17T18:24:35.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.80513599999999996 2024-08-17T18:24:35.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647180+0000", 2024-08-17T18:24:35.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0749999999999998e-06 2024-08-17T18:24:35.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.902828+0000", 2024-08-17T18:24:35.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.255647797 2024-08-17T18:24:35.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:24:35.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.902942+0000", 2024-08-17T18:24:35.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113795 2024-08-17T18:24:35.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:24:35.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.903135+0000", 2024-08-17T18:24:35.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000193043 2024-08-17T18:24:35.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:24:35.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907209+0000", 2024-08-17T18:24:35.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040746159999999997 2024-08-17T18:24:35.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907255+0000", 2024-08-17T18:24:35.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5674999999999999e-05 2024-08-17T18:24:35.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.936682+0000", 2024-08-17T18:24:35.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.029426833999999999 2024-08-17T18:24:35.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:24:35.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.936745+0000", 2024-08-17T18:24:35.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2617e-05 2024-08-17T18:24:35.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:24:35.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.937735+0000", 2024-08-17T18:24:35.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00098999000000000005 2024-08-17T18:24:35.791 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.791 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:24:35.791 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.791 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:92015 57.12 57:4fcb2167:::benchmark_data_smithi136_389235_object92014:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:24:35.791 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.850088+0000", 2024-08-17T18:24:35.791 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 574.84937932599996, 2024-08-17T18:24:35.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0964700700000001, 2024-08-17T18:24:35.791 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:24:35.792 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:24:35.792 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:24:35.792 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:24:35.792 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:24:35.792 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:24:35.792 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 92015 2024-08-17T18:24:35.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.792 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:24:35.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:24:35.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.850088+0000", 2024-08-17T18:24:35.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:24:35.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.850088+0000", 2024-08-17T18:24:35.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:24:35.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.850089+0000", 2024-08-17T18:24:35.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.063e-06 2024-08-17T18:24:35.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:24:35.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.850109+0000", 2024-08-17T18:24:35.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0438999999999999e-05 2024-08-17T18:24:35.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:24:35.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.850121+0000", 2024-08-17T18:24:35.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1982e-05 2024-08-17T18:24:35.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:24:35.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.850128+0000", 2024-08-17T18:24:35.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8990000000000004e-06 2024-08-17T18:24:35.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.850188+0000", 2024-08-17T18:24:35.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0013999999999998e-05 2024-08-17T18:24:35.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.850197+0000", 2024-08-17T18:24:35.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9679999999999995e-06 2024-08-17T18:24:35.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647204+0000", 2024-08-17T18:24:35.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.79700673499999997 2024-08-17T18:24:35.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647208+0000", 2024-08-17T18:24:35.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8940000000000003e-06 2024-08-17T18:24:35.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.904025+0000", 2024-08-17T18:24:35.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.256817004 2024-08-17T18:24:35.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:24:35.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.904124+0000", 2024-08-17T18:24:35.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9591000000000002e-05 2024-08-17T18:24:35.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:24:35.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.904271+0000", 2024-08-17T18:24:35.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014643300000000001 2024-08-17T18:24:35.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907318+0000", 2024-08-17T18:24:35.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0030469889999999999 2024-08-17T18:24:35.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:24:35.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907332+0000", 2024-08-17T18:24:35.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4800000000000001e-05 2024-08-17T18:24:35.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.944953+0000", 2024-08-17T18:24:35.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0376207 2024-08-17T18:24:35.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:24:35.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.945020+0000", 2024-08-17T18:24:35.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6569999999999995e-05 2024-08-17T18:24:35.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:24:35.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.946558+0000", 2024-08-17T18:24:35.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001537989 2024-08-17T18:24:35.799 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.799 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:24:35.799 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.799 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:92043 57.12 57:4c633a12:::benchmark_data_smithi136_389235_object92042:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:24:35.799 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.878840+0000", 2024-08-17T18:24:35.800 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 574.82062718400005, 2024-08-17T18:24:35.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0690841519999998, 2024-08-17T18:24:35.800 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:24:35.800 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:24:35.800 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:24:35.800 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:24:35.800 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:24:35.800 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:24:35.800 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 92043 2024-08-17T18:24:35.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.800 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:24:35.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:24:35.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.878840+0000", 2024-08-17T18:24:35.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:24:35.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.878840+0000", 2024-08-17T18:24:35.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:24:35.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.878842+0000", 2024-08-17T18:24:35.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.835e-06 2024-08-17T18:24:35.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:24:35.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.878872+0000", 2024-08-17T18:24:35.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0314000000000001e-05 2024-08-17T18:24:35.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:24:35.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.878887+0000", 2024-08-17T18:24:35.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5167999999999999e-05 2024-08-17T18:24:35.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:24:35.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.878894+0000", 2024-08-17T18:24:35.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3470000000000001e-06 2024-08-17T18:24:35.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.878959+0000", 2024-08-17T18:24:35.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4885e-05 2024-08-17T18:24:35.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.878973+0000", 2024-08-17T18:24:35.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3757000000000001e-05 2024-08-17T18:24:35.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647236+0000", 2024-08-17T18:24:35.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.76826348300000002 2024-08-17T18:24:35.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647244+0000", 2024-08-17T18:24:35.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.452e-06 2024-08-17T18:24:35.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.904870+0000", 2024-08-17T18:24:35.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2576263270000001 2024-08-17T18:24:35.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:24:35.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.905024+0000", 2024-08-17T18:24:35.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000153669 2024-08-17T18:24:35.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:24:35.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.905261+0000", 2024-08-17T18:24:35.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023719299999999999 2024-08-17T18:24:35.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:24:35.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907339+0000", 2024-08-17T18:24:35.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020783189999999999 2024-08-17T18:24:35.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907391+0000", 2024-08-17T18:24:35.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2040999999999999e-05 2024-08-17T18:24:35.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.946619+0000", 2024-08-17T18:24:35.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039227646999999997 2024-08-17T18:24:35.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:24:35.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.946682+0000", 2024-08-17T18:24:35.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3275999999999995e-05 2024-08-17T18:24:35.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:24:35.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.947924+0000", 2024-08-17T18:24:35.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001241439 2024-08-17T18:24:35.807 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.807 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:24:35.807 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.808 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:92046 57.12 57:49923df3:::benchmark_data_smithi136_389235_object92045:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:24:35.808 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:00.881358+0000", 2024-08-17T18:24:35.808 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 574.81810858599999, 2024-08-17T18:24:35.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0681408019999998, 2024-08-17T18:24:35.808 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:24:35.808 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:24:35.808 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:24:35.808 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:24:35.808 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:24:35.808 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:24:35.808 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 92046 2024-08-17T18:24:35.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.809 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:24:35.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:24:35.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.881358+0000", 2024-08-17T18:24:35.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:24:35.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.881358+0000", 2024-08-17T18:24:35.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:24:35.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.881360+0000", 2024-08-17T18:24:35.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7710000000000001e-06 2024-08-17T18:24:35.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.810 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:24:35.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.881387+0000", 2024-08-17T18:24:35.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6526e-05 2024-08-17T18:24:35.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:24:35.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.881401+0000", 2024-08-17T18:24:35.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4718e-05 2024-08-17T18:24:35.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:24:35.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.881410+0000", 2024-08-17T18:24:35.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8240000000000006e-06 2024-08-17T18:24:35.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.881519+0000", 2024-08-17T18:24:35.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109304 2024-08-17T18:24:35.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:00.881531+0000", 2024-08-17T18:24:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1226000000000001e-05 2024-08-17T18:24:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647335+0000", 2024-08-17T18:24:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.76580386099999997 2024-08-17T18:24:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647344+0000", 2024-08-17T18:24:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8700000000000004e-06 2024-08-17T18:24:35.872 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.872 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.872 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.872 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907434+0000", 2024-08-17T18:24:35.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.260090033 2024-08-17T18:24:35.872 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.872 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.872 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:24:35.872 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907581+0000", 2024-08-17T18:24:35.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014699299999999999 2024-08-17T18:24:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:24:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.907826+0000", 2024-08-17T18:24:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024497800000000003 2024-08-17T18:24:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:24:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.913576+0000", 2024-08-17T18:24:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0057498929999999998 2024-08-17T18:24:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.913640+0000", 2024-08-17T18:24:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3325999999999996e-05 2024-08-17T18:24:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.948005+0000", 2024-08-17T18:24:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034365035000000002 2024-08-17T18:24:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:24:35.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.948074+0000", 2024-08-17T18:24:35.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8984000000000002e-05 2024-08-17T18:24:35.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:24:35.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.949499+0000", 2024-08-17T18:24:35.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014254599999999999 2024-08-17T18:24:35.875 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.875 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:24:35.875 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.46277.0:92175 57.12 57:494e5fb2:::benchmark_data_smithi136_389235_object92174:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1623)", 2024-08-17T18:24:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:15:01.049817+0000", 2024-08-17T18:24:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 574.64964998799996, 2024-08-17T18:24:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.900916759, 2024-08-17T18:24:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:24:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:24:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:24:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:24:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.46277", 2024-08-17T18:24:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:24:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 92175 2024-08-17T18:24:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:24:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:24:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.049817+0000", 2024-08-17T18:24:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:24:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.049817+0000", 2024-08-17T18:24:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:24:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.049818+0000", 2024-08-17T18:24:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.562e-06 2024-08-17T18:24:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:24:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.049836+0000", 2024-08-17T18:24:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8045000000000001e-05 2024-08-17T18:24:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:24:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.049849+0000", 2024-08-17T18:24:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2554999999999999e-05 2024-08-17T18:24:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:24:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.049855+0000", 2024-08-17T18:24:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3169999999999998e-06 2024-08-17T18:24:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.049917+0000", 2024-08-17T18:24:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2015999999999997e-05 2024-08-17T18:24:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.049929+0000", 2024-08-17T18:24:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1861e-05 2024-08-17T18:24:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647378+0000", 2024-08-17T18:24:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59744883800000004 2024-08-17T18:24:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:24:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:01.647385+0000", 2024-08-17T18:24:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0929999999999997e-06 2024-08-17T18:24:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.908719+0000", 2024-08-17T18:24:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2613335079999999 2024-08-17T18:24:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:24:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.908865+0000", 2024-08-17T18:24:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014667200000000001 2024-08-17T18:24:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,5", 2024-08-17T18:24:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.909091+0000", 2024-08-17T18:24:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022598799999999999 2024-08-17T18:24:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:24:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.913584+0000", 2024-08-17T18:24:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0044922920000000002 2024-08-17T18:24:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.913700+0000", 2024-08-17T18:24:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011666 2024-08-17T18:24:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:24:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.949569+0000", 2024-08-17T18:24:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035869036999999999 2024-08-17T18:24:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:24:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.949631+0000", 2024-08-17T18:24:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1266000000000006e-05 2024-08-17T18:24:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:24:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:15:02.950734+0000", 2024-08-17T18:24:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001103049 2024-08-17T18:24:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:24:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.884 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:35.885 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.36936.0:141283 58.f e1730/1728 58:fabc5b79:::benchmark_data_smithi136_442124_object27667:head v 1730'8964, mlcod=1711'4451)", 2024-08-17T18:24:35.885 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:13.333086+0000", 2024-08-17T18:24:35.885 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 82.366380798999998, 2024-08-17T18:24:35.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025509171000000001, 2024-08-17T18:24:35.885 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:24:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:24:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:24:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:24:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:24:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:13.333086+0000", 2024-08-17T18:24:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:24:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:13.333086+0000", 2024-08-17T18:24:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:24:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:13.333088+0000", 2024-08-17T18:24:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.863e-06 2024-08-17T18:24:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:24:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:13.333096+0000", 2024-08-17T18:24:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5429999999999994e-06 2024-08-17T18:24:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:24:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:13.333118+0000", 2024-08-17T18:24:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1372000000000001e-05 2024-08-17T18:24:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:24:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:13.333127+0000", 2024-08-17T18:24:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6190000000000006e-06 2024-08-17T18:24:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:13.333190+0000", 2024-08-17T18:24:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2864e-05 2024-08-17T18:24:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:24:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:13.333217+0000", 2024-08-17T18:24:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6485000000000001e-05 2024-08-17T18:24:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:24:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:13.358539+0000", 2024-08-17T18:24:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025321825999999999 2024-08-17T18:24:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:24:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:13.358595+0000", 2024-08-17T18:24:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6598999999999998e-05 2024-08-17T18:24:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:24:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.36936.0:141285 58.f e1730/1728 58:faeeddee:::benchmark_data_smithi136_442124_object27669:head v 1730'8965, mlcod=1711'4451)", 2024-08-17T18:24:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:13.333152+0000", 2024-08-17T18:24:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 82.366314817000003, 2024-08-17T18:24:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025473501999999999, 2024-08-17T18:24:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:24:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:24:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:24:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:24:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:24:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:13.333152+0000", 2024-08-17T18:24:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:24:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:13.333152+0000", 2024-08-17T18:24:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:24:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:13.333152+0000", 2024-08-17T18:24:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9299999999999999e-07 2024-08-17T18:24:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:24:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:13.333156+0000", 2024-08-17T18:24:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.121e-06 2024-08-17T18:24:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:24:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:13.333164+0000", 2024-08-17T18:24:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2910000000000002e-06 2024-08-17T18:24:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:24:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:13.333168+0000", 2024-08-17T18:24:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6890000000000003e-06 2024-08-17T18:24:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:13.338025+0000", 2024-08-17T18:24:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0048563549999999997 2024-08-17T18:24:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:24:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:13.338076+0000", 2024-08-17T18:24:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0819999999999998e-05 2024-08-17T18:24:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:24:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:13.358598+0000", 2024-08-17T18:24:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020522463000000001 2024-08-17T18:24:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:24:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:13.358626+0000", 2024-08-17T18:24:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.747e-05 2024-08-17T18:24:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:24:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.36936.0:168175 58.f e1739/1728 58:f1bbfb43:::benchmark_data_smithi136_442124_object54559:head v 1739'10652, mlcod=1711'4451)", 2024-08-17T18:24:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:53.752735+0000", 2024-08-17T18:24:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.946731915000001, 2024-08-17T18:24:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.033360186999999999, 2024-08-17T18:24:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:24:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:24:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:24:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:24:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:24:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:53.752735+0000", 2024-08-17T18:24:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:24:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:53.752735+0000", 2024-08-17T18:24:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:24:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:53.752736+0000", 2024-08-17T18:24:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.206e-06 2024-08-17T18:24:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:24:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:53.752744+0000", 2024-08-17T18:24:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5039999999999996e-06 2024-08-17T18:24:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:24:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:53.752758+0000", 2024-08-17T18:24:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4055e-05 2024-08-17T18:24:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:24:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:53.752766+0000", 2024-08-17T18:24:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.875e-06 2024-08-17T18:24:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:53.756255+0000", 2024-08-17T18:24:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034892450000000002 2024-08-17T18:24:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:24:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:53.756311+0000", 2024-08-17T18:24:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6357999999999998e-05 2024-08-17T18:24:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:24:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:53.785948+0000", 2024-08-17T18:24:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.029636303999999999 2024-08-17T18:24:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:24:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:53.786095+0000", 2024-08-17T18:24:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014763999999999999 2024-08-17T18:24:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:24:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.36936.0:185370 58.3 e1748/1701 58:c0f11111:::benchmark_data_smithi136_442124_object71754:head v 1748'11593, mlcod=1748'11590)", 2024-08-17T18:24:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:24:25.254999+0000", 2024-08-17T18:24:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.444467418, 2024-08-17T18:24:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.100104384, 2024-08-17T18:24:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:24:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:24:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:24:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:24:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:24:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.254999+0000", 2024-08-17T18:24:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:24:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.254999+0000", 2024-08-17T18:24:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:24:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.255001+0000", 2024-08-17T18:24:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5e-06 2024-08-17T18:24:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:24:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.255009+0000", 2024-08-17T18:24:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5339999999999999e-06 2024-08-17T18:24:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:24:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.255025+0000", 2024-08-17T18:24:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5244e-05 2024-08-17T18:24:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:24:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.255032+0000", 2024-08-17T18:24:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6830000000000008e-06 2024-08-17T18:24:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.255124+0000", 2024-08-17T18:24:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1150999999999995e-05 2024-08-17T18:24:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:24:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.255145+0000", 2024-08-17T18:24:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1192999999999999e-05 2024-08-17T18:24:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:24:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.355059+0000", 2024-08-17T18:24:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.099914739000000002 2024-08-17T18:24:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:24:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.355104+0000", 2024-08-17T18:24:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4339999999999999e-05 2024-08-17T18:24:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:24:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.36936.0:185375 58.a e1748/1741 58:52119336:::benchmark_data_smithi136_442124_object71759:head v 1748'11637, mlcod=1748'11635)", 2024-08-17T18:24:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:24:25.261934+0000", 2024-08-17T18:24:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.437533106, 2024-08-17T18:24:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.096004961, 2024-08-17T18:24:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:24:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:24:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:24:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:24:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:24:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.261934+0000", 2024-08-17T18:24:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:24:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.261934+0000", 2024-08-17T18:24:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:24:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.261935+0000", 2024-08-17T18:24:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.421e-06 2024-08-17T18:24:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:24:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.261944+0000", 2024-08-17T18:24:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9409999999999996e-06 2024-08-17T18:24:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:24:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.349814+0000", 2024-08-17T18:24:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.087870266000000002 2024-08-17T18:24:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:24:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.349820+0000", 2024-08-17T18:24:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0870000000000003e-06 2024-08-17T18:24:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.351354+0000", 2024-08-17T18:24:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015334319999999999 2024-08-17T18:24:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:24:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.351376+0000", 2024-08-17T18:24:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2204999999999998e-05 2024-08-17T18:24:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:24:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.357887+0000", 2024-08-17T18:24:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0065109390000000003 2024-08-17T18:24:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:24:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.357939+0000", 2024-08-17T18:24:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1669999999999998e-05 2024-08-17T18:24:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:24:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.36936.0:185397 58.f e1748/1728 58:f664d8e9:::benchmark_data_smithi136_442124_object71781:head v 1748'11744, mlcod=1713'4595)", 2024-08-17T18:24:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:24:25.319399+0000", 2024-08-17T18:24:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.380067661, 2024-08-17T18:24:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035635549000000002, 2024-08-17T18:24:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:24:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:24:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:24:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:24:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:24:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.319399+0000", 2024-08-17T18:24:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:24:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.319399+0000", 2024-08-17T18:24:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:24:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:24:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.319400+0000", 2024-08-17T18:24:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.155e-06 2024-08-17T18:24:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:24:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.319408+0000", 2024-08-17T18:24:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8080000000000005e-06 2024-08-17T18:24:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:24:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.349632+0000", 2024-08-17T18:24:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030223611000000001 2024-08-17T18:24:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:24:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.349646+0000", 2024-08-17T18:24:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3884e-05 2024-08-17T18:24:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:24:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.349712+0000", 2024-08-17T18:24:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6000000000000005e-05 2024-08-17T18:24:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:24:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.349735+0000", 2024-08-17T18:24:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3302000000000002e-05 2024-08-17T18:24:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:24:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.354984+0000", 2024-08-17T18:24:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0052488250000000004 2024-08-17T18:24:35.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:24:35.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:24:35.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:24:35.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:25.355035+0000", 2024-08-17T18:24:35.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0964000000000002e-05 2024-08-17T18:24:35.914 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.914 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:24:35.914 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.914 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:24:35.914 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:24:35.914 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:24:35.949 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:35.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:35.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:36.050 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:36.078 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:36.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:36.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:36.180 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:36.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:35 smithi136 ceph-mon[109861]: pgmap v3461: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 541 op/s; 10436/115170 objects degraded (9.061%); 15845/115170 objects misplaced (13.758%); 4.5 MiB/s, 93 objects/s recovering 2024-08-17T18:24:36.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:35 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:24:36.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:35 smithi136 ceph-mon[115078]: pgmap v3461: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 541 op/s; 10436/115170 objects degraded (9.061%); 15845/115170 objects misplaced (13.758%); 4.5 MiB/s, 93 objects/s recovering 2024-08-17T18:24:36.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:35 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:24:36.205 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:36.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:36.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:36.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:35 smithi165 ceph-mon[105517]: pgmap v3461: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 541 op/s; 10436/115170 objects degraded (9.061%); 15845/115170 objects misplaced (13.758%); 4.5 MiB/s, 93 objects/s recovering 2024-08-17T18:24:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:35 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:24:36.307 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:36.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:36.339 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:36.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:36.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:36.471 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:36.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:36.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:36.573 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:36.597 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:36.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:36.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:36.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:36.731 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:36.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:36.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:36.832 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:36.869 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:36.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:36.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:36.970 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:37.003 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:37.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:37.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:37.105 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:37.135 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:37.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:37.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:37.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:24:37.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:24:37.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:24:37.273 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:37.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:37.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:37.375 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:37.405 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:37.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:37.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:37.507 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:37.540 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:37.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:37.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:37.642 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:37.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:37.676 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:37.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:37.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:24:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:24:37] "GET /metrics HTTP/1.1" 200 36721 "" "Prometheus/2.51.0" 2024-08-17T18:24:37.777 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:37.812 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:37.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:37.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:37.907 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:37 smithi165 ceph-mon[105517]: pgmap v3462: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 655 op/s; 10238/108561 objects degraded (9.431%); 14478/108561 objects misplaced (13.336%); 7.1 MiB/s, 135 objects/s recovering 2024-08-17T18:24:37.914 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:37.920 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-17T18:24:37.953 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:37.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:37.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:38.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:38.106 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:38.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:38.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:38.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:37 smithi136 ceph-mon[109861]: pgmap v3462: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 655 op/s; 10238/108561 objects degraded (9.431%); 14478/108561 objects misplaced (13.336%); 7.1 MiB/s, 135 objects/s recovering 2024-08-17T18:24:38.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:37 smithi136 ceph-mon[115078]: pgmap v3462: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 655 op/s; 10238/108561 objects degraded (9.431%); 14478/108561 objects misplaced (13.336%); 7.1 MiB/s, 135 objects/s recovering 2024-08-17T18:24:38.209 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:38.237 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:38.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:38.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:38.339 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:38.374 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:38.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:38.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:38.476 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:38.507 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:38.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:38.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:38.576 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:24:38.609 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:38.660 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:38.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:38.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:38.762 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:38.792 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:38.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:38.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:38.894 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:38.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:38.937 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:38.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:39.038 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:39.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:39.076 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:39.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:39.177 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:39.203 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:39.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:39.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:39.305 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:39.335 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:39.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:39.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:39.437 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:39.470 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:39.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:39.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:39.572 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:39.612 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:39.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:39.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:39.713 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:39.749 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:39.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:39.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:39.830 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:24:39.851 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:39.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:39.916 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:39.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:39.922 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:24:40.018 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:40.046 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:40.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:40.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:40.095 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:24:40.131 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:39 smithi136 ceph-mon[109861]: pgmap v3463: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 577 op/s; 10238/105915 objects degraded (9.666%); 14311/105915 objects misplaced (13.512%); 5.2 MiB/s, 83 objects/s recovering 2024-08-17T18:24:40.131 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:39 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3278511305' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:24:40.131 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:39 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 10238/105915 objects degraded (9.666%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:40.131 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:39 smithi136 ceph-mon[115078]: pgmap v3463: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 577 op/s; 10238/105915 objects degraded (9.666%); 14311/105915 objects misplaced (13.512%); 5.2 MiB/s, 83 objects/s recovering 2024-08-17T18:24:40.131 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:39 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3278511305' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:24:40.132 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:39 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 10238/105915 objects degraded (9.666%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:40.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:40.172 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:40.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:40.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:40.219 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:39 smithi165 ceph-mon[105517]: pgmap v3463: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 577 op/s; 10238/105915 objects degraded (9.666%); 14311/105915 objects misplaced (13.512%); 5.2 MiB/s, 83 objects/s recovering 2024-08-17T18:24:40.219 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:39 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3278511305' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:24:40.219 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:39 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 10238/105915 objects degraded (9.666%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:40.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:40.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:40.324 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:40.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:40.424 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:40.456 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:40.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:40.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:40.558 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:40.595 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:40.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:40.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:40.696 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:40.730 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:40.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:40.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:40.832 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:40.858 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:40.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:40.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:40.960 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:40.989 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:40.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:40.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:41.063 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:40 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:24:41.063 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:40 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3278511305' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:24:41.063 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:40 smithi165 ceph-mon[105517]: osdmap e1749: 8 total, 8 up, 8 in 2024-08-17T18:24:41.063 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:40 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:24:41.063 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:40 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:24:41.091 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:41.131 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:41.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:41.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:41.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:40 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:24:41.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:40 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3278511305' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:24:41.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:40 smithi136 ceph-mon[109861]: osdmap e1749: 8 total, 8 up, 8 in 2024-08-17T18:24:41.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:40 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:24:41.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:40 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:24:41.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:40 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:24:41.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:40 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3278511305' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:24:41.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:40 smithi136 ceph-mon[115078]: osdmap e1749: 8 total, 8 up, 8 in 2024-08-17T18:24:41.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:40 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:24:41.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:40 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:24:41.233 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:41.262 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:41.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:41.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:41.364 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:41.401 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:41.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:41.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:41.503 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:41.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:41.528 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:41.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:41.629 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:41.658 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:41.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:41.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:41.760 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:41.790 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:41.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:41.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:41.892 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:41.922 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:41.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:41.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:42.024 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:42.051 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:42.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:42.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:42.098 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:41 smithi136 ceph-mon[115078]: pgmap v3465: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 593 op/s; 10125/101790 objects degraded (9.947%); 13378/101790 objects misplaced (13.143%); 6.5 MiB/s, 104 objects/s recovering 2024-08-17T18:24:42.098 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:41 smithi136 ceph-mon[109861]: pgmap v3465: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 593 op/s; 10125/101790 objects degraded (9.947%); 13378/101790 objects misplaced (13.143%); 6.5 MiB/s, 104 objects/s recovering 2024-08-17T18:24:42.152 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:42.187 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:42.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:42.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:42.225 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:41 smithi165 ceph-mon[105517]: pgmap v3465: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 593 op/s; 10125/101790 objects degraded (9.947%); 13378/101790 objects misplaced (13.143%); 6.5 MiB/s, 104 objects/s recovering 2024-08-17T18:24:42.289 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:42.319 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:42.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:42.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:42.421 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:42.450 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:42.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:42.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:42.552 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:42.581 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:42.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:42.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:42.683 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:42.712 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:42.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:42.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:42.814 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:42.847 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:42.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:42.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:42.948 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:42.977 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:42.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:42.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:43.079 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:43.109 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:43.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:43.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:43.211 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:43.234 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:43.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:43.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:43.336 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:43.359 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:43.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:43.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:43.461 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:43.463 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:24:43.479 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:24:43.479 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T18:24:43.479 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-17T18:24:43.479 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:24:43.484 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:43.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:43.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:43.586 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:43.613 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:43.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:43.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:43.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:43.740 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:43.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:43.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:43.842 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:43.869 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:43.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:43.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:43.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:43.997 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:43.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:43.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:44.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:44.123 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:44.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:44.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:44.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:43 smithi136 ceph-mon[109861]: pgmap v3466: 99 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 2.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 589 op/s; 9949/99672 objects degraded (9.982%); 13202/99672 objects misplaced (13.245%); 6.6 MiB/s, 105 objects/s recovering 2024-08-17T18:24:44.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:43 smithi136 ceph-mon[115078]: pgmap v3466: 99 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 2.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 589 op/s; 9949/99672 objects degraded (9.982%); 13202/99672 objects misplaced (13.245%); 6.6 MiB/s, 105 objects/s recovering 2024-08-17T18:24:44.226 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:44.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:43 smithi165 ceph-mon[105517]: pgmap v3466: 99 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 2.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 589 op/s; 9949/99672 objects degraded (9.982%); 13202/99672 objects misplaced (13.245%); 6.6 MiB/s, 105 objects/s recovering 2024-08-17T18:24:44.263 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:44.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:44.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:44.365 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:44.392 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:44.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:44.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:44.494 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:44.519 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:44.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:44.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:44.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:44.645 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:44.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:44.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:44.747 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:44.775 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:44.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:44.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:44.877 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:44.909 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:44.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:44.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:44.976 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-17T18:24:45.011 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:45.045 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:45.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:45.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:45.147 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:45.173 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:45.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:45.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:45.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:44 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 9949/99672 objects degraded (9.982%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:45.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:44 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 9949/99672 objects degraded (9.982%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:45.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:44 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 9949/99672 objects degraded (9.982%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:45.270 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:24:45.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:45.298 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:45.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:45.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:45.317 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:24:45.399 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:45.424 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:45.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:45.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:45.526 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:45.550 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:45.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:45.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:45.652 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:45.675 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:45.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:45.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:45.776 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:45.801 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:45.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:45.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:45.903 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:45.928 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:45.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:45.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:45.946 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.3 to 1 (10000) 2024-08-17T18:24:46.030 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:46.054 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:46.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:46.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:46.156 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:46.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:46.187 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:46.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:46.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:45 smithi136 ceph-mon[109861]: pgmap v3467: 99 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 2.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 582 op/s; 9949/97677 objects degraded (10.186%); 13041/97677 objects misplaced (13.351%); 6.6 MiB/s, 105 objects/s recovering 2024-08-17T18:24:46.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:45 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3354287582' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T18:24:46.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:45 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T18:24:46.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:45 smithi136 ceph-mon[115078]: pgmap v3467: 99 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 2.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 582 op/s; 9949/97677 objects degraded (10.186%); 13041/97677 objects misplaced (13.351%); 6.6 MiB/s, 105 objects/s recovering 2024-08-17T18:24:46.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:45 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3354287582' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T18:24:46.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:45 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T18:24:46.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:45 smithi165 ceph-mon[105517]: pgmap v3467: 99 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 2.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 582 op/s; 9949/97677 objects degraded (10.186%); 13041/97677 objects misplaced (13.351%); 6.6 MiB/s, 105 objects/s recovering 2024-08-17T18:24:46.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:45 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3354287582' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T18:24:46.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:45 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T18:24:46.288 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:46.317 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:46.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:46.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:46.419 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:46.441 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:46.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:46.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:46.543 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:46.567 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:46.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:46.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:46.669 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:46.697 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:46.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:46.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:46.799 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:46.824 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:46.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:46.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:46.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:46.956 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:46.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:46.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:47.058 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:47.080 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:47.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:47.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:47.182 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:47.208 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:47.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:47.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:46 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T18:24:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:46 smithi165 ceph-mon[105517]: osdmap e1750: 8 total, 8 up, 8 in 2024-08-17T18:24:47.261 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:46 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T18:24:47.261 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:46 smithi136 ceph-mon[109861]: osdmap e1750: 8 total, 8 up, 8 in 2024-08-17T18:24:47.262 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:46 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T18:24:47.262 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:46 smithi136 ceph-mon[115078]: osdmap e1750: 8 total, 8 up, 8 in 2024-08-17T18:24:47.310 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:47.342 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:47.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:47.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:47.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:47.466 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:47.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:47.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:47.568 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:47.591 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:24:47.595 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:47.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:47.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:47.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:24:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:24:47] "GET /metrics HTTP/1.1" 200 36721 "" "Prometheus/2.51.0" 2024-08-17T18:24:47.697 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:47.719 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:47.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:47.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:47.820 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:47.842 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:47.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:47.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:47.944 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:47.969 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:47.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:47.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:48.071 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:48.098 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:48.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:48.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:48.167 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:47 smithi136 ceph-mon[109861]: pgmap v3469: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 541 op/s; 9676/92907 objects degraded (10.415%); 12126/92907 objects misplaced (13.052%); 7.6 MiB/s, 158 objects/s recovering 2024-08-17T18:24:48.167 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:47 smithi136 ceph-mon[115078]: pgmap v3469: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 541 op/s; 9676/92907 objects degraded (10.415%); 12126/92907 objects misplaced (13.052%); 7.6 MiB/s, 158 objects/s recovering 2024-08-17T18:24:48.190 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 1 1 2024-08-17T18:24:48.200 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:48.260 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:48.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:47 smithi165 ceph-mon[105517]: pgmap v3469: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 541 op/s; 9676/92907 objects degraded (10.415%); 12126/92907 objects misplaced (13.052%); 7.6 MiB/s, 158 objects/s recovering 2024-08-17T18:24:48.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:48.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:48.362 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:48.387 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:48.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:48.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:48.489 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:48.510 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:48.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:48.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:48.612 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:48.638 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:48.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:48.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:48.740 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:48.767 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:48.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:48.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:48.868 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:48.893 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:48.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:48.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:48.995 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:49.024 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:49.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:49.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:49.126 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:49.158 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:49.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:49.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:49.260 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:49.283 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:49.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:49.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:49.385 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:49.412 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:49.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:49.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:49.451 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:24:49.451 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:24:49.478 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:24:49.478 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:24:49.507 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:24:49.507 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:24:49.514 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:49.535 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:24:49.535 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:24:49.541 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:49.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:49.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:49.554 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:24:49.555 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:24:49.571 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:24:49.571 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:24:49.590 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:24:49.590 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:24:49.606 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:24:49.607 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:24:49.643 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:49.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:49.668 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:49.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:49.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:49.791 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:49.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:49.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:49.893 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:49.918 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:49.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:49.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:50.019 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:50.045 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:50.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:50.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:50.147 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:50.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:50.176 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:50.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:50.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:49 smithi165 ceph-mon[105517]: pgmap v3470: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 556 op/s; 9676/91404 objects degraded (10.586%); 12020/91404 objects misplaced (13.150%); 7.0 MiB/s, 145 objects/s recovering 2024-08-17T18:24:50.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:49 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/197313265' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:24:50.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:49 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 9676/91404 objects degraded (10.586%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:50.278 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:50.318 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:50.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:50.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:50.377 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:49 smithi136 ceph-mon[109861]: pgmap v3470: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 556 op/s; 9676/91404 objects degraded (10.586%); 12020/91404 objects misplaced (13.150%); 7.0 MiB/s, 145 objects/s recovering 2024-08-17T18:24:50.377 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:49 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/197313265' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:24:50.378 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:49 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 9676/91404 objects degraded (10.586%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:50.378 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:49 smithi136 ceph-mon[115078]: pgmap v3470: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 556 op/s; 9676/91404 objects degraded (10.586%); 12020/91404 objects misplaced (13.150%); 7.0 MiB/s, 145 objects/s recovering 2024-08-17T18:24:50.378 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:49 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/197313265' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:24:50.378 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:49 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 9676/91404 objects degraded (10.586%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:50.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:50.446 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:50.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:50.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:50.548 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:50.577 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:50.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:50.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:50.679 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:50.705 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:50.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:50.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:50.806 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:50.830 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:50.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:50.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:50.932 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:50.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:50.962 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:50.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:51.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:51.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:51.093 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:51.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:51.194 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:51.220 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:51.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:51.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:51.322 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:51.346 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:51.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:51.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:51.447 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:51.472 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:51.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:51.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:51.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:51.599 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:51.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:51.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:51.701 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:51.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:51.726 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:51.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:51.827 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:51.851 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:51.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:51.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:51.953 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:51.978 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:51.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:51.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:52.080 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:52.107 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:52.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:52.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:52.209 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:52.237 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:52.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:52.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:51 smithi165 ceph-mon[105517]: pgmap v3471: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 426 op/s; 9570/88989 objects degraded (10.754%); 11609/88989 objects misplaced (13.045%); 4.5 MiB/s, 191 objects/s recovering 2024-08-17T18:24:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:24:52.340 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:52.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:52.374 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:52.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:52.416 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:51 smithi136 ceph-mon[109861]: pgmap v3471: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 426 op/s; 9570/88989 objects degraded (10.754%); 11609/88989 objects misplaced (13.045%); 4.5 MiB/s, 191 objects/s recovering 2024-08-17T18:24:52.417 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:24:52.417 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:51 smithi136 ceph-mon[115078]: pgmap v3471: 99 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 426 op/s; 9570/88989 objects degraded (10.754%); 11609/88989 objects misplaced (13.045%); 4.5 MiB/s, 191 objects/s recovering 2024-08-17T18:24:52.417 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:24:52.476 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:52.500 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:52.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:52.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:52.602 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:52.626 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:52.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:52.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:52.728 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:52.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:52.750 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:52.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:52.851 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:52.873 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:52.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:52.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:52.976 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:52.999 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:53.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:53.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:53.101 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:53.128 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:53.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:53.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:53.230 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:53.262 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:53.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:53.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:53.364 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:53.387 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:53.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:53.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:53.489 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:53.513 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:53.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:53.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:53.615 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:53.637 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:53.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:53.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:53.739 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:53.766 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:53.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:53.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:53.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:53.900 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:53.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:53.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:54.002 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:54.027 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:54.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:54.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:54.129 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:54.152 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:54.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:54.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:54.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:53 smithi136 ceph-mon[109861]: pgmap v3472: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 397 op/s; 9450/87741 objects degraded (10.770%); 11489/87741 objects misplaced (13.094%); 4.2 MiB/s, 186 objects/s recovering 2024-08-17T18:24:54.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:53 smithi136 ceph-mon[115078]: pgmap v3472: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 397 op/s; 9450/87741 objects degraded (10.770%); 11489/87741 objects misplaced (13.094%); 4.2 MiB/s, 186 objects/s recovering 2024-08-17T18:24:54.254 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:54.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:53 smithi165 ceph-mon[105517]: pgmap v3472: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 397 op/s; 9450/87741 objects degraded (10.770%); 11489/87741 objects misplaced (13.094%); 4.2 MiB/s, 186 objects/s recovering 2024-08-17T18:24:54.283 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:54.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:54.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:54.385 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:54.409 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:54.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:54.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:54.511 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:54.537 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:54.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:54.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:54.639 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:54.664 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:54.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:54.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:54.766 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:54.792 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:54.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:54.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:54.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:54.917 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:54.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:54.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:54.966 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:24:54.970 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:24:55.019 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:55.041 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:55.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:55.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:55.143 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:55.169 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:55.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:55.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:55.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:54 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1461773970' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:24:55.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:54 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 9450/87741 objects degraded (10.770%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:55.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:54 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1461773970' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:24:55.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:54 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 9450/87741 objects degraded (10.770%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:55.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:54 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1461773970' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:24:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:54 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 9450/87741 objects degraded (10.770%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:24:55.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:55.295 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:55.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:55.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:55.396 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:55.421 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:55.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:55.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:55.523 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:55.546 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:55.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:55.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:55.648 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:55.673 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:55.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:55.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:55.775 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:55.798 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:55.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:55.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:55.900 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:55.931 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:55.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:55.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:56.033 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:56.055 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:56.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:56.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:56.157 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:56.178 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:56.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:56.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:56.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:55 smithi165 ceph-mon[105517]: pgmap v3473: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 376 op/s; 9450/86370 objects degraded (10.941%); 11384/86370 objects misplaced (13.181%); 4.2 MiB/s, 186 objects/s recovering 2024-08-17T18:24:56.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:55 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:24:56.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:55 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1461773970' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:24:56.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:55 smithi165 ceph-mon[105517]: osdmap e1751: 8 total, 8 up, 8 in 2024-08-17T18:24:56.280 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:56.304 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:56.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:56.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:56.405 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:56.428 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:56.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:56.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:56.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:55 smithi136 ceph-mon[115078]: pgmap v3473: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 376 op/s; 9450/86370 objects degraded (10.941%); 11384/86370 objects misplaced (13.181%); 4.2 MiB/s, 186 objects/s recovering 2024-08-17T18:24:56.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:55 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:24:56.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:55 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1461773970' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:24:56.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:55 smithi136 ceph-mon[115078]: osdmap e1751: 8 total, 8 up, 8 in 2024-08-17T18:24:56.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:55 smithi136 ceph-mon[109861]: pgmap v3473: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 5 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 376 op/s; 9450/86370 objects degraded (10.941%); 11384/86370 objects misplaced (13.181%); 4.2 MiB/s, 186 objects/s recovering 2024-08-17T18:24:56.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:55 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:24:56.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:55 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1461773970' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:24:56.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:55 smithi136 ceph-mon[109861]: osdmap e1751: 8 total, 8 up, 8 in 2024-08-17T18:24:56.495 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:24:56.530 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:56.572 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:56.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:56.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:56.674 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:56.697 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:56.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:56.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:56.799 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:56.832 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:56.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:56.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:56.934 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:56.960 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:56.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:56.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:57.062 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:57.092 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:57.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:57.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:57.193 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:57.217 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:57.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:57.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:57.319 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:57.352 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:57.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:57.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:57.454 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:57.480 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:57.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:57.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:57.582 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:57.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:57.605 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:57.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:57.706 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:57.729 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:57.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:57.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:57.830 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:57.857 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:57.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:57.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:57.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:24:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:24:57] "GET /metrics HTTP/1.1" 200 36719 "" "Prometheus/2.51.0" 2024-08-17T18:24:57.959 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:57.989 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:57.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:57.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:58.091 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:58.120 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:58.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:58.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:58.222 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:58.245 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:58.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:58.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:58.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:57 smithi165 ceph-mon[105517]: pgmap v3475: 99 pgs: 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 1.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 341 op/s; 9240/82653 objects degraded (11.179%); 12595/82653 objects misplaced (15.238%); 2.7 MiB/s, 201 objects/s recovering 2024-08-17T18:24:58.347 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:24:58.371 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:58.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:58.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:24:58.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:57 smithi136 ceph-mon[109861]: pgmap v3475: 99 pgs: 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 1.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 341 op/s; 9240/82653 objects degraded (11.179%); 12595/82653 objects misplaced (15.238%); 2.7 MiB/s, 201 objects/s recovering 2024-08-17T18:24:58.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:57 smithi136 ceph-mon[115078]: pgmap v3475: 99 pgs: 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 1.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 341 op/s; 9240/82653 objects degraded (11.179%); 12595/82653 objects misplaced (15.238%); 2.7 MiB/s, 201 objects/s recovering 2024-08-17T18:24:58.473 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:58.502 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:58.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:58.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:58.604 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:58.631 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:58.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:58.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:58.732 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:24:58.769 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:58.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:58.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:24:58.871 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:24:58.900 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:58.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:58.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:24:59.001 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:59.027 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:59.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:59.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:24:59.129 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:24:59.156 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:59.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:59.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:24:59.259 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:59.293 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:59.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:59.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:59.395 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:59.416 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:59.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:59.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:59.518 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:24:59.545 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:59.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:59.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:24:59.646 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:24:59.668 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:59.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:59.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:24:59.770 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:24:59.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:59.801 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:24:59.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:24:59.902 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:24:59.929 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:24:59.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:24:59.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:00.031 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:00.057 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:00.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:00.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:00.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:00.187 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:00.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:00.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:00.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:59 smithi165 ceph-mon[105517]: pgmap v3476: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 1.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 349 op/s; 8607/80916 objects degraded (10.637%); 12459/80916 objects misplaced (15.397%); 5.2 MiB/s, 241 objects/s recovering 2024-08-17T18:25:00.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:24:59 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 8607/80916 objects degraded (10.637%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:00.289 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:00.312 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:00.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:00.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:00.413 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:00.445 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:00.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:59 smithi136 ceph-mon[109861]: pgmap v3476: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 1.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 349 op/s; 8607/80916 objects degraded (10.637%); 12459/80916 objects misplaced (15.397%); 5.2 MiB/s, 241 objects/s recovering 2024-08-17T18:25:00.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:24:59 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 8607/80916 objects degraded (10.637%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:00.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:59 smithi136 ceph-mon[115078]: pgmap v3476: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 1.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 349 op/s; 8607/80916 objects degraded (10.637%); 12459/80916 objects misplaced (15.397%); 5.2 MiB/s, 241 objects/s recovering 2024-08-17T18:25:00.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:24:59 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 8607/80916 objects degraded (10.637%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:00.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:00.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:00.548 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:00.574 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:00.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:00.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:00.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:00.700 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:00.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:00.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:00.803 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:00.828 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:00.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:00.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:00.929 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:00.954 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:00.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:00.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:01.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:01.077 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:01.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:01.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:01.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:01.201 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:01.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:01.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:01.303 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:01.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:01.331 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:01.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:01.432 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:01.456 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:01.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:01.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:01.558 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:01.580 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:01.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:01.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:01.682 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:01.713 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:01.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:01.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:01.815 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:01.837 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:01.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:01.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:01.939 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:01.963 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:01.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:01.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:02.065 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:02.088 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:02.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:02.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:02.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:02.217 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:02.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:02.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:02.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:01 smithi165 ceph-mon[105517]: pgmap v3477: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 368 op/s; 8491/77931 objects degraded (10.896%); 11863/77931 objects misplaced (15.222%); 5.3 MiB/s, 152 objects/s recovering 2024-08-17T18:25:02.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:02.344 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:02.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:02.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:02.446 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:02.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:01 smithi136 ceph-mon[109861]: pgmap v3477: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 368 op/s; 8491/77931 objects degraded (10.896%); 11863/77931 objects misplaced (15.222%); 5.3 MiB/s, 152 objects/s recovering 2024-08-17T18:25:02.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:01 smithi136 ceph-mon[115078]: pgmap v3477: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 368 op/s; 8491/77931 objects degraded (10.896%); 11863/77931 objects misplaced (15.222%); 5.3 MiB/s, 152 objects/s recovering 2024-08-17T18:25:02.477 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:02.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:02.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:02.579 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:02.605 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:02.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:02.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:02.708 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:02.734 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:02.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:02.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:02.836 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:02.860 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:02.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:02.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:02.962 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:02.986 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:02.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:02.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:03.088 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:03.115 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:03.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:03.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:03.217 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:03.247 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:03.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:03.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:03.349 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:03.373 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:03.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:03.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:03.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:03.500 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:03.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:03.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:03.602 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:03.626 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:03.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:03.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:03.728 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:03.762 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:03.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:03.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:03.864 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:03.888 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:03.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:03.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:03.990 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:04.019 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:04.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:04.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:04.121 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:04.144 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:04.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:04.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:04.247 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:04.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:03 smithi165 ceph-mon[105517]: pgmap v3478: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 377 op/s; 8349/76416 objects degraded (10.926%); 11721/76416 objects misplaced (15.338%); 5.5 MiB/s, 155 objects/s recovering 2024-08-17T18:25:04.274 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:04.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:04.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:04.377 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:04.403 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:04.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:04.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:04.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:03 smithi136 ceph-mon[109861]: pgmap v3478: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 377 op/s; 8349/76416 objects degraded (10.926%); 11721/76416 objects misplaced (15.338%); 5.5 MiB/s, 155 objects/s recovering 2024-08-17T18:25:04.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:03 smithi136 ceph-mon[115078]: pgmap v3478: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 377 op/s; 8349/76416 objects degraded (10.926%); 11721/76416 objects misplaced (15.338%); 5.5 MiB/s, 155 objects/s recovering 2024-08-17T18:25:04.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:04.527 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:04.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:04.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:04.629 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:04.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:04.653 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:04.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:04.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:04.778 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:04.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:04.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:04.880 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:04.903 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:04.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:04.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:05.005 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:05.027 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:05.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:05.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:05.129 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:05.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:05.157 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:05.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:05.258 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:04 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 8349/76416 objects degraded (10.926%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:05.288 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:05.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:05.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:05.390 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:05.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:05.430 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:05.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:05.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:04 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 8349/76416 objects degraded (10.926%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:05.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:04 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 8349/76416 objects degraded (10.926%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:05.531 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:05.559 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:05.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:05.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:05.661 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:05.692 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:05.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:05.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:05.794 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:05.828 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:05.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:05.930 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:05.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:05.959 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:05.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:06.060 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:06.085 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:06.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:06.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:06.187 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:06.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:06.209 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:06.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:06.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:05 smithi165 ceph-mon[105517]: pgmap v3479: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 382 op/s; 7923/74895 objects degraded (10.579%); 11603/74895 objects misplaced (15.492%); 8.1 MiB/s, 197 objects/s recovering 2024-08-17T18:25:06.310 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:06.335 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:06.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:06.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:06.437 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:06.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:05 smithi136 ceph-mon[109861]: pgmap v3479: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 382 op/s; 7923/74895 objects degraded (10.579%); 11603/74895 objects misplaced (15.492%); 8.1 MiB/s, 197 objects/s recovering 2024-08-17T18:25:06.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:05 smithi136 ceph-mon[115078]: pgmap v3479: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 382 op/s; 7923/74895 objects degraded (10.579%); 11603/74895 objects misplaced (15.492%); 8.1 MiB/s, 197 objects/s recovering 2024-08-17T18:25:06.466 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:06.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:06.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:06.568 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:06.591 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:06.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:06.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:06.692 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:06.716 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:06.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:06.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:06.818 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:06.850 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:06.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:06.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:06.952 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:06.973 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:06.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:06.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:07.075 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:07.107 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:07.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:07.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:07.210 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:07.231 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:07.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:07.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:07.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:25:07.333 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:07.363 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:07.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:07.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:07.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:25:07.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:25:07.465 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:07.488 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:07.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:07.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:07.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:07.613 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:07.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:07.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:07.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:07.726 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:25:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:25:07] "GET /metrics HTTP/1.1" 200 36722 "" "Prometheus/2.51.0" 2024-08-17T18:25:07.738 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:07.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:07.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:07.840 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:07.862 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:07.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:07.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:07.964 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:07.991 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:07.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:07.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:08.093 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:08.118 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:08.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:08.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:08.220 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:08.250 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:08.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:08.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:08.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:07 smithi165 ceph-mon[105517]: pgmap v3480: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 435 op/s; 7690/71145 objects degraded (10.809%); 10901/71145 objects misplaced (15.322%); 8.2 MiB/s, 189 objects/s recovering 2024-08-17T18:25:08.352 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:08.372 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:25:08.372 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:25:08.372 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:07 smithi136 ceph-mon[109861]: pgmap v3480: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 435 op/s; 7690/71145 objects degraded (10.809%); 10901/71145 objects misplaced (15.322%); 8.2 MiB/s, 189 objects/s recovering 2024-08-17T18:25:08.373 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:07 smithi136 ceph-mon[115078]: pgmap v3480: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 435 op/s; 7690/71145 objects degraded (10.809%); 10901/71145 objects misplaced (15.322%); 8.2 MiB/s, 189 objects/s recovering 2024-08-17T18:25:08.374 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:08.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:08.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:08.476 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:08.502 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:08.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:08.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:08.604 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:08.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:08.634 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:08.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:08.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:08.762 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:08.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:08.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:08.864 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:08.894 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:08.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:08.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:09.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:09.024 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:09.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:09.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:09.126 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:09.147 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-17T18:25:09.158 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:09.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:09.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:09.261 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:09.286 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:09.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:09.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:09.388 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:09.417 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:09.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:09.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:09.519 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:09.555 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:09.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:09.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:09.658 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:09.681 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:09.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:09.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:09.783 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:09.805 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:09.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:09.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:09.907 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:09.931 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:09.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:09.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:10.033 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:10.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:10.059 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:10.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:10.161 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:10.184 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:10.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:10.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:10.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:09 smithi165 ceph-mon[105517]: pgmap v3481: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 368 op/s; 7128/69399 objects degraded (10.271%); 10778/69399 objects misplaced (15.530%); 9.8 MiB/s, 157 objects/s recovering 2024-08-17T18:25:10.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:09 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 7128/69399 objects degraded (10.271%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:10.287 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:10.314 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:10.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:10.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:10.416 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:10.444 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:10.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:10.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:10.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:09 smithi136 ceph-mon[109861]: pgmap v3481: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 368 op/s; 7128/69399 objects degraded (10.271%); 10778/69399 objects misplaced (15.530%); 9.8 MiB/s, 157 objects/s recovering 2024-08-17T18:25:10.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:09 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 7128/69399 objects degraded (10.271%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:10.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:09 smithi136 ceph-mon[115078]: pgmap v3481: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 368 op/s; 7128/69399 objects degraded (10.271%); 10778/69399 objects misplaced (15.530%); 9.8 MiB/s, 157 objects/s recovering 2024-08-17T18:25:10.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:09 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 7128/69399 objects degraded (10.271%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:10.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:10.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:10.587 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:10.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:10.689 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:10.718 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:10.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:10.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:10.820 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:10.845 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:10.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:10.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:10.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:10.970 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:10.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:10.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:11.022 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:25:11.024 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:25:11.072 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:11.114 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:11.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:11.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:11.216 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:11.240 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:11.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:11.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:11.342 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:11.368 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:11.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:11.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:11.470 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:11.501 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:11.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:11.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:11.603 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:11.627 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:11.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:11.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:11.729 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:11.755 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:11.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:11.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:11.857 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:11.880 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:11.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:11.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:11.982 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:12.011 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:12.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:12.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:12.113 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:12.141 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:12.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:12.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:12.243 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:12.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:11 smithi165 ceph-mon[105517]: pgmap v3482: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 399 op/s; 7002/66516 objects degraded (10.527%); 10184/66516 objects misplaced (15.311%); 8.4 MiB/s, 133 objects/s recovering 2024-08-17T18:25:12.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:11 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:25:12.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:11 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:25:12.269 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:12.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:12.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:12.371 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:12.396 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:12.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:12.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:12.438 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:11 smithi136 ceph-mon[109861]: pgmap v3482: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 399 op/s; 7002/66516 objects degraded (10.527%); 10184/66516 objects misplaced (15.311%); 8.4 MiB/s, 133 objects/s recovering 2024-08-17T18:25:12.438 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:25:12.438 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:11 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:25:12.438 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:11 smithi136 ceph-mon[115078]: pgmap v3482: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 399 op/s; 7002/66516 objects degraded (10.527%); 10184/66516 objects misplaced (15.311%); 8.4 MiB/s, 133 objects/s recovering 2024-08-17T18:25:12.439 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:11 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:25:12.439 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:11 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:25:12.498 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:12.524 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:12.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:12.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:12.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:25:12.626 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:12.653 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:12.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:12.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:12.755 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:12.788 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:12.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:12.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:12.890 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:12.913 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:12.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:12.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:13.001 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.1 to 1 (10000) 2024-08-17T18:25:13.015 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:13.038 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:13.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:13.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:13.140 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:13.164 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:13.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:13.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:13.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:12 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/348145693' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T18:25:13.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:12 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:25:13.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:12 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:25:13.265 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:13.288 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:13.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:13.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:13.347 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:12 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/348145693' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T18:25:13.347 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:25:13.347 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:25:13.347 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:12 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/348145693' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T18:25:13.347 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:25:13.347 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:25:13.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:13.412 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:13.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:13.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:13.514 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:13.535 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:13.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:13.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:13.637 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:13.658 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:13.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:13.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:13.760 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:13.782 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:13.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:13.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:13.884 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:13.906 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:13.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:13.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:13.980 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-08-17T18:25:13.980 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T18:25:14.007 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:14.056 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:14.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:14.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:14.158 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:14.186 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:14.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:14.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:14.225 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:14 smithi136 ceph-mon[115078]: pgmap v3483: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 357 op/s; 6878/65058 objects degraded (10.572%); 10061/65058 objects misplaced (15.465%); 8.4 MiB/s, 134 objects/s recovering 2024-08-17T18:25:14.225 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:14 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/348145693' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T18:25:14.226 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:14 smithi136 ceph-mon[115078]: osdmap e1752: 8 total, 8 up, 8 in 2024-08-17T18:25:14.226 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:14 smithi136 ceph-mon[109861]: pgmap v3483: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 357 op/s; 6878/65058 objects degraded (10.572%); 10061/65058 objects misplaced (15.465%); 8.4 MiB/s, 134 objects/s recovering 2024-08-17T18:25:14.226 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:14 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/348145693' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T18:25:14.226 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:14 smithi136 ceph-mon[109861]: osdmap e1752: 8 total, 8 up, 8 in 2024-08-17T18:25:14.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:14 smithi165 ceph-mon[105517]: pgmap v3483: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 6 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 357 op/s; 6878/65058 objects degraded (10.572%); 10061/65058 objects misplaced (15.465%); 8.4 MiB/s, 134 objects/s recovering 2024-08-17T18:25:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:14 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/348145693' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T18:25:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:14 smithi165 ceph-mon[105517]: osdmap e1752: 8 total, 8 up, 8 in 2024-08-17T18:25:14.289 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:14.314 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:14.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:14.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:14.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:14.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:14.438 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:14.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:14.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:14.560 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:14.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:14.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:14.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:14.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:14.685 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:14.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:14.786 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:14.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:14.811 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:14.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:14.912 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:14.955 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:14.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:14.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:15.059 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:15.088 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:15.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:15.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:15.190 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:15.214 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:15.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:15.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:15.316 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:15.339 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:15.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:15.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:15.441 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:15.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:15 smithi136 ceph-mon[109861]: 58.9 starting backfill to osd.1 from (0'0,1714'5976] MAX to 1751'13285 2024-08-17T18:25:15.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:15 smithi136 ceph-mon[109861]: osdmap e1753: 8 total, 8 up, 8 in 2024-08-17T18:25:15.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:15 smithi136 ceph-mon[109861]: pgmap v3486: 99 pgs: 2 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 309 op/s; 6330/63720 objects degraded (9.934%); 8657/63720 objects misplaced (13.586%); 9.1 MiB/s, 198 objects/s recovering 2024-08-17T18:25:15.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:15 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 6878/65058 objects degraded (10.572%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:15.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:15 smithi136 ceph-mon[109861]: osdmap e1754: 8 total, 8 up, 8 in 2024-08-17T18:25:15.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:15 smithi136 ceph-mon[115078]: 58.9 starting backfill to osd.1 from (0'0,1714'5976] MAX to 1751'13285 2024-08-17T18:25:15.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:15 smithi136 ceph-mon[115078]: osdmap e1753: 8 total, 8 up, 8 in 2024-08-17T18:25:15.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:15 smithi136 ceph-mon[115078]: pgmap v3486: 99 pgs: 2 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 309 op/s; 6330/63720 objects degraded (9.934%); 8657/63720 objects misplaced (13.586%); 9.1 MiB/s, 198 objects/s recovering 2024-08-17T18:25:15.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:15 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 6878/65058 objects degraded (10.572%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:15.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:15 smithi136 ceph-mon[115078]: osdmap e1754: 8 total, 8 up, 8 in 2024-08-17T18:25:15.462 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:15.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:15.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:15 smithi165 ceph-mon[105517]: 58.9 starting backfill to osd.1 from (0'0,1714'5976] MAX to 1751'13285 2024-08-17T18:25:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:15 smithi165 ceph-mon[105517]: osdmap e1753: 8 total, 8 up, 8 in 2024-08-17T18:25:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:15 smithi165 ceph-mon[105517]: pgmap v3486: 99 pgs: 2 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 309 op/s; 6330/63720 objects degraded (9.934%); 8657/63720 objects misplaced (13.586%); 9.1 MiB/s, 198 objects/s recovering 2024-08-17T18:25:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:15 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 6878/65058 objects degraded (10.572%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:15.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:15 smithi165 ceph-mon[105517]: osdmap e1754: 8 total, 8 up, 8 in 2024-08-17T18:25:15.511 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 18:25:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=cleanup t=2024-08-17T18:25:15.059357128Z level=info msg="Completed cleanup jobs" duration=21.629379ms 2024-08-17T18:25:15.512 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 18:25:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=plugins.update.checker t=2024-08-17T18:25:15.185875306Z level=info msg="Update check succeeded" duration=63.385553ms 2024-08-17T18:25:15.512 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 18:25:15 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=grafana.update.checker t=2024-08-17T18:25:15.19999051Z level=info msg="Update check succeeded" duration=69.279848ms 2024-08-17T18:25:15.564 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:15.585 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:15.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:15.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:15.687 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:15.711 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:15.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:15.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:15.812 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:15.843 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:15.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:15.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:15.945 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:15.980 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:15.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:15.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:16.059 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:25:16.082 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:16.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:16.109 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:16.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:16.210 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:16.245 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:16.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:16.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:16.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:16.374 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:16.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:16.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:16.476 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:16.500 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:16.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:16.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:16.575 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:25:16.602 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:16.606 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:25:16.606 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:25:16.632 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:25:16.632 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:25:16.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:16.633 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:16.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:16.659 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:25:16.659 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:25:16.687 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:25:16.688 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:25:16.713 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:25:16.713 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:25:16.734 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:16.740 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:25:16.741 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:25:16.762 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:16.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:16.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:16.767 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:25:16.767 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:25:16.797 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:25:16.798 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:25:16.864 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:16.866 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:16 smithi136 ceph-mon[109861]: osdmap e1755: 8 total, 8 up, 8 in 2024-08-17T18:25:16.866 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:16 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2875663737' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:25:16.866 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:16 smithi136 ceph-mon[115078]: osdmap e1755: 8 total, 8 up, 8 in 2024-08-17T18:25:16.866 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:16 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2875663737' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:25:16.889 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:16.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:16.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:16.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:17.015 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:17.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:17.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:17.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:17.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:17.145 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:17.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:17.246 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:17.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:16 smithi165 ceph-mon[105517]: osdmap e1755: 8 total, 8 up, 8 in 2024-08-17T18:25:17.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:16 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2875663737' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:25:17.282 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:17.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:17.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:17.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:17.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:17.415 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:17.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:17.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:17.545 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:17.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:17.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:17.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:17.676 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:17.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:17.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:17.778 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:17.807 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:17.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:17.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:17.817 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:25:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:25:17] "GET /metrics HTTP/1.1" 200 36722 "" "Prometheus/2.51.0" 2024-08-17T18:25:17.909 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:17.941 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:17.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:17.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:18.043 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:18.070 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:18.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:18.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:18.114 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:17 smithi136 ceph-mon[109861]: pgmap v3489: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 336 op/s; 6052/61014 objects degraded (9.919%); 7525/61014 objects misplaced (12.333%); 14 MiB/s, 335 objects/s recovering 2024-08-17T18:25:18.114 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:17 smithi136 ceph-mon[115078]: pgmap v3489: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 336 op/s; 6052/61014 objects degraded (9.919%); 7525/61014 objects misplaced (12.333%); 14 MiB/s, 335 objects/s recovering 2024-08-17T18:25:18.172 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:18.202 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:18.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:18.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:18.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:17 smithi165 ceph-mon[105517]: pgmap v3489: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 336 op/s; 6052/61014 objects degraded (9.919%); 7525/61014 objects misplaced (12.333%); 14 MiB/s, 335 objects/s recovering 2024-08-17T18:25:18.304 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:18.332 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:18.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:18.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:18.434 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:18.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:18.469 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:18.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:18.570 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:18.595 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:18.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:18.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:18.698 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:18.725 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:18.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:18.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:18.827 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:18.859 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:18.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:18.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:18.961 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:18.992 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:18.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:18.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:19.093 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:19.116 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:19.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:19.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:19.218 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:19.250 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:19.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:19.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:19.352 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:19.383 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:19.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:19.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:19.485 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:19.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:19.515 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:19.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:19.616 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:19.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:19.640 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:19.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:19.741 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:19.766 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:19.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:19.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:19.868 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:19.900 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:19.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:19.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:19.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:19 smithi136 ceph-mon[109861]: pgmap v3490: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 1.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 289 op/s; 5423/60171 objects degraded (9.013%); 7525/60171 objects misplaced (12.506%); 10 MiB/s, 351 objects/s recovering 2024-08-17T18:25:19.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:19 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 5423/60171 objects degraded (9.013%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:19.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:19 smithi136 ceph-mon[115078]: pgmap v3490: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 1.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 289 op/s; 5423/60171 objects degraded (9.013%); 7525/60171 objects misplaced (12.506%); 10 MiB/s, 351 objects/s recovering 2024-08-17T18:25:19.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:19 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 5423/60171 objects degraded (9.013%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:20.002 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:20.036 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:20.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:20.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:20.138 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:20.157 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:25:20.160 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:20.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:20.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:19 smithi165 ceph-mon[105517]: pgmap v3490: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 1.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 289 op/s; 5423/60171 objects degraded (9.013%); 7525/60171 objects misplaced (12.506%); 10 MiB/s, 351 objects/s recovering 2024-08-17T18:25:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:19 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 5423/60171 objects degraded (9.013%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:20.262 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:20.301 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:20.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:20.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:20.403 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:20.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:20.439 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:20.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:20.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:20.565 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:20.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:20.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:20.667 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:20.699 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:20.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:20.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:20.801 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:20.824 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:20.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:20.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:20.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:20.950 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:20.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:20.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:21.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:21.080 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:21.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:21.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:21.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:21.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:21.204 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:21.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:21.305 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:21.333 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:21.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:21.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:21.435 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:21.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:21.459 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:21.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:21.560 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:21.596 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:21.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:21.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:21.698 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:21.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:21.732 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:21.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:21.833 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:21.865 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:21.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:21.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:21.968 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:21.994 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:21.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:21.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:22.014 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:25:22.014 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-17T18:25:22.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-17T18:25:22.014 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:25:22.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.014 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:69961 58.f 58:f3ef7508:::benchmark_data_smithi136_442124_object69960:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1711)", 2024-08-17T18:25:22.014 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:21:40.753366+0000", 2024-08-17T18:25:22.014 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 220.60160619199999, 2024-08-17T18:25:22.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0239768869999999, 2024-08-17T18:25:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:25:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:25:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:25:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:25:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:25:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:25:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69961 2024-08-17T18:25:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:25:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:25:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.753366+0000", 2024-08-17T18:25:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:25:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.753366+0000", 2024-08-17T18:25:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:25:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.753367+0000", 2024-08-17T18:25:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2440000000000001e-06 2024-08-17T18:25:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:25:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.753390+0000", 2024-08-17T18:25:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3356000000000001e-05 2024-08-17T18:25:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:25:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.753399+0000", 2024-08-17T18:25:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8109999999999992e-06 2024-08-17T18:25:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:25:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.753406+0000", 2024-08-17T18:25:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9619999999999996e-06 2024-08-17T18:25:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.753811+0000", 2024-08-17T18:25:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00040489699999999998 2024-08-17T18:25:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:25:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.753963+0000", 2024-08-17T18:25:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015255800000000001 2024-08-17T18:25:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6", 2024-08-17T18:25:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.754086+0000", 2024-08-17T18:25:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000122519 2024-08-17T18:25:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:25:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.758422+0000", 2024-08-17T18:25:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0043365069999999999 2024-08-17T18:25:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:25:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.758953+0000", 2024-08-17T18:25:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00053071699999999999 2024-08-17T18:25:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.774789+0000", 2024-08-17T18:25:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015835449000000001 2024-08-17T18:25:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.774798+0000", 2024-08-17T18:25:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7070000000000004e-06 2024-08-17T18:25:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.777255+0000", 2024-08-17T18:25:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0024568549999999 2024-08-17T18:25:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:25:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.777343+0000", 2024-08-17T18:25:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7305000000000005e-05 2024-08-17T18:25:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:25:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:69971 58.f 58:f4f7aaf1:::benchmark_data_smithi136_442124_object69970:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1711)", 2024-08-17T18:25:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:21:40.764667+0000", 2024-08-17T18:25:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 220.59030522, 2024-08-17T18:25:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0130957780000001, 2024-08-17T18:25:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:25:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:25:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:25:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:25:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:25:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:25:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69971 2024-08-17T18:25:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:25:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:25:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.764667+0000", 2024-08-17T18:25:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:25:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.764667+0000", 2024-08-17T18:25:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:25:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.764667+0000", 2024-08-17T18:25:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1500000000000003e-07 2024-08-17T18:25:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:25:22.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.764689+0000", 2024-08-17T18:25:22.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1665999999999999e-05 2024-08-17T18:25:22.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:25:22.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.764721+0000", 2024-08-17T18:25:22.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2348000000000003e-05 2024-08-17T18:25:22.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:25:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.764728+0000", 2024-08-17T18:25:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.511e-06 2024-08-17T18:25:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.764777+0000", 2024-08-17T18:25:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8662000000000003e-05 2024-08-17T18:25:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:25:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.764905+0000", 2024-08-17T18:25:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012809900000000001 2024-08-17T18:25:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6", 2024-08-17T18:25:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.765020+0000", 2024-08-17T18:25:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011510000000000001 2024-08-17T18:25:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:25:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.771199+0000", 2024-08-17T18:25:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.006179192 2024-08-17T18:25:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.774827+0000", 2024-08-17T18:25:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036284049999999999 2024-08-17T18:25:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.774836+0000", 2024-08-17T18:25:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9719999999999998e-06 2024-08-17T18:25:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.777696+0000", 2024-08-17T18:25:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0028596329999999 2024-08-17T18:25:22.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:25:22.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.777762+0000", 2024-08-17T18:25:22.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6375000000000001e-05 2024-08-17T18:25:22.028 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.028 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:25:22.028 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.028 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:69962 58.1 58:8a7204ea:::benchmark_data_smithi136_442124_object69961:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1712)", 2024-08-17T18:25:22.028 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:21:40.766613+0000", 2024-08-17T18:25:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 220.58835897099999, 2024-08-17T18:25:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0229622119999999, 2024-08-17T18:25:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:25:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:25:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:25:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:25:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:25:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:25:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69962 2024-08-17T18:25:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:25:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:25:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766613+0000", 2024-08-17T18:25:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:25:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766613+0000", 2024-08-17T18:25:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:25:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766615+0000", 2024-08-17T18:25:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7349999999999999e-06 2024-08-17T18:25:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:25:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766668+0000", 2024-08-17T18:25:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3797e-05 2024-08-17T18:25:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:25:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766684+0000", 2024-08-17T18:25:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5228999999999999e-05 2024-08-17T18:25:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:25:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766699+0000", 2024-08-17T18:25:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5814000000000001e-05 2024-08-17T18:25:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.767053+0000", 2024-08-17T18:25:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00035328200000000001 2024-08-17T18:25:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-08-17T18:25:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.767062+0000", 2024-08-17T18:25:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8759999999999994e-06 2024-08-17T18:25:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.771635+0000", 2024-08-17T18:25:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0045736370000000002 2024-08-17T18:25:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.771643+0000", 2024-08-17T18:25:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.503e-06 2024-08-17T18:25:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.788257+0000", 2024-08-17T18:25:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.016613797 2024-08-17T18:25:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:25:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.789575+0000", 2024-08-17T18:25:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001318542 2024-08-17T18:25:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:25:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:69966 58.1 58:843473a0:::benchmark_data_smithi136_442124_object69965:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1712)", 2024-08-17T18:25:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:21:40.766715+0000", 2024-08-17T18:25:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 220.588257089, 2024-08-17T18:25:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0367460930000001, 2024-08-17T18:25:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:25:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:25:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:25:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:25:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:25:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:25:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69966 2024-08-17T18:25:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:25:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:25:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766715+0000", 2024-08-17T18:25:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:25:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766715+0000", 2024-08-17T18:25:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:25:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766715+0000", 2024-08-17T18:25:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.68e-07 2024-08-17T18:25:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:25:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766767+0000", 2024-08-17T18:25:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1832000000000001e-05 2024-08-17T18:25:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:25:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766776+0000", 2024-08-17T18:25:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7350000000000002e-06 2024-08-17T18:25:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:25:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766782+0000", 2024-08-17T18:25:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4370000000000004e-06 2024-08-17T18:25:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.767173+0000", 2024-08-17T18:25:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00039074599999999998 2024-08-17T18:25:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:25:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.767178+0000", 2024-08-17T18:25:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1529999999999996e-06 2024-08-17T18:25:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.772138+0000", 2024-08-17T18:25:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0049600920000000001 2024-08-17T18:25:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.772145+0000", 2024-08-17T18:25:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0439999999999998e-06 2024-08-17T18:25:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.789811+0000", 2024-08-17T18:25:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.017665101 2024-08-17T18:25:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:25:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.793179+0000", 2024-08-17T18:25:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003368046 2024-08-17T18:25:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.803397+0000", 2024-08-17T18:25:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010218158 2024-08-17T18:25:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:25:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.803461+0000", 2024-08-17T18:25:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4080999999999999e-05 2024-08-17T18:25:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:25:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:69969 58.1 58:8748654a:::benchmark_data_smithi136_442124_object69968:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1712)", 2024-08-17T18:25:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:21:40.766792+0000", 2024-08-17T18:25:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 220.58817940200001, 2024-08-17T18:25:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0368268309999999, 2024-08-17T18:25:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:25:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:25:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:25:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:25:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:25:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:25:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69969 2024-08-17T18:25:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:25:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:25:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766792+0000", 2024-08-17T18:25:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:25:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766792+0000", 2024-08-17T18:25:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:25:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766793+0000", 2024-08-17T18:25:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.82e-07 2024-08-17T18:25:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:25:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766836+0000", 2024-08-17T18:25:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2725999999999997e-05 2024-08-17T18:25:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:25:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766845+0000", 2024-08-17T18:25:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8999999999999995e-06 2024-08-17T18:25:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:25:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766851+0000", 2024-08-17T18:25:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0639999999999999e-06 2024-08-17T18:25:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.767205+0000", 2024-08-17T18:25:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00035397199999999998 2024-08-17T18:25:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:25:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.767209+0000", 2024-08-17T18:25:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3610000000000003e-06 2024-08-17T18:25:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.772165+0000", 2024-08-17T18:25:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0049556840000000001 2024-08-17T18:25:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.772170+0000", 2024-08-17T18:25:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5620000000000003e-06 2024-08-17T18:25:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.793214+0000", 2024-08-17T18:25:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.021043905 2024-08-17T18:25:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:25:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.793793+0000", 2024-08-17T18:25:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00057870299999999996 2024-08-17T18:25:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7", 2024-08-17T18:25:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.793927+0000", 2024-08-17T18:25:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013374900000000001 2024-08-17T18:25:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:25:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.800104+0000", 2024-08-17T18:25:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0061776119999999999 2024-08-17T18:25:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:25:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.803504+0000", 2024-08-17T18:25:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034001399999999998 2024-08-17T18:25:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:25:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.803549+0000", 2024-08-17T18:25:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5021999999999997e-05 2024-08-17T18:25:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:25:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.803619+0000", 2024-08-17T18:25:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9849000000000002e-05 2024-08-17T18:25:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:25:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:69972 58.1 58:86412dc6:::benchmark_data_smithi136_442124_object69971:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1712)", 2024-08-17T18:25:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:21:40.766861+0000", 2024-08-17T18:25:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 220.588111271, 2024-08-17T18:25:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0381175920000001, 2024-08-17T18:25:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:25:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:25:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:25:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:25:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:25:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:25:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69972 2024-08-17T18:25:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:25:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:25:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766861+0000", 2024-08-17T18:25:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:25:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766861+0000", 2024-08-17T18:25:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:25:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766861+0000", 2024-08-17T18:25:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2900000000000004e-07 2024-08-17T18:25:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:25:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766880+0000", 2024-08-17T18:25:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9296e-05 2024-08-17T18:25:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:25:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766888+0000", 2024-08-17T18:25:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.841e-06 2024-08-17T18:25:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:25:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.766895+0000", 2024-08-17T18:25:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5010000000000001e-06 2024-08-17T18:25:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.772192+0000", 2024-08-17T18:25:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0052969840000000002 2024-08-17T18:25:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.772197+0000", 2024-08-17T18:25:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1610000000000002e-06 2024-08-17T18:25:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.796816+0000", 2024-08-17T18:25:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.024618764 2024-08-17T18:25:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:25:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.796958+0000", 2024-08-17T18:25:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0001428 2024-08-17T18:25:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7", 2024-08-17T18:25:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.797098+0000", 2024-08-17T18:25:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000139717 2024-08-17T18:25:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:25:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.803654+0000", 2024-08-17T18:25:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0065561539999999998 2024-08-17T18:25:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:25:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.804797+0000", 2024-08-17T18:25:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001143175 2024-08-17T18:25:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:25:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.804904+0000", 2024-08-17T18:25:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010654699999999999 2024-08-17T18:25:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:25:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.804978+0000", 2024-08-17T18:25:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4122999999999995e-05 2024-08-17T18:25:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:25:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:69979 58.f 58:f004dd5c:::benchmark_data_smithi136_442124_object69978:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1712)", 2024-08-17T18:25:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:21:40.781595+0000", 2024-08-17T18:25:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 220.57337657900001, 2024-08-17T18:25:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.000482329, 2024-08-17T18:25:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:25:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:25:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:25:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:25:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:25:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:25:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69979 2024-08-17T18:25:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:25:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:25:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.781595+0000", 2024-08-17T18:25:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:25:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.781595+0000", 2024-08-17T18:25:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:25:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.781597+0000", 2024-08-17T18:25:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.288e-06 2024-08-17T18:25:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:25:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.781648+0000", 2024-08-17T18:25:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1322000000000001e-05 2024-08-17T18:25:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:25:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.781657+0000", 2024-08-17T18:25:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2760000000000006e-06 2024-08-17T18:25:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:25:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.781662+0000", 2024-08-17T18:25:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.972e-06 2024-08-17T18:25:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.781715+0000", 2024-08-17T18:25:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2861000000000001e-05 2024-08-17T18:25:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.781736+0000", 2024-08-17T18:25:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1469000000000001e-05 2024-08-17T18:25:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.777787+0000", 2024-08-17T18:25:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.99605030400000005 2024-08-17T18:25:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:25:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.777936+0000", 2024-08-17T18:25:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000149636 2024-08-17T18:25:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-08-17T18:25:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.778088+0000", 2024-08-17T18:25:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015177499999999999 2024-08-17T18:25:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:25:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.780854+0000", 2024-08-17T18:25:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0027656930000000001 2024-08-17T18:25:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:25:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.781938+0000", 2024-08-17T18:25:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001083989 2024-08-17T18:25:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:25:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.781991+0000", 2024-08-17T18:25:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2972e-05 2024-08-17T18:25:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:25:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.782078+0000", 2024-08-17T18:25:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6772000000000002e-05 2024-08-17T18:25:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:25:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:69980 58.b 58:d5b260fd:::benchmark_data_smithi136_442124_object69979:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1712)", 2024-08-17T18:25:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:21:40.781826+0000", 2024-08-17T18:25:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 220.573146302, 2024-08-17T18:25:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.00692742, 2024-08-17T18:25:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:25:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:25:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:25:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:25:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:25:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:25:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69980 2024-08-17T18:25:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:25:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:25:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.781826+0000", 2024-08-17T18:25:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:25:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.781826+0000", 2024-08-17T18:25:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:25:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.781826+0000", 2024-08-17T18:25:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6899999999999997e-07 2024-08-17T18:25:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:25:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.781842+0000", 2024-08-17T18:25:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6008e-05 2024-08-17T18:25:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:25:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.781850+0000", 2024-08-17T18:25:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.661e-06 2024-08-17T18:25:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:25:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.781854+0000", 2024-08-17T18:25:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3189999999999996e-06 2024-08-17T18:25:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.781904+0000", 2024-08-17T18:25:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9428000000000002e-05 2024-08-17T18:25:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.781912+0000", 2024-08-17T18:25:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9140000000000008e-06 2024-08-17T18:25:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.779809+0000", 2024-08-17T18:25:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.99789777899999998 2024-08-17T18:25:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:25:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.779976+0000", 2024-08-17T18:25:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016633199999999999 2024-08-17T18:25:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7", 2024-08-17T18:25:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.780134+0000", 2024-08-17T18:25:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000158818 2024-08-17T18:25:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:25:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.782704+0000", 2024-08-17T18:25:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025691939999999999 2024-08-17T18:25:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:25:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.788606+0000", 2024-08-17T18:25:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005902109 2024-08-17T18:25:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:25:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.788664+0000", 2024-08-17T18:25:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8179e-05 2024-08-17T18:25:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:25:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.788753+0000", 2024-08-17T18:25:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9010000000000003e-05 2024-08-17T18:25:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:25:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:69982 58.1 58:804eb9f4:::benchmark_data_smithi136_442124_object69981:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1712)", 2024-08-17T18:25:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:21:40.786008+0000", 2024-08-17T18:25:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 220.56896426, 2024-08-17T18:25:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0197458770000001, 2024-08-17T18:25:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:25:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:25:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:25:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:25:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:25:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:25:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 69982 2024-08-17T18:25:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:25:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:25:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.786008+0000", 2024-08-17T18:25:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:25:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.786008+0000", 2024-08-17T18:25:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:25:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.786009+0000", 2024-08-17T18:25:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3179999999999999e-06 2024-08-17T18:25:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:25:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.786027+0000", 2024-08-17T18:25:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8323999999999999e-05 2024-08-17T18:25:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:25:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.786039+0000", 2024-08-17T18:25:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1775e-05 2024-08-17T18:25:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:25:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.786045+0000", 2024-08-17T18:25:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9449999999999998e-06 2024-08-17T18:25:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.786097+0000", 2024-08-17T18:25:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2509000000000002e-05 2024-08-17T18:25:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:40.786108+0000", 2024-08-17T18:25:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0467e-05 2024-08-17T18:25:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.800161+0000", 2024-08-17T18:25:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.014052876 2024-08-17T18:25:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:25:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.800290+0000", 2024-08-17T18:25:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129479 2024-08-17T18:25:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7", 2024-08-17T18:25:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.800428+0000", 2024-08-17T18:25:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013809099999999999 2024-08-17T18:25:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:25:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.804985+0000", 2024-08-17T18:25:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0045570380000000002 2024-08-17T18:25:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:25:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.805600+0000", 2024-08-17T18:25:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00061424400000000001 2024-08-17T18:25:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:25:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.805665+0000", 2024-08-17T18:25:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5727000000000003e-05 2024-08-17T18:25:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:25:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:21:41.805753+0000", 2024-08-17T18:25:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8084000000000006e-05 2024-08-17T18:25:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:25:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:117404 58.f 58:fe4fcbb2:::benchmark_data_smithi136_442124_object3788:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1723)", 2024-08-17T18:25:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:35.804989+0000", 2024-08-17T18:25:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 165.54998240699999, 2024-08-17T18:25:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2525179280000001, 2024-08-17T18:25:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:25:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:25:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:25:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:25:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:25:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:25:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 117404 2024-08-17T18:25:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:25:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:25:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.804989+0000", 2024-08-17T18:25:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:25:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.804989+0000", 2024-08-17T18:25:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:25:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.804991+0000", 2024-08-17T18:25:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7799999999999999e-06 2024-08-17T18:25:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:25:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.804997+0000", 2024-08-17T18:25:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8069999999999996e-06 2024-08-17T18:25:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:25:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.805014+0000", 2024-08-17T18:25:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7399999999999999e-05 2024-08-17T18:25:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:25:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.805022+0000", 2024-08-17T18:25:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7579999999999992e-06 2024-08-17T18:25:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.805076+0000", 2024-08-17T18:25:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3517000000000003e-05 2024-08-17T18:25:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:25:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.805213+0000", 2024-08-17T18:25:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013738199999999999 2024-08-17T18:25:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-08-17T18:25:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.805328+0000", 2024-08-17T18:25:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115043 2024-08-17T18:25:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:25:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.811964+0000", 2024-08-17T18:25:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0066355900000000002 2024-08-17T18:25:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.940756+0000", 2024-08-17T18:25:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.12879208 2024-08-17T18:25:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.940766+0000", 2024-08-17T18:25:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0451999999999999e-05 2024-08-17T18:25:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825054+0000", 2024-08-17T18:25:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.88428820600000002 2024-08-17T18:25:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825064+0000", 2024-08-17T18:25:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.876e-06 2024-08-17T18:25:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.019962+0000", 2024-08-17T18:25:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1948974530000001 2024-08-17T18:25:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for missing object", 2024-08-17T18:25:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.020076+0000", 2024-08-17T18:25:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114209 2024-08-17T18:25:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.057414+0000", 2024-08-17T18:25:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037337813999999997 2024-08-17T18:25:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:25:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.057507+0000", 2024-08-17T18:25:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3560999999999994e-05 2024-08-17T18:25:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:25:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:117410 58.f 58:fcc716a0:::benchmark_data_smithi136_442124_object3794:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1723)", 2024-08-17T18:25:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:35.813549+0000", 2024-08-17T18:25:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 165.541422482, 2024-08-17T18:25:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.255462681, 2024-08-17T18:25:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:25:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:25:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:25:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:25:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:25:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:25:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 117410 2024-08-17T18:25:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:25:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:25:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.813549+0000", 2024-08-17T18:25:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:25:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.813549+0000", 2024-08-17T18:25:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:25:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.813551+0000", 2024-08-17T18:25:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6810000000000001e-06 2024-08-17T18:25:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:25:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.813556+0000", 2024-08-17T18:25:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9180000000000002e-06 2024-08-17T18:25:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:25:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.813574+0000", 2024-08-17T18:25:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8117e-05 2024-08-17T18:25:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:25:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.813592+0000", 2024-08-17T18:25:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7524999999999999e-05 2024-08-17T18:25:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.813662+0000", 2024-08-17T18:25:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0914999999999994e-05 2024-08-17T18:25:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:25:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.813804+0000", 2024-08-17T18:25:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000141827 2024-08-17T18:25:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-08-17T18:25:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.813935+0000", 2024-08-17T18:25:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000130541 2024-08-17T18:25:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.940795+0000", 2024-08-17T18:25:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.12686061200000001 2024-08-17T18:25:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.940803+0000", 2024-08-17T18:25:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.977e-06 2024-08-17T18:25:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825091+0000", 2024-08-17T18:25:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.88428778900000005 2024-08-17T18:25:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825099+0000", 2024-08-17T18:25:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.96e-06 2024-08-17T18:25:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.022200+0000", 2024-08-17T18:25:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1971010230000001 2024-08-17T18:25:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for missing object", 2024-08-17T18:25:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.022319+0000", 2024-08-17T18:25:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011838200000000001 2024-08-17T18:25:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.068343+0000", 2024-08-17T18:25:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.046023982999999997 2024-08-17T18:25:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:25:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.069012+0000", 2024-08-17T18:25:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00066943099999999995 2024-08-17T18:25:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:25:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:117423 58.b 58:d37fea46:::benchmark_data_smithi136_442124_object3807:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1724)", 2024-08-17T18:25:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:35.940469+0000", 2024-08-17T18:25:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 165.41450318099999, 2024-08-17T18:25:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0865856489999999, 2024-08-17T18:25:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:25:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:25:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:25:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:25:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:25:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:25:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 117423 2024-08-17T18:25:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:25:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:25:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.940469+0000", 2024-08-17T18:25:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:25:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.940469+0000", 2024-08-17T18:25:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:25:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.940469+0000", 2024-08-17T18:25:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2399999999999999e-07 2024-08-17T18:25:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:25:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.940471+0000", 2024-08-17T18:25:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.187e-06 2024-08-17T18:25:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:25:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.940477+0000", 2024-08-17T18:25:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.207e-06 2024-08-17T18:25:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:25:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.940482+0000", 2024-08-17T18:25:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.023e-06 2024-08-17T18:25:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.943392+0000", 2024-08-17T18:25:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0029097580000000001 2024-08-17T18:25:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.943403+0000", 2024-08-17T18:25:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0774999999999999e-05 2024-08-17T18:25:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.823714+0000", 2024-08-17T18:25:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.88031114200000005 2024-08-17T18:25:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.823725+0000", 2024-08-17T18:25:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0878e-05 2024-08-17T18:25:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.006984+0000", 2024-08-17T18:25:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.183258822 2024-08-17T18:25:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:25:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.007126+0000", 2024-08-17T18:25:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000142583 2024-08-17T18:25:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6,7", 2024-08-17T18:25:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.007260+0000", 2024-08-17T18:25:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013366800000000001 2024-08-17T18:25:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:25:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.014187+0000", 2024-08-17T18:25:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.006926906 2024-08-17T18:25:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:25:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.023201+0000", 2024-08-17T18:25:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0090138490000000009 2024-08-17T18:25:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:25:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.023236+0000", 2024-08-17T18:25:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4897999999999997e-05 2024-08-17T18:25:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:25:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.023337+0000", 2024-08-17T18:25:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010164499999999999 2024-08-17T18:25:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:25:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.027054+0000", 2024-08-17T18:25:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037168840000000002 2024-08-17T18:25:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:25:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:117433 58.f 58:f60495c5:::benchmark_data_smithi136_442124_object3817:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1724)", 2024-08-17T18:25:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:35.952663+0000", 2024-08-17T18:25:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 165.40230844800001, 2024-08-17T18:25:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0992727279999999, 2024-08-17T18:25:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:25:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:25:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:25:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:25:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:25:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:25:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 117433 2024-08-17T18:25:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:25:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:25:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952663+0000", 2024-08-17T18:25:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:25:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952663+0000", 2024-08-17T18:25:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:25:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952665+0000", 2024-08-17T18:25:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4339999999999999e-06 2024-08-17T18:25:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:25:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952668+0000", 2024-08-17T18:25:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4649999999999999e-06 2024-08-17T18:25:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:25:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952682+0000", 2024-08-17T18:25:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3553000000000001e-05 2024-08-17T18:25:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:25:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952711+0000", 2024-08-17T18:25:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8724000000000001e-05 2024-08-17T18:25:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952770+0000", 2024-08-17T18:25:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8974000000000002e-05 2024-08-17T18:25:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952778+0000", 2024-08-17T18:25:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.541e-06 2024-08-17T18:25:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825122+0000", 2024-08-17T18:25:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.87234362399999998 2024-08-17T18:25:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825129+0000", 2024-08-17T18:25:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8789999999999997e-06 2024-08-17T18:25:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.022505+0000", 2024-08-17T18:25:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1973766109999999 2024-08-17T18:25:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:25:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.022830+0000", 2024-08-17T18:25:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00032473300000000001 2024-08-17T18:25:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6", 2024-08-17T18:25:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.023450+0000", 2024-08-17T18:25:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00062009700000000001 2024-08-17T18:25:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:25:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.028993+0000", 2024-08-17T18:25:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0055434880000000001 2024-08-17T18:25:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:25:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.030046+0000", 2024-08-17T18:25:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010523920000000001 2024-08-17T18:25:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:25:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.049751+0000", 2024-08-17T18:25:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019705361000000001 2024-08-17T18:25:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:25:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.049831+0000", 2024-08-17T18:25:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9535000000000003e-05 2024-08-17T18:25:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:25:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.051936+0000", 2024-08-17T18:25:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021053170000000002 2024-08-17T18:25:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:25:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:117434 58.b 58:dfc4ffd4:::benchmark_data_smithi136_442124_object3818:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1724)", 2024-08-17T18:25:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:35.952917+0000", 2024-08-17T18:25:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 165.40205509899999, 2024-08-17T18:25:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0786143039999998, 2024-08-17T18:25:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:25:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:25:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:25:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:25:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:25:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:25:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 117434 2024-08-17T18:25:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:25:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:25:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952917+0000", 2024-08-17T18:25:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:25:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952917+0000", 2024-08-17T18:25:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:25:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952918+0000", 2024-08-17T18:25:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9599999999999998e-07 2024-08-17T18:25:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:25:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952921+0000", 2024-08-17T18:25:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8970000000000001e-06 2024-08-17T18:25:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:25:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952934+0000", 2024-08-17T18:25:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2991e-05 2024-08-17T18:25:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:25:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952941+0000", 2024-08-17T18:25:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5540000000000003e-06 2024-08-17T18:25:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952994+0000", 2024-08-17T18:25:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3548999999999998e-05 2024-08-17T18:25:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.953005+0000", 2024-08-17T18:25:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0739e-05 2024-08-17T18:25:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.823747+0000", 2024-08-17T18:25:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.87074168900000004 2024-08-17T18:25:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.823754+0000", 2024-08-17T18:25:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8460000000000002e-06 2024-08-17T18:25:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.014223+0000", 2024-08-17T18:25:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1904697099999999 2024-08-17T18:25:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:25:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.014370+0000", 2024-08-17T18:25:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014660199999999999 2024-08-17T18:25:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6,7", 2024-08-17T18:25:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.014513+0000", 2024-08-17T18:25:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014304699999999999 2024-08-17T18:25:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:25:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.027631+0000", 2024-08-17T18:25:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013118029 2024-08-17T18:25:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:25:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.027681+0000", 2024-08-17T18:25:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0182000000000002e-05 2024-08-17T18:25:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:25:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.029620+0000", 2024-08-17T18:25:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001938698 2024-08-17T18:25:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:25:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.029816+0000", 2024-08-17T18:25:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019562099999999999 2024-08-17T18:25:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:25:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.031531+0000", 2024-08-17T18:25:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017153540000000001 2024-08-17T18:25:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:25:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:117439 58.b 58:dd77dae4:::benchmark_data_smithi136_442124_object3823:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1724)", 2024-08-17T18:25:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:35.963232+0000", 2024-08-17T18:25:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 165.39174009000001, 2024-08-17T18:25:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0702053380000001, 2024-08-17T18:25:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:25:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:25:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:25:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:25:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:25:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:25:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 117439 2024-08-17T18:25:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:25:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:25:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.963232+0000", 2024-08-17T18:25:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:25:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.963232+0000", 2024-08-17T18:25:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:25:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.963233+0000", 2024-08-17T18:25:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.595e-06 2024-08-17T18:25:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:25:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.963237+0000", 2024-08-17T18:25:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9960000000000004e-06 2024-08-17T18:25:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:25:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.963248+0000", 2024-08-17T18:25:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0886e-05 2024-08-17T18:25:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:25:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.963254+0000", 2024-08-17T18:25:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5010000000000004e-06 2024-08-17T18:25:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.963304+0000", 2024-08-17T18:25:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9937e-05 2024-08-17T18:25:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.963311+0000", 2024-08-17T18:25:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6720000000000004e-06 2024-08-17T18:25:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.823776+0000", 2024-08-17T18:25:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.86046500800000003 2024-08-17T18:25:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.823783+0000", 2024-08-17T18:25:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5710000000000003e-06 2024-08-17T18:25:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.018529+0000", 2024-08-17T18:25:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.194746613 2024-08-17T18:25:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:25:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.018796+0000", 2024-08-17T18:25:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00026683499999999999 2024-08-17T18:25:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6,7", 2024-08-17T18:25:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.019254+0000", 2024-08-17T18:25:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00045742200000000003 2024-08-17T18:25:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:25:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.027714+0000", 2024-08-17T18:25:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0084598190000000004 2024-08-17T18:25:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:25:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.029446+0000", 2024-08-17T18:25:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017321599999999999 2024-08-17T18:25:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:25:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.031589+0000", 2024-08-17T18:25:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021433369999999999 2024-08-17T18:25:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:25:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.031865+0000", 2024-08-17T18:25:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027591600000000002 2024-08-17T18:25:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:25:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.033437+0000", 2024-08-17T18:25:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015720700000000001 2024-08-17T18:25:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:25:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:117397 58.1 58:8ed9847d:::benchmark_data_smithi136_442124_object3781:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e1725)", 2024-08-17T18:25:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:36.824050+0000", 2024-08-17T18:25:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 164.53092143999999, 2024-08-17T18:25:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1889095890000001, 2024-08-17T18:25:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:25:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:25:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:25:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:25:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:25:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:25:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 117397 2024-08-17T18:25:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:25:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:25:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.824050+0000", 2024-08-17T18:25:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:25:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.824050+0000", 2024-08-17T18:25:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:25:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.824052+0000", 2024-08-17T18:25:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5269999999999999e-06 2024-08-17T18:25:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:25:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.824057+0000", 2024-08-17T18:25:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.198e-06 2024-08-17T18:25:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:25:22.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.824069+0000", 2024-08-17T18:25:22.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2119e-05 2024-08-17T18:25:22.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:25:22.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.824080+0000", 2024-08-17T18:25:22.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0685e-05 2024-08-17T18:25:22.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825907+0000", 2024-08-17T18:25:22.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001827394 2024-08-17T18:25:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825923+0000", 2024-08-17T18:25:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5722999999999998e-05 2024-08-17T18:25:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:37.992420+0000", 2024-08-17T18:25:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.166496862 2024-08-17T18:25:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for missing object", 2024-08-17T18:25:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:37.992516+0000", 2024-08-17T18:25:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6419e-05 2024-08-17T18:25:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.010810+0000", 2024-08-17T18:25:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018294089999999999 2024-08-17T18:25:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:25:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.012960+0000", 2024-08-17T18:25:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021495720000000002 2024-08-17T18:25:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:25:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:117402 58.1 58:8c1f36cd:::benchmark_data_smithi136_442124_object3786:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e1725)", 2024-08-17T18:25:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:36.825743+0000", 2024-08-17T18:25:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 164.52922922799999, 2024-08-17T18:25:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.187210938, 2024-08-17T18:25:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:25:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:25:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:25:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:25:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:25:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:25:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 117402 2024-08-17T18:25:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:25:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:25:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825743+0000", 2024-08-17T18:25:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:25:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825743+0000", 2024-08-17T18:25:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:25:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825743+0000", 2024-08-17T18:25:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.92e-07 2024-08-17T18:25:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:25:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825747+0000", 2024-08-17T18:25:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8070000000000001e-06 2024-08-17T18:25:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:25:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825757+0000", 2024-08-17T18:25:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5109999999999992e-06 2024-08-17T18:25:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:25:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825765+0000", 2024-08-17T18:25:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8350000000000004e-06 2024-08-17T18:25:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825973+0000", 2024-08-17T18:25:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020851499999999999 2024-08-17T18:25:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825982+0000", 2024-08-17T18:25:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1740000000000006e-06 2024-08-17T18:25:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:37.993122+0000", 2024-08-17T18:25:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1671400789999999 2024-08-17T18:25:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:25:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:37.993191+0000", 2024-08-17T18:25:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8961999999999995e-05 2024-08-17T18:25:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6,7", 2024-08-17T18:25:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:37.993262+0000", 2024-08-17T18:25:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0992000000000001e-05 2024-08-17T18:25:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:25:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.005064+0000", 2024-08-17T18:25:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011801944 2024-08-17T18:25:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:25:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.005123+0000", 2024-08-17T18:25:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8857000000000002e-05 2024-08-17T18:25:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:25:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.011150+0000", 2024-08-17T18:25:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0060272779999999996 2024-08-17T18:25:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:25:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.011333+0000", 2024-08-17T18:25:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000182265 2024-08-17T18:25:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:25:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.012954+0000", 2024-08-17T18:25:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016209270000000001 2024-08-17T18:25:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:25:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:117414 58.1 58:8425588e:::benchmark_data_smithi136_442124_object3798:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e1725)", 2024-08-17T18:25:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:36.825776+0000", 2024-08-17T18:25:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 164.52919605400001, 2024-08-17T18:25:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1884016289999999, 2024-08-17T18:25:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:25:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:25:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:25:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:25:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:25:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:25:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 117414 2024-08-17T18:25:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:25:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:25:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825776+0000", 2024-08-17T18:25:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:25:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825776+0000", 2024-08-17T18:25:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:25:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825776+0000", 2024-08-17T18:25:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8599999999999998e-07 2024-08-17T18:25:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:25:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825779+0000", 2024-08-17T18:25:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.588e-06 2024-08-17T18:25:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:25:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825786+0000", 2024-08-17T18:25:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7490000000000001e-06 2024-08-17T18:25:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:25:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825791+0000", 2024-08-17T18:25:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.784e-06 2024-08-17T18:25:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.826014+0000", 2024-08-17T18:25:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022224500000000001 2024-08-17T18:25:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.826021+0000", 2024-08-17T18:25:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3919999999999997e-06 2024-08-17T18:25:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:37.996158+0000", 2024-08-17T18:25:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.17013685 2024-08-17T18:25:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:25:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:37.996280+0000", 2024-08-17T18:25:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012233799999999999 2024-08-17T18:25:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6,7", 2024-08-17T18:25:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:37.996403+0000", 2024-08-17T18:25:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123289 2024-08-17T18:25:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:25:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.005169+0000", 2024-08-17T18:25:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0087653950000000005 2024-08-17T18:25:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:25:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.005211+0000", 2024-08-17T18:25:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2447000000000002e-05 2024-08-17T18:25:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:25:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.013105+0000", 2024-08-17T18:25:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0078940360000000001 2024-08-17T18:25:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:25:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.013191+0000", 2024-08-17T18:25:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5391999999999998e-05 2024-08-17T18:25:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:25:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.014177+0000", 2024-08-17T18:25:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00098663799999999997 2024-08-17T18:25:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:25:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:117429 58.1 58:8ca0996e:::benchmark_data_smithi136_442124_object3813:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1725)", 2024-08-17T18:25:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:36.825799+0000", 2024-08-17T18:25:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 164.52917328999999, 2024-08-17T18:25:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1894211880000001, 2024-08-17T18:25:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:25:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:25:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:25:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:25:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:25:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:25:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 117429 2024-08-17T18:25:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:25:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:25:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825799+0000", 2024-08-17T18:25:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:25:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825799+0000", 2024-08-17T18:25:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:25:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825799+0000", 2024-08-17T18:25:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.75e-07 2024-08-17T18:25:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:25:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825801+0000", 2024-08-17T18:25:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1890000000000001e-06 2024-08-17T18:25:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:25:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825808+0000", 2024-08-17T18:25:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4999999999999996e-06 2024-08-17T18:25:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:25:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.825813+0000", 2024-08-17T18:25:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8659999999999997e-06 2024-08-17T18:25:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.826045+0000", 2024-08-17T18:25:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023225399999999999 2024-08-17T18:25:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.826052+0000", 2024-08-17T18:25:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5719999999999997e-06 2024-08-17T18:25:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.000270+0000", 2024-08-17T18:25:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.174217939 2024-08-17T18:25:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:25:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.000421+0000", 2024-08-17T18:25:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015111899999999999 2024-08-17T18:25:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6,7", 2024-08-17T18:25:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.000565+0000", 2024-08-17T18:25:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014338099999999999 2024-08-17T18:25:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:25:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.005589+0000", 2024-08-17T18:25:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0050238080000000003 2024-08-17T18:25:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:25:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.013089+0000", 2024-08-17T18:25:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0075006170000000002 2024-08-17T18:25:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:25:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.014186+0000", 2024-08-17T18:25:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001097198 2024-08-17T18:25:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:25:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.014276+0000", 2024-08-17T18:25:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9073000000000006e-05 2024-08-17T18:25:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:25:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.015220+0000", 2024-08-17T18:25:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00094419700000000003 2024-08-17T18:25:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:25:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:171677 58.1 58.d8e3a971 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1740)", 2024-08-17T18:25:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:58.785388+0000", 2024-08-17T18:25:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 82.569584262000006, 2024-08-17T18:25:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93420657500000004, 2024-08-17T18:25:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:25:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:25:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:25:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:25:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:25:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:25:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 171677 2024-08-17T18:25:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:25:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:25:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.785388+0000", 2024-08-17T18:25:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:25:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.785388+0000", 2024-08-17T18:25:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:25:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:25:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.785389+0000", 2024-08-17T18:25:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5969999999999999e-06 2024-08-17T18:25:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:25:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.785395+0000", 2024-08-17T18:25:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9370000000000001e-06 2024-08-17T18:25:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:25:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.785413+0000", 2024-08-17T18:25:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7768999999999999e-05 2024-08-17T18:25:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:25:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.785423+0000", 2024-08-17T18:25:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.008e-05 2024-08-17T18:25:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.785728+0000", 2024-08-17T18:25:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00030556600000000001 2024-08-17T18:25:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:25:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.785742+0000", 2024-08-17T18:25:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3033e-05 2024-08-17T18:25:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:25:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:59.719582+0000", 2024-08-17T18:25:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93383998599999996 2024-08-17T18:25:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:25:22.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:25:22.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:25:22.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:59.719594+0000", 2024-08-17T18:25:22.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2607e-05 2024-08-17T18:25:22.157 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.157 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:25:22.157 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.157 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:25:22.157 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:25:22.157 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:25:22.157 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:22.183 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:22.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:22.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:21 smithi136 ceph-mon[109861]: pgmap v3491: 99 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 1 active+recovering+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 1.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 340 op/s; 5293/56967 objects degraded (9.291%); 6039/56967 objects misplaced (10.601%); 15 MiB/s, 334 objects/s recovering 2024-08-17T18:25:22.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:25:22.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:21 smithi136 ceph-mon[115078]: pgmap v3491: 99 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 1 active+recovering+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 1.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 340 op/s; 5293/56967 objects degraded (9.291%); 6039/56967 objects misplaced (10.601%); 15 MiB/s, 334 objects/s recovering 2024-08-17T18:25:22.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:25:22.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:21 smithi165 ceph-mon[105517]: pgmap v3491: 99 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 1 active+recovering+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 1.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 340 op/s; 5293/56967 objects degraded (9.291%); 6039/56967 objects misplaced (10.601%); 15 MiB/s, 334 objects/s recovering 2024-08-17T18:25:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:25:22.285 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:22.310 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:22.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:22.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:22.412 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:22.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:22.435 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:22.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:22.536 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:22.564 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:22.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:22.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:22.666 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:22.687 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:22.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:22.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:22.789 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:22.812 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:22.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:22.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:22.915 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:22.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:22.960 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:22.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:23.061 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:23.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:23.088 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:23.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:23.189 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:23.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:22 smithi136 ceph-mon[109861]: osdmap e1756: 8 total, 8 up, 8 in 2024-08-17T18:25:23.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:22 smithi136 ceph-mon[115078]: osdmap e1756: 8 total, 8 up, 8 in 2024-08-17T18:25:23.215 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:23.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:23.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:23.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:22 smithi165 ceph-mon[105517]: osdmap e1756: 8 total, 8 up, 8 in 2024-08-17T18:25:23.317 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:23.344 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:23.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:23.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:23.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:23.471 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:23.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:23.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:23.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:23.601 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:23.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:23.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:23.702 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:23.729 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:23.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:23.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:23.831 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:23.854 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:23.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:23.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:23.959 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:23.986 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:23.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:23.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:24.088 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:24.121 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:24.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:24.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:24.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:23 smithi136 ceph-mon[109861]: pgmap v3493: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 398 op/s; 4740/54378 objects degraded (8.717%); 5486/54378 objects misplaced (10.089%); 17 MiB/s, 353 objects/s recovering 2024-08-17T18:25:24.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:23 smithi136 ceph-mon[109861]: osdmap e1757: 8 total, 8 up, 8 in 2024-08-17T18:25:24.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:23 smithi136 ceph-mon[115078]: pgmap v3493: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 398 op/s; 4740/54378 objects degraded (8.717%); 5486/54378 objects misplaced (10.089%); 17 MiB/s, 353 objects/s recovering 2024-08-17T18:25:24.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:23 smithi136 ceph-mon[115078]: osdmap e1757: 8 total, 8 up, 8 in 2024-08-17T18:25:24.223 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:24.252 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:24.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:24.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:24.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:23 smithi165 ceph-mon[105517]: pgmap v3493: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 398 op/s; 4740/54378 objects degraded (8.717%); 5486/54378 objects misplaced (10.089%); 17 MiB/s, 353 objects/s recovering 2024-08-17T18:25:24.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:23 smithi165 ceph-mon[105517]: osdmap e1757: 8 total, 8 up, 8 in 2024-08-17T18:25:24.354 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:24.382 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:24.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:24.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:24.483 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:24.505 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:24.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:24.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:24.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:24.635 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:24.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:24.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:24.737 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:24.761 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:24.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:24.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:24.863 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:24.896 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:24.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:24.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:24.998 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:25.024 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:25.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:25.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:25.126 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:25.152 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:25.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:25.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:25.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:24 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 4740/54378 objects degraded (8.717%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:25.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:24 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 4740/54378 objects degraded (8.717%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:25.253 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:25.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:24 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 4740/54378 objects degraded (8.717%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:25.276 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:25.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:25.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:25.378 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:25.402 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:25.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:25.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:25.504 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:25.529 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:25.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:25.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:25.630 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:25.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:25.668 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:25.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:25.768 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:25.796 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:25.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:25.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:25.898 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:25.933 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:25.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:25.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:26.034 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:26.047 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-17T18:25:26.059 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:26.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:26.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:26.125 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:25 smithi136 ceph-mon[115078]: pgmap v3495: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 310 op/s; 4740/53559 objects degraded (8.850%); 5486/53559 objects misplaced (10.243%); 12 MiB/s, 274 objects/s recovering 2024-08-17T18:25:26.125 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:25 smithi136 ceph-mon[109861]: pgmap v3495: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 310 op/s; 4740/53559 objects degraded (8.850%); 5486/53559 objects misplaced (10.243%); 12 MiB/s, 274 objects/s recovering 2024-08-17T18:25:26.161 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:26.185 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:26.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:26.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:26.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:25 smithi165 ceph-mon[105517]: pgmap v3495: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 90 active+clean; 1.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 310 op/s; 4740/53559 objects degraded (8.850%); 5486/53559 objects misplaced (10.243%); 12 MiB/s, 274 objects/s recovering 2024-08-17T18:25:26.287 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:26.327 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:26.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:26.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:26.429 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:26.457 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:26.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:26.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:26.559 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:26.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:26.591 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:26.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:26.692 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:26.714 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:26.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:26.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:26.816 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:26.843 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:26.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:26.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:26.945 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:26.970 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:26.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:26.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:27.072 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:27.100 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:27.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:27.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:27.201 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:27.227 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:27.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:27.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:27.329 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:27.367 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:27.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:27.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:27.469 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:27.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:27.511 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:27.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:27.612 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:27.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:27.659 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:27.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:27.707 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:25:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:25:27] "GET /metrics HTTP/1.1" 200 36713 "" "Prometheus/2.51.0" 2024-08-17T18:25:27.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:27.784 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:27.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:27.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:27.884 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:25:27.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:27.890 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T18:25:27.917 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:27.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:27.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:27.973 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:27 smithi136 ceph-mon[109861]: pgmap v3496: 99 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 1 active+clean+scrubbing, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 1002 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 500 op/s; 4003/48147 objects degraded (8.314%); 5108/48147 objects misplaced (10.609%); 19 MiB/s, 337 objects/s recovering 2024-08-17T18:25:27.973 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:27 smithi136 ceph-mon[115078]: pgmap v3496: 99 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 1 active+clean+scrubbing, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 1002 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 500 op/s; 4003/48147 objects degraded (8.314%); 5108/48147 objects misplaced (10.609%); 19 MiB/s, 337 objects/s recovering 2024-08-17T18:25:28.019 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:28.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:28.062 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:28.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:28.163 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:28.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:28.195 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:28.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:28.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:27 smithi165 ceph-mon[105517]: pgmap v3496: 99 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 1 active+clean+scrubbing, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 1002 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 500 op/s; 4003/48147 objects degraded (8.314%); 5108/48147 objects misplaced (10.609%); 19 MiB/s, 337 objects/s recovering 2024-08-17T18:25:28.296 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:28.321 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:28.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:28.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:28.423 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:28.459 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:28.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:28.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:28.561 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:28.600 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:28.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:28.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:28.702 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:28.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:28.740 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:28.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:28.841 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:28.879 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:25:28.887 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:28.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:28.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:28.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:29.011 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:29.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:29.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:29.113 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:29.144 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:29.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:29.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:28 smithi136 ceph-mon[109861]: from='client.56269 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:25:29.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:28 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/370129881' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:25:29.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:28 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:25:29.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:28 smithi136 ceph-mon[115078]: from='client.56269 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:25:29.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:28 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/370129881' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:25:29.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:28 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:25:29.244 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:29.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:28 smithi165 ceph-mon[105517]: from='client.56269 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:25:29.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:28 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/370129881' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:25:29.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:28 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:25:29.269 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:29.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:29.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:29.371 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:29.397 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:29.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:29.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:29.499 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:29.528 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:29.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:29.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:29.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:29.659 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:29.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:29.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:29.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:29.785 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:29.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:29.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:29.887 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:29.911 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:29.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:29.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:30.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:30.038 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:30.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:30.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:30.140 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:30.167 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:30.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:30.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:29 smithi136 ceph-mon[109861]: pgmap v3497: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 1 active+clean+scrubbing, 3 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 976 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 419 op/s; 3594/46893 objects degraded (7.664%); 5108/46893 objects misplaced (10.893%); 13 MiB/s, 232 objects/s recovering 2024-08-17T18:25:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:29 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:25:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:29 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:25:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:29 smithi136 ceph-mon[109861]: osdmap e1758: 8 total, 8 up, 8 in 2024-08-17T18:25:30.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:29 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 3594/46893 objects degraded (7.664%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:30.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:29 smithi136 ceph-mon[115078]: pgmap v3497: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 1 active+clean+scrubbing, 3 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 976 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 419 op/s; 3594/46893 objects degraded (7.664%); 5108/46893 objects misplaced (10.893%); 13 MiB/s, 232 objects/s recovering 2024-08-17T18:25:30.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:29 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:25:30.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:29 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:25:30.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:29 smithi136 ceph-mon[115078]: osdmap e1758: 8 total, 8 up, 8 in 2024-08-17T18:25:30.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:29 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 3594/46893 objects degraded (7.664%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:29 smithi165 ceph-mon[105517]: pgmap v3497: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+degraded+remapped, 1 active+clean+scrubbing, 3 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 976 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 419 op/s; 3594/46893 objects degraded (7.664%); 5108/46893 objects misplaced (10.893%); 13 MiB/s, 232 objects/s recovering 2024-08-17T18:25:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:29 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:25:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:29 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:25:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:29 smithi165 ceph-mon[105517]: osdmap e1758: 8 total, 8 up, 8 in 2024-08-17T18:25:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:29 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 3594/46893 objects degraded (7.664%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:30.269 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:30.292 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:30.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:30.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:30.394 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:30.418 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:30.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:30.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:30.520 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:30.552 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:30.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:30.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:30.654 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:30.681 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:30.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:30.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:30.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:30.807 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:30.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:30.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:30.909 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:30.943 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:30.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:30.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:31.045 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:31.070 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:31.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:31.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:31.172 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:31.195 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:31.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:31.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:31.297 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:31.324 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:31.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:31.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:31.426 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:31.447 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:31.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:31.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:31.549 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:31.570 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:31.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:31.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:31.673 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:31.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:31.696 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:31.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:31.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:31.818 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:31.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:31.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:31.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:31.942 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:31.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:31.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:32.044 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:32.079 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:32.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:32.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:32.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:32.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:31 smithi136 ceph-mon[109861]: pgmap v3499: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+clean+scrubbing, 3 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 887 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 488 op/s; 3407/42648 objects degraded (7.989%); 4215/42648 objects misplaced (9.883%); 9.7 MiB/s, 185 objects/s recovering 2024-08-17T18:25:32.499 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:31 smithi136 ceph-mon[109861]: osdmap e1759: 8 total, 8 up, 8 in 2024-08-17T18:25:32.500 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:31 smithi136 ceph-mon[115078]: pgmap v3499: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+clean+scrubbing, 3 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 887 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 488 op/s; 3407/42648 objects degraded (7.989%); 4215/42648 objects misplaced (9.883%); 9.7 MiB/s, 185 objects/s recovering 2024-08-17T18:25:32.500 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:31 smithi136 ceph-mon[115078]: osdmap e1759: 8 total, 8 up, 8 in 2024-08-17T18:25:32.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:32.501 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:32.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:32.502 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:31 smithi165 ceph-mon[105517]: pgmap v3499: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+clean+scrubbing, 3 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 887 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 488 op/s; 3407/42648 objects degraded (7.989%); 4215/42648 objects misplaced (9.883%); 9.7 MiB/s, 185 objects/s recovering 2024-08-17T18:25:32.502 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:31 smithi165 ceph-mon[105517]: osdmap e1759: 8 total, 8 up, 8 in 2024-08-17T18:25:32.603 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:32.632 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:32.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:32.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:32.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:32.778 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:32.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:32.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:32.880 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:32.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:32.920 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:32.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:33.021 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:33.059 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:33.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:33.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:33.161 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:33.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:32 smithi136 ceph-mon[109861]: osdmap e1760: 8 total, 8 up, 8 in 2024-08-17T18:25:33.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:32 smithi136 ceph-mon[115078]: osdmap e1760: 8 total, 8 up, 8 in 2024-08-17T18:25:33.201 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:33.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:33.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:33.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:32 smithi165 ceph-mon[105517]: osdmap e1760: 8 total, 8 up, 8 in 2024-08-17T18:25:33.302 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:33.325 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:33.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:33.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:33.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:33.457 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:33.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:33.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:33.559 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:33.585 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:33.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:33.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:33.687 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:33.708 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:33.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:33.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:33.810 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:33.833 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:33.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:33.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:33.935 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:33.961 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:33.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:33.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:34.063 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:34.095 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:34.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:34.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:34.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:33 smithi136 ceph-mon[109861]: pgmap v3502: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 825 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 471 op/s; 3407/39648 objects degraded (8.593%); 3328/39648 objects misplaced (8.394%); 12 MiB/s, 192 objects/s recovering 2024-08-17T18:25:34.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:33 smithi136 ceph-mon[115078]: pgmap v3502: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 825 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 471 op/s; 3407/39648 objects degraded (8.593%); 3328/39648 objects misplaced (8.394%); 12 MiB/s, 192 objects/s recovering 2024-08-17T18:25:34.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:34.239 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:34.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:34.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:34.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:33 smithi165 ceph-mon[105517]: pgmap v3502: 99 pgs: 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 825 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 471 op/s; 3407/39648 objects degraded (8.593%); 3328/39648 objects misplaced (8.394%); 12 MiB/s, 192 objects/s recovering 2024-08-17T18:25:34.341 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:34.388 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:34.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:34.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:34.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:34.521 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:34.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:34.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:34.623 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:34.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:34.662 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:34.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:34.764 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:34.791 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:34.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:34.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:34.893 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:34.928 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:34.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:34.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:35.030 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:35.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:35.071 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:35.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:35.172 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:35.194 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:35.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:35.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:35.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:34 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 3407/39648 objects degraded (8.593%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:35.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:34 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 3407/39648 objects degraded (8.593%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:34 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 3407/39648 objects degraded (8.593%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:35.296 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:35.345 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:35.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:35.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:35.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:35.476 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:35.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:35.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:35.578 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:35.613 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:35.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:35.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:35.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:35.749 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:35.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:35.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:35.851 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:35.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:35.891 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:35.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:35.992 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:36.038 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:36.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:36.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:36.139 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:36.166 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:36.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:36.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:36.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:35 smithi136 ceph-mon[109861]: pgmap v3503: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 3 active+remapped+backfill_wait, 92 active+clean; 804 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 457 op/s; 3048/38643 objects degraded (7.888%); 3328/38643 objects misplaced (8.612%); 14 MiB/s, 223 objects/s recovering 2024-08-17T18:25:36.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:35 smithi136 ceph-mon[115078]: pgmap v3503: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 3 active+remapped+backfill_wait, 92 active+clean; 804 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 457 op/s; 3048/38643 objects degraded (7.888%); 3328/38643 objects misplaced (8.612%); 14 MiB/s, 223 objects/s recovering 2024-08-17T18:25:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:35 smithi165 ceph-mon[105517]: pgmap v3503: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+remapped, 1 active+recovering+degraded+remapped, 3 active+remapped+backfill_wait, 92 active+clean; 804 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 457 op/s; 3048/38643 objects degraded (7.888%); 3328/38643 objects misplaced (8.612%); 14 MiB/s, 223 objects/s recovering 2024-08-17T18:25:36.268 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:36.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:36.304 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:36.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:36.405 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:36.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:36.435 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:36.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:36.536 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:36.557 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:36.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:36.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:36.659 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:36.689 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:36.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:36.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:36.791 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:36.833 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:36.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:36.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:36.935 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:36.960 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:36.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:36.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:37.062 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:37.088 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:37.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:37.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:37.119 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":3496,"stamp":"2024-08-17T18:25:26.625745+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1050667426,"num_objects":16049,"num_object_clones":0,"num_object_copies":48147,"num_objects_missing_on_primary":1214,"num_objects_missing":1214,"num_objects_degraded":4003,"num_objects_misplaced":5108,"num_objects_unfound":0,"num_objects_dirty":16049,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":211288,"num_write_kb":7271485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27476,"num_bytes_recovered":1555058270,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113668,"ondisk_log_size":113668,"up":297,"acting":295,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":316,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11091288,"kb_used_data":3213704,"kb_used_omap":474,"kb_used_meta":7877029,"kb_avail":738869928,"statfs":{"total":767960285184,"available":756602806272,"internally_reserved":0,"allocated":3290832896,"data_stored":3251201514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":485693,"internal_metadata":8066078403},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,0,1,1,1,3,2,8],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-262668288,"num_objects":-4008,"num_object_clones":0,"num_object_copies":-12024,"num_objects_missing_on_primary":-295,"num_objects_missing":-295,"num_objects_degraded":-1420,"num_objects_misplaced":-2417,"num_objects_unfound":0,"num_objects_dirty":-4008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4008,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2704,"num_bytes_recovered":161349632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001997"},"pg_stats":[{"pgid":"58.f","version":"1757'13431","reported_seq":19628,"reported_epoch":1757,"state":"active+remapped+backfilling","last_fresh":"2024-08-17T18:25:25.737324+0000","last_change":"2024-08-17T18:25:25.411524+0000","last_active":"2024-08-17T18:25:25.737324+0000","last_peered":"2024-08-17T18:25:25.737324+0000","last_clean":"2024-08-17T18:21:40.729417+0000","last_became_active":"2024-08-17T18:22:55.487609+0000","last_became_peered":"2024-08-17T18:22:55.487609+0000","last_unstale":"2024-08-17T18:25:25.737324+0000","last_undegraded":"2024-08-17T18:25:25.737324+0000","last_fullsized":"2024-08-17T18:25:25.737324+0000","mapping_epoch":1728,"log_start":"1719'6432","ondisk_log_start":"1719'6432","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":6999,"log_dups_size":0,"ondisk_log_size":6999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61407232,"num_objects":937,"num_object_clones":0,"num_object_copies":2811,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":887,"num_objects_unfound":0,"num_objects_dirty":937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13431,"num_write_kb":459776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3029,"num_bytes_recovered":182452224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,0],"avail_no_missing":["1","0","6"],"object_location_counts":[{"shards":"0,1,6","objects":937}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.e","version":"1757'13113","reported_seq":26078,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:25.735154+0000","last_change":"2024-08-17T18:25:14.864202+0000","last_active":"2024-08-17T18:25:25.735154+0000","last_peered":"2024-08-17T18:25:25.735154+0000","last_clean":"2024-08-17T18:25:25.735154+0000","last_became_active":"2024-08-17T18:25:14.817285+0000","last_became_peered":"2024-08-17T18:25:14.817285+0000","last_unstale":"2024-08-17T18:25:25.735154+0000","last_undegraded":"2024-08-17T18:25:25.735154+0000","last_fullsized":"2024-08-17T18:25:25.735154+0000","mapping_epoch":1753,"log_start":"1719'6096","ondisk_log_start":"1719'6096","created":1701,"last_epoch_clean":1754,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":7017,"log_dups_size":0,"ondisk_log_size":7017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:30.449203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63504384,"num_objects":969,"num_object_clones":0,"num_object_copies":2907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13113,"num_write_kb":450624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6414,"num_bytes_recovered":420347904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.d","version":"1757'13072","reported_seq":13261,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:25.691897+0000","last_change":"2024-08-17T18:25:14.863876+0000","last_active":"2024-08-17T18:25:25.691897+0000","last_peered":"2024-08-17T18:25:25.691897+0000","last_clean":"2024-08-17T18:25:25.691897+0000","last_became_active":"2024-08-17T18:25:14.816225+0000","last_became_peered":"2024-08-17T18:25:14.816225+0000","last_unstale":"2024-08-17T18:25:25.691897+0000","last_undegraded":"2024-08-17T18:25:25.691897+0000","last_fullsized":"2024-08-17T18:25:25.691897+0000","mapping_epoch":1753,"log_start":"1719'6061","ondisk_log_start":"1719'6061","created":1701,"last_epoch_clean":1754,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":7011,"log_dups_size":0,"ondisk_log_size":7011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:29:27.087141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64421912,"num_objects":984,"num_object_clones":0,"num_object_copies":2952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":984,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13072,"num_write_kb":449729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1703936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.c","version":"1757'12964","reported_seq":18943,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:23.026976+0000","last_change":"2024-08-17T18:25:15.870840+0000","last_active":"2024-08-17T18:25:23.026976+0000","last_peered":"2024-08-17T18:25:23.026976+0000","last_clean":"2024-08-17T18:25:23.026976+0000","last_became_active":"2024-08-17T18:25:15.822167+0000","last_became_peered":"2024-08-17T18:25:15.822167+0000","last_unstale":"2024-08-17T18:25:23.026976+0000","last_undegraded":"2024-08-17T18:25:23.026976+0000","last_fullsized":"2024-08-17T18:25:23.026976+0000","mapping_epoch":1754,"log_start":"1719'6224","ondisk_log_start":"1719'6224","created":1701,"last_epoch_clean":1755,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":6740,"log_dups_size":0,"ondisk_log_size":6740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:06:40.523482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68157440,"num_objects":1040,"num_object_clones":0,"num_object_copies":3120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12964,"num_write_kb":448128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2918,"num_bytes_recovered":122421248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"58.b","version":"1757'13105","reported_seq":19155,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:25.670873+0000","last_change":"2024-08-17T18:23:41.403273+0000","last_active":"2024-08-17T18:25:25.670873+0000","last_peered":"2024-08-17T18:25:25.670873+0000","last_clean":"2024-08-17T18:25:25.670873+0000","last_became_active":"2024-08-17T18:23:41.381137+0000","last_became_peered":"2024-08-17T18:23:41.381137+0000","last_unstale":"2024-08-17T18:25:25.670873+0000","last_undegraded":"2024-08-17T18:25:25.670873+0000","last_fullsized":"2024-08-17T18:25:25.670873+0000","mapping_epoch":1738,"log_start":"1719'6052","ondisk_log_start":"1719'6052","created":1701,"last_epoch_clean":1739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":7053,"log_dups_size":0,"ondisk_log_size":7053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:24:41.670310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13105,"num_write_kb":450816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2954,"num_bytes_recovered":177405952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.a","version":"1757'13270","reported_seq":19409,"reported_epoch":1757,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:25:25.542909+0000","last_change":"2024-08-17T18:24:22.100095+0000","last_active":"2024-08-17T18:25:25.542909+0000","last_peered":"2024-08-17T18:25:25.542909+0000","last_clean":"2024-08-17T18:21:40.754759+0000","last_became_active":"2024-08-17T18:24:00.862633+0000","last_became_peered":"2024-08-17T18:24:00.862633+0000","last_unstale":"2024-08-17T18:25:25.542909+0000","last_undegraded":"2024-08-17T18:25:25.542909+0000","last_fullsized":"2024-08-17T18:25:25.542909+0000","mapping_epoch":1741,"log_start":"1720'6798","ondisk_log_start":"1720'6798","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":6472,"log_dups_size":0,"ondisk_log_size":6472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66322432,"num_objects":1012,"num_object_clones":0,"num_object_copies":3036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1012,"num_objects_unfound":0,"num_objects_dirty":1012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13270,"num_write_kb":457024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2998,"num_bytes_recovered":180158464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[3,7,6],"avail_no_missing":["3","6","7"],"object_location_counts":[{"shards":"3,6,7","objects":1012}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"58.9","version":"1757'13585","reported_seq":28258,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:25.398259+0000","last_change":"2024-08-17T18:25:22.900981+0000","last_active":"2024-08-17T18:25:25.398259+0000","last_peered":"2024-08-17T18:25:25.398259+0000","last_clean":"2024-08-17T18:25:25.398259+0000","last_became_active":"2024-08-17T18:25:22.849930+0000","last_became_peered":"2024-08-17T18:25:22.849930+0000","last_unstale":"2024-08-17T18:25:25.398259+0000","last_undegraded":"2024-08-17T18:25:25.398259+0000","last_fullsized":"2024-08-17T18:25:25.398259+0000","mapping_epoch":1756,"log_start":"1721'7122","ondisk_log_start":"1721'7122","created":1701,"last_epoch_clean":1757,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":6463,"log_dups_size":0,"ondisk_log_size":6463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:03.127833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69009408,"num_objects":1053,"num_object_clones":0,"num_object_copies":3159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13585,"num_write_kb":468416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7257,"num_bytes_recovered":352911360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"58.8","version":"1757'13090","reported_seq":13215,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:25.699301+0000","last_change":"2024-08-17T18:20:42.999325+0000","last_active":"2024-08-17T18:25:25.699301+0000","last_peered":"2024-08-17T18:25:25.699301+0000","last_clean":"2024-08-17T18:25:25.699301+0000","last_became_active":"2024-08-17T18:20:42.998979+0000","last_became_peered":"2024-08-17T18:20:42.998979+0000","last_unstale":"2024-08-17T18:25:25.699301+0000","last_undegraded":"2024-08-17T18:25:25.699301+0000","last_fullsized":"2024-08-17T18:25:25.699301+0000","mapping_epoch":1701,"log_start":"1719'6014","ondisk_log_start":"1719'6014","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":7076,"log_dups_size":0,"ondisk_log_size":7076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:31:41.072870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63963136,"num_objects":976,"num_object_clones":0,"num_object_copies":2928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13090,"num_write_kb":450112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.7","version":"1757'13362","reported_seq":13500,"reported_epoch":1757,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:25:23.008273+0000","last_change":"2024-08-17T18:23:59.761537+0000","last_active":"2024-08-17T18:25:23.008273+0000","last_peered":"2024-08-17T18:25:23.008273+0000","last_clean":"2024-08-17T18:21:40.759597+0000","last_became_active":"2024-08-17T18:22:38.009595+0000","last_became_peered":"2024-08-17T18:22:38.009595+0000","last_unstale":"2024-08-17T18:25:23.008273+0000","last_undegraded":"2024-08-17T18:22:37.853963+0000","last_fullsized":"2024-08-17T18:22:37.829715+0000","mapping_epoch":1725,"log_start":"1720'6643","ondisk_log_start":"1720'6643","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":6719,"log_dups_size":0,"ondisk_log_size":6719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68157440,"num_objects":1040,"num_object_clones":0,"num_object_copies":3120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13362,"num_write_kb":460864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":-1718}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"58.6","version":"1757'13378","reported_seq":13503,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:25.474227+0000","last_change":"2024-08-17T18:20:43.000233+0000","last_active":"2024-08-17T18:25:25.474227+0000","last_peered":"2024-08-17T18:25:25.474227+0000","last_clean":"2024-08-17T18:25:25.474227+0000","last_became_active":"2024-08-17T18:20:42.999927+0000","last_became_peered":"2024-08-17T18:20:42.999927+0000","last_unstale":"2024-08-17T18:25:25.474227+0000","last_undegraded":"2024-08-17T18:25:25.474227+0000","last_fullsized":"2024-08-17T18:25:25.474227+0000","mapping_epoch":1701,"log_start":"1720'6867","ondisk_log_start":"1720'6867","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":6511,"log_dups_size":0,"ondisk_log_size":6511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:45:46.168515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63307776,"num_objects":966,"num_object_clones":0,"num_object_copies":2898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13378,"num_write_kb":459008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"58.5","version":"1757'13165","reported_seq":13246,"reported_epoch":1757,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:25:24.988561+0000","last_change":"2024-08-17T18:24:00.911084+0000","last_active":"2024-08-17T18:25:24.988561+0000","last_peered":"2024-08-17T18:25:24.988561+0000","last_clean":"2024-08-17T18:23:58.662864+0000","last_became_active":"2024-08-17T18:24:00.868411+0000","last_became_peered":"2024-08-17T18:24:00.868411+0000","last_unstale":"2024-08-17T18:25:24.988561+0000","last_undegraded":"2024-08-17T18:25:24.988561+0000","last_fullsized":"2024-08-17T18:25:24.988561+0000","mapping_epoch":1741,"log_start":"1719'6300","ondisk_log_start":"1719'6300","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":6865,"log_dups_size":0,"ondisk_log_size":6865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:57:41.403163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63111168,"num_objects":963,"num_object_clones":0,"num_object_copies":2889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":963,"num_objects_unfound":0,"num_objects_dirty":963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13165,"num_write_kb":452096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,2],"avail_no_missing":["4","1","2"],"object_location_counts":[{"shards":"1,2,4","objects":963}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"58.4","version":"1757'13540","reported_seq":13675,"reported_epoch":1757,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:25:25.500369+0000","last_change":"2024-08-17T18:23:59.750271+0000","last_active":"2024-08-17T18:25:25.500369+0000","last_peered":"2024-08-17T18:25:25.500369+0000","last_clean":"2024-08-17T18:23:35.298503+0000","last_became_active":"2024-08-17T18:23:37.612484+0000","last_became_peered":"2024-08-17T18:23:37.612484+0000","last_unstale":"2024-08-17T18:25:25.500369+0000","last_undegraded":"2024-08-17T18:25:25.500369+0000","last_fullsized":"2024-08-17T18:25:25.500369+0000","mapping_epoch":1734,"log_start":"1721'7105","ondisk_log_start":"1721'7105","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":6435,"log_dups_size":0,"ondisk_log_size":6435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67633152,"num_objects":1032,"num_object_clones":0,"num_object_copies":3096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1032,"num_objects_unfound":0,"num_objects_dirty":1032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13540,"num_write_kb":466304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[3,5,2],"avail_no_missing":["3","2","5"],"object_location_counts":[{"shards":"2,3,5","objects":1032}],"blocked_by":[],"up_primary":6,"acting_primary":3,"purged_snaps":[]},{"pgid":"58.3","version":"1757'13280","reported_seq":13405,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:25.454336+0000","last_change":"2024-08-17T18:20:43.426768+0000","last_active":"2024-08-17T18:25:25.454336+0000","last_peered":"2024-08-17T18:25:25.454336+0000","last_clean":"2024-08-17T18:25:25.454336+0000","last_became_active":"2024-08-17T18:20:43.426272+0000","last_became_peered":"2024-08-17T18:20:43.426272+0000","last_unstale":"2024-08-17T18:25:25.454336+0000","last_undegraded":"2024-08-17T18:25:25.454336+0000","last_fullsized":"2024-08-17T18:25:25.454336+0000","mapping_epoch":1701,"log_start":"1720'6818","ondisk_log_start":"1720'6818","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":6462,"log_dups_size":0,"ondisk_log_size":6462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:29:57.872272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63438848,"num_objects":968,"num_object_clones":0,"num_object_copies":2904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13280,"num_write_kb":455936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"58.2","version":"1756'12885","reported_seq":12951,"reported_epoch":1756,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.330131+0000","last_change":"2024-08-17T18:20:43.001236+0000","last_active":"2024-08-17T18:25:22.330131+0000","last_peered":"2024-08-17T18:25:22.330131+0000","last_clean":"2024-08-17T18:25:22.330131+0000","last_became_active":"2024-08-17T18:20:43.000940+0000","last_became_peered":"2024-08-17T18:20:43.000940+0000","last_unstale":"2024-08-17T18:25:22.330131+0000","last_undegraded":"2024-08-17T18:25:22.330131+0000","last_fullsized":"2024-08-17T18:25:22.330131+0000","mapping_epoch":1701,"log_start":"1709'2800","ondisk_log_start":"1709'2800","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:58.629399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70975488,"num_objects":1083,"num_object_clones":0,"num_object_copies":3249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12885,"num_write_kb":446976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"58.1","version":"1757'12897","reported_seq":16574,"reported_epoch":1757,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-17T18:25:25.270251+0000","last_change":"2024-08-17T18:24:02.926951+0000","last_active":"2024-08-17T18:25:25.270251+0000","last_peered":"2024-08-17T18:25:25.270251+0000","last_clean":"2024-08-17T18:21:40.735087+0000","last_became_active":"2024-08-17T18:24:02.024403+0000","last_became_peered":"2024-08-17T18:24:02.024403+0000","last_unstale":"2024-08-17T18:25:25.270251+0000","last_undegraded":"2024-08-17T18:24:01.983818+0000","last_fullsized":"2024-08-17T18:25:25.270251+0000","mapping_epoch":1741,"log_start":"1718'5703","ondisk_log_start":"1718'5703","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":7194,"log_dups_size":0,"ondisk_log_size":7194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:53:15.530966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66125824,"num_objects":1009,"num_object_clones":0,"num_object_copies":3027,"num_objects_missing_on_primary":1214,"num_objects_missing":1214,"num_objects_degraded":1009,"num_objects_misplaced":1214,"num_objects_unfound":0,"num_objects_dirty":1009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12897,"num_write_kb":444992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1776,"num_bytes_recovered":116326400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,6,7","objects":14},{"shards":"1,7","objects":995}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"58.0","version":"1757'13068","reported_seq":13135,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:25.146648+0000","last_change":"2024-08-17T18:20:43.737940+0000","last_active":"2024-08-17T18:25:25.146648+0000","last_peered":"2024-08-17T18:25:25.146648+0000","last_clean":"2024-08-17T18:25:25.146648+0000","last_became_active":"2024-08-17T18:20:43.737219+0000","last_became_peered":"2024-08-17T18:20:43.737219+0000","last_unstale":"2024-08-17T18:25:25.146648+0000","last_undegraded":"2024-08-17T18:25:25.146648+0000","last_fullsized":"2024-08-17T18:25:25.146648+0000","mapping_epoch":1701,"log_start":"1719'6200","ondisk_log_start":"1719'6200","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":6868,"log_dups_size":0,"ondisk_log_size":6868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:28:39.197553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66060288,"num_objects":1008,"num_object_clones":0,"num_object_copies":3024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13068,"num_write_kb":450432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":50,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.846484+0000","last_change":"2024-08-17T18:23:59.758838+0000","last_active":"2024-08-17T18:25:22.846484+0000","last_peered":"2024-08-17T18:25:22.846484+0000","last_clean":"2024-08-17T18:25:22.846484+0000","last_became_active":"2024-08-17T18:23:59.748445+0000","last_became_peered":"2024-08-17T18:23:59.748445+0000","last_unstale":"2024-08-17T18:25:22.846484+0000","last_undegraded":"2024-08-17T18:25:22.846484+0000","last_fullsized":"2024-08-17T18:25:22.846484+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:08:27.581912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1821,"reported_epoch":1756,"state":"active+clean","last_fresh":"2024-08-17T18:25:21.845888+0000","last_change":"2024-08-17T18:23:36.324372+0000","last_active":"2024-08-17T18:25:21.845888+0000","last_peered":"2024-08-17T18:25:21.845888+0000","last_clean":"2024-08-17T18:25:21.845888+0000","last_became_active":"2024-08-17T18:23:36.323893+0000","last_became_peered":"2024-08-17T18:23:36.323893+0000","last_unstale":"2024-08-17T18:25:21.845888+0000","last_undegraded":"2024-08-17T18:25:21.845888+0000","last_fullsized":"2024-08-17T18:25:21.845888+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:10:52.829170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":209,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.844966+0000","last_change":"2024-08-17T18:23:36.319358+0000","last_active":"2024-08-17T18:25:22.844966+0000","last_peered":"2024-08-17T18:25:22.844966+0000","last_clean":"2024-08-17T18:25:22.844966+0000","last_became_active":"2024-08-17T18:23:36.318521+0000","last_became_peered":"2024-08-17T18:23:36.318521+0000","last_unstale":"2024-08-17T18:25:22.844966+0000","last_undegraded":"2024-08-17T18:25:22.844966+0000","last_fullsized":"2024-08-17T18:25:22.844966+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:56:48.984477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":201,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.844249+0000","last_change":"2024-08-17T18:23:59.741806+0000","last_active":"2024-08-17T18:25:22.844249+0000","last_peered":"2024-08-17T18:25:22.844249+0000","last_clean":"2024-08-17T18:25:22.844249+0000","last_became_active":"2024-08-17T18:23:59.741479+0000","last_became_peered":"2024-08-17T18:23:59.741479+0000","last_unstale":"2024-08-17T18:25:22.844249+0000","last_undegraded":"2024-08-17T18:25:22.844249+0000","last_fullsized":"2024-08-17T18:25:22.844249+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:02:39.072277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":329,"reported_epoch":1755,"state":"active+clean","last_fresh":"2024-08-17T18:25:15.889304+0000","last_change":"2024-08-17T18:22:36.836343+0000","last_active":"2024-08-17T18:25:15.889304+0000","last_peered":"2024-08-17T18:25:15.889304+0000","last_clean":"2024-08-17T18:25:15.889304+0000","last_became_active":"2024-08-17T18:22:36.835580+0000","last_became_peered":"2024-08-17T18:22:36.835580+0000","last_unstale":"2024-08-17T18:25:15.889304+0000","last_undegraded":"2024-08-17T18:25:15.889304+0000","last_fullsized":"2024-08-17T18:25:15.889304+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:05:51.737218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":201,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.844474+0000","last_change":"2024-08-17T18:23:59.742425+0000","last_active":"2024-08-17T18:25:22.844474+0000","last_peered":"2024-08-17T18:25:22.844474+0000","last_clean":"2024-08-17T18:25:22.844474+0000","last_became_active":"2024-08-17T18:23:59.742155+0000","last_became_peered":"2024-08-17T18:23:59.742155+0000","last_unstale":"2024-08-17T18:25:22.844474+0000","last_undegraded":"2024-08-17T18:25:22.844474+0000","last_fullsized":"2024-08-17T18:25:22.844474+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:11:42.833289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":340,"reported_epoch":1755,"state":"active+clean","last_fresh":"2024-08-17T18:25:15.889232+0000","last_change":"2024-08-17T18:20:32.094881+0000","last_active":"2024-08-17T18:25:15.889232+0000","last_peered":"2024-08-17T18:25:15.889232+0000","last_clean":"2024-08-17T18:25:15.889232+0000","last_became_active":"2024-08-17T18:20:32.077429+0000","last_became_peered":"2024-08-17T18:20:32.077429+0000","last_unstale":"2024-08-17T18:25:15.889232+0000","last_undegraded":"2024-08-17T18:25:15.889232+0000","last_fullsized":"2024-08-17T18:25:15.889232+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:08:26.290851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1826,"reported_epoch":1756,"state":"active+clean","last_fresh":"2024-08-17T18:25:21.846254+0000","last_change":"2024-08-17T18:25:14.026295+0000","last_active":"2024-08-17T18:25:21.846254+0000","last_peered":"2024-08-17T18:25:21.846254+0000","last_clean":"2024-08-17T18:25:21.846254+0000","last_became_active":"2024-08-17T18:25:14.025999+0000","last_became_peered":"2024-08-17T18:25:14.025999+0000","last_unstale":"2024-08-17T18:25:21.846254+0000","last_undegraded":"2024-08-17T18:25:21.846254+0000","last_fullsized":"2024-08-17T18:25:21.846254+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:55:16.975299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":44,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.846177+0000","last_change":"2024-08-17T18:23:59.745317+0000","last_active":"2024-08-17T18:25:22.846177+0000","last_peered":"2024-08-17T18:25:22.846177+0000","last_clean":"2024-08-17T18:25:22.846177+0000","last_became_active":"2024-08-17T18:23:59.743236+0000","last_became_peered":"2024-08-17T18:23:59.743236+0000","last_unstale":"2024-08-17T18:25:22.846177+0000","last_undegraded":"2024-08-17T18:25:22.846177+0000","last_fullsized":"2024-08-17T18:25:22.846177+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:11.253490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":70,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.848238+0000","last_change":"2024-08-17T18:22:53.938704+0000","last_active":"2024-08-17T18:25:22.848238+0000","last_peered":"2024-08-17T18:25:22.848238+0000","last_clean":"2024-08-17T18:25:22.848238+0000","last_became_active":"2024-08-17T18:22:53.938406+0000","last_became_peered":"2024-08-17T18:22:53.938406+0000","last_unstale":"2024-08-17T18:25:22.848238+0000","last_undegraded":"2024-08-17T18:25:22.848238+0000","last_fullsized":"2024-08-17T18:25:22.848238+0000","mapping_epoch":1727,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:58:21.566014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1803,"reported_epoch":1756,"state":"active+clean","last_fresh":"2024-08-17T18:25:21.846128+0000","last_change":"2024-08-17T18:20:31.942863+0000","last_active":"2024-08-17T18:25:21.846128+0000","last_peered":"2024-08-17T18:25:21.846128+0000","last_clean":"2024-08-17T18:25:21.846128+0000","last_became_active":"2024-08-17T18:20:31.941281+0000","last_became_peered":"2024-08-17T18:20:31.941281+0000","last_unstale":"2024-08-17T18:25:21.846128+0000","last_undegraded":"2024-08-17T18:25:21.846128+0000","last_fullsized":"2024-08-17T18:25:21.846128+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:48:05.556975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":201,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.844853+0000","last_change":"2024-08-17T18:20:31.933142+0000","last_active":"2024-08-17T18:25:22.844853+0000","last_peered":"2024-08-17T18:25:22.844853+0000","last_clean":"2024-08-17T18:25:22.844853+0000","last_became_active":"2024-08-17T18:20:31.932798+0000","last_became_peered":"2024-08-17T18:20:31.932798+0000","last_unstale":"2024-08-17T18:25:22.844853+0000","last_undegraded":"2024-08-17T18:25:22.844853+0000","last_fullsized":"2024-08-17T18:25:22.844853+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:32:51.059352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":136,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.848763+0000","last_change":"2024-08-17T18:20:31.937627+0000","last_active":"2024-08-17T18:25:22.848763+0000","last_peered":"2024-08-17T18:25:22.848763+0000","last_clean":"2024-08-17T18:25:22.848763+0000","last_became_active":"2024-08-17T18:20:31.937361+0000","last_became_peered":"2024-08-17T18:20:31.937361+0000","last_unstale":"2024-08-17T18:25:22.848763+0000","last_undegraded":"2024-08-17T18:25:22.848763+0000","last_fullsized":"2024-08-17T18:25:22.848763+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:01:26.309656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":44,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.846120+0000","last_change":"2024-08-17T18:23:59.743052+0000","last_active":"2024-08-17T18:25:22.846120+0000","last_peered":"2024-08-17T18:25:22.846120+0000","last_clean":"2024-08-17T18:25:22.846120+0000","last_became_active":"2024-08-17T18:23:59.742158+0000","last_became_peered":"2024-08-17T18:23:59.742158+0000","last_unstale":"2024-08-17T18:25:22.846120+0000","last_undegraded":"2024-08-17T18:25:22.846120+0000","last_fullsized":"2024-08-17T18:25:22.846120+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:34:34.249644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":222,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.844089+0000","last_change":"2024-08-17T18:25:14.176708+0000","last_active":"2024-08-17T18:25:22.844089+0000","last_peered":"2024-08-17T18:25:22.844089+0000","last_clean":"2024-08-17T18:25:22.844089+0000","last_became_active":"2024-08-17T18:25:14.176239+0000","last_became_peered":"2024-08-17T18:25:14.176239+0000","last_unstale":"2024-08-17T18:25:22.844089+0000","last_undegraded":"2024-08-17T18:25:22.844089+0000","last_fullsized":"2024-08-17T18:25:22.844089+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:08:21.826456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1819,"reported_epoch":1756,"state":"active+clean","last_fresh":"2024-08-17T18:25:21.856196+0000","last_change":"2024-08-17T18:22:36.867005+0000","last_active":"2024-08-17T18:25:21.856196+0000","last_peered":"2024-08-17T18:25:21.856196+0000","last_clean":"2024-08-17T18:25:21.856196+0000","last_became_active":"2024-08-17T18:22:36.866297+0000","last_became_peered":"2024-08-17T18:22:36.866297+0000","last_unstale":"2024-08-17T18:25:21.856196+0000","last_undegraded":"2024-08-17T18:25:21.856196+0000","last_fullsized":"2024-08-17T18:25:21.856196+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:09:18.460006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":224,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.844194+0000","last_change":"2024-08-17T18:25:14.035767+0000","last_active":"2024-08-17T18:25:22.844194+0000","last_peered":"2024-08-17T18:25:22.844194+0000","last_clean":"2024-08-17T18:25:22.844194+0000","last_became_active":"2024-08-17T18:25:14.034592+0000","last_became_peered":"2024-08-17T18:25:14.034592+0000","last_unstale":"2024-08-17T18:25:22.844194+0000","last_undegraded":"2024-08-17T18:25:22.844194+0000","last_fullsized":"2024-08-17T18:25:22.844194+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:22:00.711481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":44,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.846584+0000","last_change":"2024-08-17T18:23:59.744879+0000","last_active":"2024-08-17T18:25:22.846584+0000","last_peered":"2024-08-17T18:25:22.846584+0000","last_clean":"2024-08-17T18:25:22.846584+0000","last_became_active":"2024-08-17T18:23:59.743333+0000","last_became_peered":"2024-08-17T18:23:59.743333+0000","last_unstale":"2024-08-17T18:25:22.846584+0000","last_undegraded":"2024-08-17T18:25:22.846584+0000","last_fullsized":"2024-08-17T18:25:22.846584+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:02:01.546706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":74,"reported_epoch":1755,"state":"active+clean","last_fresh":"2024-08-17T18:25:15.888972+0000","last_change":"2024-08-17T18:22:36.836745+0000","last_active":"2024-08-17T18:25:15.888972+0000","last_peered":"2024-08-17T18:25:15.888972+0000","last_clean":"2024-08-17T18:25:15.888972+0000","last_became_active":"2024-08-17T18:22:36.835161+0000","last_became_peered":"2024-08-17T18:22:36.835161+0000","last_unstale":"2024-08-17T18:25:15.888972+0000","last_undegraded":"2024-08-17T18:25:15.888972+0000","last_fullsized":"2024-08-17T18:25:15.888972+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:29:43.303633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":399,"reported_epoch":1755,"state":"active+clean","last_fresh":"2024-08-17T18:25:15.889613+0000","last_change":"2024-08-17T18:25:14.027677+0000","last_active":"2024-08-17T18:25:15.889613+0000","last_peered":"2024-08-17T18:25:15.889613+0000","last_clean":"2024-08-17T18:25:15.889613+0000","last_became_active":"2024-08-17T18:25:14.027309+0000","last_became_peered":"2024-08-17T18:25:14.027309+0000","last_unstale":"2024-08-17T18:25:15.889613+0000","last_undegraded":"2024-08-17T18:25:15.889613+0000","last_fullsized":"2024-08-17T18:25:15.889613+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:15:02.617263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":200,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.844749+0000","last_change":"2024-08-17T18:18:29.928751+0000","last_active":"2024-08-17T18:25:22.844749+0000","last_peered":"2024-08-17T18:25:22.844749+0000","last_clean":"2024-08-17T18:25:22.844749+0000","last_became_active":"2024-08-17T18:18:29.898102+0000","last_became_peered":"2024-08-17T18:18:29.898102+0000","last_unstale":"2024-08-17T18:25:22.844749+0000","last_undegraded":"2024-08-17T18:25:22.844749+0000","last_fullsized":"2024-08-17T18:25:22.844749+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:21.098266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":20,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.848243+0000","last_change":"2024-08-17T18:25:14.030155+0000","last_active":"2024-08-17T18:25:22.848243+0000","last_peered":"2024-08-17T18:25:22.848243+0000","last_clean":"2024-08-17T18:25:22.848243+0000","last_became_active":"2024-08-17T18:25:14.029876+0000","last_became_peered":"2024-08-17T18:25:14.029876+0000","last_unstale":"2024-08-17T18:25:22.848243+0000","last_undegraded":"2024-08-17T18:25:22.848243+0000","last_fullsized":"2024-08-17T18:25:22.848243+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:04:34.619858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1803,"reported_epoch":1756,"state":"active+clean","last_fresh":"2024-08-17T18:25:21.846438+0000","last_change":"2024-08-17T18:22:36.838096+0000","last_active":"2024-08-17T18:25:21.846438+0000","last_peered":"2024-08-17T18:25:21.846438+0000","last_clean":"2024-08-17T18:25:21.846438+0000","last_became_active":"2024-08-17T18:22:36.835757+0000","last_became_peered":"2024-08-17T18:22:36.835757+0000","last_unstale":"2024-08-17T18:25:21.846438+0000","last_undegraded":"2024-08-17T18:25:21.846438+0000","last_fullsized":"2024-08-17T18:25:21.846438+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:55:31.016446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1794,"reported_epoch":1756,"state":"active+clean","last_fresh":"2024-08-17T18:25:21.846263+0000","last_change":"2024-08-17T18:25:14.026273+0000","last_active":"2024-08-17T18:25:21.846263+0000","last_peered":"2024-08-17T18:25:21.846263+0000","last_clean":"2024-08-17T18:25:21.846263+0000","last_became_active":"2024-08-17T18:25:14.025964+0000","last_became_peered":"2024-08-17T18:25:14.025964+0000","last_unstale":"2024-08-17T18:25:21.846263+0000","last_undegraded":"2024-08-17T18:25:21.846263+0000","last_fullsized":"2024-08-17T18:25:21.846263+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:43.150679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"347'149","reported_seq":1023,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.844728+0000","last_change":"2024-08-17T18:20:33.927014+0000","last_active":"2024-08-17T18:25:22.844728+0000","last_peered":"2024-08-17T18:25:22.844728+0000","last_clean":"2024-08-17T18:25:22.844728+0000","last_became_active":"2024-08-17T18:20:33.926619+0000","last_became_peered":"2024-08-17T18:20:33.926619+0000","last_unstale":"2024-08-17T18:25:22.844728+0000","last_undegraded":"2024-08-17T18:25:22.844728+0000","last_fullsized":"2024-08-17T18:25:22.844728+0000","mapping_epoch":1697,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1698,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:57.271286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:57.271286+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:35:26.778777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1781,"reported_epoch":1756,"state":"active+clean","last_fresh":"2024-08-17T18:25:21.846768+0000","last_change":"2024-08-17T18:22:36.838651+0000","last_active":"2024-08-17T18:25:21.846768+0000","last_peered":"2024-08-17T18:25:21.846768+0000","last_clean":"2024-08-17T18:25:21.846768+0000","last_became_active":"2024-08-17T18:22:36.838262+0000","last_became_peered":"2024-08-17T18:22:36.838262+0000","last_unstale":"2024-08-17T18:25:21.846768+0000","last_undegraded":"2024-08-17T18:25:21.846768+0000","last_fullsized":"2024-08-17T18:25:21.846768+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:24:25.119257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1762,"reported_epoch":1756,"state":"active+clean","last_fresh":"2024-08-17T18:25:21.856132+0000","last_change":"2024-08-17T18:22:36.866053+0000","last_active":"2024-08-17T18:25:21.856132+0000","last_peered":"2024-08-17T18:25:21.856132+0000","last_clean":"2024-08-17T18:25:21.856132+0000","last_became_active":"2024-08-17T18:22:36.865187+0000","last_became_peered":"2024-08-17T18:22:36.865187+0000","last_unstale":"2024-08-17T18:25:21.856132+0000","last_undegraded":"2024-08-17T18:25:21.856132+0000","last_fullsized":"2024-08-17T18:25:21.856132+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:45:34.232504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":48,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.846016+0000","last_change":"2024-08-17T18:23:59.750520+0000","last_active":"2024-08-17T18:25:22.846016+0000","last_peered":"2024-08-17T18:25:22.846016+0000","last_clean":"2024-08-17T18:25:22.846016+0000","last_became_active":"2024-08-17T18:23:59.742332+0000","last_became_peered":"2024-08-17T18:23:59.742332+0000","last_unstale":"2024-08-17T18:25:22.846016+0000","last_undegraded":"2024-08-17T18:25:22.846016+0000","last_fullsized":"2024-08-17T18:25:22.846016+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:39:39.325194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"347'149","reported_seq":1069,"reported_epoch":1756,"state":"active+clean","last_fresh":"2024-08-17T18:25:21.846523+0000","last_change":"2024-08-17T18:23:39.360135+0000","last_active":"2024-08-17T18:25:21.846523+0000","last_peered":"2024-08-17T18:25:21.846523+0000","last_clean":"2024-08-17T18:25:21.846523+0000","last_became_active":"2024-08-17T18:23:39.359652+0000","last_became_peered":"2024-08-17T18:23:39.359652+0000","last_unstale":"2024-08-17T18:25:21.846523+0000","last_undegraded":"2024-08-17T18:25:21.846523+0000","last_fullsized":"2024-08-17T18:25:21.846523+0000","mapping_epoch":1736,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1737,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:53.309517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:53.309517+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:20:52.590505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":209,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.844199+0000","last_change":"2024-08-17T18:23:59.746854+0000","last_active":"2024-08-17T18:25:22.844199+0000","last_peered":"2024-08-17T18:25:22.844199+0000","last_clean":"2024-08-17T18:25:22.844199+0000","last_became_active":"2024-08-17T18:23:59.746190+0000","last_became_peered":"2024-08-17T18:23:59.746190+0000","last_unstale":"2024-08-17T18:25:22.844199+0000","last_undegraded":"2024-08-17T18:25:22.844199+0000","last_fullsized":"2024-08-17T18:25:22.844199+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:01:53.108037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.849133+0000","last_change":"2024-08-17T18:25:14.025153+0000","last_active":"2024-08-17T18:25:22.849133+0000","last_peered":"2024-08-17T18:25:22.849133+0000","last_clean":"2024-08-17T18:25:22.849133+0000","last_became_active":"2024-08-17T18:25:14.024700+0000","last_became_peered":"2024-08-17T18:25:14.024700+0000","last_unstale":"2024-08-17T18:25:22.849133+0000","last_undegraded":"2024-08-17T18:25:22.849133+0000","last_fullsized":"2024-08-17T18:25:22.849133+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:50:25.014563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":211,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.845622+0000","last_change":"2024-08-17T18:23:59.772682+0000","last_active":"2024-08-17T18:25:22.845622+0000","last_peered":"2024-08-17T18:25:22.845622+0000","last_clean":"2024-08-17T18:25:22.845622+0000","last_became_active":"2024-08-17T18:23:59.748920+0000","last_became_peered":"2024-08-17T18:23:59.748920+0000","last_unstale":"2024-08-17T18:25:22.845622+0000","last_undegraded":"2024-08-17T18:25:22.845622+0000","last_fullsized":"2024-08-17T18:25:22.845622+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:07:31.660610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"347'149","reported_seq":1060,"reported_epoch":1756,"state":"active+clean","last_fresh":"2024-08-17T18:25:21.856056+0000","last_change":"2024-08-17T18:24:02.793338+0000","last_active":"2024-08-17T18:25:21.856056+0000","last_peered":"2024-08-17T18:25:21.856056+0000","last_clean":"2024-08-17T18:25:21.856056+0000","last_became_active":"2024-08-17T18:24:02.792982+0000","last_became_peered":"2024-08-17T18:24:02.792982+0000","last_unstale":"2024-08-17T18:25:21.856056+0000","last_undegraded":"2024-08-17T18:25:21.856056+0000","last_fullsized":"2024-08-17T18:25:21.856056+0000","mapping_epoch":1743,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1744,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:52.474544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:52.474544+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:37:26.909105+0000","scrub_duration":0.001886326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":44,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.846359+0000","last_change":"2024-08-17T18:23:59.741315+0000","last_active":"2024-08-17T18:25:22.846359+0000","last_peered":"2024-08-17T18:25:22.846359+0000","last_clean":"2024-08-17T18:25:22.846359+0000","last_became_active":"2024-08-17T18:23:59.741020+0000","last_became_peered":"2024-08-17T18:23:59.741020+0000","last_unstale":"2024-08-17T18:25:22.846359+0000","last_undegraded":"2024-08-17T18:25:22.846359+0000","last_fullsized":"2024-08-17T18:25:22.846359+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:46:47.093299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":170,"reported_epoch":1755,"state":"active+clean","last_fresh":"2024-08-17T18:25:15.813330+0000","last_change":"2024-08-17T18:23:59.740824+0000","last_active":"2024-08-17T18:25:15.813330+0000","last_peered":"2024-08-17T18:25:15.813330+0000","last_clean":"2024-08-17T18:25:15.813330+0000","last_became_active":"2024-08-17T18:23:59.740478+0000","last_became_peered":"2024-08-17T18:23:59.740478+0000","last_unstale":"2024-08-17T18:25:15.813330+0000","last_undegraded":"2024-08-17T18:25:15.813330+0000","last_fullsized":"2024-08-17T18:25:15.813330+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:44:30.520216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":136,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.848407+0000","last_change":"2024-08-17T18:20:32.012874+0000","last_active":"2024-08-17T18:25:22.848407+0000","last_peered":"2024-08-17T18:25:22.848407+0000","last_clean":"2024-08-17T18:25:22.848407+0000","last_became_active":"2024-08-17T18:20:32.012553+0000","last_became_peered":"2024-08-17T18:20:32.012553+0000","last_unstale":"2024-08-17T18:25:22.848407+0000","last_undegraded":"2024-08-17T18:25:22.848407+0000","last_fullsized":"2024-08-17T18:25:22.848407+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:14:17.188745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"347'149","reported_seq":1026,"reported_epoch":1756,"state":"active+clean","last_fresh":"2024-08-17T18:25:21.856057+0000","last_change":"2024-08-17T18:22:36.866341+0000","last_active":"2024-08-17T18:25:21.856057+0000","last_peered":"2024-08-17T18:25:21.856057+0000","last_clean":"2024-08-17T18:25:21.856057+0000","last_became_active":"2024-08-17T18:22:36.865561+0000","last_became_peered":"2024-08-17T18:22:36.865561+0000","last_unstale":"2024-08-17T18:25:21.856057+0000","last_undegraded":"2024-08-17T18:25:21.856057+0000","last_fullsized":"2024-08-17T18:25:21.856057+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:49.461698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:49.461698+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:22:21.423162+0000","scrub_duration":0.0013792520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":20,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.848335+0000","last_change":"2024-08-17T18:25:14.025883+0000","last_active":"2024-08-17T18:25:22.848335+0000","last_peered":"2024-08-17T18:25:22.848335+0000","last_clean":"2024-08-17T18:25:22.848335+0000","last_became_active":"2024-08-17T18:25:14.025558+0000","last_became_peered":"2024-08-17T18:25:14.025558+0000","last_unstale":"2024-08-17T18:25:22.848335+0000","last_undegraded":"2024-08-17T18:25:22.848335+0000","last_fullsized":"2024-08-17T18:25:22.848335+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:28:59.733420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":136,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.848527+0000","last_change":"2024-08-17T18:20:32.014875+0000","last_active":"2024-08-17T18:25:22.848527+0000","last_peered":"2024-08-17T18:25:22.848527+0000","last_clean":"2024-08-17T18:25:22.848527+0000","last_became_active":"2024-08-17T18:20:32.014631+0000","last_became_peered":"2024-08-17T18:20:32.014631+0000","last_unstale":"2024-08-17T18:25:22.848527+0000","last_undegraded":"2024-08-17T18:25:22.848527+0000","last_fullsized":"2024-08-17T18:25:22.848527+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:00:50.978635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":329,"reported_epoch":1755,"state":"active+clean","last_fresh":"2024-08-17T18:25:15.889377+0000","last_change":"2024-08-17T18:18:29.735362+0000","last_active":"2024-08-17T18:25:15.889377+0000","last_peered":"2024-08-17T18:25:15.889377+0000","last_clean":"2024-08-17T18:25:15.889377+0000","last_became_active":"2024-08-17T18:18:29.734752+0000","last_became_peered":"2024-08-17T18:18:29.734752+0000","last_unstale":"2024-08-17T18:25:15.889377+0000","last_undegraded":"2024-08-17T18:25:15.889377+0000","last_fullsized":"2024-08-17T18:25:15.889377+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:25.709698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"347'149","reported_seq":1069,"reported_epoch":1755,"state":"active+clean","last_fresh":"2024-08-17T18:25:15.889061+0000","last_change":"2024-08-17T18:24:02.792784+0000","last_active":"2024-08-17T18:25:15.889061+0000","last_peered":"2024-08-17T18:25:15.889061+0000","last_clean":"2024-08-17T18:25:15.889061+0000","last_became_active":"2024-08-17T18:24:02.792286+0000","last_became_peered":"2024-08-17T18:24:02.792286+0000","last_unstale":"2024-08-17T18:25:15.889061+0000","last_undegraded":"2024-08-17T18:25:15.889061+0000","last_fullsized":"2024-08-17T18:25:15.889061+0000","mapping_epoch":1743,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1744,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:00.316884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:00.316884+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:26:57.982520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":20,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.848192+0000","last_change":"2024-08-17T18:25:14.031207+0000","last_active":"2024-08-17T18:25:22.848192+0000","last_peered":"2024-08-17T18:25:22.848192+0000","last_clean":"2024-08-17T18:25:22.848192+0000","last_became_active":"2024-08-17T18:25:14.030916+0000","last_became_peered":"2024-08-17T18:25:14.030916+0000","last_unstale":"2024-08-17T18:25:22.848192+0000","last_undegraded":"2024-08-17T18:25:22.848192+0000","last_fullsized":"2024-08-17T18:25:22.848192+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:37:34.050072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1817,"reported_epoch":1756,"state":"active+clean","last_fresh":"2024-08-17T18:25:21.846343+0000","last_change":"2024-08-17T18:23:59.741594+0000","last_active":"2024-08-17T18:25:21.846343+0000","last_peered":"2024-08-17T18:25:21.846343+0000","last_clean":"2024-08-17T18:25:21.846343+0000","last_became_active":"2024-08-17T18:23:59.741207+0000","last_became_peered":"2024-08-17T18:23:59.741207+0000","last_unstale":"2024-08-17T18:25:21.846343+0000","last_undegraded":"2024-08-17T18:25:21.846343+0000","last_fullsized":"2024-08-17T18:25:21.846343+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:02:52.545413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":398,"reported_epoch":1755,"state":"active+clean","last_fresh":"2024-08-17T18:25:15.889350+0000","last_change":"2024-08-17T18:22:36.836104+0000","last_active":"2024-08-17T18:25:15.889350+0000","last_peered":"2024-08-17T18:25:15.889350+0000","last_clean":"2024-08-17T18:25:15.889350+0000","last_became_active":"2024-08-17T18:22:36.835273+0000","last_became_peered":"2024-08-17T18:22:36.835273+0000","last_unstale":"2024-08-17T18:25:15.889350+0000","last_undegraded":"2024-08-17T18:25:15.889350+0000","last_fullsized":"2024-08-17T18:25:15.889350+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:26:23.494575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"347'149","reported_seq":836,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.846516+0000","last_change":"2024-08-17T18:24:02.792482+0000","last_active":"2024-08-17T18:25:22.846516+0000","last_peered":"2024-08-17T18:25:22.846516+0000","last_clean":"2024-08-17T18:25:22.846516+0000","last_became_active":"2024-08-17T18:24:02.792032+0000","last_became_peered":"2024-08-17T18:24:02.792032+0000","last_unstale":"2024-08-17T18:25:22.846516+0000","last_undegraded":"2024-08-17T18:25:22.846516+0000","last_fullsized":"2024-08-17T18:25:22.846516+0000","mapping_epoch":1743,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1744,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:02.363531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:02.363531+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:08:22.251652+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":20,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.848363+0000","last_change":"2024-08-17T18:25:14.176621+0000","last_active":"2024-08-17T18:25:22.848363+0000","last_peered":"2024-08-17T18:25:22.848363+0000","last_clean":"2024-08-17T18:25:22.848363+0000","last_became_active":"2024-08-17T18:25:14.176321+0000","last_became_peered":"2024-08-17T18:25:14.176321+0000","last_unstale":"2024-08-17T18:25:22.848363+0000","last_undegraded":"2024-08-17T18:25:22.848363+0000","last_fullsized":"2024-08-17T18:25:22.848363+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:40:55.297225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":147,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.848860+0000","last_change":"2024-08-17T18:22:36.834643+0000","last_active":"2024-08-17T18:25:22.848860+0000","last_peered":"2024-08-17T18:25:22.848860+0000","last_clean":"2024-08-17T18:25:22.848860+0000","last_became_active":"2024-08-17T18:22:36.834279+0000","last_became_peered":"2024-08-17T18:22:36.834279+0000","last_unstale":"2024-08-17T18:25:22.848860+0000","last_undegraded":"2024-08-17T18:25:22.848860+0000","last_fullsized":"2024-08-17T18:25:22.848860+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:29:36.002584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2817,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.848292+0000","last_change":"2024-08-17T18:18:29.742742+0000","last_active":"2024-08-17T18:25:22.848292+0000","last_peered":"2024-08-17T18:25:22.848292+0000","last_clean":"2024-08-17T18:25:22.848292+0000","last_became_active":"2024-08-17T18:18:29.742393+0000","last_became_peered":"2024-08-17T18:18:29.742393+0000","last_unstale":"2024-08-17T18:25:22.848292+0000","last_undegraded":"2024-08-17T18:25:22.848292+0000","last_fullsized":"2024-08-17T18:25:22.848292+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:57:32.858608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"347'149","reported_seq":824,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.848942+0000","last_change":"2024-08-17T18:23:39.370457+0000","last_active":"2024-08-17T18:25:22.848942+0000","last_peered":"2024-08-17T18:25:22.848942+0000","last_clean":"2024-08-17T18:25:22.848942+0000","last_became_active":"2024-08-17T18:23:39.370144+0000","last_became_peered":"2024-08-17T18:23:39.370144+0000","last_unstale":"2024-08-17T18:25:22.848942+0000","last_undegraded":"2024-08-17T18:25:22.848942+0000","last_fullsized":"2024-08-17T18:25:22.848942+0000","mapping_epoch":1736,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1737,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:20:35.854154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:20:35.854154+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:39:35.866331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":44,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.845922+0000","last_change":"2024-08-17T18:23:59.742951+0000","last_active":"2024-08-17T18:25:22.845922+0000","last_peered":"2024-08-17T18:25:22.845922+0000","last_clean":"2024-08-17T18:25:22.845922+0000","last_became_active":"2024-08-17T18:23:59.742510+0000","last_became_peered":"2024-08-17T18:23:59.742510+0000","last_unstale":"2024-08-17T18:25:22.845922+0000","last_undegraded":"2024-08-17T18:25:22.845922+0000","last_fullsized":"2024-08-17T18:25:22.845922+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:19:12.943501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":44,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.846597+0000","last_change":"2024-08-17T18:23:59.740786+0000","last_active":"2024-08-17T18:25:22.846597+0000","last_peered":"2024-08-17T18:25:22.846597+0000","last_clean":"2024-08-17T18:25:22.846597+0000","last_became_active":"2024-08-17T18:23:59.740340+0000","last_became_peered":"2024-08-17T18:23:59.740340+0000","last_unstale":"2024-08-17T18:25:22.846597+0000","last_undegraded":"2024-08-17T18:25:22.846597+0000","last_fullsized":"2024-08-17T18:25:22.846597+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:52:27.108557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.847744+0000","last_change":"2024-08-17T18:25:14.176308+0000","last_active":"2024-08-17T18:25:22.847744+0000","last_peered":"2024-08-17T18:25:22.847744+0000","last_clean":"2024-08-17T18:25:22.847744+0000","last_became_active":"2024-08-17T18:25:14.175851+0000","last_became_peered":"2024-08-17T18:25:14.175851+0000","last_unstale":"2024-08-17T18:25:22.847744+0000","last_undegraded":"2024-08-17T18:25:22.847744+0000","last_fullsized":"2024-08-17T18:25:22.847744+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:10:25.195682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":34,"reported_epoch":1756,"state":"active+clean","last_fresh":"2024-08-17T18:25:21.855824+0000","last_change":"2024-08-17T18:23:36.318955+0000","last_active":"2024-08-17T18:25:21.855824+0000","last_peered":"2024-08-17T18:25:21.855824+0000","last_clean":"2024-08-17T18:25:21.855824+0000","last_became_active":"2024-08-17T18:23:36.318684+0000","last_became_peered":"2024-08-17T18:23:36.318684+0000","last_unstale":"2024-08-17T18:25:21.855824+0000","last_undegraded":"2024-08-17T18:25:21.855824+0000","last_fullsized":"2024-08-17T18:25:21.855824+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:47.191868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":206,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.844382+0000","last_change":"2024-08-17T18:22:36.835833+0000","last_active":"2024-08-17T18:25:22.844382+0000","last_peered":"2024-08-17T18:25:22.844382+0000","last_clean":"2024-08-17T18:25:22.844382+0000","last_became_active":"2024-08-17T18:22:36.835514+0000","last_became_peered":"2024-08-17T18:22:36.835514+0000","last_unstale":"2024-08-17T18:25:22.844382+0000","last_undegraded":"2024-08-17T18:25:22.844382+0000","last_fullsized":"2024-08-17T18:25:22.844382+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:43:53.668287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":344,"reported_epoch":1755,"state":"active+clean","last_fresh":"2024-08-17T18:25:15.891237+0000","last_change":"2024-08-17T18:19:26.374702+0000","last_active":"2024-08-17T18:25:15.891237+0000","last_peered":"2024-08-17T18:25:15.891237+0000","last_clean":"2024-08-17T18:25:15.891237+0000","last_became_active":"2024-08-17T18:15:53.304106+0000","last_became_peered":"2024-08-17T18:15:53.304106+0000","last_unstale":"2024-08-17T18:25:15.891237+0000","last_undegraded":"2024-08-17T18:25:15.891237+0000","last_fullsized":"2024-08-17T18:25:15.891237+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:21.409368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":977,"reported_epoch":1757,"state":"active+clean+scrubbing","last_fresh":"2024-08-17T18:25:22.846041+0000","last_change":"2024-08-17T18:25:24.715037+0000","last_active":"2024-08-17T18:25:22.846041+0000","last_peered":"2024-08-17T18:25:22.846041+0000","last_clean":"2024-08-17T18:25:22.846041+0000","last_became_active":"2024-08-17T18:24:02.791616+0000","last_became_peered":"2024-08-17T18:24:02.791616+0000","last_unstale":"2024-08-17T18:25:22.846041+0000","last_undegraded":"2024-08-17T18:25:22.846041+0000","last_fullsized":"2024-08-17T18:25:22.846041+0000","mapping_epoch":1743,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1744,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:20:36.599786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:20:36.599786+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":74,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"347'149","reported_seq":1051,"reported_epoch":1755,"state":"active+clean","last_fresh":"2024-08-17T18:25:15.889175+0000","last_change":"2024-08-17T18:22:36.836656+0000","last_active":"2024-08-17T18:25:15.889175+0000","last_peered":"2024-08-17T18:25:15.889175+0000","last_clean":"2024-08-17T18:25:15.889175+0000","last_became_active":"2024-08-17T18:22:36.835810+0000","last_became_peered":"2024-08-17T18:22:36.835810+0000","last_unstale":"2024-08-17T18:25:15.889175+0000","last_undegraded":"2024-08-17T18:25:15.889175+0000","last_fullsized":"2024-08-17T18:25:15.889175+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:55.255983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:55.255983+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:59:27.835153+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":136,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.849364+0000","last_change":"2024-08-17T18:20:31.939871+0000","last_active":"2024-08-17T18:25:22.849364+0000","last_peered":"2024-08-17T18:25:22.849364+0000","last_clean":"2024-08-17T18:25:22.849364+0000","last_became_active":"2024-08-17T18:20:31.939535+0000","last_became_peered":"2024-08-17T18:20:31.939535+0000","last_unstale":"2024-08-17T18:25:22.849364+0000","last_undegraded":"2024-08-17T18:25:22.849364+0000","last_fullsized":"2024-08-17T18:25:22.849364+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:43:31.630668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":216,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.843991+0000","last_change":"2024-08-17T18:25:14.036186+0000","last_active":"2024-08-17T18:25:22.843991+0000","last_peered":"2024-08-17T18:25:22.843991+0000","last_clean":"2024-08-17T18:25:22.843991+0000","last_became_active":"2024-08-17T18:25:14.035138+0000","last_became_peered":"2024-08-17T18:25:14.035138+0000","last_unstale":"2024-08-17T18:25:22.843991+0000","last_undegraded":"2024-08-17T18:25:22.843991+0000","last_fullsized":"2024-08-17T18:25:22.843991+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:51.645934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1832,"reported_epoch":1756,"state":"active+clean","last_fresh":"2024-08-17T18:25:21.846615+0000","last_change":"2024-08-17T18:25:14.026973+0000","last_active":"2024-08-17T18:25:21.846615+0000","last_peered":"2024-08-17T18:25:21.846615+0000","last_clean":"2024-08-17T18:25:21.846615+0000","last_became_active":"2024-08-17T18:25:14.026478+0000","last_became_peered":"2024-08-17T18:25:14.026478+0000","last_unstale":"2024-08-17T18:25:21.846615+0000","last_undegraded":"2024-08-17T18:25:21.846615+0000","last_fullsized":"2024-08-17T18:25:21.846615+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:31.568544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1786,"reported_epoch":1756,"state":"active+clean","last_fresh":"2024-08-17T18:25:21.846830+0000","last_change":"2024-08-17T18:15:53.244851+0000","last_active":"2024-08-17T18:25:21.846830+0000","last_peered":"2024-08-17T18:25:21.846830+0000","last_clean":"2024-08-17T18:25:21.846830+0000","last_became_active":"2024-08-17T18:15:53.240074+0000","last_became_peered":"2024-08-17T18:15:53.240074+0000","last_unstale":"2024-08-17T18:25:21.846830+0000","last_undegraded":"2024-08-17T18:25:21.846830+0000","last_fullsized":"2024-08-17T18:25:21.846830+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:09:00.168413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"347'149","reported_seq":1037,"reported_epoch":1757,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:25:22.848590+0000","last_change":"2024-08-17T18:24:14.849225+0000","last_active":"2024-08-17T18:25:22.848590+0000","last_peered":"2024-08-17T18:25:22.848590+0000","last_clean":"2024-08-17T18:23:20.705230+0000","last_became_active":"2024-08-17T18:24:14.848918+0000","last_became_peered":"2024-08-17T18:24:14.848918+0000","last_unstale":"2024-08-17T18:25:22.848590+0000","last_undegraded":"2024-08-17T18:24:14.838017+0000","last_fullsized":"2024-08-17T18:24:14.837609+0000","mapping_epoch":1746,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1692,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:59.307356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:59.307356+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[2,3],"avail_no_missing":["2","3","4"],"object_location_counts":[{"shards":"2,3,4","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1790,"reported_epoch":1756,"state":"active+clean","last_fresh":"2024-08-17T18:25:21.846613+0000","last_change":"2024-08-17T18:20:31.942019+0000","last_active":"2024-08-17T18:25:21.846613+0000","last_peered":"2024-08-17T18:25:21.846613+0000","last_clean":"2024-08-17T18:25:21.846613+0000","last_became_active":"2024-08-17T18:20:31.941701+0000","last_became_peered":"2024-08-17T18:20:31.941701+0000","last_unstale":"2024-08-17T18:25:21.846613+0000","last_undegraded":"2024-08-17T18:25:21.846613+0000","last_fullsized":"2024-08-17T18:25:21.846613+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:58.908049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1767,"reported_epoch":1756,"state":"active+clean","last_fresh":"2024-08-17T18:25:21.856652+0000","last_change":"2024-08-17T18:25:14.027871+0000","last_active":"2024-08-17T18:25:21.856652+0000","last_peered":"2024-08-17T18:25:21.856652+0000","last_clean":"2024-08-17T18:25:21.856652+0000","last_became_active":"2024-08-17T18:25:14.027353+0000","last_became_peered":"2024-08-17T18:25:14.027353+0000","last_unstale":"2024-08-17T18:25:21.856652+0000","last_undegraded":"2024-08-17T18:25:21.856652+0000","last_fullsized":"2024-08-17T18:25:21.856652+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:06.568798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"347'149","reported_seq":1012,"reported_epoch":1755,"state":"active+clean","last_fresh":"2024-08-17T18:25:15.812999+0000","last_change":"2024-08-17T18:18:33.757347+0000","last_active":"2024-08-17T18:25:15.812999+0000","last_peered":"2024-08-17T18:25:15.812999+0000","last_clean":"2024-08-17T18:25:15.812999+0000","last_became_active":"2024-08-17T18:18:33.756960+0000","last_became_peered":"2024-08-17T18:18:33.756960+0000","last_unstale":"2024-08-17T18:25:15.812999+0000","last_undegraded":"2024-08-17T18:25:15.812999+0000","last_fullsized":"2024-08-17T18:25:15.812999+0000","mapping_epoch":1670,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1671,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:52.271487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:52.271487+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:37:36.389244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1784,"reported_epoch":1756,"state":"active+clean","last_fresh":"2024-08-17T18:25:21.846293+0000","last_change":"2024-08-17T18:22:36.837889+0000","last_active":"2024-08-17T18:25:21.846293+0000","last_peered":"2024-08-17T18:25:21.846293+0000","last_clean":"2024-08-17T18:25:21.846293+0000","last_became_active":"2024-08-17T18:22:36.837387+0000","last_became_peered":"2024-08-17T18:22:36.837387+0000","last_unstale":"2024-08-17T18:25:21.846293+0000","last_undegraded":"2024-08-17T18:25:21.846293+0000","last_fullsized":"2024-08-17T18:25:21.846293+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:11:29.265925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":232,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.844648+0000","last_change":"2024-08-17T18:23:59.770482+0000","last_active":"2024-08-17T18:25:22.844648+0000","last_peered":"2024-08-17T18:25:22.844648+0000","last_clean":"2024-08-17T18:25:22.844648+0000","last_became_active":"2024-08-17T18:23:59.747059+0000","last_became_peered":"2024-08-17T18:23:59.747059+0000","last_unstale":"2024-08-17T18:25:22.844648+0000","last_undegraded":"2024-08-17T18:25:22.844648+0000","last_fullsized":"2024-08-17T18:25:22.844648+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:04:17.042903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":44,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.846270+0000","last_change":"2024-08-17T18:23:59.745024+0000","last_active":"2024-08-17T18:25:22.846270+0000","last_peered":"2024-08-17T18:25:22.846270+0000","last_clean":"2024-08-17T18:25:22.846270+0000","last_became_active":"2024-08-17T18:23:59.741633+0000","last_became_peered":"2024-08-17T18:23:59.741633+0000","last_unstale":"2024-08-17T18:25:22.846270+0000","last_undegraded":"2024-08-17T18:25:22.846270+0000","last_fullsized":"2024-08-17T18:25:22.846270+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:41.514649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":353,"reported_epoch":1755,"state":"active+clean","last_fresh":"2024-08-17T18:25:15.889324+0000","last_change":"2024-08-17T18:22:36.836017+0000","last_active":"2024-08-17T18:25:15.889324+0000","last_peered":"2024-08-17T18:25:15.889324+0000","last_clean":"2024-08-17T18:25:15.889324+0000","last_became_active":"2024-08-17T18:22:36.835725+0000","last_became_peered":"2024-08-17T18:22:36.835725+0000","last_unstale":"2024-08-17T18:25:15.889324+0000","last_undegraded":"2024-08-17T18:25:15.889324+0000","last_fullsized":"2024-08-17T18:25:15.889324+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:18:07.395241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":34,"reported_epoch":1756,"state":"active+clean","last_fresh":"2024-08-17T18:25:21.855330+0000","last_change":"2024-08-17T18:23:36.324099+0000","last_active":"2024-08-17T18:25:21.855330+0000","last_peered":"2024-08-17T18:25:21.855330+0000","last_clean":"2024-08-17T18:25:21.855330+0000","last_became_active":"2024-08-17T18:23:36.323739+0000","last_became_peered":"2024-08-17T18:23:36.323739+0000","last_unstale":"2024-08-17T18:25:21.855330+0000","last_undegraded":"2024-08-17T18:25:21.855330+0000","last_fullsized":"2024-08-17T18:25:21.855330+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:54:56.319013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":356,"reported_epoch":1755,"state":"active+clean","last_fresh":"2024-08-17T18:25:15.889511+0000","last_change":"2024-08-17T18:22:36.836529+0000","last_active":"2024-08-17T18:25:15.889511+0000","last_peered":"2024-08-17T18:25:15.889511+0000","last_clean":"2024-08-17T18:25:15.889511+0000","last_became_active":"2024-08-17T18:22:36.836020+0000","last_became_peered":"2024-08-17T18:22:36.836020+0000","last_unstale":"2024-08-17T18:25:15.889511+0000","last_undegraded":"2024-08-17T18:25:15.889511+0000","last_fullsized":"2024-08-17T18:25:15.889511+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:22.088303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1841,"reported_epoch":1756,"state":"active+clean","last_fresh":"2024-08-17T18:25:21.846573+0000","last_change":"2024-08-17T18:25:14.040053+0000","last_active":"2024-08-17T18:25:21.846573+0000","last_peered":"2024-08-17T18:25:21.846573+0000","last_clean":"2024-08-17T18:25:21.846573+0000","last_became_active":"2024-08-17T18:25:14.039807+0000","last_became_peered":"2024-08-17T18:25:14.039807+0000","last_unstale":"2024-08-17T18:25:21.846573+0000","last_undegraded":"2024-08-17T18:25:21.846573+0000","last_fullsized":"2024-08-17T18:25:21.846573+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:05:53.486096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":149,"reported_epoch":1755,"state":"active+clean","last_fresh":"2024-08-17T18:25:15.813552+0000","last_change":"2024-08-17T18:20:32.006404+0000","last_active":"2024-08-17T18:25:15.813552+0000","last_peered":"2024-08-17T18:25:15.813552+0000","last_clean":"2024-08-17T18:25:15.813552+0000","last_became_active":"2024-08-17T18:20:31.947053+0000","last_became_peered":"2024-08-17T18:20:31.947053+0000","last_unstale":"2024-08-17T18:25:15.813552+0000","last_undegraded":"2024-08-17T18:25:15.813552+0000","last_fullsized":"2024-08-17T18:25:15.813552+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:21:06.537725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":356,"reported_epoch":1755,"state":"active+clean","last_fresh":"2024-08-17T18:25:15.888968+0000","last_change":"2024-08-17T18:23:36.318964+0000","last_active":"2024-08-17T18:25:15.888968+0000","last_peered":"2024-08-17T18:25:15.888968+0000","last_clean":"2024-08-17T18:25:15.888968+0000","last_became_active":"2024-08-17T18:23:36.318595+0000","last_became_peered":"2024-08-17T18:23:36.318595+0000","last_unstale":"2024-08-17T18:25:15.888968+0000","last_undegraded":"2024-08-17T18:25:15.888968+0000","last_fullsized":"2024-08-17T18:25:15.888968+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:29:16.221103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":175,"reported_epoch":1755,"state":"active+clean","last_fresh":"2024-08-17T18:25:15.813655+0000","last_change":"2024-08-17T18:23:59.752171+0000","last_active":"2024-08-17T18:25:15.813655+0000","last_peered":"2024-08-17T18:25:15.813655+0000","last_clean":"2024-08-17T18:25:15.813655+0000","last_became_active":"2024-08-17T18:23:59.745970+0000","last_became_peered":"2024-08-17T18:23:59.745970+0000","last_unstale":"2024-08-17T18:25:15.813655+0000","last_undegraded":"2024-08-17T18:25:15.813655+0000","last_fullsized":"2024-08-17T18:25:15.813655+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:24:52.861533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1914,"reported_epoch":1756,"state":"active+clean","last_fresh":"2024-08-17T18:25:21.855333+0000","last_change":"2024-08-17T18:24:22.107318+0000","last_active":"2024-08-17T18:25:21.855333+0000","last_peered":"2024-08-17T18:25:21.855333+0000","last_clean":"2024-08-17T18:25:21.855333+0000","last_became_active":"2024-08-17T18:24:14.847882+0000","last_became_peered":"2024-08-17T18:24:14.847882+0000","last_unstale":"2024-08-17T18:25:21.855333+0000","last_undegraded":"2024-08-17T18:25:21.855333+0000","last_fullsized":"2024-08-17T18:25:21.855333+0000","mapping_epoch":1746,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:45:33.740721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":208,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.845725+0000","last_change":"2024-08-17T18:21:12.645070+0000","last_active":"2024-08-17T18:25:22.845725+0000","last_peered":"2024-08-17T18:25:22.845725+0000","last_clean":"2024-08-17T18:25:22.845725+0000","last_became_active":"2024-08-17T18:21:12.644748+0000","last_became_peered":"2024-08-17T18:21:12.644748+0000","last_unstale":"2024-08-17T18:25:22.845725+0000","last_undegraded":"2024-08-17T18:25:22.845725+0000","last_fullsized":"2024-08-17T18:25:22.845725+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:19:08.683431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":514,"reported_epoch":1756,"state":"active+clean","last_fresh":"2024-08-17T18:25:21.846700+0000","last_change":"2024-08-17T18:22:36.838518+0000","last_active":"2024-08-17T18:25:21.846700+0000","last_peered":"2024-08-17T18:25:21.846700+0000","last_clean":"2024-08-17T18:25:21.846700+0000","last_became_active":"2024-08-17T18:22:36.834610+0000","last_became_peered":"2024-08-17T18:22:36.834610+0000","last_unstale":"2024-08-17T18:25:21.846700+0000","last_undegraded":"2024-08-17T18:25:21.846700+0000","last_fullsized":"2024-08-17T18:25:21.846700+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:42:18.798462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":44,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.846206+0000","last_change":"2024-08-17T18:23:59.740450+0000","last_active":"2024-08-17T18:25:22.846206+0000","last_peered":"2024-08-17T18:25:22.846206+0000","last_clean":"2024-08-17T18:25:22.846206+0000","last_became_active":"2024-08-17T18:23:59.740125+0000","last_became_peered":"2024-08-17T18:23:59.740125+0000","last_unstale":"2024-08-17T18:25:22.846206+0000","last_undegraded":"2024-08-17T18:25:22.846206+0000","last_fullsized":"2024-08-17T18:25:22.846206+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:26:38.858210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":44,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.846481+0000","last_change":"2024-08-17T18:23:59.740550+0000","last_active":"2024-08-17T18:25:22.846481+0000","last_peered":"2024-08-17T18:25:22.846481+0000","last_clean":"2024-08-17T18:25:22.846481+0000","last_became_active":"2024-08-17T18:23:59.740226+0000","last_became_peered":"2024-08-17T18:23:59.740226+0000","last_unstale":"2024-08-17T18:25:22.846481+0000","last_undegraded":"2024-08-17T18:25:22.846481+0000","last_fullsized":"2024-08-17T18:25:22.846481+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:35.745256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":346,"reported_epoch":1755,"state":"active+clean","last_fresh":"2024-08-17T18:25:15.889440+0000","last_change":"2024-08-17T18:18:29.900128+0000","last_active":"2024-08-17T18:25:15.889440+0000","last_peered":"2024-08-17T18:25:15.889440+0000","last_clean":"2024-08-17T18:25:15.889440+0000","last_became_active":"2024-08-17T18:18:29.899279+0000","last_became_peered":"2024-08-17T18:18:29.899279+0000","last_unstale":"2024-08-17T18:25:15.889440+0000","last_undegraded":"2024-08-17T18:25:15.889440+0000","last_fullsized":"2024-08-17T18:25:15.889440+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:01:29.045498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":139,"reported_epoch":1755,"state":"active+clean","last_fresh":"2024-08-17T18:25:17.139368+0000","last_change":"2024-08-17T18:15:53.271113+0000","last_active":"2024-08-17T18:25:17.139368+0000","last_peered":"2024-08-17T18:25:17.139368+0000","last_clean":"2024-08-17T18:25:17.139368+0000","last_became_active":"2024-08-17T18:15:53.230983+0000","last_became_peered":"2024-08-17T18:15:53.230983+0000","last_unstale":"2024-08-17T18:25:17.139368+0000","last_undegraded":"2024-08-17T18:25:17.139368+0000","last_fullsized":"2024-08-17T18:25:17.139368+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:00:35.315798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":226,"reported_epoch":1757,"state":"active+clean","last_fresh":"2024-08-17T18:25:22.844996+0000","last_change":"2024-08-17T18:25:14.030279+0000","last_active":"2024-08-17T18:25:22.844996+0000","last_peered":"2024-08-17T18:25:22.844996+0000","last_clean":"2024-08-17T18:25:22.844996+0000","last_became_active":"2024-08-17T18:25:14.027960+0000","last_became_peered":"2024-08-17T18:25:14.027960+0000","last_unstale":"2024-08-17T18:25:22.844996+0000","last_undegraded":"2024-08-17T18:25:22.844996+0000","last_fullsized":"2024-08-17T18:25:22.844996+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:20:46.317663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":58,"num_pg":16,"stat_sum":{"num_bytes":1050017816,"num_objects":16023,"num_object_clones":0,"num_object_copies":48069,"num_objects_missing_on_primary":1214,"num_objects_missing":1214,"num_objects_degraded":4002,"num_objects_misplaced":5108,"num_objects_unfound":0,"num_objects_dirty":16023,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":211205,"num_write_kb":7271233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27375,"num_bytes_recovered":1553858560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3157667840,"data_stored":3157655624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111970,"ondisk_log_size":111970,"up":48,"acting":47,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644037,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":74,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1186397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":32,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":13180,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234912,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1212772,"kb_used_data":355608,"kb_used_omap":1,"kb_used_meta":857150,"kb_avail":92532380,"statfs":{"total":95995035648,"available":94753157120,"internally_reserved":0,"allocated":364142592,"data_stored":359186067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":877722059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1724,"seq":7404523618339,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":483564,"kb_used_data":301348,"kb_used_omap":123,"kb_used_meta":182084,"kb_avail":93261588,"statfs":{"total":95995035648,"available":95499866112,"internally_reserved":0,"allocated":308580352,"data_stored":303621475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":126809,"internal_metadata":186454183},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,1,1,3,2,8],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565792,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1938600,"kb_used_data":661916,"kb_used_omap":72,"kb_used_meta":1276599,"kb_avail":91806552,"statfs":{"total":95995035648,"available":94009909248,"internally_reserved":0,"allocated":677801984,"data_stored":672847124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":1307237870},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":1399,"seq":6008659247393,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1525276,"kb_used_data":564368,"kb_used_omap":131,"kb_used_meta":960764,"kb_avail":92219876,"statfs":{"total":95995035648,"available":94433153024,"internally_reserved":0,"allocated":577912832,"data_stored":572961870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134291,"internal_metadata":983823213},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743187182,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1533728,"kb_used_data":581272,"kb_used_omap":66,"kb_used_meta":952381,"kb_avail":92211424,"statfs":{"total":95995035648,"available":94424498176,"internally_reserved":0,"allocated":595222528,"data_stored":590271149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":975238827},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448821,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1034656,"kb_used_data":132312,"kb_used_omap":75,"kb_used_meta":902260,"kb_avail":92710496,"statfs":{"total":95995035648,"available":94935547904,"internally_reserved":0,"allocated":135487488,"data_stored":130536670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":923914431},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208758185,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1601812,"kb_used_data":221384,"kb_used_omap":1,"kb_used_meta":1380414,"kb_avail":92143340,"statfs":{"total":95995035648,"available":94354780160,"internally_reserved":0,"allocated":226697216,"data_stored":221752774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1413544390},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019822,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1760880,"kb_used_data":395496,"kb_used_omap":1,"kb_used_meta":1365374,"kb_avail":91984272,"statfs":{"total":95995035648,"available":94191894528,"internally_reserved":0,"allocated":404987904,"data_stored":400024385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1398143430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":123797504,"data_stored":123797504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":575475712,"data_stored":575471640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":557912064,"data_stored":557907992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":653594624,"data_stored":653590552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":388300800,"data_stored":388300800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":214695936,"data_stored":214695936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":291635200,"data_stored":291635200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":352256000,"data_stored":352256000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T18:25:37.124 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T18:25:37.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:37.221 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:37.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:37.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:37.323 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:37.376 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:37.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:37.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:37.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:25:37.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:25:37.447 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:37 smithi136 ceph-mon[109861]: pgmap v3504: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 92 active+clean; 655 MiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 661 op/s; 2846/31509 objects degraded (9.032%); 2570/31509 objects misplaced (8.156%); 13 MiB/s, 200 objects/s recovering 2024-08-17T18:25:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:25:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:25:37.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:37 smithi136 ceph-mon[115078]: pgmap v3504: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 92 active+clean; 655 MiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 661 op/s; 2846/31509 objects degraded (9.032%); 2570/31509 objects misplaced (8.156%); 13 MiB/s, 200 objects/s recovering 2024-08-17T18:25:37.479 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:25:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:25:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:37 smithi165 ceph-mon[105517]: pgmap v3504: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 92 active+clean; 655 MiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 661 op/s; 2846/31509 objects degraded (9.032%); 2570/31509 objects misplaced (8.156%); 13 MiB/s, 200 objects/s recovering 2024-08-17T18:25:37.520 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:37.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:37.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:37.622 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:37.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:37.659 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:37.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:37.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:37.793 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:37.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:37.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:37.895 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:37.919 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:37.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:37.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:37.946 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:25:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:25:37] "GET /metrics HTTP/1.1" 200 36711 "" "Prometheus/2.51.0" 2024-08-17T18:25:38.021 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:38.053 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:38.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:38.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:38.155 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:38.184 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:38.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:38.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:38.286 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:38.314 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:38.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:38.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:38.416 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:38.454 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:38.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:38.556 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:38.602 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:38.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:38.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:38.704 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:38.732 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:38.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:38.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:38.834 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:38.839 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:25:38.839 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:25:38.856 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:38.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:38.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:38.958 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:38.982 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:38.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:38.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:39.085 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:39.106 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:39.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:39.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:39.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:39.229 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:39.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:39.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:39.331 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:39.363 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:39.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:39.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:39.465 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:39.507 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:39.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:39.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:39.609 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:39.636 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:39.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:39.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:39.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:39.760 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:39.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:39.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:39.861 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:39.889 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:39.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:39.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:39.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:39 smithi136 ceph-mon[109861]: pgmap v3505: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 3 active+remapped+backfill_wait, 92 active+clean; 613 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 547 op/s; 2579/29493 objects degraded (8.744%); 2570/29493 objects misplaced (8.714%); 13 MiB/s, 205 objects/s recovering 2024-08-17T18:25:39.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:39 smithi136 ceph-mon[115078]: pgmap v3505: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 3 active+remapped+backfill_wait, 92 active+clean; 613 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 547 op/s; 2579/29493 objects degraded (8.744%); 2570/29493 objects misplaced (8.714%); 13 MiB/s, 205 objects/s recovering 2024-08-17T18:25:39.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:39 smithi165 ceph-mon[105517]: pgmap v3505: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 3 active+remapped+backfill_wait, 92 active+clean; 613 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 547 op/s; 2579/29493 objects degraded (8.744%); 2570/29493 objects misplaced (8.714%); 13 MiB/s, 205 objects/s recovering 2024-08-17T18:25:40.016 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:40.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:40.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:40.118 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:40.150 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:40.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:40.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:40.252 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:40.286 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:40.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:40.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:40.388 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:40.417 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:40.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:40.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:40.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:40.543 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:40.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:40.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:40.645 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:40.666 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:40.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:40.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:40.768 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:40.798 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:40.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:40.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:40.901 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:40.937 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:40.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:40.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:40.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:40 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 2579/29493 objects degraded (8.744%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:40.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:40 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 2579/29493 objects degraded (8.744%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:41.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:40 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 2579/29493 objects degraded (8.744%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:41.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:41.066 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:41.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:41.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:41.168 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:41.204 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:41.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:41.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:41.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:41.342 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:41.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:41.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:41.444 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:41.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:41.487 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:41.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:41.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:41.613 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:41.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:41.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:41.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:41.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:41.741 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:41.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:41.842 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:41.871 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:41.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:41.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:41.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:42.001 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:42.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:42.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:42.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:41 smithi165 ceph-mon[105517]: pgmap v3506: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 3 active+remapped+backfill_wait, 92 active+clean; 503 MiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 544 op/s; 2371/24177 objects degraded (9.807%); 1667/24177 objects misplaced (6.895%); 6.7 MiB/s, 107 objects/s recovering 2024-08-17T18:25:42.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:42.133 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:42.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:42.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:42.193 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:41 smithi136 ceph-mon[109861]: pgmap v3506: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 3 active+remapped+backfill_wait, 92 active+clean; 503 MiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 544 op/s; 2371/24177 objects degraded (9.807%); 1667/24177 objects misplaced (6.895%); 6.7 MiB/s, 107 objects/s recovering 2024-08-17T18:25:42.194 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:41 smithi136 ceph-mon[115078]: pgmap v3506: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 3 active+remapped+backfill_wait, 92 active+clean; 503 MiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 544 op/s; 2371/24177 objects degraded (9.807%); 1667/24177 objects misplaced (6.895%); 6.7 MiB/s, 107 objects/s recovering 2024-08-17T18:25:42.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:42.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:42.268 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:42.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:42.369 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:42.394 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:42.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:42.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:42.496 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:42.540 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:42.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:42.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:42.642 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:42.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:42.690 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:42.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:42.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:42.818 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:42.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:42.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:42.921 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:42.968 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:42.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:42.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:43.070 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:43.106 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:43.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:43.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:43.208 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:43.254 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:43.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:43.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:43.357 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:43.387 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:43.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:43.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:43.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:43.519 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:43.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:43.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:43.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:43.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:43.649 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:43.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:43.750 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:43.778 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:43.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:43.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:43.880 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:43.906 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:43.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:43.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:44.008 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:43 smithi165 ceph-mon[105517]: pgmap v3507: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 92 active+clean; 422 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 613 op/s; 2371/20316 objects degraded (11.671%); 1667/20316 objects misplaced (8.205%); 6.1 MiB/s, 97 objects/s recovering 2024-08-17T18:25:44.049 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:44.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:44.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:44.120 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:43 smithi136 ceph-mon[109861]: pgmap v3507: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 92 active+clean; 422 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 613 op/s; 2371/20316 objects degraded (11.671%); 1667/20316 objects misplaced (8.205%); 6.1 MiB/s, 97 objects/s recovering 2024-08-17T18:25:44.120 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:43 smithi136 ceph-mon[115078]: pgmap v3507: 99 pgs: 1 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 92 active+clean; 422 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 613 op/s; 2371/20316 objects degraded (11.671%); 1667/20316 objects misplaced (8.205%); 6.1 MiB/s, 97 objects/s recovering 2024-08-17T18:25:44.151 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:44.194 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:44.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:44.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:44.296 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:44.323 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:44.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:44.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:44.326 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T18:25:44.424 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:44.452 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:44.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:44.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:44.554 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:44.582 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:44.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:44.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:44.684 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:44.726 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:44.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:44.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:44.828 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:44.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:44.870 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:44.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:44.972 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:44.999 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:44.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:45.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:45.100 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:45.157 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:45.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:45.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:45.259 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:45.285 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:45.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:45.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:45.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:45.414 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:45.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:45.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:45.482 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:25:45.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:45.581 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:45.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:45.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:45.682 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:45.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:45.735 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:45.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:45.836 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:45.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:45.883 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:45.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:45 smithi136 ceph-mon[109861]: pgmap v3508: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 3 active+remapped+backfill_wait, 92 active+clean; 394 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 574 op/s; 2191/18954 objects degraded (11.560%); 1667/18954 objects misplaced (8.795%); 6.4 MiB/s, 109 objects/s recovering 2024-08-17T18:25:45.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:45 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 2191/18954 objects degraded (11.560%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:45.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:45 smithi136 ceph-mon[115078]: pgmap v3508: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 3 active+remapped+backfill_wait, 92 active+clean; 394 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 574 op/s; 2191/18954 objects degraded (11.560%); 1667/18954 objects misplaced (8.795%); 6.4 MiB/s, 109 objects/s recovering 2024-08-17T18:25:45.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:45 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 2191/18954 objects degraded (11.560%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:45.984 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:46.009 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:46.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:46.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:46.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:45 smithi165 ceph-mon[105517]: pgmap v3508: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 3 active+remapped+backfill_wait, 92 active+clean; 394 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 574 op/s; 2191/18954 objects degraded (11.560%); 1667/18954 objects misplaced (8.795%); 6.4 MiB/s, 109 objects/s recovering 2024-08-17T18:25:46.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:45 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 2191/18954 objects degraded (11.560%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:46.110 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:46.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:46.151 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:46.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:46.252 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:46.294 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:46.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:46.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:46.396 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:46.437 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:46.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:46.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:46.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:46.575 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:46.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:46.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:46.677 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:46.717 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:46.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:46.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:46.818 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:46.857 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:46.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:46.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:46.960 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:47.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:47.000 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:47.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:47.101 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:47.129 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:47.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:47.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:47.233 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:47.260 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:47.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:47.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:47.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:47.389 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:47.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:47.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:47.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:47.538 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:47.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:47.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:47.640 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:47.664 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:47.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:47.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:47.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:25:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:25:47] "GET /metrics HTTP/1.1" 200 36711 "" "Prometheus/2.51.0" 2024-08-17T18:25:47.766 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:47.792 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:47.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:47.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:47.894 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:47.936 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:47.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:47 smithi165 ceph-mon[105517]: pgmap v3509: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 92 active+clean; 203 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 800 op/s; 1898/9810 objects degraded (19.348%); 684/9810 objects misplaced (6.972%); 6.0 MiB/s, 104 objects/s recovering 2024-08-17T18:25:48.038 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:48.080 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:48.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:48.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:48.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:48.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:47 smithi136 ceph-mon[109861]: pgmap v3509: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 92 active+clean; 203 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 800 op/s; 1898/9810 objects degraded (19.348%); 684/9810 objects misplaced (6.972%); 6.0 MiB/s, 104 objects/s recovering 2024-08-17T18:25:48.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:47 smithi136 ceph-mon[115078]: pgmap v3509: 99 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 92 active+clean; 203 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 800 op/s; 1898/9810 objects degraded (19.348%); 684/9810 objects misplaced (6.972%); 6.0 MiB/s, 104 objects/s recovering 2024-08-17T18:25:48.206 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:48.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:48.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:48.308 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:48.344 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:48.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:48.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:48.446 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:48.479 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:48.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:48.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:48.581 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:48.630 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:48.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:48.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:48.736 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:48.777 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:48.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:48.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:48.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:48.901 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:48.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:48.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:49.003 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:49.025 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:49.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:49.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:49.127 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:49.148 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:49.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:49.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:49.250 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:49.272 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:49.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:49.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:49.374 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:49.396 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:49.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:49.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:49.498 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:49.520 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:49.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:49.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:49.622 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:49.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:49.645 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:49.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:49.746 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:49.770 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:49.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:49.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:49.802 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:25:49.803 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:25:49.872 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:49.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:49.902 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:49.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:50.003 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:49 smithi165 ceph-mon[105517]: pgmap v3510: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 3 active+remapped+backfill_wait, 92 active+clean; 161 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 659 op/s; 1768/7758 objects degraded (22.789%); 684/7758 objects misplaced (8.817%); 4.9 MiB/s, 109 objects/s recovering 2024-08-17T18:25:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:49 smithi165 ceph-mon[105517]: osdmap e1761: 8 total, 8 up, 8 in 2024-08-17T18:25:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:49 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2978866315' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:25:50.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:50.030 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:50.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:50.132 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:50.134 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:49 smithi136 ceph-mon[109861]: pgmap v3510: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 3 active+remapped+backfill_wait, 92 active+clean; 161 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 659 op/s; 1768/7758 objects degraded (22.789%); 684/7758 objects misplaced (8.817%); 4.9 MiB/s, 109 objects/s recovering 2024-08-17T18:25:50.134 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:49 smithi136 ceph-mon[109861]: osdmap e1761: 8 total, 8 up, 8 in 2024-08-17T18:25:50.134 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:49 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2978866315' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:25:50.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:49 smithi136 ceph-mon[115078]: pgmap v3510: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 1 active+clean+scrubbing, 3 active+remapped+backfill_wait, 92 active+clean; 161 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 659 op/s; 1768/7758 objects degraded (22.789%); 684/7758 objects misplaced (8.817%); 4.9 MiB/s, 109 objects/s recovering 2024-08-17T18:25:50.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:49 smithi136 ceph-mon[115078]: osdmap e1761: 8 total, 8 up, 8 in 2024-08-17T18:25:50.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:49 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2978866315' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:25:50.160 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:50.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:50.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:50.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:50.285 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:50.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:50.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:50.327 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Removed 113613 objects 2024-08-17T18:25:50.327 INFO:tasks.radosbench.radosbench.0.smithi136.stdout:Clean up completed and total clean up time :199.095 2024-08-17T18:25:50.386 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:50.408 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:50.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:50.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:50.509 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:50.530 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:50.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:50.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:50.564 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-17T18:25:50.632 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:50.651 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:25:50.651 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:25:50.651 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:25:50.651 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:25:50.651 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:25:50.651 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:25:50.652 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:25:50.652 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:25:50.652 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:25:50.652 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:25:50.652 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:25:50.652 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:25:50.652 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:25:50.652 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:25:50.653 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:25:50.653 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:25:50.654 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:50.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:50.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:50.756 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:50.778 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:50.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:50.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:50.880 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:50.902 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:50.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:50.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:50.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:50 smithi136 ceph-mon[109861]: osdmap e1762: 8 total, 8 up, 8 in 2024-08-17T18:25:50.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:50 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 1768/7758 objects degraded (22.789%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:50.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:50 smithi136 ceph-mon[115078]: osdmap e1762: 8 total, 8 up, 8 in 2024-08-17T18:25:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:50 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 1768/7758 objects degraded (22.789%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:51.003 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:51.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:50 smithi165 ceph-mon[105517]: osdmap e1762: 8 total, 8 up, 8 in 2024-08-17T18:25:51.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:50 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 1768/7758 objects degraded (22.789%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:51.025 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:51.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:51.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:51.129 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:51.151 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:51.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:51.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:51.252 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:51.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:51.274 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:51.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:51.375 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:51.398 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:51.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:51.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:51.500 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:51.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:51.524 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:51.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:51.625 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:51.647 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:51.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:51.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:51.749 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:51.771 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:51.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:51.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:51.874 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:51.895 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:51.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:51.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:51.996 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:52.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:51 smithi165 ceph-mon[105517]: pgmap v3513: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfill_wait, 94 active+clean; 82 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 681 op/s; 1649/3966 objects degraded (41.578%); 194/3966 objects misplaced (4.892%); 5.5 MiB/s, 160 objects/s recovering 2024-08-17T18:25:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:25:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:25:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:25:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:25:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:25:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:25:52.018 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:52.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:52.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:52.119 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:52.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:52.142 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:52.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:52.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:51 smithi136 ceph-mon[109861]: pgmap v3513: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfill_wait, 94 active+clean; 82 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 681 op/s; 1649/3966 objects degraded (41.578%); 194/3966 objects misplaced (4.892%); 5.5 MiB/s, 160 objects/s recovering 2024-08-17T18:25:52.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:25:52.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:25:52.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:25:52.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:25:52.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:25:52.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:25:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:51 smithi136 ceph-mon[115078]: pgmap v3513: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfill_wait, 94 active+clean; 82 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 681 op/s; 1649/3966 objects degraded (41.578%); 194/3966 objects misplaced (4.892%); 5.5 MiB/s, 160 objects/s recovering 2024-08-17T18:25:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:25:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:25:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:25:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:25:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:25:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:25:52.243 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:52.264 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:52.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:52.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:52.365 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:52.387 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:52.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:52.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:52.488 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:52.509 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:52.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:52.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:52.611 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:52.633 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:52.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:52.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:52.735 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:52.757 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:52.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:52.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:52.859 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:52.881 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:52.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:52.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:52.944 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:25:52.983 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:52.989 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:25:53.007 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:53.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:53.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:53.020 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-08-17T18:25:53.021 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-08-17T18:25:53.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:53.120 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:25:53.121 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T18:25:53.130 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:53.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:53.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:53.232 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:53.255 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:53.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:53.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:53.357 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:53.378 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:53.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:53.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:53.480 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:53.501 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:53.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:53.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:53.603 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:53.624 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:53.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:53.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:53.725 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:53.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:53.747 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:53.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:53.776 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:25:53.848 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:53.870 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:53.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:53.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:53.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:53.993 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:53.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:53.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:54.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:53 smithi165 ceph-mon[105517]: pgmap v3514: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 8.9 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 769 op/s; 1649/474 objects degraded (347.890%); 4.5 MiB/s, 134 objects/s recovering 2024-08-17T18:25:54.094 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:54.116 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:54.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:54.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:54.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:53 smithi136 ceph-mon[109861]: pgmap v3514: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 8.9 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 769 op/s; 1649/474 objects degraded (347.890%); 4.5 MiB/s, 134 objects/s recovering 2024-08-17T18:25:54.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:53 smithi136 ceph-mon[115078]: pgmap v3514: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 8.9 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 769 op/s; 1649/474 objects degraded (347.890%); 4.5 MiB/s, 134 objects/s recovering 2024-08-17T18:25:54.218 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:54.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:54.240 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:54.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:54.341 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:54.363 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:54.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:54.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:54.465 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:54.487 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:54.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:54.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:54.589 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:54.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:54.611 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:54.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:54.712 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:54.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:54.735 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:54.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:54.837 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:54.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:54.860 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:54.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:54.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:54.984 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:54.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:54.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:55.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:54 smithi165 ceph-mon[105517]: from='client.47012 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:25:55.085 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:55.107 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:55.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:55.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:55.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:54 smithi136 ceph-mon[109861]: from='client.47012 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:25:55.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:54 smithi136 ceph-mon[115078]: from='client.47012 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:25:55.209 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:55.233 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:55.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:55.335 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:55.357 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:55.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:55.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:55.459 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:55.484 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:55.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:55.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:55.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:55.608 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:55.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:55.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:55.710 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:55.732 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:55.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:55.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:55.833 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:55.855 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:55.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:55.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:55.956 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:55.980 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:55.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:55.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:56.082 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:56.106 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:56.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:56.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:56.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:55 smithi136 ceph-mon[109861]: pgmap v3515: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 2.1 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 402 op/s; 423/147 objects degraded (287.755%); 2.2 MiB/s, 250 objects/s recovering 2024-08-17T18:25:56.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:55 smithi136 ceph-mon[109861]: osdmap e1763: 8 total, 8 up, 8 in 2024-08-17T18:25:56.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:55 smithi136 ceph-mon[109861]: Health check update: Degraded data redundancy: 423/147 objects degraded (287.755%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:56.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:55 smithi136 ceph-mon[115078]: pgmap v3515: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 2.1 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 402 op/s; 423/147 objects degraded (287.755%); 2.2 MiB/s, 250 objects/s recovering 2024-08-17T18:25:56.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:55 smithi136 ceph-mon[115078]: osdmap e1763: 8 total, 8 up, 8 in 2024-08-17T18:25:56.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:55 smithi136 ceph-mon[115078]: Health check update: Degraded data redundancy: 423/147 objects degraded (287.755%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:56.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:56.230 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:56.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:56.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:56.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:55 smithi165 ceph-mon[105517]: pgmap v3515: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 2.1 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 402 op/s; 423/147 objects degraded (287.755%); 2.2 MiB/s, 250 objects/s recovering 2024-08-17T18:25:56.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:55 smithi165 ceph-mon[105517]: osdmap e1763: 8 total, 8 up, 8 in 2024-08-17T18:25:56.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:55 smithi165 ceph-mon[105517]: Health check update: Degraded data redundancy: 423/147 objects degraded (287.755%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T18:25:56.333 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:56.359 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:56.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:56.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:56.460 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:56.484 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:56.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:56.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:56.586 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:56.612 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:56.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:56.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:56.714 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:56.738 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:56.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:56.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:56.781 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":3514,"stamp":"2024-08-17T18:25:52.638005+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9300362,"num_objects":158,"num_object_clones":0,"num_object_copies":474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1649,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":227179,"num_write_kb":7271485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31022,"num_bytes_recovered":1749175902,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116459,"ondisk_log_size":116459,"up":297,"acting":295,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":315,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8111356,"kb_used_data":194392,"kb_used_omap":474,"kb_used_meta":7916261,"kb_avail":741849860,"statfs":{"total":767960285184,"available":759654256640,"internally_reserved":0,"allocated":199057408,"data_stored":159118231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":485693,"internal_metadata":8106251971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-403636248,"num_objects":-6160,"num_object_clones":0,"num_object_copies":-18480,"num_objects_missing_on_primary":-631,"num_objects_missing":-631,"num_objects_degraded":-542,"num_objects_misplaced":-1667,"num_objects_unfound":0,"num_objects_dirty":-6160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6160,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1074,"num_bytes_recovered":37945344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002381"},"pg_stats":[{"pgid":"58.f","version":"1762'14368","reported_seq":22199,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:50.325342+0000","last_change":"2024-08-17T18:25:31.971413+0000","last_active":"2024-08-17T18:25:50.325342+0000","last_peered":"2024-08-17T18:25:50.325342+0000","last_clean":"2024-08-17T18:25:50.325342+0000","last_became_active":"2024-08-17T18:25:31.928297+0000","last_became_peered":"2024-08-17T18:25:31.928297+0000","last_unstale":"2024-08-17T18:25:50.325342+0000","last_undegraded":"2024-08-17T18:25:50.325342+0000","last_fullsized":"2024-08-17T18:25:50.325342+0000","mapping_epoch":1759,"log_start":"1722'7332","ondisk_log_start":"1722'7332","created":1701,"last_epoch_clean":1760,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":7036,"log_dups_size":0,"ondisk_log_size":7036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:29:41.859539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14368,"num_write_kb":459776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3836,"num_bytes_recovered":235339776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.e","version":"1762'14082","reported_seq":27057,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:50.318801+0000","last_change":"2024-08-17T18:25:14.864202+0000","last_active":"2024-08-17T18:25:50.318801+0000","last_peered":"2024-08-17T18:25:50.318801+0000","last_clean":"2024-08-17T18:25:50.318801+0000","last_became_active":"2024-08-17T18:25:14.817285+0000","last_became_peered":"2024-08-17T18:25:14.817285+0000","last_unstale":"2024-08-17T18:25:50.318801+0000","last_undegraded":"2024-08-17T18:25:50.318801+0000","last_fullsized":"2024-08-17T18:25:50.318801+0000","mapping_epoch":1753,"log_start":"1721'7096","ondisk_log_start":"1721'7096","created":1701,"last_epoch_clean":1754,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":6986,"log_dups_size":0,"ondisk_log_size":6986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:30.449203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14082,"num_write_kb":450624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6414,"num_bytes_recovered":420347904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.d","version":"1762'14056","reported_seq":14255,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:50.287688+0000","last_change":"2024-08-17T18:25:14.863876+0000","last_active":"2024-08-17T18:25:50.287688+0000","last_peered":"2024-08-17T18:25:50.287688+0000","last_clean":"2024-08-17T18:25:50.287688+0000","last_became_active":"2024-08-17T18:25:14.816225+0000","last_became_peered":"2024-08-17T18:25:14.816225+0000","last_unstale":"2024-08-17T18:25:50.287688+0000","last_undegraded":"2024-08-17T18:25:50.287688+0000","last_fullsized":"2024-08-17T18:25:50.287688+0000","mapping_epoch":1753,"log_start":"1721'7061","ondisk_log_start":"1721'7061","created":1701,"last_epoch_clean":1754,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":6995,"log_dups_size":0,"ondisk_log_size":6995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:29:27.087141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14056,"num_write_kb":449729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1703936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.c","version":"1760'13959","reported_seq":19945,"reported_epoch":1760,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.170987+0000","last_change":"2024-08-17T18:25:15.870840+0000","last_active":"2024-08-17T18:25:48.170987+0000","last_peered":"2024-08-17T18:25:48.170987+0000","last_clean":"2024-08-17T18:25:48.170987+0000","last_became_active":"2024-08-17T18:25:15.822167+0000","last_became_peered":"2024-08-17T18:25:15.822167+0000","last_unstale":"2024-08-17T18:25:48.170987+0000","last_undegraded":"2024-08-17T18:25:48.170987+0000","last_fullsized":"2024-08-17T18:25:48.170987+0000","mapping_epoch":1754,"log_start":"1723'7224","ondisk_log_start":"1723'7224","created":1701,"last_epoch_clean":1755,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":6735,"log_dups_size":0,"ondisk_log_size":6735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:06:40.523482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2949120,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13959,"num_write_kb":448128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2918,"num_bytes_recovered":122421248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"58.b","version":"1762'14088","reported_seq":20148,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:50.265942+0000","last_change":"2024-08-17T18:23:41.403273+0000","last_active":"2024-08-17T18:25:50.265942+0000","last_peered":"2024-08-17T18:25:50.265942+0000","last_clean":"2024-08-17T18:25:50.265942+0000","last_became_active":"2024-08-17T18:23:41.381137+0000","last_became_peered":"2024-08-17T18:23:41.381137+0000","last_unstale":"2024-08-17T18:25:50.265942+0000","last_undegraded":"2024-08-17T18:25:50.265942+0000","last_fullsized":"2024-08-17T18:25:50.265942+0000","mapping_epoch":1738,"log_start":"1721'7052","ondisk_log_start":"1721'7052","created":1701,"last_epoch_clean":1739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":7036,"log_dups_size":0,"ondisk_log_size":7036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:24:41.670310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14088,"num_write_kb":450816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2954,"num_bytes_recovered":177405952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.a","version":"1762'14282","reported_seq":20553,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:50.326046+0000","last_change":"2024-08-17T18:25:49.830410+0000","last_active":"2024-08-17T18:25:50.326046+0000","last_peered":"2024-08-17T18:25:50.326046+0000","last_clean":"2024-08-17T18:25:50.326046+0000","last_became_active":"2024-08-17T18:25:49.755837+0000","last_became_peered":"2024-08-17T18:25:49.755837+0000","last_unstale":"2024-08-17T18:25:50.326046+0000","last_undegraded":"2024-08-17T18:25:50.326046+0000","last_fullsized":"2024-08-17T18:25:50.326046+0000","mapping_epoch":1761,"log_start":"1726'7398","ondisk_log_start":"1726'7398","created":1701,"last_epoch_clean":1762,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":6884,"log_dups_size":0,"ondisk_log_size":6884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:15:29.876089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14282,"num_write_kb":457024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3053,"num_bytes_recovered":183762944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"58.9","version":"1762'14638","reported_seq":29321,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:50.310393+0000","last_change":"2024-08-17T18:25:22.900981+0000","last_active":"2024-08-17T18:25:50.310393+0000","last_peered":"2024-08-17T18:25:50.310393+0000","last_clean":"2024-08-17T18:25:50.310393+0000","last_became_active":"2024-08-17T18:25:22.849930+0000","last_became_peered":"2024-08-17T18:25:22.849930+0000","last_unstale":"2024-08-17T18:25:50.310393+0000","last_undegraded":"2024-08-17T18:25:50.310393+0000","last_fullsized":"2024-08-17T18:25:50.310393+0000","mapping_epoch":1756,"log_start":"1726'7722","ondisk_log_start":"1726'7722","created":1701,"last_epoch_clean":1757,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":6916,"log_dups_size":0,"ondisk_log_size":6916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:03.127833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14638,"num_write_kb":468416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7257,"num_bytes_recovered":352911360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"58.8","version":"1762'14066","reported_seq":14201,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:50.307188+0000","last_change":"2024-08-17T18:20:42.999325+0000","last_active":"2024-08-17T18:25:50.307188+0000","last_peered":"2024-08-17T18:25:50.307188+0000","last_clean":"2024-08-17T18:25:50.307188+0000","last_became_active":"2024-08-17T18:20:42.998979+0000","last_became_peered":"2024-08-17T18:20:42.998979+0000","last_unstale":"2024-08-17T18:25:50.307188+0000","last_undegraded":"2024-08-17T18:25:50.307188+0000","last_fullsized":"2024-08-17T18:25:50.307188+0000","mapping_epoch":1701,"log_start":"1721'7014","ondisk_log_start":"1721'7014","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":7052,"log_dups_size":0,"ondisk_log_size":7052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:31:41.072870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14066,"num_write_kb":450112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"58.7","version":"1760'14338","reported_seq":17173,"reported_epoch":1760,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T18:25:48.163425+0000","last_change":"2024-08-17T18:25:25.410348+0000","last_active":"2024-08-17T18:25:48.163425+0000","last_peered":"2024-08-17T18:25:48.163425+0000","last_clean":"2024-08-17T18:21:40.759597+0000","last_became_active":"2024-08-17T18:22:38.009595+0000","last_became_peered":"2024-08-17T18:22:38.009595+0000","last_unstale":"2024-08-17T18:25:48.163425+0000","last_undegraded":"2024-08-17T18:22:37.853963+0000","last_fullsized":"2024-08-17T18:22:37.829715+0000","mapping_epoch":1725,"log_start":"1726'7643","ondisk_log_start":"1726'7643","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":6695,"log_dups_size":0,"ondisk_log_size":6695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14338,"num_write_kb":460864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1346,"num_bytes_recovered":64290816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":-1584}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"58.6","version":"1762'14344","reported_seq":14479,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:50.271525+0000","last_change":"2024-08-17T18:20:43.000233+0000","last_active":"2024-08-17T18:25:50.271525+0000","last_peered":"2024-08-17T18:25:50.271525+0000","last_clean":"2024-08-17T18:25:50.271525+0000","last_became_active":"2024-08-17T18:20:42.999927+0000","last_became_peered":"2024-08-17T18:20:42.999927+0000","last_unstale":"2024-08-17T18:25:50.271525+0000","last_undegraded":"2024-08-17T18:25:50.271525+0000","last_fullsized":"2024-08-17T18:25:50.271525+0000","mapping_epoch":1701,"log_start":"1726'7467","ondisk_log_start":"1726'7467","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":6877,"log_dups_size":0,"ondisk_log_size":6877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:45:46.168515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14344,"num_write_kb":459008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"58.5","version":"1762'14116","reported_seq":14368,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:50.152338+0000","last_change":"2024-08-17T18:25:49.807420+0000","last_active":"2024-08-17T18:25:50.152338+0000","last_peered":"2024-08-17T18:25:50.152338+0000","last_clean":"2024-08-17T18:25:50.152338+0000","last_became_active":"2024-08-17T18:25:49.754553+0000","last_became_peered":"2024-08-17T18:25:49.754553+0000","last_unstale":"2024-08-17T18:25:50.152338+0000","last_undegraded":"2024-08-17T18:25:50.152338+0000","last_fullsized":"2024-08-17T18:25:50.152338+0000","mapping_epoch":1761,"log_start":"1721'6900","ondisk_log_start":"1721'6900","created":1701,"last_epoch_clean":1762,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":7216,"log_dups_size":0,"ondisk_log_size":7216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:00:30.479284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":786432,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14116,"num_write_kb":452096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":5177344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"58.4","version":"1762'14572","reported_seq":14717,"reported_epoch":1762,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T18:25:50.326190+0000","last_change":"2024-08-17T18:23:59.750271+0000","last_active":"2024-08-17T18:25:50.326190+0000","last_peered":"2024-08-17T18:25:50.326190+0000","last_clean":"2024-08-17T18:23:35.298503+0000","last_became_active":"2024-08-17T18:23:37.612484+0000","last_became_peered":"2024-08-17T18:23:37.612484+0000","last_unstale":"2024-08-17T18:25:50.326190+0000","last_undegraded":"2024-08-17T18:25:50.326190+0000","last_fullsized":"2024-08-17T18:25:50.326190+0000","mapping_epoch":1734,"log_start":"1726'7705","ondisk_log_start":"1726'7705","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":6867,"log_dups_size":0,"ondisk_log_size":6867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14572,"num_write_kb":466304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[3,5,2],"avail_no_missing":["3","2","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":3,"purged_snaps":[]},{"pgid":"58.3","version":"1762'14248","reported_seq":14383,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:50.307290+0000","last_change":"2024-08-17T18:20:43.426768+0000","last_active":"2024-08-17T18:25:50.307290+0000","last_peered":"2024-08-17T18:25:50.307290+0000","last_clean":"2024-08-17T18:25:50.307290+0000","last_became_active":"2024-08-17T18:20:43.426272+0000","last_became_peered":"2024-08-17T18:20:43.426272+0000","last_unstale":"2024-08-17T18:25:50.307290+0000","last_undegraded":"2024-08-17T18:25:50.307290+0000","last_fullsized":"2024-08-17T18:25:50.307290+0000","mapping_epoch":1701,"log_start":"1726'7418","ondisk_log_start":"1726'7418","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":6830,"log_dups_size":0,"ondisk_log_size":6830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:29:57.872272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14248,"num_write_kb":455936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"58.2","version":"1762'13968","reported_seq":14040,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:50.324895+0000","last_change":"2024-08-17T18:20:43.001236+0000","last_active":"2024-08-17T18:25:50.324895+0000","last_peered":"2024-08-17T18:25:50.324895+0000","last_clean":"2024-08-17T18:25:50.324895+0000","last_became_active":"2024-08-17T18:20:43.000940+0000","last_became_peered":"2024-08-17T18:20:43.000940+0000","last_unstale":"2024-08-17T18:25:50.324895+0000","last_undegraded":"2024-08-17T18:25:50.324895+0000","last_fullsized":"2024-08-17T18:25:50.324895+0000","mapping_epoch":1701,"log_start":"1710'3900","ondisk_log_start":"1710'3900","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:58.629399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13968,"num_write_kb":446976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"58.1","version":"1762'13904","reported_seq":20111,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:50.295495+0000","last_change":"2024-08-17T18:25:49.789563+0000","last_active":"2024-08-17T18:25:50.295495+0000","last_peered":"2024-08-17T18:25:50.295495+0000","last_clean":"2024-08-17T18:25:50.295495+0000","last_became_active":"2024-08-17T18:25:49.753802+0000","last_became_peered":"2024-08-17T18:25:49.753802+0000","last_unstale":"2024-08-17T18:25:50.295495+0000","last_undegraded":"2024-08-17T18:25:50.295495+0000","last_fullsized":"2024-08-17T18:25:50.295495+0000","mapping_epoch":1761,"log_start":"1720'6503","ondisk_log_start":"1720'6503","created":1701,"last_epoch_clean":1762,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":7401,"log_dups_size":0,"ondisk_log_size":7401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:39:51.225753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13904,"num_write_kb":444992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3037,"num_bytes_recovered":184614912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"58.0","version":"1762'14067","reported_seq":14139,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:50.130234+0000","last_change":"2024-08-17T18:20:43.737940+0000","last_active":"2024-08-17T18:25:50.130234+0000","last_peered":"2024-08-17T18:25:50.130234+0000","last_clean":"2024-08-17T18:25:50.130234+0000","last_became_active":"2024-08-17T18:20:43.737219+0000","last_became_peered":"2024-08-17T18:20:43.737219+0000","last_unstale":"2024-08-17T18:25:50.130234+0000","last_undegraded":"2024-08-17T18:25:50.130234+0000","last_fullsized":"2024-08-17T18:25:50.130234+0000","mapping_epoch":1701,"log_start":"1721'6900","ondisk_log_start":"1721'6900","created":1701,"last_epoch_clean":1702,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:20:41.972470+0000","last_clean_scrub_stamp":"2024-08-17T18:20:41.972470+0000","objects_scrubbed":0,"log_size":7167,"log_dups_size":0,"ondisk_log_size":7167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:28:39.197553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":589824,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14067,"num_write_kb":450432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":60,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.753146+0000","last_change":"2024-08-17T18:23:59.758838+0000","last_active":"2024-08-17T18:25:49.753146+0000","last_peered":"2024-08-17T18:25:49.753146+0000","last_clean":"2024-08-17T18:25:49.753146+0000","last_became_active":"2024-08-17T18:23:59.748445+0000","last_became_peered":"2024-08-17T18:23:59.748445+0000","last_unstale":"2024-08-17T18:25:49.753146+0000","last_undegraded":"2024-08-17T18:25:49.753146+0000","last_fullsized":"2024-08-17T18:25:49.753146+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:08:27.581912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1826,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.741984+0000","last_change":"2024-08-17T18:23:36.324372+0000","last_active":"2024-08-17T18:25:48.741984+0000","last_peered":"2024-08-17T18:25:48.741984+0000","last_clean":"2024-08-17T18:25:48.741984+0000","last_became_active":"2024-08-17T18:23:36.323893+0000","last_became_peered":"2024-08-17T18:23:36.323893+0000","last_unstale":"2024-08-17T18:25:48.741984+0000","last_undegraded":"2024-08-17T18:25:48.741984+0000","last_fullsized":"2024-08-17T18:25:48.741984+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:10:52.829170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":219,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.754864+0000","last_change":"2024-08-17T18:23:36.319358+0000","last_active":"2024-08-17T18:25:49.754864+0000","last_peered":"2024-08-17T18:25:49.754864+0000","last_clean":"2024-08-17T18:25:49.754864+0000","last_became_active":"2024-08-17T18:23:36.318521+0000","last_became_peered":"2024-08-17T18:23:36.318521+0000","last_unstale":"2024-08-17T18:25:49.754864+0000","last_undegraded":"2024-08-17T18:25:49.754864+0000","last_fullsized":"2024-08-17T18:25:49.754864+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:56:48.984477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":212,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.754298+0000","last_change":"2024-08-17T18:23:59.741806+0000","last_active":"2024-08-17T18:25:49.754298+0000","last_peered":"2024-08-17T18:25:49.754298+0000","last_clean":"2024-08-17T18:25:49.754298+0000","last_became_active":"2024-08-17T18:23:59.741479+0000","last_became_peered":"2024-08-17T18:23:59.741479+0000","last_unstale":"2024-08-17T18:25:49.754298+0000","last_undegraded":"2024-08-17T18:25:49.754298+0000","last_fullsized":"2024-08-17T18:25:49.754298+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:02:39.072277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":335,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.751778+0000","last_change":"2024-08-17T18:22:36.836343+0000","last_active":"2024-08-17T18:25:48.751778+0000","last_peered":"2024-08-17T18:25:48.751778+0000","last_clean":"2024-08-17T18:25:48.751778+0000","last_became_active":"2024-08-17T18:22:36.835580+0000","last_became_peered":"2024-08-17T18:22:36.835580+0000","last_unstale":"2024-08-17T18:25:48.751778+0000","last_undegraded":"2024-08-17T18:25:48.751778+0000","last_fullsized":"2024-08-17T18:25:48.751778+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:05:51.737218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":211,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.755067+0000","last_change":"2024-08-17T18:23:59.742425+0000","last_active":"2024-08-17T18:25:49.755067+0000","last_peered":"2024-08-17T18:25:49.755067+0000","last_clean":"2024-08-17T18:25:49.755067+0000","last_became_active":"2024-08-17T18:23:59.742155+0000","last_became_peered":"2024-08-17T18:23:59.742155+0000","last_unstale":"2024-08-17T18:25:49.755067+0000","last_undegraded":"2024-08-17T18:25:49.755067+0000","last_fullsized":"2024-08-17T18:25:49.755067+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:11:42.833289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":346,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.751761+0000","last_change":"2024-08-17T18:20:32.094881+0000","last_active":"2024-08-17T18:25:48.751761+0000","last_peered":"2024-08-17T18:25:48.751761+0000","last_clean":"2024-08-17T18:25:48.751761+0000","last_became_active":"2024-08-17T18:20:32.077429+0000","last_became_peered":"2024-08-17T18:20:32.077429+0000","last_unstale":"2024-08-17T18:25:48.751761+0000","last_undegraded":"2024-08-17T18:25:48.751761+0000","last_fullsized":"2024-08-17T18:25:48.751761+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:08:26.290851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1831,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.742178+0000","last_change":"2024-08-17T18:25:14.026295+0000","last_active":"2024-08-17T18:25:48.742178+0000","last_peered":"2024-08-17T18:25:48.742178+0000","last_clean":"2024-08-17T18:25:48.742178+0000","last_became_active":"2024-08-17T18:25:14.025999+0000","last_became_peered":"2024-08-17T18:25:14.025999+0000","last_unstale":"2024-08-17T18:25:48.742178+0000","last_undegraded":"2024-08-17T18:25:48.742178+0000","last_fullsized":"2024-08-17T18:25:48.742178+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:55:16.975299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":54,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.752416+0000","last_change":"2024-08-17T18:23:59.745317+0000","last_active":"2024-08-17T18:25:49.752416+0000","last_peered":"2024-08-17T18:25:49.752416+0000","last_clean":"2024-08-17T18:25:49.752416+0000","last_became_active":"2024-08-17T18:23:59.743236+0000","last_became_peered":"2024-08-17T18:23:59.743236+0000","last_unstale":"2024-08-17T18:25:49.752416+0000","last_undegraded":"2024-08-17T18:25:49.752416+0000","last_fullsized":"2024-08-17T18:25:49.752416+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:11.253490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":77,"reported_epoch":1760,"state":"active+clean","last_fresh":"2024-08-17T18:25:31.891784+0000","last_change":"2024-08-17T18:22:53.938704+0000","last_active":"2024-08-17T18:25:31.891784+0000","last_peered":"2024-08-17T18:25:31.891784+0000","last_clean":"2024-08-17T18:25:31.891784+0000","last_became_active":"2024-08-17T18:22:53.938406+0000","last_became_peered":"2024-08-17T18:22:53.938406+0000","last_unstale":"2024-08-17T18:25:31.891784+0000","last_undegraded":"2024-08-17T18:25:31.891784+0000","last_fullsized":"2024-08-17T18:25:31.891784+0000","mapping_epoch":1727,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:58:21.566014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1808,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.741801+0000","last_change":"2024-08-17T18:20:31.942863+0000","last_active":"2024-08-17T18:25:48.741801+0000","last_peered":"2024-08-17T18:25:48.741801+0000","last_clean":"2024-08-17T18:25:48.741801+0000","last_became_active":"2024-08-17T18:20:31.941281+0000","last_became_peered":"2024-08-17T18:20:31.941281+0000","last_unstale":"2024-08-17T18:25:48.741801+0000","last_undegraded":"2024-08-17T18:25:48.741801+0000","last_fullsized":"2024-08-17T18:25:48.741801+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:48:05.556975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":211,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.754670+0000","last_change":"2024-08-17T18:20:31.933142+0000","last_active":"2024-08-17T18:25:49.754670+0000","last_peered":"2024-08-17T18:25:49.754670+0000","last_clean":"2024-08-17T18:25:49.754670+0000","last_became_active":"2024-08-17T18:20:31.932798+0000","last_became_peered":"2024-08-17T18:20:31.932798+0000","last_unstale":"2024-08-17T18:25:49.754670+0000","last_undegraded":"2024-08-17T18:25:49.754670+0000","last_fullsized":"2024-08-17T18:25:49.754670+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:32:51.059352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":146,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.752894+0000","last_change":"2024-08-17T18:20:31.937627+0000","last_active":"2024-08-17T18:25:49.752894+0000","last_peered":"2024-08-17T18:25:49.752894+0000","last_clean":"2024-08-17T18:25:49.752894+0000","last_became_active":"2024-08-17T18:20:31.937361+0000","last_became_peered":"2024-08-17T18:20:31.937361+0000","last_unstale":"2024-08-17T18:25:49.752894+0000","last_undegraded":"2024-08-17T18:25:49.752894+0000","last_fullsized":"2024-08-17T18:25:49.752894+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:01:26.309656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":54,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.752179+0000","last_change":"2024-08-17T18:23:59.743052+0000","last_active":"2024-08-17T18:25:49.752179+0000","last_peered":"2024-08-17T18:25:49.752179+0000","last_clean":"2024-08-17T18:25:49.752179+0000","last_became_active":"2024-08-17T18:23:59.742158+0000","last_became_peered":"2024-08-17T18:23:59.742158+0000","last_unstale":"2024-08-17T18:25:49.752179+0000","last_undegraded":"2024-08-17T18:25:49.752179+0000","last_fullsized":"2024-08-17T18:25:49.752179+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:34:34.249644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":233,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.754124+0000","last_change":"2024-08-17T18:25:14.176708+0000","last_active":"2024-08-17T18:25:49.754124+0000","last_peered":"2024-08-17T18:25:49.754124+0000","last_clean":"2024-08-17T18:25:49.754124+0000","last_became_active":"2024-08-17T18:25:14.176239+0000","last_became_peered":"2024-08-17T18:25:14.176239+0000","last_unstale":"2024-08-17T18:25:49.754124+0000","last_undegraded":"2024-08-17T18:25:49.754124+0000","last_fullsized":"2024-08-17T18:25:49.754124+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:08:21.826456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1824,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.741492+0000","last_change":"2024-08-17T18:22:36.867005+0000","last_active":"2024-08-17T18:25:48.741492+0000","last_peered":"2024-08-17T18:25:48.741492+0000","last_clean":"2024-08-17T18:25:48.741492+0000","last_became_active":"2024-08-17T18:22:36.866297+0000","last_became_peered":"2024-08-17T18:22:36.866297+0000","last_unstale":"2024-08-17T18:25:48.741492+0000","last_undegraded":"2024-08-17T18:25:48.741492+0000","last_fullsized":"2024-08-17T18:25:48.741492+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:09:18.460006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":235,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.754147+0000","last_change":"2024-08-17T18:25:14.035767+0000","last_active":"2024-08-17T18:25:49.754147+0000","last_peered":"2024-08-17T18:25:49.754147+0000","last_clean":"2024-08-17T18:25:49.754147+0000","last_became_active":"2024-08-17T18:25:14.034592+0000","last_became_peered":"2024-08-17T18:25:14.034592+0000","last_unstale":"2024-08-17T18:25:49.754147+0000","last_undegraded":"2024-08-17T18:25:49.754147+0000","last_fullsized":"2024-08-17T18:25:49.754147+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:22:00.711481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":54,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.753258+0000","last_change":"2024-08-17T18:23:59.744879+0000","last_active":"2024-08-17T18:25:49.753258+0000","last_peered":"2024-08-17T18:25:49.753258+0000","last_clean":"2024-08-17T18:25:49.753258+0000","last_became_active":"2024-08-17T18:23:59.743333+0000","last_became_peered":"2024-08-17T18:23:59.743333+0000","last_unstale":"2024-08-17T18:25:49.753258+0000","last_undegraded":"2024-08-17T18:25:49.753258+0000","last_fullsized":"2024-08-17T18:25:49.753258+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:02:01.546706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":81,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.799269+0000","last_change":"2024-08-17T18:22:36.836745+0000","last_active":"2024-08-17T18:25:49.799269+0000","last_peered":"2024-08-17T18:25:49.799269+0000","last_clean":"2024-08-17T18:25:49.799269+0000","last_became_active":"2024-08-17T18:22:36.835161+0000","last_became_peered":"2024-08-17T18:22:36.835161+0000","last_unstale":"2024-08-17T18:25:49.799269+0000","last_undegraded":"2024-08-17T18:25:49.799269+0000","last_fullsized":"2024-08-17T18:25:49.799269+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:29:43.303633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":405,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.751863+0000","last_change":"2024-08-17T18:25:14.027677+0000","last_active":"2024-08-17T18:25:48.751863+0000","last_peered":"2024-08-17T18:25:48.751863+0000","last_clean":"2024-08-17T18:25:48.751863+0000","last_became_active":"2024-08-17T18:25:14.027309+0000","last_became_peered":"2024-08-17T18:25:14.027309+0000","last_unstale":"2024-08-17T18:25:48.751863+0000","last_undegraded":"2024-08-17T18:25:48.751863+0000","last_fullsized":"2024-08-17T18:25:48.751863+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:15:02.617263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":211,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.754567+0000","last_change":"2024-08-17T18:18:29.928751+0000","last_active":"2024-08-17T18:25:49.754567+0000","last_peered":"2024-08-17T18:25:49.754567+0000","last_clean":"2024-08-17T18:25:49.754567+0000","last_became_active":"2024-08-17T18:18:29.898102+0000","last_became_peered":"2024-08-17T18:18:29.898102+0000","last_unstale":"2024-08-17T18:25:49.754567+0000","last_undegraded":"2024-08-17T18:25:49.754567+0000","last_fullsized":"2024-08-17T18:25:49.754567+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:21.098266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":30,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.751273+0000","last_change":"2024-08-17T18:25:14.030155+0000","last_active":"2024-08-17T18:25:49.751273+0000","last_peered":"2024-08-17T18:25:49.751273+0000","last_clean":"2024-08-17T18:25:49.751273+0000","last_became_active":"2024-08-17T18:25:14.029876+0000","last_became_peered":"2024-08-17T18:25:14.029876+0000","last_unstale":"2024-08-17T18:25:49.751273+0000","last_undegraded":"2024-08-17T18:25:49.751273+0000","last_fullsized":"2024-08-17T18:25:49.751273+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:04:34.619858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1808,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.742917+0000","last_change":"2024-08-17T18:22:36.838096+0000","last_active":"2024-08-17T18:25:48.742917+0000","last_peered":"2024-08-17T18:25:48.742917+0000","last_clean":"2024-08-17T18:25:48.742917+0000","last_became_active":"2024-08-17T18:22:36.835757+0000","last_became_peered":"2024-08-17T18:22:36.835757+0000","last_unstale":"2024-08-17T18:25:48.742917+0000","last_undegraded":"2024-08-17T18:25:48.742917+0000","last_fullsized":"2024-08-17T18:25:48.742917+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:55:31.016446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1799,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.742247+0000","last_change":"2024-08-17T18:25:14.026273+0000","last_active":"2024-08-17T18:25:48.742247+0000","last_peered":"2024-08-17T18:25:48.742247+0000","last_clean":"2024-08-17T18:25:48.742247+0000","last_became_active":"2024-08-17T18:25:14.025964+0000","last_became_peered":"2024-08-17T18:25:14.025964+0000","last_unstale":"2024-08-17T18:25:48.742247+0000","last_undegraded":"2024-08-17T18:25:48.742247+0000","last_fullsized":"2024-08-17T18:25:48.742247+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:43.150679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"347'149","reported_seq":1033,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.754296+0000","last_change":"2024-08-17T18:20:33.927014+0000","last_active":"2024-08-17T18:25:49.754296+0000","last_peered":"2024-08-17T18:25:49.754296+0000","last_clean":"2024-08-17T18:25:49.754296+0000","last_became_active":"2024-08-17T18:20:33.926619+0000","last_became_peered":"2024-08-17T18:20:33.926619+0000","last_unstale":"2024-08-17T18:25:49.754296+0000","last_undegraded":"2024-08-17T18:25:49.754296+0000","last_fullsized":"2024-08-17T18:25:49.754296+0000","mapping_epoch":1697,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1698,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:57.271286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:57.271286+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:35:26.778777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1786,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.741366+0000","last_change":"2024-08-17T18:22:36.838651+0000","last_active":"2024-08-17T18:25:48.741366+0000","last_peered":"2024-08-17T18:25:48.741366+0000","last_clean":"2024-08-17T18:25:48.741366+0000","last_became_active":"2024-08-17T18:22:36.838262+0000","last_became_peered":"2024-08-17T18:22:36.838262+0000","last_unstale":"2024-08-17T18:25:48.741366+0000","last_undegraded":"2024-08-17T18:25:48.741366+0000","last_fullsized":"2024-08-17T18:25:48.741366+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:24:25.119257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1767,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.741420+0000","last_change":"2024-08-17T18:22:36.866053+0000","last_active":"2024-08-17T18:25:48.741420+0000","last_peered":"2024-08-17T18:25:48.741420+0000","last_clean":"2024-08-17T18:25:48.741420+0000","last_became_active":"2024-08-17T18:22:36.865187+0000","last_became_peered":"2024-08-17T18:22:36.865187+0000","last_unstale":"2024-08-17T18:25:48.741420+0000","last_undegraded":"2024-08-17T18:25:48.741420+0000","last_fullsized":"2024-08-17T18:25:48.741420+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:45:34.232504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":58,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.752083+0000","last_change":"2024-08-17T18:23:59.750520+0000","last_active":"2024-08-17T18:25:49.752083+0000","last_peered":"2024-08-17T18:25:49.752083+0000","last_clean":"2024-08-17T18:25:49.752083+0000","last_became_active":"2024-08-17T18:23:59.742332+0000","last_became_peered":"2024-08-17T18:23:59.742332+0000","last_unstale":"2024-08-17T18:25:49.752083+0000","last_undegraded":"2024-08-17T18:25:49.752083+0000","last_fullsized":"2024-08-17T18:25:49.752083+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:39:39.325194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"347'149","reported_seq":1074,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.743173+0000","last_change":"2024-08-17T18:23:39.360135+0000","last_active":"2024-08-17T18:25:48.743173+0000","last_peered":"2024-08-17T18:25:48.743173+0000","last_clean":"2024-08-17T18:25:48.743173+0000","last_became_active":"2024-08-17T18:23:39.359652+0000","last_became_peered":"2024-08-17T18:23:39.359652+0000","last_unstale":"2024-08-17T18:25:48.743173+0000","last_undegraded":"2024-08-17T18:25:48.743173+0000","last_fullsized":"2024-08-17T18:25:48.743173+0000","mapping_epoch":1736,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1737,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:53.309517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:53.309517+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:20:52.590505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":219,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.754179+0000","last_change":"2024-08-17T18:23:59.746854+0000","last_active":"2024-08-17T18:25:49.754179+0000","last_peered":"2024-08-17T18:25:49.754179+0000","last_clean":"2024-08-17T18:25:49.754179+0000","last_became_active":"2024-08-17T18:23:59.746190+0000","last_became_peered":"2024-08-17T18:23:59.746190+0000","last_unstale":"2024-08-17T18:25:49.754179+0000","last_undegraded":"2024-08-17T18:25:49.754179+0000","last_fullsized":"2024-08-17T18:25:49.754179+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:01:53.108037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":30,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.752170+0000","last_change":"2024-08-17T18:25:14.025153+0000","last_active":"2024-08-17T18:25:49.752170+0000","last_peered":"2024-08-17T18:25:49.752170+0000","last_clean":"2024-08-17T18:25:49.752170+0000","last_became_active":"2024-08-17T18:25:14.024700+0000","last_became_peered":"2024-08-17T18:25:14.024700+0000","last_unstale":"2024-08-17T18:25:49.752170+0000","last_undegraded":"2024-08-17T18:25:49.752170+0000","last_fullsized":"2024-08-17T18:25:49.752170+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:50:25.014563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":221,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.755133+0000","last_change":"2024-08-17T18:23:59.772682+0000","last_active":"2024-08-17T18:25:49.755133+0000","last_peered":"2024-08-17T18:25:49.755133+0000","last_clean":"2024-08-17T18:25:49.755133+0000","last_became_active":"2024-08-17T18:23:59.748920+0000","last_became_peered":"2024-08-17T18:23:59.748920+0000","last_unstale":"2024-08-17T18:25:49.755133+0000","last_undegraded":"2024-08-17T18:25:49.755133+0000","last_fullsized":"2024-08-17T18:25:49.755133+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:07:31.660610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"347'149","reported_seq":1065,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.744529+0000","last_change":"2024-08-17T18:24:02.793338+0000","last_active":"2024-08-17T18:25:48.744529+0000","last_peered":"2024-08-17T18:25:48.744529+0000","last_clean":"2024-08-17T18:25:48.744529+0000","last_became_active":"2024-08-17T18:24:02.792982+0000","last_became_peered":"2024-08-17T18:24:02.792982+0000","last_unstale":"2024-08-17T18:25:48.744529+0000","last_undegraded":"2024-08-17T18:25:48.744529+0000","last_fullsized":"2024-08-17T18:25:48.744529+0000","mapping_epoch":1743,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1744,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:52.474544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:52.474544+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:37:26.909105+0000","scrub_duration":0.001886326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":54,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.752855+0000","last_change":"2024-08-17T18:23:59.741315+0000","last_active":"2024-08-17T18:25:49.752855+0000","last_peered":"2024-08-17T18:25:49.752855+0000","last_clean":"2024-08-17T18:25:49.752855+0000","last_became_active":"2024-08-17T18:23:59.741020+0000","last_became_peered":"2024-08-17T18:23:59.741020+0000","last_unstale":"2024-08-17T18:25:49.752855+0000","last_undegraded":"2024-08-17T18:25:49.752855+0000","last_fullsized":"2024-08-17T18:25:49.752855+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:46:47.093299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":176,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.743961+0000","last_change":"2024-08-17T18:23:59.740824+0000","last_active":"2024-08-17T18:25:48.743961+0000","last_peered":"2024-08-17T18:25:48.743961+0000","last_clean":"2024-08-17T18:25:48.743961+0000","last_became_active":"2024-08-17T18:23:59.740478+0000","last_became_peered":"2024-08-17T18:23:59.740478+0000","last_unstale":"2024-08-17T18:25:48.743961+0000","last_undegraded":"2024-08-17T18:25:48.743961+0000","last_fullsized":"2024-08-17T18:25:48.743961+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:44:30.520216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":146,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.752047+0000","last_change":"2024-08-17T18:20:32.012874+0000","last_active":"2024-08-17T18:25:49.752047+0000","last_peered":"2024-08-17T18:25:49.752047+0000","last_clean":"2024-08-17T18:25:49.752047+0000","last_became_active":"2024-08-17T18:20:32.012553+0000","last_became_peered":"2024-08-17T18:20:32.012553+0000","last_unstale":"2024-08-17T18:25:49.752047+0000","last_undegraded":"2024-08-17T18:25:49.752047+0000","last_fullsized":"2024-08-17T18:25:49.752047+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:14:17.188745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"347'149","reported_seq":1031,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.741347+0000","last_change":"2024-08-17T18:22:36.866341+0000","last_active":"2024-08-17T18:25:48.741347+0000","last_peered":"2024-08-17T18:25:48.741347+0000","last_clean":"2024-08-17T18:25:48.741347+0000","last_became_active":"2024-08-17T18:22:36.865561+0000","last_became_peered":"2024-08-17T18:22:36.865561+0000","last_unstale":"2024-08-17T18:25:48.741347+0000","last_undegraded":"2024-08-17T18:25:48.741347+0000","last_fullsized":"2024-08-17T18:25:48.741347+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:49.461698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:49.461698+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:22:21.423162+0000","scrub_duration":0.0013792520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":30,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.751363+0000","last_change":"2024-08-17T18:25:14.025883+0000","last_active":"2024-08-17T18:25:49.751363+0000","last_peered":"2024-08-17T18:25:49.751363+0000","last_clean":"2024-08-17T18:25:49.751363+0000","last_became_active":"2024-08-17T18:25:14.025558+0000","last_became_peered":"2024-08-17T18:25:14.025558+0000","last_unstale":"2024-08-17T18:25:49.751363+0000","last_undegraded":"2024-08-17T18:25:49.751363+0000","last_fullsized":"2024-08-17T18:25:49.751363+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:28:59.733420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":146,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.752193+0000","last_change":"2024-08-17T18:20:32.014875+0000","last_active":"2024-08-17T18:25:49.752193+0000","last_peered":"2024-08-17T18:25:49.752193+0000","last_clean":"2024-08-17T18:25:49.752193+0000","last_became_active":"2024-08-17T18:20:32.014631+0000","last_became_peered":"2024-08-17T18:20:32.014631+0000","last_unstale":"2024-08-17T18:25:49.752193+0000","last_undegraded":"2024-08-17T18:25:49.752193+0000","last_fullsized":"2024-08-17T18:25:49.752193+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:00:50.978635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":336,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.751793+0000","last_change":"2024-08-17T18:18:29.735362+0000","last_active":"2024-08-17T18:25:48.751793+0000","last_peered":"2024-08-17T18:25:48.751793+0000","last_clean":"2024-08-17T18:25:48.751793+0000","last_became_active":"2024-08-17T18:18:29.734752+0000","last_became_peered":"2024-08-17T18:18:29.734752+0000","last_unstale":"2024-08-17T18:25:48.751793+0000","last_undegraded":"2024-08-17T18:25:48.751793+0000","last_fullsized":"2024-08-17T18:25:48.751793+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:25.709698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"347'149","reported_seq":1075,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.751638+0000","last_change":"2024-08-17T18:24:02.792784+0000","last_active":"2024-08-17T18:25:48.751638+0000","last_peered":"2024-08-17T18:25:48.751638+0000","last_clean":"2024-08-17T18:25:48.751638+0000","last_became_active":"2024-08-17T18:24:02.792286+0000","last_became_peered":"2024-08-17T18:24:02.792286+0000","last_unstale":"2024-08-17T18:25:48.751638+0000","last_undegraded":"2024-08-17T18:25:48.751638+0000","last_fullsized":"2024-08-17T18:25:48.751638+0000","mapping_epoch":1743,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1744,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:00.316884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:00.316884+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:26:57.982520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":30,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.751881+0000","last_change":"2024-08-17T18:25:14.031207+0000","last_active":"2024-08-17T18:25:49.751881+0000","last_peered":"2024-08-17T18:25:49.751881+0000","last_clean":"2024-08-17T18:25:49.751881+0000","last_became_active":"2024-08-17T18:25:14.030916+0000","last_became_peered":"2024-08-17T18:25:14.030916+0000","last_unstale":"2024-08-17T18:25:49.751881+0000","last_undegraded":"2024-08-17T18:25:49.751881+0000","last_fullsized":"2024-08-17T18:25:49.751881+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:37:34.050072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1822,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.742374+0000","last_change":"2024-08-17T18:23:59.741594+0000","last_active":"2024-08-17T18:25:48.742374+0000","last_peered":"2024-08-17T18:25:48.742374+0000","last_clean":"2024-08-17T18:25:48.742374+0000","last_became_active":"2024-08-17T18:23:59.741207+0000","last_became_peered":"2024-08-17T18:23:59.741207+0000","last_unstale":"2024-08-17T18:25:48.742374+0000","last_undegraded":"2024-08-17T18:25:48.742374+0000","last_fullsized":"2024-08-17T18:25:48.742374+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:02:52.545413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":405,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.752031+0000","last_change":"2024-08-17T18:22:36.836104+0000","last_active":"2024-08-17T18:25:48.752031+0000","last_peered":"2024-08-17T18:25:48.752031+0000","last_clean":"2024-08-17T18:25:48.752031+0000","last_became_active":"2024-08-17T18:22:36.835273+0000","last_became_peered":"2024-08-17T18:22:36.835273+0000","last_unstale":"2024-08-17T18:25:48.752031+0000","last_undegraded":"2024-08-17T18:25:48.752031+0000","last_fullsized":"2024-08-17T18:25:48.752031+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:26:23.494575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"347'149","reported_seq":846,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.752221+0000","last_change":"2024-08-17T18:24:02.792482+0000","last_active":"2024-08-17T18:25:49.752221+0000","last_peered":"2024-08-17T18:25:49.752221+0000","last_clean":"2024-08-17T18:25:49.752221+0000","last_became_active":"2024-08-17T18:24:02.792032+0000","last_became_peered":"2024-08-17T18:24:02.792032+0000","last_unstale":"2024-08-17T18:25:49.752221+0000","last_undegraded":"2024-08-17T18:25:49.752221+0000","last_fullsized":"2024-08-17T18:25:49.752221+0000","mapping_epoch":1743,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1744,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:02.363531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:02.363531+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:08:22.251652+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":30,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.751892+0000","last_change":"2024-08-17T18:25:14.176621+0000","last_active":"2024-08-17T18:25:49.751892+0000","last_peered":"2024-08-17T18:25:49.751892+0000","last_clean":"2024-08-17T18:25:49.751892+0000","last_became_active":"2024-08-17T18:25:14.176321+0000","last_became_peered":"2024-08-17T18:25:14.176321+0000","last_unstale":"2024-08-17T18:25:49.751892+0000","last_undegraded":"2024-08-17T18:25:49.751892+0000","last_fullsized":"2024-08-17T18:25:49.751892+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:40:55.297225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":157,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.751573+0000","last_change":"2024-08-17T18:22:36.834643+0000","last_active":"2024-08-17T18:25:49.751573+0000","last_peered":"2024-08-17T18:25:49.751573+0000","last_clean":"2024-08-17T18:25:49.751573+0000","last_became_active":"2024-08-17T18:22:36.834279+0000","last_became_peered":"2024-08-17T18:22:36.834279+0000","last_unstale":"2024-08-17T18:25:49.751573+0000","last_undegraded":"2024-08-17T18:25:49.751573+0000","last_fullsized":"2024-08-17T18:25:49.751573+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:29:36.002584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2823,"reported_epoch":1760,"state":"active+clean","last_fresh":"2024-08-17T18:25:31.891368+0000","last_change":"2024-08-17T18:18:29.742742+0000","last_active":"2024-08-17T18:25:31.891368+0000","last_peered":"2024-08-17T18:25:31.891368+0000","last_clean":"2024-08-17T18:25:31.891368+0000","last_became_active":"2024-08-17T18:18:29.742393+0000","last_became_peered":"2024-08-17T18:18:29.742393+0000","last_unstale":"2024-08-17T18:25:31.891368+0000","last_undegraded":"2024-08-17T18:25:31.891368+0000","last_fullsized":"2024-08-17T18:25:31.891368+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:57:32.858608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"347'149","reported_seq":834,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.751942+0000","last_change":"2024-08-17T18:23:39.370457+0000","last_active":"2024-08-17T18:25:49.751942+0000","last_peered":"2024-08-17T18:25:49.751942+0000","last_clean":"2024-08-17T18:25:49.751942+0000","last_became_active":"2024-08-17T18:23:39.370144+0000","last_became_peered":"2024-08-17T18:23:39.370144+0000","last_unstale":"2024-08-17T18:25:49.751942+0000","last_undegraded":"2024-08-17T18:25:49.751942+0000","last_fullsized":"2024-08-17T18:25:49.751942+0000","mapping_epoch":1736,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1737,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:20:35.854154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:20:35.854154+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:39:35.866331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":54,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.751989+0000","last_change":"2024-08-17T18:23:59.742951+0000","last_active":"2024-08-17T18:25:49.751989+0000","last_peered":"2024-08-17T18:25:49.751989+0000","last_clean":"2024-08-17T18:25:49.751989+0000","last_became_active":"2024-08-17T18:23:59.742510+0000","last_became_peered":"2024-08-17T18:23:59.742510+0000","last_unstale":"2024-08-17T18:25:49.751989+0000","last_undegraded":"2024-08-17T18:25:49.751989+0000","last_fullsized":"2024-08-17T18:25:49.751989+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:19:12.943501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":54,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.752184+0000","last_change":"2024-08-17T18:23:59.740786+0000","last_active":"2024-08-17T18:25:49.752184+0000","last_peered":"2024-08-17T18:25:49.752184+0000","last_clean":"2024-08-17T18:25:49.752184+0000","last_became_active":"2024-08-17T18:23:59.740340+0000","last_became_peered":"2024-08-17T18:23:59.740340+0000","last_unstale":"2024-08-17T18:25:49.752184+0000","last_undegraded":"2024-08-17T18:25:49.752184+0000","last_fullsized":"2024-08-17T18:25:49.752184+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:52:27.108557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":30,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.751340+0000","last_change":"2024-08-17T18:25:14.176308+0000","last_active":"2024-08-17T18:25:49.751340+0000","last_peered":"2024-08-17T18:25:49.751340+0000","last_clean":"2024-08-17T18:25:49.751340+0000","last_became_active":"2024-08-17T18:25:14.175851+0000","last_became_peered":"2024-08-17T18:25:14.175851+0000","last_unstale":"2024-08-17T18:25:49.751340+0000","last_undegraded":"2024-08-17T18:25:49.751340+0000","last_fullsized":"2024-08-17T18:25:49.751340+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:10:25.195682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":40,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.744540+0000","last_change":"2024-08-17T18:23:36.318955+0000","last_active":"2024-08-17T18:25:48.744540+0000","last_peered":"2024-08-17T18:25:48.744540+0000","last_clean":"2024-08-17T18:25:48.744540+0000","last_became_active":"2024-08-17T18:23:36.318684+0000","last_became_peered":"2024-08-17T18:23:36.318684+0000","last_unstale":"2024-08-17T18:25:48.744540+0000","last_undegraded":"2024-08-17T18:25:48.744540+0000","last_fullsized":"2024-08-17T18:25:48.744540+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:47.191868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":216,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.754919+0000","last_change":"2024-08-17T18:22:36.835833+0000","last_active":"2024-08-17T18:25:49.754919+0000","last_peered":"2024-08-17T18:25:49.754919+0000","last_clean":"2024-08-17T18:25:49.754919+0000","last_became_active":"2024-08-17T18:22:36.835514+0000","last_became_peered":"2024-08-17T18:22:36.835514+0000","last_unstale":"2024-08-17T18:25:49.754919+0000","last_undegraded":"2024-08-17T18:25:49.754919+0000","last_fullsized":"2024-08-17T18:25:49.754919+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:43:53.668287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":351,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.751699+0000","last_change":"2024-08-17T18:19:26.374702+0000","last_active":"2024-08-17T18:25:48.751699+0000","last_peered":"2024-08-17T18:25:48.751699+0000","last_clean":"2024-08-17T18:25:48.751699+0000","last_became_active":"2024-08-17T18:15:53.304106+0000","last_became_peered":"2024-08-17T18:15:53.304106+0000","last_unstale":"2024-08-17T18:25:48.751699+0000","last_undegraded":"2024-08-17T18:25:48.751699+0000","last_fullsized":"2024-08-17T18:25:48.751699+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:21.409368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":991,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.751775+0000","last_change":"2024-08-17T18:25:48.748617+0000","last_active":"2024-08-17T18:25:49.751775+0000","last_peered":"2024-08-17T18:25:49.751775+0000","last_clean":"2024-08-17T18:25:49.751775+0000","last_became_active":"2024-08-17T18:24:02.791616+0000","last_became_peered":"2024-08-17T18:24:02.791616+0000","last_unstale":"2024-08-17T18:25:49.751775+0000","last_undegraded":"2024-08-17T18:25:49.751775+0000","last_fullsized":"2024-08-17T18:25:49.751775+0000","mapping_epoch":1743,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1744,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:20:36.599786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:20:36.599786+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T18:25:51.726713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":74,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"347'149","reported_seq":1057,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.751833+0000","last_change":"2024-08-17T18:22:36.836656+0000","last_active":"2024-08-17T18:25:48.751833+0000","last_peered":"2024-08-17T18:25:48.751833+0000","last_clean":"2024-08-17T18:25:48.751833+0000","last_became_active":"2024-08-17T18:22:36.835810+0000","last_became_peered":"2024-08-17T18:22:36.835810+0000","last_unstale":"2024-08-17T18:25:48.751833+0000","last_undegraded":"2024-08-17T18:25:48.751833+0000","last_fullsized":"2024-08-17T18:25:48.751833+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:55.255983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:55.255983+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:59:27.835153+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":146,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.752122+0000","last_change":"2024-08-17T18:20:31.939871+0000","last_active":"2024-08-17T18:25:49.752122+0000","last_peered":"2024-08-17T18:25:49.752122+0000","last_clean":"2024-08-17T18:25:49.752122+0000","last_became_active":"2024-08-17T18:20:31.939535+0000","last_became_peered":"2024-08-17T18:20:31.939535+0000","last_unstale":"2024-08-17T18:25:49.752122+0000","last_undegraded":"2024-08-17T18:25:49.752122+0000","last_fullsized":"2024-08-17T18:25:49.752122+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:43:31.630668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":226,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.754043+0000","last_change":"2024-08-17T18:25:14.036186+0000","last_active":"2024-08-17T18:25:49.754043+0000","last_peered":"2024-08-17T18:25:49.754043+0000","last_clean":"2024-08-17T18:25:49.754043+0000","last_became_active":"2024-08-17T18:25:14.035138+0000","last_became_peered":"2024-08-17T18:25:14.035138+0000","last_unstale":"2024-08-17T18:25:49.754043+0000","last_undegraded":"2024-08-17T18:25:49.754043+0000","last_fullsized":"2024-08-17T18:25:49.754043+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:51.645934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1837,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.742700+0000","last_change":"2024-08-17T18:25:14.026973+0000","last_active":"2024-08-17T18:25:48.742700+0000","last_peered":"2024-08-17T18:25:48.742700+0000","last_clean":"2024-08-17T18:25:48.742700+0000","last_became_active":"2024-08-17T18:25:14.026478+0000","last_became_peered":"2024-08-17T18:25:14.026478+0000","last_unstale":"2024-08-17T18:25:48.742700+0000","last_undegraded":"2024-08-17T18:25:48.742700+0000","last_fullsized":"2024-08-17T18:25:48.742700+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:31.568544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1791,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.741441+0000","last_change":"2024-08-17T18:15:53.244851+0000","last_active":"2024-08-17T18:25:48.741441+0000","last_peered":"2024-08-17T18:25:48.741441+0000","last_clean":"2024-08-17T18:25:48.741441+0000","last_became_active":"2024-08-17T18:15:53.240074+0000","last_became_peered":"2024-08-17T18:15:53.240074+0000","last_unstale":"2024-08-17T18:25:48.741441+0000","last_undegraded":"2024-08-17T18:25:48.741441+0000","last_fullsized":"2024-08-17T18:25:48.741441+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:09:00.168413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"347'149","reported_seq":1043,"reported_epoch":1760,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T18:25:31.890871+0000","last_change":"2024-08-17T18:24:14.849225+0000","last_active":"2024-08-17T18:25:31.890871+0000","last_peered":"2024-08-17T18:25:31.890871+0000","last_clean":"2024-08-17T18:23:20.705230+0000","last_became_active":"2024-08-17T18:24:14.848918+0000","last_became_peered":"2024-08-17T18:24:14.848918+0000","last_unstale":"2024-08-17T18:25:31.890871+0000","last_undegraded":"2024-08-17T18:24:14.838017+0000","last_fullsized":"2024-08-17T18:24:14.837609+0000","mapping_epoch":1746,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1692,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:59.307356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:59.307356+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[2,3],"avail_no_missing":["2","3","4"],"object_location_counts":[{"shards":"2,3,4","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1795,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.743359+0000","last_change":"2024-08-17T18:20:31.942019+0000","last_active":"2024-08-17T18:25:48.743359+0000","last_peered":"2024-08-17T18:25:48.743359+0000","last_clean":"2024-08-17T18:25:48.743359+0000","last_became_active":"2024-08-17T18:20:31.941701+0000","last_became_peered":"2024-08-17T18:20:31.941701+0000","last_unstale":"2024-08-17T18:25:48.743359+0000","last_undegraded":"2024-08-17T18:25:48.743359+0000","last_fullsized":"2024-08-17T18:25:48.743359+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:58.908049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1772,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.744747+0000","last_change":"2024-08-17T18:25:14.027871+0000","last_active":"2024-08-17T18:25:48.744747+0000","last_peered":"2024-08-17T18:25:48.744747+0000","last_clean":"2024-08-17T18:25:48.744747+0000","last_became_active":"2024-08-17T18:25:14.027353+0000","last_became_peered":"2024-08-17T18:25:14.027353+0000","last_unstale":"2024-08-17T18:25:48.744747+0000","last_undegraded":"2024-08-17T18:25:48.744747+0000","last_fullsized":"2024-08-17T18:25:48.744747+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:06.568798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"347'149","reported_seq":1018,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.744145+0000","last_change":"2024-08-17T18:18:33.757347+0000","last_active":"2024-08-17T18:25:48.744145+0000","last_peered":"2024-08-17T18:25:48.744145+0000","last_clean":"2024-08-17T18:25:48.744145+0000","last_became_active":"2024-08-17T18:18:33.756960+0000","last_became_peered":"2024-08-17T18:18:33.756960+0000","last_unstale":"2024-08-17T18:25:48.744145+0000","last_undegraded":"2024-08-17T18:25:48.744145+0000","last_fullsized":"2024-08-17T18:25:48.744145+0000","mapping_epoch":1670,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1671,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:52.271487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:52.271487+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:37:36.389244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1789,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.742837+0000","last_change":"2024-08-17T18:22:36.837889+0000","last_active":"2024-08-17T18:25:48.742837+0000","last_peered":"2024-08-17T18:25:48.742837+0000","last_clean":"2024-08-17T18:25:48.742837+0000","last_became_active":"2024-08-17T18:22:36.837387+0000","last_became_peered":"2024-08-17T18:22:36.837387+0000","last_unstale":"2024-08-17T18:25:48.742837+0000","last_undegraded":"2024-08-17T18:25:48.742837+0000","last_fullsized":"2024-08-17T18:25:48.742837+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:11:29.265925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":242,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.754604+0000","last_change":"2024-08-17T18:23:59.770482+0000","last_active":"2024-08-17T18:25:49.754604+0000","last_peered":"2024-08-17T18:25:49.754604+0000","last_clean":"2024-08-17T18:25:49.754604+0000","last_became_active":"2024-08-17T18:23:59.747059+0000","last_became_peered":"2024-08-17T18:23:59.747059+0000","last_unstale":"2024-08-17T18:25:49.754604+0000","last_undegraded":"2024-08-17T18:25:49.754604+0000","last_fullsized":"2024-08-17T18:25:49.754604+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:04:17.042903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":54,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.752762+0000","last_change":"2024-08-17T18:23:59.745024+0000","last_active":"2024-08-17T18:25:49.752762+0000","last_peered":"2024-08-17T18:25:49.752762+0000","last_clean":"2024-08-17T18:25:49.752762+0000","last_became_active":"2024-08-17T18:23:59.741633+0000","last_became_peered":"2024-08-17T18:23:59.741633+0000","last_unstale":"2024-08-17T18:25:49.752762+0000","last_undegraded":"2024-08-17T18:25:49.752762+0000","last_fullsized":"2024-08-17T18:25:49.752762+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:41.514649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":359,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.751804+0000","last_change":"2024-08-17T18:22:36.836017+0000","last_active":"2024-08-17T18:25:48.751804+0000","last_peered":"2024-08-17T18:25:48.751804+0000","last_clean":"2024-08-17T18:25:48.751804+0000","last_became_active":"2024-08-17T18:22:36.835725+0000","last_became_peered":"2024-08-17T18:22:36.835725+0000","last_unstale":"2024-08-17T18:25:48.751804+0000","last_undegraded":"2024-08-17T18:25:48.751804+0000","last_fullsized":"2024-08-17T18:25:48.751804+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:18:07.395241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":40,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.749150+0000","last_change":"2024-08-17T18:23:36.324099+0000","last_active":"2024-08-17T18:25:49.749150+0000","last_peered":"2024-08-17T18:25:49.749150+0000","last_clean":"2024-08-17T18:25:49.749150+0000","last_became_active":"2024-08-17T18:23:36.323739+0000","last_became_peered":"2024-08-17T18:23:36.323739+0000","last_unstale":"2024-08-17T18:25:49.749150+0000","last_undegraded":"2024-08-17T18:25:49.749150+0000","last_fullsized":"2024-08-17T18:25:49.749150+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:54:56.319013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":362,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.751893+0000","last_change":"2024-08-17T18:22:36.836529+0000","last_active":"2024-08-17T18:25:48.751893+0000","last_peered":"2024-08-17T18:25:48.751893+0000","last_clean":"2024-08-17T18:25:48.751893+0000","last_became_active":"2024-08-17T18:22:36.836020+0000","last_became_peered":"2024-08-17T18:22:36.836020+0000","last_unstale":"2024-08-17T18:25:48.751893+0000","last_undegraded":"2024-08-17T18:25:48.751893+0000","last_fullsized":"2024-08-17T18:25:48.751893+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:22.088303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1846,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.741154+0000","last_change":"2024-08-17T18:25:14.040053+0000","last_active":"2024-08-17T18:25:48.741154+0000","last_peered":"2024-08-17T18:25:48.741154+0000","last_clean":"2024-08-17T18:25:48.741154+0000","last_became_active":"2024-08-17T18:25:14.039807+0000","last_became_peered":"2024-08-17T18:25:14.039807+0000","last_unstale":"2024-08-17T18:25:48.741154+0000","last_undegraded":"2024-08-17T18:25:48.741154+0000","last_fullsized":"2024-08-17T18:25:48.741154+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:05:53.486096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":155,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.742633+0000","last_change":"2024-08-17T18:20:32.006404+0000","last_active":"2024-08-17T18:25:48.742633+0000","last_peered":"2024-08-17T18:25:48.742633+0000","last_clean":"2024-08-17T18:25:48.742633+0000","last_became_active":"2024-08-17T18:20:31.947053+0000","last_became_peered":"2024-08-17T18:20:31.947053+0000","last_unstale":"2024-08-17T18:25:48.742633+0000","last_undegraded":"2024-08-17T18:25:48.742633+0000","last_fullsized":"2024-08-17T18:25:48.742633+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:21:06.537725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":362,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.751675+0000","last_change":"2024-08-17T18:23:36.318964+0000","last_active":"2024-08-17T18:25:48.751675+0000","last_peered":"2024-08-17T18:25:48.751675+0000","last_clean":"2024-08-17T18:25:48.751675+0000","last_became_active":"2024-08-17T18:23:36.318595+0000","last_became_peered":"2024-08-17T18:23:36.318595+0000","last_unstale":"2024-08-17T18:25:48.751675+0000","last_undegraded":"2024-08-17T18:25:48.751675+0000","last_fullsized":"2024-08-17T18:25:48.751675+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:29:16.221103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":181,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.742884+0000","last_change":"2024-08-17T18:23:59.752171+0000","last_active":"2024-08-17T18:25:48.742884+0000","last_peered":"2024-08-17T18:25:48.742884+0000","last_clean":"2024-08-17T18:25:48.742884+0000","last_became_active":"2024-08-17T18:23:59.745970+0000","last_became_peered":"2024-08-17T18:23:59.745970+0000","last_unstale":"2024-08-17T18:25:48.742884+0000","last_undegraded":"2024-08-17T18:25:48.742884+0000","last_fullsized":"2024-08-17T18:25:48.742884+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:24:52.861533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1919,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.741671+0000","last_change":"2024-08-17T18:24:22.107318+0000","last_active":"2024-08-17T18:25:48.741671+0000","last_peered":"2024-08-17T18:25:48.741671+0000","last_clean":"2024-08-17T18:25:48.741671+0000","last_became_active":"2024-08-17T18:24:14.847882+0000","last_became_peered":"2024-08-17T18:24:14.847882+0000","last_unstale":"2024-08-17T18:25:48.741671+0000","last_undegraded":"2024-08-17T18:25:48.741671+0000","last_fullsized":"2024-08-17T18:25:48.741671+0000","mapping_epoch":1746,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:45:33.740721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":219,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.755256+0000","last_change":"2024-08-17T18:21:12.645070+0000","last_active":"2024-08-17T18:25:49.755256+0000","last_peered":"2024-08-17T18:25:49.755256+0000","last_clean":"2024-08-17T18:25:49.755256+0000","last_became_active":"2024-08-17T18:21:12.644748+0000","last_became_peered":"2024-08-17T18:21:12.644748+0000","last_unstale":"2024-08-17T18:25:49.755256+0000","last_undegraded":"2024-08-17T18:25:49.755256+0000","last_fullsized":"2024-08-17T18:25:49.755256+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:19:08.683431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":519,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.741291+0000","last_change":"2024-08-17T18:22:36.838518+0000","last_active":"2024-08-17T18:25:48.741291+0000","last_peered":"2024-08-17T18:25:48.741291+0000","last_clean":"2024-08-17T18:25:48.741291+0000","last_became_active":"2024-08-17T18:22:36.834610+0000","last_became_peered":"2024-08-17T18:22:36.834610+0000","last_unstale":"2024-08-17T18:25:48.741291+0000","last_undegraded":"2024-08-17T18:25:48.741291+0000","last_fullsized":"2024-08-17T18:25:48.741291+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:42:18.798462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":54,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.752324+0000","last_change":"2024-08-17T18:23:59.740450+0000","last_active":"2024-08-17T18:25:49.752324+0000","last_peered":"2024-08-17T18:25:49.752324+0000","last_clean":"2024-08-17T18:25:49.752324+0000","last_became_active":"2024-08-17T18:23:59.740125+0000","last_became_peered":"2024-08-17T18:23:59.740125+0000","last_unstale":"2024-08-17T18:25:49.752324+0000","last_undegraded":"2024-08-17T18:25:49.752324+0000","last_fullsized":"2024-08-17T18:25:49.752324+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:26:38.858210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":54,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.752124+0000","last_change":"2024-08-17T18:23:59.740550+0000","last_active":"2024-08-17T18:25:49.752124+0000","last_peered":"2024-08-17T18:25:49.752124+0000","last_clean":"2024-08-17T18:25:49.752124+0000","last_became_active":"2024-08-17T18:23:59.740226+0000","last_became_peered":"2024-08-17T18:23:59.740226+0000","last_unstale":"2024-08-17T18:25:49.752124+0000","last_undegraded":"2024-08-17T18:25:49.752124+0000","last_fullsized":"2024-08-17T18:25:49.752124+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:35.745256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":352,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.751876+0000","last_change":"2024-08-17T18:18:29.900128+0000","last_active":"2024-08-17T18:25:48.751876+0000","last_peered":"2024-08-17T18:25:48.751876+0000","last_clean":"2024-08-17T18:25:48.751876+0000","last_became_active":"2024-08-17T18:18:29.899279+0000","last_became_peered":"2024-08-17T18:18:29.899279+0000","last_unstale":"2024-08-17T18:25:48.751876+0000","last_undegraded":"2024-08-17T18:25:48.751876+0000","last_fullsized":"2024-08-17T18:25:48.751876+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:01:29.045498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":145,"reported_epoch":1761,"state":"active+clean","last_fresh":"2024-08-17T18:25:48.743389+0000","last_change":"2024-08-17T18:15:53.271113+0000","last_active":"2024-08-17T18:25:48.743389+0000","last_peered":"2024-08-17T18:25:48.743389+0000","last_clean":"2024-08-17T18:25:48.743389+0000","last_became_active":"2024-08-17T18:15:53.230983+0000","last_became_peered":"2024-08-17T18:15:53.230983+0000","last_unstale":"2024-08-17T18:25:48.743389+0000","last_undegraded":"2024-08-17T18:25:48.743389+0000","last_fullsized":"2024-08-17T18:25:48.743389+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:00:35.315798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":237,"reported_epoch":1762,"state":"active+clean","last_fresh":"2024-08-17T18:25:49.754526+0000","last_change":"2024-08-17T18:25:14.030279+0000","last_active":"2024-08-17T18:25:49.754526+0000","last_peered":"2024-08-17T18:25:49.754526+0000","last_clean":"2024-08-17T18:25:49.754526+0000","last_became_active":"2024-08-17T18:25:14.027960+0000","last_became_peered":"2024-08-17T18:25:14.027960+0000","last_unstale":"2024-08-17T18:25:49.754526+0000","last_undegraded":"2024-08-17T18:25:49.754526+0000","last_fullsized":"2024-08-17T18:25:49.754526+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:20:46.317663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":58,"num_pg":16,"stat_sum":{"num_bytes":8650752,"num_objects":132,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":227096,"num_write_kb":7271233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30921,"num_bytes_recovered":1747976192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":56168448,"data_stored":56164376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114761,"ondisk_log_size":114761,"up":48,"acting":47,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644037,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":74,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1186397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":32,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":13180,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234918,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":874696,"kb_used_data":17528,"kb_used_omap":1,"kb_used_meta":857150,"kb_avail":92870456,"statfs":{"total":95995035648,"available":95099346944,"internally_reserved":0,"allocated":17948672,"data_stored":12947117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":877722059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1724,"seq":7404523618344,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":206720,"kb_used_data":14644,"kb_used_omap":123,"kb_used_meta":191940,"kb_avail":93538432,"statfs":{"total":95995035648,"available":95783354368,"internally_reserved":0,"allocated":14995456,"data_stored":9999017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":126809,"internal_metadata":196546727},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565797,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1323384,"kb_used_data":28652,"kb_used_omap":72,"kb_used_meta":1294647,"kb_avail":92421768,"statfs":{"total":95995035648,"available":94639890432,"internally_reserved":0,"allocated":29339648,"data_stored":24347226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":1325719022},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":1399,"seq":6008659247398,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1029464,"kb_used_data":62928,"kb_used_omap":131,"kb_used_meta":966332,"kb_avail":92715688,"statfs":{"total":95995035648,"available":94940864512,"internally_reserved":0,"allocated":64438272,"data_stored":59457385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134291,"internal_metadata":989524845},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743187187,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":981364,"kb_used_data":14952,"kb_used_omap":66,"kb_used_meta":966269,"kb_avail":92763788,"statfs":{"total":95995035648,"available":94990118912,"internally_reserved":0,"allocated":15310848,"data_stored":10321907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":989460139},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448826,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":929648,"kb_used_data":23144,"kb_used_omap":75,"kb_used_meta":906420,"kb_avail":92815504,"statfs":{"total":95995035648,"available":95043076096,"internally_reserved":0,"allocated":23699456,"data_stored":18711076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":928174271},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208758191,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1387384,"kb_used_data":11048,"kb_used_omap":1,"kb_used_meta":1376318,"kb_avail":92357768,"statfs":{"total":95995035648,"available":94574354432,"internally_reserved":0,"allocated":11313152,"data_stored":6323680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1409350086},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019827,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1378696,"kb_used_data":21496,"kb_used_omap":1,"kb_used_meta":1357182,"kb_avail":92366456,"statfs":{"total":95995035648,"available":94583250944,"internally_reserved":0,"allocated":22011904,"data_stored":17010823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1389754822},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12713984,"data_stored":12713984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":33820672,"data_stored":33816600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2883584,"data_stored":2883584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":131072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":58,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6619136,"data_stored":6619136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T18:25:56.785 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph status --format=json 2024-08-17T18:25:56.840 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:56.862 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:56.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:56.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:56.964 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:56.986 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:56.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:56.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:57.046 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:56 smithi136 ceph-mon[109861]: osdmap e1764: 8 total, 8 up, 8 in 2024-08-17T18:25:57.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:56 smithi136 ceph-mon[115078]: osdmap e1764: 8 total, 8 up, 8 in 2024-08-17T18:25:57.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:57.112 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:57.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:57.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:57.213 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:57.236 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:57.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:57.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:57.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:56 smithi165 ceph-mon[105517]: osdmap e1764: 8 total, 8 up, 8 in 2024-08-17T18:25:57.338 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:57.364 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:57.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:57.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:57.466 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:57.489 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:57.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:57.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:57.591 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:57.616 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:57.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:57.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:57.718 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:57.744 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:57.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:57.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:57.781 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:25:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:25:57] "GET /metrics HTTP/1.1" 200 36700 "" "Prometheus/2.51.0" 2024-08-17T18:25:57.846 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:57.868 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:57.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:57.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:57.969 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:57.995 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:57.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:57.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:58.080 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:57 smithi136 ceph-mon[109861]: pgmap v3518: 99 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 634 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 371 op/s; 423/78 objects degraded (542.308%); 2.6 MiB/s, 251 objects/s recovering 2024-08-17T18:25:58.080 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:57 smithi136 ceph-mon[115078]: pgmap v3518: 99 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 634 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 371 op/s; 423/78 objects degraded (542.308%); 2.6 MiB/s, 251 objects/s recovering 2024-08-17T18:25:58.097 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:58.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:58.119 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:58.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:58.220 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:58.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:58.242 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:58.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:57 smithi165 ceph-mon[105517]: pgmap v3518: 99 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 634 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 371 op/s; 423/78 objects degraded (542.308%); 2.6 MiB/s, 251 objects/s recovering 2024-08-17T18:25:58.343 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:58.365 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:58.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:58.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:58.466 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:25:58.488 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:58.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:58.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:25:58.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:58.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:58.617 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:58.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:58.718 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:25:58.739 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:58.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:58.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:25:58.841 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:58.867 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:58.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:58.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:58.969 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:25:58.991 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:58.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:58.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:25:58.995 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:58 smithi136 ceph-mon[109861]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 423/78 objects degraded (542.308%), 2 pgs degraded, 2 pgs undersized) 2024-08-17T18:25:58.996 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:58 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T18:25:58.996 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:58 smithi136 ceph-mon[115078]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 423/78 objects degraded (542.308%), 2 pgs degraded, 2 pgs undersized) 2024-08-17T18:25:58.996 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:58 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T18:25:59.093 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:25:59.114 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:25:59.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:59.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:25:59.194 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:25:59.195 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:25:59.216 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:25:59.239 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:25:59.242 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:59.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:59.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:25:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:58 smithi165 ceph-mon[105517]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 423/78 objects degraded (542.308%), 2 pgs degraded, 2 pgs undersized) 2024-08-17T18:25:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:58 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T18:25:59.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:59.370 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:59.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:59.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:59.472 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:25:59.498 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:59.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:59.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:25:59.600 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:59.625 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:59.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:59.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:59.727 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:25:59.753 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:59.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:25:59.847 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is set 2024-08-17T18:25:59.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:25:59.858 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:59 smithi136 ceph-mon[115078]: 1.0 scrub starts 2024-08-17T18:25:59.858 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:59 smithi136 ceph-mon[115078]: 1.0 scrub ok 2024-08-17T18:25:59.859 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:59 smithi136 ceph-mon[115078]: pgmap v3519: 99 pgs: 1 peering, 98 active+clean; 634 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 161 op/s; 472 KiB/s, 212 objects/s recovering 2024-08-17T18:25:59.859 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:59 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1150410738' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:25:59.859 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:25:59 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:25:59.859 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:59 smithi136 ceph-mon[109861]: 1.0 scrub starts 2024-08-17T18:25:59.859 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:59 smithi136 ceph-mon[109861]: 1.0 scrub ok 2024-08-17T18:25:59.859 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:59 smithi136 ceph-mon[109861]: pgmap v3519: 99 pgs: 1 peering, 98 active+clean; 634 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 161 op/s; 472 KiB/s, 212 objects/s recovering 2024-08-17T18:25:59.859 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:59 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1150410738' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:25:59.859 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:25:59 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:25:59.879 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:25:59.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:25:59.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:25:59.981 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:26:00.003 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:00.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:00.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:26:00.105 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:00.131 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:00.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:00.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:00.233 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:00.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:00.256 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:00.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:00.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:59 smithi165 ceph-mon[105517]: 1.0 scrub starts 2024-08-17T18:26:00.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:59 smithi165 ceph-mon[105517]: 1.0 scrub ok 2024-08-17T18:26:00.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:59 smithi165 ceph-mon[105517]: pgmap v3519: 99 pgs: 1 peering, 98 active+clean; 634 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 161 op/s; 472 KiB/s, 212 objects/s recovering 2024-08-17T18:26:00.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:59 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1150410738' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:26:00.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:25:59 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T18:26:00.357 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:26:00.383 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:00.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:00.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:26:00.484 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:26:00.508 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:00.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:00.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:26:00.610 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:00.632 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:00.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:00.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:00.734 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:26:00.759 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:00.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:00.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:26:00.860 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:26:00.883 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:00.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:00.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:26:00.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:00 smithi136 ceph-mon[109861]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:26:00.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:00 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:26:00.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:00 smithi136 ceph-mon[109861]: osdmap e1765: 8 total, 8 up, 8 in 2024-08-17T18:26:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:00 smithi136 ceph-mon[115078]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:26:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:00 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:26:00.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:00 smithi136 ceph-mon[115078]: osdmap e1765: 8 total, 8 up, 8 in 2024-08-17T18:26:00.984 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:01.007 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:01.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:01.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:01.108 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:26:01.132 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:01.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:01.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:26:01.234 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:26:01.258 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:01.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:01.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:26:01.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:00 smithi165 ceph-mon[105517]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:26:01.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:00 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T18:26:01.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:00 smithi165 ceph-mon[105517]: osdmap e1765: 8 total, 8 up, 8 in 2024-08-17T18:26:01.360 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:01.384 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:01.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:01.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:01.485 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:01.507 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:01.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:01.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:01.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:26:01.633 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:01.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:01.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:26:01.735 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:26:01.758 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:01.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:01.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:26:01.860 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:26:01.885 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:01.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:01.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:26:01.987 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:02.009 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:02.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:02.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:02.111 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:02.135 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:02.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:02.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:02.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:01 smithi136 ceph-mon[109861]: pgmap v3521: 99 pgs: 99 active+clean; 634 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 3 op/s; 43 KiB/s, 70 objects/s recovering 2024-08-17T18:26:02.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:01 smithi136 ceph-mon[115078]: pgmap v3521: 99 pgs: 99 active+clean; 634 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 3 op/s; 43 KiB/s, 70 objects/s recovering 2024-08-17T18:26:02.237 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:02.259 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:02.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:02.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:02.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:01 smithi165 ceph-mon[105517]: pgmap v3521: 99 pgs: 99 active+clean; 634 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 3 op/s; 43 KiB/s, 70 objects/s recovering 2024-08-17T18:26:02.361 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:26:02.387 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:02.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:02.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:26:02.489 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:02.514 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:02.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:02.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:02.615 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:26:02.641 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:02.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:02.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:26:02.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:26:02.767 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:02.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:02.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:26:02.869 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:02.891 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:02.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:02.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:02.993 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:26:03.014 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:03.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:03.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:26:03.116 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:26:03.185 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-17T18:26:03.185 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:26:03.206 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:03.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:03.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:26:03.308 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:03.330 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:03.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:03.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:03.431 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:26:03.432 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T18:26:03.458 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:03.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:03.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:26:03.559 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:03.581 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:03.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:03.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:03.683 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:03.709 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:03.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:03.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:03.811 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:03.833 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:03.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:03.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:03.935 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:26:03.956 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:03.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:03.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:26:04.006 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:26:04.007 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:26:04.007 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:26:04.007 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:26:04.017 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:26:04.017 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:26:04.033 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:26:04.033 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:26:04.050 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:26:04.050 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:26:04.058 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:04.067 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:26:04.067 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:26:04.080 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:04.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:04.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:04.084 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:26:04.084 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:26:04.101 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:26:04.101 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:26:04.101 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:03 smithi136 ceph-mon[109861]: pgmap v3522: 99 pgs: 99 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 0 op/s; 0 B/s, 53 objects/s recovering 2024-08-17T18:26:04.101 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:03 smithi136 ceph-mon[115078]: pgmap v3522: 99 pgs: 99 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 0 op/s; 0 B/s, 53 objects/s recovering 2024-08-17T18:26:04.181 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:26:04.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:04.204 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:04.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:26:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:03 smithi165 ceph-mon[105517]: pgmap v3522: 99 pgs: 99 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 0 op/s; 0 B/s, 53 objects/s recovering 2024-08-17T18:26:04.304 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:26:04.330 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:04.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:04.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:26:04.432 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:04.455 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:04.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:04.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:04.557 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:04.580 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:04.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:04.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:04.682 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:04.703 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:04.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:04.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:04.804 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:04.826 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:04.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:04.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:04.856 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'unique_pool_5' removed 2024-08-17T18:26:04.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:26:04.952 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:04.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:04.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:26:05.054 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:05.076 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:05.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:05.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:05.178 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:26:05.179 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:04 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2957856293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:26:05.179 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:04 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3935653743' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T18:26:05.179 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:04 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2957856293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:26:05.179 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:04 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3935653743' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T18:26:05.202 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:05.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:05.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:26:05.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:04 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2957856293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:26:05.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:04 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3935653743' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T18:26:05.304 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:26:05.326 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:05.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:05.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:26:05.428 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:26:05.449 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:05.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:05.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:26:05.551 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:26:05.572 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:05.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:05.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:26:05.674 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:05.695 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:05.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:05.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:05.735 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-17T18:26:05.797 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:26:05.818 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:05.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:05.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:26:05.920 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:26:05.941 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:05.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:05.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:26:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:05 smithi136 ceph-mon[109861]: pgmap v3523: 99 pgs: 99 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 47 objects/s recovering 2024-08-17T18:26:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:05 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3935653743' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T18:26:05.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:05 smithi136 ceph-mon[109861]: osdmap e1766: 8 total, 8 up, 8 in 2024-08-17T18:26:05.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:05 smithi136 ceph-mon[115078]: pgmap v3523: 99 pgs: 99 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 47 objects/s recovering 2024-08-17T18:26:05.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:05 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3935653743' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T18:26:05.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:05 smithi136 ceph-mon[115078]: osdmap e1766: 8 total, 8 up, 8 in 2024-08-17T18:26:06.043 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:06.065 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:06.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:06.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:06.167 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:26:06.188 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:06.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:06.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:26:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:05 smithi165 ceph-mon[105517]: pgmap v3523: 99 pgs: 99 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 47 objects/s recovering 2024-08-17T18:26:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:05 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3935653743' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T18:26:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:05 smithi165 ceph-mon[105517]: osdmap e1766: 8 total, 8 up, 8 in 2024-08-17T18:26:06.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:06.312 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:06.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:06.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:06.413 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:06.436 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:06.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:06.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:06.538 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:06.559 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:06.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:06.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:06.660 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:26:06.663 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:26:06.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:06.687 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:06.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:26:06.788 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:26:06.810 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:06.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:06.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:26:06.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:26:06.933 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:06.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:06.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:26:07.035 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:07.057 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:07.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:07.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:07.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:26:07.181 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:07.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:07.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:26:07.205 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T18:26:07.283 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:26:07.304 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:07.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:07.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:26:07.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:26:07.427 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:07.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:07.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:26:07.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:26:07.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:26:07.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:07 smithi136 ceph-mon[109861]: pgmap v3525: 83 pgs: 83 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:07.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:26:07.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:26:07.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:07 smithi136 ceph-mon[115078]: pgmap v3525: 83 pgs: 83 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:26:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:26:07.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:07 smithi165 ceph-mon[105517]: pgmap v3525: 83 pgs: 83 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:07.529 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:26:07.550 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:07.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:07.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:26:07.652 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:07.674 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:07.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:07.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:07.776 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:07.797 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:07.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:07.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:07.885 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:26:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:26:07] "GET /metrics HTTP/1.1" 200 34260 "" "Prometheus/2.51.0" 2024-08-17T18:26:07.899 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:07.910 INFO:teuthology.task.sequential:In sequential, running task print... 2024-08-17T18:26:07.910 INFO:teuthology.task.print:**** done end radosbench.yaml 2024-08-17T18:26:07.910 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-08-17T18:26:07.911 INFO:tasks.workunit:Pulling workunits from ref reef 2024-08-17T18:26:07.911 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-08-17T18:26:07.911 DEBUG:teuthology.orchestra.run.smithi136:> stat -- /home/ubuntu/cephtest/mnt.0 2024-08-17T18:26:07.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:07.921 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:07.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:07.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:07.946 INFO:teuthology.orchestra.run.smithi136.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-17T18:26:07.946 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-08-17T18:26:08.020 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-08-17T18:26:08.020 DEBUG:teuthology.orchestra.run.smithi136:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-08-17T18:26:08.023 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:26:08.046 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:08.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:08.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:26:08.076 INFO:tasks.workunit:timeout=3h 2024-08-17T18:26:08.077 INFO:tasks.workunit:cleanup=True 2024-08-17T18:26:08.078 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch reef https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2024-08-17T18:26:08.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:26:08.155 INFO:tasks.workunit.client.0.smithi136.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-08-17T18:26:08.170 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:08.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:08.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:26:08.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:26:08.299 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:08.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:08.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:26:08.401 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:26:08.424 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:08.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:08.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:26:08.526 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:08.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:08.549 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:08.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:08.650 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:08.675 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:08.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:08.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:08.777 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:26:08.798 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:08.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:08.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:26:08.900 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:08.922 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:08.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:08.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:09.023 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:26:09.049 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:09.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:09.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:26:09.151 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:26:09.177 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:09.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:09.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:26:09.279 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:26:09.304 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:09.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:09.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:26:09.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:26:09.430 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:09.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:09.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:26:09.532 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:26:09.556 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:09.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:09.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:26:09.570 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:26:09.570 INFO:teuthology.orchestra.run.smithi136.stdout:{"fsid":"8270436c-5cba-11ef-bccf-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":4065,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":1766,"num_osds":8,"num_up_osds":8,"osd_up_since":1723918955,"num_in_osds":8,"osd_in_since":1723918830,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":83}],"num_pgs":83,"num_pools":4,"num_objects":26,"data_bytes":595941,"bytes_used":8197058560,"bytes_avail":759763226624,"bytes_total":767960285184},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi136.ltsuqd","status":"up:active","gid":24557}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.136:8443/","prometheus":"http://172.21.15.136:9283/"}},"servicemap":{"epoch":44,"modified":"2024-08-17T18:25:16.623308+0000","services":{}},"progress_events":{}} 2024-08-17T18:26:09.658 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:26:09.680 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:09.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:09.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:26:09.782 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:09.807 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:09.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:09.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:09.909 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:26:09.932 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:09.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:09.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:26:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:09 smithi136 ceph-mon[109861]: pgmap v3526: 83 pgs: 83 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:09.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:09 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3682795377' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T18:26:09.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:09 smithi136 ceph-mon[115078]: pgmap v3526: 83 pgs: 83 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:09.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:09 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3682795377' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T18:26:10.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:09 smithi165 ceph-mon[105517]: pgmap v3526: 83 pgs: 83 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:10.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:09 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3682795377' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T18:26:10.017 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:26:10.017 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:26:10.034 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:10.058 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:10.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:10.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:10.160 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:26:10.181 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:10.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:10.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:26:10.283 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:26:10.309 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:10.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:10.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:26:10.411 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:26:10.436 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:10.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:10.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:26:10.538 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:10.560 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:10.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:10.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:10.572 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-08-17T18:26:10.572 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T18:26:10.661 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:10.687 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:10.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:10.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:10.707 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is set 2024-08-17T18:26:10.789 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:26:10.810 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:10.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:10.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:26:10.835 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:10 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1023336768' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:26:10.835 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:10 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:26:10.836 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:10 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1023336768' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:26:10.836 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:10 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:26:10.912 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:10.933 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:10.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:10.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:11.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:10 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1023336768' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:26:11.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:10 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:26:11.035 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:26:11.056 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:11.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:11.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:26:11.158 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:26:11.184 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:11.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:11.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:26:11.286 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:11.312 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:11.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:11.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:11.414 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:11.438 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:11.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:11.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:11.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:26:11.564 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:11.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:11.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:26:11.666 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:11.689 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:11.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:11.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:11.791 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:11.812 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:11.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:11.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:11.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:11.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:11.936 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:11.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:11.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:11 smithi136 ceph-mon[109861]: pgmap v3527: 83 pgs: 83 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:11.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:11 smithi136 ceph-mon[109861]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:26:11.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:11 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:26:11.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:11 smithi136 ceph-mon[109861]: osdmap e1767: 8 total, 8 up, 8 in 2024-08-17T18:26:11.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:11 smithi136 ceph-mon[115078]: pgmap v3527: 83 pgs: 83 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:11.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:11 smithi136 ceph-mon[115078]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:26:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:11 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:26:11.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:11 smithi136 ceph-mon[115078]: osdmap e1767: 8 total, 8 up, 8 in 2024-08-17T18:26:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:11 smithi165 ceph-mon[105517]: pgmap v3527: 83 pgs: 83 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:11 smithi165 ceph-mon[105517]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:26:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:11 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T18:26:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:11 smithi165 ceph-mon[105517]: osdmap e1767: 8 total, 8 up, 8 in 2024-08-17T18:26:12.037 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:26:12.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:12.058 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:12.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:26:12.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:26:12.185 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:12.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:12.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:26:12.287 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:12.308 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:12.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:12.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:12.410 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:26:12.436 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:12.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:12.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:26:12.538 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:12.560 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:12.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:12.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:12.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:26:12.688 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:12.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:12.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:26:12.790 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:26:12.816 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:12.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:12.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:26:12.917 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:26:12.942 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:12.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:12.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:26:12.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]} : dispatch 2024-08-17T18:26:12.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:12 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-08-17T18:26:12.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:12 smithi136 ceph-mon[115078]: osdmap e1768: 8 total, 8 up, 8 in 2024-08-17T18:26:12.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]} : dispatch 2024-08-17T18:26:12.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:12 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-08-17T18:26:12.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:12 smithi136 ceph-mon[109861]: osdmap e1768: 8 total, 8 up, 8 in 2024-08-17T18:26:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:12 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]} : dispatch 2024-08-17T18:26:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:12 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-08-17T18:26:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:12 smithi165 ceph-mon[105517]: osdmap e1768: 8 total, 8 up, 8 in 2024-08-17T18:26:13.044 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:13.065 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:13.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:13.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:13.167 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:13.189 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:13.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:13.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:13.291 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:26:13.317 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:13.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:13.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:26:13.418 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:13.441 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:13.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:13.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:13.543 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:13.569 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:13.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:13.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:13.671 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:26:13.697 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:13.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:13.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:26:13.799 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:26:13.820 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:13.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:13.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:26:13.849 DEBUG:teuthology.orchestra.run.smithi136:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-08-17T18:26:13.875 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:13 smithi136 ceph-mon[115078]: pgmap v3530: 83 pgs: 83 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:13.875 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:13 smithi136 ceph-mon[115078]: osdmap e1769: 8 total, 8 up, 8 in 2024-08-17T18:26:13.876 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:13 smithi136 ceph-mon[109861]: pgmap v3530: 83 pgs: 83 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:13.876 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:13 smithi136 ceph-mon[109861]: osdmap e1769: 8 total, 8 up, 8 in 2024-08-17T18:26:13.898 INFO:tasks.workunit.client.0.smithi136.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-08-17T18:26:13.901 INFO:tasks.workunit.client.0.smithi136.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-17T18:26:13.901 INFO:tasks.workunit.client.0.smithi136.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-08-17T18:26:13.923 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:13.944 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:13.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:13.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:14.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:13 smithi165 ceph-mon[105517]: pgmap v3530: 83 pgs: 83 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:13 smithi165 ceph-mon[105517]: osdmap e1769: 8 total, 8 up, 8 in 2024-08-17T18:26:14.045 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:14.066 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:14.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:14.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:14.140 INFO:tasks.workunit.client.0.smithi136.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-08-17T18:26:14.168 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:26:14.190 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:14.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:14.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:26:14.220 INFO:tasks.workunit.client.0.smithi136.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-08-17T18:26:14.259 INFO:tasks.workunit.client.0.smithi136.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-17T18:26:14.261 INFO:tasks.workunit.client.0.smithi136.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-17T18:26:14.261 INFO:tasks.workunit.client.0.smithi136.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-08-17T18:26:14.291 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:14.299 INFO:tasks.workunit.client.0.smithi136.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-17T18:26:14.317 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:14.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:14.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:14.324 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-17T18:26:14.324 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-08-17T18:26:14.394 INFO:tasks.workunit:Running workunits matching cls/test_cls_rbd.sh on client.0... 2024-08-17T18:26:14.395 INFO:tasks.workunit:Running workunit cls/test_cls_rbd.sh... 2024-08-17T18:26:14.395 DEBUG:teuthology.orchestra.run.smithi136:workunit test cls/test_cls_rbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh 2024-08-17T18:26:14.419 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:26:14.456 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:14.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:14.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:26:14.558 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:26:14.584 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:14.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:14.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:26:14.676 INFO:tasks.workunit.client.0.smithi136.stdout:Running main() from gmock_main.cc 2024-08-17T18:26:14.677 INFO:tasks.workunit.client.0.smithi136.stdout:[==========] Running 70 tests from 1 test suite. 2024-08-17T18:26:14.677 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] Global test environment set-up. 2024-08-17T18:26:14.677 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 70 tests from TestClsRbd 2024-08-17T18:26:14.686 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:26:14.711 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:14.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:14.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:26:14.813 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:26:14.839 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:14.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:14.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:26:14.930 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:26:14.931 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-17T18:26:14.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-17T18:26:14.931 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:26:14.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.931 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:93198 58.9 58:991f0d5b:::benchmark_data_smithi136_442124_object93197:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1715)", 2024-08-17T18:26:14.931 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:03.668789+0000", 2024-08-17T18:26:14.931 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 251.251219263, 2024-08-17T18:26:14.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2031386620000002, 2024-08-17T18:26:14.932 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:14.932 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:14.932 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:14.932 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:14.932 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:14.932 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:14.932 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 93198 2024-08-17T18:26:14.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.932 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:14.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:14.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.668789+0000", 2024-08-17T18:26:14.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:14.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:14.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.668789+0000", 2024-08-17T18:26:14.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:14.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:14.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.668791+0000", 2024-08-17T18:26:14.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4930000000000001e-06 2024-08-17T18:26:14.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:14.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.668818+0000", 2024-08-17T18:26:14.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6800000000000001e-05 2024-08-17T18:26:14.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:14.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.668834+0000", 2024-08-17T18:26:14.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5781e-05 2024-08-17T18:26:14.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:14.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.668845+0000", 2024-08-17T18:26:14.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0888e-05 2024-08-17T18:26:14.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:14.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.676822+0000", 2024-08-17T18:26:14.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0079767950000000001 2024-08-17T18:26:14.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:14.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.676832+0000", 2024-08-17T18:26:14.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0832999999999999e-05 2024-08-17T18:26:14.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:14.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.654008+0000", 2024-08-17T18:26:14.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.977175872 2024-08-17T18:26:14.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:14.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.654023+0000", 2024-08-17T18:26:14.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4810999999999999e-05 2024-08-17T18:26:14.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:14.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.851917+0000", 2024-08-17T18:26:14.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.19789346 2024-08-17T18:26:14.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:14.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.852093+0000", 2024-08-17T18:26:14.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017626100000000001 2024-08-17T18:26:14.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4", 2024-08-17T18:26:14.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.852608+0000", 2024-08-17T18:26:14.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00051531899999999998 2024-08-17T18:26:14.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:14.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.866931+0000", 2024-08-17T18:26:14.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014322516 2024-08-17T18:26:14.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:14.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.866947+0000", 2024-08-17T18:26:14.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5875999999999998e-05 2024-08-17T18:26:14.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:14.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.866982+0000", 2024-08-17T18:26:14.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5522999999999999e-05 2024-08-17T18:26:14.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:14.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.867036+0000", 2024-08-17T18:26:14.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3925000000000003e-05 2024-08-17T18:26:14.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:14.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.871928+0000", 2024-08-17T18:26:14.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0048915089999999996 2024-08-17T18:26:14.940 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:14.940 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:14.940 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:14.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.940 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:93206 58.9 58:99f4cc79:::benchmark_data_smithi136_442124_object93205:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1715)", 2024-08-17T18:26:14.940 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:03.682712+0000", 2024-08-17T18:26:14.940 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 251.237296241, 2024-08-17T18:26:14.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2199088979999999, 2024-08-17T18:26:14.940 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:14.940 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:14.941 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:14.941 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:14.941 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:14.941 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:14.941 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 93206 2024-08-17T18:26:14.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.941 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:14.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:14.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.682712+0000", 2024-08-17T18:26:14.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:14.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:14.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.682712+0000", 2024-08-17T18:26:14.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:14.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:14.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.682714+0000", 2024-08-17T18:26:14.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8059999999999999e-06 2024-08-17T18:26:14.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:14.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.682740+0000", 2024-08-17T18:26:14.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5891999999999999e-05 2024-08-17T18:26:14.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:14.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.682755+0000", 2024-08-17T18:26:14.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5727e-05 2024-08-17T18:26:14.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:14.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.682763+0000", 2024-08-17T18:26:14.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7319999999999998e-06 2024-08-17T18:26:14.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:14.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.685209+0000", 2024-08-17T18:26:14.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0024458129999999998 2024-08-17T18:26:14.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:14.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.685221+0000", 2024-08-17T18:26:14.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2026e-05 2024-08-17T18:26:14.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:14.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.654053+0000", 2024-08-17T18:26:14.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.96883200400000002 2024-08-17T18:26:14.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:14.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.654061+0000", 2024-08-17T18:26:14.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8900000000000007e-06 2024-08-17T18:26:14.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:14.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.863666+0000", 2024-08-17T18:26:14.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.209605482 2024-08-17T18:26:14.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:14.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.863838+0000", 2024-08-17T18:26:14.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017137200000000001 2024-08-17T18:26:14.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4", 2024-08-17T18:26:14.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.864014+0000", 2024-08-17T18:26:14.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017631299999999999 2024-08-17T18:26:14.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:14.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.875384+0000", 2024-08-17T18:26:14.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011370488 2024-08-17T18:26:14.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:14.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.875434+0000", 2024-08-17T18:26:14.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9812999999999999e-05 2024-08-17T18:26:14.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:14.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.897146+0000", 2024-08-17T18:26:14.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.021711672000000001 2024-08-17T18:26:14.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:14.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.897247+0000", 2024-08-17T18:26:14.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010138599999999999 2024-08-17T18:26:14.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:14.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.902621+0000", 2024-08-17T18:26:14.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0053734819999999997 2024-08-17T18:26:14.949 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:14.949 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:14.949 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:14.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.949 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:93209 58.9 58:9acddeb4:::benchmark_data_smithi136_442124_object93208:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1715)", 2024-08-17T18:26:14.949 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:03.685070+0000", 2024-08-17T18:26:14.949 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 251.234937938, 2024-08-17T18:26:14.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2247561199999999, 2024-08-17T18:26:14.949 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:14.949 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:14.949 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:14.950 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:14.950 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:14.950 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:14.950 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 93209 2024-08-17T18:26:14.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.950 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:14.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:14.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.685070+0000", 2024-08-17T18:26:14.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:14.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:14.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.685070+0000", 2024-08-17T18:26:14.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:14.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:14.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.685071+0000", 2024-08-17T18:26:14.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.89e-07 2024-08-17T18:26:14.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:14.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.685094+0000", 2024-08-17T18:26:14.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3706e-05 2024-08-17T18:26:14.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:14.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.685102+0000", 2024-08-17T18:26:14.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2819999999999999e-06 2024-08-17T18:26:14.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:14.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.685107+0000", 2024-08-17T18:26:14.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6799999999999998e-06 2024-08-17T18:26:14.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:14.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.685246+0000", 2024-08-17T18:26:14.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000138653 2024-08-17T18:26:14.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:14.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.685254+0000", 2024-08-17T18:26:14.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5800000000000003e-06 2024-08-17T18:26:14.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:14.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.654085+0000", 2024-08-17T18:26:14.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.96883142799999999 2024-08-17T18:26:14.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:14.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.654092+0000", 2024-08-17T18:26:14.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0589999999999997e-06 2024-08-17T18:26:14.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:14.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.871987+0000", 2024-08-17T18:26:14.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.217894689 2024-08-17T18:26:14.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:14.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.872145+0000", 2024-08-17T18:26:14.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015860599999999999 2024-08-17T18:26:14.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4", 2024-08-17T18:26:14.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.872310+0000", 2024-08-17T18:26:14.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016492699999999999 2024-08-17T18:26:14.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:14.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.882463+0000", 2024-08-17T18:26:14.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010152437 2024-08-17T18:26:14.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:14.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.902759+0000", 2024-08-17T18:26:14.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020296682999999999 2024-08-17T18:26:14.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:14.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.902845+0000", 2024-08-17T18:26:14.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5735000000000005e-05 2024-08-17T18:26:14.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:14.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.903012+0000", 2024-08-17T18:26:14.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016666599999999999 2024-08-17T18:26:14.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:14.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.909826+0000", 2024-08-17T18:26:14.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0068145000000000002 2024-08-17T18:26:14.957 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:14.957 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:14.957 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:14.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.958 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:93227 58.9 58:9c526ec9:::benchmark_data_smithi136_442124_object93226:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1715)", 2024-08-17T18:26:14.958 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:03.701109+0000", 2024-08-17T18:26:14.958 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 251.21889945199999, 2024-08-17T18:26:14.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2148894810000002, 2024-08-17T18:26:14.958 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:14.958 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:14.958 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:14.958 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:14.958 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:14.958 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:14.958 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 93227 2024-08-17T18:26:14.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.959 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:14.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:14.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.701109+0000", 2024-08-17T18:26:14.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:14.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:14.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.701109+0000", 2024-08-17T18:26:14.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:14.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:14.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.701109+0000", 2024-08-17T18:26:14.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6400000000000002e-07 2024-08-17T18:26:14.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:14.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.701130+0000", 2024-08-17T18:26:14.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0315000000000001e-05 2024-08-17T18:26:14.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:14.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.701136+0000", 2024-08-17T18:26:14.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9009999999999997e-06 2024-08-17T18:26:14.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:14.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.701141+0000", 2024-08-17T18:26:14.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8350000000000003e-06 2024-08-17T18:26:14.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:14.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.701196+0000", 2024-08-17T18:26:14.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4879000000000001e-05 2024-08-17T18:26:14.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:14.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.701206+0000", 2024-08-17T18:26:14.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6299999999999993e-06 2024-08-17T18:26:14.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:14.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.654116+0000", 2024-08-17T18:26:14.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.95291036900000003 2024-08-17T18:26:14.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:14.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.654123+0000", 2024-08-17T18:26:14.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2860000000000002e-06 2024-08-17T18:26:14.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:14.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.875466+0000", 2024-08-17T18:26:14.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2213430459999999 2024-08-17T18:26:14.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:14.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.875619+0000", 2024-08-17T18:26:14.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015246199999999999 2024-08-17T18:26:14.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4", 2024-08-17T18:26:14.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.875786+0000", 2024-08-17T18:26:14.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016685499999999999 2024-08-17T18:26:14.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:14.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.882542+0000", 2024-08-17T18:26:14.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.006756115 2024-08-17T18:26:14.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:14.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.902773+0000", 2024-08-17T18:26:14.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020231398000000001 2024-08-17T18:26:14.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:14.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.909882+0000", 2024-08-17T18:26:14.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071083029999999998 2024-08-17T18:26:14.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:14.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.909938+0000", 2024-08-17T18:26:14.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6504e-05 2024-08-17T18:26:14.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:14.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.915998+0000", 2024-08-17T18:26:14.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0060600189999999998 2024-08-17T18:26:14.966 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:14.966 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:14.966 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:14.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.966 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:93238 58.9 58:9041c1b1:::benchmark_data_smithi136_442124_object93237:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1715)", 2024-08-17T18:26:14.966 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:03.709016+0000", 2024-08-17T18:26:14.966 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 251.21099232200001, 2024-08-17T18:26:14.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2197829520000001, 2024-08-17T18:26:14.967 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:14.967 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:14.967 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:14.967 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:14.967 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:14.967 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:14.967 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 93238 2024-08-17T18:26:14.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.967 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:14.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:14.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.709016+0000", 2024-08-17T18:26:14.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:14.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:14.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.709016+0000", 2024-08-17T18:26:14.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:14.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:14.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.709016+0000", 2024-08-17T18:26:14.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7399999999999998e-07 2024-08-17T18:26:14.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:14.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.709035+0000", 2024-08-17T18:26:14.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9191e-05 2024-08-17T18:26:14.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:14.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.709042+0000", 2024-08-17T18:26:14.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.99e-06 2024-08-17T18:26:14.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:14.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.709047+0000", 2024-08-17T18:26:14.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9649999999999999e-06 2024-08-17T18:26:14.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:14.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.709101+0000", 2024-08-17T18:26:14.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3943000000000002e-05 2024-08-17T18:26:14.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:14.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.709111+0000", 2024-08-17T18:26:14.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3999999999999998e-06 2024-08-17T18:26:14.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:14.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.654146+0000", 2024-08-17T18:26:14.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.94503481 2024-08-17T18:26:14.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:14.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.654153+0000", 2024-08-17T18:26:14.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1960000000000002e-06 2024-08-17T18:26:14.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:14.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.879073+0000", 2024-08-17T18:26:14.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.224920027 2024-08-17T18:26:14.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:14.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.879214+0000", 2024-08-17T18:26:14.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014135299999999999 2024-08-17T18:26:14.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4", 2024-08-17T18:26:14.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.879371+0000", 2024-08-17T18:26:14.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0001572 2024-08-17T18:26:14.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:14.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.902676+0000", 2024-08-17T18:26:14.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023304281999999999 2024-08-17T18:26:14.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:14.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.902780+0000", 2024-08-17T18:26:14.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104861 2024-08-17T18:26:14.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:14.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.922365+0000", 2024-08-17T18:26:14.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019584793999999999 2024-08-17T18:26:14.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:14.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.922481+0000", 2024-08-17T18:26:14.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116043 2024-08-17T18:26:14.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:14.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.928799+0000", 2024-08-17T18:26:14.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.006317423 2024-08-17T18:26:14.975 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:14.975 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:14.975 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:14.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.975 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:93242 58.9 58:93c1b2d1:::benchmark_data_smithi136_442124_object93241:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1715)", 2024-08-17T18:26:14.975 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:03.713761+0000", 2024-08-17T18:26:14.975 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 251.20624696199999, 2024-08-17T18:26:14.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2211944949999998, 2024-08-17T18:26:14.975 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:14.976 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:14.976 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:14.976 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:14.976 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:14.976 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:14.976 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 93242 2024-08-17T18:26:14.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.976 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:14.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:14.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.713761+0000", 2024-08-17T18:26:14.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:14.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:14.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.713761+0000", 2024-08-17T18:26:14.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:14.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:14.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.713762+0000", 2024-08-17T18:26:14.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5799999999999998e-07 2024-08-17T18:26:14.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:14.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.713786+0000", 2024-08-17T18:26:14.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4338999999999999e-05 2024-08-17T18:26:14.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:14.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.713797+0000", 2024-08-17T18:26:14.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0287e-05 2024-08-17T18:26:14.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:14.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.713804+0000", 2024-08-17T18:26:14.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9360000000000002e-06 2024-08-17T18:26:14.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:14.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.713865+0000", 2024-08-17T18:26:14.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1456000000000002e-05 2024-08-17T18:26:14.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:14.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.713874+0000", 2024-08-17T18:26:14.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9730000000000003e-06 2024-08-17T18:26:14.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:14.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.654175+0000", 2024-08-17T18:26:14.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.94030137199999997 2024-08-17T18:26:14.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:14.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.654183+0000", 2024-08-17T18:26:14.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1369999999999996e-06 2024-08-17T18:26:14.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:14.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.882589+0000", 2024-08-17T18:26:14.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.228406648 2024-08-17T18:26:14.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:14.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.882759+0000", 2024-08-17T18:26:14.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016988300000000001 2024-08-17T18:26:14.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4", 2024-08-17T18:26:14.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.882924+0000", 2024-08-17T18:26:14.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016455500000000001 2024-08-17T18:26:14.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:14.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.902787+0000", 2024-08-17T18:26:14.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019863461999999998 2024-08-17T18:26:14.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:14.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.928853+0000", 2024-08-17T18:26:14.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.026066143999999999 2024-08-17T18:26:14.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:14.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.928901+0000", 2024-08-17T18:26:14.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7540000000000002e-05 2024-08-17T18:26:14.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.928965+0000", 2024-08-17T18:26:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4034000000000004e-05 2024-08-17T18:26:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.934956+0000", 2024-08-17T18:26:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0059907709999999998 2024-08-17T18:26:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.984 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:93248 58.9 58:970df5d5:::benchmark_data_smithi136_442124_object93247:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1715)", 2024-08-17T18:26:14.984 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:03.717909+0000", 2024-08-17T18:26:14.984 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 251.20209894600001, 2024-08-17T18:26:14.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2248920289999998, 2024-08-17T18:26:14.984 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:14.984 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:14.984 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:14.984 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:14.984 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:14.984 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:14.984 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 93248 2024-08-17T18:26:14.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.984 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:14.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:14.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.717909+0000", 2024-08-17T18:26:14.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:14.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:14.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.717909+0000", 2024-08-17T18:26:14.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:14.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:14.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.717910+0000", 2024-08-17T18:26:14.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1999999999999998e-07 2024-08-17T18:26:14.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:14.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.717933+0000", 2024-08-17T18:26:14.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2908000000000002e-05 2024-08-17T18:26:14.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:14.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.717959+0000", 2024-08-17T18:26:14.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5741e-05 2024-08-17T18:26:14.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:14.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.717965+0000", 2024-08-17T18:26:14.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2940000000000002e-06 2024-08-17T18:26:14.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:14.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.718021+0000", 2024-08-17T18:26:14.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5478e-05 2024-08-17T18:26:14.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:14.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.718029+0000", 2024-08-17T18:26:14.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6240000000000008e-06 2024-08-17T18:26:14.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:14.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.654204+0000", 2024-08-17T18:26:14.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93617483199999996 2024-08-17T18:26:14.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:14.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.654211+0000", 2024-08-17T18:26:14.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9290000000000001e-06 2024-08-17T18:26:14.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:14.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.889085+0000", 2024-08-17T18:26:14.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2348744869999999 2024-08-17T18:26:14.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:14.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.889451+0000", 2024-08-17T18:26:14.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036590700000000001 2024-08-17T18:26:14.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4", 2024-08-17T18:26:14.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.889644+0000", 2024-08-17T18:26:14.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019240699999999999 2024-08-17T18:26:14.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:14.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.902794+0000", 2024-08-17T18:26:14.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013150411000000001 2024-08-17T18:26:14.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:14.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.935630+0000", 2024-08-17T18:26:14.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.032835504000000001 2024-08-17T18:26:14.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:14.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.935677+0000", 2024-08-17T18:26:14.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7329e-05 2024-08-17T18:26:14.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:14.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.935856+0000", 2024-08-17T18:26:14.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017913 2024-08-17T18:26:14.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:14.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.942801+0000", 2024-08-17T18:26:14.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0069451280000000001 2024-08-17T18:26:14.991 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:14.991 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:14.991 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:14.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.992 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:14.993 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:26:14.994 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:93264 58.9 58:91fa1de9:::benchmark_data_smithi136_442124_object93263:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1715)", 2024-08-17T18:26:14.994 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:03.749514+0000", 2024-08-17T18:26:14.994 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 251.170494596, 2024-08-17T18:26:14.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2016387609999999, 2024-08-17T18:26:14.995 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:14.995 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:14.995 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:14.995 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:14.995 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:14.995 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:14.995 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 93264 2024-08-17T18:26:14.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.995 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:14.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:14.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.749514+0000", 2024-08-17T18:26:14.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:14.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:14.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.749514+0000", 2024-08-17T18:26:14.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:14.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:14.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.749514+0000", 2024-08-17T18:26:14.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4099999999999998e-07 2024-08-17T18:26:14.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:14.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.749533+0000", 2024-08-17T18:26:14.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9042000000000002e-05 2024-08-17T18:26:14.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:14.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.749543+0000", 2024-08-17T18:26:14.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3710000000000006e-06 2024-08-17T18:26:14.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:14.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.749548+0000", 2024-08-17T18:26:14.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9049999999999996e-06 2024-08-17T18:26:14.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:14.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.749594+0000", 2024-08-17T18:26:14.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6063000000000003e-05 2024-08-17T18:26:14.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:14.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.749601+0000", 2024-08-17T18:26:14.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1369999999999996e-06 2024-08-17T18:26:14.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:14.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.654233+0000", 2024-08-17T18:26:14.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.90463236400000002 2024-08-17T18:26:14.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:14.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.654241+0000", 2024-08-17T18:26:14.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.469e-06 2024-08-17T18:26:14.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:14.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:14.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:14.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.893176+0000", 2024-08-17T18:26:14.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2389356389999999 2024-08-17T18:26:15.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:15.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.893383+0000", 2024-08-17T18:26:15.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020693799999999999 2024-08-17T18:26:15.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4", 2024-08-17T18:26:15.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.893574+0000", 2024-08-17T18:26:15.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019041699999999999 2024-08-17T18:26:15.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:15.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.902801+0000", 2024-08-17T18:26:15.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0092272710000000004 2024-08-17T18:26:15.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.942849+0000", 2024-08-17T18:26:15.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040047632999999999 2024-08-17T18:26:15.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.942886+0000", 2024-08-17T18:26:15.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7289999999999997e-05 2024-08-17T18:26:15.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:15.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.943078+0000", 2024-08-17T18:26:15.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192451 2024-08-17T18:26:15.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:15.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.951152+0000", 2024-08-17T18:26:15.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0080740299999999994 2024-08-17T18:26:15.002 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.002 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:15.002 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.003 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:117430 58.a 58:52078c2b:::benchmark_data_smithi136_442124_object3814:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1724)", 2024-08-17T18:26:15.003 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:35.952289+0000", 2024-08-17T18:26:15.003 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 218.96771889300001, 2024-08-17T18:26:15.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0742568819999998, 2024-08-17T18:26:15.003 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:15.003 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:15.003 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:15.003 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:15.003 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:15.003 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:15.003 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 117430 2024-08-17T18:26:15.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.003 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:15.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:15.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952289+0000", 2024-08-17T18:26:15.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:15.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:15.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952289+0000", 2024-08-17T18:26:15.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:15.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:15.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952291+0000", 2024-08-17T18:26:15.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6929999999999999e-06 2024-08-17T18:26:15.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:15.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952296+0000", 2024-08-17T18:26:15.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3310000000000003e-06 2024-08-17T18:26:15.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:15.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952313+0000", 2024-08-17T18:26:15.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6500000000000001e-05 2024-08-17T18:26:15.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:15.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952320+0000", 2024-08-17T18:26:15.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.092e-06 2024-08-17T18:26:15.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952384+0000", 2024-08-17T18:26:15.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3952e-05 2024-08-17T18:26:15.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:15.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952393+0000", 2024-08-17T18:26:15.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1929999999999999e-06 2024-08-17T18:26:15.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.832215+0000", 2024-08-17T18:26:15.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.87982240700000003 2024-08-17T18:26:15.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:15.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.832228+0000", 2024-08-17T18:26:15.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2401e-05 2024-08-17T18:26:15.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.008785+0000", 2024-08-17T18:26:15.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.176557139 2024-08-17T18:26:15.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:15.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.008918+0000", 2024-08-17T18:26:15.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000133504 2024-08-17T18:26:15.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6,7", 2024-08-17T18:26:15.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.009053+0000", 2024-08-17T18:26:15.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000134432 2024-08-17T18:26:15.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.015178+0000", 2024-08-17T18:26:15.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0061246590000000002 2024-08-17T18:26:15.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:15.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.021507+0000", 2024-08-17T18:26:15.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063292510000000001 2024-08-17T18:26:15.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.022930+0000", 2024-08-17T18:26:15.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014231739999999999 2024-08-17T18:26:15.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:15.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.023084+0000", 2024-08-17T18:26:15.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015423300000000001 2024-08-17T18:26:15.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:15.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.026546+0000", 2024-08-17T18:26:15.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034619210000000002 2024-08-17T18:26:15.010 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.010 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:15.010 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.011 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:171648 58.a 58:504ec6a9:::benchmark_data_smithi136_442124_object58032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1739)", 2024-08-17T18:26:15.011 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:58.705108+0000", 2024-08-17T18:26:15.011 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 136.21489983199999, 2024-08-17T18:26:15.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2107198619999999, 2024-08-17T18:26:15.011 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:15.011 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:15.011 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:26:15.011 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:15.011 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:15.011 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:15.011 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 171648 2024-08-17T18:26:15.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.011 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:15.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:15.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.705108+0000", 2024-08-17T18:26:15.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:15.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:15.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.705108+0000", 2024-08-17T18:26:15.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:15.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:15.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.705109+0000", 2024-08-17T18:26:15.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0200000000000001e-07 2024-08-17T18:26:15.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:15.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.705113+0000", 2024-08-17T18:26:15.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3420000000000001e-06 2024-08-17T18:26:15.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:15.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.705122+0000", 2024-08-17T18:26:15.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.422e-06 2024-08-17T18:26:15.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:15.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.705128+0000", 2024-08-17T18:26:15.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2990000000000002e-06 2024-08-17T18:26:15.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.718062+0000", 2024-08-17T18:26:15.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012934034000000001 2024-08-17T18:26:15.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:15.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.718191+0000", 2024-08-17T18:26:15.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012908900000000001 2024-08-17T18:26:15.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6,7", 2024-08-17T18:26:15.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.718323+0000", 2024-08-17T18:26:15.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013170400000000001 2024-08-17T18:26:15.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.778486+0000", 2024-08-17T18:26:15.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.060162629000000002 2024-08-17T18:26:15.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:15.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.778497+0000", 2024-08-17T18:26:15.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1338999999999999e-05 2024-08-17T18:26:15.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:59.723819+0000", 2024-08-17T18:26:15.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.94532170000000004 2024-08-17T18:26:15.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:15.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:59.723831+0000", 2024-08-17T18:26:15.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2163e-05 2024-08-17T18:26:15.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.901835+0000", 2024-08-17T18:26:15.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.17800467 2024-08-17T18:26:15.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for missing object", 2024-08-17T18:26:15.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.902006+0000", 2024-08-17T18:26:15.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017078199999999999 2024-08-17T18:26:15.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.915757+0000", 2024-08-17T18:26:15.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013750848 2024-08-17T18:26:15.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:15.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.915828+0000", 2024-08-17T18:26:15.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1038999999999995e-05 2024-08-17T18:26:15.018 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.018 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:15.018 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.019 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:171683 58.a 58:5da76ac2:::benchmark_data_smithi136_442124_object58067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1740)", 2024-08-17T18:26:15.019 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:58.799338+0000", 2024-08-17T18:26:15.019 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 136.120670195, 2024-08-17T18:26:15.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.117701743, 2024-08-17T18:26:15.019 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:15.019 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:15.019 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:15.019 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:15.019 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:15.019 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:15.019 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 171683 2024-08-17T18:26:15.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.019 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:15.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:15.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.799338+0000", 2024-08-17T18:26:15.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:15.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:15.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.799338+0000", 2024-08-17T18:26:15.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:15.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:15.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.799339+0000", 2024-08-17T18:26:15.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.333e-06 2024-08-17T18:26:15.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:15.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.799344+0000", 2024-08-17T18:26:15.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2140000000000004e-06 2024-08-17T18:26:15.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:15.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.799358+0000", 2024-08-17T18:26:15.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3509000000000001e-05 2024-08-17T18:26:15.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:15.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.799366+0000", 2024-08-17T18:26:15.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7330000000000003e-06 2024-08-17T18:26:15.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.799434+0000", 2024-08-17T18:26:15.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7847000000000003e-05 2024-08-17T18:26:15.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:15.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.799445+0000", 2024-08-17T18:26:15.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1708e-05 2024-08-17T18:26:15.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:59.723867+0000", 2024-08-17T18:26:15.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.924421567 2024-08-17T18:26:15.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:15.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:59.723876+0000", 2024-08-17T18:26:15.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9779999999999994e-06 2024-08-17T18:26:15.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.902919+0000", 2024-08-17T18:26:15.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1790433149999999 2024-08-17T18:26:15.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:15.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.903057+0000", 2024-08-17T18:26:15.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013749500000000001 2024-08-17T18:26:15.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6,7", 2024-08-17T18:26:15.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.903217+0000", 2024-08-17T18:26:15.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016058500000000001 2024-08-17T18:26:15.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.908875+0000", 2024-08-17T18:26:15.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0056580099999999998 2024-08-17T18:26:15.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:15.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.908892+0000", 2024-08-17T18:26:15.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6470999999999999e-05 2024-08-17T18:26:15.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.915958+0000", 2024-08-17T18:26:15.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0070666879999999998 2024-08-17T18:26:15.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.916021+0000", 2024-08-17T18:26:15.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2261999999999994e-05 2024-08-17T18:26:15.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:15.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.916076+0000", 2024-08-17T18:26:15.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5569000000000003e-05 2024-08-17T18:26:15.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:15.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.917040+0000", 2024-08-17T18:26:15.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00096345899999999995 2024-08-17T18:26:15.026 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.027 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:15.027 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.027 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:171694 58.a 58:5c6ce09a:::benchmark_data_smithi136_442124_object58078:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1740)", 2024-08-17T18:26:15.027 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:58.814907+0000", 2024-08-17T18:26:15.027 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 136.10510133299999, 2024-08-17T18:26:15.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.103424994, 2024-08-17T18:26:15.027 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:15.027 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:15.027 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:15.027 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:15.027 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:15.027 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:15.028 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 171694 2024-08-17T18:26:15.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.028 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:15.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:15.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.814907+0000", 2024-08-17T18:26:15.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:15.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:15.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.814907+0000", 2024-08-17T18:26:15.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:15.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:15.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.814908+0000", 2024-08-17T18:26:15.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2270000000000001e-06 2024-08-17T18:26:15.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:15.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.814912+0000", 2024-08-17T18:26:15.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2350000000000001e-06 2024-08-17T18:26:15.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:15.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.814923+0000", 2024-08-17T18:26:15.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1072e-05 2024-08-17T18:26:15.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:15.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.814930+0000", 2024-08-17T18:26:15.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.917e-06 2024-08-17T18:26:15.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.815053+0000", 2024-08-17T18:26:15.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012267199999999999 2024-08-17T18:26:15.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:15.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.815063+0000", 2024-08-17T18:26:15.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9170000000000001e-06 2024-08-17T18:26:15.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:59.723901+0000", 2024-08-17T18:26:15.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.90883803600000002 2024-08-17T18:26:15.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:15.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:59.723909+0000", 2024-08-17T18:26:15.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9789999999999993e-06 2024-08-17T18:26:15.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.906175+0000", 2024-08-17T18:26:15.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1822659820000001 2024-08-17T18:26:15.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:15.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.906263+0000", 2024-08-17T18:26:15.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8383999999999999e-05 2024-08-17T18:26:15.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6,7", 2024-08-17T18:26:15.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.906416+0000", 2024-08-17T18:26:15.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000152931 2024-08-17T18:26:15.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:15.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.913623+0000", 2024-08-17T18:26:15.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0072071069999999999 2024-08-17T18:26:15.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.913760+0000", 2024-08-17T18:26:15.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013678799999999999 2024-08-17T18:26:15.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.917105+0000", 2024-08-17T18:26:15.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033445940000000002 2024-08-17T18:26:15.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.917221+0000", 2024-08-17T18:26:15.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116693 2024-08-17T18:26:15.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:15.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.917309+0000", 2024-08-17T18:26:15.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7737000000000004e-05 2024-08-17T18:26:15.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:15.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.918332+0000", 2024-08-17T18:26:15.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010227229999999999 2024-08-17T18:26:15.035 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.035 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:15.035 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.035 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:171700 58.a 58:53093524:::benchmark_data_smithi136_442124_object58084:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1740)", 2024-08-17T18:26:15.035 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:58.819394+0000", 2024-08-17T18:26:15.035 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 136.10061415199999, 2024-08-17T18:26:15.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1013540040000001, 2024-08-17T18:26:15.036 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:15.036 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:15.036 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:15.036 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:15.036 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:15.036 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:15.036 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 171700 2024-08-17T18:26:15.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.036 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:15.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:15.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.819394+0000", 2024-08-17T18:26:15.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:15.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:15.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.819394+0000", 2024-08-17T18:26:15.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:15.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:15.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.819395+0000", 2024-08-17T18:26:15.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2020000000000001e-06 2024-08-17T18:26:15.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:15.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.819400+0000", 2024-08-17T18:26:15.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3390000000000003e-06 2024-08-17T18:26:15.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:15.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.819409+0000", 2024-08-17T18:26:15.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5219999999999996e-06 2024-08-17T18:26:15.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:15.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.819417+0000", 2024-08-17T18:26:15.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7060000000000004e-06 2024-08-17T18:26:15.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.819473+0000", 2024-08-17T18:26:15.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6588000000000001e-05 2024-08-17T18:26:15.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:15.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.819483+0000", 2024-08-17T18:26:15.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9490000000000008e-06 2024-08-17T18:26:15.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:59.723935+0000", 2024-08-17T18:26:15.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.904451385 2024-08-17T18:26:15.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:15.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:59.723942+0000", 2024-08-17T18:26:15.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9190000000000002e-06 2024-08-17T18:26:15.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.910527+0000", 2024-08-17T18:26:15.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1865853980000001 2024-08-17T18:26:15.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:15.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.910677+0000", 2024-08-17T18:26:15.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015007200000000001 2024-08-17T18:26:15.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6,7", 2024-08-17T18:26:15.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.910849+0000", 2024-08-17T18:26:15.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000171726 2024-08-17T18:26:15.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:15.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.915826+0000", 2024-08-17T18:26:15.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0049771199999999998 2024-08-17T18:26:15.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.915892+0000", 2024-08-17T18:26:15.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5720000000000001e-05 2024-08-17T18:26:15.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.918385+0000", 2024-08-17T18:26:15.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0024930529999999998 2024-08-17T18:26:15.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.919649+0000", 2024-08-17T18:26:15.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001264296 2024-08-17T18:26:15.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:15.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.919723+0000", 2024-08-17T18:26:15.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4239999999999994e-05 2024-08-17T18:26:15.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:15.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.920748+0000", 2024-08-17T18:26:15.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001024769 2024-08-17T18:26:15.043 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.043 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:15.043 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.043 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:171702 58.a 58:5776512b:::benchmark_data_smithi136_442124_object58086:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1740)", 2024-08-17T18:26:15.043 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:58.825379+0000", 2024-08-17T18:26:15.043 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 136.09462867299999, 2024-08-17T18:26:15.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1137876950000001, 2024-08-17T18:26:15.043 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:15.044 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:15.044 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:15.044 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:15.044 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:15.044 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:15.044 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 171702 2024-08-17T18:26:15.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.044 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:15.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:15.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.825379+0000", 2024-08-17T18:26:15.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:15.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:15.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.825379+0000", 2024-08-17T18:26:15.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:15.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:15.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.825381+0000", 2024-08-17T18:26:15.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.384e-06 2024-08-17T18:26:15.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:15.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.825385+0000", 2024-08-17T18:26:15.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4939999999999997e-06 2024-08-17T18:26:15.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:15.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.825399+0000", 2024-08-17T18:26:15.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.367e-05 2024-08-17T18:26:15.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:15.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.825406+0000", 2024-08-17T18:26:15.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6719999999999998e-06 2024-08-17T18:26:15.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.825485+0000", 2024-08-17T18:26:15.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9188000000000001e-05 2024-08-17T18:26:15.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:15.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.825495+0000", 2024-08-17T18:26:15.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6930000000000002e-06 2024-08-17T18:26:15.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:59.724309+0000", 2024-08-17T18:26:15.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.89881489599999997 2024-08-17T18:26:15.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:15.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:59.724323+0000", 2024-08-17T18:26:15.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3497e-05 2024-08-17T18:26:15.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.920777+0000", 2024-08-17T18:26:15.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1964544020000001 2024-08-17T18:26:15.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:15.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.920863+0000", 2024-08-17T18:26:15.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5939000000000005e-05 2024-08-17T18:26:15.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6,7", 2024-08-17T18:26:15.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.920959+0000", 2024-08-17T18:26:15.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5463000000000004e-05 2024-08-17T18:26:15.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:15.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.930413+0000", 2024-08-17T18:26:15.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0094547290000000003 2024-08-17T18:26:15.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.937165+0000", 2024-08-17T18:26:15.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0067512659999999997 2024-08-17T18:26:15.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.937223+0000", 2024-08-17T18:26:15.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.842e-05 2024-08-17T18:26:15.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.937378+0000", 2024-08-17T18:26:15.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000155136 2024-08-17T18:26:15.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:15.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.937435+0000", 2024-08-17T18:26:15.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6348000000000003e-05 2024-08-17T18:26:15.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:15.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.939167+0000", 2024-08-17T18:26:15.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017324980000000001 2024-08-17T18:26:15.051 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.051 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:15.051 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.051 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:171707 58.a 58:54a1bf78:::benchmark_data_smithi136_442124_object58091:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1740)", 2024-08-17T18:26:15.051 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:58.834570+0000", 2024-08-17T18:26:15.051 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 136.08543793699999, 2024-08-17T18:26:15.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1059423690000001, 2024-08-17T18:26:15.051 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:15.051 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:15.051 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:15.051 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:15.052 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:15.052 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:15.052 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 171707 2024-08-17T18:26:15.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.052 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:15.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:15.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.834570+0000", 2024-08-17T18:26:15.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:15.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:15.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.834570+0000", 2024-08-17T18:26:15.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:15.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:15.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.834572+0000", 2024-08-17T18:26:15.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.81e-06 2024-08-17T18:26:15.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:15.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.834577+0000", 2024-08-17T18:26:15.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8579999999999999e-06 2024-08-17T18:26:15.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:15.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.834587+0000", 2024-08-17T18:26:15.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7690000000000008e-06 2024-08-17T18:26:15.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:15.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.834593+0000", 2024-08-17T18:26:15.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7889999999999997e-06 2024-08-17T18:26:15.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.834650+0000", 2024-08-17T18:26:15.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6592999999999998e-05 2024-08-17T18:26:15.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:15.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.834661+0000", 2024-08-17T18:26:15.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0696000000000001e-05 2024-08-17T18:26:15.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:59.724358+0000", 2024-08-17T18:26:15.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.88969762299999999 2024-08-17T18:26:15.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:15.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:59.724367+0000", 2024-08-17T18:26:15.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6200000000000005e-06 2024-08-17T18:26:15.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.923568+0000", 2024-08-17T18:26:15.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.199201172 2024-08-17T18:26:15.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:15.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.923685+0000", 2024-08-17T18:26:15.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116432 2024-08-17T18:26:15.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6,7", 2024-08-17T18:26:15.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.923862+0000", 2024-08-17T18:26:15.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017770700000000001 2024-08-17T18:26:15.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:15.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.933710+0000", 2024-08-17T18:26:15.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0098480890000000008 2024-08-17T18:26:15.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.937303+0000", 2024-08-17T18:26:15.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003592207 2024-08-17T18:26:15.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.939223+0000", 2024-08-17T18:26:15.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0019200510000000001 2024-08-17T18:26:15.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.939316+0000", 2024-08-17T18:26:15.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3669000000000001e-05 2024-08-17T18:26:15.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:15.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.939372+0000", 2024-08-17T18:26:15.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5556999999999997e-05 2024-08-17T18:26:15.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:15.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.940513+0000", 2024-08-17T18:26:15.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001140727 2024-08-17T18:26:15.059 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.059 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:15.059 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.059 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:171723 58.a 58:596fa3e1:::benchmark_data_smithi136_442124_object58107:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1740)", 2024-08-17T18:26:15.059 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:58.865379+0000", 2024-08-17T18:26:15.059 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 136.05462927100001, 2024-08-17T18:26:15.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.07641484, 2024-08-17T18:26:15.059 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:15.059 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:15.059 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:15.059 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:15.059 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:15.060 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:15.060 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 171723 2024-08-17T18:26:15.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.060 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:15.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:15.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.865379+0000", 2024-08-17T18:26:15.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:15.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:15.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.865379+0000", 2024-08-17T18:26:15.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:15.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:15.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.865380+0000", 2024-08-17T18:26:15.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9999999999999996e-07 2024-08-17T18:26:15.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:15.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.865384+0000", 2024-08-17T18:26:15.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3479999999999997e-06 2024-08-17T18:26:15.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:15.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.865393+0000", 2024-08-17T18:26:15.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5730000000000008e-06 2024-08-17T18:26:15.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:15.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.865399+0000", 2024-08-17T18:26:15.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.268e-06 2024-08-17T18:26:15.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.865456+0000", 2024-08-17T18:26:15.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6740999999999998e-05 2024-08-17T18:26:15.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:15.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.865464+0000", 2024-08-17T18:26:15.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.49e-06 2024-08-17T18:26:15.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:59.724427+0000", 2024-08-17T18:26:15.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.85896264300000003 2024-08-17T18:26:15.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:15.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:59.724435+0000", 2024-08-17T18:26:15.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1130000000000001e-06 2024-08-17T18:26:15.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.927043+0000", 2024-08-17T18:26:15.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.202608517 2024-08-17T18:26:15.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:15.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.927165+0000", 2024-08-17T18:26:15.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012163300000000001 2024-08-17T18:26:15.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6,7", 2024-08-17T18:26:15.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.927300+0000", 2024-08-17T18:26:15.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013537299999999999 2024-08-17T18:26:15.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:15.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.933717+0000", 2024-08-17T18:26:15.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0064162619999999998 2024-08-17T18:26:15.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.939272+0000", 2024-08-17T18:26:15.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0055549950000000001 2024-08-17T18:26:15.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.940604+0000", 2024-08-17T18:26:15.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001332644 2024-08-17T18:26:15.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.940642+0000", 2024-08-17T18:26:15.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7432999999999999e-05 2024-08-17T18:26:15.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:15.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.940702+0000", 2024-08-17T18:26:15.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0467000000000002e-05 2024-08-17T18:26:15.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:15.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.941794+0000", 2024-08-17T18:26:15.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010914399999999999 2024-08-17T18:26:15.070 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.070 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:15.070 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.070 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:171724 58.a 58:5c763075:::benchmark_data_smithi136_442124_object58108:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1740)", 2024-08-17T18:26:15.070 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:58.867589+0000", 2024-08-17T18:26:15.070 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 136.05241864600001, 2024-08-17T18:26:15.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0755104270000002, 2024-08-17T18:26:15.070 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:15.070 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:15.070 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:15.070 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:15.071 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:15.071 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:15.071 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 171724 2024-08-17T18:26:15.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.071 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:15.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:15.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.867589+0000", 2024-08-17T18:26:15.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:15.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:15.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.867589+0000", 2024-08-17T18:26:15.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:15.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:15.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.867590+0000", 2024-08-17T18:26:15.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.37e-07 2024-08-17T18:26:15.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:15.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.867594+0000", 2024-08-17T18:26:15.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4170000000000001e-06 2024-08-17T18:26:15.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:15.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.867600+0000", 2024-08-17T18:26:15.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7440000000000001e-06 2024-08-17T18:26:15.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:15.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.867605+0000", 2024-08-17T18:26:15.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.301e-06 2024-08-17T18:26:15.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.867658+0000", 2024-08-17T18:26:15.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3718999999999998e-05 2024-08-17T18:26:15.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:15.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.867665+0000", 2024-08-17T18:26:15.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0029999999999997e-06 2024-08-17T18:26:15.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:59.724467+0000", 2024-08-17T18:26:15.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.85680161799999999 2024-08-17T18:26:15.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:15.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:59.724475+0000", 2024-08-17T18:26:15.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.106e-06 2024-08-17T18:26:15.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.930443+0000", 2024-08-17T18:26:15.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.205968084 2024-08-17T18:26:15.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:15.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.930568+0000", 2024-08-17T18:26:15.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012520899999999999 2024-08-17T18:26:15.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6,7", 2024-08-17T18:26:15.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.930707+0000", 2024-08-17T18:26:15.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013844899999999999 2024-08-17T18:26:15.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:15.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.937183+0000", 2024-08-17T18:26:15.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0064762109999999999 2024-08-17T18:26:15.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.940565+0000", 2024-08-17T18:26:15.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033821290000000002 2024-08-17T18:26:15.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.941891+0000", 2024-08-17T18:26:15.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001325484 2024-08-17T18:26:15.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.941932+0000", 2024-08-17T18:26:15.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1309e-05 2024-08-17T18:26:15.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:15.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.941988+0000", 2024-08-17T18:26:15.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5977999999999998e-05 2024-08-17T18:26:15.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:15.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.943100+0000", 2024-08-17T18:26:15.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001111929 2024-08-17T18:26:15.078 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.078 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:15.078 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.078 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:171745 58.a 58:5030c623:::benchmark_data_smithi136_442124_object58129:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1740)", 2024-08-17T18:26:15.078 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:58.964405+0000", 2024-08-17T18:26:15.078 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 135.95560327199999, 2024-08-17T18:26:15.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.990252559, 2024-08-17T18:26:15.078 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:15.079 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:15.079 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:15.079 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:15.079 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:15.079 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:15.079 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 171745 2024-08-17T18:26:15.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.079 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:15.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:15.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.964405+0000", 2024-08-17T18:26:15.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:15.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:15.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.964405+0000", 2024-08-17T18:26:15.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:15.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:15.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.964406+0000", 2024-08-17T18:26:15.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.26e-06 2024-08-17T18:26:15.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:15.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.964410+0000", 2024-08-17T18:26:15.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0450000000000002e-06 2024-08-17T18:26:15.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:15.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.964420+0000", 2024-08-17T18:26:15.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7489999999999993e-06 2024-08-17T18:26:15.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:15.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.964425+0000", 2024-08-17T18:26:15.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7060000000000003e-06 2024-08-17T18:26:15.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.964468+0000", 2024-08-17T18:26:15.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3849e-05 2024-08-17T18:26:15.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:15.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:58.964476+0000", 2024-08-17T18:26:15.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3900000000000004e-06 2024-08-17T18:26:15.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:59.724530+0000", 2024-08-17T18:26:15.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.76005410100000004 2024-08-17T18:26:15.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:15.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:59.724539+0000", 2024-08-17T18:26:15.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6109999999999994e-06 2024-08-17T18:26:15.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.933746+0000", 2024-08-17T18:26:15.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.20920695 2024-08-17T18:26:15.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:15.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.934043+0000", 2024-08-17T18:26:15.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029786800000000002 2024-08-17T18:26:15.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6,7", 2024-08-17T18:26:15.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.934178+0000", 2024-08-17T18:26:15.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013448400000000001 2024-08-17T18:26:15.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.941848+0000", 2024-08-17T18:26:15.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0076699230000000004 2024-08-17T18:26:15.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.944183+0000", 2024-08-17T18:26:15.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023350350000000001 2024-08-17T18:26:15.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.949304+0000", 2024-08-17T18:26:15.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0051213350000000003 2024-08-17T18:26:15.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:15.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.952895+0000", 2024-08-17T18:26:15.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035906950000000001 2024-08-17T18:26:15.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:15.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.952946+0000", 2024-08-17T18:26:15.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1059e-05 2024-08-17T18:26:15.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:15.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.954657+0000", 2024-08-17T18:26:15.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001711499 2024-08-17T18:26:15.086 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.086 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:15.086 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.086 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:171750 58.9 58:9610e2b3:::benchmark_data_smithi136_442124_object58134:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1742)", 2024-08-17T18:26:15.086 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:24:00.911515+0000", 2024-08-17T18:26:15.086 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 134.00849299000001, 2024-08-17T18:26:15.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.155160084, 2024-08-17T18:26:15.087 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:15.087 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:15.087 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:15.087 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:15.087 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:15.087 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:15.087 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 171750 2024-08-17T18:26:15.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.087 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:15.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:15.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.911515+0000", 2024-08-17T18:26:15.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:15.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:15.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.911515+0000", 2024-08-17T18:26:15.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:15.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:15.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.911517+0000", 2024-08-17T18:26:15.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4890000000000001e-06 2024-08-17T18:26:15.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:15.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.911522+0000", 2024-08-17T18:26:15.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5219999999999998e-06 2024-08-17T18:26:15.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:15.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.911533+0000", 2024-08-17T18:26:15.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0995999999999999e-05 2024-08-17T18:26:15.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:15.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.911538+0000", 2024-08-17T18:26:15.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2079999999999999e-06 2024-08-17T18:26:15.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.911595+0000", 2024-08-17T18:26:15.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6419999999999999e-05 2024-08-17T18:26:15.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:26:15.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.912129+0000", 2024-08-17T18:26:15.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00053404500000000003 2024-08-17T18:26:15.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.920477+0000", 2024-08-17T18:26:15.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0083486680000000001 2024-08-17T18:26:15.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:15.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.920566+0000", 2024-08-17T18:26:15.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8207000000000004e-05 2024-08-17T18:26:15.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4,6", 2024-08-17T18:26:15.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.920713+0000", 2024-08-17T18:26:15.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014683 2024-08-17T18:26:15.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:15.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.925098+0000", 2024-08-17T18:26:15.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0043855910000000003 2024-08-17T18:26:15.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.929598+0000", 2024-08-17T18:26:15.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0044999059999999997 2024-08-17T18:26:15.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.932511+0000", 2024-08-17T18:26:15.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0029125930000000002 2024-08-17T18:26:15.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:02.064239+0000", 2024-08-17T18:26:15.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.131728678 2024-08-17T18:26:15.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:15.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:02.064294+0000", 2024-08-17T18:26:15.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4737999999999997e-05 2024-08-17T18:26:15.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:15.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:02.066675+0000", 2024-08-17T18:26:15.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023811930000000002 2024-08-17T18:26:15.093 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.093 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:15.093 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.093 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:171782 58.9 58:9855f0a2:::benchmark_data_smithi136_442124_object58166:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1742)", 2024-08-17T18:26:15.093 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:24:00.962368+0000", 2024-08-17T18:26:15.093 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 133.95763974600001, 2024-08-17T18:26:15.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.106674291, 2024-08-17T18:26:15.094 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:15.094 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:15.094 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:15.094 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:15.094 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:15.094 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:15.094 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 171782 2024-08-17T18:26:15.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.094 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:15.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:15.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.962368+0000", 2024-08-17T18:26:15.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:15.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:15.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.962368+0000", 2024-08-17T18:26:15.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:15.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:15.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.962370+0000", 2024-08-17T18:26:15.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.395e-06 2024-08-17T18:26:15.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:15.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.962374+0000", 2024-08-17T18:26:15.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.194e-06 2024-08-17T18:26:15.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:15.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.962389+0000", 2024-08-17T18:26:15.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4989e-05 2024-08-17T18:26:15.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:15.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.962396+0000", 2024-08-17T18:26:15.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3900000000000004e-06 2024-08-17T18:26:15.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.964462+0000", 2024-08-17T18:26:15.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020660829999999998 2024-08-17T18:26:15.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for degraded object", 2024-08-17T18:26:15.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.965892+0000", 2024-08-17T18:26:15.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001429975 2024-08-17T18:26:15.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:15.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.980101+0000", 2024-08-17T18:26:15.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014208159999999999 2024-08-17T18:26:15.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:15.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.980195+0000", 2024-08-17T18:26:15.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4641000000000004e-05 2024-08-17T18:26:15.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,4,6", 2024-08-17T18:26:15.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.980328+0000", 2024-08-17T18:26:15.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000132829 2024-08-17T18:26:15.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.986000+0000", 2024-08-17T18:26:15.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0056715940000000003 2024-08-17T18:26:15.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:15.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.986322+0000", 2024-08-17T18:26:15.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00032188799999999999 2024-08-17T18:26:15.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:00.993391+0000", 2024-08-17T18:26:15.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0070699719999999999 2024-08-17T18:26:15.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:15.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:02.066742+0000", 2024-08-17T18:26:15.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0733502239999999 2024-08-17T18:26:15.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:15.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:02.066783+0000", 2024-08-17T18:26:15.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1251000000000002e-05 2024-08-17T18:26:15.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:15.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:15.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:15.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:24:02.069043+0000", 2024-08-17T18:26:15.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0022597060000000002 2024-08-17T18:26:15.100 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.100 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:15.100 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.101 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:15.101 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:15.101 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:26:15.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:15.102 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:15.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:15.203 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:15.229 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:15.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:15.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:15.331 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:15.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:15.354 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:15.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:15.455 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:26:15.476 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:15.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:15.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:26:15.559 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:26:15.559 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:26:15.559 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:26:15.559 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:26:15.559 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:26:15.560 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:26:15.560 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:26:15.560 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:26:15.560 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:26:15.560 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:26:15.560 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:26:15.560 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:26:15.560 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:26:15.561 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:26:15.561 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:26:15.561 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:26:15.578 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:26:15.604 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:15.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:15.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:26:15.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:26:15.730 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:15.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:15.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:26:15.782 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.get_all_features 2024-08-17T18:26:15.798 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.get_all_features (17 ms) 2024-08-17T18:26:15.799 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.copyup 2024-08-17T18:26:15.832 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:15.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:15.856 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:15.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:15.898 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:26:15.920 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:15 smithi136 ceph-mon[115078]: pgmap v3532: 83 pgs: 1 activating, 82 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:15.920 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:15 smithi136 ceph-mon[115078]: osdmap e1770: 8 total, 8 up, 8 in 2024-08-17T18:26:15.921 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:15 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1536837953' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-479128-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T18:26:15.921 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:15 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2353048564' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:26:15.921 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:15 smithi136 ceph-mon[109861]: pgmap v3532: 83 pgs: 1 activating, 82 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:15.921 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:15 smithi136 ceph-mon[109861]: osdmap e1770: 8 total, 8 up, 8 in 2024-08-17T18:26:15.921 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:15 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1536837953' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-479128-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T18:26:15.921 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:15 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2353048564' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:26:15.957 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:15.972 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.copyup (173 ms) 2024-08-17T18:26:15.972 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.sparse_copyup 2024-08-17T18:26:15.980 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:15.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:15.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:16.022 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.sparse_copyup (50 ms) 2024-08-17T18:26:16.022 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.get_and_set_id 2024-08-17T18:26:16.029 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.get_and_set_id (7 ms) 2024-08-17T18:26:16.029 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.add_remove_child 2024-08-17T18:26:16.043 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.add_remove_child (14 ms) 2024-08-17T18:26:16.043 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.directory_methods 2024-08-17T18:26:16.070 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.directory_methods (27 ms) 2024-08-17T18:26:16.070 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.create 2024-08-17T18:26:16.082 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:26:16.082 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.create (12 ms) 2024-08-17T18:26:16.083 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.get_features 2024-08-17T18:26:16.084 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.get_features (1 ms) 2024-08-17T18:26:16.084 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.get_object_prefix 2024-08-17T18:26:16.086 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.get_object_prefix (3 ms) 2024-08-17T18:26:16.086 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.get_create_timestamp 2024-08-17T18:26:16.088 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.get_create_timestamp (2 ms) 2024-08-17T18:26:16.089 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.get_access_timestamp 2024-08-17T18:26:16.090 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.get_access_timestamp (1 ms) 2024-08-17T18:26:16.090 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.get_modify_timestamp 2024-08-17T18:26:16.091 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.get_modify_timestamp (1 ms) 2024-08-17T18:26:16.091 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.get_data_pool 2024-08-17T18:26:16.095 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.get_data_pool (4 ms) 2024-08-17T18:26:16.095 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.get_size 2024-08-17T18:26:16.099 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.get_size (4 ms) 2024-08-17T18:26:16.099 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.set_size 2024-08-17T18:26:16.103 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.set_size (5 ms) 2024-08-17T18:26:16.104 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.protection_status 2024-08-17T18:26:16.105 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:16.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:16.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:26:16.120 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.protection_status (16 ms) 2024-08-17T18:26:16.120 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.snapshot_limits 2024-08-17T18:26:16.127 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.snapshot_limits (7 ms) 2024-08-17T18:26:16.127 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.parents_v1 2024-08-17T18:26:16.165 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.parents_v1 (39 ms) 2024-08-17T18:26:16.165 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.parents_v2 2024-08-17T18:26:16.193 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.parents_v2 (27 ms) 2024-08-17T18:26:16.193 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.snapshots 2024-08-17T18:26:16.207 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:16.216 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.snapshots (24 ms) 2024-08-17T18:26:16.217 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.snapid_race 2024-08-17T18:26:16.221 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.snapid_race (4 ms) 2024-08-17T18:26:16.221 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.stripingv2 2024-08-17T18:26:16.229 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.stripingv2 (8 ms) 2024-08-17T18:26:16.229 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.object_map_save 2024-08-17T18:26:16.230 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:16.231 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.object_map_save (2 ms) 2024-08-17T18:26:16.231 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.object_map_resize 2024-08-17T18:26:16.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:16.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:16.236 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.object_map_resize (5 ms) 2024-08-17T18:26:16.236 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.object_map_update 2024-08-17T18:26:16.241 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.object_map_update (6 ms) 2024-08-17T18:26:16.242 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.object_map_load_enoent 2024-08-17T18:26:16.242 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.object_map_load_enoent (0 ms) 2024-08-17T18:26:16.242 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.object_map_snap_add 2024-08-17T18:26:16.249 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.object_map_snap_add (8 ms) 2024-08-17T18:26:16.249 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.object_map_snap_remove 2024-08-17T18:26:16.255 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.object_map_snap_remove (5 ms) 2024-08-17T18:26:16.255 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.flags 2024-08-17T18:26:16.260 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.flags (5 ms) 2024-08-17T18:26:16.260 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.metadata 2024-08-17T18:26:16.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:15 smithi165 ceph-mon[105517]: pgmap v3532: 83 pgs: 1 activating, 82 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:16.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:15 smithi165 ceph-mon[105517]: osdmap e1770: 8 total, 8 up, 8 in 2024-08-17T18:26:16.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:15 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1536837953' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-479128-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T18:26:16.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:15 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2353048564' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:26:16.285 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.metadata (25 ms) 2024-08-17T18:26:16.285 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.set_features 2024-08-17T18:26:16.291 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.set_features (6 ms) 2024-08-17T18:26:16.291 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.mirror 2024-08-17T18:26:16.323 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.mirror (32 ms) 2024-08-17T18:26:16.323 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.mirror_image 2024-08-17T18:26:16.332 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:26:16.338 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.mirror_image (15 ms) 2024-08-17T18:26:16.338 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.mirror_image_status 2024-08-17T18:26:16.354 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:16.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:16.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:26:16.455 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:16.478 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:16.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:16.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:16.534 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-17T18:26:16.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:16.605 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:16.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:16.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:16.707 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:26:16.733 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:16.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:16.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:26:16.782 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:26:16.835 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:26:16.859 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:16.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:16.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:26:16.910 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:16 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1536837953' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-479128-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T18:26:16.910 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:16 smithi165 ceph-mon[105517]: osdmap e1771: 8 total, 8 up, 8 in 2024-08-17T18:26:16.961 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:16.982 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:16.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:16.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:17.084 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:26:17.110 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:17.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:26:17.170 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:16 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1536837953' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-479128-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T18:26:17.171 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:16 smithi136 ceph-mon[109861]: osdmap e1771: 8 total, 8 up, 8 in 2024-08-17T18:26:17.171 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:16 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1536837953' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi136-479128-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T18:26:17.171 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:16 smithi136 ceph-mon[115078]: osdmap e1771: 8 total, 8 up, 8 in 2024-08-17T18:26:17.212 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:17.234 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:17.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:17.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:17.336 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:26:17.359 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:17.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:17.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:26:17.461 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:26:17.483 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:17.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:17.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:26:17.585 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:26:17.607 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:17.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:17.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:26:17.709 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:17.731 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:17.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:17.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:17.780 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:26:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:26:17] "GET /metrics HTTP/1.1" 200 34260 "" "Prometheus/2.51.0" 2024-08-17T18:26:17.833 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:17.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:17.856 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:17.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:17.957 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:17.978 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:17.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:17.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:17 smithi165 ceph-mon[105517]: pgmap v3535: 115 pgs: 14 creating+peering, 12 unknown, 1 activating, 88 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:17 smithi165 ceph-mon[105517]: osdmap e1772: 8 total, 8 up, 8 in 2024-08-17T18:26:18.031 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:17 smithi136 ceph-mon[109861]: pgmap v3535: 115 pgs: 14 creating+peering, 12 unknown, 1 activating, 88 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:18.031 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:17 smithi136 ceph-mon[109861]: osdmap e1772: 8 total, 8 up, 8 in 2024-08-17T18:26:18.031 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:17 smithi136 ceph-mon[115078]: pgmap v3535: 115 pgs: 14 creating+peering, 12 unknown, 1 activating, 88 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:18.031 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:17 smithi136 ceph-mon[115078]: osdmap e1772: 8 total, 8 up, 8 in 2024-08-17T18:26:18.081 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:18.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:18.103 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:18.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:18.204 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:18.226 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:18.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:18.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:18.328 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:26:18.352 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:18.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:18.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:26:18.454 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:18.476 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:18.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:18.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:18.578 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:18.599 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:18.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:18.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:18.624 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.mirror_image_status (2286 ms) 2024-08-17T18:26:18.624 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.mirror_image_map 2024-08-17T18:26:18.701 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:26:18.706 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.mirror_image_map (82 ms) 2024-08-17T18:26:18.706 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.mirror_instances 2024-08-17T18:26:18.715 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.mirror_instances (9 ms) 2024-08-17T18:26:18.715 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.mirror_snapshot 2024-08-17T18:26:18.724 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:18.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:18.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:26:18.727 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.mirror_snapshot (13 ms) 2024-08-17T18:26:18.727 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.group_dir_list 2024-08-17T18:26:18.730 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.group_dir_list (2 ms) 2024-08-17T18:26:18.730 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.group_dir_add 2024-08-17T18:26:18.732 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.group_dir_add (2 ms) 2024-08-17T18:26:18.732 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.dir_add_already_existing 2024-08-17T18:26:18.735 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.dir_add_already_existing (3 ms) 2024-08-17T18:26:18.735 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.group_dir_rename 2024-08-17T18:26:18.742 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.group_dir_rename (6 ms) 2024-08-17T18:26:18.743 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.group_dir_remove 2024-08-17T18:26:18.743 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.group_dir_remove (3 ms) 2024-08-17T18:26:18.743 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.group_dir_remove_missing 2024-08-17T18:26:18.747 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.group_dir_remove_missing (3 ms) 2024-08-17T18:26:18.748 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.group_image_add 2024-08-17T18:26:18.749 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.group_image_add (3 ms) 2024-08-17T18:26:18.749 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.group_image_remove 2024-08-17T18:26:18.759 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.group_image_remove (2 ms) 2024-08-17T18:26:18.759 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.group_image_list 2024-08-17T18:26:18.759 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.group_image_list (2 ms) 2024-08-17T18:26:18.759 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.group_image_clean 2024-08-17T18:26:18.760 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.group_image_clean (4 ms) 2024-08-17T18:26:18.760 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.image_group_add 2024-08-17T18:26:18.761 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.image_group_add (2 ms) 2024-08-17T18:26:18.761 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.image_group_remove 2024-08-17T18:26:18.764 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.image_group_remove (3 ms) 2024-08-17T18:26:18.765 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.image_group_get 2024-08-17T18:26:18.766 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.image_group_get (2 ms) 2024-08-17T18:26:18.767 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_name 2024-08-17T18:26:18.768 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.group_snap_set_empty_name (2 ms) 2024-08-17T18:26:18.768 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_id 2024-08-17T18:26:18.770 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.group_snap_set_empty_id (2 ms) 2024-08-17T18:26:18.770 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_id 2024-08-17T18:26:18.772 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_id (2 ms) 2024-08-17T18:26:18.772 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_name 2024-08-17T18:26:18.775 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_name (3 ms) 2024-08-17T18:26:18.775 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.group_snap_set 2024-08-17T18:26:18.777 INFO:tasks.workunit.client.0.smithi136.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-719-gced649f8/rpm/el9/BUILD/ceph-18.2.4-719-gced649f8/src/test/cls_rbd/test_cls_rbd.cc:2681: Failure 2024-08-17T18:26:18.777 INFO:tasks.workunit.client.0.smithi136.stdout:Expected equality of these values: 2024-08-17T18:26:18.777 INFO:tasks.workunit.client.0.smithi136.stdout: 1U 2024-08-17T18:26:18.778 INFO:tasks.workunit.client.0.smithi136.stdout: Which is: 1 2024-08-17T18:26:18.778 INFO:tasks.workunit.client.0.smithi136.stdout: keys.size() 2024-08-17T18:26:18.778 INFO:tasks.workunit.client.0.smithi136.stdout: Which is: 3 2024-08-17T18:26:18.778 INFO:tasks.workunit.client.0.smithi136.stdout:[ FAILED ] TestClsRbd.group_snap_set (2 ms) 2024-08-17T18:26:18.778 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.group_snap_list 2024-08-17T18:26:18.780 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.group_snap_list (3 ms) 2024-08-17T18:26:18.780 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.group_snap_list_max_return 2024-08-17T18:26:18.801 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.group_snap_list_max_return (20 ms) 2024-08-17T18:26:18.801 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.group_snap_list_max_read 2024-08-17T18:26:18.825 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:18.854 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:18.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:18.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:18.950 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:26:18.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:26:18.993 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:18.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:18.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:26:19.027 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.group_snap_list_max_read (227 ms) 2024-08-17T18:26:19.027 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.group_snap_remove 2024-08-17T18:26:19.030 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.group_snap_remove (4 ms) 2024-08-17T18:26:19.031 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.group_snap_get_by_id 2024-08-17T18:26:19.032 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.group_snap_get_by_id (2 ms) 2024-08-17T18:26:19.033 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.trash_methods 2024-08-17T18:26:19.040 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.trash_methods (7 ms) 2024-08-17T18:26:19.040 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.op_features 2024-08-17T18:26:19.047 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.op_features (7 ms) 2024-08-17T18:26:19.047 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.clone_parent 2024-08-17T18:26:19.066 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.clone_parent (19 ms) 2024-08-17T18:26:19.066 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.clone_parent_ns 2024-08-17T18:26:19.074 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.clone_parent_ns (8 ms) 2024-08-17T18:26:19.075 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.clone_child 2024-08-17T18:26:19.085 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.clone_child (11 ms) 2024-08-17T18:26:19.085 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.namespace_methods 2024-08-17T18:26:19.091 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.namespace_methods (7 ms) 2024-08-17T18:26:19.092 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.migration 2024-08-17T18:26:19.095 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:19.100 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.migration (8 ms) 2024-08-17T18:26:19.100 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.migration_v1 2024-08-17T18:26:19.113 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.migration_v1 (13 ms) 2024-08-17T18:26:19.113 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.assert_snapc_seq 2024-08-17T18:26:19.118 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:19.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:19.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:19.142 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:26:19.144 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T18:26:19.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:19.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:19.244 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:19.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:19.327 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:26:19.328 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:26:19.345 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:19.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:19.368 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:19.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:19.397 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T18:26:19.470 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:19.492 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:19.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:19.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:19.594 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:26:19.617 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:19.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:19.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:26:19.719 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:26:19.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:19.742 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:19.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:26:19.843 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:26:19.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:19.865 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:19.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:26:19.966 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:19.973 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:19 smithi165 ceph-mon[105517]: pgmap v3537: 115 pgs: 14 creating+peering, 101 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 422 KiB/s rd, 276 KiB/s wr, 712 op/s 2024-08-17T18:26:19.993 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:19.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:19.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:19.995 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-17T18:26:20.093 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:19 smithi136 ceph-mon[109861]: pgmap v3537: 115 pgs: 14 creating+peering, 101 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 422 KiB/s rd, 276 KiB/s wr, 712 op/s 2024-08-17T18:26:20.094 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:19 smithi136 ceph-mon[115078]: pgmap v3537: 115 pgs: 14 creating+peering, 101 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 422 KiB/s rd, 276 KiB/s wr, 712 op/s 2024-08-17T18:26:20.095 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:26:20.117 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":3537,"stamp":"2024-08-17T18:26:18.644435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596041,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":2567,"num_read_kb":2503,"num_write":1726,"num_write_kb":1668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":1194049,"num_keys_recovered":10,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2875,"ondisk_log_size":2875,"up":345,"acting":345,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8013788,"kb_used_data":88640,"kb_used_omap":474,"kb_used_meta":7924517,"kb_avail":741947428,"statfs":{"total":767960285184,"available":759754166272,"internally_reserved":0,"allocated":90767360,"data_stored":50354367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":485693,"internal_metadata":8114706115},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":100,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":2567,"num_read_kb":2503,"num_write":1662,"num_write_kb":1635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.933979"},"pg_stats":[{"pgid":"59.1f","version":"0'0","reported_seq":11,"reported_epoch":1772,"state":"active+clean","last_fresh":"2024-08-17T18:26:16.782208+0000","last_change":"2024-08-17T18:26:15.794436+0000","last_active":"2024-08-17T18:26:16.782208+0000","last_peered":"2024-08-17T18:26:16.782208+0000","last_clean":"2024-08-17T18:26:16.782208+0000","last_became_active":"2024-08-17T18:26:15.794093+0000","last_became_peered":"2024-08-17T18:26:15.794093+0000","last_unstale":"2024-08-17T18:26:16.782208+0000","last_undegraded":"2024-08-17T18:26:16.782208+0000","last_fullsized":"2024-08-17T18:26:16.782208+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":1771,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:10:47.559591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"59.1e","version":"1771'7","reported_seq":19,"reported_epoch":1772,"state":"active+clean","last_fresh":"2024-08-17T18:26:16.814418+0000","last_change":"2024-08-17T18:26:15.793429+0000","last_active":"2024-08-17T18:26:16.814418+0000","last_peered":"2024-08-17T18:26:16.814418+0000","last_clean":"2024-08-17T18:26:16.814418+0000","last_became_active":"2024-08-17T18:26:15.793108+0000","last_became_peered":"2024-08-17T18:26:15.793108+0000","last_unstale":"2024-08-17T18:26:16.814418+0000","last_undegraded":"2024-08-17T18:26:16.814418+0000","last_fullsized":"2024-08-17T18:26:16.814418+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":1771,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:27:32.314591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"59.1d","version":"0'0","reported_seq":11,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.793911+0000","last_change":"2024-08-17T18:26:15.793556+0000","last_active":"2024-08-17T18:26:15.793911+0000","last_peered":"2024-08-17T18:26:15.793911+0000","last_clean":"2024-08-17T18:26:15.793911+0000","last_became_active":"2024-08-17T18:26:15.792526+0000","last_became_peered":"2024-08-17T18:26:15.792526+0000","last_unstale":"2024-08-17T18:26:15.793911+0000","last_undegraded":"2024-08-17T18:26:15.793911+0000","last_fullsized":"2024-08-17T18:26:15.793911+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":1771,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:15:38.806985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"59.1c","version":"0'0","reported_seq":11,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.795386+0000","last_change":"2024-08-17T18:26:15.794531+0000","last_active":"2024-08-17T18:26:15.795386+0000","last_peered":"2024-08-17T18:26:15.795386+0000","last_clean":"2024-08-17T18:26:15.795386+0000","last_became_active":"2024-08-17T18:26:15.794246+0000","last_became_peered":"2024-08-17T18:26:15.794246+0000","last_unstale":"2024-08-17T18:26:15.795386+0000","last_undegraded":"2024-08-17T18:26:15.795386+0000","last_fullsized":"2024-08-17T18:26:15.795386+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":1771,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:32:46.250194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"59.1b","version":"0'0","reported_seq":11,"reported_epoch":1772,"state":"active+clean","last_fresh":"2024-08-17T18:26:16.814027+0000","last_change":"2024-08-17T18:26:15.789202+0000","last_active":"2024-08-17T18:26:16.814027+0000","last_peered":"2024-08-17T18:26:16.814027+0000","last_clean":"2024-08-17T18:26:16.814027+0000","last_became_active":"2024-08-17T18:26:15.788847+0000","last_became_peered":"2024-08-17T18:26:15.788847+0000","last_unstale":"2024-08-17T18:26:16.814027+0000","last_undegraded":"2024-08-17T18:26:16.814027+0000","last_fullsized":"2024-08-17T18:26:16.814027+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":1771,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:58:30.127825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"59.1a","version":"0'0","reported_seq":11,"reported_epoch":1772,"state":"active+clean","last_fresh":"2024-08-17T18:26:16.813879+0000","last_change":"2024-08-17T18:26:15.791703+0000","last_active":"2024-08-17T18:26:16.813879+0000","last_peered":"2024-08-17T18:26:16.813879+0000","last_clean":"2024-08-17T18:26:16.813879+0000","last_became_active":"2024-08-17T18:26:15.791389+0000","last_became_peered":"2024-08-17T18:26:15.791389+0000","last_unstale":"2024-08-17T18:26:16.813879+0000","last_undegraded":"2024-08-17T18:26:16.813879+0000","last_fullsized":"2024-08-17T18:26:16.813879+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":1771,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:54:48.322976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"59.19","version":"1771'7","reported_seq":20,"reported_epoch":1772,"state":"active+clean","last_fresh":"2024-08-17T18:26:16.813831+0000","last_change":"2024-08-17T18:26:15.795353+0000","last_active":"2024-08-17T18:26:16.813831+0000","last_peered":"2024-08-17T18:26:16.813831+0000","last_clean":"2024-08-17T18:26:16.813831+0000","last_became_active":"2024-08-17T18:26:15.794903+0000","last_became_peered":"2024-08-17T18:26:15.794903+0000","last_unstale":"2024-08-17T18:26:16.813831+0000","last_undegraded":"2024-08-17T18:26:16.813831+0000","last_fullsized":"2024-08-17T18:26:16.813831+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":1771,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:44:47.814057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":10,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"59.18","version":"0'0","reported_seq":1,"reported_epoch":1770,"state":"creating+peering","last_fresh":"2024-08-17T18:26:14.774095+0000","last_change":"2024-08-17T18:26:14.774095+0000","last_active":"2024-08-17T18:26:14.764205+0000","last_peered":"2024-08-17T18:26:14.764205+0000","last_clean":"2024-08-17T18:26:14.764205+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:26:14.774095+0000","last_undegraded":"2024-08-17T18:26:14.774095+0000","last_fullsized":"2024-08-17T18:26:14.774095+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"59.17","version":"1771'2","reported_seq":16,"reported_epoch":1772,"state":"active+clean","last_fresh":"2024-08-17T18:26:16.833636+0000","last_change":"2024-08-17T18:26:15.786657+0000","last_active":"2024-08-17T18:26:16.833636+0000","last_peered":"2024-08-17T18:26:16.833636+0000","last_clean":"2024-08-17T18:26:16.833636+0000","last_became_active":"2024-08-17T18:26:15.786253+0000","last_became_peered":"2024-08-17T18:26:15.786253+0000","last_unstale":"2024-08-17T18:26:16.833636+0000","last_undegraded":"2024-08-17T18:26:16.833636+0000","last_fullsized":"2024-08-17T18:26:16.833636+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":1771,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:21:05.359588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"59.16","version":"0'0","reported_seq":1,"reported_epoch":1770,"state":"creating+peering","last_fresh":"2024-08-17T18:26:14.774372+0000","last_change":"2024-08-17T18:26:14.774372+0000","last_active":"2024-08-17T18:26:14.764205+0000","last_peered":"2024-08-17T18:26:14.764205+0000","last_clean":"2024-08-17T18:26:14.764205+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:26:14.774372+0000","last_undegraded":"2024-08-17T18:26:14.774372+0000","last_fullsized":"2024-08-17T18:26:14.774372+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:37:19.299733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"59.15","version":"0'0","reported_seq":1,"reported_epoch":1770,"state":"creating+peering","last_fresh":"2024-08-17T18:26:14.774313+0000","last_change":"2024-08-17T18:26:14.774313+0000","last_active":"2024-08-17T18:26:14.764205+0000","last_peered":"2024-08-17T18:26:14.764205+0000","last_clean":"2024-08-17T18:26:14.764205+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:26:14.774313+0000","last_undegraded":"2024-08-17T18:26:14.774313+0000","last_fullsized":"2024-08-17T18:26:14.774313+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:19:56.051160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"59.14","version":"0'0","reported_seq":1,"reported_epoch":1770,"state":"creating+peering","last_fresh":"2024-08-17T18:26:14.775289+0000","last_change":"2024-08-17T18:26:14.775289+0000","last_active":"2024-08-17T18:26:14.764205+0000","last_peered":"2024-08-17T18:26:14.764205+0000","last_clean":"2024-08-17T18:26:14.764205+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:26:14.775289+0000","last_undegraded":"2024-08-17T18:26:14.775289+0000","last_fullsized":"2024-08-17T18:26:14.775289+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:13:39.975836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"59.13","version":"0'0","reported_seq":1,"reported_epoch":1770,"state":"creating+peering","last_fresh":"2024-08-17T18:26:14.774354+0000","last_change":"2024-08-17T18:26:14.774354+0000","last_active":"2024-08-17T18:26:14.764205+0000","last_peered":"2024-08-17T18:26:14.764205+0000","last_clean":"2024-08-17T18:26:14.764205+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:26:14.774354+0000","last_undegraded":"2024-08-17T18:26:14.774354+0000","last_fullsized":"2024-08-17T18:26:14.774354+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:48:27.707571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"59.12","version":"0'0","reported_seq":1,"reported_epoch":1770,"state":"creating+peering","last_fresh":"2024-08-17T18:26:14.775508+0000","last_change":"2024-08-17T18:26:14.775508+0000","last_active":"2024-08-17T18:26:14.764205+0000","last_peered":"2024-08-17T18:26:14.764205+0000","last_clean":"2024-08-17T18:26:14.764205+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:26:14.775508+0000","last_undegraded":"2024-08-17T18:26:14.775508+0000","last_fullsized":"2024-08-17T18:26:14.775508+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:49:05.902262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"59.11","version":"0'0","reported_seq":1,"reported_epoch":1770,"state":"creating+peering","last_fresh":"2024-08-17T18:26:14.775710+0000","last_change":"2024-08-17T18:26:14.775710+0000","last_active":"2024-08-17T18:26:14.764205+0000","last_peered":"2024-08-17T18:26:14.764205+0000","last_clean":"2024-08-17T18:26:14.764205+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:26:14.775710+0000","last_undegraded":"2024-08-17T18:26:14.775710+0000","last_fullsized":"2024-08-17T18:26:14.775710+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T06:18:51.592800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"59.10","version":"0'0","reported_seq":1,"reported_epoch":1770,"state":"creating+peering","last_fresh":"2024-08-17T18:26:14.773687+0000","last_change":"2024-08-17T18:26:14.773687+0000","last_active":"2024-08-17T18:26:14.764205+0000","last_peered":"2024-08-17T18:26:14.764205+0000","last_clean":"2024-08-17T18:26:14.764205+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:26:14.773687+0000","last_undegraded":"2024-08-17T18:26:14.773687+0000","last_fullsized":"2024-08-17T18:26:14.773687+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"59.f","version":"0'0","reported_seq":1,"reported_epoch":1770,"state":"creating+peering","last_fresh":"2024-08-17T18:26:14.775015+0000","last_change":"2024-08-17T18:26:14.775015+0000","last_active":"2024-08-17T18:26:14.764205+0000","last_peered":"2024-08-17T18:26:14.764205+0000","last_clean":"2024-08-17T18:26:14.764205+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:26:14.775015+0000","last_undegraded":"2024-08-17T18:26:14.775015+0000","last_fullsized":"2024-08-17T18:26:14.775015+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"59.e","version":"1771'9","reported_seq":48,"reported_epoch":1772,"state":"active+clean","last_fresh":"2024-08-17T18:26:16.782309+0000","last_change":"2024-08-17T18:26:15.793634+0000","last_active":"2024-08-17T18:26:16.782309+0000","last_peered":"2024-08-17T18:26:16.782309+0000","last_clean":"2024-08-17T18:26:16.782309+0000","last_became_active":"2024-08-17T18:26:15.793377+0000","last_became_peered":"2024-08-17T18:26:15.793377+0000","last_unstale":"2024-08-17T18:26:16.782309+0000","last_undegraded":"2024-08-17T18:26:16.782309+0000","last_fullsized":"2024-08-17T18:26:16.782309+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":1771,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:45:33.365488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":52,"num_write":7,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"59.d","version":"1771'30","reported_seq":69,"reported_epoch":1772,"state":"active+clean","last_fresh":"2024-08-17T18:26:16.832546+0000","last_change":"2024-08-17T18:26:15.794538+0000","last_active":"2024-08-17T18:26:16.832546+0000","last_peered":"2024-08-17T18:26:16.832546+0000","last_clean":"2024-08-17T18:26:16.832546+0000","last_became_active":"2024-08-17T18:26:15.794277+0000","last_became_peered":"2024-08-17T18:26:15.794277+0000","last_unstale":"2024-08-17T18:26:16.832546+0000","last_undegraded":"2024-08-17T18:26:16.832546+0000","last_fullsized":"2024-08-17T18:26:16.832546+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":1771,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:27:31.236747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":139,"num_read_kb":104,"num_write":29,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"59.c","version":"0'0","reported_seq":11,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.795290+0000","last_change":"2024-08-17T18:26:15.794510+0000","last_active":"2024-08-17T18:26:15.795290+0000","last_peered":"2024-08-17T18:26:15.795290+0000","last_clean":"2024-08-17T18:26:15.795290+0000","last_became_active":"2024-08-17T18:26:15.794228+0000","last_became_peered":"2024-08-17T18:26:15.794228+0000","last_unstale":"2024-08-17T18:26:15.795290+0000","last_undegraded":"2024-08-17T18:26:15.795290+0000","last_fullsized":"2024-08-17T18:26:15.795290+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":1771,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:46:59.459798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"59.b","version":"0'0","reported_seq":11,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.794232+0000","last_change":"2024-08-17T18:26:15.790624+0000","last_active":"2024-08-17T18:26:15.794232+0000","last_peered":"2024-08-17T18:26:15.794232+0000","last_clean":"2024-08-17T18:26:15.794232+0000","last_became_active":"2024-08-17T18:26:15.790207+0000","last_became_peered":"2024-08-17T18:26:15.790207+0000","last_unstale":"2024-08-17T18:26:15.794232+0000","last_undegraded":"2024-08-17T18:26:15.794232+0000","last_fullsized":"2024-08-17T18:26:15.794232+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":1771,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:26:02.315164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"59.a","version":"1771'14","reported_seq":31,"reported_epoch":1772,"state":"active+clean","last_fresh":"2024-08-17T18:26:16.813972+0000","last_change":"2024-08-17T18:26:15.789130+0000","last_active":"2024-08-17T18:26:16.813972+0000","last_peered":"2024-08-17T18:26:16.813972+0000","last_clean":"2024-08-17T18:26:16.813972+0000","last_became_active":"2024-08-17T18:26:15.788456+0000","last_became_peered":"2024-08-17T18:26:15.788456+0000","last_unstale":"2024-08-17T18:26:16.813972+0000","last_undegraded":"2024-08-17T18:26:16.813972+0000","last_fullsized":"2024-08-17T18:26:16.813972+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":1771,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:27:27.563808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"59.9","version":"0'0","reported_seq":1,"reported_epoch":1770,"state":"creating+peering","last_fresh":"2024-08-17T18:26:14.773103+0000","last_change":"2024-08-17T18:26:14.773103+0000","last_active":"2024-08-17T18:26:14.764205+0000","last_peered":"2024-08-17T18:26:14.764205+0000","last_clean":"2024-08-17T18:26:14.764205+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:26:14.773103+0000","last_undegraded":"2024-08-17T18:26:14.773103+0000","last_fullsized":"2024-08-17T18:26:14.773103+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"59.8","version":"0'0","reported_seq":11,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.798358+0000","last_change":"2024-08-17T18:26:15.798037+0000","last_active":"2024-08-17T18:26:15.798358+0000","last_peered":"2024-08-17T18:26:15.798358+0000","last_clean":"2024-08-17T18:26:15.798358+0000","last_became_active":"2024-08-17T18:26:15.797668+0000","last_became_peered":"2024-08-17T18:26:15.797668+0000","last_unstale":"2024-08-17T18:26:15.798358+0000","last_undegraded":"2024-08-17T18:26:15.798358+0000","last_fullsized":"2024-08-17T18:26:15.798358+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":1771,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:25:02.571609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"59.7","version":"1771'23","reported_seq":58,"reported_epoch":1772,"state":"active+clean","last_fresh":"2024-08-17T18:26:16.833473+0000","last_change":"2024-08-17T18:26:15.793719+0000","last_active":"2024-08-17T18:26:16.833473+0000","last_peered":"2024-08-17T18:26:16.833473+0000","last_clean":"2024-08-17T18:26:16.833473+0000","last_became_active":"2024-08-17T18:26:15.793452+0000","last_became_peered":"2024-08-17T18:26:15.793452+0000","last_unstale":"2024-08-17T18:26:16.833473+0000","last_undegraded":"2024-08-17T18:26:16.833473+0000","last_fullsized":"2024-08-17T18:26:16.833473+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":1771,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:21:29.808394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":39,"num_write":17,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"59.6","version":"1772'1085","reported_seq":1145,"reported_epoch":1772,"state":"active+clean","last_fresh":"2024-08-17T18:26:17.365162+0000","last_change":"2024-08-17T18:26:15.794693+0000","last_active":"2024-08-17T18:26:17.365162+0000","last_peered":"2024-08-17T18:26:17.365162+0000","last_clean":"2024-08-17T18:26:17.365162+0000","last_became_active":"2024-08-17T18:26:15.794383+0000","last_became_peered":"2024-08-17T18:26:15.794383+0000","last_unstale":"2024-08-17T18:26:17.365162+0000","last_undegraded":"2024-08-17T18:26:17.365162+0000","last_fullsized":"2024-08-17T18:26:17.365162+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":1771,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":1085,"log_dups_size":0,"ondisk_log_size":1085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:50:23.447998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2294,"num_read_kb":2261,"num_write":1594,"num_write_kb":1557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"59.5","version":"0'0","reported_seq":1,"reported_epoch":1770,"state":"creating+peering","last_fresh":"2024-08-17T18:26:14.774370+0000","last_change":"2024-08-17T18:26:14.774370+0000","last_active":"2024-08-17T18:26:14.764205+0000","last_peered":"2024-08-17T18:26:14.764205+0000","last_clean":"2024-08-17T18:26:14.764205+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:26:14.774370+0000","last_undegraded":"2024-08-17T18:26:14.774370+0000","last_fullsized":"2024-08-17T18:26:14.774370+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"59.4","version":"0'0","reported_seq":14,"reported_epoch":1772,"state":"active+clean","last_fresh":"2024-08-17T18:26:16.832563+0000","last_change":"2024-08-17T18:26:15.792995+0000","last_active":"2024-08-17T18:26:16.832563+0000","last_peered":"2024-08-17T18:26:16.832563+0000","last_clean":"2024-08-17T18:26:16.832563+0000","last_became_active":"2024-08-17T18:26:15.792717+0000","last_became_peered":"2024-08-17T18:26:15.792717+0000","last_unstale":"2024-08-17T18:26:16.832563+0000","last_undegraded":"2024-08-17T18:26:16.832563+0000","last_fullsized":"2024-08-17T18:26:16.832563+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":1771,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:43:20.926126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"59.3","version":"0'0","reported_seq":1,"reported_epoch":1770,"state":"creating+peering","last_fresh":"2024-08-17T18:26:14.774025+0000","last_change":"2024-08-17T18:26:14.774025+0000","last_active":"2024-08-17T18:26:14.764205+0000","last_peered":"2024-08-17T18:26:14.764205+0000","last_clean":"2024-08-17T18:26:14.764205+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:26:14.774025+0000","last_undegraded":"2024-08-17T18:26:14.774025+0000","last_fullsized":"2024-08-17T18:26:14.774025+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"59.2","version":"0'0","reported_seq":1,"reported_epoch":1770,"state":"creating+peering","last_fresh":"2024-08-17T18:26:14.773846+0000","last_change":"2024-08-17T18:26:14.773846+0000","last_active":"2024-08-17T18:26:14.764205+0000","last_peered":"2024-08-17T18:26:14.764205+0000","last_clean":"2024-08-17T18:26:14.764205+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:26:14.773846+0000","last_undegraded":"2024-08-17T18:26:14.773846+0000","last_fullsized":"2024-08-17T18:26:14.773846+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"59.1","version":"0'0","reported_seq":11,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.795213+0000","last_change":"2024-08-17T18:26:15.793850+0000","last_active":"2024-08-17T18:26:15.795213+0000","last_peered":"2024-08-17T18:26:15.795213+0000","last_clean":"2024-08-17T18:26:15.795213+0000","last_became_active":"2024-08-17T18:26:15.793546+0000","last_became_peered":"2024-08-17T18:26:15.793546+0000","last_unstale":"2024-08-17T18:26:15.795213+0000","last_undegraded":"2024-08-17T18:26:15.795213+0000","last_fullsized":"2024-08-17T18:26:15.795213+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":1771,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:11:59.260987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"59.0","version":"0'0","reported_seq":1,"reported_epoch":1770,"state":"creating+peering","last_fresh":"2024-08-17T18:26:14.775192+0000","last_change":"2024-08-17T18:26:14.775192+0000","last_active":"2024-08-17T18:26:14.764205+0000","last_peered":"2024-08-17T18:26:14.764205+0000","last_clean":"2024-08-17T18:26:14.764205+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T18:26:14.775192+0000","last_undegraded":"2024-08-17T18:26:14.775192+0000","last_fullsized":"2024-08-17T18:26:14.775192+0000","mapping_epoch":1770,"log_start":"0'0","ondisk_log_start":"0'0","created":1770,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T18:26:14.764205+0000","last_clean_scrub_stamp":"2024-08-17T18:26:14.764205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:22:32.844370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"70'4","reported_seq":79,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.785224+0000","last_change":"2024-08-17T18:23:59.758838+0000","last_active":"2024-08-17T18:26:15.785224+0000","last_peered":"2024-08-17T18:26:15.785224+0000","last_clean":"2024-08-17T18:26:15.785224+0000","last_became_active":"2024-08-17T18:23:59.748445+0000","last_became_peered":"2024-08-17T18:23:59.748445+0000","last_unstale":"2024-08-17T18:26:15.785224+0000","last_undegraded":"2024-08-17T18:26:15.785224+0000","last_fullsized":"2024-08-17T18:26:15.785224+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:08:27.581912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1834,"reported_epoch":1769,"state":"active+clean","last_fresh":"2024-08-17T18:26:13.336810+0000","last_change":"2024-08-17T18:23:36.324372+0000","last_active":"2024-08-17T18:26:13.336810+0000","last_peered":"2024-08-17T18:26:13.336810+0000","last_clean":"2024-08-17T18:26:13.336810+0000","last_became_active":"2024-08-17T18:23:36.323893+0000","last_became_peered":"2024-08-17T18:23:36.323893+0000","last_unstale":"2024-08-17T18:26:13.336810+0000","last_undegraded":"2024-08-17T18:26:13.336810+0000","last_fullsized":"2024-08-17T18:26:13.336810+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:10:52.829170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":235,"reported_epoch":1770,"state":"active+clean","last_fresh":"2024-08-17T18:26:14.775786+0000","last_change":"2024-08-17T18:23:36.319358+0000","last_active":"2024-08-17T18:26:14.775786+0000","last_peered":"2024-08-17T18:26:14.775786+0000","last_clean":"2024-08-17T18:26:14.775786+0000","last_became_active":"2024-08-17T18:23:36.318521+0000","last_became_peered":"2024-08-17T18:23:36.318521+0000","last_unstale":"2024-08-17T18:26:14.775786+0000","last_undegraded":"2024-08-17T18:26:14.775786+0000","last_fullsized":"2024-08-17T18:26:14.775786+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:56:48.984477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":228,"reported_epoch":1770,"state":"active+clean","last_fresh":"2024-08-17T18:26:14.777060+0000","last_change":"2024-08-17T18:23:59.741806+0000","last_active":"2024-08-17T18:26:14.777060+0000","last_peered":"2024-08-17T18:26:14.777060+0000","last_clean":"2024-08-17T18:26:14.777060+0000","last_became_active":"2024-08-17T18:23:59.741479+0000","last_became_peered":"2024-08-17T18:23:59.741479+0000","last_unstale":"2024-08-17T18:26:14.777060+0000","last_undegraded":"2024-08-17T18:26:14.777060+0000","last_fullsized":"2024-08-17T18:26:14.777060+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:02:39.072277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":345,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.780382+0000","last_change":"2024-08-17T18:22:36.836343+0000","last_active":"2024-08-17T18:26:15.780382+0000","last_peered":"2024-08-17T18:26:15.780382+0000","last_clean":"2024-08-17T18:26:15.780382+0000","last_became_active":"2024-08-17T18:22:36.835580+0000","last_became_peered":"2024-08-17T18:22:36.835580+0000","last_unstale":"2024-08-17T18:26:15.780382+0000","last_undegraded":"2024-08-17T18:26:15.780382+0000","last_fullsized":"2024-08-17T18:26:15.780382+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:05:51.737218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":228,"reported_epoch":1770,"state":"active+clean","last_fresh":"2024-08-17T18:26:14.774951+0000","last_change":"2024-08-17T18:23:59.742425+0000","last_active":"2024-08-17T18:26:14.774951+0000","last_peered":"2024-08-17T18:26:14.774951+0000","last_clean":"2024-08-17T18:26:14.774951+0000","last_became_active":"2024-08-17T18:23:59.742155+0000","last_became_peered":"2024-08-17T18:23:59.742155+0000","last_unstale":"2024-08-17T18:26:14.774951+0000","last_undegraded":"2024-08-17T18:26:14.774951+0000","last_fullsized":"2024-08-17T18:26:14.774951+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:11:42.833289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":356,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.780624+0000","last_change":"2024-08-17T18:20:32.094881+0000","last_active":"2024-08-17T18:26:15.780624+0000","last_peered":"2024-08-17T18:26:15.780624+0000","last_clean":"2024-08-17T18:26:15.780624+0000","last_became_active":"2024-08-17T18:20:32.077429+0000","last_became_peered":"2024-08-17T18:20:32.077429+0000","last_unstale":"2024-08-17T18:26:15.780624+0000","last_undegraded":"2024-08-17T18:26:15.780624+0000","last_fullsized":"2024-08-17T18:26:15.780624+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:08:26.290851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1839,"reported_epoch":1769,"state":"active+clean","last_fresh":"2024-08-17T18:26:13.337163+0000","last_change":"2024-08-17T18:25:14.026295+0000","last_active":"2024-08-17T18:26:13.337163+0000","last_peered":"2024-08-17T18:26:13.337163+0000","last_clean":"2024-08-17T18:26:13.337163+0000","last_became_active":"2024-08-17T18:25:14.025999+0000","last_became_peered":"2024-08-17T18:25:14.025999+0000","last_unstale":"2024-08-17T18:26:13.337163+0000","last_undegraded":"2024-08-17T18:26:13.337163+0000","last_fullsized":"2024-08-17T18:26:13.337163+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:55:16.975299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":72,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.786408+0000","last_change":"2024-08-17T18:23:59.745317+0000","last_active":"2024-08-17T18:26:15.786408+0000","last_peered":"2024-08-17T18:26:15.786408+0000","last_clean":"2024-08-17T18:26:15.786408+0000","last_became_active":"2024-08-17T18:23:59.743236+0000","last_became_peered":"2024-08-17T18:23:59.743236+0000","last_unstale":"2024-08-17T18:26:15.786408+0000","last_undegraded":"2024-08-17T18:26:15.786408+0000","last_fullsized":"2024-08-17T18:26:15.786408+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:11.253490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":101,"reported_epoch":1772,"state":"active+clean","last_fresh":"2024-08-17T18:26:16.832610+0000","last_change":"2024-08-17T18:22:53.938704+0000","last_active":"2024-08-17T18:26:16.832610+0000","last_peered":"2024-08-17T18:26:16.832610+0000","last_clean":"2024-08-17T18:26:16.832610+0000","last_became_active":"2024-08-17T18:22:53.938406+0000","last_became_peered":"2024-08-17T18:22:53.938406+0000","last_unstale":"2024-08-17T18:26:16.832610+0000","last_undegraded":"2024-08-17T18:26:16.832610+0000","last_fullsized":"2024-08-17T18:26:16.832610+0000","mapping_epoch":1727,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:58:21.566014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1816,"reported_epoch":1769,"state":"active+clean","last_fresh":"2024-08-17T18:26:13.337411+0000","last_change":"2024-08-17T18:20:31.942863+0000","last_active":"2024-08-17T18:26:13.337411+0000","last_peered":"2024-08-17T18:26:13.337411+0000","last_clean":"2024-08-17T18:26:13.337411+0000","last_became_active":"2024-08-17T18:20:31.941281+0000","last_became_peered":"2024-08-17T18:20:31.941281+0000","last_unstale":"2024-08-17T18:26:13.337411+0000","last_undegraded":"2024-08-17T18:26:13.337411+0000","last_fullsized":"2024-08-17T18:26:13.337411+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:48:05.556975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":228,"reported_epoch":1770,"state":"active+clean","last_fresh":"2024-08-17T18:26:14.775510+0000","last_change":"2024-08-17T18:20:31.933142+0000","last_active":"2024-08-17T18:26:14.775510+0000","last_peered":"2024-08-17T18:26:14.775510+0000","last_clean":"2024-08-17T18:26:14.775510+0000","last_became_active":"2024-08-17T18:20:31.932798+0000","last_became_peered":"2024-08-17T18:20:31.932798+0000","last_unstale":"2024-08-17T18:26:14.775510+0000","last_undegraded":"2024-08-17T18:26:14.775510+0000","last_fullsized":"2024-08-17T18:26:14.775510+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:32:51.059352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":164,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.786151+0000","last_change":"2024-08-17T18:20:31.937627+0000","last_active":"2024-08-17T18:26:15.786151+0000","last_peered":"2024-08-17T18:26:15.786151+0000","last_clean":"2024-08-17T18:26:15.786151+0000","last_became_active":"2024-08-17T18:20:31.937361+0000","last_became_peered":"2024-08-17T18:20:31.937361+0000","last_unstale":"2024-08-17T18:26:15.786151+0000","last_undegraded":"2024-08-17T18:26:15.786151+0000","last_fullsized":"2024-08-17T18:26:15.786151+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:01:26.309656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":73,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.785037+0000","last_change":"2024-08-17T18:23:59.743052+0000","last_active":"2024-08-17T18:26:15.785037+0000","last_peered":"2024-08-17T18:26:15.785037+0000","last_clean":"2024-08-17T18:26:15.785037+0000","last_became_active":"2024-08-17T18:23:59.742158+0000","last_became_peered":"2024-08-17T18:23:59.742158+0000","last_unstale":"2024-08-17T18:26:15.785037+0000","last_undegraded":"2024-08-17T18:26:15.785037+0000","last_fullsized":"2024-08-17T18:26:15.785037+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:34:34.249644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":249,"reported_epoch":1770,"state":"active+clean","last_fresh":"2024-08-17T18:26:14.776104+0000","last_change":"2024-08-17T18:25:14.176708+0000","last_active":"2024-08-17T18:26:14.776104+0000","last_peered":"2024-08-17T18:26:14.776104+0000","last_clean":"2024-08-17T18:26:14.776104+0000","last_became_active":"2024-08-17T18:25:14.176239+0000","last_became_peered":"2024-08-17T18:25:14.176239+0000","last_unstale":"2024-08-17T18:26:14.776104+0000","last_undegraded":"2024-08-17T18:26:14.776104+0000","last_fullsized":"2024-08-17T18:26:14.776104+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:08:21.826456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1832,"reported_epoch":1769,"state":"active+clean","last_fresh":"2024-08-17T18:26:12.712142+0000","last_change":"2024-08-17T18:22:36.867005+0000","last_active":"2024-08-17T18:26:12.712142+0000","last_peered":"2024-08-17T18:26:12.712142+0000","last_clean":"2024-08-17T18:26:12.712142+0000","last_became_active":"2024-08-17T18:22:36.866297+0000","last_became_peered":"2024-08-17T18:22:36.866297+0000","last_unstale":"2024-08-17T18:26:12.712142+0000","last_undegraded":"2024-08-17T18:26:12.712142+0000","last_fullsized":"2024-08-17T18:26:12.712142+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:09:18.460006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":251,"reported_epoch":1770,"state":"active+clean","last_fresh":"2024-08-17T18:26:14.776196+0000","last_change":"2024-08-17T18:25:14.035767+0000","last_active":"2024-08-17T18:26:14.776196+0000","last_peered":"2024-08-17T18:26:14.776196+0000","last_clean":"2024-08-17T18:26:14.776196+0000","last_became_active":"2024-08-17T18:25:14.034592+0000","last_became_peered":"2024-08-17T18:25:14.034592+0000","last_unstale":"2024-08-17T18:26:14.776196+0000","last_undegraded":"2024-08-17T18:26:14.776196+0000","last_fullsized":"2024-08-17T18:26:14.776196+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:22:00.711481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":72,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.785945+0000","last_change":"2024-08-17T18:23:59.744879+0000","last_active":"2024-08-17T18:26:15.785945+0000","last_peered":"2024-08-17T18:26:15.785945+0000","last_clean":"2024-08-17T18:26:15.785945+0000","last_became_active":"2024-08-17T18:23:59.743333+0000","last_became_peered":"2024-08-17T18:23:59.743333+0000","last_unstale":"2024-08-17T18:26:15.785945+0000","last_undegraded":"2024-08-17T18:26:15.785945+0000","last_fullsized":"2024-08-17T18:26:15.785945+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:02:01.546706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":91,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.780040+0000","last_change":"2024-08-17T18:22:36.836745+0000","last_active":"2024-08-17T18:26:15.780040+0000","last_peered":"2024-08-17T18:26:15.780040+0000","last_clean":"2024-08-17T18:26:15.780040+0000","last_became_active":"2024-08-17T18:22:36.835161+0000","last_became_peered":"2024-08-17T18:22:36.835161+0000","last_unstale":"2024-08-17T18:26:15.780040+0000","last_undegraded":"2024-08-17T18:26:15.780040+0000","last_fullsized":"2024-08-17T18:26:15.780040+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:29:43.303633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":415,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.780827+0000","last_change":"2024-08-17T18:25:14.027677+0000","last_active":"2024-08-17T18:26:15.780827+0000","last_peered":"2024-08-17T18:26:15.780827+0000","last_clean":"2024-08-17T18:26:15.780827+0000","last_became_active":"2024-08-17T18:25:14.027309+0000","last_became_peered":"2024-08-17T18:25:14.027309+0000","last_unstale":"2024-08-17T18:26:15.780827+0000","last_undegraded":"2024-08-17T18:26:15.780827+0000","last_fullsized":"2024-08-17T18:26:15.780827+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:15:02.617263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":227,"reported_epoch":1770,"state":"active+clean","last_fresh":"2024-08-17T18:26:14.775407+0000","last_change":"2024-08-17T18:18:29.928751+0000","last_active":"2024-08-17T18:26:14.775407+0000","last_peered":"2024-08-17T18:26:14.775407+0000","last_clean":"2024-08-17T18:26:14.775407+0000","last_became_active":"2024-08-17T18:18:29.898102+0000","last_became_peered":"2024-08-17T18:18:29.898102+0000","last_unstale":"2024-08-17T18:26:14.775407+0000","last_undegraded":"2024-08-17T18:26:14.775407+0000","last_fullsized":"2024-08-17T18:26:14.775407+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:21.098266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":48,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.785059+0000","last_change":"2024-08-17T18:25:14.030155+0000","last_active":"2024-08-17T18:26:15.785059+0000","last_peered":"2024-08-17T18:26:15.785059+0000","last_clean":"2024-08-17T18:26:15.785059+0000","last_became_active":"2024-08-17T18:25:14.029876+0000","last_became_peered":"2024-08-17T18:25:14.029876+0000","last_unstale":"2024-08-17T18:26:15.785059+0000","last_undegraded":"2024-08-17T18:26:15.785059+0000","last_fullsized":"2024-08-17T18:26:15.785059+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:04:34.619858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1816,"reported_epoch":1769,"state":"active+clean","last_fresh":"2024-08-17T18:26:13.336765+0000","last_change":"2024-08-17T18:22:36.838096+0000","last_active":"2024-08-17T18:26:13.336765+0000","last_peered":"2024-08-17T18:26:13.336765+0000","last_clean":"2024-08-17T18:26:13.336765+0000","last_became_active":"2024-08-17T18:22:36.835757+0000","last_became_peered":"2024-08-17T18:22:36.835757+0000","last_unstale":"2024-08-17T18:26:13.336765+0000","last_undegraded":"2024-08-17T18:26:13.336765+0000","last_fullsized":"2024-08-17T18:26:13.336765+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:55:31.016446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1807,"reported_epoch":1769,"state":"active+clean","last_fresh":"2024-08-17T18:26:13.337285+0000","last_change":"2024-08-17T18:25:14.026273+0000","last_active":"2024-08-17T18:26:13.337285+0000","last_peered":"2024-08-17T18:26:13.337285+0000","last_clean":"2024-08-17T18:26:13.337285+0000","last_became_active":"2024-08-17T18:25:14.025964+0000","last_became_peered":"2024-08-17T18:25:14.025964+0000","last_unstale":"2024-08-17T18:26:13.337285+0000","last_undegraded":"2024-08-17T18:26:13.337285+0000","last_fullsized":"2024-08-17T18:26:13.337285+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:43.150679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"347'149","reported_seq":1049,"reported_epoch":1770,"state":"active+clean","last_fresh":"2024-08-17T18:26:14.775499+0000","last_change":"2024-08-17T18:20:33.927014+0000","last_active":"2024-08-17T18:26:14.775499+0000","last_peered":"2024-08-17T18:26:14.775499+0000","last_clean":"2024-08-17T18:26:14.775499+0000","last_became_active":"2024-08-17T18:20:33.926619+0000","last_became_peered":"2024-08-17T18:20:33.926619+0000","last_unstale":"2024-08-17T18:26:14.775499+0000","last_undegraded":"2024-08-17T18:26:14.775499+0000","last_fullsized":"2024-08-17T18:26:14.775499+0000","mapping_epoch":1697,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1698,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:57.271286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:57.271286+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:35:26.778777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1794,"reported_epoch":1769,"state":"active+clean","last_fresh":"2024-08-17T18:26:13.336716+0000","last_change":"2024-08-17T18:22:36.838651+0000","last_active":"2024-08-17T18:26:13.336716+0000","last_peered":"2024-08-17T18:26:13.336716+0000","last_clean":"2024-08-17T18:26:13.336716+0000","last_became_active":"2024-08-17T18:22:36.838262+0000","last_became_peered":"2024-08-17T18:22:36.838262+0000","last_unstale":"2024-08-17T18:26:13.336716+0000","last_undegraded":"2024-08-17T18:26:13.336716+0000","last_fullsized":"2024-08-17T18:26:13.336716+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:24:25.119257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1775,"reported_epoch":1769,"state":"active+clean","last_fresh":"2024-08-17T18:26:12.712085+0000","last_change":"2024-08-17T18:22:36.866053+0000","last_active":"2024-08-17T18:26:12.712085+0000","last_peered":"2024-08-17T18:26:12.712085+0000","last_clean":"2024-08-17T18:26:12.712085+0000","last_became_active":"2024-08-17T18:22:36.865187+0000","last_became_peered":"2024-08-17T18:22:36.865187+0000","last_unstale":"2024-08-17T18:26:12.712085+0000","last_undegraded":"2024-08-17T18:26:12.712085+0000","last_fullsized":"2024-08-17T18:26:12.712085+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:45:34.232504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":77,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.784941+0000","last_change":"2024-08-17T18:23:59.750520+0000","last_active":"2024-08-17T18:26:15.784941+0000","last_peered":"2024-08-17T18:26:15.784941+0000","last_clean":"2024-08-17T18:26:15.784941+0000","last_became_active":"2024-08-17T18:23:59.742332+0000","last_became_peered":"2024-08-17T18:23:59.742332+0000","last_unstale":"2024-08-17T18:26:15.784941+0000","last_undegraded":"2024-08-17T18:26:15.784941+0000","last_fullsized":"2024-08-17T18:26:15.784941+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:39:39.325194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"347'149","reported_seq":1083,"reported_epoch":1769,"state":"active+clean","last_fresh":"2024-08-17T18:26:13.337284+0000","last_change":"2024-08-17T18:23:39.360135+0000","last_active":"2024-08-17T18:26:13.337284+0000","last_peered":"2024-08-17T18:26:13.337284+0000","last_clean":"2024-08-17T18:26:13.337284+0000","last_became_active":"2024-08-17T18:23:39.359652+0000","last_became_peered":"2024-08-17T18:23:39.359652+0000","last_unstale":"2024-08-17T18:26:13.337284+0000","last_undegraded":"2024-08-17T18:26:13.337284+0000","last_fullsized":"2024-08-17T18:26:13.337284+0000","mapping_epoch":1736,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1737,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:53.309517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:53.309517+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:20:52.590505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":235,"reported_epoch":1770,"state":"active+clean","last_fresh":"2024-08-17T18:26:14.776276+0000","last_change":"2024-08-17T18:23:59.746854+0000","last_active":"2024-08-17T18:26:14.776276+0000","last_peered":"2024-08-17T18:26:14.776276+0000","last_clean":"2024-08-17T18:26:14.776276+0000","last_became_active":"2024-08-17T18:23:59.746190+0000","last_became_peered":"2024-08-17T18:23:59.746190+0000","last_unstale":"2024-08-17T18:26:14.776276+0000","last_undegraded":"2024-08-17T18:26:14.776276+0000","last_fullsized":"2024-08-17T18:26:14.776276+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:01:53.108037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":48,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.786837+0000","last_change":"2024-08-17T18:25:14.025153+0000","last_active":"2024-08-17T18:26:15.786837+0000","last_peered":"2024-08-17T18:26:15.786837+0000","last_clean":"2024-08-17T18:26:15.786837+0000","last_became_active":"2024-08-17T18:25:14.024700+0000","last_became_peered":"2024-08-17T18:25:14.024700+0000","last_unstale":"2024-08-17T18:26:15.786837+0000","last_undegraded":"2024-08-17T18:26:15.786837+0000","last_fullsized":"2024-08-17T18:26:15.786837+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:50:25.014563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":237,"reported_epoch":1770,"state":"active+clean","last_fresh":"2024-08-17T18:26:14.775799+0000","last_change":"2024-08-17T18:23:59.772682+0000","last_active":"2024-08-17T18:26:14.775799+0000","last_peered":"2024-08-17T18:26:14.775799+0000","last_clean":"2024-08-17T18:26:14.775799+0000","last_became_active":"2024-08-17T18:23:59.748920+0000","last_became_peered":"2024-08-17T18:23:59.748920+0000","last_unstale":"2024-08-17T18:26:14.775799+0000","last_undegraded":"2024-08-17T18:26:14.775799+0000","last_fullsized":"2024-08-17T18:26:14.775799+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:07:31.660610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"347'149","reported_seq":1073,"reported_epoch":1769,"state":"active+clean","last_fresh":"2024-08-17T18:26:12.712045+0000","last_change":"2024-08-17T18:24:02.793338+0000","last_active":"2024-08-17T18:26:12.712045+0000","last_peered":"2024-08-17T18:26:12.712045+0000","last_clean":"2024-08-17T18:26:12.712045+0000","last_became_active":"2024-08-17T18:24:02.792982+0000","last_became_peered":"2024-08-17T18:24:02.792982+0000","last_unstale":"2024-08-17T18:26:12.712045+0000","last_undegraded":"2024-08-17T18:26:12.712045+0000","last_fullsized":"2024-08-17T18:26:12.712045+0000","mapping_epoch":1743,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1744,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:52.474544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:52.474544+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:37:26.909105+0000","scrub_duration":0.001886326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":73,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.786613+0000","last_change":"2024-08-17T18:23:59.741315+0000","last_active":"2024-08-17T18:26:15.786613+0000","last_peered":"2024-08-17T18:26:15.786613+0000","last_clean":"2024-08-17T18:26:15.786613+0000","last_became_active":"2024-08-17T18:23:59.741020+0000","last_became_peered":"2024-08-17T18:23:59.741020+0000","last_unstale":"2024-08-17T18:26:15.786613+0000","last_undegraded":"2024-08-17T18:26:15.786613+0000","last_fullsized":"2024-08-17T18:26:15.786613+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:46:47.093299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":186,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.785596+0000","last_change":"2024-08-17T18:23:59.740824+0000","last_active":"2024-08-17T18:26:15.785596+0000","last_peered":"2024-08-17T18:26:15.785596+0000","last_clean":"2024-08-17T18:26:15.785596+0000","last_became_active":"2024-08-17T18:23:59.740478+0000","last_became_peered":"2024-08-17T18:23:59.740478+0000","last_unstale":"2024-08-17T18:26:15.785596+0000","last_undegraded":"2024-08-17T18:26:15.785596+0000","last_fullsized":"2024-08-17T18:26:15.785596+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:44:30.520216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":164,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.785648+0000","last_change":"2024-08-17T18:20:32.012874+0000","last_active":"2024-08-17T18:26:15.785648+0000","last_peered":"2024-08-17T18:26:15.785648+0000","last_clean":"2024-08-17T18:26:15.785648+0000","last_became_active":"2024-08-17T18:20:32.012553+0000","last_became_peered":"2024-08-17T18:20:32.012553+0000","last_unstale":"2024-08-17T18:26:15.785648+0000","last_undegraded":"2024-08-17T18:26:15.785648+0000","last_fullsized":"2024-08-17T18:26:15.785648+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:14:17.188745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"347'149","reported_seq":1039,"reported_epoch":1769,"state":"active+clean","last_fresh":"2024-08-17T18:26:12.712098+0000","last_change":"2024-08-17T18:22:36.866341+0000","last_active":"2024-08-17T18:26:12.712098+0000","last_peered":"2024-08-17T18:26:12.712098+0000","last_clean":"2024-08-17T18:26:12.712098+0000","last_became_active":"2024-08-17T18:22:36.865561+0000","last_became_peered":"2024-08-17T18:22:36.865561+0000","last_unstale":"2024-08-17T18:26:12.712098+0000","last_undegraded":"2024-08-17T18:26:12.712098+0000","last_fullsized":"2024-08-17T18:26:12.712098+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:49.461698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:49.461698+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:22:21.423162+0000","scrub_duration":0.0013792520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":1772,"state":"active+clean","last_fresh":"2024-08-17T18:26:16.832800+0000","last_change":"2024-08-17T18:26:13.338201+0000","last_active":"2024-08-17T18:26:16.832800+0000","last_peered":"2024-08-17T18:26:16.832800+0000","last_clean":"2024-08-17T18:26:16.832800+0000","last_became_active":"2024-08-17T18:26:13.337933+0000","last_became_peered":"2024-08-17T18:26:13.337933+0000","last_unstale":"2024-08-17T18:26:16.832800+0000","last_undegraded":"2024-08-17T18:26:16.832800+0000","last_fullsized":"2024-08-17T18:26:16.832800+0000","mapping_epoch":1768,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1769,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:16:05.784689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":164,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.786574+0000","last_change":"2024-08-17T18:20:32.014875+0000","last_active":"2024-08-17T18:26:15.786574+0000","last_peered":"2024-08-17T18:26:15.786574+0000","last_clean":"2024-08-17T18:26:15.786574+0000","last_became_active":"2024-08-17T18:20:32.014631+0000","last_became_peered":"2024-08-17T18:20:32.014631+0000","last_unstale":"2024-08-17T18:26:15.786574+0000","last_undegraded":"2024-08-17T18:26:15.786574+0000","last_fullsized":"2024-08-17T18:26:15.786574+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:00:50.978635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":346,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.780293+0000","last_change":"2024-08-17T18:18:29.735362+0000","last_active":"2024-08-17T18:26:15.780293+0000","last_peered":"2024-08-17T18:26:15.780293+0000","last_clean":"2024-08-17T18:26:15.780293+0000","last_became_active":"2024-08-17T18:18:29.734752+0000","last_became_peered":"2024-08-17T18:18:29.734752+0000","last_unstale":"2024-08-17T18:26:15.780293+0000","last_undegraded":"2024-08-17T18:26:15.780293+0000","last_fullsized":"2024-08-17T18:26:15.780293+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:25.709698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"347'149","reported_seq":1086,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.780147+0000","last_change":"2024-08-17T18:24:02.792784+0000","last_active":"2024-08-17T18:26:15.780147+0000","last_peered":"2024-08-17T18:26:15.780147+0000","last_clean":"2024-08-17T18:26:15.780147+0000","last_became_active":"2024-08-17T18:24:02.792286+0000","last_became_peered":"2024-08-17T18:24:02.792286+0000","last_unstale":"2024-08-17T18:26:15.780147+0000","last_undegraded":"2024-08-17T18:26:15.780147+0000","last_fullsized":"2024-08-17T18:26:15.780147+0000","mapping_epoch":1743,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1744,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:00.316884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:00.316884+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:26:57.982520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":48,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.785563+0000","last_change":"2024-08-17T18:25:14.031207+0000","last_active":"2024-08-17T18:26:15.785563+0000","last_peered":"2024-08-17T18:26:15.785563+0000","last_clean":"2024-08-17T18:26:15.785563+0000","last_became_active":"2024-08-17T18:25:14.030916+0000","last_became_peered":"2024-08-17T18:25:14.030916+0000","last_unstale":"2024-08-17T18:26:15.785563+0000","last_undegraded":"2024-08-17T18:26:15.785563+0000","last_fullsized":"2024-08-17T18:26:15.785563+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:37:34.050072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1830,"reported_epoch":1769,"state":"active+clean","last_fresh":"2024-08-17T18:26:13.337385+0000","last_change":"2024-08-17T18:23:59.741594+0000","last_active":"2024-08-17T18:26:13.337385+0000","last_peered":"2024-08-17T18:26:13.337385+0000","last_clean":"2024-08-17T18:26:13.337385+0000","last_became_active":"2024-08-17T18:23:59.741207+0000","last_became_peered":"2024-08-17T18:23:59.741207+0000","last_unstale":"2024-08-17T18:26:13.337385+0000","last_undegraded":"2024-08-17T18:26:13.337385+0000","last_fullsized":"2024-08-17T18:26:13.337385+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:02:52.545413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":415,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.781055+0000","last_change":"2024-08-17T18:22:36.836104+0000","last_active":"2024-08-17T18:26:15.781055+0000","last_peered":"2024-08-17T18:26:15.781055+0000","last_clean":"2024-08-17T18:26:15.781055+0000","last_became_active":"2024-08-17T18:22:36.835273+0000","last_became_peered":"2024-08-17T18:22:36.835273+0000","last_unstale":"2024-08-17T18:26:15.781055+0000","last_undegraded":"2024-08-17T18:26:15.781055+0000","last_fullsized":"2024-08-17T18:26:15.781055+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:26:23.494575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"347'149","reported_seq":865,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.785935+0000","last_change":"2024-08-17T18:24:02.792482+0000","last_active":"2024-08-17T18:26:15.785935+0000","last_peered":"2024-08-17T18:26:15.785935+0000","last_clean":"2024-08-17T18:26:15.785935+0000","last_became_active":"2024-08-17T18:24:02.792032+0000","last_became_peered":"2024-08-17T18:24:02.792032+0000","last_unstale":"2024-08-17T18:26:15.785935+0000","last_undegraded":"2024-08-17T18:26:15.785935+0000","last_fullsized":"2024-08-17T18:26:15.785935+0000","mapping_epoch":1743,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1744,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:02.363531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:02.363531+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:08:22.251652+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":48,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.787673+0000","last_change":"2024-08-17T18:25:14.176621+0000","last_active":"2024-08-17T18:26:15.787673+0000","last_peered":"2024-08-17T18:26:15.787673+0000","last_clean":"2024-08-17T18:26:15.787673+0000","last_became_active":"2024-08-17T18:25:14.176321+0000","last_became_peered":"2024-08-17T18:25:14.176321+0000","last_unstale":"2024-08-17T18:26:15.787673+0000","last_undegraded":"2024-08-17T18:26:15.787673+0000","last_fullsized":"2024-08-17T18:26:15.787673+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:40:55.297225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":175,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.784282+0000","last_change":"2024-08-17T18:22:36.834643+0000","last_active":"2024-08-17T18:26:15.784282+0000","last_peered":"2024-08-17T18:26:15.784282+0000","last_clean":"2024-08-17T18:26:15.784282+0000","last_became_active":"2024-08-17T18:22:36.834279+0000","last_became_peered":"2024-08-17T18:22:36.834279+0000","last_unstale":"2024-08-17T18:26:15.784282+0000","last_undegraded":"2024-08-17T18:26:15.784282+0000","last_fullsized":"2024-08-17T18:26:15.784282+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:29:36.002584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2847,"reported_epoch":1772,"state":"active+clean","last_fresh":"2024-08-17T18:26:16.832528+0000","last_change":"2024-08-17T18:18:29.742742+0000","last_active":"2024-08-17T18:26:16.832528+0000","last_peered":"2024-08-17T18:26:16.832528+0000","last_clean":"2024-08-17T18:26:16.832528+0000","last_became_active":"2024-08-17T18:18:29.742393+0000","last_became_peered":"2024-08-17T18:18:29.742393+0000","last_unstale":"2024-08-17T18:26:16.832528+0000","last_undegraded":"2024-08-17T18:26:16.832528+0000","last_fullsized":"2024-08-17T18:26:16.832528+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:57:32.858608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"347'149","reported_seq":852,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.785025+0000","last_change":"2024-08-17T18:23:39.370457+0000","last_active":"2024-08-17T18:26:15.785025+0000","last_peered":"2024-08-17T18:26:15.785025+0000","last_clean":"2024-08-17T18:26:15.785025+0000","last_became_active":"2024-08-17T18:23:39.370144+0000","last_became_peered":"2024-08-17T18:23:39.370144+0000","last_unstale":"2024-08-17T18:26:15.785025+0000","last_undegraded":"2024-08-17T18:26:15.785025+0000","last_fullsized":"2024-08-17T18:26:15.785025+0000","mapping_epoch":1736,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1737,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:20:35.854154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:20:35.854154+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:39:35.866331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":73,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.784847+0000","last_change":"2024-08-17T18:23:59.742951+0000","last_active":"2024-08-17T18:26:15.784847+0000","last_peered":"2024-08-17T18:26:15.784847+0000","last_clean":"2024-08-17T18:26:15.784847+0000","last_became_active":"2024-08-17T18:23:59.742510+0000","last_became_peered":"2024-08-17T18:23:59.742510+0000","last_unstale":"2024-08-17T18:26:15.784847+0000","last_undegraded":"2024-08-17T18:26:15.784847+0000","last_fullsized":"2024-08-17T18:26:15.784847+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:19:12.943501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":73,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.785859+0000","last_change":"2024-08-17T18:23:59.740786+0000","last_active":"2024-08-17T18:26:15.785859+0000","last_peered":"2024-08-17T18:26:15.785859+0000","last_clean":"2024-08-17T18:26:15.785859+0000","last_became_active":"2024-08-17T18:23:59.740340+0000","last_became_peered":"2024-08-17T18:23:59.740340+0000","last_unstale":"2024-08-17T18:26:15.785859+0000","last_undegraded":"2024-08-17T18:26:15.785859+0000","last_fullsized":"2024-08-17T18:26:15.785859+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:52:27.108557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":48,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.784142+0000","last_change":"2024-08-17T18:25:14.176308+0000","last_active":"2024-08-17T18:26:15.784142+0000","last_peered":"2024-08-17T18:26:15.784142+0000","last_clean":"2024-08-17T18:26:15.784142+0000","last_became_active":"2024-08-17T18:25:14.175851+0000","last_became_peered":"2024-08-17T18:25:14.175851+0000","last_unstale":"2024-08-17T18:26:15.784142+0000","last_undegraded":"2024-08-17T18:26:15.784142+0000","last_fullsized":"2024-08-17T18:26:15.784142+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:10:25.195682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":48,"reported_epoch":1769,"state":"active+clean","last_fresh":"2024-08-17T18:26:12.711904+0000","last_change":"2024-08-17T18:23:36.318955+0000","last_active":"2024-08-17T18:26:12.711904+0000","last_peered":"2024-08-17T18:26:12.711904+0000","last_clean":"2024-08-17T18:26:12.711904+0000","last_became_active":"2024-08-17T18:23:36.318684+0000","last_became_peered":"2024-08-17T18:23:36.318684+0000","last_unstale":"2024-08-17T18:26:12.711904+0000","last_undegraded":"2024-08-17T18:26:12.711904+0000","last_fullsized":"2024-08-17T18:26:12.711904+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:47.191868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":232,"reported_epoch":1770,"state":"active+clean","last_fresh":"2024-08-17T18:26:14.774850+0000","last_change":"2024-08-17T18:22:36.835833+0000","last_active":"2024-08-17T18:26:14.774850+0000","last_peered":"2024-08-17T18:26:14.774850+0000","last_clean":"2024-08-17T18:26:14.774850+0000","last_became_active":"2024-08-17T18:22:36.835514+0000","last_became_peered":"2024-08-17T18:22:36.835514+0000","last_unstale":"2024-08-17T18:26:14.774850+0000","last_undegraded":"2024-08-17T18:26:14.774850+0000","last_fullsized":"2024-08-17T18:26:14.774850+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:43:53.668287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":361,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.780282+0000","last_change":"2024-08-17T18:19:26.374702+0000","last_active":"2024-08-17T18:26:15.780282+0000","last_peered":"2024-08-17T18:26:15.780282+0000","last_clean":"2024-08-17T18:26:15.780282+0000","last_became_active":"2024-08-17T18:15:53.304106+0000","last_became_peered":"2024-08-17T18:15:53.304106+0000","last_unstale":"2024-08-17T18:26:15.780282+0000","last_undegraded":"2024-08-17T18:26:15.780282+0000","last_fullsized":"2024-08-17T18:26:15.780282+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:21.409368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":1019,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.787747+0000","last_change":"2024-08-17T18:25:58.619520+0000","last_active":"2024-08-17T18:26:15.787747+0000","last_peered":"2024-08-17T18:26:15.787747+0000","last_clean":"2024-08-17T18:26:15.787747+0000","last_became_active":"2024-08-17T18:24:02.791616+0000","last_became_peered":"2024-08-17T18:24:02.791616+0000","last_unstale":"2024-08-17T18:26:15.787747+0000","last_undegraded":"2024-08-17T18:26:15.787747+0000","last_fullsized":"2024-08-17T18:26:15.787747+0000","mapping_epoch":1743,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1744,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:25:58.619429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:25:58.619429+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:23:08.373499+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"347'149","reported_seq":1067,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.780929+0000","last_change":"2024-08-17T18:22:36.836656+0000","last_active":"2024-08-17T18:26:15.780929+0000","last_peered":"2024-08-17T18:26:15.780929+0000","last_clean":"2024-08-17T18:26:15.780929+0000","last_became_active":"2024-08-17T18:22:36.835810+0000","last_became_peered":"2024-08-17T18:22:36.835810+0000","last_unstale":"2024-08-17T18:26:15.780929+0000","last_undegraded":"2024-08-17T18:26:15.780929+0000","last_fullsized":"2024-08-17T18:26:15.780929+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:55.255983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:55.255983+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:59:27.835153+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":164,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.787798+0000","last_change":"2024-08-17T18:20:31.939871+0000","last_active":"2024-08-17T18:26:15.787798+0000","last_peered":"2024-08-17T18:26:15.787798+0000","last_clean":"2024-08-17T18:26:15.787798+0000","last_became_active":"2024-08-17T18:20:31.939535+0000","last_became_peered":"2024-08-17T18:20:31.939535+0000","last_unstale":"2024-08-17T18:26:15.787798+0000","last_undegraded":"2024-08-17T18:26:15.787798+0000","last_fullsized":"2024-08-17T18:26:15.787798+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:43:31.630668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":242,"reported_epoch":1770,"state":"active+clean","last_fresh":"2024-08-17T18:26:14.776274+0000","last_change":"2024-08-17T18:25:14.036186+0000","last_active":"2024-08-17T18:26:14.776274+0000","last_peered":"2024-08-17T18:26:14.776274+0000","last_clean":"2024-08-17T18:26:14.776274+0000","last_became_active":"2024-08-17T18:25:14.035138+0000","last_became_peered":"2024-08-17T18:25:14.035138+0000","last_unstale":"2024-08-17T18:26:14.776274+0000","last_undegraded":"2024-08-17T18:26:14.776274+0000","last_fullsized":"2024-08-17T18:26:14.776274+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:51.645934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1845,"reported_epoch":1769,"state":"active+clean","last_fresh":"2024-08-17T18:26:13.337254+0000","last_change":"2024-08-17T18:25:14.026973+0000","last_active":"2024-08-17T18:26:13.337254+0000","last_peered":"2024-08-17T18:26:13.337254+0000","last_clean":"2024-08-17T18:26:13.337254+0000","last_became_active":"2024-08-17T18:25:14.026478+0000","last_became_peered":"2024-08-17T18:25:14.026478+0000","last_unstale":"2024-08-17T18:26:13.337254+0000","last_undegraded":"2024-08-17T18:26:13.337254+0000","last_fullsized":"2024-08-17T18:26:13.337254+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:31.568544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1800,"reported_epoch":1769,"state":"active+clean","last_fresh":"2024-08-17T18:26:13.337169+0000","last_change":"2024-08-17T18:15:53.244851+0000","last_active":"2024-08-17T18:26:13.337169+0000","last_peered":"2024-08-17T18:26:13.337169+0000","last_clean":"2024-08-17T18:26:13.337169+0000","last_became_active":"2024-08-17T18:15:53.240074+0000","last_became_peered":"2024-08-17T18:15:53.240074+0000","last_unstale":"2024-08-17T18:26:13.337169+0000","last_undegraded":"2024-08-17T18:26:13.337169+0000","last_fullsized":"2024-08-17T18:26:13.337169+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:09:00.168413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"347'149","reported_seq":1031,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.779986+0000","last_change":"2024-08-17T18:25:55.776006+0000","last_active":"2024-08-17T18:26:15.779986+0000","last_peered":"2024-08-17T18:26:15.779986+0000","last_clean":"2024-08-17T18:26:15.779986+0000","last_became_active":"2024-08-17T18:25:55.775559+0000","last_became_peered":"2024-08-17T18:25:55.775559+0000","last_unstale":"2024-08-17T18:26:15.779986+0000","last_undegraded":"2024-08-17T18:26:15.779986+0000","last_fullsized":"2024-08-17T18:26:15.779986+0000","mapping_epoch":1763,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1764,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:59.307356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:59.307356+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:52:14.784551+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1803,"reported_epoch":1769,"state":"active+clean","last_fresh":"2024-08-17T18:26:13.337401+0000","last_change":"2024-08-17T18:20:31.942019+0000","last_active":"2024-08-17T18:26:13.337401+0000","last_peered":"2024-08-17T18:26:13.337401+0000","last_clean":"2024-08-17T18:26:13.337401+0000","last_became_active":"2024-08-17T18:20:31.941701+0000","last_became_peered":"2024-08-17T18:20:31.941701+0000","last_unstale":"2024-08-17T18:26:13.337401+0000","last_undegraded":"2024-08-17T18:26:13.337401+0000","last_fullsized":"2024-08-17T18:26:13.337401+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:58.908049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1780,"reported_epoch":1769,"state":"active+clean","last_fresh":"2024-08-17T18:26:12.711737+0000","last_change":"2024-08-17T18:25:14.027871+0000","last_active":"2024-08-17T18:26:12.711737+0000","last_peered":"2024-08-17T18:26:12.711737+0000","last_clean":"2024-08-17T18:26:12.711737+0000","last_became_active":"2024-08-17T18:25:14.027353+0000","last_became_peered":"2024-08-17T18:25:14.027353+0000","last_unstale":"2024-08-17T18:26:12.711737+0000","last_undegraded":"2024-08-17T18:26:12.711737+0000","last_fullsized":"2024-08-17T18:26:12.711737+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:06.568798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"347'149","reported_seq":1028,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.784308+0000","last_change":"2024-08-17T18:18:33.757347+0000","last_active":"2024-08-17T18:26:15.784308+0000","last_peered":"2024-08-17T18:26:15.784308+0000","last_clean":"2024-08-17T18:26:15.784308+0000","last_became_active":"2024-08-17T18:18:33.756960+0000","last_became_peered":"2024-08-17T18:18:33.756960+0000","last_unstale":"2024-08-17T18:26:15.784308+0000","last_undegraded":"2024-08-17T18:26:15.784308+0000","last_fullsized":"2024-08-17T18:26:15.784308+0000","mapping_epoch":1670,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1671,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:52.271487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:52.271487+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:37:36.389244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1797,"reported_epoch":1769,"state":"active+clean","last_fresh":"2024-08-17T18:26:13.336949+0000","last_change":"2024-08-17T18:22:36.837889+0000","last_active":"2024-08-17T18:26:13.336949+0000","last_peered":"2024-08-17T18:26:13.336949+0000","last_clean":"2024-08-17T18:26:13.336949+0000","last_became_active":"2024-08-17T18:22:36.837387+0000","last_became_peered":"2024-08-17T18:22:36.837387+0000","last_unstale":"2024-08-17T18:26:13.336949+0000","last_undegraded":"2024-08-17T18:26:13.336949+0000","last_fullsized":"2024-08-17T18:26:13.336949+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:11:29.265925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":258,"reported_epoch":1770,"state":"active+clean","last_fresh":"2024-08-17T18:26:14.774986+0000","last_change":"2024-08-17T18:23:59.770482+0000","last_active":"2024-08-17T18:26:14.774986+0000","last_peered":"2024-08-17T18:26:14.774986+0000","last_clean":"2024-08-17T18:26:14.774986+0000","last_became_active":"2024-08-17T18:23:59.747059+0000","last_became_peered":"2024-08-17T18:23:59.747059+0000","last_unstale":"2024-08-17T18:26:14.774986+0000","last_undegraded":"2024-08-17T18:26:14.774986+0000","last_fullsized":"2024-08-17T18:26:14.774986+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:04:17.042903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":73,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.786560+0000","last_change":"2024-08-17T18:23:59.745024+0000","last_active":"2024-08-17T18:26:15.786560+0000","last_peered":"2024-08-17T18:26:15.786560+0000","last_clean":"2024-08-17T18:26:15.786560+0000","last_became_active":"2024-08-17T18:23:59.741633+0000","last_became_peered":"2024-08-17T18:23:59.741633+0000","last_unstale":"2024-08-17T18:26:15.786560+0000","last_undegraded":"2024-08-17T18:26:15.786560+0000","last_fullsized":"2024-08-17T18:26:15.786560+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:41.514649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":369,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.780551+0000","last_change":"2024-08-17T18:22:36.836017+0000","last_active":"2024-08-17T18:26:15.780551+0000","last_peered":"2024-08-17T18:26:15.780551+0000","last_clean":"2024-08-17T18:26:15.780551+0000","last_became_active":"2024-08-17T18:22:36.835725+0000","last_became_peered":"2024-08-17T18:22:36.835725+0000","last_unstale":"2024-08-17T18:26:15.780551+0000","last_undegraded":"2024-08-17T18:26:15.780551+0000","last_fullsized":"2024-08-17T18:26:15.780551+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:18:07.395241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":48,"reported_epoch":1769,"state":"active+clean","last_fresh":"2024-08-17T18:26:12.711497+0000","last_change":"2024-08-17T18:23:36.324099+0000","last_active":"2024-08-17T18:26:12.711497+0000","last_peered":"2024-08-17T18:26:12.711497+0000","last_clean":"2024-08-17T18:26:12.711497+0000","last_became_active":"2024-08-17T18:23:36.323739+0000","last_became_peered":"2024-08-17T18:23:36.323739+0000","last_unstale":"2024-08-17T18:26:12.711497+0000","last_undegraded":"2024-08-17T18:26:12.711497+0000","last_fullsized":"2024-08-17T18:26:12.711497+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:54:56.319013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":372,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.780486+0000","last_change":"2024-08-17T18:22:36.836529+0000","last_active":"2024-08-17T18:26:15.780486+0000","last_peered":"2024-08-17T18:26:15.780486+0000","last_clean":"2024-08-17T18:26:15.780486+0000","last_became_active":"2024-08-17T18:22:36.836020+0000","last_became_peered":"2024-08-17T18:22:36.836020+0000","last_unstale":"2024-08-17T18:26:15.780486+0000","last_undegraded":"2024-08-17T18:26:15.780486+0000","last_fullsized":"2024-08-17T18:26:15.780486+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:22.088303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1855,"reported_epoch":1769,"state":"active+clean","last_fresh":"2024-08-17T18:26:13.336968+0000","last_change":"2024-08-17T18:25:14.040053+0000","last_active":"2024-08-17T18:26:13.336968+0000","last_peered":"2024-08-17T18:26:13.336968+0000","last_clean":"2024-08-17T18:26:13.336968+0000","last_became_active":"2024-08-17T18:25:14.039807+0000","last_became_peered":"2024-08-17T18:25:14.039807+0000","last_unstale":"2024-08-17T18:26:13.336968+0000","last_undegraded":"2024-08-17T18:26:13.336968+0000","last_fullsized":"2024-08-17T18:26:13.336968+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:05:53.486096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":165,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.786465+0000","last_change":"2024-08-17T18:20:32.006404+0000","last_active":"2024-08-17T18:26:15.786465+0000","last_peered":"2024-08-17T18:26:15.786465+0000","last_clean":"2024-08-17T18:26:15.786465+0000","last_became_active":"2024-08-17T18:20:31.947053+0000","last_became_peered":"2024-08-17T18:20:31.947053+0000","last_unstale":"2024-08-17T18:26:15.786465+0000","last_undegraded":"2024-08-17T18:26:15.786465+0000","last_fullsized":"2024-08-17T18:26:15.786465+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:21:06.537725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":372,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.780051+0000","last_change":"2024-08-17T18:23:36.318964+0000","last_active":"2024-08-17T18:26:15.780051+0000","last_peered":"2024-08-17T18:26:15.780051+0000","last_clean":"2024-08-17T18:26:15.780051+0000","last_became_active":"2024-08-17T18:23:36.318595+0000","last_became_peered":"2024-08-17T18:23:36.318595+0000","last_unstale":"2024-08-17T18:26:15.780051+0000","last_undegraded":"2024-08-17T18:26:15.780051+0000","last_fullsized":"2024-08-17T18:26:15.780051+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:29:16.221103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":191,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.787073+0000","last_change":"2024-08-17T18:23:59.752171+0000","last_active":"2024-08-17T18:26:15.787073+0000","last_peered":"2024-08-17T18:26:15.787073+0000","last_clean":"2024-08-17T18:26:15.787073+0000","last_became_active":"2024-08-17T18:23:59.745970+0000","last_became_peered":"2024-08-17T18:23:59.745970+0000","last_unstale":"2024-08-17T18:26:15.787073+0000","last_undegraded":"2024-08-17T18:26:15.787073+0000","last_fullsized":"2024-08-17T18:26:15.787073+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:24:52.861533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1927,"reported_epoch":1769,"state":"active+clean","last_fresh":"2024-08-17T18:26:12.711565+0000","last_change":"2024-08-17T18:24:22.107318+0000","last_active":"2024-08-17T18:26:12.711565+0000","last_peered":"2024-08-17T18:26:12.711565+0000","last_clean":"2024-08-17T18:26:12.711565+0000","last_became_active":"2024-08-17T18:24:14.847882+0000","last_became_peered":"2024-08-17T18:24:14.847882+0000","last_unstale":"2024-08-17T18:26:12.711565+0000","last_undegraded":"2024-08-17T18:26:12.711565+0000","last_fullsized":"2024-08-17T18:26:12.711565+0000","mapping_epoch":1746,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:45:33.740721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":235,"reported_epoch":1770,"state":"active+clean","last_fresh":"2024-08-17T18:26:14.775850+0000","last_change":"2024-08-17T18:21:12.645070+0000","last_active":"2024-08-17T18:26:14.775850+0000","last_peered":"2024-08-17T18:26:14.775850+0000","last_clean":"2024-08-17T18:26:14.775850+0000","last_became_active":"2024-08-17T18:21:12.644748+0000","last_became_peered":"2024-08-17T18:21:12.644748+0000","last_unstale":"2024-08-17T18:26:14.775850+0000","last_undegraded":"2024-08-17T18:26:14.775850+0000","last_fullsized":"2024-08-17T18:26:14.775850+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:19:08.683431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":527,"reported_epoch":1769,"state":"active+clean","last_fresh":"2024-08-17T18:26:13.336744+0000","last_change":"2024-08-17T18:22:36.838518+0000","last_active":"2024-08-17T18:26:13.336744+0000","last_peered":"2024-08-17T18:26:13.336744+0000","last_clean":"2024-08-17T18:26:13.336744+0000","last_became_active":"2024-08-17T18:22:36.834610+0000","last_became_peered":"2024-08-17T18:22:36.834610+0000","last_unstale":"2024-08-17T18:26:13.336744+0000","last_undegraded":"2024-08-17T18:26:13.336744+0000","last_fullsized":"2024-08-17T18:26:13.336744+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:42:18.798462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":72,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.786119+0000","last_change":"2024-08-17T18:23:59.740450+0000","last_active":"2024-08-17T18:26:15.786119+0000","last_peered":"2024-08-17T18:26:15.786119+0000","last_clean":"2024-08-17T18:26:15.786119+0000","last_became_active":"2024-08-17T18:23:59.740125+0000","last_became_peered":"2024-08-17T18:23:59.740125+0000","last_unstale":"2024-08-17T18:26:15.786119+0000","last_undegraded":"2024-08-17T18:26:15.786119+0000","last_fullsized":"2024-08-17T18:26:15.786119+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:26:38.858210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":72,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.784903+0000","last_change":"2024-08-17T18:23:59.740550+0000","last_active":"2024-08-17T18:26:15.784903+0000","last_peered":"2024-08-17T18:26:15.784903+0000","last_clean":"2024-08-17T18:26:15.784903+0000","last_became_active":"2024-08-17T18:23:59.740226+0000","last_became_peered":"2024-08-17T18:23:59.740226+0000","last_unstale":"2024-08-17T18:26:15.784903+0000","last_undegraded":"2024-08-17T18:26:15.784903+0000","last_fullsized":"2024-08-17T18:26:15.784903+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:35.745256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":362,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.780395+0000","last_change":"2024-08-17T18:18:29.900128+0000","last_active":"2024-08-17T18:26:15.780395+0000","last_peered":"2024-08-17T18:26:15.780395+0000","last_clean":"2024-08-17T18:26:15.780395+0000","last_became_active":"2024-08-17T18:18:29.899279+0000","last_became_peered":"2024-08-17T18:18:29.899279+0000","last_unstale":"2024-08-17T18:26:15.780395+0000","last_undegraded":"2024-08-17T18:26:15.780395+0000","last_fullsized":"2024-08-17T18:26:15.780395+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:01:29.045498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":155,"reported_epoch":1771,"state":"active+clean","last_fresh":"2024-08-17T18:26:15.786580+0000","last_change":"2024-08-17T18:15:53.271113+0000","last_active":"2024-08-17T18:26:15.786580+0000","last_peered":"2024-08-17T18:26:15.786580+0000","last_clean":"2024-08-17T18:26:15.786580+0000","last_became_active":"2024-08-17T18:15:53.230983+0000","last_became_peered":"2024-08-17T18:15:53.230983+0000","last_unstale":"2024-08-17T18:26:15.786580+0000","last_undegraded":"2024-08-17T18:26:15.786580+0000","last_fullsized":"2024-08-17T18:26:15.786580+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:00:35.315798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":253,"reported_epoch":1770,"state":"active+clean","last_fresh":"2024-08-17T18:26:14.775629+0000","last_change":"2024-08-17T18:25:14.030279+0000","last_active":"2024-08-17T18:26:14.775629+0000","last_peered":"2024-08-17T18:26:14.775629+0000","last_clean":"2024-08-17T18:26:14.775629+0000","last_became_active":"2024-08-17T18:25:14.027960+0000","last_became_peered":"2024-08-17T18:25:14.027960+0000","last_unstale":"2024-08-17T18:26:14.775629+0000","last_undegraded":"2024-08-17T18:26:14.775629+0000","last_fullsized":"2024-08-17T18:26:14.775629+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:20:46.317663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":59,"num_pg":32,"stat_sum":{"num_bytes":100,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":2567,"num_read_kb":2503,"num_write":1662,"num_write_kb":1635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1177,"ondisk_log_size":1177,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":13180,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234923,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":872508,"kb_used_data":11228,"kb_used_omap":1,"kb_used_meta":861246,"kb_avail":92872644,"statfs":{"total":95995035648,"available":95101587456,"internally_reserved":0,"allocated":11497472,"data_stored":6415372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":881916363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1724,"seq":7404523618349,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":198836,"kb_used_data":10856,"kb_used_omap":123,"kb_used_meta":187844,"kb_avail":93546316,"statfs":{"total":95995035648,"available":95791427584,"internally_reserved":0,"allocated":11116544,"data_stored":6070340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":126809,"internal_metadata":192352423},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565802,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1306152,"kb_used_data":11420,"kb_used_omap":72,"kb_used_meta":1294647,"kb_avail":92439000,"statfs":{"total":95995035648,"available":94657536000,"internally_reserved":0,"allocated":11694080,"data_stored":6655965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":1325719022},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":1399,"seq":6008659247404,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":981812,"kb_used_data":11152,"kb_used_omap":131,"kb_used_meta":970492,"kb_avail":92763340,"statfs":{"total":95995035648,"available":94989660160,"internally_reserved":0,"allocated":11419648,"data_stored":6347297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134291,"internal_metadata":993784685},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743187192,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":977192,"kb_used_data":10840,"kb_used_omap":66,"kb_used_meta":966269,"kb_avail":92767960,"statfs":{"total":95995035648,"available":94994391040,"internally_reserved":0,"allocated":11100160,"data_stored":6065526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":989460139},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448831,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":917352,"kb_used_data":10844,"kb_used_omap":75,"kb_used_meta":906420,"kb_avail":92827800,"statfs":{"total":95995035648,"available":95055667200,"internally_reserved":0,"allocated":11104256,"data_stored":6066111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":928174271},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208758196,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1391892,"kb_used_data":11460,"kb_used_omap":1,"kb_used_meta":1380414,"kb_avail":92353260,"statfs":{"total":95995035648,"available":94569738240,"internally_reserved":0,"allocated":11735040,"data_stored":6673619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1413544390},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019832,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1368044,"kb_used_data":10840,"kb_used_omap":1,"kb_used_meta":1357182,"kb_avail":92377108,"statfs":{"total":95995035648,"available":94594158592,"internally_reserved":0,"allocated":11100160,"data_stored":6060137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1389754822},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":59,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":59,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":59,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":59,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":59,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":59,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":59,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":59,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T18:26:20.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:20.123 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:20.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:26:20.199 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:26:20.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:26:20.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:20.247 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:20.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:26:20.348 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T18:26:20.372 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:20.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:20.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T18:26:20.474 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:26:20.497 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:20.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:20.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:26:20.599 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:20.624 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:20.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:20.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:20.629 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:26:20.726 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:26:20.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:20.748 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:20.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:26:20.807 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.assert_snapc_seq (1694 ms) 2024-08-17T18:26:20.807 INFO:tasks.workunit.client.0.smithi136.stdout:[ RUN ] TestClsRbd.sparsify 2024-08-17T18:26:20.848 INFO:tasks.workunit.client.0.smithi136.stdout:[ OK ] TestClsRbd.sparsify (41 ms) 2024-08-17T18:26:20.850 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:26:20.871 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:20.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:20.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:26:20.948 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:20 smithi165 ceph-mon[105517]: from='client.47060 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:26:20.948 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:20 smithi165 ceph-mon[105517]: osdmap e1773: 8 total, 8 up, 8 in 2024-08-17T18:26:20.972 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T18:26:20.994 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:20.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:20.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T18:26:21.096 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T18:26:21.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:21.119 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:21.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T18:26:21.125 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:20 smithi136 ceph-mon[109861]: from='client.47060 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:26:21.125 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:20 smithi136 ceph-mon[109861]: osdmap e1773: 8 total, 8 up, 8 in 2024-08-17T18:26:21.125 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:20 smithi136 ceph-mon[115078]: from='client.47060 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:26:21.125 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:20 smithi136 ceph-mon[115078]: osdmap e1773: 8 total, 8 up, 8 in 2024-08-17T18:26:21.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T18:26:21.242 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:21.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:21.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T18:26:21.343 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T18:26:21.365 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:21.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:21.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T18:26:21.467 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T18:26:21.488 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-17T18:26:21.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:21.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T18:26:21.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:26:21.612 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:21.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:21.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:26:21.714 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:26:21.735 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:21.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:21.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:26:21.812 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] 70 tests from TestClsRbd (5066 ms total) 2024-08-17T18:26:21.812 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T18:26:21.812 INFO:tasks.workunit.client.0.smithi136.stdout:[----------] Global test environment tear-down 2024-08-17T18:26:21.812 INFO:tasks.workunit.client.0.smithi136.stdout:[==========] 70 tests from 1 test suite ran. (7135 ms total) 2024-08-17T18:26:21.812 INFO:tasks.workunit.client.0.smithi136.stdout:[ PASSED ] 69 tests. 2024-08-17T18:26:21.812 INFO:tasks.workunit.client.0.smithi136.stdout:[ FAILED ] 1 test, listed below: 2024-08-17T18:26:21.813 INFO:tasks.workunit.client.0.smithi136.stdout:[ FAILED ] TestClsRbd.group_snap_set 2024-08-17T18:26:21.813 INFO:tasks.workunit.client.0.smithi136.stdout: 2024-08-17T18:26:21.813 INFO:tasks.workunit.client.0.smithi136.stdout: 1 FAILED TEST 2024-08-17T18:26:21.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:21.823 INFO:tasks.workunit:Stopping ['cls/test_cls_rbd.sh'] on client.0... 2024-08-17T18:26:21.823 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-08-17T18:26:21.837 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T18:26:21.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:21 smithi165 ceph-mon[105517]: pgmap v3539: 115 pgs: 115 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 987 op/s 2024-08-17T18:26:21.856 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:21 smithi165 ceph-mon[105517]: osdmap e1774: 8 total, 8 up, 8 in 2024-08-17T18:26:21.856 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:26:21.858 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:21.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:21.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T18:26:21.868 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-08-17T18:26:21.868 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T18:26:21.960 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T18:26:21.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:26:21.983 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-08-17T18:26:21.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T18:26:22.043 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:21 smithi136 ceph-mon[109861]: pgmap v3539: 115 pgs: 115 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 987 op/s 2024-08-17T18:26:22.043 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:21 smithi136 ceph-mon[109861]: osdmap e1774: 8 total, 8 up, 8 in 2024-08-17T18:26:22.043 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:26:22.044 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:21 smithi136 ceph-mon[115078]: pgmap v3539: 115 pgs: 115 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 987 op/s 2024-08-17T18:26:22.044 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:21 smithi136 ceph-mon[115078]: osdmap e1774: 8 total, 8 up, 8 in 2024-08-17T18:26:22.044 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:26:22.045 INFO:tasks.thrashosds:joining thrashosds 2024-08-17T18:26:22.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-17T18:26:22.702 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:26:22.816 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:26:23.084 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:22 smithi136 ceph-mon[109861]: osdmap e1775: 8 total, 8 up, 8 in 2024-08-17T18:26:23.084 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:22 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3464061287' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:26:23.084 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:22 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:26:23.085 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:22 smithi136 ceph-mon[115078]: osdmap e1775: 8 total, 8 up, 8 in 2024-08-17T18:26:23.085 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:22 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3464061287' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:26:23.085 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:22 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:26:23.118 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:22 smithi165 ceph-mon[105517]: osdmap e1775: 8 total, 8 up, 8 in 2024-08-17T18:26:23.118 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:22 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3464061287' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:26:23.118 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:22 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:26:23.123 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T18:26:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:23 smithi165 ceph-mon[105517]: pgmap v3542: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:23 smithi165 ceph-mon[105517]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:26:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:23 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:26:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:23 smithi165 ceph-mon[105517]: osdmap e1776: 8 total, 8 up, 8 in 2024-08-17T18:26:24.175 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:23 smithi136 ceph-mon[109861]: pgmap v3542: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:24.175 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:23 smithi136 ceph-mon[109861]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:26:24.175 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:23 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:26:24.175 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:23 smithi136 ceph-mon[109861]: osdmap e1776: 8 total, 8 up, 8 in 2024-08-17T18:26:24.175 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:23 smithi136 ceph-mon[115078]: pgmap v3542: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:24.175 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:23 smithi136 ceph-mon[115078]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T18:26:24.176 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:23 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:26:24.176 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:23 smithi136 ceph-mon[115078]: osdmap e1776: 8 total, 8 up, 8 in 2024-08-17T18:26:24.275 INFO:teuthology.orchestra.run.smithi136.stdout:osd.0: {} 2024-08-17T18:26:24.276 INFO:teuthology.orchestra.run.smithi136.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T18:26:24.293 INFO:teuthology.orchestra.run.smithi136.stdout:osd.1: {} 2024-08-17T18:26:24.293 INFO:teuthology.orchestra.run.smithi136.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T18:26:24.303 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:26:24.311 INFO:teuthology.orchestra.run.smithi136.stdout:osd.2: {} 2024-08-17T18:26:24.311 INFO:teuthology.orchestra.run.smithi136.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T18:26:24.328 INFO:teuthology.orchestra.run.smithi136.stdout:osd.3: {} 2024-08-17T18:26:24.328 INFO:teuthology.orchestra.run.smithi136.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T18:26:24.345 INFO:teuthology.orchestra.run.smithi136.stdout:osd.4: {} 2024-08-17T18:26:24.346 INFO:teuthology.orchestra.run.smithi136.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T18:26:24.362 INFO:teuthology.orchestra.run.smithi136.stdout:osd.5: {} 2024-08-17T18:26:24.362 INFO:teuthology.orchestra.run.smithi136.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T18:26:24.378 INFO:teuthology.orchestra.run.smithi136.stdout:osd.6: {} 2024-08-17T18:26:24.378 INFO:teuthology.orchestra.run.smithi136.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T18:26:24.394 INFO:teuthology.orchestra.run.smithi136.stdout:osd.7: {} 2024-08-17T18:26:24.394 INFO:teuthology.orchestra.run.smithi136.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T18:26:24.811 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:26:24.811 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T18:26:24.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T18:26:24.811 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:26:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:133120 57.12 57:49924cf9:::benchmark_data_smithi136_389235_object17198:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1648)", 2024-08-17T18:26:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:35.543899+0000", 2024-08-17T18:26:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 589.24675847000003, 2024-08-17T18:26:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1856874199999998, 2024-08-17T18:26:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133120 2024-08-17T18:26:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.812 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:35.543899+0000", 2024-08-17T18:26:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:35.543899+0000", 2024-08-17T18:26:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:24.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:35.543897+0000", 2024-08-17T18:26:24.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:26:24.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:24.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:35.543905+0000", 2024-08-17T18:26:24.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7819999999999994e-06 2024-08-17T18:26:24.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:24.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:35.543919+0000", 2024-08-17T18:26:24.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.432e-05 2024-08-17T18:26:24.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:24.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:35.543926+0000", 2024-08-17T18:26:24.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8279999999999997e-06 2024-08-17T18:26:24.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:35.643933+0000", 2024-08-17T18:26:24.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.100006952 2024-08-17T18:26:24.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:26:24.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:35.646011+0000", 2024-08-17T18:26:24.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002078009 2024-08-17T18:26:24.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:36.551693+0000", 2024-08-17T18:26:24.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90568204399999996 2024-08-17T18:26:24.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:26:24.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:36.557914+0000", 2024-08-17T18:26:24.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062204959999999998 2024-08-17T18:26:24.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.616506+0000", 2024-08-17T18:26:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.058592816 2024-08-17T18:26:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.619828+0000", 2024-08-17T18:26:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033217440000000002 2024-08-17T18:26:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:26:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.626665+0000", 2024-08-17T18:26:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068367649999999999 2024-08-17T18:26:24.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.632385+0000", 2024-08-17T18:26:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057198839999999997 2024-08-17T18:26:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.633470+0000", 2024-08-17T18:26:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010857499999999999 2024-08-17T18:26:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.716484+0000", 2024-08-17T18:26:24.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.083013179000000006 2024-08-17T18:26:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.718934+0000", 2024-08-17T18:26:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024506319999999999 2024-08-17T18:26:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.729586+0000", 2024-08-17T18:26:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010652053 2024-08-17T18:26:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:133992 57.12 57:48ed4d24:::benchmark_data_smithi136_389235_object18070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1648)", 2024-08-17T18:26:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:38.781600+0000", 2024-08-17T18:26:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 586.00905719000002, 2024-08-17T18:26:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1134712009999999, 2024-08-17T18:26:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 133992 2024-08-17T18:26:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.821 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.781600+0000", 2024-08-17T18:26:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.781600+0000", 2024-08-17T18:26:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:24.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.781599+0000", 2024-08-17T18:26:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:26:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.781604+0000", 2024-08-17T18:26:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8099999999999997e-06 2024-08-17T18:26:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.781615+0000", 2024-08-17T18:26:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0771000000000001e-05 2024-08-17T18:26:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.781620+0000", 2024-08-17T18:26:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9740000000000001e-06 2024-08-17T18:26:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.847439+0000", 2024-08-17T18:26:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.065819201999999993 2024-08-17T18:26:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:26:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:38.851794+0000", 2024-08-17T18:26:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043556259999999996 2024-08-17T18:26:24.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:39.873915+0000", 2024-08-17T18:26:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0221207800000001 2024-08-17T18:26:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:26:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:39.878239+0000", 2024-08-17T18:26:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004323716 2024-08-17T18:26:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.852201+0000", 2024-08-17T18:26:24.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.973961909 2024-08-17T18:26:24.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:24.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.855640+0000", 2024-08-17T18:26:24.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034392709999999998 2024-08-17T18:26:24.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:26:24.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.862697+0000", 2024-08-17T18:26:24.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070570750000000003 2024-08-17T18:26:24.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.882488+0000", 2024-08-17T18:26:24.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019790395999999998 2024-08-17T18:26:24.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.883769+0000", 2024-08-17T18:26:24.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012813869999999999 2024-08-17T18:26:24.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:24.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.884136+0000", 2024-08-17T18:26:24.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036747899999999998 2024-08-17T18:26:24.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:24.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.885963+0000", 2024-08-17T18:26:24.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00182672 2024-08-17T18:26:24.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:24.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.895071+0000", 2024-08-17T18:26:24.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0091082130000000004 2024-08-17T18:26:24.828 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.828 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:24.829 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.829 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:134465 57.12 57:49a43719:::benchmark_data_smithi136_389235_object18543:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1648)", 2024-08-17T18:26:24.829 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:40.473026+0000", 2024-08-17T18:26:24.829 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 584.31763134100004, 2024-08-17T18:26:24.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5757722840000001, 2024-08-17T18:26:24.829 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:24.829 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:24.829 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:24.829 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:24.829 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:24.829 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 134465 2024-08-17T18:26:24.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.830 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:24.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:24.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:40.473026+0000", 2024-08-17T18:26:24.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:24.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:40.473026+0000", 2024-08-17T18:26:24.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:24.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:40.473025+0000", 2024-08-17T18:26:24.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:26:24.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:24.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:40.473032+0000", 2024-08-17T18:26:24.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1280000000000002e-06 2024-08-17T18:26:24.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:24.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:40.473049+0000", 2024-08-17T18:26:24.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7317999999999999e-05 2024-08-17T18:26:24.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:24.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:40.473057+0000", 2024-08-17T18:26:24.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4309999999999994e-06 2024-08-17T18:26:24.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:40.572818+0000", 2024-08-17T18:26:24.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.099760653000000005 2024-08-17T18:26:24.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:26:24.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:40.574727+0000", 2024-08-17T18:26:24.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019091240000000001 2024-08-17T18:26:24.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.377268+0000", 2024-08-17T18:26:24.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.80254037899999997 2024-08-17T18:26:24.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:26:24.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.382386+0000", 2024-08-17T18:26:24.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051183130000000002 2024-08-17T18:26:24.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:42.982237+0000", 2024-08-17T18:26:24.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.599851355 2024-08-17T18:26:24.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:24.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:42.987818+0000", 2024-08-17T18:26:24.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055806980000000003 2024-08-17T18:26:24.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:26:24.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:42.998928+0000", 2024-08-17T18:26:24.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011109734 2024-08-17T18:26:24.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:24.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.024007+0000", 2024-08-17T18:26:24.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.025079534000000001 2024-08-17T18:26:24.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.025175+0000", 2024-08-17T18:26:24.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011681350000000001 2024-08-17T18:26:24.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.026529+0000", 2024-08-17T18:26:24.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013539509999999999 2024-08-17T18:26:24.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:24.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.029124+0000", 2024-08-17T18:26:24.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025947209999999999 2024-08-17T18:26:24.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:24.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.048798+0000", 2024-08-17T18:26:24.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019674259999999999 2024-08-17T18:26:24.836 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.837 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:24.837 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.837 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:134821 57.12 57:49381b19:::benchmark_data_smithi136_389235_object18899:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1648)", 2024-08-17T18:26:24.837 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:41.544450+0000", 2024-08-17T18:26:24.837 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 583.24620771699995, 2024-08-17T18:26:24.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5392011000000001, 2024-08-17T18:26:24.837 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:24.837 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:24.837 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:24.837 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:24.837 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:24.837 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 134821 2024-08-17T18:26:24.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.838 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:24.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:24.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.544450+0000", 2024-08-17T18:26:24.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:24.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.544450+0000", 2024-08-17T18:26:24.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:24.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.544448+0000", 2024-08-17T18:26:24.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:26:24.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:24.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.544454+0000", 2024-08-17T18:26:24.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0630000000000002e-06 2024-08-17T18:26:24.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:24.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.544466+0000", 2024-08-17T18:26:24.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2170999999999999e-05 2024-08-17T18:26:24.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:24.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.544474+0000", 2024-08-17T18:26:24.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3069999999999997e-06 2024-08-17T18:26:24.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.566862+0000", 2024-08-17T18:26:24.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022388354999999999 2024-08-17T18:26:24.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:26:24.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:41.568805+0000", 2024-08-17T18:26:24.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019431330000000001 2024-08-17T18:26:24.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:42.238732+0000", 2024-08-17T18:26:24.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.66992708300000003 2024-08-17T18:26:24.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:26:24.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:42.243750+0000", 2024-08-17T18:26:24.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050172599999999999 2024-08-17T18:26:24.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.029453+0000", 2024-08-17T18:26:24.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.785703324 2024-08-17T18:26:24.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:24.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.032708+0000", 2024-08-17T18:26:24.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032548080000000001 2024-08-17T18:26:24.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:26:24.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.039182+0000", 2024-08-17T18:26:24.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064746630000000003 2024-08-17T18:26:24.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.050280+0000", 2024-08-17T18:26:24.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011097205000000001 2024-08-17T18:26:24.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.051571+0000", 2024-08-17T18:26:24.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001291169 2024-08-17T18:26:24.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:24.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.070117+0000", 2024-08-17T18:26:24.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018546475 2024-08-17T18:26:24.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:24.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.072184+0000", 2024-08-17T18:26:24.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020664170000000001 2024-08-17T18:26:24.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:24.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.083651+0000", 2024-08-17T18:26:24.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011467073 2024-08-17T18:26:24.845 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.845 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:24.845 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.845 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:135268 57.12 57:4ffd0969:::benchmark_data_smithi136_389235_object19346:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1648)", 2024-08-17T18:26:24.845 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:43.180362+0000", 2024-08-17T18:26:24.845 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 581.61029586799998, 2024-08-17T18:26:24.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.63819413, 2024-08-17T18:26:24.845 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:24.845 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:24.846 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:24.846 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:24.846 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:24.846 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 135268 2024-08-17T18:26:24.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.846 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:24.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:24.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.180362+0000", 2024-08-17T18:26:24.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:24.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.180362+0000", 2024-08-17T18:26:24.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:24.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.180360+0000", 2024-08-17T18:26:24.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:26:24.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:24.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.180367+0000", 2024-08-17T18:26:24.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5989999999999998e-06 2024-08-17T18:26:24.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:24.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.180379+0000", 2024-08-17T18:26:24.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1735e-05 2024-08-17T18:26:24.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:24.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.180384+0000", 2024-08-17T18:26:24.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8069999999999996e-06 2024-08-17T18:26:24.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.286732+0000", 2024-08-17T18:26:24.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.106347284 2024-08-17T18:26:24.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:26:24.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.289232+0000", 2024-08-17T18:26:24.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002500178 2024-08-17T18:26:24.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.963018+0000", 2024-08-17T18:26:24.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.67378577200000001 2024-08-17T18:26:24.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:26:24.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:43.967249+0000", 2024-08-17T18:26:24.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042313009999999998 2024-08-17T18:26:24.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.762994+0000", 2024-08-17T18:26:24.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.795744819 2024-08-17T18:26:24.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:24.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.766441+0000", 2024-08-17T18:26:24.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034468820000000001 2024-08-17T18:26:24.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:26:24.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.773636+0000", 2024-08-17T18:26:24.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071958530000000003 2024-08-17T18:26:24.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.800098+0000", 2024-08-17T18:26:24.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.026461710999999999 2024-08-17T18:26:24.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:24.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.800485+0000", 2024-08-17T18:26:24.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038679399999999998 2024-08-17T18:26:24.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.801755+0000", 2024-08-17T18:26:24.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001270427 2024-08-17T18:26:24.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:24.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.805500+0000", 2024-08-17T18:26:24.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037451469999999999 2024-08-17T18:26:24.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:24.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.818556+0000", 2024-08-17T18:26:24.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013055152 2024-08-17T18:26:24.853 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.853 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:24.853 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.853 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:135559 57.12 57:481716dc:::benchmark_data_smithi136_389235_object19637:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1649)", 2024-08-17T18:26:24.853 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:16:44.137515+0000", 2024-08-17T18:26:24.853 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 580.65314285700003, 2024-08-17T18:26:24.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0680285010000001, 2024-08-17T18:26:24.854 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:24.854 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:24.854 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:24.854 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:24.854 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:24.854 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 135559 2024-08-17T18:26:24.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.854 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:24.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:24.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.137515+0000", 2024-08-17T18:26:24.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:24.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.137515+0000", 2024-08-17T18:26:24.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:24.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.137513+0000", 2024-08-17T18:26:24.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:26:24.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:24.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.137521+0000", 2024-08-17T18:26:24.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0010000000000001e-06 2024-08-17T18:26:24.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:24.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.137538+0000", 2024-08-17T18:26:24.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6816e-05 2024-08-17T18:26:24.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:24.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.137546+0000", 2024-08-17T18:26:24.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2199999999999992e-06 2024-08-17T18:26:24.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.259176+0000", 2024-08-17T18:26:24.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.121630083 2024-08-17T18:26:24.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:26:24.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:44.261072+0000", 2024-08-17T18:26:24.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001895863 2024-08-17T18:26:24.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.204887+0000", 2024-08-17T18:26:24.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.94381471800000005 2024-08-17T18:26:24.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:26:24.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:45.209973+0000", 2024-08-17T18:26:24.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005086775 2024-08-17T18:26:24.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:47.165222+0000", 2024-08-17T18:26:24.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9552484919999999 2024-08-17T18:26:24.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:24.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:47.169551+0000", 2024-08-17T18:26:24.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043290500000000001 2024-08-17T18:26:24.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:26:24.859 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:47.180180+0000", 2024-08-17T18:26:24.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010628629000000001 2024-08-17T18:26:24.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:47.188542+0000", 2024-08-17T18:26:24.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0083620650000000001 2024-08-17T18:26:24.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:47.191455+0000", 2024-08-17T18:26:24.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029128700000000001 2024-08-17T18:26:24.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:24.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:47.191846+0000", 2024-08-17T18:26:24.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000391911 2024-08-17T18:26:24.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:24.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:47.194664+0000", 2024-08-17T18:26:24.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028179059999999998 2024-08-17T18:26:24.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:24.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:16:47.205543+0000", 2024-08-17T18:26:24.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010878649000000001 2024-08-17T18:26:24.862 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.862 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:24.862 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.862 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:141699 57.12 57:4fa5408d:::benchmark_data_smithi136_389235_object25777:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1650)", 2024-08-17T18:26:24.862 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:02.480434+0000", 2024-08-17T18:26:24.862 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 562.31022315799999, 2024-08-17T18:26:24.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.198213231, 2024-08-17T18:26:24.862 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:24.863 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:24.863 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:24.863 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:24.863 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:24.864 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 141699 2024-08-17T18:26:24.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.864 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:24.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:24.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:02.480434+0000", 2024-08-17T18:26:24.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:24.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:02.480434+0000", 2024-08-17T18:26:24.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:24.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:02.480433+0000", 2024-08-17T18:26:24.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:26:24.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:24.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:02.480440+0000", 2024-08-17T18:26:24.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2400000000000001e-06 2024-08-17T18:26:24.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:24.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:02.480454+0000", 2024-08-17T18:26:24.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3973e-05 2024-08-17T18:26:24.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.866 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:24.866 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:02.480462+0000", 2024-08-17T18:26:24.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2300000000000008e-06 2024-08-17T18:26:24.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:02.510495+0000", 2024-08-17T18:26:24.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.030032631000000001 2024-08-17T18:26:24.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:26:24.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:02.512703+0000", 2024-08-17T18:26:24.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022079579999999999 2024-08-17T18:26:24.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.293890+0000", 2024-08-17T18:26:24.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.78118731900000005 2024-08-17T18:26:24.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:26:24.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.297795+0000", 2024-08-17T18:26:24.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039044959999999999 2024-08-17T18:26:24.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.625210+0000", 2024-08-17T18:26:24.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3274149319999999 2024-08-17T18:26:24.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:24.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.629619+0000", 2024-08-17T18:26:24.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044094820000000002 2024-08-17T18:26:24.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:26:24.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.636707+0000", 2024-08-17T18:26:24.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070873990000000003 2024-08-17T18:26:24.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.665989+0000", 2024-08-17T18:26:24.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.029282077 2024-08-17T18:26:24.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:24.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.666332+0000", 2024-08-17T18:26:24.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034313799999999999 2024-08-17T18:26:24.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.668196+0000", 2024-08-17T18:26:24.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001863969 2024-08-17T18:26:24.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:24.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.669949+0000", 2024-08-17T18:26:24.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017531300000000001 2024-08-17T18:26:24.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:24.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.678647+0000", 2024-08-17T18:26:24.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0086985150000000004 2024-08-17T18:26:24.871 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.871 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:24.871 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.871 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:142338 57.12 57:4cde03a7:::benchmark_data_smithi136_389235_object26416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1650)", 2024-08-17T18:26:24.871 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:04.803270+0000", 2024-08-17T18:26:24.871 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 559.98738721799998, 2024-08-17T18:26:24.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5418123230000003, 2024-08-17T18:26:24.871 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:24.872 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:24.872 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:24.872 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:24.872 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:24.872 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 142338 2024-08-17T18:26:24.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.872 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:24.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:24.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.803270+0000", 2024-08-17T18:26:24.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:24.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.803270+0000", 2024-08-17T18:26:24.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:24.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.803269+0000", 2024-08-17T18:26:24.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:26:24.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:24.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.803277+0000", 2024-08-17T18:26:24.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9999999999999996e-06 2024-08-17T18:26:24.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:24.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.803292+0000", 2024-08-17T18:26:24.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5608e-05 2024-08-17T18:26:24.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:24.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.803300+0000", 2024-08-17T18:26:24.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5839999999999997e-06 2024-08-17T18:26:24.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.830271+0000", 2024-08-17T18:26:24.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.026971517 2024-08-17T18:26:24.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:26:24.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:04.832589+0000", 2024-08-17T18:26:24.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023174799999999998 2024-08-17T18:26:24.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:05.753306+0000", 2024-08-17T18:26:24.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.92071752200000001 2024-08-17T18:26:24.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:26:24.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:05.757990+0000", 2024-08-17T18:26:24.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004683235 2024-08-17T18:26:24.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.296145+0000", 2024-08-17T18:26:24.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5381556270000001 2024-08-17T18:26:24.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:24.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.299493+0000", 2024-08-17T18:26:24.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033477310000000001 2024-08-17T18:26:24.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:26:24.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.306116+0000", 2024-08-17T18:26:24.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006623253 2024-08-17T18:26:24.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:24.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.325542+0000", 2024-08-17T18:26:24.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019425575 2024-08-17T18:26:24.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.330732+0000", 2024-08-17T18:26:24.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051902720000000001 2024-08-17T18:26:24.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.332141+0000", 2024-08-17T18:26:24.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014088799999999999 2024-08-17T18:26:24.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:24.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.334377+0000", 2024-08-17T18:26:24.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022364239999999999 2024-08-17T18:26:24.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:24.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.345082+0000", 2024-08-17T18:26:24.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010705077 2024-08-17T18:26:24.879 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.879 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:24.880 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.880 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:143066 57.12 57:4e20cb33:::benchmark_data_smithi136_389235_object27144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1650)", 2024-08-17T18:26:24.880 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:07.229551+0000", 2024-08-17T18:26:24.880 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 557.56110671199997, 2024-08-17T18:26:24.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.970721556, 2024-08-17T18:26:24.880 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:24.880 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:24.880 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:24.880 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:24.880 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:24.880 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 143066 2024-08-17T18:26:24.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.881 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:24.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:24.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.229551+0000", 2024-08-17T18:26:24.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:24.881 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.229551+0000", 2024-08-17T18:26:24.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.881 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:24.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.229549+0000", 2024-08-17T18:26:24.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:26:24.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:24.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.229562+0000", 2024-08-17T18:26:24.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3594000000000001e-05 2024-08-17T18:26:24.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:24.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.229576+0000", 2024-08-17T18:26:24.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3481e-05 2024-08-17T18:26:24.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:24.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.229584+0000", 2024-08-17T18:26:24.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5539999999999998e-06 2024-08-17T18:26:24.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.345598+0000", 2024-08-17T18:26:24.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.116014002 2024-08-17T18:26:24.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:26:24.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:07.347905+0000", 2024-08-17T18:26:24.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023068730000000001 2024-08-17T18:26:24.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.884 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.884 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.239359+0000", 2024-08-17T18:26:24.884 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.89145372499999997 2024-08-17T18:26:24.884 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.884 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:26:24.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.243463+0000", 2024-08-17T18:26:24.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041039589999999999 2024-08-17T18:26:24.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.112003+0000", 2024-08-17T18:26:24.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.868540396 2024-08-17T18:26:24.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.885 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:24.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.116413+0000", 2024-08-17T18:26:24.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044100650000000003 2024-08-17T18:26:24.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:26:24.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.125278+0000", 2024-08-17T18:26:24.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0088651159999999993 2024-08-17T18:26:24.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.137166+0000", 2024-08-17T18:26:24.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011887246000000001 2024-08-17T18:26:24.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.138635+0000", 2024-08-17T18:26:24.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001469595 2024-08-17T18:26:24.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:24.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.186048+0000", 2024-08-17T18:26:24.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.047412965000000001 2024-08-17T18:26:24.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:24.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.188412+0000", 2024-08-17T18:26:24.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002364054 2024-08-17T18:26:24.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:24.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.200272+0000", 2024-08-17T18:26:24.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.01186006 2024-08-17T18:26:24.888 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.888 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:24.888 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.888 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:143493 57.12 57:4dd25d03:::benchmark_data_smithi136_389235_object27571:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1650)", 2024-08-17T18:26:24.888 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:08.252789+0000", 2024-08-17T18:26:24.888 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 556.53786852600001, 2024-08-17T18:26:24.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0232178630000002, 2024-08-17T18:26:24.888 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:24.888 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:24.889 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:24.889 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:24.889 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:24.889 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 143493 2024-08-17T18:26:24.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.889 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:24.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:24.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.252789+0000", 2024-08-17T18:26:24.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:24.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.252789+0000", 2024-08-17T18:26:24.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:24.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.252787+0000", 2024-08-17T18:26:24.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:26:24.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:24.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.252795+0000", 2024-08-17T18:26:24.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7840000000000004e-06 2024-08-17T18:26:24.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:24.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.252808+0000", 2024-08-17T18:26:24.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.291e-05 2024-08-17T18:26:24.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:24.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.252814+0000", 2024-08-17T18:26:24.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2600000000000002e-06 2024-08-17T18:26:24.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.299118+0000", 2024-08-17T18:26:24.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.046304317999999997 2024-08-17T18:26:24.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:26:24.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:08.301303+0000", 2024-08-17T18:26:24.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002185144 2024-08-17T18:26:24.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.358653+0000", 2024-08-17T18:26:24.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0573500520000001 2024-08-17T18:26:24.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:26:24.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.362431+0000", 2024-08-17T18:26:24.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037779129999999999 2024-08-17T18:26:24.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:11.243257+0000", 2024-08-17T18:26:24.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8808254760000001 2024-08-17T18:26:24.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:24.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:11.246415+0000", 2024-08-17T18:26:24.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003157726 2024-08-17T18:26:24.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:26:24.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:11.253380+0000", 2024-08-17T18:26:24.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069659270000000002 2024-08-17T18:26:24.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:11.263911+0000", 2024-08-17T18:26:24.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010530891000000001 2024-08-17T18:26:24.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:11.264967+0000", 2024-08-17T18:26:24.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001055769 2024-08-17T18:26:24.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:24.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:11.265311+0000", 2024-08-17T18:26:24.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034373200000000002 2024-08-17T18:26:24.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:24.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:11.267008+0000", 2024-08-17T18:26:24.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001696779 2024-08-17T18:26:24.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:24.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:11.276007+0000", 2024-08-17T18:26:24.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0089990529999999999 2024-08-17T18:26:24.896 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.896 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:24.896 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.896 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:144015 57.12 57:4d3166e5:::benchmark_data_smithi136_389235_object28093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1650)", 2024-08-17T18:26:24.896 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:09.497595+0000", 2024-08-17T18:26:24.896 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 555.29306204399995, 2024-08-17T18:26:24.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0067860620000002, 2024-08-17T18:26:24.897 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:24.897 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:24.897 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:24.897 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:24.897 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:24.897 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 144015 2024-08-17T18:26:24.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.897 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:24.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:24.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.497595+0000", 2024-08-17T18:26:24.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:24.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.497595+0000", 2024-08-17T18:26:24.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:24.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.497594+0000", 2024-08-17T18:26:24.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:26:24.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:24.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.497599+0000", 2024-08-17T18:26:24.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.096e-06 2024-08-17T18:26:24.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:24.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.497610+0000", 2024-08-17T18:26:24.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1107000000000001e-05 2024-08-17T18:26:24.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:24.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.497616+0000", 2024-08-17T18:26:24.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5459999999999999e-06 2024-08-17T18:26:24.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.599005+0000", 2024-08-17T18:26:24.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.101389497 2024-08-17T18:26:24.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:26:24.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:09.600888+0000", 2024-08-17T18:26:24.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001883006 2024-08-17T18:26:24.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.353923+0000", 2024-08-17T18:26:24.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.75303402200000003 2024-08-17T18:26:24.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:26:24.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:10.359101+0000", 2024-08-17T18:26:24.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005178958 2024-08-17T18:26:24.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.408708+0000", 2024-08-17T18:26:24.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0496060800000002 2024-08-17T18:26:24.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:24.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.413557+0000", 2024-08-17T18:26:24.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048495170000000002 2024-08-17T18:26:24.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:26:24.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.423699+0000", 2024-08-17T18:26:24.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010141554000000001 2024-08-17T18:26:24.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:24.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.453884+0000", 2024-08-17T18:26:24.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.030185025000000001 2024-08-17T18:26:24.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.489415+0000", 2024-08-17T18:26:24.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035530897999999998 2024-08-17T18:26:24.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.491456+0000", 2024-08-17T18:26:24.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020412389999999998 2024-08-17T18:26:24.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:24.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.493673+0000", 2024-08-17T18:26:24.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022170179999999999 2024-08-17T18:26:24.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:24.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.504381+0000", 2024-08-17T18:26:24.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010708611999999999 2024-08-17T18:26:24.905 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.905 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:24.905 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.906 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:145041 57.12 57:4cef9719:::benchmark_data_smithi136_389235_object29119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1651)", 2024-08-17T18:26:24.906 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:12.526739+0000", 2024-08-17T18:26:24.906 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 552.26391876900004, 2024-08-17T18:26:24.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3206357639999999, 2024-08-17T18:26:24.906 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:24.906 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:24.906 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:24.906 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:24.906 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:24.906 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 145041 2024-08-17T18:26:24.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.907 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:24.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:24.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.526739+0000", 2024-08-17T18:26:24.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:24.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.526739+0000", 2024-08-17T18:26:24.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:24.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.526737+0000", 2024-08-17T18:26:24.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:26:24.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:24.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.526745+0000", 2024-08-17T18:26:24.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0960000000000001e-06 2024-08-17T18:26:24.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:24.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.526762+0000", 2024-08-17T18:26:24.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6514000000000001e-05 2024-08-17T18:26:24.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:24.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.526770+0000", 2024-08-17T18:26:24.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7849999999999997e-06 2024-08-17T18:26:24.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.575365+0000", 2024-08-17T18:26:24.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.048594641000000001 2024-08-17T18:26:24.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:26:24.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:12.577274+0000", 2024-08-17T18:26:24.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001909371 2024-08-17T18:26:24.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.206156+0000", 2024-08-17T18:26:24.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.62888149900000001 2024-08-17T18:26:24.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:26:24.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.216312+0000", 2024-08-17T18:26:24.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010156263 2024-08-17T18:26:24.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.814900+0000", 2024-08-17T18:26:24.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5985881389999999 2024-08-17T18:26:24.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:24.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.818177+0000", 2024-08-17T18:26:24.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032767489999999998 2024-08-17T18:26:24.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:26:24.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.825676+0000", 2024-08-17T18:26:24.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0074987159999999999 2024-08-17T18:26:24.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.833778+0000", 2024-08-17T18:26:24.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0081025070000000001 2024-08-17T18:26:24.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.834861+0000", 2024-08-17T18:26:24.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010832140000000001 2024-08-17T18:26:24.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:24.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.835223+0000", 2024-08-17T18:26:24.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036148499999999999 2024-08-17T18:26:24.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:24.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.837322+0000", 2024-08-17T18:26:24.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002099212 2024-08-17T18:26:24.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:24.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.847374+0000", 2024-08-17T18:26:24.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010052195999999999 2024-08-17T18:26:24.914 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.914 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:24.914 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.914 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:145558 57.12 57:4e414c7c:::benchmark_data_smithi136_389235_object29636:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1651)", 2024-08-17T18:26:24.914 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:13.886158+0000", 2024-08-17T18:26:24.914 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 550.90449947800005, 2024-08-17T18:26:24.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5611586079999999, 2024-08-17T18:26:24.914 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:24.914 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:24.914 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:24.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:24.915 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:24.915 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 145558 2024-08-17T18:26:24.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.915 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:24.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:24.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.886158+0000", 2024-08-17T18:26:24.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:24.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.886158+0000", 2024-08-17T18:26:24.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:24.916 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.886157+0000", 2024-08-17T18:26:24.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:26:24.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:24.916 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.886162+0000", 2024-08-17T18:26:24.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7880000000000003e-06 2024-08-17T18:26:24.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:24.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.886174+0000", 2024-08-17T18:26:24.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1508999999999999e-05 2024-08-17T18:26:24.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:24.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.886180+0000", 2024-08-17T18:26:24.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8089999999999998e-06 2024-08-17T18:26:24.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.902475+0000", 2024-08-17T18:26:24.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016294869999999999 2024-08-17T18:26:24.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-08-17T18:26:24.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:13.904366+0000", 2024-08-17T18:26:24.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001891083 2024-08-17T18:26:24.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.790935+0000", 2024-08-17T18:26:24.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.88656959800000001 2024-08-17T18:26:24.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-08-17T18:26:24.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:14.795601+0000", 2024-08-17T18:26:24.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046654030000000003 2024-08-17T18:26:24.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:16.393619+0000", 2024-08-17T18:26:24.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.598018199 2024-08-17T18:26:24.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:24.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:16.397485+0000", 2024-08-17T18:26:24.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003866455 2024-08-17T18:26:24.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-08-17T18:26:24.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:16.404509+0000", 2024-08-17T18:26:24.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070232319999999999 2024-08-17T18:26:24.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:24.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:16.431091+0000", 2024-08-17T18:26:24.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.026582179000000001 2024-08-17T18:26:24.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:16.432698+0000", 2024-08-17T18:26:24.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001607107 2024-08-17T18:26:24.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:16.434119+0000", 2024-08-17T18:26:24.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001420775 2024-08-17T18:26:24.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:24.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:16.436158+0000", 2024-08-17T18:26:24.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002039482 2024-08-17T18:26:24.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:24.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:16.447317+0000", 2024-08-17T18:26:24.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011158404 2024-08-17T18:26:24.922 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.922 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:24.922 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.922 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:171647 58.5 58:a257b33a:::benchmark_data_smithi136_442124_object58031:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1739)", 2024-08-17T18:26:24.922 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:23:58.705011+0000", 2024-08-17T18:26:24.922 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 146.08564626, 2024-08-17T18:26:24.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1851860040000002, 2024-08-17T18:26:24.923 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:24.923 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:26:24.923 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:24.923 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:26:24.923 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:24.923 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 171647 2024-08-17T18:26:24.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.923 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:24.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:24.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.705011+0000", 2024-08-17T18:26:24.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:24.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.705011+0000", 2024-08-17T18:26:24.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:24.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.705010+0000", 2024-08-17T18:26:24.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:26:24.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:24.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.705014+0000", 2024-08-17T18:26:24.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0339999999999998e-06 2024-08-17T18:26:24.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:24.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.705023+0000", 2024-08-17T18:26:24.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3240000000000003e-06 2024-08-17T18:26:24.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:24.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.705027+0000", 2024-08-17T18:26:24.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9670000000000003e-06 2024-08-17T18:26:24.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.705080+0000", 2024-08-17T18:26:24.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3090999999999997e-05 2024-08-17T18:26:24.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:24.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.705206+0000", 2024-08-17T18:26:24.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012604099999999999 2024-08-17T18:26:24.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-08-17T18:26:24.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.705333+0000", 2024-08-17T18:26:24.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000127713 2024-08-17T18:26:24.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:24.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.708874+0000", 2024-08-17T18:26:24.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035400649999999998 2024-08-17T18:26:24.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.718518+0000", 2024-08-17T18:26:24.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0096442000000000003 2024-08-17T18:26:24.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.771334+0000", 2024-08-17T18:26:24.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.052815978999999999 2024-08-17T18:26:24.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:26:24.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.771349+0000", 2024-08-17T18:26:24.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5316999999999999e-05 2024-08-17T18:26:24.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.726754+0000", 2024-08-17T18:26:24.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.95540548700000005 2024-08-17T18:26:24.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:26:24.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.726768+0000", 2024-08-17T18:26:24.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3033999999999999e-05 2024-08-17T18:26:24.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.890109+0000", 2024-08-17T18:26:24.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.163341884 2024-08-17T18:26:24.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:24.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.890197+0000", 2024-08-17T18:26:24.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7718000000000003e-05 2024-08-17T18:26:24.930 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.930 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:24.930 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.930 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:171687 58.5 58:a08fd36c:::benchmark_data_smithi136_442124_object58071:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1740)", 2024-08-17T18:26:24.930 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:23:58.800517+0000", 2024-08-17T18:26:24.930 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 145.99014034300001, 2024-08-17T18:26:24.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.104712702, 2024-08-17T18:26:24.930 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:24.930 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:24.930 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:24.930 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:26:24.931 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:24.931 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 171687 2024-08-17T18:26:24.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.931 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:24.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:24.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.800517+0000", 2024-08-17T18:26:24.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:24.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.800517+0000", 2024-08-17T18:26:24.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:24.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.800516+0000", 2024-08-17T18:26:24.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:26:24.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:24.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.800522+0000", 2024-08-17T18:26:24.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9970000000000004e-06 2024-08-17T18:26:24.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:24.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.800535+0000", 2024-08-17T18:26:24.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3084999999999999e-05 2024-08-17T18:26:24.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:24.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.800540+0000", 2024-08-17T18:26:24.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1549999999999998e-06 2024-08-17T18:26:24.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.800598+0000", 2024-08-17T18:26:24.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8307999999999998e-05 2024-08-17T18:26:24.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:26:24.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.800611+0000", 2024-08-17T18:26:24.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.279e-05 2024-08-17T18:26:24.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.726800+0000", 2024-08-17T18:26:24.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.92618868899999995 2024-08-17T18:26:24.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:26:24.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.726808+0000", 2024-08-17T18:26:24.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5529999999999993e-06 2024-08-17T18:26:24.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.890396+0000", 2024-08-17T18:26:24.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.163588085 2024-08-17T18:26:24.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:24.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.890519+0000", 2024-08-17T18:26:24.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012242899999999999 2024-08-17T18:26:24.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1,2", 2024-08-17T18:26:24.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.890664+0000", 2024-08-17T18:26:24.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014479400000000001 2024-08-17T18:26:24.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.898804+0000", 2024-08-17T18:26:24.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0081403580000000003 2024-08-17T18:26:24.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:24.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.898815+0000", 2024-08-17T18:26:24.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0941000000000001e-05 2024-08-17T18:26:24.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.899256+0000", 2024-08-17T18:26:24.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000440839 2024-08-17T18:26:24.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.905034+0000", 2024-08-17T18:26:24.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057779670000000002 2024-08-17T18:26:24.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:24.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.905102+0000", 2024-08-17T18:26:24.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7951e-05 2024-08-17T18:26:24.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:24.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.905230+0000", 2024-08-17T18:26:24.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000128007 2024-08-17T18:26:24.938 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.938 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:24.938 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.938 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:171716 58.5 58:affa78ef:::benchmark_data_smithi136_442124_object58100:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1740)", 2024-08-17T18:26:24.938 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:23:58.850679+0000", 2024-08-17T18:26:24.938 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 145.93997795999999, 2024-08-17T18:26:24.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0548675809999999, 2024-08-17T18:26:24.938 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:24.938 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:24.939 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:24.939 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:26:24.939 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:24.939 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 171716 2024-08-17T18:26:24.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.939 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:24.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:24.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.850679+0000", 2024-08-17T18:26:24.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:24.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.850679+0000", 2024-08-17T18:26:24.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:24.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.850679+0000", 2024-08-17T18:26:24.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:26:24.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:24.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.850683+0000", 2024-08-17T18:26:24.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3610000000000003e-06 2024-08-17T18:26:24.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:24.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.850691+0000", 2024-08-17T18:26:24.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9119999999999998e-06 2024-08-17T18:26:24.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:24.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.850695+0000", 2024-08-17T18:26:24.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.109e-06 2024-08-17T18:26:24.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.850746+0000", 2024-08-17T18:26:24.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1112000000000001e-05 2024-08-17T18:26:24.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:26:24.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.850756+0000", 2024-08-17T18:26:24.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0465000000000001e-05 2024-08-17T18:26:24.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.726831+0000", 2024-08-17T18:26:24.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.87607475400000001 2024-08-17T18:26:24.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:26:24.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.726839+0000", 2024-08-17T18:26:24.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9789999999999993e-06 2024-08-17T18:26:24.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.893738+0000", 2024-08-17T18:26:24.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1668986619999999 2024-08-17T18:26:24.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:24.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.893823+0000", 2024-08-17T18:26:24.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5292999999999998e-05 2024-08-17T18:26:24.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1,2", 2024-08-17T18:26:24.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.893914+0000", 2024-08-17T18:26:24.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0909999999999995e-05 2024-08-17T18:26:24.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:24.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.898821+0000", 2024-08-17T18:26:24.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049072009999999999 2024-08-17T18:26:24.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.898859+0000", 2024-08-17T18:26:24.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7673999999999999e-05 2024-08-17T18:26:24.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.905289+0000", 2024-08-17T18:26:24.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006430056 2024-08-17T18:26:24.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.905393+0000", 2024-08-17T18:26:24.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010377100000000001 2024-08-17T18:26:24.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:24.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.905441+0000", 2024-08-17T18:26:24.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8661e-05 2024-08-17T18:26:24.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:24.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.905547+0000", 2024-08-17T18:26:24.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105573 2024-08-17T18:26:24.946 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.946 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:24.946 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.946 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:171729 58.5 58:a5c79663:::benchmark_data_smithi136_442124_object58113:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1740)", 2024-08-17T18:26:24.946 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:23:58.887370+0000", 2024-08-17T18:26:24.946 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 145.90328714399999, 2024-08-17T18:26:24.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.018381894, 2024-08-17T18:26:24.947 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:24.947 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:24.947 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:24.947 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:26:24.947 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:24.947 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 171729 2024-08-17T18:26:24.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.947 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:24.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:24.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.887370+0000", 2024-08-17T18:26:24.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:24.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.887370+0000", 2024-08-17T18:26:24.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:24.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:24.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.887369+0000", 2024-08-17T18:26:24.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:26:24.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:24.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.887375+0000", 2024-08-17T18:26:24.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4859999999999997e-06 2024-08-17T18:26:24.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:24.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.887404+0000", 2024-08-17T18:26:24.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9901999999999999e-05 2024-08-17T18:26:24.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:24.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.887411+0000", 2024-08-17T18:26:24.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1689999999999997e-06 2024-08-17T18:26:24.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.887455+0000", 2024-08-17T18:26:24.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4332999999999997e-05 2024-08-17T18:26:24.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:26:24.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:58.887463+0000", 2024-08-17T18:26:24.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0290000000000005e-06 2024-08-17T18:26:24.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.726860+0000", 2024-08-17T18:26:24.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.83939684999999997 2024-08-17T18:26:24.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:26:24.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.726868+0000", 2024-08-17T18:26:24.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0420000000000003e-06 2024-08-17T18:26:24.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:24.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.895946+0000", 2024-08-17T18:26:24.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1690777240000001 2024-08-17T18:26:24.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:24.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.896074+0000", 2024-08-17T18:26:24.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000128291 2024-08-17T18:26:24.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1,2", 2024-08-17T18:26:24.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.896203+0000", 2024-08-17T18:26:24.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000128825 2024-08-17T18:26:24.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:24.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.898873+0000", 2024-08-17T18:26:24.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026706999999999998 2024-08-17T18:26:24.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.899718+0000", 2024-08-17T18:26:24.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00084447099999999996 2024-08-17T18:26:24.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.905338+0000", 2024-08-17T18:26:24.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056205049999999996 2024-08-17T18:26:24.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:24.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.905605+0000", 2024-08-17T18:26:24.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026678599999999998 2024-08-17T18:26:24.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:24.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.905653+0000", 2024-08-17T18:26:24.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7404000000000002e-05 2024-08-17T18:26:24.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:24.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:24.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:24.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.905752+0000", 2024-08-17T18:26:24.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9563999999999997e-05 2024-08-17T18:26:24.955 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.955 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:24.955 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.955 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:24.955 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:24.955 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:26:25.138 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:24 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3674531282' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:26:25.138 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:24 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3674531282' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:26:25.140 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:24 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3674531282' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T18:26:25.666 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-17T18:26:25.869 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:26:25.992 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:25 smithi165 ceph-mon[105517]: pgmap v3544: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:26.022 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T18:26:26.137 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:25 smithi136 ceph-mon[109861]: pgmap v3544: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:26.137 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:25 smithi136 ceph-mon[115078]: pgmap v3544: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:26.989 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:26:27.107 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:26:27.108 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T18:26:27.637 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:26:27.637 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:26:27.637 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-08-17T18:26:27.637 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:26:27.786 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:26:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:26:27] "GET /metrics HTTP/1.1" 200 36675 "" "Prometheus/2.51.0" 2024-08-17T18:26:27.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:27 smithi165 ceph-mon[105517]: pgmap v3545: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:28.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:27 smithi136 ceph-mon[109861]: pgmap v3545: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:28.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:27 smithi136 ceph-mon[115078]: pgmap v3545: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:28.265 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-17T18:26:28.471 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:26:28.534 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":3545,"stamp":"2024-08-17T18:26:26.646514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595941,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":1194049,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1698,"ondisk_log_size":1698,"up":249,"acting":249,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":300,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8044700,"kb_used_data":89220,"kb_used_omap":474,"kb_used_meta":7954853,"kb_avail":741916516,"statfs":{"total":767960285184,"available":759722512384,"internally_reserved":0,"allocated":91361280,"data_stored":50694778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":485693,"internal_metadata":8145770179},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-336,"num_objects":-37,"num_object_clones":0,"num_object_copies":-111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-37,"num_whiteouts":0,"num_read":-3571,"num_read_kb":-12540,"num_write":-2229,"num_write_kb":-6271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001592"},"pg_stats":[{"pgid":"3.1f","version":"70'4","reported_seq":89,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:24.580164+0000","last_change":"2024-08-17T18:23:59.758838+0000","last_active":"2024-08-17T18:26:24.580164+0000","last_peered":"2024-08-17T18:26:24.580164+0000","last_clean":"2024-08-17T18:26:24.580164+0000","last_became_active":"2024-08-17T18:23:59.748445+0000","last_became_peered":"2024-08-17T18:23:59.748445+0000","last_unstale":"2024-08-17T18:26:24.580164+0000","last_undegraded":"2024-08-17T18:26:24.580164+0000","last_fullsized":"2024-08-17T18:26:24.580164+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:08:27.581912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1840,"reported_epoch":1775,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.732610+0000","last_change":"2024-08-17T18:23:36.324372+0000","last_active":"2024-08-17T18:26:22.732610+0000","last_peered":"2024-08-17T18:26:22.732610+0000","last_clean":"2024-08-17T18:26:22.732610+0000","last_became_active":"2024-08-17T18:23:36.323893+0000","last_became_peered":"2024-08-17T18:23:36.323893+0000","last_unstale":"2024-08-17T18:26:22.732610+0000","last_undegraded":"2024-08-17T18:26:22.732610+0000","last_fullsized":"2024-08-17T18:26:22.732610+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:10:52.829170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":247,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:23.893785+0000","last_change":"2024-08-17T18:23:36.319358+0000","last_active":"2024-08-17T18:26:23.893785+0000","last_peered":"2024-08-17T18:26:23.893785+0000","last_clean":"2024-08-17T18:26:23.893785+0000","last_became_active":"2024-08-17T18:23:36.318521+0000","last_became_peered":"2024-08-17T18:23:36.318521+0000","last_unstale":"2024-08-17T18:26:23.893785+0000","last_undegraded":"2024-08-17T18:26:23.893785+0000","last_fullsized":"2024-08-17T18:26:23.893785+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:56:48.984477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":240,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:23.893254+0000","last_change":"2024-08-17T18:23:59.741806+0000","last_active":"2024-08-17T18:26:23.893254+0000","last_peered":"2024-08-17T18:26:23.893254+0000","last_clean":"2024-08-17T18:26:23.893254+0000","last_became_active":"2024-08-17T18:23:59.741479+0000","last_became_peered":"2024-08-17T18:23:59.741479+0000","last_unstale":"2024-08-17T18:26:23.893254+0000","last_undegraded":"2024-08-17T18:26:23.893254+0000","last_fullsized":"2024-08-17T18:26:23.893254+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:02:39.072277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":348,"reported_epoch":1774,"state":"active+clean","last_fresh":"2024-08-17T18:26:20.937355+0000","last_change":"2024-08-17T18:22:36.836343+0000","last_active":"2024-08-17T18:26:20.937355+0000","last_peered":"2024-08-17T18:26:20.937355+0000","last_clean":"2024-08-17T18:26:20.937355+0000","last_became_active":"2024-08-17T18:22:36.835580+0000","last_became_peered":"2024-08-17T18:22:36.835580+0000","last_unstale":"2024-08-17T18:26:20.937355+0000","last_undegraded":"2024-08-17T18:26:20.937355+0000","last_fullsized":"2024-08-17T18:26:20.937355+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:05:51.737218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":240,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:23.893427+0000","last_change":"2024-08-17T18:23:59.742425+0000","last_active":"2024-08-17T18:26:23.893427+0000","last_peered":"2024-08-17T18:26:23.893427+0000","last_clean":"2024-08-17T18:26:23.893427+0000","last_became_active":"2024-08-17T18:23:59.742155+0000","last_became_peered":"2024-08-17T18:23:59.742155+0000","last_unstale":"2024-08-17T18:26:23.893427+0000","last_undegraded":"2024-08-17T18:26:23.893427+0000","last_fullsized":"2024-08-17T18:26:23.893427+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:11:42.833289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":359,"reported_epoch":1774,"state":"active+clean","last_fresh":"2024-08-17T18:26:20.937924+0000","last_change":"2024-08-17T18:20:32.094881+0000","last_active":"2024-08-17T18:26:20.937924+0000","last_peered":"2024-08-17T18:26:20.937924+0000","last_clean":"2024-08-17T18:26:20.937924+0000","last_became_active":"2024-08-17T18:20:32.077429+0000","last_became_peered":"2024-08-17T18:20:32.077429+0000","last_unstale":"2024-08-17T18:26:20.937924+0000","last_undegraded":"2024-08-17T18:26:20.937924+0000","last_fullsized":"2024-08-17T18:26:20.937924+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:08:26.290851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1845,"reported_epoch":1775,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.734636+0000","last_change":"2024-08-17T18:25:14.026295+0000","last_active":"2024-08-17T18:26:22.734636+0000","last_peered":"2024-08-17T18:26:22.734636+0000","last_clean":"2024-08-17T18:26:22.734636+0000","last_became_active":"2024-08-17T18:25:14.025999+0000","last_became_peered":"2024-08-17T18:25:14.025999+0000","last_unstale":"2024-08-17T18:26:22.734636+0000","last_undegraded":"2024-08-17T18:26:22.734636+0000","last_fullsized":"2024-08-17T18:26:22.734636+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:55:16.975299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":82,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:24.580493+0000","last_change":"2024-08-17T18:23:59.745317+0000","last_active":"2024-08-17T18:26:24.580493+0000","last_peered":"2024-08-17T18:26:24.580493+0000","last_clean":"2024-08-17T18:26:24.580493+0000","last_became_active":"2024-08-17T18:23:59.743236+0000","last_became_peered":"2024-08-17T18:23:59.743236+0000","last_unstale":"2024-08-17T18:26:24.580493+0000","last_undegraded":"2024-08-17T18:26:24.580493+0000","last_fullsized":"2024-08-17T18:26:24.580493+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:11.253490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":109,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.814111+0000","last_change":"2024-08-17T18:22:53.938704+0000","last_active":"2024-08-17T18:26:22.814111+0000","last_peered":"2024-08-17T18:26:22.814111+0000","last_clean":"2024-08-17T18:26:22.814111+0000","last_became_active":"2024-08-17T18:22:53.938406+0000","last_became_peered":"2024-08-17T18:22:53.938406+0000","last_unstale":"2024-08-17T18:26:22.814111+0000","last_undegraded":"2024-08-17T18:26:22.814111+0000","last_fullsized":"2024-08-17T18:26:22.814111+0000","mapping_epoch":1727,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:58:21.566014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1822,"reported_epoch":1775,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.732882+0000","last_change":"2024-08-17T18:20:31.942863+0000","last_active":"2024-08-17T18:26:22.732882+0000","last_peered":"2024-08-17T18:26:22.732882+0000","last_clean":"2024-08-17T18:26:22.732882+0000","last_became_active":"2024-08-17T18:20:31.941281+0000","last_became_peered":"2024-08-17T18:20:31.941281+0000","last_unstale":"2024-08-17T18:26:22.732882+0000","last_undegraded":"2024-08-17T18:26:22.732882+0000","last_fullsized":"2024-08-17T18:26:22.732882+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:48:05.556975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":240,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:23.893770+0000","last_change":"2024-08-17T18:20:31.933142+0000","last_active":"2024-08-17T18:26:23.893770+0000","last_peered":"2024-08-17T18:26:23.893770+0000","last_clean":"2024-08-17T18:26:23.893770+0000","last_became_active":"2024-08-17T18:20:31.932798+0000","last_became_peered":"2024-08-17T18:20:31.932798+0000","last_unstale":"2024-08-17T18:26:23.893770+0000","last_undegraded":"2024-08-17T18:26:23.893770+0000","last_fullsized":"2024-08-17T18:26:23.893770+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:32:51.059352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":174,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:24.263092+0000","last_change":"2024-08-17T18:20:31.937627+0000","last_active":"2024-08-17T18:26:24.263092+0000","last_peered":"2024-08-17T18:26:24.263092+0000","last_clean":"2024-08-17T18:26:24.263092+0000","last_became_active":"2024-08-17T18:20:31.937361+0000","last_became_peered":"2024-08-17T18:20:31.937361+0000","last_unstale":"2024-08-17T18:26:24.263092+0000","last_undegraded":"2024-08-17T18:26:24.263092+0000","last_fullsized":"2024-08-17T18:26:24.263092+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:01:26.309656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":83,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:24.580593+0000","last_change":"2024-08-17T18:23:59.743052+0000","last_active":"2024-08-17T18:26:24.580593+0000","last_peered":"2024-08-17T18:26:24.580593+0000","last_clean":"2024-08-17T18:26:24.580593+0000","last_became_active":"2024-08-17T18:23:59.742158+0000","last_became_peered":"2024-08-17T18:23:59.742158+0000","last_unstale":"2024-08-17T18:26:24.580593+0000","last_undegraded":"2024-08-17T18:26:24.580593+0000","last_fullsized":"2024-08-17T18:26:24.580593+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:34:34.249644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":261,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:23.893370+0000","last_change":"2024-08-17T18:25:14.176708+0000","last_active":"2024-08-17T18:26:23.893370+0000","last_peered":"2024-08-17T18:26:23.893370+0000","last_clean":"2024-08-17T18:26:23.893370+0000","last_became_active":"2024-08-17T18:25:14.176239+0000","last_became_peered":"2024-08-17T18:25:14.176239+0000","last_unstale":"2024-08-17T18:26:23.893370+0000","last_undegraded":"2024-08-17T18:26:23.893370+0000","last_fullsized":"2024-08-17T18:26:23.893370+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:08:21.826456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1838,"reported_epoch":1775,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.735529+0000","last_change":"2024-08-17T18:22:36.867005+0000","last_active":"2024-08-17T18:26:22.735529+0000","last_peered":"2024-08-17T18:26:22.735529+0000","last_clean":"2024-08-17T18:26:22.735529+0000","last_became_active":"2024-08-17T18:22:36.866297+0000","last_became_peered":"2024-08-17T18:22:36.866297+0000","last_unstale":"2024-08-17T18:26:22.735529+0000","last_undegraded":"2024-08-17T18:26:22.735529+0000","last_fullsized":"2024-08-17T18:26:22.735529+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:09:18.460006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":263,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:23.893514+0000","last_change":"2024-08-17T18:25:14.035767+0000","last_active":"2024-08-17T18:26:23.893514+0000","last_peered":"2024-08-17T18:26:23.893514+0000","last_clean":"2024-08-17T18:26:23.893514+0000","last_became_active":"2024-08-17T18:25:14.034592+0000","last_became_peered":"2024-08-17T18:25:14.034592+0000","last_unstale":"2024-08-17T18:26:23.893514+0000","last_undegraded":"2024-08-17T18:26:23.893514+0000","last_fullsized":"2024-08-17T18:26:23.893514+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:22:00.711481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":83,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:24.580237+0000","last_change":"2024-08-17T18:23:59.744879+0000","last_active":"2024-08-17T18:26:24.580237+0000","last_peered":"2024-08-17T18:26:24.580237+0000","last_clean":"2024-08-17T18:26:24.580237+0000","last_became_active":"2024-08-17T18:23:59.743333+0000","last_became_peered":"2024-08-17T18:23:59.743333+0000","last_unstale":"2024-08-17T18:26:24.580237+0000","last_undegraded":"2024-08-17T18:26:24.580237+0000","last_fullsized":"2024-08-17T18:26:24.580237+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:02:01.546706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":94,"reported_epoch":1774,"state":"active+clean","last_fresh":"2024-08-17T18:26:20.937216+0000","last_change":"2024-08-17T18:22:36.836745+0000","last_active":"2024-08-17T18:26:20.937216+0000","last_peered":"2024-08-17T18:26:20.937216+0000","last_clean":"2024-08-17T18:26:20.937216+0000","last_became_active":"2024-08-17T18:22:36.835161+0000","last_became_peered":"2024-08-17T18:22:36.835161+0000","last_unstale":"2024-08-17T18:26:20.937216+0000","last_undegraded":"2024-08-17T18:26:20.937216+0000","last_fullsized":"2024-08-17T18:26:20.937216+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:29:43.303633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":418,"reported_epoch":1774,"state":"active+clean","last_fresh":"2024-08-17T18:26:20.938141+0000","last_change":"2024-08-17T18:25:14.027677+0000","last_active":"2024-08-17T18:26:20.938141+0000","last_peered":"2024-08-17T18:26:20.938141+0000","last_clean":"2024-08-17T18:26:20.938141+0000","last_became_active":"2024-08-17T18:25:14.027309+0000","last_became_peered":"2024-08-17T18:25:14.027309+0000","last_unstale":"2024-08-17T18:26:20.938141+0000","last_undegraded":"2024-08-17T18:26:20.938141+0000","last_fullsized":"2024-08-17T18:26:20.938141+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:15:02.617263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":239,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:23.893722+0000","last_change":"2024-08-17T18:18:29.928751+0000","last_active":"2024-08-17T18:26:23.893722+0000","last_peered":"2024-08-17T18:26:23.893722+0000","last_clean":"2024-08-17T18:26:23.893722+0000","last_became_active":"2024-08-17T18:18:29.898102+0000","last_became_peered":"2024-08-17T18:18:29.898102+0000","last_unstale":"2024-08-17T18:26:23.893722+0000","last_undegraded":"2024-08-17T18:26:23.893722+0000","last_fullsized":"2024-08-17T18:26:23.893722+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:21.098266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":58,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:24.263516+0000","last_change":"2024-08-17T18:25:14.030155+0000","last_active":"2024-08-17T18:26:24.263516+0000","last_peered":"2024-08-17T18:26:24.263516+0000","last_clean":"2024-08-17T18:26:24.263516+0000","last_became_active":"2024-08-17T18:25:14.029876+0000","last_became_peered":"2024-08-17T18:25:14.029876+0000","last_unstale":"2024-08-17T18:26:24.263516+0000","last_undegraded":"2024-08-17T18:26:24.263516+0000","last_fullsized":"2024-08-17T18:26:24.263516+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:04:34.619858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1822,"reported_epoch":1775,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.736049+0000","last_change":"2024-08-17T18:22:36.838096+0000","last_active":"2024-08-17T18:26:22.736049+0000","last_peered":"2024-08-17T18:26:22.736049+0000","last_clean":"2024-08-17T18:26:22.736049+0000","last_became_active":"2024-08-17T18:22:36.835757+0000","last_became_peered":"2024-08-17T18:22:36.835757+0000","last_unstale":"2024-08-17T18:26:22.736049+0000","last_undegraded":"2024-08-17T18:26:22.736049+0000","last_fullsized":"2024-08-17T18:26:22.736049+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:55:31.016446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1813,"reported_epoch":1775,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.734801+0000","last_change":"2024-08-17T18:25:14.026273+0000","last_active":"2024-08-17T18:26:22.734801+0000","last_peered":"2024-08-17T18:26:22.734801+0000","last_clean":"2024-08-17T18:26:22.734801+0000","last_became_active":"2024-08-17T18:25:14.025964+0000","last_became_peered":"2024-08-17T18:25:14.025964+0000","last_unstale":"2024-08-17T18:26:22.734801+0000","last_undegraded":"2024-08-17T18:26:22.734801+0000","last_fullsized":"2024-08-17T18:26:22.734801+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:43.150679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"347'149","reported_seq":1061,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:23.893494+0000","last_change":"2024-08-17T18:20:33.927014+0000","last_active":"2024-08-17T18:26:23.893494+0000","last_peered":"2024-08-17T18:26:23.893494+0000","last_clean":"2024-08-17T18:26:23.893494+0000","last_became_active":"2024-08-17T18:20:33.926619+0000","last_became_peered":"2024-08-17T18:20:33.926619+0000","last_unstale":"2024-08-17T18:26:23.893494+0000","last_undegraded":"2024-08-17T18:26:23.893494+0000","last_fullsized":"2024-08-17T18:26:23.893494+0000","mapping_epoch":1697,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1698,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:57.271286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:57.271286+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:35:26.778777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1800,"reported_epoch":1775,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.734314+0000","last_change":"2024-08-17T18:22:36.838651+0000","last_active":"2024-08-17T18:26:22.734314+0000","last_peered":"2024-08-17T18:26:22.734314+0000","last_clean":"2024-08-17T18:26:22.734314+0000","last_became_active":"2024-08-17T18:22:36.838262+0000","last_became_peered":"2024-08-17T18:22:36.838262+0000","last_unstale":"2024-08-17T18:26:22.734314+0000","last_undegraded":"2024-08-17T18:26:22.734314+0000","last_fullsized":"2024-08-17T18:26:22.734314+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:24:25.119257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1781,"reported_epoch":1775,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.735455+0000","last_change":"2024-08-17T18:22:36.866053+0000","last_active":"2024-08-17T18:26:22.735455+0000","last_peered":"2024-08-17T18:26:22.735455+0000","last_clean":"2024-08-17T18:26:22.735455+0000","last_became_active":"2024-08-17T18:22:36.865187+0000","last_became_peered":"2024-08-17T18:22:36.865187+0000","last_unstale":"2024-08-17T18:26:22.735455+0000","last_undegraded":"2024-08-17T18:26:22.735455+0000","last_fullsized":"2024-08-17T18:26:22.735455+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:45:34.232504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":87,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:24.580584+0000","last_change":"2024-08-17T18:23:59.750520+0000","last_active":"2024-08-17T18:26:24.580584+0000","last_peered":"2024-08-17T18:26:24.580584+0000","last_clean":"2024-08-17T18:26:24.580584+0000","last_became_active":"2024-08-17T18:23:59.742332+0000","last_became_peered":"2024-08-17T18:23:59.742332+0000","last_unstale":"2024-08-17T18:26:24.580584+0000","last_undegraded":"2024-08-17T18:26:24.580584+0000","last_fullsized":"2024-08-17T18:26:24.580584+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:39:39.325194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"347'149","reported_seq":1089,"reported_epoch":1775,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.734579+0000","last_change":"2024-08-17T18:23:39.360135+0000","last_active":"2024-08-17T18:26:22.734579+0000","last_peered":"2024-08-17T18:26:22.734579+0000","last_clean":"2024-08-17T18:26:22.734579+0000","last_became_active":"2024-08-17T18:23:39.359652+0000","last_became_peered":"2024-08-17T18:23:39.359652+0000","last_unstale":"2024-08-17T18:26:22.734579+0000","last_undegraded":"2024-08-17T18:26:22.734579+0000","last_fullsized":"2024-08-17T18:26:22.734579+0000","mapping_epoch":1736,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1737,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:53.309517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:53.309517+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:20:52.590505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":247,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:23.893485+0000","last_change":"2024-08-17T18:23:59.746854+0000","last_active":"2024-08-17T18:26:23.893485+0000","last_peered":"2024-08-17T18:26:23.893485+0000","last_clean":"2024-08-17T18:26:23.893485+0000","last_became_active":"2024-08-17T18:23:59.746190+0000","last_became_peered":"2024-08-17T18:23:59.746190+0000","last_unstale":"2024-08-17T18:26:23.893485+0000","last_undegraded":"2024-08-17T18:26:23.893485+0000","last_fullsized":"2024-08-17T18:26:23.893485+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:01:53.108037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":58,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:24.263766+0000","last_change":"2024-08-17T18:25:14.025153+0000","last_active":"2024-08-17T18:26:24.263766+0000","last_peered":"2024-08-17T18:26:24.263766+0000","last_clean":"2024-08-17T18:26:24.263766+0000","last_became_active":"2024-08-17T18:25:14.024700+0000","last_became_peered":"2024-08-17T18:25:14.024700+0000","last_unstale":"2024-08-17T18:26:24.263766+0000","last_undegraded":"2024-08-17T18:26:24.263766+0000","last_fullsized":"2024-08-17T18:26:24.263766+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:50:25.014563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":249,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:23.893875+0000","last_change":"2024-08-17T18:23:59.772682+0000","last_active":"2024-08-17T18:26:23.893875+0000","last_peered":"2024-08-17T18:26:23.893875+0000","last_clean":"2024-08-17T18:26:23.893875+0000","last_became_active":"2024-08-17T18:23:59.748920+0000","last_became_peered":"2024-08-17T18:23:59.748920+0000","last_unstale":"2024-08-17T18:26:23.893875+0000","last_undegraded":"2024-08-17T18:26:23.893875+0000","last_fullsized":"2024-08-17T18:26:23.893875+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:07:31.660610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"347'149","reported_seq":1079,"reported_epoch":1775,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.736172+0000","last_change":"2024-08-17T18:24:02.793338+0000","last_active":"2024-08-17T18:26:22.736172+0000","last_peered":"2024-08-17T18:26:22.736172+0000","last_clean":"2024-08-17T18:26:22.736172+0000","last_became_active":"2024-08-17T18:24:02.792982+0000","last_became_peered":"2024-08-17T18:24:02.792982+0000","last_unstale":"2024-08-17T18:26:22.736172+0000","last_undegraded":"2024-08-17T18:26:22.736172+0000","last_fullsized":"2024-08-17T18:26:22.736172+0000","mapping_epoch":1743,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1744,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:52.474544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:52.474544+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:37:26.909105+0000","scrub_duration":0.001886326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":83,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:24.580158+0000","last_change":"2024-08-17T18:23:59.741315+0000","last_active":"2024-08-17T18:26:24.580158+0000","last_peered":"2024-08-17T18:26:24.580158+0000","last_clean":"2024-08-17T18:26:24.580158+0000","last_became_active":"2024-08-17T18:23:59.741020+0000","last_became_peered":"2024-08-17T18:23:59.741020+0000","last_unstale":"2024-08-17T18:26:24.580158+0000","last_undegraded":"2024-08-17T18:26:24.580158+0000","last_fullsized":"2024-08-17T18:26:24.580158+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:46:47.093299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":189,"reported_epoch":1774,"state":"active+clean","last_fresh":"2024-08-17T18:26:20.807489+0000","last_change":"2024-08-17T18:23:59.740824+0000","last_active":"2024-08-17T18:26:20.807489+0000","last_peered":"2024-08-17T18:26:20.807489+0000","last_clean":"2024-08-17T18:26:20.807489+0000","last_became_active":"2024-08-17T18:23:59.740478+0000","last_became_peered":"2024-08-17T18:23:59.740478+0000","last_unstale":"2024-08-17T18:26:20.807489+0000","last_undegraded":"2024-08-17T18:26:20.807489+0000","last_fullsized":"2024-08-17T18:26:20.807489+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:44:30.520216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":174,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:24.263253+0000","last_change":"2024-08-17T18:20:32.012874+0000","last_active":"2024-08-17T18:26:24.263253+0000","last_peered":"2024-08-17T18:26:24.263253+0000","last_clean":"2024-08-17T18:26:24.263253+0000","last_became_active":"2024-08-17T18:20:32.012553+0000","last_became_peered":"2024-08-17T18:20:32.012553+0000","last_unstale":"2024-08-17T18:26:24.263253+0000","last_undegraded":"2024-08-17T18:26:24.263253+0000","last_fullsized":"2024-08-17T18:26:24.263253+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:14:17.188745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"347'149","reported_seq":1045,"reported_epoch":1775,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.735379+0000","last_change":"2024-08-17T18:22:36.866341+0000","last_active":"2024-08-17T18:26:22.735379+0000","last_peered":"2024-08-17T18:26:22.735379+0000","last_clean":"2024-08-17T18:26:22.735379+0000","last_became_active":"2024-08-17T18:22:36.865561+0000","last_became_peered":"2024-08-17T18:22:36.865561+0000","last_unstale":"2024-08-17T18:26:22.735379+0000","last_undegraded":"2024-08-17T18:26:22.735379+0000","last_fullsized":"2024-08-17T18:26:22.735379+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:49.461698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:49.461698+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:22:21.423162+0000","scrub_duration":0.0013792520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.814321+0000","last_change":"2024-08-17T18:26:13.338201+0000","last_active":"2024-08-17T18:26:22.814321+0000","last_peered":"2024-08-17T18:26:22.814321+0000","last_clean":"2024-08-17T18:26:22.814321+0000","last_became_active":"2024-08-17T18:26:13.337933+0000","last_became_peered":"2024-08-17T18:26:13.337933+0000","last_unstale":"2024-08-17T18:26:22.814321+0000","last_undegraded":"2024-08-17T18:26:22.814321+0000","last_fullsized":"2024-08-17T18:26:22.814321+0000","mapping_epoch":1768,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1769,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:16:05.784689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":174,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:24.263771+0000","last_change":"2024-08-17T18:20:32.014875+0000","last_active":"2024-08-17T18:26:24.263771+0000","last_peered":"2024-08-17T18:26:24.263771+0000","last_clean":"2024-08-17T18:26:24.263771+0000","last_became_active":"2024-08-17T18:20:32.014631+0000","last_became_peered":"2024-08-17T18:20:32.014631+0000","last_unstale":"2024-08-17T18:26:24.263771+0000","last_undegraded":"2024-08-17T18:26:24.263771+0000","last_fullsized":"2024-08-17T18:26:24.263771+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:00:50.978635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":349,"reported_epoch":1774,"state":"active+clean","last_fresh":"2024-08-17T18:26:20.938007+0000","last_change":"2024-08-17T18:18:29.735362+0000","last_active":"2024-08-17T18:26:20.938007+0000","last_peered":"2024-08-17T18:26:20.938007+0000","last_clean":"2024-08-17T18:26:20.938007+0000","last_became_active":"2024-08-17T18:18:29.734752+0000","last_became_peered":"2024-08-17T18:18:29.734752+0000","last_unstale":"2024-08-17T18:26:20.938007+0000","last_undegraded":"2024-08-17T18:26:20.938007+0000","last_fullsized":"2024-08-17T18:26:20.938007+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:25.709698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"347'149","reported_seq":1089,"reported_epoch":1774,"state":"active+clean","last_fresh":"2024-08-17T18:26:20.937666+0000","last_change":"2024-08-17T18:24:02.792784+0000","last_active":"2024-08-17T18:26:20.937666+0000","last_peered":"2024-08-17T18:26:20.937666+0000","last_clean":"2024-08-17T18:26:20.937666+0000","last_became_active":"2024-08-17T18:24:02.792286+0000","last_became_peered":"2024-08-17T18:24:02.792286+0000","last_unstale":"2024-08-17T18:26:20.937666+0000","last_undegraded":"2024-08-17T18:26:20.937666+0000","last_fullsized":"2024-08-17T18:26:20.937666+0000","mapping_epoch":1743,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1744,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:00.316884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:00.316884+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:26:57.982520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":58,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:24.263542+0000","last_change":"2024-08-17T18:25:14.031207+0000","last_active":"2024-08-17T18:26:24.263542+0000","last_peered":"2024-08-17T18:26:24.263542+0000","last_clean":"2024-08-17T18:26:24.263542+0000","last_became_active":"2024-08-17T18:25:14.030916+0000","last_became_peered":"2024-08-17T18:25:14.030916+0000","last_unstale":"2024-08-17T18:26:24.263542+0000","last_undegraded":"2024-08-17T18:26:24.263542+0000","last_fullsized":"2024-08-17T18:26:24.263542+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:37:34.050072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1836,"reported_epoch":1775,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.734891+0000","last_change":"2024-08-17T18:23:59.741594+0000","last_active":"2024-08-17T18:26:22.734891+0000","last_peered":"2024-08-17T18:26:22.734891+0000","last_clean":"2024-08-17T18:26:22.734891+0000","last_became_active":"2024-08-17T18:23:59.741207+0000","last_became_peered":"2024-08-17T18:23:59.741207+0000","last_unstale":"2024-08-17T18:26:22.734891+0000","last_undegraded":"2024-08-17T18:26:22.734891+0000","last_fullsized":"2024-08-17T18:26:22.734891+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:02:52.545413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":418,"reported_epoch":1774,"state":"active+clean","last_fresh":"2024-08-17T18:26:20.937837+0000","last_change":"2024-08-17T18:22:36.836104+0000","last_active":"2024-08-17T18:26:20.937837+0000","last_peered":"2024-08-17T18:26:20.937837+0000","last_clean":"2024-08-17T18:26:20.937837+0000","last_became_active":"2024-08-17T18:22:36.835273+0000","last_became_peered":"2024-08-17T18:22:36.835273+0000","last_unstale":"2024-08-17T18:26:20.937837+0000","last_undegraded":"2024-08-17T18:26:20.937837+0000","last_fullsized":"2024-08-17T18:26:20.937837+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:26:23.494575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"347'149","reported_seq":875,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:24.580460+0000","last_change":"2024-08-17T18:24:02.792482+0000","last_active":"2024-08-17T18:26:24.580460+0000","last_peered":"2024-08-17T18:26:24.580460+0000","last_clean":"2024-08-17T18:26:24.580460+0000","last_became_active":"2024-08-17T18:24:02.792032+0000","last_became_peered":"2024-08-17T18:24:02.792032+0000","last_unstale":"2024-08-17T18:26:24.580460+0000","last_undegraded":"2024-08-17T18:26:24.580460+0000","last_fullsized":"2024-08-17T18:26:24.580460+0000","mapping_epoch":1743,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1744,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:02.363531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:02.363531+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:08:22.251652+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":58,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:24.263601+0000","last_change":"2024-08-17T18:25:14.176621+0000","last_active":"2024-08-17T18:26:24.263601+0000","last_peered":"2024-08-17T18:26:24.263601+0000","last_clean":"2024-08-17T18:26:24.263601+0000","last_became_active":"2024-08-17T18:25:14.176321+0000","last_became_peered":"2024-08-17T18:25:14.176321+0000","last_unstale":"2024-08-17T18:26:24.263601+0000","last_undegraded":"2024-08-17T18:26:24.263601+0000","last_fullsized":"2024-08-17T18:26:24.263601+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:40:55.297225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":185,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:24.263492+0000","last_change":"2024-08-17T18:22:36.834643+0000","last_active":"2024-08-17T18:26:24.263492+0000","last_peered":"2024-08-17T18:26:24.263492+0000","last_clean":"2024-08-17T18:26:24.263492+0000","last_became_active":"2024-08-17T18:22:36.834279+0000","last_became_peered":"2024-08-17T18:22:36.834279+0000","last_unstale":"2024-08-17T18:26:24.263492+0000","last_undegraded":"2024-08-17T18:26:24.263492+0000","last_fullsized":"2024-08-17T18:26:24.263492+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:29:36.002584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2856,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.814193+0000","last_change":"2024-08-17T18:18:29.742742+0000","last_active":"2024-08-17T18:26:22.814193+0000","last_peered":"2024-08-17T18:26:22.814193+0000","last_clean":"2024-08-17T18:26:22.814193+0000","last_became_active":"2024-08-17T18:18:29.742393+0000","last_became_peered":"2024-08-17T18:18:29.742393+0000","last_unstale":"2024-08-17T18:26:22.814193+0000","last_undegraded":"2024-08-17T18:26:22.814193+0000","last_fullsized":"2024-08-17T18:26:22.814193+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:57:32.858608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"347'149","reported_seq":862,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:24.263660+0000","last_change":"2024-08-17T18:23:39.370457+0000","last_active":"2024-08-17T18:26:24.263660+0000","last_peered":"2024-08-17T18:26:24.263660+0000","last_clean":"2024-08-17T18:26:24.263660+0000","last_became_active":"2024-08-17T18:23:39.370144+0000","last_became_peered":"2024-08-17T18:23:39.370144+0000","last_unstale":"2024-08-17T18:26:24.263660+0000","last_undegraded":"2024-08-17T18:26:24.263660+0000","last_fullsized":"2024-08-17T18:26:24.263660+0000","mapping_epoch":1736,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1737,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:20:35.854154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:20:35.854154+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:39:35.866331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":83,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:24.580158+0000","last_change":"2024-08-17T18:23:59.742951+0000","last_active":"2024-08-17T18:26:24.580158+0000","last_peered":"2024-08-17T18:26:24.580158+0000","last_clean":"2024-08-17T18:26:24.580158+0000","last_became_active":"2024-08-17T18:23:59.742510+0000","last_became_peered":"2024-08-17T18:23:59.742510+0000","last_unstale":"2024-08-17T18:26:24.580158+0000","last_undegraded":"2024-08-17T18:26:24.580158+0000","last_fullsized":"2024-08-17T18:26:24.580158+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:19:12.943501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":83,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:24.580342+0000","last_change":"2024-08-17T18:23:59.740786+0000","last_active":"2024-08-17T18:26:24.580342+0000","last_peered":"2024-08-17T18:26:24.580342+0000","last_clean":"2024-08-17T18:26:24.580342+0000","last_became_active":"2024-08-17T18:23:59.740340+0000","last_became_peered":"2024-08-17T18:23:59.740340+0000","last_unstale":"2024-08-17T18:26:24.580342+0000","last_undegraded":"2024-08-17T18:26:24.580342+0000","last_fullsized":"2024-08-17T18:26:24.580342+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:52:27.108557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":58,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:24.263403+0000","last_change":"2024-08-17T18:25:14.176308+0000","last_active":"2024-08-17T18:26:24.263403+0000","last_peered":"2024-08-17T18:26:24.263403+0000","last_clean":"2024-08-17T18:26:24.263403+0000","last_became_active":"2024-08-17T18:25:14.175851+0000","last_became_peered":"2024-08-17T18:25:14.175851+0000","last_unstale":"2024-08-17T18:26:24.263403+0000","last_undegraded":"2024-08-17T18:26:24.263403+0000","last_fullsized":"2024-08-17T18:26:24.263403+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:10:25.195682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":54,"reported_epoch":1775,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.734574+0000","last_change":"2024-08-17T18:23:36.318955+0000","last_active":"2024-08-17T18:26:22.734574+0000","last_peered":"2024-08-17T18:26:22.734574+0000","last_clean":"2024-08-17T18:26:22.734574+0000","last_became_active":"2024-08-17T18:23:36.318684+0000","last_became_peered":"2024-08-17T18:23:36.318684+0000","last_unstale":"2024-08-17T18:26:22.734574+0000","last_undegraded":"2024-08-17T18:26:22.734574+0000","last_fullsized":"2024-08-17T18:26:22.734574+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:47.191868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":244,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:23.893409+0000","last_change":"2024-08-17T18:22:36.835833+0000","last_active":"2024-08-17T18:26:23.893409+0000","last_peered":"2024-08-17T18:26:23.893409+0000","last_clean":"2024-08-17T18:26:23.893409+0000","last_became_active":"2024-08-17T18:22:36.835514+0000","last_became_peered":"2024-08-17T18:22:36.835514+0000","last_unstale":"2024-08-17T18:26:23.893409+0000","last_undegraded":"2024-08-17T18:26:23.893409+0000","last_fullsized":"2024-08-17T18:26:23.893409+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:43:53.668287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":364,"reported_epoch":1774,"state":"active+clean","last_fresh":"2024-08-17T18:26:20.937890+0000","last_change":"2024-08-17T18:19:26.374702+0000","last_active":"2024-08-17T18:26:20.937890+0000","last_peered":"2024-08-17T18:26:20.937890+0000","last_clean":"2024-08-17T18:26:20.937890+0000","last_became_active":"2024-08-17T18:15:53.304106+0000","last_became_peered":"2024-08-17T18:15:53.304106+0000","last_unstale":"2024-08-17T18:26:20.937890+0000","last_undegraded":"2024-08-17T18:26:20.937890+0000","last_fullsized":"2024-08-17T18:26:20.937890+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:21.409368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":1029,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:24.580395+0000","last_change":"2024-08-17T18:25:58.619520+0000","last_active":"2024-08-17T18:26:24.580395+0000","last_peered":"2024-08-17T18:26:24.580395+0000","last_clean":"2024-08-17T18:26:24.580395+0000","last_became_active":"2024-08-17T18:24:02.791616+0000","last_became_peered":"2024-08-17T18:24:02.791616+0000","last_unstale":"2024-08-17T18:26:24.580395+0000","last_undegraded":"2024-08-17T18:26:24.580395+0000","last_fullsized":"2024-08-17T18:26:24.580395+0000","mapping_epoch":1743,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1744,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:25:58.619429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:25:58.619429+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:23:08.373499+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"347'149","reported_seq":1070,"reported_epoch":1774,"state":"active+clean","last_fresh":"2024-08-17T18:26:20.937243+0000","last_change":"2024-08-17T18:22:36.836656+0000","last_active":"2024-08-17T18:26:20.937243+0000","last_peered":"2024-08-17T18:26:20.937243+0000","last_clean":"2024-08-17T18:26:20.937243+0000","last_became_active":"2024-08-17T18:22:36.835810+0000","last_became_peered":"2024-08-17T18:22:36.835810+0000","last_unstale":"2024-08-17T18:26:20.937243+0000","last_undegraded":"2024-08-17T18:26:20.937243+0000","last_fullsized":"2024-08-17T18:26:20.937243+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:55.255983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:55.255983+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:59:27.835153+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":174,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:24.263149+0000","last_change":"2024-08-17T18:20:31.939871+0000","last_active":"2024-08-17T18:26:24.263149+0000","last_peered":"2024-08-17T18:26:24.263149+0000","last_clean":"2024-08-17T18:26:24.263149+0000","last_became_active":"2024-08-17T18:20:31.939535+0000","last_became_peered":"2024-08-17T18:20:31.939535+0000","last_unstale":"2024-08-17T18:26:24.263149+0000","last_undegraded":"2024-08-17T18:26:24.263149+0000","last_fullsized":"2024-08-17T18:26:24.263149+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:43:31.630668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":254,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:23.893315+0000","last_change":"2024-08-17T18:25:14.036186+0000","last_active":"2024-08-17T18:26:23.893315+0000","last_peered":"2024-08-17T18:26:23.893315+0000","last_clean":"2024-08-17T18:26:23.893315+0000","last_became_active":"2024-08-17T18:25:14.035138+0000","last_became_peered":"2024-08-17T18:25:14.035138+0000","last_unstale":"2024-08-17T18:26:23.893315+0000","last_undegraded":"2024-08-17T18:26:23.893315+0000","last_fullsized":"2024-08-17T18:26:23.893315+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:51.645934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1851,"reported_epoch":1775,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.736395+0000","last_change":"2024-08-17T18:25:14.026973+0000","last_active":"2024-08-17T18:26:22.736395+0000","last_peered":"2024-08-17T18:26:22.736395+0000","last_clean":"2024-08-17T18:26:22.736395+0000","last_became_active":"2024-08-17T18:25:14.026478+0000","last_became_peered":"2024-08-17T18:25:14.026478+0000","last_unstale":"2024-08-17T18:26:22.736395+0000","last_undegraded":"2024-08-17T18:26:22.736395+0000","last_fullsized":"2024-08-17T18:26:22.736395+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:31.568544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1806,"reported_epoch":1775,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.733489+0000","last_change":"2024-08-17T18:15:53.244851+0000","last_active":"2024-08-17T18:26:22.733489+0000","last_peered":"2024-08-17T18:26:22.733489+0000","last_clean":"2024-08-17T18:26:22.733489+0000","last_became_active":"2024-08-17T18:15:53.240074+0000","last_became_peered":"2024-08-17T18:15:53.240074+0000","last_unstale":"2024-08-17T18:26:22.733489+0000","last_undegraded":"2024-08-17T18:26:22.733489+0000","last_fullsized":"2024-08-17T18:26:22.733489+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:09:00.168413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"347'149","reported_seq":1034,"reported_epoch":1774,"state":"active+clean","last_fresh":"2024-08-17T18:26:20.937468+0000","last_change":"2024-08-17T18:25:55.776006+0000","last_active":"2024-08-17T18:26:20.937468+0000","last_peered":"2024-08-17T18:26:20.937468+0000","last_clean":"2024-08-17T18:26:20.937468+0000","last_became_active":"2024-08-17T18:25:55.775559+0000","last_became_peered":"2024-08-17T18:25:55.775559+0000","last_unstale":"2024-08-17T18:26:20.937468+0000","last_undegraded":"2024-08-17T18:26:20.937468+0000","last_fullsized":"2024-08-17T18:26:20.937468+0000","mapping_epoch":1763,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1764,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:59.307356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:59.307356+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:52:14.784551+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1809,"reported_epoch":1775,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.735124+0000","last_change":"2024-08-17T18:20:31.942019+0000","last_active":"2024-08-17T18:26:22.735124+0000","last_peered":"2024-08-17T18:26:22.735124+0000","last_clean":"2024-08-17T18:26:22.735124+0000","last_became_active":"2024-08-17T18:20:31.941701+0000","last_became_peered":"2024-08-17T18:20:31.941701+0000","last_unstale":"2024-08-17T18:26:22.735124+0000","last_undegraded":"2024-08-17T18:26:22.735124+0000","last_fullsized":"2024-08-17T18:26:22.735124+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:58.908049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1786,"reported_epoch":1775,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.734357+0000","last_change":"2024-08-17T18:25:14.027871+0000","last_active":"2024-08-17T18:26:22.734357+0000","last_peered":"2024-08-17T18:26:22.734357+0000","last_clean":"2024-08-17T18:26:22.734357+0000","last_became_active":"2024-08-17T18:25:14.027353+0000","last_became_peered":"2024-08-17T18:25:14.027353+0000","last_unstale":"2024-08-17T18:26:22.734357+0000","last_undegraded":"2024-08-17T18:26:22.734357+0000","last_fullsized":"2024-08-17T18:26:22.734357+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:06.568798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"347'149","reported_seq":1031,"reported_epoch":1774,"state":"active+clean","last_fresh":"2024-08-17T18:26:20.807294+0000","last_change":"2024-08-17T18:18:33.757347+0000","last_active":"2024-08-17T18:26:20.807294+0000","last_peered":"2024-08-17T18:26:20.807294+0000","last_clean":"2024-08-17T18:26:20.807294+0000","last_became_active":"2024-08-17T18:18:33.756960+0000","last_became_peered":"2024-08-17T18:18:33.756960+0000","last_unstale":"2024-08-17T18:26:20.807294+0000","last_undegraded":"2024-08-17T18:26:20.807294+0000","last_fullsized":"2024-08-17T18:26:20.807294+0000","mapping_epoch":1670,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1671,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:52.271487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:52.271487+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:37:36.389244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1803,"reported_epoch":1775,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.735958+0000","last_change":"2024-08-17T18:22:36.837889+0000","last_active":"2024-08-17T18:26:22.735958+0000","last_peered":"2024-08-17T18:26:22.735958+0000","last_clean":"2024-08-17T18:26:22.735958+0000","last_became_active":"2024-08-17T18:22:36.837387+0000","last_became_peered":"2024-08-17T18:22:36.837387+0000","last_unstale":"2024-08-17T18:26:22.735958+0000","last_undegraded":"2024-08-17T18:26:22.735958+0000","last_fullsized":"2024-08-17T18:26:22.735958+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:11:29.265925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":270,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:23.893525+0000","last_change":"2024-08-17T18:23:59.770482+0000","last_active":"2024-08-17T18:26:23.893525+0000","last_peered":"2024-08-17T18:26:23.893525+0000","last_clean":"2024-08-17T18:26:23.893525+0000","last_became_active":"2024-08-17T18:23:59.747059+0000","last_became_peered":"2024-08-17T18:23:59.747059+0000","last_unstale":"2024-08-17T18:26:23.893525+0000","last_undegraded":"2024-08-17T18:26:23.893525+0000","last_fullsized":"2024-08-17T18:26:23.893525+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:04:17.042903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":83,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:24.580568+0000","last_change":"2024-08-17T18:23:59.745024+0000","last_active":"2024-08-17T18:26:24.580568+0000","last_peered":"2024-08-17T18:26:24.580568+0000","last_clean":"2024-08-17T18:26:24.580568+0000","last_became_active":"2024-08-17T18:23:59.741633+0000","last_became_peered":"2024-08-17T18:23:59.741633+0000","last_unstale":"2024-08-17T18:26:24.580568+0000","last_undegraded":"2024-08-17T18:26:24.580568+0000","last_fullsized":"2024-08-17T18:26:24.580568+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:41.514649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":372,"reported_epoch":1774,"state":"active+clean","last_fresh":"2024-08-17T18:26:20.937241+0000","last_change":"2024-08-17T18:22:36.836017+0000","last_active":"2024-08-17T18:26:20.937241+0000","last_peered":"2024-08-17T18:26:20.937241+0000","last_clean":"2024-08-17T18:26:20.937241+0000","last_became_active":"2024-08-17T18:22:36.835725+0000","last_became_peered":"2024-08-17T18:22:36.835725+0000","last_unstale":"2024-08-17T18:26:20.937241+0000","last_undegraded":"2024-08-17T18:26:20.937241+0000","last_fullsized":"2024-08-17T18:26:20.937241+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:18:07.395241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":54,"reported_epoch":1775,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.733995+0000","last_change":"2024-08-17T18:23:36.324099+0000","last_active":"2024-08-17T18:26:22.733995+0000","last_peered":"2024-08-17T18:26:22.733995+0000","last_clean":"2024-08-17T18:26:22.733995+0000","last_became_active":"2024-08-17T18:23:36.323739+0000","last_became_peered":"2024-08-17T18:23:36.323739+0000","last_unstale":"2024-08-17T18:26:22.733995+0000","last_undegraded":"2024-08-17T18:26:22.733995+0000","last_fullsized":"2024-08-17T18:26:22.733995+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:54:56.319013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":375,"reported_epoch":1774,"state":"active+clean","last_fresh":"2024-08-17T18:26:20.937401+0000","last_change":"2024-08-17T18:22:36.836529+0000","last_active":"2024-08-17T18:26:20.937401+0000","last_peered":"2024-08-17T18:26:20.937401+0000","last_clean":"2024-08-17T18:26:20.937401+0000","last_became_active":"2024-08-17T18:22:36.836020+0000","last_became_peered":"2024-08-17T18:22:36.836020+0000","last_unstale":"2024-08-17T18:26:20.937401+0000","last_undegraded":"2024-08-17T18:26:20.937401+0000","last_fullsized":"2024-08-17T18:26:20.937401+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:22.088303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1861,"reported_epoch":1775,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.733261+0000","last_change":"2024-08-17T18:25:14.040053+0000","last_active":"2024-08-17T18:26:22.733261+0000","last_peered":"2024-08-17T18:26:22.733261+0000","last_clean":"2024-08-17T18:26:22.733261+0000","last_became_active":"2024-08-17T18:25:14.039807+0000","last_became_peered":"2024-08-17T18:25:14.039807+0000","last_unstale":"2024-08-17T18:26:22.733261+0000","last_undegraded":"2024-08-17T18:26:22.733261+0000","last_fullsized":"2024-08-17T18:26:22.733261+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:05:53.486096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":168,"reported_epoch":1774,"state":"active+clean","last_fresh":"2024-08-17T18:26:20.807561+0000","last_change":"2024-08-17T18:20:32.006404+0000","last_active":"2024-08-17T18:26:20.807561+0000","last_peered":"2024-08-17T18:26:20.807561+0000","last_clean":"2024-08-17T18:26:20.807561+0000","last_became_active":"2024-08-17T18:20:31.947053+0000","last_became_peered":"2024-08-17T18:20:31.947053+0000","last_unstale":"2024-08-17T18:26:20.807561+0000","last_undegraded":"2024-08-17T18:26:20.807561+0000","last_fullsized":"2024-08-17T18:26:20.807561+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:21:06.537725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":375,"reported_epoch":1774,"state":"active+clean","last_fresh":"2024-08-17T18:26:20.937541+0000","last_change":"2024-08-17T18:23:36.318964+0000","last_active":"2024-08-17T18:26:20.937541+0000","last_peered":"2024-08-17T18:26:20.937541+0000","last_clean":"2024-08-17T18:26:20.937541+0000","last_became_active":"2024-08-17T18:23:36.318595+0000","last_became_peered":"2024-08-17T18:23:36.318595+0000","last_unstale":"2024-08-17T18:26:20.937541+0000","last_undegraded":"2024-08-17T18:26:20.937541+0000","last_fullsized":"2024-08-17T18:26:20.937541+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:29:16.221103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":194,"reported_epoch":1774,"state":"active+clean","last_fresh":"2024-08-17T18:26:20.807641+0000","last_change":"2024-08-17T18:23:59.752171+0000","last_active":"2024-08-17T18:26:20.807641+0000","last_peered":"2024-08-17T18:26:20.807641+0000","last_clean":"2024-08-17T18:26:20.807641+0000","last_became_active":"2024-08-17T18:23:59.745970+0000","last_became_peered":"2024-08-17T18:23:59.745970+0000","last_unstale":"2024-08-17T18:26:20.807641+0000","last_undegraded":"2024-08-17T18:26:20.807641+0000","last_fullsized":"2024-08-17T18:26:20.807641+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:24:52.861533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1933,"reported_epoch":1775,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.733537+0000","last_change":"2024-08-17T18:24:22.107318+0000","last_active":"2024-08-17T18:26:22.733537+0000","last_peered":"2024-08-17T18:26:22.733537+0000","last_clean":"2024-08-17T18:26:22.733537+0000","last_became_active":"2024-08-17T18:24:14.847882+0000","last_became_peered":"2024-08-17T18:24:14.847882+0000","last_unstale":"2024-08-17T18:26:22.733537+0000","last_undegraded":"2024-08-17T18:26:22.733537+0000","last_fullsized":"2024-08-17T18:26:22.733537+0000","mapping_epoch":1746,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:45:33.740721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":247,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:23.893897+0000","last_change":"2024-08-17T18:21:12.645070+0000","last_active":"2024-08-17T18:26:23.893897+0000","last_peered":"2024-08-17T18:26:23.893897+0000","last_clean":"2024-08-17T18:26:23.893897+0000","last_became_active":"2024-08-17T18:21:12.644748+0000","last_became_peered":"2024-08-17T18:21:12.644748+0000","last_unstale":"2024-08-17T18:26:23.893897+0000","last_undegraded":"2024-08-17T18:26:23.893897+0000","last_fullsized":"2024-08-17T18:26:23.893897+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:19:08.683431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":533,"reported_epoch":1775,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.733401+0000","last_change":"2024-08-17T18:22:36.838518+0000","last_active":"2024-08-17T18:26:22.733401+0000","last_peered":"2024-08-17T18:26:22.733401+0000","last_clean":"2024-08-17T18:26:22.733401+0000","last_became_active":"2024-08-17T18:22:36.834610+0000","last_became_peered":"2024-08-17T18:22:36.834610+0000","last_unstale":"2024-08-17T18:26:22.733401+0000","last_undegraded":"2024-08-17T18:26:22.733401+0000","last_fullsized":"2024-08-17T18:26:22.733401+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:42:18.798462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":83,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:24.580272+0000","last_change":"2024-08-17T18:23:59.740450+0000","last_active":"2024-08-17T18:26:24.580272+0000","last_peered":"2024-08-17T18:26:24.580272+0000","last_clean":"2024-08-17T18:26:24.580272+0000","last_became_active":"2024-08-17T18:23:59.740125+0000","last_became_peered":"2024-08-17T18:23:59.740125+0000","last_unstale":"2024-08-17T18:26:24.580272+0000","last_undegraded":"2024-08-17T18:26:24.580272+0000","last_fullsized":"2024-08-17T18:26:24.580272+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:26:38.858210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":83,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:24.580361+0000","last_change":"2024-08-17T18:23:59.740550+0000","last_active":"2024-08-17T18:26:24.580361+0000","last_peered":"2024-08-17T18:26:24.580361+0000","last_clean":"2024-08-17T18:26:24.580361+0000","last_became_active":"2024-08-17T18:23:59.740226+0000","last_became_peered":"2024-08-17T18:23:59.740226+0000","last_unstale":"2024-08-17T18:26:24.580361+0000","last_undegraded":"2024-08-17T18:26:24.580361+0000","last_fullsized":"2024-08-17T18:26:24.580361+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:35.745256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":366,"reported_epoch":1775,"state":"active+clean","last_fresh":"2024-08-17T18:26:22.340938+0000","last_change":"2024-08-17T18:18:29.900128+0000","last_active":"2024-08-17T18:26:22.340938+0000","last_peered":"2024-08-17T18:26:22.340938+0000","last_clean":"2024-08-17T18:26:22.340938+0000","last_became_active":"2024-08-17T18:18:29.899279+0000","last_became_peered":"2024-08-17T18:18:29.899279+0000","last_unstale":"2024-08-17T18:26:22.340938+0000","last_undegraded":"2024-08-17T18:26:22.340938+0000","last_fullsized":"2024-08-17T18:26:22.340938+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:01:29.045498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":158,"reported_epoch":1774,"state":"active+clean","last_fresh":"2024-08-17T18:26:20.807198+0000","last_change":"2024-08-17T18:15:53.271113+0000","last_active":"2024-08-17T18:26:20.807198+0000","last_peered":"2024-08-17T18:26:20.807198+0000","last_clean":"2024-08-17T18:26:20.807198+0000","last_became_active":"2024-08-17T18:15:53.230983+0000","last_became_peered":"2024-08-17T18:15:53.230983+0000","last_unstale":"2024-08-17T18:26:20.807198+0000","last_undegraded":"2024-08-17T18:26:20.807198+0000","last_fullsized":"2024-08-17T18:26:20.807198+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:00:35.315798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":265,"reported_epoch":1776,"state":"active+clean","last_fresh":"2024-08-17T18:26:23.893660+0000","last_change":"2024-08-17T18:25:14.030279+0000","last_active":"2024-08-17T18:26:23.893660+0000","last_peered":"2024-08-17T18:26:23.893660+0000","last_clean":"2024-08-17T18:26:23.893660+0000","last_became_active":"2024-08-17T18:25:14.027960+0000","last_became_peered":"2024-08-17T18:25:14.027960+0000","last_unstale":"2024-08-17T18:26:23.893660+0000","last_undegraded":"2024-08-17T18:26:23.893660+0000","last_fullsized":"2024-08-17T18:26:23.893660+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:20:46.317663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":13180,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234924,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":872520,"kb_used_data":11260,"kb_used_omap":1,"kb_used_meta":861246,"kb_avail":92872632,"statfs":{"total":95995035648,"available":95101575168,"internally_reserved":0,"allocated":11530240,"data_stored":6442214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":881916363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":1724,"seq":7404523618351,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":207124,"kb_used_data":10948,"kb_used_omap":123,"kb_used_meta":196036,"kb_avail":93538028,"statfs":{"total":95995035648,"available":95782940672,"internally_reserved":0,"allocated":11210752,"data_stored":6123931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":126809,"internal_metadata":200741031},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565804,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1302084,"kb_used_data":11512,"kb_used_omap":72,"kb_used_meta":1290487,"kb_avail":92443068,"statfs":{"total":95995035648,"available":94661701632,"internally_reserved":0,"allocated":11788288,"data_stored":6709556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":1321459182},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":1399,"seq":6008659247405,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":981876,"kb_used_data":11192,"kb_used_omap":131,"kb_used_meta":970492,"kb_avail":92763276,"statfs":{"total":95995035648,"available":94989594624,"internally_reserved":0,"allocated":11460608,"data_stored":6365566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134291,"internal_metadata":993784685},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743187194,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":981440,"kb_used_data":10932,"kb_used_omap":66,"kb_used_meta":970429,"kb_avail":92763712,"statfs":{"total":95995035648,"available":94990041088,"internally_reserved":0,"allocated":11194368,"data_stored":6119117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":993719979},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448833,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":917444,"kb_used_data":10936,"kb_used_omap":75,"kb_used_meta":906420,"kb_avail":92827708,"statfs":{"total":95995035648,"available":95055572992,"internally_reserved":0,"allocated":11198464,"data_stored":6119702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":928174271},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208758197,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1414068,"kb_used_data":11492,"kb_used_omap":1,"kb_used_meta":1402558,"kb_avail":92331084,"statfs":{"total":95995035648,"available":94547030016,"internally_reserved":0,"allocated":11767808,"data_stored":6691851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1436219846},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019834,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1368144,"kb_used_data":10948,"kb_used_omap":1,"kb_used_meta":1357182,"kb_avail":92377008,"statfs":{"total":95995035648,"available":94594056192,"internally_reserved":0,"allocated":11210752,"data_stored":6122841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1389754822},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T18:26:28.537 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-08-17T18:26:28.537 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T18:26:28.537 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-08-17T18:26:28.537 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-08-17T18:26:28.911 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:28 smithi165 ceph-mon[105517]: from='client.37191 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:26:29.142 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:28 smithi136 ceph-mon[109861]: from='client.37191 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:26:29.142 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:28 smithi136 ceph-mon[115078]: from='client.37191 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:26:29.460 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:26:29.833 INFO:teuthology.orchestra.run.smithi136.stderr:noscrub is unset 2024-08-17T18:26:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:29 smithi165 ceph-mon[105517]: pgmap v3546: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:29 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2241461556' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:26:30.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:29 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:26:30.192 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:29 smithi136 ceph-mon[109861]: pgmap v3546: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:30.192 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:29 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2241461556' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:26:30.192 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:29 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:26:30.192 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:29 smithi136 ceph-mon[115078]: pgmap v3546: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:30.192 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:29 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2241461556' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:26:30.192 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:29 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T18:26:30.247 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:26:30.247 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:26:30.247 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-08-17T18:26:30.247 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-08-17T18:26:30.247 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:26:30.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-17T18:26:30.973 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T18:26:31.124 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:30 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:26:31.124 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:30 smithi136 ceph-mon[109861]: osdmap e1777: 8 total, 8 up, 8 in 2024-08-17T18:26:31.124 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:30 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:26:31.125 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:30 smithi136 ceph-mon[115078]: osdmap e1777: 8 total, 8 up, 8 in 2024-08-17T18:26:31.125 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:26:31.233 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:30 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T18:26:31.233 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:30 smithi165 ceph-mon[105517]: osdmap e1777: 8 total, 8 up, 8 in 2024-08-17T18:26:31.518 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:26:32.003 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:31 smithi136 ceph-mon[109861]: pgmap v3548: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:32.004 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:31 smithi136 ceph-mon[115078]: pgmap v3548: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:31 smithi165 ceph-mon[105517]: pgmap v3548: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:32.835 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:26:32.835 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T18:26:32.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T18:26:32.836 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:26:32.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.836 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:160898 57.16 57:6bcdff34:::benchmark_data_smithi136_389235_object44976:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1659)", 2024-08-17T18:26:32.836 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:56.563419+0000", 2024-08-17T18:26:32.836 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 516.25242551099996, 2024-08-17T18:26:32.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0259137519999999, 2024-08-17T18:26:32.836 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:32.836 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:32.836 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:32.836 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:32.836 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:32.836 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 160898 2024-08-17T18:26:32.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.837 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:32.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:32.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.563419+0000", 2024-08-17T18:26:32.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:32.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.563419+0000", 2024-08-17T18:26:32.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:32.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.563417+0000", 2024-08-17T18:26:32.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:26:32.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:32.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.563427+0000", 2024-08-17T18:26:32.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0105000000000001e-05 2024-08-17T18:26:32.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:32.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.563454+0000", 2024-08-17T18:26:32.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.639e-05 2024-08-17T18:26:32.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:32.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.563464+0000", 2024-08-17T18:26:32.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0795000000000001e-05 2024-08-17T18:26:32.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.575209+0000", 2024-08-17T18:26:32.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011744675 2024-08-17T18:26:32.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:26:32.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.575220+0000", 2024-08-17T18:26:32.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1182999999999999e-05 2024-08-17T18:26:32.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.584432+0000", 2024-08-17T18:26:32.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.009211565 2024-08-17T18:26:32.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.840 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:32.840 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.584575+0000", 2024-08-17T18:26:32.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014335199999999999 2024-08-17T18:26:32.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T18:26:32.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.584924+0000", 2024-08-17T18:26:32.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034901399999999998 2024-08-17T18:26:32.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.589159+0000", 2024-08-17T18:26:32.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00423516 2024-08-17T18:26:32.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.589205+0000", 2024-08-17T18:26:32.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5757999999999998e-05 2024-08-17T18:26:32.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:32.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.589215+0000", 2024-08-17T18:26:32.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0037000000000001e-05 2024-08-17T18:26:32.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:32.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.589287+0000", 2024-08-17T18:26:32.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2329000000000005e-05 2024-08-17T18:26:32.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:32.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.589333+0000", 2024-08-17T18:26:32.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5303000000000003e-05 2024-08-17T18:26:32.843 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.843 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:32.843 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.843 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:160913 57.16 57:6c3e772a:::benchmark_data_smithi136_389235_object44991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1660)", 2024-08-17T18:26:32.843 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:56.598041+0000", 2024-08-17T18:26:32.843 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 516.21780392300002, 2024-08-17T18:26:32.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99375612099999999, 2024-08-17T18:26:32.844 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:32.844 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:32.844 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:32.844 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:32.844 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:32.844 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 160913 2024-08-17T18:26:32.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.844 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:32.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:32.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598041+0000", 2024-08-17T18:26:32.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:32.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598041+0000", 2024-08-17T18:26:32.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:32.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598040+0000", 2024-08-17T18:26:32.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:26:32.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:32.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598044+0000", 2024-08-17T18:26:32.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.685e-06 2024-08-17T18:26:32.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:32.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598055+0000", 2024-08-17T18:26:32.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.043e-05 2024-08-17T18:26:32.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:32.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598061+0000", 2024-08-17T18:26:32.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4269999999999996e-06 2024-08-17T18:26:32.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598122+0000", 2024-08-17T18:26:32.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.101e-05 2024-08-17T18:26:32.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:26:32.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598131+0000", 2024-08-17T18:26:32.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9679999999999995e-06 2024-08-17T18:26:32.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.587199+0000", 2024-08-17T18:26:32.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98906787600000001 2024-08-17T18:26:32.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:32.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.587308+0000", 2024-08-17T18:26:32.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108712 2024-08-17T18:26:32.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T18:26:32.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.587567+0000", 2024-08-17T18:26:32.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025900400000000002 2024-08-17T18:26:32.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.589363+0000", 2024-08-17T18:26:32.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017963650000000001 2024-08-17T18:26:32.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.591651+0000", 2024-08-17T18:26:32.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022876120000000001 2024-08-17T18:26:32.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:32.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.591719+0000", 2024-08-17T18:26:32.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8367000000000005e-05 2024-08-17T18:26:32.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:32.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.591759+0000", 2024-08-17T18:26:32.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9480999999999997e-05 2024-08-17T18:26:32.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:32.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.591797+0000", 2024-08-17T18:26:32.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7957999999999998e-05 2024-08-17T18:26:32.851 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.851 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:32.851 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.851 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:160914 57.16 57:681070b7:::benchmark_data_smithi136_389235_object44992:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1660)", 2024-08-17T18:26:32.851 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:56.598883+0000", 2024-08-17T18:26:32.851 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 516.21696120499996, 2024-08-17T18:26:32.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0028806669999999, 2024-08-17T18:26:32.851 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:32.851 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:32.852 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:32.852 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:32.852 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:32.852 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 160914 2024-08-17T18:26:32.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.852 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:32.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:32.852 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598883+0000", 2024-08-17T18:26:32.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.852 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:32.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598883+0000", 2024-08-17T18:26:32.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:32.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598882+0000", 2024-08-17T18:26:32.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:26:32.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:32.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598888+0000", 2024-08-17T18:26:32.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9750000000000004e-06 2024-08-17T18:26:32.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:32.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598903+0000", 2024-08-17T18:26:32.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4534e-05 2024-08-17T18:26:32.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:32.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598910+0000", 2024-08-17T18:26:32.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0979999999999996e-06 2024-08-17T18:26:32.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598952+0000", 2024-08-17T18:26:32.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2006999999999999e-05 2024-08-17T18:26:32.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:26:32.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.598960+0000", 2024-08-17T18:26:32.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3899999999999993e-06 2024-08-17T18:26:32.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.589545+0000", 2024-08-17T18:26:32.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.990585037 2024-08-17T18:26:32.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:32.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.589681+0000", 2024-08-17T18:26:32.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000136379 2024-08-17T18:26:32.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T18:26:32.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.590020+0000", 2024-08-17T18:26:32.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033900400000000001 2024-08-17T18:26:32.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.591710+0000", 2024-08-17T18:26:32.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016892420000000001 2024-08-17T18:26:32.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.597521+0000", 2024-08-17T18:26:32.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058114159999999998 2024-08-17T18:26:32.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:32.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.601595+0000", 2024-08-17T18:26:32.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040736549999999998 2024-08-17T18:26:32.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:32.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.601719+0000", 2024-08-17T18:26:32.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012428 2024-08-17T18:26:32.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:32.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.601764+0000", 2024-08-17T18:26:32.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4897000000000001e-05 2024-08-17T18:26:32.858 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.858 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:32.858 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.858 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:160956 57.16 57:69ce9ed4:::benchmark_data_smithi136_389235_object45034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1660)", 2024-08-17T18:26:32.859 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:56.725647+0000", 2024-08-17T18:26:32.859 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 516.09019719499997, 2024-08-17T18:26:32.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.88076719599999997, 2024-08-17T18:26:32.859 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:32.859 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:32.859 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:32.859 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:32.859 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:32.859 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 160956 2024-08-17T18:26:32.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.859 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:32.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.859 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:32.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.725647+0000", 2024-08-17T18:26:32.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:32.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.725647+0000", 2024-08-17T18:26:32.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:32.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.725646+0000", 2024-08-17T18:26:32.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:26:32.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:32.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.725652+0000", 2024-08-17T18:26:32.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2759999999999997e-06 2024-08-17T18:26:32.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:32.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.725664+0000", 2024-08-17T18:26:32.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1616e-05 2024-08-17T18:26:32.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:32.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.725670+0000", 2024-08-17T18:26:32.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4359999999999999e-06 2024-08-17T18:26:32.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.725720+0000", 2024-08-17T18:26:32.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9599999999999999e-05 2024-08-17T18:26:32.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:26:32.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.725727+0000", 2024-08-17T18:26:32.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1799999999999999e-06 2024-08-17T18:26:32.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.591831+0000", 2024-08-17T18:26:32.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.86610343700000003 2024-08-17T18:26:32.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:32.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.591902+0000", 2024-08-17T18:26:32.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1347999999999995e-05 2024-08-17T18:26:32.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T18:26:32.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.592142+0000", 2024-08-17T18:26:32.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024018299999999999 2024-08-17T18:26:32.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.597454+0000", 2024-08-17T18:26:32.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0053123390000000001 2024-08-17T18:26:32.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.597555+0000", 2024-08-17T18:26:32.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000100389 2024-08-17T18:26:32.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:32.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606321+0000", 2024-08-17T18:26:32.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0087664409999999998 2024-08-17T18:26:32.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:32.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606372+0000", 2024-08-17T18:26:32.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0843000000000001e-05 2024-08-17T18:26:32.865 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.865 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:32.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606414+0000", 2024-08-17T18:26:32.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2333999999999998e-05 2024-08-17T18:26:32.866 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.866 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:32.866 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.866 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.866 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.866 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:161012 57.16 57:6e84d2aa:::benchmark_data_smithi136_389235_object45090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1660)", 2024-08-17T18:26:32.866 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:56.934535+0000", 2024-08-17T18:26:32.866 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 515.88130956999998, 2024-08-17T18:26:32.866 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.68200467600000003, 2024-08-17T18:26:32.866 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:32.866 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:32.867 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:32.867 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:32.867 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:32.867 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161012 2024-08-17T18:26:32.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.867 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:32.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:32.867 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.934535+0000", 2024-08-17T18:26:32.867 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.867 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.867 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.867 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:32.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.934535+0000", 2024-08-17T18:26:32.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:32.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.934533+0000", 2024-08-17T18:26:32.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:26:32.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.868 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:32.868 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.934540+0000", 2024-08-17T18:26:32.868 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2890000000000003e-06 2024-08-17T18:26:32.868 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.868 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:32.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.934551+0000", 2024-08-17T18:26:32.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1762e-05 2024-08-17T18:26:32.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:32.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.934557+0000", 2024-08-17T18:26:32.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4399999999999996e-06 2024-08-17T18:26:32.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.869 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.869 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.869 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.934601+0000", 2024-08-17T18:26:32.869 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4533000000000002e-05 2024-08-17T18:26:32.869 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:26:32.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:56.934610+0000", 2024-08-17T18:26:32.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2199999999999992e-06 2024-08-17T18:26:32.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.594900+0000", 2024-08-17T18:26:32.870 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.66029078600000002 2024-08-17T18:26:32.870 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.870 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.870 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:32.870 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.595020+0000", 2024-08-17T18:26:32.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119622 2024-08-17T18:26:32.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T18:26:32.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.595434+0000", 2024-08-17T18:26:32.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000413805 2024-08-17T18:26:32.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.871 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:32.871 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606426+0000", 2024-08-17T18:26:32.871 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010991772 2024-08-17T18:26:32.871 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.871 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606470+0000", 2024-08-17T18:26:32.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4162999999999997e-05 2024-08-17T18:26:32.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616445+0000", 2024-08-17T18:26:32.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0099751089999999994 2024-08-17T18:26:32.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.872 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.872 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:32.872 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616496+0000", 2024-08-17T18:26:32.872 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1069000000000002e-05 2024-08-17T18:26:32.872 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.873 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:32.873 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616540+0000", 2024-08-17T18:26:32.873 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3649000000000002e-05 2024-08-17T18:26:32.873 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.873 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:32.873 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.873 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.873 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.873 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:161032 57.16 57:6919a724:::benchmark_data_smithi136_389235_object45110:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1660)", 2024-08-17T18:26:32.873 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:57.066208+0000", 2024-08-17T18:26:32.873 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 515.74963630100001, 2024-08-17T18:26:32.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.55043567800000004, 2024-08-17T18:26:32.874 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:32.874 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:32.874 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:32.874 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:32.874 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:32.874 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161032 2024-08-17T18:26:32.874 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.874 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:32.874 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.874 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:32.874 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.066208+0000", 2024-08-17T18:26:32.874 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:32.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.066208+0000", 2024-08-17T18:26:32.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:32.875 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.066206+0000", 2024-08-17T18:26:32.875 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:26:32.875 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.875 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.875 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:32.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.066216+0000", 2024-08-17T18:26:32.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9129999999999998e-06 2024-08-17T18:26:32.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:32.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.066235+0000", 2024-08-17T18:26:32.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8524000000000001e-05 2024-08-17T18:26:32.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.876 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:32.876 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.066244+0000", 2024-08-17T18:26:32.876 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9039999999999998e-06 2024-08-17T18:26:32.876 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.876 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.066297+0000", 2024-08-17T18:26:32.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3087999999999997e-05 2024-08-17T18:26:32.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:26:32.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.066306+0000", 2024-08-17T18:26:32.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2140000000000002e-06 2024-08-17T18:26:32.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.877 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.877 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.877 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.597579+0000", 2024-08-17T18:26:32.877 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.53127298999999994 2024-08-17T18:26:32.877 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:32.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.597711+0000", 2024-08-17T18:26:32.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013171400000000001 2024-08-17T18:26:32.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T18:26:32.878 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.598077+0000", 2024-08-17T18:26:32.878 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00036651799999999998 2024-08-17T18:26:32.878 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.878 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.878 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:32.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606431+0000", 2024-08-17T18:26:32.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0083538680000000004 2024-08-17T18:26:32.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606528+0000", 2024-08-17T18:26:32.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7275999999999995e-05 2024-08-17T18:26:32.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.879 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.879 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616574+0000", 2024-08-17T18:26:32.879 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010045554 2024-08-17T18:26:32.879 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.879 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:32.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616610+0000", 2024-08-17T18:26:32.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6565e-05 2024-08-17T18:26:32.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.880 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.880 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:32.880 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616644+0000", 2024-08-17T18:26:32.880 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3522999999999997e-05 2024-08-17T18:26:32.880 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.880 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:32.880 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.880 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.881 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.881 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:161078 57.16 57:6e3b9c80:::benchmark_data_smithi136_389235_object45156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1660)", 2024-08-17T18:26:32.881 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:57.238646+0000", 2024-08-17T18:26:32.881 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 515.57719807399997, 2024-08-17T18:26:32.881 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.37810427000000002, 2024-08-17T18:26:32.881 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:32.881 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:32.881 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:32.881 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:32.881 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:32.881 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 161078 2024-08-17T18:26:32.881 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.881 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:32.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:32.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.238646+0000", 2024-08-17T18:26:32.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:32.882 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.238646+0000", 2024-08-17T18:26:32.882 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.882 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.882 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.882 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:32.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.238645+0000", 2024-08-17T18:26:32.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:26:32.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:32.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.238652+0000", 2024-08-17T18:26:32.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8349999999999998e-06 2024-08-17T18:26:32.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:32.883 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.238666+0000", 2024-08-17T18:26:32.883 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4346e-05 2024-08-17T18:26:32.883 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.883 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.883 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:32.885 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.238674+0000", 2024-08-17T18:26:32.885 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9889999999999992e-06 2024-08-17T18:26:32.885 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.885 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.239725+0000", 2024-08-17T18:26:32.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001050743 2024-08-17T18:26:32.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:26:32.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.239737+0000", 2024-08-17T18:26:32.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2510000000000001e-05 2024-08-17T18:26:32.886 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.886 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.886 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.599446+0000", 2024-08-17T18:26:32.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.359708378 2024-08-17T18:26:32.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:32.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.599580+0000", 2024-08-17T18:26:32.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000134485 2024-08-17T18:26:32.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.887 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-08-17T18:26:32.887 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.599828+0000", 2024-08-17T18:26:32.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024764999999999998 2024-08-17T18:26:32.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:32.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606479+0000", 2024-08-17T18:26:32.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066513500000000003 2024-08-17T18:26:32.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.606563+0000", 2024-08-17T18:26:32.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3973000000000004e-05 2024-08-17T18:26:32.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616677+0000", 2024-08-17T18:26:32.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010113924999999999 2024-08-17T18:26:32.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:32.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616718+0000", 2024-08-17T18:26:32.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0836e-05 2024-08-17T18:26:32.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:32.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:57.616751+0000", 2024-08-17T18:26:32.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2685999999999998e-05 2024-08-17T18:26:32.889 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.889 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:32.889 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.890 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:164936 57.a 57:5d3ff99b:::benchmark_data_smithi136_389235_object49014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1661)", 2024-08-17T18:26:32.890 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:06.210033+0000", 2024-08-17T18:26:32.890 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 506.605811883, 2024-08-17T18:26:32.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27177604100000002, 2024-08-17T18:26:32.890 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:32.890 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:32.890 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:32.890 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:32.890 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:32.890 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164936 2024-08-17T18:26:32.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.891 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:32.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:32.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210033+0000", 2024-08-17T18:26:32.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:32.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210033+0000", 2024-08-17T18:26:32.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:32.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210031+0000", 2024-08-17T18:26:32.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:26:32.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:32.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210039+0000", 2024-08-17T18:26:32.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9470000000000003e-06 2024-08-17T18:26:32.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:32.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210051+0000", 2024-08-17T18:26:32.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2771e-05 2024-08-17T18:26:32.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:32.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210058+0000", 2024-08-17T18:26:32.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8109999999999997e-06 2024-08-17T18:26:32.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210117+0000", 2024-08-17T18:26:32.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9086000000000003e-05 2024-08-17T18:26:32.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:32.893 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210260+0000", 2024-08-17T18:26:32.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014312299999999999 2024-08-17T18:26:32.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.893 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.893 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4,7", 2024-08-17T18:26:32.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.210404+0000", 2024-08-17T18:26:32.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014415700000000001 2024-08-17T18:26:32.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.213570+0000", 2024-08-17T18:26:32.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031656869999999999 2024-08-17T18:26:32.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.213627+0000", 2024-08-17T18:26:32.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6935999999999998e-05 2024-08-17T18:26:32.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:32.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.213639+0000", 2024-08-17T18:26:32.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2143e-05 2024-08-17T18:26:32.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.481648+0000", 2024-08-17T18:26:32.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.26800850399999998 2024-08-17T18:26:32.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:32.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.481700+0000", 2024-08-17T18:26:32.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2073000000000001e-05 2024-08-17T18:26:32.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:32.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.481809+0000", 2024-08-17T18:26:32.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000108818 2024-08-17T18:26:32.896 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.896 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:32.896 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.896 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:164945 57.a 57:55ee62ab:::benchmark_data_smithi136_389235_object49023:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1661)", 2024-08-17T18:26:32.896 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:06.230920+0000", 2024-08-17T18:26:32.897 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 506.58492397399999, 2024-08-17T18:26:32.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25469593200000001, 2024-08-17T18:26:32.897 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:32.897 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:32.897 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:32.897 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:32.897 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:32.897 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 164945 2024-08-17T18:26:32.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.897 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:32.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:32.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.230920+0000", 2024-08-17T18:26:32.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:32.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.230920+0000", 2024-08-17T18:26:32.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:32.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.230916+0000", 2024-08-17T18:26:32.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999957 2024-08-17T18:26:32.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:32.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.230926+0000", 2024-08-17T18:26:32.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4390000000000006e-06 2024-08-17T18:26:32.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:32.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.230938+0000", 2024-08-17T18:26:32.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2094e-05 2024-08-17T18:26:32.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:32.899 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.396017+0000", 2024-08-17T18:26:32.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.16507881099999999 2024-08-17T18:26:32.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.396075+0000", 2024-08-17T18:26:32.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8207e-05 2024-08-17T18:26:32.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:32.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.396236+0000", 2024-08-17T18:26:32.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016120800000000001 2024-08-17T18:26:32.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,4,7", 2024-08-17T18:26:32.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.396397+0000", 2024-08-17T18:26:32.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000160571 2024-08-17T18:26:32.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.398644+0000", 2024-08-17T18:26:32.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022471930000000002 2024-08-17T18:26:32.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:32.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.400494+0000", 2024-08-17T18:26:32.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018502049999999999 2024-08-17T18:26:32.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.404981+0000", 2024-08-17T18:26:32.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044866910000000001 2024-08-17T18:26:32.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.485449+0000", 2024-08-17T18:26:32.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.080468711999999998 2024-08-17T18:26:32.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:32.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.485506+0000", 2024-08-17T18:26:32.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6317999999999998e-05 2024-08-17T18:26:32.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:32.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:06.485616+0000", 2024-08-17T18:26:32.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000110649 2024-08-17T18:26:32.903 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.903 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:32.903 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.903 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:117666 58.2 58:47c92b81:::benchmark_data_smithi136_442124_object4050:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1726)", 2024-08-17T18:26:32.903 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:38.291041+0000", 2024-08-17T18:26:32.903 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 234.52480391500001, 2024-08-17T18:26:32.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27569550999999998, 2024-08-17T18:26:32.904 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:32.904 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:32.904 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:32.904 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:26:32.904 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:32.904 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 117666 2024-08-17T18:26:32.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.904 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:32.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:32.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.291041+0000", 2024-08-17T18:26:32.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:32.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.291041+0000", 2024-08-17T18:26:32.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:32.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.291038+0000", 2024-08-17T18:26:32.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999976 2024-08-17T18:26:32.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:32.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.291049+0000", 2024-08-17T18:26:32.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0883999999999999e-05 2024-08-17T18:26:32.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:32.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.291074+0000", 2024-08-17T18:26:32.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4553999999999999e-05 2024-08-17T18:26:32.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:32.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.291083+0000", 2024-08-17T18:26:32.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6590000000000002e-06 2024-08-17T18:26:32.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.291146+0000", 2024-08-17T18:26:32.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2277e-05 2024-08-17T18:26:32.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:32.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.291278+0000", 2024-08-17T18:26:32.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000132356 2024-08-17T18:26:32.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,4", 2024-08-17T18:26:32.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.291391+0000", 2024-08-17T18:26:32.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011304899999999999 2024-08-17T18:26:32.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:32.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.295625+0000", 2024-08-17T18:26:32.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042343459999999999 2024-08-17T18:26:32.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.308090+0000", 2024-08-17T18:26:32.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012464648 2024-08-17T18:26:32.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566641+0000", 2024-08-17T18:26:32.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25855076999999999 2024-08-17T18:26:32.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:32.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566689+0000", 2024-08-17T18:26:32.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8615000000000001e-05 2024-08-17T18:26:32.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:32.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566736+0000", 2024-08-17T18:26:32.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6635999999999999e-05 2024-08-17T18:26:32.909 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.909 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:32.910 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.910 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:117670 58.2 58:47193288:::benchmark_data_smithi136_442124_object4054:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1726)", 2024-08-17T18:26:32.910 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:38.295548+0000", 2024-08-17T18:26:32.910 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 234.52029597999999, 2024-08-17T18:26:32.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27135621100000001, 2024-08-17T18:26:32.911 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:32.911 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:32.911 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:32.911 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:26:32.911 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:32.911 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 117670 2024-08-17T18:26:32.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.912 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:32.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:32.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.295548+0000", 2024-08-17T18:26:32.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:32.912 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.295548+0000", 2024-08-17T18:26:32.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:32.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.295546+0000", 2024-08-17T18:26:32.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999976 2024-08-17T18:26:32.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:32.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.295557+0000", 2024-08-17T18:26:32.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.076e-05 2024-08-17T18:26:32.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:32.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.295572+0000", 2024-08-17T18:26:32.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5009e-05 2024-08-17T18:26:32.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:32.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.295581+0000", 2024-08-17T18:26:32.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2410000000000001e-06 2024-08-17T18:26:32.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.295648+0000", 2024-08-17T18:26:32.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6795e-05 2024-08-17T18:26:32.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:32.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.295776+0000", 2024-08-17T18:26:32.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000127811 2024-08-17T18:26:32.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,4", 2024-08-17T18:26:32.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.295890+0000", 2024-08-17T18:26:32.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000114164 2024-08-17T18:26:32.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:32.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.303987+0000", 2024-08-17T18:26:32.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080971359999999996 2024-08-17T18:26:32.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.916 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566789+0000", 2024-08-17T18:26:32.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.26280209500000001 2024-08-17T18:26:32.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.916 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566836+0000", 2024-08-17T18:26:32.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6341000000000002e-05 2024-08-17T18:26:32.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.916 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:32.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566872+0000", 2024-08-17T18:26:32.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6581000000000001e-05 2024-08-17T18:26:32.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:32.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566905+0000", 2024-08-17T18:26:32.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2428000000000002e-05 2024-08-17T18:26:32.917 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.917 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:32.917 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.917 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:117672 58.2 58:45143eb0:::benchmark_data_smithi136_442124_object4056:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1726)", 2024-08-17T18:26:32.918 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:38.296417+0000", 2024-08-17T18:26:32.918 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 234.51942763700001, 2024-08-17T18:26:32.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.27061583500000003, 2024-08-17T18:26:32.918 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:32.918 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:32.918 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:32.918 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:26:32.918 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:32.918 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 117672 2024-08-17T18:26:32.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.918 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:32.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:32.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.296417+0000", 2024-08-17T18:26:32.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:32.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.296417+0000", 2024-08-17T18:26:32.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:32.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.296416+0000", 2024-08-17T18:26:32.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:26:32.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:32.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.296421+0000", 2024-08-17T18:26:32.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9779999999999996e-06 2024-08-17T18:26:32.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:32.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.296431+0000", 2024-08-17T18:26:32.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8379999999999997e-06 2024-08-17T18:26:32.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:32.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.296439+0000", 2024-08-17T18:26:32.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6920000000000008e-06 2024-08-17T18:26:32.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.299968+0000", 2024-08-17T18:26:32.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035287209999999999 2024-08-17T18:26:32.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:32.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.300084+0000", 2024-08-17T18:26:32.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011565699999999999 2024-08-17T18:26:32.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,4", 2024-08-17T18:26:32.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.300188+0000", 2024-08-17T18:26:32.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104132 2024-08-17T18:26:32.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:32.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.308107+0000", 2024-08-17T18:26:32.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0079187189999999994 2024-08-17T18:26:32.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566948+0000", 2024-08-17T18:26:32.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25884148299999998 2024-08-17T18:26:32.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566978+0000", 2024-08-17T18:26:32.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9751999999999999e-05 2024-08-17T18:26:32.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:32.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.567004+0000", 2024-08-17T18:26:32.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5573000000000001e-05 2024-08-17T18:26:32.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:32.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.567033+0000", 2024-08-17T18:26:32.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9153999999999999e-05 2024-08-17T18:26:32.924 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.924 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:32.924 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.924 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:117675 58.2 58:4a294109:::benchmark_data_smithi136_442124_object4059:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1726)", 2024-08-17T18:26:32.924 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:38.300334+0000", 2024-08-17T18:26:32.924 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 234.51551049400001, 2024-08-17T18:26:32.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.26687857300000001, 2024-08-17T18:26:32.925 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:32.925 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:32.925 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:32.925 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:26:32.925 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:32.925 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 117675 2024-08-17T18:26:32.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.925 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:32.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:32.925 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.300334+0000", 2024-08-17T18:26:32.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:32.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.300334+0000", 2024-08-17T18:26:32.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:32.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.300332+0000", 2024-08-17T18:26:32.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:26:32.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:32.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.300340+0000", 2024-08-17T18:26:32.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.79e-06 2024-08-17T18:26:32.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:32.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.300354+0000", 2024-08-17T18:26:32.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4029000000000001e-05 2024-08-17T18:26:32.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:32.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.300362+0000", 2024-08-17T18:26:32.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6140000000000003e-06 2024-08-17T18:26:32.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.304024+0000", 2024-08-17T18:26:32.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036621219999999999 2024-08-17T18:26:32.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:32.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.304165+0000", 2024-08-17T18:26:32.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014138599999999999 2024-08-17T18:26:32.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,4", 2024-08-17T18:26:32.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.304277+0000", 2024-08-17T18:26:32.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011175699999999999 2024-08-17T18:26:32.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:32.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.312292+0000", 2024-08-17T18:26:32.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080151070000000005 2024-08-17T18:26:32.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.567081+0000", 2024-08-17T18:26:32.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25478909399999999 2024-08-17T18:26:32.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.567128+0000", 2024-08-17T18:26:32.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6828e-05 2024-08-17T18:26:32.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:32.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.567173+0000", 2024-08-17T18:26:32.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5215e-05 2024-08-17T18:26:32.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.930 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:32.930 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.567213+0000", 2024-08-17T18:26:32.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9303e-05 2024-08-17T18:26:32.931 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.931 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:32.931 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.931 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:117682 58.2 58:489a1685:::benchmark_data_smithi136_442124_object4066:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1726)", 2024-08-17T18:26:32.931 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:38.304875+0000", 2024-08-17T18:26:32.931 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 234.51096955400001, 2024-08-17T18:26:32.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.26249927099999998, 2024-08-17T18:26:32.931 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:32.931 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:32.931 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:32.932 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:26:32.932 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:32.932 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 117682 2024-08-17T18:26:32.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.932 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:32.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:32.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.304875+0000", 2024-08-17T18:26:32.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:32.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.304875+0000", 2024-08-17T18:26:32.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:32.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.304873+0000", 2024-08-17T18:26:32.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:26:32.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:32.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.304880+0000", 2024-08-17T18:26:32.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5579999999999997e-06 2024-08-17T18:26:32.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:32.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.304893+0000", 2024-08-17T18:26:32.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2835e-05 2024-08-17T18:26:32.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:32.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.304910+0000", 2024-08-17T18:26:32.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7674000000000001e-05 2024-08-17T18:26:32.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.934 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.308127+0000", 2024-08-17T18:26:32.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032166270000000001 2024-08-17T18:26:32.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:32.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.308250+0000", 2024-08-17T18:26:32.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012278300000000001 2024-08-17T18:26:32.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,4", 2024-08-17T18:26:32.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.308361+0000", 2024-08-17T18:26:32.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000111047 2024-08-17T18:26:32.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:32.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.312516+0000", 2024-08-17T18:26:32.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041553759999999997 2024-08-17T18:26:32.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.567262+0000", 2024-08-17T18:26:32.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25474614499999998 2024-08-17T18:26:32.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:32.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.567300+0000", 2024-08-17T18:26:32.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7829e-05 2024-08-17T18:26:32.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:32.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.567342+0000", 2024-08-17T18:26:32.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1551000000000002e-05 2024-08-17T18:26:32.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:32.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.567374+0000", 2024-08-17T18:26:32.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2487000000000003e-05 2024-08-17T18:26:32.937 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.937 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:32.937 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.938 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.36936.0:117666 58.2 e1726/1701 ondisk, result = 0)", 2024-08-17T18:26:32.938 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:38.316274+0000", 2024-08-17T18:26:32.938 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 234.49957095799999, 2024-08-17T18:26:32.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25047019199999998, 2024-08-17T18:26:32.938 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:32.938 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:26:32.938 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:32.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:32.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316274+0000", 2024-08-17T18:26:32.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:32.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316274+0000", 2024-08-17T18:26:32.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:32.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316273+0000", 2024-08-17T18:26:32.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:26:32.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:32.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316279+0000", 2024-08-17T18:26:32.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9320000000000001e-06 2024-08-17T18:26:32.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:32.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316288+0000", 2024-08-17T18:26:32.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0250000000000008e-06 2024-08-17T18:26:32.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:32.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316294+0000", 2024-08-17T18:26:32.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3380000000000001e-06 2024-08-17T18:26:32.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316338+0000", 2024-08-17T18:26:32.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3930999999999997e-05 2024-08-17T18:26:32.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:32.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566627+0000", 2024-08-17T18:26:32.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25028892600000002 2024-08-17T18:26:32.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:32.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566744+0000", 2024-08-17T18:26:32.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011685500000000001 2024-08-17T18:26:32.941 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.941 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:32.941 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.942 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.36936.0:117670 58.2 e1726/1701 ondisk, result = 0)", 2024-08-17T18:26:32.942 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:38.316304+0000", 2024-08-17T18:26:32.942 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 234.49954040200001, 2024-08-17T18:26:32.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25048983800000002, 2024-08-17T18:26:32.942 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:32.942 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:26:32.942 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:32.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:32.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316304+0000", 2024-08-17T18:26:32.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:32.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316304+0000", 2024-08-17T18:26:32.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:32.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316304+0000", 2024-08-17T18:26:32.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:26:32.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:32.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316306+0000", 2024-08-17T18:26:32.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3250000000000002e-06 2024-08-17T18:26:32.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:32.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316311+0000", 2024-08-17T18:26:32.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2259999999999996e-06 2024-08-17T18:26:32.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:32.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316315+0000", 2024-08-17T18:26:32.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2310000000000001e-06 2024-08-17T18:26:32.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566774+0000", 2024-08-17T18:26:32.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25045969899999998 2024-08-17T18:26:32.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:32.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566783+0000", 2024-08-17T18:26:32.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5339999999999999e-06 2024-08-17T18:26:32.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:32.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566794+0000", 2024-08-17T18:26:32.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1085e-05 2024-08-17T18:26:32.945 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.945 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:32.945 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.946 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.36936.0:117672 58.2 e1726/1701 ondisk, result = 0)", 2024-08-17T18:26:32.946 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:38.316343+0000", 2024-08-17T18:26:32.946 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 234.49950186699999, 2024-08-17T18:26:32.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25060864999999999, 2024-08-17T18:26:32.946 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:32.946 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:26:32.946 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:32.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:32.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316343+0000", 2024-08-17T18:26:32.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:32.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316343+0000", 2024-08-17T18:26:32.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:32.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316342+0000", 2024-08-17T18:26:32.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:26:32.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:32.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316345+0000", 2024-08-17T18:26:32.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.988e-06 2024-08-17T18:26:32.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:32.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566609+0000", 2024-08-17T18:26:32.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.25026386699999997 2024-08-17T18:26:32.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:32.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566622+0000", 2024-08-17T18:26:32.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2916e-05 2024-08-17T18:26:32.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566937+0000", 2024-08-17T18:26:32.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000315013 2024-08-17T18:26:32.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:32.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566943+0000", 2024-08-17T18:26:32.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8660000000000002e-06 2024-08-17T18:26:32.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:32.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566951+0000", 2024-08-17T18:26:32.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2989999999999997e-06 2024-08-17T18:26:32.950 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.950 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:32.950 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.950 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop_reply(client.36936.0:117670 58.2 e1726/1701 ondisk, result = 0)", 2024-08-17T18:26:32.950 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:38.316955+0000", 2024-08-17T18:26:32.950 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 234.49888970500001, 2024-08-17T18:26:32.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.249955276, 2024-08-17T18:26:32.950 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:32.950 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:26:32.950 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:32.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:32.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316955+0000", 2024-08-17T18:26:32.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:32.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316955+0000", 2024-08-17T18:26:32.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:32.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316954+0000", 2024-08-17T18:26:32.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:26:32.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:32.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.316959+0000", 2024-08-17T18:26:32.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.921e-06 2024-08-17T18:26:32.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:32.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566597+0000", 2024-08-17T18:26:32.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.249638255 2024-08-17T18:26:32.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:32.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566612+0000", 2024-08-17T18:26:32.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.454e-05 2024-08-17T18:26:32.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566823+0000", 2024-08-17T18:26:32.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021107400000000001 2024-08-17T18:26:32.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:32.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566830+0000", 2024-08-17T18:26:32.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3320000000000003e-06 2024-08-17T18:26:32.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:32.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:38.566910+0000", 2024-08-17T18:26:32.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9765999999999995e-05 2024-08-17T18:26:32.954 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.954 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:32.954 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.954 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.25175.0:182 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e1751)", 2024-08-17T18:26:32.954 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:26:06.807335+0000", 2024-08-17T18:26:32.954 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 26.008509604, 2024-08-17T18:26:32.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048281300000000001, 2024-08-17T18:26:32.955 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:32.955 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:26:32.955 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:32.955 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.25175", 2024-08-17T18:26:32.955 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3845461328", 2024-08-17T18:26:32.955 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 182 2024-08-17T18:26:32.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.955 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:32.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:32.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:26:06.807335+0000", 2024-08-17T18:26:32.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:32.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:26:06.807335+0000", 2024-08-17T18:26:32.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:32.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:32.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:26:06.807332+0000", 2024-08-17T18:26:32.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999976 2024-08-17T18:26:32.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:32.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:26:06.807344+0000", 2024-08-17T18:26:32.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2033e-05 2024-08-17T18:26:32.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:32.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:26:06.807370+0000", 2024-08-17T18:26:32.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5536000000000001e-05 2024-08-17T18:26:32.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:32.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:26:06.807380+0000", 2024-08-17T18:26:32.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6800000000000005e-06 2024-08-17T18:26:32.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:32.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:26:06.807665+0000", 2024-08-17T18:26:32.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028566900000000001 2024-08-17T18:26:32.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:32.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:32.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:32.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:26:06.807818+0000", 2024-08-17T18:26:32.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015231399999999999 2024-08-17T18:26:32.958 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.958 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:32.958 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.958 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:32.958 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:32.958 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:26:33.679 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-08-17T18:26:33.835 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-17T18:26:33.923 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:33 smithi136 ceph-mon[109861]: pgmap v3549: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:33.923 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:33 smithi136 ceph-mon[115078]: pgmap v3549: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:34.046 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:26:34.168 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:33 smithi165 ceph-mon[105517]: pgmap v3549: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:34.201 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:26:34.860 INFO:teuthology.orchestra.run.smithi136.stderr:nodeep-scrub is unset 2024-08-17T18:26:35.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:34 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3583764170' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:26:35.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:34 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:26:35.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:34 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3583764170' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:26:35.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:34 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:26:35.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:34 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3583764170' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:26:35.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:34 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T18:26:35.770 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:26:35.770 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:26:35.770 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-08-17T18:26:35.770 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:26:35.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:35 smithi136 ceph-mon[109861]: pgmap v3550: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:35.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:35 smithi136 ceph-mon[109861]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:26:35.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:35 smithi136 ceph-mon[109861]: Cluster is now healthy 2024-08-17T18:26:35.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:35 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:26:35.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:35 smithi136 ceph-mon[109861]: osdmap e1778: 8 total, 8 up, 8 in 2024-08-17T18:26:35.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:35 smithi136 ceph-mon[115078]: pgmap v3550: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:35.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:35 smithi136 ceph-mon[115078]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:26:35.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:35 smithi136 ceph-mon[115078]: Cluster is now healthy 2024-08-17T18:26:35.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:35 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:26:35.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:35 smithi136 ceph-mon[115078]: osdmap e1778: 8 total, 8 up, 8 in 2024-08-17T18:26:36.073 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:35 smithi165 ceph-mon[105517]: pgmap v3550: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:36.073 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:35 smithi165 ceph-mon[105517]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T18:26:36.074 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:35 smithi165 ceph-mon[105517]: Cluster is now healthy 2024-08-17T18:26:36.074 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:35 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T18:26:36.074 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:35 smithi165 ceph-mon[105517]: osdmap e1778: 8 total, 8 up, 8 in 2024-08-17T18:26:36.379 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-17T18:26:36.586 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:26:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:26:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:26:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:37 smithi165 ceph-mon[105517]: pgmap v3552: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:37.394 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-08-17T18:26:37.602 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:26:37.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:26:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:26:37] "GET /metrics HTTP/1.1" 200 34256 "" "Prometheus/2.51.0" 2024-08-17T18:26:37.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:26:37.696 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:26:37.697 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:37 smithi136 ceph-mon[115078]: pgmap v3552: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:37.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:26:37.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:26:37.697 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:37 smithi136 ceph-mon[109861]: pgmap v3552: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:38.285 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:26:38.285 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:26:38.285 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-08-17T18:26:38.285 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-08-17T18:26:38.286 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:26:38.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-17T18:26:39.125 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:26:39.874 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:39 smithi136 ceph-mon[109861]: pgmap v3553: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:39.874 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:39 smithi136 ceph-mon[115078]: pgmap v3553: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:39.896 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-08-17T18:26:40.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:39 smithi165 ceph-mon[105517]: pgmap v3553: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:40.100 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:26:40.938 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:26:40.938 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T18:26:40.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T18:26:40.938 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:26:40.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.938 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:141933 57.8 e1650/1637 57:11a7cd89:::benchmark_data_smithi136_389235_object26011:head v 1650'6938, mlcod=1612'390)", 2024-08-17T18:26:40.938 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:03.208093+0000", 2024-08-17T18:26:40.938 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 577.727836131, 2024-08-17T18:26:40.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.42764199200000003, 2024-08-17T18:26:40.939 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:40.939 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:40.939 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:40.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:40.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.208093+0000", 2024-08-17T18:26:40.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:40.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:40.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.208093+0000", 2024-08-17T18:26:40.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:40.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:40.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.208092+0000", 2024-08-17T18:26:40.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:26:40.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:40.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.208099+0000", 2024-08-17T18:26:40.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8600000000000004e-06 2024-08-17T18:26:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.208111+0000", 2024-08-17T18:26:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1440000000000001e-05 2024-08-17T18:26:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.208118+0000", 2024-08-17T18:26:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9589999999999998e-06 2024-08-17T18:26:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.208208+0000", 2024-08-17T18:26:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0399e-05 2024-08-17T18:26:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.208232+0000", 2024-08-17T18:26:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3886000000000001e-05 2024-08-17T18:26:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.635686+0000", 2024-08-17T18:26:40.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.42745419899999998 2024-08-17T18:26:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.635735+0000", 2024-08-17T18:26:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9035000000000001e-05 2024-08-17T18:26:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:141934 57.f e1650/1637 57:fc154abb:::benchmark_data_smithi136_389235_object26012:head v 1650'9020, mlcod=1650'9018)", 2024-08-17T18:26:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:03.209585+0000", 2024-08-17T18:26:40.943 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 577.72634392099997, 2024-08-17T18:26:40.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.42294235000000002, 2024-08-17T18:26:40.944 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:40.944 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:40.944 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:40.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:40.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.209585+0000", 2024-08-17T18:26:40.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:40.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:40.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.209585+0000", 2024-08-17T18:26:40.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.209584+0000", 2024-08-17T18:26:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:26:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.209592+0000", 2024-08-17T18:26:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9389999999999997e-06 2024-08-17T18:26:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:40.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.209605+0000", 2024-08-17T18:26:40.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2255999999999999e-05 2024-08-17T18:26:40.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:40.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.209611+0000", 2024-08-17T18:26:40.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7340000000000002e-06 2024-08-17T18:26:40.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:40.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.209656+0000", 2024-08-17T18:26:40.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4384000000000001e-05 2024-08-17T18:26:40.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:40.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.209677+0000", 2024-08-17T18:26:40.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1656000000000001e-05 2024-08-17T18:26:40.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:40.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.632480+0000", 2024-08-17T18:26:40.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.422802601 2024-08-17T18:26:40.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:40.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.632528+0000", 2024-08-17T18:26:40.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7911000000000003e-05 2024-08-17T18:26:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:141937 57.a e1650/1638 57:5a772745:::benchmark_data_smithi136_389235_object26015:head v 1650'9015, mlcod=1650'9012)", 2024-08-17T18:26:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:03.219452+0000", 2024-08-17T18:26:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 577.71647780700005, 2024-08-17T18:26:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.42668703299999999, 2024-08-17T18:26:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:40.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:40.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.219452+0000", 2024-08-17T18:26:40.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:40.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:40.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.219452+0000", 2024-08-17T18:26:40.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:40.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:40.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.219451+0000", 2024-08-17T18:26:40.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:26:40.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.219459+0000", 2024-08-17T18:26:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4700000000000002e-06 2024-08-17T18:26:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.632648+0000", 2024-08-17T18:26:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.41318864900000002 2024-08-17T18:26:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:40.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.632656+0000", 2024-08-17T18:26:40.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7420000000000003e-06 2024-08-17T18:26:40.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:40.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.632692+0000", 2024-08-17T18:26:40.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5278999999999999e-05 2024-08-17T18:26:40.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:40.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.633241+0000", 2024-08-17T18:26:40.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00054898299999999996 2024-08-17T18:26:40.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:40.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.646102+0000", 2024-08-17T18:26:40.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012860955 2024-08-17T18:26:40.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:40.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.646139+0000", 2024-08-17T18:26:40.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7045999999999997e-05 2024-08-17T18:26:40.952 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:40.952 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:40.952 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:40.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.953 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(57.12 1650/1637 [PushOp(57:4ae6f4a6:::benchmark_data_smithi136_389235_object25728:head, version: 1613'1580, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(57:4ae6f4a6:::benchmark_data_smithi136_389235_object25728:head@1613'1580, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-17T18:26:40.953 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:17:03.278350+0000", 2024-08-17T18:26:40.953 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 577.65757955499998, 2024-08-17T18:26:40.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.35665043000000002, 2024-08-17T18:26:40.953 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:40.953 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:26:40.953 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:40.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:40.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.278350+0000", 2024-08-17T18:26:40.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:40.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:40.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.278350+0000", 2024-08-17T18:26:40.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:40.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:40.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.278348+0000", 2024-08-17T18:26:40.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:26:40.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:40.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.278384+0000", 2024-08-17T18:26:40.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5187e-05 2024-08-17T18:26:40.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:40.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.632552+0000", 2024-08-17T18:26:40.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.35416881300000003 2024-08-17T18:26:40.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:40.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.632573+0000", 2024-08-17T18:26:40.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0684999999999999e-05 2024-08-17T18:26:40.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:40.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.632644+0000", 2024-08-17T18:26:40.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1192000000000005e-05 2024-08-17T18:26:40.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:40.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.632653+0000", 2024-08-17T18:26:40.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9700000000000005e-06 2024-08-17T18:26:40.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:40.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:17:03.635000+0000", 2024-08-17T18:26:40.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002347039 2024-08-17T18:26:40.956 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:40.956 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:40.957 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:40.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.957 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:179448 57.17 e1672/1653 57:ea2f25a4:::benchmark_data_smithi136_389235_object63526:head v 1672'7871, mlcod=1614'3902)", 2024-08-17T18:26:40.957 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:45.746159+0000", 2024-08-17T18:26:40.957 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 475.189770227, 2024-08-17T18:26:40.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.047589698999999999, 2024-08-17T18:26:40.957 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:40.957 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:40.957 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:40.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:40.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.746159+0000", 2024-08-17T18:26:40.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:40.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:40.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.746159+0000", 2024-08-17T18:26:40.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:40.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:40.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.746158+0000", 2024-08-17T18:26:40.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:26:40.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:40.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.746165+0000", 2024-08-17T18:26:40.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.742e-06 2024-08-17T18:26:40.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:40.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.746176+0000", 2024-08-17T18:26:40.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0896999999999999e-05 2024-08-17T18:26:40.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:40.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.746182+0000", 2024-08-17T18:26:40.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9800000000000003e-06 2024-08-17T18:26:40.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:40.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.746245+0000", 2024-08-17T18:26:40.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3003000000000006e-05 2024-08-17T18:26:40.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:40.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.746267+0000", 2024-08-17T18:26:40.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1880000000000001e-05 2024-08-17T18:26:40.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:40.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.793718+0000", 2024-08-17T18:26:40.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.047451013 2024-08-17T18:26:40.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:40.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.793749+0000", 2024-08-17T18:26:40.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0956e-05 2024-08-17T18:26:40.961 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:40.961 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:40.961 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:40.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.961 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:179233 57.12 e1672/1667 57:4f78a0b4:::benchmark_data_smithi136_389235_object63311:head v 1672'7995, mlcod=1613'3906)", 2024-08-17T18:26:40.962 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:18:45.754516+0000", 2024-08-17T18:26:40.962 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 475.181413238, 2024-08-17T18:26:40.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.047304248, 2024-08-17T18:26:40.962 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:40.962 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:40.962 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:40.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:40.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.754516+0000", 2024-08-17T18:26:40.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:40.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:40.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.754516+0000", 2024-08-17T18:26:40.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:40.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:40.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.754515+0000", 2024-08-17T18:26:40.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:26:40.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:40.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.754523+0000", 2024-08-17T18:26:40.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5820000000000002e-06 2024-08-17T18:26:40.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:40.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.789709+0000", 2024-08-17T18:26:40.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.035185813000000003 2024-08-17T18:26:40.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:40.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.789723+0000", 2024-08-17T18:26:40.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3385e-05 2024-08-17T18:26:40.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:40.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.789829+0000", 2024-08-17T18:26:40.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010666399999999999 2024-08-17T18:26:40.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:40.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.789860+0000", 2024-08-17T18:26:40.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0341999999999999e-05 2024-08-17T18:26:40.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:40.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.801782+0000", 2024-08-17T18:26:40.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011922759 2024-08-17T18:26:40.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:40.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:18:45.801820+0000", 2024-08-17T18:26:40.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7984000000000001e-05 2024-08-17T18:26:40.966 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:40.966 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:40.966 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:40.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.966 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:187065 57.8 57:124620dd:::benchmark_data_smithi136_389235_object71143:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1676)", 2024-08-17T18:26:40.966 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:19:03.662550+0000", 2024-08-17T18:26:40.966 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 457.27337959099998, 2024-08-17T18:26:40.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.01595659, 2024-08-17T18:26:40.966 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:40.967 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:40.968 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:40.968 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:40.968 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:40.968 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 187065 2024-08-17T18:26:40.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.968 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:40.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:40.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:03.662550+0000", 2024-08-17T18:26:40.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:40.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:40.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:03.662550+0000", 2024-08-17T18:26:40.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:40.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:40.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:03.662548+0000", 2024-08-17T18:26:40.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:26:40.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:40.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:03.662556+0000", 2024-08-17T18:26:40.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.452e-06 2024-08-17T18:26:40.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:40.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:03.662567+0000", 2024-08-17T18:26:40.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1567e-05 2024-08-17T18:26:40.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:40.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:03.662574+0000", 2024-08-17T18:26:40.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2429999999999999e-06 2024-08-17T18:26:40.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:40.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:03.663807+0000", 2024-08-17T18:26:40.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001232293 2024-08-17T18:26:40.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:26:40.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:03.663819+0000", 2024-08-17T18:26:40.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1973e-05 2024-08-17T18:26:40.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:40.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.668353+0000", 2024-08-17T18:26:40.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.004534576 2024-08-17T18:26:40.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:40.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.668487+0000", 2024-08-17T18:26:40.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013356800000000001 2024-08-17T18:26:40.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,6", 2024-08-17T18:26:40.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.668599+0000", 2024-08-17T18:26:40.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000111995 2024-08-17T18:26:40.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:40.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.670655+0000", 2024-08-17T18:26:40.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020564670000000002 2024-08-17T18:26:40.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:40.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.678308+0000", 2024-08-17T18:26:40.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0076523759999999998 2024-08-17T18:26:40.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:40.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.678361+0000", 2024-08-17T18:26:40.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3672999999999999e-05 2024-08-17T18:26:40.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:40.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.678467+0000", 2024-08-17T18:26:40.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010538499999999999 2024-08-17T18:26:40.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:40.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.678506+0000", 2024-08-17T18:26:40.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9659000000000001e-05 2024-08-17T18:26:40.974 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:40.974 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:40.974 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:40.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.975 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:187159 57.8 57:16cc7ce8:::benchmark_data_smithi136_389235_object71237:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1676)", 2024-08-17T18:26:40.975 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:19:03.841045+0000", 2024-08-17T18:26:40.975 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 457.09488427799999, 2024-08-17T18:26:40.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.84213724700000003, 2024-08-17T18:26:40.975 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:40.975 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:40.975 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:40.975 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:40.975 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:40.975 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 187159 2024-08-17T18:26:40.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.976 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:40.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:40.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:03.841045+0000", 2024-08-17T18:26:40.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:40.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:40.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:03.841045+0000", 2024-08-17T18:26:40.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:40.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:40.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:03.841043+0000", 2024-08-17T18:26:40.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:26:40.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:40.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:03.841051+0000", 2024-08-17T18:26:40.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3690000000000001e-06 2024-08-17T18:26:40.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:40.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:03.841080+0000", 2024-08-17T18:26:40.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8881e-05 2024-08-17T18:26:40.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:40.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:03.841088+0000", 2024-08-17T18:26:40.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0909999999999993e-06 2024-08-17T18:26:40.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:40.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:03.841142+0000", 2024-08-17T18:26:40.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4493000000000002e-05 2024-08-17T18:26:40.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:26:40.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:03.841153+0000", 2024-08-17T18:26:40.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0295000000000001e-05 2024-08-17T18:26:40.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:40.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.671100+0000", 2024-08-17T18:26:40.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.829947463 2024-08-17T18:26:40.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:40.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.673619+0000", 2024-08-17T18:26:40.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002519018 2024-08-17T18:26:40.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,6", 2024-08-17T18:26:40.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.673740+0000", 2024-08-17T18:26:40.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012133600000000001 2024-08-17T18:26:40.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:40.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.678316+0000", 2024-08-17T18:26:40.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045757319999999999 2024-08-17T18:26:40.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:40.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.683003+0000", 2024-08-17T18:26:40.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046866069999999998 2024-08-17T18:26:40.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:40.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.683040+0000", 2024-08-17T18:26:40.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6928999999999998e-05 2024-08-17T18:26:40.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:40.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.683137+0000", 2024-08-17T18:26:40.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7102000000000006e-05 2024-08-17T18:26:40.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:40.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.683182+0000", 2024-08-17T18:26:40.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5646000000000003e-05 2024-08-17T18:26:40.982 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:40.982 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:40.982 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:40.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.982 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:187290 57.8 57:1446c80f:::benchmark_data_smithi136_389235_object71368:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1676)", 2024-08-17T18:26:40.982 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:19:04.106114+0000", 2024-08-17T18:26:40.982 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 456.829814946, 2024-08-17T18:26:40.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.57726026100000005, 2024-08-17T18:26:40.983 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:40.983 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:40.983 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:40.983 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:40.983 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:40.983 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 187290 2024-08-17T18:26:40.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.983 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:40.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:40.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.106114+0000", 2024-08-17T18:26:40.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:40.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:40.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.106114+0000", 2024-08-17T18:26:40.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:40.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:40.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.106113+0000", 2024-08-17T18:26:40.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:26:40.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:40.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.106122+0000", 2024-08-17T18:26:40.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8799999999999997e-06 2024-08-17T18:26:40.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:40.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.106135+0000", 2024-08-17T18:26:40.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3335e-05 2024-08-17T18:26:40.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:40.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.106143+0000", 2024-08-17T18:26:40.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5420000000000005e-06 2024-08-17T18:26:40.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:40.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.106299+0000", 2024-08-17T18:26:40.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000155835 2024-08-17T18:26:40.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:26:40.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.106311+0000", 2024-08-17T18:26:40.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1661999999999999e-05 2024-08-17T18:26:40.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:40.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.675871+0000", 2024-08-17T18:26:40.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.56955965399999997 2024-08-17T18:26:40.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:40.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.675980+0000", 2024-08-17T18:26:40.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109681 2024-08-17T18:26:40.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,6", 2024-08-17T18:26:40.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.676095+0000", 2024-08-17T18:26:40.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000114598 2024-08-17T18:26:40.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:40.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.678321+0000", 2024-08-17T18:26:40.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022265039999999998 2024-08-17T18:26:40.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:40.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.683223+0000", 2024-08-17T18:26:40.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049018220000000001 2024-08-17T18:26:40.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:40.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.683268+0000", 2024-08-17T18:26:40.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4681000000000001e-05 2024-08-17T18:26:40.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:40.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.683339+0000", 2024-08-17T18:26:40.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1211999999999995e-05 2024-08-17T18:26:40.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:40.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.683375+0000", 2024-08-17T18:26:40.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5685000000000001e-05 2024-08-17T18:26:40.990 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:40.990 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:40.990 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:40.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.990 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:187295 57.8 57:14763113:::benchmark_data_smithi136_389235_object71373:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1676)", 2024-08-17T18:26:40.990 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:19:04.114917+0000", 2024-08-17T18:26:40.990 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 456.82101255600003, 2024-08-17T18:26:40.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.57372016400000003, 2024-08-17T18:26:40.991 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:40.991 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:40.991 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:40.991 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:40.991 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:40.991 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 187295 2024-08-17T18:26:40.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.991 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:40.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:40.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.114917+0000", 2024-08-17T18:26:40.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:40.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:40.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.114917+0000", 2024-08-17T18:26:40.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:40.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:40.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.114915+0000", 2024-08-17T18:26:40.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:26:40.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:40.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.114923+0000", 2024-08-17T18:26:40.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3900000000000004e-06 2024-08-17T18:26:40.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:40.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.114939+0000", 2024-08-17T18:26:40.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6197000000000001e-05 2024-08-17T18:26:40.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:40.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.114946+0000", 2024-08-17T18:26:40.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9319999999999999e-06 2024-08-17T18:26:40.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:40.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.116093+0000", 2024-08-17T18:26:40.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001147314 2024-08-17T18:26:40.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:26:40.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.116106+0000", 2024-08-17T18:26:40.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3243e-05 2024-08-17T18:26:40.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:40.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.678536+0000", 2024-08-17T18:26:40.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.56242944500000003 2024-08-17T18:26:40.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:40.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.678648+0000", 2024-08-17T18:26:40.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011281 2024-08-17T18:26:40.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,6", 2024-08-17T18:26:40.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.678756+0000", 2024-08-17T18:26:40.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000107759 2024-08-17T18:26:40.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:40.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.683010+0000", 2024-08-17T18:26:40.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004253621 2024-08-17T18:26:40.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:40.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.683420+0000", 2024-08-17T18:26:40.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00040977899999999998 2024-08-17T18:26:40.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:40.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.688505+0000", 2024-08-17T18:26:40.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050849529999999997 2024-08-17T18:26:40.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:40.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.688596+0000", 2024-08-17T18:26:40.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1644000000000006e-05 2024-08-17T18:26:40.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:40.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.688637+0000", 2024-08-17T18:26:40.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0793000000000001e-05 2024-08-17T18:26:40.997 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:40.998 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:40.998 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:40.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.998 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:187412 57.8 57:14e9a78f:::benchmark_data_smithi136_389235_object71490:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1676)", 2024-08-17T18:26:40.998 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:19:04.531904+0000", 2024-08-17T18:26:40.998 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 456.40402501300002, 2024-08-17T18:26:40.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.163455505, 2024-08-17T18:26:40.998 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:40.998 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:40.998 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:40.998 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:40.998 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:40.999 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 187412 2024-08-17T18:26:40.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.999 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:40.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:40.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.531904+0000", 2024-08-17T18:26:40.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:40.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:40.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:40.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:40.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.531904+0000", 2024-08-17T18:26:40.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:40.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:41.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.531903+0000", 2024-08-17T18:26:41.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:26:41.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:41.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.531910+0000", 2024-08-17T18:26:41.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4279999999999999e-06 2024-08-17T18:26:41.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:41.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.531927+0000", 2024-08-17T18:26:41.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7343999999999999e-05 2024-08-17T18:26:41.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:41.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.531935+0000", 2024-08-17T18:26:41.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8350000000000004e-06 2024-08-17T18:26:41.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:41.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.531984+0000", 2024-08-17T18:26:41.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8770999999999998e-05 2024-08-17T18:26:41.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:26:41.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.531996+0000", 2024-08-17T18:26:41.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1583000000000001e-05 2024-08-17T18:26:41.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:41.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.680830+0000", 2024-08-17T18:26:41.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.14883479399999999 2024-08-17T18:26:41.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:41.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.680942+0000", 2024-08-17T18:26:41.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011186199999999999 2024-08-17T18:26:41.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,6", 2024-08-17T18:26:41.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.681053+0000", 2024-08-17T18:26:41.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000111013 2024-08-17T18:26:41.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:41.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.688880+0000", 2024-08-17T18:26:41.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0078267370000000003 2024-08-17T18:26:41.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:41.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.693219+0000", 2024-08-17T18:26:41.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043388350000000001 2024-08-17T18:26:41.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:41.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.695229+0000", 2024-08-17T18:26:41.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020106080000000001 2024-08-17T18:26:41.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:41.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.695318+0000", 2024-08-17T18:26:41.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8942000000000003e-05 2024-08-17T18:26:41.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:41.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.695360+0000", 2024-08-17T18:26:41.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1461999999999997e-05 2024-08-17T18:26:41.005 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:41.005 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:41.005 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:41.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.005 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:187418 57.8 57:11d50c8b:::benchmark_data_smithi136_389235_object71496:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1676)", 2024-08-17T18:26:41.006 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:19:04.546818+0000", 2024-08-17T18:26:41.006 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 456.389111329, 2024-08-17T18:26:41.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.148677054, 2024-08-17T18:26:41.006 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:41.006 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:41.006 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:41.006 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:41.006 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:41.006 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 187418 2024-08-17T18:26:41.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.006 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:41.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:41.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.546818+0000", 2024-08-17T18:26:41.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:41.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:41.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.546818+0000", 2024-08-17T18:26:41.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:41.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:41.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.546816+0000", 2024-08-17T18:26:41.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:26:41.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:41.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.546826+0000", 2024-08-17T18:26:41.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4870000000000008e-06 2024-08-17T18:26:41.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:41.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.546843+0000", 2024-08-17T18:26:41.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7872000000000001e-05 2024-08-17T18:26:41.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:41.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.546852+0000", 2024-08-17T18:26:41.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9090000000000006e-06 2024-08-17T18:26:41.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:41.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.546919+0000", 2024-08-17T18:26:41.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6839000000000002e-05 2024-08-17T18:26:41.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:26:41.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.546934+0000", 2024-08-17T18:26:41.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5213e-05 2024-08-17T18:26:41.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:41.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.683439+0000", 2024-08-17T18:26:41.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13650498899999999 2024-08-17T18:26:41.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:41.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.683543+0000", 2024-08-17T18:26:41.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103341 2024-08-17T18:26:41.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,6", 2024-08-17T18:26:41.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.683639+0000", 2024-08-17T18:26:41.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.603e-05 2024-08-17T18:26:41.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:41.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.688885+0000", 2024-08-17T18:26:41.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052465180000000004 2024-08-17T18:26:41.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:41.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.693249+0000", 2024-08-17T18:26:41.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043638649999999998 2024-08-17T18:26:41.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:41.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.695392+0000", 2024-08-17T18:26:41.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021431430000000001 2024-08-17T18:26:41.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:41.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.695460+0000", 2024-08-17T18:26:41.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7527000000000006e-05 2024-08-17T18:26:41.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:41.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.695495+0000", 2024-08-17T18:26:41.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5369e-05 2024-08-17T18:26:41.013 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:41.013 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:41.013 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:41.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.013 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.46277.0:187420 57.8 57:1675e106:::benchmark_data_smithi136_389235_object71498:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1676)", 2024-08-17T18:26:41.013 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:19:04.552860+0000", 2024-08-17T18:26:41.013 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 456.38306947000001, 2024-08-17T18:26:41.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.144910649, 2024-08-17T18:26:41.013 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:41.013 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:41.013 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:26:41.014 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.46277", 2024-08-17T18:26:41.014 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/2013118579", 2024-08-17T18:26:41.014 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 187420 2024-08-17T18:26:41.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.014 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:41.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:41.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.552860+0000", 2024-08-17T18:26:41.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:41.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:41.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.552860+0000", 2024-08-17T18:26:41.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:41.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:41.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.552858+0000", 2024-08-17T18:26:41.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:26:41.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:41.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.552866+0000", 2024-08-17T18:26:41.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8509999999999999e-06 2024-08-17T18:26:41.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:41.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.552878+0000", 2024-08-17T18:26:41.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2318e-05 2024-08-17T18:26:41.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:41.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.552885+0000", 2024-08-17T18:26:41.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5300000000000002e-06 2024-08-17T18:26:41.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:41.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.552939+0000", 2024-08-17T18:26:41.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4188999999999999e-05 2024-08-17T18:26:41.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:26:41.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.552950+0000", 2024-08-17T18:26:41.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0900000000000001e-05 2024-08-17T18:26:41.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:41.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.686178+0000", 2024-08-17T18:26:41.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13322814299999999 2024-08-17T18:26:41.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:41.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.686286+0000", 2024-08-17T18:26:41.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010812899999999999 2024-08-17T18:26:41.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,6", 2024-08-17T18:26:41.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.686389+0000", 2024-08-17T18:26:41.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010264000000000001 2024-08-17T18:26:41.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:41.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.695529+0000", 2024-08-17T18:26:41.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0091400769999999999 2024-08-17T18:26:41.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:26:41.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.697345+0000", 2024-08-17T18:26:41.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018161169999999999 2024-08-17T18:26:41.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:41.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.697646+0000", 2024-08-17T18:26:41.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00030062300000000003 2024-08-17T18:26:41.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:41.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.697730+0000", 2024-08-17T18:26:41.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3918999999999994e-05 2024-08-17T18:26:41.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:41.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:19:04.697771+0000", 2024-08-17T18:26:41.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0942999999999997e-05 2024-08-17T18:26:41.020 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:41.020 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:41.020 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:41.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.020 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.46277.0:221269 57.6 e1686/1637 57:6534dd10:::benchmark_data_smithi136_389235_object105347:head v 1686'9241, mlcod=1612'390)", 2024-08-17T18:26:41.020 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:20:01.201944+0000", 2024-08-17T18:26:41.020 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 399.73398534199998, 2024-08-17T18:26:41.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.048228082999999998, 2024-08-17T18:26:41.020 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:41.020 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:41.020 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:41.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:41.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:20:01.201944+0000", 2024-08-17T18:26:41.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:41.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:41.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:20:01.201944+0000", 2024-08-17T18:26:41.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:41.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:41.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:20:01.201943+0000", 2024-08-17T18:26:41.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:26:41.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:41.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:20:01.201949+0000", 2024-08-17T18:26:41.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7769999999999999e-06 2024-08-17T18:26:41.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:41.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:20:01.201958+0000", 2024-08-17T18:26:41.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0739999999999999e-06 2024-08-17T18:26:41.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:41.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:20:01.201965+0000", 2024-08-17T18:26:41.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4359999999999999e-06 2024-08-17T18:26:41.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:41.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:20:01.220294+0000", 2024-08-17T18:26:41.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018328807999999999 2024-08-17T18:26:41.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:41.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:20:01.220347+0000", 2024-08-17T18:26:41.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3427999999999997e-05 2024-08-17T18:26:41.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:41.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:20:01.250084+0000", 2024-08-17T18:26:41.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.029736571 2024-08-17T18:26:41.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:41.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:20:01.250172+0000", 2024-08-17T18:26:41.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8628000000000006e-05 2024-08-17T18:26:41.024 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:41.024 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:41.024 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:41.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.024 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "pg_backfill(finish 57.f e 1696/1696 lb MAX)", 2024-08-17T18:26:41.025 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:20:32.010472+0000", 2024-08-17T18:26:41.025 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 368.92545761700001, 2024-08-17T18:26:41.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.058779949999999997, 2024-08-17T18:26:41.025 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:41.025 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:26:41.025 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:41.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:41.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:20:32.010472+0000", 2024-08-17T18:26:41.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:41.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:41.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:20:32.010472+0000", 2024-08-17T18:26:41.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:41.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:41.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:20:32.010471+0000", 2024-08-17T18:26:41.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:26:41.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:41.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:20:32.010484+0000", 2024-08-17T18:26:41.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3199e-05 2024-08-17T18:26:41.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:41.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:20:32.010497+0000", 2024-08-17T18:26:41.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3137e-05 2024-08-17T18:26:41.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:41.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:20:32.010505+0000", 2024-08-17T18:26:41.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.96e-06 2024-08-17T18:26:41.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:41.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:20:32.010544+0000", 2024-08-17T18:26:41.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8383000000000002e-05 2024-08-17T18:26:41.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:41.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:20:32.010560+0000", 2024-08-17T18:26:41.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6350000000000001e-05 2024-08-17T18:26:41.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:41.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:20:32.069252+0000", 2024-08-17T18:26:41.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.058691800000000002 2024-08-17T18:26:41.028 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:41.028 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:41.028 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:41.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.029 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:66378 58.0 e1711/1701 58:0e70f639:::benchmark_data_smithi136_442124_object66377:head v 1711'4161, mlcod=1710'4159)", 2024-08-17T18:26:41.029 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:21:37.755960+0000", 2024-08-17T18:26:41.029 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 303.17996914600002, 2024-08-17T18:26:41.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.064346350999999996, 2024-08-17T18:26:41.029 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:41.029 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:41.029 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:41.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:41.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:21:37.755960+0000", 2024-08-17T18:26:41.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:41.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:41.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:21:37.755960+0000", 2024-08-17T18:26:41.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:41.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:41.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:21:37.755960+0000", 2024-08-17T18:26:41.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:26:41.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:41.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:21:37.755982+0000", 2024-08-17T18:26:41.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2356000000000001e-05 2024-08-17T18:26:41.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:41.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:21:37.815971+0000", 2024-08-17T18:26:41.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.059988644000000001 2024-08-17T18:26:41.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:41.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:21:37.815980+0000", 2024-08-17T18:26:41.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6369999999999994e-06 2024-08-17T18:26:41.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:41.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:21:37.816325+0000", 2024-08-17T18:26:41.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034467699999999999 2024-08-17T18:26:41.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:41.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:21:37.816344+0000", 2024-08-17T18:26:41.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9505999999999999e-05 2024-08-17T18:26:41.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:41.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:21:37.820270+0000", 2024-08-17T18:26:41.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003925212 2024-08-17T18:26:41.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:41.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:21:37.820307+0000", 2024-08-17T18:26:41.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7018999999999999e-05 2024-08-17T18:26:41.033 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:41.033 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:41.033 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:41.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.033 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:66382 58.b e1711/1701 58:d6b06911:::benchmark_data_smithi136_442124_object66381:head v 1711'4110, mlcod=1710'4108)", 2024-08-17T18:26:41.033 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:21:37.769557+0000", 2024-08-17T18:26:41.033 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 303.16637244499998, 2024-08-17T18:26:41.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.050932877000000001, 2024-08-17T18:26:41.033 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:41.034 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:41.034 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:41.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:41.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:21:37.769557+0000", 2024-08-17T18:26:41.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:41.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:41.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:21:37.769557+0000", 2024-08-17T18:26:41.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:41.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:41.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:21:37.769557+0000", 2024-08-17T18:26:41.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:26:41.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:41.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:21:37.769590+0000", 2024-08-17T18:26:41.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3436999999999999e-05 2024-08-17T18:26:41.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:41.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:21:37.815952+0000", 2024-08-17T18:26:41.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.046361549000000002 2024-08-17T18:26:41.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:41.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:21:37.815965+0000", 2024-08-17T18:26:41.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.343e-05 2024-08-17T18:26:41.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:41.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:21:37.816496+0000", 2024-08-17T18:26:41.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00053103600000000003 2024-08-17T18:26:41.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:41.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:21:37.816515+0000", 2024-08-17T18:26:41.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8695999999999999e-05 2024-08-17T18:26:41.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:41.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:21:37.820461+0000", 2024-08-17T18:26:41.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00394598 2024-08-17T18:26:41.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:41.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:21:37.820490+0000", 2024-08-17T18:26:41.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9127000000000001e-05 2024-08-17T18:26:41.038 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:41.038 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:41.038 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:41.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.038 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:112805 58.a e1721/1712 58:5ba22203:::benchmark_data_smithi136_442124_object112804:head v 1721'7093, mlcod=1721'7091)", 2024-08-17T18:26:41.038 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:29.126674+0000", 2024-08-17T18:26:41.038 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 251.809255049, 2024-08-17T18:26:41.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10944738699999999, 2024-08-17T18:26:41.038 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:41.039 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:41.039 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:41.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:41.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.126674+0000", 2024-08-17T18:26:41.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:41.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:41.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.126674+0000", 2024-08-17T18:26:41.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:41.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:41.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.126673+0000", 2024-08-17T18:26:41.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:26:41.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:41.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.126714+0000", 2024-08-17T18:26:41.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0930999999999998e-05 2024-08-17T18:26:41.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:41.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.126729+0000", 2024-08-17T18:26:41.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4592e-05 2024-08-17T18:26:41.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:41.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.126736+0000", 2024-08-17T18:26:41.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7319999999999998e-06 2024-08-17T18:26:41.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:41.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.126776+0000", 2024-08-17T18:26:41.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9758000000000001e-05 2024-08-17T18:26:41.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:41.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.126793+0000", 2024-08-17T18:26:41.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6558999999999998e-05 2024-08-17T18:26:41.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:41.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.131751+0000", 2024-08-17T18:26:41.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049584410000000001 2024-08-17T18:26:41.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:41.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.236122+0000", 2024-08-17T18:26:41.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10437054799999999 2024-08-17T18:26:41.043 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:41.043 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:41.043 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:41.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.043 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:112807 58.a e1721/1712 58:5e37b9be:::benchmark_data_smithi136_442124_object112806:head v 1721'7094, mlcod=1721'7091)", 2024-08-17T18:26:41.043 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:29.129433+0000", 2024-08-17T18:26:41.043 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 251.806496286, 2024-08-17T18:26:41.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.106730981, 2024-08-17T18:26:41.044 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:41.044 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:41.044 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:41.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:41.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.129433+0000", 2024-08-17T18:26:41.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:41.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:41.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.129433+0000", 2024-08-17T18:26:41.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:41.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:41.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.129432+0000", 2024-08-17T18:26:41.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:26:41.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:41.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.129463+0000", 2024-08-17T18:26:41.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0839000000000001e-05 2024-08-17T18:26:41.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:41.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.129474+0000", 2024-08-17T18:26:41.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1287999999999999e-05 2024-08-17T18:26:41.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:41.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.129481+0000", 2024-08-17T18:26:41.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0299999999999996e-06 2024-08-17T18:26:41.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:41.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.129513+0000", 2024-08-17T18:26:41.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2157999999999999e-05 2024-08-17T18:26:41.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:41.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.129530+0000", 2024-08-17T18:26:41.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6929999999999999e-05 2024-08-17T18:26:41.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:26:41.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.236130+0000", 2024-08-17T18:26:41.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.106600337 2024-08-17T18:26:41.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:41.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.236164+0000", 2024-08-17T18:26:41.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3599000000000001e-05 2024-08-17T18:26:41.048 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:41.048 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:41.048 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:41.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.048 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(58.e 1721/1716 [PushOp(58:7977f106:::benchmark_data_smithi136_442124_object90298:head, version: 1714'5601, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(58:7977f106:::benchmark_data_smithi136_442124_object90298:head@1714'5601, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-08-17T18:26:41.048 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:29.132342+0000", 2024-08-17T18:26:41.048 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 251.803587803, 2024-08-17T18:26:41.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.104282737, 2024-08-17T18:26:41.048 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:26:41.048 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-08-17T18:26:41.048 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:26:41.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:26:41.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.132342+0000", 2024-08-17T18:26:41.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:41.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:26:41.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.132342+0000", 2024-08-17T18:26:41.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:26:41.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:26:41.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.132341+0000", 2024-08-17T18:26:41.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:26:41.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:26:41.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.132392+0000", 2024-08-17T18:26:41.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1341000000000002e-05 2024-08-17T18:26:41.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:26:41.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.236053+0000", 2024-08-17T18:26:41.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10366048999999999 2024-08-17T18:26:41.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:26:41.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.236096+0000", 2024-08-17T18:26:41.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3609000000000002e-05 2024-08-17T18:26:41.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:26:41.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.236362+0000", 2024-08-17T18:26:41.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000265968 2024-08-17T18:26:41.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:26:41.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.236374+0000", 2024-08-17T18:26:41.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1455e-05 2024-08-17T18:26:41.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:26:41.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:26:41.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:26:41.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:29.236624+0000", 2024-08-17T18:26:41.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025073599999999999 2024-08-17T18:26:41.052 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:41.052 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:41.052 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:41.052 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:26:41.052 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:26:41.052 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:26:41.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:41 smithi165 ceph-mon[105517]: pgmap v3554: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:41.852 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-17T18:26:41.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:41 smithi136 ceph-mon[115078]: pgmap v3554: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:41.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:41 smithi136 ceph-mon[109861]: pgmap v3554: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:42.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-08-17T18:26:42.670 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:26:43.340 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:26:43.715 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:43 smithi136 ceph-mon[109861]: pgmap v3555: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:43.715 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:43 smithi136 ceph-mon[115078]: pgmap v3555: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:44.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:43 smithi165 ceph-mon[105517]: pgmap v3555: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:44.853 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:26:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T18:26:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-17T18:26:44.853 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:26:45.461 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-17T18:26:45.897 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:26:45.992 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:45 smithi136 ceph-mon[109861]: pgmap v3556: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:45.993 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:45 smithi136 ceph-mon[115078]: pgmap v3556: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:46.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:45 smithi165 ceph-mon[105517]: pgmap v3556: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:47.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:26:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:26:47] "GET /metrics HTTP/1.1" 200 34256 "" "Prometheus/2.51.0" 2024-08-17T18:26:47.823 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-08-17T18:26:47.967 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:47 smithi136 ceph-mon[109861]: pgmap v3557: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:47.967 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:47 smithi136 ceph-mon[115078]: pgmap v3557: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:48.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:47 smithi165 ceph-mon[105517]: pgmap v3557: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:48.876 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:26:48.896 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:26:48.896 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-17T18:26:48.897 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-17T18:26:48.897 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-17T18:26:48.897 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:26:49.782 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-17T18:26:50.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:49 smithi165 ceph-mon[105517]: pgmap v3558: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:50.060 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:49 smithi136 ceph-mon[109861]: pgmap v3558: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:50.060 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:49 smithi136 ceph-mon[115078]: pgmap v3558: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:50.720 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:26:51.998 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:51 smithi136 ceph-mon[109861]: pgmap v3559: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:51.998 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:26:51.998 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:26:51.999 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:51 smithi136 ceph-mon[115078]: pgmap v3559: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:51.999 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:26:51.999 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:26:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:51 smithi165 ceph-mon[105517]: pgmap v3559: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:26:52.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:26:53.178 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-08-17T18:26:53.415 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-17T18:26:53.415 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-17T18:26:53.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-17T18:26:53.416 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-17T18:26:53.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.416 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:93183 58.d 58:bb6a1e46:::benchmark_data_smithi136_442124_object93182:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1715)", 2024-08-17T18:26:53.416 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:03.663908+0000", 2024-08-17T18:26:53.416 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 289.74715949500001, 2024-08-17T18:26:53.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0972245300000001, 2024-08-17T18:26:53.416 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:53.416 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:53.416 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:26:53.416 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:53.416 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:53.417 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:53.417 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 93183 2024-08-17T18:26:53.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.417 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:53.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:53.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.663908+0000", 2024-08-17T18:26:53.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:53.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.663908+0000", 2024-08-17T18:26:53.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:53.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.663908+0000", 2024-08-17T18:26:53.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4099999999999999e-07 2024-08-17T18:26:53.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:53.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.663928+0000", 2024-08-17T18:26:53.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9582999999999999e-05 2024-08-17T18:26:53.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.419 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:53.419 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.663937+0000", 2024-08-17T18:26:53.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7430000000000008e-06 2024-08-17T18:26:53.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.419 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.419 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:53.419 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.663941+0000", 2024-08-17T18:26:53.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8160000000000001e-06 2024-08-17T18:26:53.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.419 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.664901+0000", 2024-08-17T18:26:53.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00095965099999999995 2024-08-17T18:26:53.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.664911+0000", 2024-08-17T18:26:53.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.007e-05 2024-08-17T18:26:53.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.659228+0000", 2024-08-17T18:26:53.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.99431681199999999 2024-08-17T18:26:53.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.659238+0000", 2024-08-17T18:26:53.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9000000000000001e-06 2024-08-17T18:26:53.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.759230+0000", 2024-08-17T18:26:53.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0999924809999999 2024-08-17T18:26:53.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:53.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.761132+0000", 2024-08-17T18:26:53.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001902033 2024-08-17T18:26:53.423 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.423 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:53.423 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.423 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:93189 58.e 58:75aea715:::benchmark_data_smithi136_442124_object93188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1715)", 2024-08-17T18:26:53.423 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:03.663949+0000", 2024-08-17T18:26:53.423 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 289.74711790800001, 2024-08-17T18:26:53.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.097522278, 2024-08-17T18:26:53.423 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:53.423 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:53.424 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:26:53.424 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:53.424 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:53.424 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:53.424 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 93189 2024-08-17T18:26:53.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.424 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:53.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.424 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:53.424 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.663949+0000", 2024-08-17T18:26:53.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:53.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.663949+0000", 2024-08-17T18:26:53.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:53.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.663950+0000", 2024-08-17T18:26:53.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2100000000000002e-07 2024-08-17T18:26:53.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:53.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.663970+0000", 2024-08-17T18:26:53.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0015999999999999e-05 2024-08-17T18:26:53.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:53.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.663979+0000", 2024-08-17T18:26:53.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5009999999999993e-06 2024-08-17T18:26:53.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:53.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.663984+0000", 2024-08-17T18:26:53.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4619999999999998e-06 2024-08-17T18:26:53.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.664038+0000", 2024-08-17T18:26:53.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4120999999999999e-05 2024-08-17T18:26:53.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.664050+0000", 2024-08-17T18:26:53.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2169e-05 2024-08-17T18:26:53.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.658052+0000", 2024-08-17T18:26:53.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.994002092 2024-08-17T18:26:53.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.658063+0000", 2024-08-17T18:26:53.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1302e-05 2024-08-17T18:26:53.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.760011+0000", 2024-08-17T18:26:53.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.101947759 2024-08-17T18:26:53.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:53.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.761472+0000", 2024-08-17T18:26:53.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014604349999999999 2024-08-17T18:26:53.429 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.429 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:53.429 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.429 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:93217 58.e 58:744f1784:::benchmark_data_smithi136_442124_object93216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1715)", 2024-08-17T18:26:53.429 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:03.689852+0000", 2024-08-17T18:26:53.429 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 289.72121574900001, 2024-08-17T18:26:53.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0842789869999998, 2024-08-17T18:26:53.429 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:53.429 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:53.429 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:53.430 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:53.430 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:53.430 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:53.430 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 93217 2024-08-17T18:26:53.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.430 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:53.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:53.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.689852+0000", 2024-08-17T18:26:53.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:53.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.689852+0000", 2024-08-17T18:26:53.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:53.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.689853+0000", 2024-08-17T18:26:53.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4929999999999999e-06 2024-08-17T18:26:53.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:53.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.689875+0000", 2024-08-17T18:26:53.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2052000000000002e-05 2024-08-17T18:26:53.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.432 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:53.432 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.689888+0000", 2024-08-17T18:26:53.432 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3040000000000001e-05 2024-08-17T18:26:53.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.432 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:53.432 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.689896+0000", 2024-08-17T18:26:53.432 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4869999999999996e-06 2024-08-17T18:26:53.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.689967+0000", 2024-08-17T18:26:53.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1371999999999994e-05 2024-08-17T18:26:53.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.689978+0000", 2024-08-17T18:26:53.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1107000000000001e-05 2024-08-17T18:26:53.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.658091+0000", 2024-08-17T18:26:53.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.96811318000000002 2024-08-17T18:26:53.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.658099+0000", 2024-08-17T18:26:53.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9899999999999997e-06 2024-08-17T18:26:53.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.761766+0000", 2024-08-17T18:26:53.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1036663390000001 2024-08-17T18:26:53.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.435 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:53.435 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.762525+0000", 2024-08-17T18:26:53.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00075985099999999997 2024-08-17T18:26:53.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.435 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.435 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:26:53.435 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.762658+0000", 2024-08-17T18:26:53.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013274000000000001 2024-08-17T18:26:53.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.435 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.435 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:53.435 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.773574+0000", 2024-08-17T18:26:53.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010915658999999999 2024-08-17T18:26:53.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.773625+0000", 2024-08-17T18:26:53.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1663000000000003e-05 2024-08-17T18:26:53.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.773747+0000", 2024-08-17T18:26:53.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121481 2024-08-17T18:26:53.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.773967+0000", 2024-08-17T18:26:53.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021986 2024-08-17T18:26:53.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:53.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.774022+0000", 2024-08-17T18:26:53.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.558e-05 2024-08-17T18:26:53.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:53.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.774130+0000", 2024-08-17T18:26:53.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000108093 2024-08-17T18:26:53.437 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.438 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:53.438 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.438 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:93230 58.e 58:7924df3e:::benchmark_data_smithi136_442124_object93229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1715)", 2024-08-17T18:26:53.438 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:03.701886+0000", 2024-08-17T18:26:53.438 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 289.70918099400001, 2024-08-17T18:26:53.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0798862979999999, 2024-08-17T18:26:53.438 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:53.438 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:53.438 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:53.438 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:53.439 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:53.439 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:53.439 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 93230 2024-08-17T18:26:53.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.439 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:53.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:53.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.701886+0000", 2024-08-17T18:26:53.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:53.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.701886+0000", 2024-08-17T18:26:53.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:53.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.701888+0000", 2024-08-17T18:26:53.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6509999999999999e-06 2024-08-17T18:26:53.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:53.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.701911+0000", 2024-08-17T18:26:53.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2719999999999999e-05 2024-08-17T18:26:53.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:53.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.701927+0000", 2024-08-17T18:26:53.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6753999999999999e-05 2024-08-17T18:26:53.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:53.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.701935+0000", 2024-08-17T18:26:53.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9840000000000001e-06 2024-08-17T18:26:53.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.702009+0000", 2024-08-17T18:26:53.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3223e-05 2024-08-17T18:26:53.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.702018+0000", 2024-08-17T18:26:53.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3759999999999997e-06 2024-08-17T18:26:53.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.658125+0000", 2024-08-17T18:26:53.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.95610726599999996 2024-08-17T18:26:53.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.658133+0000", 2024-08-17T18:26:53.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5510000000000002e-06 2024-08-17T18:26:53.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.765717+0000", 2024-08-17T18:26:53.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.107584261 2024-08-17T18:26:53.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:53.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.765886+0000", 2024-08-17T18:26:53.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000168739 2024-08-17T18:26:53.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:26:53.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.766064+0000", 2024-08-17T18:26:53.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000178402 2024-08-17T18:26:53.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.773684+0000", 2024-08-17T18:26:53.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0076195259999999997 2024-08-17T18:26:53.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:53.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.773703+0000", 2024-08-17T18:26:53.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9273e-05 2024-08-17T18:26:53.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.773815+0000", 2024-08-17T18:26:53.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111723 2024-08-17T18:26:53.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.781604+0000", 2024-08-17T18:26:53.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0077888230000000003 2024-08-17T18:26:53.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:53.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.781659+0000", 2024-08-17T18:26:53.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5350999999999999e-05 2024-08-17T18:26:53.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:53.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.781773+0000", 2024-08-17T18:26:53.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113675 2024-08-17T18:26:53.446 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.446 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:53.446 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.446 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:93244 58.e 58:7486cdb7:::benchmark_data_smithi136_442124_object93243:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1715)", 2024-08-17T18:26:53.447 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:03.713933+0000", 2024-08-17T18:26:53.447 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 289.69713410899999, 2024-08-17T18:26:53.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.068027963, 2024-08-17T18:26:53.447 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:53.447 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:53.447 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:53.447 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:53.447 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:53.447 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:53.447 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 93244 2024-08-17T18:26:53.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.447 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:53.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:53.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.713933+0000", 2024-08-17T18:26:53.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:53.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.713933+0000", 2024-08-17T18:26:53.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:53.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.713934+0000", 2024-08-17T18:26:53.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2099999999999995e-07 2024-08-17T18:26:53.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:53.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.713956+0000", 2024-08-17T18:26:53.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1810999999999998e-05 2024-08-17T18:26:53.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:53.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.713968+0000", 2024-08-17T18:26:53.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1649e-05 2024-08-17T18:26:53.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:53.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.713974+0000", 2024-08-17T18:26:53.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9750000000000001e-06 2024-08-17T18:26:53.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.714043+0000", 2024-08-17T18:26:53.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8255000000000003e-05 2024-08-17T18:26:53.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.714052+0000", 2024-08-17T18:26:53.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5200000000000003e-06 2024-08-17T18:26:53.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.658158+0000", 2024-08-17T18:26:53.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.94410581800000004 2024-08-17T18:26:53.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.658183+0000", 2024-08-17T18:26:53.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4570999999999999e-05 2024-08-17T18:26:53.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.769855+0000", 2024-08-17T18:26:53.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1116725000000001 2024-08-17T18:26:53.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:53.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.770169+0000", 2024-08-17T18:26:53.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000313987 2024-08-17T18:26:53.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:26:53.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.770299+0000", 2024-08-17T18:26:53.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013020999999999999 2024-08-17T18:26:53.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:53.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.773826+0000", 2024-08-17T18:26:53.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035269910000000002 2024-08-17T18:26:53.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.773882+0000", 2024-08-17T18:26:53.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5890000000000002e-05 2024-08-17T18:26:53.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.777918+0000", 2024-08-17T18:26:53.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040352920000000002 2024-08-17T18:26:53.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.781822+0000", 2024-08-17T18:26:53.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039042679999999998 2024-08-17T18:26:53.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:53.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.781869+0000", 2024-08-17T18:26:53.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7215000000000001e-05 2024-08-17T18:26:53.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:53.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.781961+0000", 2024-08-17T18:26:53.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2089999999999994e-05 2024-08-17T18:26:53.455 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.455 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:53.455 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.455 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:93250 58.d 58:b8cd6769:::benchmark_data_smithi136_442124_object93249:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1715)", 2024-08-17T18:26:53.455 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:03.720642+0000", 2024-08-17T18:26:53.455 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 289.69042565299998, 2024-08-17T18:26:53.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0538402869999999, 2024-08-17T18:26:53.456 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:53.456 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:53.456 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:53.456 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:53.456 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:53.456 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:53.456 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 93250 2024-08-17T18:26:53.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.456 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:53.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:53.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.720642+0000", 2024-08-17T18:26:53.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:53.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.720642+0000", 2024-08-17T18:26:53.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:53.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.720643+0000", 2024-08-17T18:26:53.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.53e-06 2024-08-17T18:26:53.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:53.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.720663+0000", 2024-08-17T18:26:53.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0086e-05 2024-08-17T18:26:53.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:53.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.720673+0000", 2024-08-17T18:26:53.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0264e-05 2024-08-17T18:26:53.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:53.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.720678+0000", 2024-08-17T18:26:53.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9309999999999999e-06 2024-08-17T18:26:53.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.720737+0000", 2024-08-17T18:26:53.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8922000000000003e-05 2024-08-17T18:26:53.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.720747+0000", 2024-08-17T18:26:53.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5349999999999993e-06 2024-08-17T18:26:53.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.659267+0000", 2024-08-17T18:26:53.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93852022300000004 2024-08-17T18:26:53.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.659277+0000", 2024-08-17T18:26:53.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9729999999999992e-06 2024-08-17T18:26:53.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.761429+0000", 2024-08-17T18:26:53.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.102152247 2024-08-17T18:26:53.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:53.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.762001+0000", 2024-08-17T18:26:53.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00057144900000000002 2024-08-17T18:26:53.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,4", 2024-08-17T18:26:53.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.762135+0000", 2024-08-17T18:26:53.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013431900000000001 2024-08-17T18:26:53.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:53.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.766248+0000", 2024-08-17T18:26:53.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0041127330000000004 2024-08-17T18:26:53.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.766508+0000", 2024-08-17T18:26:53.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00026014199999999999 2024-08-17T18:26:53.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.766643+0000", 2024-08-17T18:26:53.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000134621 2024-08-17T18:26:53.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.774301+0000", 2024-08-17T18:26:53.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0076583759999999997 2024-08-17T18:26:53.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:53.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.774364+0000", 2024-08-17T18:26:53.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3215999999999999e-05 2024-08-17T18:26:53.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:53.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.774482+0000", 2024-08-17T18:26:53.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011772 2024-08-17T18:26:53.465 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.465 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:53.465 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.465 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:93252 58.e 58:769350ec:::benchmark_data_smithi136_442124_object93251:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1715)", 2024-08-17T18:26:53.465 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:03.722287+0000", 2024-08-17T18:26:53.465 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 289.68878064, 2024-08-17T18:26:53.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0611659229999999, 2024-08-17T18:26:53.466 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:53.466 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:53.466 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:53.466 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:53.466 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:53.466 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:53.466 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 93252 2024-08-17T18:26:53.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.466 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:53.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:53.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.722287+0000", 2024-08-17T18:26:53.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.467 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:53.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.722287+0000", 2024-08-17T18:26:53.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.467 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:53.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.722288+0000", 2024-08-17T18:26:53.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1500000000000003e-07 2024-08-17T18:26:53.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:53.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.722309+0000", 2024-08-17T18:26:53.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.181e-05 2024-08-17T18:26:53.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:53.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.722322+0000", 2024-08-17T18:26:53.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2717999999999999e-05 2024-08-17T18:26:53.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:53.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.722329+0000", 2024-08-17T18:26:53.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9199999999999998e-06 2024-08-17T18:26:53.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.722383+0000", 2024-08-17T18:26:53.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4148999999999999e-05 2024-08-17T18:26:53.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.722390+0000", 2024-08-17T18:26:53.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4300000000000003e-06 2024-08-17T18:26:53.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.658210+0000", 2024-08-17T18:26:53.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93582048900000003 2024-08-17T18:26:53.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.658218+0000", 2024-08-17T18:26:53.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6669999999999996e-06 2024-08-17T18:26:53.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.774182+0000", 2024-08-17T18:26:53.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1159646409999999 2024-08-17T18:26:53.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:53.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.774362+0000", 2024-08-17T18:26:53.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000179686 2024-08-17T18:26:53.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:26:53.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.774502+0000", 2024-08-17T18:26:53.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000139672 2024-08-17T18:26:53.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.777961+0000", 2024-08-17T18:26:53.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003458914 2024-08-17T18:26:53.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:53.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.781788+0000", 2024-08-17T18:26:53.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038273410000000002 2024-08-17T18:26:53.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.781998+0000", 2024-08-17T18:26:53.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020954 2024-08-17T18:26:53.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.782351+0000", 2024-08-17T18:26:53.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00035384199999999999 2024-08-17T18:26:53.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:53.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.783346+0000", 2024-08-17T18:26:53.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00099427299999999995 2024-08-17T18:26:53.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:53.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.783453+0000", 2024-08-17T18:26:53.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106916 2024-08-17T18:26:53.474 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.474 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:53.474 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.474 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:93258 58.e 58:72d04e18:::benchmark_data_smithi136_442124_object93257:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1715)", 2024-08-17T18:26:53.474 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:03.731845+0000", 2024-08-17T18:26:53.474 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 289.679222364, 2024-08-17T18:26:53.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0608153100000002, 2024-08-17T18:26:53.474 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:53.474 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:53.475 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:53.475 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:53.475 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:53.475 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:53.475 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 93258 2024-08-17T18:26:53.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.475 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:53.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:53.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.731845+0000", 2024-08-17T18:26:53.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:53.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.731845+0000", 2024-08-17T18:26:53.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:53.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.731846+0000", 2024-08-17T18:26:53.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1939999999999999e-06 2024-08-17T18:26:53.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:53.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.731867+0000", 2024-08-17T18:26:53.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0670999999999999e-05 2024-08-17T18:26:53.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.477 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:53.477 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.731879+0000", 2024-08-17T18:26:53.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2619999999999999e-05 2024-08-17T18:26:53.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.477 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:53.477 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.731885+0000", 2024-08-17T18:26:53.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9200000000000001e-06 2024-08-17T18:26:53.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.477 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.478 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.731940+0000", 2024-08-17T18:26:53.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5204000000000002e-05 2024-08-17T18:26:53.478 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.478 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.478 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.478 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:03.731947+0000", 2024-08-17T18:26:53.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6100000000000002e-06 2024-08-17T18:26:53.478 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.478 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.478 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.478 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.658300+0000", 2024-08-17T18:26:53.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92635331700000001 2024-08-17T18:26:53.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:04.658308+0000", 2024-08-17T18:26:53.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0609999999999996e-06 2024-08-17T18:26:53.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.777994+0000", 2024-08-17T18:26:53.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.119685437 2024-08-17T18:26:53.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:53.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.778136+0000", 2024-08-17T18:26:53.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014222799999999999 2024-08-17T18:26:53.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:26:53.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.778285+0000", 2024-08-17T18:26:53.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014846799999999999 2024-08-17T18:26:53.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:53.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.782010+0000", 2024-08-17T18:26:53.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037250339999999999 2024-08-17T18:26:53.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.481 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.481 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.481 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.782043+0000", 2024-08-17T18:26:53.481 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2903e-05 2024-08-17T18:26:53.481 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.481 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.481 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.481 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.783503+0000", 2024-08-17T18:26:53.481 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014600259999999999 2024-08-17T18:26:53.481 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.481 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.481 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.482 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.792497+0000", 2024-08-17T18:26:53.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0089942779999999996 2024-08-17T18:26:53.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.482 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.482 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:53.482 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.792556+0000", 2024-08-17T18:26:53.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9027999999999998e-05 2024-08-17T18:26:53.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.482 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.482 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:53.482 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:05.792660+0000", 2024-08-17T18:26:53.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104311 2024-08-17T18:26:53.482 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.483 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:53.483 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.483 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.483 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:117416 58.7 58:eb43e51f:::benchmark_data_smithi136_442124_object3800:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1724)", 2024-08-17T18:26:53.483 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:35.831658+0000", 2024-08-17T18:26:53.483 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 257.57940890100002, 2024-08-17T18:26:53.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2346256489999998, 2024-08-17T18:26:53.483 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:53.483 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:53.483 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:53.483 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:53.483 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:53.484 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:53.484 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 117416 2024-08-17T18:26:53.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.484 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:53.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:53.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.831658+0000", 2024-08-17T18:26:53.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:53.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.831658+0000", 2024-08-17T18:26:53.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:53.485 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.831660+0000", 2024-08-17T18:26:53.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.728e-06 2024-08-17T18:26:53.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:53.485 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.831665+0000", 2024-08-17T18:26:53.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3600000000000004e-06 2024-08-17T18:26:53.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:53.486 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.831681+0000", 2024-08-17T18:26:53.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5942000000000001e-05 2024-08-17T18:26:53.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.486 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.486 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:53.486 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.831690+0000", 2024-08-17T18:26:53.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1300000000000001e-06 2024-08-17T18:26:53.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.486 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.486 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.486 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.831767+0000", 2024-08-17T18:26:53.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7917999999999995e-05 2024-08-17T18:26:53.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.487 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.487 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.831790+0000", 2024-08-17T18:26:53.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2765999999999999e-05 2024-08-17T18:26:53.487 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.487 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.487 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.831610+0000", 2024-08-17T18:26:53.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.99981979200000004 2024-08-17T18:26:53.487 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.487 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.487 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.831618+0000", 2024-08-17T18:26:53.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1249999999999993e-06 2024-08-17T18:26:53.488 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.488 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.488 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.488 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.024861+0000", 2024-08-17T18:26:53.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1932427990000001 2024-08-17T18:26:53.488 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.488 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.488 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:53.488 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.024993+0000", 2024-08-17T18:26:53.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000132256 2024-08-17T18:26:53.488 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.488 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.489 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,6", 2024-08-17T18:26:53.489 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.025120+0000", 2024-08-17T18:26:53.489 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000126348 2024-08-17T18:26:53.489 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.489 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.489 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.043085+0000", 2024-08-17T18:26:53.489 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017965954999999999 2024-08-17T18:26:53.489 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.489 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:53.489 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.043103+0000", 2024-08-17T18:26:53.489 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7897000000000002e-05 2024-08-17T18:26:53.489 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.490 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.490 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.490 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.063440+0000", 2024-08-17T18:26:53.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020336811999999999 2024-08-17T18:26:53.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.490 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.490 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:53.490 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.063575+0000", 2024-08-17T18:26:53.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013467899999999999 2024-08-17T18:26:53.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.490 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.490 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:53.490 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.066284+0000", 2024-08-17T18:26:53.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0027091419999999999 2024-08-17T18:26:53.491 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.491 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:53.491 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.491 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:117428 58.c 58:3dc497e4:::benchmark_data_smithi136_442124_object3812:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1724)", 2024-08-17T18:26:53.491 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:35.951490+0000", 2024-08-17T18:26:53.491 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 257.45957745599998, 2024-08-17T18:26:53.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.07934186, 2024-08-17T18:26:53.491 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:53.491 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:53.492 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:53.492 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:53.492 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:53.492 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:53.492 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 117428 2024-08-17T18:26:53.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.492 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:53.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:53.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.951490+0000", 2024-08-17T18:26:53.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:53.493 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.951490+0000", 2024-08-17T18:26:53.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:53.493 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.951491+0000", 2024-08-17T18:26:53.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.294e-06 2024-08-17T18:26:53.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:53.493 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.951496+0000", 2024-08-17T18:26:53.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0699999999999997e-06 2024-08-17T18:26:53.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.494 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:53.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.951512+0000", 2024-08-17T18:26:53.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5423000000000001e-05 2024-08-17T18:26:53.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.494 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:53.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.951519+0000", 2024-08-17T18:26:53.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.08e-06 2024-08-17T18:26:53.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.494 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.951605+0000", 2024-08-17T18:26:53.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6812999999999997e-05 2024-08-17T18:26:53.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.495 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.951617+0000", 2024-08-17T18:26:53.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1569999999999999e-05 2024-08-17T18:26:53.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.495 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.830453+0000", 2024-08-17T18:26:53.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.87883554100000005 2024-08-17T18:26:53.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.830464+0000", 2024-08-17T18:26:53.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1063999999999999e-05 2024-08-17T18:26:53.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:37.985647+0000", 2024-08-17T18:26:53.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.155183123 2024-08-17T18:26:53.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:53.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.003552+0000", 2024-08-17T18:26:53.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017904776000000001 2024-08-17T18:26:53.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,6", 2024-08-17T18:26:53.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.003704+0000", 2024-08-17T18:26:53.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000152526 2024-08-17T18:26:53.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.018987+0000", 2024-08-17T18:26:53.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015282706 2024-08-17T18:26:53.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.019033+0000", 2024-08-17T18:26:53.498 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6235e-05 2024-08-17T18:26:53.498 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.498 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:53.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.019143+0000", 2024-08-17T18:26:53.498 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011043 2024-08-17T18:26:53.498 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.498 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:53.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.019482+0000", 2024-08-17T18:26:53.498 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000338153 2024-08-17T18:26:53.499 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.499 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.499 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:53.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.030832+0000", 2024-08-17T18:26:53.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011350056000000001 2024-08-17T18:26:53.499 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.499 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:53.499 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.499 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.499 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.499 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:117431 58.7 58:e0ee6161:::benchmark_data_smithi136_442124_object3815:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1724)", 2024-08-17T18:26:53.499 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:35.952423+0000", 2024-08-17T18:26:53.499 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 257.45864472300002, 2024-08-17T18:26:53.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1159465389999998, 2024-08-17T18:26:53.500 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:53.500 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:53.500 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:53.500 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:53.500 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:53.500 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:53.500 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 117431 2024-08-17T18:26:53.500 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.500 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:53.500 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.500 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:53.500 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952423+0000", 2024-08-17T18:26:53.501 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:53.501 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952423+0000", 2024-08-17T18:26:53.501 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:53.501 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952423+0000", 2024-08-17T18:26:53.501 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2900000000000002e-07 2024-08-17T18:26:53.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:53.502 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952428+0000", 2024-08-17T18:26:53.502 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3320000000000002e-06 2024-08-17T18:26:53.502 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.502 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.502 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:53.502 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952439+0000", 2024-08-17T18:26:53.502 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1296e-05 2024-08-17T18:26:53.502 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.502 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.502 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:53.502 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952446+0000", 2024-08-17T18:26:53.502 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7309999999999996e-06 2024-08-17T18:26:53.502 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952503+0000", 2024-08-17T18:26:53.503 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7649999999999999e-05 2024-08-17T18:26:53.503 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.952512+0000", 2024-08-17T18:26:53.503 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9500000000000007e-06 2024-08-17T18:26:53.503 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.831645+0000", 2024-08-17T18:26:53.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.87913216699999996 2024-08-17T18:26:53.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.504 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.504 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.831651+0000", 2024-08-17T18:26:53.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7970000000000003e-06 2024-08-17T18:26:53.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.504 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.504 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.031454+0000", 2024-08-17T18:26:53.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1998025189999999 2024-08-17T18:26:53.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.505 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:53.505 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.031606+0000", 2024-08-17T18:26:53.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015214699999999999 2024-08-17T18:26:53.505 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.505 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.505 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,6", 2024-08-17T18:26:53.505 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.031761+0000", 2024-08-17T18:26:53.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015501299999999999 2024-08-17T18:26:53.505 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.505 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.505 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.505 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.043149+0000", 2024-08-17T18:26:53.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011387981 2024-08-17T18:26:53.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:53.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.043163+0000", 2024-08-17T18:26:53.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4039000000000001e-05 2024-08-17T18:26:53.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.066338+0000", 2024-08-17T18:26:53.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023175281999999998 2024-08-17T18:26:53.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:53.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.066390+0000", 2024-08-17T18:26:53.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1997999999999999e-05 2024-08-17T18:26:53.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.507 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:53.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.068369+0000", 2024-08-17T18:26:53.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001978708 2024-08-17T18:26:53.507 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.507 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:53.507 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.507 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:117438 58.c 58:3abb8eca:::benchmark_data_smithi136_442124_object3822:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1724)", 2024-08-17T18:26:53.508 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:35.961815+0000", 2024-08-17T18:26:53.508 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 257.44925179400002, 2024-08-17T18:26:53.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0707978649999998, 2024-08-17T18:26:53.508 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:53.508 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:53.508 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:53.508 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:53.508 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:53.508 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:53.508 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 117438 2024-08-17T18:26:53.508 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.508 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:53.508 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:53.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.961815+0000", 2024-08-17T18:26:53.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:53.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.961815+0000", 2024-08-17T18:26:53.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:53.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.961817+0000", 2024-08-17T18:26:53.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2890000000000001e-06 2024-08-17T18:26:53.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:53.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.961822+0000", 2024-08-17T18:26:53.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6389999999999997e-06 2024-08-17T18:26:53.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:53.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.961834+0000", 2024-08-17T18:26:53.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1892000000000001e-05 2024-08-17T18:26:53.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:53.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.961842+0000", 2024-08-17T18:26:53.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1480000000000006e-06 2024-08-17T18:26:53.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.961906+0000", 2024-08-17T18:26:53.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3990000000000002e-05 2024-08-17T18:26:53.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.961916+0000", 2024-08-17T18:26:53.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7380000000000006e-06 2024-08-17T18:26:53.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.830540+0000", 2024-08-17T18:26:53.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.86862356200000002 2024-08-17T18:26:53.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.830548+0000", 2024-08-17T18:26:53.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1689999999999992e-06 2024-08-17T18:26:53.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.010461+0000", 2024-08-17T18:26:53.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1799135590000001 2024-08-17T18:26:53.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:53.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.010600+0000", 2024-08-17T18:26:53.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000138455 2024-08-17T18:26:53.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4,6", 2024-08-17T18:26:53.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.010764+0000", 2024-08-17T18:26:53.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000164038 2024-08-17T18:26:53.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.514 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.019077+0000", 2024-08-17T18:26:53.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0083134309999999996 2024-08-17T18:26:53.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.514 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.019122+0000", 2024-08-17T18:26:53.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.494e-05 2024-08-17T18:26:53.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:53.514 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.030868+0000", 2024-08-17T18:26:53.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011746081 2024-08-17T18:26:53.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:53.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.030914+0000", 2024-08-17T18:26:53.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5902000000000002e-05 2024-08-17T18:26:53.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:53.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.032613+0000", 2024-08-17T18:26:53.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016990320000000001 2024-08-17T18:26:53.515 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.515 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:53.516 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.516 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:117440 58.7 58:e3fc2b90:::benchmark_data_smithi136_442124_object3824:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1724)", 2024-08-17T18:26:53.516 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:22:35.968816+0000", 2024-08-17T18:26:53.516 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 257.44225084700003, 2024-08-17T18:26:53.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1090719419999999, 2024-08-17T18:26:53.516 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:53.516 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:53.516 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:53.516 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:53.516 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:53.516 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:53.517 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 117440 2024-08-17T18:26:53.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.517 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:53.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:53.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.968816+0000", 2024-08-17T18:26:53.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:53.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.968816+0000", 2024-08-17T18:26:53.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:53.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.968818+0000", 2024-08-17T18:26:53.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.372e-06 2024-08-17T18:26:53.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:53.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.968822+0000", 2024-08-17T18:26:53.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.941e-06 2024-08-17T18:26:53.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:53.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.968832+0000", 2024-08-17T18:26:53.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8570000000000007e-06 2024-08-17T18:26:53.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:53.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.968836+0000", 2024-08-17T18:26:53.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9099999999999996e-06 2024-08-17T18:26:53.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.968916+0000", 2024-08-17T18:26:53.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9375999999999994e-05 2024-08-17T18:26:53.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:35.968923+0000", 2024-08-17T18:26:53.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2119999999999997e-06 2024-08-17T18:26:53.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.831745+0000", 2024-08-17T18:26:53.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.86282181700000005 2024-08-17T18:26:53.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-17T18:26:53.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:36.831752+0000", 2024-08-17T18:26:53.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0210000000000002e-06 2024-08-17T18:26:53.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.039164+0000", 2024-08-17T18:26:53.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2074123729999999 2024-08-17T18:26:53.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:53.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.039290+0000", 2024-08-17T18:26:53.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125284 2024-08-17T18:26:53.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,6", 2024-08-17T18:26:53.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.039444+0000", 2024-08-17T18:26:53.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015427299999999999 2024-08-17T18:26:53.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:53.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.047543+0000", 2024-08-17T18:26:53.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0080995270000000005 2024-08-17T18:26:53.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.051978+0000", 2024-08-17T18:26:53.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0044349530000000002 2024-08-17T18:26:53.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.068429+0000", 2024-08-17T18:26:53.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016450848000000001 2024-08-17T18:26:53.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:53.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.068479+0000", 2024-08-17T18:26:53.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0300000000000003e-05 2024-08-17T18:26:53.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:53.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:22:38.077888+0000", 2024-08-17T18:26:53.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0094088780000000007 2024-08-17T18:26:53.524 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.524 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:53.524 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.524 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:148184 58.d 58:b7f299fc:::benchmark_data_smithi136_442124_object34568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1732)", 2024-08-17T18:26:53.525 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:22.715750+0000", 2024-08-17T18:26:53.525 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 210.69531742699999, 2024-08-17T18:26:53.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.48974341100000002, 2024-08-17T18:26:53.525 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:53.525 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:53.525 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:53.525 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:53.525 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:53.525 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:53.525 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 148184 2024-08-17T18:26:53.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.525 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:53.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:53.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.715750+0000", 2024-08-17T18:26:53.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:53.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.715750+0000", 2024-08-17T18:26:53.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:53.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.715751+0000", 2024-08-17T18:26:53.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.28e-06 2024-08-17T18:26:53.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:53.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.715756+0000", 2024-08-17T18:26:53.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5929999999999999e-06 2024-08-17T18:26:53.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:53.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.715764+0000", 2024-08-17T18:26:53.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7280000000000001e-06 2024-08-17T18:26:53.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:53.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.715770+0000", 2024-08-17T18:26:53.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0440000000000001e-06 2024-08-17T18:26:53.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.715819+0000", 2024-08-17T18:26:53.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8964000000000001e-05 2024-08-17T18:26:53.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:53.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.715962+0000", 2024-08-17T18:26:53.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000142657 2024-08-17T18:26:53.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,4", 2024-08-17T18:26:53.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.716108+0000", 2024-08-17T18:26:53.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014576199999999999 2024-08-17T18:26:53.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.721179+0000", 2024-08-17T18:26:53.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0050714760000000001 2024-08-17T18:26:53.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.721262+0000", 2024-08-17T18:26:53.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2540999999999995e-05 2024-08-17T18:26:53.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:53.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.723146+0000", 2024-08-17T18:26:53.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001883771 2024-08-17T18:26:53.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:23.205206+0000", 2024-08-17T18:26:53.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.48206084999999999 2024-08-17T18:26:53.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.531 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:53.531 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:23.205326+0000", 2024-08-17T18:26:53.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119919 2024-08-17T18:26:53.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.531 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:53.531 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:23.205493+0000", 2024-08-17T18:26:53.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016682599999999999 2024-08-17T18:26:53.531 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.531 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:53.531 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.531 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:148193 58.e 58:7537a0a3:::benchmark_data_smithi136_442124_object34577:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1732)", 2024-08-17T18:26:53.532 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:22.715849+0000", 2024-08-17T18:26:53.532 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 210.69521784400001, 2024-08-17T18:26:53.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.50632263799999999, 2024-08-17T18:26:53.532 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:53.532 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:53.532 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:53.532 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:53.532 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:53.532 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:53.532 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 148193 2024-08-17T18:26:53.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.532 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:53.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:53.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.715849+0000", 2024-08-17T18:26:53.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:53.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.715849+0000", 2024-08-17T18:26:53.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:53.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.715850+0000", 2024-08-17T18:26:53.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4999999999999998e-07 2024-08-17T18:26:53.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:53.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.715852+0000", 2024-08-17T18:26:53.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6980000000000001e-06 2024-08-17T18:26:53.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:53.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.715857+0000", 2024-08-17T18:26:53.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3109999999999996e-06 2024-08-17T18:26:53.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:53.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.715861+0000", 2024-08-17T18:26:53.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4660000000000001e-06 2024-08-17T18:26:53.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.716382+0000", 2024-08-17T18:26:53.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00052041299999999995 2024-08-17T18:26:53.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:53.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.716495+0000", 2024-08-17T18:26:53.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113646 2024-08-17T18:26:53.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:26:53.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.716610+0000", 2024-08-17T18:26:53.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011455699999999999 2024-08-17T18:26:53.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:53.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.724550+0000", 2024-08-17T18:26:53.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0079402380000000005 2024-08-17T18:26:53.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.731829+0000", 2024-08-17T18:26:53.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0072786689999999998 2024-08-17T18:26:53.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.732317+0000", 2024-08-17T18:26:53.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000488087 2024-08-17T18:26:53.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:23.222006+0000", 2024-08-17T18:26:53.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.48968878900000001 2024-08-17T18:26:53.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:53.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:23.222061+0000", 2024-08-17T18:26:53.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5529000000000003e-05 2024-08-17T18:26:53.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:53.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:23.222172+0000", 2024-08-17T18:26:53.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110785 2024-08-17T18:26:53.539 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.539 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:53.539 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.539 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:148195 58.e 58:7242de7d:::benchmark_data_smithi136_442124_object34579:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1732)", 2024-08-17T18:26:53.539 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:22.720241+0000", 2024-08-17T18:26:53.539 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 210.69082647499999, 2024-08-17T18:26:53.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.50211468199999998, 2024-08-17T18:26:53.540 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:53.540 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:53.540 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:53.540 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:53.540 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:53.540 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:53.540 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 148195 2024-08-17T18:26:53.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.540 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:53.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:53.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.720241+0000", 2024-08-17T18:26:53.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:53.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.720241+0000", 2024-08-17T18:26:53.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:53.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.720242+0000", 2024-08-17T18:26:53.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3629999999999999e-06 2024-08-17T18:26:53.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:53.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.720248+0000", 2024-08-17T18:26:53.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6869999999999999e-06 2024-08-17T18:26:53.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:53.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.720260+0000", 2024-08-17T18:26:53.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2113000000000001e-05 2024-08-17T18:26:53.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:53.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.720267+0000", 2024-08-17T18:26:53.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6850000000000004e-06 2024-08-17T18:26:53.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.720369+0000", 2024-08-17T18:26:53.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010273200000000001 2024-08-17T18:26:53.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:53.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.720513+0000", 2024-08-17T18:26:53.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000143306 2024-08-17T18:26:53.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:26:53.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.720654+0000", 2024-08-17T18:26:53.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014137000000000001 2024-08-17T18:26:53.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:53.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.724737+0000", 2024-08-17T18:26:53.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040833739999999999 2024-08-17T18:26:53.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.731874+0000", 2024-08-17T18:26:53.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071365769999999999 2024-08-17T18:26:53.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.732370+0000", 2024-08-17T18:26:53.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00049560000000000001 2024-08-17T18:26:53.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:23.222212+0000", 2024-08-17T18:26:53.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.48984221 2024-08-17T18:26:53.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:53.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:23.222258+0000", 2024-08-17T18:26:53.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6452999999999997e-05 2024-08-17T18:26:53.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.546 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:53.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:23.222355+0000", 2024-08-17T18:26:53.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7212000000000004e-05 2024-08-17T18:26:53.546 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.546 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:53.546 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.546 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.36936.0:148216 58.e 58:768419fc:::benchmark_data_smithi136_442124_object34600:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1732)", 2024-08-17T18:26:53.546 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:23:22.746927+0000", 2024-08-17T18:26:53.547 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 210.664140076, 2024-08-17T18:26:53.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.47561308400000002, 2024-08-17T18:26:53.547 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:53.547 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:53.547 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:26:53.547 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:53.547 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.36936", 2024-08-17T18:26:53.547 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:26:53.547 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 148216 2024-08-17T18:26:53.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.547 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:53.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.547 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:53.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.746927+0000", 2024-08-17T18:26:53.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:53.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.746927+0000", 2024-08-17T18:26:53.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:53.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.746928+0000", 2024-08-17T18:26:53.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.114e-06 2024-08-17T18:26:53.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:53.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.746933+0000", 2024-08-17T18:26:53.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9269999999999996e-06 2024-08-17T18:26:53.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:53.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.746946+0000", 2024-08-17T18:26:53.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2927e-05 2024-08-17T18:26:53.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:53.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.746953+0000", 2024-08-17T18:26:53.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0530000000000001e-06 2024-08-17T18:26:53.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.747043+0000", 2024-08-17T18:26:53.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0070999999999999e-05 2024-08-17T18:26:53.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-17T18:26:53.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.747174+0000", 2024-08-17T18:26:53.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013036100000000001 2024-08-17T18:26:53.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,3,7", 2024-08-17T18:26:53.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.747306+0000", 2024-08-17T18:26:53.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013204300000000001 2024-08-17T18:26:53.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.750906+0000", 2024-08-17T18:26:53.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036006419999999998 2024-08-17T18:26:53.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-17T18:26:53.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.751048+0000", 2024-08-17T18:26:53.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000141271 2024-08-17T18:26:53.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:22.754083+0000", 2024-08-17T18:26:53.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0030356099999999998 2024-08-17T18:26:53.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:26:53.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:23.222393+0000", 2024-08-17T18:26:53.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.468310165 2024-08-17T18:26:53.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.553 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-17T18:26:53.553 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:23.222443+0000", 2024-08-17T18:26:53.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9617999999999998e-05 2024-08-17T18:26:53.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.553 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.553 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:53.553 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:23:23.222540+0000", 2024-08-17T18:26:53.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7281999999999995e-05 2024-08-17T18:26:53.553 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.553 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:53.553 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.553 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.554 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25175.0:181 2.1 2:88c1567c:::rbd_trash_purge_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e1751)", 2024-08-17T18:26:53.554 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-17T18:26:06.807344+0000", 2024-08-17T18:26:53.554 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.603723490999997, 2024-08-17T18:26:53.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00042747800000000002, 2024-08-17T18:26:53.554 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-17T18:26:53.554 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-17T18:26:53.554 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-17T18:26:53.554 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-17T18:26:53.554 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25175", 2024-08-17T18:26:53.554 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.136:0/3845461328", 2024-08-17T18:26:53.555 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 181 2024-08-17T18:26:53.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.555 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-17T18:26:53.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-17T18:26:53.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:26:06.807344+0000", 2024-08-17T18:26:53.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-17T18:26:53.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:26:06.807344+0000", 2024-08-17T18:26:53.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-17T18:26:53.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-17T18:26:53.556 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:26:06.807346+0000", 2024-08-17T18:26:53.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.261e-06 2024-08-17T18:26:53.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-17T18:26:53.556 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:26:06.807355+0000", 2024-08-17T18:26:53.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3840000000000003e-06 2024-08-17T18:26:53.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-17T18:26:53.557 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:26:06.807376+0000", 2024-08-17T18:26:53.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0849999999999999e-05 2024-08-17T18:26:53.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.557 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-17T18:26:53.557 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:26:06.807385+0000", 2024-08-17T18:26:53.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8510000000000005e-06 2024-08-17T18:26:53.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.557 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-17T18:26:53.557 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:26:06.807651+0000", 2024-08-17T18:26:53.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00026563900000000002 2024-08-17T18:26:53.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-17T18:26:53.558 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-17T18:26:53.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-17T18:26:53.558 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-17T18:26:06.807771+0000", 2024-08-17T18:26:53.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120493 2024-08-17T18:26:53.558 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.558 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:53.558 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.558 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-17T18:26:53.558 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-17T18:26:53.558 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-17T18:26:54.005 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:53 smithi165 ceph-mon[105517]: pgmap v3560: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:54.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:53 smithi136 ceph-mon[109861]: pgmap v3560: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:54.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:53 smithi136 ceph-mon[115078]: pgmap v3560: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:55.122 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-17T18:26:55.209 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:26:55.894 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:26:56.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:55 smithi165 ceph-mon[105517]: pgmap v3561: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:56.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:55 smithi136 ceph-mon[109861]: pgmap v3561: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:56.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:55 smithi136 ceph-mon[115078]: pgmap v3561: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:57.148 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:56 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:26:57.148 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:56 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:26:57.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:56 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:26:57.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:56 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:26:57.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:56 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:26:57.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:56 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:26:57.591 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:26:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:26:57] "GET /metrics HTTP/1.1" 200 34257 "" "Prometheus/2.51.0" 2024-08-17T18:26:57.669 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:26:57.670 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:26:57.670 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-08-17T18:26:57.670 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:26:57.895 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-08-17T18:26:58.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:57 smithi136 ceph-mon[109861]: pgmap v3562: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:58.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:57 smithi136 ceph-mon[115078]: pgmap v3562: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:57 smithi165 ceph-mon[105517]: pgmap v3562: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:26:58.328 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-17T18:26:58.532 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:26:58.701 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:27:00.158 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:26:59 smithi165 ceph-mon[105517]: pgmap v3563: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:00.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:26:59 smithi136 ceph-mon[109861]: pgmap v3563: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:00.199 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:26:59 smithi136 ceph-mon[115078]: pgmap v3563: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:00.267 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:27:00.267 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-08-17T18:27:00.267 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-08-17T18:27:00.267 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-08-17T18:27:00.268 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:27:00.986 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-17T18:27:01.195 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T18:27:02.191 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:01 smithi136 ceph-mon[115078]: pgmap v3564: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:02.191 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:01 smithi136 ceph-mon[109861]: pgmap v3564: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:02.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:01 smithi165 ceph-mon[105517]: pgmap v3564: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:02.957 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-08-17T18:27:02.957 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-08-17T18:27:02.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-08-17T18:27:02.957 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-08-17T18:27:02.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.958 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:117397 58.1 58.be219b71 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e1724)", 2024-08-17T18:27:02.958 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:35.829777+0000", 2024-08-17T18:27:02.958 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 267.122946673, 2024-08-17T18:27:02.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.000250192, 2024-08-17T18:27:02.958 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:27:02.958 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:27:02.958 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:27:02.958 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:27:02.958 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:27:02.958 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 117397 2024-08-17T18:27:02.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.958 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:27:02.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:27:02.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829777+0000", 2024-08-17T18:27:02.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:02.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:27:02.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829777+0000", 2024-08-17T18:27:02.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:02.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:27:02.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829776+0000", 2024-08-17T18:27:02.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:27:02.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:27:02.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829782+0000", 2024-08-17T18:27:02.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9379999999999997e-06 2024-08-17T18:27:02.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:27:02.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829793+0000", 2024-08-17T18:27:02.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1321999999999999e-05 2024-08-17T18:27:02.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:27:02.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829802+0000", 2024-08-17T18:27:02.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4770000000000003e-06 2024-08-17T18:27:02.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:02.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829919+0000", 2024-08-17T18:27:02.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000117695 2024-08-17T18:27:02.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:27:02.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829942+0000", 2024-08-17T18:27:02.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2864e-05 2024-08-17T18:27:02.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:02.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:36.830017+0000", 2024-08-17T18:27:02.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.000074567 2024-08-17T18:27:02.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:27:02.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:36.830028+0000", 2024-08-17T18:27:02.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0794e-05 2024-08-17T18:27:02.962 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:02.962 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:27:02.962 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:02.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.963 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:117402 58.1 58.b36cf831 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e1724)", 2024-08-17T18:27:02.963 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:35.829819+0000", 2024-08-17T18:27:02.963 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 267.12290464199998, 2024-08-17T18:27:02.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0002586389999999, 2024-08-17T18:27:02.963 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:27:02.963 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:27:02.963 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:27:02.963 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:27:02.963 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:27:02.963 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 117402 2024-08-17T18:27:02.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.963 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:27:02.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:27:02.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829819+0000", 2024-08-17T18:27:02.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:02.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:27:02.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829819+0000", 2024-08-17T18:27:02.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:02.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:27:02.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829819+0000", 2024-08-17T18:27:02.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:27:02.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:27:02.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829822+0000", 2024-08-17T18:27:02.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0450000000000001e-06 2024-08-17T18:27:02.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:27:02.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829828+0000", 2024-08-17T18:27:02.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1650000000000003e-06 2024-08-17T18:27:02.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:27:02.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829834+0000", 2024-08-17T18:27:02.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1140000000000002e-06 2024-08-17T18:27:02.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:02.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829966+0000", 2024-08-17T18:27:02.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013191200000000001 2024-08-17T18:27:02.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:27:02.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829974+0000", 2024-08-17T18:27:02.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6369999999999999e-06 2024-08-17T18:27:02.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:02.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:36.830062+0000", 2024-08-17T18:27:02.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0000880130000001 2024-08-17T18:27:02.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:27:02.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:36.830078+0000", 2024-08-17T18:27:02.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6158999999999999e-05 2024-08-17T18:27:02.967 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:02.967 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:27:02.967 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:02.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.968 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:117414 58.1 58.711aa421 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e1724)", 2024-08-17T18:27:02.968 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:22:35.829839+0000", 2024-08-17T18:27:02.968 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 267.12288514400001, 2024-08-17T18:27:02.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0002668699999999, 2024-08-17T18:27:02.968 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:27:02.968 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-08-17T18:27:02.968 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:27:02.968 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:27:02.968 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:27:02.968 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 117414 2024-08-17T18:27:02.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.968 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:27:02.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:27:02.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829839+0000", 2024-08-17T18:27:02.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:02.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:27:02.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829839+0000", 2024-08-17T18:27:02.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:02.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:27:02.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829839+0000", 2024-08-17T18:27:02.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:27:02.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:27:02.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829841+0000", 2024-08-17T18:27:02.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0200000000000001e-06 2024-08-17T18:27:02.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:27:02.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829846+0000", 2024-08-17T18:27:02.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7459999999999997e-06 2024-08-17T18:27:02.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:27:02.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829851+0000", 2024-08-17T18:27:02.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5739999999999997e-06 2024-08-17T18:27:02.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:02.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.829995+0000", 2024-08-17T18:27:02.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000144444 2024-08-17T18:27:02.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:27:02.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:35.830003+0000", 2024-08-17T18:27:02.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4390000000000003e-06 2024-08-17T18:27:02.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:02.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:36.830098+0000", 2024-08-17T18:27:02.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.000094966 2024-08-17T18:27:02.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:27:02.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:22:36.830106+0000", 2024-08-17T18:27:02.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0520000000000002e-06 2024-08-17T18:27:02.972 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:02.972 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:27:02.972 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:02.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.972 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:171677 58.1 58:8e95c71b:::benchmark_data_smithi136_442124_object58061:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1741)", 2024-08-17T18:27:02.973 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:23:59.720295+0000", 2024-08-17T18:27:02.973 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 183.232428653, 2024-08-17T18:27:02.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2225817000000001, 2024-08-17T18:27:02.973 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:27:02.973 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:27:02.973 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:27:02.973 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:27:02.973 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:27:02.973 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 171677 2024-08-17T18:27:02.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.973 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:27:02.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:27:02.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.720295+0000", 2024-08-17T18:27:02.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:02.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:27:02.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.720295+0000", 2024-08-17T18:27:02.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:02.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:27:02.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.720294+0000", 2024-08-17T18:27:02.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:27:02.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:27:02.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.720302+0000", 2024-08-17T18:27:02.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1049999999999995e-06 2024-08-17T18:27:02.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:27:02.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.720315+0000", 2024-08-17T18:27:02.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2208000000000001e-05 2024-08-17T18:27:02.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:27:02.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.720323+0000", 2024-08-17T18:27:02.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8839999999999994e-06 2024-08-17T18:27:02.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:02.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.720373+0000", 2024-08-17T18:27:02.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0237e-05 2024-08-17T18:27:02.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-08-17T18:27:02.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.720379+0000", 2024-08-17T18:27:02.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1999999999999999e-06 2024-08-17T18:27:02.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:02.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.722122+0000", 2024-08-17T18:27:02.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001743306 2024-08-17T18:27:02.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:27:02.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.722133+0000", 2024-08-17T18:27:02.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0657e-05 2024-08-17T18:27:02.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:02.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.886738+0000", 2024-08-17T18:27:02.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1646055089999998 2024-08-17T18:27:02.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:27:02.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.889936+0000", 2024-08-17T18:27:02.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031972390000000002 2024-08-17T18:27:02.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1,7", 2024-08-17T18:27:02.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.895583+0000", 2024-08-17T18:27:02.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056469759999999997 2024-08-17T18:27:02.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:27:02.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.922390+0000", 2024-08-17T18:27:02.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.026807780999999999 2024-08-17T18:27:02.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:02.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.932746+0000", 2024-08-17T18:27:02.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010355804999999999 2024-08-17T18:27:02.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:02.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.933770+0000", 2024-08-17T18:27:02.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001023335 2024-08-17T18:27:02.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:02.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.934807+0000", 2024-08-17T18:27:02.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001037477 2024-08-17T18:27:02.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:27:02.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.936539+0000", 2024-08-17T18:27:02.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017315760000000001 2024-08-17T18:27:02.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:27:02.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.942877+0000", 2024-08-17T18:27:02.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063385020000000002 2024-08-17T18:27:02.982 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:02.982 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:27:02.982 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:02.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.982 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:171698 58.1 58:80980ada:::benchmark_data_smithi136_442124_object58082:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1741)", 2024-08-17T18:27:02.982 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:23:59.720330+0000", 2024-08-17T18:27:02.982 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 183.232393629, 2024-08-17T18:27:02.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.262030443, 2024-08-17T18:27:02.982 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:27:02.982 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:27:02.982 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:27:02.982 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:27:02.983 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:27:02.983 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 171698 2024-08-17T18:27:02.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.983 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:27:02.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:27:02.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.720330+0000", 2024-08-17T18:27:02.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:02.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:27:02.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.720330+0000", 2024-08-17T18:27:02.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:02.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:27:02.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.720330+0000", 2024-08-17T18:27:02.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:27:02.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:27:02.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.720334+0000", 2024-08-17T18:27:02.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2479999999999998e-06 2024-08-17T18:27:02.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:27:02.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.720339+0000", 2024-08-17T18:27:02.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.814e-06 2024-08-17T18:27:02.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:27:02.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.720343+0000", 2024-08-17T18:27:02.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5410000000000001e-06 2024-08-17T18:27:02.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:02.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.720438+0000", 2024-08-17T18:27:02.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5383000000000005e-05 2024-08-17T18:27:02.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:27:02.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.720446+0000", 2024-08-17T18:27:02.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.571e-06 2024-08-17T18:27:02.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:02.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.722294+0000", 2024-08-17T18:27:02.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00184878 2024-08-17T18:27:02.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:27:02.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.722303+0000", 2024-08-17T18:27:02.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9749999999999996e-06 2024-08-17T18:27:02.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:02.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.944462+0000", 2024-08-17T18:27:02.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.222158302 2024-08-17T18:27:02.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:27:02.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.948512+0000", 2024-08-17T18:27:02.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040505130000000004 2024-08-17T18:27:02.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1,7", 2024-08-17T18:27:02.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.954350+0000", 2024-08-17T18:27:02.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058372880000000004 2024-08-17T18:27:02.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:02.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.973403+0000", 2024-08-17T18:27:02.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019053701999999999 2024-08-17T18:27:02.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:02.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.974192+0000", 2024-08-17T18:27:02.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00078906099999999997 2024-08-17T18:27:02.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:27:02.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.974448+0000", 2024-08-17T18:27:02.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025604799999999999 2024-08-17T18:27:02.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:02.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.975586+0000", 2024-08-17T18:27:02.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001137349 2024-08-17T18:27:02.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:27:02.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.976972+0000", 2024-08-17T18:27:02.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013866569999999999 2024-08-17T18:27:02.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:27:02.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.982361+0000", 2024-08-17T18:27:02.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0053885399999999998 2024-08-17T18:27:02.991 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:02.991 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:27:02.991 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:02.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.991 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:171738 58.1 58:8e26e2af:::benchmark_data_smithi136_442124_object58122:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1741)", 2024-08-17T18:27:02.991 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:23:59.720362+0000", 2024-08-17T18:27:02.991 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 183.23236157700001, 2024-08-17T18:27:02.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2724856170000001, 2024-08-17T18:27:02.991 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:27:02.991 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:27:02.991 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:27:02.992 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:27:02.992 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:27:02.992 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 171738 2024-08-17T18:27:02.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.992 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:27:02.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:27:02.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.720362+0000", 2024-08-17T18:27:02.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:02.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:27:02.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.720362+0000", 2024-08-17T18:27:02.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:02.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:27:02.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.720362+0000", 2024-08-17T18:27:02.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-08-17T18:27:02.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:27:02.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.720364+0000", 2024-08-17T18:27:02.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7799999999999999e-06 2024-08-17T18:27:02.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:27:02.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.720373+0000", 2024-08-17T18:27:02.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7430000000000008e-06 2024-08-17T18:27:02.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:27:02.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.720377+0000", 2024-08-17T18:27:02.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.817e-06 2024-08-17T18:27:02.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:02.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.720477+0000", 2024-08-17T18:27:02.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010045200000000001 2024-08-17T18:27:02.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-08-17T18:27:02.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.720481+0000", 2024-08-17T18:27:02.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0729999999999998e-06 2024-08-17T18:27:02.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:02.995 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.722327+0000", 2024-08-17T18:27:02.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018455889999999999 2024-08-17T18:27:02.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.995 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-08-17T18:27:02.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:23:59.722335+0000", 2024-08-17T18:27:02.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8720000000000002e-06 2024-08-17T18:27:02.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:02.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.959608+0000", 2024-08-17T18:27:02.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2372733849999999 2024-08-17T18:27:02.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:27:02.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.962761+0000", 2024-08-17T18:27:02.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031531380000000002 2024-08-17T18:27:02.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1,7", 2024-08-17T18:27:02.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.968032+0000", 2024-08-17T18:27:02.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052713289999999999 2024-08-17T18:27:02.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:27:02.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.974701+0000", 2024-08-17T18:27:02.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006668513 2024-08-17T18:27:02.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:02.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.983388+0000", 2024-08-17T18:27:02.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0086868169999999995 2024-08-17T18:27:02.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:02.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.984270+0000", 2024-08-17T18:27:02.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00088250000000000004 2024-08-17T18:27:02.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:02.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.985255+0000", 2024-08-17T18:27:02.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0009843059999999999 2024-08-17T18:27:02.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:27:02.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.986905+0000", 2024-08-17T18:27:02.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016507379999999999 2024-08-17T18:27:02.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:27:02.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.992848+0000", 2024-08-17T18:27:02.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059428579999999996 2024-08-17T18:27:02.999 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:02.999 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:27:02.999 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:02.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:02.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:02.999 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:171746 58.1 58:8252aa4a:::benchmark_data_smithi136_442124_object58130:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1742)", 2024-08-17T18:27:02.999 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:24:00.890471+0000", 2024-08-17T18:27:02.999 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 182.062252656, 2024-08-17T18:27:02.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1429057540000001, 2024-08-17T18:27:02.999 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:27:03.000 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:27:03.000 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:27:03.000 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:27:03.000 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:27:03.000 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 171746 2024-08-17T18:27:03.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.000 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:27:03.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:27:03.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.890471+0000", 2024-08-17T18:27:03.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:27:03.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.890471+0000", 2024-08-17T18:27:03.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:27:03.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.890469+0000", 2024-08-17T18:27:03.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:27:03.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:27:03.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.890477+0000", 2024-08-17T18:27:03.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6680000000000001e-06 2024-08-17T18:27:03.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:27:03.002 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.890490+0000", 2024-08-17T18:27:03.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3040000000000001e-05 2024-08-17T18:27:03.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:27:03.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.890497+0000", 2024-08-17T18:27:03.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0700000000000001e-06 2024-08-17T18:27:03.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:03.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.993176+0000", 2024-08-17T18:27:03.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1026785960000001 2024-08-17T18:27:03.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:27:03.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.996656+0000", 2024-08-17T18:27:03.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034801490000000001 2024-08-17T18:27:03.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1,7", 2024-08-17T18:27:03.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.002377+0000", 2024-08-17T18:27:03.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057211959999999996 2024-08-17T18:27:03.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:27:03.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.022291+0000", 2024-08-17T18:27:03.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019913365999999998 2024-08-17T18:27:03.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.023132+0000", 2024-08-17T18:27:03.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00084108700000000002 2024-08-17T18:27:03.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.024224+0000", 2024-08-17T18:27:03.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010928089999999999 2024-08-17T18:27:03.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.025212+0000", 2024-08-17T18:27:03.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00098729700000000004 2024-08-17T18:27:03.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:27:03.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.026824+0000", 2024-08-17T18:27:03.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001612275 2024-08-17T18:27:03.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:27:03.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.033377+0000", 2024-08-17T18:27:03.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0065531230000000001 2024-08-17T18:27:03.007 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.007 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:27:03.007 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.007 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:171750 58.9 e1742/1734 58:9610e2b3:::benchmark_data_smithi136_442124_object58134:head v 1742'11071, mlcod=0'0)", 2024-08-17T18:27:03.007 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:24:00.920847+0000", 2024-08-17T18:27:03.007 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 182.03187664699999, 2024-08-17T18:27:03.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0830032199999999, 2024-08-17T18:27:03.007 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:27:03.008 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:27:03.008 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:27:03.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:27:03.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.920847+0000", 2024-08-17T18:27:03.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:27:03.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.920847+0000", 2024-08-17T18:27:03.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:27:03.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.920846+0000", 2024-08-17T18:27:03.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:27:03.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:27:03.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.920855+0000", 2024-08-17T18:27:03.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9430000000000006e-06 2024-08-17T18:27:03.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:27:03.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.920869+0000", 2024-08-17T18:27:03.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3631e-05 2024-08-17T18:27:03.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:27:03.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.920877+0000", 2024-08-17T18:27:03.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7729999999999999e-06 2024-08-17T18:27:03.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:03.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:01.996867+0000", 2024-08-17T18:27:03.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0759899319999999 2024-08-17T18:27:03.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:27:03.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:01.996889+0000", 2024-08-17T18:27:03.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2450999999999999e-05 2024-08-17T18:27:03.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:27:03.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.003803+0000", 2024-08-17T18:27:03.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069135469999999999 2024-08-17T18:27:03.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:27:03.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.003851+0000", 2024-08-17T18:27:03.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7954000000000002e-05 2024-08-17T18:27:03.012 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.012 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:27:03.012 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.012 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:171762 58.1 58:89c9c57c:::benchmark_data_smithi136_442124_object58146:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1742)", 2024-08-17T18:27:03.012 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:24:00.932746+0000", 2024-08-17T18:27:03.012 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 182.01997813700001, 2024-08-17T18:27:03.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.110385119, 2024-08-17T18:27:03.012 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:27:03.012 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:27:03.012 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:27:03.013 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:27:03.013 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:27:03.013 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 171762 2024-08-17T18:27:03.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.013 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:27:03.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:27:03.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.932746+0000", 2024-08-17T18:27:03.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:27:03.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.932746+0000", 2024-08-17T18:27:03.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:27:03.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.932745+0000", 2024-08-17T18:27:03.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:27:03.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:27:03.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.932752+0000", 2024-08-17T18:27:03.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5360000000000004e-06 2024-08-17T18:27:03.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:27:03.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.932766+0000", 2024-08-17T18:27:03.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3828999999999999e-05 2024-08-17T18:27:03.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:27:03.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.932774+0000", 2024-08-17T18:27:03.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2609999999999993e-06 2024-08-17T18:27:03.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.015 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:03.015 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.008486+0000", 2024-08-17T18:27:03.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0757119720000001 2024-08-17T18:27:03.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:27:03.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.011633+0000", 2024-08-17T18:27:03.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00314636 2024-08-17T18:27:03.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1,7", 2024-08-17T18:27:03.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.016465+0000", 2024-08-17T18:27:03.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048320380000000003 2024-08-17T18:27:03.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:27:03.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.033784+0000", 2024-08-17T18:27:03.016 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017319511999999999 2024-08-17T18:27:03.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.034784+0000", 2024-08-17T18:27:03.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00099944399999999994 2024-08-17T18:27:03.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.035576+0000", 2024-08-17T18:27:03.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00079233400000000001 2024-08-17T18:27:03.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.036371+0000", 2024-08-17T18:27:03.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00079477699999999998 2024-08-17T18:27:03.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:27:03.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.037714+0000", 2024-08-17T18:27:03.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013435249999999999 2024-08-17T18:27:03.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:27:03.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.043131+0000", 2024-08-17T18:27:03.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005416906 2024-08-17T18:27:03.018 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.019 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:27:03.019 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.019 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:171770 58.1 58:8765f0b8:::benchmark_data_smithi136_442124_object58154:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1742)", 2024-08-17T18:27:03.019 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:24:00.944517+0000", 2024-08-17T18:27:03.019 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 182.00820678100001, 2024-08-17T18:27:03.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.138031813, 2024-08-17T18:27:03.019 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:27:03.019 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:27:03.019 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:27:03.019 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:27:03.019 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:27:03.020 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 171770 2024-08-17T18:27:03.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.020 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:27:03.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:27:03.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.944517+0000", 2024-08-17T18:27:03.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:27:03.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.944517+0000", 2024-08-17T18:27:03.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:27:03.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.944515+0000", 2024-08-17T18:27:03.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:27:03.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:27:03.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.944521+0000", 2024-08-17T18:27:03.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7810000000000002e-06 2024-08-17T18:27:03.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:27:03.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.944545+0000", 2024-08-17T18:27:03.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3923000000000001e-05 2024-08-17T18:27:03.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:27:03.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.944552+0000", 2024-08-17T18:27:03.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4439999999999996e-06 2024-08-17T18:27:03.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:03.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.043444+0000", 2024-08-17T18:27:03.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0988922639999998 2024-08-17T18:27:03.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:27:03.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.046689+0000", 2024-08-17T18:27:03.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032450399999999998 2024-08-17T18:27:03.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1,7", 2024-08-17T18:27:03.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.051542+0000", 2024-08-17T18:27:03.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004853525 2024-08-17T18:27:03.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.023 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:27:03.023 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.072788+0000", 2024-08-17T18:27:03.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021245288000000001 2024-08-17T18:27:03.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.073805+0000", 2024-08-17T18:27:03.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010175060000000001 2024-08-17T18:27:03.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.074653+0000", 2024-08-17T18:27:03.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00084750199999999998 2024-08-17T18:27:03.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.075643+0000", 2024-08-17T18:27:03.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00099075000000000009 2024-08-17T18:27:03.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:27:03.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.077107+0000", 2024-08-17T18:27:03.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001463336 2024-08-17T18:27:03.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:27:03.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.082549+0000", 2024-08-17T18:27:03.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054422999999999997 2024-08-17T18:27:03.025 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.025 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:27:03.026 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.026 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:171793 58.1 58:8981e51b:::benchmark_data_smithi136_442124_object58177:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1742)", 2024-08-17T18:27:03.026 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:24:00.979380+0000", 2024-08-17T18:27:03.026 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 181.97334420799999, 2024-08-17T18:27:03.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1133498259999999, 2024-08-17T18:27:03.026 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:27:03.026 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:27:03.026 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:27:03.026 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:27:03.026 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:27:03.026 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 171793 2024-08-17T18:27:03.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.027 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:27:03.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:27:03.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.979380+0000", 2024-08-17T18:27:03.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:27:03.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.979380+0000", 2024-08-17T18:27:03.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:27:03.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.979378+0000", 2024-08-17T18:27:03.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:27:03.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:27:03.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.979388+0000", 2024-08-17T18:27:03.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4800000000000007e-06 2024-08-17T18:27:03.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:27:03.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.979401+0000", 2024-08-17T18:27:03.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2853999999999999e-05 2024-08-17T18:27:03.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:27:03.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.979408+0000", 2024-08-17T18:27:03.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7400000000000004e-06 2024-08-17T18:27:03.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:03.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.057885+0000", 2024-08-17T18:27:03.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0784770319999999 2024-08-17T18:27:03.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:27:03.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.061097+0000", 2024-08-17T18:27:03.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032119570000000001 2024-08-17T18:27:03.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1,7", 2024-08-17T18:27:03.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.066040+0000", 2024-08-17T18:27:03.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049425579999999997 2024-08-17T18:27:03.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:27:03.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.082872+0000", 2024-08-17T18:27:03.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016831641000000001 2024-08-17T18:27:03.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.083691+0000", 2024-08-17T18:27:03.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00081966999999999999 2024-08-17T18:27:03.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.084496+0000", 2024-08-17T18:27:03.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000804721 2024-08-17T18:27:03.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.085387+0000", 2024-08-17T18:27:03.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00089143499999999999 2024-08-17T18:27:03.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:27:03.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.086748+0000", 2024-08-17T18:27:03.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013600960000000001 2024-08-17T18:27:03.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:27:03.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.092730+0000", 2024-08-17T18:27:03.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059821789999999998 2024-08-17T18:27:03.032 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.032 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:27:03.032 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.032 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:171782 58.9 e1742/1734 58:9855f0a2:::benchmark_data_smithi136_442124_object58166:head v 1742'11072, mlcod=0'0)", 2024-08-17T18:27:03.032 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:24:00.986224+0000", 2024-08-17T18:27:03.033 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 181.96649995600001, 2024-08-17T18:27:03.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0176426620000001, 2024-08-17T18:27:03.033 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:27:03.033 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:27:03.033 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:27:03.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:27:03.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.986224+0000", 2024-08-17T18:27:03.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.033 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:27:03.033 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.986224+0000", 2024-08-17T18:27:03.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:27:03.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.986223+0000", 2024-08-17T18:27:03.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:27:03.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:27:03.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.986234+0000", 2024-08-17T18:27:03.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1542999999999999e-05 2024-08-17T18:27:03.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:27:03.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.986251+0000", 2024-08-17T18:27:03.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6328e-05 2024-08-17T18:27:03.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:27:03.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.986257+0000", 2024-08-17T18:27:03.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.19e-06 2024-08-17T18:27:03.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:03.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:01.999028+0000", 2024-08-17T18:27:03.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0127710729999999 2024-08-17T18:27:03.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:27:03.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:01.999049+0000", 2024-08-17T18:27:03.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1279000000000001e-05 2024-08-17T18:27:03.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:27:03.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.003853+0000", 2024-08-17T18:27:03.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048040089999999997 2024-08-17T18:27:03.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:27:03.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.003867+0000", 2024-08-17T18:27:03.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3465000000000001e-05 2024-08-17T18:27:03.037 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.037 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:27:03.037 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.037 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:171795 58.9 e1742/1734 58:99c7f6f0:::benchmark_data_smithi136_442124_object58179:head v 1742'11073, mlcod=0'0)", 2024-08-17T18:27:03.037 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:24:00.998265+0000", 2024-08-17T18:27:03.037 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 181.954459324, 2024-08-17T18:27:03.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.005616719, 2024-08-17T18:27:03.037 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:27:03.037 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:27:03.037 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:27:03.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:27:03.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.998265+0000", 2024-08-17T18:27:03.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:27:03.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.998265+0000", 2024-08-17T18:27:03.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:27:03.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.998264+0000", 2024-08-17T18:27:03.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:27:03.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:27:03.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.998274+0000", 2024-08-17T18:27:03.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.013e-05 2024-08-17T18:27:03.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:27:03.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.998288+0000", 2024-08-17T18:27:03.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.375e-05 2024-08-17T18:27:03.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:27:03.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:00.998295+0000", 2024-08-17T18:27:03.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.922e-06 2024-08-17T18:27:03.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:03.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.000446+0000", 2024-08-17T18:27:03.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0021514010000001 2024-08-17T18:27:03.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:27:03.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.000467+0000", 2024-08-17T18:27:03.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0655000000000001e-05 2024-08-17T18:27:03.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:27:03.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.003869+0000", 2024-08-17T18:27:03.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034019860000000001 2024-08-17T18:27:03.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:27:03.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.003881+0000", 2024-08-17T18:27:03.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2867000000000001e-05 2024-08-17T18:27:03.041 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.042 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:27:03.042 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.042 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:171801 58.9 e1742/1734 58:9ce83bef:::benchmark_data_smithi136_442124_object58185:head v 1742'11074, mlcod=0'0)", 2024-08-17T18:27:03.042 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:24:01.005961+0000", 2024-08-17T18:27:03.042 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 181.946762976, 2024-08-17T18:27:03.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99988991599999999, 2024-08-17T18:27:03.042 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:27:03.043 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:27:03.043 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:27:03.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:27:03.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:01.005961+0000", 2024-08-17T18:27:03.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:27:03.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:01.005961+0000", 2024-08-17T18:27:03.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:27:03.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:01.005960+0000", 2024-08-17T18:27:03.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:27:03.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:27:03.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:01.005967+0000", 2024-08-17T18:27:03.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9800000000000001e-06 2024-08-17T18:27:03.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:27:03.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:01.005978+0000", 2024-08-17T18:27:03.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0557000000000001e-05 2024-08-17T18:27:03.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:27:03.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:01.005985+0000", 2024-08-17T18:27:03.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.165e-06 2024-08-17T18:27:03.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:03.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.001806+0000", 2024-08-17T18:27:03.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99582139400000003 2024-08-17T18:27:03.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:27:03.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.001826+0000", 2024-08-17T18:27:03.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0149e-05 2024-08-17T18:27:03.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:27:03.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.005827+0000", 2024-08-17T18:27:03.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040004410000000004 2024-08-17T18:27:03.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:27:03.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.005851+0000", 2024-08-17T18:27:03.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4159999999999999e-05 2024-08-17T18:27:03.046 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.046 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:27:03.046 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.047 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:171816 58.1 58:8d3d9e64:::benchmark_data_smithi136_442124_object58200:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1742)", 2024-08-17T18:27:03.047 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:24:01.013632+0000", 2024-08-17T18:27:03.047 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 181.93909198700001, 2024-08-17T18:27:03.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.117124397, 2024-08-17T18:27:03.047 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:27:03.047 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:27:03.047 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:27:03.048 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:27:03.048 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:27:03.048 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 171816 2024-08-17T18:27:03.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.048 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:27:03.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:27:03.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:01.013632+0000", 2024-08-17T18:27:03.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:27:03.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:01.013632+0000", 2024-08-17T18:27:03.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:27:03.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:01.013631+0000", 2024-08-17T18:27:03.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:27:03.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:27:03.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:01.013638+0000", 2024-08-17T18:27:03.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1289999999999998e-06 2024-08-17T18:27:03.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:27:03.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:01.013649+0000", 2024-08-17T18:27:03.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1600000000000001e-05 2024-08-17T18:27:03.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:27:03.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:01.013657+0000", 2024-08-17T18:27:03.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8779999999999998e-06 2024-08-17T18:27:03.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:03.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.093038+0000", 2024-08-17T18:27:03.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0793813349999999 2024-08-17T18:27:03.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:27:03.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.096585+0000", 2024-08-17T18:27:03.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003546764 2024-08-17T18:27:03.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1,7", 2024-08-17T18:27:03.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.101826+0000", 2024-08-17T18:27:03.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052407139999999996 2024-08-17T18:27:03.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:27:03.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.121001+0000", 2024-08-17T18:27:03.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019174667999999999 2024-08-17T18:27:03.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.121775+0000", 2024-08-17T18:27:03.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00077450499999999999 2024-08-17T18:27:03.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.122784+0000", 2024-08-17T18:27:03.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001009268 2024-08-17T18:27:03.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.123743+0000", 2024-08-17T18:27:03.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00095895099999999999 2024-08-17T18:27:03.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:27:03.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.125382+0000", 2024-08-17T18:27:03.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001638631 2024-08-17T18:27:03.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:27:03.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.130756+0000", 2024-08-17T18:27:03.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005374495 2024-08-17T18:27:03.053 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.054 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:27:03.054 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.054 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.36936.0:171808 58.9 e1742/1734 58:9654aa1f:::benchmark_data_smithi136_442124_object58192:head v 1742'11075, mlcod=0'0)", 2024-08-17T18:27:03.054 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:24:01.018526+0000", 2024-08-17T18:27:03.054 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 181.93419801300001, 2024-08-17T18:27:03.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.988502667, 2024-08-17T18:27:03.054 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:27:03.054 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:27:03.054 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:27:03.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:27:03.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:01.018526+0000", 2024-08-17T18:27:03.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:27:03.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:01.018526+0000", 2024-08-17T18:27:03.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:27:03.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:01.018525+0000", 2024-08-17T18:27:03.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-08-17T18:27:03.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:27:03.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:01.018534+0000", 2024-08-17T18:27:03.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2709999999999998e-06 2024-08-17T18:27:03.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:27:03.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:01.018548+0000", 2024-08-17T18:27:03.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.345e-05 2024-08-17T18:27:03.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:27:03.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:01.018554+0000", 2024-08-17T18:27:03.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0209999999999996e-06 2024-08-17T18:27:03.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:03.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.002999+0000", 2024-08-17T18:27:03.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98444536199999999 2024-08-17T18:27:03.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:27:03.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.003011+0000", 2024-08-17T18:27:03.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1664e-05 2024-08-17T18:27:03.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:27:03.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.007005+0000", 2024-08-17T18:27:03.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039942470000000002 2024-08-17T18:27:03.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:27:03.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.007029+0000", 2024-08-17T18:27:03.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3634000000000001e-05 2024-08-17T18:27:03.058 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.058 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:27:03.058 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.058 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:171834 58.1 58:8a943b8b:::benchmark_data_smithi136_442124_object58218:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1743)", 2024-08-17T18:27:03.058 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:24:02.076198+0000", 2024-08-17T18:27:03.058 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 180.87652586600001, 2024-08-17T18:27:03.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0647847260000001, 2024-08-17T18:27:03.059 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:27:03.059 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:27:03.059 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:27:03.059 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:27:03.059 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:27:03.059 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 171834 2024-08-17T18:27:03.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.059 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:27:03.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:27:03.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.076198+0000", 2024-08-17T18:27:03.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:27:03.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.076198+0000", 2024-08-17T18:27:03.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:27:03.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.076197+0000", 2024-08-17T18:27:03.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:27:03.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:27:03.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.076205+0000", 2024-08-17T18:27:03.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8499999999999994e-06 2024-08-17T18:27:03.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:27:03.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.076217+0000", 2024-08-17T18:27:03.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2901999999999999e-05 2024-08-17T18:27:03.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:27:03.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.076228+0000", 2024-08-17T18:27:03.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0902e-05 2024-08-17T18:27:03.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:03.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.107869+0000", 2024-08-17T18:27:03.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0316401580000001 2024-08-17T18:27:03.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:27:03.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.111752+0000", 2024-08-17T18:27:03.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038838039999999998 2024-08-17T18:27:03.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1,7", 2024-08-17T18:27:03.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.116281+0000", 2024-08-17T18:27:03.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004528242 2024-08-17T18:27:03.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:27:03.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.131093+0000", 2024-08-17T18:27:03.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014812120999999999 2024-08-17T18:27:03.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.132063+0000", 2024-08-17T18:27:03.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00097010399999999995 2024-08-17T18:27:03.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.132853+0000", 2024-08-17T18:27:03.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00079033700000000001 2024-08-17T18:27:03.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.133661+0000", 2024-08-17T18:27:03.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00080798700000000003 2024-08-17T18:27:03.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:27:03.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.134935+0000", 2024-08-17T18:27:03.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012736430000000001 2024-08-17T18:27:03.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:27:03.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.140983+0000", 2024-08-17T18:27:03.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060481859999999997 2024-08-17T18:27:03.065 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.065 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:27:03.065 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.065 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:171846 58.1 58:8a79b991:::benchmark_data_smithi136_442124_object58230:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1743)", 2024-08-17T18:27:03.065 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:24:02.091823+0000", 2024-08-17T18:27:03.065 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 180.860901216, 2024-08-17T18:27:03.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.08653461, 2024-08-17T18:27:03.065 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:27:03.065 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:27:03.066 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:27:03.066 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:27:03.066 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:27:03.066 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 171846 2024-08-17T18:27:03.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.066 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:27:03.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:27:03.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.091823+0000", 2024-08-17T18:27:03.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:27:03.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.091823+0000", 2024-08-17T18:27:03.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:27:03.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.091821+0000", 2024-08-17T18:27:03.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:27:03.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:27:03.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.091828+0000", 2024-08-17T18:27:03.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2110000000000003e-06 2024-08-17T18:27:03.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:27:03.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.091841+0000", 2024-08-17T18:27:03.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3028e-05 2024-08-17T18:27:03.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:27:03.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.091847+0000", 2024-08-17T18:27:03.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4960000000000001e-06 2024-08-17T18:27:03.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:03.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.141396+0000", 2024-08-17T18:27:03.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.049549348 2024-08-17T18:27:03.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:27:03.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.145163+0000", 2024-08-17T18:27:03.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037665089999999999 2024-08-17T18:27:03.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1,7", 2024-08-17T18:27:03.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.149928+0000", 2024-08-17T18:27:03.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0047651509999999996 2024-08-17T18:27:03.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.169805+0000", 2024-08-17T18:27:03.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019877260000000001 2024-08-17T18:27:03.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:27:03.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.170109+0000", 2024-08-17T18:27:03.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00030338099999999997 2024-08-17T18:27:03.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.170877+0000", 2024-08-17T18:27:03.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00076786900000000002 2024-08-17T18:27:03.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.171717+0000", 2024-08-17T18:27:03.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00084006900000000004 2024-08-17T18:27:03.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:27:03.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.173036+0000", 2024-08-17T18:27:03.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001319328 2024-08-17T18:27:03.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:27:03.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.178357+0000", 2024-08-17T18:27:03.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00532148 2024-08-17T18:27:03.072 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.072 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:27:03.072 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.072 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:171884 58.1 58:8d454228:::benchmark_data_smithi136_442124_object58268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1743)", 2024-08-17T18:27:03.072 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:24:02.181341+0000", 2024-08-17T18:27:03.072 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 180.77138292999999, 2024-08-17T18:27:03.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0069735639999999, 2024-08-17T18:27:03.072 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:27:03.072 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:27:03.072 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:27:03.072 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:27:03.073 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:27:03.073 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 171884 2024-08-17T18:27:03.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.073 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:27:03.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:27:03.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.181341+0000", 2024-08-17T18:27:03.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:27:03.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.181341+0000", 2024-08-17T18:27:03.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:27:03.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.181339+0000", 2024-08-17T18:27:03.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-08-17T18:27:03.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:27:03.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.181348+0000", 2024-08-17T18:27:03.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2989999999999997e-06 2024-08-17T18:27:03.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:27:03.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.181365+0000", 2024-08-17T18:27:03.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7155000000000001e-05 2024-08-17T18:27:03.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:27:03.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.181373+0000", 2024-08-17T18:27:03.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7659999999999998e-06 2024-08-17T18:27:03.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:03.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.155626+0000", 2024-08-17T18:27:03.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.97425358799999995 2024-08-17T18:27:03.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:27:03.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.159024+0000", 2024-08-17T18:27:03.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033974249999999999 2024-08-17T18:27:03.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1,7", 2024-08-17T18:27:03.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.163983+0000", 2024-08-17T18:27:03.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049590240000000002 2024-08-17T18:27:03.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.179211+0000", 2024-08-17T18:27:03.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015228702 2024-08-17T18:27:03.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:27:03.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.179498+0000", 2024-08-17T18:27:03.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028612599999999999 2024-08-17T18:27:03.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.180318+0000", 2024-08-17T18:27:03.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00082095499999999995 2024-08-17T18:27:03.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.181317+0000", 2024-08-17T18:27:03.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00099856199999999993 2024-08-17T18:27:03.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:27:03.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.182895+0000", 2024-08-17T18:27:03.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015781599999999999 2024-08-17T18:27:03.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:27:03.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.188315+0000", 2024-08-17T18:27:03.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054195199999999997 2024-08-17T18:27:03.078 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.078 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:27:03.078 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.079 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.36936.0:171888 58.1 58:84f041f0:::benchmark_data_smithi136_442124_object58272:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1743)", 2024-08-17T18:27:03.079 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-08-17T18:24:02.196117+0000", 2024-08-17T18:27:03.079 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 180.75660664899999, 2024-08-17T18:27:03.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.033160187, 2024-08-17T18:27:03.079 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-08-17T18:27:03.079 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T18:27:03.079 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-08-17T18:27:03.079 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.36936", 2024-08-17T18:27:03.079 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "172.21.15.136:0/3233908197", 2024-08-17T18:27:03.079 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 171888 2024-08-17T18:27:03.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.079 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-08-17T18:27:03.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-08-17T18:27:03.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.196117+0000", 2024-08-17T18:27:03.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-08-17T18:27:03.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.196117+0000", 2024-08-17T18:27:03.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-08-17T18:27:03.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-08-17T18:27:03.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.196116+0000", 2024-08-17T18:27:03.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999986 2024-08-17T18:27:03.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-08-17T18:27:03.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.196122+0000", 2024-08-17T18:27:03.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1759999999999998e-06 2024-08-17T18:27:03.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-08-17T18:27:03.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.196134+0000", 2024-08-17T18:27:03.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.146e-05 2024-08-17T18:27:03.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-08-17T18:27:03.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:02.196141+0000", 2024-08-17T18:27:03.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7819999999999996e-06 2024-08-17T18:27:03.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-08-17T18:27:03.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.188676+0000", 2024-08-17T18:27:03.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.99253566299999996 2024-08-17T18:27:03.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-08-17T18:27:03.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.192386+0000", 2024-08-17T18:27:03.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003709356 2024-08-17T18:27:03.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1,7", 2024-08-17T18:27:03.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.198231+0000", 2024-08-17T18:27:03.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058450990000000003 2024-08-17T18:27:03.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.219285+0000", 2024-08-17T18:27:03.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021054086999999999 2024-08-17T18:27:03.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.220328+0000", 2024-08-17T18:27:03.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010433770000000001 2024-08-17T18:27:03.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-08-17T18:27:03.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.220659+0000", 2024-08-17T18:27:03.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000330713 2024-08-17T18:27:03.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-08-17T18:27:03.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.221908+0000", 2024-08-17T18:27:03.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012489910000000001 2024-08-17T18:27:03.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-08-17T18:27:03.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.223532+0000", 2024-08-17T18:27:03.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001623614 2024-08-17T18:27:03.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-08-17T18:27:03.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-08-17T18:27:03.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-08-17T18:27:03.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-08-17T18:24:03.229278+0000", 2024-08-17T18:27:03.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057460790000000003 2024-08-17T18:27:03.085 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.085 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:27:03.085 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.085 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-08-17T18:27:03.085 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-08-17T18:27:03.085 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-08-17T18:27:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:27:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:27:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:03 smithi165 ceph-mon[105517]: pgmap v3565: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:27:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:27:03.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:27:03.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:27:03.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:27:03.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:03 smithi136 ceph-mon[109861]: pgmap v3565: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:03.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:27:03.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:27:03.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:27:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:27:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:27:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:03 smithi136 ceph-mon[115078]: pgmap v3565: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:27:03.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:27:03.948 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:27:06.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:05 smithi165 ceph-mon[105517]: pgmap v3566: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:06.025 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:05 smithi136 ceph-mon[109861]: pgmap v3566: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:06.025 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:05 smithi136 ceph-mon[115078]: pgmap v3566: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:06.976 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:27:06.976 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:27:07.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:27:07.039 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-08-17T18:27:07.245 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:27:07.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:27:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:27:07] "GET /metrics HTTP/1.1" 200 34256 "" "Prometheus/2.51.0" 2024-08-17T18:27:08.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:07 smithi165 ceph-mon[105517]: pgmap v3567: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:08.165 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:07 smithi136 ceph-mon[109861]: pgmap v3567: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:08.166 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:07 smithi136 ceph-mon[115078]: pgmap v3567: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:09.076 INFO:teuthology.orchestra.run.smithi136.stdout:mon.a: {} 2024-08-17T18:27:09.076 INFO:teuthology.orchestra.run.smithi136.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-17T18:27:09.086 INFO:teuthology.orchestra.run.smithi136.stdout:mon.b: {} 2024-08-17T18:27:09.086 INFO:teuthology.orchestra.run.smithi136.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-17T18:27:09.095 INFO:teuthology.orchestra.run.smithi136.stdout:mon.c: {} 2024-08-17T18:27:09.095 INFO:teuthology.orchestra.run.smithi136.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-17T18:27:09.714 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 0 1 2024-08-17T18:27:09.919 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:27:10.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:09 smithi165 ceph-mon[105517]: pgmap v3568: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:10.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:09 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/647322717' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-17T18:27:10.041 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:09 smithi136 ceph-mon[109861]: pgmap v3568: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:10.041 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:09 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/647322717' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-17T18:27:10.041 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:09 smithi136 ceph-mon[115078]: pgmap v3568: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:10.041 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:09 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/647322717' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-17T18:27:12.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:11 smithi165 ceph-mon[105517]: pgmap v3569: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:12.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:11 smithi136 ceph-mon[109861]: pgmap v3569: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:12.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:11 smithi136 ceph-mon[115078]: pgmap v3569: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:12.727 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.0 to 1 (10000) 2024-08-17T18:27:13.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:12 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/684302858' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T18:27:13.082 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:12 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/684302858' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T18:27:13.082 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:12 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/684302858' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T18:27:13.385 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 0 1 2024-08-17T18:27:13.595 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:27:14.002 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:13 smithi136 ceph-mon[109861]: pgmap v3570: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:14.002 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:13 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/684302858' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T18:27:14.002 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:13 smithi136 ceph-mon[109861]: osdmap e1779: 8 total, 8 up, 8 in 2024-08-17T18:27:14.003 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:13 smithi136 ceph-mon[115078]: pgmap v3570: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:14.003 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:13 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/684302858' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T18:27:14.003 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:13 smithi136 ceph-mon[115078]: osdmap e1779: 8 total, 8 up, 8 in 2024-08-17T18:27:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:13 smithi165 ceph-mon[105517]: pgmap v3570: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:13 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/684302858' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T18:27:14.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:13 smithi165 ceph-mon[105517]: osdmap e1779: 8 total, 8 up, 8 in 2024-08-17T18:27:15.731 INFO:teuthology.orchestra.run.smithi136.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-08-17T18:27:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:15 smithi165 ceph-mon[105517]: pgmap v3572: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:16.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:15 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3585120916' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-08-17T18:27:16.046 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:15 smithi136 ceph-mon[109861]: pgmap v3572: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:16.047 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:15 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3585120916' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-08-17T18:27:16.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:15 smithi136 ceph-mon[115078]: pgmap v3572: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:16.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:15 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3585120916' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-08-17T18:27:16.480 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 1 1 2024-08-17T18:27:16.687 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:27:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:16 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3585120916' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-08-17T18:27:17.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:16 smithi165 ceph-mon[105517]: osdmap e1780: 8 total, 8 up, 8 in 2024-08-17T18:27:17.038 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:16 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3585120916' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-08-17T18:27:17.038 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:16 smithi136 ceph-mon[109861]: osdmap e1780: 8 total, 8 up, 8 in 2024-08-17T18:27:17.039 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:16 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3585120916' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-08-17T18:27:17.039 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:16 smithi136 ceph-mon[115078]: osdmap e1780: 8 total, 8 up, 8 in 2024-08-17T18:27:17.742 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:27:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:27:17] "GET /metrics HTTP/1.1" 200 34256 "" "Prometheus/2.51.0" 2024-08-17T18:27:18.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:17 smithi165 ceph-mon[105517]: pgmap v3574: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:18.100 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:17 smithi136 ceph-mon[109861]: pgmap v3574: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:18.100 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:17 smithi136 ceph-mon[115078]: pgmap v3574: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:18.748 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.1 to 1 (10000) 2024-08-17T18:27:19.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:18 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2632625133' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T18:27:19.068 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:18 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2632625133' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T18:27:19.068 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:18 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2632625133' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T18:27:19.496 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 1 1 2024-08-17T18:27:19.706 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:27:19.999 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:19 smithi136 ceph-mon[109861]: pgmap v3575: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:19.999 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:19 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2632625133' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T18:27:20.000 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:19 smithi136 ceph-mon[109861]: osdmap e1781: 8 total, 8 up, 8 in 2024-08-17T18:27:20.000 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:19 smithi136 ceph-mon[115078]: pgmap v3575: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:20.000 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:19 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2632625133' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T18:27:20.000 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:19 smithi136 ceph-mon[115078]: osdmap e1781: 8 total, 8 up, 8 in 2024-08-17T18:27:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:19 smithi165 ceph-mon[105517]: pgmap v3575: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:19 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2632625133' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T18:27:20.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:19 smithi165 ceph-mon[105517]: osdmap e1781: 8 total, 8 up, 8 in 2024-08-17T18:27:21.756 INFO:teuthology.orchestra.run.smithi136.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-08-17T18:27:22.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:21 smithi136 ceph-mon[115078]: pgmap v3577: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:22.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:27:22.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:21 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3472982861' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-08-17T18:27:22.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:21 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-08-17T18:27:22.136 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:21 smithi136 ceph-mon[109861]: pgmap v3577: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:22.136 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:27:22.136 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:21 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3472982861' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-08-17T18:27:22.136 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:21 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-08-17T18:27:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:21 smithi165 ceph-mon[105517]: pgmap v3577: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:27:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:21 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3472982861' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-08-17T18:27:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:21 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-08-17T18:27:22.523 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 3 1 2024-08-17T18:27:22.726 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:27:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:22 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-08-17T18:27:23.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:22 smithi165 ceph-mon[105517]: osdmap e1782: 8 total, 8 up, 8 in 2024-08-17T18:27:23.017 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:22 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-08-17T18:27:23.017 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:22 smithi136 ceph-mon[109861]: osdmap e1782: 8 total, 8 up, 8 in 2024-08-17T18:27:23.017 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:22 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-08-17T18:27:23.017 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:22 smithi136 ceph-mon[115078]: osdmap e1782: 8 total, 8 up, 8 in 2024-08-17T18:27:23.869 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:23 smithi136 ceph-mon[109861]: pgmap v3579: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:23.869 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:23 smithi136 ceph-mon[115078]: pgmap v3579: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:24.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:23 smithi165 ceph-mon[105517]: pgmap v3579: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:24.778 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.3 to 1 (10000) 2024-08-17T18:27:25.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:24 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1523578507' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T18:27:25.077 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:24 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T18:27:25.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:24 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1523578507' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T18:27:25.077 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:24 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T18:27:25.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:24 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1523578507' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T18:27:25.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:24 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T18:27:25.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 3 1 2024-08-17T18:27:25.685 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:27:26.086 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:25 smithi136 ceph-mon[109861]: pgmap v3580: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:26.087 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:25 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T18:27:26.087 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:25 smithi136 ceph-mon[109861]: osdmap e1783: 8 total, 8 up, 8 in 2024-08-17T18:27:26.087 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:25 smithi136 ceph-mon[115078]: pgmap v3580: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:26.087 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:25 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T18:27:26.088 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:25 smithi136 ceph-mon[115078]: osdmap e1783: 8 total, 8 up, 8 in 2024-08-17T18:27:26.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:25 smithi165 ceph-mon[105517]: pgmap v3580: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:26.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:25 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T18:27:26.262 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:25 smithi165 ceph-mon[105517]: osdmap e1783: 8 total, 8 up, 8 in 2024-08-17T18:27:27.778 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:27:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:27:27] "GET /metrics HTTP/1.1" 200 34256 "" "Prometheus/2.51.0" 2024-08-17T18:27:28.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:27 smithi136 ceph-mon[109861]: pgmap v3582: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:28.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:27 smithi136 ceph-mon[115078]: pgmap v3582: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:28.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:27 smithi165 ceph-mon[105517]: pgmap v3582: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:28.790 INFO:teuthology.orchestra.run.smithi136.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-08-17T18:27:29.121 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:28 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1999861718' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-08-17T18:27:29.121 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:28 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-08-17T18:27:29.121 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:28 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1999861718' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-08-17T18:27:29.121 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:28 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-08-17T18:27:29.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:28 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1999861718' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-08-17T18:27:29.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:28 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-08-17T18:27:29.567 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 4 1 2024-08-17T18:27:29.774 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:27:29.897 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:29 smithi136 ceph-mon[109861]: pgmap v3583: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:29.897 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:29 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-08-17T18:27:29.897 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:29 smithi136 ceph-mon[109861]: osdmap e1784: 8 total, 8 up, 8 in 2024-08-17T18:27:29.898 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:29 smithi136 ceph-mon[115078]: pgmap v3583: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:29.898 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:29 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-08-17T18:27:29.898 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:29 smithi136 ceph-mon[115078]: osdmap e1784: 8 total, 8 up, 8 in 2024-08-17T18:27:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:29 smithi165 ceph-mon[105517]: pgmap v3583: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:29 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-08-17T18:27:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:29 smithi165 ceph-mon[105517]: osdmap e1784: 8 total, 8 up, 8 in 2024-08-17T18:27:31.787 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.4 to 1 (10000) 2024-08-17T18:27:31.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:31 smithi136 ceph-mon[109861]: pgmap v3585: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:31.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:31 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3084499806' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T18:27:31.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:31 smithi136 ceph-mon[115078]: pgmap v3585: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:31.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:31 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3084499806' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T18:27:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:31 smithi165 ceph-mon[105517]: pgmap v3585: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:31 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3084499806' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T18:27:32.685 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 4 1 2024-08-17T18:27:32.901 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:27:32.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:32 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3084499806' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-17T18:27:32.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:32 smithi136 ceph-mon[109861]: osdmap e1785: 8 total, 8 up, 8 in 2024-08-17T18:27:32.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:32 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3084499806' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-17T18:27:32.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:32 smithi136 ceph-mon[115078]: osdmap e1785: 8 total, 8 up, 8 in 2024-08-17T18:27:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:32 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3084499806' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-17T18:27:33.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:32 smithi165 ceph-mon[105517]: osdmap e1785: 8 total, 8 up, 8 in 2024-08-17T18:27:34.075 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:33 smithi136 ceph-mon[109861]: pgmap v3587: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:34.076 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:33 smithi136 ceph-mon[115078]: pgmap v3587: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:33 smithi165 ceph-mon[105517]: pgmap v3587: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:35.816 INFO:teuthology.orchestra.run.smithi136.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-08-17T18:27:36.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:35 smithi136 ceph-mon[109861]: pgmap v3588: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:36.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:35 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2223830307' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-08-17T18:27:36.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:35 smithi136 ceph-mon[115078]: pgmap v3588: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:36.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:35 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2223830307' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-08-17T18:27:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:35 smithi165 ceph-mon[105517]: pgmap v3588: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:36.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:35 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2223830307' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-08-17T18:27:36.626 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 5 1 2024-08-17T18:27:36.835 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:27:36.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:36 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2223830307' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-08-17T18:27:36.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:36 smithi136 ceph-mon[115078]: osdmap e1786: 8 total, 8 up, 8 in 2024-08-17T18:27:36.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:36 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:27:36.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:36 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2223830307' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-08-17T18:27:36.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:36 smithi136 ceph-mon[109861]: osdmap e1786: 8 total, 8 up, 8 in 2024-08-17T18:27:36.947 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:36 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:27:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:36 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2223830307' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-08-17T18:27:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:36 smithi165 ceph-mon[105517]: osdmap e1786: 8 total, 8 up, 8 in 2024-08-17T18:27:37.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:36 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:27:37.529 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:27:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:27:37] "GET /metrics HTTP/1.1" 200 34258 "" "Prometheus/2.51.0" 2024-08-17T18:27:38.142 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:37 smithi136 ceph-mon[109861]: pgmap v3590: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:38.142 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:37 smithi136 ceph-mon[109861]: osdmap e1787: 8 total, 8 up, 8 in 2024-08-17T18:27:38.142 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:37 smithi136 ceph-mon[115078]: pgmap v3590: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:38.142 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:37 smithi136 ceph-mon[115078]: osdmap e1787: 8 total, 8 up, 8 in 2024-08-17T18:27:38.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:37 smithi165 ceph-mon[105517]: pgmap v3590: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:38.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:37 smithi165 ceph-mon[105517]: osdmap e1787: 8 total, 8 up, 8 in 2024-08-17T18:27:39.835 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.5 to 1 (10000) 2024-08-17T18:27:40.184 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:39 smithi136 ceph-mon[109861]: pgmap v3592: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:40.184 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:39 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1871270346' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T18:27:40.184 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:39 smithi136 ceph-mon[115078]: pgmap v3592: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:40.184 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:39 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1871270346' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T18:27:40.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:39 smithi165 ceph-mon[105517]: pgmap v3592: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:40.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:39 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1871270346' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T18:27:40.496 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 5 1 2024-08-17T18:27:40.706 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:27:41.112 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:40 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1871270346' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T18:27:41.113 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:40 smithi136 ceph-mon[109861]: osdmap e1788: 8 total, 8 up, 8 in 2024-08-17T18:27:41.113 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:40 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1871270346' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T18:27:41.113 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:40 smithi136 ceph-mon[115078]: osdmap e1788: 8 total, 8 up, 8 in 2024-08-17T18:27:41.162 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:40 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1871270346' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T18:27:41.162 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:40 smithi165 ceph-mon[105517]: osdmap e1788: 8 total, 8 up, 8 in 2024-08-17T18:27:42.154 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:41 smithi136 ceph-mon[109861]: pgmap v3594: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:42.154 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:41 smithi136 ceph-mon[115078]: pgmap v3594: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:42.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:41 smithi165 ceph-mon[105517]: pgmap v3594: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:42.848 INFO:teuthology.orchestra.run.smithi136.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-08-17T18:27:43.191 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:42 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3051391435' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-08-17T18:27:43.191 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:42 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3051391435' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-08-17T18:27:43.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:42 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3051391435' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-08-17T18:27:43.530 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 7 1 2024-08-17T18:27:43.733 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:27:44.170 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:43 smithi136 ceph-mon[115078]: pgmap v3595: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:44.170 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:43 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3051391435' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-08-17T18:27:44.170 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:43 smithi136 ceph-mon[115078]: osdmap e1789: 8 total, 8 up, 8 in 2024-08-17T18:27:44.171 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:43 smithi136 ceph-mon[109861]: pgmap v3595: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:44.171 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:43 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3051391435' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-08-17T18:27:44.171 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:43 smithi136 ceph-mon[109861]: osdmap e1789: 8 total, 8 up, 8 in 2024-08-17T18:27:44.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:43 smithi165 ceph-mon[105517]: pgmap v3595: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:44.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:43 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3051391435' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-08-17T18:27:44.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:43 smithi165 ceph-mon[105517]: osdmap e1789: 8 total, 8 up, 8 in 2024-08-17T18:27:45.852 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.7 to 1 (10000) 2024-08-17T18:27:46.170 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:45 smithi136 ceph-mon[109861]: pgmap v3597: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:46.170 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:45 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1833945914' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T18:27:46.170 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:45 smithi136 ceph-mon[115078]: pgmap v3597: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:46.170 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:45 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1833945914' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T18:27:46.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:45 smithi165 ceph-mon[105517]: pgmap v3597: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:46.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:45 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1833945914' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T18:27:46.505 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 7 1 2024-08-17T18:27:46.707 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:27:47.154 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:46 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1833945914' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-17T18:27:47.154 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:46 smithi136 ceph-mon[109861]: osdmap e1790: 8 total, 8 up, 8 in 2024-08-17T18:27:47.154 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:46 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1833945914' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-17T18:27:47.154 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:46 smithi136 ceph-mon[115078]: osdmap e1790: 8 total, 8 up, 8 in 2024-08-17T18:27:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:46 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1833945914' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-17T18:27:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:46 smithi165 ceph-mon[105517]: osdmap e1790: 8 total, 8 up, 8 in 2024-08-17T18:27:47.866 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:27:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:27:47] "GET /metrics HTTP/1.1" 200 34258 "" "Prometheus/2.51.0" 2024-08-17T18:27:48.168 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:47 smithi136 ceph-mon[109861]: pgmap v3599: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:48.169 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:47 smithi136 ceph-mon[115078]: pgmap v3599: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:48.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:47 smithi165 ceph-mon[105517]: pgmap v3599: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:48.878 INFO:teuthology.orchestra.run.smithi136.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-08-17T18:27:49.183 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:48 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1249462442' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-17T18:27:49.183 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:48 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-17T18:27:49.183 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:48 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1249462442' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-17T18:27:49.183 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:48 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-17T18:27:49.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:48 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1249462442' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-17T18:27:49.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:48 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-17T18:27:49.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 2 1 2024-08-17T18:27:49.799 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:27:50.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:49 smithi136 ceph-mon[109861]: pgmap v3600: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:50.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:49 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-08-17T18:27:50.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:49 smithi136 ceph-mon[109861]: osdmap e1791: 8 total, 8 up, 8 in 2024-08-17T18:27:50.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:49 smithi136 ceph-mon[115078]: pgmap v3600: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:50.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:49 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-08-17T18:27:50.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:49 smithi136 ceph-mon[115078]: osdmap e1791: 8 total, 8 up, 8 in 2024-08-17T18:27:50.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:49 smithi165 ceph-mon[105517]: pgmap v3600: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:50.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:49 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-08-17T18:27:50.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:49 smithi165 ceph-mon[105517]: osdmap e1791: 8 total, 8 up, 8 in 2024-08-17T18:27:50.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:50 smithi136 ceph-mon[109861]: osdmap e1792: 8 total, 8 up, 8 in 2024-08-17T18:27:50.947 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:50 smithi136 ceph-mon[115078]: osdmap e1792: 8 total, 8 up, 8 in 2024-08-17T18:27:51.268 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:50 smithi165 ceph-mon[105517]: osdmap e1792: 8 total, 8 up, 8 in 2024-08-17T18:27:51.896 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.2 to 1 (10000) 2024-08-17T18:27:52.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:51 smithi136 ceph-mon[109861]: pgmap v3603: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:52.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:27:52.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:51 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1993714154' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T18:27:52.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:51 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T18:27:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:51 smithi136 ceph-mon[115078]: pgmap v3603: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:27:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:51 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1993714154' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T18:27:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:51 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T18:27:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:51 smithi165 ceph-mon[105517]: pgmap v3603: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:27:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:51 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1993714154' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T18:27:52.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:51 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T18:27:52.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 2 1 2024-08-17T18:27:52.811 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:27:53.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:52 smithi136 ceph-mon[115078]: 1.0 scrub starts 2024-08-17T18:27:53.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:52 smithi136 ceph-mon[115078]: 1.0 scrub ok 2024-08-17T18:27:53.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:52 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T18:27:53.163 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:52 smithi136 ceph-mon[115078]: osdmap e1793: 8 total, 8 up, 8 in 2024-08-17T18:27:53.163 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:52 smithi136 ceph-mon[109861]: 1.0 scrub starts 2024-08-17T18:27:53.164 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:52 smithi136 ceph-mon[109861]: 1.0 scrub ok 2024-08-17T18:27:53.164 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:52 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T18:27:53.164 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:52 smithi136 ceph-mon[109861]: osdmap e1793: 8 total, 8 up, 8 in 2024-08-17T18:27:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:52 smithi165 ceph-mon[105517]: 1.0 scrub starts 2024-08-17T18:27:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:52 smithi165 ceph-mon[105517]: 1.0 scrub ok 2024-08-17T18:27:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:52 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T18:27:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:52 smithi165 ceph-mon[105517]: osdmap e1793: 8 total, 8 up, 8 in 2024-08-17T18:27:54.170 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:53 smithi136 ceph-mon[109861]: pgmap v3605: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:54.170 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:53 smithi136 ceph-mon[115078]: pgmap v3605: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:54.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:53 smithi165 ceph-mon[105517]: pgmap v3605: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:54.910 INFO:teuthology.orchestra.run.smithi136.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-08-17T18:27:55.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:54 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3113487478' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-08-17T18:27:55.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:54 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3113487478' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-08-17T18:27:55.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:54 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3113487478' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-08-17T18:27:55.727 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 6 1 2024-08-17T18:27:55.933 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:27:56.055 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:55 smithi136 ceph-mon[115078]: pgmap v3606: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:56.055 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:55 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3113487478' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-08-17T18:27:56.055 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:55 smithi136 ceph-mon[115078]: osdmap e1794: 8 total, 8 up, 8 in 2024-08-17T18:27:56.055 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:55 smithi136 ceph-mon[109861]: pgmap v3606: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:56.056 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:55 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3113487478' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-08-17T18:27:56.056 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:55 smithi136 ceph-mon[109861]: osdmap e1794: 8 total, 8 up, 8 in 2024-08-17T18:27:56.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:55 smithi165 ceph-mon[105517]: pgmap v3606: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:56.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:55 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3113487478' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-08-17T18:27:56.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:55 smithi165 ceph-mon[105517]: osdmap e1794: 8 total, 8 up, 8 in 2024-08-17T18:27:57.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:27:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:27:57] "GET /metrics HTTP/1.1" 200 34260 "" "Prometheus/2.51.0" 2024-08-17T18:27:57.927 INFO:teuthology.orchestra.run.smithi136.stderr:reweighted osd.6 to 1 (10000) 2024-08-17T18:27:58.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:57 smithi136 ceph-mon[109861]: pgmap v3608: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:58.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:57 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4135378774' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T18:27:58.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:57 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T18:27:58.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:57 smithi136 ceph-mon[115078]: pgmap v3608: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:58.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:57 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4135378774' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T18:27:58.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:57 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T18:27:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:57 smithi165 ceph-mon[105517]: pgmap v3608: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:27:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:57 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4135378774' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T18:27:58.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:57 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T18:27:58.659 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 6 1 2024-08-17T18:27:58.866 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:27:59.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:58 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T18:27:59.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:58 smithi136 ceph-mon[109861]: osdmap e1795: 8 total, 8 up, 8 in 2024-08-17T18:27:59.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:58 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T18:27:59.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:58 smithi136 ceph-mon[115078]: osdmap e1795: 8 total, 8 up, 8 in 2024-08-17T18:27:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:58 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T18:27:59.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:58 smithi165 ceph-mon[105517]: osdmap e1795: 8 total, 8 up, 8 in 2024-08-17T18:28:00.074 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:27:59 smithi136 ceph-mon[109861]: pgmap v3610: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:00.075 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:27:59 smithi136 ceph-mon[115078]: pgmap v3610: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:00.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:27:59 smithi165 ceph-mon[105517]: pgmap v3610: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:00.945 INFO:teuthology.orchestra.run.smithi136.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-08-17T18:28:01.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:00 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3364892121' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-08-17T18:28:01.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:00 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-08-17T18:28:01.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:00 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3364892121' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-08-17T18:28:01.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:00 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-08-17T18:28:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:00 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3364892121' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-08-17T18:28:01.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:00 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-08-17T18:28:01.638 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-08-17T18:28:01.639 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-08-17T18:28:01.639 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-08-17T18:28:01.639 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-08-17T18:28:01.639 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-08-17T18:28:01.639 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-17T18:28:01.845 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:28:02.199 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:01 smithi136 ceph-mon[109861]: pgmap v3611: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:02.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:01 smithi136 ceph-mon[109861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-08-17T18:28:02.200 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:01 smithi136 ceph-mon[109861]: osdmap e1796: 8 total, 8 up, 8 in 2024-08-17T18:28:02.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:01 smithi136 ceph-mon[115078]: pgmap v3611: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:02.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:01 smithi136 ceph-mon[115078]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-08-17T18:28:02.200 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:01 smithi136 ceph-mon[115078]: osdmap e1796: 8 total, 8 up, 8 in 2024-08-17T18:28:02.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:01 smithi165 ceph-mon[105517]: pgmap v3611: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:02.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:01 smithi165 ceph-mon[105517]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-08-17T18:28:02.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:01 smithi165 ceph-mon[105517]: osdmap e1796: 8 total, 8 up, 8 in 2024-08-17T18:28:03.731 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:28:03.732 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":1796,"fsid":"8270436c-5cba-11ef-bccf-c7b262605968","created":"2024-08-17T17:03:17.682767+0000","modified":"2024-08-17T18:28:00.935431+0000","last_up_change":"2024-08-17T18:22:35.786566+0000","last_in_change":"2024-08-17T18:20:30.909950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":24,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":59,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-17T17:06:20.772596+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"1710","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"1491","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-17T17:08:47.840093+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-08-17T17:10:04.522913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-08-17T17:10:05.212219+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51eb458d-9f33-4b87-b7f5-8acb076c16ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":653,"up_from":659,"up_thru":1770,"down_at":653,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1285058127},{"type":"v1","addr":"172.21.15.136:6803","nonce":1285058127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1285058127},{"type":"v1","addr":"172.21.15.136:6805","nonce":1285058127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1285058127},{"type":"v1","addr":"172.21.15.136:6809","nonce":1285058127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1285058127},{"type":"v1","addr":"172.21.15.136:6807","nonce":1285058127}]},"public_addr":"172.21.15.136:6803/1285058127","cluster_addr":"172.21.15.136:6805/1285058127","heartbeat_back_addr":"172.21.15.136:6809/1285058127","heartbeat_front_addr":"172.21.15.136:6807/1285058127","state":["exists","up"]},{"osd":1,"uuid":"aa73fa16-8736-478c-b07a-e674c18698ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":662,"up_from":667,"up_thru":1770,"down_at":662,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":717195127},{"type":"v1","addr":"172.21.15.136:6811","nonce":717195127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":717195127},{"type":"v1","addr":"172.21.15.136:6813","nonce":717195127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":717195127},{"type":"v1","addr":"172.21.15.136:6817","nonce":717195127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":717195127},{"type":"v1","addr":"172.21.15.136:6815","nonce":717195127}]},"public_addr":"172.21.15.136:6811/717195127","cluster_addr":"172.21.15.136:6813/717195127","heartbeat_back_addr":"172.21.15.136:6817/717195127","heartbeat_front_addr":"172.21.15.136:6815/717195127","state":["exists","up"]},{"osd":2,"uuid":"4d2612be-7068-48d9-bb0c-992ebccaa091","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":674,"last_clean_end":1392,"up_from":1399,"up_thru":1770,"down_at":1393,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":809598197},{"type":"v1","addr":"172.21.15.136:6819","nonce":809598197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":809598197},{"type":"v1","addr":"172.21.15.136:6821","nonce":809598197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":809598197},{"type":"v1","addr":"172.21.15.136:6825","nonce":809598197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":809598197},{"type":"v1","addr":"172.21.15.136:6823","nonce":809598197}]},"public_addr":"172.21.15.136:6819/809598197","cluster_addr":"172.21.15.136:6821/809598197","heartbeat_back_addr":"172.21.15.136:6825/809598197","heartbeat_front_addr":"172.21.15.136:6823/809598197","state":["exists","up"]},{"osd":3,"uuid":"30b8a4f2-be6d-4d5f-8397-06c2e17cfc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":682,"up_from":686,"up_thru":1770,"down_at":682,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":946860336},{"type":"v1","addr":"172.21.15.136:6827","nonce":946860336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":946860336},{"type":"v1","addr":"172.21.15.136:6829","nonce":946860336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":946860336},{"type":"v1","addr":"172.21.15.136:6833","nonce":946860336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":946860336},{"type":"v1","addr":"172.21.15.136:6831","nonce":946860336}]},"public_addr":"172.21.15.136:6827/946860336","cluster_addr":"172.21.15.136:6829/946860336","heartbeat_back_addr":"172.21.15.136:6833/946860336","heartbeat_front_addr":"172.21.15.136:6831/946860336","state":["exists","up"]},{"osd":4,"uuid":"ae4074cf-0069-403f-acc7-74bedab266bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":1786,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6801","nonce":842623090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6803","nonce":842623090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6807","nonce":842623090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6805","nonce":842623090}]},"public_addr":"172.21.15.165:6801/842623090","cluster_addr":"172.21.15.165:6803/842623090","heartbeat_back_addr":"172.21.15.165:6807/842623090","heartbeat_front_addr":"172.21.15.165:6805/842623090","state":["exists","up"]},{"osd":5,"uuid":"586c23a7-4831-4b4d-9a06-c1bfcc1c5286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":1770,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6809","nonce":3602301990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6811","nonce":3602301990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6815","nonce":3602301990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6813","nonce":3602301990}]},"public_addr":"172.21.15.165:6809/3602301990","cluster_addr":"172.21.15.165:6811/3602301990","heartbeat_back_addr":"172.21.15.165:6815/3602301990","heartbeat_front_addr":"172.21.15.165:6813/3602301990","state":["exists","up"]},{"osd":6,"uuid":"e838c26b-5e4a-41c6-b35b-b438ba64b237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":46,"last_clean_end":1711,"up_from":1724,"up_thru":1770,"down_at":1712,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":2775283828},{"type":"v1","addr":"172.21.15.165:6817","nonce":2775283828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":2775283828},{"type":"v1","addr":"172.21.15.165:6819","nonce":2775283828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":2775283828},{"type":"v1","addr":"172.21.15.165:6823","nonce":2775283828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":2775283828},{"type":"v1","addr":"172.21.15.165:6821","nonce":2775283828}]},"public_addr":"172.21.15.165:6817/2775283828","cluster_addr":"172.21.15.165:6819/2775283828","heartbeat_back_addr":"172.21.15.165:6823/2775283828","heartbeat_front_addr":"172.21.15.165:6821/2775283828","state":["exists","up"]},{"osd":7,"uuid":"de2fab2e-177a-4eb0-9262-7bbef0d9837f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":1791,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6825","nonce":4195921091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6827","nonce":4195921091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6831","nonce":4195921091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6829","nonce":4195921091}]},"public_addr":"172.21.15.165:6825/4195921091","cluster_addr":"172.21.15.165:6827/4195921091","heartbeat_back_addr":"172.21.15.165:6831/4195921091","heartbeat_front_addr":"172.21.15.165:6829/4195921091","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:30.507144+0000","dead_epoch":653},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:54.456854+0000","dead_epoch":662},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:06:17.338746+0000","dead_epoch":1392},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:06:42.262947+0000","dead_epoch":682},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:02.983751+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:26.594502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:50.363918+0000","dead_epoch":1711},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:08:15.455798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.e","mappings":[{"from":4,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.11","mappings":[{"from":3,"to":5}]},{"pgid":"4.16","mappings":[{"from":3,"to":6}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.136:0/3515054633":"2024-08-17T18:28:42.746890+0000","172.21.15.136:6800/4104535444":"2024-08-18T17:16:05.797025+0000","172.21.15.136:0/3841116499":"2024-08-18T17:16:05.797025+0000","172.21.15.136:0/3340551496":"2024-08-18T17:16:05.797025+0000","172.21.15.136:0/3946052118":"2024-08-18T17:16:05.797025+0000","172.21.15.165:0/3676140772":"2024-08-18T17:13:24.578001+0000","172.21.15.165:6835/7511620":"2024-08-18T17:13:24.578001+0000","172.21.15.136:0/1414082347":"2024-08-18T17:04:10.472088+0000","172.21.15.165:6834/7511620":"2024-08-18T17:13:24.578001+0000","172.21.15.136:6800/803461847":"2024-08-18T17:12:45.319619+0000","172.21.15.136:0/1577563918":"2024-08-18T17:04:10.472088+0000","172.21.15.165:0/300548107":"2024-08-18T17:13:24.578001+0000","172.21.15.165:0/1675517659":"2024-08-18T17:13:24.578001+0000","172.21.15.136:6801/3513268997":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/972617428":"2024-08-18T17:12:45.319619+0000","172.21.15.136:6800/3058734149":"2024-08-18T17:04:10.472088+0000","172.21.15.136:6800/3513268997":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/2887741187":"2024-08-18T17:12:45.319619+0000","172.21.15.136:0/2931810343":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/1381030503":"2024-08-18T17:03:48.417471+0000","172.21.15.136:0/2540800865":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/3933059683":"2024-08-18T17:04:10.472088+0000","172.21.15.165:0/3723748982":"2024-08-18T17:13:24.578001+0000","172.21.15.136:6801/3058734149":"2024-08-18T17:04:10.472088+0000","172.21.15.136:0/3380731497":"2024-08-18T17:03:48.417471+0000","172.21.15.136:0/1905186682":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/1450531984":"2024-08-18T17:08:47.841029+0000","172.21.15.165:0/1694689688":"2024-08-18T17:13:24.578001+0000","172.21.15.136:0/2336022730":"2024-08-18T17:03:48.417471+0000","172.21.15.136:0/2251781402":"2024-08-18T17:16:05.797025+0000","172.21.15.136:0/3166566423":"2024-08-18T17:12:45.319619+0000","172.21.15.136:6801/803461847":"2024-08-18T17:12:45.319619+0000","172.21.15.136:6801/4104535444":"2024-08-18T17:16:05.797025+0000","172.21.15.136:6801/878395364":"2024-08-18T17:03:48.417471+0000","172.21.15.136:0/707907005":"2024-08-18T17:12:45.319619+0000","172.21.15.165:0/2646930649":"2024-08-18T17:13:24.578001+0000","172.21.15.136:6800/878395364":"2024-08-18T17:03:48.417471+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":1}]},{"pool":6,"snaps":[{"begin":2,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":1}]},{"pool":8,"snaps":[{"begin":2,"length":1}]},{"pool":9,"snaps":[{"begin":2,"length":1}]},{"pool":10,"snaps":[{"begin":2,"length":1}]},{"pool":11,"snaps":[{"begin":2,"length":1}]},{"pool":14,"snaps":[{"begin":5,"length":4}]},{"pool":16,"snaps":[{"begin":37,"length":1},{"begin":43,"length":1}]},{"pool":17,"snaps":[{"begin":22,"length":1}]},{"pool":18,"snaps":[{"begin":2,"length":1}]},{"pool":59,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-17T18:28:04.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:03 smithi136 ceph-mon[109861]: pgmap v3613: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:04.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:03 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:28:04.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:03 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3197576631' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-17T18:28:04.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:03 smithi136 ceph-mon[115078]: pgmap v3613: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:04.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:03 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:28:04.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:03 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3197576631' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-17T18:28:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:03 smithi165 ceph-mon[105517]: pgmap v3613: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:03 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T18:28:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:03 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3197576631' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-17T18:28:04.483 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-08-17T18:28:04.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-17T18:28:04.922 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:28:05.978 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:05 smithi136 ceph-mon[109861]: pgmap v3614: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:05.978 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:05 smithi136 ceph-mon[115078]: pgmap v3614: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:06.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:05 smithi165 ceph-mon[105517]: pgmap v3614: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:06.718 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:28:06.718 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":1796,"fsid":"8270436c-5cba-11ef-bccf-c7b262605968","created":"2024-08-17T17:03:17.682767+0000","modified":"2024-08-17T18:28:00.935431+0000","last_up_change":"2024-08-17T18:22:35.786566+0000","last_in_change":"2024-08-17T18:20:30.909950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":24,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":59,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-17T17:06:20.772596+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"1710","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"1491","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-17T17:08:47.840093+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-08-17T17:10:04.522913+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-08-17T17:10:05.212219+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51eb458d-9f33-4b87-b7f5-8acb076c16ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":653,"up_from":659,"up_thru":1770,"down_at":653,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1285058127},{"type":"v1","addr":"172.21.15.136:6803","nonce":1285058127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1285058127},{"type":"v1","addr":"172.21.15.136:6805","nonce":1285058127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":1285058127},{"type":"v1","addr":"172.21.15.136:6809","nonce":1285058127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1285058127},{"type":"v1","addr":"172.21.15.136:6807","nonce":1285058127}]},"public_addr":"172.21.15.136:6803/1285058127","cluster_addr":"172.21.15.136:6805/1285058127","heartbeat_back_addr":"172.21.15.136:6809/1285058127","heartbeat_front_addr":"172.21.15.136:6807/1285058127","state":["exists","up"]},{"osd":1,"uuid":"aa73fa16-8736-478c-b07a-e674c18698ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":662,"up_from":667,"up_thru":1770,"down_at":662,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":717195127},{"type":"v1","addr":"172.21.15.136:6811","nonce":717195127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":717195127},{"type":"v1","addr":"172.21.15.136:6813","nonce":717195127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":717195127},{"type":"v1","addr":"172.21.15.136:6817","nonce":717195127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":717195127},{"type":"v1","addr":"172.21.15.136:6815","nonce":717195127}]},"public_addr":"172.21.15.136:6811/717195127","cluster_addr":"172.21.15.136:6813/717195127","heartbeat_back_addr":"172.21.15.136:6817/717195127","heartbeat_front_addr":"172.21.15.136:6815/717195127","state":["exists","up"]},{"osd":2,"uuid":"4d2612be-7068-48d9-bb0c-992ebccaa091","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":674,"last_clean_end":1392,"up_from":1399,"up_thru":1770,"down_at":1393,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":809598197},{"type":"v1","addr":"172.21.15.136:6819","nonce":809598197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":809598197},{"type":"v1","addr":"172.21.15.136:6821","nonce":809598197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":809598197},{"type":"v1","addr":"172.21.15.136:6825","nonce":809598197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":809598197},{"type":"v1","addr":"172.21.15.136:6823","nonce":809598197}]},"public_addr":"172.21.15.136:6819/809598197","cluster_addr":"172.21.15.136:6821/809598197","heartbeat_back_addr":"172.21.15.136:6825/809598197","heartbeat_front_addr":"172.21.15.136:6823/809598197","state":["exists","up"]},{"osd":3,"uuid":"30b8a4f2-be6d-4d5f-8397-06c2e17cfc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":682,"up_from":686,"up_thru":1770,"down_at":682,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":946860336},{"type":"v1","addr":"172.21.15.136:6827","nonce":946860336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":946860336},{"type":"v1","addr":"172.21.15.136:6829","nonce":946860336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":946860336},{"type":"v1","addr":"172.21.15.136:6833","nonce":946860336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":946860336},{"type":"v1","addr":"172.21.15.136:6831","nonce":946860336}]},"public_addr":"172.21.15.136:6827/946860336","cluster_addr":"172.21.15.136:6829/946860336","heartbeat_back_addr":"172.21.15.136:6833/946860336","heartbeat_front_addr":"172.21.15.136:6831/946860336","state":["exists","up"]},{"osd":4,"uuid":"ae4074cf-0069-403f-acc7-74bedab266bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":1786,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6801","nonce":842623090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6803","nonce":842623090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6807","nonce":842623090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":842623090},{"type":"v1","addr":"172.21.15.165:6805","nonce":842623090}]},"public_addr":"172.21.15.165:6801/842623090","cluster_addr":"172.21.15.165:6803/842623090","heartbeat_back_addr":"172.21.15.165:6807/842623090","heartbeat_front_addr":"172.21.15.165:6805/842623090","state":["exists","up"]},{"osd":5,"uuid":"586c23a7-4831-4b4d-9a06-c1bfcc1c5286","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":1770,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6809","nonce":3602301990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6811","nonce":3602301990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6815","nonce":3602301990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":3602301990},{"type":"v1","addr":"172.21.15.165:6813","nonce":3602301990}]},"public_addr":"172.21.15.165:6809/3602301990","cluster_addr":"172.21.15.165:6811/3602301990","heartbeat_back_addr":"172.21.15.165:6815/3602301990","heartbeat_front_addr":"172.21.15.165:6813/3602301990","state":["exists","up"]},{"osd":6,"uuid":"e838c26b-5e4a-41c6-b35b-b438ba64b237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":46,"last_clean_end":1711,"up_from":1724,"up_thru":1770,"down_at":1712,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":2775283828},{"type":"v1","addr":"172.21.15.165:6817","nonce":2775283828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":2775283828},{"type":"v1","addr":"172.21.15.165:6819","nonce":2775283828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":2775283828},{"type":"v1","addr":"172.21.15.165:6823","nonce":2775283828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":2775283828},{"type":"v1","addr":"172.21.15.165:6821","nonce":2775283828}]},"public_addr":"172.21.15.165:6817/2775283828","cluster_addr":"172.21.15.165:6819/2775283828","heartbeat_back_addr":"172.21.15.165:6823/2775283828","heartbeat_front_addr":"172.21.15.165:6821/2775283828","state":["exists","up"]},{"osd":7,"uuid":"de2fab2e-177a-4eb0-9262-7bbef0d9837f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":1791,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6825","nonce":4195921091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6827","nonce":4195921091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6831","nonce":4195921091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":4195921091},{"type":"v1","addr":"172.21.15.165:6829","nonce":4195921091}]},"public_addr":"172.21.15.165:6825/4195921091","cluster_addr":"172.21.15.165:6827/4195921091","heartbeat_back_addr":"172.21.15.165:6831/4195921091","heartbeat_front_addr":"172.21.15.165:6829/4195921091","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:30.507144+0000","dead_epoch":653},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:54.456854+0000","dead_epoch":662},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:06:17.338746+0000","dead_epoch":1392},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:06:42.262947+0000","dead_epoch":682},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:02.983751+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:26.594502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:07:50.363918+0000","dead_epoch":1711},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:08:15.455798+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.e","mappings":[{"from":4,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.11","mappings":[{"from":3,"to":5}]},{"pgid":"4.16","mappings":[{"from":3,"to":6}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.136:0/3515054633":"2024-08-17T18:28:42.746890+0000","172.21.15.165:0/3676140772":"2024-08-18T17:13:24.578001+0000","172.21.15.136:6800/878395364":"2024-08-18T17:03:48.417471+0000","172.21.15.165:0/1675517659":"2024-08-18T17:13:24.578001+0000","172.21.15.136:6800/803461847":"2024-08-18T17:12:45.319619+0000","172.21.15.165:6835/7511620":"2024-08-18T17:13:24.578001+0000","172.21.15.165:6834/7511620":"2024-08-18T17:13:24.578001+0000","172.21.15.136:0/1414082347":"2024-08-18T17:04:10.472088+0000","172.21.15.136:6801/803461847":"2024-08-18T17:12:45.319619+0000","172.21.15.136:0/2251781402":"2024-08-18T17:16:05.797025+0000","172.21.15.136:0/3166566423":"2024-08-18T17:12:45.319619+0000","172.21.15.136:0/3946052118":"2024-08-18T17:16:05.797025+0000","172.21.15.136:0/1577563918":"2024-08-18T17:04:10.472088+0000","172.21.15.165:0/300548107":"2024-08-18T17:13:24.578001+0000","172.21.15.136:6801/3513268997":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/972617428":"2024-08-18T17:12:45.319619+0000","172.21.15.136:6800/3058734149":"2024-08-18T17:04:10.472088+0000","172.21.15.136:6800/3513268997":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/2887741187":"2024-08-18T17:12:45.319619+0000","172.21.15.136:0/2931810343":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/1381030503":"2024-08-18T17:03:48.417471+0000","172.21.15.136:0/3841116499":"2024-08-18T17:16:05.797025+0000","172.21.15.136:0/3340551496":"2024-08-18T17:16:05.797025+0000","172.21.15.136:0/1450531984":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/2540800865":"2024-08-18T17:08:47.841029+0000","172.21.15.136:0/3933059683":"2024-08-18T17:04:10.472088+0000","172.21.15.165:0/3723748982":"2024-08-18T17:13:24.578001+0000","172.21.15.136:6801/3058734149":"2024-08-18T17:04:10.472088+0000","172.21.15.136:0/3380731497":"2024-08-18T17:03:48.417471+0000","172.21.15.136:0/1905186682":"2024-08-18T17:08:47.841029+0000","172.21.15.136:6800/4104535444":"2024-08-18T17:16:05.797025+0000","172.21.15.136:0/2336022730":"2024-08-18T17:03:48.417471+0000","172.21.15.165:0/1694689688":"2024-08-18T17:13:24.578001+0000","172.21.15.136:6801/878395364":"2024-08-18T17:03:48.417471+0000","172.21.15.165:0/2646930649":"2024-08-18T17:13:24.578001+0000","172.21.15.136:6801/4104535444":"2024-08-18T17:16:05.797025+0000","172.21.15.136:0/707907005":"2024-08-18T17:12:45.319619+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":1}]},{"pool":6,"snaps":[{"begin":2,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":1}]},{"pool":8,"snaps":[{"begin":2,"length":1}]},{"pool":9,"snaps":[{"begin":2,"length":1}]},{"pool":10,"snaps":[{"begin":2,"length":1}]},{"pool":11,"snaps":[{"begin":2,"length":1}]},{"pool":14,"snaps":[{"begin":5,"length":4}]},{"pool":16,"snaps":[{"begin":37,"length":1},{"begin":43,"length":1}]},{"pool":17,"snaps":[{"begin":22,"length":1}]},{"pool":18,"snaps":[{"begin":2,"length":1}]},{"pool":59,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-17T18:28:07.095 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:06 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:28:07.095 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:06 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/2543552411' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-17T18:28:07.095 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:06 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:28:07.095 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:06 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/2543552411' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-17T18:28:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:06 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:28:07.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:06 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/2543552411' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-17T18:28:07.544 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-08-17T18:28:07.545 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-08-17T18:28:07.545 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-08-17T18:28:07.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-08-17T18:28:07.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-08-17T18:28:07.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-08-17T18:28:07.547 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-08-17T18:28:07.547 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-08-17T18:28:07.703 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:28:07 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:28:07] "GET /metrics HTTP/1.1" 200 34258 "" "Prometheus/2.51.0" 2024-08-17T18:28:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:07 smithi165 ceph-mon[105517]: pgmap v3615: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:28:08.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:07 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:28:08.359 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:07 smithi136 ceph-mon[109861]: pgmap v3615: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:08.360 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:28:08.360 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:07 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:28:08.360 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:07 smithi136 ceph-mon[115078]: pgmap v3615: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:08.361 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:28:08.361 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:07 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:28:08.803 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:28:08.834 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:28:08.897 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:28:08.899 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:28:08.901 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:28:08.905 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:28:08.907 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:28:08.911 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:28:10.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:09 smithi136 ceph-mon[109861]: pgmap v3616: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:10.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:09 smithi136 ceph-mon[115078]: pgmap v3616: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:10.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:09 smithi165 ceph-mon[105517]: pgmap v3616: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:12.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:11 smithi165 ceph-mon[105517]: pgmap v3617: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:12.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:11 smithi136 ceph-mon[109861]: pgmap v3617: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:12.446 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:11 smithi136 ceph-mon[115078]: pgmap v3617: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:14.228 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:13 smithi136 ceph-mon[109861]: pgmap v3618: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:14.228 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:13 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:28:14.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:13 smithi136 ceph-mon[115078]: pgmap v3618: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:14.229 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:13 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:28:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:13 smithi165 ceph-mon[105517]: pgmap v3618: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:14.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:13 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:28:15.182 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:14 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:28:15.182 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:14 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:28:15.183 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:14 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:28:15.183 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:14 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:28:15.183 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:14 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:28:15.183 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:14 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:28:15.183 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:14 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:28:15.183 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:14 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:28:15.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:14 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:28:15.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:14 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T18:28:15.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:14 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T18:28:15.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:14 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' 2024-08-17T18:28:16.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:15 smithi136 ceph-mon[109861]: pgmap v3619: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:16.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:15 smithi136 ceph-mon[115078]: pgmap v3619: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:16.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:15 smithi165 ceph-mon[105517]: pgmap v3619: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:17.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:28:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:28:17] "GET /metrics HTTP/1.1" 200 34258 "" "Prometheus/2.51.0" 2024-08-17T18:28:18.252 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:17 smithi136 ceph-mon[109861]: pgmap v3620: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:18.252 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:17 smithi136 ceph-mon[115078]: pgmap v3620: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:18.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:17 smithi165 ceph-mon[105517]: pgmap v3620: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:19.866 INFO:teuthology.orchestra.run.smithi136.stdout:6008659247428 2024-08-17T18:28:19.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-08-17T18:28:20.109 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:19 smithi136 ceph-mon[109861]: pgmap v3621: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:20.109 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:19 smithi136 ceph-mon[115078]: pgmap v3621: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:20.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:19 smithi165 ceph-mon[105517]: pgmap v3621: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:22.237 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:21 smithi136 ceph-mon[109861]: pgmap v3622: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:22.237 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:21 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:28:22.237 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:21 smithi136 ceph-mon[115078]: pgmap v3622: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:22.237 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:21 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:28:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:21 smithi165 ceph-mon[105517]: pgmap v3622: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:22.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:21 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:28:24.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:23 smithi165 ceph-mon[105517]: pgmap v3623: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:24.286 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:23 smithi136 ceph-mon[109861]: pgmap v3623: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:24.286 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:23 smithi136 ceph-mon[115078]: pgmap v3623: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:24.313 INFO:teuthology.orchestra.run.smithi136.stdout:128849019857 2024-08-17T18:28:24.313 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-08-17T18:28:24.444 INFO:teuthology.orchestra.run.smithi136.stdout:163208758221 2024-08-17T18:28:24.444 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-08-17T18:28:25.338 INFO:teuthology.orchestra.run.smithi136.stdout:2830383448856 2024-08-17T18:28:25.338 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-08-17T18:28:25.821 INFO:teuthology.orchestra.run.smithi136.stdout:7404523618375 2024-08-17T18:28:25.822 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-08-17T18:28:26.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:25 smithi136 ceph-mon[115078]: pgmap v3624: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:26.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:25 smithi136 ceph-mon[109861]: pgmap v3624: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:26.230 INFO:teuthology.orchestra.run.smithi136.stdout:2864743187218 2024-08-17T18:28:26.230 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-08-17T18:28:26.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:25 smithi165 ceph-mon[105517]: pgmap v3624: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:26.890 INFO:teuthology.orchestra.run.smithi136.stdout:2946347565828 2024-08-17T18:28:26.890 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-08-17T18:28:27.132 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:28:27.132 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:28:27.376 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:28:27.379 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:28:27.779 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:28:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:28:27] "GET /metrics HTTP/1.1" 200 34262 "" "Prometheus/2.51.0" 2024-08-17T18:28:28.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:27 smithi136 ceph-mon[109861]: pgmap v3625: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:28.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:27 smithi136 ceph-mon[115078]: pgmap v3625: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:28.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:27 smithi165 ceph-mon[105517]: pgmap v3625: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:30.087 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:29 smithi136 ceph-mon[109861]: pgmap v3626: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:30.088 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:29 smithi136 ceph-mon[115078]: pgmap v3626: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:30.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:29 smithi165 ceph-mon[105517]: pgmap v3626: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:30.522 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:28:30.524 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:28:30.942 INFO:teuthology.orchestra.run.smithi136.stdout:231928234949 2024-08-17T18:28:30.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-08-17T18:28:32.110 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:31 smithi136 ceph-mon[109861]: pgmap v3627: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:32.110 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:31 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3632081710' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-08-17T18:28:32.110 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:31 smithi136 ceph-mon[115078]: pgmap v3627: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:32.110 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:31 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3632081710' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-08-17T18:28:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:31 smithi165 ceph-mon[105517]: pgmap v3627: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:32.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:31 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3632081710' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-08-17T18:28:34.127 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:33 smithi136 ceph-mon[109861]: pgmap v3628: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:34.127 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:33 smithi136 ceph-mon[115078]: pgmap v3628: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:34.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:33 smithi165 ceph-mon[105517]: pgmap v3628: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:34.400 INFO:teuthology.orchestra.run.smithi136.stdout:128849019860 2024-08-17T18:28:34.783 INFO:teuthology.orchestra.run.smithi136.stdout:2830383448858 2024-08-17T18:28:34.973 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:28:35.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:35 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3975817179' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-08-17T18:28:35.431 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:35 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3975817179' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-08-17T18:28:35.432 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:35 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3975817179' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-08-17T18:28:35.457 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2830383448856 got 2830383448858 for osd.0 2024-08-17T18:28:35.457 DEBUG:teuthology.parallel:result is None 2024-08-17T18:28:36.071 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849019857 got 128849019860 for osd.4 2024-08-17T18:28:36.071 DEBUG:teuthology.parallel:result is None 2024-08-17T18:28:36.256 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:36 smithi136 ceph-mon[109861]: pgmap v3629: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:36.256 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:36 smithi136 ceph-mon[115078]: pgmap v3629: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:36.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:36 smithi165 ceph-mon[105517]: pgmap v3629: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:37.272 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:37 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:28:37.272 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:37 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:28:37.440 INFO:teuthology.orchestra.run.smithi136.stdout:2864743187221 2024-08-17T18:28:37.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:37 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:28:37.524 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:28:37 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:28:37] "GET /metrics HTTP/1.1" 200 34260 "" "Prometheus/2.51.0" 2024-08-17T18:28:38.298 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:38 smithi136 ceph-mon[109861]: pgmap v3630: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:38.298 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:38 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/589279831' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-08-17T18:28:38.298 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:38 smithi136 ceph-mon[115078]: pgmap v3630: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:38.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:38 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/589279831' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-08-17T18:28:38.368 INFO:teuthology.orchestra.run.smithi136.stdout:6008659247432 2024-08-17T18:28:38.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:38 smithi165 ceph-mon[105517]: pgmap v3630: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:38.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:38 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/589279831' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-08-17T18:28:39.331 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:39 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/1133315299' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-08-17T18:28:39.331 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:39 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/1133315299' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-08-17T18:28:39.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:39 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/1133315299' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-08-17T18:28:40.154 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:40 smithi136 ceph-mon[109861]: pgmap v3631: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:40.155 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:40 smithi136 ceph-mon[115078]: pgmap v3631: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:40.176 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2864743187218 got 2864743187221 for osd.1 2024-08-17T18:28:40.176 DEBUG:teuthology.parallel:result is None 2024-08-17T18:28:40.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:40 smithi165 ceph-mon[105517]: pgmap v3631: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:40.540 INFO:teuthology.orchestra.run.smithi136.stdout:163208758225 2024-08-17T18:28:41.008 INFO:teuthology.orchestra.run.smithi136.stdout:7404523618378 2024-08-17T18:28:41.380 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:41 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/4136143703' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-08-17T18:28:41.380 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:41 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/748249123' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-08-17T18:28:41.380 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:41 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/4136143703' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-08-17T18:28:41.380 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:41 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/748249123' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-08-17T18:28:41.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:41 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/4136143703' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-08-17T18:28:41.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:41 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/748249123' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-08-17T18:28:41.849 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208758221 got 163208758225 for osd.5 2024-08-17T18:28:41.849 DEBUG:teuthology.parallel:result is None 2024-08-17T18:28:42.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:42 smithi136 ceph-mon[109861]: pgmap v3632: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:42.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:42 smithi136 ceph-mon[115078]: pgmap v3632: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:42.352 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:28:42.375 INFO:tasks.cephadm.ceph_manager.ceph:need seq 6008659247428 got 6008659247432 for osd.2 2024-08-17T18:28:42.376 DEBUG:teuthology.parallel:result is None 2024-08-17T18:28:42.412 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:42 smithi165 ceph-mon[105517]: pgmap v3632: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:42.833 INFO:tasks.cephadm.ceph_manager.ceph:need seq 7404523618375 got 7404523618378 for osd.6 2024-08-17T18:28:42.833 DEBUG:teuthology.parallel:result is None 2024-08-17T18:28:43.944 INFO:teuthology.orchestra.run.smithi136.stdout:2946347565832 2024-08-17T18:28:44.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:44 smithi136 ceph-mon[109861]: pgmap v3633: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:44.446 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:44 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/3728427121' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-08-17T18:28:44.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:44 smithi136 ceph-mon[115078]: pgmap v3633: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:44.447 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:44 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/3728427121' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-08-17T18:28:44.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:44 smithi165 ceph-mon[105517]: pgmap v3633: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:44.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:44 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/3728427121' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-08-17T18:28:44.536 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2946347565828 got 2946347565832 for osd.3 2024-08-17T18:28:44.536 DEBUG:teuthology.parallel:result is None 2024-08-17T18:28:45.898 INFO:teuthology.orchestra.run.smithi136.stdout:231928234953 2024-08-17T18:28:46.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:45 smithi136 ceph-mon[109861]: pgmap v3634: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:46.196 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:45 smithi136 ceph-mon[109861]: osdmap e1797: 8 total, 8 up, 8 in 2024-08-17T18:28:46.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:45 smithi136 ceph-mon[115078]: pgmap v3634: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:46.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:45 smithi136 ceph-mon[115078]: osdmap e1797: 8 total, 8 up, 8 in 2024-08-17T18:28:46.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:45 smithi165 ceph-mon[105517]: pgmap v3634: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:46.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:45 smithi165 ceph-mon[105517]: osdmap e1797: 8 total, 8 up, 8 in 2024-08-17T18:28:46.647 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234949 got 231928234953 for osd.7 2024-08-17T18:28:46.648 DEBUG:teuthology.parallel:result is None 2024-08-17T18:28:46.648 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-08-17T18:28:46.649 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T18:28:46.860 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:28:46.946 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:46 smithi136 ceph-mon[109861]: from='client.? 172.21.15.136:0/284054090' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-08-17T18:28:46.946 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:46 smithi136 ceph-mon[115078]: from='client.? 172.21.15.136:0/284054090' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-08-17T18:28:47.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:46 smithi165 ceph-mon[105517]: from='client.? 172.21.15.136:0/284054090' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-08-17T18:28:47.696 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:28:47 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:28:47] "GET /metrics HTTP/1.1" 200 34260 "" "Prometheus/2.51.0" 2024-08-17T18:28:48.106 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:47 smithi136 ceph-mon[109861]: pgmap v3636: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:48.107 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:47 smithi136 ceph-mon[115078]: pgmap v3636: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:48.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:47 smithi165 ceph-mon[105517]: pgmap v3636: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:48.821 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:28:48.821 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T18:28:49.453 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":3637,"stamp":"2024-08-17T18:28:48.680869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595941,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":1193799,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1698,"ondisk_log_size":1698,"up":249,"acting":249,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":249,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8046664,"kb_used_data":91240,"kb_used_omap":474,"kb_used_meta":7954853,"kb_avail":741914552,"statfs":{"total":767960285184,"available":759720501248,"internally_reserved":0,"allocated":93429760,"data_stored":52120435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":485693,"internal_metadata":8145770179},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002635"},"pg_stats":[{"pgid":"3.1f","version":"70'4","reported_seq":132,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:45.494780+0000","last_change":"2024-08-17T18:23:59.758838+0000","last_active":"2024-08-17T18:28:45.494780+0000","last_peered":"2024-08-17T18:28:45.494780+0000","last_clean":"2024-08-17T18:28:45.494780+0000","last_became_active":"2024-08-17T18:23:59.748445+0000","last_became_peered":"2024-08-17T18:23:59.748445+0000","last_unstale":"2024-08-17T18:28:45.494780+0000","last_undegraded":"2024-08-17T18:28:45.494780+0000","last_fullsized":"2024-08-17T18:28:45.494780+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:08:27.581912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1862,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.185729+0000","last_change":"2024-08-17T18:23:36.324372+0000","last_active":"2024-08-17T18:28:02.185729+0000","last_peered":"2024-08-17T18:28:02.185729+0000","last_clean":"2024-08-17T18:28:02.185729+0000","last_became_active":"2024-08-17T18:23:36.323893+0000","last_became_peered":"2024-08-17T18:23:36.323893+0000","last_unstale":"2024-08-17T18:28:02.185729+0000","last_undegraded":"2024-08-17T18:28:02.185729+0000","last_fullsized":"2024-08-17T18:28:02.185729+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:10:52.829170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":291,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.864992+0000","last_change":"2024-08-17T18:23:36.319358+0000","last_active":"2024-08-17T18:28:44.864992+0000","last_peered":"2024-08-17T18:28:44.864992+0000","last_clean":"2024-08-17T18:28:44.864992+0000","last_became_active":"2024-08-17T18:23:36.318521+0000","last_became_peered":"2024-08-17T18:23:36.318521+0000","last_unstale":"2024-08-17T18:28:44.864992+0000","last_undegraded":"2024-08-17T18:28:44.864992+0000","last_fullsized":"2024-08-17T18:28:44.864992+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:56:48.984477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":283,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.864890+0000","last_change":"2024-08-17T18:23:59.741806+0000","last_active":"2024-08-17T18:28:44.864890+0000","last_peered":"2024-08-17T18:28:44.864890+0000","last_clean":"2024-08-17T18:28:44.864890+0000","last_became_active":"2024-08-17T18:23:59.741479+0000","last_became_peered":"2024-08-17T18:23:59.741479+0000","last_unstale":"2024-08-17T18:28:44.864890+0000","last_undegraded":"2024-08-17T18:28:44.864890+0000","last_fullsized":"2024-08-17T18:28:44.864890+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:02:39.072277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":371,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:01.502242+0000","last_change":"2024-08-17T18:22:36.836343+0000","last_active":"2024-08-17T18:28:01.502242+0000","last_peered":"2024-08-17T18:28:01.502242+0000","last_clean":"2024-08-17T18:28:01.502242+0000","last_became_active":"2024-08-17T18:22:36.835580+0000","last_became_peered":"2024-08-17T18:22:36.835580+0000","last_unstale":"2024-08-17T18:28:01.502242+0000","last_undegraded":"2024-08-17T18:28:01.502242+0000","last_fullsized":"2024-08-17T18:28:01.502242+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:05:51.737218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1515,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.185705+0000","last_change":"2024-08-17T18:27:37.426405+0000","last_active":"2024-08-17T18:28:02.185705+0000","last_peered":"2024-08-17T18:28:02.185705+0000","last_clean":"2024-08-17T18:28:02.185705+0000","last_became_active":"2024-08-17T18:27:37.426039+0000","last_became_peered":"2024-08-17T18:27:37.426039+0000","last_unstale":"2024-08-17T18:28:02.185705+0000","last_undegraded":"2024-08-17T18:28:02.185705+0000","last_fullsized":"2024-08-17T18:28:02.185705+0000","mapping_epoch":1786,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1787,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:14:11.039422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":382,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:01.502347+0000","last_change":"2024-08-17T18:20:32.094881+0000","last_active":"2024-08-17T18:28:01.502347+0000","last_peered":"2024-08-17T18:28:01.502347+0000","last_clean":"2024-08-17T18:28:01.502347+0000","last_became_active":"2024-08-17T18:20:32.077429+0000","last_became_peered":"2024-08-17T18:20:32.077429+0000","last_unstale":"2024-08-17T18:28:01.502347+0000","last_undegraded":"2024-08-17T18:28:01.502347+0000","last_fullsized":"2024-08-17T18:28:01.502347+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:08:26.290851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1867,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.185662+0000","last_change":"2024-08-17T18:25:14.026295+0000","last_active":"2024-08-17T18:28:02.185662+0000","last_peered":"2024-08-17T18:28:02.185662+0000","last_clean":"2024-08-17T18:28:02.185662+0000","last_became_active":"2024-08-17T18:25:14.025999+0000","last_became_peered":"2024-08-17T18:25:14.025999+0000","last_unstale":"2024-08-17T18:28:02.185662+0000","last_undegraded":"2024-08-17T18:28:02.185662+0000","last_fullsized":"2024-08-17T18:28:02.185662+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:55:16.975299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":125,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:45.494461+0000","last_change":"2024-08-17T18:23:59.745317+0000","last_active":"2024-08-17T18:28:45.494461+0000","last_peered":"2024-08-17T18:28:45.494461+0000","last_clean":"2024-08-17T18:28:45.494461+0000","last_became_active":"2024-08-17T18:23:59.743236+0000","last_became_peered":"2024-08-17T18:23:59.743236+0000","last_unstale":"2024-08-17T18:28:45.494461+0000","last_undegraded":"2024-08-17T18:28:45.494461+0000","last_fullsized":"2024-08-17T18:28:45.494461+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:11.253490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":152,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.865001+0000","last_change":"2024-08-17T18:22:53.938704+0000","last_active":"2024-08-17T18:28:44.865001+0000","last_peered":"2024-08-17T18:28:44.865001+0000","last_clean":"2024-08-17T18:28:44.865001+0000","last_became_active":"2024-08-17T18:22:53.938406+0000","last_became_peered":"2024-08-17T18:22:53.938406+0000","last_unstale":"2024-08-17T18:28:44.865001+0000","last_undegraded":"2024-08-17T18:28:44.865001+0000","last_fullsized":"2024-08-17T18:28:44.865001+0000","mapping_epoch":1727,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:58:21.566014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1844,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.185939+0000","last_change":"2024-08-17T18:20:31.942863+0000","last_active":"2024-08-17T18:28:02.185939+0000","last_peered":"2024-08-17T18:28:02.185939+0000","last_clean":"2024-08-17T18:28:02.185939+0000","last_became_active":"2024-08-17T18:20:31.941281+0000","last_became_peered":"2024-08-17T18:20:31.941281+0000","last_unstale":"2024-08-17T18:28:02.185939+0000","last_undegraded":"2024-08-17T18:28:02.185939+0000","last_fullsized":"2024-08-17T18:28:02.185939+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:48:05.556975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":283,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.864829+0000","last_change":"2024-08-17T18:20:31.933142+0000","last_active":"2024-08-17T18:28:44.864829+0000","last_peered":"2024-08-17T18:28:44.864829+0000","last_clean":"2024-08-17T18:28:44.864829+0000","last_became_active":"2024-08-17T18:20:31.932798+0000","last_became_peered":"2024-08-17T18:20:31.932798+0000","last_unstale":"2024-08-17T18:28:44.864829+0000","last_undegraded":"2024-08-17T18:28:44.864829+0000","last_fullsized":"2024-08-17T18:28:44.864829+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:32:51.059352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":217,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.969493+0000","last_change":"2024-08-17T18:20:31.937627+0000","last_active":"2024-08-17T18:28:44.969493+0000","last_peered":"2024-08-17T18:28:44.969493+0000","last_clean":"2024-08-17T18:28:44.969493+0000","last_became_active":"2024-08-17T18:20:31.937361+0000","last_became_peered":"2024-08-17T18:20:31.937361+0000","last_unstale":"2024-08-17T18:28:44.969493+0000","last_undegraded":"2024-08-17T18:28:44.969493+0000","last_fullsized":"2024-08-17T18:28:44.969493+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:01:26.309656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":126,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:45.494663+0000","last_change":"2024-08-17T18:23:59.743052+0000","last_active":"2024-08-17T18:28:45.494663+0000","last_peered":"2024-08-17T18:28:45.494663+0000","last_clean":"2024-08-17T18:28:45.494663+0000","last_became_active":"2024-08-17T18:23:59.742158+0000","last_became_peered":"2024-08-17T18:23:59.742158+0000","last_unstale":"2024-08-17T18:28:45.494663+0000","last_undegraded":"2024-08-17T18:28:45.494663+0000","last_fullsized":"2024-08-17T18:28:45.494663+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:34:34.249644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":304,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.864463+0000","last_change":"2024-08-17T18:25:14.176708+0000","last_active":"2024-08-17T18:28:44.864463+0000","last_peered":"2024-08-17T18:28:44.864463+0000","last_clean":"2024-08-17T18:28:44.864463+0000","last_became_active":"2024-08-17T18:25:14.176239+0000","last_became_peered":"2024-08-17T18:25:14.176239+0000","last_unstale":"2024-08-17T18:28:44.864463+0000","last_undegraded":"2024-08-17T18:28:44.864463+0000","last_fullsized":"2024-08-17T18:28:44.864463+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:08:21.826456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1860,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.526199+0000","last_change":"2024-08-17T18:22:36.867005+0000","last_active":"2024-08-17T18:28:02.526199+0000","last_peered":"2024-08-17T18:28:02.526199+0000","last_clean":"2024-08-17T18:28:02.526199+0000","last_became_active":"2024-08-17T18:22:36.866297+0000","last_became_peered":"2024-08-17T18:22:36.866297+0000","last_unstale":"2024-08-17T18:28:02.526199+0000","last_undegraded":"2024-08-17T18:28:02.526199+0000","last_fullsized":"2024-08-17T18:28:02.526199+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:09:18.460006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":306,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.864569+0000","last_change":"2024-08-17T18:25:14.035767+0000","last_active":"2024-08-17T18:28:44.864569+0000","last_peered":"2024-08-17T18:28:44.864569+0000","last_clean":"2024-08-17T18:28:44.864569+0000","last_became_active":"2024-08-17T18:25:14.034592+0000","last_became_peered":"2024-08-17T18:25:14.034592+0000","last_unstale":"2024-08-17T18:28:44.864569+0000","last_undegraded":"2024-08-17T18:28:44.864569+0000","last_fullsized":"2024-08-17T18:28:44.864569+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:22:00.711481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":125,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:45.494829+0000","last_change":"2024-08-17T18:23:59.744879+0000","last_active":"2024-08-17T18:28:45.494829+0000","last_peered":"2024-08-17T18:28:45.494829+0000","last_clean":"2024-08-17T18:28:45.494829+0000","last_became_active":"2024-08-17T18:23:59.743333+0000","last_became_peered":"2024-08-17T18:23:59.743333+0000","last_unstale":"2024-08-17T18:28:45.494829+0000","last_undegraded":"2024-08-17T18:28:45.494829+0000","last_fullsized":"2024-08-17T18:28:45.494829+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:02:01.546706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":117,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:12.378698+0000","last_change":"2024-08-17T18:22:36.836745+0000","last_active":"2024-08-17T18:28:12.378698+0000","last_peered":"2024-08-17T18:28:12.378698+0000","last_clean":"2024-08-17T18:28:12.378698+0000","last_became_active":"2024-08-17T18:22:36.835161+0000","last_became_peered":"2024-08-17T18:22:36.835161+0000","last_unstale":"2024-08-17T18:28:12.378698+0000","last_undegraded":"2024-08-17T18:28:12.378698+0000","last_fullsized":"2024-08-17T18:28:12.378698+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:29:43.303633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":441,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:01.502269+0000","last_change":"2024-08-17T18:25:14.027677+0000","last_active":"2024-08-17T18:28:01.502269+0000","last_peered":"2024-08-17T18:28:01.502269+0000","last_clean":"2024-08-17T18:28:01.502269+0000","last_became_active":"2024-08-17T18:25:14.027309+0000","last_became_peered":"2024-08-17T18:25:14.027309+0000","last_unstale":"2024-08-17T18:28:01.502269+0000","last_undegraded":"2024-08-17T18:28:01.502269+0000","last_fullsized":"2024-08-17T18:28:01.502269+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:15:02.617263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":91,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:00.943823+0000","last_change":"2024-08-17T18:27:49.885531+0000","last_active":"2024-08-17T18:28:00.943823+0000","last_peered":"2024-08-17T18:28:00.943823+0000","last_clean":"2024-08-17T18:28:00.943823+0000","last_became_active":"2024-08-17T18:27:49.884877+0000","last_became_peered":"2024-08-17T18:27:49.884877+0000","last_unstale":"2024-08-17T18:28:00.943823+0000","last_undegraded":"2024-08-17T18:28:00.943823+0000","last_fullsized":"2024-08-17T18:28:00.943823+0000","mapping_epoch":1791,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:05:23.870213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":101,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.969552+0000","last_change":"2024-08-17T18:25:14.030155+0000","last_active":"2024-08-17T18:28:44.969552+0000","last_peered":"2024-08-17T18:28:44.969552+0000","last_clean":"2024-08-17T18:28:44.969552+0000","last_became_active":"2024-08-17T18:25:14.029876+0000","last_became_peered":"2024-08-17T18:25:14.029876+0000","last_unstale":"2024-08-17T18:28:44.969552+0000","last_undegraded":"2024-08-17T18:28:44.969552+0000","last_fullsized":"2024-08-17T18:28:44.969552+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:04:34.619858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1844,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.186222+0000","last_change":"2024-08-17T18:22:36.838096+0000","last_active":"2024-08-17T18:28:02.186222+0000","last_peered":"2024-08-17T18:28:02.186222+0000","last_clean":"2024-08-17T18:28:02.186222+0000","last_became_active":"2024-08-17T18:22:36.835757+0000","last_became_peered":"2024-08-17T18:22:36.835757+0000","last_unstale":"2024-08-17T18:28:02.186222+0000","last_undegraded":"2024-08-17T18:28:02.186222+0000","last_fullsized":"2024-08-17T18:28:02.186222+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:55:31.016446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1835,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.186215+0000","last_change":"2024-08-17T18:25:14.026273+0000","last_active":"2024-08-17T18:28:02.186215+0000","last_peered":"2024-08-17T18:28:02.186215+0000","last_clean":"2024-08-17T18:28:02.186215+0000","last_became_active":"2024-08-17T18:25:14.025964+0000","last_became_peered":"2024-08-17T18:25:14.025964+0000","last_unstale":"2024-08-17T18:28:02.186215+0000","last_undegraded":"2024-08-17T18:28:02.186215+0000","last_fullsized":"2024-08-17T18:28:02.186215+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:43.150679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"347'149","reported_seq":1104,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.864385+0000","last_change":"2024-08-17T18:20:33.927014+0000","last_active":"2024-08-17T18:28:44.864385+0000","last_peered":"2024-08-17T18:28:44.864385+0000","last_clean":"2024-08-17T18:28:44.864385+0000","last_became_active":"2024-08-17T18:20:33.926619+0000","last_became_peered":"2024-08-17T18:20:33.926619+0000","last_unstale":"2024-08-17T18:28:44.864385+0000","last_undegraded":"2024-08-17T18:28:44.864385+0000","last_fullsized":"2024-08-17T18:28:44.864385+0000","mapping_epoch":1697,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1698,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:57.271286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:57.271286+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:35:26.778777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1822,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.185872+0000","last_change":"2024-08-17T18:22:36.838651+0000","last_active":"2024-08-17T18:28:02.185872+0000","last_peered":"2024-08-17T18:28:02.185872+0000","last_clean":"2024-08-17T18:28:02.185872+0000","last_became_active":"2024-08-17T18:22:36.838262+0000","last_became_peered":"2024-08-17T18:22:36.838262+0000","last_unstale":"2024-08-17T18:28:02.185872+0000","last_undegraded":"2024-08-17T18:28:02.185872+0000","last_fullsized":"2024-08-17T18:28:02.185872+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:24:25.119257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1803,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.526126+0000","last_change":"2024-08-17T18:22:36.866053+0000","last_active":"2024-08-17T18:28:02.526126+0000","last_peered":"2024-08-17T18:28:02.526126+0000","last_clean":"2024-08-17T18:28:02.526126+0000","last_became_active":"2024-08-17T18:22:36.865187+0000","last_became_peered":"2024-08-17T18:22:36.865187+0000","last_unstale":"2024-08-17T18:28:02.526126+0000","last_undegraded":"2024-08-17T18:28:02.526126+0000","last_fullsized":"2024-08-17T18:28:02.526126+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:45:34.232504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":130,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:45.494723+0000","last_change":"2024-08-17T18:23:59.750520+0000","last_active":"2024-08-17T18:28:45.494723+0000","last_peered":"2024-08-17T18:28:45.494723+0000","last_clean":"2024-08-17T18:28:45.494723+0000","last_became_active":"2024-08-17T18:23:59.742332+0000","last_became_peered":"2024-08-17T18:23:59.742332+0000","last_unstale":"2024-08-17T18:28:45.494723+0000","last_undegraded":"2024-08-17T18:28:45.494723+0000","last_fullsized":"2024-08-17T18:28:45.494723+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:39:39.325194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"347'149","reported_seq":1111,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:05.191604+0000","last_change":"2024-08-17T18:23:39.360135+0000","last_active":"2024-08-17T18:28:05.191604+0000","last_peered":"2024-08-17T18:28:05.191604+0000","last_clean":"2024-08-17T18:28:05.191604+0000","last_became_active":"2024-08-17T18:23:39.359652+0000","last_became_peered":"2024-08-17T18:23:39.359652+0000","last_unstale":"2024-08-17T18:28:05.191604+0000","last_undegraded":"2024-08-17T18:28:05.191604+0000","last_fullsized":"2024-08-17T18:28:05.191604+0000","mapping_epoch":1736,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1737,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:53.309517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:53.309517+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:20:52.590505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":291,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.864683+0000","last_change":"2024-08-17T18:23:59.746854+0000","last_active":"2024-08-17T18:28:44.864683+0000","last_peered":"2024-08-17T18:28:44.864683+0000","last_clean":"2024-08-17T18:28:44.864683+0000","last_became_active":"2024-08-17T18:23:59.746190+0000","last_became_peered":"2024-08-17T18:23:59.746190+0000","last_unstale":"2024-08-17T18:28:44.864683+0000","last_undegraded":"2024-08-17T18:28:44.864683+0000","last_fullsized":"2024-08-17T18:28:44.864683+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:01:53.108037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":101,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.969944+0000","last_change":"2024-08-17T18:25:14.025153+0000","last_active":"2024-08-17T18:28:44.969944+0000","last_peered":"2024-08-17T18:28:44.969944+0000","last_clean":"2024-08-17T18:28:44.969944+0000","last_became_active":"2024-08-17T18:25:14.024700+0000","last_became_peered":"2024-08-17T18:25:14.024700+0000","last_unstale":"2024-08-17T18:28:44.969944+0000","last_undegraded":"2024-08-17T18:28:44.969944+0000","last_fullsized":"2024-08-17T18:28:44.969944+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:50:25.014563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":292,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.865341+0000","last_change":"2024-08-17T18:23:59.772682+0000","last_active":"2024-08-17T18:28:44.865341+0000","last_peered":"2024-08-17T18:28:44.865341+0000","last_clean":"2024-08-17T18:28:44.865341+0000","last_became_active":"2024-08-17T18:23:59.748920+0000","last_became_peered":"2024-08-17T18:23:59.748920+0000","last_unstale":"2024-08-17T18:28:44.865341+0000","last_undegraded":"2024-08-17T18:28:44.865341+0000","last_fullsized":"2024-08-17T18:28:44.865341+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:07:31.660610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"347'149","reported_seq":1101,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.526228+0000","last_change":"2024-08-17T18:24:02.793338+0000","last_active":"2024-08-17T18:28:02.526228+0000","last_peered":"2024-08-17T18:28:02.526228+0000","last_clean":"2024-08-17T18:28:02.526228+0000","last_became_active":"2024-08-17T18:24:02.792982+0000","last_became_peered":"2024-08-17T18:24:02.792982+0000","last_unstale":"2024-08-17T18:28:02.526228+0000","last_undegraded":"2024-08-17T18:28:02.526228+0000","last_fullsized":"2024-08-17T18:28:02.526228+0000","mapping_epoch":1743,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1744,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:52.474544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:52.474544+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:37:26.909105+0000","scrub_duration":0.001886326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":126,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:45.494533+0000","last_change":"2024-08-17T18:23:59.741315+0000","last_active":"2024-08-17T18:28:45.494533+0000","last_peered":"2024-08-17T18:28:45.494533+0000","last_clean":"2024-08-17T18:28:45.494533+0000","last_became_active":"2024-08-17T18:23:59.741020+0000","last_became_peered":"2024-08-17T18:23:59.741020+0000","last_unstale":"2024-08-17T18:28:45.494533+0000","last_undegraded":"2024-08-17T18:28:45.494533+0000","last_fullsized":"2024-08-17T18:28:45.494533+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:46:47.093299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":213,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:32.164530+0000","last_change":"2024-08-17T18:23:59.740824+0000","last_active":"2024-08-17T18:28:32.164530+0000","last_peered":"2024-08-17T18:28:32.164530+0000","last_clean":"2024-08-17T18:28:32.164530+0000","last_became_active":"2024-08-17T18:23:59.740478+0000","last_became_peered":"2024-08-17T18:23:59.740478+0000","last_unstale":"2024-08-17T18:28:32.164530+0000","last_undegraded":"2024-08-17T18:28:32.164530+0000","last_fullsized":"2024-08-17T18:28:32.164530+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:44:30.520216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":217,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.969870+0000","last_change":"2024-08-17T18:20:32.012874+0000","last_active":"2024-08-17T18:28:44.969870+0000","last_peered":"2024-08-17T18:28:44.969870+0000","last_clean":"2024-08-17T18:28:44.969870+0000","last_became_active":"2024-08-17T18:20:32.012553+0000","last_became_peered":"2024-08-17T18:20:32.012553+0000","last_unstale":"2024-08-17T18:28:44.969870+0000","last_undegraded":"2024-08-17T18:28:44.969870+0000","last_fullsized":"2024-08-17T18:28:44.969870+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:14:17.188745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"347'149","reported_seq":1067,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.526054+0000","last_change":"2024-08-17T18:22:36.866341+0000","last_active":"2024-08-17T18:28:02.526054+0000","last_peered":"2024-08-17T18:28:02.526054+0000","last_clean":"2024-08-17T18:28:02.526054+0000","last_became_active":"2024-08-17T18:22:36.865561+0000","last_became_peered":"2024-08-17T18:22:36.865561+0000","last_unstale":"2024-08-17T18:28:02.526054+0000","last_undegraded":"2024-08-17T18:28:02.526054+0000","last_fullsized":"2024-08-17T18:28:02.526054+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:49.461698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:49.461698+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:22:21.423162+0000","scrub_duration":0.0013792520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":69,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.864638+0000","last_change":"2024-08-17T18:26:13.338201+0000","last_active":"2024-08-17T18:28:44.864638+0000","last_peered":"2024-08-17T18:28:44.864638+0000","last_clean":"2024-08-17T18:28:44.864638+0000","last_became_active":"2024-08-17T18:26:13.337933+0000","last_became_peered":"2024-08-17T18:26:13.337933+0000","last_unstale":"2024-08-17T18:28:44.864638+0000","last_undegraded":"2024-08-17T18:28:44.864638+0000","last_fullsized":"2024-08-17T18:28:44.864638+0000","mapping_epoch":1768,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1769,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:16:05.784689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":217,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.969933+0000","last_change":"2024-08-17T18:20:32.014875+0000","last_active":"2024-08-17T18:28:44.969933+0000","last_peered":"2024-08-17T18:28:44.969933+0000","last_clean":"2024-08-17T18:28:44.969933+0000","last_became_active":"2024-08-17T18:20:32.014631+0000","last_became_peered":"2024-08-17T18:20:32.014631+0000","last_unstale":"2024-08-17T18:28:44.969933+0000","last_undegraded":"2024-08-17T18:28:44.969933+0000","last_fullsized":"2024-08-17T18:28:44.969933+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:00:50.978635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":372,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:01.502236+0000","last_change":"2024-08-17T18:18:29.735362+0000","last_active":"2024-08-17T18:28:01.502236+0000","last_peered":"2024-08-17T18:28:01.502236+0000","last_clean":"2024-08-17T18:28:01.502236+0000","last_became_active":"2024-08-17T18:18:29.734752+0000","last_became_peered":"2024-08-17T18:18:29.734752+0000","last_unstale":"2024-08-17T18:28:01.502236+0000","last_undegraded":"2024-08-17T18:28:01.502236+0000","last_fullsized":"2024-08-17T18:28:01.502236+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:25.709698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"347'149","reported_seq":1112,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.377477+0000","last_change":"2024-08-17T18:24:02.792784+0000","last_active":"2024-08-17T18:28:02.377477+0000","last_peered":"2024-08-17T18:28:02.377477+0000","last_clean":"2024-08-17T18:28:02.377477+0000","last_became_active":"2024-08-17T18:24:02.792286+0000","last_became_peered":"2024-08-17T18:24:02.792286+0000","last_unstale":"2024-08-17T18:28:02.377477+0000","last_undegraded":"2024-08-17T18:28:02.377477+0000","last_fullsized":"2024-08-17T18:28:02.377477+0000","mapping_epoch":1743,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1744,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:00.316884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:00.316884+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:26:57.982520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":101,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.969721+0000","last_change":"2024-08-17T18:25:14.031207+0000","last_active":"2024-08-17T18:28:44.969721+0000","last_peered":"2024-08-17T18:28:44.969721+0000","last_clean":"2024-08-17T18:28:44.969721+0000","last_became_active":"2024-08-17T18:25:14.030916+0000","last_became_peered":"2024-08-17T18:25:14.030916+0000","last_unstale":"2024-08-17T18:28:44.969721+0000","last_undegraded":"2024-08-17T18:28:44.969721+0000","last_fullsized":"2024-08-17T18:28:44.969721+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:37:34.050072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1858,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.186329+0000","last_change":"2024-08-17T18:23:59.741594+0000","last_active":"2024-08-17T18:28:02.186329+0000","last_peered":"2024-08-17T18:28:02.186329+0000","last_clean":"2024-08-17T18:28:02.186329+0000","last_became_active":"2024-08-17T18:23:59.741207+0000","last_became_peered":"2024-08-17T18:23:59.741207+0000","last_unstale":"2024-08-17T18:28:02.186329+0000","last_undegraded":"2024-08-17T18:28:02.186329+0000","last_fullsized":"2024-08-17T18:28:02.186329+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:02:52.545413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":441,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:01.502262+0000","last_change":"2024-08-17T18:22:36.836104+0000","last_active":"2024-08-17T18:28:01.502262+0000","last_peered":"2024-08-17T18:28:01.502262+0000","last_clean":"2024-08-17T18:28:01.502262+0000","last_became_active":"2024-08-17T18:22:36.835273+0000","last_became_peered":"2024-08-17T18:22:36.835273+0000","last_unstale":"2024-08-17T18:28:01.502262+0000","last_undegraded":"2024-08-17T18:28:01.502262+0000","last_fullsized":"2024-08-17T18:28:01.502262+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:26:23.494575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"347'149","reported_seq":1042,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:00.943652+0000","last_change":"2024-08-17T18:27:49.884360+0000","last_active":"2024-08-17T18:28:00.943652+0000","last_peered":"2024-08-17T18:28:00.943652+0000","last_clean":"2024-08-17T18:28:00.943652+0000","last_became_active":"2024-08-17T18:27:49.884006+0000","last_became_peered":"2024-08-17T18:27:49.884006+0000","last_unstale":"2024-08-17T18:28:00.943652+0000","last_undegraded":"2024-08-17T18:28:00.943652+0000","last_fullsized":"2024-08-17T18:28:00.943652+0000","mapping_epoch":1791,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1792,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:02.363531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:02.363531+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:43:15.930550+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":101,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.969999+0000","last_change":"2024-08-17T18:25:14.176621+0000","last_active":"2024-08-17T18:28:44.969999+0000","last_peered":"2024-08-17T18:28:44.969999+0000","last_clean":"2024-08-17T18:28:44.969999+0000","last_became_active":"2024-08-17T18:25:14.176321+0000","last_became_peered":"2024-08-17T18:25:14.176321+0000","last_unstale":"2024-08-17T18:28:44.969999+0000","last_undegraded":"2024-08-17T18:28:44.969999+0000","last_fullsized":"2024-08-17T18:28:44.969999+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:40:55.297225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":229,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.969386+0000","last_change":"2024-08-17T18:22:36.834643+0000","last_active":"2024-08-17T18:28:44.969386+0000","last_peered":"2024-08-17T18:28:44.969386+0000","last_clean":"2024-08-17T18:28:44.969386+0000","last_became_active":"2024-08-17T18:22:36.834279+0000","last_became_peered":"2024-08-17T18:22:36.834279+0000","last_unstale":"2024-08-17T18:28:44.969386+0000","last_undegraded":"2024-08-17T18:28:44.969386+0000","last_fullsized":"2024-08-17T18:28:44.969386+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:29:36.002584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2899,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.864641+0000","last_change":"2024-08-17T18:18:29.742742+0000","last_active":"2024-08-17T18:28:44.864641+0000","last_peered":"2024-08-17T18:28:44.864641+0000","last_clean":"2024-08-17T18:28:44.864641+0000","last_became_active":"2024-08-17T18:18:29.742393+0000","last_became_peered":"2024-08-17T18:18:29.742393+0000","last_unstale":"2024-08-17T18:28:44.864641+0000","last_undegraded":"2024-08-17T18:28:44.864641+0000","last_fullsized":"2024-08-17T18:28:44.864641+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:57:32.858608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"347'149","reported_seq":905,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.969763+0000","last_change":"2024-08-17T18:23:39.370457+0000","last_active":"2024-08-17T18:28:44.969763+0000","last_peered":"2024-08-17T18:28:44.969763+0000","last_clean":"2024-08-17T18:28:44.969763+0000","last_became_active":"2024-08-17T18:23:39.370144+0000","last_became_peered":"2024-08-17T18:23:39.370144+0000","last_unstale":"2024-08-17T18:28:44.969763+0000","last_undegraded":"2024-08-17T18:28:44.969763+0000","last_fullsized":"2024-08-17T18:28:44.969763+0000","mapping_epoch":1736,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1737,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:20:35.854154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:20:35.854154+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:39:35.866331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":126,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:45.494554+0000","last_change":"2024-08-17T18:23:59.742951+0000","last_active":"2024-08-17T18:28:45.494554+0000","last_peered":"2024-08-17T18:28:45.494554+0000","last_clean":"2024-08-17T18:28:45.494554+0000","last_became_active":"2024-08-17T18:23:59.742510+0000","last_became_peered":"2024-08-17T18:23:59.742510+0000","last_unstale":"2024-08-17T18:28:45.494554+0000","last_undegraded":"2024-08-17T18:28:45.494554+0000","last_fullsized":"2024-08-17T18:28:45.494554+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:19:12.943501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":126,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:45.494293+0000","last_change":"2024-08-17T18:23:59.740786+0000","last_active":"2024-08-17T18:28:45.494293+0000","last_peered":"2024-08-17T18:28:45.494293+0000","last_clean":"2024-08-17T18:28:45.494293+0000","last_became_active":"2024-08-17T18:23:59.740340+0000","last_became_peered":"2024-08-17T18:23:59.740340+0000","last_unstale":"2024-08-17T18:28:45.494293+0000","last_undegraded":"2024-08-17T18:28:45.494293+0000","last_fullsized":"2024-08-17T18:28:45.494293+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:52:27.108557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1111,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:00.943479+0000","last_change":"2024-08-17T18:27:50.140301+0000","last_active":"2024-08-17T18:28:00.943479+0000","last_peered":"2024-08-17T18:28:00.943479+0000","last_clean":"2024-08-17T18:28:00.943479+0000","last_became_active":"2024-08-17T18:27:50.139980+0000","last_became_peered":"2024-08-17T18:27:50.139980+0000","last_unstale":"2024-08-17T18:28:00.943479+0000","last_undegraded":"2024-08-17T18:28:00.943479+0000","last_fullsized":"2024-08-17T18:28:00.943479+0000","mapping_epoch":1791,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:57:47.317296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":76,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.526043+0000","last_change":"2024-08-17T18:23:36.318955+0000","last_active":"2024-08-17T18:28:02.526043+0000","last_peered":"2024-08-17T18:28:02.526043+0000","last_clean":"2024-08-17T18:28:02.526043+0000","last_became_active":"2024-08-17T18:23:36.318684+0000","last_became_peered":"2024-08-17T18:23:36.318684+0000","last_unstale":"2024-08-17T18:28:02.526043+0000","last_undegraded":"2024-08-17T18:28:02.526043+0000","last_fullsized":"2024-08-17T18:28:02.526043+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:47.191868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":287,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.864346+0000","last_change":"2024-08-17T18:22:36.835833+0000","last_active":"2024-08-17T18:28:44.864346+0000","last_peered":"2024-08-17T18:28:44.864346+0000","last_clean":"2024-08-17T18:28:44.864346+0000","last_became_active":"2024-08-17T18:22:36.835514+0000","last_became_peered":"2024-08-17T18:22:36.835514+0000","last_unstale":"2024-08-17T18:28:44.864346+0000","last_undegraded":"2024-08-17T18:28:44.864346+0000","last_fullsized":"2024-08-17T18:28:44.864346+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:43:53.668287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":387,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:01.501890+0000","last_change":"2024-08-17T18:19:26.374702+0000","last_active":"2024-08-17T18:28:01.501890+0000","last_peered":"2024-08-17T18:28:01.501890+0000","last_clean":"2024-08-17T18:28:01.501890+0000","last_became_active":"2024-08-17T18:15:53.304106+0000","last_became_peered":"2024-08-17T18:15:53.304106+0000","last_unstale":"2024-08-17T18:28:01.501890+0000","last_undegraded":"2024-08-17T18:28:01.501890+0000","last_fullsized":"2024-08-17T18:28:01.501890+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:21.409368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":814,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:00.943397+0000","last_change":"2024-08-17T18:27:51.487506+0000","last_active":"2024-08-17T18:28:00.943397+0000","last_peered":"2024-08-17T18:28:00.943397+0000","last_clean":"2024-08-17T18:28:00.943397+0000","last_became_active":"2024-08-17T18:27:49.883665+0000","last_became_peered":"2024-08-17T18:27:49.883665+0000","last_unstale":"2024-08-17T18:28:00.943397+0000","last_undegraded":"2024-08-17T18:28:00.943397+0000","last_fullsized":"2024-08-17T18:28:00.943397+0000","mapping_epoch":1791,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1792,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:27:51.487383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:27:51.487383+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:47:51.031707+0000","scrub_duration":0.001833096,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"347'149","reported_seq":1093,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:01.502040+0000","last_change":"2024-08-17T18:22:36.836656+0000","last_active":"2024-08-17T18:28:01.502040+0000","last_peered":"2024-08-17T18:28:01.502040+0000","last_clean":"2024-08-17T18:28:01.502040+0000","last_became_active":"2024-08-17T18:22:36.835810+0000","last_became_peered":"2024-08-17T18:22:36.835810+0000","last_unstale":"2024-08-17T18:28:01.502040+0000","last_undegraded":"2024-08-17T18:28:01.502040+0000","last_fullsized":"2024-08-17T18:28:01.502040+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:55.255983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:55.255983+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:59:27.835153+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":217,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.970025+0000","last_change":"2024-08-17T18:20:31.939871+0000","last_active":"2024-08-17T18:28:44.970025+0000","last_peered":"2024-08-17T18:28:44.970025+0000","last_clean":"2024-08-17T18:28:44.970025+0000","last_became_active":"2024-08-17T18:20:31.939535+0000","last_became_peered":"2024-08-17T18:20:31.939535+0000","last_unstale":"2024-08-17T18:28:44.970025+0000","last_undegraded":"2024-08-17T18:28:44.970025+0000","last_fullsized":"2024-08-17T18:28:44.970025+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:43:31.630668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":297,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.864333+0000","last_change":"2024-08-17T18:25:14.036186+0000","last_active":"2024-08-17T18:28:44.864333+0000","last_peered":"2024-08-17T18:28:44.864333+0000","last_clean":"2024-08-17T18:28:44.864333+0000","last_became_active":"2024-08-17T18:25:14.035138+0000","last_became_peered":"2024-08-17T18:25:14.035138+0000","last_unstale":"2024-08-17T18:28:44.864333+0000","last_undegraded":"2024-08-17T18:28:44.864333+0000","last_fullsized":"2024-08-17T18:28:44.864333+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:51.645934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1873,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.186377+0000","last_change":"2024-08-17T18:25:14.026973+0000","last_active":"2024-08-17T18:28:02.186377+0000","last_peered":"2024-08-17T18:28:02.186377+0000","last_clean":"2024-08-17T18:28:02.186377+0000","last_became_active":"2024-08-17T18:25:14.026478+0000","last_became_peered":"2024-08-17T18:25:14.026478+0000","last_unstale":"2024-08-17T18:28:02.186377+0000","last_undegraded":"2024-08-17T18:28:02.186377+0000","last_fullsized":"2024-08-17T18:28:02.186377+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:31.568544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1828,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:20.193370+0000","last_change":"2024-08-17T18:15:53.244851+0000","last_active":"2024-08-17T18:28:20.193370+0000","last_peered":"2024-08-17T18:28:20.193370+0000","last_clean":"2024-08-17T18:28:20.193370+0000","last_became_active":"2024-08-17T18:15:53.240074+0000","last_became_peered":"2024-08-17T18:15:53.240074+0000","last_unstale":"2024-08-17T18:28:20.193370+0000","last_undegraded":"2024-08-17T18:28:20.193370+0000","last_fullsized":"2024-08-17T18:28:20.193370+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:09:00.168413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"347'149","reported_seq":1057,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:07.378145+0000","last_change":"2024-08-17T18:25:55.776006+0000","last_active":"2024-08-17T18:28:07.378145+0000","last_peered":"2024-08-17T18:28:07.378145+0000","last_clean":"2024-08-17T18:28:07.378145+0000","last_became_active":"2024-08-17T18:25:55.775559+0000","last_became_peered":"2024-08-17T18:25:55.775559+0000","last_unstale":"2024-08-17T18:28:07.378145+0000","last_undegraded":"2024-08-17T18:28:07.378145+0000","last_fullsized":"2024-08-17T18:28:07.378145+0000","mapping_epoch":1763,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1764,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:59.307356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:59.307356+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:52:14.784551+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1831,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.186311+0000","last_change":"2024-08-17T18:20:31.942019+0000","last_active":"2024-08-17T18:28:02.186311+0000","last_peered":"2024-08-17T18:28:02.186311+0000","last_clean":"2024-08-17T18:28:02.186311+0000","last_became_active":"2024-08-17T18:20:31.941701+0000","last_became_peered":"2024-08-17T18:20:31.941701+0000","last_unstale":"2024-08-17T18:28:02.186311+0000","last_undegraded":"2024-08-17T18:28:02.186311+0000","last_fullsized":"2024-08-17T18:28:02.186311+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:58.908049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1808,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.525969+0000","last_change":"2024-08-17T18:25:14.027871+0000","last_active":"2024-08-17T18:28:02.525969+0000","last_peered":"2024-08-17T18:28:02.525969+0000","last_clean":"2024-08-17T18:28:02.525969+0000","last_became_active":"2024-08-17T18:25:14.027353+0000","last_became_peered":"2024-08-17T18:25:14.027353+0000","last_unstale":"2024-08-17T18:28:02.525969+0000","last_undegraded":"2024-08-17T18:28:02.525969+0000","last_fullsized":"2024-08-17T18:28:02.525969+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:06.568798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"347'149","reported_seq":1054,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:00.943676+0000","last_change":"2024-08-17T18:18:33.757347+0000","last_active":"2024-08-17T18:28:00.943676+0000","last_peered":"2024-08-17T18:28:00.943676+0000","last_clean":"2024-08-17T18:28:00.943676+0000","last_became_active":"2024-08-17T18:18:33.756960+0000","last_became_peered":"2024-08-17T18:18:33.756960+0000","last_unstale":"2024-08-17T18:28:00.943676+0000","last_undegraded":"2024-08-17T18:28:00.943676+0000","last_fullsized":"2024-08-17T18:28:00.943676+0000","mapping_epoch":1670,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1671,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:52.271487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:52.271487+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:37:36.389244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1825,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.186164+0000","last_change":"2024-08-17T18:22:36.837889+0000","last_active":"2024-08-17T18:28:02.186164+0000","last_peered":"2024-08-17T18:28:02.186164+0000","last_clean":"2024-08-17T18:28:02.186164+0000","last_became_active":"2024-08-17T18:22:36.837387+0000","last_became_peered":"2024-08-17T18:22:36.837387+0000","last_unstale":"2024-08-17T18:28:02.186164+0000","last_undegraded":"2024-08-17T18:28:02.186164+0000","last_fullsized":"2024-08-17T18:28:02.186164+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:11:29.265925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":313,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.864581+0000","last_change":"2024-08-17T18:23:59.770482+0000","last_active":"2024-08-17T18:28:44.864581+0000","last_peered":"2024-08-17T18:28:44.864581+0000","last_clean":"2024-08-17T18:28:44.864581+0000","last_became_active":"2024-08-17T18:23:59.747059+0000","last_became_peered":"2024-08-17T18:23:59.747059+0000","last_unstale":"2024-08-17T18:28:44.864581+0000","last_undegraded":"2024-08-17T18:28:44.864581+0000","last_fullsized":"2024-08-17T18:28:44.864581+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:04:17.042903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":126,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:45.494469+0000","last_change":"2024-08-17T18:23:59.745024+0000","last_active":"2024-08-17T18:28:45.494469+0000","last_peered":"2024-08-17T18:28:45.494469+0000","last_clean":"2024-08-17T18:28:45.494469+0000","last_became_active":"2024-08-17T18:23:59.741633+0000","last_became_peered":"2024-08-17T18:23:59.741633+0000","last_unstale":"2024-08-17T18:28:45.494469+0000","last_undegraded":"2024-08-17T18:28:45.494469+0000","last_fullsized":"2024-08-17T18:28:45.494469+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:41.514649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":395,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:01.502339+0000","last_change":"2024-08-17T18:22:36.836017+0000","last_active":"2024-08-17T18:28:01.502339+0000","last_peered":"2024-08-17T18:28:01.502339+0000","last_clean":"2024-08-17T18:28:01.502339+0000","last_became_active":"2024-08-17T18:22:36.835725+0000","last_became_peered":"2024-08-17T18:22:36.835725+0000","last_unstale":"2024-08-17T18:28:01.502339+0000","last_undegraded":"2024-08-17T18:28:01.502339+0000","last_fullsized":"2024-08-17T18:28:01.502339+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:18:07.395241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":76,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.525724+0000","last_change":"2024-08-17T18:23:36.324099+0000","last_active":"2024-08-17T18:28:02.525724+0000","last_peered":"2024-08-17T18:28:02.525724+0000","last_clean":"2024-08-17T18:28:02.525724+0000","last_became_active":"2024-08-17T18:23:36.323739+0000","last_became_peered":"2024-08-17T18:23:36.323739+0000","last_unstale":"2024-08-17T18:28:02.525724+0000","last_undegraded":"2024-08-17T18:28:02.525724+0000","last_fullsized":"2024-08-17T18:28:02.525724+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:54:56.319013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":222,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:00.943701+0000","last_change":"2024-08-17T18:27:49.885619+0000","last_active":"2024-08-17T18:28:00.943701+0000","last_peered":"2024-08-17T18:28:00.943701+0000","last_clean":"2024-08-17T18:28:00.943701+0000","last_became_active":"2024-08-17T18:27:49.885246+0000","last_became_peered":"2024-08-17T18:27:49.885246+0000","last_unstale":"2024-08-17T18:28:00.943701+0000","last_undegraded":"2024-08-17T18:28:00.943701+0000","last_fullsized":"2024-08-17T18:28:00.943701+0000","mapping_epoch":1791,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:48:04.478804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1883,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:20.193344+0000","last_change":"2024-08-17T18:25:14.040053+0000","last_active":"2024-08-17T18:28:20.193344+0000","last_peered":"2024-08-17T18:28:20.193344+0000","last_clean":"2024-08-17T18:28:20.193344+0000","last_became_active":"2024-08-17T18:25:14.039807+0000","last_became_peered":"2024-08-17T18:25:14.039807+0000","last_unstale":"2024-08-17T18:28:20.193344+0000","last_undegraded":"2024-08-17T18:28:20.193344+0000","last_fullsized":"2024-08-17T18:28:20.193344+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:05:53.486096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":191,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:00.943912+0000","last_change":"2024-08-17T18:20:32.006404+0000","last_active":"2024-08-17T18:28:00.943912+0000","last_peered":"2024-08-17T18:28:00.943912+0000","last_clean":"2024-08-17T18:28:00.943912+0000","last_became_active":"2024-08-17T18:20:31.947053+0000","last_became_peered":"2024-08-17T18:20:31.947053+0000","last_unstale":"2024-08-17T18:28:00.943912+0000","last_undegraded":"2024-08-17T18:28:00.943912+0000","last_fullsized":"2024-08-17T18:28:00.943912+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:21:06.537725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":399,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:37.381625+0000","last_change":"2024-08-17T18:23:36.318964+0000","last_active":"2024-08-17T18:28:37.381625+0000","last_peered":"2024-08-17T18:28:37.381625+0000","last_clean":"2024-08-17T18:28:37.381625+0000","last_became_active":"2024-08-17T18:23:36.318595+0000","last_became_peered":"2024-08-17T18:23:36.318595+0000","last_unstale":"2024-08-17T18:28:37.381625+0000","last_undegraded":"2024-08-17T18:28:37.381625+0000","last_fullsized":"2024-08-17T18:28:37.381625+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:29:16.221103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":217,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:00.944020+0000","last_change":"2024-08-17T18:23:59.752171+0000","last_active":"2024-08-17T18:28:00.944020+0000","last_peered":"2024-08-17T18:28:00.944020+0000","last_clean":"2024-08-17T18:28:00.944020+0000","last_became_active":"2024-08-17T18:23:59.745970+0000","last_became_peered":"2024-08-17T18:23:59.745970+0000","last_unstale":"2024-08-17T18:28:00.944020+0000","last_undegraded":"2024-08-17T18:28:00.944020+0000","last_fullsized":"2024-08-17T18:28:00.944020+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:24:52.861533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1955,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.525964+0000","last_change":"2024-08-17T18:24:22.107318+0000","last_active":"2024-08-17T18:28:02.525964+0000","last_peered":"2024-08-17T18:28:02.525964+0000","last_clean":"2024-08-17T18:28:02.525964+0000","last_became_active":"2024-08-17T18:24:14.847882+0000","last_became_peered":"2024-08-17T18:24:14.847882+0000","last_unstale":"2024-08-17T18:28:02.525964+0000","last_undegraded":"2024-08-17T18:28:02.525964+0000","last_fullsized":"2024-08-17T18:28:02.525964+0000","mapping_epoch":1746,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:45:33.740721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":290,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.865430+0000","last_change":"2024-08-17T18:21:12.645070+0000","last_active":"2024-08-17T18:28:44.865430+0000","last_peered":"2024-08-17T18:28:44.865430+0000","last_clean":"2024-08-17T18:28:44.865430+0000","last_became_active":"2024-08-17T18:21:12.644748+0000","last_became_peered":"2024-08-17T18:21:12.644748+0000","last_unstale":"2024-08-17T18:28:44.865430+0000","last_undegraded":"2024-08-17T18:28:44.865430+0000","last_fullsized":"2024-08-17T18:28:44.865430+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:19:08.683431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":489,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:00.943492+0000","last_change":"2024-08-17T18:27:49.885293+0000","last_active":"2024-08-17T18:28:00.943492+0000","last_peered":"2024-08-17T18:28:00.943492+0000","last_clean":"2024-08-17T18:28:00.943492+0000","last_became_active":"2024-08-17T18:27:49.884978+0000","last_became_peered":"2024-08-17T18:27:49.884978+0000","last_unstale":"2024-08-17T18:28:00.943492+0000","last_undegraded":"2024-08-17T18:28:00.943492+0000","last_fullsized":"2024-08-17T18:28:00.943492+0000","mapping_epoch":1791,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:21:38.469502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":125,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:45.494878+0000","last_change":"2024-08-17T18:23:59.740450+0000","last_active":"2024-08-17T18:28:45.494878+0000","last_peered":"2024-08-17T18:28:45.494878+0000","last_clean":"2024-08-17T18:28:45.494878+0000","last_became_active":"2024-08-17T18:23:59.740125+0000","last_became_peered":"2024-08-17T18:23:59.740125+0000","last_unstale":"2024-08-17T18:28:45.494878+0000","last_undegraded":"2024-08-17T18:28:45.494878+0000","last_fullsized":"2024-08-17T18:28:45.494878+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:26:38.858210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":126,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:45.494293+0000","last_change":"2024-08-17T18:23:59.740550+0000","last_active":"2024-08-17T18:28:45.494293+0000","last_peered":"2024-08-17T18:28:45.494293+0000","last_clean":"2024-08-17T18:28:45.494293+0000","last_became_active":"2024-08-17T18:23:59.740226+0000","last_became_peered":"2024-08-17T18:23:59.740226+0000","last_unstale":"2024-08-17T18:28:45.494293+0000","last_undegraded":"2024-08-17T18:28:45.494293+0000","last_fullsized":"2024-08-17T18:28:45.494293+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:35.745256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":389,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:45.494971+0000","last_change":"2024-08-17T18:18:29.900128+0000","last_active":"2024-08-17T18:28:45.494971+0000","last_peered":"2024-08-17T18:28:45.494971+0000","last_clean":"2024-08-17T18:28:45.494971+0000","last_became_active":"2024-08-17T18:18:29.899279+0000","last_became_peered":"2024-08-17T18:18:29.899279+0000","last_unstale":"2024-08-17T18:28:45.494971+0000","last_undegraded":"2024-08-17T18:28:45.494971+0000","last_fullsized":"2024-08-17T18:28:45.494971+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:01:29.045498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":181,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:00.943761+0000","last_change":"2024-08-17T18:15:53.271113+0000","last_active":"2024-08-17T18:28:00.943761+0000","last_peered":"2024-08-17T18:28:00.943761+0000","last_clean":"2024-08-17T18:28:00.943761+0000","last_became_active":"2024-08-17T18:15:53.230983+0000","last_became_peered":"2024-08-17T18:15:53.230983+0000","last_unstale":"2024-08-17T18:28:00.943761+0000","last_undegraded":"2024-08-17T18:28:00.943761+0000","last_fullsized":"2024-08-17T18:28:00.943761+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:00:35.315798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":308,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.865033+0000","last_change":"2024-08-17T18:25:14.030279+0000","last_active":"2024-08-17T18:28:44.865033+0000","last_peered":"2024-08-17T18:28:44.865033+0000","last_clean":"2024-08-17T18:28:44.865033+0000","last_became_active":"2024-08-17T18:25:14.027960+0000","last_became_peered":"2024-08-17T18:25:14.027960+0000","last_unstale":"2024-08-17T18:28:44.865033+0000","last_undegraded":"2024-08-17T18:28:44.865033+0000","last_fullsized":"2024-08-17T18:28:44.865033+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:20:46.317663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":12930,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234954,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":872784,"kb_used_data":11524,"kb_used_omap":1,"kb_used_meta":861246,"kb_avail":92872368,"statfs":{"total":95995035648,"available":95101304832,"internally_reserved":0,"allocated":11800576,"data_stored":6626644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":881916363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1724,"seq":7404523618380,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":207360,"kb_used_data":11188,"kb_used_omap":123,"kb_used_meta":196036,"kb_avail":93537792,"statfs":{"total":95995035648,"available":95782699008,"internally_reserved":0,"allocated":11456512,"data_stored":6291615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":126809,"internal_metadata":200741031},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565833,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1302336,"kb_used_data":11764,"kb_used_omap":72,"kb_used_meta":1290487,"kb_avail":92442816,"statfs":{"total":95995035648,"available":94661443584,"internally_reserved":0,"allocated":12046336,"data_stored":6885609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":1321459182},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":1399,"seq":6008659247435,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":982084,"kb_used_data":11448,"kb_used_omap":131,"kb_used_meta":970492,"kb_avail":92763068,"statfs":{"total":95995035648,"available":94989381632,"internally_reserved":0,"allocated":11722752,"data_stored":6558455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134291,"internal_metadata":993784685},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743187223,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":981680,"kb_used_data":11172,"kb_used_omap":66,"kb_used_meta":970429,"kb_avail":92763472,"statfs":{"total":95995035648,"available":94989795328,"internally_reserved":0,"allocated":11440128,"data_stored":6286801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":993719979},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448862,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":917696,"kb_used_data":11188,"kb_used_omap":75,"kb_used_meta":906420,"kb_avail":92827456,"statfs":{"total":95995035648,"available":95055314944,"internally_reserved":0,"allocated":11456512,"data_stored":6295755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":928174271},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208758227,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1414340,"kb_used_data":11768,"kb_used_omap":1,"kb_used_meta":1402558,"kb_avail":92330812,"statfs":{"total":95995035648,"available":94546751488,"internally_reserved":0,"allocated":12050432,"data_stored":6885031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1436219846},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019863,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1368384,"kb_used_data":11188,"kb_used_omap":1,"kb_used_meta":1357182,"kb_avail":92376768,"statfs":{"total":95995035648,"available":94593810432,"internally_reserved":0,"allocated":11456512,"data_stored":6290525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1389754822},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T18:28:49.457 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 8270436c-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T18:28:49.659 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T18:28:50.111 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:49 smithi136 ceph-mon[109861]: pgmap v3637: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:50.111 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:49 smithi136 ceph-mon[109861]: from='client.47360 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:28:50.111 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:49 smithi136 ceph-mon[115078]: pgmap v3637: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:50.111 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:49 smithi136 ceph-mon[115078]: from='client.47360 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:28:50.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:49 smithi165 ceph-mon[105517]: pgmap v3637: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:50.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:49 smithi165 ceph-mon[105517]: from='client.47360 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:28:51.573 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-17T18:28:51.574 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-08-17T18:28:52.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:51 smithi136 ceph-mon[115078]: pgmap v3638: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:52.197 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:51 smithi136 ceph-mon[115078]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:28:52.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:51 smithi136 ceph-mon[109861]: pgmap v3638: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:52.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:51 smithi136 ceph-mon[109861]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:28:52.219 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":3638,"stamp":"2024-08-17T18:28:50.681259+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595941,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":1193799,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1698,"ondisk_log_size":1698,"up":249,"acting":249,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":249,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8046688,"kb_used_data":91264,"kb_used_omap":474,"kb_used_meta":7954853,"kb_avail":741914528,"statfs":{"total":767960285184,"available":759720476672,"internally_reserved":0,"allocated":93454336,"data_stored":52137173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":485693,"internal_metadata":8145770179},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002488"},"pg_stats":[{"pgid":"3.1f","version":"70'4","reported_seq":132,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:45.494780+0000","last_change":"2024-08-17T18:23:59.758838+0000","last_active":"2024-08-17T18:28:45.494780+0000","last_peered":"2024-08-17T18:28:45.494780+0000","last_clean":"2024-08-17T18:28:45.494780+0000","last_became_active":"2024-08-17T18:23:59.748445+0000","last_became_peered":"2024-08-17T18:23:59.748445+0000","last_unstale":"2024-08-17T18:28:45.494780+0000","last_undegraded":"2024-08-17T18:28:45.494780+0000","last_fullsized":"2024-08-17T18:28:45.494780+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:08:27.581912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1862,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.185729+0000","last_change":"2024-08-17T18:23:36.324372+0000","last_active":"2024-08-17T18:28:02.185729+0000","last_peered":"2024-08-17T18:28:02.185729+0000","last_clean":"2024-08-17T18:28:02.185729+0000","last_became_active":"2024-08-17T18:23:36.323893+0000","last_became_peered":"2024-08-17T18:23:36.323893+0000","last_unstale":"2024-08-17T18:28:02.185729+0000","last_undegraded":"2024-08-17T18:28:02.185729+0000","last_fullsized":"2024-08-17T18:28:02.185729+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:10:52.829170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'1","reported_seq":291,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.864992+0000","last_change":"2024-08-17T18:23:36.319358+0000","last_active":"2024-08-17T18:28:44.864992+0000","last_peered":"2024-08-17T18:28:44.864992+0000","last_clean":"2024-08-17T18:28:44.864992+0000","last_became_active":"2024-08-17T18:23:36.318521+0000","last_became_peered":"2024-08-17T18:23:36.318521+0000","last_unstale":"2024-08-17T18:28:44.864992+0000","last_undegraded":"2024-08-17T18:28:44.864992+0000","last_fullsized":"2024-08-17T18:28:44.864992+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:56:48.984477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":283,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.864890+0000","last_change":"2024-08-17T18:23:59.741806+0000","last_active":"2024-08-17T18:28:44.864890+0000","last_peered":"2024-08-17T18:28:44.864890+0000","last_clean":"2024-08-17T18:28:44.864890+0000","last_became_active":"2024-08-17T18:23:59.741479+0000","last_became_peered":"2024-08-17T18:23:59.741479+0000","last_unstale":"2024-08-17T18:28:44.864890+0000","last_undegraded":"2024-08-17T18:28:44.864890+0000","last_fullsized":"2024-08-17T18:28:44.864890+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:02:39.072277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"70'2","reported_seq":371,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:01.502242+0000","last_change":"2024-08-17T18:22:36.836343+0000","last_active":"2024-08-17T18:28:01.502242+0000","last_peered":"2024-08-17T18:28:01.502242+0000","last_clean":"2024-08-17T18:28:01.502242+0000","last_became_active":"2024-08-17T18:22:36.835580+0000","last_became_peered":"2024-08-17T18:22:36.835580+0000","last_unstale":"2024-08-17T18:28:01.502242+0000","last_undegraded":"2024-08-17T18:28:01.502242+0000","last_fullsized":"2024-08-17T18:28:01.502242+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:05:51.737218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1515,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.185705+0000","last_change":"2024-08-17T18:27:37.426405+0000","last_active":"2024-08-17T18:28:02.185705+0000","last_peered":"2024-08-17T18:28:02.185705+0000","last_clean":"2024-08-17T18:28:02.185705+0000","last_became_active":"2024-08-17T18:27:37.426039+0000","last_became_peered":"2024-08-17T18:27:37.426039+0000","last_unstale":"2024-08-17T18:28:02.185705+0000","last_undegraded":"2024-08-17T18:28:02.185705+0000","last_fullsized":"2024-08-17T18:28:02.185705+0000","mapping_epoch":1786,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1787,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:14:11.039422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"70'2","reported_seq":382,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:01.502347+0000","last_change":"2024-08-17T18:20:32.094881+0000","last_active":"2024-08-17T18:28:01.502347+0000","last_peered":"2024-08-17T18:28:01.502347+0000","last_clean":"2024-08-17T18:28:01.502347+0000","last_became_active":"2024-08-17T18:20:32.077429+0000","last_became_peered":"2024-08-17T18:20:32.077429+0000","last_unstale":"2024-08-17T18:28:01.502347+0000","last_undegraded":"2024-08-17T18:28:01.502347+0000","last_fullsized":"2024-08-17T18:28:01.502347+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:08:26.290851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1867,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.185662+0000","last_change":"2024-08-17T18:25:14.026295+0000","last_active":"2024-08-17T18:28:02.185662+0000","last_peered":"2024-08-17T18:28:02.185662+0000","last_clean":"2024-08-17T18:28:02.185662+0000","last_became_active":"2024-08-17T18:25:14.025999+0000","last_became_peered":"2024-08-17T18:25:14.025999+0000","last_unstale":"2024-08-17T18:28:02.185662+0000","last_undegraded":"2024-08-17T18:28:02.185662+0000","last_fullsized":"2024-08-17T18:28:02.185662+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:55:16.975299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"70'1","reported_seq":125,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:45.494461+0000","last_change":"2024-08-17T18:23:59.745317+0000","last_active":"2024-08-17T18:28:45.494461+0000","last_peered":"2024-08-17T18:28:45.494461+0000","last_clean":"2024-08-17T18:28:45.494461+0000","last_became_active":"2024-08-17T18:23:59.743236+0000","last_became_peered":"2024-08-17T18:23:59.743236+0000","last_unstale":"2024-08-17T18:28:45.494461+0000","last_undegraded":"2024-08-17T18:28:45.494461+0000","last_fullsized":"2024-08-17T18:28:45.494461+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:11.253490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":152,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.865001+0000","last_change":"2024-08-17T18:22:53.938704+0000","last_active":"2024-08-17T18:28:44.865001+0000","last_peered":"2024-08-17T18:28:44.865001+0000","last_clean":"2024-08-17T18:28:44.865001+0000","last_became_active":"2024-08-17T18:22:53.938406+0000","last_became_peered":"2024-08-17T18:22:53.938406+0000","last_unstale":"2024-08-17T18:28:44.865001+0000","last_undegraded":"2024-08-17T18:28:44.865001+0000","last_fullsized":"2024-08-17T18:28:44.865001+0000","mapping_epoch":1727,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1728,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:58:21.566014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"70'1","reported_seq":1844,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.185939+0000","last_change":"2024-08-17T18:20:31.942863+0000","last_active":"2024-08-17T18:28:02.185939+0000","last_peered":"2024-08-17T18:28:02.185939+0000","last_clean":"2024-08-17T18:28:02.185939+0000","last_became_active":"2024-08-17T18:20:31.941281+0000","last_became_peered":"2024-08-17T18:20:31.941281+0000","last_unstale":"2024-08-17T18:28:02.185939+0000","last_undegraded":"2024-08-17T18:28:02.185939+0000","last_fullsized":"2024-08-17T18:28:02.185939+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:48:05.556975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":283,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.864829+0000","last_change":"2024-08-17T18:20:31.933142+0000","last_active":"2024-08-17T18:28:44.864829+0000","last_peered":"2024-08-17T18:28:44.864829+0000","last_clean":"2024-08-17T18:28:44.864829+0000","last_became_active":"2024-08-17T18:20:31.932798+0000","last_became_peered":"2024-08-17T18:20:31.932798+0000","last_unstale":"2024-08-17T18:28:44.864829+0000","last_undegraded":"2024-08-17T18:28:44.864829+0000","last_fullsized":"2024-08-17T18:28:44.864829+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:32:51.059352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":217,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.969493+0000","last_change":"2024-08-17T18:20:31.937627+0000","last_active":"2024-08-17T18:28:44.969493+0000","last_peered":"2024-08-17T18:28:44.969493+0000","last_clean":"2024-08-17T18:28:44.969493+0000","last_became_active":"2024-08-17T18:20:31.937361+0000","last_became_peered":"2024-08-17T18:20:31.937361+0000","last_unstale":"2024-08-17T18:28:44.969493+0000","last_undegraded":"2024-08-17T18:28:44.969493+0000","last_fullsized":"2024-08-17T18:28:44.969493+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:01:26.309656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":126,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:45.494663+0000","last_change":"2024-08-17T18:23:59.743052+0000","last_active":"2024-08-17T18:28:45.494663+0000","last_peered":"2024-08-17T18:28:45.494663+0000","last_clean":"2024-08-17T18:28:45.494663+0000","last_became_active":"2024-08-17T18:23:59.742158+0000","last_became_peered":"2024-08-17T18:23:59.742158+0000","last_unstale":"2024-08-17T18:28:45.494663+0000","last_undegraded":"2024-08-17T18:28:45.494663+0000","last_fullsized":"2024-08-17T18:28:45.494663+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:34:34.249644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":304,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.864463+0000","last_change":"2024-08-17T18:25:14.176708+0000","last_active":"2024-08-17T18:28:44.864463+0000","last_peered":"2024-08-17T18:28:44.864463+0000","last_clean":"2024-08-17T18:28:44.864463+0000","last_became_active":"2024-08-17T18:25:14.176239+0000","last_became_peered":"2024-08-17T18:25:14.176239+0000","last_unstale":"2024-08-17T18:28:44.864463+0000","last_undegraded":"2024-08-17T18:28:44.864463+0000","last_fullsized":"2024-08-17T18:28:44.864463+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:08:21.826456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1861,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.952593+0000","last_change":"2024-08-17T18:22:36.867005+0000","last_active":"2024-08-17T18:28:44.952593+0000","last_peered":"2024-08-17T18:28:44.952593+0000","last_clean":"2024-08-17T18:28:44.952593+0000","last_became_active":"2024-08-17T18:22:36.866297+0000","last_became_peered":"2024-08-17T18:22:36.866297+0000","last_unstale":"2024-08-17T18:28:44.952593+0000","last_undegraded":"2024-08-17T18:28:44.952593+0000","last_fullsized":"2024-08-17T18:28:44.952593+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:09:18.460006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":306,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.864569+0000","last_change":"2024-08-17T18:25:14.035767+0000","last_active":"2024-08-17T18:28:44.864569+0000","last_peered":"2024-08-17T18:28:44.864569+0000","last_clean":"2024-08-17T18:28:44.864569+0000","last_became_active":"2024-08-17T18:25:14.034592+0000","last_became_peered":"2024-08-17T18:25:14.034592+0000","last_unstale":"2024-08-17T18:28:44.864569+0000","last_undegraded":"2024-08-17T18:28:44.864569+0000","last_fullsized":"2024-08-17T18:28:44.864569+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:22:00.711481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":125,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:45.494829+0000","last_change":"2024-08-17T18:23:59.744879+0000","last_active":"2024-08-17T18:28:45.494829+0000","last_peered":"2024-08-17T18:28:45.494829+0000","last_clean":"2024-08-17T18:28:45.494829+0000","last_became_active":"2024-08-17T18:23:59.743333+0000","last_became_peered":"2024-08-17T18:23:59.743333+0000","last_unstale":"2024-08-17T18:28:45.494829+0000","last_undegraded":"2024-08-17T18:28:45.494829+0000","last_fullsized":"2024-08-17T18:28:45.494829+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:02:01.546706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":117,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:12.378698+0000","last_change":"2024-08-17T18:22:36.836745+0000","last_active":"2024-08-17T18:28:12.378698+0000","last_peered":"2024-08-17T18:28:12.378698+0000","last_clean":"2024-08-17T18:28:12.378698+0000","last_became_active":"2024-08-17T18:22:36.835161+0000","last_became_peered":"2024-08-17T18:22:36.835161+0000","last_unstale":"2024-08-17T18:28:12.378698+0000","last_undegraded":"2024-08-17T18:28:12.378698+0000","last_fullsized":"2024-08-17T18:28:12.378698+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:29:43.303633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":441,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:01.502269+0000","last_change":"2024-08-17T18:25:14.027677+0000","last_active":"2024-08-17T18:28:01.502269+0000","last_peered":"2024-08-17T18:28:01.502269+0000","last_clean":"2024-08-17T18:28:01.502269+0000","last_became_active":"2024-08-17T18:25:14.027309+0000","last_became_peered":"2024-08-17T18:25:14.027309+0000","last_unstale":"2024-08-17T18:28:01.502269+0000","last_undegraded":"2024-08-17T18:28:01.502269+0000","last_fullsized":"2024-08-17T18:28:01.502269+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:15:02.617263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"70'3","reported_seq":91,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:00.943823+0000","last_change":"2024-08-17T18:27:49.885531+0000","last_active":"2024-08-17T18:28:00.943823+0000","last_peered":"2024-08-17T18:28:00.943823+0000","last_clean":"2024-08-17T18:28:00.943823+0000","last_became_active":"2024-08-17T18:27:49.884877+0000","last_became_peered":"2024-08-17T18:27:49.884877+0000","last_unstale":"2024-08-17T18:28:00.943823+0000","last_undegraded":"2024-08-17T18:28:00.943823+0000","last_fullsized":"2024-08-17T18:28:00.943823+0000","mapping_epoch":1791,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:05:23.870213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":101,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.969552+0000","last_change":"2024-08-17T18:25:14.030155+0000","last_active":"2024-08-17T18:28:44.969552+0000","last_peered":"2024-08-17T18:28:44.969552+0000","last_clean":"2024-08-17T18:28:44.969552+0000","last_became_active":"2024-08-17T18:25:14.029876+0000","last_became_peered":"2024-08-17T18:25:14.029876+0000","last_unstale":"2024-08-17T18:28:44.969552+0000","last_undegraded":"2024-08-17T18:28:44.969552+0000","last_fullsized":"2024-08-17T18:28:44.969552+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:04:34.619858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"427'23","reported_seq":1844,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.186222+0000","last_change":"2024-08-17T18:22:36.838096+0000","last_active":"2024-08-17T18:28:02.186222+0000","last_peered":"2024-08-17T18:28:02.186222+0000","last_clean":"2024-08-17T18:28:02.186222+0000","last_became_active":"2024-08-17T18:22:36.835757+0000","last_became_peered":"2024-08-17T18:22:36.835757+0000","last_unstale":"2024-08-17T18:28:02.186222+0000","last_undegraded":"2024-08-17T18:28:02.186222+0000","last_fullsized":"2024-08-17T18:28:02.186222+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:55:31.016446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1835,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.186215+0000","last_change":"2024-08-17T18:25:14.026273+0000","last_active":"2024-08-17T18:28:02.186215+0000","last_peered":"2024-08-17T18:28:02.186215+0000","last_clean":"2024-08-17T18:28:02.186215+0000","last_became_active":"2024-08-17T18:25:14.025964+0000","last_became_peered":"2024-08-17T18:25:14.025964+0000","last_unstale":"2024-08-17T18:28:02.186215+0000","last_undegraded":"2024-08-17T18:28:02.186215+0000","last_fullsized":"2024-08-17T18:28:02.186215+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:43.150679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"347'149","reported_seq":1104,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.864385+0000","last_change":"2024-08-17T18:20:33.927014+0000","last_active":"2024-08-17T18:28:44.864385+0000","last_peered":"2024-08-17T18:28:44.864385+0000","last_clean":"2024-08-17T18:28:44.864385+0000","last_became_active":"2024-08-17T18:20:33.926619+0000","last_became_peered":"2024-08-17T18:20:33.926619+0000","last_unstale":"2024-08-17T18:28:44.864385+0000","last_undegraded":"2024-08-17T18:28:44.864385+0000","last_fullsized":"2024-08-17T18:28:44.864385+0000","mapping_epoch":1697,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1698,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:57.271286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:57.271286+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:35:26.778777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1822,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.185872+0000","last_change":"2024-08-17T18:22:36.838651+0000","last_active":"2024-08-17T18:28:02.185872+0000","last_peered":"2024-08-17T18:28:02.185872+0000","last_clean":"2024-08-17T18:28:02.185872+0000","last_became_active":"2024-08-17T18:22:36.838262+0000","last_became_peered":"2024-08-17T18:22:36.838262+0000","last_unstale":"2024-08-17T18:28:02.185872+0000","last_undegraded":"2024-08-17T18:28:02.185872+0000","last_fullsized":"2024-08-17T18:28:02.185872+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:24:25.119257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1803,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.526126+0000","last_change":"2024-08-17T18:22:36.866053+0000","last_active":"2024-08-17T18:28:02.526126+0000","last_peered":"2024-08-17T18:28:02.526126+0000","last_clean":"2024-08-17T18:28:02.526126+0000","last_became_active":"2024-08-17T18:22:36.865187+0000","last_became_peered":"2024-08-17T18:22:36.865187+0000","last_unstale":"2024-08-17T18:28:02.526126+0000","last_undegraded":"2024-08-17T18:28:02.526126+0000","last_fullsized":"2024-08-17T18:28:02.526126+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:45:34.232504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033881900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"70'1","reported_seq":130,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:45.494723+0000","last_change":"2024-08-17T18:23:59.750520+0000","last_active":"2024-08-17T18:28:45.494723+0000","last_peered":"2024-08-17T18:28:45.494723+0000","last_clean":"2024-08-17T18:28:45.494723+0000","last_became_active":"2024-08-17T18:23:59.742332+0000","last_became_peered":"2024-08-17T18:23:59.742332+0000","last_unstale":"2024-08-17T18:28:45.494723+0000","last_undegraded":"2024-08-17T18:28:45.494723+0000","last_fullsized":"2024-08-17T18:28:45.494723+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:39:39.325194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"347'149","reported_seq":1111,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:05.191604+0000","last_change":"2024-08-17T18:23:39.360135+0000","last_active":"2024-08-17T18:28:05.191604+0000","last_peered":"2024-08-17T18:28:05.191604+0000","last_clean":"2024-08-17T18:28:05.191604+0000","last_became_active":"2024-08-17T18:23:39.359652+0000","last_became_peered":"2024-08-17T18:23:39.359652+0000","last_unstale":"2024-08-17T18:28:05.191604+0000","last_undegraded":"2024-08-17T18:28:05.191604+0000","last_fullsized":"2024-08-17T18:28:05.191604+0000","mapping_epoch":1736,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1737,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:53.309517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:53.309517+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:20:52.590505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":291,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.864683+0000","last_change":"2024-08-17T18:23:59.746854+0000","last_active":"2024-08-17T18:28:44.864683+0000","last_peered":"2024-08-17T18:28:44.864683+0000","last_clean":"2024-08-17T18:28:44.864683+0000","last_became_active":"2024-08-17T18:23:59.746190+0000","last_became_peered":"2024-08-17T18:23:59.746190+0000","last_unstale":"2024-08-17T18:28:44.864683+0000","last_undegraded":"2024-08-17T18:28:44.864683+0000","last_fullsized":"2024-08-17T18:28:44.864683+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:01:53.108037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":101,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.969944+0000","last_change":"2024-08-17T18:25:14.025153+0000","last_active":"2024-08-17T18:28:44.969944+0000","last_peered":"2024-08-17T18:28:44.969944+0000","last_clean":"2024-08-17T18:28:44.969944+0000","last_became_active":"2024-08-17T18:25:14.024700+0000","last_became_peered":"2024-08-17T18:25:14.024700+0000","last_unstale":"2024-08-17T18:28:44.969944+0000","last_undegraded":"2024-08-17T18:28:44.969944+0000","last_fullsized":"2024-08-17T18:28:44.969944+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:50:25.014563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"70'2","reported_seq":292,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.865341+0000","last_change":"2024-08-17T18:23:59.772682+0000","last_active":"2024-08-17T18:28:44.865341+0000","last_peered":"2024-08-17T18:28:44.865341+0000","last_clean":"2024-08-17T18:28:44.865341+0000","last_became_active":"2024-08-17T18:23:59.748920+0000","last_became_peered":"2024-08-17T18:23:59.748920+0000","last_unstale":"2024-08-17T18:28:44.865341+0000","last_undegraded":"2024-08-17T18:28:44.865341+0000","last_fullsized":"2024-08-17T18:28:44.865341+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:07:31.660610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"347'149","reported_seq":1101,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.526228+0000","last_change":"2024-08-17T18:24:02.793338+0000","last_active":"2024-08-17T18:28:02.526228+0000","last_peered":"2024-08-17T18:28:02.526228+0000","last_clean":"2024-08-17T18:28:02.526228+0000","last_became_active":"2024-08-17T18:24:02.792982+0000","last_became_peered":"2024-08-17T18:24:02.792982+0000","last_unstale":"2024-08-17T18:28:02.526228+0000","last_undegraded":"2024-08-17T18:28:02.526228+0000","last_fullsized":"2024-08-17T18:28:02.526228+0000","mapping_epoch":1743,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1744,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:52.474544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:52.474544+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:37:26.909105+0000","scrub_duration":0.001886326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":126,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:45.494533+0000","last_change":"2024-08-17T18:23:59.741315+0000","last_active":"2024-08-17T18:28:45.494533+0000","last_peered":"2024-08-17T18:28:45.494533+0000","last_clean":"2024-08-17T18:28:45.494533+0000","last_became_active":"2024-08-17T18:23:59.741020+0000","last_became_peered":"2024-08-17T18:23:59.741020+0000","last_unstale":"2024-08-17T18:28:45.494533+0000","last_undegraded":"2024-08-17T18:28:45.494533+0000","last_fullsized":"2024-08-17T18:28:45.494533+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:46:47.093299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":213,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:32.164530+0000","last_change":"2024-08-17T18:23:59.740824+0000","last_active":"2024-08-17T18:28:32.164530+0000","last_peered":"2024-08-17T18:28:32.164530+0000","last_clean":"2024-08-17T18:28:32.164530+0000","last_became_active":"2024-08-17T18:23:59.740478+0000","last_became_peered":"2024-08-17T18:23:59.740478+0000","last_unstale":"2024-08-17T18:28:32.164530+0000","last_undegraded":"2024-08-17T18:28:32.164530+0000","last_fullsized":"2024-08-17T18:28:32.164530+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:44:30.520216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027465900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":217,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.969870+0000","last_change":"2024-08-17T18:20:32.012874+0000","last_active":"2024-08-17T18:28:44.969870+0000","last_peered":"2024-08-17T18:28:44.969870+0000","last_clean":"2024-08-17T18:28:44.969870+0000","last_became_active":"2024-08-17T18:20:32.012553+0000","last_became_peered":"2024-08-17T18:20:32.012553+0000","last_unstale":"2024-08-17T18:28:44.969870+0000","last_undegraded":"2024-08-17T18:28:44.969870+0000","last_fullsized":"2024-08-17T18:28:44.969870+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:14:17.188745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"347'149","reported_seq":1067,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.526054+0000","last_change":"2024-08-17T18:22:36.866341+0000","last_active":"2024-08-17T18:28:02.526054+0000","last_peered":"2024-08-17T18:28:02.526054+0000","last_clean":"2024-08-17T18:28:02.526054+0000","last_became_active":"2024-08-17T18:22:36.865561+0000","last_became_peered":"2024-08-17T18:22:36.865561+0000","last_unstale":"2024-08-17T18:28:02.526054+0000","last_undegraded":"2024-08-17T18:28:02.526054+0000","last_fullsized":"2024-08-17T18:28:02.526054+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:09:49.461698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:09:49.461698+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:22:21.423162+0000","scrub_duration":0.0013792520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":69,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.864638+0000","last_change":"2024-08-17T18:26:13.338201+0000","last_active":"2024-08-17T18:28:44.864638+0000","last_peered":"2024-08-17T18:28:44.864638+0000","last_clean":"2024-08-17T18:28:44.864638+0000","last_became_active":"2024-08-17T18:26:13.337933+0000","last_became_peered":"2024-08-17T18:26:13.337933+0000","last_unstale":"2024-08-17T18:28:44.864638+0000","last_undegraded":"2024-08-17T18:28:44.864638+0000","last_fullsized":"2024-08-17T18:28:44.864638+0000","mapping_epoch":1768,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1769,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:16:05.784689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":217,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.969933+0000","last_change":"2024-08-17T18:20:32.014875+0000","last_active":"2024-08-17T18:28:44.969933+0000","last_peered":"2024-08-17T18:28:44.969933+0000","last_clean":"2024-08-17T18:28:44.969933+0000","last_became_active":"2024-08-17T18:20:32.014631+0000","last_became_peered":"2024-08-17T18:20:32.014631+0000","last_unstale":"2024-08-17T18:28:44.969933+0000","last_undegraded":"2024-08-17T18:28:44.969933+0000","last_fullsized":"2024-08-17T18:28:44.969933+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:00:50.978635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043819200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":372,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:01.502236+0000","last_change":"2024-08-17T18:18:29.735362+0000","last_active":"2024-08-17T18:28:01.502236+0000","last_peered":"2024-08-17T18:28:01.502236+0000","last_clean":"2024-08-17T18:28:01.502236+0000","last_became_active":"2024-08-17T18:18:29.734752+0000","last_became_peered":"2024-08-17T18:18:29.734752+0000","last_unstale":"2024-08-17T18:28:01.502236+0000","last_undegraded":"2024-08-17T18:28:01.502236+0000","last_fullsized":"2024-08-17T18:28:01.502236+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:45:25.709698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"347'149","reported_seq":1112,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.377477+0000","last_change":"2024-08-17T18:24:02.792784+0000","last_active":"2024-08-17T18:28:02.377477+0000","last_peered":"2024-08-17T18:28:02.377477+0000","last_clean":"2024-08-17T18:28:02.377477+0000","last_became_active":"2024-08-17T18:24:02.792286+0000","last_became_peered":"2024-08-17T18:24:02.792286+0000","last_unstale":"2024-08-17T18:28:02.377477+0000","last_undegraded":"2024-08-17T18:28:02.377477+0000","last_fullsized":"2024-08-17T18:28:02.377477+0000","mapping_epoch":1743,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1744,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:00.316884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:00.316884+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:26:57.982520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":101,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.969721+0000","last_change":"2024-08-17T18:25:14.031207+0000","last_active":"2024-08-17T18:28:44.969721+0000","last_peered":"2024-08-17T18:28:44.969721+0000","last_clean":"2024-08-17T18:28:44.969721+0000","last_became_active":"2024-08-17T18:25:14.030916+0000","last_became_peered":"2024-08-17T18:25:14.030916+0000","last_unstale":"2024-08-17T18:28:44.969721+0000","last_undegraded":"2024-08-17T18:28:44.969721+0000","last_fullsized":"2024-08-17T18:28:44.969721+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:37:34.050072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1858,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.186329+0000","last_change":"2024-08-17T18:23:59.741594+0000","last_active":"2024-08-17T18:28:02.186329+0000","last_peered":"2024-08-17T18:28:02.186329+0000","last_clean":"2024-08-17T18:28:02.186329+0000","last_became_active":"2024-08-17T18:23:59.741207+0000","last_became_peered":"2024-08-17T18:23:59.741207+0000","last_unstale":"2024-08-17T18:28:02.186329+0000","last_undegraded":"2024-08-17T18:28:02.186329+0000","last_fullsized":"2024-08-17T18:28:02.186329+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:02:52.545413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":441,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:01.502262+0000","last_change":"2024-08-17T18:22:36.836104+0000","last_active":"2024-08-17T18:28:01.502262+0000","last_peered":"2024-08-17T18:28:01.502262+0000","last_clean":"2024-08-17T18:28:01.502262+0000","last_became_active":"2024-08-17T18:22:36.835273+0000","last_became_peered":"2024-08-17T18:22:36.835273+0000","last_unstale":"2024-08-17T18:28:01.502262+0000","last_undegraded":"2024-08-17T18:28:01.502262+0000","last_fullsized":"2024-08-17T18:28:01.502262+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:26:23.494575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"347'149","reported_seq":1042,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:00.943652+0000","last_change":"2024-08-17T18:27:49.884360+0000","last_active":"2024-08-17T18:28:00.943652+0000","last_peered":"2024-08-17T18:28:00.943652+0000","last_clean":"2024-08-17T18:28:00.943652+0000","last_became_active":"2024-08-17T18:27:49.884006+0000","last_became_peered":"2024-08-17T18:27:49.884006+0000","last_unstale":"2024-08-17T18:28:00.943652+0000","last_undegraded":"2024-08-17T18:28:00.943652+0000","last_fullsized":"2024-08-17T18:28:00.943652+0000","mapping_epoch":1791,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1792,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:08:02.363531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:08:02.363531+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:43:15.930550+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":101,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.969999+0000","last_change":"2024-08-17T18:25:14.176621+0000","last_active":"2024-08-17T18:28:44.969999+0000","last_peered":"2024-08-17T18:28:44.969999+0000","last_clean":"2024-08-17T18:28:44.969999+0000","last_became_active":"2024-08-17T18:25:14.176321+0000","last_became_peered":"2024-08-17T18:25:14.176321+0000","last_unstale":"2024-08-17T18:28:44.969999+0000","last_undegraded":"2024-08-17T18:28:44.969999+0000","last_fullsized":"2024-08-17T18:28:44.969999+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:40:55.297225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":229,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.969386+0000","last_change":"2024-08-17T18:22:36.834643+0000","last_active":"2024-08-17T18:28:44.969386+0000","last_peered":"2024-08-17T18:28:44.969386+0000","last_clean":"2024-08-17T18:28:44.969386+0000","last_became_active":"2024-08-17T18:22:36.834279+0000","last_became_peered":"2024-08-17T18:22:36.834279+0000","last_unstale":"2024-08-17T18:28:44.969386+0000","last_undegraded":"2024-08-17T18:28:44.969386+0000","last_fullsized":"2024-08-17T18:28:44.969386+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:29:36.002584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2899,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.864641+0000","last_change":"2024-08-17T18:18:29.742742+0000","last_active":"2024-08-17T18:28:44.864641+0000","last_peered":"2024-08-17T18:28:44.864641+0000","last_clean":"2024-08-17T18:28:44.864641+0000","last_became_active":"2024-08-17T18:18:29.742393+0000","last_became_peered":"2024-08-17T18:18:29.742393+0000","last_unstale":"2024-08-17T18:28:44.864641+0000","last_undegraded":"2024-08-17T18:28:44.864641+0000","last_fullsized":"2024-08-17T18:28:44.864641+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:57:32.858608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000346258,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"347'149","reported_seq":905,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.969763+0000","last_change":"2024-08-17T18:23:39.370457+0000","last_active":"2024-08-17T18:28:44.969763+0000","last_peered":"2024-08-17T18:28:44.969763+0000","last_clean":"2024-08-17T18:28:44.969763+0000","last_became_active":"2024-08-17T18:23:39.370144+0000","last_became_peered":"2024-08-17T18:23:39.370144+0000","last_unstale":"2024-08-17T18:28:44.969763+0000","last_undegraded":"2024-08-17T18:28:44.969763+0000","last_fullsized":"2024-08-17T18:28:44.969763+0000","mapping_epoch":1736,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1737,"parent":"0.0","parent_split_bits":3,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:20:35.854154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:20:35.854154+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:39:35.866331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":126,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:45.494554+0000","last_change":"2024-08-17T18:23:59.742951+0000","last_active":"2024-08-17T18:28:45.494554+0000","last_peered":"2024-08-17T18:28:45.494554+0000","last_clean":"2024-08-17T18:28:45.494554+0000","last_became_active":"2024-08-17T18:23:59.742510+0000","last_became_peered":"2024-08-17T18:23:59.742510+0000","last_unstale":"2024-08-17T18:28:45.494554+0000","last_undegraded":"2024-08-17T18:28:45.494554+0000","last_fullsized":"2024-08-17T18:28:45.494554+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:19:12.943501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":126,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:45.494293+0000","last_change":"2024-08-17T18:23:59.740786+0000","last_active":"2024-08-17T18:28:45.494293+0000","last_peered":"2024-08-17T18:28:45.494293+0000","last_clean":"2024-08-17T18:28:45.494293+0000","last_became_active":"2024-08-17T18:23:59.740340+0000","last_became_peered":"2024-08-17T18:23:59.740340+0000","last_unstale":"2024-08-17T18:28:45.494293+0000","last_undegraded":"2024-08-17T18:28:45.494293+0000","last_fullsized":"2024-08-17T18:28:45.494293+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:52:27.108557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1111,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:00.943479+0000","last_change":"2024-08-17T18:27:50.140301+0000","last_active":"2024-08-17T18:28:00.943479+0000","last_peered":"2024-08-17T18:28:00.943479+0000","last_clean":"2024-08-17T18:28:00.943479+0000","last_became_active":"2024-08-17T18:27:50.139980+0000","last_became_peered":"2024-08-17T18:27:50.139980+0000","last_unstale":"2024-08-17T18:28:00.943479+0000","last_undegraded":"2024-08-17T18:28:00.943479+0000","last_fullsized":"2024-08-17T18:28:00.943479+0000","mapping_epoch":1791,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:57:47.317296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030259999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":76,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.526043+0000","last_change":"2024-08-17T18:23:36.318955+0000","last_active":"2024-08-17T18:28:02.526043+0000","last_peered":"2024-08-17T18:28:02.526043+0000","last_clean":"2024-08-17T18:28:02.526043+0000","last_became_active":"2024-08-17T18:23:36.318684+0000","last_became_peered":"2024-08-17T18:23:36.318684+0000","last_unstale":"2024-08-17T18:28:02.526043+0000","last_undegraded":"2024-08-17T18:28:02.526043+0000","last_fullsized":"2024-08-17T18:28:02.526043+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:47.191868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":287,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.864346+0000","last_change":"2024-08-17T18:22:36.835833+0000","last_active":"2024-08-17T18:28:44.864346+0000","last_peered":"2024-08-17T18:28:44.864346+0000","last_clean":"2024-08-17T18:28:44.864346+0000","last_became_active":"2024-08-17T18:22:36.835514+0000","last_became_peered":"2024-08-17T18:22:36.835514+0000","last_unstale":"2024-08-17T18:28:44.864346+0000","last_undegraded":"2024-08-17T18:28:44.864346+0000","last_fullsized":"2024-08-17T18:28:44.864346+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:43:53.668287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"62'1","reported_seq":387,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:01.501890+0000","last_change":"2024-08-17T18:19:26.374702+0000","last_active":"2024-08-17T18:28:01.501890+0000","last_peered":"2024-08-17T18:28:01.501890+0000","last_clean":"2024-08-17T18:28:01.501890+0000","last_became_active":"2024-08-17T18:15:53.304106+0000","last_became_peered":"2024-08-17T18:15:53.304106+0000","last_unstale":"2024-08-17T18:28:01.501890+0000","last_undegraded":"2024-08-17T18:28:01.501890+0000","last_fullsized":"2024-08-17T18:28:01.501890+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":1679,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:08:47.841065+0000","last_clean_scrub_stamp":"2024-08-17T17:08:47.841065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:21.409368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"347'149","reported_seq":814,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:00.943397+0000","last_change":"2024-08-17T18:27:51.487506+0000","last_active":"2024-08-17T18:28:00.943397+0000","last_peered":"2024-08-17T18:28:00.943397+0000","last_clean":"2024-08-17T18:28:00.943397+0000","last_became_active":"2024-08-17T18:27:49.883665+0000","last_became_peered":"2024-08-17T18:27:49.883665+0000","last_unstale":"2024-08-17T18:28:00.943397+0000","last_undegraded":"2024-08-17T18:28:00.943397+0000","last_fullsized":"2024-08-17T18:28:00.943397+0000","mapping_epoch":1791,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1792,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:27:51.487383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:27:51.487383+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:47:51.031707+0000","scrub_duration":0.001833096,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"347'149","reported_seq":1093,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:01.502040+0000","last_change":"2024-08-17T18:22:36.836656+0000","last_active":"2024-08-17T18:28:01.502040+0000","last_peered":"2024-08-17T18:28:01.502040+0000","last_clean":"2024-08-17T18:28:01.502040+0000","last_became_active":"2024-08-17T18:22:36.835810+0000","last_became_peered":"2024-08-17T18:22:36.835810+0000","last_unstale":"2024-08-17T18:28:01.502040+0000","last_undegraded":"2024-08-17T18:28:01.502040+0000","last_fullsized":"2024-08-17T18:28:01.502040+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:55.255983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:55.255983+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:59:27.835153+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":217,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.970025+0000","last_change":"2024-08-17T18:20:31.939871+0000","last_active":"2024-08-17T18:28:44.970025+0000","last_peered":"2024-08-17T18:28:44.970025+0000","last_clean":"2024-08-17T18:28:44.970025+0000","last_became_active":"2024-08-17T18:20:31.939535+0000","last_became_peered":"2024-08-17T18:20:31.939535+0000","last_unstale":"2024-08-17T18:28:44.970025+0000","last_undegraded":"2024-08-17T18:28:44.970025+0000","last_fullsized":"2024-08-17T18:28:44.970025+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:43:31.630668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":297,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.864333+0000","last_change":"2024-08-17T18:25:14.036186+0000","last_active":"2024-08-17T18:28:44.864333+0000","last_peered":"2024-08-17T18:28:44.864333+0000","last_clean":"2024-08-17T18:28:44.864333+0000","last_became_active":"2024-08-17T18:25:14.035138+0000","last_became_peered":"2024-08-17T18:25:14.035138+0000","last_unstale":"2024-08-17T18:28:44.864333+0000","last_undegraded":"2024-08-17T18:28:44.864333+0000","last_fullsized":"2024-08-17T18:28:44.864333+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:51.645934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1873,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.186377+0000","last_change":"2024-08-17T18:25:14.026973+0000","last_active":"2024-08-17T18:28:02.186377+0000","last_peered":"2024-08-17T18:28:02.186377+0000","last_clean":"2024-08-17T18:28:02.186377+0000","last_became_active":"2024-08-17T18:25:14.026478+0000","last_became_peered":"2024-08-17T18:25:14.026478+0000","last_unstale":"2024-08-17T18:28:02.186377+0000","last_undegraded":"2024-08-17T18:28:02.186377+0000","last_fullsized":"2024-08-17T18:28:02.186377+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:31.568544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1828,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:20.193370+0000","last_change":"2024-08-17T18:15:53.244851+0000","last_active":"2024-08-17T18:28:20.193370+0000","last_peered":"2024-08-17T18:28:20.193370+0000","last_clean":"2024-08-17T18:28:20.193370+0000","last_became_active":"2024-08-17T18:15:53.240074+0000","last_became_peered":"2024-08-17T18:15:53.240074+0000","last_unstale":"2024-08-17T18:28:20.193370+0000","last_undegraded":"2024-08-17T18:28:20.193370+0000","last_fullsized":"2024-08-17T18:28:20.193370+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:09:00.168413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"347'149","reported_seq":1057,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:07.378145+0000","last_change":"2024-08-17T18:25:55.776006+0000","last_active":"2024-08-17T18:28:07.378145+0000","last_peered":"2024-08-17T18:28:07.378145+0000","last_clean":"2024-08-17T18:28:07.378145+0000","last_became_active":"2024-08-17T18:25:55.775559+0000","last_became_peered":"2024-08-17T18:25:55.775559+0000","last_unstale":"2024-08-17T18:28:07.378145+0000","last_undegraded":"2024-08-17T18:28:07.378145+0000","last_fullsized":"2024-08-17T18:28:07.378145+0000","mapping_epoch":1763,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1764,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:59.307356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:59.307356+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:52:14.784551+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1831,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.186311+0000","last_change":"2024-08-17T18:20:31.942019+0000","last_active":"2024-08-17T18:28:02.186311+0000","last_peered":"2024-08-17T18:28:02.186311+0000","last_clean":"2024-08-17T18:28:02.186311+0000","last_became_active":"2024-08-17T18:20:31.941701+0000","last_became_peered":"2024-08-17T18:20:31.941701+0000","last_unstale":"2024-08-17T18:28:02.186311+0000","last_undegraded":"2024-08-17T18:28:02.186311+0000","last_fullsized":"2024-08-17T18:28:02.186311+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:58.908049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1808,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.525969+0000","last_change":"2024-08-17T18:25:14.027871+0000","last_active":"2024-08-17T18:28:02.525969+0000","last_peered":"2024-08-17T18:28:02.525969+0000","last_clean":"2024-08-17T18:28:02.525969+0000","last_became_active":"2024-08-17T18:25:14.027353+0000","last_became_peered":"2024-08-17T18:25:14.027353+0000","last_unstale":"2024-08-17T18:28:02.525969+0000","last_undegraded":"2024-08-17T18:28:02.525969+0000","last_fullsized":"2024-08-17T18:28:02.525969+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:06.568798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"347'149","reported_seq":1054,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:00.943676+0000","last_change":"2024-08-17T18:18:33.757347+0000","last_active":"2024-08-17T18:28:00.943676+0000","last_peered":"2024-08-17T18:28:00.943676+0000","last_clean":"2024-08-17T18:28:00.943676+0000","last_became_active":"2024-08-17T18:18:33.756960+0000","last_became_peered":"2024-08-17T18:18:33.756960+0000","last_unstale":"2024-08-17T18:28:00.943676+0000","last_undegraded":"2024-08-17T18:28:00.943676+0000","last_fullsized":"2024-08-17T18:28:00.943676+0000","mapping_epoch":1670,"log_start":"0'0","ondisk_log_start":"0'0","created":1491,"last_epoch_clean":1671,"parent":"0.0","parent_split_bits":4,"last_scrub":"347'149","last_scrub_stamp":"2024-08-17T18:07:52.271487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:21.174471+0000","last_clean_scrub_stamp":"2024-08-17T18:07:52.271487+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:37:36.389244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1825,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.186164+0000","last_change":"2024-08-17T18:22:36.837889+0000","last_active":"2024-08-17T18:28:02.186164+0000","last_peered":"2024-08-17T18:28:02.186164+0000","last_clean":"2024-08-17T18:28:02.186164+0000","last_became_active":"2024-08-17T18:22:36.837387+0000","last_became_peered":"2024-08-17T18:22:36.837387+0000","last_unstale":"2024-08-17T18:28:02.186164+0000","last_undegraded":"2024-08-17T18:28:02.186164+0000","last_fullsized":"2024-08-17T18:28:02.186164+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:11:29.265925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"70'2","reported_seq":313,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.864581+0000","last_change":"2024-08-17T18:23:59.770482+0000","last_active":"2024-08-17T18:28:44.864581+0000","last_peered":"2024-08-17T18:28:44.864581+0000","last_clean":"2024-08-17T18:28:44.864581+0000","last_became_active":"2024-08-17T18:23:59.747059+0000","last_became_peered":"2024-08-17T18:23:59.747059+0000","last_unstale":"2024-08-17T18:28:44.864581+0000","last_undegraded":"2024-08-17T18:28:44.864581+0000","last_fullsized":"2024-08-17T18:28:44.864581+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:04:17.042903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":126,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:45.494469+0000","last_change":"2024-08-17T18:23:59.745024+0000","last_active":"2024-08-17T18:28:45.494469+0000","last_peered":"2024-08-17T18:28:45.494469+0000","last_clean":"2024-08-17T18:28:45.494469+0000","last_became_active":"2024-08-17T18:23:59.741633+0000","last_became_peered":"2024-08-17T18:23:59.741633+0000","last_unstale":"2024-08-17T18:28:45.494469+0000","last_undegraded":"2024-08-17T18:28:45.494469+0000","last_fullsized":"2024-08-17T18:28:45.494469+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:41.514649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":395,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:01.502339+0000","last_change":"2024-08-17T18:22:36.836017+0000","last_active":"2024-08-17T18:28:01.502339+0000","last_peered":"2024-08-17T18:28:01.502339+0000","last_clean":"2024-08-17T18:28:01.502339+0000","last_became_active":"2024-08-17T18:22:36.835725+0000","last_became_peered":"2024-08-17T18:22:36.835725+0000","last_unstale":"2024-08-17T18:28:01.502339+0000","last_undegraded":"2024-08-17T18:28:01.502339+0000","last_fullsized":"2024-08-17T18:28:01.502339+0000","mapping_epoch":1724,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1725,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:18:07.395241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":76,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.525724+0000","last_change":"2024-08-17T18:23:36.324099+0000","last_active":"2024-08-17T18:28:02.525724+0000","last_peered":"2024-08-17T18:28:02.525724+0000","last_clean":"2024-08-17T18:28:02.525724+0000","last_became_active":"2024-08-17T18:23:36.323739+0000","last_became_peered":"2024-08-17T18:23:36.323739+0000","last_unstale":"2024-08-17T18:28:02.525724+0000","last_undegraded":"2024-08-17T18:28:02.525724+0000","last_fullsized":"2024-08-17T18:28:02.525724+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:54:56.319013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"70'1","reported_seq":222,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:00.943701+0000","last_change":"2024-08-17T18:27:49.885619+0000","last_active":"2024-08-17T18:28:00.943701+0000","last_peered":"2024-08-17T18:28:00.943701+0000","last_clean":"2024-08-17T18:28:00.943701+0000","last_became_active":"2024-08-17T18:27:49.885246+0000","last_became_peered":"2024-08-17T18:27:49.885246+0000","last_unstale":"2024-08-17T18:28:00.943701+0000","last_undegraded":"2024-08-17T18:28:00.943701+0000","last_fullsized":"2024-08-17T18:28:00.943701+0000","mapping_epoch":1791,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:48:04.478804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1883,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:20.193344+0000","last_change":"2024-08-17T18:25:14.040053+0000","last_active":"2024-08-17T18:28:20.193344+0000","last_peered":"2024-08-17T18:28:20.193344+0000","last_clean":"2024-08-17T18:28:20.193344+0000","last_became_active":"2024-08-17T18:25:14.039807+0000","last_became_peered":"2024-08-17T18:25:14.039807+0000","last_unstale":"2024-08-17T18:28:20.193344+0000","last_undegraded":"2024-08-17T18:28:20.193344+0000","last_fullsized":"2024-08-17T18:28:20.193344+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:05:53.486096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"70'2","reported_seq":191,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:00.943912+0000","last_change":"2024-08-17T18:20:32.006404+0000","last_active":"2024-08-17T18:28:00.943912+0000","last_peered":"2024-08-17T18:28:00.943912+0000","last_clean":"2024-08-17T18:28:00.943912+0000","last_became_active":"2024-08-17T18:20:31.947053+0000","last_became_peered":"2024-08-17T18:20:31.947053+0000","last_unstale":"2024-08-17T18:28:00.943912+0000","last_undegraded":"2024-08-17T18:28:00.943912+0000","last_fullsized":"2024-08-17T18:28:00.943912+0000","mapping_epoch":1695,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:21:06.537725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":399,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:37.381625+0000","last_change":"2024-08-17T18:23:36.318964+0000","last_active":"2024-08-17T18:28:37.381625+0000","last_peered":"2024-08-17T18:28:37.381625+0000","last_clean":"2024-08-17T18:28:37.381625+0000","last_became_active":"2024-08-17T18:23:36.318595+0000","last_became_peered":"2024-08-17T18:23:36.318595+0000","last_unstale":"2024-08-17T18:28:37.381625+0000","last_undegraded":"2024-08-17T18:28:37.381625+0000","last_fullsized":"2024-08-17T18:28:37.381625+0000","mapping_epoch":1733,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1734,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:29:16.221103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"70'1","reported_seq":217,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:00.944020+0000","last_change":"2024-08-17T18:23:59.752171+0000","last_active":"2024-08-17T18:28:00.944020+0000","last_peered":"2024-08-17T18:28:00.944020+0000","last_clean":"2024-08-17T18:28:00.944020+0000","last_became_active":"2024-08-17T18:23:59.745970+0000","last_became_peered":"2024-08-17T18:23:59.745970+0000","last_unstale":"2024-08-17T18:28:00.944020+0000","last_undegraded":"2024-08-17T18:28:00.944020+0000","last_fullsized":"2024-08-17T18:28:00.944020+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:24:52.861533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"70'8","reported_seq":1955,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:02.525964+0000","last_change":"2024-08-17T18:24:22.107318+0000","last_active":"2024-08-17T18:28:02.525964+0000","last_peered":"2024-08-17T18:28:02.525964+0000","last_clean":"2024-08-17T18:28:02.525964+0000","last_became_active":"2024-08-17T18:24:14.847882+0000","last_became_peered":"2024-08-17T18:24:14.847882+0000","last_unstale":"2024-08-17T18:28:02.525964+0000","last_undegraded":"2024-08-17T18:28:02.525964+0000","last_fullsized":"2024-08-17T18:28:02.525964+0000","mapping_epoch":1746,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:45:33.740721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":290,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.865430+0000","last_change":"2024-08-17T18:21:12.645070+0000","last_active":"2024-08-17T18:28:44.865430+0000","last_peered":"2024-08-17T18:28:44.865430+0000","last_clean":"2024-08-17T18:28:44.865430+0000","last_became_active":"2024-08-17T18:21:12.644748+0000","last_became_peered":"2024-08-17T18:21:12.644748+0000","last_unstale":"2024-08-17T18:28:44.865430+0000","last_undegraded":"2024-08-17T18:28:44.865430+0000","last_fullsized":"2024-08-17T18:28:44.865430+0000","mapping_epoch":1708,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1709,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:19:08.683431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":489,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:00.943492+0000","last_change":"2024-08-17T18:27:49.885293+0000","last_active":"2024-08-17T18:28:00.943492+0000","last_peered":"2024-08-17T18:28:00.943492+0000","last_clean":"2024-08-17T18:28:00.943492+0000","last_became_active":"2024-08-17T18:27:49.884978+0000","last_became_peered":"2024-08-17T18:27:49.884978+0000","last_unstale":"2024-08-17T18:28:00.943492+0000","last_undegraded":"2024-08-17T18:28:00.943492+0000","last_fullsized":"2024-08-17T18:28:00.943492+0000","mapping_epoch":1791,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1792,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:21:38.469502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":125,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:45.494878+0000","last_change":"2024-08-17T18:23:59.740450+0000","last_active":"2024-08-17T18:28:45.494878+0000","last_peered":"2024-08-17T18:28:45.494878+0000","last_clean":"2024-08-17T18:28:45.494878+0000","last_became_active":"2024-08-17T18:23:59.740125+0000","last_became_peered":"2024-08-17T18:23:59.740125+0000","last_unstale":"2024-08-17T18:28:45.494878+0000","last_undegraded":"2024-08-17T18:28:45.494878+0000","last_fullsized":"2024-08-17T18:28:45.494878+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:26:38.858210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":126,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:45.494293+0000","last_change":"2024-08-17T18:23:59.740550+0000","last_active":"2024-08-17T18:28:45.494293+0000","last_peered":"2024-08-17T18:28:45.494293+0000","last_clean":"2024-08-17T18:28:45.494293+0000","last_became_active":"2024-08-17T18:23:59.740226+0000","last_became_peered":"2024-08-17T18:23:59.740226+0000","last_unstale":"2024-08-17T18:28:45.494293+0000","last_undegraded":"2024-08-17T18:28:45.494293+0000","last_fullsized":"2024-08-17T18:28:45.494293+0000","mapping_epoch":1740,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1741,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:35.745256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":389,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:45.494971+0000","last_change":"2024-08-17T18:18:29.900128+0000","last_active":"2024-08-17T18:28:45.494971+0000","last_peered":"2024-08-17T18:28:45.494971+0000","last_clean":"2024-08-17T18:28:45.494971+0000","last_became_active":"2024-08-17T18:18:29.899279+0000","last_became_peered":"2024-08-17T18:18:29.899279+0000","last_unstale":"2024-08-17T18:28:45.494971+0000","last_undegraded":"2024-08-17T18:28:45.494971+0000","last_fullsized":"2024-08-17T18:28:45.494971+0000","mapping_epoch":1666,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:01:29.045498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"70'2","reported_seq":181,"reported_epoch":1796,"state":"active+clean","last_fresh":"2024-08-17T18:28:00.943761+0000","last_change":"2024-08-17T18:15:53.271113+0000","last_active":"2024-08-17T18:28:00.943761+0000","last_peered":"2024-08-17T18:28:00.943761+0000","last_clean":"2024-08-17T18:28:00.943761+0000","last_became_active":"2024-08-17T18:15:53.230983+0000","last_became_peered":"2024-08-17T18:15:53.230983+0000","last_unstale":"2024-08-17T18:28:00.943761+0000","last_undegraded":"2024-08-17T18:28:00.943761+0000","last_fullsized":"2024-08-17T18:28:00.943761+0000","mapping_epoch":1637,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:04.937295+0000","last_clean_scrub_stamp":"2024-08-17T17:10:04.937295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:00:35.315798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":308,"reported_epoch":1797,"state":"active+clean","last_fresh":"2024-08-17T18:28:44.865033+0000","last_change":"2024-08-17T18:25:14.030279+0000","last_active":"2024-08-17T18:28:44.865033+0000","last_peered":"2024-08-17T18:28:44.865033+0000","last_clean":"2024-08-17T18:28:44.865033+0000","last_became_active":"2024-08-17T18:25:14.027960+0000","last_became_peered":"2024-08-17T18:25:14.027960+0000","last_unstale":"2024-08-17T18:28:44.865033+0000","last_undegraded":"2024-08-17T18:28:44.865033+0000","last_fullsized":"2024-08-17T18:28:44.865033+0000","mapping_epoch":1752,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":1753,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:10:05.943521+0000","last_clean_scrub_stamp":"2024-08-17T17:10:05.943521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:20:46.317663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":12930,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2386,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234954,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":872784,"kb_used_data":11524,"kb_used_omap":1,"kb_used_meta":861246,"kb_avail":92872368,"statfs":{"total":95995035648,"available":95101304832,"internally_reserved":0,"allocated":11800576,"data_stored":6626644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":881916363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1724,"seq":7404523618381,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":207372,"kb_used_data":11200,"kb_used_omap":123,"kb_used_meta":196036,"kb_avail":93537780,"statfs":{"total":95995035648,"available":95782686720,"internally_reserved":0,"allocated":11468800,"data_stored":6299984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":126809,"internal_metadata":200741031},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":686,"seq":2946347565834,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1302336,"kb_used_data":11764,"kb_used_omap":72,"kb_used_meta":1290487,"kb_avail":92442816,"statfs":{"total":95995035648,"available":94661443584,"internally_reserved":0,"allocated":12046336,"data_stored":6885609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74258,"internal_metadata":1321459182},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":1399,"seq":6008659247435,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":982084,"kb_used_data":11448,"kb_used_omap":131,"kb_used_meta":970492,"kb_avail":92763068,"statfs":{"total":95995035648,"available":94989381632,"internally_reserved":0,"allocated":11722752,"data_stored":6558455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":134291,"internal_metadata":993784685},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":667,"seq":2864743187223,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":981680,"kb_used_data":11172,"kb_used_omap":66,"kb_used_meta":970429,"kb_avail":92763472,"statfs":{"total":95995035648,"available":94989795328,"internally_reserved":0,"allocated":11440128,"data_stored":6286801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67925,"internal_metadata":993719979},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":659,"seq":2830383448862,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":917696,"kb_used_data":11188,"kb_used_omap":75,"kb_used_meta":906420,"kb_avail":92827456,"statfs":{"total":95995035648,"available":95055314944,"internally_reserved":0,"allocated":11456512,"data_stored":6295755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77633,"internal_metadata":928174271},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208758227,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1414340,"kb_used_data":11768,"kb_used_omap":1,"kb_used_meta":1402558,"kb_avail":92330812,"statfs":{"total":95995035648,"available":94546751488,"internally_reserved":0,"allocated":12050432,"data_stored":6885031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1436219846},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019864,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1368396,"kb_used_data":11200,"kb_used_omap":1,"kb_used_meta":1357182,"kb_avail":92376756,"statfs":{"total":95995035648,"available":94593798144,"internally_reserved":0,"allocated":11468800,"data_stored":6298894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1389754822},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1544,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":842,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T18:28:52.223 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-08-17T18:28:52.224 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_91f88e4d132ac08dc53355fa3d9717a53236a3a9/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_91f88e4d132ac08dc53355fa3d9717a53236a3a9/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_rbd.sh) on smithi136 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2024-08-17T18:28:52.404 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1a5e0a04a189467f98b2822546a1a0c7 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_91f88e4d132ac08dc53355fa3d9717a53236a3a9/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_91f88e4d132ac08dc53355fa3d9717a53236a3a9/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_rbd.sh) on smithi136 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2024-08-17T18:28:52.407 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-08-17T18:28:52.417 INFO:tasks.cephadm:Teardown begin 2024-08-17T18:28:52.417 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-17T18:28:52.431 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:51 smithi165 ceph-mon[105517]: pgmap v3638: 83 pgs: 83 active+clean; 582 KiB data, 7.7 GiB used, 708 GiB / 715 GiB avail 2024-08-17T18:28:52.432 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:51 smithi165 ceph-mon[105517]: from='mgr.34104 172.21.15.136:0/4240442346' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T18:28:52.474 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-17T18:28:52.514 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-08-17T18:28:52.515 DEBUG:teuthology.orchestra.run.smithi136:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-08-17T18:28:52.531 DEBUG:teuthology.orchestra.run.smithi165:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-08-17T18:28:52.581 INFO:tasks.cephadm:Stopping all daemons... 2024-08-17T18:28:52.581 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-08-17T18:28:52.581 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-8270436c-5cba-11ef-bccf-c7b262605968@mon.a 2024-08-17T18:28:52.876 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:52 smithi136 systemd[1]: Stopping Ceph mon.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T18:28:53.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:52 smithi136 ceph-mon[115078]: from='client.37479 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:28:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:52 smithi136 ceph-mon[109861]: from='client.37479 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:28:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a[109838]: 2024-08-17T18:28:52.894+0000 7f5b93f88640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T18:28:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a[109838]: 2024-08-17T18:28:52.894+0000 7f5b93f88640 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-08-17T18:28:53.197 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:53 smithi136 podman[492185]: 2024-08-17 18:28:53.088987566 +0000 UTC m=+0.294129041 container died 9cf82e613626d40f8c2d9c7046718ba80ebe1693cad532de3387d5e9a487fed5 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-17T18:28:53.197 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:28:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:18:28:52] ENGINE Bus STOPPING 2024-08-17T18:28:53.198 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:28:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:18:28:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-08-17T18:28:53.198 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:28:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:18:28:52] ENGINE Bus STOPPED 2024-08-17T18:28:53.198 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:28:52 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:18:28:52] ENGINE Bus STARTING 2024-08-17T18:28:53.198 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:28:53 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:18:28:53] ENGINE Serving on http://:::9283 2024-08-17T18:28:53.198 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:28:53 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:18:28:53] ENGINE Bus STARTED 2024-08-17T18:28:53.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:52 smithi165 ceph-mon[105517]: from='client.37479 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T18:28:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:53 smithi136 podman[492185]: 2024-08-17 18:28:53.510710793 +0000 UTC m=+0.715852265 container cleanup 9cf82e613626d40f8c2d9c7046718ba80ebe1693cad532de3387d5e9a487fed5 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a, ceph=True, RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-08-17T18:28:53.696 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:53 smithi136 bash[492185]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a 2024-08-17T18:28:54.012 INFO:journalctl@ceph.mon.a.smithi136.stdout:Aug 17 18:28:53 smithi136 podman[492279]: 2024-08-17 18:28:53.76257677 +0000 UTC m=+0.667891135 container remove 9cf82e613626d40f8c2d9c7046718ba80ebe1693cad532de3387d5e9a487fed5 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-a, org.label-schema.license=GPLv2, RELEASE=main-cd8bdbe, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-08-17T18:28:54.018 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@mon.a.service' 2024-08-17T18:28:54.055 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T18:28:54.055 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-08-17T18:28:54.055 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-08-17T18:28:54.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-8270436c-5cba-11ef-bccf-c7b262605968@mon.c 2024-08-17T18:28:54.393 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:54 smithi136 systemd[1]: Stopping Ceph mon.c for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T18:28:54.672 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:54 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-c[115055]: 2024-08-17T18:28:54.391+0000 7f9ee36f4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T18:28:54.673 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:54 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-c[115055]: 2024-08-17T18:28:54.391+0000 7f9ee36f4640 -1 mon.c@2(peon) e4 *** Got Signal Terminated *** 2024-08-17T18:28:54.673 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:54 smithi136 podman[492416]: 2024-08-17 18:28:54.55625666 +0000 UTC m=+0.277927998 container died 2c2ac98ad9d499e6f24973f19adb814ccd1e5a363b677247051e0e713aecae98 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, GIT_CLEAN=True, ceph=True) 2024-08-17T18:28:54.673 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:28:54 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:18:28:54] ENGINE Bus STOPPING 2024-08-17T18:28:54.673 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:28:54 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:18:28:54] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-08-17T18:28:54.673 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:28:54 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:18:28:54] ENGINE Bus STOPPED 2024-08-17T18:28:54.674 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:28:54 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:18:28:54] ENGINE Bus STARTING 2024-08-17T18:28:54.674 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:28:54 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:18:28:54] ENGINE Serving on http://:::9283 2024-08-17T18:28:54.674 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:28:54 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: [17/Aug/2024:18:28:54] ENGINE Bus STARTED 2024-08-17T18:28:55.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:54 smithi136 podman[492416]: 2024-08-17 18:28:54.941315302 +0000 UTC m=+0.662986648 container cleanup 2c2ac98ad9d499e6f24973f19adb814ccd1e5a363b677247051e0e713aecae98 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-08-17T18:28:55.196 INFO:journalctl@ceph.mon.c.smithi136.stdout:Aug 17 18:28:54 smithi136 bash[492416]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-c 2024-08-17T18:28:55.521 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@mon.c.service' 2024-08-17T18:28:55.556 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T18:28:55.556 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-08-17T18:28:55.556 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-08-17T18:28:55.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-8270436c-5cba-11ef-bccf-c7b262605968@mon.b 2024-08-17T18:28:56.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:55 smithi165 systemd[1]: Stopping Ceph mon.b for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T18:28:56.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:56 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-b[105493]: 2024-08-17T18:28:56.023+0000 7f8801076640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T18:28:56.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:56 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-b[105493]: 2024-08-17T18:28:56.023+0000 7f8801076640 -1 mon.b@1(peon) e4 *** Got Signal Terminated *** 2024-08-17T18:28:56.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:56 smithi165 podman[307926]: 2024-08-17 18:28:56.239187254 +0000 UTC m=+0.482505083 container died 1fd357308b8d9c5756303eb35cb2497df3ef60b5fe965f83d94c0fbfeec85b53 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-b, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-cd8bdbe) 2024-08-17T18:28:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:56 smithi165 podman[307926]: 2024-08-17 18:28:56.756731587 +0000 UTC m=+1.000049417 container cleanup 1fd357308b8d9c5756303eb35cb2497df3ef60b5fe965f83d94c0fbfeec85b53 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-b, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=main-cd8bdbe, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True) 2024-08-17T18:28:57.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:56 smithi165 bash[307926]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-b 2024-08-17T18:28:57.247 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@mon.b.service' 2024-08-17T18:28:57.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:57 smithi165 podman[307945]: 2024-08-17 18:28:57.016905338 +0000 UTC m=+0.776884497 container remove 1fd357308b8d9c5756303eb35cb2497df3ef60b5fe965f83d94c0fbfeec85b53 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mon-b, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-cd8bdbe, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-08-17T18:28:57.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:57 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@mon.b.service: Deactivated successfully. 2024-08-17T18:28:57.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:57 smithi165 systemd[1]: Stopped Ceph mon.b for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T18:28:57.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:Aug 17 18:28:57 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@mon.b.service: Consumed 41.991s CPU time. 2024-08-17T18:28:57.284 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T18:28:57.284 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-08-17T18:28:57.285 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-08-17T18:28:57.285 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-8270436c-5cba-11ef-bccf-c7b262605968@mgr.y 2024-08-17T18:28:57.614 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:28:57 smithi136 systemd[1]: Stopping Ceph mgr.y for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T18:28:57.615 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:28:57 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y[91605]: ::ffff:172.21.15.165 - - [17/Aug/2024:18:28:57] "GET /metrics HTTP/1.1" 200 34258 "" "Prometheus/2.51.0" 2024-08-17T18:28:57.942 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:28:57 smithi136 podman[492561]: 2024-08-17 18:28:57.623421727 +0000 UTC m=+0.145575027 container died 9acdc6775e486c1a13f35bf433e99d16609c8fdccecae3c56dff42fb2ec39d35 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y, io.buildah.version=1.36.0, ceph=True, RELEASE=main-cd8bdbe, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T18:28:58.196 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:28:58 smithi136 podman[492561]: 2024-08-17 18:28:58.063556076 +0000 UTC m=+0.585709372 container cleanup 9acdc6775e486c1a13f35bf433e99d16609c8fdccecae3c56dff42fb2ec39d35 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y, org.label-schema.license=GPLv2, RELEASE=main-cd8bdbe, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731) 2024-08-17T18:28:58.196 INFO:journalctl@ceph.mgr.y.smithi136.stdout:Aug 17 18:28:58 smithi136 bash[492561]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-y 2024-08-17T18:28:58.525 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@mgr.y.service' 2024-08-17T18:28:58.559 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T18:28:58.560 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-08-17T18:28:58.560 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-08-17T18:28:58.560 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-8270436c-5cba-11ef-bccf-c7b262605968@mgr.x 2024-08-17T18:28:58.920 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 18:28:58 smithi165 systemd[1]: Stopping Ceph mgr.x for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T18:28:59.261 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 18:28:58 smithi165 podman[308058]: 2024-08-17 18:28:58.919256173 +0000 UTC m=+0.164559708 container died 837fd48ff7cb95d6e28e91b05d6b3c46c96f4b47af54d535abd4a912e08e261a (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x, RELEASE=main-cd8bdbe, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-08-17T18:28:59.680 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 18:28:59 smithi165 podman[308058]: 2024-08-17 18:28:59.35374747 +0000 UTC m=+0.599051014 container cleanup 837fd48ff7cb95d6e28e91b05d6b3c46c96f4b47af54d535abd4a912e08e261a (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, RELEASE=main-cd8bdbe, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-08-17T18:28:59.680 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Aug 17 18:28:59 smithi165 bash[308058]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-mgr-x 2024-08-17T18:28:59.806 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@mgr.x.service' 2024-08-17T18:28:59.841 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T18:28:59.842 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-08-17T18:28:59.842 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-08-17T18:28:59.842 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.0 2024-08-17T18:29:00.129 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 18:28:59 smithi136 systemd[1]: Stopping Ceph osd.0 for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T18:29:00.446 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 18:29:00 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0[128682]: 2024-08-17T18:29:00.127+0000 7f1cef299640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T18:29:00.446 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 18:29:00 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0[128682]: 2024-08-17T18:29:00.127+0000 7f1cef299640 -1 osd.0 1797 *** Got signal Terminated *** 2024-08-17T18:29:00.446 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 18:29:00 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0[128682]: 2024-08-17T18:29:00.127+0000 7f1cef299640 -1 osd.0 1797 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T18:29:05.828 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 18:29:05 smithi136 podman[492692]: 2024-08-17 18:29:05.563613467 +0000 UTC m=+5.528039462 container died 71b7314ae4a5b549076d7ef75cfd4ef44101068cbfeb22f4feed628422ec7557 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=main-cd8bdbe, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-08-17T18:29:06.196 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 18:29:06 smithi136 podman[492692]: 2024-08-17 18:29:06.010373991 +0000 UTC m=+5.974799986 container cleanup 71b7314ae4a5b549076d7ef75cfd4ef44101068cbfeb22f4feed628422ec7557 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=main-cd8bdbe, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T18:29:06.196 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 18:29:06 smithi136 bash[492692]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0 2024-08-17T18:29:06.696 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 18:29:06 smithi136 podman[492712]: 2024-08-17 18:29:06.370835762 +0000 UTC m=+0.809085521 container remove 71b7314ae4a5b549076d7ef75cfd4ef44101068cbfeb22f4feed628422ec7557 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-cd8bdbe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux ) 2024-08-17T18:29:07.081 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 18:29:06 smithi136 podman[492808]: 2024-08-17 18:29:06.682565829 +0000 UTC m=+0.016751771 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T18:29:07.082 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 18:29:06 smithi136 podman[492808]: 2024-08-17 18:29:06.813478397 +0000 UTC m=+0.147664374 container create e881cb78d59d67a415af3fda9b7339551afaab5043d43dff06610b5ce927eca2 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=main-cd8bdbe, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-08-17T18:29:07.082 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 18:29:07 smithi136 podman[492808]: 2024-08-17 18:29:07.081394922 +0000 UTC m=+0.415580854 container init e881cb78d59d67a415af3fda9b7339551afaab5043d43dff06610b5ce927eca2 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=main-cd8bdbe, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T18:29:07.446 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 18:29:07 smithi136 podman[492808]: 2024-08-17 18:29:07.085995182 +0000 UTC m=+0.420181121 container start e881cb78d59d67a415af3fda9b7339551afaab5043d43dff06610b5ce927eca2 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-08-17T18:29:07.447 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 18:29:07 smithi136 podman[492808]: 2024-08-17 18:29:07.191591374 +0000 UTC m=+0.525777319 container attach e881cb78d59d67a415af3fda9b7339551afaab5043d43dff06610b5ce927eca2 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.36.0, ceph=True, GIT_CLEAN=True, RELEASE=main-cd8bdbe, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T18:29:07.921 INFO:journalctl@ceph.osd.0.smithi136.stdout:Aug 17 18:29:07 smithi136 podman[492808]: 2024-08-17 18:29:07.63059511 +0000 UTC m=+0.964781052 container died e881cb78d59d67a415af3fda9b7339551afaab5043d43dff06610b5ce927eca2 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-08-17T18:29:08.321 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.0.service' 2024-08-17T18:29:08.360 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T18:29:08.360 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-08-17T18:29:08.360 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-08-17T18:29:08.361 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.1 2024-08-17T18:29:08.696 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 18:29:08 smithi136 systemd[1]: Stopping Ceph osd.1 for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T18:29:09.196 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 18:29:08 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1[134606]: 2024-08-17T18:29:08.702+0000 7f3d03b48640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T18:29:09.197 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 18:29:08 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1[134606]: 2024-08-17T18:29:08.702+0000 7f3d03b48640 -1 osd.1 1797 *** Got signal Terminated *** 2024-08-17T18:29:09.197 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 18:29:08 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1[134606]: 2024-08-17T18:29:08.702+0000 7f3d03b48640 -1 osd.1 1797 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T18:29:14.601 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 18:29:14 smithi136 podman[492949]: 2024-08-17 18:29:14.317617014 +0000 UTC m=+5.723845555 container died a0bc94c66763d209c210caf3f644d339d3c688d725a46bb6e40552dd49c09380 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-08-17T18:29:14.946 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 18:29:14 smithi136 podman[492949]: 2024-08-17 18:29:14.798664473 +0000 UTC m=+6.204893006 container cleanup a0bc94c66763d209c210caf3f644d339d3c688d725a46bb6e40552dd49c09380 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731) 2024-08-17T18:29:14.947 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 18:29:14 smithi136 bash[492949]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1 2024-08-17T18:29:15.429 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 18:29:15 smithi136 podman[492971]: 2024-08-17 18:29:15.066831319 +0000 UTC m=+0.747451985 container remove a0bc94c66763d209c210caf3f644d339d3c688d725a46bb6e40552dd49c09380 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-cd8bdbe, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-08-17T18:29:15.696 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 18:29:15 smithi136 podman[493066]: 2024-08-17 18:29:15.331206244 +0000 UTC m=+0.017211396 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T18:29:15.696 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 18:29:15 smithi136 podman[493066]: 2024-08-17 18:29:15.442915216 +0000 UTC m=+0.128920400 container create 617b526a1cd11881118753c322d2b98ffff2bc68a3a0d341a8055f1482dc5736 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, ceph=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=main-cd8bdbe, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T18:29:15.696 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 18:29:15 smithi136 podman[493066]: 2024-08-17 18:29:15.620434463 +0000 UTC m=+0.306439604 container init 617b526a1cd11881118753c322d2b98ffff2bc68a3a0d341a8055f1482dc5736 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-cd8bdbe, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-08-17T18:29:15.696 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 18:29:15 smithi136 podman[493066]: 2024-08-17 18:29:15.624942718 +0000 UTC m=+0.310947862 container start 617b526a1cd11881118753c322d2b98ffff2bc68a3a0d341a8055f1482dc5736 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-cd8bdbe, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T18:29:16.196 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 18:29:15 smithi136 podman[493066]: 2024-08-17 18:29:15.720227312 +0000 UTC m=+0.406232462 container attach 617b526a1cd11881118753c322d2b98ffff2bc68a3a0d341a8055f1482dc5736 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240731, RELEASE=main-cd8bdbe, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T18:29:16.196 INFO:journalctl@ceph.osd.1.smithi136.stdout:Aug 17 18:29:15 smithi136 podman[493066]: 2024-08-17 18:29:15.81744912 +0000 UTC m=+0.503454271 container died 617b526a1cd11881118753c322d2b98ffff2bc68a3a0d341a8055f1482dc5736 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe) 2024-08-17T18:29:16.730 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.1.service' 2024-08-17T18:29:16.775 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T18:29:16.775 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-08-17T18:29:16.776 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-08-17T18:29:16.776 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.2 2024-08-17T18:29:17.113 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:29:16 smithi136 systemd[1]: Stopping Ceph osd.2 for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T18:29:17.446 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:29:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2[291983]: 2024-08-17T18:29:17.112+0000 7f0b5eff9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T18:29:17.446 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:29:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2[291983]: 2024-08-17T18:29:17.112+0000 7f0b5eff9640 -1 osd.2 1797 *** Got signal Terminated *** 2024-08-17T18:29:17.446 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:29:17 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2[291983]: 2024-08-17T18:29:17.112+0000 7f0b5eff9640 -1 osd.2 1797 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T18:29:23.136 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:29:22 smithi136 podman[493193]: 2024-08-17 18:29:22.747622139 +0000 UTC m=+5.741700690 container died 697e0eb06edb12bcdfb22c6cd6ebdb7b4731fdcc3914ff0fa671d3c37cc55ea5 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-cd8bdbe, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-08-17T18:29:23.136 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:29:23 smithi136 podman[493193]: 2024-08-17 18:29:23.135474884 +0000 UTC m=+6.129553476 container cleanup 697e0eb06edb12bcdfb22c6cd6ebdb7b4731fdcc3914ff0fa671d3c37cc55ea5 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-cd8bdbe, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-08-17T18:29:23.397 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:29:23 smithi136 bash[493193]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2 2024-08-17T18:29:23.696 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:29:23 smithi136 podman[493213]: 2024-08-17 18:29:23.397467579 +0000 UTC m=+0.647228899 container remove 697e0eb06edb12bcdfb22c6cd6ebdb7b4731fdcc3914ff0fa671d3c37cc55ea5 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2, RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-08-17T18:29:24.035 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:29:23 smithi136 podman[493307]: 2024-08-17 18:29:23.637847433 +0000 UTC m=+0.017342277 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T18:29:24.035 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:29:23 smithi136 podman[493307]: 2024-08-17 18:29:23.739898832 +0000 UTC m=+0.119393668 container create 562a4541eb85f2334e6258aab7a1f8f3ba03ca72d3d2ea0348415023311e38e2 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=main-cd8bdbe, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, ceph=True) 2024-08-17T18:29:24.036 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:29:23 smithi136 podman[493307]: 2024-08-17 18:29:23.959338292 +0000 UTC m=+0.338833141 container init 562a4541eb85f2334e6258aab7a1f8f3ba03ca72d3d2ea0348415023311e38e2 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-cd8bdbe, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-08-17T18:29:24.036 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:29:23 smithi136 podman[493307]: 2024-08-17 18:29:23.964032276 +0000 UTC m=+0.343527114 container start 562a4541eb85f2334e6258aab7a1f8f3ba03ca72d3d2ea0348415023311e38e2 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T18:29:24.036 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:29:24 smithi136 podman[493307]: 2024-08-17 18:29:24.034224978 +0000 UTC m=+0.413719817 container attach 562a4541eb85f2334e6258aab7a1f8f3ba03ca72d3d2ea0348415023311e38e2 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_CLEAN=True, RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T18:29:24.446 INFO:journalctl@ceph.osd.2.smithi136.stdout:Aug 17 18:29:24 smithi136 podman[493307]: 2024-08-17 18:29:24.159081487 +0000 UTC m=+0.538576322 container died 562a4541eb85f2334e6258aab7a1f8f3ba03ca72d3d2ea0348415023311e38e2 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-cd8bdbe, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True) 2024-08-17T18:29:24.828 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.2.service' 2024-08-17T18:29:24.866 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T18:29:24.866 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-08-17T18:29:24.866 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-08-17T18:29:24.866 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.3 2024-08-17T18:29:25.193 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 18:29:24 smithi136 systemd[1]: Stopping Ceph osd.3 for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T18:29:25.446 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 18:29:25 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3[148816]: 2024-08-17T18:29:25.192+0000 7f453a96d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T18:29:25.446 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 18:29:25 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3[148816]: 2024-08-17T18:29:25.192+0000 7f453a96d640 -1 osd.3 1797 *** Got signal Terminated *** 2024-08-17T18:29:25.446 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 18:29:25 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3[148816]: 2024-08-17T18:29:25.192+0000 7f453a96d640 -1 osd.3 1797 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T18:29:26.011 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:25 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4[40219]: 2024-08-17T18:29:25.698+0000 7ffbdf125640 -1 osd.4 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:02.295893+0000 front 2024-08-17T18:29:02.296027+0000 (oldest deadline 2024-08-17T18:29:25.696216+0000) 2024-08-17T18:29:26.011 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:25 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:25.881+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.426891+0000 front 2024-08-17T18:29:03.426962+0000 (oldest deadline 2024-08-17T18:29:25.726741+0000) 2024-08-17T18:29:26.446 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 18:29:26 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3[148816]: 2024-08-17T18:29:26.050+0000 7f4536f82640 -1 osd.3 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:02.833557+0000 front 2024-08-17T18:29:02.833609+0000 (oldest deadline 2024-08-17T18:29:25.733806+0000) 2024-08-17T18:29:27.011 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:26 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:26.924+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.426891+0000 front 2024-08-17T18:29:03.426962+0000 (oldest deadline 2024-08-17T18:29:25.726741+0000) 2024-08-17T18:29:27.011 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:26 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4[40219]: 2024-08-17T18:29:26.708+0000 7ffbdf125640 -1 osd.4 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:02.295893+0000 front 2024-08-17T18:29:02.296027+0000 (oldest deadline 2024-08-17T18:29:25.696216+0000) 2024-08-17T18:29:27.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:26 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:26.907+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:27.282 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:27 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:27.080+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:27.446 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 18:29:27 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3[148816]: 2024-08-17T18:29:27.061+0000 7f4536f82640 -1 osd.3 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:02.833557+0000 front 2024-08-17T18:29:02.833609+0000 (oldest deadline 2024-08-17T18:29:25.733806+0000) 2024-08-17T18:29:28.011 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:27 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4[40219]: 2024-08-17T18:29:27.702+0000 7ffbdf125640 -1 osd.4 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:02.295893+0000 front 2024-08-17T18:29:02.296027+0000 (oldest deadline 2024-08-17T18:29:25.696216+0000) 2024-08-17T18:29:28.011 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:27 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:27.898+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.426891+0000 front 2024-08-17T18:29:03.426962+0000 (oldest deadline 2024-08-17T18:29:25.726741+0000) 2024-08-17T18:29:28.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:27 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:27.898+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:28.446 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 18:29:28 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3[148816]: 2024-08-17T18:29:28.070+0000 7f4536f82640 -1 osd.3 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:02.833557+0000 front 2024-08-17T18:29:02.833609+0000 (oldest deadline 2024-08-17T18:29:25.733806+0000) 2024-08-17T18:29:28.510 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:28 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:28.033+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:29.011 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:28 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4[40219]: 2024-08-17T18:29:28.733+0000 7ffbdf125640 -1 osd.4 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:02.295893+0000 front 2024-08-17T18:29:02.296027+0000 (oldest deadline 2024-08-17T18:29:25.696216+0000) 2024-08-17T18:29:29.011 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:28 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:28.939+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.426891+0000 front 2024-08-17T18:29:03.426962+0000 (oldest deadline 2024-08-17T18:29:25.726741+0000) 2024-08-17T18:29:29.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:28 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:28.863+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:29.446 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 18:29:29 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3[148816]: 2024-08-17T18:29:29.046+0000 7f4536f82640 -1 osd.3 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:02.833557+0000 front 2024-08-17T18:29:02.833609+0000 (oldest deadline 2024-08-17T18:29:25.733806+0000) 2024-08-17T18:29:29.511 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:29 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:29.062+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:30.011 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:29 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4[40219]: 2024-08-17T18:29:29.707+0000 7ffbdf125640 -1 osd.4 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:02.295893+0000 front 2024-08-17T18:29:02.296027+0000 (oldest deadline 2024-08-17T18:29:25.696216+0000) 2024-08-17T18:29:30.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:29 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:29.814+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:30.011 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:29 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:29.938+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.426891+0000 front 2024-08-17T18:29:03.426962+0000 (oldest deadline 2024-08-17T18:29:25.726741+0000) 2024-08-17T18:29:30.446 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 18:29:30 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3[148816]: 2024-08-17T18:29:30.053+0000 7f4536f82640 -1 osd.3 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:02.833557+0000 front 2024-08-17T18:29:02.833609+0000 (oldest deadline 2024-08-17T18:29:25.733806+0000) 2024-08-17T18:29:30.510 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:30 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:30.025+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:30.978 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:30 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4[40219]: 2024-08-17T18:29:30.704+0000 7ffbdf125640 -1 osd.4 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:02.295893+0000 front 2024-08-17T18:29:02.296027+0000 (oldest deadline 2024-08-17T18:29:25.696216+0000) 2024-08-17T18:29:30.978 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:30 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:30.770+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:31.142 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 18:29:30 smithi136 podman[493436]: 2024-08-17 18:29:30.85646349 +0000 UTC m=+5.765657746 container died 64f31d4557dbb44e30de087730f06fde260c03935a1d253e7d84e2357094233f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=main-cd8bdbe, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0) 2024-08-17T18:29:31.261 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:30 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:30.977+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.426891+0000 front 2024-08-17T18:29:03.426962+0000 (oldest deadline 2024-08-17T18:29:25.726741+0000) 2024-08-17T18:29:31.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:31.052+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:31.446 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 18:29:31 smithi136 podman[493436]: 2024-08-17 18:29:31.324111772 +0000 UTC m=+6.233306020 container cleanup 64f31d4557dbb44e30de087730f06fde260c03935a1d253e7d84e2357094233f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, RELEASE=main-cd8bdbe, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-17T18:29:31.446 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 18:29:31 smithi136 bash[493436]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3 2024-08-17T18:29:31.723 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 18:29:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T18:29:31.442Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.136:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.136:8765: connect: connection refused" 2024-08-17T18:29:31.723 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 18:29:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T18:29:31.444Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.136:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.136:8765: connect: connection refused" 2024-08-17T18:29:31.724 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 18:29:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T18:29:31.444Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.136:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.136:8765: connect: connection refused" 2024-08-17T18:29:31.724 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 18:29:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T18:29:31.446Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.136:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.136:8765: connect: connection refused" 2024-08-17T18:29:31.724 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 18:29:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T18:29:31.446Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.136:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.136:8765: connect: connection refused" 2024-08-17T18:29:31.724 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 18:29:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T18:29:31.446Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.136:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.136:8765: connect: connection refused" 2024-08-17T18:29:31.892 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 18:29:31 smithi136 podman[493460]: 2024-08-17 18:29:31.575707372 +0000 UTC m=+0.716157559 container remove 64f31d4557dbb44e30de087730f06fde260c03935a1d253e7d84e2357094233f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-cd8bdbe) 2024-08-17T18:29:32.011 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:31.967+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.426891+0000 front 2024-08-17T18:29:03.426962+0000 (oldest deadline 2024-08-17T18:29:25.726741+0000) 2024-08-17T18:29:32.011 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4[40219]: 2024-08-17T18:29:31.722+0000 7ffbdf125640 -1 osd.4 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:02.295893+0000 front 2024-08-17T18:29:02.296027+0000 (oldest deadline 2024-08-17T18:29:25.696216+0000) 2024-08-17T18:29:32.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:31 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:31.781+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:32.187 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 18:29:31 smithi136 podman[493550]: 2024-08-17 18:29:31.794915969 +0000 UTC m=+0.017974192 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T18:29:32.187 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 18:29:31 smithi136 podman[493550]: 2024-08-17 18:29:31.901552652 +0000 UTC m=+0.124610862 container create 66371485616c0cdcd20137d2771d9b8f1b63fa5b5e4e65d31610e4721fc287f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-08-17T18:29:32.187 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 18:29:32 smithi136 podman[493550]: 2024-08-17 18:29:32.103981369 +0000 UTC m=+0.327039592 container init 66371485616c0cdcd20137d2771d9b8f1b63fa5b5e4e65d31610e4721fc287f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-deactivate, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-cd8bdbe, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, ceph=True) 2024-08-17T18:29:32.188 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 18:29:32 smithi136 podman[493550]: 2024-08-17 18:29:32.108778565 +0000 UTC m=+0.331836776 container start 66371485616c0cdcd20137d2771d9b8f1b63fa5b5e4e65d31610e4721fc287f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-08-17T18:29:32.188 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 18:29:32 smithi136 podman[493550]: 2024-08-17 18:29:32.187221635 +0000 UTC m=+0.410279846 container attach 66371485616c0cdcd20137d2771d9b8f1b63fa5b5e4e65d31610e4721fc287f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731) 2024-08-17T18:29:32.446 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 18:29:32 smithi136 podman[493550]: 2024-08-17 18:29:32.297555764 +0000 UTC m=+0.520613985 container died 66371485616c0cdcd20137d2771d9b8f1b63fa5b5e4e65d31610e4721fc287f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-deactivate, RELEASE=main-cd8bdbe, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-08-17T18:29:32.511 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:32 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:32.024+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:32.822 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.3.service' 2024-08-17T18:29:32.845 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 18:29:32 smithi136 podman[493594]: 2024-08-17 18:29:32.807440179 +0000 UTC m=+0.504408730 container remove 66371485616c0cdcd20137d2771d9b8f1b63fa5b5e4e65d31610e4721fc287f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, RELEASE=main-cd8bdbe, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731) 2024-08-17T18:29:32.846 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 18:29:32 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.3.service: Deactivated successfully. 2024-08-17T18:29:32.846 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 18:29:32 smithi136 systemd[1]: Stopped Ceph osd.3 for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T18:29:32.846 INFO:journalctl@ceph.osd.3.smithi136.stdout:Aug 17 18:29:32 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.3.service: Consumed 38min 12.806s CPU time. 2024-08-17T18:29:32.860 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T18:29:32.861 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-08-17T18:29:32.861 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-08-17T18:29:32.861 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.4 2024-08-17T18:29:32.952 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:32 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4[40219]: 2024-08-17T18:29:32.696+0000 7ffbdf125640 -1 osd.4 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:02.295893+0000 front 2024-08-17T18:29:02.296027+0000 (oldest deadline 2024-08-17T18:29:25.696216+0000) 2024-08-17T18:29:32.952 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:32 smithi165 systemd[1]: Stopping Ceph osd.4 for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T18:29:32.952 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:32 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:32.761+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:33.261 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:32 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:32.950+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.426891+0000 front 2024-08-17T18:29:03.426962+0000 (oldest deadline 2024-08-17T18:29:25.726741+0000) 2024-08-17T18:29:33.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:33 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:33.003+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:33.262 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:33 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4[40219]: 2024-08-17T18:29:33.173+0000 7ffbe6b19640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T18:29:33.262 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:33 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4[40219]: 2024-08-17T18:29:33.173+0000 7ffbe6b19640 -1 osd.4 1797 *** Got signal Terminated *** 2024-08-17T18:29:33.262 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:33 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4[40219]: 2024-08-17T18:29:33.173+0000 7ffbe6b19640 -1 osd.4 1797 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T18:29:34.011 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:33 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4[40219]: 2024-08-17T18:29:33.728+0000 7ffbdf125640 -1 osd.4 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:02.295893+0000 front 2024-08-17T18:29:02.296027+0000 (oldest deadline 2024-08-17T18:29:25.696216+0000) 2024-08-17T18:29:34.011 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:33 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:33.930+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.426891+0000 front 2024-08-17T18:29:03.426962+0000 (oldest deadline 2024-08-17T18:29:25.726741+0000) 2024-08-17T18:29:34.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:33 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:33.735+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:34.511 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:34.017+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:35.011 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:34.929+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.426891+0000 front 2024-08-17T18:29:03.426962+0000 (oldest deadline 2024-08-17T18:29:25.726741+0000) 2024-08-17T18:29:35.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:34.779+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:35.011 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:34 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4[40219]: 2024-08-17T18:29:34.698+0000 7ffbdf125640 -1 osd.4 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:02.295893+0000 front 2024-08-17T18:29:02.296027+0000 (oldest deadline 2024-08-17T18:29:25.696216+0000) 2024-08-17T18:29:35.511 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:35 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:35.066+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:36.011 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:35 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:35.977+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.426891+0000 front 2024-08-17T18:29:03.426962+0000 (oldest deadline 2024-08-17T18:29:25.726741+0000) 2024-08-17T18:29:36.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:35 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:35.819+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:36.012 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:35 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4[40219]: 2024-08-17T18:29:35.731+0000 7ffbdf125640 -1 osd.4 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:02.295893+0000 front 2024-08-17T18:29:02.296027+0000 (oldest deadline 2024-08-17T18:29:25.696216+0000) 2024-08-17T18:29:36.012 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:35 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4[40219]: 2024-08-17T18:29:35.731+0000 7ffbdf125640 -1 osd.4 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:11.497888+0000 front 2024-08-17T18:29:11.497949+0000 (oldest deadline 2024-08-17T18:29:35.597765+0000) 2024-08-17T18:29:36.511 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:36 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:36.095+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:37.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:36 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:36.831+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:37.011 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:36 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4[40219]: 2024-08-17T18:29:36.697+0000 7ffbdf125640 -1 osd.4 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:02.295893+0000 front 2024-08-17T18:29:02.296027+0000 (oldest deadline 2024-08-17T18:29:25.696216+0000) 2024-08-17T18:29:37.011 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:36 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4[40219]: 2024-08-17T18:29:36.697+0000 7ffbdf125640 -1 osd.4 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:11.497888+0000 front 2024-08-17T18:29:11.497949+0000 (oldest deadline 2024-08-17T18:29:35.597765+0000) 2024-08-17T18:29:37.511 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:37 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:37.027+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.426891+0000 front 2024-08-17T18:29:03.426962+0000 (oldest deadline 2024-08-17T18:29:25.726741+0000) 2024-08-17T18:29:37.511 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:37 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:37.073+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:37.512 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:37 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:37.073+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:10.763387+0000 front 2024-08-17T18:29:10.763197+0000 (oldest deadline 2024-08-17T18:29:36.663042+0000) 2024-08-17T18:29:38.011 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:37 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4[40219]: 2024-08-17T18:29:37.673+0000 7ffbdf125640 -1 osd.4 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:02.295893+0000 front 2024-08-17T18:29:02.296027+0000 (oldest deadline 2024-08-17T18:29:25.696216+0000) 2024-08-17T18:29:38.011 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:37 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4[40219]: 2024-08-17T18:29:37.673+0000 7ffbdf125640 -1 osd.4 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:11.497888+0000 front 2024-08-17T18:29:11.497949+0000 (oldest deadline 2024-08-17T18:29:35.597765+0000) 2024-08-17T18:29:38.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:37 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:37.842+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:38.012 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:37 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:37.842+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:29:38.283 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:38 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:38.019+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.426891+0000 front 2024-08-17T18:29:03.426962+0000 (oldest deadline 2024-08-17T18:29:25.726741+0000) 2024-08-17T18:29:38.284 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:38 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:38.042+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:38.284 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:38 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:38.042+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:10.763387+0000 front 2024-08-17T18:29:10.763197+0000 (oldest deadline 2024-08-17T18:29:36.663042+0000) 2024-08-17T18:29:38.928 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:38 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:38.832+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:38.928 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:38 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:38.832+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:29:38.928 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:38 smithi165 podman[308205]: 2024-08-17 18:29:38.631131722 +0000 UTC m=+5.564952281 container died df366c73e1f292daabb4d2b80a81454a6894328d72f18a0a7ca1394c8d7ae7fd (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, RELEASE=reef-ced649f, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-17T18:29:39.261 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:39 smithi165 podman[308205]: 2024-08-17 18:29:39.01322394 +0000 UTC m=+5.947044489 container cleanup df366c73e1f292daabb4d2b80a81454a6894328d72f18a0a7ca1394c8d7ae7fd (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=reef-ced649f, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-08-17T18:29:39.261 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:39 smithi165 bash[308205]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4 2024-08-17T18:29:39.261 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:39.003+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.426891+0000 front 2024-08-17T18:29:03.426962+0000 (oldest deadline 2024-08-17T18:29:25.726741+0000) 2024-08-17T18:29:39.262 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:39.003+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:13.328152+0000 front 2024-08-17T18:29:13.328197+0000 (oldest deadline 2024-08-17T18:29:38.028012+0000) 2024-08-17T18:29:39.262 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:39.003+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:39.262 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:39.003+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:10.763387+0000 front 2024-08-17T18:29:10.763197+0000 (oldest deadline 2024-08-17T18:29:36.663042+0000) 2024-08-17T18:29:39.619 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:39 smithi165 podman[308224]: 2024-08-17 18:29:39.298367668 +0000 UTC m=+0.665275209 container remove df366c73e1f292daabb4d2b80a81454a6894328d72f18a0a7ca1394c8d7ae7fd (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=reef-ced649f) 2024-08-17T18:29:39.874 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:39.795+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:39.874 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:39.795+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:29:39.875 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:39 smithi165 podman[308323]: 2024-08-17 18:29:39.521835884 +0000 UTC m=+0.017724031 image pull c870753d67c2504e5ac044c83696772f8b7527a928cc764ed24eeea7fea57d6e quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408 2024-08-17T18:29:39.875 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:39 smithi165 podman[308323]: 2024-08-17 18:29:39.641014018 +0000 UTC m=+0.136902153 container create a8074e42235f5956e5398f978ee519814c6055aac4c46e8cc16093acdf1057c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=reef-ced649f, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-08-17T18:29:39.875 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:39 smithi165 podman[308323]: 2024-08-17 18:29:39.868702235 +0000 UTC m=+0.364590369 container init a8074e42235f5956e5398f978ee519814c6055aac4c46e8cc16093acdf1057c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4-deactivate, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=reef-ced649f, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-08-17T18:29:39.875 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:39 smithi165 podman[308323]: 2024-08-17 18:29:39.874256019 +0000 UTC m=+0.370144155 container start a8074e42235f5956e5398f978ee519814c6055aac4c46e8cc16093acdf1057c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.36.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=reef-ced649f, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-08-17T18:29:40.198 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:40 smithi165 podman[308323]: 2024-08-17 18:29:40.035394354 +0000 UTC m=+0.531282550 container attach a8074e42235f5956e5398f978ee519814c6055aac4c46e8cc16093acdf1057c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-ced649f, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-08-17T18:29:40.199 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:39.976+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.426891+0000 front 2024-08-17T18:29:03.426962+0000 (oldest deadline 2024-08-17T18:29:25.726741+0000) 2024-08-17T18:29:40.199 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:39 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:39.976+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:13.328152+0000 front 2024-08-17T18:29:13.328197+0000 (oldest deadline 2024-08-17T18:29:38.028012+0000) 2024-08-17T18:29:40.199 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:40 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:40.006+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:40.200 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:40 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:40.006+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:10.763387+0000 front 2024-08-17T18:29:10.763197+0000 (oldest deadline 2024-08-17T18:29:36.663042+0000) 2024-08-17T18:29:40.511 INFO:journalctl@ceph.osd.4.smithi165.stdout:Aug 17 18:29:40 smithi165 podman[308323]: 2024-08-17 18:29:40.19887835 +0000 UTC m=+0.694766490 container died a8074e42235f5956e5398f978ee519814c6055aac4c46e8cc16093acdf1057c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , RELEASE=reef-ced649f, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-08-17T18:29:40.906 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:40 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:40.749+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:40.907 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:40 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:40.749+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:29:40.930 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.4.service' 2024-08-17T18:29:40.971 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T18:29:40.971 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-08-17T18:29:40.971 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-08-17T18:29:40.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.5 2024-08-17T18:29:41.261 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:40 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:40.984+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.426891+0000 front 2024-08-17T18:29:03.426962+0000 (oldest deadline 2024-08-17T18:29:25.726741+0000) 2024-08-17T18:29:41.261 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:40 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:40.984+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:13.328152+0000 front 2024-08-17T18:29:13.328197+0000 (oldest deadline 2024-08-17T18:29:38.028012+0000) 2024-08-17T18:29:41.261 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:41 smithi165 systemd[1]: Stopping Ceph osd.5 for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T18:29:41.262 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:41.036+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:41.262 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:41.036+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:10.763387+0000 front 2024-08-17T18:29:10.763197+0000 (oldest deadline 2024-08-17T18:29:36.663042+0000) 2024-08-17T18:29:41.726 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:41.319+0000 7fedc847a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T18:29:41.726 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:41.319+0000 7fedc847a640 -1 osd.5 1797 *** Got signal Terminated *** 2024-08-17T18:29:41.727 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:41.319+0000 7fedc847a640 -1 osd.5 1797 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T18:29:42.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:41.725+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:42.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:41.725+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:29:42.011 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:41.961+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.426891+0000 front 2024-08-17T18:29:03.426962+0000 (oldest deadline 2024-08-17T18:29:25.726741+0000) 2024-08-17T18:29:42.011 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:41 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:41.961+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:13.328152+0000 front 2024-08-17T18:29:13.328197+0000 (oldest deadline 2024-08-17T18:29:38.028012+0000) 2024-08-17T18:29:42.511 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:42 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:42.019+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:42.511 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:42 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:42.019+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:10.763387+0000 front 2024-08-17T18:29:10.763197+0000 (oldest deadline 2024-08-17T18:29:36.663042+0000) 2024-08-17T18:29:42.987 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:42 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:42.689+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:42.988 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:42 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:42.689+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:29:42.988 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:42 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:42.927+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.426891+0000 front 2024-08-17T18:29:03.426962+0000 (oldest deadline 2024-08-17T18:29:25.726741+0000) 2024-08-17T18:29:42.988 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:42 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:42.927+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:13.328152+0000 front 2024-08-17T18:29:13.328197+0000 (oldest deadline 2024-08-17T18:29:38.028012+0000) 2024-08-17T18:29:43.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:42 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:42.986+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:43.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:42 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:42.986+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:10.763387+0000 front 2024-08-17T18:29:10.763197+0000 (oldest deadline 2024-08-17T18:29:36.663042+0000) 2024-08-17T18:29:44.011 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:43 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:43.958+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.426891+0000 front 2024-08-17T18:29:03.426962+0000 (oldest deadline 2024-08-17T18:29:25.726741+0000) 2024-08-17T18:29:44.011 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:43 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:43.958+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:13.328152+0000 front 2024-08-17T18:29:13.328197+0000 (oldest deadline 2024-08-17T18:29:38.028012+0000) 2024-08-17T18:29:44.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:43 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:43.728+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:44.012 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:43 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:43.728+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:29:44.012 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:43 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:43.728+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:18.155608+0000 front 2024-08-17T18:29:18.155628+0000 (oldest deadline 2024-08-17T18:29:42.855691+0000) 2024-08-17T18:29:44.511 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:44 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:44.017+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:44.511 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:44 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:44.017+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:10.763387+0000 front 2024-08-17T18:29:10.763197+0000 (oldest deadline 2024-08-17T18:29:36.663042+0000) 2024-08-17T18:29:44.989 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:44 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:44.691+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:44.989 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:44 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:44.691+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:29:44.989 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:44 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:44.691+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:18.155608+0000 front 2024-08-17T18:29:18.155628+0000 (oldest deadline 2024-08-17T18:29:42.855691+0000) 2024-08-17T18:29:45.261 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:44 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:44.988+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.426891+0000 front 2024-08-17T18:29:03.426962+0000 (oldest deadline 2024-08-17T18:29:25.726741+0000) 2024-08-17T18:29:45.261 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:44 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:44.988+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:13.328152+0000 front 2024-08-17T18:29:13.328197+0000 (oldest deadline 2024-08-17T18:29:38.028012+0000) 2024-08-17T18:29:45.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:45 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:45.011+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:45.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:45 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:45.011+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:10.763387+0000 front 2024-08-17T18:29:10.763197+0000 (oldest deadline 2024-08-17T18:29:36.663042+0000) 2024-08-17T18:29:45.262 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:45 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:45.011+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:21.964143+0000 front 2024-08-17T18:29:21.964186+0000 (oldest deadline 2024-08-17T18:29:44.263902+0000) 2024-08-17T18:29:45.995 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:45 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:45.698+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:45.995 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:45 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:45.698+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:29:45.995 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:45 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:45.698+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:18.155608+0000 front 2024-08-17T18:29:18.155628+0000 (oldest deadline 2024-08-17T18:29:42.855691+0000) 2024-08-17T18:29:46.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:46.016+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:46.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:46.016+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:10.763387+0000 front 2024-08-17T18:29:10.763197+0000 (oldest deadline 2024-08-17T18:29:36.663042+0000) 2024-08-17T18:29:46.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:46.016+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:21.964143+0000 front 2024-08-17T18:29:21.964186+0000 (oldest deadline 2024-08-17T18:29:44.263902+0000) 2024-08-17T18:29:46.261 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:45 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:45.994+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.426891+0000 front 2024-08-17T18:29:03.426962+0000 (oldest deadline 2024-08-17T18:29:25.726741+0000) 2024-08-17T18:29:46.261 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:45 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:45.994+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:13.328152+0000 front 2024-08-17T18:29:13.328197+0000 (oldest deadline 2024-08-17T18:29:38.028012+0000) 2024-08-17T18:29:46.262 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:45 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5[46420]: 2024-08-17T18:29:45.994+0000 7fedc4a8e640 -1 osd.5 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:19.128990+0000 front 2024-08-17T18:29:19.129004+0000 (oldest deadline 2024-08-17T18:29:45.028876+0000) 2024-08-17T18:29:47.011 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:46 smithi165 podman[308451]: 2024-08-17 18:29:46.866260577 +0000 UTC m=+5.667763475 container died 1131665e203acff4f0e5ced40b0f1ac78278d8f6fb8cd4050ea61de9bda7e240 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=reef-ced649f, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-17T18:29:47.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:46.713+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:47.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:46.713+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:29:47.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:46 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:46.713+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:18.155608+0000 front 2024-08-17T18:29:18.155628+0000 (oldest deadline 2024-08-17T18:29:42.855691+0000) 2024-08-17T18:29:47.269 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:47 smithi165 podman[308451]: 2024-08-17 18:29:47.267250767 +0000 UTC m=+6.068753672 container cleanup 1131665e203acff4f0e5ced40b0f1ac78278d8f6fb8cd4050ea61de9bda7e240 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=reef-ced649f, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-08-17T18:29:47.270 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:47 smithi165 bash[308451]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5 2024-08-17T18:29:47.270 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:47.017+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:47.270 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:47.017+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:10.763387+0000 front 2024-08-17T18:29:10.763197+0000 (oldest deadline 2024-08-17T18:29:36.663042+0000) 2024-08-17T18:29:47.270 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:47.017+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:21.964143+0000 front 2024-08-17T18:29:21.964186+0000 (oldest deadline 2024-08-17T18:29:44.263902+0000) 2024-08-17T18:29:47.890 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:47 smithi165 podman[308472]: 2024-08-17 18:29:47.560504348 +0000 UTC m=+0.689685168 container remove 1131665e203acff4f0e5ced40b0f1ac78278d8f6fb8cd4050ea61de9bda7e240 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, ceph=True, org.label-schema.schema-version=1.0, RELEASE=reef-ced649f, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-08-17T18:29:47.890 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:47.698+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:47.890 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:47.698+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:29:47.891 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:47.698+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:18.155608+0000 front 2024-08-17T18:29:18.155628+0000 (oldest deadline 2024-08-17T18:29:42.855691+0000) 2024-08-17T18:29:48.181 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:47.979+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:48.181 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:47.979+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:10.763387+0000 front 2024-08-17T18:29:10.763197+0000 (oldest deadline 2024-08-17T18:29:36.663042+0000) 2024-08-17T18:29:48.181 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:47 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:47.979+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:21.964143+0000 front 2024-08-17T18:29:21.964186+0000 (oldest deadline 2024-08-17T18:29:44.263902+0000) 2024-08-17T18:29:48.182 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:47 smithi165 podman[308565]: 2024-08-17 18:29:47.792290362 +0000 UTC m=+0.017422807 image pull c870753d67c2504e5ac044c83696772f8b7527a928cc764ed24eeea7fea57d6e quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408 2024-08-17T18:29:48.182 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:47 smithi165 podman[308565]: 2024-08-17 18:29:47.91124936 +0000 UTC m=+0.136381803 container create 7d28c35e10c095738b6a42e355b2e3ec68b43aa3333b036d0554486febd43d11 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=reef-ced649f, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-17T18:29:48.182 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:48 smithi165 podman[308565]: 2024-08-17 18:29:48.181014066 +0000 UTC m=+0.406146524 container init 7d28c35e10c095738b6a42e355b2e3ec68b43aa3333b036d0554486febd43d11 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, RELEASE=reef-ced649f, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-08-17T18:29:48.511 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:48 smithi165 podman[308565]: 2024-08-17 18:29:48.186567402 +0000 UTC m=+0.411699845 container start 7d28c35e10c095738b6a42e355b2e3ec68b43aa3333b036d0554486febd43d11 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=reef-ced649f) 2024-08-17T18:29:48.511 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:48 smithi165 podman[308565]: 2024-08-17 18:29:48.280851522 +0000 UTC m=+0.505983964 container attach 7d28c35e10c095738b6a42e355b2e3ec68b43aa3333b036d0554486febd43d11 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=reef-ced649f, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T18:29:48.511 INFO:journalctl@ceph.osd.5.smithi165.stdout:Aug 17 18:29:48 smithi165 podman[308565]: 2024-08-17 18:29:48.373455673 +0000 UTC m=+0.598588125 container died 7d28c35e10c095738b6a42e355b2e3ec68b43aa3333b036d0554486febd43d11 (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-ced649f, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-08-17T18:29:48.945 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:48 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:48.683+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:48.945 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:48 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:48.683+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:29:48.946 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:48 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:48.683+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:18.155608+0000 front 2024-08-17T18:29:18.155628+0000 (oldest deadline 2024-08-17T18:29:42.855691+0000) 2024-08-17T18:29:49.027 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.5.service' 2024-08-17T18:29:49.066 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T18:29:49.067 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-08-17T18:29:49.067 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-08-17T18:29:49.067 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.6 2024-08-17T18:29:49.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:48 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:48.944+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:49.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:48 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:48.944+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:10.763387+0000 front 2024-08-17T18:29:10.763197+0000 (oldest deadline 2024-08-17T18:29:36.663042+0000) 2024-08-17T18:29:49.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:48 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:48.944+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:21.964143+0000 front 2024-08-17T18:29:21.964186+0000 (oldest deadline 2024-08-17T18:29:44.263902+0000) 2024-08-17T18:29:49.262 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:49 smithi165 systemd[1]: Stopping Ceph osd.6 for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T18:29:49.761 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:49 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:49.650+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:49.761 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:49 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:49.650+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:29:49.761 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:49 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:49.650+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:18.155608+0000 front 2024-08-17T18:29:18.155628+0000 (oldest deadline 2024-08-17T18:29:42.855691+0000) 2024-08-17T18:29:49.761 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:49 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:49.430+0000 7fb1a0153640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T18:29:49.762 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:49 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:49.430+0000 7fb1a0153640 -1 osd.6 1797 *** Got signal Terminated *** 2024-08-17T18:29:49.762 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:49 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:49.430+0000 7fb1a0153640 -1 osd.6 1797 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T18:29:50.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:49 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:49.916+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:50.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:49 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:49.916+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:10.763387+0000 front 2024-08-17T18:29:10.763197+0000 (oldest deadline 2024-08-17T18:29:36.663042+0000) 2024-08-17T18:29:50.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:49 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:49.916+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:21.964143+0000 front 2024-08-17T18:29:21.964186+0000 (oldest deadline 2024-08-17T18:29:44.263902+0000) 2024-08-17T18:29:50.958 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:50 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:50.687+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:50.958 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:50 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:50.687+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:29:50.958 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:50 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:50.687+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:18.155608+0000 front 2024-08-17T18:29:18.155628+0000 (oldest deadline 2024-08-17T18:29:42.855691+0000) 2024-08-17T18:29:51.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:50 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:50.957+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:51.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:50 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:50.957+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:10.763387+0000 front 2024-08-17T18:29:10.763197+0000 (oldest deadline 2024-08-17T18:29:36.663042+0000) 2024-08-17T18:29:51.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:50 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:50.957+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:21.964143+0000 front 2024-08-17T18:29:21.964186+0000 (oldest deadline 2024-08-17T18:29:44.263902+0000) 2024-08-17T18:29:51.968 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:51 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:51.670+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:51.968 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:51 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:51.670+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:29:51.969 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:51 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:51.670+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:18.155608+0000 front 2024-08-17T18:29:18.155628+0000 (oldest deadline 2024-08-17T18:29:42.855691+0000) 2024-08-17T18:29:52.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:51 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:51.967+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:52.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:51 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:51.967+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:10.763387+0000 front 2024-08-17T18:29:10.763197+0000 (oldest deadline 2024-08-17T18:29:36.663042+0000) 2024-08-17T18:29:52.261 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:51 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:51.967+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:21.964143+0000 front 2024-08-17T18:29:21.964186+0000 (oldest deadline 2024-08-17T18:29:44.263902+0000) 2024-08-17T18:29:53.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:52 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:52.720+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:53.012 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:52 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:52.720+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:29:53.012 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:52 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:52.720+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:18.155608+0000 front 2024-08-17T18:29:18.155628+0000 (oldest deadline 2024-08-17T18:29:42.855691+0000) 2024-08-17T18:29:53.012 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:52 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:52.942+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:53.013 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:52 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:52.942+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:10.763387+0000 front 2024-08-17T18:29:10.763197+0000 (oldest deadline 2024-08-17T18:29:36.663042+0000) 2024-08-17T18:29:53.013 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:52 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:52.942+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:21.964143+0000 front 2024-08-17T18:29:21.964186+0000 (oldest deadline 2024-08-17T18:29:44.263902+0000) 2024-08-17T18:29:54.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:53 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:53.754+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:54.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:53 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:53.754+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:29:54.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:53 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:53.754+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:18.155608+0000 front 2024-08-17T18:29:18.155628+0000 (oldest deadline 2024-08-17T18:29:42.855691+0000) 2024-08-17T18:29:54.012 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:53 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:53.907+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:03.162195+0000 front 2024-08-17T18:29:03.162234+0000 (oldest deadline 2024-08-17T18:29:26.662161+0000) 2024-08-17T18:29:54.012 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:53 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:53.907+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:10.763387+0000 front 2024-08-17T18:29:10.763197+0000 (oldest deadline 2024-08-17T18:29:36.663042+0000) 2024-08-17T18:29:54.012 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:53 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6[284155]: 2024-08-17T18:29:53.907+0000 7fb19875f640 -1 osd.6 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:21.964143+0000 front 2024-08-17T18:29:21.964186+0000 (oldest deadline 2024-08-17T18:29:44.263902+0000) 2024-08-17T18:29:55.205 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:54 smithi165 podman[308692]: 2024-08-17 18:29:54.896422306 +0000 UTC m=+5.598281449 container died f0789491d7c7fae594262071cd5b815b9765d728379aa50813edd28bcba1cb5d (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-ced649f, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-08-17T18:29:55.205 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:54 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:54.782+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:55.206 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:54 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:54.782+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:29:55.206 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:54 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:54.782+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:18.155608+0000 front 2024-08-17T18:29:18.155628+0000 (oldest deadline 2024-08-17T18:29:42.855691+0000) 2024-08-17T18:29:55.497 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:55 smithi165 podman[308692]: 2024-08-17 18:29:55.295275083 +0000 UTC m=+5.997134228 container cleanup f0789491d7c7fae594262071cd5b815b9765d728379aa50813edd28bcba1cb5d (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=reef-ced649f, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0) 2024-08-17T18:29:55.497 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:55 smithi165 bash[308692]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6 2024-08-17T18:29:55.761 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:55 smithi165 podman[308716]: 2024-08-17 18:29:55.496707121 +0000 UTC m=+0.595257578 container remove f0789491d7c7fae594262071cd5b815b9765d728379aa50813edd28bcba1cb5d (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=reef-ced649f, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-08-17T18:29:56.047 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:55 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:55.783+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:56.047 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:55 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:55.783+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:29:56.048 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:55 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:55.783+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:18.155608+0000 front 2024-08-17T18:29:18.155628+0000 (oldest deadline 2024-08-17T18:29:42.855691+0000) 2024-08-17T18:29:56.048 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:55 smithi165 podman[308805]: 2024-08-17 18:29:55.810480547 +0000 UTC m=+0.017601709 image pull c870753d67c2504e5ac044c83696772f8b7527a928cc764ed24eeea7fea57d6e quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408 2024-08-17T18:29:56.048 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:55 smithi165 podman[308805]: 2024-08-17 18:29:55.931136419 +0000 UTC m=+0.138257575 container create 0f59a2c5846bb07ca0278cec0553954e8dedb25df2c54a366e1e7ee24b313d9a (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=reef-ced649f, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-08-17T18:29:56.357 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:56 smithi165 podman[308805]: 2024-08-17 18:29:56.167012363 +0000 UTC m=+0.374133537 container init 0f59a2c5846bb07ca0278cec0553954e8dedb25df2c54a366e1e7ee24b313d9a (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-ced649f, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-08-17T18:29:56.357 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:56 smithi165 podman[308805]: 2024-08-17 18:29:56.171804861 +0000 UTC m=+0.378926016 container start 0f59a2c5846bb07ca0278cec0553954e8dedb25df2c54a366e1e7ee24b313d9a (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=reef-ced649f, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0) 2024-08-17T18:29:56.357 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:56 smithi165 podman[308805]: 2024-08-17 18:29:56.258333331 +0000 UTC m=+0.465454502 container attach 0f59a2c5846bb07ca0278cec0553954e8dedb25df2c54a366e1e7ee24b313d9a (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, RELEASE=reef-ced649f, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0) 2024-08-17T18:29:56.653 INFO:journalctl@ceph.osd.6.smithi165.stdout:Aug 17 18:29:56 smithi165 podman[308805]: 2024-08-17 18:29:56.357325467 +0000 UTC m=+0.564446634 container died 0f59a2c5846bb07ca0278cec0553954e8dedb25df2c54a366e1e7ee24b313d9a (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=reef-ced649f, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-08-17T18:29:56.947 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:56 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:56.751+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:56.948 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:56 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:56.751+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:29:56.948 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:56 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:56.751+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:18.155608+0000 front 2024-08-17T18:29:18.155628+0000 (oldest deadline 2024-08-17T18:29:42.855691+0000) 2024-08-17T18:29:56.948 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:56 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:56.751+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6830 osd.3 since back 2024-08-17T18:29:29.257192+0000 front 2024-08-17T18:29:29.257143+0000 (oldest deadline 2024-08-17T18:29:56.257566+0000) 2024-08-17T18:29:56.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.6.service' 2024-08-17T18:29:57.012 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T18:29:57.012 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-08-17T18:29:57.012 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-08-17T18:29:57.012 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.7 2024-08-17T18:29:57.261 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:57 smithi165 systemd[1]: Stopping Ceph osd.7 for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T18:29:57.757 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:57 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:57.377+0000 7f63a060e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T18:29:57.757 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:57 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:57.377+0000 7f63a060e640 -1 osd.7 1797 *** Got signal Terminated *** 2024-08-17T18:29:57.757 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:57 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:57.377+0000 7f63a060e640 -1 osd.7 1797 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T18:29:57.758 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:57 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:57.755+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:57.758 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:57 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:57.755+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:29:57.758 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:57 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:57.755+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:18.155608+0000 front 2024-08-17T18:29:18.155628+0000 (oldest deadline 2024-08-17T18:29:42.855691+0000) 2024-08-17T18:29:57.758 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:57 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:57.755+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6830 osd.3 since back 2024-08-17T18:29:29.257192+0000 front 2024-08-17T18:29:29.257143+0000 (oldest deadline 2024-08-17T18:29:56.257566+0000) 2024-08-17T18:29:59.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:58 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:58.714+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:29:59.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:58 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:58.714+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:29:59.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:58 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:58.714+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:18.155608+0000 front 2024-08-17T18:29:18.155628+0000 (oldest deadline 2024-08-17T18:29:42.855691+0000) 2024-08-17T18:29:59.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:58 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:58.714+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6830 osd.3 since back 2024-08-17T18:29:29.257192+0000 front 2024-08-17T18:29:29.257143+0000 (oldest deadline 2024-08-17T18:29:56.257566+0000) 2024-08-17T18:30:00.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:59 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:59.747+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:30:00.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:59 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:59.747+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:30:00.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:59 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:59.747+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:18.155608+0000 front 2024-08-17T18:29:18.155628+0000 (oldest deadline 2024-08-17T18:29:42.855691+0000) 2024-08-17T18:30:00.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:29:59 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:29:59.747+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6830 osd.3 since back 2024-08-17T18:29:29.257192+0000 front 2024-08-17T18:29:29.257143+0000 (oldest deadline 2024-08-17T18:29:56.257566+0000) 2024-08-17T18:30:01.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:30:00 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:30:00.716+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:30:01.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:30:00 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:30:00.716+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:30:01.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:30:00 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:30:00.716+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:18.155608+0000 front 2024-08-17T18:29:18.155628+0000 (oldest deadline 2024-08-17T18:29:42.855691+0000) 2024-08-17T18:30:01.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:30:00 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:30:00.716+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6830 osd.3 since back 2024-08-17T18:29:29.257192+0000 front 2024-08-17T18:29:29.257143+0000 (oldest deadline 2024-08-17T18:29:56.257566+0000) 2024-08-17T18:30:02.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:30:01 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:30:01.752+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6806 osd.0 since back 2024-08-17T18:29:01.853585+0000 front 2024-08-17T18:29:01.853501+0000 (oldest deadline 2024-08-17T18:29:25.953363+0000) 2024-08-17T18:30:02.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:30:01 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:30:01.752+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6814 osd.1 since back 2024-08-17T18:29:12.354757+0000 front 2024-08-17T18:29:12.354687+0000 (oldest deadline 2024-08-17T18:29:37.654644+0000) 2024-08-17T18:30:02.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:30:01 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:30:01.752+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6822 osd.2 since back 2024-08-17T18:29:18.155608+0000 front 2024-08-17T18:29:18.155628+0000 (oldest deadline 2024-08-17T18:29:42.855691+0000) 2024-08-17T18:30:02.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:30:01 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:30:01.752+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.136:6830 osd.3 since back 2024-08-17T18:29:29.257192+0000 front 2024-08-17T18:29:29.257143+0000 (oldest deadline 2024-08-17T18:29:56.257566+0000) 2024-08-17T18:30:02.011 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:30:01 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7[58481]: 2024-08-17T18:30:01.752+0000 7f639cc22640 -1 osd.7 1797 heartbeat_check: no reply from 172.21.15.165:6804 osd.4 since back 2024-08-17T18:29:36.258138+0000 front 2024-08-17T18:29:36.258116+0000 (oldest deadline 2024-08-17T18:30:00.958024+0000) 2024-08-17T18:30:03.207 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:30:02 smithi165 podman[308933]: 2024-08-17 18:30:02.896848502 +0000 UTC m=+5.626785993 container died 18382e4ae3b8cb2d7a221ab611e5eeb11bb8f82e155b924b6d502b3ae221221b (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=reef-ced649f, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-08-17T18:30:03.511 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:30:03 smithi165 podman[308933]: 2024-08-17 18:30:03.291630545 +0000 UTC m=+6.021568034 container cleanup 18382e4ae3b8cb2d7a221ab611e5eeb11bb8f82e155b924b6d502b3ae221221b (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=reef-ced649f, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-08-17T18:30:03.511 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:30:03 smithi165 bash[308933]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7 2024-08-17T18:30:03.917 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:30:03 smithi165 podman[308967]: 2024-08-17 18:30:03.584666346 +0000 UTC m=+0.667755554 container remove 18382e4ae3b8cb2d7a221ab611e5eeb11bb8f82e155b924b6d502b3ae221221b (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-ced649f, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-08-17T18:30:04.205 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:30:03 smithi165 podman[309061]: 2024-08-17 18:30:03.819288309 +0000 UTC m=+0.017353537 image pull c870753d67c2504e5ac044c83696772f8b7527a928cc764ed24eeea7fea57d6e quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408 2024-08-17T18:30:04.206 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:30:03 smithi165 podman[309061]: 2024-08-17 18:30:03.918783228 +0000 UTC m=+0.116848434 container create b20a213d67eb15888f2bfeef0f83950e44f20909509b379068bdcd08d981fe0c (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=reef-ced649f) 2024-08-17T18:30:04.206 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:30:04 smithi165 podman[309061]: 2024-08-17 18:30:04.121920501 +0000 UTC m=+0.319985714 container init b20a213d67eb15888f2bfeef0f83950e44f20909509b379068bdcd08d981fe0c (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=reef-ced649f, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-08-17T18:30:04.206 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:30:04 smithi165 podman[309061]: 2024-08-17 18:30:04.12669639 +0000 UTC m=+0.324761597 container start b20a213d67eb15888f2bfeef0f83950e44f20909509b379068bdcd08d981fe0c (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7-deactivate, RELEASE=reef-ced649f, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-08-17T18:30:04.206 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:30:04 smithi165 podman[309061]: 2024-08-17 18:30:04.204978436 +0000 UTC m=+0.403043642 container attach b20a213d67eb15888f2bfeef0f83950e44f20909509b379068bdcd08d981fe0c (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, RELEASE=reef-ced649f, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-08-17T18:30:04.510 INFO:journalctl@ceph.osd.7.smithi165.stdout:Aug 17 18:30:04 smithi165 podman[309061]: 2024-08-17 18:30:04.314279137 +0000 UTC m=+0.512344343 container died b20a213d67eb15888f2bfeef0f83950e44f20909509b379068bdcd08d981fe0c (image=quay.ceph.io/ceph-ci/ceph@sha256:37f887e3bbdff4051aa7da31b6d591754a495ffdffc717f32d57523a5142a408, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=reef-ced649f, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-08-17T18:30:04.939 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@osd.7.service' 2024-08-17T18:30:04.978 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T18:30:04.978 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-08-17T18:30:04.978 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-08-17T18:30:04.979 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-8270436c-5cba-11ef-bccf-c7b262605968@prometheus.a 2024-08-17T18:30:05.261 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 18:30:05 smithi165 systemd[1]: Stopping Ceph prometheus.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T18:30:05.761 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 18:30:05 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T18:30:05.305Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-08-17T18:30:05.761 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 18:30:05 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T18:30:05.305Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-08-17T18:30:05.761 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 18:30:05 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T18:30:05.305Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-08-17T18:30:05.762 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 18:30:05 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T18:30:05.305Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-08-17T18:30:05.762 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 18:30:05 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T18:30:05.305Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-08-17T18:30:05.762 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 18:30:05 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T18:30:05.305Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-08-17T18:30:05.762 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 18:30:05 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T18:30:05.305Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-08-17T18:30:05.762 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 18:30:05 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T18:30:05.305Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-08-17T18:30:05.762 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 18:30:05 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T18:30:05.306Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-08-17T18:30:05.762 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 18:30:05 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T18:30:05.311Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-08-17T18:30:05.763 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 18:30:05 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T18:30:05.311Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-08-17T18:30:05.763 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 18:30:05 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a[91273]: ts=2024-08-17T18:30:05.311Z caller=main.go:1273 level=info msg="See you next time!" 2024-08-17T18:30:05.763 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 18:30:05 smithi165 podman[309189]: 2024-08-17 18:30:05.317177589 +0000 UTC m=+0.119249945 container died ae8f8b69fe2ffa9eabdcdb85da64ea9055fd204f3d4a8afabb23368189d35700 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T18:30:06.155 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 18:30:05 smithi165 podman[309189]: 2024-08-17 18:30:05.878422609 +0000 UTC m=+0.680494950 container cleanup ae8f8b69fe2ffa9eabdcdb85da64ea9055fd204f3d4a8afabb23368189d35700 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T18:30:06.155 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Aug 17 18:30:05 smithi165 bash[309189]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-prometheus-a 2024-08-17T18:30:06.288 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8270436c-5cba-11ef-bccf-c7b262605968@prometheus.a.service' 2024-08-17T18:30:06.322 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T18:30:06.322 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-08-17T18:30:06.322 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8270436c-5cba-11ef-bccf-c7b262605968 --force --keep-logs 2024-08-17T18:30:06.482 INFO:teuthology.orchestra.run.smithi136.stdout:Deleting cluster with fsid: 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T18:30:08.247 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 18:30:07 smithi136 systemd[1]: Stopping Ceph alertmanager.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T18:30:08.247 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 18:30:08 smithi136 ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a[104739]: ts=2024-08-17T18:30:08.232Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-08-17T18:30:08.545 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 18:30:08 smithi136 podman[493977]: 2024-08-17 18:30:08.27402913 +0000 UTC m=+0.140585149 container died c3b9b5b223ab70bb8cc2b48b8b4f8ae2dbd700fb9c45af4178846d2c47fc9ec2 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T18:30:08.946 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 18:30:08 smithi136 podman[493977]: 2024-08-17 18:30:08.744463473 +0000 UTC m=+0.611019535 container cleanup c3b9b5b223ab70bb8cc2b48b8b4f8ae2dbd700fb9c45af4178846d2c47fc9ec2 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T18:30:08.946 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 18:30:08 smithi136 bash[493977]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a 2024-08-17T18:30:09.404 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 18:30:09 smithi136 podman[493997]: 2024-08-17 18:30:09.0041949 +0000 UTC m=+0.722844497 container remove c3b9b5b223ab70bb8cc2b48b8b4f8ae2dbd700fb9c45af4178846d2c47fc9ec2 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T18:30:09.404 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 18:30:09 smithi136 podman[493997]: 2024-08-17 18:30:09.088380778 +0000 UTC m=+0.807030368 volume remove cad8abfd9b1edfc26c36e185f8586f63b536d79159c1a5deb73503a3ab385a9c 2024-08-17T18:30:09.404 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 18:30:09 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-08-17T18:30:09.404 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 18:30:09 smithi136 systemd[1]: Stopped Ceph alertmanager.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T18:30:09.404 INFO:journalctl@ceph.alertmanager.a.smithi136.stdout:Aug 17 18:30:09 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@alertmanager.a.service: Consumed 6.075s CPU time. 2024-08-17T18:30:09.662 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 18:30:09 smithi136 systemd[1]: Stopping Ceph node-exporter.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T18:30:09.941 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 18:30:09 smithi136 podman[494107]: 2024-08-17 18:30:09.662145675 +0000 UTC m=+0.117716125 container died 1e4c644ae533e78f10c35439b2f88a95d12c25e345e9429afad64f85b253c5ea (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-08-17T18:30:10.196 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 18:30:10 smithi136 podman[494107]: 2024-08-17 18:30:10.03247476 +0000 UTC m=+0.488045212 container cleanup 1e4c644ae533e78f10c35439b2f88a95d12c25e345e9429afad64f85b253c5ea (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-08-17T18:30:10.196 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 18:30:10 smithi136 bash[494107]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a 2024-08-17T18:30:10.663 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 18:30:10 smithi136 podman[494125]: 2024-08-17 18:30:10.258933354 +0000 UTC m=+0.600967442 container remove 1e4c644ae533e78f10c35439b2f88a95d12c25e345e9429afad64f85b253c5ea (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-08-17T18:30:10.663 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 18:30:10 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-08-17T18:30:10.663 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 18:30:10 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-08-17T18:30:10.663 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 18:30:10 smithi136 systemd[1]: Stopped Ceph node-exporter.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T18:30:10.664 INFO:journalctl@ceph.node-exporter.a.smithi136.stdout:Aug 17 18:30:10 smithi136 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@node-exporter.a.service: Consumed 12.284s CPU time. 2024-08-17T18:30:18.868 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8270436c-5cba-11ef-bccf-c7b262605968 --force --keep-logs 2024-08-17T18:30:19.027 INFO:teuthology.orchestra.run.smithi165.stdout:Deleting cluster with fsid: 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T18:30:20.618 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 18:30:20 smithi165 systemd[1]: Stopping Ceph grafana.a for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T18:30:20.915 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 18:30:20 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=server t=2024-08-17T18:30:20.617477132Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-08-17T18:30:20.915 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 18:30:20 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=tracing t=2024-08-17T18:30:20.617831763Z level=info msg="Closing tracing" 2024-08-17T18:30:20.915 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 18:30:20 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=ticker t=2024-08-17T18:30:20.617875037Z level=info msg=stopped last_tick=2024-08-17T18:30:20Z 2024-08-17T18:30:20.915 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 18:30:20 smithi165 ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a[96038]: logger=grafana-apiserver t=2024-08-17T18:30:20.617881363Z level=info msg="StorageObjectCountTracker pruner is exiting" 2024-08-17T18:30:20.915 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 18:30:20 smithi165 podman[309606]: 2024-08-17 18:30:20.627993699 +0000 UTC m=+0.119717376 container died 04db24fce5e9ecd97fe76d33e0bdff26ebbb2738fdce1ffc6d9eb4b3739292b0 (image=quay.io/ceph/grafana:10.4.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-08-17T18:30:21.260 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 18:30:21 smithi165 podman[309606]: 2024-08-17 18:30:21.015192144 +0000 UTC m=+0.506915884 container cleanup 04db24fce5e9ecd97fe76d33e0bdff26ebbb2738fdce1ffc6d9eb4b3739292b0 (image=quay.io/ceph/grafana:10.4.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-08-17T18:30:21.261 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 18:30:21 smithi165 bash[309606]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a 2024-08-17T18:30:21.578 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 18:30:21 smithi165 podman[309625]: 2024-08-17 18:30:21.262302634 +0000 UTC m=+0.635424368 container remove 04db24fce5e9ecd97fe76d33e0bdff26ebbb2738fdce1ffc6d9eb4b3739292b0 (image=quay.io/ceph/grafana:10.4.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-08-17T18:30:21.578 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 18:30:21 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@grafana.a.service: Deactivated successfully. 2024-08-17T18:30:21.578 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 18:30:21 smithi165 systemd[1]: Stopped Ceph grafana.a for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T18:30:21.578 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Aug 17 18:30:21 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@grafana.a.service: Consumed 50.207s CPU time. 2024-08-17T18:30:21.837 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 18:30:21 smithi165 systemd[1]: Stopping Ceph node-exporter.b for 8270436c-5cba-11ef-bccf-c7b262605968... 2024-08-17T18:30:22.466 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 18:30:21 smithi165 podman[309736]: 2024-08-17 18:30:21.836957789 +0000 UTC m=+0.115922894 container died 7af2d2d68b06803956a927505275969bec7debc0062a1c3981716600864cd82f (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-08-17T18:30:22.467 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 18:30:22 smithi165 podman[309736]: 2024-08-17 18:30:22.206868365 +0000 UTC m=+0.485833476 container cleanup 7af2d2d68b06803956a927505275969bec7debc0062a1c3981716600864cd82f (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-08-17T18:30:22.467 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 18:30:22 smithi165 bash[309736]: ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b 2024-08-17T18:30:22.761 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 18:30:22 smithi165 podman[309754]: 2024-08-17 18:30:22.466563005 +0000 UTC m=+0.633507558 container remove 7af2d2d68b06803956a927505275969bec7debc0062a1c3981716600864cd82f (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-8270436c-5cba-11ef-bccf-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-08-17T18:30:22.761 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 18:30:22 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-08-17T18:30:22.761 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 18:30:22 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-08-17T18:30:22.761 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 18:30:22 smithi165 systemd[1]: Stopped Ceph node-exporter.b for 8270436c-5cba-11ef-bccf-c7b262605968. 2024-08-17T18:30:22.761 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Aug 17 18:30:22 smithi165 systemd[1]: ceph-8270436c-5cba-11ef-bccf-c7b262605968@node-exporter.b.service: Consumed 11.512s CPU time. 2024-08-17T18:30:27.508 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-17T18:30:27.541 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-17T18:30:27.570 INFO:tasks.cephadm:Archiving crash dumps... 2024-08-17T18:30:27.572 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-08-17_03:08:15-upgrade-main-distro-default-smithi/7858334/remote/smithi136/crash 2024-08-17T18:30:27.573 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/crash -- . 2024-08-17T18:30:27.620 INFO:teuthology.orchestra.run.smithi136.stderr:tar: /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/crash: Cannot open: No such file or directory 2024-08-17T18:30:27.620 INFO:teuthology.orchestra.run.smithi136.stderr:tar: Error is not recoverable: exiting now 2024-08-17T18:30:27.622 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-08-17_03:08:15-upgrade-main-distro-default-smithi/7858334/remote/smithi165/crash 2024-08-17T18:30:27.623 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/crash -- . 2024-08-17T18:30:27.667 INFO:teuthology.orchestra.run.smithi165.stderr:tar: /var/lib/ceph/8270436c-5cba-11ef-bccf-c7b262605968/crash: Cannot open: No such file or directory 2024-08-17T18:30:27.667 INFO:teuthology.orchestra.run.smithi165.stderr:tar: Error is not recoverable: exiting now 2024-08-17T18:30:27.668 INFO:tasks.cephadm:Checking cluster log for badness... 2024-08-17T18:30:27.669 DEBUG:teuthology.orchestra.run.smithi136:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v MON_DOWN | egrep -v 'out of quorum' | egrep -v PG_AVAILABILITY | head -n 1 2024-08-17T18:30:27.767 INFO:teuthology.orchestra.run.smithi136.stdout:2024-08-17T17:08:49.854552+0000 mon.a (mon.0) 653 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T18:30:27.772 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-08-17T18:30:27.772 INFO:tasks.cephadm:Compressing logs... 2024-08-17T18:30:27.772 DEBUG:teuthology.orchestra.run.smithi136:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-17T18:30:27.775 DEBUG:teuthology.orchestra.run.smithi165:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-17T18:30:27.819 INFO:teuthology.orchestra.run.smithi165.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-08-17T18:30:27.819 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph.log 2024-08-17T18:30:27.819 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph.audit.log 2024-08-17T18:30:27.820 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-osd.7.log 2024-08-17T18:30:27.820 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-osd.5.log 2024-08-17T18:30:27.820 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-osd.4.log 2024-08-17T18:30:27.820 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-mgr.x.log 2024-08-17T18:30:27.820 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-osd.5.log: /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-osd.6.log 2024-08-17T18:30:27.820 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph.cephadm.log 2024-08-17T18:30:27.820 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-volume.log 2024-08-17T18:30:27.820 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-mon.b.log 2024-08-17T18:30:27.821 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-mds.foo.smithi165.bmrbhh.log 2024-08-17T18:30:27.821 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-08-17T18:30:27.821 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-mds.foo.smithi165.bmrbhh.log: 84.3% -- replaced with /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-mds.foo.smithi165.bmrbhh.log.gz 2024-08-17T18:30:27.835 INFO:teuthology.orchestra.run.smithi136.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-08-17T18:30:27.835 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph.log 2024-08-17T18:30:27.835 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-osd.2.log 2024-08-17T18:30:27.835 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-mds.foo.smithi136.ltsuqd.log 2024-08-17T18:30:27.835 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph.log: /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph.audit.log 2024-08-17T18:30:27.838 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-mds.foo.smithi136.ltsuqd.log: gzip -5 --verbose -- /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-osd.3.log 2024-08-17T18:30:27.838 INFO:teuthology.orchestra.run.smithi136.stderr: 85.8% -- replaced with /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-mds.foo.smithi136.ltsuqd.log.gz 2024-08-17T18:30:27.838 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-osd.1.log 2024-08-17T18:30:27.838 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-mgr.y.log 2024-08-17T18:30:27.838 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-osd.0.log 2024-08-17T18:30:27.839 INFO:teuthology.orchestra.run.smithi136.stderr:gzip/var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-mgr.y.log: -5 --verbose -- /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-mon.c.log 2024-08-17T18:30:27.839 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-mon.a.log 2024-08-17T18:30:27.839 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph.cephadm.log 2024-08-17T18:30:27.839 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-volume.log 2024-08-17T18:30:27.839 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-08-17T18:30:27.853 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-volume.log: /var/log/ceph/cephadm.log: 92.8% -- replaced with /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph.log.gz 2024-08-17T18:30:27.963 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/cephadm.log: 90.7% -- replaced with /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph.audit.log.gz 2024-08-17T18:30:28.002 INFO:teuthology.orchestra.run.smithi165.stderr: 85.9% -- replaced with /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph.cephadm.log.gz 2024-08-17T18:30:28.058 INFO:teuthology.orchestra.run.smithi165.stderr: 87.2% -- replaced with /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph.log.gz 2024-08-17T18:30:28.100 INFO:teuthology.orchestra.run.smithi136.stderr: 94.4% -- replaced with /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph.audit.log.gz 2024-08-17T18:30:28.100 INFO:teuthology.orchestra.run.smithi136.stderr: 92.0% -- replaced with /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph.cephadm.log.gz 2024-08-17T18:30:28.155 INFO:teuthology.orchestra.run.smithi165.stderr: 90.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-08-17T18:30:28.307 INFO:teuthology.orchestra.run.smithi165.stderr: 91.3% -- replaced with /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-volume.log.gz 2024-08-17T18:30:28.409 INFO:teuthology.orchestra.run.smithi136.stderr: 91.2% -- replaced with /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-volume.log.gz 2024-08-17T18:30:28.427 INFO:teuthology.orchestra.run.smithi136.stderr: 95.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-08-17T18:30:28.716 INFO:teuthology.orchestra.run.smithi165.stderr: 90.4% -- replaced with /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-mgr.x.log.gz 2024-08-17T18:30:36.469 INFO:teuthology.orchestra.run.smithi165.stderr: 92.4% -- replaced with /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-mon.b.log.gz 2024-08-17T18:30:43.799 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-mgr.y.log.gz 2024-08-17T18:30:58.754 INFO:teuthology.orchestra.run.smithi136.stderr: 91.6% -- replaced with /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-mon.c.log.gz 2024-08-17T18:31:33.923 INFO:teuthology.orchestra.run.smithi136.stderr: 90.5% -- replaced with /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-mon.a.log.gz 2024-08-17T18:41:15.759 INFO:teuthology.orchestra.run.smithi165.stderr: 94.2% -- replaced with /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-osd.7.log.gz 2024-08-17T18:44:17.351 INFO:teuthology.orchestra.run.smithi165.stderr: 92.8% -- replaced with /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-osd.5.log.gz 2024-08-17T18:49:47.387 INFO:teuthology.orchestra.run.smithi165.stderr: 93.2% -- replaced with /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-osd.4.log.gz 2024-08-17T18:50:31.098 INFO:teuthology.orchestra.run.smithi165.stderr: 93.0% -- replaced with /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-osd.6.log.gz 2024-08-17T18:50:31.100 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-08-17T18:50:31.100 INFO:teuthology.orchestra.run.smithi165.stderr:real 20m3.310s 2024-08-17T18:50:31.100 INFO:teuthology.orchestra.run.smithi165.stderr:user 8m7.684s 2024-08-17T18:50:31.100 INFO:teuthology.orchestra.run.smithi165.stderr:sys 0m25.187s 2024-08-17T19:04:25.389 INFO:teuthology.orchestra.run.smithi136.stderr: 92.1% -- replaced with /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-osd.0.log.gz 2024-08-17T19:09:17.226 INFO:teuthology.orchestra.run.smithi136.stderr: 92.7% -- replaced with /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-osd.3.log.gz 2024-08-17T19:11:14.047 INFO:teuthology.orchestra.run.smithi136.stderr: 92.3% -- replaced with /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-osd.2.log.gz 2024-08-17T19:11:14.604 INFO:teuthology.orchestra.run.smithi136.stderr: 92.4% -- replaced with /var/log/ceph/8270436c-5cba-11ef-bccf-c7b262605968/ceph-osd.1.log.gz 2024-08-17T19:11:14.649 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-08-17T19:11:14.649 INFO:teuthology.orchestra.run.smithi136.stderr:real 40m46.860s 2024-08-17T19:11:14.649 INFO:teuthology.orchestra.run.smithi136.stderr:user 12m38.901s 2024-08-17T19:11:14.650 INFO:teuthology.orchestra.run.smithi136.stderr:sys 0m37.653s 2024-08-17T19:11:14.650 INFO:tasks.cephadm:Archiving logs... 2024-08-17T19:11:14.652 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-08-17_03:08:15-upgrade-main-distro-default-smithi/7858334/remote/smithi136/log 2024-08-17T19:11:14.654 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-17T19:13:14.666 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-08-17_03:08:15-upgrade-main-distro-default-smithi/7858334/remote/smithi165/log 2024-08-17T19:13:14.668 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-17T19:14:30.234 INFO:tasks.cephadm:Removing cluster... 2024-08-17T19:14:30.235 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8270436c-5cba-11ef-bccf-c7b262605968 --force 2024-08-17T19:14:31.224 INFO:teuthology.orchestra.run.smithi136.stdout:Deleting cluster with fsid: 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T19:14:32.253 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8270436c-5cba-11ef-bccf-c7b262605968 --force 2024-08-17T19:14:32.680 INFO:teuthology.orchestra.run.smithi165.stdout:Deleting cluster with fsid: 8270436c-5cba-11ef-bccf-c7b262605968 2024-08-17T19:14:33.497 INFO:tasks.cephadm:Removing cephadm ... 2024-08-17T19:14:33.497 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf /home/ubuntu/cephtest/cephadm 2024-08-17T19:14:33.516 DEBUG:teuthology.orchestra.run.smithi165:> rm -rf /home/ubuntu/cephtest/cephadm 2024-08-17T19:14:33.532 INFO:tasks.cephadm:Teardown complete 2024-08-17T19:14:33.532 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-08-17T19:14:33.543 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-08-17T19:14:33.543 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-17T19:14:33.559 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-17T19:14:33.688 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-08-17T19:14:33.698 INFO:teuthology.task.clock:Checking final clock skew... 2024-08-17T19:14:33.698 DEBUG:teuthology.orchestra.run.smithi136:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-17T19:14:33.701 DEBUG:teuthology.orchestra.run.smithi165:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-17T19:14:33.716 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-08-17T19:14:33.716 INFO:teuthology.orchestra.run.smithi165.stderr:bash: line 1: ntpq: command not found 2024-08-17T19:14:33.737 INFO:teuthology.orchestra.run.smithi165.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-17T19:14:33.738 INFO:teuthology.orchestra.run.smithi165.stdout:=============================================================================== 2024-08-17T19:14:33.738 INFO:teuthology.orchestra.run.smithi165.stdout:^* hv01.front.sepia.ceph.com 3 6 377 20 +281us[ +280us] +/- 41ms 2024-08-17T19:14:33.738 INFO:teuthology.orchestra.run.smithi165.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 19 -487us[ -487us] +/- 62ms 2024-08-17T19:14:33.738 INFO:teuthology.orchestra.run.smithi165.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 +170us[ +170us] +/- 59ms 2024-08-17T19:14:33.738 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-08-17T19:14:33.785 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-17T19:14:33.786 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-08-17T19:14:33.786 INFO:teuthology.orchestra.run.smithi136.stdout:^* hv01.front.sepia.ceph.com 3 6 377 19 +270us[ +276us] +/- 41ms 2024-08-17T19:14:33.786 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 18 -458us[ -458us] +/- 62ms 2024-08-17T19:14:33.786 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 24 +110us[ +115us] +/- 59ms 2024-08-17T19:14:33.786 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-08-17T19:14:33.803 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-08-17T19:14:33.812 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-08-17T19:14:33.813 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-08-17T19:14:33.822 DEBUG:teuthology.orchestra.run.smithi136:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-17T19:14:33.959 DEBUG:teuthology.orchestra.run.smithi165:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-17T19:14:34.126 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-08-17T19:14:34.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T19:14:34.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-08-17T19:14:34.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T19:14:34.189 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-08-17T19:14:34.200 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-08-17T19:14:34.208 INFO:teuthology.task.internal:Duration was 8320.178449 seconds 2024-08-17T19:14:34.208 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-08-17T19:14:34.217 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-08-17T19:14:34.217 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-17T19:14:34.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-17T19:14:34.277 INFO:teuthology.orchestra.run.smithi165.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-17T19:14:34.278 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-17T19:14:34.812 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-08-17T19:14:34.812 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi136.front.sepia.ceph.com 2024-08-17T19:14:34.812 DEBUG:teuthology.orchestra.run.smithi136:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-17T19:14:34.841 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi165.front.sepia.ceph.com 2024-08-17T19:14:34.841 DEBUG:teuthology.orchestra.run.smithi165:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-17T19:14:34.873 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-08-17T19:14:34.874 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-17T19:14:34.884 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-17T19:14:34.947 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-08-17T19:14:34.947 DEBUG:teuthology.orchestra.run.smithi136:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-17T19:14:34.966 DEBUG:teuthology.orchestra.run.smithi165:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-17T19:14:36.895 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-08-17T19:14:36.906 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-08-17T19:14:36.906 DEBUG:teuthology.orchestra.run.smithi136:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-17T19:14:36.940 DEBUG:teuthology.orchestra.run.smithi165:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-17T19:14:37.074 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-08-17T19:14:37.085 DEBUG:teuthology.orchestra.run.smithi136:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-17T19:14:37.087 DEBUG:teuthology.orchestra.run.smithi165:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-17T19:14:37.119 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = core 2024-08-17T19:14:37.126 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = core 2024-08-17T19:14:37.152 DEBUG:teuthology.orchestra.run.smithi136:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-17T19:14:37.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T19:14:37.206 DEBUG:teuthology.orchestra.run.smithi165:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-17T19:14:37.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T19:14:37.223 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-08-17T19:14:37.234 INFO:teuthology.task.internal:Transferring archived files... 2024-08-17T19:14:37.235 DEBUG:teuthology.misc:Transferring archived files from smithi136:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-08-17_03:08:15-upgrade-main-distro-default-smithi/7858334/remote/smithi136 2024-08-17T19:14:37.235 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-17T19:14:37.328 DEBUG:teuthology.misc:Transferring archived files from smithi165:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-08-17_03:08:15-upgrade-main-distro-default-smithi/7858334/remote/smithi165 2024-08-17T19:14:37.329 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-17T19:14:37.384 INFO:teuthology.task.internal:Removing archive directory... 2024-08-17T19:14:37.384 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-17T19:14:37.386 DEBUG:teuthology.orchestra.run.smithi165:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-17T19:14:37.429 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-08-17T19:14:37.441 INFO:teuthology.task.internal:Not uploading archives. 2024-08-17T19:14:37.441 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-08-17T19:14:37.535 INFO:teuthology.task.internal:Tidying up after the test... 2024-08-17T19:14:37.536 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-17T19:14:37.538 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-17T19:14:37.556 INFO:teuthology.orchestra.run.smithi165.stdout: 265318 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 17 19:14 /home/ubuntu/cephtest 2024-08-17T19:14:37.562 INFO:teuthology.orchestra.run.smithi136.stdout: 265838 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 17 19:14 /home/ubuntu/cephtest 2024-08-17T19:14:37.563 INFO:teuthology.orchestra.run.smithi136.stdout: 672067 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 17 18:26 /home/ubuntu/cephtest/mnt.0 2024-08-17T19:14:37.563 INFO:teuthology.orchestra.run.smithi136.stdout: 672085 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 17 18:26 /home/ubuntu/cephtest/mnt.0/client.0 2024-08-17T19:14:37.563 INFO:teuthology.orchestra.run.smithi136.stdout: 686330 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 17 18:26 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-17T19:14:37.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T19:14:37.564 INFO:teuthology.orchestra.run.smithi136.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-08-17T19:14:37.564 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi136 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-08-17T19:14:37.571 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-08-17T19:14:37.583 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-08-17T19:14:37.594 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi136 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-08-17T19:14:37.598 INFO:teuthology.run:Summary data: description: upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} duration: 8320.178449392319 failure_reason: 'Command failed (workunit test cls/test_cls_rbd.sh) on smithi136 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=1a5e0a04a189467f98b2822546a1a0c7 status: fail success: false 2024-08-17T19:14:37.599 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-17T19:14:37.711 INFO:teuthology.run:FAIL